2024-09-13T16:56:48.116 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T16:56:48.149 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:56:48.261 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869 branch: wip-rishabh-testing-20240910.100730-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} email: null first_in_suite: false job_id: '7903869' kernel: client: branch: testing kdb: 1 sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240910.100730-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds cache max size: 1000 mds debug frag: true mds debug scatterstat: true mds log max segments: 1 mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon warn on pool no app: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - Replacing daemon mds - Scrub error on inode - Behind on trimming - Metadata damage detected - bad backtrace on inode - overall HEALTH_ - \(MDS_TRIM\) sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240910.100730-reef sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 6439 sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 sleep_before_teardown: 0 subset: 964/1024 suite: fs suite_branch: wip-rishabh-testing-20240910.100730-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOsybv/7bBp+DjNREGwjWjG0yl+b5ZsEiopw5t4b1vNIOyDWeUbGD0sCEh7te8WSOoLvJVwm+4yfjJZopHMa5o= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPY6Rv1zqe55AvVQUhAISV60J/AzOPWUgUQv63hjfij+9CuWxRR0evY2NPbA2CmmneeE0yyMcMcVRhefvy8X/A= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_scrub_checks - tasks.cephfs.test_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_14:42:16 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T16:56:48.261 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/qa; will attempt to use it 2024-09-13T16:56:48.262 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/qa/tasks 2024-09-13T16:56:48.262 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T16:56:48.263 INFO:teuthology.task.internal:Checking packages... 2024-09-13T16:56:48.287 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3' 2024-09-13T16:56:48.287 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T16:56:48.287 INFO:teuthology.packaging:ref: None 2024-09-13T16:56:48.287 INFO:teuthology.packaging:tag: None 2024-09-13T16:56:48.287 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240910.100730-reef 2024-09-13T16:56:48.287 INFO:teuthology.packaging:sha1: 6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 2024-09-13T16:56:48.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240910.100730-reef 2024-09-13T16:56:48.440 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-860.g6c6b083f 2024-09-13T16:56:48.443 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T16:56:48.450 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T16:56:48.450 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T16:56:48.474 INFO:teuthology.task.internal:Saving configuration 2024-09-13T16:56:48.492 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T16:56:48.500 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T16:56:48.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:51:34.569070', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOsybv/7bBp+DjNREGwjWjG0yl+b5ZsEiopw5t4b1vNIOyDWeUbGD0sCEh7te8WSOoLvJVwm+4yfjJZopHMa5o='} 2024-09-13T16:56:48.562 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:51:34.567153', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPY6Rv1zqe55AvVQUhAISV60J/AzOPWUgUQv63hjfij+9CuWxRR0evY2NPbA2CmmneeE0yyMcMcVRhefvy8X/A='} 2024-09-13T16:56:48.562 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T16:56:48.570 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-13T16:56:48.570 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-13T16:56:48.570 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T16:56:48.650 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7fabdd3be0>, signals=[15]) 2024-09-13T16:56:48.650 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T16:56:48.657 INFO:teuthology.task.internal:Opening connections... 2024-09-13T16:56:48.657 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T16:56:48.658 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:56:48.745 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T16:56:48.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:56:48.828 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T16:56:48.834 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-09-13T16:56:48.853 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-09-13T16:56:48.853 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-09-13T16:56:48.911 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T16:56:48.912 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T16:56:48.913 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-09-13T16:56:48.944 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-09-13T16:56:48.963 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-09-13T16:56:48.964 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-09-13T16:56:49.022 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-13T16:56:49.023 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-13T16:56:49.024 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-09-13T16:56:49.053 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T16:56:49.062 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T16:56:49.088 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T16:56:49.088 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:56:49.091 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:56:49.108 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T16:56:49.115 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T16:56:49.115 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:56:49.148 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:56:49.241 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T16:56:49.308 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T16:56:49.324 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'} 2024-09-13T16:56:49.324 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-09-13T16:56:49.324 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-09-13T16:56:49.324 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:56:49.325 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:56:49.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:56:49.342 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-13T16:56:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:56:49.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-09-13T16:56:49.539 DEBUG:teuthology.task.kernel:sha1 for client.0 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:56:49.540 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-13T16:56:49.540 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-13T16:56:49.543 DEBUG:teuthology.task.kernel:sha1 for client.1 is 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:56:49.543 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "2ca44a2f35ca8f36983aaaa46d49312239ef519a"... 2024-09-13T16:56:49.543 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-13T16:56:49.557 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-503.el9.x86_64 2024-09-13T16:56:49.558 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:56:49.558 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-13T16:56:49.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-09-13T16:56:49.561 INFO:teuthology.orchestra.run.smithi189.stdout:5.14.0-503.el9.x86_64 2024-09-13T16:56:49.561 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-503.el9.x86_64 vs 2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:56:49.562 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-09-13T16:56:49.562 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum upgrade -y linux-firmware 2024-09-13T16:56:52.730 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.2 MB 00:01 2024-09-13T16:56:52.753 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 5.7 MB/s | 8.2 MB 00:01 2024-09-13T16:56:54.431 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 37 MB/s | 20 MB 00:00 2024-09-13T16:56:55.837 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-13T16:56:58.749 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-13T16:57:00.241 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-13T16:57:01.321 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-13T16:57:01.470 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T16:57:02.714 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 100 kB/s | 19 kB 00:00 2024-09-13T16:57:03.768 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-13T16:57:07.076 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-13T16:57:09.248 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T16:57:09.252 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-13T16:57:09.253 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-09-13T16:57:09.253 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-13T16:57:09.352 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T16:57:09.353 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:57:09.517 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-13T16:57:09.517 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-13T16:57:09.520 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-09-13T16:57:11.338 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-09-13T16:57:11.339 INFO:teuthology.orchestra.run.smithi189.stdout:Nothing to do. 2024-09-13T16:57:11.339 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-09-13T16:57:11.429 INFO:teuthology.task.kernel:Downloading kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-13T16:57:11.429 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2ca44a2f35ca8f36983aaaa46d49312239ef519a 2024-09-13T16:57:11.588 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ 2024-09-13T16:57:11.588 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/ --input-file=- 2024-09-13T16:57:11.591 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi189.front.sepia.ceph.com to complete... 2024-09-13T16:57:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13 16:57:18 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-13T16:57:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-09-13 16:57:18-- 2024-09-13T16:57:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 9.1s 2024-09-13T16:57:18.621 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 343M in 8.8s (39.1 MB/s) 2024-09-13T16:57:18.624 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.0... 2024-09-13T16:57:18.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-13T16:57:20.868 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-09-13T16:57:20.977 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-09-13T16:57:21.093 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-09-13T16:57:21.751 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13 16:57:21 URL:https://3.chacra.ceph.com/r/kernel/testing/2ca44a2f35ca8f36983aaaa46d49312239ef519a/centos/9/flavors/default/x86_64/kernel-6.11.0_rc5_g2ca44a2f35ca-1.x86_64.rpm [359633503/359633503] -> "/tmp/kernel.x86_64.rpm" [1] 2024-09-13T16:57:21.751 INFO:teuthology.orchestra.run.smithi189.stderr:FINISHED --2024-09-13 16:57:21-- 2024-09-13T16:57:21.751 INFO:teuthology.orchestra.run.smithi189.stderr:Total wall clock time: 10s 2024-09-13T16:57:21.751 INFO:teuthology.orchestra.run.smithi189.stderr:Downloaded: 1 files, 343M in 9.7s (35.2 MB/s) 2024-09-13T16:57:21.753 INFO:teuthology.task.kernel:Installing kernel 2ca44a2f35ca8f36983aaaa46d49312239ef519a on client.1... 2024-09-13T16:57:21.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-09-13T16:57:23.885 INFO:teuthology.orchestra.run.smithi189.stdout:Verifying... ######################################## 2024-09-13T16:57:24.117 INFO:teuthology.orchestra.run.smithi189.stdout:Preparing... ######################################## 2024-09-13T16:57:24.230 INFO:teuthology.orchestra.run.smithi189.stdout:Updating / installing... 2024-09-13T16:57:29.413 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-13T16:57:32.216 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-6.11.0_rc5_g2ca44a2f35ca-1 ######################################## 2024-09-13T16:58:08.973 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-13T16:58:08.973 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-13T16:58:09.060 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-13T16:58:09.061 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-13T16:58:09.062 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-13T16:58:09.063 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-13T16:58:09.064 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-13T16:58:09.065 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-13T16:58:09.066 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-13T16:58:09.067 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-13T16:58:09.068 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-13T16:58:09.069 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-13T16:58:09.070 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-13T16:58:09.071 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-13T16:58:09.072 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-13T16:58:09.073 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-13T16:58:09.074 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-13T16:58:09.075 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-13T16:58:09.076 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-13T16:58:09.077 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-13T16:58:09.078 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-13T16:58:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-13T16:58:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-13T16:58:09.081 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-13T16:58:09.082 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-13T16:58:09.083 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-13T16:58:09.084 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-13T16:58:09.085 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-13T16:58:09.086 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-13T16:58:09.087 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-13T16:58:09.088 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-13T16:58:09.089 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-13T16:58:09.090 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-13T16:58:09.091 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-13T16:58:09.092 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-13T16:58:09.093 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-13T16:58:09.094 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-13T16:58:09.095 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-13T16:58:09.096 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-13T16:58:09.097 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-13T16:58:09.098 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-13T16:58:09.099 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-13T16:58:09.100 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-13T16:58:09.101 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-13T16:58:09.102 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-13T16:58:09.103 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-13T16:58:09.104 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-13T16:58:09.105 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-13T16:58:09.106 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-13T16:58:09.107 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-13T16:58:09.108 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-13T16:58:09.109 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-13T16:58:09.110 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-13T16:58:09.111 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-13T16:58:09.112 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-13T16:58:09.113 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-13T16:58:09.114 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-13T16:58:09.115 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-13T16:58:09.116 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-13T16:58:09.117 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-13T16:58:09.118 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-13T16:58:09.119 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-13T16:58:09.120 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-13T16:58:09.121 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-13T16:58:09.122 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-13T16:58:09.123 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-13T16:58:09.124 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-13T16:58:09.125 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-13T16:58:09.126 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-13T16:58:09.127 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-13T16:58:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-13T16:58:09.129 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-13T16:58:09.130 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-13T16:58:09.131 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-13T16:58:09.132 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-13T16:58:09.133 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-13T16:58:09.134 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-13T16:58:09.135 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-13T16:58:09.136 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-13T16:58:09.137 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-13T16:58:09.138 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-13T16:58:09.139 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-13T16:58:09.140 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-13T16:58:09.141 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-13T16:58:09.142 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-13T16:58:09.143 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-13T16:58:09.144 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-13T16:58:09.145 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-13T16:58:09.146 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-13T16:58:09.147 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-13T16:58:09.148 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-13T16:58:09.149 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-13T16:58:09.150 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-13T16:58:09.151 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-13T16:58:09.152 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-13T16:58:09.153 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-13T16:58:09.154 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-13T16:58:09.155 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-13T16:58:09.156 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-13T16:58:09.157 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-13T16:58:09.158 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-13T16:58:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-13T16:58:09.160 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-13T16:58:09.161 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-13T16:58:09.162 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-13T16:58:09.163 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-13T16:58:09.164 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-13T16:58:09.165 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-13T16:58:09.166 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-13T16:58:09.167 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-13T16:58:09.168 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-13T16:58:09.169 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-13T16:58:09.170 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-13T16:58:09.171 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-13T16:58:09.172 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-13T16:58:09.173 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-13T16:58:09.174 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-13T16:58:09.175 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-13T16:58:09.176 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-13T16:58:09.177 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-13T16:58:09.178 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-13T16:58:09.179 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-13T16:58:09.180 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-13T16:58:09.181 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-13T16:58:09.182 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-13T16:58:09.183 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-13T16:58:09.184 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-13T16:58:09.185 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-13T16:58:09.186 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-13T16:58:09.187 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-13T16:58:09.188 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-13T16:58:09.189 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-13T16:58:09.190 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-13T16:58:09.191 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-13T16:58:09.192 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-13T16:58:09.193 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-13T16:58:09.194 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-13T16:58:09.195 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-13T16:58:09.196 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-13T16:58:09.197 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-13T16:58:09.198 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-13T16:58:09.199 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-13T16:58:09.199 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-13T16:58:09.199 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-13T16:58:09.199 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:58:09.203 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.203 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-13T16:58:09.226 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.227 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-09-13T16:58:09.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 82.el9 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-09-13T16:58:09.301 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8274670 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:58:09.302 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T16:58:09.303 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T16:58:09.303 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T16:58:09.303 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-09-13T16:58:09.303 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T16:58:09.303 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T16:58:09.304 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T16:58:09.304 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:09.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T16:58:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-09-13T16:58:11.417 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T16:58:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-09-13T16:58:11.441 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-13T16:58:11.457 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.sQFegY9Zr6 2024-09-13T16:58:11.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sQFegY9Zr6 2024-09-13T16:58:11.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.sQFegY9Zr6 2024-09-13T16:58:11.672 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.sQFegY9Zr6 is 6KB 2024-09-13T16:58:11.722 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.sQFegY9Zr6 2024-09-13T16:58:11.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T16:58:11.799 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T16:58:11.799 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T16:58:11.800 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T16:58:11.800 INFO:teuthology.orchestra.run.smithi028.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-13T16:58:11.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:11.873 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi189.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-09-13T16:58:11.873 DEBUG:teuthology.orchestra.run.smithi189:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-09-13T16:58:11.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-09-13T16:58:11.978 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/System.map-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:11.978 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/config-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:11.978 INFO:teuthology.orchestra.run.smithi189.stdout:/boot/vmlinuz-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:11.978 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:11.978 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/System.map 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/config 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-09-13T16:58:11.979 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crc32c-intel.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-09-13T16:58:11.980 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-avx2.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-09-13T16:58:11.981 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-amd.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm-intel.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/arch/x86/kvm/kvm.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ansi_cprng.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/anubis.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/arc4.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_memcpy.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_pq.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_raid6_recov.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_tx.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/async_xor.ko 2024-09-13T16:58:11.982 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/async_tx/raid6test.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authenc.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/authencesn.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blake2b_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_common.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/blowfish_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/camellia_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast5_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast6_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cast_common.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ccm.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cmac.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crc32_generic.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_engine.ko 2024-09-13T16:58:11.983 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/crypto_user.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/cts.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/des_generic.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/dh_generic.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecc.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/ecdh_generic.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/echainiv.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/essiv.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/fcrypt.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/khazad.ko 2024-09-13T16:58:11.984 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/lz4hc.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/md4.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/michael_mic.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcbc.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/pcrypt.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/rmd160.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/seed.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/serpent_generic.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tcrypt.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/tea.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_common.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/twofish_generic.ko 2024-09-13T16:58:11.985 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/vmac.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/wp512.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xcbc.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xor.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/crypto/xxhash_generic.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_ipmi.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/acpi_pad.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/ec_sys.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/platform_profile.ko 2024-09-13T16:58:11.986 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbs.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/sbshc.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/acpi/video.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/acard-ahci.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ahci_platform.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/ata_generic.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/libahci_platform.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_acpi.ko 2024-09-13T16:58:11.987 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ali.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_amd.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_artop.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atiixp.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_atp867x.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd640.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cmd64x.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_cypress.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_efar.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt366.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt37x.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x2n.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_hpt3x3.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it8213.ko 2024-09-13T16:58:11.988 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_it821x.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_jmicron.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_marvell.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_mpiix.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_netcell.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ninja32.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87410.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_ns87415.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_oldpiix.ko 2024-09-13T16:58:11.989 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_opti.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_optidma.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pcmcia.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc2027x.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_piccolo.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_rdc.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sch.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_serverworks.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sil680.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sis.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_sl82c105.ko 2024-09-13T16:58:11.990 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_triflex.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pata_via.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/pdc_adma.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_inic162x.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_mv.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_nv.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_promise.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_qstor.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sil24.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sis.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_svw.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_sx4.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_uli.ko 2024-09-13T16:58:11.991 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_via.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ata/sata_vsc.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/auxdisplay/ks0108.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-i2c.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/base/regmap/regmap-mmio.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/bcma/bcma.ko 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe 2024-09-13T16:58:11.992 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/aoe/aoe.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/brd.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/drbd/drbd.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/floppy.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/loop.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/nbd.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/null_blk/null_blk.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/pktcdvd.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/rbd.ko 2024-09-13T16:58:11.993 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/virtio_blk.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/xen-blkfront.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/block/zram/zram.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hangcheck-timer.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/amd-rng.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/intel-rng.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/via-rng.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/hw_random/virtio-rng.ko 2024-09-13T16:58:11.994 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_si.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/lp.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/mwave/mwave.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/ppdev.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tlclk.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm.ko 2024-09-13T16:58:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_atmel.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_infineon.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_nsc.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/uv_mmtimer.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/char/virtio_console.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/p4-clockmod.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-09-13T16:58:11.996 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/powernow-k8.ko 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/cpufreq/speedstep-lib.ko 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/ccp/ccp.ko 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat 2024-09-13T16:58:11.997 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-aes.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/padlock-sha.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dca/dca.ko 2024-09-13T16:58:11.998 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/dw/dw_dmac.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/dma/ioat/ioatdma.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/amd64_edac.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/e752x_edac.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/edac_mce_amd.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3000_edac.ko 2024-09-13T16:58:11.999 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i3200_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5100_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i5400_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7300_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i7core_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/i82975x_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/ie31200_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/sb_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/edac/x38_edac.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/edd.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/firmware/iscsi_ibft.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-ich.ko 2024-09-13T16:58:12.000 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/gpio/gpio-viperboard.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-appleir.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-aureal.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-axff.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-dr.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elecom.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-elo.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-emsff.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gaff.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gt683r.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-gyration.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-kbd.ko 2024-09-13T16:58:12.001 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtek-mouse.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-holtekff.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-hyperv.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-icade.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-keytouch.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-kye.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lcpower.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-led.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-lenovo.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-dj.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-multitouch.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-ortek.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-penmount.ko 2024-09-13T16:58:12.002 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-petalynx.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-picolcd.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-pl.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-plantronics.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-primax.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-rmi.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-arvo.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-common.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-isku.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kone.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-konepure.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-lua.ko 2024-09-13T16:58:12.003 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-pyra.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-ryos.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat-savu.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-roccat.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-saitek.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-samsung.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sensor-hub.ko 2024-09-13T16:58:12.004 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sjoy.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sony.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-speedlink.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-steelseries.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-sunplus.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-thrustmaster.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tivo.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-tmff.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-topseed.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-twinhan.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-uclogic.ko 2024-09-13T16:58:12.005 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-waltop.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-wiimote.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-xinmo.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zpff.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/hid-zydacron.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/uhid.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hid/wacom.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_balloon.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_utils.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hv/hv_vmbus.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/abituguru3.ko 2024-09-13T16:58:12.006 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/acpi_power_meter.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7414.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ad7418.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adc128d818.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1025.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1026.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1029.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm1031.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adm9240.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ads7828.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7410.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7411.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7462.ko 2024-09-13T16:58:12.007 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7470.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7475.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/adt7x10.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/amc6821.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/applesmc.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asb100.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asc7621.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/asus_atk0110.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/atxp1.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/coretemp.ko 2024-09-13T16:58:12.008 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/dme1737.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds1621.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ds620.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc1403.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/emc6w201.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71805f.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f71882fg.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/f75375s.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fam15h_power.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/fschmd.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g760a.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/g762.ko 2024-09-13T16:58:12.009 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl518sm.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/gl520sm.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/hwmon-vid.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5500_temp.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/i5k_amb.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmaem.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ibmpex.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina209.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ina2xx.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/it87.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k10temp.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/k8temp.ko 2024-09-13T16:58:12.010 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lineage-pem.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm63.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm73.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm75.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm77.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm78.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm80.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm83.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm85.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm87.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm90.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm92.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm93.ko 2024-09-13T16:58:12.011 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95234.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95241.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/lm95245.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc2945.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4151.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4215.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4222.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4245.ko 2024-09-13T16:58:12.012 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4260.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/ltc4261.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max16065.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1619.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max1668.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max197.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6639.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6650.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/max6697.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/mcp3021.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6683.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775-core.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct6775.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/nct7802.ko 2024-09-13T16:58:12.013 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87360.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pc87427.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pcf8591.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max16064.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max34440.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/max8688.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-09-13T16:58:12.014 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/powr1220.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5627.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch5636.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sch56xx-common.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht15.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sht21.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/shtc1.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/sis5595.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47b397.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m1.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/smsc47m192.ko 2024-09-13T16:58:12.015 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/thmc50.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp102.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp103.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp401.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/tmp421.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via-cputemp.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/via686a.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt1211.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/vt8231.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627ehf.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83627hf.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83781d.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83791d.ko 2024-09-13T16:58:12.016 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83792d.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83793.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83795.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l785ts.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/hwmon/w83l786ng.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-09-13T16:58:12.017 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-i801.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-isch.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-09-13T16:58:12.018 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-parport.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-09-13T16:58:12.019 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-09-13T16:58:12.020 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-09-13T16:58:12.020 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-via.ko 2024-09-13T16:58:12.020 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-09-13T16:58:12.020 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-09-13T16:58:12.020 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-dev.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-mux.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-smbus.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/i2c/i2c-stub.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_cm.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_core.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_umad.ko 2024-09-13T16:58:12.021 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/iw_cm.ko 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_cm.ko 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-09-13T16:58:12.022 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp 2024-09-13T16:58:12.023 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-09-13T16:58:12.024 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport 2024-09-13T16:58:12.025 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-09-13T16:58:12.026 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/fm801-gp.ko 2024-09-13T16:58:12.026 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/gameport/gameport.ko 2024-09-13T16:58:12.026 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/joydev.ko 2024-09-13T16:58:12.026 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/keyboard/gpio_keys.ko 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/appletouch.ko 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/bcm5974.ko 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/cyapatp.ko 2024-09-13T16:58:12.027 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/elan_i2c.ko 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/sermouse.ko 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/synaptics_usb.ko 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/mouse/vsxxxaa.ko 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4 2024-09-13T16:58:12.028 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/rmi4/rmi_core.ko 2024-09-13T16:58:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio 2024-09-13T16:58:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/altera_ps2.ko 2024-09-13T16:58:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/arc_ps2.ko 2024-09-13T16:58:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-09-13T16:58:12.029 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/serio/serio_raw.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/input/sparse-keymap.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-blinkm.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lm3530.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-lp3944.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/leds-ss4200.ko 2024-09-13T16:58:12.030 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-09-13T16:58:12.031 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/bcache/bcache.ko 2024-09-13T16:58:12.032 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-bio-prison.ko 2024-09-13T16:58:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache-smq.ko 2024-09-13T16:58:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-cache.ko 2024-09-13T16:58:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-crypt.ko 2024-09-13T16:58:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-delay.ko 2024-09-13T16:58:12.033 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-flakey.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-log-userspace.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-multipath.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-queue-length.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-raid.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-round-robin.ko 2024-09-13T16:58:12.034 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-service-time.ko 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-switch.ko 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-thin-pool.ko 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/dm-verity.ko 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-09-13T16:58:12.035 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid0.ko 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid1.ko 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid10.ko 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/md/raid456.ko 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion 2024-09-13T16:58:12.036 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptbase.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptctl.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptfc.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptlan.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptsas.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptscsih.ko 2024-09-13T16:58:12.037 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/message/fusion/mptspi.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_ich.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/lpc_sch.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/mfd-core.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/sm501.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/viperboard.ko 2024-09-13T16:58:12.038 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/vx855.ko 2024-09-13T16:58:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/mfd/wl1273-core.ko 2024-09-13T16:58:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc 2024-09-13T16:58:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl 2024-09-13T16:58:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-09-13T16:58:12.039 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds9802als.ko 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/apds990x.ko 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/bh1770glc.ko 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/cb710/cb710.ko 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom 2024-09-13T16:58:12.040 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/at24.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/eeprom/max6875.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/enclosure.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/hpilo.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29003.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/isl29020.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-09-13T16:58:12.041 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-me.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei-txe.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/mei/mei.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-gru/gru.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xp.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpc.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_7xx1.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tifm_core.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/tsl2550.ko 2024-09-13T16:58:12.042 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_balloon.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/bonding/bonding.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/b53/b53_common.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/bcm-sf2.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6060.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/dummy.ko 2024-09-13T16:58:12.043 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/eql.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/8390.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-09-13T16:58:12.044 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/agere/et131x.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c 2024-09-13T16:58:12.045 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-09-13T16:58:12.046 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-09-13T16:58:12.047 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-09-13T16:58:12.048 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-09-13T16:58:12.049 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-09-13T16:58:12.050 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-09-13T16:58:12.051 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/dnet.ko 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet 2024-09-13T16:58:12.052 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ethoc.ko 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/fealnx.ko 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e100.ko 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000 2024-09-13T16:58:12.053 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-09-13T16:58:12.054 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libeth/libeth.ko 2024-09-13T16:58:12.055 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/intel/libie/libie.ko 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/jme.ko 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/skge.ko 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox 2024-09-13T16:58:12.056 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-09-13T16:58:12.057 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-09-13T16:58:12.058 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/atp.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-09-13T16:58:12.059 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis190.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sis/sis900.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-09-13T16:58:12.060 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/cassini.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/niu.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sungem.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti 2024-09-13T16:58:12.061 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/ti/tlan.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/geneve.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ieee802154/fakelb.ko 2024-09-13T16:58:12.062 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ifb.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ipvlan/ipvlan.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvlan.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/macvtap.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mdio/mdio-regmap.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/mii.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/net_failover.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/netconsole.ko 2024-09-13T16:58:12.063 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/nlmon.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs-lynx.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/amd.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm7xxx.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/bcm87xx.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/broadcom.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/cicada.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/davicom.ko 2024-09-13T16:58:12.064 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/dp83640.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/et1011c.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/icplus.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/lxt.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/marvell.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/micrel.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/national.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/phylink.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/qsemi.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/realtek.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/smsc.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/ste10Xp.ko 2024-09-13T16:58:12.065 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/phy/vitesse.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/bsd_comp.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_async.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_deflate.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_generic.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_mppe.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/ppp_synctty.ko 2024-09-13T16:58:12.066 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppoe.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pppox.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/ppp/pptp.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slhc.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/slip/slip.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/sungem_phy.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tap.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_activebackup.ko 2024-09-13T16:58:12.067 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_broadcast.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_random.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/tun.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/veth.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/virtio_net.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-09-13T16:58:12.068 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/vxlan/vxlan.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netback/xen-netback.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/net/xen-netfront.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ntb/ntb.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme-core.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/nvme/host/nvme.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport.ko 2024-09-13T16:58:12.069 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_cs.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_pc.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/parport/parport_serial.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/pcie/aer_inject.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pci/xen-pcifront.ko 2024-09-13T16:58:12.070 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/i82092.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/pd6729.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pcmcia/yenta_socket.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acer-wmi.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/acerhdf.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/apple-gmux.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-laptop.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-09-13T16:58:12.071 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/asus-wmi.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/classmate-laptop.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/hdaps.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/intel_ips.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/msi-wmi.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/mxm-wmi.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-laptop.ko 2024-09-13T16:58:12.072 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/samsung-q10.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/topstar-laptop.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/toshiba_haps.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi-bmof.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/platform/x86/wmi.ko 2024-09-13T16:58:12.073 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_common.ko 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/powercap/intel_rapl_msr.ko 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-gpio.ko 2024-09-13T16:58:12.074 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps-ldisc.ko 2024-09-13T16:58:12.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/clients/pps_parport.ko 2024-09-13T16:58:12.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/pps/pps_core.ko 2024-09-13T16:58:12.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp 2024-09-13T16:58:12.075 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp.ko 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ptp/ptp_kvm.ko 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ras/amd/fmpm.ko 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-bq32k.ko 2024-09-13T16:58:12.076 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1286.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1307.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1374.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1511.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1553.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1672.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds1742.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds2404.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-ds3232.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-em3027.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-fm3130.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl12022.ko 2024-09-13T16:58:12.077 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-isl1208.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m41t80.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t35.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-m48t59.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-max6900.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-msm6242.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf2127.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf85063.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8523.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8563.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-pcf8583.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rp5c01.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rs5c372.ko 2024-09-13T16:58:12.078 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8025.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-rx8581.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/rtc/rtc-x1205.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-9xxx.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-sas.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/3w-xxxx.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/BusLogic.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/a100u2w.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aacraid/aacraid.ko 2024-09-13T16:58:12.079 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/advansys.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/am53c974.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/atp870u.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa 2024-09-13T16:58:12.080 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bfa/bfa.ko 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ch.ko 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/csiostor/csiostor.ko 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i 2024-09-13T16:58:12.081 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dc395x.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-09-13T16:58:12.082 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/dmx3191d.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esas2r/esas2r.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/esp_scsi.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/fcoe.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/fnic/fnic.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hpsa.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hptiop.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/hv_storvsc.ko 2024-09-13T16:58:12.083 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/initio.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ipr.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ips.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/isci/isci.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/iscsi_tcp.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libfc/libfc.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libiscsi_tcp.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/libsas/libsas.ko 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc 2024-09-13T16:58:12.084 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/lpfc/lpfc.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvsas/mvsas.ko 2024-09-13T16:58:12.085 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/mvumi.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/pmcraid.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla1280.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/raid_class.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_debug.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_fc.ko 2024-09-13T16:58:12.086 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_sas.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_spi.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/scsi_transport_srp.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/ses.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/st.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/stex.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/virtio_scsi.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/vmw_pvscsi.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/scsi/wd719x.ko 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb 2024-09-13T16:58:12.087 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ssb/ssb.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/loopback/tcm_loop.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_file.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_iblock.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_mod.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_pscsi.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/target_core_user.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal 2024-09-13T16:58:12.088 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-09-13T16:58:12.089 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_gsm.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/n_hdlc.ko 2024-09-13T16:58:12.090 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/nozomi.ko 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/arc_uart.ko 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/tty/serial/jsm/jsm.ko 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/core/ufshcd-core.ko 2024-09-13T16:58:12.091 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_aec.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_cif.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_pci_generic.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/uio/uio_sercos3.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-acm.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/cdc-wdm.ko 2024-09-13T16:58:12.092 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usblp.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/class/usbtmc.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/host/sl811-hcd.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/mdc800.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/image/microtek.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/adutux.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/appledisplay.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi26.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/emi62.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ezusb.ko 2024-09-13T16:58:12.093 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/idmouse.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/iowarrior.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/isight_firmware.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/ldusb.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/legousbtower.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/trancevibrator.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usb3503.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usblcd.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/usbsevseg.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/uss720.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/misc/yurex.ko 2024-09-13T16:58:12.094 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/phy/phy-generic.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/aircable.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ark3116.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/belkin_sa.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ch341.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cp210x.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cyberjack.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/cypress_m8.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/digi_acceleport.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/empeg.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ftdi_sio.ko 2024-09-13T16:58:12.095 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/garmin_gps.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_edgeport.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/io_ti.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipaq.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ipw.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ir-usb.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/keyspan_pda.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kl5kusb105.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/kobil_sct.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mct_u232.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7720.ko 2024-09-13T16:58:12.096 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/mos7840.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/navman.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/omninet.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/opticon.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/option.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/oti6858.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/pl2303.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcaux.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/qcserial.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/quatech2.ko 2024-09-13T16:58:12.097 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/safe_serial.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/sierra.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/spcp8x5.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ssu100.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/symbolserial.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_debug.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/usb_wwan.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/visor.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/whiteheat.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/serial/xsens_mt.ko 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage 2024-09-13T16:58:12.098 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/uas.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-alauda.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-cypress.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-datafab.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-freecom.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-isd200.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-karma.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-onetouch.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-realtek.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr09.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-sddr55.ko 2024-09-13T16:58:12.099 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/ums-usbat.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/storage/usb-storage.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-core.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/usbip-host.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/pci/vfio-pci.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost 2024-09-13T16:58:12.100 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_iotlb.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_net.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/vhost/vhost_scsi.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/apple_bl.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/lcd.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/video/backlight/platform_lcd.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_balloon.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_mmio.ko 2024-09-13T16:58:12.101 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/virtio/virtio_ring.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2482.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/masters/ds2490.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-09-13T16:58:12.102 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_smem.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/slaves/w1_therm.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/w1/wire.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim1535_wdt.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/alim7101_wdt.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/f71808e_wdt.ko 2024-09-13T16:58:12.103 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/hpwdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/i6300esb.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/iTCO_wdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ib700wdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ibmasr.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it8712f_wdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/it87_wdt.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/machzwd.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/nv_tco.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_pci.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/pcwd_usb.ko 2024-09-13T16:58:12.104 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sch311x_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/softdog.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/sp5100_tco.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/via_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83877f_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/w83977f_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/wdt_pci.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/watchdog/xen_wdt.ko 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen 2024-09-13T16:58:12.105 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-acpi-processor.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-evtchn.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntalloc.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-gntdev.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xen-privcmd.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/drivers/xen/xenfs/xenfs.ko 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p 2024-09-13T16:58:12.106 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/9p/9p.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/affs/affs.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/befs/befs.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/binfmt_misc.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/btrfs/btrfs.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cachefiles/cachefiles.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ceph/ceph.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/coda/coda.ko 2024-09-13T16:58:12.107 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/cramfs/cramfs.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/dlm/dlm.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ecryptfs/ecryptfs.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/f2fs/f2fs.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/fat.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/msdos.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fat/vfat.ko 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse 2024-09-13T16:58:12.108 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/cuse.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/fuse/fuse.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/gfs2/gfs2.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfs/hfs.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/hfsplus/hfsplus.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/isofs/isofs.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/jfs/jfs.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/lockd/lockd.ko 2024-09-13T16:58:12.109 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/minix/minix.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/netfs/netfs.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfs.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv3.ko 2024-09-13T16:58:12.110 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs/nfsv4.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/grace.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfs_common/nfs_acl.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nfsd/nfsd.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nilfs2/nilfs2.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-celtic.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-centeuro.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-croatian.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-cyrillic.ko 2024-09-13T16:58:12.111 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-gaelic.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-greek.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-iceland.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-inuit.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-roman.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-romanian.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/mac-turkish.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1250.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1251.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp1255.ko 2024-09-13T16:58:12.112 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp737.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp775.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp850.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp852.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp855.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp857.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp860.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp861.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp862.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp863.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp864.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp865.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp866.ko 2024-09-13T16:58:12.113 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp869.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp874.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp932.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp936.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp949.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_cp950.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_euc-jp.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-1.ko 2024-09-13T16:58:12.114 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-13.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-14.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-15.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-2.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-3.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-4.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-5.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-6.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-7.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_iso8859-9.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-r.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-ru.ko 2024-09-13T16:58:12.115 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_koi8-u.ko 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_ucs2_utils.ko 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/nls/nls_utf8.ko 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-09-13T16:58:12.116 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2.ko 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs 2024-09-13T16:58:12.117 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/overlayfs/overlay.ko 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/pstore/ramoops.ko 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/reiserfs/reiserfs.ko 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs 2024-09-13T16:58:12.118 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/romfs/romfs.ko 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/client/cifs.ko 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_arc4.ko 2024-09-13T16:58:12.119 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/smb/common/cifs_md4.ko 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/squashfs/squashfs.ko 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/sysv/sysv.ko 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf 2024-09-13T16:58:12.120 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/udf/udf.ko 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/ufs/ufs.ko 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/fs/xfs/xfs.ko 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel 2024-09-13T16:58:12.121 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/rcu/rcutorture.ko 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/torture.ko 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/asn1_encoder.ko 2024-09-13T16:58:12.122 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc-itu-t.ko 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crc8.ko 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libaescfb.ko 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libarc4.ko 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/crypto/libdes.ko 2024-09-13T16:58:12.123 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/libcrc32c.ko 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lru_cache.ko 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4_compress.ko 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/lz4/lz4hc_compress.ko 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math 2024-09-13T16:58:12.124 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/math/cordic.ko 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/raid6/raid6_pq.ko 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/reed_solomon/reed_solomon.ko 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_bm.ko 2024-09-13T16:58:12.125 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_fsm.ko 2024-09-13T16:58:12.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/ts_kmp.ko 2024-09-13T16:58:12.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd 2024-09-13T16:58:12.126 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/lib/zstd/zstd_compress.ko 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/mm/hwpoison-inject.ko 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/6lowpan.ko 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_dest.ko 2024-09-13T16:58:12.127 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_fragment.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_hop.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_ipv6.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_mobility.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_routing.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/6lowpan/nhc_udp.ko 2024-09-13T16:58:12.128 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/garp.ko 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/mrp.ko 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/p8022.ko 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/psnap.ko 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/802/stp.ko 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q 2024-09-13T16:58:12.129 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/8021q/8021q.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_fd.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_rdma.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/9p/9pnet_virtio.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/batman-adv/batman-adv.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/br_netfilter.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/bridge.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_among.ko 2024-09-13T16:58:12.130 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arp.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_limit.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_log.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_snat.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_stp.ko 2024-09-13T16:58:12.131 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/ebtables.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ceph/libceph.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/failover.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/core/pktgen.ko 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp 2024-09-13T16:58:12.132 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_diag.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv4.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dccp/dccp_ipv6.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dns_resolver/dns_resolver.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/dsa_core.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_brcm.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_dsa.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_none.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/dsa/tag_trailer.ko 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan 2024-09-13T16:58:12.133 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ieee802154/ieee802154_socket.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ah4.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/esp4.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/fou.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/gre.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/inet_diag.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_gre.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_tunnel.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ip_vti.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipcomp.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/ipip.ko 2024-09-13T16:58:12.134 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arp_tables.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/iptable_security.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-09-13T16:58:12.135 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_bic.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_dctcp.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_diag.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_highspeed.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_htcp.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_hybla.ko 2024-09-13T16:58:12.136 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_illinois.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_lp.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_scalable.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_vegas.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_veno.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_westwood.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tcp_yeah.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/tunnel4.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_diag.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/udp_tunnel.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv4/xfrm4_tunnel.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ah6.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/esp6.ko 2024-09-13T16:58:12.137 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/fou6.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_tunnel.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ip6_vti.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/ipcomp6.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-09-13T16:58:12.138 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-09-13T16:58:12.139 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/sit.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/tunnel6.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/ipv6/xfrm6_tunnel.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/key/af_key.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_core.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_debugfs.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_eth.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ip6.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_netlink.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/l2tp/l2tp_ppp.ko 2024-09-13T16:58:12.140 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/llc/llc.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mac802154/mac802154.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/mpls/mpls_gso.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-09-13T16:58:12.141 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-09-13T16:58:12.142 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conncount.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-09-13T16:58:12.143 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_h323.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_irc.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sane.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_sip.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_dup_netdev.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table.ko 2024-09-13T16:58:12.144 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_flow_table_inet.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_log_syslog.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_amanda.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_ftp.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_irc.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_sip.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_nat_tftp.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_synproxy_core.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nf_tables.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_acct.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_log.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_osf.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nfnetlink_queue.ko 2024-09-13T16:58:12.145 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_chain_nat.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_compat.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_ct.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_dup_netdev.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fib.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_fwd_netdev.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_hash.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_limit.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_log.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_masq.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_nat.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_numgen.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_queue.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_quota.ko 2024-09-13T16:58:12.146 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_redir.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_reject_inet.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_socket.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/nft_tunnel.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_AUDIT.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CHECKSUM.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CLASSIFY.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_CT.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_DSCP.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HL.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_HMARK.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_IDLETIMER.ko 2024-09-13T16:58:12.147 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LED.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_LOG.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_MASQUERADE.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NETMAP.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFLOG.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_NFQUEUE.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_RATEEST.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_REDIRECT.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_SECMARK.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPMSS.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TEE.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TPROXY.ko 2024-09-13T16:58:12.148 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_TRACE.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_addrtype.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_bpf.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cgroup.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cluster.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_comment.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connbytes.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlabel.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connlimit.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_connmark.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_conntrack.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_cpu.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dccp.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_devgroup.ko 2024-09-13T16:58:12.149 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_dscp.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ecn.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_esp.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hashlimit.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_helper.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_hl.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipcomp.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_iprange.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_ipvs.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_l2tp.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_length.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_limit.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mac.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_mark.ko 2024-09-13T16:58:12.150 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_multiport.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nat.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_nfacct.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_osf.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_owner.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_physdev.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_pkttype.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_policy.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_quota.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_rateest.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_realm.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_recent.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_sctp.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_set.ko 2024-09-13T16:58:12.151 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_socket.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_state.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_statistic.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_string.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_tcpmss.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_time.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netfilter/xt_u32.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/netlink/netlink_diag.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/nsh/nsh.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/openvswitch.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-geneve.ko 2024-09-13T16:58:12.152 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-gre.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/openvswitch/vport-vxlan.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/packet/af_packet_diag.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_rdma.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/rds/rds_tcp.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_csum.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_gact.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_mirred.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_nat.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_pedit.ko 2024-09-13T16:58:12.153 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_police.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_simple.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_skbedit.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/act_vlan.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_basic.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_bpf.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_flow.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_fw.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_route.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/cls_u32.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_cmp.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_ipset.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_meta.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_nbyte.ko 2024-09-13T16:58:12.154 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_text.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/em_u32.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_choke.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_codel.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_drr.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_fq.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_gred.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hfsc.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_hhf.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_htb.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_ingress.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_mqprio_lib.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_multiq.ko 2024-09-13T16:58:12.155 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_netem.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_pie.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_plug.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_prio.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_qfq.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_red.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfb.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_sfq.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_tbf.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sched/sch_teql.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sctp/sctp_diag.ko 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc 2024-09-13T16:58:12.156 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/sunrpc.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/tipc/tipc_diag.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/unix/unix_diag.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-09-13T16:58:12.157 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_diag.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/vmw_vsock/vsock_loopback.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/net/xfrm/xfrm_ipcomp.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/kernel/security/keys/trusted-keys/trusted.ko 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.alias.bin 2024-09-13T16:58:12.158 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.alias.bin 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.bin 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.builtin.modinfo 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep.bin 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.devname 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.order 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.softdep 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.symbols.bin 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.weakdep 2024-09-13T16:58:12.159 INFO:teuthology.orchestra.run.smithi189.stdout:/lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:58:12.163 DEBUG:teuthology.task.kernel:get_image_version: 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:12.163 DEBUG:teuthology.orchestra.run.smithi189:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-09-13T16:58:12.187 INFO:teuthology.orchestra.run.smithi189.stdout:postinstall scriptlet (using /bin/sh): 2024-09-13T16:58:12.187 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: /usr/bin/kernel-install add 6.11.0-rc5-g2ca44a2f35ca /lib/modules/6.11.0-rc5-g2ca44a2f35ca/vmlinuz 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:for file in vmlinuz System.map config; do 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: if ! cmp --silent "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca"; then 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: cp "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/${file}" "/boot/${file}-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: fi 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:done 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -d "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" ] && \ 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: ! diff -rq "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" >/dev/null 2>&1; then 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: rm -rf "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout: cp -r "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/dtb" "/boot/dtb-6.11.0-rc5-g2ca44a2f35ca" 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-09-13T16:58:12.188 INFO:teuthology.orchestra.run.smithi189.stdout:if [ ! -e "/lib/modules/6.11.0-rc5-g2ca44a2f35ca/modules.dep" ]; then 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout: /usr/sbin/depmod 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout:preuninstall scriptlet (using /bin/sh): 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout:kernel-install remove 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:12.189 INFO:teuthology.orchestra.run.smithi189.stdout:fi 2024-09-13T16:58:12.190 DEBUG:teuthology.orchestra.run.smithi189:> sudo rpm -qi grub2-tools 2024-09-13T16:58:12.263 INFO:teuthology.orchestra.run.smithi189.stdout:Name : grub2-tools 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Epoch : 1 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Version : 2.06 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Release : 82.el9 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Architecture: x86_64 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Group : Unspecified 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Size : 8274670 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:License : GPLv3+ 2024-09-13T16:58:12.264 INFO:teuthology.orchestra.run.smithi189.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Packager : builder@centos.org 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Vendor : CentOS 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Summary : Support tools for GRUB. 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:Description : 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:hardware devices. 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T16:58:12.265 INFO:teuthology.orchestra.run.smithi189.stdout:This subpackage provides tools for support of all platforms. 2024-09-13T16:58:12.266 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-13T16:58:12.266 INFO:teuthology.task.kernel:Updating grub on smithi189 to boot 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:12.267 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-13T16:58:12.879 INFO:teuthology.orchestra.run.smithi189.stderr:Generating grub configuration file ... 2024-09-13T16:58:14.256 INFO:teuthology.orchestra.run.smithi189.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-13T16:58:14.277 INFO:teuthology.orchestra.run.smithi189.stderr:done 2024-09-13T16:58:14.280 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-13T16:58:14.295 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.iz4jOlCGba 2024-09-13T16:58:14.295 DEBUG:teuthology.orchestra.run.smithi189:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iz4jOlCGba 2024-09-13T16:58:14.369 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.iz4jOlCGba 2024-09-13T16:58:14.513 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.iz4jOlCGba is 6KB 2024-09-13T16:58:14.561 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.iz4jOlCGba 2024-09-13T16:58:14.576 DEBUG:teuthology.orchestra.run.smithi189:> sudo /bin/ls /boot/loader/entries || true 2024-09-13T16:58:14.644 INFO:teuthology.orchestra.run.smithi189.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-13T16:58:14.644 INFO:teuthology.orchestra.run.smithi189.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-13T16:58:14.644 INFO:teuthology.orchestra.run.smithi189.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-13T16:58:14.644 INFO:teuthology.orchestra.run.smithi189.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca.conf 2024-09-13T16:58:14.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:58:14.790 DEBUG:teuthology.orchestra.run.smithi189:> sudo shutdown -r now 2024-09-13T16:58:41.997 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-13T16:58:41.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:58:41.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:58:44.794 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-09-13T16:58:44.795 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T16:58:44.795 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:59:19.486 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.189 2024-09-13T16:59:28.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T16:59:28.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:59:42.022 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T16:59:51.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:59:51.032 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:59:51.385 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-13T16:59:51.863 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T16:59:51.864 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-13T16:59:51.864 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-09-13T16:59:51.898 INFO:teuthology.orchestra.run.smithi028.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:59:51.898 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T16:59:51.898 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T16:59:51.898 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T16:59:52.901 DEBUG:teuthology.parallel:result is None 2024-09-13T17:00:28.507 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T17:00:40.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T17:00:40.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T17:00:40.881 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-13T17:00:41.405 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T17:00:41.405 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.11.0-rc5-g2ca44a2f35ca"... 2024-09-13T17:00:41.406 DEBUG:teuthology.orchestra.run.smithi189:> uname -r 2024-09-13T17:00:41.440 INFO:teuthology.orchestra.run.smithi189.stdout:6.11.0-rc5-g2ca44a2f35ca 2024-09-13T17:00:41.440 DEBUG:teuthology.task.kernel:current kernel version is 6.11.0-rc5-g2ca44a2f35ca vs 6.11.0-rc5-g2ca44a2f35ca 2024-09-13T17:00:41.440 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T17:00:41.440 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-13T17:00:42.441 DEBUG:teuthology.parallel:result is None 2024-09-13T17:00:42.441 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T17:00:42.450 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T17:00:42.450 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T17:00:42.455 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T17:00:42.493 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T17:00:42.505 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T17:00:42.520 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T17:00:42.521 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T17:00:42.535 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T17:00:42.585 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T17:00:42.596 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T17:00:42.596 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T17:00:42.600 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T17:00:42.725 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T17:00:42.738 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T17:00:42.752 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T17:00:42.765 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T17:00:42.769 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T17:00:42.777 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T17:00:42.778 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T17:00:42.782 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T17:00:42.839 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T17:00:42.851 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T17:00:42.851 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T17:00:42.855 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T17:00:42.889 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T17:00:43.000 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T17:00:43.091 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T17:00:43.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T17:00:43.223 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:00:43.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T17:00:43.278 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T17:00:43.352 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T17:00:43.430 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T17:00:43.503 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T17:00:43.565 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:00:43.565 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T17:00:43.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-09-13T17:00:43.623 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-09-13T17:00:43.718 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T17:00:43.747 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T17:00:44.113 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T17:00:44.120 INFO:teuthology.task.internal:Starting timer... 2024-09-13T17:00:44.120 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T17:00:44.133 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T17:00:44.142 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-13T17:00:44.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-09-13T17:00:44.214 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-09-13T17:00:44.218 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-09-13T17:00:44.300 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-09-13T17:00:44.303 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T17:00:44.304 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-09-13T17:00:44.345 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-09-13T17:00:44.345 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-09-13T17:00:44.388 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-09-13T17:00:44.388 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-09-13T17:00:44.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T17:00:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:00:44.445 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T17:00:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:00:44.502 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T17:00:44.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-09-13T17:00:44.572 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-09-13T17:00:44.634 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T17:00:44.644 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T17:00:44.647 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T17:00:44.833 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T17:00:44.844 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T17:00:44.845 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T17:05:27.090 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-09-13T17:05:27.092 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T17:05:27.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T17:05:27.208 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-09-13T17:05:27.320 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-09-13T17:05:27.321 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T17:05:27.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T17:05:27.432 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-09-13T17:05:27.553 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-09-13T17:05:27.554 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T17:05:27.564 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T17:05:27.565 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T17:05:27.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T17:05:27.569 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T17:05:27.569 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T17:05:27.627 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T17:05:27.654 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-13T17:05:27.657 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T17:05:27.683 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-13T17:05:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-09-13T17:05:27.732 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-09-13T17:05:27.734 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-09-13T17:05:27.759 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-09-13T17:05:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T17:05:27.789 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-13T17:05:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T17:05:27.818 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-13T17:05:27.889 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.895 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.911 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.918 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:05:27.919 INFO:teuthology.run_tasks:Running task install... 2024-09-13T17:05:27.927 DEBUG:teuthology.task.install:project ceph 2024-09-13T17:05:27.927 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'}} 2024-09-13T17:05:27.927 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'} 2024-09-13T17:05:27.928 INFO:teuthology.task.install:Using flavor: default 2024-09-13T17:05:27.933 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T17:05:27.933 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-09-13T17:05:27.934 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3', 'tag': None, 'wait_for_package': False} 2024-09-13T17:05:27.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 2024-09-13T17:05:27.936 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3', 'tag': None, 'wait_for_package': False} 2024-09-13T17:05:27.936 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 2024-09-13T17:05:28.092 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/ 2024-09-13T17:05:28.092 INFO:teuthology.task.install.rpm:Package version is 18.2.4-860.g6c6b083f 2024-09-13T17:05:28.104 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/ 2024-09-13T17:05:28.104 INFO:teuthology.task.install.rpm:Package version is 18.2.4-860.g6c6b083f 2024-09-13T17:05:28.211 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T17:05:28.211 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:05:28.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T17:05:28.224 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.100730-reef/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-13T17:05:28.224 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:05:28.224 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-13T17:05:28.267 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T17:05:28.268 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T17:05:28.281 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-09-13T17:05:28.281 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-13T17:05:28.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T17:05:28.379 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-13T17:05:28.444 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T17:05:28.459 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-13T17:05:28.499 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-09-13T17:05:28.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-09-13T17:05:28.516 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-09-13T17:05:28.519 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-09-13T17:05:28.774 INFO:teuthology.orchestra.run.smithi028.stdout:45 files removed 2024-09-13T17:05:28.802 INFO:teuthology.orchestra.run.smithi189.stdout:45 files removed 2024-09-13T17:05:28.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T17:05:28.835 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-09-13T17:05:29.510 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 262 kB/s | 78 kB 00:00 2024-09-13T17:05:29.560 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 249 kB/s | 78 kB 00:00 2024-09-13T17:05:29.804 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 46 kB/s | 11 kB 00:00 2024-09-13T17:05:29.885 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 41 kB/s | 11 kB 00:00 2024-09-13T17:05:30.079 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 7.8 kB/s | 1.9 kB 00:00 2024-09-13T17:05:30.211 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-09-13T17:05:30.886 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-13T17:05:31.249 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 7.3 MB/s | 8.2 MB 00:01 2024-09-13T17:05:32.954 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-13T17:05:33.239 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-13T17:05:37.521 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-13T17:05:39.081 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.5 MB 00:01 2024-09-13T17:05:39.088 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 68 kB/s | 19 kB 00:00 2024-09-13T17:05:40.301 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T17:05:40.623 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-13T17:05:41.828 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 21 MB/s | 23 MB 00:01 2024-09-13T17:05:46.598 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2024-09-13T17:05:48.417 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-13T17:05:49.136 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T17:05:49.137 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T17:05:49.148 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T17:05:49.148 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T17:05:49.149 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T17:05:49.150 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T17:05:49.152 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T17:05:49.154 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T17:05:49.155 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T17:05:49.219 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 6.5 k 2024-09-13T17:05:49.242 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 5.1 M 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 852 k 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 142 k 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 1.5 M 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 139 k 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 3.5 M 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 7.4 M 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 49 k 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 7.7 M 2024-09-13T17:05:49.243 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 40 M 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 264 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 226 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 224 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 18 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T17:05:49.244 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 31 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 712 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 127 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 162 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 321 k 2024-09-13T17:05:49.245 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 301 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 99 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 87 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.0 M 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 172 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.3 M 2024-09-13T17:05:49.246 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.0 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 18 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 24 k 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 2.1 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 246 k 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 4.7 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 17 M 2024-09-13T17:05:49.247 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 16 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 25 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T17:05:49.248 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 166 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 478 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 4.5 M 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T17:05:49.249 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 45 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 129 k 2024-09-13T17:05:49.250 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T17:05:49.251 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T17:05:49.252 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T17:05:49.253 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T17:05:49.254 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout:====================================================================================== 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout:Install 130 Packages 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 8 Packages 2024-09-13T17:05:49.255 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:05:49.256 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 203 M 2024-09-13T17:05:49.256 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-09-13T17:05:49.666 INFO:teuthology.orchestra.run.smithi189.stdout:(1/138): ceph-18.2.4-860.g6c6b083f.el9.x86_64.r 47 kB/s | 6.5 kB 00:00 2024-09-13T17:05:49.825 INFO:teuthology.orchestra.run.smithi189.stdout:(2/138): ceph-fuse-18.2.4-860.g6c6b083f.el9.x86 5.3 MB/s | 852 kB 00:00 2024-09-13T17:05:49.909 INFO:teuthology.orchestra.run.smithi189.stdout:(3/138): ceph-immutable-object-cache-18.2.4-860 1.7 MB/s | 142 kB 00:00 2024-09-13T17:05:50.034 INFO:teuthology.orchestra.run.smithi189.stdout:(4/138): ceph-base-18.2.4-860.g6c6b083f.el9.x86 10 MB/s | 5.1 MB 00:00 2024-09-13T17:05:50.109 INFO:teuthology.orchestra.run.smithi189.stdout:(5/138): ceph-mds-18.2.4-860.g6c6b083f.el9.x86_ 10 MB/s | 2.1 MB 00:00 2024-09-13T17:05:50.160 INFO:teuthology.orchestra.run.smithi189.stdout:(6/138): ceph-mgr-18.2.4-860.g6c6b083f.el9.x86_ 12 MB/s | 1.5 MB 00:00 2024-09-13T17:05:50.602 INFO:teuthology.orchestra.run.smithi189.stdout:(7/138): ceph-common-18.2.4-860.g6c6b083f.el9.x 17 MB/s | 18 MB 00:01 2024-09-13T17:05:50.685 INFO:teuthology.orchestra.run.smithi189.stdout:(8/138): ceph-mon-18.2.4-860.g6c6b083f.el9.x86_ 8.1 MB/s | 4.7 MB 00:00 2024-09-13T17:05:50.760 INFO:teuthology.orchestra.run.smithi189.stdout:(9/138): ceph-selinux-18.2.4-860.g6c6b083f.el9. 338 kB/s | 25 kB 00:00 2024-09-13T17:05:50.972 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T17:05:50.973 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-09-13T17:05:50.984 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.48.8-1.el9.x86_64 is already installed. 2024-09-13T17:05:50.985 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-09-13T17:05:50.986 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-20.el9.x86_64 is already installed. 2024-09-13T17:05:50.989 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-09-13T17:05:50.989 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-09-13T17:05:50.991 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-09-13T17:05:50.992 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.4.0-4.el9.x86_64 is already installed. 2024-09-13T17:05:51.027 INFO:teuthology.orchestra.run.smithi189.stdout:(10/138): ceph-radosgw-18.2.4-860.g6c6b083f.el9 18 MB/s | 7.7 MB 00:00 2024-09-13T17:05:51.058 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 6.5 k 2024-09-13T17:05:51.076 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 5.1 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 852 k 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 142 k 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 1.5 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 139 k 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 3.5 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 7.4 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 49 k 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 7.7 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 40 M 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 264 k 2024-09-13T17:05:51.077 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 226 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 224 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 18 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 31 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 712 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 127 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-46.el9 appstream 581 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-20.el9 appstream 24 k 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.2-6.el9 appstream 4.4 M 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.16-1.el9 appstream 1.6 M 2024-09-13T17:05:51.078 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.16-1.el9 appstream 910 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.16-1.el9 appstream 5.8 M 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 162 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 321 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 301 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 99 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 87 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.0 M 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 172 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.4.0-4.el9 appstream 96 k 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager x86_64 1:1.49.90-1.el9 baseos 2.3 M 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm x86_64 1:1.49.90-1.el9 baseos 1.8 M 2024-09-13T17:05:51.079 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team x86_64 1:1.49.90-1.el9 baseos 41 k 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui x86_64 1:1.49.90-1.el9 baseos 249 k 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.3 M 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 3.0 M 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.2-6.el9 baseos 1.4 M 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.2-6.el9 baseos 2.4 M 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 18 M 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 24 k 2024-09-13T17:05:51.080 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 2.1 M 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 246 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 4.7 M 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 17 M 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:18.2.4-860.g6c6b083f.el9 ceph-noarch 16 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 25 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-13T17:05:51.081 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 166 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 478 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 4.5 M 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-13T17:05:51.082 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.16-1.el9 appstream 137 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 45 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:18.2.4-860.g6c6b083f.el9 ceph 129 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-13T17:05:51.083 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-13T17:05:51.084 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-13T17:05:51.085 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-13T17:05:51.086 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-09-13T17:05:51.087 INFO:teuthology.orchestra.run.smithi028.stdout:====================================================================================== 2024-09-13T17:05:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Install 130 Packages 2024-09-13T17:05:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 8 Packages 2024-09-13T17:05:51.088 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:05:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 203 M 2024-09-13T17:05:51.088 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-09-13T17:05:51.319 INFO:teuthology.orchestra.run.smithi189.stdout:(11/138): ceph-osd-18.2.4-860.g6c6b083f.el9.x86 14 MB/s | 17 MB 00:01 2024-09-13T17:05:51.345 INFO:teuthology.orchestra.run.smithi189.stdout:(12/138): cephfs-mirror-18.2.4-860.g6c6b083f.el 708 kB/s | 224 kB 00:00 2024-09-13T17:05:51.360 INFO:teuthology.orchestra.run.smithi028.stdout:(1/138): ceph-18.2.4-860.g6c6b083f.el9.x86_64.r 52 kB/s | 6.5 kB 00:00 2024-09-13T17:05:51.370 INFO:teuthology.orchestra.run.smithi189.stdout:(13/138): libcephfs-devel-18.2.4-860.g6c6b083f. 622 kB/s | 31 kB 00:00 2024-09-13T17:05:51.429 INFO:teuthology.orchestra.run.smithi189.stdout:(14/138): libcephfs2-18.2.4-860.g6c6b083f.el9.x 8.3 MB/s | 712 kB 00:00 2024-09-13T17:05:51.454 INFO:teuthology.orchestra.run.smithi189.stdout:(15/138): libcephsqlite-18.2.4-860.g6c6b083f.el 1.9 MB/s | 166 kB 00:00 2024-09-13T17:05:51.479 INFO:teuthology.orchestra.run.smithi189.stdout:(16/138): librados-devel-18.2.4-860.g6c6b083f.e 2.5 MB/s | 127 kB 00:00 2024-09-13T17:05:51.521 INFO:teuthology.orchestra.run.smithi189.stdout:(17/138): libradosstriper1-18.2.4-860.g6c6b083f 7.0 MB/s | 478 kB 00:00 2024-09-13T17:05:51.543 INFO:teuthology.orchestra.run.smithi028.stdout:(2/138): ceph-fuse-18.2.4-860.g6c6b083f.el9.x86 4.5 MB/s | 852 kB 00:00 2024-09-13T17:05:51.596 INFO:teuthology.orchestra.run.smithi189.stdout:(18/138): python3-ceph-argparse-18.2.4-860.g6c6 601 kB/s | 45 kB 00:00 2024-09-13T17:05:51.635 INFO:teuthology.orchestra.run.smithi028.stdout:(3/138): ceph-immutable-object-cache-18.2.4-860 1.5 MB/s | 142 kB 00:00 2024-09-13T17:05:51.705 INFO:teuthology.orchestra.run.smithi189.stdout:(19/138): python3-ceph-common-18.2.4-860.g6c6b0 1.2 MB/s | 129 kB 00:00 2024-09-13T17:05:51.905 INFO:teuthology.orchestra.run.smithi189.stdout:(20/138): librgw2-18.2.4-860.g6c6b083f.el9.x86_ 10 MB/s | 4.5 MB 00:00 2024-09-13T17:05:51.927 INFO:teuthology.orchestra.run.smithi028.stdout:(4/138): ceph-base-18.2.4-860.g6c6b083f.el9.x86 7.4 MB/s | 5.1 MB 00:00 2024-09-13T17:05:51.972 INFO:teuthology.orchestra.run.smithi189.stdout:(21/138): python3-rados-18.2.4-860.g6c6b083f.el 4.7 MB/s | 321 kB 00:00 2024-09-13T17:05:52.047 INFO:teuthology.orchestra.run.smithi189.stdout:(22/138): python3-rbd-18.2.4-860.g6c6b083f.el9. 3.9 MB/s | 301 kB 00:00 2024-09-13T17:05:52.052 INFO:teuthology.orchestra.run.smithi028.stdout:(5/138): ceph-mds-18.2.4-860.g6c6b083f.el9.x86_ 5.0 MB/s | 2.1 MB 00:00 2024-09-13T17:05:52.097 INFO:teuthology.orchestra.run.smithi189.stdout:(23/138): python3-rgw-18.2.4-860.g6c6b083f.el9. 1.9 MB/s | 99 kB 00:00 2024-09-13T17:05:52.127 INFO:teuthology.orchestra.run.smithi028.stdout:(6/138): ceph-mgr-18.2.4-860.g6c6b083f.el9.x86_ 7.3 MB/s | 1.5 MB 00:00 2024-09-13T17:05:52.139 INFO:teuthology.orchestra.run.smithi189.stdout:(24/138): python3-cephfs-18.2.4-860.g6c6b083f.e 373 kB/s | 162 kB 00:00 2024-09-13T17:05:52.165 INFO:teuthology.orchestra.run.smithi189.stdout:(25/138): rbd-fuse-18.2.4-860.g6c6b083f.el9.x86 1.3 MB/s | 87 kB 00:00 2024-09-13T17:05:52.257 INFO:teuthology.orchestra.run.smithi189.stdout:(26/138): rbd-nbd-18.2.4-860.g6c6b083f.el9.x86_ 1.8 MB/s | 172 kB 00:00 2024-09-13T17:05:52.332 INFO:teuthology.orchestra.run.smithi189.stdout:(27/138): ceph-grafana-dashboards-18.2.4-860.g6 326 kB/s | 24 kB 00:00 2024-09-13T17:05:52.399 INFO:teuthology.orchestra.run.smithi189.stdout:(28/138): ceph-mgr-cephadm-18.2.4-860.g6c6b083f 2.0 MB/s | 139 kB 00:00 2024-09-13T17:05:52.461 INFO:teuthology.orchestra.run.smithi028.stdout:(7/138): ceph-common-18.2.4-860.g6c6b083f.el9.x 15 MB/s | 18 MB 00:01 2024-09-13T17:05:52.466 INFO:teuthology.orchestra.run.smithi189.stdout:(29/138): rbd-mirror-18.2.4-860.g6c6b083f.el9.x 9.2 MB/s | 3.0 MB 00:00 2024-09-13T17:05:52.637 INFO:teuthology.orchestra.run.smithi028.stdout:(8/138): ceph-mon-18.2.4-860.g6c6b083f.el9.x86_ 8.0 MB/s | 4.7 MB 00:00 2024-09-13T17:05:52.641 INFO:teuthology.orchestra.run.smithi189.stdout:(30/138): ceph-mgr-dashboard-18.2.4-860.g6c6b08 15 MB/s | 3.5 MB 00:00 2024-09-13T17:05:52.703 INFO:teuthology.orchestra.run.smithi028.stdout:(9/138): ceph-selinux-18.2.4-860.g6c6b083f.el9. 380 kB/s | 25 kB 00:00 2024-09-13T17:05:52.725 INFO:teuthology.orchestra.run.smithi189.stdout:(31/138): ceph-mgr-modules-core-18.2.4-860.g6c6 2.9 MB/s | 246 kB 00:00 2024-09-13T17:05:52.825 INFO:teuthology.orchestra.run.smithi189.stdout:(32/138): ceph-mgr-rook-18.2.4-860.g6c6b083f.el 493 kB/s | 49 kB 00:00 2024-09-13T17:05:52.908 INFO:teuthology.orchestra.run.smithi189.stdout:(33/138): ceph-prometheus-alerts-18.2.4-860.g6c 198 kB/s | 16 kB 00:00 2024-09-13T17:05:52.962 INFO:teuthology.orchestra.run.smithi028.stdout:(10/138): ceph-radosgw-18.2.4-860.g6c6b083f.el9 15 MB/s | 7.7 MB 00:00 2024-09-13T17:05:52.967 INFO:teuthology.orchestra.run.smithi189.stdout:(34/138): ceph-volume-18.2.4-860.g6c6b083f.el9. 4.4 MB/s | 264 kB 00:00 2024-09-13T17:05:53.020 INFO:teuthology.orchestra.run.smithi028.stdout:(11/138): cephfs-mirror-18.2.4-860.g6c6b083f.el 3.8 MB/s | 224 kB 00:00 2024-09-13T17:05:53.042 INFO:teuthology.orchestra.run.smithi189.stdout:(35/138): cephadm-18.2.4-860.g6c6b083f.el9.noar 2.9 MB/s | 226 kB 00:00 2024-09-13T17:05:53.079 INFO:teuthology.orchestra.run.smithi028.stdout:(12/138): libcephfs-devel-18.2.4-860.g6c6b083f. 533 kB/s | 31 kB 00:00 2024-09-13T17:05:53.109 INFO:teuthology.orchestra.run.smithi189.stdout:(36/138): cephfs-top-18.2.4-860.g6c6b083f.el9.n 274 kB/s | 18 kB 00:00 2024-09-13T17:05:53.163 INFO:teuthology.orchestra.run.smithi028.stdout:(13/138): libcephfs2-18.2.4-860.g6c6b083f.el9.x 8.3 MB/s | 712 kB 00:00 2024-09-13T17:05:53.230 INFO:teuthology.orchestra.run.smithi028.stdout:(14/138): libcephsqlite-18.2.4-860.g6c6b083f.el 2.5 MB/s | 166 kB 00:00 2024-09-13T17:05:53.276 INFO:teuthology.orchestra.run.smithi189.stdout:(37/138): fuse-2.9.9-16.el9.x86_64.rpm 479 kB/s | 80 kB 00:00 2024-09-13T17:05:53.288 INFO:teuthology.orchestra.run.smithi028.stdout:(15/138): librados-devel-18.2.4-860.g6c6b083f.e 2.1 MB/s | 127 kB 00:00 2024-09-13T17:05:53.355 INFO:teuthology.orchestra.run.smithi028.stdout:(16/138): libradosstriper1-18.2.4-860.g6c6b083f 7.0 MB/s | 478 kB 00:00 2024-09-13T17:05:53.368 INFO:teuthology.orchestra.run.smithi189.stdout:(38/138): ceph-mgr-diskprediction-local-18.2.4- 8.2 MB/s | 7.4 MB 00:00 2024-09-13T17:05:53.468 INFO:teuthology.orchestra.run.smithi189.stdout:(39/138): libgfortran-11.5.0-2.el9.x86_64.rpm 4.1 MB/s | 801 kB 00:00 2024-09-13T17:05:53.527 INFO:teuthology.orchestra.run.smithi189.stdout:(40/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-09-13T17:05:53.547 INFO:teuthology.orchestra.run.smithi028.stdout:(17/138): ceph-osd-18.2.4-860.g6c6b083f.el9.x86 12 MB/s | 17 MB 00:01 2024-09-13T17:05:53.569 INFO:teuthology.orchestra.run.smithi189.stdout:(41/138): libquadmath-11.5.0-2.el9.x86_64.rpm 953 kB/s | 191 kB 00:00 2024-09-13T17:05:53.597 INFO:teuthology.orchestra.run.smithi028.stdout:(18/138): python3-ceph-argparse-18.2.4-860.g6c6 907 kB/s | 45 kB 00:00 2024-09-13T17:05:53.628 INFO:teuthology.orchestra.run.smithi189.stdout:(42/138): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-09-13T17:05:53.687 INFO:teuthology.orchestra.run.smithi189.stdout:(43/138): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-09-13T17:05:53.689 INFO:teuthology.orchestra.run.smithi028.stdout:(19/138): librgw2-18.2.4-860.g6c6b083f.el9.x86_ 13 MB/s | 4.5 MB 00:00 2024-09-13T17:05:53.715 INFO:teuthology.orchestra.run.smithi028.stdout:(20/138): python3-ceph-common-18.2.4-860.g6c6b0 1.1 MB/s | 129 kB 00:00 2024-09-13T17:05:53.745 INFO:teuthology.orchestra.run.smithi189.stdout:(44/138): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-09-13T17:05:53.748 INFO:teuthology.orchestra.run.smithi028.stdout:(21/138): python3-cephfs-18.2.4-860.g6c6b083f.e 2.7 MB/s | 162 kB 00:00 2024-09-13T17:05:53.773 INFO:teuthology.orchestra.run.smithi028.stdout:(22/138): python3-rados-18.2.4-860.g6c6b083f.el 5.4 MB/s | 321 kB 00:00 2024-09-13T17:05:53.799 INFO:teuthology.orchestra.run.smithi028.stdout:(23/138): python3-rbd-18.2.4-860.g6c6b083f.el9. 5.9 MB/s | 301 kB 00:00 2024-09-13T17:05:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:(24/138): python3-rgw-18.2.4-860.g6c6b083f.el9. 1.3 MB/s | 99 kB 00:00 2024-09-13T17:05:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:(25/138): rbd-fuse-18.2.4-860.g6c6b083f.el9.x86 1.0 MB/s | 87 kB 00:00 2024-09-13T17:05:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:(26/138): rbd-nbd-18.2.4-860.g6c6b083f.el9.x86_ 3.4 MB/s | 172 kB 00:00 2024-09-13T17:05:53.984 INFO:teuthology.orchestra.run.smithi028.stdout:(27/138): ceph-grafana-dashboards-18.2.4-860.g6 488 kB/s | 24 kB 00:00 2024-09-13T17:05:54.050 INFO:teuthology.orchestra.run.smithi028.stdout:(28/138): rbd-mirror-18.2.4-860.g6c6b083f.el9.x 15 MB/s | 3.0 MB 00:00 2024-09-13T17:05:54.075 INFO:teuthology.orchestra.run.smithi028.stdout:(29/138): ceph-mgr-cephadm-18.2.4-860.g6c6b083f 1.5 MB/s | 139 kB 00:00 2024-09-13T17:05:54.146 INFO:teuthology.orchestra.run.smithi189.stdout:(45/138): ceph-test-18.2.4-860.g6c6b083f.el9.x8 12 MB/s | 40 MB 00:03 2024-09-13T17:05:54.179 INFO:teuthology.orchestra.run.smithi189.stdout:(46/138): python3-cryptography-36.0.1-4.el9.x86 1.9 MB/s | 1.2 MB 00:00 2024-09-13T17:05:54.213 INFO:teuthology.orchestra.run.smithi189.stdout:(47/138): python3-urllib3-1.26.5-6.el9.noarch.r 460 kB/s | 215 kB 00:00 2024-09-13T17:05:54.301 INFO:teuthology.orchestra.run.smithi028.stdout:(30/138): ceph-mgr-dashboard-18.2.4-860.g6c6b08 14 MB/s | 3.5 MB 00:00 2024-09-13T17:05:54.321 INFO:teuthology.orchestra.run.smithi189.stdout:(48/138): boost-program-options-1.75.0-8.el9.x8 991 kB/s | 107 kB 00:00 2024-09-13T17:05:54.372 INFO:teuthology.orchestra.run.smithi189.stdout:(49/138): automake-1.16.2-8.el9.noarch.rpm 3.0 MB/s | 689 kB 00:00 2024-09-13T17:05:54.397 INFO:teuthology.orchestra.run.smithi189.stdout:(50/138): elfutils-libelf-devel-0.191-4.el9.x86 290 kB/s | 22 kB 00:00 2024-09-13T17:05:54.443 INFO:teuthology.orchestra.run.smithi028.stdout:(31/138): ceph-mgr-diskprediction-local-18.2.4- 20 MB/s | 7.4 MB 00:00 2024-09-13T17:05:54.455 INFO:teuthology.orchestra.run.smithi189.stdout:(51/138): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 943 kB 00:00 2024-09-13T17:05:54.468 INFO:teuthology.orchestra.run.smithi028.stdout:(32/138): ceph-mgr-modules-core-18.2.4-860.g6c6 1.4 MB/s | 246 kB 00:00 2024-09-13T17:05:54.481 INFO:teuthology.orchestra.run.smithi189.stdout:(52/138): flexiblas-3.0.4-8.el9.x86_64.rpm 391 kB/s | 33 kB 00:00 2024-09-13T17:05:54.493 INFO:teuthology.orchestra.run.smithi028.stdout:(33/138): ceph-mgr-rook-18.2.4-860.g6c6b083f.el 982 kB/s | 49 kB 00:00 2024-09-13T17:05:54.514 INFO:teuthology.orchestra.run.smithi189.stdout:(53/138): flex-2.6.4-9.el9.x86_64.rpm 2.2 MB/s | 313 kB 00:00 2024-09-13T17:05:54.518 INFO:teuthology.orchestra.run.smithi028.stdout:(34/138): ceph-prometheus-alerts-18.2.4-860.g6c 330 kB/s | 16 kB 00:00 2024-09-13T17:05:54.540 INFO:teuthology.orchestra.run.smithi189.stdout:(54/138): flexiblas-openblas-openmp-3.0.4-8.el9 310 kB/s | 18 kB 00:00 2024-09-13T17:05:54.544 INFO:teuthology.orchestra.run.smithi028.stdout:(35/138): ceph-volume-18.2.4-860.g6c6b083f.el9. 5.2 MB/s | 264 kB 00:00 2024-09-13T17:05:54.569 INFO:teuthology.orchestra.run.smithi028.stdout:(36/138): cephadm-18.2.4-860.g6c6b083f.el9.noar 4.4 MB/s | 226 kB 00:00 2024-09-13T17:05:54.573 INFO:teuthology.orchestra.run.smithi189.stdout:(55/138): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-09-13T17:05:54.595 INFO:teuthology.orchestra.run.smithi028.stdout:(37/138): cephfs-top-18.2.4-860.g6c6b083f.el9.n 364 kB/s | 18 kB 00:00 2024-09-13T17:05:54.598 INFO:teuthology.orchestra.run.smithi189.stdout:(56/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-09-13T17:05:54.624 INFO:teuthology.orchestra.run.smithi189.stdout:(57/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-09-13T17:05:54.766 INFO:teuthology.orchestra.run.smithi189.stdout:(58/138): libtool-2.4.6-46.el9.x86_64.rpm 4.0 MB/s | 581 kB 00:00 2024-09-13T17:05:54.791 INFO:teuthology.orchestra.run.smithi189.stdout:(59/138): librdkafka-1.6.1-102.el9.x86_64.rpm 3.4 MB/s | 662 kB 00:00 2024-09-13T17:05:54.816 INFO:teuthology.orchestra.run.smithi189.stdout:(60/138): libuuid-devel-2.37.4-20.el9.x86_64.rp 488 kB/s | 24 kB 00:00 2024-09-13T17:05:54.841 INFO:teuthology.orchestra.run.smithi189.stdout:(61/138): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-09-13T17:05:54.892 INFO:teuthology.orchestra.run.smithi189.stdout:(62/138): openblas-0.3.26-2.el9.x86_64.rpm 770 kB/s | 38 kB 00:00 2024-09-13T17:05:54.926 INFO:teuthology.orchestra.run.smithi189.stdout:(63/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-09-13T17:05:55.053 INFO:teuthology.orchestra.run.smithi028.stdout:(38/138): fuse-2.9.9-16.el9.x86_64.rpm 165 kB/s | 80 kB 00:00 2024-09-13T17:05:55.093 INFO:teuthology.orchestra.run.smithi189.stdout:(64/138): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.7 MB/s | 3.0 MB 00:00 2024-09-13T17:05:55.376 INFO:teuthology.orchestra.run.smithi189.stdout:(65/138): postgresql-13.16-1.el9.x86_64.rpm 5.6 MB/s | 1.6 MB 00:00 2024-09-13T17:05:55.420 INFO:teuthology.orchestra.run.smithi028.stdout:(39/138): ceph-test-18.2.4-860.g6c6b083f.el9.x8 15 MB/s | 40 MB 00:02 2024-09-13T17:05:55.454 INFO:teuthology.orchestra.run.smithi028.stdout:(40/138): libquadmath-11.5.0-2.el9.x86_64.rpm 478 kB/s | 191 kB 00:00 2024-09-13T17:05:55.496 INFO:teuthology.orchestra.run.smithi028.stdout:(41/138): libgfortran-11.5.0-2.el9.x86_64.rpm 890 kB/s | 801 kB 00:00 2024-09-13T17:05:55.593 INFO:teuthology.orchestra.run.smithi189.stdout:(66/138): postgresql-contrib-13.16-1.el9.x86_64 4.1 MB/s | 910 kB 00:00 2024-09-13T17:05:55.612 INFO:teuthology.orchestra.run.smithi028.stdout:(42/138): python3-ply-3.11-14.el9.noarch.rpm 914 kB/s | 106 kB 00:00 2024-09-13T17:05:55.660 INFO:teuthology.orchestra.run.smithi189.stdout:(67/138): openssl-devel-3.2.2-6.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-09-13T17:05:55.730 INFO:teuthology.orchestra.run.smithi028.stdout:(43/138): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-09-13T17:05:55.752 INFO:teuthology.orchestra.run.smithi189.stdout:(68/138): openblas-openmp-0.3.26-2.el9.x86_64.r 5.7 MB/s | 4.9 MB 00:00 2024-09-13T17:05:55.777 INFO:teuthology.orchestra.run.smithi189.stdout:(69/138): postgresql-private-libs-13.16-1.el9.x 748 kB/s | 137 kB 00:00 2024-09-13T17:05:55.838 INFO:teuthology.orchestra.run.smithi028.stdout:(44/138): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-09-13T17:05:55.860 INFO:teuthology.orchestra.run.smithi189.stdout:(70/138): python3-devel-3.9.19-8.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-09-13T17:05:55.944 INFO:teuthology.orchestra.run.smithi189.stdout:(71/138): python3-jinja2-2.11.3-6.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-09-13T17:05:55.955 INFO:teuthology.orchestra.run.smithi028.stdout:(45/138): python3-urllib3-1.26.5-6.el9.noarch.r 1.8 MB/s | 215 kB 00:00 2024-09-13T17:05:55.987 INFO:teuthology.orchestra.run.smithi189.stdout:(72/138): python3-jsonpatch-1.21-16.el9.noarch. 614 kB/s | 26 kB 00:00 2024-09-13T17:05:56.028 INFO:teuthology.orchestra.run.smithi189.stdout:(73/138): python3-jsonpointer-2.0-4.el9.noarch. 465 kB/s | 19 kB 00:00 2024-09-13T17:05:56.039 INFO:teuthology.orchestra.run.smithi028.stdout:(46/138): python3-cryptography-36.0.1-4.el9.x86 2.1 MB/s | 1.2 MB 00:00 2024-09-13T17:05:56.064 INFO:teuthology.orchestra.run.smithi028.stdout:(47/138): python3-cffi-1.14.5-5.el9.x86_64.rpm 394 kB/s | 253 kB 00:00 2024-09-13T17:05:56.087 INFO:teuthology.orchestra.run.smithi189.stdout:(74/138): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-13T17:05:56.129 INFO:teuthology.orchestra.run.smithi189.stdout:(75/138): python3-markupsafe-1.1.1-12.el9.x86_6 841 kB/s | 35 kB 00:00 2024-09-13T17:05:56.231 INFO:teuthology.orchestra.run.smithi028.stdout:(48/138): boost-program-options-1.75.0-8.el9.x8 642 kB/s | 107 kB 00:00 2024-09-13T17:05:56.273 INFO:teuthology.orchestra.run.smithi028.stdout:(49/138): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-09-13T17:05:56.298 INFO:teuthology.orchestra.run.smithi028.stdout:(50/138): elfutils-libelf-devel-0.191-4.el9.x86 326 kB/s | 22 kB 00:00 2024-09-13T17:05:56.331 INFO:teuthology.orchestra.run.smithi028.stdout:(51/138): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-09-13T17:05:56.357 INFO:teuthology.orchestra.run.smithi028.stdout:(52/138): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-09-13T17:05:56.382 INFO:teuthology.orchestra.run.smithi028.stdout:(53/138): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-09-13T17:05:56.407 INFO:teuthology.orchestra.run.smithi028.stdout:(54/138): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-09-13T17:05:56.432 INFO:teuthology.orchestra.run.smithi028.stdout:(55/138): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-09-13T17:05:56.488 INFO:teuthology.orchestra.run.smithi189.stdout:(76/138): postgresql-server-13.16-1.el9.x86_64. 7.0 MB/s | 5.8 MB 00:00 2024-09-13T17:05:56.517 INFO:teuthology.orchestra.run.smithi028.stdout:(56/138): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-09-13T17:05:56.534 INFO:teuthology.orchestra.run.smithi028.stdout:(57/138): librabbitmq-0.11.0-7.el9.x86_64.rpm 450 kB/s | 45 kB 00:00 2024-09-13T17:05:56.567 INFO:teuthology.orchestra.run.smithi028.stdout:(58/138): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-09-13T17:05:56.596 INFO:teuthology.orchestra.run.smithi189.stdout:(77/138): python3-babel-2.9.1-2.el9.noarch.rpm 7.1 MB/s | 6.0 MB 00:00 2024-09-13T17:05:56.600 INFO:teuthology.orchestra.run.smithi028.stdout:(59/138): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 662 kB 00:00 2024-09-13T17:05:56.621 INFO:teuthology.orchestra.run.smithi189.stdout:(78/138): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.8 MB/s | 250 kB 00:00 2024-09-13T17:05:56.626 INFO:teuthology.orchestra.run.smithi028.stdout:(60/138): libuuid-devel-2.37.4-20.el9.x86_64.rp 418 kB/s | 24 kB 00:00 2024-09-13T17:05:56.676 INFO:teuthology.orchestra.run.smithi028.stdout:(61/138): libtool-2.4.6-46.el9.x86_64.rpm 4.0 MB/s | 581 kB 00:00 2024-09-13T17:05:56.680 INFO:teuthology.orchestra.run.smithi189.stdout:(79/138): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 222 kB 00:00 2024-09-13T17:05:56.701 INFO:teuthology.orchestra.run.smithi028.stdout:(62/138): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-09-13T17:05:56.705 INFO:teuthology.orchestra.run.smithi189.stdout:(80/138): python3-packaging-20.9-5.el9.noarch.r 926 kB/s | 77 kB 00:00 2024-09-13T17:05:56.726 INFO:teuthology.orchestra.run.smithi028.stdout:(63/138): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-13T17:05:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:(64/138): openblas-0.3.26-2.el9.x86_64.rpm 511 kB/s | 38 kB 00:00 2024-09-13T17:05:56.802 INFO:teuthology.orchestra.run.smithi189.stdout:(81/138): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 159 kB 00:00 2024-09-13T17:05:56.889 INFO:teuthology.orchestra.run.smithi189.stdout:(82/138): python3-prettytable-0.7.2-27.el9.noar 202 kB/s | 42 kB 00:00 2024-09-13T17:05:56.902 INFO:teuthology.orchestra.run.smithi028.stdout:(65/138): openblas-openmp-0.3.26-2.el9.x86_64.r 25 MB/s | 4.9 MB 00:00 2024-09-13T17:05:56.922 INFO:teuthology.orchestra.run.smithi189.stdout:(83/138): python3-pyasn1-modules-0.4.8-6.el9.no 2.3 MB/s | 279 kB 00:00 2024-09-13T17:05:56.944 INFO:teuthology.orchestra.run.smithi028.stdout:(66/138): postgresql-13.16-1.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-09-13T17:05:56.977 INFO:teuthology.orchestra.run.smithi028.stdout:(67/138): postgresql-contrib-13.16-1.el9.x86_64 12 MB/s | 910 kB 00:00 2024-09-13T17:05:56.998 INFO:teuthology.orchestra.run.smithi189.stdout:(84/138): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.9 MB/s | 5.1 MB 00:00 2024-09-13T17:05:57.023 INFO:teuthology.orchestra.run.smithi189.stdout:(85/138): python3-pytz-2021.1-5.el9.noarch.rpm 382 kB/s | 51 kB 00:00 2024-09-13T17:05:57.024 INFO:teuthology.orchestra.run.smithi028.stdout:(68/138): postgresql-private-libs-13.16-1.el9.x 1.7 MB/s | 137 kB 00:00 2024-09-13T17:05:57.049 INFO:teuthology.orchestra.run.smithi189.stdout:(86/138): python3-requests-oauthlib-1.3.0-12.el 428 kB/s | 54 kB 00:00 2024-09-13T17:05:57.074 INFO:teuthology.orchestra.run.smithi189.stdout:(87/138): python3-toml-0.10.2-6.el9.noarch.rpm 829 kB/s | 42 kB 00:00 2024-09-13T17:05:57.111 INFO:teuthology.orchestra.run.smithi028.stdout:(69/138): openssl-devel-3.2.2-6.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-13T17:05:57.124 INFO:teuthology.orchestra.run.smithi189.stdout:(88/138): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-09-13T17:05:57.149 INFO:teuthology.orchestra.run.smithi189.stdout:(89/138): socat-1.7.4.1-6.el9.x86_64.rpm 3.0 MB/s | 304 kB 00:00 2024-09-13T17:05:57.178 INFO:teuthology.orchestra.run.smithi028.stdout:(70/138): python3-devel-3.9.19-8.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-09-13T17:05:57.191 INFO:teuthology.orchestra.run.smithi189.stdout:(90/138): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 1.4 MB/s | 96 kB 00:00 2024-09-13T17:05:57.224 INFO:teuthology.orchestra.run.smithi189.stdout:(91/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 853 kB/s | 64 kB 00:00 2024-09-13T17:05:57.249 INFO:teuthology.orchestra.run.smithi189.stdout:(92/138): zlib-devel-1.2.11-41.el9.x86_64.rpm 778 kB/s | 45 kB 00:00 2024-09-13T17:05:57.295 INFO:teuthology.orchestra.run.smithi028.stdout:(71/138): postgresql-server-13.16-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-09-13T17:05:57.317 INFO:teuthology.orchestra.run.smithi189.stdout:(93/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 4.5 MB/s | 308 kB 00:00 2024-09-13T17:05:57.387 INFO:teuthology.orchestra.run.smithi028.stdout:(72/138): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-13T17:05:57.412 INFO:teuthology.orchestra.run.smithi028.stdout:(73/138): python3-jinja2-2.11.3-6.el9.noarch.rp 1.0 MB/s | 248 kB 00:00 2024-09-13T17:05:57.433 INFO:teuthology.orchestra.run.smithi189.stdout:(94/138): fmt-8.1.1-5.el9.x86_64.rpm 531 kB/s | 111 kB 00:00 2024-09-13T17:05:57.454 INFO:teuthology.orchestra.run.smithi028.stdout:(74/138): python3-jsonpatch-1.21-16.el9.noarch. 385 kB/s | 26 kB 00:00 2024-09-13T17:05:57.458 INFO:teuthology.orchestra.run.smithi189.stdout:(95/138): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.0 MB/s | 26 kB 00:00 2024-09-13T17:05:57.496 INFO:teuthology.orchestra.run.smithi028.stdout:(75/138): python3-jsonpointer-2.0-4.el9.noarch. 176 kB/s | 19 kB 00:00 2024-09-13T17:05:57.525 INFO:teuthology.orchestra.run.smithi189.stdout:(96/138): libarrow-9.0.0-11.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-09-13T17:05:57.538 INFO:teuthology.orchestra.run.smithi028.stdout:(76/138): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-09-13T17:05:57.551 INFO:teuthology.orchestra.run.smithi189.stdout:(97/138): liboath-2.6.7-2.el9.x86_64.rpm 527 kB/s | 49 kB 00:00 2024-09-13T17:05:57.563 INFO:teuthology.orchestra.run.smithi028.stdout:(77/138): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-13T17:05:57.576 INFO:teuthology.orchestra.run.smithi189.stdout:(98/138): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-13T17:05:57.597 INFO:teuthology.orchestra.run.smithi028.stdout:(78/138): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-09-13T17:05:57.622 INFO:teuthology.orchestra.run.smithi028.stdout:(79/138): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-09-13T17:05:57.635 INFO:teuthology.orchestra.run.smithi189.stdout:(99/138): python3-asyncssh-2.13.2-5.el9.noarch. 9.2 MB/s | 548 kB 00:00 2024-09-13T17:05:57.647 INFO:teuthology.orchestra.run.smithi028.stdout:(80/138): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-09-13T17:05:57.660 INFO:teuthology.orchestra.run.smithi189.stdout:(100/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-09-13T17:05:57.685 INFO:teuthology.orchestra.run.smithi189.stdout:(101/138): python3-cachetools-4.2.4-1.el9.noarc 1.3 MB/s | 32 kB 00:00 2024-09-13T17:05:57.689 INFO:teuthology.orchestra.run.smithi028.stdout:(81/138): python3-prettytable-0.7.2-27.el9.noar 632 kB/s | 42 kB 00:00 2024-09-13T17:05:57.715 INFO:teuthology.orchestra.run.smithi028.stdout:(82/138): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-09-13T17:05:57.727 INFO:teuthology.orchestra.run.smithi189.stdout:(102/138): python3-certifi-2023.05.07-4.el9.noa 342 kB/s | 14 kB 00:00 2024-09-13T17:05:57.748 INFO:teuthology.orchestra.run.smithi028.stdout:(83/138): python3-pyasn1-modules-0.4.8-6.el9.no 4.7 MB/s | 279 kB 00:00 2024-09-13T17:05:57.761 INFO:teuthology.orchestra.run.smithi189.stdout:(103/138): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-09-13T17:05:57.790 INFO:teuthology.orchestra.run.smithi028.stdout:(84/138): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-13T17:05:57.794 INFO:teuthology.orchestra.run.smithi189.stdout:(104/138): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-09-13T17:05:57.828 INFO:teuthology.orchestra.run.smithi189.stdout:(105/138): python3-google-auth-2.34.0-1.el9.noa 6.5 MB/s | 220 kB 00:00 2024-09-13T17:05:57.832 INFO:teuthology.orchestra.run.smithi028.stdout:(85/138): python3-requests-oauthlib-1.3.0-12.el 644 kB/s | 54 kB 00:00 2024-09-13T17:05:57.853 INFO:teuthology.orchestra.run.smithi189.stdout:(106/138): python3-cherrypy-18.6.1-2.el9.noarch 3.8 MB/s | 358 kB 00:00 2024-09-13T17:05:57.878 INFO:teuthology.orchestra.run.smithi189.stdout:(107/138): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-09-13T17:05:57.907 INFO:teuthology.orchestra.run.smithi028.stdout:(86/138): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-09-13T17:05:57.912 INFO:teuthology.orchestra.run.smithi189.stdout:(108/138): python3-jaraco-classes-3.2.1-5.el9.n 303 kB/s | 18 kB 00:00 2024-09-13T17:05:57.938 INFO:teuthology.orchestra.run.smithi189.stdout:(109/138): python3-jaraco-collections-3.0.0-8.e 394 kB/s | 23 kB 00:00 2024-09-13T17:05:57.949 INFO:teuthology.orchestra.run.smithi028.stdout:(87/138): python3-toml-0.10.2-6.el9.noarch.rpm 356 kB/s | 42 kB 00:00 2024-09-13T17:05:57.963 INFO:teuthology.orchestra.run.smithi189.stdout:(110/138): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-09-13T17:05:57.974 INFO:teuthology.orchestra.run.smithi028.stdout:(88/138): socat-1.7.4.1-6.el9.x86_64.rpm 4.4 MB/s | 304 kB 00:00 2024-09-13T17:05:57.988 INFO:teuthology.orchestra.run.smithi189.stdout:(111/138): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-09-13T17:05:58.013 INFO:teuthology.orchestra.run.smithi189.stdout:(112/138): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-09-13T17:05:58.016 INFO:teuthology.orchestra.run.smithi028.stdout:(89/138): uuid-1.6.2-55.el9.x86_64.rpm 862 kB/s | 58 kB 00:00 2024-09-13T17:05:58.039 INFO:teuthology.orchestra.run.smithi189.stdout:(113/138): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-09-13T17:05:58.064 INFO:teuthology.orchestra.run.smithi189.stdout:(114/138): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-09-13T17:05:58.075 INFO:teuthology.orchestra.run.smithi028.stdout:(90/138): xfsprogs-devel-6.4.0-4.el9.x86_64.rpm 954 kB/s | 96 kB 00:00 2024-09-13T17:05:58.098 INFO:teuthology.orchestra.run.smithi189.stdout:(115/138): python3-pecan-1.4.2-3.el9.noarch.rpm 8.0 MB/s | 272 kB 00:00 2024-09-13T17:05:58.117 INFO:teuthology.orchestra.run.smithi028.stdout:(91/138): xmlstarlet-1.6.1-20.el9.x86_64.rpm 633 kB/s | 64 kB 00:00 2024-09-13T17:05:58.123 INFO:teuthology.orchestra.run.smithi189.stdout:(116/138): python3-portend-3.1.0-2.el9.noarch.r 659 kB/s | 16 kB 00:00 2024-09-13T17:05:58.157 INFO:teuthology.orchestra.run.smithi189.stdout:(117/138): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-09-13T17:05:58.159 INFO:teuthology.orchestra.run.smithi028.stdout:(92/138): zlib-devel-1.2.11-41.el9.x86_64.rpm 537 kB/s | 45 kB 00:00 2024-09-13T17:05:58.182 INFO:teuthology.orchestra.run.smithi189.stdout:(118/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-09-13T17:05:58.207 INFO:teuthology.orchestra.run.smithi189.stdout:(119/138): python3-repoze-lru-0.7-16.el9.noarch 615 kB/s | 31 kB 00:00 2024-09-13T17:05:58.218 INFO:teuthology.orchestra.run.smithi028.stdout:(93/138): fmt-8.1.1-5.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2024-09-13T17:05:58.224 INFO:teuthology.orchestra.run.smithi189.stdout:(120/138): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-09-13T17:05:58.249 INFO:teuthology.orchestra.run.smithi189.stdout:(121/138): python3-routes-2.5.1-5.el9.noarch.rp 2.7 MB/s | 188 kB 00:00 2024-09-13T17:05:58.260 INFO:teuthology.orchestra.run.smithi028.stdout:(94/138): gperftools-libs-2.9.1-3.el9.x86_64.rp 3.0 MB/s | 308 kB 00:00 2024-09-13T17:05:58.266 INFO:teuthology.orchestra.run.smithi189.stdout:(122/138): python3-tempora-5.0.0-2.el9.noarch.r 855 kB/s | 36 kB 00:00 2024-09-13T17:05:58.292 INFO:teuthology.orchestra.run.smithi189.stdout:(123/138): python3-typing-extensions-4.12.2-2.e 1.7 MB/s | 74 kB 00:00 2024-09-13T17:05:58.316 INFO:teuthology.orchestra.run.smithi028.stdout:(95/138): libarrow-doc-9.0.0-11.el9.noarch.rpm 462 kB/s | 26 kB 00:00 2024-09-13T17:05:58.317 INFO:teuthology.orchestra.run.smithi189.stdout:(124/138): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-09-13T17:05:58.342 INFO:teuthology.orchestra.run.smithi189.stdout:(125/138): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-13T17:05:58.360 INFO:teuthology.orchestra.run.smithi028.stdout:(96/138): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-09-13T17:05:58.367 INFO:teuthology.orchestra.run.smithi189.stdout:(126/138): python3-zc-lockfile-2.0-10.el9.noarc 813 kB/s | 20 kB 00:00 2024-09-13T17:05:58.383 INFO:teuthology.orchestra.run.smithi028.stdout:(97/138): libunwind-1.6.2-1.el9.x86_64.rpm 2.7 MB/s | 67 kB 00:00 2024-09-13T17:05:58.401 INFO:teuthology.orchestra.run.smithi189.stdout:(127/138): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-13T17:05:58.435 INFO:teuthology.orchestra.run.smithi189.stdout:(128/138): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-09-13T17:05:58.450 INFO:teuthology.orchestra.run.smithi028.stdout:(98/138): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-09-13T17:05:58.509 INFO:teuthology.orchestra.run.smithi028.stdout:(99/138): python3-asyncssh-2.13.2-5.el9.noarch. 9.2 MB/s | 548 kB 00:00 2024-09-13T17:05:58.527 INFO:teuthology.orchestra.run.smithi189.stdout:(129/138): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-09-13T17:05:58.551 INFO:teuthology.orchestra.run.smithi028.stdout:(100/138): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-09-13T17:05:58.593 INFO:teuthology.orchestra.run.smithi028.stdout:(101/138): python3-cachetools-4.2.4-1.el9.noarc 776 kB/s | 32 kB 00:00 2024-09-13T17:05:58.635 INFO:teuthology.orchestra.run.smithi028.stdout:(102/138): python3-certifi-2023.05.07-4.el9.noa 340 kB/s | 14 kB 00:00 2024-09-13T17:05:58.669 INFO:teuthology.orchestra.run.smithi028.stdout:(103/138): python3-cheroot-8.6.0-4.el9.noarch.r 5.0 MB/s | 172 kB 00:00 2024-09-13T17:05:58.710 INFO:teuthology.orchestra.run.smithi189.stdout:(130/138): librados2-18.2.4-860.g6c6b083f.el9.x 12 MB/s | 3.3 MB 00:00 2024-09-13T17:05:58.719 INFO:teuthology.orchestra.run.smithi028.stdout:(104/138): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-09-13T17:05:58.894 INFO:teuthology.orchestra.run.smithi028.stdout:(105/138): python3-scipy-1.6.2-8.el9.x86_64.rpm 15 MB/s | 16 MB 00:01 2024-09-13T17:05:58.944 INFO:teuthology.orchestra.run.smithi189.stdout:(131/138): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-09-13T17:05:58.961 INFO:teuthology.orchestra.run.smithi028.stdout:(106/138): libarrow-9.0.0-11.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-09-13T17:05:58.994 INFO:teuthology.orchestra.run.smithi189.stdout:(132/138): librbd1-18.2.4-860.g6c6b083f.el9.x86 6.4 MB/s | 3.0 MB 00:00 2024-09-13T17:05:59.012 INFO:teuthology.orchestra.run.smithi028.stdout:(107/138): python3-jaraco-classes-3.2.1-5.el9.n 448 kB/s | 18 kB 00:00 2024-09-13T17:05:59.053 INFO:teuthology.orchestra.run.smithi028.stdout:(108/138): python3-jaraco-8.2.1-3.el9.noarch.rp 67 kB/s | 11 kB 00:00 2024-09-13T17:05:59.111 INFO:teuthology.orchestra.run.smithi189.stdout:(133/138): NetworkManager-team-1.49.90-1.el9.x8 350 kB/s | 41 kB 00:00 2024-09-13T17:05:59.112 INFO:teuthology.orchestra.run.smithi028.stdout:(109/138): python3-jaraco-collections-3.0.0-8.e 236 kB/s | 23 kB 00:00 2024-09-13T17:05:59.137 INFO:teuthology.orchestra.run.smithi028.stdout:(110/138): python3-google-auth-2.34.0-1.el9.noa 527 kB/s | 220 kB 00:00 2024-09-13T17:05:59.179 INFO:teuthology.orchestra.run.smithi028.stdout:(111/138): python3-jaraco-functools-3.5.0-2.el9 292 kB/s | 19 kB 00:00 2024-09-13T17:05:59.221 INFO:teuthology.orchestra.run.smithi028.stdout:(112/138): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-09-13T17:05:59.261 INFO:teuthology.orchestra.run.smithi189.stdout:(134/138): NetworkManager-tui-1.49.90-1.el9.x86 1.6 MB/s | 249 kB 00:00 2024-09-13T17:05:59.263 INFO:teuthology.orchestra.run.smithi028.stdout:(113/138): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-09-13T17:05:59.288 INFO:teuthology.orchestra.run.smithi028.stdout:(114/138): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-09-13T17:05:59.330 INFO:teuthology.orchestra.run.smithi028.stdout:(115/138): python3-natsort-7.1.1-5.el9.noarch.r 865 kB/s | 58 kB 00:00 2024-09-13T17:05:59.372 INFO:teuthology.orchestra.run.smithi028.stdout:(116/138): python3-portend-3.1.0-2.el9.noarch.r 395 kB/s | 16 kB 00:00 2024-09-13T17:05:59.397 INFO:teuthology.orchestra.run.smithi028.stdout:(117/138): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-09-13T17:05:59.422 INFO:teuthology.orchestra.run.smithi028.stdout:(118/138): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-09-13T17:05:59.462 INFO:teuthology.orchestra.run.smithi189.stdout:(135/138): NetworkManager-1.49.90-1.el9.x86_64. 3.1 MB/s | 2.3 MB 00:00 2024-09-13T17:05:59.464 INFO:teuthology.orchestra.run.smithi028.stdout:(119/138): python3-repoze-lru-0.7-16.el9.noarch 460 kB/s | 31 kB 00:00 2024-09-13T17:05:59.503 INFO:teuthology.orchestra.run.smithi189.stdout:(136/138): openssl-3.2.2-6.el9.x86_64.rpm 5.6 MB/s | 1.4 MB 00:00 2024-09-13T17:05:59.506 INFO:teuthology.orchestra.run.smithi028.stdout:(120/138): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-13T17:05:59.548 INFO:teuthology.orchestra.run.smithi028.stdout:(121/138): python3-tempora-5.0.0-2.el9.noarch.r 858 kB/s | 36 kB 00:00 2024-09-13T17:05:59.574 INFO:teuthology.orchestra.run.smithi028.stdout:(122/138): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-09-13T17:05:59.599 INFO:teuthology.orchestra.run.smithi028.stdout:(123/138): python3-typing-extensions-4.12.2-2.e 1.4 MB/s | 74 kB 00:00 2024-09-13T17:05:59.620 INFO:teuthology.orchestra.run.smithi189.stdout:(137/138): NetworkManager-libnm-1.49.90-1.el9.x 2.7 MB/s | 1.8 MB 00:00 2024-09-13T17:05:59.624 INFO:teuthology.orchestra.run.smithi028.stdout:(124/138): python3-websocket-client-1.2.3-2.el9 3.6 MB/s | 90 kB 00:00 2024-09-13T17:05:59.658 INFO:teuthology.orchestra.run.smithi028.stdout:(125/138): python3-kubernetes-26.1.0-2.el9.noar 2.0 MB/s | 1.0 MB 00:00 2024-09-13T17:05:59.683 INFO:teuthology.orchestra.run.smithi028.stdout:(126/138): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-13T17:05:59.733 INFO:teuthology.orchestra.run.smithi028.stdout:(127/138): python3-zc-lockfile-2.0-10.el9.noarc 266 kB/s | 20 kB 00:00 2024-09-13T17:05:59.758 INFO:teuthology.orchestra.run.smithi028.stdout:(128/138): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-13T17:05:59.792 INFO:teuthology.orchestra.run.smithi028.stdout:(129/138): python3-werkzeug-2.0.3-3.el9.1.noarc 2.5 MB/s | 427 kB 00:00 2024-09-13T17:05:59.843 INFO:teuthology.orchestra.run.smithi028.stdout:(130/138): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-13T17:06:00.035 INFO:teuthology.orchestra.run.smithi028.stdout:(131/138): librados2-18.2.4-860.g6c6b083f.el9.x 12 MB/s | 3.3 MB 00:00 2024-09-13T17:06:00.102 INFO:teuthology.orchestra.run.smithi028.stdout:(132/138): librbd1-18.2.4-860.g6c6b083f.el9.x86 9.7 MB/s | 3.0 MB 00:00 2024-09-13T17:06:00.310 INFO:teuthology.orchestra.run.smithi028.stdout:(133/138): NetworkManager-team-1.49.90-1.el9.x8 195 kB/s | 41 kB 00:00 2024-09-13T17:06:00.594 INFO:teuthology.orchestra.run.smithi028.stdout:(134/138): NetworkManager-tui-1.49.90-1.el9.x86 879 kB/s | 249 kB 00:00 2024-09-13T17:06:00.679 INFO:teuthology.orchestra.run.smithi189.stdout:(138/138): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-09-13T17:06:00.685 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-09-13T17:06:00.685 INFO:teuthology.orchestra.run.smithi189.stdout:Total 18 MB/s | 203 MB 00:11 2024-09-13T17:06:00.936 INFO:teuthology.orchestra.run.smithi028.stdout:(135/138): NetworkManager-1.49.90-1.el9.x86_64. 2.1 MB/s | 2.3 MB 00:01 2024-09-13T17:06:01.065 INFO:teuthology.orchestra.run.smithi028.stdout:(136/138): NetworkManager-libnm-1.49.90-1.el9.x 1.8 MB/s | 1.8 MB 00:01 2024-09-13T17:06:01.145 INFO:teuthology.orchestra.run.smithi028.stdout:(137/138): openssl-3.2.2-6.el9.x86_64.rpm 2.5 MB/s | 1.4 MB 00:00 2024-09-13T17:06:01.894 INFO:teuthology.orchestra.run.smithi028.stdout:(138/138): openssl-libs-3.2.2-6.el9.x86_64.rpm 2.5 MB/s | 2.4 MB 00:00 2024-09-13T17:06:01.901 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-09-13T17:06:01.901 INFO:teuthology.orchestra.run.smithi028.stdout:Total 19 MB/s | 203 MB 00:10 2024-09-13T17:06:02.524 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-09-13T17:06:02.696 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-09-13T17:06:02.696 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-09-13T17:06:03.736 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-09-13T17:06:03.926 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-09-13T17:06:03.926 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-09-13T17:06:04.899 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-09-13T17:06:04.900 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-09-13T17:06:06.178 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-09-13T17:06:06.187 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-09-13T17:06:10.734 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-09-13T17:06:10.892 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/146 2024-09-13T17:06:10.951 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/146 2024-09-13T17:06:11.016 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/146 2024-09-13T17:06:11.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/146 2024-09-13T17:06:11.243 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/146 2024-09-13T17:06:11.303 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/146 2024-09-13T17:06:11.841 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/146 2024-09-13T17:06:11.934 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:06:12.088 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:06:12.119 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:06:12.202 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-09-13T17:06:12.231 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:06:12.266 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 10/146 2024-09-13T17:06:12.341 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.2-6.el9.x86_64 1/146 2024-09-13T17:06:12.392 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/146 2024-09-13T17:06:12.459 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/146 2024-09-13T17:06:12.476 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/146 2024-09-13T17:06:12.502 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/146 2024-09-13T17:06:12.534 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/146 2024-09-13T17:06:12.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/146 2024-09-13T17:06:12.573 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:06:12.591 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/146 2024-09-13T17:06:12.687 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:06:12.724 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 14/146 2024-09-13T17:06:12.975 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/146 2024-09-13T17:06:13.016 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 14/146 2024-09-13T17:06:13.084 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:06:13.187 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 15/146 2024-09-13T17:06:13.228 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:06:13.268 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:06:13.315 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:13.382 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:06:13.415 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 10/146 2024-09-13T17:06:13.422 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:13.634 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.16-1.el9.x86_64 11/146 2024-09-13T17:06:13.688 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.16-1.el9.x86_64 12/146 2024-09-13T17:06:13.738 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:06:13.783 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:13.819 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/146 2024-09-13T17:06:13.863 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/146 2024-09-13T17:06:13.863 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:06:13.889 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 14/146 2024-09-13T17:06:13.945 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/146 2024-09-13T17:06:13.990 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/146 2024-09-13T17:06:14.029 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/146 2024-09-13T17:06:14.117 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/146 2024-09-13T17:06:14.136 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 14/146 2024-09-13T17:06:14.181 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 23/146 2024-09-13T17:06:14.216 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 24/146 2024-09-13T17:06:14.285 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 25/146 2024-09-13T17:06:14.294 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 15/146 2024-09-13T17:06:14.368 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 26/146 2024-09-13T17:06:14.404 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x 27/146 2024-09-13T17:06:14.416 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:14.449 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9 28/146 2024-09-13T17:06:14.532 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:14.542 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 29/146 2024-09-13T17:06:14.715 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/146 2024-09-13T17:06:14.788 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/146 2024-09-13T17:06:14.907 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.49.90-1.el9.x86_64 16/146 2024-09-13T17:06:14.960 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/146 2024-09-13T17:06:15.004 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/146 2024-09-13T17:06:15.088 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/146 2024-09-13T17:06:15.128 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 20/146 2024-09-13T17:06:15.170 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 21/146 2024-09-13T17:06:15.260 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 22/146 2024-09-13T17:06:15.324 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 23/146 2024-09-13T17:06:15.357 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 24/146 2024-09-13T17:06:15.404 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 25/146 2024-09-13T17:06:15.452 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/146 2024-09-13T17:06:15.494 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 26/146 2024-09-13T17:06:15.535 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x 27/146 2024-09-13T17:06:15.562 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/146 2024-09-13T17:06:15.581 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9 28/146 2024-09-13T17:06:15.617 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 34/146 2024-09-13T17:06:15.655 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 35/146 2024-09-13T17:06:15.668 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 29/146 2024-09-13T17:06:15.708 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/146 2024-09-13T17:06:15.840 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/146 2024-09-13T17:06:15.890 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 37/146 2024-09-13T17:06:15.905 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/146 2024-09-13T17:06:15.946 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 38/146 2024-09-13T17:06:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-20.el9.x86_64 39/146 2024-09-13T17:06:16.125 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 40/146 2024-09-13T17:06:16.185 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 41/146 2024-09-13T17:06:16.270 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 42/146 2024-09-13T17:06:16.309 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 43/146 2024-09-13T17:06:16.346 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 44/146 2024-09-13T17:06:16.404 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 45/146 2024-09-13T17:06:16.457 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 46/146 2024-09-13T17:06:16.506 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 47/146 2024-09-13T17:06:16.592 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/146 2024-09-13T17:06:16.697 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 48/146 2024-09-13T17:06:16.697 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/146 2024-09-13T17:06:16.749 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 34/146 2024-09-13T17:06:16.795 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 35/146 2024-09-13T17:06:16.848 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/146 2024-09-13T17:06:16.870 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 49/146 2024-09-13T17:06:17.053 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 37/146 2024-09-13T17:06:17.059 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 50/146 2024-09-13T17:06:17.103 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.2-6.el9.x86_64 38/146 2024-09-13T17:06:17.186 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 51/146 2024-09-13T17:06:17.235 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 52/146 2024-09-13T17:06:17.240 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-20.el9.x86_64 39/146 2024-09-13T17:06:17.276 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 53/146 2024-09-13T17:06:17.282 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 40/146 2024-09-13T17:06:17.307 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 54/146 2024-09-13T17:06:17.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 55/146 2024-09-13T17:06:17.352 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 41/146 2024-09-13T17:06:17.367 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fmt-8.1.1-5.el9.x86_64 56/146 2024-09-13T17:06:17.404 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 57/146 2024-09-13T17:06:17.437 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 42/146 2024-09-13T17:06:17.446 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 58/146 2024-09-13T17:06:17.474 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 43/146 2024-09-13T17:06:17.485 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 59/146 2024-09-13T17:06:17.511 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 44/146 2024-09-13T17:06:17.530 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 60/146 2024-09-13T17:06:17.570 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 45/146 2024-09-13T17:06:17.573 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 61/146 2024-09-13T17:06:17.606 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 62/146 2024-09-13T17:06:17.622 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 46/146 2024-09-13T17:06:17.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 63/146 2024-09-13T17:06:17.681 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 47/146 2024-09-13T17:06:17.682 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 64/146 2024-09-13T17:06:17.717 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 65/146 2024-09-13T17:06:17.767 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 66/146 2024-09-13T17:06:17.862 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 48/146 2024-09-13T17:06:17.888 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/146 2024-09-13T17:06:17.947 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 68/146 2024-09-13T17:06:18.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 49/146 2024-09-13T17:06:18.199 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 50/146 2024-09-13T17:06:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 51/146 2024-09-13T17:06:18.375 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 52/146 2024-09-13T17:06:18.429 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 53/146 2024-09-13T17:06:18.463 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 54/146 2024-09-13T17:06:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 55/146 2024-09-13T17:06:18.524 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fmt-8.1.1-5.el9.x86_64 56/146 2024-09-13T17:06:18.561 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 57/146 2024-09-13T17:06:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 58/146 2024-09-13T17:06:18.658 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 59/146 2024-09-13T17:06:18.695 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 60/146 2024-09-13T17:06:18.730 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 61/146 2024-09-13T17:06:18.763 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 62/146 2024-09-13T17:06:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 63/146 2024-09-13T17:06:18.839 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 64/146 2024-09-13T17:06:18.844 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 69/146 2024-09-13T17:06:18.874 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 65/146 2024-09-13T17:06:18.948 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 70/146 2024-09-13T17:06:18.950 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 66/146 2024-09-13T17:06:18.991 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 71/146 2024-09-13T17:06:19.079 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 67/146 2024-09-13T17:06:19.103 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 72/146 2024-09-13T17:06:19.150 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 73/146 2024-09-13T17:06:19.155 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 68/146 2024-09-13T17:06:19.386 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.26-2.el9.x86_64 74/146 2024-09-13T17:06:19.422 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 75/146 2024-09-13T17:06:19.533 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 76/146 2024-09-13T17:06:20.090 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 69/146 2024-09-13T17:06:20.156 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 70/146 2024-09-13T17:06:20.190 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 71/146 2024-09-13T17:06:20.310 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 72/146 2024-09-13T17:06:20.335 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 77/146 2024-09-13T17:06:20.356 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 73/146 2024-09-13T17:06:20.475 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T17:06:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.26-2.el9.x86_64 74/146 2024-09-13T17:06:20.653 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 75/146 2024-09-13T17:06:20.753 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 76/146 2024-09-13T17:06:21.599 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 77/146 2024-09-13T17:06:21.743 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T17:06:22.040 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T17:06:22.134 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 80/146 2024-09-13T17:06:22.178 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 81/146 2024-09-13T17:06:22.211 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 82/146 2024-09-13T17:06:22.547 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 83/146 2024-09-13T17:06:22.589 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 84/146 2024-09-13T17:06:22.735 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 84/146 2024-09-13T17:06:22.787 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 85/146 2024-09-13T17:06:22.835 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 86/146 2024-09-13T17:06:23.276 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 87/146 2024-09-13T17:06:23.309 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 88/146 2024-09-13T17:06:23.387 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T17:06:23.450 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 88/146 2024-09-13T17:06:23.477 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 89/146 2024-09-13T17:06:23.488 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 80/146 2024-09-13T17:06:23.533 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 81/146 2024-09-13T17:06:23.565 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-20.el9.x86_64 82/146 2024-09-13T17:06:23.915 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 83/146 2024-09-13T17:06:23.976 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 84/146 2024-09-13T17:06:24.089 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 84/146 2024-09-13T17:06:24.124 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 85/146 2024-09-13T17:06:24.174 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 86/146 2024-09-13T17:06:24.640 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 87/146 2024-09-13T17:06:24.671 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 88/146 2024-09-13T17:06:24.762 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 88/146 2024-09-13T17:06:24.789 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 89/146 2024-09-13T17:06:25.759 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:25.807 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:26.422 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:26.531 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 91/146 2024-09-13T17:06:27.011 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 91/146 2024-09-13T17:06:27.011 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T17:06:27.011 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T17:06:27.011 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:27.053 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 92/146 2024-09-13T17:06:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:27.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:27.829 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 90/146 2024-09-13T17:06:27.871 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 91/146 2024-09-13T17:06:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 91/146 2024-09-13T17:06:28.024 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-13T17:06:28.024 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-13T17:06:28.024 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:28.064 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 92/146 2024-09-13T17:06:39.469 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 92/146 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-09-13T17:06:39.470 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:39.663 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 93/146 2024-09-13T17:06:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 93/146 2024-09-13T17:06:39.700 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:39.700 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T17:06:39.700 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 92/146 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-09-13T17:06:40.665 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-09-13T17:06:40.666 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-09-13T17:06:40.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:40.865 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 93/146 2024-09-13T17:06:40.900 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 93/146 2024-09-13T17:06:40.900 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:40.900 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-13T17:06:40.901 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:41.258 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 94/146 2024-09-13T17:06:41.315 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 94/146 2024-09-13T17:06:41.315 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:41.315 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T17:06:41.315 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T17:06:41.316 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T17:06:41.316 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:41.541 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 95/146 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 95/146 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T17:06:41.593 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:42.075 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 96/146 2024-09-13T17:06:42.126 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 96/146 2024-09-13T17:06:42.126 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:42.126 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T17:06:42.126 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T17:06:42.126 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T17:06:42.127 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:42.250 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 97/146 2024-09-13T17:06:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 98/146 2024-09-13T17:06:42.374 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 99/146 2024-09-13T17:06:42.543 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 94/146 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 94/146 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-13T17:06:42.602 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:42.608 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 100/146 2024-09-13T17:06:42.680 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 101/146 2024-09-13T17:06:42.765 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 102/146 2024-09-13T17:06:42.827 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 95/146 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 95/146 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-13T17:06:42.878 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:42.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 103/146 2024-09-13T17:06:43.071 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/146 2024-09-13T17:06:43.128 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 105/146 2024-09-13T17:06:43.177 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 106/146 2024-09-13T17:06:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-16.el9.x86_64 107/146 2024-09-13T17:06:43.386 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 96/146 2024-09-13T17:06:43.436 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 96/146 2024-09-13T17:06:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-13T17:06:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T17:06:43.437 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-13T17:06:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:43.579 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 97/146 2024-09-13T17:06:43.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 98/146 2024-09-13T17:06:43.685 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 108/146 2024-09-13T17:06:43.713 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 99/146 2024-09-13T17:06:43.718 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 108/146 2024-09-13T17:06:43.779 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el 109/146 2024-09-13T17:06:43.859 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.e 110/146 2024-09-13T17:06:43.932 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 111/146 2024-09-13T17:06:43.957 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 100/146 2024-09-13T17:06:44.023 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 101/146 2024-09-13T17:06:44.113 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 102/146 2024-09-13T17:06:44.223 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 103/146 2024-09-13T17:06:44.394 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/146 2024-09-13T17:06:44.456 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 105/146 2024-09-13T17:06:44.504 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 106/146 2024-09-13T17:06:44.537 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-16.el9.x86_64 107/146 2024-09-13T17:06:45.053 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 108/146 2024-09-13T17:06:45.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 108/146 2024-09-13T17:06:45.125 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el 109/146 2024-09-13T17:06:45.155 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 111/146 2024-09-13T17:06:45.192 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 112/146 2024-09-13T17:06:45.196 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.e 110/146 2024-09-13T17:06:45.225 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 111/146 2024-09-13T17:06:46.289 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 112/146 2024-09-13T17:06:46.317 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 113/146 2024-09-13T17:06:46.453 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 113/146 2024-09-13T17:06:46.519 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 111/146 2024-09-13T17:06:46.569 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 112/146 2024-09-13T17:06:46.647 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9 114/146 2024-09-13T17:06:46.686 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 115/146 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 115/146 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T17:06:46.737 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:46.795 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 116/146 2024-09-13T17:06:46.821 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 116/146 2024-09-13T17:06:46.923 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 117/146 2024-09-13T17:06:46.996 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 118/146 2024-09-13T17:06:47.719 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 112/146 2024-09-13T17:06:47.752 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 113/146 2024-09-13T17:06:47.780 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-46.el9.x86_64 119/146 2024-09-13T17:06:47.814 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 120/146 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 120/146 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T17:06:47.866 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:47.895 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 113/146 2024-09-13T17:06:47.915 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 121/146 2024-09-13T17:06:47.964 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 121/146 2024-09-13T17:06:47.965 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:47.965 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T17:06:47.965 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:48.024 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 122/146 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 122/146 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T17:06:48.075 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:48.078 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9 114/146 2024-09-13T17:06:48.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 115/146 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 115/146 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-13T17:06:48.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:48.221 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 116/146 2024-09-13T17:06:48.248 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 116/146 2024-09-13T17:06:48.354 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 117/146 2024-09-13T17:06:48.399 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 123/146 2024-09-13T17:06:48.424 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 118/146 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 123/146 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T17:06:48.451 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:49.254 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-46.el9.x86_64 119/146 2024-09-13T17:06:49.290 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 120/146 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 120/146 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-13T17:06:49.342 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:49.391 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 121/146 2024-09-13T17:06:49.441 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 121/146 2024-09-13T17:06:49.441 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:49.441 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-13T17:06:49.441 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:49.499 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 122/146 2024-09-13T17:06:49.551 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 122/146 2024-09-13T17:06:49.551 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:49.551 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-09-13T17:06:49.552 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T17:06:49.552 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-09-13T17:06:49.552 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:49.892 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 123/146 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 123/146 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-13T17:06:49.944 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:06:52.487 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 124/146 2024-09-13T17:06:52.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 125/146 2024-09-13T17:06:52.580 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 126/146 2024-09-13T17:06:52.626 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 127/146 2024-09-13T17:06:52.862 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 128/146 2024-09-13T17:06:52.907 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 129/146 2024-09-13T17:06:52.937 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_6 130/146 2024-09-13T17:06:52.968 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 131/146 2024-09-13T17:06:53.013 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 132/146 2024-09-13T17:06:53.797 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:53.844 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:53.885 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:54.172 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 124/146 2024-09-13T17:06:54.214 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 125/146 2024-09-13T17:06:54.258 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 126/146 2024-09-13T17:06:54.294 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.4.0-4.el9.x86_64 127/146 2024-09-13T17:06:54.553 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 128/146 2024-09-13T17:06:54.600 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.16-1.el9.x86_64 129/146 2024-09-13T17:06:54.639 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_6 130/146 2024-09-13T17:06:54.703 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-team-1:1.49.90-1.el9.x86_64 131/146 2024-09-13T17:06:54.755 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : NetworkManager-tui-1:1.49.90-1.el9.x86_64 132/146 2024-09-13T17:06:55.162 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 134/146 2024-09-13T17:06:55.288 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 135/146 2024-09-13T17:06:55.343 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 136/146 2024-09-13T17:06:55.446 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 137/146 2024-09-13T17:06:55.483 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:55.485 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 138/146 2024-09-13T17:06:55.510 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 139/146 2024-09-13T17:06:55.528 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:55.555 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 140/146 2024-09-13T17:06:55.555 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T17:06:55.569 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.16-1.el9.x86_64 133/146 2024-09-13T17:06:55.662 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T17:06:55.662 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T17:06:55.805 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T17:06:55.829 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 143/146 2024-09-13T17:06:55.846 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:55.846 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:56.090 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:56.090 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T17:06:56.200 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T17:06:56.305 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:06:56.509 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:06:56.510 INFO:teuthology.orchestra.run.smithi189.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-13T17:06:56.510 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:06:56.954 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 134/146 2024-09-13T17:06:57.066 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.2-6.el9.x86_64 135/146 2024-09-13T17:06:57.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 136/146 2024-09-13T17:06:57.234 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 137/146 2024-09-13T17:06:57.268 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 138/146 2024-09-13T17:06:57.304 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-tui-1:1.48.8-1.el9.x86_64 139/146 2024-09-13T17:06:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.2.2-4.el9.x86_64 140/146 2024-09-13T17:06:57.339 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T17:06:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 141/146 2024-09-13T17:06:57.446 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T17:06:57.538 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 142/146 2024-09-13T17:06:57.562 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-team-1:1.48.8-1.el9.x86_64 143/146 2024-09-13T17:06:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:57.580 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:57.865 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-1:1.48.8-1.el9.x86_64 144/146 2024-09-13T17:06:57.865 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T17:06:57.958 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: NetworkManager-libnm-1:1.48.8-1.el9.x86_64 145/146 2024-09-13T17:06:58.042 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:06:58.244 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:06:58.244 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-09-13T17:06:58.245 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:01.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 1/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 2/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 3/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 4/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 5/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 6/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 7/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 10/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 11/146 2024-09-13T17:07:01.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 12/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_6 14/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 15/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 16/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 17/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 18/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 19/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9 20/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x 21/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 22/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 23/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 24/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 25/146 2024-09-13T17:07:01.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 26/146 2024-09-13T17:07:01.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 27/146 2024-09-13T17:07:01.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 28/146 2024-09-13T17:07:01.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.e 29/146 2024-09-13T17:07:01.609 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 30/146 2024-09-13T17:07:01.609 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 31/146 2024-09-13T17:07:01.609 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 32/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9 33/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 34/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el 35/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 36/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 37/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 38/146 2024-09-13T17:07:01.610 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 39/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 40/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 41/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 42/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 43/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 44/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 45/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 46/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 47/146 2024-09-13T17:07:01.611 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 48/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 49/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 50/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 51/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 52/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 53/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 54/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 55/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 56/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 57/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 58/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 59/146 2024-09-13T17:07:01.612 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 60/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 61/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 62/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 63/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 64/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 65/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 66/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.16-1.el9.x86_64 67/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 68/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 69/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 70/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 71/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 72/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 73/146 2024-09-13T17:07:01.613 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 74/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 75/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 76/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 77/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 80/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 82/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 83/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 84/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 85/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 87/146 2024-09-13T17:07:01.614 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 88/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 89/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 90/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 91/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 93/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 94/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 95/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 96/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 97/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 100/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/146 2024-09-13T17:07:01.615 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 104/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 107/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 113/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 114/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 115/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 116/146 2024-09-13T17:07:01.616 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 117/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 118/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 120/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 121/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 122/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 123/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 124/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 125/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 126/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 127/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 128/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-20.el9.x86_64 129/146 2024-09-13T17:07:01.617 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 130/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 131/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 132/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 133/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 134/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 135/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 136/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 137/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 138/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 139/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 140/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 141/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 142/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 143/146 2024-09-13T17:07:01.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 144/146 2024-09-13T17:07:01.619 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 145/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 1/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 2/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 3/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 4/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-860.g6c6b08 5/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 6/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 7/146 2024-09-13T17:07:03.444 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 8/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 9/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 10/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 11/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 12/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 13/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_6 14/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 15/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 16/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 17/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_ 18/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 19/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9 20/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x 21/146 2024-09-13T17:07:03.445 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 22/146 2024-09-13T17:07:03.446 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 23/146 2024-09-13T17:07:03.446 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 24/146 2024-09-13T17:07:03.446 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 25/146 2024-09-13T17:07:03.446 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 26/146 2024-09-13T17:07:03.446 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 27/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 28/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.e 29/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noar 30/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.no 31/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b 32/146 2024-09-13T17:07:03.460 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9 33/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 34/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el 35/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 36/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 37/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 38/146 2024-09-13T17:07:03.461 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 39/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 40/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 41/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 42/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 43/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 44/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 45/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 46/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 47/146 2024-09-13T17:07:03.462 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 48/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 49/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 50/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 51/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 52/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 53/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 54/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 55/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 56/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 57/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 58/146 2024-09-13T17:07:03.463 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 59/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-46.el9.x86_64 60/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-20.el9.x86_64 61/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 62/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 63/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 64/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 65/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.2-6.el9.x86_64 66/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.16-1.el9.x86_64 67/146 2024-09-13T17:07:03.464 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.16-1.el9.x86_64 68/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.16-1.el9.x86_64 69/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.16-1.el9.x86_64 70/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 71/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 72/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 73/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 74/146 2024-09-13T17:07:03.465 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 75/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 76/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 77/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 78/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 79/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 80/146 2024-09-13T17:07:03.466 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 81/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 82/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 83/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 84/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 85/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/146 2024-09-13T17:07:03.467 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 87/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 88/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 89/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 90/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.4.0-4.el9.x86_64 91/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 92/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 93/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 94/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 95/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 96/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 97/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 98/146 2024-09-13T17:07:03.468 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 99/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 100/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 101/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 102/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 103/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 104/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 105/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 106/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 107/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 108/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 109/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 110/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 111/146 2024-09-13T17:07:03.469 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 112/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 113/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 114/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 115/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 116/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 117/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 118/146 2024-09-13T17:07:03.470 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 120/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 121/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 122/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 123/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 124/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 125/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 126/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 127/146 2024-09-13T17:07:03.471 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 128/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-20.el9.x86_64 129/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 130/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 131/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 132/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 133/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 134/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.49.90-1.el9.x86_64 135/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-1:1.48.8-1.el9.x86_64 136/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.49.90-1.el9.x86_64 137/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-libnm-1:1.48.8-1.el9.x86_64 138/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.49.90-1.el9.x86_64 139/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-team-1:1.48.8-1.el9.x86_64 140/146 2024-09-13T17:07:03.472 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.49.90-1.el9.x86_64 141/146 2024-09-13T17:07:03.473 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : NetworkManager-tui-1:1.48.8-1.el9.x86_64 142/146 2024-09-13T17:07:03.473 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-6.el9.x86_64 143/146 2024-09-13T17:07:03.473 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.2-4.el9.x86_64 144/146 2024-09-13T17:07:03.473 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-6.el9.x86_64 145/146 2024-09-13T17:07:03.761 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:07:03.762 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:03.762 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-09-13T17:07:03.762 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:03.762 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:03.762 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:03.763 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:03.763 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.763 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.763 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:03.763 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.764 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.765 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.765 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.765 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.765 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.765 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.766 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.767 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.767 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.767 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.767 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.767 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T17:07:03.768 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T17:07:03.769 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.770 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.771 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T17:07:03.772 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T17:07:03.773 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.774 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T17:07:03.775 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T17:07:03.776 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T17:07:03.776 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T17:07:03.776 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T17:07:03.776 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T17:07:03.776 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T17:07:03.777 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T17:07:03.778 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T17:07:03.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T17:07:03.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T17:07:03.779 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T17:07:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T17:07:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T17:07:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T17:07:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T17:07:03.780 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T17:07:03.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T17:07:03.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T17:07:03.781 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T17:07:03.782 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T17:07:03.783 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T17:07:03.784 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T17:07:03.785 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T17:07:03.785 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T17:07:03.785 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T17:07:03.786 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T17:07:03.786 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:03.786 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-09-13T17:07:04.405 DEBUG:teuthology.parallel:result is None 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.2-4.el9.x86_64 146/146 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-libnm-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-team-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: NetworkManager-tui-1:1.49.90-1.el9.x86_64 2024-09-13T17:07:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:05.652 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-09-13T17:07:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:18.2.4-860.g6c6b083f.el9.noarch 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-09-13T17:07:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-16.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-13T17:07:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-46.el9.x86_64 2024-09-13T17:07:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-20.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.2-6.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.16-1.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.16-1.el9.x86_64 2024-09-13T17:07:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.16-1.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.16-1.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-13T17:07:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-13T17:07:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-13T17:07:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-13T17:07:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-13T17:07:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:18.2.4-860.g6c6b083f.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-20.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-13T17:07:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-09-13T17:07:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.4.0-4.el9.x86_64 2024-09-13T17:07:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-13T17:07:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-09-13T17:07:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:05.665 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-09-13T17:07:06.281 DEBUG:teuthology.parallel:result is None 2024-09-13T17:07:06.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 2024-09-13T17:07:06.444 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T17:07:06.489 INFO:teuthology.orchestra.run.smithi028.stdout:18.2.4-860.g6c6b083f.el9 2024-09-13T17:07:06.490 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-860.g6c6b083f.el9 2024-09-13T17:07:06.490 INFO:teuthology.task.install:The correct ceph version 18.2.4-860.g6c6b083f is installed. 2024-09-13T17:07:06.492 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3 2024-09-13T17:07:06.648 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-13T17:07:06.693 INFO:teuthology.orchestra.run.smithi189.stdout:18.2.4-860.g6c6b083f.el9 2024-09-13T17:07:06.693 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-860.g6c6b083f.el9 2024-09-13T17:07:06.694 INFO:teuthology.task.install:The correct ceph version 18.2.4-860.g6c6b083f is installed. 2024-09-13T17:07:06.695 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T17:07:06.696 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:06.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T17:07:06.756 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:06.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T17:07:06.822 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T17:07:06.822 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:06.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T17:07:06.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T17:07:06.980 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:06.980 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T17:07:07.036 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T17:07:07.134 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T17:07:07.134 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:07.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T17:07:07.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T17:07:07.288 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:07.288 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T17:07:07.347 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T17:07:07.446 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T17:07:07.446 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:07.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T17:07:07.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T17:07:07.601 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:07.601 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T17:07:07.659 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T17:07:07.755 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T17:07:07.836 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T17:07:07.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /var/log/ceph 2024-09-13T17:07:07.841 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /var/log/ceph 2024-09-13T17:07:07.896 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T17:07:07.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T17:07:07.900 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T17:07:07.955 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T17:07:07.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T17:07:07.960 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T17:07:08.022 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-13T17:07:08.023 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-13T17:07:08.023 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-13T17:07:08.023 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:08.023 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T17:07:08.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:08.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:08.296 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:08.296 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T17:07:08.332 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:08.511 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:08.571 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T17:07:08.572 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T17:07:08.572 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869', 'branch': 'wip-rishabh-testing-20240910.100730-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub}', 'email': None, 'first_in_suite': False, 'job_id': '7903869', 'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240910.100730-reef'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240910.100730-reef', 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 6439, 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3', 'sleep_before_teardown': 0, 'subset': '964/1024', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240910.100730-reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi028.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKOsybv/7bBp+DjNREGwjWjG0yl+b5ZsEiopw5t4b1vNIOyDWeUbGD0sCEh7te8WSOoLvJVwm+4yfjJZopHMa5o=', 'smithi189.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPY6Rv1zqe55AvVQUhAISV60J/AzOPWUgUQv63hjfij+9CuWxRR0evY2NPbA2CmmneeE0yyMcMcVRhefvy8X/A='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'Replacing daemon mds', 'Scrub error on inode', 'Behind on trimming', 'Metadata damage detected', 'bad backtrace on inode', 'overall HEALTH_', '\\(MDS_TRIM\\)'], 'sha1': '6c6b083f148f500c4f56d402b9e4ffcab6f1e2d3', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_scrub_checks', 'tasks.cephfs.test_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_14:42:16', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T17:07:08.573 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T17:07:08.576 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T17:07:08.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T17:07:08.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T17:07:08.708 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:08.709 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T17:07:08.743 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T17:07:08.743 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 631 Links: 1 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:08.819 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 17:06:58.594678110 +0000 2024-09-13T17:07:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 17:05:03.678867746 +0000 2024-09-13T17:07:08.820 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 17:05:03.678867746 +0000 2024-09-13T17:07:08.820 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 17:05:03.678867746 +0000 2024-09-13T17:07:08.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T17:07:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:07:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:07:08.913 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000217106 s, 2.4 MB/s 2024-09-13T17:07:08.916 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T17:07:08.952 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:09.029 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 17:06:58.595678082 +0000 2024-09-13T17:07:09.030 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 17:05:04.124855261 +0000 2024-09-13T17:07:09.030 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 17:05:04.124855261 +0000 2024-09-13T17:07:09.030 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 17:05:04.124855261 +0000 2024-09-13T17:07:09.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T17:07:09.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:07:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:07:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000190325 s, 2.7 MB/s 2024-09-13T17:07:09.125 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T17:07:09.162 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 17:06:58.596678054 +0000 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 17:05:04.615841515 +0000 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 17:05:04.615841515 +0000 2024-09-13T17:07:09.237 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 17:05:04.615841515 +0000 2024-09-13T17:07:09.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T17:07:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:07:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:07:09.330 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000176011 s, 2.9 MB/s 2024-09-13T17:07:09.333 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T17:07:09.369 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-09-13 17:06:58.596678054 +0000 2024-09-13T17:07:09.445 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-09-13 17:05:05.124827266 +0000 2024-09-13T17:07:09.446 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-09-13 17:05:05.124827266 +0000 2024-09-13T17:07:09.446 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-09-13 17:05:05.124827266 +0000 2024-09-13T17:07:09.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T17:07:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:07:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:07:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000174405 s, 2.9 MB/s 2024-09-13T17:07:09.541 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T17:07:09.577 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:09.578 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:09.578 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T17:07:09.617 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T17:07:09.618 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 638 Links: 1 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-13 17:06:56.866962780 +0000 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-13 17:05:01.716257102 +0000 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-13 17:05:01.716257102 +0000 2024-09-13T17:07:09.696 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-09-13 17:05:01.716257102 +0000 2024-09-13T17:07:09.696 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T17:07:09.791 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-13T17:07:09.791 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-13T17:07:09.791 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000304551 s, 1.7 MB/s 2024-09-13T17:07:09.793 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T17:07:09.830 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 656 Links: 1 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-13 17:06:56.867962760 +0000 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-13 17:05:02.194247658 +0000 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-13 17:05:02.194247658 +0000 2024-09-13T17:07:09.906 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-09-13 17:05:02.194247658 +0000 2024-09-13T17:07:09.907 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T17:07:09.999 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-13T17:07:09.999 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-13T17:07:09.999 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000277145 s, 1.8 MB/s 2024-09-13T17:07:10.002 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T17:07:10.038 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-09-13T17:07:10.113 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 676 Links: 1 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-13 17:06:56.867962760 +0000 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-13 17:05:02.678238096 +0000 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-13 17:05:02.678238096 +0000 2024-09-13T17:07:10.114 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-09-13 17:05:02.678238096 +0000 2024-09-13T17:07:10.114 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T17:07:10.206 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-13T17:07:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-13T17:07:10.207 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000502402 s, 1.0 MB/s 2024-09-13T17:07:10.209 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T17:07:10.245 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-09-13T17:07:10.320 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T17:07:10.320 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T17:07:10.320 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 6h/6d Inode: 696 Links: 1 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-09-13 17:06:56.868962741 +0000 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-09-13 17:05:03.129229185 +0000 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-09-13 17:05:03.129229185 +0000 2024-09-13T17:07:10.321 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-09-13 17:05:03.129229185 +0000 2024-09-13T17:07:10.321 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T17:07:10.413 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-09-13T17:07:10.413 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-09-13T17:07:10.413 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.00029822 s, 1.7 MB/s 2024-09-13T17:07:10.416 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T17:07:10.452 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:10.452 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T17:07:10.452 INFO:tasks.ceph:Generating config... 2024-09-13T17:07:10.453 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-13T17:07:10.453 INFO:tasks.ceph:[client] debug client = 20 2024-09-13T17:07:10.454 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T17:07:10.454 INFO:tasks.ceph:[client] pid file = 2024-09-13T17:07:10.454 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-13T17:07:10.454 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-13T17:07:10.454 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds cache max size = 1000 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds log max segments = 1 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-13T17:07:10.454 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-13T17:07:10.455 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-13T17:07:10.455 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-13T17:07:10.455 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-13T17:07:10.455 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T17:07:10.456 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-13T17:07:10.457 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T17:07:10.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T17:07:10.553 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T17:07:10.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T17:07:10.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T17:07:10.702 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.28'), ('mon.b', '172.21.15.189'), ('mon.c', '[v2:172.21.15.189:3301,v1:172.21.15.189:6790]')] 2024-09-13T17:07:10.703 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds cache max size': 1000, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds log max segments': 1, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-13T17:07:10.703 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:10.704 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T17:07:10.738 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.28 --add b 172.21.15.189 --addv c '[v2:172.21.15.189:3301,v1:172.21.15.189:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: generated fsid 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:07:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = pacific 2024-09-13T17:07:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 0 2024-09-13T17:07:10.855 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:last_changed 2024-09-13T17:07:10.855001+0000 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-09-13T17:07:10.855001+0000 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:min_mon_release 16 (pacific) 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:election_strategy: 1 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.b 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:2: [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] mon.c 2024-09-13T17:07:10.856 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T17:07:10.858 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T17:07:10.893 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 99459b14-8688-430f-9c7e-baba0c3bf859... 2024-09-13T17:07:10.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:07:10.937 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T17:07:11.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:07:11.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.032 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T17:07:11.033 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.034 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-13T17:07:11.035 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.036 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:07:11.037 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.038 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:07:11.039 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T17:07:11.040 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds cache max size = 1000 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds log max segments = 1 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = True 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:07:11.041 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-13T17:07:11.042 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-13T17:07:11.043 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-13T17:07:11.044 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.045 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.046 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore allocator = bitmap 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 1/20 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 1/20 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:07:11.047 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 4/10 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:07:11.048 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-13T17:07:11.049 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-09-13T17:07:11.050 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-13T17:07:11.051 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: mds cache max size = 1000 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = True 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: mds log max segments = 1 2024-09-13T17:07:11.052 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-09-13T17:07:11.053 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = True 2024-09-13T17:07:11.053 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:07:11.053 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:07:11.053 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:07:11.060 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T17:07:11.060 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T17:07:11.157 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T17:07:11.157 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:11.157 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T17:07:11.192 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:11.192 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T17:07:11.267 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:07:11.267 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:11.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T17:07:11.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T17:07:11.384 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:11.385 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:11.459 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:07:11.460 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:11.460 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T17:07:11.460 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T17:07:11.539 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:11.539 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:11.613 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T17:07:11.613 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T17:07:11.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T17:07:11.717 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T17:07:11.723 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T17:07:11.832 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T17:07:11.838 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T17:07:11.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T17:07:11.945 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T17:07:11.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-13T17:07:12.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T17:07:12.109 INFO:teuthology.orchestra.run.smithi028.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T17:07:12.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-13T17:07:12.170 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T17:07:12.274 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T17:07:12.279 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-13T17:07:12.336 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T17:07:12.438 INFO:teuthology.orchestra.run.smithi189.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T17:07:12.444 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-13T17:07:12.499 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T17:07:12.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T17:07:12.583 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T17:07:12.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-13T17:07:12.692 INFO:teuthology.orchestra.run.smithi028.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-13T17:07:12.720 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T17:07:12.801 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-13T17:07:12.828 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-13T17:07:12.909 INFO:teuthology.orchestra.run.smithi189.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-13T17:07:12.936 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T17:07:12.937 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi028.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi189.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T17:07:12.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T17:07:12.994 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:12.994 INFO:tasks.ceph:role: osd.0 2024-09-13T17:07:12.994 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:07:12.994 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T17:07:13.048 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:13.049 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:13.082 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T17:07:13.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T17:07:13.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T17:07:13.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T17:07:13.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T17:07:13.287 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:13.287 INFO:tasks.ceph:role: osd.1 2024-09-13T17:07:13.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:07:13.287 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:13.344 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:13.345 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:13.345 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:13.377 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T17:07:13.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T17:07:13.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T17:07:13.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T17:07:13.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T17:07:13.584 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:13.584 INFO:tasks.ceph:role: osd.2 2024-09-13T17:07:13.584 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:07:13.584 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:13.640 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:13.673 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T17:07:13.680 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T17:07:13.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T17:07:13.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T17:07:13.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T17:07:13.881 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:13.881 INFO:tasks.ceph:role: osd.3 2024-09-13T17:07:13.881 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:07:13.881 DEBUG:teuthology.orchestra.run.smithi028:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:13.944 INFO:teuthology.orchestra.run.smithi028.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:13.945 INFO:teuthology.orchestra.run.smithi028.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:13.945 INFO:teuthology.orchestra.run.smithi028.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:13.945 INFO:teuthology.orchestra.run.smithi028.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:13.978 INFO:teuthology.orchestra.run.smithi028.stdout:Discarding blocks...Done. 2024-09-13T17:07:13.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi028.front.sepia.ceph.com -o noatime 2024-09-13T17:07:13.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-13T17:07:14.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T17:07:14.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.226+0000 7f4deeda2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T17:07:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.226+0000 7f4deeda2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T17:07:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.226+0000 7f4deeda2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T17:07:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.226+0000 7f4deeda2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T17:07:14.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.226+0000 7f4deeda2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T17:07:14.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:14.229+0000 7f4deeda2640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T17:07:17.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T17:07:17.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.319+0000 7f444f92e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T17:07:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.320+0000 7f444f92e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T17:07:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.320+0000 7f444f92e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T17:07:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.320+0000 7f444f92e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T17:07:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.320+0000 7f444f92e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T17:07:17.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:17.321+0000 7f444f92e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T17:07:20.246 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T17:07:20.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.396+0000 7f8640c0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T17:07:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.397+0000 7f8640c0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T17:07:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.397+0000 7f8640c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T17:07:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.397+0000 7f8640c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T17:07:20.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.397+0000 7f8640c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T17:07:20.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:20.398+0000 7f8640c0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T17:07:23.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T17:07:23.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.449+0000 7f2a770b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T17:07:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.450+0000 7f2a770b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T17:07:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.450+0000 7f2a770b6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T17:07:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.450+0000 7f2a770b6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T17:07:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.450+0000 7f2a770b6640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T17:07:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:23.451+0000 7f2a770b6640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T17:07:26.390 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T17:07:26.450 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T17:07:26.516 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:26.516 INFO:tasks.ceph:role: osd.4 2024-09-13T17:07:26.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:07:26.516 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:26.581 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:26.582 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:26.582 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:26.582 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:26.582 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:27.002 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-13T17:07:27.006 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-13T17:07:27.006 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-13T17:07:27.157 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T17:07:27.223 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T17:07:27.278 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:27.278 INFO:tasks.ceph:role: osd.5 2024-09-13T17:07:27.278 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:07:27.278 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T17:07:27.340 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:27.341 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:27.698 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-13T17:07:27.702 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-13T17:07:27.702 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T17:07:27.787 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T17:07:27.849 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T17:07:27.904 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:27.905 INFO:tasks.ceph:role: osd.6 2024-09-13T17:07:27.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:07:27.905 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T17:07:27.967 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:27.967 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:27.967 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:27.967 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:27.968 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:28.352 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-13T17:07:28.357 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-13T17:07:28.357 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-13T17:07:28.464 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T17:07:28.528 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T17:07:28.583 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T17:07:28.583 INFO:tasks.ceph:role: osd.7 2024-09-13T17:07:28.583 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:07:28.583 DEBUG:teuthology.orchestra.run.smithi189:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout: = sunit=0 swidth=0 blks 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-13T17:07:28.646 INFO:teuthology.orchestra.run.smithi189.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T17:07:28.647 INFO:teuthology.orchestra.run.smithi189.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T17:07:29.068 INFO:teuthology.orchestra.run.smithi189.stdout:Discarding blocks...Done. 2024-09-13T17:07:29.073 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi189.front.sepia.ceph.com -o noatime 2024-09-13T17:07:29.073 DEBUG:teuthology.orchestra.run.smithi189:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-13T17:07:29.211 DEBUG:teuthology.orchestra.run.smithi189:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T17:07:29.275 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:29.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.373+0000 7f165c945640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T17:07:29.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.374+0000 7f165c945640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T17:07:29.374 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.374+0000 7f165c945640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T17:07:29.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.374+0000 7f165c945640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T17:07:29.375 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.374+0000 7f165c945640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T17:07:29.381 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:29.381+0000 7f165c945640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T17:07:32.323 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T17:07:32.380 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:32.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.469+0000 7f97b5464640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T17:07:32.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.470+0000 7f97b5464640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T17:07:32.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.470+0000 7f97b5464640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T17:07:32.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.470+0000 7f97b5464640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T17:07:32.471 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.470+0000 7f97b5464640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T17:07:32.473 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:32.472+0000 7f97b5464640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T17:07:35.395 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T17:07:35.450 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:35.541 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.540+0000 7fccab43e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T17:07:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.541+0000 7fccab43e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T17:07:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.541+0000 7fccab43e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T17:07:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.541+0000 7fccab43e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T17:07:35.542 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.541+0000 7fccab43e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T17:07:35.543 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:35.543+0000 7fccab43e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T17:07:38.472 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T17:07:38.528 DEBUG:teuthology.orchestra.run.smithi189:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:38.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:38.578 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:07:38.619 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.619+0000 7fdd81a0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T17:07:38.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.619+0000 7fdd81a0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T17:07:38.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.619+0000 7fdd81a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T17:07:38.620 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.619+0000 7fdd81a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T17:07:38.621 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.619+0000 7fdd81a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T17:07:38.622 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:38.621+0000 7fdd81a0c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T17:07:41.550 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T17:07:41.606 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T17:07:41.606 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:41.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T17:07:41.669 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:41.669 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-13T17:07:41.767 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:41.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-13T17:07:41.864 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:41.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T17:07:41.962 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:41.962 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T17:07:42.058 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:42.058 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T17:07:42.153 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:42.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T17:07:42.248 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.248 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T17:07:42.309 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.309 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-13T17:07:42.404 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.404 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-13T17:07:42.501 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T17:07:42.596 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.596 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T17:07:42.691 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.691 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T17:07:42.786 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.786 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T17:07:42.882 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:42.882 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T17:07:42.920 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:42.921 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-13T17:07:42.998 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:42.998 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-13T17:07:43.033 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:43.033 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-13T17:07:43.109 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T17:07:43.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T17:07:43.113 DEBUG:teuthology.orchestra.run.smithi189:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.y] 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg== 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA== 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw== 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.0] 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g== 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.1] 2024-09-13T17:07:43.165 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.2] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.3] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr.x] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.4] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.5] 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A== 2024-09-13T17:07:43.166 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.6] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA== 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout:[osd.7] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA== 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA== 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout:[client.2] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA== 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout:[client.1] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg== 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout:[client.3] 2024-09-13T17:07:43.167 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ== 2024-09-13T17:07:43.201 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.y] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.0] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.1] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.2] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.3] 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg== 2024-09-13T17:07:43.202 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr.x] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.4] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.5] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.6] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[osd.7] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA== 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout:[client.0] 2024-09-13T17:07:43.203 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA== 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout:[client.2] 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA== 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout:[client.1] 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg== 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout:[client.3] 2024-09-13T17:07:43.204 INFO:teuthology.orchestra.run.smithi189.stdout: key = AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ== 2024-09-13T17:07:43.205 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T17:07:43.213 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T17:07:43.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:43.343 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:43.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:43.442 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:43.538 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.541 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.637 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.831 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:43.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T17:07:43.929 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T17:07:44.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:44.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:44.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:44.124 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T17:07:44.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.223 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.320 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.509 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.513 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T17:07:44.606 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.611 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.711 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.811 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:44.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T17:07:45.008 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T17:07:45.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T17:07:45.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T17:07:45.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T17:07:45.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T17:07:45.283 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T17:07:45.391 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T17:07:45.447 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T17:07:45.502 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T17:07:45.609 DEBUG:teuthology.orchestra.run.smithi189:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T17:07:45.664 DEBUG:teuthology.orchestra.run.smithi028:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:45.668 DEBUG:teuthology.orchestra.run.smithi189:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T17:07:45.702 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T17:07:45.702 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T17:07:45.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T17:07:45.744 INFO:tasks.ceph.mon.a:Started 2024-09-13T17:07:45.744 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T17:07:45.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T17:07:45.748 INFO:tasks.ceph.mon.b:Started 2024-09-13T17:07:45.748 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T17:07:45.748 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T17:07:45.751 INFO:tasks.ceph.mon.c:Started 2024-09-13T17:07:45.751 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T17:07:45.752 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T17:07:45.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T17:07:45.756 INFO:tasks.ceph.mgr.y:Started 2024-09-13T17:07:45.756 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T17:07:45.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T17:07:45.760 INFO:tasks.ceph.mgr.x:Started 2024-09-13T17:07:45.760 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T17:07:45.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T17:07:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.911+0000 7f8096800640 1 Processor -- start 2024-09-13T17:07:45.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.911+0000 7f8096800640 1 -- start start 2024-09-13T17:07:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.912+0000 7f8096800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.912+0000 7f8096800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.912+0000 7f8096800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809006c8b0 con 0x7f809006bb40 2024-09-13T17:07:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.912+0000 7f8096800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809006cb90 con 0x7f809015a780 2024-09-13T17:07:45.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.912+0000 7f8096800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900cc980 con 0x7f809015ab60 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:39756/0 (socket says 172.21.15.28:39756) 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 -- 172.21.15.28:0/2111560196 learned_addr learned my addr 172.21.15.28:0/2111560196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f80900712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f80900712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T17:07:45.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T17:07:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 msgr2=0x7f8090070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T17:07:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 msgr2=0x7f8090070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T17:07:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T17:07:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T17:07:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:45.913+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T17:07:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 msgr2=0x7f8090070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T17:07:46.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 msgr2=0x7f8090070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808ea00640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f80900712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808ea00640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f80900712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T17:07:46.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.114+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T17:07:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.119+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543084958 0 0) 0x7f809006c8b0 con 0x7f809006bb40 2024-09-13T17:07:46.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.119+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074003520 con 0x7f809006bb40 2024-09-13T17:07:46.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3766844677 0 0) 0x7f8074003520 con 0x7f809006bb40 2024-09-13T17:07:46.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f80900712c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T17:07:46.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f80900712c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 msgr2=0x7f8090070d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 --2- 172.21.15.28:0/2111560196 >> v2:172.21.15.189:3300/0 conn(0x7f809015a780 0x7f8090070d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900ccc60 con 0x7f809006bb40 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f80880019f0 con 0x7f809006bb40 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 >> v1:172.21.15.189:6789/0 conn(0x7f809006bb40 legacy=0x7f809006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f8074003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80740047c0 0x7f8074004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f8074005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809006cb90 con 0x7f8074003970 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900cc980 con 0x7f80740047c0 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807400bab0 con 0x7f809015ab60 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f8074003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808fe00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f8074005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.125+0000 7f808ea00640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80740047c0 0x7f8074004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.126+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 msgr2=0x7f8074005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.126+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809015ab60 0x7f8074005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.126+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80740047c0 msgr2=0x7f8074004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.126+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80740047c0 0x7f8074004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.126+0000 7f808f400640 1 -- 172.21.15.28:0/2111560196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8074003520 con 0x7f8074003970 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.127+0000 7f808f400640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f8074003d90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f80740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea251898aa52e7e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.128+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8080033410 con 0x7f8074003970 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.128+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808002d030 con 0x7f8074003970 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.128+0000 7f808e000640 1 -- 172.21.15.28:0/2111560196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80800337b0 con 0x7f8074003970 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 -- 172.21.15.28:0/2111560196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 msgr2=0x7f8074003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:46.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 --2- 172.21.15.28:0/2111560196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f8074003d90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8080006fd0 tx=0x7f80740057f0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f808fe00640 1 -- 172.21.15.28:0/2111560196 reap_dead start 2024-09-13T17:07:46.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 -- 172.21.15.28:0/2111560196 shutdown_connections 2024-09-13T17:07:46.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 -- 172.21.15.28:0/2111560196 >> 172.21.15.28:0/2111560196 conn(0x7f80900b0270 msgr2=0x7f80900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:46.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 -- 172.21.15.28:0/2111560196 shutdown_connections 2024-09-13T17:07:46.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.129+0000 7f8096800640 1 -- 172.21.15.28:0/2111560196 wait complete. 2024-09-13T17:07:46.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.132+0000 7f8096800640 1 Processor -- start 2024-09-13T17:07:46.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.132+0000 7f8096800640 1 -- start start 2024-09-13T17:07:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80900aa4c0 0x7f80900abb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 0x7f80900ce270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900ce7b0 con 0x7f8074003970 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80900ac430 con 0x7f80900aa4c0 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.133+0000 7f8096800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80900ac710 con 0x7f80900cddc0 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43510/0 (socket says 172.21.15.28:43510) 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 -- 172.21.15.28:0/1549574532 learned_addr learned my addr 172.21.15.28:0/1549574532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808ea00640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80900aa4c0 0x7f80900abb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808fe00640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 0x7f80900ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 -- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 msgr2=0x7f80900ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 0x7f80900ce270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 -- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80900aa4c0 msgr2=0x7f80900abb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80900aa4c0 0x7f80900abb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808f400640 1 -- 172.21.15.28:0/1549574532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900ac9f0 con 0x7f8074003970 2024-09-13T17:07:46.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.134+0000 7f808fe00640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 0x7f80900ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.135+0000 7f808f400640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8080036040 tx=0x7f80800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808003c600 con 0x7f8074003970 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8080033e90 con 0x7f8074003970 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808003c9c0 con 0x7f8074003970 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80900accd0 con 0x7f8074003970 2024-09-13T17:07:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80900ab6d0 con 0x7f8074003970 2024-09-13T17:07:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f808003a040 con 0x7f8074003970 2024-09-13T17:07:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.136+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8080047ac0 con 0x7f8074003970 2024-09-13T17:07:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.137+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8090070d80 con 0x7f8074003970 2024-09-13T17:07:46.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.142+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f808004c060 con 0x7f8074003970 2024-09-13T17:07:46.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.209+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f80900cb060 con 0x7f8074003970 2024-09-13T17:07:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.212+0000 7f8087400640 1 -- 172.21.15.28:0/1549574532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8080038070 con 0x7f8074003970 2024-09-13T17:07:46.213 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.213+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 msgr2=0x7f80900ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8080036040 tx=0x7f80800046c0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 shutdown_connections 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80900cddc0 0x7f80900ce270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80900aa4c0 0x7f80900abb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 --2- 172.21.15.28:0/1549574532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8074003970 0x7f80900ab190 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 >> 172.21.15.28:0/1549574532 conn(0x7f80900b0270 msgr2=0x7f8090156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 shutdown_connections 2024-09-13T17:07:46.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:46.214+0000 7f8096800640 1 -- 172.21.15.28:0/1549574532 wait complete. 2024-09-13T17:07:46.229 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T17:07:46.229 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T17:07:46.230 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:46.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T17:07:46.261 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.260+0000 7f929a62a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T17:07:46.263 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.261+0000 7fbede80c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T17:07:46.286 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:46.286 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T17:07:46.310 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.309+0000 7fbede80c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T17:07:46.312 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.311+0000 7f929a62a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T17:07:46.367 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:46.367 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T17:07:46.464 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:07:46.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T17:07:46.479 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.478+0000 7fbede80c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T17:07:46.504 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.503+0000 7f929a62a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T17:07:46.538 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.536+0000 7fbede80c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T17:07:46.560 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:46.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T17:07:46.567 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.566+0000 7f929a62a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T17:07:46.616 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:46.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T17:07:46.713 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:46.713 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T17:07:46.722 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.720+0000 7fbede80c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T17:07:46.768 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.768+0000 7f929a62a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T17:07:46.777 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:07:46.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T17:07:46.856 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:46.854+0000 7fbede80c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T17:07:46.874 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 83067adb-3344-4f9d-b7f5-33d98663ee01 0 2024-09-13T17:07:46.901 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:46.900+0000 7f929a62a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T17:07:47.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.060+0000 7fe290600640 1 Processor -- start 2024-09-13T17:07:47.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.061+0000 7fe290600640 1 -- start start 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe290600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2880ad5a0 0x7fe2880ad9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe290600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe28806d5b0 0x7fe28806da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe290600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2880cfab0 con 0x7fe28815cd90 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe290600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2880adf20 con 0x7fe28806d5b0 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe290600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2880cfd40 con 0x7fe2880ad5a0 2024-09-13T17:07:47.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2880ad5a0 0x7fe2880ad9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28dc00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe28815cd90 0x7fe28806b650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53728/0 (socket says 172.21.15.189:53728) 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28dc00640 1 -- 172.21.15.189:0/3507781223 learned_addr learned my addr 172.21.15.189:0/3507781223 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28e600640 1 --2- 172.21.15.189:0/3507781223 >> v2:172.21.15.28:3300/0 conn(0x7fe28806d5b0 0x7fe28806da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe287e00640 1 -- 172.21.15.189:0/3507781223 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437987990 0 0) 0x7fe2880cfab0 con 0x7fe28815cd90 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe287e00640 1 -- 172.21.15.189:0/3507781223 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe270003520 con 0x7fe28815cd90 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28d200640 1 -- 172.21.15.189:0/3507781223 >> v2:172.21.15.28:3300/0 conn(0x7fe28806d5b0 msgr2=0x7fe28806da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28d200640 1 --2- 172.21.15.189:0/3507781223 >> v2:172.21.15.28:3300/0 conn(0x7fe28806d5b0 0x7fe28806da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28d200640 1 -- 172.21.15.189:0/3507781223 >> v1:172.21.15.189:6789/0 conn(0x7fe28815cd90 legacy=0x7fe28806b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.062+0000 7fe28d200640 1 -- 172.21.15.189:0/3507781223 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28806a890 con 0x7fe2880ad5a0 2024-09-13T17:07:47.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.063+0000 7fe28d200640 1 --2- 172.21.15.189:0/3507781223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe2880ad9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe288072590 tx=0x7fe27402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb344fca61f67d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe287e00640 1 -- 172.21.15.189:0/3507781223 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274032440 con 0x7fe2880ad5a0 2024-09-13T17:07:47.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe287e00640 1 -- 172.21.15.189:0/3507781223 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2740325a0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe287e00640 1 -- 172.21.15.189:0/3507781223 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe274032910 con 0x7fe2880ad5a0 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 -- 172.21.15.189:0/3507781223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 msgr2=0x7fe2880ad9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 --2- 172.21.15.189:0/3507781223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe2880ad9e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe288072590 tx=0x7fe27402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 -- 172.21.15.189:0/3507781223 shutdown_connections 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 --2- 172.21.15.189:0/3507781223 >> v2:172.21.15.28:3300/0 conn(0x7fe28806d5b0 0x7fe28806da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 --2- 172.21.15.189:0/3507781223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe2880ad9e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 -- 172.21.15.189:0/3507781223 >> 172.21.15.189:0/3507781223 conn(0x7fe2880b22c0 msgr2=0x7fe2880b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 -- 172.21.15.189:0/3507781223 shutdown_connections 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.064+0000 7fe290600640 1 -- 172.21.15.189:0/3507781223 wait complete. 2024-09-13T17:07:47.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.066+0000 7fe290600640 1 Processor -- start 2024-09-13T17:07:47.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.066+0000 7fe290600640 1 -- start start 2024-09-13T17:07:47.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 0x7fe28815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28815cd90 0x7fe28815f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28815e280 con 0x7fe28815cd90 2024-09-13T17:07:47.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28815d600 con 0x7fe2880ad5a0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe290600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe28815d8e0 con 0x7fe28806d5b0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe28d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe28dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 0x7fe28815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe28d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39198/0 (socket says 172.21.15.189:39198) 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe28d200640 1 -- 172.21.15.189:0/2372792370 learned_addr learned my addr 172.21.15.189:0/2372792370 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.067+0000 7fe28e600640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28815cd90 0x7fe28815f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 -- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 msgr2=0x7fe28815e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 0x7fe28815e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 -- 172.21.15.189:0/2372792370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28815cd90 msgr2=0x7fe28815f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28815cd90 0x7fe28815f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 -- 172.21.15.189:0/2372792370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe274009400 con 0x7fe2880ad5a0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28dc00640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 0x7fe28815e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe28d200640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe274035040 tx=0x7fe2740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.068+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27403d8f0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe27403da70 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe27403de20 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28815dbc0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2881555a0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe274039040 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.069+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe27403b810 con 0x7fe2880ad5a0 2024-09-13T17:07:47.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.070+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28815e3c0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.072+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fe28815e3c0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.147 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.145+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "83067adb-3344-4f9d-b7f5-33d98663ee01", "id": 0} v 0) v1 -- 0x7fe2880ad9e0 con 0x7fe2880ad5a0 2024-09-13T17:07:47.150 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.148+0000 7fe285600640 1 -- 172.21.15.189:0/2372792370 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "83067adb-3344-4f9d-b7f5-33d98663ee01", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe27404b060 con 0x7fe2880ad5a0 2024-09-13T17:07:47.150 INFO:teuthology.orchestra.run.smithi189.stdout:0 2024-09-13T17:07:47.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 msgr2=0x7fe288155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe274035040 tx=0x7fe2740046c0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 shutdown_connections 2024-09-13T17:07:47.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe28815cd90 0x7fe28815f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.152 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2880ad5a0 0x7fe288155060 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 --2- 172.21.15.189:0/2372792370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe28806d5b0 0x7fe28815e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 >> 172.21.15.189:0/2372792370 conn(0x7fe2880b22c0 msgr2=0x7fe2880cb650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 shutdown_connections 2024-09-13T17:07:47.153 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.150+0000 7fe290600640 1 -- 172.21.15.189:0/2372792370 wait complete. 2024-09-13T17:07:47.166 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new ae399b9b-4170-40ef-9121-629474c54fb0 1 2024-09-13T17:07:47.232 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:47.228+0000 7fbede80c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T17:07:47.268 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:47.267+0000 7f929a62a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T17:07:47.291 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:47.289+0000 7fbede80c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T17:07:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.312+0000 7f9b9da00640 1 Processor -- start 2024-09-13T17:07:47.314 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.312+0000 7f9b9da00640 1 -- start start 2024-09-13T17:07:47.315 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b980cf980 0x7f9b980d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b980d19b0 con 0x7f9b9815ca40 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9da00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b980d1c90 con 0x7f9b980d1230 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b980d1f70 con 0x7f9b980cf980 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9d000640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f9b980d1230 0x7f9b980b5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53736/0 (socket says 172.21.15.189:53736) 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b9d000640 1 -- 172.21.15.189:0/792478261 learned_addr learned my addr 172.21.15.189:0/792478261 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-13T17:07:47.316 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b8fe00640 1 --2- 172.21.15.189:0/792478261 >> v2:172.21.15.28:3300/0 conn(0x7f9b980cf980 0x7f9b980d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530696084 0 0) 0x7f9b980d1c90 con 0x7f9b980d1230 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b7c003520 con 0x7f9b980d1230 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503830472 0 0) 0x7f9b980d19b0 con 0x7f9b9815ca40 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b980d1c90 con 0x7f9b9815ca40 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b8fe00640 1 -- 172.21.15.189:0/792478261 >> v1:172.21.15.189:6789/0 conn(0x7f9b980d1230 legacy=0x7f9b980b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.314+0000 7f9b8fe00640 1 -- 172.21.15.189:0/792478261 >> v1:172.21.15.28:6789/0 conn(0x7f9b9815ca40 legacy=0x7f9b980cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.317 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.315+0000 7f9b8fe00640 1 -- 172.21.15.189:0/792478261 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b980d2250 con 0x7f9b980cf980 2024-09-13T17:07:47.318 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b8fe00640 1 --2- 172.21.15.189:0/792478261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980cf980 0x7f9b980d0cf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9b800097c0 tx=0x7f9b8002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c63ce084762f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80032440 con 0x7f9b980cf980 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b800325a0 con 0x7f9b980cf980 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b97400640 1 -- 172.21.15.189:0/792478261 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b80032930 con 0x7f9b980cf980 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b9da00640 1 -- 172.21.15.189:0/792478261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980cf980 msgr2=0x7f9b980d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.316+0000 7f9b9da00640 1 --2- 172.21.15.189:0/792478261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980cf980 0x7f9b980d0cf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9b800097c0 tx=0x7f9b8002a250 comp rx=0 tx=0).stop 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.317+0000 7f9b9da00640 1 -- 172.21.15.189:0/792478261 shutdown_connections 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.317+0000 7f9b9da00640 1 --2- 172.21.15.189:0/792478261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980cf980 0x7f9b980d0cf0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.317+0000 7f9b9da00640 1 -- 172.21.15.189:0/792478261 >> 172.21.15.189:0/792478261 conn(0x7f9b98070ad0 msgr2=0x7f9b980b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.317+0000 7f9b9da00640 1 -- 172.21.15.189:0/792478261 shutdown_connections 2024-09-13T17:07:47.319 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.317+0000 7f9b9da00640 1 -- 172.21.15.189:0/792478261 wait complete. 2024-09-13T17:07:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.318+0000 7f9b9da00640 1 Processor -- start 2024-09-13T17:07:47.320 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.318+0000 7f9b9da00640 1 -- start start 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 0x7f9b980b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980b1f80 0x7f9b980cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9814d780 con 0x7f9b980b1f80 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9814fca0 con 0x7f9b980d1230 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.319+0000 7f9b9da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9814cde0 con 0x7f9b980cf980 2024-09-13T17:07:47.322 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 0x7f9b980b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b8fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39214/0 (socket says 172.21.15.189:39214) 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b8fe00640 1 -- 172.21.15.189:0/385283085 learned_addr learned my addr 172.21.15.189:0/385283085 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b9d000640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980b1f80 0x7f9b980cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 -- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 msgr2=0x7f9b980cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 -- 172.21.15.189:0/385283085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980b1f80 msgr2=0x7f9b980cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980b1f80 0x7f9b980cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 -- 172.21.15.189:0/385283085 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800093a0 con 0x7f9b980cf980 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b8fe00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:47.323 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.320+0000 7f9b97e00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 0x7f9b980b1a40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c3c0 tx=0x7f9b88001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b880052d0 con 0x7f9b980cf980 2024-09-13T17:07:47.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b88005430 con 0x7f9b980cf980 2024-09-13T17:07:47.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9814d0c0 con 0x7f9b980cf980 2024-09-13T17:07:47.324 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b88003830 con 0x7f9b980cf980 2024-09-13T17:07:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b980aaa50 con 0x7f9b980cf980 2024-09-13T17:07:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f9b88003c90 con 0x7f9b980cf980 2024-09-13T17:07:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.321+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f9b8801c440 con 0x7f9b980cf980 2024-09-13T17:07:47.325 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.322+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b980b4420 con 0x7f9b980cf980 2024-09-13T17:07:47.326 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.325+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f9b88017070 con 0x7f9b980cf980 2024-09-13T17:07:47.330 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:47.330+0000 7f929a62a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T17:07:47.397 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.395+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ae399b9b-4170-40ef-9121-629474c54fb0", "id": 1} v 0) v1 -- 0x7f9b980d0cf0 con 0x7f9b980cf980 2024-09-13T17:07:47.400 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.399+0000 7f9b94c00640 1 -- 172.21.15.189:0/385283085 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae399b9b-4170-40ef-9121-629474c54fb0", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9b88011590 con 0x7f9b980cf980 2024-09-13T17:07:47.401 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 msgr2=0x7f9b980b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 0x7f9b980b1a40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b8800c3c0 tx=0x7f9b88001e70 comp rx=0 tx=0).stop 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 shutdown_connections 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b980b1f80 0x7f9b980cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b980d1230 0x7f9b980cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 --2- 172.21.15.189:0/385283085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b980cf980 0x7f9b980b1a40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 >> 172.21.15.189:0/385283085 conn(0x7f9b98070ad0 msgr2=0x7f9b980b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 shutdown_connections 2024-09-13T17:07:47.402 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.400+0000 7f9b9da00640 1 -- 172.21.15.189:0/385283085 wait complete. 2024-09-13T17:07:47.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 8638f528-5f30-4d64-ad6f-25c517197b04 2 2024-09-13T17:07:47.469 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:47.467+0000 7fbede80c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T17:07:47.506 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:47.504+0000 7f929a62a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T17:07:47.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.563+0000 7f5082400640 1 Processor -- start 2024-09-13T17:07:47.565 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.563+0000 7f5082400640 1 -- start start 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.564+0000 7f5082400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f507c15dbf0 0x7f507c0cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.564+0000 7f5082400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f507c0cbc20 0x7f507c0ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.564+0000 7f5082400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507c0b4350 con 0x7f507c0ae580 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.564+0000 7f5082400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c15cf00 con 0x7f507c0cbc20 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.564+0000 7f5082400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507c0cc050 con 0x7f507c15dbf0 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f507c15dbf0 0x7f507c0cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f507c15dbf0 0x7f507c0cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39228/0 (socket says 172.21.15.189:39228) 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 -- 172.21.15.189:0/1673730756 learned_addr learned my addr 172.21.15.189:0/1673730756 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.567 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507be00640 1 --2- 172.21.15.189:0/1673730756 >> v2:172.21.15.28:3300/0 conn(0x7f507c0cbc20 0x7f507c0ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 -- 172.21.15.189:0/1673730756 >> v2:172.21.15.28:3300/0 conn(0x7f507c0cbc20 msgr2=0x7f507c0ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 --2- 172.21.15.189:0/1673730756 >> v2:172.21.15.28:3300/0 conn(0x7f507c0cbc20 0x7f507c0ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 -- 172.21.15.189:0/1673730756 >> v1:172.21.15.28:6789/0 conn(0x7f507c0ae580 legacy=0x7f507c0b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507aa00640 1 -- 172.21.15.189:0/1673730756 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507c0b4070 con 0x7f507c15dbf0 2024-09-13T17:07:47.568 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.565+0000 7f507a000640 1 -- 172.21.15.189:0/1673730756 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379686617 0 0) 0x7f506c001110 con 0x7f507c0ae580 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.566+0000 7f507aa00640 1 --2- 172.21.15.189:0/1673730756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c15dbf0 0x7f507c0cb6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f507c06e840 tx=0x7f506802a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97abbde439d338ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.566+0000 7f507a000640 1 -- 172.21.15.189:0/1673730756 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068032440 con 0x7f507c15dbf0 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.566+0000 7f507a000640 1 -- 172.21.15.189:0/1673730756 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50680325a0 con 0x7f507c15dbf0 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.566+0000 7f507a000640 1 -- 172.21.15.189:0/1673730756 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068032930 con 0x7f507c15dbf0 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 -- 172.21.15.189:0/1673730756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c15dbf0 msgr2=0x7f507c0cb6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 --2- 172.21.15.189:0/1673730756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c15dbf0 0x7f507c0cb6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f507c06e840 tx=0x7f506802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 -- 172.21.15.189:0/1673730756 shutdown_connections 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 --2- 172.21.15.189:0/1673730756 >> v2:172.21.15.28:3300/0 conn(0x7f507c0cbc20 0x7f507c0ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.569 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 --2- 172.21.15.189:0/1673730756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c15dbf0 0x7f507c0cb6e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 -- 172.21.15.189:0/1673730756 >> 172.21.15.189:0/1673730756 conn(0x7f507c0b0af0 msgr2=0x7f507c0b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 -- 172.21.15.189:0/1673730756 shutdown_connections 2024-09-13T17:07:47.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.567+0000 7f5082400640 1 -- 172.21.15.189:0/1673730756 wait complete. 2024-09-13T17:07:47.570 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.569+0000 7f5082400640 1 Processor -- start 2024-09-13T17:07:47.571 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.569+0000 7f5082400640 1 -- start start 2024-09-13T17:07:47.572 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0ae580 0x7f507c15c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f507c15dbf0 0x7f507c1f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c15a1c0 con 0x7f507c0ae580 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507c1595b0 con 0x7f507c0cbc20 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f5082400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507c159890 con 0x7f507c15dbf0 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f507aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f507b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0ae580 0x7f507c15c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f507be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f507c15dbf0 0x7f507c1f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f507aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39240/0 (socket says 172.21.15.189:39240) 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.570+0000 7f507aa00640 1 -- 172.21.15.189:0/1859461616 learned_addr learned my addr 172.21.15.189:0/1859461616 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 -- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f507c15dbf0 msgr2=0x7f507c1f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.573 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f507c15dbf0 0x7f507c1f2800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 -- 172.21.15.189:0/1859461616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0ae580 msgr2=0x7f507c15c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0ae580 0x7f507c15c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 -- 172.21.15.189:0/1859461616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50680094d0 con 0x7f507c0cbc20 2024-09-13T17:07:47.574 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f507aa00640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f506802a220 tx=0x7f50680046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068032440 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5068030980 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5068030d10 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f507c159b70 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.571+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507c156b30 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.572+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5068039040 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.572+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f5068046820 con 0x7f507c0cbc20 2024-09-13T17:07:47.575 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.572+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507c15a330 con 0x7f507c0cbc20 2024-09-13T17:07:47.577 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.575+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f5068064e30 con 0x7f507c0cbc20 2024-09-13T17:07:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.645+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8638f528-5f30-4d64-ad6f-25c517197b04", "id": 2} v 0) v1 -- 0x7f507c0ad4e0 con 0x7f507c0cbc20 2024-09-13T17:07:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.648+0000 7f5073400640 1 -- 172.21.15.189:0/1859461616 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8638f528-5f30-4d64-ad6f-25c517197b04", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5068053030 con 0x7f507c0cbc20 2024-09-13T17:07:47.650 INFO:teuthology.orchestra.run.smithi189.stdout:2 2024-09-13T17:07:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.649+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 msgr2=0x7f507c14c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.649+0000 7f5082400640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f506802a220 tx=0x7f50680046c0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.649+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 shutdown_connections 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.649+0000 7f5082400640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f507c15dbf0 0x7f507c1f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.650+0000 7f5082400640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f507c0cbc20 0x7f507c14c630 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.650+0000 7f5082400640 1 --2- 172.21.15.189:0/1859461616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c0ae580 0x7f507c15c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.650+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 >> 172.21.15.189:0/1859461616 conn(0x7f507c0b0af0 msgr2=0x7f507c0ccc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.650+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 shutdown_connections 2024-09-13T17:07:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.650+0000 7f5082400640 1 -- 172.21.15.189:0/1859461616 wait complete. 2024-09-13T17:07:47.667 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58 3 2024-09-13T17:07:47.815 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.814+0000 7ff899c00640 1 Processor -- start 2024-09-13T17:07:47.816 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.814+0000 7ff899c00640 1 -- start start 2024-09-13T17:07:47.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff89415c920 0x7ff8940d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.817 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 0x7ff8940cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8940cf1c0 0x7ff8940cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940b5b10 con 0x7ff89415c920 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8940d0b80 con 0x7ff8940cf1c0 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff899c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940d16a0 con 0x7ff89415cd00 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff893400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8940cf1c0 0x7ff8940cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff88be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 0x7ff8940cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff893400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff8940cf1c0 0x7ff8940cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39244/0 (socket says 172.21.15.189:39244) 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff893400640 1 -- 172.21.15.189:0/3828678703 learned_addr learned my addr 172.21.15.189:0/3828678703 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.815+0000 7ff892a00640 1 --2- 172.21.15.189:0/3828678703 >> v2:172.21.15.28:3300/0 conn(0x7ff89415c920 0x7ff8940d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff893400640 1 -- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 msgr2=0x7ff8940cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff893400640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 0x7ff8940cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.818 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff893400640 1 -- 172.21.15.189:0/3828678703 >> v2:172.21.15.28:3300/0 conn(0x7ff89415c920 msgr2=0x7ff8940d0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:47.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff893400640 1 --2- 172.21.15.189:0/3828678703 >> v2:172.21.15.28:3300/0 conn(0x7ff89415c920 0x7ff8940d0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff893400640 1 -- 172.21.15.189:0/3828678703 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8940d0e00 con 0x7ff8940cf1c0 2024-09-13T17:07:47.819 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.816+0000 7ff88be00640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 0x7ff8940cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff893400640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff8940cdad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff89408bdc0 tx=0x7ff88c02a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27d94de7b7aea797 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff892000640 1 -- 172.21.15.189:0/3828678703 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c031600 con 0x7ff8940cf1c0 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff892000640 1 -- 172.21.15.189:0/3828678703 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88c02d050 con 0x7ff8940cf1c0 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff892000640 1 -- 172.21.15.189:0/3828678703 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88c031980 con 0x7ff8940cf1c0 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff899c00640 1 -- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 msgr2=0x7ff8940cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.817+0000 7ff899c00640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff8940cdad0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff89408bdc0 tx=0x7ff88c02a730 comp rx=0 tx=0).stop 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 -- 172.21.15.189:0/3828678703 shutdown_connections 2024-09-13T17:07:47.820 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff8940cdad0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 --2- 172.21.15.189:0/3828678703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415cd00 0x7ff8940cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 --2- 172.21.15.189:0/3828678703 >> v2:172.21.15.28:3300/0 conn(0x7ff89415c920 0x7ff8940d0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 -- 172.21.15.189:0/3828678703 >> 172.21.15.189:0/3828678703 conn(0x7ff894069e80 msgr2=0x7ff89406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 -- 172.21.15.189:0/3828678703 shutdown_connections 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.818+0000 7ff899c00640 1 -- 172.21.15.189:0/3828678703 wait complete. 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.819+0000 7ff899c00640 1 Processor -- start 2024-09-13T17:07:47.821 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.819+0000 7ff899c00640 1 -- start start 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 0x7ff8940c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89415cd00 0x7ff8941f6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940c86a0 con 0x7ff89415cd00 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8940c6480 con 0x7ff8940cf1c0 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.820+0000 7ff899c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8941f6a70 con 0x7ff89415c920 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.823 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff88be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 0x7ff8940c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39248/0 (socket says 172.21.15.189:39248) 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 -- 172.21.15.189:0/3543172755 learned_addr learned my addr 172.21.15.189:0/3543172755 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff893400640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89415cd00 0x7ff8941f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 -- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 msgr2=0x7ff8940c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 0x7ff8940c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 -- 172.21.15.189:0/3543172755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89415cd00 msgr2=0x7ff8941f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89415cd00 0x7ff8941f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 -- 172.21.15.189:0/3543172755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c009400 con 0x7ff8940cf1c0 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff88be00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 0x7ff8940c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:47.824 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.821+0000 7ff892a00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e9b0 tx=0x7ff88400ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff884009670 con 0x7ff8940cf1c0 2024-09-13T17:07:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88400c9a0 con 0x7ff8940cf1c0 2024-09-13T17:07:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88c009b30 con 0x7ff8940cf1c0 2024-09-13T17:07:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88400cd50 con 0x7ff8940cf1c0 2024-09-13T17:07:47.825 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8940c66d0 con 0x7ff8940cf1c0 2024-09-13T17:07:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7ff884005c90 con 0x7ff8940cf1c0 2024-09-13T17:07:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.822+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7ff8840064b0 con 0x7ff8940cf1c0 2024-09-13T17:07:47.826 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.823+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8940d0640 con 0x7ff8940cf1c0 2024-09-13T17:07:47.827 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.825+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7ff884021060 con 0x7ff8940cf1c0 2024-09-13T17:07:47.899 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.897+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58", "id": 3} v 0) v1 -- 0x7ff8940c5f30 con 0x7ff8940cf1c0 2024-09-13T17:07:47.903 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.901+0000 7ff88aa00640 1 -- 172.21.15.189:0/3543172755 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff884029020 con 0x7ff8940cf1c0 2024-09-13T17:07:47.903 INFO:teuthology.orchestra.run.smithi189.stdout:3 2024-09-13T17:07:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 msgr2=0x7ff89414c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff88400e9b0 tx=0x7ff88400ee70 comp rx=0 tx=0).stop 2024-09-13T17:07:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 shutdown_connections 2024-09-13T17:07:47.904 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89415cd00 0x7ff8941f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89415c920 0x7ff8940c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 --2- 172.21.15.189:0/3543172755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8940cf1c0 0x7ff89414c940 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:47.905 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 >> 172.21.15.189:0/3543172755 conn(0x7ff894069e80 msgr2=0x7ff89415b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:47.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 shutdown_connections 2024-09-13T17:07:47.906 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:47.902+0000 7ff899c00640 1 -- 172.21.15.189:0/3543172755 wait complete. 2024-09-13T17:07:47.918 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new f390dbdf-1494-4416-9b4e-50c14a064a5d 4 2024-09-13T17:07:48.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.067+0000 7f014fe00640 1 Processor -- start 2024-09-13T17:07:48.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.067+0000 7f014fe00640 1 -- start start 2024-09-13T17:07:48.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f014815c920 0x7f01480d0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f014815cd00 0x7f01480cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480b5b10 con 0x7f014815cd00 2024-09-13T17:07:48.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01480d0b80 con 0x7f014815c920 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01480d16a0 con 0x7f01480cf1c0 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f014815c920 0x7f01480d0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f014815c920 0x7f01480d0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39264/0 (socket says 172.21.15.189:39264) 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.189:33340/0 (socket says 172.21.15.189:33340) 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014d400640 1 -- 172.21.15.189:0/2605241372 learned_addr learned my addr 172.21.15.189:0/2605241372 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.068+0000 7f014ca00640 1 --2- 172.21.15.189:0/2605241372 >> v2:172.21.15.28:3300/0 conn(0x7f014815cd00 0x7f01480cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014d400640 1 -- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 msgr2=0x7f01480cdad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014d400640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014d400640 1 -- 172.21.15.189:0/2605241372 >> v2:172.21.15.28:3300/0 conn(0x7f014815cd00 msgr2=0x7f01480cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014d400640 1 --2- 172.21.15.189:0/2605241372 >> v2:172.21.15.28:3300/0 conn(0x7f014815cd00 0x7f01480cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014d400640 1 -- 172.21.15.189:0/2605241372 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01480d0e00 con 0x7f014815c920 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.069+0000 7f014de00640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.070+0000 7f014d400640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f01480d0640 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f014806c010 tx=0x7f013c02a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=246fb6f932ecf7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.070+0000 7f0147e00640 1 -- 172.21.15.189:0/2605241372 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c032440 con 0x7f014815c920 2024-09-13T17:07:48.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.070+0000 7f0147e00640 1 -- 172.21.15.189:0/2605241372 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c0325a0 con 0x7f014815c920 2024-09-13T17:07:48.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.070+0000 7f0147e00640 1 -- 172.21.15.189:0/2605241372 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c032930 con 0x7f014815c920 2024-09-13T17:07:48.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 -- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 msgr2=0x7f01480d0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f01480d0640 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f014806c010 tx=0x7f013c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.073 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 -- 172.21.15.189:0/2605241372 shutdown_connections 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 --2- 172.21.15.189:0/2605241372 >> v2:172.21.15.28:3300/0 conn(0x7f014815cd00 0x7f01480cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480cdad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 --2- 172.21.15.189:0/2605241372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f01480d0640 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 -- 172.21.15.189:0/2605241372 >> 172.21.15.189:0/2605241372 conn(0x7f0148069e80 msgr2=0x7f014806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 -- 172.21.15.189:0/2605241372 shutdown_connections 2024-09-13T17:07:48.074 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.071+0000 7f014fe00640 1 -- 172.21.15.189:0/2605241372 wait complete. 2024-09-13T17:07:48.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.073+0000 7f014fe00640 1 Processor -- start 2024-09-13T17:07:48.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.073+0000 7f014fe00640 1 -- start start 2024-09-13T17:07:48.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.077 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014815cd00 0x7f01481f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014815ede0 con 0x7f014815cd00 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01480caf70 con 0x7f014815c920 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014815db90 con 0x7f01480cf1c0 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39266/0 (socket says 172.21.15.189:39266) 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 -- 172.21.15.189:0/3714797091 learned_addr learned my addr 172.21.15.189:0/3714797091 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014de00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014815cd00 0x7f01481f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014d400640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 -- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 msgr2=0x7f01480c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 -- 172.21.15.189:0/3714797091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014815cd00 msgr2=0x7f01481f5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014815cd00 0x7f01481f5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014ca00640 1 -- 172.21.15.189:0/3714797091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140009a70 con 0x7f014815c920 2024-09-13T17:07:48.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.074+0000 7f014d400640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f014ca00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f013400e870 tx=0x7f013400ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01340094f0 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013400ee60 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c0094d0 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013400cb80 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014815e150 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.075+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f0134005990 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.076+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f01340066b0 con 0x7f014815c920 2024-09-13T17:07:48.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.076+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01480cb0e0 con 0x7f014815c920 2024-09-13T17:07:48.080 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.078+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f01480cb0e0 con 0x7f014815c920 2024-09-13T17:07:48.169 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.167+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f390dbdf-1494-4416-9b4e-50c14a064a5d", "id": 4} v 0) v1 -- 0x7f01480cad40 con 0x7f014815c920 2024-09-13T17:07:48.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.171+0000 7f0145600640 1 -- 172.21.15.189:0/3714797091 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f390dbdf-1494-4416-9b4e-50c14a064a5d", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0134029020 con 0x7f014815c920 2024-09-13T17:07:48.173 INFO:teuthology.orchestra.run.smithi189.stdout:4 2024-09-13T17:07:48.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 msgr2=0x7f014815e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f013400e870 tx=0x7f013400ec70 comp rx=0 tx=0).stop 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 shutdown_connections 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f014815cd00 0x7f01481f5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f014815c920 0x7f014815e8a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 --2- 172.21.15.189:0/3714797091 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01480cf1c0 0x7f01480c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 >> 172.21.15.189:0/3714797091 conn(0x7f0148069e80 msgr2=0x7f01480ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.172+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 shutdown_connections 2024-09-13T17:07:48.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.173+0000 7f014fe00640 1 -- 172.21.15.189:0/3714797091 wait complete. 2024-09-13T17:07:48.189 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new fc031241-63f7-4113-a561-8bc3012719bc 5 2024-09-13T17:07:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.336+0000 7fe413e00640 1 Processor -- start 2024-09-13T17:07:48.338 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.336+0000 7fe413e00640 1 -- start start 2024-09-13T17:07:48.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.337+0000 7fe413e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe4140cd7c0 0x7fe4140c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.339 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.337+0000 7fe413e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4140c3ba0 0x7fe414070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.337+0000 7fe413e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe41415ef70 con 0x7fe414143f70 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.337+0000 7fe413e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4140c4320 con 0x7fe4140cd7c0 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.337+0000 7fe413e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41415e1b0 con 0x7fe4140c3ba0 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4140c3ba0 0x7fe414070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4140c3ba0 0x7fe414070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39274/0 (socket says 172.21.15.189:39274) 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 -- 172.21.15.189:0/3973149265 learned_addr learned my addr 172.21.15.189:0/3973149265 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe412000640 1 --2- 172.21.15.189:0/3973149265 >> v2:172.21.15.28:3300/0 conn(0x7fe4140cd7c0 0x7fe4140c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.340 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 -- 172.21.15.189:0/3973149265 >> v2:172.21.15.28:3300/0 conn(0x7fe4140cd7c0 msgr2=0x7fe4140c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:48.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 --2- 172.21.15.189:0/3973149265 >> v2:172.21.15.28:3300/0 conn(0x7fe4140cd7c0 0x7fe4140c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 -- 172.21.15.189:0/3973149265 >> v1:172.21.15.28:6789/0 conn(0x7fe414143f70 legacy=0x7fe4140cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.338+0000 7fe413400640 1 -- 172.21.15.189:0/3973149265 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41415e490 con 0x7fe4140c3ba0 2024-09-13T17:07:48.341 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.339+0000 7fe413400640 1 --2- 172.21.15.189:0/3973149265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe414070440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4140ce2b0 tx=0x7fe40c02a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfcce6c68b560094 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.339+0000 7fe411600640 1 -- 172.21.15.189:0/3973149265 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c031600 con 0x7fe4140c3ba0 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.339+0000 7fe411600640 1 -- 172.21.15.189:0/3973149265 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c02d050 con 0x7fe4140c3ba0 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.339+0000 7fe411600640 1 -- 172.21.15.189:0/3973149265 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c031980 con 0x7fe4140c3ba0 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 -- 172.21.15.189:0/3973149265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 msgr2=0x7fe414070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 --2- 172.21.15.189:0/3973149265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe414070440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe4140ce2b0 tx=0x7fe40c02a730 comp rx=0 tx=0).stop 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 -- 172.21.15.189:0/3973149265 shutdown_connections 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 --2- 172.21.15.189:0/3973149265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe414070440 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.342 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 --2- 172.21.15.189:0/3973149265 >> v2:172.21.15.28:3300/0 conn(0x7fe4140cd7c0 0x7fe4140c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 -- 172.21.15.189:0/3973149265 >> 172.21.15.189:0/3973149265 conn(0x7fe414069e80 msgr2=0x7fe41406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 -- 172.21.15.189:0/3973149265 shutdown_connections 2024-09-13T17:07:48.343 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.340+0000 7fe413e00640 1 -- 172.21.15.189:0/3973149265 wait complete. 2024-09-13T17:07:48.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.342+0000 7fe413e00640 1 Processor -- start 2024-09-13T17:07:48.344 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.342+0000 7fe413e00640 1 -- start start 2024-09-13T17:07:48.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe41413c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 0x7fe4140c6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41413c2f0 con 0x7fe4140cd7c0 2024-09-13T17:07:48.346 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41413cfa0 con 0x7fe4140c3ba0 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe41413d280 con 0x7fe414143f70 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe412a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe41413c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.189:33358/0 (socket says 172.21.15.189:33358) 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe412a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe41413c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39288/0 (socket says 172.21.15.189:39288) 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe413400640 1 -- 172.21.15.189:0/2571882069 learned_addr learned my addr 172.21.15.189:0/2571882069 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.343+0000 7fe412000640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 0x7fe4140c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 -- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 msgr2=0x7fe41413c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe41413c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 -- 172.21.15.189:0/2571882069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 msgr2=0x7fe4140c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 0x7fe4140c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 -- 172.21.15.189:0/2571882069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c009b30 con 0x7fe414143f70 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe412000640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 0x7fe4140c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:48.347 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413400640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe40c034040 tx=0x7fe40c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c03d860 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe40c03d9c0 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41413d560 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.344+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c03dd70 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.345+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41413db20 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.345+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fe40c046700 con 0x7fe414143f70 2024-09-13T17:07:48.348 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.345+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fe40c03b980 con 0x7fe414143f70 2024-09-13T17:07:48.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.346+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe414070440 con 0x7fe414143f70 2024-09-13T17:07:48.349 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.348+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fe40c064e30 con 0x7fe414143f70 2024-09-13T17:07:48.418 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.417+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fc031241-63f7-4113-a561-8bc3012719bc", "id": 5} v 0) v1 -- 0x7fe4140c81c0 con 0x7fe414143f70 2024-09-13T17:07:48.422 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.421+0000 7fe40aa00640 1 -- 172.21.15.189:0/2571882069 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc031241-63f7-4113-a561-8bc3012719bc", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe40c053030 con 0x7fe414143f70 2024-09-13T17:07:48.423 INFO:teuthology.orchestra.run.smithi189.stdout:5 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 msgr2=0x7fe4141401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe40c034040 tx=0x7fe40c004a40 comp rx=0 tx=0).stop 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 shutdown_connections 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143f70 0x7fe4141401b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4140cd7c0 0x7fe4140c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 --2- 172.21.15.189:0/2571882069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140c3ba0 0x7fe41413c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.424 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 >> 172.21.15.189:0/2571882069 conn(0x7fe414069e80 msgr2=0x7fe41406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 shutdown_connections 2024-09-13T17:07:48.425 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.422+0000 7fe413e00640 1 -- 172.21.15.189:0/2571882069 wait complete. 2024-09-13T17:07:48.438 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new 8cd60534-2b08-47c1-8cd7-69c2e3ac1599 6 2024-09-13T17:07:48.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.584+0000 7f169cc00640 1 Processor -- start 2024-09-13T17:07:48.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.585+0000 7f169cc00640 1 -- start start 2024-09-13T17:07:48.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f169cc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f169815ca40 0x7f16980cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f169cc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980d19b0 con 0x7f16980cf7b0 2024-09-13T17:07:48.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f169cc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980d1c90 con 0x7f16980d1400 2024-09-13T17:07:48.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f169cc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980cfc40 con 0x7f169815ca40 2024-09-13T17:07:48.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f1697e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f16980d1400 0x7f16980b5ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.189:53746/0 (socket says 172.21.15.189:53746) 2024-09-13T17:07:48.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f1697e00640 1 -- 172.21.15.189:0/3363201352 learned_addr learned my addr 172.21.15.189:0/3363201352 (peer_addr_for_me v1:172.21.15.189:0/0) 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.586+0000 7f1697400640 1 --2- 172.21.15.189:0/3363201352 >> v2:172.21.15.189:3300/0 conn(0x7f169815ca40 0x7f16980cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442592605 0 0) 0x7f16980d1c90 con 0x7f16980d1400 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f167c003520 con 0x7f16980d1400 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3564508632 0 0) 0x7f16980d19b0 con 0x7f16980cf7b0 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16980d1c90 con 0x7f16980cf7b0 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1697400640 1 -- 172.21.15.189:0/3363201352 >> v1:172.21.15.189:6789/0 conn(0x7f16980d1400 legacy=0x7f16980b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1697400640 1 -- 172.21.15.189:0/3363201352 >> v1:172.21.15.28:6789/0 conn(0x7f16980cf7b0 legacy=0x7f16980d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.589 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.587+0000 7f1697400640 1 -- 172.21.15.189:0/3363201352 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980d1f70 con 0x7f169815ca40 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.588+0000 7f1697400640 1 --2- 172.21.15.189:0/3363201352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169815ca40 0x7f16980cf270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16880097d0 tx=0x7f168802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6655acd96c736c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.588+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688032440 con 0x7f169815ca40 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.588+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16880325a0 con 0x7f169815ca40 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.588+0000 7f1696000640 1 -- 172.21.15.189:0/3363201352 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688032910 con 0x7f169815ca40 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 -- 172.21.15.189:0/3363201352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169815ca40 msgr2=0x7f16980cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 --2- 172.21.15.189:0/3363201352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169815ca40 0x7f16980cf270 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f16880097d0 tx=0x7f168802a250 comp rx=0 tx=0).stop 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 -- 172.21.15.189:0/3363201352 shutdown_connections 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 --2- 172.21.15.189:0/3363201352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f169815ca40 0x7f16980cf270 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 -- 172.21.15.189:0/3363201352 >> 172.21.15.189:0/3363201352 conn(0x7f1698070ad0 msgr2=0x7f16980b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 -- 172.21.15.189:0/3363201352 shutdown_connections 2024-09-13T17:07:48.591 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.589+0000 7f169cc00640 1 -- 172.21.15.189:0/3363201352 wait complete. 2024-09-13T17:07:48.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.590+0000 7f169cc00640 1 Processor -- start 2024-09-13T17:07:48.592 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.590+0000 7f169cc00640 1 -- start start 2024-09-13T17:07:48.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 0x7f1698148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.593 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698153bb0 con 0x7f16980cf7b0 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1698153e90 con 0x7f16980d1400 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.591+0000 7f169cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1698154170 con 0x7f1698153650 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1696a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.189:33360/0 (socket says 172.21.15.189:33360) 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 0x7f1698148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 -- 172.21.15.189:0/3084950128 learned_addr learned my addr 172.21.15.189:0/3084950128 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.594 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1696a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39308/0 (socket says 172.21.15.189:39308) 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 -- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 msgr2=0x7f1698153110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 -- 172.21.15.189:0/3084950128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 msgr2=0x7f1698148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 0x7f1698148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 -- 172.21.15.189:0/3084950128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1688009400 con 0x7f1698153650 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697400640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 0x7f1698148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1696a00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.592+0000 7f1697e00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f169000cb40 tx=0x7f169000a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169000aca0 con 0x7f1698153650 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169000ae20 con 0x7f1698153650 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690012640 con 0x7f1698153650 2024-09-13T17:07:48.596 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1698154450 con 0x7f1698153650 2024-09-13T17:07:48.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1698154c50 con 0x7f1698153650 2024-09-13T17:07:48.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f1690012aa0 con 0x7f1698153650 2024-09-13T17:07:48.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.593+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f1690012d50 con 0x7f1698153650 2024-09-13T17:07:48.597 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.594+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1698154970 con 0x7f1698153650 2024-09-13T17:07:48.598 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.596+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f1698154970 con 0x7f1698153650 2024-09-13T17:07:48.622 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:48.621+0000 7fbede80c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T17:07:48.632 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:48.631+0000 7f929a62a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T17:07:48.668 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.666+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8cd60534-2b08-47c1-8cd7-69c2e3ac1599", "id": 6} v 0) v1 -- 0x7f16980cf270 con 0x7f1698153650 2024-09-13T17:07:48.672 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.670+0000 7f168f400640 1 -- 172.21.15.189:0/3084950128 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cd60534-2b08-47c1-8cd7-69c2e3ac1599", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1690017070 con 0x7f1698153650 2024-09-13T17:07:48.672 INFO:teuthology.orchestra.run.smithi189.stdout:6 2024-09-13T17:07:48.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.671+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 msgr2=0x7f1698158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.671+0000 7f169cc00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f169000cb40 tx=0x7f169000a060 comp rx=0 tx=0).stop 2024-09-13T17:07:48.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 shutdown_connections 2024-09-13T17:07:48.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1698153650 0x7f1698158240 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.674 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16980d1400 0x7f1698153110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 --2- 172.21.15.189:0/3084950128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16980cf7b0 0x7f1698148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 >> 172.21.15.189:0/3084950128 conn(0x7f1698070ad0 msgr2=0x7f169815f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 shutdown_connections 2024-09-13T17:07:48.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.672+0000 7f169cc00640 1 -- 172.21.15.189:0/3084950128 wait complete. 2024-09-13T17:07:48.685 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:48.683+0000 7fbede80c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T17:07:48.687 DEBUG:teuthology.orchestra.run.smithi189:> sudo ceph --cluster ceph osd new e1be57a8-fb3f-4809-88b1-009581c546b7 7 2024-09-13T17:07:48.695 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:48.694+0000 7f929a62a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T17:07:48.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.833+0000 7f3bade00640 1 Processor -- start 2024-09-13T17:07:48.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.833+0000 7f3bade00640 1 -- start start 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bade00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ba80cf980 0x7f3ba80d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ba80d1230 0x7f3ba80b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bade00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba80d19b0 con 0x7f3ba815ca40 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bade00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba80d1c90 con 0x7f3ba80d1230 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bade00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba80d1f70 con 0x7f3ba80cf980 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3ba7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ba80cf980 0x7f3ba80d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3ba7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3ba80cf980 0x7f3ba80d0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58430/0 (socket says 172.21.15.189:58430) 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3ba7e00640 1 -- 172.21.15.189:0/3768595465 learned_addr learned my addr 172.21.15.189:0/3768595465 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.834+0000 7f3bad400640 1 --2- 172.21.15.189:0/3768595465 >> v2:172.21.15.28:3300/0 conn(0x7f3ba80d1230 0x7f3ba80b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.835+0000 7f3ba7e00640 1 -- 172.21.15.189:0/3768595465 >> v2:172.21.15.28:3300/0 conn(0x7f3ba80d1230 msgr2=0x7f3ba80b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.835+0000 7f3ba7e00640 1 --2- 172.21.15.189:0/3768595465 >> v2:172.21.15.28:3300/0 conn(0x7f3ba80d1230 0x7f3ba80b5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.835+0000 7f3ba7e00640 1 -- 172.21.15.189:0/3768595465 >> v1:172.21.15.28:6789/0 conn(0x7f3ba815ca40 legacy=0x7f3ba80cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.835+0000 7f3ba7e00640 1 -- 172.21.15.189:0/3768595465 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba80d2250 con 0x7f3ba80cf980 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.836+0000 7f3ba7e00640 1 --2- 172.21.15.189:0/3768595465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80d0cf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009850 tx=0x7f3b9402a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58700497e2bc7d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.836+0000 7f3ba7400640 1 -- 172.21.15.189:0/3768595465 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94004030 con 0x7f3ba80cf980 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.836+0000 7f3ba7400640 1 -- 172.21.15.189:0/3768595465 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b94004190 con 0x7f3ba80cf980 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.836+0000 7f3ba7400640 1 -- 172.21.15.189:0/3768595465 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94003610 con 0x7f3ba80cf980 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 -- 172.21.15.189:0/3768595465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 msgr2=0x7f3ba80d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 --2- 172.21.15.189:0/3768595465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80d0cf0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3b94009850 tx=0x7f3b9402a290 comp rx=0 tx=0).stop 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 -- 172.21.15.189:0/3768595465 shutdown_connections 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 --2- 172.21.15.189:0/3768595465 >> v2:172.21.15.28:3300/0 conn(0x7f3ba80d1230 0x7f3ba80b5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 --2- 172.21.15.189:0/3768595465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80d0cf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 -- 172.21.15.189:0/3768595465 >> 172.21.15.189:0/3768595465 conn(0x7f3ba8070ad0 msgr2=0x7f3ba80b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 -- 172.21.15.189:0/3768595465 shutdown_connections 2024-09-13T17:07:48.839 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.837+0000 7f3bade00640 1 -- 172.21.15.189:0/3768595465 wait complete. 2024-09-13T17:07:48.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.838+0000 7f3bade00640 1 Processor -- start 2024-09-13T17:07:48.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 -- start start 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ba80d1230 0x7f3ba815b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81f24d0 0x7f3ba80cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba80a9a60 con 0x7f3ba81f24d0 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba81f3530 con 0x7f3ba80cf980 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.839+0000 7f3bade00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba815bd40 con 0x7f3ba80d1230 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.842 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58434/0 (socket says 172.21.15.189:58434) 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 -- 172.21.15.189:0/2765536881 learned_addr learned my addr 172.21.15.189:0/2765536881 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3ba7e00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ba80d1230 0x7f3ba815b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3bad400640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81f24d0 0x7f3ba80cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 -- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ba80d1230 msgr2=0x7f3ba815b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ba80d1230 0x7f3ba815b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 -- 172.21.15.189:0/2765536881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81f24d0 msgr2=0x7f3ba80cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81f24d0 0x7f3ba80cb060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 -- 172.21.15.189:0/2765536881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b94009460 con 0x7f3ba80cf980 2024-09-13T17:07:48.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.840+0000 7f3baca00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c430 tx=0x7f3b9c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:48.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c0052d0 con 0x7f3ba80cf980 2024-09-13T17:07:48.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c005430 con 0x7f3ba80cf980 2024-09-13T17:07:48.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c003830 con 0x7f3ba80cf980 2024-09-13T17:07:48.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba81f2c90 con 0x7f3ba80cf980 2024-09-13T17:07:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba81f3250 con 0x7f3ba80cf980 2024-09-13T17:07:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f3b9c003c90 con 0x7f3ba80cf980 2024-09-13T17:07:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.841+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f3b9c012420 con 0x7f3ba80cf980 2024-09-13T17:07:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.842+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba80d0cf0 con 0x7f3ba80cf980 2024-09-13T17:07:48.845 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.844+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f3b9c017070 con 0x7f3ba80cf980 2024-09-13T17:07:48.915 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.913+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e1be57a8-fb3f-4809-88b1-009581c546b7", "id": 7} v 0) v1 -- 0x7f3ba80cb5a0 con 0x7f3ba80cf980 2024-09-13T17:07:48.919 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.917+0000 7f3ba4c00640 1 -- 172.21.15.189:0/2765536881 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1be57a8-fb3f-4809-88b1-009581c546b7", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3b9c011e50 con 0x7f3ba80cf980 2024-09-13T17:07:48.920 INFO:teuthology.orchestra.run.smithi189.stdout:7 2024-09-13T17:07:48.921 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 msgr2=0x7f3ba80a9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c00c430 tx=0x7f3b9c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 shutdown_connections 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ba81f24d0 0x7f3ba80cb060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ba80d1230 0x7f3ba815b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 --2- 172.21.15.189:0/2765536881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ba80cf980 0x7f3ba80a9520 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 >> 172.21.15.189:0/2765536881 conn(0x7f3ba8070ad0 msgr2=0x7f3ba80b7790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 shutdown_connections 2024-09-13T17:07:48.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-09-13T17:07:48.918+0000 7f3bade00640 1 -- 172.21.15.189:0/2765536881 wait complete. 2024-09-13T17:07:48.934 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T17:07:48.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T17:07:48.938 INFO:tasks.ceph.osd.0:Started 2024-09-13T17:07:48.938 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T17:07:48.939 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T17:07:48.942 INFO:tasks.ceph.osd.1:Started 2024-09-13T17:07:48.942 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T17:07:48.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T17:07:48.947 INFO:tasks.ceph.osd.2:Started 2024-09-13T17:07:48.947 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T17:07:48.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T17:07:48.951 INFO:tasks.ceph.osd.3:Started 2024-09-13T17:07:48.951 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T17:07:48.951 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T17:07:48.956 INFO:tasks.ceph.osd.4:Started 2024-09-13T17:07:48.956 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T17:07:48.956 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T17:07:48.961 INFO:tasks.ceph.osd.5:Started 2024-09-13T17:07:48.961 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T17:07:48.961 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T17:07:48.966 INFO:tasks.ceph.osd.6:Started 2024-09-13T17:07:48.966 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T17:07:48.966 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T17:07:48.971 INFO:tasks.ceph.osd.7:Started 2024-09-13T17:07:48.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:07:49.178 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.176+0000 7fbede80c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T17:07:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.227+0000 7fbf02200640 1 Processor -- start 2024-09-13T17:07:49.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.227+0000 7fbf02200640 1 -- start start 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.228+0000 7fbf02200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc146170 0x7fbefc15d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.228+0000 7fbf02200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbefc160310 0x7fbefc160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.228+0000 7fbf02200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbefc146550 con 0x7fbefc0ac850 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.228+0000 7fbf02200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbefc15cd20 con 0x7fbefc160310 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.228+0000 7fbf02200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc15d280 con 0x7fbefc146170 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbefc160310 0x7fbefc160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc146170 0x7fbefc15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:49.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbefc160310 0x7fbefc160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43538/0 (socket says 172.21.15.28:43538) 2024-09-13T17:07:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 -- 172.21.15.28:0/693520781 learned_addr learned my addr 172.21.15.28:0/693520781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 -- 172.21.15.28:0/693520781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc146170 msgr2=0x7fbefc15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 --2- 172.21.15.28:0/693520781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc146170 0x7fbefc15d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 -- 172.21.15.28:0/693520781 >> v1:172.21.15.189:6789/0 conn(0x7fbefc0ac850 legacy=0x7fbefc06b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:49.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.229+0000 7fbefaa00640 1 -- 172.21.15.28:0/693520781 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbefc15dc50 con 0x7fbefc160310 2024-09-13T17:07:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.230+0000 7fbefaa00640 1 --2- 172.21.15.28:0/693520781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc160310 0x7fbefc160750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009870 tx=0x7fbee802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b221877aa62934d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.230+0000 7fbefa000640 1 -- 172.21.15.28:0/693520781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8032440 con 0x7fbefc160310 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.230+0000 7fbefa000640 1 -- 172.21.15.28:0/693520781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbee80325a0 con 0x7fbefc160310 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.230+0000 7fbefa000640 1 -- 172.21.15.28:0/693520781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbee8032930 con 0x7fbefc160310 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 -- 172.21.15.28:0/693520781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc160310 msgr2=0x7fbefc160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 --2- 172.21.15.28:0/693520781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc160310 0x7fbefc160750 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbee8009870 tx=0x7fbee802a270 comp rx=0 tx=0).stop 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 -- 172.21.15.28:0/693520781 shutdown_connections 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 --2- 172.21.15.28:0/693520781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc160310 0x7fbefc160750 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 --2- 172.21.15.28:0/693520781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc146170 0x7fbefc15d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 -- 172.21.15.28:0/693520781 >> 172.21.15.28:0/693520781 conn(0x7fbefc069e80 msgr2=0x7fbefc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 -- 172.21.15.28:0/693520781 shutdown_connections 2024-09-13T17:07:49.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.231+0000 7fbf02200640 1 -- 172.21.15.28:0/693520781 wait complete. 2024-09-13T17:07:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.233+0000 7fbf02200640 1 Processor -- start 2024-09-13T17:07:49.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.233+0000 7fbf02200640 1 -- start start 2024-09-13T17:07:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbefc146170 0x7fbefc154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 0x7fbefc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc156c70 con 0x7fbefc0ac850 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc155150 con 0x7fbefc146170 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbf02200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbefc155430 con 0x7fbefc160310 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43546/0 (socket says 172.21.15.28:43546) 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 -- 172.21.15.28:0/2906805722 learned_addr learned my addr 172.21.15.28:0/2906805722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefbe00640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 0x7fbefc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefaa00640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbefc146170 0x7fbefc154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 -- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 msgr2=0x7fbefc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 0x7fbefc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.234+0000 7fbefb400640 1 -- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbefc146170 msgr2=0x7fbefc154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:49.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbefb400640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbefc146170 0x7fbefc154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbefb400640 1 -- 172.21.15.28:0/2906805722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee80094d0 con 0x7fbefc0ac850 2024-09-13T17:07:49.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbefbe00640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 0x7fbefc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbefb400640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00e9b0 tx=0x7fbeec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec009670 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec00c9a0 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec00cd50 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbefc155710 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.235+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbefc143ab0 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.236+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fbeec005c90 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.236+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbeec013470 con 0x7fbefc0ac850 2024-09-13T17:07:49.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.237+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbefc15d710 con 0x7fbefc0ac850 2024-09-13T17:07:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.240+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fbeec021060 con 0x7fbefc0ac850 2024-09-13T17:07:49.253 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.252+0000 7f929a62a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-13T17:07:49.276 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.274+0000 7fbede80c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T17:07:49.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.341+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbefc071350 con 0x7fbefc0ac850 2024-09-13T17:07:49.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.342+0000 7fbef3400640 1 -- 172.21.15.28:0/2906805722 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbeec029020 con 0x7fbefc0ac850 2024-09-13T17:07:49.343 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:07:49.343 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:07:48.915782+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 msgr2=0x7fbefc143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00e9b0 tx=0x7fbeec00ee70 comp rx=0 tx=0).stop 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 shutdown_connections 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbefc160310 0x7fbefc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbefc146170 0x7fbefc154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 --2- 172.21.15.28:0/2906805722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbefc0ac850 0x7fbefc143570 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:49.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 >> 172.21.15.28:0/2906805722 conn(0x7fbefc069e80 msgr2=0x7fbefc0af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:49.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 shutdown_connections 2024-09-13T17:07:49.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:07:49.343+0000 7fbf02200640 1 -- 172.21.15.28:0/2906805722 wait complete. 2024-09-13T17:07:49.359 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T17:07:49.359 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T17:07:49.381 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.380+0000 7f929a62a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T17:07:49.397 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.395+0000 7fbede80c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T17:07:49.453 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.451+0000 7fbede80c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T17:07:49.506 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.506+0000 7f929a62a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T17:07:49.563 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.562+0000 7f929a62a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T17:07:49.577 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.576+0000 7fbede80c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T17:07:49.616 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-13T17:07:49.615+0000 7f0dd2e0f640 -1 Falling back to public interface 2024-09-13T17:07:49.638 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-13T17:07:49.636+0000 7fd5b9a2e640 -1 Falling back to public interface 2024-09-13T17:07:49.640 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T17:07:49.640+0000 7fae75885640 -1 Falling back to public interface 2024-09-13T17:07:49.642 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.640+0000 7fbede80c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T17:07:49.648 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T17:07:49.648+0000 7f3ccca20640 -1 Falling back to public interface 2024-09-13T17:07:49.652 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T17:07:49.651+0000 7f863bc0d640 -1 Falling back to public interface 2024-09-13T17:07:49.652 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T17:07:49.651+0000 7f5368213640 -1 Falling back to public interface 2024-09-13T17:07:49.653 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T17:07:49.652+0000 7f3ccae00640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6814/0: (98) Address already in use 2024-09-13T17:07:49.659 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-09-13T17:07:49.658+0000 7f6270617640 -1 Falling back to public interface 2024-09-13T17:07:49.665 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-09-13T17:07:49.663+0000 7f7f607bb640 -1 Falling back to public interface 2024-09-13T17:07:49.697 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.695+0000 7fbede80c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T17:07:49.699 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.698+0000 7f929a62a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-13T17:07:49.764 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.763+0000 7f929a62a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T17:07:49.819 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.818+0000 7f929a62a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T17:07:49.821 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.820+0000 7fbede80c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T17:07:49.880 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.877+0000 7fbede80c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T17:07:49.941 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:49.939+0000 7fbede80c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T17:07:49.949 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:49.948+0000 7f929a62a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T17:07:50.010 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:50.009+0000 7f929a62a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T17:07:50.020 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:50.019+0000 7fbede80c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T17:07:50.072 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:50.071+0000 7f929a62a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T17:07:50.072 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:50.070+0000 7fbede80c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T17:07:50.124 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:50.122+0000 7fbede80c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T17:07:50.153 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:50.152+0000 7f929a62a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T17:07:50.163 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T17:07:50.208 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:50.207+0000 7f929a62a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T17:07:50.262 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:50.261+0000 7f929a62a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T17:07:50.301 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.300+0000 7fb684a00640 1 Processor -- start 2024-09-13T17:07:50.301 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.300+0000 7fb684a00640 1 -- start start 2024-09-13T17:07:50.303 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb684a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 0x7fb68006b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:50.303 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb684a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6800cd6c0 0x7fb6801642f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:50.303 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb684a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68006bba0 con 0x7fb68006f4b0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb684a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb68015ac20 con 0x7fb6800cd6c0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb684a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68006be80 con 0x7fb6800b0ee0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb68006f4b0 0x7fb68006f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50042/0 (socket says 172.21.15.28:50042) 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67de00640 1 -- 172.21.15.28:0/2382536421 learned_addr learned my addr 172.21.15.28:0/2382536421 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67e800640 1 --2- 172.21.15.28:0/2382536421 >> v2:172.21.15.189:3300/0 conn(0x7fb6800cd6c0 0x7fb6801642f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb677e00640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 0x7fb68006b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67d400640 1 -- 172.21.15.28:0/2382536421 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670287805 0 0) 0x7fb68006bba0 con 0x7fb68006f4b0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67d400640 1 -- 172.21.15.28:0/2382536421 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664003520 con 0x7fb68006f4b0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67e800640 1 -- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 msgr2=0x7fb68006b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.302+0000 7fb67e800640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 0x7fb68006b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.303+0000 7fb67e800640 1 -- 172.21.15.28:0/2382536421 >> v1:172.21.15.28:6789/0 conn(0x7fb68006f4b0 legacy=0x7fb68006f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.303+0000 7fb67e800640 1 -- 172.21.15.28:0/2382536421 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68006c160 con 0x7fb6800cd6c0 2024-09-13T17:07:50.304 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.303+0000 7fb677e00640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 0x7fb68006b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:50.305 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb67e800640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800cd6c0 0x7fb6801642f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb678009d90 tx=0x7fb67802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d48bc10c74329d9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb67d400640 1 -- 172.21.15.28:0/2382536421 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678031600 con 0x7fb6800cd6c0 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb67d400640 1 -- 172.21.15.28:0/2382536421 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67802d050 con 0x7fb6800cd6c0 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb67d400640 1 -- 172.21.15.28:0/2382536421 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678031980 con 0x7fb6800cd6c0 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb684a00640 1 -- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800cd6c0 msgr2=0x7fb6801642f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.304+0000 7fb684a00640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800cd6c0 0x7fb6801642f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb678009d90 tx=0x7fb67802a5d0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 -- 172.21.15.28:0/2382536421 shutdown_connections 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800cd6c0 0x7fb6801642f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 --2- 172.21.15.28:0/2382536421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6800b0ee0 0x7fb68006b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 -- 172.21.15.28:0/2382536421 >> 172.21.15.28:0/2382536421 conn(0x7fb6800b0310 msgr2=0x7fb6800b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 -- 172.21.15.28:0/2382536421 shutdown_connections 2024-09-13T17:07:50.306 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.305+0000 7fb684a00640 1 -- 172.21.15.28:0/2382536421 wait complete. 2024-09-13T17:07:50.307 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.306+0000 7fb684a00640 1 Processor -- start 2024-09-13T17:07:50.307 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.306+0000 7fb684a00640 1 -- start start 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb68006f4b0 0x7fb680150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 0x7fb6800c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68015af50 con 0x7fb6800cd6c0 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6800c6210 con 0x7fb6800b0ee0 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.307+0000 7fb684a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb68015b1a0 con 0x7fb68006f4b0 2024-09-13T17:07:50.309 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43560/0 (socket says 172.21.15.28:43560) 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 -- 172.21.15.28:0/3644336656 learned_addr learned my addr 172.21.15.28:0/3644336656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb677e00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 0x7fb6800c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67de00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb68006f4b0 0x7fb680150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 -- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb68006f4b0 msgr2=0x7fb680150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb68006f4b0 0x7fb680150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.310 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 -- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 msgr2=0x7fb6800c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:50.311 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 0x7fb6800c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.312 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 -- 172.21.15.28:0/3644336656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668009400 con 0x7fb6800cd6c0 2024-09-13T17:07:50.312 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb677e00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 0x7fb6800c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:07:50.312 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.308+0000 7fb67e800640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb67802abc0 tx=0x7fb678004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67803c830 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb67803c990 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb678009a70 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67803cd20 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68015b760 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fb678003c90 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.309+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb678004700 con 0x7fb6800cd6c0 2024-09-13T17:07:50.313 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.310+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68006f890 con 0x7fb6800cd6c0 2024-09-13T17:07:50.314 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.313+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fb678040070 con 0x7fb6800cd6c0 2024-09-13T17:07:50.375 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.374+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb680071410 con 0x7fb6800cd6c0 2024-09-13T17:07:50.376 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.375+0000 7fb676000640 1 -- 172.21.15.28:0/3644336656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb678003f90 con 0x7fb6800cd6c0 2024-09-13T17:07:50.376 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-13T17:07:50.376 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":10,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:07:48.915782+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 msgr2=0x7fb6800c53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fb67802abc0 tx=0x7fb678004a40 comp rx=0 tx=0).stop 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 shutdown_connections 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6800cd6c0 0x7fb6800c53b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6800b0ee0 0x7fb6800c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 --2- 172.21.15.28:0/3644336656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb68006f4b0 0x7fb680150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:50.377 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 >> 172.21.15.28:0/3644336656 conn(0x7fb6800b0310 msgr2=0x7fb6800acdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:50.378 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 shutdown_connections 2024-09-13T17:07:50.378 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:50.376+0000 7fb684a00640 1 -- 172.21.15.28:0/3644336656 wait complete. 2024-09-13T17:07:50.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-13T17:07:50.867 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:50.865+0000 7fbede80c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T17:07:51.027 INFO:tasks.ceph.mgr.y.smithi028.stderr:2024-09-13T17:07:51.026+0000 7f929a62a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T17:07:51.889 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:51.887+0000 7fbe7ea00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T17:07:51.889 INFO:tasks.ceph.mgr.x.smithi189.stderr:2024-09-13T17:07:51.888+0000 7fbe7ea00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T17:07:53.231 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-13T17:07:53.229+0000 7fd5b9a2e640 -1 osd.5 0 log_to_monitors true 2024-09-13T17:07:53.260 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T17:07:53.259+0000 7f3ccca20640 -1 osd.1 0 log_to_monitors true 2024-09-13T17:07:53.279 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T17:07:53.278+0000 7f863bc0d640 -1 osd.3 0 log_to_monitors true 2024-09-13T17:07:53.309 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-09-13T17:07:53.307+0000 7f6270617640 -1 osd.6 0 log_to_monitors true 2024-09-13T17:07:53.318 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-09-13T17:07:53.317+0000 7f7f607bb640 -1 osd.7 0 log_to_monitors true 2024-09-13T17:07:53.393 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-13T17:07:53.391+0000 7f0dd2e0f640 -1 osd.4 0 log_to_monitors true 2024-09-13T17:07:53.449 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T17:07:53.449+0000 7fae75885640 -1 osd.0 0 log_to_monitors true 2024-09-13T17:07:53.503 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T17:07:53.502+0000 7f5368213640 -1 osd.2 0 log_to_monitors true 2024-09-13T17:07:56.596 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T17:07:56.595+0000 7fae6fe00640 -1 osd.0 0 waiting for initial osdmap 2024-09-13T17:07:56.596 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T17:07:56.595+0000 7f8635c00640 -1 osd.3 0 waiting for initial osdmap 2024-09-13T17:07:56.603 INFO:tasks.ceph.osd.0.smithi028.stderr:2024-09-13T17:07:56.603+0000 7fae66800640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:56.604 INFO:tasks.ceph.osd.3.smithi028.stderr:2024-09-13T17:07:56.604+0000 7f862c600640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:56.624 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T17:07:56.624+0000 7f5362000640 -1 osd.2 0 waiting for initial osdmap 2024-09-13T17:07:56.635 INFO:tasks.ceph.osd.2.smithi028.stderr:2024-09-13T17:07:56.634+0000 7f5358a00640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:56.644 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T17:07:56.643+0000 7f3cc6800640 -1 osd.1 0 waiting for initial osdmap 2024-09-13T17:07:56.652 INFO:tasks.ceph.osd.1.smithi028.stderr:2024-09-13T17:07:56.651+0000 7f3cbd200640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:57.196 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T17:07:57.231 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-09-13T17:07:57.229+0000 7f626a600640 -1 osd.6 0 waiting for initial osdmap 2024-09-13T17:07:57.242 INFO:tasks.ceph.osd.6.smithi189.stderr:2024-09-13T17:07:57.240+0000 7f6261000640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:57.244 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-13T17:07:57.243+0000 7fd5b3400640 -1 osd.5 0 waiting for initial osdmap 2024-09-13T17:07:57.245 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-13T17:07:57.243+0000 7f0dccc00640 -1 osd.4 0 waiting for initial osdmap 2024-09-13T17:07:57.249 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-09-13T17:07:57.246+0000 7f7f5ac00640 -1 osd.7 0 waiting for initial osdmap 2024-09-13T17:07:57.251 INFO:tasks.ceph.osd.4.smithi189.stderr:2024-09-13T17:07:57.249+0000 7f0dc3600640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:57.254 INFO:tasks.ceph.osd.5.smithi189.stderr:2024-09-13T17:07:57.252+0000 7fd5a9e00640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:57.258 INFO:tasks.ceph.osd.7.smithi189.stderr:2024-09-13T17:07:57.256+0000 7f7f51600640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T17:07:57.339 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.338+0000 7f08f3600640 1 Processor -- start 2024-09-13T17:07:57.339 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.338+0000 7f08f3600640 1 -- start start 2024-09-13T17:07:57.340 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f3600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f08ec06f4b0 0x7f08ec06f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec0b0ee0 0x7f08ec06b230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f3600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08ec06fdd0 con 0x7f08ec0cc6b0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec15ac20 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f3600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08ec06bba0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec0b0ee0 0x7f08ec06b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f1600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08ec0cc6b0 0x7f08ec0ccae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60736/0 (socket says 172.21.15.28:60736) 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08ec0b0ee0 0x7f08ec06b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38140/0 (socket says 172.21.15.28:38140) 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f1600640 1 -- 172.21.15.28:0/2710873169 learned_addr learned my addr 172.21.15.28:0/2710873169 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.339+0000 7f08f0c00640 1 --2- 172.21.15.28:0/2710873169 >> v2:172.21.15.189:3300/0 conn(0x7f08ec06f4b0 0x7f08ec06f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.340+0000 7f08ebe00640 1 -- 172.21.15.28:0/2710873169 >> v2:172.21.15.189:3300/0 conn(0x7f08ec06f4b0 msgr2=0x7f08ec06f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.340+0000 7f08ebe00640 1 --2- 172.21.15.28:0/2710873169 >> v2:172.21.15.189:3300/0 conn(0x7f08ec06f4b0 0x7f08ec06f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.340+0000 7f08ebe00640 1 -- 172.21.15.28:0/2710873169 >> v1:172.21.15.28:6789/0 conn(0x7f08ec0cc6b0 legacy=0x7f08ec0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.340+0000 7f08ebe00640 1 -- 172.21.15.28:0/2710873169 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ec06be80 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.341 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.340+0000 7f08eb400640 1 -- 172.21.15.28:0/2710873169 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547811218 0 0) 0x7f08ec06fdd0 con 0x7f08ec0cc6b0 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08ebe00640 1 --2- 172.21.15.28:0/2710873169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec06b230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06f020 tx=0x7f08d802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331b20a7278d9cfe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08eb400640 1 -- 172.21.15.28:0/2710873169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d802ae60 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08eb400640 1 -- 172.21.15.28:0/2710873169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8032440 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08eb400640 1 -- 172.21.15.28:0/2710873169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80327d0 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08f3600640 1 -- 172.21.15.28:0/2710873169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0b0ee0 msgr2=0x7f08ec06b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.341+0000 7f08f3600640 1 --2- 172.21.15.28:0/2710873169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec06b230 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f08ec06f020 tx=0x7f08d802a110 comp rx=0 tx=0).stop 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 -- 172.21.15.28:0/2710873169 shutdown_connections 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 --2- 172.21.15.28:0/2710873169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec06b230 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 --2- 172.21.15.28:0/2710873169 >> v2:172.21.15.189:3300/0 conn(0x7f08ec06f4b0 0x7f08ec06f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 -- 172.21.15.28:0/2710873169 >> 172.21.15.28:0/2710873169 conn(0x7f08ec0b0310 msgr2=0x7f08ec0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 -- 172.21.15.28:0/2710873169 shutdown_connections 2024-09-13T17:07:57.343 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.342+0000 7f08f3600640 1 -- 172.21.15.28:0/2710873169 wait complete. 2024-09-13T17:07:57.344 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.343+0000 7f08f3600640 1 Processor -- start 2024-09-13T17:07:57.344 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.343+0000 7f08f3600640 1 -- start start 2024-09-13T17:07:57.346 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.346 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec0aab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.346 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08ec0cc6b0 0x7f08ec0ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec14f0b0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec14edd0 con 0x7f08ec0b0ee0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08ec0b18c0 con 0x7f08ec0cc6b0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38144/0 (socket says 172.21.15.28:38144) 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.344+0000 7f08f0c00640 1 -- 172.21.15.28:0/579164479 learned_addr learned my addr 172.21.15.28:0/579164479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08ebe00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec0aab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f1600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08ec0cc6b0 0x7f08ec0ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 -- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08ec0cc6b0 msgr2=0x7f08ec0ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08ec0cc6b0 0x7f08ec0ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 -- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ec0b0ee0 msgr2=0x7f08ec0aab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec0aab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 -- 172.21.15.28:0/579164479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8009400 con 0x7f08ec06f4b0 2024-09-13T17:07:57.347 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f0c00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08e000e9b0 tx=0x7f08e000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e0009670 con 0x7f08ec06f4b0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e000c9a0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e000cd50 con 0x7f08ec06f4b0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.345+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ec0b1ba0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.346+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ec0b23a0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.348 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.347+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ec06f890 con 0x7f08ec06f4b0 2024-09-13T17:07:57.349 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.347+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) v1 ==== 89089+0+0 (secure 0 0 0) 0x7f08e0013470 con 0x7f08ec06f4b0 2024-09-13T17:07:57.349 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.348+0000 7f08e8c00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 0x7f08cc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:07:57.349 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.348+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f08e00886b0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.349 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.349+0000 7f08ebe00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 0x7f08cc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:07:57.350 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.349+0000 7f08ebe00640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 0x7f08cc06d030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08d8034000 tx=0x7f08d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:07:57.354 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.353+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f08e00547a0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.534 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.533+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08ec0712a0 con 0x7f08ec06f4b0 2024-09-13T17:07:57.534 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.534+0000 7f08e8c00640 1 -- 172.21.15.28:0/579164479 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f08e0021090 con 0x7f08ec06f4b0 2024-09-13T17:07:57.535 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-13T17:07:57.535 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":12,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:07:54.887539+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 msgr2=0x7f08cc06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 0x7f08cc06d030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08d8034000 tx=0x7f08d802e000 comp rx=0 tx=0).stop 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 msgr2=0x7f08ec0c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f08e000e9b0 tx=0x7f08e000ee70 comp rx=0 tx=0).stop 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 shutdown_connections 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08cc06ab70 0x7f08cc06d030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08ec0cc6b0 0x7f08ec0ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.537 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ec0b0ee0 0x7f08ec0aab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.538 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 --2- 172.21.15.28:0/579164479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08ec06f4b0 0x7f08ec0c8620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:07:57.538 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 >> 172.21.15.28:0/579164479 conn(0x7f08ec0b0310 msgr2=0x7f08ec0ce170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:07:57.538 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 shutdown_connections 2024-09-13T17:07:57.538 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:07:57.536+0000 7f08f3600640 1 -- 172.21.15.28:0/579164479 wait complete. 2024-09-13T17:07:57.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-13T17:08:04.356 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:04.502 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.500+0000 7fb0b1400640 1 Processor -- start 2024-09-13T17:08:04.502 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.500+0000 7fb0b1400640 1 -- start start 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0b1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0ac06f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0b1400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac06c8b0 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0b1400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ac1592a0 con 0x7fb0ac06fbb0 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0b1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac070060 con 0x7fb0ac0ad830 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0aa800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0ac06c4d0 0x7fb0ac06b230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42502/0 (socket says 172.21.15.28:42502) 2024-09-13T17:08:04.503 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.501+0000 7fb0aa800640 1 -- 172.21.15.28:0/1850585592 learned_addr learned my addr 172.21.15.28:0/1850585592 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:04.505 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9e00640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0ac06f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.505 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917197382 0 0) 0x7fb0ac06c8b0 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.505 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb090003520 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.505 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872682350 0 0) 0x7fb090003520 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 msgr2=0x7fb0ac06f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0ac06f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 >> v1:172.21.15.189:6789/0 conn(0x7fb0ac06fbb0 legacy=0x7fb0ac163960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac159b40 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fb09c001670 con 0x7fb0ac06c4d0 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 >> v1:172.21.15.28:6789/0 conn(0x7fb0ac06c4d0 legacy=0x7fb0ac06b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0900039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb090003ee0 0x7fb0900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090004d00 0x7fb090005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac070060 con 0x7fb090004d00 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09000ba90 con 0x7fb090003ee0 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.502+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb09000de60 con 0x7fb0ac0ad830 2024-09-13T17:08:04.506 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090004d00 0x7fb090005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0aa800640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0900039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0a9e00640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb090003ee0 0x7fb0900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 -- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 msgr2=0x7fb0900039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac0ad830 0x7fb0900039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 -- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb090003ee0 msgr2=0x7fb0900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb090003ee0 0x7fb0900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.507 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.503+0000 7fb0ab200640 1 -- 172.21.15.28:0/1850585592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb090003520 con 0x7fb090004d00 2024-09-13T17:08:04.508 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.504+0000 7fb0ab200640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090004d00 0x7fb090005180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4002010 tx=0x7fb0900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7b9247c194874d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.508 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.504+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a4031600 con 0x7fb090004d00 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.504+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a402d050 con 0x7fb090004d00 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.504+0000 7fb0a9400640 1 -- 172.21.15.28:0/1850585592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a40319c0 con 0x7fb090004d00 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 -- 172.21.15.28:0/1850585592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090004d00 msgr2=0x7fb090005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 --2- 172.21.15.28:0/1850585592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb090004d00 0x7fb090005180 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb0a4002010 tx=0x7fb0900057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0ab200640 1 -- 172.21.15.28:0/1850585592 reap_dead start 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 -- 172.21.15.28:0/1850585592 shutdown_connections 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 -- 172.21.15.28:0/1850585592 >> 172.21.15.28:0/1850585592 conn(0x7fb0ac0b0310 msgr2=0x7fb0ac0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 -- 172.21.15.28:0/1850585592 shutdown_connections 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.505+0000 7fb0b1400640 1 -- 172.21.15.28:0/1850585592 wait complete. 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.507+0000 7fb0b1400640 1 Processor -- start 2024-09-13T17:08:04.509 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.507+0000 7fb0b1400640 1 -- start start 2024-09-13T17:08:04.511 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac158770 0x7fb0ac13e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.511 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.511 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0ac15a410 0x7fb0ac1ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.511 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac15a890 con 0x7fb0ac15d410 2024-09-13T17:08:04.511 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac15c370 con 0x7fb0ac15a410 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0b1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac15c650 con 0x7fb0ac158770 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58938/0 (socket says 172.21.15.28:58938) 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 -- 172.21.15.28:0/2680060244 learned_addr learned my addr 172.21.15.28:0/2680060244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0ab200640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0ac15a410 0x7fb0ac1ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0aa800640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac158770 0x7fb0ac13e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 -- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac158770 msgr2=0x7fb0ac13e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac158770 0x7fb0ac13e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 -- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0ac15a410 msgr2=0x7fb0ac1ed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0ac15a410 0x7fb0ac1ed860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.508+0000 7fb0a9e00640 1 -- 172.21.15.28:0/2680060244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac15c930 con 0x7fb0ac15d410 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0a9e00640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb098011910 tx=0x7fb098011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.512 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb098009590 con 0x7fb0ac15d410 2024-09-13T17:08:04.513 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09800faf0 con 0x7fb0ac15d410 2024-09-13T17:08:04.513 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09800fe60 con 0x7fb0ac15d410 2024-09-13T17:08:04.513 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac15cc10 con 0x7fb0ac15d410 2024-09-13T17:08:04.513 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.509+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac13ea50 con 0x7fb0ac15d410 2024-09-13T17:08:04.513 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.510+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac1597d0 con 0x7fb0ac15d410 2024-09-13T17:08:04.514 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.511+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb098014440 con 0x7fb0ac15d410 2024-09-13T17:08:04.514 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.512+0000 7fb0a2a00640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 0x7fb08c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.514 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.512+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb098089820 con 0x7fb0ac15d410 2024-09-13T17:08:04.514 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.512+0000 7fb0aa800640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 0x7fb08c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.515 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.513+0000 7fb0aa800640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 0x7fb08c06d0c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0010e0 tx=0x7fb09c001000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.518 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.517+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb098054d60 con 0x7fb0ac15d410 2024-09-13T17:08:04.699 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.697+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0ac0cf3f0 con 0x7fb0ac15d410 2024-09-13T17:08:04.700 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.698+0000 7fb0a2a00640 1 -- 172.21.15.28:0/2680060244 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fb098058380 con 0x7fb0ac15d410 2024-09-13T17:08:04.701 INFO:teuthology.misc.health.smithi028.stdout: 2024-09-13T17:08:04.701 INFO:teuthology.misc.health.smithi028.stdout:{"epoch":16,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:00.943360+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 msgr2=0x7fb08c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 0x7fb08c06d0c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb09c0010e0 tx=0x7fb09c001000 comp rx=0 tx=0).stop 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 msgr2=0x7fb0ac159ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb098011910 tx=0x7fb098011dd0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 shutdown_connections 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08c06ac00 0x7fb08c06d0c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.702 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0ac15a410 0x7fb0ac1ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0ac15d410 0x7fb0ac159ed0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 --2- 172.21.15.28:0/2680060244 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0ac158770 0x7fb0ac13e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 >> 172.21.15.28:0/2680060244 conn(0x7fb0ac0b0310 msgr2=0x7fb0ac155df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:04.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 shutdown_connections 2024-09-13T17:08:04.703 INFO:teuthology.misc.health.smithi028.stderr:2024-09-13T17:08:04.700+0000 7fb0b1400640 1 -- 172.21.15.28:0/2680060244 wait complete. 2024-09-13T17:08:04.715 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-13T17:08:04.716 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T17:08:04.716 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:08:04.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:08:04.720 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:08:04.720 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:08:04.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:08:04.723 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:08:04.724 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:08:04.724 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:08:04.728 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:08:04.729 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:08:04.729 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:08:04.732 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:08:04.733 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-13T17:08:04.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:04.851 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:08:04.855 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:08:04.867 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:08:04.867 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:08:04.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.897+0000 7f561c800640 1 Processor -- start 2024-09-13T17:08:04.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.898+0000 7f561c800640 1 -- start start 2024-09-13T17:08:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f561415a820 0x7f56140cbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561406dcc0 0x7f561415e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561c800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56140cc680 con 0x7f56140cef80 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56140b3190 con 0x7f561415a820 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56140cc3a0 con 0x7f561406dcc0 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f561415a820 0x7f56140cbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f561415a820 0x7f56140cbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58964/0 (socket says 172.21.15.28:58964) 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 -- 172.21.15.28:0/4229785466 learned_addr learned my addr 172.21.15.28:0/4229785466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619400640 1 --1- 172.21.15.28:0/4229785466 >> v1:172.21.15.28:6789/0 conn(0x7f56140cef80 0x7f561406d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42534/0 (socket says 172.21.15.28:42534) 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f561a800640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561406dcc0 0x7f561415e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 -- 172.21.15.28:0/4229785466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561406dcc0 msgr2=0x7f561415e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561406dcc0 0x7f561415e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 -- 172.21.15.28:0/4229785466 >> v1:172.21.15.28:6789/0 conn(0x7f56140cef80 legacy=0x7f561406d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.899+0000 7f5619e00640 1 -- 172.21.15.28:0/4229785466 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56140ce4a0 con 0x7f561415a820 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.900+0000 7f5619e00640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561415a820 0x7f56140cbd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56080098a0 tx=0x7f560802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e9ab3cde2f83fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f5618a00640 1 -- 172.21.15.28:0/4229785466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608032440 con 0x7f561415a820 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f5618a00640 1 -- 172.21.15.28:0/4229785466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56080325a0 con 0x7f561415a820 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f5618a00640 1 -- 172.21.15.28:0/4229785466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608032930 con 0x7f561415a820 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 -- 172.21.15.28:0/4229785466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561415a820 msgr2=0x7f56140cbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561415a820 0x7f56140cbd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56080098a0 tx=0x7f560802a2a0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 -- 172.21.15.28:0/4229785466 shutdown_connections 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561406dcc0 0x7f561415e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 --2- 172.21.15.28:0/4229785466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561415a820 0x7f56140cbd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 -- 172.21.15.28:0/4229785466 >> 172.21.15.28:0/4229785466 conn(0x7f56140b22c0 msgr2=0x7f56140b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:04.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 -- 172.21.15.28:0/4229785466 shutdown_connections 2024-09-13T17:08:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.901+0000 7f561c800640 1 -- 172.21.15.28:0/4229785466 wait complete. 2024-09-13T17:08:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.903+0000 7f561c800640 1 Processor -- start 2024-09-13T17:08:04.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.903+0000 7f561c800640 1 -- start start 2024-09-13T17:08:04.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56140cef80 0x7f56141e4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 0x7f561415ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56141e57c0 con 0x7f561406dcc0 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56141e5200 con 0x7f56140cef80 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f561c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56141e54e0 con 0x7f561415a820 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f5619e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f5619e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58966/0 (socket says 172.21.15.28:58966) 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f5619e00640 1 -- 172.21.15.28:0/3882556346 learned_addr learned my addr 172.21.15.28:0/3882556346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.904+0000 7f5619400640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56140cef80 0x7f56141e4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f561a800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 0x7f561415ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 -- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 msgr2=0x7f561415ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 0x7f561415ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 -- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56140cef80 msgr2=0x7f56141e4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56140cef80 0x7f56141e4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 -- 172.21.15.28:0/3882556346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56080094d0 con 0x7f561406dcc0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f561a800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 0x7f561415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.905+0000 7f5619e00640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5608000c00 tx=0x7f56080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.906+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608032440 con 0x7f561406dcc0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.906+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5608030b80 con 0x7f561406dcc0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.906+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56140c7930 con 0x7f561406dcc0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.906+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560803d440 con 0x7f561406dcc0 2024-09-13T17:08:04.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.906+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56140c9db0 con 0x7f561406dcc0 2024-09-13T17:08:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.907+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614067690 con 0x7f561406dcc0 2024-09-13T17:08:04.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.908+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5608039040 con 0x7f561406dcc0 2024-09-13T17:08:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.909+0000 7f560e000640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 0x7f55f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.909+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f56080a9a00 con 0x7f561406dcc0 2024-09-13T17:08:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.909+0000 7f5619400640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 0x7f55f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:04.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.910+0000 7f5619400640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 0x7f55f806d0c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56140d1700 tx=0x7f5604000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:04.913+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5608074fd0 con 0x7f561406dcc0 2024-09-13T17:08:05.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.093+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56140cbd50 con 0x7f561406dcc0 2024-09-13T17:08:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.094+0000 7f560e000640 1 -- 172.21.15.28:0/3882556346 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7f56080a6250 con 0x7f561406dcc0 2024-09-13T17:08:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:05.096 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:00.943360+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 msgr2=0x7f55f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 0x7f55f806d0c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56140d1700 tx=0x7f5604000ed0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 msgr2=0x7f56140c9870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5608000c00 tx=0x7f56080046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 shutdown_connections 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55f806ac00 0x7f55f806d0c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f561415a820 0x7f561415ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56140cef80 0x7f56141e4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 --2- 172.21.15.28:0/3882556346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f561406dcc0 0x7f56140c9870 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 >> 172.21.15.28:0/3882556346 conn(0x7f56140b22c0 msgr2=0x7f561406d5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 shutdown_connections 2024-09-13T17:08:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.096+0000 7f561c800640 1 -- 172.21.15.28:0/3882556346 wait complete. 2024-09-13T17:08:05.114 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:05.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.270+0000 7f21a3600640 1 Processor -- start 2024-09-13T17:08:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.270+0000 7f21a3600640 1 -- start start 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.271+0000 7f21a3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c15a810 0x7f219c06cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.271+0000 7f21a3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f219c06bcf0 0x7f219c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.271+0000 7f21a3600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c06d7c0 con 0x7f219c0cc990 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.271+0000 7f21a3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c06d140 con 0x7f219c06bcf0 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.271+0000 7f21a3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c06fe10 con 0x7f219c15a810 2024-09-13T17:08:05.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f219c06bcf0 0x7f219c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f219c06bcf0 0x7f219c0cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58976/0 (socket says 172.21.15.28:58976) 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 -- 172.21.15.28:0/891136702 learned_addr learned my addr 172.21.15.28:0/891136702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f21a0c00640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c15a810 0x7f219c06cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 -- 172.21.15.28:0/891136702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c15a810 msgr2=0x7f219c06cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c15a810 0x7f219c06cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 -- 172.21.15.28:0/891136702 >> v1:172.21.15.28:6789/0 conn(0x7f219c0cc990 legacy=0x7f219c0caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.272+0000 7f2193e00640 1 -- 172.21.15.28:0/891136702 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c06d330 con 0x7f219c06bcf0 2024-09-13T17:08:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.273+0000 7f2193e00640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c06bcf0 0x7f219c0cc450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2184009870 tx=0x7f218402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125b2bd9f5ce61b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.273+0000 7f219be00640 1 -- 172.21.15.28:0/891136702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184032640 con 0x7f219c06bcf0 2024-09-13T17:08:05.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.273+0000 7f219be00640 1 -- 172.21.15.28:0/891136702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21840327a0 con 0x7f219c06bcf0 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.273+0000 7f219be00640 1 -- 172.21.15.28:0/891136702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184032b30 con 0x7f219c06bcf0 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 -- 172.21.15.28:0/891136702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c06bcf0 msgr2=0x7f219c0cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c06bcf0 0x7f219c0cc450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2184009870 tx=0x7f218402a300 comp rx=0 tx=0).stop 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 -- 172.21.15.28:0/891136702 shutdown_connections 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c06bcf0 0x7f219c0cc450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 --2- 172.21.15.28:0/891136702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c15a810 0x7f219c06cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 -- 172.21.15.28:0/891136702 >> 172.21.15.28:0/891136702 conn(0x7f219c0b0200 msgr2=0x7f219c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 -- 172.21.15.28:0/891136702 shutdown_connections 2024-09-13T17:08:05.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.274+0000 7f21a3600640 1 -- 172.21.15.28:0/891136702 wait complete. 2024-09-13T17:08:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.276+0000 7f21a3600640 1 Processor -- start 2024-09-13T17:08:05.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.276+0000 7f21a3600640 1 -- start start 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 0x7f219c14d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 0x7f219c150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c158e00 con 0x7f219c15a810 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219c1581f0 con 0x7f219c0cc990 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c1584d0 con 0x7f219c06bcf0 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58982/0 (socket says 172.21.15.28:58982) 2024-09-13T17:08:05.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 -- 172.21.15.28:0/2316971628 learned_addr learned my addr 172.21.15.28:0/2316971628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f2193e00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 0x7f219c150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a0c00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 0x7f219c14d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 -- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 msgr2=0x7f219c14d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 0x7f219c14d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 -- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 msgr2=0x7f219c150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 0x7f219c150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a1600640 1 -- 172.21.15.28:0/2316971628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21840094d0 con 0x7f219c15a810 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f21a0c00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 0x7f219c14d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.277+0000 7f2193e00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 0x7f219c150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f21a1600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f219400c730 tx=0x7f219400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194003b70 con 0x7f219c15a810 2024-09-13T17:08:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2194003cd0 con 0x7f219c15a810 2024-09-13T17:08:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219400aa90 con 0x7f219c15a810 2024-09-13T17:08:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c1587b0 con 0x7f219c15a810 2024-09-13T17:08:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.278+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c13e490 con 0x7f219c15a810 2024-09-13T17:08:05.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.279+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c158f70 con 0x7f219c15a810 2024-09-13T17:08:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.280+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f219400abf0 con 0x7f219c15a810 2024-09-13T17:08:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.281+0000 7f2199600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 0x7f217806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.281+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2194088e30 con 0x7f219c15a810 2024-09-13T17:08:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.281+0000 7f21a0c00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 0x7f217806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.282+0000 7f21a0c00640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 0x7f217806d0f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f218c009a20 tx=0x7f218c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.285+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2194054350 con 0x7f219c15a810 2024-09-13T17:08:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.461+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f219c0cc450 con 0x7f219c15a810 2024-09-13T17:08:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.462+0000 7f2199600640 1 -- 172.21.15.28:0/2316971628 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2194057970 con 0x7f219c15a810 2024-09-13T17:08:05.464 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.463+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 msgr2=0x7f217806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.463+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 0x7f217806d0f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f218c009a20 tx=0x7f218c008040 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.463+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 msgr2=0x7f219c13d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f219400c730 tx=0x7f219400cbf0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 shutdown_connections 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f217806ac30 0x7f217806d0f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f219c15a810 0x7f219c13d6f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f219c0cc990 0x7f219c150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 --2- 172.21.15.28:0/2316971628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f219c06bcf0 0x7f219c14d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 >> 172.21.15.28:0/2316971628 conn(0x7f219c0b0200 msgr2=0x7f219c0aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:05.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 shutdown_connections 2024-09-13T17:08:05.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.464+0000 7f21a3600640 1 -- 172.21.15.28:0/2316971628 wait complete. 2024-09-13T17:08:05.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:05.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.633+0000 7fdea7e00640 1 Processor -- start 2024-09-13T17:08:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.634+0000 7fdea7e00640 1 -- start start 2024-09-13T17:08:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159440 0x7fdea80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdea8159820 0x7fdea80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea80c3850 con 0x7fdea80b22e0 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea815a930 con 0x7fdea8159440 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea8159d20 con 0x7fdea8159820 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159440 0x7fdea80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdea8159440 0x7fdea80b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58998/0 (socket says 172.21.15.28:58998) 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea6a00640 1 -- 172.21.15.28:0/991540708 learned_addr learned my addr 172.21.15.28:0/991540708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fdea7400640 1 --1- 172.21.15.28:0/991540708 >> v1:172.21.15.28:6789/0 conn(0x7fdea80b22e0 0x7fdea80b2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42544/0 (socket says 172.21.15.28:42544) 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.635+0000 7fde9e000640 1 --2- 172.21.15.28:0/991540708 >> v2:172.21.15.189:3300/0 conn(0x7fdea8159820 0x7fdea80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.636+0000 7fdea6a00640 1 -- 172.21.15.28:0/991540708 >> v2:172.21.15.189:3300/0 conn(0x7fdea8159820 msgr2=0x7fdea80b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.636+0000 7fdea6a00640 1 --2- 172.21.15.28:0/991540708 >> v2:172.21.15.189:3300/0 conn(0x7fdea8159820 0x7fdea80b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.636+0000 7fdea6a00640 1 -- 172.21.15.28:0/991540708 >> v1:172.21.15.28:6789/0 conn(0x7fdea80b22e0 legacy=0x7fdea80b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.636+0000 7fdea6a00640 1 -- 172.21.15.28:0/991540708 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea815a000 con 0x7fdea8159440 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea6a00640 1 --2- 172.21.15.28:0/991540708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159440 0x7fdea80b0440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde98009870 tx=0x7fde9802a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39a68ca0641b8654 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea6000640 1 -- 172.21.15.28:0/991540708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98032640 con 0x7fdea8159440 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea6000640 1 -- 172.21.15.28:0/991540708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980327a0 con 0x7fdea8159440 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea6000640 1 -- 172.21.15.28:0/991540708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98032b30 con 0x7fdea8159440 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea7e00640 1 -- 172.21.15.28:0/991540708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159440 msgr2=0x7fdea80b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.637+0000 7fdea7e00640 1 --2- 172.21.15.28:0/991540708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159440 0x7fdea80b0440 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fde98009870 tx=0x7fde9802a330 comp rx=0 tx=0).stop 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 -- 172.21.15.28:0/991540708 shutdown_connections 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 --2- 172.21.15.28:0/991540708 >> v2:172.21.15.189:3300/0 conn(0x7fdea8159820 0x7fdea80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 --2- 172.21.15.28:0/991540708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159440 0x7fdea80b0440 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 -- 172.21.15.28:0/991540708 >> 172.21.15.28:0/991540708 conn(0x7fdea8072e40 msgr2=0x7fdea8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 -- 172.21.15.28:0/991540708 shutdown_connections 2024-09-13T17:08:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.638+0000 7fdea7e00640 1 -- 172.21.15.28:0/991540708 wait complete. 2024-09-13T17:08:05.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.639+0000 7fdea7e00640 1 Processor -- start 2024-09-13T17:08:05.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.639+0000 7fdea7e00640 1 -- start start 2024-09-13T17:08:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdea80b22e0 0x7fdea814c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdea8159440 0x7fdea8164f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea814beb0 con 0x7fdea8159820 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea814b300 con 0x7fdea80b22e0 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.640+0000 7fdea7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea814b510 con 0x7fdea8159440 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59012/0 (socket says 172.21.15.28:59012) 2024-09-13T17:08:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 -- 172.21.15.28:0/2843536867 learned_addr learned my addr 172.21.15.28:0/2843536867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea6a00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdea80b22e0 0x7fdea814c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fde9e000640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdea8159440 0x7fdea8164f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 -- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdea8159440 msgr2=0x7fdea8164f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdea8159440 0x7fdea8164f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 -- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdea80b22e0 msgr2=0x7fdea814c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdea80b22e0 0x7fdea814c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 -- 172.21.15.28:0/2843536867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980094d0 con 0x7fdea8159820 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7400640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c730 tx=0x7fdea000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0004d60 con 0x7fdea8159820 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea000a5e0 con 0x7fdea8159820 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea814b7f0 con 0x7fdea8159820 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.641+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea000a990 con 0x7fdea8159820 2024-09-13T17:08:05.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.642+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea81536c0 con 0x7fdea8159820 2024-09-13T17:08:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.643+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea814b650 con 0x7fdea8159820 2024-09-13T17:08:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.643+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdea000ad00 con 0x7fdea8159820 2024-09-13T17:08:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.644+0000 7fde9f400640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 0x7fde8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:05.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.644+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdea00888e0 con 0x7fdea8159820 2024-09-13T17:08:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.645+0000 7fdea6a00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 0x7fde8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:05.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.646+0000 7fdea6a00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 0x7fde8406d0c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde98034000 tx=0x7fde9802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:05.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.648+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdea814b650 con 0x7fdea8159820 2024-09-13T17:08:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.861+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdea814bff0 con 0x7fdea8159820 2024-09-13T17:08:05.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.863+0000 7fde9f400640 1 -- 172.21.15.28:0/2843536867 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1256 (secure 0 0 0) 0x7fdea814bff0 con 0x7fdea8159820 2024-09-13T17:08:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:05.865 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-09-13T17:08:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-09-13T17:08:05.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.865+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 msgr2=0x7fde8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.865+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 0x7fde8406d0c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fde98034000 tx=0x7fde9802e000 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.865+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 msgr2=0x7fdea81f66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.865+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c730 tx=0x7fdea000cbf0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 shutdown_connections 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde8406ac00 0x7fde8406d0c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdea8159820 0x7fdea81f66b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdea8159440 0x7fdea8164f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 --2- 172.21.15.28:0/2843536867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdea80b22e0 0x7fdea814c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 >> 172.21.15.28:0/2843536867 conn(0x7fdea8072e40 msgr2=0x7fdea8152ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 shutdown_connections 2024-09-13T17:08:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:05.866+0000 7fdea7e00640 1 -- 172.21.15.28:0/2843536867 wait complete. 2024-09-13T17:08:07.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.050+0000 7efeefe00640 1 Processor -- start 2024-09-13T17:08:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.050+0000 7efeefe00640 1 -- start start 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeefe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef00b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efef00b0440 0x7efef00b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeefe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efef00cad20 con 0x7efef00b21c0 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeefe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efef00b0e10 con 0x7efef00b0440 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeefe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef006bb40 con 0x7efef015fb90 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeef400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efef00b0440 0x7efef00b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeef400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efef00b0440 0x7efef00b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeef400640 1 -- 172.21.15.28:0/1158119892 learned_addr learned my addr 172.21.15.28:0/1158119892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efee6000640 1 --1- 172.21.15.28:0/1158119892 >> v1:172.21.15.28:6789/0 conn(0x7efef00b21c0 0x7efef006b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42556/0 (socket says 172.21.15.28:42556) 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.051+0000 7efeeea00640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef00b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.052+0000 7efeef400640 1 -- 172.21.15.28:0/1158119892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 msgr2=0x7efef00b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:08.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.052+0000 7efeef400640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef00b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.052+0000 7efeef400640 1 -- 172.21.15.28:0/1158119892 >> v1:172.21.15.28:6789/0 conn(0x7efef00b21c0 legacy=0x7efef006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.052+0000 7efeef400640 1 -- 172.21.15.28:0/1158119892 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef006be20 con 0x7efef00b0440 2024-09-13T17:08:08.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.052+0000 7efeee000640 1 -- 172.21.15.28:0/1158119892 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906168439 0 0) 0x7efef00cad20 con 0x7efef00b21c0 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.053+0000 7efeef400640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00b08d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efee8009e50 tx=0x7efee802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231abb38d38c04a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.053+0000 7efeee000640 1 -- 172.21.15.28:0/1158119892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8031600 con 0x7efef00b0440 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.053+0000 7efeee000640 1 -- 172.21.15.28:0/1158119892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee802d050 con 0x7efef00b0440 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.053+0000 7efeee000640 1 -- 172.21.15.28:0/1158119892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8031980 con 0x7efef00b0440 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 -- 172.21.15.28:0/1158119892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 msgr2=0x7efef00b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00b08d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efee8009e50 tx=0x7efee802a5f0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 -- 172.21.15.28:0/1158119892 shutdown_connections 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00b08d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 --2- 172.21.15.28:0/1158119892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef00b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 -- 172.21.15.28:0/1158119892 >> 172.21.15.28:0/1158119892 conn(0x7efef0072e40 msgr2=0x7efef0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 -- 172.21.15.28:0/1158119892 shutdown_connections 2024-09-13T17:08:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.054+0000 7efeefe00640 1 -- 172.21.15.28:0/1158119892 wait complete. 2024-09-13T17:08:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.055+0000 7efeefe00640 1 Processor -- start 2024-09-13T17:08:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.056+0000 7efeefe00640 1 -- start start 2024-09-13T17:08:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.056+0000 7efeefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.056+0000 7efeefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef00b21c0 0x7efef015bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.056+0000 7efeefe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeefe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef00ca0c0 con 0x7efef00b0440 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeefe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef015ac10 con 0x7efef00b21c0 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeefe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef015aef0 con 0x7efef015fb90 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59020/0 (socket says 172.21.15.28:59020) 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 -- 172.21.15.28:0/3964039699 learned_addr learned my addr 172.21.15.28:0/3964039699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeef400640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efee6000640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef00b21c0 0x7efef015bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 -- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 msgr2=0x7efef015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 -- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef00b21c0 msgr2=0x7efef015bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef00b21c0 0x7efef015bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 -- 172.21.15.28:0/3964039699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee8009b30 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.057+0000 7efeeea00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9b0 tx=0x7efee000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.058+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee0009670 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.058+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efee000c9a0 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.058+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee000cd50 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.058+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef015b1d0 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.058+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef015b790 con 0x7efef00b0440 2024-09-13T17:08:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.059+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef00b08d0 con 0x7efef00b0440 2024-09-13T17:08:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.060+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efee0013470 con 0x7efef00b0440 2024-09-13T17:08:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.061+0000 7efee7400640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 0x7efecc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.061+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efee00893a0 con 0x7efef00b0440 2024-09-13T17:08:08.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.061+0000 7efee6000640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 0x7efecc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.062+0000 7efee6000640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 0x7efecc06d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efed8000f60 tx=0x7efed8007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.065+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efee0054830 con 0x7efef00b0440 2024-09-13T17:08:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.285+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efef00b31b0 con 0x7efef00b0440 2024-09-13T17:08:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.286+0000 7efee7400640 1 -- 172.21.15.28:0/3964039699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7efee0017de0 con 0x7efef00b0440 2024-09-13T17:08:08.288 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:08.288 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4294,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-09-13T17:08:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-09-13T17:08:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 msgr2=0x7efecc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 0x7efecc06d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efed8000f60 tx=0x7efed8007b50 comp rx=0 tx=0).stop 2024-09-13T17:08:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 msgr2=0x7efef00c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efee000e9b0 tx=0x7efee000ee70 comp rx=0 tx=0).stop 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 shutdown_connections 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efecc06ac00 0x7efecc06d0c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efef015fb90 0x7efef015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efef00b21c0 0x7efef015bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 --2- 172.21.15.28:0/3964039699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efef00b0440 0x7efef00c9b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 >> 172.21.15.28:0/3964039699 conn(0x7efef0072e40 msgr2=0x7efef00afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 shutdown_connections 2024-09-13T17:08:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.288+0000 7efeefe00640 1 -- 172.21.15.28:0/3964039699 wait complete. 2024-09-13T17:08:08.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.458+0000 7fbcef200640 1 Processor -- start 2024-09-13T17:08:08.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.458+0000 7fbcef200640 1 -- start start 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcef200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbcf00b0980 0x7fbcf00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcef200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcf006b900 con 0x7fbcf0154a90 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcef200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf00b0df0 con 0x7fbcf00b0980 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf006bbe0 con 0x7fbcf006b360 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbced400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbced400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55350/0 (socket says 172.21.15.28:55350) 2024-09-13T17:08:08.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbced400640 1 -- 172.21.15.28:0/1266757146 learned_addr learned my addr 172.21.15.28:0/1266757146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcede00640 1 --1- 172.21.15.28:0/1266757146 >> v1:172.21.15.189:6789/0 conn(0x7fbcf0154a90 0x7fbcf015b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:37778/0 (socket says 172.21.15.28:37778) 2024-09-13T17:08:08.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.460+0000 7fbcee800640 1 --2- 172.21.15.28:0/1266757146 >> v2:172.21.15.189:3300/0 conn(0x7fbcf00b0980 0x7fbcf00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbceca00640 1 -- 172.21.15.28:0/1266757146 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104651781 0 0) 0x7fbcf006b900 con 0x7fbcf0154a90 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbceca00640 1 -- 172.21.15.28:0/1266757146 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd4003520 con 0x7fbcf0154a90 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbcee800640 1 -- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 msgr2=0x7fbcf00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbcee800640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbcee800640 1 -- 172.21.15.28:0/1266757146 >> v1:172.21.15.189:6789/0 conn(0x7fbcf0154a90 legacy=0x7fbcf015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbcee800640 1 -- 172.21.15.28:0/1266757146 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf006bec0 con 0x7fbcf00b0980 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.461+0000 7fbced400640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbcee800640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf00b0980 0x7fbcf00af4b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009d90 tx=0x7fbce802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d58c9e8cb18167 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbceca00640 1 -- 172.21.15.28:0/1266757146 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8007740 con 0x7fbcf00b0980 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbceca00640 1 -- 172.21.15.28:0/1266757146 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce802d050 con 0x7fbcf00b0980 2024-09-13T17:08:08.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbceca00640 1 -- 172.21.15.28:0/1266757146 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8007ac0 con 0x7fbcf00b0980 2024-09-13T17:08:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbcef200640 1 -- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf00b0980 msgr2=0x7fbcf00af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.462+0000 7fbcef200640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf00b0980 0x7fbcf00af4b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009d90 tx=0x7fbce802a660 comp rx=0 tx=0).stop 2024-09-13T17:08:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 -- 172.21.15.28:0/1266757146 shutdown_connections 2024-09-13T17:08:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf00b0980 0x7fbcf00af4b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 --2- 172.21.15.28:0/1266757146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf006b360 0x7fbcf00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 -- 172.21.15.28:0/1266757146 >> 172.21.15.28:0/1266757146 conn(0x7fbcf0072e40 msgr2=0x7fbcf0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 -- 172.21.15.28:0/1266757146 shutdown_connections 2024-09-13T17:08:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.463+0000 7fbcef200640 1 -- 172.21.15.28:0/1266757146 wait complete. 2024-09-13T17:08:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.464+0000 7fbcef200640 1 Processor -- start 2024-09-13T17:08:08.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.464+0000 7fbcef200640 1 -- start start 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf006b360 0x7fbcf00ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf00b0980 0x7fbcf013b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf013c480 con 0x7fbcf0154a90 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf01f41e0 con 0x7fbcf006b360 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.465+0000 7fbcef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf01f3d50 con 0x7fbcf00b0980 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59026/0 (socket says 172.21.15.28:59026) 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 -- 172.21.15.28:0/4059039436 learned_addr learned my addr 172.21.15.28:0/4059039436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbced400640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf00b0980 0x7fbcf013b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcede00640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf006b360 0x7fbcf00ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 -- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf00b0980 msgr2=0x7fbcf013b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf00b0980 0x7fbcf013b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 -- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf006b360 msgr2=0x7fbcf00ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf006b360 0x7fbcf00ac2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 -- 172.21.15.28:0/4059039436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcdc0094d0 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.466+0000 7fbcee800640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009eb0 tx=0x7fbce8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.467+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce803c830 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.467+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce803c990 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.467+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce803cd20 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.467+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbce8009a70 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.467+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf01f3700 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.468+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf01f3ec0 con 0x7fbcf0154a90 2024-09-13T17:08:08.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.469+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbce8031400 con 0x7fbcf0154a90 2024-09-13T17:08:08.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.469+0000 7fbce6000640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 0x7fbcd006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.469+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbce80aa330 con 0x7fbcf0154a90 2024-09-13T17:08:08.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.470+0000 7fbcede00640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 0x7fbcd006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.470+0000 7fbcede00640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 0x7fbcd006d0f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbce00011e0 tx=0x7fbce0007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.474+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbce8075870 con 0x7fbcf0154a90 2024-09-13T17:08:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.655+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf006b900 con 0x7fbcf0154a90 2024-09-13T17:08:08.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.656+0000 7fbce6000640 1 -- 172.21.15.28:0/4059039436 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12850 (secure 0 0 0) 0x7fbce8009450 con 0x7fbcf0154a90 2024-09-13T17:08:08.658 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:08.658 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:00.943360+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 msgr2=0x7fbcd006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 0x7fbcd006d0f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbce00011e0 tx=0x7fbce0007d30 comp rx=0 tx=0).stop 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 msgr2=0x7fbcf013bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009eb0 tx=0x7fbce8004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 shutdown_connections 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcd006ac30 0x7fbcd006d0f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcf0154a90 0x7fbcf013bd70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbcf00b0980 0x7fbcf013b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 --2- 172.21.15.28:0/4059039436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcf006b360 0x7fbcf00ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 >> 172.21.15.28:0/4059039436 conn(0x7fbcf0072e40 msgr2=0x7fbcf006fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 shutdown_connections 2024-09-13T17:08:08.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.658+0000 7fbcef200640 1 -- 172.21.15.28:0/4059039436 wait complete. 2024-09-13T17:08:08.675 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:08.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:08.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:08:08.724 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:08:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.831+0000 7f7156400640 1 Processor -- start 2024-09-13T17:08:08.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.831+0000 7f7156400640 1 -- start start 2024-09-13T17:08:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7156400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71500cde30 0x7f7150071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7156400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f7150070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7156400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7150159e50 con 0x7f7150071100 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7156400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71500715b0 con 0x7f71500cde30 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7156400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71500ce210 con 0x7f7150071f60 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7155000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71500cde30 0x7f7150071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7155a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7150071100 0x7f715015cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42564/0 (socket says 172.21.15.28:42564) 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7155000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71500cde30 0x7f7150071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59032/0 (socket says 172.21.15.28:59032) 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f7155a00640 1 -- 172.21.15.28:0/264254537 learned_addr learned my addr 172.21.15.28:0/264254537 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.833+0000 7f714fe00640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f7150070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.834+0000 7f7155000640 1 -- 172.21.15.28:0/264254537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 msgr2=0x7f7150070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.834+0000 7f7155000640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f7150070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.834+0000 7f7155000640 1 -- 172.21.15.28:0/264254537 >> v1:172.21.15.28:6789/0 conn(0x7f7150071100 legacy=0x7f715015cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.834+0000 7f7155000640 1 -- 172.21.15.28:0/264254537 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715015a130 con 0x7f71500cde30 2024-09-13T17:08:08.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.834+0000 7f714f400640 1 -- 172.21.15.28:0/264254537 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154233850 0 0) 0x7f7150159e50 con 0x7f7150071100 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7155000640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f7150071a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f71440097a0 tx=0x7f714402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e75ad8285d05d4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f714f400640 1 -- 172.21.15.28:0/264254537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144032440 con 0x7f71500cde30 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f714f400640 1 -- 172.21.15.28:0/264254537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71440325a0 con 0x7f71500cde30 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f714f400640 1 -- 172.21.15.28:0/264254537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144032930 con 0x7f71500cde30 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 -- 172.21.15.28:0/264254537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 msgr2=0x7f7150071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f7150071a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f71440097a0 tx=0x7f714402a180 comp rx=0 tx=0).stop 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 -- 172.21.15.28:0/264254537 shutdown_connections 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f7150070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 --2- 172.21.15.28:0/264254537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f7150071a20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 -- 172.21.15.28:0/264254537 >> 172.21.15.28:0/264254537 conn(0x7f7150069e80 msgr2=0x7f715006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.835+0000 7f7156400640 1 -- 172.21.15.28:0/264254537 shutdown_connections 2024-09-13T17:08:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.836+0000 7f7156400640 1 -- 172.21.15.28:0/264254537 wait complete. 2024-09-13T17:08:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.837+0000 7f7156400640 1 Processor -- start 2024-09-13T17:08:08.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.837+0000 7f7156400640 1 -- start start 2024-09-13T17:08:08.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 0x7f7150155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f71501f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150150250 con 0x7f71500cde30 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71501f69a0 con 0x7f7150071100 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7156400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71501563f0 con 0x7f7150071f60 2024-09-13T17:08:08.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7155a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7155a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59042/0 (socket says 172.21.15.28:59042) 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7155a00640 1 -- 172.21.15.28:0/1751872084 learned_addr learned my addr 172.21.15.28:0/1751872084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f7155000640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 0x7f7150155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.838+0000 7f714fe00640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f71501f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 -- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 msgr2=0x7f71501f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f71501f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 -- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 msgr2=0x7f7150155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 0x7f7150155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 -- 172.21.15.28:0/1751872084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009400 con 0x7f71500cde30 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155000640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 0x7f7150155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7155a00640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f714800c780 tx=0x7f714800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714800ce90 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714800a640 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714800a9d0 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71501566d0 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.839+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150156c90 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.840+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7150150400 con 0x7f71500cde30 2024-09-13T17:08:08.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.841+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f714800ab30 con 0x7f71500cde30 2024-09-13T17:08:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.842+0000 7f714cc00640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 0x7f713006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.842+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f71480883a0 con 0x7f71500cde30 2024-09-13T17:08:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.843+0000 7f7155000640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 0x7f713006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.844+0000 7f7155000640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 0x7f713006d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7144034000 tx=0x7f714402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:08.846+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7150150400 con 0x7f71500cde30 2024-09-13T17:08:09.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.026+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7150070bc0 con 0x7f71500cde30 2024-09-13T17:08:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.027+0000 7f714cc00640 1 -- 172.21.15.28:0/1751872084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7148018020 con 0x7f71500cde30 2024-09-13T17:08:09.029 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 msgr2=0x7f713006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 0x7f713006d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7144034000 tx=0x7f714402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 msgr2=0x7f71500b2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f714800c780 tx=0x7f714800cc40 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 shutdown_connections 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f713006ac30 0x7f713006d0f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71500cde30 0x7f71500b2290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7150071f60 0x7f71501f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 --2- 172.21.15.28:0/1751872084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7150071100 0x7f7150155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 >> 172.21.15.28:0/1751872084 conn(0x7f7150069e80 msgr2=0x7f7150158f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 shutdown_connections 2024-09-13T17:08:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.029+0000 7f7156400640 1 -- 172.21.15.28:0/1751872084 wait complete. 2024-09-13T17:08:09.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:08:09.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.241+0000 7f4d96000640 1 Processor -- start 2024-09-13T17:08:09.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.241+0000 7f4d96000640 1 -- start start 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.242+0000 7f4d96000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d90070430 0x7f4d9015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.242+0000 7f4d96000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d9006f3f0 0x7f4d9015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.242+0000 7f4d96000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9015ab60 con 0x7f4d90159770 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.242+0000 7f4d96000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d9006f920 con 0x7f4d90070430 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.242+0000 7f4d96000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9015b290 con 0x7f4d9006f3f0 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d90070430 0x7f4d9015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d90070430 0x7f4d9015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59050/0 (socket says 172.21.15.28:59050) 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 -- 172.21.15.28:0/3047708208 learned_addr learned my addr 172.21.15.28:0/3047708208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8fe00640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d9006f3f0 0x7f4d9015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 -- 172.21.15.28:0/3047708208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d9006f3f0 msgr2=0x7f4d9015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:09.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d9006f3f0 0x7f4d9015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 -- 172.21.15.28:0/3047708208 >> v1:172.21.15.189:6789/0 conn(0x7f4d90159770 legacy=0x7f4d9006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.243+0000 7f4d8ea00640 1 -- 172.21.15.28:0/3047708208 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9015b570 con 0x7f4d90070430 2024-09-13T17:08:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.244+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d90070430 0x7f4d9015bd70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009870 tx=0x7f4d7c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5911b6b24f9d890f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.244+0000 7f4d8e000640 1 -- 172.21.15.28:0/3047708208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c032440 con 0x7f4d90070430 2024-09-13T17:08:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.244+0000 7f4d8e000640 1 -- 172.21.15.28:0/3047708208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7c0325a0 con 0x7f4d90070430 2024-09-13T17:08:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.244+0000 7f4d8e000640 1 -- 172.21.15.28:0/3047708208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c032930 con 0x7f4d90070430 2024-09-13T17:08:09.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 -- 172.21.15.28:0/3047708208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d90070430 msgr2=0x7f4d9015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d90070430 0x7f4d9015bd70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009870 tx=0x7f4d7c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 -- 172.21.15.28:0/3047708208 shutdown_connections 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d9006f3f0 0x7f4d9015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 --2- 172.21.15.28:0/3047708208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d90070430 0x7f4d9015bd70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 -- 172.21.15.28:0/3047708208 >> 172.21.15.28:0/3047708208 conn(0x7f4d900b0270 msgr2=0x7f4d900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 -- 172.21.15.28:0/3047708208 shutdown_connections 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.245+0000 7f4d96000640 1 -- 172.21.15.28:0/3047708208 wait complete. 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.247+0000 7f4d96000640 1 Processor -- start 2024-09-13T17:08:09.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.247+0000 7f4d96000640 1 -- start start 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 0x7f4d90148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 0x7f4d900ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901e0f30 con 0x7f4d9006f3f0 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901e59f0 con 0x7f4d90070430 2024-09-13T17:08:09.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d96000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d901e4de0 con 0x7f4d90159770 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59054/0 (socket says 172.21.15.28:59054) 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 -- 172.21.15.28:0/3337432689 learned_addr learned my addr 172.21.15.28:0/3337432689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 0x7f4d90148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8fe00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 0x7f4d900ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 -- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 msgr2=0x7f4d900ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 0x7f4d900ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 -- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 msgr2=0x7f4d90148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 0x7f4d90148cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8f400640 1 -- 172.21.15.28:0/3337432689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c0094d0 con 0x7f4d9006f3f0 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 0x7f4d90148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.248+0000 7f4d8fe00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 0x7f4d900ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:09.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d8f400640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c400 tx=0x7f4d8000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d800052d0 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80005430 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80003830 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d901e50c0 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.249+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d901e5680 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.250+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d90154630 con 0x7f4d9006f3f0 2024-09-13T17:08:09.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.251+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4d80003990 con 0x7f4d9006f3f0 2024-09-13T17:08:09.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.252+0000 7f4d87400640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 0x7f4d7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.252+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4d80088470 con 0x7f4d9006f3f0 2024-09-13T17:08:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.252+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 0x7f4d7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.253+0000 7f4d8ea00640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 0x7f4d7006d0f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.256+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4d90154630 con 0x7f4d9006f3f0 2024-09-13T17:08:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.436+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4d90154630 con 0x7f4d9006f3f0 2024-09-13T17:08:09.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.437+0000 7f4d87400640 1 -- 172.21.15.28:0/3337432689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4d90154630 con 0x7f4d9006f3f0 2024-09-13T17:08:09.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:08:09.439 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 msgr2=0x7f4d7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 0x7f4d7006d0f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c034000 tx=0x7f4d7c02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 msgr2=0x7f4d901e09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000c400 tx=0x7f4d8000c8c0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 shutdown_connections 2024-09-13T17:08:09.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d7006ac30 0x7f4d7006d0f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d90159770 0x7f4d900ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d90070430 0x7f4d90148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 --2- 172.21.15.28:0/3337432689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d9006f3f0 0x7f4d901e09f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 >> 172.21.15.28:0/3337432689 conn(0x7f4d900b0270 msgr2=0x7f4d90159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 shutdown_connections 2024-09-13T17:08:09.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.439+0000 7f4d96000640 1 -- 172.21.15.28:0/3337432689 wait complete. 2024-09-13T17:08:09.456 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:08:09.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:08:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.615+0000 7f250dc00640 1 Processor -- start 2024-09-13T17:08:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.615+0000 7f250dc00640 1 -- start start 2024-09-13T17:08:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.616+0000 7f250dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f250815ebd0 0x7f25080cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.616+0000 7f250dc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2508144280 con 0x7f2508143ea0 2024-09-13T17:08:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.616+0000 7f250dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25080cdad0 con 0x7f25080cd6d0 2024-09-13T17:08:09.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.616+0000 7f250dc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25080b3060 con 0x7f250815ebd0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2507e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2508143ea0 0x7f25080b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42566/0 (socket says 172.21.15.28:42566) 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2507e00640 1 -- 172.21.15.28:0/1251618645 learned_addr learned my addr 172.21.15.28:0/1251618645 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2507400640 1 --2- 172.21.15.28:0/1251618645 >> v2:172.21.15.189:3300/0 conn(0x7f250815ebd0 0x7f25080cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196544546 0 0) 0x7f24f8001110 con 0x7f2508143ea0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24ec003520 con 0x7f2508143ea0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814848030 0 0) 0x7f25080cdad0 con 0x7f25080cd6d0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24f8001110 con 0x7f25080cd6d0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1018384005 0 0) 0x7f24ec003520 con 0x7f2508143ea0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 >> v2:172.21.15.189:3300/0 conn(0x7f250815ebd0 msgr2=0x7f25080cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 --2- 172.21.15.28:0/1251618645 >> v2:172.21.15.189:3300/0 conn(0x7f250815ebd0 0x7f25080cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 >> v1:172.21.15.189:6789/0 conn(0x7f25080cd6d0 legacy=0x7f25080b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25080b23e0 con 0x7f2508143ea0 2024-09-13T17:08:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f24f8001800 con 0x7f2508143ea0 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 >> v1:172.21.15.28:6789/0 conn(0x7f2508143ea0 legacy=0x7f25080b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.617+0000 7f2506a00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 0x7f24ec003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2506a00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0047c0 0x7f24ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2506a00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec005180 0x7f24ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080b3060 con 0x7f24ec003b00 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec00c0b0 con 0x7f24ec0047c0 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec00c270 con 0x7f24ec005180 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 0x7f24ec003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507400640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0047c0 0x7f24ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f250d200640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec005180 0x7f24ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 -- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec005180 msgr2=0x7f24ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec005180 0x7f24ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 -- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0047c0 msgr2=0x7f24ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0047c0 0x7f24ec004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f250d200640 1 -- 172.21.15.28:0/1251618645 reap_dead start 2024-09-13T17:08:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.618+0000 7f2507e00640 1 -- 172.21.15.28:0/1251618645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec003520 con 0x7f24ec003b00 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.619+0000 7f2507e00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 0x7f24ec003f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c160 tx=0x7f24ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31fb72591e334f4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.619+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8005ad0 con 0x7f24ec003b00 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f802d070 con 0x7f24ec003b00 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f2506a00640 1 -- 172.21.15.28:0/1251618645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f8007070 con 0x7f24ec003b00 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 -- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 msgr2=0x7f24ec003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 0x7f24ec003f00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f24f800c160 tx=0x7f24ec0057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 -- 172.21.15.28:0/1251618645 shutdown_connections 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 --2- 172.21.15.28:0/1251618645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec003b00 0x7f24ec003f00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 -- 172.21.15.28:0/1251618645 >> 172.21.15.28:0/1251618645 conn(0x7f2508069e80 msgr2=0x7f250806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 -- 172.21.15.28:0/1251618645 shutdown_connections 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.620+0000 7f250dc00640 1 -- 172.21.15.28:0/1251618645 wait complete. 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.622+0000 7f250dc00640 1 Processor -- start 2024-09-13T17:08:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.622+0000 7f250dc00640 1 -- start start 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec003b00 0x7f2508159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 0x7f250813b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250815e760 con 0x7f250815dda0 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25081f3a90 con 0x7f250813b1a0 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250815a310 con 0x7f24ec003b00 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f2507400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f2507400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59068/0 (socket says 172.21.15.28:59068) 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f2507400640 1 -- 172.21.15.28:0/2596044393 learned_addr learned my addr 172.21.15.28:0/2596044393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f250d200640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 0x7f250813b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.623+0000 7f2507e00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec003b00 0x7f2508159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 -- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec003b00 msgr2=0x7f2508159dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec003b00 0x7f2508159dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 -- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 msgr2=0x7f250813b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 0x7f250813b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 -- 172.21.15.28:0/2596044393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250813bed0 con 0x7f250815dda0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f250d200640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 0x7f250813b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f2507400640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f24f40118c0 tx=0x7f24f4011d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f4004240 con 0x7f250815dda0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f40043c0 con 0x7f250815dda0 2024-09-13T17:08:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24f400fa00 con 0x7f250815dda0 2024-09-13T17:08:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25081f3160 con 0x7f250815dda0 2024-09-13T17:08:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.624+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25081f3720 con 0x7f250815dda0 2024-09-13T17:08:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.625+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2508067a10 con 0x7f250815dda0 2024-09-13T17:08:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.626+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f24f400fb60 con 0x7f250815dda0 2024-09-13T17:08:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.627+0000 7f24ffe00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 0x7f24e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.627+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f24f4089770 con 0x7f250815dda0 2024-09-13T17:08:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.627+0000 7f2507e00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 0x7f24e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.628+0000 7f2507e00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 0x7f24e806d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24f8032000 tx=0x7f24f8007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.631+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f24f4054c00 con 0x7f250815dda0 2024-09-13T17:08:09.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:09.813+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f250815a940 con 0x7f250815dda0 2024-09-13T17:08:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.105+0000 7f24ffe00640 1 -- 172.21.15.28:0/2596044393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f24f40190b0 con 0x7f250815dda0 2024-09-13T17:08:10.107 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 msgr2=0x7f24e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 0x7f24e806d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f24f8032000 tx=0x7f24f8007000 comp rx=0 tx=0).stop 2024-09-13T17:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 msgr2=0x7f250815e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f24f40118c0 tx=0x7f24f4011d80 comp rx=0 tx=0).stop 2024-09-13T17:08:10.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 shutdown_connections 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24e806ac00 0x7f24e806d0c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f250813b1a0 0x7f250813b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f250815dda0 0x7f250815e220 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 --2- 172.21.15.28:0/2596044393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec003b00 0x7f2508159dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 >> 172.21.15.28:0/2596044393 conn(0x7f2508069e80 msgr2=0x7f25080b0610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 shutdown_connections 2024-09-13T17:08:10.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.108+0000 7f250dc00640 1 -- 172.21.15.28:0/2596044393 wait complete. 2024-09-13T17:08:10.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:08:10.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.288+0000 7f8b0be00640 1 Processor -- start 2024-09-13T17:08:10.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.288+0000 7f8b0be00640 1 -- start start 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8b0c159820 0x7f8b0c164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0c0c3850 con 0x7f8b0c159440 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0be00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b0c15a930 con 0x7f8b0c0b0b50 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b0c159d20 con 0x7f8b0c159820 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b0c159440 0x7f8b0c0b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42578/0 (socket says 172.21.15.28:42578) 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0aa00640 1 -- 172.21.15.28:0/1075825088 learned_addr learned my addr 172.21.15.28:0/1075825088 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:10.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0b400640 1 --2- 172.21.15.28:0/1075825088 >> v2:172.21.15.189:3300/0 conn(0x7f8b0c159820 0x7f8b0c164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0a000640 1 -- 172.21.15.28:0/1075825088 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229886811 0 0) 0x7f8b0c0c3850 con 0x7f8b0c159440 2024-09-13T17:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.290+0000 7f8b0a000640 1 -- 172.21.15.28:0/1075825088 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af0003520 con 0x7f8b0c159440 2024-09-13T17:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.291+0000 7f8b0b400640 1 -- 172.21.15.28:0/1075825088 >> v1:172.21.15.189:6789/0 conn(0x7f8b0c0b0b50 legacy=0x7f8b0c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.291+0000 7f8b0b400640 1 -- 172.21.15.28:0/1075825088 >> v1:172.21.15.28:6789/0 conn(0x7f8b0c159440 legacy=0x7f8b0c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.291+0000 7f8b0b400640 1 -- 172.21.15.28:0/1075825088 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c15a000 con 0x7f8b0c159820 2024-09-13T17:08:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.292+0000 7f8b0b400640 1 --2- 172.21.15.28:0/1075825088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c159820 0x7f8b0c164260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b0402a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=677d359dfca30c12 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.292+0000 7f8b0a000640 1 -- 172.21.15.28:0/1075825088 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04007740 con 0x7f8b0c159820 2024-09-13T17:08:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.292+0000 7f8b0a000640 1 -- 172.21.15.28:0/1075825088 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b040078a0 con 0x7f8b0c159820 2024-09-13T17:08:10.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.292+0000 7f8b0a000640 1 -- 172.21.15.28:0/1075825088 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04007c30 con 0x7f8b0c159820 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 -- 172.21.15.28:0/1075825088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c159820 msgr2=0x7f8b0c164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 --2- 172.21.15.28:0/1075825088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c159820 0x7f8b0c164260 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b0402a780 comp rx=0 tx=0).stop 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 -- 172.21.15.28:0/1075825088 shutdown_connections 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 --2- 172.21.15.28:0/1075825088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c159820 0x7f8b0c164260 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 -- 172.21.15.28:0/1075825088 >> 172.21.15.28:0/1075825088 conn(0x7f8b0c072e40 msgr2=0x7f8b0c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 -- 172.21.15.28:0/1075825088 shutdown_connections 2024-09-13T17:08:10.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.293+0000 7f8b0be00640 1 -- 172.21.15.28:0/1075825088 wait complete. 2024-09-13T17:08:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.295+0000 7f8b0be00640 1 Processor -- start 2024-09-13T17:08:10.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.295+0000 7f8b0be00640 1 -- start start 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c0b0b50 0x7f8b0c1e3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b0c159440 0x7f8b0c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0703b0 con 0x7f8b0c159820 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c0acb80 con 0x7f8b0c0b0b50 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0c06f730 con 0x7f8b0c159440 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59072/0 (socket says 172.21.15.28:59072) 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0b400640 1 -- 172.21.15.28:0/14880721 learned_addr learned my addr 172.21.15.28:0/14880721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b0aa00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c0b0b50 0x7f8b0c1e3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:10.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.296+0000 7f8b03e00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b0c159440 0x7f8b0c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 -- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b0c159440 msgr2=0x7f8b0c1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b0c159440 0x7f8b0c1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 -- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c0b0b50 msgr2=0x7f8b0c1e3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c0b0b50 0x7f8b0c1e3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 -- 172.21.15.28:0/14880721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04009a70 con 0x7f8b0c159820 2024-09-13T17:08:10.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0b400640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04032c00 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b04032d60 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0403c650 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b0c06fa10 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.297+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b0c06ffd0 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.298+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0c0b22e0 con 0x7f8b0c159820 2024-09-13T17:08:10.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.300+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8b04039050 con 0x7f8b0c159820 2024-09-13T17:08:10.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.300+0000 7f8b02a00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 0x7f8ae806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:10.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.301+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8b040aa490 con 0x7f8b0c159820 2024-09-13T17:08:10.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.301+0000 7f8b0aa00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 0x7f8ae806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:10.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.302+0000 7f8b0aa00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 0x7f8ae806d0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0011e0 tx=0x7f8afc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:10.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.304+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8b04075990 con 0x7f8b0c159820 2024-09-13T17:08:10.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:10.485+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8b0c0b17e0 con 0x7f8b0c159820 2024-09-13T17:08:11.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.116+0000 7f8b02a00640 1 -- 172.21.15.28:0/14880721 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8b040370d0 con 0x7f8b0c159820 2024-09-13T17:08:11.119 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:08:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.123+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 msgr2=0x7f8ae806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.123+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 0x7f8ae806d0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f8afc0011e0 tx=0x7f8afc007d30 comp rx=0 tx=0).stop 2024-09-13T17:08:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.123+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 msgr2=0x7f8b0c1531e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.123+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8b04009d90 tx=0x7f8b04004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 shutdown_connections 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8ae806ac00 0x7f8ae806d0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b0c159820 0x7f8b0c1531e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b0c159440 0x7f8b0c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 --2- 172.21.15.28:0/14880721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b0c0b0b50 0x7f8b0c1e3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.125+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 >> 172.21.15.28:0/14880721 conn(0x7f8b0c072e40 msgr2=0x7f8b0c15c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.126+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 shutdown_connections 2024-09-13T17:08:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.126+0000 7f8b0be00640 1 -- 172.21.15.28:0/14880721 wait complete. 2024-09-13T17:08:11.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:08:11.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.322+0000 7f425f200640 1 Processor -- start 2024-09-13T17:08:11.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.322+0000 7f425f200640 1 -- start start 2024-09-13T17:08:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f425f200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42600b3e20 0x7f4260070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f425f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4260071350 con 0x7f426006cbe0 2024-09-13T17:08:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f425f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426006feb0 con 0x7f42600cefd0 2024-09-13T17:08:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f425f200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4260070ac0 con 0x7f42600b3e20 2024-09-13T17:08:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f4255400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f426006cbe0 0x7f42600ce8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42594/0 (socket says 172.21.15.28:42594) 2024-09-13T17:08:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f4255400640 1 -- 172.21.15.28:0/1861540195 learned_addr learned my addr 172.21.15.28:0/1861540195 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.323+0000 7f425de00640 1 --2- 172.21.15.28:0/1861540195 >> v2:172.21.15.189:3300/0 conn(0x7f42600b3e20 0x7f4260070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519644113 0 0) 0x7f4260071350 con 0x7f426006cbe0 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244003520 con 0x7f426006cbe0 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425de00640 1 -- 172.21.15.28:0/1861540195 >> v1:172.21.15.189:6789/0 conn(0x7f42600cefd0 legacy=0x7f426015d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126470104 0 0) 0x7f426006feb0 con 0x7f42600cefd0 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425de00640 1 -- 172.21.15.28:0/1861540195 >> v1:172.21.15.28:6789/0 conn(0x7f426006cbe0 legacy=0x7f42600ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.324+0000 7f425de00640 1 -- 172.21.15.28:0/1861540195 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260070190 con 0x7f42600b3e20 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.325+0000 7f425de00640 1 --2- 172.21.15.28:0/1861540195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f4260070e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42500097a0 tx=0x7f425002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62a965deb419fb43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.325+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250032440 con 0x7f42600b3e20 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.325+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42500325a0 con 0x7f42600b3e20 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.325+0000 7f425d400640 1 -- 172.21.15.28:0/1861540195 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250032930 con 0x7f42600b3e20 2024-09-13T17:08:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 -- 172.21.15.28:0/1861540195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 msgr2=0x7f4260070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 --2- 172.21.15.28:0/1861540195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f4260070e10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f42500097a0 tx=0x7f425002a230 comp rx=0 tx=0).stop 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 -- 172.21.15.28:0/1861540195 shutdown_connections 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 --2- 172.21.15.28:0/1861540195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f4260070e10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 -- 172.21.15.28:0/1861540195 >> 172.21.15.28:0/1861540195 conn(0x7f4260069e80 msgr2=0x7f426006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 -- 172.21.15.28:0/1861540195 shutdown_connections 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.326+0000 7f425f200640 1 -- 172.21.15.28:0/1861540195 wait complete. 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.328+0000 7f425f200640 1 Processor -- start 2024-09-13T17:08:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.328+0000 7f425f200640 1 -- start start 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f426009bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 0x7f42601562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601e76c0 con 0x7f426006cbe0 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42601e79a0 con 0x7f42600b3e20 2024-09-13T17:08:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42601e7c80 con 0x7f42600cefd0 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59088/0 (socket says 172.21.15.28:59088) 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 -- 172.21.15.28:0/619222725 learned_addr learned my addr 172.21.15.28:0/619222725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f4255400640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f426009bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425e800640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 0x7f42601562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 -- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 msgr2=0x7f42601562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 0x7f42601562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 -- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 msgr2=0x7f426009bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f426009bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425de00640 1 -- 172.21.15.28:0/619222725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4250009400 con 0x7f426006cbe0 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.329+0000 7f425e800640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 0x7f42601562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f425de00640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4250035040 tx=0x7f42500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250032440 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250030b00 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250030e90 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42601e7f60 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.330+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426009c1a0 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.331+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426009c390 con 0x7f426006cbe0 2024-09-13T17:08:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.332+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4250039040 con 0x7f426006cbe0 2024-09-13T17:08:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.333+0000 7f4256a00640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 0x7f423c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.333+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f42500a9e60 con 0x7f426006cbe0 2024-09-13T17:08:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.333+0000 7f4255400640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 0x7f423c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.334+0000 7f4255400640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 0x7f423c06d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f426015f7f0 tx=0x7f4248009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.337+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f42500750b0 con 0x7f426006cbe0 2024-09-13T17:08:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.553+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f42601e8ca0 con 0x7f426006cbe0 2024-09-13T17:08:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.560+0000 7f4256a00640 1 -- 172.21.15.28:0/619222725 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f4250006e10 con 0x7f426006cbe0 2024-09-13T17:08:11.562 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.564+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 msgr2=0x7f423c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.564+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 0x7f423c06d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f426015f7f0 tx=0x7f4248009040 comp rx=0 tx=0).stop 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.564+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 msgr2=0x7f42601e8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.564+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4250035040 tx=0x7f42500046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 shutdown_connections 2024-09-13T17:08:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f423c06ac00 0x7f423c06d0c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42600cefd0 0x7f42601562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42600b3e20 0x7f426009bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 --2- 172.21.15.28:0/619222725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f426006cbe0 0x7f42601e8760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 >> 172.21.15.28:0/619222725 conn(0x7f4260069e80 msgr2=0x7f426006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.566+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 shutdown_connections 2024-09-13T17:08:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.568+0000 7f425f200640 1 -- 172.21.15.28:0/619222725 wait complete. 2024-09-13T17:08:11.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.751+0000 7f589fe00640 1 Processor -- start 2024-09-13T17:08:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.752+0000 7f589fe00640 1 -- start start 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 0x7f58a00d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58a00b48d0 0x7f58a00b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a00b5870 con 0x7f58a00cd710 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a00d2560 con 0x7f58a00b48d0 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a015cc10 con 0x7f58a015c830 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f5896000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f58a00cd710 0x7f58a00b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42598/0 (socket says 172.21.15.28:42598) 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f5896000640 1 -- 172.21.15.28:0/2438002040 learned_addr learned my addr 172.21.15.28:0/2438002040 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589ea00640 1 --2- 172.21.15.28:0/2438002040 >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 0x7f58a00d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589f400640 1 --2- 172.21.15.28:0/2438002040 >> v2:172.21.15.28:3300/0 conn(0x7f58a00b48d0 0x7f58a00b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.753+0000 7f589e000640 1 -- 172.21.15.28:0/2438002040 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955818854 0 0) 0x7f58a00b5870 con 0x7f58a00cd710 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589e000640 1 -- 172.21.15.28:0/2438002040 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5884003520 con 0x7f58a00cd710 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589f400640 1 -- 172.21.15.28:0/2438002040 >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 msgr2=0x7f58a00d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589f400640 1 --2- 172.21.15.28:0/2438002040 >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 0x7f58a00d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589f400640 1 -- 172.21.15.28:0/2438002040 >> v1:172.21.15.28:6789/0 conn(0x7f58a00cd710 legacy=0x7f58a00b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589f400640 1 -- 172.21.15.28:0/2438002040 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a00d19f0 con 0x7f58a00b48d0 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.754+0000 7f589ea00640 1 --2- 172.21.15.28:0/2438002040 >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 0x7f58a00d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589f400640 1 --2- 172.21.15.28:0/2438002040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00b48d0 0x7f58a00b5fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58a00b4df0 tx=0x7f589802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315fb3bc3ea41fef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589e000640 1 -- 172.21.15.28:0/2438002040 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589802aea0 con 0x7f58a00b48d0 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589e000640 1 -- 172.21.15.28:0/2438002040 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589802d050 con 0x7f58a00b48d0 2024-09-13T17:08:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589e000640 1 -- 172.21.15.28:0/2438002040 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898007940 con 0x7f58a00b48d0 2024-09-13T17:08:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589fe00640 1 -- 172.21.15.28:0/2438002040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00b48d0 msgr2=0x7f58a00b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.755+0000 7f589fe00640 1 --2- 172.21.15.28:0/2438002040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00b48d0 0x7f58a00b5fc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f58a00b4df0 tx=0x7f589802a680 comp rx=0 tx=0).stop 2024-09-13T17:08:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 -- 172.21.15.28:0/2438002040 shutdown_connections 2024-09-13T17:08:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 --2- 172.21.15.28:0/2438002040 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00b48d0 0x7f58a00b5fc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 --2- 172.21.15.28:0/2438002040 >> v2:172.21.15.189:3300/0 conn(0x7f58a015c830 0x7f58a00d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 -- 172.21.15.28:0/2438002040 >> 172.21.15.28:0/2438002040 conn(0x7f58a00cf3a0 msgr2=0x7f58a00cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 -- 172.21.15.28:0/2438002040 shutdown_connections 2024-09-13T17:08:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.756+0000 7f589fe00640 1 -- 172.21.15.28:0/2438002040 wait complete. 2024-09-13T17:08:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.757+0000 7f589fe00640 1 Processor -- start 2024-09-13T17:08:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.757+0000 7f589fe00640 1 -- start start 2024-09-13T17:08:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58a00b48d0 0x7f58a00cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f58a015c830 0x7f58a015adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01e8040 con 0x7f58a00cd710 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01e85c0 con 0x7f58a00b48d0 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f589fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a01e82e0 con 0x7f58a015c830 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f5896000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.758+0000 7f5896000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59118/0 (socket says 172.21.15.28:59118) 2024-09-13T17:08:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 -- 172.21.15.28:0/3558618841 learned_addr learned my addr 172.21.15.28:0/3558618841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f589f400640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f58a015c830 0x7f58a015adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f589ea00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58a00b48d0 0x7f58a00cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 -- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f58a015c830 msgr2=0x7f58a015adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f58a015c830 0x7f58a015adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 -- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58a00b48d0 msgr2=0x7f58a00cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58a00b48d0 0x7f58a00cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 -- 172.21.15.28:0/3558618841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5890009400 con 0x7f58a00cd710 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5896000640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f588800c700 tx=0x7f588800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888007270 con 0x7f58a00cd710 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58880073f0 con 0x7f58a00cd710 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5888001d40 con 0x7f58a00cd710 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.759+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5898009a70 con 0x7f58a00cd710 2024-09-13T17:08:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.760+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a00caab0 con 0x7f58a00cd710 2024-09-13T17:08:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.761+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a00d14b0 con 0x7f58a00cd710 2024-09-13T17:08:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.762+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f58880057a0 con 0x7f58a00cd710 2024-09-13T17:08:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.763+0000 7f5897e00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 0x7f587c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:11.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.763+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5888088920 con 0x7f58a00cd710 2024-09-13T17:08:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.763+0000 7f589ea00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 0x7f587c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:11.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.764+0000 7f589ea00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 0x7f587c06d0c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5890009bc0 tx=0x7f5890008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:11.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.767+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5888053b00 con 0x7f58a00cd710 2024-09-13T17:08:11.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.948+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58a015c3a0 con 0x7f58a00cd710 2024-09-13T17:08:11.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.950+0000 7f5897e00640 1 -- 172.21.15.28:0/3558618841 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f5888017070 con 0x7f58a00cd710 2024-09-13T17:08:11.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:11.952 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:11.555100+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 msgr2=0x7f587c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 0x7f587c06d0c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5890009bc0 tx=0x7f5890008040 comp rx=0 tx=0).stop 2024-09-13T17:08:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 msgr2=0x7f58a01e7b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:11.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f588800c700 tx=0x7f588800cbc0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 shutdown_connections 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f587c06ac00 0x7f587c06d0c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f58a015c830 0x7f58a015adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58a00cd710 0x7f58a01e7b00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 --2- 172.21.15.28:0/3558618841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58a00b48d0 0x7f58a00cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 >> 172.21.15.28:0/3558618841 conn(0x7f58a00cf3a0 msgr2=0x7f58a00b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 shutdown_connections 2024-09-13T17:08:11.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:11.952+0000 7f589fe00640 1 -- 172.21.15.28:0/3558618841 wait complete. 2024-09-13T17:08:11.970 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.122+0000 7f88f2600640 1 Processor -- start 2024-09-13T17:08:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.122+0000 7f88f2600640 1 -- start start 2024-09-13T17:08:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88f2600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f88ec06c9e0 0x7f88ec15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88f2600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec159d90 con 0x7f88ec1592a0 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88f2600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec0b2040 con 0x7f88ec06f970 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88f2600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec15b0b0 con 0x7f88ec06c9e0 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f88ec1592a0 0x7f88ec159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42600/0 (socket says 172.21.15.28:42600) 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eb400640 1 -- 172.21.15.28:0/3785381289 learned_addr learned my addr 172.21.15.28:0/3785381289 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88ebe00640 1 --2- 172.21.15.28:0/3785381289 >> v2:172.21.15.189:3300/0 conn(0x7f88ec06c9e0 0x7f88ec15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209337521 0 0) 0x7f88ec159d90 con 0x7f88ec1592a0 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0003520 con 0x7f88ec1592a0 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478145499 0 0) 0x7f88ec0b2040 con 0x7f88ec06f970 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ec159d90 con 0x7f88ec06f970 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88ebe00640 1 -- 172.21.15.28:0/3785381289 >> v1:172.21.15.189:6789/0 conn(0x7f88ec06f970 legacy=0x7f88ec06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88ebe00640 1 -- 172.21.15.28:0/3785381289 >> v1:172.21.15.28:6789/0 conn(0x7f88ec1592a0 legacy=0x7f88ec159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.124+0000 7f88ebe00640 1 -- 172.21.15.28:0/3785381289 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec159ff0 con 0x7f88ec06c9e0 2024-09-13T17:08:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88ebe00640 1 --2- 172.21.15.28:0/3785381289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec06c9e0 0x7f88ec15ab70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009d90 tx=0x7f88e402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f42e88ba66962e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4007740 con 0x7f88ec06c9e0 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e402d050 con 0x7f88ec06c9e0 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88eaa00640 1 -- 172.21.15.28:0/3785381289 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e4007ac0 con 0x7f88ec06c9e0 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88f2600640 1 -- 172.21.15.28:0/3785381289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec06c9e0 msgr2=0x7f88ec15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.126+0000 7f88f2600640 1 --2- 172.21.15.28:0/3785381289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec06c9e0 0x7f88ec15ab70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009d90 tx=0x7f88e402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.127+0000 7f88f2600640 1 -- 172.21.15.28:0/3785381289 shutdown_connections 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.127+0000 7f88f2600640 1 --2- 172.21.15.28:0/3785381289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec06c9e0 0x7f88ec15ab70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.127+0000 7f88f2600640 1 -- 172.21.15.28:0/3785381289 >> 172.21.15.28:0/3785381289 conn(0x7f88ec069e80 msgr2=0x7f88ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.127+0000 7f88f2600640 1 -- 172.21.15.28:0/3785381289 shutdown_connections 2024-09-13T17:08:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.127+0000 7f88f2600640 1 -- 172.21.15.28:0/3785381289 wait complete. 2024-09-13T17:08:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.128+0000 7f88f2600640 1 Processor -- start 2024-09-13T17:08:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.128+0000 7f88f2600640 1 -- start start 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88ec06c9e0 0x7f88ec14c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec1592a0 0x7f88ec1f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec14cf80 con 0x7f88ec06f970 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec14c6e0 con 0x7f88ec1592a0 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.129+0000 7f88f2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec14c9c0 con 0x7f88ec06c9e0 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59124/0 (socket says 172.21.15.28:59124) 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 -- 172.21.15.28:0/513418719 learned_addr learned my addr 172.21.15.28:0/513418719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:12.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88ebe00640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec1592a0 0x7f88ec1f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88eb400640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88ec06c9e0 0x7f88ec14c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 -- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88ec06c9e0 msgr2=0x7f88ec14c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88ec06c9e0 0x7f88ec14c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 -- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec1592a0 msgr2=0x7f88ec1f2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec1592a0 0x7f88ec1f2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 -- 172.21.15.28:0/513418719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009a70 con 0x7f88ec06f970 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3e00640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c490 tx=0x7f88d400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.130+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d40052d0 con 0x7f88ec06f970 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.131+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88d4005430 con 0x7f88ec06f970 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.131+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88d4012870 con 0x7f88ec06f970 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.131+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec14cca0 con 0x7f88ec06f970 2024-09-13T17:08:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.131+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec155c30 con 0x7f88ec06f970 2024-09-13T17:08:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.132+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec15ab70 con 0x7f88ec06f970 2024-09-13T17:08:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.133+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f88d40129d0 con 0x7f88ec06f970 2024-09-13T17:08:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.133+0000 7f88e3400640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 0x7f88c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.133+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88d4088e40 con 0x7f88ec06f970 2024-09-13T17:08:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.134+0000 7f88eb400640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 0x7f88c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.135+0000 7f88eb400640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 0x7f88c806d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f88dc005880 tx=0x7f88dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.137+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f88d40540c0 con 0x7f88ec06f970 2024-09-13T17:08:12.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.316+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88ec14bee0 con 0x7f88ec06f970 2024-09-13T17:08:12.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.317+0000 7f88e3400640 1 -- 172.21.15.28:0/513418719 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16518 (secure 0 0 0) 0x7f88d40576e0 con 0x7f88ec06f970 2024-09-13T17:08:12.319 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:12.319 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":19,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:11.555100+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 msgr2=0x7f88c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 0x7f88c806d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f88dc005880 tx=0x7f88dc009040 comp rx=0 tx=0).stop 2024-09-13T17:08:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 msgr2=0x7f88ec1f2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f88d400c490 tx=0x7f88d400c950 comp rx=0 tx=0).stop 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 shutdown_connections 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88c806ac00 0x7f88c806d0c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f88ec1592a0 0x7f88ec1f2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88ec06f970 0x7f88ec1f2d80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 --2- 172.21.15.28:0/513418719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88ec06c9e0 0x7f88ec14c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 >> 172.21.15.28:0/513418719 conn(0x7f88ec069e80 msgr2=0x7f88ec156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 shutdown_connections 2024-09-13T17:08:12.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.319+0000 7f88f2600640 1 -- 172.21.15.28:0/513418719 wait complete. 2024-09-13T17:08:12.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:08:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.492+0000 7fd5fec00640 1 Processor -- start 2024-09-13T17:08:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.492+0000 7fd5fec00640 1 -- start start 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 0x7fd5f80c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f815ef70 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f80c4320 con 0x7fd5f8143f70 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f815e1b0 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fe200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd5f80c3ba0 0x7fd5f8070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42614/0 (socket says 172.21.15.28:42614) 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fe200640 1 -- 172.21.15.28:0/3415877852 learned_addr learned my addr 172.21.15.28:0/3415877852 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:12.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.494+0000 7fd5fce00640 1 --2- 172.21.15.28:0/3415877852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 0x7fd5f80c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783245483 0 0) 0x7fd5f815ef70 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0003520 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580722758 0 0) 0x7fd5f80c4320 con 0x7fd5f8143f70 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5fce00640 1 -- 172.21.15.28:0/3415877852 >> v1:172.21.15.189:6789/0 conn(0x7fd5f8143f70 legacy=0x7fd5f80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5fce00640 1 -- 172.21.15.28:0/3415877852 >> v1:172.21.15.28:6789/0 conn(0x7fd5f80c3ba0 legacy=0x7fd5f8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.495+0000 7fd5fce00640 1 -- 172.21.15.28:0/3415877852 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f815e490 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5fce00640 1 --2- 172.21.15.28:0/3415877852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 0x7fd5f80c3660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40097d0 tx=0x7fd5e402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ce6c1cd0fe1a435 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4032440 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e40325a0 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5f7e00640 1 -- 172.21.15.28:0/3415877852 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e4032930 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5fec00640 1 -- 172.21.15.28:0/3415877852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 msgr2=0x7fd5f80c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.496+0000 7fd5fec00640 1 --2- 172.21.15.28:0/3415877852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 0x7fd5f80c3660 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd5e40097d0 tx=0x7fd5e402a210 comp rx=0 tx=0).stop 2024-09-13T17:08:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.497+0000 7fd5fec00640 1 -- 172.21.15.28:0/3415877852 shutdown_connections 2024-09-13T17:08:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.497+0000 7fd5fec00640 1 --2- 172.21.15.28:0/3415877852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f80cd7c0 0x7fd5f80c3660 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.497+0000 7fd5fec00640 1 -- 172.21.15.28:0/3415877852 >> 172.21.15.28:0/3415877852 conn(0x7fd5f8069e80 msgr2=0x7fd5f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.497+0000 7fd5fec00640 1 -- 172.21.15.28:0/3415877852 shutdown_connections 2024-09-13T17:08:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.497+0000 7fd5fec00640 1 -- 172.21.15.28:0/3415877852 wait complete. 2024-09-13T17:08:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.498+0000 7fd5fec00640 1 Processor -- start 2024-09-13T17:08:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.498+0000 7fd5fec00640 1 -- start start 2024-09-13T17:08:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 0x7fd5f814cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 0x7fd5f814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f814e3d0 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f814faa0 con 0x7fd5f80cd7c0 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.499+0000 7fd5fec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f81f56f0 con 0x7fd5f8143f70 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59128/0 (socket says 172.21.15.28:59128) 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 -- 172.21.15.28:0/1396080786 learned_addr learned my addr 172.21.15.28:0/1396080786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fce00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 0x7fd5f814cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fe200640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 0x7fd5f814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 -- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 msgr2=0x7fd5f814d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 0x7fd5f814d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 -- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 msgr2=0x7fd5f814cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 0x7fd5f814cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 -- 172.21.15.28:0/1396080786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e4009400 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fce00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 0x7fd5f814cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fe200640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 0x7fd5f814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5fd800640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c410 tx=0x7fd5ec00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.500+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec0035f0 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.501+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ec003770 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.501+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f814d750 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.501+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec003b20 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.501+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f814dd10 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.502+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f814e510 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.503+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd5ec003c80 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.503+0000 7fd5f5600640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 0x7fd5d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:12.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.504+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5ec0889b0 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.504+0000 7fd5fce00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 0x7fd5d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:12.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.505+0000 7fd5fce00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 0x7fd5d806d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4034000 tx=0x7fd5e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:12.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.508+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd5ec053b80 con 0x7fd5f80c3ba0 2024-09-13T17:08:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:12.722+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd5f80c3660 con 0x7fd5f80c3ba0 2024-09-13T17:08:13.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.569+0000 7fd5f5600640 1 -- 172.21.15.28:0/1396080786 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd5ec017020 con 0x7fd5f80c3ba0 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.572+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 msgr2=0x7fd5d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.572+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 0x7fd5d806d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4034000 tx=0x7fd5e402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.572+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 msgr2=0x7fd5f81f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec00c410 tx=0x7fd5ec00c8d0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 shutdown_connections 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd5d806ac00 0x7fd5d806d0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5f8143f70 0x7fd5f814d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd5f80cd7c0 0x7fd5f814cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 --2- 172.21.15.28:0/1396080786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5f80c3ba0 0x7fd5f81f51b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 >> 172.21.15.28:0/1396080786 conn(0x7fd5f8069e80 msgr2=0x7fd5f8070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 shutdown_connections 2024-09-13T17:08:13.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.573+0000 7fd5fec00640 1 -- 172.21.15.28:0/1396080786 wait complete. 2024-09-13T17:08:13.592 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:08:13.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:08:13.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.754+0000 7f4c4d600640 1 Processor -- start 2024-09-13T17:08:13.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.754+0000 7f4c4d600640 1 -- start start 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c4d600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c480ae580 0x7f4c480b3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c4d600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c4815dbf0 0x7f4c480b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c4d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c480b4350 con 0x7f4c480cb6e0 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c4d600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4815cf00 con 0x7f4c480ae580 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c4d600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c480b4070 con 0x7f4c4815dbf0 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c46a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c480ae580 0x7f4c480b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c47400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4c480cb6e0 0x7f4c481604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42618/0 (socket says 172.21.15.28:42618) 2024-09-13T17:08:13.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.755+0000 7f4c47400640 1 -- 172.21.15.28:0/3126833003 learned_addr learned my addr 172.21.15.28:0/3126833003 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c3e000640 1 --2- 172.21.15.28:0/3126833003 >> v2:172.21.15.189:3300/0 conn(0x7f4c4815dbf0 0x7f4c480b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c46a00640 1 -- 172.21.15.28:0/3126833003 >> v2:172.21.15.189:3300/0 conn(0x7f4c4815dbf0 msgr2=0x7f4c480b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c46a00640 1 --2- 172.21.15.28:0/3126833003 >> v2:172.21.15.189:3300/0 conn(0x7f4c4815dbf0 0x7f4c480b3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c46a00640 1 -- 172.21.15.28:0/3126833003 >> v1:172.21.15.28:6789/0 conn(0x7f4c480cb6e0 legacy=0x7f4c481604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c46a00640 1 -- 172.21.15.28:0/3126833003 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4815d110 con 0x7f4c480ae580 2024-09-13T17:08:13.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.756+0000 7f4c46000640 1 -- 172.21.15.28:0/3126833003 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230474073 0 0) 0x7f4c40001490 con 0x7f4c480cb6e0 2024-09-13T17:08:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.757+0000 7f4c46a00640 1 --2- 172.21.15.28:0/3126833003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480ae580 0x7f4c480b3540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c4806e840 tx=0x7f4c3802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76b710b1feaab39 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.757+0000 7f4c46000640 1 -- 172.21.15.28:0/3126833003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3802ae60 con 0x7f4c480ae580 2024-09-13T17:08:13.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.757+0000 7f4c46000640 1 -- 172.21.15.28:0/3126833003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c38032440 con 0x7f4c480ae580 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.757+0000 7f4c46000640 1 -- 172.21.15.28:0/3126833003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c380327d0 con 0x7f4c480ae580 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 -- 172.21.15.28:0/3126833003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480ae580 msgr2=0x7f4c480b3540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 --2- 172.21.15.28:0/3126833003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480ae580 0x7f4c480b3540 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4c4806e840 tx=0x7f4c3802a110 comp rx=0 tx=0).stop 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 -- 172.21.15.28:0/3126833003 shutdown_connections 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 --2- 172.21.15.28:0/3126833003 >> v2:172.21.15.189:3300/0 conn(0x7f4c4815dbf0 0x7f4c480b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 --2- 172.21.15.28:0/3126833003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480ae580 0x7f4c480b3540 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 -- 172.21.15.28:0/3126833003 >> 172.21.15.28:0/3126833003 conn(0x7f4c480b0af0 msgr2=0x7f4c480b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 -- 172.21.15.28:0/3126833003 shutdown_connections 2024-09-13T17:08:13.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.758+0000 7f4c4d600640 1 -- 172.21.15.28:0/3126833003 wait complete. 2024-09-13T17:08:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.760+0000 7f4c4d600640 1 Processor -- start 2024-09-13T17:08:13.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.760+0000 7f4c4d600640 1 -- start start 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 0x7f4c48148790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c4815dbf0 0x7f4c4806cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c481494a0 con 0x7f4c480cb6e0 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c48149740 con 0x7f4c4815dbf0 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c4d600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c48149a20 con 0x7f4c480ae580 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59144/0 (socket says 172.21.15.28:59144) 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 -- 172.21.15.28:0/2704122836 learned_addr learned my addr 172.21.15.28:0/2704122836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c46a00640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 0x7f4c48148790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c47400640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c4815dbf0 0x7f4c4806cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 -- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 msgr2=0x7f4c48148790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 0x7f4c48148790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 -- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c4815dbf0 msgr2=0x7f4c4806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c4815dbf0 0x7f4c4806cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 -- 172.21.15.28:0/2704122836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38009400 con 0x7f4c480cb6e0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c46a00640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 0x7f4c48148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.761+0000 7f4c3e000640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e840 tx=0x7f4c3000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.762+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c300094f0 con 0x7f4c480cb6e0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.762+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3000c940 con 0x7f4c480cb6e0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.762+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c48149d00 con 0x7f4c480cb6e0 2024-09-13T17:08:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.762+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3000ccd0 con 0x7f4c480cb6e0 2024-09-13T17:08:13.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.762+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c48148cd0 con 0x7f4c480cb6e0 2024-09-13T17:08:13.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.763+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c480cbb10 con 0x7f4c480cb6e0 2024-09-13T17:08:13.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.764+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4c30013470 con 0x7f4c480cb6e0 2024-09-13T17:08:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.765+0000 7f4c3f400640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 0x7f4c2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.765+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c30089420 con 0x7f4c480cb6e0 2024-09-13T17:08:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.766+0000 7f4c46a00640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 0x7f4c2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.766+0000 7f4c46a00640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 0x7f4c2406d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4c38034000 tx=0x7f4c3802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.770+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4c30054610 con 0x7f4c480cb6e0 2024-09-13T17:08:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:13.987+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4c48071cc0 con 0x7f4c480cb6e0 2024-09-13T17:08:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.572+0000 7f4c3f400640 1 -- 172.21.15.28:0/2704122836 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c30057c30 con 0x7f4c480cb6e0 2024-09-13T17:08:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 msgr2=0x7f4c2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 0x7f4c2406d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4c38034000 tx=0x7f4c3802e000 comp rx=0 tx=0).stop 2024-09-13T17:08:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 msgr2=0x7f4c4815ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c3000e840 tx=0x7f4c3000ed00 comp rx=0 tx=0).stop 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 shutdown_connections 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c2406ac00 0x7f4c2406d0c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c4815dbf0 0x7f4c4806cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c480cb6e0 0x7f4c4815ac20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 --2- 172.21.15.28:0/2704122836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c480ae580 0x7f4c48148790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 >> 172.21.15.28:0/2704122836 conn(0x7f4c480b0af0 msgr2=0x7f4c4815e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 shutdown_connections 2024-09-13T17:08:14.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.574+0000 7f4c4d600640 1 -- 172.21.15.28:0/2704122836 wait complete. 2024-09-13T17:08:14.593 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:08:14.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:14.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.748+0000 7f025cc00640 1 Processor -- start 2024-09-13T17:08:14.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.749+0000 7f025cc00640 1 -- start start 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f025cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 0x7f0258160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f025cc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580c4af0 con 0x7f02580cb760 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f025cc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02580cd460 con 0x7f02580c4380 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f025cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02580cd740 con 0x7f025806b5a0 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f0257400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f02580cb760 0x7f025806ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34392/0 (socket says 172.21.15.28:34392) 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f0257400640 1 -- 172.21.15.28:0/1764966175 learned_addr learned my addr 172.21.15.28:0/1764966175 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:14.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.750+0000 7f024ea00640 1 --2- 172.21.15.28:0/1764966175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 0x7f0258160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224946678 0 0) 0x7f02580c4af0 con 0x7f02580cb760 2024-09-13T17:08:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f023c003520 con 0x7f02580cb760 2024-09-13T17:08:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f024ea00640 1 -- 172.21.15.28:0/1764966175 >> v1:172.21.15.189:6789/0 conn(0x7f02580c4380 legacy=0x7f02580ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338441253 0 0) 0x7f02580cd460 con 0x7f02580c4380 2024-09-13T17:08:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f024ea00640 1 -- 172.21.15.28:0/1764966175 >> v1:172.21.15.28:6789/0 conn(0x7f02580cb760 legacy=0x7f025806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.751+0000 7f024ea00640 1 -- 172.21.15.28:0/1764966175 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580cda20 con 0x7f025806b5a0 2024-09-13T17:08:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.752+0000 7f024ea00640 1 --2- 172.21.15.28:0/1764966175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 0x7f0258160f10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02400097d0 tx=0x7f024002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c7c7b0b6ef3d94a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.752+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240032440 con 0x7f025806b5a0 2024-09-13T17:08:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.752+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02400325a0 con 0x7f025806b5a0 2024-09-13T17:08:14.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.752+0000 7f0256a00640 1 -- 172.21.15.28:0/1764966175 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0240032930 con 0x7f025806b5a0 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 -- 172.21.15.28:0/1764966175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 msgr2=0x7f0258160f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 --2- 172.21.15.28:0/1764966175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 0x7f0258160f10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f02400097d0 tx=0x7f024002a180 comp rx=0 tx=0).stop 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 -- 172.21.15.28:0/1764966175 shutdown_connections 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 --2- 172.21.15.28:0/1764966175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f025806b5a0 0x7f0258160f10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 -- 172.21.15.28:0/1764966175 >> 172.21.15.28:0/1764966175 conn(0x7f02580703a0 msgr2=0x7f02580707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 -- 172.21.15.28:0/1764966175 shutdown_connections 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.753+0000 7f025cc00640 1 -- 172.21.15.28:0/1764966175 wait complete. 2024-09-13T17:08:14.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.754+0000 7f025cc00640 1 Processor -- start 2024-09-13T17:08:14.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 -- start start 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025806b5a0 0x7f02580ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02580c4380 0x7f025815f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580ad160 con 0x7f02580cb760 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.755+0000 7f025cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02580af200 con 0x7f025806b5a0 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f025cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02580ac850 con 0x7f02580c4380 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025806b5a0 0x7f02580ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f024ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02580c4380 0x7f025815f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33254/0 (socket says 172.21.15.28:33254) 2024-09-13T17:08:14.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 -- 172.21.15.28:0/123253610 learned_addr learned my addr 172.21.15.28:0/123253610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 -- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02580c4380 msgr2=0x7f025815f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02580c4380 0x7f025815f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 -- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025806b5a0 msgr2=0x7f02580ad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025806b5a0 0x7f02580ad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.756+0000 7f0257e00640 1 -- 172.21.15.28:0/123253610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0240009400 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f0257e00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0250001560 tx=0x7f025000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250003da0 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0250005070 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0250005420 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02580aef20 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.757+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02580acd80 con 0x7f02580cb760 2024-09-13T17:08:14.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.758+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02580ad2a0 con 0x7f02580cb760 2024-09-13T17:08:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.759+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0250005580 con 0x7f02580cb760 2024-09-13T17:08:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.760+0000 7f024fe00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 0x7f023406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:14.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.760+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0250088ba0 con 0x7f02580cb760 2024-09-13T17:08:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.760+0000 7f0257400640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 0x7f023406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:14.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.761+0000 7f0257400640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 0x7f023406d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0248005880 tx=0x7f0248009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:14.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.764+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f02580ad2a0 con 0x7f02580cb760 2024-09-13T17:08:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.981+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02580ad2a0 con 0x7f02580cb760 2024-09-13T17:08:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.982+0000 7f024fe00640 1 -- 172.21.15.28:0/123253610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f02580ad2a0 con 0x7f02580cb760 2024-09-13T17:08:14.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:14.984 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.984+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 msgr2=0x7f023406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.984+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 0x7f023406d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f0248005880 tx=0x7f0248009040 comp rx=0 tx=0).stop 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.984+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 msgr2=0x7f02580ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.984+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0250001560 tx=0x7f025000cc70 comp rx=0 tx=0).stop 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 shutdown_connections 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f023406ac00 0x7f023406d0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02580cb760 0x7f02580ae620 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02580c4380 0x7f025815f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 --2- 172.21.15.28:0/123253610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f025806b5a0 0x7f02580ad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 >> 172.21.15.28:0/123253610 conn(0x7f02580703a0 msgr2=0x7f02580c9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 shutdown_connections 2024-09-13T17:08:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:14.985+0000 7f025cc00640 1 -- 172.21.15.28:0/123253610 wait complete. 2024-09-13T17:08:15.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.156+0000 7fe476e00640 1 Processor -- start 2024-09-13T17:08:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.156+0000 7fe476e00640 1 -- start start 2024-09-13T17:08:15.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.157+0000 7fe476e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 0x7fe4701602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.157+0000 7fe476e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe47015d9e0 0x7fe47006bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.157+0000 7fe476e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47006b8c0 con 0x7fe4700ac850 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.157+0000 7fe476e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe470161340 con 0x7fe47015d9e0 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.157+0000 7fe476e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe470160820 con 0x7fe47015c5f0 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe475000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe4700ac850 0x7fe4700b1ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34400/0 (socket says 172.21.15.28:34400) 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe475000640 1 -- 172.21.15.28:0/3133216451 learned_addr learned my addr 172.21.15.28:0/3133216451 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe476400640 1 --2- 172.21.15.28:0/3133216451 >> v2:172.21.15.28:3300/0 conn(0x7fe47015d9e0 0x7fe47006bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe475a00640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 0x7fe4701602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe46fe00640 1 -- 172.21.15.28:0/3133216451 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414131338 0 0) 0x7fe47006b8c0 con 0x7fe4700ac850 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe46fe00640 1 -- 172.21.15.28:0/3133216451 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe458003520 con 0x7fe4700ac850 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe476400640 1 -- 172.21.15.28:0/3133216451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 msgr2=0x7fe4701602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe476400640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 0x7fe4701602e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe476400640 1 -- 172.21.15.28:0/3133216451 >> v1:172.21.15.28:6789/0 conn(0x7fe4700ac850 legacy=0x7fe4700b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe476400640 1 -- 172.21.15.28:0/3133216451 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470160aa0 con 0x7fe47015d9e0 2024-09-13T17:08:15.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.158+0000 7fe475a00640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 0x7fe4701602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:15.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.159+0000 7fe476400640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe47015d9e0 0x7fe47006bd50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe468009dc0 tx=0x7fe46802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e649c328504c7fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.159+0000 7fe46fe00640 1 -- 172.21.15.28:0/3133216451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468007740 con 0x7fe47015d9e0 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.159+0000 7fe46fe00640 1 -- 172.21.15.28:0/3133216451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46802d050 con 0x7fe47015d9e0 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.159+0000 7fe46fe00640 1 -- 172.21.15.28:0/3133216451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468007ac0 con 0x7fe47015d9e0 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 -- 172.21.15.28:0/3133216451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe47015d9e0 msgr2=0x7fe47006bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe47015d9e0 0x7fe47006bd50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe468009dc0 tx=0x7fe46802a680 comp rx=0 tx=0).stop 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 -- 172.21.15.28:0/3133216451 shutdown_connections 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe47015d9e0 0x7fe47006bd50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 --2- 172.21.15.28:0/3133216451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015c5f0 0x7fe4701602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 -- 172.21.15.28:0/3133216451 >> 172.21.15.28:0/3133216451 conn(0x7fe4700ce880 msgr2=0x7fe4700cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 -- 172.21.15.28:0/3133216451 shutdown_connections 2024-09-13T17:08:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.160+0000 7fe476e00640 1 -- 172.21.15.28:0/3133216451 wait complete. 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.162+0000 7fe476e00640 1 Processor -- start 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.162+0000 7fe476e00640 1 -- start start 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 0x7fe470153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015d9e0 0x7fe470153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe470150310 con 0x7fe4700ac850 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47014f790 con 0x7fe47015c5f0 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe476e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47014fa70 con 0x7fe47015d9e0 2024-09-13T17:08:15.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33276/0 (socket says 172.21.15.28:33276) 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 -- 172.21.15.28:0/2415697831 learned_addr learned my addr 172.21.15.28:0/2415697831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 -- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015d9e0 msgr2=0x7fe470153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475000640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 0x7fe470153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015d9e0 0x7fe470153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 -- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 msgr2=0x7fe470153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 0x7fe470153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475a00640 1 -- 172.21.15.28:0/2415697831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009a70 con 0x7fe4700ac850 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.163+0000 7fe475000640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 0x7fe470153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe475a00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe46400e7e0 tx=0x7fe46400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464004030 con 0x7fe4700ac850 2024-09-13T17:08:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464004190 con 0x7fe4700ac850 2024-09-13T17:08:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464004520 con 0x7fe4700ac850 2024-09-13T17:08:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe47014fd50 con 0x7fe4700ac850 2024-09-13T17:08:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.164+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4700af970 con 0x7fe4700ac850 2024-09-13T17:08:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.165+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4700afb60 con 0x7fe4700ac850 2024-09-13T17:08:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.166+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe46400c6d0 con 0x7fe4700ac850 2024-09-13T17:08:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.167+0000 7fe46d600640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 0x7fe45006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.167+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe464088ff0 con 0x7fe4700ac850 2024-09-13T17:08:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.167+0000 7fe475000640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 0x7fe45006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.168+0000 7fe475000640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 0x7fe45006d0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe4700aecd0 tx=0x7fe45c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.171+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe464054220 con 0x7fe4700ac850 2024-09-13T17:08:15.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.350+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4701f32c0 con 0x7fe4700ac850 2024-09-13T17:08:15.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.351+0000 7fe46d600640 1 -- 172.21.15.28:0/2415697831 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fe464057840 con 0x7fe4700ac850 2024-09-13T17:08:15.353 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:15.353 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 msgr2=0x7fe45006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 0x7fe45006d0c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe4700aecd0 tx=0x7fe45c009040 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 msgr2=0x7fe47014ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe46400e7e0 tx=0x7fe46400eca0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 shutdown_connections 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe45006ac00 0x7fe45006d0c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe47015d9e0 0x7fe470153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe47015c5f0 0x7fe470153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.353+0000 7fe476e00640 1 --2- 172.21.15.28:0/2415697831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4700ac850 0x7fe47014ed50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.354+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 >> 172.21.15.28:0/2415697831 conn(0x7fe4700ce880 msgr2=0x7fe47015c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.354+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 shutdown_connections 2024-09-13T17:08:15.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.354+0000 7fe476e00640 1 -- 172.21.15.28:0/2415697831 wait complete. 2024-09-13T17:08:15.372 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:08:15.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:08:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.524+0000 7f8174000640 1 Processor -- start 2024-09-13T17:08:15.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.524+0000 7f8174000640 1 -- start start 2024-09-13T17:08:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.525+0000 7f8174000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.525+0000 7f8174000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816c06f440 0x7f816c06f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.525+0000 7f8174000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816c0b3970 con 0x7f816c0b3160 2024-09-13T17:08:15.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.525+0000 7f8174000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816c159230 con 0x7f816c06f440 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.525+0000 7f8174000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c06fe10 con 0x7f816c0cdfa0 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816c06f440 0x7f816c06f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816c06f440 0x7f816c06f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33282/0 (socket says 172.21.15.28:33282) 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 -- 172.21.15.28:0/1136569190 learned_addr learned my addr 172.21.15.28:0/1136569190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8171600640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 -- 172.21.15.28:0/1136569190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 msgr2=0x7f816c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 -- 172.21.15.28:0/1136569190 >> v1:172.21.15.189:6789/0 conn(0x7f816c0b3160 legacy=0x7f816c0ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.526+0000 7f8172000640 1 -- 172.21.15.28:0/1136569190 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f816c0cec70 con 0x7f816c06f440 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.527+0000 7f8172000640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c06f8d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b625c833ff9b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.527+0000 7f816be00640 1 -- 172.21.15.28:0/1136569190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164031600 con 0x7f816c06f440 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.527+0000 7f816be00640 1 -- 172.21.15.28:0/1136569190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816402d050 con 0x7f816c06f440 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.527+0000 7f816be00640 1 -- 172.21.15.28:0/1136569190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8164031980 con 0x7f816c06f440 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 -- 172.21.15.28:0/1136569190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 msgr2=0x7f816c06f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c06f8d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8164009e50 tx=0x7f816402a710 comp rx=0 tx=0).stop 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 -- 172.21.15.28:0/1136569190 shutdown_connections 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c06f8d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 --2- 172.21.15.28:0/1136569190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 -- 172.21.15.28:0/1136569190 >> 172.21.15.28:0/1136569190 conn(0x7f816c069e80 msgr2=0x7f816c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 -- 172.21.15.28:0/1136569190 shutdown_connections 2024-09-13T17:08:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.528+0000 7f8174000640 1 -- 172.21.15.28:0/1136569190 wait complete. 2024-09-13T17:08:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.529+0000 7f8174000640 1 Processor -- start 2024-09-13T17:08:15.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 -- start start 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f816c0b3160 0x7f816c1502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c1f0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1f12b0 con 0x7f816c06f440 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1f1830 con 0x7f816c0b3160 2024-09-13T17:08:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.530+0000 7f8174000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f816c1f1b10 con 0x7f816c0cdfa0 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33290/0 (socket says 172.21.15.28:33290) 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 -- 172.21.15.28:0/1541164588 learned_addr learned my addr 172.21.15.28:0/1541164588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8172000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c1f0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8170c00640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f816c0b3160 0x7f816c1502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 -- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 msgr2=0x7f816c1f0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c1f0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 -- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f816c0b3160 msgr2=0x7f816c1502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f816c0b3160 0x7f816c1502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 -- 172.21.15.28:0/1541164588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8164009b30 con 0x7f816c06f440 2024-09-13T17:08:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.531+0000 7f8171600640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f816000d5d0 tx=0x7f816000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.532+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600125f0 con 0x7f816c06f440 2024-09-13T17:08:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.532+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160012770 con 0x7f816c06f440 2024-09-13T17:08:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.532+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816c1f1df0 con 0x7f816c06f440 2024-09-13T17:08:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.532+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160012b20 con 0x7f816c06f440 2024-09-13T17:08:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.532+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f816c1f23b0 con 0x7f816c06f440 2024-09-13T17:08:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.533+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f816c06c2d0 con 0x7f816c06f440 2024-09-13T17:08:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.534+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8160012c80 con 0x7f816c06f440 2024-09-13T17:08:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.534+0000 7f8169600640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 0x7f814c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.535+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8160089760 con 0x7f816c06f440 2024-09-13T17:08:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.535+0000 7f8170c00640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 0x7f814c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.535+0000 7f8170c00640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 0x7f814c06d0f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f816c1edee0 tx=0x7f8158009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.538+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8160054950 con 0x7f816c06f440 2024-09-13T17:08:15.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.715+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f816c1599f0 con 0x7f814c06ac30 2024-09-13T17:08:15.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.724+0000 7f8169600640 1 -- 172.21.15.28:0/1541164588 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264581 (secure 0 0 0) 0x7f81580014a0 con 0x7f814c06ac30 2024-09-13T17:08:15.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:15.731 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:08:15.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 msgr2=0x7f814c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 0x7f814c06d0f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f816c1edee0 tx=0x7f8158009040 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 msgr2=0x7f816c1edf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f816000d5d0 tx=0x7f816000da90 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 shutdown_connections 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f814c06ac30 0x7f814c06d0f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816c0cdfa0 0x7f816c1f0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f816c0b3160 0x7f816c1502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 --2- 172.21.15.28:0/1541164588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816c06f440 0x7f816c1edf30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 >> 172.21.15.28:0/1541164588 conn(0x7f816c069e80 msgr2=0x7f816c0ce500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 shutdown_connections 2024-09-13T17:08:15.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.728+0000 7f8174000640 1 -- 172.21.15.28:0/1541164588 wait complete. 2024-09-13T17:08:15.746 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T17:08:14.884520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235292,"kb_used_data":2720,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754739428,"statfs":{"total":773094113280,"available":772853174272,"internally_reserved":0,"allocated":2785280,"data_stored":1700206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.773919"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606164+0000","last_change":"2024-09-13T17:08:11.601843+0000","last_active":"2024-09-13T17:08:11.606164+0000","last_peered":"2024-09-13T17:08:11.606164+0000","last_clean":"2024-09-13T17:08:11.606164+0000","last_became_active":"2024-09-13T17:08:11.601437+0000","last_became_peered":"2024-09-13T17:08:11.601437+0000","last_unstale":"2024-09-13T17:08:11.606164+0000","last_undegraded":"2024-09-13T17:08:11.606164+0000","last_fullsized":"2024-09-13T17:08:11.606164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:28.411961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566115+0000","last_change":"2024-09-13T17:08:11.576946+0000","last_active":"2024-09-13T17:08:12.566115+0000","last_peered":"2024-09-13T17:08:12.566115+0000","last_clean":"2024-09-13T17:08:12.566115+0000","last_became_active":"2024-09-13T17:08:11.576700+0000","last_became_peered":"2024-09-13T17:08:11.576700+0000","last_unstale":"2024-09-13T17:08:12.566115+0000","last_undegraded":"2024-09-13T17:08:12.566115+0000","last_fullsized":"2024-09-13T17:08:12.566115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.580583+0000","last_change":"2024-09-13T17:08:11.576052+0000","last_active":"2024-09-13T17:08:11.580583+0000","last_peered":"2024-09-13T17:08:11.580583+0000","last_clean":"2024-09-13T17:08:11.580583+0000","last_became_active":"2024-09-13T17:08:11.575297+0000","last_became_peered":"2024-09-13T17:08:11.575297+0000","last_unstale":"2024-09-13T17:08:11.580583+0000","last_undegraded":"2024-09-13T17:08:11.580583+0000","last_fullsized":"2024-09-13T17:08:11.580583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567600+0000","last_change":"2024-09-13T17:08:11.582113+0000","last_active":"2024-09-13T17:08:12.567600+0000","last_peered":"2024-09-13T17:08:12.567600+0000","last_clean":"2024-09-13T17:08:12.567600+0000","last_became_active":"2024-09-13T17:08:11.579185+0000","last_became_peered":"2024-09-13T17:08:11.579185+0000","last_unstale":"2024-09-13T17:08:12.567600+0000","last_undegraded":"2024-09-13T17:08:12.567600+0000","last_fullsized":"2024-09-13T17:08:12.567600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606007+0000","last_change":"2024-09-13T17:08:11.600241+0000","last_active":"2024-09-13T17:08:11.606007+0000","last_peered":"2024-09-13T17:08:11.606007+0000","last_clean":"2024-09-13T17:08:11.606007+0000","last_became_active":"2024-09-13T17:08:11.596443+0000","last_became_peered":"2024-09-13T17:08:11.596443+0000","last_unstale":"2024-09-13T17:08:11.606007+0000","last_undegraded":"2024-09-13T17:08:11.606007+0000","last_fullsized":"2024-09-13T17:08:11.606007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567274+0000","last_change":"2024-09-13T17:08:11.583211+0000","last_active":"2024-09-13T17:08:12.567274+0000","last_peered":"2024-09-13T17:08:12.567274+0000","last_clean":"2024-09-13T17:08:12.567274+0000","last_became_active":"2024-09-13T17:08:11.583040+0000","last_became_peered":"2024-09-13T17:08:11.583040+0000","last_unstale":"2024-09-13T17:08:12.567274+0000","last_undegraded":"2024-09-13T17:08:12.567274+0000","last_fullsized":"2024-09-13T17:08:12.567274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596334+0000","last_change":"2024-09-13T17:08:11.592666+0000","last_active":"2024-09-13T17:08:11.596334+0000","last_peered":"2024-09-13T17:08:11.596334+0000","last_clean":"2024-09-13T17:08:11.596334+0000","last_became_active":"2024-09-13T17:08:11.592373+0000","last_became_peered":"2024-09-13T17:08:11.592373+0000","last_unstale":"2024-09-13T17:08:11.596334+0000","last_undegraded":"2024-09-13T17:08:11.596334+0000","last_fullsized":"2024-09-13T17:08:11.596334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.583252+0000","last_change":"2024-09-13T17:08:11.578631+0000","last_active":"2024-09-13T17:08:11.583252+0000","last_peered":"2024-09-13T17:08:11.583252+0000","last_clean":"2024-09-13T17:08:11.583252+0000","last_became_active":"2024-09-13T17:08:11.575801+0000","last_became_peered":"2024-09-13T17:08:11.575801+0000","last_unstale":"2024-09-13T17:08:11.583252+0000","last_undegraded":"2024-09-13T17:08:11.583252+0000","last_fullsized":"2024-09-13T17:08:11.583252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606141+0000","last_change":"2024-09-13T17:08:11.601554+0000","last_active":"2024-09-13T17:08:11.606141+0000","last_peered":"2024-09-13T17:08:11.606141+0000","last_clean":"2024-09-13T17:08:11.606141+0000","last_became_active":"2024-09-13T17:08:11.601069+0000","last_became_peered":"2024-09-13T17:08:11.601069+0000","last_unstale":"2024-09-13T17:08:11.606141+0000","last_undegraded":"2024-09-13T17:08:11.606141+0000","last_fullsized":"2024-09-13T17:08:11.606141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:24.360659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593128+0000","last_change":"2024-09-13T17:08:11.587154+0000","last_active":"2024-09-13T17:08:11.593128+0000","last_peered":"2024-09-13T17:08:11.593128+0000","last_clean":"2024-09-13T17:08:11.593128+0000","last_became_active":"2024-09-13T17:08:11.586876+0000","last_became_peered":"2024-09-13T17:08:11.586876+0000","last_unstale":"2024-09-13T17:08:11.593128+0000","last_undegraded":"2024-09-13T17:08:11.593128+0000","last_fullsized":"2024-09-13T17:08:11.593128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591336+0000","last_change":"2024-09-13T17:08:11.586451+0000","last_active":"2024-09-13T17:08:11.591336+0000","last_peered":"2024-09-13T17:08:11.591336+0000","last_clean":"2024-09-13T17:08:11.591336+0000","last_became_active":"2024-09-13T17:08:11.586127+0000","last_became_peered":"2024-09-13T17:08:11.586127+0000","last_unstale":"2024-09-13T17:08:11.591336+0000","last_undegraded":"2024-09-13T17:08:11.591336+0000","last_fullsized":"2024-09-13T17:08:11.591336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569157+0000","last_change":"2024-09-13T17:08:11.591207+0000","last_active":"2024-09-13T17:08:12.569157+0000","last_peered":"2024-09-13T17:08:12.569157+0000","last_clean":"2024-09-13T17:08:12.569157+0000","last_became_active":"2024-09-13T17:08:11.590920+0000","last_became_peered":"2024-09-13T17:08:11.590920+0000","last_unstale":"2024-09-13T17:08:12.569157+0000","last_undegraded":"2024-09-13T17:08:12.569157+0000","last_fullsized":"2024-09-13T17:08:12.569157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:25.618116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591159+0000","last_change":"2024-09-13T17:08:11.586067+0000","last_active":"2024-09-13T17:08:11.591159+0000","last_peered":"2024-09-13T17:08:11.591159+0000","last_clean":"2024-09-13T17:08:11.591159+0000","last_became_active":"2024-09-13T17:08:11.585787+0000","last_became_peered":"2024-09-13T17:08:11.585787+0000","last_unstale":"2024-09-13T17:08:11.591159+0000","last_undegraded":"2024-09-13T17:08:11.591159+0000","last_fullsized":"2024-09-13T17:08:11.591159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572983+0000","last_change":"2024-09-13T17:08:11.570021+0000","last_active":"2024-09-13T17:08:11.572983+0000","last_peered":"2024-09-13T17:08:11.572983+0000","last_clean":"2024-09-13T17:08:11.572983+0000","last_became_active":"2024-09-13T17:08:11.569663+0000","last_became_peered":"2024-09-13T17:08:11.569663+0000","last_unstale":"2024-09-13T17:08:11.572983+0000","last_undegraded":"2024-09-13T17:08:11.572983+0000","last_fullsized":"2024-09-13T17:08:11.572983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565777+0000","last_change":"2024-09-13T17:08:11.578244+0000","last_active":"2024-09-13T17:08:12.565777+0000","last_peered":"2024-09-13T17:08:12.565777+0000","last_clean":"2024-09-13T17:08:12.565777+0000","last_became_active":"2024-09-13T17:08:11.577989+0000","last_became_peered":"2024-09-13T17:08:11.577989+0000","last_unstale":"2024-09-13T17:08:12.565777+0000","last_undegraded":"2024-09-13T17:08:12.565777+0000","last_fullsized":"2024-09-13T17:08:12.565777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567131+0000","last_change":"2024-09-13T17:08:11.582930+0000","last_active":"2024-09-13T17:08:12.567131+0000","last_peered":"2024-09-13T17:08:12.567131+0000","last_clean":"2024-09-13T17:08:12.567131+0000","last_became_active":"2024-09-13T17:08:11.582621+0000","last_became_peered":"2024-09-13T17:08:11.582621+0000","last_unstale":"2024-09-13T17:08:12.567131+0000","last_undegraded":"2024-09-13T17:08:12.567131+0000","last_fullsized":"2024-09-13T17:08:12.567131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568144+0000","last_change":"2024-09-13T17:08:11.601899+0000","last_active":"2024-09-13T17:08:12.568144+0000","last_peered":"2024-09-13T17:08:12.568144+0000","last_clean":"2024-09-13T17:08:12.568144+0000","last_became_active":"2024-09-13T17:08:11.601452+0000","last_became_peered":"2024-09-13T17:08:11.601452+0000","last_unstale":"2024-09-13T17:08:12.568144+0000","last_undegraded":"2024-09-13T17:08:12.568144+0000","last_fullsized":"2024-09-13T17:08:12.568144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593019+0000","last_change":"2024-09-13T17:08:11.591991+0000","last_active":"2024-09-13T17:08:11.593019+0000","last_peered":"2024-09-13T17:08:11.593019+0000","last_clean":"2024-09-13T17:08:11.593019+0000","last_became_active":"2024-09-13T17:08:11.591554+0000","last_became_peered":"2024-09-13T17:08:11.591554+0000","last_unstale":"2024-09-13T17:08:11.593019+0000","last_undegraded":"2024-09-13T17:08:11.593019+0000","last_fullsized":"2024-09-13T17:08:11.593019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.599818+0000","last_change":"2024-09-13T17:08:11.596861+0000","last_active":"2024-09-13T17:08:11.599818+0000","last_peered":"2024-09-13T17:08:11.599818+0000","last_clean":"2024-09-13T17:08:11.599818+0000","last_became_active":"2024-09-13T17:08:11.596669+0000","last_became_peered":"2024-09-13T17:08:11.596669+0000","last_unstale":"2024-09-13T17:08:11.599818+0000","last_undegraded":"2024-09-13T17:08:11.599818+0000","last_fullsized":"2024-09-13T17:08:11.599818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591158+0000","last_change":"2024-09-13T17:08:11.582225+0000","last_active":"2024-09-13T17:08:11.591158+0000","last_peered":"2024-09-13T17:08:11.591158+0000","last_clean":"2024-09-13T17:08:11.591158+0000","last_became_active":"2024-09-13T17:08:11.581900+0000","last_became_peered":"2024-09-13T17:08:11.581900+0000","last_unstale":"2024-09-13T17:08:11.591158+0000","last_undegraded":"2024-09-13T17:08:11.591158+0000","last_fullsized":"2024-09-13T17:08:11.591158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591212+0000","last_change":"2024-09-13T17:08:11.588070+0000","last_active":"2024-09-13T17:08:11.591212+0000","last_peered":"2024-09-13T17:08:11.591212+0000","last_clean":"2024-09-13T17:08:11.591212+0000","last_became_active":"2024-09-13T17:08:11.587859+0000","last_became_peered":"2024-09-13T17:08:11.587859+0000","last_unstale":"2024-09-13T17:08:11.591212+0000","last_undegraded":"2024-09-13T17:08:11.591212+0000","last_fullsized":"2024-09-13T17:08:11.591212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567322+0000","last_change":"2024-09-13T17:08:11.597424+0000","last_active":"2024-09-13T17:08:12.567322+0000","last_peered":"2024-09-13T17:08:12.567322+0000","last_clean":"2024-09-13T17:08:12.567322+0000","last_became_active":"2024-09-13T17:08:11.597093+0000","last_became_peered":"2024-09-13T17:08:11.597093+0000","last_unstale":"2024-09-13T17:08:12.567322+0000","last_undegraded":"2024-09-13T17:08:12.567322+0000","last_fullsized":"2024-09-13T17:08:12.567322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.585950+0000","last_change":"2024-09-13T17:08:11.581514+0000","last_active":"2024-09-13T17:08:11.585950+0000","last_peered":"2024-09-13T17:08:11.585950+0000","last_clean":"2024-09-13T17:08:11.585950+0000","last_became_active":"2024-09-13T17:08:11.581178+0000","last_became_peered":"2024-09-13T17:08:11.581178+0000","last_unstale":"2024-09-13T17:08:11.585950+0000","last_undegraded":"2024-09-13T17:08:11.585950+0000","last_fullsized":"2024-09-13T17:08:11.585950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567215+0000","last_change":"2024-09-13T17:08:11.575094+0000","last_active":"2024-09-13T17:08:12.567215+0000","last_peered":"2024-09-13T17:08:12.567215+0000","last_clean":"2024-09-13T17:08:12.567215+0000","last_became_active":"2024-09-13T17:08:11.574833+0000","last_became_peered":"2024-09-13T17:08:11.574833+0000","last_unstale":"2024-09-13T17:08:12.567215+0000","last_undegraded":"2024-09-13T17:08:12.567215+0000","last_fullsized":"2024-09-13T17:08:12.567215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.586697+0000","last_change":"2024-09-13T17:08:11.581636+0000","last_active":"2024-09-13T17:08:11.586697+0000","last_peered":"2024-09-13T17:08:11.586697+0000","last_clean":"2024-09-13T17:08:11.586697+0000","last_became_active":"2024-09-13T17:08:11.581264+0000","last_became_peered":"2024-09-13T17:08:11.581264+0000","last_unstale":"2024-09-13T17:08:11.586697+0000","last_undegraded":"2024-09-13T17:08:11.586697+0000","last_fullsized":"2024-09-13T17:08:11.586697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.603877+0000","last_change":"2024-09-13T17:08:11.601141+0000","last_active":"2024-09-13T17:08:11.603877+0000","last_peered":"2024-09-13T17:08:11.603877+0000","last_clean":"2024-09-13T17:08:11.603877+0000","last_became_active":"2024-09-13T17:08:11.600877+0000","last_became_peered":"2024-09-13T17:08:11.600877+0000","last_unstale":"2024-09-13T17:08:11.603877+0000","last_undegraded":"2024-09-13T17:08:11.603877+0000","last_fullsized":"2024-09-13T17:08:11.603877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565968+0000","last_change":"2024-09-13T17:08:11.576209+0000","last_active":"2024-09-13T17:08:12.565968+0000","last_peered":"2024-09-13T17:08:12.565968+0000","last_clean":"2024-09-13T17:08:12.565968+0000","last_became_active":"2024-09-13T17:08:11.575917+0000","last_became_peered":"2024-09-13T17:08:11.575917+0000","last_unstale":"2024-09-13T17:08:12.565968+0000","last_undegraded":"2024-09-13T17:08:12.565968+0000","last_fullsized":"2024-09-13T17:08:12.565968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.590314+0000","last_change":"2024-09-13T17:08:11.582222+0000","last_active":"2024-09-13T17:08:11.590314+0000","last_peered":"2024-09-13T17:08:11.590314+0000","last_clean":"2024-09-13T17:08:11.590314+0000","last_became_active":"2024-09-13T17:08:11.582057+0000","last_became_peered":"2024-09-13T17:08:11.582057+0000","last_unstale":"2024-09-13T17:08:11.590314+0000","last_undegraded":"2024-09-13T17:08:11.590314+0000","last_fullsized":"2024-09-13T17:08:11.590314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.582927+0000","last_change":"2024-09-13T17:08:11.575089+0000","last_active":"2024-09-13T17:08:11.582927+0000","last_peered":"2024-09-13T17:08:11.582927+0000","last_clean":"2024-09-13T17:08:11.582927+0000","last_became_active":"2024-09-13T17:08:11.574665+0000","last_became_peered":"2024-09-13T17:08:11.574665+0000","last_unstale":"2024-09-13T17:08:11.582927+0000","last_undegraded":"2024-09-13T17:08:11.582927+0000","last_fullsized":"2024-09-13T17:08:11.582927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567404+0000","last_change":"2024-09-13T17:08:11.579735+0000","last_active":"2024-09-13T17:08:12.567404+0000","last_peered":"2024-09-13T17:08:12.567404+0000","last_clean":"2024-09-13T17:08:12.567404+0000","last_became_active":"2024-09-13T17:08:11.578749+0000","last_became_peered":"2024-09-13T17:08:11.578749+0000","last_unstale":"2024-09-13T17:08:12.567404+0000","last_undegraded":"2024-09-13T17:08:12.567404+0000","last_fullsized":"2024-09-13T17:08:12.567404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572297+0000","last_change":"2024-09-13T17:08:11.570273+0000","last_active":"2024-09-13T17:08:11.572297+0000","last_peered":"2024-09-13T17:08:11.572297+0000","last_clean":"2024-09-13T17:08:11.572297+0000","last_became_active":"2024-09-13T17:08:11.569987+0000","last_became_peered":"2024-09-13T17:08:11.569987+0000","last_unstale":"2024-09-13T17:08:11.572297+0000","last_undegraded":"2024-09-13T17:08:11.572297+0000","last_fullsized":"2024-09-13T17:08:11.572297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591341+0000","last_change":"2024-09-13T17:08:11.586071+0000","last_active":"2024-09-13T17:08:11.591341+0000","last_peered":"2024-09-13T17:08:11.591341+0000","last_clean":"2024-09-13T17:08:11.591341+0000","last_became_active":"2024-09-13T17:08:11.585507+0000","last_became_peered":"2024-09-13T17:08:11.585507+0000","last_unstale":"2024-09-13T17:08:11.591341+0000","last_undegraded":"2024-09-13T17:08:11.591341+0000","last_fullsized":"2024-09-13T17:08:11.591341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567646+0000","last_change":"2024-09-13T17:08:11.578066+0000","last_active":"2024-09-13T17:08:12.567646+0000","last_peered":"2024-09-13T17:08:12.567646+0000","last_clean":"2024-09-13T17:08:12.567646+0000","last_became_active":"2024-09-13T17:08:11.577655+0000","last_became_peered":"2024-09-13T17:08:11.577655+0000","last_unstale":"2024-09-13T17:08:12.567646+0000","last_undegraded":"2024-09-13T17:08:12.567646+0000","last_fullsized":"2024-09-13T17:08:12.567646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.318727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.601896+0000","last_change":"2024-09-13T17:08:11.147841+0000","last_active":"2024-09-13T17:08:11.601896+0000","last_peered":"2024-09-13T17:08:11.601896+0000","last_clean":"2024-09-13T17:08:11.601896+0000","last_became_active":"2024-09-13T17:08:11.147449+0000","last_became_peered":"2024-09-13T17:08:11.147449+0000","last_unstale":"2024-09-13T17:08:11.601896+0000","last_undegraded":"2024-09-13T17:08:11.601896+0000","last_fullsized":"2024-09-13T17:08:11.601896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.582944+0000","last_change":"2024-09-13T17:08:11.580301+0000","last_active":"2024-09-13T17:08:11.582944+0000","last_peered":"2024-09-13T17:08:11.582944+0000","last_clean":"2024-09-13T17:08:11.582944+0000","last_became_active":"2024-09-13T17:08:11.579029+0000","last_became_peered":"2024-09-13T17:08:11.579029+0000","last_unstale":"2024-09-13T17:08:11.582944+0000","last_undegraded":"2024-09-13T17:08:11.582944+0000","last_fullsized":"2024-09-13T17:08:11.582944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:30.800655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.598106+0000","last_change":"2024-09-13T17:08:11.146319+0000","last_active":"2024-09-13T17:08:11.598106+0000","last_peered":"2024-09-13T17:08:11.598106+0000","last_clean":"2024-09-13T17:08:11.598106+0000","last_became_active":"2024-09-13T17:08:11.146057+0000","last_became_peered":"2024-09-13T17:08:11.146057+0000","last_unstale":"2024-09-13T17:08:11.598106+0000","last_undegraded":"2024-09-13T17:08:11.598106+0000","last_fullsized":"2024-09-13T17:08:11.598106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572346+0000","last_change":"2024-09-13T17:08:11.138543+0000","last_active":"2024-09-13T17:08:11.572346+0000","last_peered":"2024-09-13T17:08:11.572346+0000","last_clean":"2024-09-13T17:08:11.572346+0000","last_became_active":"2024-09-13T17:08:11.133550+0000","last_became_peered":"2024-09-13T17:08:11.133550+0000","last_unstale":"2024-09-13T17:08:11.572346+0000","last_undegraded":"2024-09-13T17:08:11.572346+0000","last_fullsized":"2024-09-13T17:08:11.572346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.584696+0000","last_change":"2024-09-13T17:08:11.580334+0000","last_active":"2024-09-13T17:08:11.584696+0000","last_peered":"2024-09-13T17:08:11.584696+0000","last_clean":"2024-09-13T17:08:11.584696+0000","last_became_active":"2024-09-13T17:08:11.574885+0000","last_became_peered":"2024-09-13T17:08:11.574885+0000","last_unstale":"2024-09-13T17:08:11.584696+0000","last_undegraded":"2024-09-13T17:08:11.584696+0000","last_fullsized":"2024-09-13T17:08:11.584696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567242+0000","last_change":"2024-09-13T17:08:11.142227+0000","last_active":"2024-09-13T17:08:12.567242+0000","last_peered":"2024-09-13T17:08:12.567242+0000","last_clean":"2024-09-13T17:08:12.567242+0000","last_became_active":"2024-09-13T17:08:11.142009+0000","last_became_peered":"2024-09-13T17:08:11.142009+0000","last_unstale":"2024-09-13T17:08:12.567242+0000","last_undegraded":"2024-09-13T17:08:12.567242+0000","last_fullsized":"2024-09-13T17:08:12.567242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565757+0000","last_change":"2024-09-13T17:08:11.569231+0000","last_active":"2024-09-13T17:08:12.565757+0000","last_peered":"2024-09-13T17:08:12.565757+0000","last_clean":"2024-09-13T17:08:12.565757+0000","last_became_active":"2024-09-13T17:08:11.568934+0000","last_became_peered":"2024-09-13T17:08:11.568934+0000","last_unstale":"2024-09-13T17:08:12.565757+0000","last_undegraded":"2024-09-13T17:08:12.565757+0000","last_fullsized":"2024-09-13T17:08:12.565757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.578842+0000","last_change":"2024-09-13T17:08:11.140675+0000","last_active":"2024-09-13T17:08:11.578842+0000","last_peered":"2024-09-13T17:08:11.578842+0000","last_clean":"2024-09-13T17:08:11.578842+0000","last_became_active":"2024-09-13T17:08:11.139918+0000","last_became_peered":"2024-09-13T17:08:11.139918+0000","last_unstale":"2024-09-13T17:08:11.578842+0000","last_undegraded":"2024-09-13T17:08:11.578842+0000","last_fullsized":"2024-09-13T17:08:11.578842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568038+0000","last_change":"2024-09-13T17:08:11.599453+0000","last_active":"2024-09-13T17:08:12.568038+0000","last_peered":"2024-09-13T17:08:12.568038+0000","last_clean":"2024-09-13T17:08:12.568038+0000","last_became_active":"2024-09-13T17:08:11.599107+0000","last_became_peered":"2024-09-13T17:08:11.599107+0000","last_unstale":"2024-09-13T17:08:12.568038+0000","last_undegraded":"2024-09-13T17:08:12.568038+0000","last_fullsized":"2024-09-13T17:08:12.568038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568117+0000","last_change":"2024-09-13T17:08:11.136652+0000","last_active":"2024-09-13T17:08:12.568117+0000","last_peered":"2024-09-13T17:08:12.568117+0000","last_clean":"2024-09-13T17:08:12.568117+0000","last_became_active":"2024-09-13T17:08:11.135676+0000","last_became_peered":"2024-09-13T17:08:11.135676+0000","last_unstale":"2024-09-13T17:08:12.568117+0000","last_undegraded":"2024-09-13T17:08:12.568117+0000","last_fullsized":"2024-09-13T17:08:12.568117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591687+0000","last_change":"2024-09-13T17:08:11.589078+0000","last_active":"2024-09-13T17:08:11.591687+0000","last_peered":"2024-09-13T17:08:11.591687+0000","last_clean":"2024-09-13T17:08:11.591687+0000","last_became_active":"2024-09-13T17:08:11.588613+0000","last_became_peered":"2024-09-13T17:08:11.588613+0000","last_unstale":"2024-09-13T17:08:11.591687+0000","last_undegraded":"2024-09-13T17:08:11.591687+0000","last_fullsized":"2024-09-13T17:08:11.591687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565993+0000","last_change":"2024-09-13T17:08:11.139774+0000","last_active":"2024-09-13T17:08:12.565993+0000","last_peered":"2024-09-13T17:08:12.565993+0000","last_clean":"2024-09-13T17:08:12.565993+0000","last_became_active":"2024-09-13T17:08:11.139257+0000","last_became_peered":"2024-09-13T17:08:11.139257+0000","last_unstale":"2024-09-13T17:08:12.565993+0000","last_undegraded":"2024-09-13T17:08:12.565993+0000","last_fullsized":"2024-09-13T17:08:12.565993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569193+0000","last_change":"2024-09-13T17:08:11.590978+0000","last_active":"2024-09-13T17:08:12.569193+0000","last_peered":"2024-09-13T17:08:12.569193+0000","last_clean":"2024-09-13T17:08:12.569193+0000","last_became_active":"2024-09-13T17:08:11.590763+0000","last_became_peered":"2024-09-13T17:08:11.590763+0000","last_unstale":"2024-09-13T17:08:12.569193+0000","last_undegraded":"2024-09-13T17:08:12.569193+0000","last_fullsized":"2024-09-13T17:08:12.569193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.608756+0000","last_change":"2024-09-13T17:08:11.134868+0000","last_active":"2024-09-13T17:08:11.608756+0000","last_peered":"2024-09-13T17:08:11.608756+0000","last_clean":"2024-09-13T17:08:11.608756+0000","last_became_active":"2024-09-13T17:08:11.134516+0000","last_became_peered":"2024-09-13T17:08:11.134516+0000","last_unstale":"2024-09-13T17:08:11.608756+0000","last_undegraded":"2024-09-13T17:08:11.608756+0000","last_fullsized":"2024-09-13T17:08:11.608756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596787+0000","last_change":"2024-09-13T17:08:11.587369+0000","last_active":"2024-09-13T17:08:11.596787+0000","last_peered":"2024-09-13T17:08:11.596787+0000","last_clean":"2024-09-13T17:08:11.596787+0000","last_became_active":"2024-09-13T17:08:11.586043+0000","last_became_peered":"2024-09-13T17:08:11.586043+0000","last_unstale":"2024-09-13T17:08:11.596787+0000","last_undegraded":"2024-09-13T17:08:11.596787+0000","last_fullsized":"2024-09-13T17:08:11.596787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.597601+0000","last_change":"2024-09-13T17:08:11.145171+0000","last_active":"2024-09-13T17:08:11.597601+0000","last_peered":"2024-09-13T17:08:11.597601+0000","last_clean":"2024-09-13T17:08:11.597601+0000","last_became_active":"2024-09-13T17:08:11.144862+0000","last_became_peered":"2024-09-13T17:08:11.144862+0000","last_unstale":"2024-09-13T17:08:11.597601+0000","last_undegraded":"2024-09-13T17:08:11.597601+0000","last_fullsized":"2024-09-13T17:08:11.597601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596904+0000","last_change":"2024-09-13T17:08:11.596289+0000","last_active":"2024-09-13T17:08:11.596904+0000","last_peered":"2024-09-13T17:08:11.596904+0000","last_clean":"2024-09-13T17:08:11.596904+0000","last_became_active":"2024-09-13T17:08:11.595983+0000","last_became_peered":"2024-09-13T17:08:11.595983+0000","last_unstale":"2024-09-13T17:08:11.596904+0000","last_undegraded":"2024-09-13T17:08:11.596904+0000","last_fullsized":"2024-09-13T17:08:11.596904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:39.915077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567483+0000","last_change":"2024-09-13T17:08:11.143981+0000","last_active":"2024-09-13T17:08:12.567483+0000","last_peered":"2024-09-13T17:08:12.567483+0000","last_clean":"2024-09-13T17:08:12.567483+0000","last_became_active":"2024-09-13T17:08:11.143705+0000","last_became_peered":"2024-09-13T17:08:11.143705+0000","last_unstale":"2024-09-13T17:08:12.567483+0000","last_undegraded":"2024-09-13T17:08:12.567483+0000","last_fullsized":"2024-09-13T17:08:12.567483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606067+0000","last_change":"2024-09-13T17:08:11.601054+0000","last_active":"2024-09-13T17:08:11.606067+0000","last_peered":"2024-09-13T17:08:11.606067+0000","last_clean":"2024-09-13T17:08:11.606067+0000","last_became_active":"2024-09-13T17:08:11.600160+0000","last_became_peered":"2024-09-13T17:08:11.600160+0000","last_unstale":"2024-09-13T17:08:11.606067+0000","last_undegraded":"2024-09-13T17:08:11.606067+0000","last_fullsized":"2024-09-13T17:08:11.606067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.569969+0000","last_change":"2024-09-13T17:08:11.145590+0000","last_active":"2024-09-13T17:08:11.569969+0000","last_peered":"2024-09-13T17:08:11.569969+0000","last_clean":"2024-09-13T17:08:11.569969+0000","last_became_active":"2024-09-13T17:08:11.145267+0000","last_became_peered":"2024-09-13T17:08:11.145267+0000","last_unstale":"2024-09-13T17:08:11.569969+0000","last_undegraded":"2024-09-13T17:08:11.569969+0000","last_fullsized":"2024-09-13T17:08:11.569969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591322+0000","last_change":"2024-09-13T17:08:11.584806+0000","last_active":"2024-09-13T17:08:11.591322+0000","last_peered":"2024-09-13T17:08:11.591322+0000","last_clean":"2024-09-13T17:08:11.591322+0000","last_became_active":"2024-09-13T17:08:11.584303+0000","last_became_peered":"2024-09-13T17:08:11.584303+0000","last_unstale":"2024-09-13T17:08:11.591322+0000","last_undegraded":"2024-09-13T17:08:11.591322+0000","last_fullsized":"2024-09-13T17:08:11.591322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.589937+0000","last_change":"2024-09-13T17:08:11.140195+0000","last_active":"2024-09-13T17:08:11.589937+0000","last_peered":"2024-09-13T17:08:11.589937+0000","last_clean":"2024-09-13T17:08:11.589937+0000","last_became_active":"2024-09-13T17:08:11.139870+0000","last_became_peered":"2024-09-13T17:08:11.139870+0000","last_unstale":"2024-09-13T17:08:11.589937+0000","last_undegraded":"2024-09-13T17:08:11.589937+0000","last_fullsized":"2024-09-13T17:08:11.589937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596116+0000","last_change":"2024-09-13T17:08:11.588867+0000","last_active":"2024-09-13T17:08:11.596116+0000","last_peered":"2024-09-13T17:08:11.596116+0000","last_clean":"2024-09-13T17:08:11.596116+0000","last_became_active":"2024-09-13T17:08:11.588564+0000","last_became_peered":"2024-09-13T17:08:11.588564+0000","last_unstale":"2024-09-13T17:08:11.596116+0000","last_undegraded":"2024-09-13T17:08:11.596116+0000","last_fullsized":"2024-09-13T17:08:11.596116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569202+0000","last_change":"2024-09-13T17:08:11.138154+0000","last_active":"2024-09-13T17:08:12.569202+0000","last_peered":"2024-09-13T17:08:12.569202+0000","last_clean":"2024-09-13T17:08:12.569202+0000","last_became_active":"2024-09-13T17:08:11.137611+0000","last_became_peered":"2024-09-13T17:08:11.137611+0000","last_unstale":"2024-09-13T17:08:12.569202+0000","last_undegraded":"2024-09-13T17:08:12.569202+0000","last_fullsized":"2024-09-13T17:08:12.569202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:48.182859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.608822+0000","last_change":"2024-09-13T17:08:11.604320+0000","last_active":"2024-09-13T17:08:11.608822+0000","last_peered":"2024-09-13T17:08:11.608822+0000","last_clean":"2024-09-13T17:08:11.608822+0000","last_became_active":"2024-09-13T17:08:11.603337+0000","last_became_peered":"2024-09-13T17:08:11.603337+0000","last_unstale":"2024-09-13T17:08:11.608822+0000","last_undegraded":"2024-09-13T17:08:11.608822+0000","last_fullsized":"2024-09-13T17:08:11.608822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.590544+0000","last_change":"2024-09-13T17:08:11.138593+0000","last_active":"2024-09-13T17:08:11.590544+0000","last_peered":"2024-09-13T17:08:11.590544+0000","last_clean":"2024-09-13T17:08:11.590544+0000","last_became_active":"2024-09-13T17:08:11.138131+0000","last_became_peered":"2024-09-13T17:08:11.138131+0000","last_unstale":"2024-09-13T17:08:11.590544+0000","last_undegraded":"2024-09-13T17:08:11.590544+0000","last_fullsized":"2024-09-13T17:08:11.590544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606181+0000","last_change":"2024-09-13T17:08:11.602024+0000","last_active":"2024-09-13T17:08:11.606181+0000","last_peered":"2024-09-13T17:08:11.606181+0000","last_clean":"2024-09-13T17:08:11.606181+0000","last_became_active":"2024-09-13T17:08:11.601178+0000","last_became_peered":"2024-09-13T17:08:11.601178+0000","last_unstale":"2024-09-13T17:08:11.606181+0000","last_undegraded":"2024-09-13T17:08:11.606181+0000","last_fullsized":"2024-09-13T17:08:11.606181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566047+0000","last_change":"2024-09-13T17:08:11.132903+0000","last_active":"2024-09-13T17:08:12.566047+0000","last_peered":"2024-09-13T17:08:12.566047+0000","last_clean":"2024-09-13T17:08:12.566047+0000","last_became_active":"2024-09-13T17:08:11.131326+0000","last_became_peered":"2024-09-13T17:08:11.131326+0000","last_unstale":"2024-09-13T17:08:12.566047+0000","last_undegraded":"2024-09-13T17:08:12.566047+0000","last_fullsized":"2024-09-13T17:08:12.566047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567836+0000","last_change":"2024-09-13T17:08:11.581340+0000","last_active":"2024-09-13T17:08:12.567836+0000","last_peered":"2024-09-13T17:08:12.567836+0000","last_clean":"2024-09-13T17:08:12.567836+0000","last_became_active":"2024-09-13T17:08:11.578762+0000","last_became_peered":"2024-09-13T17:08:11.578762+0000","last_unstale":"2024-09-13T17:08:12.567836+0000","last_undegraded":"2024-09-13T17:08:12.567836+0000","last_fullsized":"2024-09-13T17:08:12.567836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.565331+0000","last_change":"2024-09-13T17:08:11.138090+0000","last_active":"2024-09-13T17:08:11.565331+0000","last_peered":"2024-09-13T17:08:11.565331+0000","last_clean":"2024-09-13T17:08:11.565331+0000","last_became_active":"2024-09-13T17:08:11.137846+0000","last_became_peered":"2024-09-13T17:08:11.137846+0000","last_unstale":"2024-09-13T17:08:11.565331+0000","last_undegraded":"2024-09-13T17:08:11.565331+0000","last_fullsized":"2024-09-13T17:08:11.565331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596798+0000","last_change":"2024-09-13T17:08:11.589182+0000","last_active":"2024-09-13T17:08:11.596798+0000","last_peered":"2024-09-13T17:08:11.596798+0000","last_clean":"2024-09-13T17:08:11.596798+0000","last_became_active":"2024-09-13T17:08:11.588462+0000","last_became_peered":"2024-09-13T17:08:11.588462+0000","last_unstale":"2024-09-13T17:08:11.596798+0000","last_undegraded":"2024-09-13T17:08:11.596798+0000","last_fullsized":"2024-09-13T17:08:11.596798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:40.591592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.140691+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:08:11.140691+0000","last_peered":"2024-09-13T17:08:11.140691+0000","last_clean":"2024-09-13T17:08:11.140691+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:08:11.140691+0000","last_undegraded":"2024-09-13T17:08:11.140691+0000","last_fullsized":"2024-09-13T17:08:11.140691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567696+0000","last_change":"2024-09-13T17:08:11.132650+0000","last_active":"2024-09-13T17:08:12.567696+0000","last_peered":"2024-09-13T17:08:12.567696+0000","last_clean":"2024-09-13T17:08:12.567696+0000","last_became_active":"2024-09-13T17:08:11.132311+0000","last_became_peered":"2024-09-13T17:08:11.132311+0000","last_unstale":"2024-09-13T17:08:12.567696+0000","last_undegraded":"2024-09-13T17:08:12.567696+0000","last_fullsized":"2024-09-13T17:08:12.567696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591524+0000","last_change":"2024-09-13T17:08:11.586834+0000","last_active":"2024-09-13T17:08:11.591524+0000","last_peered":"2024-09-13T17:08:11.591524+0000","last_clean":"2024-09-13T17:08:11.591524+0000","last_became_active":"2024-09-13T17:08:11.585548+0000","last_became_peered":"2024-09-13T17:08:11.585548+0000","last_unstale":"2024-09-13T17:08:11.591524+0000","last_undegraded":"2024-09-13T17:08:11.591524+0000","last_fullsized":"2024-09-13T17:08:11.591524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573891+0000","last_change":"2024-09-13T17:08:11.148517+0000","last_active":"2024-09-13T17:08:11.573891+0000","last_peered":"2024-09-13T17:08:11.573891+0000","last_clean":"2024-09-13T17:08:11.573891+0000","last_became_active":"2024-09-13T17:08:11.148250+0000","last_became_peered":"2024-09-13T17:08:11.148250+0000","last_unstale":"2024-09-13T17:08:11.573891+0000","last_undegraded":"2024-09-13T17:08:11.573891+0000","last_fullsized":"2024-09-13T17:08:11.573891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591072+0000","last_change":"2024-09-13T17:08:11.586386+0000","last_active":"2024-09-13T17:08:11.591072+0000","last_peered":"2024-09-13T17:08:11.591072+0000","last_clean":"2024-09-13T17:08:11.591072+0000","last_became_active":"2024-09-13T17:08:11.577192+0000","last_became_peered":"2024-09-13T17:08:11.577192+0000","last_unstale":"2024-09-13T17:08:11.591072+0000","last_undegraded":"2024-09-13T17:08:11.591072+0000","last_fullsized":"2024-09-13T17:08:11.591072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.565301+0000","last_change":"2024-09-13T17:08:11.136035+0000","last_active":"2024-09-13T17:08:11.565301+0000","last_peered":"2024-09-13T17:08:11.565301+0000","last_clean":"2024-09-13T17:08:11.565301+0000","last_became_active":"2024-09-13T17:08:11.135742+0000","last_became_peered":"2024-09-13T17:08:11.135742+0000","last_unstale":"2024-09-13T17:08:11.565301+0000","last_undegraded":"2024-09-13T17:08:11.565301+0000","last_fullsized":"2024-09-13T17:08:11.565301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565763+0000","last_change":"2024-09-13T17:08:11.571193+0000","last_active":"2024-09-13T17:08:12.565763+0000","last_peered":"2024-09-13T17:08:12.565763+0000","last_clean":"2024-09-13T17:08:12.565763+0000","last_became_active":"2024-09-13T17:08:11.570542+0000","last_became_peered":"2024-09-13T17:08:11.570542+0000","last_unstale":"2024-09-13T17:08:12.565763+0000","last_undegraded":"2024-09-13T17:08:12.565763+0000","last_fullsized":"2024-09-13T17:08:12.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567567+0000","last_change":"2024-09-13T17:08:11.140490+0000","last_active":"2024-09-13T17:08:12.567567+0000","last_peered":"2024-09-13T17:08:12.567567+0000","last_clean":"2024-09-13T17:08:12.567567+0000","last_became_active":"2024-09-13T17:08:11.139782+0000","last_became_peered":"2024-09-13T17:08:11.139782+0000","last_unstale":"2024-09-13T17:08:12.567567+0000","last_undegraded":"2024-09-13T17:08:12.567567+0000","last_fullsized":"2024-09-13T17:08:12.567567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568964+0000","last_change":"2024-09-13T17:08:11.578078+0000","last_active":"2024-09-13T17:08:12.568964+0000","last_peered":"2024-09-13T17:08:12.568964+0000","last_clean":"2024-09-13T17:08:12.568964+0000","last_became_active":"2024-09-13T17:08:11.577379+0000","last_became_peered":"2024-09-13T17:08:11.577379+0000","last_unstale":"2024-09-13T17:08:12.568964+0000","last_undegraded":"2024-09-13T17:08:12.568964+0000","last_fullsized":"2024-09-13T17:08:12.568964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565979+0000","last_change":"2024-09-13T17:08:11.135658+0000","last_active":"2024-09-13T17:08:12.565979+0000","last_peered":"2024-09-13T17:08:12.565979+0000","last_clean":"2024-09-13T17:08:12.565979+0000","last_became_active":"2024-09-13T17:08:11.134906+0000","last_became_peered":"2024-09-13T17:08:11.134906+0000","last_unstale":"2024-09-13T17:08:12.565979+0000","last_undegraded":"2024-09-13T17:08:12.565979+0000","last_fullsized":"2024-09-13T17:08:12.565979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568073+0000","last_change":"2024-09-13T17:08:11.580253+0000","last_active":"2024-09-13T17:08:12.568073+0000","last_peered":"2024-09-13T17:08:12.568073+0000","last_clean":"2024-09-13T17:08:12.568073+0000","last_became_active":"2024-09-13T17:08:11.579547+0000","last_became_peered":"2024-09-13T17:08:11.579547+0000","last_unstale":"2024-09-13T17:08:12.568073+0000","last_undegraded":"2024-09-13T17:08:12.568073+0000","last_fullsized":"2024-09-13T17:08:12.568073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.576475+0000","last_change":"2024-09-13T17:08:11.149945+0000","last_active":"2024-09-13T17:08:11.576475+0000","last_peered":"2024-09-13T17:08:11.576475+0000","last_clean":"2024-09-13T17:08:11.576475+0000","last_became_active":"2024-09-13T17:08:11.149641+0000","last_became_peered":"2024-09-13T17:08:11.149641+0000","last_unstale":"2024-09-13T17:08:11.576475+0000","last_undegraded":"2024-09-13T17:08:11.576475+0000","last_fullsized":"2024-09-13T17:08:11.576475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565952+0000","last_change":"2024-09-13T17:08:11.578587+0000","last_active":"2024-09-13T17:08:12.565952+0000","last_peered":"2024-09-13T17:08:12.565952+0000","last_clean":"2024-09-13T17:08:12.565952+0000","last_became_active":"2024-09-13T17:08:11.576109+0000","last_became_peered":"2024-09-13T17:08:11.576109+0000","last_unstale":"2024-09-13T17:08:12.565952+0000","last_undegraded":"2024-09-13T17:08:12.565952+0000","last_fullsized":"2024-09-13T17:08:12.565952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566142+0000","last_change":"2024-09-13T17:08:11.141871+0000","last_active":"2024-09-13T17:08:12.566142+0000","last_peered":"2024-09-13T17:08:12.566142+0000","last_clean":"2024-09-13T17:08:12.566142+0000","last_became_active":"2024-09-13T17:08:11.141567+0000","last_became_peered":"2024-09-13T17:08:11.141567+0000","last_unstale":"2024-09-13T17:08:12.566142+0000","last_undegraded":"2024-09-13T17:08:12.566142+0000","last_fullsized":"2024-09-13T17:08:12.566142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591451+0000","last_change":"2024-09-13T17:08:11.589765+0000","last_active":"2024-09-13T17:08:11.591451+0000","last_peered":"2024-09-13T17:08:11.591451+0000","last_clean":"2024-09-13T17:08:11.591451+0000","last_became_active":"2024-09-13T17:08:11.589278+0000","last_became_peered":"2024-09-13T17:08:11.589278+0000","last_unstale":"2024-09-13T17:08:11.591451+0000","last_undegraded":"2024-09-13T17:08:11.591451+0000","last_fullsized":"2024-09-13T17:08:11.591451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.603021+0000","last_change":"2024-09-13T17:08:11.148235+0000","last_active":"2024-09-13T17:08:11.603021+0000","last_peered":"2024-09-13T17:08:11.603021+0000","last_clean":"2024-09-13T17:08:11.603021+0000","last_became_active":"2024-09-13T17:08:11.147994+0000","last_became_peered":"2024-09-13T17:08:11.147994+0000","last_unstale":"2024-09-13T17:08:11.603021+0000","last_undegraded":"2024-09-13T17:08:11.603021+0000","last_fullsized":"2024-09-13T17:08:11.603021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573772+0000","last_change":"2024-09-13T17:08:11.570444+0000","last_active":"2024-09-13T17:08:11.573772+0000","last_peered":"2024-09-13T17:08:11.573772+0000","last_clean":"2024-09-13T17:08:11.573772+0000","last_became_active":"2024-09-13T17:08:11.570207+0000","last_became_peered":"2024-09-13T17:08:11.570207+0000","last_unstale":"2024-09-13T17:08:11.573772+0000","last_undegraded":"2024-09-13T17:08:11.573772+0000","last_fullsized":"2024-09-13T17:08:11.573772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573767+0000","last_change":"2024-09-13T17:08:11.147785+0000","last_active":"2024-09-13T17:08:11.573767+0000","last_peered":"2024-09-13T17:08:11.573767+0000","last_clean":"2024-09-13T17:08:11.573767+0000","last_became_active":"2024-09-13T17:08:11.147222+0000","last_became_peered":"2024-09-13T17:08:11.147222+0000","last_unstale":"2024-09-13T17:08:11.573767+0000","last_undegraded":"2024-09-13T17:08:11.573767+0000","last_fullsized":"2024-09-13T17:08:11.573767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567547+0000","last_change":"2024-09-13T17:08:11.574148+0000","last_active":"2024-09-13T17:08:12.567547+0000","last_peered":"2024-09-13T17:08:12.567547+0000","last_clean":"2024-09-13T17:08:12.567547+0000","last_became_active":"2024-09-13T17:08:11.573879+0000","last_became_peered":"2024-09-13T17:08:11.573879+0000","last_unstale":"2024-09-13T17:08:12.567547+0000","last_undegraded":"2024-09-13T17:08:12.567547+0000","last_fullsized":"2024-09-13T17:08:12.567547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:10.871670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565831+0000","last_change":"2024-09-13T17:08:11.146000+0000","last_active":"2024-09-13T17:08:12.565831+0000","last_peered":"2024-09-13T17:08:12.565831+0000","last_clean":"2024-09-13T17:08:12.565831+0000","last_became_active":"2024-09-13T17:08:11.145714+0000","last_became_peered":"2024-09-13T17:08:11.145714+0000","last_unstale":"2024-09-13T17:08:12.565831+0000","last_undegraded":"2024-09-13T17:08:12.565831+0000","last_fullsized":"2024-09-13T17:08:12.565831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.580470+0000","last_change":"2024-09-13T17:08:11.573192+0000","last_active":"2024-09-13T17:08:11.580470+0000","last_peered":"2024-09-13T17:08:11.580470+0000","last_clean":"2024-09-13T17:08:11.580470+0000","last_became_active":"2024-09-13T17:08:11.572009+0000","last_became_peered":"2024-09-13T17:08:11.572009+0000","last_unstale":"2024-09-13T17:08:11.580470+0000","last_undegraded":"2024-09-13T17:08:11.580470+0000","last_fullsized":"2024-09-13T17:08:11.580470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565978+0000","last_change":"2024-09-13T17:08:11.145299+0000","last_active":"2024-09-13T17:08:12.565978+0000","last_peered":"2024-09-13T17:08:12.565978+0000","last_clean":"2024-09-13T17:08:12.565978+0000","last_became_active":"2024-09-13T17:08:11.143966+0000","last_became_peered":"2024-09-13T17:08:11.143966+0000","last_unstale":"2024-09-13T17:08:12.565978+0000","last_undegraded":"2024-09-13T17:08:12.565978+0000","last_fullsized":"2024-09-13T17:08:12.565978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.577272+0000","last_change":"2024-09-13T17:08:11.573641+0000","last_active":"2024-09-13T17:08:11.577272+0000","last_peered":"2024-09-13T17:08:11.577272+0000","last_clean":"2024-09-13T17:08:11.577272+0000","last_became_active":"2024-09-13T17:08:11.573434+0000","last_became_peered":"2024-09-13T17:08:11.573434+0000","last_unstale":"2024-09-13T17:08:11.577272+0000","last_undegraded":"2024-09-13T17:08:11.577272+0000","last_fullsized":"2024-09-13T17:08:11.577272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.602613+0000","last_change":"2024-09-13T17:08:11.148180+0000","last_active":"2024-09-13T17:08:11.602613+0000","last_peered":"2024-09-13T17:08:11.602613+0000","last_clean":"2024-09-13T17:08:11.602613+0000","last_became_active":"2024-09-13T17:08:11.147892+0000","last_became_peered":"2024-09-13T17:08:11.147892+0000","last_unstale":"2024-09-13T17:08:11.602613+0000","last_undegraded":"2024-09-13T17:08:11.602613+0000","last_fullsized":"2024-09-13T17:08:11.602613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593208+0000","last_change":"2024-09-13T17:08:11.589293+0000","last_active":"2024-09-13T17:08:11.593208+0000","last_peered":"2024-09-13T17:08:11.593208+0000","last_clean":"2024-09-13T17:08:11.593208+0000","last_became_active":"2024-09-13T17:08:11.589022+0000","last_became_peered":"2024-09-13T17:08:11.589022+0000","last_unstale":"2024-09-13T17:08:11.593208+0000","last_undegraded":"2024-09-13T17:08:11.593208+0000","last_fullsized":"2024-09-13T17:08:11.593208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:39.460619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568004+0000","last_change":"2024-09-13T17:08:11.594776+0000","last_active":"2024-09-13T17:08:12.568004+0000","last_peered":"2024-09-13T17:08:12.568004+0000","last_clean":"2024-09-13T17:08:12.568004+0000","last_became_active":"2024-09-13T17:08:11.594498+0000","last_became_peered":"2024-09-13T17:08:11.594498+0000","last_unstale":"2024-09-13T17:08:12.568004+0000","last_undegraded":"2024-09-13T17:08:12.568004+0000","last_fullsized":"2024-09-13T17:08:12.568004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.598145+0000","last_change":"2024-09-13T17:08:11.149488+0000","last_active":"2024-09-13T17:08:11.598145+0000","last_peered":"2024-09-13T17:08:11.598145+0000","last_clean":"2024-09-13T17:08:11.598145+0000","last_became_active":"2024-09-13T17:08:11.149198+0000","last_became_peered":"2024-09-13T17:08:11.149198+0000","last_unstale":"2024-09-13T17:08:11.598145+0000","last_undegraded":"2024-09-13T17:08:11.598145+0000","last_fullsized":"2024-09-13T17:08:11.598145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:24.611621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567674+0000","last_change":"2024-09-13T17:08:11.597049+0000","last_active":"2024-09-13T17:08:12.567674+0000","last_peered":"2024-09-13T17:08:12.567674+0000","last_clean":"2024-09-13T17:08:12.567674+0000","last_became_active":"2024-09-13T17:08:11.594306+0000","last_became_peered":"2024-09-13T17:08:11.594306+0000","last_unstale":"2024-09-13T17:08:12.567674+0000","last_undegraded":"2024-09-13T17:08:12.567674+0000","last_fullsized":"2024-09-13T17:08:12.567674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.600666+0000","last_change":"2024-09-13T17:08:11.152688+0000","last_active":"2024-09-13T17:08:11.600666+0000","last_peered":"2024-09-13T17:08:11.600666+0000","last_clean":"2024-09-13T17:08:11.600666+0000","last_became_active":"2024-09-13T17:08:11.152286+0000","last_became_peered":"2024-09-13T17:08:11.152286+0000","last_unstale":"2024-09-13T17:08:11.600666+0000","last_undegraded":"2024-09-13T17:08:11.600666+0000","last_fullsized":"2024-09-13T17:08:11.600666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591670+0000","last_change":"2024-09-13T17:08:11.588908+0000","last_active":"2024-09-13T17:08:11.591670+0000","last_peered":"2024-09-13T17:08:11.591670+0000","last_clean":"2024-09-13T17:08:11.591670+0000","last_became_active":"2024-09-13T17:08:11.588299+0000","last_became_peered":"2024-09-13T17:08:11.588299+0000","last_unstale":"2024-09-13T17:08:11.591670+0000","last_undegraded":"2024-09-13T17:08:11.591670+0000","last_fullsized":"2024-09-13T17:08:11.591670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591284+0000","last_change":"2024-09-13T17:08:11.140760+0000","last_active":"2024-09-13T17:08:11.591284+0000","last_peered":"2024-09-13T17:08:11.591284+0000","last_clean":"2024-09-13T17:08:11.591284+0000","last_became_active":"2024-09-13T17:08:11.140489+0000","last_became_peered":"2024-09-13T17:08:11.140489+0000","last_unstale":"2024-09-13T17:08:11.591284+0000","last_undegraded":"2024-09-13T17:08:11.591284+0000","last_fullsized":"2024-09-13T17:08:11.591284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593002+0000","last_change":"2024-09-13T17:08:11.591976+0000","last_active":"2024-09-13T17:08:11.593002+0000","last_peered":"2024-09-13T17:08:11.593002+0000","last_clean":"2024-09-13T17:08:11.593002+0000","last_became_active":"2024-09-13T17:08:11.591484+0000","last_became_peered":"2024-09-13T17:08:11.591484+0000","last_unstale":"2024-09-13T17:08:11.593002+0000","last_undegraded":"2024-09-13T17:08:11.593002+0000","last_fullsized":"2024-09-13T17:08:11.593002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.601650+0000","last_change":"2024-09-13T17:08:11.145298+0000","last_active":"2024-09-13T17:08:11.601650+0000","last_peered":"2024-09-13T17:08:11.601650+0000","last_clean":"2024-09-13T17:08:11.601650+0000","last_became_active":"2024-09-13T17:08:11.145029+0000","last_became_peered":"2024-09-13T17:08:11.145029+0000","last_unstale":"2024-09-13T17:08:11.601650+0000","last_undegraded":"2024-09-13T17:08:11.601650+0000","last_fullsized":"2024-09-13T17:08:11.601650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":655548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":204800,"data_stored":66162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":64758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":794624,"data_stored":654585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:08:15.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:08:15.876 INFO:tasks.ceph.mds.d.smithi189.stderr:2024-09-13T17:08:15.874+0000 7f8715200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:08:15.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.939+0000 7f438be00640 1 Processor -- start 2024-09-13T17:08:15.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.940+0000 7f438be00640 1 -- start start 2024-09-13T17:08:15.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f438c159440 0x7f438c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f438c159820 0x7f438c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f438c0c3850 con 0x7f438c0b22e0 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c15a930 con 0x7f438c159820 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f438c159d20 con 0x7f438c159440 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f4382000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f438c159820 0x7f438c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f438c0b22e0 0x7f438c0b2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34406/0 (socket says 172.21.15.28:34406) 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438b400640 1 -- 172.21.15.28:0/826773835 learned_addr learned my addr 172.21.15.28:0/826773835 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.941+0000 7f438aa00640 1 --2- 172.21.15.28:0/826773835 >> v2:172.21.15.189:3300/0 conn(0x7f438c159440 0x7f438c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.942+0000 7f4382000640 1 -- 172.21.15.28:0/826773835 >> v2:172.21.15.189:3300/0 conn(0x7f438c159440 msgr2=0x7f438c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.942+0000 7f4382000640 1 --2- 172.21.15.28:0/826773835 >> v2:172.21.15.189:3300/0 conn(0x7f438c159440 0x7f438c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.942+0000 7f4382000640 1 -- 172.21.15.28:0/826773835 >> v1:172.21.15.28:6789/0 conn(0x7f438c0b22e0 legacy=0x7f438c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.942+0000 7f438a000640 1 -- 172.21.15.28:0/826773835 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957720089 0 0) 0x7f438c0c3850 con 0x7f438c0b22e0 2024-09-13T17:08:15.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.942+0000 7f4382000640 1 -- 172.21.15.28:0/826773835 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f438c15a000 con 0x7f438c159820 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f4382000640 1 --2- 172.21.15.28:0/826773835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c159820 0x7f438c0b0980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43740097a0 tx=0x7f437402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579f32bc4afc8077 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f438a000640 1 -- 172.21.15.28:0/826773835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374032440 con 0x7f438c159820 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f438a000640 1 -- 172.21.15.28:0/826773835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43740325a0 con 0x7f438c159820 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f438a000640 1 -- 172.21.15.28:0/826773835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374032930 con 0x7f438c159820 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f438be00640 1 -- 172.21.15.28:0/826773835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c159820 msgr2=0x7f438c0b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.943+0000 7f438be00640 1 --2- 172.21.15.28:0/826773835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c159820 0x7f438c0b0980 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f43740097a0 tx=0x7f437402a270 comp rx=0 tx=0).stop 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 -- 172.21.15.28:0/826773835 shutdown_connections 2024-09-13T17:08:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 --2- 172.21.15.28:0/826773835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c159820 0x7f438c0b0980 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 --2- 172.21.15.28:0/826773835 >> v2:172.21.15.189:3300/0 conn(0x7f438c159440 0x7f438c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 -- 172.21.15.28:0/826773835 >> 172.21.15.28:0/826773835 conn(0x7f438c072e40 msgr2=0x7f438c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 -- 172.21.15.28:0/826773835 shutdown_connections 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.944+0000 7f438be00640 1 -- 172.21.15.28:0/826773835 wait complete. 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.945+0000 7f438be00640 1 Processor -- start 2024-09-13T17:08:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.945+0000 7f438be00640 1 -- start start 2024-09-13T17:08:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f438c159440 0x7f438c1506b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 0x7f438c1536f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c06f290 con 0x7f438c0b22e0 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438c165ae0 con 0x7f438c159820 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.946+0000 7f438be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f438c165dc0 con 0x7f438c159440 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33312/0 (socket says 172.21.15.28:33312) 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 -- 172.21.15.28:0/332278119 learned_addr learned my addr 172.21.15.28:0/332278119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438b400640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 0x7f438c1536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f4382000640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f438c159440 0x7f438c1506b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 -- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f438c159440 msgr2=0x7f438c1506b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f438c159440 0x7f438c1506b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 -- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 msgr2=0x7f438c1536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:15.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 0x7f438c1536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 -- 172.21.15.28:0/332278119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4374009400 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438b400640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 0x7f438c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f438aa00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f437c00d800 tx=0x7f437c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.947+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00b6c0 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.948+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c00b820 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.948+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437c00bbb0 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.948+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f438c1660a0 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.948+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438c15cce0 con 0x7f438c0b22e0 2024-09-13T17:08:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.949+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f438c166290 con 0x7f438c0b22e0 2024-09-13T17:08:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.950+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f437c00bd10 con 0x7f438c0b22e0 2024-09-13T17:08:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.950+0000 7f4383e00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 0x7f436806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.951+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f437c089900 con 0x7f438c0b22e0 2024-09-13T17:08:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.951+0000 7f4382000640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 0x7f436806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.952+0000 7f4382000640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 0x7f436806d0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4374034000 tx=0x7f437402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:15.955+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f437c054ae0 con 0x7f438c0b22e0 2024-09-13T17:08:16.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.130+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f438c165f30 con 0x7f436806ac00 2024-09-13T17:08:16.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.137+0000 7f4383e00640 1 -- 172.21.15.28:0/332278119 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264581 (secure 0 0 0) 0x7f438c165f30 con 0x7f436806ac00 2024-09-13T17:08:16.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:16.143 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 msgr2=0x7f436806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 0x7f436806d0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4374034000 tx=0x7f437402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 msgr2=0x7f438c150170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f437c00d800 tx=0x7f437c00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 shutdown_connections 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f436806ac00 0x7f436806d0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f438c159820 0x7f438c1536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f438c159440 0x7f438c1506b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 --2- 172.21.15.28:0/332278119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438c0b22e0 0x7f438c150170 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 >> 172.21.15.28:0/332278119 conn(0x7f438c072e40 msgr2=0x7f438c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 shutdown_connections 2024-09-13T17:08:16.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.141+0000 7f438be00640 1 -- 172.21.15.28:0/332278119 wait complete. 2024-09-13T17:08:16.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-13T17:08:14.884520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235292,"kb_used_data":2720,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754739428,"statfs":{"total":773094113280,"available":772853174272,"internally_reserved":0,"allocated":2785280,"data_stored":1700206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.773919"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606164+0000","last_change":"2024-09-13T17:08:11.601843+0000","last_active":"2024-09-13T17:08:11.606164+0000","last_peered":"2024-09-13T17:08:11.606164+0000","last_clean":"2024-09-13T17:08:11.606164+0000","last_became_active":"2024-09-13T17:08:11.601437+0000","last_became_peered":"2024-09-13T17:08:11.601437+0000","last_unstale":"2024-09-13T17:08:11.606164+0000","last_undegraded":"2024-09-13T17:08:11.606164+0000","last_fullsized":"2024-09-13T17:08:11.606164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:28.411961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566115+0000","last_change":"2024-09-13T17:08:11.576946+0000","last_active":"2024-09-13T17:08:12.566115+0000","last_peered":"2024-09-13T17:08:12.566115+0000","last_clean":"2024-09-13T17:08:12.566115+0000","last_became_active":"2024-09-13T17:08:11.576700+0000","last_became_peered":"2024-09-13T17:08:11.576700+0000","last_unstale":"2024-09-13T17:08:12.566115+0000","last_undegraded":"2024-09-13T17:08:12.566115+0000","last_fullsized":"2024-09-13T17:08:12.566115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.580583+0000","last_change":"2024-09-13T17:08:11.576052+0000","last_active":"2024-09-13T17:08:11.580583+0000","last_peered":"2024-09-13T17:08:11.580583+0000","last_clean":"2024-09-13T17:08:11.580583+0000","last_became_active":"2024-09-13T17:08:11.575297+0000","last_became_peered":"2024-09-13T17:08:11.575297+0000","last_unstale":"2024-09-13T17:08:11.580583+0000","last_undegraded":"2024-09-13T17:08:11.580583+0000","last_fullsized":"2024-09-13T17:08:11.580583+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567600+0000","last_change":"2024-09-13T17:08:11.582113+0000","last_active":"2024-09-13T17:08:12.567600+0000","last_peered":"2024-09-13T17:08:12.567600+0000","last_clean":"2024-09-13T17:08:12.567600+0000","last_became_active":"2024-09-13T17:08:11.579185+0000","last_became_peered":"2024-09-13T17:08:11.579185+0000","last_unstale":"2024-09-13T17:08:12.567600+0000","last_undegraded":"2024-09-13T17:08:12.567600+0000","last_fullsized":"2024-09-13T17:08:12.567600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606007+0000","last_change":"2024-09-13T17:08:11.600241+0000","last_active":"2024-09-13T17:08:11.606007+0000","last_peered":"2024-09-13T17:08:11.606007+0000","last_clean":"2024-09-13T17:08:11.606007+0000","last_became_active":"2024-09-13T17:08:11.596443+0000","last_became_peered":"2024-09-13T17:08:11.596443+0000","last_unstale":"2024-09-13T17:08:11.606007+0000","last_undegraded":"2024-09-13T17:08:11.606007+0000","last_fullsized":"2024-09-13T17:08:11.606007+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567274+0000","last_change":"2024-09-13T17:08:11.583211+0000","last_active":"2024-09-13T17:08:12.567274+0000","last_peered":"2024-09-13T17:08:12.567274+0000","last_clean":"2024-09-13T17:08:12.567274+0000","last_became_active":"2024-09-13T17:08:11.583040+0000","last_became_peered":"2024-09-13T17:08:11.583040+0000","last_unstale":"2024-09-13T17:08:12.567274+0000","last_undegraded":"2024-09-13T17:08:12.567274+0000","last_fullsized":"2024-09-13T17:08:12.567274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596334+0000","last_change":"2024-09-13T17:08:11.592666+0000","last_active":"2024-09-13T17:08:11.596334+0000","last_peered":"2024-09-13T17:08:11.596334+0000","last_clean":"2024-09-13T17:08:11.596334+0000","last_became_active":"2024-09-13T17:08:11.592373+0000","last_became_peered":"2024-09-13T17:08:11.592373+0000","last_unstale":"2024-09-13T17:08:11.596334+0000","last_undegraded":"2024-09-13T17:08:11.596334+0000","last_fullsized":"2024-09-13T17:08:11.596334+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.583252+0000","last_change":"2024-09-13T17:08:11.578631+0000","last_active":"2024-09-13T17:08:11.583252+0000","last_peered":"2024-09-13T17:08:11.583252+0000","last_clean":"2024-09-13T17:08:11.583252+0000","last_became_active":"2024-09-13T17:08:11.575801+0000","last_became_peered":"2024-09-13T17:08:11.575801+0000","last_unstale":"2024-09-13T17:08:11.583252+0000","last_undegraded":"2024-09-13T17:08:11.583252+0000","last_fullsized":"2024-09-13T17:08:11.583252+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606141+0000","last_change":"2024-09-13T17:08:11.601554+0000","last_active":"2024-09-13T17:08:11.606141+0000","last_peered":"2024-09-13T17:08:11.606141+0000","last_clean":"2024-09-13T17:08:11.606141+0000","last_became_active":"2024-09-13T17:08:11.601069+0000","last_became_peered":"2024-09-13T17:08:11.601069+0000","last_unstale":"2024-09-13T17:08:11.606141+0000","last_undegraded":"2024-09-13T17:08:11.606141+0000","last_fullsized":"2024-09-13T17:08:11.606141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:24.360659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593128+0000","last_change":"2024-09-13T17:08:11.587154+0000","last_active":"2024-09-13T17:08:11.593128+0000","last_peered":"2024-09-13T17:08:11.593128+0000","last_clean":"2024-09-13T17:08:11.593128+0000","last_became_active":"2024-09-13T17:08:11.586876+0000","last_became_peered":"2024-09-13T17:08:11.586876+0000","last_unstale":"2024-09-13T17:08:11.593128+0000","last_undegraded":"2024-09-13T17:08:11.593128+0000","last_fullsized":"2024-09-13T17:08:11.593128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591336+0000","last_change":"2024-09-13T17:08:11.586451+0000","last_active":"2024-09-13T17:08:11.591336+0000","last_peered":"2024-09-13T17:08:11.591336+0000","last_clean":"2024-09-13T17:08:11.591336+0000","last_became_active":"2024-09-13T17:08:11.586127+0000","last_became_peered":"2024-09-13T17:08:11.586127+0000","last_unstale":"2024-09-13T17:08:11.591336+0000","last_undegraded":"2024-09-13T17:08:11.591336+0000","last_fullsized":"2024-09-13T17:08:11.591336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569157+0000","last_change":"2024-09-13T17:08:11.591207+0000","last_active":"2024-09-13T17:08:12.569157+0000","last_peered":"2024-09-13T17:08:12.569157+0000","last_clean":"2024-09-13T17:08:12.569157+0000","last_became_active":"2024-09-13T17:08:11.590920+0000","last_became_peered":"2024-09-13T17:08:11.590920+0000","last_unstale":"2024-09-13T17:08:12.569157+0000","last_undegraded":"2024-09-13T17:08:12.569157+0000","last_fullsized":"2024-09-13T17:08:12.569157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:25.618116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591159+0000","last_change":"2024-09-13T17:08:11.586067+0000","last_active":"2024-09-13T17:08:11.591159+0000","last_peered":"2024-09-13T17:08:11.591159+0000","last_clean":"2024-09-13T17:08:11.591159+0000","last_became_active":"2024-09-13T17:08:11.585787+0000","last_became_peered":"2024-09-13T17:08:11.585787+0000","last_unstale":"2024-09-13T17:08:11.591159+0000","last_undegraded":"2024-09-13T17:08:11.591159+0000","last_fullsized":"2024-09-13T17:08:11.591159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572983+0000","last_change":"2024-09-13T17:08:11.570021+0000","last_active":"2024-09-13T17:08:11.572983+0000","last_peered":"2024-09-13T17:08:11.572983+0000","last_clean":"2024-09-13T17:08:11.572983+0000","last_became_active":"2024-09-13T17:08:11.569663+0000","last_became_peered":"2024-09-13T17:08:11.569663+0000","last_unstale":"2024-09-13T17:08:11.572983+0000","last_undegraded":"2024-09-13T17:08:11.572983+0000","last_fullsized":"2024-09-13T17:08:11.572983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565777+0000","last_change":"2024-09-13T17:08:11.578244+0000","last_active":"2024-09-13T17:08:12.565777+0000","last_peered":"2024-09-13T17:08:12.565777+0000","last_clean":"2024-09-13T17:08:12.565777+0000","last_became_active":"2024-09-13T17:08:11.577989+0000","last_became_peered":"2024-09-13T17:08:11.577989+0000","last_unstale":"2024-09-13T17:08:12.565777+0000","last_undegraded":"2024-09-13T17:08:12.565777+0000","last_fullsized":"2024-09-13T17:08:12.565777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567131+0000","last_change":"2024-09-13T17:08:11.582930+0000","last_active":"2024-09-13T17:08:12.567131+0000","last_peered":"2024-09-13T17:08:12.567131+0000","last_clean":"2024-09-13T17:08:12.567131+0000","last_became_active":"2024-09-13T17:08:11.582621+0000","last_became_peered":"2024-09-13T17:08:11.582621+0000","last_unstale":"2024-09-13T17:08:12.567131+0000","last_undegraded":"2024-09-13T17:08:12.567131+0000","last_fullsized":"2024-09-13T17:08:12.567131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568144+0000","last_change":"2024-09-13T17:08:11.601899+0000","last_active":"2024-09-13T17:08:12.568144+0000","last_peered":"2024-09-13T17:08:12.568144+0000","last_clean":"2024-09-13T17:08:12.568144+0000","last_became_active":"2024-09-13T17:08:11.601452+0000","last_became_peered":"2024-09-13T17:08:11.601452+0000","last_unstale":"2024-09-13T17:08:12.568144+0000","last_undegraded":"2024-09-13T17:08:12.568144+0000","last_fullsized":"2024-09-13T17:08:12.568144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593019+0000","last_change":"2024-09-13T17:08:11.591991+0000","last_active":"2024-09-13T17:08:11.593019+0000","last_peered":"2024-09-13T17:08:11.593019+0000","last_clean":"2024-09-13T17:08:11.593019+0000","last_became_active":"2024-09-13T17:08:11.591554+0000","last_became_peered":"2024-09-13T17:08:11.591554+0000","last_unstale":"2024-09-13T17:08:11.593019+0000","last_undegraded":"2024-09-13T17:08:11.593019+0000","last_fullsized":"2024-09-13T17:08:11.593019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.599818+0000","last_change":"2024-09-13T17:08:11.596861+0000","last_active":"2024-09-13T17:08:11.599818+0000","last_peered":"2024-09-13T17:08:11.599818+0000","last_clean":"2024-09-13T17:08:11.599818+0000","last_became_active":"2024-09-13T17:08:11.596669+0000","last_became_peered":"2024-09-13T17:08:11.596669+0000","last_unstale":"2024-09-13T17:08:11.599818+0000","last_undegraded":"2024-09-13T17:08:11.599818+0000","last_fullsized":"2024-09-13T17:08:11.599818+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591158+0000","last_change":"2024-09-13T17:08:11.582225+0000","last_active":"2024-09-13T17:08:11.591158+0000","last_peered":"2024-09-13T17:08:11.591158+0000","last_clean":"2024-09-13T17:08:11.591158+0000","last_became_active":"2024-09-13T17:08:11.581900+0000","last_became_peered":"2024-09-13T17:08:11.581900+0000","last_unstale":"2024-09-13T17:08:11.591158+0000","last_undegraded":"2024-09-13T17:08:11.591158+0000","last_fullsized":"2024-09-13T17:08:11.591158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591212+0000","last_change":"2024-09-13T17:08:11.588070+0000","last_active":"2024-09-13T17:08:11.591212+0000","last_peered":"2024-09-13T17:08:11.591212+0000","last_clean":"2024-09-13T17:08:11.591212+0000","last_became_active":"2024-09-13T17:08:11.587859+0000","last_became_peered":"2024-09-13T17:08:11.587859+0000","last_unstale":"2024-09-13T17:08:11.591212+0000","last_undegraded":"2024-09-13T17:08:11.591212+0000","last_fullsized":"2024-09-13T17:08:11.591212+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567322+0000","last_change":"2024-09-13T17:08:11.597424+0000","last_active":"2024-09-13T17:08:12.567322+0000","last_peered":"2024-09-13T17:08:12.567322+0000","last_clean":"2024-09-13T17:08:12.567322+0000","last_became_active":"2024-09-13T17:08:11.597093+0000","last_became_peered":"2024-09-13T17:08:11.597093+0000","last_unstale":"2024-09-13T17:08:12.567322+0000","last_undegraded":"2024-09-13T17:08:12.567322+0000","last_fullsized":"2024-09-13T17:08:12.567322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.585950+0000","last_change":"2024-09-13T17:08:11.581514+0000","last_active":"2024-09-13T17:08:11.585950+0000","last_peered":"2024-09-13T17:08:11.585950+0000","last_clean":"2024-09-13T17:08:11.585950+0000","last_became_active":"2024-09-13T17:08:11.581178+0000","last_became_peered":"2024-09-13T17:08:11.581178+0000","last_unstale":"2024-09-13T17:08:11.585950+0000","last_undegraded":"2024-09-13T17:08:11.585950+0000","last_fullsized":"2024-09-13T17:08:11.585950+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567215+0000","last_change":"2024-09-13T17:08:11.575094+0000","last_active":"2024-09-13T17:08:12.567215+0000","last_peered":"2024-09-13T17:08:12.567215+0000","last_clean":"2024-09-13T17:08:12.567215+0000","last_became_active":"2024-09-13T17:08:11.574833+0000","last_became_peered":"2024-09-13T17:08:11.574833+0000","last_unstale":"2024-09-13T17:08:12.567215+0000","last_undegraded":"2024-09-13T17:08:12.567215+0000","last_fullsized":"2024-09-13T17:08:12.567215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.586697+0000","last_change":"2024-09-13T17:08:11.581636+0000","last_active":"2024-09-13T17:08:11.586697+0000","last_peered":"2024-09-13T17:08:11.586697+0000","last_clean":"2024-09-13T17:08:11.586697+0000","last_became_active":"2024-09-13T17:08:11.581264+0000","last_became_peered":"2024-09-13T17:08:11.581264+0000","last_unstale":"2024-09-13T17:08:11.586697+0000","last_undegraded":"2024-09-13T17:08:11.586697+0000","last_fullsized":"2024-09-13T17:08:11.586697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.603877+0000","last_change":"2024-09-13T17:08:11.601141+0000","last_active":"2024-09-13T17:08:11.603877+0000","last_peered":"2024-09-13T17:08:11.603877+0000","last_clean":"2024-09-13T17:08:11.603877+0000","last_became_active":"2024-09-13T17:08:11.600877+0000","last_became_peered":"2024-09-13T17:08:11.600877+0000","last_unstale":"2024-09-13T17:08:11.603877+0000","last_undegraded":"2024-09-13T17:08:11.603877+0000","last_fullsized":"2024-09-13T17:08:11.603877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565968+0000","last_change":"2024-09-13T17:08:11.576209+0000","last_active":"2024-09-13T17:08:12.565968+0000","last_peered":"2024-09-13T17:08:12.565968+0000","last_clean":"2024-09-13T17:08:12.565968+0000","last_became_active":"2024-09-13T17:08:11.575917+0000","last_became_peered":"2024-09-13T17:08:11.575917+0000","last_unstale":"2024-09-13T17:08:12.565968+0000","last_undegraded":"2024-09-13T17:08:12.565968+0000","last_fullsized":"2024-09-13T17:08:12.565968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.590314+0000","last_change":"2024-09-13T17:08:11.582222+0000","last_active":"2024-09-13T17:08:11.590314+0000","last_peered":"2024-09-13T17:08:11.590314+0000","last_clean":"2024-09-13T17:08:11.590314+0000","last_became_active":"2024-09-13T17:08:11.582057+0000","last_became_peered":"2024-09-13T17:08:11.582057+0000","last_unstale":"2024-09-13T17:08:11.590314+0000","last_undegraded":"2024-09-13T17:08:11.590314+0000","last_fullsized":"2024-09-13T17:08:11.590314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.582927+0000","last_change":"2024-09-13T17:08:11.575089+0000","last_active":"2024-09-13T17:08:11.582927+0000","last_peered":"2024-09-13T17:08:11.582927+0000","last_clean":"2024-09-13T17:08:11.582927+0000","last_became_active":"2024-09-13T17:08:11.574665+0000","last_became_peered":"2024-09-13T17:08:11.574665+0000","last_unstale":"2024-09-13T17:08:11.582927+0000","last_undegraded":"2024-09-13T17:08:11.582927+0000","last_fullsized":"2024-09-13T17:08:11.582927+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567404+0000","last_change":"2024-09-13T17:08:11.579735+0000","last_active":"2024-09-13T17:08:12.567404+0000","last_peered":"2024-09-13T17:08:12.567404+0000","last_clean":"2024-09-13T17:08:12.567404+0000","last_became_active":"2024-09-13T17:08:11.578749+0000","last_became_peered":"2024-09-13T17:08:11.578749+0000","last_unstale":"2024-09-13T17:08:12.567404+0000","last_undegraded":"2024-09-13T17:08:12.567404+0000","last_fullsized":"2024-09-13T17:08:12.567404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572297+0000","last_change":"2024-09-13T17:08:11.570273+0000","last_active":"2024-09-13T17:08:11.572297+0000","last_peered":"2024-09-13T17:08:11.572297+0000","last_clean":"2024-09-13T17:08:11.572297+0000","last_became_active":"2024-09-13T17:08:11.569987+0000","last_became_peered":"2024-09-13T17:08:11.569987+0000","last_unstale":"2024-09-13T17:08:11.572297+0000","last_undegraded":"2024-09-13T17:08:11.572297+0000","last_fullsized":"2024-09-13T17:08:11.572297+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591341+0000","last_change":"2024-09-13T17:08:11.586071+0000","last_active":"2024-09-13T17:08:11.591341+0000","last_peered":"2024-09-13T17:08:11.591341+0000","last_clean":"2024-09-13T17:08:11.591341+0000","last_became_active":"2024-09-13T17:08:11.585507+0000","last_became_peered":"2024-09-13T17:08:11.585507+0000","last_unstale":"2024-09-13T17:08:11.591341+0000","last_undegraded":"2024-09-13T17:08:11.591341+0000","last_fullsized":"2024-09-13T17:08:11.591341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567646+0000","last_change":"2024-09-13T17:08:11.578066+0000","last_active":"2024-09-13T17:08:12.567646+0000","last_peered":"2024-09-13T17:08:12.567646+0000","last_clean":"2024-09-13T17:08:12.567646+0000","last_became_active":"2024-09-13T17:08:11.577655+0000","last_became_peered":"2024-09-13T17:08:11.577655+0000","last_unstale":"2024-09-13T17:08:12.567646+0000","last_undegraded":"2024-09-13T17:08:12.567646+0000","last_fullsized":"2024-09-13T17:08:12.567646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.318727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.601896+0000","last_change":"2024-09-13T17:08:11.147841+0000","last_active":"2024-09-13T17:08:11.601896+0000","last_peered":"2024-09-13T17:08:11.601896+0000","last_clean":"2024-09-13T17:08:11.601896+0000","last_became_active":"2024-09-13T17:08:11.147449+0000","last_became_peered":"2024-09-13T17:08:11.147449+0000","last_unstale":"2024-09-13T17:08:11.601896+0000","last_undegraded":"2024-09-13T17:08:11.601896+0000","last_fullsized":"2024-09-13T17:08:11.601896+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.582944+0000","last_change":"2024-09-13T17:08:11.580301+0000","last_active":"2024-09-13T17:08:11.582944+0000","last_peered":"2024-09-13T17:08:11.582944+0000","last_clean":"2024-09-13T17:08:11.582944+0000","last_became_active":"2024-09-13T17:08:11.579029+0000","last_became_peered":"2024-09-13T17:08:11.579029+0000","last_unstale":"2024-09-13T17:08:11.582944+0000","last_undegraded":"2024-09-13T17:08:11.582944+0000","last_fullsized":"2024-09-13T17:08:11.582944+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:30.800655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.598106+0000","last_change":"2024-09-13T17:08:11.146319+0000","last_active":"2024-09-13T17:08:11.598106+0000","last_peered":"2024-09-13T17:08:11.598106+0000","last_clean":"2024-09-13T17:08:11.598106+0000","last_became_active":"2024-09-13T17:08:11.146057+0000","last_became_peered":"2024-09-13T17:08:11.146057+0000","last_unstale":"2024-09-13T17:08:11.598106+0000","last_undegraded":"2024-09-13T17:08:11.598106+0000","last_fullsized":"2024-09-13T17:08:11.598106+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.572346+0000","last_change":"2024-09-13T17:08:11.138543+0000","last_active":"2024-09-13T17:08:11.572346+0000","last_peered":"2024-09-13T17:08:11.572346+0000","last_clean":"2024-09-13T17:08:11.572346+0000","last_became_active":"2024-09-13T17:08:11.133550+0000","last_became_peered":"2024-09-13T17:08:11.133550+0000","last_unstale":"2024-09-13T17:08:11.572346+0000","last_undegraded":"2024-09-13T17:08:11.572346+0000","last_fullsized":"2024-09-13T17:08:11.572346+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.584696+0000","last_change":"2024-09-13T17:08:11.580334+0000","last_active":"2024-09-13T17:08:11.584696+0000","last_peered":"2024-09-13T17:08:11.584696+0000","last_clean":"2024-09-13T17:08:11.584696+0000","last_became_active":"2024-09-13T17:08:11.574885+0000","last_became_peered":"2024-09-13T17:08:11.574885+0000","last_unstale":"2024-09-13T17:08:11.584696+0000","last_undegraded":"2024-09-13T17:08:11.584696+0000","last_fullsized":"2024-09-13T17:08:11.584696+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567242+0000","last_change":"2024-09-13T17:08:11.142227+0000","last_active":"2024-09-13T17:08:12.567242+0000","last_peered":"2024-09-13T17:08:12.567242+0000","last_clean":"2024-09-13T17:08:12.567242+0000","last_became_active":"2024-09-13T17:08:11.142009+0000","last_became_peered":"2024-09-13T17:08:11.142009+0000","last_unstale":"2024-09-13T17:08:12.567242+0000","last_undegraded":"2024-09-13T17:08:12.567242+0000","last_fullsized":"2024-09-13T17:08:12.567242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565757+0000","last_change":"2024-09-13T17:08:11.569231+0000","last_active":"2024-09-13T17:08:12.565757+0000","last_peered":"2024-09-13T17:08:12.565757+0000","last_clean":"2024-09-13T17:08:12.565757+0000","last_became_active":"2024-09-13T17:08:11.568934+0000","last_became_peered":"2024-09-13T17:08:11.568934+0000","last_unstale":"2024-09-13T17:08:12.565757+0000","last_undegraded":"2024-09-13T17:08:12.565757+0000","last_fullsized":"2024-09-13T17:08:12.565757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.578842+0000","last_change":"2024-09-13T17:08:11.140675+0000","last_active":"2024-09-13T17:08:11.578842+0000","last_peered":"2024-09-13T17:08:11.578842+0000","last_clean":"2024-09-13T17:08:11.578842+0000","last_became_active":"2024-09-13T17:08:11.139918+0000","last_became_peered":"2024-09-13T17:08:11.139918+0000","last_unstale":"2024-09-13T17:08:11.578842+0000","last_undegraded":"2024-09-13T17:08:11.578842+0000","last_fullsized":"2024-09-13T17:08:11.578842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568038+0000","last_change":"2024-09-13T17:08:11.599453+0000","last_active":"2024-09-13T17:08:12.568038+0000","last_peered":"2024-09-13T17:08:12.568038+0000","last_clean":"2024-09-13T17:08:12.568038+0000","last_became_active":"2024-09-13T17:08:11.599107+0000","last_became_peered":"2024-09-13T17:08:11.599107+0000","last_unstale":"2024-09-13T17:08:12.568038+0000","last_undegraded":"2024-09-13T17:08:12.568038+0000","last_fullsized":"2024-09-13T17:08:12.568038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568117+0000","last_change":"2024-09-13T17:08:11.136652+0000","last_active":"2024-09-13T17:08:12.568117+0000","last_peered":"2024-09-13T17:08:12.568117+0000","last_clean":"2024-09-13T17:08:12.568117+0000","last_became_active":"2024-09-13T17:08:11.135676+0000","last_became_peered":"2024-09-13T17:08:11.135676+0000","last_unstale":"2024-09-13T17:08:12.568117+0000","last_undegraded":"2024-09-13T17:08:12.568117+0000","last_fullsized":"2024-09-13T17:08:12.568117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591687+0000","last_change":"2024-09-13T17:08:11.589078+0000","last_active":"2024-09-13T17:08:11.591687+0000","last_peered":"2024-09-13T17:08:11.591687+0000","last_clean":"2024-09-13T17:08:11.591687+0000","last_became_active":"2024-09-13T17:08:11.588613+0000","last_became_peered":"2024-09-13T17:08:11.588613+0000","last_unstale":"2024-09-13T17:08:11.591687+0000","last_undegraded":"2024-09-13T17:08:11.591687+0000","last_fullsized":"2024-09-13T17:08:11.591687+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565993+0000","last_change":"2024-09-13T17:08:11.139774+0000","last_active":"2024-09-13T17:08:12.565993+0000","last_peered":"2024-09-13T17:08:12.565993+0000","last_clean":"2024-09-13T17:08:12.565993+0000","last_became_active":"2024-09-13T17:08:11.139257+0000","last_became_peered":"2024-09-13T17:08:11.139257+0000","last_unstale":"2024-09-13T17:08:12.565993+0000","last_undegraded":"2024-09-13T17:08:12.565993+0000","last_fullsized":"2024-09-13T17:08:12.565993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569193+0000","last_change":"2024-09-13T17:08:11.590978+0000","last_active":"2024-09-13T17:08:12.569193+0000","last_peered":"2024-09-13T17:08:12.569193+0000","last_clean":"2024-09-13T17:08:12.569193+0000","last_became_active":"2024-09-13T17:08:11.590763+0000","last_became_peered":"2024-09-13T17:08:11.590763+0000","last_unstale":"2024-09-13T17:08:12.569193+0000","last_undegraded":"2024-09-13T17:08:12.569193+0000","last_fullsized":"2024-09-13T17:08:12.569193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.608756+0000","last_change":"2024-09-13T17:08:11.134868+0000","last_active":"2024-09-13T17:08:11.608756+0000","last_peered":"2024-09-13T17:08:11.608756+0000","last_clean":"2024-09-13T17:08:11.608756+0000","last_became_active":"2024-09-13T17:08:11.134516+0000","last_became_peered":"2024-09-13T17:08:11.134516+0000","last_unstale":"2024-09-13T17:08:11.608756+0000","last_undegraded":"2024-09-13T17:08:11.608756+0000","last_fullsized":"2024-09-13T17:08:11.608756+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596787+0000","last_change":"2024-09-13T17:08:11.587369+0000","last_active":"2024-09-13T17:08:11.596787+0000","last_peered":"2024-09-13T17:08:11.596787+0000","last_clean":"2024-09-13T17:08:11.596787+0000","last_became_active":"2024-09-13T17:08:11.586043+0000","last_became_peered":"2024-09-13T17:08:11.586043+0000","last_unstale":"2024-09-13T17:08:11.596787+0000","last_undegraded":"2024-09-13T17:08:11.596787+0000","last_fullsized":"2024-09-13T17:08:11.596787+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.597601+0000","last_change":"2024-09-13T17:08:11.145171+0000","last_active":"2024-09-13T17:08:11.597601+0000","last_peered":"2024-09-13T17:08:11.597601+0000","last_clean":"2024-09-13T17:08:11.597601+0000","last_became_active":"2024-09-13T17:08:11.144862+0000","last_became_peered":"2024-09-13T17:08:11.144862+0000","last_unstale":"2024-09-13T17:08:11.597601+0000","last_undegraded":"2024-09-13T17:08:11.597601+0000","last_fullsized":"2024-09-13T17:08:11.597601+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596904+0000","last_change":"2024-09-13T17:08:11.596289+0000","last_active":"2024-09-13T17:08:11.596904+0000","last_peered":"2024-09-13T17:08:11.596904+0000","last_clean":"2024-09-13T17:08:11.596904+0000","last_became_active":"2024-09-13T17:08:11.595983+0000","last_became_peered":"2024-09-13T17:08:11.595983+0000","last_unstale":"2024-09-13T17:08:11.596904+0000","last_undegraded":"2024-09-13T17:08:11.596904+0000","last_fullsized":"2024-09-13T17:08:11.596904+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:39.915077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567483+0000","last_change":"2024-09-13T17:08:11.143981+0000","last_active":"2024-09-13T17:08:12.567483+0000","last_peered":"2024-09-13T17:08:12.567483+0000","last_clean":"2024-09-13T17:08:12.567483+0000","last_became_active":"2024-09-13T17:08:11.143705+0000","last_became_peered":"2024-09-13T17:08:11.143705+0000","last_unstale":"2024-09-13T17:08:12.567483+0000","last_undegraded":"2024-09-13T17:08:12.567483+0000","last_fullsized":"2024-09-13T17:08:12.567483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606067+0000","last_change":"2024-09-13T17:08:11.601054+0000","last_active":"2024-09-13T17:08:11.606067+0000","last_peered":"2024-09-13T17:08:11.606067+0000","last_clean":"2024-09-13T17:08:11.606067+0000","last_became_active":"2024-09-13T17:08:11.600160+0000","last_became_peered":"2024-09-13T17:08:11.600160+0000","last_unstale":"2024-09-13T17:08:11.606067+0000","last_undegraded":"2024-09-13T17:08:11.606067+0000","last_fullsized":"2024-09-13T17:08:11.606067+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.569969+0000","last_change":"2024-09-13T17:08:11.145590+0000","last_active":"2024-09-13T17:08:11.569969+0000","last_peered":"2024-09-13T17:08:11.569969+0000","last_clean":"2024-09-13T17:08:11.569969+0000","last_became_active":"2024-09-13T17:08:11.145267+0000","last_became_peered":"2024-09-13T17:08:11.145267+0000","last_unstale":"2024-09-13T17:08:11.569969+0000","last_undegraded":"2024-09-13T17:08:11.569969+0000","last_fullsized":"2024-09-13T17:08:11.569969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591322+0000","last_change":"2024-09-13T17:08:11.584806+0000","last_active":"2024-09-13T17:08:11.591322+0000","last_peered":"2024-09-13T17:08:11.591322+0000","last_clean":"2024-09-13T17:08:11.591322+0000","last_became_active":"2024-09-13T17:08:11.584303+0000","last_became_peered":"2024-09-13T17:08:11.584303+0000","last_unstale":"2024-09-13T17:08:11.591322+0000","last_undegraded":"2024-09-13T17:08:11.591322+0000","last_fullsized":"2024-09-13T17:08:11.591322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.589937+0000","last_change":"2024-09-13T17:08:11.140195+0000","last_active":"2024-09-13T17:08:11.589937+0000","last_peered":"2024-09-13T17:08:11.589937+0000","last_clean":"2024-09-13T17:08:11.589937+0000","last_became_active":"2024-09-13T17:08:11.139870+0000","last_became_peered":"2024-09-13T17:08:11.139870+0000","last_unstale":"2024-09-13T17:08:11.589937+0000","last_undegraded":"2024-09-13T17:08:11.589937+0000","last_fullsized":"2024-09-13T17:08:11.589937+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596116+0000","last_change":"2024-09-13T17:08:11.588867+0000","last_active":"2024-09-13T17:08:11.596116+0000","last_peered":"2024-09-13T17:08:11.596116+0000","last_clean":"2024-09-13T17:08:11.596116+0000","last_became_active":"2024-09-13T17:08:11.588564+0000","last_became_peered":"2024-09-13T17:08:11.588564+0000","last_unstale":"2024-09-13T17:08:11.596116+0000","last_undegraded":"2024-09-13T17:08:11.596116+0000","last_fullsized":"2024-09-13T17:08:11.596116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569202+0000","last_change":"2024-09-13T17:08:11.138154+0000","last_active":"2024-09-13T17:08:12.569202+0000","last_peered":"2024-09-13T17:08:12.569202+0000","last_clean":"2024-09-13T17:08:12.569202+0000","last_became_active":"2024-09-13T17:08:11.137611+0000","last_became_peered":"2024-09-13T17:08:11.137611+0000","last_unstale":"2024-09-13T17:08:12.569202+0000","last_undegraded":"2024-09-13T17:08:12.569202+0000","last_fullsized":"2024-09-13T17:08:12.569202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:48.182859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.608822+0000","last_change":"2024-09-13T17:08:11.604320+0000","last_active":"2024-09-13T17:08:11.608822+0000","last_peered":"2024-09-13T17:08:11.608822+0000","last_clean":"2024-09-13T17:08:11.608822+0000","last_became_active":"2024-09-13T17:08:11.603337+0000","last_became_peered":"2024-09-13T17:08:11.603337+0000","last_unstale":"2024-09-13T17:08:11.608822+0000","last_undegraded":"2024-09-13T17:08:11.608822+0000","last_fullsized":"2024-09-13T17:08:11.608822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.590544+0000","last_change":"2024-09-13T17:08:11.138593+0000","last_active":"2024-09-13T17:08:11.590544+0000","last_peered":"2024-09-13T17:08:11.590544+0000","last_clean":"2024-09-13T17:08:11.590544+0000","last_became_active":"2024-09-13T17:08:11.138131+0000","last_became_peered":"2024-09-13T17:08:11.138131+0000","last_unstale":"2024-09-13T17:08:11.590544+0000","last_undegraded":"2024-09-13T17:08:11.590544+0000","last_fullsized":"2024-09-13T17:08:11.590544+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.606181+0000","last_change":"2024-09-13T17:08:11.602024+0000","last_active":"2024-09-13T17:08:11.606181+0000","last_peered":"2024-09-13T17:08:11.606181+0000","last_clean":"2024-09-13T17:08:11.606181+0000","last_became_active":"2024-09-13T17:08:11.601178+0000","last_became_peered":"2024-09-13T17:08:11.601178+0000","last_unstale":"2024-09-13T17:08:11.606181+0000","last_undegraded":"2024-09-13T17:08:11.606181+0000","last_fullsized":"2024-09-13T17:08:11.606181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566047+0000","last_change":"2024-09-13T17:08:11.132903+0000","last_active":"2024-09-13T17:08:12.566047+0000","last_peered":"2024-09-13T17:08:12.566047+0000","last_clean":"2024-09-13T17:08:12.566047+0000","last_became_active":"2024-09-13T17:08:11.131326+0000","last_became_peered":"2024-09-13T17:08:11.131326+0000","last_unstale":"2024-09-13T17:08:12.566047+0000","last_undegraded":"2024-09-13T17:08:12.566047+0000","last_fullsized":"2024-09-13T17:08:12.566047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567836+0000","last_change":"2024-09-13T17:08:11.581340+0000","last_active":"2024-09-13T17:08:12.567836+0000","last_peered":"2024-09-13T17:08:12.567836+0000","last_clean":"2024-09-13T17:08:12.567836+0000","last_became_active":"2024-09-13T17:08:11.578762+0000","last_became_peered":"2024-09-13T17:08:11.578762+0000","last_unstale":"2024-09-13T17:08:12.567836+0000","last_undegraded":"2024-09-13T17:08:12.567836+0000","last_fullsized":"2024-09-13T17:08:12.567836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.565331+0000","last_change":"2024-09-13T17:08:11.138090+0000","last_active":"2024-09-13T17:08:11.565331+0000","last_peered":"2024-09-13T17:08:11.565331+0000","last_clean":"2024-09-13T17:08:11.565331+0000","last_became_active":"2024-09-13T17:08:11.137846+0000","last_became_peered":"2024-09-13T17:08:11.137846+0000","last_unstale":"2024-09-13T17:08:11.565331+0000","last_undegraded":"2024-09-13T17:08:11.565331+0000","last_fullsized":"2024-09-13T17:08:11.565331+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.596798+0000","last_change":"2024-09-13T17:08:11.589182+0000","last_active":"2024-09-13T17:08:11.596798+0000","last_peered":"2024-09-13T17:08:11.596798+0000","last_clean":"2024-09-13T17:08:11.596798+0000","last_became_active":"2024-09-13T17:08:11.588462+0000","last_became_peered":"2024-09-13T17:08:11.588462+0000","last_unstale":"2024-09-13T17:08:11.596798+0000","last_undegraded":"2024-09-13T17:08:11.596798+0000","last_fullsized":"2024-09-13T17:08:11.596798+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:40.591592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":163,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.140691+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:08:11.140691+0000","last_peered":"2024-09-13T17:08:11.140691+0000","last_clean":"2024-09-13T17:08:11.140691+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:08:11.140691+0000","last_undegraded":"2024-09-13T17:08:11.140691+0000","last_fullsized":"2024-09-13T17:08:11.140691+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567696+0000","last_change":"2024-09-13T17:08:11.132650+0000","last_active":"2024-09-13T17:08:12.567696+0000","last_peered":"2024-09-13T17:08:12.567696+0000","last_clean":"2024-09-13T17:08:12.567696+0000","last_became_active":"2024-09-13T17:08:11.132311+0000","last_became_peered":"2024-09-13T17:08:11.132311+0000","last_unstale":"2024-09-13T17:08:12.567696+0000","last_undegraded":"2024-09-13T17:08:12.567696+0000","last_fullsized":"2024-09-13T17:08:12.567696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591524+0000","last_change":"2024-09-13T17:08:11.586834+0000","last_active":"2024-09-13T17:08:11.591524+0000","last_peered":"2024-09-13T17:08:11.591524+0000","last_clean":"2024-09-13T17:08:11.591524+0000","last_became_active":"2024-09-13T17:08:11.585548+0000","last_became_peered":"2024-09-13T17:08:11.585548+0000","last_unstale":"2024-09-13T17:08:11.591524+0000","last_undegraded":"2024-09-13T17:08:11.591524+0000","last_fullsized":"2024-09-13T17:08:11.591524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573891+0000","last_change":"2024-09-13T17:08:11.148517+0000","last_active":"2024-09-13T17:08:11.573891+0000","last_peered":"2024-09-13T17:08:11.573891+0000","last_clean":"2024-09-13T17:08:11.573891+0000","last_became_active":"2024-09-13T17:08:11.148250+0000","last_became_peered":"2024-09-13T17:08:11.148250+0000","last_unstale":"2024-09-13T17:08:11.573891+0000","last_undegraded":"2024-09-13T17:08:11.573891+0000","last_fullsized":"2024-09-13T17:08:11.573891+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591072+0000","last_change":"2024-09-13T17:08:11.586386+0000","last_active":"2024-09-13T17:08:11.591072+0000","last_peered":"2024-09-13T17:08:11.591072+0000","last_clean":"2024-09-13T17:08:11.591072+0000","last_became_active":"2024-09-13T17:08:11.577192+0000","last_became_peered":"2024-09-13T17:08:11.577192+0000","last_unstale":"2024-09-13T17:08:11.591072+0000","last_undegraded":"2024-09-13T17:08:11.591072+0000","last_fullsized":"2024-09-13T17:08:11.591072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.565301+0000","last_change":"2024-09-13T17:08:11.136035+0000","last_active":"2024-09-13T17:08:11.565301+0000","last_peered":"2024-09-13T17:08:11.565301+0000","last_clean":"2024-09-13T17:08:11.565301+0000","last_became_active":"2024-09-13T17:08:11.135742+0000","last_became_peered":"2024-09-13T17:08:11.135742+0000","last_unstale":"2024-09-13T17:08:11.565301+0000","last_undegraded":"2024-09-13T17:08:11.565301+0000","last_fullsized":"2024-09-13T17:08:11.565301+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565763+0000","last_change":"2024-09-13T17:08:11.571193+0000","last_active":"2024-09-13T17:08:12.565763+0000","last_peered":"2024-09-13T17:08:12.565763+0000","last_clean":"2024-09-13T17:08:12.565763+0000","last_became_active":"2024-09-13T17:08:11.570542+0000","last_became_peered":"2024-09-13T17:08:11.570542+0000","last_unstale":"2024-09-13T17:08:12.565763+0000","last_undegraded":"2024-09-13T17:08:12.565763+0000","last_fullsized":"2024-09-13T17:08:12.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567567+0000","last_change":"2024-09-13T17:08:11.140490+0000","last_active":"2024-09-13T17:08:12.567567+0000","last_peered":"2024-09-13T17:08:12.567567+0000","last_clean":"2024-09-13T17:08:12.567567+0000","last_became_active":"2024-09-13T17:08:11.139782+0000","last_became_peered":"2024-09-13T17:08:11.139782+0000","last_unstale":"2024-09-13T17:08:12.567567+0000","last_undegraded":"2024-09-13T17:08:12.567567+0000","last_fullsized":"2024-09-13T17:08:12.567567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568964+0000","last_change":"2024-09-13T17:08:11.578078+0000","last_active":"2024-09-13T17:08:12.568964+0000","last_peered":"2024-09-13T17:08:12.568964+0000","last_clean":"2024-09-13T17:08:12.568964+0000","last_became_active":"2024-09-13T17:08:11.577379+0000","last_became_peered":"2024-09-13T17:08:11.577379+0000","last_unstale":"2024-09-13T17:08:12.568964+0000","last_undegraded":"2024-09-13T17:08:12.568964+0000","last_fullsized":"2024-09-13T17:08:12.568964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565979+0000","last_change":"2024-09-13T17:08:11.135658+0000","last_active":"2024-09-13T17:08:12.565979+0000","last_peered":"2024-09-13T17:08:12.565979+0000","last_clean":"2024-09-13T17:08:12.565979+0000","last_became_active":"2024-09-13T17:08:11.134906+0000","last_became_peered":"2024-09-13T17:08:11.134906+0000","last_unstale":"2024-09-13T17:08:12.565979+0000","last_undegraded":"2024-09-13T17:08:12.565979+0000","last_fullsized":"2024-09-13T17:08:12.565979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568073+0000","last_change":"2024-09-13T17:08:11.580253+0000","last_active":"2024-09-13T17:08:12.568073+0000","last_peered":"2024-09-13T17:08:12.568073+0000","last_clean":"2024-09-13T17:08:12.568073+0000","last_became_active":"2024-09-13T17:08:11.579547+0000","last_became_peered":"2024-09-13T17:08:11.579547+0000","last_unstale":"2024-09-13T17:08:12.568073+0000","last_undegraded":"2024-09-13T17:08:12.568073+0000","last_fullsized":"2024-09-13T17:08:12.568073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.576475+0000","last_change":"2024-09-13T17:08:11.149945+0000","last_active":"2024-09-13T17:08:11.576475+0000","last_peered":"2024-09-13T17:08:11.576475+0000","last_clean":"2024-09-13T17:08:11.576475+0000","last_became_active":"2024-09-13T17:08:11.149641+0000","last_became_peered":"2024-09-13T17:08:11.149641+0000","last_unstale":"2024-09-13T17:08:11.576475+0000","last_undegraded":"2024-09-13T17:08:11.576475+0000","last_fullsized":"2024-09-13T17:08:11.576475+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565952+0000","last_change":"2024-09-13T17:08:11.578587+0000","last_active":"2024-09-13T17:08:12.565952+0000","last_peered":"2024-09-13T17:08:12.565952+0000","last_clean":"2024-09-13T17:08:12.565952+0000","last_became_active":"2024-09-13T17:08:11.576109+0000","last_became_peered":"2024-09-13T17:08:11.576109+0000","last_unstale":"2024-09-13T17:08:12.565952+0000","last_undegraded":"2024-09-13T17:08:12.565952+0000","last_fullsized":"2024-09-13T17:08:12.565952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566142+0000","last_change":"2024-09-13T17:08:11.141871+0000","last_active":"2024-09-13T17:08:12.566142+0000","last_peered":"2024-09-13T17:08:12.566142+0000","last_clean":"2024-09-13T17:08:12.566142+0000","last_became_active":"2024-09-13T17:08:11.141567+0000","last_became_peered":"2024-09-13T17:08:11.141567+0000","last_unstale":"2024-09-13T17:08:12.566142+0000","last_undegraded":"2024-09-13T17:08:12.566142+0000","last_fullsized":"2024-09-13T17:08:12.566142+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591451+0000","last_change":"2024-09-13T17:08:11.589765+0000","last_active":"2024-09-13T17:08:11.591451+0000","last_peered":"2024-09-13T17:08:11.591451+0000","last_clean":"2024-09-13T17:08:11.591451+0000","last_became_active":"2024-09-13T17:08:11.589278+0000","last_became_peered":"2024-09-13T17:08:11.589278+0000","last_unstale":"2024-09-13T17:08:11.591451+0000","last_undegraded":"2024-09-13T17:08:11.591451+0000","last_fullsized":"2024-09-13T17:08:11.591451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.603021+0000","last_change":"2024-09-13T17:08:11.148235+0000","last_active":"2024-09-13T17:08:11.603021+0000","last_peered":"2024-09-13T17:08:11.603021+0000","last_clean":"2024-09-13T17:08:11.603021+0000","last_became_active":"2024-09-13T17:08:11.147994+0000","last_became_peered":"2024-09-13T17:08:11.147994+0000","last_unstale":"2024-09-13T17:08:11.603021+0000","last_undegraded":"2024-09-13T17:08:11.603021+0000","last_fullsized":"2024-09-13T17:08:11.603021+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573772+0000","last_change":"2024-09-13T17:08:11.570444+0000","last_active":"2024-09-13T17:08:11.573772+0000","last_peered":"2024-09-13T17:08:11.573772+0000","last_clean":"2024-09-13T17:08:11.573772+0000","last_became_active":"2024-09-13T17:08:11.570207+0000","last_became_peered":"2024-09-13T17:08:11.570207+0000","last_unstale":"2024-09-13T17:08:11.573772+0000","last_undegraded":"2024-09-13T17:08:11.573772+0000","last_fullsized":"2024-09-13T17:08:11.573772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573767+0000","last_change":"2024-09-13T17:08:11.147785+0000","last_active":"2024-09-13T17:08:11.573767+0000","last_peered":"2024-09-13T17:08:11.573767+0000","last_clean":"2024-09-13T17:08:11.573767+0000","last_became_active":"2024-09-13T17:08:11.147222+0000","last_became_peered":"2024-09-13T17:08:11.147222+0000","last_unstale":"2024-09-13T17:08:11.573767+0000","last_undegraded":"2024-09-13T17:08:11.573767+0000","last_fullsized":"2024-09-13T17:08:11.573767+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567547+0000","last_change":"2024-09-13T17:08:11.574148+0000","last_active":"2024-09-13T17:08:12.567547+0000","last_peered":"2024-09-13T17:08:12.567547+0000","last_clean":"2024-09-13T17:08:12.567547+0000","last_became_active":"2024-09-13T17:08:11.573879+0000","last_became_peered":"2024-09-13T17:08:11.573879+0000","last_unstale":"2024-09-13T17:08:12.567547+0000","last_undegraded":"2024-09-13T17:08:12.567547+0000","last_fullsized":"2024-09-13T17:08:12.567547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:10.871670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565831+0000","last_change":"2024-09-13T17:08:11.146000+0000","last_active":"2024-09-13T17:08:12.565831+0000","last_peered":"2024-09-13T17:08:12.565831+0000","last_clean":"2024-09-13T17:08:12.565831+0000","last_became_active":"2024-09-13T17:08:11.145714+0000","last_became_peered":"2024-09-13T17:08:11.145714+0000","last_unstale":"2024-09-13T17:08:12.565831+0000","last_undegraded":"2024-09-13T17:08:12.565831+0000","last_fullsized":"2024-09-13T17:08:12.565831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.580470+0000","last_change":"2024-09-13T17:08:11.573192+0000","last_active":"2024-09-13T17:08:11.580470+0000","last_peered":"2024-09-13T17:08:11.580470+0000","last_clean":"2024-09-13T17:08:11.580470+0000","last_became_active":"2024-09-13T17:08:11.572009+0000","last_became_peered":"2024-09-13T17:08:11.572009+0000","last_unstale":"2024-09-13T17:08:11.580470+0000","last_undegraded":"2024-09-13T17:08:11.580470+0000","last_fullsized":"2024-09-13T17:08:11.580470+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565978+0000","last_change":"2024-09-13T17:08:11.145299+0000","last_active":"2024-09-13T17:08:12.565978+0000","last_peered":"2024-09-13T17:08:12.565978+0000","last_clean":"2024-09-13T17:08:12.565978+0000","last_became_active":"2024-09-13T17:08:11.143966+0000","last_became_peered":"2024-09-13T17:08:11.143966+0000","last_unstale":"2024-09-13T17:08:12.565978+0000","last_undegraded":"2024-09-13T17:08:12.565978+0000","last_fullsized":"2024-09-13T17:08:12.565978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.577272+0000","last_change":"2024-09-13T17:08:11.573641+0000","last_active":"2024-09-13T17:08:11.577272+0000","last_peered":"2024-09-13T17:08:11.577272+0000","last_clean":"2024-09-13T17:08:11.577272+0000","last_became_active":"2024-09-13T17:08:11.573434+0000","last_became_peered":"2024-09-13T17:08:11.573434+0000","last_unstale":"2024-09-13T17:08:11.577272+0000","last_undegraded":"2024-09-13T17:08:11.577272+0000","last_fullsized":"2024-09-13T17:08:11.577272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.602613+0000","last_change":"2024-09-13T17:08:11.148180+0000","last_active":"2024-09-13T17:08:11.602613+0000","last_peered":"2024-09-13T17:08:11.602613+0000","last_clean":"2024-09-13T17:08:11.602613+0000","last_became_active":"2024-09-13T17:08:11.147892+0000","last_became_peered":"2024-09-13T17:08:11.147892+0000","last_unstale":"2024-09-13T17:08:11.602613+0000","last_undegraded":"2024-09-13T17:08:11.602613+0000","last_fullsized":"2024-09-13T17:08:11.602613+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593208+0000","last_change":"2024-09-13T17:08:11.589293+0000","last_active":"2024-09-13T17:08:11.593208+0000","last_peered":"2024-09-13T17:08:11.593208+0000","last_clean":"2024-09-13T17:08:11.593208+0000","last_became_active":"2024-09-13T17:08:11.589022+0000","last_became_peered":"2024-09-13T17:08:11.589022+0000","last_unstale":"2024-09-13T17:08:11.593208+0000","last_undegraded":"2024-09-13T17:08:11.593208+0000","last_fullsized":"2024-09-13T17:08:11.593208+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:39.460619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568004+0000","last_change":"2024-09-13T17:08:11.594776+0000","last_active":"2024-09-13T17:08:12.568004+0000","last_peered":"2024-09-13T17:08:12.568004+0000","last_clean":"2024-09-13T17:08:12.568004+0000","last_became_active":"2024-09-13T17:08:11.594498+0000","last_became_peered":"2024-09-13T17:08:11.594498+0000","last_unstale":"2024-09-13T17:08:12.568004+0000","last_undegraded":"2024-09-13T17:08:12.568004+0000","last_fullsized":"2024-09-13T17:08:12.568004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.598145+0000","last_change":"2024-09-13T17:08:11.149488+0000","last_active":"2024-09-13T17:08:11.598145+0000","last_peered":"2024-09-13T17:08:11.598145+0000","last_clean":"2024-09-13T17:08:11.598145+0000","last_became_active":"2024-09-13T17:08:11.149198+0000","last_became_peered":"2024-09-13T17:08:11.149198+0000","last_unstale":"2024-09-13T17:08:11.598145+0000","last_undegraded":"2024-09-13T17:08:11.598145+0000","last_fullsized":"2024-09-13T17:08:11.598145+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:24.611621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567674+0000","last_change":"2024-09-13T17:08:11.597049+0000","last_active":"2024-09-13T17:08:12.567674+0000","last_peered":"2024-09-13T17:08:12.567674+0000","last_clean":"2024-09-13T17:08:12.567674+0000","last_became_active":"2024-09-13T17:08:11.594306+0000","last_became_peered":"2024-09-13T17:08:11.594306+0000","last_unstale":"2024-09-13T17:08:12.567674+0000","last_undegraded":"2024-09-13T17:08:12.567674+0000","last_fullsized":"2024-09-13T17:08:12.567674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.600666+0000","last_change":"2024-09-13T17:08:11.152688+0000","last_active":"2024-09-13T17:08:11.600666+0000","last_peered":"2024-09-13T17:08:11.600666+0000","last_clean":"2024-09-13T17:08:11.600666+0000","last_became_active":"2024-09-13T17:08:11.152286+0000","last_became_peered":"2024-09-13T17:08:11.152286+0000","last_unstale":"2024-09-13T17:08:11.600666+0000","last_undegraded":"2024-09-13T17:08:11.600666+0000","last_fullsized":"2024-09-13T17:08:11.600666+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591670+0000","last_change":"2024-09-13T17:08:11.588908+0000","last_active":"2024-09-13T17:08:11.591670+0000","last_peered":"2024-09-13T17:08:11.591670+0000","last_clean":"2024-09-13T17:08:11.591670+0000","last_became_active":"2024-09-13T17:08:11.588299+0000","last_became_peered":"2024-09-13T17:08:11.588299+0000","last_unstale":"2024-09-13T17:08:11.591670+0000","last_undegraded":"2024-09-13T17:08:11.591670+0000","last_fullsized":"2024-09-13T17:08:11.591670+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.591284+0000","last_change":"2024-09-13T17:08:11.140760+0000","last_active":"2024-09-13T17:08:11.591284+0000","last_peered":"2024-09-13T17:08:11.591284+0000","last_clean":"2024-09-13T17:08:11.591284+0000","last_became_active":"2024-09-13T17:08:11.140489+0000","last_became_peered":"2024-09-13T17:08:11.140489+0000","last_unstale":"2024-09-13T17:08:11.591284+0000","last_undegraded":"2024-09-13T17:08:11.591284+0000","last_fullsized":"2024-09-13T17:08:11.591284+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.593002+0000","last_change":"2024-09-13T17:08:11.591976+0000","last_active":"2024-09-13T17:08:11.593002+0000","last_peered":"2024-09-13T17:08:11.593002+0000","last_clean":"2024-09-13T17:08:11.593002+0000","last_became_active":"2024-09-13T17:08:11.591484+0000","last_became_peered":"2024-09-13T17:08:11.591484+0000","last_unstale":"2024-09-13T17:08:11.593002+0000","last_undegraded":"2024-09-13T17:08:11.593002+0000","last_fullsized":"2024-09-13T17:08:11.593002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.601650+0000","last_change":"2024-09-13T17:08:11.145298+0000","last_active":"2024-09-13T17:08:11.601650+0000","last_peered":"2024-09-13T17:08:11.601650+0000","last_clean":"2024-09-13T17:08:11.601650+0000","last_became_active":"2024-09-13T17:08:11.145029+0000","last_became_peered":"2024-09-13T17:08:11.145029+0000","last_unstale":"2024-09-13T17:08:11.601650+0000","last_undegraded":"2024-09-13T17:08:11.601650+0000","last_fullsized":"2024-09-13T17:08:11.601650+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":790528,"data_stored":655548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96699999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":66162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":64688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72599999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574852,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":204800,"data_stored":66162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31304,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340536,"statfs":{"total":96636764160,"available":96604708864,"internally_reserved":0,"allocated":188416,"data_stored":63579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31316,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340524,"statfs":{"total":96636764160,"available":96604696576,"internally_reserved":0,"allocated":200704,"data_stored":64758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72799999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":794624,"data_stored":654585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.437}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:08:16.163 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:08:16.164 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T17:08:16.164 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T17:08:16.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T17:08:16.165 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T17:08:16.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.352+0000 7fbf80000640 1 Processor -- start 2024-09-13T17:08:16.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.352+0000 7fbf80000640 1 -- start start 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.353+0000 7fbf80000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf80000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 0x7fbf7815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf80000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7815edb0 0x7fbf78070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf80000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf780adb00 con 0x7fbf7815edb0 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf80000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf7815e100 con 0x7fbf780cabc0 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf80000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf7815e3e0 con 0x7fbf7815a540 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7815edb0 0x7fbf78070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf7815edb0 0x7fbf78070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33326/0 (socket says 172.21.15.28:33326) 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 -- 172.21.15.28:0/2301552301 learned_addr learned my addr 172.21.15.28:0/2301552301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:16.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7d600640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7cc00640 1 --2- 172.21.15.28:0/2301552301 >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 0x7fbf7815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 -- 172.21.15.28:0/2301552301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 msgr2=0x7fbf7806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 -- 172.21.15.28:0/2301552301 >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 msgr2=0x7fbf7815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 --2- 172.21.15.28:0/2301552301 >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 0x7fbf7815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7e000640 1 -- 172.21.15.28:0/2301552301 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7806ce80 con 0x7fbf7815edb0 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7d600640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:16.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.354+0000 7fbf7cc00640 1 --2- 172.21.15.28:0/2301552301 >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 0x7fbf7815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:16.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.355+0000 7fbf7e000640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7815edb0 0x7fbf78070cc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbf7806ae00 tx=0x7fbf7002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a754cf07447725da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf77e00640 1 -- 172.21.15.28:0/2301552301 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70031600 con 0x7fbf7815edb0 2024-09-13T17:08:16.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf77e00640 1 -- 172.21.15.28:0/2301552301 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7002d050 con 0x7fbf7815edb0 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf77e00640 1 -- 172.21.15.28:0/2301552301 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf70031980 con 0x7fbf7815edb0 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 -- 172.21.15.28:0/2301552301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7815edb0 msgr2=0x7fbf78070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7815edb0 0x7fbf78070cc0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbf7806ae00 tx=0x7fbf7002a610 comp rx=0 tx=0).stop 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 -- 172.21.15.28:0/2301552301 shutdown_connections 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf7815edb0 0x7fbf78070cc0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 --2- 172.21.15.28:0/2301552301 >> v2:172.21.15.189:3300/0 conn(0x7fbf780cabc0 0x7fbf7815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 --2- 172.21.15.28:0/2301552301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 -- 172.21.15.28:0/2301552301 >> 172.21.15.28:0/2301552301 conn(0x7fbf78069e80 msgr2=0x7fbf7806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 -- 172.21.15.28:0/2301552301 shutdown_connections 2024-09-13T17:08:16.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.356+0000 7fbf80000640 1 -- 172.21.15.28:0/2301552301 wait complete. 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.358+0000 7fbf80000640 1 Processor -- start 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.358+0000 7fbf80000640 1 -- start start 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7815c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf78150e70 0x7fbf7815cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf780b11c0 con 0x7fbf780cabc0 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf781f3200 con 0x7fbf78150e70 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.359+0000 7fbf80000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf781f2330 con 0x7fbf7815a540 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33340/0 (socket says 172.21.15.28:33340) 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 -- 172.21.15.28:0/788879381 learned_addr learned my addr 172.21.15.28:0/788879381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7e000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf78150e70 0x7fbf7815cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7cc00640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7815c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 -- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 msgr2=0x7fbf7815c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7815c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 -- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf78150e70 msgr2=0x7fbf7815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf78150e70 0x7fbf7815cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 -- 172.21.15.28:0/788879381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf70009b30 con 0x7fbf780cabc0 2024-09-13T17:08:16.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.360+0000 7fbf7d600640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e760 tx=0x7fbf6c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.361+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c004230 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.361+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6c004390 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.361+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf6c00c8f0 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.361+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf781f25f0 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.361+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf781f2bb0 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.362+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf7815e870 con 0x7fbf780cabc0 2024-09-13T17:08:16.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.363+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf6c00ca50 con 0x7fbf780cabc0 2024-09-13T17:08:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.363+0000 7fbf75600640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 0x7fbf5806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.364+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf6c088e40 con 0x7fbf780cabc0 2024-09-13T17:08:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.364+0000 7fbf7cc00640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 0x7fbf5806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.364+0000 7fbf7cc00640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 0x7fbf5806d0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf78154f10 tx=0x7fbf64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.367+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf6c0540c0 con 0x7fbf780cabc0 2024-09-13T17:08:16.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.594+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fbf78150c10 con 0x7fbf780cabc0 2024-09-13T17:08:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.597+0000 7fbf75600640 1 -- 172.21.15.28:0/788879381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170950 (secure 0 0 0) 0x7fbf6c0576e0 con 0x7fbf780cabc0 2024-09-13T17:08:16.641 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:16.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 msgr2=0x7fbf5806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 0x7fbf5806d0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbf78154f10 tx=0x7fbf64008040 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 msgr2=0x7fbf781506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c00e760 tx=0x7fbf6c00ec20 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 shutdown_connections 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5806ac00 0x7fbf5806d0c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf78150e70 0x7fbf7815cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf7815a540 0x7fbf7815c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 --2- 172.21.15.28:0/788879381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf780cabc0 0x7fbf781506d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 >> 172.21.15.28:0/788879381 conn(0x7fbf78069e80 msgr2=0x7fbf7806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 shutdown_connections 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.600+0000 7fbf80000640 1 -- 172.21.15.28:0/788879381 wait complete. 2024-09-13T17:08:16.643 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":19110189},{"type":"v1","addr":"172.21.15.189:6833","nonce":19110189}]},"active_addr":"172.21.15.189:6833/19110189","active_change":"2024-09-13T17:07:50.870013+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":3573895939}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":4286587077}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":2319693}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.189:0","nonce":1670862219}]}]} 2024-09-13T17:08:16.648 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T17:08:16.648 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T17:08:16.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.802+0000 7ffa19c00640 1 Processor -- start 2024-09-13T17:08:16.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.802+0000 7ffa19c00640 1 -- start start 2024-09-13T17:08:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.803+0000 7ffa19c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffa1406c9e0 0x7ffa1415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.803+0000 7ffa19c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa14159d90 con 0x7ffa1406f970 2024-09-13T17:08:16.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.803+0000 7ffa19c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa140b2040 con 0x7ffa141592a0 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.803+0000 7ffa19c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa1415b0b0 con 0x7ffa1406c9e0 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffa1406c9e0 0x7ffa1415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffa1406c9e0 0x7ffa1415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33356/0 (socket says 172.21.15.28:33356) 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 -- 172.21.15.28:0/4078495140 learned_addr learned my addr 172.21.15.28:0/4078495140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 -- 172.21.15.28:0/4078495140 >> v1:172.21.15.189:6789/0 conn(0x7ffa141592a0 legacy=0x7ffa14159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 -- 172.21.15.28:0/4078495140 >> v1:172.21.15.28:6789/0 conn(0x7ffa1406f970 legacy=0x7ffa1406c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa13400640 1 -- 172.21.15.28:0/4078495140 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa14159ff0 con 0x7ffa1406c9e0 2024-09-13T17:08:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.804+0000 7ffa11600640 1 -- 172.21.15.28:0/4078495140 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782720099 0 0) 0x7ffa14159d90 con 0x7ffa1406f970 2024-09-13T17:08:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.805+0000 7ffa13400640 1 --2- 172.21.15.28:0/4078495140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406c9e0 0x7ffa1415ab70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00d150 tx=0x7ffa0c02c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef60e7564867ea2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.805+0000 7ffa11600640 1 -- 172.21.15.28:0/4078495140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c00b2d0 con 0x7ffa1406c9e0 2024-09-13T17:08:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.805+0000 7ffa11600640 1 -- 172.21.15.28:0/4078495140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c00b430 con 0x7ffa1406c9e0 2024-09-13T17:08:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.805+0000 7ffa11600640 1 -- 172.21.15.28:0/4078495140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c009820 con 0x7ffa1406c9e0 2024-09-13T17:08:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 -- 172.21.15.28:0/4078495140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406c9e0 msgr2=0x7ffa1415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 --2- 172.21.15.28:0/4078495140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406c9e0 0x7ffa1415ab70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c00d150 tx=0x7ffa0c02c9c0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 -- 172.21.15.28:0/4078495140 shutdown_connections 2024-09-13T17:08:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 --2- 172.21.15.28:0/4078495140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406c9e0 0x7ffa1415ab70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 -- 172.21.15.28:0/4078495140 >> 172.21.15.28:0/4078495140 conn(0x7ffa14069e80 msgr2=0x7ffa1406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 -- 172.21.15.28:0/4078495140 shutdown_connections 2024-09-13T17:08:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.806+0000 7ffa19c00640 1 -- 172.21.15.28:0/4078495140 wait complete. 2024-09-13T17:08:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.808+0000 7ffa19c00640 1 Processor -- start 2024-09-13T17:08:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.808+0000 7ffa19c00640 1 -- start start 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa1406c9e0 0x7ffa140cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa141592a0 0x7ffa140ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa141447b0 con 0x7ffa1406f970 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140cbbd0 con 0x7ffa1406c9e0 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa19c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa140cf110 con 0x7ffa141592a0 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33358/0 (socket says 172.21.15.28:33358) 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 -- 172.21.15.28:0/136926103 learned_addr learned my addr 172.21.15.28:0/136926103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12a00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa1406c9e0 0x7ffa140cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa13400640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa141592a0 0x7ffa140ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 -- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa141592a0 msgr2=0x7ffa140ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa141592a0 0x7ffa140ce6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 -- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa1406c9e0 msgr2=0x7ffa140cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa1406c9e0 0x7ffa140cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.809+0000 7ffa12000640 1 -- 172.21.15.28:0/136926103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0c00bd40 con 0x7ffa1406f970 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa12000640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000c520 tx=0x7ffa0000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa000052d0 con 0x7ffa1406f970 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa00005430 con 0x7ffa1406f970 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa00012870 con 0x7ffa1406f970 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa140cb580 con 0x7ffa1406f970 2024-09-13T17:08:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.810+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa140cf3f0 con 0x7ffa1406f970 2024-09-13T17:08:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.811+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1415ab70 con 0x7ffa1406f970 2024-09-13T17:08:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.812+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffa00003610 con 0x7ffa1406f970 2024-09-13T17:08:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.813+0000 7ffa0aa00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 0x7ff9f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.813+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffa00088960 con 0x7ffa1406f970 2024-09-13T17:08:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.813+0000 7ffa12a00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 0x7ff9f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.814+0000 7ffa12a00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 0x7ff9f406d0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa040016c0 tx=0x7ffa04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:16.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.817+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffa00053ab0 con 0x7ffa1406f970 2024-09-13T17:08:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:16.999+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa1406c630 con 0x7ffa1406f970 2024-09-13T17:08:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.000+0000 7ffa0aa00640 1 -- 172.21.15.28:0/136926103 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7ffa000570d0 con 0x7ffa1406f970 2024-09-13T17:08:17.002 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:17.003 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.002+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 msgr2=0x7ff9f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.002+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 0x7ff9f406d0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ffa040016c0 tx=0x7ffa04009040 comp rx=0 tx=0).stop 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.002+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 msgr2=0x7ffa14144270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.002+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000c520 tx=0x7ffa0000c9e0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 shutdown_connections 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff9f406ac00 0x7ff9f406d0c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa141592a0 0x7ffa140ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa1406f970 0x7ffa14144270 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 --2- 172.21.15.28:0/136926103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa1406c9e0 0x7ffa140cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 >> 172.21.15.28:0/136926103 conn(0x7ffa14069e80 msgr2=0x7ffa1415b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 shutdown_connections 2024-09-13T17:08:17.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.003+0000 7ffa19c00640 1 -- 172.21.15.28:0/136926103 wait complete. 2024-09-13T17:08:17.020 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T17:08:17.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:17.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.173+0000 7fd12be00640 1 Processor -- start 2024-09-13T17:08:17.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.174+0000 7fd12be00640 1 -- start start 2024-09-13T17:08:17.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd12c15c8d0 0x7fd12c06b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c06ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12be00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c06bb50 con 0x7fd12c06f480 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0b38a0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c069f20 con 0x7fd12c0ccb60 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd12c15c8d0 0x7fd12c06b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd12c15c8d0 0x7fd12c06b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33372/0 (socket says 172.21.15.28:33372) 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 -- 172.21.15.28:0/1160481517 learned_addr learned my addr 172.21.15.28:0/1160481517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd123e00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c06ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 -- 172.21.15.28:0/1160481517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 msgr2=0x7fd12c06ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c06ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 -- 172.21.15.28:0/1160481517 >> v1:172.21.15.189:6789/0 conn(0x7fd12c06f480 legacy=0x7fd12c0c9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.176+0000 7fd12aa00640 1 -- 172.21.15.28:0/1160481517 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c06a200 con 0x7fd12c15c8d0 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12aa00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c06b610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009870 tx=0x7fd11c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=827c28d0c0c8cf28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12a000640 1 -- 172.21.15.28:0/1160481517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c032440 con 0x7fd12c15c8d0 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12a000640 1 -- 172.21.15.28:0/1160481517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c0325a0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12a000640 1 -- 172.21.15.28:0/1160481517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11c032930 con 0x7fd12c15c8d0 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12be00640 1 -- 172.21.15.28:0/1160481517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 msgr2=0x7fd12c06b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.177+0000 7fd12be00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c06b610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd11c009870 tx=0x7fd11c02a270 comp rx=0 tx=0).stop 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 -- 172.21.15.28:0/1160481517 shutdown_connections 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c06ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 --2- 172.21.15.28:0/1160481517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c06b610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 -- 172.21.15.28:0/1160481517 >> 172.21.15.28:0/1160481517 conn(0x7fd12c06d4e0 msgr2=0x7fd12c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 -- 172.21.15.28:0/1160481517 shutdown_connections 2024-09-13T17:08:17.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.178+0000 7fd12be00640 1 -- 172.21.15.28:0/1160481517 wait complete. 2024-09-13T17:08:17.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.179+0000 7fd12be00640 1 Processor -- start 2024-09-13T17:08:17.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.179+0000 7fd12be00640 1 -- start start 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 0x7fd12c149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c1f64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1f60c0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c14a370 con 0x7fd12c06f480 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.180+0000 7fd12be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1f5d70 con 0x7fd12c0ccb60 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd123e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c1f64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 0x7fd12c149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33380/0 (socket says 172.21.15.28:33380) 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 -- 172.21.15.28:0/2095281373 learned_addr learned my addr 172.21.15.28:0/2095281373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 -- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 msgr2=0x7fd12c1f64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c1f64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 -- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 msgr2=0x7fd12c149e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 0x7fd12c149e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 -- 172.21.15.28:0/2095281373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c0094d0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12aa00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 0x7fd12c149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.181+0000 7fd12b400640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c7d0 tx=0x7fd12400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.182+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd124003b70 con 0x7fd12c15c8d0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.182+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd124003cf0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.182+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd12400aa30 con 0x7fd12c15c8d0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.182+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c15f1d0 con 0x7fd12c15c8d0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.182+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c15f790 con 0x7fd12c15c8d0 2024-09-13T17:08:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.183+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c1f6200 con 0x7fd12c15c8d0 2024-09-13T17:08:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.184+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd12400ab90 con 0x7fd12c15c8d0 2024-09-13T17:08:17.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.185+0000 7fd122a00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 0x7fd10806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.185+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd124088f30 con 0x7fd12c15c8d0 2024-09-13T17:08:17.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.186+0000 7fd12aa00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 0x7fd10806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.186+0000 7fd12aa00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 0x7fd10806d0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd11c034000 tx=0x7fd11c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.190+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd12c1f6200 con 0x7fd12c15c8d0 2024-09-13T17:08:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.368+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd12c06bb50 con 0x7fd12c15c8d0 2024-09-13T17:08:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.369+0000 7fd122a00640 1 -- 172.21.15.28:0/2095281373 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd12c06bb50 con 0x7fd12c15c8d0 2024-09-13T17:08:17.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:17.372 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 msgr2=0x7fd10806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 0x7fd10806d0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd11c034000 tx=0x7fd11c02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 msgr2=0x7fd12c151130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd12400c7d0 tx=0x7fd12400cc90 comp rx=0 tx=0).stop 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 shutdown_connections 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd10806ac00 0x7fd10806d0c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd12c15c8d0 0x7fd12c151130 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd12c0ccb60 0x7fd12c1f64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 --2- 172.21.15.28:0/2095281373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd12c06f480 0x7fd12c149e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 >> 172.21.15.28:0/2095281373 conn(0x7fd12c06d4e0 msgr2=0x7fd12c0b4ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 shutdown_connections 2024-09-13T17:08:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.372+0000 7fd12be00640 1 -- 172.21.15.28:0/2095281373 wait complete. 2024-09-13T17:08:17.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T17:08:17.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T17:08:17.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T17:08:17.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T17:08:17.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T17:08:17.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T17:08:17.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T17:08:17.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T17:08:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.671+0000 7f14fe200640 1 Processor -- start 2024-09-13T17:08:17.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.671+0000 7f14fe200640 1 -- start start 2024-09-13T17:08:17.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.672+0000 7f9612a00640 1 Processor -- start 2024-09-13T17:08:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.673+0000 7f9612a00640 1 -- start start 2024-09-13T17:08:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.673+0000 7ff597200640 1 Processor -- start 2024-09-13T17:08:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.673+0000 7ff597200640 1 -- start start 2024-09-13T17:08:17.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.674+0000 7f14fe200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.674+0000 7f14fe200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f14f80d0480 0x7f14f80cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.674+0000 7f14fce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f14fce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55604/0 (socket says 172.21.15.28:55604) 2024-09-13T17:08:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f14fce00640 1 -- 172.21.15.28:0/4179512030 learned_addr learned my addr 172.21.15.28:0/4179512030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9604055180 0x7f9604035ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9604007040 0x7f9604036530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96040e5840 con 0x7f9604055180 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96040398e0 con 0x7f9604007040 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7ff597200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 0x7ff588053f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7ff597200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5880520a0 con 0x7ff588005a20 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7ff597200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5880522f0 con 0x7ff5880e70e0 2024-09-13T17:08:17.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7ff597200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff588006c40 con 0x7ff58801dc00 2024-09-13T17:08:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7f9612a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9604039bc0 con 0x7f9604054c80 2024-09-13T17:08:17.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7f9611600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7f14f7e00640 1 --2- 172.21.15.28:0/4179512030 >> v2:172.21.15.28:3300/0 conn(0x7f14f80d0480 0x7f14f80cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7f9611600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55610/0 (socket says 172.21.15.28:55610) 2024-09-13T17:08:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7ff58f400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff5880e70e0 0x7ff58801d4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:38082/0 (socket says 172.21.15.28:38082) 2024-09-13T17:08:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7ff58f400640 1 -- 172.21.15.28:0/1250067457 learned_addr learned my addr 172.21.15.28:0/1250067457 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7ff595200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 0x7ff588053f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.675+0000 7fb8d8c00640 1 Processor -- start 2024-09-13T17:08:17.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7fb8d8c00640 1 -- start start 2024-09-13T17:08:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7ff58fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff588005a20 0x7ff588005e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34420/0 (socket says 172.21.15.28:34420) 2024-09-13T17:08:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7f9611600640 1 -- 172.21.15.28:0/390235046 learned_addr learned my addr 172.21.15.28:0/390235046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.676+0000 7f9610c00640 1 --2- 172.21.15.28:0/390235046 >> v2:172.21.15.28:3300/0 conn(0x7f9604055180 0x7f9604035ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.677+0000 7ff595200640 1 -- 172.21.15.28:0/1250067457 >> v1:172.21.15.189:6789/0 conn(0x7ff5880e70e0 legacy=0x7ff58801d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.677+0000 7ff595200640 1 -- 172.21.15.28:0/1250067457 >> v1:172.21.15.28:6789/0 conn(0x7ff588005a20 legacy=0x7ff588005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.677+0000 7ff58ea00640 1 -- 172.21.15.28:0/1250067457 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135974828 0 0) 0x7ff5880522f0 con 0x7ff5880e70e0 2024-09-13T17:08:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d8c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb8cc053880 0x7fb8cc053ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.677+0000 7ff595200640 1 -- 172.21.15.28:0/1250067457 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588006f20 con 0x7ff58801dc00 2024-09-13T17:08:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb8cc0e7000 0x7fb8cc006400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34434/0 (socket says 172.21.15.28:34434) 2024-09-13T17:08:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d1e00640 1 -- 172.21.15.28:0/2775039834 learned_addr learned my addr 172.21.15.28:0/2775039834 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7ff595200640 1 --2- 172.21.15.28:0/1250067457 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 0x7ff588053f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff590000b60 tx=0x7ff5900a4510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cdfb09bd00b8f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc047d40 con 0x7fb8cc0e7000 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc007390 con 0x7fb8cc01c7c0 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8cc032830 con 0x7fb8cc053880 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7fb8d1400640 1 --2- 172.21.15.28:0/2775039834 >> v2:172.21.15.189:3300/0 conn(0x7fb8cc053880 0x7fb8cc053ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7f9611600640 1 -- 172.21.15.28:0/390235046 >> v2:172.21.15.189:3300/0 conn(0x7f9604007040 msgr2=0x7f9604036530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7f9611600640 1 --2- 172.21.15.28:0/390235046 >> v2:172.21.15.189:3300/0 conn(0x7f9604007040 0x7f9604036530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7f9611600640 1 -- 172.21.15.28:0/390235046 >> v2:172.21.15.28:3300/0 conn(0x7f9604055180 msgr2=0x7f9604035ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7f9611600640 1 --2- 172.21.15.28:0/390235046 >> v2:172.21.15.28:3300/0 conn(0x7f9604055180 0x7f9604035ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.678+0000 7f9611600640 1 -- 172.21.15.28:0/390235046 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9604039ea0 con 0x7f9604054c80 2024-09-13T17:08:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f80d0bf0 con 0x7f14f80cf7a0 2024-09-13T17:08:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f80cfc50 con 0x7f14f80d0480 2024-09-13T17:08:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80d0ed0 con 0x7f14f815cb30 2024-09-13T17:08:17.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7f9611600640 1 --2- 172.21.15.28:0/390235046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f960c02c970 tx=0x7f960c0a4440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17ad805da0d74d77 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7ff58ea00640 1 -- 172.21.15.28:0/1250067457 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5900a97d0 con 0x7ff58801dc00 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7ff58ea00640 1 -- 172.21.15.28:0/1250067457 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5900a7050 con 0x7ff58801dc00 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7ff58ea00640 1 -- 172.21.15.28:0/1250067457 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5900a9bb0 con 0x7ff58801dc00 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7fb8d1400640 1 -- 172.21.15.28:0/2775039834 >> v1:172.21.15.189:6789/0 conn(0x7fb8cc01c7c0 legacy=0x7fb8cc01cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7fb8d0a00640 1 -- 172.21.15.28:0/2775039834 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918926739 0 0) 0x7fb8cc007390 con 0x7fb8cc01c7c0 2024-09-13T17:08:17.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.679+0000 7fb8d1400640 1 -- 172.21.15.28:0/2775039834 >> v1:172.21.15.28:6789/0 conn(0x7fb8cc0e7000 legacy=0x7fb8cc006400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f960be00640 1 -- 172.21.15.28:0/390235046 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0ab600 con 0x7f9604054c80 2024-09-13T17:08:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f960be00640 1 -- 172.21.15.28:0/390235046 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c0a7050 con 0x7f9604054c80 2024-09-13T17:08:17.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f960be00640 1 -- 172.21.15.28:0/390235046 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0aba60 con 0x7f9604054c80 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f14fce00640 1 -- 172.21.15.28:0/4179512030 >> v2:172.21.15.28:3300/0 conn(0x7f14f80d0480 msgr2=0x7f14f80cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f14fce00640 1 --2- 172.21.15.28:0/4179512030 >> v2:172.21.15.28:3300/0 conn(0x7f14f80d0480 0x7f14f80cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f14fce00640 1 -- 172.21.15.28:0/4179512030 >> v1:172.21.15.28:6789/0 conn(0x7f14f80cf7a0 legacy=0x7f14f8069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f14fce00640 1 -- 172.21.15.28:0/4179512030 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f80d11b0 con 0x7f14f815cb30 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7f14f7400640 1 -- 172.21.15.28:0/4179512030 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274193144 0 0) 0x7f14f80d0bf0 con 0x7f14f80cf7a0 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 -- 172.21.15.28:0/390235046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 msgr2=0x7f960400a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 --2- 172.21.15.28:0/390235046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f960c02c970 tx=0x7f960c0a4440 comp rx=0 tx=0).stop 2024-09-13T17:08:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.680+0000 7fb8d1400640 1 -- 172.21.15.28:0/2775039834 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc006b10 con 0x7fb8cc053880 2024-09-13T17:08:17.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f14ce400640 1 Processor -- start 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 -- 172.21.15.28:0/390235046 shutdown_connections 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 --2- 172.21.15.28:0/390235046 >> v2:172.21.15.28:3300/0 conn(0x7f9604055180 0x7f9604035ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 --2- 172.21.15.28:0/390235046 >> v2:172.21.15.189:3300/0 conn(0x7f9604007040 0x7f9604036530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 --2- 172.21.15.28:0/390235046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604054c80 0x7f960400a250 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 -- 172.21.15.28:0/390235046 >> 172.21.15.28:0/390235046 conn(0x7f960401d900 msgr2=0x7f9604054570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f9612a00640 1 -- 172.21.15.28:0/390235046 shutdown_connections 2024-09-13T17:08:17.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.681+0000 7f14ce400640 1 -- start start 2024-09-13T17:08:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7f9612a00640 1 -- 172.21.15.28:0/390235046 wait complete. 2024-09-13T17:08:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7ff597200640 1 -- 172.21.15.28:0/1250067457 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 msgr2=0x7ff588053f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7ff597200640 1 --2- 172.21.15.28:0/1250067457 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 0x7ff588053f10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff590000b60 tx=0x7ff5900a4510 comp rx=0 tx=0).stop 2024-09-13T17:08:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7fb8d1400640 1 --2- 172.21.15.28:0/2775039834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc053880 0x7fb8cc053ca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00097a0 tx=0x7fb8c002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dc235cadbabc40a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7f14fce00640 1 --2- 172.21.15.28:0/4179512030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009870 tx=0x7f14e802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ffcf15a2e9f7620 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7ff597200640 1 -- 172.21.15.28:0/1250067457 shutdown_connections 2024-09-13T17:08:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7ff597200640 1 --2- 172.21.15.28:0/1250067457 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff58801dc00 0x7ff588053f10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7ff597200640 1 -- 172.21.15.28:0/1250067457 >> 172.21.15.28:0/1250067457 conn(0x7ff588036050 msgr2=0x7ff588036460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7fb8d0a00640 1 -- 172.21.15.28:0/2775039834 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0032440 con 0x7fb8cc053880 2024-09-13T17:08:17.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7fb8d0a00640 1 -- 172.21.15.28:0/2775039834 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c00325a0 con 0x7fb8cc053880 2024-09-13T17:08:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.682+0000 7fb8d0a00640 1 -- 172.21.15.28:0/2775039834 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0032930 con 0x7fb8cc053880 2024-09-13T17:08:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7ff597200640 1 -- 172.21.15.28:0/1250067457 shutdown_connections 2024-09-13T17:08:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14f7400640 1 -- 172.21.15.28:0/4179512030 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8032440 con 0x7f14f815cb30 2024-09-13T17:08:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14f7400640 1 -- 172.21.15.28:0/4179512030 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e80325a0 con 0x7f14f815cb30 2024-09-13T17:08:17.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14f7400640 1 -- 172.21.15.28:0/4179512030 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e8032930 con 0x7f14f815cb30 2024-09-13T17:08:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7ff597200640 1 -- 172.21.15.28:0/1250067457 wait complete. 2024-09-13T17:08:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14ce400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14ce400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c815ccf0 con 0x7f14c80b41c0 2024-09-13T17:08:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14ce400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c80ccd90 con 0x7f14c80b5870 2024-09-13T17:08:17.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.683+0000 7f14ce400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80cdf40 con 0x7f14c80cd7a0 2024-09-13T17:08:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7f14cd000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f14c80b41c0 0x7f14c80b45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34436/0 (socket says 172.21.15.28:34436) 2024-09-13T17:08:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7f14cd000640 1 -- 172.21.15.28:0/2543953482 learned_addr learned my addr 172.21.15.28:0/2543953482 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc053880 msgr2=0x7fb8cc053ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7f9612a00640 1 Processor -- start 2024-09-13T17:08:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7fb8d8c00640 1 --2- 172.21.15.28:0/2775039834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc053880 0x7fb8cc053ca0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb8c00097a0 tx=0x7fb8c002a230 comp rx=0 tx=0).stop 2024-09-13T17:08:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873120055 0 0) 0x7f14c815ccf0 con 0x7f14c80b41c0 2024-09-13T17:08:17.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.684+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14ac003520 con 0x7f14c80b41c0 2024-09-13T17:08:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 shutdown_connections 2024-09-13T17:08:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7fb8d8c00640 1 --2- 172.21.15.28:0/2775039834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc053880 0x7fb8cc053ca0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 >> 172.21.15.28:0/2775039834 conn(0x7fb8cc036050 msgr2=0x7fb8cc036460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7e00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257929469 0 0) 0x7f14c80ccd90 con 0x7f14c80b5870 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c815ccf0 con 0x7f14c80b5870 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3334358631 0 0) 0x7f14ac003520 con 0x7f14c80b41c0 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c80cd7a0 msgr2=0x7f14c80b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c80cd7a0 0x7f14c80b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 >> v1:172.21.15.189:6789/0 conn(0x7f14c80b5870 legacy=0x7f14c80d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c80cd070 con 0x7f14c80b41c0 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f14c00019f0 con 0x7f14c80b41c0 2024-09-13T17:08:17.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 >> v1:172.21.15.28:6789/0 conn(0x7f14c80b41c0 legacy=0x7f14c80b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14ac003b30 0x7f14ac003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14ac0047c0 0x7f14ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac005180 0x7f14ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80cdf40 con 0x7f14ac005180 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00bfd0 con 0x7f14ac003b30 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14ac00c190 con 0x7f14ac0047c0 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 msgr2=0x7f14f80cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f14fe200640 1 --2- 172.21.15.28:0/4179512030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f14e8009870 tx=0x7f14e802a250 comp rx=0 tx=0).stop 2024-09-13T17:08:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7f9612a00640 1 -- start start 2024-09-13T17:08:17.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.685+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 shutdown_connections 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.686+0000 7f14cd000640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14ac003b30 0x7f14ac003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.686+0000 7f14c7e00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14ac0047c0 0x7f14ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.686+0000 7f14cda00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac005180 0x7f14ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14c7e00640 1 -- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14ac003b30 msgr2=0x7f14ac003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14c7e00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14ac003b30 0x7f14ac003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14c7e00640 1 -- 172.21.15.28:0/2543953482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac005180 msgr2=0x7f14ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14c7e00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ac005180 0x7f14ac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f9612a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 0x7f96040e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14c7e00640 1 -- 172.21.15.28:0/2543953482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ac003520 con 0x7f14ac0047c0 2024-09-13T17:08:17.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7ff597200640 1 Processor -- start 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 shutdown_connections 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14fe200640 1 --2- 172.21.15.28:0/4179512030 >> v2:172.21.15.28:3300/0 conn(0x7f14f80d0480 0x7f14f80cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14fe200640 1 --2- 172.21.15.28:0/4179512030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f815cb30 0x7f14f80cdc00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 >> 172.21.15.28:0/4179512030 conn(0x7f14f8070ad0 msgr2=0x7f14f80b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.686+0000 7fb8d8c00640 1 -- 172.21.15.28:0/2775039834 wait complete. 2024-09-13T17:08:17.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 shutdown_connections 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7ff597200640 1 -- start start 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.687+0000 7f9611600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 0x7f96040e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9611600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 0x7f96040e52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44698/0 (socket says 172.21.15.28:44698) 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9611600640 1 -- 172.21.15.28:0/4099585575 learned_addr learned my addr 172.21.15.28:0/4099585575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9612a00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9604054c80 0x7f96040e3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9612a00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604055180 0x7f960404a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604049c30 con 0x7f9604054c80 2024-09-13T17:08:17.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9604049f10 con 0x7f9604007040 2024-09-13T17:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7fb8d8c00640 1 Processor -- start 2024-09-13T17:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f1a86e00640 1 Processor -- start 2024-09-13T17:08:17.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f1a86e00640 1 -- start start 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96040e3340 con 0x7f9604055180 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f9611600640 1 -- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604055180 msgr2=0x7f960404a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f9611600640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9604055180 0x7f960404a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f9611600640 1 -- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9604054c80 msgr2=0x7f96040e3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f8888000640 1 Processor -- start 2024-09-13T17:08:17.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7fb8d8c00640 1 -- start start 2024-09-13T17:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f8c9d200640 1 Processor -- start 2024-09-13T17:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f8888000640 1 -- start start 2024-09-13T17:08:17.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 0x7ff5880495a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58801dc00 0x7ff58817be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5880e70e0 0x7ff588039050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588048500 con 0x7ff58801dc00 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880487e0 con 0x7ff588005a20 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7ff597200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff588048ac0 con 0x7ff5880e70e0 2024-09-13T17:08:17.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f8c9d200640 1 -- start start 2024-09-13T17:08:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 0x7ff5880495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 0x7ff5880495a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44708/0 (socket says 172.21.15.28:44708) 2024-09-13T17:08:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 -- 172.21.15.28:0/2888784017 learned_addr learned my addr 172.21.15.28:0/2888784017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.689+0000 7f14c7e00640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14ac0047c0 0x7f14ac004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14c80b7e20 tx=0x7f14ac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83ec5e9196ba1d6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff595200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5880e70e0 0x7ff588039050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc01c7c0 0x7fb8cc0db140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b403b410 con 0x7f14ac0047c0 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc053880 0x7fb8cc04a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 0x7fb8cc0e0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc037cc0 con 0x7fb8cc053880 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0db680 con 0x7fb8cc01c7c0 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7fb8d8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc04ad40 con 0x7fb8cc0e7000 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f1a86e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a80159770 0x7f1a8006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f1a86e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a8015cd80 0x7f1a80070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f1a86e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8006f920 con 0x7f1a8006f3f0 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f1a86e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8015b290 con 0x7f1a80159770 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7f1a86e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8015b570 con 0x7f1a8015cd80 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 -- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5880e70e0 msgr2=0x7ff588039050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5880e70e0 0x7ff588039050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 -- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58801dc00 msgr2=0x7ff58817be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58801dc00 0x7ff58817be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.690+0000 7ff58fe00640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588048da0 con 0x7ff588005a20 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8888000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f888006d070 0x7f88800b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8888000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88801610a0 con 0x7f88800b17e0 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8888000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800b2620 con 0x7f888006c540 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.688+0000 7f14fe200640 1 -- 172.21.15.28:0/4179512030 wait complete. 2024-09-13T17:08:17.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14fe200640 1 Processor -- start 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8888000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88800b2900 con 0x7f888006d070 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f888006d070 0x7f88800b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f888006d070 0x7f88800b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44726/0 (socket says 172.21.15.28:44726) 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 -- 172.21.15.28:0/779643348 learned_addr learned my addr 172.21.15.28:0/779643348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 0x7fb8cc0e0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 0x7fb8cc0e0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55670/0 (socket says 172.21.15.28:55670) 2024-09-13T17:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 -- 172.21.15.28:0/568593636 learned_addr learned my addr 172.21.15.28:0/568593636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b4035030 con 0x7f14ac0047c0 2024-09-13T17:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 0x7f8c980cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 0x7f8c980cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a8015cd80 0x7f1a80070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a8015cd80 0x7f1a80070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44720/0 (socket says 172.21.15.28:44720) 2024-09-13T17:08:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 -- 172.21.15.28:0/1424690100 learned_addr learned my addr 172.21.15.28:0/1424690100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d1e00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc01c7c0 0x7fb8cc0db140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f9610c00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9604054c80 0x7f96040e3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f9611600640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9604054c80 0x7f96040e3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7ff58fe00640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 0x7ff5880495a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c4d0 tx=0x7ff58400c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c9815cea0 con 0x7f8c980cd020 2024-09-13T17:08:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c980ce230 con 0x7f8c9815cac0 2024-09-13T17:08:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c9d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980cd9d0 con 0x7f8c9815e1a0 2024-09-13T17:08:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c95e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f9611600640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96040e3620 con 0x7f9604007040 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c95e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44742/0 (socket says 172.21.15.28:44742) 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c95e00640 1 -- 172.21.15.28:0/1227083281 learned_addr learned my addr 172.21.15.28:0/1227083281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 -- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc01c7c0 msgr2=0x7fb8cc0db140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc01c7c0 0x7fb8cc0db140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 -- 172.21.15.28:0/568593636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc053880 msgr2=0x7fb8cc04a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc053880 0x7fb8cc04a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0009400 con 0x7fb8cc0e7000 2024-09-13T17:08:17.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8c95400640 1 --2- 172.21.15.28:0/1227083281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 0x7f8c980cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f9610c00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9604054c80 0x7f96040e3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:17.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 -- 172.21.15.28:0/779643348 >> v1:172.21.15.189:6789/0 conn(0x7f888006c540 legacy=0x7f888016a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 -- 172.21.15.28:0/779643348 >> v1:172.21.15.28:6789/0 conn(0x7f88800b17e0 legacy=0x7f88800b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f8884c00640 1 -- 172.21.15.28:0/779643348 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88800b2be0 con 0x7f888006d070 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584007480 con 0x7ff588005a20 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff584007600 con 0x7ff588005a20 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584004a20 con 0x7ff588005a20 2024-09-13T17:08:17.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7fb8d2800640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 0x7fb8cc0e0980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c400c460 tx=0x7fb8c400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 -- 172.21.15.28:0/1424690100 >> v2:172.21.15.28:3300/0 conn(0x7f1a80159770 msgr2=0x7f1a8006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 --2- 172.21.15.28:0/1424690100 >> v2:172.21.15.28:3300/0 conn(0x7f1a80159770 0x7f1a8006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 -- 172.21.15.28:0/1424690100 >> v1:172.21.15.28:6789/0 conn(0x7f1a8006f3f0 legacy=0x7f1a80159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f1a84e00640 1 -- 172.21.15.28:0/1424690100 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8015b850 con 0x7f1a8015cd80 2024-09-13T17:08:17.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f9611600640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 0x7f96040e52e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f960c067e50 tx=0x7f960c0698a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff588049080 con 0x7ff588005a20 2024-09-13T17:08:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58817c540 con 0x7ff588005a20 2024-09-13T17:08:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4005480 con 0x7fb8cc0e7000 2024-09-13T17:08:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14c7400640 1 -- 172.21.15.28:0/2543953482 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b403b7b0 con 0x7f14ac0047c0 2024-09-13T17:08:17.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14ce400640 1 -- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14ac0047c0 msgr2=0x7f14ac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14ce400640 1 --2- 172.21.15.28:0/2543953482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14ac0047c0 0x7f14ac004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14c80b7e20 tx=0x7f14ac0057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14cda00640 1 -- 172.21.15.28:0/2543953482 reap_dead start 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14ce400640 1 -- 172.21.15.28:0/2543953482 shutdown_connections 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.692+0000 7f14ce400640 1 -- 172.21.15.28:0/2543953482 >> 172.21.15.28:0/2543953482 conn(0x7f14c80cf3a0 msgr2=0x7f14c80cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0b68c0 con 0x7f9604007040 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960c06ae90 con 0x7f9604007040 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960c0b6c20 con 0x7f9604007040 2024-09-13T17:08:17.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96040e4080 con 0x7f9604007040 2024-09-13T17:08:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c40035f0 con 0x7fb8cc0e7000 2024-09-13T17:08:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960404ac40 con 0x7f9604007040 2024-09-13T17:08:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f14ce400640 1 -- 172.21.15.28:0/2543953482 shutdown_connections 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f8c95400640 1 -- 172.21.15.28:0/1227083281 >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 msgr2=0x7f8c980cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7f8c96800640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 0x7f8c980cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f8c95400640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f8c95400640 1 -- 172.21.15.28:0/1227083281 >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 msgr2=0x7f8c980cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c40039a0 con 0x7fb8cc0e7000 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc037390 con 0x7fb8cc0e7000 2024-09-13T17:08:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.693+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc037950 con 0x7fb8cc0e7000 2024-09-13T17:08:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f8c95400640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 0x7f8c980cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f8c95400640 1 -- 172.21.15.28:0/1227083281 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c980cdc70 con 0x7f8c9815e1a0 2024-09-13T17:08:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f14ce400640 1 -- 172.21.15.28:0/2543953482 wait complete. 2024-09-13T17:08:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f960404a080 con 0x7f9604007040 2024-09-13T17:08:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.694+0000 7f14fe200640 1 -- start start 2024-09-13T17:08:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f80cf7a0 0x7f14f806f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f80d0480 0x7f14f81ff940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 0x7f14f815a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f80cabd0 con 0x7f14f815cb30 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f806fcd0 con 0x7f14f80d0480 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f14fe200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f81ffe80 con 0x7f14f80cf7a0 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f14f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f80d0480 0x7f14f81ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f14fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 0x7f14f815a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f14fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 0x7f14f815a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33468/0 (socket says 172.21.15.28:33468) 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f14fd800640 1 -- 172.21.15.28:0/2102179288 learned_addr learned my addr 172.21.15.28:0/2102179288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f14fce00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f80cf7a0 0x7f14f806f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.695+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f960c0b2050 con 0x7f9604007040 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff588048950 con 0x7ff588005a20 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f8c95e00640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f8c96800640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 0x7f8c980cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f8c95400640 1 --2- 172.21.15.28:0/1227083281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 0x7f8c980cfea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c980d3a70 tx=0x7f8c8402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8f83261d9892f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f1a84e00640 1 --2- 172.21.15.28:0/1424690100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a80070260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1a8006f380 tx=0x7f1a7802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d73eb3f74e8099 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f1a7ea00640 1 -- 172.21.15.28:0/1424690100 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78031600 con 0x7f1a8015cd80 2024-09-13T17:08:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f1a7ea00640 1 -- 172.21.15.28:0/1424690100 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a7802d050 con 0x7f1a8015cd80 2024-09-13T17:08:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.696+0000 7f1a7ea00640 1 -- 172.21.15.28:0/1424690100 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78031980 con 0x7f1a8015cd80 2024-09-13T17:08:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb8cc006d80 con 0x7fb8cc0e7000 2024-09-13T17:08:17.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f14ce400640 1 Processor -- start 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f8884c00640 1 --2- 172.21.15.28:0/779643348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006d070 0x7f88800b3480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8874009870 tx=0x7f887402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73a54d670469292 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9609600640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f95ec06ac00 0x7f95ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f960c123b90 con 0x7f9604007040 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9610c00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f95ec06ac00 0x7f95ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9609600640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7f95ec072da0 0x7f95ec0751c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95ec075870 con 0x7f95ec072da0 2024-09-13T17:08:17.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f960c1282a0 con 0x7f9604007040 2024-09-13T17:08:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f14ce400640 1 -- start start 2024-09-13T17:08:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb8c4003b00 con 0x7fb8cc0e7000 2024-09-13T17:08:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f8c94a00640 1 -- 172.21.15.28:0/1227083281 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8402ae00 con 0x7f8c9815e1a0 2024-09-13T17:08:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f8c94a00640 1 -- 172.21.15.28:0/1227083281 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c84032440 con 0x7f8c9815e1a0 2024-09-13T17:08:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f8c94a00640 1 -- 172.21.15.28:0/1227083281 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c84032810 con 0x7f8c9815e1a0 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f1a86e00640 1 -- 172.21.15.28:0/1424690100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 msgr2=0x7f1a80070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.697+0000 7f1a86e00640 1 --2- 172.21.15.28:0/1424690100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a80070260 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1a8006f380 tx=0x7f1a7802a660 comp rx=0 tx=0).stop 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f14fd800640 1 -- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f80cf7a0 msgr2=0x7f14f806f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f14fd800640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14f80cf7a0 0x7f14f806f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f14fd800640 1 -- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f80d0480 msgr2=0x7f14f81ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f14fd800640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14f80d0480 0x7f14f81ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f14fd800640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80094d0 con 0x7f14f815cb30 2024-09-13T17:08:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7f9612000640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7f95ec072da0 0x7f95ec0751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f887fe00640 1 -- 172.21.15.28:0/779643348 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874032440 con 0x7f888006d070 2024-09-13T17:08:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f887fe00640 1 -- 172.21.15.28:0/779643348 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88740325a0 con 0x7f888006d070 2024-09-13T17:08:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f887fe00640 1 -- 172.21.15.28:0/779643348 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874032930 con 0x7f888006d070 2024-09-13T17:08:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7fb8ca000640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8b406acd0 0x7fb8b406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8c4088920 con 0x7fb8cc0e7000 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7fb8ca000640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7fb8b4072e70 0x7fb8b4075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb8b4075940 con 0x7fb8b4072e70 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8c4007ea0 con 0x7fb8cc0e7000 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 -- 172.21.15.28:0/1227083281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 msgr2=0x7f8c980cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 --2- 172.21.15.28:0/1227083281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 0x7f8c980cfea0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c980d3a70 tx=0x7f8c8402a0b0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 -- 172.21.15.28:0/1227083281 shutdown_connections 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.28:3300/0 conn(0x7f8c980cd020 0x7f8c980cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 --2- 172.21.15.28:0/1227083281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c9815e1a0 0x7f8c980cfea0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 --2- 172.21.15.28:0/1227083281 >> v2:172.21.15.189:3300/0 conn(0x7f8c9815cac0 0x7f8c980cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f8c9d200640 1 -- 172.21.15.28:0/1227083281 >> 172.21.15.28:0/1227083281 conn(0x7f8c980708f0 msgr2=0x7f8c98070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f1a86e00640 1 -- 172.21.15.28:0/1424690100 shutdown_connections 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f1a86e00640 1 --2- 172.21.15.28:0/1424690100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a80070260 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f1a86e00640 1 --2- 172.21.15.28:0/1424690100 >> v2:172.21.15.28:3300/0 conn(0x7f1a80159770 0x7f1a8006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f1a86e00640 1 -- 172.21.15.28:0/1424690100 >> 172.21.15.28:0/1424690100 conn(0x7f1a800b0270 msgr2=0x7f1a800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.698+0000 7f1a86e00640 1 -- 172.21.15.28:0/1424690100 shutdown_connections 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7f1a86e00640 1 -- 172.21.15.28:0/1424690100 wait complete. 2024-09-13T17:08:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff584004b80 con 0x7ff588005a20 2024-09-13T17:08:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7f8c9d200640 1 -- 172.21.15.28:0/1227083281 shutdown_connections 2024-09-13T17:08:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.699+0000 7f8c9d200640 1 -- 172.21.15.28:0/1227083281 wait complete. 2024-09-13T17:08:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f8888000640 1 -- 172.21.15.28:0/779643348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006d070 msgr2=0x7f88800b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f8888000640 1 --2- 172.21.15.28:0/779643348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006d070 0x7f88800b3480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8874009870 tx=0x7f887402a270 comp rx=0 tx=0).stop 2024-09-13T17:08:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7fb8d1e00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8b406acd0 0x7fb8b406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f14fd800640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 0x7f14f815a8e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c7e0 tx=0x7f14f000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f8888000640 1 -- 172.21.15.28:0/779643348 shutdown_connections 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f8888000640 1 --2- 172.21.15.28:0/779643348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006d070 0x7f88800b3480 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f8888000640 1 -- 172.21.15.28:0/779643348 >> 172.21.15.28:0/779643348 conn(0x7f88800b0270 msgr2=0x7f88800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f8888000640 1 -- 172.21.15.28:0/779643348 shutdown_connections 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0003da0 con 0x7f14f815cb30 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f0005070 con 0x7f14f815cb30 2024-09-13T17:08:17.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0005420 con 0x7f14f815cb30 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f9610c00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f95ec06ac00 0x7f95ec06d0c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9600006fd0 tx=0x7f9600008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7ff583e00640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff57006ac30 0x7ff57006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff584088730 con 0x7ff588005a20 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7ff583e00640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7ff570072e50 0x7ff570075270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff570075920 con 0x7ff570072e50 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff584088ae0 con 0x7ff588005a20 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7ff58f400640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff57006ac30 0x7ff57006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7ff595200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7ff570072e50 0x7ff570075270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7fb8d1400640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7fb8b4072e70 0x7fb8b4075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7fb8d1400640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7fb8b4072e70 0x7fb8b4075290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f14ce400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 0x7f14c80abcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.700+0000 7f14ce400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c8151f40 0x7f14c80ac230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7f8c9d200640 1 Processor -- start 2024-09-13T17:08:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7f8c9d200640 1 -- start start 2024-09-13T17:08:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f1a86e00640 1 Processor -- start 2024-09-13T17:08:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14ce400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c80ab280 0x7f14c80a94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14ce400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80d2090 con 0x7f14c8151b60 2024-09-13T17:08:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14ce400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80aadf0 con 0x7f14c80ab280 2024-09-13T17:08:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14ce400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c80aa1e0 con 0x7f14c8151f40 2024-09-13T17:08:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.702+0000 7f14cd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 0x7f14c80abcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f9612000640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7f95ec072da0 0x7f95ec0751c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== osd.4 v2:172.21.15.189:6800/531106775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb8b4075940 con 0x7fb8b4072e70 2024-09-13T17:08:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f80c9fc0 con 0x7f14f815cb30 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f80ca580 con 0x7f14f815cb30 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14f80d08c0 con 0x7f14f815cb30 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== osd.1 v2:172.21.15.28:6808/2722659727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f95ec075870 con 0x7f95ec072da0 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7ff595200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7ff570072e50 0x7ff570075270 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== osd.2 v2:172.21.15.28:6816/2959928040 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff570075920 con 0x7ff570072e50 2024-09-13T17:08:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7ff58f400640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff57006ac30 0x7ff57006d0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c0079c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f8888000640 1 -- 172.21.15.28:0/779643348 wait complete. 2024-09-13T17:08:17.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f8888000640 1 Processor -- start 2024-09-13T17:08:17.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f8888000640 1 -- start start 2024-09-13T17:08:17.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7fb8d1e00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8b406acd0 0x7fb8b406d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4063b50 tx=0x7fb8d406c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f14c7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c8151f40 0x7f14c80ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f14cd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 0x7f14c80abcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33476/0 (socket says 172.21.15.28:33476) 2024-09-13T17:08:17.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f14cd000640 1 -- 172.21.15.28:0/1824864930 learned_addr learned my addr 172.21.15.28:0/1824864930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f14cda00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c80ab280 0x7f14c80a94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 -- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c8151f40 msgr2=0x7f14c80ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c8151f40 0x7f14c80ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 -- 172.21.15.28:0/1824864930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 msgr2=0x7f14c80abcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 0x7f14c80abcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c80aa4c0 con 0x7f14c80ab280 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14c7e00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f14c8151f40 0x7f14c80ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cd000640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14c8151b60 0x7f14c80abcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.705+0000 7f14cda00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c80ab280 0x7f14c80a94b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c80b2150 tx=0x7f14b800fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b8005030 con 0x7f14c80ab280 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14b8005190 con 0x7f14c80ab280 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14b80067d0 con 0x7f14c80ab280 2024-09-13T17:08:17.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.701+0000 7f1a86e00640 1 -- start start 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 0x7f1a800cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a80159770 0x7f1a80155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a8014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80152fa0 con 0x7f1a8006f3f0 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a80151f00 con 0x7f1a8015cd80 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.703+0000 7f1a86e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a801521e0 con 0x7f1a80159770 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 0x7f1a800cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 0x7f1a800cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33480/0 (socket says 172.21.15.28:33480) 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f1a7fe00640 1 -- 172.21.15.28:0/3279988069 learned_addr learned my addr 172.21.15.28:0/3279988069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f1a84e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a8014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f1a7fe00640 1 -- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a80159770 msgr2=0x7f1a80155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c80aa7a0 con 0x7f14c80ab280 2024-09-13T17:08:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c80a99f0 con 0x7f14c80ab280 2024-09-13T17:08:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.706+0000 7f1a7f400640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a80159770 0x7f1a80155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7fe00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a80159770 0x7f1a80155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7fe00640 1 -- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 msgr2=0x7f1a8014cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7fe00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a8014cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7fe00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a78009b30 con 0x7f1a8006f3f0 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a84e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a8015cd80 0x7f1a8014cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7f400640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a80159770 0x7f1a80155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.707+0000 7f1a7fe00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 0x7f1a800cc050 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000b4a0 tx=0x7f1a7000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.704+0000 7f8c9d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 0x7f8c981f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8888000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006c540 0x7f88801e8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c9d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c9815cac0 0x7f8c981ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c9d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c9815e1a0 0x7f8c980cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f14f0005580 con 0x7f14f815cb30 2024-09-13T17:08:17.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8888000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f888006d070 0x7f88801ea720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a70010610 con 0x7f1a8006f3f0 2024-09-13T17:08:17.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c9d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980cb090 con 0x7f8c9815e1a0 2024-09-13T17:08:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8888000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88800b17e0 0x7f8880154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8885600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006c540 0x7f88801e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a70010770 con 0x7f1a8006f3f0 2024-09-13T17:08:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a70010b20 con 0x7f1a8006f3f0 2024-09-13T17:08:17.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8885600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006c540 0x7f88801e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44780/0 (socket says 172.21.15.28:44780) 2024-09-13T17:08:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c9d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980cb370 con 0x7f8c9815cac0 2024-09-13T17:08:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c9d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980cb650 con 0x7f8c980cd020 2024-09-13T17:08:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c95e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 0x7f8c981f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c95e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 0x7f8c981f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55734/0 (socket says 172.21.15.28:55734) 2024-09-13T17:08:17.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.708+0000 7f8c95e00640 1 -- 172.21.15.28:0/738928235 learned_addr learned my addr 172.21.15.28:0/738928235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8885600640 1 -- 172.21.15.28:0/1827714470 learned_addr learned my addr 172.21.15.28:0/1827714470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88801eac60 con 0x7f888006d070 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8886000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88800b17e0 0x7f8880154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88801e97d0 con 0x7f888006c540 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88801e91c0 con 0x7f88800b17e0 2024-09-13T17:08:17.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f14f4c00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14d806acd0 0x7f14d806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a801524c0 con 0x7f1a8006f3f0 2024-09-13T17:08:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14f0089290 con 0x7f14f815cb30 2024-09-13T17:08:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f14fce00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14d806acd0 0x7f14d806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f14f4c00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f14d8072ef0 0x7f14d8075310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14c80d1cb0 con 0x7f14c80ab280 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f14b8006930 con 0x7f14c80ab280 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8c95e00640 1 -- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c9815cac0 msgr2=0x7f8c981ef8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8c95e00640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c9815cac0 0x7f8c981ef8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8c95e00640 1 -- 172.21.15.28:0/738928235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c9815e1a0 msgr2=0x7f8c980cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8c95e00640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c9815e1a0 0x7f8c980cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.709+0000 7f8c95e00640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c880094d0 con 0x7f8c980cd020 2024-09-13T17:08:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a80152a80 con 0x7f1a8006f3f0 2024-09-13T17:08:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14d80759c0 con 0x7f14d8072ef0 2024-09-13T17:08:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14f0089640 con 0x7f14f815cb30 2024-09-13T17:08:17.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14c4c00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14a806ac30 0x7f14a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.710+0000 7f14fce00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14d806acd0 0x7f14d806d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14e8034000 tx=0x7f14e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14b8088130 con 0x7f14c80ab280 2024-09-13T17:08:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14c4c00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f14a8072dd0 0x7f14a80751f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14a80758a0 con 0x7f14a8072dd0 2024-09-13T17:08:17.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14b8053660 con 0x7f14c80ab280 2024-09-13T17:08:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14f7e00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f14d8072ef0 0x7f14d8075310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.711+0000 7f14cd000640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14a806ac30 0x7f14a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f14c7e00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f14a8072dd0 0x7f14a80751f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a8006d160 con 0x7f1a8006f3f0 2024-09-13T17:08:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f8884c00640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f888006d070 0x7f88801ea720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f14cd000640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14a806ac30 0x7f14a806d0f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000f80 tx=0x7f14c00017f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f14c7e00640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f14a8072dd0 0x7f14a80751f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f8c95e00640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 0x7f8c981f2510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800e9d0 tx=0x7f8c8800ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.712+0000 7f14f7e00640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f14d8072ef0 0x7f14d8075310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8800ca60 con 0x7f8c980cd020 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8800cbc0 con 0x7f8c980cd020 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c88013470 con 0x7f8c980cd020 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c84009400 con 0x7f8c980cd020 2024-09-13T17:08:17.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8884c00640 1 -- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88800b17e0 msgr2=0x7f8880154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8884c00640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f88800b17e0 0x7f8880154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8884c00640 1 -- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006c540 msgr2=0x7f88801e8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8884c00640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f888006c540 0x7f88801e8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8884c00640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88740094d0 con 0x7f888006d070 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a70010c80 con 0x7f1a8006f3f0 2024-09-13T17:08:17.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.713+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c980cbc10 con 0x7f8c980cd020 2024-09-13T17:08:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.714+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== osd.5 v2:172.21.15.189:6808/1971214256 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f14a80758a0 con 0x7f14a8072dd0 2024-09-13T17:08:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.714+0000 7f8884c00640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f888006d070 0x7f88801ea720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8874031000 tx=0x7f88740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.714+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== osd.0 v2:172.21.15.28:6800/3372255106 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f14d80759c0 con 0x7f14d8072ef0 2024-09-13T17:08:17.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874030b80 con 0x7f888006d070 2024-09-13T17:08:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a77e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a6006ac80 0x7f1a6006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874030d20 con 0x7f888006d070 2024-09-13T17:08:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874004260 con 0x7f888006d070 2024-09-13T17:08:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a70088da0 con 0x7f1a8006f3f0 2024-09-13T17:08:17.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a77e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f1a60072ea0 0x7f1a600752c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a60075970 con 0x7f1a60072ea0 2024-09-13T17:08:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a7008b4b0 con 0x7f1a8006f3f0 2024-09-13T17:08:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8c980d1300 con 0x7f8c980cd020 2024-09-13T17:08:17.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f1a7f400640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a6006ac80 0x7f1a6006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.715+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88801eae90 con 0x7f888006d070 2024-09-13T17:08:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.716+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8880154d80 con 0x7f888006d070 2024-09-13T17:08:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.716+0000 7f1a7f400640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a6006ac80 0x7f1a6006d140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a80154660 tx=0x7f1a6c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.717+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c880135d0 con 0x7f8c980cd020 2024-09-13T17:08:17.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.717+0000 7f1a84e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f1a60072ea0 0x7f1a600752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.718+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f888006f3f0 con 0x7f888006d070 2024-09-13T17:08:17.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.718+0000 7f1a84e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f1a60072ea0 0x7f1a600752c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.718+0000 7f8c8e000640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c7806ac00 0x7f8c7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.718+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c88089840 con 0x7f8c980cd020 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f8c8e000640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f8c78072da0 0x7f8c780751c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8c78075870 con 0x7f8c78072da0 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8c8808be20 con 0x7f8c980cd020 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f8c95400640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c7806ac00 0x7f8c7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f8c96800640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f8c78072da0 0x7f8c780751c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.719+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8874039040 con 0x7f888006d070 2024-09-13T17:08:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.720+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== osd.7 v2:172.21.15.189:6822/2656371491 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f1a60075970 con 0x7f1a60072ea0 2024-09-13T17:08:17.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f887d600640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f886006ac50 0x7f886006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88740a9840 con 0x7f888006d070 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f887d600640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f8860072e70 0x7f8860075290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8860075940 con 0x7f8860072e70 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88740abe20 con 0x7f888006d070 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.720+0000 7f8c95400640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c7806ac00 0x7f8c7806d0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8c84034000 tx=0x7f8c8402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.722+0000 7f8886000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f8860072e70 0x7f8860075290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.722+0000 7f8885600640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f886006ac50 0x7f886006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.721+0000 7f8c96800640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f8c78072da0 0x7f8c780751c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.723+0000 7f8886000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f8860072e70 0x7f8860075290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.723+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== osd.6 v2:172.21.15.189:6816/2946605912 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f8c78075870 con 0x7f8c78072da0 2024-09-13T17:08:17.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.724+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== osd.3 v2:172.21.15.28:6814/3941122226 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f8860075940 con 0x7f8860072e70 2024-09-13T17:08:17.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.727+0000 7f8885600640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f886006ac50 0x7f886006d110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8878007fd0 tx=0x7f8878009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:17.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.739+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 --> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb8cc0e73e0 con 0x7fb8b4072e70 2024-09-13T17:08:17.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8ca000640 1 -- 172.21.15.28:0/568593636 <== osd.4 v2:172.21.15.189:6800/531106775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb8c00327f0 con 0x7fb8b4072e70 2024-09-13T17:08:17.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7fb8b4072e70 msgr2=0x7fb8b4075290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7fb8b4072e70 0x7fb8b4075290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8b406acd0 msgr2=0x7fb8b406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8b406acd0 0x7fb8b406d190 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8d4063b50 tx=0x7fb8d406c040 comp rx=0 tx=0).stop 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 msgr2=0x7fb8cc0e0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.740+0000 7fb8d8c00640 1 --2- 172.21.15.28:0/568593636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc0e7000 0x7fb8cc0e0980 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb8c400c460 tx=0x7fb8c400c920 comp rx=0 tx=0).stop 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.741+0000 7fb8d2800640 1 -- 172.21.15.28:0/568593636 reap_dead start 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.741+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 shutdown_connections 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.741+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 >> 172.21.15.28:0/568593636 conn(0x7fb8cc036050 msgr2=0x7fb8cc00a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.741+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 shutdown_connections 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.741+0000 7fb8d8c00640 1 -- 172.21.15.28:0/568593636 wait complete. 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.742+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 --> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5880539d0 con 0x7ff570072e50 2024-09-13T17:08:17.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff583e00640 1 -- 172.21.15.28:0/2888784017 <== osd.2 v2:172.21.15.28:6816/2959928040 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff59006ae90 con 0x7ff570072e50 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7ff570072e50 msgr2=0x7ff570075270 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7ff570072e50 0x7ff570075270 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff57006ac30 msgr2=0x7ff57006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff57006ac30 0x7ff57006d0f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff57c000c00 tx=0x7ff57c0079c0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 msgr2=0x7ff5880495a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 --2- 172.21.15.28:0/2888784017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff588005a20 0x7ff5880495a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff58400c4d0 tx=0x7ff58400c990 comp rx=0 tx=0).stop 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff595200640 1 -- 172.21.15.28:0/2888784017 reap_dead start 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 shutdown_connections 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.743+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 >> 172.21.15.28:0/2888784017 conn(0x7ff588036050 msgr2=0x7ff58804f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.745+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 --> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14f80b5a70 con 0x7f14d8072ef0 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.745+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 shutdown_connections 2024-09-13T17:08:17.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.745+0000 7ff597200640 1 -- 172.21.15.28:0/2888784017 wait complete. 2024-09-13T17:08:17.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.746+0000 7f14f4c00640 1 -- 172.21.15.28:0/2102179288 <== osd.0 v2:172.21.15.28:6800/3372255106 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f14e4009870 con 0x7f14d8072ef0 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f14d8072ef0 msgr2=0x7f14d8075310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f14d8072ef0 0x7f14d8075310 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14d806acd0 msgr2=0x7f14d806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14d806acd0 0x7f14d806d190 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f14e8034000 tx=0x7f14e802e000 comp rx=0 tx=0).stop 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 msgr2=0x7f14f815a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 --2- 172.21.15.28:0/2102179288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14f815cb30 0x7f14f815a8e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f14f000c7e0 tx=0x7f14f000cca0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fd800640 1 -- 172.21.15.28:0/2102179288 reap_dead start 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 shutdown_connections 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 >> 172.21.15.28:0/2102179288 conn(0x7f14f8070ad0 msgr2=0x7f14f80d16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 shutdown_connections 2024-09-13T17:08:17.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f14fe200640 1 -- 172.21.15.28:0/2102179288 wait complete. 2024-09-13T17:08:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.747+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f96040e31c0 con 0x7f95ec072da0 2024-09-13T17:08:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.749+0000 7f9609600640 1 -- 172.21.15.28:0/4099585575 <== osd.1 v2:172.21.15.28:6808/2722659727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f95f8009280 con 0x7f95ec072da0 2024-09-13T17:08:17.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.749+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 --> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14c80b5d20 con 0x7f14a8072dd0 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7f95ec072da0 msgr2=0x7f95ec0751c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7f95ec072da0 0x7f95ec0751c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f14c4c00640 1 -- 172.21.15.28:0/1824864930 <== osd.5 v2:172.21.15.189:6808/1971214256 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f14c80b5d20 con 0x7f14a8072dd0 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f95ec06ac00 msgr2=0x7f95ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f95ec06ac00 0x7f95ec06d0c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9600006fd0 tx=0x7f9600008040 comp rx=0 tx=0).stop 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 msgr2=0x7f96040e52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.750+0000 7f9612a00640 1 --2- 172.21.15.28:0/4099585575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9604007040 0x7f96040e52e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f960c067e50 tx=0x7f960c0698a0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f14a8072dd0 msgr2=0x7f14a80751f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f14a8072dd0 0x7f14a80751f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14a806ac30 msgr2=0x7f14a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f14a806ac30 0x7f14a806d0f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c0000f80 tx=0x7f14c00017f0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c80ab280 msgr2=0x7f14c80a94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 --2- 172.21.15.28:0/1824864930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14c80ab280 0x7f14c80a94b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f14c80b2150 tx=0x7f14b800fc00 comp rx=0 tx=0).stop 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14cda00640 1 -- 172.21.15.28:0/1824864930 reap_dead start 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f9612000640 1 -- 172.21.15.28:0/4099585575 reap_dead start 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 shutdown_connections 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 >> 172.21.15.28:0/4099585575 conn(0x7f960401d900 msgr2=0x7f96040049c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 shutdown_connections 2024-09-13T17:08:17.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 >> 172.21.15.28:0/1824864930 conn(0x7f14c80cf3a0 msgr2=0x7f14c80cb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.751+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 shutdown_connections 2024-09-13T17:08:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.752+0000 7f9612a00640 1 -- 172.21.15.28:0/4099585575 wait complete. 2024-09-13T17:08:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.752+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 shutdown_connections 2024-09-13T17:08:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.752+0000 7f14ce400640 1 -- 172.21.15.28:0/1824864930 wait complete. 2024-09-13T17:08:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.754+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 --> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a80152ca0 con 0x7f1a60072ea0 2024-09-13T17:08:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.755+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8c980d1020 con 0x7f8c78072da0 2024-09-13T17:08:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.755+0000 7f1a77e00640 1 -- 172.21.15.28:0/3279988069 <== osd.7 v2:172.21.15.189:6822/2656371491 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a80152ca0 con 0x7f1a60072ea0 2024-09-13T17:08:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 --> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8880154d80 con 0x7f8860072e70 2024-09-13T17:08:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.755+0000 7f8c8e000640 1 -- 172.21.15.28:0/738928235 <== osd.6 v2:172.21.15.189:6816/2946605912 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8c900079c0 con 0x7f8c78072da0 2024-09-13T17:08:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f1a60072ea0 msgr2=0x7f1a600752c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f1a60072ea0 0x7f1a600752c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f8c78072da0 msgr2=0x7f8c780751c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f8c9d200640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f8c78072da0 0x7f8c780751c0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a6006ac80 msgr2=0x7f1a6006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c7806ac00 msgr2=0x7f8c7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c7806ac00 0x7f8c7806d0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8c84034000 tx=0x7f8c8402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 msgr2=0x7f8c981f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 --2- 172.21.15.28:0/738928235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c980cd020 0x7f8c981f2510 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8c8800e9d0 tx=0x7f8c8800ee90 comp rx=0 tx=0).stop 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a6006ac80 0x7f1a6006d140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1a80154660 tx=0x7f1a6c008040 comp rx=0 tx=0).stop 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c96800640 1 -- 172.21.15.28:0/738928235 reap_dead start 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 shutdown_connections 2024-09-13T17:08:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 >> 172.21.15.28:0/738928235 conn(0x7f8c980708f0 msgr2=0x7f8c980d2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 msgr2=0x7f1a800cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a86e00640 1 --2- 172.21.15.28:0/3279988069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a8006f3f0 0x7f1a800cc050 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1a7000b4a0 tx=0x7f1a7000b960 comp rx=0 tx=0).stop 2024-09-13T17:08:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f887d600640 1 -- 172.21.15.28:0/1827714470 <== osd.3 v2:172.21.15.28:6814/3941122226 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8880154d80 con 0x7f8860072e70 2024-09-13T17:08:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.757+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 shutdown_connections 2024-09-13T17:08:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.756+0000 7f1a84e00640 1 -- 172.21.15.28:0/3279988069 reap_dead start 2024-09-13T17:08:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.758+0000 7f8c9d200640 1 -- 172.21.15.28:0/738928235 wait complete. 2024-09-13T17:08:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.758+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 shutdown_connections 2024-09-13T17:08:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.758+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 >> 172.21.15.28:0/3279988069 conn(0x7f1a800b0270 msgr2=0x7f1a8015a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.758+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 shutdown_connections 2024-09-13T17:08:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.758+0000 7f1a86e00640 1 -- 172.21.15.28:0/3279988069 wait complete. 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.761+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f8860072e70 msgr2=0x7f8860075290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.761+0000 7f8888000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f8860072e70 0x7f8860075290 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.762+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f886006ac50 msgr2=0x7f886006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.762+0000 7f8888000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f886006ac50 0x7f886006d110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8878007fd0 tx=0x7f8878009040 comp rx=0 tx=0).stop 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.762+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f888006d070 msgr2=0x7f88801ea720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.762+0000 7f8888000640 1 --2- 172.21.15.28:0/1827714470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f888006d070 0x7f88801ea720 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8874031000 tx=0x7f88740046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.765+0000 7f8886000640 1 -- 172.21.15.28:0/1827714470 reap_dead start 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.765+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 shutdown_connections 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.765+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 >> 172.21.15.28:0/1827714470 conn(0x7f88800b0270 msgr2=0x7f88800ce4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.769+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 shutdown_connections 2024-09-13T17:08:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:17.769+0000 7f8888000640 1 -- 172.21.15.28:0/1827714470 wait complete. 2024-09-13T17:08:17.799 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:17.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T17:08:17.800 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:17.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T17:08:17.801 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:17.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T17:08:17.802 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:17.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T17:08:17.803 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:17.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T17:08:17.804 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:17.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T17:08:17.805 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:17.805 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T17:08:17.806 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:17.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T17:08:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.110+0000 7f5255000640 1 Processor -- start 2024-09-13T17:08:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.111+0000 7f5255000640 1 -- start start 2024-09-13T17:08:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f970ac00640 1 Processor -- start 2024-09-13T17:08:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f5255000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f524800a4b0 0x7f5248035fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f970ac00640 1 -- start start 2024-09-13T17:08:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f5255000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f524d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.113+0000 7f524d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55752/0 (socket says 172.21.15.28:55752) 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f5255000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52480395f0 con 0x7f5248007030 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f5255000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52480398d0 con 0x7f524800a4b0 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f5255000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5248039bb0 con 0x7f5248055170 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f524d400640 1 -- 172.21.15.28:0/1002819926 learned_addr learned my addr 172.21.15.28:0/1002819926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f524de00640 1 --2- 172.21.15.28:0/1002819926 >> v2:172.21.15.28:3300/0 conn(0x7f524800a4b0 0x7f5248035fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f524d400640 1 -- 172.21.15.28:0/1002819926 >> v2:172.21.15.28:3300/0 conn(0x7f524800a4b0 msgr2=0x7f5248035fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f524d400640 1 --2- 172.21.15.28:0/1002819926 >> v2:172.21.15.28:3300/0 conn(0x7f524800a4b0 0x7f5248035fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.114+0000 7f524d400640 1 -- 172.21.15.28:0/1002819926 >> v1:172.21.15.28:6789/0 conn(0x7f5248007030 legacy=0x7f52480059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f524d400640 1 -- 172.21.15.28:0/1002819926 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248039e90 con 0x7f5248055170 2024-09-13T17:08:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f970ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970406a970 0x7f970406cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f970ac00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f970406b2b0 0x7f970406b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f970ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9704159c70 con 0x7f97040b31c0 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f970ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970406bc30 con 0x7f970406a970 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.115+0000 7f970ac00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9704159190 con 0x7f970406b2b0 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.116+0000 7f9703e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970406a970 0x7f970406cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.116+0000 7f9703e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970406a970 0x7f970406cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33504/0 (socket says 172.21.15.28:33504) 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.116+0000 7f9703e00640 1 -- 172.21.15.28:0/2835091209 learned_addr learned my addr 172.21.15.28:0/2835091209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.116+0000 7f524d400640 1 --2- 172.21.15.28:0/1002819926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52500634d0 tx=0x7f52500accf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a594f39915c1245 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f9703e00640 1 -- 172.21.15.28:0/2835091209 >> v2:172.21.15.189:3300/0 conn(0x7f970406b2b0 msgr2=0x7f970406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f9708c00640 1 --2- 172.21.15.28:0/2835091209 >> v2:172.21.15.189:3300/0 conn(0x7f970406b2b0 0x7f970406b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f9703e00640 1 --2- 172.21.15.28:0/2835091209 >> v2:172.21.15.189:3300/0 conn(0x7f970406b2b0 0x7f970406b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f9703e00640 1 -- 172.21.15.28:0/2835091209 >> v1:172.21.15.28:6789/0 conn(0x7f97040b31c0 legacy=0x7f9704158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f9703e00640 1 -- 172.21.15.28:0/2835091209 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704159470 con 0x7f970406a970 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f524ca00640 1 -- 172.21.15.28:0/1002819926 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5250065ea0 con 0x7f5248055170 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f524ca00640 1 -- 172.21.15.28:0/1002819926 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f525006ae90 con 0x7f5248055170 2024-09-13T17:08:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.117+0000 7f524ca00640 1 -- 172.21.15.28:0/1002819926 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52500adab0 con 0x7f5248055170 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 -- 172.21.15.28:0/1002819926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 msgr2=0x7f5248036520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 --2- 172.21.15.28:0/1002819926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f52500634d0 tx=0x7f52500accf0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 -- 172.21.15.28:0/1002819926 shutdown_connections 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 --2- 172.21.15.28:0/1002819926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248055170 0x7f5248036520 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 --2- 172.21.15.28:0/1002819926 >> v2:172.21.15.28:3300/0 conn(0x7f524800a4b0 0x7f5248035fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.118+0000 7f5255000640 1 -- 172.21.15.28:0/1002819926 >> 172.21.15.28:0/1002819926 conn(0x7f524801d8f0 msgr2=0x7f5248054560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.119+0000 7f9703e00640 1 --2- 172.21.15.28:0/2835091209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f970406cac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9704159090 tx=0x7f96f002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8aaed7a38f6f51 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.120+0000 7f9702a00640 1 -- 172.21.15.28:0/2835091209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0032640 con 0x7f970406a970 2024-09-13T17:08:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.120+0000 7f9702a00640 1 -- 172.21.15.28:0/2835091209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f00327a0 con 0x7f970406a970 2024-09-13T17:08:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.120+0000 7f9702a00640 1 -- 172.21.15.28:0/2835091209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f0032b30 con 0x7f970406a970 2024-09-13T17:08:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.121+0000 7f970ac00640 1 -- 172.21.15.28:0/2835091209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 msgr2=0x7f970406cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.121+0000 7f970ac00640 1 --2- 172.21.15.28:0/2835091209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f970406cac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9704159090 tx=0x7f96f002a300 comp rx=0 tx=0).stop 2024-09-13T17:08:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.120+0000 7f5255000640 1 -- 172.21.15.28:0/1002819926 shutdown_connections 2024-09-13T17:08:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.120+0000 7f5255000640 1 -- 172.21.15.28:0/1002819926 wait complete. 2024-09-13T17:08:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.122+0000 7f8802000640 1 Processor -- start 2024-09-13T17:08:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f5255000640 1 Processor -- start 2024-09-13T17:08:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f5255000640 1 -- start start 2024-09-13T17:08:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f970ac00640 1 -- 172.21.15.28:0/2835091209 shutdown_connections 2024-09-13T17:08:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f970ac00640 1 --2- 172.21.15.28:0/2835091209 >> v2:172.21.15.189:3300/0 conn(0x7f970406b2b0 0x7f970406b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f970ac00640 1 --2- 172.21.15.28:0/2835091209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f970406cac0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.123+0000 7f970ac00640 1 -- 172.21.15.28:0/2835091209 >> 172.21.15.28:0/2835091209 conn(0x7f9704069e80 msgr2=0x7f970406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f970ac00640 1 -- 172.21.15.28:0/2835091209 shutdown_connections 2024-09-13T17:08:18.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.122+0000 7f8802000640 1 -- start start 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f8802000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f87f4037800 0x7f87f400c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f8802000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f400ad00 con 0x7f87f40df9f0 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f8802000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f4008d30 con 0x7f87f40343a0 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f8802000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f400afa0 con 0x7f87f4037800 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.124+0000 7f87fb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f87f40df9f0 0x7f87f4008140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34496/0 (socket says 172.21.15.28:34496) 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f87fb400640 1 -- 172.21.15.28:0/3873876545 learned_addr learned my addr 172.21.15.28:0/3873876545 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 0x7f524804a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 0x7f524804b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f524804aa10 con 0x7f524804a560 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52480d74b0 con 0x7f5248055170 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f5255000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f524804ce00 con 0x7f5248007030 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f0217800640 1 Processor -- start 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f524e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 0x7f524804b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f524de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f524de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55754/0 (socket says 172.21.15.28:55754) 2024-09-13T17:08:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f524de00640 1 -- 172.21.15.28:0/1145089947 learned_addr learned my addr 172.21.15.28:0/1145089947 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497087356 0 0) 0x7f87f400ad00 con 0x7f87f40df9f0 2024-09-13T17:08:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87e4003520 con 0x7f87f40df9f0 2024-09-13T17:08:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f524d400640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 0x7f524804a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fbe00640 1 --2- 172.21.15.28:0/3873876545 >> v2:172.21.15.189:3300/0 conn(0x7f87f4037800 0x7f87f400c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 596560358 0 0) 0x7f87e4003520 con 0x7f87f40df9f0 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 >> v2:172.21.15.189:3300/0 conn(0x7f87f4037800 msgr2=0x7f87f400c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 --2- 172.21.15.28:0/3873876545 >> v2:172.21.15.189:3300/0 conn(0x7f87f4037800 0x7f87f400c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 >> v1:172.21.15.189:6789/0 conn(0x7f87f40343a0 legacy=0x7f87f40370f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.126+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4009010 con 0x7f87f40df9f0 2024-09-13T17:08:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f524e800640 1 -- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 msgr2=0x7f5248173cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f87e80016c0 con 0x7f87f40df9f0 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 >> v1:172.21.15.28:6789/0 conn(0x7f87f40df9f0 legacy=0x7f87f4008140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e4003970 0x7f87e4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e4005380 0x7f87e400fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f524e800640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f400afa0 con 0x7f87e4005380 2024-09-13T17:08:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87faa00640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f87fbe00640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e4005380 0x7f87e400fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4010630 con 0x7f87e4003970 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e40107f0 con 0x7f87e40047c0 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f0217800640 1 -- start start 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f0217800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0210159cc0 0x7f021006cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f0217800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f021006f3f0 con 0x7f02100b1570 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f0217800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100b3070 con 0x7f02100b1950 2024-09-13T17:08:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f0217800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02100b3350 con 0x7f0210159cc0 2024-09-13T17:08:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.125+0000 7f970ac00640 1 -- 172.21.15.28:0/2835091209 wait complete. 2024-09-13T17:08:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f970ac00640 1 Processor -- start 2024-09-13T17:08:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87fb400640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e4003970 0x7f87e4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f0214e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f02100b1570 0x7f021006bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34508/0 (socket says 172.21.15.28:34508) 2024-09-13T17:08:18.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f0214e00640 1 -- 172.21.15.28:0/1126242065 learned_addr learned my addr 172.21.15.28:0/1126242065 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f524e800640 1 -- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 msgr2=0x7f524804a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.127+0000 7f524e800640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 0x7f524804a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f524e800640 1 -- 172.21.15.28:0/1145089947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5248174200 con 0x7f524804a560 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f524de00640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f524d400640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 0x7f524804a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:18.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f020fe00640 1 --2- 172.21.15.28:0/1126242065 >> v2:172.21.15.28:3300/0 conn(0x7f0210159cc0 0x7f021006cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87faa00640 1 -- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e4003970 msgr2=0x7f87e4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87faa00640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e4003970 0x7f87e4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87faa00640 1 -- 172.21.15.28:0/3873876545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e4005380 msgr2=0x7f87e400fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87faa00640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87e4005380 0x7f87e400fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.128+0000 7f87faa00640 1 -- 172.21.15.28:0/3873876545 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4003520 con 0x7f87e40047c0 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.129+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379206867 0 0) 0x7f02100b3070 con 0x7f02100b1950 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.129+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f4003440 con 0x7f02100b1950 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.129+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431137193 0 0) 0x7f01f4003440 con 0x7f02100b1950 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 >> v2:172.21.15.28:3300/0 conn(0x7f0210159cc0 msgr2=0x7f021006cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 --2- 172.21.15.28:0/1126242065 >> v2:172.21.15.28:3300/0 conn(0x7f0210159cc0 0x7f021006cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f524e800640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 0x7f524804b5a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f523c00c470 tx=0x7f523c00c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f87fbe00640 1 -- 172.21.15.28:0/3873876545 reap_dead start 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f87fb400640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e4003970 0x7f87e4003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 >> v1:172.21.15.28:6789/0 conn(0x7f02100b1570 legacy=0x7f021006bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100b3630 con 0x7f02100b1950 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f0200001670 con 0x7f02100b1950 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 >> v1:172.21.15.189:6789/0 conn(0x7f02100b1950 legacy=0x7f02100b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01f4003a60 0x7f01f4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f4006690 0x7f01f4008b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 0x7f01f4009870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02100b3350 con 0x7f01f4006690 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f400a150 con 0x7f01f40093c0 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f400a310 con 0x7f01f4003a60 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f87faa00640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87e4004c40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009860 tx=0x7f87e40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6abc3232a8180fd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f970ac00640 1 -- start start 2024-09-13T17:08:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c011790 con 0x7f524804a560 2024-09-13T17:08:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f0215800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 0x7f01f4009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f0214e00640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01f4003a60 0x7f01f4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7fba69600640 1 Processor -- start 2024-09-13T17:08:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.130+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f002aa10 con 0x7f87e40047c0 2024-09-13T17:08:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f002ab90 con 0x7f87e40047c0 2024-09-13T17:08:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f87fa000640 1 -- 172.21.15.28:0/3873876545 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f0032440 con 0x7f87e40047c0 2024-09-13T17:08:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7fba69600640 1 -- start start 2024-09-13T17:08:18.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c0118f0 con 0x7f524804a560 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c011ca0 con 0x7f524804a560 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f97040cc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 0x7f9704145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 0x7f97040aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97040ccf30 con 0x7f970406a970 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97040cdd70 con 0x7f97040b31c0 2024-09-13T17:08:18.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f970ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97040cd160 con 0x7f970406b2b0 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f9703400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 0x7f9704145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f9708c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 0x7f97040aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f9703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f97040cc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f9708c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 0x7f97040aaa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44838/0 (socket says 172.21.15.28:44838) 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f9708c00640 1 -- 172.21.15.28:0/4069737822 learned_addr learned my addr 172.21.15.28:0/4069737822 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.131+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524804c1f0 con 0x7f524804a560 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524804c7b0 con 0x7f524804a560 2024-09-13T17:08:18.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f020fe00640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f4006690 0x7f01f4008b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f0215800640 1 -- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01f4003a60 msgr2=0x7f01f4003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f0215800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f01f4003a60 0x7f01f4003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f0215800640 1 -- 172.21.15.28:0/1126242065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f4006690 msgr2=0x7f01f4008b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f0215800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f4006690 0x7f01f4008b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f0215800640 1 -- 172.21.15.28:0/1126242065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100b3630 con 0x7f01f40093c0 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7f9703400640 1 -- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 msgr2=0x7f97040aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7f9703400640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 0x7f97040aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7f9703400640 1 -- 172.21.15.28:0/4069737822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 msgr2=0x7f97040cc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7f9703400640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f97040cc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f8802000640 1 -- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 msgr2=0x7f87e4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.132+0000 7f8802000640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87e4004c40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f87f0009860 tx=0x7f87e40057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7f0215800640 1 -- 172.21.15.28:0/1126242065 reap_dead start 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7fba69600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fba5c0070e0 0x7fba5c058350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7fba69600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba5c0536d0 0x7fba5c005ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7fba69600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c0e1a20 con 0x7fba5c0e7f90 2024-09-13T17:08:18.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7fba69600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0565e0 con 0x7fba5c0536d0 2024-09-13T17:08:18.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.133+0000 7fba69600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0568c0 con 0x7fba5c0070e0 2024-09-13T17:08:18.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f8802000640 1 -- 172.21.15.28:0/3873876545 shutdown_connections 2024-09-13T17:08:18.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f8802000640 1 --2- 172.21.15.28:0/3873876545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87e4004c40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f8802000640 1 -- 172.21.15.28:0/3873876545 >> 172.21.15.28:0/3873876545 conn(0x7f87f401ea00 msgr2=0x7f87f4055670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f0215800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 0x7f01f4009870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f020000c0e0 tx=0x7f01f40036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc1d55636345339 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f9703400640 1 -- 172.21.15.28:0/4069737822 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f00094d0 con 0x7f970406b2b0 2024-09-13T17:08:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7fba63400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fba5c0e7f90 0x7fba5c052fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:38124/0 (socket says 172.21.15.28:38124) 2024-09-13T17:08:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7fba63400640 1 -- 172.21.15.28:0/2021661932 learned_addr learned my addr 172.21.15.28:0/2021661932 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:18.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f9703400640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 0x7f9704145f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c390 tx=0x7f96f400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524804cfb0 con 0x7f524804a560 2024-09-13T17:08:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f8802000640 1 -- 172.21.15.28:0/3873876545 shutdown_connections 2024-09-13T17:08:18.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7fba63e00640 1 --2- 172.21.15.28:0/2021661932 >> v2:172.21.15.189:3300/0 conn(0x7fba5c0070e0 0x7fba5c058350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200031600 con 0x7f01f40093c0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020002d050 con 0x7f01f40093c0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.134+0000 7f020f400640 1 -- 172.21.15.28:0/1126242065 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02000319c0 con 0x7f01f40093c0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f40052d0 con 0x7f970406b2b0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4005430 con 0x7f970406b2b0 2024-09-13T17:08:18.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f970ac00640 1 -- 172.21.15.28:0/4069737822 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97040cd440 con 0x7f970406b2b0 2024-09-13T17:08:18.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f8802000640 1 -- 172.21.15.28:0/3873876545 wait complete. 2024-09-13T17:08:18.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96f4003830 con 0x7f970406b2b0 2024-09-13T17:08:18.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7fba68c00640 1 --2- 172.21.15.28:0/2021661932 >> v2:172.21.15.28:3300/0 conn(0x7fba5c0536d0 0x7fba5c005ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.135+0000 7f970ac00640 1 -- 172.21.15.28:0/4069737822 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97040cda00 con 0x7f970406b2b0 2024-09-13T17:08:18.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.136+0000 7f3f33c00640 1 Processor -- start 2024-09-13T17:08:18.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.136+0000 7f3f33c00640 1 -- start start 2024-09-13T17:08:18.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d8005250 con 0x7f970406b2b0 2024-09-13T17:08:18.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba62a00640 1 -- 172.21.15.28:0/2021661932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939723303 0 0) 0x7fba5c0e1a20 con 0x7fba5c0e7f90 2024-09-13T17:08:18.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba62a00640 1 -- 172.21.15.28:0/2021661932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba4c003520 con 0x7fba5c0e7f90 2024-09-13T17:08:18.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7f0217800640 1 -- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 msgr2=0x7f01f4009870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7f0217800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 0x7f01f4009870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f020000c0e0 tx=0x7f01f40036f0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f523c01c460 con 0x7f524804a560 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.138+0000 7f0217800640 1 -- 172.21.15.28:0/1126242065 shutdown_connections 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.138+0000 7f0217800640 1 --2- 172.21.15.28:0/1126242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f01f40093c0 0x7f01f4009870 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.138+0000 7f0217800640 1 -- 172.21.15.28:0/1126242065 >> 172.21.15.28:0/1126242065 conn(0x7f0210069e80 msgr2=0x7f021006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.138+0000 7f0217800640 1 -- 172.21.15.28:0/1126242065 shutdown_connections 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.138+0000 7f3f33c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f2c06d4e0 0x7f3f2c158760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f33c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c071700 con 0x7f3f2c15c3e0 2024-09-13T17:08:18.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f33c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f2c071420 con 0x7f3f2c0b2580 2024-09-13T17:08:18.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f33c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f2c06f500 con 0x7f3f2c06d4e0 2024-09-13T17:08:18.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba63e00640 1 -- 172.21.15.28:0/2021661932 >> v2:172.21.15.28:3300/0 conn(0x7fba5c0536d0 msgr2=0x7fba5c005ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba63e00640 1 --2- 172.21.15.28:0/2021661932 >> v2:172.21.15.28:3300/0 conn(0x7fba5c0536d0 0x7fba5c005ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7fe9e7e00640 1 Processor -- start 2024-09-13T17:08:18.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f5246000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 0x7f523006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f523c0890c0 con 0x7f524804a560 2024-09-13T17:08:18.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f31200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f2c06d4e0 0x7f3f2c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba63e00640 1 -- 172.21.15.28:0/2021661932 >> v1:172.21.15.189:6789/0 conn(0x7fba5c0e7f90 legacy=0x7fba5c052fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.137+0000 7fba63e00640 1 -- 172.21.15.28:0/2021661932 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c056ba0 con 0x7fba5c0070e0 2024-09-13T17:08:18.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f31200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f2c06d4e0 0x7f3f2c158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33554/0 (socket says 172.21.15.28:33554) 2024-09-13T17:08:18.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f3f31200640 1 -- 172.21.15.28:0/359360641 learned_addr learned my addr 172.21.15.28:0/359360641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7fe9e7e00640 1 -- start start 2024-09-13T17:08:18.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7fba63e00640 1 --2- 172.21.15.28:0/2021661932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c058350 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba540097a0 tx=0x7fba5402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d71163b3110ed7d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7f524de00640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 0x7f523006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.139+0000 7f0217800640 1 -- 172.21.15.28:0/1126242065 wait complete. 2024-09-13T17:08:18.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f96f4003990 con 0x7f970406b2b0 2024-09-13T17:08:18.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7f8802000640 1 Processor -- start 2024-09-13T17:08:18.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7f8802000640 1 -- start start 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7fba62a00640 1 -- 172.21.15.28:0/2021661932 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5402ae90 con 0x7fba5c0070e0 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7fba62a00640 1 -- 172.21.15.28:0/2021661932 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba54032440 con 0x7fba5c0070e0 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7fba62a00640 1 -- 172.21.15.28:0/2021661932 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba540327d0 con 0x7fba5c0070e0 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7f3f31200640 1 -- 172.21.15.28:0/359360641 >> v1:172.21.15.189:6789/0 conn(0x7f3f2c0b2580 legacy=0x7f3f2c06c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7f3f31200640 1 -- 172.21.15.28:0/359360641 >> v1:172.21.15.28:6789/0 conn(0x7f3f2c15c3e0 legacy=0x7f3f2c06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.140+0000 7f3f31200640 1 -- 172.21.15.28:0/359360641 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f2c06f7e0 con 0x7f3f2c06d4e0 2024-09-13T17:08:18.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f3f31200640 1 --2- 172.21.15.28:0/359360641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c06d4e0 0x7f3f2c158760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009e50 tx=0x7f3f2402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caba969c9504ff2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f96fbe00640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 0x7f96e406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.141+0000 7f524de00640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 0x7f523006d1c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5240009a10 tx=0x7f5240008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f0217800640 1 Processor -- start 2024-09-13T17:08:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f96f4035020 con 0x7f970406b2b0 2024-09-13T17:08:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fba69600640 1 -- 172.21.15.28:0/2021661932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 msgr2=0x7fba5c058350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fba69600640 1 --2- 172.21.15.28:0/2021661932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c058350 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fba540097a0 tx=0x7fba5402a180 comp rx=0 tx=0).stop 2024-09-13T17:08:18.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fe9e7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe9e80b6eb0 0x7fe9e8167460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f0217800640 1 -- start start 2024-09-13T17:08:18.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fe9e7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80d19b0 con 0x7fe9e815ca40 2024-09-13T17:08:18.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f8802000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87f40dab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fe9e7e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80d1c90 con 0x7fe9e80cf980 2024-09-13T17:08:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7f9703e00640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 0x7f96e406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7f8802000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 0x7f87f404d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 -- 172.21.15.28:0/2021661932 shutdown_connections 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 --2- 172.21.15.28:0/2021661932 >> v2:172.21.15.28:3300/0 conn(0x7fba5c0536d0 0x7fba5c005ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 --2- 172.21.15.28:0/2021661932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c058350 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 -- 172.21.15.28:0/2021661932 >> 172.21.15.28:0/2021661932 conn(0x7fba5c03a4f0 msgr2=0x7fba5c03a900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.142+0000 7fe9e7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80d1f70 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 -- 172.21.15.28:0/2021661932 shutdown_connections 2024-09-13T17:08:18.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fe9e6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9e815ca40 0x7fe9e80cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34532/0 (socket says 172.21.15.28:34532) 2024-09-13T17:08:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fe9e6a00640 1 -- 172.21.15.28:0/2803424246 learned_addr learned my addr 172.21.15.28:0/2803424246 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7f37a0c00640 1 Processor -- start 2024-09-13T17:08:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fe9e7400640 1 --2- 172.21.15.28:0/2803424246 >> v2:172.21.15.189:3300/0 conn(0x7fe9e80b6eb0 0x7fe9e8167460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7f37a0c00640 1 -- start start 2024-09-13T17:08:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7f8802000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f404c6b0 0x7f87f404d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f8802000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f404cc70 con 0x7f87f40487a0 2024-09-13T17:08:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f8802000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4053c60 con 0x7f87f404c6b0 2024-09-13T17:08:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f8802000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4053f40 con 0x7f87e40047c0 2024-09-13T17:08:18.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7fe9e7400640 1 -- 172.21.15.28:0/2803424246 >> v1:172.21.15.189:6789/0 conn(0x7fe9e80cf980 legacy=0x7fe9e80d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7f87fb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87f40dab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87fb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87f40dab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55782/0 (socket says 172.21.15.28:55782) 2024-09-13T17:08:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 0x7f87f404d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87fb400640 1 -- 172.21.15.28:0/1849505840 learned_addr learned my addr 172.21.15.28:0/1849505840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f9703e00640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 0x7f96e406d1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f96f0034000 tx=0x7f96f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f0217800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 0x7f0210148820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f0217800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02101e7bc0 0x7f02101fe240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f0217800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 0x7f021015a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 -- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 msgr2=0x7f87f40dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f0214e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 0x7f0210148820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7fe9e7400640 1 -- 172.21.15.28:0/2803424246 >> v1:172.21.15.28:6789/0 conn(0x7fe9e815ca40 legacy=0x7fe9e80cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7fe9e7400640 1 -- 172.21.15.28:0/2803424246 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80d2250 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.143+0000 7fba69600640 1 -- 172.21.15.28:0/2021661932 wait complete. 2024-09-13T17:08:18.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87f40dab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 -- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f404c6b0 msgr2=0x7f87f404d8a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f404c6b0 0x7f87f404d8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f87faa00640 1 -- 172.21.15.28:0/1849505840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4054220 con 0x7f87f40487a0 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f87faa00640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 0x7f87f404d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f87f0031000 tx=0x7f87f0004700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.144+0000 7f0214e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 0x7f0210148820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33576/0 (socket says 172.21.15.28:33576) 2024-09-13T17:08:18.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f0214e00640 1 -- 172.21.15.28:0/3898649297 learned_addr learned my addr 172.21.15.28:0/3898649297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f37a0c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 0x7f379c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f37a0c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f379c0d0070 0x7f379c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f37a0c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f379c15cea0 con 0x7f379c15cac0 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f37a0c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379c0ce230 con 0x7f379c0d0070 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f37a0c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c0cdc70 con 0x7f379c15e1a0 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f3799e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f379c15cac0 0x7f379c0cf960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:38142/0 (socket says 172.21.15.28:38142) 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f3799e00640 1 -- 172.21.15.28:0/1143634471 learned_addr learned my addr 172.21.15.28:0/1143634471 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f0215800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 0x7f021015a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f020fe00640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02101e7bc0 0x7f02101fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021015aa60 con 0x7f01f40093c0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0210202cd0 con 0x7f021015a0a0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.145+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02102020c0 con 0x7f02101e7bc0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7fe9e7400640 1 --2- 172.21.15.28:0/2803424246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e8167460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009dc0 tx=0x7fe9e002a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52c3299acab5b34c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f379a800640 1 --2- 172.21.15.28:0/1143634471 >> v2:172.21.15.189:3300/0 conn(0x7f379c0d0070 0x7f379c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f3798a00640 1 -- 172.21.15.28:0/1143634471 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941880862 0 0) 0x7f379c15cea0 con 0x7f379c15cac0 2024-09-13T17:08:18.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f3798a00640 1 -- 172.21.15.28:0/1143634471 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003520 con 0x7f379c15cac0 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f0215800640 1 -- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02101e7bc0 msgr2=0x7f02101fe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f0215800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02101e7bc0 0x7f02101fe240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f0215800640 1 -- 172.21.15.28:0/3898649297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 msgr2=0x7f0210148820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f0215800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 0x7f0210148820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f3799400640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 0x7f379c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f379a800640 1 -- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 msgr2=0x7f379c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f379a800640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 0x7f379c0cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f379a800640 1 -- 172.21.15.28:0/1143634471 >> v1:172.21.15.189:6789/0 conn(0x7f379c15cac0 legacy=0x7f379c0cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f379a800640 1 -- 172.21.15.28:0/1143634471 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379c0cdf50 con 0x7f379c0d0070 2024-09-13T17:08:18.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7f0215800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02102023a0 con 0x7f021015a0a0 2024-09-13T17:08:18.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7fe9e5600640 1 -- 172.21.15.28:0/2803424246 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0007740 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7fe9e5600640 1 -- 172.21.15.28:0/2803424246 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e002d050 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.146+0000 7fe9e5600640 1 -- 172.21.15.28:0/2803424246 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0007ac0 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f0215800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 0x7f021015a520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f021006d160 tx=0x7f0200000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f3799400640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 0x7f379c0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f96f4053bd0 con 0x7f970406b2b0 2024-09-13T17:08:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0200003b40 con 0x7f021015a0a0 2024-09-13T17:08:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0200003ce0 con 0x7f021015a0a0 2024-09-13T17:08:18.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020003b670 con 0x7f021015a0a0 2024-09-13T17:08:18.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2803424246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 msgr2=0x7fe9e8167460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2803424246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e8167460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009dc0 tx=0x7fe9e002a730 comp rx=0 tx=0).stop 2024-09-13T17:08:18.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0210202680 con 0x7f021015a0a0 2024-09-13T17:08:18.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.147+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0210148d60 con 0x7f021015a0a0 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2803424246 shutdown_connections 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2803424246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e8167460 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2803424246 >> 172.21.15.28:0/2803424246 conn(0x7fe9e8070ad0 msgr2=0x7fe9e80b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2803424246 shutdown_connections 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2803424246 wait complete. 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7fba69600640 1 Processor -- start 2024-09-13T17:08:18.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7fba69600640 1 -- start start 2024-09-13T17:08:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021006cc20 con 0x7f021015a0a0 2024-09-13T17:08:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7fe9e7e00640 1 Processor -- start 2024-09-13T17:08:18.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fe9e7e00640 1 -- start start 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0200039040 con 0x7f021015a0a0 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba5c0536d0 0x7fba5c0d9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c0e7f90 0x7fba5c03c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0e2d30 con 0x7fba5c0e7f90 2024-09-13T17:08:18.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0e8640 con 0x7fba5c0070e0 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7fba69600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c0e8920 con 0x7fba5c0536d0 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9e80cf980 0x7fe9e80cc620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e814b340 con 0x7fe9e815bca0 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80cd930 con 0x7fe9e80b6eb0 2024-09-13T17:08:18.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80cd090 con 0x7fe9e80cf980 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.148+0000 7f379a800640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c0d0070 0x7f379c0cd560 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3794009d90 tx=0x7f379402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a28e7107e119f59 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7f3798a00640 1 -- 172.21.15.28:0/1143634471 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794031600 con 0x7f379c0d0070 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7f3798a00640 1 -- 172.21.15.28:0/1143634471 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379402d050 con 0x7f379c0d0070 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.149+0000 7f3798a00640 1 -- 172.21.15.28:0/1143634471 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794031980 con 0x7f379c0d0070 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 -- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c0d0070 msgr2=0x7f379c0cd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c0d0070 0x7f379c0cd560 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3794009d90 tx=0x7f379402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 -- 172.21.15.28:0/1143634471 shutdown_connections 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c0d0070 0x7f379c0cd560 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 --2- 172.21.15.28:0/1143634471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c15e1a0 0x7f379c0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.150+0000 7f37a0c00640 1 -- 172.21.15.28:0/1143634471 >> 172.21.15.28:0/1143634471 conn(0x7f379c0708f0 msgr2=0x7f379c070d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f37a0c00640 1 -- 172.21.15.28:0/1143634471 shutdown_connections 2024-09-13T17:08:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f524804cfb0 con 0x7f524804a560 2024-09-13T17:08:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f003c610 con 0x7f87f40487a0 2024-09-13T17:08:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87f002aa20 con 0x7f87f40487a0 2024-09-13T17:08:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87f003c970 con 0x7f87f40487a0 2024-09-13T17:08:18.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33588/0 (socket says 172.21.15.28:33588) 2024-09-13T17:08:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7fe9e7400640 1 -- 172.21.15.28:0/2966495523 learned_addr learned my addr 172.21.15.28:0/2966495523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f020cc00640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 0x7f01f006d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02000a8de0 con 0x7f021015a0a0 2024-09-13T17:08:18.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e6a00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f37a0c00640 1 -- 172.21.15.28:0/1143634471 wait complete. 2024-09-13T17:08:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.151+0000 7f3f2b400640 1 -- 172.21.15.28:0/359360641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24031600 con 0x7f3f2c06d4e0 2024-09-13T17:08:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f3f2b400640 1 -- 172.21.15.28:0/359360641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f2402d050 con 0x7f3f2c06d4e0 2024-09-13T17:08:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f3f2b400640 1 -- 172.21.15.28:0/359360641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24031980 con 0x7f3f2c06d4e0 2024-09-13T17:08:18.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e7400640 1 -- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9e80cf980 msgr2=0x7fe9e80cc620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e7400640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9e80cf980 0x7fe9e80cc620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e7400640 1 -- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 msgr2=0x7fe9e814ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e7400640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e814ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7fe9e7400640 1 -- 172.21.15.28:0/2966495523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0009a70 con 0x7fe9e815bca0 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.153+0000 7fe9e7400640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009ee0 tx=0x7fe9e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f0214e00640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 0x7f01f006d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.153+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e00319d0 con 0x7fe9e815bca0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.153+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e0031b30 con 0x7fe9e815bca0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e80cd370 con 0x7fe9e815bca0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0045460 con 0x7fe9e815bca0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba5c0536d0 0x7fba5c0d9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44898/0 (socket says 172.21.15.28:44898) 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 -- 172.21.15.28:0/1213849454 learned_addr learned my addr 172.21.15.28:0/1213849454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 -- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba5c0536d0 msgr2=0x7fba5c0d9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba5c0536d0 0x7fba5c0d9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 -- 172.21.15.28:0/1213849454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c0e7f90 msgr2=0x7fba5c03c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c0e7f90 0x7fba5c03c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fba63e00640 1 -- 172.21.15.28:0/1213849454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba54009400 con 0x7fba5c0070e0 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.153+0000 7f37a0c00640 1 Processor -- start 2024-09-13T17:08:18.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f37a0c00640 1 -- start start 2024-09-13T17:08:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4054500 con 0x7f87f40487a0 2024-09-13T17:08:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.152+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4054a20 con 0x7f87f40487a0 2024-09-13T17:08:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d8005250 con 0x7f87f40487a0 2024-09-13T17:08:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.154+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80c7b40 con 0x7fe9e815bca0 2024-09-13T17:08:18.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7fba63e00640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fba54035040 tx=0x7fba540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f3f33c00640 1 -- 172.21.15.28:0/359360641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c06d4e0 msgr2=0x7f3f2c158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f3f33c00640 1 --2- 172.21.15.28:0/359360641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c06d4e0 0x7f3f2c158760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3f24009e50 tx=0x7f3f2402a710 comp rx=0 tx=0).stop 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f3f33c00640 1 -- 172.21.15.28:0/359360641 shutdown_connections 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f3f33c00640 1 --2- 172.21.15.28:0/359360641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c06d4e0 0x7f3f2c158760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.155+0000 7f3f33c00640 1 -- 172.21.15.28:0/359360641 >> 172.21.15.28:0/359360641 conn(0x7f3f2c0b0270 msgr2=0x7f3f2c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e80d0cf0 con 0x7fe9e815bca0 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f379c15cac0 0x7f379c14acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 0x7f379c0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c14b200 con 0x7f379c15cac0 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c152ed0 con 0x7f379c15e1a0 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f37a0c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379c153190 con 0x7f379c0d0070 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55826/0 (socket says 172.21.15.28:55826) 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f0214e00640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 0x7f01f006d140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02101e81a0 tx=0x7f0208009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 -- 172.21.15.28:0/3482451482 learned_addr learned my addr 172.21.15.28:0/3482451482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f379a800640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 0x7f379c0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 -- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 msgr2=0x7f379c0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 0x7f379c0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 -- 172.21.15.28:0/3482451482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f379c15cac0 msgr2=0x7f379c14acc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:08:18.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f379c15cac0 0x7f379c14acc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f3799e00640 1 -- 172.21.15.28:0/3482451482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794009a70 con 0x7f379c0d0070 2024-09-13T17:08:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.156+0000 7f379a800640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 0x7f379c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba54004230 con 0x7fba5c0070e0 2024-09-13T17:08:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba54004390 con 0x7fba5c0070e0 2024-09-13T17:08:18.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5403d910 con 0x7fba5c0070e0 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f3799e00640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c430 tx=0x7f378c00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c005ce0 con 0x7f379c0d0070 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c005e40 con 0x7f379c0d0070 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c0072d0 con 0x7f379c0d0070 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.158+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe9e003c420 con 0x7fe9e815bca0 2024-09-13T17:08:18.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.158+0000 7f3f33c00640 1 -- 172.21.15.28:0/359360641 shutdown_connections 2024-09-13T17:08:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f37a0c00640 1 -- 172.21.15.28:0/3482451482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c153470 con 0x7f379c0d0070 2024-09-13T17:08:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.158+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c0e8c00 con 0x7fba5c0070e0 2024-09-13T17:08:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.158+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c0e91c0 con 0x7fba5c0070e0 2024-09-13T17:08:18.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.158+0000 7f37a0c00640 1 -- 172.21.15.28:0/3482451482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c153a30 con 0x7f379c0d0070 2024-09-13T17:08:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.157+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f02000740e0 con 0x7f021015a0a0 2024-09-13T17:08:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.160+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c057680 con 0x7fba5c0070e0 2024-09-13T17:08:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.160+0000 7fe9dea00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 0x7fe9c806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.160+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9e00abba0 con 0x7fe9e815bca0 2024-09-13T17:08:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.159+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3770005250 con 0x7f379c0d0070 2024-09-13T17:08:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.162+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f378c007430 con 0x7f379c0d0070 2024-09-13T17:08:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.162+0000 7f3f33c00640 1 -- 172.21.15.28:0/359360641 wait complete. 2024-09-13T17:08:18.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.163+0000 7fe9e6a00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 0x7fe9c806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.163+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f87f0039050 con 0x7f87f40487a0 2024-09-13T17:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.164+0000 7f3792000640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 0x7f377c06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.164+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f378c088dd0 con 0x7f379c0d0070 2024-09-13T17:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.164+0000 7f3799400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 0x7f377c06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7fe9e6a00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 0x7fe9c806d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4005880 tx=0x7fe9d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.163+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fba54039040 con 0x7fba5c0070e0 2024-09-13T17:08:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7fba53e00640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 0x7fba4806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7f87ef400640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 0x7f87dc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7f3f33c00640 1 Processor -- start 2024-09-13T17:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba540a9c80 con 0x7fba5c0070e0 2024-09-13T17:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.165+0000 7fba63400640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 0x7fba4806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.166+0000 7f3f33c00640 1 -- start start 2024-09-13T17:08:18.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.166+0000 7f3799400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 0x7f377c06d140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f379c150d50 tx=0x7f3788008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.167+0000 7fba63400640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 0x7fba4806d1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba5c04f720 tx=0x7fba58007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.167+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f87f00a9a20 con 0x7f87f40487a0 2024-09-13T17:08:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.167+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f87f0074710 con 0x7f87f40487a0 2024-09-13T17:08:18.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.167+0000 7f87fb400640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 0x7f87dc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 0x7f3f2c153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c15c3e0 0x7f3f2c146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 0x7f3f2c0aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f31200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 0x7f3f2c153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f87fb400640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 0x7f87dc06d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f87e8007cc0 tx=0x7f87e8007ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c147520 con 0x7f3f2c15c3e0 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0a86f0 con 0x7f3f2c157300 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f33c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f2c0a95d0 con 0x7f3f2c0b2580 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f31200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 0x7f3f2c153fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55830/0 (socket says 172.21.15.28:55830) 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f31200640 1 -- 172.21.15.28:0/796642231 learned_addr learned my addr 172.21.15.28:0/796642231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f31c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 0x7f3f2c0aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.168+0000 7f3f2be00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c15c3e0 0x7f3f2c146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.169+0000 7f3f31c00640 1 -- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 msgr2=0x7f3f2c153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.169+0000 7f3f31c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 0x7f3f2c153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.169+0000 7f3f31c00640 1 -- 172.21.15.28:0/796642231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c15c3e0 msgr2=0x7f3f2c146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.169+0000 7f3f31c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c15c3e0 0x7f3f2c146fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.169+0000 7f3f31c00640 1 -- 172.21.15.28:0/796642231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f24009b30 con 0x7f3f2c157300 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.170+0000 7f3f31c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 0x7f3f2c0aa610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f180041c0 tx=0x7f3f18014420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.171+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18017070 con 0x7f3f2c157300 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.171+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f18014b30 con 0x7f3f2c157300 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.171+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f18005030 con 0x7f3f2c157300 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.171+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fba540b0080 con 0x7fba5c0070e0 2024-09-13T17:08:18.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.171+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f378c0540b0 con 0x7f379c0d0070 2024-09-13T17:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.172+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f2c0a98b0 con 0x7f3f2c157300 2024-09-13T17:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.172+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f2c157790 con 0x7f3f2c157300 2024-09-13T17:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.175+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3f18005190 con 0x7f3f2c157300 2024-09-13T17:08:18.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.176+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe9e0076290 con 0x7fe9e815bca0 2024-09-13T17:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.177+0000 7f3f28c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 0x7f3f0c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.177+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f18090bd0 con 0x7f3f2c157300 2024-09-13T17:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.178+0000 7f3f31200640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 0x7f3f0c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.179+0000 7f3f31200640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 0x7f3f0c06d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000f80 tx=0x7f3f24009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.182+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f2c15d0d0 con 0x7f3f2c157300 2024-09-13T17:08:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.188+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3f1805bed0 con 0x7f3f2c157300 2024-09-13T17:08:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.546+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f524804cfb0 con 0x7f524804a560 2024-09-13T17:08:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.548+0000 7f5246000640 1 -- 172.21.15.28:0/1145089947 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f524804cfb0 con 0x7f524804a560 2024-09-13T17:08:18.553 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T17:08:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.553+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 msgr2=0x7f523006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.553+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 0x7f523006d1c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f5240009a10 tx=0x7f5240008040 comp rx=0 tx=0).stop 2024-09-13T17:08:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.553+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 msgr2=0x7f524804b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.553+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 0x7f524804b5a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f523c00c470 tx=0x7f523c00c930 comp rx=0 tx=0).stop 2024-09-13T17:08:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 shutdown_connections 2024-09-13T17:08:18.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f523006ad00 0x7f523006d1c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f524804a560 0x7f524804b5a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5248055170 0x7f524804a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 --2- 172.21.15.28:0/1145089947 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5248007030 0x7f5248173cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.554+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 >> 172.21.15.28:0/1145089947 conn(0x7f524801d8f0 msgr2=0x7f5248032960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.555+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 shutdown_connections 2024-09-13T17:08:18.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.555+0000 7f5255000640 1 -- 172.21.15.28:0/1145089947 wait complete. 2024-09-13T17:08:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.565+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f96d8005500 con 0x7f970406b2b0 2024-09-13T17:08:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.566+0000 7f96fbe00640 1 -- 172.21.15.28:0/4069737822 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96f40571f0 con 0x7f970406b2b0 2024-09-13T17:08:18.569 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.569+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 msgr2=0x7f96e406d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.569+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 0x7f96e406d1c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f96f0034000 tx=0x7f96f002e000 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 msgr2=0x7f9704145f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 0x7f9704145f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96f400c390 tx=0x7f96f400c850 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 shutdown_connections 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e406ad00 0x7f96e406d1c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97040b31c0 0x7f97040aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970406b2b0 0x7f9704145f90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 --2- 172.21.15.28:0/4069737822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970406a970 0x7f97040cc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 >> 172.21.15.28:0/4069737822 conn(0x7f9704069e80 msgr2=0x7f97040aefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 shutdown_connections 2024-09-13T17:08:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.570+0000 7f96f9600640 1 -- 172.21.15.28:0/4069737822 wait complete. 2024-09-13T17:08:18.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.578+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f0210202e10 con 0x7f021015a0a0 2024-09-13T17:08:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.581+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fba5c037d30 con 0x7fba5c0070e0 2024-09-13T17:08:18.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.581+0000 7f020cc00640 1 -- 172.21.15.28:0/3898649297 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0200037020 con 0x7f021015a0a0 2024-09-13T17:08:18.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.4 2024-09-13T17:08:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.582+0000 7fba53e00640 1 -- 172.21.15.28:0/1213849454 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba54074e40 con 0x7fba5c0070e0 2024-09-13T17:08:18.584 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:18.584 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 msgr2=0x7fba4806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 0x7fba4806d1c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fba5c04f720 tx=0x7fba58007dc0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 msgr2=0x7fba5c03e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fba54035040 tx=0x7fba540046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 msgr2=0x7f01f006d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 0x7f01f006d140 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f02101e81a0 tx=0x7f0208009040 comp rx=0 tx=0).stop 2024-09-13T17:08:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 msgr2=0x7f021015a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 0x7f021015a520 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f021006d160 tx=0x7f0200000fe0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 shutdown_connections 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba4806ad00 0x7fba4806d1c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba5c0e7f90 0x7fba5c03c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba5c0536d0 0x7fba5c0d9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 --2- 172.21.15.28:0/1213849454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba5c0070e0 0x7fba5c03e260 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 >> 172.21.15.28:0/1213849454 conn(0x7fba5c03a4f0 msgr2=0x7fba5c004f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 shutdown_connections 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f01f006ac80 0x7f01f006d140 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f021015a0a0 0x7f021015a520 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f02101e7bc0 0x7f02101fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 --2- 172.21.15.28:0/3898649297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f01f40093c0 0x7f0210148820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 >> 172.21.15.28:0/3898649297 conn(0x7f0210069e80 msgr2=0x7f02100ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.585+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 shutdown_connections 2024-09-13T17:08:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.586+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 shutdown_connections 2024-09-13T17:08:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.586+0000 7fba69600640 1 -- 172.21.15.28:0/1213849454 wait complete. 2024-09-13T17:08:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.586+0000 7f0217800640 1 -- 172.21.15.28:0/3898649297 wait complete. 2024-09-13T17:08:18.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.589+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f37700057d0 con 0x7f379c0d0070 2024-09-13T17:08:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.589+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f87d80050c0 con 0x7f87f40487a0 2024-09-13T17:08:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.589+0000 7f3792000640 1 -- 172.21.15.28:0/3482451482 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f378c017090 con 0x7f379c0d0070 2024-09-13T17:08:18.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.590+0000 7f87ef400640 1 -- 172.21.15.28:0/1849505840 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f87f004c090 con 0x7f87f40487a0 2024-09-13T17:08:18.591 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:18.591 INFO:teuthology.orchestra.run.smithi028.stdout:55834574853 2024-09-13T17:08:18.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2024-09-13T17:08:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 msgr2=0x7f377c06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 0x7f377c06d140 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f379c150d50 tx=0x7f3788008040 comp rx=0 tx=0).stop 2024-09-13T17:08:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 msgr2=0x7f379c1f4b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f378c00c430 tx=0x7f378c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 msgr2=0x7f87dc06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 0x7f87dc06d190 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f87e8007cc0 tx=0x7f87e8007ef0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 msgr2=0x7f87f404d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 0x7f87f404d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f87f0031000 tx=0x7f87f0004700 comp rx=0 tx=0).stop 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 shutdown_connections 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f377c06ac80 0x7f377c06d140 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f379c15e1a0 0x7f379c0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f379c15cac0 0x7f379c14acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 --2- 172.21.15.28:0/3482451482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f379c0d0070 0x7f379c1f4b30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.593+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 >> 172.21.15.28:0/3482451482 conn(0x7f379c0708f0 msgr2=0x7f379c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 shutdown_connections 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f3787400640 1 -- 172.21.15.28:0/3482451482 wait complete. 2024-09-13T17:08:18.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 shutdown_connections 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f87dc06acd0 0x7f87dc06d190 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f87e8007cc0 tx=0x7f87e8007ef0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87f404c6b0 0x7f87f404d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87f40487a0 0x7f87f404d360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 --2- 172.21.15.28:0/1849505840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87e40047c0 0x7f87f40dab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 >> 172.21.15.28:0/1849505840 conn(0x7f87f401ea00 msgr2=0x7f87f40dda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 shutdown_connections 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.594+0000 7f8802000640 1 -- 172.21.15.28:0/1849505840 wait complete. 2024-09-13T17:08:18.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.596+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fe9e80c7d00 con 0x7fe9e815bca0 2024-09-13T17:08:18.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.596+0000 7fe9dea00640 1 -- 172.21.15.28:0/2966495523 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe9e00798b0 con 0x7fe9e815bca0 2024-09-13T17:08:18.598 INFO:teuthology.orchestra.run.smithi028.stdout:55834574852 2024-09-13T17:08:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 msgr2=0x7fe9c806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 0x7fe9c806d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4005880 tx=0x7fe9d4009040 comp rx=0 tx=0).stop 2024-09-13T17:08:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 msgr2=0x7fe9e8159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0009ee0 tx=0x7fe9e0004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:18.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 shutdown_connections 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe9c806ac30 0x7fe9c806d0f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9e815bca0 0x7fe9e8159d80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9e80cf980 0x7fe9e80cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 --2- 172.21.15.28:0/2966495523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9e80b6eb0 0x7fe9e814ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 >> 172.21.15.28:0/2966495523 conn(0x7fe9e8070ad0 msgr2=0x7fe9e80b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 shutdown_connections 2024-09-13T17:08:18.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.600+0000 7fe9e7e00640 1 -- 172.21.15.28:0/2966495523 wait complete. 2024-09-13T17:08:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.603+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3f2c15c050 con 0x7f3f2c157300 2024-09-13T17:08:18.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.604+0000 7f3f28c00640 1 -- 172.21.15.28:0/796642231 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3f18020030 con 0x7f3f2c157300 2024-09-13T17:08:18.605 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.607+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 msgr2=0x7f3f0c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.607+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 0x7f3f0c06d190 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000f80 tx=0x7f3f24009000 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.607+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 msgr2=0x7f3f2c0aa610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.607+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 0x7f3f2c0aa610 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3f180041c0 tx=0x7f3f18014420 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 shutdown_connections 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f0c06acd0 0x7f3f0c06d190 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3f24000f80 tx=0x7f3f24009000 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f2c157300 0x7f3f2c0aa610 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f2c15c3e0 0x7f3f2c146fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 --2- 172.21.15.28:0/796642231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f2c0b2580 0x7f3f2c153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:18.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.608+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 >> 172.21.15.28:0/796642231 conn(0x7f3f2c0b0270 msgr2=0x7f3f2c15ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.610+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 shutdown_connections 2024-09-13T17:08:18.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:18.610+0000 7f3f33c00640 1 -- 172.21.15.28:0/796642231 wait complete. 2024-09-13T17:08:18.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2024-09-13T17:08:18.616 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2024-09-13T17:08:18.623 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.0 2024-09-13T17:08:18.624 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2024-09-13T17:08:18.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-09-13T17:08:18.631 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.6 2024-09-13T17:08:19.584 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T17:08:19.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T17:08:19.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T17:08:19.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T17:08:19.624 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T17:08:19.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T17:08:19.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T17:08:19.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T17:08:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.834+0000 7f950a400640 1 Processor -- start 2024-09-13T17:08:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.834+0000 7f950a400640 1 -- start start 2024-09-13T17:08:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.836+0000 7f950a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95040cf980 0x7f95040d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.836+0000 7f950a400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95040d19b0 con 0x7f950415ca40 2024-09-13T17:08:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.836+0000 7f950a400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95040d1c90 con 0x7f95040d1230 2024-09-13T17:08:19.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.836+0000 7f950a400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95040d1f70 con 0x7f95040cf980 2024-09-13T17:08:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.838+0000 7f9509000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f950415ca40 0x7f95040cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34546/0 (socket says 172.21.15.28:34546) 2024-09-13T17:08:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.838+0000 7f9503e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95040cf980 0x7f95040d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.838+0000 7f9503e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f95040cf980 0x7f95040d0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33618/0 (socket says 172.21.15.28:33618) 2024-09-13T17:08:19.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.838+0000 7f9503e00640 1 -- 172.21.15.28:0/3466138086 learned_addr learned my addr 172.21.15.28:0/3466138086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.839+0000 7f9503400640 1 -- 172.21.15.28:0/3466138086 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133935434 0 0) 0x7f95040d1c90 con 0x7f95040d1230 2024-09-13T17:08:19.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.839+0000 7f9503400640 1 -- 172.21.15.28:0/3466138086 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e8003520 con 0x7f95040d1230 2024-09-13T17:08:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.840+0000 7f9503e00640 1 -- 172.21.15.28:0/3466138086 >> v1:172.21.15.189:6789/0 conn(0x7f95040d1230 legacy=0x7f95040b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.840+0000 7f9503e00640 1 -- 172.21.15.28:0/3466138086 >> v1:172.21.15.28:6789/0 conn(0x7f950415ca40 legacy=0x7f95040cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.840+0000 7f9509000640 1 --1- 172.21.15.28:0/3466138086 >> v1:172.21.15.28:6789/0 conn(0x7f950415ca40 0x7f95040cf270 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-09-13T17:08:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.840+0000 7f9503e00640 1 -- 172.21.15.28:0/3466138086 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95040d2250 con 0x7f95040cf980 2024-09-13T17:08:19.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.842+0000 7f9503e00640 1 --2- 172.21.15.28:0/3466138086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cf980 0x7f95040d0cf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009880 tx=0x7f94f402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c8b8d4949a3e317 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.844+0000 7f9503400640 1 -- 172.21.15.28:0/3466138086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4004030 con 0x7f95040cf980 2024-09-13T17:08:19.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.844+0000 7f9503400640 1 -- 172.21.15.28:0/3466138086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4004190 con 0x7f95040cf980 2024-09-13T17:08:19.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.844+0000 7f9503400640 1 -- 172.21.15.28:0/3466138086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4004030 con 0x7f95040cf980 2024-09-13T17:08:19.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.845+0000 7f950a400640 1 -- 172.21.15.28:0/3466138086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cf980 msgr2=0x7f95040d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.845+0000 7f950a400640 1 --2- 172.21.15.28:0/3466138086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cf980 0x7f95040d0cf0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009880 tx=0x7f94f402a2c0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.846+0000 7f950a400640 1 -- 172.21.15.28:0/3466138086 shutdown_connections 2024-09-13T17:08:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.846+0000 7f950a400640 1 --2- 172.21.15.28:0/3466138086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cf980 0x7f95040d0cf0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.846+0000 7f950a400640 1 -- 172.21.15.28:0/3466138086 >> 172.21.15.28:0/3466138086 conn(0x7f9504070ad0 msgr2=0x7f95040b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.846+0000 7f950a400640 1 -- 172.21.15.28:0/3466138086 shutdown_connections 2024-09-13T17:08:19.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.847+0000 7f950a400640 1 -- 172.21.15.28:0/3466138086 wait complete. 2024-09-13T17:08:19.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.850+0000 7f950a400640 1 Processor -- start 2024-09-13T17:08:19.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.851+0000 7f950a400640 1 -- start start 2024-09-13T17:08:19.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.853+0000 7f950a400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.853+0000 7f950a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.854+0000 7f9509000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.854+0000 7f9509000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55846/0 (socket says 172.21.15.28:55846) 2024-09-13T17:08:19.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.854+0000 7f9503e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.854+0000 7f9503e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44934/0 (socket says 172.21.15.28:44934) 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.854+0000 7f9509000640 1 -- 172.21.15.28:0/4247056560 learned_addr learned my addr 172.21.15.28:0/4247056560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.855+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 0x7f95040cca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.855+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95040ccfd0 con 0x7f95040cc5e0 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.855+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950415b9c0 con 0x7f95040d1230 2024-09-13T17:08:19.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.856+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95040ac980 con 0x7f95040cf980 2024-09-13T17:08:19.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.856+0000 7f9509a00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 0x7f95040cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.857+0000 7f9503e00640 1 -- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 msgr2=0x7f95040b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.857+0000 7f9503e00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.857+0000 7f9503e00640 1 -- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 msgr2=0x7f95040cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.857+0000 7f9503e00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 0x7f95040cca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.857+0000 7f9503e00640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f4009460 con 0x7f95040d1230 2024-09-13T17:08:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.858+0000 7f9509a00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 0x7f95040cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.858+0000 7f9509000640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:19.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.859+0000 7f9503e00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004000 tx=0x7f94f40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.860+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4004030 con 0x7f95040d1230 2024-09-13T17:08:19.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.860+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95040b21a0 con 0x7f95040d1230 2024-09-13T17:08:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.860+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95040aceb0 con 0x7f95040d1230 2024-09-13T17:08:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.860+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f4003c50 con 0x7f95040d1230 2024-09-13T17:08:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.860+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4006040 con 0x7f95040d1230 2024-09-13T17:08:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.862+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94dc005250 con 0x7f95040d1230 2024-09-13T17:08:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.865+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f94f40061a0 con 0x7f95040d1230 2024-09-13T17:08:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.866+0000 7f9500c00640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 0x7f94e006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.867+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94f40aa870 con 0x7f95040d1230 2024-09-13T17:08:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.868+0000 7f9509000640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 0x7f94e006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.869+0000 7f9509000640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 0x7f94e006d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00c550 tx=0x7f94fc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.874+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f94f4075b50 con 0x7f95040d1230 2024-09-13T17:08:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.917+0000 7f1c48c00640 1 Processor -- start 2024-09-13T17:08:19.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.917+0000 7f1c48c00640 1 -- start start 2024-09-13T17:08:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.918+0000 7f287d400640 1 Processor -- start 2024-09-13T17:08:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.918+0000 7f287d400640 1 -- start start 2024-09-13T17:08:19.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.919+0000 7fa52b200640 1 Processor -- start 2024-09-13T17:08:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.919+0000 7f1c48c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 0x7f1c4406fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.919+0000 7f1c48c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c44159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.919+0000 7f1c48c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c44071020 con 0x7f1c44070220 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f1c48c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c44070d40 con 0x7f1c4406d960 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f1c48c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c44159f70 con 0x7f1c4415b780 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f1c42a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1c44070220 0x7f1c44158c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34562/0 (socket says 172.21.15.28:34562) 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f1c42a00640 1 -- 172.21.15.28:0/1670787750 learned_addr learned my addr 172.21.15.28:0/1670787750 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:19.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f1c43e00640 1 --2- 172.21.15.28:0/1670787750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c44159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f287d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28780b17e0 0x7f28780b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f287d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f287806c540 0x7f287816a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f287d400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28781610a0 con 0x7f287806d070 2024-09-13T17:08:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f287d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780b2620 con 0x7f28780b17e0 2024-09-13T17:08:19.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7f287d400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28780b2900 con 0x7f287806c540 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c43400640 1 --2- 172.21.15.28:0/1670787750 >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 0x7f1c4406fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c43e00640 1 -- 172.21.15.28:0/1670787750 >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 msgr2=0x7f1c4406fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c42000640 1 -- 172.21.15.28:0/1670787750 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621660128 0 0) 0x7f1c44071020 con 0x7f1c44070220 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f2876800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28780b17e0 0x7f28780b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7f2876800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28780b17e0 0x7f28780b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33634/0 (socket says 172.21.15.28:33634) 2024-09-13T17:08:19.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7f2876800640 1 -- 172.21.15.28:0/209531483 learned_addr learned my addr 172.21.15.28:0/209531483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.920+0000 7fa52b200640 1 -- start start 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5240cd7a0 0x7fa5240b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52415ccf0 con 0x7fa5240b5870 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5240ccd90 con 0x7fa5240b41c0 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5240cdf40 con 0x7fa5240cd7a0 2024-09-13T17:08:19.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa529e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fa5240b41c0 0x7fa5240b45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:38176/0 (socket says 172.21.15.28:38176) 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7f2877200640 1 --2- 172.21.15.28:0/209531483 >> v2:172.21.15.189:3300/0 conn(0x7f287806c540 0x7f287816a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7f2876800640 1 -- 172.21.15.28:0/209531483 >> v2:172.21.15.189:3300/0 conn(0x7f287806c540 msgr2=0x7f287816a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7f2876800640 1 --2- 172.21.15.28:0/209531483 >> v2:172.21.15.189:3300/0 conn(0x7f287806c540 0x7f287816a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.923+0000 7f2876800640 1 -- 172.21.15.28:0/209531483 >> v1:172.21.15.189:6789/0 conn(0x7f287806d070 legacy=0x7f28780b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.923+0000 7f2876800640 1 -- 172.21.15.28:0/209531483 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28780b2be0 con 0x7f28780b17e0 2024-09-13T17:08:19.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c43e00640 1 --2- 172.21.15.28:0/1670787750 >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 0x7f1c4406fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c43e00640 1 -- 172.21.15.28:0/1670787750 >> v1:172.21.15.28:6789/0 conn(0x7f1c44070220 legacy=0x7f1c44158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.921+0000 7f1c43e00640 1 -- 172.21.15.28:0/1670787750 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4415a250 con 0x7f1c4415b780 2024-09-13T17:08:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7f2876800640 1 --2- 172.21.15.28:0/209531483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780b3480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2878071880 tx=0x7f287002a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e345d4506dfb2c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7f1c43400640 1 --2- 172.21.15.28:0/1670787750 >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 0x7f1c4406fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5240b5870 0x7fa5240d0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34570/0 (socket says 172.21.15.28:34570) 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa52a800640 1 -- 172.21.15.28:0/1624992260 learned_addr learned my addr 172.21.15.28:0/1624992260 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.922+0000 7fa529400640 1 --2- 172.21.15.28:0/1624992260 >> v2:172.21.15.189:3300/0 conn(0x7fa5240cd7a0 0x7fa5240b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.923+0000 7fa529400640 1 -- 172.21.15.28:0/1624992260 >> v1:172.21.15.189:6789/0 conn(0x7fa5240b41c0 legacy=0x7fa5240b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.923+0000 7fa528a00640 1 -- 172.21.15.28:0/1624992260 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966885420 0 0) 0x7fa52415ccf0 con 0x7fa5240b5870 2024-09-13T17:08:19.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.925+0000 7f1c43e00640 1 --2- 172.21.15.28:0/1670787750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c44159330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097a0 tx=0x7f1c3002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57f63883ec1942fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.925+0000 7fa340200640 1 Processor -- start 2024-09-13T17:08:19.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.925+0000 7fa340200640 1 -- start start 2024-09-13T17:08:19.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7f2875400640 1 -- 172.21.15.28:0/209531483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870031600 con 0x7f28780b17e0 2024-09-13T17:08:19.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7f2875400640 1 -- 172.21.15.28:0/209531483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287002d050 con 0x7f28780b17e0 2024-09-13T17:08:19.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7f2875400640 1 -- 172.21.15.28:0/209531483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870031980 con 0x7f28780b17e0 2024-09-13T17:08:19.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.926+0000 7f287d400640 1 -- 172.21.15.28:0/209531483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 msgr2=0x7f28780b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.926+0000 7f287d400640 1 --2- 172.21.15.28:0/209531483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780b3480 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2878071880 tx=0x7f287002a740 comp rx=0 tx=0).stop 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.926+0000 7f1c42000640 1 -- 172.21.15.28:0/1670787750 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032440 con 0x7f1c4415b780 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.926+0000 7f1c42000640 1 -- 172.21.15.28:0/1670787750 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c300325a0 con 0x7f1c4415b780 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.926+0000 7f1c42000640 1 -- 172.21.15.28:0/1670787750 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032440 con 0x7f1c4415b780 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7fa529400640 1 -- 172.21.15.28:0/1624992260 >> v1:172.21.15.28:6789/0 conn(0x7fa5240b5870 legacy=0x7fa5240d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.924+0000 7fa529400640 1 -- 172.21.15.28:0/1624992260 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5240cd070 con 0x7fa5240cd7a0 2024-09-13T17:08:19.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 -- 172.21.15.28:0/1670787750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 msgr2=0x7f1c44159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7fa340200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa330036730 0x7fa33004ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 --2- 172.21.15.28:0/1670787750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c44159330 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f1c300097a0 tx=0x7f1c3002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7fa33d800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa33000a4f0 0x7fa330036020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34572/0 (socket says 172.21.15.28:34572) 2024-09-13T17:08:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 -- 172.21.15.28:0/1670787750 shutdown_connections 2024-09-13T17:08:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 --2- 172.21.15.28:0/1670787750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c44159330 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 --2- 172.21.15.28:0/1670787750 >> v2:172.21.15.28:3300/0 conn(0x7f1c4406d960 0x7f1c4406fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7fa33d800640 1 -- 172.21.15.28:0/185362070 learned_addr learned my addr 172.21.15.28:0/185362070 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 -- 172.21.15.28:0/1670787750 >> 172.21.15.28:0/1670787750 conn(0x7f1c4406b360 msgr2=0x7f1c4406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7f1c48c00640 1 -- 172.21.15.28:0/1670787750 shutdown_connections 2024-09-13T17:08:19.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.927+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa330039630 con 0x7fa33000a4f0 2024-09-13T17:08:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f1c48c00640 1 -- 172.21.15.28:0/1670787750 wait complete. 2024-09-13T17:08:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 -- 172.21.15.28:0/209531483 shutdown_connections 2024-09-13T17:08:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 --2- 172.21.15.28:0/209531483 >> v2:172.21.15.189:3300/0 conn(0x7f287806c540 0x7f287816a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 --2- 172.21.15.28:0/209531483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780b3480 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 -- 172.21.15.28:0/209531483 >> 172.21.15.28:0/209531483 conn(0x7f28780b0270 msgr2=0x7f28780b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa330039910 con 0x7fa3300551b0 2024-09-13T17:08:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa330039bf0 con 0x7fa330036730 2024-09-13T17:08:19.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 -- 172.21.15.28:0/209531483 shutdown_connections 2024-09-13T17:08:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7f287d400640 1 -- 172.21.15.28:0/209531483 wait complete. 2024-09-13T17:08:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569390725 0 0) 0x7fa330039630 con 0x7fa33000a4f0 2024-09-13T17:08:19.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320003520 con 0x7fa33000a4f0 2024-09-13T17:08:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.928+0000 7fcd49200640 1 Processor -- start 2024-09-13T17:08:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.929+0000 7fcd49200640 1 -- start start 2024-09-13T17:08:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7f1c48c00640 1 Processor -- start 2024-09-13T17:08:19.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7fa529400640 1 --2- 172.21.15.28:0/1624992260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240cd7a0 0x7fa5240b8060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa5140097a0 tx=0x7fa51402a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5a1db9c342d062e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7f1c48c00640 1 -- start start 2024-09-13T17:08:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7fa528a00640 1 -- 172.21.15.28:0/1624992260 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514032440 con 0x7fa5240cd7a0 2024-09-13T17:08:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7fa528a00640 1 -- 172.21.15.28:0/1624992260 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5140325a0 con 0x7fa5240cd7a0 2024-09-13T17:08:19.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.930+0000 7fa528a00640 1 -- 172.21.15.28:0/1624992260 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa514032930 con 0x7fa5240cd7a0 2024-09-13T17:08:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7f287d400640 1 Processor -- start 2024-09-13T17:08:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa52b200640 1 -- 172.21.15.28:0/1624992260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240cd7a0 msgr2=0x7fa5240b8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa52b200640 1 --2- 172.21.15.28:0/1624992260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240cd7a0 0x7fa5240b8060 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa5140097a0 tx=0x7fa51402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa33ce00640 1 --2- 172.21.15.28:0/185362070 >> v2:172.21.15.189:3300/0 conn(0x7fa330036730 0x7fa33004ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.929+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754230098 0 0) 0x7fa330039910 con 0x7fa3300551b0 2024-09-13T17:08:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa330039630 con 0x7fa3300551b0 2024-09-13T17:08:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa52b200640 1 -- 172.21.15.28:0/1624992260 shutdown_connections 2024-09-13T17:08:19.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa52b200640 1 --2- 172.21.15.28:0/1624992260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240cd7a0 0x7fa5240b8060 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 986947813 0 0) 0x7fa320003520 con 0x7fa33000a4f0 2024-09-13T17:08:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 >> v2:172.21.15.189:3300/0 conn(0x7fa330036730 msgr2=0x7fa33004ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 --2- 172.21.15.28:0/185362070 >> v2:172.21.15.189:3300/0 conn(0x7fa330036730 0x7fa33004ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 >> v1:172.21.15.189:6789/0 conn(0x7fa3300551b0 legacy=0x7fa33004f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa330039ed0 con 0x7fa33000a4f0 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa33804e730 con 0x7fa33000a4f0 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 >> v1:172.21.15.28:6789/0 conn(0x7fa33000a4f0 legacy=0x7fa330036020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa320003b00 0x7fa320003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3200047c0 0x7fa320004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa32000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330039bf0 con 0x7fa320003b00 2024-09-13T17:08:19.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa32000bfa0 con 0x7fa3200047c0 2024-09-13T17:08:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa52b200640 1 -- 172.21.15.28:0/1624992260 >> 172.21.15.28:0/1624992260 conn(0x7fa5240cf3a0 msgr2=0x7fa5240cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32000c160 con 0x7fa320005180 2024-09-13T17:08:19.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7fa33e200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa32000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa52b200640 1 -- 172.21.15.28:0/1624992260 shutdown_connections 2024-09-13T17:08:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa52b200640 1 -- 172.21.15.28:0/1624992260 wait complete. 2024-09-13T17:08:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fcd49200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd4406cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f3b69600640 1 Processor -- start 2024-09-13T17:08:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa33e200640 1 -- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3200047c0 msgr2=0x7fa320004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T17:08:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa33e200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3200047c0 0x7fa320004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa33e200640 1 -- 172.21.15.28:0/185362070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa320003b00 msgr2=0x7fa320003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f1c48c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 0x7f1c44158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.931+0000 7f287d400640 1 -- start start 2024-09-13T17:08:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fcd49200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcd44159cc0 0x7fcd44162480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 0x7f287814d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 0x7f28780a9660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780aa040 con 0x7f28780b17e0 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28780aa320 con 0x7f287806d070 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7f287d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28780aa600 con 0x7f287806c540 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa33e200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa320003b00 0x7fa320003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.932+0000 7fa33e200640 1 -- 172.21.15.28:0/185362070 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa330039ed0 con 0x7fa320005180 2024-09-13T17:08:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd4406cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f1c48c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd4406cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55876/0 (socket says 172.21.15.28:55876) 2024-09-13T17:08:19.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7fcd41e00640 1 -- 172.21.15.28:0/3256301369 learned_addr learned my addr 172.21.15.28:0/3256301369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f1c48c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c441478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f2875e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 0x7f28780a9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd4406f3f0 con 0x7fcd4406bb80 2024-09-13T17:08:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f2875e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 0x7f28780a9660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44964/0 (socket says 172.21.15.28:44964) 2024-09-13T17:08:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f1c42a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f3b69600640 1 -- start start 2024-09-13T17:08:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f2876800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 0x7f287814d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f2875e00640 1 -- 172.21.15.28:0/3315634229 learned_addr learned my addr 172.21.15.28:0/3315634229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd440b3070 con 0x7fcd44159cc0 2024-09-13T17:08:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440b3350 con 0x7fcd440b1570 2024-09-13T17:08:19.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139842082 0 0) 0x7fcd4406f3f0 con 0x7fcd4406bb80 2024-09-13T17:08:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7fa33e200640 1 -- 172.21.15.28:0/185362070 reap_dead start 2024-09-13T17:08:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.933+0000 7f1c42a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44972/0 (socket says 172.21.15.28:44972) 2024-09-13T17:08:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c42a00640 1 -- 172.21.15.28:0/2964656065 learned_addr learned my addr 172.21.15.28:0/2964656065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c43400640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 0x7f1c44158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c441459a0 con 0x7f1c4406d960 2024-09-13T17:08:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4414e0c0 con 0x7f1c44070220 2024-09-13T17:08:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4414f2f0 con 0x7f1c4415b780 2024-09-13T17:08:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f1c43e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c441478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2877200640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7fcd42800640 1 --2- 172.21.15.28:0/3256301369 >> v2:172.21.15.28:3300/0 conn(0x7fcd44159cc0 0x7fcd44162480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7fa33e200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa32000b980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa324001970 tx=0x7fa3200057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db9de580c818cc0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2876800640 1 -- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 msgr2=0x7f28780a9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2876800640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 0x7f28780a9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2876800640 1 -- 172.21.15.28:0/3315634229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 msgr2=0x7f28780ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2876800640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.934+0000 7f2876800640 1 -- 172.21.15.28:0/3315634229 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870009b30 con 0x7f287806c540 2024-09-13T17:08:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32402acf0 con 0x7fa320005180 2024-09-13T17:08:19.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f3b69600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3b640cd7a0 0x7f3b640b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd28004630 con 0x7fcd4406bb80 2024-09-13T17:08:19.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f1c43e00640 1 -- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 msgr2=0x7f1c44145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f1c43e00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f1c43e00640 1 -- 172.21.15.28:0/2964656065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 msgr2=0x7f1c44158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f2876800640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 0x7f287814d6d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2870030000 tx=0x7f2870004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32402ae50 con 0x7fa320005180 2024-09-13T17:08:19.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7fa337e00640 1 -- 172.21.15.28:0/185362070 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa324007240 con 0x7fa320005180 2024-09-13T17:08:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7fa52b200640 1 Processor -- start 2024-09-13T17:08:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f3b69600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b640b5870 0x7f3b640b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f3b69600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b6415ccf0 con 0x7f3b640b41c0 2024-09-13T17:08:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f3b69600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b640cdf40 con 0x7f3b640cd7a0 2024-09-13T17:08:19.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.935+0000 7f3b69600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b640d2090 con 0x7f3b640b5870 2024-09-13T17:08:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f1c43e00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 0x7f1c44158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f1c43e00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c30009400 con 0x7f1c4415b780 2024-09-13T17:08:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f1c43400640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 0x7f1c44158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f3b63e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3b640b41c0 0x7f3b640b45a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34574/0 (socket says 172.21.15.28:34574) 2024-09-13T17:08:19.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287003b820 con 0x7f287806c540 2024-09-13T17:08:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287003b980 con 0x7f287806c540 2024-09-13T17:08:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287003bd30 con 0x7f287806c540 2024-09-13T17:08:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fa52b200640 1 -- start start 2024-09-13T17:08:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f3b63e00640 1 -- 172.21.15.28:0/3704090878 learned_addr learned my addr 172.21.15.28:0/3704090878 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:19.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f3b68c00640 1 --2- 172.21.15.28:0/3704090878 >> v2:172.21.15.189:3300/0 conn(0x7f3b640b5870 0x7f3b640b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995693980 0 0) 0x7fcd28004630 con 0x7fcd4406bb80 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 msgr2=0x7fcd4406cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd4406cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 >> v2:172.21.15.28:3300/0 conn(0x7fcd44159cc0 msgr2=0x7fcd44162480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 --2- 172.21.15.28:0/3256301369 >> v2:172.21.15.28:3300/0 conn(0x7fcd44159cc0 0x7fcd44162480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd440b3630 con 0x7fcd4406bb80 2024-09-13T17:08:19.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28780aa8e0 con 0x7f287806c540 2024-09-13T17:08:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fcd34001670 con 0x7fcd4406bb80 2024-09-13T17:08:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd4406cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 >> v1:172.21.15.189:6789/0 conn(0x7fcd4406bb80 legacy=0x7fcd440b1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.936+0000 7f1c42a00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287814dc10 con 0x7f287806c540 2024-09-13T17:08:19.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fa3b9200640 1 Processor -- start 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd28004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd280058d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28005e10 0x7fcd28006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440b3070 con 0x7fcd28004a80 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440b3350 con 0x7fcd28005e10 2024-09-13T17:08:19.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28003440 con 0x7fcd440b1570 2024-09-13T17:08:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fa3b9200640 1 -- start start 2024-09-13T17:08:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd28004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7fa52b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fcd42800640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28005e10 0x7fcd28006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fcd39400640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd280058d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa52b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5240b5870 0x7fa5240a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa52b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5240cd7a0 0x7fa52414ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa52b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5240d2090 con 0x7fa5240b5870 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa52b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa524148c00 con 0x7fa5240b41c0 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa52b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa524148ea0 con 0x7fa5240cd7a0 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa529e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa529e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44994/0 (socket says 172.21.15.28:44994) 2024-09-13T17:08:19.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa529e00640 1 -- 172.21.15.28:0/1820326060 learned_addr learned my addr 172.21.15.28:0/1820326060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.937+0000 7f3b63400640 1 --2- 172.21.15.28:0/3704090878 >> v2:172.21.15.28:3300/0 conn(0x7f3b640cd7a0 0x7f3b640b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 msgr2=0x7fa32000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa340200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa32000b980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa324001970 tx=0x7fa3200057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7f3b63400640 1 -- 172.21.15.28:0/3704090878 >> v2:172.21.15.189:3300/0 conn(0x7f3b640b5870 msgr2=0x7f3b640b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7f3b63400640 1 --2- 172.21.15.28:0/3704090878 >> v2:172.21.15.189:3300/0 conn(0x7f3b640b5870 0x7f3b640b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7f3b63400640 1 -- 172.21.15.28:0/3704090878 >> v1:172.21.15.28:6789/0 conn(0x7f3b640b41c0 legacy=0x7f3b640b45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28780aae00 con 0x7f287806c540 2024-09-13T17:08:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7f3b63400640 1 -- 172.21.15.28:0/3704090878 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b640ccd90 con 0x7f3b640cd7a0 2024-09-13T17:08:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa52a800640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5240cd7a0 0x7fa52414ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.938+0000 7f1c43e00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c441478a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c30031000 tx=0x7f1c300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 shutdown_connections 2024-09-13T17:08:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa340200640 1 --2- 172.21.15.28:0/185362070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa32000b980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 >> 172.21.15.28:0/185362070 conn(0x7fa33001d930 msgr2=0x7fa3300545a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa529e00640 1 -- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5240cd7a0 msgr2=0x7fa52414ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa529e00640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5240cd7a0 0x7fa52414ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa529e00640 1 -- 172.21.15.28:0/1820326060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5240b5870 msgr2=0x7fa5240a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa529e00640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5240b5870 0x7fa5240a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fcd41e00640 1 -- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 msgr2=0x7fcd280058d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd440b1570 0x7fcd280058d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fcd41e00640 1 -- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28005e10 msgr2=0x7fcd28006290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28005e10 0x7fcd28006290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fcd41e00640 1 -- 172.21.15.28:0/3256301369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28004630 con 0x7fcd28004a80 2024-09-13T17:08:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 shutdown_connections 2024-09-13T17:08:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2870038050 con 0x7f287806c540 2024-09-13T17:08:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.939+0000 7fa529e00640 1 -- 172.21.15.28:0/1820326060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514009400 con 0x7fa5240b41c0 2024-09-13T17:08:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f3b63400640 1 --2- 172.21.15.28:0/3704090878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b640b8060 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b640b2150 tx=0x7f3b5c032ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49486be53e27345 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7fcd42800640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd28005e10 0x7fcd28006290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7fa340200640 1 -- 172.21.15.28:0/185362070 wait complete. 2024-09-13T17:08:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f3b62a00640 1 -- 172.21.15.28:0/3704090878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c033c20 con 0x7f3b640cd7a0 2024-09-13T17:08:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7fa529e00640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c480 tx=0x7fa51800c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f3b62a00640 1 -- 172.21.15.28:0/3704090878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5c033d80 con 0x7f3b640cd7a0 2024-09-13T17:08:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f3b62a00640 1 -- 172.21.15.28:0/3704090878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b5c03b660 con 0x7f3b640cd7a0 2024-09-13T17:08:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 0x7fa3b40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f286ea00640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 0x7f285806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b9200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3b406d070 0x7fa3b40b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b41610a0 con 0x7fa3b40b2100 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.940+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30032440 con 0x7f1c4415b780 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b9200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b40b2620 con 0x7fa3b406d070 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b40b2900 con 0x7fa3b40b17e0 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c30030900 con 0x7f1c4415b780 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c30030cb0 con 0x7f1c4415b780 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c44145db0 con 0x7f1c4415b780 2024-09-13T17:08:19.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c4414f010 con 0x7f1c4415b780 2024-09-13T17:08:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28700a9d40 con 0x7f287806c540 2024-09-13T17:08:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7f2875e00640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 0x7f285806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b1400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3b406d070 0x7fa3b40b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa3b40b2100 0x7fa3b406b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34576/0 (socket says 172.21.15.28:34576) 2024-09-13T17:08:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b2800640 1 -- 172.21.15.28:0/3682986034 learned_addr learned my addr 172.21.15.28:0/3682986034 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.941+0000 7fa3b1e00640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 0x7fa3b40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 -- 172.21.15.28:0/3704090878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 msgr2=0x7f3b640b8060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 --2- 172.21.15.28:0/3704090878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b640b8060 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3b640b2150 tx=0x7f3b5c032ed0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518007480 con 0x7fa5240b41c0 2024-09-13T17:08:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5180075e0 con 0x7fa5240b41c0 2024-09-13T17:08:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518005a50 con 0x7fa5240b41c0 2024-09-13T17:08:19.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa3b1400640 1 -- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 msgr2=0x7fa3b40b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa524149180 con 0x7fa5240b41c0 2024-09-13T17:08:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa524149740 con 0x7fa5240b41c0 2024-09-13T17:08:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7fa3b1400640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 0x7fa3b40b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7fa3b1400640 1 -- 172.21.15.28:0/3682986034 >> v1:172.21.15.28:6789/0 conn(0x7fa3b40b2100 legacy=0x7fa3b406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7fa3b1400640 1 -- 172.21.15.28:0/3682986034 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b40b2be0 con 0x7fa3b406d070 2024-09-13T17:08:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7fa3b1e00640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 0x7fa3b40b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c440b0bd0 con 0x7f1c4415b780 2024-09-13T17:08:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 -- 172.21.15.28:0/3704090878 shutdown_connections 2024-09-13T17:08:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 --2- 172.21.15.28:0/3704090878 >> v2:172.21.15.189:3300/0 conn(0x7f3b640b5870 0x7f3b640b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 --2- 172.21.15.28:0/3704090878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b640b8060 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.942+0000 7f3b69600640 1 -- 172.21.15.28:0/3704090878 >> 172.21.15.28:0/3704090878 conn(0x7f3b640cf3a0 msgr2=0x7f3b640cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7f3b69600640 1 -- 172.21.15.28:0/3704090878 shutdown_connections 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7f2875e00640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 0x7f285806d110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2878156a60 tx=0x7f2868009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fcd41e00640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd28004ea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009470 tx=0x7fcd28006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b33a6628b290d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.943+0000 7f3b69600640 1 -- 172.21.15.28:0/3704090878 wait complete. 2024-09-13T17:08:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5240d1490 con 0x7fa5240b41c0 2024-09-13T17:08:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fa3b1400640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b40b1bc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00097a0 tx=0x7fa3a002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4625c7d79c305958 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fa340200640 1 Processor -- start 2024-09-13T17:08:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c03a630 con 0x7fcd28004a80 2024-09-13T17:08:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c03a790 con 0x7fcd28004a80 2024-09-13T17:08:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.944+0000 7fcd41400640 1 -- 172.21.15.28:0/3256301369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2c03ab00 con 0x7fcd28004a80 2024-09-13T17:08:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fa340200640 1 -- start start 2024-09-13T17:08:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fa3b0a00640 1 -- 172.21.15.28:0/3682986034 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0032440 con 0x7fa3b406d070 2024-09-13T17:08:19.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fa3b0a00640 1 -- 172.21.15.28:0/3682986034 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a00325a0 con 0x7fa3b406d070 2024-09-13T17:08:19.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fa3b0a00640 1 -- 172.21.15.28:0/3682986034 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0032930 con 0x7fa3b406d070 2024-09-13T17:08:19.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 msgr2=0x7fcd28004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fcd49200640 1 --2- 172.21.15.28:0/3256301369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd28004ea0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c009470 tx=0x7fcd28006900 comp rx=0 tx=0).stop 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fcd42800640 1 -- 172.21.15.28:0/3256301369 reap_dead start 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.945+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 shutdown_connections 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 >> 172.21.15.28:0/3256301369 conn(0x7fcd44069e80 msgr2=0x7fcd4406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 -- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 msgr2=0x7fa3b40b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b40b1bc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa3a00097a0 tx=0x7fa3a002a180 comp rx=0 tx=0).stop 2024-09-13T17:08:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7f3b69600640 1 Processor -- start 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 -- 172.21.15.28:0/3682986034 shutdown_connections 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b40b1bc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 --2- 172.21.15.28:0/3682986034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b17e0 0x7fa3b40b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 -- 172.21.15.28:0/3682986034 >> 172.21.15.28:0/3682986034 conn(0x7fa3b40b0270 msgr2=0x7fa3b40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 shutdown_connections 2024-09-13T17:08:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa340200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 -- 172.21.15.28:0/3682986034 shutdown_connections 2024-09-13T17:08:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa518005bb0 con 0x7fa5240b41c0 2024-09-13T17:08:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.947+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28780aae00 con 0x7f287806c540 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.947+0000 7fa51e000640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 0x7fa50806d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.947+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa518088930 con 0x7fa5240b41c0 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.947+0000 7f3b69600640 1 -- start start 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.948+0000 7f3b69600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.948+0000 7f3b69600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b640b5870 0x7f3b6406cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b69600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b641597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b69600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64155700 con 0x7f3b640cd7a0 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b69600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b641559e0 con 0x7f3b640b5870 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b69600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b64155cc0 con 0x7f3b640b41c0 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fcd49200640 1 -- 172.21.15.28:0/3256301369 wait complete. 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1c3003e450 con 0x7f1c4415b780 2024-09-13T17:08:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7fa340200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa330035030 0x7fa330052a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b63e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b63e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55926/0 (socket says 172.21.15.28:55926) 2024-09-13T17:08:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7f3b63e00640 1 -- 172.21.15.28:0/600633793 learned_addr learned my addr 172.21.15.28:0/600633793 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7fa340200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa330052f40 0x7fa3300533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7fa33d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7fa33d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55936/0 (socket says 172.21.15.28:55936) 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa33d800640 1 -- 172.21.15.28:0/4014684145 learned_addr learned my addr 172.21.15.28:0/4014684145 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa33ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa330035030 0x7fa330052a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330033f90 con 0x7fa330052f40 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330034270 con 0x7fa330035030 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa330034550 con 0x7fa320005180 2024-09-13T17:08:19.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa33e200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa330052f40 0x7fa3300533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7f3b63e00640 1 -- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b640b5870 msgr2=0x7f3b6406cfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-13T17:08:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7f3b63e00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b640b5870 0x7f3b6406cfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7f3b63e00640 1 -- 172.21.15.28:0/600633793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 msgr2=0x7f3b641597e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:08:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.946+0000 7fa3b9200640 1 -- 172.21.15.28:0/3682986034 wait complete. 2024-09-13T17:08:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fa529400640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 0x7fa50806d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.949+0000 7fa3b9200640 1 Processor -- start 2024-09-13T17:08:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7fcd49200640 1 Processor -- start 2024-09-13T17:08:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.950+0000 7f3b68c00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b641597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7fa3b9200640 1 -- start start 2024-09-13T17:08:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7f1c3b400640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 0x7f1c2406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c300aa510 con 0x7f1c4415b780 2024-09-13T17:08:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7f3b63e00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b641597e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7f3b63e00640 1 -- 172.21.15.28:0/600633793 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b5c009a70 con 0x7f3b640b41c0 2024-09-13T17:08:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa518053b80 con 0x7fa5240b41c0 2024-09-13T17:08:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.951+0000 7f3b68c00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b641597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7f1c43400640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 0x7f1c2406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1c30075930 con 0x7f1c4415b780 2024-09-13T17:08:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7f3b63e00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c4a0 tx=0x7f3b5000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7fcd49200640 1 -- start start 2024-09-13T17:08:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b414da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7fa33d800640 1 -- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa330035030 msgr2=0x7fa330052a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.952+0000 7fa33d800640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa330035030 0x7fa330052a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa33d800640 1 -- 172.21.15.28:0/4014684145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa330052f40 msgr2=0x7fa3300533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:19.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa33d800640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa330052f40 0x7fa3300533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:19.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa33d800640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa330034830 con 0x7fa320005180 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 0x7fa3b415d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b414da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b414da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:45046/0 (socket says 172.21.15.28:45046) 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b1e00640 1 -- 172.21.15.28:0/2498553920 learned_addr learned my addr 172.21.15.28:0/2498553920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b2100 0x7fa3b415d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.953+0000 7f1c43400640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 0x7f1c2406d190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c440cd220 tx=0x7f1c34008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa33d800640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa338061a70 tx=0x7fa338074220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa529400640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 0x7fa50806d190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa514034000 tx=0x7fa51402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa3b2800640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b2100 0x7fa3b415d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50005ce0 con 0x7f3b640b41c0 2024-09-13T17:08:19.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b50005e40 con 0x7f3b640b41c0 2024-09-13T17:08:20.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50007270 con 0x7f3b640b41c0 2024-09-13T17:08:20.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b414d670 con 0x7fa3b40b17e0 2024-09-13T17:08:20.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa3b1400640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 0x7fa3b415d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b414c9f0 con 0x7fa3b406d070 2024-09-13T17:08:20.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b414ccd0 con 0x7fa3b40b2100 2024-09-13T17:08:20.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.954+0000 7fcd49200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd4414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd49200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 0x7fcd440a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd49200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd440a8980 0x7fcd440a9c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd49200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440abce0 con 0x7fcd28004a80 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd49200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440a8fb0 con 0x7fcd440a8980 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd49200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd440a9290 con 0x7fcd44145e10 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd4414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd41e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd4414de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33718/0 (socket says 172.21.15.28:33718) 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd41e00640 1 -- 172.21.15.28:0/1713983004 learned_addr learned my addr 172.21.15.28:0/1713983004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd39400640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 0x7fcd440a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fa3b1400640 1 -- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b2100 msgr2=0x7fa3b415d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7fcd42800640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd440a8980 0x7fcd440a9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa3b1400640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b2100 0x7fa3b415d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa338072460 con 0x7fa320005180 2024-09-13T17:08:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33806ae90 con 0x7fa320005180 2024-09-13T17:08:20.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33806f870 con 0x7fa320005180 2024-09-13T17:08:20.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fcd39400640 1 -- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd440a8980 msgr2=0x7fcd440a9c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fcd39400640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd440a8980 0x7fcd440a9c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fcd39400640 1 -- 172.21.15.28:0/1713983004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 msgr2=0x7fcd4414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fcd39400640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd4414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fcd39400640 1 -- 172.21.15.28:0/1713983004 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd440abf80 con 0x7fcd44145e10 2024-09-13T17:08:20.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa3b1400640 1 -- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 msgr2=0x7fa3b414da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa3b1400640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b414da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.956+0000 7fa3b1400640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4009400 con 0x7fa3b40b17e0 2024-09-13T17:08:20.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.955+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b64155fa0 con 0x7f3b640b41c0 2024-09-13T17:08:20.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.957+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b64155140 con 0x7f3b640b41c0 2024-09-13T17:08:20.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.957+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa330034b10 con 0x7fa320005180 2024-09-13T17:08:20.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.957+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33017c0f0 con 0x7fa320005180 2024-09-13T17:08:20.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.957+0000 7fcd39400640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 0x7fcd440a8440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcd440718a0 tx=0x7fcd3400fc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.957+0000 7fa3b1400640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 0x7fa3b415d040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0031000 tx=0x7fa3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0030970 con 0x7fa3b40b17e0 2024-09-13T17:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3400fe70 con 0x7fcd44145e10 2024-09-13T17:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd34005320 con 0x7fcd44145e10 2024-09-13T17:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd34006a10 con 0x7fcd44145e10 2024-09-13T17:08:20.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a0030ad0 con 0x7fa3b40b17e0 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.959+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd440ac260 con 0x7fcd44145e10 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.959+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd44146290 con 0x7fcd44145e10 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a0030e80 con 0x7fa3b40b17e0 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a0009400 con 0x7fa3b40b17e0 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.958+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b414d290 con 0x7fa3b40b17e0 2024-09-13T17:08:20.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.961+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b640b8580 con 0x7f3b640b41c0 2024-09-13T17:08:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.961+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330039630 con 0x7fa320005180 2024-09-13T17:08:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.961+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa38c005250 con 0x7fa3b40b17e0 2024-09-13T17:08:20.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.962+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa33806f9d0 con 0x7fa320005180 2024-09-13T17:08:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.962+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3b50007490 con 0x7f3b640b41c0 2024-09-13T17:08:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.963+0000 7fa335600640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 0x7fa31806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.962+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd440a8d60 con 0x7fcd44145e10 2024-09-13T17:08:20.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.963+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcd34006b70 con 0x7fcd44145e10 2024-09-13T17:08:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.964+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa338102f70 con 0x7fa320005180 2024-09-13T17:08:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.963+0000 7f3b5be00640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 0x7f3b4406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.964+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b50088760 con 0x7f3b640b41c0 2024-09-13T17:08:20.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.964+0000 7fa33ce00640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 0x7fa31806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.964+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa3a0039040 con 0x7fa3b40b17e0 2024-09-13T17:08:20.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.965+0000 7f3b63400640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 0x7f3b4406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.965+0000 7fa33ce00640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 0x7fa31806d1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa3300323b0 tx=0x7fa32c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.965+0000 7fcd3aa00640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 0x7fcd2006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.966+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd34088dc0 con 0x7fcd44145e10 2024-09-13T17:08:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.966+0000 7fcd41e00640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 0x7fcd2006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.966+0000 7f3b63400640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 0x7f3b4406d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c03d000 tx=0x7f3b5c037000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.967+0000 7fa3aa000640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 0x7fa39406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.967+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3a00a9e70 con 0x7fa3b40b17e0 2024-09-13T17:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.967+0000 7fa3b1e00640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 0x7fa39406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.967+0000 7fcd41e00640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 0x7fcd2006d1c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c006c80 tx=0x7fcd2c045040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.968+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3b640b8580 con 0x7f3b640b41c0 2024-09-13T17:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.968+0000 7fa3b1e00640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 0x7fa39406d0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009d20 tx=0x7fa3a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.972+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa3380ce270 con 0x7fa320005180 2024-09-13T17:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.973+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa3a0075040 con 0x7fa3b40b17e0 2024-09-13T17:08:20.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:19.976+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcd34054030 con 0x7fcd44145e10 2024-09-13T17:08:20.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.296+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f94dc0057d0 con 0x7f95040d1230 2024-09-13T17:08:20.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.297+0000 7f9500c00640 1 -- 172.21.15.28:0/4247056560 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f94f4041020 con 0x7f95040d1230 2024-09-13T17:08:20.301 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:20.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.300+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 msgr2=0x7f94e006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.300+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 0x7f94e006d190 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f94fc00c550 tx=0x7f94fc00b040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.300+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 msgr2=0x7f95040ad990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.300+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f94f4004000 tx=0x7f94f40046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 shutdown_connections 2024-09-13T17:08:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f94e006acd0 0x7f94e006d190 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f95040cc5e0 0x7f95040cca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95040d1230 0x7f95040ad990 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 --2- 172.21.15.28:0/4247056560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f95040cf980 0x7f95040b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.301+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 >> 172.21.15.28:0/4247056560 conn(0x7f9504070ad0 msgr2=0x7f95040b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.302+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 shutdown_connections 2024-09-13T17:08:20.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.302+0000 7f950a400640 1 -- 172.21.15.28:0/4247056560 wait complete. 2024-09-13T17:08:20.329 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.4 2024-09-13T17:08:20.329 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.339+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f28780aae00 con 0x7f287806c540 2024-09-13T17:08:20.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.340+0000 7f286ea00640 1 -- 172.21.15.28:0/3315634229 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f28780aae00 con 0x7f287806c540 2024-09-13T17:08:20.341 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 msgr2=0x7f285806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 0x7f285806d110 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2878156a60 tx=0x7f2868009040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 msgr2=0x7f287814d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 0x7f287814d6d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2870030000 tx=0x7f2870004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 shutdown_connections 2024-09-13T17:08:20.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f285806ac50 0x7f285806d110 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2878156a60 tx=0x7f2868009040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28780b17e0 0x7f28780ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f287806d070 0x7f28780a9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 --2- 172.21.15.28:0/3315634229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f287806c540 0x7f287814d6d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 >> 172.21.15.28:0/3315634229 conn(0x7f28780b0270 msgr2=0x7f28780af6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 shutdown_connections 2024-09-13T17:08:20.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.342+0000 7f287d400640 1 -- 172.21.15.28:0/3315634229 wait complete. 2024-09-13T17:08:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.354+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1c4406fce0 con 0x7f1c4415b780 2024-09-13T17:08:20.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.355+0000 7f1c3b400640 1 -- 172.21.15.28:0/2964656065 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c30037020 con 0x7f1c4415b780 2024-09-13T17:08:20.356 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 msgr2=0x7f1c2406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 0x7f1c2406d190 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1c440cd220 tx=0x7f1c34008040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 msgr2=0x7f1c441478a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c441478a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c30031000 tx=0x7f1c300046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 shutdown_connections 2024-09-13T17:08:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2406acd0 0x7f1c2406d190 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c4415b780 0x7f1c441478a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c44070220 0x7f1c44145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 --2- 172.21.15.28:0/2964656065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c4406d960 0x7f1c44158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.358+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 >> 172.21.15.28:0/2964656065 conn(0x7f1c4406b360 msgr2=0x7f1c440b29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.359+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 shutdown_connections 2024-09-13T17:08:20.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.359+0000 7f1c48c00640 1 -- 172.21.15.28:0/2964656065 wait complete. 2024-09-13T17:08:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.359+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa524149930 con 0x7fa5240b41c0 2024-09-13T17:08:20.361 INFO:teuthology.orchestra.run.smithi028.stdout:55834574856 2024-09-13T17:08:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.360+0000 7fa51e000640 1 -- 172.21.15.28:0/1820326060 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa518005e60 con 0x7fa5240b41c0 2024-09-13T17:08:20.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.2 2024-09-13T17:08:20.362 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 msgr2=0x7fa50806d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 0x7fa50806d190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa514034000 tx=0x7fa51402c000 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 msgr2=0x7fa5241e5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa51800c480 tx=0x7fa51800c940 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 shutdown_connections 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa50806acd0 0x7fa50806d190 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5240cd7a0 0x7fa52414ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5240b5870 0x7fa5240a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 --2- 172.21.15.28:0/1820326060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5240b41c0 0x7fa5241e5f90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.363+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 >> 172.21.15.28:0/1820326060 conn(0x7fa5240cf3a0 msgr2=0x7fa5240b7170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.364+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 shutdown_connections 2024-09-13T17:08:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.364+0000 7fa52b200640 1 -- 172.21.15.28:0/1820326060 wait complete. 2024-09-13T17:08:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.371+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa3300323e0 con 0x7fa320005180 2024-09-13T17:08:20.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.371+0000 7fa335600640 1 -- 172.21.15.28:0/4014684145 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa338099090 con 0x7fa320005180 2024-09-13T17:08:20.372 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:20.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 msgr2=0x7fa31806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 0x7fa31806d1c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa3300323b0 tx=0x7fa32c008040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 msgr2=0x7fa3300339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fa338061a70 tx=0x7fa338074220 comp rx=0 tx=0).stop 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 shutdown_connections 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa31806ad00 0x7fa31806d1c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa330052f40 0x7fa3300533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa330035030 0x7fa330052a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 --2- 172.21.15.28:0/4014684145 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa320005180 0x7fa3300339b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 >> 172.21.15.28:0/4014684145 conn(0x7fa33001d930 msgr2=0x7fa330032900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.374+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 shutdown_connections 2024-09-13T17:08:20.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.375+0000 7fa340200640 1 -- 172.21.15.28:0/4014684145 wait complete. 2024-09-13T17:08:20.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.377+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3b641564c0 con 0x7f3b640b41c0 2024-09-13T17:08:20.379 INFO:teuthology.orchestra.run.smithi028.stdout:55834574854 2024-09-13T17:08:20.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.378+0000 7f3b5be00640 1 -- 172.21.15.28:0/600633793 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3b641564c0 con 0x7f3b640b41c0 2024-09-13T17:08:20.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.5 2024-09-13T17:08:20.380 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.381+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 msgr2=0x7f3b4406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.381+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 0x7f3b4406d190 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c03d000 tx=0x7f3b5c037000 comp rx=0 tx=0).stop 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.381+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 msgr2=0x7f3b64154c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.381+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000c4a0 tx=0x7f3b5000c960 comp rx=0 tx=0).stop 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.381+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 shutdown_connections 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b4406acd0 0x7f3b4406d190 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b640cd7a0 0x7f3b641597e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b640b5870 0x7f3b6406cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 --2- 172.21.15.28:0/600633793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b640b41c0 0x7f3b64154c00 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 >> 172.21.15.28:0/600633793 conn(0x7f3b640cf3a0 msgr2=0x7f3b640cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 shutdown_connections 2024-09-13T17:08:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.382+0000 7f3b69600640 1 -- 172.21.15.28:0/600633793 wait complete. 2024-09-13T17:08:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.383+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa38c005500 con 0x7fa3b40b17e0 2024-09-13T17:08:20.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.383+0000 7fa3aa000640 1 -- 172.21.15.28:0/2498553920 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3a004b090 con 0x7fa3b40b17e0 2024-09-13T17:08:20.385 INFO:teuthology.orchestra.run.smithi028.stdout:55834574856 2024-09-13T17:08:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.384+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fcd440ae3f0 con 0x7fcd44145e10 2024-09-13T17:08:20.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.384+0000 7fcd3aa00640 1 -- 172.21.15.28:0/1713983004 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcd3401a020 con 0x7fcd44145e10 2024-09-13T17:08:20.385 INFO:teuthology.orchestra.run.smithi028.stdout:55834574855 2024-09-13T17:08:20.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 msgr2=0x7fa39406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 0x7fa39406d0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009d20 tx=0x7fa3a4008040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 msgr2=0x7fa3b415d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 0x7fa3b415d040 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0031000 tx=0x7fa3a00046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 msgr2=0x7fcd2006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 0x7fcd2006d1c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fcd2c006c80 tx=0x7fcd2c045040 comp rx=0 tx=0).stop 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 msgr2=0x7fcd440a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 0x7fcd440a8440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcd440718a0 tx=0x7fcd3400fc80 comp rx=0 tx=0).stop 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 shutdown_connections 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa39406ac00 0x7fa39406d0c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3b40b2100 0x7fa3b415d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3b40b17e0 0x7fa3b415d040 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 --2- 172.21.15.28:0/2498553920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3b406d070 0x7fa3b414da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 >> 172.21.15.28:0/2498553920 conn(0x7fa3b40b0270 msgr2=0x7fa3b4160600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 shutdown_connections 2024-09-13T17:08:20.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fa3b9200640 1 -- 172.21.15.28:0/2498553920 wait complete. 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 shutdown_connections 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd2006ad00 0x7fcd2006d1c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd440a8980 0x7fcd440a9c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd44145e10 0x7fcd440a8440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 --2- 172.21.15.28:0/1713983004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd28004a80 0x7fcd4414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.387+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 >> 172.21.15.28:0/1713983004 conn(0x7fcd44069e80 msgr2=0x7fcd44155200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.388+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 shutdown_connections 2024-09-13T17:08:20.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.388+0000 7fcd49200640 1 -- 172.21.15.28:0/1713983004 wait complete. 2024-09-13T17:08:20.392 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.0 2024-09-13T17:08:20.392 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.397 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.7 2024-09-13T17:08:20.397 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.1 2024-09-13T17:08:20.408 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.3 2024-09-13T17:08:20.409 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.409 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.6 2024-09-13T17:08:20.409 DEBUG:teuthology.parallel:result is None 2024-09-13T17:08:20.409 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T17:08:20.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:08:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.603+0000 7ff48a600640 1 Processor -- start 2024-09-13T17:08:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.603+0000 7ff48a600640 1 -- start start 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff48a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 0x7ff48406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff48a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48415e100 con 0x7ff484070cc0 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff48a600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48415e3e0 con 0x7ff4840cabc0 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff48a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840adb00 con 0x7ff48415a540 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff483e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff484070cc0 0x7ff48406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34580/0 (socket says 172.21.15.28:34580) 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.604+0000 7ff483e00640 1 -- 172.21.15.28:0/1382278587 learned_addr learned my addr 172.21.15.28:0/1382278587 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff483400640 1 --2- 172.21.15.28:0/1382278587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 0x7ff48406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197243016 0 0) 0x7ff48415e100 con 0x7ff484070cc0 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff468003520 con 0x7ff484070cc0 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3240120996 0 0) 0x7ff48415e3e0 con 0x7ff4840cabc0 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff48415e100 con 0x7ff4840cabc0 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff483400640 1 -- 172.21.15.28:0/1382278587 >> v1:172.21.15.189:6789/0 conn(0x7ff4840cabc0 legacy=0x7ff48415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff483400640 1 -- 172.21.15.28:0/1382278587 >> v1:172.21.15.28:6789/0 conn(0x7ff484070cc0 legacy=0x7ff48406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.605+0000 7ff483400640 1 -- 172.21.15.28:0/1382278587 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48415ef80 con 0x7ff48415a540 2024-09-13T17:08:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.606+0000 7ff483400640 1 --2- 172.21.15.28:0/1382278587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 0x7ff48406c430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097d0 tx=0x7ff47402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c41c72e23f0634fa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474032440 con 0x7ff48415a540 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4740325a0 con 0x7ff48415a540 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff482a00640 1 -- 172.21.15.28:0/1382278587 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474032930 con 0x7ff48415a540 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 -- 172.21.15.28:0/1382278587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 msgr2=0x7ff48406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 --2- 172.21.15.28:0/1382278587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 0x7ff48406c430 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff4740097d0 tx=0x7ff47402a180 comp rx=0 tx=0).stop 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 -- 172.21.15.28:0/1382278587 shutdown_connections 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 --2- 172.21.15.28:0/1382278587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff48415a540 0x7ff48406c430 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 -- 172.21.15.28:0/1382278587 >> 172.21.15.28:0/1382278587 conn(0x7ff484069e80 msgr2=0x7ff48406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 -- 172.21.15.28:0/1382278587 shutdown_connections 2024-09-13T17:08:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.607+0000 7ff48a600640 1 -- 172.21.15.28:0/1382278587 wait complete. 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.609+0000 7ff48a600640 1 Processor -- start 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.609+0000 7ff48a600640 1 -- start start 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff484070cc0 0x7ff48414faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 0x7ff48413ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48414f820 con 0x7ff4840cabc0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48413f8a0 con 0x7ff48415a540 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff48a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48413f490 con 0x7ff484070cc0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff47aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff47aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33726/0 (socket says 172.21.15.28:33726) 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.610+0000 7ff47aa00640 1 -- 172.21.15.28:0/2872957208 learned_addr learned my addr 172.21.15.28:0/2872957208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff483400640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff484070cc0 0x7ff48414faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff483e00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 0x7ff48413ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 -- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff484070cc0 msgr2=0x7ff48414faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff484070cc0 0x7ff48414faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 -- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 msgr2=0x7ff48413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 0x7ff48413ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 -- 172.21.15.28:0/2872957208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474009400 con 0x7ff4840cabc0 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff483e00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 0x7ff48413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:20.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47aa00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00c450 tx=0x7ff46c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c0052d0 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c005430 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.611+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c012870 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.612+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48415bb70 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.612+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48415c130 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.613+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48406c430 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.614+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff46c0129d0 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.614+0000 7ff47be00640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.615+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff46c088dd0 con 0x7ff4840cabc0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.615+0000 7ff483400640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.616+0000 7ff483400640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff474034000 tx=0x7ff47402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:20.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.620+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff46c054060 con 0x7ff4840cabc0 2024-09-13T17:08:20.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.794+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff48415a920 con 0x7ff46006ac00 2024-09-13T17:08:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.801+0000 7ff47be00640 1 -- 172.21.15.28:0/2872957208 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264574 (secure 0 0 0) 0x7ff474032890 con 0x7ff46006ac00 2024-09-13T17:08:20.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:20.807 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 msgr2=0x7ff46006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff474034000 tx=0x7ff47402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 msgr2=0x7ff484150440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff46c00c450 tx=0x7ff46c00c910 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 shutdown_connections 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff48415a540 0x7ff48413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840cabc0 0x7ff484150440 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.805+0000 7ff48a600640 1 --2- 172.21.15.28:0/2872957208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff484070cc0 0x7ff48414faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:20.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.806+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 >> 172.21.15.28:0/2872957208 conn(0x7ff484069e80 msgr2=0x7ff4840ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.806+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 shutdown_connections 2024-09-13T17:08:20.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:20.806+0000 7ff48a600640 1 -- 172.21.15.28:0/2872957208 wait complete. 2024-09-13T17:08:20.822 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-13T17:08:18.885496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235408,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754739312,"statfs":{"total":773094113280,"available":772853055488,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.769853"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901739+0000","last_change":"2024-09-13T17:08:11.601843+0000","last_active":"2024-09-13T17:08:12.901739+0000","last_peered":"2024-09-13T17:08:12.901739+0000","last_clean":"2024-09-13T17:08:12.901739+0000","last_became_active":"2024-09-13T17:08:11.601437+0000","last_became_peered":"2024-09-13T17:08:11.601437+0000","last_unstale":"2024-09-13T17:08:12.901739+0000","last_undegraded":"2024-09-13T17:08:12.901739+0000","last_fullsized":"2024-09-13T17:08:12.901739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:28.411961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566115+0000","last_change":"2024-09-13T17:08:11.576946+0000","last_active":"2024-09-13T17:08:12.566115+0000","last_peered":"2024-09-13T17:08:12.566115+0000","last_clean":"2024-09-13T17:08:12.566115+0000","last_became_active":"2024-09-13T17:08:11.576700+0000","last_became_peered":"2024-09-13T17:08:11.576700+0000","last_unstale":"2024-09-13T17:08:12.566115+0000","last_undegraded":"2024-09-13T17:08:12.566115+0000","last_fullsized":"2024-09-13T17:08:12.566115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905558+0000","last_change":"2024-09-13T17:08:11.576052+0000","last_active":"2024-09-13T17:08:12.905558+0000","last_peered":"2024-09-13T17:08:12.905558+0000","last_clean":"2024-09-13T17:08:12.905558+0000","last_became_active":"2024-09-13T17:08:11.575297+0000","last_became_peered":"2024-09-13T17:08:11.575297+0000","last_unstale":"2024-09-13T17:08:12.905558+0000","last_undegraded":"2024-09-13T17:08:12.905558+0000","last_fullsized":"2024-09-13T17:08:12.905558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567600+0000","last_change":"2024-09-13T17:08:11.582113+0000","last_active":"2024-09-13T17:08:12.567600+0000","last_peered":"2024-09-13T17:08:12.567600+0000","last_clean":"2024-09-13T17:08:12.567600+0000","last_became_active":"2024-09-13T17:08:11.579185+0000","last_became_peered":"2024-09-13T17:08:11.579185+0000","last_unstale":"2024-09-13T17:08:12.567600+0000","last_undegraded":"2024-09-13T17:08:12.567600+0000","last_fullsized":"2024-09-13T17:08:12.567600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901581+0000","last_change":"2024-09-13T17:08:11.600241+0000","last_active":"2024-09-13T17:08:12.901581+0000","last_peered":"2024-09-13T17:08:12.901581+0000","last_clean":"2024-09-13T17:08:12.901581+0000","last_became_active":"2024-09-13T17:08:11.596443+0000","last_became_peered":"2024-09-13T17:08:11.596443+0000","last_unstale":"2024-09-13T17:08:12.901581+0000","last_undegraded":"2024-09-13T17:08:12.901581+0000","last_fullsized":"2024-09-13T17:08:12.901581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567274+0000","last_change":"2024-09-13T17:08:11.583211+0000","last_active":"2024-09-13T17:08:12.567274+0000","last_peered":"2024-09-13T17:08:12.567274+0000","last_clean":"2024-09-13T17:08:12.567274+0000","last_became_active":"2024-09-13T17:08:11.583040+0000","last_became_peered":"2024-09-13T17:08:11.583040+0000","last_unstale":"2024-09-13T17:08:12.567274+0000","last_undegraded":"2024-09-13T17:08:12.567274+0000","last_fullsized":"2024-09-13T17:08:12.567274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901512+0000","last_change":"2024-09-13T17:08:11.592666+0000","last_active":"2024-09-13T17:08:12.901512+0000","last_peered":"2024-09-13T17:08:12.901512+0000","last_clean":"2024-09-13T17:08:12.901512+0000","last_became_active":"2024-09-13T17:08:11.592373+0000","last_became_peered":"2024-09-13T17:08:11.592373+0000","last_unstale":"2024-09-13T17:08:12.901512+0000","last_undegraded":"2024-09-13T17:08:12.901512+0000","last_fullsized":"2024-09-13T17:08:12.901512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904617+0000","last_change":"2024-09-13T17:08:11.578631+0000","last_active":"2024-09-13T17:08:12.904617+0000","last_peered":"2024-09-13T17:08:12.904617+0000","last_clean":"2024-09-13T17:08:12.904617+0000","last_became_active":"2024-09-13T17:08:11.575801+0000","last_became_peered":"2024-09-13T17:08:11.575801+0000","last_unstale":"2024-09-13T17:08:12.904617+0000","last_undegraded":"2024-09-13T17:08:12.904617+0000","last_fullsized":"2024-09-13T17:08:12.904617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901752+0000","last_change":"2024-09-13T17:08:11.601554+0000","last_active":"2024-09-13T17:08:12.901752+0000","last_peered":"2024-09-13T17:08:12.901752+0000","last_clean":"2024-09-13T17:08:12.901752+0000","last_became_active":"2024-09-13T17:08:11.601069+0000","last_became_peered":"2024-09-13T17:08:11.601069+0000","last_unstale":"2024-09-13T17:08:12.901752+0000","last_undegraded":"2024-09-13T17:08:12.901752+0000","last_fullsized":"2024-09-13T17:08:12.901752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:24.360659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904240+0000","last_change":"2024-09-13T17:08:11.587154+0000","last_active":"2024-09-13T17:08:12.904240+0000","last_peered":"2024-09-13T17:08:12.904240+0000","last_clean":"2024-09-13T17:08:12.904240+0000","last_became_active":"2024-09-13T17:08:11.586876+0000","last_became_peered":"2024-09-13T17:08:11.586876+0000","last_unstale":"2024-09-13T17:08:12.904240+0000","last_undegraded":"2024-09-13T17:08:12.904240+0000","last_fullsized":"2024-09-13T17:08:12.904240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904210+0000","last_change":"2024-09-13T17:08:11.586451+0000","last_active":"2024-09-13T17:08:12.904210+0000","last_peered":"2024-09-13T17:08:12.904210+0000","last_clean":"2024-09-13T17:08:12.904210+0000","last_became_active":"2024-09-13T17:08:11.586127+0000","last_became_peered":"2024-09-13T17:08:11.586127+0000","last_unstale":"2024-09-13T17:08:12.904210+0000","last_undegraded":"2024-09-13T17:08:12.904210+0000","last_fullsized":"2024-09-13T17:08:12.904210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569157+0000","last_change":"2024-09-13T17:08:11.591207+0000","last_active":"2024-09-13T17:08:12.569157+0000","last_peered":"2024-09-13T17:08:12.569157+0000","last_clean":"2024-09-13T17:08:12.569157+0000","last_became_active":"2024-09-13T17:08:11.590920+0000","last_became_peered":"2024-09-13T17:08:11.590920+0000","last_unstale":"2024-09-13T17:08:12.569157+0000","last_undegraded":"2024-09-13T17:08:12.569157+0000","last_fullsized":"2024-09-13T17:08:12.569157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:25.618116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904684+0000","last_change":"2024-09-13T17:08:11.586067+0000","last_active":"2024-09-13T17:08:12.904684+0000","last_peered":"2024-09-13T17:08:12.904684+0000","last_clean":"2024-09-13T17:08:12.904684+0000","last_became_active":"2024-09-13T17:08:11.585787+0000","last_became_peered":"2024-09-13T17:08:11.585787+0000","last_unstale":"2024-09-13T17:08:12.904684+0000","last_undegraded":"2024-09-13T17:08:12.904684+0000","last_fullsized":"2024-09-13T17:08:12.904684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905488+0000","last_change":"2024-09-13T17:08:11.570021+0000","last_active":"2024-09-13T17:08:12.905488+0000","last_peered":"2024-09-13T17:08:12.905488+0000","last_clean":"2024-09-13T17:08:12.905488+0000","last_became_active":"2024-09-13T17:08:11.569663+0000","last_became_peered":"2024-09-13T17:08:11.569663+0000","last_unstale":"2024-09-13T17:08:12.905488+0000","last_undegraded":"2024-09-13T17:08:12.905488+0000","last_fullsized":"2024-09-13T17:08:12.905488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565777+0000","last_change":"2024-09-13T17:08:11.578244+0000","last_active":"2024-09-13T17:08:12.565777+0000","last_peered":"2024-09-13T17:08:12.565777+0000","last_clean":"2024-09-13T17:08:12.565777+0000","last_became_active":"2024-09-13T17:08:11.577989+0000","last_became_peered":"2024-09-13T17:08:11.577989+0000","last_unstale":"2024-09-13T17:08:12.565777+0000","last_undegraded":"2024-09-13T17:08:12.565777+0000","last_fullsized":"2024-09-13T17:08:12.565777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567131+0000","last_change":"2024-09-13T17:08:11.582930+0000","last_active":"2024-09-13T17:08:12.567131+0000","last_peered":"2024-09-13T17:08:12.567131+0000","last_clean":"2024-09-13T17:08:12.567131+0000","last_became_active":"2024-09-13T17:08:11.582621+0000","last_became_peered":"2024-09-13T17:08:11.582621+0000","last_unstale":"2024-09-13T17:08:12.567131+0000","last_undegraded":"2024-09-13T17:08:12.567131+0000","last_fullsized":"2024-09-13T17:08:12.567131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568144+0000","last_change":"2024-09-13T17:08:11.601899+0000","last_active":"2024-09-13T17:08:12.568144+0000","last_peered":"2024-09-13T17:08:12.568144+0000","last_clean":"2024-09-13T17:08:12.568144+0000","last_became_active":"2024-09-13T17:08:11.601452+0000","last_became_peered":"2024-09-13T17:08:11.601452+0000","last_unstale":"2024-09-13T17:08:12.568144+0000","last_undegraded":"2024-09-13T17:08:12.568144+0000","last_fullsized":"2024-09-13T17:08:12.568144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902746+0000","last_change":"2024-09-13T17:08:11.591991+0000","last_active":"2024-09-13T17:08:12.902746+0000","last_peered":"2024-09-13T17:08:12.902746+0000","last_clean":"2024-09-13T17:08:12.902746+0000","last_became_active":"2024-09-13T17:08:11.591554+0000","last_became_peered":"2024-09-13T17:08:11.591554+0000","last_unstale":"2024-09-13T17:08:12.902746+0000","last_undegraded":"2024-09-13T17:08:12.902746+0000","last_fullsized":"2024-09-13T17:08:12.902746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901703+0000","last_change":"2024-09-13T17:08:11.596861+0000","last_active":"2024-09-13T17:08:12.901703+0000","last_peered":"2024-09-13T17:08:12.901703+0000","last_clean":"2024-09-13T17:08:12.901703+0000","last_became_active":"2024-09-13T17:08:11.596669+0000","last_became_peered":"2024-09-13T17:08:11.596669+0000","last_unstale":"2024-09-13T17:08:12.901703+0000","last_undegraded":"2024-09-13T17:08:12.901703+0000","last_fullsized":"2024-09-13T17:08:12.901703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904190+0000","last_change":"2024-09-13T17:08:11.582225+0000","last_active":"2024-09-13T17:08:12.904190+0000","last_peered":"2024-09-13T17:08:12.904190+0000","last_clean":"2024-09-13T17:08:12.904190+0000","last_became_active":"2024-09-13T17:08:11.581900+0000","last_became_peered":"2024-09-13T17:08:11.581900+0000","last_unstale":"2024-09-13T17:08:12.904190+0000","last_undegraded":"2024-09-13T17:08:12.904190+0000","last_fullsized":"2024-09-13T17:08:12.904190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905507+0000","last_change":"2024-09-13T17:08:11.588070+0000","last_active":"2024-09-13T17:08:12.905507+0000","last_peered":"2024-09-13T17:08:12.905507+0000","last_clean":"2024-09-13T17:08:12.905507+0000","last_became_active":"2024-09-13T17:08:11.587859+0000","last_became_peered":"2024-09-13T17:08:11.587859+0000","last_unstale":"2024-09-13T17:08:12.905507+0000","last_undegraded":"2024-09-13T17:08:12.905507+0000","last_fullsized":"2024-09-13T17:08:12.905507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567322+0000","last_change":"2024-09-13T17:08:11.597424+0000","last_active":"2024-09-13T17:08:12.567322+0000","last_peered":"2024-09-13T17:08:12.567322+0000","last_clean":"2024-09-13T17:08:12.567322+0000","last_became_active":"2024-09-13T17:08:11.597093+0000","last_became_peered":"2024-09-13T17:08:11.597093+0000","last_unstale":"2024-09-13T17:08:12.567322+0000","last_undegraded":"2024-09-13T17:08:12.567322+0000","last_fullsized":"2024-09-13T17:08:12.567322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904647+0000","last_change":"2024-09-13T17:08:11.581514+0000","last_active":"2024-09-13T17:08:12.904647+0000","last_peered":"2024-09-13T17:08:12.904647+0000","last_clean":"2024-09-13T17:08:12.904647+0000","last_became_active":"2024-09-13T17:08:11.581178+0000","last_became_peered":"2024-09-13T17:08:11.581178+0000","last_unstale":"2024-09-13T17:08:12.904647+0000","last_undegraded":"2024-09-13T17:08:12.904647+0000","last_fullsized":"2024-09-13T17:08:12.904647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567215+0000","last_change":"2024-09-13T17:08:11.575094+0000","last_active":"2024-09-13T17:08:12.567215+0000","last_peered":"2024-09-13T17:08:12.567215+0000","last_clean":"2024-09-13T17:08:12.567215+0000","last_became_active":"2024-09-13T17:08:11.574833+0000","last_became_peered":"2024-09-13T17:08:11.574833+0000","last_unstale":"2024-09-13T17:08:12.567215+0000","last_undegraded":"2024-09-13T17:08:12.567215+0000","last_fullsized":"2024-09-13T17:08:12.567215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904757+0000","last_change":"2024-09-13T17:08:11.581636+0000","last_active":"2024-09-13T17:08:12.904757+0000","last_peered":"2024-09-13T17:08:12.904757+0000","last_clean":"2024-09-13T17:08:12.904757+0000","last_became_active":"2024-09-13T17:08:11.581264+0000","last_became_peered":"2024-09-13T17:08:11.581264+0000","last_unstale":"2024-09-13T17:08:12.904757+0000","last_undegraded":"2024-09-13T17:08:12.904757+0000","last_fullsized":"2024-09-13T17:08:12.904757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904250+0000","last_change":"2024-09-13T17:08:11.601141+0000","last_active":"2024-09-13T17:08:12.904250+0000","last_peered":"2024-09-13T17:08:12.904250+0000","last_clean":"2024-09-13T17:08:12.904250+0000","last_became_active":"2024-09-13T17:08:11.600877+0000","last_became_peered":"2024-09-13T17:08:11.600877+0000","last_unstale":"2024-09-13T17:08:12.904250+0000","last_undegraded":"2024-09-13T17:08:12.904250+0000","last_fullsized":"2024-09-13T17:08:12.904250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565968+0000","last_change":"2024-09-13T17:08:11.576209+0000","last_active":"2024-09-13T17:08:12.565968+0000","last_peered":"2024-09-13T17:08:12.565968+0000","last_clean":"2024-09-13T17:08:12.565968+0000","last_became_active":"2024-09-13T17:08:11.575917+0000","last_became_peered":"2024-09-13T17:08:11.575917+0000","last_unstale":"2024-09-13T17:08:12.565968+0000","last_undegraded":"2024-09-13T17:08:12.565968+0000","last_fullsized":"2024-09-13T17:08:12.565968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904731+0000","last_change":"2024-09-13T17:08:11.582222+0000","last_active":"2024-09-13T17:08:12.904731+0000","last_peered":"2024-09-13T17:08:12.904731+0000","last_clean":"2024-09-13T17:08:12.904731+0000","last_became_active":"2024-09-13T17:08:11.582057+0000","last_became_peered":"2024-09-13T17:08:11.582057+0000","last_unstale":"2024-09-13T17:08:12.904731+0000","last_undegraded":"2024-09-13T17:08:12.904731+0000","last_fullsized":"2024-09-13T17:08:12.904731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905515+0000","last_change":"2024-09-13T17:08:11.575089+0000","last_active":"2024-09-13T17:08:12.905515+0000","last_peered":"2024-09-13T17:08:12.905515+0000","last_clean":"2024-09-13T17:08:12.905515+0000","last_became_active":"2024-09-13T17:08:11.574665+0000","last_became_peered":"2024-09-13T17:08:11.574665+0000","last_unstale":"2024-09-13T17:08:12.905515+0000","last_undegraded":"2024-09-13T17:08:12.905515+0000","last_fullsized":"2024-09-13T17:08:12.905515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567404+0000","last_change":"2024-09-13T17:08:11.579735+0000","last_active":"2024-09-13T17:08:12.567404+0000","last_peered":"2024-09-13T17:08:12.567404+0000","last_clean":"2024-09-13T17:08:12.567404+0000","last_became_active":"2024-09-13T17:08:11.578749+0000","last_became_peered":"2024-09-13T17:08:11.578749+0000","last_unstale":"2024-09-13T17:08:12.567404+0000","last_undegraded":"2024-09-13T17:08:12.567404+0000","last_fullsized":"2024-09-13T17:08:12.567404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902716+0000","last_change":"2024-09-13T17:08:11.570273+0000","last_active":"2024-09-13T17:08:12.902716+0000","last_peered":"2024-09-13T17:08:12.902716+0000","last_clean":"2024-09-13T17:08:12.902716+0000","last_became_active":"2024-09-13T17:08:11.569987+0000","last_became_peered":"2024-09-13T17:08:11.569987+0000","last_unstale":"2024-09-13T17:08:12.902716+0000","last_undegraded":"2024-09-13T17:08:12.902716+0000","last_fullsized":"2024-09-13T17:08:12.902716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904126+0000","last_change":"2024-09-13T17:08:11.586071+0000","last_active":"2024-09-13T17:08:12.904126+0000","last_peered":"2024-09-13T17:08:12.904126+0000","last_clean":"2024-09-13T17:08:12.904126+0000","last_became_active":"2024-09-13T17:08:11.585507+0000","last_became_peered":"2024-09-13T17:08:11.585507+0000","last_unstale":"2024-09-13T17:08:12.904126+0000","last_undegraded":"2024-09-13T17:08:12.904126+0000","last_fullsized":"2024-09-13T17:08:12.904126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567646+0000","last_change":"2024-09-13T17:08:11.578066+0000","last_active":"2024-09-13T17:08:12.567646+0000","last_peered":"2024-09-13T17:08:12.567646+0000","last_clean":"2024-09-13T17:08:12.567646+0000","last_became_active":"2024-09-13T17:08:11.577655+0000","last_became_peered":"2024-09-13T17:08:11.577655+0000","last_unstale":"2024-09-13T17:08:12.567646+0000","last_undegraded":"2024-09-13T17:08:12.567646+0000","last_fullsized":"2024-09-13T17:08:12.567646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.318727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904721+0000","last_change":"2024-09-13T17:08:11.147841+0000","last_active":"2024-09-13T17:08:12.904721+0000","last_peered":"2024-09-13T17:08:12.904721+0000","last_clean":"2024-09-13T17:08:12.904721+0000","last_became_active":"2024-09-13T17:08:11.147449+0000","last_became_peered":"2024-09-13T17:08:11.147449+0000","last_unstale":"2024-09-13T17:08:12.904721+0000","last_undegraded":"2024-09-13T17:08:12.904721+0000","last_fullsized":"2024-09-13T17:08:12.904721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905524+0000","last_change":"2024-09-13T17:08:11.580301+0000","last_active":"2024-09-13T17:08:12.905524+0000","last_peered":"2024-09-13T17:08:12.905524+0000","last_clean":"2024-09-13T17:08:12.905524+0000","last_became_active":"2024-09-13T17:08:11.579029+0000","last_became_peered":"2024-09-13T17:08:11.579029+0000","last_unstale":"2024-09-13T17:08:12.905524+0000","last_undegraded":"2024-09-13T17:08:12.905524+0000","last_fullsized":"2024-09-13T17:08:12.905524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:30.800655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904179+0000","last_change":"2024-09-13T17:08:11.146319+0000","last_active":"2024-09-13T17:08:12.904179+0000","last_peered":"2024-09-13T17:08:12.904179+0000","last_clean":"2024-09-13T17:08:12.904179+0000","last_became_active":"2024-09-13T17:08:11.146057+0000","last_became_peered":"2024-09-13T17:08:11.146057+0000","last_unstale":"2024-09-13T17:08:12.904179+0000","last_undegraded":"2024-09-13T17:08:12.904179+0000","last_fullsized":"2024-09-13T17:08:12.904179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904667+0000","last_change":"2024-09-13T17:08:11.138543+0000","last_active":"2024-09-13T17:08:12.904667+0000","last_peered":"2024-09-13T17:08:12.904667+0000","last_clean":"2024-09-13T17:08:12.904667+0000","last_became_active":"2024-09-13T17:08:11.133550+0000","last_became_peered":"2024-09-13T17:08:11.133550+0000","last_unstale":"2024-09-13T17:08:12.904667+0000","last_undegraded":"2024-09-13T17:08:12.904667+0000","last_fullsized":"2024-09-13T17:08:12.904667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904629+0000","last_change":"2024-09-13T17:08:11.580334+0000","last_active":"2024-09-13T17:08:12.904629+0000","last_peered":"2024-09-13T17:08:12.904629+0000","last_clean":"2024-09-13T17:08:12.904629+0000","last_became_active":"2024-09-13T17:08:11.574885+0000","last_became_peered":"2024-09-13T17:08:11.574885+0000","last_unstale":"2024-09-13T17:08:12.904629+0000","last_undegraded":"2024-09-13T17:08:12.904629+0000","last_fullsized":"2024-09-13T17:08:12.904629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567242+0000","last_change":"2024-09-13T17:08:11.142227+0000","last_active":"2024-09-13T17:08:12.567242+0000","last_peered":"2024-09-13T17:08:12.567242+0000","last_clean":"2024-09-13T17:08:12.567242+0000","last_became_active":"2024-09-13T17:08:11.142009+0000","last_became_peered":"2024-09-13T17:08:11.142009+0000","last_unstale":"2024-09-13T17:08:12.567242+0000","last_undegraded":"2024-09-13T17:08:12.567242+0000","last_fullsized":"2024-09-13T17:08:12.567242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565757+0000","last_change":"2024-09-13T17:08:11.569231+0000","last_active":"2024-09-13T17:08:12.565757+0000","last_peered":"2024-09-13T17:08:12.565757+0000","last_clean":"2024-09-13T17:08:12.565757+0000","last_became_active":"2024-09-13T17:08:11.568934+0000","last_became_peered":"2024-09-13T17:08:11.568934+0000","last_unstale":"2024-09-13T17:08:12.565757+0000","last_undegraded":"2024-09-13T17:08:12.565757+0000","last_fullsized":"2024-09-13T17:08:12.565757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904259+0000","last_change":"2024-09-13T17:08:11.140675+0000","last_active":"2024-09-13T17:08:12.904259+0000","last_peered":"2024-09-13T17:08:12.904259+0000","last_clean":"2024-09-13T17:08:12.904259+0000","last_became_active":"2024-09-13T17:08:11.139918+0000","last_became_peered":"2024-09-13T17:08:11.139918+0000","last_unstale":"2024-09-13T17:08:12.904259+0000","last_undegraded":"2024-09-13T17:08:12.904259+0000","last_fullsized":"2024-09-13T17:08:12.904259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568038+0000","last_change":"2024-09-13T17:08:11.599453+0000","last_active":"2024-09-13T17:08:12.568038+0000","last_peered":"2024-09-13T17:08:12.568038+0000","last_clean":"2024-09-13T17:08:12.568038+0000","last_became_active":"2024-09-13T17:08:11.599107+0000","last_became_peered":"2024-09-13T17:08:11.599107+0000","last_unstale":"2024-09-13T17:08:12.568038+0000","last_undegraded":"2024-09-13T17:08:12.568038+0000","last_fullsized":"2024-09-13T17:08:12.568038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568117+0000","last_change":"2024-09-13T17:08:11.136652+0000","last_active":"2024-09-13T17:08:12.568117+0000","last_peered":"2024-09-13T17:08:12.568117+0000","last_clean":"2024-09-13T17:08:12.568117+0000","last_became_active":"2024-09-13T17:08:11.135676+0000","last_became_peered":"2024-09-13T17:08:11.135676+0000","last_unstale":"2024-09-13T17:08:12.568117+0000","last_undegraded":"2024-09-13T17:08:12.568117+0000","last_fullsized":"2024-09-13T17:08:12.568117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905584+0000","last_change":"2024-09-13T17:08:11.589078+0000","last_active":"2024-09-13T17:08:12.905584+0000","last_peered":"2024-09-13T17:08:12.905584+0000","last_clean":"2024-09-13T17:08:12.905584+0000","last_became_active":"2024-09-13T17:08:11.588613+0000","last_became_peered":"2024-09-13T17:08:11.588613+0000","last_unstale":"2024-09-13T17:08:12.905584+0000","last_undegraded":"2024-09-13T17:08:12.905584+0000","last_fullsized":"2024-09-13T17:08:12.905584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565993+0000","last_change":"2024-09-13T17:08:11.139774+0000","last_active":"2024-09-13T17:08:12.565993+0000","last_peered":"2024-09-13T17:08:12.565993+0000","last_clean":"2024-09-13T17:08:12.565993+0000","last_became_active":"2024-09-13T17:08:11.139257+0000","last_became_peered":"2024-09-13T17:08:11.139257+0000","last_unstale":"2024-09-13T17:08:12.565993+0000","last_undegraded":"2024-09-13T17:08:12.565993+0000","last_fullsized":"2024-09-13T17:08:12.565993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569193+0000","last_change":"2024-09-13T17:08:11.590978+0000","last_active":"2024-09-13T17:08:12.569193+0000","last_peered":"2024-09-13T17:08:12.569193+0000","last_clean":"2024-09-13T17:08:12.569193+0000","last_became_active":"2024-09-13T17:08:11.590763+0000","last_became_peered":"2024-09-13T17:08:11.590763+0000","last_unstale":"2024-09-13T17:08:12.569193+0000","last_undegraded":"2024-09-13T17:08:12.569193+0000","last_fullsized":"2024-09-13T17:08:12.569193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904200+0000","last_change":"2024-09-13T17:08:11.134868+0000","last_active":"2024-09-13T17:08:12.904200+0000","last_peered":"2024-09-13T17:08:12.904200+0000","last_clean":"2024-09-13T17:08:12.904200+0000","last_became_active":"2024-09-13T17:08:11.134516+0000","last_became_peered":"2024-09-13T17:08:11.134516+0000","last_unstale":"2024-09-13T17:08:12.904200+0000","last_undegraded":"2024-09-13T17:08:12.904200+0000","last_fullsized":"2024-09-13T17:08:12.904200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901654+0000","last_change":"2024-09-13T17:08:11.587369+0000","last_active":"2024-09-13T17:08:12.901654+0000","last_peered":"2024-09-13T17:08:12.901654+0000","last_clean":"2024-09-13T17:08:12.901654+0000","last_became_active":"2024-09-13T17:08:11.586043+0000","last_became_peered":"2024-09-13T17:08:11.586043+0000","last_unstale":"2024-09-13T17:08:12.901654+0000","last_undegraded":"2024-09-13T17:08:12.901654+0000","last_fullsized":"2024-09-13T17:08:12.901654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905533+0000","last_change":"2024-09-13T17:08:11.145171+0000","last_active":"2024-09-13T17:08:12.905533+0000","last_peered":"2024-09-13T17:08:12.905533+0000","last_clean":"2024-09-13T17:08:12.905533+0000","last_became_active":"2024-09-13T17:08:11.144862+0000","last_became_peered":"2024-09-13T17:08:11.144862+0000","last_unstale":"2024-09-13T17:08:12.905533+0000","last_undegraded":"2024-09-13T17:08:12.905533+0000","last_fullsized":"2024-09-13T17:08:12.905533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905497+0000","last_change":"2024-09-13T17:08:11.596289+0000","last_active":"2024-09-13T17:08:12.905497+0000","last_peered":"2024-09-13T17:08:12.905497+0000","last_clean":"2024-09-13T17:08:12.905497+0000","last_became_active":"2024-09-13T17:08:11.595983+0000","last_became_peered":"2024-09-13T17:08:11.595983+0000","last_unstale":"2024-09-13T17:08:12.905497+0000","last_undegraded":"2024-09-13T17:08:12.905497+0000","last_fullsized":"2024-09-13T17:08:12.905497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:39.915077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567483+0000","last_change":"2024-09-13T17:08:11.143981+0000","last_active":"2024-09-13T17:08:12.567483+0000","last_peered":"2024-09-13T17:08:12.567483+0000","last_clean":"2024-09-13T17:08:12.567483+0000","last_became_active":"2024-09-13T17:08:11.143705+0000","last_became_peered":"2024-09-13T17:08:11.143705+0000","last_unstale":"2024-09-13T17:08:12.567483+0000","last_undegraded":"2024-09-13T17:08:12.567483+0000","last_fullsized":"2024-09-13T17:08:12.567483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901597+0000","last_change":"2024-09-13T17:08:11.601054+0000","last_active":"2024-09-13T17:08:12.901597+0000","last_peered":"2024-09-13T17:08:12.901597+0000","last_clean":"2024-09-13T17:08:12.901597+0000","last_became_active":"2024-09-13T17:08:11.600160+0000","last_became_peered":"2024-09-13T17:08:11.600160+0000","last_unstale":"2024-09-13T17:08:12.901597+0000","last_undegraded":"2024-09-13T17:08:12.901597+0000","last_fullsized":"2024-09-13T17:08:12.901597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904157+0000","last_change":"2024-09-13T17:08:11.145590+0000","last_active":"2024-09-13T17:08:12.904157+0000","last_peered":"2024-09-13T17:08:12.904157+0000","last_clean":"2024-09-13T17:08:12.904157+0000","last_became_active":"2024-09-13T17:08:11.145267+0000","last_became_peered":"2024-09-13T17:08:11.145267+0000","last_unstale":"2024-09-13T17:08:12.904157+0000","last_undegraded":"2024-09-13T17:08:12.904157+0000","last_fullsized":"2024-09-13T17:08:12.904157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904144+0000","last_change":"2024-09-13T17:08:11.584806+0000","last_active":"2024-09-13T17:08:12.904144+0000","last_peered":"2024-09-13T17:08:12.904144+0000","last_clean":"2024-09-13T17:08:12.904144+0000","last_became_active":"2024-09-13T17:08:11.584303+0000","last_became_peered":"2024-09-13T17:08:11.584303+0000","last_unstale":"2024-09-13T17:08:12.904144+0000","last_undegraded":"2024-09-13T17:08:12.904144+0000","last_fullsized":"2024-09-13T17:08:12.904144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902697+0000","last_change":"2024-09-13T17:08:11.140195+0000","last_active":"2024-09-13T17:08:12.902697+0000","last_peered":"2024-09-13T17:08:12.902697+0000","last_clean":"2024-09-13T17:08:12.902697+0000","last_became_active":"2024-09-13T17:08:11.139870+0000","last_became_peered":"2024-09-13T17:08:11.139870+0000","last_unstale":"2024-09-13T17:08:12.902697+0000","last_undegraded":"2024-09-13T17:08:12.902697+0000","last_fullsized":"2024-09-13T17:08:12.902697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901563+0000","last_change":"2024-09-13T17:08:11.588867+0000","last_active":"2024-09-13T17:08:12.901563+0000","last_peered":"2024-09-13T17:08:12.901563+0000","last_clean":"2024-09-13T17:08:12.901563+0000","last_became_active":"2024-09-13T17:08:11.588564+0000","last_became_peered":"2024-09-13T17:08:11.588564+0000","last_unstale":"2024-09-13T17:08:12.901563+0000","last_undegraded":"2024-09-13T17:08:12.901563+0000","last_fullsized":"2024-09-13T17:08:12.901563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569202+0000","last_change":"2024-09-13T17:08:11.138154+0000","last_active":"2024-09-13T17:08:12.569202+0000","last_peered":"2024-09-13T17:08:12.569202+0000","last_clean":"2024-09-13T17:08:12.569202+0000","last_became_active":"2024-09-13T17:08:11.137611+0000","last_became_peered":"2024-09-13T17:08:11.137611+0000","last_unstale":"2024-09-13T17:08:12.569202+0000","last_undegraded":"2024-09-13T17:08:12.569202+0000","last_fullsized":"2024-09-13T17:08:12.569202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:48.182859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901727+0000","last_change":"2024-09-13T17:08:11.604320+0000","last_active":"2024-09-13T17:08:12.901727+0000","last_peered":"2024-09-13T17:08:12.901727+0000","last_clean":"2024-09-13T17:08:12.901727+0000","last_became_active":"2024-09-13T17:08:11.603337+0000","last_became_peered":"2024-09-13T17:08:11.603337+0000","last_unstale":"2024-09-13T17:08:12.901727+0000","last_undegraded":"2024-09-13T17:08:12.901727+0000","last_fullsized":"2024-09-13T17:08:12.901727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902754+0000","last_change":"2024-09-13T17:08:11.138593+0000","last_active":"2024-09-13T17:08:12.902754+0000","last_peered":"2024-09-13T17:08:12.902754+0000","last_clean":"2024-09-13T17:08:12.902754+0000","last_became_active":"2024-09-13T17:08:11.138131+0000","last_became_peered":"2024-09-13T17:08:11.138131+0000","last_unstale":"2024-09-13T17:08:12.902754+0000","last_undegraded":"2024-09-13T17:08:12.902754+0000","last_fullsized":"2024-09-13T17:08:12.902754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901773+0000","last_change":"2024-09-13T17:08:11.602024+0000","last_active":"2024-09-13T17:08:12.901773+0000","last_peered":"2024-09-13T17:08:12.901773+0000","last_clean":"2024-09-13T17:08:12.901773+0000","last_became_active":"2024-09-13T17:08:11.601178+0000","last_became_peered":"2024-09-13T17:08:11.601178+0000","last_unstale":"2024-09-13T17:08:12.901773+0000","last_undegraded":"2024-09-13T17:08:12.901773+0000","last_fullsized":"2024-09-13T17:08:12.901773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566047+0000","last_change":"2024-09-13T17:08:11.132903+0000","last_active":"2024-09-13T17:08:12.566047+0000","last_peered":"2024-09-13T17:08:12.566047+0000","last_clean":"2024-09-13T17:08:12.566047+0000","last_became_active":"2024-09-13T17:08:11.131326+0000","last_became_peered":"2024-09-13T17:08:11.131326+0000","last_unstale":"2024-09-13T17:08:12.566047+0000","last_undegraded":"2024-09-13T17:08:12.566047+0000","last_fullsized":"2024-09-13T17:08:12.566047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567836+0000","last_change":"2024-09-13T17:08:11.581340+0000","last_active":"2024-09-13T17:08:12.567836+0000","last_peered":"2024-09-13T17:08:12.567836+0000","last_clean":"2024-09-13T17:08:12.567836+0000","last_became_active":"2024-09-13T17:08:11.578762+0000","last_became_peered":"2024-09-13T17:08:11.578762+0000","last_unstale":"2024-09-13T17:08:12.567836+0000","last_undegraded":"2024-09-13T17:08:12.567836+0000","last_fullsized":"2024-09-13T17:08:12.567836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902734+0000","last_change":"2024-09-13T17:08:11.138090+0000","last_active":"2024-09-13T17:08:12.902734+0000","last_peered":"2024-09-13T17:08:12.902734+0000","last_clean":"2024-09-13T17:08:12.902734+0000","last_became_active":"2024-09-13T17:08:11.137846+0000","last_became_peered":"2024-09-13T17:08:11.137846+0000","last_unstale":"2024-09-13T17:08:12.902734+0000","last_undegraded":"2024-09-13T17:08:12.902734+0000","last_fullsized":"2024-09-13T17:08:12.902734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901665+0000","last_change":"2024-09-13T17:08:11.589182+0000","last_active":"2024-09-13T17:08:12.901665+0000","last_peered":"2024-09-13T17:08:12.901665+0000","last_clean":"2024-09-13T17:08:12.901665+0000","last_became_active":"2024-09-13T17:08:11.588462+0000","last_became_peered":"2024-09-13T17:08:11.588462+0000","last_unstale":"2024-09-13T17:08:12.901665+0000","last_undegraded":"2024-09-13T17:08:12.901665+0000","last_fullsized":"2024-09-13T17:08:12.901665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:40.591592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573999+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:08:11.573999+0000","last_peered":"2024-09-13T17:08:11.573999+0000","last_clean":"2024-09-13T17:08:11.573999+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:08:11.573999+0000","last_undegraded":"2024-09-13T17:08:11.573999+0000","last_fullsized":"2024-09-13T17:08:11.573999+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567696+0000","last_change":"2024-09-13T17:08:11.132650+0000","last_active":"2024-09-13T17:08:12.567696+0000","last_peered":"2024-09-13T17:08:12.567696+0000","last_clean":"2024-09-13T17:08:12.567696+0000","last_became_active":"2024-09-13T17:08:11.132311+0000","last_became_peered":"2024-09-13T17:08:11.132311+0000","last_unstale":"2024-09-13T17:08:12.567696+0000","last_undegraded":"2024-09-13T17:08:12.567696+0000","last_fullsized":"2024-09-13T17:08:12.567696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904676+0000","last_change":"2024-09-13T17:08:11.586834+0000","last_active":"2024-09-13T17:08:12.904676+0000","last_peered":"2024-09-13T17:08:12.904676+0000","last_clean":"2024-09-13T17:08:12.904676+0000","last_became_active":"2024-09-13T17:08:11.585548+0000","last_became_peered":"2024-09-13T17:08:11.585548+0000","last_unstale":"2024-09-13T17:08:12.904676+0000","last_undegraded":"2024-09-13T17:08:12.904676+0000","last_fullsized":"2024-09-13T17:08:12.904676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901478+0000","last_change":"2024-09-13T17:08:11.148517+0000","last_active":"2024-09-13T17:08:12.901478+0000","last_peered":"2024-09-13T17:08:12.901478+0000","last_clean":"2024-09-13T17:08:12.901478+0000","last_became_active":"2024-09-13T17:08:11.148250+0000","last_became_peered":"2024-09-13T17:08:11.148250+0000","last_unstale":"2024-09-13T17:08:12.901478+0000","last_undegraded":"2024-09-13T17:08:12.901478+0000","last_fullsized":"2024-09-13T17:08:12.901478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905473+0000","last_change":"2024-09-13T17:08:11.586386+0000","last_active":"2024-09-13T17:08:12.905473+0000","last_peered":"2024-09-13T17:08:12.905473+0000","last_clean":"2024-09-13T17:08:12.905473+0000","last_became_active":"2024-09-13T17:08:11.577192+0000","last_became_peered":"2024-09-13T17:08:11.577192+0000","last_unstale":"2024-09-13T17:08:12.905473+0000","last_undegraded":"2024-09-13T17:08:12.905473+0000","last_fullsized":"2024-09-13T17:08:12.905473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902726+0000","last_change":"2024-09-13T17:08:11.136035+0000","last_active":"2024-09-13T17:08:12.902726+0000","last_peered":"2024-09-13T17:08:12.902726+0000","last_clean":"2024-09-13T17:08:12.902726+0000","last_became_active":"2024-09-13T17:08:11.135742+0000","last_became_peered":"2024-09-13T17:08:11.135742+0000","last_unstale":"2024-09-13T17:08:12.902726+0000","last_undegraded":"2024-09-13T17:08:12.902726+0000","last_fullsized":"2024-09-13T17:08:12.902726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565763+0000","last_change":"2024-09-13T17:08:11.571193+0000","last_active":"2024-09-13T17:08:12.565763+0000","last_peered":"2024-09-13T17:08:12.565763+0000","last_clean":"2024-09-13T17:08:12.565763+0000","last_became_active":"2024-09-13T17:08:11.570542+0000","last_became_peered":"2024-09-13T17:08:11.570542+0000","last_unstale":"2024-09-13T17:08:12.565763+0000","last_undegraded":"2024-09-13T17:08:12.565763+0000","last_fullsized":"2024-09-13T17:08:12.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567567+0000","last_change":"2024-09-13T17:08:11.140490+0000","last_active":"2024-09-13T17:08:12.567567+0000","last_peered":"2024-09-13T17:08:12.567567+0000","last_clean":"2024-09-13T17:08:12.567567+0000","last_became_active":"2024-09-13T17:08:11.139782+0000","last_became_peered":"2024-09-13T17:08:11.139782+0000","last_unstale":"2024-09-13T17:08:12.567567+0000","last_undegraded":"2024-09-13T17:08:12.567567+0000","last_fullsized":"2024-09-13T17:08:12.567567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568964+0000","last_change":"2024-09-13T17:08:11.578078+0000","last_active":"2024-09-13T17:08:12.568964+0000","last_peered":"2024-09-13T17:08:12.568964+0000","last_clean":"2024-09-13T17:08:12.568964+0000","last_became_active":"2024-09-13T17:08:11.577379+0000","last_became_peered":"2024-09-13T17:08:11.577379+0000","last_unstale":"2024-09-13T17:08:12.568964+0000","last_undegraded":"2024-09-13T17:08:12.568964+0000","last_fullsized":"2024-09-13T17:08:12.568964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565979+0000","last_change":"2024-09-13T17:08:11.135658+0000","last_active":"2024-09-13T17:08:12.565979+0000","last_peered":"2024-09-13T17:08:12.565979+0000","last_clean":"2024-09-13T17:08:12.565979+0000","last_became_active":"2024-09-13T17:08:11.134906+0000","last_became_peered":"2024-09-13T17:08:11.134906+0000","last_unstale":"2024-09-13T17:08:12.565979+0000","last_undegraded":"2024-09-13T17:08:12.565979+0000","last_fullsized":"2024-09-13T17:08:12.565979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568073+0000","last_change":"2024-09-13T17:08:11.580253+0000","last_active":"2024-09-13T17:08:12.568073+0000","last_peered":"2024-09-13T17:08:12.568073+0000","last_clean":"2024-09-13T17:08:12.568073+0000","last_became_active":"2024-09-13T17:08:11.579547+0000","last_became_peered":"2024-09-13T17:08:11.579547+0000","last_unstale":"2024-09-13T17:08:12.568073+0000","last_undegraded":"2024-09-13T17:08:12.568073+0000","last_fullsized":"2024-09-13T17:08:12.568073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901714+0000","last_change":"2024-09-13T17:08:11.149945+0000","last_active":"2024-09-13T17:08:12.901714+0000","last_peered":"2024-09-13T17:08:12.901714+0000","last_clean":"2024-09-13T17:08:12.901714+0000","last_became_active":"2024-09-13T17:08:11.149641+0000","last_became_peered":"2024-09-13T17:08:11.149641+0000","last_unstale":"2024-09-13T17:08:12.901714+0000","last_undegraded":"2024-09-13T17:08:12.901714+0000","last_fullsized":"2024-09-13T17:08:12.901714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565952+0000","last_change":"2024-09-13T17:08:11.578587+0000","last_active":"2024-09-13T17:08:12.565952+0000","last_peered":"2024-09-13T17:08:12.565952+0000","last_clean":"2024-09-13T17:08:12.565952+0000","last_became_active":"2024-09-13T17:08:11.576109+0000","last_became_peered":"2024-09-13T17:08:11.576109+0000","last_unstale":"2024-09-13T17:08:12.565952+0000","last_undegraded":"2024-09-13T17:08:12.565952+0000","last_fullsized":"2024-09-13T17:08:12.565952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901668+0000","last_change":"2024-09-13T17:08:11.141871+0000","last_active":"2024-09-13T17:08:12.901668+0000","last_peered":"2024-09-13T17:08:12.901668+0000","last_clean":"2024-09-13T17:08:12.901668+0000","last_became_active":"2024-09-13T17:08:11.141567+0000","last_became_peered":"2024-09-13T17:08:11.141567+0000","last_unstale":"2024-09-13T17:08:12.901668+0000","last_undegraded":"2024-09-13T17:08:12.901668+0000","last_fullsized":"2024-09-13T17:08:12.901668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904765+0000","last_change":"2024-09-13T17:08:11.589765+0000","last_active":"2024-09-13T17:08:12.904765+0000","last_peered":"2024-09-13T17:08:12.904765+0000","last_clean":"2024-09-13T17:08:12.904765+0000","last_became_active":"2024-09-13T17:08:11.589278+0000","last_became_peered":"2024-09-13T17:08:11.589278+0000","last_unstale":"2024-09-13T17:08:12.904765+0000","last_undegraded":"2024-09-13T17:08:12.904765+0000","last_fullsized":"2024-09-13T17:08:12.904765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904269+0000","last_change":"2024-09-13T17:08:11.148235+0000","last_active":"2024-09-13T17:08:12.904269+0000","last_peered":"2024-09-13T17:08:12.904269+0000","last_clean":"2024-09-13T17:08:12.904269+0000","last_became_active":"2024-09-13T17:08:11.147994+0000","last_became_peered":"2024-09-13T17:08:11.147994+0000","last_unstale":"2024-09-13T17:08:12.904269+0000","last_undegraded":"2024-09-13T17:08:12.904269+0000","last_fullsized":"2024-09-13T17:08:12.904269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905567+0000","last_change":"2024-09-13T17:08:11.570444+0000","last_active":"2024-09-13T17:08:12.905567+0000","last_peered":"2024-09-13T17:08:12.905567+0000","last_clean":"2024-09-13T17:08:12.905567+0000","last_became_active":"2024-09-13T17:08:11.570207+0000","last_became_peered":"2024-09-13T17:08:11.570207+0000","last_unstale":"2024-09-13T17:08:12.905567+0000","last_undegraded":"2024-09-13T17:08:12.905567+0000","last_fullsized":"2024-09-13T17:08:12.905567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901456+0000","last_change":"2024-09-13T17:08:11.147785+0000","last_active":"2024-09-13T17:08:12.901456+0000","last_peered":"2024-09-13T17:08:12.901456+0000","last_clean":"2024-09-13T17:08:12.901456+0000","last_became_active":"2024-09-13T17:08:11.147222+0000","last_became_peered":"2024-09-13T17:08:11.147222+0000","last_unstale":"2024-09-13T17:08:12.901456+0000","last_undegraded":"2024-09-13T17:08:12.901456+0000","last_fullsized":"2024-09-13T17:08:12.901456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567547+0000","last_change":"2024-09-13T17:08:11.574148+0000","last_active":"2024-09-13T17:08:12.567547+0000","last_peered":"2024-09-13T17:08:12.567547+0000","last_clean":"2024-09-13T17:08:12.567547+0000","last_became_active":"2024-09-13T17:08:11.573879+0000","last_became_peered":"2024-09-13T17:08:11.573879+0000","last_unstale":"2024-09-13T17:08:12.567547+0000","last_undegraded":"2024-09-13T17:08:12.567547+0000","last_fullsized":"2024-09-13T17:08:12.567547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:10.871670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565831+0000","last_change":"2024-09-13T17:08:11.146000+0000","last_active":"2024-09-13T17:08:12.565831+0000","last_peered":"2024-09-13T17:08:12.565831+0000","last_clean":"2024-09-13T17:08:12.565831+0000","last_became_active":"2024-09-13T17:08:11.145714+0000","last_became_peered":"2024-09-13T17:08:11.145714+0000","last_unstale":"2024-09-13T17:08:12.565831+0000","last_undegraded":"2024-09-13T17:08:12.565831+0000","last_fullsized":"2024-09-13T17:08:12.565831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904638+0000","last_change":"2024-09-13T17:08:11.573192+0000","last_active":"2024-09-13T17:08:12.904638+0000","last_peered":"2024-09-13T17:08:12.904638+0000","last_clean":"2024-09-13T17:08:12.904638+0000","last_became_active":"2024-09-13T17:08:11.572009+0000","last_became_peered":"2024-09-13T17:08:11.572009+0000","last_unstale":"2024-09-13T17:08:12.904638+0000","last_undegraded":"2024-09-13T17:08:12.904638+0000","last_fullsized":"2024-09-13T17:08:12.904638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565978+0000","last_change":"2024-09-13T17:08:11.145299+0000","last_active":"2024-09-13T17:08:12.565978+0000","last_peered":"2024-09-13T17:08:12.565978+0000","last_clean":"2024-09-13T17:08:12.565978+0000","last_became_active":"2024-09-13T17:08:11.143966+0000","last_became_peered":"2024-09-13T17:08:11.143966+0000","last_unstale":"2024-09-13T17:08:12.565978+0000","last_undegraded":"2024-09-13T17:08:12.565978+0000","last_fullsized":"2024-09-13T17:08:12.565978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904711+0000","last_change":"2024-09-13T17:08:11.573641+0000","last_active":"2024-09-13T17:08:12.904711+0000","last_peered":"2024-09-13T17:08:12.904711+0000","last_clean":"2024-09-13T17:08:12.904711+0000","last_became_active":"2024-09-13T17:08:11.573434+0000","last_became_peered":"2024-09-13T17:08:11.573434+0000","last_unstale":"2024-09-13T17:08:12.904711+0000","last_undegraded":"2024-09-13T17:08:12.904711+0000","last_fullsized":"2024-09-13T17:08:12.904711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901614+0000","last_change":"2024-09-13T17:08:11.148180+0000","last_active":"2024-09-13T17:08:12.901614+0000","last_peered":"2024-09-13T17:08:12.901614+0000","last_clean":"2024-09-13T17:08:12.901614+0000","last_became_active":"2024-09-13T17:08:11.147892+0000","last_became_peered":"2024-09-13T17:08:11.147892+0000","last_unstale":"2024-09-13T17:08:12.901614+0000","last_undegraded":"2024-09-13T17:08:12.901614+0000","last_fullsized":"2024-09-13T17:08:12.901614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904167+0000","last_change":"2024-09-13T17:08:11.589293+0000","last_active":"2024-09-13T17:08:12.904167+0000","last_peered":"2024-09-13T17:08:12.904167+0000","last_clean":"2024-09-13T17:08:12.904167+0000","last_became_active":"2024-09-13T17:08:11.589022+0000","last_became_peered":"2024-09-13T17:08:11.589022+0000","last_unstale":"2024-09-13T17:08:12.904167+0000","last_undegraded":"2024-09-13T17:08:12.904167+0000","last_fullsized":"2024-09-13T17:08:12.904167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:39.460619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568004+0000","last_change":"2024-09-13T17:08:11.594776+0000","last_active":"2024-09-13T17:08:12.568004+0000","last_peered":"2024-09-13T17:08:12.568004+0000","last_clean":"2024-09-13T17:08:12.568004+0000","last_became_active":"2024-09-13T17:08:11.594498+0000","last_became_peered":"2024-09-13T17:08:11.594498+0000","last_unstale":"2024-09-13T17:08:12.568004+0000","last_undegraded":"2024-09-13T17:08:12.568004+0000","last_fullsized":"2024-09-13T17:08:12.568004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901634+0000","last_change":"2024-09-13T17:08:11.149488+0000","last_active":"2024-09-13T17:08:12.901634+0000","last_peered":"2024-09-13T17:08:12.901634+0000","last_clean":"2024-09-13T17:08:12.901634+0000","last_became_active":"2024-09-13T17:08:11.149198+0000","last_became_peered":"2024-09-13T17:08:11.149198+0000","last_unstale":"2024-09-13T17:08:12.901634+0000","last_undegraded":"2024-09-13T17:08:12.901634+0000","last_fullsized":"2024-09-13T17:08:12.901634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:24.611621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567674+0000","last_change":"2024-09-13T17:08:11.597049+0000","last_active":"2024-09-13T17:08:12.567674+0000","last_peered":"2024-09-13T17:08:12.567674+0000","last_clean":"2024-09-13T17:08:12.567674+0000","last_became_active":"2024-09-13T17:08:11.594306+0000","last_became_peered":"2024-09-13T17:08:11.594306+0000","last_unstale":"2024-09-13T17:08:12.567674+0000","last_undegraded":"2024-09-13T17:08:12.567674+0000","last_fullsized":"2024-09-13T17:08:12.567674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901675+0000","last_change":"2024-09-13T17:08:11.152688+0000","last_active":"2024-09-13T17:08:12.901675+0000","last_peered":"2024-09-13T17:08:12.901675+0000","last_clean":"2024-09-13T17:08:12.901675+0000","last_became_active":"2024-09-13T17:08:11.152286+0000","last_became_peered":"2024-09-13T17:08:11.152286+0000","last_unstale":"2024-09-13T17:08:12.901675+0000","last_undegraded":"2024-09-13T17:08:12.901675+0000","last_fullsized":"2024-09-13T17:08:12.901675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905575+0000","last_change":"2024-09-13T17:08:11.588908+0000","last_active":"2024-09-13T17:08:12.905575+0000","last_peered":"2024-09-13T17:08:12.905575+0000","last_clean":"2024-09-13T17:08:12.905575+0000","last_became_active":"2024-09-13T17:08:11.588299+0000","last_became_peered":"2024-09-13T17:08:11.588299+0000","last_unstale":"2024-09-13T17:08:12.905575+0000","last_undegraded":"2024-09-13T17:08:12.905575+0000","last_fullsized":"2024-09-13T17:08:12.905575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904748+0000","last_change":"2024-09-13T17:08:11.140760+0000","last_active":"2024-09-13T17:08:12.904748+0000","last_peered":"2024-09-13T17:08:12.904748+0000","last_clean":"2024-09-13T17:08:12.904748+0000","last_became_active":"2024-09-13T17:08:11.140489+0000","last_became_peered":"2024-09-13T17:08:11.140489+0000","last_unstale":"2024-09-13T17:08:12.904748+0000","last_undegraded":"2024-09-13T17:08:12.904748+0000","last_fullsized":"2024-09-13T17:08:12.904748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904739+0000","last_change":"2024-09-13T17:08:11.591976+0000","last_active":"2024-09-13T17:08:12.904739+0000","last_peered":"2024-09-13T17:08:12.904739+0000","last_clean":"2024-09-13T17:08:12.904739+0000","last_became_active":"2024-09-13T17:08:11.591484+0000","last_became_peered":"2024-09-13T17:08:11.591484+0000","last_unstale":"2024-09-13T17:08:12.904739+0000","last_undegraded":"2024-09-13T17:08:12.904739+0000","last_fullsized":"2024-09-13T17:08:12.904739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905593+0000","last_change":"2024-09-13T17:08:11.145298+0000","last_active":"2024-09-13T17:08:12.905593+0000","last_peered":"2024-09-13T17:08:12.905593+0000","last_clean":"2024-09-13T17:08:12.905593+0000","last_became_active":"2024-09-13T17:08:11.145029+0000","last_became_peered":"2024-09-13T17:08:11.145029+0000","last_unstale":"2024-09-13T17:08:12.905593+0000","last_undegraded":"2024-09-13T17:08:12.905593+0000","last_fullsized":"2024-09-13T17:08:12.905593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8610000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.401}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.899}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":200704,"data_stored":69879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:08:20.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:08:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.016+0000 7f4b40200640 1 Processor -- start 2024-09-13T17:08:21.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.016+0000 7f4b40200640 1 -- start start 2024-09-13T17:08:21.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b40200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b38071c00 0x7f4b38146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b40200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b380ad260 0x7f4b380b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b40200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b380b2a40 con 0x7f4b380b25f0 2024-09-13T17:08:21.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b40200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b38143070 con 0x7f4b380ad260 2024-09-13T17:08:21.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b40200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b38071fe0 con 0x7f4b38071c00 2024-09-13T17:08:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.017+0000 7f4b3ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b380ad260 0x7f4b380b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4b380ad260 0x7f4b380b20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33742/0 (socket says 172.21.15.28:33742) 2024-09-13T17:08:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 -- 172.21.15.28:0/1210183682 learned_addr learned my addr 172.21.15.28:0/1210183682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3d800640 1 --2- 172.21.15.28:0/1210183682 >> v2:172.21.15.189:3300/0 conn(0x7f4b38071c00 0x7f4b38146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 -- 172.21.15.28:0/1210183682 >> v2:172.21.15.189:3300/0 conn(0x7f4b38071c00 msgr2=0x7f4b38146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 --2- 172.21.15.28:0/1210183682 >> v2:172.21.15.189:3300/0 conn(0x7f4b38071c00 0x7f4b38146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 -- 172.21.15.28:0/1210183682 >> v1:172.21.15.189:6789/0 conn(0x7f4b380b25f0 legacy=0x7f4b381604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.018+0000 7f4b3ce00640 1 -- 172.21.15.28:0/1210183682 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b380ada70 con 0x7f4b380ad260 2024-09-13T17:08:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.019+0000 7f4b3ce00640 1 --2- 172.21.15.28:0/1210183682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380ad260 0x7f4b380b20b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b380b4fd0 tx=0x7f4b2402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79437ddf96ed85bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.019+0000 7f4b37e00640 1 -- 172.21.15.28:0/1210183682 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24032440 con 0x7f4b380ad260 2024-09-13T17:08:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.019+0000 7f4b37e00640 1 -- 172.21.15.28:0/1210183682 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b240325a0 con 0x7f4b380ad260 2024-09-13T17:08:21.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.019+0000 7f4b37e00640 1 -- 172.21.15.28:0/1210183682 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b24032930 con 0x7f4b380ad260 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 -- 172.21.15.28:0/1210183682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380ad260 msgr2=0x7f4b380b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 --2- 172.21.15.28:0/1210183682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380ad260 0x7f4b380b20b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4b380b4fd0 tx=0x7f4b2402a270 comp rx=0 tx=0).stop 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 -- 172.21.15.28:0/1210183682 shutdown_connections 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 --2- 172.21.15.28:0/1210183682 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380ad260 0x7f4b380b20b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 --2- 172.21.15.28:0/1210183682 >> v2:172.21.15.189:3300/0 conn(0x7f4b38071c00 0x7f4b38146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 -- 172.21.15.28:0/1210183682 >> 172.21.15.28:0/1210183682 conn(0x7f4b380b0b60 msgr2=0x7f4b380b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 -- 172.21.15.28:0/1210183682 shutdown_connections 2024-09-13T17:08:21.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.020+0000 7f4b40200640 1 -- 172.21.15.28:0/1210183682 wait complete. 2024-09-13T17:08:21.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.021+0000 7f4b40200640 1 Processor -- start 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.022+0000 7f4b40200640 1 -- start start 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.022+0000 7f4b40200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b38071c00 0x7f4b3815c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.022+0000 7f4b40200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b380ad260 0x7f4b381f2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b40200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b40200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3815c590 con 0x7f4b380b25f0 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b40200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380ccb20 con 0x7f4b38071c00 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b40200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b380cce00 con 0x7f4b380ad260 2024-09-13T17:08:21.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b38071c00 0x7f4b3815c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b380ad260 0x7f4b381f2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33744/0 (socket says 172.21.15.28:33744) 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 -- 172.21.15.28:0/3095750128 learned_addr learned my addr 172.21.15.28:0/3095750128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 -- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b380ad260 msgr2=0x7f4b381f2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b380ad260 0x7f4b381f2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 -- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b38071c00 msgr2=0x7f4b3815c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b38071c00 0x7f4b3815c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.023+0000 7f4b3e200640 1 -- 172.21.15.28:0/3095750128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b240094d0 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b3e200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c7a0 tx=0x7f4b3000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30003b70 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b30003cf0 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b380cd0e0 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b30003ea0 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.024+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b380cd6a0 con 0x7f4b380b25f0 2024-09-13T17:08:21.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.025+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b380ad6d0 con 0x7f4b380b25f0 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.026+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4b3000aa90 con 0x7f4b380b25f0 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.027+0000 7f4b35600640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 0x7f4b1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.027+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b30088ef0 con 0x7f4b380b25f0 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.028+0000 7f4b3d800640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 0x7f4b1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.028+0000 7f4b3d800640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 0x7f4b1806d0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4b381461a0 tx=0x7f4b2c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.032+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4b30054170 con 0x7f4b380b25f0 2024-09-13T17:08:21.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.206+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4b380ae310 con 0x7f4b1806ac00 2024-09-13T17:08:21.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.214+0000 7f4b35600640 1 -- 172.21.15.28:0/3095750128 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264574 (secure 0 0 0) 0x7f4b2c011a00 con 0x7f4b1806ac00 2024-09-13T17:08:21.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:21.219 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 msgr2=0x7f4b1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 0x7f4b1806d0c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4b381461a0 tx=0x7f4b2c009190 comp rx=0 tx=0).stop 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 msgr2=0x7f4b38139470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b3000c7a0 tx=0x7f4b3000cc60 comp rx=0 tx=0).stop 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 shutdown_connections 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4b1806ac00 0x7f4b1806d0c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b380b25f0 0x7f4b38139470 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b380ad260 0x7f4b381f2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 --2- 172.21.15.28:0/3095750128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b38071c00 0x7f4b3815c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 >> 172.21.15.28:0/3095750128 conn(0x7f4b380b0b60 msgr2=0x7f4b38070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 shutdown_connections 2024-09-13T17:08:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.217+0000 7f4b40200640 1 -- 172.21.15.28:0/3095750128 wait complete. 2024-09-13T17:08:21.234 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-09-13T17:08:20.886075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235408,"kb_used_data":2816,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754739312,"statfs":{"total":773094113280,"available":772853055488,"internally_reserved":0,"allocated":2883584,"data_stored":1750606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.326471"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901739+0000","last_change":"2024-09-13T17:08:11.601843+0000","last_active":"2024-09-13T17:08:12.901739+0000","last_peered":"2024-09-13T17:08:12.901739+0000","last_clean":"2024-09-13T17:08:12.901739+0000","last_became_active":"2024-09-13T17:08:11.601437+0000","last_became_peered":"2024-09-13T17:08:11.601437+0000","last_unstale":"2024-09-13T17:08:12.901739+0000","last_undegraded":"2024-09-13T17:08:12.901739+0000","last_fullsized":"2024-09-13T17:08:12.901739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:28.411961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566115+0000","last_change":"2024-09-13T17:08:11.576946+0000","last_active":"2024-09-13T17:08:12.566115+0000","last_peered":"2024-09-13T17:08:12.566115+0000","last_clean":"2024-09-13T17:08:12.566115+0000","last_became_active":"2024-09-13T17:08:11.576700+0000","last_became_peered":"2024-09-13T17:08:11.576700+0000","last_unstale":"2024-09-13T17:08:12.566115+0000","last_undegraded":"2024-09-13T17:08:12.566115+0000","last_fullsized":"2024-09-13T17:08:12.566115+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905558+0000","last_change":"2024-09-13T17:08:11.576052+0000","last_active":"2024-09-13T17:08:12.905558+0000","last_peered":"2024-09-13T17:08:12.905558+0000","last_clean":"2024-09-13T17:08:12.905558+0000","last_became_active":"2024-09-13T17:08:11.575297+0000","last_became_peered":"2024-09-13T17:08:11.575297+0000","last_unstale":"2024-09-13T17:08:12.905558+0000","last_undegraded":"2024-09-13T17:08:12.905558+0000","last_fullsized":"2024-09-13T17:08:12.905558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567600+0000","last_change":"2024-09-13T17:08:11.582113+0000","last_active":"2024-09-13T17:08:12.567600+0000","last_peered":"2024-09-13T17:08:12.567600+0000","last_clean":"2024-09-13T17:08:12.567600+0000","last_became_active":"2024-09-13T17:08:11.579185+0000","last_became_peered":"2024-09-13T17:08:11.579185+0000","last_unstale":"2024-09-13T17:08:12.567600+0000","last_undegraded":"2024-09-13T17:08:12.567600+0000","last_fullsized":"2024-09-13T17:08:12.567600+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901581+0000","last_change":"2024-09-13T17:08:11.600241+0000","last_active":"2024-09-13T17:08:12.901581+0000","last_peered":"2024-09-13T17:08:12.901581+0000","last_clean":"2024-09-13T17:08:12.901581+0000","last_became_active":"2024-09-13T17:08:11.596443+0000","last_became_peered":"2024-09-13T17:08:11.596443+0000","last_unstale":"2024-09-13T17:08:12.901581+0000","last_undegraded":"2024-09-13T17:08:12.901581+0000","last_fullsized":"2024-09-13T17:08:12.901581+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567274+0000","last_change":"2024-09-13T17:08:11.583211+0000","last_active":"2024-09-13T17:08:12.567274+0000","last_peered":"2024-09-13T17:08:12.567274+0000","last_clean":"2024-09-13T17:08:12.567274+0000","last_became_active":"2024-09-13T17:08:11.583040+0000","last_became_peered":"2024-09-13T17:08:11.583040+0000","last_unstale":"2024-09-13T17:08:12.567274+0000","last_undegraded":"2024-09-13T17:08:12.567274+0000","last_fullsized":"2024-09-13T17:08:12.567274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901512+0000","last_change":"2024-09-13T17:08:11.592666+0000","last_active":"2024-09-13T17:08:12.901512+0000","last_peered":"2024-09-13T17:08:12.901512+0000","last_clean":"2024-09-13T17:08:12.901512+0000","last_became_active":"2024-09-13T17:08:11.592373+0000","last_became_peered":"2024-09-13T17:08:11.592373+0000","last_unstale":"2024-09-13T17:08:12.901512+0000","last_undegraded":"2024-09-13T17:08:12.901512+0000","last_fullsized":"2024-09-13T17:08:12.901512+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904617+0000","last_change":"2024-09-13T17:08:11.578631+0000","last_active":"2024-09-13T17:08:12.904617+0000","last_peered":"2024-09-13T17:08:12.904617+0000","last_clean":"2024-09-13T17:08:12.904617+0000","last_became_active":"2024-09-13T17:08:11.575801+0000","last_became_peered":"2024-09-13T17:08:11.575801+0000","last_unstale":"2024-09-13T17:08:12.904617+0000","last_undegraded":"2024-09-13T17:08:12.904617+0000","last_fullsized":"2024-09-13T17:08:12.904617+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901752+0000","last_change":"2024-09-13T17:08:11.601554+0000","last_active":"2024-09-13T17:08:12.901752+0000","last_peered":"2024-09-13T17:08:12.901752+0000","last_clean":"2024-09-13T17:08:12.901752+0000","last_became_active":"2024-09-13T17:08:11.601069+0000","last_became_peered":"2024-09-13T17:08:11.601069+0000","last_unstale":"2024-09-13T17:08:12.901752+0000","last_undegraded":"2024-09-13T17:08:12.901752+0000","last_fullsized":"2024-09-13T17:08:12.901752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:24.360659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904240+0000","last_change":"2024-09-13T17:08:11.587154+0000","last_active":"2024-09-13T17:08:12.904240+0000","last_peered":"2024-09-13T17:08:12.904240+0000","last_clean":"2024-09-13T17:08:12.904240+0000","last_became_active":"2024-09-13T17:08:11.586876+0000","last_became_peered":"2024-09-13T17:08:11.586876+0000","last_unstale":"2024-09-13T17:08:12.904240+0000","last_undegraded":"2024-09-13T17:08:12.904240+0000","last_fullsized":"2024-09-13T17:08:12.904240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904210+0000","last_change":"2024-09-13T17:08:11.586451+0000","last_active":"2024-09-13T17:08:12.904210+0000","last_peered":"2024-09-13T17:08:12.904210+0000","last_clean":"2024-09-13T17:08:12.904210+0000","last_became_active":"2024-09-13T17:08:11.586127+0000","last_became_peered":"2024-09-13T17:08:11.586127+0000","last_unstale":"2024-09-13T17:08:12.904210+0000","last_undegraded":"2024-09-13T17:08:12.904210+0000","last_fullsized":"2024-09-13T17:08:12.904210+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569157+0000","last_change":"2024-09-13T17:08:11.591207+0000","last_active":"2024-09-13T17:08:12.569157+0000","last_peered":"2024-09-13T17:08:12.569157+0000","last_clean":"2024-09-13T17:08:12.569157+0000","last_became_active":"2024-09-13T17:08:11.590920+0000","last_became_peered":"2024-09-13T17:08:11.590920+0000","last_unstale":"2024-09-13T17:08:12.569157+0000","last_undegraded":"2024-09-13T17:08:12.569157+0000","last_fullsized":"2024-09-13T17:08:12.569157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:25.618116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904684+0000","last_change":"2024-09-13T17:08:11.586067+0000","last_active":"2024-09-13T17:08:12.904684+0000","last_peered":"2024-09-13T17:08:12.904684+0000","last_clean":"2024-09-13T17:08:12.904684+0000","last_became_active":"2024-09-13T17:08:11.585787+0000","last_became_peered":"2024-09-13T17:08:11.585787+0000","last_unstale":"2024-09-13T17:08:12.904684+0000","last_undegraded":"2024-09-13T17:08:12.904684+0000","last_fullsized":"2024-09-13T17:08:12.904684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905488+0000","last_change":"2024-09-13T17:08:11.570021+0000","last_active":"2024-09-13T17:08:12.905488+0000","last_peered":"2024-09-13T17:08:12.905488+0000","last_clean":"2024-09-13T17:08:12.905488+0000","last_became_active":"2024-09-13T17:08:11.569663+0000","last_became_peered":"2024-09-13T17:08:11.569663+0000","last_unstale":"2024-09-13T17:08:12.905488+0000","last_undegraded":"2024-09-13T17:08:12.905488+0000","last_fullsized":"2024-09-13T17:08:12.905488+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565777+0000","last_change":"2024-09-13T17:08:11.578244+0000","last_active":"2024-09-13T17:08:12.565777+0000","last_peered":"2024-09-13T17:08:12.565777+0000","last_clean":"2024-09-13T17:08:12.565777+0000","last_became_active":"2024-09-13T17:08:11.577989+0000","last_became_peered":"2024-09-13T17:08:11.577989+0000","last_unstale":"2024-09-13T17:08:12.565777+0000","last_undegraded":"2024-09-13T17:08:12.565777+0000","last_fullsized":"2024-09-13T17:08:12.565777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567131+0000","last_change":"2024-09-13T17:08:11.582930+0000","last_active":"2024-09-13T17:08:12.567131+0000","last_peered":"2024-09-13T17:08:12.567131+0000","last_clean":"2024-09-13T17:08:12.567131+0000","last_became_active":"2024-09-13T17:08:11.582621+0000","last_became_peered":"2024-09-13T17:08:11.582621+0000","last_unstale":"2024-09-13T17:08:12.567131+0000","last_undegraded":"2024-09-13T17:08:12.567131+0000","last_fullsized":"2024-09-13T17:08:12.567131+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568144+0000","last_change":"2024-09-13T17:08:11.601899+0000","last_active":"2024-09-13T17:08:12.568144+0000","last_peered":"2024-09-13T17:08:12.568144+0000","last_clean":"2024-09-13T17:08:12.568144+0000","last_became_active":"2024-09-13T17:08:11.601452+0000","last_became_peered":"2024-09-13T17:08:11.601452+0000","last_unstale":"2024-09-13T17:08:12.568144+0000","last_undegraded":"2024-09-13T17:08:12.568144+0000","last_fullsized":"2024-09-13T17:08:12.568144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902746+0000","last_change":"2024-09-13T17:08:11.591991+0000","last_active":"2024-09-13T17:08:12.902746+0000","last_peered":"2024-09-13T17:08:12.902746+0000","last_clean":"2024-09-13T17:08:12.902746+0000","last_became_active":"2024-09-13T17:08:11.591554+0000","last_became_peered":"2024-09-13T17:08:11.591554+0000","last_unstale":"2024-09-13T17:08:12.902746+0000","last_undegraded":"2024-09-13T17:08:12.902746+0000","last_fullsized":"2024-09-13T17:08:12.902746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901703+0000","last_change":"2024-09-13T17:08:11.596861+0000","last_active":"2024-09-13T17:08:12.901703+0000","last_peered":"2024-09-13T17:08:12.901703+0000","last_clean":"2024-09-13T17:08:12.901703+0000","last_became_active":"2024-09-13T17:08:11.596669+0000","last_became_peered":"2024-09-13T17:08:11.596669+0000","last_unstale":"2024-09-13T17:08:12.901703+0000","last_undegraded":"2024-09-13T17:08:12.901703+0000","last_fullsized":"2024-09-13T17:08:12.901703+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904190+0000","last_change":"2024-09-13T17:08:11.582225+0000","last_active":"2024-09-13T17:08:12.904190+0000","last_peered":"2024-09-13T17:08:12.904190+0000","last_clean":"2024-09-13T17:08:12.904190+0000","last_became_active":"2024-09-13T17:08:11.581900+0000","last_became_peered":"2024-09-13T17:08:11.581900+0000","last_unstale":"2024-09-13T17:08:12.904190+0000","last_undegraded":"2024-09-13T17:08:12.904190+0000","last_fullsized":"2024-09-13T17:08:12.904190+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905507+0000","last_change":"2024-09-13T17:08:11.588070+0000","last_active":"2024-09-13T17:08:12.905507+0000","last_peered":"2024-09-13T17:08:12.905507+0000","last_clean":"2024-09-13T17:08:12.905507+0000","last_became_active":"2024-09-13T17:08:11.587859+0000","last_became_peered":"2024-09-13T17:08:11.587859+0000","last_unstale":"2024-09-13T17:08:12.905507+0000","last_undegraded":"2024-09-13T17:08:12.905507+0000","last_fullsized":"2024-09-13T17:08:12.905507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567322+0000","last_change":"2024-09-13T17:08:11.597424+0000","last_active":"2024-09-13T17:08:12.567322+0000","last_peered":"2024-09-13T17:08:12.567322+0000","last_clean":"2024-09-13T17:08:12.567322+0000","last_became_active":"2024-09-13T17:08:11.597093+0000","last_became_peered":"2024-09-13T17:08:11.597093+0000","last_unstale":"2024-09-13T17:08:12.567322+0000","last_undegraded":"2024-09-13T17:08:12.567322+0000","last_fullsized":"2024-09-13T17:08:12.567322+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904647+0000","last_change":"2024-09-13T17:08:11.581514+0000","last_active":"2024-09-13T17:08:12.904647+0000","last_peered":"2024-09-13T17:08:12.904647+0000","last_clean":"2024-09-13T17:08:12.904647+0000","last_became_active":"2024-09-13T17:08:11.581178+0000","last_became_peered":"2024-09-13T17:08:11.581178+0000","last_unstale":"2024-09-13T17:08:12.904647+0000","last_undegraded":"2024-09-13T17:08:12.904647+0000","last_fullsized":"2024-09-13T17:08:12.904647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567215+0000","last_change":"2024-09-13T17:08:11.575094+0000","last_active":"2024-09-13T17:08:12.567215+0000","last_peered":"2024-09-13T17:08:12.567215+0000","last_clean":"2024-09-13T17:08:12.567215+0000","last_became_active":"2024-09-13T17:08:11.574833+0000","last_became_peered":"2024-09-13T17:08:11.574833+0000","last_unstale":"2024-09-13T17:08:12.567215+0000","last_undegraded":"2024-09-13T17:08:12.567215+0000","last_fullsized":"2024-09-13T17:08:12.567215+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904757+0000","last_change":"2024-09-13T17:08:11.581636+0000","last_active":"2024-09-13T17:08:12.904757+0000","last_peered":"2024-09-13T17:08:12.904757+0000","last_clean":"2024-09-13T17:08:12.904757+0000","last_became_active":"2024-09-13T17:08:11.581264+0000","last_became_peered":"2024-09-13T17:08:11.581264+0000","last_unstale":"2024-09-13T17:08:12.904757+0000","last_undegraded":"2024-09-13T17:08:12.904757+0000","last_fullsized":"2024-09-13T17:08:12.904757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:37.010115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904250+0000","last_change":"2024-09-13T17:08:11.601141+0000","last_active":"2024-09-13T17:08:12.904250+0000","last_peered":"2024-09-13T17:08:12.904250+0000","last_clean":"2024-09-13T17:08:12.904250+0000","last_became_active":"2024-09-13T17:08:11.600877+0000","last_became_peered":"2024-09-13T17:08:11.600877+0000","last_unstale":"2024-09-13T17:08:12.904250+0000","last_undegraded":"2024-09-13T17:08:12.904250+0000","last_fullsized":"2024-09-13T17:08:12.904250+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565968+0000","last_change":"2024-09-13T17:08:11.576209+0000","last_active":"2024-09-13T17:08:12.565968+0000","last_peered":"2024-09-13T17:08:12.565968+0000","last_clean":"2024-09-13T17:08:12.565968+0000","last_became_active":"2024-09-13T17:08:11.575917+0000","last_became_peered":"2024-09-13T17:08:11.575917+0000","last_unstale":"2024-09-13T17:08:12.565968+0000","last_undegraded":"2024-09-13T17:08:12.565968+0000","last_fullsized":"2024-09-13T17:08:12.565968+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904731+0000","last_change":"2024-09-13T17:08:11.582222+0000","last_active":"2024-09-13T17:08:12.904731+0000","last_peered":"2024-09-13T17:08:12.904731+0000","last_clean":"2024-09-13T17:08:12.904731+0000","last_became_active":"2024-09-13T17:08:11.582057+0000","last_became_peered":"2024-09-13T17:08:11.582057+0000","last_unstale":"2024-09-13T17:08:12.904731+0000","last_undegraded":"2024-09-13T17:08:12.904731+0000","last_fullsized":"2024-09-13T17:08:12.904731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905515+0000","last_change":"2024-09-13T17:08:11.575089+0000","last_active":"2024-09-13T17:08:12.905515+0000","last_peered":"2024-09-13T17:08:12.905515+0000","last_clean":"2024-09-13T17:08:12.905515+0000","last_became_active":"2024-09-13T17:08:11.574665+0000","last_became_peered":"2024-09-13T17:08:11.574665+0000","last_unstale":"2024-09-13T17:08:12.905515+0000","last_undegraded":"2024-09-13T17:08:12.905515+0000","last_fullsized":"2024-09-13T17:08:12.905515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567404+0000","last_change":"2024-09-13T17:08:11.579735+0000","last_active":"2024-09-13T17:08:12.567404+0000","last_peered":"2024-09-13T17:08:12.567404+0000","last_clean":"2024-09-13T17:08:12.567404+0000","last_became_active":"2024-09-13T17:08:11.578749+0000","last_became_peered":"2024-09-13T17:08:11.578749+0000","last_unstale":"2024-09-13T17:08:12.567404+0000","last_undegraded":"2024-09-13T17:08:12.567404+0000","last_fullsized":"2024-09-13T17:08:12.567404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:29.750286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902716+0000","last_change":"2024-09-13T17:08:11.570273+0000","last_active":"2024-09-13T17:08:12.902716+0000","last_peered":"2024-09-13T17:08:12.902716+0000","last_clean":"2024-09-13T17:08:12.902716+0000","last_became_active":"2024-09-13T17:08:11.569987+0000","last_became_peered":"2024-09-13T17:08:11.569987+0000","last_unstale":"2024-09-13T17:08:12.902716+0000","last_undegraded":"2024-09-13T17:08:12.902716+0000","last_fullsized":"2024-09-13T17:08:12.902716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904126+0000","last_change":"2024-09-13T17:08:11.586071+0000","last_active":"2024-09-13T17:08:12.904126+0000","last_peered":"2024-09-13T17:08:12.904126+0000","last_clean":"2024-09-13T17:08:12.904126+0000","last_became_active":"2024-09-13T17:08:11.585507+0000","last_became_peered":"2024-09-13T17:08:11.585507+0000","last_unstale":"2024-09-13T17:08:12.904126+0000","last_undegraded":"2024-09-13T17:08:12.904126+0000","last_fullsized":"2024-09-13T17:08:12.904126+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567646+0000","last_change":"2024-09-13T17:08:11.578066+0000","last_active":"2024-09-13T17:08:12.567646+0000","last_peered":"2024-09-13T17:08:12.567646+0000","last_clean":"2024-09-13T17:08:12.567646+0000","last_became_active":"2024-09-13T17:08:11.577655+0000","last_became_peered":"2024-09-13T17:08:11.577655+0000","last_unstale":"2024-09-13T17:08:12.567646+0000","last_undegraded":"2024-09-13T17:08:12.567646+0000","last_fullsized":"2024-09-13T17:08:12.567646+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.318727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904721+0000","last_change":"2024-09-13T17:08:11.147841+0000","last_active":"2024-09-13T17:08:12.904721+0000","last_peered":"2024-09-13T17:08:12.904721+0000","last_clean":"2024-09-13T17:08:12.904721+0000","last_became_active":"2024-09-13T17:08:11.147449+0000","last_became_peered":"2024-09-13T17:08:11.147449+0000","last_unstale":"2024-09-13T17:08:12.904721+0000","last_undegraded":"2024-09-13T17:08:12.904721+0000","last_fullsized":"2024-09-13T17:08:12.904721+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905524+0000","last_change":"2024-09-13T17:08:11.580301+0000","last_active":"2024-09-13T17:08:12.905524+0000","last_peered":"2024-09-13T17:08:12.905524+0000","last_clean":"2024-09-13T17:08:12.905524+0000","last_became_active":"2024-09-13T17:08:11.579029+0000","last_became_peered":"2024-09-13T17:08:11.579029+0000","last_unstale":"2024-09-13T17:08:12.905524+0000","last_undegraded":"2024-09-13T17:08:12.905524+0000","last_fullsized":"2024-09-13T17:08:12.905524+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:30.800655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904179+0000","last_change":"2024-09-13T17:08:11.146319+0000","last_active":"2024-09-13T17:08:12.904179+0000","last_peered":"2024-09-13T17:08:12.904179+0000","last_clean":"2024-09-13T17:08:12.904179+0000","last_became_active":"2024-09-13T17:08:11.146057+0000","last_became_peered":"2024-09-13T17:08:11.146057+0000","last_unstale":"2024-09-13T17:08:12.904179+0000","last_undegraded":"2024-09-13T17:08:12.904179+0000","last_fullsized":"2024-09-13T17:08:12.904179+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904667+0000","last_change":"2024-09-13T17:08:11.138543+0000","last_active":"2024-09-13T17:08:12.904667+0000","last_peered":"2024-09-13T17:08:12.904667+0000","last_clean":"2024-09-13T17:08:12.904667+0000","last_became_active":"2024-09-13T17:08:11.133550+0000","last_became_peered":"2024-09-13T17:08:11.133550+0000","last_unstale":"2024-09-13T17:08:12.904667+0000","last_undegraded":"2024-09-13T17:08:12.904667+0000","last_fullsized":"2024-09-13T17:08:12.904667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904629+0000","last_change":"2024-09-13T17:08:11.580334+0000","last_active":"2024-09-13T17:08:12.904629+0000","last_peered":"2024-09-13T17:08:12.904629+0000","last_clean":"2024-09-13T17:08:12.904629+0000","last_became_active":"2024-09-13T17:08:11.574885+0000","last_became_peered":"2024-09-13T17:08:11.574885+0000","last_unstale":"2024-09-13T17:08:12.904629+0000","last_undegraded":"2024-09-13T17:08:12.904629+0000","last_fullsized":"2024-09-13T17:08:12.904629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567242+0000","last_change":"2024-09-13T17:08:11.142227+0000","last_active":"2024-09-13T17:08:12.567242+0000","last_peered":"2024-09-13T17:08:12.567242+0000","last_clean":"2024-09-13T17:08:12.567242+0000","last_became_active":"2024-09-13T17:08:11.142009+0000","last_became_peered":"2024-09-13T17:08:11.142009+0000","last_unstale":"2024-09-13T17:08:12.567242+0000","last_undegraded":"2024-09-13T17:08:12.567242+0000","last_fullsized":"2024-09-13T17:08:12.567242+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565757+0000","last_change":"2024-09-13T17:08:11.569231+0000","last_active":"2024-09-13T17:08:12.565757+0000","last_peered":"2024-09-13T17:08:12.565757+0000","last_clean":"2024-09-13T17:08:12.565757+0000","last_became_active":"2024-09-13T17:08:11.568934+0000","last_became_peered":"2024-09-13T17:08:11.568934+0000","last_unstale":"2024-09-13T17:08:12.565757+0000","last_undegraded":"2024-09-13T17:08:12.565757+0000","last_fullsized":"2024-09-13T17:08:12.565757+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:24:18.929214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904259+0000","last_change":"2024-09-13T17:08:11.140675+0000","last_active":"2024-09-13T17:08:12.904259+0000","last_peered":"2024-09-13T17:08:12.904259+0000","last_clean":"2024-09-13T17:08:12.904259+0000","last_became_active":"2024-09-13T17:08:11.139918+0000","last_became_peered":"2024-09-13T17:08:11.139918+0000","last_unstale":"2024-09-13T17:08:12.904259+0000","last_undegraded":"2024-09-13T17:08:12.904259+0000","last_fullsized":"2024-09-13T17:08:12.904259+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568038+0000","last_change":"2024-09-13T17:08:11.599453+0000","last_active":"2024-09-13T17:08:12.568038+0000","last_peered":"2024-09-13T17:08:12.568038+0000","last_clean":"2024-09-13T17:08:12.568038+0000","last_became_active":"2024-09-13T17:08:11.599107+0000","last_became_peered":"2024-09-13T17:08:11.599107+0000","last_unstale":"2024-09-13T17:08:12.568038+0000","last_undegraded":"2024-09-13T17:08:12.568038+0000","last_fullsized":"2024-09-13T17:08:12.568038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568117+0000","last_change":"2024-09-13T17:08:11.136652+0000","last_active":"2024-09-13T17:08:12.568117+0000","last_peered":"2024-09-13T17:08:12.568117+0000","last_clean":"2024-09-13T17:08:12.568117+0000","last_became_active":"2024-09-13T17:08:11.135676+0000","last_became_peered":"2024-09-13T17:08:11.135676+0000","last_unstale":"2024-09-13T17:08:12.568117+0000","last_undegraded":"2024-09-13T17:08:12.568117+0000","last_fullsized":"2024-09-13T17:08:12.568117+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905584+0000","last_change":"2024-09-13T17:08:11.589078+0000","last_active":"2024-09-13T17:08:12.905584+0000","last_peered":"2024-09-13T17:08:12.905584+0000","last_clean":"2024-09-13T17:08:12.905584+0000","last_became_active":"2024-09-13T17:08:11.588613+0000","last_became_peered":"2024-09-13T17:08:11.588613+0000","last_unstale":"2024-09-13T17:08:12.905584+0000","last_undegraded":"2024-09-13T17:08:12.905584+0000","last_fullsized":"2024-09-13T17:08:12.905584+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565993+0000","last_change":"2024-09-13T17:08:11.139774+0000","last_active":"2024-09-13T17:08:12.565993+0000","last_peered":"2024-09-13T17:08:12.565993+0000","last_clean":"2024-09-13T17:08:12.565993+0000","last_became_active":"2024-09-13T17:08:11.139257+0000","last_became_peered":"2024-09-13T17:08:11.139257+0000","last_unstale":"2024-09-13T17:08:12.565993+0000","last_undegraded":"2024-09-13T17:08:12.565993+0000","last_fullsized":"2024-09-13T17:08:12.565993+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569193+0000","last_change":"2024-09-13T17:08:11.590978+0000","last_active":"2024-09-13T17:08:12.569193+0000","last_peered":"2024-09-13T17:08:12.569193+0000","last_clean":"2024-09-13T17:08:12.569193+0000","last_became_active":"2024-09-13T17:08:11.590763+0000","last_became_peered":"2024-09-13T17:08:11.590763+0000","last_unstale":"2024-09-13T17:08:12.569193+0000","last_undegraded":"2024-09-13T17:08:12.569193+0000","last_fullsized":"2024-09-13T17:08:12.569193+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:46.889218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":17,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904200+0000","last_change":"2024-09-13T17:08:11.134868+0000","last_active":"2024-09-13T17:08:12.904200+0000","last_peered":"2024-09-13T17:08:12.904200+0000","last_clean":"2024-09-13T17:08:12.904200+0000","last_became_active":"2024-09-13T17:08:11.134516+0000","last_became_peered":"2024-09-13T17:08:11.134516+0000","last_unstale":"2024-09-13T17:08:12.904200+0000","last_undegraded":"2024-09-13T17:08:12.904200+0000","last_fullsized":"2024-09-13T17:08:12.904200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901654+0000","last_change":"2024-09-13T17:08:11.587369+0000","last_active":"2024-09-13T17:08:12.901654+0000","last_peered":"2024-09-13T17:08:12.901654+0000","last_clean":"2024-09-13T17:08:12.901654+0000","last_became_active":"2024-09-13T17:08:11.586043+0000","last_became_peered":"2024-09-13T17:08:11.586043+0000","last_unstale":"2024-09-13T17:08:12.901654+0000","last_undegraded":"2024-09-13T17:08:12.901654+0000","last_fullsized":"2024-09-13T17:08:12.901654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:00.664076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905533+0000","last_change":"2024-09-13T17:08:11.145171+0000","last_active":"2024-09-13T17:08:12.905533+0000","last_peered":"2024-09-13T17:08:12.905533+0000","last_clean":"2024-09-13T17:08:12.905533+0000","last_became_active":"2024-09-13T17:08:11.144862+0000","last_became_peered":"2024-09-13T17:08:11.144862+0000","last_unstale":"2024-09-13T17:08:12.905533+0000","last_undegraded":"2024-09-13T17:08:12.905533+0000","last_fullsized":"2024-09-13T17:08:12.905533+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905497+0000","last_change":"2024-09-13T17:08:11.596289+0000","last_active":"2024-09-13T17:08:12.905497+0000","last_peered":"2024-09-13T17:08:12.905497+0000","last_clean":"2024-09-13T17:08:12.905497+0000","last_became_active":"2024-09-13T17:08:11.595983+0000","last_became_peered":"2024-09-13T17:08:11.595983+0000","last_unstale":"2024-09-13T17:08:12.905497+0000","last_undegraded":"2024-09-13T17:08:12.905497+0000","last_fullsized":"2024-09-13T17:08:12.905497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:39.915077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567483+0000","last_change":"2024-09-13T17:08:11.143981+0000","last_active":"2024-09-13T17:08:12.567483+0000","last_peered":"2024-09-13T17:08:12.567483+0000","last_clean":"2024-09-13T17:08:12.567483+0000","last_became_active":"2024-09-13T17:08:11.143705+0000","last_became_peered":"2024-09-13T17:08:11.143705+0000","last_unstale":"2024-09-13T17:08:12.567483+0000","last_undegraded":"2024-09-13T17:08:12.567483+0000","last_fullsized":"2024-09-13T17:08:12.567483+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901597+0000","last_change":"2024-09-13T17:08:11.601054+0000","last_active":"2024-09-13T17:08:12.901597+0000","last_peered":"2024-09-13T17:08:12.901597+0000","last_clean":"2024-09-13T17:08:12.901597+0000","last_became_active":"2024-09-13T17:08:11.600160+0000","last_became_peered":"2024-09-13T17:08:11.600160+0000","last_unstale":"2024-09-13T17:08:12.901597+0000","last_undegraded":"2024-09-13T17:08:12.901597+0000","last_fullsized":"2024-09-13T17:08:12.901597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:22.045781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904157+0000","last_change":"2024-09-13T17:08:11.145590+0000","last_active":"2024-09-13T17:08:12.904157+0000","last_peered":"2024-09-13T17:08:12.904157+0000","last_clean":"2024-09-13T17:08:12.904157+0000","last_became_active":"2024-09-13T17:08:11.145267+0000","last_became_peered":"2024-09-13T17:08:11.145267+0000","last_unstale":"2024-09-13T17:08:12.904157+0000","last_undegraded":"2024-09-13T17:08:12.904157+0000","last_fullsized":"2024-09-13T17:08:12.904157+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904144+0000","last_change":"2024-09-13T17:08:11.584806+0000","last_active":"2024-09-13T17:08:12.904144+0000","last_peered":"2024-09-13T17:08:12.904144+0000","last_clean":"2024-09-13T17:08:12.904144+0000","last_became_active":"2024-09-13T17:08:11.584303+0000","last_became_peered":"2024-09-13T17:08:11.584303+0000","last_unstale":"2024-09-13T17:08:12.904144+0000","last_undegraded":"2024-09-13T17:08:12.904144+0000","last_fullsized":"2024-09-13T17:08:12.904144+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:27.468098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902697+0000","last_change":"2024-09-13T17:08:11.140195+0000","last_active":"2024-09-13T17:08:12.902697+0000","last_peered":"2024-09-13T17:08:12.902697+0000","last_clean":"2024-09-13T17:08:12.902697+0000","last_became_active":"2024-09-13T17:08:11.139870+0000","last_became_peered":"2024-09-13T17:08:11.139870+0000","last_unstale":"2024-09-13T17:08:12.902697+0000","last_undegraded":"2024-09-13T17:08:12.902697+0000","last_fullsized":"2024-09-13T17:08:12.902697+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:37.847205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901563+0000","last_change":"2024-09-13T17:08:11.588867+0000","last_active":"2024-09-13T17:08:12.901563+0000","last_peered":"2024-09-13T17:08:12.901563+0000","last_clean":"2024-09-13T17:08:12.901563+0000","last_became_active":"2024-09-13T17:08:11.588564+0000","last_became_peered":"2024-09-13T17:08:11.588564+0000","last_unstale":"2024-09-13T17:08:12.901563+0000","last_undegraded":"2024-09-13T17:08:12.901563+0000","last_fullsized":"2024-09-13T17:08:12.901563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:43.962242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.569202+0000","last_change":"2024-09-13T17:08:11.138154+0000","last_active":"2024-09-13T17:08:12.569202+0000","last_peered":"2024-09-13T17:08:12.569202+0000","last_clean":"2024-09-13T17:08:12.569202+0000","last_became_active":"2024-09-13T17:08:11.137611+0000","last_became_peered":"2024-09-13T17:08:11.137611+0000","last_unstale":"2024-09-13T17:08:12.569202+0000","last_undegraded":"2024-09-13T17:08:12.569202+0000","last_fullsized":"2024-09-13T17:08:12.569202+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:48.182859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901727+0000","last_change":"2024-09-13T17:08:11.604320+0000","last_active":"2024-09-13T17:08:12.901727+0000","last_peered":"2024-09-13T17:08:12.901727+0000","last_clean":"2024-09-13T17:08:12.901727+0000","last_became_active":"2024-09-13T17:08:11.603337+0000","last_became_peered":"2024-09-13T17:08:11.603337+0000","last_unstale":"2024-09-13T17:08:12.901727+0000","last_undegraded":"2024-09-13T17:08:12.901727+0000","last_fullsized":"2024-09-13T17:08:12.901727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:44.377934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902754+0000","last_change":"2024-09-13T17:08:11.138593+0000","last_active":"2024-09-13T17:08:12.902754+0000","last_peered":"2024-09-13T17:08:12.902754+0000","last_clean":"2024-09-13T17:08:12.902754+0000","last_became_active":"2024-09-13T17:08:11.138131+0000","last_became_peered":"2024-09-13T17:08:11.138131+0000","last_unstale":"2024-09-13T17:08:12.902754+0000","last_undegraded":"2024-09-13T17:08:12.902754+0000","last_fullsized":"2024-09-13T17:08:12.902754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:12.414178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901773+0000","last_change":"2024-09-13T17:08:11.602024+0000","last_active":"2024-09-13T17:08:12.901773+0000","last_peered":"2024-09-13T17:08:12.901773+0000","last_clean":"2024-09-13T17:08:12.901773+0000","last_became_active":"2024-09-13T17:08:11.601178+0000","last_became_peered":"2024-09-13T17:08:11.601178+0000","last_unstale":"2024-09-13T17:08:12.901773+0000","last_undegraded":"2024-09-13T17:08:12.901773+0000","last_fullsized":"2024-09-13T17:08:12.901773+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.566047+0000","last_change":"2024-09-13T17:08:11.132903+0000","last_active":"2024-09-13T17:08:12.566047+0000","last_peered":"2024-09-13T17:08:12.566047+0000","last_clean":"2024-09-13T17:08:12.566047+0000","last_became_active":"2024-09-13T17:08:11.131326+0000","last_became_peered":"2024-09-13T17:08:11.131326+0000","last_unstale":"2024-09-13T17:08:12.566047+0000","last_undegraded":"2024-09-13T17:08:12.566047+0000","last_fullsized":"2024-09-13T17:08:12.566047+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567836+0000","last_change":"2024-09-13T17:08:11.581340+0000","last_active":"2024-09-13T17:08:12.567836+0000","last_peered":"2024-09-13T17:08:12.567836+0000","last_clean":"2024-09-13T17:08:12.567836+0000","last_became_active":"2024-09-13T17:08:11.578762+0000","last_became_peered":"2024-09-13T17:08:11.578762+0000","last_unstale":"2024-09-13T17:08:12.567836+0000","last_undegraded":"2024-09-13T17:08:12.567836+0000","last_fullsized":"2024-09-13T17:08:12.567836+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:32.158022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902734+0000","last_change":"2024-09-13T17:08:11.138090+0000","last_active":"2024-09-13T17:08:12.902734+0000","last_peered":"2024-09-13T17:08:12.902734+0000","last_clean":"2024-09-13T17:08:12.902734+0000","last_became_active":"2024-09-13T17:08:11.137846+0000","last_became_peered":"2024-09-13T17:08:11.137846+0000","last_unstale":"2024-09-13T17:08:12.902734+0000","last_undegraded":"2024-09-13T17:08:12.902734+0000","last_fullsized":"2024-09-13T17:08:12.902734+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901665+0000","last_change":"2024-09-13T17:08:11.589182+0000","last_active":"2024-09-13T17:08:12.901665+0000","last_peered":"2024-09-13T17:08:12.901665+0000","last_clean":"2024-09-13T17:08:12.901665+0000","last_became_active":"2024-09-13T17:08:11.588462+0000","last_became_peered":"2024-09-13T17:08:11.588462+0000","last_unstale":"2024-09-13T17:08:12.901665+0000","last_undegraded":"2024-09-13T17:08:12.901665+0000","last_fullsized":"2024-09-13T17:08:12.901665+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:40.591592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:11.573999+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:08:11.573999+0000","last_peered":"2024-09-13T17:08:11.573999+0000","last_clean":"2024-09-13T17:08:11.573999+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:08:11.573999+0000","last_undegraded":"2024-09-13T17:08:11.573999+0000","last_fullsized":"2024-09-13T17:08:11.573999+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567696+0000","last_change":"2024-09-13T17:08:11.132650+0000","last_active":"2024-09-13T17:08:12.567696+0000","last_peered":"2024-09-13T17:08:12.567696+0000","last_clean":"2024-09-13T17:08:12.567696+0000","last_became_active":"2024-09-13T17:08:11.132311+0000","last_became_peered":"2024-09-13T17:08:11.132311+0000","last_unstale":"2024-09-13T17:08:12.567696+0000","last_undegraded":"2024-09-13T17:08:12.567696+0000","last_fullsized":"2024-09-13T17:08:12.567696+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904676+0000","last_change":"2024-09-13T17:08:11.586834+0000","last_active":"2024-09-13T17:08:12.904676+0000","last_peered":"2024-09-13T17:08:12.904676+0000","last_clean":"2024-09-13T17:08:12.904676+0000","last_became_active":"2024-09-13T17:08:11.585548+0000","last_became_peered":"2024-09-13T17:08:11.585548+0000","last_unstale":"2024-09-13T17:08:12.904676+0000","last_undegraded":"2024-09-13T17:08:12.904676+0000","last_fullsized":"2024-09-13T17:08:12.904676+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901478+0000","last_change":"2024-09-13T17:08:11.148517+0000","last_active":"2024-09-13T17:08:12.901478+0000","last_peered":"2024-09-13T17:08:12.901478+0000","last_clean":"2024-09-13T17:08:12.901478+0000","last_became_active":"2024-09-13T17:08:11.148250+0000","last_became_peered":"2024-09-13T17:08:11.148250+0000","last_unstale":"2024-09-13T17:08:12.901478+0000","last_undegraded":"2024-09-13T17:08:12.901478+0000","last_fullsized":"2024-09-13T17:08:12.901478+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:45.882723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905473+0000","last_change":"2024-09-13T17:08:11.586386+0000","last_active":"2024-09-13T17:08:12.905473+0000","last_peered":"2024-09-13T17:08:12.905473+0000","last_clean":"2024-09-13T17:08:12.905473+0000","last_became_active":"2024-09-13T17:08:11.577192+0000","last_became_peered":"2024-09-13T17:08:11.577192+0000","last_unstale":"2024-09-13T17:08:12.905473+0000","last_undegraded":"2024-09-13T17:08:12.905473+0000","last_fullsized":"2024-09-13T17:08:12.905473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.902726+0000","last_change":"2024-09-13T17:08:11.136035+0000","last_active":"2024-09-13T17:08:12.902726+0000","last_peered":"2024-09-13T17:08:12.902726+0000","last_clean":"2024-09-13T17:08:12.902726+0000","last_became_active":"2024-09-13T17:08:11.135742+0000","last_became_peered":"2024-09-13T17:08:11.135742+0000","last_unstale":"2024-09-13T17:08:12.902726+0000","last_undegraded":"2024-09-13T17:08:12.902726+0000","last_fullsized":"2024-09-13T17:08:12.902726+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565763+0000","last_change":"2024-09-13T17:08:11.571193+0000","last_active":"2024-09-13T17:08:12.565763+0000","last_peered":"2024-09-13T17:08:12.565763+0000","last_clean":"2024-09-13T17:08:12.565763+0000","last_became_active":"2024-09-13T17:08:11.570542+0000","last_became_peered":"2024-09-13T17:08:11.570542+0000","last_unstale":"2024-09-13T17:08:12.565763+0000","last_undegraded":"2024-09-13T17:08:12.565763+0000","last_fullsized":"2024-09-13T17:08:12.565763+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567567+0000","last_change":"2024-09-13T17:08:11.140490+0000","last_active":"2024-09-13T17:08:12.567567+0000","last_peered":"2024-09-13T17:08:12.567567+0000","last_clean":"2024-09-13T17:08:12.567567+0000","last_became_active":"2024-09-13T17:08:11.139782+0000","last_became_peered":"2024-09-13T17:08:11.139782+0000","last_unstale":"2024-09-13T17:08:12.567567+0000","last_undegraded":"2024-09-13T17:08:12.567567+0000","last_fullsized":"2024-09-13T17:08:12.567567+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568964+0000","last_change":"2024-09-13T17:08:11.578078+0000","last_active":"2024-09-13T17:08:12.568964+0000","last_peered":"2024-09-13T17:08:12.568964+0000","last_clean":"2024-09-13T17:08:12.568964+0000","last_became_active":"2024-09-13T17:08:11.577379+0000","last_became_peered":"2024-09-13T17:08:11.577379+0000","last_unstale":"2024-09-13T17:08:12.568964+0000","last_undegraded":"2024-09-13T17:08:12.568964+0000","last_fullsized":"2024-09-13T17:08:12.568964+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.265042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565979+0000","last_change":"2024-09-13T17:08:11.135658+0000","last_active":"2024-09-13T17:08:12.565979+0000","last_peered":"2024-09-13T17:08:12.565979+0000","last_clean":"2024-09-13T17:08:12.565979+0000","last_became_active":"2024-09-13T17:08:11.134906+0000","last_became_peered":"2024-09-13T17:08:11.134906+0000","last_unstale":"2024-09-13T17:08:12.565979+0000","last_undegraded":"2024-09-13T17:08:12.565979+0000","last_fullsized":"2024-09-13T17:08:12.565979+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568073+0000","last_change":"2024-09-13T17:08:11.580253+0000","last_active":"2024-09-13T17:08:12.568073+0000","last_peered":"2024-09-13T17:08:12.568073+0000","last_clean":"2024-09-13T17:08:12.568073+0000","last_became_active":"2024-09-13T17:08:11.579547+0000","last_became_peered":"2024-09-13T17:08:11.579547+0000","last_unstale":"2024-09-13T17:08:12.568073+0000","last_undegraded":"2024-09-13T17:08:12.568073+0000","last_fullsized":"2024-09-13T17:08:12.568073+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:05.552998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901714+0000","last_change":"2024-09-13T17:08:11.149945+0000","last_active":"2024-09-13T17:08:12.901714+0000","last_peered":"2024-09-13T17:08:12.901714+0000","last_clean":"2024-09-13T17:08:12.901714+0000","last_became_active":"2024-09-13T17:08:11.149641+0000","last_became_peered":"2024-09-13T17:08:11.149641+0000","last_unstale":"2024-09-13T17:08:12.901714+0000","last_undegraded":"2024-09-13T17:08:12.901714+0000","last_fullsized":"2024-09-13T17:08:12.901714+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565952+0000","last_change":"2024-09-13T17:08:11.578587+0000","last_active":"2024-09-13T17:08:12.565952+0000","last_peered":"2024-09-13T17:08:12.565952+0000","last_clean":"2024-09-13T17:08:12.565952+0000","last_became_active":"2024-09-13T17:08:11.576109+0000","last_became_peered":"2024-09-13T17:08:11.576109+0000","last_unstale":"2024-09-13T17:08:12.565952+0000","last_undegraded":"2024-09-13T17:08:12.565952+0000","last_fullsized":"2024-09-13T17:08:12.565952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901668+0000","last_change":"2024-09-13T17:08:11.141871+0000","last_active":"2024-09-13T17:08:12.901668+0000","last_peered":"2024-09-13T17:08:12.901668+0000","last_clean":"2024-09-13T17:08:12.901668+0000","last_became_active":"2024-09-13T17:08:11.141567+0000","last_became_peered":"2024-09-13T17:08:11.141567+0000","last_unstale":"2024-09-13T17:08:12.901668+0000","last_undegraded":"2024-09-13T17:08:12.901668+0000","last_fullsized":"2024-09-13T17:08:12.901668+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:37.629550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904765+0000","last_change":"2024-09-13T17:08:11.589765+0000","last_active":"2024-09-13T17:08:12.904765+0000","last_peered":"2024-09-13T17:08:12.904765+0000","last_clean":"2024-09-13T17:08:12.904765+0000","last_became_active":"2024-09-13T17:08:11.589278+0000","last_became_peered":"2024-09-13T17:08:11.589278+0000","last_unstale":"2024-09-13T17:08:12.904765+0000","last_undegraded":"2024-09-13T17:08:12.904765+0000","last_fullsized":"2024-09-13T17:08:12.904765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:20.875376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904269+0000","last_change":"2024-09-13T17:08:11.148235+0000","last_active":"2024-09-13T17:08:12.904269+0000","last_peered":"2024-09-13T17:08:12.904269+0000","last_clean":"2024-09-13T17:08:12.904269+0000","last_became_active":"2024-09-13T17:08:11.147994+0000","last_became_peered":"2024-09-13T17:08:11.147994+0000","last_unstale":"2024-09-13T17:08:12.904269+0000","last_undegraded":"2024-09-13T17:08:12.904269+0000","last_fullsized":"2024-09-13T17:08:12.904269+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905567+0000","last_change":"2024-09-13T17:08:11.570444+0000","last_active":"2024-09-13T17:08:12.905567+0000","last_peered":"2024-09-13T17:08:12.905567+0000","last_clean":"2024-09-13T17:08:12.905567+0000","last_became_active":"2024-09-13T17:08:11.570207+0000","last_became_peered":"2024-09-13T17:08:11.570207+0000","last_unstale":"2024-09-13T17:08:12.905567+0000","last_undegraded":"2024-09-13T17:08:12.905567+0000","last_fullsized":"2024-09-13T17:08:12.905567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901456+0000","last_change":"2024-09-13T17:08:11.147785+0000","last_active":"2024-09-13T17:08:12.901456+0000","last_peered":"2024-09-13T17:08:12.901456+0000","last_clean":"2024-09-13T17:08:12.901456+0000","last_became_active":"2024-09-13T17:08:11.147222+0000","last_became_peered":"2024-09-13T17:08:11.147222+0000","last_unstale":"2024-09-13T17:08:12.901456+0000","last_undegraded":"2024-09-13T17:08:12.901456+0000","last_fullsized":"2024-09-13T17:08:12.901456+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:42.056644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567547+0000","last_change":"2024-09-13T17:08:11.574148+0000","last_active":"2024-09-13T17:08:12.567547+0000","last_peered":"2024-09-13T17:08:12.567547+0000","last_clean":"2024-09-13T17:08:12.567547+0000","last_became_active":"2024-09-13T17:08:11.573879+0000","last_became_peered":"2024-09-13T17:08:11.573879+0000","last_unstale":"2024-09-13T17:08:12.567547+0000","last_undegraded":"2024-09-13T17:08:12.567547+0000","last_fullsized":"2024-09-13T17:08:12.567547+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:10.871670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565831+0000","last_change":"2024-09-13T17:08:11.146000+0000","last_active":"2024-09-13T17:08:12.565831+0000","last_peered":"2024-09-13T17:08:12.565831+0000","last_clean":"2024-09-13T17:08:12.565831+0000","last_became_active":"2024-09-13T17:08:11.145714+0000","last_became_peered":"2024-09-13T17:08:11.145714+0000","last_unstale":"2024-09-13T17:08:12.565831+0000","last_undegraded":"2024-09-13T17:08:12.565831+0000","last_fullsized":"2024-09-13T17:08:12.565831+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:50:09.865175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904638+0000","last_change":"2024-09-13T17:08:11.573192+0000","last_active":"2024-09-13T17:08:12.904638+0000","last_peered":"2024-09-13T17:08:12.904638+0000","last_clean":"2024-09-13T17:08:12.904638+0000","last_became_active":"2024-09-13T17:08:11.572009+0000","last_became_peered":"2024-09-13T17:08:11.572009+0000","last_unstale":"2024-09-13T17:08:12.904638+0000","last_undegraded":"2024-09-13T17:08:12.904638+0000","last_fullsized":"2024-09-13T17:08:12.904638+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:10.582386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.565978+0000","last_change":"2024-09-13T17:08:11.145299+0000","last_active":"2024-09-13T17:08:12.565978+0000","last_peered":"2024-09-13T17:08:12.565978+0000","last_clean":"2024-09-13T17:08:12.565978+0000","last_became_active":"2024-09-13T17:08:11.143966+0000","last_became_peered":"2024-09-13T17:08:11.143966+0000","last_unstale":"2024-09-13T17:08:12.565978+0000","last_undegraded":"2024-09-13T17:08:12.565978+0000","last_fullsized":"2024-09-13T17:08:12.565978+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:19:06.312938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904711+0000","last_change":"2024-09-13T17:08:11.573641+0000","last_active":"2024-09-13T17:08:12.904711+0000","last_peered":"2024-09-13T17:08:12.904711+0000","last_clean":"2024-09-13T17:08:12.904711+0000","last_became_active":"2024-09-13T17:08:11.573434+0000","last_became_peered":"2024-09-13T17:08:11.573434+0000","last_unstale":"2024-09-13T17:08:12.904711+0000","last_undegraded":"2024-09-13T17:08:12.904711+0000","last_fullsized":"2024-09-13T17:08:12.904711+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:23.922041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901614+0000","last_change":"2024-09-13T17:08:11.148180+0000","last_active":"2024-09-13T17:08:12.901614+0000","last_peered":"2024-09-13T17:08:12.901614+0000","last_clean":"2024-09-13T17:08:12.901614+0000","last_became_active":"2024-09-13T17:08:11.147892+0000","last_became_peered":"2024-09-13T17:08:11.147892+0000","last_unstale":"2024-09-13T17:08:12.901614+0000","last_undegraded":"2024-09-13T17:08:12.901614+0000","last_fullsized":"2024-09-13T17:08:12.901614+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:38.312232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904167+0000","last_change":"2024-09-13T17:08:11.589293+0000","last_active":"2024-09-13T17:08:12.904167+0000","last_peered":"2024-09-13T17:08:12.904167+0000","last_clean":"2024-09-13T17:08:12.904167+0000","last_became_active":"2024-09-13T17:08:11.589022+0000","last_became_peered":"2024-09-13T17:08:11.589022+0000","last_unstale":"2024-09-13T17:08:12.904167+0000","last_undegraded":"2024-09-13T17:08:12.904167+0000","last_fullsized":"2024-09-13T17:08:12.904167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:14:39.460619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.568004+0000","last_change":"2024-09-13T17:08:11.594776+0000","last_active":"2024-09-13T17:08:12.568004+0000","last_peered":"2024-09-13T17:08:12.568004+0000","last_clean":"2024-09-13T17:08:12.568004+0000","last_became_active":"2024-09-13T17:08:11.594498+0000","last_became_peered":"2024-09-13T17:08:11.594498+0000","last_unstale":"2024-09-13T17:08:12.568004+0000","last_undegraded":"2024-09-13T17:08:12.568004+0000","last_fullsized":"2024-09-13T17:08:12.568004+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:41:27.016260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901634+0000","last_change":"2024-09-13T17:08:11.149488+0000","last_active":"2024-09-13T17:08:12.901634+0000","last_peered":"2024-09-13T17:08:12.901634+0000","last_clean":"2024-09-13T17:08:12.901634+0000","last_became_active":"2024-09-13T17:08:11.149198+0000","last_became_peered":"2024-09-13T17:08:11.149198+0000","last_unstale":"2024-09-13T17:08:12.901634+0000","last_undegraded":"2024-09-13T17:08:12.901634+0000","last_fullsized":"2024-09-13T17:08:12.901634+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:24.611621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":10,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.567674+0000","last_change":"2024-09-13T17:08:11.597049+0000","last_active":"2024-09-13T17:08:12.567674+0000","last_peered":"2024-09-13T17:08:12.567674+0000","last_clean":"2024-09-13T17:08:12.567674+0000","last_became_active":"2024-09-13T17:08:11.594306+0000","last_became_peered":"2024-09-13T17:08:11.594306+0000","last_unstale":"2024-09-13T17:08:12.567674+0000","last_undegraded":"2024-09-13T17:08:12.567674+0000","last_fullsized":"2024-09-13T17:08:12.567674+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:39.895672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.901675+0000","last_change":"2024-09-13T17:08:11.152688+0000","last_active":"2024-09-13T17:08:12.901675+0000","last_peered":"2024-09-13T17:08:12.901675+0000","last_clean":"2024-09-13T17:08:12.901675+0000","last_became_active":"2024-09-13T17:08:11.152286+0000","last_became_peered":"2024-09-13T17:08:11.152286+0000","last_unstale":"2024-09-13T17:08:12.901675+0000","last_undegraded":"2024-09-13T17:08:12.901675+0000","last_fullsized":"2024-09-13T17:08:12.901675+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:35.258547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905575+0000","last_change":"2024-09-13T17:08:11.588908+0000","last_active":"2024-09-13T17:08:12.905575+0000","last_peered":"2024-09-13T17:08:12.905575+0000","last_clean":"2024-09-13T17:08:12.905575+0000","last_became_active":"2024-09-13T17:08:11.588299+0000","last_became_peered":"2024-09-13T17:08:11.588299+0000","last_unstale":"2024-09-13T17:08:12.905575+0000","last_undegraded":"2024-09-13T17:08:12.905575+0000","last_fullsized":"2024-09-13T17:08:12.905575+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:31.674714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904748+0000","last_change":"2024-09-13T17:08:11.140760+0000","last_active":"2024-09-13T17:08:12.904748+0000","last_peered":"2024-09-13T17:08:12.904748+0000","last_clean":"2024-09-13T17:08:12.904748+0000","last_became_active":"2024-09-13T17:08:11.140489+0000","last_became_peered":"2024-09-13T17:08:11.140489+0000","last_unstale":"2024-09-13T17:08:12.904748+0000","last_undegraded":"2024-09-13T17:08:12.904748+0000","last_fullsized":"2024-09-13T17:08:12.904748+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.904739+0000","last_change":"2024-09-13T17:08:11.591976+0000","last_active":"2024-09-13T17:08:12.904739+0000","last_peered":"2024-09-13T17:08:12.904739+0000","last_clean":"2024-09-13T17:08:12.904739+0000","last_became_active":"2024-09-13T17:08:11.591484+0000","last_became_peered":"2024-09-13T17:08:11.591484+0000","last_unstale":"2024-09-13T17:08:12.904739+0000","last_undegraded":"2024-09-13T17:08:12.904739+0000","last_fullsized":"2024-09-13T17:08:12.904739+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:11.105795+0000","last_clean_scrub_stamp":"2024-09-13T17:08:11.105795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:18.578644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-13T17:08:12.905593+0000","last_change":"2024-09-13T17:08:11.145298+0000","last_active":"2024-09-13T17:08:12.905593+0000","last_peered":"2024-09-13T17:08:12.905593+0000","last_clean":"2024-09-13T17:08:12.905593+0000","last_became_active":"2024-09-13T17:08:11.145029+0000","last_became_peered":"2024-09-13T17:08:11.145029+0000","last_unstale":"2024-09-13T17:08:12.905593+0000","last_undegraded":"2024-09-13T17:08:12.905593+0000","last_fullsized":"2024-09-13T17:08:12.905593+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:08:10.099300+0000","last_clean_scrub_stamp":"2024-09-13T17:08:10.099300+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:27.460994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.87}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31336,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340504,"statfs":{"total":96636764160,"available":96604676096,"internally_reserved":0,"allocated":217088,"data_stored":72462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8610000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2070000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.401}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.899}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31320,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340520,"statfs":{"total":96636764160,"available":96604692480,"internally_reserved":0,"allocated":200704,"data_stored":69879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31332,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340508,"statfs":{"total":96636764160,"available":96604680192,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83299999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31908,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339932,"statfs":{"total":96636764160,"available":96604090368,"internally_reserved":0,"allocated":806912,"data_stored":660885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:08:21.238 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T17:08:21.238 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T17:08:21.238 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T17:08:21.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T17:08:21.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.427+0000 7f031a800640 1 Processor -- start 2024-09-13T17:08:21.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.427+0000 7f031a800640 1 -- start start 2024-09-13T17:08:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.428+0000 7f031a800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f031406f970 0x7f031415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f031a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0314159d90 con 0x7f031406c2d0 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f031a800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03140b2040 con 0x7f03141592a0 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f031a800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031415b0b0 con 0x7f031406f970 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f0313e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f031406c2d0 0x7f03140b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34596/0 (socket says 172.21.15.28:34596) 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f0313e00640 1 -- 172.21.15.28:0/236632540 learned_addr learned my addr 172.21.15.28:0/236632540 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f030aa00640 1 --2- 172.21.15.28:0/236632540 >> v2:172.21.15.189:3300/0 conn(0x7f031406f970 0x7f031415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f0312a00640 1 -- 172.21.15.28:0/236632540 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984903062 0 0) 0x7f0314159d90 con 0x7f031406c2d0 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f0312a00640 1 -- 172.21.15.28:0/236632540 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f8005730 con 0x7f031406c2d0 2024-09-13T17:08:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f030aa00640 1 -- 172.21.15.28:0/236632540 >> v1:172.21.15.189:6789/0 conn(0x7f03141592a0 legacy=0x7f0314159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f030aa00640 1 -- 172.21.15.28:0/236632540 >> v1:172.21.15.28:6789/0 conn(0x7f031406c2d0 legacy=0x7f03140b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.429+0000 7f030aa00640 1 -- 172.21.15.28:0/236632540 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314159ff0 con 0x7f031406f970 2024-09-13T17:08:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.430+0000 7f030aa00640 1 --2- 172.21.15.28:0/236632540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f031406f970 0x7f031415ab70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0097a0 tx=0x7f02fc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=448eed1474a35259 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f0312a00640 1 -- 172.21.15.28:0/236632540 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc032440 con 0x7f031406f970 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f0312a00640 1 -- 172.21.15.28:0/236632540 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc0325a0 con 0x7f031406f970 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f0312a00640 1 -- 172.21.15.28:0/236632540 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc032910 con 0x7f031406f970 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f031a800640 1 -- 172.21.15.28:0/236632540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f031406f970 msgr2=0x7f031415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f031a800640 1 --2- 172.21.15.28:0/236632540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f031406f970 0x7f031415ab70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0097a0 tx=0x7f02fc02a230 comp rx=0 tx=0).stop 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f031a800640 1 -- 172.21.15.28:0/236632540 shutdown_connections 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f031a800640 1 --2- 172.21.15.28:0/236632540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f031406f970 0x7f031415ab70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.431+0000 7f031a800640 1 -- 172.21.15.28:0/236632540 >> 172.21.15.28:0/236632540 conn(0x7f0314069e80 msgr2=0x7f031406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:21.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.432+0000 7f031a800640 1 -- 172.21.15.28:0/236632540 shutdown_connections 2024-09-13T17:08:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.432+0000 7f031a800640 1 -- 172.21.15.28:0/236632540 wait complete. 2024-09-13T17:08:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.433+0000 7f031a800640 1 Processor -- start 2024-09-13T17:08:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.433+0000 7f031a800640 1 -- start start 2024-09-13T17:08:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 0x7f031414afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141592a0 0x7f031414dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031414ce90 con 0x7f031406c2d0 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03140b0e30 con 0x7f03141592a0 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f031a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03140b0b50 con 0x7f031406f970 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f0313400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f0313400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33758/0 (socket says 172.21.15.28:33758) 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.434+0000 7f0313400640 1 -- 172.21.15.28:0/1826440171 learned_addr learned my addr 172.21.15.28:0/1826440171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313e00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141592a0 0x7f031414dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f030aa00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 0x7f031414afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 -- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 msgr2=0x7f031414afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 0x7f031414afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 -- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141592a0 msgr2=0x7f031414dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141592a0 0x7f031414dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 -- 172.21.15.28:0/1826440171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc009400 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f030aa00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 0x7f031414afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f0313400640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f030400c400 tx=0x7f030400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03040052d0 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304005430 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304003830 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.435+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031414d0c0 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.436+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031414d680 con 0x7f031406c2d0 2024-09-13T17:08:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.436+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031415ab70 con 0x7f031406c2d0 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.437+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0304003990 con 0x7f031406c2d0 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.438+0000 7f030be00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 0x7f02f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.438+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0304088980 con 0x7f031406c2d0 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.439+0000 7f030aa00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 0x7f02f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.439+0000 7f030aa00640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 0x7f02f006d0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02fc034000 tx=0x7f02fc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.443+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0304053af0 con 0x7f031406c2d0 2024-09-13T17:08:21.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.661+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03141570a0 con 0x7f031406c2d0 2024-09-13T17:08:21.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.662+0000 7f030be00640 1 -- 172.21.15.28:0/1826440171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0304057110 con 0x7f031406c2d0 2024-09-13T17:08:21.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:21.663 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T17:08:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 msgr2=0x7f02f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 0x7f02f006d0c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02fc034000 tx=0x7f02fc02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:21.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 msgr2=0x7f03140b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f030400c400 tx=0x7f030400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 shutdown_connections 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f02f006ac00 0x7f02f006d0c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141592a0 0x7f031414dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f031406f970 0x7f031414afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 --2- 172.21.15.28:0/1826440171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f031406c2d0 0x7f03140b0320 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 >> 172.21.15.28:0/1826440171 conn(0x7f0314069e80 msgr2=0x7f03140ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 shutdown_connections 2024-09-13T17:08:21.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.664+0000 7f031a800640 1 -- 172.21.15.28:0/1826440171 wait complete. 2024-09-13T17:08:21.680 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T17:08:21.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.835+0000 7fd2d6800640 1 Processor -- start 2024-09-13T17:08:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.835+0000 7fd2d6800640 1 -- start start 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d015c870 0x7fd2d00d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d6800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2d00b4930 0x7fd2d00b8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d6800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d00b8750 con 0x7fd2d00d0b00 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d6800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d00d0f70 con 0x7fd2d015c870 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d6800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d00d1190 con 0x7fd2d00b4930 2024-09-13T17:08:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d015c870 0x7fd2d00d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2d015c870 0x7fd2d00d2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33760/0 (socket says 172.21.15.28:33760) 2024-09-13T17:08:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d5400640 1 -- 172.21.15.28:0/1044326354 learned_addr learned my addr 172.21.15.28:0/1044326354 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:21.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.836+0000 7fd2d5e00640 1 --2- 172.21.15.28:0/1044326354 >> v2:172.21.15.189:3300/0 conn(0x7fd2d00b4930 0x7fd2d00b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.837+0000 7fd2d5400640 1 -- 172.21.15.28:0/1044326354 >> v2:172.21.15.189:3300/0 conn(0x7fd2d00b4930 msgr2=0x7fd2d00b8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.837+0000 7fd2d5400640 1 --2- 172.21.15.28:0/1044326354 >> v2:172.21.15.189:3300/0 conn(0x7fd2d00b4930 0x7fd2d00b8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.837+0000 7fd2d5400640 1 -- 172.21.15.28:0/1044326354 >> v1:172.21.15.189:6789/0 conn(0x7fd2d00d0b00 legacy=0x7fd2d00b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.837+0000 7fd2d5400640 1 -- 172.21.15.28:0/1044326354 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d00d1470 con 0x7fd2d015c870 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2d5400640 1 --2- 172.21.15.28:0/1044326354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00d2230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00d0470 tx=0x7fd2c402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fd3f3c1a90224f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2cfe00640 1 -- 172.21.15.28:0/1044326354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4032640 con 0x7fd2d015c870 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2cfe00640 1 -- 172.21.15.28:0/1044326354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c40327a0 con 0x7fd2d015c870 2024-09-13T17:08:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2cfe00640 1 -- 172.21.15.28:0/1044326354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4032b30 con 0x7fd2d015c870 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2d6800640 1 -- 172.21.15.28:0/1044326354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 msgr2=0x7fd2d00d2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.838+0000 7fd2d6800640 1 --2- 172.21.15.28:0/1044326354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00d2230 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00d0470 tx=0x7fd2c402a300 comp rx=0 tx=0).stop 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 -- 172.21.15.28:0/1044326354 shutdown_connections 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 --2- 172.21.15.28:0/1044326354 >> v2:172.21.15.189:3300/0 conn(0x7fd2d00b4930 0x7fd2d00b8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 --2- 172.21.15.28:0/1044326354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00d2230 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 -- 172.21.15.28:0/1044326354 >> 172.21.15.28:0/1044326354 conn(0x7fd2d00cf400 msgr2=0x7fd2d00cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 -- 172.21.15.28:0/1044326354 shutdown_connections 2024-09-13T17:08:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.839+0000 7fd2d6800640 1 -- 172.21.15.28:0/1044326354 wait complete. 2024-09-13T17:08:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.840+0000 7fd2d6800640 1 Processor -- start 2024-09-13T17:08:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 -- start start 2024-09-13T17:08:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2d00d0b00 0x7fd2d00c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00c7000 con 0x7fd2d015c870 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d00ccbc0 con 0x7fd2d00b4930 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.841+0000 7fd2d6800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d015ce60 con 0x7fd2d00d0b00 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2d00d0b00 0x7fd2d00c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:45108/0 (socket says 172.21.15.28:45108) 2024-09-13T17:08:21.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 -- 172.21.15.28:0/934861929 learned_addr learned my addr 172.21.15.28:0/934861929 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 -- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2d00d0b00 msgr2=0x7fd2d00c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2d00d0b00 0x7fd2d00c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 -- 172.21.15.28:0/934861929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 msgr2=0x7fd2d00c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.842+0000 7fd2d5400640 1 -- 172.21.15.28:0/934861929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40094d0 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2d5400640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c402a340 tx=0x7fd2c40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c403d830 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c403d990 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c403dd40 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d00c71f0 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.843+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d015d0b0 con 0x7fd2d00b4930 2024-09-13T17:08:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.844+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d00b8210 con 0x7fd2d00b4930 2024-09-13T17:08:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.845+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd2c4039040 con 0x7fd2d00b4930 2024-09-13T17:08:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.846+0000 7fd2cd600640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 0x7fd2b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.846+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2c40a9d80 con 0x7fd2d00b4930 2024-09-13T17:08:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.846+0000 7fd2d4a00640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 0x7fd2b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.847+0000 7fd2d4a00640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 0x7fd2b006d0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00b84d0 tx=0x7fd2bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:21.851+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd2c40ad030 con 0x7fd2d00b4930 2024-09-13T17:08:22.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.031+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d015bf00 con 0x7fd2d00b4930 2024-09-13T17:08:22.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.033+0000 7fd2cd600640 1 -- 172.21.15.28:0/934861929 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fd2c4074ff0 con 0x7fd2d00b4930 2024-09-13T17:08:22.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:22.034 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 msgr2=0x7fd2b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 0x7fd2b006d0c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00b84d0 tx=0x7fd2bc009040 comp rx=0 tx=0).stop 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 msgr2=0x7fd2d014ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd2c402a340 tx=0x7fd2c40046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 shutdown_connections 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2b006ac00 0x7fd2b006d0c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2d015c870 0x7fd2d00c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2d00d0b00 0x7fd2d00c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 --2- 172.21.15.28:0/934861929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2d00b4930 0x7fd2d014ff10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 >> 172.21.15.28:0/934861929 conn(0x7fd2d00cf400 msgr2=0x7fd2d00715e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 shutdown_connections 2024-09-13T17:08:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.035+0000 7fd2d6800640 1 -- 172.21.15.28:0/934861929 wait complete. 2024-09-13T17:08:22.052 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:22.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.208+0000 7f2373200640 1 Processor -- start 2024-09-13T17:08:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.209+0000 7f2373200640 1 -- start start 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2373200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 0x7f2374069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2373200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f237406a3c0 0x7f23740b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2373200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f237406a810 con 0x7f237415c890 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2373200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23740cdf80 con 0x7f237406a3c0 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2373200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f237415cc70 con 0x7f23740d0370 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2371e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f237415c890 0x7f23740cd870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34612/0 (socket says 172.21.15.28:34612) 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2371e00640 1 -- 172.21.15.28:0/980108890 learned_addr learned my addr 172.21.15.28:0/980108890 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2369400640 1 --2- 172.21.15.28:0/980108890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 0x7f2374069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.210+0000 7f2372800640 1 --2- 172.21.15.28:0/980108890 >> v2:172.21.15.189:3300/0 conn(0x7f237406a3c0 0x7f23740b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667031449 0 0) 0x7f2364001110 con 0x7f237415c890 2024-09-13T17:08:22.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2358003520 con 0x7f237415c890 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2369400640 1 -- 172.21.15.28:0/980108890 >> v2:172.21.15.189:3300/0 conn(0x7f237406a3c0 msgr2=0x7f23740b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2369400640 1 --2- 172.21.15.28:0/980108890 >> v2:172.21.15.189:3300/0 conn(0x7f237406a3c0 0x7f23740b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2369400640 1 -- 172.21.15.28:0/980108890 >> v1:172.21.15.28:6789/0 conn(0x7f237415c890 legacy=0x7f23740cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520848877 0 0) 0x7f2358003520 con 0x7f237415c890 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.211+0000 7f2369400640 1 -- 172.21.15.28:0/980108890 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23740d08e0 con 0x7f23740d0370 2024-09-13T17:08:22.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.212+0000 7f2369400640 1 --2- 172.21.15.28:0/980108890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 0x7f2374069e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097a0 tx=0x7f235c02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=172d7fb47b80a11a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.212+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364001800 con 0x7f23740d0370 2024-09-13T17:08:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.212+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c02ac30 con 0x7f23740d0370 2024-09-13T17:08:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.212+0000 7f2371400640 1 -- 172.21.15.28:0/980108890 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c032440 con 0x7f23740d0370 2024-09-13T17:08:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 -- 172.21.15.28:0/980108890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 msgr2=0x7f2374069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 --2- 172.21.15.28:0/980108890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 0x7f2374069e80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f235c0097a0 tx=0x7f235c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 -- 172.21.15.28:0/980108890 shutdown_connections 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 --2- 172.21.15.28:0/980108890 >> v2:172.21.15.189:3300/0 conn(0x7f237406a3c0 0x7f23740b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 --2- 172.21.15.28:0/980108890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23740d0370 0x7f2374069e80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 -- 172.21.15.28:0/980108890 >> 172.21.15.28:0/980108890 conn(0x7f23740708c0 msgr2=0x7f2374070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 -- 172.21.15.28:0/980108890 shutdown_connections 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.213+0000 7f2373200640 1 -- 172.21.15.28:0/980108890 wait complete. 2024-09-13T17:08:22.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.215+0000 7f2373200640 1 Processor -- start 2024-09-13T17:08:22.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.215+0000 7f2373200640 1 -- start start 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f237406a3c0 0x7f23741f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 0x7f2374143720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23740c8bb0 con 0x7f23740d0370 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23741445a0 con 0x7f237406a3c0 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2373200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23741f2fa0 con 0x7f237415c890 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33792/0 (socket says 172.21.15.28:33792) 2024-09-13T17:08:22.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 -- 172.21.15.28:0/2225205097 learned_addr learned my addr 172.21.15.28:0/2225205097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2371e00640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f237406a3c0 0x7f23741f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2372800640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 0x7f2374143720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 -- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 msgr2=0x7f2374143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 0x7f2374143720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 -- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f237406a3c0 msgr2=0x7f23741f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f237406a3c0 0x7f23741f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2369400640 1 -- 172.21.15.28:0/2225205097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c009400 con 0x7f23740d0370 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.216+0000 7f2372800640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 0x7f2374143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f2369400640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f235c02a790 tx=0x7f235c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c03c820 con 0x7f23740d0370 2024-09-13T17:08:22.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235c03c980 con 0x7f23740d0370 2024-09-13T17:08:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235c03ccf0 con 0x7f23740d0370 2024-09-13T17:08:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23740c7f30 con 0x7f23740d0370 2024-09-13T17:08:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.217+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23740c84f0 con 0x7f23740d0370 2024-09-13T17:08:22.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.218+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23740c8cf0 con 0x7f23740d0370 2024-09-13T17:08:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.219+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f235c039050 con 0x7f23740d0370 2024-09-13T17:08:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.220+0000 7f236aa00640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 0x7f235006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.220+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f235c0a9ac0 con 0x7f23740d0370 2024-09-13T17:08:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.220+0000 7f2371e00640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 0x7f235006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.221+0000 7f2371e00640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 0x7f235006d0f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2364000f60 tx=0x7f2364001110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.224+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f23740c8cf0 con 0x7f23740d0370 2024-09-13T17:08:22.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.401+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f23740c8cf0 con 0x7f23740d0370 2024-09-13T17:08:22.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.402+0000 7f236aa00640 1 -- 172.21.15.28:0/2225205097 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f23740c8cf0 con 0x7f23740d0370 2024-09-13T17:08:22.403 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.403+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 msgr2=0x7f235006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.403+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 0x7f235006d0f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2364000f60 tx=0x7f2364001110 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.403+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 msgr2=0x7f23740cae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.403+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f235c02a790 tx=0x7f235c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 shutdown_connections 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f235006ac30 0x7f235006d0f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f237415c890 0x7f2374143720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23740d0370 0x7f23740cae40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 --2- 172.21.15.28:0/2225205097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f237406a3c0 0x7f23741f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 >> 172.21.15.28:0/2225205097 conn(0x7f23740708c0 msgr2=0x7f2374158320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 shutdown_connections 2024-09-13T17:08:22.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.404+0000 7f2373200640 1 -- 172.21.15.28:0/2225205097 wait complete. 2024-09-13T17:08:22.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:22.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.577+0000 7fa632e00640 1 Processor -- start 2024-09-13T17:08:22.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.577+0000 7fa632e00640 1 -- start start 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa632e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 0x7fa62c0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa632e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c1610a0 con 0x7fa62c0b17e0 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa632e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62c0b2620 con 0x7fa62c06d070 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa632e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0b2900 con 0x7fa62c06c540 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa62be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa62c0b17e0 0x7fa62c0b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34626/0 (socket says 172.21.15.28:34626) 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa62be00640 1 -- 172.21.15.28:0/3218767834 learned_addr learned my addr 172.21.15.28:0/3218767834 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa630e00640 1 --2- 172.21.15.28:0/3218767834 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 0x7fa62c0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897021040 0 0) 0x7fa62c1610a0 con 0x7fa62c0b17e0 2024-09-13T17:08:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.579+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa610003520 con 0x7fa62c0b17e0 2024-09-13T17:08:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.580+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1834706401 0 0) 0x7fa62c0b2620 con 0x7fa62c06d070 2024-09-13T17:08:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.580+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa62c1610a0 con 0x7fa62c06d070 2024-09-13T17:08:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.580+0000 7fa630e00640 1 -- 172.21.15.28:0/3218767834 >> v1:172.21.15.189:6789/0 conn(0x7fa62c06d070 legacy=0x7fa62c1687d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.580+0000 7fa630e00640 1 -- 172.21.15.28:0/3218767834 >> v1:172.21.15.28:6789/0 conn(0x7fa62c0b17e0 legacy=0x7fa62c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.580+0000 7fa630e00640 1 -- 172.21.15.28:0/3218767834 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c0b2be0 con 0x7fa62c06c540 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.581+0000 7fa630e00640 1 --2- 172.21.15.28:0/3218767834 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 0x7fa62c0b3480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa624009dc0 tx=0x7fa62402a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd3bf2c4e091cb3e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.581+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624007740 con 0x7fa62c06c540 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.581+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa62402d050 con 0x7fa62c06c540 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.581+0000 7fa62b400640 1 -- 172.21.15.28:0/3218767834 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624007ac0 con 0x7fa62c06c540 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 -- 172.21.15.28:0/3218767834 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 msgr2=0x7fa62c0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 --2- 172.21.15.28:0/3218767834 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 0x7fa62c0b3480 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa624009dc0 tx=0x7fa62402a6d0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 -- 172.21.15.28:0/3218767834 shutdown_connections 2024-09-13T17:08:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 --2- 172.21.15.28:0/3218767834 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c06c540 0x7fa62c0b3480 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 -- 172.21.15.28:0/3218767834 >> 172.21.15.28:0/3218767834 conn(0x7fa62c0b0270 msgr2=0x7fa62c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 -- 172.21.15.28:0/3218767834 shutdown_connections 2024-09-13T17:08:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.582+0000 7fa632e00640 1 -- 172.21.15.28:0/3218767834 wait complete. 2024-09-13T17:08:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.583+0000 7fa632e00640 1 Processor -- start 2024-09-13T17:08:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 -- start start 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c06d070 0x7fa62c0a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c0b17e0 0x7fa62c0ac650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0ac230 con 0x7fa62c06c540 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c0abbb0 con 0x7fa62c06d070 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.584+0000 7fa632e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c1581c0 con 0x7fa62c0b17e0 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33808/0 (socket says 172.21.15.28:33808) 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 -- 172.21.15.28:0/2128890674 learned_addr learned my addr 172.21.15.28:0/2128890674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa623400640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c06d070 0x7fa62c0a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa630e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c0b17e0 0x7fa62c0ac650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 -- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c0b17e0 msgr2=0x7fa62c0ac650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c0b17e0 0x7fa62c0ac650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 -- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c06d070 msgr2=0x7fa62c0a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c06d070 0x7fa62c0a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 -- 172.21.15.28:0/2128890674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa624009a70 con 0x7fa62c06c540 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.585+0000 7fa62be00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c500 tx=0x7fa61c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.586+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c005ce0 con 0x7fa62c06c540 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.586+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61c005e40 con 0x7fa62c06c540 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.586+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c0072f0 con 0x7fa62c06c540 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.586+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c0abe50 con 0x7fa62c06c540 2024-09-13T17:08:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.586+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c0cdf60 con 0x7fa62c06c540 2024-09-13T17:08:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.587+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c0b3480 con 0x7fa62c06c540 2024-09-13T17:08:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.588+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa61c007450 con 0x7fa62c06c540 2024-09-13T17:08:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.589+0000 7fa628c00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 0x7fa60806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.589+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa61c0888c0 con 0x7fa62c06c540 2024-09-13T17:08:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.589+0000 7fa623400640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 0x7fa60806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.590+0000 7fa623400640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 0x7fa60806d0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa62c1ed470 tx=0x7fa614009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.593+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa61c053b40 con 0x7fa62c06c540 2024-09-13T17:08:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.770+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa62c0aedd0 con 0x7fa62c06c540 2024-09-13T17:08:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.770+0000 7fa628c00640 1 -- 172.21.15.28:0/2128890674 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa61c0170a0 con 0x7fa62c06c540 2024-09-13T17:08:22.772 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 msgr2=0x7fa60806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 0x7fa60806d0c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa62c1ed470 tx=0x7fa614009040 comp rx=0 tx=0).stop 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 msgr2=0x7fa62c0cda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa61c00c500 tx=0x7fa61c00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 shutdown_connections 2024-09-13T17:08:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa60806ac00 0x7fa60806d0c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa62c0b17e0 0x7fa62c0ac650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa62c06d070 0x7fa62c0a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 --2- 172.21.15.28:0/2128890674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa62c06c540 0x7fa62c0cda20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 >> 172.21.15.28:0/2128890674 conn(0x7fa62c0b0270 msgr2=0x7fa62c15ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 shutdown_connections 2024-09-13T17:08:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.772+0000 7fa632e00640 1 -- 172.21.15.28:0/2128890674 wait complete. 2024-09-13T17:08:22.789 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:22.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.945+0000 7f472d000640 1 Processor -- start 2024-09-13T17:08:22.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.945+0000 7f472d000640 1 -- start start 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.946+0000 7f472d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4728159770 0x7f472806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.946+0000 7f472d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 0x7f4728070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.946+0000 7f472d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472806f920 con 0x7f472806f3f0 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.946+0000 7f472d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f472815b290 con 0x7f4728159770 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.946+0000 7f472d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472815b570 con 0x7f472815cd80 2024-09-13T17:08:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4728159770 0x7f472806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4728159770 0x7f472806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33822/0 (socket says 172.21.15.28:33822) 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 -- 172.21.15.28:0/3570389111 learned_addr learned my addr 172.21.15.28:0/3570389111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 -- 172.21.15.28:0/3570389111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 msgr2=0x7f4728070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4726800640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 0x7f4728070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 0x7f4728070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 -- 172.21.15.28:0/3570389111 >> v1:172.21.15.189:6789/0 conn(0x7f472806f3f0 legacy=0x7f4728159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4725e00640 1 -- 172.21.15.28:0/3570389111 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472815b850 con 0x7f4728159770 2024-09-13T17:08:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.947+0000 7f4726800640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 0x7f4728070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.948+0000 7f4725e00640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728159770 0x7f472806fd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4718009870 tx=0x7f471802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bc372044a19ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.948+0000 7f4725400640 1 -- 172.21.15.28:0/3570389111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471802ae90 con 0x7f4728159770 2024-09-13T17:08:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.948+0000 7f4725400640 1 -- 172.21.15.28:0/3570389111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4718032440 con 0x7f4728159770 2024-09-13T17:08:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.948+0000 7f4725400640 1 -- 172.21.15.28:0/3570389111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47180327d0 con 0x7f4728159770 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 -- 172.21.15.28:0/3570389111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728159770 msgr2=0x7f472806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728159770 0x7f472806fd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4718009870 tx=0x7f471802a180 comp rx=0 tx=0).stop 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 -- 172.21.15.28:0/3570389111 shutdown_connections 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f472815cd80 0x7f4728070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 --2- 172.21.15.28:0/3570389111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4728159770 0x7f472806fd20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 -- 172.21.15.28:0/3570389111 >> 172.21.15.28:0/3570389111 conn(0x7f47280b0270 msgr2=0x7f47280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 -- 172.21.15.28:0/3570389111 shutdown_connections 2024-09-13T17:08:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.949+0000 7f472d000640 1 -- 172.21.15.28:0/3570389111 wait complete. 2024-09-13T17:08:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.951+0000 7f472d000640 1 Processor -- start 2024-09-13T17:08:22.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.951+0000 7f472d000640 1 -- start start 2024-09-13T17:08:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 0x7f4728154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f472815cd80 0x7f472813fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728156180 con 0x7f472806f3f0 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728155570 con 0x7f472815cd80 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f472d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4728155850 con 0x7f4728159770 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33824/0 (socket says 172.21.15.28:33824) 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 -- 172.21.15.28:0/3342283351 learned_addr learned my addr 172.21.15.28:0/3342283351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4726800640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f472815cd80 0x7f472813fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f471d400640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 0x7f4728154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 -- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 msgr2=0x7f4728154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 0x7f4728154d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 -- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f472815cd80 msgr2=0x7f472813fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f472815cd80 0x7f472813fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f4725e00640 1 -- 172.21.15.28:0/3342283351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47180094d0 con 0x7f472806f3f0 2024-09-13T17:08:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.952+0000 7f471d400640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 0x7f4728154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f4725e00640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4718009870 tx=0x7f47180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4718030b80 con 0x7f472806f3f0 2024-09-13T17:08:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4718030ce0 con 0x7f472806f3f0 2024-09-13T17:08:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4728155b30 con 0x7f472806f3f0 2024-09-13T17:08:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471803d660 con 0x7f472806f3f0 2024-09-13T17:08:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.953+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4728155260 con 0x7f472806f3f0 2024-09-13T17:08:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.954+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472806fd20 con 0x7f472806f3f0 2024-09-13T17:08:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.955+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4718039040 con 0x7f472806f3f0 2024-09-13T17:08:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.956+0000 7f471ea00640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 0x7f470406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.956+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47180aa110 con 0x7f472806f3f0 2024-09-13T17:08:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.956+0000 7f471d400640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 0x7f470406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:22.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.957+0000 7f471d400640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 0x7f470406d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4728143940 tx=0x7f4710009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:22.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:22.960+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4718075270 con 0x7f472806f3f0 2024-09-13T17:08:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.138+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f47280ae820 con 0x7f472806f3f0 2024-09-13T17:08:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.139+0000 7f471ea00640 1 -- 172.21.15.28:0/3342283351 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4718078890 con 0x7f472806f3f0 2024-09-13T17:08:23.140 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 msgr2=0x7f470406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 0x7f470406d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4728143940 tx=0x7f4710009040 comp rx=0 tx=0).stop 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 msgr2=0x7f4728166e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4718009870 tx=0x7f47180046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 shutdown_connections 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f470406ac00 0x7f470406d0c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f472815cd80 0x7f472813fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4728159770 0x7f4728154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 --2- 172.21.15.28:0/3342283351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f472806f3f0 0x7f4728166e90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 >> 172.21.15.28:0/3342283351 conn(0x7f47280b0270 msgr2=0x7f47280cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 shutdown_connections 2024-09-13T17:08:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.141+0000 7f472d000640 1 -- 172.21.15.28:0/3342283351 wait complete. 2024-09-13T17:08:23.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.311+0000 7f62b7200640 1 Processor -- start 2024-09-13T17:08:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.311+0000 7f62b7200640 1 -- start start 2024-09-13T17:08:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 0x7f62b80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b8159820 0x7f62b80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62b80c3850 con 0x7f62b80b22e0 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62b815a930 con 0x7f62b8159820 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b8159d20 con 0x7f62b8159440 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b8159820 0x7f62b80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f62b8159820 0x7f62b80b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33836/0 (socket says 172.21.15.28:33836) 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62b80b22e0 0x7f62b80b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34638/0 (socket says 172.21.15.28:34638) 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 -- 172.21.15.28:0/3534476996 learned_addr learned my addr 172.21.15.28:0/3534476996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62b5e00640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 0x7f62b80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 -- 172.21.15.28:0/3534476996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 msgr2=0x7f62b80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 0x7f62b80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.313+0000 7f62afe00640 1 -- 172.21.15.28:0/3534476996 >> v1:172.21.15.28:6789/0 conn(0x7f62b80b22e0 legacy=0x7f62b80b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.314+0000 7f62afe00640 1 -- 172.21.15.28:0/3534476996 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b815a000 con 0x7f62b8159820 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.314+0000 7f62b5400640 1 -- 172.21.15.28:0/3534476996 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038188664 0 0) 0x7f62b80c3850 con 0x7f62b80b22e0 2024-09-13T17:08:23.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.314+0000 7f62b5e00640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 0x7f62b80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62afe00640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b8159820 0x7f62b80b0980 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009870 tx=0x7f62a002a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2391d662e8dd07c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b5400640 1 -- 172.21.15.28:0/3534476996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0032640 con 0x7f62b8159820 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b5400640 1 -- 172.21.15.28:0/3534476996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a00327a0 con 0x7f62b8159820 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b5400640 1 -- 172.21.15.28:0/3534476996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0032b30 con 0x7f62b8159820 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 -- 172.21.15.28:0/3534476996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b8159820 msgr2=0x7f62b80b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b8159820 0x7f62b80b0980 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f62a0009870 tx=0x7f62a002a2e0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 -- 172.21.15.28:0/3534476996 shutdown_connections 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b8159820 0x7f62b80b0980 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 --2- 172.21.15.28:0/3534476996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159440 0x7f62b80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 -- 172.21.15.28:0/3534476996 >> 172.21.15.28:0/3534476996 conn(0x7f62b8072e40 msgr2=0x7f62b8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.315+0000 7f62b7200640 1 -- 172.21.15.28:0/3534476996 shutdown_connections 2024-09-13T17:08:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.316+0000 7f62b7200640 1 -- 172.21.15.28:0/3534476996 wait complete. 2024-09-13T17:08:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.317+0000 7f62b7200640 1 Processor -- start 2024-09-13T17:08:23.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.317+0000 7f62b7200640 1 -- start start 2024-09-13T17:08:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62b8159440 0x7f62b814e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159820 0x7f62b814ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b81f3630 con 0x7f62b80b22e0 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b81f4170 con 0x7f62b8159440 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b81f3840 con 0x7f62b8159820 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33842/0 (socket says 172.21.15.28:33842) 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b5e00640 1 -- 172.21.15.28:0/138031475 learned_addr learned my addr 172.21.15.28:0/138031475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62afe00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62b8159440 0x7f62b814e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.318+0000 7f62b6800640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159820 0x7f62b814ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 -- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159820 msgr2=0x7f62b814ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159820 0x7f62b814ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 -- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62b8159440 msgr2=0x7f62b814e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62b8159440 0x7f62b814e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 -- 172.21.15.28:0/138031475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a00094d0 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b5e00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c7f0 tx=0x7f62a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a8007a30 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62a8007b90 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a800ce40 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.319+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b81f3b20 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.320+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b814b560 con 0x7f62b80b22e0 2024-09-13T17:08:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.321+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b80b2790 con 0x7f62b80b22e0 2024-09-13T17:08:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.322+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f62a80113f0 con 0x7f62b80b22e0 2024-09-13T17:08:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.322+0000 7f62ae000640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 0x7f629406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.322+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62a80894a0 con 0x7f62b80b22e0 2024-09-13T17:08:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.323+0000 7f62afe00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 0x7f629406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.324+0000 7f62afe00640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 0x7f629406d0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62a0034000 tx=0x7f62a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.327+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f62a8054720 con 0x7f62b80b22e0 2024-09-13T17:08:23.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.539+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62b81556c0 con 0x7f62b80b22e0 2024-09-13T17:08:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.540+0000 7f62ae000640 1 -- 172.21.15.28:0/138031475 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f62a8057d40 con 0x7f62b80b22e0 2024-09-13T17:08:23.541 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:23.541 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:23.541 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 msgr2=0x7f629406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 0x7f629406d0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f62a0034000 tx=0x7f62a002e000 comp rx=0 tx=0).stop 2024-09-13T17:08:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 msgr2=0x7f62b814b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f62a800c7f0 tx=0x7f62a800cbf0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 shutdown_connections 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f629406ac00 0x7f629406d0c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62b8159820 0x7f62b814ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62b8159440 0x7f62b814e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 --2- 172.21.15.28:0/138031475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62b80b22e0 0x7f62b814b020 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 >> 172.21.15.28:0/138031475 conn(0x7f62b8072e40 msgr2=0x7f62b80b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 shutdown_connections 2024-09-13T17:08:23.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.542+0000 7f62b7200640 1 -- 172.21.15.28:0/138031475 wait complete. 2024-09-13T17:08:23.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:23.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.713+0000 7f8bbb200640 1 Processor -- start 2024-09-13T17:08:23.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.713+0000 7f8bbb200640 1 -- start start 2024-09-13T17:08:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bbb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bbc159440 0x7f8bbc0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bbb200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 0x7f8bbc0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bbb200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bbc0c3850 con 0x7f8bbc159820 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bbb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bbc15a930 con 0x7f8bbc159440 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bbb200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bbc159d20 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bbc159440 0x7f8bbc0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bbc159440 0x7f8bbc0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33848/0 (socket says 172.21.15.28:33848) 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3811476811 learned_addr learned my addr 172.21.15.28:0/3811476811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bba800640 1 --2- 172.21.15.28:0/3811476811 >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 0x7f8bbc0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3811476811 >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 msgr2=0x7f8bbc0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 --2- 172.21.15.28:0/3811476811 >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 0x7f8bbc0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3811476811 >> v1:172.21.15.189:6789/0 conn(0x7f8bbc159820 legacy=0x7f8bbc0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.715+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3811476811 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc15a000 con 0x7f8bbc159440 2024-09-13T17:08:23.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.716+0000 7f8bba800640 1 --2- 172.21.15.28:0/3811476811 >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 0x7f8bbc0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bb9e00640 1 --2- 172.21.15.28:0/3811476811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc159440 0x7f8bbc0b0440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009870 tx=0x7f8bac02a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa93e9cb029c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bb9400640 1 -- 172.21.15.28:0/3811476811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac032640 con 0x7f8bbc159440 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bb9400640 1 -- 172.21.15.28:0/3811476811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac0327a0 con 0x7f8bbc159440 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bb9400640 1 -- 172.21.15.28:0/3811476811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac032b30 con 0x7f8bbc159440 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 -- 172.21.15.28:0/3811476811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc159440 msgr2=0x7f8bbc0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3811476811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc159440 0x7f8bbc0b0440 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8bac009870 tx=0x7f8bac02a330 comp rx=0 tx=0).stop 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 -- 172.21.15.28:0/3811476811 shutdown_connections 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3811476811 >> v2:172.21.15.189:3300/0 conn(0x7f8bbc0b29f0 0x7f8bbc0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3811476811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc159440 0x7f8bbc0b0440 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.717+0000 7f8bbb200640 1 -- 172.21.15.28:0/3811476811 >> 172.21.15.28:0/3811476811 conn(0x7f8bbc072e40 msgr2=0x7f8bbc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:23.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.718+0000 7f8bbb200640 1 -- 172.21.15.28:0/3811476811 shutdown_connections 2024-09-13T17:08:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.718+0000 7f8bbb200640 1 -- 172.21.15.28:0/3811476811 wait complete. 2024-09-13T17:08:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.719+0000 7f8bbb200640 1 Processor -- start 2024-09-13T17:08:23.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.719+0000 7f8bbb200640 1 -- start start 2024-09-13T17:08:23.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 0x7f8bbc151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bbc159820 0x7f8bbc0ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc152160 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc156870 con 0x7f8bbc159440 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bbb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc0adde0 con 0x7f8bbc159820 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bb9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bb9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33860/0 (socket says 172.21.15.28:33860) 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.720+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3800423703 learned_addr learned my addr 172.21.15.28:0/3800423703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb1400640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 0x7f8bbc151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bba800640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bbc159820 0x7f8bbc0ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bbc159820 msgr2=0x7f8bbc0ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bbc159820 0x7f8bbc0ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 msgr2=0x7f8bbc151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 0x7f8bbc151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 -- 172.21.15.28:0/3800423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac0094d0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb1400640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 0x7f8bbc151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb9e00640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8bac000c00 tx=0x7f8bac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac030850 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bac0309b0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bac030d60 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.721+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bbc156b10 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.722+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bbc1570d0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.722+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bbc0b22e0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.723+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8bac039040 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.724+0000 7f8bb2a00640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 0x7f8b9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:23.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.724+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8bac02ae40 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.725+0000 7f8bb1400640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 0x7f8b9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:23.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.725+0000 7f8bb1400640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 0x7f8b9806d0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4007fd0 tx=0x7f8ba4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:23.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.728+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8bac075090 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.906+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bbc15b4b0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.907+0000 7f8bb2a00640 1 -- 172.21.15.28:0/3800423703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f8bac0786b0 con 0x7f8bbc0b29f0 2024-09-13T17:08:23.909 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:23.909 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.909+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 msgr2=0x7f8b9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.909+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 0x7f8b9806d0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4007fd0 tx=0x7f8ba4009040 comp rx=0 tx=0).stop 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.909+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 msgr2=0x7f8bbc0c4480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.909+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8bac000c00 tx=0x7f8bac0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 shutdown_connections 2024-09-13T17:08:23.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b9806ac30 0x7f8b9806d0f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bbc159820 0x7f8bbc0ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bbc159440 0x7f8bbc151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 --2- 172.21.15.28:0/3800423703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bbc0b29f0 0x7f8bbc0c4480 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 >> 172.21.15.28:0/3800423703 conn(0x7f8bbc072e40 msgr2=0x7f8bbc0ccc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 shutdown_connections 2024-09-13T17:08:23.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:23.910+0000 7f8bbb200640 1 -- 172.21.15.28:0/3800423703 wait complete. 2024-09-13T17:08:23.926 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:23.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:24.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.083+0000 7fbf06800640 1 Processor -- start 2024-09-13T17:08:24.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.084+0000 7fbf06800640 1 -- start start 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf00159770 0x7fbf0006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 0x7fbf00070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf000add00 0x7fbf00159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0006f920 con 0x7fbf000add00 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0015b290 con 0x7fbf0006f3f0 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbf06800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0015b570 con 0x7fbf00159770 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbeffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf000add00 0x7fbf00159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbeffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf000add00 0x7fbf00159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59634/0 (socket says 172.21.15.28:59634) 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbeffe00640 1 -- 172.21.15.28:0/445735321 learned_addr learned my addr 172.21.15.28:0/445735321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbeff400640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf00159770 0x7fbf0006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.085+0000 7fbef6a00640 1 --2- 172.21.15.28:0/445735321 >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 0x7fbf00070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbeffe00640 1 -- 172.21.15.28:0/445735321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf00159770 msgr2=0x7fbf0006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:24.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbeffe00640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf00159770 0x7fbf0006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbeffe00640 1 -- 172.21.15.28:0/445735321 >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 msgr2=0x7fbf00070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:24.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbeffe00640 1 --2- 172.21.15.28:0/445735321 >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 0x7fbf00070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbeffe00640 1 -- 172.21.15.28:0/445735321 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0015b850 con 0x7fbf000add00 2024-09-13T17:08:24.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.086+0000 7fbef6a00640 1 --2- 172.21.15.28:0/445735321 >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 0x7fbf00070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.087+0000 7fbeffe00640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf000add00 0x7fbf00159b50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbf000cf5c0 tx=0x7fbef802c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a225cf503927d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.087+0000 7fbefea00640 1 -- 172.21.15.28:0/445735321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8009600 con 0x7fbf000add00 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.087+0000 7fbefea00640 1 -- 172.21.15.28:0/445735321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef8030050 con 0x7fbf000add00 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.087+0000 7fbefea00640 1 -- 172.21.15.28:0/445735321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef80099a0 con 0x7fbf000add00 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 -- 172.21.15.28:0/445735321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf000add00 msgr2=0x7fbf00159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf000add00 0x7fbf00159b50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbf000cf5c0 tx=0x7fbef802c810 comp rx=0 tx=0).stop 2024-09-13T17:08:24.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 -- 172.21.15.28:0/445735321 shutdown_connections 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf000add00 0x7fbf00159b50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 --2- 172.21.15.28:0/445735321 >> v2:172.21.15.189:3300/0 conn(0x7fbf0006f3f0 0x7fbf00070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 --2- 172.21.15.28:0/445735321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf00159770 0x7fbf0006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 -- 172.21.15.28:0/445735321 >> 172.21.15.28:0/445735321 conn(0x7fbf000b0270 msgr2=0x7fbf000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 -- 172.21.15.28:0/445735321 shutdown_connections 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.088+0000 7fbf06800640 1 -- 172.21.15.28:0/445735321 wait complete. 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.089+0000 7fbf06800640 1 Processor -- start 2024-09-13T17:08:24.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.090+0000 7fbf06800640 1 -- start start 2024-09-13T17:08:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf0006f3f0 0x7fbf0014a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf000add00 0x7fbf000c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000ca090 con 0x7fbf00159770 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0014aac0 con 0x7fbf0006f3f0 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbf06800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf000c9410 con 0x7fbf000add00 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59636/0 (socket says 172.21.15.28:59636) 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 -- 172.21.15.28:0/2276085599 learned_addr learned my addr 172.21.15.28:0/2276085599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbef6a00640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf000add00 0x7fbf000c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeff400640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf0006f3f0 0x7fbf0014a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 -- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf000add00 msgr2=0x7fbf000c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf000add00 0x7fbf000c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 -- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf0006f3f0 msgr2=0x7fbf0014a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf0006f3f0 0x7fbf0014a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.091+0000 7fbeffe00640 1 -- 172.21.15.28:0/2276085599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef800bd40 con 0x7fbf00159770 2024-09-13T17:08:24.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbeffe00640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbef800d240 tx=0x7fbef8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8004800 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef800b720 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef8009840 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf000c96f0 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.092+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf000c9cb0 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.093+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf00159b50 con 0x7fbf00159770 2024-09-13T17:08:24.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.094+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbef803b070 con 0x7fbf00159770 2024-09-13T17:08:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.095+0000 7fbef7e00640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 0x7fbedc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.095+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbef80aa320 con 0x7fbf00159770 2024-09-13T17:08:24.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.096+0000 7fbeff400640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 0x7fbedc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.096+0000 7fbeff400640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 0x7fbedc06d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009a20 tx=0x7fbef0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.099+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbef8075640 con 0x7fbf00159770 2024-09-13T17:08:24.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.277+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf000b3140 con 0x7fbf00159770 2024-09-13T17:08:24.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.278+0000 7fbef7e00640 1 -- 172.21.15.28:0/2276085599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbef8078c60 con 0x7fbf00159770 2024-09-13T17:08:24.279 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 msgr2=0x7fbedc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 0x7fbedc06d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fbef0009a20 tx=0x7fbef0009190 comp rx=0 tx=0).stop 2024-09-13T17:08:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 msgr2=0x7fbf000c6ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbef800d240 tx=0x7fbef8004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 shutdown_connections 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbedc06ac00 0x7fbedc06d0c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf00159770 0x7fbf000c6ae0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf000add00 0x7fbf000c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 --2- 172.21.15.28:0/2276085599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf0006f3f0 0x7fbf0014a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 >> 172.21.15.28:0/2276085599 conn(0x7fbf000b0270 msgr2=0x7fbf000b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 shutdown_connections 2024-09-13T17:08:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.280+0000 7fbf06800640 1 -- 172.21.15.28:0/2276085599 wait complete. 2024-09-13T17:08:24.296 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:24.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.453+0000 7f41a1800640 1 Processor -- start 2024-09-13T17:08:24.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.453+0000 7f41a1800640 1 -- start start 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f41a1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f419c15ab60 0x7f419c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f41a1800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 0x7f419c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f41a1800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c06c8b0 con 0x7f419c15a780 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f41a1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f419c06cb90 con 0x7f419c15ab60 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f41a1800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0cc980 con 0x7f419c071490 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.454+0000 7f419a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f419c15ab60 0x7f419c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f419c15a780 0x7f419c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37614/0 (socket says 172.21.15.28:37614) 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f419c15ab60 0x7f419c06bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59638/0 (socket says 172.21.15.28:59638) 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419aa00640 1 -- 172.21.15.28:0/34018494 learned_addr learned my addr 172.21.15.28:0/34018494 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:24.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419b400640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 0x7f419c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419a000640 1 -- 172.21.15.28:0/34018494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 msgr2=0x7f419c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:24.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419a000640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 0x7f419c06c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419a000640 1 -- 172.21.15.28:0/34018494 >> v1:172.21.15.28:6789/0 conn(0x7f419c15a780 legacy=0x7f419c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419a000640 1 -- 172.21.15.28:0/34018494 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0ccc60 con 0x7f419c15ab60 2024-09-13T17:08:24.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.455+0000 7f419b400640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 0x7f419c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.456+0000 7f419a000640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15ab60 0x7f419c06bb40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f41880097a0 tx=0x7f418802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df59c4cb7fcb0c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.456+0000 7f4199600640 1 -- 172.21.15.28:0/34018494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188032440 con 0x7f419c15ab60 2024-09-13T17:08:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.456+0000 7f4199600640 1 -- 172.21.15.28:0/34018494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41880325a0 con 0x7f419c15ab60 2024-09-13T17:08:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.456+0000 7f4199600640 1 -- 172.21.15.28:0/34018494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188032930 con 0x7f419c15ab60 2024-09-13T17:08:24.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 -- 172.21.15.28:0/34018494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15ab60 msgr2=0x7f419c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15ab60 0x7f419c06bb40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f41880097a0 tx=0x7f418802a180 comp rx=0 tx=0).stop 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 -- 172.21.15.28:0/34018494 shutdown_connections 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c071490 0x7f419c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 --2- 172.21.15.28:0/34018494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15ab60 0x7f419c06bb40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 -- 172.21.15.28:0/34018494 >> 172.21.15.28:0/34018494 conn(0x7f419c0b0270 msgr2=0x7f419c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 -- 172.21.15.28:0/34018494 shutdown_connections 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.457+0000 7f41a1800640 1 -- 172.21.15.28:0/34018494 wait complete. 2024-09-13T17:08:24.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.458+0000 7f41a1800640 1 Processor -- start 2024-09-13T17:08:24.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.459+0000 7f41a1800640 1 -- start start 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.459+0000 7f41a1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 0x7f419c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.459+0000 7f41a1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.459+0000 7f41a1800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c15ab60 0x7f419c154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f41a1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c13cae0 con 0x7f419c15a780 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f41a1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0b2e90 con 0x7f419c071490 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f41a1800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c13cd80 con 0x7f419c15ab60 2024-09-13T17:08:24.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59646/0 (socket says 172.21.15.28:59646) 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 -- 172.21.15.28:0/2832556512 learned_addr learned my addr 172.21.15.28:0/2832556512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419aa00640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 0x7f419c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419b400640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c15ab60 0x7f419c154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 -- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c15ab60 msgr2=0x7f419c154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c15ab60 0x7f419c154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 -- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 msgr2=0x7f419c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 0x7f419c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419a000640 1 -- 172.21.15.28:0/2832556512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188009400 con 0x7f419c15a780 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.460+0000 7f419aa00640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 0x7f419c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f419a000640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4188035040 tx=0x7f41880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188032440 con 0x7f419c15a780 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4188006e10 con 0x7f419c15a780 2024-09-13T17:08:24.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188030be0 con 0x7f419c15a780 2024-09-13T17:08:24.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c13d060 con 0x7f419c15a780 2024-09-13T17:08:24.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.461+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c1519c0 con 0x7f419c15a780 2024-09-13T17:08:24.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.462+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c13d580 con 0x7f419c15a780 2024-09-13T17:08:24.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.463+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4188039040 con 0x7f419c15a780 2024-09-13T17:08:24.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.464+0000 7f4192a00640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 0x7f417c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.464+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41880a9ac0 con 0x7f419c15a780 2024-09-13T17:08:24.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.464+0000 7f419aa00640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 0x7f417c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.465+0000 7f419aa00640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 0x7f417c06d0f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f418c005880 tx=0x7f418c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.468+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f419c13d580 con 0x7f419c15a780 2024-09-13T17:08:24.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.648+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f419c13d580 con 0x7f419c15a780 2024-09-13T17:08:24.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.649+0000 7f4192a00640 1 -- 172.21.15.28:0/2832556512 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f419c13d580 con 0x7f419c15a780 2024-09-13T17:08:24.650 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 msgr2=0x7f417c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 0x7f417c06d0f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f418c005880 tx=0x7f418c009040 comp rx=0 tx=0).stop 2024-09-13T17:08:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 msgr2=0x7f419c154320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4188035040 tx=0x7f41880046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 shutdown_connections 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f417c06ac30 0x7f417c06d0f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f419c15ab60 0x7f419c154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f419c15a780 0x7f419c154320 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 --2- 172.21.15.28:0/2832556512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f419c071490 0x7f419c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 >> 172.21.15.28:0/2832556512 conn(0x7f419c0b0270 msgr2=0x7f419c0cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 shutdown_connections 2024-09-13T17:08:24.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.651+0000 7f41a1800640 1 -- 172.21.15.28:0/2832556512 wait complete. 2024-09-13T17:08:24.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.825+0000 7f5da3200640 1 Processor -- start 2024-09-13T17:08:24.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.825+0000 7f5da3200640 1 -- start start 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da406d2f0 con 0x7f5da4073550 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da406a8e0 con 0x7f5da406af70 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da406c400 con 0x7f5da41460f0 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5da4073550 0x7f5da406d780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37628/0 (socket says 172.21.15.28:37628) 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da2800640 1 -- 172.21.15.28:0/1233318935 learned_addr learned my addr 172.21.15.28:0/1233318935 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da1e00640 1 --2- 172.21.15.28:0/1233318935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552797193 0 0) 0x7f5da406d2f0 con 0x7f5da4073550 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.827+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d88003520 con 0x7f5da4073550 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.828+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770785672 0 0) 0x7f5da406a8e0 con 0x7f5da406af70 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.828+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5da406d2f0 con 0x7f5da406af70 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.828+0000 7f5da1e00640 1 -- 172.21.15.28:0/1233318935 >> v1:172.21.15.189:6789/0 conn(0x7f5da406af70 legacy=0x7f5da4072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.828+0000 7f5da1e00640 1 -- 172.21.15.28:0/1233318935 >> v1:172.21.15.28:6789/0 conn(0x7f5da4073550 legacy=0x7f5da406d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.828+0000 7f5da1e00640 1 -- 172.21.15.28:0/1233318935 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da406c6e0 con 0x7f5da41460f0 2024-09-13T17:08:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da1e00640 1 --2- 172.21.15.28:0/1233318935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da40ac850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d940097d0 tx=0x7f5d9402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=353f628e53c2229c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94032440 con 0x7f5da41460f0 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d940325a0 con 0x7f5da41460f0 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da1400640 1 -- 172.21.15.28:0/1233318935 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d94032930 con 0x7f5da41460f0 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da3200640 1 -- 172.21.15.28:0/1233318935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 msgr2=0x7f5da40ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.829+0000 7f5da3200640 1 --2- 172.21.15.28:0/1233318935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da40ac850 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5d940097d0 tx=0x7f5d9402a210 comp rx=0 tx=0).stop 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.830+0000 7f5da3200640 1 -- 172.21.15.28:0/1233318935 shutdown_connections 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.830+0000 7f5da3200640 1 --2- 172.21.15.28:0/1233318935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da40ac850 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.830+0000 7f5da3200640 1 -- 172.21.15.28:0/1233318935 >> 172.21.15.28:0/1233318935 conn(0x7f5da40b1280 msgr2=0x7f5da40b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.830+0000 7f5da3200640 1 -- 172.21.15.28:0/1233318935 shutdown_connections 2024-09-13T17:08:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.830+0000 7f5da3200640 1 -- 172.21.15.28:0/1233318935 wait complete. 2024-09-13T17:08:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.831+0000 7f5da3200640 1 Processor -- start 2024-09-13T17:08:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.831+0000 7f5da3200640 1 -- start start 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da406af70 0x7f5da41f5c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da41423e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da4140d30 con 0x7f5da4073550 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da415ca20 con 0x7f5da406af70 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41f6150 con 0x7f5da41460f0 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5d9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5d9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59654/0 (socket says 172.21.15.28:59654) 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5d9be00640 1 -- 172.21.15.28:0/4258662531 learned_addr learned my addr 172.21.15.28:0/4258662531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:24.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.832+0000 7f5da1e00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da406af70 0x7f5da41f5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5da2800640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da41423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 -- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 msgr2=0x7f5da41423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da41423e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 -- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da406af70 msgr2=0x7f5da41f5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da406af70 0x7f5da41f5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 -- 172.21.15.28:0/4258662531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d94009400 con 0x7f5da4073550 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5da2800640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da41423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9be00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c490 tx=0x7f5d8c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c0052d0 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d8c005430 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c012870 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.833+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41400b0 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.834+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da4140670 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.835+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da41464d0 con 0x7f5da4073550 2024-09-13T17:08:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.836+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5d8c0129d0 con 0x7f5da4073550 2024-09-13T17:08:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.836+0000 7f5d9a000640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 0x7f5d8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.837+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d8c088e80 con 0x7f5da4073550 2024-09-13T17:08:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.837+0000 7f5da1e00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 0x7f5d8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.838+0000 7f5da1e00640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 0x7f5d8006d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5d94034000 tx=0x7f5d9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:24.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:24.841+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5d8c054050 con 0x7f5da4073550 2024-09-13T17:08:25.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.018+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5da4073250 con 0x7f5da4073550 2024-09-13T17:08:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.019+0000 7f5d9a000640 1 -- 172.21.15.28:0/4258662531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d8c057670 con 0x7f5da4073550 2024-09-13T17:08:25.020 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 msgr2=0x7f5d8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 0x7f5d8006d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5d94034000 tx=0x7f5d9402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 msgr2=0x7f5da4141ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c00c490 tx=0x7f5d8c00c950 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 shutdown_connections 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d8006ac00 0x7f5d8006d0c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5da41460f0 0x7f5da41423e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5da4073550 0x7f5da4141ea0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 --2- 172.21.15.28:0/4258662531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da406af70 0x7f5da41f5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 >> 172.21.15.28:0/4258662531 conn(0x7f5da40b1280 msgr2=0x7f5da4070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 shutdown_connections 2024-09-13T17:08:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.021+0000 7f5da3200640 1 -- 172.21.15.28:0/4258662531 wait complete. 2024-09-13T17:08:25.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.192+0000 7f67e4e00640 1 Processor -- start 2024-09-13T17:08:25.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.192+0000 7f67e4e00640 1 -- start start 2024-09-13T17:08:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.193+0000 7f67e4e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006f970 0x7f67e015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.193+0000 7f67e4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67e006c2d0 0x7f67e006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.193+0000 7f67e4e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67e0159d90 con 0x7f67e01592a0 2024-09-13T17:08:25.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.193+0000 7f67e4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67e015b0b0 con 0x7f67e006c2d0 2024-09-13T17:08:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.193+0000 7f67e4e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e006cca0 con 0x7f67e006f970 2024-09-13T17:08:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67d5400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006f970 0x7f67e015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67dde00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f67e01592a0 0x7f67e0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:55476/0 (socket says 172.21.15.28:55476) 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67dde00640 1 -- 172.21.15.28:0/1699641250 learned_addr learned my addr 172.21.15.28:0/1699641250 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67de800640 1 --2- 172.21.15.28:0/1699641250 >> v2:172.21.15.28:3300/0 conn(0x7f67e006c2d0 0x7f67e006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67dd400640 1 -- 172.21.15.28:0/1699641250 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558536992 0 0) 0x7f67e0159d90 con 0x7f67e01592a0 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67dd400640 1 -- 172.21.15.28:0/1699641250 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67c4005730 con 0x7f67e01592a0 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67de800640 1 -- 172.21.15.28:0/1699641250 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006f970 msgr2=0x7f67e015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67de800640 1 --2- 172.21.15.28:0/1699641250 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006f970 0x7f67e015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67de800640 1 -- 172.21.15.28:0/1699641250 >> v1:172.21.15.189:6789/0 conn(0x7f67e01592a0 legacy=0x7f67e0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.194+0000 7f67de800640 1 -- 172.21.15.28:0/1699641250 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00b2040 con 0x7f67e006c2d0 2024-09-13T17:08:25.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.195+0000 7f67de800640 1 --2- 172.21.15.28:0/1699641250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006c2d0 0x7f67e006c760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009d90 tx=0x7f67d802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63dadc8edc225b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.195+0000 7f67dd400640 1 -- 172.21.15.28:0/1699641250 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8007740 con 0x7f67e006c2d0 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67dd400640 1 -- 172.21.15.28:0/1699641250 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d802d050 con 0x7f67e006c2d0 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67dd400640 1 -- 172.21.15.28:0/1699641250 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8007aa0 con 0x7f67e006c2d0 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 -- 172.21.15.28:0/1699641250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006c2d0 msgr2=0x7f67e006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 --2- 172.21.15.28:0/1699641250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006c2d0 0x7f67e006c760 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009d90 tx=0x7f67d802a660 comp rx=0 tx=0).stop 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 -- 172.21.15.28:0/1699641250 shutdown_connections 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 --2- 172.21.15.28:0/1699641250 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006c2d0 0x7f67e006c760 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 --2- 172.21.15.28:0/1699641250 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006f970 0x7f67e015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 -- 172.21.15.28:0/1699641250 >> 172.21.15.28:0/1699641250 conn(0x7f67e0069e80 msgr2=0x7f67e006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 -- 172.21.15.28:0/1699641250 shutdown_connections 2024-09-13T17:08:25.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.196+0000 7f67e4e00640 1 -- 172.21.15.28:0/1699641250 wait complete. 2024-09-13T17:08:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.198+0000 7f67e4e00640 1 Processor -- start 2024-09-13T17:08:25.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.198+0000 7f67e4e00640 1 -- start start 2024-09-13T17:08:25.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006c2d0 0x7f67e00cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 0x7f67e014ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e00cc670 con 0x7f67e006f970 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e0153cd0 con 0x7f67e01592a0 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67e4e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67e01539f0 con 0x7f67e006c2d0 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59666/0 (socket says 172.21.15.28:59666) 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67d5400640 1 -- 172.21.15.28:0/22678454 learned_addr learned my addr 172.21.15.28:0/22678454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67dde00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006c2d0 0x7f67e00cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67de800640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 0x7f67e014ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67d5400640 1 -- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006c2d0 msgr2=0x7f67e00cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.199+0000 7f67d5400640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006c2d0 0x7f67e00cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d5400640 1 -- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 msgr2=0x7f67e014ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d5400640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 0x7f67e014ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d5400640 1 -- 172.21.15.28:0/22678454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d8009a70 con 0x7f67e006f970 2024-09-13T17:08:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67de800640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 0x7f67e014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:25.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d5400640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f67c800e9b0 tx=0x7f67c800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c8009670 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c800c9a0 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e00cb650 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c800cd50 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.200+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e00cbc10 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.201+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e00b3740 con 0x7f67e006f970 2024-09-13T17:08:25.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.202+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f67c8013470 con 0x7f67e006f970 2024-09-13T17:08:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.203+0000 7f67d6a00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 0x7f67bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.203+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67c8089640 con 0x7f67e006f970 2024-09-13T17:08:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.203+0000 7f67dde00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 0x7f67bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.204+0000 7f67dde00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 0x7f67bc06d0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67d0005880 tx=0x7f67d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.207+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f67c8054780 con 0x7f67e006f970 2024-09-13T17:08:25.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.418+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67e015bf40 con 0x7f67e006f970 2024-09-13T17:08:25.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.419+0000 7f67d6a00640 1 -- 172.21.15.28:0/22678454 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f67c8057da0 con 0x7f67e006f970 2024-09-13T17:08:25.421 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:25.421 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:25.422 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 msgr2=0x7f67bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 0x7f67bc06d0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f67d0005880 tx=0x7f67d0009040 comp rx=0 tx=0).stop 2024-09-13T17:08:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 msgr2=0x7f67e006b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f67c800e9b0 tx=0x7f67c800ee70 comp rx=0 tx=0).stop 2024-09-13T17:08:25.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 shutdown_connections 2024-09-13T17:08:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67bc06ac00 0x7f67bc06d0c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e01592a0 0x7f67e014ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67e006f970 0x7f67e006b430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 --2- 172.21.15.28:0/22678454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67e006c2d0 0x7f67e00cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 >> 172.21.15.28:0/22678454 conn(0x7f67e0069e80 msgr2=0x7f67e006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 shutdown_connections 2024-09-13T17:08:25.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.421+0000 7f67e4e00640 1 -- 172.21.15.28:0/22678454 wait complete. 2024-09-13T17:08:25.437 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.590+0000 7f98fbc00640 1 Processor -- start 2024-09-13T17:08:25.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.590+0000 7f98fbc00640 1 -- start start 2024-09-13T17:08:25.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98fbc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f98f415a540 0x7f98f406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98fbc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f415e100 con 0x7f98f40cabc0 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98fbc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f415e3e0 con 0x7f98f4070cc0 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98fbc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f40adb00 con 0x7f98f415a540 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98f3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98f40cabc0 0x7f98f415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37636/0 (socket says 172.21.15.28:37636) 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98f3e00640 1 -- 172.21.15.28:0/2832327313 learned_addr learned my addr 172.21.15.28:0/2832327313 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98f9200640 1 --2- 172.21.15.28:0/2832327313 >> v2:172.21.15.189:3300/0 conn(0x7f98f415a540 0x7f98f406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92734104 0 0) 0x7f98f415e100 con 0x7f98f40cabc0 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.592+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98dc003520 con 0x7f98f40cabc0 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2116789990 0 0) 0x7f98dc003520 con 0x7f98f40cabc0 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 >> v2:172.21.15.189:3300/0 conn(0x7f98f415a540 msgr2=0x7f98f406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 --2- 172.21.15.28:0/2832327313 >> v2:172.21.15.189:3300/0 conn(0x7f98f415a540 0x7f98f406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 >> v1:172.21.15.189:6789/0 conn(0x7f98f4070cc0 legacy=0x7f98f406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f415ef80 con 0x7f98f40cabc0 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f98e0001670 con 0x7f98f40cabc0 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 >> v1:172.21.15.28:6789/0 conn(0x7f98f40cabc0 legacy=0x7f98f415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98dc003970 0x7f98dc003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c050 con 0x7f98dc005180 2024-09-13T17:08:25.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c1c0 con 0x7f98dc003970 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc00c380 con 0x7f98dc0047c0 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f9c00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f9200640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98dc003970 0x7f98dc003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.593+0000 7f98f3e00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 -- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98dc0047c0 msgr2=0x7f98dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:25.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98dc0047c0 0x7f98dc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 -- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98dc003970 msgr2=0x7f98dc003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98dc003970 0x7f98dc003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 -- 172.21.15.28:0/2832327313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc003520 con 0x7f98dc005180 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.594+0000 7f98f9c00640 1 -- 172.21.15.28:0/2832327313 reap_dead start 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98f9c00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f98ec001f60 tx=0x7f98dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ecbc50eacc011c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec003b70 con 0x7f98dc005180 2024-09-13T17:08:25.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec02d050 con 0x7f98dc005180 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98f3400640 1 -- 172.21.15.28:0/2832327313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec005110 con 0x7f98dc005180 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 -- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 msgr2=0x7f98dc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f98ec001f60 tx=0x7f98dc0057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 -- 172.21.15.28:0/2832327313 shutdown_connections 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 --2- 172.21.15.28:0/2832327313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98dc00b980 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 -- 172.21.15.28:0/2832327313 >> 172.21.15.28:0/2832327313 conn(0x7f98f4069e80 msgr2=0x7f98f406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.595+0000 7f98fbc00640 1 -- 172.21.15.28:0/2832327313 shutdown_connections 2024-09-13T17:08:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.596+0000 7f98fbc00640 1 -- 172.21.15.28:0/2832327313 wait complete. 2024-09-13T17:08:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.597+0000 7f98fbc00640 1 Processor -- start 2024-09-13T17:08:25.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.597+0000 7f98fbc00640 1 -- start start 2024-09-13T17:08:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 0x7f98f4158a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 0x7f98f406ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f4070500 con 0x7f98dc005180 2024-09-13T17:08:25.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40aef70 con 0x7f98f406fb40 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98fbc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f40af250 con 0x7f98f414b130 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.598+0000 7f98f9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59678/0 (socket says 172.21.15.28:59678) 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 -- 172.21.15.28:0/3461572853 learned_addr learned my addr 172.21.15.28:0/3461572853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f3e00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 0x7f98f4158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 -- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 msgr2=0x7f98f4158a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 0x7f98f4158a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9c00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 0x7f98f406ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 -- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 msgr2=0x7f98f406ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 0x7f98f406ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 -- 172.21.15.28:0/3461572853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f414b5b0 con 0x7f98dc005180 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f3e00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 0x7f98f4158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9c00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 0x7f98f406ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f9200640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f98e8011910 tx=0x7f98e8011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e800fa90 con 0x7f98dc005180 2024-09-13T17:08:25.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e800fbf0 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8014440 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.599+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f40b3010 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.600+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f40b35d0 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.601+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f4061090 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.601+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98e80145a0 con 0x7f98dc005180 2024-09-13T17:08:25.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.602+0000 7f98f0c00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.602+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98e8089b90 con 0x7f98dc005180 2024-09-13T17:08:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.603+0000 7f98f3e00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.603+0000 7f98f3e00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98f40ac870 tx=0x7f98e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.606+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98e8054d70 con 0x7f98dc005180 2024-09-13T17:08:25.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.786+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98f406c7c0 con 0x7f98dc005180 2024-09-13T17:08:25.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.787+0000 7f98f0c00640 1 -- 172.21.15.28:0/3461572853 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f98e8058390 con 0x7f98dc005180 2024-09-13T17:08:25.788 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:25.788 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 msgr2=0x7f98d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f98f40ac870 tx=0x7f98e0009040 comp rx=0 tx=0).stop 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 msgr2=0x7f98f40b2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f98e8011910 tx=0x7f98e8011dd0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 shutdown_connections 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f406fb40 0x7f98f406ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f414b130 0x7f98f4158a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.789+0000 7f98fbc00640 1 --2- 172.21.15.28:0/3461572853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98dc005180 0x7f98f40b2a30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.790+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 >> 172.21.15.28:0/3461572853 conn(0x7f98f4069e80 msgr2=0x7f98f415b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.790+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 shutdown_connections 2024-09-13T17:08:25.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.790+0000 7f98fbc00640 1 -- 172.21.15.28:0/3461572853 wait complete. 2024-09-13T17:08:25.807 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:25.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.962+0000 7f98f7200640 1 Processor -- start 2024-09-13T17:08:25.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.962+0000 7f98f7200640 1 -- start start 2024-09-13T17:08:25.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f8159440 0x7f98f80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98f8159820 0x7f98f80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f7200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f80c3850 con 0x7f98f80b22e0 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f815a930 con 0x7f98f8159820 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8159d20 con 0x7f98f8159440 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98f8159820 0x7f98f80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98f8159820 0x7f98f80b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59682/0 (socket says 172.21.15.28:59682) 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98efe00640 1 -- 172.21.15.28:0/3027799472 learned_addr learned my addr 172.21.15.28:0/3027799472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.963+0000 7f98f5e00640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f8159440 0x7f98f80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.964+0000 7f98efe00640 1 -- 172.21.15.28:0/3027799472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f8159440 msgr2=0x7f98f80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.964+0000 7f98efe00640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f8159440 0x7f98f80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.964+0000 7f98efe00640 1 -- 172.21.15.28:0/3027799472 >> v1:172.21.15.189:6789/0 conn(0x7f98f80b22e0 legacy=0x7f98f80b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.964+0000 7f98efe00640 1 -- 172.21.15.28:0/3027799472 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f815a000 con 0x7f98f8159820 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98efe00640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159820 0x7f98f80b0980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009870 tx=0x7f98e002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d850c08e59e40311 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98f5400640 1 -- 172.21.15.28:0/3027799472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0032640 con 0x7f98f8159820 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98f5400640 1 -- 172.21.15.28:0/3027799472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e00327a0 con 0x7f98f8159820 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98f5400640 1 -- 172.21.15.28:0/3027799472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0032b30 con 0x7f98f8159820 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98f7200640 1 -- 172.21.15.28:0/3027799472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159820 msgr2=0x7f98f80b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.965+0000 7f98f7200640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159820 0x7f98f80b0980 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f98e0009870 tx=0x7f98e002a300 comp rx=0 tx=0).stop 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 -- 172.21.15.28:0/3027799472 shutdown_connections 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159820 0x7f98f80b0980 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 --2- 172.21.15.28:0/3027799472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f8159440 0x7f98f80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 -- 172.21.15.28:0/3027799472 >> 172.21.15.28:0/3027799472 conn(0x7f98f8072e40 msgr2=0x7f98f8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 -- 172.21.15.28:0/3027799472 shutdown_connections 2024-09-13T17:08:25.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.966+0000 7f98f7200640 1 -- 172.21.15.28:0/3027799472 wait complete. 2024-09-13T17:08:25.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.967+0000 7f98f7200640 1 Processor -- start 2024-09-13T17:08:25.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.967+0000 7f98f7200640 1 -- start start 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f80b22e0 0x7f98f813d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f8159820 0x7f98f815b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f813de30 con 0x7f98f8159440 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f814e4d0 con 0x7f98f8159820 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98f7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8140e90 con 0x7f98f80b22e0 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59692/0 (socket says 172.21.15.28:59692) 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.968+0000 7f98efe00640 1 -- 172.21.15.28:0/629801625 learned_addr learned my addr 172.21.15.28:0/629801625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 -- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f80b22e0 msgr2=0x7f98f813d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f80b22e0 0x7f98f813d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 -- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f8159820 msgr2=0x7f98f815b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f8159820 0x7f98f815b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 -- 172.21.15.28:0/629801625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00094d0 con 0x7f98f8159440 2024-09-13T17:08:25.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98efe00640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98e00090b0 tx=0x7f98e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0030850 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e00309f0 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e0030da0 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.969+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f814e750 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.970+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f814ed10 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.970+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f80b2790 con 0x7f98f8159440 2024-09-13T17:08:25.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.971+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98e0039040 con 0x7f98f8159440 2024-09-13T17:08:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.972+0000 7f98ee000640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.972+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98e0041030 con 0x7f98f8159440 2024-09-13T17:08:25.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.973+0000 7f98f5e00640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:25.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.973+0000 7f98f5e00640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f98f815c350 tx=0x7f98e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:25.977+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98e0074e30 con 0x7f98f8159440 2024-09-13T17:08:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.191+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98f80cd5b0 con 0x7f98f8159440 2024-09-13T17:08:26.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.192+0000 7f98ee000640 1 -- 172.21.15.28:0/629801625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f98e004b090 con 0x7f98f8159440 2024-09-13T17:08:26.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:26.193 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 msgr2=0x7f98d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f98f815c350 tx=0x7f98e8008040 comp rx=0 tx=0).stop 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 msgr2=0x7f98f815b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98e00090b0 tx=0x7f98e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 shutdown_connections 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98d406ac00 0x7f98d406d0c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98f8159820 0x7f98f815b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98f8159440 0x7f98f815b3d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 --2- 172.21.15.28:0/629801625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98f80b22e0 0x7f98f813d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 >> 172.21.15.28:0/629801625 conn(0x7f98f8072e40 msgr2=0x7f98f815d7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 shutdown_connections 2024-09-13T17:08:26.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.194+0000 7f98f7200640 1 -- 172.21.15.28:0/629801625 wait complete. 2024-09-13T17:08:26.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:08:11.555431+0000', 'modified': '2024-09-13T17:08:14.569832+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6837/946583009', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 946583009}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 946583009}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:08:26.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:08:26.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T17:08:26.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.772+0000 7fadc5600640 1 Processor -- start 2024-09-13T17:08:26.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.772+0000 7fadc5600640 1 -- start start 2024-09-13T17:08:26.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadc5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc0070430 0x7fadc015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadc5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadc006f3f0 0x7fadc015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadc5600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadc015ab60 con 0x7fadc0159770 2024-09-13T17:08:26.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadc5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadc006f920 con 0x7fadc006f3f0 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadc5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc015b290 con 0x7fadc0070430 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadc006f3f0 0x7fadc015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fadc006f3f0 0x7fadc015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59702/0 (socket says 172.21.15.28:59702) 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 -- 172.21.15.28:0/3442075936 learned_addr learned my addr 172.21.15.28:0/3442075936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadb7e00640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc0070430 0x7fadc015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 -- 172.21.15.28:0/3442075936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc0070430 msgr2=0x7fadc015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc0070430 0x7fadc015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 -- 172.21.15.28:0/3442075936 >> v1:172.21.15.189:6789/0 conn(0x7fadc0159770 legacy=0x7fadc006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.774+0000 7fadbf400640 1 -- 172.21.15.28:0/3442075936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc015b570 con 0x7fadc006f3f0 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.775+0000 7fadbf400640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc006f3f0 0x7fadc015c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009e50 tx=0x7fadb802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64bfdf334ee1d22 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadbe000640 1 -- 172.21.15.28:0/3442075936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb802aea0 con 0x7fadc006f3f0 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadbe000640 1 -- 172.21.15.28:0/3442075936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb802d050 con 0x7fadc006f3f0 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadbe000640 1 -- 172.21.15.28:0/3442075936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb8031820 con 0x7fadc006f3f0 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 -- 172.21.15.28:0/3442075936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc006f3f0 msgr2=0x7fadc015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc006f3f0 0x7fadc015c2b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009e50 tx=0x7fadb802a680 comp rx=0 tx=0).stop 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 -- 172.21.15.28:0/3442075936 shutdown_connections 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc006f3f0 0x7fadc015c2b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 --2- 172.21.15.28:0/3442075936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc0070430 0x7fadc015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 -- 172.21.15.28:0/3442075936 >> 172.21.15.28:0/3442075936 conn(0x7fadc00b0270 msgr2=0x7fadc00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 -- 172.21.15.28:0/3442075936 shutdown_connections 2024-09-13T17:08:26.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.776+0000 7fadc5600640 1 -- 172.21.15.28:0/3442075936 wait complete. 2024-09-13T17:08:26.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.778+0000 7fadc5600640 1 Processor -- start 2024-09-13T17:08:26.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.778+0000 7fadc5600640 1 -- start start 2024-09-13T17:08:26.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc006f3f0 0x7fadc00acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc0159770 0x7fadc0155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc0154e20 con 0x7fadc0070430 2024-09-13T17:08:26.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadc01547f0 con 0x7fadc0159770 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadc5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc00b22a0 con 0x7fadc006f3f0 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59712/0 (socket says 172.21.15.28:59712) 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadb7e00640 1 -- 172.21.15.28:0/2951146350 learned_addr learned my addr 172.21.15.28:0/2951146350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadbf400640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc0159770 0x7fadc0155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.779+0000 7fadbea00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc006f3f0 0x7fadc00acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 -- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc006f3f0 msgr2=0x7fadc00acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc006f3f0 0x7fadc00acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 -- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc0159770 msgr2=0x7fadc0155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc0159770 0x7fadc0155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 -- 172.21.15.28:0/2951146350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb8009b30 con 0x7fadc0070430 2024-09-13T17:08:26.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fada800e9b0 tx=0x7fada800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb6a00640 1 -- 172.21.15.28:0/2951146350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada8009670 con 0x7fadc0070430 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb6a00640 1 -- 172.21.15.28:0/2951146350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada800c9a0 con 0x7fadc0070430 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadc00b2580 con 0x7fadc0070430 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadb6a00640 1 -- 172.21.15.28:0/2951146350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada800cd50 con 0x7fadc0070430 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.780+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadc00ad000 con 0x7fadc0070430 2024-09-13T17:08:26.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.782+0000 7fadb6a00640 1 -- 172.21.15.28:0/2951146350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fada8013470 con 0x7fadc0070430 2024-09-13T17:08:26.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.783+0000 7fadb6a00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 0x7fad9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.783+0000 7fadb6a00640 1 -- 172.21.15.28:0/2951146350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fada8089550 con 0x7fadc0070430 2024-09-13T17:08:26.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.784+0000 7fadbea00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 0x7fad9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.784+0000 7fadc8468740 1 Processor -- start 2024-09-13T17:08:26.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.784+0000 7fadc8468740 1 -- start start 2024-09-13T17:08:26.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.784+0000 7fadbea00640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 0x7fad9c06d0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fadb0000f60 tx=0x7fadb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad3308ce90 0x55ad330dcba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330dd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad32ff8ab0 con 0x55ad3308f6d0 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad32ff6240 con 0x55ad330b6b80 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad32ff1b30 con 0x55ad3308ce90 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59722/0 (socket says 172.21.15.28:59722) 2024-09-13T17:08:26.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 -- 172.21.15.28:0/1734325072 learned_addr learned my addr 172.21.15.28:0/1734325072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbf400640 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330dd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadb7e00640 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad3308ce90 0x55ad330dcba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 -- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad3308ce90 msgr2=0x55ad330dcba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad3308ce90 0x55ad330dcba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 -- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 msgr2=0x55ad330dd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330dd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.785+0000 7fadbea00640 1 -- 172.21.15.28:0/1734325072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad32fbcbd0 con 0x55ad3308f6d0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadbea00640 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fadb000f9e0 tx=0x7fadb0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7dd7591795d9e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadaf400640 1 -- 172.21.15.28:0/1734325072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0001400 con 0x55ad3308f6d0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadaf400640 1 -- 172.21.15.28:0/1734325072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0014be0 con 0x55ad3308f6d0 2024-09-13T17:08:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadaf400640 1 -- 172.21.15.28:0/1734325072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0020410 con 0x55ad3308f6d0 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadc8468740 1 -- 172.21.15.28:0/1734325072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 msgr2=0x55ad3308fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.786+0000 7fadc8468740 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fadb000f9e0 tx=0x7fadb0012000 comp rx=0 tx=0).stop 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 -- 172.21.15.28:0/1734325072 shutdown_connections 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330dd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad3308ce90 0x55ad330dcba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 --2- 172.21.15.28:0/1734325072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad3308fab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 -- 172.21.15.28:0/1734325072 >> 172.21.15.28:0/1734325072 conn(0x55ad3308be00 msgr2=0x55ad330d42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 -- 172.21.15.28:0/1734325072 shutdown_connections 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 -- 172.21.15.28:0/1734325072 wait complete. 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 -- start start 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.787+0000 7fadc8468740 1 Processor -- start 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b6b80 0x55ad330b82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b8800 0x55ad330d9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad32f93260 con 0x55ad3308f6d0 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad32f92f00 con 0x55ad330b8800 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad32d58f20 con 0x55ad330b6b80 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.788+0000 7fadc8468740 10 client.0 _ll_register_callbacks cb 0x55ad32feb1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59732/0 (socket says 172.21.15.28:59732) 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 -- 172.21.15.28:0/137178850 learned_addr learned my addr 172.21.15.28:0/137178850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbf400640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b8800 0x55ad330d9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadb7e00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b6b80 0x55ad330b82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 -- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b6b80 msgr2=0x55ad330b82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b6b80 0x55ad330b82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 -- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b8800 msgr2=0x55ad330d9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b8800 0x55ad330d9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 -- 172.21.15.28:0/137178850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb0001d10 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fadbea00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fadb0023040 tx=0x7fadb0001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c25982b87bc78aad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.789+0000 7fada3e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0020b90 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb001c040 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad3301bcd0 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadb0020cf0 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 10 client.4665 fetch_fsmap learned FSMap version 8 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55ad32ff01b0 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad3301bcd0 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fadb0012070 con 0x55ad3308f6d0 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 10 client.4665 fetch_fsmap finished waiting for FSMap version 8 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 10 client.4665 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:08:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 20 client.4665 populate_metadata read hostname 'smithi028' 2024-09-13T17:08:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330ef260 0x55ad330f1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 4 client.4665 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:08:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.790+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 --> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad32ce53c0 con 0x55ad330ef260 2024-09-13T17:08:26.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.791+0000 7fadb7e00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330ef260 0x55ad330f1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.791+0000 7fadb7e00640 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330ef260 0x55ad330f1650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd95f40a893dc0c9 server_cookie=4c16bcc8212e3e75 in_seq=0 out_seq=0 2024-09-13T17:08:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.791+0000 7fada3e00640 10 client.4665 ms_handle_connect on v2:172.21.15.189:6835/946583009 2024-09-13T17:08:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.792+0000 7fada3e00640 1 -- 172.21.15.28:0/137178850 <== mds.0 v2:172.21.15.189:6835/946583009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55ad32ce53c0 con 0x55ad330ef260 2024-09-13T17:08:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.792+0000 7fada3e00640 10 client.4665 handle_command_reply: tid=0 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.792+0000 7fadc8468740 1 client.4665 shutdown 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.792+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 msgr2=0x55ad330dc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.792+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fadb0023040 tx=0x7fadb0001080 comp rx=0 tx=0).stop 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 shutdown_connections 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330ef260 0x55ad330f1650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b8800 0x55ad330d9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b6b80 0x55ad330b82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 --2- 172.21.15.28:0/137178850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330dc1b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fada3e00640 0 client.4665 ms_handle_reset on v2:172.21.15.189:6835/946583009 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 >> 172.21.15.28:0/137178850 conn(0x55ad3308ce90 msgr2=0x55ad330d5b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 shutdown_connections 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 1 -- 172.21.15.28:0/137178850 wait complete. 2024-09-13T17:08:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.793+0000 7fadc8468740 20 client.4665 trim_cache size 0 max 16384 2024-09-13T17:08:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.808+0000 7fadc8468740 1 Processor -- start 2024-09-13T17:08:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.808+0000 7fadc8468740 1 -- start start 2024-09-13T17:08:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad3308af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad3308b4a0 con 0x55ad3308f6d0 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad330b40a0 con 0x55ad330b6b80 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.809+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad3308b860 con 0x55ad330b8800 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59738/0 (socket says 172.21.15.28:59738) 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 -- 172.21.15.28:0/804377641 learned_addr learned my addr 172.21.15.28:0/804377641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbf400640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadb7e00640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad3308af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 -- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 msgr2=0x55ad330d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 -- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 msgr2=0x55ad3308af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad3308af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 -- 172.21.15.28:0/804377641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad33025170 con 0x55ad3308f6d0 2024-09-13T17:08:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadb7e00640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad3308af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadbea00640 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadb0008fd0 tx=0x7fadb002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadbc8a63b38a49d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadafe00640 1 -- 172.21.15.28:0/804377641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0020ea0 con 0x55ad3308f6d0 2024-09-13T17:08:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadafe00640 1 -- 172.21.15.28:0/804377641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb0033520 con 0x55ad3308f6d0 2024-09-13T17:08:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.810+0000 7fadafe00640 1 -- 172.21.15.28:0/804377641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0007ea0 con 0x55ad3308f6d0 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 -- 172.21.15.28:0/804377641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 msgr2=0x55ad330f3880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fadb0008fd0 tx=0x7fadb002a000 comp rx=0 tx=0).stop 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 -- 172.21.15.28:0/804377641 shutdown_connections 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad3308af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 --2- 172.21.15.28:0/804377641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad330f3880 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 -- 172.21.15.28:0/804377641 >> 172.21.15.28:0/804377641 conn(0x55ad3308ce90 msgr2=0x55ad330d1980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 -- 172.21.15.28:0/804377641 shutdown_connections 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.811+0000 7fadc8468740 1 -- 172.21.15.28:0/804377641 wait complete. 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.812+0000 7fadc8468740 1 -- start start 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.812+0000 7fadc8468740 1 Processor -- start 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330ef640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad33128390 con 0x55ad3308f6d0 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad330b7050 con 0x55ad330b6b80 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ad33140080 con 0x55ad330b8800 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadc8468740 10 client.0 _ll_register_callbacks cb 0x7fadb0023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:08:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59744/0 (socket says 172.21.15.28:59744) 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 -- 172.21.15.28:0/2754421006 learned_addr learned my addr 172.21.15.28:0/2754421006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbf400640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330ef640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 -- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 msgr2=0x55ad330b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330b34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 -- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 msgr2=0x55ad330ef640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330ef640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.813+0000 7fadbea00640 1 -- 172.21.15.28:0/2754421006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad32ff01b0 con 0x55ad3308f6d0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fadbea00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fadb000f640 tx=0x7fadb000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca879d3cf0ad0ade server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fada3e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0007ea0 con 0x55ad3308f6d0 2024-09-13T17:08:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadb0033cd0 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad330bbc00 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7fadb0020ea0 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fadc8468740 10 client.4671 fetch_fsmap learned FSMap version 8 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.814+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55ad32f92470 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ad330bbc00 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 8) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fadb0012030 con 0x55ad3308f6d0 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 10 client.4671 fetch_fsmap finished waiting for FSMap version 8 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 10 client.4671 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 20 client.4671 populate_metadata read hostname 'smithi028' 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330d2300 0x55ad330d26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 4 client.4671 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 --> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fadb0014be0 con 0x55ad330d2300 2024-09-13T17:08:26.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.815+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330d2300 0x55ad330d26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadb7e00640 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330d2300 0x55ad330d26e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ed339eabfb3c944 server_cookie=3401fc120c90cbd9 in_seq=0 out_seq=0 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fada3e00640 10 client.4671 ms_handle_connect on v2:172.21.15.189:6835/946583009 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fada3e00640 1 -- 172.21.15.28:0/2754421006 <== mds.0 v2:172.21.15.189:6835/946583009 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7fadb0014be0 con 0x55ad330d2300 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fada3e00640 10 client.4671 handle_command_reply: tid=0 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 client.4671 shutdown 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 msgr2=0x55ad33127e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fadb000f640 tx=0x7fadb000a000 comp rx=0 tx=0).stop 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 shutdown_connections 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:6835/946583009,v1:172.21.15.189:6837/946583009] conn(0x55ad330d2300 0x55ad330d26e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ad330b8800 0x55ad330b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ad330b6b80 0x55ad330ef640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 --2- 172.21.15.28:0/2754421006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ad3308f6d0 0x55ad33127e50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 >> 172.21.15.28:0/2754421006 conn(0x55ad3308ce90 msgr2=0x55ad330d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fada3e00640 0 client.4671 ms_handle_reset on v2:172.21.15.189:6835/946583009 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 shutdown_connections 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 1 -- 172.21.15.28:0/2754421006 wait complete. 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.816+0000 7fadc8468740 20 client.4671 trim_cache size 0 max 16384 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 4294, 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:08:26.822 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 15.256888309000001, 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 8, 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 20, 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 20, 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 21.944681782 2024-09-13T17:08:26.823 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 msgr2=0x7fad9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 0x7fad9c06d0c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fadb0000f60 tx=0x7fadb0009040 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 msgr2=0x7fadc0150630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fada800e9b0 tx=0x7fada800ee70 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 shutdown_connections 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad9c06ac00 0x7fad9c06d0c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fadc0159770 0x7fadc0155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadc0070430 0x7fadc0150630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 --2- 172.21.15.28:0/2951146350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fadc006f3f0 0x7fadc00acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 >> 172.21.15.28:0/2951146350 conn(0x7fadc00b0270 msgr2=0x7fadc00af810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 shutdown_connections 2024-09-13T17:08:26.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:26.817+0000 7fadc5600640 1 -- 172.21.15.28:0/2951146350 wait complete. 2024-09-13T17:08:26.834 INFO:teuthology.run_tasks:Running task sequential... 2024-09-13T17:08:26.846 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-13T17:08:26.846 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T17:08:26.847 INFO:teuthology.task.print:Enabling mgr modules 2024-09-13T17:08:26.847 INFO:teuthology.run_tasks:Running task kclient... 2024-09-13T17:08:26.885 INFO:tasks.kclient:Mounting kernel clients... 2024-09-13T17:08:26.885 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-13T17:08:26.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:26.886 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:08:26.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:26.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T17:08:27.020 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-13T17:08:27.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:08:27.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.023 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:08:27.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T17:08:27.159 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-13T17:08:27.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:08:27.162 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.162 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:08:27.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.202 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-09-13T17:08:27.305 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-09-13T17:08:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:08:27.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.309 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:08:27.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.345 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-09-13T17:08:27.443 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-09-13T17:08:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:08:27.446 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-09-13T17:08:27.447 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T17:08:27.447 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:08:27.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:08:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.603+0000 7f3b9fc00640 1 Processor -- start 2024-09-13T17:08:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.603+0000 7f3b9fc00640 1 -- start start 2024-09-13T17:08:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9fc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9fc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b98069f10 con 0x7f3b9815cb60 2024-09-13T17:08:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9fc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b9806e990 con 0x7f3b980afe40 2024-09-13T17:08:27.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9806dcc0 con 0x7f3b9806e500 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b9dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.604+0000 7f3b97e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b97e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:53182/0 (socket says 172.21.15.28:53182) 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b97e00640 1 -- 172.21.15.28:0/2560260695 learned_addr learned my addr 172.21.15.28:0/2560260695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2560260695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:44250/0 (socket says 172.21.15.28:44250) 2024-09-13T17:08:27.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9d200640 1 --1- 172.21.15.28:0/2560260695 >> v1:172.21.15.189:6789/0 conn(0x7f3b9815cb60 0x7f3b9806d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:55510/0 (socket says 172.21.15.28:55510) 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2560260695 >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 msgr2=0x7f3b9806dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2560260695 >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2560260695 >> v1:172.21.15.189:6789/0 conn(0x7f3b9815cb60 legacy=0x7f3b9806d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2560260695 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b980cf5b0 con 0x7f3b9806e500 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b97e00640 1 --2- 172.21.15.28:0/2560260695 >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:27.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.605+0000 7f3b97400640 1 -- 172.21.15.28:0/2560260695 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001645823 0 0) 0x7f3b98069f10 con 0x7f3b9815cb60 2024-09-13T17:08:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.606+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2560260695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009d90 tx=0x7f3b9002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f06367625afe44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.606+0000 7f3b97400640 1 -- 172.21.15.28:0/2560260695 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90007740 con 0x7f3b9806e500 2024-09-13T17:08:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.606+0000 7f3b97400640 1 -- 172.21.15.28:0/2560260695 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9002d050 con 0x7f3b9806e500 2024-09-13T17:08:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.606+0000 7f3b97400640 1 -- 172.21.15.28:0/2560260695 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90007ac0 con 0x7f3b9806e500 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2560260695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 msgr2=0x7f3b9806a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2560260695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3b90009d90 tx=0x7f3b9002a660 comp rx=0 tx=0).stop 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2560260695 shutdown_connections 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2560260695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9806e500 0x7f3b9806a3a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2560260695 >> v2:172.21.15.189:3300/0 conn(0x7f3b980afe40 0x7f3b9806dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2560260695 >> 172.21.15.28:0/2560260695 conn(0x7f3b980b22c0 msgr2=0x7f3b980b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2560260695 shutdown_connections 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.607+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2560260695 wait complete. 2024-09-13T17:08:27.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.609+0000 7f3b9fc00640 1 Processor -- start 2024-09-13T17:08:27.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.609+0000 7f3b9fc00640 1 -- start start 2024-09-13T17:08:27.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b9806e500 0x7f3b98158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9815f810 0x7f3b980b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b980c9a30 con 0x7f3b980c7800 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9815e2f0 con 0x7f3b9806e500 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b981585a0 con 0x7f3b9815f810 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59754/0 (socket says 172.21.15.28:59754) 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2275296379 learned_addr learned my addr 172.21.15.28:0/2275296379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b9d200640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b9806e500 0x7f3b98158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.610+0000 7f3b97e00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9815f810 0x7f3b980b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9815f810 msgr2=0x7f3b980b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9815f810 0x7f3b980b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b9806e500 msgr2=0x7f3b98158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b9806e500 0x7f3b98158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 -- 172.21.15.28:0/2275296379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840094d0 con 0x7f3b980c7800 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9dc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002ac50 tx=0x7f3b90004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9003c830 con 0x7f3b980c7800 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9003c990 con 0x7f3b980c7800 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b90009a70 con 0x7f3b980c7800 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.611+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9003cd20 con 0x7f3b980c7800 2024-09-13T17:08:27.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.612+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b980b40c0 con 0x7f3b980c7800 2024-09-13T17:08:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.612+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9815e430 con 0x7f3b980c7800 2024-09-13T17:08:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.613+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3b90031400 con 0x7f3b980c7800 2024-09-13T17:08:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.614+0000 7f3b94c00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 0x7f3b7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.614+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b900aaf60 con 0x7f3b980c7800 2024-09-13T17:08:27.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.615+0000 7f3b9d200640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 0x7f3b7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:27.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.615+0000 7f3b9d200640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 0x7f3b7806d0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c007fd0 tx=0x7f3b8c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:27.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.618+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3b900761e0 con 0x7f3b980c7800 2024-09-13T17:08:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.831+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3b9806cd80 con 0x7f3b980c7800 2024-09-13T17:08:27.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.831+0000 7f3b94c00640 1 -- 172.21.15.28:0/2275296379 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3b90079800 con 0x7f3b980c7800 2024-09-13T17:08:27.833 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.833+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 msgr2=0x7f3b7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.833+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 0x7f3b7806d0c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c007fd0 tx=0x7f3b8c007de0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.833+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 msgr2=0x7f3b981596c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.833+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002ac50 tx=0x7f3b90004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 shutdown_connections 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b7806ac00 0x7f3b7806d0c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3b980c7800 0x7f3b981596c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3b9815f810 0x7f3b980b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 --2- 172.21.15.28:0/2275296379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b9806e500 0x7f3b98158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:27.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 >> 172.21.15.28:0/2275296379 conn(0x7f3b980b22c0 msgr2=0x7f3b980b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 shutdown_connections 2024-09-13T17:08:27.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:27.834+0000 7f3b9fc00640 1 -- 172.21.15.28:0/2275296379 wait complete. 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T17:08:27.850 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:08:27.851 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:08:27.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.851 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:08:27.885 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:27.885 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:08:27.886 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42673sec preferred_lft 42673sec 2024-09-13T17:08:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:08:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:27.887 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:27.887 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:08:27.888 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T17:08:27.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-09-13T17:08:27.888 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T17:08:27.889 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:28.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:28.250 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:08:28.307 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:08:28.310 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:28.310 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:08:28.355 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:08:28.356 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:08:28.356 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:08:28.356 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:08:28.356 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:08:28.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:28.356 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:28.356 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:28.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:08:28.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:08:28.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:08:28.357 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:28.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:28.937 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:08:28.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:28.973 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-13T17:08:29.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:29.048 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:29.048 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:29.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:08:29.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-13T17:08:29.049 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:29.177 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-13T17:08:29.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-13T17:08:29.178 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:29.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:29.370 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:29.370 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:29.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:08:29.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T17:08:29.371 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:29.486 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:29.487 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:08:29.522 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:08:29.522 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:29.522 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:08:29.596 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:08:29.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:29.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:29.741 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:29.741 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:30.367 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:30.367 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:30.368 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:08:30.368 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:08:30.368 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:08:30.369 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:08:30.370 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:08:30.370 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:08:30.370 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:08:30.370 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:08:30.372 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:08:30.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:30.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:08:30.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:30.430 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:30.478 INFO:teuthology.orchestra.run.smithi028.stdout:client4677 2024-09-13T17:08:30.479 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:30.479 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:30.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:30.514 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:30.605 INFO:teuthology.orchestra.run.smithi028.stdout:client4677 2024-09-13T17:08:30.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:30.605 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:30.657 INFO:teuthology.orchestra.run.smithi028.stdout:client4677 2024-09-13T17:08:30.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:30.658 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/99459b14-8688-430f-9c7e-baba0c3bf859.client4677/status') 2024-09-13T17:08:30.720 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4677 (3)192.168.0.1:0/1877417967 2024-09-13T17:08:30.720 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-13T17:08:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-13T17:08:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-13T17:08:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:69 bytes copied, 5.7807e-05 s, 1.2 MB/s 2024-09-13T17:08:30.724 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-09-13T17:08:30.724 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T17:08:30.724 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T17:08:30.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:08:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.879+0000 7f924fa00640 1 Processor -- start 2024-09-13T17:08:30.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.879+0000 7f924fa00640 1 -- start start 2024-09-13T17:08:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.880+0000 7f924fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.880+0000 7f924fa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9248070cc0 0x7f9248071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.880+0000 7f924fa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f924815e100 con 0x7f924815a540 2024-09-13T17:08:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f924fa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92480716b0 con 0x7f9248070cc0 2024-09-13T17:08:30.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f924fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92480adb00 con 0x7f92480cabc0 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f923fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f924da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9248070cc0 0x7f9248071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f923fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:44274/0 (socket says 172.21.15.28:44274) 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f924da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9248070cc0 0x7f9248071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:53208/0 (socket says 172.21.15.28:53208) 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f923fe00640 1 -- 172.21.15.28:0/2864362980 learned_addr learned my addr 172.21.15.28:0/2864362980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.881+0000 7f924d000640 1 --1- 172.21.15.28:0/2864362980 >> v1:172.21.15.189:6789/0 conn(0x7f924815a540 0x7f924815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:55512/0 (socket says 172.21.15.28:55512) 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f924da00640 1 -- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 msgr2=0x7f924806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:30.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f924da00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f924da00640 1 -- 172.21.15.28:0/2864362980 >> v1:172.21.15.189:6789/0 conn(0x7f924815a540 legacy=0x7f924815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f924da00640 1 -- 172.21.15.28:0/2864362980 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f924815e3e0 con 0x7f9248070cc0 2024-09-13T17:08:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f9247e00640 1 -- 172.21.15.28:0/2864362980 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227782428 0 0) 0x7f924815e100 con 0x7f924815a540 2024-09-13T17:08:30.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.882+0000 7f923fe00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.883+0000 7f924da00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9248070cc0 0x7f9248071170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9240009dc0 tx=0x7f924002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfa72fd533d18a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.883+0000 7f9247e00640 1 -- 172.21.15.28:0/2864362980 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240007740 con 0x7f9248070cc0 2024-09-13T17:08:30.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.883+0000 7f9247e00640 1 -- 172.21.15.28:0/2864362980 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924002d050 con 0x7f9248070cc0 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.883+0000 7f9247e00640 1 -- 172.21.15.28:0/2864362980 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240007ac0 con 0x7f9248070cc0 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 -- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9248070cc0 msgr2=0x7f9248071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9248070cc0 0x7f9248071170 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9240009dc0 tx=0x7f924002a5f0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 -- 172.21.15.28:0/2864362980 shutdown_connections 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9248070cc0 0x7f9248071170 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 --2- 172.21.15.28:0/2864362980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 -- 172.21.15.28:0/2864362980 >> 172.21.15.28:0/2864362980 conn(0x7f9248069e80 msgr2=0x7f924806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 -- 172.21.15.28:0/2864362980 shutdown_connections 2024-09-13T17:08:30.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.884+0000 7f924fa00640 1 -- 172.21.15.28:0/2864362980 wait complete. 2024-09-13T17:08:30.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.886+0000 7f924fa00640 1 Processor -- start 2024-09-13T17:08:30.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.886+0000 7f924fa00640 1 -- start start 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924815b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 0x7f9248155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481f2d90 con 0x7f9248070cc0 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92481f2fd0 con 0x7f924815a540 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f924815da40 con 0x7f92480cabc0 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59766/0 (socket says 172.21.15.28:59766) 2024-09-13T17:08:30.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 -- 172.21.15.28:0/256859565 learned_addr learned my addr 172.21.15.28:0/256859565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f923fe00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924815b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924da00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 0x7f9248155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 -- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 msgr2=0x7f924815b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924815b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 -- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 msgr2=0x7f9248155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 0x7f9248155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924d000640 1 -- 172.21.15.28:0/256859565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92300094d0 con 0x7f9248070cc0 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f923fe00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:30.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.887+0000 7f924da00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 0x7f9248155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f924d000640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f923800c430 tx=0x7f923800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92380074e0 con 0x7f9248070cc0 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9238007660 con 0x7f9248070cc0 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238004a20 con 0x7f9248070cc0 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9240009a70 con 0x7f9248070cc0 2024-09-13T17:08:30.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.888+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92481f22b0 con 0x7f9248070cc0 2024-09-13T17:08:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.889+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92480cbeb0 con 0x7f9248070cc0 2024-09-13T17:08:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.890+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9238004b80 con 0x7f9248070cc0 2024-09-13T17:08:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.891+0000 7f9245600640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 0x7f922406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.891+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9238089290 con 0x7f9248070cc0 2024-09-13T17:08:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.892+0000 7f923fe00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 0x7f922406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.893+0000 7f923fe00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 0x7f922406d0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92480aeba0 tx=0x7f9230008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:30.896+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f92380543d0 con 0x7f9248070cc0 2024-09-13T17:08:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.112+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9248071170 con 0x7f9248070cc0 2024-09-13T17:08:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.112+0000 7f9245600640 1 -- 172.21.15.28:0/256859565 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f92380579f0 con 0x7f9248070cc0 2024-09-13T17:08:31.114 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 msgr2=0x7f922406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 0x7f922406d0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f92480aeba0 tx=0x7f9230008040 comp rx=0 tx=0).stop 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 msgr2=0x7f924815b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f923800c430 tx=0x7f923800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 shutdown_connections 2024-09-13T17:08:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f922406ac00 0x7f922406d0c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f924815a540 0x7f9248155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92480cabc0 0x7f924815b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 --2- 172.21.15.28:0/256859565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9248070cc0 0x7f924815b0c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 >> 172.21.15.28:0/256859565 conn(0x7f9248069e80 msgr2=0x7f92480ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 shutdown_connections 2024-09-13T17:08:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:31.115+0000 7f924fa00640 1 -- 172.21.15.28:0/256859565 wait complete. 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:08:31.131 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-13T17:08:31.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.131 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:08:31.167 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:08:31.168 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42670sec preferred_lft 42670sec 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.169 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.170 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:08:31.262 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:08:31.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.265 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:08:31.301 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:08:31.302 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:08:31.302 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:08:31.302 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:08:31.302 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:08:31.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.302 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:31.302 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:31.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:08:31.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:08:31.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:08:31.303 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:31.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.448 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:08:31.483 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:08:31.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.483 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-09-13T17:08:31.558 INFO:teuthology.orchestra.run.smithi028.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-13T17:08:31.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.558 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:31.558 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:31.558 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:08:31.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-13T17:08:31.559 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:31.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether f2:36:01:0c:5a:1c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T17:08:31.738 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.739 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::f036:1ff:fe0c:5a1c/64 scope link 2024-09-13T17:08:31.739 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:31.796 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether f2:36:01:0c:5a:1c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T17:08:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T17:08:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::f036:1ff:fe0c:5a1c/64 scope link 2024-09-13T17:08:31.797 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:31.799 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-13T17:08:31.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.799 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:31.799 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-13T17:08:31.800 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:31.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:31.966 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:08:31.966 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:08:31.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.1 up 2024-09-13T17:08:31.966 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T17:08:31.966 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:08:32.081 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:32.081 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-13T17:08:32.116 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-13T17:08:32.116 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:32.117 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-13T17:08:32.192 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-09-13T17:08:32.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.192 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:32.319 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:32.320 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:32.347 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:32.347 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:32.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:08:32.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 2@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:08:32.348 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:08:32.350 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-09-13T17:08:32.350 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:08:32.350 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:08:32.350 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:08:32.350 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:08:32.352 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:08:32.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.352 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-13T17:08:32.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.409 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:32.447 INFO:teuthology.orchestra.run.smithi028.stdout:client4514 2024-09-13T17:08:32.447 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:32.447 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:32.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.523 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:32.600 INFO:teuthology.orchestra.run.smithi028.stdout:client4514 2024-09-13T17:08:32.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.601 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:32.679 INFO:teuthology.orchestra.run.smithi028.stdout:client4514 2024-09-13T17:08:32.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:32.680 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/99459b14-8688-430f-9c7e-baba0c3bf859.client4514/status') 2024-09-13T17:08:32.776 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.4514 (3)172.21.15.28:0/247949665 2024-09-13T17:08:32.776 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-09-13T17:08:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-09-13T17:08:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-09-13T17:08:32.776 INFO:teuthology.orchestra.run.smithi028.stderr:69 bytes copied, 7.1758e-05 s, 962 kB/s 2024-09-13T17:08:32.778 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-09-13T17:08:32.779 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T17:08:32.779 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T17:08:32.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:08:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.934+0000 7f8f67e00640 1 Processor -- start 2024-09-13T17:08:32.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.934+0000 7f8f67e00640 1 -- start start 2024-09-13T17:08:32.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8f68160e70 0x7f8f680ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f67e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6806bf50 con 0x7f8f68061110 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f67e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f68161300 con 0x7f8f6806ce80 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f67e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f680ccd70 con 0x7f8f68160e70 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f67400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8f68160e70 0x7f8f680ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f5fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8f68061110 0x7f8f6806c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37638/0 (socket says 172.21.15.28:37638) 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.935+0000 7f8f5fe00640 1 -- 172.21.15.28:0/728102511 learned_addr learned my addr 172.21.15.28:0/728102511 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.936+0000 7f8f67400640 1 -- 172.21.15.28:0/728102511 >> v1:172.21.15.189:6789/0 conn(0x7f8f6806ce80 legacy=0x7f8f6806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.936+0000 7f8f67400640 1 -- 172.21.15.28:0/728102511 >> v1:172.21.15.28:6789/0 conn(0x7f8f68061110 legacy=0x7f8f6806c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.936+0000 7f8f67400640 1 -- 172.21.15.28:0/728102511 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f680ce3d0 con 0x7f8f68160e70 2024-09-13T17:08:32.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.936+0000 7f8f66000640 1 -- 172.21.15.28:0/728102511 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415677150 0 0) 0x7f8f6806bf50 con 0x7f8f68061110 2024-09-13T17:08:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.937+0000 7f8f67400640 1 --2- 172.21.15.28:0/728102511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68160e70 0x7f8f680ad870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009d90 tx=0x7f8f6002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca28f18963aab72 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.937+0000 7f8f66000640 1 -- 172.21.15.28:0/728102511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60007740 con 0x7f8f68160e70 2024-09-13T17:08:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.937+0000 7f8f66000640 1 -- 172.21.15.28:0/728102511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6002d050 con 0x7f8f68160e70 2024-09-13T17:08:32.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.937+0000 7f8f66000640 1 -- 172.21.15.28:0/728102511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60007ac0 con 0x7f8f68160e70 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 -- 172.21.15.28:0/728102511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68160e70 msgr2=0x7f8f680ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 --2- 172.21.15.28:0/728102511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68160e70 0x7f8f680ad870 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009d90 tx=0x7f8f6002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 -- 172.21.15.28:0/728102511 shutdown_connections 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 --2- 172.21.15.28:0/728102511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68160e70 0x7f8f680ad870 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 -- 172.21.15.28:0/728102511 >> 172.21.15.28:0/728102511 conn(0x7f8f680cb6d0 msgr2=0x7f8f680cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 -- 172.21.15.28:0/728102511 shutdown_connections 2024-09-13T17:08:32.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.938+0000 7f8f67e00640 1 -- 172.21.15.28:0/728102511 wait complete. 2024-09-13T17:08:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.939+0000 7f8f67e00640 1 Processor -- start 2024-09-13T17:08:32.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.939+0000 7f8f67e00640 1 -- start start 2024-09-13T17:08:32.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 0x7f8f680b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 0x7f8f6814e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6814e5b0 con 0x7f8f68061110 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6815c520 con 0x7f8f68160e70 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.940+0000 7f8f67e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f680b21e0 con 0x7f8f6806ce80 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59788/0 (socket says 172.21.15.28:59788) 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 -- 172.21.15.28:0/1893083698 learned_addr learned my addr 172.21.15.28:0/1893083698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f67400640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 0x7f8f6814e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f5fe00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 0x7f8f680b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 -- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 msgr2=0x7f8f680b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 0x7f8f680b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:32.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 -- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 msgr2=0x7f8f6814e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 0x7f8f6814e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 -- 172.21.15.28:0/1893083698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f60009a70 con 0x7f8f68061110 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f67400640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 0x7f8f6814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f5fe00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 0x7f8f680b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.941+0000 7f8f66a00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800c450 tx=0x7f8f5800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.942+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f580052d0 con 0x7f8f68061110 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.942+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f58005430 con 0x7f8f68061110 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.942+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f680b24c0 con 0x7f8f68061110 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.942+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f58012870 con 0x7f8f68061110 2024-09-13T17:08:32.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.942+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f680b2a80 con 0x7f8f68061110 2024-09-13T17:08:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.943+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6806c3e0 con 0x7f8f68061110 2024-09-13T17:08:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.944+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8f580129d0 con 0x7f8f68061110 2024-09-13T17:08:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.945+0000 7f8f5ea00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 0x7f8f4406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:32.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.945+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f58088e20 con 0x7f8f68061110 2024-09-13T17:08:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.945+0000 7f8f5fe00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 0x7f8f4406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:32.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.946+0000 7f8f5fe00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 0x7f8f4406d0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f680c7170 tx=0x7f8f50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:32.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:32.950+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8f58053ff0 con 0x7f8f68061110 2024-09-13T17:08:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.163+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8f680ae330 con 0x7f8f68061110 2024-09-13T17:08:33.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.163+0000 7f8f5ea00640 1 -- 172.21.15.28:0/1893083698 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8f58057610 con 0x7f8f68061110 2024-09-13T17:08:33.164 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:08:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 msgr2=0x7f8f4406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 0x7f8f4406d0f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8f680c7170 tx=0x7f8f50009040 comp rx=0 tx=0).stop 2024-09-13T17:08:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 msgr2=0x7f8f680c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8f5800c450 tx=0x7f8f5800c910 comp rx=0 tx=0).stop 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 shutdown_connections 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4406ac30 0x7f8f4406d0f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f68160e70 0x7f8f6814e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6806ce80 0x7f8f680b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 --2- 172.21.15.28:0/1893083698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f68061110 0x7f8f680c8160 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 >> 172.21.15.28:0/1893083698 conn(0x7f8f680cb6d0 msgr2=0x7f8f680ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 shutdown_connections 2024-09-13T17:08:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:33.165+0000 7f8f67e00640 1 -- 172.21.15.28:0/1893083698 wait complete. 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:08:33.182 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T17:08:33.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:33.183 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-09-13T17:08:33.222 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:33.223 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42691sec preferred_lft 42691sec 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::ec4:7aff:fe88:6df1/64 scope link noprefixroute 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:33.224 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T17:08:33.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:33.224 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:33.224 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:33.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add name ceph-brx type bridge 2024-09-13T17:08:33.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr flush dev ceph-brx 2024-09-13T17:08:33.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set ceph-brx up 2024-09-13T17:08:33.225 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T17:08:33.225 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:33.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:33.594 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:08:33.655 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-09-13T17:08:33.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:33.658 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-09-13T17:08:33.705 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-09-13T17:08:33.705 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:08:33.705 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T17:08:33.705 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-13T17:08:33.705 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:08:33.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T17:08:33.706 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:34.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.229 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:08:34.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.266 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-09-13T17:08:34.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.344 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:34.344 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:34.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T17:08:34.344 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-13T17:08:34.344 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:34.470 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-13T17:08:34.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-13T17:08:34.471 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:34.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.673 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:34.674 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:34.674 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.0 up 2024-09-13T17:08:34.674 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T17:08:34.674 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:34.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:34.793 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T17:08:34.828 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T17:08:34.829 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:34.829 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T17:08:34.903 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-09-13T17:08:34.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:34.904 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:35.037 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:35.037 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:35.657 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:35.657 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:35.657 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:08:35.657 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with new device syntax: 1@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:08:35.657 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:08:35.660 INFO:teuthology.orchestra.run.smithi189.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-09-13T17:08:35.660 INFO:teuthology.orchestra.run.smithi189.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:08:35.660 INFO:teuthology.orchestra.run.smithi189.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:08:35.660 INFO:teuthology.orchestra.run.smithi189.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:08:35.660 INFO:teuthology.orchestra.run.smithi189.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:08:35.663 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:08:35.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:35.664 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T17:08:35.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:35.720 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:35.759 INFO:teuthology.orchestra.run.smithi189.stdout:client4692 2024-09-13T17:08:35.759 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:35.759 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:35.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:35.836 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:35.913 INFO:teuthology.orchestra.run.smithi189.stdout:client4692 2024-09-13T17:08:35.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:35.914 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:35.992 INFO:teuthology.orchestra.run.smithi189.stdout:client4692 2024-09-13T17:08:35.993 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:35.993 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/99459b14-8688-430f-9c7e-baba0c3bf859.client4692/status') 2024-09-13T17:08:36.088 INFO:teuthology.orchestra.run.smithi189.stdout:instance: client.4692 (3)172.21.15.189:0/2751510236 2024-09-13T17:08:36.088 INFO:teuthology.orchestra.run.smithi189.stdout:blocklisted: false 2024-09-13T17:08:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2024-09-13T17:08:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2024-09-13T17:08:36.088 INFO:teuthology.orchestra.run.smithi189.stderr:71 bytes copied, 5.9809e-05 s, 1.2 MB/s 2024-09-13T17:08:36.090 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-09-13T17:08:36.091 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T17:08:36.091 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T17:08:36.091 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:08:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.255+0000 7fd6b9000640 1 Processor -- start 2024-09-13T17:08:36.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.255+0000 7fd6b9000640 1 -- start start 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b9000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b40cabc0 0x7fd6b406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b9000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415e100 con 0x7fd6b415a540 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b9000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b415e3e0 con 0x7fd6b4070cc0 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b9000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b40adb00 con 0x7fd6b40cabc0 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6b415a540 0x7fd6b415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51474/0 (socket says 172.21.15.28:51474) 2024-09-13T17:08:36.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b1e00640 1 -- 172.21.15.28:0/4277882353 learned_addr learned my addr 172.21.15.28:0/4277882353 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6abe00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b40cabc0 0x7fd6b406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103820430 0 0) 0x7fd6b415e100 con 0x7fd6b415a540 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.257+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd698003520 con 0x7fd6b415a540 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325354761 0 0) 0x7fd6b415e3e0 con 0x7fd6b4070cc0 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6b415e100 con 0x7fd6b4070cc0 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 130895827 0 0) 0x7fd698003520 con 0x7fd6b415a540 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b40cabc0 msgr2=0x7fd6b406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b40cabc0 0x7fd6b406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 >> v1:172.21.15.189:6789/0 conn(0x7fd6b4070cc0 legacy=0x7fd6b406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b415ef80 con 0x7fd6b415a540 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fd6a4001670 con 0x7fd6b415a540 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 >> v1:172.21.15.28:6789/0 conn(0x7fd6b415a540 legacy=0x7fd6b415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd698003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd698005180 0x7fd69800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800c050 con 0x7fd698003b30 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800c190 con 0x7fd698005180 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69800c350 con 0x7fd6980047c0 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd698003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6b2800640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd698005180 0x7fd69800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.258+0000 7fd6abe00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 -- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6980047c0 msgr2=0x7fd698004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:36.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 -- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd698005180 msgr2=0x7fd69800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd698005180 0x7fd69800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 -- 172.21.15.28:0/4277882353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd698003520 con 0x7fd698003b30 2024-09-13T17:08:36.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6abe00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6980047c0 0x7fd698004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:36.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.259+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd698003f30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4007f90 tx=0x7fd6980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dc97eb0f813f4d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:36.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4005ad0 con 0x7fd698003b30 2024-09-13T17:08:36.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a402d070 con 0x7fd698003b30 2024-09-13T17:08:36.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b1400640 1 -- 172.21.15.28:0/4277882353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4005e90 con 0x7fd698003b30 2024-09-13T17:08:36.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 -- 172.21.15.28:0/4277882353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 msgr2=0x7fd698003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 --2- 172.21.15.28:0/4277882353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd698003f30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4007f90 tx=0x7fd6980057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b2800640 1 -- 172.21.15.28:0/4277882353 reap_dead start 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 -- 172.21.15.28:0/4277882353 shutdown_connections 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 -- 172.21.15.28:0/4277882353 >> 172.21.15.28:0/4277882353 conn(0x7fd6b4069e80 msgr2=0x7fd6b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 -- 172.21.15.28:0/4277882353 shutdown_connections 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.260+0000 7fd6b9000640 1 -- 172.21.15.28:0/4277882353 wait complete. 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.262+0000 7fd6b9000640 1 Processor -- start 2024-09-13T17:08:36.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.262+0000 7fd6b9000640 1 -- start start 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6b414f010 0x7fd6b4141d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b414fa40 0x7fd6b414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4150400 con 0x7fd698003b30 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b415b660 con 0x7fd6b414f010 2024-09-13T17:08:36.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b9000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b414f790 con 0x7fd6b414fa40 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39336/0 (socket says 172.21.15.28:39336) 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.263+0000 7fd6b1e00640 1 -- 172.21.15.28:0/1301738638 learned_addr learned my addr 172.21.15.28:0/1301738638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6abe00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6b414f010 0x7fd6b4141d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b2800640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b414fa40 0x7fd6b414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 -- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b414fa40 msgr2=0x7fd6b414fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b414fa40 0x7fd6b414fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 -- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6b414f010 msgr2=0x7fd6b4141d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6b414f010 0x7fd6b4141d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 -- 172.21.15.28:0/1301738638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b41540d0 con 0x7fd698003b30 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b1e00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c1d0 tx=0x7fd6a4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4004bf0 con 0x7fd698003b30 2024-09-13T17:08:36.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a4004d50 con 0x7fd698003b30 2024-09-13T17:08:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a4007270 con 0x7fd698003b30 2024-09-13T17:08:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.264+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b41543b0 con 0x7fd698003b30 2024-09-13T17:08:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.265+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b4154970 con 0x7fd698003b30 2024-09-13T17:08:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.266+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b406c430 con 0x7fd698003b30 2024-09-13T17:08:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.267+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6a402f030 con 0x7fd698003b30 2024-09-13T17:08:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.268+0000 7fd6aa000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 0x7fd69006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.268+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6a40a9050 con 0x7fd698003b30 2024-09-13T17:08:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.268+0000 7fd6abe00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 0x7fd69006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.269+0000 7fd6abe00640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 0x7fd69006d0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4152f70 tx=0x7fd69c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.272+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6a4074310 con 0x7fd698003b30 2024-09-13T17:08:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.489+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd6b406c430 con 0x7fd698003b30 2024-09-13T17:08:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.490+0000 7fd6aa000640 1 -- 172.21.15.28:0/1301738638 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd6a4077930 con 0x7fd698003b30 2024-09-13T17:08:36.492 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 msgr2=0x7fd69006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 0x7fd69006d0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4152f70 tx=0x7fd69c008040 comp rx=0 tx=0).stop 2024-09-13T17:08:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 msgr2=0x7fd6b4153a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd6a400c1d0 tx=0x7fd6a4001930 comp rx=0 tx=0).stop 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 shutdown_connections 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd69006ac00 0x7fd69006d0c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6b414fa40 0x7fd6b414fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6b414f010 0x7fd6b4141d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 --2- 172.21.15.28:0/1301738638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd698003b30 0x7fd6b4153a40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 >> 172.21.15.28:0/1301738638 conn(0x7fd6b4069e80 msgr2=0x7fd6b40cb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 shutdown_connections 2024-09-13T17:08:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:36.492+0000 7fd6b9000640 1 -- 172.21.15.28:0/1301738638 wait complete. 2024-09-13T17:08:36.509 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T17:08:36.509 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi189.front.sepia.ceph.com 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:08:36.510 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-13T17:08:36.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.510 DEBUG:teuthology.orchestra.run.smithi189:> ip addr 2024-09-13T17:08:36.546 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:36.546 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:36.546 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:36.546 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:18 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f0 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:d9:3b:19 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: altname enp7s0f1 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f0 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 0c:c4:7a:88:6d:f1 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: inet 172.21.15.189/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-09-13T17:08:36.547 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft 42687sec preferred_lft 42687sec 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::ec4:7aff:fe88:6df1/64 scope link noprefixroute 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 06:d0:e0:a0:0f:12 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::d48f:58ff:fe75:98a/64 scope link 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether 06:d0:e0:a0:0f:12 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::4d0:e0ff:fea0:f12/64 scope link 2024-09-13T17:08:36.548 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:36.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.549 DEBUG:teuthology.orchestra.run.smithi189:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:08:36.641 INFO:teuthology.orchestra.run.smithi189.stdout:1 2024-09-13T17:08:36.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.645 DEBUG:teuthology.orchestra.run.smithi189:> route 2024-09-13T17:08:36.681 INFO:teuthology.orchestra.run.smithi189.stdout:Kernel IP routing table 2024-09-13T17:08:36.681 INFO:teuthology.orchestra.run.smithi189.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:08:36.682 INFO:teuthology.orchestra.run.smithi189.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T17:08:36.682 INFO:teuthology.orchestra.run.smithi189.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-09-13T17:08:36.682 INFO:teuthology.orchestra.run.smithi189.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:08:36.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T17:08:36.682 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:36.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.828 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:08:36.863 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T17:08:36.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.864 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list-id 2024-09-13T17:08:36.938 INFO:teuthology.orchestra.run.smithi189.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-13T17:08:36.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:36.939 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:36.939 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:36.939 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T17:08:36.939 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-13T17:08:36.939 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:37.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.064 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:37.119 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether ba:0e:55:06:06:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T17:08:37.120 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T17:08:37.120 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.120 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::b80e:55ff:fe06:6eb/64 scope link 2024-09-13T17:08:37.120 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T17:08:37.177 INFO:teuthology.orchestra.run.smithi189.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:08:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:08:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:08:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.177 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 ::1/128 scope host 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: link/ether ba:0e:55:06:06:eb brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: inet6 fe80::b80e:55ff:fe06:6eb/64 scope link 2024-09-13T17:08:37.178 INFO:teuthology.orchestra.run.smithi189.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:08:37.180 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-13T17:08:37.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.180 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-13T17:08:37.181 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:37.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.346 DEBUG:teuthology.orchestra.run.smithi189:> (cd / && exec bash -c ' 2024-09-13T17:08:37.346 DEBUG:teuthology.orchestra.run.smithi189:> set -e 2024-09-13T17:08:37.346 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set brx.1 up 2024-09-13T17:08:37.346 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T17:08:37.346 DEBUG:teuthology.orchestra.run.smithi189:> ') 2024-09-13T17:08:37.459 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:37.460 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-13T17:08:37.495 INFO:teuthology.orchestra.run.smithi189.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-13T17:08:37.495 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:08:37.495 DEBUG:teuthology.orchestra.run.smithi189:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-13T17:08:37.569 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-09-13T17:08:37.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.569 DEBUG:teuthology.orchestra.run.smithi189:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:37.697 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:37.697 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:37.724 INFO:teuthology.orchestra.run.smithi189.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:08:37.724 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:08:37.724 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:08:37.724 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: trying mount with new device syntax: 3@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:08:37.724 INFO:teuthology.orchestra.run.smithi189.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:08:37.726 INFO:teuthology.orchestra.run.smithi189.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-09-13T17:08:37.726 INFO:teuthology.orchestra.run.smithi189.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:08:37.726 INFO:teuthology.orchestra.run.smithi189.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:08:37.726 INFO:teuthology.orchestra.run.smithi189.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:08:37.726 INFO:teuthology.orchestra.run.smithi189.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:08:37.729 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:08:37.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.729 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-13T17:08:37.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.785 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:37.823 INFO:teuthology.orchestra.run.smithi189.stdout:client4701 2024-09-13T17:08:37.824 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:37.824 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:37.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.899 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:37.977 INFO:teuthology.orchestra.run.smithi189.stdout:client4701 2024-09-13T17:08:37.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:37.977 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:08:38.055 INFO:teuthology.orchestra.run.smithi189.stdout:client4701 2024-09-13T17:08:38.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:38.055 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/99459b14-8688-430f-9c7e-baba0c3bf859.client4701/status') 2024-09-13T17:08:38.150 INFO:teuthology.orchestra.run.smithi189.stdout:instance: client.4701 (3)172.21.15.189:0/1667933243 2024-09-13T17:08:38.150 INFO:teuthology.orchestra.run.smithi189.stdout:blocklisted: false 2024-09-13T17:08:38.151 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records in 2024-09-13T17:08:38.151 INFO:teuthology.orchestra.run.smithi189.stderr:0+1 records out 2024-09-13T17:08:38.151 INFO:teuthology.orchestra.run.smithi189.stderr:71 bytes copied, 5.3005e-05 s, 1.3 MB/s 2024-09-13T17:08:38.154 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-13T17:08:38.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.335+0000 7fe366200640 1 Processor -- start 2024-09-13T17:08:38.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.335+0000 7fe366200640 1 -- start start 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe366200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 0x7fe360160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe366200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 0x7fe36006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe366200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3600cf230 con 0x7fe36006b3d0 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe366200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36006b860 con 0x7fe3600cb760 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe366200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600c4af0 con 0x7fe3600c4380 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe365800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe36006b3d0 0x7fe360072e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51486/0 (socket says 172.21.15.28:51486) 2024-09-13T17:08:38.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe365800640 1 -- 172.21.15.28:0/3908076957 learned_addr learned my addr 172.21.15.28:0/3908076957 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe35fe00640 1 --2- 172.21.15.28:0/3908076957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 0x7fe36006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.336+0000 7fe364e00640 1 --2- 172.21.15.28:0/3908076957 >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 0x7fe360160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35f400640 1 -- 172.21.15.28:0/3908076957 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260829469 0 0) 0x7fe3600cf230 con 0x7fe36006b3d0 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35f400640 1 -- 172.21.15.28:0/3908076957 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe348003520 con 0x7fe36006b3d0 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35fe00640 1 -- 172.21.15.28:0/3908076957 >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 msgr2=0x7fe360160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:38.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35fe00640 1 --2- 172.21.15.28:0/3908076957 >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 0x7fe360160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35fe00640 1 -- 172.21.15.28:0/3908076957 >> v1:172.21.15.28:6789/0 conn(0x7fe36006b3d0 legacy=0x7fe360072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe35fe00640 1 -- 172.21.15.28:0/3908076957 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3600cd460 con 0x7fe3600c4380 2024-09-13T17:08:38.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.337+0000 7fe364e00640 1 --2- 172.21.15.28:0/3908076957 >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 0x7fe360160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.338+0000 7fe35fe00640 1 --2- 172.21.15.28:0/3908076957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 0x7fe36006ae90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097a0 tx=0x7fe34c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae45347dca1be4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.338+0000 7fe35f400640 1 -- 172.21.15.28:0/3908076957 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c02ae50 con 0x7fe3600c4380 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.338+0000 7fe35f400640 1 -- 172.21.15.28:0/3908076957 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c032440 con 0x7fe3600c4380 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.338+0000 7fe35f400640 1 -- 172.21.15.28:0/3908076957 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0327d0 con 0x7fe3600c4380 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 -- 172.21.15.28:0/3908076957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 msgr2=0x7fe36006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 --2- 172.21.15.28:0/3908076957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 0x7fe36006ae90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe34c0097a0 tx=0x7fe34c02a100 comp rx=0 tx=0).stop 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 -- 172.21.15.28:0/3908076957 shutdown_connections 2024-09-13T17:08:38.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 --2- 172.21.15.28:0/3908076957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3600c4380 0x7fe36006ae90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 --2- 172.21.15.28:0/3908076957 >> v2:172.21.15.189:3300/0 conn(0x7fe3600cb760 0x7fe360160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 -- 172.21.15.28:0/3908076957 >> 172.21.15.28:0/3908076957 conn(0x7fe3600703a0 msgr2=0x7fe3600707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 -- 172.21.15.28:0/3908076957 shutdown_connections 2024-09-13T17:08:38.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.339+0000 7fe366200640 1 -- 172.21.15.28:0/3908076957 wait complete. 2024-09-13T17:08:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.341+0000 7fe366200640 1 Processor -- start 2024-09-13T17:08:38.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.341+0000 7fe366200640 1 -- start start 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe36006b3d0 0x7fe36015c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3600cb760 0x7fe36015fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3600c79b0 con 0x7fe3600c4380 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36015d8f0 con 0x7fe3600cb760 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe366200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36015e3e0 con 0x7fe36006b3d0 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe35fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe35fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39346/0 (socket says 172.21.15.28:39346) 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.342+0000 7fe35fe00640 1 -- 172.21.15.28:0/246117156 learned_addr learned my addr 172.21.15.28:0/246117156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe365800640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3600cb760 0x7fe36015fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe364e00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe36006b3d0 0x7fe36015c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 -- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe36006b3d0 msgr2=0x7fe36015c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe36006b3d0 0x7fe36015c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 -- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3600cb760 msgr2=0x7fe36015fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3600cb760 0x7fe36015fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 -- 172.21.15.28:0/246117156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe354009400 con 0x7fe3600c4380 2024-09-13T17:08:38.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35fe00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe34c035040 tx=0x7fe34c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.343+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c03d710 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.344+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c03d890 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.344+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c03dc40 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.344+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c009400 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.344+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36015de20 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.345+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3600c63b0 con 0x7fe3600c4380 2024-09-13T17:08:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.346+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe34c039040 con 0x7fe3600c4380 2024-09-13T17:08:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.347+0000 7fe35cc00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 0x7fe34006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.347+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe34c0a9c10 con 0x7fe3600c4380 2024-09-13T17:08:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.347+0000 7fe364e00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 0x7fe34006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.348+0000 7fe364e00640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 0x7fe34006d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe354009bc0 tx=0x7fe354008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.351+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe34c074d50 con 0x7fe3600c4380 2024-09-13T17:08:38.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.533+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3600c9b10 con 0x7fe3600c4380 2024-09-13T17:08:38.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.534+0000 7fe35cc00640 1 -- 172.21.15.28:0/246117156 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fe34c078370 con 0x7fe3600c4380 2024-09-13T17:08:38.536 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:38.536 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 msgr2=0x7fe34006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 0x7fe34006d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe354009bc0 tx=0x7fe354008040 comp rx=0 tx=0).stop 2024-09-13T17:08:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 msgr2=0x7fe36015f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe34c035040 tx=0x7fe34c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 shutdown_connections 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe34006ac00 0x7fe34006d0c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3600cb760 0x7fe36015fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3600c4380 0x7fe36015f540 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 --2- 172.21.15.28:0/246117156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe36006b3d0 0x7fe36015c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 >> 172.21.15.28:0/246117156 conn(0x7fe3600703a0 msgr2=0x7fe36006e0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 shutdown_connections 2024-09-13T17:08:38.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.536+0000 7fe366200640 1 -- 172.21.15.28:0/246117156 wait complete. 2024-09-13T17:08:38.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:38.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:38.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.711+0000 7f55e0a00640 1 Processor -- start 2024-09-13T17:08:38.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.712+0000 7f55e0a00640 1 -- start start 2024-09-13T17:08:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55e0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55dc0cc4a0 0x7f55dc0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55e0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 0x7f55dc0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55e0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0722e0 con 0x7f55dc0b0c30 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55e0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0cb3e0 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55e0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0cbdb0 con 0x7f55dc0cc880 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55da800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f55dc0cc4a0 0x7f55dc0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55db200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f55dc0b0c30 0x7f55dc0712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51498/0 (socket says 172.21.15.28:51498) 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.713+0000 7f55db200640 1 -- 172.21.15.28:0/3683781871 learned_addr learned my addr 172.21.15.28:0/3683781871 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55d1e00640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 0x7f55dc0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55da800640 1 -- 172.21.15.28:0/3683781871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 msgr2=0x7f55dc0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55da800640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 0x7f55dc0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55da800640 1 -- 172.21.15.28:0/3683781871 >> v1:172.21.15.28:6789/0 conn(0x7f55dc0b0c30 legacy=0x7f55dc0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55da800640 1 -- 172.21.15.28:0/3683781871 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0b1080 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55d1e00640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 0x7f55dc0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.714+0000 7f55d9e00640 1 -- 172.21.15.28:0/3683781871 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828187488 0 0) 0x7f55dc0722e0 con 0x7f55dc0b0c30 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.715+0000 7f55da800640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc0cb870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0097a0 tx=0x7f55cc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e301e10f94b9b7f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.715+0000 7f55d9e00640 1 -- 172.21.15.28:0/3683781871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc02ae50 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.715+0000 7f55d9e00640 1 -- 172.21.15.28:0/3683781871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc032440 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.715+0000 7f55d9e00640 1 -- 172.21.15.28:0/3683781871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0327d0 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 -- 172.21.15.28:0/3683781871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 msgr2=0x7f55dc0cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc0cb870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f55cc0097a0 tx=0x7f55cc02a100 comp rx=0 tx=0).stop 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 -- 172.21.15.28:0/3683781871 shutdown_connections 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0cc880 0x7f55dc0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 --2- 172.21.15.28:0/3683781871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc0cb870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 -- 172.21.15.28:0/3683781871 >> 172.21.15.28:0/3683781871 conn(0x7f55dc069e80 msgr2=0x7f55dc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:38.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 -- 172.21.15.28:0/3683781871 shutdown_connections 2024-09-13T17:08:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.716+0000 7f55e0a00640 1 -- 172.21.15.28:0/3683781871 wait complete. 2024-09-13T17:08:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.717+0000 7f55e0a00640 1 Processor -- start 2024-09-13T17:08:38.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 -- start start 2024-09-13T17:08:38.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0b0c30 0x7f55dc1f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55dc0cc880 0x7f55dc1f4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc154550 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc155140 con 0x7f55dc0cc880 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.718+0000 7f55e0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc1f3100 con 0x7f55dc0b0c30 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39358/0 (socket says 172.21.15.28:39358) 2024-09-13T17:08:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55da800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0b0c30 0x7f55dc1f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55dc0cc880 0x7f55dc1f4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 -- 172.21.15.28:0/572006683 learned_addr learned my addr 172.21.15.28:0/572006683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:38.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 -- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0b0c30 msgr2=0x7f55dc1f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0b0c30 0x7f55dc1f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 -- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55dc0cc880 msgr2=0x7f55dc1f4e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55dc0cc880 0x7f55dc1f4e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.719+0000 7f55d1e00640 1 -- 172.21.15.28:0/572006683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55cc009400 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55d1e00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c820 tx=0x7f55c400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c4007a30 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c4007b90 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c400ce10 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc1539d0 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.720+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc153f90 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.721+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc14c9d0 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.722+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f55c40113f0 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.723+0000 7f55d3400640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 0x7f55b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.723+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55c4089390 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.723+0000 7f55da800640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 0x7f55b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.724+0000 7f55da800640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 0x7f55b806d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f55cc034000 tx=0x7f55cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.727+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f55c4054610 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.827 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:08:38.832 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:08:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.908+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55dc0cb850 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.909+0000 7f55d3400640 1 -- 172.21.15.28:0/572006683 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f55c4057c30 con 0x7f55dc0cc4a0 2024-09-13T17:08:38.911 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 msgr2=0x7f55b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 0x7f55b806d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f55cc034000 tx=0x7f55cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 msgr2=0x7f55dc1f2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f55c400c820 tx=0x7f55c400cce0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 shutdown_connections 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55b806ac00 0x7f55b806d0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55dc0cc880 0x7f55dc1f4e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55dc0cc4a0 0x7f55dc1f2bc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 --2- 172.21.15.28:0/572006683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55dc0b0c30 0x7f55dc1f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 >> 172.21.15.28:0/572006683 conn(0x7f55dc069e80 msgr2=0x7f55dc15e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 shutdown_connections 2024-09-13T17:08:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:38.911+0000 7f55e0a00640 1 -- 172.21.15.28:0/572006683 wait complete. 2024-09-13T17:08:38.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.126+0000 7f0c2fa00640 1 Processor -- start 2024-09-13T17:08:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.126+0000 7f0c2fa00640 1 -- start start 2024-09-13T17:08:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2815a780 0x7f0c28070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0c2806c250 0x7f0c280712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2fa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c2806c8b0 con 0x7f0c2815ab60 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c2806c6a0 con 0x7f0c2806c250 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c2806cb90 con 0x7f0c2815a780 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c27e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0c2815ab60 0x7f0c2806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51508/0 (socket says 172.21.15.28:51508) 2024-09-13T17:08:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c27e00640 1 -- 172.21.15.28:0/2244557581 learned_addr learned my addr 172.21.15.28:0/2244557581 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2da00640 1 --2- 172.21.15.28:0/2244557581 >> v2:172.21.15.28:3300/0 conn(0x7f0c2806c250 0x7f0c280712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2d000640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2815a780 0x7f0c28070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c27400640 1 -- 172.21.15.28:0/2244557581 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329807374 0 0) 0x7f0c2806c8b0 con 0x7f0c2815ab60 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c27400640 1 -- 172.21.15.28:0/2244557581 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c10003520 con 0x7f0c2815ab60 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2da00640 1 -- 172.21.15.28:0/2244557581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2815a780 msgr2=0x7f0c28070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2da00640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2815a780 0x7f0c28070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2da00640 1 -- 172.21.15.28:0/2244557581 >> v1:172.21.15.28:6789/0 conn(0x7f0c2815ab60 legacy=0x7f0c2806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.128+0000 7f0c2da00640 1 -- 172.21.15.28:0/2244557581 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280cc980 con 0x7f0c2806c250 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.129+0000 7f0c2da00640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2806c250 0x7f0c280712c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009d90 tx=0x7f0c2002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06b6e4ef88311fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c27400640 1 -- 172.21.15.28:0/2244557581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20007740 con 0x7f0c2806c250 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c27400640 1 -- 172.21.15.28:0/2244557581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2002d050 con 0x7f0c2806c250 2024-09-13T17:08:39.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c27400640 1 -- 172.21.15.28:0/2244557581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c20007ac0 con 0x7f0c2806c250 2024-09-13T17:08:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 -- 172.21.15.28:0/2244557581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2806c250 msgr2=0x7f0c280712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2806c250 0x7f0c280712c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009d90 tx=0x7f0c2002a660 comp rx=0 tx=0).stop 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 -- 172.21.15.28:0/2244557581 shutdown_connections 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2806c250 0x7f0c280712c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/2244557581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2815a780 0x7f0c28070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 -- 172.21.15.28:0/2244557581 >> 172.21.15.28:0/2244557581 conn(0x7f0c280b0270 msgr2=0x7f0c280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 -- 172.21.15.28:0/2244557581 shutdown_connections 2024-09-13T17:08:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.130+0000 7f0c2fa00640 1 -- 172.21.15.28:0/2244557581 wait complete. 2024-09-13T17:08:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.132+0000 7f0c2fa00640 1 Processor -- start 2024-09-13T17:08:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.132+0000 7f0c2fa00640 1 -- start start 2024-09-13T17:08:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2806c250 0x7f0c28070870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2815a780 0x7f0c281f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28154b00 con 0x7f0c2815ab60 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c281f5b20 con 0x7f0c2815a780 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28153e80 con 0x7f0c2806c250 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.133+0000 7f0c2da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39376/0 (socket says 172.21.15.28:39376) 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 -- 172.21.15.28:0/1284914375 learned_addr learned my addr 172.21.15.28:0/1284914375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2d000640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2806c250 0x7f0c28070870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c27e00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2815a780 0x7f0c281f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 -- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2806c250 msgr2=0x7f0c28070870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2806c250 0x7f0c28070870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 -- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2815a780 msgr2=0x7f0c281f50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2815a780 0x7f0c281f50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 -- 172.21.15.28:0/1284914375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c20009a70 con 0x7f0c2815ab60 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.134+0000 7f0c2da00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009eb0 tx=0x7f0c20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.135+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2003c830 con 0x7f0c2815ab60 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.135+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2003c990 con 0x7f0c2815ab60 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.135+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2003cd00 con 0x7f0c2815ab60 2024-09-13T17:08:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.135+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c28154160 con 0x7f0c2815ab60 2024-09-13T17:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.135+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c28154720 con 0x7f0c2815ab60 2024-09-13T17:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.136+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c28154c40 con 0x7f0c2815ab60 2024-09-13T17:08:39.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.137+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0c20031400 con 0x7f0c2815ab60 2024-09-13T17:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.137+0000 7f0c24c00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 0x7f0c0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.138+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c200aa830 con 0x7f0c2815ab60 2024-09-13T17:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.138+0000 7f0c2d000640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 0x7f0c0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.139+0000 7f0c2d000640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 0x7f0c0806d0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009b40 tx=0x7f0c1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.142+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0c28154c40 con 0x7f0c2815ab60 2024-09-13T17:08:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.319+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0c28154c40 con 0x7f0c2815ab60 2024-09-13T17:08:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.320+0000 7f0c24c00640 1 -- 172.21.15.28:0/1284914375 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0c20040020 con 0x7f0c2815ab60 2024-09-13T17:08:39.322 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 msgr2=0x7f0c0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 0x7f0c0806d0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009b40 tx=0x7f0c1c009190 comp rx=0 tx=0).stop 2024-09-13T17:08:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 msgr2=0x7f0c281f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0c20009eb0 tx=0x7f0c20004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 shutdown_connections 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c0806ac00 0x7f0c0806d0c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c2815ab60 0x7f0c281f55e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c2815a780 0x7f0c281f50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 --2- 172.21.15.28:0/1284914375 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c2806c250 0x7f0c28070870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 >> 172.21.15.28:0/1284914375 conn(0x7f0c280b0270 msgr2=0x7f0c280ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 shutdown_connections 2024-09-13T17:08:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.322+0000 7f0c2fa00640 1 -- 172.21.15.28:0/1284914375 wait complete. 2024-09-13T17:08:39.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.496+0000 7f98dde00640 1 Processor -- start 2024-09-13T17:08:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.497+0000 7f98dde00640 1 -- start start 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98d8143f70 0x7f98d80cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dde00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d815ef70 con 0x7f98d80cd5f0 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dde00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d80cda80 con 0x7f98d8143f70 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80c4320 con 0x7f98d80c3d70 2024-09-13T17:08:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98d8143f70 0x7f98d80cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98d7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98d80cd5f0 0x7f98d80c3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51524/0 (socket says 172.21.15.28:51524) 2024-09-13T17:08:39.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98d7e00640 1 -- 172.21.15.28:0/433831752 learned_addr learned my addr 172.21.15.28:0/433831752 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:39.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.498+0000 7f98dd400640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98dca00640 1 -- 172.21.15.28:0/433831752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 msgr2=0x7f98d8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98dca00640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d8070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98dca00640 1 -- 172.21.15.28:0/433831752 >> v1:172.21.15.28:6789/0 conn(0x7f98d80cd5f0 legacy=0x7f98d80c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98dca00640 1 -- 172.21.15.28:0/433831752 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d815e1b0 con 0x7f98d8143f70 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98dd400640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d8070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.499+0000 7f98d7400640 1 -- 172.21.15.28:0/433831752 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658725387 0 0) 0x7f98d815ef70 con 0x7f98d80cd5f0 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98dca00640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d8143f70 0x7f98d80cd0b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097a0 tx=0x7f98cc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a9757de01b859e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98d7400640 1 -- 172.21.15.28:0/433831752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc032440 con 0x7f98d8143f70 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98d7400640 1 -- 172.21.15.28:0/433831752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc0325a0 con 0x7f98d8143f70 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98d7400640 1 -- 172.21.15.28:0/433831752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc032930 con 0x7f98d8143f70 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98dde00640 1 -- 172.21.15.28:0/433831752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d8143f70 msgr2=0x7f98d80cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.500+0000 7f98dde00640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d8143f70 0x7f98d80cd0b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0097a0 tx=0x7f98cc02a180 comp rx=0 tx=0).stop 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 -- 172.21.15.28:0/433831752 shutdown_connections 2024-09-13T17:08:39.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d8070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 --2- 172.21.15.28:0/433831752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d8143f70 0x7f98d80cd0b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 -- 172.21.15.28:0/433831752 >> 172.21.15.28:0/433831752 conn(0x7f98d8069e80 msgr2=0x7f98d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 -- 172.21.15.28:0/433831752 shutdown_connections 2024-09-13T17:08:39.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.501+0000 7f98dde00640 1 -- 172.21.15.28:0/433831752 wait complete. 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.502+0000 7f98dde00640 1 Processor -- start 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.502+0000 7f98dde00640 1 -- start start 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d80a9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8143f70 0x7f98d80ab310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80a8d80 con 0x7f98d80cd5f0 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80aaf50 con 0x7f98d8143f70 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.503+0000 7f98dde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80a9040 con 0x7f98d80c3d70 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39386/0 (socket says 172.21.15.28:39386) 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 -- 172.21.15.28:0/3037220320 learned_addr learned my addr 172.21.15.28:0/3037220320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98dca00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d80a9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98dd400640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8143f70 0x7f98d80ab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 -- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 msgr2=0x7f98d80a9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d80a9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 -- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8143f70 msgr2=0x7f98d80ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8143f70 0x7f98d80ab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 -- 172.21.15.28:0/3037220320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc009400 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d7e00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c4d0 tx=0x7f98c400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.504+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40074e0 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.505+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4007660 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.505+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d80a9320 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.505+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c4004a20 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.505+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d80aa060 con 0x7f98d80cd5f0 2024-09-13T17:08:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.506+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d80a9840 con 0x7f98d80cd5f0 2024-09-13T17:08:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.507+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98c4004b80 con 0x7f98d80cd5f0 2024-09-13T17:08:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.507+0000 7f98d4c00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 0x7f98b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.508+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98c4088850 con 0x7f98d80cd5f0 2024-09-13T17:08:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.508+0000 7f98dca00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 0x7f98b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.508+0000 7f98dca00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 0x7f98b806d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98cc034000 tx=0x7f98cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.511+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98d80a9840 con 0x7f98d80cd5f0 2024-09-13T17:08:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.689+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f98d80a9840 con 0x7f98d80cd5f0 2024-09-13T17:08:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.690+0000 7f98d4c00640 1 -- 172.21.15.28:0/3037220320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f98d80a9840 con 0x7f98d80cd5f0 2024-09-13T17:08:39.693 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 msgr2=0x7f98b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 0x7f98b806d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98cc034000 tx=0x7f98cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 msgr2=0x7f98d80aa590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c4d0 tx=0x7f98c400c990 comp rx=0 tx=0).stop 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 shutdown_connections 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98b806ac00 0x7f98b806d0c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98d8143f70 0x7f98d80ab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98d80cd5f0 0x7f98d80aa590 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 --2- 172.21.15.28:0/3037220320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98d80c3d70 0x7f98d80a9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 >> 172.21.15.28:0/3037220320 conn(0x7f98d8069e80 msgr2=0x7f98d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 shutdown_connections 2024-09-13T17:08:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.692+0000 7f98dde00640 1 -- 172.21.15.28:0/3037220320 wait complete. 2024-09-13T17:08:39.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.864+0000 7f62c6200640 1 Processor -- start 2024-09-13T17:08:39.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.864+0000 7f62c6200640 1 -- start start 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62c6200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62c00cabc0 0x7f62c006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62c6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c015e100 con 0x7f62c015a540 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62c6200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c015e3e0 con 0x7f62c0070cc0 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62c6200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c00adb00 con 0x7f62c00cabc0 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62bf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62c015a540 0x7f62c015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51532/0 (socket says 172.21.15.28:51532) 2024-09-13T17:08:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.865+0000 7f62bf400640 1 -- 172.21.15.28:0/963335806 learned_addr learned my addr 172.21.15.28:0/963335806 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62bea00640 1 --2- 172.21.15.28:0/963335806 >> v2:172.21.15.189:3300/0 conn(0x7f62c00cabc0 0x7f62c006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721124000 0 0) 0x7f62c015e100 con 0x7f62c015a540 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a4003520 con 0x7f62c015a540 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401120855 0 0) 0x7f62a4003520 con 0x7f62c015a540 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 >> v2:172.21.15.189:3300/0 conn(0x7f62c00cabc0 msgr2=0x7f62c006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 --2- 172.21.15.28:0/963335806 >> v2:172.21.15.189:3300/0 conn(0x7f62c00cabc0 0x7f62c006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 >> v1:172.21.15.189:6789/0 conn(0x7f62c0070cc0 legacy=0x7f62c006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c015ef80 con 0x7f62c015a540 2024-09-13T17:08:39.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f62b0001670 con 0x7f62c015a540 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 >> v1:172.21.15.28:6789/0 conn(0x7f62c015a540 legacy=0x7f62c015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40039c0 0x7f62a4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 0x7f62a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a40051d0 0x7f62a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a400c0f0 con 0x7f62a40047c0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a400c260 con 0x7f62a40039c0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.866+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a400c420 con 0x7f62a40051d0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 0x7f62a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bf400640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40039c0 0x7f62a4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bfe00640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a40051d0 0x7f62a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 -- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a40051d0 msgr2=0x7f62a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a40051d0 0x7f62a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 -- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40039c0 msgr2=0x7f62a4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40039c0 0x7f62a4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bea00640 1 -- 172.21.15.28:0/963335806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a4003520 con 0x7f62a40047c0 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.867+0000 7f62bfe00640 1 -- 172.21.15.28:0/963335806 reap_dead start 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.868+0000 7f62bea00640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 0x7f62a4004c40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009a90 tx=0x7f62a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc539d7a79fc4d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.868+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac031400 con 0x7f62a40047c0 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.868+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ac02d050 con 0x7f62a40047c0 2024-09-13T17:08:39.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.868+0000 7f62be000640 1 -- 172.21.15.28:0/963335806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ac031780 con 0x7f62a40047c0 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 -- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 msgr2=0x7f62a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 0x7f62a4004c40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009a90 tx=0x7f62a40057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 -- 172.21.15.28:0/963335806 shutdown_connections 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 --2- 172.21.15.28:0/963335806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a40047c0 0x7f62a4004c40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 -- 172.21.15.28:0/963335806 >> 172.21.15.28:0/963335806 conn(0x7f62c0069e80 msgr2=0x7f62c006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 -- 172.21.15.28:0/963335806 shutdown_connections 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.869+0000 7f62c6200640 1 -- 172.21.15.28:0/963335806 wait complete. 2024-09-13T17:08:39.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.871+0000 7f62c6200640 1 Processor -- start 2024-09-13T17:08:39.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.871+0000 7f62c6200640 1 -- start start 2024-09-13T17:08:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.871+0000 7f62c6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40047c0 0x7f62c015b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62c6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62c015cc80 0x7f62c015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62c6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62c6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c01f62f0 con 0x7f62c00b0290 2024-09-13T17:08:39.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62c6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c00b0c80 con 0x7f62a40047c0 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62c6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c015c830 con 0x7f62c015cc80 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39414/0 (socket says 172.21.15.28:39414) 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 -- 172.21.15.28:0/3004349397 learned_addr learned my addr 172.21.15.28:0/3004349397 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bf400640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40047c0 0x7f62c015b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bea00640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62c015cc80 0x7f62c015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 -- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62c015cc80 msgr2=0x7f62c015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62c015cc80 0x7f62c015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 -- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40047c0 msgr2=0x7f62c015b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40047c0 0x7f62c015b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.872+0000 7f62bfe00640 1 -- 172.21.15.28:0/3004349397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c01f5670 con 0x7f62c00b0290 2024-09-13T17:08:39.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62bfe00640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b8007fd0 tx=0x7f62b800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b800eed0 con 0x7f62c00b0290 2024-09-13T17:08:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b8003970 con 0x7f62c00b0290 2024-09-13T17:08:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8003d40 con 0x7f62c00b0290 2024-09-13T17:08:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c01f5950 con 0x7f62c00b0290 2024-09-13T17:08:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.873+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c01f5f10 con 0x7f62c00b0290 2024-09-13T17:08:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.874+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c01f6430 con 0x7f62c00b0290 2024-09-13T17:08:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.875+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f62b80050d0 con 0x7f62c00b0290 2024-09-13T17:08:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.876+0000 7f62b7400640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 0x7f62a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.876+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62b8088fb0 con 0x7f62c00b0290 2024-09-13T17:08:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.876+0000 7f62bf400640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 0x7f62a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:39.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.877+0000 7f62bf400640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 0x7f62a006d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62b0001050 tx=0x7f62b0000f80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:39.880+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f62c01f6430 con 0x7f62c00b0290 2024-09-13T17:08:40.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.063+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62c01f6430 con 0x7f62c00b0290 2024-09-13T17:08:40.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.064+0000 7f62b7400640 1 -- 172.21.15.28:0/3004349397 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f62c01f6430 con 0x7f62c00b0290 2024-09-13T17:08:40.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:40.069 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 msgr2=0x7f62a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 0x7f62a006d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f62b0001050 tx=0x7f62b0000f80 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 msgr2=0x7f62c00b0740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f62b8007fd0 tx=0x7f62b800eb90 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 shutdown_connections 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62a006ac00 0x7f62a006d0c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62c00b0290 0x7f62c00b0740 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62c015cc80 0x7f62c015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 --2- 172.21.15.28:0/3004349397 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a40047c0 0x7f62c015b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 >> 172.21.15.28:0/3004349397 conn(0x7f62c0069e80 msgr2=0x7f62c0155cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 shutdown_connections 2024-09-13T17:08:40.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.067+0000 7f62c6200640 1 -- 172.21.15.28:0/3004349397 wait complete. 2024-09-13T17:08:40.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:40.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.242+0000 7f4f33600640 1 Processor -- start 2024-09-13T17:08:40.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.242+0000 7f4f33600640 1 -- start start 2024-09-13T17:08:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f33600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f2c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f33600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c070470 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f33600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c0ace30 con 0x7f4f2c0b2400 2024-09-13T17:08:40.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f33600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0ad110 con 0x7f4f2c0ad940 2024-09-13T17:08:40.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f30c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4f2c15c9a0 0x7f4f2c0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51534/0 (socket says 172.21.15.28:51534) 2024-09-13T17:08:40.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.243+0000 7f4f30c00640 1 -- 172.21.15.28:0/1796520803 learned_addr learned my addr 172.21.15.28:0/1796520803 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f31600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f2c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676917684 0 0) 0x7f4f2c070470 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14003520 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4179441113 0 0) 0x7f4f14003520 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 msgr2=0x7f4f2c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f2c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 >> v1:172.21.15.189:6789/0 conn(0x7f4f2c0b2400 legacy=0x7f4f2c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c06b170 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4f20001670 con 0x7f4f2c15c9a0 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 >> v1:172.21.15.28:6789/0 conn(0x7f4f2c15c9a0 legacy=0x7f4f2c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f140039a0 0x7f4f14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f14004d00 0x7f4f14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0ad110 con 0x7f4f14004d00 2024-09-13T17:08:40.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1400bac0 con 0x7f4f140039a0 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.244+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1400bc80 con 0x7f4f2c0ad940 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f14004d00 0x7f4f14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f30c00640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f140039a0 0x7f4f14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f2be00640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 -- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 msgr2=0x7f4f140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c0ad940 0x7f4f140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 -- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f140039a0 msgr2=0x7f4f14003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f140039a0 0x7f4f14003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.245+0000 7f4f31600640 1 -- 172.21.15.28:0/1796520803 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f14003520 con 0x7f4f14004d00 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.246+0000 7f4f31600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f14004d00 0x7f4f14005180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f90 tx=0x7f4f140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8279e9181bd2ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.246+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24031610 con 0x7f4f14004d00 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.246+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2400f040 con 0x7f4f14004d00 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.246+0000 7f4f2b400640 1 -- 172.21.15.28:0/1796520803 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f240319b0 con 0x7f4f14004d00 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 -- 172.21.15.28:0/1796520803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f14004d00 msgr2=0x7f4f14005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 --2- 172.21.15.28:0/1796520803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f14004d00 0x7f4f14005180 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f90 tx=0x7f4f140057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f31600640 1 -- 172.21.15.28:0/1796520803 reap_dead start 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 -- 172.21.15.28:0/1796520803 shutdown_connections 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 -- 172.21.15.28:0/1796520803 >> 172.21.15.28:0/1796520803 conn(0x7f4f2c0b0270 msgr2=0x7f4f2c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 -- 172.21.15.28:0/1796520803 shutdown_connections 2024-09-13T17:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.247+0000 7f4f33600640 1 -- 172.21.15.28:0/1796520803 wait complete. 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.249+0000 7f4f33600640 1 Processor -- start 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.249+0000 7f4f33600640 1 -- start start 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f2c1ef980 0x7f4f2c15d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c1efd60 0x7f4f2c154af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15df20 con 0x7f4f2c15fe60 2024-09-13T17:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c15dc40 con 0x7f4f2c1ef980 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f33600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c151870 con 0x7f4f2c1efd60 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39446/0 (socket says 172.21.15.28:39446) 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 -- 172.21.15.28:0/1914820039 learned_addr learned my addr 172.21.15.28:0/1914820039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f30c00640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f2c1ef980 0x7f4f2c15d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f2be00640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c1efd60 0x7f4f2c154af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 -- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c1efd60 msgr2=0x7f4f2c154af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c1efd60 0x7f4f2c154af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 -- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f2c1ef980 msgr2=0x7f4f2c15d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f2c1ef980 0x7f4f2c15d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.250+0000 7f4f31600640 1 -- 172.21.15.28:0/1914820039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c070930 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f31600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f90 tx=0x7f4f24009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24003990 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f24040070 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c15df20 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f24003d30 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.251+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c151e30 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.252+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2c1516d0 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.253+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f24042030 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.254+0000 7f4f28c00640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 0x7f4f0c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.254+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f240aa590 con 0x7f4f2c15fe60 2024-09-13T17:08:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.254+0000 7f4f30c00640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 0x7f4f0c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.255+0000 7f4f30c00640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 0x7f4f0c06d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f20004840 tx=0x7f4f20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.258+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4f2c1516d0 con 0x7f4f2c15fe60 2024-09-13T17:08:40.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.435+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f2c1516d0 con 0x7f4f2c15fe60 2024-09-13T17:08:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.436+0000 7f4f28c00640 1 -- 172.21.15.28:0/1914820039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f2c1516d0 con 0x7f4f2c15fe60 2024-09-13T17:08:40.438 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:40.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 msgr2=0x7f4f0c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 0x7f4f0c06d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4f20004840 tx=0x7f4f20009040 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 msgr2=0x7f4f2c1f5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f4f24009f90 tx=0x7f4f24009750 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 shutdown_connections 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f0c06ac00 0x7f4f0c06d0c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f2c15fe60 0x7f4f2c1f5c20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f2c1efd60 0x7f4f2c154af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 --2- 172.21.15.28:0/1914820039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f2c1ef980 0x7f4f2c15d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 >> 172.21.15.28:0/1914820039 conn(0x7f4f2c0b0270 msgr2=0x7f4f2c06bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 shutdown_connections 2024-09-13T17:08:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.438+0000 7f4f33600640 1 -- 172.21.15.28:0/1914820039 wait complete. 2024-09-13T17:08:40.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.611+0000 7f339fe00640 1 Processor -- start 2024-09-13T17:08:40.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.612+0000 7f339fe00640 1 -- start start 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33a0143f70 0x7f33a00cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a015ef70 con 0x7f33a00c3d70 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a00c4320 con 0x7f33a00cd5f0 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a00cda80 con 0x7f33a0143f70 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f33a00c3d70 0x7f33a0070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51536/0 (socket says 172.21.15.28:51536) 2024-09-13T17:08:40.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339f400640 1 -- 172.21.15.28:0/764283891 learned_addr learned my addr 172.21.15.28:0/764283891 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.613+0000 7f339ea00640 1 --2- 172.21.15.28:0/764283891 >> v2:172.21.15.189:3300/0 conn(0x7f33a0143f70 0x7f33a00cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.614+0000 7f339e000640 1 -- 172.21.15.28:0/764283891 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087846608 0 0) 0x7f33a015ef70 con 0x7f33a00c3d70 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.614+0000 7f339e000640 1 -- 172.21.15.28:0/764283891 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3384003520 con 0x7f33a00c3d70 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.614+0000 7f339ea00640 1 -- 172.21.15.28:0/764283891 >> v1:172.21.15.189:6789/0 conn(0x7f33a00cd5f0 legacy=0x7f33a00c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.614+0000 7f339ea00640 1 -- 172.21.15.28:0/764283891 >> v1:172.21.15.28:6789/0 conn(0x7f33a00c3d70 legacy=0x7f33a0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.614+0000 7f339ea00640 1 -- 172.21.15.28:0/764283891 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a015e1b0 con 0x7f33a0143f70 2024-09-13T17:08:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.615+0000 7f339ea00640 1 --2- 172.21.15.28:0/764283891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a0143f70 0x7f33a00cd0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f33900097a0 tx=0x7f339002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abde2ebd602738c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.615+0000 7f339e000640 1 -- 172.21.15.28:0/764283891 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390032440 con 0x7f33a0143f70 2024-09-13T17:08:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.615+0000 7f339e000640 1 -- 172.21.15.28:0/764283891 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33900325a0 con 0x7f33a0143f70 2024-09-13T17:08:40.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.615+0000 7f339e000640 1 -- 172.21.15.28:0/764283891 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3390032930 con 0x7f33a0143f70 2024-09-13T17:08:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 -- 172.21.15.28:0/764283891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a0143f70 msgr2=0x7f33a00cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 --2- 172.21.15.28:0/764283891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a0143f70 0x7f33a00cd0b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f33900097a0 tx=0x7f339002a230 comp rx=0 tx=0).stop 2024-09-13T17:08:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 -- 172.21.15.28:0/764283891 shutdown_connections 2024-09-13T17:08:40.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 --2- 172.21.15.28:0/764283891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a0143f70 0x7f33a00cd0b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 -- 172.21.15.28:0/764283891 >> 172.21.15.28:0/764283891 conn(0x7f33a0069e80 msgr2=0x7f33a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 -- 172.21.15.28:0/764283891 shutdown_connections 2024-09-13T17:08:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.616+0000 7f339fe00640 1 -- 172.21.15.28:0/764283891 wait complete. 2024-09-13T17:08:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.618+0000 7f339fe00640 1 Processor -- start 2024-09-13T17:08:40.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.618+0000 7f339fe00640 1 -- start start 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 0x7f33a0142e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a00cd5f0 0x7f33a015d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a00c72c0 con 0x7f33a0143f70 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0143350 con 0x7f33a00c3d70 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a00a99f0 con 0x7f33a00cd5f0 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39462/0 (socket says 172.21.15.28:39462) 2024-09-13T17:08:40.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 -- 172.21.15.28:0/2520924029 learned_addr learned my addr 172.21.15.28:0/2520924029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 -- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a00cd5f0 msgr2=0x7f33a015d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a00cd5f0 0x7f33a015d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 -- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 msgr2=0x7f33a0142e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339ea00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 0x7f33a0142e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 0x7f33a0142e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339f400640 1 -- 172.21.15.28:0/2520924029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3390009400 con 0x7f33a0143f70 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.619+0000 7f339ea00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 0x7f33a0142e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f339f400640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f339800cb40 tx=0x7f339800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339800aca0 con 0x7f33a0143f70 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339800ae00 con 0x7f33a0143f70 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a00c76e0 con 0x7f33a0143f70 2024-09-13T17:08:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398012620 con 0x7f33a0143f70 2024-09-13T17:08:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.620+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a00c6c00 con 0x7f33a0143f70 2024-09-13T17:08:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.621+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a00c7400 con 0x7f33a0143f70 2024-09-13T17:08:40.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.622+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3398007070 con 0x7f33a0143f70 2024-09-13T17:08:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.623+0000 7f3396a00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 0x7f337c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.623+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33980892f0 con 0x7f33a0143f70 2024-09-13T17:08:40.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.623+0000 7f339ea00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 0x7f337c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.624+0000 7f339ea00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 0x7f337c06d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3390034000 tx=0x7f339002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.627+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f33a00c7400 con 0x7f33a0143f70 2024-09-13T17:08:40.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.804+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f33a00c7400 con 0x7f33a0143f70 2024-09-13T17:08:40.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.804+0000 7f3396a00640 1 -- 172.21.15.28:0/2520924029 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f33a00c7400 con 0x7f33a0143f70 2024-09-13T17:08:40.806 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 msgr2=0x7f337c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 0x7f337c06d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3390034000 tx=0x7f339002e000 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 msgr2=0x7f33a015a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f339800cb40 tx=0x7f339800a060 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 shutdown_connections 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f337c06ac00 0x7f337c06d0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0143f70 0x7f33a015a770 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a00cd5f0 0x7f33a015d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 --2- 172.21.15.28:0/2520924029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00c3d70 0x7f33a0142e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 >> 172.21.15.28:0/2520924029 conn(0x7f33a0069e80 msgr2=0x7f33a015c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.806+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 shutdown_connections 2024-09-13T17:08:40.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.807+0000 7f339fe00640 1 -- 172.21.15.28:0/2520924029 wait complete. 2024-09-13T17:08:40.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.981+0000 7f6a0a200640 1 Processor -- start 2024-09-13T17:08:40.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.981+0000 7f6a0a200640 1 -- start start 2024-09-13T17:08:40.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.982+0000 7f6a0a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a0415a780 0x7f6a04070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.982+0000 7f6a0a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0406c660 con 0x7f6a0415ab60 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.982+0000 7f6a0a200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0406c8b0 con 0x7f6a0406c250 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.982+0000 7f6a0a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0406cb90 con 0x7f6a0415a780 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a03400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a0415a780 0x7f6a04070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a02a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a0415ab60 0x7f6a0406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51542/0 (socket says 172.21.15.28:51542) 2024-09-13T17:08:40.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a02a00640 1 -- 172.21.15.28:0/2634229760 learned_addr learned my addr 172.21.15.28:0/2634229760 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a02000640 1 -- 172.21.15.28:0/2634229760 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051033844 0 0) 0x7f6a0406c660 con 0x7f6a0415ab60 2024-09-13T17:08:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a02000640 1 -- 172.21.15.28:0/2634229760 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69e8003520 con 0x7f6a0415ab60 2024-09-13T17:08:40.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a03400640 1 -- 172.21.15.28:0/2634229760 >> v1:172.21.15.189:6789/0 conn(0x7f6a0406c250 legacy=0x7f6a040cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a03400640 1 -- 172.21.15.28:0/2634229760 >> v1:172.21.15.28:6789/0 conn(0x7f6a0415ab60 legacy=0x7f6a0406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.983+0000 7f6a03400640 1 -- 172.21.15.28:0/2634229760 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040cc980 con 0x7f6a0415a780 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.984+0000 7f6a03400640 1 --2- 172.21.15.28:0/2634229760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415a780 0x7f6a04070d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f69f40097a0 tx=0x7f69f402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5da15c50a3ad19 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.984+0000 7f6a02000640 1 -- 172.21.15.28:0/2634229760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4032440 con 0x7f6a0415a780 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a02000640 1 -- 172.21.15.28:0/2634229760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f40325a0 con 0x7f6a0415a780 2024-09-13T17:08:40.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a02000640 1 -- 172.21.15.28:0/2634229760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4032930 con 0x7f6a0415a780 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 -- 172.21.15.28:0/2634229760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415a780 msgr2=0x7f6a04070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 --2- 172.21.15.28:0/2634229760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415a780 0x7f6a04070d80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f69f40097a0 tx=0x7f69f402a230 comp rx=0 tx=0).stop 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 -- 172.21.15.28:0/2634229760 shutdown_connections 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 --2- 172.21.15.28:0/2634229760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415a780 0x7f6a04070d80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 -- 172.21.15.28:0/2634229760 >> 172.21.15.28:0/2634229760 conn(0x7f6a040b0270 msgr2=0x7f6a040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 -- 172.21.15.28:0/2634229760 shutdown_connections 2024-09-13T17:08:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.985+0000 7f6a0a200640 1 -- 172.21.15.28:0/2634229760 wait complete. 2024-09-13T17:08:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.987+0000 7f6a0a200640 1 Processor -- start 2024-09-13T17:08:40.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.987+0000 7f6a0a200640 1 -- start start 2024-09-13T17:08:40.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0406c250 0x7f6a04148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 0x7f6a04144e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04147d10 con 0x7f6a0415ab60 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04148670 con 0x7f6a0415a780 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a0a200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a04147090 con 0x7f6a0406c250 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39476/0 (socket says 172.21.15.28:39476) 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03e00640 1 -- 172.21.15.28:0/901611892 learned_addr learned my addr 172.21.15.28:0/901611892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a02a00640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 0x7f6a04144e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03e00640 1 -- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0406c250 msgr2=0x7f6a04148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03400640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0406c250 0x7f6a04148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.988+0000 7f6a03e00640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0406c250 0x7f6a04148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a03e00640 1 -- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 msgr2=0x7f6a04144e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a03e00640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 0x7f6a04144e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a03e00640 1 -- 172.21.15.28:0/901611892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4009400 con 0x7f6a0415ab60 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a02a00640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 0x7f6a04144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a03e00640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c7a0 tx=0x7f69fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc003b70 con 0x7f6a0415ab60 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc003cf0 con 0x7f6a0415ab60 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc003ea0 con 0x7f6a0415ab60 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a04147370 con 0x7f6a0415ab60 2024-09-13T17:08:40.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.989+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04147930 con 0x7f6a0415ab60 2024-09-13T17:08:40.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.990+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a04147e50 con 0x7f6a0415ab60 2024-09-13T17:08:40.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.991+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f69fc00aa90 con 0x7f6a0415ab60 2024-09-13T17:08:40.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.992+0000 7f69fb400640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 0x7f69e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:40.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.992+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69fc0891d0 con 0x7f6a0415ab60 2024-09-13T17:08:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.993+0000 7f6a03400640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 0x7f69e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:40.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.994+0000 7f6a03400640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 0x7f69e406d0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69f4034000 tx=0x7f69f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:40.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:40.997+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6a04147e50 con 0x7f6a0415ab60 2024-09-13T17:08:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.178+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6a04147e50 con 0x7f6a0415ab60 2024-09-13T17:08:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.178+0000 7f69fb400640 1 -- 172.21.15.28:0/901611892 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6a04147e50 con 0x7f6a0415ab60 2024-09-13T17:08:41.180 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 msgr2=0x7f69e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 0x7f69e406d0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f69f4034000 tx=0x7f69f402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 msgr2=0x7f6a04146340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f69fc00c7a0 tx=0x7f69fc00cc60 comp rx=0 tx=0).stop 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 shutdown_connections 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f69e406ac00 0x7f69e406d0c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0415ab60 0x7f6a04146340 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0415a780 0x7f6a04144e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 --2- 172.21.15.28:0/901611892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0406c250 0x7f6a04148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 >> 172.21.15.28:0/901611892 conn(0x7f6a040b0270 msgr2=0x7f6a04157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 shutdown_connections 2024-09-13T17:08:41.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.180+0000 7f6a0a200640 1 -- 172.21.15.28:0/901611892 wait complete. 2024-09-13T17:08:41.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:41.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.351+0000 7f1efd400640 1 Processor -- start 2024-09-13T17:08:41.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.351+0000 7f1efd400640 1 -- start start 2024-09-13T17:08:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ef806ce80 0x7f1ef8160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efd400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef8061110 0x7f1ef806d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efd400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef806bf50 con 0x7f1ef806c3e0 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efd400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef80ccd70 con 0x7f1ef806ce80 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efd400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81613b0 con 0x7f1ef8061110 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1ef7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ef806ce80 0x7f1ef8160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1ef7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ef806ce80 0x7f1ef8160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39486/0 (socket says 172.21.15.28:39486) 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1ef7e00640 1 -- 172.21.15.28:0/2709632823 learned_addr learned my addr 172.21.15.28:0/2709632823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1efca00640 1 --1- 172.21.15.28:0/2709632823 >> v1:172.21.15.28:6789/0 conn(0x7f1ef806c3e0 0x7f1ef815cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51550/0 (socket says 172.21.15.28:51550) 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.352+0000 7f1eef400640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef8061110 0x7f1ef806d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.353+0000 7f1ef7e00640 1 -- 172.21.15.28:0/2709632823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef8061110 msgr2=0x7f1ef806d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.353+0000 7f1ef7e00640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef8061110 0x7f1ef806d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.353+0000 7f1ef7e00640 1 -- 172.21.15.28:0/2709632823 >> v1:172.21.15.28:6789/0 conn(0x7f1ef806c3e0 legacy=0x7f1ef815cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.353+0000 7f1ef7400640 1 -- 172.21.15.28:0/2709632823 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547729395 0 0) 0x7f1ef806bf50 con 0x7f1ef806c3e0 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.353+0000 7f1ef7e00640 1 -- 172.21.15.28:0/2709632823 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef806d7a0 con 0x7f1ef806ce80 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.354+0000 7f1ef7e00640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef806ce80 0x7f1ef8160e70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009870 tx=0x7f1ee802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228c235932996976 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.354+0000 7f1ef7400640 1 -- 172.21.15.28:0/2709632823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8032440 con 0x7f1ef806ce80 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.354+0000 7f1ef7400640 1 -- 172.21.15.28:0/2709632823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee80325a0 con 0x7f1ef806ce80 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.354+0000 7f1ef7400640 1 -- 172.21.15.28:0/2709632823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8032930 con 0x7f1ef806ce80 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 -- 172.21.15.28:0/2709632823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef806ce80 msgr2=0x7f1ef8160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef806ce80 0x7f1ef8160e70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8009870 tx=0x7f1ee802a270 comp rx=0 tx=0).stop 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 -- 172.21.15.28:0/2709632823 shutdown_connections 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef8061110 0x7f1ef806d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 --2- 172.21.15.28:0/2709632823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef806ce80 0x7f1ef8160e70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 -- 172.21.15.28:0/2709632823 >> 172.21.15.28:0/2709632823 conn(0x7f1ef80cb6d0 msgr2=0x7f1ef80cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 -- 172.21.15.28:0/2709632823 shutdown_connections 2024-09-13T17:08:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.355+0000 7f1efd400640 1 -- 172.21.15.28:0/2709632823 wait complete. 2024-09-13T17:08:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.356+0000 7f1efd400640 1 Processor -- start 2024-09-13T17:08:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.356+0000 7f1efd400640 1 -- start start 2024-09-13T17:08:41.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 0x7f1ef80c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef806ce80 0x7f1ef8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c6260 con 0x7f1ef8061110 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef8149de0 con 0x7f1ef806c3e0 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1efd400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c6a10 con 0x7f1ef806ce80 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.357+0000 7f1ef7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39500/0 (socket says 172.21.15.28:39500) 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 -- 172.21.15.28:0/3644492830 learned_addr learned my addr 172.21.15.28:0/3644492830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1eef400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 0x7f1ef80c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1efca00640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef806ce80 0x7f1ef8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 -- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef806ce80 msgr2=0x7f1ef8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef806ce80 0x7f1ef8070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 -- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 msgr2=0x7f1ef80c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 0x7f1ef80c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 -- 172.21.15.28:0/3644492830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee80094d0 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1eef400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 0x7f1ef80c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.358+0000 7f1ef7e00640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8000c00 tx=0x7f1ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.359+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee8032440 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.359+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee8004030 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.359+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee80043e0 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.359+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef80c64b0 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.359+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef80c8110 con 0x7f1ef8061110 2024-09-13T17:08:41.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.360+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef80caf40 con 0x7f1ef8061110 2024-09-13T17:08:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.361+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ee8039050 con 0x7f1ef8061110 2024-09-13T17:08:41.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.361+0000 7f1ef4c00640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 0x7f1ed406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.361+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ee80a99e0 con 0x7f1ef8061110 2024-09-13T17:08:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.362+0000 7f1eef400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 0x7f1ed406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.362+0000 7f1eef400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 0x7f1ed406d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0006fd0 tx=0x7f1ee0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.366+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1ee8074cf0 con 0x7f1ef8061110 2024-09-13T17:08:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.544+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ef80c2520 con 0x7f1ef8061110 2024-09-13T17:08:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.545+0000 7f1ef4c00640 1 -- 172.21.15.28:0/3644492830 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f1ee804b090 con 0x7f1ef8061110 2024-09-13T17:08:41.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:41.548 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 msgr2=0x7f1ed406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 0x7f1ed406d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0006fd0 tx=0x7f1ee0008040 comp rx=0 tx=0).stop 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 msgr2=0x7f1ef80c5d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8000c00 tx=0x7f1ee80046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 shutdown_connections 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ed406ac00 0x7f1ed406d0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ef806ce80 0x7f1ef8070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ef806c3e0 0x7f1ef80c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 --2- 172.21.15.28:0/3644492830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef8061110 0x7f1ef80c5d20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 >> 172.21.15.28:0/3644492830 conn(0x7f1ef80cb6d0 msgr2=0x7f1ef815cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 shutdown_connections 2024-09-13T17:08:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.548+0000 7f1efd400640 1 -- 172.21.15.28:0/3644492830 wait complete. 2024-09-13T17:08:41.566 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:41.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.722+0000 7ff939e00640 1 Processor -- start 2024-09-13T17:08:41.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.722+0000 7ff939e00640 1 -- start start 2024-09-13T17:08:41.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff939e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff93415a780 0x7ff934070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff939e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff9340712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff939e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93406c660 con 0x7ff93406bb40 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff939e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93406c8b0 con 0x7ff93415a780 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff939e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93406cb90 con 0x7ff93415ab60 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff932a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff93415a780 0x7ff934070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff933400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff93406bb40 0x7ff93406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51556/0 (socket says 172.21.15.28:51556) 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff933400640 1 -- 172.21.15.28:0/2376717198 learned_addr learned my addr 172.21.15.28:0/2376717198 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.723+0000 7ff92be00640 1 --2- 172.21.15.28:0/2376717198 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff9340712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.724+0000 7ff932a00640 1 -- 172.21.15.28:0/2376717198 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 msgr2=0x7ff9340712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.724+0000 7ff932a00640 1 --2- 172.21.15.28:0/2376717198 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff9340712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.724+0000 7ff932a00640 1 -- 172.21.15.28:0/2376717198 >> v1:172.21.15.28:6789/0 conn(0x7ff93406bb40 legacy=0x7ff93406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.724+0000 7ff932a00640 1 -- 172.21.15.28:0/2376717198 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9340cc980 con 0x7ff93415a780 2024-09-13T17:08:41.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.724+0000 7ff932000640 1 -- 172.21.15.28:0/2376717198 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791261435 0 0) 0x7ff93406c660 con 0x7ff93406bb40 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.725+0000 7ff932a00640 1 --2- 172.21.15.28:0/2376717198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415a780 0x7ff934070d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff93406f380 tx=0x7ff92402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd298b72106107e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.725+0000 7ff932000640 1 -- 172.21.15.28:0/2376717198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924032440 con 0x7ff93415a780 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.725+0000 7ff932000640 1 -- 172.21.15.28:0/2376717198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9240325a0 con 0x7ff93415a780 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.725+0000 7ff932000640 1 -- 172.21.15.28:0/2376717198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924032930 con 0x7ff93415a780 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 -- 172.21.15.28:0/2376717198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415a780 msgr2=0x7ff934070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 --2- 172.21.15.28:0/2376717198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415a780 0x7ff934070d80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff93406f380 tx=0x7ff92402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 -- 172.21.15.28:0/2376717198 shutdown_connections 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 --2- 172.21.15.28:0/2376717198 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff9340712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 --2- 172.21.15.28:0/2376717198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415a780 0x7ff934070d80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 -- 172.21.15.28:0/2376717198 >> 172.21.15.28:0/2376717198 conn(0x7ff9340b0270 msgr2=0x7ff9340b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 -- 172.21.15.28:0/2376717198 shutdown_connections 2024-09-13T17:08:41.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.726+0000 7ff939e00640 1 -- 172.21.15.28:0/2376717198 wait complete. 2024-09-13T17:08:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.727+0000 7ff939e00640 1 Processor -- start 2024-09-13T17:08:41.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.728+0000 7ff939e00640 1 -- start start 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.728+0000 7ff939e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff93406bb40 0x7ff9340b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff939e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 0x7ff934156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff939e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff939e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340b2750 con 0x7ff93415ab60 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff939e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340ad2e0 con 0x7ff93415a780 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff939e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340ac9f0 con 0x7ff93406bb40 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39522/0 (socket says 172.21.15.28:39522) 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 -- 172.21.15.28:0/1872668724 learned_addr learned my addr 172.21.15.28:0/1872668724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff932a00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff93406bb40 0x7ff9340b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff92be00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 0x7ff934156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 -- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff93406bb40 msgr2=0x7ff9340b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff93406bb40 0x7ff9340b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 -- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 msgr2=0x7ff934156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 0x7ff934156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff933400640 1 -- 172.21.15.28:0/1872668724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924009400 con 0x7ff93415ab60 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.729+0000 7ff92be00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 0x7ff934156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff933400640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00c7e0 tx=0x7ff92c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00a640 con 0x7ff93415ab60 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c00a7c0 con 0x7ff93415ab60 2024-09-13T17:08:41.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00ab70 con 0x7ff93415ab60 2024-09-13T17:08:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9340acc90 con 0x7ff93415ab60 2024-09-13T17:08:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.730+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9340b2440 con 0x7ff93415ab60 2024-09-13T17:08:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.731+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff93406bf50 con 0x7ff93415ab60 2024-09-13T17:08:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.732+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff92c00acd0 con 0x7ff93415ab60 2024-09-13T17:08:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.733+0000 7ff92b400640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.733+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff92c0887f0 con 0x7ff93415ab60 2024-09-13T17:08:41.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.733+0000 7ff932a00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:41.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.734+0000 7ff932a00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff924034000 tx=0x7ff92402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.737+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff92c0539e0 con 0x7ff93415ab60 2024-09-13T17:08:41.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.914+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff9340cb060 con 0x7ff93415ab60 2024-09-13T17:08:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.914+0000 7ff92b400640 1 -- 172.21.15.28:0/1872668724 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff92c0180a0 con 0x7ff93415ab60 2024-09-13T17:08:41.917 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 msgr2=0x7ff91006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff924034000 tx=0x7ff92402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 msgr2=0x7ff9340b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff92c00c7e0 tx=0x7ff92c00cca0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 shutdown_connections 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93415ab60 0x7ff9340b1f00 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff93415a780 0x7ff934156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 --2- 172.21.15.28:0/1872668724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff93406bb40 0x7ff9340b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 >> 172.21.15.28:0/1872668724 conn(0x7ff9340b0270 msgr2=0x7ff9340b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 shutdown_connections 2024-09-13T17:08:41.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:41.917+0000 7ff939e00640 1 -- 172.21.15.28:0/1872668724 wait complete. 2024-09-13T17:08:41.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.092+0000 7f34d6200640 1 Processor -- start 2024-09-13T17:08:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.092+0000 7f34d6200640 1 -- start start 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34d015a780 0x7f34d0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d015ab60 0x7f34d00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34d006bb40 0x7f34d006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d006c8b0 con 0x7f34d006bb40 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d006cb90 con 0x7f34d015a780 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34d6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d00cc980 con 0x7f34d015ab60 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34d006bb40 0x7f34d006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f34d006bb40 0x7f34d006bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39538/0 (socket says 172.21.15.28:39538) 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.093+0000 7f34cfe00640 1 -- 172.21.15.28:0/3043785750 learned_addr learned my addr 172.21.15.28:0/3043785750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cf400640 1 --2- 172.21.15.28:0/3043785750 >> v2:172.21.15.189:3300/0 conn(0x7f34d015a780 0x7f34d0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cea00640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d015ab60 0x7f34d00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cfe00640 1 -- 172.21.15.28:0/3043785750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d015ab60 msgr2=0x7f34d00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cfe00640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d015ab60 0x7f34d00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cfe00640 1 -- 172.21.15.28:0/3043785750 >> v2:172.21.15.189:3300/0 conn(0x7f34d015a780 msgr2=0x7f34d0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cfe00640 1 --2- 172.21.15.28:0/3043785750 >> v2:172.21.15.189:3300/0 conn(0x7f34d015a780 0x7f34d0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.094+0000 7f34cfe00640 1 -- 172.21.15.28:0/3043785750 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00ccc60 con 0x7f34d006bb40 2024-09-13T17:08:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.095+0000 7f34cfe00640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d006bb40 0x7f34d006bfd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34d00b1330 tx=0x7f34c802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=707d07187ec9e2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.095+0000 7f34ce000640 1 -- 172.21.15.28:0/3043785750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8031600 con 0x7f34d006bb40 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.095+0000 7f34ce000640 1 -- 172.21.15.28:0/3043785750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c802d050 con 0x7f34d006bb40 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.095+0000 7f34ce000640 1 -- 172.21.15.28:0/3043785750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8031980 con 0x7f34d006bb40 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 -- 172.21.15.28:0/3043785750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d006bb40 msgr2=0x7f34d006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d006bb40 0x7f34d006bfd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34d00b1330 tx=0x7f34c802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 -- 172.21.15.28:0/3043785750 shutdown_connections 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d006bb40 0x7f34d006bfd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 --2- 172.21.15.28:0/3043785750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d015ab60 0x7f34d00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 --2- 172.21.15.28:0/3043785750 >> v2:172.21.15.189:3300/0 conn(0x7f34d015a780 0x7f34d0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 -- 172.21.15.28:0/3043785750 >> 172.21.15.28:0/3043785750 conn(0x7f34d00b0270 msgr2=0x7f34d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 -- 172.21.15.28:0/3043785750 shutdown_connections 2024-09-13T17:08:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.096+0000 7f34d6200640 1 -- 172.21.15.28:0/3043785750 wait complete. 2024-09-13T17:08:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.097+0000 7f34d6200640 1 Processor -- start 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 -- start start 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d006bb40 0x7f34d0148d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d015a780 0x7f34d00ab610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d01e17a0 con 0x7f34d015ab60 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d01e1bc0 con 0x7f34d015a780 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.098+0000 7f34d6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d01e0b20 con 0x7f34d006bb40 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39554/0 (socket says 172.21.15.28:39554) 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 -- 172.21.15.28:0/20921464 learned_addr learned my addr 172.21.15.28:0/20921464 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cea00640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d015a780 0x7f34d00ab610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cf400640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d006bb40 0x7f34d0148d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 -- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d006bb40 msgr2=0x7f34d0148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d006bb40 0x7f34d0148d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 -- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d015a780 msgr2=0x7f34d00ab610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d015a780 0x7f34d00ab610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.099+0000 7f34cfe00640 1 -- 172.21.15.28:0/20921464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009b30 con 0x7f34d015ab60 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34cfe00640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34c8034040 tx=0x7f34c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c803d860 con 0x7f34d015ab60 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c803d9e0 con 0x7f34d015ab60 2024-09-13T17:08:42.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c803dd90 con 0x7f34d015ab60 2024-09-13T17:08:42.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d01e0e00 con 0x7f34d015ab60 2024-09-13T17:08:42.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.100+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d01e13c0 con 0x7f34d015ab60 2024-09-13T17:08:42.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.101+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d006bfd0 con 0x7f34d015ab60 2024-09-13T17:08:42.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.102+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f34c803b400 con 0x7f34d015ab60 2024-09-13T17:08:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.103+0000 7f34c7400640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 0x7f34b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.103+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34c80a9e20 con 0x7f34d015ab60 2024-09-13T17:08:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.103+0000 7f34cf400640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 0x7f34b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.104+0000 7f34cf400640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 0x7f34b006d0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.107+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f34c80750a0 con 0x7f34d015ab60 2024-09-13T17:08:42.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.284+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34d0070e40 con 0x7f34d015ab60 2024-09-13T17:08:42.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.285+0000 7f34c7400640 1 -- 172.21.15.28:0/20921464 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34c8009510 con 0x7f34d015ab60 2024-09-13T17:08:42.287 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:42.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 msgr2=0x7f34b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 0x7f34b006d0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34c0006fd0 tx=0x7f34c0008040 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 msgr2=0x7f34d00ac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f34c8034040 tx=0x7f34c8004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 shutdown_connections 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f34b006ac00 0x7f34b006d0c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34d015ab60 0x7f34d00ac6a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34d015a780 0x7f34d00ab610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 --2- 172.21.15.28:0/20921464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34d006bb40 0x7f34d0148d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 >> 172.21.15.28:0/20921464 conn(0x7f34d00b0270 msgr2=0x7f34d00ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 shutdown_connections 2024-09-13T17:08:42.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.287+0000 7f34d6200640 1 -- 172.21.15.28:0/20921464 wait complete. 2024-09-13T17:08:42.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.460+0000 7efcb6c00640 1 Processor -- start 2024-09-13T17:08:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.461+0000 7efcb6c00640 1 -- start start 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb00b3e20 0x7efcb0070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb6c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efcb00cefd0 0x7efcb015d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb6c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcb0071350 con 0x7efcb006cbe0 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb6c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcb0070ac0 con 0x7efcb00cefd0 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb006d830 con 0x7efcb00b3e20 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efcb006cbe0 0x7efcb00ce8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51558/0 (socket says 172.21.15.28:51558) 2024-09-13T17:08:42.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.462+0000 7efcb4e00640 1 -- 172.21.15.28:0/1878160644 learned_addr learned my addr 172.21.15.28:0/1878160644 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:42.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb6200640 1 --2- 172.21.15.28:0/1878160644 >> v2:172.21.15.189:3300/0 conn(0x7efcb00cefd0 0x7efcb015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb5800640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb00b3e20 0x7efcb0070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583728085 0 0) 0x7efcb0071350 con 0x7efcb006cbe0 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc98003520 con 0x7efcb006cbe0 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb6200640 1 -- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb00b3e20 msgr2=0x7efcb0070e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826524493 0 0) 0x7efc98003520 con 0x7efcb006cbe0 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb6200640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb00b3e20 0x7efcb0070e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb6200640 1 -- 172.21.15.28:0/1878160644 >> v1:172.21.15.28:6789/0 conn(0x7efcb006cbe0 legacy=0x7efcb00ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.463+0000 7efcb6200640 1 -- 172.21.15.28:0/1878160644 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb006feb0 con 0x7efcb00cefd0 2024-09-13T17:08:42.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.464+0000 7efcb6200640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00cefd0 0x7efcb015d010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bae52c90cc0fe1d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.464+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9c001670 con 0x7efcb00cefd0 2024-09-13T17:08:42.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca802d050 con 0x7efcb00cefd0 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcafe00640 1 -- 172.21.15.28:0/1878160644 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca8007740 con 0x7efcb00cefd0 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 -- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00cefd0 msgr2=0x7efcb015d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00cefd0 0x7efcb015d010 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efca8009d90 tx=0x7efca802a5f0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 -- 172.21.15.28:0/1878160644 shutdown_connections 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00cefd0 0x7efcb015d010 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 --2- 172.21.15.28:0/1878160644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb00b3e20 0x7efcb0070e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 -- 172.21.15.28:0/1878160644 >> 172.21.15.28:0/1878160644 conn(0x7efcb0069e80 msgr2=0x7efcb006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 -- 172.21.15.28:0/1878160644 shutdown_connections 2024-09-13T17:08:42.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.465+0000 7efcb6c00640 1 -- 172.21.15.28:0/1878160644 wait complete. 2024-09-13T17:08:42.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.467+0000 7efcb6c00640 1 Processor -- start 2024-09-13T17:08:42.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.467+0000 7efcb6c00640 1 -- start start 2024-09-13T17:08:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb006cbe0 0x7efcb0153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 0x7efcb009b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb009ac40 con 0x7efcb00cefd0 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb0154510 con 0x7efcb00b3e20 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcb009a960 con 0x7efcb006cbe0 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39570/0 (socket says 172.21.15.28:39570) 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb6200640 1 -- 172.21.15.28:0/2217149005 learned_addr learned my addr 172.21.15.28:0/2217149005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb4e00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 0x7efcb009b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.468+0000 7efcb5800640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb006cbe0 0x7efcb0153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 -- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb006cbe0 msgr2=0x7efcb0153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb006cbe0 0x7efcb0153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 -- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 msgr2=0x7efcb009b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 0x7efcb009b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 -- 172.21.15.28:0/2217149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca8009a70 con 0x7efcb00cefd0 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb4e00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 0x7efcb009b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:42.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6200640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efca8009eb0 tx=0x7efca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca803b820 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca803b980 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb0157440 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca803bd10 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.469+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb0157a00 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.470+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb006d370 con 0x7efcb00cefd0 2024-09-13T17:08:42.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.471+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efca802ad50 con 0x7efcb00cefd0 2024-09-13T17:08:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.472+0000 7efcad600640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 0x7efc9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.473+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efca80aa360 con 0x7efcb00cefd0 2024-09-13T17:08:42.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.473+0000 7efcb5800640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 0x7efc9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.474+0000 7efcb5800640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 0x7efc9006d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efca4009b40 tx=0x7efca4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.477+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efca80755d0 con 0x7efcb00cefd0 2024-09-13T17:08:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.653+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efcb006d020 con 0x7efcb00cefd0 2024-09-13T17:08:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.654+0000 7efcad600640 1 -- 172.21.15.28:0/2217149005 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7efca8078bf0 con 0x7efcb00cefd0 2024-09-13T17:08:42.656 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 msgr2=0x7efc9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 0x7efc9006d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efca4009b40 tx=0x7efca4009190 comp rx=0 tx=0).stop 2024-09-13T17:08:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 msgr2=0x7efcb015f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7efca8009eb0 tx=0x7efca8004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 shutdown_connections 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc9006ac00 0x7efc9006d0c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcb00cefd0 0x7efcb015f1e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcb00b3e20 0x7efcb009b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 --2- 172.21.15.28:0/2217149005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcb006cbe0 0x7efcb0153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 >> 172.21.15.28:0/2217149005 conn(0x7efcb0069e80 msgr2=0x7efcb00afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 shutdown_connections 2024-09-13T17:08:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.656+0000 7efcb6c00640 1 -- 172.21.15.28:0/2217149005 wait complete. 2024-09-13T17:08:42.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.828+0000 7fbab8000640 1 Processor -- start 2024-09-13T17:08:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.828+0000 7fbab8000640 1 -- start start 2024-09-13T17:08:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab8000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 0x7fbab015d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab8000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 0x7fbab0160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab8000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbab0146550 con 0x7fbab00ac850 2024-09-13T17:08:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab8000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab015cd20 con 0x7fbab0160310 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab8000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab015d280 con 0x7fbab0146170 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 0x7fbab0160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbab00ac850 0x7fbab006b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51564/0 (socket says 172.21.15.28:51564) 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.829+0000 7fbab6000640 1 -- 172.21.15.28:0/536801005 learned_addr learned my addr 172.21.15.28:0/536801005 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab5600640 1 --2- 172.21.15.28:0/536801005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 0x7fbab015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbaafe00640 1 -- 172.21.15.28:0/536801005 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017919924 0 0) 0x7fbab0146550 con 0x7fbab00ac850 2024-09-13T17:08:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbaafe00640 1 -- 172.21.15.28:0/536801005 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba98003520 con 0x7fbab00ac850 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab5600640 1 -- 172.21.15.28:0/536801005 >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 msgr2=0x7fbab0160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab5600640 1 --2- 172.21.15.28:0/536801005 >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 0x7fbab0160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab5600640 1 -- 172.21.15.28:0/536801005 >> v1:172.21.15.28:6789/0 conn(0x7fbab00ac850 legacy=0x7fbab006b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab5600640 1 -- 172.21.15.28:0/536801005 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab015dc50 con 0x7fbab0146170 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.830+0000 7fbab4c00640 1 --2- 172.21.15.28:0/536801005 >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 0x7fbab0160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.831+0000 7fbab5600640 1 --2- 172.21.15.28:0/536801005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 0x7fbab015d710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40097a0 tx=0x7fbaa402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f5bfe1cc3a205db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.832+0000 7fbaafe00640 1 -- 172.21.15.28:0/536801005 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032440 con 0x7fbab0146170 2024-09-13T17:08:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.832+0000 7fbaafe00640 1 -- 172.21.15.28:0/536801005 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa40325a0 con 0x7fbab0146170 2024-09-13T17:08:42.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.832+0000 7fbaafe00640 1 -- 172.21.15.28:0/536801005 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa4032930 con 0x7fbab0146170 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.832+0000 7fbab8000640 1 -- 172.21.15.28:0/536801005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 msgr2=0x7fbab015d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.832+0000 7fbab8000640 1 --2- 172.21.15.28:0/536801005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 0x7fbab015d710 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbaa40097a0 tx=0x7fbaa402a180 comp rx=0 tx=0).stop 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 -- 172.21.15.28:0/536801005 shutdown_connections 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 --2- 172.21.15.28:0/536801005 >> v2:172.21.15.28:3300/0 conn(0x7fbab0160310 0x7fbab0160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 --2- 172.21.15.28:0/536801005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab0146170 0x7fbab015d710 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 -- 172.21.15.28:0/536801005 >> 172.21.15.28:0/536801005 conn(0x7fbab0069e80 msgr2=0x7fbab006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 -- 172.21.15.28:0/536801005 shutdown_connections 2024-09-13T17:08:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.833+0000 7fbab8000640 1 -- 172.21.15.28:0/536801005 wait complete. 2024-09-13T17:08:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.834+0000 7fbab8000640 1 Processor -- start 2024-09-13T17:08:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.834+0000 7fbab8000640 1 -- start start 2024-09-13T17:08:42.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab00ac850 0x7fbab01f28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbab0160310 0x7fbab00ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01f6770 con 0x7fbab0146170 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01f5bf0 con 0x7fbab0160310 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab8000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab01f5ed0 con 0x7fbab00ac850 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.835+0000 7fbab4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39582/0 (socket says 172.21.15.28:39582) 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 -- 172.21.15.28:0/1678450510 learned_addr learned my addr 172.21.15.28:0/1678450510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab5600640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab00ac850 0x7fbab01f28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab6000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbab0160310 0x7fbab00ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 -- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab00ac850 msgr2=0x7fbab01f28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab00ac850 0x7fbab01f28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 -- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbab0160310 msgr2=0x7fbab00ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbab0160310 0x7fbab00ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 -- 172.21.15.28:0/1678450510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9c009400 con 0x7fbab0146170 2024-09-13T17:08:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbab4c00640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00ebb0 tx=0x7fba9c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00eeb0 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba9c00cc30 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.836+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba9c013470 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.837+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa4009400 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.837+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab01f6490 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.838+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab015d6f0 con 0x7fbab0146170 2024-09-13T17:08:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.839+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fba9c0135d0 con 0x7fbab0146170 2024-09-13T17:08:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.839+0000 7fbaad600640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 0x7fba9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.840+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba9c0895c0 con 0x7fbab0146170 2024-09-13T17:08:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.840+0000 7fbab5600640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 0x7fba9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.841+0000 7fbab5600640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 0x7fba9006d0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4034000 tx=0x7fbaa402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:42.844+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fba9c054830 con 0x7fbab0146170 2024-09-13T17:08:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.026+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbab015e3b0 con 0x7fbab0146170 2024-09-13T17:08:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.027+0000 7fbaad600640 1 -- 172.21.15.28:0/1678450510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fba9c057e50 con 0x7fbab0146170 2024-09-13T17:08:43.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:43.030 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 msgr2=0x7fba9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 0x7fba9006d0c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbaa4034000 tx=0x7fbaa402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 msgr2=0x7fbab01f2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fba9c00ebb0 tx=0x7fba9c00c520 comp rx=0 tx=0).stop 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 shutdown_connections 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9006ac00 0x7fba9006d0c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbab0160310 0x7fbab00ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbab0146170 0x7fbab01f2e00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 --2- 172.21.15.28:0/1678450510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbab00ac850 0x7fbab01f28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 >> 172.21.15.28:0/1678450510 conn(0x7fbab0069e80 msgr2=0x7fbab00b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 shutdown_connections 2024-09-13T17:08:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.030+0000 7fbab8000640 1 -- 172.21.15.28:0/1678450510 wait complete. 2024-09-13T17:08:43.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:43.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.206+0000 7fc9be400640 1 Processor -- start 2024-09-13T17:08:43.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.206+0000 7fc9be400640 1 -- start start 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9be400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc9b815c910 0x7fc9b8160e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9be400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9b80732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9be400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9b80737e0 con 0x7fc9b815ccf0 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9be400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9b8160990 con 0x7fc9b815c910 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9be400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b815d930 con 0x7fc9b8072e40 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9bda00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc9b815ccf0 0x7fc9b8069e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51576/0 (socket says 172.21.15.28:51576) 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9bda00640 1 -- 172.21.15.28:0/3919451294 learned_addr learned my addr 172.21.15.28:0/3919451294 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:43.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9b7e00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9b80732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3919451294 >> v2:172.21.15.189:3300/0 conn(0x7fc9b815c910 0x7fc9b8160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.208+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804109946 0 0) 0x7fc9b80737e0 con 0x7fc9b815ccf0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a0003520 con 0x7fc9b815ccf0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381738040 0 0) 0x7fc9a0003520 con 0x7fc9b815ccf0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 msgr2=0x7fc9b80732a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9b80732a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 >> v2:172.21.15.189:3300/0 conn(0x7fc9b815c910 msgr2=0x7fc9b8160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 --2- 172.21.15.28:0/3919451294 >> v2:172.21.15.189:3300/0 conn(0x7fc9b815c910 0x7fc9b8160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b815dc10 con 0x7fc9b815ccf0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc9b00019f0 con 0x7fc9b815ccf0 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 >> v1:172.21.15.28:6789/0 conn(0x7fc9b815ccf0 legacy=0x7fc9b8069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0003970 0x7fc9a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a0004d00 0x7fc9a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8160990 con 0x7fc9a0004d00 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b815d930 con 0x7fc9a0003970 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a000b980 con 0x7fc9b8072e40 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3919451294 >> v2:172.21.15.189:3300/0 conn(0x7fc9b815c910 0x7fc9b8160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9bda00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a0004d00 0x7fc9a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.209+0000 7fc9b7e00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0003970 0x7fc9a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bda00640 1 -- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 msgr2=0x7fc9a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bda00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8072e40 0x7fc9a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bda00640 1 -- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0003970 msgr2=0x7fc9a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bda00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0003970 0x7fc9a0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.210+0000 7fc9bda00640 1 -- 172.21.15.28:0/3919451294 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a0003520 con 0x7fc9a0004d00 2024-09-13T17:08:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9bda00640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a0004d00 0x7fc9a0005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000a860 tx=0x7fc9a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f843d9207bbc35b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b002ba80 con 0x7fc9a0004d00 2024-09-13T17:08:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b002bbe0 con 0x7fc9a0004d00 2024-09-13T17:08:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9b7400640 1 -- 172.21.15.28:0/3919451294 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b0034470 con 0x7fc9a0004d00 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9be400640 1 -- 172.21.15.28:0/3919451294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a0004d00 msgr2=0x7fc9a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.211+0000 7fc9be400640 1 --2- 172.21.15.28:0/3919451294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9a0004d00 0x7fc9a0005180 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9b000a860 tx=0x7fc9a00057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.212+0000 7fc9bda00640 1 -- 172.21.15.28:0/3919451294 reap_dead start 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.212+0000 7fc9be400640 1 -- 172.21.15.28:0/3919451294 shutdown_connections 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.212+0000 7fc9be400640 1 -- 172.21.15.28:0/3919451294 >> 172.21.15.28:0/3919451294 conn(0x7fc9b80cbec0 msgr2=0x7fc9b80cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.212+0000 7fc9be400640 1 -- 172.21.15.28:0/3919451294 shutdown_connections 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.212+0000 7fc9be400640 1 -- 172.21.15.28:0/3919451294 wait complete. 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.213+0000 7fc9be400640 1 Processor -- start 2024-09-13T17:08:43.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.214+0000 7fc9be400640 1 -- start start 2024-09-13T17:08:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.214+0000 7fc9be400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 0x7fc9b8147fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.214+0000 7fc9be400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9be400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8149620 0x7fc9b8158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9be400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8148580 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9be400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8148860 con 0x7fc9a0004d00 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9be400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b8148b40 con 0x7fc9b8149620 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39596/0 (socket says 172.21.15.28:39596) 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 -- 172.21.15.28:0/3358221909 learned_addr learned my addr 172.21.15.28:0/3358221909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 -- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8149620 msgr2=0x7fc9b8158830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9bda00640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8149620 0x7fc9b8158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8149620 0x7fc9b8158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 0x7fc9b8147fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 -- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 msgr2=0x7fc9b8147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 0x7fc9b8147fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9b7e00640 1 -- 172.21.15.28:0/3358221909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b8148e20 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.215+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 0x7fc9b8147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9b7e00640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4011910 tx=0x7fc9a4011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a4009590 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9a400faf0 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b8149100 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a400fe60 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.216+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b8147b40 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.217+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9b8160e20 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.218+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc9a4014440 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.219+0000 7fc9b4c00640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 0x7fc99806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.219+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc9a4089bf0 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.219+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 0x7fc99806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.220+0000 7fc9bd000640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 0x7fc99806d0f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c40 tx=0x7fc9ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.223+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc9a4054dc0 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.402+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc9b806b810 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.402+0000 7fc9b4c00640 1 -- 172.21.15.28:0/3358221909 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc9a40583e0 con 0x7fc9b80b4ea0 2024-09-13T17:08:43.404 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 msgr2=0x7fc99806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 0x7fc99806d0f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc9ac009c40 tx=0x7fc9ac008040 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 msgr2=0x7fc9b8147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc9a4011910 tx=0x7fc9a4011dd0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 shutdown_connections 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc99806ac30 0x7fc99806d0f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc9b8149620 0x7fc9b8158830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc9b80b4ea0 0x7fc9b8147600 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 --2- 172.21.15.28:0/3358221909 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9a0004d00 0x7fc9b8147fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 >> 172.21.15.28:0/3358221909 conn(0x7fc9b80cbec0 msgr2=0x7fc9b80cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 shutdown_connections 2024-09-13T17:08:43.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.405+0000 7fc9be400640 1 -- 172.21.15.28:0/3358221909 wait complete. 2024-09-13T17:08:43.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.580+0000 7f037b200640 1 Processor -- start 2024-09-13T17:08:43.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.580+0000 7f037b200640 1 -- start start 2024-09-13T17:08:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.581+0000 7f037b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f037c06a2f0 0x7f037c0b6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.581+0000 7f037b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c0b1f50 con 0x7f037c069e80 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.581+0000 7f037b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f037c15cca0 con 0x7f037c15c8c0 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.581+0000 7f037b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f037c0d25a0 con 0x7f037c06a2f0 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f0379400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f037c069e80 0x7f037c0b6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51584/0 (socket says 172.21.15.28:51584) 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f0379400640 1 -- 172.21.15.28:0/2769661057 learned_addr learned my addr 172.21.15.28:0/2769661057 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f037a800640 1 --2- 172.21.15.28:0/2769661057 >> v2:172.21.15.189:3300/0 conn(0x7f037c06a2f0 0x7f037c0b6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f0378a00640 1 -- 172.21.15.28:0/2769661057 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416244043 0 0) 0x7f0368001110 con 0x7f037c069e80 2024-09-13T17:08:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f0378a00640 1 -- 172.21.15.28:0/2769661057 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0360003520 con 0x7f037c069e80 2024-09-13T17:08:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f037a800640 1 -- 172.21.15.28:0/2769661057 >> v1:172.21.15.189:6789/0 conn(0x7f037c15c8c0 legacy=0x7f037c0d1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f037a800640 1 -- 172.21.15.28:0/2769661057 >> v1:172.21.15.28:6789/0 conn(0x7f037c069e80 legacy=0x7f037c0b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.582+0000 7f037a800640 1 -- 172.21.15.28:0/2769661057 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c0d1fe0 con 0x7f037c06a2f0 2024-09-13T17:08:43.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.583+0000 7f037a800640 1 --2- 172.21.15.28:0/2769661057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c06a2f0 0x7f037c0b6910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0374009d90 tx=0x7f037402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a6cfd6da9314b07 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f0378a00640 1 -- 172.21.15.28:0/2769661057 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374007740 con 0x7f037c06a2f0 2024-09-13T17:08:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f0378a00640 1 -- 172.21.15.28:0/2769661057 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037402d050 con 0x7f037c06a2f0 2024-09-13T17:08:43.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f0378a00640 1 -- 172.21.15.28:0/2769661057 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374007aa0 con 0x7f037c06a2f0 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f037b200640 1 -- 172.21.15.28:0/2769661057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c06a2f0 msgr2=0x7f037c0b6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f037b200640 1 --2- 172.21.15.28:0/2769661057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c06a2f0 0x7f037c0b6910 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f0374009d90 tx=0x7f037402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f037b200640 1 -- 172.21.15.28:0/2769661057 shutdown_connections 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f037b200640 1 --2- 172.21.15.28:0/2769661057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c06a2f0 0x7f037c0b6910 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.584+0000 7f037b200640 1 -- 172.21.15.28:0/2769661057 >> 172.21.15.28:0/2769661057 conn(0x7f037c0708c0 msgr2=0x7f037c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.585+0000 7f037b200640 1 -- 172.21.15.28:0/2769661057 shutdown_connections 2024-09-13T17:08:43.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.585+0000 7f037b200640 1 -- 172.21.15.28:0/2769661057 wait complete. 2024-09-13T17:08:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.586+0000 7f037b200640 1 Processor -- start 2024-09-13T17:08:43.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.586+0000 7f037b200640 1 -- start start 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c069e80 0x7f037c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f037c06a2f0 0x7f037c152700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c152c40 con 0x7f037c15c8c0 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c142950 con 0x7f037c069e80 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.587+0000 7f037b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c142c30 con 0x7f037c06a2f0 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f0379400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f037c06a2f0 0x7f037c152700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39608/0 (socket says 172.21.15.28:39608) 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 -- 172.21.15.28:0/3794704062 learned_addr learned my addr 172.21.15.28:0/3794704062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f0379e00640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c069e80 0x7f037c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 -- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f037c06a2f0 msgr2=0x7f037c152700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f037c06a2f0 0x7f037c152700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 -- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c069e80 msgr2=0x7f037c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c069e80 0x7f037c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.588+0000 7f037a800640 1 -- 172.21.15.28:0/3794704062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374009a70 con 0x7f037c15c8c0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f037a800640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0374009d90 tx=0x7f0374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374003b90 con 0x7f037c15c8c0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0374003cf0 con 0x7f037c15c8c0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037403c910 con 0x7f037c15c8c0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037c0b3350 con 0x7f037c15c8c0 2024-09-13T17:08:43.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.589+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c1598d0 con 0x7f037c15c8c0 2024-09-13T17:08:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.590+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c0b3870 con 0x7f037c15c8c0 2024-09-13T17:08:43.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.591+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f037403ca70 con 0x7f037c15c8c0 2024-09-13T17:08:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.592+0000 7f0372000640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 0x7f035c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.592+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03740aa4f0 con 0x7f037c15c8c0 2024-09-13T17:08:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.592+0000 7f0379e00640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 0x7f035c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.593+0000 7f0379e00640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 0x7f035c06d0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f036c000f60 tx=0x7f036c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.596+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f037c0b3870 con 0x7f037c15c8c0 2024-09-13T17:08:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.777+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f037c0b3870 con 0x7f037c15c8c0 2024-09-13T17:08:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.778+0000 7f0372000640 1 -- 172.21.15.28:0/3794704062 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f037c0b3870 con 0x7f037c15c8c0 2024-09-13T17:08:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 msgr2=0x7f035c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 0x7f035c06d0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f036c000f60 tx=0x7f036c009040 comp rx=0 tx=0).stop 2024-09-13T17:08:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 msgr2=0x7f037c158c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0374009d90 tx=0x7f0374004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:43.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 shutdown_connections 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f035c06ac00 0x7f035c06d0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f037c15c8c0 0x7f037c158c10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f037c06a2f0 0x7f037c152700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 --2- 172.21.15.28:0/3794704062 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f037c069e80 0x7f037c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 >> 172.21.15.28:0/3794704062 conn(0x7f037c0708c0 msgr2=0x7f037c15a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 shutdown_connections 2024-09-13T17:08:43.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.780+0000 7f037b200640 1 -- 172.21.15.28:0/3794704062 wait complete. 2024-09-13T17:08:43.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:43.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.953+0000 7fe551200640 1 Processor -- start 2024-09-13T17:08:43.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.953+0000 7fe551200640 1 -- start start 2024-09-13T17:08:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.954+0000 7fe551200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c159770 0x7fe54c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.954+0000 7fe551200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.954+0000 7fe551200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54c06f920 con 0x7fe54c06f3f0 2024-09-13T17:08:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.954+0000 7fe551200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c15b290 con 0x7fe54c159770 2024-09-13T17:08:43.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.954+0000 7fe551200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c15b570 con 0x7fe54c15cd80 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c159770 0x7fe54c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c159770 0x7fe54c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33564/0 (socket says 172.21.15.28:33564) 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 -- 172.21.15.28:0/3597181248 learned_addr learned my addr 172.21.15.28:0/3597181248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe54a800640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 -- 172.21.15.28:0/3597181248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 msgr2=0x7fe54c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 -- 172.21.15.28:0/3597181248 >> v1:172.21.15.189:6789/0 conn(0x7fe54c06f3f0 legacy=0x7fe54c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.955+0000 7fe549e00640 1 -- 172.21.15.28:0/3597181248 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c15b850 con 0x7fe54c159770 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.956+0000 7fe549e00640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c06fd20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009870 tx=0x7fe53c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4323d130bda343 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.956+0000 7fe548a00640 1 -- 172.21.15.28:0/3597181248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c032440 con 0x7fe54c159770 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.956+0000 7fe548a00640 1 -- 172.21.15.28:0/3597181248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c0325a0 con 0x7fe54c159770 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.956+0000 7fe548a00640 1 -- 172.21.15.28:0/3597181248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c032930 con 0x7fe54c159770 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 -- 172.21.15.28:0/3597181248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 msgr2=0x7fe54c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c06fd20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009870 tx=0x7fe53c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 -- 172.21.15.28:0/3597181248 shutdown_connections 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 --2- 172.21.15.28:0/3597181248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c06fd20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 -- 172.21.15.28:0/3597181248 >> 172.21.15.28:0/3597181248 conn(0x7fe54c0b0270 msgr2=0x7fe54c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 -- 172.21.15.28:0/3597181248 shutdown_connections 2024-09-13T17:08:43.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.957+0000 7fe551200640 1 -- 172.21.15.28:0/3597181248 wait complete. 2024-09-13T17:08:43.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.959+0000 7fe551200640 1 Processor -- start 2024-09-13T17:08:43.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.959+0000 7fe551200640 1 -- start start 2024-09-13T17:08:43.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c06f3f0 0x7fe54c145810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c0acd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c146ab0 con 0x7fe54c159770 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c156280 con 0x7fe54c06f3f0 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe551200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0ad2a0 con 0x7fe54c15cd80 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe549400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe549400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33568/0 (socket says 172.21.15.28:33568) 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe549400640 1 -- 172.21.15.28:0/1336069327 learned_addr learned my addr 172.21.15.28:0/1336069327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe549e00640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c06f3f0 0x7fe54c145810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.960+0000 7fe54a800640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c0acd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 -- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 msgr2=0x7fe54c0acd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c0acd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 -- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c06f3f0 msgr2=0x7fe54c145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c06f3f0 0x7fe54c145810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 -- 172.21.15.28:0/1336069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0094d0 con 0x7fe54c159770 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe54a800640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe549400640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c400 tx=0x7fe53800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380052d0 con 0x7fe54c159770 2024-09-13T17:08:43.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538005430 con 0x7fe54c159770 2024-09-13T17:08:43.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538003830 con 0x7fe54c159770 2024-09-13T17:08:43.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c145ea0 con 0x7fe54c159770 2024-09-13T17:08:43.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.961+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c146460 con 0x7fe54c159770 2024-09-13T17:08:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.962+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c146c20 con 0x7fe54c159770 2024-09-13T17:08:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.963+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe538003990 con 0x7fe54c159770 2024-09-13T17:08:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.964+0000 7fe542000640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.964+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe538088820 con 0x7fe54c159770 2024-09-13T17:08:43.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.965+0000 7fe549e00640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:43.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.966+0000 7fe549e00640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe53c034000 tx=0x7fe53c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:43.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:43.968+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe538053ab0 con 0x7fe54c159770 2024-09-13T17:08:44.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.146+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe54c06fd20 con 0x7fe54c159770 2024-09-13T17:08:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.147+0000 7fe542000640 1 -- 172.21.15.28:0/1336069327 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe5380170a0 con 0x7fe54c159770 2024-09-13T17:08:44.149 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 msgr2=0x7fe52c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe53c034000 tx=0x7fe53c02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 msgr2=0x7fe54c155d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe53800c400 tx=0x7fe53800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 shutdown_connections 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c15cd80 0x7fe54c0acd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c159770 0x7fe54c155d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 --2- 172.21.15.28:0/1336069327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c06f3f0 0x7fe54c145810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 >> 172.21.15.28:0/1336069327 conn(0x7fe54c0b0270 msgr2=0x7fe54c0af480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 shutdown_connections 2024-09-13T17:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.149+0000 7fe551200640 1 -- 172.21.15.28:0/1336069327 wait complete. 2024-09-13T17:08:44.192 INFO:tasks.cephfs_test_runner:Starting test: test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) test flag scrub_mdsdir 2024-09-13T17:08:44.192 INFO:tasks.cephfs_test_runner:test_flag_scrub_mdsdir (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:08:44.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.348+0000 7fa621600640 1 Processor -- start 2024-09-13T17:08:44.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.348+0000 7fa621600640 1 -- start start 2024-09-13T17:08:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa621600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa61c15a540 0x7fa61c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa621600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa61c0cabc0 0x7fa61c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa621600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c15e100 con 0x7fa61c15edb0 2024-09-13T17:08:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa621600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0adb00 con 0x7fa61c15a540 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa621600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c15e3e0 con 0x7fa61c0cabc0 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa61c15a540 0x7fa61c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa61c15edb0 0x7fa61c070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42078/0 (socket says 172.21.15.28:42078) 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa61c15a540 0x7fa61c06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33572/0 (socket says 172.21.15.28:33572) 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61b400640 1 -- 172.21.15.28:0/1765514404 learned_addr learned my addr 172.21.15.28:0/1765514404 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa613e00640 1 --2- 172.21.15.28:0/1765514404 >> v2:172.21.15.189:3300/0 conn(0x7fa61c0cabc0 0x7fa61c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 -- 172.21.15.28:0/1765514404 >> v2:172.21.15.189:3300/0 conn(0x7fa61c0cabc0 msgr2=0x7fa61c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 --2- 172.21.15.28:0/1765514404 >> v2:172.21.15.189:3300/0 conn(0x7fa61c0cabc0 0x7fa61c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 -- 172.21.15.28:0/1765514404 >> v1:172.21.15.28:6789/0 conn(0x7fa61c15edb0 legacy=0x7fa61c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.350+0000 7fa61aa00640 1 -- 172.21.15.28:0/1765514404 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c06ce80 con 0x7fa61c15a540 2024-09-13T17:08:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa61aa00640 1 --2- 172.21.15.28:0/1765514404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c15a540 0x7fa61c06c430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0ae250 tx=0x7fa60c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed88eddc3b501b61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa61a000640 1 -- 172.21.15.28:0/1765514404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c02ae60 con 0x7fa61c15a540 2024-09-13T17:08:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa61a000640 1 -- 172.21.15.28:0/1765514404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60c032440 con 0x7fa61c15a540 2024-09-13T17:08:44.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa61a000640 1 -- 172.21.15.28:0/1765514404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60c0327d0 con 0x7fa61c15a540 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa621600640 1 -- 172.21.15.28:0/1765514404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c15a540 msgr2=0x7fa61c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.352+0000 7fa621600640 1 --2- 172.21.15.28:0/1765514404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c15a540 0x7fa61c06c430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa61c0ae250 tx=0x7fa60c02a110 comp rx=0 tx=0).stop 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 -- 172.21.15.28:0/1765514404 shutdown_connections 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 --2- 172.21.15.28:0/1765514404 >> v2:172.21.15.189:3300/0 conn(0x7fa61c0cabc0 0x7fa61c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 --2- 172.21.15.28:0/1765514404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c15a540 0x7fa61c06c430 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 -- 172.21.15.28:0/1765514404 >> 172.21.15.28:0/1765514404 conn(0x7fa61c069e80 msgr2=0x7fa61c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:44.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 -- 172.21.15.28:0/1765514404 shutdown_connections 2024-09-13T17:08:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.353+0000 7fa621600640 1 -- 172.21.15.28:0/1765514404 wait complete. 2024-09-13T17:08:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.354+0000 7fa621600640 1 Processor -- start 2024-09-13T17:08:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.354+0000 7fa621600640 1 -- start start 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 0x7fa61c142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa61c15a540 0x7fa61c0af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c142570 con 0x7fa61c14bfe0 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c156760 con 0x7fa61c0cabc0 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.355+0000 7fa621600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c156a40 con 0x7fa61c15a540 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33578/0 (socket says 172.21.15.28:33578) 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 -- 172.21.15.28:0/368789785 learned_addr learned my addr 172.21.15.28:0/368789785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61aa00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 0x7fa61c142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa613e00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa61c15a540 0x7fa61c0af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 -- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa61c15a540 msgr2=0x7fa61c0af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa61c15a540 0x7fa61c0af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 -- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 msgr2=0x7fa61c142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 0x7fa61c142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61b400640 1 -- 172.21.15.28:0/368789785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c009400 con 0x7fa61c14bfe0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.356+0000 7fa61aa00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 0x7fa61c142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa61b400640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa61400caa0 tx=0x7fa61400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61400ac00 con 0x7fa61c14bfe0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61400ad60 con 0x7fa61c14bfe0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa614012600 con 0x7fa61c14bfe0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c156d20 con 0x7fa61c14bfe0 2024-09-13T17:08:44.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.357+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c1572e0 con 0x7fa61c14bfe0 2024-09-13T17:08:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.358+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c1574d0 con 0x7fa61c14bfe0 2024-09-13T17:08:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.359+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa614007070 con 0x7fa61c14bfe0 2024-09-13T17:08:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.360+0000 7fa612a00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 0x7fa5f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.360+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa614089120 con 0x7fa61c14bfe0 2024-09-13T17:08:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.360+0000 7fa61aa00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 0x7fa5f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.361+0000 7fa61aa00640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 0x7fa5f806d0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa60c034000 tx=0x7fa60c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.364+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa6140543a0 con 0x7fa61c14bfe0 2024-09-13T17:08:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.545+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa61c15e850 con 0x7fa61c14bfe0 2024-09-13T17:08:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.546+0000 7fa612a00640 1 -- 172.21.15.28:0/368789785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fa6140579c0 con 0x7fa61c14bfe0 2024-09-13T17:08:44.549 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:44.549 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 msgr2=0x7fa5f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 0x7fa5f806d0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa60c034000 tx=0x7fa60c02e000 comp rx=0 tx=0).stop 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 msgr2=0x7fa61c0af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa61400caa0 tx=0x7fa61400cf60 comp rx=0 tx=0).stop 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 shutdown_connections 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5f806ac00 0x7fa5f806d0c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61c14bfe0 0x7fa61c0af9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa61c15a540 0x7fa61c0af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 --2- 172.21.15.28:0/368789785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa61c0cabc0 0x7fa61c142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 >> 172.21.15.28:0/368789785 conn(0x7fa61c069e80 msgr2=0x7fa61c0cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 shutdown_connections 2024-09-13T17:08:44.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.549+0000 7fa621600640 1 -- 172.21.15.28:0/368789785 wait complete. 2024-09-13T17:08:44.568 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:44.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:44.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.729+0000 7fc8c7e00640 1 Processor -- start 2024-09-13T17:08:44.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.730+0000 7fc8c7e00640 1 -- start start 2024-09-13T17:08:44.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.731+0000 7fc8c7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8c815c900 0x7fc8c8069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.731+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8c806a440 0x7fc8c806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c806ae10 con 0x7fc8c806a440 2024-09-13T17:08:44.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8c80ae200 con 0x7fc8c815c900 2024-09-13T17:08:44.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80ae4e0 con 0x7fc8c80b2dc0 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8c806a440 0x7fc8c806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc8c806a440 0x7fc8c806a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33586/0 (socket says 172.21.15.28:33586) 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 -- 172.21.15.28:0/1320162869 learned_addr learned my addr 172.21.15.28:0/1320162869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c6a00640 1 --2- 172.21.15.28:0/1320162869 >> v2:172.21.15.189:3300/0 conn(0x7fc8c815c900 0x7fc8c8069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8bfe00640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 -- 172.21.15.28:0/1320162869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 msgr2=0x7fc8c80af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80af2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 -- 172.21.15.28:0/1320162869 >> v2:172.21.15.189:3300/0 conn(0x7fc8c815c900 msgr2=0x7fc8c8069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 --2- 172.21.15.28:0/1320162869 >> v2:172.21.15.189:3300/0 conn(0x7fc8c815c900 0x7fc8c8069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8c7400640 1 -- 172.21.15.28:0/1320162869 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80ae7c0 con 0x7fc8c806a440 2024-09-13T17:08:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.732+0000 7fc8bfe00640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80af2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.734+0000 7fc8c7400640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c806a8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80d1690 tx=0x7fc8c002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc91c5bd9a711ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.734+0000 7fc8c6000640 1 -- 172.21.15.28:0/1320162869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0031600 con 0x7fc8c806a440 2024-09-13T17:08:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.734+0000 7fc8c6000640 1 -- 172.21.15.28:0/1320162869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8c002d050 con 0x7fc8c806a440 2024-09-13T17:08:44.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.734+0000 7fc8c6000640 1 -- 172.21.15.28:0/1320162869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c0031980 con 0x7fc8c806a440 2024-09-13T17:08:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 -- 172.21.15.28:0/1320162869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 msgr2=0x7fc8c806a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c806a8d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80d1690 tx=0x7fc8c002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 -- 172.21.15.28:0/1320162869 shutdown_connections 2024-09-13T17:08:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c806a8d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/1320162869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80af2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/1320162869 >> v2:172.21.15.189:3300/0 conn(0x7fc8c815c900 0x7fc8c8069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 -- 172.21.15.28:0/1320162869 >> 172.21.15.28:0/1320162869 conn(0x7fc8c80b2150 msgr2=0x7fc8c80b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 -- 172.21.15.28:0/1320162869 shutdown_connections 2024-09-13T17:08:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.735+0000 7fc8c7e00640 1 -- 172.21.15.28:0/1320162869 wait complete. 2024-09-13T17:08:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.738+0000 7fc8c7e00640 1 Processor -- start 2024-09-13T17:08:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.738+0000 7fc8c7e00640 1 -- start start 2024-09-13T17:08:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8c815c900 0x7fc8c80c6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c815a460 con 0x7fc8c806a440 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80b58b0 con 0x7fc8c815c900 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8c80b4a00 con 0x7fc8c80b2dc0 2024-09-13T17:08:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33590/0 (socket says 172.21.15.28:33590) 2024-09-13T17:08:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 -- 172.21.15.28:0/2534495737 learned_addr learned my addr 172.21.15.28:0/2534495737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8bfe00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c7400640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8c815c900 0x7fc8c80c6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 -- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 msgr2=0x7fc8c80b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 -- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8c815c900 msgr2=0x7fc8c80c6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.740+0000 7fc8c6a00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8c815c900 0x7fc8c80c6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8c6a00640 1 -- 172.21.15.28:0/2534495737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c0009b30 con 0x7fc8c806a440 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8c6a00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800d5f0 tx=0x7fc8b800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b80125f0 con 0x7fc8c806a440 2024-09-13T17:08:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b8012750 con 0x7fc8c806a440 2024-09-13T17:08:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8c80b4ca0 con 0x7fc8c806a440 2024-09-13T17:08:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8b8012b00 con 0x7fc8c806a440 2024-09-13T17:08:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.741+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8c80b5260 con 0x7fc8c806a440 2024-09-13T17:08:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.743+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8c815cce0 con 0x7fc8c806a440 2024-09-13T17:08:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.743+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc8b8012c60 con 0x7fc8c806a440 2024-09-13T17:08:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.745+0000 7fc8bea00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 0x7fc8a406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.745+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8b8089760 con 0x7fc8c806a440 2024-09-13T17:08:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.745+0000 7fc8bfe00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 0x7fc8a406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.746+0000 7fc8bfe00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 0x7fc8a406d0f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80c5d40 tx=0x7fc8b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.753+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc8b80549d0 con 0x7fc8c806a440 2024-09-13T17:08:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.943+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc8c80ce8e0 con 0x7fc8c806a440 2024-09-13T17:08:44.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.943+0000 7fc8bea00640 1 -- 172.21.15.28:0/2534495737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc8b8057ff0 con 0x7fc8c806a440 2024-09-13T17:08:44.945 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.945+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 msgr2=0x7fc8a406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.945+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 0x7fc8a406d0f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80c5d40 tx=0x7fc8b0008040 comp rx=0 tx=0).stop 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 msgr2=0x7fc8c8159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc8b800d5f0 tx=0x7fc8b800dab0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 shutdown_connections 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8a406ac30 0x7fc8a406d0f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8c815c900 0x7fc8c80c6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8c80b2dc0 0x7fc8c80b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 --2- 172.21.15.28:0/2534495737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8c806a440 0x7fc8c8159f20 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 >> 172.21.15.28:0/2534495737 conn(0x7fc8c80b2150 msgr2=0x7fc8c806af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 shutdown_connections 2024-09-13T17:08:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:44.946+0000 7fc8c7e00640 1 -- 172.21.15.28:0/2534495737 wait complete. 2024-09-13T17:08:44.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.125+0000 7fb166000640 1 Processor -- start 2024-09-13T17:08:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.125+0000 7fb166000640 1 -- start start 2024-09-13T17:08:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.126+0000 7fb166000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb16006f3f0 0x7fb16015c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.126+0000 7fb166000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16015ab60 con 0x7fb160070430 2024-09-13T17:08:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.126+0000 7fb166000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16006f920 con 0x7fb160159770 2024-09-13T17:08:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.126+0000 7fb166000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16015b290 con 0x7fb16006f3f0 2024-09-13T17:08:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb157e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb160070430 0x7fb16015bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42090/0 (socket says 172.21.15.28:42090) 2024-09-13T17:08:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb157e00640 1 -- 172.21.15.28:0/3408838955 learned_addr learned my addr 172.21.15.28:0/3408838955 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:45.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15fe00640 1 --2- 172.21.15.28:0/3408838955 >> v2:172.21.15.189:3300/0 conn(0x7fb16006f3f0 0x7fb16015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503180879 0 0) 0x7fb148001110 con 0x7fb160070430 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb144003520 con 0x7fb160070430 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993476795 0 0) 0x7fb16006f920 con 0x7fb160159770 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb148001110 con 0x7fb160159770 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15fe00640 1 -- 172.21.15.28:0/3408838955 >> v1:172.21.15.189:6789/0 conn(0x7fb160159770 legacy=0x7fb16006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15fe00640 1 -- 172.21.15.28:0/3408838955 >> v1:172.21.15.28:6789/0 conn(0x7fb160070430 legacy=0x7fb16015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.127+0000 7fb15fe00640 1 -- 172.21.15.28:0/3408838955 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16015b570 con 0x7fb16006f3f0 2024-09-13T17:08:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.128+0000 7fb15fe00640 1 --2- 172.21.15.28:0/3408838955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16015c480 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb158009d90 tx=0x7fb15802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a748c9f21a63eebc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158007740 con 0x7fb16006f3f0 2024-09-13T17:08:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15802d050 con 0x7fb16006f3f0 2024-09-13T17:08:45.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb15ea00640 1 -- 172.21.15.28:0/3408838955 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158007ac0 con 0x7fb16006f3f0 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 -- 172.21.15.28:0/3408838955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 msgr2=0x7fb16015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 --2- 172.21.15.28:0/3408838955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16015c480 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb158009d90 tx=0x7fb15802a660 comp rx=0 tx=0).stop 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 -- 172.21.15.28:0/3408838955 shutdown_connections 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 --2- 172.21.15.28:0/3408838955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16015c480 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 -- 172.21.15.28:0/3408838955 >> 172.21.15.28:0/3408838955 conn(0x7fb1600b0270 msgr2=0x7fb1600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 -- 172.21.15.28:0/3408838955 shutdown_connections 2024-09-13T17:08:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.129+0000 7fb166000640 1 -- 172.21.15.28:0/3408838955 wait complete. 2024-09-13T17:08:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.131+0000 7fb166000640 1 Processor -- start 2024-09-13T17:08:45.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.131+0000 7fb166000640 1 -- start start 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb160070430 0x7fb1601e1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16014eaf0 con 0x7fb160159770 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1601e1bb0 con 0x7fb16006f3f0 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb166000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1601f5d00 con 0x7fb160070430 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.132+0000 7fb15fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33594/0 (socket says 172.21.15.28:33594) 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 -- 172.21.15.28:0/1796351833 learned_addr learned my addr 172.21.15.28:0/1796351833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb157e00640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb160070430 0x7fb1601e1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15f400640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 -- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb160070430 msgr2=0x7fb1601e1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb160070430 0x7fb1601e1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 -- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 msgr2=0x7fb16014e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16014e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 -- 172.21.15.28:0/1796351833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb158009a70 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.133+0000 7fb15fe00640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb15802ac50 tx=0x7fb158004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.134+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15803c830 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.134+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15803c990 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.134+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15803cd00 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.134+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1601f5fe0 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.134+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1601f65a0 con 0x7fb160159770 2024-09-13T17:08:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.135+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1600ce310 con 0x7fb160159770 2024-09-13T17:08:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.136+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb158031400 con 0x7fb160159770 2024-09-13T17:08:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.137+0000 7fb157400640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 0x7fb13c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.137+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb1580aae30 con 0x7fb160159770 2024-09-13T17:08:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.138+0000 7fb15f400640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 0x7fb13c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.138+0000 7fb15f400640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 0x7fb13c06d0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1500011e0 tx=0x7fb150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.141+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb158076140 con 0x7fb160159770 2024-09-13T17:08:45.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.319+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb16006f920 con 0x7fb160159770 2024-09-13T17:08:45.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.320+0000 7fb157400640 1 -- 172.21.15.28:0/1796351833 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb158040020 con 0x7fb160159770 2024-09-13T17:08:45.322 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:45.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 msgr2=0x7fb13c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 0x7fb13c06d0c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb1500011e0 tx=0x7fb150009040 comp rx=0 tx=0).stop 2024-09-13T17:08:45.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 msgr2=0x7fb1600ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb15802ac50 tx=0x7fb158004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 shutdown_connections 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13c06ac00 0x7fb13c06d0c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb160159770 0x7fb1600ac850 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb160070430 0x7fb1601e1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 --2- 172.21.15.28:0/1796351833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb16006f3f0 0x7fb16014e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 >> 172.21.15.28:0/1796351833 conn(0x7fb1600b0270 msgr2=0x7fb1600af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 shutdown_connections 2024-09-13T17:08:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.322+0000 7fb166000640 1 -- 172.21.15.28:0/1796351833 wait complete. 2024-09-13T17:08:45.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.497+0000 7f346be00640 1 Processor -- start 2024-09-13T17:08:45.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.497+0000 7f346be00640 1 -- start start 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 0x7f346c0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f346c0cf7b0 0x7f346c0d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346c0cfc10 con 0x7f346c0d1230 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346c0d19b0 con 0x7f346c0cf7b0 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c0d1c90 con 0x7f346c15ca40 2024-09-13T17:08:45.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f346c0d1230 0x7f346c069e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42094/0 (socket says 172.21.15.28:42094) 2024-09-13T17:08:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346b400640 1 -- 172.21.15.28:0/268201978 learned_addr learned my addr 172.21.15.28:0/268201978 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f346aa00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 0x7f346c0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.498+0000 7f3463e00640 1 --2- 172.21.15.28:0/268201978 >> v2:172.21.15.189:3300/0 conn(0x7f346c0cf7b0 0x7f346c0d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f346a000640 1 -- 172.21.15.28:0/268201978 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097025975 0 0) 0x7f346c0cfc10 con 0x7f346c0d1230 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f346a000640 1 -- 172.21.15.28:0/268201978 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3450003520 con 0x7f346c0d1230 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f3463e00640 1 -- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 msgr2=0x7f346c0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f3463e00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 0x7f346c0cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f3463e00640 1 -- 172.21.15.28:0/268201978 >> v1:172.21.15.28:6789/0 conn(0x7f346c0d1230 legacy=0x7f346c069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f3463e00640 1 -- 172.21.15.28:0/268201978 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346c0d1f70 con 0x7f346c0cf7b0 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.499+0000 7f346aa00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 0x7f346c0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.500+0000 7f3463e00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0d0cf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3454009790 tx=0x7f345402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=499cb7267906f6bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.500+0000 7f346a000640 1 -- 172.21.15.28:0/268201978 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454032440 con 0x7f346c0cf7b0 2024-09-13T17:08:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346a000640 1 -- 172.21.15.28:0/268201978 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34540325a0 con 0x7f346c0cf7b0 2024-09-13T17:08:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346a000640 1 -- 172.21.15.28:0/268201978 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454032910 con 0x7f346c0cf7b0 2024-09-13T17:08:45.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 -- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 msgr2=0x7f346c0d0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0d0cf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3454009790 tx=0x7f345402a180 comp rx=0 tx=0).stop 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 -- 172.21.15.28:0/268201978 shutdown_connections 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0d0cf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 --2- 172.21.15.28:0/268201978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c15ca40 0x7f346c0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 -- 172.21.15.28:0/268201978 >> 172.21.15.28:0/268201978 conn(0x7f346c070ad0 msgr2=0x7f346c0b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 -- 172.21.15.28:0/268201978 shutdown_connections 2024-09-13T17:08:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.501+0000 7f346be00640 1 -- 172.21.15.28:0/268201978 wait complete. 2024-09-13T17:08:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.503+0000 7f346be00640 1 Processor -- start 2024-09-13T17:08:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.503+0000 7f346be00640 1 -- start start 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0c8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 0x7f346c1478b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c1565d0 con 0x7f346c0d1230 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346c147df0 con 0x7f346c0cf7b0 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f346be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346c156010 con 0x7f346c147420 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.504+0000 7f3463e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33598/0 (socket says 172.21.15.28:33598) 2024-09-13T17:08:45.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 -- 172.21.15.28:0/953444069 learned_addr learned my addr 172.21.15.28:0/953444069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f346aa00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0c8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f346b400640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 0x7f346c1478b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 -- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 msgr2=0x7f346c1478b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 0x7f346c1478b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 -- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 msgr2=0x7f346c0c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0c8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 -- 172.21.15.28:0/953444069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009400 con 0x7f346c0d1230 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f346b400640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 0x7f346c1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.505+0000 7f3463e00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3454000c00 tx=0x7f34540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.506+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454006db0 con 0x7f346c0d1230 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.506+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34540044c0 con 0x7f346c0d1230 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.506+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454003850 con 0x7f346c0d1230 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.506+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34540093a0 con 0x7f346c0d1230 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.506+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346c144ff0 con 0x7f346c0d1230 2024-09-13T17:08:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.507+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346c144e50 con 0x7f346c0d1230 2024-09-13T17:08:45.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.509+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3454039050 con 0x7f346c0d1230 2024-09-13T17:08:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.509+0000 7f3462a00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 0x7f344806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.509+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34540a9c70 con 0x7f346c0d1230 2024-09-13T17:08:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.510+0000 7f346aa00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 0x7f344806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.511+0000 7f346aa00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 0x7f344806d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f345c009c30 tx=0x7f345c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.514+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f346c144e50 con 0x7f346c0d1230 2024-09-13T17:08:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.692+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f346c144e50 con 0x7f346c0d1230 2024-09-13T17:08:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.692+0000 7f3462a00640 1 -- 172.21.15.28:0/953444069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f346c144e50 con 0x7f346c0d1230 2024-09-13T17:08:45.694 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.694+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 msgr2=0x7f344806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.694+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 0x7f344806d0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f345c009c30 tx=0x7f345c009190 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.694+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 msgr2=0x7f346c146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.694+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3454000c00 tx=0x7f34540046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 shutdown_connections 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344806ac00 0x7f344806d0c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346c147420 0x7f346c1478b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f346c0d1230 0x7f346c146ee0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 --2- 172.21.15.28:0/953444069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346c0cf7b0 0x7f346c0c8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 >> 172.21.15.28:0/953444069 conn(0x7f346c070ad0 msgr2=0x7f346c0ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 shutdown_connections 2024-09-13T17:08:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.695+0000 7f346be00640 1 -- 172.21.15.28:0/953444069 wait complete. 2024-09-13T17:08:45.712 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir' 2024-09-13T17:08:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.867+0000 7f98ebc00640 1 Processor -- start 2024-09-13T17:08:45.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.867+0000 7f98ebc00640 1 -- start start 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.868+0000 7f98ebc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f98e406f970 0x7f98e415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.868+0000 7f98ebc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98e406c2d0 0x7f98e406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.868+0000 7f98ebc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e406cc70 con 0x7f98e41592a0 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.868+0000 7f98ebc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e415b0b0 con 0x7f98e406c2d0 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.868+0000 7f98ebc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e4159d90 con 0x7f98e406f970 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98e406c2d0 0x7f98e406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98e406c2d0 0x7f98e406c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33614/0 (socket says 172.21.15.28:33614) 2024-09-13T17:08:45.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 -- 172.21.15.28:0/72518430 learned_addr learned my addr 172.21.15.28:0/72518430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9200640 1 --1- 172.21.15.28:0/72518430 >> v1:172.21.15.28:6789/0 conn(0x7f98e41592a0 0x7f98e4159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42110/0 (socket says 172.21.15.28:42110) 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98dbe00640 1 --2- 172.21.15.28:0/72518430 >> v2:172.21.15.189:3300/0 conn(0x7f98e406f970 0x7f98e415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 -- 172.21.15.28:0/72518430 >> v2:172.21.15.189:3300/0 conn(0x7f98e406f970 msgr2=0x7f98e415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 --2- 172.21.15.28:0/72518430 >> v2:172.21.15.189:3300/0 conn(0x7f98e406f970 0x7f98e415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 -- 172.21.15.28:0/72518430 >> v1:172.21.15.28:6789/0 conn(0x7f98e41592a0 legacy=0x7f98e4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.869+0000 7f98e9c00640 1 -- 172.21.15.28:0/72518430 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40b2040 con 0x7f98e406c2d0 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.870+0000 7f98e9c00640 1 --2- 172.21.15.28:0/72518430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e406c730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f98e406ac10 tx=0x7f98dc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8e655993076f8c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98e3e00640 1 -- 172.21.15.28:0/72518430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc031600 con 0x7f98e406c2d0 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98e3e00640 1 -- 172.21.15.28:0/72518430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc02d050 con 0x7f98e406c2d0 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98e3e00640 1 -- 172.21.15.28:0/72518430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc031980 con 0x7f98e406c2d0 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 -- 172.21.15.28:0/72518430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 msgr2=0x7f98e406c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 --2- 172.21.15.28:0/72518430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e406c730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f98e406ac10 tx=0x7f98dc02a730 comp rx=0 tx=0).stop 2024-09-13T17:08:45.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 -- 172.21.15.28:0/72518430 shutdown_connections 2024-09-13T17:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 --2- 172.21.15.28:0/72518430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e406c730 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 --2- 172.21.15.28:0/72518430 >> v2:172.21.15.189:3300/0 conn(0x7f98e406f970 0x7f98e415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 -- 172.21.15.28:0/72518430 >> 172.21.15.28:0/72518430 conn(0x7f98e4069e80 msgr2=0x7f98e406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 -- 172.21.15.28:0/72518430 shutdown_connections 2024-09-13T17:08:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.871+0000 7f98ebc00640 1 -- 172.21.15.28:0/72518430 wait complete. 2024-09-13T17:08:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.873+0000 7f98ebc00640 1 Processor -- start 2024-09-13T17:08:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.873+0000 7f98ebc00640 1 -- start start 2024-09-13T17:08:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 0x7f98e40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e41592a0 0x7f98e40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4149330 con 0x7f98e406c2d0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4146450 con 0x7f98e41592a0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98ebc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e4146730 con 0x7f98e406f970 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98e9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98e9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33628/0 (socket says 172.21.15.28:33628) 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98e9200640 1 -- 172.21.15.28:0/2333785270 learned_addr learned my addr 172.21.15.28:0/2333785270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98e9c00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e41592a0 0x7f98e40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.874+0000 7f98dbe00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 0x7f98e40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 -- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 msgr2=0x7f98e40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 0x7f98e40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 -- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e41592a0 msgr2=0x7f98e40acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e41592a0 0x7f98e40acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 -- 172.21.15.28:0/2333785270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc009b30 con 0x7f98e406c2d0 2024-09-13T17:08:45.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98dbe00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 0x7f98e40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e9200640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c3e0 tx=0x7f98d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40074e0 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d4007640 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40049e0 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e4146a10 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.875+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e40cb600 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.876+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e415a2b0 con 0x7f98e406c2d0 2024-09-13T17:08:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.877+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98d4004b40 con 0x7f98e406c2d0 2024-09-13T17:08:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.878+0000 7f98e1600640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 0x7f98c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.878+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98d4088cd0 con 0x7f98e406c2d0 2024-09-13T17:08:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.879+0000 7f98dbe00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 0x7f98c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.879+0000 7f98dbe00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 0x7f98c006d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98cc006fd0 tx=0x7f98cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:45.882+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98e415a2b0 con 0x7f98e406c2d0 2024-09-13T17:08:46.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.132+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]} v 0) v1 -- 0x7f98e415a2b0 con 0x7f98e406c2d0 2024-09-13T17:08:46.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.991+0000 7f98e1600640 1 -- 172.21.15.28:0/2333785270 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]}]=0 v56) v1 ==== 151+0+0 (secure 0 0 0) 0x7f98e415a2b0 con 0x7f98e406c2d0 2024-09-13T17:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 msgr2=0x7f98c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:46.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 0x7f98c006d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f98cc006fd0 tx=0x7f98cc008040 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 msgr2=0x7f98e4145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c3e0 tx=0x7f98d400c8a0 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 shutdown_connections 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98c006ac00 0x7f98c006d0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98e41592a0 0x7f98e40acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98e406f970 0x7f98e40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 --2- 172.21.15.28:0/2333785270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98e406c2d0 0x7f98e4145c00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 >> 172.21.15.28:0/2333785270 conn(0x7f98e4069e80 msgr2=0x7f98e415be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 shutdown_connections 2024-09-13T17:08:46.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:46.994+0000 7f98ebc00640 1 -- 172.21.15.28:0/2333785270 wait complete. 2024-09-13T17:08:47.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:08:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.175+0000 7fdd81400640 1 Processor -- start 2024-09-13T17:08:47.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.175+0000 7fdd81400640 1 -- start start 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd81400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd7c15a780 0x7fdd7c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd81400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd81400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7c06c8b0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd81400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7c06cb90 con 0x7fdd7c15a780 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd81400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c0cc980 con 0x7fdd7c15ab60 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.176+0000 7fdd7a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd7c15a780 0x7fdd7c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdd7c15a780 0x7fdd7c070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33644/0 (socket says 172.21.15.28:33644) 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7b200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdd7c06bb40 0x7fdd7c06bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42118/0 (socket says 172.21.15.28:42118) 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 -- 172.21.15.28:0/2049556393 learned_addr learned my addr 172.21.15.28:0/2049556393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 -- 172.21.15.28:0/2049556393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 msgr2=0x7fdd7c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:47.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 --2- 172.21.15.28:0/2049556393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 -- 172.21.15.28:0/2049556393 >> v1:172.21.15.28:6789/0 conn(0x7fdd7c06bb40 legacy=0x7fdd7c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.177+0000 7fdd7a800640 1 -- 172.21.15.28:0/2049556393 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7c0ccc60 con 0x7fdd7c15a780 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.178+0000 7fdd7a800640 1 --2- 172.21.15.28:0/2049556393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c15a780 0x7fdd7c070d80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e96c0371c052d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.178+0000 7fdd79400640 1 -- 172.21.15.28:0/2049556393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c032440 con 0x7fdd7c15a780 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.178+0000 7fdd79400640 1 -- 172.21.15.28:0/2049556393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c0325a0 con 0x7fdd7c15a780 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.178+0000 7fdd79400640 1 -- 172.21.15.28:0/2049556393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c032930 con 0x7fdd7c15a780 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 -- 172.21.15.28:0/2049556393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c15a780 msgr2=0x7fdd7c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 --2- 172.21.15.28:0/2049556393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c15a780 0x7fdd7c070d80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c02a250 comp rx=0 tx=0).stop 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 -- 172.21.15.28:0/2049556393 shutdown_connections 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 --2- 172.21.15.28:0/2049556393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 --2- 172.21.15.28:0/2049556393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c15a780 0x7fdd7c070d80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 -- 172.21.15.28:0/2049556393 >> 172.21.15.28:0/2049556393 conn(0x7fdd7c0b0270 msgr2=0x7fdd7c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 -- 172.21.15.28:0/2049556393 shutdown_connections 2024-09-13T17:08:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.179+0000 7fdd81400640 1 -- 172.21.15.28:0/2049556393 wait complete. 2024-09-13T17:08:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.180+0000 7fdd81400640 1 Processor -- start 2024-09-13T17:08:47.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 -- start start 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 0x7fdd7c152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c1f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c153640 con 0x7fdd7c06bb40 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c153920 con 0x7fdd7c15a780 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.181+0000 7fdd81400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7c153c00 con 0x7fdd7c15ab60 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33660/0 (socket says 172.21.15.28:33660) 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 -- 172.21.15.28:0/238758594 learned_addr learned my addr 172.21.15.28:0/238758594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7b200640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c1f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd79e00640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 0x7fdd7c152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 -- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 msgr2=0x7fdd7c1f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c1f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 -- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 msgr2=0x7fdd7c152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 0x7fdd7c152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 -- 172.21.15.28:0/238758594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd6c0094d0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd79e00640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 0x7fdd7c152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.182+0000 7fdd7a800640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.183+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c030b00 con 0x7fdd7c06bb40 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.183+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6c030c60 con 0x7fdd7c06bb40 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.183+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd7c153ee0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.183+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6c03d420 con 0x7fdd7c06bb40 2024-09-13T17:08:47.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.183+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd7c14ef30 con 0x7fdd7c06bb40 2024-09-13T17:08:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.184+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7c06bfd0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.185+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdd6c039040 con 0x7fdd7c06bb40 2024-09-13T17:08:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.186+0000 7fdd72a00640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 0x7fdd5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.186+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdd6c0aa0b0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.186+0000 7fdd79e00640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 0x7fdd5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.187+0000 7fdd79e00640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 0x7fdd5c06d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c14f6f0 tx=0x7fdd68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.190+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdd6c075280 con 0x7fdd7c06bb40 2024-09-13T17:08:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.369+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdd7c06b780 con 0x7fdd7c06bb40 2024-09-13T17:08:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.373+0000 7fdd72a00640 1 -- 172.21.15.28:0/238758594 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdd6c0788a0 con 0x7fdd7c06bb40 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.375+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 msgr2=0x7fdd5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.375+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 0x7fdd5c06d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c14f6f0 tx=0x7fdd68008040 comp rx=0 tx=0).stop 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 msgr2=0x7fdd7c14e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c009870 tx=0x7fdd6c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 shutdown_connections 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd5c06ac00 0x7fdd5c06d0c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd7c15ab60 0x7fdd7c1f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd7c15a780 0x7fdd7c152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 --2- 172.21.15.28:0/238758594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd7c06bb40 0x7fdd7c14e9f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 >> 172.21.15.28:0/238758594 conn(0x7fdd7c0b0270 msgr2=0x7fdd7c0ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 shutdown_connections 2024-09-13T17:08:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.376+0000 7fdd81400640 1 -- 172.21.15.28:0/238758594 wait complete. 2024-09-13T17:08:47.394 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:08:47.394 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:08:47.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:08:47.574 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:08:47.574 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:47.575 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:47.610 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:47.610 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:47.686 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:47.687 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:47.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.957+0000 7f854f000640 1 Processor -- start 2024-09-13T17:08:47.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.957+0000 7f854f000640 1 -- start start 2024-09-13T17:08:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.958+0000 7f854f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 0x7f854806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.958+0000 7f854f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.958+0000 7f854f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854815e100 con 0x7f854815a540 2024-09-13T17:08:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.958+0000 7f854f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85480adb00 con 0x7f85480cabc0 2024-09-13T17:08:47.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.958+0000 7f854f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8548071670 con 0x7f8548070cc0 2024-09-13T17:08:47.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8547400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 0x7f854806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8547e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f854815a540 0x7f854815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:43030/0 (socket says 172.21.15.28:43030) 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8547e00640 1 -- 172.21.15.28:0/2713686363 learned_addr learned my addr 172.21.15.28:0/2713686363 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 --2- 172.21.15.28:0/2713686363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56338/0 (socket says 172.21.15.28:56338) 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8546a00640 1 -- 172.21.15.28:0/2713686363 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377595603 0 0) 0x7f854815e100 con 0x7f854815a540 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8546a00640 1 -- 172.21.15.28:0/2713686363 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c003520 con 0x7f854815a540 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 -- 172.21.15.28:0/2713686363 >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 msgr2=0x7f854806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 --2- 172.21.15.28:0/2713686363 >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 0x7f854806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 -- 172.21.15.28:0/2713686363 >> v1:172.21.15.189:6789/0 conn(0x7f854815a540 legacy=0x7f854815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f854d000640 1 -- 172.21.15.28:0/2713686363 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854815e3e0 con 0x7f8548070cc0 2024-09-13T17:08:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.959+0000 7f8547400640 1 --2- 172.21.15.28:0/2713686363 >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 0x7f854806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.960+0000 7f854d000640 1 --2- 172.21.15.28:0/2713686363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8540009d90 tx=0x7f854002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c657ee4c54794b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f8546a00640 1 -- 172.21.15.28:0/2713686363 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031600 con 0x7f8548070cc0 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f8546a00640 1 -- 172.21.15.28:0/2713686363 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854002d050 con 0x7f8548070cc0 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f8546a00640 1 -- 172.21.15.28:0/2713686363 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8540031980 con 0x7f8548070cc0 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 -- 172.21.15.28:0/2713686363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 msgr2=0x7f8548071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 --2- 172.21.15.28:0/2713686363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8540009d90 tx=0x7f854002a5d0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 -- 172.21.15.28:0/2713686363 shutdown_connections 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 --2- 172.21.15.28:0/2713686363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8548070cc0 0x7f8548071130 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 --2- 172.21.15.28:0/2713686363 >> v2:172.21.15.189:3300/0 conn(0x7f85480cabc0 0x7f854806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 -- 172.21.15.28:0/2713686363 >> 172.21.15.28:0/2713686363 conn(0x7f8548069e80 msgr2=0x7f854806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:47.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 -- 172.21.15.28:0/2713686363 shutdown_connections 2024-09-13T17:08:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.961+0000 7f854f000640 1 -- 172.21.15.28:0/2713686363 wait complete. 2024-09-13T17:08:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.963+0000 7f854f000640 1 Processor -- start 2024-09-13T17:08:47.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.963+0000 7f854f000640 1 -- start start 2024-09-13T17:08:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 0x7f85481437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854815a540 0x7f85481403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8548143cf0 con 0x7f8548070cc0 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854814fa50 con 0x7f854815a540 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f854f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854814fd30 con 0x7f85480cabc0 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f8547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f8547e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33672/0 (socket says 172.21.15.28:33672) 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.964+0000 7f8547e00640 1 -- 172.21.15.28:0/191590680 learned_addr learned my addr 172.21.15.28:0/191590680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f854d000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854815a540 0x7f85481403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547400640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 0x7f85481437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 -- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 msgr2=0x7f85481437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 0x7f85481437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 -- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854815a540 msgr2=0x7f85481403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854815a540 0x7f85481403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 -- 172.21.15.28:0/191590680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8534009400 con 0x7f8548070cc0 2024-09-13T17:08:47.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547400640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 0x7f85481437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f8547e00640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f853800c390 tx=0x7f853800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.965+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8538005ce0 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.966+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8538005e40 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.966+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8540009a70 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.966+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85380072f0 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.966+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8548151a80 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.967+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8548151600 con 0x7f8548070cc0 2024-09-13T17:08:47.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.967+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8538007450 con 0x7f8548070cc0 2024-09-13T17:08:47.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.968+0000 7f853fe00640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 0x7f852806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:47.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.968+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85380889e0 con 0x7f8548070cc0 2024-09-13T17:08:47.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.969+0000 7f8547400640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 0x7f852806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:47.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.969+0000 7f8547400640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 0x7f852806d0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85480b3820 tx=0x7f8534008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:47.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:47.972+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8548151600 con 0x7f8548070cc0 2024-09-13T17:08:48.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.153+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8548151600 con 0x7f8548070cc0 2024-09-13T17:08:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.155+0000 7f853fe00640 1 -- 172.21.15.28:0/191590680 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7f8548151600 con 0x7f8548070cc0 2024-09-13T17:08:48.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:48.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 msgr2=0x7f852806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 0x7f852806d0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f85480b3820 tx=0x7f8534008040 comp rx=0 tx=0).stop 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 msgr2=0x7f8548140d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f853800c390 tx=0x7f853800c850 comp rx=0 tx=0).stop 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 shutdown_connections 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f852806ac00 0x7f852806d0c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f854815a540 0x7f85481403d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85480cabc0 0x7f85481437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 --2- 172.21.15.28:0/191590680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8548070cc0 0x7f8548140d10 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 >> 172.21.15.28:0/191590680 conn(0x7f8548069e80 msgr2=0x7f85480ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 shutdown_connections 2024-09-13T17:08:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.157+0000 7f854f000640 1 -- 172.21.15.28:0/191590680 wait complete. 2024-09-13T17:08:48.175 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:48.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.331+0000 7fb4c5a00640 1 Processor -- start 2024-09-13T17:08:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.331+0000 7fb4c5a00640 1 -- start start 2024-09-13T17:08:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.332+0000 7fb4c5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c015c9a0 0x7fb4c00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.332+0000 7fb4c5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4c00b2400 0x7fb4c006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.332+0000 7fb4c5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c0070470 con 0x7fb4c00ad940 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.332+0000 7fb4c5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c00ace30 con 0x7fb4c00b2400 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.332+0000 7fb4c5a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00ad110 con 0x7fb4c015c9a0 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4c00b2400 0x7fb4c006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4c00b2400 0x7fb4c006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33676/0 (socket says 172.21.15.28:33676) 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4be000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4c00ad940 0x7fb4c00b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42134/0 (socket says 172.21.15.28:42134) 2024-09-13T17:08:48.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 -- 172.21.15.28:0/1540743884 learned_addr learned my addr 172.21.15.28:0/1540743884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bea00640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c015c9a0 0x7fb4c00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 -- 172.21.15.28:0/1540743884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c015c9a0 msgr2=0x7fb4c00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c015c9a0 0x7fb4c00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 -- 172.21.15.28:0/1540743884 >> v1:172.21.15.28:6789/0 conn(0x7fb4c00ad940 legacy=0x7fb4c00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.333+0000 7fb4bf400640 1 -- 172.21.15.28:0/1540743884 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c006b170 con 0x7fb4c00b2400 2024-09-13T17:08:48.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.334+0000 7fb4bf400640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00b2400 0x7fb4c006bc50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009e50 tx=0x7fb4b802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cbd3089be9a512 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.334+0000 7fb4bd600640 1 -- 172.21.15.28:0/1540743884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8031600 con 0x7fb4c00b2400 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.334+0000 7fb4bd600640 1 -- 172.21.15.28:0/1540743884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b802d050 con 0x7fb4c00b2400 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.334+0000 7fb4bd600640 1 -- 172.21.15.28:0/1540743884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b80319a0 con 0x7fb4c00b2400 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 -- 172.21.15.28:0/1540743884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00b2400 msgr2=0x7fb4c006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00b2400 0x7fb4c006bc50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8009e50 tx=0x7fb4b802a7c0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 -- 172.21.15.28:0/1540743884 shutdown_connections 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00b2400 0x7fb4c006bc50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/1540743884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c015c9a0 0x7fb4c00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 -- 172.21.15.28:0/1540743884 >> 172.21.15.28:0/1540743884 conn(0x7fb4c00b0270 msgr2=0x7fb4c00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 -- 172.21.15.28:0/1540743884 shutdown_connections 2024-09-13T17:08:48.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.335+0000 7fb4c5a00640 1 -- 172.21.15.28:0/1540743884 wait complete. 2024-09-13T17:08:48.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.337+0000 7fb4c5a00640 1 Processor -- start 2024-09-13T17:08:48.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.337+0000 7fb4c5a00640 1 -- start start 2024-09-13T17:08:48.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 0x7fb4c01f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 0x7fb4c015fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c00cd090 con 0x7fb4c00ad940 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01f60a0 con 0x7fb4c015c9a0 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4c5a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c01f6380 con 0x7fb4c00b2400 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33680/0 (socket says 172.21.15.28:33680) 2024-09-13T17:08:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bea00640 1 -- 172.21.15.28:0/3677354504 learned_addr learned my addr 172.21.15.28:0/3677354504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bf400640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 0x7fb4c015fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4be000640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 0x7fb4c01f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bea00640 1 -- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 msgr2=0x7fb4c01f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.338+0000 7fb4bea00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 0x7fb4c01f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4bea00640 1 -- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 msgr2=0x7fb4c015fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4bea00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 0x7fb4c015fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4bea00640 1 -- 172.21.15.28:0/3677354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b8009b30 con 0x7fb4c00ad940 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4bf400640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 0x7fb4c015fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4be000640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 0x7fb4c01f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4bea00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000d5f0 tx=0x7fb4b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b00125f0 con 0x7fb4c00ad940 2024-09-13T17:08:48.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b0012750 con 0x7fb4c00ad940 2024-09-13T17:08:48.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c0070930 con 0x7fb4c00ad940 2024-09-13T17:08:48.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4b0012b00 con 0x7fb4c00ad940 2024-09-13T17:08:48.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.339+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c015f220 con 0x7fb4c00ad940 2024-09-13T17:08:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.340+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c01f2110 con 0x7fb4c00ad940 2024-09-13T17:08:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.341+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4b0012c60 con 0x7fb4c00ad940 2024-09-13T17:08:48.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.342+0000 7fb4b6a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 0x7fb4a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.342+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4b00897b0 con 0x7fb4c00ad940 2024-09-13T17:08:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.342+0000 7fb4be000640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 0x7fb4a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.343+0000 7fb4be000640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 0x7fb4a006d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4c01f57b0 tx=0x7fb4ac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.346+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb4b0054a20 con 0x7fb4c00ad940 2024-09-13T17:08:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.522+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb4c00af6a0 con 0x7fb4c00ad940 2024-09-13T17:08:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.522+0000 7fb4b6a00640 1 -- 172.21.15.28:0/3677354504 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb4b0058040 con 0x7fb4c00ad940 2024-09-13T17:08:48.524 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 msgr2=0x7fb4a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 0x7fb4a006d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4c01f57b0 tx=0x7fb4ac009040 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 msgr2=0x7fb4c01f1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb4b000d5f0 tx=0x7fb4b000dab0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 shutdown_connections 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4a006ac00 0x7fb4a006d0c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4c015c9a0 0x7fb4c015fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4c00b2400 0x7fb4c01f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 --2- 172.21.15.28:0/3677354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4c00ad940 0x7fb4c01f1bd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 >> 172.21.15.28:0/3677354504 conn(0x7fb4c00b0270 msgr2=0x7fb4c00b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 shutdown_connections 2024-09-13T17:08:48.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.524+0000 7fb4c5a00640 1 -- 172.21.15.28:0/3677354504 wait complete. 2024-09-13T17:08:48.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.697+0000 7fbe2b800640 1 Processor -- start 2024-09-13T17:08:48.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.697+0000 7fbe2b800640 1 -- start start 2024-09-13T17:08:48.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.698+0000 7fbe2b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24070430 0x7fbe2415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.698+0000 7fbe2b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbe2406f3f0 0x7fbe2415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.698+0000 7fbe2b800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415ab60 con 0x7fbe24159770 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.698+0000 7fbe2b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe2406f920 con 0x7fbe2406f3f0 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.698+0000 7fbe2b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2415b290 con 0x7fbe24070430 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe28e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe24159770 0x7fbe2406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42136/0 (socket says 172.21.15.28:42136) 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe28e00640 1 -- 172.21.15.28:0/126174554 learned_addr learned my addr 172.21.15.28:0/126174554 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe29800640 1 --2- 172.21.15.28:0/126174554 >> v2:172.21.15.28:3300/0 conn(0x7fbe2406f3f0 0x7fbe2415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe23e00640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24070430 0x7fbe2415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe29800640 1 -- 172.21.15.28:0/126174554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24070430 msgr2=0x7fbe2415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe29800640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24070430 0x7fbe2415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe29800640 1 -- 172.21.15.28:0/126174554 >> v1:172.21.15.28:6789/0 conn(0x7fbe24159770 legacy=0x7fbe2406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe29800640 1 -- 172.21.15.28:0/126174554 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2415b570 con 0x7fbe2406f3f0 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.699+0000 7fbe23400640 1 -- 172.21.15.28:0/126174554 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536846745 0 0) 0x7fbe18001110 con 0x7fbe24159770 2024-09-13T17:08:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.700+0000 7fbe29800640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe2415c2b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009d90 tx=0x7fbe1c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6076dc3ab50f439a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.700+0000 7fbe23400640 1 -- 172.21.15.28:0/126174554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c031600 con 0x7fbe2406f3f0 2024-09-13T17:08:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.700+0000 7fbe23400640 1 -- 172.21.15.28:0/126174554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1c02d050 con 0x7fbe2406f3f0 2024-09-13T17:08:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe23400640 1 -- 172.21.15.28:0/126174554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe1c031980 con 0x7fbe2406f3f0 2024-09-13T17:08:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 -- 172.21.15.28:0/126174554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 msgr2=0x7fbe2415c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe2415c2b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbe1c009d90 tx=0x7fbe1c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 -- 172.21.15.28:0/126174554 shutdown_connections 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe2415c2b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 --2- 172.21.15.28:0/126174554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24070430 0x7fbe2415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 -- 172.21.15.28:0/126174554 >> 172.21.15.28:0/126174554 conn(0x7fbe240b0270 msgr2=0x7fbe240b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 -- 172.21.15.28:0/126174554 shutdown_connections 2024-09-13T17:08:48.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.701+0000 7fbe2b800640 1 -- 172.21.15.28:0/126174554 wait complete. 2024-09-13T17:08:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.703+0000 7fbe2b800640 1 Processor -- start 2024-09-13T17:08:48.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.703+0000 7fbe2b800640 1 -- start start 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe24070430 0x7fbe24150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 0x7fbe240ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240cc870 con 0x7fbe2406f3f0 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe24150580 con 0x7fbe24070430 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe2b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe240cbc60 con 0x7fbe24159770 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33706/0 (socket says 172.21.15.28:33706) 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe28e00640 1 -- 172.21.15.28:0/2087206795 learned_addr learned my addr 172.21.15.28:0/2087206795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe23e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe24070430 0x7fbe24150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.704+0000 7fbe29800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 0x7fbe240ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 -- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 msgr2=0x7fbe240ccd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 0x7fbe240ccd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 -- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe24070430 msgr2=0x7fbe24150040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe24070430 0x7fbe24150040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 -- 172.21.15.28:0/2087206795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c009a70 con 0x7fbe2406f3f0 2024-09-13T17:08:48.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe29800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 0x7fbe240ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe28e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c430 tx=0x7fbe1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe180074e0 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe18007660 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe18004a20 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe240cbf40 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.705+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe240cc500 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.706+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2406fd20 con 0x7fbe2406f3f0 2024-09-13T17:08:48.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.707+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbe18004b80 con 0x7fbe2406f3f0 2024-09-13T17:08:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.708+0000 7fbe20c00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 0x7fbe0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.708+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbe18088fe0 con 0x7fbe2406f3f0 2024-09-13T17:08:48.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.709+0000 7fbe23e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 0x7fbe0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.709+0000 7fbe23e00640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 0x7fbe0406d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe241f39f0 tx=0x7fbe10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:48.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.712+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbe180541b0 con 0x7fbe2406f3f0 2024-09-13T17:08:48.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.892+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe2415ab60 con 0x7fbe2406f3f0 2024-09-13T17:08:48.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.893+0000 7fbe20c00640 1 -- 172.21.15.28:0/2087206795 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe2415ab60 con 0x7fbe2406f3f0 2024-09-13T17:08:48.895 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 msgr2=0x7fbe0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 0x7fbe0406d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe241f39f0 tx=0x7fbe10008040 comp rx=0 tx=0).stop 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 msgr2=0x7fbe240cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbe1800c430 tx=0x7fbe1800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 shutdown_connections 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe0406ac00 0x7fbe0406d0c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe24159770 0x7fbe240ccd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe24070430 0x7fbe24150040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 --2- 172.21.15.28:0/2087206795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe2406f3f0 0x7fbe240cb5f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 >> 172.21.15.28:0/2087206795 conn(0x7fbe240b0270 msgr2=0x7fbe24158710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 shutdown_connections 2024-09-13T17:08:48.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:48.895+0000 7fbe2b800640 1 -- 172.21.15.28:0/2087206795 wait complete. 2024-09-13T17:08:48.914 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.071+0000 7f575ca00640 1 Processor -- start 2024-09-13T17:08:49.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.071+0000 7f575ca00640 1 -- start start 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f575ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f575815ab60 0x7f575806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f575ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575806c8b0 con 0x7f5758071490 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f575ca00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575806cb90 con 0x7f575815a780 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f575ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57580cc980 con 0x7f575815ab60 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5755400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f575815ab60 0x7f575806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5756800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5758071490 0x7f575806c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42144/0 (socket says 172.21.15.28:42144) 2024-09-13T17:08:49.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5756800640 1 -- 172.21.15.28:0/2234720827 learned_addr learned my addr 172.21.15.28:0/2234720827 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5755400640 1 -- 172.21.15.28:0/2234720827 >> v1:172.21.15.189:6789/0 conn(0x7f575815a780 legacy=0x7f5758070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5755400640 1 -- 172.21.15.28:0/2234720827 >> v1:172.21.15.28:6789/0 conn(0x7f5758071490 legacy=0x7f575806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.073+0000 7f5755400640 1 -- 172.21.15.28:0/2234720827 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580ccc60 con 0x7f575815ab60 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.074+0000 7f5755400640 1 --2- 172.21.15.28:0/2234720827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575815ab60 0x7f575806bb40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f57440097a0 tx=0x7f574402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e5fba0a23fe3cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f5754a00640 1 -- 172.21.15.28:0/2234720827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744032440 con 0x7f575815ab60 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f5754a00640 1 -- 172.21.15.28:0/2234720827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57440325a0 con 0x7f575815ab60 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f5754a00640 1 -- 172.21.15.28:0/2234720827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744032930 con 0x7f575815ab60 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 -- 172.21.15.28:0/2234720827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575815ab60 msgr2=0x7f575806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 --2- 172.21.15.28:0/2234720827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575815ab60 0x7f575806bb40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f57440097a0 tx=0x7f574402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 -- 172.21.15.28:0/2234720827 shutdown_connections 2024-09-13T17:08:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 --2- 172.21.15.28:0/2234720827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575815ab60 0x7f575806bb40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 -- 172.21.15.28:0/2234720827 >> 172.21.15.28:0/2234720827 conn(0x7f57580b0270 msgr2=0x7f57580b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 -- 172.21.15.28:0/2234720827 shutdown_connections 2024-09-13T17:08:49.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.075+0000 7f575ca00640 1 -- 172.21.15.28:0/2234720827 wait complete. 2024-09-13T17:08:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.077+0000 7f575ca00640 1 Processor -- start 2024-09-13T17:08:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.077+0000 7f575ca00640 1 -- start start 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 0x7f57580b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575815ab60 0x7f57580acd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57580b1670 con 0x7f5758071490 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57580b1c30 con 0x7f575815ab60 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f575ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57580b2600 con 0x7f575815a780 2024-09-13T17:08:49.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f5755e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f5755e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33728/0 (socket says 172.21.15.28:33728) 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f5755e00640 1 -- 172.21.15.28:0/4273654532 learned_addr learned my addr 172.21.15.28:0/4273654532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f5756800640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575815ab60 0x7f57580acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.078+0000 7f5755400640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 0x7f57580b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 -- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 msgr2=0x7f57580b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 0x7f57580b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 -- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575815ab60 msgr2=0x7f57580acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575815ab60 0x7f57580acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 -- 172.21.15.28:0/4273654532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5744009400 con 0x7f5758071490 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755400640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 0x7f57580b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f5755e00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f574800c430 tx=0x7f574800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011790 con 0x7f5758071490 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57480118f0 con 0x7f5758071490 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5748011ce0 con 0x7f5758071490 2024-09-13T17:08:49.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57580b2e50 con 0x7f5758071490 2024-09-13T17:08:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.079+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57580b18c0 con 0x7f5758071490 2024-09-13T17:08:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.081+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57580b3370 con 0x7f5758071490 2024-09-13T17:08:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.081+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f574801c460 con 0x7f5758071490 2024-09-13T17:08:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.082+0000 7f574e000640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 0x7f573806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.082+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5748089260 con 0x7f5758071490 2024-09-13T17:08:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.083+0000 7f5755400640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 0x7f573806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.083+0000 7f5755400640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 0x7f573806d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5744034000 tx=0x7f574402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.086+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f57480543a0 con 0x7f5758071490 2024-09-13T17:08:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.264+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5758070d80 con 0x7f5758071490 2024-09-13T17:08:49.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.264+0000 7f574e000640 1 -- 172.21.15.28:0/4273654532 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57480579c0 con 0x7f5758071490 2024-09-13T17:08:49.266 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:49.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 msgr2=0x7f573806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 0x7f573806d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5744034000 tx=0x7f574402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 msgr2=0x7f57580b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f574800c430 tx=0x7f574800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 shutdown_connections 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac30 0x7f573806d0f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575815ab60 0x7f57580acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575815a780 0x7f57580b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 --2- 172.21.15.28:0/4273654532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5758071490 0x7f57580b2850 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 >> 172.21.15.28:0/4273654532 conn(0x7f57580b0270 msgr2=0x7f5758159700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 shutdown_connections 2024-09-13T17:08:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.266+0000 7f575ca00640 1 -- 172.21.15.28:0/4273654532 wait complete. 2024-09-13T17:08:49.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:08:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.440+0000 7fb703a00640 1 Processor -- start 2024-09-13T17:08:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.440+0000 7fb703a00640 1 -- start start 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb703a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc070430 0x7fb6fc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb703a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc15ab60 con 0x7fb6fc159770 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb703a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6fc06f920 con 0x7fb6fc06f3f0 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb703a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6fc15b290 con 0x7fb6fc070430 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb6f3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc070430 0x7fb6fc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb701000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb6fc159770 0x7fb6fc06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42146/0 (socket says 172.21.15.28:42146) 2024-09-13T17:08:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb6f3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6fc070430 0x7fb6fc15bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33730/0 (socket says 172.21.15.28:33730) 2024-09-13T17:08:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.441+0000 7fb701000640 1 -- 172.21.15.28:0/1579235929 learned_addr learned my addr 172.21.15.28:0/1579235929 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.442+0000 7fb6f3e00640 1 -- 172.21.15.28:0/1579235929 >> v1:172.21.15.189:6789/0 conn(0x7fb6fc06f3f0 legacy=0x7fb6fc15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.442+0000 7fb6f3e00640 1 -- 172.21.15.28:0/1579235929 >> v1:172.21.15.28:6789/0 conn(0x7fb6fc159770 legacy=0x7fb6fc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.442+0000 7fb6f3e00640 1 -- 172.21.15.28:0/1579235929 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc15b570 con 0x7fb6fc070430 2024-09-13T17:08:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.443+0000 7fb6f3e00640 1 --2- 172.21.15.28:0/1579235929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc070430 0x7fb6fc15bd70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40097a0 tx=0x7fb6e402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b2f0ac78e1eea5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.443+0000 7fb6fbe00640 1 -- 172.21.15.28:0/1579235929 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e402ae50 con 0x7fb6fc070430 2024-09-13T17:08:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.443+0000 7fb6fbe00640 1 -- 172.21.15.28:0/1579235929 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e4032440 con 0x7fb6fc070430 2024-09-13T17:08:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.443+0000 7fb6fbe00640 1 -- 172.21.15.28:0/1579235929 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e40327d0 con 0x7fb6fc070430 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 -- 172.21.15.28:0/1579235929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc070430 msgr2=0x7fb6fc15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 --2- 172.21.15.28:0/1579235929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc070430 0x7fb6fc15bd70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb6e40097a0 tx=0x7fb6e402a140 comp rx=0 tx=0).stop 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 -- 172.21.15.28:0/1579235929 shutdown_connections 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 --2- 172.21.15.28:0/1579235929 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc070430 0x7fb6fc15bd70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 -- 172.21.15.28:0/1579235929 >> 172.21.15.28:0/1579235929 conn(0x7fb6fc0b0270 msgr2=0x7fb6fc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 -- 172.21.15.28:0/1579235929 shutdown_connections 2024-09-13T17:08:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.444+0000 7fb703a00640 1 -- 172.21.15.28:0/1579235929 wait complete. 2024-09-13T17:08:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.445+0000 7fb703a00640 1 Processor -- start 2024-09-13T17:08:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.445+0000 7fb703a00640 1 -- start start 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc06f3f0 0x7fb6fc1e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 0x7fb6fc156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1e7f10 con 0x7fb6fc159770 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1e83a0 con 0x7fb6fc06f3f0 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.446+0000 7fb703a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6fc1e7040 con 0x7fb6fc070430 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33732/0 (socket says 172.21.15.28:33732) 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 -- 172.21.15.28:0/3575088619 learned_addr learned my addr 172.21.15.28:0/3575088619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb6f3e00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 0x7fb6fc156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701000640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc06f3f0 0x7fb6fc1e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 -- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 msgr2=0x7fb6fc156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 0x7fb6fc156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 -- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc06f3f0 msgr2=0x7fb6fc1e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc06f3f0 0x7fb6fc1e6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 -- 172.21.15.28:0/3575088619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e4009400 con 0x7fb6fc159770 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb6f3e00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 0x7fb6fc156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.447+0000 7fb701a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400c780 tx=0x7fb6f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.448+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4004b60 con 0x7fb6fc159770 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.448+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f400ce90 con 0x7fb6fc159770 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.448+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f400a850 con 0x7fb6fc159770 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.448+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6fc1e7300 con 0x7fb6fc159770 2024-09-13T17:08:49.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.448+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6fc1e78c0 con 0x7fb6fc159770 2024-09-13T17:08:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.449+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6fc15a9d0 con 0x7fb6fc159770 2024-09-13T17:08:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.450+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb6f400a9b0 con 0x7fb6fc159770 2024-09-13T17:08:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.451+0000 7fb6f9600640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 0x7fb6d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.451+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f4089600 con 0x7fb6fc159770 2024-09-13T17:08:49.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.451+0000 7fb701000640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 0x7fb6d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:49.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.452+0000 7fb701000640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 0x7fb6d806d0f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec005880 tx=0x7fb6ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:49.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.455+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb6f4054740 con 0x7fb6fc159770 2024-09-13T17:08:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.643+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb6fc06fd20 con 0x7fb6fc159770 2024-09-13T17:08:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.643+0000 7fb6f9600640 1 -- 172.21.15.28:0/3575088619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb6f4057d60 con 0x7fb6fc159770 2024-09-13T17:08:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.645+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 msgr2=0x7fb6d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.645+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 0x7fb6d806d0f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec005880 tx=0x7fb6ec009040 comp rx=0 tx=0).stop 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 msgr2=0x7fb6fc157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb6f400c780 tx=0x7fb6f400cc40 comp rx=0 tx=0).stop 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 shutdown_connections 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb6d806ac30 0x7fb6d806d0f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6fc159770 0x7fb6fc157110 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6fc070430 0x7fb6fc156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 --2- 172.21.15.28:0/3575088619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6fc06f3f0 0x7fb6fc1e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 >> 172.21.15.28:0/3575088619 conn(0x7fb6fc0b0270 msgr2=0x7fb6fc0ac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 shutdown_connections 2024-09-13T17:08:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:49.646+0000 7fb703a00640 1 -- 172.21.15.28:0/3575088619 wait complete. 2024-09-13T17:08:49.663 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:08:49.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:49.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-09-13T17:08:49.726 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:08:49.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:49.726 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:08:49.772 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:49.772 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:49.813 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:49.813 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:49.892 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:49.892 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:49.969 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.124+0000 7fa690800640 1 Processor -- start 2024-09-13T17:08:50.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.124+0000 7fa690800640 1 -- start start 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa690800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa68815a540 0x7fa68806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa690800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa688070cc0 0x7fa688071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa690800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68815e100 con 0x7fa6880cabc0 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa690800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6880adb00 con 0x7fa68815a540 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa690800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa688071670 con 0x7fa688070cc0 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa68de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa68815a540 0x7fa68806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa68de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa68815a540 0x7fa68806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33734/0 (socket says 172.21.15.28:33734) 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.125+0000 7fa68de00640 1 -- 172.21.15.28:0/1991341024 learned_addr learned my addr 172.21.15.28:0/1991341024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.126+0000 7fa68e800640 1 --2- 172.21.15.28:0/1991341024 >> v2:172.21.15.189:3300/0 conn(0x7fa688070cc0 0x7fa688071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.126+0000 7fa68de00640 1 -- 172.21.15.28:0/1991341024 >> v2:172.21.15.189:3300/0 conn(0x7fa688070cc0 msgr2=0x7fa688071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.126+0000 7fa68de00640 1 --2- 172.21.15.28:0/1991341024 >> v2:172.21.15.189:3300/0 conn(0x7fa688070cc0 0x7fa688071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.126+0000 7fa68de00640 1 -- 172.21.15.28:0/1991341024 >> v1:172.21.15.28:6789/0 conn(0x7fa6880cabc0 legacy=0x7fa68815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.126+0000 7fa68de00640 1 -- 172.21.15.28:0/1991341024 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa68815e3e0 con 0x7fa68815a540 2024-09-13T17:08:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.127+0000 7fa68de00640 1 --2- 172.21.15.28:0/1991341024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa68815a540 0x7fa68806c430 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa688070900 tx=0x7fa67c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4b52c6ef2326c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.127+0000 7fa68ca00640 1 -- 172.21.15.28:0/1991341024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c032440 con 0x7fa68815a540 2024-09-13T17:08:50.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.127+0000 7fa68ca00640 1 -- 172.21.15.28:0/1991341024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c0325a0 con 0x7fa68815a540 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.127+0000 7fa68ca00640 1 -- 172.21.15.28:0/1991341024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c032930 con 0x7fa68815a540 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 -- 172.21.15.28:0/1991341024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa68815a540 msgr2=0x7fa68806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 --2- 172.21.15.28:0/1991341024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa68815a540 0x7fa68806c430 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa688070900 tx=0x7fa67c02a210 comp rx=0 tx=0).stop 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 -- 172.21.15.28:0/1991341024 shutdown_connections 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 --2- 172.21.15.28:0/1991341024 >> v2:172.21.15.189:3300/0 conn(0x7fa688070cc0 0x7fa688071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 --2- 172.21.15.28:0/1991341024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa68815a540 0x7fa68806c430 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 -- 172.21.15.28:0/1991341024 >> 172.21.15.28:0/1991341024 conn(0x7fa688069e80 msgr2=0x7fa68806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 -- 172.21.15.28:0/1991341024 shutdown_connections 2024-09-13T17:08:50.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.128+0000 7fa690800640 1 -- 172.21.15.28:0/1991341024 wait complete. 2024-09-13T17:08:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.129+0000 7fa690800640 1 Processor -- start 2024-09-13T17:08:50.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 -- start start 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6880cabc0 0x7fa68814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa68815a540 0x7fa68813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68814cc10 con 0x7fa688070cc0 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68814e2f0 con 0x7fa6880cabc0 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.130+0000 7fa690800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68814e5d0 con 0x7fa68815a540 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33746/0 (socket says 172.21.15.28:33746) 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 -- 172.21.15.28:0/2128196512 learned_addr learned my addr 172.21.15.28:0/2128196512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68d400640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6880cabc0 0x7fa68814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68e800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa68815a540 0x7fa68813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 -- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa68815a540 msgr2=0x7fa68813baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa68815a540 0x7fa68813baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 -- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6880cabc0 msgr2=0x7fa68814f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6880cabc0 0x7fa68814f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 -- 172.21.15.28:0/2128196512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67c0094d0 con 0x7fa688070cc0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.131+0000 7fa68de00640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa67c000c00 tx=0x7fa67c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.132+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c032440 con 0x7fa688070cc0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.132+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c030980 con 0x7fa688070cc0 2024-09-13T17:08:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.132+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa68814e8b0 con 0x7fa688070cc0 2024-09-13T17:08:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.132+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c030d30 con 0x7fa688070cc0 2024-09-13T17:08:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.132+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa68814dfe0 con 0x7fa688070cc0 2024-09-13T17:08:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.133+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa688071130 con 0x7fa688070cc0 2024-09-13T17:08:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.134+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa67c039040 con 0x7fa688070cc0 2024-09-13T17:08:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.135+0000 7fa682000640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 0x7fa66c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.135+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa67c0a9c00 con 0x7fa688070cc0 2024-09-13T17:08:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.135+0000 7fa68d400640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 0x7fa66c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.136+0000 7fa68d400640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 0x7fa66c06d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa68814dd00 tx=0x7fa678009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.139+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa67c074df0 con 0x7fa688070cc0 2024-09-13T17:08:50.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.321+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa68814e710 con 0x7fa688070cc0 2024-09-13T17:08:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.322+0000 7fa682000640 1 -- 172.21.15.28:0/2128196512 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16505 (secure 0 0 0) 0x7fa68814e710 con 0x7fa688070cc0 2024-09-13T17:08:50.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:50.324 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":20,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:12.558212+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 msgr2=0x7fa66c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 0x7fa66c06d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa68814dd00 tx=0x7fa678009040 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 msgr2=0x7fa68814c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa67c000c00 tx=0x7fa67c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 shutdown_connections 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa66c06ac00 0x7fa66c06d0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa68815a540 0x7fa68813baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6880cabc0 0x7fa68814f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 --2- 172.21.15.28:0/2128196512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa688070cc0 0x7fa68814c510 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 >> 172.21.15.28:0/2128196512 conn(0x7fa688069e80 msgr2=0x7fa6880cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 shutdown_connections 2024-09-13T17:08:50.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.324+0000 7fa690800640 1 -- 172.21.15.28:0/2128196512 wait complete. 2024-09-13T17:08:50.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:50.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.497+0000 7f6480800640 1 Processor -- start 2024-09-13T17:08:50.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.497+0000 7f6480800640 1 -- start start 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f6480800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f6480800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f647806c250 0x7f64780712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f6480800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647806c8b0 con 0x7f647815ab60 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f6480800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647806c6a0 con 0x7f647806c250 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f6480800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647806cb90 con 0x7f647815a780 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f647e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f647806c250 0x7f64780712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.498+0000 7f647e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f647806c250 0x7f64780712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33750/0 (socket says 172.21.15.28:33750) 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647e800640 1 -- 172.21.15.28:0/485394890 learned_addr learned my addr 172.21.15.28:0/485394890 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647de00640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647e800640 1 -- 172.21.15.28:0/485394890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 msgr2=0x7f6478070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647e800640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647e800640 1 -- 172.21.15.28:0/485394890 >> v1:172.21.15.28:6789/0 conn(0x7f647815ab60 legacy=0x7f647806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647e800640 1 -- 172.21.15.28:0/485394890 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64780cc980 con 0x7f647806c250 2024-09-13T17:08:50.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.499+0000 7f647de00640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.500+0000 7f647e800640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f64780712c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6474009e50 tx=0x7f647402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297058cc77eafe6b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.500+0000 7f647ca00640 1 -- 172.21.15.28:0/485394890 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474031600 con 0x7f647806c250 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.500+0000 7f647ca00640 1 -- 172.21.15.28:0/485394890 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647402d050 con 0x7f647806c250 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.500+0000 7f647ca00640 1 -- 172.21.15.28:0/485394890 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474031980 con 0x7f647806c250 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 -- 172.21.15.28:0/485394890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 msgr2=0x7f64780712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f64780712c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f6474009e50 tx=0x7f647402a710 comp rx=0 tx=0).stop 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 -- 172.21.15.28:0/485394890 shutdown_connections 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f64780712c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 --2- 172.21.15.28:0/485394890 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 -- 172.21.15.28:0/485394890 >> 172.21.15.28:0/485394890 conn(0x7f64780b0270 msgr2=0x7f64780b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:50.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 -- 172.21.15.28:0/485394890 shutdown_connections 2024-09-13T17:08:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.501+0000 7f6480800640 1 -- 172.21.15.28:0/485394890 wait complete. 2024-09-13T17:08:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.503+0000 7f6480800640 1 Processor -- start 2024-09-13T17:08:50.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.503+0000 7f6480800640 1 -- start start 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 0x7f64781f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64780a94a0 con 0x7f647806c250 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64780a8890 con 0x7f647815ab60 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f6480800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780a8b70 con 0x7f647815a780 2024-09-13T17:08:50.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33762/0 (socket says 172.21.15.28:33762) 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 -- 172.21.15.28:0/917842080 learned_addr learned my addr 172.21.15.28:0/917842080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647d400640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647e800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 0x7f64781f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 -- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 msgr2=0x7f6478154670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478154670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 -- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 msgr2=0x7f64781f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 0x7f64781f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647de00640 1 -- 172.21.15.28:0/917842080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474009b30 con 0x7f647806c250 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647e800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 0x7f64781f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.504+0000 7f647d400640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f647de00640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e730 tx=0x7f646c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c004230 con 0x7f647806c250 2024-09-13T17:08:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f646c004390 con 0x7f647806c250 2024-09-13T17:08:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f646c00c8f0 con 0x7f647806c250 2024-09-13T17:08:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64780a8e50 con 0x7f647806c250 2024-09-13T17:08:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.505+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64780ac470 con 0x7f647806c250 2024-09-13T17:08:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.506+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647806bb40 con 0x7f647806c250 2024-09-13T17:08:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.507+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f646c00ca50 con 0x7f647806c250 2024-09-13T17:08:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.508+0000 7f6472000640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 0x7f645c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.508+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f646c088fe0 con 0x7f647806c250 2024-09-13T17:08:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.508+0000 7f647d400640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 0x7f645c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.509+0000 7f647d400640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 0x7f645c06d0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f64780a9780 tx=0x7f6468009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.512+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f646c054160 con 0x7f647806c250 2024-09-13T17:08:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.688+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64780ad8f0 con 0x7f647806c250 2024-09-13T17:08:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.688+0000 7f6472000640 1 -- 172.21.15.28:0/917842080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f646c057780 con 0x7f647806c250 2024-09-13T17:08:50.690 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:50.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 msgr2=0x7f645c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 0x7f645c06d0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f64780a9780 tx=0x7f6468009040 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 msgr2=0x7f6478154130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e730 tx=0x7f646c00ebf0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 shutdown_connections 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f645c06ac00 0x7f645c06d0c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f647815ab60 0x7f64781f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f647815a780 0x7f6478154670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 --2- 172.21.15.28:0/917842080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f647806c250 0x7f6478154130 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 >> 172.21.15.28:0/917842080 conn(0x7f64780b0270 msgr2=0x7f6478070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 shutdown_connections 2024-09-13T17:08:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.690+0000 7f6480800640 1 -- 172.21.15.28:0/917842080 wait complete. 2024-09-13T17:08:50.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.865+0000 7fc68da00640 1 Processor -- start 2024-09-13T17:08:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.865+0000 7fc68da00640 1 -- start start 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.866+0000 7fc68da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc68815a780 0x7fc688070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.866+0000 7fc68da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.866+0000 7fc68da00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68806c8b0 con 0x7fc68806bb40 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.866+0000 7fc68da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68806cb90 con 0x7fc68815a780 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.866+0000 7fc68da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6880cc980 con 0x7fc68815ab60 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc686a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc68815a780 0x7fc688070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56436/0 (socket says 172.21.15.28:56436) 2024-09-13T17:08:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 -- 172.21.15.28:0/1493470469 learned_addr learned my addr 172.21.15.28:0/1493470469 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 -- 172.21.15.28:0/1493470469 >> v2:172.21.15.189:3300/0 conn(0x7fc68815a780 msgr2=0x7fc688070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc686000640 1 -- 172.21.15.28:0/1493470469 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378124914 0 0) 0x7fc68806c8b0 con 0x7fc68806bb40 2024-09-13T17:08:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 --2- 172.21.15.28:0/1493470469 >> v2:172.21.15.189:3300/0 conn(0x7fc68815a780 0x7fc688070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 -- 172.21.15.28:0/1493470469 >> v1:172.21.15.189:6789/0 conn(0x7fc68806bb40 legacy=0x7fc68806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.867+0000 7fc67fe00640 1 -- 172.21.15.28:0/1493470469 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6880ccc60 con 0x7fc68815ab60 2024-09-13T17:08:50.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc67fe00640 1 --2- 172.21.15.28:0/1493470469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc670009870 tx=0x7fc67002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b83644010490e8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc686000640 1 -- 172.21.15.28:0/1493470469 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670032440 con 0x7fc68815ab60 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc686000640 1 -- 172.21.15.28:0/1493470469 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6700325a0 con 0x7fc68815ab60 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc686000640 1 -- 172.21.15.28:0/1493470469 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670032930 con 0x7fc68815ab60 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc68da00640 1 -- 172.21.15.28:0/1493470469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 msgr2=0x7fc6880712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.869+0000 7fc68da00640 1 --2- 172.21.15.28:0/1493470469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc670009870 tx=0x7fc67002a270 comp rx=0 tx=0).stop 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 -- 172.21.15.28:0/1493470469 shutdown_connections 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 --2- 172.21.15.28:0/1493470469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68815ab60 0x7fc6880712c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 --2- 172.21.15.28:0/1493470469 >> v2:172.21.15.189:3300/0 conn(0x7fc68815a780 0x7fc688070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 -- 172.21.15.28:0/1493470469 >> 172.21.15.28:0/1493470469 conn(0x7fc6880b0270 msgr2=0x7fc6880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 -- 172.21.15.28:0/1493470469 shutdown_connections 2024-09-13T17:08:50.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.870+0000 7fc68da00640 1 -- 172.21.15.28:0/1493470469 wait complete. 2024-09-13T17:08:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.871+0000 7fc68da00640 1 Processor -- start 2024-09-13T17:08:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.871+0000 7fc68da00640 1 -- start start 2024-09-13T17:08:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68806bb40 0x7fc688149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68815ab60 0x7fc6880c37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6881528b0 con 0x7fc68815a780 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc688152350 con 0x7fc68815ab60 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.872+0000 7fc68da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc688153950 con 0x7fc68806bb40 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33766/0 (socket says 172.21.15.28:33766) 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc686a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68806bb40 0x7fc688149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 -- 172.21.15.28:0/203244719 learned_addr learned my addr 172.21.15.28:0/203244719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc687400640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68815ab60 0x7fc6880c37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 -- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68806bb40 msgr2=0x7fc688149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68806bb40 0x7fc688149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 -- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68815ab60 msgr2=0x7fc6880c37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68815ab60 0x7fc6880c37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 -- 172.21.15.28:0/203244719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700094d0 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.873+0000 7fc67fe00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc670000c00 tx=0x7fc6700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.874+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670032440 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.874+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670032e20 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.874+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670004270 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.874+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc688153c30 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.874+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6881541f0 con 0x7fc68815a780 2024-09-13T17:08:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.875+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6880c34e0 con 0x7fc68815a780 2024-09-13T17:08:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.876+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc670039050 con 0x7fc68815a780 2024-09-13T17:08:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.877+0000 7fc67ea00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 0x7fc66406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.877+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6700a98c0 con 0x7fc68815a780 2024-09-13T17:08:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.877+0000 7fc686a00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 0x7fc66406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.878+0000 7fc686a00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 0x7fc66406d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc688154900 tx=0x7fc678009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:50.881+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6880c34e0 con 0x7fc68815a780 2024-09-13T17:08:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.065+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc6880c34e0 con 0x7fc68815a780 2024-09-13T17:08:51.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.066+0000 7fc67ea00640 1 -- 172.21.15.28:0/203244719 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc67004b020 con 0x7fc68815a780 2024-09-13T17:08:51.068 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 msgr2=0x7fc66406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 0x7fc66406d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc688154900 tx=0x7fc678009190 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 msgr2=0x7fc68814a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc670000c00 tx=0x7fc6700046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 shutdown_connections 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc66406ac00 0x7fc66406d0c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc68815ab60 0x7fc6880c37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc68815a780 0x7fc68814a500 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 --2- 172.21.15.28:0/203244719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc68806bb40 0x7fc688149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 >> 172.21.15.28:0/203244719 conn(0x7fc6880b0270 msgr2=0x7fc6880cd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:51.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 shutdown_connections 2024-09-13T17:08:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.068+0000 7fc68da00640 1 -- 172.21.15.28:0/203244719 wait complete. 2024-09-13T17:08:51.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.244+0000 7f0425000640 1 Processor -- start 2024-09-13T17:08:51.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.244+0000 7f0425000640 1 -- start start 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f0425000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0420154ce0 0x7f042015b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f0425000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f04200cbec0 0x7f0420070300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f0425000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04201550c0 con 0x7f042006d610 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f0425000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042006e690 con 0x7f0420154ce0 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f0425000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042006ddf0 con 0x7f04200cbec0 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f041de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0420154ce0 0x7f042015b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f041de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0420154ce0 0x7f042015b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33770/0 (socket says 172.21.15.28:33770) 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.245+0000 7f041de00640 1 -- 172.21.15.28:0/1285181889 learned_addr learned my addr 172.21.15.28:0/1285181889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.246+0000 7f041e800640 1 --2- 172.21.15.28:0/1285181889 >> v2:172.21.15.189:3300/0 conn(0x7f04200cbec0 0x7f0420070300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.246+0000 7f041de00640 1 -- 172.21.15.28:0/1285181889 >> v2:172.21.15.189:3300/0 conn(0x7f04200cbec0 msgr2=0x7f0420070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.246+0000 7f041de00640 1 --2- 172.21.15.28:0/1285181889 >> v2:172.21.15.189:3300/0 conn(0x7f04200cbec0 0x7f0420070300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.246+0000 7f041de00640 1 -- 172.21.15.28:0/1285181889 >> v1:172.21.15.28:6789/0 conn(0x7f042006d610 legacy=0x7f04200cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.246+0000 7f041de00640 1 -- 172.21.15.28:0/1285181889 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042006e0d0 con 0x7f0420154ce0 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f041de00640 1 --2- 172.21.15.28:0/1285181889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015b7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04200cc800 tx=0x7f041002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129d07193c066e9e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f041d400640 1 -- 172.21.15.28:0/1285181889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410032440 con 0x7f0420154ce0 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f041d400640 1 -- 172.21.15.28:0/1285181889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04100325a0 con 0x7f0420154ce0 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f041d400640 1 -- 172.21.15.28:0/1285181889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410032930 con 0x7f0420154ce0 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f0425000640 1 -- 172.21.15.28:0/1285181889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 msgr2=0x7f042015b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.247+0000 7f0425000640 1 --2- 172.21.15.28:0/1285181889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015b7e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f04200cc800 tx=0x7f041002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 -- 172.21.15.28:0/1285181889 shutdown_connections 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 --2- 172.21.15.28:0/1285181889 >> v2:172.21.15.189:3300/0 conn(0x7f04200cbec0 0x7f0420070300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 --2- 172.21.15.28:0/1285181889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015b7e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 -- 172.21.15.28:0/1285181889 >> 172.21.15.28:0/1285181889 conn(0x7f04200b0af0 msgr2=0x7f04200b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:51.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 -- 172.21.15.28:0/1285181889 shutdown_connections 2024-09-13T17:08:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.248+0000 7f0425000640 1 -- 172.21.15.28:0/1285181889 wait complete. 2024-09-13T17:08:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.249+0000 7f0425000640 1 Processor -- start 2024-09-13T17:08:51.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 -- start start 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f042006d610 0x7f04201e4080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 0x7f042013ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042013b900 con 0x7f0420154ce0 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0420142eb0 con 0x7f042006d610 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.250+0000 7f0425000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042013d950 con 0x7f04200cbec0 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33784/0 (socket says 172.21.15.28:33784) 2024-09-13T17:08:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 -- 172.21.15.28:0/59762706 learned_addr learned my addr 172.21.15.28:0/59762706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f0417e00640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 0x7f042013ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041de00640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f042006d610 0x7f04201e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 -- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 msgr2=0x7f042013ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 0x7f042013ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 -- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f042006d610 msgr2=0x7f04201e4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f042006d610 0x7f04201e4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f041e800640 1 -- 172.21.15.28:0/59762706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04100094d0 con 0x7f0420154ce0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.251+0000 7f0417e00640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 0x7f042013ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f041e800640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0418007de0 tx=0x7f041800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418014620 con 0x7f0420154ce0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041800ee90 con 0x7f0420154ce0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04180149e0 con 0x7f0420154ce0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0420142580 con 0x7f0420154ce0 2024-09-13T17:08:51.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.252+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0420142b40 con 0x7f0420154ce0 2024-09-13T17:08:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.253+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0420143020 con 0x7f0420154ce0 2024-09-13T17:08:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.254+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0418014b40 con 0x7f0420154ce0 2024-09-13T17:08:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.254+0000 7f0416000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 0x7f03fc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.255+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f0418089db0 con 0x7f0420154ce0 2024-09-13T17:08:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.255+0000 7f041de00640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 0x7f03fc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.256+0000 7f041de00640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 0x7f03fc06d0f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0410034000 tx=0x7f041002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.259+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f041808e070 con 0x7f0420154ce0 2024-09-13T17:08:51.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.440+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0420070300 con 0x7f0420154ce0 2024-09-13T17:08:51.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.441+0000 7f0416000640 1 -- 172.21.15.28:0/59762706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0418054f20 con 0x7f0420154ce0 2024-09-13T17:08:51.443 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 msgr2=0x7f03fc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 0x7f03fc06d0f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f0410034000 tx=0x7f041002e000 comp rx=0 tx=0).stop 2024-09-13T17:08:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 msgr2=0x7f042015e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0418007de0 tx=0x7f041800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 shutdown_connections 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03fc06ac30 0x7f03fc06d0f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0420154ce0 0x7f042015e000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f04200cbec0 0x7f042013ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 --2- 172.21.15.28:0/59762706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f042006d610 0x7f04201e4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 >> 172.21.15.28:0/59762706 conn(0x7f04200b0af0 msgr2=0x7f042015ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 shutdown_connections 2024-09-13T17:08:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.443+0000 7f0425000640 1 -- 172.21.15.28:0/59762706 wait complete. 2024-09-13T17:08:51.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:08:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.617+0000 7ffbf1000640 1 Processor -- start 2024-09-13T17:08:51.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.617+0000 7ffbf1000640 1 -- start start 2024-09-13T17:08:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbf1000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffbec070430 0x7ffbec15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbf1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec15ab60 con 0x7ffbec06f3f0 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbf1000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec06f920 con 0x7ffbec159770 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbf1000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec15b290 con 0x7ffbec070430 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbea800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbec06f3f0 0x7ffbec15c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42176/0 (socket says 172.21.15.28:42176) 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbea800640 1 -- 172.21.15.28:0/639940332 learned_addr learned my addr 172.21.15.28:0/639940332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbe3e00640 1 --2- 172.21.15.28:0/639940332 >> v2:172.21.15.189:3300/0 conn(0x7ffbec070430 0x7ffbec15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbe9400640 1 -- 172.21.15.28:0/639940332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943580800 0 0) 0x7ffbe4001490 con 0x7ffbec06f3f0 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.619+0000 7ffbe9400640 1 -- 172.21.15.28:0/639940332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd0003520 con 0x7ffbec06f3f0 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.620+0000 7ffbe3e00640 1 -- 172.21.15.28:0/639940332 >> v1:172.21.15.189:6789/0 conn(0x7ffbec159770 legacy=0x7ffbec06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.620+0000 7ffbe3e00640 1 -- 172.21.15.28:0/639940332 >> v1:172.21.15.28:6789/0 conn(0x7ffbec06f3f0 legacy=0x7ffbec15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.620+0000 7ffbe3e00640 1 -- 172.21.15.28:0/639940332 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec15b570 con 0x7ffbec070430 2024-09-13T17:08:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.621+0000 7ffbe3e00640 1 --2- 172.21.15.28:0/639940332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec15bd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40097a0 tx=0x7ffbd402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e5ce4914fd2f48b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.621+0000 7ffbe9400640 1 -- 172.21.15.28:0/639940332 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4032440 con 0x7ffbec070430 2024-09-13T17:08:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.621+0000 7ffbe9400640 1 -- 172.21.15.28:0/639940332 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd40325a0 con 0x7ffbec070430 2024-09-13T17:08:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.621+0000 7ffbe9400640 1 -- 172.21.15.28:0/639940332 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4032910 con 0x7ffbec070430 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 -- 172.21.15.28:0/639940332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 msgr2=0x7ffbec15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 --2- 172.21.15.28:0/639940332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec15bd70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40097a0 tx=0x7ffbd402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 -- 172.21.15.28:0/639940332 shutdown_connections 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 --2- 172.21.15.28:0/639940332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec15bd70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 -- 172.21.15.28:0/639940332 >> 172.21.15.28:0/639940332 conn(0x7ffbec0b0270 msgr2=0x7ffbec0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 -- 172.21.15.28:0/639940332 shutdown_connections 2024-09-13T17:08:51.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.622+0000 7ffbf1000640 1 -- 172.21.15.28:0/639940332 wait complete. 2024-09-13T17:08:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.623+0000 7ffbf1000640 1 Processor -- start 2024-09-13T17:08:51.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.624+0000 7ffbf1000640 1 -- start start 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.624+0000 7ffbf1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbec06f3f0 0x7ffbec14ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbf1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec14efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbf1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbf1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1e2780 con 0x7ffbec159770 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbf1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec14d350 con 0x7ffbec070430 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbf1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec1e1b70 con 0x7ffbec06f3f0 2024-09-13T17:08:51.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33788/0 (socket says 172.21.15.28:33788) 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbe3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec14efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 -- 172.21.15.28:0/4275320424 learned_addr learned my addr 172.21.15.28:0/4275320424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbe9e00640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbec06f3f0 0x7ffbec14ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 -- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbec06f3f0 msgr2=0x7ffbec14ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbec06f3f0 0x7ffbec14ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 -- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 msgr2=0x7ffbec14efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec14efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.625+0000 7ffbea800640 1 -- 172.21.15.28:0/4275320424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd4009400 con 0x7ffbec159770 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbea800640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400cb40 tx=0x7ffbe400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe400aca0 con 0x7ffbec159770 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe400ae00 con 0x7ffbec159770 2024-09-13T17:08:51.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec1e1e50 con 0x7ffbec159770 2024-09-13T17:08:51.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbe4012620 con 0x7ffbec159770 2024-09-13T17:08:51.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.626+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec1e2410 con 0x7ffbec159770 2024-09-13T17:08:51.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.627+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec1e2930 con 0x7ffbec159770 2024-09-13T17:08:51.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.628+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffbe4007070 con 0x7ffbec159770 2024-09-13T17:08:51.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.629+0000 7ffbe2000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 0x7ffbc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:51.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.629+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5804+0+0 (secure 0 0 0) 0x7ffbe4089180 con 0x7ffbec159770 2024-09-13T17:08:51.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.629+0000 7ffbe9e00640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 0x7ffbc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:51.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.630+0000 7ffbe9e00640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 0x7ffbc806d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0cae60 tx=0x7ffbdc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:51.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.633+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffbec1e2930 con 0x7ffbec159770 2024-09-13T17:08:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.823+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffbec15ab60 con 0x7ffbec159770 2024-09-13T17:08:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.823+0000 7ffbe2000640 1 -- 172.21.15.28:0/4275320424 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffbec15ab60 con 0x7ffbec159770 2024-09-13T17:08:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:08:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 msgr2=0x7ffbc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 0x7ffbc806d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ffbec0cae60 tx=0x7ffbdc009040 comp rx=0 tx=0).stop 2024-09-13T17:08:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 msgr2=0x7ffbec0b26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ffbe400cb40 tx=0x7ffbe400a060 comp rx=0 tx=0).stop 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 shutdown_connections 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbc806ac00 0x7ffbc806d0c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbec159770 0x7ffbec0b26f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbec070430 0x7ffbec14efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 --2- 172.21.15.28:0/4275320424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbec06f3f0 0x7ffbec14ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 >> 172.21.15.28:0/4275320424 conn(0x7ffbec0b0270 msgr2=0x7ffbec0ad490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 shutdown_connections 2024-09-13T17:08:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:51.825+0000 7ffbf1000640 1 -- 172.21.15.28:0/4275320424 wait complete. 2024-09-13T17:08:51.843 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-09-13T17:08:51.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:51.843 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-09-13T17:08:51.911 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:08:51.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:51.912 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T17:08:51.961 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:51.961 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:51.997 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:51.998 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:52.074 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:08:52.074 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:52.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.349+0000 7fabb7400640 1 Processor -- start 2024-09-13T17:08:52.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.349+0000 7fabb7400640 1 -- start start 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fabb015a540 0x7fabb006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fabb0070cc0 0x7fabb0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb7400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb015e100 con 0x7fabb00cabc0 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb7400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb00adb00 con 0x7fabb015a540 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb7400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb0071670 con 0x7fabb0070cc0 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fabb015a540 0x7fabb006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fabb015a540 0x7fabb006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33794/0 (socket says 172.21.15.28:33794) 2024-09-13T17:08:52.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb4a00640 1 -- 172.21.15.28:0/198798591 learned_addr learned my addr 172.21.15.28:0/198798591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.350+0000 7fabb5400640 1 --2- 172.21.15.28:0/198798591 >> v2:172.21.15.189:3300/0 conn(0x7fabb0070cc0 0x7fabb0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.351+0000 7fabb4a00640 1 -- 172.21.15.28:0/198798591 >> v2:172.21.15.189:3300/0 conn(0x7fabb0070cc0 msgr2=0x7fabb0071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.351+0000 7fabb4a00640 1 --2- 172.21.15.28:0/198798591 >> v2:172.21.15.189:3300/0 conn(0x7fabb0070cc0 0x7fabb0071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.351+0000 7fabb4a00640 1 -- 172.21.15.28:0/198798591 >> v1:172.21.15.189:6789/0 conn(0x7fabb00cabc0 legacy=0x7fabb015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.351+0000 7fabb4a00640 1 -- 172.21.15.28:0/198798591 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb015e3e0 con 0x7fabb015a540 2024-09-13T17:08:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.352+0000 7fabb4a00640 1 --2- 172.21.15.28:0/198798591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb015a540 0x7fabb006c430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fabb0070900 tx=0x7faba402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fcfa0a5f13acadc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.352+0000 7fabaf400640 1 -- 172.21.15.28:0/198798591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4032440 con 0x7fabb015a540 2024-09-13T17:08:52.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.352+0000 7fabaf400640 1 -- 172.21.15.28:0/198798591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba40325a0 con 0x7fabb015a540 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.352+0000 7fabaf400640 1 -- 172.21.15.28:0/198798591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4032930 con 0x7fabb015a540 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 -- 172.21.15.28:0/198798591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb015a540 msgr2=0x7fabb006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 --2- 172.21.15.28:0/198798591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb015a540 0x7fabb006c430 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fabb0070900 tx=0x7faba402a210 comp rx=0 tx=0).stop 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 -- 172.21.15.28:0/198798591 shutdown_connections 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 --2- 172.21.15.28:0/198798591 >> v2:172.21.15.189:3300/0 conn(0x7fabb0070cc0 0x7fabb0071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 --2- 172.21.15.28:0/198798591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb015a540 0x7fabb006c430 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 -- 172.21.15.28:0/198798591 >> 172.21.15.28:0/198798591 conn(0x7fabb0069e80 msgr2=0x7fabb006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 -- 172.21.15.28:0/198798591 shutdown_connections 2024-09-13T17:08:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.353+0000 7fabb7400640 1 -- 172.21.15.28:0/198798591 wait complete. 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.354+0000 7fabb7400640 1 Processor -- start 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.354+0000 7fabb7400640 1 -- start start 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabb00cabc0 0x7fabb014f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb015a540 0x7fabb013baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb014cc10 con 0x7fabb0070cc0 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb014e2f0 con 0x7fabb015a540 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.355+0000 7fabb7400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb014e5d0 con 0x7fabb00cabc0 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33800/0 (socket says 172.21.15.28:33800) 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 -- 172.21.15.28:0/3396469683 learned_addr learned my addr 172.21.15.28:0/3396469683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb5400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb015a540 0x7fabb013baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabafe00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabb00cabc0 0x7fabb014f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 -- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabb00cabc0 msgr2=0x7fabb014f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabb00cabc0 0x7fabb014f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 -- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb015a540 msgr2=0x7fabb013baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb015a540 0x7fabb013baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 -- 172.21.15.28:0/3396469683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba40094d0 con 0x7fabb0070cc0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.356+0000 7fabb4a00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faba4000c00 tx=0x7faba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.357+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4032440 con 0x7fabb0070cc0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.357+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba4030980 con 0x7fabb0070cc0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.357+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba4030d30 con 0x7fabb0070cc0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.357+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb014e8b0 con 0x7fabb0070cc0 2024-09-13T17:08:52.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.357+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb014dfe0 con 0x7fabb0070cc0 2024-09-13T17:08:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.358+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb014edd0 con 0x7fabb0070cc0 2024-09-13T17:08:52.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.359+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faba4039040 con 0x7fabb0070cc0 2024-09-13T17:08:52.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.360+0000 7fabacc00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 0x7fab9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.360+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7faba40a9bc0 con 0x7fabb0070cc0 2024-09-13T17:08:52.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.360+0000 7fabafe00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 0x7fab9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.361+0000 7fabafe00640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 0x7fab9006d0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fabb014dd00 tx=0x7fab9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.364+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faba4074df0 con 0x7fabb0070cc0 2024-09-13T17:08:52.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.541+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabb014e710 con 0x7fabb0070cc0 2024-09-13T17:08:52.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.542+0000 7fabacc00640 1 -- 172.21.15.28:0/3396469683 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16725 (secure 0 0 0) 0x7fabb014e710 con 0x7fabb0070cc0 2024-09-13T17:08:52.545 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:52.545 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:52.005252+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.544+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 msgr2=0x7fab9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.544+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 0x7fab9006d0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fabb014dd00 tx=0x7fab9c009040 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.544+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 msgr2=0x7fabb014c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.544+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faba4000c00 tx=0x7faba40046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 shutdown_connections 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab9006ac00 0x7fab9006d0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabb015a540 0x7fabb013baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabb00cabc0 0x7fabb014f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 --2- 172.21.15.28:0/3396469683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabb0070cc0 0x7fabb014c510 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 >> 172.21.15.28:0/3396469683 conn(0x7fabb0069e80 msgr2=0x7fabb00cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 shutdown_connections 2024-09-13T17:08:52.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.545+0000 7fabb7400640 1 -- 172.21.15.28:0/3396469683 wait complete. 2024-09-13T17:08:52.563 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:52.564 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:52.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.721+0000 7fad48c00640 1 Processor -- start 2024-09-13T17:08:52.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.721+0000 7fad48c00640 1 -- start start 2024-09-13T17:08:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.722+0000 7fad48c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fad4415ab60 0x7fad4406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.722+0000 7fad48c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad44071490 0x7fad4406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.722+0000 7fad48c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad4406c8b0 con 0x7fad4415a780 2024-09-13T17:08:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad48c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad4406cb90 con 0x7fad44071490 2024-09-13T17:08:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad48c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440cc980 con 0x7fad4415ab60 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad42800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad44071490 0x7fad4406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad41e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad4415a780 0x7fad44070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42184/0 (socket says 172.21.15.28:42184) 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad41e00640 1 -- 172.21.15.28:0/2624997363 learned_addr learned my addr 172.21.15.28:0/2624997363 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad3be00640 1 --2- 172.21.15.28:0/2624997363 >> v2:172.21.15.189:3300/0 conn(0x7fad4415ab60 0x7fad4406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad42800640 1 -- 172.21.15.28:0/2624997363 >> v2:172.21.15.189:3300/0 conn(0x7fad4415ab60 msgr2=0x7fad4406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad42800640 1 --2- 172.21.15.28:0/2624997363 >> v2:172.21.15.189:3300/0 conn(0x7fad4415ab60 0x7fad4406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad42800640 1 -- 172.21.15.28:0/2624997363 >> v1:172.21.15.28:6789/0 conn(0x7fad4415a780 legacy=0x7fad44070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad42800640 1 -- 172.21.15.28:0/2624997363 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440ccc60 con 0x7fad44071490 2024-09-13T17:08:52.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.723+0000 7fad41400640 1 -- 172.21.15.28:0/2624997363 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353961864 0 0) 0x7fad4406c8b0 con 0x7fad4415a780 2024-09-13T17:08:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.724+0000 7fad42800640 1 --2- 172.21.15.28:0/2624997363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44071490 0x7fad4406c080 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad4406f380 tx=0x7fad3c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b970dce15baa0917 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad41400640 1 -- 172.21.15.28:0/2624997363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c02aea0 con 0x7fad44071490 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad41400640 1 -- 172.21.15.28:0/2624997363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c02d050 con 0x7fad44071490 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad41400640 1 -- 172.21.15.28:0/2624997363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c007960 con 0x7fad44071490 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 -- 172.21.15.28:0/2624997363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44071490 msgr2=0x7fad4406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 --2- 172.21.15.28:0/2624997363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44071490 0x7fad4406c080 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad4406f380 tx=0x7fad3c02a680 comp rx=0 tx=0).stop 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 -- 172.21.15.28:0/2624997363 shutdown_connections 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 --2- 172.21.15.28:0/2624997363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad44071490 0x7fad4406c080 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 --2- 172.21.15.28:0/2624997363 >> v2:172.21.15.189:3300/0 conn(0x7fad4415ab60 0x7fad4406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 -- 172.21.15.28:0/2624997363 >> 172.21.15.28:0/2624997363 conn(0x7fad440b0270 msgr2=0x7fad440b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 -- 172.21.15.28:0/2624997363 shutdown_connections 2024-09-13T17:08:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.725+0000 7fad48c00640 1 -- 172.21.15.28:0/2624997363 wait complete. 2024-09-13T17:08:52.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.727+0000 7fad48c00640 1 Processor -- start 2024-09-13T17:08:52.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.727+0000 7fad48c00640 1 -- start start 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad44071490 0x7fad441f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 0x7fad44156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4415bfc0 con 0x7fad4415ab60 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44156620 con 0x7fad4415a780 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad48c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad4415c280 con 0x7fad44071490 2024-09-13T17:08:52.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad42800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad42800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33828/0 (socket says 172.21.15.28:33828) 2024-09-13T17:08:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad42800640 1 -- 172.21.15.28:0/1497001544 learned_addr learned my addr 172.21.15.28:0/1497001544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad3be00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 0x7fad44156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.728+0000 7fad41e00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad44071490 0x7fad441f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 -- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad44071490 msgr2=0x7fad441f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad44071490 0x7fad441f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 -- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 msgr2=0x7fad44156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 0x7fad44156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 -- 172.21.15.28:0/1497001544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3c009a70 con 0x7fad4415ab60 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad3be00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 0x7fad44156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad42800640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad3c02ac70 tx=0x7fad3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c03c830 con 0x7fad4415ab60 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c03c990 con 0x7fad4415ab60 2024-09-13T17:08:52.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad4415c560 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c03cd40 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.729+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4415cb20 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.730+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad44070d80 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.731+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad3c044410 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.732+0000 7fad3a000640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 0x7fad2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.732+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fad3c0aab50 con 0x7fad4415ab60 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.732+0000 7fad41e00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 0x7fad2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.733+0000 7fad41e00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 0x7fad2006d0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fad34000f60 tx=0x7fad34007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.736+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fad3c075c30 con 0x7fad4415ab60 2024-09-13T17:08:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.913+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad440cdf10 con 0x7fad4415ab60 2024-09-13T17:08:52.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.914+0000 7fad3a000640 1 -- 172.21.15.28:0/1497001544 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad3c040020 con 0x7fad4415ab60 2024-09-13T17:08:52.916 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 msgr2=0x7fad2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 0x7fad2006d0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fad34000f60 tx=0x7fad34007b50 comp rx=0 tx=0).stop 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 msgr2=0x7fad44154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fad3c02ac70 tx=0x7fad3c004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 shutdown_connections 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad2006ac00 0x7fad2006d0c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad4415ab60 0x7fad44154760 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad4415a780 0x7fad44156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 --2- 172.21.15.28:0/1497001544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad44071490 0x7fad441f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:52.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 >> 172.21.15.28:0/1497001544 conn(0x7fad440b0270 msgr2=0x7fad440cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:52.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 shutdown_connections 2024-09-13T17:08:52.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:52.916+0000 7fad48c00640 1 -- 172.21.15.28:0/1497001544 wait complete. 2024-09-13T17:08:52.933 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:53.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.092+0000 7f6045a00640 1 Processor -- start 2024-09-13T17:08:53.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.092+0000 7f6045a00640 1 -- start start 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 0x7f60400b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 0x7f60400ce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60400cf560 con 0x7f60400b11d0 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60400cb5f0 con 0x7f6040160300 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604006b1a0 con 0x7f604015ff20 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f60400b11d0 0x7f60400cee50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42192/0 (socket says 172.21.15.28:42192) 2024-09-13T17:08:53.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6045000640 1 -- 172.21.15.28:0/4022695721 learned_addr learned my addr 172.21.15.28:0/4022695721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f6037e00640 1 --2- 172.21.15.28:0/4022695721 >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 0x7f60400ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f603fe00640 1 --2- 172.21.15.28:0/4022695721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 0x7f60400b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.093+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1742918796 0 0) 0x7f60400cf560 con 0x7f60400b11d0 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003520 con 0x7f60400b11d0 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603fe00640 1 -- 172.21.15.28:0/4022695721 >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 msgr2=0x7f60400ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603fe00640 1 --2- 172.21.15.28:0/4022695721 >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 0x7f60400ce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603fe00640 1 -- 172.21.15.28:0/4022695721 >> v1:172.21.15.28:6789/0 conn(0x7f60400b11d0 legacy=0x7f60400cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966957300 0 0) 0x7f6024003520 con 0x7f60400b11d0 2024-09-13T17:08:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f603fe00640 1 -- 172.21.15.28:0/4022695721 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400cf820 con 0x7f604015ff20 2024-09-13T17:08:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.094+0000 7f6037e00640 1 --2- 172.21.15.28:0/4022695721 >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 0x7f60400ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.095+0000 7f603fe00640 1 --2- 172.21.15.28:0/4022695721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 0x7f60400b2c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60300097a0 tx=0x7f6030029ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73a1d6253d5ed632 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.095+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60380019f0 con 0x7f604015ff20 2024-09-13T17:08:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.095+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603002aa80 con 0x7f604015ff20 2024-09-13T17:08:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.095+0000 7f603f400640 1 -- 172.21.15.28:0/4022695721 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603002ae10 con 0x7f604015ff20 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 -- 172.21.15.28:0/4022695721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 msgr2=0x7f60400b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 --2- 172.21.15.28:0/4022695721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 0x7f60400b2c70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f60300097a0 tx=0x7f6030029ff0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 -- 172.21.15.28:0/4022695721 shutdown_connections 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 --2- 172.21.15.28:0/4022695721 >> v2:172.21.15.189:3300/0 conn(0x7f6040160300 0x7f60400ce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 --2- 172.21.15.28:0/4022695721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f604015ff20 0x7f60400b2c70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 -- 172.21.15.28:0/4022695721 >> 172.21.15.28:0/4022695721 conn(0x7f60400b2150 msgr2=0x7f60400b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 -- 172.21.15.28:0/4022695721 shutdown_connections 2024-09-13T17:08:53.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.096+0000 7f6045a00640 1 -- 172.21.15.28:0/4022695721 wait complete. 2024-09-13T17:08:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.098+0000 7f6045a00640 1 Processor -- start 2024-09-13T17:08:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.098+0000 7f6045a00640 1 -- start start 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f604015ff20 0x7f60400b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 0x7f60400b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400c7a00 con 0x7f60400b11d0 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60400b61b0 con 0x7f604015ff20 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6040151030 con 0x7f60400b5d20 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33832/0 (socket says 172.21.15.28:33832) 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 -- 172.21.15.28:0/626040079 learned_addr learned my addr 172.21.15.28:0/626040079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6037e00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f604015ff20 0x7f60400b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045000640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 0x7f60400b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 -- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 msgr2=0x7f60400b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 0x7f60400b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 -- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f604015ff20 msgr2=0x7f60400b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f604015ff20 0x7f60400b57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f603fe00640 1 -- 172.21.15.28:0/626040079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028009400 con 0x7f60400b11d0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.099+0000 7f6045000640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 0x7f60400b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f603fe00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6030035040 tx=0x7f60300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603003b810 con 0x7f60400b11d0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603003b970 con 0x7f60400b11d0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603003bd20 con 0x7f60400b11d0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6030009400 con 0x7f60400b11d0 2024-09-13T17:08:53.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.100+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60400b4cb0 con 0x7f60400b11d0 2024-09-13T17:08:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.101+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60400c7500 con 0x7f60400b11d0 2024-09-13T17:08:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.102+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6030039040 con 0x7f60400b11d0 2024-09-13T17:08:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.103+0000 7f603cc00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 0x7f601c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.103+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f60300a99f0 con 0x7f60400b11d0 2024-09-13T17:08:53.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.103+0000 7f6037e00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 0x7f601c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.104+0000 7f6037e00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 0x7f601c06d0c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60400b54b0 tx=0x7f6028009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.107+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6030074b80 con 0x7f60400b11d0 2024-09-13T17:08:53.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.287+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f60400b4e40 con 0x7f60400b11d0 2024-09-13T17:08:53.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.287+0000 7f603cc00640 1 -- 172.21.15.28:0/626040079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f60300781a0 con 0x7f60400b11d0 2024-09-13T17:08:53.289 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 msgr2=0x7f601c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 0x7f601c06d0c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f60400b54b0 tx=0x7f6028009190 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 msgr2=0x7f6040150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6030035040 tx=0x7f60300046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 shutdown_connections 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f601c06ac00 0x7f601c06d0c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60400b5d20 0x7f60400b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f604015ff20 0x7f60400b57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 --2- 172.21.15.28:0/626040079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60400b11d0 0x7f6040150af0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 >> 172.21.15.28:0/626040079 conn(0x7f60400b2150 msgr2=0x7f60400b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 shutdown_connections 2024-09-13T17:08:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.290+0000 7f6045a00640 1 -- 172.21.15.28:0/626040079 wait complete. 2024-09-13T17:08:53.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.466+0000 7f0e1ca00640 1 Processor -- start 2024-09-13T17:08:53.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.466+0000 7f0e1ca00640 1 -- start start 2024-09-13T17:08:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e1815a780 0x7f0e18070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e1806bb40 0x7f0e1806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1806c8b0 con 0x7f0e1815a780 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1806cb90 con 0x7f0e1806bb40 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e1ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180cc980 con 0x7f0e1815ab60 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e15e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e1815a780 0x7f0e18070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e15e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e1815a780 0x7f0e18070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33838/0 (socket says 172.21.15.28:33838) 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.467+0000 7f0e15e00640 1 -- 172.21.15.28:0/4042129345 learned_addr learned my addr 172.21.15.28:0/4042129345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e16800640 1 --2- 172.21.15.28:0/4042129345 >> v2:172.21.15.189:3300/0 conn(0x7f0e1806bb40 0x7f0e1806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e0d400640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e15e00640 1 -- 172.21.15.28:0/4042129345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 msgr2=0x7f0e180712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e15e00640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e15e00640 1 -- 172.21.15.28:0/4042129345 >> v2:172.21.15.189:3300/0 conn(0x7f0e1806bb40 msgr2=0x7f0e1806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e15e00640 1 --2- 172.21.15.28:0/4042129345 >> v2:172.21.15.189:3300/0 conn(0x7f0e1806bb40 0x7f0e1806bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.468+0000 7f0e15e00640 1 -- 172.21.15.28:0/4042129345 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e180ccc60 con 0x7f0e1815a780 2024-09-13T17:08:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.469+0000 7f0e15e00640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1815a780 0x7f0e18070d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e1806f1b0 tx=0x7f0e0802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9eab36777a93206 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.469+0000 7f0e15400640 1 -- 172.21.15.28:0/4042129345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08032440 con 0x7f0e1815a780 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.469+0000 7f0e15400640 1 -- 172.21.15.28:0/4042129345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e080325a0 con 0x7f0e1815a780 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.469+0000 7f0e15400640 1 -- 172.21.15.28:0/4042129345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08032930 con 0x7f0e1815a780 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 -- 172.21.15.28:0/4042129345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1815a780 msgr2=0x7f0e18070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1815a780 0x7f0e18070d80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f0e1806f1b0 tx=0x7f0e0802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 -- 172.21.15.28:0/4042129345 shutdown_connections 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/4042129345 >> v2:172.21.15.189:3300/0 conn(0x7f0e1806bb40 0x7f0e1806bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/4042129345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1815a780 0x7f0e18070d80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 -- 172.21.15.28:0/4042129345 >> 172.21.15.28:0/4042129345 conn(0x7f0e180b0270 msgr2=0x7f0e180b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 -- 172.21.15.28:0/4042129345 shutdown_connections 2024-09-13T17:08:53.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.470+0000 7f0e1ca00640 1 -- 172.21.15.28:0/4042129345 wait complete. 2024-09-13T17:08:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.471+0000 7f0e1ca00640 1 Processor -- start 2024-09-13T17:08:53.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.472+0000 7f0e1ca00640 1 -- start start 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.472+0000 7f0e1ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.472+0000 7f0e1ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 0x7f0e180c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e1ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e1ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180c4820 con 0x7f0e1806bb40 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e1ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180c34e0 con 0x7f0e1815a780 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e1ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e180c3780 con 0x7f0e1815ab60 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33840/0 (socket says 172.21.15.28:33840) 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 -- 172.21.15.28:0/129498278 learned_addr learned my addr 172.21.15.28:0/129498278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e16800640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e0d400640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 0x7f0e180c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 -- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 msgr2=0x7f0e180b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 -- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 msgr2=0x7f0e180c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 0x7f0e180c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 -- 172.21.15.28:0/129498278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e080094d0 con 0x7f0e1806bb40 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e16800640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e0d400640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 0x7f0e180c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.473+0000 7f0e15e00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0e0802a220 tx=0x7f0e080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.474+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08032440 con 0x7f0e1806bb40 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.474+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08030ba0 con 0x7f0e1806bb40 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.474+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0803d440 con 0x7f0e1806bb40 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.474+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e180c3a60 con 0x7f0e1806bb40 2024-09-13T17:08:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.474+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e180c4020 con 0x7f0e1806bb40 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.475+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e1806bfd0 con 0x7f0e1806bb40 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.476+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e08039040 con 0x7f0e1806bb40 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.477+0000 7f0e0ea00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 0x7f0df406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.477+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f0e080a9ff0 con 0x7f0e1806bb40 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.477+0000 7f0e0d400640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 0x7f0df406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.478+0000 7f0e0d400640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 0x7f0df406d0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e181f5900 tx=0x7f0e00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.481+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0e080dd9c0 con 0x7f0e1806bb40 2024-09-13T17:08:53.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.660+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0e180ca830 con 0x7f0e1806bb40 2024-09-13T17:08:53.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.661+0000 7f0e0ea00640 1 -- 172.21.15.28:0/129498278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0e080410a0 con 0x7f0e1806bb40 2024-09-13T17:08:53.662 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 msgr2=0x7f0df406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 0x7f0df406d0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0e181f5900 tx=0x7f0e00008040 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 msgr2=0x7f0e181e8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0e0802a220 tx=0x7f0e080046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 shutdown_connections 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df406ac00 0x7f0df406d0c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e1815ab60 0x7f0e180b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e1815a780 0x7f0e180c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 --2- 172.21.15.28:0/129498278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e1806bb40 0x7f0e181e8430 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.663+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 >> 172.21.15.28:0/129498278 conn(0x7f0e180b0270 msgr2=0x7f0e180aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.664+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 shutdown_connections 2024-09-13T17:08:53.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.664+0000 7f0e1ca00640 1 -- 172.21.15.28:0/129498278 wait complete. 2024-09-13T17:08:53.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:08:53.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.835+0000 7fca2da00640 1 Processor -- start 2024-09-13T17:08:53.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.835+0000 7fca2da00640 1 -- start start 2024-09-13T17:08:53.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca2da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca28070430 0x7fca2815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca2da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca2806f3f0 0x7fca2815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca2da00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca2815ab60 con 0x7fca28159770 2024-09-13T17:08:53.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca2da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca2806f920 con 0x7fca28070430 2024-09-13T17:08:53.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca2da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca2815b290 con 0x7fca2806f3f0 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca1e000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca28070430 0x7fca2815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca26a00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fca28159770 0x7fca2806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:56490/0 (socket says 172.21.15.28:56490) 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca1e000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca28070430 0x7fca2815bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58602/0 (socket says 172.21.15.28:58602) 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca26a00640 1 -- 172.21.15.28:0/3555894124 learned_addr learned my addr 172.21.15.28:0/3555894124 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.837+0000 7fca27400640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca2806f3f0 0x7fca2815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.838+0000 7fca1e000640 1 -- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca2806f3f0 msgr2=0x7fca2815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.838+0000 7fca1e000640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca2806f3f0 0x7fca2815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.838+0000 7fca1e000640 1 -- 172.21.15.28:0/3555894124 >> v1:172.21.15.189:6789/0 conn(0x7fca28159770 legacy=0x7fca2806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.838+0000 7fca1e000640 1 -- 172.21.15.28:0/3555894124 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca2815b570 con 0x7fca28070430 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.839+0000 7fca1e000640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28070430 0x7fca2815bd70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fca1000ba40 tx=0x7fca1002c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb26356ca68a766 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.839+0000 7fca26000640 1 -- 172.21.15.28:0/3555894124 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10031710 con 0x7fca28070430 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.839+0000 7fca26000640 1 -- 172.21.15.28:0/3555894124 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca1002f050 con 0x7fca28070430 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.839+0000 7fca26000640 1 -- 172.21.15.28:0/3555894124 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca10031a90 con 0x7fca28070430 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 -- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28070430 msgr2=0x7fca2815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28070430 0x7fca2815bd70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fca1000ba40 tx=0x7fca1002c7b0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 -- 172.21.15.28:0/3555894124 shutdown_connections 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca2806f3f0 0x7fca2815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 --2- 172.21.15.28:0/3555894124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28070430 0x7fca2815bd70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 -- 172.21.15.28:0/3555894124 >> 172.21.15.28:0/3555894124 conn(0x7fca280b0270 msgr2=0x7fca280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 -- 172.21.15.28:0/3555894124 shutdown_connections 2024-09-13T17:08:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.840+0000 7fca2da00640 1 -- 172.21.15.28:0/3555894124 wait complete. 2024-09-13T17:08:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.841+0000 7fca2da00640 1 Processor -- start 2024-09-13T17:08:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 -- start start 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 0x7fca281e1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 0x7fca2806bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca281e1750 con 0x7fca2806f3f0 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca280ca410 con 0x7fca28159770 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.842+0000 7fca2da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca280acee0 con 0x7fca28070430 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48166/0 (socket says 172.21.15.28:48166) 2024-09-13T17:08:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 -- 172.21.15.28:0/806044654 learned_addr learned my addr 172.21.15.28:0/806044654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca1e000640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 0x7fca281e1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca27400640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 0x7fca2806bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 -- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 msgr2=0x7fca281e1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 0x7fca281e1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 -- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 msgr2=0x7fca2806bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 0x7fca2806bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca26a00640 1 -- 172.21.15.28:0/806044654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca1000b6a0 con 0x7fca2806f3f0 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca1e000640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 0x7fca281e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:53.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.843+0000 7fca27400640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 0x7fca2806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca26a00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c480 tx=0x7fca1800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca18005ce0 con 0x7fca2806f3f0 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca18005e40 con 0x7fca2806f3f0 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca180072f0 con 0x7fca2806f3f0 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca280ad1c0 con 0x7fca2806f3f0 2024-09-13T17:08:53.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.844+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2806c1b0 con 0x7fca2806f3f0 2024-09-13T17:08:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.845+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca2806c3a0 con 0x7fca2806f3f0 2024-09-13T17:08:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.846+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca18007510 con 0x7fca2806f3f0 2024-09-13T17:08:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.847+0000 7fca1f400640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 0x7fca0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.847+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fca18088710 con 0x7fca2806f3f0 2024-09-13T17:08:53.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.847+0000 7fca1e000640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 0x7fca0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:53.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.848+0000 7fca1e000640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 0x7fca0406d0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca28068040 tx=0x7fca10031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:53.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:53.851+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fca18053930 con 0x7fca2806f3f0 2024-09-13T17:08:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.042+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca280ad3b0 con 0x7fca2806f3f0 2024-09-13T17:08:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.043+0000 7fca1f400640 1 -- 172.21.15.28:0/806044654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fca280ad3b0 con 0x7fca2806f3f0 2024-09-13T17:08:54.044 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:08:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 msgr2=0x7fca0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 0x7fca0406d0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fca28068040 tx=0x7fca10031000 comp rx=0 tx=0).stop 2024-09-13T17:08:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 msgr2=0x7fca280c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca1800c480 tx=0x7fca1800c940 comp rx=0 tx=0).stop 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 shutdown_connections 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca0406ac00 0x7fca0406d0c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca28159770 0x7fca2806bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca28070430 0x7fca281e1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 --2- 172.21.15.28:0/806044654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca2806f3f0 0x7fca280c9c20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 >> 172.21.15.28:0/806044654 conn(0x7fca280b0270 msgr2=0x7fca28158780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 shutdown_connections 2024-09-13T17:08:54.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.045+0000 7fca2da00640 1 -- 172.21.15.28:0/806044654 wait complete. 2024-09-13T17:08:54.062 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-09-13T17:08:54.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:54.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-09-13T17:08:54.122 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:08:54.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:54.122 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T17:08:54.158 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:54.158 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:54.201 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:54.202 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:54.279 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:08:54.280 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:08:54.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:54.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.511+0000 7f20aa600640 1 Processor -- start 2024-09-13T17:08:54.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.512+0000 7f20aa600640 1 -- start start 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20aa600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20a40b0b50 0x7f20a40b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20aa600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f20a4159820 0x7f20a40b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20aa600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a40c3850 con 0x7f20a4159440 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20aa600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a40b2d60 con 0x7f20a40b0b50 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20aa600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a415a930 con 0x7f20a4159820 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20a3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20a40b0b50 0x7f20a40b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20a3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f20a40b0b50 0x7f20a40b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48178/0 (socket says 172.21.15.28:48178) 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20a3e00640 1 -- 172.21.15.28:0/509062737 learned_addr learned my addr 172.21.15.28:0/509062737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.513+0000 7f20a9c00640 1 --2- 172.21.15.28:0/509062737 >> v2:172.21.15.189:3300/0 conn(0x7f20a4159820 0x7f20a40b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.514+0000 7f20a3e00640 1 -- 172.21.15.28:0/509062737 >> v2:172.21.15.189:3300/0 conn(0x7f20a4159820 msgr2=0x7f20a40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.514+0000 7f20a3e00640 1 --2- 172.21.15.28:0/509062737 >> v2:172.21.15.189:3300/0 conn(0x7f20a4159820 0x7f20a40b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.514+0000 7f20a3e00640 1 -- 172.21.15.28:0/509062737 >> v1:172.21.15.28:6789/0 conn(0x7f20a4159440 legacy=0x7f20a40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.514+0000 7f20a3e00640 1 -- 172.21.15.28:0/509062737 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4159d20 con 0x7f20a40b0b50 2024-09-13T17:08:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.514+0000 7f20a3400640 1 -- 172.21.15.28:0/509062737 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075591379 0 0) 0x7f20a40c3850 con 0x7f20a4159440 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.515+0000 7f20a3e00640 1 --2- 172.21.15.28:0/509062737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a40b22e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f20a406d2d0 tx=0x7f209002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ff9075b8ce366e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.515+0000 7f20a3400640 1 -- 172.21.15.28:0/509062737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090032640 con 0x7f20a40b0b50 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.515+0000 7f20a3400640 1 -- 172.21.15.28:0/509062737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900327a0 con 0x7f20a40b0b50 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.515+0000 7f20a3400640 1 -- 172.21.15.28:0/509062737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090032b30 con 0x7f20a40b0b50 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 -- 172.21.15.28:0/509062737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 msgr2=0x7f20a40b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 --2- 172.21.15.28:0/509062737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a40b22e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f20a406d2d0 tx=0x7f209002a300 comp rx=0 tx=0).stop 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 -- 172.21.15.28:0/509062737 shutdown_connections 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 --2- 172.21.15.28:0/509062737 >> v2:172.21.15.189:3300/0 conn(0x7f20a4159820 0x7f20a40b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 --2- 172.21.15.28:0/509062737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a40b22e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 -- 172.21.15.28:0/509062737 >> 172.21.15.28:0/509062737 conn(0x7f20a4072e40 msgr2=0x7f20a4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 -- 172.21.15.28:0/509062737 shutdown_connections 2024-09-13T17:08:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.516+0000 7f20aa600640 1 -- 172.21.15.28:0/509062737 wait complete. 2024-09-13T17:08:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.517+0000 7f20aa600640 1 Processor -- start 2024-09-13T17:08:54.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 -- start start 2024-09-13T17:08:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20a4159440 0x7f20a4147620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4159820 0x7f20a41f2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4147380 con 0x7f20a40b0b50 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4156a10 con 0x7f20a4159820 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.518+0000 7f20aa600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a4156cf0 con 0x7f20a4159440 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48184/0 (socket says 172.21.15.28:48184) 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 -- 172.21.15.28:0/3929671644 learned_addr learned my addr 172.21.15.28:0/3929671644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a3e00640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20a4159440 0x7f20a4147620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9c00640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4159820 0x7f20a41f2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 -- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20a4159440 msgr2=0x7f20a4147620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20a4159440 0x7f20a4147620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 -- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4159820 msgr2=0x7f20a41f2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4159820 0x7f20a41f2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 -- 172.21.15.28:0/3929671644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900094d0 con 0x7f20a40b0b50 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.519+0000 7f20a9200640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f209800c4b0 tx=0x7f209800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.520+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2098005480 con 0x7f20a40b0b50 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.520+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20980035f0 con 0x7f20a40b0b50 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.520+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20980039e0 con 0x7f20a40b0b50 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.520+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a4156fd0 con 0x7f20a40b0b50 2024-09-13T17:08:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.520+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a4157590 con 0x7f20a40b0b50 2024-09-13T17:08:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.521+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a406f270 con 0x7f20a40b0b50 2024-09-13T17:08:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.522+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2098003b40 con 0x7f20a40b0b50 2024-09-13T17:08:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.523+0000 7f20a0c00640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 0x7f208406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.523+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f2098088840 con 0x7f20a40b0b50 2024-09-13T17:08:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.523+0000 7f20a3e00640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 0x7f208406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.524+0000 7f20a3e00640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 0x7f208406d0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2090034000 tx=0x7f209002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.527+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2098053b00 con 0x7f20a40b0b50 2024-09-13T17:08:54.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.706+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20a41577b0 con 0x7f20a40b0b50 2024-09-13T17:08:54.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.707+0000 7f20a0c00640 1 -- 172.21.15.28:0/3929671644 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16725 (secure 0 0 0) 0x7f2098017070 con 0x7f20a40b0b50 2024-09-13T17:08:54.709 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:54.709 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:52.005252+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 msgr2=0x7f208406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 0x7f208406d0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2090034000 tx=0x7f209002e000 comp rx=0 tx=0).stop 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 msgr2=0x7f20a4146e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f209800c4b0 tx=0x7f209800c970 comp rx=0 tx=0).stop 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 shutdown_connections 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f208406ac30 0x7f208406d0f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.709+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20a4159820 0x7f20a41f2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.710+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20a4159440 0x7f20a4147620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.710+0000 7f20aa600640 1 --2- 172.21.15.28:0/3929671644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20a40b0b50 0x7f20a4146e40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.710+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 >> 172.21.15.28:0/3929671644 conn(0x7f20a4072e40 msgr2=0x7f20a40ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:54.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.710+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 shutdown_connections 2024-09-13T17:08:54.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.710+0000 7f20aa600640 1 -- 172.21.15.28:0/3929671644 wait complete. 2024-09-13T17:08:54.727 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:54.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:54.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.885+0000 7fdc08e00640 1 Processor -- start 2024-09-13T17:08:54.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.885+0000 7fdc08e00640 1 -- start start 2024-09-13T17:08:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.886+0000 7fdc08e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc0415a780 0x7fdc04070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc08e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc0415ab60 0x7fdc040712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc08e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc0406bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc08e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0406c8b0 con 0x7fdc0415ab60 2024-09-13T17:08:54.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc08e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0406cb90 con 0x7fdc0415a780 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc08e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040cc980 con 0x7fdc0406bb40 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc0415ab60 0x7fdc040712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc0415ab60 0x7fdc040712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48188/0 (socket says 172.21.15.28:48188) 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 -- 172.21.15.28:0/2720324419 learned_addr learned my addr 172.21.15.28:0/2720324419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01e00640 1 --2- 172.21.15.28:0/2720324419 >> v2:172.21.15.189:3300/0 conn(0x7fdc0415a780 0x7fdc04070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc02800640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc0406bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 -- 172.21.15.28:0/2720324419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 msgr2=0x7fdc0406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc0406bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 -- 172.21.15.28:0/2720324419 >> v2:172.21.15.189:3300/0 conn(0x7fdc0415a780 msgr2=0x7fdc04070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 --2- 172.21.15.28:0/2720324419 >> v2:172.21.15.189:3300/0 conn(0x7fdc0415a780 0x7fdc04070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.887+0000 7fdc01400640 1 -- 172.21.15.28:0/2720324419 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040ccc60 con 0x7fdc0415ab60 2024-09-13T17:08:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.888+0000 7fdc01400640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415ab60 0x7fdc040712c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdc040b1330 tx=0x7fdbf002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=910c7bbf50f1242f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc00a00640 1 -- 172.21.15.28:0/2720324419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0032440 con 0x7fdc0415ab60 2024-09-13T17:08:54.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc00a00640 1 -- 172.21.15.28:0/2720324419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf00325a0 con 0x7fdc0415ab60 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc00a00640 1 -- 172.21.15.28:0/2720324419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf0032930 con 0x7fdc0415ab60 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 -- 172.21.15.28:0/2720324419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415ab60 msgr2=0x7fdc040712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415ab60 0x7fdc040712c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdc040b1330 tx=0x7fdbf002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 -- 172.21.15.28:0/2720324419 shutdown_connections 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415ab60 0x7fdc040712c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 --2- 172.21.15.28:0/2720324419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc0406bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 --2- 172.21.15.28:0/2720324419 >> v2:172.21.15.189:3300/0 conn(0x7fdc0415a780 0x7fdc04070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 -- 172.21.15.28:0/2720324419 >> 172.21.15.28:0/2720324419 conn(0x7fdc040b0270 msgr2=0x7fdc040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 -- 172.21.15.28:0/2720324419 shutdown_connections 2024-09-13T17:08:54.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.889+0000 7fdc08e00640 1 -- 172.21.15.28:0/2720324419 wait complete. 2024-09-13T17:08:54.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.891+0000 7fdc08e00640 1 Processor -- start 2024-09-13T17:08:54.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.891+0000 7fdc08e00640 1 -- start start 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc04158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 0x7fdc040aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc04158a40 con 0x7fdc0415a780 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040a98b0 con 0x7fdc0415ab60 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc08e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040a8f80 con 0x7fdc0406bb40 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48192/0 (socket says 172.21.15.28:48192) 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.892+0000 7fdc01400640 1 -- 172.21.15.28:0/1630867247 learned_addr learned my addr 172.21.15.28:0/1630867247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc02800640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 0x7fdc040aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc04158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 -- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 msgr2=0x7fdc04158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc04158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 -- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 msgr2=0x7fdc040aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 0x7fdc040aa6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 -- 172.21.15.28:0/1630867247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf00094d0 con 0x7fdc0415a780 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc04158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc02800640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 0x7fdc040aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:54.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc01400640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdbf002a220 tx=0x7fdbf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf003b840 con 0x7fdc0415a780 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf003b9a0 con 0x7fdc0415a780 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf003bd10 con 0x7fdc0415a780 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.893+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc040a9260 con 0x7fdc0415a780 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.894+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc040ada10 con 0x7fdc0415a780 2024-09-13T17:08:54.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.895+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc040a9a20 con 0x7fdc0415a780 2024-09-13T17:08:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.895+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdbf0039040 con 0x7fdc0415a780 2024-09-13T17:08:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.896+0000 7fdbfa000640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 0x7fdbe406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:54.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.896+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fdbf00a9d70 con 0x7fdc0415a780 2024-09-13T17:08:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.897+0000 7fdc01e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 0x7fdbe406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:54.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.897+0000 7fdc01e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 0x7fdbe406d0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009b10 tx=0x7fdbf4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:54.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:54.901+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdbf0074ee0 con 0x7fdc0415a780 2024-09-13T17:08:55.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.080+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc0406bf70 con 0x7fdc0415a780 2024-09-13T17:08:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.080+0000 7fdbfa000640 1 -- 172.21.15.28:0/1630867247 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdbf0078500 con 0x7fdc0415a780 2024-09-13T17:08:55.081 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 msgr2=0x7fdbe406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 0x7fdbe406d0c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009b10 tx=0x7fdbf4008040 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 msgr2=0x7fdc0414a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fdbf002a220 tx=0x7fdbf00046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 shutdown_connections 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe406ac00 0x7fdbe406d0c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0415ab60 0x7fdc040aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0415a780 0x7fdc0414a0a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 --2- 172.21.15.28:0/1630867247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0406bb40 0x7fdc04158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 >> 172.21.15.28:0/1630867247 conn(0x7fdc040b0270 msgr2=0x7fdc040cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.082+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 shutdown_connections 2024-09-13T17:08:55.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.083+0000 7fdc08e00640 1 -- 172.21.15.28:0/1630867247 wait complete. 2024-09-13T17:08:55.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:55.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.258+0000 7f066c600640 1 Processor -- start 2024-09-13T17:08:55.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.258+0000 7f066c600640 1 -- start start 2024-09-13T17:08:55.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.259+0000 7f066c600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f066415a780 0x7f0664070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.259+0000 7f066c600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066406c660 con 0x7f066415ab60 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f066c600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f066406c8b0 con 0x7f066406c250 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f066c600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f066406cb90 con 0x7f066415a780 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0669200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f066415ab60 0x7f066406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37812/0 (socket says 172.21.15.28:37812) 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0669200640 1 -- 172.21.15.28:0/4277209323 learned_addr learned my addr 172.21.15.28:0/4277209323 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0669c00640 1 --2- 172.21.15.28:0/4277209323 >> v2:172.21.15.189:3300/0 conn(0x7f066415a780 0x7f0664070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0663e00640 1 -- 172.21.15.28:0/4277209323 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313396448 0 0) 0x7f066406c660 con 0x7f066415ab60 2024-09-13T17:08:55.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0663e00640 1 -- 172.21.15.28:0/4277209323 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f064c003520 con 0x7f066415ab60 2024-09-13T17:08:55.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.260+0000 7f0669c00640 1 -- 172.21.15.28:0/4277209323 >> v1:172.21.15.189:6789/0 conn(0x7f066406c250 legacy=0x7f06640cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.261+0000 7f0669c00640 1 -- 172.21.15.28:0/4277209323 >> v1:172.21.15.28:6789/0 conn(0x7f066415ab60 legacy=0x7f066406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.261+0000 7f0669c00640 1 -- 172.21.15.28:0/4277209323 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06640cc980 con 0x7f066415a780 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.262+0000 7f0669c00640 1 --2- 172.21.15.28:0/4277209323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066415a780 0x7f0664070d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f06580097a0 tx=0x7f065802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa6c434349bacf05 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.262+0000 7f0663e00640 1 -- 172.21.15.28:0/4277209323 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658032440 con 0x7f066415a780 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.262+0000 7f0663e00640 1 -- 172.21.15.28:0/4277209323 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06580325a0 con 0x7f066415a780 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.262+0000 7f0663e00640 1 -- 172.21.15.28:0/4277209323 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658032930 con 0x7f066415a780 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 -- 172.21.15.28:0/4277209323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066415a780 msgr2=0x7f0664070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 --2- 172.21.15.28:0/4277209323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066415a780 0x7f0664070d80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f06580097a0 tx=0x7f065802a230 comp rx=0 tx=0).stop 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 -- 172.21.15.28:0/4277209323 shutdown_connections 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 --2- 172.21.15.28:0/4277209323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066415a780 0x7f0664070d80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 -- 172.21.15.28:0/4277209323 >> 172.21.15.28:0/4277209323 conn(0x7f06640b0270 msgr2=0x7f06640b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 -- 172.21.15.28:0/4277209323 shutdown_connections 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.263+0000 7f066c600640 1 -- 172.21.15.28:0/4277209323 wait complete. 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.265+0000 7f066c600640 1 Processor -- start 2024-09-13T17:08:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.265+0000 7f066c600640 1 -- start start 2024-09-13T17:08:55.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 0x7f0664150050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f066415ab60 0x7f06641f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0664150590 con 0x7f066415a780 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0664150950 con 0x7f066406c250 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066c600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0664150c30 con 0x7f066415ab60 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f0669200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f0669200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48208/0 (socket says 172.21.15.28:48208) 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f0669200640 1 -- 172.21.15.28:0/3245556017 learned_addr learned my addr 172.21.15.28:0/3245556017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f0669c00640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 0x7f0664150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.266+0000 7f066a600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f066415ab60 0x7f06641f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 -- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f066415ab60 msgr2=0x7f06641f5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f066415ab60 0x7f06641f5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 -- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 msgr2=0x7f0664150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 0x7f0664150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 -- 172.21.15.28:0/3245556017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0658009400 con 0x7f066415a780 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669c00640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 0x7f0664150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0669200640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f065000c410 tx=0x7f065000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0650005ce0 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0650005e40 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06500072f0 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0664150f10 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.267+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06641514d0 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.269+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0664071680 con 0x7f066415a780 2024-09-13T17:08:55.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.270+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0650007510 con 0x7f066415a780 2024-09-13T17:08:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.270+0000 7f0661600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 0x7f064406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.270+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f0650088a10 con 0x7f066415a780 2024-09-13T17:08:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.271+0000 7f0669c00640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 0x7f064406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.272+0000 7f0669c00640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 0x7f064406d0c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0658034000 tx=0x7f065802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.275+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0650053c60 con 0x7f066415a780 2024-09-13T17:08:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.452+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0664067070 con 0x7f066415a780 2024-09-13T17:08:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.452+0000 7f0661600640 1 -- 172.21.15.28:0/3245556017 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0650017020 con 0x7f066415a780 2024-09-13T17:08:55.453 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:55.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 msgr2=0x7f064406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 0x7f064406d0c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0658034000 tx=0x7f065802e000 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 msgr2=0x7f06640ced40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f065000c410 tx=0x7f065000c8d0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 shutdown_connections 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f064406ac00 0x7f064406d0c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f066415ab60 0x7f06641f5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f066415a780 0x7f06640ced40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 --2- 172.21.15.28:0/3245556017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f066406c250 0x7f0664150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 >> 172.21.15.28:0/3245556017 conn(0x7f06640b0270 msgr2=0x7f06640ae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.454+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 shutdown_connections 2024-09-13T17:08:55.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.455+0000 7f066c600640 1 -- 172.21.15.28:0/3245556017 wait complete. 2024-09-13T17:08:55.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:55.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.628+0000 7f8972a00640 1 Processor -- start 2024-09-13T17:08:55.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.628+0000 7f8972a00640 1 -- start start 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8972a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f896c15ab60 0x7f896c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8972a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 0x7f896c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8972a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f896c06c8b0 con 0x7f896c15a780 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8972a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f896c06cb90 con 0x7f896c15ab60 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8972a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c0cc980 con 0x7f896c071490 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f896c15ab60 0x7f896c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f896c15ab60 0x7f896c06bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48214/0 (socket says 172.21.15.28:48214) 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 -- 172.21.15.28:0/1874254922 learned_addr learned my addr 172.21.15.28:0/1874254922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8970a00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 0x7f896c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 -- 172.21.15.28:0/1874254922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 msgr2=0x7f896c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:55.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 0x7f896c06c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.630+0000 7f8963e00640 1 -- 172.21.15.28:0/1874254922 >> v1:172.21.15.189:6789/0 conn(0x7f896c15a780 legacy=0x7f896c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.631+0000 7f8963e00640 1 -- 172.21.15.28:0/1874254922 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c0ccc60 con 0x7f896c15ab60 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.631+0000 7f8970a00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 0x7f896c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.631+0000 7f8963e00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c15ab60 0x7f896c06bb40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8954009870 tx=0x7f895402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8293010ef8a1d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f896b400640 1 -- 172.21.15.28:0/1874254922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954032440 con 0x7f896c15ab60 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f896b400640 1 -- 172.21.15.28:0/1874254922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540325a0 con 0x7f896c15ab60 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f896b400640 1 -- 172.21.15.28:0/1874254922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954032930 con 0x7f896c15ab60 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 -- 172.21.15.28:0/1874254922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c15ab60 msgr2=0x7f896c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c15ab60 0x7f896c06bb40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f8954009870 tx=0x7f895402a250 comp rx=0 tx=0).stop 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 -- 172.21.15.28:0/1874254922 shutdown_connections 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c071490 0x7f896c06c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 --2- 172.21.15.28:0/1874254922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c15ab60 0x7f896c06bb40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 -- 172.21.15.28:0/1874254922 >> 172.21.15.28:0/1874254922 conn(0x7f896c0b0270 msgr2=0x7f896c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:55.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.632+0000 7f8972a00640 1 -- 172.21.15.28:0/1874254922 shutdown_connections 2024-09-13T17:08:55.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.633+0000 7f8972a00640 1 -- 172.21.15.28:0/1874254922 wait complete. 2024-09-13T17:08:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.634+0000 7f8972a00640 1 Processor -- start 2024-09-13T17:08:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.634+0000 7f8972a00640 1 -- start start 2024-09-13T17:08:55.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c15a780 0x7f896c153f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 0x7f896c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c150190 con 0x7f896c071490 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896c151810 con 0x7f896c15ab60 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f8972a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896c151530 con 0x7f896c15a780 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f896be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f896be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48218/0 (socket says 172.21.15.28:48218) 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.635+0000 7f896be00640 1 -- 172.21.15.28:0/1936554210 learned_addr learned my addr 172.21.15.28:0/1936554210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8970a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 0x7f896c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8963e00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c15a780 0x7f896c153f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 -- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c15a780 msgr2=0x7f896c153f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c15a780 0x7f896c153f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 -- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 msgr2=0x7f896c1544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 0x7f896c1544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 -- 172.21.15.28:0/1936554210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89540094d0 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8970a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 0x7f896c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f896be00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f895c00c480 tx=0x7f895c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c0052d0 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c005430 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c012870 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896c14fb40 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.636+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f896c0cb0c0 con 0x7f896c071490 2024-09-13T17:08:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.637+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896c14f9a0 con 0x7f896c071490 2024-09-13T17:08:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.638+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f895c0129d0 con 0x7f896c071490 2024-09-13T17:08:55.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.640+0000 7f8968c00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 0x7f894806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.640+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f895c088f40 con 0x7f896c071490 2024-09-13T17:08:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.640+0000 7f8963e00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 0x7f894806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:55.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.641+0000 7f8963e00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 0x7f894806d0f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8954034000 tx=0x7f895402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.644+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f896c14f9a0 con 0x7f896c071490 2024-09-13T17:08:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.825+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f896c14f9a0 con 0x7f896c071490 2024-09-13T17:08:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.825+0000 7f8968c00640 1 -- 172.21.15.28:0/1936554210 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f896c14f9a0 con 0x7f896c071490 2024-09-13T17:08:55.826 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:55.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 msgr2=0x7f894806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 0x7f894806d0f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8954034000 tx=0x7f895402e000 comp rx=0 tx=0).stop 2024-09-13T17:08:55.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 msgr2=0x7f896c0cab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f895c00c480 tx=0x7f895c00c940 comp rx=0 tx=0).stop 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 shutdown_connections 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f894806ac30 0x7f894806d0f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c15ab60 0x7f896c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f896c15a780 0x7f896c153f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 --2- 172.21.15.28:0/1936554210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f896c071490 0x7f896c0cab80 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.827+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 >> 172.21.15.28:0/1936554210 conn(0x7f896c0b0270 msgr2=0x7f896c0ce240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.828+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 shutdown_connections 2024-09-13T17:08:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:55.828+0000 7f8972a00640 1 -- 172.21.15.28:0/1936554210 wait complete. 2024-09-13T17:08:55.846 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:08:56.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.002+0000 7f869be00640 1 Processor -- start 2024-09-13T17:08:56.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.002+0000 7f869be00640 1 -- start start 2024-09-13T17:08:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.003+0000 7f869be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f869c0cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.003+0000 7f869be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f869c06bba0 0x7f869c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.003+0000 7f869be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f869c06fe50 con 0x7f869c0cdc20 2024-09-13T17:08:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.003+0000 7f869be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f869c06cc50 con 0x7f869c06bba0 2024-09-13T17:08:56.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.003+0000 7f869be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15a8c0 con 0x7f869c15a4e0 2024-09-13T17:08:56.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f869c0cdc20 0x7f869c15e080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37814/0 (socket says 172.21.15.28:37814) 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869b400640 1 -- 172.21.15.28:0/3681604727 learned_addr learned my addr 172.21.15.28:0/3681604727 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f8692000640 1 --2- 172.21.15.28:0/3681604727 >> v2:172.21.15.189:3300/0 conn(0x7f869c06bba0 0x7f869c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869aa00640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f869c0cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701006105 0 0) 0x7f869c06fe50 con 0x7f869c0cdc20 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8680003520 con 0x7f869c0cdc20 2024-09-13T17:08:56.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.004+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838148703 0 0) 0x7f8680003520 con 0x7f869c0cdc20 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 msgr2=0x7f869c0cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f869c0cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 >> v2:172.21.15.189:3300/0 conn(0x7f869c06bba0 msgr2=0x7f869c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 --2- 172.21.15.28:0/3681604727 >> v2:172.21.15.189:3300/0 conn(0x7f869c06bba0 0x7f869c0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c06c3b0 con 0x7f869c0cdc20 2024-09-13T17:08:56.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f86940019f0 con 0x7f869c0cdc20 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 >> v1:172.21.15.28:6789/0 conn(0x7f869c0cdc20 legacy=0x7f869c15e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8680003970 0x7f8680003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86800047c0 0x7f8680004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f8680005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c06cc50 con 0x7f86800047c0 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c15a8c0 con 0x7f8680003970 2024-09-13T17:08:56.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868000fd90 con 0x7f869c15a4e0 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86800047c0 0x7f8680004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869aa00640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8680003970 0x7f8680003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f869b400640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f8680005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 -- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 msgr2=0x7f8680005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f869c15a4e0 0x7f8680005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 -- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8680003970 msgr2=0x7f8680003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8680003970 0x7f8680003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.005+0000 7f8692000640 1 -- 172.21.15.28:0/3681604727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8680003520 con 0x7f86800047c0 2024-09-13T17:08:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.006+0000 7f8692000640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86800047c0 0x7f8680004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8684009a90 tx=0x7f86800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297dba36d4a2651c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8684031400 con 0x7f86800047c0 2024-09-13T17:08:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868402d050 con 0x7f86800047c0 2024-09-13T17:08:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869a000640 1 -- 172.21.15.28:0/3681604727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86840317a0 con 0x7f86800047c0 2024-09-13T17:08:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 -- 172.21.15.28:0/3681604727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86800047c0 msgr2=0x7f8680004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 --2- 172.21.15.28:0/3681604727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86800047c0 0x7f8680004c40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f8684009a90 tx=0x7f86800057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869b400640 1 -- 172.21.15.28:0/3681604727 reap_dead start 2024-09-13T17:08:56.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 -- 172.21.15.28:0/3681604727 shutdown_connections 2024-09-13T17:08:56.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 -- 172.21.15.28:0/3681604727 >> 172.21.15.28:0/3681604727 conn(0x7f869c069e80 msgr2=0x7f869c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:56.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 -- 172.21.15.28:0/3681604727 shutdown_connections 2024-09-13T17:08:56.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.007+0000 7f869be00640 1 -- 172.21.15.28:0/3681604727 wait complete. 2024-09-13T17:08:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.009+0000 7f869be00640 1 Processor -- start 2024-09-13T17:08:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.009+0000 7f869be00640 1 -- start start 2024-09-13T17:08:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 0x7f869c146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 0x7f869c0ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0a9240 con 0x7f869c1469a0 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0a9520 con 0x7f869c146e20 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f869be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869c0a9800 con 0x7f86800047c0 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f8692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f8692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48238/0 (socket says 172.21.15.28:48238) 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.010+0000 7f8692000640 1 -- 172.21.15.28:0/3495961219 learned_addr learned my addr 172.21.15.28:0/3495961219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f869aa00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 0x7f869c146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f869b400640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 0x7f869c0ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 -- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 msgr2=0x7f869c146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 0x7f869c146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 -- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 msgr2=0x7f869c0ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 0x7f869c0ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 -- 172.21.15.28:0/3495961219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c0a9ae0 con 0x7f869c1469a0 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f869aa00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 0x7f869c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f869b400640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 0x7f869c0ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.011+0000 7f8692000640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8684034040 tx=0x7f8684004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.012+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868403a600 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.012+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868403a760 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.012+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868403aaf0 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.012+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f869c0a9dc0 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.012+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f869c0a8cc0 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.013+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869c0cd6e0 con 0x7f869c1469a0 2024-09-13T17:08:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.014+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8684038040 con 0x7f869c1469a0 2024-09-13T17:08:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.015+0000 7f8693400640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 0x7f867806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.015+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f86840a9850 con 0x7f869c1469a0 2024-09-13T17:08:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.015+0000 7f869aa00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 0x7f867806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.016+0000 7f869aa00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 0x7f867806d0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f868c009ae0 tx=0x7f868c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.019+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f86840749e0 con 0x7f869c1469a0 2024-09-13T17:08:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.212+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f869c071310 con 0x7f869c1469a0 2024-09-13T17:08:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.213+0000 7f8693400640 1 -- 172.21.15.28:0/3495961219 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f868403ae60 con 0x7f869c1469a0 2024-09-13T17:08:56.214 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:08:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 msgr2=0x7f867806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 0x7f867806d0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f868c009ae0 tx=0x7f868c008040 comp rx=0 tx=0).stop 2024-09-13T17:08:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 msgr2=0x7f869c0a8780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8684034040 tx=0x7f8684004730 comp rx=0 tx=0).stop 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 shutdown_connections 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f867806ac30 0x7f867806d0f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f869c146e20 0x7f869c0ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f869c1469a0 0x7f869c0a8780 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 --2- 172.21.15.28:0/3495961219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86800047c0 0x7f869c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 >> 172.21.15.28:0/3495961219 conn(0x7f869c069e80 msgr2=0x7f869c15aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 shutdown_connections 2024-09-13T17:08:56.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.215+0000 7f869be00640 1 -- 172.21.15.28:0/3495961219 wait complete. 2024-09-13T17:08:56.232 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-09-13T17:08:56.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:56.232 DEBUG:teuthology.orchestra.run.smithi189:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-09-13T17:08:56.299 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:08:56.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:08:56.299 DEBUG:teuthology.orchestra.run.smithi189:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T17:08:56.336 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:08:56.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.491+0000 7f4549c00640 1 Processor -- start 2024-09-13T17:08:56.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.491+0000 7f4549c00640 1 -- start start 2024-09-13T17:08:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4549c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4544154a90 0x7f454415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4549c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45440b0b50 0x7f45440af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4549c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454415bb10 con 0x7f454406b360 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4549c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454406b900 con 0x7f4544154a90 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4549c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454406bbe0 con 0x7f45440b0b50 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4543e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4544154a90 0x7f454415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4543e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4544154a90 0x7f454415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48244/0 (socket says 172.21.15.28:48244) 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f4543e00640 1 -- 172.21.15.28:0/1390296239 learned_addr learned my addr 172.21.15.28:0/1390296239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:56.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.492+0000 7f453b400640 1 --1- 172.21.15.28:0/1390296239 >> v1:172.21.15.28:6789/0 conn(0x7f454406b360 0x7f45440b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37830/0 (socket says 172.21.15.28:37830) 2024-09-13T17:08:56.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.493+0000 7f4549200640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45440b0b50 0x7f45440af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.493+0000 7f4543e00640 1 -- 172.21.15.28:0/1390296239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45440b0b50 msgr2=0x7f45440af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.493+0000 7f4543e00640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45440b0b50 0x7f45440af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.493+0000 7f4543e00640 1 -- 172.21.15.28:0/1390296239 >> v1:172.21.15.28:6789/0 conn(0x7f454406b360 legacy=0x7f45440b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.493+0000 7f4543e00640 1 -- 172.21.15.28:0/1390296239 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454406bec0 con 0x7f4544154a90 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.494+0000 7f4543e00640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4544154a90 0x7f454415b5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4502361814533852 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.494+0000 7f4543400640 1 -- 172.21.15.28:0/1390296239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534032440 con 0x7f4544154a90 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.494+0000 7f4543400640 1 -- 172.21.15.28:0/1390296239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45340325a0 con 0x7f4544154a90 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.494+0000 7f4543400640 1 -- 172.21.15.28:0/1390296239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534032930 con 0x7f4544154a90 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 -- 172.21.15.28:0/1390296239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4544154a90 msgr2=0x7f454415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4544154a90 0x7f454415b5d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4534009870 tx=0x7f453402a250 comp rx=0 tx=0).stop 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 -- 172.21.15.28:0/1390296239 shutdown_connections 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45440b0b50 0x7f45440af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 --2- 172.21.15.28:0/1390296239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4544154a90 0x7f454415b5d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 -- 172.21.15.28:0/1390296239 >> 172.21.15.28:0/1390296239 conn(0x7f4544072e40 msgr2=0x7f4544073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 -- 172.21.15.28:0/1390296239 shutdown_connections 2024-09-13T17:08:56.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.495+0000 7f4549c00640 1 -- 172.21.15.28:0/1390296239 wait complete. 2024-09-13T17:08:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.496+0000 7f4549c00640 1 Processor -- start 2024-09-13T17:08:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 -- start start 2024-09-13T17:08:56.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45440b0b50 0x7f45441f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4544154a90 0x7f45440ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441f5240 con 0x7f454406b360 2024-09-13T17:08:56.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441f5490 con 0x7f45440b0b50 2024-09-13T17:08:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.497+0000 7f4549c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441f5770 con 0x7f4544154a90 2024-09-13T17:08:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48252/0 (socket says 172.21.15.28:48252) 2024-09-13T17:08:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 -- 172.21.15.28:0/702013842 learned_addr learned my addr 172.21.15.28:0/702013842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:56.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4549200640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4544154a90 0x7f45440ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f453b400640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45440b0b50 0x7f45441f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 -- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4544154a90 msgr2=0x7f45440ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4544154a90 0x7f45440ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 -- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45440b0b50 msgr2=0x7f45441f1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45440b0b50 0x7f45441f1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.498+0000 7f4543e00640 1 -- 172.21.15.28:0/702013842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340094d0 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4543e00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4534000c00 tx=0x7f45340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534032440 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4534030980 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534030d10 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45441f5a50 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.499+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45441f6010 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.500+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45440af4b0 con 0x7f454406b360 2024-09-13T17:08:56.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.501+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4534039040 con 0x7f454406b360 2024-09-13T17:08:56.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.502+0000 7f4540c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 0x7f452006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.502+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f45340a9be0 con 0x7f454406b360 2024-09-13T17:08:56.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.502+0000 7f453b400640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 0x7f452006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.503+0000 7f453b400640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 0x7f452006d0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f454406c970 tx=0x7f452c007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.506+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4534074e00 con 0x7f454406b360 2024-09-13T17:08:56.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.722+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4544157f40 con 0x7f454406b360 2024-09-13T17:08:56.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.722+0000 7f4540c00640 1 -- 172.21.15.28:0/702013842 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f4534078420 con 0x7f454406b360 2024-09-13T17:08:56.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:56.724 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:56.724 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 msgr2=0x7f452006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 0x7f452006d0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f454406c970 tx=0x7f452c007e00 comp rx=0 tx=0).stop 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 msgr2=0x7f45440ae850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4534000c00 tx=0x7f45340046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 shutdown_connections 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452006ac00 0x7f452006d0c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4544154a90 0x7f45440ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45440b0b50 0x7f45441f1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 --2- 172.21.15.28:0/702013842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f454406b360 0x7f45440ae850 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 >> 172.21.15.28:0/702013842 conn(0x7f4544072e40 msgr2=0x7f45440b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 shutdown_connections 2024-09-13T17:08:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.725+0000 7f4549c00640 1 -- 172.21.15.28:0/702013842 wait complete. 2024-09-13T17:08:56.741 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.897+0000 7fbf48800640 1 Processor -- start 2024-09-13T17:08:56.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.897+0000 7fbf48800640 1 -- start start 2024-09-13T17:08:56.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf48800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf4015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf48800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf4006c2d0 0x7fbf4006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf48800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf40159d90 con 0x7fbf401592a0 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf48800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4015b0b0 con 0x7fbf4006c2d0 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf48800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4006cca0 con 0x7fbf4006f970 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf45e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbf401592a0 0x7fbf40159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37832/0 (socket says 172.21.15.28:37832) 2024-09-13T17:08:56.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf45e00640 1 -- 172.21.15.28:0/514791688 learned_addr learned my addr 172.21.15.28:0/514791688 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf45400640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf4015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf46800640 1 --2- 172.21.15.28:0/514791688 >> v2:172.21.15.189:3300/0 conn(0x7fbf4006c2d0 0x7fbf4006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624989841 0 0) 0x7fbf40159d90 con 0x7fbf401592a0 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.899+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf28005730 con 0x7fbf401592a0 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332677288 0 0) 0x7fbf28005730 con 0x7fbf401592a0 2024-09-13T17:08:56.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 msgr2=0x7fbf4015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf4015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 >> v2:172.21.15.189:3300/0 conn(0x7fbf4006c2d0 msgr2=0x7fbf4006c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 --2- 172.21.15.28:0/514791688 >> v2:172.21.15.189:3300/0 conn(0x7fbf4006c2d0 0x7fbf4006c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf400b2040 con 0x7fbf401592a0 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fbf34001690 con 0x7fbf401592a0 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 >> v1:172.21.15.28:6789/0 conn(0x7fbf401592a0 legacy=0x7fbf40159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf28005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf280060c0 0x7fbf280069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf28006540 0x7fbf28006f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4015b0b0 con 0x7fbf280060c0 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4006cca0 con 0x7fbf28006540 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280077d0 con 0x7fbf4006f970 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf45400640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf280060c0 0x7fbf280069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf45e00640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.900+0000 7fbf46800640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf28006540 0x7fbf28006f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.901+0000 7fbf45400640 1 -- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 msgr2=0x7fbf28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.901+0000 7fbf45400640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf4006f970 0x7fbf28005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.901+0000 7fbf45400640 1 -- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf28006540 msgr2=0x7fbf28006f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.901+0000 7fbf45400640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf28006540 0x7fbf28006f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.901+0000 7fbf45400640 1 -- 172.21.15.28:0/514791688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28005730 con 0x7fbf280060c0 2024-09-13T17:08:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf45400640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf280060c0 0x7fbf280069d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009ac0 tx=0x7fbf28007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8168df36128ecd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30031400 con 0x7fbf280060c0 2024-09-13T17:08:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf3002d050 con 0x7fbf280060c0 2024-09-13T17:08:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf44a00640 1 -- 172.21.15.28:0/514791688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30031780 con 0x7fbf280060c0 2024-09-13T17:08:56.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf48800640 1 -- 172.21.15.28:0/514791688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf280060c0 msgr2=0x7fbf280069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf48800640 1 --2- 172.21.15.28:0/514791688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf280060c0 0x7fbf280069d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fbf30009ac0 tx=0x7fbf28007a00 comp rx=0 tx=0).stop 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.902+0000 7fbf46800640 1 -- 172.21.15.28:0/514791688 reap_dead start 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.903+0000 7fbf48800640 1 -- 172.21.15.28:0/514791688 shutdown_connections 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.903+0000 7fbf48800640 1 -- 172.21.15.28:0/514791688 >> 172.21.15.28:0/514791688 conn(0x7fbf40069e80 msgr2=0x7fbf4006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.903+0000 7fbf48800640 1 -- 172.21.15.28:0/514791688 shutdown_connections 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.903+0000 7fbf48800640 1 -- 172.21.15.28:0/514791688 wait complete. 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.904+0000 7fbf48800640 1 Processor -- start 2024-09-13T17:08:56.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.904+0000 7fbf48800640 1 -- start start 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf280060c0 0x7fbf40147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf40146340 0x7fbf400adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401478c0 con 0x7fbf40149030 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401460d0 con 0x7fbf280060c0 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.905+0000 7fbf48800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf40146c40 con 0x7fbf40146340 2024-09-13T17:08:56.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48284/0 (socket says 172.21.15.28:48284) 2024-09-13T17:08:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 -- 172.21.15.28:0/3213791776 learned_addr learned my addr 172.21.15.28:0/3213791776 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45e00640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf280060c0 0x7fbf40147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf46800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf40146340 0x7fbf400adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 -- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf40146340 msgr2=0x7fbf400adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf40146340 0x7fbf400adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 -- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf280060c0 msgr2=0x7fbf40147ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf280060c0 0x7fbf40147ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 -- 172.21.15.28:0/3213791776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40146f20 con 0x7fbf40149030 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.906+0000 7fbf45400640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbf3002aae0 tx=0x7fbf30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.907+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30003c90 con 0x7fbf40149030 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.907+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf30003df0 con 0x7fbf40149030 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.907+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf40147200 con 0x7fbf40149030 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.907+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30044650 con 0x7fbf40149030 2024-09-13T17:08:56.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.907+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf401467c0 con 0x7fbf40149030 2024-09-13T17:08:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.908+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf40147a00 con 0x7fbf40149030 2024-09-13T17:08:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.909+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf30038070 con 0x7fbf40149030 2024-09-13T17:08:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.910+0000 7fbf3a000640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 0x7fbf2406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.910+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fbf300a98a0 con 0x7fbf40149030 2024-09-13T17:08:56.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.910+0000 7fbf45e00640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 0x7fbf2406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.911+0000 7fbf45e00640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 0x7fbf2406d0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf34001c10 tx=0x7fbf34001df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:56.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:56.914+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf40147a00 con 0x7fbf40149030 2024-09-13T17:08:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.097+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf40147a00 con 0x7fbf40149030 2024-09-13T17:08:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.098+0000 7fbf3a000640 1 -- 172.21.15.28:0/3213791776 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16725 (secure 0 0 0) 0x7fbf40147a00 con 0x7fbf40149030 2024-09-13T17:08:57.099 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:57.099 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:52.005252+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 msgr2=0x7fbf2406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 0x7fbf2406d0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbf34001c10 tx=0x7fbf34001df0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 msgr2=0x7fbf40145b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbf3002aae0 tx=0x7fbf30004730 comp rx=0 tx=0).stop 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 shutdown_connections 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac30 0x7fbf2406d0f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf40146340 0x7fbf400adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf40149030 0x7fbf40145b90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 --2- 172.21.15.28:0/3213791776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf280060c0 0x7fbf40147ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 >> 172.21.15.28:0/3213791776 conn(0x7fbf40069e80 msgr2=0x7fbf4006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 shutdown_connections 2024-09-13T17:08:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.100+0000 7fbf48800640 1 -- 172.21.15.28:0/3213791776 wait complete. 2024-09-13T17:08:57.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:08:09.855355+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:08:10.527313+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:08:57.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:08:57.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.276+0000 7f22c5600640 1 Processor -- start 2024-09-13T17:08:57.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.276+0000 7f22c5600640 1 -- start start 2024-09-13T17:08:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22c5600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f22c015c9a0 0x7f22c00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22c5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f22c00b2400 0x7f22c006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22c5600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0070470 con 0x7f22c00ad940 2024-09-13T17:08:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22c5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c00ace30 con 0x7f22c00b2400 2024-09-13T17:08:57.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22c5600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c00ad110 con 0x7f22c015c9a0 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22bf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f22c00b2400 0x7f22c006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22b7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f22c00ad940 0x7f22c00b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37840/0 (socket says 172.21.15.28:37840) 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.277+0000 7f22b7e00640 1 -- 172.21.15.28:0/3875532684 learned_addr learned my addr 172.21.15.28:0/3875532684 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.278+0000 7f22bea00640 1 --2- 172.21.15.28:0/3875532684 >> v2:172.21.15.189:3300/0 conn(0x7f22c015c9a0 0x7f22c00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.278+0000 7f22bf400640 1 -- 172.21.15.28:0/3875532684 >> v2:172.21.15.189:3300/0 conn(0x7f22c015c9a0 msgr2=0x7f22c00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.278+0000 7f22bf400640 1 --2- 172.21.15.28:0/3875532684 >> v2:172.21.15.189:3300/0 conn(0x7f22c015c9a0 0x7f22c00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.278+0000 7f22bf400640 1 -- 172.21.15.28:0/3875532684 >> v1:172.21.15.28:6789/0 conn(0x7f22c00ad940 legacy=0x7f22c00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.278+0000 7f22bf400640 1 -- 172.21.15.28:0/3875532684 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c006b170 con 0x7f22c00b2400 2024-09-13T17:08:57.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.279+0000 7f22bf400640 1 --2- 172.21.15.28:0/3875532684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c006bc50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009d90 tx=0x7f22b802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab399878f7aebacd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.279+0000 7f22be000640 1 -- 172.21.15.28:0/3875532684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8007740 con 0x7f22c00b2400 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.279+0000 7f22be000640 1 -- 172.21.15.28:0/3875532684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b802d050 con 0x7f22c00b2400 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.279+0000 7f22be000640 1 -- 172.21.15.28:0/3875532684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8007ac0 con 0x7f22c00b2400 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 -- 172.21.15.28:0/3875532684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 msgr2=0x7f22c006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 --2- 172.21.15.28:0/3875532684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c006bc50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009d90 tx=0x7f22b802a730 comp rx=0 tx=0).stop 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 -- 172.21.15.28:0/3875532684 shutdown_connections 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 --2- 172.21.15.28:0/3875532684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c006bc50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 --2- 172.21.15.28:0/3875532684 >> v2:172.21.15.189:3300/0 conn(0x7f22c015c9a0 0x7f22c00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 -- 172.21.15.28:0/3875532684 >> 172.21.15.28:0/3875532684 conn(0x7f22c00b0270 msgr2=0x7f22c00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 -- 172.21.15.28:0/3875532684 shutdown_connections 2024-09-13T17:08:57.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.280+0000 7f22c5600640 1 -- 172.21.15.28:0/3875532684 wait complete. 2024-09-13T17:08:57.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.282+0000 7f22c5600640 1 Processor -- start 2024-09-13T17:08:57.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.282+0000 7f22c5600640 1 -- start start 2024-09-13T17:08:57.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f22c00ad940 0x7f22c01f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c015c9a0 0x7f22c0153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c014f780 con 0x7f22c00b2400 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c01f6b90 con 0x7f22c015c9a0 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22c5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c0152f50 con 0x7f22c00ad940 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48304/0 (socket says 172.21.15.28:48304) 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 -- 172.21.15.28:0/604552608 learned_addr learned my addr 172.21.15.28:0/604552608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22bea00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f22c00ad940 0x7f22c01f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22bf400640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c015c9a0 0x7f22c0153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 -- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f22c00ad940 msgr2=0x7f22c01f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f22c00ad940 0x7f22c01f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 -- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c015c9a0 msgr2=0x7f22c0153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c015c9a0 0x7f22c0153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.283+0000 7f22b7e00640 1 -- 172.21.15.28:0/604552608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b8009a70 con 0x7f22c00b2400 2024-09-13T17:08:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22b7e00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f22a800c400 tx=0x7f22a800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a8005ce0 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a8005e40 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a80072f0 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c0070930 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.284+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c0153510 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.285+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c015cd80 con 0x7f22c00b2400 2024-09-13T17:08:57.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.286+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f22a8007510 con 0x7f22c00b2400 2024-09-13T17:08:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.287+0000 7f22b6a00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 0x7f229c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.287+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f22a80889e0 con 0x7f22c00b2400 2024-09-13T17:08:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.288+0000 7f22bea00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 0x7f229c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.288+0000 7f22bea00640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 0x7f229c06d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f22b00079a0 tx=0x7f22b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.291+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f22a8053b70 con 0x7f22c00b2400 2024-09-13T17:08:57.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.466+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f22c00ae4b0 con 0x7f22c00b2400 2024-09-13T17:08:57.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.467+0000 7f22b6a00640 1 -- 172.21.15.28:0/604552608 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22a8017020 con 0x7f22c00b2400 2024-09-13T17:08:57.468 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 msgr2=0x7f229c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 0x7f229c06d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f22b00079a0 tx=0x7f22b0008040 comp rx=0 tx=0).stop 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 msgr2=0x7f22c0152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f22a800c400 tx=0x7f22a800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 shutdown_connections 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f229c06ac00 0x7f229c06d0c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22c015c9a0 0x7f22c0153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22c00b2400 0x7f22c0152460 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 --2- 172.21.15.28:0/604552608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f22c00ad940 0x7f22c01f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 >> 172.21.15.28:0/604552608 conn(0x7f22c00b0270 msgr2=0x7f22c0159e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:57.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 shutdown_connections 2024-09-13T17:08:57.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.469+0000 7f22c5600640 1 -- 172.21.15.28:0/604552608 wait complete. 2024-09-13T17:08:57.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:08:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.642+0000 7feb54800640 1 Processor -- start 2024-09-13T17:08:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.642+0000 7feb54800640 1 -- start start 2024-09-13T17:08:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.643+0000 7feb54800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb4c0b2400 0x7feb4c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.643+0000 7feb54800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c070470 con 0x7feb4c0ad940 2024-09-13T17:08:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.643+0000 7feb54800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4c0ace30 con 0x7feb4c15c9a0 2024-09-13T17:08:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.643+0000 7feb54800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4c0ad110 con 0x7feb4c0b2400 2024-09-13T17:08:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.643+0000 7feb51400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb4c0b2400 0x7feb4c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.644+0000 7feb51400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb4c0b2400 0x7feb4c0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48312/0 (socket says 172.21.15.28:48312) 2024-09-13T17:08:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.644+0000 7feb51400640 1 -- 172.21.15.28:0/4008749477 learned_addr learned my addr 172.21.15.28:0/4008749477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:57.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.644+0000 7feb51400640 1 -- 172.21.15.28:0/4008749477 >> v1:172.21.15.189:6789/0 conn(0x7feb4c15c9a0 legacy=0x7feb4c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.644+0000 7feb51400640 1 -- 172.21.15.28:0/4008749477 >> v1:172.21.15.28:6789/0 conn(0x7feb4c0ad940 legacy=0x7feb4c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.644+0000 7feb51400640 1 -- 172.21.15.28:0/4008749477 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c06b170 con 0x7feb4c0b2400 2024-09-13T17:08:57.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.645+0000 7feb51400640 1 --2- 172.21.15.28:0/4008749477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c0ac850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009870 tx=0x7feb3c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7aff45842e906c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.645+0000 7feb50a00640 1 -- 172.21.15.28:0/4008749477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c032640 con 0x7feb4c0b2400 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.645+0000 7feb50a00640 1 -- 172.21.15.28:0/4008749477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c0327a0 con 0x7feb4c0b2400 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.645+0000 7feb50a00640 1 -- 172.21.15.28:0/4008749477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c032b30 con 0x7feb4c0b2400 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 -- 172.21.15.28:0/4008749477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 msgr2=0x7feb4c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 --2- 172.21.15.28:0/4008749477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c0ac850 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009870 tx=0x7feb3c02a300 comp rx=0 tx=0).stop 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 -- 172.21.15.28:0/4008749477 shutdown_connections 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 --2- 172.21.15.28:0/4008749477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c0ac850 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 -- 172.21.15.28:0/4008749477 >> 172.21.15.28:0/4008749477 conn(0x7feb4c0b0270 msgr2=0x7feb4c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 -- 172.21.15.28:0/4008749477 shutdown_connections 2024-09-13T17:08:57.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.646+0000 7feb54800640 1 -- 172.21.15.28:0/4008749477 wait complete. 2024-09-13T17:08:57.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.647+0000 7feb54800640 1 Processor -- start 2024-09-13T17:08:57.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.647+0000 7feb54800640 1 -- start start 2024-09-13T17:08:57.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4c0ad940 0x7feb4c1f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb4c15c9a0 0x7feb4c1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c152340 con 0x7feb4c0b2400 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c1f20a0 con 0x7feb4c0ad940 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.648+0000 7feb54800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4c151a10 con 0x7feb4c15c9a0 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48314/0 (socket says 172.21.15.28:48314) 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 -- 172.21.15.28:0/2459587439 learned_addr learned my addr 172.21.15.28:0/2459587439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb52800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb4c15c9a0 0x7feb4c1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51e00640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4c0ad940 0x7feb4c1f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 -- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb4c15c9a0 msgr2=0x7feb4c1540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb4c15c9a0 0x7feb4c1540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 -- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4c0ad940 msgr2=0x7feb4c1f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4c0ad940 0x7feb4c1f33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 -- 172.21.15.28:0/2459587439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0094d0 con 0x7feb4c0b2400 2024-09-13T17:08:57.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.649+0000 7feb51400640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.650+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c030b80 con 0x7feb4c0b2400 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.650+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3c030d00 con 0x7feb4c0b2400 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.650+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3c03d660 con 0x7feb4c0b2400 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.650+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4c070930 con 0x7feb4c0b2400 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.650+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4c151fd0 con 0x7feb4c0b2400 2024-09-13T17:08:57.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.651+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4c152160 con 0x7feb4c0b2400 2024-09-13T17:08:57.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.652+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feb3c039040 con 0x7feb4c0b2400 2024-09-13T17:08:57.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.652+0000 7feb46000640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 0x7feb3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:57.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.653+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7feb3c0a9ff0 con 0x7feb4c0b2400 2024-09-13T17:08:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.653+0000 7feb51e00640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 0x7feb3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:57.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.654+0000 7feb51e00640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 0x7feb3006d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb40000f60 tx=0x7feb40009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.657+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feb3c0dd9c0 con 0x7feb4c0b2400 2024-09-13T17:08:57.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.835+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feb4c152480 con 0x7feb4c0b2400 2024-09-13T17:08:57.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.836+0000 7feb46000640 1 -- 172.21.15.28:0/2459587439 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7feb3c075250 con 0x7feb4c0b2400 2024-09-13T17:08:57.837 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 msgr2=0x7feb3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 0x7feb3006d0c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7feb40000f60 tx=0x7feb40009040 comp rx=0 tx=0).stop 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 msgr2=0x7feb4c1f1b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feb3c000c00 tx=0x7feb3c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 shutdown_connections 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb3006ac00 0x7feb3006d0c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb4c15c9a0 0x7feb4c1540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb4c0b2400 0x7feb4c1f1b60 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 --2- 172.21.15.28:0/2459587439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb4c0ad940 0x7feb4c1f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 >> 172.21.15.28:0/2459587439 conn(0x7feb4c0b0270 msgr2=0x7feb4c0b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 shutdown_connections 2024-09-13T17:08:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:57.838+0000 7feb54800640 1 -- 172.21.15.28:0/2459587439 wait complete. 2024-09-13T17:08:57.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:08:58.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.012+0000 7f67f6400640 1 Processor -- start 2024-09-13T17:08:58.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.013+0000 7f67f6400640 1 -- start start 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67f6400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f67f0143f70 0x7f67f00cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67f6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f015ef70 con 0x7f67f00cd5f0 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67f6400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f00c4320 con 0x7f67f00c3d70 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67f6400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f00cda80 con 0x7f67f0143f70 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67e7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f67f00cd5f0 0x7f67f00c3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37872/0 (socket says 172.21.15.28:37872) 2024-09-13T17:08:58.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.014+0000 7f67e7e00640 1 -- 172.21.15.28:0/784036121 learned_addr learned my addr 172.21.15.28:0/784036121 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67f5000640 1 --2- 172.21.15.28:0/784036121 >> v2:172.21.15.189:3300/0 conn(0x7f67f0143f70 0x7f67f00cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334272018 0 0) 0x7f67f015ef70 con 0x7f67f00cd5f0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d4003520 con 0x7f67f00cd5f0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864070542 0 0) 0x7f67f00c4320 con 0x7f67f00c3d70 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67f015ef70 con 0x7f67f00c3d70 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 630333559 0 0) 0x7f67d4003520 con 0x7f67f00cd5f0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 >> v2:172.21.15.189:3300/0 conn(0x7f67f0143f70 msgr2=0x7f67f00cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 --2- 172.21.15.28:0/784036121 >> v2:172.21.15.189:3300/0 conn(0x7f67f0143f70 0x7f67f00cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 >> v1:172.21.15.189:6789/0 conn(0x7f67f00c3d70 legacy=0x7f67f0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f015e1b0 con 0x7f67f00cd5f0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f67d8001670 con 0x7f67f00cd5f0 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 >> v1:172.21.15.28:6789/0 conn(0x7f67f00cd5f0 legacy=0x7f67f00c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67d4003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67d40047c0 0x7f67d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d4005180 0x7f67d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00cda80 con 0x7f67d4003b00 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d400c000 con 0x7f67d4005180 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.015+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67d400c1c0 con 0x7f67d40047c0 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67d4003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67e7e00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67d40047c0 0x7f67d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5a00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d4005180 0x7f67d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 -- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67d40047c0 msgr2=0x7f67d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67d40047c0 0x7f67d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 -- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d4005180 msgr2=0x7f67d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d4005180 0x7f67d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5000640 1 -- 172.21.15.28:0/784036121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4003520 con 0x7f67d4003b00 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5a00640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67d4005180 0x7f67d400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:58.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.016+0000 7f67f5a00640 1 -- 172.21.15.28:0/784036121 reap_dead start 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.017+0000 7f67f5000640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67d4003f00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009ac0 tx=0x7f67d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb93fb3df25b2e1d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.017+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031400 con 0x7f67d4003b00 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e002d050 con 0x7f67d4003b00 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67efe00640 1 -- 172.21.15.28:0/784036121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031780 con 0x7f67d4003b00 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 -- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 msgr2=0x7f67d4003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67d4003f00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009ac0 tx=0x7f67d40057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 -- 172.21.15.28:0/784036121 shutdown_connections 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 --2- 172.21.15.28:0/784036121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67d4003f00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 -- 172.21.15.28:0/784036121 >> 172.21.15.28:0/784036121 conn(0x7f67f0069e80 msgr2=0x7f67f006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 -- 172.21.15.28:0/784036121 shutdown_connections 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.018+0000 7f67f6400640 1 -- 172.21.15.28:0/784036121 wait complete. 2024-09-13T17:08:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.020+0000 7f67f6400640 1 Processor -- start 2024-09-13T17:08:58.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.020+0000 7f67f6400640 1 -- start start 2024-09-13T17:08:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f014d670 0x7f67f013bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0150520 0x7f67f015d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f014c990 con 0x7f67d4003b00 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00c8570 con 0x7f67f0150520 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f014cb90 con 0x7f67f014d670 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48318/0 (socket says 172.21.15.28:48318) 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 -- 172.21.15.28:0/2471729541 learned_addr learned my addr 172.21.15.28:0/2471729541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 -- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f014d670 msgr2=0x7f67f013bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f014d670 0x7f67f013bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 -- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0150520 msgr2=0x7f67f015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.021+0000 7f67f5000640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0150520 0x7f67f015d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67f5000640 1 -- 172.21.15.28:0/2471729541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f014ce70 con 0x7f67d4003b00 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67f5000640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f67e002aae0 tx=0x7f67e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0003e10 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e003b400 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f014d150 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e003b790 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.022+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f013c330 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.023+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f014d2e0 con 0x7f67d4003b00 2024-09-13T17:08:58.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.024+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f67e003b8f0 con 0x7f67d4003b00 2024-09-13T17:08:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.025+0000 7f67ed600640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.025+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f67e00a9ac0 con 0x7f67d4003b00 2024-09-13T17:08:58.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.025+0000 7f67e7e00640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.026+0000 7f67e7e00640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67d8000f60 tx=0x7f67d8000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.029+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f67e0074c50 con 0x7f67d4003b00 2024-09-13T17:08:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.207+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f67f014c990 con 0x7f67d4003b00 2024-09-13T17:08:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.207+0000 7f67ed600640 1 -- 172.21.15.28:0/2471729541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f67e00480a0 con 0x7f67d4003b00 2024-09-13T17:08:58.208 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 msgr2=0x7f67cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67d8000f60 tx=0x7f67d8000ed0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 msgr2=0x7f67f00c7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f67e002aae0 tx=0x7f67e0004730 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 shutdown_connections 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0150520 0x7f67f015d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f014d670 0x7f67f013bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 --2- 172.21.15.28:0/2471729541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67d4003b00 0x7f67f00c7cb0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 >> 172.21.15.28:0/2471729541 conn(0x7f67f0069e80 msgr2=0x7f67f013f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.209+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 shutdown_connections 2024-09-13T17:08:58.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.210+0000 7f67f6400640 1 -- 172.21.15.28:0/2471729541 wait complete. 2024-09-13T17:08:58.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.379+0000 7f7146a00640 1 Processor -- start 2024-09-13T17:08:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.380+0000 7f7146a00640 1 -- start start 2024-09-13T17:08:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 0x7f71400b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71400b0440 0x7f71400b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71400cad20 con 0x7f714015fb90 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71400b0df0 con 0x7f71400b0440 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714006bb40 con 0x7f71400b21c0 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7146000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71400b0440 0x7f71400b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7145600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f714015fb90 0x7f714006b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37886/0 (socket says 172.21.15.28:37886) 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7145600640 1 -- 172.21.15.28:0/4066028679 learned_addr learned my addr 172.21.15.28:0/4066028679 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.381+0000 7f7144c00640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 0x7f71400b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f7146000640 1 -- 172.21.15.28:0/4066028679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 msgr2=0x7f71400b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f7146000640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 0x7f71400b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f7146000640 1 -- 172.21.15.28:0/4066028679 >> v1:172.21.15.28:6789/0 conn(0x7f714015fb90 legacy=0x7f714006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f7146000640 1 -- 172.21.15.28:0/4066028679 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714006be20 con 0x7f71400b0440 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f7144c00640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 0x7f71400b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.382+0000 7f713fe00640 1 -- 172.21.15.28:0/4066028679 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235324692 0 0) 0x7f71400cad20 con 0x7f714015fb90 2024-09-13T17:08:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f7146000640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71400b0440 0x7f71400b08b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7138009d90 tx=0x7f713802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceae64edfc125b4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f713fe00640 1 -- 172.21.15.28:0/4066028679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138007740 con 0x7f71400b0440 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f713fe00640 1 -- 172.21.15.28:0/4066028679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713802d050 con 0x7f71400b0440 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f713fe00640 1 -- 172.21.15.28:0/4066028679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138007ac0 con 0x7f71400b0440 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f7146a00640 1 -- 172.21.15.28:0/4066028679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71400b0440 msgr2=0x7f71400b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.383+0000 7f7146a00640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71400b0440 0x7f71400b08b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7138009d90 tx=0x7f713802a530 comp rx=0 tx=0).stop 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 -- 172.21.15.28:0/4066028679 shutdown_connections 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71400b0440 0x7f71400b08b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 --2- 172.21.15.28:0/4066028679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b21c0 0x7f71400b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 -- 172.21.15.28:0/4066028679 >> 172.21.15.28:0/4066028679 conn(0x7f7140072e40 msgr2=0x7f7140073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 -- 172.21.15.28:0/4066028679 shutdown_connections 2024-09-13T17:08:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.384+0000 7f7146a00640 1 -- 172.21.15.28:0/4066028679 wait complete. 2024-09-13T17:08:58.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.385+0000 7f7146a00640 1 Processor -- start 2024-09-13T17:08:58.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.385+0000 7f7146a00640 1 -- start start 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b0440 0x7f71400a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 0x7f7140157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71400aa000 con 0x7f714015fb90 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714015ab30 con 0x7f71400b21c0 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.386+0000 7f7146a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7140157860 con 0x7f71400b0440 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48338/0 (socket says 172.21.15.28:48338) 2024-09-13T17:08:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 -- 172.21.15.28:0/3829445177 learned_addr learned my addr 172.21.15.28:0/3829445177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7145600640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b0440 0x7f71400a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7144c00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 0x7f7140157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 -- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b0440 msgr2=0x7f71400a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b0440 0x7f71400a9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 -- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 msgr2=0x7f7140157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 0x7f7140157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 -- 172.21.15.28:0/3829445177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138009a70 con 0x7f714015fb90 2024-09-13T17:08:58.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7144c00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 0x7f7140157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.387+0000 7f7146000640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7138009eb0 tx=0x7f7138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.388+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713803c830 con 0x7f714015fb90 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.388+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713803c990 con 0x7f714015fb90 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.388+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713803cd00 con 0x7f714015fb90 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.388+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714015a850 con 0x7f714015fb90 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.388+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71400a92c0 con 0x7f714015fb90 2024-09-13T17:08:58.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.389+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71400b08b0 con 0x7f714015fb90 2024-09-13T17:08:58.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.390+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f713802ad50 con 0x7f714015fb90 2024-09-13T17:08:58.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.391+0000 7f713d600640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 0x7f712006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.391+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f71380aad30 con 0x7f714015fb90 2024-09-13T17:08:58.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.391+0000 7f7145600640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 0x7f712006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.392+0000 7f7145600640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 0x7f712006d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7134007fd0 tx=0x7f7134007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.395+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7138075e20 con 0x7f714015fb90 2024-09-13T17:08:58.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.613+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f714015eae0 con 0x7f714015fb90 2024-09-13T17:08:58.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.613+0000 7f713d600640 1 -- 172.21.15.28:0/3829445177 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f713803ce60 con 0x7f714015fb90 2024-09-13T17:08:58.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:58.614 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:58.615 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:58.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 msgr2=0x7f712006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 0x7f712006d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7134007fd0 tx=0x7f7134007de0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 msgr2=0x7f71400cb390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7138009eb0 tx=0x7f7138004a40 comp rx=0 tx=0).stop 2024-09-13T17:08:58.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 shutdown_connections 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f712006ac00 0x7f712006d0c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f714015fb90 0x7f71400cb390 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71400b21c0 0x7f7140157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 --2- 172.21.15.28:0/3829445177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71400b0440 0x7f71400a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 >> 172.21.15.28:0/3829445177 conn(0x7f7140072e40 msgr2=0x7f7140070b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 shutdown_connections 2024-09-13T17:08:58.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.616+0000 7f7146a00640 1 -- 172.21.15.28:0/3829445177 wait complete. 2024-09-13T17:08:58.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:58.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.786+0000 7fa22a600640 1 Processor -- start 2024-09-13T17:08:58.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.786+0000 7fa22a600640 1 -- start start 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa22a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2241592a0 0x7fa22415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa22a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa22406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa22a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa22406cc70 con 0x7fa22406f970 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa22a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22415b0b0 con 0x7fa2241592a0 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa22a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224159d90 con 0x7fa22406c2d0 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa223400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2241592a0 0x7fa22415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa223400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa2241592a0 0x7fa22415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48342/0 (socket says 172.21.15.28:48342) 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.787+0000 7fa223400640 1 -- 172.21.15.28:0/492593219 learned_addr learned my addr 172.21.15.28:0/492593219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:58.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.788+0000 7fa223400640 1 -- 172.21.15.28:0/492593219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 msgr2=0x7fa22406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.788+0000 7fa223400640 1 --2- 172.21.15.28:0/492593219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa22406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.788+0000 7fa223400640 1 -- 172.21.15.28:0/492593219 >> v1:172.21.15.28:6789/0 conn(0x7fa22406f970 legacy=0x7fa224159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.788+0000 7fa223400640 1 -- 172.21.15.28:0/492593219 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240b2040 con 0x7fa2241592a0 2024-09-13T17:08:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.789+0000 7fa223400640 1 --2- 172.21.15.28:0/492593219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa22415ab70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa214009870 tx=0x7fa21402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502c4b69d766637a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.789+0000 7fa222a00640 1 -- 172.21.15.28:0/492593219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032440 con 0x7fa2241592a0 2024-09-13T17:08:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.789+0000 7fa222a00640 1 -- 172.21.15.28:0/492593219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2140325a0 con 0x7fa2241592a0 2024-09-13T17:08:58.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.789+0000 7fa222a00640 1 -- 172.21.15.28:0/492593219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214032930 con 0x7fa2241592a0 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 -- 172.21.15.28:0/492593219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 msgr2=0x7fa22415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 --2- 172.21.15.28:0/492593219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa22415ab70 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fa214009870 tx=0x7fa21402a250 comp rx=0 tx=0).stop 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 -- 172.21.15.28:0/492593219 shutdown_connections 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 --2- 172.21.15.28:0/492593219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa22406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 --2- 172.21.15.28:0/492593219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa22415ab70 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 -- 172.21.15.28:0/492593219 >> 172.21.15.28:0/492593219 conn(0x7fa224069e80 msgr2=0x7fa22406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 -- 172.21.15.28:0/492593219 shutdown_connections 2024-09-13T17:08:58.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.790+0000 7fa22a600640 1 -- 172.21.15.28:0/492593219 wait complete. 2024-09-13T17:08:58.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.792+0000 7fa22a600640 1 Processor -- start 2024-09-13T17:08:58.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.792+0000 7fa22a600640 1 -- start start 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa224148220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 0x7fa2241582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22414b5e0 con 0x7fa2241592a0 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22414a9d0 con 0x7fa22406f970 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa22a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22414acb0 con 0x7fa22406c2d0 2024-09-13T17:08:58.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48354/0 (socket says 172.21.15.28:48354) 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 -- 172.21.15.28:0/3574156495 learned_addr learned my addr 172.21.15.28:0/3574156495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 -- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 msgr2=0x7fa224148220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa224148220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 -- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 msgr2=0x7fa2241582c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa21be00640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 0x7fa2241582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 0x7fa2241582c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa223e00640 1 -- 172.21.15.28:0/3574156495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2140094d0 con 0x7fa2241592a0 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.793+0000 7fa21be00640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 0x7fa2241582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa223e00640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00b970 tx=0x7fa21c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c00cb40 con 0x7fa2241592a0 2024-09-13T17:08:58.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c00ccc0 con 0x7fa2241592a0 2024-09-13T17:08:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21c014680 con 0x7fa2241592a0 2024-09-13T17:08:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa22414af90 con 0x7fa2241592a0 2024-09-13T17:08:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.794+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa224156fd0 con 0x7fa2241592a0 2024-09-13T17:08:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.795+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2240b3740 con 0x7fa2241592a0 2024-09-13T17:08:58.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.796+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa21c007370 con 0x7fa2241592a0 2024-09-13T17:08:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.797+0000 7fa21b400640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 0x7fa20006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.797+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fa21c089260 con 0x7fa2241592a0 2024-09-13T17:08:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.798+0000 7fa223400640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 0x7fa20006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:58.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.798+0000 7fa223400640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 0x7fa20006d0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa214034000 tx=0x7fa21402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:58.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.802+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa21c054470 con 0x7fa2241592a0 2024-09-13T17:08:58.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.981+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2240ae8b0 con 0x7fa2241592a0 2024-09-13T17:08:58.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.982+0000 7fa21b400640 1 -- 172.21.15.28:0/3574156495 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16725 (secure 0 0 0) 0x7fa2240ae8b0 con 0x7fa2241592a0 2024-09-13T17:08:58.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:58.983 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:52.005252+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:08:58.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 msgr2=0x7fa20006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 0x7fa20006d0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa214034000 tx=0x7fa21402c000 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 msgr2=0x7fa224157770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00b970 tx=0x7fa21c00be30 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 shutdown_connections 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa20006ac00 0x7fa20006d0c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2241592a0 0x7fa224157770 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa22406f970 0x7fa2241582c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 --2- 172.21.15.28:0/3574156495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa22406c2d0 0x7fa224148220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.984+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 >> 172.21.15.28:0/3574156495 conn(0x7fa224069e80 msgr2=0x7fa224152030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.985+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 shutdown_connections 2024-09-13T17:08:58.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:58.985+0000 7fa22a600640 1 -- 172.21.15.28:0/3574156495 wait complete. 2024-09-13T17:08:59.001 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:08:59.001 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.159+0000 7f0403000640 1 Processor -- start 2024-09-13T17:08:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.159+0000 7f0403000640 1 -- start start 2024-09-13T17:08:59.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f0403000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03fc15a540 0x7f03fc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f0403000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc15e100 con 0x7f03fc0cabc0 2024-09-13T17:08:59.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f0403000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03fc15e3e0 con 0x7f03fc070cc0 2024-09-13T17:08:59.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f0403000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03fc0adb00 con 0x7f03fc15a540 2024-09-13T17:08:59.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f03fb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f03fc0cabc0 0x7f03fc15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37912/0 (socket says 172.21.15.28:37912) 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.160+0000 7f03fb400640 1 -- 172.21.15.28:0/3953513078 learned_addr learned my addr 172.21.15.28:0/3953513078 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03fbe00640 1 --2- 172.21.15.28:0/3953513078 >> v2:172.21.15.189:3300/0 conn(0x7f03fc15a540 0x7f03fc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211274691 0 0) 0x7f03fc15e100 con 0x7f03fc0cabc0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03e0003520 con 0x7f03fc0cabc0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1241814046 0 0) 0x7f03e0003520 con 0x7f03fc0cabc0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 >> v2:172.21.15.189:3300/0 conn(0x7f03fc15a540 msgr2=0x7f03fc06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 --2- 172.21.15.28:0/3953513078 >> v2:172.21.15.189:3300/0 conn(0x7f03fc15a540 0x7f03fc06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 >> v1:172.21.15.189:6789/0 conn(0x7f03fc070cc0 legacy=0x7f03fc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc15ef80 con 0x7f03fc0cabc0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f03e8001670 con 0x7f03fc0cabc0 2024-09-13T17:08:59.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 >> v1:172.21.15.28:6789/0 conn(0x7f03fc0cabc0 legacy=0x7f03fc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00051d0 0x7f03e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e000c0f0 con 0x7f03e00039c0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e000c260 con 0x7f03e00047c0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.161+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e000c420 con 0x7f03e00051d0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fb400640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f0401000640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00051d0 0x7f03e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 -- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00051d0 msgr2=0x7f03e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:59.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00051d0 0x7f03e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 -- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00047c0 msgr2=0x7f03e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fbe00640 1 -- 172.21.15.28:0/3953513078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e0003520 con 0x7f03e00039c0 2024-09-13T17:08:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f03fb400640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03e00047c0 0x7f03e0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:59.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.162+0000 7f0401000640 1 -- 172.21.15.28:0/3953513078 reap_dead start 2024-09-13T17:08:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.163+0000 7f03fbe00640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 0x7f03e0003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009a90 tx=0x7f03e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0681b2423fdf622 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.163+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec031400 con 0x7f03e00039c0 2024-09-13T17:08:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.163+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ec02d050 con 0x7f03e00039c0 2024-09-13T17:08:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.163+0000 7f03faa00640 1 -- 172.21.15.28:0/3953513078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ec031780 con 0x7f03e00039c0 2024-09-13T17:08:59.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 -- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 msgr2=0x7f03e0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 0x7f03e0003de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f03ec009a90 tx=0x7f03e00057f0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 -- 172.21.15.28:0/3953513078 shutdown_connections 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 --2- 172.21.15.28:0/3953513078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03e00039c0 0x7f03e0003de0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 -- 172.21.15.28:0/3953513078 >> 172.21.15.28:0/3953513078 conn(0x7f03fc069e80 msgr2=0x7f03fc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 -- 172.21.15.28:0/3953513078 shutdown_connections 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.164+0000 7f0403000640 1 -- 172.21.15.28:0/3953513078 wait complete. 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.165+0000 7f0403000640 1 Processor -- start 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.166+0000 7f0403000640 1 -- start start 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.166+0000 7f0403000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 0x7f03fc0b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.166+0000 7f0403000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.166+0000 7f0403000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 0x7f03fc13ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.166+0000 7f0403000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0b0a60 con 0x7f03fc14f1c0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f0403000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc13e420 con 0x7f03fc13dfa0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f0403000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03fc0af480 con 0x7f03e00039c0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48368/0 (socket says 172.21.15.28:48368) 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 -- 172.21.15.28:0/2227377922 learned_addr learned my addr 172.21.15.28:0/2227377922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fbe00640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 0x7f03fc0b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f0401000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 0x7f03fc13ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 -- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 msgr2=0x7f03fc0b0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 0x7f03fc0b0520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 -- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 msgr2=0x7f03fc13ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 0x7f03fc13ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fb400640 1 -- 172.21.15.28:0/2227377922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc0af760 con 0x7f03fc14f1c0 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f03fbe00640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 0x7f03fc0b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.167+0000 7f0401000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 0x7f03fc13ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f03fb400640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f03e800e7f0 tx=0x7f03e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e800c870 con 0x7f03fc14f1c0 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e800c9d0 con 0x7f03fc14f1c0 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e800cd40 con 0x7f03fc14f1c0 2024-09-13T17:08:59.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03fc0afa40 con 0x7f03fc14f1c0 2024-09-13T17:08:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.168+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03fc0b0000 con 0x7f03fc14f1c0 2024-09-13T17:08:59.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.169+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03fc06c7c0 con 0x7f03fc14f1c0 2024-09-13T17:08:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.170+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f03e8013420 con 0x7f03fc14f1c0 2024-09-13T17:08:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.171+0000 7f03f3e00640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 0x7f03dc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.171+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f03e80890a0 con 0x7f03fc14f1c0 2024-09-13T17:08:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.171+0000 7f03fbe00640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 0x7f03dc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.172+0000 7f03fbe00640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 0x7f03dc06d0f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03ec033000 tx=0x7f03ec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.175+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f03e8054190 con 0x7f03fc14f1c0 2024-09-13T17:08:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.389+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03fc15cce0 con 0x7f03fc14f1c0 2024-09-13T17:08:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.389+0000 7f03f3e00640 1 -- 172.21.15.28:0/2227377922 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f03e8058e60 con 0x7f03fc14f1c0 2024-09-13T17:08:59.391 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:59.391 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.391+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 msgr2=0x7f03dc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 0x7f03dc06d0f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f03ec033000 tx=0x7f03ec008030 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 msgr2=0x7f03fc13da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f03e800e7f0 tx=0x7f03e800ecb0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 shutdown_connections 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03dc06ac30 0x7f03dc06d0f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03fc13dfa0 0x7f03fc13ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03fc14f1c0 0x7f03fc13da60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 --2- 172.21.15.28:0/2227377922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03e00039c0 0x7f03fc0b0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 >> 172.21.15.28:0/2227377922 conn(0x7f03fc069e80 msgr2=0x7f03fc15aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 shutdown_connections 2024-09-13T17:08:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.392+0000 7f0403000640 1 -- 172.21.15.28:0/2227377922 wait complete. 2024-09-13T17:08:59.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:08:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.563+0000 7f4e99000640 1 Processor -- start 2024-09-13T17:08:59.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.563+0000 7f4e99000640 1 -- start start 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e99000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e941592a0 0x7f4e9415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e99000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406c2d0 0x7f4e9406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e99000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9406cc70 con 0x7f4e9406f970 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e99000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9415b0b0 con 0x7f4e941592a0 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e99000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e94159d90 con 0x7f4e9406c2d0 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e91e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e941592a0 0x7f4e9415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e91e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e941592a0 0x7f4e9415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48374/0 (socket says 172.21.15.28:48374) 2024-09-13T17:08:59.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.564+0000 7f4e91e00640 1 -- 172.21.15.28:0/374179460 learned_addr learned my addr 172.21.15.28:0/374179460 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.565+0000 7f4e92800640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406c2d0 0x7f4e9406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.565+0000 7f4e91e00640 1 -- 172.21.15.28:0/374179460 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406c2d0 msgr2=0x7f4e9406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.565+0000 7f4e91e00640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406c2d0 0x7f4e9406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.565+0000 7f4e91e00640 1 -- 172.21.15.28:0/374179460 >> v1:172.21.15.189:6789/0 conn(0x7f4e9406f970 legacy=0x7f4e94159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.565+0000 7f4e91e00640 1 -- 172.21.15.28:0/374179460 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e940b2040 con 0x7f4e941592a0 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.566+0000 7f4e91e00640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e9415ab70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009870 tx=0x7f4e8402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834d88f5a18189b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.566+0000 7f4e90a00640 1 -- 172.21.15.28:0/374179460 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84032440 con 0x7f4e941592a0 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.566+0000 7f4e90a00640 1 -- 172.21.15.28:0/374179460 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e840325a0 con 0x7f4e941592a0 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.566+0000 7f4e90a00640 1 -- 172.21.15.28:0/374179460 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84032930 con 0x7f4e941592a0 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 -- 172.21.15.28:0/374179460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 msgr2=0x7f4e9415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e9415ab70 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4e84009870 tx=0x7f4e8402a270 comp rx=0 tx=0).stop 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 -- 172.21.15.28:0/374179460 shutdown_connections 2024-09-13T17:08:59.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406c2d0 0x7f4e9406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 --2- 172.21.15.28:0/374179460 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e9415ab70 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 -- 172.21.15.28:0/374179460 >> 172.21.15.28:0/374179460 conn(0x7f4e94069e80 msgr2=0x7f4e9406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 -- 172.21.15.28:0/374179460 shutdown_connections 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.567+0000 7f4e99000640 1 -- 172.21.15.28:0/374179460 wait complete. 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.568+0000 7f4e99000640 1 Processor -- start 2024-09-13T17:08:59.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.569+0000 7f4e99000640 1 -- start start 2024-09-13T17:08:59.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.569+0000 7f4e99000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e99000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f970 0x7f4e941e3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e99000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e941e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e99000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940cf4d0 con 0x7f4e941592a0 2024-09-13T17:08:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e99000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941515a0 con 0x7f4e9406c2d0 2024-09-13T17:08:59.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e99000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941e42a0 con 0x7f4e9406f970 2024-09-13T17:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e91e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e941e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e91e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58844/0 (socket says 172.21.15.28:58844) 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e91e00640 1 -- 172.21.15.28:0/2156017435 learned_addr learned my addr 172.21.15.28:0/2156017435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.570+0000 7f4e91400640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f970 0x7f4e941e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 -- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f970 msgr2=0x7f4e941e3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f970 0x7f4e941e3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 -- 172.21.15.28:0/2156017435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 msgr2=0x7f4e941e3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e941e3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 -- 172.21.15.28:0/2156017435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e840094d0 con 0x7f4e9406c2d0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e92800640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e941e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e91e00640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84032440 con 0x7f4e9406c2d0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e84030980 con 0x7f4e9406c2d0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.571+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84030d30 con 0x7f4e9406c2d0 2024-09-13T17:08:59.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.572+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e941e4580 con 0x7f4e9406c2d0 2024-09-13T17:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.572+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9413dfb0 con 0x7f4e9406c2d0 2024-09-13T17:08:59.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.573+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e940b3740 con 0x7f4e9406c2d0 2024-09-13T17:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.574+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e84039040 con 0x7f4e9406c2d0 2024-09-13T17:08:59.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.575+0000 7f4e8a000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 0x7f4e7406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.575+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f4e840a9d00 con 0x7f4e9406c2d0 2024-09-13T17:08:59.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.575+0000 7f4e91400640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 0x7f4e7406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.576+0000 7f4e91400640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 0x7f4e7406d0c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000f60 tx=0x7f4e80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.581+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4e84074d80 con 0x7f4e9406c2d0 2024-09-13T17:08:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.794+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e9415b720 con 0x7f4e9406c2d0 2024-09-13T17:08:59.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.795+0000 7f4e8a000640 1 -- 172.21.15.28:0/2156017435 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4728 (secure 0 0 0) 0x7f4e840783a0 con 0x7f4e9406c2d0 2024-09-13T17:08:59.796 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:08:59.797 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3394061369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3394061369},{"type":"v1","addr":"172.21.15.28:6835","nonce":3394061369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/456709939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":456709939},{"type":"v1","addr":"172.21.15.189:6836","nonce":456709939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4275,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1782735359","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1782735359},{"type":"v1","addr":"172.21.15.28:6833","nonce":1782735359}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:08:11.555431+0000","modified":"2024-09-13T17:08:14.569832+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/946583009","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":946583009},{"type":"v1","addr":"172.21.15.189:6837","nonce":946583009}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T17:08:59.797 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-09-13T17:08:59.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 msgr2=0x7f4e7406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 0x7f4e7406d0c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4e80000f60 tx=0x7f4e80009040 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 msgr2=0x7f4e940cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e840046c0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 shutdown_connections 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7406ac00 0x7f4e7406d0c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e941592a0 0x7f4e941e3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f970 0x7f4e941e3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 --2- 172.21.15.28:0/2156017435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9406c2d0 0x7f4e940cef90 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 >> 172.21.15.28:0/2156017435 conn(0x7f4e94069e80 msgr2=0x7f4e9406fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 shutdown_connections 2024-09-13T17:08:59.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.797+0000 7f4e99000640 1 -- 172.21.15.28:0/2156017435 wait complete. 2024-09-13T17:08:59.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:08:59.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.968+0000 7f28ab200640 1 Processor -- start 2024-09-13T17:08:59.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.969+0000 7f28ab200640 1 -- start start 2024-09-13T17:08:59.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28ab200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f28ac154a90 0x7f28ac15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28ab200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac06b360 0x7f28ac0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28ab200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ac15bb10 con 0x7f28ac0b0980 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28ab200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ac0b0df0 con 0x7f28ac154a90 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28ab200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac06b900 con 0x7f28ac06b360 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28aa800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28ac0b0980 0x7f28ac0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37920/0 (socket says 172.21.15.28:37920) 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28aa800640 1 -- 172.21.15.28:0/589331535 learned_addr learned my addr 172.21.15.28:0/589331535 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28a9e00640 1 --2- 172.21.15.28:0/589331535 >> v2:172.21.15.189:3300/0 conn(0x7f28ac154a90 0x7f28ac15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.970+0000 7f28a1400640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac06b360 0x7f28ac0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9400640 1 -- 172.21.15.28:0/589331535 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203350237 0 0) 0x7f28ac15bb10 con 0x7f28ac0b0980 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9400640 1 -- 172.21.15.28:0/589331535 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2890003520 con 0x7f28ac0b0980 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9e00640 1 -- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac06b360 msgr2=0x7f28ac0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:08:59.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9e00640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac06b360 0x7f28ac0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9e00640 1 -- 172.21.15.28:0/589331535 >> v1:172.21.15.28:6789/0 conn(0x7f28ac0b0980 legacy=0x7f28ac0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.971+0000 7f28a9e00640 1 -- 172.21.15.28:0/589331535 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac06bbe0 con 0x7f28ac154a90 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.972+0000 7f28a9e00640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac154a90 0x7f28ac15b5d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f289c0097a0 tx=0x7f289c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66500f7e0b13a59d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.972+0000 7f28a9400640 1 -- 172.21.15.28:0/589331535 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c032440 con 0x7f28ac154a90 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.972+0000 7f28a9400640 1 -- 172.21.15.28:0/589331535 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c0325a0 con 0x7f28ac154a90 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.972+0000 7f28a9400640 1 -- 172.21.15.28:0/589331535 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f289c032930 con 0x7f28ac154a90 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 -- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac154a90 msgr2=0x7f28ac15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac154a90 0x7f28ac15b5d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f289c0097a0 tx=0x7f289c02a180 comp rx=0 tx=0).stop 2024-09-13T17:08:59.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 -- 172.21.15.28:0/589331535 shutdown_connections 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac06b360 0x7f28ac0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 --2- 172.21.15.28:0/589331535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac154a90 0x7f28ac15b5d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 -- 172.21.15.28:0/589331535 >> 172.21.15.28:0/589331535 conn(0x7f28ac072e40 msgr2=0x7f28ac073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 -- 172.21.15.28:0/589331535 shutdown_connections 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.973+0000 7f28ab200640 1 -- 172.21.15.28:0/589331535 wait complete. 2024-09-13T17:08:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.975+0000 7f28ab200640 1 Processor -- start 2024-09-13T17:08:59.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.975+0000 7f28ab200640 1 -- start start 2024-09-13T17:08:59.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac06b360 0x7f28ac0ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 0x7f28ac0ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac0aa3c0 con 0x7f28ac154a90 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac0aa6a0 con 0x7f28ac06b360 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28ab200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac0aa980 con 0x7f28ac0b0980 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28a9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac06b360 0x7f28ac0ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48396/0 (socket says 172.21.15.28:48396) 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28aa800640 1 -- 172.21.15.28:0/1203663122 learned_addr learned my addr 172.21.15.28:0/1203663122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.976+0000 7f28a1400640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 0x7f28ac0ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 -- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 msgr2=0x7f28ac0ae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 0x7f28ac0ae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 -- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac06b360 msgr2=0x7f28ac0ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:08:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac06b360 0x7f28ac0ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:08:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 -- 172.21.15.28:0/1203663122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c009400 con 0x7f28ac154a90 2024-09-13T17:08:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28a1400640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 0x7f28ac0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:08:59.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28aa800640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f28a400caa0 tx=0x7f28a400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a400ac00 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a400ad60 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac0aac60 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a4012620 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.977+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac0a9620 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.978+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ac0aae80 con 0x7f28ac154a90 2024-09-13T17:08:59.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.979+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28a4007070 con 0x7f28ac154a90 2024-09-13T17:08:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.980+0000 7f28a2a00640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 0x7f288806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:08:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.980+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7f28a4089380 con 0x7f28ac154a90 2024-09-13T17:08:59.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.980+0000 7f28a9e00640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 0x7f288806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:08:59.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.981+0000 7f28a9e00640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 0x7f288806d0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f289c034000 tx=0x7f289c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:08:59.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:08:59.984+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28a4054510 con 0x7f28ac154a90 2024-09-13T17:09:00.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.165+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28ac070f80 con 0x7f28ac154a90 2024-09-13T17:09:00.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.166+0000 7f28a2a00640 1 -- 172.21.15.28:0/1203663122 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16725 (secure 0 0 0) 0x7f28a4057b30 con 0x7f28ac154a90 2024-09-13T17:09:00.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:00.167 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":22,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:08:52.005252+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:08:09.855355+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T17:08:10.527313+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.7","mappings":[{"from":3,"to":2}]},{"pgid":"3.22","mappings":[{"from":5,"to":2},{"from":3,"to":1}]},{"pgid":"3.3a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 msgr2=0x7f288806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 0x7f288806d0c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f289c034000 tx=0x7f289c02e000 comp rx=0 tx=0).stop 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 msgr2=0x7f28ac0aef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f28a400caa0 tx=0x7f28a400cf60 comp rx=0 tx=0).stop 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 shutdown_connections 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f288806ac00 0x7f288806d0c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28ac154a90 0x7f28ac0aef00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28ac0b0980 0x7f28ac0ae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 --2- 172.21.15.28:0/1203663122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28ac06b360 0x7f28ac0ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 >> 172.21.15.28:0/1203663122 conn(0x7f28ac072e40 msgr2=0x7f28ac070180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 shutdown_connections 2024-09-13T17:09:00.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.168+0000 7f28ab200640 1 -- 172.21.15.28:0/1203663122 wait complete. 2024-09-13T17:09:00.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:09:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.341+0000 7fd00a800640 1 Processor -- start 2024-09-13T17:09:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.341+0000 7fd00a800640 1 -- start start 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd00a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd00406f3f0 0x7fd00415c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd00a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00415ab60 con 0x7fd004159770 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd00a800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00406f920 con 0x7fd004070430 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd00a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00415b290 con 0x7fd00406f3f0 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd003e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd00406f3f0 0x7fd00415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd003400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd004159770 0x7fd00406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37924/0 (socket says 172.21.15.28:37924) 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.342+0000 7fd003400640 1 -- 172.21.15.28:0/3210762793 learned_addr learned my addr 172.21.15.28:0/3210762793 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.343+0000 7fd003e00640 1 -- 172.21.15.28:0/3210762793 >> v1:172.21.15.189:6789/0 conn(0x7fd004070430 legacy=0x7fd00415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.343+0000 7fd003e00640 1 -- 172.21.15.28:0/3210762793 >> v1:172.21.15.28:6789/0 conn(0x7fd004159770 legacy=0x7fd00406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.343+0000 7fd003e00640 1 -- 172.21.15.28:0/3210762793 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00415b570 con 0x7fd00406f3f0 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.343+0000 7fd002a00640 1 -- 172.21.15.28:0/3210762793 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450155566 0 0) 0x7fcff4001110 con 0x7fd004159770 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.344+0000 7fd003e00640 1 --2- 172.21.15.28:0/3210762793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00406f3f0 0x7fd00415c480 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f92928419582f925 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.344+0000 7fd002a00640 1 -- 172.21.15.28:0/3210762793 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc007740 con 0x7fd00406f3f0 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.344+0000 7fd002a00640 1 -- 172.21.15.28:0/3210762793 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc02d050 con 0x7fd00406f3f0 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.344+0000 7fd002a00640 1 -- 172.21.15.28:0/3210762793 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc007ac0 con 0x7fd00406f3f0 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 -- 172.21.15.28:0/3210762793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00406f3f0 msgr2=0x7fd00415c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 --2- 172.21.15.28:0/3210762793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00406f3f0 0x7fd00415c480 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc02a660 comp rx=0 tx=0).stop 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 -- 172.21.15.28:0/3210762793 shutdown_connections 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 --2- 172.21.15.28:0/3210762793 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd00406f3f0 0x7fd00415c480 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 -- 172.21.15.28:0/3210762793 >> 172.21.15.28:0/3210762793 conn(0x7fd0040b0270 msgr2=0x7fd0040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 -- 172.21.15.28:0/3210762793 shutdown_connections 2024-09-13T17:09:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.345+0000 7fd00a800640 1 -- 172.21.15.28:0/3210762793 wait complete. 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.346+0000 7fd00a800640 1 Processor -- start 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.346+0000 7fd00a800640 1 -- start start 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd00406f3f0 0x7fd00406bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd004070430 0x7fd0041e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0040b37e0 con 0x7fd004159770 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0041e7d10 con 0x7fd00406f3f0 2024-09-13T17:09:00.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.347+0000 7fd00a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0040b2c60 con 0x7fd004070430 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48410/0 (socket says 172.21.15.28:48410) 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 -- 172.21.15.28:0/3498950748 learned_addr learned my addr 172.21.15.28:0/3498950748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003400640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd00406f3f0 0x7fd00406bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fcffbe00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd004070430 0x7fd0041e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 -- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd004070430 msgr2=0x7fd0041e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd004070430 0x7fd0041e7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 -- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd00406f3f0 msgr2=0x7fd00406bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd00406f3f0 0x7fd00406bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 -- 172.21.15.28:0/3498950748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc009a70 con 0x7fd004159770 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.348+0000 7fd003e00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.349+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc031910 con 0x7fd004159770 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.349+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc031a70 con 0x7fd004159770 2024-09-13T17:09:00.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.349+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc031de0 con 0x7fd004159770 2024-09-13T17:09:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.349+0000 7fd00a800640 1 -- 172.21.15.28:0/3498950748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0040b2f40 con 0x7fd004159770 2024-09-13T17:09:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.349+0000 7fd00a800640 1 -- 172.21.15.28:0/3498950748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0040b3500 con 0x7fd004159770 2024-09-13T17:09:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.350+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00406fd20 con 0x7fd004159770 2024-09-13T17:09:00.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.351+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcffc03c420 con 0x7fd004159770 2024-09-13T17:09:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.352+0000 7fcffb400640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 0x7fcfe006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.352+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5804+0+0 (secure 0 0 0) 0x7fcffc0aad40 con 0x7fd004159770 2024-09-13T17:09:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.352+0000 7fd003400640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 0x7fcfe006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.353+0000 7fd003400640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 0x7fcfe006d0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000f60 tx=0x7fcff4007e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.356+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcffc075ed0 con 0x7fd004159770 2024-09-13T17:09:00.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.569+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd004070210 con 0x7fd004159770 2024-09-13T17:09:00.570 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:09:00.570+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:09:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.580+0000 7fcffb400640 1 -- 172.21.15.28:0/3498950748 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7fcffc0794f0 con 0x7fd004159770 2024-09-13T17:09:00.581 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 msgr2=0x7fcfe006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 0x7fcfe006d0c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcff4000f60 tx=0x7fcff4007e40 comp rx=0 tx=0).stop 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 msgr2=0x7fd0041e77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fcffc009d90 tx=0x7fcffc004a40 comp rx=0 tx=0).stop 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 shutdown_connections 2024-09-13T17:09:00.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfe006ac00 0x7fcfe006d0c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd004159770 0x7fd0041e77d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd004070430 0x7fd0041e7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 --2- 172.21.15.28:0/3498950748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd00406f3f0 0x7fd00406bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 >> 172.21.15.28:0/3498950748 conn(0x7fd0040b0270 msgr2=0x7fd0041e87f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 shutdown_connections 2024-09-13T17:09:00.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.582+0000 7fcff8c00640 1 -- 172.21.15.28:0/3498950748 wait complete. 2024-09-13T17:09:00.594 INFO:tasks.ceph.mds.d.smithi189.stderr: -7111> 2024-09-13T17:08:15.874+0000 7f8715200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:09:00.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:09:00.647 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:09:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.756+0000 7fa12d000640 1 Processor -- start 2024-09-13T17:09:00.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.757+0000 7fa12d000640 1 -- start start 2024-09-13T17:09:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa12d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa128143320 0x7fa128070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa12d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1280cd120 0x7fa1280cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa12d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa128071630 con 0x7fa128143910 2024-09-13T17:09:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa12d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1280cdad0 con 0x7fa1280cd120 2024-09-13T17:09:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa12d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa128143700 con 0x7fa128143320 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa11ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa128143910 0x7fa128071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37940/0 (socket says 172.21.15.28:37940) 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa11ea00640 1 -- 172.21.15.28:0/3181341939 learned_addr learned my addr 172.21.15.28:0/3181341939 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa127e00640 1 --2- 172.21.15.28:0/3181341939 >> v2:172.21.15.28:3300/0 conn(0x7fa1280cd120 0x7fa1280cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.758+0000 7fa127400640 1 --2- 172.21.15.28:0/3181341939 >> v2:172.21.15.189:3300/0 conn(0x7fa128143320 0x7fa128070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa126a00640 1 -- 172.21.15.28:0/3181341939 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 65330399 0 0) 0x7fa128071630 con 0x7fa128143910 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa126a00640 1 -- 172.21.15.28:0/3181341939 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa10c003520 con 0x7fa128143910 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa127e00640 1 -- 172.21.15.28:0/3181341939 >> v2:172.21.15.189:3300/0 conn(0x7fa128143320 msgr2=0x7fa128070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa127e00640 1 --2- 172.21.15.28:0/3181341939 >> v2:172.21.15.189:3300/0 conn(0x7fa128143320 0x7fa128070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa127e00640 1 -- 172.21.15.28:0/3181341939 >> v1:172.21.15.28:6789/0 conn(0x7fa128143910 legacy=0x7fa128071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.759+0000 7fa127e00640 1 -- 172.21.15.28:0/3181341939 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128070a20 con 0x7fa1280cd120 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.760+0000 7fa127e00640 1 --2- 172.21.15.28:0/3181341939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa1280cd590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa12806c690 tx=0x7fa12002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f16f5bf7c64cbde server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.760+0000 7fa126a00640 1 -- 172.21.15.28:0/3181341939 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12002aea0 con 0x7fa1280cd120 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.760+0000 7fa126a00640 1 -- 172.21.15.28:0/3181341939 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12002d050 con 0x7fa1280cd120 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.760+0000 7fa126a00640 1 -- 172.21.15.28:0/3181341939 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa120007960 con 0x7fa1280cd120 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 -- 172.21.15.28:0/3181341939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 msgr2=0x7fa1280cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 --2- 172.21.15.28:0/3181341939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa1280cd590 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa12806c690 tx=0x7fa12002a680 comp rx=0 tx=0).stop 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 -- 172.21.15.28:0/3181341939 shutdown_connections 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 --2- 172.21.15.28:0/3181341939 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa1280cd590 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 --2- 172.21.15.28:0/3181341939 >> v2:172.21.15.189:3300/0 conn(0x7fa128143320 0x7fa128070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 -- 172.21.15.28:0/3181341939 >> 172.21.15.28:0/3181341939 conn(0x7fa128069e80 msgr2=0x7fa12806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 -- 172.21.15.28:0/3181341939 shutdown_connections 2024-09-13T17:09:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.761+0000 7fa12d000640 1 -- 172.21.15.28:0/3181341939 wait complete. 2024-09-13T17:09:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.762+0000 7fa12d000640 1 Processor -- start 2024-09-13T17:09:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.763+0000 7fa12d000640 1 -- start start 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.763+0000 7fa12d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa12d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 0x7fa1281552c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa12d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 0x7fa1281f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa12d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12813da00 con 0x7fa1280cd120 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa12d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12813cdf0 con 0x7fa128143910 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa12d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12813d0d0 con 0x7fa128143320 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48428/0 (socket says 172.21.15.28:48428) 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 -- 172.21.15.28:0/1162710978 learned_addr learned my addr 172.21.15.28:0/1162710978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 -- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 msgr2=0x7fa1281552c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa11ea00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 0x7fa1281552c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127e00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 0x7fa1281f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 0x7fa1281552c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 -- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 msgr2=0x7fa1281f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 0x7fa1281f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 -- 172.21.15.28:0/1162710978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa120009a70 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa11ea00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 0x7fa1281552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127e00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 0x7fa1281f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.764+0000 7fa127400640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa11800e9b0 tx=0x7fa11800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.765+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa118009670 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.765+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11800c9a0 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.765+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12813d3b0 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.765+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11800cd50 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.765+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12813ca50 con 0x7fa1280cd120 2024-09-13T17:09:00.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.766+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1280b3800 con 0x7fa1280cd120 2024-09-13T17:09:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.767+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa118013470 con 0x7fa1280cd120 2024-09-13T17:09:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.768+0000 7fa11fe00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 0x7fa10406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.768+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5890+0+0 (secure 0 0 0) 0x7fa118089740 con 0x7fa1280cd120 2024-09-13T17:09:00.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.768+0000 7fa11ea00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 0x7fa10406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.769+0000 7fa11ea00640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 0x7fa10406d0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa128155d10 tx=0x7fa110009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.772+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa1180547c0 con 0x7fa1280cd120 2024-09-13T17:09:00.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:00.988+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa128144010 con 0x7fa1280cd120 2024-09-13T17:09:01.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.588+0000 7fa11fe00640 1 -- 172.21.15.28:0/1162710978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa118057de0 con 0x7fa1280cd120 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 msgr2=0x7fa10406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 0x7fa10406d0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa128155d10 tx=0x7fa110009040 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 msgr2=0x7fa12813c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa11800e9b0 tx=0x7fa11800ee70 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 shutdown_connections 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa10406ac00 0x7fa10406d0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa128143910 0x7fa1281f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa128143320 0x7fa1281552c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 --2- 172.21.15.28:0/1162710978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1280cd120 0x7fa12813c510 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.590+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 >> 172.21.15.28:0/1162710978 conn(0x7fa128069e80 msgr2=0x7fa1280b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.591+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 shutdown_connections 2024-09-13T17:09:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.591+0000 7fa12d000640 1 -- 172.21.15.28:0/1162710978 wait complete. 2024-09-13T17:09:01.610 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:09:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.771+0000 7f7970a00640 1 Processor -- start 2024-09-13T17:09:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.771+0000 7f7970a00640 1 -- start start 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7970a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f796c15a7a0 0x7f796c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7970a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c0ce9d0 0x7f796c0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7970a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c06c950 con 0x7f796c0b1ab0 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7970a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0cee40 con 0x7f796c15a7a0 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7970a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0b1ee0 con 0x7f796c0ce9d0 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7963e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c0ce9d0 0x7f796c0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7969e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f796c15a7a0 0x7f796c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f796a800640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f796c0b1ab0 0x7f796c06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:56554/0 (socket says 172.21.15.28:56554) 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f7969e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f796c15a7a0 0x7f796c0ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58892/0 (socket says 172.21.15.28:58892) 2024-09-13T17:09:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.773+0000 7f796a800640 1 -- 172.21.15.28:0/3060387306 learned_addr learned my addr 172.21.15.28:0/3060387306 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.774+0000 7f7969e00640 1 -- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c0ce9d0 msgr2=0x7f796c0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.774+0000 7f7969e00640 1 --2- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c0ce9d0 0x7f796c0b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.774+0000 7f7969e00640 1 -- 172.21.15.28:0/3060387306 >> v1:172.21.15.189:6789/0 conn(0x7f796c0b1ab0 legacy=0x7f796c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.774+0000 7f7969e00640 1 -- 172.21.15.28:0/3060387306 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0b2150 con 0x7f796c15a7a0 2024-09-13T17:09:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.775+0000 7f7969e00640 1 --2- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c15a7a0 0x7f796c0ce490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e70547627b11c92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.775+0000 7f7969400640 1 -- 172.21.15.28:0/3060387306 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c02ae40 con 0x7f796c15a7a0 2024-09-13T17:09:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.775+0000 7f7969400640 1 -- 172.21.15.28:0/3060387306 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c032440 con 0x7f796c15a7a0 2024-09-13T17:09:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.775+0000 7f7969400640 1 -- 172.21.15.28:0/3060387306 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0327d0 con 0x7f796c15a7a0 2024-09-13T17:09:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 -- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c15a7a0 msgr2=0x7f796c0ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 --2- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c15a7a0 0x7f796c0ce490 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f795c0097a0 tx=0x7f795c02a0f0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 -- 172.21.15.28:0/3060387306 shutdown_connections 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 --2- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c0ce9d0 0x7f796c0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 --2- 172.21.15.28:0/3060387306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c15a7a0 0x7f796c0ce490 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 -- 172.21.15.28:0/3060387306 >> 172.21.15.28:0/3060387306 conn(0x7f796c069e80 msgr2=0x7f796c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 -- 172.21.15.28:0/3060387306 shutdown_connections 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.776+0000 7f7970a00640 1 -- 172.21.15.28:0/3060387306 wait complete. 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.777+0000 7f7970a00640 1 Processor -- start 2024-09-13T17:09:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.777+0000 7f7970a00640 1 -- start start 2024-09-13T17:09:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 0x7f796c0c9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 0x7f796c1544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0c8940 con 0x7f796c0b1ab0 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c15df40 con 0x7f796c0ce9d0 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.778+0000 7f7970a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c0c8b90 con 0x7f796c15a7a0 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48444/0 (socket says 172.21.15.28:48444) 2024-09-13T17:09:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 -- 172.21.15.28:0/2415712470 learned_addr learned my addr 172.21.15.28:0/2415712470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f796a800640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 0x7f796c1544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7963e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 0x7f796c0c9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 -- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 msgr2=0x7f796c1544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 0x7f796c1544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 -- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 msgr2=0x7f796c0c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 0x7f796c0c9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7969e00640 1 -- 172.21.15.28:0/2415712470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c009400 con 0x7f796c0b1ab0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f7963e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 0x7f796c0c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.779+0000 7f796a800640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 0x7f796c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7969e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f795c035040 tx=0x7f795c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c03d8f0 con 0x7f796c0b1ab0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c03da70 con 0x7f796c0b1ab0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c03de20 con 0x7f796c0b1ab0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c0c8e70 con 0x7f796c0b1ab0 2024-09-13T17:09:01.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.780+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c0c9430 con 0x7f796c0b1ab0 2024-09-13T17:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.781+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c15ab80 con 0x7f796c0b1ab0 2024-09-13T17:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.782+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f795c039040 con 0x7f796c0b1ab0 2024-09-13T17:09:01.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.783+0000 7f7962000640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 0x7f794806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.783+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5890+0+0 (secure 0 0 0) 0x7f795c0a9d80 con 0x7f796c0b1ab0 2024-09-13T17:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.783+0000 7f7963e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 0x7f794806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.784+0000 7f7963e00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 0x7f794806d0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f796c0cd560 tx=0x7f7954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:01.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.787+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f795c074f20 con 0x7f796c0b1ab0 2024-09-13T17:09:01.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:01.965+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f796c0cd2b0 con 0x7f796c0b1ab0 2024-09-13T17:09:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.591+0000 7f7962000640 1 -- 172.21.15.28:0/2415712470 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f795c078540 con 0x7f796c0b1ab0 2024-09-13T17:09:02.592 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:09:02.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 msgr2=0x7f794806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 0x7f794806d0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f796c0cd560 tx=0x7f7954008040 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 msgr2=0x7f796c15da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f795c035040 tx=0x7f795c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 shutdown_connections 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f794806ac00 0x7f794806d0c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f796c15a7a0 0x7f796c1544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f796c0ce9d0 0x7f796c0c9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 --2- 172.21.15.28:0/2415712470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f796c0b1ab0 0x7f796c15da00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 >> 172.21.15.28:0/2415712470 conn(0x7f796c069e80 msgr2=0x7f796c0b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 shutdown_connections 2024-09-13T17:09:02.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.594+0000 7f7970a00640 1 -- 172.21.15.28:0/2415712470 wait complete. 2024-09-13T17:09:02.614 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:09:02.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.773+0000 7fef18800640 1 Processor -- start 2024-09-13T17:09:02.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.773+0000 7fef18800640 1 -- start start 2024-09-13T17:09:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef18800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fef10159970 0x7fef100b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef18800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef100b2630 0x7fef1015a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef18800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1015a4c0 con 0x7fef10159590 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef18800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef100b2ae0 con 0x7fef100b2630 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef18800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef10071390 con 0x7fef10159970 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef16800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef100b2630 0x7fef1015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef16800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fef100b2630 0x7fef1015a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48456/0 (socket says 172.21.15.28:48456) 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef16800640 1 -- 172.21.15.28:0/244237756 learned_addr learned my addr 172.21.15.28:0/244237756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef15400640 1 --2- 172.21.15.28:0/244237756 >> v2:172.21.15.189:3300/0 conn(0x7fef10159970 0x7fef100b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef16800640 1 -- 172.21.15.28:0/244237756 >> v2:172.21.15.189:3300/0 conn(0x7fef10159970 msgr2=0x7fef100b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.775+0000 7fef16800640 1 --2- 172.21.15.28:0/244237756 >> v2:172.21.15.189:3300/0 conn(0x7fef10159970 0x7fef100b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.776+0000 7fef16800640 1 -- 172.21.15.28:0/244237756 >> v1:172.21.15.189:6789/0 conn(0x7fef10159590 legacy=0x7fef10070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.776+0000 7fef16800640 1 -- 172.21.15.28:0/244237756 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10071580 con 0x7fef100b2630 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef16800640 1 --2- 172.21.15.28:0/244237756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef1015a950 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fef100cf5e0 tx=0x7fef0c02c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4f52e4c65acc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef14a00640 1 -- 172.21.15.28:0/244237756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c00b2e0 con 0x7fef100b2630 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef14a00640 1 -- 172.21.15.28:0/244237756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef0c00b440 con 0x7fef100b2630 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef14a00640 1 -- 172.21.15.28:0/244237756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c009820 con 0x7fef100b2630 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 -- 172.21.15.28:0/244237756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 msgr2=0x7fef1015a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 --2- 172.21.15.28:0/244237756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef1015a950 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fef100cf5e0 tx=0x7fef0c02c940 comp rx=0 tx=0).stop 2024-09-13T17:09:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 -- 172.21.15.28:0/244237756 shutdown_connections 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 --2- 172.21.15.28:0/244237756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef1015a950 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 --2- 172.21.15.28:0/244237756 >> v2:172.21.15.189:3300/0 conn(0x7fef10159970 0x7fef100b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.777+0000 7fef18800640 1 -- 172.21.15.28:0/244237756 >> 172.21.15.28:0/244237756 conn(0x7fef10069e80 msgr2=0x7fef1006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.778+0000 7fef18800640 1 -- 172.21.15.28:0/244237756 shutdown_connections 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.778+0000 7fef18800640 1 -- 172.21.15.28:0/244237756 wait complete. 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.779+0000 7fef18800640 1 Processor -- start 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.779+0000 7fef18800640 1 -- start start 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef10159590 0x7fef1015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 0x7fef101f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef101394f0 con 0x7fef100b2630 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1013f4b0 con 0x7fef10159590 2024-09-13T17:09:02.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef18800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef1015b000 con 0x7fef10159970 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.780+0000 7fef15e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:48468/0 (socket says 172.21.15.28:48468) 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 -- 172.21.15.28:0/3824539482 learned_addr learned my addr 172.21.15.28:0/3824539482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15400640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef10159590 0x7fef1015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef16800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 0x7fef101f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 -- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 msgr2=0x7fef101f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 0x7fef101f31b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 -- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef10159590 msgr2=0x7fef1015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef10159590 0x7fef1015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 -- 172.21.15.28:0/3824539482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c00bd40 con 0x7fef100b2630 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef16800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 0x7fef101f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.781+0000 7fef15e00640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c450 tx=0x7fef0400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.782+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef040052d0 con 0x7fef100b2630 2024-09-13T17:09:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.782+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef04005430 con 0x7fef100b2630 2024-09-13T17:09:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.782+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1015b2e0 con 0x7fef100b2630 2024-09-13T17:09:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.782+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04012870 con 0x7fef100b2630 2024-09-13T17:09:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.782+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1015b8a0 con 0x7fef100b2630 2024-09-13T17:09:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.783+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1006ce00 con 0x7fef100b2630 2024-09-13T17:09:02.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.784+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fef040129d0 con 0x7fef100b2630 2024-09-13T17:09:02.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.785+0000 7fef0a000640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 0x7feef406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:02.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.785+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5452+0+0 (secure 0 0 0) 0x7fef04088c30 con 0x7fef100b2630 2024-09-13T17:09:02.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.786+0000 7fef15400640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 0x7feef406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:02.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.786+0000 7fef15400640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 0x7feef406d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef10153bb0 tx=0x7fef00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:02.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.789+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fef04053fa0 con 0x7fef100b2630 2024-09-13T17:09:02.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:02.971+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fef1006cb20 con 0x7fef100b2630 2024-09-13T17:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.598+0000 7fef0a000640 1 -- 172.21.15.28:0/3824539482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fef040575c0 con 0x7fef100b2630 2024-09-13T17:09:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 msgr2=0x7feef406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 0x7feef406d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fef10153bb0 tx=0x7fef00008040 comp rx=0 tx=0).stop 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 msgr2=0x7fef10138fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fef0400c450 tx=0x7fef0400c910 comp rx=0 tx=0).stop 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 shutdown_connections 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feef406ac00 0x7feef406d0c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fef10159970 0x7fef101f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef10159590 0x7fef1015c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 --2- 172.21.15.28:0/3824539482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fef100b2630 0x7fef10138fb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.602+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 >> 172.21.15.28:0/3824539482 conn(0x7fef10069e80 msgr2=0x7fef1006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.603+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 shutdown_connections 2024-09-13T17:09:03.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.603+0000 7fef18800640 1 -- 172.21.15.28:0/3824539482 wait complete. 2024-09-13T17:09:03.624 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:09:03.624 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:09:03.625 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:09:03.625 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:09:03.625 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:09:03.625 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:09:03.626 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:09:03.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:09:03.626 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:09:03.627 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:09:03.627 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:09:03.627 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:09:03.627 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:09:03.628 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:09:03.628 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:09:03.628 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:09:03.641 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:09:03.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:09:03.643 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:09:03.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:09:03.645 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:09:03.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:09:03.647 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:09:03.647 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:09:03.686 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:09:03.687 DEBUG:teuthology.parallel:result is None 2024-09-13T17:09:03.689 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:09:03.690 DEBUG:teuthology.parallel:result is None 2024-09-13T17:09:03.690 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:09:03.691 DEBUG:teuthology.parallel:result is None 2024-09-13T17:09:03.694 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:09:03.694 DEBUG:teuthology.parallel:result is None 2024-09-13T17:09:03.694 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:09:03.821 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:09:03.821 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:09:03.825 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:09:03.825 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:09:03.827 INFO:tasks.ceph.mds.b.smithi189.stderr:2024-09-13T17:09:03.825+0000 7fb2dd400640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-13T17:09:03.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.864+0000 7fe012600640 1 Processor -- start 2024-09-13T17:09:03.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.864+0000 7fe012600640 1 -- start start 2024-09-13T17:09:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe00c146f20 0x7fe00c06c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe00c0b1660 0x7fe00c0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0ce3e0 con 0x7fe00c146f20 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c0b2590 con 0x7fe00c0b1660 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.866+0000 7fe012600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0b1fe0 con 0x7fe00c161eb0 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.867+0000 7fe00be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.867+0000 7fe00be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:46318/0 (socket says 172.21.15.28:46318) 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.867+0000 7fe00be00640 1 -- 172.21.15.28:0/4291903214 learned_addr learned my addr 172.21.15.28:0/4291903214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:03.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.867+0000 7fe011c00640 1 --2- 172.21.15.28:0/4291903214 >> v2:172.21.15.189:3300/0 conn(0x7fe00c0b1660 0x7fe00c0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.868+0000 7fe00be00640 1 -- 172.21.15.28:0/4291903214 >> v2:172.21.15.189:3300/0 conn(0x7fe00c0b1660 msgr2=0x7fe00c0b1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.868+0000 7fe00be00640 1 --2- 172.21.15.28:0/4291903214 >> v2:172.21.15.189:3300/0 conn(0x7fe00c0b1660 0x7fe00c0b1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.868+0000 7fe00be00640 1 -- 172.21.15.28:0/4291903214 >> v2:172.21.15.28:3300/0 conn(0x7fe00c146f20 msgr2=0x7fe00c06c370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.868+0000 7fe00be00640 1 --2- 172.21.15.28:0/4291903214 >> v2:172.21.15.28:3300/0 conn(0x7fe00c146f20 0x7fe00c06c370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.868+0000 7fe00be00640 1 -- 172.21.15.28:0/4291903214 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c0b22b0 con 0x7fe00c161eb0 2024-09-13T17:09:03.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.869+0000 7fe00be00640 1 --2- 172.21.15.28:0/4291903214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0b3d00 tx=0x7fdff802a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e06483544c44e5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.872+0000 7fe00b400640 1 -- 172.21.15.28:0/4291903214 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8031600 con 0x7fe00c161eb0 2024-09-13T17:09:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.872+0000 7fe00b400640 1 -- 172.21.15.28:0/4291903214 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdff802d050 con 0x7fe00c161eb0 2024-09-13T17:09:03.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.872+0000 7fe00b400640 1 -- 172.21.15.28:0/4291903214 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdff8031980 con 0x7fe00c161eb0 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 -- 172.21.15.28:0/4291903214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 msgr2=0x7fe00c06c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 --2- 172.21.15.28:0/4291903214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe00c0b3d00 tx=0x7fdff802a6c0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 -- 172.21.15.28:0/4291903214 shutdown_connections 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 --2- 172.21.15.28:0/4291903214 >> v2:172.21.15.189:3300/0 conn(0x7fe00c0b1660 0x7fe00c0b1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 --2- 172.21.15.28:0/4291903214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c161eb0 0x7fe00c06c8b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 --2- 172.21.15.28:0/4291903214 >> v2:172.21.15.28:3300/0 conn(0x7fe00c146f20 0x7fe00c06c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 -- 172.21.15.28:0/4291903214 >> 172.21.15.28:0/4291903214 conn(0x7fe00c073470 msgr2=0x7fe00c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.873+0000 7fe012600640 1 -- 172.21.15.28:0/4291903214 shutdown_connections 2024-09-13T17:09:03.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.874+0000 7fe012600640 1 -- 172.21.15.28:0/4291903214 wait complete. 2024-09-13T17:09:03.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.876+0000 7fe012600640 1 Processor -- start 2024-09-13T17:09:03.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.876+0000 7fe012600640 1 -- start start 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 0x7fe00c145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 0x7fe00c145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00c161eb0 0x7fe00c15a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c160be0 con 0x7fe00c146f20 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c15fe20 con 0x7fe00c161eb0 2024-09-13T17:09:03.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe012600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c160100 con 0x7fe00c0b1660 2024-09-13T17:09:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe00be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 0x7fe00c145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe011200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 0x7fe00c145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe00be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 0x7fe00c145d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52928/0 (socket says 172.21.15.28:52928) 2024-09-13T17:09:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe00be00640 1 -- 172.21.15.28:0/3243320525 learned_addr learned my addr 172.21.15.28:0/3243320525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:03.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.877+0000 7fe011c00640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00c161eb0 0x7fe00c15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 -- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00c161eb0 msgr2=0x7fe00c15a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00c161eb0 0x7fe00c15a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 -- 172.21.15.28:0/3243320525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 msgr2=0x7fe00c145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 0x7fe00c145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 -- 172.21.15.28:0/3243320525 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdff8009b30 con 0x7fe00c0b1660 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe011200640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 0x7fe00c145370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe004009fd0 tx=0x7fe00400b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004011830 con 0x7fe00c0b1660 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0040119b0 con 0x7fe00c0b1660 2024-09-13T17:09:03.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe004011da0 con 0x7fe00c0b1660 2024-09-13T17:09:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c1603e0 con 0x7fe00c0b1660 2024-09-13T17:09:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.878+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c13ef20 con 0x7fe00c0b1660 2024-09-13T17:09:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.880+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c0b1aa0 con 0x7fe00c0b1660 2024-09-13T17:09:03.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.880+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe00401a400 con 0x7fe00c0b1660 2024-09-13T17:09:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.881+0000 7fe008c00640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 0x7fdfec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.881+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe004089180 con 0x7fe00c0b1660 2024-09-13T17:09:03.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.881+0000 7fe00be00640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 0x7fdfec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:03.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.882+0000 7fe00be00640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 0x7fdfec06d0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe00c164120 tx=0x7fdff8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:03.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:03.886+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe0040545e0 con 0x7fe00c0b1660 2024-09-13T17:09:04.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.078+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe00c160240 con 0x7fe00c0b1660 2024-09-13T17:09:04.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.879+0000 7fe008c00640 1 -- 172.21.15.28:0/3243320525 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe00c160240 con 0x7fe00c0b1660 2024-09-13T17:09:04.880 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 msgr2=0x7fdfec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 0x7fdfec06d0c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe00c164120 tx=0x7fdff8009000 comp rx=0 tx=0).stop 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 msgr2=0x7fe00c145370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 0x7fe00c145370 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe004009fd0 tx=0x7fe00400b870 comp rx=0 tx=0).stop 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 shutdown_connections 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdfec06ac00 0x7fdfec06d0c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:04.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe00c161eb0 0x7fe00c15a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe00c146f20 0x7fe00c145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 --2- 172.21.15.28:0/3243320525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe00c0b1660 0x7fe00c145370 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 >> 172.21.15.28:0/3243320525 conn(0x7fe00c073470 msgr2=0x7fe00c070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 shutdown_connections 2024-09-13T17:09:04.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:04.882+0000 7fe012600640 1 -- 172.21.15.28:0/3243320525 wait complete. 2024-09-13T17:09:04.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:09:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.062+0000 7fbe07e00640 1 Processor -- start 2024-09-13T17:09:05.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.062+0000 7fbe07e00640 1 -- start start 2024-09-13T17:09:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe080b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe080b29f0 0x7fbe080b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe080c3850 con 0x7fbe08159820 2024-09-13T17:09:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0815a930 con 0x7fbe080b29f0 2024-09-13T17:09:05.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe08159d20 con 0x7fbe08159440 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe06a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe080b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe080b29f0 0x7fbe080b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbe080b29f0 0x7fbe080b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48004/0 (socket says 172.21.15.28:48004) 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe06a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe080b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:46332/0 (socket says 172.21.15.28:46332) 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbe07400640 1 -- 172.21.15.28:0/469018092 learned_addr learned my addr 172.21.15.28:0/469018092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.064+0000 7fbdffe00640 1 --1- 172.21.15.28:0/469018092 >> v1:172.21.15.189:6789/0 conn(0x7fbe08159820 0x7fbe080b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:37102/0 (socket says 172.21.15.28:37102) 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.065+0000 7fbe07400640 1 -- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 msgr2=0x7fbe080b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:05.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.065+0000 7fbe07400640 1 --2- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe080b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.065+0000 7fbe07400640 1 -- 172.21.15.28:0/469018092 >> v1:172.21.15.189:6789/0 conn(0x7fbe08159820 legacy=0x7fbe080b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.065+0000 7fbe07400640 1 -- 172.21.15.28:0/469018092 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0815a000 con 0x7fbe080b29f0 2024-09-13T17:09:05.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.065+0000 7fbe06000640 1 -- 172.21.15.28:0/469018092 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873743188 0 0) 0x7fbe080c3850 con 0x7fbe08159820 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.066+0000 7fbe07400640 1 --2- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe080b0980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009e50 tx=0x7fbe0002a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=568b0231a64f97d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.066+0000 7fbe06000640 1 -- 172.21.15.28:0/469018092 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00031600 con 0x7fbe080b29f0 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.066+0000 7fbe06000640 1 -- 172.21.15.28:0/469018092 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0002d050 con 0x7fbe080b29f0 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.066+0000 7fbe06000640 1 -- 172.21.15.28:0/469018092 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00031980 con 0x7fbe080b29f0 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 -- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 msgr2=0x7fbe080b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 --2- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe080b0980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009e50 tx=0x7fbe0002a7a0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 -- 172.21.15.28:0/469018092 shutdown_connections 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 --2- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe080b0980 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 --2- 172.21.15.28:0/469018092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe080b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 -- 172.21.15.28:0/469018092 >> 172.21.15.28:0/469018092 conn(0x7fbe08072e40 msgr2=0x7fbe08073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:05.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 -- 172.21.15.28:0/469018092 shutdown_connections 2024-09-13T17:09:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.067+0000 7fbe07e00640 1 -- 172.21.15.28:0/469018092 wait complete. 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.069+0000 7fbe07e00640 1 Processor -- start 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.069+0000 7fbe07e00640 1 -- start start 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe08150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe081513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0814b140 con 0x7fbe08159820 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0814bb70 con 0x7fbe080b29f0 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0814be50 con 0x7fbe08159440 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52942/0 (socket says 172.21.15.28:52942) 2024-09-13T17:09:05.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 -- 172.21.15.28:0/3694771044 learned_addr learned my addr 172.21.15.28:0/3694771044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe06a00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe08150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbdffe00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe081513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 -- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 msgr2=0x7fbe081513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe081513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 -- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 msgr2=0x7fbe08150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe08150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.070+0000 7fbe07400640 1 -- 172.21.15.28:0/3694771044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00009b30 con 0x7fbe08159820 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbe07400640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009f70 tx=0x7fbe00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00003b90 con 0x7fbe08159820 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe00003cf0 con 0x7fbe08159820 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe00004270 con 0x7fbe08159820 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0814c130 con 0x7fbe08159820 2024-09-13T17:09:05.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.071+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0814c6f0 con 0x7fbe08159820 2024-09-13T17:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.072+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe080b22e0 con 0x7fbe08159820 2024-09-13T17:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.073+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbe000043d0 con 0x7fbe08159820 2024-09-13T17:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.074+0000 7fbdfea00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 0x7fbde406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.074+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbe000a97b0 con 0x7fbe08159820 2024-09-13T17:09:05.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.074+0000 7fbe06a00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 0x7fbde406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.075+0000 7fbe06a00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 0x7fbde406d0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009ad0 tx=0x7fbdf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.078+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbe00074d90 con 0x7fbe08159820 2024-09-13T17:09:05.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.323+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbe0814c8b0 con 0x7fbe08159820 2024-09-13T17:09:05.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.323+0000 7fbdfea00640 1 -- 172.21.15.28:0/3694771044 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fbe00009510 con 0x7fbe08159820 2024-09-13T17:09:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:09:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:09:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:09:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:09:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:09:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:09:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mds", 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRmKLdlBhAAWm2b9zUylTpSMTiq+Rs2fw==", 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mds" 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-mgr", 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRmf8FlBhAAxdPkKE2kv5fNzT7LLPkQYw==", 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-osd", 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRmA8plBhAAecc+yWDMOI7W8D/46kg2FA==", 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-osd" 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd", 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRmTdJlBhAAtiNCM2893h5A5V/wu23dOw==", 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRm7NplBhAAhkxPYHdU1RvrNFXyqkz42w==", 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.bootstrap-rgw", 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBiceRmLeNlBhAAwFjn3AviUOpt08g/2egVwg==", 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:09:05.342 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:09:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 msgr2=0x7fbde406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 0x7fbde406d0c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8009ad0 tx=0x7fbdf8008040 comp rx=0 tx=0).stop 2024-09-13T17:09:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 msgr2=0x7fbe081563f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbe00009f70 tx=0x7fbe00004a40 comp rx=0 tx=0).stop 2024-09-13T17:09:05.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 shutdown_connections 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbde406ac00 0x7fbde406d0c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe08159820 0x7fbe081563f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe08159440 0x7fbe081513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 --2- 172.21.15.28:0/3694771044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe080b29f0 0x7fbe08150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 >> 172.21.15.28:0/3694771044 conn(0x7fbe08072e40 msgr2=0x7fbe0815d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 shutdown_connections 2024-09-13T17:09:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.326+0000 7fbe07e00640 1 -- 172.21.15.28:0/3694771044 wait complete. 2024-09-13T17:09:05.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-13T17:09:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.500+0000 7f060d200640 1 Processor -- start 2024-09-13T17:09:05.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.500+0000 7f060d200640 1 -- start start 2024-09-13T17:09:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.501+0000 7f060d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.501+0000 7f060d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f060806c540 0x7f060816a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.501+0000 7f060d200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06081610a0 con 0x7f060806d070 2024-09-13T17:09:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.501+0000 7f060d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06080b2620 con 0x7f060806c540 2024-09-13T17:09:05.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.501+0000 7f060d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080b2900 con 0x7f06080b17e0 2024-09-13T17:09:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f060806c540 0x7f060816a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0605e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f060806c540 0x7f060816a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48010/0 (socket says 172.21.15.28:48010) 2024-09-13T17:09:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 -- 172.21.15.28:0/1794987029 learned_addr learned my addr 172.21.15.28:0/1794987029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:05.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 -- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 msgr2=0x7f06080b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 --2- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 -- 172.21.15.28:0/1794987029 >> v1:172.21.15.189:6789/0 conn(0x7f060806d070 legacy=0x7f06080b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0606800640 1 -- 172.21.15.28:0/1794987029 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06080b2be0 con 0x7f060806c540 2024-09-13T17:09:05.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.502+0000 7f0604a00640 1 -- 172.21.15.28:0/1794987029 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286776438 0 0) 0x7f06081610a0 con 0x7f060806d070 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.503+0000 7f0606800640 1 --2- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f060816a7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0600009e50 tx=0x7f060002a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a944419edd9728 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.503+0000 7f0604a00640 1 -- 172.21.15.28:0/1794987029 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600031600 con 0x7f060806c540 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.503+0000 7f0604a00640 1 -- 172.21.15.28:0/1794987029 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060002d050 con 0x7f060806c540 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.503+0000 7f0604a00640 1 -- 172.21.15.28:0/1794987029 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0600031980 con 0x7f060806c540 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 -- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 msgr2=0x7f060816a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 --2- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f060816a7e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0600009e50 tx=0x7f060002a730 comp rx=0 tx=0).stop 2024-09-13T17:09:05.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 -- 172.21.15.28:0/1794987029 shutdown_connections 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 --2- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f060816a7e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 --2- 172.21.15.28:0/1794987029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 -- 172.21.15.28:0/1794987029 >> 172.21.15.28:0/1794987029 conn(0x7f06080b0270 msgr2=0x7f06080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 -- 172.21.15.28:0/1794987029 shutdown_connections 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.504+0000 7f060d200640 1 -- 172.21.15.28:0/1794987029 wait complete. 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.506+0000 7f060d200640 1 Processor -- start 2024-09-13T17:09:05.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.506+0000 7f060d200640 1 -- start start 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f06080ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080a98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080ab230 con 0x7f060806d070 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06080aada0 con 0x7f060806c540 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f060d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06080aa190 con 0x7f06080b17e0 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52952/0 (socket says 172.21.15.28:52952) 2024-09-13T17:09:05.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 -- 172.21.15.28:0/2397954305 learned_addr learned my addr 172.21.15.28:0/2397954305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605e00640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f06080ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0606800640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080a98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 -- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 msgr2=0x7f06080a98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080a98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 -- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 msgr2=0x7f06080ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f06080ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.507+0000 7f0605400640 1 -- 172.21.15.28:0/2397954305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0600009b30 con 0x7f060806d070 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f0605400640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c3c0 tx=0x7f05f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f40035f0 con 0x7f060806d070 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f4003750 con 0x7f060806d070 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06080aa470 con 0x7f060806d070 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4003b00 con 0x7f060806d070 2024-09-13T17:09:05.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.508+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06080aaa30 con 0x7f060806d070 2024-09-13T17:09:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.509+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06080b3480 con 0x7f060806d070 2024-09-13T17:09:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.510+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f05f4003c60 con 0x7f060806d070 2024-09-13T17:09:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.511+0000 7f05fe000640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 0x7f05e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.512+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f05f40885f0 con 0x7f060806d070 2024-09-13T17:09:05.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.512+0000 7f0605e00640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 0x7f05e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.513+0000 7f0605e00640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 0x7f05e806d0c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b40 tx=0x7f05f8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.516+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f05f4053ab0 con 0x7f060806d070 2024-09-13T17:09:05.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.762+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f0608071440 con 0x7f060806d070 2024-09-13T17:09:05.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.766+0000 7f05fe000640 1 -- 172.21.15.28:0/2397954305 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f05f40570d0 con 0x7f060806d070 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.768+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 msgr2=0x7f05e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.768+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 0x7f05e806d0c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f05f8009b40 tx=0x7f05f8009190 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.768+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 msgr2=0x7f06081545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.768+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f05f400c3c0 tx=0x7f05f400c880 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 shutdown_connections 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05e806ac00 0x7f05e806d0c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06080b17e0 0x7f06080a98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f060806d070 0x7f06081545a0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 --2- 172.21.15.28:0/2397954305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f060806c540 0x7f06080ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 >> 172.21.15.28:0/2397954305 conn(0x7f06080b0270 msgr2=0x7f06080ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 shutdown_connections 2024-09-13T17:09:05.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.769+0000 7f060d200640 1 -- 172.21.15.28:0/2397954305 wait complete. 2024-09-13T17:09:05.785 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-13T17:09:05.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.941+0000 7fd677e00640 1 Processor -- start 2024-09-13T17:09:05.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.941+0000 7fd677e00640 1 -- start start 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6780cd7c0 0x7fd6780c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67815ef70 con 0x7fd6780c3ba0 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6780c4320 con 0x7fd678143f70 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67815e1b0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd66fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6780cd7c0 0x7fd6780c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6780c3ba0 0x7fd678070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33796/0 (socket says 172.21.15.28:33796) 2024-09-13T17:09:05.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.943+0000 7fd677400640 1 -- 172.21.15.28:0/2655337036 learned_addr learned my addr 172.21.15.28:0/2655337036 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:05.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.944+0000 7fd66fe00640 1 -- 172.21.15.28:0/2655337036 >> v1:172.21.15.189:6789/0 conn(0x7fd678143f70 legacy=0x7fd6780cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.944+0000 7fd66fe00640 1 -- 172.21.15.28:0/2655337036 >> v1:172.21.15.28:6789/0 conn(0x7fd6780c3ba0 legacy=0x7fd678070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.944+0000 7fd66fe00640 1 -- 172.21.15.28:0/2655337036 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67815e490 con 0x7fd6780cd7c0 2024-09-13T17:09:05.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.944+0000 7fd676000640 1 -- 172.21.15.28:0/2655337036 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583016675 0 0) 0x7fd67815ef70 con 0x7fd6780c3ba0 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd66fe00640 1 --2- 172.21.15.28:0/2655337036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd6780c3660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd6600097a0 tx=0x7fd66002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ddf8cc747aa9187 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd676000640 1 -- 172.21.15.28:0/2655337036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660032440 con 0x7fd6780cd7c0 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd676000640 1 -- 172.21.15.28:0/2655337036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6600325a0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd676000640 1 -- 172.21.15.28:0/2655337036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660032930 con 0x7fd6780cd7c0 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd677e00640 1 -- 172.21.15.28:0/2655337036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 msgr2=0x7fd6780c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd677e00640 1 --2- 172.21.15.28:0/2655337036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd6780c3660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd6600097a0 tx=0x7fd66002a230 comp rx=0 tx=0).stop 2024-09-13T17:09:05.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd677e00640 1 -- 172.21.15.28:0/2655337036 shutdown_connections 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd677e00640 1 --2- 172.21.15.28:0/2655337036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd6780c3660 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.945+0000 7fd677e00640 1 -- 172.21.15.28:0/2655337036 >> 172.21.15.28:0/2655337036 conn(0x7fd678069e80 msgr2=0x7fd67806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.946+0000 7fd677e00640 1 -- 172.21.15.28:0/2655337036 shutdown_connections 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.946+0000 7fd677e00640 1 -- 172.21.15.28:0/2655337036 wait complete. 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.947+0000 7fd677e00640 1 Processor -- start 2024-09-13T17:09:05.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.947+0000 7fd677e00640 1 -- start start 2024-09-13T17:09:05.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 0x7fd678142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd678143f70 0x7fd67815afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6781410a0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678140420 con 0x7fd6780c3ba0 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd677e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd678140700 con 0x7fd678143f70 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd66fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd66fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52966/0 (socket says 172.21.15.28:52966) 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.948+0000 7fd66fe00640 1 -- 172.21.15.28:0/96014069 learned_addr learned my addr 172.21.15.28:0/96014069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd676a00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 0x7fd678142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd677400640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd678143f70 0x7fd67815afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 -- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd678143f70 msgr2=0x7fd67815afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd678143f70 0x7fd67815afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 -- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 msgr2=0x7fd678142820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 0x7fd678142820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 -- 172.21.15.28:0/96014069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd660009400 con 0x7fd6780cd7c0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd676a00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 0x7fd678142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66fe00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd6600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd660030b80 con 0x7fd6780cd7c0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd660030ce0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66003d680 con 0x7fd6780cd7c0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.949+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6781409e0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.950+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd678142d60 con 0x7fd6780cd7c0 2024-09-13T17:09:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.951+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6780c3660 con 0x7fd6780cd7c0 2024-09-13T17:09:05.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.951+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd660039040 con 0x7fd6780cd7c0 2024-09-13T17:09:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.953+0000 7fd66ea00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 0x7fd65406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.953+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd6600a9bd0 con 0x7fd6780cd7c0 2024-09-13T17:09:05.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.953+0000 7fd676a00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 0x7fd65406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:05.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.954+0000 7fd676a00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 0x7fd65406d0c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd668007fd0 tx=0x7fd668009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:05.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:05.957+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6600751a0 con 0x7fd6780cd7c0 2024-09-13T17:09:06.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.200+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fd6780722d0 con 0x7fd6780cd7c0 2024-09-13T17:09:06.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.204+0000 7fd66ea00640 1 -- 172.21.15.28:0/96014069 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7fd6600787c0 con 0x7fd6780cd7c0 2024-09-13T17:09:06.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 msgr2=0x7fd65406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 0x7fd65406d0c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd668007fd0 tx=0x7fd668009040 comp rx=0 tx=0).stop 2024-09-13T17:09:06.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 msgr2=0x7fd67815aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd660000c00 tx=0x7fd6600046c0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 shutdown_connections 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd65406ac00 0x7fd65406d0c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd678143f70 0x7fd67815afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6780cd7c0 0x7fd67815aa60 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 --2- 172.21.15.28:0/96014069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6780c3ba0 0x7fd678142820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 >> 172.21.15.28:0/96014069 conn(0x7fd678069e80 msgr2=0x7fd6780b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 shutdown_connections 2024-09-13T17:09:06.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.206+0000 7fd677e00640 1 -- 172.21.15.28:0/96014069 wait complete. 2024-09-13T17:09:06.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-13T17:09:06.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.379+0000 7f968f600640 1 Processor -- start 2024-09-13T17:09:06.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.379+0000 7f968f600640 1 -- start start 2024-09-13T17:09:06.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968f600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f968815a780 0x7f9688070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968f600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968806c660 con 0x7f968806c250 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968f600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968806c8b0 con 0x7f968815ab60 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968f600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968806cb90 con 0x7f968815a780 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f968815a780 0x7f9688070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968d600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f968806c250 0x7f96880cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33812/0 (socket says 172.21.15.28:33812) 2024-09-13T17:09:06.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968d600640 1 -- 172.21.15.28:0/2233958311 learned_addr learned my addr 172.21.15.28:0/2233958311 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:06.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968cc00640 1 -- 172.21.15.28:0/2233958311 >> v1:172.21.15.189:6789/0 conn(0x7f968815ab60 legacy=0x7f968806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f968cc00640 1 -- 172.21.15.28:0/2233958311 >> v1:172.21.15.28:6789/0 conn(0x7f968806c250 legacy=0x7f96880cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.381+0000 7f9687e00640 1 -- 172.21.15.28:0/2233958311 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089444052 0 0) 0x7f968806c660 con 0x7f968806c250 2024-09-13T17:09:06.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.382+0000 7f968cc00640 1 -- 172.21.15.28:0/2233958311 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96880cc980 con 0x7f968815a780 2024-09-13T17:09:06.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f968cc00640 1 --2- 172.21.15.28:0/2233958311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f9688070d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f96780097a0 tx=0x7f967802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e3a82abcdc98308 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f9687e00640 1 -- 172.21.15.28:0/2233958311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678032440 con 0x7f968815a780 2024-09-13T17:09:06.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f9687e00640 1 -- 172.21.15.28:0/2233958311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96780325a0 con 0x7f968815a780 2024-09-13T17:09:06.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f9687e00640 1 -- 172.21.15.28:0/2233958311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678032930 con 0x7f968815a780 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f968f600640 1 -- 172.21.15.28:0/2233958311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 msgr2=0x7f9688070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.383+0000 7f968f600640 1 --2- 172.21.15.28:0/2233958311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f9688070d80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f96780097a0 tx=0x7f967802a230 comp rx=0 tx=0).stop 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.384+0000 7f968f600640 1 -- 172.21.15.28:0/2233958311 shutdown_connections 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.384+0000 7f968f600640 1 --2- 172.21.15.28:0/2233958311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f9688070d80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.384+0000 7f968f600640 1 -- 172.21.15.28:0/2233958311 >> 172.21.15.28:0/2233958311 conn(0x7f96880b0270 msgr2=0x7f96880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.384+0000 7f968f600640 1 -- 172.21.15.28:0/2233958311 shutdown_connections 2024-09-13T17:09:06.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.384+0000 7f968f600640 1 -- 172.21.15.28:0/2233958311 wait complete. 2024-09-13T17:09:06.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.385+0000 7f968f600640 1 Processor -- start 2024-09-13T17:09:06.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.385+0000 7f968f600640 1 -- start start 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f968806c250 0x7f9688155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f968815ab60 0x7f96880c3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96880caa60 con 0x7f968815a780 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96881561c0 con 0x7f968815ab60 2024-09-13T17:09:06.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f968f600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880aaff0 con 0x7f968806c250 2024-09-13T17:09:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f967fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f967fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52982/0 (socket says 172.21.15.28:52982) 2024-09-13T17:09:06.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.386+0000 7f967fe00640 1 -- 172.21.15.28:0/185996300 learned_addr learned my addr 172.21.15.28:0/185996300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f968cc00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f968806c250 0x7f9688155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f968d600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f968815ab60 0x7f96880c3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 -- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f968806c250 msgr2=0x7f9688155c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f968806c250 0x7f9688155c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 -- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f968815ab60 msgr2=0x7f96880c3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f968815ab60 0x7f96880c3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 -- 172.21.15.28:0/185996300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9678009400 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f967fe00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f967000c400 tx=0x7f967000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96700052d0 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9670005430 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9670003830 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.387+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96880ab2d0 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.388+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96880ab890 con 0x7f968815a780 2024-09-13T17:09:06.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.389+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96880abab0 con 0x7f968815a780 2024-09-13T17:09:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.390+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9670003990 con 0x7f968815a780 2024-09-13T17:09:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.390+0000 7f9685600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 0x7f966406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.391+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9670088580 con 0x7f968815a780 2024-09-13T17:09:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.391+0000 7f968cc00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 0x7f966406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.392+0000 7f968cc00640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 0x7f966406d0c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9678034000 tx=0x7f967802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.395+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9670053ab0 con 0x7f968815a780 2024-09-13T17:09:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.638+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f9688071680 con 0x7f968815a780 2024-09-13T17:09:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.641+0000 7f9685600640 1 -- 172.21.15.28:0/185996300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f96700570d0 con 0x7f968815a780 2024-09-13T17:09:06.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 msgr2=0x7f966406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 0x7f966406d0c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9678034000 tx=0x7f967802e000 comp rx=0 tx=0).stop 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 msgr2=0x7f96880c34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f967000c400 tx=0x7f967000c8c0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 shutdown_connections 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f966406ac00 0x7f966406d0c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f968815ab60 0x7f96880c3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f968815a780 0x7f96880c34e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 --2- 172.21.15.28:0/185996300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f968806c250 0x7f9688155c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 >> 172.21.15.28:0/185996300 conn(0x7f96880b0270 msgr2=0x7f96880af590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:06.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 shutdown_connections 2024-09-13T17:09:06.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.643+0000 7f968f600640 1 -- 172.21.15.28:0/185996300 wait complete. 2024-09-13T17:09:06.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-13T17:09:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.815+0000 7fc819a00640 1 Processor -- start 2024-09-13T17:09:06.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.815+0000 7fc819a00640 1 -- start start 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc819a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc81415cb30 0x7fc8140cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc819a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 0x7fc8140cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc819a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8140cfbd0 con 0x7fc8140cf7a0 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc819a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8140d0bf0 con 0x7fc81415cb30 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc819a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8140d0ed0 con 0x7fc8140d0480 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc813e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc81415cb30 0x7fc8140cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc813e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc81415cb30 0x7fc8140cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52998/0 (socket says 172.21.15.28:52998) 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.816+0000 7fc813e00640 1 -- 172.21.15.28:0/4238892111 learned_addr learned my addr 172.21.15.28:0/4238892111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc80be00640 1 --2- 172.21.15.28:0/4238892111 >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 0x7fc8140cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc813e00640 1 -- 172.21.15.28:0/4238892111 >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 msgr2=0x7fc8140cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc813e00640 1 --2- 172.21.15.28:0/4238892111 >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 0x7fc8140cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc813e00640 1 -- 172.21.15.28:0/4238892111 >> v1:172.21.15.189:6789/0 conn(0x7fc8140cf7a0 legacy=0x7fc814069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:06.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc813e00640 1 -- 172.21.15.28:0/4238892111 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8140d11b0 con 0x7fc81415cb30 2024-09-13T17:09:06.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.817+0000 7fc80be00640 1 --2- 172.21.15.28:0/4238892111 >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 0x7fc8140cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.818+0000 7fc813e00640 1 --2- 172.21.15.28:0/4238892111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc81415cb30 0x7fc8140cdc00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8140d2880 tx=0x7fc80402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd5d946a20085d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.818+0000 7fc813400640 1 -- 172.21.15.28:0/4238892111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804032440 con 0x7fc81415cb30 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.818+0000 7fc813400640 1 -- 172.21.15.28:0/4238892111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8040325a0 con 0x7fc81415cb30 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.818+0000 7fc813400640 1 -- 172.21.15.28:0/4238892111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc804032930 con 0x7fc81415cb30 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 -- 172.21.15.28:0/4238892111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc81415cb30 msgr2=0x7fc8140cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 --2- 172.21.15.28:0/4238892111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc81415cb30 0x7fc8140cdc00 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fc8140d2880 tx=0x7fc80402a230 comp rx=0 tx=0).stop 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 -- 172.21.15.28:0/4238892111 shutdown_connections 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 --2- 172.21.15.28:0/4238892111 >> v2:172.21.15.189:3300/0 conn(0x7fc8140d0480 0x7fc8140cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 --2- 172.21.15.28:0/4238892111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc81415cb30 0x7fc8140cdc00 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 -- 172.21.15.28:0/4238892111 >> 172.21.15.28:0/4238892111 conn(0x7fc814070ad0 msgr2=0x7fc8140b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 -- 172.21.15.28:0/4238892111 shutdown_connections 2024-09-13T17:09:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.819+0000 7fc819a00640 1 -- 172.21.15.28:0/4238892111 wait complete. 2024-09-13T17:09:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.820+0000 7fc819a00640 1 Processor -- start 2024-09-13T17:09:06.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.821+0000 7fc819a00640 1 -- start start 2024-09-13T17:09:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.821+0000 7fc819a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8140cf7a0 0x7fc8141f19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.821+0000 7fc819a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc819a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc81415cb30 0x7fc8140a91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc819a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81415edd0 con 0x7fc8140d0480 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc819a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81415f050 con 0x7fc81415cb30 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc819a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc81415f330 con 0x7fc8140cf7a0 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53006/0 (socket says 172.21.15.28:53006) 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 -- 172.21.15.28:0/2657075026 learned_addr learned my addr 172.21.15.28:0/2657075026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc819000640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc81415cb30 0x7fc8140a91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc813e00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8140cf7a0 0x7fc8141f19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 -- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8140cf7a0 msgr2=0x7fc8141f19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8140cf7a0 0x7fc8141f19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 -- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc81415cb30 msgr2=0x7fc8140a91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc81415cb30 0x7fc8140a91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.822+0000 7fc80be00640 1 -- 172.21.15.28:0/2657075026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8040094d0 con 0x7fc8140d0480 2024-09-13T17:09:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc80be00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e7e0 tx=0x7fc7fc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc004030 con 0x7fc8140d0480 2024-09-13T17:09:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7fc004190 con 0x7fc8140d0480 2024-09-13T17:09:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81415f610 con 0x7fc8140d0480 2024-09-13T17:09:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc004520 con 0x7fc8140d0480 2024-09-13T17:09:06.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.823+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc81415fbd0 con 0x7fc8140d0480 2024-09-13T17:09:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.824+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8140cf260 con 0x7fc8140d0480 2024-09-13T17:09:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.825+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7fc00c6d0 con 0x7fc8140d0480 2024-09-13T17:09:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.826+0000 7fc810c00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 0x7fc7f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.826+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc7fc088ce0 con 0x7fc8140d0480 2024-09-13T17:09:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.827+0000 7fc813e00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 0x7fc7f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:06.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.827+0000 7fc813e00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 0x7fc7f006d0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc804034000 tx=0x7fc80402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:06.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:06.831+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc7fc054220 con 0x7fc8140d0480 2024-09-13T17:09:07.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.073+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7fc8140d3550 con 0x7fc8140d0480 2024-09-13T17:09:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.077+0000 7fc810c00640 1 -- 172.21.15.28:0/2657075026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc7fc057840 con 0x7fc8140d0480 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.079+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 msgr2=0x7fc7f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.079+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 0x7fc7f006d0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc804034000 tx=0x7fc80402e000 comp rx=0 tx=0).stop 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.079+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 msgr2=0x7fc8140b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.079+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e7e0 tx=0x7fc7fc00eca0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 shutdown_connections 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7f006ac00 0x7fc7f006d0c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc81415cb30 0x7fc8140a91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8140d0480 0x7fc8140b1030 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 --2- 172.21.15.28:0/2657075026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8140cf7a0 0x7fc8141f19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 >> 172.21.15.28:0/2657075026 conn(0x7fc814070ad0 msgr2=0x7fc8140b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 shutdown_connections 2024-09-13T17:09:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.080+0000 7fc819a00640 1 -- 172.21.15.28:0/2657075026 wait complete. 2024-09-13T17:09:07.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-13T17:09:07.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.252+0000 7efe24600640 1 Processor -- start 2024-09-13T17:09:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.252+0000 7efe24600640 1 -- start start 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.253+0000 7efe24600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe1c159770 0x7efe1c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.253+0000 7efe24600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 0x7efe1c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.253+0000 7efe24600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c06f920 con 0x7efe1c0add00 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.253+0000 7efe24600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1c15b290 con 0x7efe1c159770 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.253+0000 7efe24600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c15b570 con 0x7efe1c06f3f0 2024-09-13T17:09:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe1c159770 0x7efe1c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe1c159770 0x7efe1c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53010/0 (socket says 172.21.15.28:53010) 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 -- 172.21.15.28:0/2234028463 learned_addr learned my addr 172.21.15.28:0/2234028463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21200640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 0x7efe1c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 -- 172.21.15.28:0/2234028463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 msgr2=0x7efe1c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 0x7efe1c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 -- 172.21.15.28:0/2234028463 >> v1:172.21.15.189:6789/0 conn(0x7efe1c0add00 legacy=0x7efe1c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21c00640 1 -- 172.21.15.28:0/2234028463 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c15b850 con 0x7efe1c159770 2024-09-13T17:09:07.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.254+0000 7efe21200640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 0x7efe1c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:07.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.255+0000 7efe21c00640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c06fd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efe1000bb10 tx=0x7efe1002c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10943318a100386 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe1be00640 1 -- 172.21.15.28:0/2234028463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10032790 con 0x7efe1c159770 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe1be00640 1 -- 172.21.15.28:0/2234028463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe100328f0 con 0x7efe1c159770 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe1be00640 1 -- 172.21.15.28:0/2234028463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe10032c80 con 0x7efe1c159770 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 -- 172.21.15.28:0/2234028463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 msgr2=0x7efe1c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c06fd20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efe1000bb10 tx=0x7efe1002c890 comp rx=0 tx=0).stop 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 -- 172.21.15.28:0/2234028463 shutdown_connections 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c06f3f0 0x7efe1c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 --2- 172.21.15.28:0/2234028463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c06fd20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 -- 172.21.15.28:0/2234028463 >> 172.21.15.28:0/2234028463 conn(0x7efe1c0b0270 msgr2=0x7efe1c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 -- 172.21.15.28:0/2234028463 shutdown_connections 2024-09-13T17:09:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.256+0000 7efe24600640 1 -- 172.21.15.28:0/2234028463 wait complete. 2024-09-13T17:09:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.258+0000 7efe24600640 1 Processor -- start 2024-09-13T17:09:07.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.258+0000 7efe24600640 1 -- start start 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1c06f3f0 0x7efe1c145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c0add00 0x7efe1c0abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0a9260 con 0x7efe1c159770 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c1458b0 con 0x7efe1c06f3f0 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe24600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe1c0a85e0 con 0x7efe1c0add00 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe22600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe22600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53016/0 (socket says 172.21.15.28:53016) 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe22600640 1 -- 172.21.15.28:0/47108592 learned_addr learned my addr 172.21.15.28:0/47108592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe21c00640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1c06f3f0 0x7efe1c145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.259+0000 7efe21200640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c0add00 0x7efe1c0abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 -- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c0add00 msgr2=0x7efe1c0abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c0add00 0x7efe1c0abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 -- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1c06f3f0 msgr2=0x7efe1c145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1c06f3f0 0x7efe1c145370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 -- 172.21.15.28:0/47108592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1000b770 con 0x7efe1c159770 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe22600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c7a0 tx=0x7efe1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14003b70 con 0x7efe1c159770 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14003cf0 con 0x7efe1c159770 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe14003ea0 con 0x7efe1c159770 2024-09-13T17:09:07.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.260+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c0a88c0 con 0x7efe1c159770 2024-09-13T17:09:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.261+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c0a8e80 con 0x7efe1c159770 2024-09-13T17:09:07.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.262+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1c06fd20 con 0x7efe1c159770 2024-09-13T17:09:07.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.263+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efe1400aa30 con 0x7efe1c159770 2024-09-13T17:09:07.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.263+0000 7efe19600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 0x7efdfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.264+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7efe14088cb0 con 0x7efe1c159770 2024-09-13T17:09:07.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.264+0000 7efe21c00640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 0x7efdfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.265+0000 7efe21c00640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 0x7efdfc06d0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe10037000 tx=0x7efe1003e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.268+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efe14054280 con 0x7efe1c159770 2024-09-13T17:09:07.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.512+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7efe1c06d080 con 0x7efe1c159770 2024-09-13T17:09:07.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.516+0000 7efe19600640 1 -- 172.21.15.28:0/47108592 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7efe140578a0 con 0x7efe1c159770 2024-09-13T17:09:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 msgr2=0x7efdfc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 0x7efdfc06d0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe10037000 tx=0x7efe1003e040 comp rx=0 tx=0).stop 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 msgr2=0x7efe1c0ac130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efe1400c7a0 tx=0x7efe1400cc60 comp rx=0 tx=0).stop 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 shutdown_connections 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdfc06ac00 0x7efdfc06d0c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe1c159770 0x7efe1c0ac130 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe1c0add00 0x7efe1c0abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 --2- 172.21.15.28:0/47108592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe1c06f3f0 0x7efe1c145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.518+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 >> 172.21.15.28:0/47108592 conn(0x7efe1c0b0270 msgr2=0x7efe1c0ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:07.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.519+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 shutdown_connections 2024-09-13T17:09:07.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.519+0000 7efe24600640 1 -- 172.21.15.28:0/47108592 wait complete. 2024-09-13T17:09:07.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-13T17:09:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.694+0000 7f7ab3600640 1 Processor -- start 2024-09-13T17:09:07.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.694+0000 7f7ab3600640 1 -- start start 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7aac159770 0x7f7aac06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac06f3f0 0x7f7aac070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 0x7f7aac159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac06f920 con 0x7f7aac159770 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac15b290 con 0x7f7aac0add00 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac15b570 con 0x7f7aac06f3f0 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7aac159770 0x7f7aac06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7aac159770 0x7f7aac06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53024/0 (socket says 172.21.15.28:53024) 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.695+0000 7f7ab0c00640 1 -- 172.21.15.28:0/111702581 learned_addr learned my addr 172.21.15.28:0/111702581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab1600640 1 --2- 172.21.15.28:0/111702581 >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 0x7f7aac159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7aabe00640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac06f3f0 0x7f7aac070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab0c00640 1 -- 172.21.15.28:0/111702581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac06f3f0 msgr2=0x7f7aac070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac06f3f0 0x7f7aac070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab0c00640 1 -- 172.21.15.28:0/111702581 >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 msgr2=0x7f7aac159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/111702581 >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 0x7f7aac159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab0c00640 1 -- 172.21.15.28:0/111702581 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac15b850 con 0x7f7aac159770 2024-09-13T17:09:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.696+0000 7f7ab1600640 1 --2- 172.21.15.28:0/111702581 >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 0x7f7aac159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.697+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac159770 0x7f7aac06fd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0b0e90 tx=0x7f7aa002c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280552ca4ae7f8bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.697+0000 7f7aab400640 1 -- 172.21.15.28:0/111702581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0030050 con 0x7f7aac159770 2024-09-13T17:09:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.697+0000 7f7aab400640 1 -- 172.21.15.28:0/111702581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0032700 con 0x7f7aac159770 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.697+0000 7f7aab400640 1 -- 172.21.15.28:0/111702581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0032a90 con 0x7f7aac159770 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 -- 172.21.15.28:0/111702581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac159770 msgr2=0x7f7aac06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac159770 0x7f7aac06fd20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7aac0b0e90 tx=0x7f7aa002c800 comp rx=0 tx=0).stop 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 -- 172.21.15.28:0/111702581 shutdown_connections 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 --2- 172.21.15.28:0/111702581 >> v2:172.21.15.189:3300/0 conn(0x7f7aac0add00 0x7f7aac159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac06f3f0 0x7f7aac070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 --2- 172.21.15.28:0/111702581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac159770 0x7f7aac06fd20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 -- 172.21.15.28:0/111702581 >> 172.21.15.28:0/111702581 conn(0x7f7aac0b0270 msgr2=0x7f7aac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 -- 172.21.15.28:0/111702581 shutdown_connections 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.698+0000 7f7ab3600640 1 -- 172.21.15.28:0/111702581 wait complete. 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.700+0000 7f7ab3600640 1 Processor -- start 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.700+0000 7f7ab3600640 1 -- start start 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aac0add00 0x7f7aac0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac159770 0x7f7aac157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac14f590 con 0x7f7aac06f3f0 2024-09-13T17:09:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0ad360 con 0x7f7aac0add00 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0a9ae0 con 0x7f7aac159770 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53038/0 (socket says 172.21.15.28:53038) 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 -- 172.21.15.28:0/2217143168 learned_addr learned my addr 172.21.15.28:0/2217143168 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab1600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac159770 0x7f7aac157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7aabe00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aac0add00 0x7f7aac0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 -- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac159770 msgr2=0x7f7aac157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac159770 0x7f7aac157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 -- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aac0add00 msgr2=0x7f7aac0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aac0add00 0x7f7aac0ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.701+0000 7f7ab0c00640 1 -- 172.21.15.28:0/2217143168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa000b770 con 0x7f7aac06f3f0 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7ab0c00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7aa002c840 tx=0x7f7aa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa0034c30 con 0x7f7aac06f3f0 2024-09-13T17:09:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7aa0034db0 con 0x7f7aac06f3f0 2024-09-13T17:09:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7aa003d680 con 0x7f7aac06f3f0 2024-09-13T17:09:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac14ebf0 con 0x7f7aac06f3f0 2024-09-13T17:09:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.702+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac14f1b0 con 0x7f7aac06f3f0 2024-09-13T17:09:07.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.703+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac159b50 con 0x7f7aac06f3f0 2024-09-13T17:09:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.704+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7aa003d7e0 con 0x7f7aac06f3f0 2024-09-13T17:09:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.705+0000 7f7aa8c00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 0x7f7a8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.705+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7aa00a9f40 con 0x7f7aac06f3f0 2024-09-13T17:09:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.705+0000 7f7aabe00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 0x7f7a8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:07.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.706+0000 7f7aabe00640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 0x7f7a8c06d0c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7aac158510 tx=0x7f7a98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.709+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7aa0075460 con 0x7f7aac06f3f0 2024-09-13T17:09:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.956+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f7aac157430 con 0x7f7aac06f3f0 2024-09-13T17:09:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.960+0000 7f7aa8c00640 1 -- 172.21.15.28:0/2217143168 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f7aa0078a80 con 0x7f7aac06f3f0 2024-09-13T17:09:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 msgr2=0x7f7a8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 0x7f7a8c06d0c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7aac158510 tx=0x7f7a98008040 comp rx=0 tx=0).stop 2024-09-13T17:09:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 msgr2=0x7f7aac0a9490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7aa002c840 tx=0x7f7aa0004730 comp rx=0 tx=0).stop 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 shutdown_connections 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a8c06ac00 0x7f7a8c06d0c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7aac159770 0x7f7aac157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7aac0add00 0x7f7aac0ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 --2- 172.21.15.28:0/2217143168 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7aac06f3f0 0x7f7aac0a9490 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 >> 172.21.15.28:0/2217143168 conn(0x7f7aac0b0270 msgr2=0x7f7aac0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 shutdown_connections 2024-09-13T17:09:07.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:07.962+0000 7f7ab3600640 1 -- 172.21.15.28:0/2217143168 wait complete. 2024-09-13T17:09:07.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.133+0000 7fd461200640 1 Processor -- start 2024-09-13T17:09:08.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.133+0000 7fd461200640 1 -- start start 2024-09-13T17:09:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.134+0000 7fd461200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd45c0cabc0 0x7fd45c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.134+0000 7fd461200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd45c070cc0 0x7fd45c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.134+0000 7fd461200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd45c15e100 con 0x7fd45c15a540 2024-09-13T17:09:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.134+0000 7fd461200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c0adb00 con 0x7fd45c070cc0 2024-09-13T17:09:08.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.134+0000 7fd461200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45c071670 con 0x7fd45c0cabc0 2024-09-13T17:09:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd45c070cc0 0x7fd45c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd45c070cc0 0x7fd45c071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53044/0 (socket says 172.21.15.28:53044) 2024-09-13T17:09:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 -- 172.21.15.28:0/1886661579 learned_addr learned my addr 172.21.15.28:0/1886661579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd459400640 1 --2- 172.21.15.28:0/1886661579 >> v2:172.21.15.189:3300/0 conn(0x7fd45c0cabc0 0x7fd45c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 -- 172.21.15.28:0/1886661579 >> v2:172.21.15.189:3300/0 conn(0x7fd45c0cabc0 msgr2=0x7fd45c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 --2- 172.21.15.28:0/1886661579 >> v2:172.21.15.189:3300/0 conn(0x7fd45c0cabc0 0x7fd45c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 -- 172.21.15.28:0/1886661579 >> v1:172.21.15.189:6789/0 conn(0x7fd45c15a540 legacy=0x7fd45c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:08.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.135+0000 7fd45a800640 1 -- 172.21.15.28:0/1886661579 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c15e3e0 con 0x7fd45c070cc0 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.136+0000 7fd45a800640 1 --2- 172.21.15.28:0/1886661579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c070cc0 0x7fd45c071130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0ae250 tx=0x7fd45402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f79be7f25ad13c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.136+0000 7fd458a00640 1 -- 172.21.15.28:0/1886661579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454031600 con 0x7fd45c070cc0 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.136+0000 7fd458a00640 1 -- 172.21.15.28:0/1886661579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd45402d050 con 0x7fd45c070cc0 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.136+0000 7fd458a00640 1 -- 172.21.15.28:0/1886661579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd454031980 con 0x7fd45c070cc0 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 -- 172.21.15.28:0/1886661579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c070cc0 msgr2=0x7fd45c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 --2- 172.21.15.28:0/1886661579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c070cc0 0x7fd45c071130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd45c0ae250 tx=0x7fd45402a680 comp rx=0 tx=0).stop 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 -- 172.21.15.28:0/1886661579 shutdown_connections 2024-09-13T17:09:08.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 --2- 172.21.15.28:0/1886661579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c070cc0 0x7fd45c071130 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 --2- 172.21.15.28:0/1886661579 >> v2:172.21.15.189:3300/0 conn(0x7fd45c0cabc0 0x7fd45c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 -- 172.21.15.28:0/1886661579 >> 172.21.15.28:0/1886661579 conn(0x7fd45c069e80 msgr2=0x7fd45c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 -- 172.21.15.28:0/1886661579 shutdown_connections 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.137+0000 7fd461200640 1 -- 172.21.15.28:0/1886661579 wait complete. 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.139+0000 7fd461200640 1 Processor -- start 2024-09-13T17:09:08.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.139+0000 7fd461200640 1 -- start start 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 0x7fd45c14c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 0x7fd45c15d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c14e670 con 0x7fd45c0cabc0 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c14e950 con 0x7fd45c070cc0 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd461200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45c14ec30 con 0x7fd45c15a540 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53050/0 (socket says 172.21.15.28:53050) 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 -- 172.21.15.28:0/2113226743 learned_addr learned my addr 172.21.15.28:0/2113226743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:08.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459e00640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 0x7fd45c14c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd45a800640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 0x7fd45c15d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 -- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 msgr2=0x7fd45c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 0x7fd45c15d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 -- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 msgr2=0x7fd45c14c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 0x7fd45c14c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459400640 1 -- 172.21.15.28:0/2113226743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd454009b30 con 0x7fd45c0cabc0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd45a800640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 0x7fd45c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.140+0000 7fd459e00640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 0x7fd45c14c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd459400640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd44800d800 tx=0x7fd44800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44800b6c0 con 0x7fd45c0cabc0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd44800b820 con 0x7fd45c0cabc0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44800bbb0 con 0x7fd45c0cabc0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45c14ef10 con 0x7fd45c0cabc0 2024-09-13T17:09:08.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.141+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45c14cdf0 con 0x7fd45c0cabc0 2024-09-13T17:09:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.142+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45c0b1710 con 0x7fd45c0cabc0 2024-09-13T17:09:08.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.143+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd44800bd10 con 0x7fd45c0cabc0 2024-09-13T17:09:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.144+0000 7fd452000640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 0x7fd43c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.144+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd448089680 con 0x7fd45c0cabc0 2024-09-13T17:09:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.144+0000 7fd459e00640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 0x7fd43c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.145+0000 7fd459e00640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 0x7fd43c06d0c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd44c006fd0 tx=0x7fd44c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.148+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd448054a90 con 0x7fd45c0cabc0 2024-09-13T17:09:08.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.329+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd45c071130 con 0x7fd45c0cabc0 2024-09-13T17:09:08.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.330+0000 7fd452000640 1 -- 172.21.15.28:0/2113226743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fd4480580b0 con 0x7fd45c0cabc0 2024-09-13T17:09:08.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:08.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":30,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:09:04.874119+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:09:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 msgr2=0x7fd43c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 0x7fd43c06d0c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd44c006fd0 tx=0x7fd44c008040 comp rx=0 tx=0).stop 2024-09-13T17:09:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 msgr2=0x7fd45c15d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd44800d800 tx=0x7fd44800dcc0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 shutdown_connections 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd43c06ac00 0x7fd43c06d0c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd45c15a540 0x7fd45c15d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd45c0cabc0 0x7fd45c15d110 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 --2- 172.21.15.28:0/2113226743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd45c070cc0 0x7fd45c14c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 >> 172.21.15.28:0/2113226743 conn(0x7fd45c069e80 msgr2=0x7fd45c0ccae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 shutdown_connections 2024-09-13T17:09:08.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.332+0000 7fd461200640 1 -- 172.21.15.28:0/2113226743 wait complete. 2024-09-13T17:09:08.348 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:09:08.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:09:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.506+0000 7fbe85800640 1 Processor -- start 2024-09-13T17:09:08.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.506+0000 7fbe85800640 1 -- start start 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe85800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbe80070430 0x7fbe8015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe85800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 0x7fbe8015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe85800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe8015ab60 con 0x7fbe80159770 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe85800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8006f920 con 0x7fbe80070430 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe85800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8015b290 con 0x7fbe8006f3f0 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe7ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbe80159770 0x7fbe8006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33824/0 (socket says 172.21.15.28:33824) 2024-09-13T17:09:08.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.507+0000 7fbe7ea00640 1 -- 172.21.15.28:0/1244934359 learned_addr learned my addr 172.21.15.28:0/1244934359 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7e000640 1 --2- 172.21.15.28:0/1244934359 >> v2:172.21.15.28:3300/0 conn(0x7fbe80070430 0x7fbe8015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7f400640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 0x7fbe8015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7d600640 1 -- 172.21.15.28:0/1244934359 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083440690 0 0) 0x7fbe70001110 con 0x7fbe80159770 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7d600640 1 -- 172.21.15.28:0/1244934359 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe64003520 con 0x7fbe80159770 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7e000640 1 -- 172.21.15.28:0/1244934359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 msgr2=0x7fbe8015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7e000640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 0x7fbe8015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7e000640 1 -- 172.21.15.28:0/1244934359 >> v1:172.21.15.28:6789/0 conn(0x7fbe80159770 legacy=0x7fbe8006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7e000640 1 -- 172.21.15.28:0/1244934359 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8015b570 con 0x7fbe80070430 2024-09-13T17:09:08.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.508+0000 7fbe7f400640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 0x7fbe8015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.509+0000 7fbe7e000640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80070430 0x7fbe8015bd70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0097a0 tx=0x7fbe6c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91277d4c2a8a7c67 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.509+0000 7fbe7d600640 1 -- 172.21.15.28:0/1244934359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c02ae50 con 0x7fbe80070430 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.509+0000 7fbe7d600640 1 -- 172.21.15.28:0/1244934359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c032440 con 0x7fbe80070430 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.509+0000 7fbe7d600640 1 -- 172.21.15.28:0/1244934359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0327d0 con 0x7fbe80070430 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 -- 172.21.15.28:0/1244934359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80070430 msgr2=0x7fbe8015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80070430 0x7fbe8015bd70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c0097a0 tx=0x7fbe6c02a100 comp rx=0 tx=0).stop 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 -- 172.21.15.28:0/1244934359 shutdown_connections 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe8006f3f0 0x7fbe8015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 --2- 172.21.15.28:0/1244934359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80070430 0x7fbe8015bd70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 -- 172.21.15.28:0/1244934359 >> 172.21.15.28:0/1244934359 conn(0x7fbe800b0270 msgr2=0x7fbe800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 -- 172.21.15.28:0/1244934359 shutdown_connections 2024-09-13T17:09:08.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.510+0000 7fbe85800640 1 -- 172.21.15.28:0/1244934359 wait complete. 2024-09-13T17:09:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.512+0000 7fbe85800640 1 Processor -- start 2024-09-13T17:09:08.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.512+0000 7fbe85800640 1 -- start start 2024-09-13T17:09:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 0x7fbe801f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 0x7fbe80150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe801f3960 con 0x7fbe80159770 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe800ad670 con 0x7fbe8006f3f0 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe85800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe800ad950 con 0x7fbe80070430 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53066/0 (socket says 172.21.15.28:53066) 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe7f400640 1 -- 172.21.15.28:0/563661541 learned_addr learned my addr 172.21.15.28:0/563661541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe7ea00640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 0x7fbe801f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.513+0000 7fbe7e000640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 0x7fbe80150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 -- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 msgr2=0x7fbe80150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 0x7fbe80150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 -- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 msgr2=0x7fbe801f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 0x7fbe801f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 -- 172.21.15.28:0/563661541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6c009400 con 0x7fbe80159770 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7ea00640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 0x7fbe801f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7e000640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 0x7fbe80150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe7f400640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbe78007ef0 tx=0x7fbe7800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe78015600 con 0x7fbe80159770 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7800f040 con 0x7fbe80159770 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe800a8cc0 con 0x7fbe80159770 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe780159a0 con 0x7fbe80159770 2024-09-13T17:09:08.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.514+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe800a9310 con 0x7fbe80159770 2024-09-13T17:09:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.516+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8006fd20 con 0x7fbe80159770 2024-09-13T17:09:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.516+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbe7801e3b0 con 0x7fbe80159770 2024-09-13T17:09:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.517+0000 7fbe76a00640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 0x7fbe6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.517+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbe780893c0 con 0x7fbe80159770 2024-09-13T17:09:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.518+0000 7fbe7ea00640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 0x7fbe6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.518+0000 7fbe7ea00640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 0x7fbe6006d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe70001270 tx=0x7fbe70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.521+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbe780548e0 con 0x7fbe80159770 2024-09-13T17:09:08.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.701+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe801f3b40 con 0x7fbe80159770 2024-09-13T17:09:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.702+0000 7fbe76a00640 1 -- 172.21.15.28:0/563661541 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe78057f00 con 0x7fbe80159770 2024-09-13T17:09:08.703 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:09:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 msgr2=0x7fbe6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 0x7fbe6006d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbe70001270 tx=0x7fbe70009040 comp rx=0 tx=0).stop 2024-09-13T17:09:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 msgr2=0x7fbe801536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fbe78007ef0 tx=0x7fbe7800d4a0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 shutdown_connections 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbe6006ac00 0x7fbe6006d0c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbe80159770 0x7fbe801536d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbe80070430 0x7fbe80150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 --2- 172.21.15.28:0/563661541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8006f3f0 0x7fbe801f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 >> 172.21.15.28:0/563661541 conn(0x7fbe800b0270 msgr2=0x7fbe80159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 shutdown_connections 2024-09-13T17:09:08.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.704+0000 7fbe85800640 1 -- 172.21.15.28:0/563661541 wait complete. 2024-09-13T17:09:08.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:09:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.875+0000 7fcad6e00640 1 Processor -- start 2024-09-13T17:09:08.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.875+0000 7fcad6e00640 1 -- start start 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 0x7fcad015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad6e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcad006f3f0 0x7fcad015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad6e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad015ab60 con 0x7fcad0159770 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad6e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad006f920 con 0x7fcad006f3f0 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad015b290 con 0x7fcad0070430 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad4e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcad006f3f0 0x7fcad015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcac7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 0x7fcad015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad4e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcad006f3f0 0x7fcad015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48112/0 (socket says 172.21.15.28:48112) 2024-09-13T17:09:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.877+0000 7fcad4e00640 1 -- 172.21.15.28:0/3705658673 learned_addr learned my addr 172.21.15.28:0/3705658673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcad4e00640 1 -- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 msgr2=0x7fcad015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcad4e00640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 0x7fcad015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcad4e00640 1 -- 172.21.15.28:0/3705658673 >> v1:172.21.15.189:6789/0 conn(0x7fcad0159770 legacy=0x7fcad006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcad4e00640 1 -- 172.21.15.28:0/3705658673 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad015b570 con 0x7fcad006f3f0 2024-09-13T17:09:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcacf400640 1 -- 172.21.15.28:0/3705658673 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777029470 0 0) 0x7fcac0001110 con 0x7fcad0159770 2024-09-13T17:09:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.878+0000 7fcac7400640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 0x7fcad015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:09:08.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.879+0000 7fcad4e00640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad006f3f0 0x7fcad015c2b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac802c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b22ba987d8625d43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.879+0000 7fcacf400640 1 -- 172.21.15.28:0/3705658673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8009600 con 0x7fcad006f3f0 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.879+0000 7fcacf400640 1 -- 172.21.15.28:0/3705658673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac8009760 con 0x7fcad006f3f0 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.879+0000 7fcacf400640 1 -- 172.21.15.28:0/3705658673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8009af0 con 0x7fcad006f3f0 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 -- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad006f3f0 msgr2=0x7fcad015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad006f3f0 0x7fcad015c2b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac802c890 comp rx=0 tx=0).stop 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 -- 172.21.15.28:0/3705658673 shutdown_connections 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad006f3f0 0x7fcad015c2b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3705658673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad0070430 0x7fcad015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 -- 172.21.15.28:0/3705658673 >> 172.21.15.28:0/3705658673 conn(0x7fcad00b0270 msgr2=0x7fcad00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 -- 172.21.15.28:0/3705658673 shutdown_connections 2024-09-13T17:09:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.880+0000 7fcad6e00640 1 -- 172.21.15.28:0/3705658673 wait complete. 2024-09-13T17:09:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.881+0000 7fcad6e00640 1 Processor -- start 2024-09-13T17:09:08.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 -- start start 2024-09-13T17:09:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad006f3f0 0x7fcad014de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0070430 0x7fcad01e0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad01e3a20 con 0x7fcad0159770 2024-09-13T17:09:08.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad014e380 con 0x7fcad0070430 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.882+0000 7fcad6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00cc720 con 0x7fcad006f3f0 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53074/0 (socket says 172.21.15.28:53074) 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 -- 172.21.15.28:0/3247299572 learned_addr learned my addr 172.21.15.28:0/3247299572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcac7400640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0070430 0x7fcad01e0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcacfe00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad006f3f0 0x7fcad014de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 -- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad006f3f0 msgr2=0x7fcad014de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad006f3f0 0x7fcad014de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 -- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0070430 msgr2=0x7fcad01e0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0070430 0x7fcad01e0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.883+0000 7fcad4e00640 1 -- 172.21.15.28:0/3247299572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab8009400 con 0x7fcad0159770 2024-09-13T17:09:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcad4e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8009600 con 0x7fcad0159770 2024-09-13T17:09:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac800b720 con 0x7fcad0159770 2024-09-13T17:09:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac80346c0 con 0x7fcad0159770 2024-09-13T17:09:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac800bd40 con 0x7fcad0159770 2024-09-13T17:09:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.884+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad00ccce0 con 0x7fcad0159770 2024-09-13T17:09:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.885+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad006fd20 con 0x7fcad0159770 2024-09-13T17:09:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.886+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcac803b070 con 0x7fcad0159770 2024-09-13T17:09:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.887+0000 7fcaccc00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 0x7fcaac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.887+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcac80ab650 con 0x7fcad0159770 2024-09-13T17:09:08.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.888+0000 7fcacfe00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 0x7fcaac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:08.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.888+0000 7fcacfe00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 0x7fcaac06d0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcac0000f60 tx=0x7fcac0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:08.892+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcac8076b90 con 0x7fcad0159770 2024-09-13T17:09:08.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:09:08.933 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.078+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fcad01545b0 con 0x7fcad0159770 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.078+0000 7fcaccc00640 1 -- 172.21.15.28:0/3247299572 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7fcac807a1b0 con 0x7fcad0159770 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:09:09.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:09:09.080 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:09:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 msgr2=0x7fcaac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 0x7fcaac06d0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcac0000f60 tx=0x7fcac0009040 comp rx=0 tx=0).stop 2024-09-13T17:09:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 msgr2=0x7fcad01e11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fcac800d150 tx=0x7fcac8004a40 comp rx=0 tx=0).stop 2024-09-13T17:09:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 shutdown_connections 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcaac06ac00 0x7fcaac06d0c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad0159770 0x7fcad01e11b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad0070430 0x7fcad01e0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 --2- 172.21.15.28:0/3247299572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad006f3f0 0x7fcad014de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 >> 172.21.15.28:0/3247299572 conn(0x7fcad00b0270 msgr2=0x7fcad00ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 shutdown_connections 2024-09-13T17:09:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.080+0000 7fcad6e00640 1 -- 172.21.15.28:0/3247299572 wait complete. 2024-09-13T17:09:09.096 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:09:09.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:09:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.255+0000 7f6ed1600640 1 Processor -- start 2024-09-13T17:09:09.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.255+0000 7f6ed1600640 1 -- start start 2024-09-13T17:09:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.256+0000 7f6ed1600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.256+0000 7f6ed1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 0x7f6ecc15d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.256+0000 7f6ed1600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc15d950 con 0x7f6ecc157000 2024-09-13T17:09:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.256+0000 7f6ed1600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc15dc30 con 0x7f6ecc0b2380 2024-09-13T17:09:09.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.256+0000 7f6ed1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0b1b80 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6ecaa00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6ecc157000 0x7f6ecc06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:37186/0 (socket says 172.21.15.28:37186) 2024-09-13T17:09:09.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48144/0 (socket says 172.21.15.28:48144) 2024-09-13T17:09:09.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 -- 172.21.15.28:0/192935112 learned_addr learned my addr 172.21.15.28:0/192935112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6ecb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 0x7f6ecc15d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 -- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 msgr2=0x7f6ecc15d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 0x7f6ecc15d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 -- 172.21.15.28:0/192935112 >> v1:172.21.15.189:6789/0 conn(0x7f6ecc157000 legacy=0x7f6ecc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6eca000640 1 -- 172.21.15.28:0/192935112 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc0b1e60 con 0x7f6ecc0b2380 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6ec9600640 1 -- 172.21.15.28:0/192935112 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029441134 0 0) 0x7f6ecc15d950 con 0x7f6ecc157000 2024-09-13T17:09:09.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.257+0000 7f6ecb400640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 0x7f6ecc15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:09:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.258+0000 7f6eca000640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb014462345fffa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.258+0000 7f6ec9600640 1 -- 172.21.15.28:0/192935112 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8032440 con 0x7f6ecc0b2380 2024-09-13T17:09:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ec9600640 1 -- 172.21.15.28:0/192935112 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb80325a0 con 0x7f6ecc0b2380 2024-09-13T17:09:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ec9600640 1 -- 172.21.15.28:0/192935112 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb8032930 con 0x7f6ecc0b2380 2024-09-13T17:09:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 -- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 msgr2=0x7f6ecc0b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009870 tx=0x7f6eb802a250 comp rx=0 tx=0).stop 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 -- 172.21.15.28:0/192935112 shutdown_connections 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc15cdd0 0x7f6ecc15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 --2- 172.21.15.28:0/192935112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc0b27e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 -- 172.21.15.28:0/192935112 >> 172.21.15.28:0/192935112 conn(0x7f6ecc069e80 msgr2=0x7f6ecc06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 -- 172.21.15.28:0/192935112 shutdown_connections 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.259+0000 7f6ed1600640 1 -- 172.21.15.28:0/192935112 wait complete. 2024-09-13T17:09:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.261+0000 7f6ed1600640 1 Processor -- start 2024-09-13T17:09:09.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.261+0000 7f6ed1600640 1 -- start start 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc157000 0x7f6ecc1f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1553c0 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc0cbdc0 con 0x7f6ecc0b2380 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ed1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc1f2ce0 con 0x7f6ecc157000 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ecb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ecb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53082/0 (socket says 172.21.15.28:53082) 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ecb400640 1 -- 172.21.15.28:0/3623810402 learned_addr learned my addr 172.21.15.28:0/3623810402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6eca000640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc157000 0x7f6ecc1f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.262+0000 7f6ecaa00640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 -- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc157000 msgr2=0x7f6ecc1f25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc157000 0x7f6ecc1f25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 -- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 msgr2=0x7f6ecc1594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc1594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 -- 172.21.15.28:0/3623810402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4009a70 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecaa00640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ecb400640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400efc0 tx=0x7f6ec4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4009d70 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec400f040 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4003990 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.263+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb80094d0 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.264+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc1f32a0 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.265+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc0cbf30 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.265+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6ec40144a0 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.266+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 0x7f6eac06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.266+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6ec4089c80 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.267+0000 7f6ecaa00640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 0x7f6eac06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.267+0000 7f6ecaa00640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 0x7f6eac06d0f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc000f60 tx=0x7f6ebc007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.270+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6ec4055110 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.452+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6ecc0b27e0 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.651+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3623810402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6ec4058730 con 0x7f6ecc15cdd0 2024-09-13T17:09:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:09:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.654+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 msgr2=0x7f6eac06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.654+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 0x7f6eac06d0f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc000f60 tx=0x7f6ebc007c10 comp rx=0 tx=0).stop 2024-09-13T17:09:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.654+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 msgr2=0x7f6ecc14beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.654+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400efc0 tx=0x7f6ec4009f90 comp rx=0 tx=0).stop 2024-09-13T17:09:09.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 shutdown_connections 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6eac06ac30 0x7f6eac06d0f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ecc15cdd0 0x7f6ecc14beb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6ecc157000 0x7f6ecc1f25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 --2- 172.21.15.28:0/3623810402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ecc0b2380 0x7f6ecc1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 >> 172.21.15.28:0/3623810402 conn(0x7f6ecc069e80 msgr2=0x7f6ecc06bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 shutdown_connections 2024-09-13T17:09:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.655+0000 7f6ed1600640 1 -- 172.21.15.28:0/3623810402 wait complete. 2024-09-13T17:09:09.675 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:09:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.836+0000 7fc587e00640 1 Processor -- start 2024-09-13T17:09:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.837+0000 7fc587e00640 1 -- start start 2024-09-13T17:09:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc58806b360 0x7fc5880b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5880b0980 0x7fc5880af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b0dd0 con 0x7fc588154a90 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58806b900 con 0x7fc58806b360 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58806bbe0 con 0x7fc5880b0980 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc57fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc58806b360 0x7fc5880b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc57fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc58806b360 0x7fc5880b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53098/0 (socket says 172.21.15.28:53098) 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc57fe00640 1 -- 172.21.15.28:0/2795046503 learned_addr learned my addr 172.21.15.28:0/2795046503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.838+0000 7fc587400640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5880b0980 0x7fc5880af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.839+0000 7fc57fe00640 1 -- 172.21.15.28:0/2795046503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5880b0980 msgr2=0x7fc5880af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.839+0000 7fc57fe00640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5880b0980 0x7fc5880af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.839+0000 7fc57fe00640 1 -- 172.21.15.28:0/2795046503 >> v1:172.21.15.189:6789/0 conn(0x7fc588154a90 legacy=0x7fc58815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.839+0000 7fc57fe00640 1 -- 172.21.15.28:0/2795046503 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58806bec0 con 0x7fc58806b360 2024-09-13T17:09:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc57fe00640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58806b360 0x7fc5880b0440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc570009870 tx=0x7fc57002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb9e55e8c1fc2e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc586000640 1 -- 172.21.15.28:0/2795046503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570032440 con 0x7fc58806b360 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc586000640 1 -- 172.21.15.28:0/2795046503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5700325a0 con 0x7fc58806b360 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc586000640 1 -- 172.21.15.28:0/2795046503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570032930 con 0x7fc58806b360 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 -- 172.21.15.28:0/2795046503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58806b360 msgr2=0x7fc5880b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58806b360 0x7fc5880b0440 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc570009870 tx=0x7fc57002a250 comp rx=0 tx=0).stop 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 -- 172.21.15.28:0/2795046503 shutdown_connections 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5880b0980 0x7fc5880af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 --2- 172.21.15.28:0/2795046503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc58806b360 0x7fc5880b0440 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.840+0000 7fc587e00640 1 -- 172.21.15.28:0/2795046503 >> 172.21.15.28:0/2795046503 conn(0x7fc588072e40 msgr2=0x7fc588073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.841+0000 7fc587e00640 1 -- 172.21.15.28:0/2795046503 shutdown_connections 2024-09-13T17:09:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.841+0000 7fc587e00640 1 -- 172.21.15.28:0/2795046503 wait complete. 2024-09-13T17:09:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.842+0000 7fc587e00640 1 Processor -- start 2024-09-13T17:09:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.842+0000 7fc587e00640 1 -- start start 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58806b360 0x7fc58814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880b0980 0x7fc588147ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc588157a10 con 0x7fc588154a90 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5881557d0 con 0x7fc5880b0980 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.843+0000 7fc587e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc588147c80 con 0x7fc58806b360 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53112/0 (socket says 172.21.15.28:53112) 2024-09-13T17:09:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 -- 172.21.15.28:0/3064447614 learned_addr learned my addr 172.21.15.28:0/3064447614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc586a00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58806b360 0x7fc58814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc57fe00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880b0980 0x7fc588147ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 -- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58806b360 msgr2=0x7fc58814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58806b360 0x7fc58814a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 -- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880b0980 msgr2=0x7fc588147ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880b0980 0x7fc588147ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 -- 172.21.15.28:0/3064447614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700094d0 con 0x7fc588154a90 2024-09-13T17:09:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.844+0000 7fc587400640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc580007dc0 tx=0x7fc58000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.845+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580014420 con 0x7fc588154a90 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.845+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5800145a0 con 0x7fc588154a90 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.845+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc580014950 con 0x7fc588154a90 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.845+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58814a6a0 con 0x7fc588154a90 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.845+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5881554f0 con 0x7fc588154a90 2024-09-13T17:09:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.846+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc588157b50 con 0x7fc588154a90 2024-09-13T17:09:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.847+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc580014ab0 con 0x7fc588154a90 2024-09-13T17:09:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.848+0000 7fc57ea00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 0x7fc56406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.848+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc58008a5d0 con 0x7fc588154a90 2024-09-13T17:09:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.849+0000 7fc586a00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 0x7fc56406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:09.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.849+0000 7fc586a00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 0x7fc56406d0f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc578000f60 tx=0x7fc578009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:09.851+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc580055ab0 con 0x7fc588154a90 2024-09-13T17:09:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.031+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc588157b50 con 0x7fc588154a90 2024-09-13T17:09:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:09:10.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.657+0000 7fc57ea00640 1 -- 172.21.15.28:0/3064447614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc588157b50 con 0x7fc588154a90 2024-09-13T17:09:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.662+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 msgr2=0x7fc56406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.662+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 0x7fc56406d0f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc578000f60 tx=0x7fc578009040 comp rx=0 tx=0).stop 2024-09-13T17:09:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.662+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 msgr2=0x7fc588137000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.662+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc580007dc0 tx=0x7fc58000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 shutdown_connections 2024-09-13T17:09:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc56406ac30 0x7fc56406d0f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc588154a90 0x7fc588137000 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5880b0980 0x7fc588147ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 --2- 172.21.15.28:0/3064447614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58806b360 0x7fc58814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.663+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 >> 172.21.15.28:0/3064447614 conn(0x7fc588072e40 msgr2=0x7fc5880b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.666+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 shutdown_connections 2024-09-13T17:09:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.666+0000 7fc587e00640 1 -- 172.21.15.28:0/3064447614 wait complete. 2024-09-13T17:09:10.697 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:09:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.857+0000 7f800b200640 1 Processor -- start 2024-09-13T17:09:10.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.857+0000 7f800b200640 1 -- start start 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f800b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f800c1643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f800b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c159e10 con 0x7f800c070e30 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f800b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f800c15a0f0 con 0x7f800c071210 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f800b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c15a3d0 con 0x7f800c0cdf80 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f8009e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f800c070e30 0x7f800c071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33836/0 (socket says 172.21.15.28:33836) 2024-09-13T17:09:10.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.858+0000 7f8009e00640 1 -- 172.21.15.28:0/2531368609 learned_addr learned my addr 172.21.15.28:0/2531368609 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f800a800640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f800c1643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365735683 0 0) 0x7f800c159e10 con 0x7f800c070e30 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff0003520 con 0x7f800c070e30 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2466123123 0 0) 0x7f7ff0003520 con 0x7f800c070e30 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 msgr2=0x7f800c1643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f800c1643b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 >> v1:172.21.15.189:6789/0 conn(0x7f800c071210 legacy=0x7f800c0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c15a6b0 con 0x7f800c070e30 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7ffc001670 con 0x7f800c070e30 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 >> v1:172.21.15.28:6789/0 conn(0x7f800c070e30 legacy=0x7f800c071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f7ff00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff0003f20 0x7f7ff00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0004d00 0x7f7ff0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c15a3d0 con 0x7f7ff0003f20 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff000dc70 con 0x7f7ff0004d00 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.859+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff000de30 con 0x7f800c0cdf80 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff0003f20 0x7f7ff00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f800a800640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0004d00 0x7f7ff0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8009e00640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f7ff00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 -- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 msgr2=0x7f7ff00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:10.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c0cdf80 0x7f7ff00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 -- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0004d00 msgr2=0x7f7ff0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0004d00 0x7f7ff0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.860+0000 7f8001400640 1 -- 172.21.15.28:0/2531368609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff0003520 con 0x7f7ff0003f20 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.861+0000 7f8001400640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff0003f20 0x7f7ff00047c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4001750 tx=0x7f7ff00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d778c5b316a3b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.861+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff40052d0 con 0x7f7ff0003f20 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.861+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4005430 con 0x7f7ff0003f20 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.861+0000 7f8009400640 1 -- 172.21.15.28:0/2531368609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff402abe0 con 0x7f7ff0003f20 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 -- 172.21.15.28:0/2531368609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff0003f20 msgr2=0x7f7ff00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 --2- 172.21.15.28:0/2531368609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ff0003f20 0x7f7ff00047c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4001750 tx=0x7f7ff00057f0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800a800640 1 -- 172.21.15.28:0/2531368609 reap_dead start 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 -- 172.21.15.28:0/2531368609 shutdown_connections 2024-09-13T17:09:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 -- 172.21.15.28:0/2531368609 >> 172.21.15.28:0/2531368609 conn(0x7f800c069e80 msgr2=0x7f800c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 -- 172.21.15.28:0/2531368609 shutdown_connections 2024-09-13T17:09:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.862+0000 7f800b200640 1 -- 172.21.15.28:0/2531368609 wait complete. 2024-09-13T17:09:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.864+0000 7f800b200640 1 Processor -- start 2024-09-13T17:09:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.864+0000 7f800b200640 1 -- start start 2024-09-13T17:09:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0003f20 0x7f800c1476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c147e40 0x7f800c148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c14d620 con 0x7f800c0cbe80 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800c0cc870 con 0x7f7ff0003f20 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f800c147be0 con 0x7f800c147e40 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53138/0 (socket says 172.21.15.28:53138) 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 -- 172.21.15.28:0/233218521 learned_addr learned my addr 172.21.15.28:0/233218521 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f8009e00640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0003f20 0x7f800c1476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f8001400640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c147e40 0x7f800c148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 -- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c147e40 msgr2=0x7f800c148c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c147e40 0x7f800c148c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 -- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0003f20 msgr2=0x7f800c1476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0003f20 0x7f800c1476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.865+0000 7f800a800640 1 -- 172.21.15.28:0/233218521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c1495a0 con 0x7f800c0cbe80 2024-09-13T17:09:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f800a800640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f800c0b3a80 tx=0x7f800400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400e040 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f800400f040 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004015430 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800c149880 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.866+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800c149e40 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.867+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800c14d760 con 0x7f800c0cbe80 2024-09-13T17:09:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.869+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8004015590 con 0x7f800c0cbe80 2024-09-13T17:09:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.869+0000 7f8002a00640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 0x7f7fe806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.869+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f8004089eb0 con 0x7f800c0cbe80 2024-09-13T17:09:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.870+0000 7f8009e00640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 0x7f7fe806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.871+0000 7f8009e00640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 0x7f7fe806d0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc001c10 tx=0x7f7ffc001b20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:10.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:10.873+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8004055210 con 0x7f800c0cbe80 2024-09-13T17:09:11.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.088+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f800c14d760 con 0x7f800c0cbe80 2024-09-13T17:09:11.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.095+0000 7f8002a00640 1 -- 172.21.15.28:0/233218521 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f800c14d760 con 0x7f800c0cbe80 2024-09-13T17:09:11.098 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 4 and data pool 5 2024-09-13T17:09:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.100+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 msgr2=0x7f7fe806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.100+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 0x7f7fe806d0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc001c10 tx=0x7f7ffc001b20 comp rx=0 tx=0).stop 2024-09-13T17:09:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.100+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 msgr2=0x7f800c0cc330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.100+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f800c0b3a80 tx=0x7f800400f580 comp rx=0 tx=0).stop 2024-09-13T17:09:11.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 shutdown_connections 2024-09-13T17:09:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7fe806ac30 0x7f7fe806d0f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f800c0cbe80 0x7f800c0cc330 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f800c147e40 0x7f800c148c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 --2- 172.21.15.28:0/233218521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ff0003f20 0x7f800c1476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.102+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 >> 172.21.15.28:0/233218521 conn(0x7f800c069e80 msgr2=0x7f800c15d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.105+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 shutdown_connections 2024-09-13T17:09:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.105+0000 7f800b200640 1 -- 172.21.15.28:0/233218521 wait complete. 2024-09-13T17:09:11.138 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:09:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.292+0000 7fbff6000640 1 Processor -- start 2024-09-13T17:09:11.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.292+0000 7fbff6000640 1 -- start start 2024-09-13T17:09:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff6000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbff015c8d0 0x7fbff006b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 0x7fbff006f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff6000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff006bb50 con 0x7fbff00ccb60 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff6000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff006b3f0 con 0x7fbff006f480 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff6000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff006fe00 con 0x7fbff015c8d0 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 0x7fbff006f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 0x7fbff006f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53148/0 (socket says 172.21.15.28:53148) 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff4c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbff015c8d0 0x7fbff006b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.293+0000 7fbff5600640 1 -- 172.21.15.28:0/1297808268 learned_addr learned my addr 172.21.15.28:0/1297808268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.294+0000 7fbff4c00640 1 -- 172.21.15.28:0/1297808268 >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 msgr2=0x7fbff006f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.294+0000 7fbff4c00640 1 --2- 172.21.15.28:0/1297808268 >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 0x7fbff006f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.294+0000 7fbff4c00640 1 -- 172.21.15.28:0/1297808268 >> v1:172.21.15.189:6789/0 conn(0x7fbff00ccb60 legacy=0x7fbff006ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.294+0000 7fbff4c00640 1 -- 172.21.15.28:0/1297808268 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00b38a0 con 0x7fbff015c8d0 2024-09-13T17:09:11.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.295+0000 7fbff4c00640 1 --2- 172.21.15.28:0/1297808268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff015c8d0 0x7fbff006b610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40097a0 tx=0x7fbfe402a1d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb30a039fe35c69 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.295+0000 7fbfef400640 1 -- 172.21.15.28:0/1297808268 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4032440 con 0x7fbff015c8d0 2024-09-13T17:09:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.295+0000 7fbfef400640 1 -- 172.21.15.28:0/1297808268 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfe40325a0 con 0x7fbff015c8d0 2024-09-13T17:09:11.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.295+0000 7fbfef400640 1 -- 172.21.15.28:0/1297808268 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4032930 con 0x7fbff015c8d0 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 -- 172.21.15.28:0/1297808268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff015c8d0 msgr2=0x7fbff006b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 --2- 172.21.15.28:0/1297808268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff015c8d0 0x7fbff006b610 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40097a0 tx=0x7fbfe402a1d0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 -- 172.21.15.28:0/1297808268 shutdown_connections 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 --2- 172.21.15.28:0/1297808268 >> v2:172.21.15.28:3300/0 conn(0x7fbff006f480 0x7fbff006f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 --2- 172.21.15.28:0/1297808268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff015c8d0 0x7fbff006b610 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 -- 172.21.15.28:0/1297808268 >> 172.21.15.28:0/1297808268 conn(0x7fbff006d4e0 msgr2=0x7fbff006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 -- 172.21.15.28:0/1297808268 shutdown_connections 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.296+0000 7fbff6000640 1 -- 172.21.15.28:0/1297808268 wait complete. 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.297+0000 7fbff6000640 1 Processor -- start 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.298+0000 7fbff6000640 1 -- start start 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.298+0000 7fbff6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff006f480 0x7fbff015d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff6000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbff015c8d0 0x7fbff01f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff015e5c0 con 0x7fbff00ccb60 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff6000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff015f100 con 0x7fbff006f480 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff6000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff015fad0 con 0x7fbff015c8d0 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53162/0 (socket says 172.21.15.28:53162) 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 -- 172.21.15.28:0/2100899128 learned_addr learned my addr 172.21.15.28:0/2100899128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:11.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff4c00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff006f480 0x7fbff015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbff5600640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbff015c8d0 0x7fbff01f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 -- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbff015c8d0 msgr2=0x7fbff01f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbff015c8d0 0x7fbff01f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 -- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff006f480 msgr2=0x7fbff015d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff006f480 0x7fbff015d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.299+0000 7fbfefe00640 1 -- 172.21.15.28:0/2100899128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe4009400 con 0x7fbff00ccb60 2024-09-13T17:09:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbfefe00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00c450 tx=0x7fbfdc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0052d0 con 0x7fbff00ccb60 2024-09-13T17:09:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc005430 con 0x7fbff00ccb60 2024-09-13T17:09:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc012870 con 0x7fbff00ccb60 2024-09-13T17:09:11.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff015e7d0 con 0x7fbff00ccb60 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.300+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff015ed90 con 0x7fbff00ccb60 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.301+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff015f290 con 0x7fbff00ccb60 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.302+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbfdc0129d0 con 0x7fbff00ccb60 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.303+0000 7fbfecc00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 0x7fbfd006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.303+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfdc088c40 con 0x7fbff00ccb60 2024-09-13T17:09:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.303+0000 7fbff4c00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 0x7fbfd006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.304+0000 7fbff4c00640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 0x7fbfd006d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4034000 tx=0x7fbfe402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.307+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbfdc053e30 con 0x7fbff00ccb60 2024-09-13T17:09:11.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.487+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbff006bb50 con 0x7fbff00ccb60 2024-09-13T17:09:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.488+0000 7fbfecc00640 1 -- 172.21.15.28:0/2100899128 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7fbff006bb50 con 0x7fbff00ccb60 2024-09-13T17:09:11.491 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:11.491 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":33,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:09:11.090788+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:09:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 msgr2=0x7fbfd006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 0x7fbfd006d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fbfe4034000 tx=0x7fbfe402e000 comp rx=0 tx=0).stop 2024-09-13T17:09:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 msgr2=0x7fbff015f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00c450 tx=0x7fbfdc00c910 comp rx=0 tx=0).stop 2024-09-13T17:09:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 shutdown_connections 2024-09-13T17:09:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbfd006ac00 0x7fbfd006d0c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbff015c8d0 0x7fbff01f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbff00ccb60 0x7fbff015f590 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 --2- 172.21.15.28:0/2100899128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbff006f480 0x7fbff015d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 >> 172.21.15.28:0/2100899128 conn(0x7fbff006d4e0 msgr2=0x7fbff015b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.490+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 shutdown_connections 2024-09-13T17:09:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.491+0000 7fbff6000640 1 -- 172.21.15.28:0/2100899128 wait complete. 2024-09-13T17:09:11.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:09:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.664+0000 7efea6c00640 1 Processor -- start 2024-09-13T17:09:11.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.664+0000 7efea6c00640 1 -- start start 2024-09-13T17:09:11.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efea00b3710 0x7efea00714a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efea00719e0 0x7efea015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efea006aaf0 con 0x7efea00afd70 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea00b0150 con 0x7efea00b3710 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efea0071e10 con 0x7efea00719e0 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efea00b3710 0x7efea00714a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efea00b3710 0x7efea00714a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53168/0 (socket says 172.21.15.28:53168) 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea4e00640 1 -- 172.21.15.28:0/1198791026 learned_addr learned my addr 172.21.15.28:0/1198791026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.665+0000 7efea6200640 1 --2- 172.21.15.28:0/1198791026 >> v2:172.21.15.189:3300/0 conn(0x7efea00719e0 0x7efea015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.666+0000 7efea4e00640 1 -- 172.21.15.28:0/1198791026 >> v2:172.21.15.189:3300/0 conn(0x7efea00719e0 msgr2=0x7efea015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.666+0000 7efea4e00640 1 --2- 172.21.15.28:0/1198791026 >> v2:172.21.15.189:3300/0 conn(0x7efea00719e0 0x7efea015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.666+0000 7efea4e00640 1 -- 172.21.15.28:0/1198791026 >> v1:172.21.15.189:6789/0 conn(0x7efea00afd70 legacy=0x7efea006b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.666+0000 7efea4e00640 1 -- 172.21.15.28:0/1198791026 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0069f20 con 0x7efea00b3710 2024-09-13T17:09:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.667+0000 7efea4e00640 1 --2- 172.21.15.28:0/1198791026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00714a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efea006b520 tx=0x7efe8c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51424ca3ea636d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.667+0000 7efe9fe00640 1 -- 172.21.15.28:0/1198791026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c032440 con 0x7efea00b3710 2024-09-13T17:09:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.667+0000 7efe9fe00640 1 -- 172.21.15.28:0/1198791026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c0325a0 con 0x7efea00b3710 2024-09-13T17:09:11.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.667+0000 7efe9fe00640 1 -- 172.21.15.28:0/1198791026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe8c032930 con 0x7efea00b3710 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 -- 172.21.15.28:0/1198791026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 msgr2=0x7efea00714a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 --2- 172.21.15.28:0/1198791026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00714a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7efea006b520 tx=0x7efe8c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 -- 172.21.15.28:0/1198791026 shutdown_connections 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 --2- 172.21.15.28:0/1198791026 >> v2:172.21.15.189:3300/0 conn(0x7efea00719e0 0x7efea015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 --2- 172.21.15.28:0/1198791026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00714a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 -- 172.21.15.28:0/1198791026 >> 172.21.15.28:0/1198791026 conn(0x7efea00b2380 msgr2=0x7efea00b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:11.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 -- 172.21.15.28:0/1198791026 shutdown_connections 2024-09-13T17:09:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.668+0000 7efea6c00640 1 -- 172.21.15.28:0/1198791026 wait complete. 2024-09-13T17:09:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.669+0000 7efea6c00640 1 Processor -- start 2024-09-13T17:09:11.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.670+0000 7efea6c00640 1 -- start start 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.670+0000 7efea6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efea00719e0 0x7efea01f3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.670+0000 7efea6c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00afd70 0x7efea00af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea01f3d50 con 0x7efea00b3710 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea00a8c30 con 0x7efea00afd70 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efea00ab680 con 0x7efea00719e0 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53172/0 (socket says 172.21.15.28:53172) 2024-09-13T17:09:11.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 -- 172.21.15.28:0/383172265 learned_addr learned my addr 172.21.15.28:0/383172265 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea4e00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00afd70 0x7efea00af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea5800640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efea00719e0 0x7efea01f3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 -- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efea00719e0 msgr2=0x7efea01f3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efea00719e0 0x7efea01f3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 -- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00afd70 msgr2=0x7efea00af2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00afd70 0x7efea00af2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.671+0000 7efea6200640 1 -- 172.21.15.28:0/383172265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8c0094d0 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efea6200640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7efe9800efc0 tx=0x7efe9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe98009ba0 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe9800f040 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe980146c0 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea00ab3a0 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.672+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea00a88c0 con 0x7efea00b3710 2024-09-13T17:09:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.673+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea00a8440 con 0x7efea00b3710 2024-09-13T17:09:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.674+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efe98014820 con 0x7efea00b3710 2024-09-13T17:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.675+0000 7efe9d600640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 0x7efe8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.675+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe9808a020 con 0x7efea00b3710 2024-09-13T17:09:11.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.675+0000 7efea5800640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 0x7efe8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:11.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.676+0000 7efea5800640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 0x7efe8006d0c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efea00d0030 tx=0x7efe94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:11.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.679+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efea00a8440 con 0x7efea00b3710 2024-09-13T17:09:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.860+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efea00a8440 con 0x7efea00b3710 2024-09-13T17:09:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.861+0000 7efe9d600640 1 -- 172.21.15.28:0/383172265 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16582 (secure 0 0 0) 0x7efea00a8440 con 0x7efea00b3710 2024-09-13T17:09:11.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:11.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":33,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:09:11.090788+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 msgr2=0x7efe8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 0x7efe8006d0c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7efea00d0030 tx=0x7efe94009040 comp rx=0 tx=0).stop 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 msgr2=0x7efea00aa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7efe9800efc0 tx=0x7efe9800c490 comp rx=0 tx=0).stop 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 shutdown_connections 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe8006ac00 0x7efe8006d0c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efea00b3710 0x7efea00aa8e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efea00afd70 0x7efea00af2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 --2- 172.21.15.28:0/383172265 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efea00719e0 0x7efea01f3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 >> 172.21.15.28:0/383172265 conn(0x7efea00b2380 msgr2=0x7efea006ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 shutdown_connections 2024-09-13T17:09:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:11.864+0000 7efea6c00640 1 -- 172.21.15.28:0/383172265 wait complete. 2024-09-13T17:09:11.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:09:12.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.039+0000 7f7999800640 1 Processor -- start 2024-09-13T17:09:12.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.040+0000 7f7999800640 1 -- start start 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7999800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7994071210 0x7f79940cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7999800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f79940cddb0 0x7f799415aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7999800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7994159e10 con 0x7f7994070e30 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7999800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79940ce240 con 0x7f7994071210 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7999800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f799415a0f0 con 0x7f79940cddb0 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f798be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7994071210 0x7f79940cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f798be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7994071210 0x7f79940cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53184/0 (socket says 172.21.15.28:53184) 2024-09-13T17:09:12.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f798be00640 1 -- 172.21.15.28:0/3583586102 learned_addr learned my addr 172.21.15.28:0/3583586102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.041+0000 7f7998e00640 1 --2- 172.21.15.28:0/3583586102 >> v2:172.21.15.189:3300/0 conn(0x7f79940cddb0 0x7f799415aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.042+0000 7f798be00640 1 -- 172.21.15.28:0/3583586102 >> v2:172.21.15.189:3300/0 conn(0x7f79940cddb0 msgr2=0x7f799415aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.042+0000 7f798be00640 1 --2- 172.21.15.28:0/3583586102 >> v2:172.21.15.189:3300/0 conn(0x7f79940cddb0 0x7f799415aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.042+0000 7f798be00640 1 -- 172.21.15.28:0/3583586102 >> v1:172.21.15.189:6789/0 conn(0x7f7994070e30 legacy=0x7f7994071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:12.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.042+0000 7f798be00640 1 -- 172.21.15.28:0/3583586102 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799415a3d0 con 0x7f7994071210 2024-09-13T17:09:12.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f798be00640 1 --2- 172.21.15.28:0/3583586102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f79940cd870 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f79940c33c0 tx=0x7f797c02c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776edbf4409b74ee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7993400640 1 -- 172.21.15.28:0/3583586102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c031710 con 0x7f7994071210 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7993400640 1 -- 172.21.15.28:0/3583586102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c02f050 con 0x7f7994071210 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7993400640 1 -- 172.21.15.28:0/3583586102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c031a90 con 0x7f7994071210 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 -- 172.21.15.28:0/3583586102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 msgr2=0x7f79940cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 --2- 172.21.15.28:0/3583586102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f79940cd870 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f79940c33c0 tx=0x7f797c02c480 comp rx=0 tx=0).stop 2024-09-13T17:09:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 -- 172.21.15.28:0/3583586102 shutdown_connections 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 --2- 172.21.15.28:0/3583586102 >> v2:172.21.15.189:3300/0 conn(0x7f79940cddb0 0x7f799415aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 --2- 172.21.15.28:0/3583586102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f79940cd870 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.043+0000 7f7999800640 1 -- 172.21.15.28:0/3583586102 >> 172.21.15.28:0/3583586102 conn(0x7f7994069e80 msgr2=0x7f799406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.044+0000 7f7999800640 1 -- 172.21.15.28:0/3583586102 shutdown_connections 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.044+0000 7f7999800640 1 -- 172.21.15.28:0/3583586102 wait complete. 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.045+0000 7f7999800640 1 Processor -- start 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.045+0000 7f7999800640 1 -- start start 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7994070e30 0x7f79940702a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79940cddb0 0x7f799413c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799414c810 con 0x7f7994071210 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799414caf0 con 0x7f79940cddb0 2024-09-13T17:09:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.046+0000 7f7999800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799414cdd0 con 0x7f7994070e30 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53200/0 (socket says 172.21.15.28:53200) 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 -- 172.21.15.28:0/1892520169 learned_addr learned my addr 172.21.15.28:0/1892520169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f7993e00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7994070e30 0x7f79940702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f7998e00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79940cddb0 0x7f799413c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 -- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7994070e30 msgr2=0x7f79940702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7994070e30 0x7f79940702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 -- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79940cddb0 msgr2=0x7f799413c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79940cddb0 0x7f799413c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 -- 172.21.15.28:0/1892520169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f797c0094d0 con 0x7f7994071210 2024-09-13T17:09:12.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.047+0000 7f798be00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f797c02c4c0 tx=0x7f797c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.048+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c03c830 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.048+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c03c9b0 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.048+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c03cd60 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.048+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799414d0b0 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.048+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799414bf00 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.049+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799414d5d0 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.050+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f797c03a070 con 0x7f7994071210 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.051+0000 7f7990c00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 0x7f797006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.051+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f797c0aa0e0 con 0x7f7994071210 2024-09-13T17:09:12.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.051+0000 7f7993e00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 0x7f797006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:12.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.052+0000 7f7993e00640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 0x7f797006d0f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7984000f60 tx=0x7f7984009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:12.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.055+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f799414d5d0 con 0x7f7994071210 2024-09-13T17:09:12.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:12.273+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f799414cf10 con 0x7f7994071210 2024-09-13T17:09:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.103+0000 7f7990c00640 1 -- 172.21.15.28:0/1892520169 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f799414cf10 con 0x7f7994071210 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 msgr2=0x7f797006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 0x7f797006d0f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7984000f60 tx=0x7f7984009040 comp rx=0 tx=0).stop 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 msgr2=0x7f799414b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f797c02c4c0 tx=0x7f797c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 shutdown_connections 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f797006ac30 0x7f797006d0f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79940cddb0 0x7f799413c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7994071210 0x7f799414b9c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 --2- 172.21.15.28:0/1892520169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7994070e30 0x7f79940702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 >> 172.21.15.28:0/1892520169 conn(0x7f7994069e80 msgr2=0x7f79940b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 shutdown_connections 2024-09-13T17:09:13.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.107+0000 7f7999800640 1 -- 172.21.15.28:0/1892520169 wait complete. 2024-09-13T17:09:13.128 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:09:13.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:09:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.291+0000 7fa373800640 1 Processor -- start 2024-09-13T17:09:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.291+0000 7fa373800640 1 -- start start 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa373800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c159770 0x7fa36c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa373800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa36c06f3f0 0x7fa36c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa373800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa36c06f920 con 0x7fa36c0add00 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa373800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa36c15b290 con 0x7fa36c06f3f0 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa373800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c15b570 con 0x7fa36c159770 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa36be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa36c06f3f0 0x7fa36c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa36be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa36c06f3f0 0x7fa36c070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53208/0 (socket says 172.21.15.28:53208) 2024-09-13T17:09:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.292+0000 7fa36be00640 1 -- 172.21.15.28:0/1812589189 learned_addr learned my addr 172.21.15.28:0/1812589189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.293+0000 7fa36be00640 1 -- 172.21.15.28:0/1812589189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c159770 msgr2=0x7fa36c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.293+0000 7fa36be00640 1 --2- 172.21.15.28:0/1812589189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c159770 0x7fa36c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.293+0000 7fa36be00640 1 -- 172.21.15.28:0/1812589189 >> v1:172.21.15.189:6789/0 conn(0x7fa36c0add00 legacy=0x7fa36c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.293+0000 7fa36be00640 1 -- 172.21.15.28:0/1812589189 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa36c15b850 con 0x7fa36c06f3f0 2024-09-13T17:09:13.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa36be00640 1 --2- 172.21.15.28:0/1812589189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c06f3f0 0x7fa36c070260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa358009870 tx=0x7fa35802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18fbbca98a4fa0c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa36b400640 1 -- 172.21.15.28:0/1812589189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35802ae90 con 0x7fa36c06f3f0 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa36b400640 1 -- 172.21.15.28:0/1812589189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa358032440 con 0x7fa36c06f3f0 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa36b400640 1 -- 172.21.15.28:0/1812589189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3580327d0 con 0x7fa36c06f3f0 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 -- 172.21.15.28:0/1812589189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c06f3f0 msgr2=0x7fa36c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 --2- 172.21.15.28:0/1812589189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c06f3f0 0x7fa36c070260 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa358009870 tx=0x7fa35802a180 comp rx=0 tx=0).stop 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 -- 172.21.15.28:0/1812589189 shutdown_connections 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 --2- 172.21.15.28:0/1812589189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c06f3f0 0x7fa36c070260 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 --2- 172.21.15.28:0/1812589189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c159770 0x7fa36c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.294+0000 7fa373800640 1 -- 172.21.15.28:0/1812589189 >> 172.21.15.28:0/1812589189 conn(0x7fa36c0b0270 msgr2=0x7fa36c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.295+0000 7fa373800640 1 -- 172.21.15.28:0/1812589189 shutdown_connections 2024-09-13T17:09:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.295+0000 7fa373800640 1 -- 172.21.15.28:0/1812589189 wait complete. 2024-09-13T17:09:13.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.296+0000 7fa373800640 1 Processor -- start 2024-09-13T17:09:13.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.296+0000 7fa373800640 1 -- start start 2024-09-13T17:09:13.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c06f3f0 0x7fa36c1f0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0add00 0x7fa36c0cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c14e9c0 con 0x7fa36c159770 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c1f13f0 con 0x7fa36c0add00 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.297+0000 7fa373800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa36c151d00 con 0x7fa36c06f3f0 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53212/0 (socket says 172.21.15.28:53212) 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 -- 172.21.15.28:0/2120440218 learned_addr learned my addr 172.21.15.28:0/2120440218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa36be00640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0add00 0x7fa36c0cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:13.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa370e00640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c06f3f0 0x7fa36c1f0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:13.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 -- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c06f3f0 msgr2=0x7fa36c1f0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:13.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c06f3f0 0x7fa36c1f0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 -- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0add00 msgr2=0x7fa36c0cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:13.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0add00 0x7fa36c0cc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 -- 172.21.15.28:0/2120440218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580094d0 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.298+0000 7fa371800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa36400b9e0 tx=0x7fa36400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.299+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36400cac0 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.299+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36400cc20 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.299+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa364014440 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.299+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa36c0b18c0 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.299+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa36c0b1e80 con 0x7fa36c159770 2024-09-13T17:09:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.300+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36c06fd20 con 0x7fa36c159770 2024-09-13T17:09:13.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.301+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa36401e440 con 0x7fa36c159770 2024-09-13T17:09:13.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.302+0000 7fa368c00640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 0x7fa34c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:13.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.302+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa364089080 con 0x7fa36c159770 2024-09-13T17:09:13.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.302+0000 7fa370e00640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 0x7fa34c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:13.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.303+0000 7fa370e00640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 0x7fa34c06d0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa360009990 tx=0x7fa360009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.306+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa3640bc9c0 con 0x7fa36c159770 2024-09-13T17:09:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:13.518+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa36c158b30 con 0x7fa36c159770 2024-09-13T17:09:14.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.105+0000 7fa368c00640 1 -- 172.21.15.28:0/2120440218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa3640190c0 con 0x7fa36c159770 2024-09-13T17:09:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 msgr2=0x7fa34c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 0x7fa34c06d0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa360009990 tx=0x7fa360009110 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 msgr2=0x7fa36c0ac740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa36400b9e0 tx=0x7fa36400bde0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 shutdown_connections 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa34c06ac00 0x7fa34c06d0c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa36c159770 0x7fa36c0ac740 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa36c0add00 0x7fa36c0cc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 --2- 172.21.15.28:0/2120440218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa36c06f3f0 0x7fa36c1f0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 >> 172.21.15.28:0/2120440218 conn(0x7fa36c0b0270 msgr2=0x7fa36c1581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 shutdown_connections 2024-09-13T17:09:14.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.107+0000 7fa373800640 1 -- 172.21.15.28:0/2120440218 wait complete. 2024-09-13T17:09:14.128 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:09:14.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:09:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.284+0000 7f407e400640 1 Processor -- start 2024-09-13T17:09:14.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.284+0000 7f407e400640 1 -- start start 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f407e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f407806f970 0x7f407815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f407e400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f407806c2d0 0x7f407806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f407e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407806cc70 con 0x7f40781592a0 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f407e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407815b0b0 con 0x7f407806f970 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f407e400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4078159d90 con 0x7f407806c2d0 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f4077400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f40781592a0 0x7f4078159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43124/0 (socket says 172.21.15.28:43124) 2024-09-13T17:09:14.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.285+0000 7f4077400640 1 -- 172.21.15.28:0/3293878758 learned_addr learned my addr 172.21.15.28:0/3293878758 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076a00640 1 --2- 172.21.15.28:0/3293878758 >> v2:172.21.15.28:3300/0 conn(0x7f407806f970 0x7f407815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4077e00640 1 --2- 172.21.15.28:0/3293878758 >> v2:172.21.15.189:3300/0 conn(0x7f407806c2d0 0x7f407806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076000640 1 -- 172.21.15.28:0/3293878758 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170334283 0 0) 0x7f407806cc70 con 0x7f40781592a0 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076000640 1 -- 172.21.15.28:0/3293878758 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003520 con 0x7f40781592a0 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076a00640 1 -- 172.21.15.28:0/3293878758 >> v2:172.21.15.189:3300/0 conn(0x7f407806c2d0 msgr2=0x7f407806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076a00640 1 --2- 172.21.15.28:0/3293878758 >> v2:172.21.15.189:3300/0 conn(0x7f407806c2d0 0x7f407806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076a00640 1 -- 172.21.15.28:0/3293878758 >> v1:172.21.15.28:6789/0 conn(0x7f40781592a0 legacy=0x7f4078159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.286+0000 7f4076a00640 1 -- 172.21.15.28:0/3293878758 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780b2040 con 0x7f407806f970 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.287+0000 7f4076a00640 1 --2- 172.21.15.28:0/3293878758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806f970 0x7f407815ab70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f40780b1dd0 tx=0x7f406402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aca8c6264666cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.287+0000 7f4076000640 1 -- 172.21.15.28:0/3293878758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064032440 con 0x7f407806f970 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f4076000640 1 -- 172.21.15.28:0/3293878758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40640325a0 con 0x7f407806f970 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f4076000640 1 -- 172.21.15.28:0/3293878758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064032910 con 0x7f407806f970 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 -- 172.21.15.28:0/3293878758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806f970 msgr2=0x7f407815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 --2- 172.21.15.28:0/3293878758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806f970 0x7f407815ab70 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f40780b1dd0 tx=0x7f406402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 -- 172.21.15.28:0/3293878758 shutdown_connections 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 --2- 172.21.15.28:0/3293878758 >> v2:172.21.15.189:3300/0 conn(0x7f407806c2d0 0x7f407806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 --2- 172.21.15.28:0/3293878758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806f970 0x7f407815ab70 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 -- 172.21.15.28:0/3293878758 >> 172.21.15.28:0/3293878758 conn(0x7f4078069e80 msgr2=0x7f407806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:14.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 -- 172.21.15.28:0/3293878758 shutdown_connections 2024-09-13T17:09:14.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.288+0000 7f407e400640 1 -- 172.21.15.28:0/3293878758 wait complete. 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.290+0000 7f407e400640 1 Processor -- start 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.290+0000 7f407e400640 1 -- start start 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f407806f970 0x7f40780cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40781592a0 0x7f40781f4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4078151000 con 0x7f407806c2d0 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4078150d20 con 0x7f40781592a0 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f407e400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40780cda70 con 0x7f407806f970 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4077400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4077400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47750/0 (socket says 172.21.15.28:47750) 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4077400640 1 -- 172.21.15.28:0/1883515473 learned_addr learned my addr 172.21.15.28:0/1883515473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4077e00640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40781592a0 0x7f40781f4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4076a00640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f407806f970 0x7f40780cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.291+0000 7f4077400640 1 -- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f407806f970 msgr2=0x7f40780cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f4077400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f407806f970 0x7f40780cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f4077400640 1 -- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40781592a0 msgr2=0x7f40781f4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f4077400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40781592a0 0x7f40781f4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f4077400640 1 -- 172.21.15.28:0/1883515473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4064009400 con 0x7f407806c2d0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f4077400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f406800c500 tx=0x7f406800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068005ce0 con 0x7f407806c2d0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4068005e40 con 0x7f407806c2d0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4068007330 con 0x7f407806c2d0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40780cdd50 con 0x7f407806c2d0 2024-09-13T17:09:14.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.292+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40781509d0 con 0x7f407806c2d0 2024-09-13T17:09:14.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.293+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40780b3740 con 0x7f407806c2d0 2024-09-13T17:09:14.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.294+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4068007550 con 0x7f407806c2d0 2024-09-13T17:09:14.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.295+0000 7f406f400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 0x7f405806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.295+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4068088a70 con 0x7f407806c2d0 2024-09-13T17:09:14.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.296+0000 7f4076a00640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 0x7f405806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.296+0000 7f4076a00640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 0x7f405806d0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4064034000 tx=0x7f406402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.299+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4068053d00 con 0x7f407806c2d0 2024-09-13T17:09:14.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.517+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4078157750 con 0x7f407806c2d0 2024-09-13T17:09:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.518+0000 7f406f400640 1 -- 172.21.15.28:0/1883515473 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f4068017020 con 0x7f407806c2d0 2024-09-13T17:09:14.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:14.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:09:14.102658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:09:14.520 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-13T17:09:14.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 msgr2=0x7f405806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 0x7f405806d0c0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4064034000 tx=0x7f406402e000 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 msgr2=0x7f4078144760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f406800c500 tx=0x7f406800c9c0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 shutdown_connections 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f405806ac00 0x7f405806d0c0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40781592a0 0x7f40781f4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f407806f970 0x7f40780cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 --2- 172.21.15.28:0/1883515473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f407806c2d0 0x7f4078144760 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 >> 172.21.15.28:0/1883515473 conn(0x7f4078069e80 msgr2=0x7f4078156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 shutdown_connections 2024-09-13T17:09:14.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.520+0000 7f407e400640 1 -- 172.21.15.28:0/1883515473 wait complete. 2024-09-13T17:09:14.539 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:09:14.605 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:09:14.603+0000 7f3014200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:09:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.693+0000 7f7628200640 1 Processor -- start 2024-09-13T17:09:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.693+0000 7f7628200640 1 -- start start 2024-09-13T17:09:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7628200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f762015c750 0x7f76200b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7628200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76200aeca0 0x7f76200ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7628200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76200b2b30 con 0x7f762015cb30 2024-09-13T17:09:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7628200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762006e000 con 0x7f762015c750 2024-09-13T17:09:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7628200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762006e2e0 con 0x7f76200aeca0 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7626200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f762015cb30 0x7f762006d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43126/0 (socket says 172.21.15.28:43126) 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7625800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f762015c750 0x7f76200b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.694+0000 7f7626200640 1 -- 172.21.15.28:0/1563010543 learned_addr learned my addr 172.21.15.28:0/1563010543 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f7624e00640 1 --2- 172.21.15.28:0/1563010543 >> v2:172.21.15.189:3300/0 conn(0x7f76200aeca0 0x7f76200ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f7625800640 1 -- 172.21.15.28:0/1563010543 >> v2:172.21.15.189:3300/0 conn(0x7f76200aeca0 msgr2=0x7f76200ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f7625800640 1 --2- 172.21.15.28:0/1563010543 >> v2:172.21.15.189:3300/0 conn(0x7f76200aeca0 0x7f76200ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f7625800640 1 -- 172.21.15.28:0/1563010543 >> v1:172.21.15.28:6789/0 conn(0x7f762015cb30 legacy=0x7f762006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f7625800640 1 -- 172.21.15.28:0/1563010543 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762006c5a0 con 0x7f762015c750 2024-09-13T17:09:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.695+0000 7f761fe00640 1 -- 172.21.15.28:0/1563010543 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711723501 0 0) 0x7f7618001490 con 0x7f762015cb30 2024-09-13T17:09:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.696+0000 7f7625800640 1 --2- 172.21.15.28:0/1563010543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762015c750 0x7f76200b25f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76200b3a90 tx=0x7f761402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180288ab3cab9e5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.696+0000 7f761fe00640 1 -- 172.21.15.28:0/1563010543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614032440 con 0x7f762015c750 2024-09-13T17:09:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.696+0000 7f761fe00640 1 -- 172.21.15.28:0/1563010543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76140325a0 con 0x7f762015c750 2024-09-13T17:09:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.696+0000 7f761fe00640 1 -- 172.21.15.28:0/1563010543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614032930 con 0x7f762015c750 2024-09-13T17:09:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 -- 172.21.15.28:0/1563010543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762015c750 msgr2=0x7f76200b25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 --2- 172.21.15.28:0/1563010543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762015c750 0x7f76200b25f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f76200b3a90 tx=0x7f761402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 -- 172.21.15.28:0/1563010543 shutdown_connections 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 --2- 172.21.15.28:0/1563010543 >> v2:172.21.15.189:3300/0 conn(0x7f76200aeca0 0x7f76200ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 --2- 172.21.15.28:0/1563010543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f762015c750 0x7f76200b25f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 -- 172.21.15.28:0/1563010543 >> 172.21.15.28:0/1563010543 conn(0x7f76200cf580 msgr2=0x7f76200cf990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 -- 172.21.15.28:0/1563010543 shutdown_connections 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.697+0000 7f7628200640 1 -- 172.21.15.28:0/1563010543 wait complete. 2024-09-13T17:09:14.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.699+0000 7f7628200640 1 Processor -- start 2024-09-13T17:09:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.699+0000 7f7628200640 1 -- start start 2024-09-13T17:09:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762015c750 0x7f76200add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 0x7f76200ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762015e8e0 con 0x7f76200aeca0 2024-09-13T17:09:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762015ebc0 con 0x7f762015c750 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7628200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762015eea0 con 0x7f762015cb30 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47766/0 (socket says 172.21.15.28:47766) 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 -- 172.21.15.28:0/2919019393 learned_addr learned my addr 172.21.15.28:0/2919019393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7624e00640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762015c750 0x7f76200add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7626200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 0x7f76200ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 -- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 msgr2=0x7f76200ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 0x7f76200ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 -- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762015c750 msgr2=0x7f76200add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762015c750 0x7f76200add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7625800640 1 -- 172.21.15.28:0/2919019393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009400 con 0x7f76200aeca0 2024-09-13T17:09:14.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.700+0000 7f7626200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 0x7f76200ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:14.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f7625800640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7614009890 tx=0x7f76140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614032440 con 0x7f76200aeca0 2024-09-13T17:09:14.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7614004250 con 0x7f76200aeca0 2024-09-13T17:09:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762015f180 con 0x7f76200aeca0 2024-09-13T17:09:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614033ea0 con 0x7f76200aeca0 2024-09-13T17:09:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.701+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762014ecb0 con 0x7f76200aeca0 2024-09-13T17:09:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.702+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762004e210 con 0x7f76200aeca0 2024-09-13T17:09:14.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.703+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7614039050 con 0x7f76200aeca0 2024-09-13T17:09:14.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.704+0000 7f761d600640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 0x7f760006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:14.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.704+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76140a9c30 con 0x7f76200aeca0 2024-09-13T17:09:14.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.705+0000 7f7624e00640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 0x7f760006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:14.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.705+0000 7f7624e00640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 0x7f760006d110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f76201f6060 tx=0x7f760c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:14.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.708+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7614074ea0 con 0x7f76200aeca0 2024-09-13T17:09:14.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.886+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7620073ed0 con 0x7f76200aeca0 2024-09-13T17:09:14.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.887+0000 7f761d600640 1 -- 172.21.15.28:0/2919019393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16569 (secure 0 0 0) 0x7f761404b090 con 0x7f76200aeca0 2024-09-13T17:09:14.889 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:14.890 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":34,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:09:12.093680+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.889+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 msgr2=0x7f760006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.889+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 0x7f760006d110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f76201f6060 tx=0x7f760c009190 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.889+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 msgr2=0x7f762014e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.889+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7614009890 tx=0x7f76140046c0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 shutdown_connections 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f760006ac50 0x7f760006d110 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f762015cb30 0x7f76200ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f762015c750 0x7f76200add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 --2- 172.21.15.28:0/2919019393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76200aeca0 0x7f762014e770 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 >> 172.21.15.28:0/2919019393 conn(0x7f76200cf580 msgr2=0x7f76200b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:14.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 shutdown_connections 2024-09-13T17:09:14.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:14.890+0000 7f7628200640 1 -- 172.21.15.28:0/2919019393 wait complete. 2024-09-13T17:09:14.908 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:09:14.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:09:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.062+0000 7f8162800640 1 Processor -- start 2024-09-13T17:09:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.063+0000 7f8162800640 1 -- start start 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 0x7f815c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c0cf3d0 0x7f815c0cf830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f815c160580 0x7f815c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815c0d1690 con 0x7f815c160580 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f815c0b6200 con 0x7f815c15c970 2024-09-13T17:09:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8162800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0cfd70 con 0x7f815c0cf3d0 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f815c160580 0x7f815c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f815c160580 0x7f815c1609f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47778/0 (socket says 172.21.15.28:47778) 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 -- 172.21.15.28:0/776115068 learned_addr learned my addr 172.21.15.28:0/776115068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815b400640 1 --2- 172.21.15.28:0/776115068 >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 0x7f815c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f8153e00640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c0cf3d0 0x7f815c0cf830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 -- 172.21.15.28:0/776115068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c0cf3d0 msgr2=0x7f815c0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c0cf3d0 0x7f815c0cf830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 -- 172.21.15.28:0/776115068 >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 msgr2=0x7f815c0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 --2- 172.21.15.28:0/776115068 >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 0x7f815c0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815be00640 1 -- 172.21.15.28:0/776115068 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c160f30 con 0x7f815c160580 2024-09-13T17:09:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.064+0000 7f815b400640 1 --2- 172.21.15.28:0/776115068 >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 0x7f815c0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.065+0000 7f815be00640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c160580 0x7f815c1609f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f815c0d3b50 tx=0x7f815402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4688bb615b6a1bfc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f815aa00640 1 -- 172.21.15.28:0/776115068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154031600 con 0x7f815c160580 2024-09-13T17:09:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f815aa00640 1 -- 172.21.15.28:0/776115068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f815402d050 con 0x7f815c160580 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f815aa00640 1 -- 172.21.15.28:0/776115068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8154031980 con 0x7f815c160580 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 -- 172.21.15.28:0/776115068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c160580 msgr2=0x7f815c1609f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c160580 0x7f815c1609f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f815c0d3b50 tx=0x7f815402a610 comp rx=0 tx=0).stop 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 -- 172.21.15.28:0/776115068 shutdown_connections 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c160580 0x7f815c1609f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 --2- 172.21.15.28:0/776115068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c0cf3d0 0x7f815c0cf830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 --2- 172.21.15.28:0/776115068 >> v2:172.21.15.189:3300/0 conn(0x7f815c15c970 0x7f815c0b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 -- 172.21.15.28:0/776115068 >> 172.21.15.28:0/776115068 conn(0x7f815c070ad0 msgr2=0x7f815c0b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 -- 172.21.15.28:0/776115068 shutdown_connections 2024-09-13T17:09:15.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.066+0000 7f8162800640 1 -- 172.21.15.28:0/776115068 wait complete. 2024-09-13T17:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.068+0000 7f8162800640 1 Processor -- start 2024-09-13T17:09:15.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.068+0000 7f8162800640 1 -- start start 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f815c15c970 0x7f815c1f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 0x7f815c0aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c15f8d0 con 0x7f815c0cf3d0 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1519c0 con 0x7f815c15c970 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f8162800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c15ce80 con 0x7f815c160580 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f815b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f815b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47794/0 (socket says 172.21.15.28:47794) 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.069+0000 7f815b400640 1 -- 172.21.15.28:0/3136414700 learned_addr learned my addr 172.21.15.28:0/3136414700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f8153e00640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f815c15c970 0x7f815c1f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815be00640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 0x7f815c0aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 -- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 msgr2=0x7f815c0aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 0x7f815c0aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 -- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f815c15c970 msgr2=0x7f815c1f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f815c15c970 0x7f815c1f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 -- 172.21.15.28:0/3136414700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8154009b30 con 0x7f815c0cf3d0 2024-09-13T17:09:15.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815be00640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 0x7f815c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f815b400640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e7e0 tx=0x7f814c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.070+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c004030 con 0x7f815c0cf3d0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.071+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c004190 con 0x7f815c0cf3d0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.071+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c15d160 con 0x7f815c0cf3d0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.071+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c004520 con 0x7f815c0cf3d0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.071+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c15d720 con 0x7f815c0cf3d0 2024-09-13T17:09:15.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.072+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c0d12b0 con 0x7f815c0cf3d0 2024-09-13T17:09:15.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.073+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f814c00c6d0 con 0x7f815c0cf3d0 2024-09-13T17:09:15.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.073+0000 7f8153400640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 0x7f813806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.074+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f814c088fe0 con 0x7f815c0cf3d0 2024-09-13T17:09:15.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.074+0000 7f8153e00640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 0x7f813806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.075+0000 7f8153e00640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 0x7f813806d0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f815c0b6ac0 tx=0x7f8144008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.078+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f814c054260 con 0x7f815c0cf3d0 2024-09-13T17:09:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.253+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f815c161470 con 0x7f813806ac00 2024-09-13T17:09:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.263+0000 7f8153400640 1 -- 172.21.15.28:0/3136414700 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289143 (secure 0 0 0) 0x7f8144010960 con 0x7f813806ac00 2024-09-13T17:09:15.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.266+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 msgr2=0x7f813806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.266+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 0x7f813806d0c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f815c0b6ac0 tx=0x7f8144008040 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.266+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 msgr2=0x7f815c151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.266+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f814c00e7e0 tx=0x7f814c00eca0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 shutdown_connections 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813806ac00 0x7f813806d0c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f815c160580 0x7f815c0aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f815c15c970 0x7f815c1f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 --2- 172.21.15.28:0/3136414700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f815c0cf3d0 0x7f815c151480 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 >> 172.21.15.28:0/3136414700 conn(0x7f815c070ad0 msgr2=0x7f815c06a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 shutdown_connections 2024-09-13T17:09:15.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.267+0000 7f8162800640 1 -- 172.21.15.28:0/3136414700 wait complete. 2024-09-13T17:09:15.285 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T17:09:14.900040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236752,"kb_used_data":4176,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754737968,"statfs":{"total":773094113280,"available":772851679232,"internally_reserved":0,"allocated":4276224,"data_stored":2372278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.249225"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826697+0000","last_change":"2024-09-13T17:09:11.114498+0000","last_active":"2024-09-13T17:09:12.826697+0000","last_peered":"2024-09-13T17:09:12.826697+0000","last_clean":"2024-09-13T17:09:12.826697+0000","last_became_active":"2024-09-13T17:09:11.114077+0000","last_became_peered":"2024-09-13T17:09:11.114077+0000","last_unstale":"2024-09-13T17:09:12.826697+0000","last_undegraded":"2024-09-13T17:09:12.826697+0000","last_fullsized":"2024-09-13T17:09:12.826697+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:02.757328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103517+0000","last_change":"2024-09-13T17:09:11.130697+0000","last_active":"2024-09-13T17:09:12.103517+0000","last_peered":"2024-09-13T17:09:12.103517+0000","last_clean":"2024-09-13T17:09:12.103517+0000","last_became_active":"2024-09-13T17:09:11.129323+0000","last_became_peered":"2024-09-13T17:09:11.129323+0000","last_unstale":"2024-09-13T17:09:12.103517+0000","last_undegraded":"2024-09-13T17:09:12.103517+0000","last_fullsized":"2024-09-13T17:09:12.103517+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:52.626256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105973+0000","last_change":"2024-09-13T17:09:11.124311+0000","last_active":"2024-09-13T17:09:12.105973+0000","last_peered":"2024-09-13T17:09:12.105973+0000","last_clean":"2024-09-13T17:09:12.105973+0000","last_became_active":"2024-09-13T17:09:11.124015+0000","last_became_peered":"2024-09-13T17:09:11.124015+0000","last_unstale":"2024-09-13T17:09:12.105973+0000","last_undegraded":"2024-09-13T17:09:12.105973+0000","last_fullsized":"2024-09-13T17:09:12.105973+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475414+0000","last_change":"2024-09-13T17:09:11.127945+0000","last_active":"2024-09-13T17:09:12.475414+0000","last_peered":"2024-09-13T17:09:12.475414+0000","last_clean":"2024-09-13T17:09:12.475414+0000","last_became_active":"2024-09-13T17:09:11.127711+0000","last_became_peered":"2024-09-13T17:09:11.127711+0000","last_unstale":"2024-09-13T17:09:12.475414+0000","last_undegraded":"2024-09-13T17:09:12.475414+0000","last_fullsized":"2024-09-13T17:09:12.475414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125413+0000","last_change":"2024-09-13T17:09:11.113062+0000","last_active":"2024-09-13T17:09:11.125413+0000","last_peered":"2024-09-13T17:09:11.125413+0000","last_clean":"2024-09-13T17:09:11.125413+0000","last_became_active":"2024-09-13T17:09:11.112493+0000","last_became_peered":"2024-09-13T17:09:11.112493+0000","last_unstale":"2024-09-13T17:09:11.125413+0000","last_undegraded":"2024-09-13T17:09:11.125413+0000","last_fullsized":"2024-09-13T17:09:11.125413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103016+0000","last_change":"2024-09-13T17:09:11.122759+0000","last_active":"2024-09-13T17:09:12.103016+0000","last_peered":"2024-09-13T17:09:12.103016+0000","last_clean":"2024-09-13T17:09:12.103016+0000","last_became_active":"2024-09-13T17:09:11.121272+0000","last_became_peered":"2024-09-13T17:09:11.121272+0000","last_unstale":"2024-09-13T17:09:12.103016+0000","last_undegraded":"2024-09-13T17:09:12.103016+0000","last_fullsized":"2024-09-13T17:09:12.103016+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108400+0000","last_change":"2024-09-13T17:09:11.114090+0000","last_active":"2024-09-13T17:09:12.108400+0000","last_peered":"2024-09-13T17:09:12.108400+0000","last_clean":"2024-09-13T17:09:12.108400+0000","last_became_active":"2024-09-13T17:09:11.113798+0000","last_became_peered":"2024-09-13T17:09:11.113798+0000","last_unstale":"2024-09-13T17:09:12.108400+0000","last_undegraded":"2024-09-13T17:09:12.108400+0000","last_fullsized":"2024-09-13T17:09:12.108400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:02.205155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108218+0000","last_change":"2024-09-13T17:09:11.113340+0000","last_active":"2024-09-13T17:09:12.108218+0000","last_peered":"2024-09-13T17:09:12.108218+0000","last_clean":"2024-09-13T17:09:12.108218+0000","last_became_active":"2024-09-13T17:09:11.112989+0000","last_became_peered":"2024-09-13T17:09:11.112989+0000","last_unstale":"2024-09-13T17:09:12.108218+0000","last_undegraded":"2024-09-13T17:09:12.108218+0000","last_fullsized":"2024-09-13T17:09:12.108218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.130728+0000","last_change":"2024-09-13T17:09:11.126185+0000","last_active":"2024-09-13T17:09:11.130728+0000","last_peered":"2024-09-13T17:09:11.130728+0000","last_clean":"2024-09-13T17:09:11.130728+0000","last_became_active":"2024-09-13T17:09:11.125901+0000","last_became_peered":"2024-09-13T17:09:11.125901+0000","last_unstale":"2024-09-13T17:09:11.130728+0000","last_undegraded":"2024-09-13T17:09:11.130728+0000","last_fullsized":"2024-09-13T17:09:11.130728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:20.961969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475676+0000","last_change":"2024-09-13T17:09:11.134007+0000","last_active":"2024-09-13T17:09:12.475676+0000","last_peered":"2024-09-13T17:09:12.475676+0000","last_clean":"2024-09-13T17:09:12.475676+0000","last_became_active":"2024-09-13T17:09:11.133652+0000","last_became_peered":"2024-09-13T17:09:11.133652+0000","last_unstale":"2024-09-13T17:09:12.475676+0000","last_undegraded":"2024-09-13T17:09:12.475676+0000","last_fullsized":"2024-09-13T17:09:12.475676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:15.296969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108787+0000","last_change":"2024-09-13T17:09:11.129885+0000","last_active":"2024-09-13T17:09:12.108787+0000","last_peered":"2024-09-13T17:09:12.108787+0000","last_clean":"2024-09-13T17:09:12.108787+0000","last_became_active":"2024-09-13T17:09:11.129668+0000","last_became_peered":"2024-09-13T17:09:11.129668+0000","last_unstale":"2024-09-13T17:09:12.108787+0000","last_undegraded":"2024-09-13T17:09:12.108787+0000","last_fullsized":"2024-09-13T17:09:12.108787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105895+0000","last_change":"2024-09-13T17:09:11.120792+0000","last_active":"2024-09-13T17:09:12.105895+0000","last_peered":"2024-09-13T17:09:12.105895+0000","last_clean":"2024-09-13T17:09:12.105895+0000","last_became_active":"2024-09-13T17:09:11.120497+0000","last_became_peered":"2024-09-13T17:09:11.120497+0000","last_unstale":"2024-09-13T17:09:12.105895+0000","last_undegraded":"2024-09-13T17:09:12.105895+0000","last_fullsized":"2024-09-13T17:09:12.105895+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.129091+0000","last_change":"2024-09-13T17:09:11.125541+0000","last_active":"2024-09-13T17:09:11.129091+0000","last_peered":"2024-09-13T17:09:11.129091+0000","last_clean":"2024-09-13T17:09:11.129091+0000","last_became_active":"2024-09-13T17:09:11.125257+0000","last_became_peered":"2024-09-13T17:09:11.125257+0000","last_unstale":"2024-09-13T17:09:11.129091+0000","last_undegraded":"2024-09-13T17:09:11.129091+0000","last_fullsized":"2024-09-13T17:09:11.129091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:50.054986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103117+0000","last_change":"2024-09-13T17:09:11.123092+0000","last_active":"2024-09-13T17:09:12.103117+0000","last_peered":"2024-09-13T17:09:12.103117+0000","last_clean":"2024-09-13T17:09:12.103117+0000","last_became_active":"2024-09-13T17:09:11.122870+0000","last_became_peered":"2024-09-13T17:09:11.122870+0000","last_unstale":"2024-09-13T17:09:12.103117+0000","last_undegraded":"2024-09-13T17:09:12.103117+0000","last_fullsized":"2024-09-13T17:09:12.103117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.599876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105745+0000","last_change":"2024-09-13T17:09:11.109715+0000","last_active":"2024-09-13T17:09:12.105745+0000","last_peered":"2024-09-13T17:09:12.105745+0000","last_clean":"2024-09-13T17:09:12.105745+0000","last_became_active":"2024-09-13T17:09:11.107677+0000","last_became_peered":"2024-09-13T17:09:11.107677+0000","last_unstale":"2024-09-13T17:09:12.105745+0000","last_undegraded":"2024-09-13T17:09:12.105745+0000","last_fullsized":"2024-09-13T17:09:12.105745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108481+0000","last_change":"2024-09-13T17:09:11.122482+0000","last_active":"2024-09-13T17:09:12.108481+0000","last_peered":"2024-09-13T17:09:12.108481+0000","last_clean":"2024-09-13T17:09:12.108481+0000","last_became_active":"2024-09-13T17:09:11.121649+0000","last_became_peered":"2024-09-13T17:09:11.121649+0000","last_unstale":"2024-09-13T17:09:12.108481+0000","last_undegraded":"2024-09-13T17:09:12.108481+0000","last_fullsized":"2024-09-13T17:09:12.108481+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108789+0000","last_change":"2024-09-13T17:09:11.115663+0000","last_active":"2024-09-13T17:09:12.108789+0000","last_peered":"2024-09-13T17:09:12.108789+0000","last_clean":"2024-09-13T17:09:12.108789+0000","last_became_active":"2024-09-13T17:09:11.113066+0000","last_became_peered":"2024-09-13T17:09:11.113066+0000","last_unstale":"2024-09-13T17:09:12.108789+0000","last_undegraded":"2024-09-13T17:09:12.108789+0000","last_fullsized":"2024-09-13T17:09:12.108789+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103500+0000","last_change":"2024-09-13T17:09:11.126604+0000","last_active":"2024-09-13T17:09:12.103500+0000","last_peered":"2024-09-13T17:09:12.103500+0000","last_clean":"2024-09-13T17:09:12.103500+0000","last_became_active":"2024-09-13T17:09:11.126263+0000","last_became_peered":"2024-09-13T17:09:11.126263+0000","last_unstale":"2024-09-13T17:09:12.103500+0000","last_undegraded":"2024-09-13T17:09:12.103500+0000","last_fullsized":"2024-09-13T17:09:12.103500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103350+0000","last_change":"2024-09-13T17:09:11.133584+0000","last_active":"2024-09-13T17:09:12.103350+0000","last_peered":"2024-09-13T17:09:12.103350+0000","last_clean":"2024-09-13T17:09:12.103350+0000","last_became_active":"2024-09-13T17:09:11.133027+0000","last_became_peered":"2024-09-13T17:09:11.133027+0000","last_unstale":"2024-09-13T17:09:12.103350+0000","last_undegraded":"2024-09-13T17:09:12.103350+0000","last_fullsized":"2024-09-13T17:09:12.103350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126715+0000","last_change":"2024-09-13T17:09:11.117305+0000","last_active":"2024-09-13T17:09:11.126715+0000","last_peered":"2024-09-13T17:09:11.126715+0000","last_clean":"2024-09-13T17:09:11.126715+0000","last_became_active":"2024-09-13T17:09:11.117081+0000","last_became_peered":"2024-09-13T17:09:11.117081+0000","last_unstale":"2024-09-13T17:09:11.126715+0000","last_undegraded":"2024-09-13T17:09:11.126715+0000","last_fullsized":"2024-09-13T17:09:11.126715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:49.078243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122786+0000","last_change":"2024-09-13T17:09:11.115971+0000","last_active":"2024-09-13T17:09:11.122786+0000","last_peered":"2024-09-13T17:09:11.122786+0000","last_clean":"2024-09-13T17:09:11.122786+0000","last_became_active":"2024-09-13T17:09:11.115562+0000","last_became_peered":"2024-09-13T17:09:11.115562+0000","last_unstale":"2024-09-13T17:09:11.122786+0000","last_undegraded":"2024-09-13T17:09:11.122786+0000","last_fullsized":"2024-09-13T17:09:11.122786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125446+0000","last_change":"2024-09-13T17:09:11.121800+0000","last_active":"2024-09-13T17:09:11.125446+0000","last_peered":"2024-09-13T17:09:11.125446+0000","last_clean":"2024-09-13T17:09:11.125446+0000","last_became_active":"2024-09-13T17:09:11.121495+0000","last_became_peered":"2024-09-13T17:09:11.121495+0000","last_unstale":"2024-09-13T17:09:11.125446+0000","last_undegraded":"2024-09-13T17:09:11.125446+0000","last_fullsized":"2024-09-13T17:09:11.125446+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:09.468121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164726+0000","last_change":"2024-09-13T17:09:11.111698+0000","last_active":"2024-09-13T17:09:12.164726+0000","last_peered":"2024-09-13T17:09:12.164726+0000","last_clean":"2024-09-13T17:09:12.164726+0000","last_became_active":"2024-09-13T17:09:11.111548+0000","last_became_peered":"2024-09-13T17:09:11.111548+0000","last_unstale":"2024-09-13T17:09:12.164726+0000","last_undegraded":"2024-09-13T17:09:12.164726+0000","last_fullsized":"2024-09-13T17:09:12.164726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105756+0000","last_change":"2024-09-13T17:09:11.109940+0000","last_active":"2024-09-13T17:09:12.105756+0000","last_peered":"2024-09-13T17:09:12.105756+0000","last_clean":"2024-09-13T17:09:12.105756+0000","last_became_active":"2024-09-13T17:09:11.109658+0000","last_became_peered":"2024-09-13T17:09:11.109658+0000","last_unstale":"2024-09-13T17:09:12.105756+0000","last_undegraded":"2024-09-13T17:09:12.105756+0000","last_fullsized":"2024-09-13T17:09:12.105756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.599876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475696+0000","last_change":"2024-09-13T17:09:11.127029+0000","last_active":"2024-09-13T17:09:12.475696+0000","last_peered":"2024-09-13T17:09:12.475696+0000","last_clean":"2024-09-13T17:09:12.475696+0000","last_became_active":"2024-09-13T17:09:11.126794+0000","last_became_peered":"2024-09-13T17:09:11.126794+0000","last_unstale":"2024-09-13T17:09:12.475696+0000","last_undegraded":"2024-09-13T17:09:12.475696+0000","last_fullsized":"2024-09-13T17:09:12.475696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103585+0000","last_change":"2024-09-13T17:09:11.129233+0000","last_active":"2024-09-13T17:09:12.103585+0000","last_peered":"2024-09-13T17:09:12.103585+0000","last_clean":"2024-09-13T17:09:12.103585+0000","last_became_active":"2024-09-13T17:09:11.127089+0000","last_became_peered":"2024-09-13T17:09:11.127089+0000","last_unstale":"2024-09-13T17:09:12.103585+0000","last_undegraded":"2024-09-13T17:09:12.103585+0000","last_fullsized":"2024-09-13T17:09:12.103585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475647+0000","last_change":"2024-09-13T17:09:11.126487+0000","last_active":"2024-09-13T17:09:12.475647+0000","last_peered":"2024-09-13T17:09:12.475647+0000","last_clean":"2024-09-13T17:09:12.475647+0000","last_became_active":"2024-09-13T17:09:11.126246+0000","last_became_peered":"2024-09-13T17:09:11.126246+0000","last_unstale":"2024-09-13T17:09:12.475647+0000","last_undegraded":"2024-09-13T17:09:12.475647+0000","last_fullsized":"2024-09-13T17:09:12.475647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:44.149902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105883+0000","last_change":"2024-09-13T17:09:11.123482+0000","last_active":"2024-09-13T17:09:12.105883+0000","last_peered":"2024-09-13T17:09:12.105883+0000","last_clean":"2024-09-13T17:09:12.105883+0000","last_became_active":"2024-09-13T17:09:11.123180+0000","last_became_peered":"2024-09-13T17:09:11.123180+0000","last_unstale":"2024-09-13T17:09:12.105883+0000","last_undegraded":"2024-09-13T17:09:12.105883+0000","last_fullsized":"2024-09-13T17:09:12.105883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122802+0000","last_change":"2024-09-13T17:09:11.116112+0000","last_active":"2024-09-13T17:09:11.122802+0000","last_peered":"2024-09-13T17:09:11.122802+0000","last_clean":"2024-09-13T17:09:11.122802+0000","last_became_active":"2024-09-13T17:09:11.115876+0000","last_became_peered":"2024-09-13T17:09:11.115876+0000","last_unstale":"2024-09-13T17:09:11.122802+0000","last_undegraded":"2024-09-13T17:09:11.122802+0000","last_fullsized":"2024-09-13T17:09:11.122802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108641+0000","last_change":"2024-09-13T17:09:11.112400+0000","last_active":"2024-09-13T17:09:12.108641+0000","last_peered":"2024-09-13T17:09:12.108641+0000","last_clean":"2024-09-13T17:09:12.108641+0000","last_became_active":"2024-09-13T17:09:11.112098+0000","last_became_peered":"2024-09-13T17:09:11.112098+0000","last_unstale":"2024-09-13T17:09:12.108641+0000","last_undegraded":"2024-09-13T17:09:12.108641+0000","last_fullsized":"2024-09-13T17:09:12.108641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826637+0000","last_change":"2024-09-13T17:09:11.113348+0000","last_active":"2024-09-13T17:09:12.826637+0000","last_peered":"2024-09-13T17:09:12.826637+0000","last_clean":"2024-09-13T17:09:12.826637+0000","last_became_active":"2024-09-13T17:09:11.111583+0000","last_became_peered":"2024-09-13T17:09:11.111583+0000","last_unstale":"2024-09-13T17:09:12.826637+0000","last_undegraded":"2024-09-13T17:09:12.826637+0000","last_fullsized":"2024-09-13T17:09:12.826637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:55.503431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108222+0000","last_change":"2024-09-13T17:09:11.117549+0000","last_active":"2024-09-13T17:09:12.108222+0000","last_peered":"2024-09-13T17:09:12.108222+0000","last_clean":"2024-09-13T17:09:12.108222+0000","last_became_active":"2024-09-13T17:09:11.116981+0000","last_became_peered":"2024-09-13T17:09:11.116981+0000","last_unstale":"2024-09-13T17:09:12.108222+0000","last_undegraded":"2024-09-13T17:09:12.108222+0000","last_fullsized":"2024-09-13T17:09:12.108222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164997+0000","last_change":"2024-09-13T17:09:11.111335+0000","last_active":"2024-09-13T17:09:12.164997+0000","last_peered":"2024-09-13T17:09:12.164997+0000","last_clean":"2024-09-13T17:09:12.164997+0000","last_became_active":"2024-09-13T17:09:11.111049+0000","last_became_peered":"2024-09-13T17:09:11.111049+0000","last_unstale":"2024-09-13T17:09:12.164997+0000","last_undegraded":"2024-09-13T17:09:12.164997+0000","last_fullsized":"2024-09-13T17:09:12.164997+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:50.054986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826878+0000","last_change":"2024-09-13T17:09:10.681887+0000","last_active":"2024-09-13T17:09:12.826878+0000","last_peered":"2024-09-13T17:09:12.826878+0000","last_clean":"2024-09-13T17:09:12.826878+0000","last_became_active":"2024-09-13T17:09:10.680423+0000","last_became_peered":"2024-09-13T17:09:10.680423+0000","last_unstale":"2024-09-13T17:09:12.826878+0000","last_undegraded":"2024-09-13T17:09:12.826878+0000","last_fullsized":"2024-09-13T17:09:12.826878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:21.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826795+0000","last_change":"2024-09-13T17:09:11.112682+0000","last_active":"2024-09-13T17:09:12.826795+0000","last_peered":"2024-09-13T17:09:12.826795+0000","last_clean":"2024-09-13T17:09:12.826795+0000","last_became_active":"2024-09-13T17:09:11.111830+0000","last_became_peered":"2024-09-13T17:09:11.111830+0000","last_unstale":"2024-09-13T17:09:12.826795+0000","last_undegraded":"2024-09-13T17:09:12.826795+0000","last_fullsized":"2024-09-13T17:09:12.826795+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:51.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164973+0000","last_change":"2024-09-13T17:09:10.680542+0000","last_active":"2024-09-13T17:09:12.164973+0000","last_peered":"2024-09-13T17:09:12.164973+0000","last_clean":"2024-09-13T17:09:12.164973+0000","last_became_active":"2024-09-13T17:09:10.680153+0000","last_became_peered":"2024-09-13T17:09:10.680153+0000","last_unstale":"2024-09-13T17:09:12.164973+0000","last_undegraded":"2024-09-13T17:09:12.164973+0000","last_fullsized":"2024-09-13T17:09:12.164973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122855+0000","last_change":"2024-09-13T17:09:11.119429+0000","last_active":"2024-09-13T17:09:11.122855+0000","last_peered":"2024-09-13T17:09:11.122855+0000","last_clean":"2024-09-13T17:09:11.122855+0000","last_became_active":"2024-09-13T17:09:11.119138+0000","last_became_peered":"2024-09-13T17:09:11.119138+0000","last_unstale":"2024-09-13T17:09:11.122855+0000","last_undegraded":"2024-09-13T17:09:11.122855+0000","last_fullsized":"2024-09-13T17:09:11.122855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:44.149902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103347+0000","last_change":"2024-09-13T17:09:10.673266+0000","last_active":"2024-09-13T17:09:12.103347+0000","last_peered":"2024-09-13T17:09:12.103347+0000","last_clean":"2024-09-13T17:09:12.103347+0000","last_became_active":"2024-09-13T17:09:10.672994+0000","last_became_peered":"2024-09-13T17:09:10.672994+0000","last_unstale":"2024-09-13T17:09:12.103347+0000","last_undegraded":"2024-09-13T17:09:12.103347+0000","last_fullsized":"2024-09-13T17:09:12.103347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:50.428518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475693+0000","last_change":"2024-09-13T17:09:11.126063+0000","last_active":"2024-09-13T17:09:12.475693+0000","last_peered":"2024-09-13T17:09:12.475693+0000","last_clean":"2024-09-13T17:09:12.475693+0000","last_became_active":"2024-09-13T17:09:11.125816+0000","last_became_peered":"2024-09-13T17:09:11.125816+0000","last_unstale":"2024-09-13T17:09:12.475693+0000","last_undegraded":"2024-09-13T17:09:12.475693+0000","last_fullsized":"2024-09-13T17:09:12.475693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:07.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.132373+0000","last_change":"2024-09-13T17:09:10.686826+0000","last_active":"2024-09-13T17:09:11.132373+0000","last_peered":"2024-09-13T17:09:11.132373+0000","last_clean":"2024-09-13T17:09:11.132373+0000","last_became_active":"2024-09-13T17:09:10.686541+0000","last_became_peered":"2024-09-13T17:09:10.686541+0000","last_unstale":"2024-09-13T17:09:11.132373+0000","last_undegraded":"2024-09-13T17:09:11.132373+0000","last_fullsized":"2024-09-13T17:09:11.132373+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108656+0000","last_change":"2024-09-13T17:09:11.117807+0000","last_active":"2024-09-13T17:09:12.108656+0000","last_peered":"2024-09-13T17:09:12.108656+0000","last_clean":"2024-09-13T17:09:12.108656+0000","last_became_active":"2024-09-13T17:09:11.117036+0000","last_became_peered":"2024-09-13T17:09:11.117036+0000","last_unstale":"2024-09-13T17:09:12.108656+0000","last_undegraded":"2024-09-13T17:09:12.108656+0000","last_fullsized":"2024-09-13T17:09:12.108656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:52.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.120031+0000","last_change":"2024-09-13T17:09:10.680350+0000","last_active":"2024-09-13T17:09:11.120031+0000","last_peered":"2024-09-13T17:09:11.120031+0000","last_clean":"2024-09-13T17:09:11.120031+0000","last_became_active":"2024-09-13T17:09:10.680081+0000","last_became_peered":"2024-09-13T17:09:10.680081+0000","last_unstale":"2024-09-13T17:09:11.120031+0000","last_undegraded":"2024-09-13T17:09:11.120031+0000","last_fullsized":"2024-09-13T17:09:11.120031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:38.198840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105781+0000","last_change":"2024-09-13T17:09:11.123632+0000","last_active":"2024-09-13T17:09:12.105781+0000","last_peered":"2024-09-13T17:09:12.105781+0000","last_clean":"2024-09-13T17:09:12.105781+0000","last_became_active":"2024-09-13T17:09:11.123341+0000","last_became_peered":"2024-09-13T17:09:11.123341+0000","last_unstale":"2024-09-13T17:09:12.105781+0000","last_undegraded":"2024-09-13T17:09:12.105781+0000","last_fullsized":"2024-09-13T17:09:12.105781+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:52.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.124377+0000","last_change":"2024-09-13T17:09:10.677543+0000","last_active":"2024-09-13T17:09:11.124377+0000","last_peered":"2024-09-13T17:09:11.124377+0000","last_clean":"2024-09-13T17:09:11.124377+0000","last_became_active":"2024-09-13T17:09:10.677232+0000","last_became_peered":"2024-09-13T17:09:10.677232+0000","last_unstale":"2024-09-13T17:09:11.124377+0000","last_undegraded":"2024-09-13T17:09:11.124377+0000","last_fullsized":"2024-09-13T17:09:11.124377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:02.036847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.116223+0000","last_change":"2024-09-13T17:09:11.111933+0000","last_active":"2024-09-13T17:09:11.116223+0000","last_peered":"2024-09-13T17:09:11.116223+0000","last_clean":"2024-09-13T17:09:11.116223+0000","last_became_active":"2024-09-13T17:09:11.111637+0000","last_became_peered":"2024-09-13T17:09:11.111637+0000","last_unstale":"2024-09-13T17:09:11.116223+0000","last_undegraded":"2024-09-13T17:09:11.116223+0000","last_fullsized":"2024-09-13T17:09:11.116223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:49.078243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102881+0000","last_change":"2024-09-13T17:09:10.685098+0000","last_active":"2024-09-13T17:09:12.102881+0000","last_peered":"2024-09-13T17:09:12.102881+0000","last_clean":"2024-09-13T17:09:12.102881+0000","last_became_active":"2024-09-13T17:09:10.684813+0000","last_became_peered":"2024-09-13T17:09:10.684813+0000","last_unstale":"2024-09-13T17:09:12.102881+0000","last_undegraded":"2024-09-13T17:09:12.102881+0000","last_fullsized":"2024-09-13T17:09:12.102881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:49.049253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475259+0000","last_change":"2024-09-13T17:09:11.119710+0000","last_active":"2024-09-13T17:09:12.475259+0000","last_peered":"2024-09-13T17:09:12.475259+0000","last_clean":"2024-09-13T17:09:12.475259+0000","last_became_active":"2024-09-13T17:09:11.119415+0000","last_became_peered":"2024-09-13T17:09:11.119415+0000","last_unstale":"2024-09-13T17:09:12.475259+0000","last_undegraded":"2024-09-13T17:09:12.475259+0000","last_fullsized":"2024-09-13T17:09:12.475259+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:02.205155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.098883+0000","last_change":"2024-09-13T17:09:10.678609+0000","last_active":"2024-09-13T17:09:11.098883+0000","last_peered":"2024-09-13T17:09:11.098883+0000","last_clean":"2024-09-13T17:09:11.098883+0000","last_became_active":"2024-09-13T17:09:10.678288+0000","last_became_peered":"2024-09-13T17:09:10.678288+0000","last_unstale":"2024-09-13T17:09:11.098883+0000","last_undegraded":"2024-09-13T17:09:11.098883+0000","last_fullsized":"2024-09-13T17:09:11.098883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106084+0000","last_change":"2024-09-13T17:09:11.128190+0000","last_active":"2024-09-13T17:09:12.106084+0000","last_peered":"2024-09-13T17:09:12.106084+0000","last_clean":"2024-09-13T17:09:12.106084+0000","last_became_active":"2024-09-13T17:09:11.127856+0000","last_became_peered":"2024-09-13T17:09:11.127856+0000","last_unstale":"2024-09-13T17:09:12.106084+0000","last_undegraded":"2024-09-13T17:09:12.106084+0000","last_fullsized":"2024-09-13T17:09:12.106084+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:52.626256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103525+0000","last_change":"2024-09-13T17:09:10.681781+0000","last_active":"2024-09-13T17:09:12.103525+0000","last_peered":"2024-09-13T17:09:12.103525+0000","last_clean":"2024-09-13T17:09:12.103525+0000","last_became_active":"2024-09-13T17:09:10.681228+0000","last_became_peered":"2024-09-13T17:09:10.681228+0000","last_unstale":"2024-09-13T17:09:12.103525+0000","last_undegraded":"2024-09-13T17:09:12.103525+0000","last_fullsized":"2024-09-13T17:09:12.103525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:29.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103525+0000","last_change":"2024-09-13T17:09:11.134679+0000","last_active":"2024-09-13T17:09:12.103525+0000","last_peered":"2024-09-13T17:09:12.103525+0000","last_clean":"2024-09-13T17:09:12.103525+0000","last_became_active":"2024-09-13T17:09:11.133887+0000","last_became_peered":"2024-09-13T17:09:11.133887+0000","last_unstale":"2024-09-13T17:09:12.103525+0000","last_undegraded":"2024-09-13T17:09:12.103525+0000","last_fullsized":"2024-09-13T17:09:12.103525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.165015+0000","last_change":"2024-09-13T17:09:10.674536+0000","last_active":"2024-09-13T17:09:12.165015+0000","last_peered":"2024-09-13T17:09:12.165015+0000","last_clean":"2024-09-13T17:09:12.165015+0000","last_became_active":"2024-09-13T17:09:10.674183+0000","last_became_peered":"2024-09-13T17:09:10.674183+0000","last_unstale":"2024-09-13T17:09:12.165015+0000","last_undegraded":"2024-09-13T17:09:12.165015+0000","last_fullsized":"2024-09-13T17:09:12.165015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103377+0000","last_change":"2024-09-13T17:09:11.125874+0000","last_active":"2024-09-13T17:09:12.103377+0000","last_peered":"2024-09-13T17:09:12.103377+0000","last_clean":"2024-09-13T17:09:12.103377+0000","last_became_active":"2024-09-13T17:09:11.125649+0000","last_became_peered":"2024-09-13T17:09:11.125649+0000","last_unstale":"2024-09-13T17:09:12.103377+0000","last_undegraded":"2024-09-13T17:09:12.103377+0000","last_fullsized":"2024-09-13T17:09:12.103377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106091+0000","last_change":"2024-09-13T17:09:10.681100+0000","last_active":"2024-09-13T17:09:12.106091+0000","last_peered":"2024-09-13T17:09:12.106091+0000","last_clean":"2024-09-13T17:09:12.106091+0000","last_became_active":"2024-09-13T17:09:10.680814+0000","last_became_peered":"2024-09-13T17:09:10.680814+0000","last_unstale":"2024-09-13T17:09:12.106091+0000","last_undegraded":"2024-09-13T17:09:12.106091+0000","last_fullsized":"2024-09-13T17:09:12.106091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:49.049253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105973+0000","last_change":"2024-09-13T17:09:11.124755+0000","last_active":"2024-09-13T17:09:12.105973+0000","last_peered":"2024-09-13T17:09:12.105973+0000","last_clean":"2024-09-13T17:09:12.105973+0000","last_became_active":"2024-09-13T17:09:11.124164+0000","last_became_peered":"2024-09-13T17:09:11.124164+0000","last_unstale":"2024-09-13T17:09:12.105973+0000","last_undegraded":"2024-09-13T17:09:12.105973+0000","last_fullsized":"2024-09-13T17:09:12.105973+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103364+0000","last_change":"2024-09-13T17:09:10.677273+0000","last_active":"2024-09-13T17:09:12.103364+0000","last_peered":"2024-09-13T17:09:12.103364+0000","last_clean":"2024-09-13T17:09:12.103364+0000","last_became_active":"2024-09-13T17:09:10.677015+0000","last_became_peered":"2024-09-13T17:09:10.677015+0000","last_unstale":"2024-09-13T17:09:12.103364+0000","last_undegraded":"2024-09-13T17:09:12.103364+0000","last_fullsized":"2024-09-13T17:09:12.103364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:01.751596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475295+0000","last_change":"2024-09-13T17:09:11.117305+0000","last_active":"2024-09-13T17:09:12.475295+0000","last_peered":"2024-09-13T17:09:12.475295+0000","last_clean":"2024-09-13T17:09:12.475295+0000","last_became_active":"2024-09-13T17:09:11.117020+0000","last_became_peered":"2024-09-13T17:09:11.117020+0000","last_unstale":"2024-09-13T17:09:12.475295+0000","last_undegraded":"2024-09-13T17:09:12.475295+0000","last_fullsized":"2024-09-13T17:09:12.475295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122874+0000","last_change":"2024-09-13T17:09:10.678353+0000","last_active":"2024-09-13T17:09:11.122874+0000","last_peered":"2024-09-13T17:09:11.122874+0000","last_clean":"2024-09-13T17:09:11.122874+0000","last_became_active":"2024-09-13T17:09:10.678049+0000","last_became_peered":"2024-09-13T17:09:10.678049+0000","last_unstale":"2024-09-13T17:09:11.122874+0000","last_undegraded":"2024-09-13T17:09:11.122874+0000","last_fullsized":"2024-09-13T17:09:11.122874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103223+0000","last_change":"2024-09-13T17:09:11.124757+0000","last_active":"2024-09-13T17:09:12.103223+0000","last_peered":"2024-09-13T17:09:12.103223+0000","last_clean":"2024-09-13T17:09:12.103223+0000","last_became_active":"2024-09-13T17:09:11.124485+0000","last_became_peered":"2024-09-13T17:09:11.124485+0000","last_unstale":"2024-09-13T17:09:12.103223+0000","last_undegraded":"2024-09-13T17:09:12.103223+0000","last_fullsized":"2024-09-13T17:09:12.103223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475365+0000","last_change":"2024-09-13T17:09:10.679337+0000","last_active":"2024-09-13T17:09:12.475365+0000","last_peered":"2024-09-13T17:09:12.475365+0000","last_clean":"2024-09-13T17:09:12.475365+0000","last_became_active":"2024-09-13T17:09:10.678804+0000","last_became_peered":"2024-09-13T17:09:10.678804+0000","last_unstale":"2024-09-13T17:09:12.475365+0000","last_undegraded":"2024-09-13T17:09:12.475365+0000","last_fullsized":"2024-09-13T17:09:12.475365+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108347+0000","last_change":"2024-09-13T17:09:11.117723+0000","last_active":"2024-09-13T17:09:12.108347+0000","last_peered":"2024-09-13T17:09:12.108347+0000","last_clean":"2024-09-13T17:09:12.108347+0000","last_became_active":"2024-09-13T17:09:11.117307+0000","last_became_peered":"2024-09-13T17:09:11.117307+0000","last_unstale":"2024-09-13T17:09:12.108347+0000","last_undegraded":"2024-09-13T17:09:12.108347+0000","last_fullsized":"2024-09-13T17:09:12.108347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102858+0000","last_change":"2024-09-13T17:09:10.679490+0000","last_active":"2024-09-13T17:09:12.102858+0000","last_peered":"2024-09-13T17:09:12.102858+0000","last_clean":"2024-09-13T17:09:12.102858+0000","last_became_active":"2024-09-13T17:09:10.679222+0000","last_became_peered":"2024-09-13T17:09:10.679222+0000","last_unstale":"2024-09-13T17:09:12.102858+0000","last_undegraded":"2024-09-13T17:09:12.102858+0000","last_fullsized":"2024-09-13T17:09:12.102858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826690+0000","last_change":"2024-09-13T17:09:11.119436+0000","last_active":"2024-09-13T17:09:12.826690+0000","last_peered":"2024-09-13T17:09:12.826690+0000","last_clean":"2024-09-13T17:09:12.826690+0000","last_became_active":"2024-09-13T17:09:11.117438+0000","last_became_peered":"2024-09-13T17:09:11.117438+0000","last_unstale":"2024-09-13T17:09:12.826690+0000","last_undegraded":"2024-09-13T17:09:12.826690+0000","last_fullsized":"2024-09-13T17:09:12.826690+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:38.504248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102925+0000","last_change":"2024-09-13T17:09:10.676478+0000","last_active":"2024-09-13T17:09:12.102925+0000","last_peered":"2024-09-13T17:09:12.102925+0000","last_clean":"2024-09-13T17:09:12.102925+0000","last_became_active":"2024-09-13T17:09:10.675860+0000","last_became_peered":"2024-09-13T17:09:10.675860+0000","last_unstale":"2024-09-13T17:09:12.102925+0000","last_undegraded":"2024-09-13T17:09:12.102925+0000","last_fullsized":"2024-09-13T17:09:12.102925+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122757+0000","last_change":"2024-09-13T17:09:11.114884+0000","last_active":"2024-09-13T17:09:11.122757+0000","last_peered":"2024-09-13T17:09:11.122757+0000","last_clean":"2024-09-13T17:09:11.122757+0000","last_became_active":"2024-09-13T17:09:11.114634+0000","last_became_peered":"2024-09-13T17:09:11.114634+0000","last_unstale":"2024-09-13T17:09:11.122757+0000","last_undegraded":"2024-09-13T17:09:11.122757+0000","last_fullsized":"2024-09-13T17:09:11.122757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475768+0000","last_change":"2024-09-13T17:09:10.678443+0000","last_active":"2024-09-13T17:09:12.475768+0000","last_peered":"2024-09-13T17:09:12.475768+0000","last_clean":"2024-09-13T17:09:12.475768+0000","last_became_active":"2024-09-13T17:09:10.676390+0000","last_became_peered":"2024-09-13T17:09:10.676390+0000","last_unstale":"2024-09-13T17:09:12.475768+0000","last_undegraded":"2024-09-13T17:09:12.475768+0000","last_fullsized":"2024-09-13T17:09:12.475768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:37.498516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106144+0000","last_change":"2024-09-13T17:09:11.118052+0000","last_active":"2024-09-13T17:09:12.106144+0000","last_peered":"2024-09-13T17:09:12.106144+0000","last_clean":"2024-09-13T17:09:12.106144+0000","last_became_active":"2024-09-13T17:09:11.116981+0000","last_became_peered":"2024-09-13T17:09:11.116981+0000","last_unstale":"2024-09-13T17:09:12.106144+0000","last_undegraded":"2024-09-13T17:09:12.106144+0000","last_fullsized":"2024-09-13T17:09:12.106144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:32.685108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.119904+0000","last_change":"2024-09-13T17:09:10.674921+0000","last_active":"2024-09-13T17:09:11.119904+0000","last_peered":"2024-09-13T17:09:11.119904+0000","last_clean":"2024-09-13T17:09:11.119904+0000","last_became_active":"2024-09-13T17:09:10.674586+0000","last_became_peered":"2024-09-13T17:09:10.674586+0000","last_unstale":"2024-09-13T17:09:11.119904+0000","last_undegraded":"2024-09-13T17:09:11.119904+0000","last_fullsized":"2024-09-13T17:09:11.119904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:53.442943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475567+0000","last_change":"2024-09-13T17:09:11.126719+0000","last_active":"2024-09-13T17:09:12.475567+0000","last_peered":"2024-09-13T17:09:12.475567+0000","last_clean":"2024-09-13T17:09:12.475567+0000","last_became_active":"2024-09-13T17:09:11.123207+0000","last_became_peered":"2024-09-13T17:09:11.123207+0000","last_unstale":"2024-09-13T17:09:12.475567+0000","last_undegraded":"2024-09-13T17:09:12.475567+0000","last_fullsized":"2024-09-13T17:09:12.475567+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:09.468121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.120711+0000","last_change":"2024-09-13T17:09:10.682727+0000","last_active":"2024-09-13T17:09:11.120711+0000","last_peered":"2024-09-13T17:09:11.120711+0000","last_clean":"2024-09-13T17:09:11.120711+0000","last_became_active":"2024-09-13T17:09:10.682437+0000","last_became_peered":"2024-09-13T17:09:10.682437+0000","last_unstale":"2024-09-13T17:09:11.120711+0000","last_undegraded":"2024-09-13T17:09:11.120711+0000","last_fullsized":"2024-09-13T17:09:11.120711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:21.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126774+0000","last_change":"2024-09-13T17:09:11.120060+0000","last_active":"2024-09-13T17:09:11.126774+0000","last_peered":"2024-09-13T17:09:11.126774+0000","last_clean":"2024-09-13T17:09:11.126774+0000","last_became_active":"2024-09-13T17:09:11.118745+0000","last_became_peered":"2024-09-13T17:09:11.118745+0000","last_unstale":"2024-09-13T17:09:11.126774+0000","last_undegraded":"2024-09-13T17:09:11.126774+0000","last_fullsized":"2024-09-13T17:09:11.126774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:30.168489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100339+0000","last_change":"2024-09-13T17:09:10.677405+0000","last_active":"2024-09-13T17:09:11.100339+0000","last_peered":"2024-09-13T17:09:11.100339+0000","last_clean":"2024-09-13T17:09:11.100339+0000","last_became_active":"2024-09-13T17:09:10.677113+0000","last_became_peered":"2024-09-13T17:09:10.677113+0000","last_unstale":"2024-09-13T17:09:11.100339+0000","last_undegraded":"2024-09-13T17:09:11.100339+0000","last_fullsized":"2024-09-13T17:09:11.100339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126743+0000","last_change":"2024-09-13T17:09:11.116724+0000","last_active":"2024-09-13T17:09:11.126743+0000","last_peered":"2024-09-13T17:09:11.126743+0000","last_clean":"2024-09-13T17:09:11.126743+0000","last_became_active":"2024-09-13T17:09:11.114532+0000","last_became_peered":"2024-09-13T17:09:11.114532+0000","last_unstale":"2024-09-13T17:09:11.126743+0000","last_undegraded":"2024-09-13T17:09:11.126743+0000","last_fullsized":"2024-09-13T17:09:11.126743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:02.757328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164853+0000","last_change":"2024-09-13T17:09:10.680509+0000","last_active":"2024-09-13T17:09:12.164853+0000","last_peered":"2024-09-13T17:09:12.164853+0000","last_clean":"2024-09-13T17:09:12.164853+0000","last_became_active":"2024-09-13T17:09:10.680218+0000","last_became_peered":"2024-09-13T17:09:10.680218+0000","last_unstale":"2024-09-13T17:09:12.164853+0000","last_undegraded":"2024-09-13T17:09:12.164853+0000","last_fullsized":"2024-09-13T17:09:12.164853+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125430+0000","last_change":"2024-09-13T17:09:11.113188+0000","last_active":"2024-09-13T17:09:11.125430+0000","last_peered":"2024-09-13T17:09:11.125430+0000","last_clean":"2024-09-13T17:09:11.125430+0000","last_became_active":"2024-09-13T17:09:11.112842+0000","last_became_peered":"2024-09-13T17:09:11.112842+0000","last_unstale":"2024-09-13T17:09:11.125430+0000","last_undegraded":"2024-09-13T17:09:11.125430+0000","last_fullsized":"2024-09-13T17:09:11.125430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:51.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826569+0000","last_change":"2024-09-13T17:09:10.683670+0000","last_active":"2024-09-13T17:09:12.826569+0000","last_peered":"2024-09-13T17:09:12.826569+0000","last_clean":"2024-09-13T17:09:12.826569+0000","last_became_active":"2024-09-13T17:09:10.681296+0000","last_became_peered":"2024-09-13T17:09:10.681296+0000","last_unstale":"2024-09-13T17:09:12.826569+0000","last_undegraded":"2024-09-13T17:09:12.826569+0000","last_fullsized":"2024-09-13T17:09:12.826569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.109332+0000","last_change":"2024-09-13T17:09:11.107205+0000","last_active":"2024-09-13T17:09:11.109332+0000","last_peered":"2024-09-13T17:09:11.109332+0000","last_clean":"2024-09-13T17:09:11.109332+0000","last_became_active":"2024-09-13T17:09:11.106769+0000","last_became_peered":"2024-09-13T17:09:11.106769+0000","last_unstale":"2024-09-13T17:09:11.109332+0000","last_undegraded":"2024-09-13T17:09:11.109332+0000","last_fullsized":"2024-09-13T17:09:11.109332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100414+0000","last_change":"2024-09-13T17:09:10.671120+0000","last_active":"2024-09-13T17:09:11.100414+0000","last_peered":"2024-09-13T17:09:11.100414+0000","last_clean":"2024-09-13T17:09:11.100414+0000","last_became_active":"2024-09-13T17:09:10.670790+0000","last_became_peered":"2024-09-13T17:09:10.670790+0000","last_unstale":"2024-09-13T17:09:11.100414+0000","last_undegraded":"2024-09-13T17:09:11.100414+0000","last_fullsized":"2024-09-13T17:09:11.100414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:29.934709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108332+0000","last_change":"2024-09-13T17:09:11.110016+0000","last_active":"2024-09-13T17:09:12.108332+0000","last_peered":"2024-09-13T17:09:12.108332+0000","last_clean":"2024-09-13T17:09:12.108332+0000","last_became_active":"2024-09-13T17:09:11.109777+0000","last_became_peered":"2024-09-13T17:09:11.109777+0000","last_unstale":"2024-09-13T17:09:12.108332+0000","last_undegraded":"2024-09-13T17:09:12.108332+0000","last_fullsized":"2024-09-13T17:09:12.108332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:32.685108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105621+0000","last_change":"2024-09-13T17:09:10.681520+0000","last_active":"2024-09-13T17:09:12.105621+0000","last_peered":"2024-09-13T17:09:12.105621+0000","last_clean":"2024-09-13T17:09:12.105621+0000","last_became_active":"2024-09-13T17:09:10.681188+0000","last_became_peered":"2024-09-13T17:09:10.681188+0000","last_unstale":"2024-09-13T17:09:12.105621+0000","last_undegraded":"2024-09-13T17:09:12.105621+0000","last_fullsized":"2024-09-13T17:09:12.105621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:50.428518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126749+0000","last_change":"2024-09-13T17:09:11.116597+0000","last_active":"2024-09-13T17:09:11.126749+0000","last_peered":"2024-09-13T17:09:11.126749+0000","last_clean":"2024-09-13T17:09:11.126749+0000","last_became_active":"2024-09-13T17:09:11.116290+0000","last_became_peered":"2024-09-13T17:09:11.116290+0000","last_unstale":"2024-09-13T17:09:11.126749+0000","last_undegraded":"2024-09-13T17:09:11.126749+0000","last_fullsized":"2024-09-13T17:09:11.126749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:20.961969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475783+0000","last_change":"2024-09-13T17:09:10.679585+0000","last_active":"2024-09-13T17:09:12.475783+0000","last_peered":"2024-09-13T17:09:12.475783+0000","last_clean":"2024-09-13T17:09:12.475783+0000","last_became_active":"2024-09-13T17:09:10.679255+0000","last_became_peered":"2024-09-13T17:09:10.679255+0000","last_unstale":"2024-09-13T17:09:12.475783+0000","last_undegraded":"2024-09-13T17:09:12.475783+0000","last_fullsized":"2024-09-13T17:09:12.475783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106152+0000","last_change":"2024-09-13T17:09:11.125856+0000","last_active":"2024-09-13T17:09:12.106152+0000","last_peered":"2024-09-13T17:09:12.106152+0000","last_clean":"2024-09-13T17:09:12.106152+0000","last_became_active":"2024-09-13T17:09:11.125618+0000","last_became_peered":"2024-09-13T17:09:11.125618+0000","last_unstale":"2024-09-13T17:09:12.106152+0000","last_undegraded":"2024-09-13T17:09:12.106152+0000","last_fullsized":"2024-09-13T17:09:12.106152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.165089+0000","last_change":"2024-09-13T17:09:10.681092+0000","last_active":"2024-09-13T17:09:12.165089+0000","last_peered":"2024-09-13T17:09:12.165089+0000","last_clean":"2024-09-13T17:09:12.165089+0000","last_became_active":"2024-09-13T17:09:10.680311+0000","last_became_peered":"2024-09-13T17:09:10.680311+0000","last_unstale":"2024-09-13T17:09:12.165089+0000","last_undegraded":"2024-09-13T17:09:12.165089+0000","last_fullsized":"2024-09-13T17:09:12.165089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:38.198840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164875+0000","last_change":"2024-09-13T17:09:11.126127+0000","last_active":"2024-09-13T17:09:12.164875+0000","last_peered":"2024-09-13T17:09:12.164875+0000","last_clean":"2024-09-13T17:09:12.164875+0000","last_became_active":"2024-09-13T17:09:11.125698+0000","last_became_peered":"2024-09-13T17:09:11.125698+0000","last_unstale":"2024-09-13T17:09:12.164875+0000","last_undegraded":"2024-09-13T17:09:12.164875+0000","last_fullsized":"2024-09-13T17:09:12.164875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:30.168489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475665+0000","last_change":"2024-09-13T17:09:10.684142+0000","last_active":"2024-09-13T17:09:12.475665+0000","last_peered":"2024-09-13T17:09:12.475665+0000","last_clean":"2024-09-13T17:09:12.475665+0000","last_became_active":"2024-09-13T17:09:10.683853+0000","last_became_peered":"2024-09-13T17:09:10.683853+0000","last_unstale":"2024-09-13T17:09:12.475665+0000","last_undegraded":"2024-09-13T17:09:12.475665+0000","last_fullsized":"2024-09-13T17:09:12.475665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:31.818414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826611+0000","last_change":"2024-09-13T17:09:11.123301+0000","last_active":"2024-09-13T17:09:12.826611+0000","last_peered":"2024-09-13T17:09:12.826611+0000","last_clean":"2024-09-13T17:09:12.826611+0000","last_became_active":"2024-09-13T17:09:11.122785+0000","last_became_peered":"2024-09-13T17:09:11.122785+0000","last_unstale":"2024-09-13T17:09:12.826611+0000","last_undegraded":"2024-09-13T17:09:12.826611+0000","last_fullsized":"2024-09-13T17:09:12.826611+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:59.294698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164960+0000","last_change":"2024-09-13T17:09:10.669375+0000","last_active":"2024-09-13T17:09:12.164960+0000","last_peered":"2024-09-13T17:09:12.164960+0000","last_clean":"2024-09-13T17:09:12.164960+0000","last_became_active":"2024-09-13T17:09:10.669088+0000","last_became_peered":"2024-09-13T17:09:10.669088+0000","last_unstale":"2024-09-13T17:09:12.164960+0000","last_undegraded":"2024-09-13T17:09:12.164960+0000","last_fullsized":"2024-09-13T17:09:12.164960+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:02.036847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164869+0000","last_change":"2024-09-13T17:09:11.112160+0000","last_active":"2024-09-13T17:09:12.164869+0000","last_peered":"2024-09-13T17:09:12.164869+0000","last_clean":"2024-09-13T17:09:12.164869+0000","last_became_active":"2024-09-13T17:09:11.111915+0000","last_became_peered":"2024-09-13T17:09:11.111915+0000","last_unstale":"2024-09-13T17:09:12.164869+0000","last_undegraded":"2024-09-13T17:09:12.164869+0000","last_fullsized":"2024-09-13T17:09:12.164869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475603+0000","last_change":"2024-09-13T17:09:10.681264+0000","last_active":"2024-09-13T17:09:12.475603+0000","last_peered":"2024-09-13T17:09:12.475603+0000","last_clean":"2024-09-13T17:09:12.475603+0000","last_became_active":"2024-09-13T17:09:10.680742+0000","last_became_peered":"2024-09-13T17:09:10.680742+0000","last_unstale":"2024-09-13T17:09:12.475603+0000","last_undegraded":"2024-09-13T17:09:12.475603+0000","last_fullsized":"2024-09-13T17:09:12.475603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102825+0000","last_change":"2024-09-13T17:09:11.126061+0000","last_active":"2024-09-13T17:09:12.102825+0000","last_peered":"2024-09-13T17:09:12.102825+0000","last_clean":"2024-09-13T17:09:12.102825+0000","last_became_active":"2024-09-13T17:09:11.125808+0000","last_became_peered":"2024-09-13T17:09:11.125808+0000","last_unstale":"2024-09-13T17:09:12.102825+0000","last_undegraded":"2024-09-13T17:09:12.102825+0000","last_fullsized":"2024-09-13T17:09:12.102825+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:15.296969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100545+0000","last_change":"2024-09-13T17:09:10.675570+0000","last_active":"2024-09-13T17:09:11.100545+0000","last_peered":"2024-09-13T17:09:11.100545+0000","last_clean":"2024-09-13T17:09:11.100545+0000","last_became_active":"2024-09-13T17:09:10.675264+0000","last_became_peered":"2024-09-13T17:09:10.675264+0000","last_unstale":"2024-09-13T17:09:11.100545+0000","last_undegraded":"2024-09-13T17:09:11.100545+0000","last_fullsized":"2024-09-13T17:09:11.100545+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:40.804455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125456+0000","last_change":"2024-09-13T17:09:11.121833+0000","last_active":"2024-09-13T17:09:11.125456+0000","last_peered":"2024-09-13T17:09:11.125456+0000","last_clean":"2024-09-13T17:09:11.125456+0000","last_became_active":"2024-09-13T17:09:11.121546+0000","last_became_peered":"2024-09-13T17:09:11.121546+0000","last_unstale":"2024-09-13T17:09:11.125456+0000","last_undegraded":"2024-09-13T17:09:11.125456+0000","last_fullsized":"2024-09-13T17:09:11.125456+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:07.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102849+0000","last_change":"2024-09-13T17:09:10.682643+0000","last_active":"2024-09-13T17:09:12.102849+0000","last_peered":"2024-09-13T17:09:12.102849+0000","last_clean":"2024-09-13T17:09:12.102849+0000","last_became_active":"2024-09-13T17:09:10.682307+0000","last_became_peered":"2024-09-13T17:09:10.682307+0000","last_unstale":"2024-09-13T17:09:12.102849+0000","last_undegraded":"2024-09-13T17:09:12.102849+0000","last_fullsized":"2024-09-13T17:09:12.102849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108653+0000","last_change":"2024-09-13T17:09:11.112831+0000","last_active":"2024-09-13T17:09:12.108653+0000","last_peered":"2024-09-13T17:09:12.108653+0000","last_clean":"2024-09-13T17:09:12.108653+0000","last_became_active":"2024-09-13T17:09:11.112578+0000","last_became_peered":"2024-09-13T17:09:11.112578+0000","last_unstale":"2024-09-13T17:09:12.108653+0000","last_undegraded":"2024-09-13T17:09:12.108653+0000","last_fullsized":"2024-09-13T17:09:12.108653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826487+0000","last_change":"2024-09-13T17:09:10.682484+0000","last_active":"2024-09-13T17:09:12.826487+0000","last_peered":"2024-09-13T17:09:12.826487+0000","last_clean":"2024-09-13T17:09:12.826487+0000","last_became_active":"2024-09-13T17:09:10.682224+0000","last_became_peered":"2024-09-13T17:09:10.682224+0000","last_unstale":"2024-09-13T17:09:12.826487+0000","last_undegraded":"2024-09-13T17:09:12.826487+0000","last_fullsized":"2024-09-13T17:09:12.826487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.103092+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:09:11.103092+0000","last_peered":"2024-09-13T17:09:11.103092+0000","last_clean":"2024-09-13T17:09:11.103092+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:09:11.103092+0000","last_undegraded":"2024-09-13T17:09:11.103092+0000","last_fullsized":"2024-09-13T17:09:11.103092+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":742215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.80200000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.90200000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.82799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.81899999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.77400000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.755},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.79000000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.88},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.94799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.75}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56699999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.751}]},{"osd":3,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.871}]},{"osd":5,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.88900000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":401408,"data_stored":153343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.6829999999999998,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.8610000000000002,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.76500000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.82699999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":0.753},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.89500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":372736,"data_stored":144508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.34999999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.372},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.749}]},{"osd":6,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":389120,"data_stored":147043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.438}]},{"osd":2,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.38900000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":1.0740000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.94799999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.91100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":974848,"data_stored":741106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.5259999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.4889999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.0430000000000001,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6760000000000002,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":4.7549999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.5940000000000003,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.53500000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":4.734,"5min":4.734,"15min":4.734},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.8079999999999998,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.93999999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":0.997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.9710000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":5.0170000000000003,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.96599999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":4.9950000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.86799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:09:15.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:09:15.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.478+0000 7f3240a00640 1 Processor -- start 2024-09-13T17:09:15.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.478+0000 7f3240a00640 1 -- start start 2024-09-13T17:09:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323c154a90 0x7f323c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 0x7f323c0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c0b0df0 con 0x7f323c154a90 2024-09-13T17:09:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323c15bb10 con 0x7f323c0b0980 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3240a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c06b900 con 0x7f323c06b360 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f323a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323c154a90 0x7f323c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f323a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323c154a90 0x7f323c15b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47798/0 (socket says 172.21.15.28:47798) 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f323a800640 1 -- 172.21.15.28:0/479591005 learned_addr learned my addr 172.21.15.28:0/479591005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f323b200640 1 --2- 172.21.15.28:0/479591005 >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 0x7f323c0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.479+0000 7f3233e00640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323a800640 1 -- 172.21.15.28:0/479591005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 msgr2=0x7f323c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323a800640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323a800640 1 -- 172.21.15.28:0/479591005 >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 msgr2=0x7f323c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323a800640 1 --2- 172.21.15.28:0/479591005 >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 0x7f323c0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323a800640 1 -- 172.21.15.28:0/479591005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323c06bbe0 con 0x7f323c154a90 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f3233e00640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.480+0000 7f323b200640 1 --2- 172.21.15.28:0/479591005 >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 0x7f323c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.481+0000 7f323a800640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c15b5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f323c15e080 tx=0x7f322c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a508b0d9a70521d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.481+0000 7f3239e00640 1 -- 172.21.15.28:0/479591005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c032440 con 0x7f323c154a90 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.481+0000 7f3239e00640 1 -- 172.21.15.28:0/479591005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c0325a0 con 0x7f323c154a90 2024-09-13T17:09:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.481+0000 7f3239e00640 1 -- 172.21.15.28:0/479591005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c032930 con 0x7f323c154a90 2024-09-13T17:09:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 -- 172.21.15.28:0/479591005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 msgr2=0x7f323c15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c15b5d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f323c15e080 tx=0x7f322c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 -- 172.21.15.28:0/479591005 shutdown_connections 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 --2- 172.21.15.28:0/479591005 >> v2:172.21.15.189:3300/0 conn(0x7f323c0b0980 0x7f323c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 --2- 172.21.15.28:0/479591005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c15b5d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 -- 172.21.15.28:0/479591005 >> 172.21.15.28:0/479591005 conn(0x7f323c072e40 msgr2=0x7f323c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 -- 172.21.15.28:0/479591005 shutdown_connections 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.482+0000 7f3240a00640 1 -- 172.21.15.28:0/479591005 wait complete. 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.483+0000 7f3240a00640 1 Processor -- start 2024-09-13T17:09:15.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 -- start start 2024-09-13T17:09:15.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c1f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 0x7f323c1f5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0ac480 con 0x7f323c154a90 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0ac760 con 0x7f323c0b0980 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.484+0000 7f3240a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323c0aca40 con 0x7f323c06b360 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47808/0 (socket says 172.21.15.28:47808) 2024-09-13T17:09:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 -- 172.21.15.28:0/689842316 learned_addr learned my addr 172.21.15.28:0/689842316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f3233e00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 0x7f323c1f5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323a800640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c1f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 -- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 msgr2=0x7f323c1f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c1f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 -- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 msgr2=0x7f323c1f5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 0x7f323c1f5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 -- 172.21.15.28:0/689842316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0094d0 con 0x7f323c154a90 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f3233e00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 0x7f323c1f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323a800640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c1f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f323b200640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3234007de0 tx=0x7f323400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.485+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234003990 con 0x7f323c154a90 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.486+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323400f040 con 0x7f323c154a90 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.486+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3234003d30 con 0x7f323c154a90 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.486+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c0acd20 con 0x7f323c154a90 2024-09-13T17:09:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.486+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c0af070 con 0x7f323c154a90 2024-09-13T17:09:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.487+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c0ad240 con 0x7f323c154a90 2024-09-13T17:09:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.488+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3234004030 con 0x7f323c154a90 2024-09-13T17:09:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.488+0000 7f3232a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 0x7f321806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.488+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3234089790 con 0x7f323c154a90 2024-09-13T17:09:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.489+0000 7f323a800640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 0x7f321806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.490+0000 7f323a800640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 0x7f321806d0f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f322c034000 tx=0x7f322c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.493+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f323c0ad240 con 0x7f323c154a90 2024-09-13T17:09:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.667+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f323c06c1a0 con 0x7f321806ac30 2024-09-13T17:09:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.675+0000 7f3232a00640 1 -- 172.21.15.28:0/689842316 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289143 (secure 0 0 0) 0x7f322c041090 con 0x7f321806ac30 2024-09-13T17:09:15.680 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 msgr2=0x7f321806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 0x7f321806d0f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f322c034000 tx=0x7f322c02e000 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 msgr2=0x7f323c0aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3234007de0 tx=0x7f323400d4a0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 shutdown_connections 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321806ac30 0x7f321806d0f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323c154a90 0x7f323c0aeb30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f323c0b0980 0x7f323c1f5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 --2- 172.21.15.28:0/689842316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323c06b360 0x7f323c1f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 >> 172.21.15.28:0/689842316 conn(0x7f323c072e40 msgr2=0x7f323c159970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 shutdown_connections 2024-09-13T17:09:15.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.679+0000 7f3240a00640 1 -- 172.21.15.28:0/689842316 wait complete. 2024-09-13T17:09:15.696 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-13T17:09:14.900040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236752,"kb_used_data":4176,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754737968,"statfs":{"total":773094113280,"available":772851679232,"internally_reserved":0,"allocated":4276224,"data_stored":2372278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.249225"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826697+0000","last_change":"2024-09-13T17:09:11.114498+0000","last_active":"2024-09-13T17:09:12.826697+0000","last_peered":"2024-09-13T17:09:12.826697+0000","last_clean":"2024-09-13T17:09:12.826697+0000","last_became_active":"2024-09-13T17:09:11.114077+0000","last_became_peered":"2024-09-13T17:09:11.114077+0000","last_unstale":"2024-09-13T17:09:12.826697+0000","last_undegraded":"2024-09-13T17:09:12.826697+0000","last_fullsized":"2024-09-13T17:09:12.826697+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:02.757328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103517+0000","last_change":"2024-09-13T17:09:11.130697+0000","last_active":"2024-09-13T17:09:12.103517+0000","last_peered":"2024-09-13T17:09:12.103517+0000","last_clean":"2024-09-13T17:09:12.103517+0000","last_became_active":"2024-09-13T17:09:11.129323+0000","last_became_peered":"2024-09-13T17:09:11.129323+0000","last_unstale":"2024-09-13T17:09:12.103517+0000","last_undegraded":"2024-09-13T17:09:12.103517+0000","last_fullsized":"2024-09-13T17:09:12.103517+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:52.626256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105973+0000","last_change":"2024-09-13T17:09:11.124311+0000","last_active":"2024-09-13T17:09:12.105973+0000","last_peered":"2024-09-13T17:09:12.105973+0000","last_clean":"2024-09-13T17:09:12.105973+0000","last_became_active":"2024-09-13T17:09:11.124015+0000","last_became_peered":"2024-09-13T17:09:11.124015+0000","last_unstale":"2024-09-13T17:09:12.105973+0000","last_undegraded":"2024-09-13T17:09:12.105973+0000","last_fullsized":"2024-09-13T17:09:12.105973+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475414+0000","last_change":"2024-09-13T17:09:11.127945+0000","last_active":"2024-09-13T17:09:12.475414+0000","last_peered":"2024-09-13T17:09:12.475414+0000","last_clean":"2024-09-13T17:09:12.475414+0000","last_became_active":"2024-09-13T17:09:11.127711+0000","last_became_peered":"2024-09-13T17:09:11.127711+0000","last_unstale":"2024-09-13T17:09:12.475414+0000","last_undegraded":"2024-09-13T17:09:12.475414+0000","last_fullsized":"2024-09-13T17:09:12.475414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125413+0000","last_change":"2024-09-13T17:09:11.113062+0000","last_active":"2024-09-13T17:09:11.125413+0000","last_peered":"2024-09-13T17:09:11.125413+0000","last_clean":"2024-09-13T17:09:11.125413+0000","last_became_active":"2024-09-13T17:09:11.112493+0000","last_became_peered":"2024-09-13T17:09:11.112493+0000","last_unstale":"2024-09-13T17:09:11.125413+0000","last_undegraded":"2024-09-13T17:09:11.125413+0000","last_fullsized":"2024-09-13T17:09:11.125413+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103016+0000","last_change":"2024-09-13T17:09:11.122759+0000","last_active":"2024-09-13T17:09:12.103016+0000","last_peered":"2024-09-13T17:09:12.103016+0000","last_clean":"2024-09-13T17:09:12.103016+0000","last_became_active":"2024-09-13T17:09:11.121272+0000","last_became_peered":"2024-09-13T17:09:11.121272+0000","last_unstale":"2024-09-13T17:09:12.103016+0000","last_undegraded":"2024-09-13T17:09:12.103016+0000","last_fullsized":"2024-09-13T17:09:12.103016+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108400+0000","last_change":"2024-09-13T17:09:11.114090+0000","last_active":"2024-09-13T17:09:12.108400+0000","last_peered":"2024-09-13T17:09:12.108400+0000","last_clean":"2024-09-13T17:09:12.108400+0000","last_became_active":"2024-09-13T17:09:11.113798+0000","last_became_peered":"2024-09-13T17:09:11.113798+0000","last_unstale":"2024-09-13T17:09:12.108400+0000","last_undegraded":"2024-09-13T17:09:12.108400+0000","last_fullsized":"2024-09-13T17:09:12.108400+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:02.205155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108218+0000","last_change":"2024-09-13T17:09:11.113340+0000","last_active":"2024-09-13T17:09:12.108218+0000","last_peered":"2024-09-13T17:09:12.108218+0000","last_clean":"2024-09-13T17:09:12.108218+0000","last_became_active":"2024-09-13T17:09:11.112989+0000","last_became_peered":"2024-09-13T17:09:11.112989+0000","last_unstale":"2024-09-13T17:09:12.108218+0000","last_undegraded":"2024-09-13T17:09:12.108218+0000","last_fullsized":"2024-09-13T17:09:12.108218+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.130728+0000","last_change":"2024-09-13T17:09:11.126185+0000","last_active":"2024-09-13T17:09:11.130728+0000","last_peered":"2024-09-13T17:09:11.130728+0000","last_clean":"2024-09-13T17:09:11.130728+0000","last_became_active":"2024-09-13T17:09:11.125901+0000","last_became_peered":"2024-09-13T17:09:11.125901+0000","last_unstale":"2024-09-13T17:09:11.130728+0000","last_undegraded":"2024-09-13T17:09:11.130728+0000","last_fullsized":"2024-09-13T17:09:11.130728+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:20.961969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475676+0000","last_change":"2024-09-13T17:09:11.134007+0000","last_active":"2024-09-13T17:09:12.475676+0000","last_peered":"2024-09-13T17:09:12.475676+0000","last_clean":"2024-09-13T17:09:12.475676+0000","last_became_active":"2024-09-13T17:09:11.133652+0000","last_became_peered":"2024-09-13T17:09:11.133652+0000","last_unstale":"2024-09-13T17:09:12.475676+0000","last_undegraded":"2024-09-13T17:09:12.475676+0000","last_fullsized":"2024-09-13T17:09:12.475676+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:15.296969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108787+0000","last_change":"2024-09-13T17:09:11.129885+0000","last_active":"2024-09-13T17:09:12.108787+0000","last_peered":"2024-09-13T17:09:12.108787+0000","last_clean":"2024-09-13T17:09:12.108787+0000","last_became_active":"2024-09-13T17:09:11.129668+0000","last_became_peered":"2024-09-13T17:09:11.129668+0000","last_unstale":"2024-09-13T17:09:12.108787+0000","last_undegraded":"2024-09-13T17:09:12.108787+0000","last_fullsized":"2024-09-13T17:09:12.108787+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105895+0000","last_change":"2024-09-13T17:09:11.120792+0000","last_active":"2024-09-13T17:09:12.105895+0000","last_peered":"2024-09-13T17:09:12.105895+0000","last_clean":"2024-09-13T17:09:12.105895+0000","last_became_active":"2024-09-13T17:09:11.120497+0000","last_became_peered":"2024-09-13T17:09:11.120497+0000","last_unstale":"2024-09-13T17:09:12.105895+0000","last_undegraded":"2024-09-13T17:09:12.105895+0000","last_fullsized":"2024-09-13T17:09:12.105895+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.129091+0000","last_change":"2024-09-13T17:09:11.125541+0000","last_active":"2024-09-13T17:09:11.129091+0000","last_peered":"2024-09-13T17:09:11.129091+0000","last_clean":"2024-09-13T17:09:11.129091+0000","last_became_active":"2024-09-13T17:09:11.125257+0000","last_became_peered":"2024-09-13T17:09:11.125257+0000","last_unstale":"2024-09-13T17:09:11.129091+0000","last_undegraded":"2024-09-13T17:09:11.129091+0000","last_fullsized":"2024-09-13T17:09:11.129091+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:50.054986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103117+0000","last_change":"2024-09-13T17:09:11.123092+0000","last_active":"2024-09-13T17:09:12.103117+0000","last_peered":"2024-09-13T17:09:12.103117+0000","last_clean":"2024-09-13T17:09:12.103117+0000","last_became_active":"2024-09-13T17:09:11.122870+0000","last_became_peered":"2024-09-13T17:09:11.122870+0000","last_unstale":"2024-09-13T17:09:12.103117+0000","last_undegraded":"2024-09-13T17:09:12.103117+0000","last_fullsized":"2024-09-13T17:09:12.103117+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.599876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105745+0000","last_change":"2024-09-13T17:09:11.109715+0000","last_active":"2024-09-13T17:09:12.105745+0000","last_peered":"2024-09-13T17:09:12.105745+0000","last_clean":"2024-09-13T17:09:12.105745+0000","last_became_active":"2024-09-13T17:09:11.107677+0000","last_became_peered":"2024-09-13T17:09:11.107677+0000","last_unstale":"2024-09-13T17:09:12.105745+0000","last_undegraded":"2024-09-13T17:09:12.105745+0000","last_fullsized":"2024-09-13T17:09:12.105745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108481+0000","last_change":"2024-09-13T17:09:11.122482+0000","last_active":"2024-09-13T17:09:12.108481+0000","last_peered":"2024-09-13T17:09:12.108481+0000","last_clean":"2024-09-13T17:09:12.108481+0000","last_became_active":"2024-09-13T17:09:11.121649+0000","last_became_peered":"2024-09-13T17:09:11.121649+0000","last_unstale":"2024-09-13T17:09:12.108481+0000","last_undegraded":"2024-09-13T17:09:12.108481+0000","last_fullsized":"2024-09-13T17:09:12.108481+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108789+0000","last_change":"2024-09-13T17:09:11.115663+0000","last_active":"2024-09-13T17:09:12.108789+0000","last_peered":"2024-09-13T17:09:12.108789+0000","last_clean":"2024-09-13T17:09:12.108789+0000","last_became_active":"2024-09-13T17:09:11.113066+0000","last_became_peered":"2024-09-13T17:09:11.113066+0000","last_unstale":"2024-09-13T17:09:12.108789+0000","last_undegraded":"2024-09-13T17:09:12.108789+0000","last_fullsized":"2024-09-13T17:09:12.108789+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103500+0000","last_change":"2024-09-13T17:09:11.126604+0000","last_active":"2024-09-13T17:09:12.103500+0000","last_peered":"2024-09-13T17:09:12.103500+0000","last_clean":"2024-09-13T17:09:12.103500+0000","last_became_active":"2024-09-13T17:09:11.126263+0000","last_became_peered":"2024-09-13T17:09:11.126263+0000","last_unstale":"2024-09-13T17:09:12.103500+0000","last_undegraded":"2024-09-13T17:09:12.103500+0000","last_fullsized":"2024-09-13T17:09:12.103500+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:37.140050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103350+0000","last_change":"2024-09-13T17:09:11.133584+0000","last_active":"2024-09-13T17:09:12.103350+0000","last_peered":"2024-09-13T17:09:12.103350+0000","last_clean":"2024-09-13T17:09:12.103350+0000","last_became_active":"2024-09-13T17:09:11.133027+0000","last_became_peered":"2024-09-13T17:09:11.133027+0000","last_unstale":"2024-09-13T17:09:12.103350+0000","last_undegraded":"2024-09-13T17:09:12.103350+0000","last_fullsized":"2024-09-13T17:09:12.103350+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126715+0000","last_change":"2024-09-13T17:09:11.117305+0000","last_active":"2024-09-13T17:09:11.126715+0000","last_peered":"2024-09-13T17:09:11.126715+0000","last_clean":"2024-09-13T17:09:11.126715+0000","last_became_active":"2024-09-13T17:09:11.117081+0000","last_became_peered":"2024-09-13T17:09:11.117081+0000","last_unstale":"2024-09-13T17:09:11.126715+0000","last_undegraded":"2024-09-13T17:09:11.126715+0000","last_fullsized":"2024-09-13T17:09:11.126715+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:49.078243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122786+0000","last_change":"2024-09-13T17:09:11.115971+0000","last_active":"2024-09-13T17:09:11.122786+0000","last_peered":"2024-09-13T17:09:11.122786+0000","last_clean":"2024-09-13T17:09:11.122786+0000","last_became_active":"2024-09-13T17:09:11.115562+0000","last_became_peered":"2024-09-13T17:09:11.115562+0000","last_unstale":"2024-09-13T17:09:11.122786+0000","last_undegraded":"2024-09-13T17:09:11.122786+0000","last_fullsized":"2024-09-13T17:09:11.122786+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125446+0000","last_change":"2024-09-13T17:09:11.121800+0000","last_active":"2024-09-13T17:09:11.125446+0000","last_peered":"2024-09-13T17:09:11.125446+0000","last_clean":"2024-09-13T17:09:11.125446+0000","last_became_active":"2024-09-13T17:09:11.121495+0000","last_became_peered":"2024-09-13T17:09:11.121495+0000","last_unstale":"2024-09-13T17:09:11.125446+0000","last_undegraded":"2024-09-13T17:09:11.125446+0000","last_fullsized":"2024-09-13T17:09:11.125446+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:09.468121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164726+0000","last_change":"2024-09-13T17:09:11.111698+0000","last_active":"2024-09-13T17:09:12.164726+0000","last_peered":"2024-09-13T17:09:12.164726+0000","last_clean":"2024-09-13T17:09:12.164726+0000","last_became_active":"2024-09-13T17:09:11.111548+0000","last_became_peered":"2024-09-13T17:09:11.111548+0000","last_unstale":"2024-09-13T17:09:12.164726+0000","last_undegraded":"2024-09-13T17:09:12.164726+0000","last_fullsized":"2024-09-13T17:09:12.164726+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105756+0000","last_change":"2024-09-13T17:09:11.109940+0000","last_active":"2024-09-13T17:09:12.105756+0000","last_peered":"2024-09-13T17:09:12.105756+0000","last_clean":"2024-09-13T17:09:12.105756+0000","last_became_active":"2024-09-13T17:09:11.109658+0000","last_became_peered":"2024-09-13T17:09:11.109658+0000","last_unstale":"2024-09-13T17:09:12.105756+0000","last_undegraded":"2024-09-13T17:09:12.105756+0000","last_fullsized":"2024-09-13T17:09:12.105756+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.599876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475696+0000","last_change":"2024-09-13T17:09:11.127029+0000","last_active":"2024-09-13T17:09:12.475696+0000","last_peered":"2024-09-13T17:09:12.475696+0000","last_clean":"2024-09-13T17:09:12.475696+0000","last_became_active":"2024-09-13T17:09:11.126794+0000","last_became_peered":"2024-09-13T17:09:11.126794+0000","last_unstale":"2024-09-13T17:09:12.475696+0000","last_undegraded":"2024-09-13T17:09:12.475696+0000","last_fullsized":"2024-09-13T17:09:12.475696+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103585+0000","last_change":"2024-09-13T17:09:11.129233+0000","last_active":"2024-09-13T17:09:12.103585+0000","last_peered":"2024-09-13T17:09:12.103585+0000","last_clean":"2024-09-13T17:09:12.103585+0000","last_became_active":"2024-09-13T17:09:11.127089+0000","last_became_peered":"2024-09-13T17:09:11.127089+0000","last_unstale":"2024-09-13T17:09:12.103585+0000","last_undegraded":"2024-09-13T17:09:12.103585+0000","last_fullsized":"2024-09-13T17:09:12.103585+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475647+0000","last_change":"2024-09-13T17:09:11.126487+0000","last_active":"2024-09-13T17:09:12.475647+0000","last_peered":"2024-09-13T17:09:12.475647+0000","last_clean":"2024-09-13T17:09:12.475647+0000","last_became_active":"2024-09-13T17:09:11.126246+0000","last_became_peered":"2024-09-13T17:09:11.126246+0000","last_unstale":"2024-09-13T17:09:12.475647+0000","last_undegraded":"2024-09-13T17:09:12.475647+0000","last_fullsized":"2024-09-13T17:09:12.475647+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:44.149902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105883+0000","last_change":"2024-09-13T17:09:11.123482+0000","last_active":"2024-09-13T17:09:12.105883+0000","last_peered":"2024-09-13T17:09:12.105883+0000","last_clean":"2024-09-13T17:09:12.105883+0000","last_became_active":"2024-09-13T17:09:11.123180+0000","last_became_peered":"2024-09-13T17:09:11.123180+0000","last_unstale":"2024-09-13T17:09:12.105883+0000","last_undegraded":"2024-09-13T17:09:12.105883+0000","last_fullsized":"2024-09-13T17:09:12.105883+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122802+0000","last_change":"2024-09-13T17:09:11.116112+0000","last_active":"2024-09-13T17:09:11.122802+0000","last_peered":"2024-09-13T17:09:11.122802+0000","last_clean":"2024-09-13T17:09:11.122802+0000","last_became_active":"2024-09-13T17:09:11.115876+0000","last_became_peered":"2024-09-13T17:09:11.115876+0000","last_unstale":"2024-09-13T17:09:11.122802+0000","last_undegraded":"2024-09-13T17:09:11.122802+0000","last_fullsized":"2024-09-13T17:09:11.122802+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.731519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108641+0000","last_change":"2024-09-13T17:09:11.112400+0000","last_active":"2024-09-13T17:09:12.108641+0000","last_peered":"2024-09-13T17:09:12.108641+0000","last_clean":"2024-09-13T17:09:12.108641+0000","last_became_active":"2024-09-13T17:09:11.112098+0000","last_became_peered":"2024-09-13T17:09:11.112098+0000","last_unstale":"2024-09-13T17:09:12.108641+0000","last_undegraded":"2024-09-13T17:09:12.108641+0000","last_fullsized":"2024-09-13T17:09:12.108641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826637+0000","last_change":"2024-09-13T17:09:11.113348+0000","last_active":"2024-09-13T17:09:12.826637+0000","last_peered":"2024-09-13T17:09:12.826637+0000","last_clean":"2024-09-13T17:09:12.826637+0000","last_became_active":"2024-09-13T17:09:11.111583+0000","last_became_peered":"2024-09-13T17:09:11.111583+0000","last_unstale":"2024-09-13T17:09:12.826637+0000","last_undegraded":"2024-09-13T17:09:12.826637+0000","last_fullsized":"2024-09-13T17:09:12.826637+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:55.503431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108222+0000","last_change":"2024-09-13T17:09:11.117549+0000","last_active":"2024-09-13T17:09:12.108222+0000","last_peered":"2024-09-13T17:09:12.108222+0000","last_clean":"2024-09-13T17:09:12.108222+0000","last_became_active":"2024-09-13T17:09:11.116981+0000","last_became_peered":"2024-09-13T17:09:11.116981+0000","last_unstale":"2024-09-13T17:09:12.108222+0000","last_undegraded":"2024-09-13T17:09:12.108222+0000","last_fullsized":"2024-09-13T17:09:12.108222+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164997+0000","last_change":"2024-09-13T17:09:11.111335+0000","last_active":"2024-09-13T17:09:12.164997+0000","last_peered":"2024-09-13T17:09:12.164997+0000","last_clean":"2024-09-13T17:09:12.164997+0000","last_became_active":"2024-09-13T17:09:11.111049+0000","last_became_peered":"2024-09-13T17:09:11.111049+0000","last_unstale":"2024-09-13T17:09:12.164997+0000","last_undegraded":"2024-09-13T17:09:12.164997+0000","last_fullsized":"2024-09-13T17:09:12.164997+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:50.054986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826878+0000","last_change":"2024-09-13T17:09:10.681887+0000","last_active":"2024-09-13T17:09:12.826878+0000","last_peered":"2024-09-13T17:09:12.826878+0000","last_clean":"2024-09-13T17:09:12.826878+0000","last_became_active":"2024-09-13T17:09:10.680423+0000","last_became_peered":"2024-09-13T17:09:10.680423+0000","last_unstale":"2024-09-13T17:09:12.826878+0000","last_undegraded":"2024-09-13T17:09:12.826878+0000","last_fullsized":"2024-09-13T17:09:12.826878+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:21.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826795+0000","last_change":"2024-09-13T17:09:11.112682+0000","last_active":"2024-09-13T17:09:12.826795+0000","last_peered":"2024-09-13T17:09:12.826795+0000","last_clean":"2024-09-13T17:09:12.826795+0000","last_became_active":"2024-09-13T17:09:11.111830+0000","last_became_peered":"2024-09-13T17:09:11.111830+0000","last_unstale":"2024-09-13T17:09:12.826795+0000","last_undegraded":"2024-09-13T17:09:12.826795+0000","last_fullsized":"2024-09-13T17:09:12.826795+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:51.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164973+0000","last_change":"2024-09-13T17:09:10.680542+0000","last_active":"2024-09-13T17:09:12.164973+0000","last_peered":"2024-09-13T17:09:12.164973+0000","last_clean":"2024-09-13T17:09:12.164973+0000","last_became_active":"2024-09-13T17:09:10.680153+0000","last_became_peered":"2024-09-13T17:09:10.680153+0000","last_unstale":"2024-09-13T17:09:12.164973+0000","last_undegraded":"2024-09-13T17:09:12.164973+0000","last_fullsized":"2024-09-13T17:09:12.164973+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122855+0000","last_change":"2024-09-13T17:09:11.119429+0000","last_active":"2024-09-13T17:09:11.122855+0000","last_peered":"2024-09-13T17:09:11.122855+0000","last_clean":"2024-09-13T17:09:11.122855+0000","last_became_active":"2024-09-13T17:09:11.119138+0000","last_became_peered":"2024-09-13T17:09:11.119138+0000","last_unstale":"2024-09-13T17:09:11.122855+0000","last_undegraded":"2024-09-13T17:09:11.122855+0000","last_fullsized":"2024-09-13T17:09:11.122855+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:25:44.149902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103347+0000","last_change":"2024-09-13T17:09:10.673266+0000","last_active":"2024-09-13T17:09:12.103347+0000","last_peered":"2024-09-13T17:09:12.103347+0000","last_clean":"2024-09-13T17:09:12.103347+0000","last_became_active":"2024-09-13T17:09:10.672994+0000","last_became_peered":"2024-09-13T17:09:10.672994+0000","last_unstale":"2024-09-13T17:09:12.103347+0000","last_undegraded":"2024-09-13T17:09:12.103347+0000","last_fullsized":"2024-09-13T17:09:12.103347+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:50.428518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475693+0000","last_change":"2024-09-13T17:09:11.126063+0000","last_active":"2024-09-13T17:09:12.475693+0000","last_peered":"2024-09-13T17:09:12.475693+0000","last_clean":"2024-09-13T17:09:12.475693+0000","last_became_active":"2024-09-13T17:09:11.125816+0000","last_became_peered":"2024-09-13T17:09:11.125816+0000","last_unstale":"2024-09-13T17:09:12.475693+0000","last_undegraded":"2024-09-13T17:09:12.475693+0000","last_fullsized":"2024-09-13T17:09:12.475693+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:07.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.132373+0000","last_change":"2024-09-13T17:09:10.686826+0000","last_active":"2024-09-13T17:09:11.132373+0000","last_peered":"2024-09-13T17:09:11.132373+0000","last_clean":"2024-09-13T17:09:11.132373+0000","last_became_active":"2024-09-13T17:09:10.686541+0000","last_became_peered":"2024-09-13T17:09:10.686541+0000","last_unstale":"2024-09-13T17:09:11.132373+0000","last_undegraded":"2024-09-13T17:09:11.132373+0000","last_fullsized":"2024-09-13T17:09:11.132373+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108656+0000","last_change":"2024-09-13T17:09:11.117807+0000","last_active":"2024-09-13T17:09:12.108656+0000","last_peered":"2024-09-13T17:09:12.108656+0000","last_clean":"2024-09-13T17:09:12.108656+0000","last_became_active":"2024-09-13T17:09:11.117036+0000","last_became_peered":"2024-09-13T17:09:11.117036+0000","last_unstale":"2024-09-13T17:09:12.108656+0000","last_undegraded":"2024-09-13T17:09:12.108656+0000","last_fullsized":"2024-09-13T17:09:12.108656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:52.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.120031+0000","last_change":"2024-09-13T17:09:10.680350+0000","last_active":"2024-09-13T17:09:11.120031+0000","last_peered":"2024-09-13T17:09:11.120031+0000","last_clean":"2024-09-13T17:09:11.120031+0000","last_became_active":"2024-09-13T17:09:10.680081+0000","last_became_peered":"2024-09-13T17:09:10.680081+0000","last_unstale":"2024-09-13T17:09:11.120031+0000","last_undegraded":"2024-09-13T17:09:11.120031+0000","last_fullsized":"2024-09-13T17:09:11.120031+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:38.198840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105781+0000","last_change":"2024-09-13T17:09:11.123632+0000","last_active":"2024-09-13T17:09:12.105781+0000","last_peered":"2024-09-13T17:09:12.105781+0000","last_clean":"2024-09-13T17:09:12.105781+0000","last_became_active":"2024-09-13T17:09:11.123341+0000","last_became_peered":"2024-09-13T17:09:11.123341+0000","last_unstale":"2024-09-13T17:09:12.105781+0000","last_undegraded":"2024-09-13T17:09:12.105781+0000","last_fullsized":"2024-09-13T17:09:12.105781+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:52.005203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.124377+0000","last_change":"2024-09-13T17:09:10.677543+0000","last_active":"2024-09-13T17:09:11.124377+0000","last_peered":"2024-09-13T17:09:11.124377+0000","last_clean":"2024-09-13T17:09:11.124377+0000","last_became_active":"2024-09-13T17:09:10.677232+0000","last_became_peered":"2024-09-13T17:09:10.677232+0000","last_unstale":"2024-09-13T17:09:11.124377+0000","last_undegraded":"2024-09-13T17:09:11.124377+0000","last_fullsized":"2024-09-13T17:09:11.124377+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:02.036847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.116223+0000","last_change":"2024-09-13T17:09:11.111933+0000","last_active":"2024-09-13T17:09:11.116223+0000","last_peered":"2024-09-13T17:09:11.116223+0000","last_clean":"2024-09-13T17:09:11.116223+0000","last_became_active":"2024-09-13T17:09:11.111637+0000","last_became_peered":"2024-09-13T17:09:11.111637+0000","last_unstale":"2024-09-13T17:09:11.116223+0000","last_undegraded":"2024-09-13T17:09:11.116223+0000","last_fullsized":"2024-09-13T17:09:11.116223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:49.078243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102881+0000","last_change":"2024-09-13T17:09:10.685098+0000","last_active":"2024-09-13T17:09:12.102881+0000","last_peered":"2024-09-13T17:09:12.102881+0000","last_clean":"2024-09-13T17:09:12.102881+0000","last_became_active":"2024-09-13T17:09:10.684813+0000","last_became_peered":"2024-09-13T17:09:10.684813+0000","last_unstale":"2024-09-13T17:09:12.102881+0000","last_undegraded":"2024-09-13T17:09:12.102881+0000","last_fullsized":"2024-09-13T17:09:12.102881+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:49.049253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475259+0000","last_change":"2024-09-13T17:09:11.119710+0000","last_active":"2024-09-13T17:09:12.475259+0000","last_peered":"2024-09-13T17:09:12.475259+0000","last_clean":"2024-09-13T17:09:12.475259+0000","last_became_active":"2024-09-13T17:09:11.119415+0000","last_became_peered":"2024-09-13T17:09:11.119415+0000","last_unstale":"2024-09-13T17:09:12.475259+0000","last_undegraded":"2024-09-13T17:09:12.475259+0000","last_fullsized":"2024-09-13T17:09:12.475259+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:02.205155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.098883+0000","last_change":"2024-09-13T17:09:10.678609+0000","last_active":"2024-09-13T17:09:11.098883+0000","last_peered":"2024-09-13T17:09:11.098883+0000","last_clean":"2024-09-13T17:09:11.098883+0000","last_became_active":"2024-09-13T17:09:10.678288+0000","last_became_peered":"2024-09-13T17:09:10.678288+0000","last_unstale":"2024-09-13T17:09:11.098883+0000","last_undegraded":"2024-09-13T17:09:11.098883+0000","last_fullsized":"2024-09-13T17:09:11.098883+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106084+0000","last_change":"2024-09-13T17:09:11.128190+0000","last_active":"2024-09-13T17:09:12.106084+0000","last_peered":"2024-09-13T17:09:12.106084+0000","last_clean":"2024-09-13T17:09:12.106084+0000","last_became_active":"2024-09-13T17:09:11.127856+0000","last_became_peered":"2024-09-13T17:09:11.127856+0000","last_unstale":"2024-09-13T17:09:12.106084+0000","last_undegraded":"2024-09-13T17:09:12.106084+0000","last_fullsized":"2024-09-13T17:09:12.106084+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:52.626256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103525+0000","last_change":"2024-09-13T17:09:10.681781+0000","last_active":"2024-09-13T17:09:12.103525+0000","last_peered":"2024-09-13T17:09:12.103525+0000","last_clean":"2024-09-13T17:09:12.103525+0000","last_became_active":"2024-09-13T17:09:10.681228+0000","last_became_peered":"2024-09-13T17:09:10.681228+0000","last_unstale":"2024-09-13T17:09:12.103525+0000","last_undegraded":"2024-09-13T17:09:12.103525+0000","last_fullsized":"2024-09-13T17:09:12.103525+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:29.162757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103525+0000","last_change":"2024-09-13T17:09:11.134679+0000","last_active":"2024-09-13T17:09:12.103525+0000","last_peered":"2024-09-13T17:09:12.103525+0000","last_clean":"2024-09-13T17:09:12.103525+0000","last_became_active":"2024-09-13T17:09:11.133887+0000","last_became_peered":"2024-09-13T17:09:11.133887+0000","last_unstale":"2024-09-13T17:09:12.103525+0000","last_undegraded":"2024-09-13T17:09:12.103525+0000","last_fullsized":"2024-09-13T17:09:12.103525+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:01.525043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.165015+0000","last_change":"2024-09-13T17:09:10.674536+0000","last_active":"2024-09-13T17:09:12.165015+0000","last_peered":"2024-09-13T17:09:12.165015+0000","last_clean":"2024-09-13T17:09:12.165015+0000","last_became_active":"2024-09-13T17:09:10.674183+0000","last_became_peered":"2024-09-13T17:09:10.674183+0000","last_unstale":"2024-09-13T17:09:12.165015+0000","last_undegraded":"2024-09-13T17:09:12.165015+0000","last_fullsized":"2024-09-13T17:09:12.165015+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103377+0000","last_change":"2024-09-13T17:09:11.125874+0000","last_active":"2024-09-13T17:09:12.103377+0000","last_peered":"2024-09-13T17:09:12.103377+0000","last_clean":"2024-09-13T17:09:12.103377+0000","last_became_active":"2024-09-13T17:09:11.125649+0000","last_became_peered":"2024-09-13T17:09:11.125649+0000","last_unstale":"2024-09-13T17:09:12.103377+0000","last_undegraded":"2024-09-13T17:09:12.103377+0000","last_fullsized":"2024-09-13T17:09:12.103377+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:32.207419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106091+0000","last_change":"2024-09-13T17:09:10.681100+0000","last_active":"2024-09-13T17:09:12.106091+0000","last_peered":"2024-09-13T17:09:12.106091+0000","last_clean":"2024-09-13T17:09:12.106091+0000","last_became_active":"2024-09-13T17:09:10.680814+0000","last_became_peered":"2024-09-13T17:09:10.680814+0000","last_unstale":"2024-09-13T17:09:12.106091+0000","last_undegraded":"2024-09-13T17:09:12.106091+0000","last_fullsized":"2024-09-13T17:09:12.106091+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:49.049253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105973+0000","last_change":"2024-09-13T17:09:11.124755+0000","last_active":"2024-09-13T17:09:12.105973+0000","last_peered":"2024-09-13T17:09:12.105973+0000","last_clean":"2024-09-13T17:09:12.105973+0000","last_became_active":"2024-09-13T17:09:11.124164+0000","last_became_peered":"2024-09-13T17:09:11.124164+0000","last_unstale":"2024-09-13T17:09:12.105973+0000","last_undegraded":"2024-09-13T17:09:12.105973+0000","last_fullsized":"2024-09-13T17:09:12.105973+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103364+0000","last_change":"2024-09-13T17:09:10.677273+0000","last_active":"2024-09-13T17:09:12.103364+0000","last_peered":"2024-09-13T17:09:12.103364+0000","last_clean":"2024-09-13T17:09:12.103364+0000","last_became_active":"2024-09-13T17:09:10.677015+0000","last_became_peered":"2024-09-13T17:09:10.677015+0000","last_unstale":"2024-09-13T17:09:12.103364+0000","last_undegraded":"2024-09-13T17:09:12.103364+0000","last_fullsized":"2024-09-13T17:09:12.103364+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:01.751596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475295+0000","last_change":"2024-09-13T17:09:11.117305+0000","last_active":"2024-09-13T17:09:12.475295+0000","last_peered":"2024-09-13T17:09:12.475295+0000","last_clean":"2024-09-13T17:09:12.475295+0000","last_became_active":"2024-09-13T17:09:11.117020+0000","last_became_peered":"2024-09-13T17:09:11.117020+0000","last_unstale":"2024-09-13T17:09:12.475295+0000","last_undegraded":"2024-09-13T17:09:12.475295+0000","last_fullsized":"2024-09-13T17:09:12.475295+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:19:16.849690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122874+0000","last_change":"2024-09-13T17:09:10.678353+0000","last_active":"2024-09-13T17:09:11.122874+0000","last_peered":"2024-09-13T17:09:11.122874+0000","last_clean":"2024-09-13T17:09:11.122874+0000","last_became_active":"2024-09-13T17:09:10.678049+0000","last_became_peered":"2024-09-13T17:09:10.678049+0000","last_unstale":"2024-09-13T17:09:11.122874+0000","last_undegraded":"2024-09-13T17:09:11.122874+0000","last_fullsized":"2024-09-13T17:09:11.122874+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.103223+0000","last_change":"2024-09-13T17:09:11.124757+0000","last_active":"2024-09-13T17:09:12.103223+0000","last_peered":"2024-09-13T17:09:12.103223+0000","last_clean":"2024-09-13T17:09:12.103223+0000","last_became_active":"2024-09-13T17:09:11.124485+0000","last_became_peered":"2024-09-13T17:09:11.124485+0000","last_unstale":"2024-09-13T17:09:12.103223+0000","last_undegraded":"2024-09-13T17:09:12.103223+0000","last_fullsized":"2024-09-13T17:09:12.103223+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:24.071785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475365+0000","last_change":"2024-09-13T17:09:10.679337+0000","last_active":"2024-09-13T17:09:12.475365+0000","last_peered":"2024-09-13T17:09:12.475365+0000","last_clean":"2024-09-13T17:09:12.475365+0000","last_became_active":"2024-09-13T17:09:10.678804+0000","last_became_peered":"2024-09-13T17:09:10.678804+0000","last_unstale":"2024-09-13T17:09:12.475365+0000","last_undegraded":"2024-09-13T17:09:12.475365+0000","last_fullsized":"2024-09-13T17:09:12.475365+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108347+0000","last_change":"2024-09-13T17:09:11.117723+0000","last_active":"2024-09-13T17:09:12.108347+0000","last_peered":"2024-09-13T17:09:12.108347+0000","last_clean":"2024-09-13T17:09:12.108347+0000","last_became_active":"2024-09-13T17:09:11.117307+0000","last_became_peered":"2024-09-13T17:09:11.117307+0000","last_unstale":"2024-09-13T17:09:12.108347+0000","last_undegraded":"2024-09-13T17:09:12.108347+0000","last_fullsized":"2024-09-13T17:09:12.108347+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102858+0000","last_change":"2024-09-13T17:09:10.679490+0000","last_active":"2024-09-13T17:09:12.102858+0000","last_peered":"2024-09-13T17:09:12.102858+0000","last_clean":"2024-09-13T17:09:12.102858+0000","last_became_active":"2024-09-13T17:09:10.679222+0000","last_became_peered":"2024-09-13T17:09:10.679222+0000","last_unstale":"2024-09-13T17:09:12.102858+0000","last_undegraded":"2024-09-13T17:09:12.102858+0000","last_fullsized":"2024-09-13T17:09:12.102858+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:24:54.497699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826690+0000","last_change":"2024-09-13T17:09:11.119436+0000","last_active":"2024-09-13T17:09:12.826690+0000","last_peered":"2024-09-13T17:09:12.826690+0000","last_clean":"2024-09-13T17:09:12.826690+0000","last_became_active":"2024-09-13T17:09:11.117438+0000","last_became_peered":"2024-09-13T17:09:11.117438+0000","last_unstale":"2024-09-13T17:09:12.826690+0000","last_undegraded":"2024-09-13T17:09:12.826690+0000","last_fullsized":"2024-09-13T17:09:12.826690+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:38.504248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102925+0000","last_change":"2024-09-13T17:09:10.676478+0000","last_active":"2024-09-13T17:09:12.102925+0000","last_peered":"2024-09-13T17:09:12.102925+0000","last_clean":"2024-09-13T17:09:12.102925+0000","last_became_active":"2024-09-13T17:09:10.675860+0000","last_became_peered":"2024-09-13T17:09:10.675860+0000","last_unstale":"2024-09-13T17:09:12.102925+0000","last_undegraded":"2024-09-13T17:09:12.102925+0000","last_fullsized":"2024-09-13T17:09:12.102925+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.122757+0000","last_change":"2024-09-13T17:09:11.114884+0000","last_active":"2024-09-13T17:09:11.122757+0000","last_peered":"2024-09-13T17:09:11.122757+0000","last_clean":"2024-09-13T17:09:11.122757+0000","last_became_active":"2024-09-13T17:09:11.114634+0000","last_became_peered":"2024-09-13T17:09:11.114634+0000","last_unstale":"2024-09-13T17:09:11.122757+0000","last_undegraded":"2024-09-13T17:09:11.122757+0000","last_fullsized":"2024-09-13T17:09:11.122757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475768+0000","last_change":"2024-09-13T17:09:10.678443+0000","last_active":"2024-09-13T17:09:12.475768+0000","last_peered":"2024-09-13T17:09:12.475768+0000","last_clean":"2024-09-13T17:09:12.475768+0000","last_became_active":"2024-09-13T17:09:10.676390+0000","last_became_peered":"2024-09-13T17:09:10.676390+0000","last_unstale":"2024-09-13T17:09:12.475768+0000","last_undegraded":"2024-09-13T17:09:12.475768+0000","last_fullsized":"2024-09-13T17:09:12.475768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:37.498516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106144+0000","last_change":"2024-09-13T17:09:11.118052+0000","last_active":"2024-09-13T17:09:12.106144+0000","last_peered":"2024-09-13T17:09:12.106144+0000","last_clean":"2024-09-13T17:09:12.106144+0000","last_became_active":"2024-09-13T17:09:11.116981+0000","last_became_peered":"2024-09-13T17:09:11.116981+0000","last_unstale":"2024-09-13T17:09:12.106144+0000","last_undegraded":"2024-09-13T17:09:12.106144+0000","last_fullsized":"2024-09-13T17:09:12.106144+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:32.685108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.119904+0000","last_change":"2024-09-13T17:09:10.674921+0000","last_active":"2024-09-13T17:09:11.119904+0000","last_peered":"2024-09-13T17:09:11.119904+0000","last_clean":"2024-09-13T17:09:11.119904+0000","last_became_active":"2024-09-13T17:09:10.674586+0000","last_became_peered":"2024-09-13T17:09:10.674586+0000","last_unstale":"2024-09-13T17:09:11.119904+0000","last_undegraded":"2024-09-13T17:09:11.119904+0000","last_fullsized":"2024-09-13T17:09:11.119904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:53.442943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475567+0000","last_change":"2024-09-13T17:09:11.126719+0000","last_active":"2024-09-13T17:09:12.475567+0000","last_peered":"2024-09-13T17:09:12.475567+0000","last_clean":"2024-09-13T17:09:12.475567+0000","last_became_active":"2024-09-13T17:09:11.123207+0000","last_became_peered":"2024-09-13T17:09:11.123207+0000","last_unstale":"2024-09-13T17:09:12.475567+0000","last_undegraded":"2024-09-13T17:09:12.475567+0000","last_fullsized":"2024-09-13T17:09:12.475567+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:09.468121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":11,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.120711+0000","last_change":"2024-09-13T17:09:10.682727+0000","last_active":"2024-09-13T17:09:11.120711+0000","last_peered":"2024-09-13T17:09:11.120711+0000","last_clean":"2024-09-13T17:09:11.120711+0000","last_became_active":"2024-09-13T17:09:10.682437+0000","last_became_peered":"2024-09-13T17:09:10.682437+0000","last_unstale":"2024-09-13T17:09:11.120711+0000","last_undegraded":"2024-09-13T17:09:11.120711+0000","last_fullsized":"2024-09-13T17:09:11.120711+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:11:21.165481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126774+0000","last_change":"2024-09-13T17:09:11.120060+0000","last_active":"2024-09-13T17:09:11.126774+0000","last_peered":"2024-09-13T17:09:11.126774+0000","last_clean":"2024-09-13T17:09:11.126774+0000","last_became_active":"2024-09-13T17:09:11.118745+0000","last_became_peered":"2024-09-13T17:09:11.118745+0000","last_unstale":"2024-09-13T17:09:11.126774+0000","last_undegraded":"2024-09-13T17:09:11.126774+0000","last_fullsized":"2024-09-13T17:09:11.126774+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:30.168489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100339+0000","last_change":"2024-09-13T17:09:10.677405+0000","last_active":"2024-09-13T17:09:11.100339+0000","last_peered":"2024-09-13T17:09:11.100339+0000","last_clean":"2024-09-13T17:09:11.100339+0000","last_became_active":"2024-09-13T17:09:10.677113+0000","last_became_peered":"2024-09-13T17:09:10.677113+0000","last_unstale":"2024-09-13T17:09:11.100339+0000","last_undegraded":"2024-09-13T17:09:11.100339+0000","last_fullsized":"2024-09-13T17:09:11.100339+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126743+0000","last_change":"2024-09-13T17:09:11.116724+0000","last_active":"2024-09-13T17:09:11.126743+0000","last_peered":"2024-09-13T17:09:11.126743+0000","last_clean":"2024-09-13T17:09:11.126743+0000","last_became_active":"2024-09-13T17:09:11.114532+0000","last_became_peered":"2024-09-13T17:09:11.114532+0000","last_unstale":"2024-09-13T17:09:11.126743+0000","last_undegraded":"2024-09-13T17:09:11.126743+0000","last_fullsized":"2024-09-13T17:09:11.126743+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:02.757328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164853+0000","last_change":"2024-09-13T17:09:10.680509+0000","last_active":"2024-09-13T17:09:12.164853+0000","last_peered":"2024-09-13T17:09:12.164853+0000","last_clean":"2024-09-13T17:09:12.164853+0000","last_became_active":"2024-09-13T17:09:10.680218+0000","last_became_peered":"2024-09-13T17:09:10.680218+0000","last_unstale":"2024-09-13T17:09:12.164853+0000","last_undegraded":"2024-09-13T17:09:12.164853+0000","last_fullsized":"2024-09-13T17:09:12.164853+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125430+0000","last_change":"2024-09-13T17:09:11.113188+0000","last_active":"2024-09-13T17:09:11.125430+0000","last_peered":"2024-09-13T17:09:11.125430+0000","last_clean":"2024-09-13T17:09:11.125430+0000","last_became_active":"2024-09-13T17:09:11.112842+0000","last_became_peered":"2024-09-13T17:09:11.112842+0000","last_unstale":"2024-09-13T17:09:11.125430+0000","last_undegraded":"2024-09-13T17:09:11.125430+0000","last_fullsized":"2024-09-13T17:09:11.125430+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:51.434251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826569+0000","last_change":"2024-09-13T17:09:10.683670+0000","last_active":"2024-09-13T17:09:12.826569+0000","last_peered":"2024-09-13T17:09:12.826569+0000","last_clean":"2024-09-13T17:09:12.826569+0000","last_became_active":"2024-09-13T17:09:10.681296+0000","last_became_peered":"2024-09-13T17:09:10.681296+0000","last_unstale":"2024-09-13T17:09:12.826569+0000","last_undegraded":"2024-09-13T17:09:12.826569+0000","last_fullsized":"2024-09-13T17:09:12.826569+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:09.166659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.109332+0000","last_change":"2024-09-13T17:09:11.107205+0000","last_active":"2024-09-13T17:09:11.109332+0000","last_peered":"2024-09-13T17:09:11.109332+0000","last_clean":"2024-09-13T17:09:11.109332+0000","last_became_active":"2024-09-13T17:09:11.106769+0000","last_became_peered":"2024-09-13T17:09:11.106769+0000","last_unstale":"2024-09-13T17:09:11.109332+0000","last_undegraded":"2024-09-13T17:09:11.109332+0000","last_fullsized":"2024-09-13T17:09:11.109332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:46.428704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100414+0000","last_change":"2024-09-13T17:09:10.671120+0000","last_active":"2024-09-13T17:09:11.100414+0000","last_peered":"2024-09-13T17:09:11.100414+0000","last_clean":"2024-09-13T17:09:11.100414+0000","last_became_active":"2024-09-13T17:09:10.670790+0000","last_became_peered":"2024-09-13T17:09:10.670790+0000","last_unstale":"2024-09-13T17:09:11.100414+0000","last_undegraded":"2024-09-13T17:09:11.100414+0000","last_fullsized":"2024-09-13T17:09:11.100414+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:29.934709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108332+0000","last_change":"2024-09-13T17:09:11.110016+0000","last_active":"2024-09-13T17:09:12.108332+0000","last_peered":"2024-09-13T17:09:12.108332+0000","last_clean":"2024-09-13T17:09:12.108332+0000","last_became_active":"2024-09-13T17:09:11.109777+0000","last_became_peered":"2024-09-13T17:09:11.109777+0000","last_unstale":"2024-09-13T17:09:12.108332+0000","last_undegraded":"2024-09-13T17:09:12.108332+0000","last_fullsized":"2024-09-13T17:09:12.108332+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:21:32.685108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.105621+0000","last_change":"2024-09-13T17:09:10.681520+0000","last_active":"2024-09-13T17:09:12.105621+0000","last_peered":"2024-09-13T17:09:12.105621+0000","last_clean":"2024-09-13T17:09:12.105621+0000","last_became_active":"2024-09-13T17:09:10.681188+0000","last_became_peered":"2024-09-13T17:09:10.681188+0000","last_unstale":"2024-09-13T17:09:12.105621+0000","last_undegraded":"2024-09-13T17:09:12.105621+0000","last_fullsized":"2024-09-13T17:09:12.105621+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:46:50.428518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.126749+0000","last_change":"2024-09-13T17:09:11.116597+0000","last_active":"2024-09-13T17:09:11.126749+0000","last_peered":"2024-09-13T17:09:11.126749+0000","last_clean":"2024-09-13T17:09:11.126749+0000","last_became_active":"2024-09-13T17:09:11.116290+0000","last_became_peered":"2024-09-13T17:09:11.116290+0000","last_unstale":"2024-09-13T17:09:11.126749+0000","last_undegraded":"2024-09-13T17:09:11.126749+0000","last_fullsized":"2024-09-13T17:09:11.126749+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:20.961969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475783+0000","last_change":"2024-09-13T17:09:10.679585+0000","last_active":"2024-09-13T17:09:12.475783+0000","last_peered":"2024-09-13T17:09:12.475783+0000","last_clean":"2024-09-13T17:09:12.475783+0000","last_became_active":"2024-09-13T17:09:10.679255+0000","last_became_peered":"2024-09-13T17:09:10.679255+0000","last_unstale":"2024-09-13T17:09:12.475783+0000","last_undegraded":"2024-09-13T17:09:12.475783+0000","last_fullsized":"2024-09-13T17:09:12.475783+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.106152+0000","last_change":"2024-09-13T17:09:11.125856+0000","last_active":"2024-09-13T17:09:12.106152+0000","last_peered":"2024-09-13T17:09:12.106152+0000","last_clean":"2024-09-13T17:09:12.106152+0000","last_became_active":"2024-09-13T17:09:11.125618+0000","last_became_peered":"2024-09-13T17:09:11.125618+0000","last_unstale":"2024-09-13T17:09:12.106152+0000","last_undegraded":"2024-09-13T17:09:12.106152+0000","last_fullsized":"2024-09-13T17:09:12.106152+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:03.526335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.165089+0000","last_change":"2024-09-13T17:09:10.681092+0000","last_active":"2024-09-13T17:09:12.165089+0000","last_peered":"2024-09-13T17:09:12.165089+0000","last_clean":"2024-09-13T17:09:12.165089+0000","last_became_active":"2024-09-13T17:09:10.680311+0000","last_became_peered":"2024-09-13T17:09:10.680311+0000","last_unstale":"2024-09-13T17:09:12.165089+0000","last_undegraded":"2024-09-13T17:09:12.165089+0000","last_fullsized":"2024-09-13T17:09:12.165089+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:38.198840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164875+0000","last_change":"2024-09-13T17:09:11.126127+0000","last_active":"2024-09-13T17:09:12.164875+0000","last_peered":"2024-09-13T17:09:12.164875+0000","last_clean":"2024-09-13T17:09:12.164875+0000","last_became_active":"2024-09-13T17:09:11.125698+0000","last_became_peered":"2024-09-13T17:09:11.125698+0000","last_unstale":"2024-09-13T17:09:12.164875+0000","last_undegraded":"2024-09-13T17:09:12.164875+0000","last_fullsized":"2024-09-13T17:09:12.164875+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:30.168489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475665+0000","last_change":"2024-09-13T17:09:10.684142+0000","last_active":"2024-09-13T17:09:12.475665+0000","last_peered":"2024-09-13T17:09:12.475665+0000","last_clean":"2024-09-13T17:09:12.475665+0000","last_became_active":"2024-09-13T17:09:10.683853+0000","last_became_peered":"2024-09-13T17:09:10.683853+0000","last_unstale":"2024-09-13T17:09:12.475665+0000","last_undegraded":"2024-09-13T17:09:12.475665+0000","last_fullsized":"2024-09-13T17:09:12.475665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:31.818414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826611+0000","last_change":"2024-09-13T17:09:11.123301+0000","last_active":"2024-09-13T17:09:12.826611+0000","last_peered":"2024-09-13T17:09:12.826611+0000","last_clean":"2024-09-13T17:09:12.826611+0000","last_became_active":"2024-09-13T17:09:11.122785+0000","last_became_peered":"2024-09-13T17:09:11.122785+0000","last_unstale":"2024-09-13T17:09:12.826611+0000","last_undegraded":"2024-09-13T17:09:12.826611+0000","last_fullsized":"2024-09-13T17:09:12.826611+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:59.294698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164960+0000","last_change":"2024-09-13T17:09:10.669375+0000","last_active":"2024-09-13T17:09:12.164960+0000","last_peered":"2024-09-13T17:09:12.164960+0000","last_clean":"2024-09-13T17:09:12.164960+0000","last_became_active":"2024-09-13T17:09:10.669088+0000","last_became_peered":"2024-09-13T17:09:10.669088+0000","last_unstale":"2024-09-13T17:09:12.164960+0000","last_undegraded":"2024-09-13T17:09:12.164960+0000","last_fullsized":"2024-09-13T17:09:12.164960+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:02.036847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.164869+0000","last_change":"2024-09-13T17:09:11.112160+0000","last_active":"2024-09-13T17:09:12.164869+0000","last_peered":"2024-09-13T17:09:12.164869+0000","last_clean":"2024-09-13T17:09:12.164869+0000","last_became_active":"2024-09-13T17:09:11.111915+0000","last_became_peered":"2024-09-13T17:09:11.111915+0000","last_unstale":"2024-09-13T17:09:12.164869+0000","last_undegraded":"2024-09-13T17:09:12.164869+0000","last_fullsized":"2024-09-13T17:09:12.164869+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:56.809834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.475603+0000","last_change":"2024-09-13T17:09:10.681264+0000","last_active":"2024-09-13T17:09:12.475603+0000","last_peered":"2024-09-13T17:09:12.475603+0000","last_clean":"2024-09-13T17:09:12.475603+0000","last_became_active":"2024-09-13T17:09:10.680742+0000","last_became_peered":"2024-09-13T17:09:10.680742+0000","last_unstale":"2024-09-13T17:09:12.475603+0000","last_undegraded":"2024-09-13T17:09:12.475603+0000","last_fullsized":"2024-09-13T17:09:12.475603+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:51.539813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102825+0000","last_change":"2024-09-13T17:09:11.126061+0000","last_active":"2024-09-13T17:09:12.102825+0000","last_peered":"2024-09-13T17:09:12.102825+0000","last_clean":"2024-09-13T17:09:12.102825+0000","last_became_active":"2024-09-13T17:09:11.125808+0000","last_became_peered":"2024-09-13T17:09:11.125808+0000","last_unstale":"2024-09-13T17:09:12.102825+0000","last_undegraded":"2024-09-13T17:09:12.102825+0000","last_fullsized":"2024-09-13T17:09:12.102825+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:15.296969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.100545+0000","last_change":"2024-09-13T17:09:10.675570+0000","last_active":"2024-09-13T17:09:11.100545+0000","last_peered":"2024-09-13T17:09:11.100545+0000","last_clean":"2024-09-13T17:09:11.100545+0000","last_became_active":"2024-09-13T17:09:10.675264+0000","last_became_peered":"2024-09-13T17:09:10.675264+0000","last_unstale":"2024-09-13T17:09:11.100545+0000","last_undegraded":"2024-09-13T17:09:11.100545+0000","last_fullsized":"2024-09-13T17:09:11.100545+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:15:40.804455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":9,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.125456+0000","last_change":"2024-09-13T17:09:11.121833+0000","last_active":"2024-09-13T17:09:11.125456+0000","last_peered":"2024-09-13T17:09:11.125456+0000","last_clean":"2024-09-13T17:09:11.125456+0000","last_became_active":"2024-09-13T17:09:11.121546+0000","last_became_peered":"2024-09-13T17:09:11.121546+0000","last_unstale":"2024-09-13T17:09:11.125456+0000","last_undegraded":"2024-09-13T17:09:11.125456+0000","last_fullsized":"2024-09-13T17:09:11.125456+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:07.994196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.102849+0000","last_change":"2024-09-13T17:09:10.682643+0000","last_active":"2024-09-13T17:09:12.102849+0000","last_peered":"2024-09-13T17:09:12.102849+0000","last_clean":"2024-09-13T17:09:12.102849+0000","last_became_active":"2024-09-13T17:09:10.682307+0000","last_became_peered":"2024-09-13T17:09:10.682307+0000","last_unstale":"2024-09-13T17:09:12.102849+0000","last_undegraded":"2024-09-13T17:09:12.102849+0000","last_fullsized":"2024-09-13T17:09:12.102849+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:48.224278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.108653+0000","last_change":"2024-09-13T17:09:11.112831+0000","last_active":"2024-09-13T17:09:12.108653+0000","last_peered":"2024-09-13T17:09:12.108653+0000","last_clean":"2024-09-13T17:09:12.108653+0000","last_became_active":"2024-09-13T17:09:11.112578+0000","last_became_peered":"2024-09-13T17:09:11.112578+0000","last_unstale":"2024-09-13T17:09:12.108653+0000","last_undegraded":"2024-09-13T17:09:12.108653+0000","last_fullsized":"2024-09-13T17:09:12.108653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:10.650101+0000","last_clean_scrub_stamp":"2024-09-13T17:09:10.650101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:47.324888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-13T17:09:12.826487+0000","last_change":"2024-09-13T17:09:10.682484+0000","last_active":"2024-09-13T17:09:12.826487+0000","last_peered":"2024-09-13T17:09:12.826487+0000","last_clean":"2024-09-13T17:09:12.826487+0000","last_became_active":"2024-09-13T17:09:10.682224+0000","last_became_peered":"2024-09-13T17:09:10.682224+0000","last_unstale":"2024-09-13T17:09:12.826487+0000","last_undegraded":"2024-09-13T17:09:12.826487+0000","last_fullsized":"2024-09-13T17:09:12.826487+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:09:09.644369+0000","last_clean_scrub_stamp":"2024-09-13T17:09:09.644369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:56.576124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":174,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-13T17:09:11.103092+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:09:11.103092+0000","last_peered":"2024-09-13T17:09:11.103092+0000","last_clean":"2024-09-13T17:09:11.103092+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:09:11.103092+0000","last_undegraded":"2024-09-13T17:09:11.103092+0000","last_fullsized":"2024-09-13T17:09:11.103092+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":742215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.76300000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.80200000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.84999999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.497},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.90200000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.82799999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.81899999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.56200000000000006}]},{"osd":2,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":1.077,"5min":1.077,"15min":1.077},"last":0.77400000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.755},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.79000000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.63600000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.88},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.94799999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":152429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.61599999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.629}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.75}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59199999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.56699999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":145561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.341,"5min":1.341,"15min":1.341},"last":0.79100000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.63900000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.751}]},{"osd":3,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.871}]},{"osd":5,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.88900000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:09:00 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574866,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31516,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340324,"statfs":{"total":96636764160,"available":96604491776,"internally_reserved":0,"allocated":401408,"data_stored":153343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":2.96,"5min":2.96,"15min":2.96},"last":0.39900000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":3.6829999999999998,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":2.8610000000000002,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.76500000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.321,"5min":1.321,"15min":1.321},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.9300000000000002,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.82699999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.25,"5min":1.25,"15min":1.25},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:09:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":0.753},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.89500000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31480,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340360,"statfs":{"total":96636764160,"available":96604528640,"internally_reserved":0,"allocated":372736,"data_stored":144508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.34999999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.372},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.58499999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.55000000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.2949999999999999},"last":0.749}]},{"osd":6,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.012,"5min":1.012,"15min":1.012},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:08:58 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.56399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574865,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31500,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340340,"statfs":{"total":96636764160,"available":96604508160,"internally_reserved":0,"allocated":389120,"data_stored":147043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.438}]},{"osd":2,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.40799999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.38900000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.3560000000000001,"5min":1.3560000000000001,"15min":1.3560000000000001},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.599,"5min":1.599,"15min":1.599},"last":1.0740000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6870000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.94799999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7310000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.83199999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:08:57 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.91100000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32068,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339772,"statfs":{"total":96636764160,"available":96603926528,"internally_reserved":0,"allocated":974848,"data_stored":741106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":5.5259999999999998,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":5.4889999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":6.0430000000000001,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":4.6760000000000002,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":4.7549999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":4.5940000000000003,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.53500000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":4.734,"5min":4.734,"15min":4.734},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.8079999999999998,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.93999999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":4.944,"5min":4.944,"15min":4.944},"last":0.997},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":4.9710000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.216,"15min":1.216},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":4.8470000000000004,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.016,"5min":1.016,"15min":1.016},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":5.0170000000000003,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.96599999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:08:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":5.0540000000000003,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":4.9950000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.86799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:09:15.701 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:09:15.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:09:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.897+0000 7f3564600640 1 Processor -- start 2024-09-13T17:09:15.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.897+0000 7f3564600640 1 -- start start 2024-09-13T17:09:15.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 0x7f355c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 0x7f355c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06bb40 0x7f355c06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c06c8b0 con 0x7f355c06bb40 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c06cb90 con 0x7f355c15ab60 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3564600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c0cc980 con 0x7f355c15a780 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3562600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06bb40 0x7f355c06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3562600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06bb40 0x7f355c06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47812/0 (socket says 172.21.15.28:47812) 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.898+0000 7f3562600640 1 -- 172.21.15.28:0/3417682992 learned_addr learned my addr 172.21.15.28:0/3417682992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3562600640 1 -- 172.21.15.28:0/3417682992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 msgr2=0x7f355c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3561c00640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 0x7f355c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3561200640 1 --2- 172.21.15.28:0/3417682992 >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 0x7f355c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3562600640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 0x7f355c070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3562600640 1 -- 172.21.15.28:0/3417682992 >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 msgr2=0x7f355c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3562600640 1 --2- 172.21.15.28:0/3417682992 >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 0x7f355c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3562600640 1 -- 172.21.15.28:0/3417682992 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0ccc60 con 0x7f355c06bb40 2024-09-13T17:09:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3561c00640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 0x7f355c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:15.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.899+0000 7f3561200640 1 --2- 172.21.15.28:0/3417682992 >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 0x7f355c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f3562600640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c06bf90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f355c0cf5c0 tx=0x7f355402a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c921acdc5f342c0a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f355be00640 1 -- 172.21.15.28:0/3417682992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554031600 con 0x7f355c06bb40 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f355be00640 1 -- 172.21.15.28:0/3417682992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355402d050 con 0x7f355c06bb40 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f355be00640 1 -- 172.21.15.28:0/3417682992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554031980 con 0x7f355c06bb40 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f3564600640 1 -- 172.21.15.28:0/3417682992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 msgr2=0x7f355c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.900+0000 7f3564600640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c06bf90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f355c0cf5c0 tx=0x7f355402a690 comp rx=0 tx=0).stop 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 -- 172.21.15.28:0/3417682992 shutdown_connections 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c06bf90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 --2- 172.21.15.28:0/3417682992 >> v2:172.21.15.189:3300/0 conn(0x7f355c15ab60 0x7f355c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 --2- 172.21.15.28:0/3417682992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15a780 0x7f355c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 -- 172.21.15.28:0/3417682992 >> 172.21.15.28:0/3417682992 conn(0x7f355c0b0270 msgr2=0x7f355c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 -- 172.21.15.28:0/3417682992 shutdown_connections 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.901+0000 7f3564600640 1 -- 172.21.15.28:0/3417682992 wait complete. 2024-09-13T17:09:15.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.902+0000 7f3564600640 1 Processor -- start 2024-09-13T17:09:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.902+0000 7f3564600640 1 -- start start 2024-09-13T17:09:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c15a780 0x7f355c152270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15ab60 0x7f355c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c153d20 con 0x7f355c06bb40 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c152e70 con 0x7f355c15a780 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3564600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c1509e0 con 0x7f355c15ab60 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.903+0000 7f3561c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47820/0 (socket says 172.21.15.28:47820) 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 -- 172.21.15.28:0/2140107761 learned_addr learned my addr 172.21.15.28:0/2140107761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561200640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c15a780 0x7f355c152270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3562600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15ab60 0x7f355c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 -- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15ab60 msgr2=0x7f355c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15ab60 0x7f355c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 -- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c15a780 msgr2=0x7f355c152270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c15a780 0x7f355c152270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 -- 172.21.15.28:0/2140107761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3554009b30 con 0x7f355c06bb40 2024-09-13T17:09:15.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.904+0000 7f3561c00640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f355000c7f0 tx=0x7f355000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.905+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3550007a30 con 0x7f355c06bb40 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.905+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3550007b90 con 0x7f355c06bb40 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.905+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355000ce40 con 0x7f355c06bb40 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.905+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c153110 con 0x7f355c06bb40 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.905+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1536d0 con 0x7f355c06bb40 2024-09-13T17:09:15.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.906+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c06bf90 con 0x7f355c06bb40 2024-09-13T17:09:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.907+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f35500113f0 con 0x7f355c06bb40 2024-09-13T17:09:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.908+0000 7f3559600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 0x7f353c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:15.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.908+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3550089460 con 0x7f355c06bb40 2024-09-13T17:09:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.908+0000 7f3561200640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 0x7f353c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:15.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.909+0000 7f3561200640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 0x7f353c06d0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f355c1f6bc0 tx=0x7f3548008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:15.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:15.912+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f35500546f0 con 0x7f355c06bb40 2024-09-13T17:09:16.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.154+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f355c071410 con 0x7f355c06bb40 2024-09-13T17:09:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.158+0000 7f3559600640 1 -- 172.21.15.28:0/2140107761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3550057d10 con 0x7f355c06bb40 2024-09-13T17:09:16.160 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:09:16.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 msgr2=0x7f353c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 0x7f353c06d0f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f355c1f6bc0 tx=0x7f3548008040 comp rx=0 tx=0).stop 2024-09-13T17:09:16.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 msgr2=0x7f355c1504a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f355000c7f0 tx=0x7f355000cbf0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 shutdown_connections 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353c06ac30 0x7f353c06d0f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c15ab60 0x7f355c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c15a780 0x7f355c152270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 --2- 172.21.15.28:0/2140107761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06bb40 0x7f355c1504a0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 >> 172.21.15.28:0/2140107761 conn(0x7f355c0b0270 msgr2=0x7f355c070600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 shutdown_connections 2024-09-13T17:09:16.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.160+0000 7f3564600640 1 -- 172.21.15.28:0/2140107761 wait complete. 2024-09-13T17:09:16.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:09:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.332+0000 7fdc33e00640 1 Processor -- start 2024-09-13T17:09:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.333+0000 7fdc33e00640 1 -- start start 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc340b0b50 0x7fdc340b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159820 0x7fdc340b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc340c3850 con 0x7fdc34159440 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc340b2d60 con 0x7fdc340b0b50 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3415a930 con 0x7fdc34159820 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc2be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc340b0b50 0x7fdc340b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc2be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc340b0b50 0x7fdc340b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47832/0 (socket says 172.21.15.28:47832) 2024-09-13T17:09:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc2be00640 1 -- 172.21.15.28:0/2395946704 learned_addr learned my addr 172.21.15.28:0/2395946704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.334+0000 7fdc33400640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159820 0x7fdc340b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.335+0000 7fdc2be00640 1 -- 172.21.15.28:0/2395946704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159820 msgr2=0x7fdc340b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.335+0000 7fdc2be00640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159820 0x7fdc340b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.335+0000 7fdc2be00640 1 -- 172.21.15.28:0/2395946704 >> v1:172.21.15.189:6789/0 conn(0x7fdc34159440 legacy=0x7fdc340b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.335+0000 7fdc2be00640 1 -- 172.21.15.28:0/2395946704 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc34159d20 con 0x7fdc340b0b50 2024-09-13T17:09:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc2be00640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc340b22e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009870 tx=0x7fdc1c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca270e7c6bb6807 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:16.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc32000640 1 -- 172.21.15.28:0/2395946704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c032640 con 0x7fdc340b0b50 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc32000640 1 -- 172.21.15.28:0/2395946704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc1c0327a0 con 0x7fdc340b0b50 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc32000640 1 -- 172.21.15.28:0/2395946704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1c032b30 con 0x7fdc340b0b50 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 -- 172.21.15.28:0/2395946704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 msgr2=0x7fdc340b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc340b22e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009870 tx=0x7fdc1c02a300 comp rx=0 tx=0).stop 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 -- 172.21.15.28:0/2395946704 shutdown_connections 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159820 0x7fdc340b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 --2- 172.21.15.28:0/2395946704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc340b22e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.336+0000 7fdc33e00640 1 -- 172.21.15.28:0/2395946704 >> 172.21.15.28:0/2395946704 conn(0x7fdc34072e40 msgr2=0x7fdc34073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.337+0000 7fdc33e00640 1 -- 172.21.15.28:0/2395946704 shutdown_connections 2024-09-13T17:09:16.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.337+0000 7fdc33e00640 1 -- 172.21.15.28:0/2395946704 wait complete. 2024-09-13T17:09:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.338+0000 7fdc33e00640 1 Processor -- start 2024-09-13T17:09:16.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.338+0000 7fdc33e00640 1 -- start start 2024-09-13T17:09:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 0x7fdc34158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 0x7fdc340ad8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341f2f10 con 0x7fdc340b0b50 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341585a0 con 0x7fdc34159820 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc33e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341566c0 con 0x7fdc34159440 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc32a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc32a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47838/0 (socket says 172.21.15.28:47838) 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.339+0000 7fdc32a00640 1 -- 172.21.15.28:0/3612056089 learned_addr learned my addr 172.21.15.28:0/3612056089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc2be00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 0x7fdc34158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc33400640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 0x7fdc340ad8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 -- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 msgr2=0x7fdc34158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 0x7fdc34158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 -- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 msgr2=0x7fdc340ad8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 0x7fdc340ad8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 -- 172.21.15.28:0/3612056089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1c0094d0 con 0x7fdc340b0b50 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc33400640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 0x7fdc340ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc2be00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 0x7fdc34158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc32a00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400c450 tx=0x7fdc2400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc240052d0 con 0x7fdc340b0b50 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc24005430 con 0x7fdc340b0b50 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24012870 con 0x7fdc340b0b50 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.340+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc34157580 con 0x7fdc340b0b50 2024-09-13T17:09:16.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.341+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc34157b40 con 0x7fdc340b0b50 2024-09-13T17:09:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.342+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc34152ec0 con 0x7fdc340b0b50 2024-09-13T17:09:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.342+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdc240129d0 con 0x7fdc340b0b50 2024-09-13T17:09:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.343+0000 7fdc2b400640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 0x7fdc1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.343+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc24088f60 con 0x7fdc340b0b50 2024-09-13T17:09:16.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.344+0000 7fdc2be00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 0x7fdc1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:16.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.344+0000 7fdc2be00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 0x7fdc1006d0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c034000 tx=0x7fdc1c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:16.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.348+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdc240540a0 con 0x7fdc340b0b50 2024-09-13T17:09:16.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.562+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc3406c410 con 0x7fdc340b0b50 2024-09-13T17:09:16.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.562+0000 7fdc2b400640 1 -- 172.21.15.28:0/3612056089 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fdc240576c0 con 0x7fdc340b0b50 2024-09-13T17:09:16.564 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:09:16.564 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:09:14.102658+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:09:16.565 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 16 2024-09-13T17:09:16.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 msgr2=0x7fdc1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 0x7fdc1006d0c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c034000 tx=0x7fdc1c02e000 comp rx=0 tx=0).stop 2024-09-13T17:09:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 msgr2=0x7fdc341f29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fdc2400c450 tx=0x7fdc2400c910 comp rx=0 tx=0).stop 2024-09-13T17:09:16.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 shutdown_connections 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc1006ac00 0x7fdc1006d0c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc34159820 0x7fdc340ad8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc34159440 0x7fdc34158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 --2- 172.21.15.28:0/3612056089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc340b0b50 0x7fdc341f29d0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 >> 172.21.15.28:0/3612056089 conn(0x7fdc34072e40 msgr2=0x7fdc3415b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 shutdown_connections 2024-09-13T17:09:16.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:16.565+0000 7fdc33e00640 1 -- 172.21.15.28:0/3612056089 wait complete. 2024-09-13T17:09:16.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:09:11.091132+0000', 'modified': '2024-09-13T17:09:14.102658+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5121}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5121': {'gid': 5121, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/696058040', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 696058040}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 696058040}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:09:16.986 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:09:16.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:09:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.142+0000 7fb371c00640 1 Processor -- start 2024-09-13T17:09:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.142+0000 7fb371c00640 1 -- start start 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb371c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 0x7fb36c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb371c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb36c071490 0x7fb36c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb371c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb36c06c8b0 con 0x7fb36c15a780 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb371c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb36c06cb90 con 0x7fb36c071490 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb371c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c0cc980 con 0x7fb36c15ab60 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb36b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb36c071490 0x7fb36c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb36a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 0x7fb36c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb36b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb36c071490 0x7fb36c06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:57810/0 (socket says 172.21.15.28:57810) 2024-09-13T17:09:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.144+0000 7fb36b400640 1 -- 172.21.15.28:0/1003565311 learned_addr learned my addr 172.21.15.28:0/1003565311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.145+0000 7fb36b400640 1 -- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 msgr2=0x7fb36c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.145+0000 7fb36b400640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 0x7fb36c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.145+0000 7fb36b400640 1 -- 172.21.15.28:0/1003565311 >> v1:172.21.15.189:6789/0 conn(0x7fb36c15a780 legacy=0x7fb36c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.145+0000 7fb36b400640 1 -- 172.21.15.28:0/1003565311 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb36c0ccc60 con 0x7fb36c071490 2024-09-13T17:09:17.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.145+0000 7fb36a000640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 0x7fb36c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.146+0000 7fb36b400640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c071490 0x7fb36c06c080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb364009e50 tx=0x7fb36402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddc82b970a2e9447 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.146+0000 7fb369600640 1 -- 172.21.15.28:0/1003565311 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364031600 con 0x7fb36c071490 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.146+0000 7fb369600640 1 -- 172.21.15.28:0/1003565311 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb36402d050 con 0x7fb36c071490 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.146+0000 7fb369600640 1 -- 172.21.15.28:0/1003565311 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364031980 con 0x7fb36c071490 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 -- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c071490 msgr2=0x7fb36c06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c071490 0x7fb36c06c080 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb364009e50 tx=0x7fb36402a710 comp rx=0 tx=0).stop 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 -- 172.21.15.28:0/1003565311 shutdown_connections 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c071490 0x7fb36c06c080 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 --2- 172.21.15.28:0/1003565311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c15ab60 0x7fb36c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 -- 172.21.15.28:0/1003565311 >> 172.21.15.28:0/1003565311 conn(0x7fb36c0b0270 msgr2=0x7fb36c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 -- 172.21.15.28:0/1003565311 shutdown_connections 2024-09-13T17:09:17.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.147+0000 7fb371c00640 1 -- 172.21.15.28:0/1003565311 wait complete. 2024-09-13T17:09:17.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.148+0000 7fb371c00640 1 Processor -- start 2024-09-13T17:09:17.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 -- start start 2024-09-13T17:09:17.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c071490 0x7fb36c0ad550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c15ab60 0x7fb36c154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c14ede0 con 0x7fb36c15a780 2024-09-13T17:09:17.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c14fbd0 con 0x7fb36c15ab60 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.149+0000 7fb371c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c14efc0 con 0x7fb36c071490 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c071490 0x7fb36c0ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47854/0 (socket says 172.21.15.28:47854) 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 -- 172.21.15.28:0/1301075530 learned_addr learned my addr 172.21.15.28:0/1301075530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36b400640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c15ab60 0x7fb36c154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 -- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c071490 msgr2=0x7fb36c0ad550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c071490 0x7fb36c0ad550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 -- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c15ab60 msgr2=0x7fb36c154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c15ab60 0x7fb36c154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.150+0000 7fb36a000640 1 -- 172.21.15.28:0/1301075530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb358009400 con 0x7fb36c15a780 2024-09-13T17:09:17.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb36a000640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb35800e960 tx=0x7fb35800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb362a00640 1 -- 172.21.15.28:0/1301075530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35800ca00 con 0x7fb36c15a780 2024-09-13T17:09:17.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb362a00640 1 -- 172.21.15.28:0/1301075530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35800cb60 con 0x7fb36c15a780 2024-09-13T17:09:17.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb362a00640 1 -- 172.21.15.28:0/1301075530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb358013470 con 0x7fb36c15a780 2024-09-13T17:09:17.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb364009b30 con 0x7fb36c15a780 2024-09-13T17:09:17.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.151+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c14f580 con 0x7fb36c15a780 2024-09-13T17:09:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.153+0000 7fb362a00640 1 -- 172.21.15.28:0/1301075530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb3580135d0 con 0x7fb36c15a780 2024-09-13T17:09:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.154+0000 7fb362a00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 0x7fb34c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.154+0000 7fb362a00640 1 -- 172.21.15.28:0/1301075530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb358089640 con 0x7fb36c15a780 2024-09-13T17:09:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.154+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 0x7fb34c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.154+0000 7fb374a2a740 1 Processor -- start 2024-09-13T17:09:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.154+0000 7fb374a2a740 1 -- start start 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 0x7fb34c06d0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb35c000f60 tx=0x7fb35c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 0x5646f4a32ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f494eab0 con 0x5646f49e56d0 2024-09-13T17:09:17.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f494c240 con 0x5646f49e2e90 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.155+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646f4947b30 con 0x5646f4a0cb80 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47858/0 (socket says 172.21.15.28:47858) 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 -- 172.21.15.28:0/2483244836 learned_addr learned my addr 172.21.15.28:0/2483244836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36b400640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36a000640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 0x5646f4a32ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 -- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 msgr2=0x5646f4a330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 -- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 msgr2=0x5646f4a32ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 0x5646f4a32ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 -- 172.21.15.28:0/2483244836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646f4912bd0 con 0x5646f49e56d0 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36a000640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 0x5646f4a32ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.156+0000 7fb36aa00640 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00f950 tx=0x7fb35c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d14e7805a826e99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.157+0000 7fb357400640 1 -- 172.21.15.28:0/2483244836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00fed0 con 0x5646f49e56d0 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.157+0000 7fb357400640 1 -- 172.21.15.28:0/2483244836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c014be0 con 0x5646f49e56d0 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.157+0000 7fb357400640 1 -- 172.21.15.28:0/2483244836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c020410 con 0x5646f49e56d0 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.157+0000 7fb374a2a740 1 -- 172.21.15.28:0/2483244836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 msgr2=0x5646f49e5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.157+0000 7fb374a2a740 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00f950 tx=0x7fb35c012000 comp rx=0 tx=0).stop 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 -- 172.21.15.28:0/2483244836 shutdown_connections 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e2e90 0x5646f4a32ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 --2- 172.21.15.28:0/2483244836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f49e5ab0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 -- 172.21.15.28:0/2483244836 >> 172.21.15.28:0/2483244836 conn(0x5646f49e1e00 msgr2=0x5646f4a2a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 -- 172.21.15.28:0/2483244836 shutdown_connections 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.158+0000 7fb374a2a740 1 -- 172.21.15.28:0/2483244836 wait complete. 2024-09-13T17:09:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.159+0000 7fb374a2a740 1 -- start start 2024-09-13T17:09:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.159+0000 7fb374a2a740 1 Processor -- start 2024-09-13T17:09:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 0x5646f4a0e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a0e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f48e9260 con 0x5646f49f1900 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f48e8f00 con 0x5646f49e56d0 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646f46aef20 con 0x5646f4a0cb80 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.160+0000 7fb374a2a740 10 client.0 _ll_register_callbacks cb 0x5646f49411a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 0x5646f4a0e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a0e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 0x5646f4a0e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:57842/0 (socket says 172.21.15.28:57842) 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a0e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:40970/0 (socket says 172.21.15.28:40970) 2024-09-13T17:09:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36a000640 1 -- 172.21.15.28:0/1897778129 learned_addr learned my addr 172.21.15.28:0/1897778129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47874/0 (socket says 172.21.15.28:47874) 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 -- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 msgr2=0x5646f4a0e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a0e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 -- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 msgr2=0x5646f4a0e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 0x5646f4a0e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.161+0000 7fb36b400640 1 -- 172.21.15.28:0/1897778129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c06c8b0 con 0x5646f49f1900 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb36b400640 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb364030000 tx=0x7fb364004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d20b4b3e7ee20a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364048680 con 0x5646f49f1900 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5646f4971cd0 con 0x5646f49f1900 2024-09-13T17:09:17.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb364043070 con 0x5646f49f1900 2024-09-13T17:09:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb364031e20 con 0x5646f49f1900 2024-09-13T17:09:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb374a2a740 10 client.5277 fetch_fsmap learned FSMap version 16 2024-09-13T17:09:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5646f49461b0 con 0x5646f49f1900 2024-09-13T17:09:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5646f4971cd0 con 0x5646f49f1900 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.162+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb3640487e0 con 0x5646f49f1900 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 10 client.5277 fetch_fsmap finished waiting for FSMap version 16 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 10 client.5277 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 20 client.5277 populate_metadata read hostname 'smithi028' 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0bbd0 0x5646f4a4b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 4 client.5277 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5646f463b3c0 con 0x5646f4a0bbd0 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0bbd0 0x5646f4a4b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0bbd0 0x5646f4a4b6f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29e1fc313f60a76c server_cookie=849baed0ff77cefb in_seq=0 out_seq=0 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.163+0000 7fb34be00640 10 client.5277 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb34be00640 1 -- 172.21.15.28:0/1897778129 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5646f463b3c0 con 0x5646f4a0bbd0 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb34be00640 10 client.5277 handle_command_reply: tid=0 2024-09-13T17:09:17.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 client.5277 shutdown 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 msgr2=0x5646f49f1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb364030000 tx=0x7fb364004ab0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 shutdown_connections 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0bbd0 0x5646f4a4b6f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49f1900 0x5646f49f1d80 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f4a0cb80 0x5646f4a0e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1897778129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49e56d0 0x5646f4a0e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb34be00640 0 client.5277 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.164+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 >> 172.21.15.28:0/1897778129 conn(0x5646f49e2e90 msgr2=0x5646f4a2dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.165+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 shutdown_connections 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.165+0000 7fb374a2a740 1 -- 172.21.15.28:0/1897778129 wait complete. 2024-09-13T17:09:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.165+0000 7fb374a2a740 20 client.5277 trim_cache size 0 max 16384 2024-09-13T17:09:17.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.180+0000 7fb374a2a740 1 Processor -- start 2024-09-13T17:09:17.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 -- start start 2024-09-13T17:09:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49e56d0 0x5646f4a39400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 0x5646f49f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a06d20 con 0x5646f4a0cb80 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a2c6b0 con 0x5646f49f1900 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.181+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a27a00 con 0x5646f49e56d0 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47888/0 (socket says 172.21.15.28:47888) 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 -- 172.21.15.28:0/657288454 learned_addr learned my addr 172.21.15.28:0/657288454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36aa00640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49e56d0 0x5646f4a39400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36a000640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 0x5646f49f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 -- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49e56d0 msgr2=0x5646f4a39400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49e56d0 0x5646f4a39400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 -- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 msgr2=0x5646f49f1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 0x5646f49f1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 -- 172.21.15.28:0/657288454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5646f497b170 con 0x5646f4a0cb80 2024-09-13T17:09:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36a000640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 0x5646f49f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:09:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.182+0000 7fb36b400640 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb364031b70 tx=0x7fb36403f070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=180017457befa4dc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb357e00640 1 -- 172.21.15.28:0/657288454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36403dcb0 con 0x5646f4a0cb80 2024-09-13T17:09:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb357e00640 1 -- 172.21.15.28:0/657288454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb364048e90 con 0x5646f4a0cb80 2024-09-13T17:09:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb357e00640 1 -- 172.21.15.28:0/657288454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364048680 con 0x5646f4a0cb80 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 -- 172.21.15.28:0/657288454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 msgr2=0x5646f4a29400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb364031b70 tx=0x7fb36403f070 comp rx=0 tx=0).stop 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 -- 172.21.15.28:0/657288454 shutdown_connections 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f4a0cb80 0x5646f4a29400 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f49f1900 0x5646f49f1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 --2- 172.21.15.28:0/657288454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49e56d0 0x5646f4a39400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 -- 172.21.15.28:0/657288454 >> 172.21.15.28:0/657288454 conn(0x5646f49e2e90 msgr2=0x5646f49db300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 -- 172.21.15.28:0/657288454 shutdown_connections 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.183+0000 7fb374a2a740 1 -- 172.21.15.28:0/657288454 wait complete. 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.184+0000 7fb374a2a740 1 -- start start 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.184+0000 7fb374a2a740 1 Processor -- start 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49f1900 0x5646f4a266b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 0x5646f4a2fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a98620 con 0x5646f49e56d0 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a306e0 con 0x5646f4a0cb80 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5646f4a4fa30 con 0x5646f49f1900 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb374a2a740 10 client.0 _ll_register_callbacks cb 0x7fb364009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb36aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47904/0 (socket says 172.21.15.28:47904) 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.185+0000 7fb36aa00640 1 -- 172.21.15.28:0/1177969163 learned_addr learned my addr 172.21.15.28:0/1177969163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36a000640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49f1900 0x5646f4a266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36b400640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 0x5646f4a2fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 -- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49f1900 msgr2=0x5646f4a266b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49f1900 0x5646f4a266b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 -- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 msgr2=0x5646f4a2fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 0x5646f4a2fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 -- 172.21.15.28:0/1177969163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5646f49461b0 con 0x5646f49e56d0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36b400640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 0x5646f4a2fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb36aa00640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb35c001630 tx=0x7fb35c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4754fd991deea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb34be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c014e10 con 0x5646f49e56d0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb35c01c040 con 0x5646f49e56d0 2024-09-13T17:09:17.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.186+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb35c014be0 con 0x5646f49e56d0 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb35c0209a0 con 0x5646f49e56d0 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 10 client.5283 fetch_fsmap learned FSMap version 16 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5646f48e8470 con 0x5646f49e56d0 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb35c014be0 con 0x5646f49e56d0 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb35c012070 con 0x5646f49e56d0 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 10 client.5283 fetch_fsmap finished waiting for FSMap version 16 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 10 client.5283 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 20 client.5283 populate_metadata read hostname 'smithi028' 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0f160 0x5646f4a0f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 4 client.5283 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5646f464c6c0 con 0x5646f4a0f160 2024-09-13T17:09:17.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.187+0000 7fb36a000640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0f160 0x5646f4a0f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb36a000640 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0f160 0x5646f4a0f540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dcbb9747ccbf20b server_cookie=8f8884e1d9786e1a in_seq=0 out_seq=0 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb34be00640 10 client.5283 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb34be00640 1 -- 172.21.15.28:0/1177969163 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5646f464c6c0 con 0x5646f4a0f160 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb34be00640 10 client.5283 handle_command_reply: tid=0 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 client.5283 shutdown 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 msgr2=0x5646f4a980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb35c001630 tx=0x7fb35c001120 comp rx=0 tx=0).stop 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 shutdown_connections 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5646f4a0f160 0x5646f4a0f540 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5646f4a0cb80 0x5646f4a2fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5646f49f1900 0x5646f4a266b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 --2- 172.21.15.28:0/1177969163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5646f49e56d0 0x5646f4a980e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb34be00640 0 client.5283 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:09:17.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.188+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 >> 172.21.15.28:0/1177969163 conn(0x5646f49e2e90 msgr2=0x5646f4a29f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 shutdown_connections 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb374a2a740 1 -- 172.21.15.28:0/1177969163 wait complete. 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb374a2a740 20 client.5283 trim_cache size 0 max 16384 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5121, 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:09:17.195 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.0959396809999999, 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 16, 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 34, 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 34, 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 13.363352265 2024-09-13T17:09:17.196 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 msgr2=0x7fb34c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 0x7fb34c06d0c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb35c000f60 tx=0x7fb35c009040 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 msgr2=0x7fb36c1e5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb35800e960 tx=0x7fb35800ee20 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 shutdown_connections 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb34c06ac00 0x7fb34c06d0c0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb36c15ab60 0x7fb36c154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb36c15a780 0x7fb36c1e5000 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 --2- 172.21.15.28:0/1301075530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb36c071490 0x7fb36c0ad550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 >> 172.21.15.28:0/1301075530 conn(0x7fb36c0b0270 msgr2=0x7fb36c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 shutdown_connections 2024-09-13T17:09:17.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.189+0000 7fb371c00640 1 -- 172.21.15.28:0/1301075530 wait complete. 2024-09-13T17:09:17.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:09:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.361+0000 7fcba3200640 1 Processor -- start 2024-09-13T17:09:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.361+0000 7fcba3200640 1 -- start start 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcba40b0b50 0x7fcba40b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba40c3850 con 0x7fcba4159440 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcba415a930 con 0x7fcba4159820 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcba4159d20 con 0x7fcba40b0b50 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcba4159440 0x7fcba40b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43134/0 (socket says 172.21.15.28:43134) 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcba1e00640 1 -- 172.21.15.28:0/2669245836 learned_addr learned my addr 172.21.15.28:0/2669245836 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.362+0000 7fcb99400640 1 --2- 172.21.15.28:0/2669245836 >> v2:172.21.15.28:3300/0 conn(0x7fcba40b0b50 0x7fcba40b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.363+0000 7fcb99400640 1 -- 172.21.15.28:0/2669245836 >> v1:172.21.15.189:6789/0 conn(0x7fcba4159820 legacy=0x7fcba415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.363+0000 7fcb99400640 1 -- 172.21.15.28:0/2669245836 >> v1:172.21.15.28:6789/0 conn(0x7fcba4159440 legacy=0x7fcba40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.363+0000 7fcb99400640 1 -- 172.21.15.28:0/2669245836 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba415a000 con 0x7fcba40b0b50 2024-09-13T17:09:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcb99400640 1 --2- 172.21.15.28:0/2669245836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba40b0b50 0x7fcba40b22e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0097a0 tx=0x7fcb8c02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4be8cd92152fc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcba1400640 1 -- 172.21.15.28:0/2669245836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c032640 con 0x7fcba40b0b50 2024-09-13T17:09:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcba1400640 1 -- 172.21.15.28:0/2669245836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c0327a0 con 0x7fcba40b0b50 2024-09-13T17:09:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcba1400640 1 -- 172.21.15.28:0/2669245836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c032b30 con 0x7fcba40b0b50 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcba3200640 1 -- 172.21.15.28:0/2669245836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba40b0b50 msgr2=0x7fcba40b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.364+0000 7fcba3200640 1 --2- 172.21.15.28:0/2669245836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba40b0b50 0x7fcba40b22e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c0097a0 tx=0x7fcb8c02a2c0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.365+0000 7fcba3200640 1 -- 172.21.15.28:0/2669245836 shutdown_connections 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.365+0000 7fcba3200640 1 --2- 172.21.15.28:0/2669245836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba40b0b50 0x7fcba40b22e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.365+0000 7fcba3200640 1 -- 172.21.15.28:0/2669245836 >> 172.21.15.28:0/2669245836 conn(0x7fcba4072e40 msgr2=0x7fcba4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.365+0000 7fcba3200640 1 -- 172.21.15.28:0/2669245836 shutdown_connections 2024-09-13T17:09:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.365+0000 7fcba3200640 1 -- 172.21.15.28:0/2669245836 wait complete. 2024-09-13T17:09:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.366+0000 7fcba3200640 1 Processor -- start 2024-09-13T17:09:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.366+0000 7fcba3200640 1 -- start start 2024-09-13T17:09:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcba40b0b50 0x7fcba41e30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcba4159440 0x7fcba41e4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41e4860 con 0x7fcba4159820 2024-09-13T17:09:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41e5310 con 0x7fcba40b0b50 2024-09-13T17:09:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.367+0000 7fcba3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba41e4a70 con 0x7fcba4159440 2024-09-13T17:09:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47922/0 (socket says 172.21.15.28:47922) 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 -- 172.21.15.28:0/3211304763 learned_addr learned my addr 172.21.15.28:0/3211304763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba1e00640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcba40b0b50 0x7fcba41e30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcb99400640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcba4159440 0x7fcba41e4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 -- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcba4159440 msgr2=0x7fcba41e4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcba4159440 0x7fcba41e4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 -- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcba40b0b50 msgr2=0x7fcba41e30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcba40b0b50 0x7fcba41e30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 -- 172.21.15.28:0/3211304763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c009400 con 0x7fcba4159820 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.368+0000 7fcba2800640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00c730 tx=0x7fcb9c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.369+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c003b70 con 0x7fcba4159820 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.369+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9c003cd0 con 0x7fcba4159820 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.369+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9c005310 con 0x7fcba4159820 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.369+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba41e4d50 con 0x7fcba4159820 2024-09-13T17:09:17.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.369+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba415bf40 con 0x7fcba4159820 2024-09-13T17:09:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.370+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba415c130 con 0x7fcba4159820 2024-09-13T17:09:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.371+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcb9c005470 con 0x7fcba4159820 2024-09-13T17:09:17.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.372+0000 7fcb9aa00640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 0x7fcb8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.372+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb9c088ef0 con 0x7fcba4159820 2024-09-13T17:09:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.373+0000 7fcba1e00640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 0x7fcb8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.373+0000 7fcba1e00640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 0x7fcb8006d0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcb94005880 tx=0x7fcb94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.376+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcb9c054170 con 0x7fcba4159820 2024-09-13T17:09:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.591+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcba40b22e0 con 0x7fcba4159820 2024-09-13T17:09:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.591+0000 7fcb9aa00640 1 -- 172.21.15.28:0/3211304763 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcb9c057790 con 0x7fcba4159820 2024-09-13T17:09:17.594 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:09:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 msgr2=0x7fcb8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 0x7fcb8006d0c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fcb94005880 tx=0x7fcb94009040 comp rx=0 tx=0).stop 2024-09-13T17:09:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 msgr2=0x7fcba41e58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c00c730 tx=0x7fcb9c00cbf0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 shutdown_connections 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb8006ac00 0x7fcb8006d0c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcba4159820 0x7fcba41e58b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcba4159440 0x7fcba41e4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 --2- 172.21.15.28:0/3211304763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcba40b0b50 0x7fcba41e30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 >> 172.21.15.28:0/3211304763 conn(0x7fcba4072e40 msgr2=0x7fcba41560c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 shutdown_connections 2024-09-13T17:09:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:17.594+0000 7fcba3200640 1 -- 172.21.15.28:0/3211304763 wait complete. 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:09:17.612 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:09:17.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.613 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.649 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42624sec preferred_lft 42624sec 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:09:17.650 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:09:17.651 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:09:17.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.652 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:09:17.743 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:09:17.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.746 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:09:17.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:09:17.783 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:09:17.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.929 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:09:17.964 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:09:17.965 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:09:17.965 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:09:17.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:17.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:09:18.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:09:18.106 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:09:18.141 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:09:18.141 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:09:18.142 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:09:18.217 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:09:18.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:18.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:09:18.344 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:09:18.345 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:09:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:09:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:09:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:09:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:09:18.371 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:09:18.373 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:09:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:09:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:09:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:09:18.373 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:09:18.376 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:09:18.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:18.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:09:18.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:18.433 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:09:18.479 INFO:teuthology.orchestra.run.smithi028.stdout:client5295 2024-09-13T17:09:18.480 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:09:18.480 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:09:18.516 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2024-09-13T17:09:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.716+0000 7fe252c00640 1 Processor -- start 2024-09-13T17:09:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.717+0000 7fe252c00640 1 -- start start 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe252c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe24c0b1c80 0x7fe24c1652f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe252c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c06c950 con 0x7fe24c15a7a0 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe252c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24c0b2150 con 0x7fe24c0ceba0 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe252c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24c0b2430 con 0x7fe24c0b1c80 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe24be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe24c15a7a0 0x7fe24c0ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43148/0 (socket says 172.21.15.28:43148) 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe24be00640 1 -- 172.21.15.28:0/2348500386 learned_addr learned my addr 172.21.15.28:0/2348500386 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:09:18.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe250c00640 1 --2- 172.21.15.28:0/2348500386 >> v2:172.21.15.189:3300/0 conn(0x7fe24c0b1c80 0x7fe24c1652f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.718+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992463924 0 0) 0x7fe23c001110 con 0x7fe24c15a7a0 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.719+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe230003520 con 0x7fe24c15a7a0 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.719+0000 7fe250c00640 1 -- 172.21.15.28:0/2348500386 >> v1:172.21.15.189:6789/0 conn(0x7fe24c0ceba0 legacy=0x7fe24c0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.719+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742889345 0 0) 0x7fe24c0b2150 con 0x7fe24c0ceba0 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.719+0000 7fe250c00640 1 -- 172.21.15.28:0/2348500386 >> v1:172.21.15.28:6789/0 conn(0x7fe24c15a7a0 legacy=0x7fe24c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:18.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.719+0000 7fe250c00640 1 -- 172.21.15.28:0/2348500386 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c0cc960 con 0x7fe24c0b1c80 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.720+0000 7fe250c00640 1 --2- 172.21.15.28:0/2348500386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c1652f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe244009d90 tx=0x7fe24402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9292734bcf5799dc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.720+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244007740 con 0x7fe24c0b1c80 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.720+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24402d050 con 0x7fe24c0b1c80 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.720+0000 7fe24b400640 1 -- 172.21.15.28:0/2348500386 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe244007ac0 con 0x7fe24c0b1c80 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 -- 172.21.15.28:0/2348500386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 msgr2=0x7fe24c1652f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 --2- 172.21.15.28:0/2348500386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c1652f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe244009d90 tx=0x7fe24402a660 comp rx=0 tx=0).stop 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 -- 172.21.15.28:0/2348500386 shutdown_connections 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 --2- 172.21.15.28:0/2348500386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c1652f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:18.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 -- 172.21.15.28:0/2348500386 >> 172.21.15.28:0/2348500386 conn(0x7fe24c069e80 msgr2=0x7fe24c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 -- 172.21.15.28:0/2348500386 shutdown_connections 2024-09-13T17:09:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.721+0000 7fe252c00640 1 -- 172.21.15.28:0/2348500386 wait complete. 2024-09-13T17:09:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.722+0000 7fe252c00640 1 Processor -- start 2024-09-13T17:09:18.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 -- start start 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c06f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24c15a7a0 0x7fe24c158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c158ab0 con 0x7fe24c0ceba0 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c06fa60 con 0x7fe24c0b1c80 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.723+0000 7fe252c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24c06fd40 con 0x7fe24c15a7a0 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47924/0 (socket says 172.21.15.28:47924) 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 -- 172.21.15.28:0/3327424234 learned_addr learned my addr 172.21.15.28:0/3327424234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe24be00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c06f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:18.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe250c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24c15a7a0 0x7fe24c158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 -- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24c15a7a0 msgr2=0x7fe24c158570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24c15a7a0 0x7fe24c158570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 -- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 msgr2=0x7fe24c06f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c06f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:18.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 -- 172.21.15.28:0/3327424234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244009a70 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.724+0000 7fe243e00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c410 tx=0x7fe23400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.725+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2340035f0 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.725+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe234003770 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.725+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe234003b20 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.725+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c070020 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.725+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe24c0705e0 con 0x7fe24c0ceba0 2024-09-13T17:09:18.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.726+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe24c15aca0 con 0x7fe24c0ceba0 2024-09-13T17:09:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.727+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe234003c80 con 0x7fe24c0ceba0 2024-09-13T17:09:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.728+0000 7fe248c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 0x7fe22806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:09:18.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.728+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe234088870 con 0x7fe24c0ceba0 2024-09-13T17:09:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.728+0000 7fe24be00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 0x7fe22806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:09:18.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.729+0000 7fe24be00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 0x7fe22806d0f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe23c005880 tx=0x7fe23c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:09:18.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.732+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe234053bb0 con 0x7fe24c0ceba0 2024-09-13T17:09:18.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:18.948+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe24c15e020 con 0x7fe24c0ceba0 2024-09-13T17:09:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.124+0000 7fe248c00640 1 -- 172.21.15.28:0/3327424234 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v17) v1 ==== 167+0+0 (secure 0 0 0) 0x7fe2340170d0 con 0x7fe24c0ceba0 2024-09-13T17:09:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:enabled new snapshots 2024-09-13T17:09:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 msgr2=0x7fe22806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 0x7fe22806d0f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe23c005880 tx=0x7fe23c009040 comp rx=0 tx=0).stop 2024-09-13T17:09:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 msgr2=0x7fe24c152290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:09:19.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe23400c410 tx=0x7fe23400c8d0 comp rx=0 tx=0).stop 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 shutdown_connections 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22806ac30 0x7fe22806d0f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24c15a7a0 0x7fe24c158570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24c0ceba0 0x7fe24c152290 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 --2- 172.21.15.28:0/3327424234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24c0b1c80 0x7fe24c06f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:09:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 >> 172.21.15.28:0/3327424234 conn(0x7fe24c069e80 msgr2=0x7fe24c1547f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:09:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 shutdown_connections 2024-09-13T17:09:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:09:19.127+0000 7fe252c00640 1 -- 172.21.15.28:0/3327424234 wait complete. 2024-09-13T17:09:19.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:19.146 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir stray_eval_dir) 2024-09-13T17:09:19.186 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:09:19.186 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:09:19.262 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:09:19.262 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:09:19.262 DEBUG:teuthology.orchestra.run.smithi028:> import uuid 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> n = 2000 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = False 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> if 3 > 0: 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = True 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> path = "/home/ubuntu/cephtest/mnt.0/stray_eval_dir/file" 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> dpath = os.path.dirname(path) 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> fnameprefix = os.path.basename(path) 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> os.makedirs(dpath, exist_ok=True) 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:09:19.263 DEBUG:teuthology.orchestra.run.smithi028:> for i in range(n): 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> with open(fpath, '"'"'w'"'"') as f: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> f.write(f"{i}") 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> f.flush() 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(f.fileno()) 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> os.unlink(fpath) 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> if create_hard_links: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> for j in range(3): 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:09:19.264 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:09:19.265 DEBUG:teuthology.orchestra.run.smithi028:> finally: 2024-09-13T17:09:19.265 DEBUG:teuthology.orchestra.run.smithi028:> os.close(dirfd) 2024-09-13T17:09:19.265 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:09:39.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:09:39.031 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:09:50.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:50.225 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-09-13T17:09:50.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:09:50.293 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec find /home/ubuntu/cephtest/mnt.0/stray_eval_dir/ -type f -delete) 2024-09-13T17:10:09.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:10:09.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:10:18.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:10:18.267 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-09-13T17:10:18.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 perf dump 2024-09-13T17:10:18.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.523+0000 7f449a400640 1 Processor -- start 2024-09-13T17:10:18.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.523+0000 7f449a400640 1 -- start start 2024-09-13T17:10:18.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f449a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4494159770 0x7f449406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f449a400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449406f920 con 0x7f449415cd80 2024-09-13T17:10:18.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f449a400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449415b290 con 0x7f449406f3f0 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f449a400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449415b570 con 0x7f4494159770 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f4493400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4494159770 0x7f449406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.524+0000 7f4493400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4494159770 0x7f449406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51676/0 (socket says 172.21.15.28:51676) 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.525+0000 7f4493400640 1 -- 172.21.15.28:0/2656305761 learned_addr learned my addr 172.21.15.28:0/2656305761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.525+0000 7f4493400640 1 -- 172.21.15.28:0/2656305761 >> v1:172.21.15.189:6789/0 conn(0x7f449406f3f0 legacy=0x7f4494159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.525+0000 7f4493400640 1 -- 172.21.15.28:0/2656305761 >> v1:172.21.15.28:6789/0 conn(0x7f449415cd80 legacy=0x7f449415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.525+0000 7f4493400640 1 -- 172.21.15.28:0/2656305761 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449415b850 con 0x7f4494159770 2024-09-13T17:10:18.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.525+0000 7f4492000640 1 -- 172.21.15.28:0/2656305761 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775508103 0 0) 0x7f449406f920 con 0x7f449415cd80 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.526+0000 7f4493400640 1 --2- 172.21.15.28:0/2656305761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f449406fd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f448400bb10 tx=0x7f448402c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bb0f4f2ae809c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.526+0000 7f4492000640 1 -- 172.21.15.28:0/2656305761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484032640 con 0x7f4494159770 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.526+0000 7f4492000640 1 -- 172.21.15.28:0/2656305761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44840327a0 con 0x7f4494159770 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.526+0000 7f4492000640 1 -- 172.21.15.28:0/2656305761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484032b30 con 0x7f4494159770 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 -- 172.21.15.28:0/2656305761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 msgr2=0x7f449406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 --2- 172.21.15.28:0/2656305761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f449406fd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f448400bb10 tx=0x7f448402c940 comp rx=0 tx=0).stop 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 -- 172.21.15.28:0/2656305761 shutdown_connections 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 --2- 172.21.15.28:0/2656305761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f449406fd20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 -- 172.21.15.28:0/2656305761 >> 172.21.15.28:0/2656305761 conn(0x7f44940b0270 msgr2=0x7f44940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 -- 172.21.15.28:0/2656305761 shutdown_connections 2024-09-13T17:10:18.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.527+0000 7f449a400640 1 -- 172.21.15.28:0/2656305761 wait complete. 2024-09-13T17:10:18.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.529+0000 7f449a400640 1 Processor -- start 2024-09-13T17:10:18.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.529+0000 7f449a400640 1 -- start start 2024-09-13T17:10:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 0x7f44941e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f449415cd80 0x7f44940cc890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449414c910 con 0x7f4494159770 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44941e37a0 con 0x7f449406f3f0 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f449a400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44941e3a80 con 0x7f449415cd80 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f4492a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f4492a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51688/0 (socket says 172.21.15.28:51688) 2024-09-13T17:10:18.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f4493400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 0x7f44941e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f4492a00640 1 -- 172.21.15.28:0/165934737 learned_addr learned my addr 172.21.15.28:0/165934737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.530+0000 7f4493e00640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f449415cd80 0x7f44940cc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 -- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f449415cd80 msgr2=0x7f44940cc890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f449415cd80 0x7f44940cc890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 -- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 msgr2=0x7f44941e2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 0x7f44941e2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 -- 172.21.15.28:0/165934737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f448400b770 con 0x7f4494159770 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4493400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 0x7f44941e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f4492a00640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f448000c450 tx=0x7f448000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f448b400640 1 -- 172.21.15.28:0/165934737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44800052d0 con 0x7f4494159770 2024-09-13T17:10:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f448b400640 1 -- 172.21.15.28:0/165934737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4480005430 con 0x7f4494159770 2024-09-13T17:10:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f448b400640 1 -- 172.21.15.28:0/165934737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480012870 con 0x7f4494159770 2024-09-13T17:10:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44941e3d60 con 0x7f4494159770 2024-09-13T17:10:18.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.531+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44941e4320 con 0x7f4494159770 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.533+0000 7f448b400640 1 -- 172.21.15.28:0/165934737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f44800129d0 con 0x7f4494159770 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.534+0000 7f448b400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 0x7f447406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.534+0000 7f448b400640 1 -- 172.21.15.28:0/165934737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4480088da0 con 0x7f4494159770 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.534+0000 7f4493400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 0x7f447406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.535+0000 7f449d20d740 1 Processor -- start 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.535+0000 7f449d20d740 1 -- start start 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.535+0000 7f4493400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 0x7f447406d0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4484037000 tx=0x7f448403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a782e1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782deb40 0x562a7832e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a781ec000 con 0x562a78308830 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a78249d40 con 0x562a782e1380 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a782474d0 con 0x562a782deb40 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51692/0 (socket says 172.21.15.28:51692) 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 -- 172.21.15.28:0/3340677818 learned_addr learned my addr 172.21.15.28:0/3340677818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493400640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a782e1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4492a00640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782deb40 0x562a7832e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 -- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782deb40 msgr2=0x562a7832e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782deb40 0x562a7832e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 -- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 msgr2=0x562a782e1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a782e1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493e00640 1 -- 172.21.15.28:0/3340677818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a781fe3a0 con 0x562a78308830 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.536+0000 7f4493400640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a782e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f4493e00640 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f448c001f00 tx=0x7f448c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8a547f7f3b7252 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f447fe00640 1 -- 172.21.15.28:0/3340677818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0019d0 con 0x562a78308830 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f447fe00640 1 -- 172.21.15.28:0/3340677818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c00fe50 con 0x562a78308830 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f447fe00640 1 -- 172.21.15.28:0/3340677818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c003b70 con 0x562a78308830 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f449d20d740 1 -- 172.21.15.28:0/3340677818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 msgr2=0x562a7832ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f449d20d740 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f448c001f00 tx=0x7f448c0015b0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f449d20d740 1 -- 172.21.15.28:0/3340677818 shutdown_connections 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.537+0000 7f449d20d740 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832ed90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782deb40 0x562a7832e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 --2- 172.21.15.28:0/3340677818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a782e1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 -- 172.21.15.28:0/3340677818 >> 172.21.15.28:0/3340677818 conn(0x562a782ddab0 msgr2=0x562a78325f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 -- 172.21.15.28:0/3340677818 shutdown_connections 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 -- 172.21.15.28:0/3340677818 wait complete. 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 -- start start 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.538+0000 7f449d20d740 1 Processor -- start 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a78306b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 0x562a782dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a78242dc0 con 0x562a78308830 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a782013e0 con 0x562a782db7f0 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a78201080 con 0x562a782e1380 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.539+0000 7f449d20d740 10 client.0 _ll_register_callbacks cb 0x562a7823c430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51704/0 (socket says 172.21.15.28:51704) 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 -- 172.21.15.28:0/1043802980 learned_addr learned my addr 172.21.15.28:0/1043802980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4493400640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a78306b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4493e00640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 0x562a782dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 -- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 msgr2=0x562a78306b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a78306b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 -- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 msgr2=0x562a782dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 0x562a782dbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 -- 172.21.15.28:0/1043802980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448c001b30 con 0x562a78308830 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4493400640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a78306b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4493e00640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 0x562a782dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f4492a00640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4480057950 tx=0x7f448005ff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abbb1d30e4094082 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f447cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4480082060 con 0x562a78308830 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f448005fb00 con 0x562a78308830 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.540+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a7826c600 con 0x562a78308830 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4480012e80 con 0x562a78308830 2024-09-13T17:10:18.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 10 client.5316 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x562a78241440 con 0x562a78308830 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562a7826c600 con 0x562a78308830 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4480019020 con 0x562a78308830 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 10 client.5316 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 10 client.5316 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 20 client.5316 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a7832ac00 0x562a7832afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 4 client.5316 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a78234c90 con 0x562a7832ac00 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.541+0000 7f4493400640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a7832ac00 0x562a7832afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.542+0000 7f4493400640 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a7832ac00 0x562a7832afe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b42a7b5551672d36 server_cookie=5a3b19fb5362985a in_seq=0 out_seq=0 2024-09-13T17:10:18.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.542+0000 7f447cc00640 10 client.5316 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.542+0000 7f447cc00640 1 -- 172.21.15.28:0/1043802980 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562a78234c90 con 0x562a7832ac00 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.542+0000 7f447cc00640 10 client.5316 handle_command_reply: tid=0 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.542+0000 7f449d20d740 1 client.5316 shutdown 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 msgr2=0x562a7832c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4480057950 tx=0x7f448005ff90 comp rx=0 tx=0).stop 2024-09-13T17:10:18.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 shutdown_connections 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a7832ac00 0x562a7832afe0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782db7f0 0x562a782dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a78308830 0x562a7832c470 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 --2- 172.21.15.28:0/1043802980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a78306b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f447cc00640 0 client.5316 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 >> 172.21.15.28:0/1043802980 conn(0x562a782deb40 msgr2=0x562a78329420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 shutdown_connections 2024-09-13T17:10:18.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 1 -- 172.21.15.28:0/1043802980 wait complete. 2024-09-13T17:10:18.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.543+0000 7f449d20d740 20 client.5316 trim_cache size 0 max 16384 2024-09-13T17:10:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.557+0000 7f449d20d740 1 Processor -- start 2024-09-13T17:10:18.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.557+0000 7f449d20d740 1 -- start start 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a783259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 0x562a78325ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a77faa040 con 0x562a782db7f0 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a7832f6b0 con 0x562a782e1380 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a783236a0 con 0x562a78308830 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f4493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f4493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51712/0 (socket says 172.21.15.28:51712) 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.558+0000 7f4493400640 1 -- 172.21.15.28:0/3220850049 learned_addr learned my addr 172.21.15.28:0/3220850049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493e00640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 0x562a78325ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4492a00640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a783259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 -- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 msgr2=0x562a78325ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 0x562a78325ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 -- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 msgr2=0x562a783259b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a783259b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 -- 172.21.15.28:0/3220850049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a781e35d0 con 0x562a782db7f0 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4492a00640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a783259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493e00640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 0x562a78325ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:18.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4493400640 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f448400b2c0 tx=0x7f4484032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c685d32ddffab4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4488c00640 1 -- 172.21.15.28:0/3220850049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448404c040 con 0x562a782db7f0 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4488c00640 1 -- 172.21.15.28:0/3220850049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4484041040 con 0x562a782db7f0 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.559+0000 7f4488c00640 1 -- 172.21.15.28:0/3220850049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44840478c0 con 0x562a782db7f0 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 -- 172.21.15.28:0/3220850049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 msgr2=0x562a7832ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f448400b2c0 tx=0x7f4484032000 comp rx=0 tx=0).stop 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 -- 172.21.15.28:0/3220850049 shutdown_connections 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a78308830 0x562a78325ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a782e1380 0x562a783259b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 --2- 172.21.15.28:0/3220850049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a7832ef50 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 -- 172.21.15.28:0/3220850049 >> 172.21.15.28:0/3220850049 conn(0x562a782deb40 msgr2=0x562a78334d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 -- 172.21.15.28:0/3220850049 shutdown_connections 2024-09-13T17:10:18.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.560+0000 7f449d20d740 1 -- 172.21.15.28:0/3220850049 wait complete. 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.561+0000 7f449d20d740 1 -- start start 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.561+0000 7f449d20d740 1 Processor -- start 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a783919d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a78308830 0x562a78326a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a782dcb80 con 0x562a782db7f0 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a782dc590 con 0x562a78308830 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562a78397bf0 con 0x562a782e1380 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f449d20d740 10 client.0 _ll_register_callbacks cb 0x7f4480001280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51716/0 (socket says 172.21.15.28:51716) 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 -- 172.21.15.28:0/1610006267 learned_addr learned my addr 172.21.15.28:0/1610006267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493e00640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a78308830 0x562a78326a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4492a00640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a783919d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 -- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 msgr2=0x562a783919d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a783919d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 -- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a78308830 msgr2=0x562a78326a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a78308830 0x562a78326a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.562+0000 7f4493400640 1 -- 172.21.15.28:0/1610006267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449415b850 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f4493400640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4484008010 tx=0x7f4484004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f2b45e14780c9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f447cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4484047ad0 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f448404c070 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f448c00fe50 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4484047c30 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f449d20d740 10 client.5322 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x562a782089b0 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.563+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c00fe50 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4484055020 con 0x562a782db7f0 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 10 client.5322 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 10 client.5322 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 20 client.5322 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:18.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a782dcf40 0x562a78326f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 4 client.5322 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x562a77f364e0 con 0x562a782dcf40 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f4492a00640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a782dcf40 0x562a78326f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f4492a00640 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a782dcf40 0x562a78326f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be1213ba31768b9 server_cookie=112f5b05b344735e in_seq=0 out_seq=0 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.564+0000 7f447cc00640 10 client.5322 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f447cc00640 1 -- 172.21.15.28:0/1610006267 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17913 (crc 0 0 0) 0x562a77f364e0 con 0x562a782dcf40 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f447cc00640 10 client.5322 handle_command_reply: tid=0 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 client.5322 shutdown 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 msgr2=0x562a78343ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4484008010 tx=0x7f4484004730 comp rx=0 tx=0).stop 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 shutdown_connections 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x562a782dcf40 0x562a78326f60 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562a78308830 0x562a78326a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562a782e1380 0x562a783919d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f447cc00640 0 client.5322 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 --2- 172.21.15.28:0/1610006267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a782db7f0 0x562a78343ba0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 >> 172.21.15.28:0/1610006267 conn(0x562a782deb40 msgr2=0x562a78329980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 shutdown_connections 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.565+0000 7f449d20d740 1 -- 172.21.15.28:0/1610006267 wait complete. 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449d20d740 20 client.5322 trim_cache size 0 max 16384 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:10:18.572 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3573, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 3574, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 990322, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 46854749, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.533818770, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.301975805, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.076969764, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.123974037, 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:18.573 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3572, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5.961847433, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001669050 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000170, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000085 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 29690, 2024-09-13T17:10:18.574 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 50466, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 10720429, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 46275369, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 4.904313987, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 2.878970064, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 2.604091929, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.077354139, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 50466, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.189079131, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000043377 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.575 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 70647, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017213982, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000243 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 123728, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 20528 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3758, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 3775, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1228020, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 40078458, 2024-09-13T17:10:18.576 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.423881571, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.178172724, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.080874211, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.134894977, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3774, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.155458719, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000306162 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.577 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 78800 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4521, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 15.942007285, 2024-09-13T17:10:18.578 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003526212 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1920, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.624840296, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000325437 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 25668, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 25668, 2024-09-13T17:10:18.579 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 25668, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 303.963110858, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011842103 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 36, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 18289, 2024-09-13T17:10:18.580 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 8131, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 10158, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 6673, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 157, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 157, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 2, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 47717, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 47147, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-13T17:10:18.581 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 169, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-13T17:10:18.582 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 1034, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 4038, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-13T17:10:18.583 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 4203, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 2000, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 27, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 14028 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 1844, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-13T17:10:18.584 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 4142, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 1845, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 2249, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-13T17:10:18.585 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 29294, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 21327, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 18973, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 10321, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 2354, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 45, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 32, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 27, 2024-09-13T17:10:18.586 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 18, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 5, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 62566861, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 99516190, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4192, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 94.230827065, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.022478727 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-13T17:10:18.587 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 2014, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 2014, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 0, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 13, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 13, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 18289, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 19966, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 1677, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 157, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 3604, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 3447, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 108136, 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 207200 2024-09-13T17:10:18.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 29153, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 25668, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 4, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2000, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.428036337, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000714018 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-13T17:10:18.589 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 24, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.005078376, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000211599 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6000, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.911972892, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001151995 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.590 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7382, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.471541636, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000876665 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000053613, 2024-09-13T17:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000053613 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002543020, 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002543020 2024-09-13T17:10:18.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-13T17:10:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.594 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006798440, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006798440 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-09-13T17:10:18.595 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.552415877, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.069051984 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2249, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 271.609319740, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.120768928 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.004082261, 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.004082261 2024-09-13T17:10:18.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000378026, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000378026 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.598 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-13T17:10:18.599 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8000, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.970890640, 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002121361 2024-09-13T17:10:18.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-09-13T17:10:18.601 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 18076, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 18076, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 55, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-13T17:10:18.602 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-13T17:10:18.603 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-13T17:10:18.604 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 18085510, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 6572, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 440, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 5, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-13T17:10:18.605 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 5312, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 75, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 19837814, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 852615, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-13T17:10:18.606 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 9180, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 104556684, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 37, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 9180, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9180, 2024-09-13T17:10:18.607 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 102.788116479, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011196962 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9180, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 15132 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 9180, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 36, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 9144, 2024-09-13T17:10:18.608 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 23, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 2932, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 36, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 4228, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 35, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 1689, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-13T17:10:18.609 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 203, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 5864, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-13T17:10:18.610 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-13T17:10:18.611 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 34, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 1, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 8, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 8, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 93, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 29 2024-09-13T17:10:18.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 32, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-13T17:10:18.613 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 32, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 1845, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 1845, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 37017, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 9602110, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 37017, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:18.614 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 37017, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 9602110, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:10:18.615 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 9180, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 104756541, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 9180, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 104756541, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-13T17:10:18.616 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 9180, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 9180, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 9180, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 9180, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.617 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 1845, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 199857, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 1845, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:18.618 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 36, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 199857, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x555b9c4640a0": { 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 93528, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 29294, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 95321886, 2024-09-13T17:10:18.619 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 29294, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4192, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 95228358, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:18.620 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 msgr2=0x7f447406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 0x7f447406d0c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4484037000 tx=0x7f448403d040 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 msgr2=0x7f44940cc350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f448000c450 tx=0x7f448000c910 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 shutdown_connections 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f447406ac00 0x7f447406d0c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f449415cd80 0x7f44940cc890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4494159770 0x7f44940cc350 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 --2- 172.21.15.28:0/165934737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f449406f3f0 0x7f44941e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 >> 172.21.15.28:0/165934737 conn(0x7f44940b0270 msgr2=0x7f449415a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 shutdown_connections 2024-09-13T17:10:18.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.566+0000 7f449a400640 1 -- 172.21.15.28:0/165934737 wait complete. 2024-09-13T17:10:18.623 INFO:tasks.cephfs.test_scrub_checks:num of strays: 1844 2024-09-13T17:10:18.623 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 scrub start / recursive,scrub_mdsdir 2024-09-13T17:10:18.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.781+0000 7f5c8a800640 1 Processor -- start 2024-09-13T17:10:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.782+0000 7f5c8a800640 1 -- start start 2024-09-13T17:10:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c84159590 0x7f5c84070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5c84159970 0x7f5c840b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c840b2630 0x7f5c8406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c840b2ae0 con 0x7f5c84159970 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8415a4c0 con 0x7f5c84159590 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c8a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84071580 con 0x7f5c840b2630 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5c84159970 0x7f5c840b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c7aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5c84159970 0x7f5c840b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51722/0 (socket says 172.21.15.28:51722) 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c7aa00640 1 -- 172.21.15.28:0/2635214393 learned_addr learned my addr 172.21.15.28:0/2635214393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c83400640 1 --2- 172.21.15.28:0/2635214393 >> v2:172.21.15.189:3300/0 conn(0x7f5c84159590 0x7f5c84070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c83e00640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c840b2630 0x7f5c8406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c7aa00640 1 -- 172.21.15.28:0/2635214393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c840b2630 msgr2=0x7f5c8406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.783+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c840b2630 0x7f5c8406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.784+0000 7f5c7aa00640 1 -- 172.21.15.28:0/2635214393 >> v2:172.21.15.189:3300/0 conn(0x7f5c84159590 msgr2=0x7f5c84070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.784+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/2635214393 >> v2:172.21.15.189:3300/0 conn(0x7f5c84159590 0x7f5c84070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.784+0000 7f5c7aa00640 1 -- 172.21.15.28:0/2635214393 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c84159e70 con 0x7f5c84159970 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84159970 0x7f5c840b20f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5c8406d320 tx=0x7f5c6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c62ec06338759a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c82a00640 1 -- 172.21.15.28:0/2635214393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c032440 con 0x7f5c84159970 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c82a00640 1 -- 172.21.15.28:0/2635214393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c0325a0 con 0x7f5c84159970 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c82a00640 1 -- 172.21.15.28:0/2635214393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c032930 con 0x7f5c84159970 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 -- 172.21.15.28:0/2635214393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84159970 msgr2=0x7f5c840b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84159970 0x7f5c840b20f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5c8406d320 tx=0x7f5c6c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 -- 172.21.15.28:0/2635214393 shutdown_connections 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c84159970 0x7f5c840b20f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 --2- 172.21.15.28:0/2635214393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c840b2630 0x7f5c8406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 --2- 172.21.15.28:0/2635214393 >> v2:172.21.15.189:3300/0 conn(0x7f5c84159590 0x7f5c84070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.785+0000 7f5c8a800640 1 -- 172.21.15.28:0/2635214393 >> 172.21.15.28:0/2635214393 conn(0x7f5c84069e80 msgr2=0x7f5c8406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.786+0000 7f5c8a800640 1 -- 172.21.15.28:0/2635214393 shutdown_connections 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.786+0000 7f5c8a800640 1 -- 172.21.15.28:0/2635214393 wait complete. 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.787+0000 7f5c8a800640 1 Processor -- start 2024-09-13T17:10:18.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.787+0000 7f5c8a800640 1 -- start start 2024-09-13T17:10:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c84159590 0x7f5c8414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c84159970 0x7f5c8413feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c84150680 con 0x7f5c840b2630 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8414d460 con 0x7f5c84159590 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c8a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8414fa00 con 0x7f5c84159970 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51732/0 (socket says 172.21.15.28:51732) 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.788+0000 7f5c83400640 1 -- 172.21.15.28:0/4122525857 learned_addr learned my addr 172.21.15.28:0/4122525857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83e00640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c84159970 0x7f5c8413feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c84159590 0x7f5c8414cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 -- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c84159970 msgr2=0x7f5c8413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c84159970 0x7f5c8413feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 -- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c84159590 msgr2=0x7f5c8414cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c84159590 0x7f5c8414cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 -- 172.21.15.28:0/4122525857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0094d0 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c83400640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400e9b0 tx=0x7f5c7400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c7be00640 1 -- 172.21.15.28:0/4122525857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74009670 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c7be00640 1 -- 172.21.15.28:0/4122525857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7400c9a0 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8414fce0 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.789+0000 7f5c7be00640 1 -- 172.21.15.28:0/4122525857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7400cd50 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.790+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c841502a0 con 0x7f5c840b2630 2024-09-13T17:10:18.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.791+0000 7f5c7be00640 1 -- 172.21.15.28:0/4122525857 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5c74013470 con 0x7f5c840b2630 2024-09-13T17:10:18.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.792+0000 7f5c7be00640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 0x7f5c6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.792+0000 7f5c7be00640 1 -- 172.21.15.28:0/4122525857 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c74089550 con 0x7f5c840b2630 2024-09-13T17:10:18.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.793+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 0x7f5c6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.793+0000 7f5c8d7eb740 1 Processor -- start 2024-09-13T17:10:18.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.793+0000 7f5c8d7eb740 1 -- start start 2024-09-13T17:10:18.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.793+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 0x7f5c6006d0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c034000 tx=0x7f5c6c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f7e4610 0x55bf4f7e8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f82cb30 0x55bf4f812c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f6f3c20 con 0x55bf4f7e9240 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f751cd0 con 0x55bf4f82cb30 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f74f460 con 0x55bf4f7e4610 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51738/0 (socket says 172.21.15.28:51738) 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c7aa00640 1 -- 172.21.15.28:0/820048700 learned_addr learned my addr 172.21.15.28:0/820048700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c83e00640 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f82cb30 0x55bf4f812c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.794+0000 7f5c83400640 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f7e4610 0x55bf4f7e8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 -- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f7e4610 msgr2=0x55bf4f7e8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f7e4610 0x55bf4f7e8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 -- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f82cb30 msgr2=0x55bf4f812c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f82cb30 0x55bf4f812c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 -- 172.21.15.28:0/820048700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf4f6ec9b0 con 0x55bf4f7e9240 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0090b0 tx=0x7f5c6c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb6d44f397f7642 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c73e00640 1 -- 172.21.15.28:0/820048700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c051660 con 0x55bf4f7e9240 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c73e00640 1 -- 172.21.15.28:0/820048700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c0517c0 con 0x55bf4f7e9240 2024-09-13T17:10:18.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.795+0000 7f5c73e00640 1 -- 172.21.15.28:0/820048700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c051b50 con 0x55bf4f7e9240 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/820048700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 msgr2=0x55bf4f812750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0090b0 tx=0x7f5c6c02c000 comp rx=0 tx=0).stop 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/820048700 shutdown_connections 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f82cb30 0x55bf4f812c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f812750 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/820048700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f7e4610 0x55bf4f7e8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/820048700 >> 172.21.15.28:0/820048700 conn(0x55bf4f7e52a0 msgr2=0x55bf4f82d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/820048700 shutdown_connections 2024-09-13T17:10:18.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.796+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/820048700 wait complete. 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.797+0000 7f5c8d7eb740 1 -- start start 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.797+0000 7f5c8d7eb740 1 Processor -- start 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 0x55bf4f81b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f808c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f74ad50 con 0x55bf4f8127f0 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f709180 con 0x55bf4f7e9240 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f708e20 con 0x55bf4f82cb30 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c8d7eb740 10 client.0 _ll_register_callbacks cb 0x55bf4f73acd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51754/0 (socket says 172.21.15.28:51754) 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.798+0000 7f5c83e00640 1 -- 172.21.15.28:0/1386171086 learned_addr learned my addr 172.21.15.28:0/1386171086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f808c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83400640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 0x55bf4f81b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 -- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 msgr2=0x55bf4f808c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f808c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 -- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 msgr2=0x55bf4f81b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 0x55bf4f81b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 -- 172.21.15.28:0/1386171086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c84159e70 con 0x55bf4f8127f0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83400640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 0x55bf4f81b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c83e00640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c014ca0 tx=0x7f5c7c0125f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10cbba2a1b27a105 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7c010040 con 0x55bf4f8127f0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf4f774ef0 con 0x55bf4f8127f0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c7c014e50 con 0x55bf4f8127f0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c7c012e90 con 0x55bf4f8127f0 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c8d7eb740 10 client.5334 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.799+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55bf4f7493d0 con 0x55bf4f8127f0 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bf4f774ef0 con 0x55bf4f8127f0 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5c7c003b90 con 0x55bf4f8127f0 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 10 client.5334 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 10 client.5334 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 20 client.5334 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f81d300 0x55bf4f81dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 4 client.5334 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf4f73da30 con 0x55bf4f81d300 2024-09-13T17:10:18.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.800+0000 7f5c83400640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f81d300 0x55bf4f81dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.801+0000 7f5c83400640 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f81d300 0x55bf4f81dc60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfe4ed339d90ff0f server_cookie=80925d3f100880a3 in_seq=0 out_seq=0 2024-09-13T17:10:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.801+0000 7f5c70c00640 10 client.5334 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.801+0000 7f5c70c00640 1 -- 172.21.15.28:0/1386171086 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55bf4f73da30 con 0x55bf4f81d300 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.801+0000 7f5c70c00640 10 client.5334 handle_command_reply: tid=0 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.801+0000 7f5c8d7eb740 1 client.5334 shutdown 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 msgr2=0x55bf4f808270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c014ca0 tx=0x7f5c7c0125f0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 shutdown_connections 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f81d300 0x55bf4f81dc60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f8127f0 0x55bf4f808270 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f808c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1386171086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f7e9240 0x55bf4f81b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 >> 172.21.15.28:0/1386171086 conn(0x55bf4f7e4610 msgr2=0x55bf4f82a8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c70c00640 0 client.5334 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 shutdown_connections 2024-09-13T17:10:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1386171086 wait complete. 2024-09-13T17:10:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.802+0000 7f5c8d7eb740 20 client.5334 trim_cache size 0 max 16384 2024-09-13T17:10:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.817+0000 7f5c8d7eb740 1 Processor -- start 2024-09-13T17:10:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.817+0000 7f5c8d7eb740 1 -- start start 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f811ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f8150e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f4b2180 con 0x55bf4f7e9240 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f81c140 con 0x55bf4f8127f0 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f809120 con 0x55bf4f82cb30 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51764/0 (socket says 172.21.15.28:51764) 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 -- 172.21.15.28:0/1010487110 learned_addr learned my addr 172.21.15.28:0/1010487110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f811ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83e00640 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f8150e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 -- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 msgr2=0x55bf4f8150e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f8150e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 -- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 msgr2=0x55bf4f811ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f811ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.818+0000 7f5c83400640 1 -- 172.21.15.28:0/1010487110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf4f6eb1f0 con 0x55bf4f7e9240 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c83400640 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5c740602a0 tx=0x7f5c74089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362f5e5be6fb1579 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c78c00640 1 -- 172.21.15.28:0/1010487110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7405e040 con 0x55bf4f7e9240 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c78c00640 1 -- 172.21.15.28:0/1010487110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c74087af0 con 0x55bf4f7e9240 2024-09-13T17:10:18.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c78c00640 1 -- 172.21.15.28:0/1010487110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74054ba0 con 0x55bf4f7e9240 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1010487110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 msgr2=0x55bf4f814a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5c740602a0 tx=0x7f5c74089040 comp rx=0 tx=0).stop 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.819+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1010487110 shutdown_connections 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f8150e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f811ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/1010487110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f814a00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1010487110 >> 172.21.15.28:0/1010487110 conn(0x55bf4f7e4610 msgr2=0x55bf4f808870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1010487110 shutdown_connections 2024-09-13T17:10:18.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/1010487110 wait complete. 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 -- start start 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.820+0000 7f5c8d7eb740 1 Processor -- start 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f89c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f82a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f89d0e0 con 0x55bf4f7e9240 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f8146c0 con 0x55bf4f8127f0 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bf4f814e90 con 0x55bf4f82cb30 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.821+0000 7f5c8d7eb740 10 client.0 _ll_register_callbacks cb 0x7f5c6c040050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51772/0 (socket says 172.21.15.28:51772) 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 -- 172.21.15.28:0/3948585339 learned_addr learned my addr 172.21.15.28:0/3948585339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83e00640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f82a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f89c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 -- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 msgr2=0x55bf4f82a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f82a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 -- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 msgr2=0x55bf4f89c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f89c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 -- 172.21.15.28:0/3948585339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c841502a0 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c83400640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5c74058190 tx=0x7f5c7c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af24e336810ab55 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.822+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c74078020 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c74087e40 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5c6c0517c0 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c7405e070 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 10 client.5340 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x55bf4f7107c0 con 0x55bf4f7e9240 2024-09-13T17:10:18.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c0517c0 con 0x55bf4f7e9240 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5c7407b020 con 0x55bf4f7e9240 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 10 client.5340 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 10 client.5340 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 20 client.5340 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f881d90 0x55bf4f81d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 4 client.5340 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "scrub_mdsdir"]}] 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "scrub_mdsdir"]}) v1 -- 0x55bf4f43e620 con 0x55bf4f881d90 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.823+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f881d90 0x55bf4f81d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.824+0000 7f5c7aa00640 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f881d90 0x55bf4f81d760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fd4572ddc002cc9 server_cookie=8e69802ca7182ffc in_seq=0 out_seq=0 2024-09-13T17:10:18.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.824+0000 7f5c70c00640 10 client.5340 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.827+0000 7f5c70c00640 1 -- 172.21.15.28:0/3948585339 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55bf4f43e620 con 0x55bf4f881d90 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.827+0000 7f5c70c00640 10 client.5340 handle_command_reply: tid=0 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.827+0000 7f5c8d7eb740 1 client.5340 shutdown 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 msgr2=0x55bf4f8168d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5c74058190 tx=0x7f5c7c00b040 comp rx=0 tx=0).stop 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 shutdown_connections 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x55bf4f881d90 0x55bf4f81d760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55bf4f82cb30 0x55bf4f82a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bf4f8127f0 0x55bf4f89c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 --2- 172.21.15.28:0/3948585339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bf4f7e9240 0x55bf4f8168d0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c70c00640 0 client.5340 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 >> 172.21.15.28:0/3948585339 conn(0x55bf4f7e4610 msgr2=0x55bf4f84ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 shutdown_connections 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 1 -- 172.21.15.28:0/3948585339 wait complete. 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.828+0000 7f5c8d7eb740 20 client.5340 trim_cache size 0 max 16384 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "4d52004c-cd83-4ef3-a289-c30820d23279", 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:10:18.831 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:10:18.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 msgr2=0x7f5c6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 0x7f5c6006d0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c034000 tx=0x7f5c6c02e000 comp rx=0 tx=0).stop 2024-09-13T17:10:18.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 msgr2=0x7f5c8414f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:18.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5c7400e9b0 tx=0x7f5c7400ee70 comp rx=0 tx=0).stop 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 shutdown_connections 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c6006ac00 0x7f5c6006d0c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c84159970 0x7f5c8413feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c84159590 0x7f5c8414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 --2- 172.21.15.28:0/4122525857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c840b2630 0x7f5c8414f370 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 >> 172.21.15.28:0/4122525857 conn(0x7f5c84069e80 msgr2=0x7f5c8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 shutdown_connections 2024-09-13T17:10:18.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:18.829+0000 7f5c8a800640 1 -- 172.21.15.28:0/4122525857 wait complete. 2024-09-13T17:10:18.849 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 scrub status 2024-09-13T17:10:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.089+0000 7f0c1a800640 1 Processor -- start 2024-09-13T17:10:19.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.089+0000 7f0c1a800640 1 -- start start 2024-09-13T17:10:19.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c1a800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0c1415c840 0x7f0c140ca870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 0x7f0c1406f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c1a800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140cb330 con 0x7f0c140cadb0 2024-09-13T17:10:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c1a800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c1415cc20 con 0x7f0c1415c840 2024-09-13T17:10:19.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c1a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140cb610 con 0x7f0c140ccbb0 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 0x7f0c1406f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.091+0000 7f0c18a00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0c140cadb0 0x7f0c140cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:57556/0 (socket says 172.21.15.28:57556) 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c18a00640 1 -- 172.21.15.28:0/738947168 learned_addr learned my addr 172.21.15.28:0/738947168 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c19400640 1 --2- 172.21.15.28:0/738947168 >> v2:172.21.15.189:3300/0 conn(0x7f0c1415c840 0x7f0c140ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c19e00640 1 -- 172.21.15.28:0/738947168 >> v2:172.21.15.189:3300/0 conn(0x7f0c1415c840 msgr2=0x7f0c140ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c19e00640 1 --2- 172.21.15.28:0/738947168 >> v2:172.21.15.189:3300/0 conn(0x7f0c1415c840 0x7f0c140ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c19e00640 1 -- 172.21.15.28:0/738947168 >> v1:172.21.15.189:6789/0 conn(0x7f0c140cadb0 legacy=0x7f0c140cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.092+0000 7f0c19e00640 1 -- 172.21.15.28:0/738947168 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140cb8f0 con 0x7f0c140ccbb0 2024-09-13T17:10:19.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.093+0000 7f0c19e00640 1 --2- 172.21.15.28:0/738947168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 0x7f0c1406f220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009d90 tx=0x7f0c0c02a610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a51ba075dd1b3903 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.094+0000 7f0c13e00640 1 -- 172.21.15.28:0/738947168 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c031600 con 0x7f0c140ccbb0 2024-09-13T17:10:19.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.094+0000 7f0c13e00640 1 -- 172.21.15.28:0/738947168 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c02d050 con 0x7f0c140ccbb0 2024-09-13T17:10:19.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.094+0000 7f0c13e00640 1 -- 172.21.15.28:0/738947168 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c031980 con 0x7f0c140ccbb0 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.094+0000 7f0c1a800640 1 -- 172.21.15.28:0/738947168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 msgr2=0x7f0c1406f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.094+0000 7f0c1a800640 1 --2- 172.21.15.28:0/738947168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 0x7f0c1406f220 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009d90 tx=0x7f0c0c02a610 comp rx=0 tx=0).stop 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.095+0000 7f0c1a800640 1 -- 172.21.15.28:0/738947168 shutdown_connections 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.095+0000 7f0c1a800640 1 --2- 172.21.15.28:0/738947168 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140ccbb0 0x7f0c1406f220 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.095+0000 7f0c1a800640 1 --2- 172.21.15.28:0/738947168 >> v2:172.21.15.189:3300/0 conn(0x7f0c1415c840 0x7f0c140ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.095+0000 7f0c1a800640 1 -- 172.21.15.28:0/738947168 >> 172.21.15.28:0/738947168 conn(0x7f0c1406b330 msgr2=0x7f0c1406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.095+0000 7f0c1a800640 1 -- 172.21.15.28:0/738947168 shutdown_connections 2024-09-13T17:10:19.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.096+0000 7f0c1a800640 1 -- 172.21.15.28:0/738947168 wait complete. 2024-09-13T17:10:19.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.100+0000 7f0c1a800640 1 Processor -- start 2024-09-13T17:10:19.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.101+0000 7f0c1a800640 1 -- start start 2024-09-13T17:10:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140cadb0 0x7f0c14150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1415c840 0x7f0c1415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1414b400 con 0x7f0c140ccbb0 2024-09-13T17:10:19.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1414fd40 con 0x7f0c1415c840 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.103+0000 7f0c1a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c1415eaf0 con 0x7f0c140cadb0 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51786/0 (socket says 172.21.15.28:51786) 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 -- 172.21.15.28:0/1347905729 learned_addr learned my addr 172.21.15.28:0/1347905729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c19e00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1415c840 0x7f0c1415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c19400640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140cadb0 0x7f0c14150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 -- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140cadb0 msgr2=0x7f0c14150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140cadb0 0x7f0c14150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 -- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1415c840 msgr2=0x7f0c1415dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1415c840 0x7f0c1415dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.104+0000 7f0c18a00640 1 -- 172.21.15.28:0/1347905729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c009a70 con 0x7f0c140ccbb0 2024-09-13T17:10:19.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.105+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c460 tx=0x7f0c0800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.106+0000 7f0c11600640 1 -- 172.21.15.28:0/1347905729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08005ce0 con 0x7f0c140ccbb0 2024-09-13T17:10:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.106+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1415edd0 con 0x7f0c140ccbb0 2024-09-13T17:10:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.106+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c141506f0 con 0x7f0c140ccbb0 2024-09-13T17:10:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.106+0000 7f0c11600640 1 -- 172.21.15.28:0/1347905729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c08005e60 con 0x7f0c140ccbb0 2024-09-13T17:10:19.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.106+0000 7f0c11600640 1 -- 172.21.15.28:0/1347905729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c080072b0 con 0x7f0c140ccbb0 2024-09-13T17:10:19.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.111+0000 7f0c1ecd9740 1 Processor -- start 2024-09-13T17:10:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.112+0000 7f0c1ecd9740 1 -- start start 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.113+0000 7f0c11600640 1 -- 172.21.15.28:0/1347905729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0c08007410 con 0x7f0c140ccbb0 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 0x5562e836b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e836b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e82246a0 con 0x5562e831b590 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e8287cc0 con 0x5562e831ddd0 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e8285450 con 0x5562e8345280 2024-09-13T17:10:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.114+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:36660/0 (socket says 172.21.15.28:36660) 2024-09-13T17:10:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c19e00640 1 -- 172.21.15.28:0/305626833 learned_addr learned my addr 172.21.15.28:0/305626833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c19400640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 0x5562e836b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c11600640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 0x7f0bf406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c18a00640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e836b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.115+0000 7f0c11600640 1 -- 172.21.15.28:0/1347905729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0c08088e50 con 0x7f0c140ccbb0 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 -- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 msgr2=0x5562e836b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e836b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 -- 172.21.15.28:0/305626833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 msgr2=0x5562e836b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 0x5562e836b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 -- 172.21.15.28:0/305626833 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562e82512e0 con 0x5562e831ddd0 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19e00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 0x7f0bf406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.116+0000 7f0c19400640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 0x5562e836b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:10:19.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.117+0000 7f0c19e00640 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009d90 tx=0x7f0c0c02e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d2e0d85803c9f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.118+0000 7f0c19e00640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 0x7f0bf406d0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c04b000 tx=0x7f0c0c045000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.118+0000 7f0c06000640 1 -- 172.21.15.28:0/305626833 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0487e0 con 0x5562e831ddd0 2024-09-13T17:10:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.118+0000 7f0c06000640 1 -- 172.21.15.28:0/305626833 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c048940 con 0x5562e831ddd0 2024-09-13T17:10:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.118+0000 7f0c06000640 1 -- 172.21.15.28:0/305626833 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c048cb0 con 0x5562e831ddd0 2024-09-13T17:10:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.119+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 msgr2=0x5562e831e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.119+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c009d90 tx=0x7f0c0c02e000 comp rx=0 tx=0).stop 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/305626833 shutdown_connections 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e836b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831b590 0x5562e836b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/305626833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e831ddd0 0x5562e831e1b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/305626833 >> 172.21.15.28:0/305626833 conn(0x5562e831a500 msgr2=0x5562e83629b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/305626833 shutdown_connections 2024-09-13T17:10:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.120+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/305626833 wait complete. 2024-09-13T17:10:19.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.121+0000 7f0c1ecd9740 1 -- start start 2024-09-13T17:10:19.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.122+0000 7f0c1ecd9740 1 Processor -- start 2024-09-13T17:10:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e8346750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e8346c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e8280d40 con 0x5562e831ddd0 2024-09-13T17:10:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e8222280 con 0x5562e8367ee0 2024-09-13T17:10:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e8221f20 con 0x5562e8345280 2024-09-13T17:10:19.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.123+0000 7f0c1ecd9740 10 client.0 _ll_register_callbacks cb 0x5562e827a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.124+0000 7f0c19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e8346c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.124+0000 7f0c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.124+0000 7f0c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51796/0 (socket says 172.21.15.28:51796) 2024-09-13T17:10:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.124+0000 7f0c19400640 1 -- 172.21.15.28:0/1712711472 learned_addr learned my addr 172.21.15.28:0/1712711472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.125+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e8346750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.126+0000 7f0c19400640 1 -- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 msgr2=0x5562e8346750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.126+0000 7f0c19400640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e8346750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.126+0000 7f0c19400640 1 -- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 msgr2=0x5562e8346c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.126+0000 7f0c19400640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e8346c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.126+0000 7f0c19400640 1 -- 172.21.15.28:0/1712711472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c140cb8f0 con 0x5562e831ddd0 2024-09-13T17:10:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.127+0000 7f0c19400640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0c000038d0 tx=0x7f0c00003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22085de61e787cb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.127+0000 7f0bfaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.128+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c00009c50 con 0x5562e831ddd0 2024-09-13T17:10:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.128+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0c00009db0 con 0x5562e831ddd0 2024-09-13T17:10:19.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.128+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5562e7f744e0 con 0x5562e831ddd0 2024-09-13T17:10:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.130+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c00020460 con 0x5562e831ddd0 2024-09-13T17:10:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.130+0000 7f0c1ecd9740 10 client.5349 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:19.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x5562e827f3c0 con 0x5562e831ddd0 2024-09-13T17:10:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5562e7f744e0 con 0x5562e831ddd0 2024-09-13T17:10:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0c0001e040 con 0x5562e831ddd0 2024-09-13T17:10:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0c1ecd9740 10 client.5349 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:19.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0c1ecd9740 10 client.5349 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.131+0000 7f0c1ecd9740 20 client.5349 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.132+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e836b230 0x5562e836b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.132+0000 7f0c1ecd9740 4 client.5349 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:10:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.132+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5562e7f857e0 con 0x5562e836b230 2024-09-13T17:10:19.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.133+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e836b230 0x5562e836b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.135+0000 7f0c18a00640 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e836b230 0x5562e836b640 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0ce9d2236258a8f server_cookie=5144c5062eb45fb1 in_seq=0 out_seq=0 2024-09-13T17:10:19.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.135+0000 7f0bfaa00640 10 client.5349 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.147+0000 7f0bfaa00640 1 -- 172.21.15.28:0/1712711472 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5562e7f857e0 con 0x5562e836b230 2024-09-13T17:10:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.147+0000 7f0bfaa00640 10 client.5349 handle_command_reply: tid=0 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.147+0000 7f0c1ecd9740 1 client.5349 shutdown 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 msgr2=0x5562e8344290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0c000038d0 tx=0x7f0c00003d90 comp rx=0 tx=0).stop 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 shutdown_connections 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e836b230 0x5562e836b640 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e8346c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8345280 0x5562e8346750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/1712711472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e831ddd0 0x5562e8344290 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 >> 172.21.15.28:0/1712711472 conn(0x5562e831b590 msgr2=0x5562e83666c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0bfaa00640 0 client.5349 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 shutdown_connections 2024-09-13T17:10:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.148+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/1712711472 wait complete. 2024-09-13T17:10:19.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.150+0000 7f0c1ecd9740 20 client.5349 trim_cache size 0 max 16384 2024-09-13T17:10:19.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.194+0000 7f0c1ecd9740 1 Processor -- start 2024-09-13T17:10:19.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.195+0000 7f0c1ecd9740 1 -- start start 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8343ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8367ee0 0x5562e8363470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 0x5562e83639b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e7fe8040 con 0x5562e8345280 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e835feb0 con 0x5562e836b0e0 2024-09-13T17:10:19.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.196+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e8349c30 con 0x5562e8367ee0 2024-09-13T17:10:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.197+0000 7f0c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8343ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.197+0000 7f0c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8343ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51810/0 (socket says 172.21.15.28:51810) 2024-09-13T17:10:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.197+0000 7f0c19400640 1 -- 172.21.15.28:0/3923971741 learned_addr learned my addr 172.21.15.28:0/3923971741 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.197+0000 7f0c19e00640 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 0x5562e83639b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.197+0000 7f0c18a00640 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8367ee0 0x5562e8363470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.198+0000 7f0c19e00640 1 -- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8367ee0 msgr2=0x5562e8363470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.198+0000 7f0c19e00640 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8367ee0 0x5562e8363470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.198+0000 7f0c19e00640 1 -- 172.21.15.28:0/3923971741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 msgr2=0x5562e8343ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.198+0000 7f0c19e00640 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8343ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.198+0000 7f0c19e00640 1 -- 172.21.15.28:0/3923971741 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562e8221610 con 0x5562e836b0e0 2024-09-13T17:10:19.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.199+0000 7f0c19e00640 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 0x5562e83639b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c048000 tx=0x7f0c0c030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a25cada45e5506 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.199+0000 7f0c06a00640 1 -- 172.21.15.28:0/3923971741 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c0487e0 con 0x5562e836b0e0 2024-09-13T17:10:19.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.200+0000 7f0c06a00640 1 -- 172.21.15.28:0/3923971741 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c048ab0 con 0x5562e836b0e0 2024-09-13T17:10:19.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.200+0000 7f0c06a00640 1 -- 172.21.15.28:0/3923971741 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c048e20 con 0x5562e836b0e0 2024-09-13T17:10:19.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.202+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 msgr2=0x5562e83639b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.202+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 0x5562e83639b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c048000 tx=0x7f0c0c030000 comp rx=0 tx=0).stop 2024-09-13T17:10:19.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.203+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/3923971741 shutdown_connections 2024-09-13T17:10:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.203+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e836b0e0 0x5562e83639b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.203+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e8367ee0 0x5562e8363470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.203+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/3923971741 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8343ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.203+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/3923971741 >> 172.21.15.28:0/3923971741 conn(0x5562e831b590 msgr2=0x5562e8384d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.204+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/3923971741 shutdown_connections 2024-09-13T17:10:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.204+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/3923971741 wait complete. 2024-09-13T17:10:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.205+0000 7f0c1ecd9740 1 -- start start 2024-09-13T17:10:19.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.206+0000 7f0c1ecd9740 1 Processor -- start 2024-09-13T17:10:19.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8362c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e836b0e0 0x5562e8343ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e83195c0 con 0x5562e8345280 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e835e0d0 con 0x5562e8367ee0 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e8366ab0 con 0x5562e836b0e0 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.207+0000 7f0c1ecd9740 10 client.0 _ll_register_callbacks cb 0x7f0c000043d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.208+0000 7f0c19400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8362c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.209+0000 7f0c18a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.209+0000 7f0c18a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:36694/0 (socket says 172.21.15.28:36694) 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.209+0000 7f0c18a00640 1 -- 172.21.15.28:0/2906132118 learned_addr learned my addr 172.21.15.28:0/2906132118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.210+0000 7f0c19e00640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e836b0e0 0x5562e8343ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.210+0000 7f0c18a00640 1 -- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e836b0e0 msgr2=0x5562e8343ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.210+0000 7f0c18a00640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e836b0e0 0x5562e8343ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.210+0000 7f0c18a00640 1 -- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 msgr2=0x5562e8362c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:19.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.211+0000 7f0c18a00640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8362c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.211+0000 7f0c18a00640 1 -- 172.21.15.28:0/2906132118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562e82512e0 con 0x5562e8367ee0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.211+0000 7f0c18a00640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c430 tx=0x7f0c08087200 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f21ceadac88d7d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0bfaa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08087330 con 0x5562e8367ee0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5765+0+0 (secure 0 0 0) 0x7f0c080543b0 con 0x5562e8367ee0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c0c048940 con 0x5562e8367ee0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c08087490 con 0x5562e8367ee0 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0c1ecd9740 10 client.5227 fetch_fsmap learned FSMap version 22 2024-09-13T17:10:19.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.212+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x5562e8246b20 con 0x5562e8367ee0 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0c048940 con 0x5562e8367ee0 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 22) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0c08019020 con 0x5562e8367ee0 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 10 client.5227 fetch_fsmap finished waiting for FSMap version 22 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 10 client.5227 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 20 client.5227 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e8344220 0x5562e83d0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 4 client.5227 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:10:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.213+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5562e7fd8e30 con 0x5562e8344220 2024-09-13T17:10:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.214+0000 7f0c19400640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e8344220 0x5562e83d0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.217+0000 7f0c19400640 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e8344220 0x5562e83d0810 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d143fefa496b4e server_cookie=6dcc959aedb348ad in_seq=0 out_seq=0 2024-09-13T17:10:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.217+0000 7f0bfaa00640 10 client.5227 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0bfaa00640 1 -- 172.21.15.28:0/2906132118 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+275 (crc 0 0 0) 0x5562e7fd8e30 con 0x5562e8344220 2024-09-13T17:10:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0bfaa00640 10 client.5227 handle_command_reply: tid=0 2024-09-13T17:10:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 client.5227 shutdown 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 msgr2=0x5562e83b2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c430 tx=0x7f0c08087200 comp rx=0 tx=0).stop 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 shutdown_connections 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x5562e8344220 0x5562e83d0810 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5562e836b0e0 0x5562e8343ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5562e8367ee0 0x5562e83b2db0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0bfaa00640 0 client.5227 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 --2- 172.21.15.28:0/2906132118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5562e8345280 0x5562e8362c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.218+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 >> 172.21.15.28:0/2906132118 conn(0x5562e831b590 msgr2=0x5562e836b500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.219+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 shutdown_connections 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.219+0000 7f0c1ecd9740 1 -- 172.21.15.28:0/2906132118 wait complete. 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.220+0000 7f0c1ecd9740 20 client.5227 trim_cache size 0 max 16384 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:10:19.222 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (1170 inodes in the stack)", 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "4d52004c-cd83-4ef3-a289-c30820d23279": { 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "4d52004c-cd83-4ef3-a289-c30820d23279", 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,scrub_mdsdir" 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:19.223 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:10:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 msgr2=0x7f0bf406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 0x7f0bf406d0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c04b000 tx=0x7f0c0c045000 comp rx=0 tx=0).stop 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 msgr2=0x7f0c1414aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c460 tx=0x7f0c0800c920 comp rx=0 tx=0).stop 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 shutdown_connections 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bf406ac30 0x7f0bf406d0f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0c1415c840 0x7f0c1415dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0c140ccbb0 0x7f0c1414aec0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 --2- 172.21.15.28:0/1347905729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0c140cadb0 0x7f0c14150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.221+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 >> 172.21.15.28:0/1347905729 conn(0x7f0c1406b330 msgr2=0x7f0c140b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.222+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 shutdown_connections 2024-09-13T17:10:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:19.222+0000 7f0c1a800640 1 -- 172.21.15.28:0/1347905729 wait complete. 2024-09-13T17:10:19.251 INFO:tasks.cephfs.filesystem:scrub status for tag:4d52004c-cd83-4ef3-a289-c30820d23279 - {'path': '/', 'tag': '4d52004c-cd83-4ef3-a289-c30820d23279', 'options': 'recursive,scrub_mdsdir'} 2024-09-13T17:10:39.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:10:39.234 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:10:49.252 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 scrub status 2024-09-13T17:10:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.419+0000 7f361f200640 1 Processor -- start 2024-09-13T17:10:49.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.419+0000 7f361f200640 1 -- start start 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361f200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3620143320 0x7f3620070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3620071630 con 0x7f36200cd120 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3620143700 con 0x7f3620143910 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361f200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3620070a20 con 0x7f3620143320 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f36200cd120 0x7f362015ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44726/0 (socket says 172.21.15.28:44726) 2024-09-13T17:10:49.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361e800640 1 -- 172.21.15.28:0/4025851427 learned_addr learned my addr 172.21.15.28:0/4025851427 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361de00640 1 --2- 172.21.15.28:0/4025851427 >> v2:172.21.15.189:3300/0 conn(0x7f3620143320 0x7f3620070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643378421 0 0) 0x7f3620071630 con 0x7f36200cd120 2024-09-13T17:10:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.421+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3604003520 con 0x7f36200cd120 2024-09-13T17:10:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2894007735 0 0) 0x7f3604003520 con 0x7f36200cd120 2024-09-13T17:10:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 >> v2:172.21.15.189:3300/0 conn(0x7f3620143320 msgr2=0x7f3620070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 --2- 172.21.15.28:0/4025851427 >> v2:172.21.15.189:3300/0 conn(0x7f3620143320 0x7f3620070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 >> v1:172.21.15.189:6789/0 conn(0x7f3620143910 legacy=0x7f3620071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620070d00 con 0x7f36200cd120 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f36180019f0 con 0x7f36200cd120 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 >> v1:172.21.15.28:6789/0 conn(0x7f36200cd120 legacy=0x7f362015ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36040039c0 0x7f3604003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36040047c0 0x7f3604004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 0x7f360400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3620070a20 con 0x7f3604005180 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f360400c080 con 0x7f36040039c0 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f360400c240 con 0x7f36040047c0 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361e800640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 0x7f360400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f361de00640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36040039c0 0x7f3604003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.422+0000 7f3615400640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36040047c0 0x7f3604004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.423+0000 7f361e800640 1 -- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36040047c0 msgr2=0x7f3604004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.423+0000 7f361e800640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36040047c0 0x7f3604004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.423+0000 7f361e800640 1 -- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36040039c0 msgr2=0x7f3604003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.423+0000 7f361e800640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36040039c0 0x7f3604003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.423+0000 7f361e800640 1 -- 172.21.15.28:0/4025851427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3604003520 con 0x7f3604005180 2024-09-13T17:10:49.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361e800640 1 -- 172.21.15.28:0/4025851427 reap_dead start 2024-09-13T17:10:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361e800640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 0x7f360400b980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f361800a760 tx=0x7f36040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f108e5573c6c4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361802b980 con 0x7f3604005180 2024-09-13T17:10:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361802bb20 con 0x7f3604005180 2024-09-13T17:10:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361d400640 1 -- 172.21.15.28:0/4025851427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618034470 con 0x7f3604005180 2024-09-13T17:10:49.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361f200640 1 -- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 msgr2=0x7f360400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.424+0000 7f361f200640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 0x7f360400b980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f361800a760 tx=0x7f36040057f0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.425+0000 7f361f200640 1 -- 172.21.15.28:0/4025851427 shutdown_connections 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.425+0000 7f361f200640 1 --2- 172.21.15.28:0/4025851427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3604005180 0x7f360400b980 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.425+0000 7f361f200640 1 -- 172.21.15.28:0/4025851427 >> 172.21.15.28:0/4025851427 conn(0x7f3620069e80 msgr2=0x7f362006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.425+0000 7f361f200640 1 -- 172.21.15.28:0/4025851427 shutdown_connections 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.425+0000 7f361f200640 1 -- 172.21.15.28:0/4025851427 wait complete. 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.426+0000 7f361f200640 1 Processor -- start 2024-09-13T17:10:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 -- start start 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604005180 0x7f362015cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f362015bc40 0x7f36200cc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3620138b60 con 0x7f36200ca140 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362013cd80 con 0x7f3604005180 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.427+0000 7f361f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362015d160 con 0x7f362015bc40 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53416/0 (socket says 172.21.15.28:53416) 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 -- 172.21.15.28:0/2879460591 learned_addr learned my addr 172.21.15.28:0/2879460591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f361de00640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604005180 0x7f362015cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f361e800640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f362015bc40 0x7f36200cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 -- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f362015bc40 msgr2=0x7f36200cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f362015bc40 0x7f36200cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 -- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604005180 msgr2=0x7f362015cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604005180 0x7f362015cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 -- 172.21.15.28:0/2879460591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362013d020 con 0x7f36200ca140 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.428+0000 7f3615400640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f360800f5f0 tx=0x7f360800f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.429+0000 7f3616a00640 1 -- 172.21.15.28:0/2879460591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080052e0 con 0x7f36200ca140 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.429+0000 7f3616a00640 1 -- 172.21.15.28:0/2879460591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608005440 con 0x7f36200ca140 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.429+0000 7f3616a00640 1 -- 172.21.15.28:0/2879460591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608003810 con 0x7f36200ca140 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.429+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362013d300 con 0x7f36200ca140 2024-09-13T17:10:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.429+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362015dcf0 con 0x7f36200ca140 2024-09-13T17:10:49.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.432+0000 7f3616a00640 1 -- 172.21.15.28:0/2879460591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3608003970 con 0x7f36200ca140 2024-09-13T17:10:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.433+0000 7f3616a00640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 0x7f35fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.433+0000 7f3616a00640 1 -- 172.21.15.28:0/2879460591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f36080890d0 con 0x7f36200ca140 2024-09-13T17:10:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.433+0000 7f3627c2f740 1 Processor -- start 2024-09-13T17:10:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.433+0000 7f361de00640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 0x7f35fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.433+0000 7f3627c2f740 1 -- start start 2024-09-13T17:10:49.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f361de00640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 0x7f35fc06d0c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f362013f950 tx=0x7f3610009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70b590 0x564d4f75b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f75b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f6146a0 con 0x564d4f70ddd0 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f677cc0 con 0x564d4f735280 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d4f675450 con 0x564d4f70b590 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.434+0000 7f361de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53418/0 (socket says 172.21.15.28:53418) 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 -- 172.21.15.28:0/4282898033 learned_addr learned my addr 172.21.15.28:0/4282898033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f3615400640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70b590 0x564d4f75b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361e800640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f75b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 -- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70b590 msgr2=0x564d4f75b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70b590 0x564d4f75b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 -- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 msgr2=0x564d4f75b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f75b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 -- 172.21.15.28:0/4282898033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d4f6412e0 con 0x564d4f70ddd0 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361e800640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f75b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.435+0000 7f361de00640 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3610006d40 tx=0x7f3610017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d59ae97c06ac4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f360ea00640 1 -- 172.21.15.28:0/4282898033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610015e30 con 0x564d4f70ddd0 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f360ea00640 1 -- 172.21.15.28:0/4282898033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3610025070 con 0x564d4f70ddd0 2024-09-13T17:10:49.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f360ea00640 1 -- 172.21.15.28:0/4282898033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361002a630 con 0x564d4f70ddd0 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 -- 172.21.15.28:0/4282898033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 msgr2=0x564d4f70e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3610006d40 tx=0x7f3610017000 comp rx=0 tx=0).stop 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 -- 172.21.15.28:0/4282898033 shutdown_connections 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f75b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70b590 0x564d4f75b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4282898033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f70e1b0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 -- 172.21.15.28:0/4282898033 >> 172.21.15.28:0/4282898033 conn(0x564d4f70a500 msgr2=0x564d4f7529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 -- 172.21.15.28:0/4282898033 shutdown_connections 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.436+0000 7f3627c2f740 1 -- 172.21.15.28:0/4282898033 wait complete. 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.437+0000 7f3627c2f740 1 -- start start 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.437+0000 7f3627c2f740 1 Processor -- start 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f738d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f74ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f670d40 con 0x564d4f75ac90 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f612280 con 0x564d4f735280 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d4f611f20 con 0x564d4f70ddd0 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f3627c2f740 10 client.0 _ll_register_callbacks cb 0x564d4f66a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f361e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f361e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53434/0 (socket says 172.21.15.28:53434) 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.438+0000 7f361e800640 1 -- 172.21.15.28:0/4051492224 learned_addr learned my addr 172.21.15.28:0/4051492224 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361de00640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f738d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f3615400640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f74ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 -- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 msgr2=0x564d4f738d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f738d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 -- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 msgr2=0x564d4f74ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f74ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 -- 172.21.15.28:0/4051492224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3610014c00 con 0x564d4f75ac90 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f361e800640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f361800a850 tx=0x7f361802d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae75866111bf3b2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f3603400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f361803b070 con 0x564d4f75ac90 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3618034880 con 0x564d4f75ac90 2024-09-13T17:10:49.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.439+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d4f3644e0 con 0x564d4f75ac90 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3618001250 con 0x564d4f75ac90 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 10 client.5367 fetch_fsmap learned FSMap version 27 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x564d4f66f3c0 con 0x564d4f75ac90 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564d4f3644e0 con 0x564d4f75ac90 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3618034a40 con 0x564d4f75ac90 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 10 client.5367 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 10 client.5367 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 20 client.5367 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f757550 0x564d4f757960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 4 client.5367 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d4f3757e0 con 0x564d4f757550 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.440+0000 7f361de00640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f757550 0x564d4f757960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.441+0000 7f361de00640 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f757550 0x564d4f757960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e40e12d61fb8afec server_cookie=dd7e2374129e50dd in_seq=0 out_seq=0 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.441+0000 7f3603400640 10 client.5367 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.441+0000 7f3603400640 1 -- 172.21.15.28:0/4051492224 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x564d4f3757e0 con 0x564d4f757550 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.441+0000 7f3603400640 10 client.5367 handle_command_reply: tid=0 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.441+0000 7f3627c2f740 1 client.5367 shutdown 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 msgr2=0x564d4f7367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f361800a850 tx=0x7f361802d070 comp rx=0 tx=0).stop 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 shutdown_connections 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f757550 0x564d4f757960 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f7367d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f74ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4051492224 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f738d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3603400640 0 client.5367 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 >> 172.21.15.28:0/4051492224 conn(0x564d4f70b590 msgr2=0x564d4f751f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 shutdown_connections 2024-09-13T17:10:49.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 1 -- 172.21.15.28:0/4051492224 wait complete. 2024-09-13T17:10:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.442+0000 7f3627c2f740 20 client.5367 trim_cache size 0 max 16384 2024-09-13T17:10:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.457+0000 7f3627c2f740 1 Processor -- start 2024-09-13T17:10:49.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.457+0000 7f3627c2f740 1 -- start start 2024-09-13T17:10:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f735280 0x564d4f750a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 0x564d4f750f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f3d8040 con 0x564d4f70ddd0 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f66fee0 con 0x564d4f75ac90 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d4f737140 con 0x564d4f735280 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53444/0 (socket says 172.21.15.28:53444) 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 -- 172.21.15.28:0/3811742864 learned_addr learned my addr 172.21.15.28:0/3811742864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361e800640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 0x564d4f750f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f3615400640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f735280 0x564d4f750a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 -- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f735280 msgr2=0x564d4f750a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f735280 0x564d4f750a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 -- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 msgr2=0x564d4f750f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 0x564d4f750f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361de00640 1 -- 172.21.15.28:0/3811742864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d4f709d40 con 0x564d4f70ddd0 2024-09-13T17:10:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.458+0000 7f361e800640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 0x564d4f750f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f361de00640 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3610009ff0 tx=0x7f361001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acb74ef1ba71b57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f360f400640 1 -- 172.21.15.28:0/3811742864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3610006b20 con 0x564d4f70ddd0 2024-09-13T17:10:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f360f400640 1 -- 172.21.15.28:0/3811742864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3610019070 con 0x564d4f70ddd0 2024-09-13T17:10:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f360f400640 1 -- 172.21.15.28:0/3811742864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36100372f0 con 0x564d4f70ddd0 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f3627c2f740 1 -- 172.21.15.28:0/3811742864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 msgr2=0x564d4f735850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.459+0000 7f3627c2f740 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3610009ff0 tx=0x7f361001c000 comp rx=0 tx=0).stop 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 -- 172.21.15.28:0/3811742864 shutdown_connections 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f75ac90 0x564d4f750f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f735280 0x564d4f750a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 --2- 172.21.15.28:0/3811742864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f70ddd0 0x564d4f735850 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 -- 172.21.15.28:0/3811742864 >> 172.21.15.28:0/3811742864 conn(0x564d4f70b590 msgr2=0x564d4f759fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 -- 172.21.15.28:0/3811742864 shutdown_connections 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 -- 172.21.15.28:0/3811742864 wait complete. 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 -- start start 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.460+0000 7f3627c2f740 1 Processor -- start 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f737a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f7bd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f70d430 con 0x564d4f75ac90 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d4f7568f0 con 0x564d4f735280 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564d4f70d5c0 con 0x564d4f70ddd0 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.461+0000 7f3627c2f740 10 client.0 _ll_register_callbacks cb 0x7f3610009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53448/0 (socket says 172.21.15.28:53448) 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 -- 172.21.15.28:0/4000586993 learned_addr learned my addr 172.21.15.28:0/4000586993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361de00640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f737a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f3615400640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f7bd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 -- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 msgr2=0x564d4f737a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f737a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 -- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 msgr2=0x564d4f7bd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f7bd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 -- 172.21.15.28:0/4000586993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f361002e040 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f3615400640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f7bd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f361e800640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f361802b430 tx=0x7f3618038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d40dfff076c997 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.462+0000 7f3603400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618004d60 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3618052460 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3610025070 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3618052620 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 10 client.5373 fetch_fsmap learned FSMap version 27 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x564d4f611610 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3610025070 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3618052780 con 0x564d4f75ac90 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 10 client.5373 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T17:10:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 10 client.5373 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 20 client.5373 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f752b20 0x564d4f70c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 4 client.5373 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:10:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d4f3c8e30 con 0x564d4f752b20 2024-09-13T17:10:49.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.463+0000 7f361de00640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f752b20 0x564d4f70c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f361de00640 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f752b20 0x564d4f70c300 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c90ad07a25d20ff9 server_cookie=2bac68dfd981a0d9 in_seq=0 out_seq=0 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3603400640 10 client.5373 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3603400640 1 -- 172.21.15.28:0/4000586993 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564d4f3c8e30 con 0x564d4f752b20 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3603400640 10 client.5373 handle_command_reply: tid=0 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 client.5373 shutdown 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 msgr2=0x564d4f755e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f361802b430 tx=0x7f3618038040 comp rx=0 tx=0).stop 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 shutdown_connections 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x564d4f752b20 0x564d4f70c300 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564d4f75ac90 0x564d4f755e70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d4f735280 0x564d4f7bd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 --2- 172.21.15.28:0/4000586993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564d4f70ddd0 0x564d4f737a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 >> 172.21.15.28:0/4000586993 conn(0x564d4f70b590 msgr2=0x564d4f759fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.464+0000 7f3603400640 0 client.5373 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 shutdown_connections 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f3627c2f740 1 -- 172.21.15.28:0/4000586993 wait complete. 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f3627c2f740 20 client.5373 trim_cache size 0 max 16384 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:10:49.471 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 msgr2=0x7f35fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 0x7f35fc06d0c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f362013f950 tx=0x7f3610009190 comp rx=0 tx=0).stop 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 msgr2=0x7f3620138620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f360800f5f0 tx=0x7f360800f9f0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 shutdown_connections 2024-09-13T17:10:49.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35fc06ac00 0x7f35fc06d0c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f362015bc40 0x7f36200cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36200ca140 0x7f3620138620 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 --2- 172.21.15.28:0/2879460591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3604005180 0x7f362015cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 >> 172.21.15.28:0/2879460591 conn(0x7f3620069e80 msgr2=0x7f3620071910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 shutdown_connections 2024-09-13T17:10:49.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.465+0000 7f361f200640 1 -- 172.21.15.28:0/2879460591 wait complete. 2024-09-13T17:10:49.481 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:10:49.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 perf dump 2024-09-13T17:10:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.638+0000 7fe604c00640 1 Processor -- start 2024-09-13T17:10:49.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.638+0000 7fe604c00640 1 -- start start 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe604c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe60015c9a0 0x7fe6000ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe604c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe60006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe604c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe600070470 con 0x7fe6000ad940 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe604c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6000ace30 con 0x7fe60015c9a0 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe604c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000ad110 con 0x7fe6000b2400 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe5fd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe6000ad940 0x7fe6000b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44736/0 (socket says 172.21.15.28:44736) 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.639+0000 7fe5fd400640 1 -- 172.21.15.28:0/1613815856 learned_addr learned my addr 172.21.15.28:0/1613815856 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fe800640 1 --2- 172.21.15.28:0/1613815856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe60006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1613815856 >> v2:172.21.15.189:3300/0 conn(0x7fe60015c9a0 0x7fe6000ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fca00640 1 -- 172.21.15.28:0/1613815856 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652840600 0 0) 0x7fe600070470 con 0x7fe6000ad940 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fca00640 1 -- 172.21.15.28:0/1613815856 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5e4003520 con 0x7fe6000ad940 2024-09-13T17:10:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fe800640 1 -- 172.21.15.28:0/1613815856 >> v2:172.21.15.189:3300/0 conn(0x7fe60015c9a0 msgr2=0x7fe6000ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fe800640 1 --2- 172.21.15.28:0/1613815856 >> v2:172.21.15.189:3300/0 conn(0x7fe60015c9a0 0x7fe6000ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fe800640 1 -- 172.21.15.28:0/1613815856 >> v1:172.21.15.28:6789/0 conn(0x7fe6000ad940 legacy=0x7fe6000b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.640+0000 7fe5fe800640 1 -- 172.21.15.28:0/1613815856 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60006b170 con 0x7fe6000b2400 2024-09-13T17:10:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.641+0000 7fe5fe800640 1 --2- 172.21.15.28:0/1613815856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe60006bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009dc0 tx=0x7fe5f802a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd160c9f70d813c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.641+0000 7fe5fca00640 1 -- 172.21.15.28:0/1613815856 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8007740 con 0x7fe6000b2400 2024-09-13T17:10:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.641+0000 7fe5fca00640 1 -- 172.21.15.28:0/1613815856 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f802d050 con 0x7fe6000b2400 2024-09-13T17:10:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe5fca00640 1 -- 172.21.15.28:0/1613815856 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8007ac0 con 0x7fe6000b2400 2024-09-13T17:10:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 -- 172.21.15.28:0/1613815856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 msgr2=0x7fe60006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 --2- 172.21.15.28:0/1613815856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe60006bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009dc0 tx=0x7fe5f802a710 comp rx=0 tx=0).stop 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 -- 172.21.15.28:0/1613815856 shutdown_connections 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 --2- 172.21.15.28:0/1613815856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe60006bc50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 --2- 172.21.15.28:0/1613815856 >> v2:172.21.15.189:3300/0 conn(0x7fe60015c9a0 0x7fe6000ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 -- 172.21.15.28:0/1613815856 >> 172.21.15.28:0/1613815856 conn(0x7fe6000b0270 msgr2=0x7fe6000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 -- 172.21.15.28:0/1613815856 shutdown_connections 2024-09-13T17:10:49.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.642+0000 7fe604c00640 1 -- 172.21.15.28:0/1613815856 wait complete. 2024-09-13T17:10:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.644+0000 7fe604c00640 1 Processor -- start 2024-09-13T17:10:49.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.644+0000 7fe604c00640 1 -- start start 2024-09-13T17:10:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6000ad940 0x7fe60015f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe6001f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000c3580 con 0x7fe60015c9a0 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000c3860 con 0x7fe6000ad940 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe604c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000c3b40 con 0x7fe6000b2400 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe5fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe5fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53462/0 (socket says 172.21.15.28:53462) 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe5fe800640 1 -- 172.21.15.28:0/125212766 learned_addr learned my addr 172.21.15.28:0/125212766 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe5fde00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6000ad940 0x7fe60015f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.645+0000 7fe5fd400640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe6001f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 -- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 msgr2=0x7fe6001f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe6001f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 -- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6000ad940 msgr2=0x7fe60015f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6000ad940 0x7fe60015f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 -- 172.21.15.28:0/125212766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f8009a70 con 0x7fe60015c9a0 2024-09-13T17:10:49.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5fe800640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009ee0 tx=0x7fe5f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5f6000640 1 -- 172.21.15.28:0/125212766 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f803c830 con 0x7fe60015c9a0 2024-09-13T17:10:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5f6000640 1 -- 172.21.15.28:0/125212766 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f803c990 con 0x7fe60015c9a0 2024-09-13T17:10:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe5f6000640 1 -- 172.21.15.28:0/125212766 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f803cd00 con 0x7fe60015c9a0 2024-09-13T17:10:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.646+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe600070930 con 0x7fe60015c9a0 2024-09-13T17:10:49.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.647+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6000cf010 con 0x7fe60015c9a0 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.649+0000 7fe5f6000640 1 -- 172.21.15.28:0/125212766 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe5f8031400 con 0x7fe60015c9a0 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.649+0000 7fe5f6000640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 0x7fe5e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.650+0000 7fe5f6000640 1 -- 172.21.15.28:0/125212766 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe5f80aa730 con 0x7fe60015c9a0 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.650+0000 7fe5fde00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 0x7fe5e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.650+0000 7fe6079c3740 1 Processor -- start 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.650+0000 7fe6079c3740 1 -- start start 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe5fde00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 0x7fe5e006d0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009aa0 tx=0x7fe5f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e0c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 0x56337e10e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337dfcc000 con 0x56337e0e8830 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337e029d40 con 0x56337e0c1380 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56337e0274d0 con 0x56337e0beb40 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe5fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe5fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53474/0 (socket says 172.21.15.28:53474) 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.651+0000 7fe5fe800640 1 -- 172.21.15.28:0/47236646 learned_addr learned my addr 172.21.15.28:0/47236646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fde00640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e0c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fd400640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 0x56337e10e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 -- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 msgr2=0x56337e10e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 0x56337e10e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 -- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 msgr2=0x56337e0c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e0c1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 -- 172.21.15.28:0/47236646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56337dfde3a0 con 0x56337e0e8830 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fde00640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e0c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fd400640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 0x56337e10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5fe800640 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8079670 tx=0x7fe5f8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5bbe170acaec545 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5eaa00640 1 -- 172.21.15.28:0/47236646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8081a20 con 0x56337e0e8830 2024-09-13T17:10:49.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5eaa00640 1 -- 172.21.15.28:0/47236646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8077a80 con 0x56337e0e8830 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.652+0000 7fe5eaa00640 1 -- 172.21.15.28:0/47236646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8075df0 con 0x56337e0e8830 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- 172.21.15.28:0/47236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 msgr2=0x56337e10ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8079670 tx=0x7fe5f8009750 comp rx=0 tx=0).stop 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- 172.21.15.28:0/47236646 shutdown_connections 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10ed90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0beb40 0x56337e10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 --2- 172.21.15.28:0/47236646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e0c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- 172.21.15.28:0/47236646 >> 172.21.15.28:0/47236646 conn(0x56337e0bdab0 msgr2=0x56337e105f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- 172.21.15.28:0/47236646 shutdown_connections 2024-09-13T17:10:49.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- 172.21.15.28:0/47236646 wait complete. 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.653+0000 7fe6079c3740 1 -- start start 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.654+0000 7fe6079c3740 1 Processor -- start 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e10dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e10e2c0 0x56337e10b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337e022dc0 con 0x56337e0e8830 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337dfe13e0 con 0x56337e0c1380 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56337dfe1080 con 0x56337e10e2c0 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe6079c3740 10 client.0 _ll_register_callbacks cb 0x56337e01c430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53486/0 (socket says 172.21.15.28:53486) 2024-09-13T17:10:49.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 -- 172.21.15.28:0/1849561865 learned_addr learned my addr 172.21.15.28:0/1849561865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e10dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fe800640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e10e2c0 0x56337e10b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 -- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e10e2c0 msgr2=0x56337e10b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e10e2c0 0x56337e10b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 -- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 msgr2=0x56337e10dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e10dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.655+0000 7fe5fd400640 1 -- 172.21.15.28:0/1849561865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6000cf010 con 0x56337e0e8830 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007420 tx=0x7fe5ec007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d78ad8745e53d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec001c00 con 0x56337e0e8830 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe5ec014400 con 0x56337e0e8830 2024-09-13T17:10:49.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56337e04c600 con 0x56337e0e8830 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5ec001d60 con 0x56337e0e8830 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe6079c3740 10 client.5385 fetch_fsmap learned FSMap version 27 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x56337e021440 con 0x56337e0e8830 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.656+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56337e04c600 con 0x56337e0e8830 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe5ec0145c0 con 0x56337e0e8830 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 10 client.5385 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 10 client.5385 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 20 client.5385 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e0e6f00 0x56337e0e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 4 client.5385 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56337e014c90 con 0x56337e0e6f00 2024-09-13T17:10:49.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e0e6f00 0x56337e0e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e0e6f00 0x56337e0e72e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d683ba236a57654a server_cookie=d5730cd312d5196d in_seq=0 out_seq=0 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.657+0000 7fe5df400640 10 client.5385 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe5df400640 1 -- 172.21.15.28:0/1849561865 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56337e014c90 con 0x56337e0e6f00 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe5df400640 10 client.5385 handle_command_reply: tid=0 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 client.5385 shutdown 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 msgr2=0x56337e10b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007420 tx=0x7fe5ec007820 comp rx=0 tx=0).stop 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 shutdown_connections 2024-09-13T17:10:49.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e0e6f00 0x56337e0e72e0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e10e2c0 0x56337e10b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e10b1b0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1849561865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e0c1380 0x56337e10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe5df400640 0 client.5385 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.658+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 >> 172.21.15.28:0/1849561865 conn(0x56337e0beb40 msgr2=0x56337e1042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.659+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 shutdown_connections 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.659+0000 7fe6079c3740 1 -- 172.21.15.28:0/1849561865 wait complete. 2024-09-13T17:10:49.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.659+0000 7fe6079c3740 20 client.5385 trim_cache size 0 max 16384 2024-09-13T17:10:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.673+0000 7fe6079c3740 1 Processor -- start 2024-09-13T17:10:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.673+0000 7fe6079c3740 1 -- start start 2024-09-13T17:10:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 0x56337e0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0bb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337dd8a040 con 0x56337e0e8830 2024-09-13T17:10:49.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337e108fa0 con 0x56337e10e2c0 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56337e0e5690 con 0x56337e0c1380 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53500/0 (socket says 172.21.15.28:53500) 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 -- 172.21.15.28:0/3508417695 learned_addr learned my addr 172.21.15.28:0/3508417695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fde00640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 0x56337e0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fe800640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0bb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 -- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 msgr2=0x56337e0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 0x56337e0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 -- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 msgr2=0x56337e0bb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0bb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fd400640 1 -- 172.21.15.28:0/3508417695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56337dfc35d0 con 0x56337e0e8830 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fde00640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 0x56337e0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.674+0000 7fe5fe800640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0bb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe5fd400640 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007ab0 tx=0x7fe5ec00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77555e8f2e1286a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe5eb400640 1 -- 172.21.15.28:0/3508417695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec007030 con 0x56337e0e8830 2024-09-13T17:10:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe5eb400640 1 -- 172.21.15.28:0/3508417695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec014b70 con 0x56337e0e8830 2024-09-13T17:10:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe5eb400640 1 -- 172.21.15.28:0/3508417695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5ec001c00 con 0x56337e0e8830 2024-09-13T17:10:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 -- 172.21.15.28:0/3508417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 msgr2=0x56337e1041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe5ec007ab0 tx=0x7fe5ec00e000 comp rx=0 tx=0).stop 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 -- 172.21.15.28:0/3508417695 shutdown_connections 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0bb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0e8830 0x56337e1041a0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 --2- 172.21.15.28:0/3508417695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0c1380 0x56337e0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.675+0000 7fe6079c3740 1 -- 172.21.15.28:0/3508417695 >> 172.21.15.28:0/3508417695 conn(0x56337e0beb40 msgr2=0x56337e10b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.676+0000 7fe6079c3740 1 -- 172.21.15.28:0/3508417695 shutdown_connections 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.676+0000 7fe6079c3740 1 -- 172.21.15.28:0/3508417695 wait complete. 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.676+0000 7fe6079c3740 1 -- start start 2024-09-13T17:10:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.676+0000 7fe6079c3740 1 Processor -- start 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0e8830 0x56337e158250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0c0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337e0bfab0 con 0x56337e0c1380 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56337e0ecc00 con 0x56337e10e2c0 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56337e121280 con 0x56337e0e8830 2024-09-13T17:10:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.677+0000 7fe6079c3740 10 client.0 _ll_register_callbacks cb 0x7fe5ec005990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53504/0 (socket says 172.21.15.28:53504) 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 -- 172.21.15.28:0/1054738713 learned_addr learned my addr 172.21.15.28:0/1054738713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0e8830 0x56337e158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 -- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0e8830 msgr2=0x56337e158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fe800640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0c0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0e8830 0x56337e158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 -- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 msgr2=0x56337e0c0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0c0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 -- 172.21.15.28:0/1054738713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56337e021440 con 0x56337e0c1380 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5fde00640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0011690 tx=0x7fe5f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5b3a29eb262428 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.678+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f002a070 con 0x56337e0c1380 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe5f0003ca0 con 0x56337e0c1380 2024-09-13T17:10:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5f8077a80 con 0x56337e0c1380 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5f0003e60 con 0x56337e0c1380 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 10 client.5391 fetch_fsmap learned FSMap version 27 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=2+}) v3 -- 0x56337dfe89b0 con 0x56337e0c1380 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8077a80 con 0x56337e0c1380 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 27) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe5f0005990 con 0x56337e0c1380 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 10 client.5391 fetch_fsmap finished waiting for FSMap version 27 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 10 client.5391 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.c 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 20 client.5391 populate_metadata read hostname 'smithi028' 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e158a00 0x56337e0bf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 4 client.5391 mds_command: new command op to 5121 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 --> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x56337dd164e0 con 0x56337e158a00 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.679+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e158a00 0x56337e0bf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.680+0000 7fe5df400640 10 client.5391 ms_handle_connect on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.680+0000 7fe5fd400640 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e158a00 0x56337e0bf050 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f77085db1b099240 server_cookie=8ad7976f74d22eb1 in_seq=0 out_seq=0 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.680+0000 7fe5df400640 1 -- 172.21.15.28:0/1054738713 <== mds.0 v2:172.21.15.28:6833/696058040 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17929 (crc 0 0 0) 0x56337dd164e0 con 0x56337e158a00 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.680+0000 7fe5df400640 10 client.5391 handle_command_reply: tid=0 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.680+0000 7fe6079c3740 1 client.5391 shutdown 2024-09-13T17:10:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 msgr2=0x56337e0c24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0011690 tx=0x7fe5f0004730 comp rx=0 tx=0).stop 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 shutdown_connections 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:6833/696058040,v1:172.21.15.28:6835/696058040] conn(0x56337e158a00 0x56337e0bf050 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56337e10e2c0 0x56337e0c0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56337e0e8830 0x56337e158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 --2- 172.21.15.28:0/1054738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56337e0c1380 0x56337e0c24b0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe5df400640 0 client.5391 ms_handle_reset on v2:172.21.15.28:6833/696058040 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 >> 172.21.15.28:0/1054738713 conn(0x56337e0beb40 msgr2=0x56337e0bbe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 shutdown_connections 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 1 -- 172.21.15.28:0/1054738713 wait complete. 2024-09-13T17:10:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.681+0000 7fe6079c3740 20 client.5391 trim_cache size 0 max 16384 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 5182, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 5183, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1492991, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 48424395, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 17, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.712307712, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.377081116, 2024-09-13T17:10:49.691 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.115325804, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.170869131, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5181, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.417754732, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001238709 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 10, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000957, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000095 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-13T17:10:49.692 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 30744, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 51483, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 11061508, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 46749650, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 5.036656609, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 2.941401306, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 2.630011725, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.108993175, 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:49.693 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 51483, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.436501719, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000047326 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 70683, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017219974, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000243 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 134192, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 27680 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 5294, 2024-09-13T17:10:49.694 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 5318, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1893871, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 42021514, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.571419655, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.227457954, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.113569143, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.183633632, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5317, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.444980929, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000271766 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.695 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 91328 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-13T17:10:49.696 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5793, 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.646680578, 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002873585 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.697 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3681, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.964305160, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000261968 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 25668, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 25668, 2024-09-13T17:10:49.698 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 25668, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 303.963110858, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011842103 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 53, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-13T17:10:49.699 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 14690, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 9877, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 4813, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 492, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 2, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 2, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 2, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 47718, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 47148, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-13T17:10:49.700 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 169, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 1857, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-13T17:10:49.701 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 1857, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 13, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 2, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 11, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 1844, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 1034, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 4038, 2024-09-13T17:10:49.702 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 4203, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 2000, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 28, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 14028 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-13T17:10:49.703 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 4142, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 3689, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 2249, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-13T17:10:49.704 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 1, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 31145, 2024-09-13T17:10:49.705 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 30392, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 30392, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 2801, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 2048, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 47, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 46, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 46, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 3, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 2, 2024-09-13T17:10:49.706 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 100181593, 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 100636329, 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4196, 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 94.306770263, 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.022475398 2024-09-13T17:10:49.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 14, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 2016, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 2002, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 13, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 13, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 14690, 2024-09-13T17:10:49.708 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 19966, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 5276, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 2, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 3604, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 3602, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 94948, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 207200 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.709 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 29153, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 25668, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 6, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2000, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.428036337, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000714018 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.710 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 24, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.005078376, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000211599 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6000, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.911972892, 2024-09-13T17:10:49.711 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001151995 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7382, 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.471541636, 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000876665 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-13T17:10:49.712 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-13T17:10:49.713 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-13T17:10:49.714 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000053613, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000053613 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:49.715 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002543020, 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002543020 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.716 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-13T17:10:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006798440, 2024-09-13T17:10:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006798440 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.552415877, 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.069051984 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.719 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-13T17:10:49.720 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2249, 2024-09-13T17:10:49.720 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 271.609319740, 2024-09-13T17:10:49.720 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.120768928 2024-09-13T17:10:49.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.721 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-13T17:10:49.721 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.721 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.722 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.722 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.722 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-13T17:10:49.722 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.723 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.004082261, 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.004082261 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.724 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000378026, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000378026 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.725 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-13T17:10:49.726 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8000, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.970890640, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002121361 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.727 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-13T17:10:49.728 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 18076, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 18076, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 90, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-13T17:10:49.729 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-13T17:10:49.730 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-13T17:10:49.731 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 250670, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 92, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 440, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 5, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-13T17:10:49.732 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 5416, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 77, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 11110132, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 590689, 2024-09-13T17:10:49.733 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 13317, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 107078948, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 37, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 13317, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 13317, 2024-09-13T17:10:49.734 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 128.280401850, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.009632830 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 13317, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 20064 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 13317, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 1903, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 11414, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-13T17:10:49.735 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 40, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 3308, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 44, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 4238, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 45, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 1689, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 2056, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-13T17:10:49.736 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 1859, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 6616, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-13T17:10:49.737 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-13T17:10:49.738 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 36, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 3, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 8, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 8, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 123, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 46 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-13T17:10:49.739 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 32, 2024-09-13T17:10:49.740 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 32, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 3689, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 3689, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 41216, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 10732874, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 41216, 2024-09-13T17:10:49.741 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 41216, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 10732874, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 13317, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 107465049, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 13317, 2024-09-13T17:10:49.743 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 107465049, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:10:49.744 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 13317, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 13317, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 13317, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 13317, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.745 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 3689, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 386101, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 3689, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 42, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 386101, 2024-09-13T17:10:49.746 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x555b9c4640a0": { 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 31145, 2024-09-13T17:10:49.747 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 96442025, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 31145, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4196, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 96442025, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:10:49.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.749 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:10:49.749 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:10:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 msgr2=0x7fe5e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 0x7fe5e006d0c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe5f0009aa0 tx=0x7fe5f0009190 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 msgr2=0x7fe6000cead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe5f8009ee0 tx=0x7fe5f8004a40 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 shutdown_connections 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe5e006ac00 0x7fe5e006d0c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe60015c9a0 0x7fe6000cead0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe6000b2400 0x7fe6001f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 --2- 172.21.15.28:0/125212766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6000ad940 0x7fe60015f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 >> 172.21.15.28:0/125212766 conn(0x7fe6000b0270 msgr2=0x7fe60006cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 shutdown_connections 2024-09-13T17:10:49.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:49.682+0000 7fe604c00640 1 -- 172.21.15.28:0/125212766 wait complete. 2024-09-13T17:10:49.752 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:10:49.753 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:10:49.842 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:10:49.926 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:10:49.926 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:10:49.926 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:10:49.962 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:10:49.962 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:10:50.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:10:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.231+0000 7fc217200640 1 Processor -- start 2024-09-13T17:10:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.232+0000 7fc217200640 1 -- start start 2024-09-13T17:10:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc217200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc218154a90 0x7fc21815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc217200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2180b0b50 0x7fc2180af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc217200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc21815bb10 con 0x7fc21806b360 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc217200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21806b900 con 0x7fc218154a90 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc217200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc21806bbe0 con 0x7fc2180b0b50 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc215e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc218154a90 0x7fc21815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc215e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc218154a90 0x7fc21815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53512/0 (socket says 172.21.15.28:53512) 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.233+0000 7fc215e00640 1 -- 172.21.15.28:0/666117956 learned_addr learned my addr 172.21.15.28:0/666117956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc216800640 1 --2- 172.21.15.28:0/666117956 >> v2:172.21.15.189:3300/0 conn(0x7fc2180b0b50 0x7fc2180af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc215e00640 1 -- 172.21.15.28:0/666117956 >> v2:172.21.15.189:3300/0 conn(0x7fc2180b0b50 msgr2=0x7fc2180af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc215e00640 1 --2- 172.21.15.28:0/666117956 >> v2:172.21.15.189:3300/0 conn(0x7fc2180b0b50 0x7fc2180af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc215e00640 1 -- 172.21.15.28:0/666117956 >> v1:172.21.15.28:6789/0 conn(0x7fc21806b360 legacy=0x7fc2180b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc215e00640 1 -- 172.21.15.28:0/666117956 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc21806bec0 con 0x7fc218154a90 2024-09-13T17:10:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.234+0000 7fc215400640 1 -- 172.21.15.28:0/666117956 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606945518 0 0) 0x7fc21815bb10 con 0x7fc21806b360 2024-09-13T17:10:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc215e00640 1 --2- 172.21.15.28:0/666117956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc21815b5d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc2180b14d0 tx=0x7fc20802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6911dbcc02a4a0fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc215400640 1 -- 172.21.15.28:0/666117956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208032440 con 0x7fc218154a90 2024-09-13T17:10:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc215400640 1 -- 172.21.15.28:0/666117956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2080325a0 con 0x7fc218154a90 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc215400640 1 -- 172.21.15.28:0/666117956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc208032930 con 0x7fc218154a90 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc217200640 1 -- 172.21.15.28:0/666117956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 msgr2=0x7fc21815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.235+0000 7fc217200640 1 --2- 172.21.15.28:0/666117956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc21815b5d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fc2180b14d0 tx=0x7fc20802a270 comp rx=0 tx=0).stop 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 -- 172.21.15.28:0/666117956 shutdown_connections 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 --2- 172.21.15.28:0/666117956 >> v2:172.21.15.189:3300/0 conn(0x7fc2180b0b50 0x7fc2180af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 --2- 172.21.15.28:0/666117956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc21815b5d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 -- 172.21.15.28:0/666117956 >> 172.21.15.28:0/666117956 conn(0x7fc218072e40 msgr2=0x7fc218073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 -- 172.21.15.28:0/666117956 shutdown_connections 2024-09-13T17:10:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.236+0000 7fc217200640 1 -- 172.21.15.28:0/666117956 wait complete. 2024-09-13T17:10:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.237+0000 7fc217200640 1 Processor -- start 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.237+0000 7fc217200640 1 -- start start 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 0x7fc218153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2180b0b50 0x7fc2180adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc218154000 con 0x7fc218154a90 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180ad960 con 0x7fc21806b360 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc217200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180acd50 con 0x7fc2180b0b50 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.238+0000 7fc216800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53516/0 (socket says 172.21.15.28:53516) 2024-09-13T17:10:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 -- 172.21.15.28:0/888748466 learned_addr learned my addr 172.21.15.28:0/888748466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc215e00640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 0x7fc218153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 -- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2180b0b50 msgr2=0x7fc2180adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2180b0b50 0x7fc2180adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 -- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 msgr2=0x7fc218153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 0x7fc218153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 -- 172.21.15.28:0/888748466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2080094d0 con 0x7fc218154a90 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc215e00640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 0x7fc218153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc216800640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc210007d90 tx=0x7fc21000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210014420 con 0x7fc218154a90 2024-09-13T17:10:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2100145a0 con 0x7fc218154a90 2024-09-13T17:10:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc210014950 con 0x7fc218154a90 2024-09-13T17:10:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.239+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2180ad030 con 0x7fc218154a90 2024-09-13T17:10:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.240+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2180ad5f0 con 0x7fc218154a90 2024-09-13T17:10:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.241+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21815a740 con 0x7fc218154a90 2024-09-13T17:10:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.242+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc210014ab0 con 0x7fc218154a90 2024-09-13T17:10:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.242+0000 7fc20e000640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 0x7fc1f406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.242+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc210089ca0 con 0x7fc218154a90 2024-09-13T17:10:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.243+0000 7fc215e00640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 0x7fc1f406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.244+0000 7fc215e00640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 0x7fc1f406d0f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc208034000 tx=0x7fc20802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.247+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc210054e50 con 0x7fc218154a90 2024-09-13T17:10:50.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.426+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2180af4b0 con 0x7fc218154a90 2024-09-13T17:10:50.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.427+0000 7fc20e000640 1 -- 172.21.15.28:0/888748466 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7fc210058470 con 0x7fc218154a90 2024-09-13T17:10:50.429 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:50.429 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:10:25.911635+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 msgr2=0x7fc1f406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 0x7fc1f406d0f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc208034000 tx=0x7fc20802e000 comp rx=0 tx=0).stop 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 msgr2=0x7fc2180ae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fc210007d90 tx=0x7fc21000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 shutdown_connections 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1f406ac30 0x7fc1f406d0f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc218154a90 0x7fc2180ae400 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2180b0b50 0x7fc2180adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 --2- 172.21.15.28:0/888748466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc21806b360 0x7fc218153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 >> 172.21.15.28:0/888748466 conn(0x7fc218072e40 msgr2=0x7fc218159830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 shutdown_connections 2024-09-13T17:10:50.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.430+0000 7fc217200640 1 -- 172.21.15.28:0/888748466 wait complete. 2024-09-13T17:10:50.446 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:09:09.494172+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:09:10.074208+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:10:50.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:10:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.604+0000 7f4411200640 1 Processor -- start 2024-09-13T17:10:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.604+0000 7f4411200640 1 -- start start 2024-09-13T17:10:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.605+0000 7f4411200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f440c15a780 0x7f440c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.605+0000 7f4411200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15ab60 0x7f440c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.605+0000 7f4411200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440c06c8b0 con 0x7f440c06bb40 2024-09-13T17:10:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.605+0000 7f4411200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440c06cb90 con 0x7f440c15a780 2024-09-13T17:10:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.605+0000 7f4411200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0cc980 con 0x7f440c15ab60 2024-09-13T17:10:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f440c15a780 0x7f440c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f440c15a780 0x7f440c070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53522/0 (socket says 172.21.15.28:53522) 2024-09-13T17:10:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 -- 172.21.15.28:0/682287110 learned_addr learned my addr 172.21.15.28:0/682287110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f440a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f440c06bb40 0x7f440c06bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44750/0 (socket says 172.21.15.28:44750) 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409400640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15ab60 0x7f440c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 -- 172.21.15.28:0/682287110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15ab60 msgr2=0x7f440c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15ab60 0x7f440c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 -- 172.21.15.28:0/682287110 >> v1:172.21.15.28:6789/0 conn(0x7f440c06bb40 legacy=0x7f440c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4409e00640 1 -- 172.21.15.28:0/682287110 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c0ccc60 con 0x7f440c15a780 2024-09-13T17:10:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.606+0000 7f4408a00640 1 -- 172.21.15.28:0/682287110 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806113550 0 0) 0x7f440c06c8b0 con 0x7f440c06bb40 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.607+0000 7f4409e00640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c15a780 0x7f440c070d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009870 tx=0x7f43fc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9582a1b1120c58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.607+0000 7f4408a00640 1 -- 172.21.15.28:0/682287110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc032440 con 0x7f440c15a780 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.607+0000 7f4408a00640 1 -- 172.21.15.28:0/682287110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc0325a0 con 0x7f440c15a780 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.607+0000 7f4408a00640 1 -- 172.21.15.28:0/682287110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc032930 con 0x7f440c15a780 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 -- 172.21.15.28:0/682287110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c15a780 msgr2=0x7f440c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c15a780 0x7f440c070d80 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009870 tx=0x7f43fc02a250 comp rx=0 tx=0).stop 2024-09-13T17:10:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 -- 172.21.15.28:0/682287110 shutdown_connections 2024-09-13T17:10:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15ab60 0x7f440c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 --2- 172.21.15.28:0/682287110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c15a780 0x7f440c070d80 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 -- 172.21.15.28:0/682287110 >> 172.21.15.28:0/682287110 conn(0x7f440c0b0270 msgr2=0x7f440c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 -- 172.21.15.28:0/682287110 shutdown_connections 2024-09-13T17:10:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.608+0000 7f4411200640 1 -- 172.21.15.28:0/682287110 wait complete. 2024-09-13T17:10:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.610+0000 7f4411200640 1 Processor -- start 2024-09-13T17:10:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.610+0000 7f4411200640 1 -- start start 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15a780 0x7f440c1567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f440c15ab60 0x7f440c1f6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0a8440 con 0x7f440c06bb40 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0a86e0 con 0x7f440c15ab60 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4411200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440c156d30 con 0x7f440c15a780 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53524/0 (socket says 172.21.15.28:53524) 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 -- 172.21.15.28:0/2401597761 learned_addr learned my addr 172.21.15.28:0/2401597761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f440a800640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f440c15ab60 0x7f440c1f6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409400640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15a780 0x7f440c1567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 -- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15a780 msgr2=0x7f440c1567f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15a780 0x7f440c1567f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 -- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f440c15ab60 msgr2=0x7f440c1f6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f440c15ab60 0x7f440c1f6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.611+0000 7f4409e00640 1 -- 172.21.15.28:0/2401597761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0094d0 con 0x7f440c06bb40 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4409e00640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc032440 con 0x7f440c06bb40 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc030b80 con 0x7f440c06bb40 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c1f70c0 con 0x7f440c06bb40 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc03d440 con 0x7f440c06bb40 2024-09-13T17:10:50.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.612+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c1f7680 con 0x7f440c06bb40 2024-09-13T17:10:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.613+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c06c8b0 con 0x7f440c06bb40 2024-09-13T17:10:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.614+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f43fc039040 con 0x7f440c06bb40 2024-09-13T17:10:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.615+0000 7f4402000640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 0x7f43ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.615+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f43fc0a9e70 con 0x7f440c06bb40 2024-09-13T17:10:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.615+0000 7f4409400640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 0x7f43ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.616+0000 7f4409400640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 0x7f43ec06d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f43f8006fd0 tx=0x7f43f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.619+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f43fc075020 con 0x7f440c06bb40 2024-09-13T17:10:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.797+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f440c1f7810 con 0x7f440c06bb40 2024-09-13T17:10:50.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.797+0000 7f4402000640 1 -- 172.21.15.28:0/2401597761 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43fc078640 con 0x7f440c06bb40 2024-09-13T17:10:50.798 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:10:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 msgr2=0x7f43ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 0x7f43ec06d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f43f8006fd0 tx=0x7f43f8008040 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 msgr2=0x7f440c0c5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 shutdown_connections 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f43ec06ac00 0x7f43ec06d0c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f440c15ab60 0x7f440c1f6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f440c15a780 0x7f440c1567f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 --2- 172.21.15.28:0/2401597761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f440c06bb40 0x7f440c0c5070 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 >> 172.21.15.28:0/2401597761 conn(0x7f440c0b0270 msgr2=0x7f440c0ce560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 shutdown_connections 2024-09-13T17:10:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.799+0000 7f4411200640 1 -- 172.21.15.28:0/2401597761 wait complete. 2024-09-13T17:10:50.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:10:50.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.973+0000 7f9058a00640 1 Processor -- start 2024-09-13T17:10:50.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.973+0000 7f9058a00640 1 -- start start 2024-09-13T17:10:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9054070e30 0x7f9054071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 0x7f90540cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054071210 0x7f90540cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9054159e10 con 0x7f9054070e30 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f905415a0f0 con 0x7f9054072140 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9058a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905415a3d0 con 0x7f9054071210 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9054070e30 0x7f9054071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9054070e30 0x7f9054071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53534/0 (socket says 172.21.15.28:53534) 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 -- 172.21.15.28:0/2374565228 learned_addr learned my addr 172.21.15.28:0/2374565228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9053200640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054071210 0x7f90540cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f904be00640 1 --2- 172.21.15.28:0/2374565228 >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 0x7f90540cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 -- 172.21.15.28:0/2374565228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054071210 msgr2=0x7f90540cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054071210 0x7f90540cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 -- 172.21.15.28:0/2374565228 >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 msgr2=0x7f90540cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 --2- 172.21.15.28:0/2374565228 >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 0x7f90540cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f9052800640 1 -- 172.21.15.28:0/2374565228 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f905415a6b0 con 0x7f9054070e30 2024-09-13T17:10:50.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.975+0000 7f904be00640 1 --2- 172.21.15.28:0/2374565228 >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 0x7f90540cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:50.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.976+0000 7f9052800640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054070e30 0x7f9054071c00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f90540b08d0 tx=0x7f904402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad7d5cc94cdcc93 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9051e00640 1 -- 172.21.15.28:0/2374565228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044032440 con 0x7f9054070e30 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9051e00640 1 -- 172.21.15.28:0/2374565228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90440325a0 con 0x7f9054070e30 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9051e00640 1 -- 172.21.15.28:0/2374565228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9044032930 con 0x7f9054070e30 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 -- 172.21.15.28:0/2374565228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054070e30 msgr2=0x7f9054071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054070e30 0x7f9054071c00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f90540b08d0 tx=0x7f904402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 -- 172.21.15.28:0/2374565228 shutdown_connections 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054071210 0x7f90540cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 --2- 172.21.15.28:0/2374565228 >> v2:172.21.15.189:3300/0 conn(0x7f9054072140 0x7f90540cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 --2- 172.21.15.28:0/2374565228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054070e30 0x7f9054071c00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 -- 172.21.15.28:0/2374565228 >> 172.21.15.28:0/2374565228 conn(0x7f9054069e80 msgr2=0x7f905406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:50.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 -- 172.21.15.28:0/2374565228 shutdown_connections 2024-09-13T17:10:50.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.977+0000 7f9058a00640 1 -- 172.21.15.28:0/2374565228 wait complete. 2024-09-13T17:10:50.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.979+0000 7f9058a00640 1 Processor -- start 2024-09-13T17:10:50.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.979+0000 7f9058a00640 1 -- start start 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054070e30 0x7f905414a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9054071210 0x7f90540b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905414aee0 con 0x7f9054072140 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f905414b1c0 con 0x7f9054071210 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.980+0000 7f9058a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f905414b4a0 con 0x7f9054070e30 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53540/0 (socket says 172.21.15.28:53540) 2024-09-13T17:10:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 -- 172.21.15.28:0/4003777099 learned_addr learned my addr 172.21.15.28:0/4003777099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f904be00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9054071210 0x7f90540b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9052800640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054070e30 0x7f905414a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 -- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054070e30 msgr2=0x7f905414a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054070e30 0x7f905414a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 -- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9054071210 msgr2=0x7f90540b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9054071210 0x7f90540b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:50.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 -- 172.21.15.28:0/4003777099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90440094d0 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.981+0000 7f9053200640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f904c00eed0 tx=0x7f904c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.982+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c010040 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.982+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f904c014400 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.982+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f904c009ea0 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.982+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f905414b780 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.982+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90540b23a0 con 0x7f9054072140 2024-09-13T17:10:50.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.983+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f905414bca0 con 0x7f9054072140 2024-09-13T17:10:50.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.984+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f904c0147b0 con 0x7f9054072140 2024-09-13T17:10:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.985+0000 7f904aa00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 0x7f903006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.985+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f904c089a90 con 0x7f9054072140 2024-09-13T17:10:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.985+0000 7f9052800640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 0x7f903006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.986+0000 7f9052800640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 0x7f903006d0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9044034000 tx=0x7f904402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:50.990+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f905414bca0 con 0x7f9054072140 2024-09-13T17:10:51.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.169+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f905414bca0 con 0x7f9054072140 2024-09-13T17:10:51.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.169+0000 7f904aa00640 1 -- 172.21.15.28:0/4003777099 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7f904c018020 con 0x7f9054072140 2024-09-13T17:10:51.170 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:10:51.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 msgr2=0x7f903006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 0x7f903006d0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9044034000 tx=0x7f904402e000 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 msgr2=0x7f90541f2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f904c00eed0 tx=0x7f904c00b480 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 shutdown_connections 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f903006ac00 0x7f903006d0c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9054072140 0x7f90541f2180 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9054071210 0x7f90540b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 --2- 172.21.15.28:0/4003777099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9054070e30 0x7f905414a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.171+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 >> 172.21.15.28:0/4003777099 conn(0x7f9054069e80 msgr2=0x7f90540cc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.172+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 shutdown_connections 2024-09-13T17:10:51.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.172+0000 7f9058a00640 1 -- 172.21.15.28:0/4003777099 wait complete. 2024-09-13T17:10:51.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:10:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.346+0000 7fd90e600640 1 Processor -- start 2024-09-13T17:10:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.346+0000 7fd90e600640 1 -- start start 2024-09-13T17:10:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd90815a780 0x7fd908070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815ab60 0x7fd9080712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd90806bb40 0x7fd90806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90806c8b0 con 0x7fd90806bb40 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90806cb90 con 0x7fd90815a780 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd90e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080cc980 con 0x7fd90815ab60 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd907e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd90806bb40 0x7fd90806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd907e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd90806bb40 0x7fd90806bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53546/0 (socket says 172.21.15.28:53546) 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.347+0000 7fd907e00640 1 -- 172.21.15.28:0/3171068125 learned_addr learned my addr 172.21.15.28:0/3171068125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd8fea00640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815ab60 0x7fd9080712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907400640 1 --2- 172.21.15.28:0/3171068125 >> v2:172.21.15.189:3300/0 conn(0x7fd90815a780 0x7fd908070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907e00640 1 -- 172.21.15.28:0/3171068125 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815ab60 msgr2=0x7fd9080712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907e00640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815ab60 0x7fd9080712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907e00640 1 -- 172.21.15.28:0/3171068125 >> v2:172.21.15.189:3300/0 conn(0x7fd90815a780 msgr2=0x7fd908070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907e00640 1 --2- 172.21.15.28:0/3171068125 >> v2:172.21.15.189:3300/0 conn(0x7fd90815a780 0x7fd908070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.348+0000 7fd907e00640 1 -- 172.21.15.28:0/3171068125 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080ccc60 con 0x7fd90806bb40 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.349+0000 7fd907e00640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90806bb40 0x7fd90806bfd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd9080b1330 tx=0x7fd90002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53c482bdc5e79e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.349+0000 7fd906a00640 1 -- 172.21.15.28:0/3171068125 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900031600 con 0x7fd90806bb40 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.349+0000 7fd906a00640 1 -- 172.21.15.28:0/3171068125 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90002d050 con 0x7fd90806bb40 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.349+0000 7fd906a00640 1 -- 172.21.15.28:0/3171068125 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900031980 con 0x7fd90806bb40 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 -- 172.21.15.28:0/3171068125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90806bb40 msgr2=0x7fd90806bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90806bb40 0x7fd90806bfd0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd9080b1330 tx=0x7fd90002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 -- 172.21.15.28:0/3171068125 shutdown_connections 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90806bb40 0x7fd90806bfd0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 --2- 172.21.15.28:0/3171068125 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815ab60 0x7fd9080712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 --2- 172.21.15.28:0/3171068125 >> v2:172.21.15.189:3300/0 conn(0x7fd90815a780 0x7fd908070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 -- 172.21.15.28:0/3171068125 >> 172.21.15.28:0/3171068125 conn(0x7fd9080b0270 msgr2=0x7fd9080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 -- 172.21.15.28:0/3171068125 shutdown_connections 2024-09-13T17:10:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.350+0000 7fd90e600640 1 -- 172.21.15.28:0/3171068125 wait complete. 2024-09-13T17:10:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.351+0000 7fd90e600640 1 Processor -- start 2024-09-13T17:10:51.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.352+0000 7fd90e600640 1 -- start start 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.352+0000 7fd90e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd90806bb40 0x7fd908142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd90e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815a780 0x7fd9080ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd90e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd90e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908142d40 con 0x7fd90815ab60 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd90e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd908142fa0 con 0x7fd90806bb40 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd90e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9080c9d60 con 0x7fd90815a780 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd90806bb40 0x7fd908142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53556/0 (socket says 172.21.15.28:53556) 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 -- 172.21.15.28:0/1534485278 learned_addr learned my addr 172.21.15.28:0/1534485278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd8fea00640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815a780 0x7fd9080ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 -- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815a780 msgr2=0x7fd9080ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815a780 0x7fd9080ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 -- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd90806bb40 msgr2=0x7fd908142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd90806bb40 0x7fd908142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.353+0000 7fd907e00640 1 -- 172.21.15.28:0/1534485278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009b30 con 0x7fd90815ab60 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd907e00640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd900034040 tx=0x7fd900004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90003d860 con 0x7fd90815ab60 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90003d9e0 con 0x7fd90815ab60 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9080ca040 con 0x7fd90815ab60 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90003dd90 con 0x7fd90815ab60 2024-09-13T17:10:51.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.354+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9080ca600 con 0x7fd90815ab60 2024-09-13T17:10:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.355+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd908143150 con 0x7fd90815ab60 2024-09-13T17:10:51.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.356+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd90003b400 con 0x7fd90815ab60 2024-09-13T17:10:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.357+0000 7fd8ffe00640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 0x7fd8e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.358+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd9000a9ed0 con 0x7fd90815ab60 2024-09-13T17:10:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.358+0000 7fd907400640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 0x7fd8e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.359+0000 7fd907400640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 0x7fd8e406d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8006fd0 tx=0x7fd8f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.362+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd908143150 con 0x7fd90815ab60 2024-09-13T17:10:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.542+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd908143110 con 0x7fd90815ab60 2024-09-13T17:10:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.542+0000 7fd8ffe00640 1 -- 172.21.15.28:0/1534485278 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9000750a0 con 0x7fd90815ab60 2024-09-13T17:10:51.543 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 msgr2=0x7fd8e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 0x7fd8e406d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8006fd0 tx=0x7fd8f8008040 comp rx=0 tx=0).stop 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 msgr2=0x7fd9080ae7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fd900034040 tx=0x7fd900004a40 comp rx=0 tx=0).stop 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 shutdown_connections 2024-09-13T17:10:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8e406ac00 0x7fd8e406d0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd90815ab60 0x7fd9080ae7c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd90815a780 0x7fd9080ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 --2- 172.21.15.28:0/1534485278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd90806bb40 0x7fd908142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 >> 172.21.15.28:0/1534485278 conn(0x7fd9080b0270 msgr2=0x7fd9080b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 shutdown_connections 2024-09-13T17:10:51.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.544+0000 7fd90e600640 1 -- 172.21.15.28:0/1534485278 wait complete. 2024-09-13T17:10:51.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:10:51.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.716+0000 7f7cb6200640 1 Processor -- start 2024-09-13T17:10:51.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.716+0000 7f7cb6200640 1 -- start start 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7cb6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7cb006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7cb6200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7cb006bf40 0x7f7cb00b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7cb6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cb01598c0 con 0x7f7cb0158cd0 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7cb6200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cb006c3d0 con 0x7f7cb006bf40 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7cb6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0159ba0 con 0x7f7cb006ccd0 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7caf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7cb0158cd0 0x7f7cb01590b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44754/0 (socket says 172.21.15.28:44754) 2024-09-13T17:10:51.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.717+0000 7f7caf400640 1 -- 172.21.15.28:0/1017460912 learned_addr learned my addr 172.21.15.28:0/1017460912 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:51.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7cafe00640 1 --2- 172.21.15.28:0/1017460912 >> v2:172.21.15.189:3300/0 conn(0x7f7cb006bf40 0x7f7cb00b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7ca6a00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7cb006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695889066 0 0) 0x7f7cb01598c0 con 0x7f7cb0158cd0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c94003520 con 0x7f7cb0158cd0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 940860715 0 0) 0x7f7c94003520 con 0x7f7cb0158cd0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 msgr2=0x7f7cb006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7cb006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 >> v2:172.21.15.189:3300/0 conn(0x7f7cb006bf40 msgr2=0x7f7cb00b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 --2- 172.21.15.28:0/1017460912 >> v2:172.21.15.189:3300/0 conn(0x7f7cb006bf40 0x7f7cb00b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb0159e80 con 0x7f7cb0158cd0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7ca0001670 con 0x7f7cb0158cd0 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 >> v1:172.21.15.28:6789/0 conn(0x7f7cb0158cd0 legacy=0x7f7cb01590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7c94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c940047c0 0x7f7c94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb006c3d0 con 0x7f7c94003970 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb0159ba0 con 0x7f7c940047c0 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.718+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9400db90 con 0x7f7cb006ccd0 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7c94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7ca6a00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c940047c0 0x7f7c94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7cafe00640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 -- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 msgr2=0x7f7c94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb006ccd0 0x7f7c94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 -- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c940047c0 msgr2=0x7f7c94004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c940047c0 0x7f7c94004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.719+0000 7f7caf400640 1 -- 172.21.15.28:0/1017460912 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c94003520 con 0x7f7c94003970 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.720+0000 7f7caf400640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7c94003d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c240 tx=0x7f7c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8222f2650026113a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.720+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0005ad0 con 0x7f7c94003970 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.720+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0005c30 con 0x7f7c94003970 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.720+0000 7f7caea00640 1 -- 172.21.15.28:0/1017460912 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0007070 con 0x7f7c94003970 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 -- 172.21.15.28:0/1017460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 msgr2=0x7f7c94003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 --2- 172.21.15.28:0/1017460912 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7c94003d90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c240 tx=0x7f7c940057f0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cafe00640 1 -- 172.21.15.28:0/1017460912 reap_dead start 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 -- 172.21.15.28:0/1017460912 shutdown_connections 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 -- 172.21.15.28:0/1017460912 >> 172.21.15.28:0/1017460912 conn(0x7f7cb00b0270 msgr2=0x7f7cb00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 -- 172.21.15.28:0/1017460912 shutdown_connections 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.721+0000 7f7cb6200640 1 -- 172.21.15.28:0/1017460912 wait complete. 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.723+0000 7f7cb6200640 1 Processor -- start 2024-09-13T17:10:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.723+0000 7f7cb6200640 1 -- start start 2024-09-13T17:10:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb01f2390 0x7f7cb00ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cb00cd050 0x7f7cb00cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb00cd500 con 0x7f7c94003970 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb014c500 con 0x7f7cb00cd050 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cb6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cb01e4c20 con 0x7f7cb01f2390 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53584/0 (socket says 172.21.15.28:53584) 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 -- 172.21.15.28:0/3696682026 learned_addr learned my addr 172.21.15.28:0/3696682026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7cafe00640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cb00cd050 0x7f7cb00cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7ca6a00640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb01f2390 0x7f7cb00ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 -- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb01f2390 msgr2=0x7f7cb00ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb01f2390 0x7f7cb00ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 -- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cb00cd050 msgr2=0x7f7cb00cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cb00cd050 0x7f7cb00cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.724+0000 7f7caf400640 1 -- 172.21.15.28:0/3696682026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb01e4f00 con 0x7f7c94003970 2024-09-13T17:10:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7caf400640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c240 tx=0x7f7ca0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0007480 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0007600 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca003e690 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cb01e51e0 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.725+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cb00cbc30 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.726+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cb01e4d60 con 0x7f7c94003970 2024-09-13T17:10:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.727+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7ca003e7f0 con 0x7f7c94003970 2024-09-13T17:10:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.728+0000 7f7ca7e00640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 0x7f7c8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.728+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f7ca00aa430 con 0x7f7c94003970 2024-09-13T17:10:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.729+0000 7f7ca6a00640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 0x7f7c8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.729+0000 7f7ca6a00640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 0x7f7c8c06d0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7cb015b820 tx=0x7f7c98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.732+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7cb01e4d60 con 0x7f7c94003970 2024-09-13T17:10:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.922+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7cb01e4d60 con 0x7f7c94003970 2024-09-13T17:10:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.923+0000 7f7ca7e00640 1 -- 172.21.15.28:0/3696682026 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7cb01e4d60 con 0x7f7c94003970 2024-09-13T17:10:51.924 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:10:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 msgr2=0x7f7c8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 0x7f7c8c06d0c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7cb015b820 tx=0x7f7c98008040 comp rx=0 tx=0).stop 2024-09-13T17:10:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 msgr2=0x7f7cb014bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:51.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000c240 tx=0x7f7ca0001a00 comp rx=0 tx=0).stop 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 shutdown_connections 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8c06ac00 0x7f7c8c06d0c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cb00cd050 0x7f7cb00cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cb01f2390 0x7f7cb00ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 --2- 172.21.15.28:0/3696682026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c94003970 0x7f7cb014bfc0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 >> 172.21.15.28:0/3696682026 conn(0x7f7cb00b0270 msgr2=0x7f7cb015c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 shutdown_connections 2024-09-13T17:10:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:51.925+0000 7f7cb6200640 1 -- 172.21.15.28:0/3696682026 wait complete. 2024-09-13T17:10:51.942 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:10:51.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:10:51.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:10:52.021 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:10:52.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:10:52.021 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:10:52.063 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:10:52.063 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:10:52.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:10:52.101 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:10:52.101 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:10:52.139 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:10:52.139 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:10:52.140 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:10:52.176 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:10:52.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:10:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.334+0000 7fe53be00640 1 Processor -- start 2024-09-13T17:10:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.334+0000 7fe53be00640 1 -- start start 2024-09-13T17:10:52.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c154a90 0x7fe53c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe53c06b360 0x7fe53c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe53c0b0980 0x7fe53c0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53c0b0df0 con 0x7fe53c06b360 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53c15bb10 con 0x7fe53c0b0980 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c06b900 con 0x7fe53c154a90 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe533e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe53c06b360 0x7fe53c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe533e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe53c06b360 0x7fe53c0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53598/0 (socket says 172.21.15.28:53598) 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe533e00640 1 -- 172.21.15.28:0/515988788 learned_addr learned my addr 172.21.15.28:0/515988788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53aa00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c154a90 0x7fe53c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe53b400640 1 --2- 172.21.15.28:0/515988788 >> v2:172.21.15.189:3300/0 conn(0x7fe53c0b0980 0x7fe53c0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe533e00640 1 -- 172.21.15.28:0/515988788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c154a90 msgr2=0x7fe53c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.336+0000 7fe533e00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c154a90 0x7fe53c15b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.337+0000 7fe533e00640 1 -- 172.21.15.28:0/515988788 >> v2:172.21.15.189:3300/0 conn(0x7fe53c0b0980 msgr2=0x7fe53c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:52.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.337+0000 7fe533e00640 1 --2- 172.21.15.28:0/515988788 >> v2:172.21.15.189:3300/0 conn(0x7fe53c0b0980 0x7fe53c0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.337+0000 7fe533e00640 1 -- 172.21.15.28:0/515988788 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c06bbe0 con 0x7fe53c06b360 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe533e00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c0b0440 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe53c15e080 tx=0x7fe52402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c176a404223ccc17 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53a000640 1 -- 172.21.15.28:0/515988788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524032440 con 0x7fe53c06b360 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53a000640 1 -- 172.21.15.28:0/515988788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5240325a0 con 0x7fe53c06b360 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53a000640 1 -- 172.21.15.28:0/515988788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe524032930 con 0x7fe53c06b360 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 -- 172.21.15.28:0/515988788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 msgr2=0x7fe53c0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c0b0440 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe53c15e080 tx=0x7fe52402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 -- 172.21.15.28:0/515988788 shutdown_connections 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c0b0440 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 --2- 172.21.15.28:0/515988788 >> v2:172.21.15.189:3300/0 conn(0x7fe53c0b0980 0x7fe53c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 --2- 172.21.15.28:0/515988788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c154a90 0x7fe53c15b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.338+0000 7fe53be00640 1 -- 172.21.15.28:0/515988788 >> 172.21.15.28:0/515988788 conn(0x7fe53c072e40 msgr2=0x7fe53c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.339+0000 7fe53be00640 1 -- 172.21.15.28:0/515988788 shutdown_connections 2024-09-13T17:10:52.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.339+0000 7fe53be00640 1 -- 172.21.15.28:0/515988788 wait complete. 2024-09-13T17:10:52.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.340+0000 7fe53be00640 1 Processor -- start 2024-09-13T17:10:52.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.340+0000 7fe53be00640 1 -- start start 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c0b0980 0x7fe53c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe53c154a90 0x7fe53c1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c1f43a0 con 0x7fe53c06b360 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c154140 con 0x7fe53c154a90 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.341+0000 7fe53be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c1f3de0 con 0x7fe53c0b0980 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53608/0 (socket says 172.21.15.28:53608) 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 -- 172.21.15.28:0/3405149320 learned_addr learned my addr 172.21.15.28:0/3405149320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe533e00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c0b0980 0x7fe53c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53b400640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe53c154a90 0x7fe53c1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 -- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c0b0980 msgr2=0x7fe53c155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c0b0980 0x7fe53c155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 -- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe53c154a90 msgr2=0x7fe53c1f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe53c154a90 0x7fe53c1f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 -- 172.21.15.28:0/3405149320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240094d0 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.342+0000 7fe53aa00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00e930 tx=0x7fe52c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.343+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c009690 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.343+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe52c00ca00 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.343+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53c1f40c0 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.343+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe52c00cdb0 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.343+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53c155620 con 0x7fe53c06b360 2024-09-13T17:10:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.344+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53c1551a0 con 0x7fe53c06b360 2024-09-13T17:10:52.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.345+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe52c013470 con 0x7fe53c06b360 2024-09-13T17:10:52.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.346+0000 7fe532a00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 0x7fe51806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.346+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe52c089990 con 0x7fe53c06b360 2024-09-13T17:10:52.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.346+0000 7fe533e00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 0x7fe51806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.347+0000 7fe533e00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 0x7fe51806d0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe524034000 tx=0x7fe52402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.350+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe53c1551a0 con 0x7fe53c06b360 2024-09-13T17:10:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.562+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe53c1551a0 con 0x7fe53c06b360 2024-09-13T17:10:52.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.563+0000 7fe532a00640 1 -- 172.21.15.28:0/3405149320 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fe53c1551a0 con 0x7fe53c06b360 2024-09-13T17:10:52.564 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:52.564 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:10:40.139143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":25,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:10:52.564 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 msgr2=0x7fe51806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 0x7fe51806d0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fe524034000 tx=0x7fe52402e000 comp rx=0 tx=0).stop 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 msgr2=0x7fe53c153c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe52c00e930 tx=0x7fe52c00edf0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 shutdown_connections 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe51806ac00 0x7fe51806d0c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe53c154a90 0x7fe53c1f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe53c0b0980 0x7fe53c155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 --2- 172.21.15.28:0/3405149320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe53c06b360 0x7fe53c153c00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 >> 172.21.15.28:0/3405149320 conn(0x7fe53c072e40 msgr2=0x7fe53c15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:52.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 shutdown_connections 2024-09-13T17:10:52.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.565+0000 7fe53be00640 1 -- 172.21.15.28:0/3405149320 wait complete. 2024-09-13T17:10:52.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:10:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.737+0000 7f347d800640 1 Processor -- start 2024-09-13T17:10:52.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.738+0000 7f347d800640 1 -- start start 2024-09-13T17:10:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f347806b360 0x7f34780b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347d800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34780b0980 0x7f34780af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34780b0d80 con 0x7f3478154a90 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347806b900 con 0x7f347806b360 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347d800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f347806bbe0 con 0x7f34780b0980 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f346f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f347806b360 0x7f34780b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f3477e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3478154a90 0x7f347815b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44760/0 (socket says 172.21.15.28:44760) 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f3477e00640 1 -- 172.21.15.28:0/823480657 learned_addr learned my addr 172.21.15.28:0/823480657 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.739+0000 7f347ce00640 1 --2- 172.21.15.28:0/823480657 >> v2:172.21.15.189:3300/0 conn(0x7f34780b0980 0x7f34780af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f3477400640 1 -- 172.21.15.28:0/823480657 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370936013 0 0) 0x7f34780b0d80 con 0x7f3478154a90 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f3477400640 1 -- 172.21.15.28:0/823480657 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f345c003520 con 0x7f3478154a90 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f346f400640 1 -- 172.21.15.28:0/823480657 >> v2:172.21.15.189:3300/0 conn(0x7f34780b0980 msgr2=0x7f34780af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f346f400640 1 --2- 172.21.15.28:0/823480657 >> v2:172.21.15.189:3300/0 conn(0x7f34780b0980 0x7f34780af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f346f400640 1 -- 172.21.15.28:0/823480657 >> v1:172.21.15.28:6789/0 conn(0x7f3478154a90 legacy=0x7f347815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.740+0000 7f346f400640 1 -- 172.21.15.28:0/823480657 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f347806bec0 con 0x7f347806b360 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f346f400640 1 --2- 172.21.15.28:0/823480657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347806b360 0x7f34780b0440 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f34780b14d0 tx=0x7f346002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193d0744865c82d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f3477400640 1 -- 172.21.15.28:0/823480657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460032440 con 0x7f347806b360 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f3477400640 1 -- 172.21.15.28:0/823480657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34600325a0 con 0x7f347806b360 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f3477400640 1 -- 172.21.15.28:0/823480657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3460032930 con 0x7f347806b360 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f347d800640 1 -- 172.21.15.28:0/823480657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347806b360 msgr2=0x7f34780b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.741+0000 7f347d800640 1 --2- 172.21.15.28:0/823480657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347806b360 0x7f34780b0440 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f34780b14d0 tx=0x7f346002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 -- 172.21.15.28:0/823480657 shutdown_connections 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 --2- 172.21.15.28:0/823480657 >> v2:172.21.15.189:3300/0 conn(0x7f34780b0980 0x7f34780af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 --2- 172.21.15.28:0/823480657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f347806b360 0x7f34780b0440 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 -- 172.21.15.28:0/823480657 >> 172.21.15.28:0/823480657 conn(0x7f3478072e40 msgr2=0x7f3478073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 -- 172.21.15.28:0/823480657 shutdown_connections 2024-09-13T17:10:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.742+0000 7f347d800640 1 -- 172.21.15.28:0/823480657 wait complete. 2024-09-13T17:10:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.743+0000 7f347d800640 1 Processor -- start 2024-09-13T17:10:52.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.744+0000 7f347d800640 1 -- start start 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.744+0000 7f347d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f347806b360 0x7f347815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34780b0980 0x7f3478153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780ade50 con 0x7f3478154a90 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34780ad260 con 0x7f34780b0980 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34780ad520 con 0x7f347806b360 2024-09-13T17:10:52.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53634/0 (socket says 172.21.15.28:53634) 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 -- 172.21.15.28:0/3383120260 learned_addr learned my addr 172.21.15.28:0/3383120260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f346f400640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34780b0980 0x7f3478153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f3477e00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f347806b360 0x7f347815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 -- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f347806b360 msgr2=0x7f347815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f347806b360 0x7f347815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 -- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34780b0980 msgr2=0x7f3478153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34780b0980 0x7f3478153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.745+0000 7f347ce00640 1 -- 172.21.15.28:0/3383120260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009400 con 0x7f3478154a90 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f347ce00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f347000efc0 tx=0x7f347000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470009b10 con 0x7f3478154a90 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000f040 con 0x7f3478154a90 2024-09-13T17:10:52.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700146c0 con 0x7f3478154a90 2024-09-13T17:10:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34780ad800 con 0x7f3478154a90 2024-09-13T17:10:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.746+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f347815dc90 con 0x7f3478154a90 2024-09-13T17:10:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.747+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34780c45b0 con 0x7f3478154a90 2024-09-13T17:10:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.748+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3470014820 con 0x7f3478154a90 2024-09-13T17:10:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.749+0000 7f3474c00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 0x7f345406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.749+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3470089ea0 con 0x7f3478154a90 2024-09-13T17:10:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.749+0000 7f3477e00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 0x7f345406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.750+0000 7f3477e00640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 0x7f345406d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34680011e0 tx=0x7f3468009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.753+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3470054fc0 con 0x7f3478154a90 2024-09-13T17:10:52.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.936+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3478156110 con 0x7f3478154a90 2024-09-13T17:10:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.937+0000 7f3474c00640 1 -- 172.21.15.28:0/3383120260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f3470018090 con 0x7f3478154a90 2024-09-13T17:10:52.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:52.938 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:10:25.911635+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 msgr2=0x7f345406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 0x7f345406d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f34680011e0 tx=0x7f3468009040 comp rx=0 tx=0).stop 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 msgr2=0x7f34780ae800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f347000efc0 tx=0x7f347000c490 comp rx=0 tx=0).stop 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 shutdown_connections 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f345406ac00 0x7f345406d0c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3478154a90 0x7f34780ae800 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34780b0980 0x7f3478153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 --2- 172.21.15.28:0/3383120260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f347806b360 0x7f347815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 >> 172.21.15.28:0/3383120260 conn(0x7f3478072e40 msgr2=0x7f3478159110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 shutdown_connections 2024-09-13T17:10:52.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:52.940+0000 7f347d800640 1 -- 172.21.15.28:0/3383120260 wait complete. 2024-09-13T17:10:52.957 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:09:09.494172+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:09:10.074208+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 35, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:10:52.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:10:53.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.115+0000 7f3444a00640 1 Processor -- start 2024-09-13T17:10:53.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.115+0000 7f3444a00640 1 -- start start 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.116+0000 7f3444a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f344015c9a0 0x7f34400ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3444a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34400ad940 0x7f34400b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3444a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 0x7f344006f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3444a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400b26c0 con 0x7f344015c9a0 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3444a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34400ace30 con 0x7f34400b2230 2024-09-13T17:10:53.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3444a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34400ad110 con 0x7f34400ad940 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f344015c9a0 0x7f34400ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f344015c9a0 0x7f34400ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53636/0 (socket says 172.21.15.28:53636) 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 -- 172.21.15.28:0/2443691747 learned_addr learned my addr 172.21.15.28:0/2443691747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f3437e00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34400ad940 0x7f34400b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343e800640 1 --2- 172.21.15.28:0/2443691747 >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 0x7f344006f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 -- 172.21.15.28:0/2443691747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34400ad940 msgr2=0x7f34400b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34400ad940 0x7f34400b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 -- 172.21.15.28:0/2443691747 >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 msgr2=0x7f344006f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 --2- 172.21.15.28:0/2443691747 >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 0x7f344006f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343de00640 1 -- 172.21.15.28:0/2443691747 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344006b170 con 0x7f344015c9a0 2024-09-13T17:10:53.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.117+0000 7f343e800640 1 --2- 172.21.15.28:0/2443691747 >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 0x7f344006f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.118+0000 7f343de00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344015c9a0 0x7f34400ac850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f34400ad3f0 tx=0x7f343002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcff5dc68ed370f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f343d400640 1 -- 172.21.15.28:0/2443691747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430032440 con 0x7f344015c9a0 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f343d400640 1 -- 172.21.15.28:0/2443691747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34300325a0 con 0x7f344015c9a0 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f343d400640 1 -- 172.21.15.28:0/2443691747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430032930 con 0x7f344015c9a0 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 -- 172.21.15.28:0/2443691747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344015c9a0 msgr2=0x7f34400ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344015c9a0 0x7f34400ac850 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f34400ad3f0 tx=0x7f343002a270 comp rx=0 tx=0).stop 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 -- 172.21.15.28:0/2443691747 shutdown_connections 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 --2- 172.21.15.28:0/2443691747 >> v2:172.21.15.189:3300/0 conn(0x7f34400b2230 0x7f344006f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34400ad940 0x7f34400b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 --2- 172.21.15.28:0/2443691747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344015c9a0 0x7f34400ac850 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 -- 172.21.15.28:0/2443691747 >> 172.21.15.28:0/2443691747 conn(0x7f34400b0270 msgr2=0x7f34400b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 -- 172.21.15.28:0/2443691747 shutdown_connections 2024-09-13T17:10:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.119+0000 7f3444a00640 1 -- 172.21.15.28:0/2443691747 wait complete. 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.121+0000 7f3444a00640 1 Processor -- start 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.121+0000 7f3444a00640 1 -- start start 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34400b2230 0x7f34400cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f344015c9a0 0x7f34400cd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440149830 con 0x7f34400ad940 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3440148c80 con 0x7f34400b2230 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f3444a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3440148ed0 con 0x7f344015c9a0 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f343de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f343de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53642/0 (socket says 172.21.15.28:53642) 2024-09-13T17:10:53.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.122+0000 7f343de00640 1 -- 172.21.15.28:0/310524003 learned_addr learned my addr 172.21.15.28:0/310524003 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 -- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f344015c9a0 msgr2=0x7f34400cd7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f344015c9a0 0x7f34400cd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 -- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34400b2230 msgr2=0x7f34400cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34400b2230 0x7f34400cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 -- 172.21.15.28:0/310524003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34300094d0 con 0x7f34400ad940 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f343de00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3430000c00 tx=0x7f34300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430032440 con 0x7f34400ad940 2024-09-13T17:10:53.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3430030b80 con 0x7f34400ad940 2024-09-13T17:10:53.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343003b400 con 0x7f34400ad940 2024-09-13T17:10:53.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3440070930 con 0x7f34400ad940 2024-09-13T17:10:53.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.123+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3440149170 con 0x7f34400ad940 2024-09-13T17:10:53.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.124+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34400b28f0 con 0x7f34400ad940 2024-09-13T17:10:53.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.125+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3430039040 con 0x7f34400ad940 2024-09-13T17:10:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.126+0000 7f3436000640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 0x7f341c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.126+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f34300a9bb0 con 0x7f34400ad940 2024-09-13T17:10:53.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.126+0000 7f3437e00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 0x7f341c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.127+0000 7f3437e00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 0x7f341c06d0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3428006fd0 tx=0x7f3428008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.130+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3430074e80 con 0x7f34400ad940 2024-09-13T17:10:53.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.306+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f344015cd80 con 0x7f34400ad940 2024-09-13T17:10:53.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.306+0000 7f3436000640 1 -- 172.21.15.28:0/310524003 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34300370a0 con 0x7f34400ad940 2024-09-13T17:10:53.307 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 msgr2=0x7f341c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 0x7f341c06d0c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3428006fd0 tx=0x7f3428008040 comp rx=0 tx=0).stop 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 msgr2=0x7f34400ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3430000c00 tx=0x7f34300046c0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 shutdown_connections 2024-09-13T17:10:53.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341c06ac00 0x7f341c06d0c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f344015c9a0 0x7f34400cd7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34400b2230 0x7f34400cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 --2- 172.21.15.28:0/310524003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34400ad940 0x7f34400ca310 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 >> 172.21.15.28:0/310524003 conn(0x7f34400b0270 msgr2=0x7f344006ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 shutdown_connections 2024-09-13T17:10:53.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.309+0000 7f3444a00640 1 -- 172.21.15.28:0/310524003 wait complete. 2024-09-13T17:10:53.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:10:53.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.482+0000 7feee4600640 1 Processor -- start 2024-09-13T17:10:53.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.482+0000 7feee4600640 1 -- start start 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.483+0000 7feee4600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feedc15a780 0x7feedc070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.483+0000 7feee4600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 0x7feedc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.483+0000 7feee4600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feedc06c660 con 0x7feedc15ab60 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.483+0000 7feee4600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc06c8b0 con 0x7feedc15a780 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.483+0000 7feee4600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feedc06cb90 con 0x7feedc06c250 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feedc15ab60 0x7feedc06bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44774/0 (socket says 172.21.15.28:44774) 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feedc15a780 0x7feedc070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1200640 1 -- 172.21.15.28:0/3765859892 learned_addr learned my addr 172.21.15.28:0/3765859892 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee2600640 1 --2- 172.21.15.28:0/3765859892 >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 0x7feedc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1c00640 1 -- 172.21.15.28:0/3765859892 >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 msgr2=0x7feedc0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1c00640 1 --2- 172.21.15.28:0/3765859892 >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 0x7feedc0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1c00640 1 -- 172.21.15.28:0/3765859892 >> v1:172.21.15.28:6789/0 conn(0x7feedc15ab60 legacy=0x7feedc06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee1c00640 1 -- 172.21.15.28:0/3765859892 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feedc0cc980 con 0x7feedc15a780 2024-09-13T17:10:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.484+0000 7feee2600640 1 --2- 172.21.15.28:0/3765859892 >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 0x7feedc0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.485+0000 7feee1c00640 1 --2- 172.21.15.28:0/3765859892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc15a780 0x7feedc070d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7feedc0cc370 tx=0x7feed002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0dc467cb7c1ce46 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.485+0000 7feedbe00640 1 -- 172.21.15.28:0/3765859892 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0032440 con 0x7feedc15a780 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feedbe00640 1 -- 172.21.15.28:0/3765859892 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed00325a0 con 0x7feedc15a780 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feedbe00640 1 -- 172.21.15.28:0/3765859892 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0032930 con 0x7feedc15a780 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 -- 172.21.15.28:0/3765859892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc15a780 msgr2=0x7feedc070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 --2- 172.21.15.28:0/3765859892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc15a780 0x7feedc070d80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7feedc0cc370 tx=0x7feed002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 -- 172.21.15.28:0/3765859892 shutdown_connections 2024-09-13T17:10:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 --2- 172.21.15.28:0/3765859892 >> v2:172.21.15.189:3300/0 conn(0x7feedc06c250 0x7feedc0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 --2- 172.21.15.28:0/3765859892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc15a780 0x7feedc070d80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 -- 172.21.15.28:0/3765859892 >> 172.21.15.28:0/3765859892 conn(0x7feedc0b0270 msgr2=0x7feedc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 -- 172.21.15.28:0/3765859892 shutdown_connections 2024-09-13T17:10:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.486+0000 7feee4600640 1 -- 172.21.15.28:0/3765859892 wait complete. 2024-09-13T17:10:53.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.488+0000 7feee4600640 1 Processor -- start 2024-09-13T17:10:53.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.488+0000 7feee4600640 1 -- start start 2024-09-13T17:10:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 0x7feedc15d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc15ab60 0x7feedc15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc155060 con 0x7feedc06c250 2024-09-13T17:10:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feedc15c4e0 con 0x7feedc15ab60 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee4600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feedc1543e0 con 0x7feedc15a780 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53662/0 (socket says 172.21.15.28:53662) 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.489+0000 7feee1c00640 1 -- 172.21.15.28:0/2032223651 learned_addr learned my addr 172.21.15.28:0/2032223651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee2600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc15ab60 0x7feedc15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1200640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 0x7feedc15d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 -- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 msgr2=0x7feedc15d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 0x7feedc15d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 -- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc15ab60 msgr2=0x7feedc15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc15ab60 0x7feedc15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 -- 172.21.15.28:0/2032223651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed0009400 con 0x7feedc06c250 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1200640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 0x7feedc15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:53.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee1c00640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0032440 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed0030b80 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0006e10 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feedc1546c0 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.490+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feedc154c80 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.492+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feedc0b1670 con 0x7feedc06c250 2024-09-13T17:10:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.492+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feed003c400 con 0x7feedc06c250 2024-09-13T17:10:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.493+0000 7feed9600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 0x7feebc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.493+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feed00aa1c0 con 0x7feedc06c250 2024-09-13T17:10:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.494+0000 7feee1200640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 0x7feebc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.495+0000 7feee1200640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 0x7feebc06d0f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feedc15a190 tx=0x7feec8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.498+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feed0075370 con 0x7feedc06c250 2024-09-13T17:10:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.677+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feedc06bb40 con 0x7feedc06c250 2024-09-13T17:10:53.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.678+0000 7feed9600640 1 -- 172.21.15.28:0/2032223651 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) v1 ==== 104+0+11 (secure 0 0 0) 0x7feed0078990 con 0x7feedc06c250 2024-09-13T17:10:53.679 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 msgr2=0x7feebc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 0x7feebc06d0f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7feedc15a190 tx=0x7feec8009040 comp rx=0 tx=0).stop 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 msgr2=0x7feedc147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7feed0000c00 tx=0x7feed00046c0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 shutdown_connections 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feebc06ac30 0x7feebc06d0f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feedc15ab60 0x7feedc15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feedc15a780 0x7feedc15d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 --2- 172.21.15.28:0/2032223651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feedc06c250 0x7feedc147f10 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 >> 172.21.15.28:0/2032223651 conn(0x7feedc0b0270 msgr2=0x7feedc0acf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 shutdown_connections 2024-09-13T17:10:53.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.680+0000 7feee4600640 1 -- 172.21.15.28:0/2032223651 wait complete. 2024-09-13T17:10:53.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:10:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.854+0000 7fa596400640 1 Processor -- start 2024-09-13T17:10:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.854+0000 7fa596400640 1 -- start start 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa596400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5900cf890 0x7fa5900cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa596400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5900b6340 0x7fa5900b6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa596400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59015cf00 con 0x7fa59015cb20 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa596400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5900cd5e0 con 0x7fa5900cf890 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa596400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5900b5e60 con 0x7fa5900b6340 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.855+0000 7fa58ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5900cf890 0x7fa5900cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5900cf890 0x7fa5900cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60226/0 (socket says 172.21.15.28:60226) 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 -- 172.21.15.28:0/1062581406 learned_addr learned my addr 172.21.15.28:0/1062581406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58fe00640 1 --2- 172.21.15.28:0/1062581406 >> v2:172.21.15.189:3300/0 conn(0x7fa5900b6340 0x7fa5900b6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 -- 172.21.15.28:0/1062581406 >> v2:172.21.15.189:3300/0 conn(0x7fa5900b6340 msgr2=0x7fa5900b6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 --2- 172.21.15.28:0/1062581406 >> v2:172.21.15.189:3300/0 conn(0x7fa5900b6340 0x7fa5900b6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 -- 172.21.15.28:0/1062581406 >> v1:172.21.15.189:6789/0 conn(0x7fa59015cb20 legacy=0x7fa5900cf180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.856+0000 7fa58ea00640 1 -- 172.21.15.28:0/1062581406 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5900b4890 con 0x7fa5900cf890 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.857+0000 7fa58ea00640 1 --2- 172.21.15.28:0/1062581406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5900cf890 0x7fa5900cd0a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cfd00 tx=0x7fa57c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6777aacd24fff8a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.857+0000 7fa58e000640 1 -- 172.21.15.28:0/1062581406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c032440 con 0x7fa5900cf890 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.857+0000 7fa58e000640 1 -- 172.21.15.28:0/1062581406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa57c0325a0 con 0x7fa5900cf890 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.857+0000 7fa58e000640 1 -- 172.21.15.28:0/1062581406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c032930 con 0x7fa5900cf890 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 -- 172.21.15.28:0/1062581406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5900cf890 msgr2=0x7fa5900cd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 --2- 172.21.15.28:0/1062581406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5900cf890 0x7fa5900cd0a0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cfd00 tx=0x7fa57c02a270 comp rx=0 tx=0).stop 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 -- 172.21.15.28:0/1062581406 shutdown_connections 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 --2- 172.21.15.28:0/1062581406 >> v2:172.21.15.189:3300/0 conn(0x7fa5900b6340 0x7fa5900b6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 --2- 172.21.15.28:0/1062581406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5900cf890 0x7fa5900cd0a0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 -- 172.21.15.28:0/1062581406 >> 172.21.15.28:0/1062581406 conn(0x7fa5900708c0 msgr2=0x7fa590070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 -- 172.21.15.28:0/1062581406 shutdown_connections 2024-09-13T17:10:53.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.858+0000 7fa596400640 1 -- 172.21.15.28:0/1062581406 wait complete. 2024-09-13T17:10:53.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.860+0000 7fa596400640 1 Processor -- start 2024-09-13T17:10:53.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.860+0000 7fa596400640 1 -- start start 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5900b6340 0x7fa59014f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5900cf890 0x7fa5900cbd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59014fd80 con 0x7fa59015cb20 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5901f5fe0 con 0x7fa5900cf890 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa596400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5901f62c0 con 0x7fa5900b6340 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60230/0 (socket says 172.21.15.28:60230) 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 -- 172.21.15.28:0/2277635980 learned_addr learned my addr 172.21.15.28:0/2277635980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58f400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5900b6340 0x7fa59014f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58ea00640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5900cf890 0x7fa5900cbd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 -- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5900b6340 msgr2=0x7fa59014f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5900b6340 0x7fa59014f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 -- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5900cf890 msgr2=0x7fa5900cbd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5900cf890 0x7fa5900cbd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.861+0000 7fa58fe00640 1 -- 172.21.15.28:0/2277635980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57c0094d0 con 0x7fa59015cb20 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa58fe00640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa58800ec10 tx=0x7fa58800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58800cd20 con 0x7fa59015cb20 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58800f040 con 0x7fa59015cb20 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5880146c0 con 0x7fa59015cb20 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5901f65a0 con 0x7fa59015cb20 2024-09-13T17:10:53.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.862+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5901f6b60 con 0x7fa59015cb20 2024-09-13T17:10:53.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.863+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590067a10 con 0x7fa59015cb20 2024-09-13T17:10:53.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.864+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa588014a30 con 0x7fa59015cb20 2024-09-13T17:10:53.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.865+0000 7fa587400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 0x7fa57006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:53.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.865+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa588089e80 con 0x7fa59015cb20 2024-09-13T17:10:53.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.865+0000 7fa58f400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 0x7fa57006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.866+0000 7fa58f400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 0x7fa57006d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa580000f60 tx=0x7fa580009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:53.869+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa588055180 con 0x7fa59015cb20 2024-09-13T17:10:54.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.048+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5901f6d50 con 0x7fa59015cb20 2024-09-13T17:10:54.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.049+0000 7fa587400640 1 -- 172.21.15.28:0/2277635980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa5880180a0 con 0x7fa59015cb20 2024-09-13T17:10:54.050 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:10:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 msgr2=0x7fa57006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 0x7fa57006d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa580000f60 tx=0x7fa580009040 comp rx=0 tx=0).stop 2024-09-13T17:10:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 msgr2=0x7fa590158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa58800ec10 tx=0x7fa58800c490 comp rx=0 tx=0).stop 2024-09-13T17:10:54.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 shutdown_connections 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa57006ac00 0x7fa57006d0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa59015cb20 0x7fa590158670 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5900cf890 0x7fa5900cbd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 --2- 172.21.15.28:0/2277635980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5900b6340 0x7fa59014f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 >> 172.21.15.28:0/2277635980 conn(0x7fa5900708c0 msgr2=0x7fa590159f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 shutdown_connections 2024-09-13T17:10:54.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.051+0000 7fa596400640 1 -- 172.21.15.28:0/2277635980 wait complete. 2024-09-13T17:10:54.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:10:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.221+0000 7f70fbe00640 1 Processor -- start 2024-09-13T17:10:54.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.221+0000 7f70fbe00640 1 -- start start 2024-09-13T17:10:54.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70fbe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70f4146170 0x7f70f415d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70fbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 0x7f70f4160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70fbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f4146550 con 0x7f70f4160310 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70fbe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f415cd20 con 0x7f70f4146170 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70fbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f415d280 con 0x7f70f40acf60 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.222+0000 7f70f8a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70f4160310 0x7f70f40ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38450/0 (socket says 172.21.15.28:38450) 2024-09-13T17:10:54.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f8a00640 1 -- 172.21.15.28:0/3295435128 learned_addr learned my addr 172.21.15.28:0/3295435128 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:54.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f9400640 1 --2- 172.21.15.28:0/3295435128 >> v2:172.21.15.189:3300/0 conn(0x7f70f4146170 0x7f70f415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f9e00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 0x7f70f4160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966767173 0 0) 0x7f70f4146550 con 0x7f70f4160310 2024-09-13T17:10:54.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70dc003520 con 0x7f70f4160310 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570550029 0 0) 0x7f70dc003520 con 0x7f70f4160310 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 msgr2=0x7f70f4160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 0x7f70f4160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 >> v2:172.21.15.189:3300/0 conn(0x7f70f4146170 msgr2=0x7f70f415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 --2- 172.21.15.28:0/3295435128 >> v2:172.21.15.189:3300/0 conn(0x7f70f4146170 0x7f70f415d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f415dc50 con 0x7f70f4160310 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f70e0001670 con 0x7f70f4160310 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 >> v1:172.21.15.28:6789/0 conn(0x7f70f4160310 legacy=0x7f70f40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.223+0000 7f70f3e00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70dc003970 0x7f70dc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f3e00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70dc0047c0 0x7f70dc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f3e00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 0x7f70dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f415d280 con 0x7f70dc003970 2024-09-13T17:10:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc00b980 con 0x7f70dc0047c0 2024-09-13T17:10:54.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70dc00bac0 con 0x7f70f40acf60 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70dc003970 0x7f70dc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 -- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 msgr2=0x7f70dc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70f40acf60 0x7f70dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 -- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70dc0047c0 msgr2=0x7f70dc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70dc0047c0 0x7f70dc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.224+0000 7f70f9400640 1 -- 172.21.15.28:0/3295435128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70dc003520 con 0x7f70dc003970 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.225+0000 7f70f9400640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70dc003970 0x7f70dc003d90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009ac0 tx=0x7f70dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa26120a4a406943 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8031400 con 0x7f70dc003970 2024-09-13T17:10:54.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e802d050 con 0x7f70dc003970 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70f3e00640 1 -- 172.21.15.28:0/3295435128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8031780 con 0x7f70dc003970 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 -- 172.21.15.28:0/3295435128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70dc003970 msgr2=0x7f70dc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 --2- 172.21.15.28:0/3295435128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70dc003970 0x7f70dc003d90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f70e8009ac0 tx=0x7f70dc0057f0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70f9e00640 1 -- 172.21.15.28:0/3295435128 reap_dead start 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 -- 172.21.15.28:0/3295435128 shutdown_connections 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 -- 172.21.15.28:0/3295435128 >> 172.21.15.28:0/3295435128 conn(0x7f70f4069e80 msgr2=0x7f70f406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 -- 172.21.15.28:0/3295435128 shutdown_connections 2024-09-13T17:10:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.226+0000 7f70fbe00640 1 -- 172.21.15.28:0/3295435128 wait complete. 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.228+0000 7f70fbe00640 1 Processor -- start 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.228+0000 7f70fbe00640 1 -- start start 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70dc003970 0x7f70f413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70f4141a10 0x7f70f4141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f4155630 con 0x7f70f4070470 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f413faf0 con 0x7f70f4141a10 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70fbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f4070e10 con 0x7f70dc003970 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70f8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70f8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60248/0 (socket says 172.21.15.28:60248) 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70f8a00640 1 -- 172.21.15.28:0/1380508313 learned_addr learned my addr 172.21.15.28:0/1380508313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70f9400640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70dc003970 0x7f70f413eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.229+0000 7f70f9e00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70f4141a10 0x7f70f4141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 -- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70dc003970 msgr2=0x7f70f413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70dc003970 0x7f70f413eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 -- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70f4141a10 msgr2=0x7f70f4141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70f4141a10 0x7f70f4141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 -- 172.21.15.28:0/1380508313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f4142740 con 0x7f70f4070470 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f8a00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c730 tx=0x7f70e000cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e00072d0 con 0x7f70f4070470 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e0007450 con 0x7f70f4070470 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f4142a20 con 0x7f70f4070470 2024-09-13T17:10:54.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e000ce70 con 0x7f70f4070470 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.230+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f4155250 con 0x7f70f4070470 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.231+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f4155770 con 0x7f70f4070470 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.232+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f70e00047e0 con 0x7f70f4070470 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.233+0000 7f70f1600640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 0x7f70d406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.233+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f70e0088a90 con 0x7f70f4070470 2024-09-13T17:10:54.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.234+0000 7f70f9400640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 0x7f70d406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.234+0000 7f70f9400640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 0x7f70d406d0b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70e8033000 tx=0x7f70e8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.237+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f70f4155770 con 0x7f70f4070470 2024-09-13T17:10:54.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.452+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70f4155770 con 0x7f70f4070470 2024-09-13T17:10:54.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.452+0000 7f70f1600640 1 -- 172.21.15.28:0/1380508313 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f70f4155770 con 0x7f70f4070470 2024-09-13T17:10:54.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:54.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:10:40.139143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":25,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:10:54.454 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-13T17:10:54.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 msgr2=0x7f70d406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 0x7f70d406d0b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f70e8033000 tx=0x7f70e8008030 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 msgr2=0x7f70f40708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70e000c730 tx=0x7f70e000cb30 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 shutdown_connections 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70d406abf0 0x7f70d406d0b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70f4141a10 0x7f70f4141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70f4070470 0x7f70f40708d0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 --2- 172.21.15.28:0/1380508313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70dc003970 0x7f70f413eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.454+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 >> 172.21.15.28:0/1380508313 conn(0x7f70f4069e80 msgr2=0x7f70f40aed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.455+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 shutdown_connections 2024-09-13T17:10:54.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.455+0000 7f70fbe00640 1 -- 172.21.15.28:0/1380508313 wait complete. 2024-09-13T17:10:54.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:10:54.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.626+0000 7f551da00640 1 Processor -- start 2024-09-13T17:10:54.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.626+0000 7f551da00640 1 -- start start 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f551da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5518159440 0x7f55180b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f551da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55180c3850 con 0x7f5518159820 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f551da00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551815a930 con 0x7f55180b29f0 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f551da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5518159d20 con 0x7f5518159440 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f550fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5518159820 0x7f55180b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38460/0 (socket says 172.21.15.28:38460) 2024-09-13T17:10:54.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f550fe00640 1 -- 172.21.15.28:0/4258940878 learned_addr learned my addr 172.21.15.28:0/4258940878 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.627+0000 7f5517e00640 1 --2- 172.21.15.28:0/4258940878 >> v2:172.21.15.189:3300/0 conn(0x7f5518159440 0x7f55180b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188158674 0 0) 0x7f55180c3850 con 0x7f5518159820 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc005730 con 0x7f5518159820 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517e00640 1 -- 172.21.15.28:0/4258940878 >> v1:172.21.15.189:6789/0 conn(0x7f55180b29f0 legacy=0x7f551815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1357594915 0 0) 0x7f551815a930 con 0x7f55180b29f0 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517e00640 1 -- 172.21.15.28:0/4258940878 >> v1:172.21.15.28:6789/0 conn(0x7f5518159820 legacy=0x7f55180b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.628+0000 7f5517e00640 1 -- 172.21.15.28:0/4258940878 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551815a000 con 0x7f5518159440 2024-09-13T17:10:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.629+0000 7f5517e00640 1 --2- 172.21.15.28:0/4258940878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f55180b0440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55080097d0 tx=0x7f550802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae482ad34ec95b99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.629+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508032640 con 0x7f5518159440 2024-09-13T17:10:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.629+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55080327a0 con 0x7f5518159440 2024-09-13T17:10:54.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.629+0000 7f5517400640 1 -- 172.21.15.28:0/4258940878 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5508032b30 con 0x7f5518159440 2024-09-13T17:10:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 -- 172.21.15.28:0/4258940878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 msgr2=0x7f55180b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 --2- 172.21.15.28:0/4258940878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f55180b0440 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f55080097d0 tx=0x7f550802a300 comp rx=0 tx=0).stop 2024-09-13T17:10:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 -- 172.21.15.28:0/4258940878 shutdown_connections 2024-09-13T17:10:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 --2- 172.21.15.28:0/4258940878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f55180b0440 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 -- 172.21.15.28:0/4258940878 >> 172.21.15.28:0/4258940878 conn(0x7f5518072e40 msgr2=0x7f5518073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:54.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 -- 172.21.15.28:0/4258940878 shutdown_connections 2024-09-13T17:10:54.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.630+0000 7f551da00640 1 -- 172.21.15.28:0/4258940878 wait complete. 2024-09-13T17:10:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.632+0000 7f551da00640 1 Processor -- start 2024-09-13T17:10:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.632+0000 7f551da00640 1 -- start start 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f5518149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5518159820 0x7f551814a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551814ff90 con 0x7f55180b29f0 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551814f380 con 0x7f5518159440 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551814f660 con 0x7f5518159820 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60262/0 (socket says 172.21.15.28:60262) 2024-09-13T17:10:54.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 -- 172.21.15.28:0/517763783 learned_addr learned my addr 172.21.15.28:0/517763783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:54.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f550fe00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f5518149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f551d000640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5518159820 0x7f551814a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 -- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5518159820 msgr2=0x7f551814a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5518159820 0x7f551814a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 -- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 msgr2=0x7f5518149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f5518149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f5517e00640 1 -- 172.21.15.28:0/517763783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5508009400 con 0x7f55180b29f0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.633+0000 7f550fe00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f5518149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f5517e00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5508035040 tx=0x7f55080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550803d8f0 con 0x7f55180b29f0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550803da50 con 0x7f55180b29f0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550803dde0 con 0x7f55180b29f0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551814f940 con 0x7f55180b29f0 2024-09-13T17:10:54.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.634+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551814f030 con 0x7f55180b29f0 2024-09-13T17:10:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.635+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551815c3f0 con 0x7f55180b29f0 2024-09-13T17:10:54.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.636+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5508039040 con 0x7f55180b29f0 2024-09-13T17:10:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.637+0000 7f5514c00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.637+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f55080aa330 con 0x7f55180b29f0 2024-09-13T17:10:54.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.637+0000 7f550fe00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:54.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.638+0000 7f550fe00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 0x7f54f406d0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5500007fd0 tx=0x7f5500009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.641+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5508075570 con 0x7f55180b29f0 2024-09-13T17:10:54.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.827+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f55180b0440 con 0x7f55180b29f0 2024-09-13T17:10:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.828+0000 7f5514c00640 1 -- 172.21.15.28:0/517763783 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f5508078b90 con 0x7f55180b29f0 2024-09-13T17:10:54.829 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:54.830 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:10:25.911635+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:10:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 msgr2=0x7f54f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 0x7f54f406d0c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5500007fd0 tx=0x7f5500009040 comp rx=0 tx=0).stop 2024-09-13T17:10:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 msgr2=0x7f5518149090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:54.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5508035040 tx=0x7f55080046c0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 shutdown_connections 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5518159820 0x7f551814a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5518159440 0x7f5518149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 --2- 172.21.15.28:0/517763783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55180b29f0 0x7f5518149090 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 >> 172.21.15.28:0/517763783 conn(0x7f5518072e40 msgr2=0x7f551815b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 shutdown_connections 2024-09-13T17:10:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:54.831+0000 7f551da00640 1 -- 172.21.15.28:0/517763783 wait complete. 2024-09-13T17:10:54.848 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:10:54.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:10:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.004+0000 7fbed5c00640 1 Processor -- start 2024-09-13T17:10:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.004+0000 7fbed5c00640 1 -- start start 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbed5c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbed01592a0 0x7fbed015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbed5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbed006c2d0 0x7fbed006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbed5c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbed006cc70 con 0x7fbed006f970 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbed5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed015b0b0 con 0x7fbed006c2d0 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbed5c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbed0159d90 con 0x7fbed01592a0 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbed006c2d0 0x7fbed006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbed006c2d0 0x7fbed006c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 -- 172.21.15.28:0/335122073 learned_addr learned my addr 172.21.15.28:0/335122073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecea00640 1 --2- 172.21.15.28:0/335122073 >> v2:172.21.15.189:3300/0 conn(0x7fbed01592a0 0x7fbed015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 -- 172.21.15.28:0/335122073 >> v2:172.21.15.189:3300/0 conn(0x7fbed01592a0 msgr2=0x7fbed015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 --2- 172.21.15.28:0/335122073 >> v2:172.21.15.189:3300/0 conn(0x7fbed01592a0 0x7fbed015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 -- 172.21.15.28:0/335122073 >> v1:172.21.15.189:6789/0 conn(0x7fbed006f970 legacy=0x7fbed0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.006+0000 7fbecf400640 1 -- 172.21.15.28:0/335122073 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbed00b2040 con 0x7fbed006c2d0 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.007+0000 7fbecf400640 1 --2- 172.21.15.28:0/335122073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed006c2d0 0x7fbed006c730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbed006d370 tx=0x7fbec802c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb8a4f33fc90972 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbece000640 1 -- 172.21.15.28:0/335122073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec800b2e0 con 0x7fbed006c2d0 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbece000640 1 -- 172.21.15.28:0/335122073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec800b440 con 0x7fbed006c2d0 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbece000640 1 -- 172.21.15.28:0/335122073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8009820 con 0x7fbed006c2d0 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 -- 172.21.15.28:0/335122073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed006c2d0 msgr2=0x7fbed006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 --2- 172.21.15.28:0/335122073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed006c2d0 0x7fbed006c730 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbed006d370 tx=0x7fbec802c940 comp rx=0 tx=0).stop 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 -- 172.21.15.28:0/335122073 shutdown_connections 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 --2- 172.21.15.28:0/335122073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed006c2d0 0x7fbed006c730 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 --2- 172.21.15.28:0/335122073 >> v2:172.21.15.189:3300/0 conn(0x7fbed01592a0 0x7fbed015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 -- 172.21.15.28:0/335122073 >> 172.21.15.28:0/335122073 conn(0x7fbed0069e80 msgr2=0x7fbed006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 -- 172.21.15.28:0/335122073 shutdown_connections 2024-09-13T17:10:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.008+0000 7fbed5c00640 1 -- 172.21.15.28:0/335122073 wait complete. 2024-09-13T17:10:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.010+0000 7fbed5c00640 1 Processor -- start 2024-09-13T17:10:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.010+0000 7fbed5c00640 1 -- start start 2024-09-13T17:10:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 0x7fbed00aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed006f970 0x7fbed00cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed00b02a0 con 0x7fbed01592a0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbed00af620 con 0x7fbed006f970 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbed5c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbed00af900 con 0x7fbed006c2d0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbecf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbecf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60276/0 (socket says 172.21.15.28:60276) 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.011+0000 7fbecf400640 1 -- 172.21.15.28:0/2455399624 learned_addr learned my addr 172.21.15.28:0/2455399624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecea00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 0x7fbed00aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbec7e00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed006f970 0x7fbed00cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 -- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 msgr2=0x7fbed00aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 0x7fbed00aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 -- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed006f970 msgr2=0x7fbed00cdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed006f970 0x7fbed00cdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 -- 172.21.15.28:0/2455399624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec800bd40 con 0x7fbed01592a0 2024-09-13T17:10:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecea00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 0x7fbed00aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbecf400640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbec800b5f0 tx=0x7fbec8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec800b2e0 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.012+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec8032bf0 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.013+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbed00afbe0 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.013+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec800b720 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.013+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbed00af330 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.014+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbed015a2b0 con 0x7fbed01592a0 2024-09-13T17:10:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.014+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbec803b3f0 con 0x7fbed01592a0 2024-09-13T17:10:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.015+0000 7fbec6a00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 0x7fbeac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.015+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fbec80aaac0 con 0x7fbed01592a0 2024-09-13T17:10:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.016+0000 7fbecea00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 0x7fbeac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.017+0000 7fbecea00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 0x7fbeac06d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbec000b9f0 tx=0x7fbec0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.020+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbed015a2b0 con 0x7fbed01592a0 2024-09-13T17:10:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.236+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbed00b03e0 con 0x7fbed01592a0 2024-09-13T17:10:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.237+0000 7fbec6a00640 1 -- 172.21.15.28:0/2455399624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fbed00b03e0 con 0x7fbed01592a0 2024-09-13T17:10:55.238 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:55.238 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:10:40.139143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":25,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:10:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-13T17:10:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 msgr2=0x7fbeac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 0x7fbeac06d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbec000b9f0 tx=0x7fbec0008040 comp rx=0 tx=0).stop 2024-09-13T17:10:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 msgr2=0x7fbed00ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fbec800b5f0 tx=0x7fbec8004a40 comp rx=0 tx=0).stop 2024-09-13T17:10:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 shutdown_connections 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbeac06ac00 0x7fbeac06d0c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbed01592a0 0x7fbed00ce2b0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbed006f970 0x7fbed00cdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 --2- 172.21.15.28:0/2455399624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbed006c2d0 0x7fbed00aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 >> 172.21.15.28:0/2455399624 conn(0x7fbed0069e80 msgr2=0x7fbed0156890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 shutdown_connections 2024-09-13T17:10:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.239+0000 7fbed5c00640 1 -- 172.21.15.28:0/2455399624 wait complete. 2024-09-13T17:10:55.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:10:55.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.409+0000 7f648be00640 1 Processor -- start 2024-09-13T17:10:55.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.410+0000 7f648be00640 1 -- start start 2024-09-13T17:10:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c154a90 0x7f648c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f648c06b360 0x7f648c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f648c0b0980 0x7f648c0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c0b0df0 con 0x7f648c06b360 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f648c15bb10 con 0x7f648c0b0980 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c06b900 con 0x7f648c154a90 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f6483e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f648c06b360 0x7f648c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f6483e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f648c06b360 0x7f648c0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60288/0 (socket says 172.21.15.28:60288) 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f6483e00640 1 -- 172.21.15.28:0/1449928366 learned_addr learned my addr 172.21.15.28:0/1449928366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648b400640 1 --2- 172.21.15.28:0/1449928366 >> v2:172.21.15.189:3300/0 conn(0x7f648c0b0980 0x7f648c0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.411+0000 7f648aa00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c154a90 0x7f648c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.412+0000 7f6483e00640 1 -- 172.21.15.28:0/1449928366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c154a90 msgr2=0x7f648c15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.412+0000 7f6483e00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c154a90 0x7f648c15b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.412+0000 7f6483e00640 1 -- 172.21.15.28:0/1449928366 >> v2:172.21.15.189:3300/0 conn(0x7f648c0b0980 msgr2=0x7f648c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.412+0000 7f6483e00640 1 --2- 172.21.15.28:0/1449928366 >> v2:172.21.15.189:3300/0 conn(0x7f648c0b0980 0x7f648c0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.412+0000 7f6483e00640 1 -- 172.21.15.28:0/1449928366 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c06bbe0 con 0x7f648c06b360 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f6483e00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c06b360 0x7f648c0b0440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f648c15e080 tx=0x7f647402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd42ad05f3d378c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f648a000640 1 -- 172.21.15.28:0/1449928366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474032440 con 0x7f648c06b360 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f648a000640 1 -- 172.21.15.28:0/1449928366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64740325a0 con 0x7f648c06b360 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f648a000640 1 -- 172.21.15.28:0/1449928366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6474032930 con 0x7f648c06b360 2024-09-13T17:10:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f648be00640 1 -- 172.21.15.28:0/1449928366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c06b360 msgr2=0x7f648c0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.413+0000 7f648be00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c06b360 0x7f648c0b0440 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f648c15e080 tx=0x7f647402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 -- 172.21.15.28:0/1449928366 shutdown_connections 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c06b360 0x7f648c0b0440 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 --2- 172.21.15.28:0/1449928366 >> v2:172.21.15.189:3300/0 conn(0x7f648c0b0980 0x7f648c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 --2- 172.21.15.28:0/1449928366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c154a90 0x7f648c15b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 -- 172.21.15.28:0/1449928366 >> 172.21.15.28:0/1449928366 conn(0x7f648c072e40 msgr2=0x7f648c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 -- 172.21.15.28:0/1449928366 shutdown_connections 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.414+0000 7f648be00640 1 -- 172.21.15.28:0/1449928366 wait complete. 2024-09-13T17:10:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.415+0000 7f648be00640 1 Processor -- start 2024-09-13T17:10:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.415+0000 7f648be00640 1 -- start start 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f648c06b360 0x7f648c1ef340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c0b0980 0x7f648c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0ad920 con 0x7f648c154a90 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0adc00 con 0x7f648c06b360 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.416+0000 7f648be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f648c0adee0 con 0x7f648c0b0980 2024-09-13T17:10:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60298/0 (socket says 172.21.15.28:60298) 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 -- 172.21.15.28:0/3425440088 learned_addr learned my addr 172.21.15.28:0/3425440088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f6483e00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c0b0980 0x7f648c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648aa00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f648c06b360 0x7f648c1ef340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 -- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c0b0980 msgr2=0x7f648c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c0b0980 0x7f648c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 -- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f648c06b360 msgr2=0x7f648c1ef340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f648c06b360 0x7f648c1ef340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 -- 172.21.15.28:0/3425440088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64740094d0 con 0x7f648c154a90 2024-09-13T17:10:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.417+0000 7f648b400640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f648400eed0 tx=0x7f648400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.418+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484010040 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.418+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648400bd10 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.418+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484014640 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.418+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f648c0ae1c0 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.418+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648c1ef880 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.419+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648c0afcc0 con 0x7f648c154a90 2024-09-13T17:10:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.420+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64840147a0 con 0x7f648c154a90 2024-09-13T17:10:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.421+0000 7f6482a00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 0x7f646806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.421+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6484089900 con 0x7f648c154a90 2024-09-13T17:10:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.421+0000 7f648aa00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 0x7f646806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.422+0000 7f648aa00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 0x7f646806d0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f647c009b40 tx=0x7f647c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.425+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6484054b60 con 0x7f648c154a90 2024-09-13T17:10:55.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.642+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648c0b03c0 con 0x7f648c154a90 2024-09-13T17:10:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.643+0000 7f6482a00640 1 -- 172.21.15.28:0/3425440088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f6484018090 con 0x7f648c154a90 2024-09-13T17:10:55.644 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:55.644 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":27,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:09:11.091132+0000","modified":"2024-09-13T17:10:40.139143+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5121},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5121":{"gid":5121,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":25,"addr":"172.21.15.28:6835/696058040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":696058040},{"type":"v1","addr":"172.21.15.28:6835","nonce":696058040}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T17:10:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 27 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 msgr2=0x7f646806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 0x7f646806d0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f647c009b40 tx=0x7f647c009190 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 msgr2=0x7f648c0c5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f648400eed0 tx=0x7f648400b480 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 shutdown_connections 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f646806ac00 0x7f646806d0c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f648c154a90 0x7f648c0c5850 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f648c0b0980 0x7f648c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 --2- 172.21.15.28:0/3425440088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f648c06b360 0x7f648c1ef340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 >> 172.21.15.28:0/3425440088 conn(0x7f648c072e40 msgr2=0x7f648c15bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 shutdown_connections 2024-09-13T17:10:55.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.645+0000 7f648be00640 1 -- 172.21.15.28:0/3425440088 wait complete. 2024-09-13T17:10:55.662 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:10:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.816+0000 7f03b7800640 1 Processor -- start 2024-09-13T17:10:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.816+0000 7f03b7800640 1 -- start start 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.817+0000 7f03b7800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 0x7f03b015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.817+0000 7f03b7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00b3970 con 0x7f03b006fb50 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.817+0000 7f03b7800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b0159430 con 0x7f03b00b3160 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.817+0000 7f03b7800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0159230 con 0x7f03b00cdfa0 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.817+0000 7f03b5800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f03b006fb50 0x7f03b006b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38464/0 (socket says 172.21.15.28:38464) 2024-09-13T17:10:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03b5800640 1 -- 172.21.15.28:0/352363401 learned_addr learned my addr 172.21.15.28:0/352363401 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03b4e00640 1 --2- 172.21.15.28:0/352363401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 0x7f03b015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3638644521 0 0) 0x7f03a8001490 con 0x7f03b006fb50 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0398003520 con 0x7f03b006fb50 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040272311 0 0) 0x7f03b0159430 con 0x7f03b00b3160 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03a8001490 con 0x7f03b00b3160 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03b4e00640 1 -- 172.21.15.28:0/352363401 >> v1:172.21.15.189:6789/0 conn(0x7f03b00b3160 legacy=0x7f03b006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03b4e00640 1 -- 172.21.15.28:0/352363401 >> v1:172.21.15.28:6789/0 conn(0x7f03b006fb50 legacy=0x7f03b006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.818+0000 7f03b4e00640 1 -- 172.21.15.28:0/352363401 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b0159710 con 0x7f03b00cdfa0 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03b4e00640 1 --2- 172.21.15.28:0/352363401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 0x7f03b015d340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f03a40097d0 tx=0x7f03a402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ede3f293e909a0ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4032440 con 0x7f03b00cdfa0 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40325a0 con 0x7f03b00cdfa0 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03af400640 1 -- 172.21.15.28:0/352363401 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4032910 con 0x7f03b00cdfa0 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03b7800640 1 -- 172.21.15.28:0/352363401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 msgr2=0x7f03b015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03b7800640 1 --2- 172.21.15.28:0/352363401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 0x7f03b015d340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f03a40097d0 tx=0x7f03a402a210 comp rx=0 tx=0).stop 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03b7800640 1 -- 172.21.15.28:0/352363401 shutdown_connections 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.820+0000 7f03b7800640 1 --2- 172.21.15.28:0/352363401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b00cdfa0 0x7f03b015d340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.821+0000 7f03b7800640 1 -- 172.21.15.28:0/352363401 >> 172.21.15.28:0/352363401 conn(0x7f03b0069e80 msgr2=0x7f03b006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.821+0000 7f03b7800640 1 -- 172.21.15.28:0/352363401 shutdown_connections 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.821+0000 7f03b7800640 1 -- 172.21.15.28:0/352363401 wait complete. 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.822+0000 7f03b7800640 1 Processor -- start 2024-09-13T17:10:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.822+0000 7f03b7800640 1 -- start start 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b006fb50 0x7f03b013b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b00cdfa0 0x7f03b0157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0156af0 con 0x7f03b00b3160 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b006b260 con 0x7f03b00cdfa0 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.823+0000 7f03b7800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b006b540 con 0x7f03b006fb50 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60314/0 (socket says 172.21.15.28:60314) 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 -- 172.21.15.28:0/1210897711 learned_addr learned my addr 172.21.15.28:0/1210897711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03b4e00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b006fb50 0x7f03b013b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03b5800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b00cdfa0 0x7f03b0157040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 -- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b006fb50 msgr2=0x7f03b013b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b006fb50 0x7f03b013b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 -- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b00cdfa0 msgr2=0x7f03b0157040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b00cdfa0 0x7f03b0157040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 -- 172.21.15.28:0/1210897711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a4009400 con 0x7f03b00b3160 2024-09-13T17:10:55.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.824+0000 7f03afe00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f039c00c400 tx=0x7f039c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.825+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c0052d0 con 0x7f03b00b3160 2024-09-13T17:10:55.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.825+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039c005430 con 0x7f03b00b3160 2024-09-13T17:10:55.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.825+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039c003810 con 0x7f03b00b3160 2024-09-13T17:10:55.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.825+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b006b820 con 0x7f03b00b3160 2024-09-13T17:10:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.825+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b013bb90 con 0x7f03b00b3160 2024-09-13T17:10:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.826+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b006c020 con 0x7f03b00b3160 2024-09-13T17:10:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.827+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f039c003970 con 0x7f03b00b3160 2024-09-13T17:10:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.828+0000 7f03acc00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 0x7f039006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.828+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f039c0889a0 con 0x7f03b00b3160 2024-09-13T17:10:55.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.828+0000 7f03b4e00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 0x7f039006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:55.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.829+0000 7f03b4e00640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 0x7f039006d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f03a4034000 tx=0x7f03a402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:55.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:55.832+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f03b006c020 con 0x7f03b00b3160 2024-09-13T17:10:56.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.019+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b006c020 con 0x7f03b00b3160 2024-09-13T17:10:56.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.020+0000 7f03acc00640 1 -- 172.21.15.28:0/1210897711 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+16634 (secure 0 0 0) 0x7f03b006c020 con 0x7f03b00b3160 2024-09-13T17:10:56.022 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:10:56.023 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":36,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:10:25.911635+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:09:09.494172+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-09-13T17:09:10.074208+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":35,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":5,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:10:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 msgr2=0x7f039006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 0x7f039006d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f03a4034000 tx=0x7f03a402e000 comp rx=0 tx=0).stop 2024-09-13T17:10:56.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 msgr2=0x7f03b0157d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f039c00c400 tx=0x7f039c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 shutdown_connections 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f039006ac00 0x7f039006d0c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b00cdfa0 0x7f03b0157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b00b3160 0x7f03b0157d00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 --2- 172.21.15.28:0/1210897711 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03b006fb50 0x7f03b013b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 >> 172.21.15.28:0/1210897711 conn(0x7f03b0069e80 msgr2=0x7f03b0158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 shutdown_connections 2024-09-13T17:10:56.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.023+0000 7f03b7800640 1 -- 172.21.15.28:0/1210897711 wait complete. 2024-09-13T17:10:56.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:10:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.196+0000 7fceb7000640 1 Processor -- start 2024-09-13T17:10:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.196+0000 7fceb7000640 1 -- start start 2024-09-13T17:10:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb7000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 0x7fceb015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb7000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceb015ab60 con 0x7fceb0159770 2024-09-13T17:10:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb7000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceb006f920 con 0x7fceb0070430 2024-09-13T17:10:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb015b290 con 0x7fceb006f3f0 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fcea7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 0x7fceb015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56686/0 (socket says 172.21.15.28:56686) 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fcea7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 0x7fceb015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:51876/0 (socket says 172.21.15.28:51876) 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.197+0000 7fceb5000640 1 -- 172.21.15.28:0/3689038137 learned_addr learned my addr 172.21.15.28:0/3689038137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.198+0000 7fceb5000640 1 -- 172.21.15.28:0/3689038137 >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 msgr2=0x7fceb015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.198+0000 7fceb5000640 1 --2- 172.21.15.28:0/3689038137 >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 0x7fceb015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.198+0000 7fceb5000640 1 -- 172.21.15.28:0/3689038137 >> v1:172.21.15.189:6789/0 conn(0x7fceb0159770 legacy=0x7fceb006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.198+0000 7fceb5000640 1 -- 172.21.15.28:0/3689038137 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb015b570 con 0x7fceb006f3f0 2024-09-13T17:10:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.199+0000 7fceb5000640 1 --2- 172.21.15.28:0/3689038137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009e50 tx=0x7fcea802a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31169eb8c3c34221 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.199+0000 7fceaf400640 1 -- 172.21.15.28:0/3689038137 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea802aea0 con 0x7fceb006f3f0 2024-09-13T17:10:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.199+0000 7fceaf400640 1 -- 172.21.15.28:0/3689038137 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea802d050 con 0x7fceb006f3f0 2024-09-13T17:10:56.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.199+0000 7fceaf400640 1 -- 172.21.15.28:0/3689038137 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea8031820 con 0x7fceb006f3f0 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 -- 172.21.15.28:0/3689038137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 msgr2=0x7fceb015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 --2- 172.21.15.28:0/3689038137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fcea8009e50 tx=0x7fcea802a680 comp rx=0 tx=0).stop 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 -- 172.21.15.28:0/3689038137 shutdown_connections 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 --2- 172.21.15.28:0/3689038137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb015c2b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 --2- 172.21.15.28:0/3689038137 >> v2:172.21.15.189:3300/0 conn(0x7fceb0070430 0x7fceb015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 -- 172.21.15.28:0/3689038137 >> 172.21.15.28:0/3689038137 conn(0x7fceb00b0270 msgr2=0x7fceb00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 -- 172.21.15.28:0/3689038137 shutdown_connections 2024-09-13T17:10:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.200+0000 7fceb7000640 1 -- 172.21.15.28:0/3689038137 wait complete. 2024-09-13T17:10:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.202+0000 7fceb7000640 1 Processor -- start 2024-09-13T17:10:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.202+0000 7fceb7000640 1 -- start start 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb01428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0159770 0x7fceb00b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01553c0 con 0x7fceb0070430 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01f5b40 con 0x7fceb0159770 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceb01f5e20 con 0x7fceb006f3f0 2024-09-13T17:10:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fcea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fcea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60326/0 (socket says 172.21.15.28:60326) 2024-09-13T17:10:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fcea7e00640 1 -- 172.21.15.28:0/2919125489 learned_addr learned my addr 172.21.15.28:0/2919125489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceb5000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0159770 0x7fceb00b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.203+0000 7fceafe00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb01428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 -- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 msgr2=0x7fceb01428b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb01428b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 -- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0159770 msgr2=0x7fceb00b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0159770 0x7fceb00b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 -- 172.21.15.28:0/2919125489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea8009b30 con 0x7fceb0070430 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fcea7e00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fce9800eb40 tx=0x7fce9800d350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98010440 con 0x7fceb0070430 2024-09-13T17:10:56.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce980105a0 con 0x7fceb0070430 2024-09-13T17:10:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce98010950 con 0x7fceb0070430 2024-09-13T17:10:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceb01f6100 con 0x7fceb0070430 2024-09-13T17:10:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.204+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceb01f66c0 con 0x7fceb0070430 2024-09-13T17:10:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.205+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceb006fd20 con 0x7fceb0070430 2024-09-13T17:10:56.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.206+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fce98010ab0 con 0x7fceb0070430 2024-09-13T17:10:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.207+0000 7fceacc00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 0x7fce8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.207+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fce98089c20 con 0x7fceb0070430 2024-09-13T17:10:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.208+0000 7fceafe00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 0x7fce8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.208+0000 7fceafe00640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 0x7fce8c06d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcea0000f60 tx=0x7fcea0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.211+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fce98054df0 con 0x7fceb0070430 2024-09-13T17:10:56.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.429+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fceb0153010 con 0x7fceb0070430 2024-09-13T17:10:56.431 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:10:56.430+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:10:56.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.438+0000 7fceacc00640 1 -- 172.21.15.28:0/2919125489 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) v1 ==== 161+0+0 (secure 0 0 0) 0x7fce9801d090 con 0x7fceb0070430 2024-09-13T17:10:56.439 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 msgr2=0x7fce8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 0x7fce8c06d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fcea0000f60 tx=0x7fcea0009040 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 msgr2=0x7fceb01e2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fce9800eb40 tx=0x7fce9800d350 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 shutdown_connections 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce8c06ac00 0x7fce8c06d0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fceb0159770 0x7fceb00b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fceb0070430 0x7fceb01e2110 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 --2- 172.21.15.28:0/2919125489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fceb006f3f0 0x7fceb01428b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 >> 172.21.15.28:0/2919125489 conn(0x7fceb00b0270 msgr2=0x7fceb0159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 shutdown_connections 2024-09-13T17:10:56.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.441+0000 7fceb7000640 1 -- 172.21.15.28:0/2919125489 wait complete. 2024-09-13T17:10:56.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:10:56.511 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:10:56.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.618+0000 7f0d8f400640 1 Processor -- start 2024-09-13T17:10:56.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.618+0000 7f0d8f400640 1 -- start start 2024-09-13T17:10:56.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.619+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.619+0000 7f0d8f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.619+0000 7f0d8f400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8806f920 con 0x7f0d880add00 2024-09-13T17:10:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d8f400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8815b290 con 0x7f0d8806f3f0 2024-09-13T17:10:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d8f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815b570 con 0x7f0d88159770 2024-09-13T17:10:56.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d7fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d8ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d7fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:51920/0 (socket says 172.21.15.28:51920) 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.620+0000 7f0d7fe00640 1 -- 172.21.15.28:0/2911849964 learned_addr learned my addr 172.21.15.28:0/2911849964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.621+0000 7f0d8ca00640 1 -- 172.21.15.28:0/2911849964 >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 msgr2=0x7f0d88070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.621+0000 7f0d8ca00640 1 --2- 172.21.15.28:0/2911849964 >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.621+0000 7f0d8ca00640 1 -- 172.21.15.28:0/2911849964 >> v1:172.21.15.189:6789/0 conn(0x7f0d880add00 legacy=0x7f0d88159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.621+0000 7f0d8ca00640 1 -- 172.21.15.28:0/2911849964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8815b850 con 0x7f0d88159770 2024-09-13T17:10:56.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.621+0000 7f0d7fe00640 1 --2- 172.21.15.28:0/2911849964 >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:10:56.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.622+0000 7f0d8ca00640 1 --2- 172.21.15.28:0/2911849964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806fd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800ba40 tx=0x7f0d7802c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92f1d5d5ba56b422 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.622+0000 7f0d87e00640 1 -- 172.21.15.28:0/2911849964 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78031710 con 0x7f0d88159770 2024-09-13T17:10:56.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.622+0000 7f0d87e00640 1 -- 172.21.15.28:0/2911849964 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7802f050 con 0x7f0d88159770 2024-09-13T17:10:56.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d87e00640 1 -- 172.21.15.28:0/2911849964 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78031a90 con 0x7f0d88159770 2024-09-13T17:10:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2911849964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 msgr2=0x7f0d8806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2911849964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806fd20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800ba40 tx=0x7f0d7802c7b0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2911849964 shutdown_connections 2024-09-13T17:10:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2911849964 >> v2:172.21.15.189:3300/0 conn(0x7f0d8806f3f0 0x7f0d88070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2911849964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806fd20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2911849964 >> 172.21.15.28:0/2911849964 conn(0x7f0d880b0270 msgr2=0x7f0d880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:56.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2911849964 shutdown_connections 2024-09-13T17:10:56.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2911849964 wait complete. 2024-09-13T17:10:56.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.625+0000 7f0d8f400640 1 Processor -- start 2024-09-13T17:10:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.625+0000 7f0d8f400640 1 -- start start 2024-09-13T17:10:56.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d880add00 0x7f0d880c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8814a570 con 0x7f0d8806f3f0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880ca3a0 con 0x7f0d880add00 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880c9300 con 0x7f0d88159770 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60348/0 (socket says 172.21.15.28:60348) 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8ca00640 1 -- 172.21.15.28:0/4075357185 learned_addr learned my addr 172.21.15.28:0/4075357185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d7fe00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d880add00 0x7f0d880c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.626+0000 7f0d8d400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 -- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 msgr2=0x7f0d8806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 -- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d880add00 msgr2=0x7f0d880c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d880add00 0x7f0d880c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 -- 172.21.15.28:0/4075357185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d700094d0 con 0x7f0d8806f3f0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8ca00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800bb60 tx=0x7f0d780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78033910 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d78033a70 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7800b6a0 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78033e20 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.627+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d880c98c0 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.629+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8806fd20 con 0x7f0d8806f3f0 2024-09-13T17:10:56.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.629+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d7803c430 con 0x7f0d8806f3f0 2024-09-13T17:10:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.630+0000 7f0d85600640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 0x7f0d6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.630+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0d780aa250 con 0x7f0d8806f3f0 2024-09-13T17:10:56.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.631+0000 7f0d7fe00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 0x7f0d6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.632+0000 7f0d7fe00640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 0x7f0d6406d0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0d88158f20 tx=0x7f0d70009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.634+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d78075460 con 0x7f0d8806f3f0 2024-09-13T17:10:56.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:56.847+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0d880aff20 con 0x7f0d8806f3f0 2024-09-13T17:10:57.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.442+0000 7f0d85600640 1 -- 172.21.15.28:0/4075357185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0d78078a80 con 0x7f0d8806f3f0 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 msgr2=0x7f0d6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 0x7f0d6406d0c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0d88158f20 tx=0x7f0d70009240 comp rx=0 tx=0).stop 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 msgr2=0x7f0d8814a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0d7800bb60 tx=0x7f0d780046c0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 shutdown_connections 2024-09-13T17:10:57.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6406ac00 0x7f0d6406d0c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d88159770 0x7f0d8806ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d880add00 0x7f0d880c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 --2- 172.21.15.28:0/4075357185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d8806f3f0 0x7f0d8814a030 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 >> 172.21.15.28:0/4075357185 conn(0x7f0d880b0270 msgr2=0x7f0d880ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 shutdown_connections 2024-09-13T17:10:57.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.445+0000 7f0d8f400640 1 -- 172.21.15.28:0/4075357185 wait complete. 2024-09-13T17:10:57.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:10:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.625+0000 7fcf21e00640 1 Processor -- start 2024-09-13T17:10:57.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.626+0000 7fcf21e00640 1 -- start start 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf21e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 0x7fcf1c06fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf21e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c06bff0 con 0x7fcf1c0b1d30 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf21e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c0b3a00 con 0x7fcf1c0b2880 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf21e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c070160 con 0x7fcf1c15caf0 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf1a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf1c0b1d30 0x7fcf1c0b2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38476/0 (socket says 172.21.15.28:38476) 2024-09-13T17:10:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf1a000640 1 -- 172.21.15.28:0/670346559 learned_addr learned my addr 172.21.15.28:0/670346559 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.627+0000 7fcf1aa00640 1 --2- 172.21.15.28:0/670346559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 0x7fcf1c06fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271225550 0 0) 0x7fcf1c06bff0 con 0x7fcf1c0b1d30 2024-09-13T17:10:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf00003520 con 0x7fcf1c0b1d30 2024-09-13T17:10:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864984941 0 0) 0x7fcf1c0b3a00 con 0x7fcf1c0b2880 2024-09-13T17:10:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf1c06bff0 con 0x7fcf1c0b2880 2024-09-13T17:10:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2373661210 0 0) 0x7fcf00003520 con 0x7fcf1c0b1d30 2024-09-13T17:10:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf1aa00640 1 -- 172.21.15.28:0/670346559 >> v1:172.21.15.189:6789/0 conn(0x7fcf1c0b2880 legacy=0x7fcf1c06c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf1aa00640 1 -- 172.21.15.28:0/670346559 >> v1:172.21.15.28:6789/0 conn(0x7fcf1c0b1d30 legacy=0x7fcf1c0b2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.628+0000 7fcf1aa00640 1 -- 172.21.15.28:0/670346559 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0b3c70 con 0x7fcf1c15caf0 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.629+0000 7fcf1aa00640 1 --2- 172.21.15.28:0/670346559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 0x7fcf1c06fc20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097d0 tx=0x7fcf0c02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81d69aa0d92b249e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.629+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf08001670 con 0x7fcf1c15caf0 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.629+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c02ab90 con 0x7fcf1c15caf0 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.629+0000 7fcf19600640 1 -- 172.21.15.28:0/670346559 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c032440 con 0x7fcf1c15caf0 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 -- 172.21.15.28:0/670346559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 msgr2=0x7fcf1c06fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 --2- 172.21.15.28:0/670346559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 0x7fcf1c06fc20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0097d0 tx=0x7fcf0c02a100 comp rx=0 tx=0).stop 2024-09-13T17:10:57.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 -- 172.21.15.28:0/670346559 shutdown_connections 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 --2- 172.21.15.28:0/670346559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15caf0 0x7fcf1c06fc20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 -- 172.21.15.28:0/670346559 >> 172.21.15.28:0/670346559 conn(0x7fcf1c0cee00 msgr2=0x7fcf1c0cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 -- 172.21.15.28:0/670346559 shutdown_connections 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.630+0000 7fcf21e00640 1 -- 172.21.15.28:0/670346559 wait complete. 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.632+0000 7fcf21e00640 1 Processor -- start 2024-09-13T17:10:57.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.632+0000 7fcf21e00640 1 -- start start 2024-09-13T17:10:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c0b1d30 0x7fcf1c09beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 0x7fcf1c153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:57.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c09b1a0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c09c3f0 con 0x7fcf1c0b1d30 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf21e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c09a520 con 0x7fcf1c0b2880 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60360/0 (socket says 172.21.15.28:60360) 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 -- 172.21.15.28:0/4069651061 learned_addr learned my addr 172.21.15.28:0/4069651061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1aa00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c0b1d30 0x7fcf1c09beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1a000640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 0x7fcf1c153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 -- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 msgr2=0x7fcf1c153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 0x7fcf1c153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 -- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c0b1d30 msgr2=0x7fcf1c09beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:57.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c0b1d30 0x7fcf1c09beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1b400640 1 -- 172.21.15.28:0/4069651061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c009400 con 0x7fcf1c15caf0 2024-09-13T17:10:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.633+0000 7fcf1a000640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 0x7fcf1c153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:57.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf1b400640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400c7a0 tx=0x7fcf1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14003b70 con 0x7fcf1c15caf0 2024-09-13T17:10:57.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14003cd0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c09a800 con 0x7fcf1c15caf0 2024-09-13T17:10:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1400aa90 con 0x7fcf1c15caf0 2024-09-13T17:10:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.634+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c09adc0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.635+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c067a10 con 0x7fcf1c15caf0 2024-09-13T17:10:57.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.636+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf1400abf0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.637+0000 7fcf13400640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.637+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcf140892b0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.637+0000 7fcf1aa00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:57.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.638+0000 7fcf1aa00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c034000 tx=0x7fcf0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:57.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.642+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcf140543a0 con 0x7fcf1c15caf0 2024-09-13T17:10:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:57.826+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcf1c06bc00 con 0x7fcf1c15caf0 2024-09-13T17:10:58.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.447+0000 7fcf13400640 1 -- 172.21.15.28:0/4069651061 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) v1 ==== 181+0+0 (secure 0 0 0) 0x7fcf140579c0 con 0x7fcf1c15caf0 2024-09-13T17:10:58.449 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 msgr2=0x7fcefc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c034000 tx=0x7fcf0c02e000 comp rx=0 tx=0).stop 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 msgr2=0x7fcf1c153a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400c7a0 tx=0x7fcf1400cc60 comp rx=0 tx=0).stop 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 shutdown_connections 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c15caf0 0x7fcf1c153a50 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c0b2880 0x7fcf1c153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 --2- 172.21.15.28:0/4069651061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c0b1d30 0x7fcf1c09beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 >> 172.21.15.28:0/4069651061 conn(0x7fcf1c0cee00 msgr2=0x7fcf1c0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.450+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 shutdown_connections 2024-09-13T17:10:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.451+0000 7fcf21e00640 1 -- 172.21.15.28:0/4069651061 wait complete. 2024-09-13T17:10:58.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:10:58.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.635+0000 7fe12be00640 1 Processor -- start 2024-09-13T17:10:58.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.635+0000 7fe12be00640 1 -- start start 2024-09-13T17:10:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe12c0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12c15cc70 con 0x7fe12c06a590 2024-09-13T17:10:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12be00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12c0d08e0 con 0x7fe12c0d0370 2024-09-13T17:10:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0d0bc0 con 0x7fe12c15c890 2024-09-13T17:10:58.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe12c06a590 0x7fe12c0b5c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38478/0 (socket says 172.21.15.28:38478) 2024-09-13T17:10:58.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.636+0000 7fe12b400640 1 -- 172.21.15.28:0/700668589 learned_addr learned my addr 172.21.15.28:0/700668589 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:10:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302897662 0 0) 0x7fe12c15cc70 con 0x7fe12c06a590 2024-09-13T17:10:58.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe12aa00640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe12c0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110003520 con 0x7fe12c06a590 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2758621694 0 0) 0x7fe110003520 con 0x7fe12c06a590 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 msgr2=0x7fe12c0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe12c0cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 >> v1:172.21.15.189:6789/0 conn(0x7fe12c0d0370 legacy=0x7fe12c069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c0d0ea0 con 0x7fe12c06a590 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fe1240019f0 con 0x7fe12c06a590 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 >> v1:172.21.15.28:6789/0 conn(0x7fe12c06a590 legacy=0x7fe12c0b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1100039a0 0x7fe110003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe1100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110004d00 0x7fe110005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0d0bc0 con 0x7fe110004d00 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11000bac0 con 0x7fe1100039a0 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.637+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11000bc80 con 0x7fe12c15c890 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110004d00 0x7fe110005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12a000640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe1100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12aa00640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1100039a0 0x7fe110003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 -- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 msgr2=0x7fe1100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c15c890 0x7fe1100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 -- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1100039a0 msgr2=0x7fe110003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1100039a0 0x7fe110003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.638+0000 7fe12b400640 1 -- 172.21.15.28:0/700668589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110003520 con 0x7fe110004d00 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.639+0000 7fe12b400640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110004d00 0x7fe110005180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe12c0c9820 tx=0x7fe1100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd777cb4e95e8173 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.639+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12402b9c0 con 0x7fe110004d00 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.639+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12402bb20 con 0x7fe110004d00 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.639+0000 7fe129600640 1 -- 172.21.15.28:0/700668589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124034470 con 0x7fe110004d00 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 -- 172.21.15.28:0/700668589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110004d00 msgr2=0x7fe110005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 --2- 172.21.15.28:0/700668589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe110004d00 0x7fe110005180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fe12c0c9820 tx=0x7fe1100057f0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12b400640 1 -- 172.21.15.28:0/700668589 reap_dead start 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 -- 172.21.15.28:0/700668589 shutdown_connections 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 -- 172.21.15.28:0/700668589 >> 172.21.15.28:0/700668589 conn(0x7fe12c0708c0 msgr2=0x7fe12c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 -- 172.21.15.28:0/700668589 shutdown_connections 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.640+0000 7fe12be00640 1 -- 172.21.15.28:0/700668589 wait complete. 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.642+0000 7fe12be00640 1 Processor -- start 2024-09-13T17:10:58.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.642+0000 7fe12be00640 1 -- start start 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 0x7fe12c0c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe12c1f2a00 0x7fe12c155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c1f2eb0 con 0x7fe12c1f3170 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0c7b10 con 0x7fe12c1f2a00 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c0c7df0 con 0x7fe12c14f830 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60372/0 (socket says 172.21.15.28:60372) 2024-09-13T17:10:58.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 -- 172.21.15.28:0/3600188376 learned_addr learned my addr 172.21.15.28:0/3600188376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12aa00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 0x7fe12c0c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12b400640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe12c1f2a00 0x7fe12c155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 -- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 msgr2=0x7fe12c0c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 0x7fe12c0c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 -- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe12c1f2a00 msgr2=0x7fe12c155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe12c1f2a00 0x7fe12c155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12a000640 1 -- 172.21.15.28:0/3600188376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12c0c80d0 con 0x7fe12c1f3170 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.643+0000 7fe12aa00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 0x7fe12c0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe12a000640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe11800f7f0 tx=0x7fe11800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1180050d0 con 0x7fe12c1f3170 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe118005230 con 0x7fe12c1f3170 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c155700 con 0x7fe12c1f3170 2024-09-13T17:10:58.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11800d730 con 0x7fe12c1f3170 2024-09-13T17:10:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.644+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c14fc10 con 0x7fe12c1f3170 2024-09-13T17:10:58.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.645+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c155500 con 0x7fe12c1f3170 2024-09-13T17:10:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.646+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe11800d890 con 0x7fe12c1f3170 2024-09-13T17:10:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.647+0000 7fe122a00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 0x7fe10c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.647+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5357+0+0 (secure 0 0 0) 0x7fe1180891a0 con 0x7fe12c1f3170 2024-09-13T17:10:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.648+0000 7fe12aa00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 0x7fe10c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:58.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.648+0000 7fe12aa00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 0x7fe10c06d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe12c1f3620 tx=0x7fe11c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:58.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.651+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe118054440 con 0x7fe12c1f3170 2024-09-13T17:10:58.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:58.833+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe12c0c8260 con 0x7fe12c1f3170 2024-09-13T17:10:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.454+0000 7fe122a00640 1 -- 172.21.15.28:0/3600188376 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe118057a60 con 0x7fe12c1f3170 2024-09-13T17:10:59.456 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:10:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.458+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 msgr2=0x7fe10c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.458+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 0x7fe10c06d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe12c1f3620 tx=0x7fe11c008040 comp rx=0 tx=0).stop 2024-09-13T17:10:59.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.458+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 msgr2=0x7fe12c1f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.458+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe11800f7f0 tx=0x7fe11800fcb0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 shutdown_connections 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe10c06ac30 0x7fe10c06d0f0 secure :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe12c1f3620 tx=0x7fe11c008040 comp rx=0 tx=0).stop 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe12c1f2a00 0x7fe12c155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe12c1f3170 0x7fe12c1f24c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 --2- 172.21.15.28:0/3600188376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe12c14f830 0x7fe12c0c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 >> 172.21.15.28:0/3600188376 conn(0x7fe12c0708c0 msgr2=0x7fe12c0d15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 shutdown_connections 2024-09-13T17:10:59.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.459+0000 7fe12be00640 1 -- 172.21.15.28:0/3600188376 wait complete. 2024-09-13T17:10:59.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:10:59.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.648+0000 7fd7d2600640 1 Processor -- start 2024-09-13T17:10:59.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.648+0000 7fd7d2600640 1 -- start start 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7d2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7cc158cd0 0x7fd7cc1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7d2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 0x7fd7cc0b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7d2600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc1595f0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7d2600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc1598c0 con 0x7fd7cc158cd0 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7d2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc159ba0 con 0x7fd7cc06c110 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7cc158cd0 0x7fd7cc1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7cc158cd0 0x7fd7cc1590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60374/0 (socket says 172.21.15.28:60374) 2024-09-13T17:10:59.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 -- 172.21.15.28:0/88000696 learned_addr learned my addr 172.21.15.28:0/88000696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cbe00640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 0x7fd7cc0b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 -- 172.21.15.28:0/88000696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 msgr2=0x7fd7cc0b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 0x7fd7cc0b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 -- 172.21.15.28:0/88000696 >> v1:172.21.15.189:6789/0 conn(0x7fd7cc06ccd0 legacy=0x7fd7cc06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.650+0000 7fd7cb400640 1 -- 172.21.15.28:0/88000696 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc159e80 con 0x7fd7cc158cd0 2024-09-13T17:10:59.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.651+0000 7fd7cbe00640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 0x7fd7cc0b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:10:59.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.651+0000 7fd7cb400640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc158cd0 0x7fd7cc1590b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009870 tx=0x7fd7bc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d53028af44e4a19 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7caa00640 1 -- 172.21.15.28:0/88000696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc032440 con 0x7fd7cc158cd0 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7caa00640 1 -- 172.21.15.28:0/88000696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc0325a0 con 0x7fd7cc158cd0 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7caa00640 1 -- 172.21.15.28:0/88000696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc032930 con 0x7fd7cc158cd0 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 -- 172.21.15.28:0/88000696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc158cd0 msgr2=0x7fd7cc1590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc158cd0 0x7fd7cc1590b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009870 tx=0x7fd7bc02a210 comp rx=0 tx=0).stop 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 -- 172.21.15.28:0/88000696 shutdown_connections 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc06c110 0x7fd7cc0b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 --2- 172.21.15.28:0/88000696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc158cd0 0x7fd7cc1590b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.652+0000 7fd7d2600640 1 -- 172.21.15.28:0/88000696 >> 172.21.15.28:0/88000696 conn(0x7fd7cc0b0270 msgr2=0x7fd7cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.653+0000 7fd7d2600640 1 -- 172.21.15.28:0/88000696 shutdown_connections 2024-09-13T17:10:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.653+0000 7fd7d2600640 1 -- 172.21.15.28:0/88000696 wait complete. 2024-09-13T17:10:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.654+0000 7fd7d2600640 1 Processor -- start 2024-09-13T17:10:59.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.654+0000 7fd7d2600640 1 -- start start 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7cc06c110 0x7fd7cc149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc158cd0 0x7fd7cc143ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc148800 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc14a4c0 con 0x7fd7cc06c110 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7d2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc14a7a0 con 0x7fd7cc158cd0 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.655+0000 7fd7c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7cb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7cc06c110 0x7fd7cc149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60386/0 (socket says 172.21.15.28:60386) 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 -- 172.21.15.28:0/3647316199 learned_addr learned my addr 172.21.15.28:0/3647316199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:10:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7cbe00640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc158cd0 0x7fd7cc143ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 -- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc158cd0 msgr2=0x7fd7cc143ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc158cd0 0x7fd7cc143ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 -- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7cc06c110 msgr2=0x7fd7cc149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7cc06c110 0x7fd7cc149c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 -- 172.21.15.28:0/3647316199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0094d0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.656+0000 7fd7c2a00640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c430 tx=0x7fd7b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.657+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4011790 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.657+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b40118f0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.657+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc14aa80 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.657+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4011ce0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.657+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc0cb8d0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.658+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc1590b0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.659+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd7b401c460 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.660+0000 7fd7c3e00640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 0x7fd7a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:10:59.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.660+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd7b4088e90 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.660+0000 7fd7cb400640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 0x7fd7a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:10:59.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.661+0000 7fd7cb400640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 0x7fd7a806d0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc034000 tx=0x7fd7bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:10:59.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.664+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd7b4054410 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.849+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd7cc0ace10 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.853+0000 7fd7c3e00640 1 -- 172.21.15.28:0/3647316199 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd7b408a0a0 con 0x7fd7cc06ccd0 2024-09-13T17:10:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 msgr2=0x7fd7a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 0x7fd7a806d0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc034000 tx=0x7fd7bc02e000 comp rx=0 tx=0).stop 2024-09-13T17:10:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 msgr2=0x7fd7cc0b2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:10:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400c430 tx=0x7fd7b400c8f0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 shutdown_connections 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7a806ac00 0x7fd7a806d0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7cc158cd0 0x7fd7cc143ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7cc06ccd0 0x7fd7cc0b2950 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 --2- 172.21.15.28:0/3647316199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7cc06c110 0x7fd7cc149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 >> 172.21.15.28:0/3647316199 conn(0x7fd7cc0b0270 msgr2=0x7fd7cc1512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 shutdown_connections 2024-09-13T17:10:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:10:59.855+0000 7fd7d2600640 1 -- 172.21.15.28:0/3647316199 wait complete. 2024-09-13T17:10:59.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir' 2024-09-13T17:11:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.030+0000 7f517d200640 1 Processor -- start 2024-09-13T17:11:00.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.030+0000 7f517d200640 1 -- start start 2024-09-13T17:11:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f517d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f517815c830 0x7f51780d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f517d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f517d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51780b5870 con 0x7f51780b4700 2024-09-13T17:11:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f517d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51780b4b70 con 0x7f517815c830 2024-09-13T17:11:00.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f517d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51780d2560 con 0x7f51780cd710 2024-09-13T17:11:00.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f5177e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f51780b4700 0x7f51780704a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38490/0 (socket says 172.21.15.28:38490) 2024-09-13T17:11:00.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.031+0000 7f5177e00640 1 -- 172.21.15.28:0/4084271715 learned_addr learned my addr 172.21.15.28:0/4084271715 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176a00640 1 --2- 172.21.15.28:0/4084271715 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5177400640 1 --2- 172.21.15.28:0/4084271715 >> v2:172.21.15.189:3300/0 conn(0x7f517815c830 0x7f51780d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530364262 0 0) 0x7f51780b5870 con 0x7f51780b4700 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f515c003520 con 0x7f51780b4700 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176a00640 1 -- 172.21.15.28:0/4084271715 >> v2:172.21.15.189:3300/0 conn(0x7f517815c830 msgr2=0x7f51780d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176a00640 1 --2- 172.21.15.28:0/4084271715 >> v2:172.21.15.189:3300/0 conn(0x7f517815c830 0x7f51780d14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287953676 0 0) 0x7f515c003520 con 0x7f51780b4700 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176a00640 1 -- 172.21.15.28:0/4084271715 >> v1:172.21.15.28:6789/0 conn(0x7f51780b4700 legacy=0x7f51780704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.032+0000 7f5176a00640 1 -- 172.21.15.28:0/4084271715 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517815cc10 con 0x7f51780cd710 2024-09-13T17:11:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.033+0000 7f5176a00640 1 --2- 172.21.15.28:0/4084271715 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780b41c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5164009790 tx=0x7f516402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5addcd51c431c0c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51700019f0 con 0x7f51780cd710 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516402ab90 con 0x7f51780cd710 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f5176000640 1 -- 172.21.15.28:0/4084271715 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5164032440 con 0x7f51780cd710 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 -- 172.21.15.28:0/4084271715 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 msgr2=0x7f51780b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 --2- 172.21.15.28:0/4084271715 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780b41c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5164009790 tx=0x7f516402a100 comp rx=0 tx=0).stop 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 -- 172.21.15.28:0/4084271715 shutdown_connections 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 --2- 172.21.15.28:0/4084271715 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780b41c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 --2- 172.21.15.28:0/4084271715 >> v2:172.21.15.189:3300/0 conn(0x7f517815c830 0x7f51780d14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 -- 172.21.15.28:0/4084271715 >> 172.21.15.28:0/4084271715 conn(0x7f51780cf3a0 msgr2=0x7f51780cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 -- 172.21.15.28:0/4084271715 shutdown_connections 2024-09-13T17:11:00.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.034+0000 7f517d200640 1 -- 172.21.15.28:0/4084271715 wait complete. 2024-09-13T17:11:00.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.036+0000 7f517d200640 1 Processor -- start 2024-09-13T17:11:00.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.036+0000 7f517d200640 1 -- start start 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51780b4700 0x7f51780cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178157190 con 0x7f517815c830 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178158db0 con 0x7f51780b4700 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.037+0000 7f517d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51781573e0 con 0x7f51780cd710 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60390/0 (socket says 172.21.15.28:60390) 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 -- 172.21.15.28:0/166155241 learned_addr learned my addr 172.21.15.28:0/166155241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177400640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51780b4700 0x7f51780cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5176a00640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 -- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 msgr2=0x7f51780cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 -- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51780b4700 msgr2=0x7f51780cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51780b4700 0x7f51780cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 -- 172.21.15.28:0/166155241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640093a0 con 0x7f517815c830 2024-09-13T17:11:00.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.038+0000 7f5177e00640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f517000cc70 tx=0x7f517000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.039+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517000ce80 con 0x7f517815c830 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.039+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5170012400 con 0x7f517815c830 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.039+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51700127f0 con 0x7f517815c830 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.039+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51781576c0 con 0x7f517815c830 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.039+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51780ca940 con 0x7f517815c830 2024-09-13T17:11:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.040+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5178157be0 con 0x7f517815c830 2024-09-13T17:11:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.041+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5170007070 con 0x7f517815c830 2024-09-13T17:11:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.042+0000 7f516f400640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.042+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5170089200 con 0x7f517815c830 2024-09-13T17:11:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.042+0000 7f5177400640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.043+0000 7f5177400640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f51780ce510 tx=0x7f5168008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.046+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5178157be0 con 0x7f517815c830 2024-09-13T17:11:00.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.307+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]} v 0) v1 -- 0x7f5178157be0 con 0x7f517815c830 2024-09-13T17:11:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.454+0000 7f516f400640 1 -- 172.21.15.28:0/166155241 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_flag_scrub_mdsdir"]}]=0 v154) v1 ==== 148+0+0 (secure 0 0 0) 0x7f5178157be0 con 0x7f517815c830 2024-09-13T17:11:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 msgr2=0x7f515806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f51780ce510 tx=0x7f5168008040 comp rx=0 tx=0).stop 2024-09-13T17:11:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 msgr2=0x7f51781558c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f517000cc70 tx=0x7f517000a430 comp rx=0 tx=0).stop 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 shutdown_connections 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517815c830 0x7f51781558c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f51780cd710 0x7f51780cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 --2- 172.21.15.28:0/166155241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51780b4700 0x7f51780cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 >> 172.21.15.28:0/166155241 conn(0x7f51780cf3a0 msgr2=0x7f51780b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 shutdown_connections 2024-09-13T17:11:00.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.457+0000 7f517d200640 1 -- 172.21.15.28:0/166155241 wait complete. 2024-09-13T17:11:00.475 INFO:tasks.cephfs_test_runner:test flag scrub_mdsdir ... ok 2024-09-13T17:11:00.475 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:11:00.476 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.630+0000 7f3551a00640 1 Processor -- start 2024-09-13T17:11:00.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.630+0000 7f3551a00640 1 -- start start 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f3551a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f354c0d2230 0x7f354c0d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f3551a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c0d1190 con 0x7f354c0b4930 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f3551a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354c0d1470 con 0x7f354c15c870 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f3551a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f354c0d1750 con 0x7f354c0d2230 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f354b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f354c0b4930 0x7f354c0b58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38498/0 (socket says 172.21.15.28:38498) 2024-09-13T17:11:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.631+0000 7f354b400640 1 -- 172.21.15.28:0/3676947807 learned_addr learned my addr 172.21.15.28:0/3676947807 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f3551000640 1 --2- 172.21.15.28:0/3676947807 >> v2:172.21.15.189:3300/0 conn(0x7f354c0d2230 0x7f354c0d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869649374 0 0) 0x7f354c0d1190 con 0x7f354c0b4930 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530003520 con 0x7f354c0b4930 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538741549 0 0) 0x7f354c0d1470 con 0x7f354c15c870 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f354c0d1190 con 0x7f354c15c870 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884790668 0 0) 0x7f3530003520 con 0x7f354c0b4930 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 >> v2:172.21.15.189:3300/0 conn(0x7f354c0d2230 msgr2=0x7f354c0d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 --2- 172.21.15.28:0/3676947807 >> v2:172.21.15.189:3300/0 conn(0x7f354c0d2230 0x7f354c0d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 >> v1:172.21.15.189:6789/0 conn(0x7f354c15c870 legacy=0x7f354c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0d1a30 con 0x7f354c0b4930 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3538001670 con 0x7f354c0b4930 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 >> v1:172.21.15.28:6789/0 conn(0x7f354c0b4930 legacy=0x7f354c0b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 0x7f3530003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35300047c0 0x7f3530004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3530005180 0x7f353000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0d1750 con 0x7f3530003b50 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f353000bfa0 con 0x7f35300047c0 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.632+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f353000c160 con 0x7f3530005180 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 0x7f3530003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354b400640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35300047c0 0x7f3530004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f3551000640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3530005180 0x7f353000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 -- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3530005180 msgr2=0x7f353000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3530005180 0x7f353000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 -- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35300047c0 msgr2=0x7f3530004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35300047c0 0x7f3530004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f354be00640 1 -- 172.21.15.28:0/3676947807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3530003520 con 0x7f3530003b50 2024-09-13T17:11:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.633+0000 7f3551000640 1 -- 172.21.15.28:0/3676947807 reap_dead start 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.634+0000 7f354be00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 0x7f3530003f50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f353c0018d0 tx=0x7f35300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c48ced3eee272b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.634+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c0052d0 con 0x7f3530003b50 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.634+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c005430 con 0x7f3530003b50 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.634+0000 7f354aa00640 1 -- 172.21.15.28:0/3676947807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c02ac60 con 0x7f3530003b50 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 -- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 msgr2=0x7f3530003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 0x7f3530003f50 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f353c0018d0 tx=0x7f35300057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 -- 172.21.15.28:0/3676947807 shutdown_connections 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 --2- 172.21.15.28:0/3676947807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3530003b50 0x7f3530003f50 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 -- 172.21.15.28:0/3676947807 >> 172.21.15.28:0/3676947807 conn(0x7f354c0cf400 msgr2=0x7f354c0cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 -- 172.21.15.28:0/3676947807 shutdown_connections 2024-09-13T17:11:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.635+0000 7f3551a00640 1 -- 172.21.15.28:0/3676947807 wait complete. 2024-09-13T17:11:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.637+0000 7f3551a00640 1 Processor -- start 2024-09-13T17:11:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.637+0000 7f3551a00640 1 -- start start 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3530003b50 0x7f354c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f354c15f480 0x7f354c15f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0b3560 con 0x7f354c0b2ba0 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354c14e680 con 0x7f3530003b50 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f354c0c8620 con 0x7f354c15f480 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60412/0 (socket says 172.21.15.28:60412) 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f3551000640 1 -- 172.21.15.28:0/816332765 learned_addr learned my addr 172.21.15.28:0/816332765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.638+0000 7f354b400640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f354c15f480 0x7f354c15f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f354be00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3530003b50 0x7f354c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 -- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f354c15f480 msgr2=0x7f354c15f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f354c15f480 0x7f354c15f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 -- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3530003b50 msgr2=0x7f354c14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3530003b50 0x7f354c14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 -- 172.21.15.28:0/816332765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0c8900 con 0x7f354c0b2ba0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3551000640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b5000 tx=0x7f354400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354400e040 con 0x7f354c0b2ba0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354400f040 con 0x7f354c0b2ba0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.639+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544015430 con 0x7f354c0b2ba0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.640+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f354c0c8be0 con 0x7f354c0b2ba0 2024-09-13T17:11:00.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.640+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f354c0c8310 con 0x7f354c0b2ba0 2024-09-13T17:11:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.641+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c0d1470 con 0x7f354c0b2ba0 2024-09-13T17:11:00.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.642+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3544015590 con 0x7f354c0b2ba0 2024-09-13T17:11:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.642+0000 7f3543e00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 0x7f352c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.643+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f35440896b0 con 0x7f354c0b2ba0 2024-09-13T17:11:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.643+0000 7f354be00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 0x7f352c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.644+0000 7f354be00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 0x7f352c06d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f353c032000 tx=0x7f353c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:00.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.646+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3544054c30 con 0x7f354c0b2ba0 2024-09-13T17:11:00.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.828+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f354c15b7d0 con 0x7f354c0b2ba0 2024-09-13T17:11:00.831 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:00.832 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:10:59.445602+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/696058040":"2024-09-14T17:10:56.430299+0000","172.21.15.28:6833/696058040":"2024-09-14T17:10:56.430299+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:01.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.829+0000 7f3543e00640 1 -- 172.21.15.28:0/816332765 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+12984 (secure 0 0 0) 0x7f3544058250 con 0x7f354c0b2ba0 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 msgr2=0x7f352c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 0x7f352c06d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f353c032000 tx=0x7f353c009030 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 msgr2=0x7f354c0b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f354c0b5000 tx=0x7f354400f580 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 shutdown_connections 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352c06ac00 0x7f352c06d0c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f354c0b2ba0 0x7f354c0b3020 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f354c15f480 0x7f354c15f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 --2- 172.21.15.28:0/816332765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3530003b50 0x7f354c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 >> 172.21.15.28:0/816332765 conn(0x7f354c0cf400 msgr2=0x7f354c0b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 shutdown_connections 2024-09-13T17:11:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:00.832+0000 7f3551a00640 1 -- 172.21.15.28:0/816332765 wait complete. 2024-09-13T17:11:01.119 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:11:01.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:11:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.282+0000 7f1afe600640 1 Processor -- start 2024-09-13T17:11:01.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.282+0000 7f1afe600640 1 -- start start 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1afe600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 0x7f1af806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1afe600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af806c8b0 con 0x7f1af815ab60 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1afe600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af806cb90 con 0x7f1af815a780 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1afe600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af80cc980 con 0x7f1af8071490 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1af6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1af815ab60 0x7f1af806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38508/0 (socket says 172.21.15.28:38508) 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1af6a00640 1 -- 172.21.15.28:0/110089389 learned_addr learned my addr 172.21.15.28:0/110089389 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:01.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1af7e00640 1 --2- 172.21.15.28:0/110089389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 0x7f1af806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074572898 0 0) 0x7f1af806c8b0 con 0x7f1af815ab60 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.284+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1adc003520 con 0x7f1af815ab60 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.285+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717399608 0 0) 0x7f1af806cb90 con 0x7f1af815a780 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.285+0000 7f1af7e00640 1 -- 172.21.15.28:0/110089389 >> v1:172.21.15.189:6789/0 conn(0x7f1af815a780 legacy=0x7f1af8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.285+0000 7f1af7e00640 1 -- 172.21.15.28:0/110089389 >> v1:172.21.15.28:6789/0 conn(0x7f1af815ab60 legacy=0x7f1af806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.285+0000 7f1af7e00640 1 -- 172.21.15.28:0/110089389 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af80ccc60 con 0x7f1af8071490 2024-09-13T17:11:01.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.286+0000 7f1af7e00640 1 --2- 172.21.15.28:0/110089389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 0x7f1af806c250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009dc0 tx=0x7f1af002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b8af56851c130ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.286+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0007740 con 0x7f1af8071490 2024-09-13T17:11:01.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.286+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af002d050 con 0x7f1af8071490 2024-09-13T17:11:01.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.286+0000 7f1af6000640 1 -- 172.21.15.28:0/110089389 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0007ac0 con 0x7f1af8071490 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 -- 172.21.15.28:0/110089389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 msgr2=0x7f1af806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 --2- 172.21.15.28:0/110089389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 0x7f1af806c250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009dc0 tx=0x7f1af002a6d0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 -- 172.21.15.28:0/110089389 shutdown_connections 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 --2- 172.21.15.28:0/110089389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af8071490 0x7f1af806c250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 -- 172.21.15.28:0/110089389 >> 172.21.15.28:0/110089389 conn(0x7f1af80b0270 msgr2=0x7f1af80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 -- 172.21.15.28:0/110089389 shutdown_connections 2024-09-13T17:11:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.287+0000 7f1afe600640 1 -- 172.21.15.28:0/110089389 wait complete. 2024-09-13T17:11:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.288+0000 7f1afe600640 1 Processor -- start 2024-09-13T17:11:01.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.288+0000 7f1afe600640 1 -- start start 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af8071490 0x7f1af8150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af815ab60 0x7f1af814ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af814f7c0 con 0x7f1af815a780 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af814faa0 con 0x7f1af8071490 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.289+0000 7f1afe600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af814fd80 con 0x7f1af815ab60 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60416/0 (socket says 172.21.15.28:60416) 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 -- 172.21.15.28:0/2390485205 learned_addr learned my addr 172.21.15.28:0/2390485205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af7e00640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af815ab60 0x7f1af814ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af7400640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af8071490 0x7f1af8150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 -- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af815ab60 msgr2=0x7f1af814ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af815ab60 0x7f1af814ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 -- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af8071490 msgr2=0x7f1af8150860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af8071490 0x7f1af8150860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.290+0000 7f1af6a00640 1 -- 172.21.15.28:0/2390485205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0009a70 con 0x7f1af815a780 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1af6a00640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400c390 tx=0x7f1ae400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4005ce0 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae4005e40 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40072f0 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af8150060 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.291+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af815df30 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.292+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af8070d80 con 0x7f1af815a780 2024-09-13T17:11:01.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.293+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ae4007450 con 0x7f1af815a780 2024-09-13T17:11:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.294+0000 7f1aef400640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.294+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1ae40885c0 con 0x7f1af815a780 2024-09-13T17:11:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.294+0000 7f1af7400640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.295+0000 7f1af7400640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8000f60 tx=0x7f1ae8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.298+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1ae4053b40 con 0x7f1af815a780 2024-09-13T17:11:01.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.479+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1af8070510 con 0x7f1af815a780 2024-09-13T17:11:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.480+0000 7f1aef400640 1 -- 172.21.15.28:0/2390485205 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ae40170a0 con 0x7f1af815a780 2024-09-13T17:11:01.482 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 msgr2=0x7f1ad806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8000f60 tx=0x7f1ae8009040 comp rx=0 tx=0).stop 2024-09-13T17:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 msgr2=0x7f1af814d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400c390 tx=0x7f1ae400c850 comp rx=0 tx=0).stop 2024-09-13T17:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 shutdown_connections 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af815ab60 0x7f1af814ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af815a780 0x7f1af814d180 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 --2- 172.21.15.28:0/2390485205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af8071490 0x7f1af8150860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.483+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 >> 172.21.15.28:0/2390485205 conn(0x7f1af80b0270 msgr2=0x7f1af80cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.484+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 shutdown_connections 2024-09-13T17:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.484+0000 7f1afe600640 1 -- 172.21.15.28:0/2390485205 wait complete. 2024-09-13T17:11:01.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks' 2024-09-13T17:11:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.657+0000 7f2137000640 1 Processor -- start 2024-09-13T17:11:01.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.657+0000 7f2137000640 1 -- start start 2024-09-13T17:11:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21301592a0 0x7f213015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f213006f970 0x7f2130159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f213006c2d0 0x7f213006c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213015b0b0 con 0x7f213006c2d0 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f213006ccc0 con 0x7f213006f970 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2137000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21300b2040 con 0x7f21301592a0 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2135000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f213006c2d0 0x7f213006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2135000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f213006c2d0 0x7f213006c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60418/0 (socket says 172.21.15.28:60418) 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2135000640 1 -- 172.21.15.28:0/4278499445 learned_addr learned my addr 172.21.15.28:0/4278499445 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f212fe00640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21301592a0 0x7f213015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2127e00640 1 --2- 172.21.15.28:0/4278499445 >> v2:172.21.15.189:3300/0 conn(0x7f213006f970 0x7f2130159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2135000640 1 -- 172.21.15.28:0/4278499445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21301592a0 msgr2=0x7f213015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.659+0000 7f2135000640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21301592a0 0x7f213015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.660+0000 7f2135000640 1 -- 172.21.15.28:0/4278499445 >> v2:172.21.15.189:3300/0 conn(0x7f213006f970 msgr2=0x7f2130159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.660+0000 7f2135000640 1 --2- 172.21.15.28:0/4278499445 >> v2:172.21.15.189:3300/0 conn(0x7f213006f970 0x7f2130159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.660+0000 7f2135000640 1 -- 172.21.15.28:0/4278499445 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130159ff0 con 0x7f213006c2d0 2024-09-13T17:11:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2135000640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006c2d0 0x7f213006c780 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f213006ae00 tx=0x7f212802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88da13df273a059 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f212f400640 1 -- 172.21.15.28:0/4278499445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128031600 con 0x7f213006c2d0 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f212f400640 1 -- 172.21.15.28:0/4278499445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212802d050 con 0x7f213006c2d0 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f212f400640 1 -- 172.21.15.28:0/4278499445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2128031980 con 0x7f213006c2d0 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 -- 172.21.15.28:0/4278499445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006c2d0 msgr2=0x7f213006c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006c2d0 0x7f213006c780 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f213006ae00 tx=0x7f212802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 -- 172.21.15.28:0/4278499445 shutdown_connections 2024-09-13T17:11:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006c2d0 0x7f213006c780 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 --2- 172.21.15.28:0/4278499445 >> v2:172.21.15.189:3300/0 conn(0x7f213006f970 0x7f2130159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 --2- 172.21.15.28:0/4278499445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21301592a0 0x7f213015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 -- 172.21.15.28:0/4278499445 >> 172.21.15.28:0/4278499445 conn(0x7f2130069e80 msgr2=0x7f213006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.661+0000 7f2137000640 1 -- 172.21.15.28:0/4278499445 shutdown_connections 2024-09-13T17:11:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.662+0000 7f2137000640 1 -- 172.21.15.28:0/4278499445 wait complete. 2024-09-13T17:11:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.663+0000 7f2137000640 1 Processor -- start 2024-09-13T17:11:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.663+0000 7f2137000640 1 -- start start 2024-09-13T17:11:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f213006c2d0 0x7f2130150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21301592a0 0x7f2130148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213014dfe0 con 0x7f213006f970 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2130148b90 con 0x7f21301592a0 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2137000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213014a090 con 0x7f213006c2d0 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2127e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2127e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60432/0 (socket says 172.21.15.28:60432) 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.664+0000 7f2127e00640 1 -- 172.21.15.28:0/2875131223 learned_addr learned my addr 172.21.15.28:0/2875131223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 -- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f213006c2d0 msgr2=0x7f2130150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f213006c2d0 0x7f2130150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 -- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21301592a0 msgr2=0x7f2130148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21301592a0 0x7f2130148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 -- 172.21.15.28:0/2875131223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128009b30 con 0x7f213006f970 2024-09-13T17:11:01.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2127e00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f211800e9b0 tx=0x7f211800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2118009670 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211800c9a0 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211800cd30 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2130149250 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.665+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2130149760 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.666+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213006c780 con 0x7f213006f970 2024-09-13T17:11:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.667+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2118013470 con 0x7f213006f970 2024-09-13T17:11:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.668+0000 7f212cc00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 0x7f210c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.669+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2118089310 con 0x7f213006f970 2024-09-13T17:11:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.669+0000 7f212fe00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 0x7f210c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.670+0000 7f212fe00640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 0x7f210c06d0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f21200099a0 tx=0x7f2120009460 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:01.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.673+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2118054750 con 0x7f213006f970 2024-09-13T17:11:01.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:01.917+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]} v 0) v1 -- 0x7f21301545a0 con 0x7f213006f970 2024-09-13T17:11:02.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.462+0000 7f212cc00640 1 -- 172.21.15.28:0/2875131223 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]}]=0 v156) v1 ==== 146+0+0 (secure 0 0 0) 0x7f2118057d70 con 0x7f213006f970 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 msgr2=0x7f210c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 0x7f210c06d0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f21200099a0 tx=0x7f2120009460 comp rx=0 tx=0).stop 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 msgr2=0x7f2130148650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f211800e9b0 tx=0x7f211800ee70 comp rx=0 tx=0).stop 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 shutdown_connections 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f210c06ac00 0x7f210c06d0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21301592a0 0x7f2130148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f213006f970 0x7f2130148650 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 --2- 172.21.15.28:0/2875131223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f213006c2d0 0x7f2130150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 >> 172.21.15.28:0/2875131223 conn(0x7f2130069e80 msgr2=0x7f21300afd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 shutdown_connections 2024-09-13T17:11:02.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.465+0000 7f2137000640 1 -- 172.21.15.28:0/2875131223 wait complete. 2024-09-13T17:11:02.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:11:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.647+0000 7fb057200640 1 Processor -- start 2024-09-13T17:11:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.647+0000 7fb057200640 1 -- start start 2024-09-13T17:11:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb058158080 0x7fb0580b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb0580cee50 0x7fb058070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0580cf290 0x7fb0580711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb05815b9d0 con 0x7fb0580cf290 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0580b2090 con 0x7fb0580cee50 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb057200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb05815ae50 con 0x7fb058158080 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0580cf290 0x7fb0580711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb0580cf290 0x7fb0580711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60442/0 (socket says 172.21.15.28:60442) 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 -- 172.21.15.28:0/133889270 learned_addr learned my addr 172.21.15.28:0/133889270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb055e00640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb058158080 0x7fb0580b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb055400640 1 --2- 172.21.15.28:0/133889270 >> v2:172.21.15.189:3300/0 conn(0x7fb0580cee50 0x7fb058070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 -- 172.21.15.28:0/133889270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb058158080 msgr2=0x7fb0580b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb058158080 0x7fb0580b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 -- 172.21.15.28:0/133889270 >> v2:172.21.15.189:3300/0 conn(0x7fb0580cee50 msgr2=0x7fb058070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 --2- 172.21.15.28:0/133889270 >> v2:172.21.15.189:3300/0 conn(0x7fb0580cee50 0x7fb058070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.649+0000 7fb056800640 1 -- 172.21.15.28:0/133889270 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05815b130 con 0x7fb0580cf290 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.650+0000 7fb056800640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580711f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb0580cfad0 tx=0x7fb05002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba82f8556ef255b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb054a00640 1 -- 172.21.15.28:0/133889270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050031600 con 0x7fb0580cf290 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb054a00640 1 -- 172.21.15.28:0/133889270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05002d050 con 0x7fb0580cf290 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb054a00640 1 -- 172.21.15.28:0/133889270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050031980 con 0x7fb0580cf290 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 -- 172.21.15.28:0/133889270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 msgr2=0x7fb0580711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580711f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fb0580cfad0 tx=0x7fb05002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 -- 172.21.15.28:0/133889270 shutdown_connections 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580711f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 --2- 172.21.15.28:0/133889270 >> v2:172.21.15.189:3300/0 conn(0x7fb0580cee50 0x7fb058070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 --2- 172.21.15.28:0/133889270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb058158080 0x7fb0580b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 -- 172.21.15.28:0/133889270 >> 172.21.15.28:0/133889270 conn(0x7fb058069e80 msgr2=0x7fb05806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:02.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 -- 172.21.15.28:0/133889270 shutdown_connections 2024-09-13T17:11:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.651+0000 7fb057200640 1 -- 172.21.15.28:0/133889270 wait complete. 2024-09-13T17:11:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.653+0000 7fb057200640 1 Processor -- start 2024-09-13T17:11:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.653+0000 7fb057200640 1 -- start start 2024-09-13T17:11:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0580cee50 0x7fb058159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb058158080 0x7fb058158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580cd6b0 con 0x7fb0580cf290 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580cd990 con 0x7fb058158080 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb057200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580cdc70 con 0x7fb0580cee50 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb055400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb055e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0580cee50 0x7fb058159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb055400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60454/0 (socket says 172.21.15.28:60454) 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.654+0000 7fb055400640 1 -- 172.21.15.28:0/946513255 learned_addr learned my addr 172.21.15.28:0/946513255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb056800640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb058158080 0x7fb058158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 -- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0580cee50 msgr2=0x7fb058159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0580cee50 0x7fb058159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 -- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb058158080 msgr2=0x7fb058158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb058158080 0x7fb058158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 -- 172.21.15.28:0/946513255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb050009b30 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb055400640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb04400d7d0 tx=0x7fb04400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.655+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044004020 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.656+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0580cdf50 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.656+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0440041a0 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.656+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb044004350 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.656+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05815a560 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.657+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0580b1b50 con 0x7fb0580cf290 2024-09-13T17:11:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.658+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb0440096a0 con 0x7fb0580cf290 2024-09-13T17:11:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.658+0000 7fb04e000640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 0x7fb03806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.659+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb0440891a0 con 0x7fb0580cf290 2024-09-13T17:11:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.659+0000 7fb055e00640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 0x7fb03806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.660+0000 7fb055e00640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 0x7fb03806d0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb0580b41f0 tx=0x7fb048008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.663+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb044054610 con 0x7fb0580cf290 2024-09-13T17:11:02.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.842+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb05815a560 con 0x7fb0580cf290 2024-09-13T17:11:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.847+0000 7fb04e000640 1 -- 172.21.15.28:0/946513255 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb05815a560 con 0x7fb0580cf290 2024-09-13T17:11:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 msgr2=0x7fb03806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 0x7fb03806d0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb0580b41f0 tx=0x7fb048008040 comp rx=0 tx=0).stop 2024-09-13T17:11:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 msgr2=0x7fb0580cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fb04400d7d0 tx=0x7fb04400dc90 comp rx=0 tx=0).stop 2024-09-13T17:11:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 shutdown_connections 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb03806ac00 0x7fb03806d0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb058158080 0x7fb058158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0580cf290 0x7fb0580cb5d0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 --2- 172.21.15.28:0/946513255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0580cee50 0x7fb058159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 >> 172.21.15.28:0/946513255 conn(0x7fb058069e80 msgr2=0x7fb0580ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 shutdown_connections 2024-09-13T17:11:02.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:02.849+0000 7fb057200640 1 -- 172.21.15.28:0/946513255 wait complete. 2024-09-13T17:11:02.866 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:11:02.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:02.952 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:03.044 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:11:03.044 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:03.044 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:03.080 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:11:03.080 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:03.080 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:03.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:03.118 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:03.118 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:03.156 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:11:03.156 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:03.156 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:03.194 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:03.194 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:11:03.194 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.350+0000 7f07d0800640 1 Processor -- start 2024-09-13T17:11:03.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.350+0000 7f07d0800640 1 -- start start 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.351+0000 7f07d0800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8146170 0x7f07c815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.351+0000 7f07d0800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07c8160310 0x7f07c8160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.351+0000 7f07d0800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c8146550 con 0x7f07c80ac850 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.351+0000 7f07d0800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c815cd20 con 0x7f07c8160310 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.351+0000 7f07d0800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c815d280 con 0x7f07c8146170 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07c8160310 0x7f07c8160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f07c8160310 0x7f07c8160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60470/0 (socket says 172.21.15.28:60470) 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 -- 172.21.15.28:0/1885307967 learned_addr learned my addr 172.21.15.28:0/1885307967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cde00640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8146170 0x7f07c815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 -- 172.21.15.28:0/1885307967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8146170 msgr2=0x7f07c815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:03.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8146170 0x7f07c815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 -- 172.21.15.28:0/1885307967 >> v1:172.21.15.189:6789/0 conn(0x7f07c80ac850 legacy=0x7f07c806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.352+0000 7f07cd400640 1 -- 172.21.15.28:0/1885307967 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c815dc50 con 0x7f07c8160310 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.353+0000 7f07cd400640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c8160310 0x7f07c8160750 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009870 tx=0x7f07b802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d736a5242c50d74c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.353+0000 7f07cca00640 1 -- 172.21.15.28:0/1885307967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8032440 con 0x7f07c8160310 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.353+0000 7f07cca00640 1 -- 172.21.15.28:0/1885307967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b80325a0 con 0x7f07c8160310 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.353+0000 7f07cca00640 1 -- 172.21.15.28:0/1885307967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8032930 con 0x7f07c8160310 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 -- 172.21.15.28:0/1885307967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c8160310 msgr2=0x7f07c8160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c8160310 0x7f07c8160750 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009870 tx=0x7f07b802a250 comp rx=0 tx=0).stop 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 -- 172.21.15.28:0/1885307967 shutdown_connections 2024-09-13T17:11:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c8160310 0x7f07c8160750 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 --2- 172.21.15.28:0/1885307967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8146170 0x7f07c815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 -- 172.21.15.28:0/1885307967 >> 172.21.15.28:0/1885307967 conn(0x7f07c8069e80 msgr2=0x7f07c806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 -- 172.21.15.28:0/1885307967 shutdown_connections 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.354+0000 7f07d0800640 1 -- 172.21.15.28:0/1885307967 wait complete. 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.356+0000 7f07d0800640 1 Processor -- start 2024-09-13T17:11:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.356+0000 7f07d0800640 1 -- start start 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c8146170 0x7f07c8154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8160310 0x7f07c8159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8156c70 con 0x7f07c80ac850 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8155150 con 0x7f07c8146170 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07d0800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8155430 con 0x7f07c8160310 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60480/0 (socket says 172.21.15.28:60480) 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 -- 172.21.15.28:0/836022877 learned_addr learned my addr 172.21.15.28:0/836022877 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cd400640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c8146170 0x7f07c8154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07ce800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8160310 0x7f07c8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 -- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8160310 msgr2=0x7f07c8159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8160310 0x7f07c8159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 -- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c8146170 msgr2=0x7f07c8154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c8146170 0x7f07c8154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.357+0000 7f07cde00640 1 -- 172.21.15.28:0/836022877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b80094d0 con 0x7f07c80ac850 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07cde00640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e9b0 tx=0x7f07bc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc009670 con 0x7f07c80ac850 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc00c9a0 con 0x7f07c80ac850 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c8155710 con 0x7f07c80ac850 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00cd50 con 0x7f07c80ac850 2024-09-13T17:11:03.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.358+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c8143ab0 con 0x7f07c80ac850 2024-09-13T17:11:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.359+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c815d710 con 0x7f07c80ac850 2024-09-13T17:11:03.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.360+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f07bc013470 con 0x7f07c80ac850 2024-09-13T17:11:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.361+0000 7f07c2000640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 0x7f07ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.362+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f07bc089250 con 0x7f07c80ac850 2024-09-13T17:11:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.362+0000 7f07cd400640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 0x7f07ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.363+0000 7f07cd400640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 0x7f07ac06d0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f07b8034000 tx=0x7f07b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.366+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f07bc0547d0 con 0x7f07c80ac850 2024-09-13T17:11:03.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.578+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07c81558a0 con 0x7f07c80ac850 2024-09-13T17:11:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.579+0000 7f07c2000640 1 -- 172.21.15.28:0/836022877 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f07bc017de0 con 0x7f07c80ac850 2024-09-13T17:11:03.581 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:03.581 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":29,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3883232845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3883232845},{"type":"v1","addr":"172.21.15.189:6836","nonce":3883232845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5011,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3366434063","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3366434063},{"type":"v1","addr":"172.21.15.189:6837","nonce":3366434063}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5118,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2501858310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2501858310},{"type":"v1","addr":"172.21.15.28:6834","nonce":2501858310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5502,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/222983847","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":222983847},{"type":"v1","addr":"172.21.15.28:6835","nonce":222983847}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":29}],"filesystems":[]} 2024-09-13T17:11:03.581 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 29 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 msgr2=0x7f07ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 0x7f07ac06d0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f07b8034000 tx=0x7f07b802e000 comp rx=0 tx=0).stop 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 msgr2=0x7f07c8143570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00e9b0 tx=0x7f07bc00ee70 comp rx=0 tx=0).stop 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 shutdown_connections 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07ac06ac00 0x7f07ac06d0c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07c8160310 0x7f07c8159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07c8146170 0x7f07c8154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 --2- 172.21.15.28:0/836022877 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07c80ac850 0x7f07c8143570 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 >> 172.21.15.28:0/836022877 conn(0x7f07c8069e80 msgr2=0x7f07c80af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 shutdown_connections 2024-09-13T17:11:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.581+0000 7f07d0800640 1 -- 172.21.15.28:0/836022877 wait complete. 2024-09-13T17:11:03.599 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:11:03.599 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:11:03.599 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:11:03.599 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:03.599 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:11:03.600 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:11:03.600 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:11:03.600 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:03.600 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:11:03.600 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:11:03.600 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:11:03.600 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:03.601 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:11:03.601 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:11:03.601 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:11:03.601 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:03.616 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:11:03.616 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:11:03.617 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:11:03.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:11:03.618 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:11:03.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:11:03.619 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:11:03.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:11:03.661 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:11:03.661 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:03.663 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:11:03.663 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:03.663 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:11:03.664 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:03.665 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:11:03.665 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:03.665 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:11:03.791 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:11:03.791 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:11:03.794 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:11:03.800 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:11:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.836+0000 7fa79b200640 1 Processor -- start 2024-09-13T17:11:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.836+0000 7fa79b200640 1 -- start start 2024-09-13T17:11:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa79b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa79c15c8a0 0x7fa79c0b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa79b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa79c0b42c0 0x7fa79c0b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa79b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79c0b4c50 con 0x7fa79c0b1970 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa79b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c0cdfb0 con 0x7fa79c15c8a0 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa79b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa79c0d0a40 con 0x7fa79c0b42c0 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa799e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa79c15c8a0 0x7fa79c0b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa799e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa79c15c8a0 0x7fa79c0b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50648/0 (socket says 172.21.15.28:50648) 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.837+0000 7fa799e00640 1 -- 172.21.15.28:0/3066170035 learned_addr learned my addr 172.21.15.28:0/3066170035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.838+0000 7fa79a800640 1 --2- 172.21.15.28:0/3066170035 >> v2:172.21.15.189:3300/0 conn(0x7fa79c0b42c0 0x7fa79c0b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.838+0000 7fa799e00640 1 -- 172.21.15.28:0/3066170035 >> v2:172.21.15.189:3300/0 conn(0x7fa79c0b42c0 msgr2=0x7fa79c0b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.838+0000 7fa799e00640 1 --2- 172.21.15.28:0/3066170035 >> v2:172.21.15.189:3300/0 conn(0x7fa79c0b42c0 0x7fa79c0b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.838+0000 7fa799e00640 1 -- 172.21.15.28:0/3066170035 >> v1:172.21.15.189:6789/0 conn(0x7fa79c0b1970 legacy=0x7fa79c0cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:03.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.838+0000 7fa799e00640 1 -- 172.21.15.28:0/3066170035 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79c0d0c90 con 0x7fa79c15c8a0 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.839+0000 7fa799e00640 1 --2- 172.21.15.28:0/3066170035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c15c8a0 0x7fa79c0b5a50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa79c08b950 tx=0x7fa78c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87cfd3a29b61ab53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.839+0000 7fa798a00640 1 -- 172.21.15.28:0/3066170035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c032640 con 0x7fa79c15c8a0 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.839+0000 7fa798a00640 1 -- 172.21.15.28:0/3066170035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78c0327a0 con 0x7fa79c15c8a0 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.839+0000 7fa798a00640 1 -- 172.21.15.28:0/3066170035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c032b30 con 0x7fa79c15c8a0 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 -- 172.21.15.28:0/3066170035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c15c8a0 msgr2=0x7fa79c0b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 --2- 172.21.15.28:0/3066170035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c15c8a0 0x7fa79c0b5a50 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fa79c08b950 tx=0x7fa78c02a300 comp rx=0 tx=0).stop 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 -- 172.21.15.28:0/3066170035 shutdown_connections 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 --2- 172.21.15.28:0/3066170035 >> v2:172.21.15.189:3300/0 conn(0x7fa79c0b42c0 0x7fa79c0b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 --2- 172.21.15.28:0/3066170035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c15c8a0 0x7fa79c0b5a50 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 -- 172.21.15.28:0/3066170035 >> 172.21.15.28:0/3066170035 conn(0x7fa79c0cf2c0 msgr2=0x7fa79c0cf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 -- 172.21.15.28:0/3066170035 shutdown_connections 2024-09-13T17:11:03.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.840+0000 7fa79b200640 1 -- 172.21.15.28:0/3066170035 wait complete. 2024-09-13T17:11:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.841+0000 7fa79b200640 1 Processor -- start 2024-09-13T17:11:03.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.842+0000 7fa79b200640 1 -- start start 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.842+0000 7fa79b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79c0b1970 0x7fa79c156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.842+0000 7fa79b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.842+0000 7fa79b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa79c15c8a0 0x7fa79c14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa79b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c1598b0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa79b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0c99b0 con 0x7fa79c0b1970 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa79b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa79c0c9c90 con 0x7fa79c15c8a0 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50654/0 (socket says 172.21.15.28:50654) 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 -- 172.21.15.28:0/2386382945 learned_addr learned my addr 172.21.15.28:0/2386382945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799e00640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79c0b1970 0x7fa79c156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa79a800640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa79c15c8a0 0x7fa79c14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 -- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa79c15c8a0 msgr2=0x7fa79c14ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa79c15c8a0 0x7fa79c14ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 -- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79c0b1970 msgr2=0x7fa79c156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79c0b1970 0x7fa79c156300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.843+0000 7fa799400640 1 -- 172.21.15.28:0/2386382945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78c0094d0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa799400640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c450 tx=0x7fa78800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880052d0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa788005430 con 0x7fa79c0b42c0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79c0c9f70 con 0x7fa79c0b42c0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788012870 con 0x7fa79c0b42c0 2024-09-13T17:11:03.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.844+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa79c14cfc0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.845+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79c0732d0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.846+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa7880129d0 con 0x7fa79c0b42c0 2024-09-13T17:11:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.847+0000 7fa792000640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 0x7fa77c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.847+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa788088810 con 0x7fa79c0b42c0 2024-09-13T17:11:03.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.848+0000 7fa799e00640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 0x7fa77c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:03.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.849+0000 7fa799e00640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 0x7fa77c06d0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa78c034000 tx=0x7fa78c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:03.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:03.853+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa788053e30 con 0x7fa79c0b42c0 2024-09-13T17:11:04.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.045+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa79c0b4c50 con 0x7fa79c0b42c0 2024-09-13T17:11:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.851+0000 7fa792000640 1 -- 172.21.15.28:0/2386382945 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa79c0b4c50 con 0x7fa79c0b42c0 2024-09-13T17:11:04.853 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 msgr2=0x7fa77c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 0x7fa77c06d0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa78c034000 tx=0x7fa78c02e000 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 msgr2=0x7fa79c0ca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c450 tx=0x7fa78800c910 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 shutdown_connections 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa77c06ac00 0x7fa77c06d0c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa79c15c8a0 0x7fa79c14ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa79c0b42c0 0x7fa79c0ca470 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 --2- 172.21.15.28:0/2386382945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa79c0b1970 0x7fa79c156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:04.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 >> 172.21.15.28:0/2386382945 conn(0x7fa79c0cf2c0 msgr2=0x7fa79c0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 shutdown_connections 2024-09-13T17:11:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:04.854+0000 7fa79b200640 1 -- 172.21.15.28:0/2386382945 wait complete. 2024-09-13T17:11:04.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:11:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.035+0000 7f2303200640 1 Processor -- start 2024-09-13T17:11:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.035+0000 7f2303200640 1 -- start start 2024-09-13T17:11:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.036+0000 7f2303200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2304154a90 0x7f230415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.036+0000 7f2303200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 0x7f23040b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.036+0000 7f2303200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230415bb10 con 0x7f23040b0980 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.036+0000 7f2303200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23040b0df0 con 0x7f2304154a90 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.036+0000 7f2303200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230406b900 con 0x7f230406b360 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2304154a90 0x7f230415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2302800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f23040b0980 0x7f23040af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57968/0 (socket says 172.21.15.28:57968) 2024-09-13T17:11:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2302800640 1 -- 172.21.15.28:0/1368206885 learned_addr learned my addr 172.21.15.28:0/1368206885 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301e00640 1 -- 172.21.15.28:0/1368206885 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 msgr2=0x7f23040b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301400640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 0x7f23040b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2300a00640 1 -- 172.21.15.28:0/1368206885 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464007058 0 0) 0x7f230415bb10 con 0x7f23040b0980 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301e00640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 0x7f23040b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301e00640 1 -- 172.21.15.28:0/1368206885 >> v1:172.21.15.28:6789/0 conn(0x7f23040b0980 legacy=0x7f23040af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301e00640 1 -- 172.21.15.28:0/1368206885 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230406bbe0 con 0x7f2304154a90 2024-09-13T17:11:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.037+0000 7f2301400640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 0x7f23040b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.038+0000 7f2301e00640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2304154a90 0x7f230415b5d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f22f40097a0 tx=0x7f22f402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79348a816c121281 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.038+0000 7f2300a00640 1 -- 172.21.15.28:0/1368206885 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4032440 con 0x7f2304154a90 2024-09-13T17:11:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.038+0000 7f2300a00640 1 -- 172.21.15.28:0/1368206885 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f40325a0 con 0x7f2304154a90 2024-09-13T17:11:05.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.038+0000 7f2300a00640 1 -- 172.21.15.28:0/1368206885 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4032930 con 0x7f2304154a90 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 -- 172.21.15.28:0/1368206885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2304154a90 msgr2=0x7f230415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2304154a90 0x7f230415b5d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f22f40097a0 tx=0x7f22f402a180 comp rx=0 tx=0).stop 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 -- 172.21.15.28:0/1368206885 shutdown_connections 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f230406b360 0x7f23040b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 --2- 172.21.15.28:0/1368206885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2304154a90 0x7f230415b5d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 -- 172.21.15.28:0/1368206885 >> 172.21.15.28:0/1368206885 conn(0x7f2304072e40 msgr2=0x7f2304073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 -- 172.21.15.28:0/1368206885 shutdown_connections 2024-09-13T17:11:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.039+0000 7f2303200640 1 -- 172.21.15.28:0/1368206885 wait complete. 2024-09-13T17:11:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.041+0000 7f2303200640 1 Processor -- start 2024-09-13T17:11:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.041+0000 7f2303200640 1 -- start start 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 0x7f23040c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 0x7f23040c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23040ae770 con 0x7f230406b360 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23040ad8a0 con 0x7f2304154a90 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2303200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23040adaf0 con 0x7f23040b0980 2024-09-13T17:11:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50666/0 (socket says 172.21.15.28:50666) 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 -- 172.21.15.28:0/1414498374 learned_addr learned my addr 172.21.15.28:0/1414498374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301400640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 0x7f23040c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2302800640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 0x7f23040c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 -- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 msgr2=0x7f23040c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 0x7f23040c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 -- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 msgr2=0x7f23040c5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 0x7f23040c5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301e00640 1 -- 172.21.15.28:0/1414498374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f4009400 con 0x7f230406b360 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2301400640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 0x7f23040c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.042+0000 7f2302800640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 0x7f23040c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f2301e00640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f22f4035040 tx=0x7f22f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4006e10 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f4030980 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f4030d50 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23040addd0 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.043+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23040ae390 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.044+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230415d870 con 0x7f230406b360 2024-09-13T17:11:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.045+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f22f4039040 con 0x7f230406b360 2024-09-13T17:11:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.046+0000 7f22fa000640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 0x7f22e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.046+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f22f40a9b40 con 0x7f230406b360 2024-09-13T17:11:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.046+0000 7f2301400640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 0x7f22e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.047+0000 7f2301400640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 0x7f22e406d0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22f0006fd0 tx=0x7f22f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.050+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f22f4075060 con 0x7f230406b360 2024-09-13T17:11:05.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.294+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f23040b0030 con 0x7f230406b360 2024-09-13T17:11:05.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.294+0000 7f22fa000640 1 -- 172.21.15.28:0/1414498374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f22f4078680 con 0x7f230406b360 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:11:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:11:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:11:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:11:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:11:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:11:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:11:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:11:05.311 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:05.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.296+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 msgr2=0x7f22e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.296+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 0x7f22e406d0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f22f0006fd0 tx=0x7f22f0008040 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.296+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 msgr2=0x7f23041f5f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.296+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f22f4035040 tx=0x7f22f40046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 shutdown_connections 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f22e406ac00 0x7f22e406d0c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2304154a90 0x7f23040c5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23040b0980 0x7f23040c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 --2- 172.21.15.28:0/1414498374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f230406b360 0x7f23041f5f70 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 >> 172.21.15.28:0/1414498374 conn(0x7f2304072e40 msgr2=0x7f2304157fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 shutdown_connections 2024-09-13T17:11:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.297+0000 7f2303200640 1 -- 172.21.15.28:0/1414498374 wait complete. 2024-09-13T17:11:05.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:05.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.469+0000 7fb452400640 1 Processor -- start 2024-09-13T17:11:05.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.469+0000 7fb452400640 1 -- start start 2024-09-13T17:11:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb452400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb44c0cd710 0x7fb44c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb452400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0b5870 con 0x7fb44c0b4700 2024-09-13T17:11:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb452400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0d2560 con 0x7fb44c15c830 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb452400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0b4b30 con 0x7fb44c0cd710 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb44be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb44c0cd710 0x7fb44c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb451a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb44c0b4700 0x7fb44c0b5fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57982/0 (socket says 172.21.15.28:57982) 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb44be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb44c0cd710 0x7fb44c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50672/0 (socket says 172.21.15.28:50672) 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.471+0000 7fb451a00640 1 -- 172.21.15.28:0/610467943 learned_addr learned my addr 172.21.15.28:0/610467943 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.472+0000 7fb44be00640 1 -- 172.21.15.28:0/610467943 >> v1:172.21.15.189:6789/0 conn(0x7fb44c15c830 legacy=0x7fb44c0d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.472+0000 7fb44be00640 1 -- 172.21.15.28:0/610467943 >> v1:172.21.15.28:6789/0 conn(0x7fb44c0b4700 legacy=0x7fb44c0b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.472+0000 7fb44be00640 1 -- 172.21.15.28:0/610467943 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c15cc10 con 0x7fb44c0cd710 2024-09-13T17:11:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.473+0000 7fb44be00640 1 --2- 172.21.15.28:0/610467943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c0cd710 0x7fb44c0b41c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb438009790 tx=0x7fb43802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402b30b7a66b43f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.473+0000 7fb44b400640 1 -- 172.21.15.28:0/610467943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438032440 con 0x7fb44c0cd710 2024-09-13T17:11:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.473+0000 7fb44b400640 1 -- 172.21.15.28:0/610467943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4380325a0 con 0x7fb44c0cd710 2024-09-13T17:11:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.473+0000 7fb44b400640 1 -- 172.21.15.28:0/610467943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb438032930 con 0x7fb44c0cd710 2024-09-13T17:11:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 -- 172.21.15.28:0/610467943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c0cd710 msgr2=0x7fb44c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 --2- 172.21.15.28:0/610467943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c0cd710 0x7fb44c0b41c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb438009790 tx=0x7fb43802a210 comp rx=0 tx=0).stop 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 -- 172.21.15.28:0/610467943 shutdown_connections 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 --2- 172.21.15.28:0/610467943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c0cd710 0x7fb44c0b41c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 -- 172.21.15.28:0/610467943 >> 172.21.15.28:0/610467943 conn(0x7fb44c0cf3a0 msgr2=0x7fb44c0cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 -- 172.21.15.28:0/610467943 shutdown_connections 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.474+0000 7fb452400640 1 -- 172.21.15.28:0/610467943 wait complete. 2024-09-13T17:11:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.476+0000 7fb452400640 1 Processor -- start 2024-09-13T17:11:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.476+0000 7fb452400640 1 -- start start 2024-09-13T17:11:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb44c0b4700 0x7fb44c14cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb44c0cd710 0x7fb44c157290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1588d0 con 0x7fb44c15c830 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c14d130 con 0x7fb44c0cd710 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb452400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0c9a20 con 0x7fb44c0b4700 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb451a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb451a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50674/0 (socket says 172.21.15.28:50674) 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb451a00640 1 -- 172.21.15.28:0/438518231 learned_addr learned my addr 172.21.15.28:0/438518231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb44be00640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb44c0cd710 0x7fb44c157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.477+0000 7fb451000640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb44c0b4700 0x7fb44c14cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 -- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb44c0b4700 msgr2=0x7fb44c14cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb44c0b4700 0x7fb44c14cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 -- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb44c0cd710 msgr2=0x7fb44c157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb44c0cd710 0x7fb44c157290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 -- 172.21.15.28:0/438518231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4380093a0 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb451a00640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb44400cbc0 tx=0x7fb44400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44400ad40 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb444012400 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c157c50 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44400aea0 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.478+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c158210 con 0x7fb44c15c830 2024-09-13T17:11:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.479+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c158a10 con 0x7fb44c15c830 2024-09-13T17:11:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.480+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb444007070 con 0x7fb44c15c830 2024-09-13T17:11:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.481+0000 7fb448c00640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 0x7fb42c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.481+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb444088e60 con 0x7fb44c15c830 2024-09-13T17:11:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.482+0000 7fb451000640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 0x7fb42c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.483+0000 7fb451000640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 0x7fb42c06d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb440000f60 tx=0x7fb440009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.486+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb44c158a10 con 0x7fb44c15c830 2024-09-13T17:11:05.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.666+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb44c158a10 con 0x7fb44c15c830 2024-09-13T17:11:05.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.667+0000 7fb448c00640 1 -- 172.21.15.28:0/438518231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fb44c158a10 con 0x7fb44c15c830 2024-09-13T17:11:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:05.669 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":44,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:04.847668+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 msgr2=0x7fb42c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 0x7fb42c06d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb440000f60 tx=0x7fb440009040 comp rx=0 tx=0).stop 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 msgr2=0x7fb44c15b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb44400cbc0 tx=0x7fb44400a430 comp rx=0 tx=0).stop 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 shutdown_connections 2024-09-13T17:11:05.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb42c06ac00 0x7fb42c06d0c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb44c15c830 0x7fb44c15b310 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb44c0cd710 0x7fb44c157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 --2- 172.21.15.28:0/438518231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb44c0b4700 0x7fb44c14cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 >> 172.21.15.28:0/438518231 conn(0x7fb44c0cf3a0 msgr2=0x7fb44c0b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 shutdown_connections 2024-09-13T17:11:05.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.669+0000 7fb452400640 1 -- 172.21.15.28:0/438518231 wait complete. 2024-09-13T17:11:05.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:11:05.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:11:05.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.845+0000 7f3c26000640 1 Processor -- start 2024-09-13T17:11:05.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.845+0000 7f3c26000640 1 -- start start 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.846+0000 7f3c26000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c2015c9a0 0x7f3c200ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.846+0000 7f3c26000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c200b2400 0x7f3c2006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.846+0000 7f3c26000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c20070470 con 0x7f3c200ad940 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.846+0000 7f3c26000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c200ace30 con 0x7f3c200b2400 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.846+0000 7f3c26000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200ad110 con 0x7f3c2015c9a0 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c200b2400 0x7f3c2006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3c200b2400 0x7f3c2006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50682/0 (socket says 172.21.15.28:50682) 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 -- 172.21.15.28:0/4085829687 learned_addr learned my addr 172.21.15.28:0/4085829687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:05.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1f400640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c2015c9a0 0x7f3c200ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 -- 172.21.15.28:0/4085829687 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c2015c9a0 msgr2=0x7f3c200ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:05.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c2015c9a0 0x7f3c200ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 -- 172.21.15.28:0/4085829687 >> v1:172.21.15.189:6789/0 conn(0x7f3c200ad940 legacy=0x7f3c200b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.847+0000 7f3c1fe00640 1 -- 172.21.15.28:0/4085829687 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2006b170 con 0x7f3c200b2400 2024-09-13T17:11:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.848+0000 7f3c1fe00640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c2006bc50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009e50 tx=0x7f3c1802a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3337fa3276dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.848+0000 7f3c1ea00640 1 -- 172.21.15.28:0/4085829687 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18031600 con 0x7f3c200b2400 2024-09-13T17:11:05.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.848+0000 7f3c1ea00640 1 -- 172.21.15.28:0/4085829687 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1802d050 con 0x7f3c200b2400 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.848+0000 7f3c1ea00640 1 -- 172.21.15.28:0/4085829687 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c18031980 con 0x7f3c200b2400 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 -- 172.21.15.28:0/4085829687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 msgr2=0x7f3c2006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c2006bc50 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009e50 tx=0x7f3c1802a7a0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 -- 172.21.15.28:0/4085829687 shutdown_connections 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c2006bc50 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 --2- 172.21.15.28:0/4085829687 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c2015c9a0 0x7f3c200ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 -- 172.21.15.28:0/4085829687 >> 172.21.15.28:0/4085829687 conn(0x7f3c200b0270 msgr2=0x7f3c200b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 -- 172.21.15.28:0/4085829687 shutdown_connections 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.849+0000 7f3c26000640 1 -- 172.21.15.28:0/4085829687 wait complete. 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.851+0000 7f3c26000640 1 Processor -- start 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.851+0000 7f3c26000640 1 -- start start 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 0x7f3c201f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c201f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 0x7f3c2015fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c200cd090 con 0x7f3c200b2400 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201f60a0 con 0x7f3c2015c9a0 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c26000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c201f6380 con 0x7f3c200ad940 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c201f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c201f5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50686/0 (socket says 172.21.15.28:50686) 2024-09-13T17:11:05.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c16a00640 1 -- 172.21.15.28:0/1601135744 learned_addr learned my addr 172.21.15.28:0/1601135744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c1f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 0x7f3c201f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.852+0000 7f3c1fe00640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 0x7f3c2015fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 -- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 msgr2=0x7f3c2015fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 0x7f3c2015fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 -- 172.21.15.28:0/1601135744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 msgr2=0x7f3c201f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c201f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:05.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 -- 172.21.15.28:0/1601135744 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c18009b30 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1fe00640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 0x7f3c2015fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.853+0000 7f3c1f400640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 0x7f3c201f1bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e870 tx=0x7f3c1000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.854+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000ee60 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.854+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c20070930 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.854+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1000c940 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.854+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000ccd0 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.854+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2015f220 con 0x7f3c200ad940 2024-09-13T17:11:05.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.855+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c201f2110 con 0x7f3c200ad940 2024-09-13T17:11:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.857+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3c10013470 con 0x7f3c200ad940 2024-09-13T17:11:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.858+0000 7f3c17e00640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 0x7f3bfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:05.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.858+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3c10089060 con 0x7f3c200ad940 2024-09-13T17:11:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.858+0000 7f3c16a00640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 0x7f3bfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:05.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.859+0000 7f3c16a00640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 0x7f3bfc06d0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c201f57b0 tx=0x7f3c08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:05.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:05.863+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3c100bc9c0 con 0x7f3c200ad940 2024-09-13T17:11:06.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.040+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3c200af6a0 con 0x7f3c200ad940 2024-09-13T17:11:06.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.041+0000 7f3c17e00640 1 -- 172.21.15.28:0/1601135744 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3c100545b0 con 0x7f3c200ad940 2024-09-13T17:11:06.043 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:11:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 msgr2=0x7f3bfc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 0x7f3bfc06d0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3c201f57b0 tx=0x7f3c08009040 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 msgr2=0x7f3c201f1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 0x7f3c201f1bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000e870 tx=0x7f3c1000ed30 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 shutdown_connections 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bfc06ac00 0x7f3bfc06d0c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c2015c9a0 0x7f3c2015fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c200b2400 0x7f3c201f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 --2- 172.21.15.28:0/1601135744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3c200ad940 0x7f3c201f1bd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.043+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 >> 172.21.15.28:0/1601135744 conn(0x7f3c200b0270 msgr2=0x7f3c200b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.044+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 shutdown_connections 2024-09-13T17:11:06.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.044+0000 7f3c26000640 1 -- 172.21.15.28:0/1601135744 wait complete. 2024-09-13T17:11:06.062 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.216+0000 7f18fda00640 1 Processor -- start 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.216+0000 7f18fda00640 1 -- start start 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.217+0000 7f18fda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f8071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.217+0000 7f18fda00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f18f80b2ce0 0x7f18f80b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18fda00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f80b1980 con 0x7f18f806c610 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18fda00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f80cf000 con 0x7f18f80b2ce0 2024-09-13T17:11:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18fda00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f80b31e0 con 0x7f18f80cec20 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18f7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18f806c610 0x7f18f8169af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57992/0 (socket says 172.21.15.28:57992) 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18f7400640 1 -- 172.21.15.28:0/1081488391 learned_addr learned my addr 172.21.15.28:0/1081488391 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18f6000640 1 --2- 172.21.15.28:0/1081488391 >> v2:172.21.15.189:3300/0 conn(0x7f18f80b2ce0 0x7f18f80b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18f6a00640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f8071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.218+0000 7f18f5600640 1 -- 172.21.15.28:0/1081488391 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120657304 0 0) 0x7f18f80b1980 con 0x7f18f806c610 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.475+0000 7f18f5600640 1 -- 172.21.15.28:0/1081488391 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc003500 con 0x7f18f806c610 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.476+0000 7f18f6000640 1 -- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 msgr2=0x7f18f8071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.476+0000 7f18f6000640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f8071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.476+0000 7f18f6000640 1 -- 172.21.15.28:0/1081488391 >> v1:172.21.15.28:6789/0 conn(0x7f18f806c610 legacy=0x7f18f8169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.476+0000 7f18f6000640 1 -- 172.21.15.28:0/1081488391 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f80b34c0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.478+0000 7f18f6000640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f80b2ce0 0x7f18f80b1e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c860 tx=0x7f18e402a830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51094e5f2fcc9e2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.478+0000 7f18f5600640 1 -- 172.21.15.28:0/1081488391 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e40316a0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.478+0000 7f18f5600640 1 -- 172.21.15.28:0/1081488391 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e4031800 con 0x7f18f80b2ce0 2024-09-13T17:11:06.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.478+0000 7f18f5600640 1 -- 172.21.15.28:0/1081488391 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4031bb0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 -- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f80b2ce0 msgr2=0x7f18f80b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f80b2ce0 0x7f18f80b1e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f18e400c860 tx=0x7f18e402a830 comp rx=0 tx=0).stop 2024-09-13T17:11:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 -- 172.21.15.28:0/1081488391 shutdown_connections 2024-09-13T17:11:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f80b2ce0 0x7f18f80b1e10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 --2- 172.21.15.28:0/1081488391 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f8071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 -- 172.21.15.28:0/1081488391 >> 172.21.15.28:0/1081488391 conn(0x7f18f8069e80 msgr2=0x7f18f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 -- 172.21.15.28:0/1081488391 shutdown_connections 2024-09-13T17:11:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.479+0000 7f18fda00640 1 -- 172.21.15.28:0/1081488391 wait complete. 2024-09-13T17:11:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.481+0000 7f18fda00640 1 Processor -- start 2024-09-13T17:11:06.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.482+0000 7f18fda00640 1 -- start start 2024-09-13T17:11:06.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f806c610 0x7f18f80c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f816a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8158fd0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8159270 con 0x7f18f806c610 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18fda00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f8159550 con 0x7f18f80cec20 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18f6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50700/0 (socket says 172.21.15.28:50700) 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18f6000640 1 -- 172.21.15.28:0/3335618852 learned_addr learned my addr 172.21.15.28:0/3335618852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18f6a00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f806c610 0x7f18f80c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.483+0000 7f18f7400640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f816a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 -- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 msgr2=0x7f18f816a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f816a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 -- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f806c610 msgr2=0x7f18f80c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f806c610 0x7f18f80c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 -- 172.21.15.28:0/3335618852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e400c4c0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.484+0000 7f18f6000640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f18e4037040 tx=0x7f18e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.485+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4003810 con 0x7f18f80b2ce0 2024-09-13T17:11:06.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.485+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e40039b0 con 0x7f18f80b2ce0 2024-09-13T17:11:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.485+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e4003d60 con 0x7f18f80b2ce0 2024-09-13T17:11:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.485+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f8159830 con 0x7f18f80b2ce0 2024-09-13T17:11:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.485+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f808a360 con 0x7f18f80b2ce0 2024-09-13T17:11:06.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.486+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f8159d50 con 0x7f18f80b2ce0 2024-09-13T17:11:06.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.488+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f18e403b040 con 0x7f18f80b2ce0 2024-09-13T17:11:06.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.489+0000 7f18eea00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 0x7f18d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:06.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.489+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f18e40a9e90 con 0x7f18f80b2ce0 2024-09-13T17:11:06.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.490+0000 7f18f6a00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 0x7f18d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:06.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.490+0000 7f18f6a00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 0x7f18d806d0f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f18e800df40 tx=0x7f18e800f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:06.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.494+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f18f8159d50 con 0x7f18f80b2ce0 2024-09-13T17:11:06.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.681+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f18f8159d50 con 0x7f18f80b2ce0 2024-09-13T17:11:06.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.681+0000 7f18eea00640 1 -- 172.21.15.28:0/3335618852 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) v1 ==== 84+0+65 (secure 0 0 0) 0x7f18f8159d50 con 0x7f18f80b2ce0 2024-09-13T17:11:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:06.683 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:11:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:11:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:11:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:06.684 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.683+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 msgr2=0x7f18d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 0x7f18d806d0f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f18e800df40 tx=0x7f18e800f040 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 msgr2=0x7f18f8089e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f18e4037040 tx=0x7f18e40046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 shutdown_connections 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18d806ac30 0x7f18d806d0f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18f80cec20 0x7f18f816a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18f80b2ce0 0x7f18f8089e20 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 --2- 172.21.15.28:0/3335618852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18f806c610 0x7f18f80c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:06.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 >> 172.21.15.28:0/3335618852 conn(0x7f18f8069e80 msgr2=0x7f18f816b680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:06.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 shutdown_connections 2024-09-13T17:11:06.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.684+0000 7f18fda00640 1 -- 172.21.15.28:0/3335618852 wait complete. 2024-09-13T17:11:06.701 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:11:06.701 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.858+0000 7fd061a00640 1 Processor -- start 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.859+0000 7fd061a00640 1 -- start start 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06c360 0x7fd05c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd05c0b3c50 0x7fd05c0b4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05c0b2480 con 0x7fd05c06b2b0 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd05c08ae20 con 0x7fd05c0b3c50 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c0b37c0 con 0x7fd05c06c360 2024-09-13T17:11:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd05c0b3c50 0x7fd05c0b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd05c0b3c50 0x7fd05c0b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50708/0 (socket says 172.21.15.28:50708) 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd061000640 1 -- 172.21.15.28:0/2587704136 learned_addr learned my addr 172.21.15.28:0/2587704136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.861+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835853593 0 0) 0x7fd05c0b2480 con 0x7fd05c06b2b0 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:06.860+0000 7fd053e00640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06c360 0x7fd05c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd040003500 con 0x7fd05c06b2b0 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd061000640 1 -- 172.21.15.28:0/2587704136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06c360 msgr2=0x7fd05c0b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd061000640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06c360 0x7fd05c0b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 898719974 0 0) 0x7fd040003500 con 0x7fd05c06b2b0 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd061000640 1 -- 172.21.15.28:0/2587704136 >> v1:172.21.15.189:6789/0 conn(0x7fd05c06b2b0 legacy=0x7fd05c06b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.207+0000 7fd061000640 1 -- 172.21.15.28:0/2587704136 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c0b2bb0 con 0x7fd05c0b3c50 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.208+0000 7fd061000640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c0b3c50 0x7fd05c0b4100 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd05400bef0 tx=0x7fd0540093b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3750b02b4779eb1d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.209+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c001670 con 0x7fd05c0b3c50 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.209+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05402d050 con 0x7fd05c0b3c50 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.209+0000 7fd05b400640 1 -- 172.21.15.28:0/2587704136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054009600 con 0x7fd05c0b3c50 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.209+0000 7fd061a00640 1 -- 172.21.15.28:0/2587704136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c0b3c50 msgr2=0x7fd05c0b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.209+0000 7fd061a00640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c0b3c50 0x7fd05c0b4100 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd05400bef0 tx=0x7fd0540093b0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 -- 172.21.15.28:0/2587704136 shutdown_connections 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c0b3c50 0x7fd05c0b4100 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 --2- 172.21.15.28:0/2587704136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06c360 0x7fd05c0b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 -- 172.21.15.28:0/2587704136 >> 172.21.15.28:0/2587704136 conn(0x7fd05c069e80 msgr2=0x7fd05c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 -- 172.21.15.28:0/2587704136 shutdown_connections 2024-09-13T17:11:07.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.210+0000 7fd061a00640 1 -- 172.21.15.28:0/2587704136 wait complete. 2024-09-13T17:11:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.212+0000 7fd061a00640 1 Processor -- start 2024-09-13T17:11:07.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.212+0000 7fd061a00640 1 -- start start 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.213+0000 7fd061a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06b2b0 0x7fd05c169c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.213+0000 7fd061a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd061a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c0b3c50 0x7fd05c159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd061a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c202f40 con 0x7fd05c06c360 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd061a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c15ea00 con 0x7fd05c0b3c50 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd061a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05c16cac0 con 0x7fd05c06b2b0 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50722/0 (socket says 172.21.15.28:50722) 2024-09-13T17:11:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 -- 172.21.15.28:0/249197311 learned_addr learned my addr 172.21.15.28:0/249197311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd061000640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c0b3c50 0x7fd05c159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd05be00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06b2b0 0x7fd05c169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 -- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06b2b0 msgr2=0x7fd05c169c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06b2b0 0x7fd05c169c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 -- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c0b3c50 msgr2=0x7fd05c159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.214+0000 7fd053e00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c0b3c50 0x7fd05c159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd053e00640 1 -- 172.21.15.28:0/249197311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05400bb50 con 0x7fd05c06c360 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd053e00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd0440097c0 tx=0x7fd044009c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04400ebb0 con 0x7fd05c06c360 2024-09-13T17:11:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04400d040 con 0x7fd05c06c360 2024-09-13T17:11:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd044009e70 con 0x7fd05c06c360 2024-09-13T17:11:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.215+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05c15f280 con 0x7fd05c06c360 2024-09-13T17:11:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.216+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05c15ec30 con 0x7fd05c06c360 2024-09-13T17:11:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.217+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05c067a10 con 0x7fd05c06c360 2024-09-13T17:11:07.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.219+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd044013440 con 0x7fd05c06c360 2024-09-13T17:11:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.220+0000 7fd058c00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 0x7fd03806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.220+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd0440899a0 con 0x7fd05c06c360 2024-09-13T17:11:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.221+0000 7fd05be00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 0x7fd03806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.222+0000 7fd05be00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 0x7fd03806d0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd04c001b00 tx=0x7fd04c001a90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.225+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd044054f70 con 0x7fd05c06c360 2024-09-13T17:11:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.410+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd05c15f410 con 0x7fd05c06c360 2024-09-13T17:11:07.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.480+0000 7fd058c00640 1 -- 172.21.15.28:0/249197311 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd044058590 con 0x7fd05c06c360 2024-09-13T17:11:07.482 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:11:07.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.484+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 msgr2=0x7fd03806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.484+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 0x7fd03806d0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd04c001b00 tx=0x7fd04c001a90 comp rx=0 tx=0).stop 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 msgr2=0x7fd05c202a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fd0440097c0 tx=0x7fd044009c80 comp rx=0 tx=0).stop 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 shutdown_connections 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd03806ac00 0x7fd03806d0c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c0b3c50 0x7fd05c159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd05c06c360 0x7fd05c202a00 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 --2- 172.21.15.28:0/249197311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd05c06b2b0 0x7fd05c169c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 >> 172.21.15.28:0/249197311 conn(0x7fd05c069e80 msgr2=0x7fd05c167930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.485+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 shutdown_connections 2024-09-13T17:11:07.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.486+0000 7fd061a00640 1 -- 172.21.15.28:0/249197311 wait complete. 2024-09-13T17:11:07.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:11:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.671+0000 7f21fce00640 1 Processor -- start 2024-09-13T17:11:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.671+0000 7f21fce00640 1 -- start start 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21fce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21f80b0b50 0x7f21f80b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21fce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f8159820 0x7f21f80b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21fce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21f80c3850 con 0x7f21f8159440 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21fce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21f80b2d60 con 0x7f21f80b0b50 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21fce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f815a930 con 0x7f21f8159820 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21f7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f21f8159440 0x7f21f80b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58000/0 (socket says 172.21.15.28:58000) 2024-09-13T17:11:07.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21f7400640 1 -- 172.21.15.28:0/398893081 learned_addr learned my addr 172.21.15.28:0/398893081 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.672+0000 7f21efe00640 1 --2- 172.21.15.28:0/398893081 >> v2:172.21.15.28:3300/0 conn(0x7f21f80b0b50 0x7f21f80b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.673+0000 7f21f7e00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f8159820 0x7f21f80b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.673+0000 7f21efe00640 1 -- 172.21.15.28:0/398893081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f8159820 msgr2=0x7f21f80b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.673+0000 7f21efe00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f8159820 0x7f21f80b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.673+0000 7f21efe00640 1 -- 172.21.15.28:0/398893081 >> v1:172.21.15.28:6789/0 conn(0x7f21f8159440 legacy=0x7f21f80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.673+0000 7f21efe00640 1 -- 172.21.15.28:0/398893081 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21f8159d20 con 0x7f21f80b0b50 2024-09-13T17:11:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.674+0000 7f21efe00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f80b0b50 0x7f21f80b22e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097a0 tx=0x7f21e002c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129e8e9f04d0ebb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.674+0000 7f21f6a00640 1 -- 172.21.15.28:0/398893081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0031710 con 0x7f21f80b0b50 2024-09-13T17:11:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.674+0000 7f21f6a00640 1 -- 172.21.15.28:0/398893081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e002f050 con 0x7f21f80b0b50 2024-09-13T17:11:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.674+0000 7f21f6a00640 1 -- 172.21.15.28:0/398893081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21e0031a90 con 0x7f21f80b0b50 2024-09-13T17:11:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 -- 172.21.15.28:0/398893081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f80b0b50 msgr2=0x7f21f80b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f80b0b50 0x7f21f80b22e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f21e00097a0 tx=0x7f21e002c420 comp rx=0 tx=0).stop 2024-09-13T17:11:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 -- 172.21.15.28:0/398893081 shutdown_connections 2024-09-13T17:11:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f8159820 0x7f21f80b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 --2- 172.21.15.28:0/398893081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f80b0b50 0x7f21f80b22e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 -- 172.21.15.28:0/398893081 >> 172.21.15.28:0/398893081 conn(0x7f21f8072e40 msgr2=0x7f21f8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 -- 172.21.15.28:0/398893081 shutdown_connections 2024-09-13T17:11:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.675+0000 7f21fce00640 1 -- 172.21.15.28:0/398893081 wait complete. 2024-09-13T17:11:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.678+0000 7f21fce00640 1 Processor -- start 2024-09-13T17:11:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.678+0000 7f21fce00640 1 -- start start 2024-09-13T17:11:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f80b0b50 0x7f21f81f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21f8159440 0x7f21f813feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f81403f0 con 0x7f21f8159820 2024-09-13T17:11:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21f814f160 con 0x7f21f8159440 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21fce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21f81f68e0 con 0x7f21f80b0b50 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50748/0 (socket says 172.21.15.28:50748) 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 -- 172.21.15.28:0/2603497245 learned_addr learned my addr 172.21.15.28:0/2603497245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7400640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f80b0b50 0x7f21f81f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21efe00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21f8159440 0x7f21f813feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 -- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f80b0b50 msgr2=0x7f21f81f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f80b0b50 0x7f21f81f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 -- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21f8159440 msgr2=0x7f21f813feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.679+0000 7f21f7e00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21f8159440 0x7f21f813feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21f7e00640 1 -- 172.21.15.28:0/2603497245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e0009400 con 0x7f21f8159820 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21f7e00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f21f0007c90 tx=0x7f21f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f0007ea0 con 0x7f21f8159820 2024-09-13T17:11:07.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f0014420 con 0x7f21f8159820 2024-09-13T17:11:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f00147f0 con 0x7f21f8159820 2024-09-13T17:11:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21f814e830 con 0x7f21f8159820 2024-09-13T17:11:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.680+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21f814edf0 con 0x7f21f8159820 2024-09-13T17:11:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.681+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21f814f2d0 con 0x7f21f8159820 2024-09-13T17:11:07.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.682+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f21f0014950 con 0x7f21f8159820 2024-09-13T17:11:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.683+0000 7f21ef400640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 0x7f21d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.683+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f21f008a730 con 0x7f21f8159820 2024-09-13T17:11:07.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.684+0000 7f21f7400640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 0x7f21d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:07.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.685+0000 7f21f7400640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 0x7f21d406d0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21e8005880 tx=0x7f21e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:07.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.687+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f21f0055b80 con 0x7f21f8159820 2024-09-13T17:11:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:07.885+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f21f806b2e0 con 0x7f21f8159820 2024-09-13T17:11:08.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.486+0000 7f21ef400640 1 -- 172.21.15.28:0/2603497245 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) v1 ==== 170+0+0 (secure 0 0 0) 0x7f21f00591a0 con 0x7f21f8159820 2024-09-13T17:11:08.489 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:11:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.492+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 msgr2=0x7f21d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.492+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 0x7f21d406d0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f21e8005880 tx=0x7f21e8009040 comp rx=0 tx=0).stop 2024-09-13T17:11:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.492+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 msgr2=0x7f21f814f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.492+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f21f0007c90 tx=0x7f21f000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 shutdown_connections 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21d406ac00 0x7f21d406d0c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21f8159820 0x7f21f814f5f0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21f8159440 0x7f21f813feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 --2- 172.21.15.28:0/2603497245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21f80b0b50 0x7f21f81f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 >> 172.21.15.28:0/2603497245 conn(0x7f21f8072e40 msgr2=0x7f21f8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:08.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.495+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 shutdown_connections 2024-09-13T17:11:08.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.496+0000 7f21fce00640 1 -- 172.21.15.28:0/2603497245 wait complete. 2024-09-13T17:11:08.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:11:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.686+0000 7fce71600640 1 Processor -- start 2024-09-13T17:11:08.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.686+0000 7fce71600640 1 -- start start 2024-09-13T17:11:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.687+0000 7fce71600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 0x7fce6c1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.687+0000 7fce71600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c1595f0 con 0x7fce6c06c110 2024-09-13T17:11:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.687+0000 7fce71600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6c1598c0 con 0x7fce6c06ccd0 2024-09-13T17:11:08.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.687+0000 7fce71600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c159ba0 con 0x7fce6c158cd0 2024-09-13T17:11:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fce6c06c110 0x7fce6c0b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58002/0 (socket says 172.21.15.28:58002) 2024-09-13T17:11:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6b400640 1 -- 172.21.15.28:0/687649623 learned_addr learned my addr 172.21.15.28:0/687649623 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6aa00640 1 --2- 172.21.15.28:0/687649623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 0x7fce6c1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:08.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159858066 0 0) 0x7fce6c1595f0 con 0x7fce6c06c110 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce50003520 con 0x7fce6c06c110 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544513722 0 0) 0x7fce6c1598c0 con 0x7fce6c06ccd0 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce6c1595f0 con 0x7fce6c06ccd0 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6aa00640 1 -- 172.21.15.28:0/687649623 >> v1:172.21.15.189:6789/0 conn(0x7fce6c06ccd0 legacy=0x7fce6c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6aa00640 1 -- 172.21.15.28:0/687649623 >> v1:172.21.15.28:6789/0 conn(0x7fce6c06c110 legacy=0x7fce6c0b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.688+0000 7fce6aa00640 1 -- 172.21.15.28:0/687649623 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce6c159e80 con 0x7fce6c158cd0 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.689+0000 7fce6aa00640 1 --2- 172.21.15.28:0/687649623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 0x7fce6c1590b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00a970 tx=0x7fce5c02b550 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53575a5fcab5f2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c030710 con 0x7fce6c158cd0 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c02e050 con 0x7fce6c158cd0 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce69600640 1 -- 172.21.15.28:0/687649623 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c030a70 con 0x7fce6c158cd0 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 -- 172.21.15.28:0/687649623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 msgr2=0x7fce6c1590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 --2- 172.21.15.28:0/687649623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 0x7fce6c1590b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fce5c00a970 tx=0x7fce5c02b550 comp rx=0 tx=0).stop 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 -- 172.21.15.28:0/687649623 shutdown_connections 2024-09-13T17:11:08.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 --2- 172.21.15.28:0/687649623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c158cd0 0x7fce6c1590b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 -- 172.21.15.28:0/687649623 >> 172.21.15.28:0/687649623 conn(0x7fce6c0b0270 msgr2=0x7fce6c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 -- 172.21.15.28:0/687649623 shutdown_connections 2024-09-13T17:11:08.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.690+0000 7fce71600640 1 -- 172.21.15.28:0/687649623 wait complete. 2024-09-13T17:11:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.692+0000 7fce71600640 1 Processor -- start 2024-09-13T17:11:08.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.692+0000 7fce71600640 1 -- start start 2024-09-13T17:11:08.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c06c110 0x7fce6c146270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:08.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce6c06ccd0 0x7fce6c0acff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:08.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0ad760 con 0x7fce6c158cd0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c0ad530 con 0x7fce6c06ccd0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce71600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce6c146050 con 0x7fce6c06c110 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50764/0 (socket says 172.21.15.28:50764) 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.693+0000 7fce6b400640 1 -- 172.21.15.28:0/2932890622 learned_addr learned my addr 172.21.15.28:0/2932890622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6aa00640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c06c110 0x7fce6c146270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6a000640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce6c06ccd0 0x7fce6c0acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 -- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c06c110 msgr2=0x7fce6c146270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c06c110 0x7fce6c146270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 -- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce6c06ccd0 msgr2=0x7fce6c0acff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce6c06ccd0 0x7fce6c0acff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 -- 172.21.15.28:0/2932890622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c00a5a0 con 0x7fce6c158cd0 2024-09-13T17:11:08.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce6b400640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fce6400cb10 tx=0x7fce6400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6400ac70 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce6400add0 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce6c1506c0 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce64012620 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.694+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce6c150070 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.695+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6c150850 con 0x7fce6c158cd0 2024-09-13T17:11:08.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.697+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fce64007070 con 0x7fce6c158cd0 2024-09-13T17:11:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.698+0000 7fce62a00640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 0x7fce4c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:08.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.698+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fce640896d0 con 0x7fce6c158cd0 2024-09-13T17:11:08.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.698+0000 7fce6aa00640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 0x7fce4c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:08.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.699+0000 7fce6aa00640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 0x7fce4c06d0f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fce5c035000 tx=0x7fce5c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:08.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.701+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fce640548f0 con 0x7fce6c158cd0 2024-09-13T17:11:08.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.918+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fce6c1590b0 con 0x7fce6c158cd0 2024-09-13T17:11:08.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.925+0000 7fce62a00640 1 -- 172.21.15.28:0/2932890622 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v31) v1 ==== 172+0+0 (secure 0 0 0) 0x7fce64057f10 con 0x7fce6c158cd0 2024-09-13T17:11:08.932 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 6 and data pool 7 2024-09-13T17:11:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.931+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 msgr2=0x7fce4c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.931+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 0x7fce4c06d0f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fce5c035000 tx=0x7fce5c030000 comp rx=0 tx=0).stop 2024-09-13T17:11:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.931+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 msgr2=0x7fce6c145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:08.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.931+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fce6400cb10 tx=0x7fce6400a060 comp rx=0 tx=0).stop 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 shutdown_connections 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4c06ac30 0x7fce4c06d0f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6c158cd0 0x7fce6c145b10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce6c06ccd0 0x7fce6c0acff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 --2- 172.21.15.28:0/2932890622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce6c06c110 0x7fce6c146270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.934+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 >> 172.21.15.28:0/2932890622 conn(0x7fce6c0b0270 msgr2=0x7fce6c0cdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.935+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 shutdown_connections 2024-09-13T17:11:08.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:08.935+0000 7fce71600640 1 -- 172.21.15.28:0/2932890622 wait complete. 2024-09-13T17:11:08.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.121+0000 7fb75ae00640 1 Processor -- start 2024-09-13T17:11:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.121+0000 7fb75ae00640 1 -- start start 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.122+0000 7fb75ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7540cabc0 0x7fb75406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.122+0000 7fb75ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb754070cc0 0x7fb754071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.122+0000 7fb75ae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb75415e100 con 0x7fb75415a540 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.122+0000 7fb75ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7540716b0 con 0x7fb754070cc0 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.122+0000 7fb75ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540adb00 con 0x7fb7540cabc0 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb754070cc0 0x7fb754071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb754070cc0 0x7fb754071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50772/0 (socket says 172.21.15.28:50772) 2024-09-13T17:11:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 -- 172.21.15.28:0/3382722216 learned_addr learned my addr 172.21.15.28:0/3382722216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb74b400640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7540cabc0 0x7fb75406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 -- 172.21.15.28:0/3382722216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7540cabc0 msgr2=0x7fb75406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7540cabc0 0x7fb75406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 -- 172.21.15.28:0/3382722216 >> v1:172.21.15.189:6789/0 conn(0x7fb75415a540 legacy=0x7fb75415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.123+0000 7fb758e00640 1 -- 172.21.15.28:0/3382722216 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75415e3e0 con 0x7fb754070cc0 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.124+0000 7fb758e00640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb754070cc0 0x7fb754071170 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009e80 tx=0x7fb74c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feee5295a52a78fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb753400640 1 -- 172.21.15.28:0/3382722216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c02aea0 con 0x7fb754070cc0 2024-09-13T17:11:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb753400640 1 -- 172.21.15.28:0/3382722216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74c02d050 con 0x7fb754070cc0 2024-09-13T17:11:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb753400640 1 -- 172.21.15.28:0/3382722216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74c031820 con 0x7fb754070cc0 2024-09-13T17:11:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 -- 172.21.15.28:0/3382722216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb754070cc0 msgr2=0x7fb754071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb754070cc0 0x7fb754071170 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb74c009e80 tx=0x7fb74c02a6c0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 -- 172.21.15.28:0/3382722216 shutdown_connections 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb754070cc0 0x7fb754071170 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 --2- 172.21.15.28:0/3382722216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7540cabc0 0x7fb75406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 -- 172.21.15.28:0/3382722216 >> 172.21.15.28:0/3382722216 conn(0x7fb754069e80 msgr2=0x7fb75406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 -- 172.21.15.28:0/3382722216 shutdown_connections 2024-09-13T17:11:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.125+0000 7fb75ae00640 1 -- 172.21.15.28:0/3382722216 wait complete. 2024-09-13T17:11:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.127+0000 7fb75ae00640 1 Processor -- start 2024-09-13T17:11:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.127+0000 7fb75ae00640 1 -- start start 2024-09-13T17:11:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb754070cc0 0x7fb754150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75415a540 0x7fb754155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540afda0 con 0x7fb7540cabc0 2024-09-13T17:11:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540b0d30 con 0x7fb75415a540 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb75ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7540af190 con 0x7fb754070cc0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb74b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb74b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50776/0 (socket says 172.21.15.28:50776) 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb74b400640 1 -- 172.21.15.28:0/1921043300 learned_addr learned my addr 172.21.15.28:0/1921043300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb753e00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb754070cc0 0x7fb754150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.128+0000 7fb758e00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75415a540 0x7fb754155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 -- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb754070cc0 msgr2=0x7fb754150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb754070cc0 0x7fb754150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 -- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75415a540 msgr2=0x7fb754155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75415a540 0x7fb754155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 -- 172.21.15.28:0/1921043300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c009b30 con 0x7fb7540cabc0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb74b400640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00e980 tx=0x7fb73c00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c009670 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73c00ca00 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73c00cd90 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7540af470 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.129+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7540afa30 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.130+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb754071170 con 0x7fb7540cabc0 2024-09-13T17:11:09.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.131+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb73c013470 con 0x7fb7540cabc0 2024-09-13T17:11:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.132+0000 7fb750c00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 0x7fb73006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.132+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb73c0899b0 con 0x7fb7540cabc0 2024-09-13T17:11:09.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.133+0000 7fb753e00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 0x7fb73006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.133+0000 7fb753e00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 0x7fb73006d0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb744000f60 tx=0x7fb744009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.136+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb73c054af0 con 0x7fb7540cabc0 2024-09-13T17:11:09.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.317+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb75406fd80 con 0x7fb7540cabc0 2024-09-13T17:11:09.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.318+0000 7fb750c00640 1 -- 172.21.15.28:0/1921043300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16526 (secure 0 0 0) 0x7fb73c017090 con 0x7fb7540cabc0 2024-09-13T17:11:09.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:09.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:08.920182+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.320+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 msgr2=0x7fb73006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.320+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 0x7fb73006d0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb744000f60 tx=0x7fb744009040 comp rx=0 tx=0).stop 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.320+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 msgr2=0x7fb754155210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.320+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fb73c00e980 tx=0x7fb73c00ee40 comp rx=0 tx=0).stop 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 shutdown_connections 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb73006ac00 0x7fb73006d0c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb75415a540 0x7fb754155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7540cabc0 0x7fb754155210 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 --2- 172.21.15.28:0/1921043300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb754070cc0 0x7fb754150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 >> 172.21.15.28:0/1921043300 conn(0x7fb754069e80 msgr2=0x7fb75406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 shutdown_connections 2024-09-13T17:11:09.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.321+0000 7fb75ae00640 1 -- 172.21.15.28:0/1921043300 wait complete. 2024-09-13T17:11:09.324 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:11:09.328 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:11:09.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.534+0000 7f2413e00640 1 Processor -- start 2024-09-13T17:11:09.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.534+0000 7f2413e00640 1 -- start start 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.535+0000 7f2413e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2414159820 0x7f2414164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140c3850 con 0x7f2414159440 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241415a930 con 0x7f24140b0b50 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2414159d20 con 0x7f2414159820 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2414159820 0x7f2414164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2412a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2414159440 0x7f24140b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58008/0 (socket says 172.21.15.28:58008) 2024-09-13T17:11:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2412a00640 1 -- 172.21.15.28:0/284925094 learned_addr learned my addr 172.21.15.28:0/284925094 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413400640 1 -- 172.21.15.28:0/284925094 >> v1:172.21.15.189:6789/0 conn(0x7f24140b0b50 legacy=0x7f24140b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413400640 1 -- 172.21.15.28:0/284925094 >> v1:172.21.15.28:6789/0 conn(0x7f2414159440 legacy=0x7f24140b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.536+0000 7f2413400640 1 -- 172.21.15.28:0/284925094 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241415a000 con 0x7f2414159820 2024-09-13T17:11:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.537+0000 7f2413400640 1 --2- 172.21.15.28:0/284925094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159820 0x7f2414164260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f240c009d90 tx=0x7f240c02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7982daf995e2bd81 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2412000640 1 -- 172.21.15.28:0/284925094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c007740 con 0x7f2414159820 2024-09-13T17:11:09.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2412000640 1 -- 172.21.15.28:0/284925094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f240c0078a0 con 0x7f2414159820 2024-09-13T17:11:09.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2412000640 1 -- 172.21.15.28:0/284925094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c007c30 con 0x7f2414159820 2024-09-13T17:11:09.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 -- 172.21.15.28:0/284925094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159820 msgr2=0x7f2414164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 --2- 172.21.15.28:0/284925094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159820 0x7f2414164260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f240c009d90 tx=0x7f240c02a780 comp rx=0 tx=0).stop 2024-09-13T17:11:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 -- 172.21.15.28:0/284925094 shutdown_connections 2024-09-13T17:11:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 --2- 172.21.15.28:0/284925094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159820 0x7f2414164260 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 -- 172.21.15.28:0/284925094 >> 172.21.15.28:0/284925094 conn(0x7f2414072e40 msgr2=0x7f2414073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 -- 172.21.15.28:0/284925094 shutdown_connections 2024-09-13T17:11:09.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.538+0000 7f2413e00640 1 -- 172.21.15.28:0/284925094 wait complete. 2024-09-13T17:11:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.540+0000 7f2413e00640 1 Processor -- start 2024-09-13T17:11:09.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.540+0000 7f2413e00640 1 -- start start 2024-09-13T17:11:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24140b0b50 0x7f241414f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414159820 0x7f2414147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414151010 con 0x7f2414159440 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24141563b0 con 0x7f2414159820 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2414156c50 con 0x7f24140b0b50 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f240be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f240be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50796/0 (socket says 172.21.15.28:50796) 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f240be00640 1 -- 172.21.15.28:0/1975015362 learned_addr learned my addr 172.21.15.28:0/1975015362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2413400640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414159820 0x7f2414147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.541+0000 7f2412a00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24140b0b50 0x7f241414f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 -- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24140b0b50 msgr2=0x7f241414f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24140b0b50 0x7f241414f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 -- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414159820 msgr2=0x7f2414147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414159820 0x7f2414147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 -- 172.21.15.28:0/1975015362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240c009a70 con 0x7f2414159440 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240be00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c480 tx=0x7f23fc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc0052d0 con 0x7f2414159440 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc005430 con 0x7f2414159440 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc012870 con 0x7f2414159440 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.542+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241414f0f0 con 0x7f2414159440 2024-09-13T17:11:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.543+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2414156600 con 0x7f2414159440 2024-09-13T17:11:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.544+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24140b22e0 con 0x7f2414159440 2024-09-13T17:11:09.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.545+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f23fc0129d0 con 0x7f2414159440 2024-09-13T17:11:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.545+0000 7f240aa00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 0x7f23f006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.546+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23fc088d30 con 0x7f2414159440 2024-09-13T17:11:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.546+0000 7f2412a00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 0x7f23f006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.547+0000 7f2412a00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 0x7f23f006d0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2404005880 tx=0x7f2404009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.550+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f23fc054050 con 0x7f2414159440 2024-09-13T17:11:09.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.733+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2414156d90 con 0x7f2414159440 2024-09-13T17:11:09.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.734+0000 7f240aa00640 1 -- 172.21.15.28:0/1975015362 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16526 (secure 0 0 0) 0x7f23fc057670 con 0x7f2414159440 2024-09-13T17:11:09.737 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:09.737 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":47,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:08.920182+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 msgr2=0x7f23f006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 0x7f23f006d0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2404005880 tx=0x7f2404009040 comp rx=0 tx=0).stop 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 msgr2=0x7f24141523e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c480 tx=0x7f23fc00c940 comp rx=0 tx=0).stop 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 shutdown_connections 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23f006ac30 0x7f23f006d0f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2414159820 0x7f2414147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2414159440 0x7f24141523e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 --2- 172.21.15.28:0/1975015362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24140b0b50 0x7f241414f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 >> 172.21.15.28:0/1975015362 conn(0x7f2414072e40 msgr2=0x7f241415adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 shutdown_connections 2024-09-13T17:11:09.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.737+0000 7f2413e00640 1 -- 172.21.15.28:0/1975015362 wait complete. 2024-09-13T17:11:09.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:11:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.911+0000 7f718cc00640 1 Processor -- start 2024-09-13T17:11:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.911+0000 7f718cc00640 1 -- start start 2024-09-13T17:11:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f718cc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71880cc910 0x7f7188072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f718cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f718cc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71880c78e0 con 0x7f71880cc530 2024-09-13T17:11:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f718cc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7188070c90 con 0x7f71880cc910 2024-09-13T17:11:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f718cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71880b1080 con 0x7f71880b06f0 2024-09-13T17:11:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f7187e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f717fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f71880cc910 0x7f7188072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f7187e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:33872/0 (socket says 172.21.15.28:33872) 2024-09-13T17:11:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f7187e00640 1 -- 172.21.15.28:0/2975037516 learned_addr learned my addr 172.21.15.28:0/2975037516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.913+0000 7f7187400640 1 --1- 172.21.15.28:0/2975037516 >> v1:172.21.15.189:6789/0 conn(0x7f71880cc530 0x7f718815d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:53452/0 (socket says 172.21.15.28:53452) 2024-09-13T17:11:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.914+0000 7f7187e00640 1 -- 172.21.15.28:0/2975037516 >> v2:172.21.15.189:3300/0 conn(0x7f71880cc910 msgr2=0x7f7188072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.914+0000 7f7187e00640 1 --2- 172.21.15.28:0/2975037516 >> v2:172.21.15.189:3300/0 conn(0x7f71880cc910 0x7f7188072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.914+0000 7f7187e00640 1 -- 172.21.15.28:0/2975037516 >> v1:172.21.15.189:6789/0 conn(0x7f71880cc530 legacy=0x7f718815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.914+0000 7f7187e00640 1 -- 172.21.15.28:0/2975037516 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71880cbe40 con 0x7f71880b06f0 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.915+0000 7f7187e00640 1 --2- 172.21.15.28:0/2975037516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f718002a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bc4fcaf54403df6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.915+0000 7f7186a00640 1 -- 172.21.15.28:0/2975037516 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180031600 con 0x7f71880b06f0 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.915+0000 7f7186a00640 1 -- 172.21.15.28:0/2975037516 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718002d050 con 0x7f71880b06f0 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.915+0000 7f7186a00640 1 -- 172.21.15.28:0/2975037516 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180031980 con 0x7f71880b06f0 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 -- 172.21.15.28:0/2975037516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 msgr2=0x7f71880b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 --2- 172.21.15.28:0/2975037516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f718002a7a0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 -- 172.21.15.28:0/2975037516 shutdown_connections 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 --2- 172.21.15.28:0/2975037516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880b06f0 0x7f71880b0b40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 --2- 172.21.15.28:0/2975037516 >> v2:172.21.15.189:3300/0 conn(0x7f71880cc910 0x7f7188072060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 -- 172.21.15.28:0/2975037516 >> 172.21.15.28:0/2975037516 conn(0x7f7188069e80 msgr2=0x7f718806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 -- 172.21.15.28:0/2975037516 shutdown_connections 2024-09-13T17:11:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.916+0000 7f718cc00640 1 -- 172.21.15.28:0/2975037516 wait complete. 2024-09-13T17:11:09.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.917+0000 7f718cc00640 1 Processor -- start 2024-09-13T17:11:09.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.917+0000 7f718cc00640 1 -- start start 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 0x7f7188145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 0x7f71880c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880c98c0 con 0x7f71880cc910 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880c8cb0 con 0x7f71880b06f0 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.918+0000 7f718cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71880c8f90 con 0x7f71880cc530 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50812/0 (socket says 172.21.15.28:50812) 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 -- 172.21.15.28:0/613878364 learned_addr learned my addr 172.21.15.28:0/613878364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f717fe00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 0x7f71880c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187400640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 0x7f7188145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 -- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 msgr2=0x7f71880c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 0x7f71880c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 -- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 msgr2=0x7f7188145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 0x7f7188145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 -- 172.21.15.28:0/613878364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7180009b30 con 0x7f71880cc910 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f717fe00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 0x7f71880c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187400640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 0x7f7188145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.919+0000 7f7187e00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f7180004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.920+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718003d860 con 0x7f71880cc910 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.920+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718003d9e0 con 0x7f71880cc910 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.920+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718003dd90 con 0x7f71880cc910 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.920+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71880c9270 con 0x7f71880cc910 2024-09-13T17:11:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.920+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71881461b0 con 0x7f71880cc910 2024-09-13T17:11:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.921+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71880c9a70 con 0x7f71880cc910 2024-09-13T17:11:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.922+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f718003b400 con 0x7f71880cc910 2024-09-13T17:11:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.923+0000 7f717f400640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 0x7f716406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.923+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71800aa230 con 0x7f71880cc910 2024-09-13T17:11:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.923+0000 7f7187400640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 0x7f716406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:09.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.924+0000 7f7187400640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 0x7f716406d0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7178000f60 tx=0x7f7178009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:09.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:09.929+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f71880c9a70 con 0x7f71880cc910 2024-09-13T17:11:10.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.143+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7188071b40 con 0x7f71880cc910 2024-09-13T17:11:10.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.938+0000 7f717f400640 1 -- 172.21.15.28:0/613878364 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7180075370 con 0x7f71880cc910 2024-09-13T17:11:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 msgr2=0x7f716406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 0x7f716406d0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7178000f60 tx=0x7f7178009040 comp rx=0 tx=0).stop 2024-09-13T17:11:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 msgr2=0x7f71880b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:10.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f7180009e50 tx=0x7f7180004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 shutdown_connections 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f716406ac00 0x7f716406d0c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71880cc910 0x7f71880b1ac0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71880cc530 0x7f71880c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 --2- 172.21.15.28:0/613878364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71880b06f0 0x7f7188145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 >> 172.21.15.28:0/613878364 conn(0x7f7188069e80 msgr2=0x7f71880b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:10.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 shutdown_connections 2024-09-13T17:11:10.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:10.941+0000 7f718cc00640 1 -- 172.21.15.28:0/613878364 wait complete. 2024-09-13T17:11:10.962 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:11:10.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:11:11.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.123+0000 7f40c1600640 1 Processor -- start 2024-09-13T17:11:11.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.123+0000 7f40c1600640 1 -- start start 2024-09-13T17:11:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.124+0000 7f40c1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40bc159770 0x7f40bc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.124+0000 7f40c1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc06f3f0 0x7f40bc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.124+0000 7f40c1600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 0x7f40bc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40c1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc06f920 con 0x7f40bc159770 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40c1600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc15b290 con 0x7f40bc0add00 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40c1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc15b570 con 0x7f40bc06f3f0 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40bc159770 0x7f40bc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f40bc159770 0x7f40bc06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50816/0 (socket says 172.21.15.28:50816) 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 -- 172.21.15.28:0/606700010 learned_addr learned my addr 172.21.15.28:0/606700010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40bb400640 1 --2- 172.21.15.28:0/606700010 >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 0x7f40bc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40ba000640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc06f3f0 0x7f40bc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 -- 172.21.15.28:0/606700010 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc06f3f0 msgr2=0x7f40bc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc06f3f0 0x7f40bc070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 -- 172.21.15.28:0/606700010 >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 msgr2=0x7f40bc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 --2- 172.21.15.28:0/606700010 >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 0x7f40bc159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40baa00640 1 -- 172.21.15.28:0/606700010 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc15b850 con 0x7f40bc159770 2024-09-13T17:11:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.125+0000 7f40bb400640 1 --2- 172.21.15.28:0/606700010 >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 0x7f40bc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.126+0000 7f40baa00640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc06fd20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0b0e90 tx=0x7f40ac02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f583bde162646c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.127+0000 7f40b9600640 1 -- 172.21.15.28:0/606700010 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac02ae90 con 0x7f40bc159770 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.127+0000 7f40b9600640 1 -- 172.21.15.28:0/606700010 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac032440 con 0x7f40bc159770 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.127+0000 7f40b9600640 1 -- 172.21.15.28:0/606700010 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0327d0 con 0x7f40bc159770 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.127+0000 7f40c1600640 1 -- 172.21.15.28:0/606700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 msgr2=0x7f40bc06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.127+0000 7f40c1600640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc06fd20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f40bc0b0e90 tx=0x7f40ac02a140 comp rx=0 tx=0).stop 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 -- 172.21.15.28:0/606700010 shutdown_connections 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 --2- 172.21.15.28:0/606700010 >> v2:172.21.15.189:3300/0 conn(0x7f40bc0add00 0x7f40bc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc06f3f0 0x7f40bc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 --2- 172.21.15.28:0/606700010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc06fd20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 -- 172.21.15.28:0/606700010 >> 172.21.15.28:0/606700010 conn(0x7f40bc0b0270 msgr2=0x7f40bc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 -- 172.21.15.28:0/606700010 shutdown_connections 2024-09-13T17:11:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.128+0000 7f40c1600640 1 -- 172.21.15.28:0/606700010 wait complete. 2024-09-13T17:11:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.130+0000 7f40c1600640 1 Processor -- start 2024-09-13T17:11:11.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.130+0000 7f40c1600640 1 -- start start 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 0x7f40bc1499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc0add00 0x7f40bc146680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc147eb0 con 0x7f40bc159770 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc149ee0 con 0x7f40bc06f3f0 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.131+0000 7f40c1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc1472a0 con 0x7f40bc0add00 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40baa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 0x7f40bc1499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40baa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 0x7f40bc1499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:41112/0 (socket says 172.21.15.28:41112) 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50824/0 (socket says 172.21.15.28:50824) 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40baa00640 1 -- 172.21.15.28:0/3814305827 learned_addr learned my addr 172.21.15.28:0/3814305827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40ba000640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc0add00 0x7f40bc146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 -- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc0add00 msgr2=0x7f40bc146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc0add00 0x7f40bc146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 -- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 msgr2=0x7f40bc1499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:11.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 0x7f40bc1499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 -- 172.21.15.28:0/3814305827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac0094d0 con 0x7f40bc159770 2024-09-13T17:11:11.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.132+0000 7f40bb400640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40b4007f20 tx=0x7f40b400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.133+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4003b90 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.133+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b400f040 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.133+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4004030 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.133+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc147580 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.133+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc147b40 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.134+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc148060 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.135+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f40b4004190 con 0x7f40bc159770 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.136+0000 7f40b2a00640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 0x7f409c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:11.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.136+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40b40895e0 con 0x7f40bc159770 2024-09-13T17:11:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.137+0000 7f40baa00640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 0x7f409c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:11.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.137+0000 7f40baa00640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 0x7f409c06d0c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0090b0 tx=0x7f40ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:11.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.140+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f40bc148060 con 0x7f40bc159770 2024-09-13T17:11:11.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.359+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f40bc148060 con 0x7f40bc159770 2024-09-13T17:11:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.939+0000 7f40b2a00640 1 -- 172.21.15.28:0/3814305827 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7f40bc148060 con 0x7f40bc159770 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 msgr2=0x7f409c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 0x7f409c06d0c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0090b0 tx=0x7f40ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 msgr2=0x7f40bc14a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f40b4007f20 tx=0x7f40b400d560 comp rx=0 tx=0).stop 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 shutdown_connections 2024-09-13T17:11:11.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f409c06ac00 0x7f409c06d0c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40bc159770 0x7f40bc14a3f0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f40bc0add00 0x7f40bc146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 --2- 172.21.15.28:0/3814305827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40bc06f3f0 0x7f40bc1499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 >> 172.21.15.28:0/3814305827 conn(0x7f40bc0b0270 msgr2=0x7f40bc06fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 shutdown_connections 2024-09-13T17:11:11.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:11.942+0000 7f40c1600640 1 -- 172.21.15.28:0/3814305827 wait complete. 2024-09-13T17:11:11.962 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:11:11.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.117+0000 7f2033e00640 1 Processor -- start 2024-09-13T17:11:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.117+0000 7f2033e00640 1 -- start start 2024-09-13T17:11:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20340b0b50 0x7f20340b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2034159820 0x7f20340b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20340c3850 con 0x7f2034159440 2024-09-13T17:11:12.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20340b2d60 con 0x7f2034159820 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203415a930 con 0x7f20340b0b50 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2032a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2034159440 0x7f20340b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58020/0 (socket says 172.21.15.28:58020) 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2032a00640 1 -- 172.21.15.28:0/4093477029 learned_addr learned my addr 172.21.15.28:0/4093477029 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2033400640 1 --2- 172.21.15.28:0/4093477029 >> v2:172.21.15.28:3300/0 conn(0x7f2034159820 0x7f20340b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.119+0000 7f2032000640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20340b0b50 0x7f20340b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.120+0000 7f2033400640 1 -- 172.21.15.28:0/4093477029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20340b0b50 msgr2=0x7f20340b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.120+0000 7f2033400640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20340b0b50 0x7f20340b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.120+0000 7f2033400640 1 -- 172.21.15.28:0/4093477029 >> v1:172.21.15.28:6789/0 conn(0x7f2034159440 legacy=0x7f20340b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.120+0000 7f2033400640 1 -- 172.21.15.28:0/4093477029 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2034159d20 con 0x7f2034159820 2024-09-13T17:11:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.120+0000 7f2031600640 1 -- 172.21.15.28:0/4093477029 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562593496 0 0) 0x7f20340c3850 con 0x7f2034159440 2024-09-13T17:11:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033400640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159820 0x7f20340b2820 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb15bbf6337a45d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2031600640 1 -- 172.21.15.28:0/4093477029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c007740 con 0x7f2034159820 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2031600640 1 -- 172.21.15.28:0/4093477029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c02d050 con 0x7f2034159820 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2031600640 1 -- 172.21.15.28:0/4093477029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c007ac0 con 0x7f2034159820 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 -- 172.21.15.28:0/4093477029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159820 msgr2=0x7f20340b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159820 0x7f20340b2820 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f202c009d90 tx=0x7f202c02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 -- 172.21.15.28:0/4093477029 shutdown_connections 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159820 0x7f20340b2820 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 --2- 172.21.15.28:0/4093477029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f20340b0b50 0x7f20340b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.121+0000 7f2033e00640 1 -- 172.21.15.28:0/4093477029 >> 172.21.15.28:0/4093477029 conn(0x7f2034072e40 msgr2=0x7f2034073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.122+0000 7f2033e00640 1 -- 172.21.15.28:0/4093477029 shutdown_connections 2024-09-13T17:11:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.122+0000 7f2033e00640 1 -- 172.21.15.28:0/4093477029 wait complete. 2024-09-13T17:11:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.123+0000 7f2033e00640 1 Processor -- start 2024-09-13T17:11:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.123+0000 7f2033e00640 1 -- start start 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20340b0b50 0x7f20341516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2034159820 0x7f203415d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20341529b0 con 0x7f2034159440 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203414d6e0 con 0x7f20340b0b50 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2033e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2034149780 con 0x7f2034159820 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.124+0000 7f2032000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50852/0 (socket says 172.21.15.28:50852) 2024-09-13T17:11:12.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 -- 172.21.15.28:0/2423857079 learned_addr learned my addr 172.21.15.28:0/2423857079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2033400640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2034159820 0x7f203415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032a00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20340b0b50 0x7f20341516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 -- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2034159820 msgr2=0x7f203415d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2034159820 0x7f203415d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 -- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20340b0b50 msgr2=0x7f20341516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20340b0b50 0x7f20341516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 -- 172.21.15.28:0/2423857079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202c009a70 con 0x7f2034159440 2024-09-13T17:11:12.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f2032000640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f202000d800 tx=0x7f202000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.125+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000b6c0 con 0x7f2034159440 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.126+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202000b820 con 0x7f2034159440 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.126+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2034151da0 con 0x7f2034159440 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.126+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202000bbb0 con 0x7f2034159440 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.126+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2034152360 con 0x7f2034159440 2024-09-13T17:11:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.127+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203415ada0 con 0x7f2034159440 2024-09-13T17:11:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.127+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f202000bd10 con 0x7f2034159440 2024-09-13T17:11:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.128+0000 7f202aa00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 0x7f201406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.128+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2020089980 con 0x7f2034159440 2024-09-13T17:11:12.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.129+0000 7f2032a00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 0x7f201406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.129+0000 7f2032a00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 0x7f201406d0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2024000f60 tx=0x7f2024007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.132+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f203415ada0 con 0x7f2034159440 2024-09-13T17:11:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.345+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f203415ada0 con 0x7f2034159440 2024-09-13T17:11:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.345+0000 7f202aa00640 1 -- 172.21.15.28:0/2423857079 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f203415ada0 con 0x7f2034159440 2024-09-13T17:11:12.348 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:12.348 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 msgr2=0x7f201406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 0x7f201406d0c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2024000f60 tx=0x7f2024007b50 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 msgr2=0x7f2034149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f202000d800 tx=0x7f202000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 shutdown_connections 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f201406ac00 0x7f201406d0c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.347+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2034159820 0x7f203415d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.348+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2034159440 0x7f2034149240 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.348+0000 7f2033e00640 1 --2- 172.21.15.28:0/2423857079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20340b0b50 0x7f20341516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.348+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 >> 172.21.15.28:0/2423857079 conn(0x7f2034072e40 msgr2=0x7f2034069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.348+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 shutdown_connections 2024-09-13T17:11:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.348+0000 7f2033e00640 1 -- 172.21.15.28:0/2423857079 wait complete. 2024-09-13T17:11:12.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.519+0000 7fc25ae00640 1 Processor -- start 2024-09-13T17:11:12.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.519+0000 7fc25ae00640 1 -- start start 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.520+0000 7fc25ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc25415a540 0x7fc25406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.520+0000 7fc25ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25415e100 con 0x7fc254070cc0 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc25ae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc25415e3e0 con 0x7fc2540cabc0 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc25ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2540adb00 con 0x7fc25415a540 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc25415a540 0x7fc25406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc25415a540 0x7fc25406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50862/0 (socket says 172.21.15.28:50862) 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 -- 172.21.15.28:0/68279398 learned_addr learned my addr 172.21.15.28:0/68279398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:12.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc258e00640 1 --1- 172.21.15.28:0/68279398 >> v1:172.21.15.28:6789/0 conn(0x7fc254070cc0 0x7fc25406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58022/0 (socket says 172.21.15.28:58022) 2024-09-13T17:11:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 -- 172.21.15.28:0/68279398 >> v1:172.21.15.189:6789/0 conn(0x7fc2540cabc0 legacy=0x7fc25415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 -- 172.21.15.28:0/68279398 >> v1:172.21.15.28:6789/0 conn(0x7fc254070cc0 legacy=0x7fc25406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.521+0000 7fc253e00640 1 -- 172.21.15.28:0/68279398 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25415ef80 con 0x7fc25415a540 2024-09-13T17:11:12.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.522+0000 7fc253e00640 1 --2- 172.21.15.28:0/68279398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25415a540 0x7fc25406c430 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c36e7c1a0785735 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.522+0000 7fc253400640 1 -- 172.21.15.28:0/68279398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244032440 con 0x7fc25415a540 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.522+0000 7fc253400640 1 -- 172.21.15.28:0/68279398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2440325a0 con 0x7fc25415a540 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.522+0000 7fc253400640 1 -- 172.21.15.28:0/68279398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244032930 con 0x7fc25415a540 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 -- 172.21.15.28:0/68279398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25415a540 msgr2=0x7fc25406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 --2- 172.21.15.28:0/68279398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25415a540 0x7fc25406c430 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fc244009870 tx=0x7fc24402a270 comp rx=0 tx=0).stop 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 -- 172.21.15.28:0/68279398 shutdown_connections 2024-09-13T17:11:12.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 --2- 172.21.15.28:0/68279398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc25415a540 0x7fc25406c430 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 -- 172.21.15.28:0/68279398 >> 172.21.15.28:0/68279398 conn(0x7fc254069e80 msgr2=0x7fc25406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 -- 172.21.15.28:0/68279398 shutdown_connections 2024-09-13T17:11:12.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.523+0000 7fc25ae00640 1 -- 172.21.15.28:0/68279398 wait complete. 2024-09-13T17:11:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.525+0000 7fc25ae00640 1 Processor -- start 2024-09-13T17:11:12.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.525+0000 7fc25ae00640 1 -- start start 2024-09-13T17:11:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2540cabc0 0x7fc254140f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc25415a540 0x7fc2540ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540b2f00 con 0x7fc254070cc0 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540b0e80 con 0x7fc25415a540 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc25ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2540b22f0 con 0x7fc2540cabc0 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc253e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc253e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50870/0 (socket says 172.21.15.28:50870) 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc253e00640 1 -- 172.21.15.28:0/965768194 learned_addr learned my addr 172.21.15.28:0/965768194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc258e00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc25415a540 0x7fc2540ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.526+0000 7fc24be00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2540cabc0 0x7fc254140f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 -- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2540cabc0 msgr2=0x7fc254140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2540cabc0 0x7fc254140f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 -- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc25415a540 msgr2=0x7fc2540ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc25415a540 0x7fc2540ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 -- 172.21.15.28:0/965768194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2440094d0 con 0x7fc254070cc0 2024-09-13T17:11:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc253e00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc244030b80 con 0x7fc254070cc0 2024-09-13T17:11:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc244030d00 con 0x7fc254070cc0 2024-09-13T17:11:12.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2540b25d0 con 0x7fc254070cc0 2024-09-13T17:11:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.527+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc24403d680 con 0x7fc254070cc0 2024-09-13T17:11:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.528+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2540b2b90 con 0x7fc254070cc0 2024-09-13T17:11:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.529+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc254061090 con 0x7fc254070cc0 2024-09-13T17:11:12.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.529+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc244039040 con 0x7fc254070cc0 2024-09-13T17:11:12.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.530+0000 7fc250c00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 0x7fc23006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.530+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2440aa100 con 0x7fc254070cc0 2024-09-13T17:11:12.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.531+0000 7fc24be00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 0x7fc23006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.531+0000 7fc24be00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 0x7fc23006d0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc2541e5f80 tx=0x7fc23c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.535+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc2440752f0 con 0x7fc254070cc0 2024-09-13T17:11:12.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.715+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc25406c430 con 0x7fc254070cc0 2024-09-13T17:11:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.716+0000 7fc250c00640 1 -- 172.21.15.28:0/965768194 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16513 (secure 0 0 0) 0x7fc244078910 con 0x7fc254070cc0 2024-09-13T17:11:12.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:12.719 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:09.922940+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 msgr2=0x7fc23006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 0x7fc23006d0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc2541e5f80 tx=0x7fc23c009040 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 msgr2=0x7fc2540b3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fc244000c00 tx=0x7fc2440046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 shutdown_connections 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc23006ac00 0x7fc23006d0c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc25415a540 0x7fc2540ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2540cabc0 0x7fc254140f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 --2- 172.21.15.28:0/965768194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc254070cc0 0x7fc2540b3390 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.718+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 >> 172.21.15.28:0/965768194 conn(0x7fc254069e80 msgr2=0x7fc2540ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.719+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 shutdown_connections 2024-09-13T17:11:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.719+0000 7fc25ae00640 1 -- 172.21.15.28:0/965768194 wait complete. 2024-09-13T17:11:12.737 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:11:12.737 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:11:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.891+0000 7f21cf600640 1 Processor -- start 2024-09-13T17:11:12.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.891+0000 7f21cf600640 1 -- start start 2024-09-13T17:11:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 0x7f21c806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c80cabc0 0x7f21c815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21c815edb0 0x7f21c8070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c80adb00 con 0x7f21c815edb0 2024-09-13T17:11:12.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21c815e100 con 0x7f21c815a540 2024-09-13T17:11:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cf600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c815e3e0 con 0x7f21c80cabc0 2024-09-13T17:11:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21c815edb0 0x7f21c8070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21c815edb0 0x7f21c8070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50886/0 (socket says 172.21.15.28:50886) 2024-09-13T17:11:12.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.892+0000 7f21cd600640 1 -- 172.21.15.28:0/4044889223 learned_addr learned my addr 172.21.15.28:0/4044889223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21c7e00640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c80cabc0 0x7f21c815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21ccc00640 1 --2- 172.21.15.28:0/4044889223 >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 0x7f21c806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21cd600640 1 -- 172.21.15.28:0/4044889223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c80cabc0 msgr2=0x7f21c815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21cd600640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c80cabc0 0x7f21c815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21cd600640 1 -- 172.21.15.28:0/4044889223 >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 msgr2=0x7f21c806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21cd600640 1 --2- 172.21.15.28:0/4044889223 >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 0x7f21c806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21cd600640 1 -- 172.21.15.28:0/4044889223 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c806ce80 con 0x7f21c815edb0 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.893+0000 7f21ccc00640 1 --2- 172.21.15.28:0/4044889223 >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 0x7f21c806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:12.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.894+0000 7f21cd600640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c815edb0 0x7f21c8070cc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f21c80b3a00 tx=0x7f21c002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62bbe11a2e810d09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.894+0000 7f21c7400640 1 -- 172.21.15.28:0/4044889223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0031600 con 0x7f21c815edb0 2024-09-13T17:11:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.894+0000 7f21c7400640 1 -- 172.21.15.28:0/4044889223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c002d050 con 0x7f21c815edb0 2024-09-13T17:11:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.894+0000 7f21c7400640 1 -- 172.21.15.28:0/4044889223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0031980 con 0x7f21c815edb0 2024-09-13T17:11:12.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 -- 172.21.15.28:0/4044889223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c815edb0 msgr2=0x7f21c8070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c815edb0 0x7f21c8070cc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f21c80b3a00 tx=0x7f21c002a610 comp rx=0 tx=0).stop 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 -- 172.21.15.28:0/4044889223 shutdown_connections 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c815edb0 0x7f21c8070cc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 --2- 172.21.15.28:0/4044889223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c80cabc0 0x7f21c815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 --2- 172.21.15.28:0/4044889223 >> v2:172.21.15.189:3300/0 conn(0x7f21c815a540 0x7f21c806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 -- 172.21.15.28:0/4044889223 >> 172.21.15.28:0/4044889223 conn(0x7f21c8069e80 msgr2=0x7f21c806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 -- 172.21.15.28:0/4044889223 shutdown_connections 2024-09-13T17:11:12.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.895+0000 7f21cf600640 1 -- 172.21.15.28:0/4044889223 wait complete. 2024-09-13T17:11:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.897+0000 7f21cf600640 1 Processor -- start 2024-09-13T17:11:12.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.897+0000 7f21cf600640 1 -- start start 2024-09-13T17:11:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 0x7f21c80cccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 0x7f21c8151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c815ae10 con 0x7f21c80cabc0 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c815b0f0 con 0x7f21c81f0540 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cf600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c815b3d0 con 0x7f21c815a540 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50898/0 (socket says 172.21.15.28:50898) 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 -- 172.21.15.28:0/1166706462 learned_addr learned my addr 172.21.15.28:0/1166706462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21c7e00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 0x7f21c80cccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cd600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 0x7f21c8151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 -- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 msgr2=0x7f21c80cccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 0x7f21c80cccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 -- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 msgr2=0x7f21c8151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 0x7f21c8151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:12.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21ccc00640 1 -- 172.21.15.28:0/1166706462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c0009b30 con 0x7f21c80cabc0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21c7e00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 0x7f21c80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.898+0000 7f21cd600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 0x7f21c8151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21ccc00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00c7f0 tx=0x7f21bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc007a30 con 0x7f21c80cabc0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc007b90 con 0x7f21c80cabc0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c815b6b0 con 0x7f21c80cabc0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc00ce40 con 0x7f21c80cabc0 2024-09-13T17:11:12.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.899+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c814bee0 con 0x7f21c80cabc0 2024-09-13T17:11:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.900+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c815bbd0 con 0x7f21c80cabc0 2024-09-13T17:11:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.901+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f21bc0113f0 con 0x7f21c80cabc0 2024-09-13T17:11:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.902+0000 7f21c4c00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 0x7f21a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.902+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f21bc089480 con 0x7f21c80cabc0 2024-09-13T17:11:12.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.903+0000 7f21c7e00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 0x7f21a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:12.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.903+0000 7f21c7e00640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 0x7f21a806d0f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f21b4006fd0 tx=0x7f21b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:12.906+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f21c815bbd0 con 0x7f21c80cabc0 2024-09-13T17:11:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.085+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21c815b260 con 0x7f21a806ac30 2024-09-13T17:11:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.095+0000 7f21c4c00640 1 -- 172.21.15.28:0/1166706462 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289531 (secure 0 0 0) 0x7f21c815b260 con 0x7f21a806ac30 2024-09-13T17:11:13.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:13.102 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 msgr2=0x7f21a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 0x7f21a806d0f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f21b4006fd0 tx=0x7f21b4008040 comp rx=0 tx=0).stop 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 msgr2=0x7f21c814b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00c7f0 tx=0x7f21bc00cbf0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 shutdown_connections 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21a806ac30 0x7f21a806d0f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21c81f0540 0x7f21c8151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21c815a540 0x7f21c80cccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 --2- 172.21.15.28:0/1166706462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21c80cabc0 0x7f21c814b9a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 >> 172.21.15.28:0/1166706462 conn(0x7f21c8069e80 msgr2=0x7f21c80b0aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 shutdown_connections 2024-09-13T17:11:13.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.099+0000 7f21cf600640 1 -- 172.21.15.28:0/1166706462 wait complete. 2024-09-13T17:11:13.117 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T17:11:12.928972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319340,"kb_used_data":5632,"kb_used_omap":12,"kb_used_meta":313587,"kb_avail":754655380,"statfs":{"total":773094113280,"available":772767109120,"internally_reserved":0,"allocated":5767168,"data_stored":3044326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":321113657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.447975"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131268+0000","last_change":"2024-09-13T17:11:08.947894+0000","last_active":"2024-09-13T17:11:11.131268+0000","last_peered":"2024-09-13T17:11:11.131268+0000","last_clean":"2024-09-13T17:11:11.131268+0000","last_became_active":"2024-09-13T17:11:08.947556+0000","last_became_peered":"2024-09-13T17:11:08.947556+0000","last_unstale":"2024-09-13T17:11:11.131268+0000","last_undegraded":"2024-09-13T17:11:11.131268+0000","last_fullsized":"2024-09-13T17:11:11.131268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:17.772673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935086+0000","last_change":"2024-09-13T17:11:08.945962+0000","last_active":"2024-09-13T17:11:09.935086+0000","last_peered":"2024-09-13T17:11:09.935086+0000","last_clean":"2024-09-13T17:11:09.935086+0000","last_became_active":"2024-09-13T17:11:08.945672+0000","last_became_peered":"2024-09-13T17:11:08.945672+0000","last_unstale":"2024-09-13T17:11:09.935086+0000","last_undegraded":"2024-09-13T17:11:09.935086+0000","last_fullsized":"2024-09-13T17:11:09.935086+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935077+0000","last_change":"2024-09-13T17:11:08.948950+0000","last_active":"2024-09-13T17:11:09.935077+0000","last_peered":"2024-09-13T17:11:09.935077+0000","last_clean":"2024-09-13T17:11:09.935077+0000","last_became_active":"2024-09-13T17:11:08.948692+0000","last_became_peered":"2024-09-13T17:11:08.948692+0000","last_unstale":"2024-09-13T17:11:09.935077+0000","last_undegraded":"2024-09-13T17:11:09.935077+0000","last_fullsized":"2024-09-13T17:11:09.935077+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:00.729934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131456+0000","last_change":"2024-09-13T17:11:08.946655+0000","last_active":"2024-09-13T17:11:11.131456+0000","last_peered":"2024-09-13T17:11:11.131456+0000","last_clean":"2024-09-13T17:11:11.131456+0000","last_became_active":"2024-09-13T17:11:08.946418+0000","last_became_peered":"2024-09-13T17:11:08.946418+0000","last_unstale":"2024-09-13T17:11:11.131456+0000","last_undegraded":"2024-09-13T17:11:11.131456+0000","last_fullsized":"2024-09-13T17:11:11.131456+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:58.261986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934865+0000","last_change":"2024-09-13T17:11:08.942166+0000","last_active":"2024-09-13T17:11:09.934865+0000","last_peered":"2024-09-13T17:11:09.934865+0000","last_clean":"2024-09-13T17:11:09.934865+0000","last_became_active":"2024-09-13T17:11:08.941636+0000","last_became_peered":"2024-09-13T17:11:08.941636+0000","last_unstale":"2024-09-13T17:11:09.934865+0000","last_undegraded":"2024-09-13T17:11:09.934865+0000","last_fullsized":"2024-09-13T17:11:09.934865+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130404+0000","last_change":"2024-09-13T17:11:08.940394+0000","last_active":"2024-09-13T17:11:11.130404+0000","last_peered":"2024-09-13T17:11:11.130404+0000","last_clean":"2024-09-13T17:11:11.130404+0000","last_became_active":"2024-09-13T17:11:08.937877+0000","last_became_peered":"2024-09-13T17:11:08.937877+0000","last_unstale":"2024-09-13T17:11:11.130404+0000","last_undegraded":"2024-09-13T17:11:11.130404+0000","last_fullsized":"2024-09-13T17:11:11.130404+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:16:19.951894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130918+0000","last_change":"2024-09-13T17:11:08.943653+0000","last_active":"2024-09-13T17:11:11.130918+0000","last_peered":"2024-09-13T17:11:11.130918+0000","last_clean":"2024-09-13T17:11:11.130918+0000","last_became_active":"2024-09-13T17:11:08.942874+0000","last_became_peered":"2024-09-13T17:11:08.942874+0000","last_unstale":"2024-09-13T17:11:11.130918+0000","last_undegraded":"2024-09-13T17:11:11.130918+0000","last_fullsized":"2024-09-13T17:11:11.130918+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:58.261986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130961+0000","last_change":"2024-09-13T17:11:08.946312+0000","last_active":"2024-09-13T17:11:11.130961+0000","last_peered":"2024-09-13T17:11:11.130961+0000","last_clean":"2024-09-13T17:11:11.130961+0000","last_became_active":"2024-09-13T17:11:08.945920+0000","last_became_peered":"2024-09-13T17:11:08.945920+0000","last_unstale":"2024-09-13T17:11:11.130961+0000","last_undegraded":"2024-09-13T17:11:11.130961+0000","last_fullsized":"2024-09-13T17:11:11.130961+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:47.991171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131840+0000","last_change":"2024-09-13T17:11:08.952918+0000","last_active":"2024-09-13T17:11:11.131840+0000","last_peered":"2024-09-13T17:11:11.131840+0000","last_clean":"2024-09-13T17:11:11.131840+0000","last_became_active":"2024-09-13T17:11:08.952326+0000","last_became_peered":"2024-09-13T17:11:08.952326+0000","last_unstale":"2024-09-13T17:11:11.131840+0000","last_undegraded":"2024-09-13T17:11:11.131840+0000","last_fullsized":"2024-09-13T17:11:11.131840+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:06.694348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131635+0000","last_change":"2024-09-13T17:11:08.948383+0000","last_active":"2024-09-13T17:11:11.131635+0000","last_peered":"2024-09-13T17:11:11.131635+0000","last_clean":"2024-09-13T17:11:11.131635+0000","last_became_active":"2024-09-13T17:11:08.944675+0000","last_became_peered":"2024-09-13T17:11:08.944675+0000","last_unstale":"2024-09-13T17:11:11.131635+0000","last_undegraded":"2024-09-13T17:11:11.131635+0000","last_fullsized":"2024-09-13T17:11:11.131635+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:11.796310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131501+0000","last_change":"2024-09-13T17:11:08.949752+0000","last_active":"2024-09-13T17:11:11.131501+0000","last_peered":"2024-09-13T17:11:11.131501+0000","last_clean":"2024-09-13T17:11:11.131501+0000","last_became_active":"2024-09-13T17:11:08.949514+0000","last_became_peered":"2024-09-13T17:11:08.949514+0000","last_unstale":"2024-09-13T17:11:11.131501+0000","last_undegraded":"2024-09-13T17:11:11.131501+0000","last_fullsized":"2024-09-13T17:11:11.131501+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935017+0000","last_change":"2024-09-13T17:11:08.939465+0000","last_active":"2024-09-13T17:11:09.935017+0000","last_peered":"2024-09-13T17:11:09.935017+0000","last_clean":"2024-09-13T17:11:09.935017+0000","last_became_active":"2024-09-13T17:11:08.939107+0000","last_became_peered":"2024-09-13T17:11:08.939107+0000","last_unstale":"2024-09-13T17:11:09.935017+0000","last_undegraded":"2024-09-13T17:11:09.935017+0000","last_fullsized":"2024-09-13T17:11:09.935017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:06.694348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735182+0000","last_change":"2024-09-13T17:11:08.949121+0000","last_active":"2024-09-13T17:11:10.735182+0000","last_peered":"2024-09-13T17:11:10.735182+0000","last_clean":"2024-09-13T17:11:10.735182+0000","last_became_active":"2024-09-13T17:11:08.948876+0000","last_became_peered":"2024-09-13T17:11:08.948876+0000","last_unstale":"2024-09-13T17:11:10.735182+0000","last_undegraded":"2024-09-13T17:11:10.735182+0000","last_fullsized":"2024-09-13T17:11:10.735182+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:54.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131017+0000","last_change":"2024-09-13T17:11:08.955847+0000","last_active":"2024-09-13T17:11:11.131017+0000","last_peered":"2024-09-13T17:11:11.131017+0000","last_clean":"2024-09-13T17:11:11.131017+0000","last_became_active":"2024-09-13T17:11:08.955609+0000","last_became_peered":"2024-09-13T17:11:08.955609+0000","last_unstale":"2024-09-13T17:11:11.131017+0000","last_undegraded":"2024-09-13T17:11:11.131017+0000","last_fullsized":"2024-09-13T17:11:11.131017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130297+0000","last_change":"2024-09-13T17:11:08.941554+0000","last_active":"2024-09-13T17:11:11.130297+0000","last_peered":"2024-09-13T17:11:11.130297+0000","last_clean":"2024-09-13T17:11:11.130297+0000","last_became_active":"2024-09-13T17:11:08.941307+0000","last_became_peered":"2024-09-13T17:11:08.941307+0000","last_unstale":"2024-09-13T17:11:11.130297+0000","last_undegraded":"2024-09-13T17:11:11.130297+0000","last_fullsized":"2024-09-13T17:11:11.130297+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:18.081627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147530+0000","last_change":"2024-09-13T17:11:08.960818+0000","last_active":"2024-09-13T17:11:10.147530+0000","last_peered":"2024-09-13T17:11:10.147530+0000","last_clean":"2024-09-13T17:11:10.147530+0000","last_became_active":"2024-09-13T17:11:08.960593+0000","last_became_peered":"2024-09-13T17:11:08.960593+0000","last_unstale":"2024-09-13T17:11:10.147530+0000","last_undegraded":"2024-09-13T17:11:10.147530+0000","last_fullsized":"2024-09-13T17:11:10.147530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.456687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935553+0000","last_change":"2024-09-13T17:11:08.952153+0000","last_active":"2024-09-13T17:11:09.935553+0000","last_peered":"2024-09-13T17:11:09.935553+0000","last_clean":"2024-09-13T17:11:09.935553+0000","last_became_active":"2024-09-13T17:11:08.951210+0000","last_became_peered":"2024-09-13T17:11:08.951210+0000","last_unstale":"2024-09-13T17:11:09.935553+0000","last_undegraded":"2024-09-13T17:11:09.935553+0000","last_fullsized":"2024-09-13T17:11:09.935553+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:39.355617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935246+0000","last_change":"2024-09-13T17:11:08.955918+0000","last_active":"2024-09-13T17:11:09.935246+0000","last_peered":"2024-09-13T17:11:09.935246+0000","last_clean":"2024-09-13T17:11:09.935246+0000","last_became_active":"2024-09-13T17:11:08.955640+0000","last_became_peered":"2024-09-13T17:11:08.955640+0000","last_unstale":"2024-09-13T17:11:09.935246+0000","last_undegraded":"2024-09-13T17:11:09.935246+0000","last_fullsized":"2024-09-13T17:11:09.935246+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:06.452065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148082+0000","last_change":"2024-09-13T17:11:08.955973+0000","last_active":"2024-09-13T17:11:10.148082+0000","last_peered":"2024-09-13T17:11:10.148082+0000","last_clean":"2024-09-13T17:11:10.148082+0000","last_became_active":"2024-09-13T17:11:08.955672+0000","last_became_peered":"2024-09-13T17:11:08.955672+0000","last_unstale":"2024-09-13T17:11:10.148082+0000","last_undegraded":"2024-09-13T17:11:10.148082+0000","last_fullsized":"2024-09-13T17:11:10.148082+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:05.881069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735287+0000","last_change":"2024-09-13T17:11:08.955292+0000","last_active":"2024-09-13T17:11:10.735287+0000","last_peered":"2024-09-13T17:11:10.735287+0000","last_clean":"2024-09-13T17:11:10.735287+0000","last_became_active":"2024-09-13T17:11:08.955026+0000","last_became_peered":"2024-09-13T17:11:08.955026+0000","last_unstale":"2024-09-13T17:11:10.735287+0000","last_undegraded":"2024-09-13T17:11:10.735287+0000","last_fullsized":"2024-09-13T17:11:10.735287+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:27.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130389+0000","last_change":"2024-09-13T17:11:08.957526+0000","last_active":"2024-09-13T17:11:11.130389+0000","last_peered":"2024-09-13T17:11:11.130389+0000","last_clean":"2024-09-13T17:11:11.130389+0000","last_became_active":"2024-09-13T17:11:08.957056+0000","last_became_peered":"2024-09-13T17:11:08.957056+0000","last_unstale":"2024-09-13T17:11:11.130389+0000","last_undegraded":"2024-09-13T17:11:11.130389+0000","last_fullsized":"2024-09-13T17:11:11.130389+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:08.483718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131111+0000","last_change":"2024-09-13T17:11:08.943501+0000","last_active":"2024-09-13T17:11:11.131111+0000","last_peered":"2024-09-13T17:11:11.131111+0000","last_clean":"2024-09-13T17:11:11.131111+0000","last_became_active":"2024-09-13T17:11:08.943259+0000","last_became_peered":"2024-09-13T17:11:08.943259+0000","last_unstale":"2024-09-13T17:11:11.131111+0000","last_undegraded":"2024-09-13T17:11:11.131111+0000","last_fullsized":"2024-09-13T17:11:11.131111+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.456687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147848+0000","last_change":"2024-09-13T17:11:08.946631+0000","last_active":"2024-09-13T17:11:10.147848+0000","last_peered":"2024-09-13T17:11:10.147848+0000","last_clean":"2024-09-13T17:11:10.147848+0000","last_became_active":"2024-09-13T17:11:08.945998+0000","last_became_peered":"2024-09-13T17:11:08.945998+0000","last_unstale":"2024-09-13T17:11:10.147848+0000","last_undegraded":"2024-09-13T17:11:10.147848+0000","last_fullsized":"2024-09-13T17:11:10.147848+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:54.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147610+0000","last_change":"2024-09-13T17:11:08.949674+0000","last_active":"2024-09-13T17:11:10.147610+0000","last_peered":"2024-09-13T17:11:10.147610+0000","last_clean":"2024-09-13T17:11:10.147610+0000","last_became_active":"2024-09-13T17:11:08.949352+0000","last_became_peered":"2024-09-13T17:11:08.949352+0000","last_unstale":"2024-09-13T17:11:10.147610+0000","last_undegraded":"2024-09-13T17:11:10.147610+0000","last_fullsized":"2024-09-13T17:11:10.147610+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:27.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148016+0000","last_change":"2024-09-13T17:11:08.954704+0000","last_active":"2024-09-13T17:11:10.148016+0000","last_peered":"2024-09-13T17:11:10.148016+0000","last_clean":"2024-09-13T17:11:10.148016+0000","last_became_active":"2024-09-13T17:11:08.954434+0000","last_became_peered":"2024-09-13T17:11:08.954434+0000","last_unstale":"2024-09-13T17:11:10.148016+0000","last_undegraded":"2024-09-13T17:11:10.148016+0000","last_fullsized":"2024-09-13T17:11:10.148016+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:39.983381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735168+0000","last_change":"2024-09-13T17:11:08.949307+0000","last_active":"2024-09-13T17:11:10.735168+0000","last_peered":"2024-09-13T17:11:10.735168+0000","last_clean":"2024-09-13T17:11:10.735168+0000","last_became_active":"2024-09-13T17:11:08.948037+0000","last_became_peered":"2024-09-13T17:11:08.948037+0000","last_unstale":"2024-09-13T17:11:10.735168+0000","last_undegraded":"2024-09-13T17:11:10.735168+0000","last_fullsized":"2024-09-13T17:11:10.735168+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:53.597936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935083+0000","last_change":"2024-09-13T17:11:08.947410+0000","last_active":"2024-09-13T17:11:09.935083+0000","last_peered":"2024-09-13T17:11:09.935083+0000","last_clean":"2024-09-13T17:11:09.935083+0000","last_became_active":"2024-09-13T17:11:08.942341+0000","last_became_peered":"2024-09-13T17:11:08.942341+0000","last_unstale":"2024-09-13T17:11:09.935083+0000","last_undegraded":"2024-09-13T17:11:09.935083+0000","last_fullsized":"2024-09-13T17:11:09.935083+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935138+0000","last_change":"2024-09-13T17:11:08.939334+0000","last_active":"2024-09-13T17:11:09.935138+0000","last_peered":"2024-09-13T17:11:09.935138+0000","last_clean":"2024-09-13T17:11:09.935138+0000","last_became_active":"2024-09-13T17:11:08.939007+0000","last_became_peered":"2024-09-13T17:11:08.939007+0000","last_unstale":"2024-09-13T17:11:09.935138+0000","last_undegraded":"2024-09-13T17:11:09.935138+0000","last_fullsized":"2024-09-13T17:11:09.935138+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:11.796310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147988+0000","last_change":"2024-09-13T17:11:08.954868+0000","last_active":"2024-09-13T17:11:10.147988+0000","last_peered":"2024-09-13T17:11:10.147988+0000","last_clean":"2024-09-13T17:11:10.147988+0000","last_became_active":"2024-09-13T17:11:08.953532+0000","last_became_peered":"2024-09-13T17:11:08.953532+0000","last_unstale":"2024-09-13T17:11:10.147988+0000","last_undegraded":"2024-09-13T17:11:10.147988+0000","last_fullsized":"2024-09-13T17:11:10.147988+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934546+0000","last_change":"2024-09-13T17:11:08.942858+0000","last_active":"2024-09-13T17:11:09.934546+0000","last_peered":"2024-09-13T17:11:09.934546+0000","last_clean":"2024-09-13T17:11:09.934546+0000","last_became_active":"2024-09-13T17:11:08.942395+0000","last_became_peered":"2024-09-13T17:11:08.942395+0000","last_unstale":"2024-09-13T17:11:09.934546+0000","last_undegraded":"2024-09-13T17:11:09.934546+0000","last_fullsized":"2024-09-13T17:11:09.934546+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:26.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130368+0000","last_change":"2024-09-13T17:11:08.934543+0000","last_active":"2024-09-13T17:11:11.130368+0000","last_peered":"2024-09-13T17:11:11.130368+0000","last_clean":"2024-09-13T17:11:11.130368+0000","last_became_active":"2024-09-13T17:11:08.932596+0000","last_became_peered":"2024-09-13T17:11:08.932596+0000","last_unstale":"2024-09-13T17:11:11.130368+0000","last_undegraded":"2024-09-13T17:11:11.130368+0000","last_fullsized":"2024-09-13T17:11:11.130368+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:17.898561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735311+0000","last_change":"2024-09-13T17:11:08.953304+0000","last_active":"2024-09-13T17:11:10.735311+0000","last_peered":"2024-09-13T17:11:10.735311+0000","last_clean":"2024-09-13T17:11:10.735311+0000","last_became_active":"2024-09-13T17:11:08.951792+0000","last_became_peered":"2024-09-13T17:11:08.951792+0000","last_unstale":"2024-09-13T17:11:10.735311+0000","last_undegraded":"2024-09-13T17:11:10.735311+0000","last_fullsized":"2024-09-13T17:11:10.735311+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:51.187481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933432+0000","last_change":"2024-09-13T17:11:08.949135+0000","last_active":"2024-09-13T17:11:09.933432+0000","last_peered":"2024-09-13T17:11:09.933432+0000","last_clean":"2024-09-13T17:11:09.933432+0000","last_became_active":"2024-09-13T17:11:08.948880+0000","last_became_peered":"2024-09-13T17:11:08.948880+0000","last_unstale":"2024-09-13T17:11:09.933432+0000","last_undegraded":"2024-09-13T17:11:09.933432+0000","last_fullsized":"2024-09-13T17:11:09.933432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:39.983381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960698+0000","last_change":"2024-09-13T17:11:08.508913+0000","last_active":"2024-09-13T17:11:08.960698+0000","last_peered":"2024-09-13T17:11:08.960698+0000","last_clean":"2024-09-13T17:11:08.960698+0000","last_became_active":"2024-09-13T17:11:08.508643+0000","last_became_peered":"2024-09-13T17:11:08.508643+0000","last_unstale":"2024-09-13T17:11:08.960698+0000","last_undegraded":"2024-09-13T17:11:08.960698+0000","last_fullsized":"2024-09-13T17:11:08.960698+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:52.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148080+0000","last_change":"2024-09-13T17:11:08.953202+0000","last_active":"2024-09-13T17:11:10.148080+0000","last_peered":"2024-09-13T17:11:10.148080+0000","last_clean":"2024-09-13T17:11:10.148080+0000","last_became_active":"2024-09-13T17:11:08.952916+0000","last_became_peered":"2024-09-13T17:11:08.952916+0000","last_unstale":"2024-09-13T17:11:10.148080+0000","last_undegraded":"2024-09-13T17:11:10.148080+0000","last_fullsized":"2024-09-13T17:11:10.148080+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:06.426163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.955348+0000","last_change":"2024-09-13T17:11:08.511916+0000","last_active":"2024-09-13T17:11:08.955348+0000","last_peered":"2024-09-13T17:11:08.955348+0000","last_clean":"2024-09-13T17:11:08.955348+0000","last_became_active":"2024-09-13T17:11:08.511641+0000","last_became_peered":"2024-09-13T17:11:08.511641+0000","last_unstale":"2024-09-13T17:11:08.955348+0000","last_undegraded":"2024-09-13T17:11:08.955348+0000","last_fullsized":"2024-09-13T17:11:08.955348+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935245+0000","last_change":"2024-09-13T17:11:08.944416+0000","last_active":"2024-09-13T17:11:09.935245+0000","last_peered":"2024-09-13T17:11:09.935245+0000","last_clean":"2024-09-13T17:11:09.935245+0000","last_became_active":"2024-09-13T17:11:08.943865+0000","last_became_peered":"2024-09-13T17:11:08.943865+0000","last_unstale":"2024-09-13T17:11:09.935245+0000","last_undegraded":"2024-09-13T17:11:09.935245+0000","last_fullsized":"2024-09-13T17:11:09.935245+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:54.026958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957342+0000","last_change":"2024-09-13T17:11:08.513124+0000","last_active":"2024-09-13T17:11:08.957342+0000","last_peered":"2024-09-13T17:11:08.957342+0000","last_clean":"2024-09-13T17:11:08.957342+0000","last_became_active":"2024-09-13T17:11:08.512830+0000","last_became_peered":"2024-09-13T17:11:08.512830+0000","last_unstale":"2024-09-13T17:11:08.957342+0000","last_undegraded":"2024-09-13T17:11:08.957342+0000","last_fullsized":"2024-09-13T17:11:08.957342+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:15.751862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933496+0000","last_change":"2024-09-13T17:11:08.947611+0000","last_active":"2024-09-13T17:11:09.933496+0000","last_peered":"2024-09-13T17:11:09.933496+0000","last_clean":"2024-09-13T17:11:09.933496+0000","last_became_active":"2024-09-13T17:11:08.946913+0000","last_became_peered":"2024-09-13T17:11:08.946913+0000","last_unstale":"2024-09-13T17:11:09.933496+0000","last_undegraded":"2024-09-13T17:11:09.933496+0000","last_fullsized":"2024-09-13T17:11:09.933496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:06.426163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957281+0000","last_change":"2024-09-13T17:11:08.514397+0000","last_active":"2024-09-13T17:11:08.957281+0000","last_peered":"2024-09-13T17:11:08.957281+0000","last_clean":"2024-09-13T17:11:08.957281+0000","last_became_active":"2024-09-13T17:11:08.514134+0000","last_became_peered":"2024-09-13T17:11:08.514134+0000","last_unstale":"2024-09-13T17:11:08.957281+0000","last_undegraded":"2024-09-13T17:11:08.957281+0000","last_fullsized":"2024-09-13T17:11:08.957281+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:42.476669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934596+0000","last_change":"2024-09-13T17:11:08.940677+0000","last_active":"2024-09-13T17:11:09.934596+0000","last_peered":"2024-09-13T17:11:09.934596+0000","last_clean":"2024-09-13T17:11:09.934596+0000","last_became_active":"2024-09-13T17:11:08.940390+0000","last_became_peered":"2024-09-13T17:11:08.940390+0000","last_unstale":"2024-09-13T17:11:09.934596+0000","last_undegraded":"2024-09-13T17:11:09.934596+0000","last_fullsized":"2024-09-13T17:11:09.934596+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:21.531576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957368+0000","last_change":"2024-09-13T17:11:08.509255+0000","last_active":"2024-09-13T17:11:08.957368+0000","last_peered":"2024-09-13T17:11:08.957368+0000","last_clean":"2024-09-13T17:11:08.957368+0000","last_became_active":"2024-09-13T17:11:08.508943+0000","last_became_peered":"2024-09-13T17:11:08.508943+0000","last_unstale":"2024-09-13T17:11:08.957368+0000","last_undegraded":"2024-09-13T17:11:08.957368+0000","last_fullsized":"2024-09-13T17:11:08.957368+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:53.020411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131112+0000","last_change":"2024-09-13T17:11:08.938576+0000","last_active":"2024-09-13T17:11:11.131112+0000","last_peered":"2024-09-13T17:11:11.131112+0000","last_clean":"2024-09-13T17:11:11.131112+0000","last_became_active":"2024-09-13T17:11:08.938308+0000","last_became_peered":"2024-09-13T17:11:08.938308+0000","last_unstale":"2024-09-13T17:11:11.131112+0000","last_undegraded":"2024-09-13T17:11:11.131112+0000","last_fullsized":"2024-09-13T17:11:11.131112+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:06.452065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.949613+0000","last_change":"2024-09-13T17:11:08.511045+0000","last_active":"2024-09-13T17:11:08.949613+0000","last_peered":"2024-09-13T17:11:08.949613+0000","last_clean":"2024-09-13T17:11:08.949613+0000","last_became_active":"2024-09-13T17:11:08.510781+0000","last_became_peered":"2024-09-13T17:11:08.510781+0000","last_unstale":"2024-09-13T17:11:08.949613+0000","last_undegraded":"2024-09-13T17:11:08.949613+0000","last_fullsized":"2024-09-13T17:11:08.949613+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:18:37.254451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131002+0000","last_change":"2024-09-13T17:11:08.946157+0000","last_active":"2024-09-13T17:11:11.131002+0000","last_peered":"2024-09-13T17:11:11.131002+0000","last_clean":"2024-09-13T17:11:11.131002+0000","last_became_active":"2024-09-13T17:11:08.945553+0000","last_became_peered":"2024-09-13T17:11:08.945553+0000","last_unstale":"2024-09-13T17:11:11.131002+0000","last_undegraded":"2024-09-13T17:11:11.131002+0000","last_fullsized":"2024-09-13T17:11:11.131002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:26.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927820+0000","last_change":"2024-09-13T17:11:08.504455+0000","last_active":"2024-09-13T17:11:08.927820+0000","last_peered":"2024-09-13T17:11:08.927820+0000","last_clean":"2024-09-13T17:11:08.927820+0000","last_became_active":"2024-09-13T17:11:08.504166+0000","last_became_peered":"2024-09-13T17:11:08.504166+0000","last_unstale":"2024-09-13T17:11:08.927820+0000","last_undegraded":"2024-09-13T17:11:08.927820+0000","last_fullsized":"2024-09-13T17:11:08.927820+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:22.582466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147645+0000","last_change":"2024-09-13T17:11:08.946784+0000","last_active":"2024-09-13T17:11:10.147645+0000","last_peered":"2024-09-13T17:11:10.147645+0000","last_clean":"2024-09-13T17:11:10.147645+0000","last_became_active":"2024-09-13T17:11:08.946299+0000","last_became_peered":"2024-09-13T17:11:08.946299+0000","last_unstale":"2024-09-13T17:11:10.147645+0000","last_undegraded":"2024-09-13T17:11:10.147645+0000","last_fullsized":"2024-09-13T17:11:10.147645+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:53.597936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.934280+0000","last_change":"2024-09-13T17:11:08.504442+0000","last_active":"2024-09-13T17:11:08.934280+0000","last_peered":"2024-09-13T17:11:08.934280+0000","last_clean":"2024-09-13T17:11:08.934280+0000","last_became_active":"2024-09-13T17:11:08.504168+0000","last_became_peered":"2024-09-13T17:11:08.504168+0000","last_unstale":"2024-09-13T17:11:08.934280+0000","last_undegraded":"2024-09-13T17:11:08.934280+0000","last_fullsized":"2024-09-13T17:11:08.934280+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:55.602732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935274+0000","last_change":"2024-09-13T17:11:08.956686+0000","last_active":"2024-09-13T17:11:09.935274+0000","last_peered":"2024-09-13T17:11:09.935274+0000","last_clean":"2024-09-13T17:11:09.935274+0000","last_became_active":"2024-09-13T17:11:08.956387+0000","last_became_peered":"2024-09-13T17:11:08.956387+0000","last_unstale":"2024-09-13T17:11:09.935274+0000","last_undegraded":"2024-09-13T17:11:09.935274+0000","last_fullsized":"2024-09-13T17:11:09.935274+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:43.483216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.942828+0000","last_change":"2024-09-13T17:11:08.511559+0000","last_active":"2024-09-13T17:11:08.942828+0000","last_peered":"2024-09-13T17:11:08.942828+0000","last_clean":"2024-09-13T17:11:08.942828+0000","last_became_active":"2024-09-13T17:11:08.511251+0000","last_became_peered":"2024-09-13T17:11:08.511251+0000","last_unstale":"2024-09-13T17:11:08.942828+0000","last_undegraded":"2024-09-13T17:11:08.942828+0000","last_fullsized":"2024-09-13T17:11:08.942828+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:55.414078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131268+0000","last_change":"2024-09-13T17:11:08.938024+0000","last_active":"2024-09-13T17:11:11.131268+0000","last_peered":"2024-09-13T17:11:11.131268+0000","last_clean":"2024-09-13T17:11:11.131268+0000","last_became_active":"2024-09-13T17:11:08.936316+0000","last_became_peered":"2024-09-13T17:11:08.936316+0000","last_unstale":"2024-09-13T17:11:11.131268+0000","last_undegraded":"2024-09-13T17:11:11.131268+0000","last_fullsized":"2024-09-13T17:11:11.131268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.203136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.936620+0000","last_change":"2024-09-13T17:11:08.505900+0000","last_active":"2024-09-13T17:11:08.936620+0000","last_peered":"2024-09-13T17:11:08.936620+0000","last_clean":"2024-09-13T17:11:08.936620+0000","last_became_active":"2024-09-13T17:11:08.505659+0000","last_became_peered":"2024-09-13T17:11:08.505659+0000","last_unstale":"2024-09-13T17:11:08.936620+0000","last_undegraded":"2024-09-13T17:11:08.936620+0000","last_fullsized":"2024-09-13T17:11:08.936620+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:28.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130788+0000","last_change":"2024-09-13T17:11:08.953384+0000","last_active":"2024-09-13T17:11:11.130788+0000","last_peered":"2024-09-13T17:11:11.130788+0000","last_clean":"2024-09-13T17:11:11.130788+0000","last_became_active":"2024-09-13T17:11:08.953110+0000","last_became_peered":"2024-09-13T17:11:08.953110+0000","last_unstale":"2024-09-13T17:11:11.130788+0000","last_undegraded":"2024-09-13T17:11:11.130788+0000","last_fullsized":"2024-09-13T17:11:11.130788+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:11.104415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.958240+0000","last_change":"2024-09-13T17:11:08.510876+0000","last_active":"2024-09-13T17:11:08.958240+0000","last_peered":"2024-09-13T17:11:08.958240+0000","last_clean":"2024-09-13T17:11:08.958240+0000","last_became_active":"2024-09-13T17:11:08.510236+0000","last_became_peered":"2024-09-13T17:11:08.510236+0000","last_unstale":"2024-09-13T17:11:08.958240+0000","last_undegraded":"2024-09-13T17:11:08.958240+0000","last_fullsized":"2024-09-13T17:11:08.958240+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:38.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131184+0000","last_change":"2024-09-13T17:11:08.940929+0000","last_active":"2024-09-13T17:11:11.131184+0000","last_peered":"2024-09-13T17:11:11.131184+0000","last_clean":"2024-09-13T17:11:11.131184+0000","last_became_active":"2024-09-13T17:11:08.940670+0000","last_became_peered":"2024-09-13T17:11:08.940670+0000","last_unstale":"2024-09-13T17:11:11.131184+0000","last_undegraded":"2024-09-13T17:11:11.131184+0000","last_fullsized":"2024-09-13T17:11:11.131184+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:47.991171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.961852+0000","last_change":"2024-09-13T17:11:08.509610+0000","last_active":"2024-09-13T17:11:08.961852+0000","last_peered":"2024-09-13T17:11:08.961852+0000","last_clean":"2024-09-13T17:11:08.961852+0000","last_became_active":"2024-09-13T17:11:08.509323+0000","last_became_peered":"2024-09-13T17:11:08.509323+0000","last_unstale":"2024-09-13T17:11:08.961852+0000","last_undegraded":"2024-09-13T17:11:08.961852+0000","last_fullsized":"2024-09-13T17:11:08.961852+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:55.047800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147908+0000","last_change":"2024-09-13T17:11:08.955884+0000","last_active":"2024-09-13T17:11:10.147908+0000","last_peered":"2024-09-13T17:11:10.147908+0000","last_clean":"2024-09-13T17:11:10.147908+0000","last_became_active":"2024-09-13T17:11:08.955558+0000","last_became_peered":"2024-09-13T17:11:08.955558+0000","last_unstale":"2024-09-13T17:11:10.147908+0000","last_undegraded":"2024-09-13T17:11:10.147908+0000","last_fullsized":"2024-09-13T17:11:10.147908+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:51.187481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960010+0000","last_change":"2024-09-13T17:11:08.510858+0000","last_active":"2024-09-13T17:11:08.960010+0000","last_peered":"2024-09-13T17:11:08.960010+0000","last_clean":"2024-09-13T17:11:08.960010+0000","last_became_active":"2024-09-13T17:11:08.509886+0000","last_became_peered":"2024-09-13T17:11:08.509886+0000","last_unstale":"2024-09-13T17:11:08.960010+0000","last_undegraded":"2024-09-13T17:11:08.960010+0000","last_fullsized":"2024-09-13T17:11:08.960010+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:05.445518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131371+0000","last_change":"2024-09-13T17:11:08.940863+0000","last_active":"2024-09-13T17:11:11.131371+0000","last_peered":"2024-09-13T17:11:11.131371+0000","last_clean":"2024-09-13T17:11:11.131371+0000","last_became_active":"2024-09-13T17:11:08.940591+0000","last_became_peered":"2024-09-13T17:11:08.940591+0000","last_unstale":"2024-09-13T17:11:11.131371+0000","last_undegraded":"2024-09-13T17:11:11.131371+0000","last_fullsized":"2024-09-13T17:11:11.131371+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.977161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928572+0000","last_change":"2024-09-13T17:11:08.505765+0000","last_active":"2024-09-13T17:11:08.928572+0000","last_peered":"2024-09-13T17:11:08.928572+0000","last_clean":"2024-09-13T17:11:08.928572+0000","last_became_active":"2024-09-13T17:11:08.505511+0000","last_became_peered":"2024-09-13T17:11:08.505511+0000","last_unstale":"2024-09-13T17:11:08.928572+0000","last_undegraded":"2024-09-13T17:11:08.928572+0000","last_fullsized":"2024-09-13T17:11:08.928572+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:51.970613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130702+0000","last_change":"2024-09-13T17:11:08.944939+0000","last_active":"2024-09-13T17:11:11.130702+0000","last_peered":"2024-09-13T17:11:11.130702+0000","last_clean":"2024-09-13T17:11:11.130702+0000","last_became_active":"2024-09-13T17:11:08.944655+0000","last_became_peered":"2024-09-13T17:11:08.944655+0000","last_unstale":"2024-09-13T17:11:11.130702+0000","last_undegraded":"2024-09-13T17:11:11.130702+0000","last_fullsized":"2024-09-13T17:11:11.130702+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:53.374179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.925788+0000","last_change":"2024-09-13T17:11:08.511713+0000","last_active":"2024-09-13T17:11:08.925788+0000","last_peered":"2024-09-13T17:11:08.925788+0000","last_clean":"2024-09-13T17:11:08.925788+0000","last_became_active":"2024-09-13T17:11:08.511362+0000","last_became_peered":"2024-09-13T17:11:08.511362+0000","last_unstale":"2024-09-13T17:11:08.925788+0000","last_undegraded":"2024-09-13T17:11:08.925788+0000","last_fullsized":"2024-09-13T17:11:08.925788+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:58.529901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131153+0000","last_change":"2024-09-13T17:11:08.939252+0000","last_active":"2024-09-13T17:11:11.131153+0000","last_peered":"2024-09-13T17:11:11.131153+0000","last_clean":"2024-09-13T17:11:11.131153+0000","last_became_active":"2024-09-13T17:11:08.938975+0000","last_became_peered":"2024-09-13T17:11:08.938975+0000","last_unstale":"2024-09-13T17:11:11.131153+0000","last_undegraded":"2024-09-13T17:11:11.131153+0000","last_fullsized":"2024-09-13T17:11:11.131153+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960876+0000","last_change":"2024-09-13T17:11:08.510567+0000","last_active":"2024-09-13T17:11:08.960876+0000","last_peered":"2024-09-13T17:11:08.960876+0000","last_clean":"2024-09-13T17:11:08.960876+0000","last_became_active":"2024-09-13T17:11:08.509318+0000","last_became_peered":"2024-09-13T17:11:08.509318+0000","last_unstale":"2024-09-13T17:11:08.960876+0000","last_undegraded":"2024-09-13T17:11:08.960876+0000","last_fullsized":"2024-09-13T17:11:08.960876+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:43.667809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735291+0000","last_change":"2024-09-13T17:11:08.936837+0000","last_active":"2024-09-13T17:11:10.735291+0000","last_peered":"2024-09-13T17:11:10.735291+0000","last_clean":"2024-09-13T17:11:10.735291+0000","last_became_active":"2024-09-13T17:11:08.935945+0000","last_became_peered":"2024-09-13T17:11:08.935945+0000","last_unstale":"2024-09-13T17:11:10.735291+0000","last_undegraded":"2024-09-13T17:11:10.735291+0000","last_fullsized":"2024-09-13T17:11:10.735291+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.955990+0000","last_change":"2024-09-13T17:11:08.500720+0000","last_active":"2024-09-13T17:11:08.955990+0000","last_peered":"2024-09-13T17:11:08.955990+0000","last_clean":"2024-09-13T17:11:08.955990+0000","last_became_active":"2024-09-13T17:11:08.500406+0000","last_became_peered":"2024-09-13T17:11:08.500406+0000","last_unstale":"2024-09-13T17:11:08.955990+0000","last_undegraded":"2024-09-13T17:11:08.955990+0000","last_fullsized":"2024-09-13T17:11:08.955990+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:28.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131833+0000","last_change":"2024-09-13T17:11:08.942728+0000","last_active":"2024-09-13T17:11:11.131833+0000","last_peered":"2024-09-13T17:11:11.131833+0000","last_clean":"2024-09-13T17:11:11.131833+0000","last_became_active":"2024-09-13T17:11:08.942434+0000","last_became_peered":"2024-09-13T17:11:08.942434+0000","last_unstale":"2024-09-13T17:11:11.131833+0000","last_undegraded":"2024-09-13T17:11:11.131833+0000","last_fullsized":"2024-09-13T17:11:11.131833+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:36.572460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.953820+0000","last_change":"2024-09-13T17:11:08.506379+0000","last_active":"2024-09-13T17:11:08.953820+0000","last_peered":"2024-09-13T17:11:08.953820+0000","last_clean":"2024-09-13T17:11:08.953820+0000","last_became_active":"2024-09-13T17:11:08.506074+0000","last_became_peered":"2024-09-13T17:11:08.506074+0000","last_unstale":"2024-09-13T17:11:08.953820+0000","last_undegraded":"2024-09-13T17:11:08.953820+0000","last_fullsized":"2024-09-13T17:11:08.953820+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:15.751862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147919+0000","last_change":"2024-09-13T17:11:08.952135+0000","last_active":"2024-09-13T17:11:10.147919+0000","last_peered":"2024-09-13T17:11:10.147919+0000","last_clean":"2024-09-13T17:11:10.147919+0000","last_became_active":"2024-09-13T17:11:08.951223+0000","last_became_peered":"2024-09-13T17:11:08.951223+0000","last_unstale":"2024-09-13T17:11:10.147919+0000","last_undegraded":"2024-09-13T17:11:10.147919+0000","last_fullsized":"2024-09-13T17:11:10.147919+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.589014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.962381+0000","last_change":"2024-09-13T17:11:08.504709+0000","last_active":"2024-09-13T17:11:08.962381+0000","last_peered":"2024-09-13T17:11:08.962381+0000","last_clean":"2024-09-13T17:11:08.962381+0000","last_became_active":"2024-09-13T17:11:08.504412+0000","last_became_peered":"2024-09-13T17:11:08.504412+0000","last_unstale":"2024-09-13T17:11:08.962381+0000","last_undegraded":"2024-09-13T17:11:08.962381+0000","last_fullsized":"2024-09-13T17:11:08.962381+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:14.450139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147901+0000","last_change":"2024-09-13T17:11:08.953098+0000","last_active":"2024-09-13T17:11:10.147901+0000","last_peered":"2024-09-13T17:11:10.147901+0000","last_clean":"2024-09-13T17:11:10.147901+0000","last_became_active":"2024-09-13T17:11:08.952816+0000","last_became_peered":"2024-09-13T17:11:08.952816+0000","last_unstale":"2024-09-13T17:11:10.147901+0000","last_undegraded":"2024-09-13T17:11:10.147901+0000","last_fullsized":"2024-09-13T17:11:10.147901+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:38.612605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928253+0000","last_change":"2024-09-13T17:11:08.507342+0000","last_active":"2024-09-13T17:11:08.928253+0000","last_peered":"2024-09-13T17:11:08.928253+0000","last_clean":"2024-09-13T17:11:08.928253+0000","last_became_active":"2024-09-13T17:11:08.506077+0000","last_became_peered":"2024-09-13T17:11:08.506077+0000","last_unstale":"2024-09-13T17:11:08.928253+0000","last_undegraded":"2024-09-13T17:11:08.928253+0000","last_fullsized":"2024-09-13T17:11:08.928253+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:38.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130520+0000","last_change":"2024-09-13T17:11:08.955283+0000","last_active":"2024-09-13T17:11:11.130520+0000","last_peered":"2024-09-13T17:11:11.130520+0000","last_clean":"2024-09-13T17:11:11.130520+0000","last_became_active":"2024-09-13T17:11:08.954324+0000","last_became_peered":"2024-09-13T17:11:08.954324+0000","last_unstale":"2024-09-13T17:11:11.130520+0000","last_undegraded":"2024-09-13T17:11:11.130520+0000","last_fullsized":"2024-09-13T17:11:11.130520+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:38.172171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957915+0000","last_change":"2024-09-13T17:11:08.498279+0000","last_active":"2024-09-13T17:11:08.957915+0000","last_peered":"2024-09-13T17:11:08.957915+0000","last_clean":"2024-09-13T17:11:08.957915+0000","last_became_active":"2024-09-13T17:11:08.497833+0000","last_became_peered":"2024-09-13T17:11:08.497833+0000","last_unstale":"2024-09-13T17:11:08.957915+0000","last_undegraded":"2024-09-13T17:11:08.957915+0000","last_fullsized":"2024-09-13T17:11:08.957915+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:25.567018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933122+0000","last_change":"2024-09-13T17:11:08.935400+0000","last_active":"2024-09-13T17:11:09.933122+0000","last_peered":"2024-09-13T17:11:09.933122+0000","last_clean":"2024-09-13T17:11:09.933122+0000","last_became_active":"2024-09-13T17:11:08.935093+0000","last_became_peered":"2024-09-13T17:11:08.935093+0000","last_unstale":"2024-09-13T17:11:09.933122+0000","last_undegraded":"2024-09-13T17:11:09.933122+0000","last_fullsized":"2024-09-13T17:11:09.933122+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.589014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927574+0000","last_change":"2024-09-13T17:11:08.510342+0000","last_active":"2024-09-13T17:11:08.927574+0000","last_peered":"2024-09-13T17:11:08.927574+0000","last_clean":"2024-09-13T17:11:08.927574+0000","last_became_active":"2024-09-13T17:11:08.510079+0000","last_became_peered":"2024-09-13T17:11:08.510079+0000","last_unstale":"2024-09-13T17:11:08.927574+0000","last_undegraded":"2024-09-13T17:11:08.927574+0000","last_fullsized":"2024-09-13T17:11:08.927574+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:37.606057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735321+0000","last_change":"2024-09-13T17:11:08.944120+0000","last_active":"2024-09-13T17:11:10.735321+0000","last_peered":"2024-09-13T17:11:10.735321+0000","last_clean":"2024-09-13T17:11:10.735321+0000","last_became_active":"2024-09-13T17:11:08.943870+0000","last_became_peered":"2024-09-13T17:11:08.943870+0000","last_unstale":"2024-09-13T17:11:10.735321+0000","last_undegraded":"2024-09-13T17:11:10.735321+0000","last_fullsized":"2024-09-13T17:11:10.735321+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.932354+0000","last_change":"2024-09-13T17:11:08.505319+0000","last_active":"2024-09-13T17:11:08.932354+0000","last_peered":"2024-09-13T17:11:08.932354+0000","last_clean":"2024-09-13T17:11:08.932354+0000","last_became_active":"2024-09-13T17:11:08.505029+0000","last_became_peered":"2024-09-13T17:11:08.505029+0000","last_unstale":"2024-09-13T17:11:08.932354+0000","last_undegraded":"2024-09-13T17:11:08.932354+0000","last_fullsized":"2024-09-13T17:11:08.932354+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:19.732809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131317+0000","last_change":"2024-09-13T17:11:08.956632+0000","last_active":"2024-09-13T17:11:11.131317+0000","last_peered":"2024-09-13T17:11:11.131317+0000","last_clean":"2024-09-13T17:11:11.131317+0000","last_became_active":"2024-09-13T17:11:08.956385+0000","last_became_peered":"2024-09-13T17:11:08.956385+0000","last_unstale":"2024-09-13T17:11:11.131317+0000","last_undegraded":"2024-09-13T17:11:11.131317+0000","last_fullsized":"2024-09-13T17:11:11.131317+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:54.026958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.926953+0000","last_change":"2024-09-13T17:11:08.505825+0000","last_active":"2024-09-13T17:11:08.926953+0000","last_peered":"2024-09-13T17:11:08.926953+0000","last_clean":"2024-09-13T17:11:08.926953+0000","last_became_active":"2024-09-13T17:11:08.505500+0000","last_became_peered":"2024-09-13T17:11:08.505500+0000","last_unstale":"2024-09-13T17:11:08.926953+0000","last_undegraded":"2024-09-13T17:11:08.926953+0000","last_fullsized":"2024-09-13T17:11:08.926953+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:20.525029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148124+0000","last_change":"2024-09-13T17:11:08.962536+0000","last_active":"2024-09-13T17:11:10.148124+0000","last_peered":"2024-09-13T17:11:10.148124+0000","last_clean":"2024-09-13T17:11:10.148124+0000","last_became_active":"2024-09-13T17:11:08.962247+0000","last_became_peered":"2024-09-13T17:11:08.962247+0000","last_unstale":"2024-09-13T17:11:10.148124+0000","last_undegraded":"2024-09-13T17:11:10.148124+0000","last_fullsized":"2024-09-13T17:11:10.148124+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927518+0000","last_change":"2024-09-13T17:11:08.514288+0000","last_active":"2024-09-13T17:11:08.927518+0000","last_peered":"2024-09-13T17:11:08.927518+0000","last_clean":"2024-09-13T17:11:08.927518+0000","last_became_active":"2024-09-13T17:11:08.513978+0000","last_became_peered":"2024-09-13T17:11:08.513978+0000","last_unstale":"2024-09-13T17:11:08.927518+0000","last_undegraded":"2024-09-13T17:11:08.927518+0000","last_fullsized":"2024-09-13T17:11:08.927518+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:35.565913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131565+0000","last_change":"2024-09-13T17:11:08.950475+0000","last_active":"2024-09-13T17:11:11.131565+0000","last_peered":"2024-09-13T17:11:11.131565+0000","last_clean":"2024-09-13T17:11:11.131565+0000","last_became_active":"2024-09-13T17:11:08.949141+0000","last_became_peered":"2024-09-13T17:11:08.949141+0000","last_unstale":"2024-09-13T17:11:11.131565+0000","last_undegraded":"2024-09-13T17:11:11.131565+0000","last_fullsized":"2024-09-13T17:11:11.131565+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.930241+0000","last_change":"2024-09-13T17:11:08.514093+0000","last_active":"2024-09-13T17:11:08.930241+0000","last_peered":"2024-09-13T17:11:08.930241+0000","last_clean":"2024-09-13T17:11:08.930241+0000","last_became_active":"2024-09-13T17:11:08.513508+0000","last_became_peered":"2024-09-13T17:11:08.513508+0000","last_unstale":"2024-09-13T17:11:08.930241+0000","last_undegraded":"2024-09-13T17:11:08.930241+0000","last_fullsized":"2024-09-13T17:11:08.930241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:59.723386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131083+0000","last_change":"2024-09-13T17:11:08.947653+0000","last_active":"2024-09-13T17:11:11.131083+0000","last_peered":"2024-09-13T17:11:11.131083+0000","last_clean":"2024-09-13T17:11:11.131083+0000","last_became_active":"2024-09-13T17:11:08.947330+0000","last_became_peered":"2024-09-13T17:11:08.947330+0000","last_unstale":"2024-09-13T17:11:11.131083+0000","last_undegraded":"2024-09-13T17:11:11.131083+0000","last_fullsized":"2024-09-13T17:11:11.131083+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.932196+0000","last_change":"2024-09-13T17:11:08.512643+0000","last_active":"2024-09-13T17:11:08.932196+0000","last_peered":"2024-09-13T17:11:08.932196+0000","last_clean":"2024-09-13T17:11:08.932196+0000","last_became_active":"2024-09-13T17:11:08.512339+0000","last_became_peered":"2024-09-13T17:11:08.512339+0000","last_unstale":"2024-09-13T17:11:08.932196+0000","last_undegraded":"2024-09-13T17:11:08.932196+0000","last_fullsized":"2024-09-13T17:11:08.932196+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:04.874522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130618+0000","last_change":"2024-09-13T17:11:08.947717+0000","last_active":"2024-09-13T17:11:11.130618+0000","last_peered":"2024-09-13T17:11:11.130618+0000","last_clean":"2024-09-13T17:11:11.130618+0000","last_became_active":"2024-09-13T17:11:08.947413+0000","last_became_peered":"2024-09-13T17:11:08.947413+0000","last_unstale":"2024-09-13T17:11:11.130618+0000","last_undegraded":"2024-09-13T17:11:11.130618+0000","last_fullsized":"2024-09-13T17:11:11.130618+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:54.824131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928857+0000","last_change":"2024-09-13T17:11:08.509387+0000","last_active":"2024-09-13T17:11:08.928857+0000","last_peered":"2024-09-13T17:11:08.928857+0000","last_clean":"2024-09-13T17:11:08.928857+0000","last_became_active":"2024-09-13T17:11:08.509096+0000","last_became_peered":"2024-09-13T17:11:08.509096+0000","last_unstale":"2024-09-13T17:11:08.928857+0000","last_undegraded":"2024-09-13T17:11:08.928857+0000","last_fullsized":"2024-09-13T17:11:08.928857+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:35.565913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130437+0000","last_change":"2024-09-13T17:11:08.954550+0000","last_active":"2024-09-13T17:11:11.130437+0000","last_peered":"2024-09-13T17:11:11.130437+0000","last_clean":"2024-09-13T17:11:11.130437+0000","last_became_active":"2024-09-13T17:11:08.953284+0000","last_became_peered":"2024-09-13T17:11:08.953284+0000","last_unstale":"2024-09-13T17:11:11.130437+0000","last_undegraded":"2024-09-13T17:11:11.130437+0000","last_fullsized":"2024-09-13T17:11:11.130437+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.933426+0000","last_change":"2024-09-13T17:11:08.510581+0000","last_active":"2024-09-13T17:11:08.933426+0000","last_peered":"2024-09-13T17:11:08.933426+0000","last_clean":"2024-09-13T17:11:08.933426+0000","last_became_active":"2024-09-13T17:11:08.509338+0000","last_became_peered":"2024-09-13T17:11:08.509338+0000","last_unstale":"2024-09-13T17:11:08.933426+0000","last_undegraded":"2024-09-13T17:11:08.933426+0000","last_fullsized":"2024-09-13T17:11:08.933426+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:04.874522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934229+0000","last_change":"2024-09-13T17:11:08.948768+0000","last_active":"2024-09-13T17:11:09.934229+0000","last_peered":"2024-09-13T17:11:09.934229+0000","last_clean":"2024-09-13T17:11:09.934229+0000","last_became_active":"2024-09-13T17:11:08.948467+0000","last_became_peered":"2024-09-13T17:11:08.948467+0000","last_unstale":"2024-09-13T17:11:09.934229+0000","last_undegraded":"2024-09-13T17:11:09.934229+0000","last_fullsized":"2024-09-13T17:11:09.934229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.930666+0000","last_change":"2024-09-13T17:11:08.505124+0000","last_active":"2024-09-13T17:11:08.930666+0000","last_peered":"2024-09-13T17:11:08.930666+0000","last_clean":"2024-09-13T17:11:08.930666+0000","last_became_active":"2024-09-13T17:11:08.504849+0000","last_became_peered":"2024-09-13T17:11:08.504849+0000","last_unstale":"2024-09-13T17:11:08.930666+0000","last_undegraded":"2024-09-13T17:11:08.930666+0000","last_fullsized":"2024-09-13T17:11:08.930666+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934260+0000","last_change":"2024-09-13T17:11:08.941857+0000","last_active":"2024-09-13T17:11:09.934260+0000","last_peered":"2024-09-13T17:11:09.934260+0000","last_clean":"2024-09-13T17:11:09.934260+0000","last_became_active":"2024-09-13T17:11:08.941341+0000","last_became_peered":"2024-09-13T17:11:08.941341+0000","last_unstale":"2024-09-13T17:11:09.934260+0000","last_undegraded":"2024-09-13T17:11:09.934260+0000","last_fullsized":"2024-09-13T17:11:09.934260+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:36.572460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.935655+0000","last_change":"2024-09-13T17:11:08.507664+0000","last_active":"2024-09-13T17:11:08.935655+0000","last_peered":"2024-09-13T17:11:08.935655+0000","last_clean":"2024-09-13T17:11:08.935655+0000","last_became_active":"2024-09-13T17:11:08.507326+0000","last_became_peered":"2024-09-13T17:11:08.507326+0000","last_unstale":"2024-09-13T17:11:08.935655+0000","last_undegraded":"2024-09-13T17:11:08.935655+0000","last_fullsized":"2024-09-13T17:11:08.935655+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:18.202120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131688+0000","last_change":"2024-09-13T17:11:08.947814+0000","last_active":"2024-09-13T17:11:11.131688+0000","last_peered":"2024-09-13T17:11:11.131688+0000","last_clean":"2024-09-13T17:11:11.131688+0000","last_became_active":"2024-09-13T17:11:08.945042+0000","last_became_peered":"2024-09-13T17:11:08.945042+0000","last_unstale":"2024-09-13T17:11:11.131688+0000","last_undegraded":"2024-09-13T17:11:11.131688+0000","last_fullsized":"2024-09-13T17:11:11.131688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.958962+0000","last_change":"2024-09-13T17:11:08.505921+0000","last_active":"2024-09-13T17:11:08.958962+0000","last_peered":"2024-09-13T17:11:08.958962+0000","last_clean":"2024-09-13T17:11:08.958962+0000","last_became_active":"2024-09-13T17:11:08.505613+0000","last_became_peered":"2024-09-13T17:11:08.505613+0000","last_unstale":"2024-09-13T17:11:08.958962+0000","last_undegraded":"2024-09-13T17:11:08.958962+0000","last_fullsized":"2024-09-13T17:11:08.958962+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:05.419615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":187,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928303+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:11:08.928303+0000","last_peered":"2024-09-13T17:11:08.928303+0000","last_clean":"2024-09-13T17:11:08.928303+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:11:08.928303+0000","last_undegraded":"2024-09-13T17:11:08.928303+0000","last_fullsized":"2024-09-13T17:11:08.928303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40444,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331396,"statfs":{"total":96636764160,"available":96595349504,"internally_reserved":0,"allocated":1163264,"data_stored":823071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40237515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.495,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.67900000000000005,"15min":0.68000000000000005},"min":{"1min":0.51200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.056,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.64000000000000001,"15min":0.63300000000000001},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.61099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.96099999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.53900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.60499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.59599999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.73799999999999999,"15min":0.73199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.78900000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.48299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.45000000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.198,"5min":1.337,"15min":1.337},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.54300000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.54400000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35772,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336068,"statfs":{"total":96636764160,"available":96600133632,"internally_reserved":0,"allocated":573440,"data_stored":233285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88400000000000001,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.47099999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.67900000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65900000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0129999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82699999999999996,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84199999999999997,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95499999999999996,"5min":3.742,"15min":3.742},"last":0.60599999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0349999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.72499999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.71999999999999997,"15min":0.73199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93600000000000005,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.72799999999999998,"15min":0.73299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.0269999999999999,"5min":3.609,"15min":3.609},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.069,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.81100000000000005,"15min":0.80900000000000005},"min":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0069999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.103}]}]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35776,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336064,"statfs":{"total":96636764160,"available":96600129536,"internally_reserved":0,"allocated":569344,"data_stored":232683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.64000000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.004,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.82199999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.84399999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74199999999999999,"15min":0.75700000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.76700000000000002},"min":{"1min":0.56399999999999995,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93100000000000005,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.78000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.622,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.52600000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.749,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.61899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39872,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331968,"statfs":{"total":96636764160,"available":96595935232,"internally_reserved":0,"allocated":577536,"data_stored":234177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64000000000000001,"15min":0.64100000000000001},"min":{"1min":0.46000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94299999999999995,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.746},"min":{"1min":0.48999999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.94199999999999995,"5min":1.341,"15min":1.341},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.51100000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85599999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.64500000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.43099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90100000000000002,"5min":1.377,"15min":1.377},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.47099999999999997,"5min":0.434,"15min":0.434},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.65600000000000003},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.874,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.82699999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64100000000000001,"15min":0.65100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.82799999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72499999999999998,"15min":0.75},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":1.413,"15min":1.413},"last":0.84399999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94199999999999995,"5min":0.998,"15min":0.998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.54600000000000004,"5min":0.371,"15min":0.371},"max":{"1min":0.95899999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.86599999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45636,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326204,"statfs":{"total":96636764160,"available":96590032896,"internally_reserved":0,"allocated":577536,"data_stored":233251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.97099999999999997,"5min":2.96,"15min":2.96},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2809999999999999,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.73099999999999998,"15min":0.75700000000000001},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1830000000000001,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.376}]},{"osd":2,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.40600000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80600000000000005,"15min":0.83099999999999996},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3560000000000001,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.56399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83899999999999997,"15min":0.85499999999999998},"min":{"1min":0.61099999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.252,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78300000000000003,"15min":0.76800000000000002},"min":{"1min":0.623,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.244,"5min":1.401,"15min":1.401},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80500000000000005,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.75900000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.86599999999999999,"15min":0.88100000000000001},"min":{"1min":0.65400000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.4139999999999999,"5min":2.899,"15min":2.899},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.278,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.77300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35776,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336064,"statfs":{"total":96636764160,"available":96600129536,"internally_reserved":0,"allocated":577536,"data_stored":233180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.438,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":13.476000000000001,"5min":3.077,"15min":1.3420000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":260.53399999999999,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":13.653,"5min":3.117,"15min":1.355},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":262.61399999999998,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.48899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.05,"5min":1.121,"15min":1.121},"last":0.49099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.77200000000000002,"15min":0.78400000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.44800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77800000000000002},"min":{"1min":0.54700000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":0.93300000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79900000000000004,"15min":0.81100000000000005},"min":{"1min":0.57099999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91500000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77900000000000003,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.80300000000000005},"min":{"1min":0.52600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.77200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45624,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326216,"statfs":{"total":96636764160,"available":96590045184,"internally_reserved":0,"allocated":569344,"data_stored":232683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.1779999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.437},{"interface":"front","average":{"1min":0.623,"5min":0.51500000000000001,"15min":0.497},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.48599999999999999,"15min":0.46800000000000003},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56200000000000006,"15min":0.57199999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.2350000000000001,"5min":1.49,"15min":1.49},"last":0.497}]},{"osd":3,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.65800000000000003},"min":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.42499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54200000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.83199999999999996,"15min":0.82399999999999995},"min":{"1min":0.61599999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.78200000000000003,"15min":0.77500000000000002},"min":{"1min":0.55300000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.754},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80300000000000005,"15min":0.79600000000000004},"min":{"1min":0.59999999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.82499999999999996,"15min":0.81999999999999995},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4710000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.59099999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.63}]},{"osd":7,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.83499999999999996,"15min":0.83999999999999997},"min":{"1min":0.56399999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.84299999999999997,"15min":0.83099999999999996},"min":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5049999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.67100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40440,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331400,"statfs":{"total":96636764160,"available":96595353600,"internally_reserved":0,"allocated":1159168,"data_stored":821996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.65600000000000003,"15min":0.749},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.64700000000000002,"15min":0.72899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.68400000000000005,"15min":0.76600000000000001},"min":{"1min":0.376,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64200000000000002,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.70199999999999996,"15min":0.78400000000000003},"min":{"1min":0.38500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.52100000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64800000000000002,"15min":0.71399999999999997},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.59399999999999997,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70099999999999996,"15min":0.76300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80500000000000005,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.499}]},{"osd":4,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1779999999999999,"5min":4.734,"15min":4.734},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.89200000000000002,"15min":0.95799999999999996},"min":{"1min":0.59199999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.92300000000000004,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.90700000000000003,"15min":0.97899999999999998},"min":{"1min":0.55000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98399999999999999,"5min":4.944,"15min":4.944},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.89400000000000002,"15min":0.95399999999999996},"min":{"1min":0.627,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.117,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.034,"15min":1.155},"min":{"1min":0.64200000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.242,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.92400000000000004,"15min":0.98499999999999999},"min":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.95299999999999996,"15min":1.0249999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93799999999999994,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.93200000000000005,"15min":1.002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.03,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:11:13.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:11:13.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.311+0000 7fcf5a200640 1 Processor -- start 2024-09-13T17:11:13.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.311+0000 7fcf5a200640 1 -- start start 2024-09-13T17:11:13.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.312+0000 7fcf5a200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf5415a540 0x7fcf5406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf5a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5415e100 con 0x7fcf540cabc0 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf5a200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5415e3e0 con 0x7fcf54070cc0 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf5a200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540adb00 con 0x7fcf5415a540 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf4aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcf540cabc0 0x7fcf5415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58034/0 (socket says 172.21.15.28:58034) 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf4aa00640 1 -- 172.21.15.28:0/3068811655 learned_addr learned my addr 172.21.15.28:0/3068811655 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf53400640 1 --2- 172.21.15.28:0/3068811655 >> v2:172.21.15.189:3300/0 conn(0x7fcf5415a540 0x7fcf5406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf52a00640 1 -- 172.21.15.28:0/3068811655 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631170493 0 0) 0x7fcf5415e100 con 0x7fcf540cabc0 2024-09-13T17:11:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf52a00640 1 -- 172.21.15.28:0/3068811655 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf38003520 con 0x7fcf540cabc0 2024-09-13T17:11:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf53400640 1 -- 172.21.15.28:0/3068811655 >> v1:172.21.15.189:6789/0 conn(0x7fcf54070cc0 legacy=0x7fcf5406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf53400640 1 -- 172.21.15.28:0/3068811655 >> v1:172.21.15.28:6789/0 conn(0x7fcf540cabc0 legacy=0x7fcf5415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.313+0000 7fcf53400640 1 -- 172.21.15.28:0/3068811655 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf5415ef80 con 0x7fcf5415a540 2024-09-13T17:11:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf53400640 1 --2- 172.21.15.28:0/3068811655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5406c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097a0 tx=0x7fcf4402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80ef587dbfa5b4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf52a00640 1 -- 172.21.15.28:0/3068811655 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44032440 con 0x7fcf5415a540 2024-09-13T17:11:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf52a00640 1 -- 172.21.15.28:0/3068811655 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf440325a0 con 0x7fcf5415a540 2024-09-13T17:11:13.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf52a00640 1 -- 172.21.15.28:0/3068811655 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf44032910 con 0x7fcf5415a540 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf5a200640 1 -- 172.21.15.28:0/3068811655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 msgr2=0x7fcf5406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.315+0000 7fcf5a200640 1 --2- 172.21.15.28:0/3068811655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5406c430 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcf440097a0 tx=0x7fcf4402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.316+0000 7fcf5a200640 1 -- 172.21.15.28:0/3068811655 shutdown_connections 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.316+0000 7fcf5a200640 1 --2- 172.21.15.28:0/3068811655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5406c430 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.316+0000 7fcf5a200640 1 -- 172.21.15.28:0/3068811655 >> 172.21.15.28:0/3068811655 conn(0x7fcf54069e80 msgr2=0x7fcf5406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.316+0000 7fcf5a200640 1 -- 172.21.15.28:0/3068811655 shutdown_connections 2024-09-13T17:11:13.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.316+0000 7fcf5a200640 1 -- 172.21.15.28:0/3068811655 wait complete. 2024-09-13T17:11:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.317+0000 7fcf5a200640 1 Processor -- start 2024-09-13T17:11:13.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.317+0000 7fcf5a200640 1 -- start start 2024-09-13T17:11:13.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf54070cc0 0x7fcf54153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5414ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5414aa40 con 0x7fcf540cabc0 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5414dc00 con 0x7fcf5415a540 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.318+0000 7fcf5a200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf5414d920 con 0x7fcf54070cc0 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50914/0 (socket says 172.21.15.28:50914) 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 -- 172.21.15.28:0/1714800525 learned_addr learned my addr 172.21.15.28:0/1714800525 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:13.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf53e00640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5414ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 -- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf54070cc0 msgr2=0x7fcf54153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf54070cc0 0x7fcf54153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 -- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 msgr2=0x7fcf5414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5414ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 -- 172.21.15.28:0/1714800525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf44009400 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.319+0000 7fcf4aa00640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00c500 tx=0x7fcf3c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.320+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c005ce0 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.320+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c005e40 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.320+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c007330 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.320+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf54153370 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.320+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf54153930 con 0x7fcf540cabc0 2024-09-13T17:11:13.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.321+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf541531d0 con 0x7fcf540cabc0 2024-09-13T17:11:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.322+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf3c007550 con 0x7fcf540cabc0 2024-09-13T17:11:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.322+0000 7fcf4be00640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 0x7fcf3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.323+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf3c0889c0 con 0x7fcf540cabc0 2024-09-13T17:11:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.323+0000 7fcf53400640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 0x7fcf3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.324+0000 7fcf53400640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 0x7fcf3006d0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf4402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.327+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcf541531d0 con 0x7fcf540cabc0 2024-09-13T17:11:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.500+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcf5415a920 con 0x7fcf3006ac00 2024-09-13T17:11:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.508+0000 7fcf4be00640 1 -- 172.21.15.28:0/1714800525 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289531 (secure 0 0 0) 0x7fcf440337a0 con 0x7fcf3006ac00 2024-09-13T17:11:13.514 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:13.515 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 msgr2=0x7fcf3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 0x7fcf3006d0c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fcf44034000 tx=0x7fcf4402c000 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 msgr2=0x7fcf5414a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00c500 tx=0x7fcf3c00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 shutdown_connections 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf3006ac00 0x7fcf3006d0c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf5415a540 0x7fcf5414ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf540cabc0 0x7fcf5414a500 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 --2- 172.21.15.28:0/1714800525 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf54070cc0 0x7fcf54153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 >> 172.21.15.28:0/1714800525 conn(0x7fcf54069e80 msgr2=0x7fcf5406feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:13.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.512+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 shutdown_connections 2024-09-13T17:11:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.513+0000 7fcf5a200640 1 -- 172.21.15.28:0/1714800525 wait complete. 2024-09-13T17:11:13.531 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-13T17:11:12.928972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":319340,"kb_used_data":5632,"kb_used_omap":12,"kb_used_meta":313587,"kb_avail":754655380,"statfs":{"total":773094113280,"available":772767109120,"internally_reserved":0,"allocated":5767168,"data_stored":3044326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":321113657},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.447975"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131268+0000","last_change":"2024-09-13T17:11:08.947894+0000","last_active":"2024-09-13T17:11:11.131268+0000","last_peered":"2024-09-13T17:11:11.131268+0000","last_clean":"2024-09-13T17:11:11.131268+0000","last_became_active":"2024-09-13T17:11:08.947556+0000","last_became_peered":"2024-09-13T17:11:08.947556+0000","last_unstale":"2024-09-13T17:11:11.131268+0000","last_undegraded":"2024-09-13T17:11:11.131268+0000","last_fullsized":"2024-09-13T17:11:11.131268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:17.772673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935086+0000","last_change":"2024-09-13T17:11:08.945962+0000","last_active":"2024-09-13T17:11:09.935086+0000","last_peered":"2024-09-13T17:11:09.935086+0000","last_clean":"2024-09-13T17:11:09.935086+0000","last_became_active":"2024-09-13T17:11:08.945672+0000","last_became_peered":"2024-09-13T17:11:08.945672+0000","last_unstale":"2024-09-13T17:11:09.935086+0000","last_undegraded":"2024-09-13T17:11:09.935086+0000","last_fullsized":"2024-09-13T17:11:09.935086+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935077+0000","last_change":"2024-09-13T17:11:08.948950+0000","last_active":"2024-09-13T17:11:09.935077+0000","last_peered":"2024-09-13T17:11:09.935077+0000","last_clean":"2024-09-13T17:11:09.935077+0000","last_became_active":"2024-09-13T17:11:08.948692+0000","last_became_peered":"2024-09-13T17:11:08.948692+0000","last_unstale":"2024-09-13T17:11:09.935077+0000","last_undegraded":"2024-09-13T17:11:09.935077+0000","last_fullsized":"2024-09-13T17:11:09.935077+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:00.729934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131456+0000","last_change":"2024-09-13T17:11:08.946655+0000","last_active":"2024-09-13T17:11:11.131456+0000","last_peered":"2024-09-13T17:11:11.131456+0000","last_clean":"2024-09-13T17:11:11.131456+0000","last_became_active":"2024-09-13T17:11:08.946418+0000","last_became_peered":"2024-09-13T17:11:08.946418+0000","last_unstale":"2024-09-13T17:11:11.131456+0000","last_undegraded":"2024-09-13T17:11:11.131456+0000","last_fullsized":"2024-09-13T17:11:11.131456+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:58.261986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934865+0000","last_change":"2024-09-13T17:11:08.942166+0000","last_active":"2024-09-13T17:11:09.934865+0000","last_peered":"2024-09-13T17:11:09.934865+0000","last_clean":"2024-09-13T17:11:09.934865+0000","last_became_active":"2024-09-13T17:11:08.941636+0000","last_became_peered":"2024-09-13T17:11:08.941636+0000","last_unstale":"2024-09-13T17:11:09.934865+0000","last_undegraded":"2024-09-13T17:11:09.934865+0000","last_fullsized":"2024-09-13T17:11:09.934865+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130404+0000","last_change":"2024-09-13T17:11:08.940394+0000","last_active":"2024-09-13T17:11:11.130404+0000","last_peered":"2024-09-13T17:11:11.130404+0000","last_clean":"2024-09-13T17:11:11.130404+0000","last_became_active":"2024-09-13T17:11:08.937877+0000","last_became_peered":"2024-09-13T17:11:08.937877+0000","last_unstale":"2024-09-13T17:11:11.130404+0000","last_undegraded":"2024-09-13T17:11:11.130404+0000","last_fullsized":"2024-09-13T17:11:11.130404+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:16:19.951894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130918+0000","last_change":"2024-09-13T17:11:08.943653+0000","last_active":"2024-09-13T17:11:11.130918+0000","last_peered":"2024-09-13T17:11:11.130918+0000","last_clean":"2024-09-13T17:11:11.130918+0000","last_became_active":"2024-09-13T17:11:08.942874+0000","last_became_peered":"2024-09-13T17:11:08.942874+0000","last_unstale":"2024-09-13T17:11:11.130918+0000","last_undegraded":"2024-09-13T17:11:11.130918+0000","last_fullsized":"2024-09-13T17:11:11.130918+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:58.261986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130961+0000","last_change":"2024-09-13T17:11:08.946312+0000","last_active":"2024-09-13T17:11:11.130961+0000","last_peered":"2024-09-13T17:11:11.130961+0000","last_clean":"2024-09-13T17:11:11.130961+0000","last_became_active":"2024-09-13T17:11:08.945920+0000","last_became_peered":"2024-09-13T17:11:08.945920+0000","last_unstale":"2024-09-13T17:11:11.130961+0000","last_undegraded":"2024-09-13T17:11:11.130961+0000","last_fullsized":"2024-09-13T17:11:11.130961+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:47.991171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131840+0000","last_change":"2024-09-13T17:11:08.952918+0000","last_active":"2024-09-13T17:11:11.131840+0000","last_peered":"2024-09-13T17:11:11.131840+0000","last_clean":"2024-09-13T17:11:11.131840+0000","last_became_active":"2024-09-13T17:11:08.952326+0000","last_became_peered":"2024-09-13T17:11:08.952326+0000","last_unstale":"2024-09-13T17:11:11.131840+0000","last_undegraded":"2024-09-13T17:11:11.131840+0000","last_fullsized":"2024-09-13T17:11:11.131840+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:06.694348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131635+0000","last_change":"2024-09-13T17:11:08.948383+0000","last_active":"2024-09-13T17:11:11.131635+0000","last_peered":"2024-09-13T17:11:11.131635+0000","last_clean":"2024-09-13T17:11:11.131635+0000","last_became_active":"2024-09-13T17:11:08.944675+0000","last_became_peered":"2024-09-13T17:11:08.944675+0000","last_unstale":"2024-09-13T17:11:11.131635+0000","last_undegraded":"2024-09-13T17:11:11.131635+0000","last_fullsized":"2024-09-13T17:11:11.131635+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:11.796310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131501+0000","last_change":"2024-09-13T17:11:08.949752+0000","last_active":"2024-09-13T17:11:11.131501+0000","last_peered":"2024-09-13T17:11:11.131501+0000","last_clean":"2024-09-13T17:11:11.131501+0000","last_became_active":"2024-09-13T17:11:08.949514+0000","last_became_peered":"2024-09-13T17:11:08.949514+0000","last_unstale":"2024-09-13T17:11:11.131501+0000","last_undegraded":"2024-09-13T17:11:11.131501+0000","last_fullsized":"2024-09-13T17:11:11.131501+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935017+0000","last_change":"2024-09-13T17:11:08.939465+0000","last_active":"2024-09-13T17:11:09.935017+0000","last_peered":"2024-09-13T17:11:09.935017+0000","last_clean":"2024-09-13T17:11:09.935017+0000","last_became_active":"2024-09-13T17:11:08.939107+0000","last_became_peered":"2024-09-13T17:11:08.939107+0000","last_unstale":"2024-09-13T17:11:09.935017+0000","last_undegraded":"2024-09-13T17:11:09.935017+0000","last_fullsized":"2024-09-13T17:11:09.935017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:06.694348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735182+0000","last_change":"2024-09-13T17:11:08.949121+0000","last_active":"2024-09-13T17:11:10.735182+0000","last_peered":"2024-09-13T17:11:10.735182+0000","last_clean":"2024-09-13T17:11:10.735182+0000","last_became_active":"2024-09-13T17:11:08.948876+0000","last_became_peered":"2024-09-13T17:11:08.948876+0000","last_unstale":"2024-09-13T17:11:10.735182+0000","last_undegraded":"2024-09-13T17:11:10.735182+0000","last_fullsized":"2024-09-13T17:11:10.735182+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:54.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131017+0000","last_change":"2024-09-13T17:11:08.955847+0000","last_active":"2024-09-13T17:11:11.131017+0000","last_peered":"2024-09-13T17:11:11.131017+0000","last_clean":"2024-09-13T17:11:11.131017+0000","last_became_active":"2024-09-13T17:11:08.955609+0000","last_became_peered":"2024-09-13T17:11:08.955609+0000","last_unstale":"2024-09-13T17:11:11.131017+0000","last_undegraded":"2024-09-13T17:11:11.131017+0000","last_fullsized":"2024-09-13T17:11:11.131017+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130297+0000","last_change":"2024-09-13T17:11:08.941554+0000","last_active":"2024-09-13T17:11:11.130297+0000","last_peered":"2024-09-13T17:11:11.130297+0000","last_clean":"2024-09-13T17:11:11.130297+0000","last_became_active":"2024-09-13T17:11:08.941307+0000","last_became_peered":"2024-09-13T17:11:08.941307+0000","last_unstale":"2024-09-13T17:11:11.130297+0000","last_undegraded":"2024-09-13T17:11:11.130297+0000","last_fullsized":"2024-09-13T17:11:11.130297+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:18.081627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147530+0000","last_change":"2024-09-13T17:11:08.960818+0000","last_active":"2024-09-13T17:11:10.147530+0000","last_peered":"2024-09-13T17:11:10.147530+0000","last_clean":"2024-09-13T17:11:10.147530+0000","last_became_active":"2024-09-13T17:11:08.960593+0000","last_became_peered":"2024-09-13T17:11:08.960593+0000","last_unstale":"2024-09-13T17:11:10.147530+0000","last_undegraded":"2024-09-13T17:11:10.147530+0000","last_fullsized":"2024-09-13T17:11:10.147530+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.456687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935553+0000","last_change":"2024-09-13T17:11:08.952153+0000","last_active":"2024-09-13T17:11:09.935553+0000","last_peered":"2024-09-13T17:11:09.935553+0000","last_clean":"2024-09-13T17:11:09.935553+0000","last_became_active":"2024-09-13T17:11:08.951210+0000","last_became_peered":"2024-09-13T17:11:08.951210+0000","last_unstale":"2024-09-13T17:11:09.935553+0000","last_undegraded":"2024-09-13T17:11:09.935553+0000","last_fullsized":"2024-09-13T17:11:09.935553+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:39.355617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935246+0000","last_change":"2024-09-13T17:11:08.955918+0000","last_active":"2024-09-13T17:11:09.935246+0000","last_peered":"2024-09-13T17:11:09.935246+0000","last_clean":"2024-09-13T17:11:09.935246+0000","last_became_active":"2024-09-13T17:11:08.955640+0000","last_became_peered":"2024-09-13T17:11:08.955640+0000","last_unstale":"2024-09-13T17:11:09.935246+0000","last_undegraded":"2024-09-13T17:11:09.935246+0000","last_fullsized":"2024-09-13T17:11:09.935246+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:06.452065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148082+0000","last_change":"2024-09-13T17:11:08.955973+0000","last_active":"2024-09-13T17:11:10.148082+0000","last_peered":"2024-09-13T17:11:10.148082+0000","last_clean":"2024-09-13T17:11:10.148082+0000","last_became_active":"2024-09-13T17:11:08.955672+0000","last_became_peered":"2024-09-13T17:11:08.955672+0000","last_unstale":"2024-09-13T17:11:10.148082+0000","last_undegraded":"2024-09-13T17:11:10.148082+0000","last_fullsized":"2024-09-13T17:11:10.148082+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:05.881069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735287+0000","last_change":"2024-09-13T17:11:08.955292+0000","last_active":"2024-09-13T17:11:10.735287+0000","last_peered":"2024-09-13T17:11:10.735287+0000","last_clean":"2024-09-13T17:11:10.735287+0000","last_became_active":"2024-09-13T17:11:08.955026+0000","last_became_peered":"2024-09-13T17:11:08.955026+0000","last_unstale":"2024-09-13T17:11:10.735287+0000","last_undegraded":"2024-09-13T17:11:10.735287+0000","last_fullsized":"2024-09-13T17:11:10.735287+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:27.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130389+0000","last_change":"2024-09-13T17:11:08.957526+0000","last_active":"2024-09-13T17:11:11.130389+0000","last_peered":"2024-09-13T17:11:11.130389+0000","last_clean":"2024-09-13T17:11:11.130389+0000","last_became_active":"2024-09-13T17:11:08.957056+0000","last_became_peered":"2024-09-13T17:11:08.957056+0000","last_unstale":"2024-09-13T17:11:11.130389+0000","last_undegraded":"2024-09-13T17:11:11.130389+0000","last_fullsized":"2024-09-13T17:11:11.130389+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:08.483718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131111+0000","last_change":"2024-09-13T17:11:08.943501+0000","last_active":"2024-09-13T17:11:11.131111+0000","last_peered":"2024-09-13T17:11:11.131111+0000","last_clean":"2024-09-13T17:11:11.131111+0000","last_became_active":"2024-09-13T17:11:08.943259+0000","last_became_peered":"2024-09-13T17:11:08.943259+0000","last_unstale":"2024-09-13T17:11:11.131111+0000","last_undegraded":"2024-09-13T17:11:11.131111+0000","last_fullsized":"2024-09-13T17:11:11.131111+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.456687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147848+0000","last_change":"2024-09-13T17:11:08.946631+0000","last_active":"2024-09-13T17:11:10.147848+0000","last_peered":"2024-09-13T17:11:10.147848+0000","last_clean":"2024-09-13T17:11:10.147848+0000","last_became_active":"2024-09-13T17:11:08.945998+0000","last_became_peered":"2024-09-13T17:11:08.945998+0000","last_unstale":"2024-09-13T17:11:10.147848+0000","last_undegraded":"2024-09-13T17:11:10.147848+0000","last_fullsized":"2024-09-13T17:11:10.147848+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:54.425921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147610+0000","last_change":"2024-09-13T17:11:08.949674+0000","last_active":"2024-09-13T17:11:10.147610+0000","last_peered":"2024-09-13T17:11:10.147610+0000","last_clean":"2024-09-13T17:11:10.147610+0000","last_became_active":"2024-09-13T17:11:08.949352+0000","last_became_peered":"2024-09-13T17:11:08.949352+0000","last_unstale":"2024-09-13T17:11:10.147610+0000","last_undegraded":"2024-09-13T17:11:10.147610+0000","last_fullsized":"2024-09-13T17:11:10.147610+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:27.950671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148016+0000","last_change":"2024-09-13T17:11:08.954704+0000","last_active":"2024-09-13T17:11:10.148016+0000","last_peered":"2024-09-13T17:11:10.148016+0000","last_clean":"2024-09-13T17:11:10.148016+0000","last_became_active":"2024-09-13T17:11:08.954434+0000","last_became_peered":"2024-09-13T17:11:08.954434+0000","last_unstale":"2024-09-13T17:11:10.148016+0000","last_undegraded":"2024-09-13T17:11:10.148016+0000","last_fullsized":"2024-09-13T17:11:10.148016+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:39.983381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735168+0000","last_change":"2024-09-13T17:11:08.949307+0000","last_active":"2024-09-13T17:11:10.735168+0000","last_peered":"2024-09-13T17:11:10.735168+0000","last_clean":"2024-09-13T17:11:10.735168+0000","last_became_active":"2024-09-13T17:11:08.948037+0000","last_became_peered":"2024-09-13T17:11:08.948037+0000","last_unstale":"2024-09-13T17:11:10.735168+0000","last_undegraded":"2024-09-13T17:11:10.735168+0000","last_fullsized":"2024-09-13T17:11:10.735168+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:53.597936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935083+0000","last_change":"2024-09-13T17:11:08.947410+0000","last_active":"2024-09-13T17:11:09.935083+0000","last_peered":"2024-09-13T17:11:09.935083+0000","last_clean":"2024-09-13T17:11:09.935083+0000","last_became_active":"2024-09-13T17:11:08.942341+0000","last_became_peered":"2024-09-13T17:11:08.942341+0000","last_unstale":"2024-09-13T17:11:09.935083+0000","last_undegraded":"2024-09-13T17:11:09.935083+0000","last_fullsized":"2024-09-13T17:11:09.935083+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935138+0000","last_change":"2024-09-13T17:11:08.939334+0000","last_active":"2024-09-13T17:11:09.935138+0000","last_peered":"2024-09-13T17:11:09.935138+0000","last_clean":"2024-09-13T17:11:09.935138+0000","last_became_active":"2024-09-13T17:11:08.939007+0000","last_became_peered":"2024-09-13T17:11:08.939007+0000","last_unstale":"2024-09-13T17:11:09.935138+0000","last_undegraded":"2024-09-13T17:11:09.935138+0000","last_fullsized":"2024-09-13T17:11:09.935138+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:11.796310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147988+0000","last_change":"2024-09-13T17:11:08.954868+0000","last_active":"2024-09-13T17:11:10.147988+0000","last_peered":"2024-09-13T17:11:10.147988+0000","last_clean":"2024-09-13T17:11:10.147988+0000","last_became_active":"2024-09-13T17:11:08.953532+0000","last_became_peered":"2024-09-13T17:11:08.953532+0000","last_unstale":"2024-09-13T17:11:10.147988+0000","last_undegraded":"2024-09-13T17:11:10.147988+0000","last_fullsized":"2024-09-13T17:11:10.147988+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934546+0000","last_change":"2024-09-13T17:11:08.942858+0000","last_active":"2024-09-13T17:11:09.934546+0000","last_peered":"2024-09-13T17:11:09.934546+0000","last_clean":"2024-09-13T17:11:09.934546+0000","last_became_active":"2024-09-13T17:11:08.942395+0000","last_became_peered":"2024-09-13T17:11:08.942395+0000","last_unstale":"2024-09-13T17:11:09.934546+0000","last_undegraded":"2024-09-13T17:11:09.934546+0000","last_fullsized":"2024-09-13T17:11:09.934546+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:26.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130368+0000","last_change":"2024-09-13T17:11:08.934543+0000","last_active":"2024-09-13T17:11:11.130368+0000","last_peered":"2024-09-13T17:11:11.130368+0000","last_clean":"2024-09-13T17:11:11.130368+0000","last_became_active":"2024-09-13T17:11:08.932596+0000","last_became_peered":"2024-09-13T17:11:08.932596+0000","last_unstale":"2024-09-13T17:11:11.130368+0000","last_undegraded":"2024-09-13T17:11:11.130368+0000","last_fullsized":"2024-09-13T17:11:11.130368+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:17.898561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735311+0000","last_change":"2024-09-13T17:11:08.953304+0000","last_active":"2024-09-13T17:11:10.735311+0000","last_peered":"2024-09-13T17:11:10.735311+0000","last_clean":"2024-09-13T17:11:10.735311+0000","last_became_active":"2024-09-13T17:11:08.951792+0000","last_became_peered":"2024-09-13T17:11:08.951792+0000","last_unstale":"2024-09-13T17:11:10.735311+0000","last_undegraded":"2024-09-13T17:11:10.735311+0000","last_fullsized":"2024-09-13T17:11:10.735311+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:51.187481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933432+0000","last_change":"2024-09-13T17:11:08.949135+0000","last_active":"2024-09-13T17:11:09.933432+0000","last_peered":"2024-09-13T17:11:09.933432+0000","last_clean":"2024-09-13T17:11:09.933432+0000","last_became_active":"2024-09-13T17:11:08.948880+0000","last_became_peered":"2024-09-13T17:11:08.948880+0000","last_unstale":"2024-09-13T17:11:09.933432+0000","last_undegraded":"2024-09-13T17:11:09.933432+0000","last_fullsized":"2024-09-13T17:11:09.933432+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:39.983381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"47'4","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960698+0000","last_change":"2024-09-13T17:11:08.508913+0000","last_active":"2024-09-13T17:11:08.960698+0000","last_peered":"2024-09-13T17:11:08.960698+0000","last_clean":"2024-09-13T17:11:08.960698+0000","last_became_active":"2024-09-13T17:11:08.508643+0000","last_became_peered":"2024-09-13T17:11:08.508643+0000","last_unstale":"2024-09-13T17:11:08.960698+0000","last_undegraded":"2024-09-13T17:11:08.960698+0000","last_fullsized":"2024-09-13T17:11:08.960698+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:52.107862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148080+0000","last_change":"2024-09-13T17:11:08.953202+0000","last_active":"2024-09-13T17:11:10.148080+0000","last_peered":"2024-09-13T17:11:10.148080+0000","last_clean":"2024-09-13T17:11:10.148080+0000","last_became_active":"2024-09-13T17:11:08.952916+0000","last_became_peered":"2024-09-13T17:11:08.952916+0000","last_unstale":"2024-09-13T17:11:10.148080+0000","last_undegraded":"2024-09-13T17:11:10.148080+0000","last_fullsized":"2024-09-13T17:11:10.148080+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:06.426163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.955348+0000","last_change":"2024-09-13T17:11:08.511916+0000","last_active":"2024-09-13T17:11:08.955348+0000","last_peered":"2024-09-13T17:11:08.955348+0000","last_clean":"2024-09-13T17:11:08.955348+0000","last_became_active":"2024-09-13T17:11:08.511641+0000","last_became_peered":"2024-09-13T17:11:08.511641+0000","last_unstale":"2024-09-13T17:11:08.955348+0000","last_undegraded":"2024-09-13T17:11:08.955348+0000","last_fullsized":"2024-09-13T17:11:08.955348+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935245+0000","last_change":"2024-09-13T17:11:08.944416+0000","last_active":"2024-09-13T17:11:09.935245+0000","last_peered":"2024-09-13T17:11:09.935245+0000","last_clean":"2024-09-13T17:11:09.935245+0000","last_became_active":"2024-09-13T17:11:08.943865+0000","last_became_peered":"2024-09-13T17:11:08.943865+0000","last_unstale":"2024-09-13T17:11:09.935245+0000","last_undegraded":"2024-09-13T17:11:09.935245+0000","last_fullsized":"2024-09-13T17:11:09.935245+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:54.026958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957342+0000","last_change":"2024-09-13T17:11:08.513124+0000","last_active":"2024-09-13T17:11:08.957342+0000","last_peered":"2024-09-13T17:11:08.957342+0000","last_clean":"2024-09-13T17:11:08.957342+0000","last_became_active":"2024-09-13T17:11:08.512830+0000","last_became_peered":"2024-09-13T17:11:08.512830+0000","last_unstale":"2024-09-13T17:11:08.957342+0000","last_undegraded":"2024-09-13T17:11:08.957342+0000","last_fullsized":"2024-09-13T17:11:08.957342+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:15.751862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933496+0000","last_change":"2024-09-13T17:11:08.947611+0000","last_active":"2024-09-13T17:11:09.933496+0000","last_peered":"2024-09-13T17:11:09.933496+0000","last_clean":"2024-09-13T17:11:09.933496+0000","last_became_active":"2024-09-13T17:11:08.946913+0000","last_became_peered":"2024-09-13T17:11:08.946913+0000","last_unstale":"2024-09-13T17:11:09.933496+0000","last_undegraded":"2024-09-13T17:11:09.933496+0000","last_fullsized":"2024-09-13T17:11:09.933496+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:06.426163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957281+0000","last_change":"2024-09-13T17:11:08.514397+0000","last_active":"2024-09-13T17:11:08.957281+0000","last_peered":"2024-09-13T17:11:08.957281+0000","last_clean":"2024-09-13T17:11:08.957281+0000","last_became_active":"2024-09-13T17:11:08.514134+0000","last_became_peered":"2024-09-13T17:11:08.514134+0000","last_unstale":"2024-09-13T17:11:08.957281+0000","last_undegraded":"2024-09-13T17:11:08.957281+0000","last_fullsized":"2024-09-13T17:11:08.957281+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:42.476669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934596+0000","last_change":"2024-09-13T17:11:08.940677+0000","last_active":"2024-09-13T17:11:09.934596+0000","last_peered":"2024-09-13T17:11:09.934596+0000","last_clean":"2024-09-13T17:11:09.934596+0000","last_became_active":"2024-09-13T17:11:08.940390+0000","last_became_peered":"2024-09-13T17:11:08.940390+0000","last_unstale":"2024-09-13T17:11:09.934596+0000","last_undegraded":"2024-09-13T17:11:09.934596+0000","last_fullsized":"2024-09-13T17:11:09.934596+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:21.531576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957368+0000","last_change":"2024-09-13T17:11:08.509255+0000","last_active":"2024-09-13T17:11:08.957368+0000","last_peered":"2024-09-13T17:11:08.957368+0000","last_clean":"2024-09-13T17:11:08.957368+0000","last_became_active":"2024-09-13T17:11:08.508943+0000","last_became_peered":"2024-09-13T17:11:08.508943+0000","last_unstale":"2024-09-13T17:11:08.957368+0000","last_undegraded":"2024-09-13T17:11:08.957368+0000","last_fullsized":"2024-09-13T17:11:08.957368+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:53.020411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131112+0000","last_change":"2024-09-13T17:11:08.938576+0000","last_active":"2024-09-13T17:11:11.131112+0000","last_peered":"2024-09-13T17:11:11.131112+0000","last_clean":"2024-09-13T17:11:11.131112+0000","last_became_active":"2024-09-13T17:11:08.938308+0000","last_became_peered":"2024-09-13T17:11:08.938308+0000","last_unstale":"2024-09-13T17:11:11.131112+0000","last_undegraded":"2024-09-13T17:11:11.131112+0000","last_fullsized":"2024-09-13T17:11:11.131112+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:06.452065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.949613+0000","last_change":"2024-09-13T17:11:08.511045+0000","last_active":"2024-09-13T17:11:08.949613+0000","last_peered":"2024-09-13T17:11:08.949613+0000","last_clean":"2024-09-13T17:11:08.949613+0000","last_became_active":"2024-09-13T17:11:08.510781+0000","last_became_peered":"2024-09-13T17:11:08.510781+0000","last_unstale":"2024-09-13T17:11:08.949613+0000","last_undegraded":"2024-09-13T17:11:08.949613+0000","last_fullsized":"2024-09-13T17:11:08.949613+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:18:37.254451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131002+0000","last_change":"2024-09-13T17:11:08.946157+0000","last_active":"2024-09-13T17:11:11.131002+0000","last_peered":"2024-09-13T17:11:11.131002+0000","last_clean":"2024-09-13T17:11:11.131002+0000","last_became_active":"2024-09-13T17:11:08.945553+0000","last_became_peered":"2024-09-13T17:11:08.945553+0000","last_unstale":"2024-09-13T17:11:11.131002+0000","last_undegraded":"2024-09-13T17:11:11.131002+0000","last_fullsized":"2024-09-13T17:11:11.131002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:26.573566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927820+0000","last_change":"2024-09-13T17:11:08.504455+0000","last_active":"2024-09-13T17:11:08.927820+0000","last_peered":"2024-09-13T17:11:08.927820+0000","last_clean":"2024-09-13T17:11:08.927820+0000","last_became_active":"2024-09-13T17:11:08.504166+0000","last_became_peered":"2024-09-13T17:11:08.504166+0000","last_unstale":"2024-09-13T17:11:08.927820+0000","last_undegraded":"2024-09-13T17:11:08.927820+0000","last_fullsized":"2024-09-13T17:11:08.927820+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:22.582466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147645+0000","last_change":"2024-09-13T17:11:08.946784+0000","last_active":"2024-09-13T17:11:10.147645+0000","last_peered":"2024-09-13T17:11:10.147645+0000","last_clean":"2024-09-13T17:11:10.147645+0000","last_became_active":"2024-09-13T17:11:08.946299+0000","last_became_peered":"2024-09-13T17:11:08.946299+0000","last_unstale":"2024-09-13T17:11:10.147645+0000","last_undegraded":"2024-09-13T17:11:10.147645+0000","last_fullsized":"2024-09-13T17:11:10.147645+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:53.597936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.934280+0000","last_change":"2024-09-13T17:11:08.504442+0000","last_active":"2024-09-13T17:11:08.934280+0000","last_peered":"2024-09-13T17:11:08.934280+0000","last_clean":"2024-09-13T17:11:08.934280+0000","last_became_active":"2024-09-13T17:11:08.504168+0000","last_became_peered":"2024-09-13T17:11:08.504168+0000","last_unstale":"2024-09-13T17:11:08.934280+0000","last_undegraded":"2024-09-13T17:11:08.934280+0000","last_fullsized":"2024-09-13T17:11:08.934280+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:55.602732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.935274+0000","last_change":"2024-09-13T17:11:08.956686+0000","last_active":"2024-09-13T17:11:09.935274+0000","last_peered":"2024-09-13T17:11:09.935274+0000","last_clean":"2024-09-13T17:11:09.935274+0000","last_became_active":"2024-09-13T17:11:08.956387+0000","last_became_peered":"2024-09-13T17:11:08.956387+0000","last_unstale":"2024-09-13T17:11:09.935274+0000","last_undegraded":"2024-09-13T17:11:09.935274+0000","last_fullsized":"2024-09-13T17:11:09.935274+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:43.483216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.942828+0000","last_change":"2024-09-13T17:11:08.511559+0000","last_active":"2024-09-13T17:11:08.942828+0000","last_peered":"2024-09-13T17:11:08.942828+0000","last_clean":"2024-09-13T17:11:08.942828+0000","last_became_active":"2024-09-13T17:11:08.511251+0000","last_became_peered":"2024-09-13T17:11:08.511251+0000","last_unstale":"2024-09-13T17:11:08.942828+0000","last_undegraded":"2024-09-13T17:11:08.942828+0000","last_fullsized":"2024-09-13T17:11:08.942828+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:55.414078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131268+0000","last_change":"2024-09-13T17:11:08.938024+0000","last_active":"2024-09-13T17:11:11.131268+0000","last_peered":"2024-09-13T17:11:11.131268+0000","last_clean":"2024-09-13T17:11:11.131268+0000","last_became_active":"2024-09-13T17:11:08.936316+0000","last_became_peered":"2024-09-13T17:11:08.936316+0000","last_unstale":"2024-09-13T17:11:11.131268+0000","last_undegraded":"2024-09-13T17:11:11.131268+0000","last_fullsized":"2024-09-13T17:11:11.131268+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:46.203136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.936620+0000","last_change":"2024-09-13T17:11:08.505900+0000","last_active":"2024-09-13T17:11:08.936620+0000","last_peered":"2024-09-13T17:11:08.936620+0000","last_clean":"2024-09-13T17:11:08.936620+0000","last_became_active":"2024-09-13T17:11:08.505659+0000","last_became_peered":"2024-09-13T17:11:08.505659+0000","last_unstale":"2024-09-13T17:11:08.936620+0000","last_undegraded":"2024-09-13T17:11:08.936620+0000","last_fullsized":"2024-09-13T17:11:08.936620+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:28.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130788+0000","last_change":"2024-09-13T17:11:08.953384+0000","last_active":"2024-09-13T17:11:11.130788+0000","last_peered":"2024-09-13T17:11:11.130788+0000","last_clean":"2024-09-13T17:11:11.130788+0000","last_became_active":"2024-09-13T17:11:08.953110+0000","last_became_peered":"2024-09-13T17:11:08.953110+0000","last_unstale":"2024-09-13T17:11:11.130788+0000","last_undegraded":"2024-09-13T17:11:11.130788+0000","last_fullsized":"2024-09-13T17:11:11.130788+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:11.104415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"47'3","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.958240+0000","last_change":"2024-09-13T17:11:08.510876+0000","last_active":"2024-09-13T17:11:08.958240+0000","last_peered":"2024-09-13T17:11:08.958240+0000","last_clean":"2024-09-13T17:11:08.958240+0000","last_became_active":"2024-09-13T17:11:08.510236+0000","last_became_peered":"2024-09-13T17:11:08.510236+0000","last_unstale":"2024-09-13T17:11:08.958240+0000","last_undegraded":"2024-09-13T17:11:08.958240+0000","last_fullsized":"2024-09-13T17:11:08.958240+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:38.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131184+0000","last_change":"2024-09-13T17:11:08.940929+0000","last_active":"2024-09-13T17:11:11.131184+0000","last_peered":"2024-09-13T17:11:11.131184+0000","last_clean":"2024-09-13T17:11:11.131184+0000","last_became_active":"2024-09-13T17:11:08.940670+0000","last_became_peered":"2024-09-13T17:11:08.940670+0000","last_unstale":"2024-09-13T17:11:11.131184+0000","last_undegraded":"2024-09-13T17:11:11.131184+0000","last_fullsized":"2024-09-13T17:11:11.131184+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:47.991171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"47'6","reported_seq":17,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.961852+0000","last_change":"2024-09-13T17:11:08.509610+0000","last_active":"2024-09-13T17:11:08.961852+0000","last_peered":"2024-09-13T17:11:08.961852+0000","last_clean":"2024-09-13T17:11:08.961852+0000","last_became_active":"2024-09-13T17:11:08.509323+0000","last_became_peered":"2024-09-13T17:11:08.509323+0000","last_unstale":"2024-09-13T17:11:08.961852+0000","last_undegraded":"2024-09-13T17:11:08.961852+0000","last_fullsized":"2024-09-13T17:11:08.961852+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:55.047800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147908+0000","last_change":"2024-09-13T17:11:08.955884+0000","last_active":"2024-09-13T17:11:10.147908+0000","last_peered":"2024-09-13T17:11:10.147908+0000","last_clean":"2024-09-13T17:11:10.147908+0000","last_became_active":"2024-09-13T17:11:08.955558+0000","last_became_peered":"2024-09-13T17:11:08.955558+0000","last_unstale":"2024-09-13T17:11:10.147908+0000","last_undegraded":"2024-09-13T17:11:10.147908+0000","last_fullsized":"2024-09-13T17:11:10.147908+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:51.187481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960010+0000","last_change":"2024-09-13T17:11:08.510858+0000","last_active":"2024-09-13T17:11:08.960010+0000","last_peered":"2024-09-13T17:11:08.960010+0000","last_clean":"2024-09-13T17:11:08.960010+0000","last_became_active":"2024-09-13T17:11:08.509886+0000","last_became_peered":"2024-09-13T17:11:08.509886+0000","last_unstale":"2024-09-13T17:11:08.960010+0000","last_undegraded":"2024-09-13T17:11:08.960010+0000","last_fullsized":"2024-09-13T17:11:08.960010+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:05.445518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131371+0000","last_change":"2024-09-13T17:11:08.940863+0000","last_active":"2024-09-13T17:11:11.131371+0000","last_peered":"2024-09-13T17:11:11.131371+0000","last_clean":"2024-09-13T17:11:11.131371+0000","last_became_active":"2024-09-13T17:11:08.940591+0000","last_became_peered":"2024-09-13T17:11:08.940591+0000","last_unstale":"2024-09-13T17:11:11.131371+0000","last_undegraded":"2024-09-13T17:11:11.131371+0000","last_fullsized":"2024-09-13T17:11:11.131371+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.977161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928572+0000","last_change":"2024-09-13T17:11:08.505765+0000","last_active":"2024-09-13T17:11:08.928572+0000","last_peered":"2024-09-13T17:11:08.928572+0000","last_clean":"2024-09-13T17:11:08.928572+0000","last_became_active":"2024-09-13T17:11:08.505511+0000","last_became_peered":"2024-09-13T17:11:08.505511+0000","last_unstale":"2024-09-13T17:11:08.928572+0000","last_undegraded":"2024-09-13T17:11:08.928572+0000","last_fullsized":"2024-09-13T17:11:08.928572+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:51.970613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130702+0000","last_change":"2024-09-13T17:11:08.944939+0000","last_active":"2024-09-13T17:11:11.130702+0000","last_peered":"2024-09-13T17:11:11.130702+0000","last_clean":"2024-09-13T17:11:11.130702+0000","last_became_active":"2024-09-13T17:11:08.944655+0000","last_became_peered":"2024-09-13T17:11:08.944655+0000","last_unstale":"2024-09-13T17:11:11.130702+0000","last_undegraded":"2024-09-13T17:11:11.130702+0000","last_fullsized":"2024-09-13T17:11:11.130702+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:53.374179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.925788+0000","last_change":"2024-09-13T17:11:08.511713+0000","last_active":"2024-09-13T17:11:08.925788+0000","last_peered":"2024-09-13T17:11:08.925788+0000","last_clean":"2024-09-13T17:11:08.925788+0000","last_became_active":"2024-09-13T17:11:08.511362+0000","last_became_peered":"2024-09-13T17:11:08.511362+0000","last_unstale":"2024-09-13T17:11:08.925788+0000","last_undegraded":"2024-09-13T17:11:08.925788+0000","last_fullsized":"2024-09-13T17:11:08.925788+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:58.529901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131153+0000","last_change":"2024-09-13T17:11:08.939252+0000","last_active":"2024-09-13T17:11:11.131153+0000","last_peered":"2024-09-13T17:11:11.131153+0000","last_clean":"2024-09-13T17:11:11.131153+0000","last_became_active":"2024-09-13T17:11:08.938975+0000","last_became_peered":"2024-09-13T17:11:08.938975+0000","last_unstale":"2024-09-13T17:11:11.131153+0000","last_undegraded":"2024-09-13T17:11:11.131153+0000","last_fullsized":"2024-09-13T17:11:11.131153+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:59.536449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"47'8","reported_seq":19,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.960876+0000","last_change":"2024-09-13T17:11:08.510567+0000","last_active":"2024-09-13T17:11:08.960876+0000","last_peered":"2024-09-13T17:11:08.960876+0000","last_clean":"2024-09-13T17:11:08.960876+0000","last_became_active":"2024-09-13T17:11:08.509318+0000","last_became_peered":"2024-09-13T17:11:08.509318+0000","last_unstale":"2024-09-13T17:11:08.960876+0000","last_undegraded":"2024-09-13T17:11:08.960876+0000","last_fullsized":"2024-09-13T17:11:08.960876+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:43.667809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735291+0000","last_change":"2024-09-13T17:11:08.936837+0000","last_active":"2024-09-13T17:11:10.735291+0000","last_peered":"2024-09-13T17:11:10.735291+0000","last_clean":"2024-09-13T17:11:10.735291+0000","last_became_active":"2024-09-13T17:11:08.935945+0000","last_became_peered":"2024-09-13T17:11:08.935945+0000","last_unstale":"2024-09-13T17:11:10.735291+0000","last_undegraded":"2024-09-13T17:11:10.735291+0000","last_fullsized":"2024-09-13T17:11:10.735291+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.955990+0000","last_change":"2024-09-13T17:11:08.500720+0000","last_active":"2024-09-13T17:11:08.955990+0000","last_peered":"2024-09-13T17:11:08.955990+0000","last_clean":"2024-09-13T17:11:08.955990+0000","last_became_active":"2024-09-13T17:11:08.500406+0000","last_became_peered":"2024-09-13T17:11:08.500406+0000","last_unstale":"2024-09-13T17:11:08.955990+0000","last_undegraded":"2024-09-13T17:11:08.955990+0000","last_fullsized":"2024-09-13T17:11:08.955990+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:28.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131833+0000","last_change":"2024-09-13T17:11:08.942728+0000","last_active":"2024-09-13T17:11:11.131833+0000","last_peered":"2024-09-13T17:11:11.131833+0000","last_clean":"2024-09-13T17:11:11.131833+0000","last_became_active":"2024-09-13T17:11:08.942434+0000","last_became_peered":"2024-09-13T17:11:08.942434+0000","last_unstale":"2024-09-13T17:11:11.131833+0000","last_undegraded":"2024-09-13T17:11:11.131833+0000","last_fullsized":"2024-09-13T17:11:11.131833+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:36.572460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.953820+0000","last_change":"2024-09-13T17:11:08.506379+0000","last_active":"2024-09-13T17:11:08.953820+0000","last_peered":"2024-09-13T17:11:08.953820+0000","last_clean":"2024-09-13T17:11:08.953820+0000","last_became_active":"2024-09-13T17:11:08.506074+0000","last_became_peered":"2024-09-13T17:11:08.506074+0000","last_unstale":"2024-09-13T17:11:08.953820+0000","last_undegraded":"2024-09-13T17:11:08.953820+0000","last_fullsized":"2024-09-13T17:11:08.953820+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:15.751862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147919+0000","last_change":"2024-09-13T17:11:08.952135+0000","last_active":"2024-09-13T17:11:10.147919+0000","last_peered":"2024-09-13T17:11:10.147919+0000","last_clean":"2024-09-13T17:11:10.147919+0000","last_became_active":"2024-09-13T17:11:08.951223+0000","last_became_peered":"2024-09-13T17:11:08.951223+0000","last_unstale":"2024-09-13T17:11:10.147919+0000","last_undegraded":"2024-09-13T17:11:10.147919+0000","last_fullsized":"2024-09-13T17:11:10.147919+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.589014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"47'1","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.962381+0000","last_change":"2024-09-13T17:11:08.504709+0000","last_active":"2024-09-13T17:11:08.962381+0000","last_peered":"2024-09-13T17:11:08.962381+0000","last_clean":"2024-09-13T17:11:08.962381+0000","last_became_active":"2024-09-13T17:11:08.504412+0000","last_became_peered":"2024-09-13T17:11:08.504412+0000","last_unstale":"2024-09-13T17:11:08.962381+0000","last_undegraded":"2024-09-13T17:11:08.962381+0000","last_fullsized":"2024-09-13T17:11:08.962381+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:14.450139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.147901+0000","last_change":"2024-09-13T17:11:08.953098+0000","last_active":"2024-09-13T17:11:10.147901+0000","last_peered":"2024-09-13T17:11:10.147901+0000","last_clean":"2024-09-13T17:11:10.147901+0000","last_became_active":"2024-09-13T17:11:08.952816+0000","last_became_peered":"2024-09-13T17:11:08.952816+0000","last_unstale":"2024-09-13T17:11:10.147901+0000","last_undegraded":"2024-09-13T17:11:10.147901+0000","last_fullsized":"2024-09-13T17:11:10.147901+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:38.612605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928253+0000","last_change":"2024-09-13T17:11:08.507342+0000","last_active":"2024-09-13T17:11:08.928253+0000","last_peered":"2024-09-13T17:11:08.928253+0000","last_clean":"2024-09-13T17:11:08.928253+0000","last_became_active":"2024-09-13T17:11:08.506077+0000","last_became_peered":"2024-09-13T17:11:08.506077+0000","last_unstale":"2024-09-13T17:11:08.928253+0000","last_undegraded":"2024-09-13T17:11:08.928253+0000","last_fullsized":"2024-09-13T17:11:08.928253+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:06:38.349069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130520+0000","last_change":"2024-09-13T17:11:08.955283+0000","last_active":"2024-09-13T17:11:11.130520+0000","last_peered":"2024-09-13T17:11:11.130520+0000","last_clean":"2024-09-13T17:11:11.130520+0000","last_became_active":"2024-09-13T17:11:08.954324+0000","last_became_peered":"2024-09-13T17:11:08.954324+0000","last_unstale":"2024-09-13T17:11:11.130520+0000","last_undegraded":"2024-09-13T17:11:11.130520+0000","last_fullsized":"2024-09-13T17:11:11.130520+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:38.172171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.957915+0000","last_change":"2024-09-13T17:11:08.498279+0000","last_active":"2024-09-13T17:11:08.957915+0000","last_peered":"2024-09-13T17:11:08.957915+0000","last_clean":"2024-09-13T17:11:08.957915+0000","last_became_active":"2024-09-13T17:11:08.497833+0000","last_became_peered":"2024-09-13T17:11:08.497833+0000","last_unstale":"2024-09-13T17:11:08.957915+0000","last_undegraded":"2024-09-13T17:11:08.957915+0000","last_fullsized":"2024-09-13T17:11:08.957915+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:25:25.567018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.933122+0000","last_change":"2024-09-13T17:11:08.935400+0000","last_active":"2024-09-13T17:11:09.933122+0000","last_peered":"2024-09-13T17:11:09.933122+0000","last_clean":"2024-09-13T17:11:09.933122+0000","last_became_active":"2024-09-13T17:11:08.935093+0000","last_became_peered":"2024-09-13T17:11:08.935093+0000","last_unstale":"2024-09-13T17:11:09.933122+0000","last_undegraded":"2024-09-13T17:11:09.933122+0000","last_fullsized":"2024-09-13T17:11:09.933122+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:49:23.589014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927574+0000","last_change":"2024-09-13T17:11:08.510342+0000","last_active":"2024-09-13T17:11:08.927574+0000","last_peered":"2024-09-13T17:11:08.927574+0000","last_clean":"2024-09-13T17:11:08.927574+0000","last_became_active":"2024-09-13T17:11:08.510079+0000","last_became_peered":"2024-09-13T17:11:08.510079+0000","last_unstale":"2024-09-13T17:11:08.927574+0000","last_undegraded":"2024-09-13T17:11:08.927574+0000","last_fullsized":"2024-09-13T17:11:08.927574+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:37.606057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.735321+0000","last_change":"2024-09-13T17:11:08.944120+0000","last_active":"2024-09-13T17:11:10.735321+0000","last_peered":"2024-09-13T17:11:10.735321+0000","last_clean":"2024-09-13T17:11:10.735321+0000","last_became_active":"2024-09-13T17:11:08.943870+0000","last_became_peered":"2024-09-13T17:11:08.943870+0000","last_unstale":"2024-09-13T17:11:10.735321+0000","last_undegraded":"2024-09-13T17:11:10.735321+0000","last_fullsized":"2024-09-13T17:11:10.735321+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.932354+0000","last_change":"2024-09-13T17:11:08.505319+0000","last_active":"2024-09-13T17:11:08.932354+0000","last_peered":"2024-09-13T17:11:08.932354+0000","last_clean":"2024-09-13T17:11:08.932354+0000","last_became_active":"2024-09-13T17:11:08.505029+0000","last_became_peered":"2024-09-13T17:11:08.505029+0000","last_unstale":"2024-09-13T17:11:08.932354+0000","last_undegraded":"2024-09-13T17:11:08.932354+0000","last_fullsized":"2024-09-13T17:11:08.932354+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:19.732809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131317+0000","last_change":"2024-09-13T17:11:08.956632+0000","last_active":"2024-09-13T17:11:11.131317+0000","last_peered":"2024-09-13T17:11:11.131317+0000","last_clean":"2024-09-13T17:11:11.131317+0000","last_became_active":"2024-09-13T17:11:08.956385+0000","last_became_peered":"2024-09-13T17:11:08.956385+0000","last_unstale":"2024-09-13T17:11:11.131317+0000","last_undegraded":"2024-09-13T17:11:11.131317+0000","last_fullsized":"2024-09-13T17:11:11.131317+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:54.026958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.926953+0000","last_change":"2024-09-13T17:11:08.505825+0000","last_active":"2024-09-13T17:11:08.926953+0000","last_peered":"2024-09-13T17:11:08.926953+0000","last_clean":"2024-09-13T17:11:08.926953+0000","last_became_active":"2024-09-13T17:11:08.505500+0000","last_became_peered":"2024-09-13T17:11:08.505500+0000","last_unstale":"2024-09-13T17:11:08.926953+0000","last_undegraded":"2024-09-13T17:11:08.926953+0000","last_fullsized":"2024-09-13T17:11:08.926953+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:20.525029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:10.148124+0000","last_change":"2024-09-13T17:11:08.962536+0000","last_active":"2024-09-13T17:11:10.148124+0000","last_peered":"2024-09-13T17:11:10.148124+0000","last_clean":"2024-09-13T17:11:10.148124+0000","last_became_active":"2024-09-13T17:11:08.962247+0000","last_became_peered":"2024-09-13T17:11:08.962247+0000","last_unstale":"2024-09-13T17:11:10.148124+0000","last_undegraded":"2024-09-13T17:11:10.148124+0000","last_fullsized":"2024-09-13T17:11:10.148124+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:16.758409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.927518+0000","last_change":"2024-09-13T17:11:08.514288+0000","last_active":"2024-09-13T17:11:08.927518+0000","last_peered":"2024-09-13T17:11:08.927518+0000","last_clean":"2024-09-13T17:11:08.927518+0000","last_became_active":"2024-09-13T17:11:08.513978+0000","last_became_peered":"2024-09-13T17:11:08.513978+0000","last_unstale":"2024-09-13T17:11:08.927518+0000","last_undegraded":"2024-09-13T17:11:08.927518+0000","last_fullsized":"2024-09-13T17:11:08.927518+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:35.565913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131565+0000","last_change":"2024-09-13T17:11:08.950475+0000","last_active":"2024-09-13T17:11:11.131565+0000","last_peered":"2024-09-13T17:11:11.131565+0000","last_clean":"2024-09-13T17:11:11.131565+0000","last_became_active":"2024-09-13T17:11:08.949141+0000","last_became_peered":"2024-09-13T17:11:08.949141+0000","last_unstale":"2024-09-13T17:11:11.131565+0000","last_undegraded":"2024-09-13T17:11:11.131565+0000","last_fullsized":"2024-09-13T17:11:11.131565+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.930241+0000","last_change":"2024-09-13T17:11:08.514093+0000","last_active":"2024-09-13T17:11:08.930241+0000","last_peered":"2024-09-13T17:11:08.930241+0000","last_clean":"2024-09-13T17:11:08.930241+0000","last_became_active":"2024-09-13T17:11:08.513508+0000","last_became_peered":"2024-09-13T17:11:08.513508+0000","last_unstale":"2024-09-13T17:11:08.930241+0000","last_undegraded":"2024-09-13T17:11:08.930241+0000","last_fullsized":"2024-09-13T17:11:08.930241+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:59.723386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131083+0000","last_change":"2024-09-13T17:11:08.947653+0000","last_active":"2024-09-13T17:11:11.131083+0000","last_peered":"2024-09-13T17:11:11.131083+0000","last_clean":"2024-09-13T17:11:11.131083+0000","last_became_active":"2024-09-13T17:11:08.947330+0000","last_became_peered":"2024-09-13T17:11:08.947330+0000","last_unstale":"2024-09-13T17:11:11.131083+0000","last_undegraded":"2024-09-13T17:11:11.131083+0000","last_fullsized":"2024-09-13T17:11:11.131083+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:47.532814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.932196+0000","last_change":"2024-09-13T17:11:08.512643+0000","last_active":"2024-09-13T17:11:08.932196+0000","last_peered":"2024-09-13T17:11:08.932196+0000","last_clean":"2024-09-13T17:11:08.932196+0000","last_became_active":"2024-09-13T17:11:08.512339+0000","last_became_peered":"2024-09-13T17:11:08.512339+0000","last_unstale":"2024-09-13T17:11:08.932196+0000","last_undegraded":"2024-09-13T17:11:08.932196+0000","last_fullsized":"2024-09-13T17:11:08.932196+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:04.874522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130618+0000","last_change":"2024-09-13T17:11:08.947717+0000","last_active":"2024-09-13T17:11:11.130618+0000","last_peered":"2024-09-13T17:11:11.130618+0000","last_clean":"2024-09-13T17:11:11.130618+0000","last_became_active":"2024-09-13T17:11:08.947413+0000","last_became_peered":"2024-09-13T17:11:08.947413+0000","last_unstale":"2024-09-13T17:11:11.130618+0000","last_undegraded":"2024-09-13T17:11:11.130618+0000","last_fullsized":"2024-09-13T17:11:11.130618+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:28:54.824131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928857+0000","last_change":"2024-09-13T17:11:08.509387+0000","last_active":"2024-09-13T17:11:08.928857+0000","last_peered":"2024-09-13T17:11:08.928857+0000","last_clean":"2024-09-13T17:11:08.928857+0000","last_became_active":"2024-09-13T17:11:08.509096+0000","last_became_peered":"2024-09-13T17:11:08.509096+0000","last_unstale":"2024-09-13T17:11:08.928857+0000","last_undegraded":"2024-09-13T17:11:08.928857+0000","last_fullsized":"2024-09-13T17:11:08.928857+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:35.565913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.130437+0000","last_change":"2024-09-13T17:11:08.954550+0000","last_active":"2024-09-13T17:11:11.130437+0000","last_peered":"2024-09-13T17:11:11.130437+0000","last_clean":"2024-09-13T17:11:11.130437+0000","last_became_active":"2024-09-13T17:11:08.953284+0000","last_became_peered":"2024-09-13T17:11:08.953284+0000","last_unstale":"2024-09-13T17:11:11.130437+0000","last_undegraded":"2024-09-13T17:11:11.130437+0000","last_fullsized":"2024-09-13T17:11:11.130437+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:39.600510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.933426+0000","last_change":"2024-09-13T17:11:08.510581+0000","last_active":"2024-09-13T17:11:08.933426+0000","last_peered":"2024-09-13T17:11:08.933426+0000","last_clean":"2024-09-13T17:11:08.933426+0000","last_became_active":"2024-09-13T17:11:08.509338+0000","last_became_peered":"2024-09-13T17:11:08.509338+0000","last_unstale":"2024-09-13T17:11:08.933426+0000","last_undegraded":"2024-09-13T17:11:08.933426+0000","last_fullsized":"2024-09-13T17:11:08.933426+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:04.874522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934229+0000","last_change":"2024-09-13T17:11:08.948768+0000","last_active":"2024-09-13T17:11:09.934229+0000","last_peered":"2024-09-13T17:11:09.934229+0000","last_clean":"2024-09-13T17:11:09.934229+0000","last_became_active":"2024-09-13T17:11:08.948467+0000","last_became_peered":"2024-09-13T17:11:08.948467+0000","last_unstale":"2024-09-13T17:11:09.934229+0000","last_undegraded":"2024-09-13T17:11:09.934229+0000","last_fullsized":"2024-09-13T17:11:09.934229+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:53.114410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.930666+0000","last_change":"2024-09-13T17:11:08.505124+0000","last_active":"2024-09-13T17:11:08.930666+0000","last_peered":"2024-09-13T17:11:08.930666+0000","last_clean":"2024-09-13T17:11:08.930666+0000","last_became_active":"2024-09-13T17:11:08.504849+0000","last_became_peered":"2024-09-13T17:11:08.504849+0000","last_unstale":"2024-09-13T17:11:08.930666+0000","last_undegraded":"2024-09-13T17:11:08.930666+0000","last_fullsized":"2024-09-13T17:11:08.930666+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:50.180934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:09.934260+0000","last_change":"2024-09-13T17:11:08.941857+0000","last_active":"2024-09-13T17:11:09.934260+0000","last_peered":"2024-09-13T17:11:09.934260+0000","last_clean":"2024-09-13T17:11:09.934260+0000","last_became_active":"2024-09-13T17:11:08.941341+0000","last_became_peered":"2024-09-13T17:11:08.941341+0000","last_unstale":"2024-09-13T17:11:09.934260+0000","last_undegraded":"2024-09-13T17:11:09.934260+0000","last_fullsized":"2024-09-13T17:11:09.934260+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:36.572460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.935655+0000","last_change":"2024-09-13T17:11:08.507664+0000","last_active":"2024-09-13T17:11:08.935655+0000","last_peered":"2024-09-13T17:11:08.935655+0000","last_clean":"2024-09-13T17:11:08.935655+0000","last_became_active":"2024-09-13T17:11:08.507326+0000","last_became_peered":"2024-09-13T17:11:08.507326+0000","last_unstale":"2024-09-13T17:11:08.935655+0000","last_undegraded":"2024-09-13T17:11:08.935655+0000","last_fullsized":"2024-09-13T17:11:08.935655+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:18.202120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T17:11:11.131688+0000","last_change":"2024-09-13T17:11:08.947814+0000","last_active":"2024-09-13T17:11:11.131688+0000","last_peered":"2024-09-13T17:11:11.131688+0000","last_clean":"2024-09-13T17:11:11.131688+0000","last_became_active":"2024-09-13T17:11:08.945042+0000","last_became_peered":"2024-09-13T17:11:08.945042+0000","last_unstale":"2024-09-13T17:11:11.131688+0000","last_undegraded":"2024-09-13T17:11:11.131688+0000","last_fullsized":"2024-09-13T17:11:11.131688+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:08.479475+0000","last_clean_scrub_stamp":"2024-09-13T17:11:08.479475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:07:23.681484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"47'2","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.958962+0000","last_change":"2024-09-13T17:11:08.505921+0000","last_active":"2024-09-13T17:11:08.958962+0000","last_peered":"2024-09-13T17:11:08.958962+0000","last_clean":"2024-09-13T17:11:08.958962+0000","last_became_active":"2024-09-13T17:11:08.505613+0000","last_became_peered":"2024-09-13T17:11:08.505613+0000","last_unstale":"2024-09-13T17:11:08.958962+0000","last_undegraded":"2024-09-13T17:11:08.958962+0000","last_fullsized":"2024-09-13T17:11:08.958962+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:11:07.472928+0000","last_clean_scrub_stamp":"2024-09-13T17:11:07.472928+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:05.419615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":187,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-13T17:11:08.928303+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:11:08.928303+0000","last_peered":"2024-09-13T17:11:08.928303+0000","last_clean":"2024-09-13T17:11:08.928303+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:11:08.928303+0000","last_undegraded":"2024-09-13T17:11:08.928303+0000","last_fullsized":"2024-09-13T17:11:08.928303+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574890,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40444,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331396,"statfs":{"total":96636764160,"available":96595349504,"internally_reserved":0,"allocated":1163264,"data_stored":823071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":40237515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.495,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.67900000000000005,"15min":0.68000000000000005},"min":{"1min":0.51200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.056,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.64000000000000001,"15min":0.63300000000000001},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.61099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.96099999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.53900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.60499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.64200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.59599999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.73799999999999999,"15min":0.73199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.78900000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.748,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.48299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.45000000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.198,"5min":1.337,"15min":1.337},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.54300000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.54400000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.61599999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35772,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336068,"statfs":{"total":96636764160,"available":96600133632,"internally_reserved":0,"allocated":573440,"data_stored":233285,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88400000000000001,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.47099999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.67900000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65900000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0129999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82699999999999996,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84199999999999997,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95499999999999996,"5min":3.742,"15min":3.742},"last":0.60599999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0349999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.72499999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.71999999999999997,"15min":0.73199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93600000000000005,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.0269999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.72799999999999998,"15min":0.73299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.0269999999999999,"5min":3.609,"15min":3.609},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.069,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.747,"5min":0.81100000000000005,"15min":0.80900000000000005},"min":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0069999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.103}]}]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35776,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336064,"statfs":{"total":96636764160,"available":96600129536,"internally_reserved":0,"allocated":569344,"data_stored":232683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.64000000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.004,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.82199999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.84399999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74199999999999999,"15min":0.75700000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.76700000000000002},"min":{"1min":0.56399999999999995,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93100000000000005,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.78000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.622,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.52600000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.749,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.61899999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39872,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331968,"statfs":{"total":96636764160,"available":96595935232,"internally_reserved":0,"allocated":577536,"data_stored":234177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64000000000000001,"15min":0.64100000000000001},"min":{"1min":0.46000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94299999999999995,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.746},"min":{"1min":0.48999999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.94199999999999995,"5min":1.341,"15min":1.341},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.51100000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85599999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.64500000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.43099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90100000000000002,"5min":1.377,"15min":1.377},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.47099999999999997,"5min":0.434,"15min":0.434},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.81499999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.65600000000000003},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.874,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.82699999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64100000000000001,"15min":0.65100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.82799999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72499999999999998,"15min":0.75},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":1.413,"15min":1.413},"last":0.84399999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94199999999999995,"5min":0.998,"15min":0.998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.54600000000000004,"5min":0.371,"15min":0.371},"max":{"1min":0.95899999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.86599999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45636,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326204,"statfs":{"total":96636764160,"available":96590032896,"internally_reserved":0,"allocated":577536,"data_stored":233251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.97099999999999997,"5min":2.96,"15min":2.96},"last":0.41399999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2809999999999999,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.73099999999999998,"15min":0.75700000000000001},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1830000000000001,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.376}]},{"osd":2,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.40600000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80600000000000005,"15min":0.83099999999999996},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3560000000000001,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.56399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.86699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83899999999999997,"15min":0.85499999999999998},"min":{"1min":0.61099999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.252,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78300000000000003,"15min":0.76800000000000002},"min":{"1min":0.623,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.244,"5min":1.401,"15min":1.401},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80500000000000005,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.75900000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.86599999999999999,"15min":0.88100000000000001},"min":{"1min":0.65400000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.4139999999999999,"5min":2.899,"15min":2.899},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.278,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.77300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35776,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94336064,"statfs":{"total":96636764160,"available":96600129536,"internally_reserved":0,"allocated":577536,"data_stored":233180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.41399999999999998,"15min":0.42399999999999999},"min":{"1min":0.30199999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.68999999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.438,"15min":0.42799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":13.476000000000001,"5min":3.077,"15min":1.3420000000000001},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":260.53399999999999,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":13.653,"5min":3.117,"15min":1.355},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":262.61399999999998,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.48999999999999999,"15min":0.48199999999999998},"min":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.73899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.48899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.05,"5min":1.121,"15min":1.121},"last":0.49099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.752,"15min":0.75800000000000001},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.77200000000000002,"15min":0.78400000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.79600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.44800000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77800000000000002},"min":{"1min":0.54700000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.569,"5min":1.569,"15min":1.569},"last":0.93300000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.76500000000000001,"15min":0.76300000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.413,"5min":1.413,"15min":1.413},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.79900000000000004,"15min":0.81100000000000005},"min":{"1min":0.57099999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.91500000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.77900000000000003,"15min":0.77800000000000002},"min":{"1min":0.56299999999999994,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.4730000000000001,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79100000000000004,"15min":0.80300000000000005},"min":{"1min":0.52600000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.5269999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.77200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45624,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326216,"statfs":{"total":96636764160,"available":96590045184,"internally_reserved":0,"allocated":569344,"data_stored":232683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48899999999999999,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.1779999999999999,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.437},{"interface":"front","average":{"1min":0.623,"5min":0.51500000000000001,"15min":0.497},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.48599999999999999,"15min":0.46800000000000003},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.1800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56200000000000006,"15min":0.57199999999999995},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.2350000000000001,"5min":1.49,"15min":1.49},"last":0.497}]},{"osd":3,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.66700000000000004,"15min":0.65800000000000003},"min":{"1min":0.46899999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.59799999999999998,"15min":0.56999999999999995},"min":{"1min":0.42499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":2.0299999999999998,"5min":2.0299999999999998,"15min":2.0299999999999998},"last":0.53600000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.54200000000000004,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":2.0920000000000001,"5min":2.0920000000000001,"15min":2.0920000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.83199999999999996,"15min":0.82399999999999995},"min":{"1min":0.61599999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.78,"5min":1.78,"15min":1.78},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.78200000000000003,"15min":0.77500000000000002},"min":{"1min":0.55300000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.754},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.80300000000000005,"15min":0.79600000000000004},"min":{"1min":0.59999999999999998,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.1400000000000001,"5min":2.1400000000000001,"15min":2.1400000000000001},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.82499999999999996,"15min":0.81999999999999995},"min":{"1min":0.59499999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.4710000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.82799999999999996,"15min":0.81899999999999995},"min":{"1min":0.59099999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.391,"5min":2.391,"15min":2.391},"last":0.63}]},{"osd":7,"last update":"Fri Sep 13 17:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.83499999999999996,"15min":0.83999999999999997},"min":{"1min":0.56399999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.8320000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.84299999999999997,"15min":0.83099999999999996},"min":{"1min":0.63200000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5049999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.67100000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40440,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331400,"statfs":{"total":96636764160,"available":96595353600,"internally_reserved":0,"allocated":1159168,"data_stored":821996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.65600000000000003,"15min":0.749},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.86899999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.64700000000000002,"15min":0.72899999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66300000000000003,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.68400000000000005,"15min":0.76600000000000001},"min":{"1min":0.376,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64200000000000002,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.70199999999999996,"15min":0.78400000000000003},"min":{"1min":0.38500000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68500000000000005,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.52100000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.64800000000000002,"15min":0.71399999999999997},"min":{"1min":0.36199999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.59399999999999997,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.70099999999999996,"15min":0.76300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80500000000000005,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.499}]},{"osd":4,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.88100000000000001,"15min":0.93700000000000006},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1779999999999999,"5min":4.734,"15min":4.734},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.89200000000000002,"15min":0.95799999999999996},"min":{"1min":0.59199999999999997,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.92300000000000004,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.79800000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.90700000000000003,"15min":0.97899999999999998},"min":{"1min":0.55000000000000004,"5min":0.504,"15min":0.504},"max":{"1min":0.98399999999999999,"5min":4.944,"15min":4.944},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.89400000000000002,"15min":0.95399999999999996},"min":{"1min":0.627,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.117,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":1.034,"15min":1.155},"min":{"1min":0.64200000000000002,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.242,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.92400000000000004,"15min":0.98499999999999999},"min":{"1min":0.54500000000000004,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.95299999999999996,"15min":1.0249999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93799999999999994,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.93200000000000005,"15min":1.002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.03,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.76200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:11:13.536 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:11:13.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:11:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.731+0000 7fb6bb200640 1 Processor -- start 2024-09-13T17:11:13.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.731+0000 7fb6bb200640 1 -- start start 2024-09-13T17:11:13.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b415a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6b4159b30 0x7fb6b40ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6b40b0a90 0x7fb6b40cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b4070480 con 0x7fb6b40b0a90 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6b415a010 con 0x7fb6b4159b30 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6bb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b406fb50 con 0x7fb6b4156650 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6b40b0a90 0x7fb6b40cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6ba800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6b40b0a90 0x7fb6b40cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50918/0 (socket says 172.21.15.28:50918) 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6ba800640 1 -- 172.21.15.28:0/3504713507 learned_addr learned my addr 172.21.15.28:0/3504713507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6b9400640 1 --2- 172.21.15.28:0/3504713507 >> v2:172.21.15.189:3300/0 conn(0x7fb6b4159b30 0x7fb6b40ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6b9e00640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b415a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6ba800640 1 -- 172.21.15.28:0/3504713507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 msgr2=0x7fb6b415a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.733+0000 7fb6ba800640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b415a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.734+0000 7fb6ba800640 1 -- 172.21.15.28:0/3504713507 >> v2:172.21.15.189:3300/0 conn(0x7fb6b4159b30 msgr2=0x7fb6b40ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.734+0000 7fb6ba800640 1 --2- 172.21.15.28:0/3504713507 >> v2:172.21.15.189:3300/0 conn(0x7fb6b4159b30 0x7fb6b40ccb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.734+0000 7fb6ba800640 1 -- 172.21.15.28:0/3504713507 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b406fe30 con 0x7fb6b40b0a90 2024-09-13T17:11:13.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6ba800640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40cd090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406aa10 tx=0x7fb6b002c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8764da6fbf24563 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6b8a00640 1 -- 172.21.15.28:0/3504713507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b002f070 con 0x7fb6b40b0a90 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6b8a00640 1 -- 172.21.15.28:0/3504713507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6b0035400 con 0x7fb6b40b0a90 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6b8a00640 1 -- 172.21.15.28:0/3504713507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6b0035790 con 0x7fb6b40b0a90 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 -- 172.21.15.28:0/3504713507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 msgr2=0x7fb6b40cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40cd090 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb6b406aa10 tx=0x7fb6b002c8a0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 -- 172.21.15.28:0/3504713507 shutdown_connections 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40cd090 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 --2- 172.21.15.28:0/3504713507 >> v2:172.21.15.189:3300/0 conn(0x7fb6b4159b30 0x7fb6b40ccb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 --2- 172.21.15.28:0/3504713507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b415a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.735+0000 7fb6bb200640 1 -- 172.21.15.28:0/3504713507 >> 172.21.15.28:0/3504713507 conn(0x7fb6b4069e80 msgr2=0x7fb6b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.736+0000 7fb6bb200640 1 -- 172.21.15.28:0/3504713507 shutdown_connections 2024-09-13T17:11:13.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.736+0000 7fb6bb200640 1 -- 172.21.15.28:0/3504713507 wait complete. 2024-09-13T17:11:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.737+0000 7fb6bb200640 1 Processor -- start 2024-09-13T17:11:13.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.737+0000 7fb6bb200640 1 -- start start 2024-09-13T17:11:13.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b414dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4159b30 0x7fb6b40acc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b414e810 con 0x7fb6b40b0a90 2024-09-13T17:11:13.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40af510 con 0x7fb6b4159b30 2024-09-13T17:11:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6bb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b40af7f0 con 0x7fb6b4156650 2024-09-13T17:11:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6b9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6b9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50934/0 (socket says 172.21.15.28:50934) 2024-09-13T17:11:13.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.738+0000 7fb6b9e00640 1 -- 172.21.15.28:0/1268638807 learned_addr learned my addr 172.21.15.28:0/1268638807 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 -- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 msgr2=0x7fb6b414dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b414dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 -- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4159b30 msgr2=0x7fb6b40acc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4159b30 0x7fb6b40acc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 -- 172.21.15.28:0/1268638807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b000bd40 con 0x7fb6b40b0a90 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6b9e00640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800d800 tx=0x7fb6a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800b6c0 con 0x7fb6b40b0a90 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6a800b820 con 0x7fb6b40b0a90 2024-09-13T17:11:13.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6a800bbb0 con 0x7fb6b40b0a90 2024-09-13T17:11:13.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.739+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b40afad0 con 0x7fb6b40b0a90 2024-09-13T17:11:13.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.740+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b40b0090 con 0x7fb6b40b0a90 2024-09-13T17:11:13.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.741+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b40af280 con 0x7fb6b40b0a90 2024-09-13T17:11:13.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.741+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb6a800bd10 con 0x7fb6b40b0a90 2024-09-13T17:11:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.742+0000 7fb6ae000640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 0x7fb69806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.742+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6a80899a0 con 0x7fb6b40b0a90 2024-09-13T17:11:13.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.743+0000 7fb6b9400640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 0x7fb69806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:13.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.744+0000 7fb6b9400640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 0x7fb69806d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb6b414e9a0 tx=0x7fb6a4009460 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:13.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.746+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb6a8054ae0 con 0x7fb6b40b0a90 2024-09-13T17:11:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.990+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb6b415a270 con 0x7fb6b40b0a90 2024-09-13T17:11:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.994+0000 7fb6ae000640 1 -- 172.21.15.28:0/1268638807 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7fb6a8058100 con 0x7fb6b40b0a90 2024-09-13T17:11:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:11:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 msgr2=0x7fb69806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 0x7fb69806d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb6b414e9a0 tx=0x7fb6a4009460 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 msgr2=0x7fb6b40aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb6a800d800 tx=0x7fb6a800dcc0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 shutdown_connections 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb69806ac00 0x7fb69806d0c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6b4159b30 0x7fb6b40acc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6b4156650 0x7fb6b414dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.996+0000 7fb6bb200640 1 --2- 172.21.15.28:0/1268638807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6b40b0a90 0x7fb6b40aece0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.997+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 >> 172.21.15.28:0/1268638807 conn(0x7fb6b4069e80 msgr2=0x7fb6b4154230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.997+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 shutdown_connections 2024-09-13T17:11:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:13.997+0000 7fb6bb200640 1 -- 172.21.15.28:0/1268638807 wait complete. 2024-09-13T17:11:14.015 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.169+0000 7f1fdb000640 1 Processor -- start 2024-09-13T17:11:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.169+0000 7f1fdb000640 1 -- start start 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fdb000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fd415e860 0x7f1fd4161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fdb000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd40713a0 0x7f1fd406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fdb000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fd40b35c0 con 0x7f1fd406cde0 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fdb000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fd41618f0 con 0x7f1fd415e860 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fdb000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd40b38a0 con 0x7f1fd40713a0 2024-09-13T17:11:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fd3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fd415e860 0x7f1fd4161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fd3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1fd415e860 0x7f1fd4161d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43878/0 (socket says 172.21.15.28:43878) 2024-09-13T17:11:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.170+0000 7f1fd3e00640 1 -- 172.21.15.28:0/3480862297 learned_addr learned my addr 172.21.15.28:0/3480862297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.171+0000 7f1fd9000640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd40713a0 0x7f1fd406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.171+0000 7f1fd3e00640 1 -- 172.21.15.28:0/3480862297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd40713a0 msgr2=0x7f1fd406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.171+0000 7f1fd3e00640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd40713a0 0x7f1fd406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.171+0000 7f1fd3e00640 1 -- 172.21.15.28:0/3480862297 >> v1:172.21.15.28:6789/0 conn(0x7f1fd406cde0 legacy=0x7f1fd4070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.171+0000 7f1fd3e00640 1 -- 172.21.15.28:0/3480862297 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd4161230 con 0x7f1fd415e860 2024-09-13T17:11:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.172+0000 7f1fd3e00640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4161d10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009870 tx=0x7f1fc402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5243f2818cbdb8ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.172+0000 7f1fd3400640 1 -- 172.21.15.28:0/3480862297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc402ae80 con 0x7f1fd415e860 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.172+0000 7f1fd3400640 1 -- 172.21.15.28:0/3480862297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc4032440 con 0x7f1fd415e860 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.172+0000 7f1fd3400640 1 -- 172.21.15.28:0/3480862297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc40327b0 con 0x7f1fd415e860 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 -- 172.21.15.28:0/3480862297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 msgr2=0x7f1fd4161d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4161d10 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4009870 tx=0x7f1fc402a130 comp rx=0 tx=0).stop 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 -- 172.21.15.28:0/3480862297 shutdown_connections 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd40713a0 0x7f1fd406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 --2- 172.21.15.28:0/3480862297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4161d10 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 -- 172.21.15.28:0/3480862297 >> 172.21.15.28:0/3480862297 conn(0x7f1fd4069e80 msgr2=0x7f1fd406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 -- 172.21.15.28:0/3480862297 shutdown_connections 2024-09-13T17:11:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.173+0000 7f1fdb000640 1 -- 172.21.15.28:0/3480862297 wait complete. 2024-09-13T17:11:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.174+0000 7f1fdb000640 1 Processor -- start 2024-09-13T17:11:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 -- start start 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd406cde0 0x7f1fd41652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fd40713a0 0x7f1fd4146640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4153d10 con 0x7f1fd415e860 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.175+0000 7f1fdb000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4146b80 con 0x7f1fd40713a0 2024-09-13T17:11:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fdb000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd4153f60 con 0x7f1fd406cde0 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd406cde0 0x7f1fd41652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43884/0 (socket says 172.21.15.28:43884) 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 -- 172.21.15.28:0/851232170 learned_addr learned my addr 172.21.15.28:0/851232170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fcb400640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fd40713a0 0x7f1fd4146640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 -- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd406cde0 msgr2=0x7f1fd41652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd406cde0 0x7f1fd41652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 -- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fd40713a0 msgr2=0x7f1fd4146640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fd40713a0 0x7f1fd4146640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.176+0000 7f1fd9000640 1 -- 172.21.15.28:0/851232170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc40094d0 con 0x7f1fd415e860 2024-09-13T17:11:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fd9000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc007e40 tx=0x7f1fcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc014620 con 0x7f1fd415e860 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fcc00ee90 con 0x7f1fd415e860 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fcc0149e0 con 0x7f1fd415e860 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd4154240 con 0x7f1fd415e860 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.177+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd4154800 con 0x7f1fd415e860 2024-09-13T17:11:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.178+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fd40b31c0 con 0x7f1fd415e860 2024-09-13T17:11:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.179+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1fcc014b40 con 0x7f1fd415e860 2024-09-13T17:11:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.180+0000 7f1fd0c00640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 0x7f1fb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.180+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fcc089cb0 con 0x7f1fd415e860 2024-09-13T17:11:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.180+0000 7f1fd3e00640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 0x7f1fb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.181+0000 7f1fd3e00640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 0x7f1fb006d0c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4034000 tx=0x7f1fc402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.184+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1fcc054e10 con 0x7f1fd415e860 2024-09-13T17:11:14.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.397+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fd415fcc0 con 0x7f1fd415e860 2024-09-13T17:11:14.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.397+0000 7f1fd0c00640 1 -- 172.21.15.28:0/851232170 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f1fcc058430 con 0x7f1fd415e860 2024-09-13T17:11:14.399 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:14.400 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:14.400 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 msgr2=0x7f1fb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 0x7f1fb006d0c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4034000 tx=0x7f1fc402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 msgr2=0x7f1fd4155000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc007e40 tx=0x7f1fcc00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 shutdown_connections 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1fb006ac00 0x7f1fb006d0c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1fd415e860 0x7f1fd4155000 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1fd40713a0 0x7f1fd4146640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 --2- 172.21.15.28:0/851232170 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1fd406cde0 0x7f1fd41652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 >> 172.21.15.28:0/851232170 conn(0x7f1fd4069e80 msgr2=0x7f1fd415ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 shutdown_connections 2024-09-13T17:11:14.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.400+0000 7f1fdb000640 1 -- 172.21.15.28:0/851232170 wait complete. 2024-09-13T17:11:14.477 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:11:14.475+0000 7f2042800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:11:14.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:11:08.920544+0000', 'modified': '2024-09-13T17:11:11.937604+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5580}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5580': {'gid': 5580, 'name': 'c', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6833/1182054705', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 1182054705}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 1182054705}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:11:14.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:11:14.821 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:11:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.978+0000 7f3d33000640 1 Processor -- start 2024-09-13T17:11:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.978+0000 7f3d33000640 1 -- start start 2024-09-13T17:11:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.979+0000 7f3d33000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.979+0000 7f3d33000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d2c071490 0x7f3d2c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.979+0000 7f3d33000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2c06c8b0 con 0x7f3d2c15a780 2024-09-13T17:11:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.979+0000 7f3d33000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2c06cb90 con 0x7f3d2c071490 2024-09-13T17:11:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.979+0000 7f3d33000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0cc980 con 0x7f3d2c15ab60 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d2be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d2c15a780 0x7f3d2c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50008/0 (socket says 172.21.15.28:50008) 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d31000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d2c071490 0x7f3d2c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d2be00640 1 -- 172.21.15.28:0/1311157774 learned_addr learned my addr 172.21.15.28:0/1311157774 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d2b400640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d31000640 1 -- 172.21.15.28:0/1311157774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 msgr2=0x7f3d2c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d31000640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d31000640 1 -- 172.21.15.28:0/1311157774 >> v1:172.21.15.28:6789/0 conn(0x7f3d2c15a780 legacy=0x7f3d2c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.980+0000 7f3d31000640 1 -- 172.21.15.28:0/1311157774 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d2c0ccc60 con 0x7f3d2c071490 2024-09-13T17:11:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.981+0000 7f3d31000640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c071490 0x7f3d2c06c080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d2402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1e26ada285aca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.981+0000 7f3d2aa00640 1 -- 172.21.15.28:0/1311157774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24007740 con 0x7f3d2c071490 2024-09-13T17:11:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.981+0000 7f3d2aa00640 1 -- 172.21.15.28:0/1311157774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2402d050 con 0x7f3d2c071490 2024-09-13T17:11:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.981+0000 7f3d2aa00640 1 -- 172.21.15.28:0/1311157774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24007ac0 con 0x7f3d2c071490 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 -- 172.21.15.28:0/1311157774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c071490 msgr2=0x7f3d2c06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c071490 0x7f3d2c06c080 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d2402a660 comp rx=0 tx=0).stop 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 -- 172.21.15.28:0/1311157774 shutdown_connections 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c071490 0x7f3d2c06c080 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 --2- 172.21.15.28:0/1311157774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 -- 172.21.15.28:0/1311157774 >> 172.21.15.28:0/1311157774 conn(0x7f3d2c0b0270 msgr2=0x7f3d2c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 -- 172.21.15.28:0/1311157774 shutdown_connections 2024-09-13T17:11:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.982+0000 7f3d33000640 1 -- 172.21.15.28:0/1311157774 wait complete. 2024-09-13T17:11:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.984+0000 7f3d33000640 1 Processor -- start 2024-09-13T17:11:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.984+0000 7f3d33000640 1 -- start start 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d2c071490 0x7f3d2c1450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0c5090 con 0x7f3d2c15a780 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0c4410 con 0x7f3d2c071490 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d33000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2c0c46f0 con 0x7f3d2c15ab60 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43904/0 (socket says 172.21.15.28:43904) 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d2b400640 1 -- 172.21.15.28:0/3732671529 learned_addr learned my addr 172.21.15.28:0/3732671529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d2c071490 0x7f3d2c1450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.985+0000 7f3d31000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 -- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 msgr2=0x7f3d2c154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 -- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d2c071490 msgr2=0x7f3d2c1450e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d2c071490 0x7f3d2c1450e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 -- 172.21.15.28:0/3732671529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d24009a70 con 0x7f3d2c15a780 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d2b400640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800d5f0 tx=0x7f3d1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d23e00640 1 -- 172.21.15.28:0/3732671529 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d180125f0 con 0x7f3d2c15a780 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d23e00640 1 -- 172.21.15.28:0/3732671529 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d18012750 con 0x7f3d2c15a780 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d23e00640 1 -- 172.21.15.28:0/3732671529 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18012b00 con 0x7f3d2c15a780 2024-09-13T17:11:14.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d2c0c49d0 con 0x7f3d2c15a780 2024-09-13T17:11:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.986+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c145620 con 0x7f3d2c15a780 2024-09-13T17:11:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.988+0000 7f3d23e00640 1 -- 172.21.15.28:0/3732671529 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3d18012c60 con 0x7f3d2c15a780 2024-09-13T17:11:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.989+0000 7f3d23e00640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 0x7f3d0c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.989+0000 7f3d23e00640 1 -- 172.21.15.28:0/3732671529 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d18089700 con 0x7f3d2c15a780 2024-09-13T17:11:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.990+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 0x7f3d0c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.990+0000 7f3d35e7b740 1 Processor -- start 2024-09-13T17:11:14.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.990+0000 7f3d35e7b740 1 -- start start 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 0x7f3d0c06d0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c007fd0 tx=0x7f3d1c007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f3eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f8c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153ea7ab0 con 0x55d153f3be90 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153ea5240 con 0x55d153f65b80 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.991+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d153ea0b30 con 0x55d153f3e6d0 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43908/0 (socket says 172.21.15.28:43908) 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 -- 172.21.15.28:0/336413590 learned_addr learned my addr 172.21.15.28:0/336413590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2be00640 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f3eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d31000640 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f8c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:14.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 -- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 msgr2=0x55d153f3eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f3eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 -- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 msgr2=0x55d153f8c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f8c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 -- 172.21.15.28:0/336413590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d153e6bbd0 con 0x55d153f3be90 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d2b400640 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d18058340 tx=0x7f3d18058640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cf07084c74c8a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d20c00640 1 -- 172.21.15.28:0/336413590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18060980 con 0x55d153f3be90 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d20c00640 1 -- 172.21.15.28:0/336413590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1805c170 con 0x55d153f3be90 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.992+0000 7f3d20c00640 1 -- 172.21.15.28:0/336413590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d18054d50 con 0x55d153f3be90 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 -- 172.21.15.28:0/336413590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 msgr2=0x55d153f8bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d18058340 tx=0x7f3d18058640 comp rx=0 tx=0).stop 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 -- 172.21.15.28:0/336413590 shutdown_connections 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f8c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f8bba0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/336413590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f3eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 -- 172.21.15.28:0/336413590 >> 172.21.15.28:0/336413590 conn(0x55d153f3ae00 msgr2=0x55d153f832b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 -- 172.21.15.28:0/336413590 shutdown_connections 2024-09-13T17:11:14.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.993+0000 7f3d35e7b740 1 -- 172.21.15.28:0/336413590 wait complete. 2024-09-13T17:11:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.994+0000 7f3d35e7b740 1 -- start start 2024-09-13T17:11:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.994+0000 7f3d35e7b740 1 Processor -- start 2024-09-13T17:11:15.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f8b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f672c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153e42260 con 0x55d153f67800 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153e41f00 con 0x55d153f3e6d0 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d153c07f20 con 0x55d153f65b80 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.995+0000 7f3d35e7b740 10 client.0 _ll_register_callbacks cb 0x55d153e9a1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d2b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f672c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43922/0 (socket says 172.21.15.28:43922) 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 -- 172.21.15.28:0/3187431171 learned_addr learned my addr 172.21.15.28:0/3187431171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f8b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 -- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 msgr2=0x55d153f672c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f672c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 -- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 msgr2=0x55d153f8b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f8b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 -- 172.21.15.28:0/3187431171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c145620 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d31000640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3d2402ac50 tx=0x7f3d24004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fca3f1a2ec12f6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.996+0000 7f3d15600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24042070 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d2403c420 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d153ecacd0 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d2403c5e0 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 10 client.5721 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55d153e9f1b0 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d153ecacd0 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3d24038040 con 0x55d153f67800 2024-09-13T17:11:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 10 client.5721 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 10 client.5721 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 20 client.5721 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.997+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153f87980 0x55d153f89070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.998+0000 7f3d35e7b740 4 client.5721 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.998+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d153b943c0 con 0x55d153f87980 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.998+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153f87980 0x55d153f89070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.998+0000 7f3d2be00640 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153f87980 0x55d153f89070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8561eaf3a9b2572 server_cookie=5f430cde58f8fc21 in_seq=0 out_seq=0 2024-09-13T17:11:15.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.998+0000 7f3d15600640 10 client.5721 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:15.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.999+0000 7f3d15600640 1 -- 172.21.15.28:0/3187431171 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55d153b943c0 con 0x55d153f87980 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.999+0000 7f3d15600640 10 client.5721 handle_command_reply: tid=0 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.999+0000 7f3d35e7b740 1 client.5721 shutdown 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.999+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 msgr2=0x55d153f63180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:14.999+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f3d2402ac50 tx=0x7f3d24004ab0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 shutdown_connections 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153f87980 0x55d153f89070 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153f63180 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f672c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d15600640 0 client.5721 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:15.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/3187431171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f8b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 >> 172.21.15.28:0/3187431171 conn(0x55d153f3be90 msgr2=0x55d153f83da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 shutdown_connections 2024-09-13T17:11:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 1 -- 172.21.15.28:0/3187431171 wait complete. 2024-09-13T17:11:15.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.000+0000 7f3d35e7b740 20 client.5721 trim_cache size 0 max 16384 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.015+0000 7f3d35e7b740 1 Processor -- start 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 -- start start 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f9c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f9c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153f34e30 con 0x55d153f3be90 2024-09-13T17:11:15.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153f86b40 con 0x55d153f65b80 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.016+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d153f9cce0 con 0x55d153f3e6d0 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43936/0 (socket says 172.21.15.28:43936) 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 -- 172.21.15.28:0/1869777908 learned_addr learned my addr 172.21.15.28:0/1869777908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d31000640 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f9c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 -- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 msgr2=0x55d153f9c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f9c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 -- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 msgr2=0x55d153f9c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f9c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 -- 172.21.15.28:0/1869777908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d153ed4170 con 0x55d153f3be90 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d2be00640 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c000f60 tx=0x7f3d1c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a7a4450184e021e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d21600640 1 -- 172.21.15.28:0/1869777908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c023880 con 0x55d153f3be90 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d21600640 1 -- 172.21.15.28:0/1869777908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0239e0 con 0x55d153f3be90 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.017+0000 7f3d21600640 1 -- 172.21.15.28:0/1869777908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c023b50 con 0x55d153f3be90 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 -- 172.21.15.28:0/1869777908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 msgr2=0x55d153f67180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c000f60 tx=0x7f3d1c013000 comp rx=0 tx=0).stop 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 -- 172.21.15.28:0/1869777908 shutdown_connections 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f65b80 0x55d153f9c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f3e6d0 0x55d153f9c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/1869777908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f3be90 0x55d153f67180 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 -- 172.21.15.28:0/1869777908 >> 172.21.15.28:0/1869777908 conn(0x55d153f34670 msgr2=0x55d153f85120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 -- 172.21.15.28:0/1869777908 shutdown_connections 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.018+0000 7f3d35e7b740 1 -- 172.21.15.28:0/1869777908 wait complete. 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.019+0000 7f3d35e7b740 1 -- start start 2024-09-13T17:11:15.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.019+0000 7f3d35e7b740 1 Processor -- start 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f89d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f9c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153fa3250 con 0x55d153f67800 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d153f9c8f0 con 0x55d153f3e6d0 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d153f8a260 con 0x55d153f65b80 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d35e7b740 10 client.0 _ll_register_callbacks cb 0x7f3d240079b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43946/0 (socket says 172.21.15.28:43946) 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d31000640 1 -- 172.21.15.28:0/2257954454 learned_addr learned my addr 172.21.15.28:0/2257954454 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d2be00640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f89d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.020+0000 7f3d2b400640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f9c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 -- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 msgr2=0x55d153f9c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f9c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 -- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 msgr2=0x55d153f89d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f89d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 -- 172.21.15.28:0/2257954454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2c06c8b0 con 0x55d153f67800 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d2be00640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f89d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d31000640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d24009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d429fbeed069c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d15600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24003da0 con 0x55d153f67800 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d2403c8d0 con 0x55d153f67800 2024-09-13T17:11:15.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d153f838b0 con 0x55d153f67800 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d2403ca90 con 0x55d153f67800 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d35e7b740 10 client.5727 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.021+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55d153e41470 con 0x55d153f67800 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d153f838b0 con 0x55d153f67800 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3d24038040 con 0x55d153f67800 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 10 client.5727 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 10 client.5727 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 20 client.5727 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153fa2660 0x55d153fa2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 4 client.5727 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f3d1805c170 con 0x55d153fa2660 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.022+0000 7f3d2be00640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153fa2660 0x55d153fa2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d2be00640 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153fa2660 0x55d153fa2a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=566f941d950d4849 server_cookie=d9fa248f8307700f in_seq=0 out_seq=0 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d15600640 10 client.5727 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d15600640 1 -- 172.21.15.28:0/2257954454 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x7f3d1805c170 con 0x55d153fa2660 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d15600640 10 client.5727 handle_command_reply: tid=0 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 client.5727 shutdown 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 msgr2=0x55d153fed880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d24009750 comp rx=0 tx=0).stop 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 shutdown_connections 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55d153fa2660 0x55d153fa2a40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d153f67800 0x55d153fed880 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d153f65b80 0x55d153f9c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 --2- 172.21.15.28:0/2257954454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d153f3e6d0 0x55d153f89d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 >> 172.21.15.28:0/2257954454 conn(0x55d153f34670 msgr2=0x55d153f83d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d15600640 0 client.5727 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 shutdown_connections 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 1 -- 172.21.15.28:0/2257954454 wait complete. 2024-09-13T17:11:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.023+0000 7f3d35e7b740 20 client.5727 trim_cache size 0 max 16384 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 5580, 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.101344653, 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 35, 2024-09-13T17:11:15.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 48, 2024-09-13T17:11:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 48, 2024-09-13T17:11:15.034 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.226574178 2024-09-13T17:11:15.034 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 msgr2=0x7f3d0c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 0x7f3d0c06d0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c007fd0 tx=0x7f3d1c007e80 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 msgr2=0x7f3d2c153ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3d1800d5f0 tx=0x7f3d1800dab0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 shutdown_connections 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d0c06ac00 0x7f3d0c06d0c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d2c15ab60 0x7f3d2c154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d2c15a780 0x7f3d2c153ed0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 --2- 172.21.15.28:0/3732671529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d2c071490 0x7f3d2c1450e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 >> 172.21.15.28:0/3732671529 conn(0x7f3d2c0b0270 msgr2=0x7f3d2c0acd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 shutdown_connections 2024-09-13T17:11:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.024+0000 7f3d33000640 1 -- 172.21.15.28:0/3732671529 wait complete. 2024-09-13T17:11:15.041 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:11:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.196+0000 7fdb83e00640 1 Processor -- start 2024-09-13T17:11:15.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.196+0000 7fdb83e00640 1 -- start start 2024-09-13T17:11:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb83e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdb7c1592a0 0x7fdb7c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb7c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb83e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb7c0b2040 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb83e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb7c15b0b0 con 0x7fdb7c1592a0 2024-09-13T17:11:15.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb83e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c159ff0 con 0x7fdb7c06f970 2024-09-13T17:11:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.197+0000 7fdb81e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdb7c06c2d0 0x7fdb7c0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50010/0 (socket says 172.21.15.28:50010) 2024-09-13T17:11:15.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb81e00640 1 -- 172.21.15.28:0/676784804 learned_addr learned my addr 172.21.15.28:0/676784804 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:15.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb81400640 1 --2- 172.21.15.28:0/676784804 >> v2:172.21.15.189:3300/0 conn(0x7fdb7c1592a0 0x7fdb7c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb80a00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb7c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996153216 0 0) 0x7fdb7c0b2040 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb64003520 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408604419 0 0) 0x7fdb64003520 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 msgr2=0x7fdb7c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb7c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 >> v2:172.21.15.189:3300/0 conn(0x7fdb7c1592a0 msgr2=0x7fdb7c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 --2- 172.21.15.28:0/676784804 >> v2:172.21.15.189:3300/0 conn(0x7fdb7c1592a0 0x7fdb7c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c15a2d0 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fdb740019f0 con 0x7fdb7c06c2d0 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 >> v1:172.21.15.28:6789/0 conn(0x7fdb7c06c2d0 legacy=0x7fdb7c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb64003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb64003ef0 0x7fdb640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64004d00 0x7fdb64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c15b0b0 con 0x7fdb64004d00 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c159ff0 con 0x7fdb64003ef0 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.198+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6400b980 con 0x7fdb7c06f970 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64004d00 0x7fdb64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81400640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb64003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb80a00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb64003ef0 0x7fdb640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 -- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 msgr2=0x7fdb64003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb7c06f970 0x7fdb64003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 -- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb64003ef0 msgr2=0x7fdb640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb64003ef0 0x7fdb640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.199+0000 7fdb81e00640 1 -- 172.21.15.28:0/676784804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb64003520 con 0x7fdb64004d00 2024-09-13T17:11:15.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.200+0000 7fdb81e00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64004d00 0x7fdb64005180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400a860 tx=0x7fdb640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5245e2088adf0af2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.200+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7402ba70 con 0x7fdb64004d00 2024-09-13T17:11:15.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.200+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7402bbd0 con 0x7fdb64004d00 2024-09-13T17:11:15.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.200+0000 7fdb7be00640 1 -- 172.21.15.28:0/676784804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74034470 con 0x7fdb64004d00 2024-09-13T17:11:15.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 -- 172.21.15.28:0/676784804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64004d00 msgr2=0x7fdb64005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 --2- 172.21.15.28:0/676784804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb64004d00 0x7fdb64005180 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400a860 tx=0x7fdb640057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb81e00640 1 -- 172.21.15.28:0/676784804 reap_dead start 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 -- 172.21.15.28:0/676784804 shutdown_connections 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 -- 172.21.15.28:0/676784804 >> 172.21.15.28:0/676784804 conn(0x7fdb7c069e80 msgr2=0x7fdb7c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 -- 172.21.15.28:0/676784804 shutdown_connections 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.201+0000 7fdb83e00640 1 -- 172.21.15.28:0/676784804 wait complete. 2024-09-13T17:11:15.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.203+0000 7fdb83e00640 1 Processor -- start 2024-09-13T17:11:15.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.203+0000 7fdb83e00640 1 -- start start 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 0x7fdb7c146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 0x7fdb7c0cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c0cea50 con 0x7fdb7c147020 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c149a90 con 0x7fdb7c0ceee0 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb83e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb7c0ce400 con 0x7fdb64004d00 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43956/0 (socket says 172.21.15.28:43956) 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 -- 172.21.15.28:0/2476076552 learned_addr learned my addr 172.21.15.28:0/2476076552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:15.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb80a00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 0x7fdb7c0cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81400640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 0x7fdb7c146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 -- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 msgr2=0x7fdb7c146950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 0x7fdb7c146950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 -- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 msgr2=0x7fdb7c0cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 0x7fdb7c0cc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81e00640 1 -- 172.21.15.28:0/2476076552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb7c0ce6e0 con 0x7fdb7c147020 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb81400640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 0x7fdb7c146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.204+0000 7fdb80a00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 0x7fdb7c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb81e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400a860 tx=0x7fdb74001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7403b040 con 0x7fdb7c147020 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb74034670 con 0x7fdb7c147020 2024-09-13T17:11:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb740349e0 con 0x7fdb7c147020 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb7c1496f0 con 0x7fdb7c147020 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.205+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c158440 con 0x7fdb7c147020 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.206+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb7c0cebc0 con 0x7fdb7c147020 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.207+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdb74031070 con 0x7fdb7c147020 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.208+0000 7fdb79600640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 0x7fdb5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.208+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb740a8ff0 con 0x7fdb7c147020 2024-09-13T17:11:15.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.209+0000 7fdb81400640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 0x7fdb5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:15.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.209+0000 7fdb81400640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 0x7fdb5c06d0c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009c40 tx=0x7fdb70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.213+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdb74074130 con 0x7fdb7c147020 2024-09-13T17:11:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.426+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdb7c0ce200 con 0x7fdb7c147020 2024-09-13T17:11:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.427+0000 7fdb79600640 1 -- 172.21.15.28:0/2476076552 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdb74045070 con 0x7fdb7c147020 2024-09-13T17:11:15.429 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 msgr2=0x7fdb5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 0x7fdb5c06d0c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdb70009c40 tx=0x7fdb70008040 comp rx=0 tx=0).stop 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 msgr2=0x7fdb7c157f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fdb7400a860 tx=0x7fdb74001d80 comp rx=0 tx=0).stop 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 shutdown_connections 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb5c06ac00 0x7fdb5c06d0c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb7c147020 0x7fdb7c157f00 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb7c0ceee0 0x7fdb7c0cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 --2- 172.21.15.28:0/2476076552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb64004d00 0x7fdb7c146950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 >> 172.21.15.28:0/2476076552 conn(0x7fdb7c069e80 msgr2=0x7fdb7c070060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 shutdown_connections 2024-09-13T17:11:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:15.429+0000 7fdb83e00640 1 -- 172.21.15.28:0/2476076552 wait complete. 2024-09-13T17:11:15.447 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:11:15.447 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:11:15.447 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:11:15.447 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:11:15.447 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:15.448 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:11:15.448 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:11:15.448 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:11:15.448 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:11:15.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.448 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:11:15.484 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42506sec preferred_lft 42506sec 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:11:15.485 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:11:15.486 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:11:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:11:15.487 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:11:15.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.487 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:11:15.578 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:11:15.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.581 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:11:15.617 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:11:15.617 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:11:15.617 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:11:15.617 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:11:15.618 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:11:15.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:11:15.618 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:11:15.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.763 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:11:15.798 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:11:15.799 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:11:15.799 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:11:15.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:15.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:11:15.895 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:11:15.895 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:15.930 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:11:15.930 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:11:15.930 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:16.006 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:11:16.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:16.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:11:16.134 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:11:16.134 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:11:16.163 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:11:16.163 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:11:16.163 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:11:16.163 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:11:16.163 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:11:16.165 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:11:16.167 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:11:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:11:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:11:16.168 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:11:16.169 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:11:16.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:16.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:16.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:16.225 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:11:16.269 INFO:teuthology.orchestra.run.smithi028.stdout:client5566 2024-09-13T17:11:16.269 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:16.269 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:16.345 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:16.346 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_test_path 2024-09-13T17:11:16.346 INFO:tasks.cephfs.test_scrub_checks:Cloning repo into place 2024-09-13T17:11:16.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:16.346 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p /home/ubuntu/cephtest/mnt.0/scrub_test_path) 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:11:16.427 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'"'"') 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-09-13T17:11:16.428 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:11:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:daemon-helper: command failed with exit status 2 2024-09-13T17:11:16.752 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T17:11:16.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:16.753 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec git clone --branch giant http://github.com/ceph/ceph-qa-suite /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite) 2024-09-13T17:11:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'... 2024-09-13T17:11:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:warning: redirecting to https://github.com/ceph/ceph-qa-suite/ 2024-09-13T17:11:20.264 INFO:teuthology.orchestra.run.smithi028.stderr:Updating files: 54% (760/1398) Updating files: 55% (769/1398) Updating files: 56% (783/1398) Updating files: 57% (797/1398) Updating files: 58% (811/1398) Updating files: 59% (825/1398) Updating files: 60% (839/1398) Updating files: 61% (853/1398) Updating files: 62% (867/1398) Updating files: 63% (881/1398) Updating files: 64% (895/1398) Updating files: 65% (909/1398) Updating files: 66% (923/1398) Updating files: 67% (937/1398) Updating files: 68% (951/1398) Updating files: 69% (965/1398) Updating files: 70% (979/1398) Updating files: 71% (993/1398) Updating files: 72% (1007/1398) Updating files: 73% (1021/1398) Updating files: 74% (1035/1398) Updating files: 75% (1049/1398) Updating files: 76% (1063/1398) Updating files: 77% (1077/1398) Updating files: 78% (1091/1398) Updating files: 79% (1105/1398) Updating files: 80% (1119/1398) Updating files: 81% (1133/1398) Updating files: 82% (1147/1398) Updating files: 83% (1161/1398) Updating files: 84% (1175/1398) Updating files: 84% (1180/1398) Updating files: 85% (1189/1398) Updating files: 86% (1203/1398) Updating files: 87% (1217/1398) Updating files: 88% (1231/1398) Updating files: 89% (1245/1398) Updating files: 90% (1259/1398) Updating files: 91% (1273/1398) Updating files: 92% (1287/1398) Updating files: 93% (1301/1398) Updating files: 94% (1315/1398) Updating files: 95% (1329/1398) Updating files: 96% (1343/1398) Updating files: 97% (1357/1398) Updating files: 98% (1371/1398) Updating files: 99% (1385/1398) Updating files: 100% (1398/1398) Updating files: 100% (1398/1398), done. 2024-09-13T17:11:20.271 INFO:tasks.cephfs.test_scrub_checks:Initiating mds_scrub_checks on mds.0 test_path /scrub_test_path, run_seq 0 2024-09-13T17:11:20.271 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/i/dont/exist' 2024-09-13T17:11:20.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.467+0000 7fa70da00640 1 Processor -- start 2024-09-13T17:11:20.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.467+0000 7fa70da00640 1 -- start start 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa708146170 0x7fa70815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa708160310 0x7fa708160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa7080ac850 0x7fa7080acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7080ad240 con 0x7fa708146170 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa70815cd20 con 0x7fa708160310 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa70da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa708146550 con 0x7fa7080ac850 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa706a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa708146170 0x7fa70815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa706a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa708146170 0x7fa70815d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43966/0 (socket says 172.21.15.28:43966) 2024-09-13T17:11:20.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.468+0000 7fa706a00640 1 -- 172.21.15.28:0/1280620399 learned_addr learned my addr 172.21.15.28:0/1280620399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706000640 1 --2- 172.21.15.28:0/1280620399 >> v2:172.21.15.189:3300/0 conn(0x7fa708160310 0x7fa708160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa707400640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa7080ac850 0x7fa7080acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706a00640 1 -- 172.21.15.28:0/1280620399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa7080ac850 msgr2=0x7fa7080acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706a00640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa7080ac850 0x7fa7080acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706a00640 1 -- 172.21.15.28:0/1280620399 >> v2:172.21.15.189:3300/0 conn(0x7fa708160310 msgr2=0x7fa708160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706a00640 1 --2- 172.21.15.28:0/1280620399 >> v2:172.21.15.189:3300/0 conn(0x7fa708160310 0x7fa708160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.469+0000 7fa706a00640 1 -- 172.21.15.28:0/1280620399 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa70815d280 con 0x7fa708146170 2024-09-13T17:11:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.470+0000 7fa706a00640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa708146170 0x7fa70815d710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa70806d550 tx=0x7fa6f802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5595567228a3293 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.470+0000 7fa705600640 1 -- 172.21.15.28:0/1280620399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8032440 con 0x7fa708146170 2024-09-13T17:11:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.470+0000 7fa705600640 1 -- 172.21.15.28:0/1280620399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f80325a0 con 0x7fa708146170 2024-09-13T17:11:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.470+0000 7fa705600640 1 -- 172.21.15.28:0/1280620399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8032930 con 0x7fa708146170 2024-09-13T17:11:20.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 -- 172.21.15.28:0/1280620399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa708146170 msgr2=0x7fa70815d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa708146170 0x7fa70815d710 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fa70806d550 tx=0x7fa6f802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 -- 172.21.15.28:0/1280620399 shutdown_connections 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa7080ac850 0x7fa7080acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 --2- 172.21.15.28:0/1280620399 >> v2:172.21.15.189:3300/0 conn(0x7fa708160310 0x7fa708160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 --2- 172.21.15.28:0/1280620399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa708146170 0x7fa70815d710 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 -- 172.21.15.28:0/1280620399 >> 172.21.15.28:0/1280620399 conn(0x7fa708069e80 msgr2=0x7fa70806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 -- 172.21.15.28:0/1280620399 shutdown_connections 2024-09-13T17:11:20.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.471+0000 7fa70da00640 1 -- 172.21.15.28:0/1280620399 wait complete. 2024-09-13T17:11:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.472+0000 7fa70da00640 1 Processor -- start 2024-09-13T17:11:20.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 -- start start 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa708146170 0x7fa7081369a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa708160310 0x7fa708136ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081352e0 con 0x7fa7080ac850 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa708156950 con 0x7fa708160310 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.473+0000 7fa70da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7081349b0 con 0x7fa708146170 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43976/0 (socket says 172.21.15.28:43976) 2024-09-13T17:11:20.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 -- 172.21.15.28:0/897434276 learned_addr learned my addr 172.21.15.28:0/897434276 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa707400640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa708160310 0x7fa708136ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706000640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa708146170 0x7fa7081369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 -- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa708146170 msgr2=0x7fa7081369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa708146170 0x7fa7081369a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 -- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa708160310 msgr2=0x7fa708136ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa708160310 0x7fa708136ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.474+0000 7fa706a00640 1 -- 172.21.15.28:0/897434276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6f80094d0 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa706a00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa6f802a220 tx=0x7fa6f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f8032440 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f8030ba0 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f803d440 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa708134c90 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.475+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa70813adf0 con 0x7fa7080ac850 2024-09-13T17:11:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.476+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7080ade10 con 0x7fa7080ac850 2024-09-13T17:11:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.477+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa6f8039040 con 0x7fa7080ac850 2024-09-13T17:11:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.478+0000 7fa6fea00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 0x7fa6e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.478+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6f80a9df0 con 0x7fa7080ac850 2024-09-13T17:11:20.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.478+0000 7fa706000640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 0x7fa6e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.479+0000 7fa706000640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 0x7fa6e806d0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7080cdf30 tx=0x7fa6f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:20.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.482+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa6f8075070 con 0x7fa7080ac850 2024-09-13T17:11:20.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.697+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa708134af0 con 0x7fa7080ac850 2024-09-13T17:11:20.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.698+0000 7fa6fea00640 1 -- 172.21.15.28:0/897434276 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fa708134af0 con 0x7fa7080ac850 2024-09-13T17:11:20.700 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:20.700 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:20.701 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:20.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 msgr2=0x7fa6e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:20.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 0x7fa6e806d0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7080cdf30 tx=0x7fa6f4008040 comp rx=0 tx=0).stop 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 msgr2=0x7fa70813a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa6f802a220 tx=0x7fa6f80046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 shutdown_connections 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6e806ac00 0x7fa6e806d0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa708160310 0x7fa708136ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa708146170 0x7fa7081369a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 --2- 172.21.15.28:0/897434276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa7080ac850 0x7fa70813a8b0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:20.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 >> 172.21.15.28:0/897434276 conn(0x7fa708069e80 msgr2=0x7fa7080af260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:20.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 shutdown_connections 2024-09-13T17:11:20.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:20.700+0000 7fa70da00640 1 -- 172.21.15.28:0/897434276 wait complete. 2024-09-13T17:11:20.717 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/i/dont/exist 2024-09-13T17:11:20.877 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:20.877 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": -2 2024-09-13T17:11:20.877 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:20.889 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/i/dont/exist' got response code '0' and stdout '{ "return_code": -2 } ' 2024-09-13T17:11:20.889 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/i/dont/exist' 2024-09-13T17:11:20.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/i/dont/exist 2024-09-13T17:11:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.047+0000 7f4192000640 1 Processor -- start 2024-09-13T17:11:21.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.048+0000 7f4192000640 1 -- start start 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4192000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f418c143f70 0x7f418c0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4192000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f418c070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4192000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418c15ef70 con 0x7f418c0cd5f0 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4192000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418c0cda80 con 0x7f418c143f70 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4192000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0c4320 con 0x7f418c0c3d70 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4183e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f418c0cd5f0 0x7f418c0c3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50016/0 (socket says 172.21.15.28:50016) 2024-09-13T17:11:21.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4183e00640 1 -- 172.21.15.28:0/1770319187 learned_addr learned my addr 172.21.15.28:0/1770319187 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:21.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.049+0000 7f4191600640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f418c070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f4190c00640 1 --2- 172.21.15.28:0/1770319187 >> v2:172.21.15.189:3300/0 conn(0x7f418c143f70 0x7f418c0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739443471 0 0) 0x7f418c15ef70 con 0x7f418c0cd5f0 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4170003520 con 0x7f418c0cd5f0 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 196949323 0 0) 0x7f4170003520 con 0x7f418c0cd5f0 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 msgr2=0x7f418c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f418c070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 >> v2:172.21.15.189:3300/0 conn(0x7f418c143f70 msgr2=0x7f418c0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 --2- 172.21.15.28:0/1770319187 >> v2:172.21.15.189:3300/0 conn(0x7f418c143f70 0x7f418c0cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c15e1b0 con 0x7f418c0cd5f0 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4174001670 con 0x7f418c0cd5f0 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 >> v1:172.21.15.28:6789/0 conn(0x7f418c0cd5f0 legacy=0x7f418c0c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f4170003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170003ef0 0x7f41700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170004d00 0x7f4170005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0cda80 con 0x7f4170003ef0 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0c4320 con 0x7f4170004d00 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f417000b980 con 0x7f418c0c3d70 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f4183e00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170003ef0 0x7f41700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.050+0000 7f4190c00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f4170003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4191600640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170004d00 0x7f4170005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4183e00640 1 -- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 msgr2=0x7f4170003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4183e00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0c3d70 0x7f4170003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4183e00640 1 -- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170004d00 msgr2=0x7f4170005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4183e00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170004d00 0x7f4170005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.051+0000 7f4183e00640 1 -- 172.21.15.28:0/1770319187 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170003520 con 0x7f4170003ef0 2024-09-13T17:11:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.052+0000 7f4183e00640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170003ef0 0x7f41700047c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f417400c240 tx=0x7f41700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8e98dfe573901d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.052+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174005ad0 con 0x7f4170003ef0 2024-09-13T17:11:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.052+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417402e070 con 0x7f4170003ef0 2024-09-13T17:11:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.052+0000 7f418be00640 1 -- 172.21.15.28:0/1770319187 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174007070 con 0x7f4170003ef0 2024-09-13T17:11:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 -- 172.21.15.28:0/1770319187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170003ef0 msgr2=0x7f41700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 --2- 172.21.15.28:0/1770319187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4170003ef0 0x7f41700047c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f417400c240 tx=0x7f41700057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4191600640 1 -- 172.21.15.28:0/1770319187 reap_dead start 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 -- 172.21.15.28:0/1770319187 shutdown_connections 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 -- 172.21.15.28:0/1770319187 >> 172.21.15.28:0/1770319187 conn(0x7f418c069e80 msgr2=0x7f418c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 -- 172.21.15.28:0/1770319187 shutdown_connections 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.053+0000 7f4192000640 1 -- 172.21.15.28:0/1770319187 wait complete. 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.054+0000 7f4192000640 1 Processor -- start 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 -- start start 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170003ef0 0x7f418c15b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 0x7f418c0a89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c15b570 con 0x7f418c0aacd0 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0ab670 con 0x7f4170003ef0 2024-09-13T17:11:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.055+0000 7f4192000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418c0a92c0 con 0x7f418c0a8560 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43988/0 (socket says 172.21.15.28:43988) 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 -- 172.21.15.28:0/220567560 learned_addr learned my addr 172.21.15.28:0/220567560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4191600640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 0x7f418c0a89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4190c00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170003ef0 0x7f418c15b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 -- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 msgr2=0x7f418c0a89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 0x7f418c0a89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 -- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170003ef0 msgr2=0x7f418c15b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170003ef0 0x7f418c15b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 -- 172.21.15.28:0/220567560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418c0a95a0 con 0x7f418c0aacd0 2024-09-13T17:11:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4191600640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 0x7f418c0a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.056+0000 7f4183e00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f417402ad30 tx=0x7f4174001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.057+0000 7f4189600640 1 -- 172.21.15.28:0/220567560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174001670 con 0x7f418c0aacd0 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.057+0000 7f4189600640 1 -- 172.21.15.28:0/220567560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41740049f0 con 0x7f418c0aacd0 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.057+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c0a9880 con 0x7f418c0aacd0 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.057+0000 7f4189600640 1 -- 172.21.15.28:0/220567560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4174004d60 con 0x7f418c0aacd0 2024-09-13T17:11:21.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.057+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f418c0a9e40 con 0x7f418c0aacd0 2024-09-13T17:11:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.059+0000 7f4189600640 1 -- 172.21.15.28:0/220567560 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4174030030 con 0x7f418c0aacd0 2024-09-13T17:11:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.059+0000 7f4189600640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.060+0000 7f4189600640 1 -- 172.21.15.28:0/220567560 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f41740a8350 con 0x7f418c0aacd0 2024-09-13T17:11:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.060+0000 7f4190c00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.060+0000 7f41965f6740 1 Processor -- start 2024-09-13T17:11:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 -- start start 2024-09-13T17:11:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f4190c00640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 0x7f416806d0f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f417c009bc0 tx=0x7f417c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3125410 0x5566d3175120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d3175660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d30e2a60 con 0x5566d3127c50 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3090df0 con 0x5566d314f100 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.061+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566d308e580 con 0x5566d3125410 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44002/0 (socket says 172.21.15.28:44002) 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 -- 172.21.15.28:0/2102479850 learned_addr learned my addr 172.21.15.28:0/2102479850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4191600640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d3175660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4183e00640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3125410 0x5566d3175120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 -- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3125410 msgr2=0x5566d3175120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3125410 0x5566d3175120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 -- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 msgr2=0x5566d3175660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d3175660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 -- 172.21.15.28:0/2102479850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566d30510e0 con 0x5566d3127c50 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4191600640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d3175660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.062+0000 7f4190c00640 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f417c0154a0 tx=0x7f417c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a03b1e6c91bb2a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f4181600640 1 -- 172.21.15.28:0/2102479850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c02a430 con 0x5566d3127c50 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f4181600640 1 -- 172.21.15.28:0/2102479850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c025070 con 0x5566d3127c50 2024-09-13T17:11:21.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f4181600640 1 -- 172.21.15.28:0/2102479850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c02a790 con 0x5566d3127c50 2024-09-13T17:11:21.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 -- 172.21.15.28:0/2102479850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 msgr2=0x5566d3128030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f417c0154a0 tx=0x7f417c017000 comp rx=0 tx=0).stop 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 -- 172.21.15.28:0/2102479850 shutdown_connections 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d3175660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3125410 0x5566d3175120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 --2- 172.21.15.28:0/2102479850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3128030 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 -- 172.21.15.28:0/2102479850 >> 172.21.15.28:0/2102479850 conn(0x5566d3124380 msgr2=0x5566d316c830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.063+0000 7f41965f6740 1 -- 172.21.15.28:0/2102479850 shutdown_connections 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.064+0000 7f41965f6740 1 -- 172.21.15.28:0/2102479850 wait complete. 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.064+0000 7f41965f6740 1 -- start start 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.064+0000 7f41965f6740 1 Processor -- start 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d313b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d314dcc0 0x5566d316ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3089e70 con 0x5566d3127c50 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d302b1d0 con 0x5566d314f100 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566d302ae70 con 0x5566d314dcc0 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.065+0000 7f41965f6740 10 client.0 _ll_register_callbacks cb 0x5566d30508c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44008/0 (socket says 172.21.15.28:44008) 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 -- 172.21.15.28:0/287194370 learned_addr learned my addr 172.21.15.28:0/287194370 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4191600640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d314dcc0 0x5566d316ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4183e00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d313b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 -- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d314dcc0 msgr2=0x5566d316ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d314dcc0 0x5566d316ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 -- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 msgr2=0x5566d313b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d313b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 -- 172.21.15.28:0/287194370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417c014d50 con 0x5566d3127c50 2024-09-13T17:11:21.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4183e00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d313b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f4190c00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f417c0042f0 tx=0x7f417c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e20ac036577aeb9b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.066+0000 7f417a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c003820 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f417c003980 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566d307bd10 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f417c003b40 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 10 client.5751 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x5566d30884f0 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5566d307bd10 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f417c01d040 con 0x5566d3127c50 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 10 client.5751 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 10 client.5751 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:21.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 20 client.5751 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d316b2a0 0x5566d314e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 4 client.5751 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.067+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566d2d7d6d0 con 0x5566d316b2a0 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.068+0000 7f4183e00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d316b2a0 0x5566d314e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.068+0000 7f4183e00640 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d316b2a0 0x5566d314e0a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6baf5dfb5dca93e7 server_cookie=e53577051d5fb3b1 in_seq=0 out_seq=0 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.068+0000 7f417a000640 10 client.5751 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.068+0000 7f417a000640 1 -- 172.21.15.28:0/287194370 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5566d2d7d6d0 con 0x5566d316b2a0 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.068+0000 7f417a000640 10 client.5751 handle_command_reply: tid=0 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 client.5751 shutdown 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 msgr2=0x5566d3139030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f417c0042f0 tx=0x7f417c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 shutdown_connections 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d314dcc0 0x5566d316ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314f100 0x5566d313b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d316b2a0 0x5566d314e0a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 --2- 172.21.15.28:0/287194370 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d3127c50 0x5566d3139030 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 >> 172.21.15.28:0/287194370 conn(0x5566d3125410 msgr2=0x5566d3170510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f417a000640 0 client.5751 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 shutdown_connections 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 1 -- 172.21.15.28:0/287194370 wait complete. 2024-09-13T17:11:21.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.069+0000 7f41965f6740 20 client.5751 trim_cache size 0 max 16384 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.084+0000 7f41965f6740 1 Processor -- start 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.084+0000 7f41965f6740 1 -- start start 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d31734f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d316c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d2df1230 con 0x5566d314f100 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3127650 con 0x5566d314dcc0 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566d30433a0 con 0x5566d3127c50 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f4191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f4191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44012/0 (socket says 172.21.15.28:44012) 2024-09-13T17:11:21.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.085+0000 7f4191600640 1 -- 172.21.15.28:0/2024685499 learned_addr learned my addr 172.21.15.28:0/2024685499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4183e00640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d316c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4190c00640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d31734f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 -- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 msgr2=0x5566d31734f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d31734f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 -- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 msgr2=0x5566d316c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d316c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 -- 172.21.15.28:0/2024685499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566d302a370 con 0x5566d314f100 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4183e00640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d316c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:21.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4191600640 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f418c15ba80 tx=0x7f4184009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25280b43bb57756c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4182000640 1 -- 172.21.15.28:0/2024685499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418400e040 con 0x5566d314f100 2024-09-13T17:11:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4182000640 1 -- 172.21.15.28:0/2024685499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418400f040 con 0x5566d314f100 2024-09-13T17:11:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.086+0000 7f4182000640 1 -- 172.21.15.28:0/2024685499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184004030 con 0x5566d314f100 2024-09-13T17:11:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 -- 172.21.15.28:0/2024685499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 msgr2=0x5566d317eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f418c15ba80 tx=0x7f4184009750 comp rx=0 tx=0).stop 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 -- 172.21.15.28:0/2024685499 shutdown_connections 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d317eae0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d316c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 --2- 172.21.15.28:0/2024685499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d31734f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 -- 172.21.15.28:0/2024685499 >> 172.21.15.28:0/2024685499 conn(0x5566d3125410 msgr2=0x5566d316a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 -- 172.21.15.28:0/2024685499 shutdown_connections 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.087+0000 7f41965f6740 1 -- 172.21.15.28:0/2024685499 wait complete. 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.088+0000 7f41965f6740 1 -- start start 2024-09-13T17:11:21.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.088+0000 7f41965f6740 1 Processor -- start 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d3137ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d3137ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3189cc0 con 0x5566d314f100 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566d3169700 con 0x5566d314dcc0 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5566d318ee30 con 0x5566d3127c50 2024-09-13T17:11:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f41965f6740 10 client.0 _ll_register_callbacks cb 0x7f417c014f70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f4191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f4191600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44022/0 (socket says 172.21.15.28:44022) 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f4191600640 1 -- 172.21.15.28:0/2704655602 learned_addr learned my addr 172.21.15.28:0/2704655602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f4183e00640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d3137ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.089+0000 7f4190c00640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d3137ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 -- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 msgr2=0x5566d3137ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d3137ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 -- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 msgr2=0x5566d3137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d3137ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 -- 172.21.15.28:0/2704655602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4184009e50 con 0x5566d314f100 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f4191600640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x5566d3173a30 tx=0x7f4184004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9ee2569fc3734d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f417a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184026070 con 0x5566d314f100 2024-09-13T17:11:21.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f418402b470 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f417c025070 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f418400e040 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f41965f6740 10 client.5757 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.090+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x5566d304f940 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c025070 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4184014040 con 0x5566d314f100 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 10 client.5757 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 10 client.5757 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 20 client.5757 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d31734f0 0x5566d3188ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 4 client.5757 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}] 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}) v1 -- 0x5566d2d8e9d0 con 0x5566d31734f0 2024-09-13T17:11:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.091+0000 7f4190c00640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d31734f0 0x5566d3188ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f4190c00640 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d31734f0 0x5566d3188ec0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34fc4e646d0cd033 server_cookie=ebff771cf1d00998 in_seq=0 out_seq=0 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f417a000640 10 client.5757 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f417a000640 1 -- 172.21.15.28:0/2704655602 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26 (crc 0 0 0) 0x5566d2d8e9d0 con 0x5566d31734f0 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f417a000640 10 client.5757 handle_command_reply: tid=0 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 client.5757 shutdown 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 msgr2=0x5566d3188980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x5566d3173a30 tx=0x7f4184004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 shutdown_connections 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5566d31734f0 0x5566d3188ec0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d314f100 0x5566d3188980 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5566d314dcc0 0x5566d3137ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 --2- 172.21.15.28:0/2704655602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5566d3127c50 0x5566d3137ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f417a000640 0 client.5757 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 >> 172.21.15.28:0/2704655602 conn(0x5566d3125410 msgr2=0x5566d314c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 shutdown_connections 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.092+0000 7f41965f6740 1 -- 172.21.15.28:0/2704655602 wait complete. 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f41965f6740 20 client.5757 trim_cache size 0 max 16384 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": -2 2024-09-13T17:11:21.098 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 msgr2=0x7f416806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 0x7f416806d0f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f417c009bc0 tx=0x7f417c009190 comp rx=0 tx=0).stop 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 msgr2=0x7f418c0ab130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f417402ad30 tx=0x7f4174001a00 comp rx=0 tx=0).stop 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 shutdown_connections 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f418c0a8560 0x7f418c0a89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f418c0aacd0 0x7f418c0ab130 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 --2- 172.21.15.28:0/220567560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4170003ef0 0x7f418c15b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 >> 172.21.15.28:0/220567560 conn(0x7f418c069e80 msgr2=0x7f418c0cb210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 shutdown_connections 2024-09-13T17:11:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.093+0000 7f4192000640 1 -- 172.21.15.28:0/220567560 wait complete. 2024-09-13T17:11:21.108 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/i/dont/exist' returned '{'return_code': -2}' 2024-09-13T17:11:21.109 INFO:tasks.cephfs.test_scrub_checks:First run: flushing /scrub_test_path/ceph-qa-suite/suites 2024-09-13T17:11:21.109 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/suites' 2024-09-13T17:11:21.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.263+0000 7f3ae9000640 1 Processor -- start 2024-09-13T17:11:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.263+0000 7f3ae9000640 1 -- start start 2024-09-13T17:11:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ae415a540 0x7f3ae406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae9000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 0x7f3ae415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae9000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae415e100 con 0x7f3ae415edb0 2024-09-13T17:11:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae9000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae40adb00 con 0x7f3ae415a540 2024-09-13T17:11:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae9000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae415e3e0 con 0x7f3ae40cabc0 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ae415a540 0x7f3ae406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ae415a540 0x7f3ae406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44026/0 (socket says 172.21.15.28:44026) 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.264+0000 7f3ae1e00640 1 -- 172.21.15.28:0/1760352325 learned_addr learned my addr 172.21.15.28:0/1760352325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1e00640 1 -- 172.21.15.28:0/1760352325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 msgr2=0x7f3ae415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1400640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 0x7f3ae415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1e00640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 0x7f3ae415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1e00640 1 -- 172.21.15.28:0/1760352325 >> v1:172.21.15.28:6789/0 conn(0x7f3ae415edb0 legacy=0x7f3ae4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1e00640 1 -- 172.21.15.28:0/1760352325 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae406ce80 con 0x7f3ae415a540 2024-09-13T17:11:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.265+0000 7f3ae1400640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 0x7f3ae415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.266+0000 7f3ae1e00640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae415a540 0x7f3ae406c430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009870 tx=0x7f3ad402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f9a408171d436c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.266+0000 7f3ae0a00640 1 -- 172.21.15.28:0/1760352325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4032440 con 0x7f3ae415a540 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.266+0000 7f3ae0a00640 1 -- 172.21.15.28:0/1760352325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad40325a0 con 0x7f3ae415a540 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.266+0000 7f3ae0a00640 1 -- 172.21.15.28:0/1760352325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4032930 con 0x7f3ae415a540 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 -- 172.21.15.28:0/1760352325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae415a540 msgr2=0x7f3ae406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae415a540 0x7f3ae406c430 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4009870 tx=0x7f3ad402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 -- 172.21.15.28:0/1760352325 shutdown_connections 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae40cabc0 0x7f3ae415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 --2- 172.21.15.28:0/1760352325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae415a540 0x7f3ae406c430 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 -- 172.21.15.28:0/1760352325 >> 172.21.15.28:0/1760352325 conn(0x7f3ae4069e80 msgr2=0x7f3ae406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 -- 172.21.15.28:0/1760352325 shutdown_connections 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.267+0000 7f3ae9000640 1 -- 172.21.15.28:0/1760352325 wait complete. 2024-09-13T17:11:21.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.268+0000 7f3ae9000640 1 Processor -- start 2024-09-13T17:11:21.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.268+0000 7f3ae9000640 1 -- start start 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 0x7f3ae414cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae415d880 0x7f3ae414e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41f5cf0 con 0x7f3ae40cabc0 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae414d4a0 con 0x7f3ae415d880 2024-09-13T17:11:21.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.269+0000 7f3ae9000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41f2b90 con 0x7f3ae415a540 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44032/0 (socket says 172.21.15.28:44032) 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 -- 172.21.15.28:0/375313390 learned_addr learned my addr 172.21.15.28:0/375313390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae2800640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae415d880 0x7f3ae414e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1400640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 0x7f3ae414cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 -- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 msgr2=0x7f3ae414cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 0x7f3ae414cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 -- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae415d880 msgr2=0x7f3ae414e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae415d880 0x7f3ae414e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 -- 172.21.15.28:0/375313390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad40094d0 con 0x7f3ae40cabc0 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1400640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 0x7f3ae414cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:21.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.270+0000 7f3ae1e00640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4000c00 tx=0x7f3ad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.271+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4030b80 con 0x7f3ae40cabc0 2024-09-13T17:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.271+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad4030d00 con 0x7f3ae40cabc0 2024-09-13T17:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.271+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad403d680 con 0x7f3ae40cabc0 2024-09-13T17:11:21.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.271+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae41f2e70 con 0x7f3ae40cabc0 2024-09-13T17:11:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.271+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae41f3430 con 0x7f3ae40cabc0 2024-09-13T17:11:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.272+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae415e870 con 0x7f3ae40cabc0 2024-09-13T17:11:21.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.273+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3ad4039040 con 0x7f3ae40cabc0 2024-09-13T17:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.274+0000 7f3ada000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 0x7f3ac406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.274+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ad40aa070 con 0x7f3ae40cabc0 2024-09-13T17:11:21.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.274+0000 7f3ae1400640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 0x7f3ac406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.275+0000 7f3ae1400640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 0x7f3ac406d0c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ae414fae0 tx=0x7f3ad0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.278+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3ad40dd9c0 con 0x7f3ae40cabc0 2024-09-13T17:11:21.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.492+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ae40ce190 con 0x7f3ae40cabc0 2024-09-13T17:11:21.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.492+0000 7f3ada000640 1 -- 172.21.15.28:0/375313390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f3ad40752f0 con 0x7f3ae40cabc0 2024-09-13T17:11:21.494 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:21.494 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:21.494 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 msgr2=0x7f3ac406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 0x7f3ac406d0c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3ae414fae0 tx=0x7f3ad0008040 comp rx=0 tx=0).stop 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 msgr2=0x7f3ae414bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3ad4000c00 tx=0x7f3ad40046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 shutdown_connections 2024-09-13T17:11:21.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ac406ac00 0x7f3ac406d0c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ae415d880 0x7f3ae414e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ae415a540 0x7f3ae414cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 --2- 172.21.15.28:0/375313390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ae40cabc0 0x7f3ae414bf20 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 >> 172.21.15.28:0/375313390 conn(0x7f3ae4069e80 msgr2=0x7f3ae40ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 shutdown_connections 2024-09-13T17:11:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.495+0000 7f3ae9000640 1 -- 172.21.15.28:0/375313390 wait complete. 2024-09-13T17:11:21.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/suites 2024-09-13T17:11:21.786 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:21.786 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:21.786 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:21.798 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/suites' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:21.798 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites' 2024-09-13T17:11:21.799 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites 2024-09-13T17:11:21.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.957+0000 7f744fa00640 1 Processor -- start 2024-09-13T17:11:21.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.957+0000 7f744fa00640 1 -- start start 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7448071490 0x7f744806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744fa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f744806c8b0 con 0x7f744815a780 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f744806cb90 con 0x7f7448071490 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74480cc980 con 0x7f744815ab60 2024-09-13T17:11:21.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7448071490 0x7f744806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7448071490 0x7f744806c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44048/0 (socket says 172.21.15.28:44048) 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 -- 172.21.15.28:0/2719025137 learned_addr learned my addr 172.21.15.28:0/2719025137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f743fe00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 -- 172.21.15.28:0/2719025137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 msgr2=0x7f744806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744806bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 -- 172.21.15.28:0/2719025137 >> v1:172.21.15.189:6789/0 conn(0x7f744815a780 legacy=0x7f7448070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.959+0000 7f744da00640 1 -- 172.21.15.28:0/2719025137 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74480ccc60 con 0x7f7448071490 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.960+0000 7f744da00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f744806c080 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7440009e50 tx=0x7f744002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c094a1d1b9c85116 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f7447e00640 1 -- 172.21.15.28:0/2719025137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440031600 con 0x7f7448071490 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f7447e00640 1 -- 172.21.15.28:0/2719025137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744002d050 con 0x7f7448071490 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f7447e00640 1 -- 172.21.15.28:0/2719025137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7440031980 con 0x7f7448071490 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 -- 172.21.15.28:0/2719025137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 msgr2=0x7f744806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f744806c080 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f7440009e50 tx=0x7f744002a710 comp rx=0 tx=0).stop 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 -- 172.21.15.28:0/2719025137 shutdown_connections 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f744806c080 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 --2- 172.21.15.28:0/2719025137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744806bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 -- 172.21.15.28:0/2719025137 >> 172.21.15.28:0/2719025137 conn(0x7f74480b0270 msgr2=0x7f74480b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 -- 172.21.15.28:0/2719025137 shutdown_connections 2024-09-13T17:11:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.961+0000 7f744fa00640 1 -- 172.21.15.28:0/2719025137 wait complete. 2024-09-13T17:11:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.963+0000 7f744fa00640 1 Processor -- start 2024-09-13T17:11:21.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.963+0000 7f744fa00640 1 -- start start 2024-09-13T17:11:21.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 0x7f744814e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744814fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7448147520 con 0x7f7448071490 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744814f7f0 con 0x7f744815a780 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744814e940 con 0x7f744815ab60 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44054/0 (socket says 172.21.15.28:44054) 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744d000640 1 -- 172.21.15.28:0/1789488103 learned_addr learned my addr 172.21.15.28:0/1789488103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f743fe00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 0x7f744814e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.964+0000 7f744da00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744814fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 -- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 msgr2=0x7f744814fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744814fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 -- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 msgr2=0x7f744814e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 0x7f744814e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 -- 172.21.15.28:0/1789488103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7440009b30 con 0x7f7448071490 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f743fe00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 0x7f744814e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744d000640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f743800c450 tx=0x7f743800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f7445600640 1 -- 172.21.15.28:0/1789488103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380052d0 con 0x7f7448071490 2024-09-13T17:11:21.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f7445600640 1 -- 172.21.15.28:0/1789488103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438005430 con 0x7f7448071490 2024-09-13T17:11:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f7445600640 1 -- 172.21.15.28:0/1789488103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438003830 con 0x7f7448071490 2024-09-13T17:11:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744814ebe0 con 0x7f7448071490 2024-09-13T17:11:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.965+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744814f1a0 con 0x7f7448071490 2024-09-13T17:11:21.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.967+0000 7f7445600640 1 -- 172.21.15.28:0/1789488103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7438003990 con 0x7f7448071490 2024-09-13T17:11:21.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.969+0000 7f7445600640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 0x7f742406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.969+0000 7f7445600640 1 -- 172.21.15.28:0/1789488103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74380889c0 con 0x7f7448071490 2024-09-13T17:11:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.969+0000 7f743fe00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 0x7f742406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.969+0000 7f74529ef740 1 Processor -- start 2024-09-13T17:11:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 -- start start 2024-09-13T17:11:21.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f743fe00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 0x7f742406d0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7430006fd0 tx=0x7f7430008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9cbceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9cbd3f0 0x5611a9d036c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9c26ed0 con 0x5611a9ce6b20 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9c24660 con 0x5611a9cbd3f0 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.970+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9c1ff50 con 0x5611a9d03060 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44066/0 (socket says 172.21.15.28:44066) 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 -- 172.21.15.28:0/3184480761 learned_addr learned my addr 172.21.15.28:0/3184480761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744da00640 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9cbd3f0 0x5611a9d036c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f743fe00640 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9cbceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 -- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 msgr2=0x5611a9cbceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9cbceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 -- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9cbd3f0 msgr2=0x5611a9d036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9cbd3f0 0x5611a9d036c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 -- 172.21.15.28:0/3184480761 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a9be7110 con 0x5611a9ce6b20 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f744d000640 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7438086df0 tx=0x7f743800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3a9d3bbb688e99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f743d600640 1 -- 172.21.15.28:0/3184480761 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743805fbb0 con 0x5611a9ce6b20 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f743d600640 1 -- 172.21.15.28:0/3184480761 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438055900 con 0x5611a9ce6b20 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.971+0000 7f743d600640 1 -- 172.21.15.28:0/3184480761 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438053ed0 con 0x5611a9ce6b20 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 -- 172.21.15.28:0/3184480761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 msgr2=0x5611a9ce6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f7438086df0 tx=0x7f743800a000 comp rx=0 tx=0).stop 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 -- 172.21.15.28:0/3184480761 shutdown_connections 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9cbd3f0 0x5611a9d036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9cbceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 --2- 172.21.15.28:0/3184480761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9ce6f00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 -- 172.21.15.28:0/3184480761 >> 172.21.15.28:0/3184480761 conn(0x5611a9cb94a0 msgr2=0x5611a9d01930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 -- 172.21.15.28:0/3184480761 shutdown_connections 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.972+0000 7f74529ef740 1 -- 172.21.15.28:0/3184480761 wait complete. 2024-09-13T17:11:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.973+0000 7f74529ef740 1 -- start start 2024-09-13T17:11:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.973+0000 7f74529ef740 1 Processor -- start 2024-09-13T17:11:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9ce6b20 0x5611a9ce2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9cfefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9bde260 con 0x5611a9d03060 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9bddf00 con 0x5611a9ceab10 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9987340 con 0x5611a9ce6b20 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f74529ef740 10 client.0 _ll_register_callbacks cb 0x5611a9c195c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f743fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f743fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44074/0 (socket says 172.21.15.28:44074) 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f743fe00640 1 -- 172.21.15.28:0/274085618 learned_addr learned my addr 172.21.15.28:0/274085618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f744da00640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9cfefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.974+0000 7f744d000640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9ce6b20 0x5611a9ce2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 -- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9ce6b20 msgr2=0x5611a9ce2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9ce6b20 0x5611a9ce2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 -- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 msgr2=0x5611a9cfefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9cfefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 -- 172.21.15.28:0/274085618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744814f1a0 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f743fe00640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7430014510 tx=0x7f7430004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43733c9377788bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f7436000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74300159a0 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611a9c118d0 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.975+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7430015b00 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7430015cc0 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 10 client.5778 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x5611a9c1e5d0 con 0x5611a9d03060 2024-09-13T17:11:21.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5611a9c118d0 con 0x5611a9d03060 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7430021420 con 0x5611a9d03060 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 10 client.5778 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 10 client.5778 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 20 client.5778 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9d0b5b0 0x5611a9d0b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 4 client.5778 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611a99137e0 con 0x5611a9d0b5b0 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.976+0000 7f744d000640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9d0b5b0 0x5611a9d0b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.977+0000 7f744d000640 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9d0b5b0 0x5611a9d0b990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13c782a34f85c3b4 server_cookie=fb92e97de5f6cc9c in_seq=0 out_seq=0 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.977+0000 7f7436000640 10 client.5778 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.977+0000 7f7436000640 1 -- 172.21.15.28:0/274085618 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5611a99137e0 con 0x5611a9d0b5b0 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.977+0000 7f7436000640 10 client.5778 handle_command_reply: tid=0 2024-09-13T17:11:21.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 client.5778 shutdown 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 msgr2=0x5611a9cfea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7430014510 tx=0x7f7430004730 comp rx=0 tx=0).stop 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 shutdown_connections 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9cfefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9d0b5b0 0x5611a9d0b990 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9d03060 0x5611a9cfea90 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 --2- 172.21.15.28:0/274085618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9ce6b20 0x5611a9ce2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f7436000640 0 client.5778 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 >> 172.21.15.28:0/274085618 conn(0x5611a9cbd3f0 msgr2=0x5611a9cffb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 shutdown_connections 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 1 -- 172.21.15.28:0/274085618 wait complete. 2024-09-13T17:11:21.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.978+0000 7f74529ef740 20 client.5778 trim_cache size 0 max 16384 2024-09-13T17:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 Processor -- start 2024-09-13T17:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 -- start start 2024-09-13T17:11:21.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9ce33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d02950 con 0x5611a9ce6b20 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9a93c50 con 0x5611a9ceab10 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.994+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9cbe4c0 con 0x5611a9d03060 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44078/0 (socket says 172.21.15.28:44078) 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 -- 172.21.15.28:0/1927295495 learned_addr learned my addr 172.21.15.28:0/1927295495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744da00640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9ce33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f743fe00640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 -- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 msgr2=0x5611a9ce33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9ce33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 -- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 msgr2=0x5611a9d04d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:21.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d04d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 -- 172.21.15.28:0/1927295495 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611a9bc03a0 con 0x5611a9ce6b20 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f743fe00640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f744d000640 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7438053c70 tx=0x7f74380576c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4a6bdb8cb5214b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f743e000640 1 -- 172.21.15.28:0/1927295495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7438055b60 con 0x5611a9ce6b20 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.995+0000 7f743e000640 1 -- 172.21.15.28:0/1927295495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438074e90 con 0x5611a9ce6b20 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f743e000640 1 -- 172.21.15.28:0/1927295495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380760b0 con 0x5611a9ce6b20 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 -- 172.21.15.28:0/1927295495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 msgr2=0x5611a9d1f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f7438053c70 tx=0x7f74380576c0 comp rx=0 tx=0).stop 2024-09-13T17:11:21.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 -- 172.21.15.28:0/1927295495 shutdown_connections 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9ce33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 --2- 172.21.15.28:0/1927295495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d1f2d0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 -- 172.21.15.28:0/1927295495 >> 172.21.15.28:0/1927295495 conn(0x5611a9cbd3f0 msgr2=0x5611a9cfebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 -- 172.21.15.28:0/1927295495 shutdown_connections 2024-09-13T17:11:22.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.996+0000 7f74529ef740 1 -- 172.21.15.28:0/1927295495 wait complete. 2024-09-13T17:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.997+0000 7f74529ef740 1 -- start start 2024-09-13T17:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.997+0000 7f74529ef740 1 Processor -- start 2024-09-13T17:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d56860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9d09950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d56da0 con 0x5611a9ce6b20 2024-09-13T17:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d08610 con 0x5611a9ceab10 2024-09-13T17:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611a9d0a1c0 con 0x5611a9d03060 2024-09-13T17:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f74529ef740 10 client.0 _ll_register_callbacks cb 0x7f743000b2a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f744d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44090/0 (socket says 172.21.15.28:44090) 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f744d000640 1 -- 172.21.15.28:0/4146531332 learned_addr learned my addr 172.21.15.28:0/4146531332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f743fe00640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d56860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.998+0000 7f744da00640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9d09950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 -- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 msgr2=0x5611a9d09950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9d09950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 -- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 msgr2=0x5611a9d56860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d56860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 -- 172.21.15.28:0/4146531332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611a9be7110 con 0x5611a9ce6b20 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f743fe00640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f744d000640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f74380574c0 tx=0x7f743801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad92e26456519dc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f7436000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74380760b0 con 0x5611a9ce6b20 2024-09-13T17:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7438076210 con 0x5611a9ce6b20 2024-09-13T17:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7438055900 con 0x5611a9ce6b20 2024-09-13T17:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7438053ed0 con 0x5611a9ce6b20 2024-09-13T17:11:22.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f74529ef740 10 client.5784 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x5611a9be5970 con 0x5611a9ce6b20 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:21.999+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7438055900 con 0x5611a9ce6b20 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7438088040 con 0x5611a9ce6b20 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 10 client.5784 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 10 client.5784 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 20 client.5784 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9ce1860 0x5611a9ce1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 4 client.5784 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}] 2024-09-13T17:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}) v1 -- 0x5611a9924ae0 con 0x5611a9ce1860 2024-09-13T17:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f743fe00640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9ce1860 0x5611a9ce1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f7436000640 10 client.5784 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.000+0000 7f743fe00640 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9ce1860 0x5611a9ce1c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d38b04b149a085d server_cookie=ef83f955c4c0dcae in_seq=0 out_seq=0 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f7436000640 1 -- 172.21.15.28:0/4146531332 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5611a9924ae0 con 0x5611a9ce1860 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f7436000640 10 client.5784 handle_command_reply: tid=0 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 client.5784 shutdown 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 msgr2=0x5611a9d07df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f74380574c0 tx=0x7f743801e040 comp rx=0 tx=0).stop 2024-09-13T17:11:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 shutdown_connections 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611a9d03060 0x5611a9d09950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611a9ceab10 0x5611a9d56860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5611a9ce1860 0x5611a9ce1c40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 --2- 172.21.15.28:0/4146531332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611a9ce6b20 0x5611a9d07df0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 >> 172.21.15.28:0/4146531332 conn(0x5611a9cbd3f0 msgr2=0x5611a9d27fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f7436000640 0 client.5784 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 shutdown_connections 2024-09-13T17:11:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.001+0000 7f74529ef740 1 -- 172.21.15.28:0/4146531332 wait complete. 2024-09-13T17:11:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f74529ef740 20 client.5784 trim_cache size 0 max 16384 2024-09-13T17:11:22.014 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:22.015 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:22.015 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "41e22025-057e-4c86-a44d-6ffa2f0c1680", 2024-09-13T17:11:22.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:22.015 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 msgr2=0x7f742406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 0x7f742406d0c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7430006fd0 tx=0x7f7430008040 comp rx=0 tx=0).stop 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 msgr2=0x7f7448146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f743800c450 tx=0x7f743800c910 comp rx=0 tx=0).stop 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 shutdown_connections 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f742406ac00 0x7f742406d0c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f744815ab60 0x7f744814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f744815a780 0x7f744814e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 --2- 172.21.15.28:0/1789488103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7448071490 0x7f7448146fe0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 >> 172.21.15.28:0/1789488103 conn(0x7f74480b0270 msgr2=0x7f74480ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 shutdown_connections 2024-09-13T17:11:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.002+0000 7f744fa00640 1 -- 172.21.15.28:0/1789488103 wait complete. 2024-09-13T17:11:22.020 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites' returned '{'return_code': 0, 'scrub_tag': '41e22025-057e-4c86-a44d-6ffa2f0c1680', 'mode': 'asynchronous'}' 2024-09-13T17:11:22.020 INFO:tasks.cephfs.test_scrub_checks:First run: flushing /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-09-13T17:11:22.020 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-09-13T17:11:22.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.175+0000 7f77a1c00640 1 Processor -- start 2024-09-13T17:11:22.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.175+0000 7f77a1c00640 1 -- start start 2024-09-13T17:11:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.176+0000 7f77a1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.176+0000 7f77a1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f779c06c2d0 0x7f779c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f77a1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c159d90 con 0x7f779c1592a0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f77a1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779c15b0b0 con 0x7f779c06c2d0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f77a1c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c06cca0 con 0x7f779c06f970 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f779c1592a0 0x7f779c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50042/0 (socket says 172.21.15.28:50042) 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779aa00640 1 -- 172.21.15.28:0/420718515 learned_addr learned my addr 172.21.15.28:0/420718515 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779b400640 1 --2- 172.21.15.28:0/420718515 >> v2:172.21.15.28:3300/0 conn(0x7f779c06c2d0 0x7f779c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f7792000640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779a000640 1 -- 172.21.15.28:0/420718515 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278367942 0 0) 0x7f779c159d90 con 0x7f779c1592a0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779b400640 1 -- 172.21.15.28:0/420718515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 msgr2=0x7f779c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779b400640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779b400640 1 -- 172.21.15.28:0/420718515 >> v1:172.21.15.28:6789/0 conn(0x7f779c1592a0 legacy=0x7f779c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f779b400640 1 -- 172.21.15.28:0/420718515 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c0b2040 con 0x7f779c06c2d0 2024-09-13T17:11:22.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.177+0000 7f7792000640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.178+0000 7f779b400640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c06c760 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7794009d90 tx=0x7f779402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff27d1cb2201eef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f779a000640 1 -- 172.21.15.28:0/420718515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794007740 con 0x7f779c06c2d0 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f779a000640 1 -- 172.21.15.28:0/420718515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779402d050 con 0x7f779c06c2d0 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f779a000640 1 -- 172.21.15.28:0/420718515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794007ac0 con 0x7f779c06c2d0 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 -- 172.21.15.28:0/420718515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 msgr2=0x7f779c06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c06c760 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f7794009d90 tx=0x7f779402a660 comp rx=0 tx=0).stop 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 -- 172.21.15.28:0/420718515 shutdown_connections 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c06c760 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 --2- 172.21.15.28:0/420718515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 -- 172.21.15.28:0/420718515 >> 172.21.15.28:0/420718515 conn(0x7f779c069e80 msgr2=0x7f779c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 -- 172.21.15.28:0/420718515 shutdown_connections 2024-09-13T17:11:22.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.179+0000 7f77a1c00640 1 -- 172.21.15.28:0/420718515 wait complete. 2024-09-13T17:11:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.181+0000 7f77a1c00640 1 Processor -- start 2024-09-13T17:11:22.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.181+0000 7f77a1c00640 1 -- start start 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c14cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 0x7f779c14d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c1406f0 con 0x7f779c06c2d0 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c1409d0 con 0x7f779c1592a0 2024-09-13T17:11:22.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f77a1c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c140cb0 con 0x7f779c06f970 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f779aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f779aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44118/0 (socket says 172.21.15.28:44118) 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.182+0000 7f779aa00640 1 -- 172.21.15.28:0/3234022228 learned_addr learned my addr 172.21.15.28:0/3234022228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 -- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 msgr2=0x7f779c14cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c14cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 -- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 msgr2=0x7f779c14d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779b400640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 0x7f779c14d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 0x7f779c14d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 -- 172.21.15.28:0/3234022228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7794009a70 con 0x7f779c06c2d0 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779b400640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 0x7f779c14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f779aa00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f778c00c400 tx=0x7f778c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c005ce0 con 0x7f779c06c2d0 2024-09-13T17:11:22.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c005e40 con 0x7f779c06c2d0 2024-09-13T17:11:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c007310 con 0x7f779c06c2d0 2024-09-13T17:11:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c140f90 con 0x7f779c06c2d0 2024-09-13T17:11:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.183+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c141ea0 con 0x7f779c06c2d0 2024-09-13T17:11:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.184+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c1414b0 con 0x7f779c06c2d0 2024-09-13T17:11:22.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.185+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f778c007470 con 0x7f779c06c2d0 2024-09-13T17:11:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.186+0000 7f7793400640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 0x7f777806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.186+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f778c088900 con 0x7f779c06c2d0 2024-09-13T17:11:22.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.187+0000 7f7792000640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 0x7f777806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.188+0000 7f7792000640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 0x7f777806d0f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.191+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f779c1414b0 con 0x7f779c06c2d0 2024-09-13T17:11:22.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.405+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f779c1414b0 con 0x7f779c06c2d0 2024-09-13T17:11:22.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.406+0000 7f7793400640 1 -- 172.21.15.28:0/3234022228 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f779c1414b0 con 0x7f779c06c2d0 2024-09-13T17:11:22.407 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:22.407 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:22.407 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:22.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 msgr2=0x7f777806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 0x7f777806d0f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7784006fd0 tx=0x7f7784008040 comp rx=0 tx=0).stop 2024-09-13T17:11:22.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 msgr2=0x7f779c0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f778c00c400 tx=0x7f778c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 shutdown_connections 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777806ac30 0x7f777806d0f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c1592a0 0x7f779c14d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06f970 0x7f779c14cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 --2- 172.21.15.28:0/3234022228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06c2d0 0x7f779c0b0540 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 >> 172.21.15.28:0/3234022228 conn(0x7f779c069e80 msgr2=0x7f779c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 shutdown_connections 2024-09-13T17:11:22.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.408+0000 7f77a1c00640 1 -- 172.21.15.28:0/3234022228 wait complete. 2024-09-13T17:11:22.425 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-09-13T17:11:22.587 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:22.588 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:22.588 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:22.600 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:22.601 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-09-13T17:11:22.601 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-09-13T17:11:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.758+0000 7f4549800640 1 Processor -- start 2024-09-13T17:11:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.758+0000 7f4549800640 1 -- start start 2024-09-13T17:11:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4549800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f45440b2400 0x7f45440ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4549800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4544070470 con 0x7f454415c9a0 2024-09-13T17:11:22.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4549800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440ace30 con 0x7f45440ad940 2024-09-13T17:11:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4549800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45440ad110 con 0x7f45440b2400 2024-09-13T17:11:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4542a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f454415c9a0 0x7f45440b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50056/0 (socket says 172.21.15.28:50056) 2024-09-13T17:11:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.759+0000 7f4542a00640 1 -- 172.21.15.28:0/3564278847 learned_addr learned my addr 172.21.15.28:0/3564278847 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4542000640 1 --2- 172.21.15.28:0/3564278847 >> v2:172.21.15.189:3300/0 conn(0x7f45440b2400 0x7f45440ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985070260 0 0) 0x7f4544070470 con 0x7f454415c9a0 2024-09-13T17:11:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4528003520 con 0x7f454415c9a0 2024-09-13T17:11:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469909281 0 0) 0x7f4528003520 con 0x7f454415c9a0 2024-09-13T17:11:22.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 >> v2:172.21.15.189:3300/0 conn(0x7f45440b2400 msgr2=0x7f45440ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 --2- 172.21.15.28:0/3564278847 >> v2:172.21.15.189:3300/0 conn(0x7f45440b2400 0x7f45440ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 >> v1:172.21.15.189:6789/0 conn(0x7f45440ad940 legacy=0x7f454406bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454406b170 con 0x7f454415c9a0 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4534001670 con 0x7f454415c9a0 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 >> v1:172.21.15.28:6789/0 conn(0x7f454415c9a0 legacy=0x7f45440b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4528003970 0x7f4528003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45280047c0 0x7f4528004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f452800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440ad110 con 0x7f4528005180 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452800c020 con 0x7f4528003970 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.760+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f452800c1e0 con 0x7f45280047c0 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f452800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4542a00640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4528003970 0x7f4528003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4542000640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45280047c0 0x7f4528004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 -- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45280047c0 msgr2=0x7f4528004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45280047c0 0x7f4528004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 -- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4528003970 msgr2=0x7f4528003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4528003970 0x7f4528003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.761+0000 7f4543400640 1 -- 172.21.15.28:0/3564278847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4528003520 con 0x7f4528005180 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.762+0000 7f4543400640 1 -- 172.21.15.28:0/3564278847 reap_dead start 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.762+0000 7f4543400640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f452800b980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f453c001e50 tx=0x7f45280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05d98f2e059f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.762+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c02ae70 con 0x7f4528005180 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.762+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c003970 con 0x7f4528005180 2024-09-13T17:11:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.762+0000 7f4541600640 1 -- 172.21.15.28:0/3564278847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c003d20 con 0x7f4528005180 2024-09-13T17:11:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 -- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 msgr2=0x7f452800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f452800b980 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f453c001e50 tx=0x7f45280057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 -- 172.21.15.28:0/3564278847 shutdown_connections 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 --2- 172.21.15.28:0/3564278847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f452800b980 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 -- 172.21.15.28:0/3564278847 >> 172.21.15.28:0/3564278847 conn(0x7f45440b0270 msgr2=0x7f45440b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 -- 172.21.15.28:0/3564278847 shutdown_connections 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.763+0000 7f4549800640 1 -- 172.21.15.28:0/3564278847 wait complete. 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.765+0000 7f4549800640 1 Processor -- start 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.765+0000 7f4549800640 1 -- start start 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 0x7f454413f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45441416c0 0x7f4544141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544140620 con 0x7f4528005180 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4544140900 con 0x7f45441416c0 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4549800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4544140be0 con 0x7f45441e1f50 2024-09-13T17:11:22.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44142/0 (socket says 172.21.15.28:44142) 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 -- 172.21.15.28:0/673366214 learned_addr learned my addr 172.21.15.28:0/673366214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4543400640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45441416c0 0x7f4544141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542000640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 0x7f454413f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 -- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 msgr2=0x7f454413f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 0x7f454413f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 -- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45441416c0 msgr2=0x7f4544141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45441416c0 0x7f4544141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542a00640 1 -- 172.21.15.28:0/673366214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544070930 con 0x7f4528005180 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.766+0000 7f4542000640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 0x7f454413f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f4542a00640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4534007290 tx=0x7f4534007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f453aa00640 1 -- 172.21.15.28:0/673366214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534004ea0 con 0x7f4528005180 2024-09-13T17:11:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f453aa00640 1 -- 172.21.15.28:0/673366214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453400f9c0 con 0x7f4528005180 2024-09-13T17:11:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f453aa00640 1 -- 172.21.15.28:0/673366214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453400fd90 con 0x7f4528005180 2024-09-13T17:11:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4544140ec0 con 0x7f4528005180 2024-09-13T17:11:22.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.767+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454413fb50 con 0x7f4528005180 2024-09-13T17:11:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.769+0000 7f453aa00640 1 -- 172.21.15.28:0/673366214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f453401d460 con 0x7f4528005180 2024-09-13T17:11:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.770+0000 7f453aa00640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 0x7f452406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.770+0000 7f453aa00640 1 -- 172.21.15.28:0/673366214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4534089590 con 0x7f4528005180 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.770+0000 7f4542000640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 0x7f452406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.771+0000 7f454c5e0740 1 Processor -- start 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.771+0000 7f454c5e0740 1 -- start start 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.771+0000 7f4542000640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 0x7f452406d0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45440b0140 tx=0x7f4530008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e80525980 0x560e80575690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80575bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e80492130 con 0x560e805281c0 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e8048f8c0 con 0x560e8054f670 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e8048b1b0 con 0x560e80525980 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44152/0 (socket says 172.21.15.28:44152) 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 -- 172.21.15.28:0/1305521252 learned_addr learned my addr 172.21.15.28:0/1305521252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4543400640 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80575bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542000640 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e80525980 0x560e80575690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 -- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e80525980 msgr2=0x560e80575690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e80525980 0x560e80575690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 -- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 msgr2=0x560e80575bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80575bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.772+0000 7f4542a00640 1 -- 172.21.15.28:0/1305521252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e80455510 con 0x560e805281c0 2024-09-13T17:11:22.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f4542a00640 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4534060250 tx=0x7f45340112b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=972dde77ac1c1d4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f452f400640 1 -- 172.21.15.28:0/1305521252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534058390 con 0x560e805281c0 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f452f400640 1 -- 172.21.15.28:0/1305521252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453405c1c0 con 0x560e805281c0 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f452f400640 1 -- 172.21.15.28:0/1305521252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534060780 con 0x560e805281c0 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 -- 172.21.15.28:0/1305521252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 msgr2=0x560e805285a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4534060250 tx=0x7f45340112b0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 -- 172.21.15.28:0/1305521252 shutdown_connections 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80575bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e80525980 0x560e80575690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 --2- 172.21.15.28:0/1305521252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e805285a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 -- 172.21.15.28:0/1305521252 >> 172.21.15.28:0/1305521252 conn(0x560e80524b80 msgr2=0x560e8056cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 -- 172.21.15.28:0/1305521252 shutdown_connections 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.773+0000 7f454c5e0740 1 -- 172.21.15.28:0/1305521252 wait complete. 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.774+0000 7f454c5e0740 1 -- start start 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.774+0000 7f454c5e0740 1 Processor -- start 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80572440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054e780 0x560e80572980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e80449570 con 0x560e805281c0 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e80449210 con 0x560e8054f670 2024-09-13T17:11:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e801f2740 con 0x560e8054e780 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.775+0000 7f454c5e0740 10 client.0 _ll_register_callbacks cb 0x560e801a12e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44168/0 (socket says 172.21.15.28:44168) 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 -- 172.21.15.28:0/965240799 learned_addr learned my addr 172.21.15.28:0/965240799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4543400640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054e780 0x560e80572980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542000640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80572440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 -- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054e780 msgr2=0x560e80572980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054e780 0x560e80572980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 -- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 msgr2=0x560e80572440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80572440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 -- 172.21.15.28:0/965240799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454413fb50 con 0x560e805281c0 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542000640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80572440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4542a00640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4534060250 tx=0x7f4534001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46feb618dcd34e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4523e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4534060780 con 0x560e805281c0 2024-09-13T17:11:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45340602f0 con 0x560e805281c0 2024-09-13T17:11:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.776+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560e8017ebe0 con 0x560e805281c0 2024-09-13T17:11:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45340604b0 con 0x560e805281c0 2024-09-13T17:11:22.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 10 client.5805 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x560e80489830 con 0x560e805281c0 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560e8017ebe0 con 0x560e805281c0 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f453407c080 con 0x560e805281c0 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 10 client.5805 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 10 client.5805 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 20 client.5805 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:22.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80551310 0x560e80572ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 4 client.5805 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e8018fee0 con 0x560e80551310 2024-09-13T17:11:22.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.777+0000 7f4542000640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80551310 0x560e80572ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.778+0000 7f4542000640 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80551310 0x560e80572ec0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25b54a41ab67a569 server_cookie=f7cb05ac049a7807 in_seq=0 out_seq=0 2024-09-13T17:11:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.778+0000 7f4523e00640 10 client.5805 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.778+0000 7f4523e00640 1 -- 172.21.15.28:0/965240799 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x560e8018fee0 con 0x560e80551310 2024-09-13T17:11:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.778+0000 7f4523e00640 10 client.5805 handle_command_reply: tid=0 2024-09-13T17:11:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 client.5805 shutdown 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 msgr2=0x560e80550dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4534060250 tx=0x7f4534001a00 comp rx=0 tx=0).stop 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 shutdown_connections 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054e780 0x560e80572980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e80572440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80551310 0x560e80572ec0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 --2- 172.21.15.28:0/965240799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e805281c0 0x560e80550dd0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 >> 172.21.15.28:0/965240799 conn(0x560e80525980 msgr2=0x560e8056c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f4523e00640 0 client.5805 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 shutdown_connections 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 1 -- 172.21.15.28:0/965240799 wait complete. 2024-09-13T17:11:22.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.779+0000 7f454c5e0740 20 client.5805 trim_cache size 0 max 16384 2024-09-13T17:11:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.794+0000 7f454c5e0740 1 Processor -- start 2024-09-13T17:11:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 -- start start 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 0x560e8054c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e8056ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e80527d70 con 0x560e8054e780 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e805526d0 con 0x560e8054f670 2024-09-13T17:11:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.795+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e8051d5c0 con 0x560e805281c0 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44170/0 (socket says 172.21.15.28:44170) 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 -- 172.21.15.28:0/2365060515 learned_addr learned my addr 172.21.15.28:0/2365060515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542a00640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 0x560e8054c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 -- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 msgr2=0x560e8054c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4543400640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e8056ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 0x560e8054c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 -- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 msgr2=0x560e8056ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e8056ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 -- 172.21.15.28:0/2365060515 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e8042b7e0 con 0x560e8054e780 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542a00640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 0x560e8054c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f4542000640 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45300126a0 tx=0x7f4530018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2af521ebdb712f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f452fe00640 1 -- 172.21.15.28:0/2365060515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530029040 con 0x560e8054e780 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f452fe00640 1 -- 172.21.15.28:0/2365060515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453002bd00 con 0x560e8054e780 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.796+0000 7f452fe00640 1 -- 172.21.15.28:0/2365060515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453002b4c0 con 0x560e8054e780 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 -- 172.21.15.28:0/2365060515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 msgr2=0x560e8054c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f45300126a0 tx=0x7f4530018000 comp rx=0 tx=0).stop 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 -- 172.21.15.28:0/2365060515 shutdown_connections 2024-09-13T17:11:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e8054f670 0x560e8056ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8054c600 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 --2- 172.21.15.28:0/2365060515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e805281c0 0x560e8054c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 -- 172.21.15.28:0/2365060515 >> 172.21.15.28:0/2365060515 conn(0x560e80525980 msgr2=0x560e8051e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 -- 172.21.15.28:0/2365060515 shutdown_connections 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.797+0000 7f454c5e0740 1 -- 172.21.15.28:0/2365060515 wait complete. 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.798+0000 7f454c5e0740 1 -- start start 2024-09-13T17:11:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.798+0000 7f454c5e0740 1 Processor -- start 2024-09-13T17:11:22.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 0x560e805d9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054f670 0x560e8056e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e8058ac50 con 0x560e8054e780 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e8056c9b0 con 0x560e805281c0 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e8056cc90 con 0x560e8054f670 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f454c5e0740 10 client.0 _ll_register_callbacks cb 0x7f453400ff70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:22.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44180/0 (socket says 172.21.15.28:44180) 2024-09-13T17:11:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 -- 172.21.15.28:0/243194754 learned_addr learned my addr 172.21.15.28:0/243194754 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542a00640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 0x560e805d9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4543400640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054f670 0x560e8056e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 -- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054f670 msgr2=0x560e8056e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054f670 0x560e8056e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 -- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 msgr2=0x560e805d9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 0x560e805d9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542000640 1 -- 172.21.15.28:0/243194754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4530012950 con 0x560e8054e780 2024-09-13T17:11:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.799+0000 7f4542a00640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 0x560e805d9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f4542000640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4530017040 tx=0x7f4530004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36551bd34216514 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f4523e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4530029050 con 0x560e8054e780 2024-09-13T17:11:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4530039070 con 0x560e8054e780 2024-09-13T17:11:22.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f453405c1c0 con 0x560e8054e780 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f453002be90 con 0x560e8054e780 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f454c5e0740 10 client.5811 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.800+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x560e80450c80 con 0x560e8054e780 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453405c1c0 con 0x560e8054e780 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f453003e480 con 0x560e8054e780 2024-09-13T17:11:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 10 client.5811 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 10 client.5811 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 20 client.5811 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80526450 0x560e8057e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 4 client.5811 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}] 2024-09-13T17:11:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}) v1 -- 0x560e801e3530 con 0x560e80526450 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f4542a00640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80526450 0x560e8057e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f4542a00640 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80526450 0x560e8057e3a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb6cac9449bdb476 server_cookie=10096c9a754cc643 in_seq=0 out_seq=0 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.801+0000 7f4523e00640 10 client.5811 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f4523e00640 1 -- 172.21.15.28:0/243194754 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560e801e3530 con 0x560e80526450 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f4523e00640 10 client.5811 handle_command_reply: tid=0 2024-09-13T17:11:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 client.5811 shutdown 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 msgr2=0x560e8056db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f4530017040 tx=0x7f4530004730 comp rx=0 tx=0).stop 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 shutdown_connections 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e8054f670 0x560e8056e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e80526450 0x560e8057e3a0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e8054e780 0x560e8056db80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 --2- 172.21.15.28:0/243194754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e805281c0 0x560e805d9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f4523e00640 0 client.5811 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 >> 172.21.15.28:0/243194754 conn(0x560e80525980 msgr2=0x560e8054c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 shutdown_connections 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.802+0000 7f454c5e0740 1 -- 172.21.15.28:0/243194754 wait complete. 2024-09-13T17:11:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f454c5e0740 20 client.5811 trim_cache size 0 max 16384 2024-09-13T17:11:22.816 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:22.816 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:22.816 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "7d924af1-b32f-4533-9a52-00a03aca5951", 2024-09-13T17:11:22.817 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:22.817 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 msgr2=0x7f452406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 0x7f452406d0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f45440b0140 tx=0x7f4530008040 comp rx=0 tx=0).stop 2024-09-13T17:11:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 msgr2=0x7f454413ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f4534007290 tx=0x7f4534007750 comp rx=0 tx=0).stop 2024-09-13T17:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 shutdown_connections 2024-09-13T17:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f452406ac00 0x7f452406d0c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45441416c0 0x7f4544141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45441e1f50 0x7f454413f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 --2- 172.21.15.28:0/673366214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4528005180 0x7f454413ff20 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 >> 172.21.15.28:0/673366214 conn(0x7f45440b0270 msgr2=0x7f454406c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 shutdown_connections 2024-09-13T17:11:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.803+0000 7f4549800640 1 -- 172.21.15.28:0/673366214 wait complete. 2024-09-13T17:11:22.822 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' returned '{'return_code': 0, 'scrub_tag': '7d924af1-b32f-4533-9a52-00a03aca5951', 'mode': 'asynchronous'}' 2024-09-13T17:11:22.822 INFO:tasks.cephfs.test_scrub_checks:First run: flushing base dir / 2024-09-13T17:11:22.822 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-09-13T17:11:22.822 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.979+0000 7febb3200640 1 Processor -- start 2024-09-13T17:11:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.979+0000 7febb3200640 1 -- start start 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febb3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7febac15a540 0x7febac06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febb3200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7febac070cc0 0x7febac071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febb3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febac15e100 con 0x7febac0cabc0 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febb3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac0adb00 con 0x7febac15a540 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febb3200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febac071670 con 0x7febac070cc0 2024-09-13T17:11:22.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7febac15a540 0x7febac06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7febac15a540 0x7febac06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44190/0 (socket says 172.21.15.28:44190) 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7feba3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7febac0cabc0 0x7febac15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50070/0 (socket says 172.21.15.28:50070) 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.980+0000 7febabe00640 1 -- 172.21.15.28:0/3339955412 learned_addr learned my addr 172.21.15.28:0/3339955412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.981+0000 7febb1200640 1 --2- 172.21.15.28:0/3339955412 >> v2:172.21.15.189:3300/0 conn(0x7febac070cc0 0x7febac071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.981+0000 7febabe00640 1 -- 172.21.15.28:0/3339955412 >> v2:172.21.15.189:3300/0 conn(0x7febac070cc0 msgr2=0x7febac071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:22.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.981+0000 7febabe00640 1 --2- 172.21.15.28:0/3339955412 >> v2:172.21.15.189:3300/0 conn(0x7febac070cc0 0x7febac071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.981+0000 7febabe00640 1 -- 172.21.15.28:0/3339955412 >> v1:172.21.15.28:6789/0 conn(0x7febac0cabc0 legacy=0x7febac15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.981+0000 7febabe00640 1 -- 172.21.15.28:0/3339955412 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac15e3e0 con 0x7febac15a540 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.982+0000 7febabe00640 1 --2- 172.21.15.28:0/3339955412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac15a540 0x7febac06c430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7febac070900 tx=0x7feb9c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68795c344d45bb7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.982+0000 7febab400640 1 -- 172.21.15.28:0/3339955412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c032440 con 0x7febac15a540 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.982+0000 7febab400640 1 -- 172.21.15.28:0/3339955412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9c0325a0 con 0x7febac15a540 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.982+0000 7febab400640 1 -- 172.21.15.28:0/3339955412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb9c032930 con 0x7febac15a540 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 -- 172.21.15.28:0/3339955412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac15a540 msgr2=0x7febac06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 --2- 172.21.15.28:0/3339955412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac15a540 0x7febac06c430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7febac070900 tx=0x7feb9c02a210 comp rx=0 tx=0).stop 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 -- 172.21.15.28:0/3339955412 shutdown_connections 2024-09-13T17:11:22.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 --2- 172.21.15.28:0/3339955412 >> v2:172.21.15.189:3300/0 conn(0x7febac070cc0 0x7febac071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 --2- 172.21.15.28:0/3339955412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac15a540 0x7febac06c430 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 -- 172.21.15.28:0/3339955412 >> 172.21.15.28:0/3339955412 conn(0x7febac069e80 msgr2=0x7febac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 -- 172.21.15.28:0/3339955412 shutdown_connections 2024-09-13T17:11:22.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.983+0000 7febb3200640 1 -- 172.21.15.28:0/3339955412 wait complete. 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.984+0000 7febb3200640 1 Processor -- start 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.985+0000 7febb3200640 1 -- start start 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.985+0000 7febb3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febac070cc0 0x7febac0b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac15a540 0x7febac0ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac0afde0 con 0x7febac0cabc0 2024-09-13T17:11:22.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac0b2870 con 0x7febac15a540 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac0b1cf0 con 0x7febac070cc0 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44202/0 (socket says 172.21.15.28:44202) 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 -- 172.21.15.28:0/4009835405 learned_addr learned my addr 172.21.15.28:0/4009835405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febabe00640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febac070cc0 0x7febac0b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7febb1200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac15a540 0x7febac0ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 -- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febac070cc0 msgr2=0x7febac0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febac070cc0 0x7febac0b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 -- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac15a540 msgr2=0x7febac0ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac15a540 0x7febac0ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.986+0000 7feba3400640 1 -- 172.21.15.28:0/4009835405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c0094d0 con 0x7febac0cabc0 2024-09-13T17:11:22.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7feba3400640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c4d0 tx=0x7feb9400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb94007480 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb940075e0 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb940049e0 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febac0b1fd0 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.987+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febac0b2590 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.988+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febac0aff70 con 0x7febac0cabc0 2024-09-13T17:11:22.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.989+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feb94004b40 con 0x7febac0cabc0 2024-09-13T17:11:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.990+0000 7feba8c00640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 0x7feb8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.990+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb94088570 con 0x7febac0cabc0 2024-09-13T17:11:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.990+0000 7febabe00640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 0x7feb8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.991+0000 7febabe00640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 0x7feb8806d0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb9c034000 tx=0x7feb9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:22.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:22.994+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7febac0aff70 con 0x7febac0cabc0 2024-09-13T17:11:23.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.211+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febac0aff70 con 0x7febac0cabc0 2024-09-13T17:11:23.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.212+0000 7feba8c00640 1 -- 172.21.15.28:0/4009835405 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7febac0aff70 con 0x7febac0cabc0 2024-09-13T17:11:23.213 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:23.213 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:23.214 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:23.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 msgr2=0x7feb8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 0x7feb8806d0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7feb9c034000 tx=0x7feb9c02e000 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 msgr2=0x7febac1440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7feb9400c4d0 tx=0x7feb9400c990 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 shutdown_connections 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb8806ac00 0x7feb8806d0c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febac15a540 0x7febac0ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7febac0cabc0 0x7febac1440c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 --2- 172.21.15.28:0/4009835405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febac070cc0 0x7febac0b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.214+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 >> 172.21.15.28:0/4009835405 conn(0x7febac069e80 msgr2=0x7febac0cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.215+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 shutdown_connections 2024-09-13T17:11:23.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.215+0000 7febb3200640 1 -- 172.21.15.28:0/4009835405 wait complete. 2024-09-13T17:11:23.231 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path / 2024-09-13T17:11:23.392 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:23.392 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:23.392 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:23.404 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:23.404 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /' 2024-09-13T17:11:23.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / 2024-09-13T17:11:23.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.563+0000 7fcda2a00640 1 Processor -- start 2024-09-13T17:11:23.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.563+0000 7fcda2a00640 1 -- start start 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcda2a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcd9c159770 0x7fcd9c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcda2a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c06f920 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcda2a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd9c15b290 con 0x7fcd9c15cd80 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcda2a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd9c15b570 con 0x7fcd9c159770 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcd93400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcd9c06f3f0 0x7fcd9c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50086/0 (socket says 172.21.15.28:50086) 2024-09-13T17:11:23.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.564+0000 7fcd93400640 1 -- 172.21.15.28:0/1829803735 learned_addr learned my addr 172.21.15.28:0/1829803735 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9be00640 1 --2- 172.21.15.28:0/1829803735 >> v2:172.21.15.189:3300/0 conn(0x7fcd9c159770 0x7fcd9c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197094765 0 0) 0x7fcd9c06f920 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd80003520 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126971447 0 0) 0x7fcd80003520 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 >> v2:172.21.15.189:3300/0 conn(0x7fcd9c159770 msgr2=0x7fcd9c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 --2- 172.21.15.28:0/1829803735 >> v2:172.21.15.189:3300/0 conn(0x7fcd9c159770 0x7fcd9c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 >> v1:172.21.15.189:6789/0 conn(0x7fcd9c15cd80 legacy=0x7fcd9c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c15b850 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fcd84001670 con 0x7fcd9c06f3f0 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 >> v1:172.21.15.28:6789/0 conn(0x7fcd9c06f3f0 legacy=0x7fcd9c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800039c0 0x7fcd80003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd800047c0 0x7fcd80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 0x7fcd8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c15b570 con 0x7fcd800051d0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8000c120 con 0x7fcd800047c0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.565+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8000c2e0 con 0x7fcd800039c0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 0x7fcd8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcd9be00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800039c0 0x7fcd80003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcd93400640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd800047c0 0x7fcd80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 -- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800039c0 msgr2=0x7fcd80003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800039c0 0x7fcd80003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 -- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd800047c0 msgr2=0x7fcd80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd800047c0 0x7fcd80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.566+0000 7fcda0a00640 1 -- 172.21.15.28:0/1829803735 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80003520 con 0x7fcd800051d0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.567+0000 7fcda0a00640 1 -- 172.21.15.28:0/1829803735 reap_dead start 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.567+0000 7fcda0a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 0x7fcd8000b980 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcd94001e20 tx=0x7fcd800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f07c0ffd800f2097 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.567+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9402ae40 con 0x7fcd800051d0 2024-09-13T17:11:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.567+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94003970 con 0x7fcd800051d0 2024-09-13T17:11:23.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.567+0000 7fcd9b400640 1 -- 172.21.15.28:0/1829803735 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94003d00 con 0x7fcd800051d0 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 -- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 msgr2=0x7fcd8000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 0x7fcd8000b980 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcd94001e20 tx=0x7fcd800057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 -- 172.21.15.28:0/1829803735 shutdown_connections 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 --2- 172.21.15.28:0/1829803735 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd800051d0 0x7fcd8000b980 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 -- 172.21.15.28:0/1829803735 >> 172.21.15.28:0/1829803735 conn(0x7fcd9c0b0270 msgr2=0x7fcd9c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 -- 172.21.15.28:0/1829803735 shutdown_connections 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.568+0000 7fcda2a00640 1 -- 172.21.15.28:0/1829803735 wait complete. 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.570+0000 7fcda2a00640 1 Processor -- start 2024-09-13T17:11:23.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.570+0000 7fcda2a00640 1 -- start start 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800051d0 0x7fcd9c155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd9c1f6050 0x7fcd9c159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c155890 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c156300 con 0x7fcd9c1f6050 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcda2a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c1f3090 con 0x7fcd800051d0 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44222/0 (socket says 172.21.15.28:44222) 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 -- 172.21.15.28:0/3015647659 learned_addr learned my addr 172.21.15.28:0/3015647659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 -- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800051d0 msgr2=0x7fcd9c155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800051d0 0x7fcd9c155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 -- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd9c1f6050 msgr2=0x7fcd9c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd9c1f6050 0x7fcd9c159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.571+0000 7fcd93400640 1 -- 172.21.15.28:0/3015647659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c1e66a0 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcd93400640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcd84001d70 tx=0x7fcd84007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcd98c00640 1 -- 172.21.15.28:0/3015647659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84007790 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcd98c00640 1 -- 172.21.15.28:0/3015647659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84001610 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcd98c00640 1 -- 172.21.15.28:0/3015647659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8400fbe0 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c1e6980 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.572+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c1f5b30 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.574+0000 7fcd98c00640 1 -- 172.21.15.28:0/3015647659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcd8400fd40 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.575+0000 7fcd98c00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 0x7fcd7806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.575+0000 7fcd98c00640 1 -- 172.21.15.28:0/3015647659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd84089460 con 0x7fcd9c1f2c10 2024-09-13T17:11:23.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.575+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 0x7fcd7806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcda5843740 1 Processor -- start 2024-09-13T17:11:23.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcda5843740 1 -- start start 2024-09-13T17:11:23.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 0x7fcd7806d0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009a40 tx=0x7fcd8c009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b14bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 0x55745b145e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.576+0000 7fcda5843740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda5843740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b107ae0 con 0x55745b18ed50 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b0b4de0 con 0x55745b14b820 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745b0b2570 con 0x55745b148fe0 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44228/0 (socket says 172.21.15.28:44228) 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 -- 172.21.15.28:0/3307164541 learned_addr learned my addr 172.21.15.28:0/3307164541 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b14bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcd93400640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 0x55745b145e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 -- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 msgr2=0x55745b145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 0x55745b145e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 -- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 msgr2=0x55745b14bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b14bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcda0a00640 1 -- 172.21.15.28:0/3307164541 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55745b04fa50 con 0x55745b18ed50 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.577+0000 7fcd93400640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 0x55745b145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcda0a00640 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcd9402a8f0 tx=0x7fcd9402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4914ad90b88a3c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcd90c00640 1 -- 172.21.15.28:0/3307164541 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94038070 con 0x55745b18ed50 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcd90c00640 1 -- 172.21.15.28:0/3307164541 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94004c30 con 0x55745b18ed50 2024-09-13T17:11:23.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcd90c00640 1 -- 172.21.15.28:0/3307164541 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94006b50 con 0x55745b18ed50 2024-09-13T17:11:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcda5843740 1 -- 172.21.15.28:0/3307164541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 msgr2=0x55745b18f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.578+0000 7fcda5843740 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fcd9402a8f0 tx=0x7fcd9402c070 comp rx=0 tx=0).stop 2024-09-13T17:11:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 -- 172.21.15.28:0/3307164541 shutdown_connections 2024-09-13T17:11:23.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b18ed50 0x55745b18f200 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b148fe0 0x55745b145e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 --2- 172.21.15.28:0/3307164541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b14bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 -- 172.21.15.28:0/3307164541 >> 172.21.15.28:0/3307164541 conn(0x55745b147fe0 msgr2=0x55745b1906a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 -- 172.21.15.28:0/3307164541 shutdown_connections 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 -- 172.21.15.28:0/3307164541 wait complete. 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.579+0000 7fcda5843740 1 -- start start 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 Processor -- start 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 0x55745b187f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b18ed50 0x55745b15f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b0ade60 con 0x55745b160ed0 2024-09-13T17:11:23.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b06c220 con 0x55745b18ed50 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745b06bec0 con 0x55745b14b820 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.580+0000 7fcda5843740 10 client.0 _ll_register_callbacks cb 0x55745b0a74d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44232/0 (socket says 172.21.15.28:44232) 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 -- 172.21.15.28:0/569205404 learned_addr learned my addr 172.21.15.28:0/569205404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcd9be00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 0x55745b187f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcd93400640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b18ed50 0x55745b15f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 -- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 msgr2=0x55745b187f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 0x55745b187f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 -- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b18ed50 msgr2=0x55745b15f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b18ed50 0x55745b15f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 -- 172.21.15.28:0/569205404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd80003520 con 0x55745b160ed0 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcd9be00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 0x55745b187f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:23.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcda0a00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005020 tx=0x7fcd94001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314f66c9628d88a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.581+0000 7fcd89600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd94042070 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55745b0d7340 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd940052e0 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd940054a0 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 10 client.5832 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55745b0ac4e0 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55745b0d7340 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcd94005600 con 0x55745b160ed0 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 10 client.5832 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 10 client.5832 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 20 client.5832 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:23.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.582+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b144eb0 0x55745b1452c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.583+0000 7fcda5843740 4 client.5832 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.583+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55745b0a2640 con 0x55745b144eb0 2024-09-13T17:11:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.583+0000 7fcd9be00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b144eb0 0x55745b1452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.583+0000 7fcd9be00640 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b144eb0 0x55745b1452c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e02f8973212c953 server_cookie=1fa23000a372c564 in_seq=0 out_seq=0 2024-09-13T17:11:23.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.583+0000 7fcd89600640 10 client.5832 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:23.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcd89600640 1 -- 172.21.15.28:0/569205404 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55745b0a2640 con 0x55745b144eb0 2024-09-13T17:11:23.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcd89600640 10 client.5832 handle_command_reply: tid=0 2024-09-13T17:11:23.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 client.5832 shutdown 2024-09-13T17:11:23.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 msgr2=0x55745b191190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcd94005020 tx=0x7fcd94001510 comp rx=0 tx=0).stop 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 shutdown_connections 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b144eb0 0x55745b1452c0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b191190 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b18ed50 0x55745b15f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 --2- 172.21.15.28:0/569205404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b14b820 0x55745b187f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 >> 172.21.15.28:0/569205404 conn(0x55745b148fe0 msgr2=0x55745b193ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcd89600640 0 client.5832 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 shutdown_connections 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.584+0000 7fcda5843740 1 -- 172.21.15.28:0/569205404 wait complete. 2024-09-13T17:11:23.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.585+0000 7fcda5843740 20 client.5832 trim_cache size 0 max 16384 2024-09-13T17:11:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.600+0000 7fcda5843740 1 Processor -- start 2024-09-13T17:11:23.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.600+0000 7fcda5843740 1 -- start start 2024-09-13T17:11:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.600+0000 7fcda5843740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.600+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 0x55745b1415d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.600+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b147420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcda5843740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745ae14ed0 con 0x55745b14b820 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b1862e0 con 0x55745b160ed0 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745b193ba0 con 0x55745b18ed50 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44242/0 (socket says 172.21.15.28:44242) 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 -- 172.21.15.28:0/1368500444 learned_addr learned my addr 172.21.15.28:0/1368500444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 -- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 msgr2=0x55745b147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd93400640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 0x55745b1415d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcda0a00640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b147420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 -- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 msgr2=0x55745b1415d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 0x55745b1415d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 -- 172.21.15.28:0/1368500444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55745b0e0130 con 0x55745b14b820 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd93400640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 0x55745b1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.601+0000 7fcd9be00640 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x55745b186ea0 tx=0x7fcd8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec339e1a30d6c5ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcd91600640 1 -- 172.21.15.28:0/1368500444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c02be20 con 0x55745b14b820 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcd91600640 1 -- 172.21.15.28:0/1368500444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8c02b3a0 con 0x55745b14b820 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcd91600640 1 -- 172.21.15.28:0/1368500444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8c02b730 con 0x55745b14b820 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 -- 172.21.15.28:0/1368500444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 msgr2=0x55745b188090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x55745b186ea0 tx=0x7fcd8c018000 comp rx=0 tx=0).stop 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 -- 172.21.15.28:0/1368500444 shutdown_connections 2024-09-13T17:11:23.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b160ed0 0x55745b1415d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 --2- 172.21.15.28:0/1368500444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b14b820 0x55745b188090 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 -- 172.21.15.28:0/1368500444 >> 172.21.15.28:0/1368500444 conn(0x55745b148fe0 msgr2=0x55745b193f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 -- 172.21.15.28:0/1368500444 shutdown_connections 2024-09-13T17:11:23.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.602+0000 7fcda5843740 1 -- 172.21.15.28:0/1368500444 wait complete. 2024-09-13T17:11:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.603+0000 7fcda5843740 1 -- start start 2024-09-13T17:11:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.603+0000 7fcda5843740 1 Processor -- start 2024-09-13T17:11:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b1fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b1900b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b18bfb0 con 0x55745b160ed0 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55745b1fdf50 con 0x55745b14b820 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55745b1fe0f0 con 0x55745b18ed50 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcda5843740 10 client.0 _ll_register_callbacks cb 0x7fcd8c0096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcd93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.604+0000 7fcd93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44258/0 (socket says 172.21.15.28:44258) 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 -- 172.21.15.28:0/3534436103 learned_addr learned my addr 172.21.15.28:0/3534436103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcda0a00640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b1900b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b1fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 -- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 msgr2=0x55745b1900b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b1900b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 -- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 msgr2=0x55745b1fa1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b1fa1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 -- 172.21.15.28:0/3534436103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c0119e0 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcda0a00640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b1900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd93400640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcd84054760 tx=0x7fcd84001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f448fe29f732258 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd89600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84057950 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcd840601c0 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.605+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd94004c30 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd84060380 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 10 client.5838 fetch_fsmap learned FSMap version 35 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55745b04e290 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd94004c30 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 35) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcd84012070 con 0x55745b160ed0 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 10 client.5838 fetch_fsmap finished waiting for FSMap version 35 2024-09-13T17:11:23.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 10 client.5838 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 20 client.5838 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b192160 0x55745b149b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 4 client.5838 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/"}] 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/"}) v1 -- 0x55745b09ff50 con 0x55745b192160 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.606+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b192160 0x55745b149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcd9be00640 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b192160 0x55745b149b40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=134cd3c7e494e543 server_cookie=1b53bdebc066d38 in_seq=0 out_seq=0 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcd89600640 10 client.5838 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcd89600640 1 -- 172.21.15.28:0/3534436103 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55745b09ff50 con 0x55745b192160 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcd89600640 10 client.5838 handle_command_reply: tid=0 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcda5843740 1 client.5838 shutdown 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 msgr2=0x55745b18fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.607+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fcd84054760 tx=0x7fcd84001da0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 shutdown_connections 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55745b18ed50 0x55745b1900b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55745b192160 0x55745b149b40 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55745b160ed0 0x55745b18fb70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 --2- 172.21.15.28:0/3534436103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55745b14b820 0x55745b1fa1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcd89600640 0 client.5838 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 >> 172.21.15.28:0/3534436103 conn(0x55745b148fe0 msgr2=0x55745b193070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 shutdown_connections 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 1 -- 172.21.15.28:0/3534436103 wait complete. 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.608+0000 7fcda5843740 20 client.5838 trim_cache size 0 max 16384 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:23.613 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:23.614 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "2ea0111d-9d29-4c1f-806a-2982bf12f20d", 2024-09-13T17:11:23.614 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:23.614 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:23.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 msgr2=0x7fcd7806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 0x7fcd7806d0f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c009a40 tx=0x7fcd8c009320 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 msgr2=0x7fcd9c155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fcd84001d70 tx=0x7fcd84007660 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 shutdown_connections 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd7806ac30 0x7fcd7806d0f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcd9c1f6050 0x7fcd9c159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd9c1f2c10 0x7fcd9c155350 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 --2- 172.21.15.28:0/3015647659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcd800051d0 0x7fcd9c155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 >> 172.21.15.28:0/3015647659 conn(0x7fcd9c0b0270 msgr2=0x7fcd9c0af1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 shutdown_connections 2024-09-13T17:11:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.609+0000 7fcda2a00640 1 -- 172.21.15.28:0/3015647659 wait complete. 2024-09-13T17:11:23.625 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /' returned '{'return_code': 0, 'scrub_tag': '2ea0111d-9d29-4c1f-806a-2982bf12f20d', 'mode': 'asynchronous'}' 2024-09-13T17:11:23.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:23.626 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_dir_0) 2024-09-13T17:11:23.663 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_0' 2024-09-13T17:11:23.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.878+0000 7f30b3400640 1 Processor -- start 2024-09-13T17:11:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.878+0000 7f30b3400640 1 -- start start 2024-09-13T17:11:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.879+0000 7f30b3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06f970 0x7f30ac15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.879+0000 7f30b3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30ac06c2d0 0x7f30ac06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.879+0000 7f30b3400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ac159d90 con 0x7f30ac1592a0 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.879+0000 7f30b3400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ac15b0b0 con 0x7f30ac06c2d0 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.879+0000 7f30b3400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac06cca0 con 0x7f30ac06f970 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30ac06c2d0 0x7f30ac06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30ac06c2d0 0x7f30ac06c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42486/0 (socket says 172.21.15.28:42486) 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 -- 172.21.15.28:0/465843432 learned_addr learned my addr 172.21.15.28:0/465843432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30a3e00640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06f970 0x7f30ac15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 -- 172.21.15.28:0/465843432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06f970 msgr2=0x7f30ac15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06f970 0x7f30ac15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 -- 172.21.15.28:0/465843432 >> v1:172.21.15.189:6789/0 conn(0x7f30ac1592a0 legacy=0x7f30ac159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30abe00640 1 -- 172.21.15.28:0/465843432 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036343038 0 0) 0x7f30ac159d90 con 0x7f30ac1592a0 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.880+0000 7f30b1400640 1 -- 172.21.15.28:0/465843432 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ac0b2040 con 0x7f30ac06c2d0 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.881+0000 7f30b1400640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06c2d0 0x7f30ac06c760 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009e50 tx=0x7f30a402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340bbea28df2fb28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.881+0000 7f30abe00640 1 -- 172.21.15.28:0/465843432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4031600 con 0x7f30ac06c2d0 2024-09-13T17:11:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30abe00640 1 -- 172.21.15.28:0/465843432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a402d050 con 0x7f30ac06c2d0 2024-09-13T17:11:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30abe00640 1 -- 172.21.15.28:0/465843432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4031980 con 0x7f30ac06c2d0 2024-09-13T17:11:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 -- 172.21.15.28:0/465843432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06c2d0 msgr2=0x7f30ac06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06c2d0 0x7f30ac06c760 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009e50 tx=0x7f30a402a710 comp rx=0 tx=0).stop 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 -- 172.21.15.28:0/465843432 shutdown_connections 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac06c2d0 0x7f30ac06c760 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 --2- 172.21.15.28:0/465843432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06f970 0x7f30ac15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 -- 172.21.15.28:0/465843432 >> 172.21.15.28:0/465843432 conn(0x7f30ac069e80 msgr2=0x7f30ac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.882+0000 7f30b3400640 1 -- 172.21.15.28:0/465843432 shutdown_connections 2024-09-13T17:11:23.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.883+0000 7f30b3400640 1 -- 172.21.15.28:0/465843432 wait complete. 2024-09-13T17:11:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.885+0000 7f30b3400640 1 Processor -- start 2024-09-13T17:11:23.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.885+0000 7f30b3400640 1 -- start start 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 0x7f30ac143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30ac06f970 0x7f30ac12f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac146d50 con 0x7f30ac1592a0 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac12fc20 con 0x7f30ac06f970 2024-09-13T17:11:23.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b3400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ac145630 con 0x7f30ac06c2d0 2024-09-13T17:11:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30a3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30ac06f970 0x7f30ac12f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42502/0 (socket says 172.21.15.28:42502) 2024-09-13T17:11:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.887+0000 7f30b1400640 1 -- 172.21.15.28:0/2040808573 learned_addr learned my addr 172.21.15.28:0/2040808573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:23.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b0a00640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 0x7f30ac143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b1400640 1 -- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 msgr2=0x7f30ac143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b1400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 0x7f30ac143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b1400640 1 -- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30ac06f970 msgr2=0x7f30ac12f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b1400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30ac06f970 0x7f30ac12f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b1400640 1 -- 172.21.15.28:0/2040808573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a4009b30 con 0x7f30ac1592a0 2024-09-13T17:11:23.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.888+0000 7f30b0a00640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 0x7f30ac143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30b1400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009e50 tx=0x7f30a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4003b90 con 0x7f30ac1592a0 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ac144370 con 0x7f30ac1592a0 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ac1448f0 con 0x7f30ac1592a0 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a4003cf0 con 0x7f30ac1592a0 2024-09-13T17:11:23.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.889+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a4004260 con 0x7f30ac1592a0 2024-09-13T17:11:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.891+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3084005250 con 0x7f30ac1592a0 2024-09-13T17:11:23.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.892+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f30a4038050 con 0x7f30ac1592a0 2024-09-13T17:11:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.893+0000 7f30a9600640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 0x7f308806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.893+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f30a40a9bf0 con 0x7f30ac1592a0 2024-09-13T17:11:23.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.893+0000 7f30b0a00640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 0x7f308806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:23.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.894+0000 7f30b0a00640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 0x7f308806d0c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:23.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:23.898+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f30a4074de0 con 0x7f30ac1592a0 2024-09-13T17:11:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.295+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3084005a80 con 0x7f30ac1592a0 2024-09-13T17:11:24.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.295+0000 7f30a9600640 1 -- 172.21.15.28:0/2040808573 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f30a4036090 con 0x7f30ac1592a0 2024-09-13T17:11:24.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:24.297 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:24.297 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 msgr2=0x7f308806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 0x7f308806d0c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f309c000f60 tx=0x7f309c009040 comp rx=0 tx=0).stop 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 msgr2=0x7f30ac1450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009e50 tx=0x7f30a4004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 shutdown_connections 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f308806ac00 0x7f308806d0c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30ac1592a0 0x7f30ac1450f0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30ac06f970 0x7f30ac12f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 --2- 172.21.15.28:0/2040808573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30ac06c2d0 0x7f30ac143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 >> 172.21.15.28:0/2040808573 conn(0x7f30ac069e80 msgr2=0x7f30ac157db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 shutdown_connections 2024-09-13T17:11:24.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.298+0000 7f30b3400640 1 -- 172.21.15.28:0/2040808573 wait complete. 2024-09-13T17:11:24.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/new_dir_0 2024-09-13T17:11:24.476 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:24.476 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:24.476 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:24.488 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_0' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:24.489 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:24.489 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:24.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:24.526 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_0 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T17:11:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:11:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:11:24.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01644 s, 63.8 MB/s 2024-09-13T17:11:24.619 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_0' 2024-09-13T17:11:24.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:24.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.773+0000 7f7e50a00640 1 Processor -- start 2024-09-13T17:11:24.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.774+0000 7f7e50a00640 1 -- start start 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e50a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e4c154a90 0x7f7e4c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e50a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e4c06b360 0x7f7e4c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e50a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c15bb10 con 0x7f7e4c0b0980 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e50a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0b0d80 con 0x7f7e4c154a90 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e50a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c06b900 con 0x7f7e4c06b360 2024-09-13T17:11:24.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e4b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e4c154a90 0x7f7e4c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e4be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e4c0b0980 0x7f7e4c0af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42294/0 (socket says 172.21.15.28:42294) 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e4be00640 1 -- 172.21.15.28:0/2393857717 learned_addr learned my addr 172.21.15.28:0/2393857717 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.775+0000 7f7e42a00640 1 --2- 172.21.15.28:0/2393857717 >> v2:172.21.15.189:3300/0 conn(0x7f7e4c06b360 0x7f7e4c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.776+0000 7f7e4b400640 1 -- 172.21.15.28:0/2393857717 >> v2:172.21.15.189:3300/0 conn(0x7f7e4c06b360 msgr2=0x7f7e4c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.776+0000 7f7e4b400640 1 --2- 172.21.15.28:0/2393857717 >> v2:172.21.15.189:3300/0 conn(0x7f7e4c06b360 0x7f7e4c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.776+0000 7f7e4b400640 1 -- 172.21.15.28:0/2393857717 >> v1:172.21.15.28:6789/0 conn(0x7f7e4c0b0980 legacy=0x7f7e4c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.776+0000 7f7e4b400640 1 -- 172.21.15.28:0/2393857717 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c06bbe0 con 0x7f7e4c154a90 2024-09-13T17:11:24.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.777+0000 7f7e4b400640 1 --2- 172.21.15.28:0/2393857717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c154a90 0x7f7e4c15b5d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c13dab0 tx=0x7f7e3c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb9602993cdc7e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.777+0000 7f7e4aa00640 1 -- 172.21.15.28:0/2393857717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c032440 con 0x7f7e4c154a90 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.777+0000 7f7e4aa00640 1 -- 172.21.15.28:0/2393857717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c0325a0 con 0x7f7e4c154a90 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.777+0000 7f7e4aa00640 1 -- 172.21.15.28:0/2393857717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c032930 con 0x7f7e4c154a90 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 -- 172.21.15.28:0/2393857717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c154a90 msgr2=0x7f7e4c15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 --2- 172.21.15.28:0/2393857717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c154a90 0x7f7e4c15b5d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c13dab0 tx=0x7f7e3c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 -- 172.21.15.28:0/2393857717 shutdown_connections 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 --2- 172.21.15.28:0/2393857717 >> v2:172.21.15.189:3300/0 conn(0x7f7e4c06b360 0x7f7e4c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 --2- 172.21.15.28:0/2393857717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c154a90 0x7f7e4c15b5d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 -- 172.21.15.28:0/2393857717 >> 172.21.15.28:0/2393857717 conn(0x7f7e4c072e40 msgr2=0x7f7e4c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 -- 172.21.15.28:0/2393857717 shutdown_connections 2024-09-13T17:11:24.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.778+0000 7f7e50a00640 1 -- 172.21.15.28:0/2393857717 wait complete. 2024-09-13T17:11:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.779+0000 7f7e50a00640 1 Processor -- start 2024-09-13T17:11:24.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 -- start start 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e4c0b0980 0x7f7e4c0ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e4c154a90 0x7f7e4c0abf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0ad5e0 con 0x7f7e4c06b360 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0ac9d0 con 0x7f7e4c0b0980 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.780+0000 7f7e50a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0accb0 con 0x7f7e4c154a90 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42510/0 (socket says 172.21.15.28:42510) 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 -- 172.21.15.28:0/19799235 learned_addr learned my addr 172.21.15.28:0/19799235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e42a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e4c0b0980 0x7f7e4c0ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4be00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e4c154a90 0x7f7e4c0abf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 -- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e4c154a90 msgr2=0x7f7e4c0abf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e4c154a90 0x7f7e4c0abf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 -- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e4c0b0980 msgr2=0x7f7e4c0ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e4c0b0980 0x7f7e4c0ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 -- 172.21.15.28:0/19799235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3c009400 con 0x7f7e4c06b360 2024-09-13T17:11:24.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.781+0000 7f7e4b400640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c000c00 tx=0x7f7e3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.782+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c030b80 con 0x7f7e4c06b360 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.782+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c032e90 con 0x7f7e4c06b360 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.782+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c006e10 con 0x7f7e4c06b360 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.782+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c0acf90 con 0x7f7e4c06b360 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.782+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c133fa0 con 0x7f7e4c06b360 2024-09-13T17:11:24.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.783+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c15c710 con 0x7f7e4c06b360 2024-09-13T17:11:24.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.784+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e3c039040 con 0x7f7e4c06b360 2024-09-13T17:11:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.785+0000 7f7e43e00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 0x7f7e2806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.785+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e3c0a9c10 con 0x7f7e4c06b360 2024-09-13T17:11:24.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.786+0000 7f7e42a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 0x7f7e2806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:24.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.786+0000 7f7e42a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 0x7f7e2806d0c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c1348e0 tx=0x7f7e34009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:24.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:24.790+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e3c074e90 con 0x7f7e4c06b360 2024-09-13T17:11:25.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.006+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e4c06fe10 con 0x7f7e4c06b360 2024-09-13T17:11:25.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.006+0000 7f7e43e00640 1 -- 172.21.15.28:0/19799235 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f7e3c0784b0 con 0x7f7e4c06b360 2024-09-13T17:11:25.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:25.008 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":35,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:11.937604+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:25.008 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 35 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 msgr2=0x7f7e2806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 0x7f7e2806d0c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c1348e0 tx=0x7f7e34009190 comp rx=0 tx=0).stop 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 msgr2=0x7f7e4c133a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c000c00 tx=0x7f7e3c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 shutdown_connections 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e2806ac00 0x7f7e2806d0c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e4c154a90 0x7f7e4c0abf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e4c0b0980 0x7f7e4c0ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 --2- 172.21.15.28:0/19799235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e4c06b360 0x7f7e4c133a60 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 >> 172.21.15.28:0/19799235 conn(0x7f7e4c072e40 msgr2=0x7f7e4c0608f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 shutdown_connections 2024-09-13T17:11:25.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.009+0000 7f7e50a00640 1 -- 172.21.15.28:0/19799235 wait complete. 2024-09-13T17:11:25.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/new_file_0 2024-09-13T17:11:25.191 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:25.192 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:25.192 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:25.204 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_0' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:25.204 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:11:25.204 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:11:25.204 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:11:25.205 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:25.205 DEBUG:teuthology.orchestra.run.smithi028:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_0").st_ino) 2024-09-13T17:11:25.205 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:11:25.304 INFO:teuthology.orchestra.run.smithi028.stdout:1099511630186 2024-09-13T17:11:25.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-09-13T17:11:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.686+0000 7ff939200640 1 Processor -- start 2024-09-13T17:11:25.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.686+0000 7ff939200640 1 -- start start 2024-09-13T17:11:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.687+0000 7ff939200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff93406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.687+0000 7ff939200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff93406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.687+0000 7ff939200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff93406c8b0 con 0x7ff93415a780 2024-09-13T17:11:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.687+0000 7ff939200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff93406cb90 con 0x7ff93415ab60 2024-09-13T17:11:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.687+0000 7ff939200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340cc980 con 0x7ff934071490 2024-09-13T17:11:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff93415a780 0x7ff934070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42298/0 (socket says 172.21.15.28:42298) 2024-09-13T17:11:25.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931e00640 1 -- 172.21.15.28:0/3545902910 learned_addr learned my addr 172.21.15.28:0/3545902910 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff92be00640 1 --2- 172.21.15.28:0/3545902910 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff93406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff932800640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff93406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229110416 0 0) 0x7ff93406c8b0 con 0x7ff93415a780 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff918003520 con 0x7ff93415a780 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542934256 0 0) 0x7ff918003520 con 0x7ff93415a780 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 msgr2=0x7ff93406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.688+0000 7ff931400640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff93406c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 msgr2=0x7ff93406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 --2- 172.21.15.28:0/3545902910 >> v2:172.21.15.189:3300/0 conn(0x7ff93415ab60 0x7ff93406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9340ccc60 con 0x7ff93415a780 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7ff924001670 con 0x7ff93415a780 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 >> v1:172.21.15.28:6789/0 conn(0x7ff93415a780 legacy=0x7ff934070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff918003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff918003ef0 0x7ff9180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918004d00 0x7ff918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff93406cb90 con 0x7ff918003ef0 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340cc980 con 0x7ff918004d00 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff91800b980 con 0x7ff934071490 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff918003ef0 0x7ff9180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff931e00640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff918003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff932800640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918004d00 0x7ff918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 -- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 msgr2=0x7ff918003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff934071490 0x7ff918003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 -- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918004d00 msgr2=0x7ff918005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918004d00 0x7ff918005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.689+0000 7ff92be00640 1 -- 172.21.15.28:0/3545902910 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff918003520 con 0x7ff918003ef0 2024-09-13T17:11:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.690+0000 7ff92be00640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff918003ef0 0x7ff9180047c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009ac0 tx=0x7ff9180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c1de6674d9768b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c031400 con 0x7ff918003ef0 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c02d050 con 0x7ff918003ef0 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff931400640 1 -- 172.21.15.28:0/3545902910 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c031780 con 0x7ff918003ef0 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 -- 172.21.15.28:0/3545902910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff918003ef0 msgr2=0x7ff9180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 --2- 172.21.15.28:0/3545902910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff918003ef0 0x7ff9180047c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff91c009ac0 tx=0x7ff9180057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff932800640 1 -- 172.21.15.28:0/3545902910 reap_dead start 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 -- 172.21.15.28:0/3545902910 shutdown_connections 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 -- 172.21.15.28:0/3545902910 >> 172.21.15.28:0/3545902910 conn(0x7ff9340b0270 msgr2=0x7ff9340b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 -- 172.21.15.28:0/3545902910 shutdown_connections 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.691+0000 7ff939200640 1 -- 172.21.15.28:0/3545902910 wait complete. 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.693+0000 7ff939200640 1 Processor -- start 2024-09-13T17:11:25.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.693+0000 7ff939200640 1 -- start start 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918003ef0 0x7ff9341f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 0x7ff93414fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9341f5fa0 con 0x7ff93414aec0 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9340b2730 con 0x7ff918003ef0 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff939200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9341f61d0 con 0x7ff9340b2ea0 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff92be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff92be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42518/0 (socket says 172.21.15.28:42518) 2024-09-13T17:11:25.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.694+0000 7ff92be00640 1 -- 172.21.15.28:0/1488572859 learned_addr learned my addr 172.21.15.28:0/1488572859 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff932800640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 0x7ff93414fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff931e00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918003ef0 0x7ff9341f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 -- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 msgr2=0x7ff93414fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 0x7ff93414fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 -- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918003ef0 msgr2=0x7ff9341f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918003ef0 0x7ff9341f5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 -- 172.21.15.28:0/1488572859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9341f64b0 con 0x7ff93414aec0 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff932800640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 0x7ff93414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92be00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff91c02aaf0 tx=0x7ff91c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.695+0000 7ff92a000640 1 -- 172.21.15.28:0/1488572859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c003c90 con 0x7ff93414aec0 2024-09-13T17:11:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.696+0000 7ff92a000640 1 -- 172.21.15.28:0/1488572859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c003df0 con 0x7ff93414aec0 2024-09-13T17:11:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.696+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9341f6790 con 0x7ff93414aec0 2024-09-13T17:11:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.696+0000 7ff92a000640 1 -- 172.21.15.28:0/1488572859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c044650 con 0x7ff93414aec0 2024-09-13T17:11:25.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.696+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff93414f860 con 0x7ff93414aec0 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.698+0000 7ff92a000640 1 -- 172.21.15.28:0/1488572859 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff91c038070 con 0x7ff93414aec0 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.699+0000 7ff92a000640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.699+0000 7ff92a000640 1 -- 172.21.15.28:0/1488572859 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff91c0a9870 con 0x7ff93414aec0 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.700+0000 7ff931e00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.700+0000 7ff93c067740 1 Processor -- start 2024-09-13T17:11:25.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.700+0000 7ff93c067740 1 -- start start 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff931e00640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff924001430 tx=0x7ff924009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d0271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d0245d0 0x56358d0742e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358cf33020 con 0x56358d04e2c0 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358cf90ba0 con 0x56358d026e10 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358cf8e330 con 0x56358d0245d0 2024-09-13T17:11:25.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42524/0 (socket says 172.21.15.28:42524) 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff931e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d0271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff932800640 1 -- 172.21.15.28:0/2192189171 learned_addr learned my addr 172.21.15.28:0/2192189171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.701+0000 7ff92be00640 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d0245d0 0x56358d0742e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 -- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d0245d0 msgr2=0x56358d0742e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d0245d0 0x56358d0742e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 -- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 msgr2=0x56358d0271f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d0271f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 -- 172.21.15.28:0/2192189171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358cf3bcf0 con 0x56358d04e2c0 2024-09-13T17:11:25.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff932800640 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff9340b12e0 tx=0x7ff92c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7207db7cfc49dd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff922a00640 1 -- 172.21.15.28:0/2192189171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00e040 con 0x56358d04e2c0 2024-09-13T17:11:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff922a00640 1 -- 172.21.15.28:0/2192189171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c00f040 con 0x56358d04e2c0 2024-09-13T17:11:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.702+0000 7ff922a00640 1 -- 172.21.15.28:0/2192189171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c013e70 con 0x56358d04e2c0 2024-09-13T17:11:25.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 -- 172.21.15.28:0/2192189171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 msgr2=0x56358d074820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff9340b12e0 tx=0x7ff92c009750 comp rx=0 tx=0).stop 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 -- 172.21.15.28:0/2192189171 shutdown_connections 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d074820 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d0245d0 0x56358d0742e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 --2- 172.21.15.28:0/2192189171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d0271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 -- 172.21.15.28:0/2192189171 >> 172.21.15.28:0/2192189171 conn(0x56358d023540 msgr2=0x56358d06b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 -- 172.21.15.28:0/2192189171 shutdown_connections 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.703+0000 7ff93c067740 1 -- 172.21.15.28:0/2192189171 wait complete. 2024-09-13T17:11:25.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.704+0000 7ff93c067740 1 -- start start 2024-09-13T17:11:25.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.704+0000 7ff93c067740 1 Processor -- start 2024-09-13T17:11:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d020e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d04d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358cf89c20 con 0x56358d04e2c0 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358cf2b1b0 con 0x56358d04d3f0 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358cf2ae50 con 0x56358d026e10 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff93c067740 10 client.0 _ll_register_callbacks cb 0x56358cf83290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42528/0 (socket says 172.21.15.28:42528) 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 -- 172.21.15.28:0/713578456 learned_addr learned my addr 172.21.15.28:0/713578456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 -- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 msgr2=0x56358d020e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d020e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 -- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 msgr2=0x56358d04d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d04d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.705+0000 7ff92be00640 1 -- 172.21.15.28:0/713578456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c009ca0 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff92be00640 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff91c02abc0 tx=0x7ff91c080cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90845b0b81189bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c078b30 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff91c078c90 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56358cfb2eb0 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff91c077dc0 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 10 client.5865 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x56358cf882a0 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56358cfb2eb0 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff91c04c070 con 0x56358d04e2c0 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 10 client.5865 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 10 client.5865 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:25.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 20 client.5865 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.706+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d0499d0 0x56358d049de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.707+0000 7ff93c067740 4 client.5865 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.707+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56358cf7e340 con 0x56358d0499d0 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.707+0000 7ff931e00640 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d0499d0 0x56358d049de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.707+0000 7ff931e00640 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d0499d0 0x56358d049de0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be598296d3b865a server_cookie=e48cf45131c7cefc in_seq=0 out_seq=0 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.707+0000 7ff917400640 10 client.5865 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff917400640 1 -- 172.21.15.28:0/713578456 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56358cf7e340 con 0x56358d0499d0 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff917400640 10 client.5865 handle_command_reply: tid=0 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 client.5865 shutdown 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 msgr2=0x56358d068f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff91c02abc0 tx=0x7ff91c080cd0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 shutdown_connections 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d04d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d0499d0 0x56358d049de0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d068f50 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 --2- 172.21.15.28:0/713578456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d020e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff917400640 0 client.5865 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.708+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 >> 172.21.15.28:0/713578456 conn(0x56358d0245d0 msgr2=0x56358d04b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.709+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 shutdown_connections 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.709+0000 7ff93c067740 1 -- 172.21.15.28:0/713578456 wait complete. 2024-09-13T17:11:25.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.709+0000 7ff93c067740 20 client.5865 trim_cache size 0 max 16384 2024-09-13T17:11:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.720+0000 7ff93c067740 1 Processor -- start 2024-09-13T17:11:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.720+0000 7ff93c067740 1 -- start start 2024-09-13T17:11:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d08de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d04d3f0 0x56358d020500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358ccf1040 con 0x56358d04e2c0 2024-09-13T17:11:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358d026c70 con 0x56358d026e10 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358d043600 con 0x56358d04d3f0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42532/0 (socket says 172.21.15.28:42532) 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.721+0000 7ff932800640 1 -- 172.21.15.28:0/1816032002 learned_addr learned my addr 172.21.15.28:0/1816032002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff931e00640 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d08de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff92be00640 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d04d3f0 0x56358d020500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 -- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d04d3f0 msgr2=0x56358d020500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d04d3f0 0x56358d020500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 -- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 msgr2=0x56358d08de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d08de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 -- 172.21.15.28:0/1816032002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56358cfbc200 con 0x56358d04e2c0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff932800640 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x56358d06e8f0 tx=0x7ff92c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d382fc172ae00d18 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff923400640 1 -- 172.21.15.28:0/1816032002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00e050 con 0x56358d04e2c0 2024-09-13T17:11:25.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff923400640 1 -- 172.21.15.28:0/1816032002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c016650 con 0x56358d04e2c0 2024-09-13T17:11:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.722+0000 7ff923400640 1 -- 172.21.15.28:0/1816032002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c016a00 con 0x56358d04e2c0 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 -- 172.21.15.28:0/1816032002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 msgr2=0x56358d04ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x56358d06e8f0 tx=0x7ff92c016000 comp rx=0 tx=0).stop 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 -- 172.21.15.28:0/1816032002 shutdown_connections 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d04ba00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d04d3f0 0x56358d020500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 --2- 172.21.15.28:0/1816032002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d026e10 0x56358d08de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 -- 172.21.15.28:0/1816032002 >> 172.21.15.28:0/1816032002 conn(0x56358d0245d0 msgr2=0x56358d06b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 -- 172.21.15.28:0/1816032002 shutdown_connections 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.723+0000 7ff93c067740 1 -- 172.21.15.28:0/1816032002 wait complete. 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.724+0000 7ff93c067740 1 -- start start 2024-09-13T17:11:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.724+0000 7ff93c067740 1 Processor -- start 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d048d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d0d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358d067d40 con 0x56358d04e2c0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56358d0681a0 con 0x56358d04d3f0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56358d06a730 con 0x56358d026e10 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff93c067740 10 client.0 _ll_register_callbacks cb 0x7ff91c078230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42540/0 (socket says 172.21.15.28:42540) 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 -- 172.21.15.28:0/1499939956 learned_addr learned my addr 172.21.15.28:0/1499939956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff931e00640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d048d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff92be00640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d0d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 -- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 msgr2=0x56358d048d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d048d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 -- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 msgr2=0x56358d0d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d0d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff932800640 1 -- 172.21.15.28:0/1499939956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56358cf3bcf0 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.725+0000 7ff92be00640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d0d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff932800640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009000 tx=0x7ff92c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=effcc8a8b9c40282 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff917400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92c00e040 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff92c013e70 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff92c00f040 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff92c02c420 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff93c067740 10 client.5871 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.726+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x56358cf2a5f0 con 0x56358d04e2c0 2024-09-13T17:11:25.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92c00f040 con 0x56358d04e2c0 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff92c009030 con 0x56358d04e2c0 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 10 client.5871 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 10 client.5871 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 20 client.5871 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d06c9f0 0x56358d0d59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 4 client.5871 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56358cc7d4e0 con 0x56358d06c9f0 2024-09-13T17:11:25.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff931e00640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d06c9f0 0x56358d0d59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff931e00640 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d06c9f0 0x56358d0d59e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61cbff9528c2cca1 server_cookie=b62c68b5626ded05 in_seq=0 out_seq=0 2024-09-13T17:11:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.727+0000 7ff917400640 10 client.5871 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff917400640 1 -- 172.21.15.28:0/1499939956 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56358cc7d4e0 con 0x56358d06c9f0 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff917400640 10 client.5871 handle_command_reply: tid=0 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 client.5871 shutdown 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 msgr2=0x56358d06c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009000 tx=0x7ff92c004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 shutdown_connections 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x56358d06c9f0 0x56358d0d59e0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56358d04e2c0 0x56358d06c4b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff917400640 0 client.5871 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358d04d3f0 0x56358d0d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 --2- 172.21.15.28:0/1499939956 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56358d026e10 0x56358d048d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 >> 172.21.15.28:0/1499939956 conn(0x56358d0245d0 msgr2=0x56358d046310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 shutdown_connections 2024-09-13T17:11:25.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 1 -- 172.21.15.28:0/1499939956 wait complete. 2024-09-13T17:11:25.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.728+0000 7ff93c067740 20 client.5871 trim_cache size 0 max 16384 2024-09-13T17:11:25.734 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 msgr2=0x7ff91006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff924001430 tx=0x7ff924009040 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 msgr2=0x7ff93414f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff91c02aaf0 tx=0x7ff91c004730 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 shutdown_connections 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff91006ac30 0x7ff91006d0f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff9340b2ea0 0x7ff93414fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff93414aec0 0x7ff93414f320 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 --2- 172.21.15.28:0/1488572859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff918003ef0 0x7ff9341f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 >> 172.21.15.28:0/1488572859 conn(0x7ff9340b0270 msgr2=0x7ff9340cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 shutdown_connections 2024-09-13T17:11:25.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.729+0000 7ff939200640 1 -- 172.21.15.28:0/1488572859 wait complete. 2024-09-13T17:11:25.745 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data rmxattr 1000000096a.00000000 parent 2024-09-13T17:11:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.820+0000 7ff48f631f40 1 Processor -- start 2024-09-13T17:11:25.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.820+0000 7ff48f631f40 1 -- start start 2024-09-13T17:11:25.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e92254f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e922532c40 0x55e9224a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e922421400 con 0x55e92247a8a0 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9223bcfe0 con 0x55e922532c40 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48f631f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e9223bcc80 con 0x55e922472c00 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42548/0 (socket says 172.21.15.28:42548) 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 -- 172.21.15.28:0/4283961133 learned_addr learned my addr 172.21.15.28:0/4283961133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48c800640 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e92254f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48d200640 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e922532c40 0x55e9224a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 -- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 msgr2=0x55e92254f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e92254f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 -- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e922532c40 msgr2=0x55e9224a88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e922532c40 0x55e9224a88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.821+0000 7ff48be00640 1 -- 172.21.15.28:0/4283961133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff480005de0 con 0x55e92247a8a0 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48be00640 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff478009910 tx=0x7ff478005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee39a5fc5cb1745 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48aa00640 1 -- 172.21.15.28:0/4283961133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47800f040 con 0x55e92247a8a0 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48aa00640 1 -- 172.21.15.28:0/4283961133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4780089b0 con 0x55e92247a8a0 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48aa00640 1 -- 172.21.15.28:0/4283961133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff478008d60 con 0x55e92247a8a0 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48f631f40 1 -- 172.21.15.28:0/4283961133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 msgr2=0x55e922530b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.822+0000 7ff48f631f40 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff478009910 tx=0x7ff478005a00 comp rx=0 tx=0).stop 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 -- 172.21.15.28:0/4283961133 shutdown_connections 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e922532c40 0x55e9224a88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e92247a8a0 0x55e922530b90 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 --2- 172.21.15.28:0/4283961133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e92254f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 -- 172.21.15.28:0/4283961133 >> 172.21.15.28:0/4283961133 conn(0x55e9224a5270 msgr2=0x55e9224a6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 -- 172.21.15.28:0/4283961133 shutdown_connections 2024-09-13T17:11:25.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 -- 172.21.15.28:0/4283961133 wait complete. 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.823+0000 7ff48f631f40 1 Processor -- start 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 -- start start 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e9224c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e92247a8a0 0x55e92252fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 0x55e9224c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9223b38f0 con 0x55e922532c40 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9224a6150 con 0x55e92247a8a0 2024-09-13T17:11:25.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.824+0000 7ff48f631f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e922543130 con 0x55e922472c00 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 0x55e9224c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 0x55e9224c2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42554/0 (socket says 172.21.15.28:42554) 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48be00640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e92247a8a0 0x55e92252fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 -- 172.21.15.28:0/2323103662 learned_addr learned my addr 172.21.15.28:0/2323103662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48c800640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e9224c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 -- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 msgr2=0x55e9224c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e922472c00 0x55e9224c2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 -- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e92247a8a0 msgr2=0x55e92252fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e92247a8a0 0x55e92252fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.825+0000 7ff48d200640 1 -- 172.21.15.28:0/2323103662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478009510 con 0x55e922532c40 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff48d200640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 0x55e9224c2d70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff484004e50 tx=0x7ff4840018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff489600640 1 -- 172.21.15.28:0/2323103662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff484011380 con 0x55e922532c40 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff489600640 1 -- 172.21.15.28:0/2323103662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff484011500 con 0x55e922532c40 2024-09-13T17:11:25.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e9223cbb20 con 0x55e922532c40 2024-09-13T17:11:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff489600640 1 -- 172.21.15.28:0/2323103662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4840118b0 con 0x55e922532c40 2024-09-13T17:11:25.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.826+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9223bef60 con 0x55e922532c40 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.828+0000 7ff489600640 1 -- 172.21.15.28:0/2323103662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff484011a10 con 0x55e922532c40 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.828+0000 7ff489600640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46c0695e0 0x7ff46c06baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.829+0000 7ff489600640 1 -- 172.21.15.28:0/2323103662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff484087300 con 0x55e922532c40 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.829+0000 7ff48f631f40 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x55e922548bf0 0x55e922544fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.829+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 --> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] -- osd_op(unknown.0.0:1 7.11 7:88e5eea5:::1000000096a.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e9225454e0 con 0x55e922548bf0 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.829+0000 7ff48c800640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46c0695e0 0x7ff46c06baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.829+0000 7ff48be00640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x55e922548bf0 0x55e922544fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.830+0000 7ff48c800640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46c0695e0 0x7ff46c06baa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff480057000 tx=0x7ff480052000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.830+0000 7ff48be00640 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x55e922548bf0 0x55e922544fa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48be00640 1 -- 172.21.15.28:0/2323103662 <== osd.7 v2:172.21.15.189:6822/2656371491 1 ==== osd_op_reply(1 1000000096a.00000000 [rmxattr] v48'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7ff478019020 con 0x55e922548bf0 2024-09-13T17:11:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x55e922548bf0 msgr2=0x55e922544fa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x55e922548bf0 0x55e922544fa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46c0695e0 msgr2=0x7ff46c06baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46c0695e0 0x7ff46c06baa0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff480057000 tx=0x7ff480052000 comp rx=0 tx=0).stop 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 msgr2=0x55e9224c2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.831+0000 7ff48f631f40 1 --2- 172.21.15.28:0/2323103662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e922532c40 0x55e9224c2d70 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff484004e50 tx=0x7ff4840018e0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.832+0000 7ff48d200640 1 -- 172.21.15.28:0/2323103662 reap_dead start 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.832+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 shutdown_connections 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.832+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 >> 172.21.15.28:0/2323103662 conn(0x55e9224a5270 msgr2=0x55e922549b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.832+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 shutdown_connections 2024-09-13T17:11:25.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.832+0000 7ff48f631f40 1 -- 172.21.15.28:0/2323103662 wait complete. 2024-09-13T17:11:25.837 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_0' 2024-09-13T17:11:25.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/new_file_0 2024-09-13T17:11:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.994+0000 7fba35000640 1 Processor -- start 2024-09-13T17:11:25.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.994+0000 7fba35000640 1 -- start start 2024-09-13T17:11:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.995+0000 7fba35000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3015a780 0x7fba30070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.995+0000 7fba35000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba3006c250 0x7fba300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.995+0000 7fba35000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3006c8b0 con 0x7fba3015ab60 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.995+0000 7fba35000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba3006c6a0 con 0x7fba3006c250 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.995+0000 7fba35000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3006cb90 con 0x7fba3015a780 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba3006c250 0x7fba300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3015a780 0x7fba30070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba3006c250 0x7fba300712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42566/0 (socket says 172.21.15.28:42566) 2024-09-13T17:11:25.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 -- 172.21.15.28:0/850893382 learned_addr learned my addr 172.21.15.28:0/850893382 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 -- 172.21.15.28:0/850893382 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3015a780 msgr2=0x7fba30070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 --2- 172.21.15.28:0/850893382 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3015a780 0x7fba30070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 -- 172.21.15.28:0/850893382 >> v1:172.21.15.189:6789/0 conn(0x7fba3015ab60 legacy=0x7fba3006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.996+0000 7fba2e800640 1 -- 172.21.15.28:0/850893382 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba300cc980 con 0x7fba3006c250 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.997+0000 7fba2e800640 1 --2- 172.21.15.28:0/850893382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3006c250 0x7fba300712c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fba28009e50 tx=0x7fba2802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39ea8067bda9a67 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.997+0000 7fba2ca00640 1 -- 172.21.15.28:0/850893382 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28031600 con 0x7fba3006c250 2024-09-13T17:11:25.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.997+0000 7fba2ca00640 1 -- 172.21.15.28:0/850893382 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2802d050 con 0x7fba3006c250 2024-09-13T17:11:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.997+0000 7fba2ca00640 1 -- 172.21.15.28:0/850893382 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28031980 con 0x7fba3006c250 2024-09-13T17:11:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 -- 172.21.15.28:0/850893382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3006c250 msgr2=0x7fba300712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:25.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 --2- 172.21.15.28:0/850893382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3006c250 0x7fba300712c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fba28009e50 tx=0x7fba2802a710 comp rx=0 tx=0).stop 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 -- 172.21.15.28:0/850893382 shutdown_connections 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 --2- 172.21.15.28:0/850893382 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3006c250 0x7fba300712c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 --2- 172.21.15.28:0/850893382 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3015a780 0x7fba30070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 -- 172.21.15.28:0/850893382 >> 172.21.15.28:0/850893382 conn(0x7fba300b0270 msgr2=0x7fba300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 -- 172.21.15.28:0/850893382 shutdown_connections 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:25.998+0000 7fba35000640 1 -- 172.21.15.28:0/850893382 wait complete. 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.000+0000 7fba35000640 1 Processor -- start 2024-09-13T17:11:26.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.000+0000 7fba35000640 1 -- start start 2024-09-13T17:11:26.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 0x7fba30155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 0x7fba301438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba30143e00 con 0x7fba3015ab60 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba300caff0 con 0x7fba3015a780 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba35000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba301f1420 con 0x7fba3006c250 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42582/0 (socket says 172.21.15.28:42582) 2024-09-13T17:11:26.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 -- 172.21.15.28:0/1606474411 learned_addr learned my addr 172.21.15.28:0/1606474411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2de00640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 0x7fba30155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2d400640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 0x7fba301438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 -- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 msgr2=0x7fba30155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 0x7fba30155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 -- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 msgr2=0x7fba301438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 0x7fba301438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2e800640 1 -- 172.21.15.28:0/1606474411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28009b30 con 0x7fba3015ab60 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2de00640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 0x7fba30155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.001+0000 7fba2d400640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 0x7fba301438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba2e800640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fba28009e50 tx=0x7fba28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba26000640 1 -- 172.21.15.28:0/1606474411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2803d860 con 0x7fba3015ab60 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba26000640 1 -- 172.21.15.28:0/1606474411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2803d9c0 con 0x7fba3015ab60 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba30155de0 con 0x7fba3015ab60 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba26000640 1 -- 172.21.15.28:0/1606474411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2803dd70 con 0x7fba3015ab60 2024-09-13T17:11:26.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.002+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba301563a0 con 0x7fba3015ab60 2024-09-13T17:11:26.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.004+0000 7fba26000640 1 -- 172.21.15.28:0/1606474411 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fba2803b400 con 0x7fba3015ab60 2024-09-13T17:11:26.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.005+0000 7fba26000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 0x7fba1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.005+0000 7fba26000640 1 -- 172.21.15.28:0/1606474411 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba280aa210 con 0x7fba3015ab60 2024-09-13T17:11:26.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.006+0000 7fba2de00640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 0x7fba1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.006+0000 7fba37dd5740 1 Processor -- start 2024-09-13T17:11:26.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.006+0000 7fba37dd5740 1 -- start start 2024-09-13T17:11:26.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.006+0000 7fba2de00640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 0x7fba1006d0c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fba30067f30 tx=0x7fba20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d061c0 0x55a089d0a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 0x55a089d34300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089c15e60 con 0x55a089d4e6e0 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089c73f10 con 0x55a089d0adf0 2024-09-13T17:11:26.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a089c716a0 con 0x55a089d061c0 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42596/0 (socket says 172.21.15.28:42596) 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba2de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d061c0 0x55a089d0a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.007+0000 7fba2e800640 1 -- 172.21.15.28:0/2995633184 learned_addr learned my addr 172.21.15.28:0/2995633184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2d400640 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 0x55a089d34300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 -- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d061c0 msgr2=0x55a089d0a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d061c0 0x55a089d0a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 -- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 msgr2=0x55a089d34300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 0x55a089d34300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 -- 172.21.15.28:0/2995633184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a089c0ebf0 con 0x55a089d4e6e0 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2d400640 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 0x55a089d34300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba2e800640 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fba28079030 tx=0x7fba280095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7cdcbcf7104952 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba1aa00640 1 -- 172.21.15.28:0/2995633184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2807ced0 con 0x55a089d4e6e0 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba1aa00640 1 -- 172.21.15.28:0/2995633184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28081490 con 0x55a089d4e6e0 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.008+0000 7fba1aa00640 1 -- 172.21.15.28:0/2995633184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28077440 con 0x55a089d4e6e0 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 -- 172.21.15.28:0/2995633184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 msgr2=0x55a089d34840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fba28079030 tx=0x7fba280095d0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 -- 172.21.15.28:0/2995633184 shutdown_connections 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d4e6e0 0x55a089d34840 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d0adf0 0x55a089d34300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2995633184 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d061c0 0x55a089d0a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 -- 172.21.15.28:0/2995633184 >> 172.21.15.28:0/2995633184 conn(0x55a089d06e50 msgr2=0x55a089d4f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 -- 172.21.15.28:0/2995633184 shutdown_connections 2024-09-13T17:11:26.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.009+0000 7fba37dd5740 1 -- 172.21.15.28:0/2995633184 wait complete. 2024-09-13T17:11:26.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.010+0000 7fba37dd5740 1 -- start start 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.010+0000 7fba37dd5740 1 Processor -- start 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d63020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d63560 0x55a089d3fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089c6cf90 con 0x55a089d0adf0 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089c2b3c0 con 0x55a089d4e6e0 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a089c2b060 con 0x55a089d63560 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba37dd5740 10 client.0 _ll_register_callbacks cb 0x55a089c5cd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba2de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba2de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42604/0 (socket says 172.21.15.28:42604) 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba2de00640 1 -- 172.21.15.28:0/3427338271 learned_addr learned my addr 172.21.15.28:0/3427338271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba2d400640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d63020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.011+0000 7fba2e800640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d63560 0x55a089d3fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 -- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d63560 msgr2=0x55a089d3fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d63560 0x55a089d3fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 -- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 msgr2=0x55a089d63020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d63020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 -- 172.21.15.28:0/3427338271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba301563a0 con 0x55a089d0adf0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2d400640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba2de00640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fba20010f40 tx=0x7fba200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb80f1b60ef9c450 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba0f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200177c0 con 0x55a089d0adf0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba20017920 con 0x55a089d0adf0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a089c5fab0 con 0x55a089d0adf0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba20017ae0 con 0x55a089d0adf0 2024-09-13T17:11:26.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba37dd5740 10 client.5895 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.012+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55a089c6b610 con 0x55a089d0adf0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a089c5fab0 con 0x55a089d0adf0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fba20022070 con 0x55a089d0adf0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 10 client.5895 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 10 client.5895 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 20 client.5895 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d376f0 0x55a089d37b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 4 client.5895 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0899608e0 con 0x55a089d376f0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.013+0000 7fba2d400640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d376f0 0x55a089d37b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba2d400640 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d376f0 0x55a089d37b00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74d8d09eb96f6b10 server_cookie=38429f465ac3dae3 in_seq=0 out_seq=0 2024-09-13T17:11:26.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba0f400640 10 client.5895 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba0f400640 1 -- 172.21.15.28:0/3427338271 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55a0899608e0 con 0x55a089d376f0 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba0f400640 10 client.5895 handle_command_reply: tid=0 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba37dd5740 1 client.5895 shutdown 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 msgr2=0x55a089d4c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.014+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fba20010f40 tx=0x7fba200046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 shutdown_connections 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d63560 0x55a089d3fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d63020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d376f0 0x55a089d37b00 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 --2- 172.21.15.28:0/3427338271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d4c5e0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 >> 172.21.15.28:0/3427338271 conn(0x55a089d061c0 msgr2=0x55a089d4d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba0f400640 0 client.5895 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 shutdown_connections 2024-09-13T17:11:26.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 1 -- 172.21.15.28:0/3427338271 wait complete. 2024-09-13T17:11:26.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.015+0000 7fba37dd5740 20 client.5895 trim_cache size 0 max 16384 2024-09-13T17:11:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.030+0000 7fba37dd5740 1 Processor -- start 2024-09-13T17:11:26.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.030+0000 7fba37dd5740 1 -- start start 2024-09-13T17:11:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 0x55a089d6c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 0x55a089d37b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089d62bb0 con 0x55a089d0adf0 2024-09-13T17:11:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089d26930 con 0x55a089d63560 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0899d4440 con 0x55a089d4e6e0 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42620/0 (socket says 172.21.15.28:42620) 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 -- 172.21.15.28:0/946262795 learned_addr learned my addr 172.21.15.28:0/946262795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2d400640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 0x55a089d6c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2e800640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 0x55a089d37b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 -- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 msgr2=0x55a089d6c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 0x55a089d6c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 -- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 msgr2=0x55a089d37b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 0x55a089d37b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2de00640 1 -- 172.21.15.28:0/946262795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a089c0d430 con 0x55a089d0adf0 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2d400640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 0x55a089d6c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.031+0000 7fba2e800640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 0x55a089d37b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.032+0000 7fba2de00640 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fba20008010 tx=0x7fba20022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8c56ed1d0e1d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.032+0000 7fba1b400640 1 -- 172.21.15.28:0/946262795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba200179c0 con 0x55a089d0adf0 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.032+0000 7fba1b400640 1 -- 172.21.15.28:0/946262795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2001b070 con 0x55a089d0adf0 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.032+0000 7fba1b400640 1 -- 172.21.15.28:0/946262795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba20014610 con 0x55a089d0adf0 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 -- 172.21.15.28:0/946262795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 msgr2=0x55a089d6c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fba20008010 tx=0x7fba20022000 comp rx=0 tx=0).stop 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 -- 172.21.15.28:0/946262795 shutdown_connections 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d63560 0x55a089d37b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d4e6e0 0x55a089d6c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 --2- 172.21.15.28:0/946262795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d0adf0 0x55a089d6c2a0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 -- 172.21.15.28:0/946262795 >> 172.21.15.28:0/946262795 conn(0x55a089d061c0 msgr2=0x55a089d4d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 -- 172.21.15.28:0/946262795 shutdown_connections 2024-09-13T17:11:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.033+0000 7fba37dd5740 1 -- 172.21.15.28:0/946262795 wait complete. 2024-09-13T17:11:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.034+0000 7fba37dd5740 1 -- start start 2024-09-13T17:11:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.034+0000 7fba37dd5740 1 Processor -- start 2024-09-13T17:11:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d0adf0 0x55a089da5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d339c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089d33f00 con 0x55a089d63560 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a089d33290 con 0x55a089d4e6e0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a089d09410 con 0x55a089d0adf0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba37dd5740 10 client.0 _ll_register_callbacks cb 0x7fba28075480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba2e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42634/0 (socket says 172.21.15.28:42634) 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba2e800640 1 -- 172.21.15.28:0/2346197490 learned_addr learned my addr 172.21.15.28:0/2346197490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba2de00640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d0adf0 0x55a089da5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.035+0000 7fba2d400640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d339c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 -- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d0adf0 msgr2=0x55a089da5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d0adf0 0x55a089da5d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 -- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 msgr2=0x55a089d339c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d339c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 -- 172.21.15.28:0/2346197490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20017e70 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2d400640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba2e800640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fba280755a0 tx=0x7fba280aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25890795f8c3db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba0f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba28078e70 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba28081490 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba280a7040 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba2807ced0 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba37dd5740 10 client.5901 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.036+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x55a089c32a00 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba28081490 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fba2804d020 con 0x55a089d63560 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 10 client.5901 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 10 client.5901 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 20 client.5901 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d08c30 0x55a089d50c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 4 client.5901 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_0"}] 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_0"}) v1 -- 0x55a089971be0 con 0x55a089d08c30 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba2de00640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d08c30 0x55a089d50c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba2de00640 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d08c30 0x55a089d50c00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d29d45cb5c30b7da server_cookie=a59bd95e9f4dba1 in_seq=0 out_seq=0 2024-09-13T17:11:26.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.037+0000 7fba0f400640 10 client.5901 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba0f400640 1 -- 172.21.15.28:0/2346197490 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a089971be0 con 0x55a089d08c30 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba0f400640 10 client.5901 handle_command_reply: tid=0 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 client.5901 shutdown 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 msgr2=0x55a089d086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fba280755a0 tx=0x7fba280aa040 comp rx=0 tx=0).stop 2024-09-13T17:11:26.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 shutdown_connections 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a089d08c30 0x55a089d50c00 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a089d63560 0x55a089d086f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a089d4e6e0 0x55a089d339c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 --2- 172.21.15.28:0/2346197490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a089d0adf0 0x55a089da5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba0f400640 0 client.5901 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 >> 172.21.15.28:0/2346197490 conn(0x55a089d061c0 msgr2=0x55a089d283e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 shutdown_connections 2024-09-13T17:11:26.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.038+0000 7fba37dd5740 1 -- 172.21.15.28:0/2346197490 wait complete. 2024-09-13T17:11:26.043 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:11:26.039+0000 7f203e200640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096a [2,head] /scrub_test_path/ceph-qa-suite/new_file_0 auth v679 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5566=0-4194304@1} caps={5566=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@3},l=5566 | request=0 lock=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55b5cf7acb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1000000096a:[<0x10000000001/new_file_0 v679>,<0x10000000000/ceph-qa-suite v53>,<0x1/scrub_test_path v15>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61} 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba37dd5740 20 client.5901 trim_cache size 0 max 16384 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "80aabcc8-0bea-4055-ab72-6df3920381ff", 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:26.044 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 msgr2=0x7fba1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 0x7fba1006d0c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fba30067f30 tx=0x7fba20008040 comp rx=0 tx=0).stop 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 msgr2=0x7fba301f0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fba28009e50 tx=0x7fba28004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 shutdown_connections 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba1006ac00 0x7fba1006d0c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba3015ab60 0x7fba301f0ee0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba3015a780 0x7fba301438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 --2- 172.21.15.28:0/1606474411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba3006c250 0x7fba30155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 >> 172.21.15.28:0/1606474411 conn(0x7fba300b0270 msgr2=0x7fba300afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 shutdown_connections 2024-09-13T17:11:26.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.039+0000 7fba35000640 1 -- 172.21.15.28:0/1606474411 wait complete. 2024-09-13T17:11:26.054 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_0' returned '{'return_code': 0, 'scrub_tag': '80aabcc8-0bea-4055-ab72-6df3920381ff', 'mode': 'asynchronous'}' 2024-09-13T17:11:26.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T17:11:26.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.211+0000 7fa0bea00640 1 Processor -- start 2024-09-13T17:11:26.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.211+0000 7fa0bea00640 1 -- start start 2024-09-13T17:11:26.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b815a780 0x7fa0b8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0b806c250 0x7fa0b80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b806c8b0 con 0x7fa0b815ab60 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b806c6a0 con 0x7fa0b806c250 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b806cb90 con 0x7fa0b815a780 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0bca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0b806c250 0x7fa0b80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0afe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0b815ab60 0x7fa0b806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42314/0 (socket says 172.21.15.28:42314) 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.212+0000 7fa0afe00640 1 -- 172.21.15.28:0/4002366936 learned_addr learned my addr 172.21.15.28:0/4002366936 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b815a780 0x7fa0b8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0bca00640 1 -- 172.21.15.28:0/4002366936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b815a780 msgr2=0x7fa0b8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0bca00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b815a780 0x7fa0b8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0bca00640 1 -- 172.21.15.28:0/4002366936 >> v1:172.21.15.28:6789/0 conn(0x7fa0b815ab60 legacy=0x7fa0b806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0bca00640 1 -- 172.21.15.28:0/4002366936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b80cc980 con 0x7fa0b806c250 2024-09-13T17:11:26.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.213+0000 7fa0b7400640 1 -- 172.21.15.28:0/4002366936 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245788744 0 0) 0x7fa0b806c8b0 con 0x7fa0b815ab60 2024-09-13T17:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.214+0000 7fa0bca00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b806c250 0x7fa0b80712c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009d90 tx=0x7fa0b002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b84d1c0abdf4da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.214+0000 7fa0b7400640 1 -- 172.21.15.28:0/4002366936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0007740 con 0x7fa0b806c250 2024-09-13T17:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.214+0000 7fa0b7400640 1 -- 172.21.15.28:0/4002366936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b002d050 con 0x7fa0b806c250 2024-09-13T17:11:26.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.214+0000 7fa0b7400640 1 -- 172.21.15.28:0/4002366936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0007ac0 con 0x7fa0b806c250 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 -- 172.21.15.28:0/4002366936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b806c250 msgr2=0x7fa0b80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b806c250 0x7fa0b80712c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009d90 tx=0x7fa0b002a660 comp rx=0 tx=0).stop 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 -- 172.21.15.28:0/4002366936 shutdown_connections 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b806c250 0x7fa0b80712c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4002366936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b815a780 0x7fa0b8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 -- 172.21.15.28:0/4002366936 >> 172.21.15.28:0/4002366936 conn(0x7fa0b80b0270 msgr2=0x7fa0b80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 -- 172.21.15.28:0/4002366936 shutdown_connections 2024-09-13T17:11:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.215+0000 7fa0bea00640 1 -- 172.21.15.28:0/4002366936 wait complete. 2024-09-13T17:11:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.216+0000 7fa0bea00640 1 Processor -- start 2024-09-13T17:11:26.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.217+0000 7fa0bea00640 1 -- start start 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.217+0000 7fa0bea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b806c250 0x7fa0b814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.217+0000 7fa0bea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b815a780 0x7fa0b80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b80b1e80 con 0x7fa0b815ab60 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b814a900 con 0x7fa0b815a780 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b814ede0 con 0x7fa0b806c250 2024-09-13T17:11:26.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42666/0 (socket says 172.21.15.28:42666) 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 -- 172.21.15.28:0/4144193931 learned_addr learned my addr 172.21.15.28:0/4144193931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0afe00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b815a780 0x7fa0b80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b806c250 0x7fa0b814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 -- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b806c250 msgr2=0x7fa0b814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b806c250 0x7fa0b814a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 -- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b815a780 msgr2=0x7fa0b80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b815a780 0x7fa0b80ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.218+0000 7fa0bca00640 1 -- 172.21.15.28:0/4144193931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b0009a70 con 0x7fa0b815ab60 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0bca00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009eb0 tx=0x7fa0b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0b4c00640 1 -- 172.21.15.28:0/4144193931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0031890 con 0x7fa0b815ab60 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0b4c00640 1 -- 172.21.15.28:0/4144193931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b00319f0 con 0x7fa0b815ab60 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0b4c00640 1 -- 172.21.15.28:0/4144193931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0031d60 con 0x7fa0b815ab60 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b81542c0 con 0x7fa0b815ab60 2024-09-13T17:11:26.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.219+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b8154880 con 0x7fa0b815ab60 2024-09-13T17:11:26.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.221+0000 7fa0b4c00640 1 -- 172.21.15.28:0/4144193931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0b003c420 con 0x7fa0b815ab60 2024-09-13T17:11:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.223+0000 7fa0b4c00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.223+0000 7fa0b4c00640 1 -- 172.21.15.28:0/4144193931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0b00aac00 con 0x7fa0b815ab60 2024-09-13T17:11:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.223+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.223+0000 7fa0c18cf740 1 Processor -- start 2024-09-13T17:11:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.223+0000 7fa0c18cf740 1 -- start start 2024-09-13T17:11:26.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006fd0 tx=0x7fa0a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e07598420 0x555e075e8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075c2110 0x555e075e8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e074a16a0 con 0x555e0759ac60 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e07504cc0 con 0x555e075c2110 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.224+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e07502450 con 0x555e07598420 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42670/0 (socket says 172.21.15.28:42670) 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 -- 172.21.15.28:0/2832594638 learned_addr learned my addr 172.21.15.28:0/2832594638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0afe00640 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e07598420 0x555e075e8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0bca00640 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075c2110 0x555e075e8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 -- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e07598420 msgr2=0x555e075e8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e07598420 0x555e075e8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 -- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075c2110 msgr2=0x555e075e8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075c2110 0x555e075e8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 -- 172.21.15.28:0/2832594638 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e074ce2e0 con 0x555e0759ac60 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.225+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8007fd0 tx=0x7fa0a8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1dddc4c9210a29 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0acc00640 1 -- 172.21.15.28:0/2832594638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8016ca0 con 0x555e0759ac60 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0acc00640 1 -- 172.21.15.28:0/2832594638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a801a070 con 0x555e0759ac60 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0acc00640 1 -- 172.21.15.28:0/2832594638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8021440 con 0x555e0759ac60 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 -- 172.21.15.28:0/2832594638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 msgr2=0x555e0759b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8007fd0 tx=0x7fa0a8013000 comp rx=0 tx=0).stop 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 -- 172.21.15.28:0/2832594638 shutdown_connections 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075c2110 0x555e075e8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e07598420 0x555e075e8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/2832594638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e0759b040 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 -- 172.21.15.28:0/2832594638 >> 172.21.15.28:0/2832594638 conn(0x555e07597390 msgr2=0x555e075df840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 -- 172.21.15.28:0/2832594638 shutdown_connections 2024-09-13T17:11:26.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.226+0000 7fa0c18cf740 1 -- 172.21.15.28:0/2832594638 wait complete. 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.227+0000 7fa0c18cf740 1 -- start start 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.227+0000 7fa0c18cf740 1 Processor -- start 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e075e6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 0x555e075e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e074fdd40 con 0x555e075c2110 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0749f280 con 0x555e0759ac60 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e0749ef20 con 0x555e075e74e0 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0c18cf740 10 client.0 _ll_register_callbacks cb 0x555e074f73b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:26.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.228+0000 7fa0afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42672/0 (socket says 172.21.15.28:42672) 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 -- 172.21.15.28:0/153528376 learned_addr learned my addr 172.21.15.28:0/153528376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0bca00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 0x555e075e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e075e6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 -- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 msgr2=0x555e075e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 0x555e075e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 -- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 msgr2=0x555e075e6fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e075e6fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 -- 172.21.15.28:0/153528376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a800e040 con 0x555e075c2110 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0bca00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 0x555e075e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e075e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0afe00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa0b814e850 tx=0x7fa0a00015e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4469662435bd3f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0a5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:26.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00145b0 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0a000c780 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.229+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e071f14e0 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0a000c940 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 10 client.5916 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x555e074fc3c0 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a0024020 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa0a0012050 con 0x555e075c2110 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 10 client.5916 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 10 client.5916 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 20 client.5916 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e075f9880 0x555e075fbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 4 client.5916 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e072027e0 con 0x555e075f9880 2024-09-13T17:11:26.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.230+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e075f9880 0x555e075fbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.231+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e075f9880 0x555e075fbc70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9a2e7ac05a6c76 server_cookie=716b0dd31106b493 in_seq=0 out_seq=0 2024-09-13T17:11:26.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.231+0000 7fa0a5600640 10 client.5916 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0a5600640 1 -- 172.21.15.28:0/153528376 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x555e072027e0 con 0x555e075f9880 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0a5600640 10 client.5916 handle_command_reply: tid=0 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 client.5916 shutdown 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 msgr2=0x555e075e4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa0b814e850 tx=0x7fa0a00015e0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 shutdown_connections 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075e74e0 0x555e075e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e075f9880 0x555e075fbc70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075c2110 0x555e075e4690 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/153528376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e075e6fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 >> 172.21.15.28:0/153528376 conn(0x555e07598420 msgr2=0x555e075dedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0a5600640 0 client.5916 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 shutdown_connections 2024-09-13T17:11:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 1 -- 172.21.15.28:0/153528376 wait complete. 2024-09-13T17:11:26.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.232+0000 7fa0c18cf740 20 client.5916 trim_cache size 0 max 16384 2024-09-13T17:11:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.247+0000 7fa0c18cf740 1 Processor -- start 2024-09-13T17:11:26.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.247+0000 7fa0c18cf740 1 -- start start 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075be750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075e74e0 0x555e075bec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e07265040 con 0x555e0759ac60 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0759a410 con 0x555e075e74e0 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e075dd5b0 con 0x555e075c2110 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42678/0 (socket says 172.21.15.28:42678) 2024-09-13T17:11:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.248+0000 7fa0b7e00640 1 -- 172.21.15.28:0/553953902 learned_addr learned my addr 172.21.15.28:0/553953902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 -- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 msgr2=0x555e075be750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075be750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 -- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075e74e0 msgr2=0x555e075bec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075e74e0 0x555e075bec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 -- 172.21.15.28:0/553953902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e0749e610 con 0x555e0759ac60 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8022000 tx=0x7fa0a8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c83cafeaefb96cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0ad600640 1 -- 172.21.15.28:0/553953902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8016de0 con 0x555e0759ac60 2024-09-13T17:11:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0ad600640 1 -- 172.21.15.28:0/553953902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a8035be0 con 0x555e0759ac60 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.249+0000 7fa0ad600640 1 -- 172.21.15.28:0/553953902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8004140 con 0x555e0759ac60 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- 172.21.15.28:0/553953902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 msgr2=0x555e075e3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8022000 tx=0x7fa0a8015000 comp rx=0 tx=0).stop 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- 172.21.15.28:0/553953902 shutdown_connections 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e075e74e0 0x555e075bec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075be750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/553953902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e0759ac60 0x555e075e3340 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- 172.21.15.28:0/553953902 >> 172.21.15.28:0/553953902 conn(0x555e07598420 msgr2=0x555e075ded80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- 172.21.15.28:0/553953902 shutdown_connections 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- 172.21.15.28:0/553953902 wait complete. 2024-09-13T17:11:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.250+0000 7fa0c18cf740 1 -- start start 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.251+0000 7fa0c18cf740 1 Processor -- start 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.251+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e07648fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.251+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075e9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0c18cf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e075c1d50 con 0x555e075e74e0 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0764dcf0 con 0x555e0759ac60 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0c18cf740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555e07649930 con 0x555e075c2110 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0c18cf740 10 client.0 _ll_register_callbacks cb 0x7fa0a00072d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42694/0 (socket says 172.21.15.28:42694) 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 -- 172.21.15.28:0/146617153 learned_addr learned my addr 172.21.15.28:0/146617153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e07648fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0afe00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075e9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 -- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 msgr2=0x555e075e9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075e9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 -- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 msgr2=0x555e07648fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e07648fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0bca00640 1 -- 172.21.15.28:0/146617153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e074ce2e0 con 0x555e075e74e0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.252+0000 7fa0afe00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075e9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0bca00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa0b002ac50 tx=0x7fa0b007d8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8305bfb950c7686a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0077f10 con 0x555e075e74e0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0b0078fa0 con 0x555e075e74e0 2024-09-13T17:11:26.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0a801a070 con 0x555e075e74e0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0b0079160 con 0x555e075e74e0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 10 client.5922 fetch_fsmap learned FSMap version 36 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=2+}) v3 -- 0x555e074c3b20 con 0x555e075e74e0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a801a070 con 0x555e075e74e0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 36) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa0b004c070 con 0x555e075e74e0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 10 client.5922 fetch_fsmap finished waiting for FSMap version 36 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 10 client.5922 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 20 client.5922 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e07649520 0x555e0764cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 4 client.5922 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.253+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555e07255e30 con 0x555e07649520 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e07649520 0x555e0764cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:26.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0b7e00640 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e07649520 0x555e0764cd80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c2af10e3dc488a6 server_cookie=1f818a89e5570bdc in_seq=0 out_seq=0 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0a5600640 10 client.5922 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0a5600640 1 -- 172.21.15.28:0/146617153 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+284 (crc 0 0 0) 0x555e07255e30 con 0x555e07649520 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0a5600640 10 client.5922 handle_command_reply: tid=0 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.254+0000 7fa0c18cf740 1 client.5922 shutdown 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 msgr2=0x555e075ea3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fa0b002ac50 tx=0x7fa0b007d8f0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 shutdown_connections 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x555e07649520 0x555e0764cd80 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555e075e74e0 0x555e075ea3b0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555e075c2110 0x555e075e9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 --2- 172.21.15.28:0/146617153 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555e0759ac60 0x555e07648fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0a5600640 0 client.5922 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 >> 172.21.15.28:0/146617153 conn(0x555e07598420 msgr2=0x555e076003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 shutdown_connections 2024-09-13T17:11:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 1 -- 172.21.15.28:0/146617153 wait complete. 2024-09-13T17:11:26.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.255+0000 7fa0c18cf740 20 client.5922 trim_cache size 0 max 16384 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "80aabcc8-0bea-4055-ab72-6df3920381ff": { 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_0", 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "80aabcc8-0bea-4055-ab72-6df3920381ff", 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "" 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:26.261 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 msgr2=0x7fa09406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006fd0 tx=0x7fa0a8008040 comp rx=0 tx=0).stop 2024-09-13T17:11:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 msgr2=0x7fa0b80acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:26.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0009eb0 tx=0x7fa0b0004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 shutdown_connections 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b815ab60 0x7fa0b80acd90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b815a780 0x7fa0b80ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 --2- 172.21.15.28:0/4144193931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b806c250 0x7fa0b814a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 >> 172.21.15.28:0/4144193931 conn(0x7fa0b80b0270 msgr2=0x7fa0b806f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 shutdown_connections 2024-09-13T17:11:26.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:26.256+0000 7fa0bea00640 1 -- 172.21.15.28:0/4144193931 wait complete. 2024-09-13T17:11:29.039 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:11:29.038+0000 7f58f6400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-13T17:11:31.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T17:11:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.441+0000 7fd691e00640 1 Processor -- start 2024-09-13T17:11:31.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.441+0000 7fd691e00640 1 -- start start 2024-09-13T17:11:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.442+0000 7fd691e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 0x7fd68c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.442+0000 7fd691e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd68c15cd80 0x7fd68c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.442+0000 7fd691e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd68c06f920 con 0x7fd68c06f3f0 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.442+0000 7fd691e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd68c15b290 con 0x7fd68c15cd80 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.442+0000 7fd691e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c15b570 con 0x7fd68c159770 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd68c15cd80 0x7fd68c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd68c15cd80 0x7fd68c070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42702/0 (socket says 172.21.15.28:42702) 2024-09-13T17:11:31.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 -- 172.21.15.28:0/3891228346 learned_addr learned my addr 172.21.15.28:0/3891228346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 0x7fd68c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 -- 172.21.15.28:0/3891228346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 msgr2=0x7fd68c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 0x7fd68c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 -- 172.21.15.28:0/3891228346 >> v1:172.21.15.189:6789/0 conn(0x7fd68c06f3f0 legacy=0x7fd68c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68b400640 1 -- 172.21.15.28:0/3891228346 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd68c15b850 con 0x7fd68c15cd80 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.443+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 0x7fd68c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.444+0000 7fd68b400640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c15cd80 0x7fd68c070260 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd684009e50 tx=0x7fd68402a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc9207a13c20a7cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.444+0000 7fd689600640 1 -- 172.21.15.28:0/3891228346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684031600 con 0x7fd68c15cd80 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.444+0000 7fd689600640 1 -- 172.21.15.28:0/3891228346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68402d050 con 0x7fd68c15cd80 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.444+0000 7fd689600640 1 -- 172.21.15.28:0/3891228346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684031980 con 0x7fd68c15cd80 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 -- 172.21.15.28:0/3891228346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c15cd80 msgr2=0x7fd68c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c15cd80 0x7fd68c070260 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd684009e50 tx=0x7fd68402a640 comp rx=0 tx=0).stop 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 -- 172.21.15.28:0/3891228346 shutdown_connections 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c15cd80 0x7fd68c070260 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 --2- 172.21.15.28:0/3891228346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c159770 0x7fd68c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 -- 172.21.15.28:0/3891228346 >> 172.21.15.28:0/3891228346 conn(0x7fd68c0b0270 msgr2=0x7fd68c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 -- 172.21.15.28:0/3891228346 shutdown_connections 2024-09-13T17:11:31.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.445+0000 7fd691e00640 1 -- 172.21.15.28:0/3891228346 wait complete. 2024-09-13T17:11:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.447+0000 7fd691e00640 1 Processor -- start 2024-09-13T17:11:31.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.447+0000 7fd691e00640 1 -- start start 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 0x7fd68c0cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c15cd80 0x7fd68c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0ccf10 con 0x7fd68c06f3f0 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c0cb990 con 0x7fd68c159770 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd691e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd68c1eccd0 con 0x7fd68c15cd80 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42714/0 (socket says 172.21.15.28:42714) 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68aa00640 1 -- 172.21.15.28:0/3966316975 learned_addr learned my addr 172.21.15.28:0/3966316975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68b400640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c15cd80 0x7fd68c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68a000640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 0x7fd68c0cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.448+0000 7fd68aa00640 1 -- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c15cd80 msgr2=0x7fd68c1547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c15cd80 0x7fd68c1547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68aa00640 1 -- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 msgr2=0x7fd68c0cc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 0x7fd68c0cc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68aa00640 1 -- 172.21.15.28:0/3966316975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd684009b30 con 0x7fd68c06f3f0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68a000640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 0x7fd68c0cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00c820 tx=0x7fd67c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd683400640 1 -- 172.21.15.28:0/3966316975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c007a30 con 0x7fd68c06f3f0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd683400640 1 -- 172.21.15.28:0/3966316975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c007b90 con 0x7fd68c06f3f0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd683400640 1 -- 172.21.15.28:0/3966316975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00ce70 con 0x7fd68c06f3f0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd68c0ae260 con 0x7fd68c06f3f0 2024-09-13T17:11:31.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.449+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c0cc5a0 con 0x7fd68c06f3f0 2024-09-13T17:11:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.451+0000 7fd683400640 1 -- 172.21.15.28:0/3966316975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd67c0113f0 con 0x7fd68c06f3f0 2024-09-13T17:11:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.453+0000 7fd683400640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 0x7fd66c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.453+0000 7fd683400640 1 -- 172.21.15.28:0/3966316975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd67c089490 con 0x7fd68c06f3f0 2024-09-13T17:11:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.453+0000 7fd68a000640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 0x7fd66c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.453+0000 7fd694cd1740 1 Processor -- start 2024-09-13T17:11:31.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.453+0000 7fd694cd1740 1 -- start start 2024-09-13T17:11:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd68a000640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 0x7fd66c06d0c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd68c06b3e0 tx=0x7fd678008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aac89040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac86420 0x5649aacd6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aab8f6a0 con 0x5649aacb0110 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aabf2cc0 con 0x5649aac86420 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.454+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649aabf0450 con 0x5649aac88c60 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42716/0 (socket says 172.21.15.28:42716) 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 -- 172.21.15.28:0/4195700923 learned_addr learned my addr 172.21.15.28:0/4195700923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68a000640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac86420 0x5649aacd6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68aa00640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aac89040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 -- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 msgr2=0x5649aac89040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aac89040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 -- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac86420 msgr2=0x5649aacd6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac86420 0x5649aacd6130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 -- 172.21.15.28:0/4195700923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649aabbc2e0 con 0x5649aacb0110 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68aa00640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aac89040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:31.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd68b400640 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd684034040 tx=0x7fd68402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3770cbdcc2c261 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd677e00640 1 -- 172.21.15.28:0/4195700923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684031700 con 0x5649aacb0110 2024-09-13T17:11:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd677e00640 1 -- 172.21.15.28:0/4195700923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd684031860 con 0x5649aacb0110 2024-09-13T17:11:31.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.455+0000 7fd677e00640 1 -- 172.21.15.28:0/4195700923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684031c10 con 0x5649aacb0110 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 -- 172.21.15.28:0/4195700923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 msgr2=0x5649aacd6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd684034040 tx=0x7fd68402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 -- 172.21.15.28:0/4195700923 shutdown_connections 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb0110 0x5649aacd6670 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac86420 0x5649aacd6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 --2- 172.21.15.28:0/4195700923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aac89040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 -- 172.21.15.28:0/4195700923 >> 172.21.15.28:0/4195700923 conn(0x5649aac85390 msgr2=0x5649aaccd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 -- 172.21.15.28:0/4195700923 shutdown_connections 2024-09-13T17:11:31.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.456+0000 7fd694cd1740 1 -- 172.21.15.28:0/4195700923 wait complete. 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.457+0000 7fd694cd1740 1 -- start start 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.457+0000 7fd694cd1740 1 Processor -- start 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac88c60 0x5649aacb1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacd32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aabebd40 con 0x5649aacb1db0 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aab8d280 con 0x5649aac88c60 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649aab8cf20 con 0x5649aacb0110 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd694cd1740 10 client.0 _ll_register_callbacks cb 0x5649aabe53b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42718/0 (socket says 172.21.15.28:42718) 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd68b400640 1 -- 172.21.15.28:0/3581418925 learned_addr learned my addr 172.21.15.28:0/3581418925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac88c60 0x5649aacb1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.458+0000 7fd68a000640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacd32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 -- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 msgr2=0x5649aacd32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacd32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 -- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac88c60 msgr2=0x5649aacb1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac88c60 0x5649aacb1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 -- 172.21.15.28:0/3581418925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd68c15b850 con 0x5649aacb1db0 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd68b400640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd684009f70 tx=0x7fd684004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc43f9cfd65b7f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd674c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd684003da0 con 0x5649aacb1db0 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649aa8df4e0 con 0x5649aacb1db0 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.459+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd684004040 con 0x5649aacb1db0 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd684004200 con 0x5649aacb1db0 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 10 client.5934 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:31.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x5649aabea3c0 con 0x5649aacb1db0 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649aa8df4e0 con 0x5649aacb1db0 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd684004360 con 0x5649aacb1db0 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 10 client.5934 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 10 client.5934 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 20 client.5934 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aacaeac0 0x5649aacaeed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 4 client.5934 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649aa8f07e0 con 0x5649aacaeac0 2024-09-13T17:11:31.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.460+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aacaeac0 0x5649aacaeed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.461+0000 7fd68aa00640 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aacaeac0 0x5649aacaeed0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7f47f2589c05f58 server_cookie=b91cea2b4e32be25 in_seq=0 out_seq=0 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.461+0000 7fd674c00640 10 client.5934 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.461+0000 7fd674c00640 1 -- 172.21.15.28:0/3581418925 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5649aa8f07e0 con 0x5649aacaeac0 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.461+0000 7fd674c00640 10 client.5934 handle_command_reply: tid=0 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.461+0000 7fd694cd1740 1 client.5934 shutdown 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 msgr2=0x5649aacd3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fd684009f70 tx=0x7fd684004ab0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 shutdown_connections 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aacaeac0 0x5649aacaeed0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3800 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacd32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 --2- 172.21.15.28:0/3581418925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aac88c60 0x5649aacb1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd674c00640 0 client.5934 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 >> 172.21.15.28:0/3581418925 conn(0x5649aac86420 msgr2=0x5649aacce330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 shutdown_connections 2024-09-13T17:11:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 1 -- 172.21.15.28:0/3581418925 wait complete. 2024-09-13T17:11:31.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.462+0000 7fd694cd1740 20 client.5934 trim_cache size 0 max 16384 2024-09-13T17:11:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.477+0000 7fd694cd1740 1 Processor -- start 2024-09-13T17:11:31.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.477+0000 7fd694cd1740 1 -- start start 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacad210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 0x5649aacd3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aa953040 con 0x5649aac88c60 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aaccd840 con 0x5649aacb1db0 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649aaccdd30 con 0x5649aacb0110 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42728/0 (socket says 172.21.15.28:42728) 2024-09-13T17:11:31.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68aa00640 1 -- 172.21.15.28:0/1444721079 learned_addr learned my addr 172.21.15.28:0/1444721079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68aa00640 1 -- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 msgr2=0x5649aacad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68aa00640 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.478+0000 7fd68b400640 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 0x5649aacd3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd68aa00640 1 -- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 msgr2=0x5649aacd3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd68aa00640 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 0x5649aacd3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd68aa00640 1 -- 172.21.15.28:0/1444721079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649aab8c610 con 0x5649aac88c60 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd68b400640 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 0x5649aacd3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd68aa00640 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00cfd0 tx=0x7fd67c089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b959697c54e105b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd680c00640 1 -- 172.21.15.28:0/1444721079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c056460 con 0x5649aac88c60 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd680c00640 1 -- 172.21.15.28:0/1444721079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67c087a30 con 0x5649aac88c60 2024-09-13T17:11:31.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.479+0000 7fd680c00640 1 -- 172.21.15.28:0/1444721079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd67c054a30 con 0x5649aac88c60 2024-09-13T17:11:31.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- 172.21.15.28:0/1444721079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 msgr2=0x5649aaccd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00cfd0 tx=0x7fd67c089040 comp rx=0 tx=0).stop 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- 172.21.15.28:0/1444721079 shutdown_connections 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb1db0 0x5649aacd3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aacb0110 0x5649aacad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 --2- 172.21.15.28:0/1444721079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aac88c60 0x5649aaccd300 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- 172.21.15.28:0/1444721079 >> 172.21.15.28:0/1444721079 conn(0x5649aac86420 msgr2=0x5649aacd4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- 172.21.15.28:0/1444721079 shutdown_connections 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- 172.21.15.28:0/1444721079 wait complete. 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 -- start start 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.480+0000 7fd694cd1740 1 Processor -- start 2024-09-13T17:11:31.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aaccf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb0110 0x5649aacd3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aacb4b80 con 0x5649aacb1db0 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649aad37e50 con 0x5649aacb0110 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5649aaccea80 con 0x5649aac88c60 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.481+0000 7fd694cd1740 10 client.0 _ll_register_callbacks cb 0x7fd684040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42734/0 (socket says 172.21.15.28:42734) 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb0110 0x5649aacd3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 -- 172.21.15.28:0/194376901 learned_addr learned my addr 172.21.15.28:0/194376901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68aa00640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aaccf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 -- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 msgr2=0x5649aaccf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aaccf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 -- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb0110 msgr2=0x5649aacd3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb0110 0x5649aacd3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 -- 172.21.15.28:0/194376901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c06d040 con 0x5649aacb1db0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd68b400640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd684031830 tx=0x7fd684009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af897cf418bd045 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.482+0000 7fd674c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68403c650 con 0x5649aacb1db0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd684004470 con 0x5649aacb1db0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649aacd4ae0 con 0x5649aacb1db0 2024-09-13T17:11:31.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd684007080 con 0x5649aacb1db0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 10 client.5940 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x5649aabb1b20 con 0x5649aacb1db0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5649aacd4ae0 con 0x5649aacb1db0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd68403c7b0 con 0x5649aacb1db0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 10 client.5940 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 10 client.5940 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 20 client.5940 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aad352d0 0x5649aace6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 4 client.5940 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fd684031860 con 0x5649aad352d0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.483+0000 7fd68aa00640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aad352d0 0x5649aace6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd68aa00640 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aad352d0 0x5649aace6070 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2606ec4208e6bb62 server_cookie=9aae3d32dfb133f3 in_seq=0 out_seq=0 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd674c00640 10 client.5940 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd674c00640 1 -- 172.21.15.28:0/194376901 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fd684031860 con 0x5649aad352d0 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd674c00640 10 client.5940 handle_command_reply: tid=0 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 client.5940 shutdown 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 msgr2=0x5649aacd3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fd684031830 tx=0x7fd684009000 comp rx=0 tx=0).stop 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 shutdown_connections 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5649aad352d0 0x5649aace6070 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5649aacb1db0 0x5649aacd3650 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5649aacb0110 0x5649aacd3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 --2- 172.21.15.28:0/194376901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5649aac88c60 0x5649aaccf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd674c00640 0 client.5940 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.484+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 >> 172.21.15.28:0/194376901 conn(0x5649aac86420 msgr2=0x5649aacd1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 shutdown_connections 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd694cd1740 1 -- 172.21.15.28:0/194376901 wait complete. 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd694cd1740 20 client.5940 trim_cache size 0 max 16384 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:11:31.492 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 msgr2=0x7fd66c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 0x7fd66c06d0c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fd68c06b3e0 tx=0x7fd678008040 comp rx=0 tx=0).stop 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 msgr2=0x7fd68c1ec790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00c820 tx=0x7fd67c00cc20 comp rx=0 tx=0).stop 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 shutdown_connections 2024-09-13T17:11:31.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac00 0x7fd66c06d0c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd68c15cd80 0x7fd68c1547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd68c159770 0x7fd68c0cc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 --2- 172.21.15.28:0/3966316975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd68c06f3f0 0x7fd68c1ec790 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 >> 172.21.15.28:0/3966316975 conn(0x7fd68c0b0270 msgr2=0x7fd68c157570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.485+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 shutdown_connections 2024-09-13T17:11:31.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.486+0000 7fd691e00640 1 -- 172.21.15.28:0/3966316975 wait complete. 2024-09-13T17:11:31.502 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:11:31.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-09-13T17:11:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.658+0000 7ff03ec00640 1 Processor -- start 2024-09-13T17:11:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.658+0000 7ff03ec00640 1 -- start start 2024-09-13T17:11:31.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.659+0000 7ff03ec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff038070430 0x7ff03815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.659+0000 7ff03ec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff03806f3f0 0x7ff03815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.659+0000 7ff03ec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03815ab60 con 0x7ff038159770 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.659+0000 7ff03ec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff03806f920 con 0x7ff038070430 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.659+0000 7ff03ec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03815b290 con 0x7ff03806f3f0 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff038070430 0x7ff03815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff03806f3f0 0x7ff03815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff038070430 0x7ff03815bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:59394/0 (socket says 172.21.15.28:59394) 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 -- 172.21.15.28:0/2042995512 learned_addr learned my addr 172.21.15.28:0/2042995512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff037e00640 1 --1- 172.21.15.28:0/2042995512 >> v1:172.21.15.189:6789/0 conn(0x7ff038159770 0x7ff03806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:46554/0 (socket says 172.21.15.28:46554) 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 -- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff03806f3f0 msgr2=0x7ff03815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 --2- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff03806f3f0 0x7ff03815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 -- 172.21.15.28:0/2042995512 >> v1:172.21.15.189:6789/0 conn(0x7ff038159770 legacy=0x7ff03806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff02f400640 1 -- 172.21.15.28:0/2042995512 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03815b570 con 0x7ff038070430 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.660+0000 7ff037400640 1 -- 172.21.15.28:0/2042995512 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242964925 0 0) 0x7ff028001110 con 0x7ff038159770 2024-09-13T17:11:31.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.661+0000 7ff02f400640 1 --2- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff03815bd70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff020009870 tx=0x7ff02002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eda6e5cb19ee5c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.661+0000 7ff037400640 1 -- 172.21.15.28:0/2042995512 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020032440 con 0x7ff038070430 2024-09-13T17:11:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff037400640 1 -- 172.21.15.28:0/2042995512 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0200325a0 con 0x7ff038070430 2024-09-13T17:11:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff037400640 1 -- 172.21.15.28:0/2042995512 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff020032930 con 0x7ff038070430 2024-09-13T17:11:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 -- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 msgr2=0x7ff03815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 --2- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff03815bd70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff020009870 tx=0x7ff02002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 -- 172.21.15.28:0/2042995512 shutdown_connections 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 --2- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff03806f3f0 0x7ff03815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 --2- 172.21.15.28:0/2042995512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff03815bd70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 -- 172.21.15.28:0/2042995512 >> 172.21.15.28:0/2042995512 conn(0x7ff0380b0270 msgr2=0x7ff0380b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 -- 172.21.15.28:0/2042995512 shutdown_connections 2024-09-13T17:11:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.662+0000 7ff03ec00640 1 -- 172.21.15.28:0/2042995512 wait complete. 2024-09-13T17:11:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.664+0000 7ff03ec00640 1 Processor -- start 2024-09-13T17:11:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.664+0000 7ff03ec00640 1 -- start start 2024-09-13T17:11:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff0381e1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff038159770 0x7ff0381f1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381e3370 con 0x7ff03806f3f0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381e2470 con 0x7ff038070430 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff03ec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381e26f0 con 0x7ff038159770 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff037e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff037e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42740/0 (socket says 172.21.15.28:42740) 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff037e00640 1 -- 172.21.15.28:0/3241180674 learned_addr learned my addr 172.21.15.28:0/3241180674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.665+0000 7ff02f400640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff0381e1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff03cc00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff038159770 0x7ff0381f1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 -- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff038159770 msgr2=0x7ff0381f1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff038159770 0x7ff0381f1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 -- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 msgr2=0x7ff0381e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff0381e1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 -- 172.21.15.28:0/3241180674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200094d0 con 0x7ff03806f3f0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff037e00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c430 tx=0x7ff02800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff034c00640 1 -- 172.21.15.28:0/3241180674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0280074e0 con 0x7ff03806f3f0 2024-09-13T17:11:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff034c00640 1 -- 172.21.15.28:0/3241180674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff028007660 con 0x7ff03806f3f0 2024-09-13T17:11:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff034c00640 1 -- 172.21.15.28:0/3241180674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff028004a20 con 0x7ff03806f3f0 2024-09-13T17:11:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0381e29d0 con 0x7ff03806f3f0 2024-09-13T17:11:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.666+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0381e2f90 con 0x7ff03806f3f0 2024-09-13T17:11:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.669+0000 7ff034c00640 1 -- 172.21.15.28:0/3241180674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff028004b80 con 0x7ff03806f3f0 2024-09-13T17:11:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.669+0000 7ff034c00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 0x7ff01406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.669+0000 7ff034c00640 1 -- 172.21.15.28:0/3241180674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff028088f30 con 0x7ff03806f3f0 2024-09-13T17:11:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.670+0000 7ff02f400640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 0x7ff01406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.670+0000 7ff0419f1740 1 Processor -- start 2024-09-13T17:11:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.670+0000 7ff0419f1740 1 -- start start 2024-09-13T17:11:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.670+0000 7ff02f400640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 0x7ff01406d0f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff020034000 tx=0x7ff02002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c4749b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 0x557c474e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c473a7020 con 0x557c474c22c0 2024-09-13T17:11:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c47404ba0 con 0x557c474985d0 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c47402330 con 0x557c4749ae10 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42752/0 (socket says 172.21.15.28:42752) 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.671+0000 7ff03cc00640 1 -- 172.21.15.28:0/1444682870 learned_addr learned my addr 172.21.15.28:0/1444682870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff02f400640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 0x557c474e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff037e00640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c4749b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 -- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 msgr2=0x557c4749b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c4749b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 -- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 msgr2=0x557c474e82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 0x557c474e82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 -- 172.21.15.28:0/1444682870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c473afcf0 con 0x557c474c22c0 2024-09-13T17:11:31.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff02f400640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 0x557c474e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff0300074c0 tx=0x7ff030009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c537e1849aa0fd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff02cc00640 1 -- 172.21.15.28:0/1444682870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03000e030 con 0x557c474c22c0 2024-09-13T17:11:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff02cc00640 1 -- 172.21.15.28:0/1444682870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030015600 con 0x557c474c22c0 2024-09-13T17:11:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.672+0000 7ff02cc00640 1 -- 172.21.15.28:0/1444682870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030009d00 con 0x557c474c22c0 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 -- 172.21.15.28:0/1444682870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 msgr2=0x557c474e8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ff0300074c0 tx=0x7ff030009750 comp rx=0 tx=0).stop 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 -- 172.21.15.28:0/1444682870 shutdown_connections 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e8820 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474985d0 0x557c474e82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1444682870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c4749b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 -- 172.21.15.28:0/1444682870 >> 172.21.15.28:0/1444682870 conn(0x557c47497540 msgr2=0x557c474df9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 -- 172.21.15.28:0/1444682870 shutdown_connections 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.673+0000 7ff0419f1740 1 -- 172.21.15.28:0/1444682870 wait complete. 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.674+0000 7ff0419f1740 1 -- start start 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.674+0000 7ff0419f1740 1 Processor -- start 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c474dcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 0x557c474954c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c473fdc20 con 0x557c474c0e80 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4739f1b0 con 0x557c474c22c0 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c4739ee50 con 0x557c4749ae10 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff0419f1740 10 client.0 _ll_register_callbacks cb 0x557c473f7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42756/0 (socket says 172.21.15.28:42756) 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff03cc00640 1 -- 172.21.15.28:0/1980760263 learned_addr learned my addr 172.21.15.28:0/1980760263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff037e00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c474dcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.675+0000 7ff02f400640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 0x557c474954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 -- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 msgr2=0x557c474dcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c474dcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 -- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 msgr2=0x557c474954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 0x557c474954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 -- 172.21.15.28:0/1980760263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c473fc2a0 con 0x557c474c0e80 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff02f400640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 0x557c474954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff03cc00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff030018040 tx=0x7ff030004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ef024039400f89 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff025600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03001e830 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff030024030 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c47426eb0 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.676+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff030009410 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 10 client.5949 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x557c4739e5f0 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557c47426eb0 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff030014070 con 0x557c474c0e80 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 10 client.5949 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 10 client.5949 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 20 client.5949 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c474ae8c0 0x557c474aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 4 client.5949 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c473f2340 con 0x557c474ae8c0 2024-09-13T17:11:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.677+0000 7ff037e00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c474ae8c0 0x557c474aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff037e00640 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c474ae8c0 0x557c474aeca0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23d6693113cf2339 server_cookie=de143021867a8278 in_seq=0 out_seq=0 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff025600640 10 client.5949 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff025600640 1 -- 172.21.15.28:0/1980760263 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x557c473f2340 con 0x557c474ae8c0 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff025600640 10 client.5949 handle_command_reply: tid=0 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff0419f1740 1 client.5949 shutdown 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 msgr2=0x557c47495a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.678+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ff030018040 tx=0x7ff030004ab0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 shutdown_connections 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c474ae8c0 0x557c474aeca0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c47495a00 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c22c0 0x557c474954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 --2- 172.21.15.28:0/1980760263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c474dcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff025600640 0 client.5949 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 >> 172.21.15.28:0/1980760263 conn(0x557c474985d0 msgr2=0x557c474ddd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 shutdown_connections 2024-09-13T17:11:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 1 -- 172.21.15.28:0/1980760263 wait complete. 2024-09-13T17:11:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.679+0000 7ff0419f1740 20 client.5949 trim_cache size 0 max 16384 2024-09-13T17:11:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.691+0000 7ff0419f1740 1 Processor -- start 2024-09-13T17:11:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.691+0000 7ff0419f1740 1 -- start start 2024-09-13T17:11:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.691+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c47495ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c0e80 0x557c47490c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff0419f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c47165040 con 0x557c474c22c0 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c474dfcd0 con 0x557c474c0e80 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c4749b1f0 con 0x557c4749ae10 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42758/0 (socket says 172.21.15.28:42758) 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 -- 172.21.15.28:0/3014154818 learned_addr learned my addr 172.21.15.28:0/3014154818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff02f400640 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c0e80 0x557c47490c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff037e00640 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c47495ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 -- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 msgr2=0x557c47495ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c47495ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 -- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c0e80 msgr2=0x557c47490c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c0e80 0x557c47490c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 -- 172.21.15.28:0/3014154818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c47430200 con 0x557c474c22c0 2024-09-13T17:11:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.692+0000 7ff03cc00640 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff030002230 tx=0x7ff030020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59841880f8ba7ab5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff02d600640 1 -- 172.21.15.28:0/3014154818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03000e040 con 0x557c474c22c0 2024-09-13T17:11:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff02d600640 1 -- 172.21.15.28:0/3014154818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff03001e830 con 0x557c474c22c0 2024-09-13T17:11:31.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff02d600640 1 -- 172.21.15.28:0/3014154818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff030029920 con 0x557c474c22c0 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 -- 172.21.15.28:0/3014154818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 msgr2=0x557c474e2eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff030002230 tx=0x7ff030020070 comp rx=0 tx=0).stop 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 -- 172.21.15.28:0/3014154818 shutdown_connections 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c22c0 0x557c474e2eb0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c474c0e80 0x557c47490c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 --2- 172.21.15.28:0/3014154818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c4749ae10 0x557c47495ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 -- 172.21.15.28:0/3014154818 >> 172.21.15.28:0/3014154818 conn(0x557c474985d0 msgr2=0x557c47499df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 -- 172.21.15.28:0/3014154818 shutdown_connections 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.693+0000 7ff0419f1740 1 -- 172.21.15.28:0/3014154818 wait complete. 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.694+0000 7ff0419f1740 1 -- start start 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.694+0000 7ff0419f1740 1 Processor -- start 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 0x557c4754d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 0x557c474db8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4754e070 con 0x557c474c0e80 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4754a320 con 0x557c4749ae10 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c474bffe0 con 0x557c474c22c0 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff0419f1740 10 client.0 _ll_register_callbacks cb 0x7ff0300073c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff02f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff02f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42774/0 (socket says 172.21.15.28:42774) 2024-09-13T17:11:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.695+0000 7ff02f400640 1 -- 172.21.15.28:0/228413889 learned_addr learned my addr 172.21.15.28:0/228413889 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff03cc00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 0x557c474db8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff037e00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 0x557c4754d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 -- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 msgr2=0x557c474db8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 0x557c474db8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 -- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 msgr2=0x557c4754d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 0x557c4754d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 -- 172.21.15.28:0/228413889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c4739e5f0 con 0x557c474c0e80 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff03cc00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 0x557c474db8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff037e00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 0x557c4754d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff02f400640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff020000c00 tx=0x7ff020004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0a4bb1dfb969a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff025600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.696+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02003ce60 con 0x557c474c0e80 2024-09-13T17:11:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff02004b040 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff030015600 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff020051440 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 10 client.5955 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x557c473c3b00 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030015600 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff020058060 con 0x557c474c0e80 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 10 client.5955 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 10 client.5955 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 20 client.5955 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c4754a800 0x557c4754abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 4 client.5955 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c470f14e0 con 0x557c4754a800 2024-09-13T17:11:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.697+0000 7ff037e00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c4754a800 0x557c4754abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff037e00640 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c4754a800 0x557c4754abe0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3eff3170cee81e server_cookie=d2c9ff7a4c22f19e in_seq=0 out_seq=0 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff025600640 10 client.5955 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff025600640 1 -- 172.21.15.28:0/228413889 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+168 (crc 0 0 0) 0x557c470f14e0 con 0x557c4754a800 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff025600640 10 client.5955 handle_command_reply: tid=0 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 client.5955 shutdown 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 msgr2=0x557c4752fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff020000c00 tx=0x7ff020004730 comp rx=0 tx=0).stop 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 shutdown_connections 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557c474c22c0 0x557c474db8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557c4754a800 0x557c4754abe0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557c474c0e80 0x557c4752fdb0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 --2- 172.21.15.28:0/228413889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4749ae10 0x557c4754d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 >> 172.21.15.28:0/228413889 conn(0x557c474985d0 msgr2=0x557c47501cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.698+0000 7ff025600640 0 client.5955 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 shutdown_connections 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff0419f1740 1 -- 172.21.15.28:0/228413889 wait complete. 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff0419f1740 20 client.5955 trim_cache size 0 max 16384 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 3553196049, 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630186, 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_0" 2024-09-13T17:11:31.704 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:31.705 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:11:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 msgr2=0x7ff01406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 0x7ff01406d0f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff020034000 tx=0x7ff02002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 msgr2=0x7ff0381f67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7ff02800c430 tx=0x7ff02800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 shutdown_connections 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff01406ac30 0x7ff01406d0f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff038159770 0x7ff0381f1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff038070430 0x7ff0381e1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 --2- 172.21.15.28:0/3241180674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff03806f3f0 0x7ff0381f67e0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.699+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 >> 172.21.15.28:0/3241180674 conn(0x7ff0380b0270 msgr2=0x7ff0380ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.700+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 shutdown_connections 2024-09-13T17:11:31.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.700+0000 7ff03ec00640 1 -- 172.21.15.28:0/3241180674 wait complete. 2024-09-13T17:11:31.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:31.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.871+0000 7fe26f200640 1 Processor -- start 2024-09-13T17:11:31.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.871+0000 7fe26f200640 1 -- start start 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe270159820 0x7fe270164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2700c3850 con 0x7fe270159440 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe27015a930 con 0x7fe2700b0b50 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe270159d20 con 0x7fe270159820 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe270159440 0x7fe2700b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42318/0 (socket says 172.21.15.28:42318) 2024-09-13T17:11:31.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.872+0000 7fe26de00640 1 -- 172.21.15.28:0/514789211 learned_addr learned my addr 172.21.15.28:0/514789211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730027549 0 0) 0x7fe2700c3850 con 0x7fe270159440 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26e800640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe270159820 0x7fe270164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe254005730 con 0x7fe270159440 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2892131285 0 0) 0x7fe254005730 con 0x7fe270159440 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe270159820 msgr2=0x7fe270164260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe270159820 0x7fe270164260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 >> v1:172.21.15.189:6789/0 conn(0x7fe2700b0b50 legacy=0x7fe2700b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe27015a000 con 0x7fe270159440 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fe260001670 con 0x7fe270159440 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 >> v1:172.21.15.28:6789/0 conn(0x7fe270159440 legacy=0x7fe2700b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe254005bb0 0x7fe254005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2540069d0 0x7fe254006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe25400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe270159d20 con 0x7fe254007390 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25400e240 con 0x7fe254005bb0 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25400e400 con 0x7fe2540069d0 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.873+0000 7fe26e800640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe25400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26de00640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe254005bb0 0x7fe254005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe267e00640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2540069d0 0x7fe254006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26e800640 1 -- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2540069d0 msgr2=0x7fe254006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26e800640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2540069d0 0x7fe254006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26e800640 1 -- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe254005bb0 msgr2=0x7fe254005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:31.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26e800640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe254005bb0 0x7fe254005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.874+0000 7fe26e800640 1 -- 172.21.15.28:0/514789211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe254005730 con 0x7fe254007390 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.875+0000 7fe26e800640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe25400db90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe270159c00 tx=0x7fe254007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50717ef754c8b328 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.875+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26802d070 con 0x7fe254007390 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.875+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe268031430 con 0x7fe254007390 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.875+0000 7fe26d400640 1 -- 172.21.15.28:0/514789211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2680317c0 con 0x7fe254007390 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 -- 172.21.15.28:0/514789211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 msgr2=0x7fe25400db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 --2- 172.21.15.28:0/514789211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe25400db90 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fe270159c00 tx=0x7fe254007a00 comp rx=0 tx=0).stop 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26e800640 1 -- 172.21.15.28:0/514789211 reap_dead start 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 -- 172.21.15.28:0/514789211 shutdown_connections 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 -- 172.21.15.28:0/514789211 >> 172.21.15.28:0/514789211 conn(0x7fe270072e40 msgr2=0x7fe270073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 -- 172.21.15.28:0/514789211 shutdown_connections 2024-09-13T17:11:31.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.876+0000 7fe26f200640 1 -- 172.21.15.28:0/514789211 wait complete. 2024-09-13T17:11:31.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 Processor -- start 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 -- start start 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270157fe0 0x7fe27014a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe27014db80 0x7fe27014e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe27014b030 con 0x7fe254007390 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2701554c0 con 0x7fe270157fe0 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.878+0000 7fe26f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2701557a0 con 0x7fe27014db80 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42784/0 (socket says 172.21.15.28:42784) 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 -- 172.21.15.28:0/1803315140 learned_addr learned my addr 172.21.15.28:0/1803315140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26e800640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe27014db80 0x7fe27014e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe267e00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270157fe0 0x7fe27014a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 -- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe27014db80 msgr2=0x7fe27014e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe27014db80 0x7fe27014e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 -- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270157fe0 msgr2=0x7fe27014a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270157fe0 0x7fe27014a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.879+0000 7fe26de00640 1 -- 172.21.15.28:0/1803315140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe270155a80 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe26de00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe2600072c0 tx=0x7fe260007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe260004ea0 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26000f960 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26000fd10 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe270155d60 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.880+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe27014ac00 con 0x7fe254007390 2024-09-13T17:11:31.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.881+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe270155fa0 con 0x7fe254007390 2024-09-13T17:11:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.882+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe26001d460 con 0x7fe254007390 2024-09-13T17:11:31.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.883+0000 7fe266000640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 0x7fe24c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:31.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.883+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2600893f0 con 0x7fe254007390 2024-09-13T17:11:31.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.883+0000 7fe267e00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 0x7fe24c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:31.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.884+0000 7fe267e00640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 0x7fe24c06d0f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe27015b970 tx=0x7fe258009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:31.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:31.887+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe270155fa0 con 0x7fe254007390 2024-09-13T17:11:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.101+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe270155fa0 con 0x7fe254007390 2024-09-13T17:11:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.102+0000 7fe266000640 1 -- 172.21.15.28:0/1803315140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fe270155fa0 con 0x7fe254007390 2024-09-13T17:11:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:32.103 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:29.038624+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 msgr2=0x7fe24c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 0x7fe24c06d0f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe27015b970 tx=0x7fe258009040 comp rx=0 tx=0).stop 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 msgr2=0x7fe270148cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fe2600072c0 tx=0x7fe260007780 comp rx=0 tx=0).stop 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 shutdown_connections 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe24c06ac30 0x7fe24c06d0f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe27014db80 0x7fe27014e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe270157fe0 0x7fe27014a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 --2- 172.21.15.28:0/1803315140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe254007390 0x7fe270148cd0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 >> 172.21.15.28:0/1803315140 conn(0x7fe270072e40 msgr2=0x7fe27006f760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 shutdown_connections 2024-09-13T17:11:32.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.104+0000 7fe26f200640 1 -- 172.21.15.28:0/1803315140 wait complete. 2024-09-13T17:11:32.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 3553196049 2024-09-13T17:11:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.279+0000 7f9df4600640 1 Processor -- start 2024-09-13T17:11:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.279+0000 7f9df4600640 1 -- start start 2024-09-13T17:11:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.280+0000 7f9df4600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dec159770 0x7f9dec06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.280+0000 7f9df4600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec06f920 con 0x7f9dec15cd80 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df4600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dec15b290 con 0x7f9dec06f3f0 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df4600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9dec15b570 con 0x7f9dec159770 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9dec159770 0x7f9dec06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df2600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9dec15cd80 0x7f9dec15a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42328/0 (socket says 172.21.15.28:42328) 2024-09-13T17:11:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df2600640 1 -- 172.21.15.28:0/3158532486 learned_addr learned my addr 172.21.15.28:0/3158532486 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df1c00640 1 -- 172.21.15.28:0/3158532486 >> v1:172.21.15.189:6789/0 conn(0x7f9dec06f3f0 legacy=0x7f9dec159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df1c00640 1 -- 172.21.15.28:0/3158532486 >> v1:172.21.15.28:6789/0 conn(0x7f9dec15cd80 legacy=0x7f9dec15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9df1c00640 1 -- 172.21.15.28:0/3158532486 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dec15b850 con 0x7f9dec159770 2024-09-13T17:11:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.281+0000 7f9debe00640 1 -- 172.21.15.28:0/3158532486 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001747014 0 0) 0x7f9dec06f920 con 0x7f9dec15cd80 2024-09-13T17:11:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.282+0000 7f9df1c00640 1 --2- 172.21.15.28:0/3158532486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec159770 0x7f9dec06fd20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c52d51a52fe46370 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9debe00640 1 -- 172.21.15.28:0/3158532486 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de002ae50 con 0x7f9dec159770 2024-09-13T17:11:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9debe00640 1 -- 172.21.15.28:0/3158532486 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0032440 con 0x7f9dec159770 2024-09-13T17:11:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9debe00640 1 -- 172.21.15.28:0/3158532486 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00327d0 con 0x7f9dec159770 2024-09-13T17:11:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 -- 172.21.15.28:0/3158532486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec159770 msgr2=0x7f9dec06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 --2- 172.21.15.28:0/3158532486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec159770 0x7f9dec06fd20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002a140 comp rx=0 tx=0).stop 2024-09-13T17:11:32.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 -- 172.21.15.28:0/3158532486 shutdown_connections 2024-09-13T17:11:32.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 --2- 172.21.15.28:0/3158532486 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec159770 0x7f9dec06fd20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 -- 172.21.15.28:0/3158532486 >> 172.21.15.28:0/3158532486 conn(0x7f9dec0b0270 msgr2=0x7f9dec0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 -- 172.21.15.28:0/3158532486 shutdown_connections 2024-09-13T17:11:32.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.283+0000 7f9df4600640 1 -- 172.21.15.28:0/3158532486 wait complete. 2024-09-13T17:11:32.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.285+0000 7f9df4600640 1 Processor -- start 2024-09-13T17:11:32.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.285+0000 7f9df4600640 1 -- start start 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9dec06f3f0 0x7f9dec149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 0x7f9dec155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec158fb0 con 0x7f9dec15cd80 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec150c90 con 0x7f9dec159770 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df4600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dec14a3b0 con 0x7f9dec06f3f0 2024-09-13T17:11:32.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42808/0 (socket says 172.21.15.28:42808) 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 -- 172.21.15.28:0/4257110264 learned_addr learned my addr 172.21.15.28:0/4257110264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9dec06f3f0 0x7f9dec149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 -- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9dec06f3f0 msgr2=0x7f9dec149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df1200640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 0x7f9dec155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9dec06f3f0 0x7f9dec149e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.286+0000 7f9df2600640 1 -- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 msgr2=0x7f9dec155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df2600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 0x7f9dec155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df2600640 1 -- 172.21.15.28:0/4257110264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de0009400 con 0x7f9dec15cd80 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df1200640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 0x7f9dec155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df2600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c780 tx=0x7f9de400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9de9600640 1 -- 172.21.15.28:0/4257110264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400ce90 con 0x7f9dec15cd80 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9de9600640 1 -- 172.21.15.28:0/4257110264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de400a5e0 con 0x7f9dec15cd80 2024-09-13T17:11:32.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9de9600640 1 -- 172.21.15.28:0/4257110264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de400a970 con 0x7f9dec15cd80 2024-09-13T17:11:32.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dec158cd0 con 0x7f9dec15cd80 2024-09-13T17:11:32.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.287+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec1577a0 con 0x7f9dec15cd80 2024-09-13T17:11:32.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.289+0000 7f9de9600640 1 -- 172.21.15.28:0/4257110264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9de400aad0 con 0x7f9dec15cd80 2024-09-13T17:11:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.290+0000 7f9de9600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 0x7f9dcc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.290+0000 7f9de9600640 1 -- 172.21.15.28:0/4257110264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9de4088850 con 0x7f9dec15cd80 2024-09-13T17:11:32.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.291+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 0x7f9dcc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.291+0000 7f9df7403740 1 Processor -- start 2024-09-13T17:11:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.291+0000 7f9df7403740 1 -- start start 2024-09-13T17:11:32.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.291+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 0x7f9dcc06d0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9de0034000 tx=0x7f9de002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 0x560e2c1016c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c14eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c069ba0 con 0x560e2c14f000 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c067330 con 0x560e2c1012e0 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e2c062c20 con 0x560e2c128b40 2024-09-13T17:11:32.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42816/0 (socket says 172.21.15.28:42816) 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.292+0000 7f9df2600640 1 -- 172.21.15.28:0/1986245500 learned_addr learned my addr 172.21.15.28:0/1986245500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df1c00640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 0x560e2c1016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df1200640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c14eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 -- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 msgr2=0x560e2c14eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c14eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 -- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 msgr2=0x560e2c1016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 0x560e2c1016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 -- 172.21.15.28:0/1986245500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e2c007f70 con 0x560e2c14f000 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df1200640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c14eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df1c00640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 0x560e2c1016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9df2600640 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9de4086ce0 tx=0x7f9de4001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbbc149702ea0668 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9dde000640 1 -- 172.21.15.28:0/1986245500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4055a80 con 0x560e2c14f000 2024-09-13T17:11:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9dde000640 1 -- 172.21.15.28:0/1986245500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de4088040 con 0x560e2c14f000 2024-09-13T17:11:32.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.293+0000 7f9dde000640 1 -- 172.21.15.28:0/1986245500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4053df0 con 0x560e2c14f000 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 -- 172.21.15.28:0/1986245500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 msgr2=0x560e2c14f4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9de4086ce0 tx=0x7f9de4001250 comp rx=0 tx=0).stop 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 -- 172.21.15.28:0/1986245500 shutdown_connections 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c14f4b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c14eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 --2- 172.21.15.28:0/1986245500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1012e0 0x560e2c1016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 -- 172.21.15.28:0/1986245500 >> 172.21.15.28:0/1986245500 conn(0x560e2c0fdb30 msgr2=0x560e2c145fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 -- 172.21.15.28:0/1986245500 shutdown_connections 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.294+0000 7f9df7403740 1 -- 172.21.15.28:0/1986245500 wait complete. 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.295+0000 7f9df7403740 1 -- start start 2024-09-13T17:11:32.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.295+0000 7f9df7403740 1 Processor -- start 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c14f000 0x560e2c11fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1449c0 0x560e2c120090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c020fb0 con 0x560e2c128b40 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c020c50 con 0x560e2c1449c0 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e2bdc9f50 con 0x560e2c14f000 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df7403740 10 client.0 _ll_register_callbacks cb 0x560e2c05c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42820/0 (socket says 172.21.15.28:42820) 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 -- 172.21.15.28:0/1399848043 learned_addr learned my addr 172.21.15.28:0/1399848043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 -- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c14f000 msgr2=0x560e2c11fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c14f000 0x560e2c11fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 -- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1449c0 msgr2=0x560e2c120090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1449c0 0x560e2c120090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.296+0000 7f9df1c00640 1 -- 172.21.15.28:0/1399848043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dec1577a0 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9df1c00640 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9de002a730 tx=0x7f9de002c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c345fe7f463264 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9dd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de00495f0 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9de0049750 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560e2c08ca60 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9de003ce90 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9df7403740 10 client.5976 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x560e2c01e090 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.297+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560e2c08ca60 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9de0048070 con 0x560e2c128b40 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 10 client.5976 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 10 client.5976 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 20 client.5976 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c147760 0x560e2c147b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 4 client.5976 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560e2c0556f0 con 0x560e2c147760 2024-09-13T17:11:32.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df1200640 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c147760 0x560e2c147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9dd2a00640 10 client.5976 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.298+0000 7f9df1200640 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c147760 0x560e2c147b40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6bd35cb85786fe server_cookie=b84db61f0aab4c33 in_seq=0 out_seq=0 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9dd2a00640 1 -- 172.21.15.28:0/1399848043 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x560e2c0556f0 con 0x560e2c147760 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9dd2a00640 10 client.5976 handle_command_reply: tid=0 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 client.5976 shutdown 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 msgr2=0x560e2c144480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9de002a730 tx=0x7f9de002c070 comp rx=0 tx=0).stop 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 shutdown_connections 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c1449c0 0x560e2c120090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c14f000 0x560e2c11fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c147760 0x560e2c147b40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 --2- 172.21.15.28:0/1399848043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c128b40 0x560e2c144480 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 >> 172.21.15.28:0/1399848043 conn(0x560e2c1012e0 msgr2=0x560e2c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.299+0000 7f9dd2a00640 0 client.5976 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.300+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 shutdown_connections 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.300+0000 7f9df7403740 1 -- 172.21.15.28:0/1399848043 wait complete. 2024-09-13T17:11:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.300+0000 7f9df7403740 20 client.5976 trim_cache size 0 max 16384 2024-09-13T17:11:32.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.312+0000 7f9df7403740 1 Processor -- start 2024-09-13T17:11:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.312+0000 7f9df7403740 1 -- start start 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.312+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 0x560e2c149180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.312+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c1449c0 0x560e2c129160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df7403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df7403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2bfe5ea0 con 0x560e2c14f000 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c098990 con 0x560e2c128b40 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e2c1023e0 con 0x560e2c1449c0 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42828/0 (socket says 172.21.15.28:42828) 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 -- 172.21.15.28:0/522790258 learned_addr learned my addr 172.21.15.28:0/522790258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df1c00640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 0x560e2c149180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df1200640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c1449c0 0x560e2c129160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 -- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c1449c0 msgr2=0x560e2c129160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c1449c0 0x560e2c129160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 -- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 msgr2=0x560e2c149180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 0x560e2c149180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 -- 172.21.15.28:0/522790258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560e2c0031f0 con 0x560e2c14f000 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df1c00640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 0x560e2c149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.313+0000 7f9df2600640 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9de4053be0 tx=0x7f9de405a070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d39494b63118403 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9ddea00640 1 -- 172.21.15.28:0/522790258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4053a20 con 0x560e2c14f000 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9ddea00640 1 -- 172.21.15.28:0/522790258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de4055a80 con 0x560e2c14f000 2024-09-13T17:11:32.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9ddea00640 1 -- 172.21.15.28:0/522790258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de4053df0 con 0x560e2c14f000 2024-09-13T17:11:32.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 -- 172.21.15.28:0/522790258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 msgr2=0x560e2c11f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9de4053be0 tx=0x7f9de405a070 comp rx=0 tx=0).stop 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 -- 172.21.15.28:0/522790258 shutdown_connections 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c14f000 0x560e2c11f280 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c1449c0 0x560e2c129160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 --2- 172.21.15.28:0/522790258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c128b40 0x560e2c149180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 -- 172.21.15.28:0/522790258 >> 172.21.15.28:0/522790258 conn(0x560e2c1012e0 msgr2=0x560e2c1016c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 -- 172.21.15.28:0/522790258 shutdown_connections 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.314+0000 7f9df7403740 1 -- 172.21.15.28:0/522790258 wait complete. 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.315+0000 7f9df7403740 1 -- start start 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.315+0000 7f9df7403740 1 Processor -- start 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c191f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 0x560e2c127f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c144da0 con 0x560e2c1449c0 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560e2c1b0230 con 0x560e2c14f000 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560e2c147cd0 con 0x560e2c128b40 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df7403740 10 client.0 _ll_register_callbacks cb 0x7f9de0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42838/0 (socket says 172.21.15.28:42838) 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.316+0000 7f9df1200640 1 -- 172.21.15.28:0/4085273016 learned_addr learned my addr 172.21.15.28:0/4085273016 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c191f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df2600640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 0x560e2c127f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 -- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 msgr2=0x560e2c191f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c191f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 -- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 msgr2=0x560e2c127f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 0x560e2c127f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 -- 172.21.15.28:0/4085273016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560e2c007f70 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df2600640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 0x560e2c127f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df1200640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x560e2c128f40 tx=0x7f9dd80052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5ab8f07edddbf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9dd2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dd801d070 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dd8018770 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.317+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9de4088040 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9dd8018930 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 10 client.5982 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x560e2c028690 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de4088040 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9dd8012040 con 0x560e2c1449c0 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 10 client.5982 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:32.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 10 client.5982 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 20 client.5982 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c1454d0 0x560e2c1458b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 4 client.5982 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 3553196049}] 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage rm", "damage_id": 3553196049}) v1 -- 0x560e2bd563f0 con 0x560e2c1454d0 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.318+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c1454d0 0x560e2c1458b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df1c00640 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c1454d0 0x560e2c1458b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b1049fae89f2ad server_cookie=8e86a83d1b06365f in_seq=0 out_seq=0 2024-09-13T17:11:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9dd2a00640 10 client.5982 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9dd2a00640 1 -- 172.21.15.28:0/4085273016 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x560e2bd563f0 con 0x560e2c1454d0 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9dd2a00640 10 client.5982 handle_command_reply: tid=0 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 client.5982 shutdown 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 msgr2=0x560e2c1a97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x560e2c128f40 tx=0x7f9dd80052f0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 shutdown_connections 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560e2c14f000 0x560e2c127f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x560e2c1454d0 0x560e2c1458b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2c1449c0 0x560e2c1a97e0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 --2- 172.21.15.28:0/4085273016 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560e2c128b40 0x560e2c191f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 >> 172.21.15.28:0/4085273016 conn(0x560e2c1012e0 msgr2=0x560e2c149180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.319+0000 7f9dd2a00640 0 client.5982 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 shutdown_connections 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df7403740 1 -- 172.21.15.28:0/4085273016 wait complete. 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df7403740 20 client.5982 trim_cache size 0 max 16384 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 msgr2=0x7f9dcc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 0x7f9dcc06d0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9de0034000 tx=0x7f9de002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 msgr2=0x7f9dec0ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9de400c780 tx=0x7f9de400cc40 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 shutdown_connections 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9dcc06ac00 0x7f9dcc06d0c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9dec15cd80 0x7f9dec0ccbe0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9dec159770 0x7f9dec155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 --2- 172.21.15.28:0/4257110264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9dec06f3f0 0x7f9dec149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 >> 172.21.15.28:0/4257110264 conn(0x7f9dec0b0270 msgr2=0x7f9dec06ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 shutdown_connections 2024-09-13T17:11:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.320+0000 7f9df4600640 1 -- 172.21.15.28:0/4257110264 wait complete. 2024-09-13T17:11:32.336 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-09-13T17:11:32.336 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.489+0000 7fe980a00640 1 Processor -- start 2024-09-13T17:11:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.489+0000 7fe980a00640 1 -- start start 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 0x7fe97c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c0b3160 0x7fe97c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c06f980 0x7fe97c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c159230 con 0x7fe97c06f980 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe97c06fe10 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.490+0000 7fe980a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0b3970 con 0x7fe97c0b3160 2024-09-13T17:11:32.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c06f980 0x7fe97c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe97c06f980 0x7fe97c0b1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42852/0 (socket says 172.21.15.28:42852) 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 -- 172.21.15.28:0/1141533740 learned_addr learned my addr 172.21.15.28:0/1141533740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe971400640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c0b3160 0x7fe97c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe979e00640 1 --2- 172.21.15.28:0/1141533740 >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 0x7fe97c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 -- 172.21.15.28:0/1141533740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c0b3160 msgr2=0x7fe97c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c0b3160 0x7fe97c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 -- 172.21.15.28:0/1141533740 >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 msgr2=0x7fe97c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 --2- 172.21.15.28:0/1141533740 >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 0x7fe97c15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe97a800640 1 -- 172.21.15.28:0/1141533740 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c159430 con 0x7fe97c06f980 2024-09-13T17:11:32.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.491+0000 7fe979e00640 1 --2- 172.21.15.28:0/1141533740 >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 0x7fe97c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.492+0000 7fe97a800640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c06f980 0x7fe97c0b1c20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe97c06ae20 tx=0x7fe97402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c839fa458be146c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.492+0000 7fe979400640 1 -- 172.21.15.28:0/1141533740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974031600 con 0x7fe97c06f980 2024-09-13T17:11:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe979400640 1 -- 172.21.15.28:0/1141533740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97402d050 con 0x7fe97c06f980 2024-09-13T17:11:32.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe979400640 1 -- 172.21.15.28:0/1141533740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe974031980 con 0x7fe97c06f980 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 -- 172.21.15.28:0/1141533740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c06f980 msgr2=0x7fe97c0b1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c06f980 0x7fe97c0b1c20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fe97c06ae20 tx=0x7fe97402a6a0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 -- 172.21.15.28:0/1141533740 shutdown_connections 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c06f980 0x7fe97c0b1c20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 --2- 172.21.15.28:0/1141533740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c0b3160 0x7fe97c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 --2- 172.21.15.28:0/1141533740 >> v2:172.21.15.189:3300/0 conn(0x7fe97c0cdfa0 0x7fe97c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 -- 172.21.15.28:0/1141533740 >> 172.21.15.28:0/1141533740 conn(0x7fe97c069e80 msgr2=0x7fe97c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 -- 172.21.15.28:0/1141533740 shutdown_connections 2024-09-13T17:11:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.493+0000 7fe980a00640 1 -- 172.21.15.28:0/1141533740 wait complete. 2024-09-13T17:11:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.495+0000 7fe980a00640 1 Processor -- start 2024-09-13T17:11:32.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.495+0000 7fe980a00640 1 -- start start 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c06f980 0x7fe97c0b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 0x7fe97c0a9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c147b90 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0a9b20 con 0x7fe97c0b3160 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe980a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe97c0a9e00 con 0x7fe97c06f980 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe97a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe97a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42864/0 (socket says 172.21.15.28:42864) 2024-09-13T17:11:32.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe97a800640 1 -- 172.21.15.28:0/305770165 learned_addr learned my addr 172.21.15.28:0/305770165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe979e00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c06f980 0x7fe97c0b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.496+0000 7fe971400640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 0x7fe97c0a9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 -- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c06f980 msgr2=0x7fe97c0b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c06f980 0x7fe97c0b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 -- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 msgr2=0x7fe97c0a9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 0x7fe97c0a9490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 -- 172.21.15.28:0/305770165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe974009b30 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe971400640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 0x7fe97c0a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe97a800640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe974034040 tx=0x7fe974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97403b820 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe97403b980 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97c0aa0e0 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97403bd30 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.497+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe97c0aa6a0 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.499+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe97c0b3580 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.499+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe974038050 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.500+0000 7fe972a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 0x7fe95806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.500+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9740a9e50 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.501+0000 7fe979e00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 0x7fe95806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:32.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.501+0000 7fe979e00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 0x7fe95806d0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe96c006fd0 tx=0x7fe96c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:32.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.505+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe974075040 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.725+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe97c0cf280 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.725+0000 7fe972a00640 1 -- 172.21.15.28:0/305770165 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fe97404b090 con 0x7fe97c0cdfa0 2024-09-13T17:11:32.727 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:32.727 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:29.038624+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:32.728 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.727+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 msgr2=0x7fe95806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 0x7fe95806d0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe96c006fd0 tx=0x7fe96c008040 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 msgr2=0x7fe97c147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fe974034040 tx=0x7fe974004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 shutdown_connections 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe95806ac00 0x7fe95806d0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe97c0cdfa0 0x7fe97c147180 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe97c0b3160 0x7fe97c0a9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 --2- 172.21.15.28:0/305770165 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe97c06f980 0x7fe97c0b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:32.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 >> 172.21.15.28:0/305770165 conn(0x7fe97c069e80 msgr2=0x7fe97c1525e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 shutdown_connections 2024-09-13T17:11:32.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:32.728+0000 7fe980a00640 1 -- 172.21.15.28:0/305770165 wait complete. 2024-09-13T17:11:32.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path / 2024-09-13T17:11:32.907 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:32.908 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:32.908 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:32.920 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:32.920 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:32.920 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_test_path 2024-09-13T17:11:32.920 INFO:tasks.cephfs.test_scrub_checks:Cloning repo into place 2024-09-13T17:11:32.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:32.920 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p /home/ubuntu/cephtest/mnt.0/scrub_test_path) 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:11:32.960 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite'"'"') 2024-09-13T17:11:32.961 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-09-13T17:11:32.961 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-09-13T17:11:32.961 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:32.962 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-13T17:11:32.962 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-09-13T17:11:32.962 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-13T17:11:32.962 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-09-13T17:11:32.962 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mode": 16877, 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ino": 1099511627777, 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout: "st_dev": 42, 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout: "st_nlink": 16, 2024-09-13T17:11:33.103 INFO:teuthology.orchestra.run.smithi028.stdout: "st_uid": 1000, 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout: "st_gid": 1333, 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout: "st_size": 18, 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout: "st_atime": 1726247476.8304355, 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mtime": 1726247484.6002176, 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ctime": 1726247484.6002176 2024-09-13T17:11:33.104 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:33.287 INFO:tasks.cephfs.test_scrub_checks:Initiating mds_scrub_checks on mds.0 test_path /scrub_test_path, run_seq 1 2024-09-13T17:11:33.287 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/i/dont/exist' 2024-09-13T17:11:33.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:33.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.482+0000 7f8dabe00640 1 Processor -- start 2024-09-13T17:11:33.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.483+0000 7f8dabe00640 1 -- start start 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8dabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac06b360 0x7f8dac0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8dabe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac0b0d80 con 0x7f8dac154a90 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8dabe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dac06b900 con 0x7f8dac0b0980 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8dabe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dac06bbe0 con 0x7f8dac06b360 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac06b360 0x7f8dac0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8dac06b360 0x7f8dac0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42880/0 (socket says 172.21.15.28:42880) 2024-09-13T17:11:33.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 -- 172.21.15.28:0/3497833232 learned_addr learned my addr 172.21.15.28:0/3497833232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 -- 172.21.15.28:0/3497833232 >> v1:172.21.15.189:6789/0 conn(0x7f8dac0b0980 legacy=0x7f8dac0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 -- 172.21.15.28:0/3497833232 >> v1:172.21.15.28:6789/0 conn(0x7f8dac154a90 legacy=0x7f8dac15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.484+0000 7f8da3e00640 1 -- 172.21.15.28:0/3497833232 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dac06bec0 con 0x7f8dac06b360 2024-09-13T17:11:33.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8da3e00640 1 --2- 172.21.15.28:0/3497833232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0b0440 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009870 tx=0x7f8d9402a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b615737c9c4edf73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8daa000640 1 -- 172.21.15.28:0/3497833232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94032440 con 0x7f8dac06b360 2024-09-13T17:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8daa000640 1 -- 172.21.15.28:0/3497833232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d940325a0 con 0x7f8dac06b360 2024-09-13T17:11:33.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8daa000640 1 -- 172.21.15.28:0/3497833232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94032930 con 0x7f8dac06b360 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8dabe00640 1 -- 172.21.15.28:0/3497833232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 msgr2=0x7f8dac0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8dabe00640 1 --2- 172.21.15.28:0/3497833232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0b0440 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009870 tx=0x7f8d9402a2a0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8dabe00640 1 -- 172.21.15.28:0/3497833232 shutdown_connections 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8dabe00640 1 --2- 172.21.15.28:0/3497833232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0b0440 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.486+0000 7f8dabe00640 1 -- 172.21.15.28:0/3497833232 >> 172.21.15.28:0/3497833232 conn(0x7f8dac072e40 msgr2=0x7f8dac073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.487+0000 7f8dabe00640 1 -- 172.21.15.28:0/3497833232 shutdown_connections 2024-09-13T17:11:33.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.487+0000 7f8dabe00640 1 -- 172.21.15.28:0/3497833232 wait complete. 2024-09-13T17:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.488+0000 7f8dabe00640 1 Processor -- start 2024-09-13T17:11:33.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.488+0000 7f8dabe00640 1 -- start start 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dac0b0980 0x7f8dac1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8dac154a90 0x7f8dac1f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac0af2d0 con 0x7f8dac06b360 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac1f13c0 con 0x7f8dac0b0980 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8dabe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dac1f3e50 con 0x7f8dac154a90 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8daaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8daaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42886/0 (socket says 172.21.15.28:42886) 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.489+0000 7f8daaa00640 1 -- 172.21.15.28:0/2883678570 learned_addr learned my addr 172.21.15.28:0/2883678570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8da3e00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dac0b0980 0x7f8dac1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:33.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8dab400640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8dac154a90 0x7f8dac1f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 -- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8dac154a90 msgr2=0x7f8dac1f0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8dac154a90 0x7f8dac1f0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 -- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dac0b0980 msgr2=0x7f8dac1f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dac0b0980 0x7f8dac1f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 -- 172.21.15.28:0/2883678570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d940094d0 con 0x7f8dac06b360 2024-09-13T17:11:33.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8daaa00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c400 tx=0x7f8d9c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0052d0 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9c005430 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dac1f4130 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.490+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c003830 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.491+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dac070b00 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.492+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dac070cf0 con 0x7f8dac06b360 2024-09-13T17:11:33.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.492+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d9c003990 con 0x7f8dac06b360 2024-09-13T17:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.493+0000 7f8da2a00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 0x7f8d8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.493+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d9c088830 con 0x7f8dac06b360 2024-09-13T17:11:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.494+0000 7f8da3e00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 0x7f8d8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.495+0000 7f8da3e00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 0x7f8d8806d0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034000 tx=0x7f8d9402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.498+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8d9c053ab0 con 0x7f8dac06b360 2024-09-13T17:11:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.715+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8dac0b0440 con 0x7f8dac06b360 2024-09-13T17:11:33.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.715+0000 7f8da2a00640 1 -- 172.21.15.28:0/2883678570 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f8d9c0570d0 con 0x7f8dac06b360 2024-09-13T17:11:33.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:33.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:29.038624+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:33.718 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-09-13T17:11:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 msgr2=0x7f8d8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 0x7f8d8806d0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8d94034000 tx=0x7f8d9402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 msgr2=0x7f8dac0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c00c400 tx=0x7f8d9c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 shutdown_connections 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d8806ac00 0x7f8d8806d0c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8dac154a90 0x7f8dac1f0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8dac0b0980 0x7f8dac1f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 --2- 172.21.15.28:0/2883678570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8dac06b360 0x7f8dac0705c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 >> 172.21.15.28:0/2883678570 conn(0x7f8dac072e40 msgr2=0x7f8dac157180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 shutdown_connections 2024-09-13T17:11:33.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:33.718+0000 7f8dabe00640 1 -- 172.21.15.28:0/2883678570 wait complete. 2024-09-13T17:11:33.736 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/i/dont/exist 2024-09-13T17:11:33.897 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:33.897 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": -2 2024-09-13T17:11:33.897 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:33.909 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/i/dont/exist' got response code '0' and stdout '{ "return_code": -2 } ' 2024-09-13T17:11:33.909 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/i/dont/exist' 2024-09-13T17:11:33.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/i/dont/exist 2024-09-13T17:11:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.066+0000 7ff2f9400640 1 Processor -- start 2024-09-13T17:11:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.066+0000 7ff2f9400640 1 -- start start 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.067+0000 7ff2f9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2f415a780 0x7ff2f4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.067+0000 7ff2f9400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2f415ab60 0x7ff2f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f406c660 con 0x7ff2f406bb40 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f9400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f406c8b0 con 0x7ff2f415a780 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f9400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f406cb90 con 0x7ff2f415ab60 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2f415a780 0x7ff2f4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2f415a780 0x7ff2f4070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53638/0 (socket says 172.21.15.28:53638) 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 -- 172.21.15.28:0/3400783479 learned_addr learned my addr 172.21.15.28:0/3400783479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/3400783479 >> v2:172.21.15.189:3300/0 conn(0x7ff2f415ab60 0x7ff2f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 -- 172.21.15.28:0/3400783479 >> v2:172.21.15.189:3300/0 conn(0x7ff2f415ab60 msgr2=0x7ff2f40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 --2- 172.21.15.28:0/3400783479 >> v2:172.21.15.189:3300/0 conn(0x7ff2f415ab60 0x7ff2f40712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 -- 172.21.15.28:0/3400783479 >> v1:172.21.15.28:6789/0 conn(0x7ff2f406bb40 legacy=0x7ff2f406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.068+0000 7ff2f2800640 1 -- 172.21.15.28:0/3400783479 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f40cc980 con 0x7ff2f415a780 2024-09-13T17:11:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.069+0000 7ff2f2800640 1 --2- 172.21.15.28:0/3400783479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415a780 0x7ff2f4070d80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406f380 tx=0x7ff2e402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa117844b99be597 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f1e00640 1 -- 172.21.15.28:0/3400783479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4032440 con 0x7ff2f415a780 2024-09-13T17:11:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f1e00640 1 -- 172.21.15.28:0/3400783479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e40325a0 con 0x7ff2f415a780 2024-09-13T17:11:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f1e00640 1 -- 172.21.15.28:0/3400783479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e4032930 con 0x7ff2f415a780 2024-09-13T17:11:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 -- 172.21.15.28:0/3400783479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415a780 msgr2=0x7ff2f4070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 --2- 172.21.15.28:0/3400783479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415a780 0x7ff2f4070d80 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406f380 tx=0x7ff2e402a270 comp rx=0 tx=0).stop 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 -- 172.21.15.28:0/3400783479 shutdown_connections 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 --2- 172.21.15.28:0/3400783479 >> v2:172.21.15.189:3300/0 conn(0x7ff2f415ab60 0x7ff2f40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 --2- 172.21.15.28:0/3400783479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415a780 0x7ff2f4070d80 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 -- 172.21.15.28:0/3400783479 >> 172.21.15.28:0/3400783479 conn(0x7ff2f40b0270 msgr2=0x7ff2f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 -- 172.21.15.28:0/3400783479 shutdown_connections 2024-09-13T17:11:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.070+0000 7ff2f9400640 1 -- 172.21.15.28:0/3400783479 wait complete. 2024-09-13T17:11:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.072+0000 7ff2f9400640 1 Processor -- start 2024-09-13T17:11:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.072+0000 7ff2f9400640 1 -- start start 2024-09-13T17:11:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f406bb40 0x7ff2f415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f415a780 0x7ff2f4142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f415b5c0 con 0x7ff2f415ab60 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f41f6cd0 con 0x7ff2f415a780 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f41f6840 con 0x7ff2f406bb40 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53644/0 (socket says 172.21.15.28:53644) 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f3200640 1 -- 172.21.15.28:0/1786081659 learned_addr learned my addr 172.21.15.28:0/1786081659 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f406bb40 0x7ff2f415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.073+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f415a780 0x7ff2f4142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 -- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f406bb40 msgr2=0x7ff2f415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f406bb40 0x7ff2f415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 -- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f415a780 msgr2=0x7ff2f4142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f415a780 0x7ff2f4142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 -- 172.21.15.28:0/1786081659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e40094d0 con 0x7ff2f415ab60 2024-09-13T17:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c7a0 tx=0x7ff2ec00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2eaa00640 1 -- 172.21.15.28:0/1786081659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec003b70 con 0x7ff2f415ab60 2024-09-13T17:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2eaa00640 1 -- 172.21.15.28:0/1786081659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec003cf0 con 0x7ff2f415ab60 2024-09-13T17:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2eaa00640 1 -- 172.21.15.28:0/1786081659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec003ea0 con 0x7ff2f415ab60 2024-09-13T17:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.074+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f41f5f10 con 0x7ff2f415ab60 2024-09-13T17:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.075+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f41f64d0 con 0x7ff2f415ab60 2024-09-13T17:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.076+0000 7ff2eaa00640 1 -- 172.21.15.28:0/1786081659 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff2ec00aa90 con 0x7ff2f415ab60 2024-09-13T17:11:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.077+0000 7ff2eaa00640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 0x7ff2d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.077+0000 7ff2eaa00640 1 -- 172.21.15.28:0/1786081659 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2ec089010 con 0x7ff2f415ab60 2024-09-13T17:11:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.078+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 0x7ff2d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.078+0000 7ff2fc1ba740 1 Processor -- start 2024-09-13T17:11:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.078+0000 7ff2fc1ba740 1 -- start start 2024-09-13T17:11:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 0x7ff2d006d0f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4034000 tx=0x7ff2e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff996e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9941f0 0x557fff9e3f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff951a60 con 0x557fff9bdee0 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff8ffdf0 con 0x557fff9941f0 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fff8fd580 con 0x557fff996a30 2024-09-13T17:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53656/0 (socket says 172.21.15.28:53656) 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2f3200640 1 -- 172.21.15.28:0/423016608 learned_addr learned my addr 172.21.15.28:0/423016608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2f2800640 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff996e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.079+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9941f0 0x557fff9e3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 -- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 msgr2=0x557fff996e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff996e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 -- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9941f0 msgr2=0x557fff9e3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9941f0 0x557fff9e3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 -- 172.21.15.28:0/423016608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fff8c00e0 con 0x557fff9bdee0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2f3200640 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec05fd70 tx=0x7ff2ec0578c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=812b5c5e86d272b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2e3400640 1 -- 172.21.15.28:0/423016608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0602a0 con 0x557fff9bdee0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2e3400640 1 -- 172.21.15.28:0/423016608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec056300 con 0x557fff9bdee0 2024-09-13T17:11:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.080+0000 7ff2e3400640 1 -- 172.21.15.28:0/423016608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec054670 con 0x557fff9bdee0 2024-09-13T17:11:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/423016608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 msgr2=0x557fff9e4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec05fd70 tx=0x7ff2ec0578c0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/423016608 shutdown_connections 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e4440 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9941f0 0x557fff9e3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/423016608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff996e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/423016608 >> 172.21.15.28:0/423016608 conn(0x557fff993160 msgr2=0x557fff9db610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/423016608 shutdown_connections 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.081+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/423016608 wait complete. 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.082+0000 7ff2fc1ba740 1 -- start start 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.082+0000 7ff2fc1ba740 1 Processor -- start 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9e3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9e3990 0x557fff9e1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff8f8e70 con 0x557fff9bdee0 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff89a1d0 con 0x557fff9e3990 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fff899e70 con 0x557fff996a30 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2fc1ba740 10 client.0 _ll_register_callbacks cb 0x557fff8bf8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53660/0 (socket says 172.21.15.28:53660) 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3968674639 learned_addr learned my addr 172.21.15.28:0/3968674639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2f2800640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9e3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.083+0000 7ff2f3200640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9e3990 0x557fff9e1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 msgr2=0x557fff9e3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9e3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9e3990 msgr2=0x557fff9e1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9e3990 0x557fff9e1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3968674639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f41f64d0 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0135f0 tx=0x7ff2dc0139f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaba121e911194a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc004300 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2dc004460 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.084+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557fff8ead10 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2dc003610 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 10 client.6012 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x557fff8f74f0 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557fff8ead10 con 0x557fff9bdee0 2024-09-13T17:11:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff2dc01b020 con 0x557fff9bdee0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 10 client.6012 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 10 client.6012 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 20 client.6012 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9f56c0 0x557fff9f7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 4 client.6012 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557fff5ec6d0 con 0x557fff9f56c0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.085+0000 7ff2f2800640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9f56c0 0x557fff9f7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.086+0000 7ff2f2800640 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9f56c0 0x557fff9f7ab0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b34eef14b92ba030 server_cookie=eafcf7d15ff39a3c in_seq=0 out_seq=0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.086+0000 7ff2d7e00640 10 client.6012 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.086+0000 7ff2d7e00640 1 -- 172.21.15.28:0/3968674639 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x557fff5ec6d0 con 0x557fff9f56c0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.086+0000 7ff2d7e00640 10 client.6012 handle_command_reply: tid=0 2024-09-13T17:11:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.086+0000 7ff2fc1ba740 1 client.6012 shutdown 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 msgr2=0x557fff9e0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc0135f0 tx=0x7ff2dc0139f0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 shutdown_connections 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9e3990 0x557fff9e1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9f56c0 0x557fff9f7ab0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9bdee0 0x557fff9e0e20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/3968674639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9e3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2d7e00640 0 client.6012 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 >> 172.21.15.28:0/3968674639 conn(0x557fff9941f0 msgr2=0x557fff9d9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 shutdown_connections 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/3968674639 wait complete. 2024-09-13T17:11:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.087+0000 7ff2fc1ba740 20 client.6012 trim_cache size 0 max 16384 2024-09-13T17:11:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.102+0000 7ff2fc1ba740 1 Processor -- start 2024-09-13T17:11:34.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.102+0000 7ff2fc1ba740 1 -- start start 2024-09-13T17:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff996a30 0x557fff9db1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff9bdee0 0x557fff9db700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff660230 con 0x557fff9e3990 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff9bda70 con 0x557fff996a30 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fff9d9d70 con 0x557fff9bdee0 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53668/0 (socket says 172.21.15.28:53668) 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2f3200640 1 -- 172.21.15.28:0/4245991009 learned_addr learned my addr 172.21.15.28:0/4245991009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff9bdee0 0x557fff9db700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.103+0000 7ff2f2800640 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff996a30 0x557fff9db1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 -- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff9bdee0 msgr2=0x557fff9db700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff9bdee0 0x557fff9db700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 -- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff996a30 msgr2=0x557fff9db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff996a30 0x557fff9db1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 -- 172.21.15.28:0/4245991009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557fff899370 con 0x557fff9e3990 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2f3200640 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec057c60 tx=0x7ff2ec005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3acfdcfcde2c17c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2e3e00640 1 -- 172.21.15.28:0/4245991009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0602a0 con 0x557fff9e3990 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2e3e00640 1 -- 172.21.15.28:0/4245991009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec00ad40 con 0x557fff9e3990 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.104+0000 7ff2e3e00640 1 -- 172.21.15.28:0/4245991009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec05fe10 con 0x557fff9e3990 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/4245991009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 msgr2=0x557fff9f1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec057c60 tx=0x7ff2ec005000 comp rx=0 tx=0).stop 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/4245991009 shutdown_connections 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fff9f1de0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff9bdee0 0x557fff9db700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/4245991009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff996a30 0x557fff9db1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/4245991009 >> 172.21.15.28:0/4245991009 conn(0x557fff9941f0 msgr2=0x557fff9d9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/4245991009 shutdown_connections 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/4245991009 wait complete. 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.105+0000 7ff2fc1ba740 1 -- start start 2024-09-13T17:11:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.106+0000 7ff2fc1ba740 1 Processor -- start 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9bdee0 0x557fff9f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff9aeb00 con 0x557fff9e3990 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557fff9e0860 con 0x557fff9bdee0 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557fff9ba280 con 0x557fff996a30 2024-09-13T17:11:34.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2fc1ba740 10 client.0 _ll_register_callbacks cb 0x7ff2ec00cf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53680/0 (socket says 172.21.15.28:53680) 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 -- 172.21.15.28:0/1459646879 learned_addr learned my addr 172.21.15.28:0/1459646879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9bdee0 0x557fff9f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 -- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 msgr2=0x557fff9ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 -- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9bdee0 msgr2=0x557fff9f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9bdee0 0x557fff9f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.107+0000 7ff2f3200640 1 -- 172.21.15.28:0/1459646879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557fff8c00e0 con 0x557fff9e3990 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2f3200640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c770 tx=0x7ff2ec01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7769910889bd08b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec060010 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2ec053d50 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2ec056300 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2ec053f10 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 10 client.6018 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x557fff8be940 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec056300 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff2ec019020 con 0x557fff9e3990 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 10 client.6018 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 10 client.6018 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 20 client.6018 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9da100 0x557fffa49000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 4 client.6018 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}] 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.108+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/i/dont/exist"}) v1 -- 0x557fff5fd9d0 con 0x557fff9da100 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.109+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9da100 0x557fffa49000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.109+0000 7ff2d7e00640 10 client.6018 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.109+0000 7ff2f2800640 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9da100 0x557fffa49000 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=afcffda540edc3df server_cookie=2332761cff07a2c9 in_seq=0 out_seq=0 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.109+0000 7ff2d7e00640 1 -- 172.21.15.28:0/1459646879 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+26 (crc 0 0 0) 0x557fff5fd9d0 con 0x557fff9da100 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.109+0000 7ff2d7e00640 10 client.6018 handle_command_reply: tid=0 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 client.6018 shutdown 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 msgr2=0x557fffa48ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c770 tx=0x7ff2ec01d040 comp rx=0 tx=0).stop 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 shutdown_connections 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x557fff9da100 0x557fffa49000 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557fff9e3990 0x557fffa48ac0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557fff9bdee0 0x557fff9f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 --2- 172.21.15.28:0/1459646879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557fff996a30 0x557fff9ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2d7e00640 0 client.6018 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 >> 172.21.15.28:0/1459646879 conn(0x557fff9941f0 msgr2=0x557fff9ba860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 shutdown_connections 2024-09-13T17:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 1 -- 172.21.15.28:0/1459646879 wait complete. 2024-09-13T17:11:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.110+0000 7ff2fc1ba740 20 client.6018 trim_cache size 0 max 16384 2024-09-13T17:11:34.116 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:34.116 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": -2 2024-09-13T17:11:34.116 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 msgr2=0x7ff2d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 0x7ff2d006d0f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4034000 tx=0x7ff2e402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 msgr2=0x7ff2f4156da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec00c7a0 tx=0x7ff2ec00cc60 comp rx=0 tx=0).stop 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 shutdown_connections 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d006ac30 0x7ff2d006d0f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f415ab60 0x7ff2f4156da0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f415a780 0x7ff2f4142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 --2- 172.21.15.28:0/1786081659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f406bb40 0x7ff2f415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 >> 172.21.15.28:0/1786081659 conn(0x7ff2f40b0270 msgr2=0x7ff2f406f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 shutdown_connections 2024-09-13T17:11:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.111+0000 7ff2f9400640 1 -- 172.21.15.28:0/1786081659 wait complete. 2024-09-13T17:11:34.127 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/i/dont/exist' returned '{'return_code': -2}' 2024-09-13T17:11:34.127 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites' 2024-09-13T17:11:34.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites 2024-09-13T17:11:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.284+0000 7f1658400640 1 Processor -- start 2024-09-13T17:11:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.284+0000 7f1658400640 1 -- start start 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165015a780 0x7f1650070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 0x7f16500712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f165006bb40 0x7f165006bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165006c8b0 con 0x7f165006bb40 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165006cb90 con 0x7f165015ab60 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1658400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16500cc980 con 0x7f165015a780 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f165006bb40 0x7f165006bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f165006bb40 0x7f165006bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53692/0 (socket says 172.21.15.28:53692) 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 -- 172.21.15.28:0/620221299 learned_addr learned my addr 172.21.15.28:0/620221299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1655000640 1 --2- 172.21.15.28:0/620221299 >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 0x7f16500712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1655a00640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165015a780 0x7f1650070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 -- 172.21.15.28:0/620221299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165015a780 msgr2=0x7f1650070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165015a780 0x7f1650070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 -- 172.21.15.28:0/620221299 >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 msgr2=0x7f16500712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 --2- 172.21.15.28:0/620221299 >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 0x7f16500712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1656400640 1 -- 172.21.15.28:0/620221299 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16500ccc60 con 0x7f165006bb40 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.286+0000 7f1655000640 1 --2- 172.21.15.28:0/620221299 >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 0x7f16500712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.287+0000 7f1656400640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165006bb40 0x7f165006bf90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f16500cf5c0 tx=0x7f164802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27697797dcafc404 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f164fe00640 1 -- 172.21.15.28:0/620221299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648031600 con 0x7f165006bb40 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f164fe00640 1 -- 172.21.15.28:0/620221299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164802d050 con 0x7f165006bb40 2024-09-13T17:11:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f164fe00640 1 -- 172.21.15.28:0/620221299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648031980 con 0x7f165006bb40 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 -- 172.21.15.28:0/620221299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165006bb40 msgr2=0x7f165006bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165006bb40 0x7f165006bf90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f16500cf5c0 tx=0x7f164802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 -- 172.21.15.28:0/620221299 shutdown_connections 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165006bb40 0x7f165006bf90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 --2- 172.21.15.28:0/620221299 >> v2:172.21.15.189:3300/0 conn(0x7f165015ab60 0x7f16500712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 --2- 172.21.15.28:0/620221299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165015a780 0x7f1650070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 -- 172.21.15.28:0/620221299 >> 172.21.15.28:0/620221299 conn(0x7f16500b0270 msgr2=0x7f16500b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 -- 172.21.15.28:0/620221299 shutdown_connections 2024-09-13T17:11:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.288+0000 7f1658400640 1 -- 172.21.15.28:0/620221299 wait complete. 2024-09-13T17:11:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.290+0000 7f1658400640 1 Processor -- start 2024-09-13T17:11:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.290+0000 7f1658400640 1 -- start start 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165006bb40 0x7f16500b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165015a780 0x7f1650148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16501567a0 con 0x7f165015ab60 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16501494d0 con 0x7f165015a780 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1658400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1650155aa0 con 0x7f165006bb40 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1655a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165006bb40 0x7f16500b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53696/0 (socket says 172.21.15.28:53696) 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1656400640 1 -- 172.21.15.28:0/1947904721 learned_addr learned my addr 172.21.15.28:0/1947904721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.291+0000 7f1655000640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165015a780 0x7f1650148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 -- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165006bb40 msgr2=0x7f16500b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165006bb40 0x7f16500b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 -- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165015a780 msgr2=0x7f1650148f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165015a780 0x7f1650148f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 -- 172.21.15.28:0/1947904721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1648009b30 con 0x7f165015ab60 2024-09-13T17:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f1656400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1648009f40 tx=0x7f1648004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f164d600640 1 -- 172.21.15.28:0/1947904721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164803b810 con 0x7f165015ab60 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f164d600640 1 -- 172.21.15.28:0/1947904721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164803b990 con 0x7f165015ab60 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.292+0000 7f164d600640 1 -- 172.21.15.28:0/1947904721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164803bd40 con 0x7f165015ab60 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.293+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16501567a0 con 0x7f165015ab60 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.293+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1650156150 con 0x7f165015ab60 2024-09-13T17:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.295+0000 7f164d600640 1 -- 172.21.15.28:0/1947904721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f164803a3a0 con 0x7f165015ab60 2024-09-13T17:11:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.295+0000 7f164d600640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 0x7f163006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.296+0000 7f164d600640 1 -- 172.21.15.28:0/1947904721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16480a9ca0 con 0x7f165015ab60 2024-09-13T17:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.296+0000 7f1655a00640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 0x7f163006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.296+0000 7f165b272740 1 Processor -- start 2024-09-13T17:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.296+0000 7f165b272740 1 -- start start 2024-09-13T17:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f1655a00640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 0x7f163006d0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1644006fd0 tx=0x7f1644008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dbcbc50 0x55a59dbcc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc19430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59db35ed0 con 0x55a59dc19970 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59db33660 con 0x55a59dbf34b0 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a59db2ef50 con 0x55a59dbcbc50 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53704/0 (socket says 172.21.15.28:53704) 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f1656400640 1 -- 172.21.15.28:0/895803758 learned_addr learned my addr 172.21.15.28:0/895803758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.297+0000 7f1655000640 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc19430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1655a00640 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dbcbc50 0x55a59dbcc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 -- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dbcbc50 msgr2=0x55a59dbcc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dbcbc50 0x55a59dbcc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 -- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 msgr2=0x55a59dc19430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc19430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 -- 172.21.15.28:0/895803758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a59daf6110 con 0x55a59dc19970 2024-09-13T17:11:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1656400640 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1648080960 tx=0x7f1648078a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8916898628e5aae5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1642000640 1 -- 172.21.15.28:0/895803758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648090070 con 0x55a59dc19970 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1642000640 1 -- 172.21.15.28:0/895803758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1648078b50 con 0x55a59dc19970 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.298+0000 7f1642000640 1 -- 172.21.15.28:0/895803758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164807c8d0 con 0x55a59dc19970 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 -- 172.21.15.28:0/895803758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 msgr2=0x55a59dc19e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f1648080960 tx=0x7f1648078a50 comp rx=0 tx=0).stop 2024-09-13T17:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 -- 172.21.15.28:0/895803758 shutdown_connections 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc19e20 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc19430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 --2- 172.21.15.28:0/895803758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dbcbc50 0x55a59dbcc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 -- 172.21.15.28:0/895803758 >> 172.21.15.28:0/895803758 conn(0x55a59dbc84a0 msgr2=0x55a59dc10930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 -- 172.21.15.28:0/895803758 shutdown_connections 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.299+0000 7f165b272740 1 -- 172.21.15.28:0/895803758 wait complete. 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.300+0000 7f165b272740 1 -- start start 2024-09-13T17:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.300+0000 7f165b272740 1 Processor -- start 2024-09-13T17:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dbe1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc15f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59daed260 con 0x55a59dc16490 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59daecf00 con 0x55a59dbf34b0 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a59d896340 con 0x55a59dc19970 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f165b272740 10 client.0 _ll_register_callbacks cb 0x55a59db285c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53706/0 (socket says 172.21.15.28:53706) 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.301+0000 7f1656400640 1 -- 172.21.15.28:0/1831871717 learned_addr learned my addr 172.21.15.28:0/1831871717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1655000640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc15f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 -- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 msgr2=0x55a59dc15f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1655a00640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dbe1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc15f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 -- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 msgr2=0x55a59dbe1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dbe1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 -- 172.21.15.28:0/1831871717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1650156150 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1655000640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc15f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1655a00640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dbe1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1656400640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1648075050 tx=0x7f1648009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abf7cfdc0502672e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1636a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648090040 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16480a6070 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.302+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a59db208d0 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f164807c8d0 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 10 client.6030 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55a59db2d5d0 con 0x55a59dc16490 2024-09-13T17:11:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a59db208d0 con 0x55a59dc16490 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1648080a00 con 0x55a59dc16490 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 10 client.6030 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 10 client.6030 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 20 client.6030 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc23270 0x55a59dc23680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 4 client.6030 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a59d8227e0 con 0x55a59dc23270 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.303+0000 7f1655a00640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc23270 0x55a59dc23680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.304+0000 7f1655a00640 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc23270 0x55a59dc23680 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d3538334d254b0f server_cookie=c72ca1aa69a45794 in_seq=0 out_seq=0 2024-09-13T17:11:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.304+0000 7f1636a00640 10 client.6030 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.304+0000 7f1636a00640 1 -- 172.21.15.28:0/1831871717 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55a59d8227e0 con 0x55a59dc23270 2024-09-13T17:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.304+0000 7f1636a00640 10 client.6030 handle_command_reply: tid=0 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.304+0000 7f165b272740 1 client.6030 shutdown 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 msgr2=0x55a59dc17230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f1648075050 tx=0x7f1648009810 comp rx=0 tx=0).stop 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 shutdown_connections 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc23270 0x55a59dc23680 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc16490 0x55a59dc17230 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc15f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 --2- 172.21.15.28:0/1831871717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dbe1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f1636a00640 0 client.6030 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 >> 172.21.15.28:0/1831871717 conn(0x55a59dbcbc50 msgr2=0x55a59dbf5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 shutdown_connections 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 1 -- 172.21.15.28:0/1831871717 wait complete. 2024-09-13T17:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.305+0000 7f165b272740 20 client.6030 trim_cache size 0 max 16384 2024-09-13T17:11:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.320+0000 7f165b272740 1 Processor -- start 2024-09-13T17:11:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.320+0000 7f165b272740 1 -- start start 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc32790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 0x55a59dc154d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59dc195d0 con 0x55a59dc19970 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59dc0f660 con 0x55a59dbf34b0 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a59dbc7c50 con 0x55a59dc16490 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f1656400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53708/0 (socket says 172.21.15.28:53708) 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.321+0000 7f1656400640 1 -- 172.21.15.28:0/1453712780 learned_addr learned my addr 172.21.15.28:0/1453712780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1655a00640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc32790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 -- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 msgr2=0x55a59dc154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1655000640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 0x55a59dc154d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 0x55a59dc154d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 -- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 msgr2=0x55a59dc32790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc32790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 -- 172.21.15.28:0/1453712780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a59dc1a0c0 con 0x55a59dc19970 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1655a00640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc32790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1655000640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 0x55a59dc154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1656400640 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1648076ea0 tx=0x7f16480a6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5869200b59f2c9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1642a00640 1 -- 172.21.15.28:0/1453712780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1648090040 con 0x55a59dc19970 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1642a00640 1 -- 172.21.15.28:0/1453712780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164807c8d0 con 0x55a59dc19970 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.322+0000 7f1642a00640 1 -- 172.21.15.28:0/1453712780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164803bea0 con 0x55a59dc19970 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- 172.21.15.28:0/1453712780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 msgr2=0x55a59dc23680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f1648076ea0 tx=0x7f16480a6000 comp rx=0 tx=0).stop 2024-09-13T17:11:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- 172.21.15.28:0/1453712780 shutdown_connections 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dc19970 0x55a59dc23680 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc16490 0x55a59dc154d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 --2- 172.21.15.28:0/1453712780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dbf34b0 0x55a59dc32790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- 172.21.15.28:0/1453712780 >> 172.21.15.28:0/1453712780 conn(0x55a59dbcbc50 msgr2=0x55a59dbcc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- 172.21.15.28:0/1453712780 shutdown_connections 2024-09-13T17:11:34.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- 172.21.15.28:0/1453712780 wait complete. 2024-09-13T17:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.323+0000 7f165b272740 1 -- start start 2024-09-13T17:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.324+0000 7f165b272740 1 Processor -- start 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dc16490 0x55a59dbddb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc12400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59dbde610 con 0x55a59dbf34b0 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a59dc141c0 con 0x55a59dc16490 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a59dc13470 con 0x55a59dc19970 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f165b272740 10 client.0 _ll_register_callbacks cb 0x7f16480a4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53718/0 (socket says 172.21.15.28:53718) 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 -- 172.21.15.28:0/3143254967 learned_addr learned my addr 172.21.15.28:0/3143254967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1656400640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc12400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655000640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dc16490 0x55a59dbddb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 -- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 msgr2=0x55a59dc12400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc12400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 -- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dc16490 msgr2=0x55a59dbddb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dc16490 0x55a59dbddb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.325+0000 7f1655a00640 1 -- 172.21.15.28:0/3143254967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a59db2d5d0 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1655a00640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x55a59dbdefb0 tx=0x7f1644004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c91ae118e3048a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1644029070 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1648078b50 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1644018110 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1644018310 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f165b272740 10 client.6036 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55a59db616c0 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1648078b50 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.326+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1644004030 con 0x55a59dbf34b0 2024-09-13T17:11:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 10 client.6036 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 10 client.6036 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 20 client.6036 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc14a00 0x55a59dc23680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 4 client.6036 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}] 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites"}) v1 -- 0x55a59d833ae0 con 0x55a59dc14a00 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f1655000640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc14a00 0x55a59dc23680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f1655000640 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc14a00 0x55a59dc23680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bb3f1e236cc480c server_cookie=4c6abfcab7016504 in_seq=0 out_seq=0 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.327+0000 7f1636a00640 10 client.6036 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f1636a00640 1 -- 172.21.15.28:0/3143254967 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a59d833ae0 con 0x55a59dc14a00 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f1636a00640 10 client.6036 handle_command_reply: tid=0 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 client.6036 shutdown 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 msgr2=0x55a59dc7dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x55a59dbdefb0 tx=0x7f1644004730 comp rx=0 tx=0).stop 2024-09-13T17:11:34.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 shutdown_connections 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a59dc19970 0x55a59dc12400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a59dc16490 0x55a59dbddb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55a59dc14a00 0x55a59dc23680 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 --2- 172.21.15.28:0/3143254967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a59dbf34b0 0x55a59dc7dcb0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 >> 172.21.15.28:0/3143254967 conn(0x55a59dbcbc50 msgr2=0x55a59dc17c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f1636a00640 0 client.6036 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 shutdown_connections 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.328+0000 7f165b272740 1 -- 172.21.15.28:0/3143254967 wait complete. 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f165b272740 20 client.6036 trim_cache size 0 max 16384 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:34.334 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "02bac7df-bf12-46e8-b9bf-f7c6b76fcc05", 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 msgr2=0x7f163006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 0x7f163006d0c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1644006fd0 tx=0x7f1644008040 comp rx=0 tx=0).stop 2024-09-13T17:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 msgr2=0x7f1650156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f1648009f40 tx=0x7f1648004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 shutdown_connections 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f163006ac00 0x7f163006d0c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f165015ab60 0x7f1650156c30 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165015a780 0x7f1650148f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 --2- 172.21.15.28:0/1947904721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165006bb40 0x7f16500b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 >> 172.21.15.28:0/1947904721 conn(0x7f16500b0270 msgr2=0x7f16500af020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 shutdown_connections 2024-09-13T17:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.329+0000 7f1658400640 1 -- 172.21.15.28:0/1947904721 wait complete. 2024-09-13T17:11:34.346 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites' returned '{'return_code': 0, 'scrub_tag': '02bac7df-bf12-46e8-b9bf-f7c6b76fcc05', 'mode': 'asynchronous'}' 2024-09-13T17:11:34.346 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' 2024-09-13T17:11:34.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml 2024-09-13T17:11:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.510+0000 7fd14d400640 1 Processor -- start 2024-09-13T17:11:34.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.511+0000 7fd14d400640 1 -- start start 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd14d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd14806c540 0x7fd1480b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd14d400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1481610a0 con 0x7fd1480b17e0 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd14d400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1480b2620 con 0x7fd14806d070 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd14d400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480b2900 con 0x7fd14806c540 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd146800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd1480b17e0 0x7fd1480b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33842/0 (socket says 172.21.15.28:33842) 2024-09-13T17:11:34.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd146800640 1 -- 172.21.15.28:0/3605812900 learned_addr learned my addr 172.21.15.28:0/3605812900 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.512+0000 7fd147200640 1 --2- 172.21.15.28:0/3605812900 >> v2:172.21.15.189:3300/0 conn(0x7fd14806c540 0x7fd1480b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.513+0000 7fd145e00640 1 -- 172.21.15.28:0/3605812900 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881914588 0 0) 0x7fd1481610a0 con 0x7fd1480b17e0 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.513+0000 7fd145e00640 1 -- 172.21.15.28:0/3605812900 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd12c003520 con 0x7fd1480b17e0 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.513+0000 7fd147200640 1 -- 172.21.15.28:0/3605812900 >> v1:172.21.15.189:6789/0 conn(0x7fd14806d070 legacy=0x7fd1481687d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.513+0000 7fd147200640 1 -- 172.21.15.28:0/3605812900 >> v1:172.21.15.28:6789/0 conn(0x7fd1480b17e0 legacy=0x7fd1480b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.513+0000 7fd147200640 1 -- 172.21.15.28:0/3605812900 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480b2be0 con 0x7fd14806c540 2024-09-13T17:11:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.514+0000 7fd147200640 1 --2- 172.21.15.28:0/3605812900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806c540 0x7fd1480b3480 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd140009d90 tx=0x7fd14002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcb102e65faa3cfe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.514+0000 7fd145e00640 1 -- 172.21.15.28:0/3605812900 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140007740 con 0x7fd14806c540 2024-09-13T17:11:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd145e00640 1 -- 172.21.15.28:0/3605812900 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14002d050 con 0x7fd14806c540 2024-09-13T17:11:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd145e00640 1 -- 172.21.15.28:0/3605812900 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140007ac0 con 0x7fd14806c540 2024-09-13T17:11:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 -- 172.21.15.28:0/3605812900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806c540 msgr2=0x7fd1480b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 --2- 172.21.15.28:0/3605812900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806c540 0x7fd1480b3480 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd140009d90 tx=0x7fd14002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 -- 172.21.15.28:0/3605812900 shutdown_connections 2024-09-13T17:11:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 --2- 172.21.15.28:0/3605812900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806c540 0x7fd1480b3480 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 -- 172.21.15.28:0/3605812900 >> 172.21.15.28:0/3605812900 conn(0x7fd1480b0270 msgr2=0x7fd1480b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 -- 172.21.15.28:0/3605812900 shutdown_connections 2024-09-13T17:11:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.515+0000 7fd14d400640 1 -- 172.21.15.28:0/3605812900 wait complete. 2024-09-13T17:11:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.517+0000 7fd14d400640 1 Processor -- start 2024-09-13T17:11:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.517+0000 7fd14d400640 1 -- start start 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 0x7fd1480c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806d070 0x7fd14815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480cd700 con 0x7fd1480b17e0 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480ab180 con 0x7fd14806d070 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd14d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480ab460 con 0x7fd14806c540 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53720/0 (socket says 172.21.15.28:53720) 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd147200640 1 -- 172.21.15.28:0/1648942285 learned_addr learned my addr 172.21.15.28:0/1648942285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd13fe00640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806d070 0x7fd14815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.518+0000 7fd146800640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 0x7fd1480c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 -- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 msgr2=0x7fd1480c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 0x7fd1480c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 -- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806d070 msgr2=0x7fd14815a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806d070 0x7fd14815a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 -- 172.21.15.28:0/1648942285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140009a70 con 0x7fd1480b17e0 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd146800640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 0x7fd1480c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:34.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd147200640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd140009eb0 tx=0x7fd140004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd13ea00640 1 -- 172.21.15.28:0/1648942285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14003b820 con 0x7fd1480b17e0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd13ea00640 1 -- 172.21.15.28:0/1648942285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14003b980 con 0x7fd1480b17e0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd13ea00640 1 -- 172.21.15.28:0/1648942285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14003bcf0 con 0x7fd1480b17e0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.519+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1480ab740 con 0x7fd1480b17e0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.520+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1480ac470 con 0x7fd1480b17e0 2024-09-13T17:11:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.521+0000 7fd13ea00640 1 -- 172.21.15.28:0/1648942285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd140031400 con 0x7fd1480b17e0 2024-09-13T17:11:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.522+0000 7fd13ea00640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 0x7fd12406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.522+0000 7fd13ea00640 1 -- 172.21.15.28:0/1648942285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1400aac00 con 0x7fd1480b17e0 2024-09-13T17:11:34.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.523+0000 7fd146800640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 0x7fd12406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.523+0000 7fd150226740 1 Processor -- start 2024-09-13T17:11:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.523+0000 7fd150226740 1 -- start start 2024-09-13T17:11:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.523+0000 7fd146800640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 0x7fd12406d0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd1380011e0 tx=0x7fd138009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e155a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 0x562e51e62690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51d7f130 con 0x562e51e3c670 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51d7c8c0 con 0x562e51e151c0 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e51d781b0 con 0x562e51e12980 2024-09-13T17:11:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53734/0 (socket says 172.21.15.28:53734) 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.524+0000 7fd147200640 1 -- 172.21.15.28:0/3817160231 learned_addr learned my addr 172.21.15.28:0/3817160231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd146800640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e155a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 0x562e51e62690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 -- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 msgr2=0x562e51e62690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 0x562e51e62690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 -- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 msgr2=0x562e51e155a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e155a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 -- 172.21.15.28:0/3817160231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e51d42510 con 0x562e51e3c670 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 0x562e51e62690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd147200640 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd140079710 tx=0x7fd14007a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a51b7a02c96f504 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd137400640 1 -- 172.21.15.28:0/3817160231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140081e00 con 0x562e51e3c670 2024-09-13T17:11:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd137400640 1 -- 172.21.15.28:0/3817160231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd140077db0 con 0x562e51e3c670 2024-09-13T17:11:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.525+0000 7fd137400640 1 -- 172.21.15.28:0/3817160231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140079c70 con 0x562e51e3c670 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 -- 172.21.15.28:0/3817160231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 msgr2=0x562e51e62bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fd140079710 tx=0x7fd14007a110 comp rx=0 tx=0).stop 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 -- 172.21.15.28:0/3817160231 shutdown_connections 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c670 0x562e51e62bd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e12980 0x562e51e62690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 --2- 172.21.15.28:0/3817160231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e155a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 -- 172.21.15.28:0/3817160231 >> 172.21.15.28:0/3817160231 conn(0x562e51e11b80 msgr2=0x562e51e59da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 -- 172.21.15.28:0/3817160231 shutdown_connections 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.526+0000 7fd150226740 1 -- 172.21.15.28:0/3817160231 wait complete. 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.527+0000 7fd150226740 1 -- start start 2024-09-13T17:11:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.527+0000 7fd150226740 1 Processor -- start 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e39e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c670 0x562e51e3dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51d36570 con 0x562e51e3c1a0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51d36210 con 0x562e51e151c0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e51adf740 con 0x562e51e3c670 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd150226740 10 client.0 _ll_register_callbacks cb 0x562e51a8e2e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd147200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53746/0 (socket says 172.21.15.28:53746) 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd147200640 1 -- 172.21.15.28:0/1987679103 learned_addr learned my addr 172.21.15.28:0/1987679103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd13fe00640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c670 0x562e51e3dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd146800640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e39e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.528+0000 7fd147200640 1 -- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c670 msgr2=0x562e51e3dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd147200640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c670 0x562e51e3dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd147200640 1 -- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 msgr2=0x562e51e39e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd147200640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e39e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd147200640 1 -- 172.21.15.28:0/1987679103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1480ac470 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd147200640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd140075f10 tx=0x7fd140009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac4fcd32a54d0f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd12be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd140079100 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd140079c70 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e51a6bbe0 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd140079e30 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd150226740 10 client.6048 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x562e51d76830 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.529+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562e51a6bbe0 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd14004c070 con 0x562e51e3c1a0 2024-09-13T17:11:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 10 client.6048 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 10 client.6048 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 20 client.6048 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51e5f950 0x562e51e5fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 4 client.6048 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e51a7cee0 con 0x562e51e5f950 2024-09-13T17:11:34.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd146800640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51e5f950 0x562e51e5fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd146800640 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51e5f950 0x562e51e5fd60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec510a07c7928134 server_cookie=44732012b2b3d54 in_seq=0 out_seq=0 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.530+0000 7fd12be00640 10 client.6048 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd12be00640 1 -- 172.21.15.28:0/1987679103 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562e51a7cee0 con 0x562e51e5f950 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd12be00640 10 client.6048 handle_command_reply: tid=0 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 client.6048 shutdown 2024-09-13T17:11:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 msgr2=0x562e51e3e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd140075f10 tx=0x7fd140009750 comp rx=0 tx=0).stop 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 shutdown_connections 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51e5f950 0x562e51e5fd60 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e3e100 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c670 0x562e51e3dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd12be00640 0 client.6048 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 --2- 172.21.15.28:0/1987679103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e151c0 0x562e51e39e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.531+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 >> 172.21.15.28:0/1987679103 conn(0x562e51e12980 msgr2=0x562e51e580e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.532+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 shutdown_connections 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.532+0000 7fd150226740 1 -- 172.21.15.28:0/1987679103 wait complete. 2024-09-13T17:11:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.532+0000 7fd150226740 20 client.6048 trim_cache size 0 max 16384 2024-09-13T17:11:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.547+0000 7fd150226740 1 Processor -- start 2024-09-13T17:11:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.547+0000 7fd150226740 1 -- start start 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c1a0 0x562e51e10ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e11430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51e5c3e0 con 0x562e51e151c0 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51e60730 con 0x562e51e3c670 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e51e3a100 con 0x562e51e3c1a0 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53752/0 (socket says 172.21.15.28:53752) 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 -- 172.21.15.28:0/3717275069 learned_addr learned my addr 172.21.15.28:0/3717275069 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 -- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c1a0 msgr2=0x562e51e10ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c1a0 0x562e51e10ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 -- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 msgr2=0x562e51e11430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e11430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.548+0000 7fd146800640 1 -- 172.21.15.28:0/3717275069 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e51d187e0 con 0x562e51e151c0 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd146800640 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd138000f60 tx=0x7fd138017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7deb4e0f4ef84863 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd137e00640 1 -- 172.21.15.28:0/3717275069 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd138001470 con 0x562e51e151c0 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd137e00640 1 -- 172.21.15.28:0/3717275069 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd138007560 con 0x562e51e151c0 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd137e00640 1 -- 172.21.15.28:0/3717275069 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1380015d0 con 0x562e51e151c0 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 -- 172.21.15.28:0/3717275069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 msgr2=0x562e51e5a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd138000f60 tx=0x7fd138017040 comp rx=0 tx=0).stop 2024-09-13T17:11:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 -- 172.21.15.28:0/3717275069 shutdown_connections 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e11430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e3c1a0 0x562e51e10ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 --2- 172.21.15.28:0/3717275069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e151c0 0x562e51e5a860 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.549+0000 7fd150226740 1 -- 172.21.15.28:0/3717275069 >> 172.21.15.28:0/3717275069 conn(0x562e51e12980 msgr2=0x562e51e5fd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.550+0000 7fd150226740 1 -- 172.21.15.28:0/3717275069 shutdown_connections 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.550+0000 7fd150226740 1 -- 172.21.15.28:0/3717275069 wait complete. 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.550+0000 7fd150226740 1 -- start start 2024-09-13T17:11:34.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.550+0000 7fd150226740 1 Processor -- start 2024-09-13T17:11:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e151c0 0x562e51ec3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e5ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51ec5ce0 con 0x562e51e3c1a0 2024-09-13T17:11:34.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e51e39bb0 con 0x562e51e3c670 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e51e6bba0 con 0x562e51e151c0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd150226740 10 client.0 _ll_register_callbacks cb 0x7fd140075dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.551+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53764/0 (socket says 172.21.15.28:53764) 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 -- 172.21.15.28:0/3577015066 learned_addr learned my addr 172.21.15.28:0/3577015066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd147200640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e5ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd146800640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e151c0 0x562e51ec3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 -- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e151c0 msgr2=0x562e51ec3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e151c0 0x562e51ec3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 -- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 msgr2=0x562e51e5ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e5ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 -- 172.21.15.28:0/3577015066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd13800f040 con 0x562e51e3c1a0 2024-09-13T17:11:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd13fe00640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd148066f40 tx=0x7fd1300015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20160e9c8fdc3b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd12be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd130016450 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.552+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1300165b0 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd140077db0 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd130016770 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 10 client.6054 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x562e51d3dc80 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd140077db0 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd1300168d0 con 0x562e51e3c1a0 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 10 client.6054 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 10 client.6054 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 20 client.6054 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51ec4220 0x562e51e55700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 4 client.6054 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}] 2024-09-13T17:11:34.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml"}) v1 -- 0x562e51ad0530 con 0x562e51ec4220 2024-09-13T17:11:34.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.553+0000 7fd146800640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51ec4220 0x562e51e55700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.554+0000 7fd146800640 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51ec4220 0x562e51e55700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a84e896057028f94 server_cookie=48406e54db1672a2 in_seq=0 out_seq=0 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.554+0000 7fd12be00640 10 client.6054 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.554+0000 7fd12be00640 1 -- 172.21.15.28:0/3577015066 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562e51ad0530 con 0x562e51ec4220 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.554+0000 7fd12be00640 10 client.6054 handle_command_reply: tid=0 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 client.6054 shutdown 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 msgr2=0x562e51e5a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fd148066f40 tx=0x7fd1300015d0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 shutdown_connections 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e51e3c670 0x562e51e5ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e51ec4220 0x562e51e55700 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e51e3c1a0 0x562e51e5a860 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 --2- 172.21.15.28:0/3577015066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e51e151c0 0x562e51ec3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 >> 172.21.15.28:0/3577015066 conn(0x562e51e12980 msgr2=0x562e51e10d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd12be00640 0 client.6054 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 shutdown_connections 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 1 -- 172.21.15.28:0/3577015066 wait complete. 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.555+0000 7fd150226740 20 client.6054 trim_cache size 0 max 16384 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e79607ce-3d21-4ce8-8a85-7f54fe34e373", 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:34.560 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 msgr2=0x7fd12406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 0x7fd12406d0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd1380011e0 tx=0x7fd138009040 comp rx=0 tx=0).stop 2024-09-13T17:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 msgr2=0x7fd1480aa610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fd140009eb0 tx=0x7fd140004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 shutdown_connections 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12406ac00 0x7fd12406d0c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1480b17e0 0x7fd1480aa610 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd14806d070 0x7fd14815a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 --2- 172.21.15.28:0/1648942285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd14806c540 0x7fd1480c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 >> 172.21.15.28:0/1648942285 conn(0x7fd1480b0270 msgr2=0x7fd1480ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 shutdown_connections 2024-09-13T17:11:34.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.556+0000 7fd14d400640 1 -- 172.21.15.28:0/1648942285 wait complete. 2024-09-13T17:11:34.573 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml' returned '{'return_code': 0, 'scrub_tag': 'e79607ce-3d21-4ce8-8a85-7f54fe34e373', 'mode': 'asynchronous'}' 2024-09-13T17:11:34.573 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /' 2024-09-13T17:11:34.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / 2024-09-13T17:11:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.728+0000 7fcfaec00640 1 Processor -- start 2024-09-13T17:11:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.728+0000 7fcfaec00640 1 -- start start 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfaec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfaec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcfa80b0e00 0x7fcfa80712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfaec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80722e0 con 0x7fcfa80cc880 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfaec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa80cb3e0 con 0x7fcfa80b0e00 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfaec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80cbdb0 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfad800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfad800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:40908/0 (socket says 172.21.15.28:40908) 2024-09-13T17:11:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfad800640 1 -- 172.21.15.28:0/3902654494 learned_addr learned my addr 172.21.15.28:0/3902654494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.730+0000 7fcfae200640 1 --2- 172.21.15.28:0/3902654494 >> v2:172.21.15.189:3300/0 conn(0x7fcfa80b0e00 0x7fcfa80712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.731+0000 7fcfad800640 1 -- 172.21.15.28:0/3902654494 >> v2:172.21.15.189:3300/0 conn(0x7fcfa80b0e00 msgr2=0x7fcfa80712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.731+0000 7fcfad800640 1 --2- 172.21.15.28:0/3902654494 >> v2:172.21.15.189:3300/0 conn(0x7fcfa80b0e00 0x7fcfa80712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.731+0000 7fcfad800640 1 -- 172.21.15.28:0/3902654494 >> v1:172.21.15.189:6789/0 conn(0x7fcfa80cc880 legacy=0x7fcfa80b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.731+0000 7fcfad800640 1 -- 172.21.15.28:0/3902654494 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8072000 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfad800640 1 --2- 172.21.15.28:0/3902654494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02a1d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51018867333f7c76 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfa7e00640 1 -- 172.21.15.28:0/3902654494 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c032440 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfa7e00640 1 -- 172.21.15.28:0/3902654494 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c0325a0 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfa7e00640 1 -- 172.21.15.28:0/3902654494 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c032930 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfaec00640 1 -- 172.21.15.28:0/3902654494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 msgr2=0x7fcfa80cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.732+0000 7fcfaec00640 1 --2- 172.21.15.28:0/3902654494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c009870 tx=0x7fcf9c02a1d0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 -- 172.21.15.28:0/3902654494 shutdown_connections 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 --2- 172.21.15.28:0/3902654494 >> v2:172.21.15.189:3300/0 conn(0x7fcfa80b0e00 0x7fcfa80712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 --2- 172.21.15.28:0/3902654494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc4a0 0x7fcfa80cb870 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 -- 172.21.15.28:0/3902654494 >> 172.21.15.28:0/3902654494 conn(0x7fcfa8069e80 msgr2=0x7fcfa806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 -- 172.21.15.28:0/3902654494 shutdown_connections 2024-09-13T17:11:34.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.733+0000 7fcfaec00640 1 -- 172.21.15.28:0/3902654494 wait complete. 2024-09-13T17:11:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.734+0000 7fcfaec00640 1 Processor -- start 2024-09-13T17:11:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.734+0000 7fcfaec00640 1 -- start start 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfa80cc4a0 0x7fcfa814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc880 0x7fcfa8152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8142270 con 0x7fcfa80b0e00 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8151670 con 0x7fcfa80cc4a0 2024-09-13T17:11:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.735+0000 7fcfaec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa8151950 con 0x7fcfa80cc880 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53778/0 (socket says 172.21.15.28:53778) 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 -- 172.21.15.28:0/188867760 learned_addr learned my addr 172.21.15.28:0/188867760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfae200640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc880 0x7fcfa8152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcface00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfa80cc4a0 0x7fcfa814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 -- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc880 msgr2=0x7fcfa8152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc880 0x7fcfa8152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 -- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfa80cc4a0 msgr2=0x7fcfa814b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfa80cc4a0 0x7fcfa814b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 -- 172.21.15.28:0/188867760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf9c0094d0 con 0x7fcfa80b0e00 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.736+0000 7fcfad800640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c000c00 tx=0x7fcf9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.737+0000 7fcfa5600640 1 -- 172.21.15.28:0/188867760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c032440 con 0x7fcfa80b0e00 2024-09-13T17:11:34.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.737+0000 7fcfa5600640 1 -- 172.21.15.28:0/188867760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9c030980 con 0x7fcfa80b0e00 2024-09-13T17:11:34.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.737+0000 7fcfa5600640 1 -- 172.21.15.28:0/188867760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c030d30 con 0x7fcfa80b0e00 2024-09-13T17:11:34.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.737+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa8151c30 con 0x7fcfa80b0e00 2024-09-13T17:11:34.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.737+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa8151290 con 0x7fcfa80b0e00 2024-09-13T17:11:34.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.739+0000 7fcfa5600640 1 -- 172.21.15.28:0/188867760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf9c039040 con 0x7fcfa80b0e00 2024-09-13T17:11:34.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.740+0000 7fcfa5600640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 0x7fcf8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.740+0000 7fcfa5600640 1 -- 172.21.15.28:0/188867760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf9c0a9c00 con 0x7fcfa80b0e00 2024-09-13T17:11:34.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.740+0000 7fcface00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 0x7fcf8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.740+0000 7fcfb31e0740 1 Processor -- start 2024-09-13T17:11:34.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 -- start start 2024-09-13T17:11:34.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcface00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 0x7fcf8806d0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcfa814c1b0 tx=0x7fcf94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ad32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad75f10 0x55ef2ad763c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2aceeae0 con 0x55ef2ad301a0 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ac9bde0 con 0x55ef2ad329e0 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.741+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ac99570 con 0x55ef2ad75f10 2024-09-13T17:11:34.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53794/0 (socket says 172.21.15.28:53794) 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 -- 172.21.15.28:0/2425527243 learned_addr learned my addr 172.21.15.28:0/2425527243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcfad800640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ad32dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcfae200640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad75f10 0x55ef2ad763c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 -- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad75f10 msgr2=0x55ef2ad763c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad75f10 0x55ef2ad763c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 -- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 msgr2=0x55ef2ad32dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ad32dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 -- 172.21.15.28:0/2425527243 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef2ac36a50 con 0x55ef2ad301a0 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcfad800640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ad32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:34.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.742+0000 7fcface00640 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf94001590 tx=0x7fcf94012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43cab2a9d205ada1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcf9a000640 1 -- 172.21.15.28:0/2425527243 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940149d0 con 0x55ef2ad301a0 2024-09-13T17:11:34.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcf9a000640 1 -- 172.21.15.28:0/2425527243 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94014b30 con 0x55ef2ad301a0 2024-09-13T17:11:34.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcf9a000640 1 -- 172.21.15.28:0/2425527243 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94014ea0 con 0x55ef2ad301a0 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 -- 172.21.15.28:0/2425527243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 msgr2=0x55ef2ad2cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fcf94001590 tx=0x7fcf94012000 comp rx=0 tx=0).stop 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 -- 172.21.15.28:0/2425527243 shutdown_connections 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad75f10 0x55ef2ad763c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad301a0 0x55ef2ad2cfd0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/2425527243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ad32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 -- 172.21.15.28:0/2425527243 >> 172.21.15.28:0/2425527243 conn(0x55ef2ad2f1a0 msgr2=0x55ef2ad77860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 -- 172.21.15.28:0/2425527243 shutdown_connections 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.743+0000 7fcfb31e0740 1 -- 172.21.15.28:0/2425527243 wait complete. 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.744+0000 7fcfb31e0740 1 -- start start 2024-09-13T17:11:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.744+0000 7fcfb31e0740 1 Processor -- start 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad5c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad75f10 0x55ef2ad58950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ac94e60 con 0x55ef2ad58e90 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ac53220 con 0x55ef2ad75f10 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ac52ec0 con 0x55ef2ad329e0 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.745+0000 7fcfb31e0740 10 client.0 _ll_register_callbacks cb 0x55ef2ac8e4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53796/0 (socket says 172.21.15.28:53796) 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 -- 172.21.15.28:0/920486683 learned_addr learned my addr 172.21.15.28:0/920486683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfad800640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad5c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcface00640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad75f10 0x55ef2ad58950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 -- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 msgr2=0x55ef2ad5c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad5c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 -- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad75f10 msgr2=0x55ef2ad58950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad75f10 0x55ef2ad58950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 -- 172.21.15.28:0/920486683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa80722e0 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcfae200640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ce0 tx=0x7fcfa0013680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8142c2cf86fece9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.746+0000 7fcf8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000e040 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ef2acbe340 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfa0016630 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfa0009e10 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 10 client.6063 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55ef2ac934e0 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ef2acbe340 con 0x55ef2ad58e90 2024-09-13T17:11:34.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcfa0014040 con 0x55ef2ad58e90 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 10 client.6063 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 10 client.6063 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 20 client.6063 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad5cde0 0x55ef2ad5b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 4 client.6063 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.747+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef2ac89640 con 0x55ef2ad5cde0 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.748+0000 7fcfad800640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad5cde0 0x55ef2ad5b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.748+0000 7fcfad800640 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad5cde0 0x55ef2ad5b3f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=338cfefd8d3e25f4 server_cookie=f2d25c1fd5f814c in_seq=0 out_seq=0 2024-09-13T17:11:34.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.748+0000 7fcf8ea00640 10 client.6063 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcf8ea00640 1 -- 172.21.15.28:0/920486683 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55ef2ac89640 con 0x55ef2ad5cde0 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcf8ea00640 10 client.6063 handle_command_reply: tid=0 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 client.6063 shutdown 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 msgr2=0x55ef2ad48010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ce0 tx=0x7fcfa0013680 comp rx=0 tx=0).stop 2024-09-13T17:11:34.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 shutdown_connections 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad5cde0 0x55ef2ad5b3f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad58e90 0x55ef2ad48010 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad75f10 0x55ef2ad58950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/920486683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad5c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcf8ea00640 0 client.6063 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 >> 172.21.15.28:0/920486683 conn(0x55ef2ad301a0 msgr2=0x55ef2ad2e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 shutdown_connections 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 1 -- 172.21.15.28:0/920486683 wait complete. 2024-09-13T17:11:34.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.749+0000 7fcfb31e0740 20 client.6063 trim_cache size 0 max 16384 2024-09-13T17:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.764+0000 7fcfb31e0740 1 Processor -- start 2024-09-13T17:11:34.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.764+0000 7fcfb31e0740 1 -- start start 2024-09-13T17:11:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad798a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad58e90 0x55ef2ad6ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2a9fbed0 con 0x55ef2ad75f10 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ad77e30 con 0x55ef2ad58e90 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.765+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ad5cc50 con 0x55ef2ad329e0 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53812/0 (socket says 172.21.15.28:53812) 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 -- 172.21.15.28:0/227585829 learned_addr learned my addr 172.21.15.28:0/227585829 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfad800640 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad798a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcface00640 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad58e90 0x55ef2ad6ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 -- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 msgr2=0x55ef2ad798a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad798a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 -- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad58e90 msgr2=0x55ef2ad6ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad58e90 0x55ef2ad6ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 -- 172.21.15.28:0/227585829 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef2acc7130 con 0x55ef2ad75f10 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.766+0000 7fcfae200640 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000b010 tx=0x7fcfa0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9995dec39652d818 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcf9aa00640 1 -- 172.21.15.28:0/227585829 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000e040 con 0x55ef2ad75f10 2024-09-13T17:11:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcf9aa00640 1 -- 172.21.15.28:0/227585829 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa000f040 con 0x55ef2ad75f10 2024-09-13T17:11:34.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcf9aa00640 1 -- 172.21.15.28:0/227585829 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0009e10 con 0x55ef2ad75f10 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 -- 172.21.15.28:0/227585829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 msgr2=0x55ef2ad6d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fcfa000b010 tx=0x7fcfa0019000 comp rx=0 tx=0).stop 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 -- 172.21.15.28:0/227585829 shutdown_connections 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad6d9f0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad58e90 0x55ef2ad6ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/227585829 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad329e0 0x55ef2ad798a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 -- 172.21.15.28:0/227585829 >> 172.21.15.28:0/227585829 conn(0x55ef2ad301a0 msgr2=0x55ef2ad28820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 -- 172.21.15.28:0/227585829 shutdown_connections 2024-09-13T17:11:34.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.767+0000 7fcfb31e0740 1 -- 172.21.15.28:0/227585829 wait complete. 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.768+0000 7fcfb31e0740 1 -- start start 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.768+0000 7fcfb31e0740 1 Processor -- start 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ade4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 0x55ef2ad85e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ad575b0 con 0x55ef2ad75f10 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ad6ef20 con 0x55ef2ad329e0 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ef2ad30fb0 con 0x55ef2ad58e90 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfb31e0740 10 client.0 _ll_register_callbacks cb 0x7fcfa0009d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53828/0 (socket says 172.21.15.28:53828) 2024-09-13T17:11:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.769+0000 7fcfae200640 1 -- 172.21.15.28:0/4148395157 learned_addr learned my addr 172.21.15.28:0/4148395157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcface00640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 0x55ef2ad85e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfad800640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ade4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 -- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 msgr2=0x55ef2ad85e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 0x55ef2ad85e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 -- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 msgr2=0x55ef2ade4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ade4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 -- 172.21.15.28:0/4148395157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef2ac934e0 con 0x55ef2ad75f10 2024-09-13T17:11:34.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfad800640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ade4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcface00640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 0x55ef2ad85e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfae200640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ce0 tx=0x7fcfa0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1683b599ec084d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcf8ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa000e040 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfa00166f0 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.770+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf94014b30 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfa00168b0 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 10 client.6069 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x55ef2ac35290 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf94014b30 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcfa0003990 con 0x55ef2ad75f10 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 10 client.6069 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 10 client.6069 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:34.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 20 client.6069 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad87030 0x55ef2ad46cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 4 client.6069 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/"}] 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/"}) v1 -- 0x55ef2ac86f50 con 0x55ef2ad87030 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.771+0000 7fcfad800640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad87030 0x55ef2ad46cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.772+0000 7fcfad800640 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad87030 0x55ef2ad46cb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6836c95c29191e3 server_cookie=a73b9759d5af4519 in_seq=0 out_seq=0 2024-09-13T17:11:34.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.772+0000 7fcf8ea00640 10 client.6069 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.772+0000 7fcf8ea00640 1 -- 172.21.15.28:0/4148395157 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55ef2ac86f50 con 0x55ef2ad87030 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.772+0000 7fcf8ea00640 10 client.6069 handle_command_reply: tid=0 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.772+0000 7fcfb31e0740 1 client.6069 shutdown 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 msgr2=0x55ef2ad86350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007ce0 tx=0x7fcfa0004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 shutdown_connections 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55ef2ad87030 0x55ef2ad46cb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ef2ad75f10 0x55ef2ad86350 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ef2ad58e90 0x55ef2ad85e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 --2- 172.21.15.28:0/4148395157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ef2ad329e0 0x55ef2ade4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcf8ea00640 0 client.6069 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 >> 172.21.15.28:0/4148395157 conn(0x55ef2ad301a0 msgr2=0x55ef2ad6d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 shutdown_connections 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 1 -- 172.21.15.28:0/4148395157 wait complete. 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.773+0000 7fcfb31e0740 20 client.6069 trim_cache size 0 max 16384 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "93b417fd-2902-46cf-8244-e4e94ed67653", 2024-09-13T17:11:34.780 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:34.781 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 msgr2=0x7fcf8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 0x7fcf8806d0c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcfa814c1b0 tx=0x7fcf94009040 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 msgr2=0x7fcfa8141d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c000c00 tx=0x7fcf9c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 shutdown_connections 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf8806ac00 0x7fcf8806d0c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfa80cc880 0x7fcfa8152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfa80cc4a0 0x7fcfa814b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 --2- 172.21.15.28:0/188867760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfa80b0e00 0x7fcfa8141d30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 >> 172.21.15.28:0/188867760 conn(0x7fcfa8069e80 msgr2=0x7fcfa806fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 shutdown_connections 2024-09-13T17:11:34.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:34.774+0000 7fcfaec00640 1 -- 172.21.15.28:0/188867760 wait complete. 2024-09-13T17:11:34.791 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /' returned '{'return_code': 0, 'scrub_tag': '93b417fd-2902-46cf-8244-e4e94ed67653', 'mode': 'asynchronous'}' 2024-09-13T17:11:34.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:34.791 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_dir_1) 2024-09-13T17:11:34.829 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_1' 2024-09-13T17:11:34.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.023+0000 7fc6ba200640 1 Processor -- start 2024-09-13T17:11:35.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.023+0000 7fc6ba200640 1 -- start start 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6ba200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b40cabc0 0x7fc6b406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6ba200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b415e100 con 0x7fc6b4070cc0 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6ba200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b415e3e0 con 0x7fc6b415a540 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6ba200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40adb00 con 0x7fc6b40cabc0 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b40cabc0 0x7fc6b406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6b3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc6b4070cc0 0x7fc6b406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33844/0 (socket says 172.21.15.28:33844) 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6abe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6b40cabc0 0x7fc6b406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53836/0 (socket says 172.21.15.28:53836) 2024-09-13T17:11:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6b3e00640 1 -- 172.21.15.28:0/1348671043 learned_addr learned my addr 172.21.15.28:0/1348671043 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:35.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6b2a00640 1 -- 172.21.15.28:0/1348671043 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718790964 0 0) 0x7fc6b415e100 con 0x7fc6b4070cc0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.025+0000 7fc6b2a00640 1 -- 172.21.15.28:0/1348671043 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc698003520 con 0x7fc6b4070cc0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.026+0000 7fc6abe00640 1 -- 172.21.15.28:0/1348671043 >> v1:172.21.15.189:6789/0 conn(0x7fc6b415a540 legacy=0x7fc6b415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.026+0000 7fc6abe00640 1 -- 172.21.15.28:0/1348671043 >> v1:172.21.15.28:6789/0 conn(0x7fc6b4070cc0 legacy=0x7fc6b406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.026+0000 7fc6abe00640 1 -- 172.21.15.28:0/1348671043 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b415ef80 con 0x7fc6b40cabc0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6abe00640 1 --2- 172.21.15.28:0/1348671043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40cabc0 0x7fc6b406c430 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0097a0 tx=0x7fc69c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb55b8dd0463543b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6b2a00640 1 -- 172.21.15.28:0/1348671043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c032440 con 0x7fc6b40cabc0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6b2a00640 1 -- 172.21.15.28:0/1348671043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c0325a0 con 0x7fc6b40cabc0 2024-09-13T17:11:35.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6b2a00640 1 -- 172.21.15.28:0/1348671043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c032930 con 0x7fc6b40cabc0 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 -- 172.21.15.28:0/1348671043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40cabc0 msgr2=0x7fc6b406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 --2- 172.21.15.28:0/1348671043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40cabc0 0x7fc6b406c430 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc69c0097a0 tx=0x7fc69c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 -- 172.21.15.28:0/1348671043 shutdown_connections 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 --2- 172.21.15.28:0/1348671043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b40cabc0 0x7fc6b406c430 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 -- 172.21.15.28:0/1348671043 >> 172.21.15.28:0/1348671043 conn(0x7fc6b4069e80 msgr2=0x7fc6b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.027+0000 7fc6ba200640 1 -- 172.21.15.28:0/1348671043 shutdown_connections 2024-09-13T17:11:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.028+0000 7fc6ba200640 1 -- 172.21.15.28:0/1348671043 wait complete. 2024-09-13T17:11:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.029+0000 7fc6ba200640 1 Processor -- start 2024-09-13T17:11:35.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.029+0000 7fc6ba200640 1 -- start start 2024-09-13T17:11:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4070cc0 0x7fc6b41e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6b40cabc0 0x7fc6b40adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40ae9a0 con 0x7fc6b415a540 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41e6910 con 0x7fc6b4070cc0 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6ba200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b41e5870 con 0x7fc6b40cabc0 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53842/0 (socket says 172.21.15.28:53842) 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.030+0000 7fc6b3e00640 1 -- 172.21.15.28:0/3156620299 learned_addr learned my addr 172.21.15.28:0/3156620299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3400640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4070cc0 0x7fc6b41e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6abe00640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6b40cabc0 0x7fc6b40adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 -- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6b40cabc0 msgr2=0x7fc6b40adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6b40cabc0 0x7fc6b40adf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 -- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4070cc0 msgr2=0x7fc6b41e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4070cc0 0x7fc6b41e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 -- 172.21.15.28:0/3156620299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c009400 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6b3e00640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00cb40 tx=0x7fc6ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.031+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac00aca0 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.032+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac00ae00 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.032+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac012620 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.032+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b41e5b50 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.032+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b41e6110 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.033+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b406c430 con 0x7fc6b415a540 2024-09-13T17:11:35.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.034+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6ac007070 con 0x7fc6b415a540 2024-09-13T17:11:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.034+0000 7fc6ab400640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 0x7fc69006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.035+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6ac0892c0 con 0x7fc6b415a540 2024-09-13T17:11:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.035+0000 7fc6b3400640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 0x7fc69006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.036+0000 7fc6b3400640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 0x7fc69006d0f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4000f60 tx=0x7fc6a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.039+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6ac054540 con 0x7fc6b415a540 2024-09-13T17:11:35.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.254+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6b4156770 con 0x7fc6b415a540 2024-09-13T17:11:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.255+0000 7fc6ab400640 1 -- 172.21.15.28:0/3156620299 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4737 (secure 0 0 0) 0x7fc6ac057b60 con 0x7fc6b415a540 2024-09-13T17:11:35.257 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:35.257 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:29.038624+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:35.257 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 msgr2=0x7fc69006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 0x7fc69006d0f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4000f60 tx=0x7fc6a4009040 comp rx=0 tx=0).stop 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 msgr2=0x7fc6b40ae460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac00cb40 tx=0x7fc6ac00a060 comp rx=0 tx=0).stop 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 shutdown_connections 2024-09-13T17:11:35.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69006ac30 0x7fc69006d0f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6b415a540 0x7fc6b40ae460 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6b40cabc0 0x7fc6b40adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 --2- 172.21.15.28:0/3156620299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6b4070cc0 0x7fc6b41e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 >> 172.21.15.28:0/3156620299 conn(0x7fc6b4069e80 msgr2=0x7fc6b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 shutdown_connections 2024-09-13T17:11:35.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.257+0000 7fc6ba200640 1 -- 172.21.15.28:0/3156620299 wait complete. 2024-09-13T17:11:35.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/new_dir_1 2024-09-13T17:11:35.439 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:35.439 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:35.439 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:35.452 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_dir_1' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:35.453 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:35.453 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:35.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:35.488 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_1 bs=1M conv=fdatasync count=1 seek=0) 2024-09-13T17:11:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:11:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:11:35.581 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172432 s, 60.8 MB/s 2024-09-13T17:11:35.582 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_1' 2024-09-13T17:11:35.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:35.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.736+0000 7f4f21600640 1 Processor -- start 2024-09-13T17:11:35.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.737+0000 7f4f21600640 1 -- start start 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f21600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4f1c159440 0x7f4f1c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f21600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4f1c0b29f0 0x7f4f1c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f21600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1c0c3850 con 0x7f4f1c159820 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f21600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c15a930 con 0x7f4f1c159440 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f21600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1c159d20 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f1be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4f1c159440 0x7f4f1c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f1be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4f1c159440 0x7f4f1c0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53850/0 (socket says 172.21.15.28:53850) 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.738+0000 7f4f1be00640 1 -- 172.21.15.28:0/2828089972 learned_addr learned my addr 172.21.15.28:0/2828089972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.739+0000 7f4f20c00640 1 --2- 172.21.15.28:0/2828089972 >> v2:172.21.15.189:3300/0 conn(0x7f4f1c0b29f0 0x7f4f1c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.739+0000 7f4f1be00640 1 -- 172.21.15.28:0/2828089972 >> v2:172.21.15.189:3300/0 conn(0x7f4f1c0b29f0 msgr2=0x7f4f1c0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.739+0000 7f4f1be00640 1 --2- 172.21.15.28:0/2828089972 >> v2:172.21.15.189:3300/0 conn(0x7f4f1c0b29f0 0x7f4f1c0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.739+0000 7f4f1be00640 1 -- 172.21.15.28:0/2828089972 >> v1:172.21.15.189:6789/0 conn(0x7f4f1c159820 legacy=0x7f4f1c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.739+0000 7f4f1be00640 1 -- 172.21.15.28:0/2828089972 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1c15a000 con 0x7f4f1c159440 2024-09-13T17:11:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f1be00640 1 --2- 172.21.15.28:0/2828089972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c159440 0x7f4f1c0b0440 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009870 tx=0x7f4f0c02c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68a6ce959c2648d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f1b400640 1 -- 172.21.15.28:0/2828089972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c031710 con 0x7f4f1c159440 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f1b400640 1 -- 172.21.15.28:0/2828089972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c02f050 con 0x7f4f1c159440 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f1b400640 1 -- 172.21.15.28:0/2828089972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c031a90 con 0x7f4f1c159440 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f21600640 1 -- 172.21.15.28:0/2828089972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c159440 msgr2=0x7f4f1c0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.740+0000 7f4f21600640 1 --2- 172.21.15.28:0/2828089972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c159440 0x7f4f1c0b0440 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c009870 tx=0x7f4f0c02c510 comp rx=0 tx=0).stop 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 -- 172.21.15.28:0/2828089972 shutdown_connections 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 --2- 172.21.15.28:0/2828089972 >> v2:172.21.15.189:3300/0 conn(0x7f4f1c0b29f0 0x7f4f1c0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 --2- 172.21.15.28:0/2828089972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c159440 0x7f4f1c0b0440 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 -- 172.21.15.28:0/2828089972 >> 172.21.15.28:0/2828089972 conn(0x7f4f1c072e40 msgr2=0x7f4f1c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 -- 172.21.15.28:0/2828089972 shutdown_connections 2024-09-13T17:11:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.741+0000 7f4f21600640 1 -- 172.21.15.28:0/2828089972 wait complete. 2024-09-13T17:11:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.742+0000 7f4f21600640 1 Processor -- start 2024-09-13T17:11:35.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.742+0000 7f4f21600640 1 -- start start 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f1c159440 0x7f4f1c1485f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f1c159820 0x7f4f1c15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c156340 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c148b30 con 0x7f4f1c159440 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f21600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1c155730 con 0x7f4f1c159820 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.743+0000 7f4f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53866/0 (socket says 172.21.15.28:53866) 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 -- 172.21.15.28:0/2529615385 learned_addr learned my addr 172.21.15.28:0/2529615385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:35.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f13400640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f1c159440 0x7f4f1c1485f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f20c00640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f1c159820 0x7f4f1c15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 -- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f1c159820 msgr2=0x7f4f1c15bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f1c159820 0x7f4f1c15bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 -- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f1c159440 msgr2=0x7f4f1c1485f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f1c159440 0x7f4f1c1485f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 -- 172.21.15.28:0/2529615385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c0094d0 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.744+0000 7f4f1be00640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c000c00 tx=0x7f4f0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.745+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c004230 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.745+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f0c004390 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.745+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c003850 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.745+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f1c155a10 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.745+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f1c155fd0 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.746+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f1c156160 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.747+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f0c044030 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.748+0000 7f4f18c00640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 0x7f4ef806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.748+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f0c042070 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.748+0000 7f4f13400640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 0x7f4ef806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.749+0000 7f4f13400640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 0x7f4ef806d0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c14d4c0 tx=0x7f4f04009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:35.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.752+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4f0c076e50 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.965+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f1c0b22e0 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.966+0000 7f4f18c00640 1 -- 172.21.15.28:0/2529615385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+4737 (secure 0 0 0) 0x7f4f0c038090 con 0x7f4f1c0b29f0 2024-09-13T17:11:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:35.968 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:29.038624+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":8,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:35.968 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 37 2024-09-13T17:11:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 msgr2=0x7f4ef806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 0x7f4ef806d0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c14d4c0 tx=0x7f4f04009060 comp rx=0 tx=0).stop 2024-09-13T17:11:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 msgr2=0x7f4f1c14cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c000c00 tx=0x7f4f0c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 shutdown_connections 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ef806ac00 0x7f4ef806d0c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f1c159820 0x7f4f1c15bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f1c159440 0x7f4f1c1485f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 --2- 172.21.15.28:0/2529615385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f1c0b29f0 0x7f4f1c14cf80 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 >> 172.21.15.28:0/2529615385 conn(0x7f4f1c072e40 msgr2=0x7f4f1c153880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 shutdown_connections 2024-09-13T17:11:35.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:35.968+0000 7f4f21600640 1 -- 172.21.15.28:0/2529615385 wait complete. 2024-09-13T17:11:35.986 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path /scrub_test_path/ceph-qa-suite/new_file_1 2024-09-13T17:11:36.150 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:36.150 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:36.150 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:36.163 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /scrub_test_path/ceph-qa-suite/new_file_1' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_test_path/ceph-qa-suite/new_file_1").st_ino) 2024-09-13T17:11:36.164 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:11:36.263 INFO:teuthology.orchestra.run.smithi028.stdout:1099511630188 2024-09-13T17:11:36.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-09-13T17:11:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.646+0000 7fdc11000640 1 Processor -- start 2024-09-13T17:11:36.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.646+0000 7fdc11000640 1 -- start start 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc11000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdc0c15a780 0x7fdc0c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc11000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 0x7fdc0c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc11000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc0c06c8b0 con 0x7fdc0c06bb40 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc11000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0c06cb90 con 0x7fdc0c15a780 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc11000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c0cc980 con 0x7fdc0c15ab60 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc0a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdc0c06bb40 0x7fdc0c06bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33856/0 (socket says 172.21.15.28:33856) 2024-09-13T17:11:36.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.647+0000 7fdc0a800640 1 -- 172.21.15.28:0/1509423474 learned_addr learned my addr 172.21.15.28:0/1509423474 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1509423474 >> v2:172.21.15.189:3300/0 conn(0x7fdc0c15a780 0x7fdc0c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 0x7fdc0c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09400640 1 -- 172.21.15.28:0/1509423474 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584105983 0 0) 0x7fdc0c06c8b0 con 0x7fdc0c06bb40 2024-09-13T17:11:36.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09400640 1 -- 172.21.15.28:0/1509423474 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0003520 con 0x7fdc0c06bb40 2024-09-13T17:11:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09e00640 1 -- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 msgr2=0x7fdc0c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 0x7fdc0c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09e00640 1 -- 172.21.15.28:0/1509423474 >> v1:172.21.15.28:6789/0 conn(0x7fdc0c06bb40 legacy=0x7fdc0c06bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc09e00640 1 -- 172.21.15.28:0/1509423474 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c0ccc60 con 0x7fdc0c15a780 2024-09-13T17:11:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.648+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 0x7fdc0c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.649+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c070d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097a0 tx=0x7fdbfc02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41ffa8c467445ef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc09400640 1 -- 172.21.15.28:0/1509423474 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc032440 con 0x7fdc0c15a780 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc09400640 1 -- 172.21.15.28:0/1509423474 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc0325a0 con 0x7fdc0c15a780 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc09400640 1 -- 172.21.15.28:0/1509423474 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc032930 con 0x7fdc0c15a780 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 -- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 msgr2=0x7fdc0c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c070d80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0097a0 tx=0x7fdbfc02a180 comp rx=0 tx=0).stop 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 -- 172.21.15.28:0/1509423474 shutdown_connections 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c15ab60 0x7fdc0c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 --2- 172.21.15.28:0/1509423474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c070d80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 -- 172.21.15.28:0/1509423474 >> 172.21.15.28:0/1509423474 conn(0x7fdc0c0b0270 msgr2=0x7fdc0c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 -- 172.21.15.28:0/1509423474 shutdown_connections 2024-09-13T17:11:36.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.650+0000 7fdc11000640 1 -- 172.21.15.28:0/1509423474 wait complete. 2024-09-13T17:11:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.652+0000 7fdc11000640 1 Processor -- start 2024-09-13T17:11:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.652+0000 7fdc11000640 1 -- start start 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c06bb40 0x7fdc0c14ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c0af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c1f1d80 con 0x7fdc0c15ab60 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c1eec20 con 0x7fdc0c15a780 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc11000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c1eef00 con 0x7fdc0c06bb40 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53880/0 (socket says 172.21.15.28:53880) 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.653+0000 7fdc0a800640 1 -- 172.21.15.28:0/2386453713 learned_addr learned my addr 172.21.15.28:0/2386453713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc09e00640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c06bb40 0x7fdc0c14ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc03e00640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c0af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 -- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c06bb40 msgr2=0x7fdc0c14ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c06bb40 0x7fdc0c14ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 -- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 msgr2=0x7fdc0c0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c0af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 -- 172.21.15.28:0/2386453713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf4009400 con 0x7fdc0c15ab60 2024-09-13T17:11:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.654+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400cb40 tx=0x7fdc0400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.655+0000 7fdc02000640 1 -- 172.21.15.28:0/2386453713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400aca0 con 0x7fdc0c15ab60 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.655+0000 7fdc02000640 1 -- 172.21.15.28:0/2386453713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc0400ae00 con 0x7fdc0c15ab60 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.655+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc009400 con 0x7fdc0c15ab60 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.655+0000 7fdc02000640 1 -- 172.21.15.28:0/2386453713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04012620 con 0x7fdc0c15ab60 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.655+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c1ef4c0 con 0x7fdc0c15ab60 2024-09-13T17:11:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.657+0000 7fdc02000640 1 -- 172.21.15.28:0/2386453713 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdc04007070 con 0x7fdc0c15ab60 2024-09-13T17:11:36.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.658+0000 7fdc02000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 0x7fdbe806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.658+0000 7fdc02000640 1 -- 172.21.15.28:0/2386453713 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdc040893f0 con 0x7fdc0c15ab60 2024-09-13T17:11:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.659+0000 7fdc09e00640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 0x7fdbe806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.659+0000 7fdc13e66740 1 Processor -- start 2024-09-13T17:11:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.659+0000 7fdc13e66740 1 -- start start 2024-09-13T17:11:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc09e00640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 0x7fdbe806d0c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x56225459b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545e8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622544a7020 con 0x5622545985d0 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562254504ba0 con 0x56225459ae10 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562254502330 con 0x5622545c22c0 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53888/0 (socket says 172.21.15.28:53888) 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 -- 172.21.15.28:0/1774849299 learned_addr learned my addr 172.21.15.28:0/1774849299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x56225459b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc0a800640 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545e8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 -- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 msgr2=0x5622545e8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545e8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 -- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 msgr2=0x56225459b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.660+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x56225459b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.661+0000 7fdc03e00640 1 -- 172.21.15.28:0/1774849299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622544afcf0 con 0x5622545985d0 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.661+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009c80 tx=0x7fdbf4011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b181e4fcc7d2793 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.661+0000 7fdbfaa00640 1 -- 172.21.15.28:0/1774849299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4011ec0 con 0x5622545985d0 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.661+0000 7fdbfaa00640 1 -- 172.21.15.28:0/1774849299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4004030 con 0x5622545985d0 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.661+0000 7fdbfaa00640 1 -- 172.21.15.28:0/1774849299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40043a0 con 0x5622545985d0 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- 172.21.15.28:0/1774849299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 msgr2=0x5622545e82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009c80 tx=0x7fdbf4011690 comp rx=0 tx=0).stop 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- 172.21.15.28:0/1774849299 shutdown_connections 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545e8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545985d0 0x5622545e82e0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1774849299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x56225459b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- 172.21.15.28:0/1774849299 >> 172.21.15.28:0/1774849299 conn(0x562254597540 msgr2=0x5622545df9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- 172.21.15.28:0/1774849299 shutdown_connections 2024-09-13T17:11:36.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- 172.21.15.28:0/1774849299 wait complete. 2024-09-13T17:11:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.662+0000 7fdc13e66740 1 -- start start 2024-09-13T17:11:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.663+0000 7fdc13e66740 1 Processor -- start 2024-09-13T17:11:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545b7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545bbf10 0x5622545b9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622544fdc20 con 0x56225459ae10 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56225449f1b0 con 0x5622545bbf10 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56225449ee50 con 0x5622545c22c0 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc13e66740 10 client.0 _ll_register_callbacks cb 0x5622544f7290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53896/0 (socket says 172.21.15.28:53896) 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 -- 172.21.15.28:0/1803992252 learned_addr learned my addr 172.21.15.28:0/1803992252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc0a800640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545bbf10 0x5622545b9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 -- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 msgr2=0x5622545b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 -- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545bbf10 msgr2=0x5622545b9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545bbf10 0x5622545b9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.664+0000 7fdc09e00640 1 -- 172.21.15.28:0/1803992252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbf40098b0 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdc09e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0098c0 tx=0x7fdbfc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e47f25eb1e824d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdbef400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc03cd10 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbfc053ae0 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562254526eb0 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbfc03ce70 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdc13e66740 10 client.6096 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x5622544fc2a0 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.665+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562254526eb0 con 0x56225459ae10 2024-09-13T17:11:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdbfc044040 con 0x56225459ae10 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 10 client.6096 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 10 client.6096 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 20 client.6096 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545bc970 0x5622545bcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 4 client.6096 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622544f2340 con 0x5622545bc970 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545bc970 0x5622545bcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdc03e00640 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545bc970 0x5622545bcd80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa0886d8237b96b1 server_cookie=8d694aca41b758fc in_seq=0 out_seq=0 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.666+0000 7fdbef400640 10 client.6096 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdbef400640 1 -- 172.21.15.28:0/1803992252 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5622544f2340 con 0x5622545bc970 2024-09-13T17:11:36.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdbef400640 10 client.6096 handle_command_reply: tid=0 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 client.6096 shutdown 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 msgr2=0x5622545b8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc0098c0 tx=0x7fdbfc004730 comp rx=0 tx=0).stop 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 shutdown_connections 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545bbf10 0x5622545b9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545c22c0 0x5622545b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545bc970 0x5622545bcd80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 --2- 172.21.15.28:0/1803992252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56225459ae10 0x5622545b8de0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 >> 172.21.15.28:0/1803992252 conn(0x5622545985d0 msgr2=0x5622545e2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.667+0000 7fdbef400640 0 client.6096 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.668+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 shutdown_connections 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.668+0000 7fdc13e66740 1 -- 172.21.15.28:0/1803992252 wait complete. 2024-09-13T17:11:36.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.668+0000 7fdc13e66740 20 client.6096 trim_cache size 0 max 16384 2024-09-13T17:11:36.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.679+0000 7fdc13e66740 1 Processor -- start 2024-09-13T17:11:36.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.679+0000 7fdc13e66740 1 -- start start 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x5622545dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545bbf10 0x5622545df3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562254265040 con 0x5622545c22c0 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562254595dc0 con 0x56225459ae10 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.680+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562254594ea0 con 0x5622545bbf10 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53898/0 (socket says 172.21.15.28:53898) 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 -- 172.21.15.28:0/2889360514 learned_addr learned my addr 172.21.15.28:0/2889360514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc03e00640 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545bbf10 0x5622545df3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc09e00640 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x5622545dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 -- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545bbf10 msgr2=0x5622545df3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545bbf10 0x5622545df3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 -- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 msgr2=0x5622545dd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x5622545dd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 -- 172.21.15.28:0/2889360514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562254530200 con 0x5622545c22c0 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdc0a800640 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc04057f20 tx=0x7fdc04089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c28aee9ed4476d5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.681+0000 7fdbfb400640 1 -- 172.21.15.28:0/2889360514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc040890b0 con 0x5622545c22c0 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdbfb400640 1 -- 172.21.15.28:0/2889360514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04089210 con 0x5622545c22c0 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdbfb400640 1 -- 172.21.15.28:0/2889360514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04058480 con 0x5622545c22c0 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 -- 172.21.15.28:0/2889360514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 msgr2=0x5622545c0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdc04057f20 tx=0x7fdc04089040 comp rx=0 tx=0).stop 2024-09-13T17:11:36.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 -- 172.21.15.28:0/2889360514 shutdown_connections 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545c22c0 0x5622545c0a80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5622545bbf10 0x5622545df3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 --2- 172.21.15.28:0/2889360514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56225459ae10 0x5622545dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 -- 172.21.15.28:0/2889360514 >> 172.21.15.28:0/2889360514 conn(0x5622545985d0 msgr2=0x5622545b9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 -- 172.21.15.28:0/2889360514 shutdown_connections 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.682+0000 7fdc13e66740 1 -- 172.21.15.28:0/2889360514 wait complete. 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.683+0000 7fdc13e66740 1 -- start start 2024-09-13T17:11:36.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.683+0000 7fdc13e66740 1 Processor -- start 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56225459ae10 0x5622545f8440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545c22c0 0x5622545f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56225464ae30 con 0x5622545bbf10 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622545f1380 con 0x5622545c22c0 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5622545bda50 con 0x56225459ae10 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc13e66740 10 client.0 _ll_register_callbacks cb 0x7fdbf00036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:36.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc03e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53910/0 (socket says 172.21.15.28:53910) 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc03e00640 1 -- 172.21.15.28:0/62044378 learned_addr learned my addr 172.21.15.28:0/62044378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc09e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56225459ae10 0x5622545f8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.684+0000 7fdc0a800640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545c22c0 0x5622545f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 -- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56225459ae10 msgr2=0x5622545f8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56225459ae10 0x5622545f8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 -- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545c22c0 msgr2=0x5622545f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545c22c0 0x5622545f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 -- 172.21.15.28:0/62044378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c1ef4c0 con 0x5622545bbf10 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc03e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4011b90 tx=0x7fdbf4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390d13a22ed6e13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdbef400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:36.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf402a670 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbf4004030 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.685+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbf40044d0 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbf402a7d0 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 10 client.6102 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x56225449e5f0 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf4004030 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdbf4003610 con 0x5622545bbf10 2024-09-13T17:11:36.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 10 client.6102 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 10 client.6102 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 20 client.6102 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545b7950 0x5622545f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 4 client.6102 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5622541f14e0 con 0x5622545b7950 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.686+0000 7fdc09e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545b7950 0x5622545f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc09e00640 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545b7950 0x5622545f1d50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92571f72e24951e5 server_cookie=d042ddc3ead2ccc5 in_seq=0 out_seq=0 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdbef400640 10 client.6102 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdbef400640 1 -- 172.21.15.28:0/62044378 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5622541f14e0 con 0x5622545b7950 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdbef400640 10 client.6102 handle_command_reply: tid=0 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 client.6102 shutdown 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 msgr2=0x5622545f8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4011b90 tx=0x7fdbf4004730 comp rx=0 tx=0).stop 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 shutdown_connections 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622545c22c0 0x5622545f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5622545b7950 0x5622545f1d50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5622545bbf10 0x5622545f8980 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 --2- 172.21.15.28:0/62044378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56225459ae10 0x5622545f8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdbef400640 0 client.6102 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 >> 172.21.15.28:0/62044378 conn(0x5622545985d0 msgr2=0x5622545b9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 shutdown_connections 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.687+0000 7fdc13e66740 1 -- 172.21.15.28:0/62044378 wait complete. 2024-09-13T17:11:36.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc13e66740 20 client.6102 trim_cache size 0 max 16384 2024-09-13T17:11:36.695 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 msgr2=0x7fdbe806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 0x7fdbe806d0c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc02e000 comp rx=0 tx=0).stop 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 msgr2=0x7fdc0c14f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400cb40 tx=0x7fdc0400a060 comp rx=0 tx=0).stop 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 shutdown_connections 2024-09-13T17:11:36.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdbe806ac00 0x7fdbe806d0c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc0c15ab60 0x7fdc0c14f8c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c15a780 0x7fdc0c0af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 --2- 172.21.15.28:0/2386453713 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc0c06bb40 0x7fdc0c14ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 >> 172.21.15.28:0/2386453713 conn(0x7fdc0c0b0270 msgr2=0x7fdc0c0ce820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 shutdown_connections 2024-09-13T17:11:36.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.688+0000 7fdc11000640 1 -- 172.21.15.28:0/2386453713 wait complete. 2024-09-13T17:11:36.705 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data rmxattr 1000000096c.00000000 parent 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.779+0000 7f54a8fd1f40 1 Processor -- start 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 -- start start 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd926f48a0 0x55dd927bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd927acc40 0x55dd927bfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd92636fe0 con 0x55dd926ecc00 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd92636c80 con 0x55dd927acc40 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.780+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd9262d8f0 con 0x55dd926f48a0 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53922/0 (socket says 172.21.15.28:53922) 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 -- 172.21.15.28:0/762929413 learned_addr learned my addr 172.21.15.28:0/762929413 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a5800640 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd926f48a0 0x55dd927bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6c00640 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd927acc40 0x55dd927bfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 -- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd926f48a0 msgr2=0x55dd927bf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd926f48a0 0x55dd927bf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 -- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd927acc40 msgr2=0x55dd927bfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd927acc40 0x55dd927bfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 -- 172.21.15.28:0/762929413 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498005de0 con 0x55dd926ecc00 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.781+0000 7f54a6200640 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5498025b60 tx=0x7f5498051000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b83b08337666984 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f549fe00640 1 -- 172.21.15.28:0/762929413 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498055680 con 0x55dd926ecc00 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f549fe00640 1 -- 172.21.15.28:0/762929413 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498055950 con 0x55dd926ecc00 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f549fe00640 1 -- 172.21.15.28:0/762929413 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498055ce0 con 0x55dd926ecc00 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/762929413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 msgr2=0x55dd926ed260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f5498025b60 tx=0x7f5498051000 comp rx=0 tx=0).stop 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/762929413 shutdown_connections 2024-09-13T17:11:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd927acc40 0x55dd927bfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd926f48a0 0x55dd927bf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/762929413 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd926ed260 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/762929413 >> 172.21.15.28:0/762929413 conn(0x55dd9271f270 msgr2=0x55dd9271e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/762929413 shutdown_connections 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.782+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/762929413 wait complete. 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.783+0000 7f54a8fd1f40 1 Processor -- start 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.783+0000 7f54a8fd1f40 1 -- start start 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd927c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd926f48a0 0x55dd92739530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd927acc40 0x55dd927bf5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd927a9b50 con 0x55dd926ecc00 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd927c1fd0 con 0x55dd926f48a0 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a8fd1f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd927a47e0 con 0x55dd927acc40 2024-09-13T17:11:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd927c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd927c45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53924/0 (socket says 172.21.15.28:53924) 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a6200640 1 -- 172.21.15.28:0/2408370008 learned_addr learned my addr 172.21.15.28:0/2408370008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.784+0000 7f54a6c00640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd927acc40 0x55dd927bf5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a5800640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd926f48a0 0x55dd92739530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 -- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd927acc40 msgr2=0x55dd927bf5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd927acc40 0x55dd927bf5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 -- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd926f48a0 msgr2=0x55dd92739530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd926f48a0 0x55dd92739530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 -- 172.21.15.28:0/2408370008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd92638bd0 con 0x55dd926ecc00 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6c00640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55dd927acc40 0x55dd927bf5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a5800640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dd926f48a0 0x55dd92739530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a6200640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd927c45b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5498064040 tx=0x7f5498003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f549ea00640 1 -- 172.21.15.28:0/2408370008 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498055ac0 con 0x55dd926ecc00 2024-09-13T17:11:36.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f549ea00640 1 -- 172.21.15.28:0/2408370008 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5498055c20 con 0x55dd926ecc00 2024-09-13T17:11:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f549ea00640 1 -- 172.21.15.28:0/2408370008 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5498061450 con 0x55dd926ecc00 2024-09-13T17:11:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.785+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dd9265fcc0 con 0x55dd926ecc00 2024-09-13T17:11:36.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.786+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd92645b20 con 0x55dd926ecc00 2024-09-13T17:11:36.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.787+0000 7f549ea00640 1 -- 172.21.15.28:0/2408370008 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5498068040 con 0x55dd926ecc00 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.788+0000 7f549ea00640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54880695e0 0x7f548806baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.788+0000 7f549ea00640 1 -- 172.21.15.28:0/2408370008 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54980cc150 con 0x55dd926ecc00 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.789+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x55dd927a76a0 0x55dd927aa490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.789+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 --> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] -- osd_op(unknown.0.0:1 7.2b 7:d716f75e:::1000000096c.00000000:head [rmxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) v8 -- 0x55dd927aa9d0 con 0x55dd927a76a0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.789+0000 7f54a5800640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54880695e0 0x7f548806baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.789+0000 7f54a6c00640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x55dd927a76a0 0x55dd927aa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.789+0000 7f54a5800640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54880695e0 0x7f548806baa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55dd9285fc50 tx=0x7f549400e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.790+0000 7f54a6c00640 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x55dd927a76a0 0x55dd927aa490 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.791+0000 7f54a6c00640 1 -- 172.21.15.28:0/2408370008 <== osd.3 v2:172.21.15.28:6814/3941122226 1 ==== osd_op_reply(1 1000000096c.00000000 [rmxattr] v48'32 uv32 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f54a0002af0 con 0x55dd927a76a0 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.791+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x55dd927a76a0 msgr2=0x55dd927aa490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x55dd927a76a0 0x55dd927aa490 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54880695e0 msgr2=0x7f548806baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f54880695e0 0x7f548806baa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55dd9285fc50 tx=0x7f549400e040 comp rx=0 tx=0).stop 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 msgr2=0x55dd927c45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 --2- 172.21.15.28:0/2408370008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dd926ecc00 0x55dd927c45b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f5498064040 tx=0x7f5498003020 comp rx=0 tx=0).stop 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a6c00640 1 -- 172.21.15.28:0/2408370008 reap_dead start 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 shutdown_connections 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 >> 172.21.15.28:0/2408370008 conn(0x55dd9271f270 msgr2=0x55dd926ec320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 shutdown_connections 2024-09-13T17:11:36.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.792+0000 7f54a8fd1f40 1 -- 172.21.15.28:0/2408370008 wait complete. 2024-09-13T17:11:36.798 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_1' 2024-09-13T17:11:36.798 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start /scrub_test_path/ceph-qa-suite/new_file_1 2024-09-13T17:11:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.955+0000 7fb693e00640 1 Processor -- start 2024-09-13T17:11:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.955+0000 7fb693e00640 1 -- start start 2024-09-13T17:11:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb6940cf980 0x7fb6940d0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6940d1230 0x7fb6940b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6940d19b0 con 0x7fb69415ca40 2024-09-13T17:11:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940d1c90 con 0x7fb6940d1230 2024-09-13T17:11:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6940d1f70 con 0x7fb6940cf980 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6940d1230 0x7fb6940b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6940d1230 0x7fb6940b5ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53928/0 (socket says 172.21.15.28:53928) 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.956+0000 7fb693400640 1 -- 172.21.15.28:0/1492109197 learned_addr learned my addr 172.21.15.28:0/1492109197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.957+0000 7fb692000640 1 --2- 172.21.15.28:0/1492109197 >> v2:172.21.15.189:3300/0 conn(0x7fb6940cf980 0x7fb6940d0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.957+0000 7fb693400640 1 -- 172.21.15.28:0/1492109197 >> v2:172.21.15.189:3300/0 conn(0x7fb6940cf980 msgr2=0x7fb6940d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.957+0000 7fb693400640 1 --2- 172.21.15.28:0/1492109197 >> v2:172.21.15.189:3300/0 conn(0x7fb6940cf980 0x7fb6940d0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.957+0000 7fb693400640 1 -- 172.21.15.28:0/1492109197 >> v1:172.21.15.189:6789/0 conn(0x7fb69415ca40 legacy=0x7fb6940cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.957+0000 7fb693400640 1 -- 172.21.15.28:0/1492109197 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6940d2250 con 0x7fb6940d1230 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.958+0000 7fb693400640 1 --2- 172.21.15.28:0/1492109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6940b5ea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009e50 tx=0x7fb68c02a760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94a8832f340e71e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.958+0000 7fb691600640 1 -- 172.21.15.28:0/1492109197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c031600 con 0x7fb6940d1230 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.958+0000 7fb691600640 1 -- 172.21.15.28:0/1492109197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c02d050 con 0x7fb6940d1230 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.958+0000 7fb691600640 1 -- 172.21.15.28:0/1492109197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c031980 con 0x7fb6940d1230 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 -- 172.21.15.28:0/1492109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 msgr2=0x7fb6940b5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 --2- 172.21.15.28:0/1492109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6940b5ea0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009e50 tx=0x7fb68c02a760 comp rx=0 tx=0).stop 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 -- 172.21.15.28:0/1492109197 shutdown_connections 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 --2- 172.21.15.28:0/1492109197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6940b5ea0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 --2- 172.21.15.28:0/1492109197 >> v2:172.21.15.189:3300/0 conn(0x7fb6940cf980 0x7fb6940d0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 -- 172.21.15.28:0/1492109197 >> 172.21.15.28:0/1492109197 conn(0x7fb694070ad0 msgr2=0x7fb6940b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 -- 172.21.15.28:0/1492109197 shutdown_connections 2024-09-13T17:11:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.959+0000 7fb693e00640 1 -- 172.21.15.28:0/1492109197 wait complete. 2024-09-13T17:11:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.960+0000 7fb693e00640 1 Processor -- start 2024-09-13T17:11:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.961+0000 7fb693e00640 1 -- start start 2024-09-13T17:11:36.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.961+0000 7fb693e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6940cf980 0x7fb6940c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.961+0000 7fb693e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb693e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6941e6160 0x7fb6940c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb693e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940c8760 con 0x7fb6940d1230 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb693e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940c8a40 con 0x7fb6940cf980 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb693e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6940c8d20 con 0x7fb6941e6160 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53934/0 (socket says 172.21.15.28:53934) 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 -- 172.21.15.28:0/1020715734 learned_addr learned my addr 172.21.15.28:0/1020715734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692a00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6940cf980 0x7fb6940c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb693400640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6941e6160 0x7fb6940c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 -- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6941e6160 msgr2=0x7fb6940c6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6941e6160 0x7fb6940c6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 -- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6940cf980 msgr2=0x7fb6940c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6940cf980 0x7fb6940c7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.962+0000 7fb692000640 1 -- 172.21.15.28:0/1020715734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c009b30 con 0x7fb6940d1230 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb692000640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb68000d7d0 tx=0x7fb68000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb68b400640 1 -- 172.21.15.28:0/1020715734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000b6c0 con 0x7fb6940d1230 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb68b400640 1 -- 172.21.15.28:0/1020715734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68000b820 con 0x7fb6940d1230 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb68b400640 1 -- 172.21.15.28:0/1020715734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68000bbb0 con 0x7fb6940d1230 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6940c9000 con 0x7fb6940d1230 2024-09-13T17:11:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.963+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6940c83a0 con 0x7fb6940d1230 2024-09-13T17:11:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.965+0000 7fb68b400640 1 -- 172.21.15.28:0/1020715734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb68000bd10 con 0x7fb6940d1230 2024-09-13T17:11:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.966+0000 7fb68b400640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 0x7fb67406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.966+0000 7fb68b400640 1 -- 172.21.15.28:0/1020715734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb680089800 con 0x7fb6940d1230 2024-09-13T17:11:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.966+0000 7fb692a00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 0x7fb67406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.966+0000 7fb69cc27740 1 Processor -- start 2024-09-13T17:11:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb69cc27740 1 -- start start 2024-09-13T17:11:36.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb692a00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 0x7fb67406d0c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb684000f60 tx=0x7fb684009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd1f0d0 0x562e3fd6ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd6f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.967+0000 7fb69cc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc2ce60 con 0x562e3fd21910 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc8af10 con 0x562e3fd48dc0 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc886a0 con 0x562e3fd1f0d0 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53950/0 (socket says 172.21.15.28:53950) 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 -- 172.21.15.28:0/3967311593 learned_addr learned my addr 172.21.15.28:0/3967311593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692000640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd1f0d0 0x562e3fd6ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb693400640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd6f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 -- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd1f0d0 msgr2=0x562e3fd6ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd1f0d0 0x562e3fd6ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 -- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 msgr2=0x562e3fd6f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd6f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb692a00640 1 -- 172.21.15.28:0/3967311593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e3fc25bf0 con 0x562e3fd21910 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.968+0000 7fb693400640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd6f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb692a00640 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb684001ea0 tx=0x7fb684012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc88650ccb3af43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb67fe00640 1 -- 172.21.15.28:0/3967311593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6840149d0 con 0x562e3fd21910 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb67fe00640 1 -- 172.21.15.28:0/3967311593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684014b30 con 0x562e3fd21910 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb67fe00640 1 -- 172.21.15.28:0/3967311593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684014ea0 con 0x562e3fd21910 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb69cc27740 1 -- 172.21.15.28:0/3967311593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 msgr2=0x562e3fd21cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.969+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fb684001ea0 tx=0x7fb684012000 comp rx=0 tx=0).stop 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 -- 172.21.15.28:0/3967311593 shutdown_connections 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd6f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd1f0d0 0x562e3fd6ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3967311593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd21cf0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 -- 172.21.15.28:0/3967311593 >> 172.21.15.28:0/3967311593 conn(0x562e3fd1e040 msgr2=0x562e3fd664f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 -- 172.21.15.28:0/3967311593 shutdown_connections 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 -- 172.21.15.28:0/3967311593 wait complete. 2024-09-13T17:11:36.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.970+0000 7fb69cc27740 1 -- start start 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.971+0000 7fb69cc27740 1 Processor -- start 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd48dc0 0x562e3fd4c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 0x562e3fd4a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc83f90 con 0x562e3fd21910 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc423c0 con 0x562e3fd4a330 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc42060 con 0x562e3fd48dc0 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb69cc27740 10 client.0 _ll_register_callbacks cb 0x562e3fc73d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53954/0 (socket says 172.21.15.28:53954) 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 -- 172.21.15.28:0/2978925701 learned_addr learned my addr 172.21.15.28:0/2978925701 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692000640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd48dc0 0x562e3fd4c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb693400640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 0x562e3fd4a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 -- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd48dc0 msgr2=0x562e3fd4c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd48dc0 0x562e3fd4c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 -- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 msgr2=0x562e3fd4a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 0x562e3fd4a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb692a00640 1 -- 172.21.15.28:0/2978925701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6940d19b0 con 0x562e3fd21910 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.972+0000 7fb693400640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 0x562e3fd4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb692a00640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb684008fd0 tx=0x7fb684001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5f69d50abcd68ca server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6840149d0 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb68401c040 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e3fc76ab0 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb684020c50 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb69cc27740 10 client.6123 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x562e3fc82610 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562e3fc76ab0 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.973+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb684012070 con 0x562e3fd21910 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 10 client.6123 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 10 client.6123 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 20 client.6123 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd796d0 0x562e3fd79ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 4 client.6123 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e3f9778e0 con 0x562e3fd796d0 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb692000640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd796d0 0x562e3fd79ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb692000640 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd796d0 0x562e3fd79ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7427270f1c4fb6b9 server_cookie=3dbd0a8b150b3174 in_seq=0 out_seq=0 2024-09-13T17:11:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.974+0000 7fb67cc00640 10 client.6123 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb67cc00640 1 -- 172.21.15.28:0/2978925701 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562e3f9778e0 con 0x562e3fd796d0 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb67cc00640 10 client.6123 handle_command_reply: tid=0 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 client.6123 shutdown 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 msgr2=0x562e3fd47d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb684008fd0 tx=0x7fb684001080 comp rx=0 tx=0).stop 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 shutdown_connections 2024-09-13T17:11:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd4a330 0x562e3fd4a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd48dc0 0x562e3fd4c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd796d0 0x562e3fd79ae0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 --2- 172.21.15.28:0/2978925701 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd21910 0x562e3fd47d40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 >> 172.21.15.28:0/2978925701 conn(0x562e3fd1f0d0 msgr2=0x562e3fd67bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb67cc00640 0 client.6123 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 shutdown_connections 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.975+0000 7fb69cc27740 1 -- 172.21.15.28:0/2978925701 wait complete. 2024-09-13T17:11:36.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.976+0000 7fb69cc27740 20 client.6123 trim_cache size 0 max 16384 2024-09-13T17:11:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.990+0000 7fb69cc27740 1 Processor -- start 2024-09-13T17:11:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.990+0000 7fb69cc27740 1 -- start start 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd21910 0x562e3fd67700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd4a330 0x562e3fd62310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3f9eb440 con 0x562e3fd48dc0 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fd17b30 con 0x562e3fd21910 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3fc8a090 con 0x562e3fd4a330 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.991+0000 7fb692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53956/0 (socket says 172.21.15.28:53956) 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 -- 172.21.15.28:0/963165671 learned_addr learned my addr 172.21.15.28:0/963165671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692a00640 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd21910 0x562e3fd67700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb693400640 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd4a330 0x562e3fd62310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 -- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd4a330 msgr2=0x562e3fd62310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd4a330 0x562e3fd62310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 -- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd21910 msgr2=0x562e3fd67700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd21910 0x562e3fd67700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 -- 172.21.15.28:0/963165671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e3fc24430 con 0x562e3fd48dc0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb692000640 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb680054bb0 tx=0x7fb68005b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b33593c639432d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb688c00640 1 -- 172.21.15.28:0/963165671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6800568b0 con 0x562e3fd48dc0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb688c00640 1 -- 172.21.15.28:0/963165671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb680056a50 con 0x562e3fd48dc0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.992+0000 7fb688c00640 1 -- 172.21.15.28:0/963165671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb680054dc0 con 0x562e3fd48dc0 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 -- 172.21.15.28:0/963165671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 msgr2=0x562e3fd67c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:36.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fb680054bb0 tx=0x7fb68005b040 comp rx=0 tx=0).stop 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 -- 172.21.15.28:0/963165671 shutdown_connections 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd4a330 0x562e3fd62310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd48dc0 0x562e3fd67c40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 --2- 172.21.15.28:0/963165671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd21910 0x562e3fd67700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 -- 172.21.15.28:0/963165671 >> 172.21.15.28:0/963165671 conn(0x562e3fd1f0d0 msgr2=0x562e3fd662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 -- 172.21.15.28:0/963165671 shutdown_connections 2024-09-13T17:11:36.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.993+0000 7fb69cc27740 1 -- 172.21.15.28:0/963165671 wait complete. 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.994+0000 7fb69cc27740 1 -- start start 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.994+0000 7fb69cc27740 1 Processor -- start 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd21910 0x562e3fdd3af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd7ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fd7e410 con 0x562e3fd4a330 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3fd7e840 con 0x562e3fd48dc0 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3fd87460 con 0x562e3fd21910 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb69cc27740 10 client.0 _ll_register_callbacks cb 0x7fb684017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb693400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb693400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53968/0 (socket says 172.21.15.28:53968) 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb693400640 1 -- 172.21.15.28:0/3927899085 learned_addr learned my addr 172.21.15.28:0/3927899085 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb692a00640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd21910 0x562e3fdd3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb692000640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd7ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb693400640 1 -- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd21910 msgr2=0x562e3fdd3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.995+0000 7fb693400640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd21910 0x562e3fdd3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb693400640 1 -- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 msgr2=0x562e3fd7ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb693400640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd7ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb693400640 1 -- 172.21.15.28:0/3927899085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6940c83a0 con 0x562e3fd4a330 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb693400640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0093e0 tx=0x7fb68c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8547053bde3749f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb67cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c041040 con 0x562e3fd4a330 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb68c031600 con 0x562e3fd4a330 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb684014b30 con 0x562e3fd4a330 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb68c0317c0 con 0x562e3fd4a330 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb69cc27740 10 client.6129 fetch_fsmap learned FSMap version 37 2024-09-13T17:11:37.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.996+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) v3 -- 0x562e3fc49a00 con 0x562e3fd4a330 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684014b30 con 0x562e3fd4a330 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 37) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb68c031920 con 0x562e3fd4a330 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 10 client.6129 fetch_fsmap finished waiting for FSMap version 37 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 10 client.6129 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 20 client.6129 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd49e50 0x562e3fd674f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 4 client.6129 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_1"}] 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_test_path/ceph-qa-suite/new_file_1"}) v1 -- 0x562e3f988be0 con 0x562e3fd49e50 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.997+0000 7fb692a00640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd49e50 0x562e3fd674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb692a00640 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd49e50 0x562e3fd674f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35046a3895a25783 server_cookie=416a36a3c2aabf75 in_seq=0 out_seq=0 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb67cc00640 10 client.6129 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb67cc00640 1 -- 172.21.15.28:0/3927899085 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562e3f988be0 con 0x562e3fd49e50 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb67cc00640 10 client.6129 handle_command_reply: tid=0 2024-09-13T17:11:37.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 client.6129 shutdown 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 msgr2=0x562e3fd49910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fb68c0093e0 tx=0x7fb68c02c000 comp rx=0 tx=0).stop 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 shutdown_connections 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x562e3fd49e50 0x562e3fd674f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e3fd4a330 0x562e3fd49910 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e3fd48dc0 0x562e3fd7ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 --2- 172.21.15.28:0/3927899085 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e3fd21910 0x562e3fdd3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb67cc00640 0 client.6129 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.998+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 >> 172.21.15.28:0/3927899085 conn(0x562e3fd1f0d0 msgr2=0x562e3fd664b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 shutdown_connections 2024-09-13T17:11:37.004 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:11:36.999+0000 7f203e200640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000096c [2,head] /scrub_test_path/ceph-qa-suite/new_file_1 auth v698 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={5566=0-4194304@1} caps={5566=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@3},l=5566 | request=0 lock=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55b5cf7ad080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(7)0x1000000096c:[<0x10000000001/new_file_1 v698>,<0x10000000000/ceph-qa-suite v70>,<0x1/scrub_test_path v23>]//[]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61} 2024-09-13T17:11:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb69cc27740 1 -- 172.21.15.28:0/3927899085 wait complete. 2024-09-13T17:11:37.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb69cc27740 20 client.6129 trim_cache size 0 max 16384 2024-09-13T17:11:37.005 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:11:37.005 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "dbdfdf6f-bea7-4266-be17-6430bfabc9c1", 2024-09-13T17:11:37.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:11:37.006 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:37.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 msgr2=0x7fb67406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 0x7fb67406d0c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb684000f60 tx=0x7fb684009040 comp rx=0 tx=0).stop 2024-09-13T17:11:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 msgr2=0x7fb6941e5c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fb68000d7d0 tx=0x7fb68000dc90 comp rx=0 tx=0).stop 2024-09-13T17:11:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 shutdown_connections 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb67406ac00 0x7fb67406d0c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6941e6160 0x7fb6940c6440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6940d1230 0x7fb6941e5c20 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 --2- 172.21.15.28:0/1020715734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6940cf980 0x7fb6940c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 >> 172.21.15.28:0/1020715734 conn(0x7fb694070ad0 msgr2=0x7fb6940b68f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:36.999+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 shutdown_connections 2024-09-13T17:11:37.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.000+0000 7fb693e00640 1 -- 172.21.15.28:0/1020715734 wait complete. 2024-09-13T17:11:37.020 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_test_path/ceph-qa-suite/new_file_1' returned '{'return_code': 0, 'scrub_tag': 'dbdfdf6f-bea7-4266-be17-6430bfabc9c1', 'mode': 'asynchronous'}' 2024-09-13T17:11:37.020 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T17:11:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.177+0000 7fb5e3600640 1 Processor -- start 2024-09-13T17:11:37.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.177+0000 7fb5e3600640 1 -- start start 2024-09-13T17:11:37.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.178+0000 7fb5e3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5dc15a780 0x7fb5dc070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.178+0000 7fb5e3600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc06c660 con 0x7fb5dc15ab60 2024-09-13T17:11:37.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.178+0000 7fb5e3600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5dc06c8b0 con 0x7fb5dc06c250 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.178+0000 7fb5e3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5dc06cb90 con 0x7fb5dc15a780 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5dc15a780 0x7fb5dc070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5dc15a780 0x7fb5dc070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53972/0 (socket says 172.21.15.28:53972) 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 -- 172.21.15.28:0/770799351 learned_addr learned my addr 172.21.15.28:0/770799351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5dbe00640 1 --1- 172.21.15.28:0/770799351 >> v1:172.21.15.28:6789/0 conn(0x7fb5dc15ab60 0x7fb5dc06bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33860/0 (socket says 172.21.15.28:33860) 2024-09-13T17:11:37.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 -- 172.21.15.28:0/770799351 >> v1:172.21.15.189:6789/0 conn(0x7fb5dc06c250 legacy=0x7fb5dc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 -- 172.21.15.28:0/770799351 >> v1:172.21.15.28:6789/0 conn(0x7fb5dc15ab60 legacy=0x7fb5dc06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.179+0000 7fb5e0c00640 1 -- 172.21.15.28:0/770799351 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc0cc980 con 0x7fb5dc15a780 2024-09-13T17:11:37.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.180+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/770799351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc070d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009870 tx=0x7fb5d002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9754ae9693d3e1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.180+0000 7fb5db400640 1 -- 172.21.15.28:0/770799351 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0032440 con 0x7fb5dc15a780 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.180+0000 7fb5db400640 1 -- 172.21.15.28:0/770799351 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d00325a0 con 0x7fb5dc15a780 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.180+0000 7fb5db400640 1 -- 172.21.15.28:0/770799351 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0032930 con 0x7fb5dc15a780 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 -- 172.21.15.28:0/770799351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 msgr2=0x7fb5dc070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 --2- 172.21.15.28:0/770799351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc070d80 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009870 tx=0x7fb5d002a2a0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 -- 172.21.15.28:0/770799351 shutdown_connections 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 --2- 172.21.15.28:0/770799351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc070d80 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 -- 172.21.15.28:0/770799351 >> 172.21.15.28:0/770799351 conn(0x7fb5dc0b0270 msgr2=0x7fb5dc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 -- 172.21.15.28:0/770799351 shutdown_connections 2024-09-13T17:11:37.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.181+0000 7fb5e3600640 1 -- 172.21.15.28:0/770799351 wait complete. 2024-09-13T17:11:37.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.182+0000 7fb5e3600640 1 Processor -- start 2024-09-13T17:11:37.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.183+0000 7fb5e3600640 1 -- start start 2024-09-13T17:11:37.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.183+0000 7fb5e3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5dc06c250 0x7fb5dc1549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.183+0000 7fb5e3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc15ab60 0x7fb5dc0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc0b1a80 con 0x7fb5dc15a780 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc14f230 con 0x7fb5dc15ab60 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc14f510 con 0x7fb5dc06c250 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53980/0 (socket says 172.21.15.28:53980) 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 -- 172.21.15.28:0/819486208 learned_addr learned my addr 172.21.15.28:0/819486208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5dc06c250 0x7fb5dc1549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5e1600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc15ab60 0x7fb5dc0acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 -- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5dc06c250 msgr2=0x7fb5dc1549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5dc06c250 0x7fb5dc1549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 -- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc15ab60 msgr2=0x7fb5dc0acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc15ab60 0x7fb5dc0acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.184+0000 7fb5dbe00640 1 -- 172.21.15.28:0/819486208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d00094d0 con 0x7fb5dc15a780 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800c3e0 tx=0x7fb5c800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5d8c00640 1 -- 172.21.15.28:0/819486208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80074e0 con 0x7fb5dc15a780 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5d8c00640 1 -- 172.21.15.28:0/819486208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c8007640 con 0x7fb5dc15a780 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc14f7f0 con 0x7fb5dc15a780 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5d8c00640 1 -- 172.21.15.28:0/819486208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80049e0 con 0x7fb5dc15a780 2024-09-13T17:11:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.185+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc154ee0 con 0x7fb5dc15a780 2024-09-13T17:11:37.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.187+0000 7fb5d8c00640 1 -- 172.21.15.28:0/819486208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb5c8004b40 con 0x7fb5dc15a780 2024-09-13T17:11:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.189+0000 7fb5d8c00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 0x7fb5bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.189+0000 7fb5d8c00640 1 -- 172.21.15.28:0/819486208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5c8088c20 con 0x7fb5dc15a780 2024-09-13T17:11:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.189+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 0x7fb5bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.189+0000 7fb5e643e740 1 Processor -- start 2024-09-13T17:11:37.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.189+0000 7fb5e643e740 1 -- start start 2024-09-13T17:11:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 0x7fb5bc06d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0034000 tx=0x7fb5d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 0x5572cb8ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 0x5572cb8ad670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb7666a0 con 0x5572cb85fc60 2024-09-13T17:11:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb7c9cc0 con 0x5572cb887110 2024-09-13T17:11:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572cb7c7450 con 0x5572cb85d420 2024-09-13T17:11:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53996/0 (socket says 172.21.15.28:53996) 2024-09-13T17:11:37.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.190+0000 7fb5e0c00640 1 -- 172.21.15.28:0/1160232745 learned_addr learned my addr 172.21.15.28:0/1160232745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e1600640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 0x5572cb8ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 0x5572cb8ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 -- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 msgr2=0x5572cb8ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 0x5572cb8ad130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 -- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 msgr2=0x5572cb8ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 0x5572cb8ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 -- 172.21.15.28:0/1160232745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572cb7932e0 con 0x5572cb85fc60 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e1600640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 0x5572cb8ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 0x5572cb8ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb5d002a2e0 tx=0x7fb5d002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c632603b91d19f7d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5cd600640 1 -- 172.21.15.28:0/1160232745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d003c780 con 0x5572cb85fc60 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5cd600640 1 -- 172.21.15.28:0/1160232745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d003c8e0 con 0x5572cb85fc60 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.191+0000 7fb5cd600640 1 -- 172.21.15.28:0/1160232745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d003cc50 con 0x5572cb85fc60 2024-09-13T17:11:37.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 -- 172.21.15.28:0/1160232745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 msgr2=0x5572cb860040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fb5d002a2e0 tx=0x7fb5d002c000 comp rx=0 tx=0).stop 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 -- 172.21.15.28:0/1160232745 shutdown_connections 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb887110 0x5572cb8ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85d420 0x5572cb8ad130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 --2- 172.21.15.28:0/1160232745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb860040 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 -- 172.21.15.28:0/1160232745 >> 172.21.15.28:0/1160232745 conn(0x5572cb85c390 msgr2=0x5572cb8a4840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 -- 172.21.15.28:0/1160232745 shutdown_connections 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.192+0000 7fb5e643e740 1 -- 172.21.15.28:0/1160232745 wait complete. 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.193+0000 7fb5e643e740 1 -- start start 2024-09-13T17:11:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.193+0000 7fb5e643e740 1 Processor -- start 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb8ab3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb8ad0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb7c2d40 con 0x5572cb887110 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb764280 con 0x5572cb88abf0 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572cb763f20 con 0x5572cb85fc60 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e643e740 10 client.0 _ll_register_callbacks cb 0x5572cb7bc3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53998/0 (socket says 172.21.15.28:53998) 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3972944538 learned_addr learned my addr 172.21.15.28:0/3972944538 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb8ab3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e1600640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb8ad0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 msgr2=0x5572cb8ab3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb8ab3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 msgr2=0x5572cb8ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb8ad0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3972944538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc0cc980 con 0x5572cb887110 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.194+0000 7fb5e1600640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb8ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb5c805f970 tx=0x7fb5c805b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e5753ffd54f5df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c806f070 con 0x5572cb887110 2024-09-13T17:11:37.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5c8057fc0 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572cb4b64e0 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5c8055d60 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5e643e740 10 client.6144 fetch_fsmap learned FSMap version 38 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) v3 -- 0x5572cb7c13c0 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5572cb4b64e0 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb5c8057050 con 0x5572cb887110 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5e643e740 10 client.6144 fetch_fsmap finished waiting for FSMap version 38 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.195+0000 7fb5e643e740 10 client.6144 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e643e740 20 client.6144 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8866f0 0x5572cb888b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e643e740 4 client.6144 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572cb4c77e0 con 0x5572cb8866f0 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8866f0 0x5572cb888b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8866f0 0x5572cb888b30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d3c9c84179e8273 server_cookie=35cd452797bdc73 in_seq=0 out_seq=0 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.196+0000 7fb5c2000640 10 client.6144 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5c2000640 1 -- 172.21.15.28:0/3972944538 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5572cb4c77e0 con 0x5572cb8866f0 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5c2000640 10 client.6144 handle_command_reply: tid=0 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 client.6144 shutdown 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 msgr2=0x5572cb885e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fb5c805f970 tx=0x7fb5c805b9a0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 shutdown_connections 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb8ad0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8866f0 0x5572cb888b30 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb885e60 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3972944538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb8ab3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5c2000640 0 client.6144 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 >> 172.21.15.28:0/3972944538 conn(0x5572cb85d420 msgr2=0x5572cb8a8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 shutdown_connections 2024-09-13T17:11:37.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.197+0000 7fb5e643e740 1 -- 172.21.15.28:0/3972944538 wait complete. 2024-09-13T17:11:37.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.198+0000 7fb5e643e740 20 client.6144 trim_cache size 0 max 16384 2024-09-13T17:11:37.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.212+0000 7fb5e643e740 1 Processor -- start 2024-09-13T17:11:37.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 -- start start 2024-09-13T17:11:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 0x5572cb88b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb52a040 con 0x5572cb85fc60 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb4fcb00 con 0x5572cb88abf0 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.213+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572cb8a3990 con 0x5572cb887110 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54014/0 (socket says 172.21.15.28:54014) 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 -- 172.21.15.28:0/3074491060 learned_addr learned my addr 172.21.15.28:0/3074491060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 0x5572cb88b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e1600640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 -- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 msgr2=0x5572cb88b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 0x5572cb88b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 -- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 msgr2=0x5572cb885a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 -- 172.21.15.28:0/3074491060 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572cb763610 con 0x5572cb85fc60 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 0x5572cb88b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.214+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0035040 tx=0x7fb5d0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cb814847ff638 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5ce000640 1 -- 172.21.15.28:0/3074491060 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0057040 con 0x5572cb85fc60 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5ce000640 1 -- 172.21.15.28:0/3074491060 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d003cc50 con 0x5572cb85fc60 2024-09-13T17:11:37.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5ce000640 1 -- 172.21.15.28:0/3074491060 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d005c410 con 0x5572cb85fc60 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 -- 172.21.15.28:0/3074491060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 msgr2=0x5572cb8aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0035040 tx=0x7fb5d0009240 comp rx=0 tx=0).stop 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 -- 172.21.15.28:0/3074491060 shutdown_connections 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb887110 0x5572cb88b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3074491060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb85fc60 0x5572cb8aa4f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 -- 172.21.15.28:0/3074491060 >> 172.21.15.28:0/3074491060 conn(0x5572cb85d420 msgr2=0x5572cb8a1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 -- 172.21.15.28:0/3074491060 shutdown_connections 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.215+0000 7fb5e643e740 1 -- 172.21.15.28:0/3074491060 wait complete. 2024-09-13T17:11:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.216+0000 7fb5e643e740 1 -- start start 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.216+0000 7fb5e643e740 1 Processor -- start 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb912960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb90f560 con 0x5572cb887110 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572cb9124d0 con 0x5572cb88abf0 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572cb8a3f90 con 0x5572cb85fc60 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5e643e740 10 client.0 _ll_register_callbacks cb 0x7fb5c8000f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54024/0 (socket says 172.21.15.28:54024) 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.217+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3672697984 learned_addr learned my addr 172.21.15.28:0/3672697984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5e1600640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb912960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 msgr2=0x5572cb912960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb912960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 msgr2=0x5572cb885930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 -- 172.21.15.28:0/3672697984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5572cb7932e0 con 0x5572cb887110 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5dbe00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800cda0 tx=0x7fb5c801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09fe889c23b4cca server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5c2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c806f070 con 0x5572cb887110 2024-09-13T17:11:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5c8077eb0 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5d003c8e0 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5c800e040 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5e643e740 10 client.6150 fetch_fsmap learned FSMap version 38 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.218+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=2+}) v3 -- 0x5572cb788b20 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c8062080 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 38) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb5c807d020 con 0x5572cb887110 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 10 client.6150 fetch_fsmap finished waiting for FSMap version 38 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 10 client.6150 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 20 client.6150 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8807c0 0x5572cb885e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 4 client.6150 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5572cb51ae30 con 0x5572cb8807c0 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.219+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8807c0 0x5572cb885e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e0c00640 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8807c0 0x5572cb885e70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8c92e04bcb75bb7 server_cookie=a23730d621f45371 in_seq=0 out_seq=0 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5c2000640 10 client.6150 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5c2000640 1 -- 172.21.15.28:0/3672697984 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+903 (crc 0 0 0) 0x5572cb51ae30 con 0x5572cb8807c0 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5c2000640 10 client.6150 handle_command_reply: tid=0 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 client.6150 shutdown 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 msgr2=0x5572cb8f7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800cda0 tx=0x7fb5c801e040 comp rx=0 tx=0).stop 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 shutdown_connections 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572cb88abf0 0x5572cb885930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x5572cb8807c0 0x5572cb885e70 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572cb887110 0x5572cb8f7920 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 --2- 172.21.15.28:0/3672697984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572cb85fc60 0x5572cb912960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5c2000640 0 client.6150 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 >> 172.21.15.28:0/3672697984 conn(0x5572cb85d420 msgr2=0x5572cb8a2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 shutdown_connections 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 1 -- 172.21.15.28:0/3672697984 wait complete. 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.220+0000 7fb5e643e740 20 client.6150 trim_cache size 0 max 16384 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "02bac7df-bf12-46e8-b9bf-f7c6b76fcc05": { 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/suites", 2024-09-13T17:11:37.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "02bac7df-bf12-46e8-b9bf-f7c6b76fcc05", 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "" 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "93b417fd-2902-46cf-8244-e4e94ed67653": { 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "93b417fd-2902-46cf-8244-e4e94ed67653", 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "" 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "dbdfdf6f-bea7-4266-be17-6430bfabc9c1": { 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_1", 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "dbdfdf6f-bea7-4266-be17-6430bfabc9c1", 2024-09-13T17:11:37.229 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "" 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "e79607ce-3d21-4ce8-8a85-7f54fe34e373": { 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/suites/fs/verify/validater/valgrind.yaml", 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e79607ce-3d21-4ce8-8a85-7f54fe34e373", 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "" 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:37.230 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 msgr2=0x7fb5bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 0x7fb5bc06d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0034000 tx=0x7fb5d002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 msgr2=0x7fb5dc0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800c3e0 tx=0x7fb5c800c8a0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 shutdown_connections 2024-09-13T17:11:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5bc06ac00 0x7fb5bc06d0c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5dc15ab60 0x7fb5dc0acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5dc15a780 0x7fb5dc0ac850 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 --2- 172.21.15.28:0/819486208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5dc06c250 0x7fb5dc1549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 >> 172.21.15.28:0/819486208 conn(0x7fb5dc0b0270 msgr2=0x7fb5dc157800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 shutdown_connections 2024-09-13T17:11:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:37.221+0000 7fb5e3600640 1 -- 172.21.15.28:0/819486208 wait complete. 2024-09-13T17:11:39.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:11:39.435 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:11:41.069 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:11:41.067+0000 7f58f6400640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-13T17:11:42.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T17:11:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.405+0000 7fc3b0e00640 1 Processor -- start 2024-09-13T17:11:42.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.405+0000 7fc3b0e00640 1 -- start start 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.406+0000 7fc3b0e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac15a780 0x7fc3ac070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.406+0000 7fc3b0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3ac06c250 0x7fc3ac0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3b0e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3ac06c8b0 con 0x7fc3ac15ab60 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3b0e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3ac06c6a0 con 0x7fc3ac06c250 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3b0e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac06cb90 con 0x7fc3ac15a780 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3ac06c250 0x7fc3ac0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3ac06c250 0x7fc3ac0712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54032/0 (socket says 172.21.15.28:54032) 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 -- 172.21.15.28:0/1573200323 learned_addr learned my addr 172.21.15.28:0/1573200323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac15a780 0x7fc3ac070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 -- 172.21.15.28:0/1573200323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac15a780 msgr2=0x7fc3ac070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac15a780 0x7fc3ac070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 -- 172.21.15.28:0/1573200323 >> v1:172.21.15.189:6789/0 conn(0x7fc3ac15ab60 legacy=0x7fc3ac06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.407+0000 7fc3aa800640 1 -- 172.21.15.28:0/1573200323 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0cc980 con 0x7fc3ac06c250 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.408+0000 7fc3aa800640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac06c250 0x7fc3ac0712c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8994eb76798c0c6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3a9400640 1 -- 172.21.15.28:0/1573200323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4031600 con 0x7fc3ac06c250 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3a9400640 1 -- 172.21.15.28:0/1573200323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a402d050 con 0x7fc3ac06c250 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3a9400640 1 -- 172.21.15.28:0/1573200323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4031980 con 0x7fc3ac06c250 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 -- 172.21.15.28:0/1573200323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac06c250 msgr2=0x7fc3ac0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac06c250 0x7fc3ac0712c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a402a710 comp rx=0 tx=0).stop 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 -- 172.21.15.28:0/1573200323 shutdown_connections 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac06c250 0x7fc3ac0712c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/1573200323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac15a780 0x7fc3ac070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 -- 172.21.15.28:0/1573200323 >> 172.21.15.28:0/1573200323 conn(0x7fc3ac0b0270 msgr2=0x7fc3ac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 -- 172.21.15.28:0/1573200323 shutdown_connections 2024-09-13T17:11:42.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.409+0000 7fc3b0e00640 1 -- 172.21.15.28:0/1573200323 wait complete. 2024-09-13T17:11:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.411+0000 7fc3b0e00640 1 Processor -- start 2024-09-13T17:11:42.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.411+0000 7fc3b0e00640 1 -- start start 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac06c250 0x7fc3ac0a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac15ab60 0x7fc3ac0ce380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac150a70 con 0x7fc3ac15a780 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac14a470 con 0x7fc3ac15ab60 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3b0e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac0a84e0 con 0x7fc3ac06c250 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54034/0 (socket says 172.21.15.28:54034) 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3a3e00640 1 -- 172.21.15.28:0/3268257466 learned_addr learned my addr 172.21.15.28:0/3268257466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac06c250 0x7fc3ac0a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.412+0000 7fc3aa800640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac15ab60 0x7fc3ac0ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 -- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac06c250 msgr2=0x7fc3ac0a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac06c250 0x7fc3ac0a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 -- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac15ab60 msgr2=0x7fc3ac0ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac15ab60 0x7fc3ac0ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 -- 172.21.15.28:0/3268257466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4009b30 con 0x7fc3ac15a780 2024-09-13T17:11:42.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c3c0 tx=0x7fc39400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.413+0000 7fc3a2000640 1 -- 172.21.15.28:0/3268257466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3940052d0 con 0x7fc3ac15a780 2024-09-13T17:11:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.414+0000 7fc3a2000640 1 -- 172.21.15.28:0/3268257466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394005450 con 0x7fc3ac15a780 2024-09-13T17:11:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.414+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac0a87c0 con 0x7fc3ac15a780 2024-09-13T17:11:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.414+0000 7fc3a2000640 1 -- 172.21.15.28:0/3268257466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394003830 con 0x7fc3ac15a780 2024-09-13T17:11:42.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.414+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac0a8d80 con 0x7fc3ac15a780 2024-09-13T17:11:42.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.417+0000 7fc3a2000640 1 -- 172.21.15.28:0/3268257466 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc394003990 con 0x7fc3ac15a780 2024-09-13T17:11:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.418+0000 7fc3a2000640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 0x7fc38806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.418+0000 7fc3a2000640 1 -- 172.21.15.28:0/3268257466 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc394088780 con 0x7fc3ac15a780 2024-09-13T17:11:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.418+0000 7fc3b3c68740 1 Processor -- start 2024-09-13T17:11:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.418+0000 7fc3b3c68740 1 -- start start 2024-09-13T17:11:42.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.418+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 0x7fc38806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757bdf420 0x55f757c2f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 0x55f757c2f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757ae86a0 con 0x55f757be1c60 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757b4bcc0 con 0x55f757c09110 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f757b49450 con 0x55f757bdf420 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 0x7fc38806d0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006fd0 tx=0x7fc39c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54040/0 (socket says 172.21.15.28:54040) 2024-09-13T17:11:42.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3a3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757bdf420 0x55f757c2f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3a9e00640 1 -- 172.21.15.28:0/397779185 learned_addr learned my addr 172.21.15.28:0/397779185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.419+0000 7fc3aa800640 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 0x55f757c2f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 -- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757bdf420 msgr2=0x55f757c2f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757bdf420 0x55f757c2f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 -- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 msgr2=0x55f757c2f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 0x55f757c2f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 -- 172.21.15.28:0/397779185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f757b152e0 con 0x55f757be1c60 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3aa800640 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 0x55f757c2f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc39c010b40 tx=0x7fc39c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f9940e88e9e46f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc39aa00640 1 -- 172.21.15.28:0/397779185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c016cd0 con 0x55f757be1c60 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc39aa00640 1 -- 172.21.15.28:0/397779185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c01a070 con 0x55f757be1c60 2024-09-13T17:11:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.420+0000 7fc39aa00640 1 -- 172.21.15.28:0/397779185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c021640 con 0x55f757be1c60 2024-09-13T17:11:42.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 -- 172.21.15.28:0/397779185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 msgr2=0x55f757be2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc39c010b40 tx=0x7fc39c013000 comp rx=0 tx=0).stop 2024-09-13T17:11:42.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 -- 172.21.15.28:0/397779185 shutdown_connections 2024-09-13T17:11:42.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c09110 0x55f757c2f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757bdf420 0x55f757c2f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/397779185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757be2040 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 -- 172.21.15.28:0/397779185 >> 172.21.15.28:0/397779185 conn(0x55f757bde390 msgr2=0x55f757c26840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 -- 172.21.15.28:0/397779185 shutdown_connections 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.421+0000 7fc3b3c68740 1 -- 172.21.15.28:0/397779185 wait complete. 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.422+0000 7fc3b3c68740 1 -- start start 2024-09-13T17:11:42.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.422+0000 7fc3b3c68740 1 Processor -- start 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c09110 0x55f757c0b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757b44d40 con 0x55f757be1c60 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757ae6280 con 0x55f757c081b0 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f757ae5f20 con 0x55f757c09110 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3b3c68740 10 client.0 _ll_register_callbacks cb 0x55f757b3e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54048/0 (socket says 172.21.15.28:54048) 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.423+0000 7fc3a9e00640 1 -- 172.21.15.28:0/3100869388 learned_addr learned my addr 172.21.15.28:0/3100869388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c09110 0x55f757c0b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3aa800640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 -- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c09110 msgr2=0x55f757c0b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c09110 0x55f757c0b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 -- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 msgr2=0x55f757c0b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 -- 172.21.15.28:0/3100869388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc39c00e040 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc39c010620 tx=0x7fc39c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78d817bd82d5fd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc38f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c004240 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc39c0043a0 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.424+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7578384e0 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c004560 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 10 client.6162 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55f757b433c0 con 0x55f757be1c60 2024-09-13T17:11:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f7578384e0 con 0x55f757be1c60 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc39c0217f0 con 0x55f757be1c60 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 10 client.6162 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 10 client.6162 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 20 client.6162 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757bf4fe0 0x55f757bf53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 4 client.6162 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7578497e0 con 0x55f757bf4fe0 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.425+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757bf4fe0 0x55f757bf53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.426+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757bf4fe0 0x55f757bf53c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90774ee6cfdf6381 server_cookie=3c030b61e7b5312f in_seq=0 out_seq=0 2024-09-13T17:11:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.426+0000 7fc38f400640 10 client.6162 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.426+0000 7fc38f400640 1 -- 172.21.15.28:0/3100869388 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55f7578497e0 con 0x55f757bf4fe0 2024-09-13T17:11:42.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.426+0000 7fc38f400640 10 client.6162 handle_command_reply: tid=0 2024-09-13T17:11:42.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.426+0000 7fc3b3c68740 1 client.6162 shutdown 2024-09-13T17:11:42.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 msgr2=0x55f757c23e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc39c010620 tx=0x7fc39c004730 comp rx=0 tx=0).stop 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 shutdown_connections 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c09110 0x55f757c0b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757bf4fe0 0x55f757bf53c0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3100869388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757be1c60 0x55f757c23e20 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 >> 172.21.15.28:0/3100869388 conn(0x55f757bdf420 msgr2=0x55f757c25ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc38f400640 0 client.6162 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 shutdown_connections 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3100869388 wait complete. 2024-09-13T17:11:42.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.427+0000 7fc3b3c68740 20 client.6162 trim_cache size 0 max 16384 2024-09-13T17:11:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.442+0000 7fc3b3c68740 1 Processor -- start 2024-09-13T17:11:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.442+0000 7fc3b3c68740 1 -- start start 2024-09-13T17:11:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757be1c60 0x55f757be12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 0x55f757c0b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7578ac040 con 0x55f757c09110 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757c2de50 con 0x55f757be1c60 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f757bdf9d0 con 0x55f757c081b0 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54056/0 (socket says 172.21.15.28:54056) 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3aa800640 1 -- 172.21.15.28:0/3683403142 learned_addr learned my addr 172.21.15.28:0/3683403142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757be1c60 0x55f757be12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.443+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 0x55f757c0b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 -- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 msgr2=0x55f757c0b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 0x55f757c0b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 -- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757be1c60 msgr2=0x55f757be12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757be1c60 0x55f757be12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 -- 172.21.15.28:0/3683403142 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f757ae5610 con 0x55f757c09110 2024-09-13T17:11:42.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 0x55f757c0b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc3aa800640 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee770bfc598f733b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc39b400640 1 -- 172.21.15.28:0/3683403142 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4041070 con 0x55f757c09110 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc39b400640 1 -- 172.21.15.28:0/3683403142 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a4031750 con 0x55f757c09110 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.444+0000 7fc39b400640 1 -- 172.21.15.28:0/3683403142 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4031ae0 con 0x55f757c09110 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3683403142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 msgr2=0x55f757c28de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3683403142 shutdown_connections 2024-09-13T17:11:42.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c28de0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757c081b0 0x55f757c0b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/3683403142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757be1c60 0x55f757be12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3683403142 >> 172.21.15.28:0/3683403142 conn(0x55f757bdf420 msgr2=0x55f757c095e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3683403142 shutdown_connections 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.445+0000 7fc3b3c68740 1 -- 172.21.15.28:0/3683403142 wait complete. 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.446+0000 7fc3b3c68740 1 -- start start 2024-09-13T17:11:42.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.446+0000 7fc3b3c68740 1 Processor -- start 2024-09-13T17:11:42.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757be1c60 0x55f757c09e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757c07980 con 0x55f757c09110 2024-09-13T17:11:42.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f757c3f660 con 0x55f757c081b0 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f757c29270 con 0x55f757be1c60 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3b3c68740 10 client.0 _ll_register_callbacks cb 0x7fc3a4009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54064/0 (socket says 172.21.15.28:54064) 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 -- 172.21.15.28:0/1231472750 learned_addr learned my addr 172.21.15.28:0/1231472750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3a3e00640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757be1c60 0x55f757c09e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 -- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757be1c60 msgr2=0x55f757c09e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757be1c60 0x55f757c09e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 -- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 msgr2=0x55f757c0a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.447+0000 7fc3aa800640 1 -- 172.21.15.28:0/1231472750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a4031400 con 0x55f757c09110 2024-09-13T17:11:42.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc3aa800640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5536aba51ba46ab3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4031cd0 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc39c01a070 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3a4031e30 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3a4007160 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc3b3c68740 10 client.6168 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55f757b0ab20 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c01a070 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.448+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc3a4004040 con 0x55f757c09110 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 10 client.6168 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 10 client.6168 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 20 client.6168 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757c28de0 0x55f757c27cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 4 client.6168 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:11:42.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f75789ce30 con 0x55f757c28de0 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757c28de0 0x55f757c27cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc3a9e00640 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757c28de0 0x55f757c27cd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ece873b36e6a1188 server_cookie=db5890d2b9b9477a in_seq=0 out_seq=0 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc38f400640 10 client.6168 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc38f400640 1 -- 172.21.15.28:0/1231472750 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f75789ce30 con 0x55f757c28de0 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.449+0000 7fc38f400640 10 client.6168 handle_command_reply: tid=0 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 client.6168 shutdown 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 msgr2=0x55f757c27790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4009e50 tx=0x7fc3a4004ab0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 shutdown_connections 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55f757c28de0 0x55f757c27cd0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f757c09110 0x55f757c27790 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f757c081b0 0x55f757c0a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 --2- 172.21.15.28:0/1231472750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f757be1c60 0x55f757c09e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 >> 172.21.15.28:0/1231472750 conn(0x55f757bdf420 msgr2=0x55f757c48b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc38f400640 0 client.6168 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 shutdown_connections 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 1 -- 172.21.15.28:0/1231472750 wait complete. 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.450+0000 7fc3b3c68740 20 client.6168 trim_cache size 0 max 16384 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:11:42.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 msgr2=0x7fc38806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 0x7fc38806d0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc39c006fd0 tx=0x7fc39c008040 comp rx=0 tx=0).stop 2024-09-13T17:11:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 msgr2=0x7fc3ac149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c3c0 tx=0x7fc39400c880 comp rx=0 tx=0).stop 2024-09-13T17:11:42.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 shutdown_connections 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38806ac00 0x7fc38806d0c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3ac15ab60 0x7fc3ac0ce380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3ac15a780 0x7fc3ac149f30 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 --2- 172.21.15.28:0/3268257466 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3ac06c250 0x7fc3ac0a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 >> 172.21.15.28:0/3268257466 conn(0x7fc3ac0b0270 msgr2=0x7fc3ac0ae390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 shutdown_connections 2024-09-13T17:11:42.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.451+0000 7fc3b0e00640 1 -- 172.21.15.28:0/3268257466 wait complete. 2024-09-13T17:11:42.468 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:11:42.468 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 damage ls 2024-09-13T17:11:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.624+0000 7fd845400640 1 Processor -- start 2024-09-13T17:11:42.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.624+0000 7fd845400640 1 -- start start 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd845400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd84015a780 0x7fd840070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd845400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd84006c250 0x7fd8400712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd845400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd84006c660 con 0x7fd84015ab60 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd845400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84006c8b0 con 0x7fd84006c250 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd845400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84006cb90 con 0x7fd84015a780 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd83f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd84006c250 0x7fd8400712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd83f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd84006c250 0x7fd8400712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54080/0 (socket says 172.21.15.28:54080) 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.625+0000 7fd83f200640 1 -- 172.21.15.28:0/4021815918 learned_addr learned my addr 172.21.15.28:0/4021815918 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.626+0000 7fd83e800640 1 --2- 172.21.15.28:0/4021815918 >> v2:172.21.15.189:3300/0 conn(0x7fd84015a780 0x7fd840070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.626+0000 7fd83f200640 1 -- 172.21.15.28:0/4021815918 >> v2:172.21.15.189:3300/0 conn(0x7fd84015a780 msgr2=0x7fd840070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.626+0000 7fd83f200640 1 --2- 172.21.15.28:0/4021815918 >> v2:172.21.15.189:3300/0 conn(0x7fd84015a780 0x7fd840070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.626+0000 7fd83f200640 1 -- 172.21.15.28:0/4021815918 >> v1:172.21.15.189:6789/0 conn(0x7fd84015ab60 legacy=0x7fd84006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.626+0000 7fd83f200640 1 -- 172.21.15.28:0/4021815918 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8400cc980 con 0x7fd84006c250 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.627+0000 7fd83f200640 1 --2- 172.21.15.28:0/4021815918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84006c250 0x7fd8400712c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd8400b1330 tx=0x7fd83802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184fdb80e468d0e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.627+0000 7fd83d400640 1 -- 172.21.15.28:0/4021815918 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838032440 con 0x7fd84006c250 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.627+0000 7fd83d400640 1 -- 172.21.15.28:0/4021815918 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8380325a0 con 0x7fd84006c250 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.627+0000 7fd83d400640 1 -- 172.21.15.28:0/4021815918 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838032930 con 0x7fd84006c250 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 -- 172.21.15.28:0/4021815918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84006c250 msgr2=0x7fd8400712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 --2- 172.21.15.28:0/4021815918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84006c250 0x7fd8400712c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fd8400b1330 tx=0x7fd83802a270 comp rx=0 tx=0).stop 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 -- 172.21.15.28:0/4021815918 shutdown_connections 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 --2- 172.21.15.28:0/4021815918 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84006c250 0x7fd8400712c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 --2- 172.21.15.28:0/4021815918 >> v2:172.21.15.189:3300/0 conn(0x7fd84015a780 0x7fd840070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 -- 172.21.15.28:0/4021815918 >> 172.21.15.28:0/4021815918 conn(0x7fd8400b0270 msgr2=0x7fd8400b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 -- 172.21.15.28:0/4021815918 shutdown_connections 2024-09-13T17:11:42.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.628+0000 7fd845400640 1 -- 172.21.15.28:0/4021815918 wait complete. 2024-09-13T17:11:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.630+0000 7fd845400640 1 Processor -- start 2024-09-13T17:11:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.630+0000 7fd845400640 1 -- start start 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 0x7fd840154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 0x7fd840140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8401414f0 con 0x7fd84015a780 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd84015b3e0 con 0x7fd84006c250 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd845400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd84015b6c0 con 0x7fd84015ab60 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54096/0 (socket says 172.21.15.28:54096) 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 -- 172.21.15.28:0/3091986005 learned_addr learned my addr 172.21.15.28:0/3091986005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83e800640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 0x7fd840154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83f200640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 0x7fd840140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 -- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 msgr2=0x7fd840140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 0x7fd840140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 -- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 msgr2=0x7fd840154410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 0x7fd840154410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83de00640 1 -- 172.21.15.28:0/3091986005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380094d0 con 0x7fd84015a780 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83e800640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 0x7fd840154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.631+0000 7fd83f200640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 0x7fd840140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd83de00640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c400 tx=0x7fd82c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd836a00640 1 -- 172.21.15.28:0/3091986005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c0052d0 con 0x7fd84015a780 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd836a00640 1 -- 172.21.15.28:0/3091986005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c005430 con 0x7fd84015a780 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd836a00640 1 -- 172.21.15.28:0/3091986005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c003830 con 0x7fd84015a780 2024-09-13T17:11:42.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd840156fa0 con 0x7fd84015a780 2024-09-13T17:11:42.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.632+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd840157560 con 0x7fd84015a780 2024-09-13T17:11:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.635+0000 7fd836a00640 1 -- 172.21.15.28:0/3091986005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd82c003990 con 0x7fd84015a780 2024-09-13T17:11:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.636+0000 7fd836a00640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 0x7fd82006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.636+0000 7fd836a00640 1 -- 172.21.15.28:0/3091986005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd82c088970 con 0x7fd84015a780 2024-09-13T17:11:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.636+0000 7fd84820c740 1 Processor -- start 2024-09-13T17:11:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.636+0000 7fd83e800640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 0x7fd82006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.636+0000 7fd84820c740 1 -- start start 2024-09-13T17:11:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd83e800640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 0x7fd82006d0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd830009b20 tx=0x7fd8300039d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f5051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 0x55e28f552820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f411020 con 0x55e28f5025d0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f46eba0 con 0x55e28f504e10 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e28f46c330 con 0x55e28f52c2c0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd83e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f5051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.637+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54104/0 (socket says 172.21.15.28:54104) 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 -- 172.21.15.28:0/945827955 learned_addr learned my addr 172.21.15.28:0/945827955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83f200640 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 0x55e28f552820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 -- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 msgr2=0x55e28f552820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 0x55e28f552820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 -- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 msgr2=0x55e28f5051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f5051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 -- 172.21.15.28:0/945827955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e28f419cf0 con 0x55e28f5025d0 2024-09-13T17:11:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83f200640 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 0x55e28f552820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd83de00640 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd82c053c70 tx=0x7fd82c0573e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa607c17bf69423 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd82b400640 1 -- 172.21.15.28:0/945827955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c05fb60 con 0x55e28f5025d0 2024-09-13T17:11:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd82b400640 1 -- 172.21.15.28:0/945827955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c055b10 con 0x55e28f5025d0 2024-09-13T17:11:42.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.638+0000 7fd82b400640 1 -- 172.21.15.28:0/945827955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c05b170 con 0x55e28f5025d0 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 -- 172.21.15.28:0/945827955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 msgr2=0x55e28f5522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fd82c053c70 tx=0x7fd82c0573e0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 -- 172.21.15.28:0/945827955 shutdown_connections 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f52c2c0 0x55e28f552820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f5025d0 0x55e28f5522e0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 --2- 172.21.15.28:0/945827955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f5051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 -- 172.21.15.28:0/945827955 >> 172.21.15.28:0/945827955 conn(0x55e28f501540 msgr2=0x55e28f5499f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 -- 172.21.15.28:0/945827955 shutdown_connections 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.639+0000 7fd84820c740 1 -- 172.21.15.28:0/945827955 wait complete. 2024-09-13T17:11:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.640+0000 7fd84820c740 1 -- start start 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.640+0000 7fd84820c740 1 Processor -- start 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f521a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f527150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f467c20 con 0x55e28f52c2c0 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f4091b0 con 0x55e28f504e10 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e28f408e50 con 0x55e28f523d70 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd84820c740 10 client.0 _ll_register_callbacks cb 0x55e28f461290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54116/0 (socket says 172.21.15.28:54116) 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.641+0000 7fd83de00640 1 -- 172.21.15.28:0/4081975100 learned_addr learned my addr 172.21.15.28:0/4081975100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83f200640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f527150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83e800640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f521a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 -- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 msgr2=0x55e28f527150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f527150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 -- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 msgr2=0x55e28f521a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f521a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 -- 172.21.15.28:0/4081975100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd840157560 con 0x55e28f52c2c0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83f200640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f527150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83e800640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f521a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd83de00640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd82c057470 tx=0x7fd82c001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f7c5cadbd9237c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd81fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c05b370 con 0x55e28f52c2c0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd82c05fb60 con 0x55e28f52c2c0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e28f490eb0 con 0x55e28f52c2c0 2024-09-13T17:11:42.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd82c0760a0 con 0x55e28f52c2c0 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd84820c740 10 client.6180 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55e28f4662a0 con 0x55e28f52c2c0 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.642+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e28f490eb0 con 0x55e28f52c2c0 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd82c05f6d0 con 0x55e28f52c2c0 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 10 client.6180 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 10 client.6180 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 20 client.6180 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f5246f0 0x55e28f524ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 4 client.6180 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e28f45c340 con 0x55e28f5246f0 2024-09-13T17:11:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd83e800640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f5246f0 0x55e28f524ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd83e800640 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f5246f0 0x55e28f524ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=814b080244cba6be server_cookie=1f89148c4324381d in_seq=0 out_seq=0 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.643+0000 7fd81fe00640 10 client.6180 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd81fe00640 1 -- 172.21.15.28:0/4081975100 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55e28f45c340 con 0x55e28f5246f0 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd81fe00640 10 client.6180 handle_command_reply: tid=0 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 client.6180 shutdown 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 msgr2=0x55e28f526c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd82c057470 tx=0x7fd82c001080 comp rx=0 tx=0).stop 2024-09-13T17:11:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 shutdown_connections 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f527150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f5246f0 0x55e28f524ad0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f52c2c0 0x55e28f526c10 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 --2- 172.21.15.28:0/4081975100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f504e10 0x55e28f521a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd81fe00640 0 client.6180 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.644+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 >> 172.21.15.28:0/4081975100 conn(0x55e28f5025d0 msgr2=0x55e28f547d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.645+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 shutdown_connections 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.645+0000 7fd84820c740 1 -- 172.21.15.28:0/4081975100 wait complete. 2024-09-13T17:11:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.645+0000 7fd84820c740 20 client.6180 trim_cache size 0 max 16384 2024-09-13T17:11:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.656+0000 7fd84820c740 1 Processor -- start 2024-09-13T17:11:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.656+0000 7fd84820c740 1 -- start start 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f54d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f4ffb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f1cf040 con 0x55e28f504e10 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f5626b0 con 0x55e28f52c2c0 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e28f546cc0 con 0x55e28f523d70 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd83e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.657+0000 7fd83e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54126/0 (socket says 172.21.15.28:54126) 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 -- 172.21.15.28:0/2874118516 learned_addr learned my addr 172.21.15.28:0/2874118516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83de00640 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f54d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83f200640 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f4ffb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 -- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 msgr2=0x55e28f54d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f54d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 -- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 msgr2=0x55e28f4ffb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f4ffb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 -- 172.21.15.28:0/2874118516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e28f49a200 con 0x55e28f504e10 2024-09-13T17:11:42.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd83e800640 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd830004410 tx=0x7fd830015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4de3498cd257c040 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd82be00640 1 -- 172.21.15.28:0/2874118516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830016b80 con 0x55e28f504e10 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd82be00640 1 -- 172.21.15.28:0/2874118516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd830016ce0 con 0x55e28f504e10 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.658+0000 7fd82be00640 1 -- 172.21.15.28:0/2874118516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd830028360 con 0x55e28f504e10 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 -- 172.21.15.28:0/2874118516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 msgr2=0x55e28f5483a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fd830004410 tx=0x7fd830015000 comp rx=0 tx=0).stop 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 -- 172.21.15.28:0/2874118516 shutdown_connections 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f4ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f523d70 0x55e28f54d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 --2- 172.21.15.28:0/2874118516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f504e10 0x55e28f5483a0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 -- 172.21.15.28:0/2874118516 >> 172.21.15.28:0/2874118516 conn(0x55e28f5025d0 msgr2=0x55e28f521e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 -- 172.21.15.28:0/2874118516 shutdown_connections 2024-09-13T17:11:42.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.659+0000 7fd84820c740 1 -- 172.21.15.28:0/2874118516 wait complete. 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.660+0000 7fd84820c740 1 -- start start 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.660+0000 7fd84820c740 1 Processor -- start 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f504e10 0x55e28f5b4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f59cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f563280 con 0x55e28f523d70 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e28f527690 con 0x55e28f52c2c0 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e28f529d60 con 0x55e28f504e10 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd84820c740 10 client.0 _ll_register_callbacks cb 0x7fd8300097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54140/0 (socket says 172.21.15.28:54140) 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83de00640 1 -- 172.21.15.28:0/2093390531 learned_addr learned my addr 172.21.15.28:0/2093390531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83e800640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f504e10 0x55e28f5b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83f200640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f59cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83de00640 1 -- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f504e10 msgr2=0x55e28f5b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.661+0000 7fd83de00640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f504e10 0x55e28f5b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd83de00640 1 -- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 msgr2=0x55e28f59cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd83de00640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f59cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd83de00640 1 -- 172.21.15.28:0/2093390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83001e040 con 0x55e28f523d70 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd83f200640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f59cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd83de00640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd82c057420 tx=0x7fd82c053b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ad3879a7fc4dd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd81fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd82c053760 con 0x55e28f523d70 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd82c0538c0 con 0x55e28f523d70 2024-09-13T17:11:42.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd82c055b10 con 0x55e28f523d70 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd82c0577f0 con 0x55e28f523d70 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd84820c740 10 client.6186 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55e28f4085f0 con 0x55e28f523d70 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.662+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd82c055b10 con 0x55e28f523d70 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd82c0566d0 con 0x55e28f523d70 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 10 client.6186 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 10 client.6186 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.c 2024-09-13T17:11:42.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 20 client.6186 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f4ffb50 0x55e28f4fff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 4 client.6186 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e28f15b4e0 con 0x55e28f4ffb50 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd83e800640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f4ffb50 0x55e28f4fff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd83e800640 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f4ffb50 0x55e28f4fff30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=149d2196dd9c5172 server_cookie=3da0377e4cb68894 in_seq=0 out_seq=0 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.663+0000 7fd81fe00640 10 client.6186 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd81fe00640 1 -- 172.21.15.28:0/2093390531 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+168 (crc 0 0 0) 0x55e28f15b4e0 con 0x55e28f4ffb50 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd81fe00640 10 client.6186 handle_command_reply: tid=0 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 client.6186 shutdown 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 msgr2=0x55e28f549010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fd82c057420 tx=0x7fd82c053b30 comp rx=0 tx=0).stop 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 shutdown_connections 2024-09-13T17:11:42.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e28f52c2c0 0x55e28f59cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55e28f4ffb50 0x55e28f4fff30 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e28f523d70 0x55e28f549010 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 --2- 172.21.15.28:0/2093390531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e28f504e10 0x55e28f5b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 >> 172.21.15.28:0/2093390531 conn(0x55e28f5025d0 msgr2=0x55e28f5479a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd81fe00640 0 client.6186 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 shutdown_connections 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 1 -- 172.21.15.28:0/2093390531 wait complete. 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.664+0000 7fd84820c740 20 client.6186 trim_cache size 0 max 16384 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:11:42.670 INFO:teuthology.orchestra.run.smithi028.stdout: "damage_type": "backtrace", 2024-09-13T17:11:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 2979790141, 2024-09-13T17:11:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511630188, 2024-09-13T17:11:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_test_path/ceph-qa-suite/new_file_1" 2024-09-13T17:11:42.671 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:11:42.671 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 msgr2=0x7fd82006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 0x7fd82006d0c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd830009b20 tx=0x7fd8300039d0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 msgr2=0x7fd840140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd82c00c400 tx=0x7fd82c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 shutdown_connections 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd82006ac00 0x7fd82006d0c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd84015ab60 0x7fd840140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84015a780 0x7fd840140a70 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 --2- 172.21.15.28:0/3091986005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd84006c250 0x7fd840154410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 >> 172.21.15.28:0/3091986005 conn(0x7fd8400b0270 msgr2=0x7fd8400ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 shutdown_connections 2024-09-13T17:11:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.665+0000 7fd845400640 1 -- 172.21.15.28:0/3091986005 wait complete. 2024-09-13T17:11:42.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.836+0000 7f55f9a00640 1 Processor -- start 2024-09-13T17:11:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.836+0000 7f55f9a00640 1 -- start start 2024-09-13T17:11:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.837+0000 7f55f9a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 0x7f55f415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.837+0000 7f55f9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f4159d90 con 0x7f55f406c2d0 2024-09-13T17:11:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.837+0000 7f55f9a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f40b2040 con 0x7f55f406f970 2024-09-13T17:11:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.837+0000 7f55f9a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f415b0b0 con 0x7f55f41592a0 2024-09-13T17:11:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f55f406c2d0 0x7f55f40b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33876/0 (socket says 172.21.15.28:33876) 2024-09-13T17:11:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f3400640 1 -- 172.21.15.28:0/567662696 learned_addr learned my addr 172.21.15.28:0/567662696 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f2a00640 1 --2- 172.21.15.28:0/567662696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 0x7f55f415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074314609 0 0) 0x7f55f4159d90 con 0x7f55f406c2d0 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55d8003520 con 0x7f55f406c2d0 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962011451 0 0) 0x7f55f40b2040 con 0x7f55f406f970 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55f4159d90 con 0x7f55f406f970 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f2a00640 1 -- 172.21.15.28:0/567662696 >> v1:172.21.15.189:6789/0 conn(0x7f55f406f970 legacy=0x7f55f4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f2a00640 1 -- 172.21.15.28:0/567662696 >> v1:172.21.15.28:6789/0 conn(0x7f55f406c2d0 legacy=0x7f55f40b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.838+0000 7f55f2a00640 1 -- 172.21.15.28:0/567662696 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f4159ff0 con 0x7f55f41592a0 2024-09-13T17:11:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.839+0000 7f55f2a00640 1 --2- 172.21.15.28:0/567662696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 0x7f55f415ab70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55e40097d0 tx=0x7f55e402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe30345924bcc53e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4032440 con 0x7f55f41592a0 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e40325a0 con 0x7f55f41592a0 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f1600640 1 -- 172.21.15.28:0/567662696 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e4032930 con 0x7f55f41592a0 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 -- 172.21.15.28:0/567662696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 msgr2=0x7f55f415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 --2- 172.21.15.28:0/567662696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 0x7f55f415ab70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f55e40097d0 tx=0x7f55e402a210 comp rx=0 tx=0).stop 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 -- 172.21.15.28:0/567662696 shutdown_connections 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 --2- 172.21.15.28:0/567662696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f41592a0 0x7f55f415ab70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 -- 172.21.15.28:0/567662696 >> 172.21.15.28:0/567662696 conn(0x7f55f4069e80 msgr2=0x7f55f406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 -- 172.21.15.28:0/567662696 shutdown_connections 2024-09-13T17:11:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.840+0000 7f55f9a00640 1 -- 172.21.15.28:0/567662696 wait complete. 2024-09-13T17:11:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.842+0000 7f55f9a00640 1 Processor -- start 2024-09-13T17:11:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.842+0000 7f55f9a00640 1 -- start start 2024-09-13T17:11:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f406c2d0 0x7f55f4156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55f406f970 0x7f55f41520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f414ea00 con 0x7f55f41592a0 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f414ece0 con 0x7f55f406f970 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f9a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f414efc0 con 0x7f55f406c2d0 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54148/0 (socket says 172.21.15.28:54148) 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.843+0000 7f55f3400640 1 -- 172.21.15.28:0/3288129455 learned_addr learned my addr 172.21.15.28:0/3288129455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 -- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f406c2d0 msgr2=0x7f55f4156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f406c2d0 0x7f55f4156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 -- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55f406f970 msgr2=0x7f55f41520c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:42.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55f406f970 0x7f55f41520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 -- 172.21.15.28:0/3288129455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e4009400 con 0x7f55f41592a0 2024-09-13T17:11:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f3400640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00cb40 tx=0x7f55ec00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec00aca0 con 0x7f55f41592a0 2024-09-13T17:11:42.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ec00ae00 con 0x7f55f41592a0 2024-09-13T17:11:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ec012600 con 0x7f55f41592a0 2024-09-13T17:11:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.844+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f414f2a0 con 0x7f55f41592a0 2024-09-13T17:11:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.845+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f4156a80 con 0x7f55f41592a0 2024-09-13T17:11:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.846+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f414faf0 con 0x7f55f41592a0 2024-09-13T17:11:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.846+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f55ec012760 con 0x7f55f41592a0 2024-09-13T17:11:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.847+0000 7f55eaa00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 0x7f55d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.847+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f55ec0892d0 con 0x7f55f41592a0 2024-09-13T17:11:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.848+0000 7f55f2a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 0x7f55d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.848+0000 7f55f2a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 0x7f55d406d0c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f55e4034000 tx=0x7f55e402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:42.852+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f55f414faf0 con 0x7f55f41592a0 2024-09-13T17:11:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.064+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f414faf0 con 0x7f55f41592a0 2024-09-13T17:11:43.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.065+0000 7f55eaa00640 1 -- 172.21.15.28:0/3288129455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f55f414faf0 con 0x7f55f41592a0 2024-09-13T17:11:43.067 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:43.068 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:41.068760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":11,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:43.068 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 39 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 msgr2=0x7f55d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 0x7f55d406d0c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f55e4034000 tx=0x7f55e402c000 comp rx=0 tx=0).stop 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 msgr2=0x7f55f40ce780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f55ec00cb40 tx=0x7f55ec00a060 comp rx=0 tx=0).stop 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 shutdown_connections 2024-09-13T17:11:43.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f55d406ac00 0x7f55d406d0c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55f41592a0 0x7f55f40ce780 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f55f406f970 0x7f55f41520c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 --2- 172.21.15.28:0/3288129455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f55f406c2d0 0x7f55f4156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.067+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 >> 172.21.15.28:0/3288129455 conn(0x7f55f4069e80 msgr2=0x7f55f40ada80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.068+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 shutdown_connections 2024-09-13T17:11:43.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.068+0000 7f55f9a00640 1 -- 172.21.15.28:0/3288129455 wait complete. 2024-09-13T17:11:43.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage rm 2979790141 2024-09-13T17:11:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.244+0000 7f600ae00640 1 Processor -- start 2024-09-13T17:11:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.244+0000 7f600ae00640 1 -- start start 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f600ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f600406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f600ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600406f920 con 0x7f600415cd80 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f600ae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600415b290 con 0x7f600406f3f0 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f600ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f600415b570 con 0x7f6004159770 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f6008e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f600415cd80 0x7f600415a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33886/0 (socket says 172.21.15.28:33886) 2024-09-13T17:11:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.245+0000 7f6008e00640 1 -- 172.21.15.28:0/494849677 learned_addr learned my addr 172.21.15.28:0/494849677 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003e00640 1 --2- 172.21.15.28:0/494849677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f600406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 700765395 0 0) 0x7f600406f920 con 0x7f600415cd80 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe8003520 con 0x7f600415cd80 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145527760 0 0) 0x7f600415b290 con 0x7f600406f3f0 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600406f920 con 0x7f600406f3f0 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003e00640 1 -- 172.21.15.28:0/494849677 >> v1:172.21.15.189:6789/0 conn(0x7f600406f3f0 legacy=0x7f6004159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003e00640 1 -- 172.21.15.28:0/494849677 >> v1:172.21.15.28:6789/0 conn(0x7f600415cd80 legacy=0x7f600415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3392904754 0 0) 0x7f5fe8003520 con 0x7f600415cd80 2024-09-13T17:11:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.246+0000 7f6003e00640 1 -- 172.21.15.28:0/494849677 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600415b850 con 0x7f6004159770 2024-09-13T17:11:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.247+0000 7f6003e00640 1 --2- 172.21.15.28:0/494849677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f600406fd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40097d0 tx=0x7f5ff402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1af7c3fb14e8b684 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0019f0 con 0x7f6004159770 2024-09-13T17:11:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff402ab90 con 0x7f6004159770 2024-09-13T17:11:43.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f6003400640 1 -- 172.21.15.28:0/494849677 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4032440 con 0x7f6004159770 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 -- 172.21.15.28:0/494849677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 msgr2=0x7f600406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 --2- 172.21.15.28:0/494849677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f600406fd20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40097d0 tx=0x7f5ff402a100 comp rx=0 tx=0).stop 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 -- 172.21.15.28:0/494849677 shutdown_connections 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 --2- 172.21.15.28:0/494849677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f600406fd20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 -- 172.21.15.28:0/494849677 >> 172.21.15.28:0/494849677 conn(0x7f60040b0270 msgr2=0x7f60040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 -- 172.21.15.28:0/494849677 shutdown_connections 2024-09-13T17:11:43.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.248+0000 7f600ae00640 1 -- 172.21.15.28:0/494849677 wait complete. 2024-09-13T17:11:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.250+0000 7f600ae00640 1 Processor -- start 2024-09-13T17:11:43.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.250+0000 7f600ae00640 1 -- start start 2024-09-13T17:11:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f6004158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600415cd80 0x7f60040ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6004156580 con 0x7f600406f3f0 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60041e2390 con 0x7f600415cd80 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f600ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60041e2670 con 0x7f6004159770 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54162/0 (socket says 172.21.15.28:54162) 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.251+0000 7f6003e00640 1 -- 172.21.15.28:0/2922461443 learned_addr learned my addr 172.21.15.28:0/2922461443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6008e00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600415cd80 0x7f60040ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f5ffb400640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f6004158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 -- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 msgr2=0x7f6004158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f6004158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 -- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600415cd80 msgr2=0x7f60040ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600415cd80 0x7f60040ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 -- 172.21.15.28:0/2922461443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4009400 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6003e00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4035040 tx=0x7f5ff40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6000c00640 1 -- 172.21.15.28:0/2922461443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff403b810 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6000c00640 1 -- 172.21.15.28:0/2922461443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff403b990 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f6000c00640 1 -- 172.21.15.28:0/2922461443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff403bd40 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.252+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60041e2950 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.253+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60041e2f10 con 0x7f600406f3f0 2024-09-13T17:11:43.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.255+0000 7f6000c00640 1 -- 172.21.15.28:0/2922461443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5ff4039040 con 0x7f600406f3f0 2024-09-13T17:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.255+0000 7f6000c00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 0x7f5fe006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.255+0000 7f6000c00640 1 -- 172.21.15.28:0/2922461443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ff40a9940 con 0x7f600406f3f0 2024-09-13T17:11:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.256+0000 7f5ffb400640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 0x7f5fe006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.256+0000 7f600dc84740 1 Processor -- start 2024-09-13T17:11:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.256+0000 7f600dc84740 1 -- start start 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f5ffb400640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 0x7f5fe006d0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6004067260 tx=0x7f5fec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb149322e0 0x55fb149326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb149804b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb1489aba0 con 0x55fb14959b40 2024-09-13T17:11:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb14898330 con 0x55fb149322e0 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.257+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb14893c20 con 0x55fb14980000 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54178/0 (socket says 172.21.15.28:54178) 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f6003e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb149322e0 0x55fb149326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 -- 172.21.15.28:0/3138126966 learned_addr learned my addr 172.21.15.28:0/3138126966 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f6008e00640 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb149804b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 -- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 msgr2=0x55fb149804b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb149804b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 -- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb149322e0 msgr2=0x55fb149326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb149322e0 0x55fb149326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 -- 172.21.15.28:0/3138126966 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb14838f70 con 0x55fb14959b40 2024-09-13T17:11:43.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5fec001df0 tx=0x7f5fec011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a890f9eae9df25e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ff8c00640 1 -- 172.21.15.28:0/3138126966 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0035f0 con 0x55fb14959b40 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ff8c00640 1 -- 172.21.15.28:0/3138126966 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec003750 con 0x55fb14959b40 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.258+0000 7f5ff8c00640 1 -- 172.21.15.28:0/3138126966 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec003ac0 con 0x55fb14959b40 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 -- 172.21.15.28:0/3138126966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 msgr2=0x55fb1497fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f5fec001df0 tx=0x7f5fec011000 comp rx=0 tx=0).stop 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 -- 172.21.15.28:0/3138126966 shutdown_connections 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb149804b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1497fac0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 --2- 172.21.15.28:0/3138126966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb149322e0 0x55fb149326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 -- 172.21.15.28:0/3138126966 >> 172.21.15.28:0/3138126966 conn(0x55fb1492eb30 msgr2=0x55fb14976fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 -- 172.21.15.28:0/3138126966 shutdown_connections 2024-09-13T17:11:43.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.259+0000 7f600dc84740 1 -- 172.21.15.28:0/3138126966 wait complete. 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.260+0000 7f600dc84740 1 -- start start 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.260+0000 7f600dc84740 1 Processor -- start 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb14957d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1492c1b0 0x55fb14958290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb14851fb0 con 0x55fb14959b40 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb14851c50 con 0x55fb1492c1b0 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb145faf50 con 0x55fb14980000 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f600dc84740 10 client.0 _ll_register_callbacks cb 0x55fb1488d290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:43.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54190/0 (socket says 172.21.15.28:54190) 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f6003e00640 1 -- 172.21.15.28:0/3650121226 learned_addr learned my addr 172.21.15.28:0/3650121226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f6008e00640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1492c1b0 0x55fb14958290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.261+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb14957d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 -- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 msgr2=0x55fb14957d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb14957d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 -- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1492c1b0 msgr2=0x55fb14958290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1492c1b0 0x55fb14958290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 -- 172.21.15.28:0/3650121226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec00f040 con 0x55fb14959b40 2024-09-13T17:11:43.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f6003e00640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4006a90 tx=0x7f5ff4006f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a245f78cb686de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f5ff1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff407c690 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ff4080c50 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fb148bda60 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ff4080ea0 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f600dc84740 10 client.6204 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.262+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55fb1484f090 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff4084080 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5ff4077d50 con 0x55fb14959b40 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 10 client.6204 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 10 client.6204 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 20 client.6204 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:43.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb14958910 0x55fb1495a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 4 client.6204 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb148866f0 con 0x55fb14958910 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.263+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb14958910 0x55fb1495a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb14958910 0x55fb1495a5d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae77db9cc40950e3 server_cookie=b144e0647cbf0d93 in_seq=0 out_seq=0 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f5ff1600640 10 client.6204 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f5ff1600640 1 -- 172.21.15.28:0/3650121226 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55fb148866f0 con 0x55fb14958910 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f5ff1600640 10 client.6204 handle_command_reply: tid=0 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f600dc84740 1 client.6204 shutdown 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 msgr2=0x55fb1492bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.264+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4006a90 tx=0x7f5ff4006f40 comp rx=0 tx=0).stop 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 shutdown_connections 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb1492c1b0 0x55fb14958290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14980000 0x55fb14957d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb14958910 0x55fb1495a5d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 --2- 172.21.15.28:0/3650121226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb1492bc70 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 >> 172.21.15.28:0/3650121226 conn(0x55fb149322e0 msgr2=0x55fb14977880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f5ff1600640 0 client.6204 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 shutdown_connections 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 1 -- 172.21.15.28:0/3650121226 wait complete. 2024-09-13T17:11:43.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.265+0000 7f600dc84740 20 client.6204 trim_cache size 0 max 16384 2024-09-13T17:11:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.277+0000 7f600dc84740 1 Processor -- start 2024-09-13T17:11:43.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.277+0000 7f600dc84740 1 -- start start 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb1492c1b0 0x55fb1495a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb145b7150 con 0x55fb14959b40 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb1492e8b0 con 0x55fb14980000 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb1497e580 con 0x55fb1492c1b0 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54206/0 (socket says 172.21.15.28:54206) 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 -- 172.21.15.28:0/3770645695 learned_addr learned my addr 172.21.15.28:0/3770645695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 -- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb1492c1b0 msgr2=0x55fb1495a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb1492c1b0 0x55fb1495a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 -- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 msgr2=0x55fb14976210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f6008e00640 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.278+0000 7f5ffb400640 1 -- 172.21.15.28:0/3770645695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb148341f0 con 0x55fb14959b40 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f5ffb400640 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007df0 tx=0x7f5fec005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a0d23eaa14c731 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f5ff9600640 1 -- 172.21.15.28:0/3770645695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec0035f0 con 0x55fb14959b40 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f5ff9600640 1 -- 172.21.15.28:0/3770645695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec003ac0 con 0x55fb14959b40 2024-09-13T17:11:43.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f5ff9600640 1 -- 172.21.15.28:0/3770645695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fec003e30 con 0x55fb14959b40 2024-09-13T17:11:43.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f600dc84740 1 -- 172.21.15.28:0/3770645695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 msgr2=0x55fb149d80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f600dc84740 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f5fec007df0 tx=0x7f5fec005000 comp rx=0 tx=0).stop 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.279+0000 7f600dc84740 1 -- 172.21.15.28:0/3770645695 shutdown_connections 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb14959b40 0x55fb149d80e0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 --2- 172.21.15.28:0/3770645695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb1492c1b0 0x55fb1495a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 -- 172.21.15.28:0/3770645695 >> 172.21.15.28:0/3770645695 conn(0x55fb149322e0 msgr2=0x55fb1495b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 -- 172.21.15.28:0/3770645695 shutdown_connections 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 -- 172.21.15.28:0/3770645695 wait complete. 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 -- start start 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.280+0000 7f600dc84740 1 Processor -- start 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14959b40 0x55fb149d6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb14977bd0 con 0x55fb1492c1b0 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb1495b230 con 0x55fb14980000 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.281+0000 7f600dc84740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fb1495bc20 con 0x55fb14959b40 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f600dc84740 10 client.0 _ll_register_callbacks cb 0x7f5ff4074c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54212/0 (socket says 172.21.15.28:54212) 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 -- 172.21.15.28:0/989872846 learned_addr learned my addr 172.21.15.28:0/989872846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f5ffb400640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14959b40 0x55fb149d6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6008e00640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 -- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14959b40 msgr2=0x55fb149d6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14959b40 0x55fb149d6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 -- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 msgr2=0x55fb14976fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 -- 172.21.15.28:0/989872846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb14838f70 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f6003e00640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4074d60 tx=0x7f5ff40a7ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c4a13d7d5a74e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.282+0000 7f5ff1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4088070 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ff4080c50 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fec003750 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ff4080ea0 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 10 client.6210 fetch_fsmap learned FSMap version 39 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=2+}) v3 -- 0x55fb14859690 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fec003750 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 39) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5ff40807c0 con 0x55fb1492c1b0 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 10 client.6210 fetch_fsmap finished waiting for FSMap version 39 2024-09-13T17:11:43.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 10 client.6210 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:11:43.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 20 client.6210 populate_metadata read hostname 'smithi028' 2024-09-13T17:11:43.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb1495a790 0x55fb1495ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 4 client.6210 mds_command: new command op to 5580 tid=0 multi_id=0 [{"prefix": "damage rm", "damage_id": 2979790141}] 2024-09-13T17:11:43.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 --> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] -- command(tid 0: {"prefix": "damage rm", "damage_id": 2979790141}) v1 -- 0x55fb145873f0 con 0x55fb1495a790 2024-09-13T17:11:43.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.283+0000 7f5ffb400640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb1495a790 0x55fb1495ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f5ffb400640 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb1495a790 0x55fb1495ab70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2a2adddc2014e8a server_cookie=a0d67bd1bf5eb13a in_seq=0 out_seq=0 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f5ff1600640 10 client.6210 ms_handle_connect on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f5ff1600640 1 -- 172.21.15.28:0/989872846 <== mds.0 v2:172.21.15.28:6832/1182054705 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+0 (crc 0 0 0) 0x55fb145873f0 con 0x55fb1495a790 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f5ff1600640 10 client.6210 handle_command_reply: tid=0 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f600dc84740 1 client.6210 shutdown 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 msgr2=0x55fb1495b6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.284+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4074d60 tx=0x7f5ff40a7ca0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 shutdown_connections 2024-09-13T17:11:43.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fb14980000 0x55fb14976fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fb14959b40 0x55fb149d6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:6832/1182054705,v1:172.21.15.28:6833/1182054705] conn(0x55fb1495a790 0x55fb1495ab70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 --2- 172.21.15.28:0/989872846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fb1492c1b0 0x55fb1495b6e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 >> 172.21.15.28:0/989872846 conn(0x55fb149322e0 msgr2=0x55fb1497ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f5ff1600640 0 client.6210 ms_handle_reset on v2:172.21.15.28:6832/1182054705 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 shutdown_connections 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 1 -- 172.21.15.28:0/989872846 wait complete. 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600dc84740 20 client.6210 trim_cache size 0 max 16384 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 msgr2=0x7f5fe006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 0x7f5fe006d0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6004067260 tx=0x7f5fec009040 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 msgr2=0x7f6004156040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f5ff4035040 tx=0x7f5ff40046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 shutdown_connections 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5fe006ac00 0x7f5fe006d0c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f600415cd80 0x7f60040ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6004159770 0x7f6004158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 --2- 172.21.15.28:0/2922461443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f600406f3f0 0x7f6004156040 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.285+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 >> 172.21.15.28:0/2922461443 conn(0x7f60040b0270 msgr2=0x7f60040cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.286+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 shutdown_connections 2024-09-13T17:11:43.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.286+0000 7f600ae00640 1 -- 172.21.15.28:0/2922461443 wait complete. 2024-09-13T17:11:43.303 INFO:tasks.cephfs.test_scrub_checks:Running command 'flush_path /' 2024-09-13T17:11:43.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:43.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.456+0000 7fd896600640 1 Processor -- start 2024-09-13T17:11:43.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.456+0000 7fd896600640 1 -- start start 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd896600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 0x7fd89015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd896600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 0x7fd89006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd896600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89006cc70 con 0x7fd8901592a0 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd896600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89015b0b0 con 0x7fd89006f970 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd896600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd890159d90 con 0x7fd89006c2d0 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd88f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd8901592a0 0x7fd890159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33902/0 (socket says 172.21.15.28:33902) 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd88f400640 1 -- 172.21.15.28:0/2441061127 learned_addr learned my addr 172.21.15.28:0/2441061127 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd887e00640 1 --2- 172.21.15.28:0/2441061127 >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 0x7fd89015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2441061127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 0x7fd89006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.458+0000 7fd88ea00640 1 -- 172.21.15.28:0/2441061127 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648881566 0 0) 0x7fd89006cc70 con 0x7fd8901592a0 2024-09-13T17:11:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd88ea00640 1 -- 172.21.15.28:0/2441061127 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd874003520 con 0x7fd8901592a0 2024-09-13T17:11:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd88fe00640 1 -- 172.21.15.28:0/2441061127 >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 msgr2=0x7fd89015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2441061127 >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 0x7fd89015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd88fe00640 1 -- 172.21.15.28:0/2441061127 >> v1:172.21.15.28:6789/0 conn(0x7fd8901592a0 legacy=0x7fd890159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd88fe00640 1 -- 172.21.15.28:0/2441061127 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8900b2040 con 0x7fd89006c2d0 2024-09-13T17:11:43.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.459+0000 7fd887e00640 1 --2- 172.21.15.28:0/2441061127 >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 0x7fd89015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.460+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2441061127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 0x7fd89006c730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd888009d90 tx=0x7fd88802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b47b8bf5e4581d6e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.460+0000 7fd88ea00640 1 -- 172.21.15.28:0/2441061127 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888007740 con 0x7fd89006c2d0 2024-09-13T17:11:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.460+0000 7fd88ea00640 1 -- 172.21.15.28:0/2441061127 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88802d050 con 0x7fd89006c2d0 2024-09-13T17:11:43.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.460+0000 7fd88ea00640 1 -- 172.21.15.28:0/2441061127 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888007ac0 con 0x7fd89006c2d0 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 -- 172.21.15.28:0/2441061127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 msgr2=0x7fd89006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 --2- 172.21.15.28:0/2441061127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 0x7fd89006c730 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd888009d90 tx=0x7fd88802a660 comp rx=0 tx=0).stop 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 -- 172.21.15.28:0/2441061127 shutdown_connections 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 --2- 172.21.15.28:0/2441061127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006c2d0 0x7fd89006c730 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 --2- 172.21.15.28:0/2441061127 >> v2:172.21.15.189:3300/0 conn(0x7fd89006f970 0x7fd89015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 -- 172.21.15.28:0/2441061127 >> 172.21.15.28:0/2441061127 conn(0x7fd890069e80 msgr2=0x7fd89006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 -- 172.21.15.28:0/2441061127 shutdown_connections 2024-09-13T17:11:43.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.461+0000 7fd896600640 1 -- 172.21.15.28:0/2441061127 wait complete. 2024-09-13T17:11:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.462+0000 7fd896600640 1 Processor -- start 2024-09-13T17:11:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 -- start start 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 0x7fd89014f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 0x7fd8900ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890137c80 con 0x7fd8901592a0 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89014fae0 con 0x7fd89006c2d0 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.463+0000 7fd896600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd890137ed0 con 0x7fd89006f970 2024-09-13T17:11:43.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54226/0 (socket says 172.21.15.28:54226) 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88fe00640 1 -- 172.21.15.28:0/4124224923 learned_addr learned my addr 172.21.15.28:0/4124224923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 0x7fd8900ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88f400640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 0x7fd89014f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 -- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 msgr2=0x7fd89014f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 0x7fd89014f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 -- 172.21.15.28:0/4124224923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 msgr2=0x7fd8900acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd887e00640 1 -- 172.21.15.28:0/4124224923 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878009400 con 0x7fd89006f970 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88fe00640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:43.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.464+0000 7fd88f400640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 0x7fd89014f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd887e00640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 0x7fd8900ac850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd87800e900 tx=0x7fd87800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87800ca00 con 0x7fd89006f970 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87800cb60 con 0x7fd89006f970 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878013470 con 0x7fd89006f970 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd888009a70 con 0x7fd89006f970 2024-09-13T17:11:43.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.465+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890138490 con 0x7fd89006f970 2024-09-13T17:11:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.466+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8900b3740 con 0x7fd89006f970 2024-09-13T17:11:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.467+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd8780135d0 con 0x7fd89006f970 2024-09-13T17:11:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.468+0000 7fd887400640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.468+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd878089590 con 0x7fd89006f970 2024-09-13T17:11:43.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.469+0000 7fd88f400640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:43.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.469+0000 7fd88f400640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd880005880 tx=0x7fd880009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:43.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.474+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd878054810 con 0x7fd89006f970 2024-09-13T17:11:43.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.687+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd890138010 con 0x7fd89006f970 2024-09-13T17:11:43.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.688+0000 7fd887400640 1 -- 172.21.15.28:0/4124224923 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fd890138010 con 0x7fd89006f970 2024-09-13T17:11:43.690 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:43.690 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":39,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:41.068760+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":11,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:43.690 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 39 2024-09-13T17:11:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 msgr2=0x7fd86c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd880005880 tx=0x7fd880009040 comp rx=0 tx=0).stop 2024-09-13T17:11:43.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 msgr2=0x7fd8900ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 0x7fd8900ac850 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd87800e900 tx=0x7fd87800edc0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 shutdown_connections 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8901592a0 0x7fd8900acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89006f970 0x7fd8900ac850 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 --2- 172.21.15.28:0/4124224923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd89006c2d0 0x7fd89014f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 >> 172.21.15.28:0/4124224923 conn(0x7fd890069e80 msgr2=0x7fd890159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 shutdown_connections 2024-09-13T17:11:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:43.690+0000 7fd896600640 1 -- 172.21.15.28:0/4124224923 wait complete. 2024-09-13T17:11:43.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok flush_path / 2024-09-13T17:11:43.871 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:11:43.872 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:11:43.872 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:11:43.884 INFO:tasks.cephfs.test_scrub_checks:command 'flush_path /' got response code '0' and stdout '{ "return_code": 0 } ' 2024-09-13T17:11:43.884 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:11:43.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:43.972 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:44.064 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:11:44.064 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:44.064 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:44.101 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:44.101 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:44.178 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.374+0000 7f1f9c800640 1 Processor -- start 2024-09-13T17:11:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.374+0000 7f1f9c800640 1 -- start start 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9c800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f940cabc0 0x7f1f9406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9c800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9415e100 con 0x7f1f94070cc0 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9c800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f9415e3e0 con 0x7f1f9415a540 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9c800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f940adb00 con 0x7f1f940cabc0 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1f94070cc0 0x7f1f9406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39186/0 (socket says 172.21.15.28:39186) 2024-09-13T17:11:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f9a800640 1 -- 172.21.15.28:0/2166687706 learned_addr learned my addr 172.21.15.28:0/2166687706 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f99400640 1 --2- 172.21.15.28:0/2166687706 >> v2:172.21.15.189:3300/0 conn(0x7f1f940cabc0 0x7f1f9406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.376+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401386198 0 0) 0x7f1f9415e100 con 0x7f1f94070cc0 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f7c003520 con 0x7f1f94070cc0 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679737754 0 0) 0x7f1f9415e3e0 con 0x7f1f9415a540 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f9415e100 con 0x7f1f9415a540 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f99400640 1 -- 172.21.15.28:0/2166687706 >> v1:172.21.15.189:6789/0 conn(0x7f1f9415a540 legacy=0x7f1f9415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f99400640 1 -- 172.21.15.28:0/2166687706 >> v1:172.21.15.28:6789/0 conn(0x7f1f94070cc0 legacy=0x7f1f9406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.377+0000 7f1f99400640 1 -- 172.21.15.28:0/2166687706 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9415ef80 con 0x7f1f940cabc0 2024-09-13T17:11:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.378+0000 7f1f99400640 1 --2- 172.21.15.28:0/2166687706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cabc0 0x7f1f9406c430 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097a0 tx=0x7f1f8402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54d8ccd3f77d0ee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.378+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84032440 con 0x7f1f940cabc0 2024-09-13T17:11:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.378+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f840325a0 con 0x7f1f940cabc0 2024-09-13T17:11:44.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.378+0000 7f1f98a00640 1 -- 172.21.15.28:0/2166687706 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f84032930 con 0x7f1f940cabc0 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 -- 172.21.15.28:0/2166687706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cabc0 msgr2=0x7f1f9406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 --2- 172.21.15.28:0/2166687706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cabc0 0x7f1f9406c430 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f840097a0 tx=0x7f1f8402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 -- 172.21.15.28:0/2166687706 shutdown_connections 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 --2- 172.21.15.28:0/2166687706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f940cabc0 0x7f1f9406c430 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 -- 172.21.15.28:0/2166687706 >> 172.21.15.28:0/2166687706 conn(0x7f1f94069e80 msgr2=0x7f1f9406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 -- 172.21.15.28:0/2166687706 shutdown_connections 2024-09-13T17:11:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.379+0000 7f1f9c800640 1 -- 172.21.15.28:0/2166687706 wait complete. 2024-09-13T17:11:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.381+0000 7f1f9c800640 1 Processor -- start 2024-09-13T17:11:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.381+0000 7f1f9c800640 1 -- start start 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f94070cc0 0x7f1f94156990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 0x7f1f940ce760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f941f56d0 con 0x7f1f940cabc0 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f941f5920 con 0x7f1f94070cc0 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f941f5c00 con 0x7f1f9415a540 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50406/0 (socket says 172.21.15.28:50406) 2024-09-13T17:11:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 -- 172.21.15.28:0/1930394906 learned_addr learned my addr 172.21.15.28:0/1930394906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99e00640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f94070cc0 0x7f1f94156990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9a800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 0x7f1f940ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 -- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 msgr2=0x7f1f940ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 0x7f1f940ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 -- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f94070cc0 msgr2=0x7f1f94156990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f94070cc0 0x7f1f94156990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f99400640 1 -- 172.21.15.28:0/1930394906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f84009400 con 0x7f1f940cabc0 2024-09-13T17:11:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.382+0000 7f1f9a800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 0x7f1f940ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f99400640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1f84035040 tx=0x7f1f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8403d8f0 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f8403da70 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f8403de20 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f941f5ee0 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.383+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f941f64a0 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.385+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f940aff30 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.385+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1f84039040 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.387+0000 7f1f8e000640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 0x7f1f7806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.387+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f840a9d80 con 0x7f1f940cabc0 2024-09-13T17:11:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.387+0000 7f1f99e00640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 0x7f1f7806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.388+0000 7f1f99e00640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 0x7f1f7806d0f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1f880011e0 tx=0x7f1f88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.391+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1f940aff30 con 0x7f1f940cabc0 2024-09-13T17:11:44.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.576+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f940aff30 con 0x7f1f940cabc0 2024-09-13T17:11:44.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.577+0000 7f1f8e000640 1 -- 172.21.15.28:0/1930394906 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16513 (secure 0 0 0) 0x7f1f940aff30 con 0x7f1f940cabc0 2024-09-13T17:11:44.580 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:44.580 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:09.922940+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.579+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 msgr2=0x7f1f7806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 0x7f1f7806d0f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f1f880011e0 tx=0x7f1f88009040 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 msgr2=0x7f1f9415ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1f84035040 tx=0x7f1f840046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 shutdown_connections 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f7806ac30 0x7f1f7806d0f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f9415a540 0x7f1f940ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f940cabc0 0x7f1f9415ceb0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 --2- 172.21.15.28:0/1930394906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f94070cc0 0x7f1f94156990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 >> 172.21.15.28:0/1930394906 conn(0x7f1f94069e80 msgr2=0x7f1f94155c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 shutdown_connections 2024-09-13T17:11:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.580+0000 7f1f9c800640 1 -- 172.21.15.28:0/1930394906 wait complete. 2024-09-13T17:11:44.598 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:11:07.454143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:11:07.927877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:11:44.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:11:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.756+0000 7f8629c00640 1 Processor -- start 2024-09-13T17:11:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.757+0000 7f8629c00640 1 -- start start 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8629c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86240cde30 0x7f8624071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8629c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f862415af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8629c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8624159e50 con 0x7f8624071f60 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8629c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86240ce210 con 0x7f86240cde30 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8629c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862415a130 con 0x7f86240712d0 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8623e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86240cde30 0x7f8624071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8623e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86240cde30 0x7f8624071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50420/0 (socket says 172.21.15.28:50420) 2024-09-13T17:11:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.758+0000 7f8623e00640 1 -- 172.21.15.28:0/3656713842 learned_addr learned my addr 172.21.15.28:0/3656713842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8629200640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f862415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8623e00640 1 -- 172.21.15.28:0/3656713842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 msgr2=0x7f862415af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8623e00640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f862415af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8623e00640 1 -- 172.21.15.28:0/3656713842 >> v1:172.21.15.189:6789/0 conn(0x7f8624071f60 legacy=0x7f8624070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:44.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8623e00640 1 -- 172.21.15.28:0/3656713842 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862415a410 con 0x7f86240cde30 2024-09-13T17:11:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.759+0000 7f8629200640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f862415af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8623e00640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f8624071a20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f861400bb10 tx=0x7f861402c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f92582143686949 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8623400640 1 -- 172.21.15.28:0/3656713842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614032790 con 0x7f86240cde30 2024-09-13T17:11:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8623400640 1 -- 172.21.15.28:0/3656713842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86140328f0 con 0x7f86240cde30 2024-09-13T17:11:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8623400640 1 -- 172.21.15.28:0/3656713842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8614032c80 con 0x7f86240cde30 2024-09-13T17:11:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8629c00640 1 -- 172.21.15.28:0/3656713842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 msgr2=0x7f8624071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.760+0000 7f8629c00640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f8624071a20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f861400bb10 tx=0x7f861402c8d0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 -- 172.21.15.28:0/3656713842 shutdown_connections 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f862415af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 --2- 172.21.15.28:0/3656713842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f8624071a20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 -- 172.21.15.28:0/3656713842 >> 172.21.15.28:0/3656713842 conn(0x7f8624069e80 msgr2=0x7f862406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 -- 172.21.15.28:0/3656713842 shutdown_connections 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.761+0000 7f8629c00640 1 -- 172.21.15.28:0/3656713842 wait complete. 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.762+0000 7f8629c00640 1 Processor -- start 2024-09-13T17:11:44.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.762+0000 7f8629c00640 1 -- start start 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f86241f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8624071f60 0x7f86240af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b0380 con 0x7f86240cde30 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240afdc0 con 0x7f8624071f60 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.763+0000 7f8629c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b00a0 con 0x7f86240712d0 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8623e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f86241f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50432/0 (socket says 172.21.15.28:50432) 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 -- 172.21.15.28:0/1534603094 learned_addr learned my addr 172.21.15.28:0/1534603094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f861b400640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8624071f60 0x7f86240af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 -- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 msgr2=0x7f86241f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f86241f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 -- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8624071f60 msgr2=0x7f86240af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8624071f60 0x7f86240af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 -- 172.21.15.28:0/1534603094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861400b770 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.764+0000 7f8629200640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f861c00ce70 tx=0x7f861c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.765+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c003b90 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.765+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c003d10 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.765+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c004260 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.765+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862414d4c0 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.765+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8624137970 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.766+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862414d320 con 0x7f86240cde30 2024-09-13T17:11:44.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.767+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f861c0043c0 con 0x7f86240cde30 2024-09-13T17:11:44.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.767+0000 7f8620c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 0x7f860006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:44.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.768+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f861c089bb0 con 0x7f86240cde30 2024-09-13T17:11:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.768+0000 7f8623e00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 0x7f860006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.769+0000 7f8623e00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 0x7f860006d0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8614037000 tx=0x7f8614032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:44.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.772+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f862414d320 con 0x7f86240cde30 2024-09-13T17:11:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.953+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f862414d9e0 con 0x7f86240cde30 2024-09-13T17:11:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.954+0000 7f8620c00640 1 -- 172.21.15.28:0/1534603094 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f862414d9e0 con 0x7f86240cde30 2024-09-13T17:11:44.956 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:11:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 msgr2=0x7f860006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 0x7f860006d0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8614037000 tx=0x7f8614032000 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 msgr2=0x7f86240ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f861c00ce70 tx=0x7f861c00a710 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 shutdown_connections 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f860006ac00 0x7f860006d0c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86240cde30 0x7f86240ac850 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8624071f60 0x7f86240af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 --2- 172.21.15.28:0/1534603094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86240712d0 0x7f86241f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 >> 172.21.15.28:0/1534603094 conn(0x7f8624069e80 msgr2=0x7f86241560d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.956+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 shutdown_connections 2024-09-13T17:11:44.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:44.957+0000 7f8629c00640 1 -- 172.21.15.28:0/1534603094 wait complete. 2024-09-13T17:11:44.976 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:11:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.134+0000 7fda55400640 1 Processor -- start 2024-09-13T17:11:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.134+0000 7fda55400640 1 -- start start 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda55400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda5015ca40 0x7fda500cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda55400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 0x7fda500b5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda55400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda500d19b0 con 0x7fda500cf7b0 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda55400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda500cfc40 con 0x7fda5015ca40 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda55400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500d1c90 con 0x7fda500d1400 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda5015ca40 0x7fda500cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda5015ca40 0x7fda500cf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50438/0 (socket says 172.21.15.28:50438) 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 -- 172.21.15.28:0/4193387936 learned_addr learned my addr 172.21.15.28:0/4193387936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4f400640 1 --1- 172.21.15.28:0/4193387936 >> v1:172.21.15.28:6789/0 conn(0x7fda500cf7b0 0x7fda500d0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda54a00640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 0x7fda500b5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 -- 172.21.15.28:0/4193387936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 msgr2=0x7fda500b5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 0x7fda500b5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 -- 172.21.15.28:0/4193387936 >> v1:172.21.15.28:6789/0 conn(0x7fda500cf7b0 legacy=0x7fda500d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda4fe00640 1 -- 172.21.15.28:0/4193387936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda500d1f70 con 0x7fda5015ca40 2024-09-13T17:11:45.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.136+0000 7fda54a00640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 0x7fda500b5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.137+0000 7fda4fe00640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda5015ca40 0x7fda500cf270 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fda400098a0 tx=0x7fda4002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e993dd482f0606f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.138+0000 7fda4ea00640 1 -- 172.21.15.28:0/4193387936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40032440 con 0x7fda5015ca40 2024-09-13T17:11:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.138+0000 7fda4ea00640 1 -- 172.21.15.28:0/4193387936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda400325a0 con 0x7fda5015ca40 2024-09-13T17:11:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.138+0000 7fda4ea00640 1 -- 172.21.15.28:0/4193387936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40032930 con 0x7fda5015ca40 2024-09-13T17:11:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.138+0000 7fda55400640 1 -- 172.21.15.28:0/4193387936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda5015ca40 msgr2=0x7fda500cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.138+0000 7fda55400640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda5015ca40 0x7fda500cf270 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fda400098a0 tx=0x7fda4002a2a0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 -- 172.21.15.28:0/4193387936 shutdown_connections 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500d1400 0x7fda500b5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 --2- 172.21.15.28:0/4193387936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda5015ca40 0x7fda500cf270 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 -- 172.21.15.28:0/4193387936 >> 172.21.15.28:0/4193387936 conn(0x7fda50070ad0 msgr2=0x7fda500b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 -- 172.21.15.28:0/4193387936 shutdown_connections 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.139+0000 7fda55400640 1 -- 172.21.15.28:0/4193387936 wait complete. 2024-09-13T17:11:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.140+0000 7fda55400640 1 Processor -- start 2024-09-13T17:11:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.140+0000 7fda55400640 1 -- start start 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500cf7b0 0x7fda50149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 0x7fda500cab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5015fa50 con 0x7fda500d1400 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda501587e0 con 0x7fda5015f5d0 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.141+0000 7fda55400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda50159640 con 0x7fda500cf7b0 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50442/0 (socket says 172.21.15.28:50442) 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 -- 172.21.15.28:0/2535766140 learned_addr learned my addr 172.21.15.28:0/2535766140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:45.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4fe00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500cf7b0 0x7fda50149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda54a00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 0x7fda500cab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 -- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500cf7b0 msgr2=0x7fda50149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500cf7b0 0x7fda50149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 -- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 msgr2=0x7fda500cab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 0x7fda500cab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 -- 172.21.15.28:0/2535766140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda400094d0 con 0x7fda500d1400 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda54a00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 0x7fda500cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.142+0000 7fda4f400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c500 tx=0x7fda3c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.143+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c007500 con 0x7fda500d1400 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.143+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3c0057a0 con 0x7fda500d1400 2024-09-13T17:11:45.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.143+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3c007660 con 0x7fda500d1400 2024-09-13T17:11:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.143+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda50158a30 con 0x7fda500d1400 2024-09-13T17:11:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.143+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda50158ff0 con 0x7fda500d1400 2024-09-13T17:11:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.144+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda500b5ea0 con 0x7fda500d1400 2024-09-13T17:11:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.145+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fda3c005bd0 con 0x7fda500d1400 2024-09-13T17:11:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.146+0000 7fda47e00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 0x7fda3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.146+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda3c088900 con 0x7fda500d1400 2024-09-13T17:11:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.147+0000 7fda4fe00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 0x7fda3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.147+0000 7fda4fe00640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 0x7fda3006d0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda40034000 tx=0x7fda4002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.150+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fda3c053b80 con 0x7fda500d1400 2024-09-13T17:11:45.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.329+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fda500d3420 con 0x7fda500d1400 2024-09-13T17:11:45.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.330+0000 7fda47e00640 1 -- 172.21.15.28:0/2535766140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7fda3c017020 con 0x7fda500d1400 2024-09-13T17:11:45.332 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:11:45.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 msgr2=0x7fda3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 0x7fda3006d0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda40034000 tx=0x7fda4002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:45.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 msgr2=0x7fda5015f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c500 tx=0x7fda3c00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 shutdown_connections 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda3006ac00 0x7fda3006d0c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda5015f5d0 0x7fda500cab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda500d1400 0x7fda5015f090 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 --2- 172.21.15.28:0/2535766140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda500cf7b0 0x7fda50149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 >> 172.21.15.28:0/2535766140 conn(0x7fda50070ad0 msgr2=0x7fda500b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 shutdown_connections 2024-09-13T17:11:45.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.332+0000 7fda55400640 1 -- 172.21.15.28:0/2535766140 wait complete. 2024-09-13T17:11:45.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:11:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.506+0000 7f9f98e00640 1 Processor -- start 2024-09-13T17:11:45.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.506+0000 7f9f98e00640 1 -- start start 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f98e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 0x7f9f94070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f98e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f9415ab60 0x7f9f940712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f98e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9406c8b0 con 0x7f9f9406bb40 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f98e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9406c6a0 con 0x7f9f9415ab60 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f98e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9406cb90 con 0x7f9f9415a780 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f92800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f9406bb40 0x7f9f9406bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39208/0 (socket says 172.21.15.28:39208) 2024-09-13T17:11:45.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.507+0000 7f9f92800640 1 -- 172.21.15.28:0/3690494039 learned_addr learned my addr 172.21.15.28:0/3690494039 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:45.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3690494039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 0x7f9f94070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f89400640 1 --2- 172.21.15.28:0/3690494039 >> v2:172.21.15.189:3300/0 conn(0x7f9f9415ab60 0x7f9f940712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91400640 1 -- 172.21.15.28:0/3690494039 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1986897360 0 0) 0x7f9f9406c8b0 con 0x7f9f9406bb40 2024-09-13T17:11:45.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91400640 1 -- 172.21.15.28:0/3690494039 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f78003520 con 0x7f9f9406bb40 2024-09-13T17:11:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91e00640 1 -- 172.21.15.28:0/3690494039 >> v2:172.21.15.189:3300/0 conn(0x7f9f9415ab60 msgr2=0x7f9f940712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3690494039 >> v2:172.21.15.189:3300/0 conn(0x7f9f9415ab60 0x7f9f940712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91e00640 1 -- 172.21.15.28:0/3690494039 >> v1:172.21.15.28:6789/0 conn(0x7f9f9406bb40 legacy=0x7f9f9406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.508+0000 7f9f91e00640 1 -- 172.21.15.28:0/3690494039 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940cc980 con 0x7f9f9415a780 2024-09-13T17:11:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.509+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3690494039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 0x7f9f94070d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f840097a0 tx=0x7f9f8402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b0a35f46a921057 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f91400640 1 -- 172.21.15.28:0/3690494039 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84032440 con 0x7f9f9415a780 2024-09-13T17:11:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f91400640 1 -- 172.21.15.28:0/3690494039 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f840325a0 con 0x7f9f9415a780 2024-09-13T17:11:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f91400640 1 -- 172.21.15.28:0/3690494039 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f84032930 con 0x7f9f9415a780 2024-09-13T17:11:45.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 -- 172.21.15.28:0/3690494039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 msgr2=0x7f9f94070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3690494039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 0x7f9f94070d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9f840097a0 tx=0x7f9f8402a180 comp rx=0 tx=0).stop 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 -- 172.21.15.28:0/3690494039 shutdown_connections 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3690494039 >> v2:172.21.15.189:3300/0 conn(0x7f9f9415ab60 0x7f9f940712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3690494039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415a780 0x7f9f94070d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 -- 172.21.15.28:0/3690494039 >> 172.21.15.28:0/3690494039 conn(0x7f9f940b0270 msgr2=0x7f9f940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 -- 172.21.15.28:0/3690494039 shutdown_connections 2024-09-13T17:11:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.510+0000 7f9f98e00640 1 -- 172.21.15.28:0/3690494039 wait complete. 2024-09-13T17:11:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.512+0000 7f9f98e00640 1 Processor -- start 2024-09-13T17:11:45.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.512+0000 7f9f98e00640 1 -- start start 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 0x7f9f940c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 0x7f9f940c99d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f940c44f0 con 0x7f9f9406bb40 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94151fb0 con 0x7f9f9415a780 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f98e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f940c83f0 con 0x7f9f9415ab60 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50456/0 (socket says 172.21.15.28:50456) 2024-09-13T17:11:45.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f91e00640 1 -- 172.21.15.28:0/3256521344 learned_addr learned my addr 172.21.15.28:0/3256521344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.513+0000 7f9f89400640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 0x7f9f940c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f92800640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 0x7f9f940c99d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 -- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 msgr2=0x7f9f940c99d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 0x7f9f940c99d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 -- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 msgr2=0x7f9f940c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 0x7f9f940c7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 -- 172.21.15.28:0/3256521344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f84009400 con 0x7f9f9406bb40 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f92800640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 0x7f9f940c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f89400640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 0x7f9f940c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:45.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f91e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9f840098c0 tx=0x7f9f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8403d8f0 con 0x7f9f9406bb40 2024-09-13T17:11:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8403da70 con 0x7f9f9406bb40 2024-09-13T17:11:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8403de20 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.514+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f940c34e0 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.515+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f940c3a10 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.516+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9406c8b0 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.516+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9f84039040 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.517+0000 7f9f8aa00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 0x7f9f7006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.517+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f840a98c0 con 0x7f9f9406bb40 2024-09-13T17:11:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.518+0000 7f9f89400640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 0x7f9f7006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.518+0000 7f9f89400640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 0x7f9f7006d110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406b420 tx=0x7f9f7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.521+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9f84074d30 con 0x7f9f9406bb40 2024-09-13T17:11:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.701+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9f94159a50 con 0x7f9f9406bb40 2024-09-13T17:11:45.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.702+0000 7f9f8aa00640 1 -- 172.21.15.28:0/3256521344 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9f84078350 con 0x7f9f9406bb40 2024-09-13T17:11:45.704 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 msgr2=0x7f9f7006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 0x7f9f7006d110 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9f9406b420 tx=0x7f9f7c008040 comp rx=0 tx=0).stop 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 msgr2=0x7f9f940c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f9f840098c0 tx=0x7f9f840046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 shutdown_connections 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f7006ac50 0x7f9f7006d110 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f9415ab60 0x7f9f940c99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f9415a780 0x7f9f940c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 --2- 172.21.15.28:0/3256521344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f9406bb40 0x7f9f940c7970 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 >> 172.21.15.28:0/3256521344 conn(0x7f9f940b0270 msgr2=0x7f9f940cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 shutdown_connections 2024-09-13T17:11:45.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.704+0000 7f9f98e00640 1 -- 172.21.15.28:0/3256521344 wait complete. 2024-09-13T17:11:45.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:11:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.877+0000 7ff1a4800640 1 Processor -- start 2024-09-13T17:11:45.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.877+0000 7ff1a4800640 1 -- start start 2024-09-13T17:11:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a4800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff19c159770 0x7ff19c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a4800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff19c15cd80 0x7ff19c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a4800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c06f920 con 0x7ff19c06f3f0 2024-09-13T17:11:45.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a4800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c15b290 con 0x7ff19c159770 2024-09-13T17:11:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a4800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c15b570 con 0x7ff19c15cd80 2024-09-13T17:11:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff19c159770 0x7ff19c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.878+0000 7ff1a1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff19c06f3f0 0x7ff19c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39214/0 (socket says 172.21.15.28:39214) 2024-09-13T17:11:45.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a1400640 1 -- 172.21.15.28:0/4103823109 learned_addr learned my addr 172.21.15.28:0/4103823109 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a2800640 1 --2- 172.21.15.28:0/4103823109 >> v2:172.21.15.189:3300/0 conn(0x7ff19c15cd80 0x7ff19c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a1e00640 1 -- 172.21.15.28:0/4103823109 >> v2:172.21.15.189:3300/0 conn(0x7ff19c15cd80 msgr2=0x7ff19c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a1e00640 1 --2- 172.21.15.28:0/4103823109 >> v2:172.21.15.189:3300/0 conn(0x7ff19c15cd80 0x7ff19c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a1e00640 1 -- 172.21.15.28:0/4103823109 >> v1:172.21.15.28:6789/0 conn(0x7ff19c06f3f0 legacy=0x7ff19c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a1e00640 1 -- 172.21.15.28:0/4103823109 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c15b850 con 0x7ff19c159770 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.879+0000 7ff1a0a00640 1 -- 172.21.15.28:0/4103823109 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921350398 0 0) 0x7ff19c06f920 con 0x7ff19c06f3f0 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.880+0000 7ff1a1e00640 1 --2- 172.21.15.28:0/4103823109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c159770 0x7ff19c06fd20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff19c06f380 tx=0x7ff19002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90064e3bdbddb09f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.880+0000 7ff1a0a00640 1 -- 172.21.15.28:0/4103823109 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002ae60 con 0x7ff19c159770 2024-09-13T17:11:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.880+0000 7ff1a0a00640 1 -- 172.21.15.28:0/4103823109 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190032440 con 0x7ff19c159770 2024-09-13T17:11:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.880+0000 7ff1a0a00640 1 -- 172.21.15.28:0/4103823109 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1900327d0 con 0x7ff19c159770 2024-09-13T17:11:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 -- 172.21.15.28:0/4103823109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c159770 msgr2=0x7ff19c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 --2- 172.21.15.28:0/4103823109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c159770 0x7ff19c06fd20 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7ff19c06f380 tx=0x7ff19002a110 comp rx=0 tx=0).stop 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 -- 172.21.15.28:0/4103823109 shutdown_connections 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 --2- 172.21.15.28:0/4103823109 >> v2:172.21.15.189:3300/0 conn(0x7ff19c15cd80 0x7ff19c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 --2- 172.21.15.28:0/4103823109 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c159770 0x7ff19c06fd20 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 -- 172.21.15.28:0/4103823109 >> 172.21.15.28:0/4103823109 conn(0x7ff19c0b0270 msgr2=0x7ff19c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 -- 172.21.15.28:0/4103823109 shutdown_connections 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.881+0000 7ff1a4800640 1 -- 172.21.15.28:0/4103823109 wait complete. 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.882+0000 7ff1a4800640 1 Processor -- start 2024-09-13T17:11:45.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.883+0000 7ff1a4800640 1 -- start start 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.883+0000 7ff1a4800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.883+0000 7ff1a4800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19c159770 0x7ff19c151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.883+0000 7ff1a4800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19c15cd80 0x7ff19c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a4800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c150350 con 0x7ff19c06f3f0 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a4800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c146510 con 0x7ff19c15cd80 2024-09-13T17:11:45.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a4800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c152910 con 0x7ff19c159770 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50474/0 (socket says 172.21.15.28:50474) 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 -- 172.21.15.28:0/896967009 learned_addr learned my addr 172.21.15.28:0/896967009 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a2800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19c15cd80 0x7ff19c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1400640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19c159770 0x7ff19c151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 -- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19c159770 msgr2=0x7ff19c151da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19c159770 0x7ff19c151da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 -- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19c15cd80 msgr2=0x7ff19c1522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19c15cd80 0x7ff19c1522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 -- 172.21.15.28:0/896967009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff190009400 con 0x7ff19c06f3f0 2024-09-13T17:11:45.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.884+0000 7ff1a1e00640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff190000c00 tx=0x7ff1900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.885+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190030990 con 0x7ff19c06f3f0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.885+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190030b10 con 0x7ff19c06f3f0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.885+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19003d420 con 0x7ff19c06f3f0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.885+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c152bf0 con 0x7ff19c06f3f0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.885+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c1531b0 con 0x7ff19c06f3f0 2024-09-13T17:11:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.886+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c06fd20 con 0x7ff19c06f3f0 2024-09-13T17:11:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.887+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff190039040 con 0x7ff19c06f3f0 2024-09-13T17:11:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.888+0000 7ff196000640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 0x7ff18006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.888+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1900a9f40 con 0x7ff19c06f3f0 2024-09-13T17:11:45.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.889+0000 7ff1a1400640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 0x7ff18006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:45.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.889+0000 7ff1a1400640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 0x7ff18006d110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff19c1552d0 tx=0x7ff18c007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:45.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:45.892+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff190075100 con 0x7ff19c06f3f0 2024-09-13T17:11:46.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.082+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff19c155a80 con 0x7ff19c06f3f0 2024-09-13T17:11:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.082+0000 7ff196000640 1 -- 172.21.15.28:0/896967009 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff190078720 con 0x7ff19c06f3f0 2024-09-13T17:11:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:11:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 msgr2=0x7ff18006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 0x7ff18006d110 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff19c1552d0 tx=0x7ff18c007c00 comp rx=0 tx=0).stop 2024-09-13T17:11:46.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 msgr2=0x7ff19c145fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff190000c00 tx=0x7ff1900046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 shutdown_connections 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff18006ac50 0x7ff18006d110 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19c15cd80 0x7ff19c1522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19c159770 0x7ff19c151da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 --2- 172.21.15.28:0/896967009 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19c06f3f0 0x7ff19c145fd0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 >> 172.21.15.28:0/896967009 conn(0x7ff19c0b0270 msgr2=0x7ff19c0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 shutdown_connections 2024-09-13T17:11:46.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:46.084+0000 7ff1a4800640 1 -- 172.21.15.28:0/896967009 wait complete. 2024-09-13T17:11:46.103 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:11:46.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:46.103 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:11:47.168 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:11:47.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:11:47.169 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:11:47.206 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:47.206 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:47.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:47.243 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:47.243 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:47.290 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:11:47.290 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:47.290 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:47.332 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:47.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.491+0000 7f381aa00640 1 Processor -- start 2024-09-13T17:11:47.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.491+0000 7f381aa00640 1 -- start start 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f381aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38140b0b50 0x7f38140af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f381aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381406b900 con 0x7f381406b360 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f381aa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381406bbe0 con 0x7f3814154a90 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f381aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381406bec0 con 0x7f38140b0b50 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f3818c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f381406b360 0x7f38140b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39216/0 (socket says 172.21.15.28:39216) 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f3818c00640 1 -- 172.21.15.28:0/1552844225 learned_addr learned my addr 172.21.15.28:0/1552844225 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:47.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f381a000640 1 --2- 172.21.15.28:0/1552844225 >> v2:172.21.15.189:3300/0 conn(0x7f38140b0b50 0x7f38140af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.493+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173166123 0 0) 0x7f381406b900 con 0x7f381406b360 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37fc003520 con 0x7f381406b360 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4213800954 0 0) 0x7f381406bbe0 con 0x7f3814154a90 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f381406b900 con 0x7f3814154a90 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f381a000640 1 -- 172.21.15.28:0/1552844225 >> v1:172.21.15.189:6789/0 conn(0x7f3814154a90 legacy=0x7f381415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f381a000640 1 -- 172.21.15.28:0/1552844225 >> v1:172.21.15.28:6789/0 conn(0x7f381406b360 legacy=0x7f38140b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.494+0000 7f381a000640 1 -- 172.21.15.28:0/1552844225 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381406c1a0 con 0x7f38140b0b50 2024-09-13T17:11:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f381a000640 1 --2- 172.21.15.28:0/1552844225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38140b0b50 0x7f38140af4b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f380c009d90 tx=0x7f380c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b63c24d3743da7c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c007740 con 0x7f38140b0b50 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c02d050 con 0x7f38140b0b50 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f3813e00640 1 -- 172.21.15.28:0/1552844225 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c007aa0 con 0x7f38140b0b50 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f381aa00640 1 -- 172.21.15.28:0/1552844225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38140b0b50 msgr2=0x7f38140af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.495+0000 7f381aa00640 1 --2- 172.21.15.28:0/1552844225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38140b0b50 0x7f38140af4b0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f380c009d90 tx=0x7f380c02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.496+0000 7f381aa00640 1 -- 172.21.15.28:0/1552844225 shutdown_connections 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.496+0000 7f381aa00640 1 --2- 172.21.15.28:0/1552844225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38140b0b50 0x7f38140af4b0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.496+0000 7f381aa00640 1 -- 172.21.15.28:0/1552844225 >> 172.21.15.28:0/1552844225 conn(0x7f3814072e40 msgr2=0x7f3814073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.496+0000 7f381aa00640 1 -- 172.21.15.28:0/1552844225 shutdown_connections 2024-09-13T17:11:47.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.496+0000 7f381aa00640 1 -- 172.21.15.28:0/1552844225 wait complete. 2024-09-13T17:11:47.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.497+0000 7f381aa00640 1 Processor -- start 2024-09-13T17:11:47.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.497+0000 7f381aa00640 1 -- start start 2024-09-13T17:11:47.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38140b0b50 0x7f38140a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3814154a90 0x7f381415ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3814155080 con 0x7f381406b360 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3814155320 con 0x7f3814154a90 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.498+0000 7f381aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3814155600 con 0x7f38140b0b50 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50484/0 (socket says 172.21.15.28:50484) 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 -- 172.21.15.28:0/845611306 learned_addr learned my addr 172.21.15.28:0/845611306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3818c00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38140b0b50 0x7f38140a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f381a000640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3814154a90 0x7f381415ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 -- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38140b0b50 msgr2=0x7f38140a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38140b0b50 0x7f38140a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 -- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3814154a90 msgr2=0x7f381415ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3814154a90 0x7f381415ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 -- 172.21.15.28:0/845611306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380c009a70 con 0x7f381406b360 2024-09-13T17:11:47.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.499+0000 7f3819600640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f380800c430 tx=0x7f380800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.500+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38080052d0 con 0x7f381406b360 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.500+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808005450 con 0x7f381406b360 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.500+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808003830 con 0x7f381406b360 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.500+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38141558e0 con 0x7f381406b360 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.500+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3814155ea0 con 0x7f381406b360 2024-09-13T17:11:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.501+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38140b0440 con 0x7f381406b360 2024-09-13T17:11:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.502+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3808003990 con 0x7f381406b360 2024-09-13T17:11:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.503+0000 7f3811600640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 0x7f37f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.503+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f38080888f0 con 0x7f381406b360 2024-09-13T17:11:47.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.503+0000 7f3818c00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 0x7f37f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.504+0000 7f3818c00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 0x7f37f406d0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f381413ebe0 tx=0x7f3800007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.507+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3808053ae0 con 0x7f381406b360 2024-09-13T17:11:47.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.723+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38141efc20 con 0x7f381406b360 2024-09-13T17:11:47.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.724+0000 7f3811600640 1 -- 172.21.15.28:0/845611306 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f3808017090 con 0x7f381406b360 2024-09-13T17:11:47.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:47.726 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":40,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:45.129903+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":12,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:47.727 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 40 2024-09-13T17:11:47.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 msgr2=0x7f37f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 0x7f37f406d0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f381413ebe0 tx=0x7f3800007d30 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 msgr2=0x7f38141ef6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f380800c430 tx=0x7f380800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 shutdown_connections 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37f406ac00 0x7f37f406d0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3814154a90 0x7f381415ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38140b0b50 0x7f38140a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 --2- 172.21.15.28:0/845611306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381406b360 0x7f38141ef6e0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.726+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 >> 172.21.15.28:0/845611306 conn(0x7f3814072e40 msgr2=0x7f381406fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.727+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 shutdown_connections 2024-09-13T17:11:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.727+0000 7f381aa00640 1 -- 172.21.15.28:0/845611306 wait complete. 2024-09-13T17:11:47.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.897+0000 7f5379c00640 1 Processor -- start 2024-09-13T17:11:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.897+0000 7f5379c00640 1 -- start start 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5379c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5374158b20 0x7f53740afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5379c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f537406c3f0 0x7f537406b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5379c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f537406bcf0 con 0x7f537406b8a0 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5379c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53740b39d0 con 0x7f5374158b20 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5379c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53740b2dc0 con 0x7f537406c3f0 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.898+0000 7f5373e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5374158b20 0x7f53740afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5379200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f537406b8a0 0x7f53740b1740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39218/0 (socket says 172.21.15.28:39218) 2024-09-13T17:11:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5379200640 1 -- 172.21.15.28:0/4140435750 learned_addr learned my addr 172.21.15.28:0/4140435750 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5373400640 1 --2- 172.21.15.28:0/4140435750 >> v2:172.21.15.189:3300/0 conn(0x7f537406c3f0 0x7f537406b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5373e00640 1 -- 172.21.15.28:0/4140435750 >> v2:172.21.15.189:3300/0 conn(0x7f537406c3f0 msgr2=0x7f537406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5373e00640 1 --2- 172.21.15.28:0/4140435750 >> v2:172.21.15.189:3300/0 conn(0x7f537406c3f0 0x7f537406b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5373e00640 1 -- 172.21.15.28:0/4140435750 >> v1:172.21.15.28:6789/0 conn(0x7f537406b8a0 legacy=0x7f53740b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5373e00640 1 -- 172.21.15.28:0/4140435750 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53740b30a0 con 0x7f5374158b20 2024-09-13T17:11:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.899+0000 7f5372a00640 1 -- 172.21.15.28:0/4140435750 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837891745 0 0) 0x7f536c001490 con 0x7f537406b8a0 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.900+0000 7f5373e00640 1 --2- 172.21.15.28:0/4140435750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f53740afbd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f53740b0110 tx=0x7f536402c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8de08e66ed8a3c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.900+0000 7f5372a00640 1 -- 172.21.15.28:0/4140435750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364034640 con 0x7f5374158b20 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.900+0000 7f5372a00640 1 -- 172.21.15.28:0/4140435750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53640347a0 con 0x7f5374158b20 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.900+0000 7f5372a00640 1 -- 172.21.15.28:0/4140435750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5364034b30 con 0x7f5374158b20 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 -- 172.21.15.28:0/4140435750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 msgr2=0x7f53740afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 --2- 172.21.15.28:0/4140435750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f53740afbd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f53740b0110 tx=0x7f536402c330 comp rx=0 tx=0).stop 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 -- 172.21.15.28:0/4140435750 shutdown_connections 2024-09-13T17:11:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 --2- 172.21.15.28:0/4140435750 >> v2:172.21.15.189:3300/0 conn(0x7f537406c3f0 0x7f537406b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 --2- 172.21.15.28:0/4140435750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f53740afbd0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 -- 172.21.15.28:0/4140435750 >> 172.21.15.28:0/4140435750 conn(0x7f5374072e40 msgr2=0x7f5374073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 -- 172.21.15.28:0/4140435750 shutdown_connections 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.901+0000 7f5379c00640 1 -- 172.21.15.28:0/4140435750 wait complete. 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.903+0000 7f5379c00640 1 Processor -- start 2024-09-13T17:11:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.903+0000 7f5379c00640 1 -- start start 2024-09-13T17:11:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 0x7f53741e1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f537406c3f0 0x7f53741f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53741e2da0 con 0x7f5374158b20 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53741e2a90 con 0x7f537406b8a0 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53741f6380 con 0x7f537406c3f0 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50510/0 (socket says 172.21.15.28:50510) 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 -- 172.21.15.28:0/3521375143 learned_addr learned my addr 172.21.15.28:0/3521375143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5373e00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 0x7f53741e1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5373400640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f537406c3f0 0x7f53741f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 -- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f537406c3f0 msgr2=0x7f53741f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f537406c3f0 0x7f53741f5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 -- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 msgr2=0x7f53741e1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 0x7f53741e1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5379200640 1 -- 172.21.15.28:0/3521375143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5364009400 con 0x7f5374158b20 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.904+0000 7f5373e00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 0x7f53741e1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:47.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f5379200640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f536c002230 tx=0x7f536c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00ce30 con 0x7f5374158b20 2024-09-13T17:11:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c00a640 con 0x7f5374158b20 2024-09-13T17:11:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53741f65b0 con 0x7f5374158b20 2024-09-13T17:11:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f536c00a9f0 con 0x7f5374158b20 2024-09-13T17:11:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.905+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53741f6b70 con 0x7f5374158b20 2024-09-13T17:11:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.906+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53741e2ee0 con 0x7f5374158b20 2024-09-13T17:11:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.907+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f536c00ab50 con 0x7f5374158b20 2024-09-13T17:11:47.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.908+0000 7f536be00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 0x7f535406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:47.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.908+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f536c0888e0 con 0x7f5374158b20 2024-09-13T17:11:47.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.908+0000 7f5373e00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 0x7f535406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:47.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.909+0000 7f5373e00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 0x7f535406d0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5364036000 tx=0x7f5364030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:47.913+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f53741e2ee0 con 0x7f5374158b20 2024-09-13T17:11:48.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.095+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f537406bcf0 con 0x7f5374158b20 2024-09-13T17:11:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.096+0000 7f536be00640 1 -- 172.21.15.28:0/3521375143 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16513 (secure 0 0 0) 0x7f537406bcf0 con 0x7f5374158b20 2024-09-13T17:11:48.098 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:48.098 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:09.922940+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 msgr2=0x7f535406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 0x7f535406d0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5364036000 tx=0x7f5364030000 comp rx=0 tx=0).stop 2024-09-13T17:11:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 msgr2=0x7f537414f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f536c002230 tx=0x7f536c00cbe0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 shutdown_connections 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f535406ac00 0x7f535406d0c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5374158b20 0x7f537414f320 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f537406c3f0 0x7f53741f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 --2- 172.21.15.28:0/3521375143 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f537406b8a0 0x7f53741e1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 >> 172.21.15.28:0/3521375143 conn(0x7f5374072e40 msgr2=0x7f5374158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 shutdown_connections 2024-09-13T17:11:48.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.098+0000 7f5379c00640 1 -- 172.21.15.28:0/3521375143 wait complete. 2024-09-13T17:11:48.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:11:07.454143+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:11:07.927877+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:11:48.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:11:48.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.273+0000 7fd6ef800640 1 Processor -- start 2024-09-13T17:11:48.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.273+0000 7fd6ef800640 1 -- start start 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6ef800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 0x7fd6e8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6ef800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd6e806c250 0x7fd6e80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6ef800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e806c8b0 con 0x7fd6e815ab60 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6ef800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e806c6a0 con 0x7fd6e806c250 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6ef800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e806cb90 con 0x7fd6e815a780 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6e7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6e815ab60 0x7fd6e806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39230/0 (socket says 172.21.15.28:39230) 2024-09-13T17:11:48.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.274+0000 7fd6e7e00640 1 -- 172.21.15.28:0/1760639178 learned_addr learned my addr 172.21.15.28:0/1760639178 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ece00640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 0x7fd6e8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ed800640 1 --2- 172.21.15.28:0/1760639178 >> v2:172.21.15.189:3300/0 conn(0x7fd6e806c250 0x7fd6e80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6e7400640 1 -- 172.21.15.28:0/1760639178 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478409506 0 0) 0x7fd6e806c8b0 con 0x7fd6e815ab60 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6e7400640 1 -- 172.21.15.28:0/1760639178 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d0003520 con 0x7fd6e815ab60 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ed800640 1 -- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 msgr2=0x7fd6e8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ed800640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 0x7fd6e8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ed800640 1 -- 172.21.15.28:0/1760639178 >> v1:172.21.15.28:6789/0 conn(0x7fd6e815ab60 legacy=0x7fd6e806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ed800640 1 -- 172.21.15.28:0/1760639178 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80cc980 con 0x7fd6e806c250 2024-09-13T17:11:48.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.275+0000 7fd6ece00640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 0x7fd6e8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.276+0000 7fd6ed800640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e80712c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009d90 tx=0x7fd6e002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d9bbecfa1ae43d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6e7400640 1 -- 172.21.15.28:0/1760639178 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0007740 con 0x7fd6e806c250 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6e7400640 1 -- 172.21.15.28:0/1760639178 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e002d050 con 0x7fd6e806c250 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6e7400640 1 -- 172.21.15.28:0/1760639178 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e0007ac0 con 0x7fd6e806c250 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 -- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 msgr2=0x7fd6e80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e80712c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd6e0009d90 tx=0x7fd6e002a660 comp rx=0 tx=0).stop 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 -- 172.21.15.28:0/1760639178 shutdown_connections 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e80712c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1760639178 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815a780 0x7fd6e8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 -- 172.21.15.28:0/1760639178 >> 172.21.15.28:0/1760639178 conn(0x7fd6e80b0270 msgr2=0x7fd6e80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 -- 172.21.15.28:0/1760639178 shutdown_connections 2024-09-13T17:11:48.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.277+0000 7fd6ef800640 1 -- 172.21.15.28:0/1760639178 wait complete. 2024-09-13T17:11:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.279+0000 7fd6ef800640 1 Processor -- start 2024-09-13T17:11:48.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.279+0000 7fd6ef800640 1 -- start start 2024-09-13T17:11:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e814f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815ab60 0x7fd6e814d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e813d410 con 0x7fd6e815a780 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e814e3d0 con 0x7fd6e806c250 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ef800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e814e6b0 con 0x7fd6e815ab60 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50518/0 (socket says 172.21.15.28:50518) 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6e7e00640 1 -- 172.21.15.28:0/1778667351 learned_addr learned my addr 172.21.15.28:0/1778667351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.280+0000 7fd6ece00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e814f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6ed800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815ab60 0x7fd6e814d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 -- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815ab60 msgr2=0x7fd6e814d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815ab60 0x7fd6e814d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 -- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 msgr2=0x7fd6e814f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e814f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 -- 172.21.15.28:0/1778667351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009400 con 0x7fd6e815a780 2024-09-13T17:11:48.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e7e00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c3e0 tx=0x7fd6d400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4005ce0 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4005e40 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40072d0 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.281+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e0009a70 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.282+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e814ec70 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.283+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e806c070 con 0x7fd6e815a780 2024-09-13T17:11:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.283+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6d40074f0 con 0x7fd6e815a780 2024-09-13T17:11:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.284+0000 7fd6e4c00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 0x7fd6c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.284+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d40889f0 con 0x7fd6e815a780 2024-09-13T17:11:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.285+0000 7fd6ece00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 0x7fd6c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.286+0000 7fd6ece00640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 0x7fd6c806d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009cd0 tx=0x7fd6dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.289+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6d4053b50 con 0x7fd6e815a780 2024-09-13T17:11:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.465+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6e806c8b0 con 0x7fd6e815a780 2024-09-13T17:11:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.466+0000 7fd6e4c00640 1 -- 172.21.15.28:0/1778667351 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd6d4017020 con 0x7fd6e815a780 2024-09-13T17:11:48.468 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:11:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 msgr2=0x7fd6c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 0x7fd6c806d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc009cd0 tx=0x7fd6dc009190 comp rx=0 tx=0).stop 2024-09-13T17:11:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 msgr2=0x7fd6e813ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fd6d400c3e0 tx=0x7fd6d400c8a0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 shutdown_connections 2024-09-13T17:11:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c806ac00 0x7fd6c806d0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e815ab60 0x7fd6e814d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e815a780 0x7fd6e813ced0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 --2- 172.21.15.28:0/1778667351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e806c250 0x7fd6e814f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 >> 172.21.15.28:0/1778667351 conn(0x7fd6e80b0270 msgr2=0x7fd6e80ced30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 shutdown_connections 2024-09-13T17:11:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.468+0000 7fd6ef800640 1 -- 172.21.15.28:0/1778667351 wait complete. 2024-09-13T17:11:48.486 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:11:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.643+0000 7f8d49e00640 1 Processor -- start 2024-09-13T17:11:48.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.643+0000 7f8d49e00640 1 -- start start 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.644+0000 7f8d49e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d44159770 0x7f8d4406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4415cd80 0x7f8d44070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d49e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d4406f920 con 0x7f8d4406f3f0 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d49e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4415b290 con 0x7f8d44159770 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d49e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4415b570 con 0x7f8d4415cd80 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d3a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8d4406f3f0 0x7f8d44159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39242/0 (socket says 172.21.15.28:39242) 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d3a000640 1 -- 172.21.15.28:0/2442641575 learned_addr learned my addr 172.21.15.28:0/2442641575 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d42a00640 1 --2- 172.21.15.28:0/2442641575 >> v2:172.21.15.189:3300/0 conn(0x7f8d44159770 0x7f8d4406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d43400640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4415cd80 0x7f8d44070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d42000640 1 -- 172.21.15.28:0/2442641575 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532151406 0 0) 0x7f8d4406f920 con 0x7f8d4406f3f0 2024-09-13T17:11:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d42000640 1 -- 172.21.15.28:0/2442641575 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28005730 con 0x7f8d4406f3f0 2024-09-13T17:11:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d42a00640 1 -- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4415cd80 msgr2=0x7f8d44070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.645+0000 7f8d42a00640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4415cd80 0x7f8d44070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.646+0000 7f8d42a00640 1 -- 172.21.15.28:0/2442641575 >> v1:172.21.15.28:6789/0 conn(0x7f8d4406f3f0 legacy=0x7f8d44159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.646+0000 7f8d42a00640 1 -- 172.21.15.28:0/2442641575 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d4415b850 con 0x7f8d44159770 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.647+0000 7f8d42a00640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d4406fd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8d340097a0 tx=0x7f8d3402a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792dfefb90c82bbe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.647+0000 7f8d42000640 1 -- 172.21.15.28:0/2442641575 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3402ae50 con 0x7f8d44159770 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.647+0000 7f8d42000640 1 -- 172.21.15.28:0/2442641575 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34032440 con 0x7f8d44159770 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.647+0000 7f8d42000640 1 -- 172.21.15.28:0/2442641575 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d340327d0 con 0x7f8d44159770 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 -- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 msgr2=0x7f8d4406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d4406fd20 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8d340097a0 tx=0x7f8d3402a100 comp rx=0 tx=0).stop 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 -- 172.21.15.28:0/2442641575 shutdown_connections 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4415cd80 0x7f8d44070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 --2- 172.21.15.28:0/2442641575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d4406fd20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 -- 172.21.15.28:0/2442641575 >> 172.21.15.28:0/2442641575 conn(0x7f8d440b0270 msgr2=0x7f8d440b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 -- 172.21.15.28:0/2442641575 shutdown_connections 2024-09-13T17:11:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.648+0000 7f8d49e00640 1 -- 172.21.15.28:0/2442641575 wait complete. 2024-09-13T17:11:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.649+0000 7f8d49e00640 1 Processor -- start 2024-09-13T17:11:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 -- start start 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4406f3f0 0x7f8d441584b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d440aa170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d441589f0 con 0x7f8d4415cd80 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44157410 con 0x7f8d44159770 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.650+0000 7f8d49e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d441576f0 con 0x7f8d4406f3f0 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50520/0 (socket says 172.21.15.28:50520) 2024-09-13T17:11:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 -- 172.21.15.28:0/649738880 learned_addr learned my addr 172.21.15.28:0/649738880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d3a000640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d440aa170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d42a00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4406f3f0 0x7f8d441584b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 -- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4406f3f0 msgr2=0x7f8d441584b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4406f3f0 0x7f8d441584b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 -- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 msgr2=0x7f8d440aa170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d440aa170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 -- 172.21.15.28:0/649738880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d34009400 con 0x7f8d4415cd80 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.651+0000 7f8d43400640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00efc0 tx=0x7f8d3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.652+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c009b20 con 0x7f8d4415cd80 2024-09-13T17:11:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.652+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3c00f040 con 0x7f8d4415cd80 2024-09-13T17:11:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.652+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0146c0 con 0x7f8d4415cd80 2024-09-13T17:11:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.652+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d441579d0 con 0x7f8d4415cd80 2024-09-13T17:11:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.652+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d44157f90 con 0x7f8d4415cd80 2024-09-13T17:11:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.653+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d4406c8b0 con 0x7f8d4415cd80 2024-09-13T17:11:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.654+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d3c014820 con 0x7f8d4415cd80 2024-09-13T17:11:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.655+0000 7f8d3be00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.655+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d3c089fa0 con 0x7f8d4415cd80 2024-09-13T17:11:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.656+0000 7f8d42a00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.656+0000 7f8d42a00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 0x7f8d2006d0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034000 tx=0x7f8d3402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.660+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8d3c055220 con 0x7f8d4415cd80 2024-09-13T17:11:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.841+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d441581b0 con 0x7f8d4415cd80 2024-09-13T17:11:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.842+0000 7f8d3be00640 1 -- 172.21.15.28:0/649738880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d3c018020 con 0x7f8d4415cd80 2024-09-13T17:11:48.844 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:11:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 msgr2=0x7f8d2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 0x7f8d2006d0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8d34034000 tx=0x7f8d3402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 msgr2=0x7f8d441ef080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c00efc0 tx=0x7f8d3c00c490 comp rx=0 tx=0).stop 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 shutdown_connections 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d4415cd80 0x7f8d441ef080 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d44159770 0x7f8d440aa170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 --2- 172.21.15.28:0/649738880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d4406f3f0 0x7f8d441584b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 >> 172.21.15.28:0/649738880 conn(0x7f8d440b0270 msgr2=0x7f8d440ced40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.844+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 shutdown_connections 2024-09-13T17:11:48.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:48.845+0000 7f8d49e00640 1 -- 172.21.15.28:0/649738880 wait complete. 2024-09-13T17:11:48.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:11:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.114+0000 7f49e8000640 1 Processor -- start 2024-09-13T17:11:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.114+0000 7f49e8000640 1 -- start start 2024-09-13T17:11:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.115+0000 7f49e8000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.115+0000 7f49e8000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49e015cd80 0x7f49e0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.115+0000 7f49e8000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e006f920 con 0x7f49e006f3f0 2024-09-13T17:11:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.115+0000 7f49e8000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e015b290 con 0x7f49e015cd80 2024-09-13T17:11:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.115+0000 7f49e8000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e015b570 con 0x7f49e0159770 2024-09-13T17:11:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49e015cd80 0x7f49e0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49e015cd80 0x7f49e0070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44406/0 (socket says 172.21.15.28:44406) 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 -- 172.21.15.28:0/2453181060 learned_addr learned my addr 172.21.15.28:0/2453181060 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56530/0 (socket says 172.21.15.28:56530) 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 -- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 msgr2=0x7f49e006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 -- 172.21.15.28:0/2453181060 >> v1:172.21.15.189:6789/0 conn(0x7f49e006f3f0 legacy=0x7f49e0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e6000640 1 -- 172.21.15.28:0/2453181060 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e015b850 con 0x7f49e015cd80 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49dfe00640 1 -- 172.21.15.28:0/2453181060 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975258523 0 0) 0x7f49e006f920 con 0x7f49e006f3f0 2024-09-13T17:11:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.116+0000 7f49e5600640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.117+0000 7f49e6000640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e015cd80 0x7f49e0070260 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e50 tx=0x7f49d802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffd171d79fd048fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49dfe00640 1 -- 172.21.15.28:0/2453181060 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d802aea0 con 0x7f49e015cd80 2024-09-13T17:11:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49dfe00640 1 -- 172.21.15.28:0/2453181060 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d802d050 con 0x7f49e015cd80 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49dfe00640 1 -- 172.21.15.28:0/2453181060 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8031820 con 0x7f49e015cd80 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 -- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e015cd80 msgr2=0x7f49e0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e015cd80 0x7f49e0070260 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e50 tx=0x7f49d802a680 comp rx=0 tx=0).stop 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 -- 172.21.15.28:0/2453181060 shutdown_connections 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e015cd80 0x7f49e0070260 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 --2- 172.21.15.28:0/2453181060 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e0159770 0x7f49e006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 -- 172.21.15.28:0/2453181060 >> 172.21.15.28:0/2453181060 conn(0x7f49e00b0270 msgr2=0x7f49e00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 -- 172.21.15.28:0/2453181060 shutdown_connections 2024-09-13T17:11:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.118+0000 7f49e8000640 1 -- 172.21.15.28:0/2453181060 wait complete. 2024-09-13T17:11:49.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.120+0000 7f49e8000640 1 Processor -- start 2024-09-13T17:11:49.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.120+0000 7f49e8000640 1 -- start start 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 0x7f49e01500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e015cd80 0x7f49e01583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00cd210 con 0x7f49e0159770 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e0150620 con 0x7f49e006f3f0 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e8000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e01560f0 con 0x7f49e015cd80 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50536/0 (socket says 172.21.15.28:50536) 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e4c00640 1 -- 172.21.15.28:0/3721250924 learned_addr learned my addr 172.21.15.28:0/3721250924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e5600640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 0x7f49e01500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.121+0000 7f49e6000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e015cd80 0x7f49e01583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 -- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e015cd80 msgr2=0x7f49e01583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e015cd80 0x7f49e01583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 -- 172.21.15.28:0/3721250924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 msgr2=0x7f49e0155650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 -- 172.21.15.28:0/3721250924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d4009400 con 0x7f49e006f3f0 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e4c00640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e5600640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 0x7f49e01500e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f49d400e900 tx=0x7f49d400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d400ca00 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d400cb60 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d4013470 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.122+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8009b30 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.123+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e00ccb50 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.124+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e006fd20 con 0x7f49e006f3f0 2024-09-13T17:11:49.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.125+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f49d40135d0 con 0x7f49e006f3f0 2024-09-13T17:11:49.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.125+0000 7f49dd600640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 0x7f49c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.126+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49d4089580 con 0x7f49e006f3f0 2024-09-13T17:11:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.126+0000 7f49e4c00640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 0x7f49c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.126+0000 7f49e4c00640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 0x7f49c006d0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49e0158fe0 tx=0x7f49cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.131+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f49d4054770 con 0x7f49e006f3f0 2024-09-13T17:11:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.313+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f49e00b3850 con 0x7f49e006f3f0 2024-09-13T17:11:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.314+0000 7f49dd600640 1 -- 172.21.15.28:0/3721250924 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7f49d4057d90 con 0x7f49e006f3f0 2024-09-13T17:11:49.317 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:11:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 msgr2=0x7f49c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 0x7f49c006d0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f49e0158fe0 tx=0x7f49cc009040 comp rx=0 tx=0).stop 2024-09-13T17:11:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 msgr2=0x7f49e01500e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 0x7f49e01500e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f49d400e900 tx=0x7f49d400edc0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 shutdown_connections 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49c006ac00 0x7f49c006d0c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49e015cd80 0x7f49e01583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49e0159770 0x7f49e0155650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 --2- 172.21.15.28:0/3721250924 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49e006f3f0 0x7f49e01500e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 >> 172.21.15.28:0/3721250924 conn(0x7f49e00b0270 msgr2=0x7f49e00cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 shutdown_connections 2024-09-13T17:11:49.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.316+0000 7f49e8000640 1 -- 172.21.15.28:0/3721250924 wait complete. 2024-09-13T17:11:49.333 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:49.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.486+0000 7f7759000640 1 Processor -- start 2024-09-13T17:11:49.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.486+0000 7f7759000640 1 -- start start 2024-09-13T17:11:49.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.487+0000 7f7759000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 0x7f775406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.487+0000 7f7759000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 0x7f7754071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.487+0000 7f7759000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f775415e100 con 0x7f77540cabc0 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.487+0000 7f7759000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77540adb00 con 0x7f775415a540 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.487+0000 7f7759000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7754071670 con 0x7f7754070cc0 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f774be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f77540cabc0 0x7f775415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39258/0 (socket says 172.21.15.28:39258) 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f774be00640 1 -- 172.21.15.28:0/3241833611 learned_addr learned my addr 172.21.15.28:0/3241833611 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7752800640 1 --2- 172.21.15.28:0/3241833611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 0x7f7754071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7751e00640 1 --2- 172.21.15.28:0/3241833611 >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 0x7f775406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7751400640 1 -- 172.21.15.28:0/3241833611 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116647019 0 0) 0x7f775415e100 con 0x7f77540cabc0 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7751400640 1 -- 172.21.15.28:0/3241833611 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7738003520 con 0x7f77540cabc0 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7752800640 1 -- 172.21.15.28:0/3241833611 >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 msgr2=0x7f775406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7752800640 1 --2- 172.21.15.28:0/3241833611 >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 0x7f775406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7752800640 1 -- 172.21.15.28:0/3241833611 >> v1:172.21.15.28:6789/0 conn(0x7f77540cabc0 legacy=0x7f775415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7752800640 1 -- 172.21.15.28:0/3241833611 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f775415e3e0 con 0x7f7754070cc0 2024-09-13T17:11:49.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.488+0000 7f7751e00640 1 --2- 172.21.15.28:0/3241833611 >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 0x7f775406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:49.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.489+0000 7f7752800640 1 --2- 172.21.15.28:0/3241833611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 0x7f7754071130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f774c009d90 tx=0x7f774c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e689b1e319e87c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7751400640 1 -- 172.21.15.28:0/3241833611 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c031600 con 0x7f7754070cc0 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7751400640 1 -- 172.21.15.28:0/3241833611 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774c02d050 con 0x7f7754070cc0 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7751400640 1 -- 172.21.15.28:0/3241833611 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774c031980 con 0x7f7754070cc0 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 -- 172.21.15.28:0/3241833611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 msgr2=0x7f7754071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 --2- 172.21.15.28:0/3241833611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 0x7f7754071130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f774c009d90 tx=0x7f774c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 -- 172.21.15.28:0/3241833611 shutdown_connections 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 --2- 172.21.15.28:0/3241833611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7754070cc0 0x7f7754071130 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 --2- 172.21.15.28:0/3241833611 >> v2:172.21.15.189:3300/0 conn(0x7f775415a540 0x7f775406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 -- 172.21.15.28:0/3241833611 >> 172.21.15.28:0/3241833611 conn(0x7f7754069e80 msgr2=0x7f775406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:49.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.490+0000 7f7759000640 1 -- 172.21.15.28:0/3241833611 shutdown_connections 2024-09-13T17:11:49.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.491+0000 7f7759000640 1 -- 172.21.15.28:0/3241833611 wait complete. 2024-09-13T17:11:49.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.492+0000 7f7759000640 1 Processor -- start 2024-09-13T17:11:49.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.492+0000 7f7759000640 1 -- start start 2024-09-13T17:11:49.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7754070cc0 0x7f775414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f775415a540 0x7f77540ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775415c080 con 0x7f77540cabc0 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f775415bd80 con 0x7f7754070cc0 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7759000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f775414d600 con 0x7f775415a540 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f774be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f774be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50542/0 (socket says 172.21.15.28:50542) 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f774be00640 1 -- 172.21.15.28:0/113081899 learned_addr learned my addr 172.21.15.28:0/113081899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7751e00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7754070cc0 0x7f775414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.493+0000 7f7752800640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f775415a540 0x7f77540ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 -- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f775415a540 msgr2=0x7f77540ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f775415a540 0x7f77540ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 -- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7754070cc0 msgr2=0x7f775414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7754070cc0 0x7f775414d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 -- 172.21.15.28:0/113081899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744009400 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774be00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c530 tx=0x7f773c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c005ce0 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c005e40 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c0072b0 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.494+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c009a70 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.495+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f775414f510 con 0x7f77540cabc0 2024-09-13T17:11:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.495+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7754071130 con 0x7f77540cabc0 2024-09-13T17:11:49.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.496+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f773c007410 con 0x7f77540cabc0 2024-09-13T17:11:49.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.497+0000 7f774a000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 0x7f773006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.497+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f773c088890 con 0x7f77540cabc0 2024-09-13T17:11:49.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.498+0000 7f7751e00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 0x7f773006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.498+0000 7f7751e00640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 0x7f773006d0f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7744009c30 tx=0x7f7744009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.501+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f773c053bd0 con 0x7f77540cabc0 2024-09-13T17:11:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.715+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f775415c230 con 0x7f77540cabc0 2024-09-13T17:11:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.715+0000 7f774a000640 1 -- 172.21.15.28:0/113081899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f773c017020 con 0x7f77540cabc0 2024-09-13T17:11:49.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:49.718 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:49.140632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 msgr2=0x7f773006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 0x7f773006d0f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7744009c30 tx=0x7f7744009190 comp rx=0 tx=0).stop 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 msgr2=0x7f775414da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f773c00c530 tx=0x7f773c00c9f0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 shutdown_connections 2024-09-13T17:11:49.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f773006ac30 0x7f773006d0f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f775415a540 0x7f77540ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77540cabc0 0x7f775414da80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 --2- 172.21.15.28:0/113081899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7754070cc0 0x7f775414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 >> 172.21.15.28:0/113081899 conn(0x7f7754069e80 msgr2=0x7f7754155ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 shutdown_connections 2024-09-13T17:11:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.718+0000 7f7759000640 1 -- 172.21.15.28:0/113081899 wait complete. 2024-09-13T17:11:49.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.888+0000 7f48f3e00640 1 Processor -- start 2024-09-13T17:11:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.888+0000 7f48f3e00640 1 -- start start 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f3e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48ec1592a0 0x7f48ec15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48ec06f970 0x7f48ec159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f3e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48ec0b2040 con 0x7f48ec06c2d0 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec15b0b0 con 0x7f48ec06f970 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f3e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48ec159ff0 con 0x7f48ec1592a0 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48ec06f970 0x7f48ec159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f48ec06f970 0x7f48ec159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50552/0 (socket says 172.21.15.28:50552) 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.889+0000 7f48f0a00640 1 -- 172.21.15.28:0/3640538573 learned_addr learned my addr 172.21.15.28:0/3640538573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.890+0000 7f48f1400640 1 --2- 172.21.15.28:0/3640538573 >> v2:172.21.15.189:3300/0 conn(0x7f48ec1592a0 0x7f48ec15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.890+0000 7f48f0a00640 1 -- 172.21.15.28:0/3640538573 >> v2:172.21.15.189:3300/0 conn(0x7f48ec1592a0 msgr2=0x7f48ec15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.890+0000 7f48f0a00640 1 --2- 172.21.15.28:0/3640538573 >> v2:172.21.15.189:3300/0 conn(0x7f48ec1592a0 0x7f48ec15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.890+0000 7f48f0a00640 1 -- 172.21.15.28:0/3640538573 >> v1:172.21.15.189:6789/0 conn(0x7f48ec06c2d0 legacy=0x7f48ec0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.890+0000 7f48f0a00640 1 -- 172.21.15.28:0/3640538573 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48ec15a2d0 con 0x7f48ec06f970 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.891+0000 7f48f0a00640 1 --2- 172.21.15.28:0/3640538573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec06f970 0x7f48ec159680 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f48ec06d370 tx=0x7f48d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b5f288f7b68a46 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.891+0000 7f48ebe00640 1 -- 172.21.15.28:0/3640538573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8032440 con 0x7f48ec06f970 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.891+0000 7f48ebe00640 1 -- 172.21.15.28:0/3640538573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48d80325a0 con 0x7f48ec06f970 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.891+0000 7f48ebe00640 1 -- 172.21.15.28:0/3640538573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d8032930 con 0x7f48ec06f970 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 -- 172.21.15.28:0/3640538573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec06f970 msgr2=0x7f48ec159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3640538573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec06f970 0x7f48ec159680 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f48ec06d370 tx=0x7f48d802a270 comp rx=0 tx=0).stop 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 -- 172.21.15.28:0/3640538573 shutdown_connections 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3640538573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec06f970 0x7f48ec159680 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3640538573 >> v2:172.21.15.189:3300/0 conn(0x7f48ec1592a0 0x7f48ec15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 -- 172.21.15.28:0/3640538573 >> 172.21.15.28:0/3640538573 conn(0x7f48ec069e80 msgr2=0x7f48ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 -- 172.21.15.28:0/3640538573 shutdown_connections 2024-09-13T17:11:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.892+0000 7f48f3e00640 1 -- 172.21.15.28:0/3640538573 wait complete. 2024-09-13T17:11:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.893+0000 7f48f3e00640 1 Processor -- start 2024-09-13T17:11:49.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 -- start start 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ec06f970 0x7f48ec0cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 0x7f48ec0ad250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec0cbf20 con 0x7f48ec1592a0 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec0cc200 con 0x7f48ec06f970 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.894+0000 7f48f3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48ec0cc4e0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 0x7f48ec0ad250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ec06f970 0x7f48ec0cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56568/0 (socket says 172.21.15.28:56568) 2024-09-13T17:11:49.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 -- 172.21.15.28:0/3992297238 learned_addr learned my addr 172.21.15.28:0/3992297238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 -- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ec06f970 msgr2=0x7f48ec0cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ec06f970 0x7f48ec0cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 -- 172.21.15.28:0/3992297238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 msgr2=0x7f48ec0ad250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 0x7f48ec0ad250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1400640 1 -- 172.21.15.28:0/3992297238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d80094d0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.895+0000 7f48f1e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 0x7f48ec0ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48f1400640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48e000e870 tx=0x7f48e000ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e00094f0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e000c940 con 0x7f48ec06c2d0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e000ccf0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ec0cc7c0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.896+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ec0cbaf0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.897+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ec15ab70 con 0x7f48ec06c2d0 2024-09-13T17:11:49.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.898+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f48e0013470 con 0x7f48ec06c2d0 2024-09-13T17:11:49.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.899+0000 7f48e9600640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 0x7f48cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:49.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.899+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48e00893a0 con 0x7f48ec06c2d0 2024-09-13T17:11:49.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.899+0000 7f48f0a00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 0x7f48cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:49.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.900+0000 7f48f0a00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 0x7f48cc06d0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48d8034000 tx=0x7f48d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:49.904+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f48e0054620 con 0x7f48ec06c2d0 2024-09-13T17:11:50.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.082+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48ec0b0c50 con 0x7f48ec06c2d0 2024-09-13T17:11:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.084+0000 7f48e9600640 1 -- 172.21.15.28:0/3992297238 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+16513 (secure 0 0 0) 0x7f48e0057c40 con 0x7f48ec06c2d0 2024-09-13T17:11:50.086 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:50.086 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":48,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:09.922940+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 msgr2=0x7f48cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 0x7f48cc06d0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48d8034000 tx=0x7f48d802e000 comp rx=0 tx=0).stop 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 msgr2=0x7f48ec151220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f48e000e870 tx=0x7f48e000ed30 comp rx=0 tx=0).stop 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 shutdown_connections 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48cc06ac00 0x7f48cc06d0c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48ec1592a0 0x7f48ec0ad250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48ec06f970 0x7f48ec0cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 --2- 172.21.15.28:0/3992297238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48ec06c2d0 0x7f48ec151220 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 >> 172.21.15.28:0/3992297238 conn(0x7f48ec069e80 msgr2=0x7f48ec06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 shutdown_connections 2024-09-13T17:11:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.086+0000 7f48f3e00640 1 -- 172.21.15.28:0/3992297238 wait complete. 2024-09-13T17:11:50.104 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:11:50.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:50.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.257+0000 7f340a800640 1 Processor -- start 2024-09-13T17:11:50.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.257+0000 7f340a800640 1 -- start start 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f340a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 0x7f340406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f340a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340415e100 con 0x7f3404070cc0 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f340a800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f340415e3e0 con 0x7f340415a540 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f340a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040adb00 con 0x7f34040cabc0 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3403400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f340415a540 0x7f340415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:45702/0 (socket says 172.21.15.28:45702) 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f33fbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 0x7f340406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3403e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3404070cc0 0x7f340406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39274/0 (socket says 172.21.15.28:39274) 2024-09-13T17:11:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3403400640 1 -- 172.21.15.28:0/1697904379 learned_addr learned my addr 172.21.15.28:0/1697904379 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828691717 0 0) 0x7f340415e3e0 con 0x7f340415a540 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e8003520 con 0x7f340415a540 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871421341 0 0) 0x7f340415e100 con 0x7f3404070cc0 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f33fbe00640 1 -- 172.21.15.28:0/1697904379 >> v1:172.21.15.189:6789/0 conn(0x7f340415a540 legacy=0x7f340415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f33fbe00640 1 -- 172.21.15.28:0/1697904379 >> v1:172.21.15.28:6789/0 conn(0x7f3404070cc0 legacy=0x7f340406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.259+0000 7f33fbe00640 1 -- 172.21.15.28:0/1697904379 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340415ef80 con 0x7f34040cabc0 2024-09-13T17:11:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f33fbe00640 1 --2- 172.21.15.28:0/1697904379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 0x7f340406c430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0097d0 tx=0x7f33ec02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5c6e6f9b3b42e73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec032440 con 0x7f34040cabc0 2024-09-13T17:11:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0325a0 con 0x7f34040cabc0 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f3402a00640 1 -- 172.21.15.28:0/1697904379 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec032930 con 0x7f34040cabc0 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 -- 172.21.15.28:0/1697904379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 msgr2=0x7f340406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 --2- 172.21.15.28:0/1697904379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 0x7f340406c430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0097d0 tx=0x7f33ec02a180 comp rx=0 tx=0).stop 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 -- 172.21.15.28:0/1697904379 shutdown_connections 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 --2- 172.21.15.28:0/1697904379 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34040cabc0 0x7f340406c430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 -- 172.21.15.28:0/1697904379 >> 172.21.15.28:0/1697904379 conn(0x7f3404069e80 msgr2=0x7f340406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.261+0000 7f340a800640 1 -- 172.21.15.28:0/1697904379 shutdown_connections 2024-09-13T17:11:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.262+0000 7f340a800640 1 -- 172.21.15.28:0/1697904379 wait complete. 2024-09-13T17:11:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.263+0000 7f340a800640 1 Processor -- start 2024-09-13T17:11:50.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.263+0000 7f340a800640 1 -- start start 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34040cabc0 0x7f34040b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f340415a540 0x7f340413b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340414b620 con 0x7f3404070cc0 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34040af390 con 0x7f34040cabc0 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f340a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34040af670 con 0x7f340415a540 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f3403400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f3403400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50576/0 (socket says 172.21.15.28:50576) 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.264+0000 7f3403400640 1 -- 172.21.15.28:0/2231223202 learned_addr learned my addr 172.21.15.28:0/2231223202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f33fbe00640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34040cabc0 0x7f34040b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403e00640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f340415a540 0x7f340413b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 -- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f340415a540 msgr2=0x7f340413b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f340415a540 0x7f340413b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 -- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34040cabc0 msgr2=0x7f34040b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34040cabc0 0x7f34040b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 -- 172.21.15.28:0/2231223202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec009400 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f3403400640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c3c0 tx=0x7f33f400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4005ce0 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f4005e40 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4007310 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.265+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34040af950 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.266+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34040aff10 con 0x7f3404070cc0 2024-09-13T17:11:50.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.267+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f340406fad0 con 0x7f3404070cc0 2024-09-13T17:11:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.267+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f33f4007530 con 0x7f3404070cc0 2024-09-13T17:11:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.268+0000 7f33fb400640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 0x7f33e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.268+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33f4088970 con 0x7f3404070cc0 2024-09-13T17:11:50.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.269+0000 7f33fbe00640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 0x7f33e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.269+0000 7f33fbe00640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 0x7f33e006d0c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f33ec034000 tx=0x7f33ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.272+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f33f4053b40 con 0x7f3404070cc0 2024-09-13T17:11:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.485+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f340406c430 con 0x7f3404070cc0 2024-09-13T17:11:50.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.486+0000 7f33fb400640 1 -- 172.21.15.28:0/2231223202 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f33f4017020 con 0x7f3404070cc0 2024-09-13T17:11:50.488 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:50.488 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:49.140632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:50.488 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 msgr2=0x7f33e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 0x7f33e006d0c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f33ec034000 tx=0x7f33ec02e000 comp rx=0 tx=0).stop 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 msgr2=0x7f340413aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f33f400c3c0 tx=0x7f33f400c880 comp rx=0 tx=0).stop 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 shutdown_connections 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33e006ac00 0x7f33e006d0c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f340415a540 0x7f340413b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34040cabc0 0x7f34040b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 --2- 172.21.15.28:0/2231223202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3404070cc0 0x7f340413aae0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 >> 172.21.15.28:0/2231223202 conn(0x7f3404069e80 msgr2=0x7f340415b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:50.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 shutdown_connections 2024-09-13T17:11:50.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.488+0000 7f340a800640 1 -- 172.21.15.28:0/2231223202 wait complete. 2024-09-13T17:11:50.505 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:50.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.659+0000 7f7619600640 1 Processor -- start 2024-09-13T17:11:50.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.659+0000 7f7619600640 1 -- start start 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7619600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76141592a0 0x7f761415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7619600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 0x7f761406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7619600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761406cc70 con 0x7f761406f970 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7619600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761415b0b0 con 0x7f76141592a0 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7619600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7614159d90 con 0x7f761406c2d0 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7612a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76141592a0 0x7f761415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f760be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f761406f970 0x7f7614159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39286/0 (socket says 172.21.15.28:39286) 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f7612a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f76141592a0 0x7f761415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50592/0 (socket says 172.21.15.28:50592) 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.660+0000 7f760be00640 1 -- 172.21.15.28:0/244468017 learned_addr learned my addr 172.21.15.28:0/244468017 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7613400640 1 --2- 172.21.15.28:0/244468017 >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 0x7f761406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7612a00640 1 -- 172.21.15.28:0/244468017 >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 msgr2=0x7f761406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7612a00640 1 --2- 172.21.15.28:0/244468017 >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 0x7f761406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7612000640 1 -- 172.21.15.28:0/244468017 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270908283 0 0) 0x7f761406cc70 con 0x7f761406f970 2024-09-13T17:11:50.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7612a00640 1 -- 172.21.15.28:0/244468017 >> v1:172.21.15.28:6789/0 conn(0x7f761406f970 legacy=0x7f7614159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7612a00640 1 -- 172.21.15.28:0/244468017 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76140b2040 con 0x7f76141592a0 2024-09-13T17:11:50.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.661+0000 7f7613400640 1 --2- 172.21.15.28:0/244468017 >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 0x7f761406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.662+0000 7f7612a00640 1 --2- 172.21.15.28:0/244468017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76141592a0 0x7f761415ab70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f76140b1dd0 tx=0x7f760402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d78235d7ed37168 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.662+0000 7f7612000640 1 -- 172.21.15.28:0/244468017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604032440 con 0x7f76141592a0 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.662+0000 7f7612000640 1 -- 172.21.15.28:0/244468017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76040325a0 con 0x7f76141592a0 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.662+0000 7f7612000640 1 -- 172.21.15.28:0/244468017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604032930 con 0x7f76141592a0 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 -- 172.21.15.28:0/244468017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76141592a0 msgr2=0x7f761415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 --2- 172.21.15.28:0/244468017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76141592a0 0x7f761415ab70 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f76140b1dd0 tx=0x7f760402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 -- 172.21.15.28:0/244468017 shutdown_connections 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 --2- 172.21.15.28:0/244468017 >> v2:172.21.15.189:3300/0 conn(0x7f761406c2d0 0x7f761406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 --2- 172.21.15.28:0/244468017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76141592a0 0x7f761415ab70 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 -- 172.21.15.28:0/244468017 >> 172.21.15.28:0/244468017 conn(0x7f7614069e80 msgr2=0x7f761406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 -- 172.21.15.28:0/244468017 shutdown_connections 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.663+0000 7f7619600640 1 -- 172.21.15.28:0/244468017 wait complete. 2024-09-13T17:11:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.665+0000 7f7619600640 1 Processor -- start 2024-09-13T17:11:50.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.665+0000 7f7619600640 1 -- start start 2024-09-13T17:11:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f761406c2d0 0x7f7614148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 0x7f76140acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141e3e50 con 0x7f761406f970 2024-09-13T17:11:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76141e4410 con 0x7f76141592a0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7619600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76140af910 con 0x7f761406c2d0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f760be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f760be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50606/0 (socket says 172.21.15.28:50606) 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f760be00640 1 -- 172.21.15.28:0/371925605 learned_addr learned my addr 172.21.15.28:0/371925605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7613400640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 0x7f76140acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.666+0000 7f7612a00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f761406c2d0 0x7f7614148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 -- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f761406c2d0 msgr2=0x7f7614148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f761406c2d0 0x7f7614148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 -- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 msgr2=0x7f76140acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 0x7f76140acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 -- 172.21.15.28:0/371925605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604009400 con 0x7f761406f970 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f7613400640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 0x7f76140acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760be00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c500 tx=0x7f75fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc007560 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc0047e0 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc004b90 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76140afbf0 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.667+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7614149420 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.668+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76140b3740 con 0x7f761406f970 2024-09-13T17:11:50.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.669+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f75fc004cf0 con 0x7f761406f970 2024-09-13T17:11:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.670+0000 7f760aa00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 0x7f75f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.670+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75fc088480 con 0x7f761406f970 2024-09-13T17:11:50.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.671+0000 7f7612a00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 0x7f75f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:50.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.671+0000 7f7612a00640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 0x7f75f006d0c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7604034000 tx=0x7f760402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.674+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f75fc053700 con 0x7f761406f970 2024-09-13T17:11:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.892+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76141e4550 con 0x7f761406f970 2024-09-13T17:11:50.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.893+0000 7f760aa00640 1 -- 172.21.15.28:0/371925605 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f75fc056d20 con 0x7f761406f970 2024-09-13T17:11:50.895 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:50.895 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":41,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:11:08.920544+0000","modified":"2024-09-13T17:11:49.140632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5580},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5580":{"gid":5580,"name":"c","rank":0,"incarnation":31,"state":"up:active","state_seq":13,"addr":"172.21.15.28:6833/1182054705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1182054705},{"type":"v1","addr":"172.21.15.28:6833","nonce":1182054705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T17:11:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 41 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 msgr2=0x7f75f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 0x7f75f006d0c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7604034000 tx=0x7f760402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 msgr2=0x7f76140ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f75fc00c500 tx=0x7f75fc00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 shutdown_connections 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75f006ac00 0x7f75f006d0c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76141592a0 0x7f76140acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f761406f970 0x7f76140ac850 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 --2- 172.21.15.28:0/371925605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f761406c2d0 0x7f7614148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 >> 172.21.15.28:0/371925605 conn(0x7f7614069e80 msgr2=0x7f7614159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 shutdown_connections 2024-09-13T17:11:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:50.895+0000 7f7619600640 1 -- 172.21.15.28:0/371925605 wait complete. 2024-09-13T17:11:50.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.067+0000 7f7041a00640 1 Processor -- start 2024-09-13T17:11:51.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.067+0000 7f7041a00640 1 -- start start 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.068+0000 7f7041a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.068+0000 7f7041a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f703c06c2d0 0x7f703c06c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.068+0000 7f7041a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703c159d90 con 0x7f703c06f970 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.068+0000 7f7041a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f703c15b0b0 con 0x7f703c06c2d0 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.068+0000 7f7041a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c06ccc0 con 0x7f703c1592a0 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f703c06c2d0 0x7f703c06c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56608/0 (socket says 172.21.15.28:56608) 2024-09-13T17:11:51.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 -- 172.21.15.28:0/3259676313 learned_addr learned my addr 172.21.15.28:0/3259676313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 -- 172.21.15.28:0/3259676313 >> v2:172.21.15.189:3300/0 conn(0x7f703c06c2d0 msgr2=0x7f703c06c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 --2- 172.21.15.28:0/3259676313 >> v2:172.21.15.189:3300/0 conn(0x7f703c06c2d0 0x7f703c06c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 -- 172.21.15.28:0/3259676313 >> v1:172.21.15.189:6789/0 conn(0x7f703c06f970 legacy=0x7f703c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.069+0000 7f703aa00640 1 -- 172.21.15.28:0/3259676313 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703c0b2040 con 0x7f703c1592a0 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.070+0000 7f703aa00640 1 --2- 172.21.15.28:0/3259676313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f702c00bb10 tx=0x7f702c02c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbd188dd239e0870 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f703a000640 1 -- 172.21.15.28:0/3259676313 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c032820 con 0x7f703c1592a0 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f703a000640 1 -- 172.21.15.28:0/3259676313 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c02e040 con 0x7f703c1592a0 2024-09-13T17:11:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f703a000640 1 -- 172.21.15.28:0/3259676313 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c032ba0 con 0x7f703c1592a0 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 -- 172.21.15.28:0/3259676313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 msgr2=0x7f703c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 --2- 172.21.15.28:0/3259676313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f702c00bb10 tx=0x7f702c02c920 comp rx=0 tx=0).stop 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 -- 172.21.15.28:0/3259676313 shutdown_connections 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 --2- 172.21.15.28:0/3259676313 >> v2:172.21.15.189:3300/0 conn(0x7f703c06c2d0 0x7f703c06c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 --2- 172.21.15.28:0/3259676313 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c1592a0 0x7f703c15ab70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 -- 172.21.15.28:0/3259676313 >> 172.21.15.28:0/3259676313 conn(0x7f703c069e80 msgr2=0x7f703c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.071+0000 7f7041a00640 1 -- 172.21.15.28:0/3259676313 shutdown_connections 2024-09-13T17:11:51.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.072+0000 7f7041a00640 1 -- 172.21.15.28:0/3259676313 wait complete. 2024-09-13T17:11:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.073+0000 7f7041a00640 1 Processor -- start 2024-09-13T17:11:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.073+0000 7f7041a00640 1 -- start start 2024-09-13T17:11:51.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c06f970 0x7f703c152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 0x7f703c0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0cc6e0 con 0x7f703c06c2d0 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703c0ccc60 con 0x7f703c1592a0 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f7041a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703c1531a0 con 0x7f703c06f970 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f703aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f703aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50610/0 (socket says 172.21.15.28:50610) 2024-09-13T17:11:51.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.074+0000 7f703aa00640 1 -- 172.21.15.28:0/959164390 learned_addr learned my addr 172.21.15.28:0/959164390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703b400640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 0x7f703c0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f7032000640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c06f970 0x7f703c152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 -- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c06f970 msgr2=0x7f703c152c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c06f970 0x7f703c152c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 -- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 msgr2=0x7f703c0aa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 0x7f703c0aa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 -- 172.21.15.28:0/959164390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c00b770 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703b400640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 0x7f703c0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f703aa00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f702c02c960 tx=0x7f702c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c03b7b0 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c03b910 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c03bcc0 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.075+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f703c0cea10 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.076+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703c0cefd0 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.077+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703c0b3740 con 0x7f703c06c2d0 2024-09-13T17:11:51.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.077+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f702c03a3b0 con 0x7f703c06c2d0 2024-09-13T17:11:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.078+0000 7f7033400640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 0x7f701806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.078+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f702c0ac400 con 0x7f703c06c2d0 2024-09-13T17:11:51.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.079+0000 7f7032000640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 0x7f701806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.080+0000 7f7032000640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 0x7f701806d0f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f703c06bd50 tx=0x7f7024009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.083+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f702c0775d0 con 0x7f703c06c2d0 2024-09-13T17:11:51.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.266+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f703c071330 con 0x7f703c06c2d0 2024-09-13T17:11:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.267+0000 7f7033400640 1 -- 172.21.15.28:0/959164390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16609 (secure 0 0 0) 0x7f702c07abf0 con 0x7f703c06c2d0 2024-09-13T17:11:51.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:51.269 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:51.147420+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:11:07.454143+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-09-13T17:11:07.927877+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.16","mappings":[{"from":0,"to":5}]},{"pgid":"7.3b","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 msgr2=0x7f701806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 0x7f701806d0f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f703c06bd50 tx=0x7f7024009040 comp rx=0 tx=0).stop 2024-09-13T17:11:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 msgr2=0x7f703c0ca4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f702c02c960 tx=0x7f702c004730 comp rx=0 tx=0).stop 2024-09-13T17:11:51.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 shutdown_connections 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f701806ac30 0x7f701806d0f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f703c1592a0 0x7f703c0aa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f703c06f970 0x7f703c152c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 --2- 172.21.15.28:0/959164390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f703c06c2d0 0x7f703c0ca4c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 >> 172.21.15.28:0/959164390 conn(0x7f703c069e80 msgr2=0x7f703c0b07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 shutdown_connections 2024-09-13T17:11:51.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.269+0000 7f7041a00640 1 -- 172.21.15.28:0/959164390 wait complete. 2024-09-13T17:11:51.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:11:51.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.443+0000 7fb1d7000640 1 Processor -- start 2024-09-13T17:11:51.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.443+0000 7fb1d7000640 1 -- start start 2024-09-13T17:11:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1d7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1d00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1d7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d0070470 con 0x7fb1d015c9a0 2024-09-13T17:11:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1d7000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d00ace30 con 0x7fb1d00b2400 2024-09-13T17:11:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1d7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00ad110 con 0x7fb1d00ad940 2024-09-13T17:11:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb1d015c9a0 0x7fb1d00b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39288/0 (socket says 172.21.15.28:39288) 2024-09-13T17:11:51.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cfe00640 1 -- 172.21.15.28:0/771922896 learned_addr learned my addr 172.21.15.28:0/771922896 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:51.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556687819 0 0) 0x7fb1d0070470 con 0x7fb1d015c9a0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1d5000640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1d00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1b4003520 con 0x7fb1d015c9a0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 411327402 0 0) 0x7fb1b4003520 con 0x7fb1d015c9a0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 msgr2=0x7fb1d00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1d00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 >> v1:172.21.15.189:6789/0 conn(0x7fb1d00b2400 legacy=0x7fb1d006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d006b170 con 0x7fb1d015c9a0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fb1c0001670 con 0x7fb1d015c9a0 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 >> v1:172.21.15.28:6789/0 conn(0x7fb1d015c9a0 legacy=0x7fb1d00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.444+0000 7fb1cea00640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00ad110 con 0x7fb1b40047c0 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400bac0 con 0x7fb1b40039a0 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b400bc80 con 0x7fb1d00ad940 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1d5000640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 -- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 msgr2=0x7fb1b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00ad940 0x7fb1b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 -- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1b40039a0 msgr2=0x7fb1b4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1b40039a0 0x7fb1b4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.445+0000 7fb1cf400640 1 -- 172.21.15.28:0/771922896 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b4003520 con 0x7fb1b40047c0 2024-09-13T17:11:51.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.446+0000 7fb1cf400640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1b4004c40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc001720 tx=0x7fb1b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c818e40f13e8f713 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc0052d0 con 0x7fb1b40047c0 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1bc005430 con 0x7fb1b40047c0 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1cea00640 1 -- 172.21.15.28:0/771922896 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc02ab90 con 0x7fb1b40047c0 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 -- 172.21.15.28:0/771922896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 msgr2=0x7fb1b4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 --2- 172.21.15.28:0/771922896 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1b4004c40 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc001720 tx=0x7fb1b40057f0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d5000640 1 -- 172.21.15.28:0/771922896 reap_dead start 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 -- 172.21.15.28:0/771922896 shutdown_connections 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 -- 172.21.15.28:0/771922896 >> 172.21.15.28:0/771922896 conn(0x7fb1d00b0270 msgr2=0x7fb1d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 -- 172.21.15.28:0/771922896 shutdown_connections 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.447+0000 7fb1d7000640 1 -- 172.21.15.28:0/771922896 wait complete. 2024-09-13T17:11:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.449+0000 7fb1d7000640 1 Processor -- start 2024-09-13T17:11:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.449+0000 7fb1d7000640 1 -- start start 2024-09-13T17:11:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1d00cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00cb260 0x7fb1d015f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d01f3310 0x7fb1d01f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d00cc620 con 0x7fb1b40047c0 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d015f470 con 0x7fb1d01f3310 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d015e580 con 0x7fb1d00cb260 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1d00cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1d00cc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50630/0 (socket says 172.21.15.28:50630) 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1cfe00640 1 -- 172.21.15.28:0/2068732462 learned_addr learned my addr 172.21.15.28:0/2068732462 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1cf400640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00cb260 0x7fb1d015f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.450+0000 7fb1d5000640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d01f3310 0x7fb1d01f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 -- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00cb260 msgr2=0x7fb1d015f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb1d00cb260 0x7fb1d015f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 -- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d01f3310 msgr2=0x7fb1d01f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb1d01f3310 0x7fb1d01f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 -- 172.21.15.28:0/2068732462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d0070930 con 0x7fb1b40047c0 2024-09-13T17:11:51.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1cfe00640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb1b40047c0 0x7fb1d00cc0e0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00072c0 tx=0x7fb1c0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c0001650 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c000f960 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c000fcd0 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1d7000640 1 -- 172.21.15.28:0/2068732462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d015e860 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.451+0000 7fb1d7000640 1 -- 172.21.15.28:0/2068732462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d015ee20 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.452+0000 7fb1d7000640 1 -- 172.21.15.28:0/2068732462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d015cd80 con 0x7fb1b40047c0 2024-09-13T17:11:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.454+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb1c001d360 con 0x7fb1b40047c0 2024-09-13T17:11:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.454+0000 7fb1c7e00640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb1b006ac00 0x7fb1b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.454+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1c0088cc0 con 0x7fb1b40047c0 2024-09-13T17:11:51.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.455+0000 7fb1cf400640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb1b006ac00 0x7fb1b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.455+0000 7fb1cf400640 1 --2- 172.21.15.28:0/2068732462 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb1b006ac00 0x7fb1b006d0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc032000 tx=0x7fb1bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.459+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb1c008f080 con 0x7fb1b40047c0 2024-09-13T17:11:51.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.672+0000 7fb1d7000640 1 -- 172.21.15.28:0/2068732462 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb1d006c7b0 con 0x7fb1b40047c0 2024-09-13T17:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.673+0000 7fb1c7e00640 1 -- 172.21.15.28:0/2068732462 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=-1 MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it v41) v1 ==== 316+0+0 (secure 0 0 0) 0x7fb1c0053de0 con 0x7fb1b40047c0 2024-09-13T17:11:51.674 INFO:teuthology.orchestra.run.smithi028.stderr:Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it 2024-09-13T17:11:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:11:51.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs --yes-i-really-mean-it 2024-09-13T17:11:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.842+0000 7fd6e1000640 1 Processor -- start 2024-09-13T17:11:51.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.842+0000 7fd6e1000640 1 -- start start 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.843+0000 7fd6e1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6dc06d070 0x7fd6dc0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.843+0000 7fd6e1000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd6dc06c540 0x7fd6dc16a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.843+0000 7fd6e1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc1610a0 con 0x7fd6dc0b17e0 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.843+0000 7fd6e1000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc0b2620 con 0x7fd6dc06d070 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.843+0000 7fd6e1000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc0b2900 con 0x7fd6dc06c540 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6dc06d070 0x7fd6dc0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6dc0b17e0 0x7fd6dc0b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39292/0 (socket says 172.21.15.28:39292) 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9e00640 1 -- 172.21.15.28:0/2043221053 learned_addr learned my addr 172.21.15.28:0/2043221053 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6da800640 1 --2- 172.21.15.28:0/2043221053 >> v2:172.21.15.189:3300/0 conn(0x7fd6dc06c540 0x7fd6dc16a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9400640 1 -- 172.21.15.28:0/2043221053 >> v2:172.21.15.189:3300/0 conn(0x7fd6dc06c540 msgr2=0x7fd6dc16a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9400640 1 --2- 172.21.15.28:0/2043221053 >> v2:172.21.15.189:3300/0 conn(0x7fd6dc06c540 0x7fd6dc16a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9400640 1 -- 172.21.15.28:0/2043221053 >> v1:172.21.15.28:6789/0 conn(0x7fd6dc0b17e0 legacy=0x7fd6dc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d9400640 1 -- 172.21.15.28:0/2043221053 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0b2be0 con 0x7fd6dc06d070 2024-09-13T17:11:51.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.844+0000 7fd6d8a00640 1 -- 172.21.15.28:0/2043221053 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304818795 0 0) 0x7fd6dc1610a0 con 0x7fd6dc0b17e0 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.845+0000 7fd6d9400640 1 --2- 172.21.15.28:0/2043221053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06d070 0x7fd6dc0b3480 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc071880 tx=0x7fd6c802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bed8030db41e887 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.845+0000 7fd6d8a00640 1 -- 172.21.15.28:0/2043221053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8032440 con 0x7fd6dc06d070 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.845+0000 7fd6d8a00640 1 -- 172.21.15.28:0/2043221053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c80325a0 con 0x7fd6dc06d070 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.845+0000 7fd6d8a00640 1 -- 172.21.15.28:0/2043221053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6c8032930 con 0x7fd6dc06d070 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 -- 172.21.15.28:0/2043221053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06d070 msgr2=0x7fd6dc0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 --2- 172.21.15.28:0/2043221053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06d070 0x7fd6dc0b3480 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc071880 tx=0x7fd6c802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 -- 172.21.15.28:0/2043221053 shutdown_connections 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 --2- 172.21.15.28:0/2043221053 >> v2:172.21.15.189:3300/0 conn(0x7fd6dc06c540 0x7fd6dc16a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 --2- 172.21.15.28:0/2043221053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06d070 0x7fd6dc0b3480 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 -- 172.21.15.28:0/2043221053 >> 172.21.15.28:0/2043221053 conn(0x7fd6dc0b0270 msgr2=0x7fd6dc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 -- 172.21.15.28:0/2043221053 shutdown_connections 2024-09-13T17:11:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.846+0000 7fd6e1000640 1 -- 172.21.15.28:0/2043221053 wait complete. 2024-09-13T17:11:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.848+0000 7fd6e1000640 1 Processor -- start 2024-09-13T17:11:51.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.848+0000 7fd6e1000640 1 -- start start 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6dc06d070 0x7fd6dc0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 0x7fd6dc0ab080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc1f7870 con 0x7fd6dc06c540 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc15ba10 con 0x7fd6dc0b17e0 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6e1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0a89b0 con 0x7fd6dc06d070 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50656/0 (socket says 172.21.15.28:50656) 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 -- 172.21.15.28:0/3723007006 learned_addr learned my addr 172.21.15.28:0/3723007006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:51.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9400640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6dc06d070 0x7fd6dc0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6da800640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 0x7fd6dc0ab080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 -- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6dc06d070 msgr2=0x7fd6dc0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6dc06d070 0x7fd6dc0ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 -- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 msgr2=0x7fd6dc0ab080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 0x7fd6dc0ab080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6d9e00640 1 -- 172.21.15.28:0/3723007006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8009400 con 0x7fd6dc06c540 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.849+0000 7fd6da800640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 0x7fd6dc0ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6d9e00640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c500 tx=0x7fd6cc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc007560 con 0x7fd6dc06c540 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc0047e0 con 0x7fd6dc06c540 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc004b90 con 0x7fd6dc06c540 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc0ad360 con 0x7fd6dc06c540 2024-09-13T17:11:51.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.850+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc0ad5e0 con 0x7fd6dc06c540 2024-09-13T17:11:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.851+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc06cbd0 con 0x7fd6dc06c540 2024-09-13T17:11:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.852+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6cc004cf0 con 0x7fd6dc06c540 2024-09-13T17:11:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.853+0000 7fd6d2000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 0x7fd6bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:51.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.853+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd6cc088530 con 0x7fd6dc06c540 2024-09-13T17:11:51.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.853+0000 7fd6d9400640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 0x7fd6bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.854+0000 7fd6d9400640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 0x7fd6bc06d0c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8034000 tx=0x7fd6c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:51.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:51.857+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6cc0536c0 con 0x7fd6dc06c540 2024-09-13T17:11:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.076+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd6dc0b3480 con 0x7fd6dc06c540 2024-09-13T17:11:52.078 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:11:52.077+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:11:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.087+0000 7fd6d2000640 1 -- 172.21.15.28:0/3723007006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 191+0+0 (secure 0 0 0) 0x7fd6cc056ce0 con 0x7fd6dc06c540 2024-09-13T17:11:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.089+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 msgr2=0x7fd6bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.089+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 0x7fd6bc06d0c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd6c8034000 tx=0x7fd6c802e000 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.089+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 msgr2=0x7fd6dc1f7330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.089+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00c500 tx=0x7fd6cc00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 shutdown_connections 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6bc06ac00 0x7fd6bc06d0c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6dc0b17e0 0x7fd6dc0ab080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6dc06d070 0x7fd6dc0ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 --2- 172.21.15.28:0/3723007006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6dc06c540 0x7fd6dc1f7330 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 >> 172.21.15.28:0/3723007006 conn(0x7fd6dc0b0270 msgr2=0x7fd6dc0afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 shutdown_connections 2024-09-13T17:11:52.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.090+0000 7fd6e1000640 1 -- 172.21.15.28:0/3723007006 wait complete. 2024-09-13T17:11:52.106 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:11:52.164 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:11:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.266+0000 7f1cf2600640 1 Processor -- start 2024-09-13T17:11:52.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.266+0000 7f1cf2600640 1 -- start start 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.267+0000 7f1cf2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cec159770 0x7f1cec06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.267+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec15cd80 0x7f1cec070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1cf2600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cec06f920 con 0x7f1cec06f3f0 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1cf2600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cec15b290 con 0x7f1cec159770 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1cf2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec15b570 con 0x7f1cec15cd80 2024-09-13T17:11:52.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cec159770 0x7f1cec06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cec159770 0x7f1cec06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50698/0 (socket says 172.21.15.28:50698) 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 -- 172.21.15.28:0/606125610 learned_addr learned my addr 172.21.15.28:0/606125610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1cebe00640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec15cd80 0x7f1cec070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 -- 172.21.15.28:0/606125610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec15cd80 msgr2=0x7f1cec070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec15cd80 0x7f1cec070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 -- 172.21.15.28:0/606125610 >> v1:172.21.15.189:6789/0 conn(0x7f1cec06f3f0 legacy=0x7f1cec159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.268+0000 7f1ceb400640 1 -- 172.21.15.28:0/606125610 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec15b850 con 0x7f1cec159770 2024-09-13T17:11:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.269+0000 7f1ceb400640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec159770 0x7f1cec06fd20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009870 tx=0x7f1cdc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887ce0645aee6073 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1ceaa00640 1 -- 172.21.15.28:0/606125610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc032440 con 0x7f1cec159770 2024-09-13T17:11:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1ceaa00640 1 -- 172.21.15.28:0/606125610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc0325a0 con 0x7f1cec159770 2024-09-13T17:11:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1ceaa00640 1 -- 172.21.15.28:0/606125610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc032930 con 0x7f1cec159770 2024-09-13T17:11:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 -- 172.21.15.28:0/606125610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec159770 msgr2=0x7f1cec06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec159770 0x7f1cec06fd20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009870 tx=0x7f1cdc02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 -- 172.21.15.28:0/606125610 shutdown_connections 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec15cd80 0x7f1cec070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 --2- 172.21.15.28:0/606125610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec159770 0x7f1cec06fd20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 -- 172.21.15.28:0/606125610 >> 172.21.15.28:0/606125610 conn(0x7f1cec0b0270 msgr2=0x7f1cec0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.270+0000 7f1cf2600640 1 -- 172.21.15.28:0/606125610 shutdown_connections 2024-09-13T17:11:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.271+0000 7f1cf2600640 1 -- 172.21.15.28:0/606125610 wait complete. 2024-09-13T17:11:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.272+0000 7f1cf2600640 1 Processor -- start 2024-09-13T17:11:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.272+0000 7f1cf2600640 1 -- start start 2024-09-13T17:11:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec159770 0x7f1cec0ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cec15cd80 0x7f1cec14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec14f6d0 con 0x7f1cec06f3f0 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec14f9b0 con 0x7f1cec15cd80 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1cf2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cec14fc90 con 0x7f1cec159770 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1ceb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1ceb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50712/0 (socket says 172.21.15.28:50712) 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1ceb400640 1 -- 172.21.15.28:0/646562502 learned_addr learned my addr 172.21.15.28:0/646562502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.273+0000 7f1ce2a00640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec159770 0x7f1cec0ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1cebe00640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cec15cd80 0x7f1cec14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 -- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec159770 msgr2=0x7f1cec0ce230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec159770 0x7f1cec0ce230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 -- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cec15cd80 msgr2=0x7f1cec14e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cec15cd80 0x7f1cec14e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 -- 172.21.15.28:0/646562502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc0094d0 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.274+0000 7f1ceb400640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000c00 tx=0x7f1cdc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.275+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc032440 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.275+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc030b80 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.275+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc03d440 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.275+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cec14ff70 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.275+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cec13ae20 con 0x7f1cec06f3f0 2024-09-13T17:11:52.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.276+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cec06fd20 con 0x7f1cec06f3f0 2024-09-13T17:11:52.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.277+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1cdc039040 con 0x7f1cec06f3f0 2024-09-13T17:11:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.278+0000 7f1ce3e00640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 0x7f1cc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.278+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1cdc0a9d70 con 0x7f1cec06f3f0 2024-09-13T17:11:52.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.279+0000 7f1ce2a00640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 0x7f1cc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:52.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.279+0000 7f1ce2a00640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 0x7f1cc806d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1cec1f3670 tx=0x7f1cd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.282+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1cdc075000 con 0x7f1cec06f3f0 2024-09-13T17:11:52.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:52.496+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1cec06d120 con 0x7f1cec06f3f0 2024-09-13T17:11:53.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.161+0000 7f1ce3e00640 1 -- 172.21.15.28:0/646562502 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1cdc078620 con 0x7f1cec06f3f0 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 msgr2=0x7f1cc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 0x7f1cc806d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f1cec1f3670 tx=0x7f1cd4009040 comp rx=0 tx=0).stop 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 msgr2=0x7f1cec13a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc000c00 tx=0x7f1cdc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 shutdown_connections 2024-09-13T17:11:53.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cc806ac00 0x7f1cc806d0c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cec15cd80 0x7f1cec14e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cec159770 0x7f1cec0ce230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 --2- 172.21.15.28:0/646562502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cec06f3f0 0x7f1cec13a8e0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 >> 172.21.15.28:0/646562502 conn(0x7f1cec0b0270 msgr2=0x7f1cec0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 shutdown_connections 2024-09-13T17:11:53.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.164+0000 7f1cf2600640 1 -- 172.21.15.28:0/646562502 wait complete. 2024-09-13T17:11:53.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:11:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.345+0000 7fb77f000640 1 Processor -- start 2024-09-13T17:11:53.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.345+0000 7fb77f000640 1 -- start start 2024-09-13T17:11:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.346+0000 7fb77f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7780ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.346+0000 7fb77f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7780ce9d0 0x7fb7780b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.346+0000 7fb77f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77806c950 con 0x7fb7780b1ab0 2024-09-13T17:11:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.346+0000 7fb77f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7780cee10 con 0x7fb7780ce9d0 2024-09-13T17:11:53.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.346+0000 7fb77f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780b1ec0 con 0x7fb77815a7a0 2024-09-13T17:11:53.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb77d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb7780b1ab0 0x7fb77806b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39294/0 (socket says 172.21.15.28:39294) 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb77d000640 1 -- 172.21.15.28:0/753190274 learned_addr learned my addr 172.21.15.28:0/753190274 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb777e00640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7780ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb76f400640 1 --2- 172.21.15.28:0/753190274 >> v2:172.21.15.189:3300/0 conn(0x7fb7780ce9d0 0x7fb7780b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb777400640 1 -- 172.21.15.28:0/753190274 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1526816720 0 0) 0x7fb770001490 con 0x7fb7780b1ab0 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb777400640 1 -- 172.21.15.28:0/753190274 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb75c003520 con 0x7fb7780b1ab0 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb76f400640 1 -- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 msgr2=0x7fb7780ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb76f400640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7780ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb76f400640 1 -- 172.21.15.28:0/753190274 >> v1:172.21.15.28:6789/0 conn(0x7fb7780b1ab0 legacy=0x7fb77806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb76f400640 1 -- 172.21.15.28:0/753190274 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b2150 con 0x7fb7780ce9d0 2024-09-13T17:11:53.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.347+0000 7fb777e00640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7780ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:11:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb76f400640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780b1570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7600097a0 tx=0x7fb76002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bc7badd3c5cc145 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb777400640 1 -- 172.21.15.28:0/753190274 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76002ae50 con 0x7fb7780ce9d0 2024-09-13T17:11:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb777400640 1 -- 172.21.15.28:0/753190274 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb760032440 con 0x7fb7780ce9d0 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb777400640 1 -- 172.21.15.28:0/753190274 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7600327d0 con 0x7fb7780ce9d0 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 -- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 msgr2=0x7fb7780b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780b1570 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fb7600097a0 tx=0x7fb76002a100 comp rx=0 tx=0).stop 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 -- 172.21.15.28:0/753190274 shutdown_connections 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780b1570 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 --2- 172.21.15.28:0/753190274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7780ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.349+0000 7fb77f000640 1 -- 172.21.15.28:0/753190274 >> 172.21.15.28:0/753190274 conn(0x7fb778069e80 msgr2=0x7fb77806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.350+0000 7fb77f000640 1 -- 172.21.15.28:0/753190274 shutdown_connections 2024-09-13T17:11:53.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.350+0000 7fb77f000640 1 -- 172.21.15.28:0/753190274 wait complete. 2024-09-13T17:11:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.351+0000 7fb77f000640 1 Processor -- start 2024-09-13T17:11:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.351+0000 7fb77f000640 1 -- start start 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780a9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7781f38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780aa4d0 con 0x7fb7780b1ab0 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780a98c0 con 0x7fb7780ce9d0 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb77f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7780a9ba0 con 0x7fb77815a7a0 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb777e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb777e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50724/0 (socket says 172.21.15.28:50724) 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.352+0000 7fb777e00640 1 -- 172.21.15.28:0/423102414 learned_addr learned my addr 172.21.15.28:0/423102414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb76f400640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb77d000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7781f38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 -- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 msgr2=0x7fb7781f38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7781f38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 -- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 msgr2=0x7fb7780a9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780a9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:53.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 -- 172.21.15.28:0/423102414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb768009400 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb777e00640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb76800ebe0 tx=0x7fb76800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.353+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb768009760 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.354+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76800cc30 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.354+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76800eee0 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.354+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb760009400 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.354+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7780aa160 con 0x7fb7780b1ab0 2024-09-13T17:11:53.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.355+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780ce200 con 0x7fb7780b1ab0 2024-09-13T17:11:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.356+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb768013470 con 0x7fb7780b1ab0 2024-09-13T17:11:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.356+0000 7fb774c00640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 0x7fb75406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:53.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.357+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fb768089630 con 0x7fb7780b1ab0 2024-09-13T17:11:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.357+0000 7fb76f400640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 0x7fb75406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:53.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.358+0000 7fb76f400640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 0x7fb75406d0f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb760034000 tx=0x7fb76002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:53.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.361+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb768054820 con 0x7fb7780b1ab0 2024-09-13T17:11:53.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:53.539+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb778157800 con 0x7fb7780b1ab0 2024-09-13T17:11:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.164+0000 7fb774c00640 1 -- 172.21.15.28:0/423102414 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v51) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb768057e40 con 0x7fb7780b1ab0 2024-09-13T17:11:54.165 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 msgr2=0x7fb75406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 0x7fb75406d0f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb760034000 tx=0x7fb76002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 msgr2=0x7fb7781529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb76800ebe0 tx=0x7fb76800c520 comp rx=0 tx=0).stop 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 shutdown_connections 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb75406ac30 0x7fb75406d0f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb77815a7a0 0x7fb7781f38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7780ce9d0 0x7fb7780a9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 --2- 172.21.15.28:0/423102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7780b1ab0 0x7fb7781529d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.167+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 >> 172.21.15.28:0/423102414 conn(0x7fb778069e80 msgr2=0x7fb778157bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.168+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 shutdown_connections 2024-09-13T17:11:54.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.168+0000 7fb77f000640 1 -- 172.21.15.28:0/423102414 wait complete. 2024-09-13T17:11:54.188 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:11:54.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.350+0000 7fb74be00640 1 Processor -- start 2024-09-13T17:11:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.351+0000 7fb74be00640 1 -- start start 2024-09-13T17:11:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 0x7fb74c0b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb74c0b2dd0 0x7fb74c159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb74c0b3600 con 0x7fb74c159440 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0b46a0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb74c0b38e0 con 0x7fb74c06b9c0 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb74c159440 0x7fb74c06b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56588/0 (socket says 172.21.15.28:56588) 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74aa00640 1 -- 172.21.15.28:0/2169082331 learned_addr learned my addr 172.21.15.28:0/2169082331 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb74b400640 1 --2- 172.21.15.28:0/2169082331 >> v2:172.21.15.28:3300/0 conn(0x7fb74c0b2dd0 0x7fb74c159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.352+0000 7fb742000640 1 --2- 172.21.15.28:0/2169082331 >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 0x7fb74c0b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74a000640 1 -- 172.21.15.28:0/2169082331 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408164342 0 0) 0x7fb74c0b3600 con 0x7fb74c159440 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74a000640 1 -- 172.21.15.28:0/2169082331 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb730003520 con 0x7fb74c159440 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74b400640 1 -- 172.21.15.28:0/2169082331 >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 msgr2=0x7fb74c0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74b400640 1 --2- 172.21.15.28:0/2169082331 >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 0x7fb74c0b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74b400640 1 -- 172.21.15.28:0/2169082331 >> v1:172.21.15.28:6789/0 conn(0x7fb74c159440 legacy=0x7fb74c06b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb74b400640 1 -- 172.21.15.28:0/2169082331 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74c0b3bc0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.353+0000 7fb742000640 1 --2- 172.21.15.28:0/2169082331 >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 0x7fb74c0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74b400640 1 --2- 172.21.15.28:0/2169082331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c159820 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb74c06acd0 tx=0x7fb74402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f15f851da75296 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74a000640 1 -- 172.21.15.28:0/2169082331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74402aea0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74a000640 1 -- 172.21.15.28:0/2169082331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb74402d050 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74a000640 1 -- 172.21.15.28:0/2169082331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb744007960 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74be00640 1 -- 172.21.15.28:0/2169082331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 msgr2=0x7fb74c159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74be00640 1 --2- 172.21.15.28:0/2169082331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c159820 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb74c06acd0 tx=0x7fb74402a680 comp rx=0 tx=0).stop 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74be00640 1 -- 172.21.15.28:0/2169082331 shutdown_connections 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74be00640 1 --2- 172.21.15.28:0/2169082331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c159820 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.354+0000 7fb74be00640 1 --2- 172.21.15.28:0/2169082331 >> v2:172.21.15.189:3300/0 conn(0x7fb74c06b9c0 0x7fb74c0b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.355+0000 7fb74be00640 1 -- 172.21.15.28:0/2169082331 >> 172.21.15.28:0/2169082331 conn(0x7fb74c069e80 msgr2=0x7fb74c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.355+0000 7fb74be00640 1 -- 172.21.15.28:0/2169082331 shutdown_connections 2024-09-13T17:11:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.355+0000 7fb74be00640 1 -- 172.21.15.28:0/2169082331 wait complete. 2024-09-13T17:11:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.356+0000 7fb74be00640 1 Processor -- start 2024-09-13T17:11:54.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.356+0000 7fb74be00640 1 -- start start 2024-09-13T17:11:54.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74c06b9c0 0x7fb74c13ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 0x7fb74c1469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c1564a0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c155b70 con 0x7fb74c06b9c0 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb74be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb74c155e50 con 0x7fb74c159440 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb742000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb742000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60830/0 (socket says 172.21.15.28:60830) 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.357+0000 7fb742000640 1 -- 172.21.15.28:0/2156791274 learned_addr learned my addr 172.21.15.28:0/2156791274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb74aa00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74c06b9c0 0x7fb74c13ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb74b400640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 0x7fb74c1469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 -- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 msgr2=0x7fb74c1469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 0x7fb74c1469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 -- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74c06b9c0 msgr2=0x7fb74c13ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74c06b9c0 0x7fb74c13ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 -- 172.21.15.28:0/2156791274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb744009a70 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb74b400640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 0x7fb74c1469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb742000640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb73400e730 tx=0x7fb73400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb734004230 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb734004390 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb73400c8f0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.358+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74c156130 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.359+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74c140be0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.360+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74c067a10 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.361+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb73400ca50 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.361+0000 7fb743e00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 0x7fb72806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:54.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.361+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fb734088db0 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.362+0000 7fb74aa00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 0x7fb72806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:54.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.362+0000 7fb74aa00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 0x7fb72806d0f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0011e0 tx=0x7fb73c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.366+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb73408d070 con 0x7fb74c0b2dd0 2024-09-13T17:11:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:54.551+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb74c1562c0 con 0x7fb74c0b2dd0 2024-09-13T17:11:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.176+0000 7fb743e00640 1 -- 172.21.15.28:0/2156791274 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v52) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb734057be0 con 0x7fb74c0b2dd0 2024-09-13T17:11:55.177 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.179+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 msgr2=0x7fb72806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.179+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 0x7fb72806d0f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0011e0 tx=0x7fb73c007d30 comp rx=0 tx=0).stop 2024-09-13T17:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.179+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 msgr2=0x7fb74c1421c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.179+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb73400e730 tx=0x7fb73400ebf0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 shutdown_connections 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb72806ac30 0x7fb72806d0f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb74c159440 0x7fb74c1469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb74c0b2dd0 0x7fb74c1421c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 --2- 172.21.15.28:0/2156791274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb74c06b9c0 0x7fb74c13ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.180+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 >> 172.21.15.28:0/2156791274 conn(0x7fb74c069e80 msgr2=0x7fb74c0cb780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.181+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 shutdown_connections 2024-09-13T17:11:55.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.181+0000 7fb74be00640 1 -- 172.21.15.28:0/2156791274 wait complete. 2024-09-13T17:11:55.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:11:55.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.358+0000 7f83ca000640 1 Processor -- start 2024-09-13T17:11:55.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.358+0000 7f83ca000640 1 -- start start 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 0x7f83c406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c406f3f0 0x7f83c4070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c40add00 0x7f83c4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c406f920 con 0x7f83c406f3f0 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83c415b290 con 0x7f83c4159770 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.359+0000 7f83ca000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c415b570 con 0x7f83c40add00 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c406f3f0 0x7f83c4070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83c406f3f0 0x7f83c4070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60836/0 (socket says 172.21.15.28:60836) 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 -- 172.21.15.28:0/2469366160 learned_addr learned my addr 172.21.15.28:0/2469366160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83c3e00640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c40add00 0x7f83c4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83c3400640 1 --2- 172.21.15.28:0/2469366160 >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 0x7f83c406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 -- 172.21.15.28:0/2469366160 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c40add00 msgr2=0x7f83c4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c40add00 0x7f83c4159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 -- 172.21.15.28:0/2469366160 >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 msgr2=0x7f83c406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 --2- 172.21.15.28:0/2469366160 >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 0x7f83c406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83baa00640 1 -- 172.21.15.28:0/2469366160 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c415b850 con 0x7f83c406f3f0 2024-09-13T17:11:55.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.360+0000 7f83c3400640 1 --2- 172.21.15.28:0/2469366160 >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 0x7f83c406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:55.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.361+0000 7f83baa00640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406f3f0 0x7f83c4070260 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f83c4067f80 tx=0x7f83ac02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=403addbb307c1189 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.361+0000 7f83c2a00640 1 -- 172.21.15.28:0/2469366160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac02ae90 con 0x7f83c406f3f0 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83c2a00640 1 -- 172.21.15.28:0/2469366160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac032440 con 0x7f83c406f3f0 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83c2a00640 1 -- 172.21.15.28:0/2469366160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac0327d0 con 0x7f83c406f3f0 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 -- 172.21.15.28:0/2469366160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406f3f0 msgr2=0x7f83c4070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406f3f0 0x7f83c4070260 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f83c4067f80 tx=0x7f83ac02a140 comp rx=0 tx=0).stop 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 -- 172.21.15.28:0/2469366160 shutdown_connections 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c406f3f0 0x7f83c4070260 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 --2- 172.21.15.28:0/2469366160 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c40add00 0x7f83c4159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 --2- 172.21.15.28:0/2469366160 >> v2:172.21.15.189:3300/0 conn(0x7f83c4159770 0x7f83c406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 -- 172.21.15.28:0/2469366160 >> 172.21.15.28:0/2469366160 conn(0x7f83c40b0270 msgr2=0x7f83c40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:55.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 -- 172.21.15.28:0/2469366160 shutdown_connections 2024-09-13T17:11:55.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.362+0000 7f83ca000640 1 -- 172.21.15.28:0/2469366160 wait complete. 2024-09-13T17:11:55.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.365+0000 7f83ca000640 1 Processor -- start 2024-09-13T17:11:55.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.365+0000 7f83ca000640 1 -- start start 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c406f3f0 0x7f83c4140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c4159770 0x7f83c4158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c41410d0 con 0x7f83c40add00 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4141370 con 0x7f83c406f3f0 2024-09-13T17:11:55.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83ca000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c4141650 con 0x7f83c4159770 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60850/0 (socket says 172.21.15.28:60850) 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83baa00640 1 -- 172.21.15.28:0/881928675 learned_addr learned my addr 172.21.15.28:0/881928675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83c3400640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c406f3f0 0x7f83c4140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.366+0000 7f83c3e00640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c4159770 0x7f83c4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 -- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c4159770 msgr2=0x7f83c4158000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c4159770 0x7f83c4158000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 -- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c406f3f0 msgr2=0x7f83c4140b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c406f3f0 0x7f83c4140b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 -- 172.21.15.28:0/881928675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83ac0094d0 con 0x7f83c40add00 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83baa00640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f83ac02a180 tx=0x7f83ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac030850 con 0x7f83c40add00 2024-09-13T17:11:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83ac0309b0 con 0x7f83c40add00 2024-09-13T17:11:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83ac030d60 con 0x7f83c40add00 2024-09-13T17:11:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.367+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c4141930 con 0x7f83c40add00 2024-09-13T17:11:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.368+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c4141ef0 con 0x7f83c40add00 2024-09-13T17:11:55.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.369+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c4141790 con 0x7f83c40add00 2024-09-13T17:11:55.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.369+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f83ac039040 con 0x7f83c40add00 2024-09-13T17:11:55.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.370+0000 7f83bbe00640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 0x7f83a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.370+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f83ac0a9bf0 con 0x7f83c40add00 2024-09-13T17:11:55.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.371+0000 7f83c3400640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 0x7f83a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.372+0000 7f83c3400640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 0x7f83a006d0c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f83b4006e80 tx=0x7f83b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.374+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f83c4141790 con 0x7f83c40add00 2024-09-13T17:11:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.584+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f83c4141790 con 0x7f83c40add00 2024-09-13T17:11:55.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.588+0000 7f83bbe00640 1 -- 172.21.15.28:0/881928675 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f83c4141790 con 0x7f83c40add00 2024-09-13T17:11:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 msgr2=0x7f83a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 0x7f83a006d0c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f83b4006e80 tx=0x7f83b4008040 comp rx=0 tx=0).stop 2024-09-13T17:11:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 msgr2=0x7f83c4142410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f83ac02a180 tx=0x7f83ac0046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 shutdown_connections 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f83a006ac00 0x7f83a006d0c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83c4159770 0x7f83c4158000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83c40add00 0x7f83c4142410 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 --2- 172.21.15.28:0/881928675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f83c406f3f0 0x7f83c4140b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 >> 172.21.15.28:0/881928675 conn(0x7f83c40b0270 msgr2=0x7f83c40ae670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 shutdown_connections 2024-09-13T17:11:55.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.590+0000 7f83ca000640 1 -- 172.21.15.28:0/881928675 wait complete. 2024-09-13T17:11:55.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks' 2024-09-13T17:11:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.761+0000 7f6082400640 1 Processor -- start 2024-09-13T17:11:55.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.761+0000 7f6082400640 1 -- start start 2024-09-13T17:11:55.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.762+0000 7f6082400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f607c15ab60 0x7f607c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.762+0000 7f6082400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c06c8b0 con 0x7f607c071490 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.762+0000 7f6082400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f607c06cb90 con 0x7f607c15a780 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.762+0000 7f6082400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f607c0cc980 con 0x7f607c15ab60 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f607c15ab60 0x7f607c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f607c15ab60 0x7f607c06bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60852/0 (socket says 172.21.15.28:60852) 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 -- 172.21.15.28:0/2187530153 learned_addr learned my addr 172.21.15.28:0/2187530153 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:55.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 -- 172.21.15.28:0/2187530153 >> v1:172.21.15.189:6789/0 conn(0x7f607c15a780 legacy=0x7f607c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 -- 172.21.15.28:0/2187530153 >> v1:172.21.15.28:6789/0 conn(0x7f607c071490 legacy=0x7f607c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607aa00640 1 -- 172.21.15.28:0/2187530153 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c0ccc60 con 0x7f607c15ab60 2024-09-13T17:11:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.763+0000 7f607a000640 1 -- 172.21.15.28:0/2187530153 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009743351 0 0) 0x7f607c06c8b0 con 0x7f607c071490 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.764+0000 7f607aa00640 1 --2- 172.21.15.28:0/2187530153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15ab60 0x7f607c06bb40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6068009870 tx=0x7f606802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a79a481e65976fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.764+0000 7f607a000640 1 -- 172.21.15.28:0/2187530153 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068032640 con 0x7f607c15ab60 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.764+0000 7f607a000640 1 -- 172.21.15.28:0/2187530153 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60680327a0 con 0x7f607c15ab60 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.764+0000 7f607a000640 1 -- 172.21.15.28:0/2187530153 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068032b30 con 0x7f607c15ab60 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 -- 172.21.15.28:0/2187530153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15ab60 msgr2=0x7f607c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 --2- 172.21.15.28:0/2187530153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15ab60 0x7f607c06bb40 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f6068009870 tx=0x7f606802a300 comp rx=0 tx=0).stop 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 -- 172.21.15.28:0/2187530153 shutdown_connections 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 --2- 172.21.15.28:0/2187530153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15ab60 0x7f607c06bb40 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 -- 172.21.15.28:0/2187530153 >> 172.21.15.28:0/2187530153 conn(0x7f607c0b0270 msgr2=0x7f607c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 -- 172.21.15.28:0/2187530153 shutdown_connections 2024-09-13T17:11:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.765+0000 7f6082400640 1 -- 172.21.15.28:0/2187530153 wait complete. 2024-09-13T17:11:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.767+0000 7f6082400640 1 Processor -- start 2024-09-13T17:11:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.767+0000 7f6082400640 1 -- start start 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f607c071490 0x7f607c154240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607c15ab60 0x7f607c14b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c154780 con 0x7f607c15a780 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c1503c0 con 0x7f607c15ab60 2024-09-13T17:11:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f6082400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f607c1506a0 con 0x7f607c071490 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60862/0 (socket says 172.21.15.28:60862) 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 -- 172.21.15.28:0/3200697303 learned_addr learned my addr 172.21.15.28:0/3200697303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607be00640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607c15ab60 0x7f607c14b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607b400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f607c071490 0x7f607c154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 -- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f607c071490 msgr2=0x7f607c154240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f607c071490 0x7f607c154240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 -- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607c15ab60 msgr2=0x7f607c14b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607c15ab60 0x7f607c14b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.768+0000 7f607aa00640 1 -- 172.21.15.28:0/3200697303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60680094d0 con 0x7f607c15a780 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f607aa00640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6068009870 tx=0x7f60680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068030b80 con 0x7f607c15a780 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6068030ce0 con 0x7f607c15a780 2024-09-13T17:11:55.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c150980 con 0x7f607c15a780 2024-09-13T17:11:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f606803d660 con 0x7f607c15a780 2024-09-13T17:11:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.769+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c150f40 con 0x7f607c15a780 2024-09-13T17:11:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.770+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c1507e0 con 0x7f607c15a780 2024-09-13T17:11:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.771+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6068039040 con 0x7f607c15a780 2024-09-13T17:11:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.772+0000 7f6073400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 0x7f605c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.772+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60680a9e90 con 0x7f607c15a780 2024-09-13T17:11:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.773+0000 7f607b400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 0x7f605c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:55.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.773+0000 7f607b400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 0x7f605c06d0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f606c000f60 tx=0x7f606c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:55.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:55.777+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f607c1507e0 con 0x7f607c15a780 2024-09-13T17:11:56.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.027+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]} v 0) v1 -- 0x7f607c1507e0 con 0x7f607c15a780 2024-09-13T17:11:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.177+0000 7f6073400640 1 -- 172.21.15.28:0/3200697303 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_checks"]}]=0 v204) v1 ==== 143+0+0 (secure 0 0 0) 0x7f607c1507e0 con 0x7f607c15a780 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 msgr2=0x7f605c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 0x7f605c06d0c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f606c000f60 tx=0x7f606c009040 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 msgr2=0x7f607c151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6068009870 tx=0x7f60680046c0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 shutdown_connections 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f605c06ac00 0x7f605c06d0c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f607c15ab60 0x7f607c14b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f607c15a780 0x7f607c151460 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 --2- 172.21.15.28:0/3200697303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f607c071490 0x7f607c154240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 >> 172.21.15.28:0/3200697303 conn(0x7f607c0b0270 msgr2=0x7f607c06bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 shutdown_connections 2024-09-13T17:11:56.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.179+0000 7f6082400640 1 -- 172.21.15.28:0/3200697303 wait complete. 2024-09-13T17:11:56.196 INFO:tasks.cephfs_test_runner:test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ok 2024-09-13T17:11:56.197 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) That a directory is merged during scrub. 2024-09-13T17:11:56.197 INFO:tasks.cephfs_test_runner:test_scrub_merge_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:11:56.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:11:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.352+0000 7f1719c00640 1 Processor -- start 2024-09-13T17:11:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.352+0000 7f1719c00640 1 -- start start 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.353+0000 7f1719c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f171415a540 0x7f171406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.353+0000 7f1719c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 0x7f1714071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.353+0000 7f1719c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171415e100 con 0x7f17140cabc0 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.353+0000 7f1719c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17140adb00 con 0x7f171415a540 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.353+0000 7f1719c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1714071670 con 0x7f1714070cc0 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f171415a540 0x7f171406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f171415a540 0x7f171406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60868/0 (socket says 172.21.15.28:60868) 2024-09-13T17:11:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 -- 172.21.15.28:0/938962980 learned_addr learned my addr 172.21.15.28:0/938962980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1713400640 1 --2- 172.21.15.28:0/938962980 >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 0x7f1714071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 -- 172.21.15.28:0/938962980 >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 msgr2=0x7f1714071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 --2- 172.21.15.28:0/938962980 >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 0x7f1714071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 -- 172.21.15.28:0/938962980 >> v1:172.21.15.189:6789/0 conn(0x7f17140cabc0 legacy=0x7f171415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1712a00640 1 -- 172.21.15.28:0/938962980 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171415e3e0 con 0x7f171415a540 2024-09-13T17:11:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.354+0000 7f1713400640 1 --2- 172.21.15.28:0/938962980 >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 0x7f1714071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.355+0000 7f1712a00640 1 --2- 172.21.15.28:0/938962980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171415a540 0x7f171406c430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1714070900 tx=0x7f170402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624598d88556a521 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1711600640 1 -- 172.21.15.28:0/938962980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704032440 con 0x7f171415a540 2024-09-13T17:11:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1711600640 1 -- 172.21.15.28:0/938962980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17040325a0 con 0x7f171415a540 2024-09-13T17:11:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1711600640 1 -- 172.21.15.28:0/938962980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704032930 con 0x7f171415a540 2024-09-13T17:11:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 -- 172.21.15.28:0/938962980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171415a540 msgr2=0x7f171406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 --2- 172.21.15.28:0/938962980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171415a540 0x7f171406c430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f1714070900 tx=0x7f170402a230 comp rx=0 tx=0).stop 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 -- 172.21.15.28:0/938962980 shutdown_connections 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 --2- 172.21.15.28:0/938962980 >> v2:172.21.15.189:3300/0 conn(0x7f1714070cc0 0x7f1714071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 --2- 172.21.15.28:0/938962980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171415a540 0x7f171406c430 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 -- 172.21.15.28:0/938962980 >> 172.21.15.28:0/938962980 conn(0x7f1714069e80 msgr2=0x7f171406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 -- 172.21.15.28:0/938962980 shutdown_connections 2024-09-13T17:11:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.356+0000 7f1719c00640 1 -- 172.21.15.28:0/938962980 wait complete. 2024-09-13T17:11:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.358+0000 7f1719c00640 1 Processor -- start 2024-09-13T17:11:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.358+0000 7f1719c00640 1 -- start start 2024-09-13T17:11:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 0x7f171414ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171415a540 0x7f17140ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17140ae420 con 0x7f17140cabc0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17140ae700 con 0x7f171415a540 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1719c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17140ae9e0 con 0x7f1714070cc0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1712000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1712000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60884/0 (socket says 172.21.15.28:60884) 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1712000640 1 -- 172.21.15.28:0/3217954013 learned_addr learned my addr 172.21.15.28:0/3217954013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1713400640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171415a540 0x7f17140ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.359+0000 7f1712a00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 0x7f171414ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 -- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 msgr2=0x7f171414ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 0x7f171414ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 -- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171415a540 msgr2=0x7f17140ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171415a540 0x7f17140ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 -- 172.21.15.28:0/3217954013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17040094d0 con 0x7f17140cabc0 2024-09-13T17:11:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712a00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 0x7f171414ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1712000640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f170000c450 tx=0x7f170000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17000052d0 con 0x7f17140cabc0 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1700005430 con 0x7f17140cabc0 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1700012870 con 0x7f17140cabc0 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17140aecc0 con 0x7f17140cabc0 2024-09-13T17:11:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.360+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17140b36c0 con 0x7f17140cabc0 2024-09-13T17:11:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.361+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17140af1e0 con 0x7f17140cabc0 2024-09-13T17:11:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.362+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f17000129d0 con 0x7f17140cabc0 2024-09-13T17:11:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.363+0000 7f170aa00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 0x7f16f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.363+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1700088b50 con 0x7f17140cabc0 2024-09-13T17:11:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.364+0000 7f1712a00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 0x7f16f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.364+0000 7f1712a00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 0x7f16f406d0c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1704034000 tx=0x7f170402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.368+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f17140af1e0 con 0x7f17140cabc0 2024-09-13T17:11:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.548+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17140af1e0 con 0x7f17140cabc0 2024-09-13T17:11:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.549+0000 7f170aa00640 1 -- 172.21.15.28:0/3217954013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+12986 (secure 0 0 0) 0x7f17140af1e0 con 0x7f17140cabc0 2024-09-13T17:11:56.551 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:56.551 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":52,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:11:55.168563+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6833/1182054705":"2024-09-14T17:11:52.077282+0000","172.21.15.28:6832/1182054705":"2024-09-14T17:11:52.077282+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 msgr2=0x7f16f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 0x7f16f406d0c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1704034000 tx=0x7f170402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 msgr2=0x7f17140afdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f170000c450 tx=0x7f170000c910 comp rx=0 tx=0).stop 2024-09-13T17:11:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 shutdown_connections 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f406ac00 0x7f16f406d0c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171415a540 0x7f17140ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17140cabc0 0x7f17140afdb0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 --2- 172.21.15.28:0/3217954013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1714070cc0 0x7f171414ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 >> 172.21.15.28:0/3217954013 conn(0x7f1714069e80 msgr2=0x7f17140cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 shutdown_connections 2024-09-13T17:11:56.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.551+0000 7f1719c00640 1 -- 172.21.15.28:0/3217954013 wait complete. 2024-09-13T17:11:56.569 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:11:56.569 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:11:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.726+0000 7f53b0a00640 1 Processor -- start 2024-09-13T17:11:56.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.726+0000 7f53b0a00640 1 -- start start 2024-09-13T17:11:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.727+0000 7f53b0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac15ab60 0x7f53ac06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.727+0000 7f53b0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53ac071490 0x7f53ac06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.727+0000 7f53b0a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53ac06c8b0 con 0x7f53ac15a780 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.727+0000 7f53b0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53ac06cb90 con 0x7f53ac071490 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.727+0000 7f53b0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac0cc980 con 0x7f53ac15ab60 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53ac071490 0x7f53ac06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53ac071490 0x7f53ac06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60892/0 (socket says 172.21.15.28:60892) 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 -- 172.21.15.28:0/1434896043 learned_addr learned my addr 172.21.15.28:0/1434896043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53a1400640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac15ab60 0x7f53ac06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 -- 172.21.15.28:0/1434896043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac15ab60 msgr2=0x7f53ac06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac15ab60 0x7f53ac06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 -- 172.21.15.28:0/1434896043 >> v1:172.21.15.189:6789/0 conn(0x7f53ac15a780 legacy=0x7f53ac070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.728+0000 7f53aa800640 1 -- 172.21.15.28:0/1434896043 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53ac0ccc60 con 0x7f53ac071490 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.729+0000 7f53aa800640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac071490 0x7f53ac06c080 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009e50 tx=0x7f53a402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c20ee25bb6309f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.729+0000 7f53a9400640 1 -- 172.21.15.28:0/1434896043 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4031600 con 0x7f53ac071490 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53a9400640 1 -- 172.21.15.28:0/1434896043 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a402d050 con 0x7f53ac071490 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53a9400640 1 -- 172.21.15.28:0/1434896043 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a4031980 con 0x7f53ac071490 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 -- 172.21.15.28:0/1434896043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac071490 msgr2=0x7f53ac06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac071490 0x7f53ac06c080 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009e50 tx=0x7f53a402a710 comp rx=0 tx=0).stop 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 -- 172.21.15.28:0/1434896043 shutdown_connections 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac071490 0x7f53ac06c080 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 --2- 172.21.15.28:0/1434896043 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac15ab60 0x7f53ac06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 -- 172.21.15.28:0/1434896043 >> 172.21.15.28:0/1434896043 conn(0x7f53ac0b0270 msgr2=0x7f53ac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:56.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 -- 172.21.15.28:0/1434896043 shutdown_connections 2024-09-13T17:11:56.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.730+0000 7f53b0a00640 1 -- 172.21.15.28:0/1434896043 wait complete. 2024-09-13T17:11:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.732+0000 7f53b0a00640 1 Processor -- start 2024-09-13T17:11:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.732+0000 7f53b0a00640 1 -- start start 2024-09-13T17:11:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 0x7f53ac0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 0x7f53ac1ebfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac14d880 con 0x7f53ac15a780 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac14ed30 con 0x7f53ac15ab60 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53b0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53ac1f3070 con 0x7f53ac071490 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53a1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60908/0 (socket says 172.21.15.28:60908) 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53a1400640 1 -- 172.21.15.28:0/2334197383 learned_addr learned my addr 172.21.15.28:0/2334197383 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53a1400640 1 -- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 msgr2=0x7f53ac0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53a9e00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 0x7f53ac0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.733+0000 7f53aa800640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 0x7f53ac1ebfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a1400640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 0x7f53ac0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a1400640 1 -- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 msgr2=0x7f53ac1ebfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a1400640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 0x7f53ac1ebfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a1400640 1 -- 172.21.15.28:0/2334197383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a4009b30 con 0x7f53ac15a780 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a9e00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 0x7f53ac0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53aa800640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 0x7f53ac1ebfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a1400640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f539400e870 tx=0x7f539400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53940094f0 con 0x7f53ac15a780 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539400c940 con 0x7f53ac15a780 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400ccf0 con 0x7f53ac15a780 2024-09-13T17:11:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53ac0acee0 con 0x7f53ac15a780 2024-09-13T17:11:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.734+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53ac0ad4a0 con 0x7f53ac15a780 2024-09-13T17:11:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.735+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac070d80 con 0x7f53ac15a780 2024-09-13T17:11:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.736+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5394013470 con 0x7f53ac15a780 2024-09-13T17:11:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.737+0000 7f53a2a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 0x7f538806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.737+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53940891e0 con 0x7f53ac15a780 2024-09-13T17:11:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.738+0000 7f53a9e00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 0x7f538806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.739+0000 7f53a9e00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 0x7f538806d0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f539c000f60 tx=0x7f539c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.742+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5394054620 con 0x7f53ac15a780 2024-09-13T17:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.919+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53ac06c080 con 0x7f53ac15a780 2024-09-13T17:11:56.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.919+0000 7f53a2a00640 1 -- 172.21.15.28:0/2334197383 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5394057c40 con 0x7f53ac15a780 2024-09-13T17:11:56.921 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 msgr2=0x7f538806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 0x7f538806d0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f539c000f60 tx=0x7f539c009040 comp rx=0 tx=0).stop 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 msgr2=0x7f53ac1f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f539400e870 tx=0x7f539400ed30 comp rx=0 tx=0).stop 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 shutdown_connections 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538806ac00 0x7f538806d0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53ac15ab60 0x7f53ac1ebfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53ac15a780 0x7f53ac1f2b30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 --2- 172.21.15.28:0/2334197383 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53ac071490 0x7f53ac0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 >> 172.21.15.28:0/2334197383 conn(0x7f53ac0b0270 msgr2=0x7f53ac0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 shutdown_connections 2024-09-13T17:11:56.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:56.922+0000 7f53b0a00640 1 -- 172.21.15.28:0/2334197383 wait complete. 2024-09-13T17:11:56.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags' 2024-09-13T17:11:57.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.093+0000 7f18ed400640 1 Processor -- start 2024-09-13T17:11:57.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.093+0000 7f18ed400640 1 -- start start 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.094+0000 7f18ed400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18e80cabc0 0x7f18e806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.094+0000 7f18ed400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 0x7f18e8071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.094+0000 7f18ed400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e815e100 con 0x7f18e815a540 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.094+0000 7f18ed400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e80adb00 con 0x7f18e80cabc0 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.094+0000 7f18ed400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e8071670 con 0x7f18e8070cc0 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18e80cabc0 0x7f18e806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18e80cabc0 0x7f18e806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60918/0 (socket says 172.21.15.28:60918) 2024-09-13T17:11:57.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 -- 172.21.15.28:0/2747357672 learned_addr learned my addr 172.21.15.28:0/2747357672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18e7200640 1 --2- 172.21.15.28:0/2747357672 >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 0x7f18e8071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 -- 172.21.15.28:0/2747357672 >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 msgr2=0x7f18e8071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 --2- 172.21.15.28:0/2747357672 >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 0x7f18e8071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 -- 172.21.15.28:0/2747357672 >> v1:172.21.15.189:6789/0 conn(0x7f18e815a540 legacy=0x7f18e815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18dfe00640 1 -- 172.21.15.28:0/2747357672 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e815e3e0 con 0x7f18e80cabc0 2024-09-13T17:11:57.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.095+0000 7f18e7200640 1 --2- 172.21.15.28:0/2747357672 >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 0x7f18e8071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.096+0000 7f18dfe00640 1 --2- 172.21.15.28:0/2747357672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e80cabc0 0x7f18e806c430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f18e8070900 tx=0x7f18d002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87bbb491c9aafd54 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.096+0000 7f18e5e00640 1 -- 172.21.15.28:0/2747357672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0032440 con 0x7f18e80cabc0 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.096+0000 7f18e5e00640 1 -- 172.21.15.28:0/2747357672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d00325a0 con 0x7f18e80cabc0 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.096+0000 7f18e5e00640 1 -- 172.21.15.28:0/2747357672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0032930 con 0x7f18e80cabc0 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 -- 172.21.15.28:0/2747357672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e80cabc0 msgr2=0x7f18e806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:57.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 --2- 172.21.15.28:0/2747357672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e80cabc0 0x7f18e806c430 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f18e8070900 tx=0x7f18d002a210 comp rx=0 tx=0).stop 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 -- 172.21.15.28:0/2747357672 shutdown_connections 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 --2- 172.21.15.28:0/2747357672 >> v2:172.21.15.189:3300/0 conn(0x7f18e8070cc0 0x7f18e8071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 --2- 172.21.15.28:0/2747357672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e80cabc0 0x7f18e806c430 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 -- 172.21.15.28:0/2747357672 >> 172.21.15.28:0/2747357672 conn(0x7f18e8069e80 msgr2=0x7f18e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 -- 172.21.15.28:0/2747357672 shutdown_connections 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.097+0000 7f18ed400640 1 -- 172.21.15.28:0/2747357672 wait complete. 2024-09-13T17:11:57.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.099+0000 7f18ed400640 1 Processor -- start 2024-09-13T17:11:57.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.099+0000 7f18ed400640 1 -- start start 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80cabc0 0x7f18e8144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e815a540 0x7f18e8145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8145590 con 0x7f18e815a540 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8156a30 con 0x7f18e80cabc0 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18ed400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8156cd0 con 0x7f18e8070cc0 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18e6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80cabc0 0x7f18e8144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18e6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:38300/0 (socket says 172.21.15.28:38300) 2024-09-13T17:11:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18e6800640 1 -- 172.21.15.28:0/1341957964 learned_addr learned my addr 172.21.15.28:0/1341957964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:57.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.100+0000 7f18e7200640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e815a540 0x7f18e8145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 -- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80cabc0 msgr2=0x7f18e8144b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80cabc0 0x7f18e8144b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 -- 172.21.15.28:0/1341957964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e815a540 msgr2=0x7f18e8145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e815a540 0x7f18e8145050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:57.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 -- 172.21.15.28:0/1341957964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d00094d0 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18e6800640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18d800c400 tx=0x7f18d800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d80052d0 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d8005430 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d8003830 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e8156fb0 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.101+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e8157570 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.102+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e8157700 con 0x7f18e8070cc0 2024-09-13T17:11:57.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.104+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f18d8003990 con 0x7f18e8070cc0 2024-09-13T17:11:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.104+0000 7f18dea00640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 0x7f18c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:57.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.105+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f18d80887f0 con 0x7f18e8070cc0 2024-09-13T17:11:57.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.105+0000 7f18dfe00640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 0x7f18c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:57.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.106+0000 7f18dfe00640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 0x7f18c406d0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f18d0034000 tx=0x7f18d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.108+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f18d8053d60 con 0x7f18e8070cc0 2024-09-13T17:11:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:57.353+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags"]} v 0) v1 -- 0x7f18e8071130 con 0x7f18e8070cc0 2024-09-13T17:11:58.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.185+0000 7f18dea00640 1 -- 172.21.15.28:0/1341957964 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags"]}]=0 v206) v1 ==== 154+0+0 (secure 0 0 0) 0x7f18d80170e0 con 0x7f18e8070cc0 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 msgr2=0x7f18c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 0x7f18c406d0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f18d0034000 tx=0x7f18d002e000 comp rx=0 tx=0).stop 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 msgr2=0x7f18e814da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18d800c400 tx=0x7f18d800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 shutdown_connections 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f18c406ac00 0x7f18c406d0c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18e815a540 0x7f18e8145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18e80cabc0 0x7f18e8144b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 --2- 172.21.15.28:0/1341957964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18e8070cc0 0x7f18e814da70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 >> 172.21.15.28:0/1341957964 conn(0x7f18e8069e80 msgr2=0x7f18e815e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:58.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 shutdown_connections 2024-09-13T17:11:58.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.189+0000 7f18ed400640 1 -- 172.21.15.28:0/1341957964 wait complete. 2024-09-13T17:11:58.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:11:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.371+0000 7fdc8fe00640 1 Processor -- start 2024-09-13T17:11:58.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.371+0000 7fdc8fe00640 1 -- start start 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.372+0000 7fdc8fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdc88070d20 0x7fdc8806cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.372+0000 7fdc8fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8815cc30 con 0x7fdc8815c850 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.372+0000 7fdc8fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc8806b020 con 0x7fdc880708e0 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.372+0000 7fdc8fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc8806b590 con 0x7fdc88070d20 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdc8815c850 0x7fdc8806dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56610/0 (socket says 172.21.15.28:56610) 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8d400640 1 -- 172.21.15.28:0/115784935 learned_addr learned my addr 172.21.15.28:0/115784935 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8de00640 1 --2- 172.21.15.28:0/115784935 >> v2:172.21.15.28:3300/0 conn(0x7fdc88070d20 0x7fdc8806cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:58.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8de00640 1 -- 172.21.15.28:0/115784935 >> v1:172.21.15.189:6789/0 conn(0x7fdc880708e0 legacy=0x7fdc8806c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8de00640 1 -- 172.21.15.28:0/115784935 >> v1:172.21.15.28:6789/0 conn(0x7fdc8815c850 legacy=0x7fdc8806dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.373+0000 7fdc8de00640 1 -- 172.21.15.28:0/115784935 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8806ad40 con 0x7fdc88070d20 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.374+0000 7fdc8de00640 1 --2- 172.21.15.28:0/115784935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc88070d20 0x7fdc8806cbf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009d90 tx=0x7fdc8002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b695e1e17a09727 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.374+0000 7fdc87e00640 1 -- 172.21.15.28:0/115784935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80007740 con 0x7fdc88070d20 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc87e00640 1 -- 172.21.15.28:0/115784935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc8002d050 con 0x7fdc88070d20 2024-09-13T17:11:58.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc87e00640 1 -- 172.21.15.28:0/115784935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80007ac0 con 0x7fdc88070d20 2024-09-13T17:11:58.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 -- 172.21.15.28:0/115784935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc88070d20 msgr2=0x7fdc8806cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/115784935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc88070d20 0x7fdc8806cbf0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009d90 tx=0x7fdc8002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 -- 172.21.15.28:0/115784935 shutdown_connections 2024-09-13T17:11:58.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/115784935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc88070d20 0x7fdc8806cbf0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 -- 172.21.15.28:0/115784935 >> 172.21.15.28:0/115784935 conn(0x7fdc880b1740 msgr2=0x7fdc880b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:58.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 -- 172.21.15.28:0/115784935 shutdown_connections 2024-09-13T17:11:58.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.375+0000 7fdc8fe00640 1 -- 172.21.15.28:0/115784935 wait complete. 2024-09-13T17:11:58.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.377+0000 7fdc8fe00640 1 Processor -- start 2024-09-13T17:11:58.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.377+0000 7fdc8fe00640 1 -- start start 2024-09-13T17:11:58.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc880708e0 0x7fdc88152430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:58.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc88070d20 0x7fdc8814e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:58.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8815ded0 con 0x7fdc8815c850 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8815e1b0 con 0x7fdc88070d20 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc8815e490 con 0x7fdc880708e0 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60940/0 (socket says 172.21.15.28:60940) 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8de00640 1 -- 172.21.15.28:0/3398122395 learned_addr learned my addr 172.21.15.28:0/3398122395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8d400640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc880708e0 0x7fdc88152430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.378+0000 7fdc8ca00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc88070d20 0x7fdc8814e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 -- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc880708e0 msgr2=0x7fdc88152430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc880708e0 0x7fdc88152430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 -- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc88070d20 msgr2=0x7fdc8814e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc88070d20 0x7fdc8814e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 -- 172.21.15.28:0/3398122395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc80009a70 con 0x7fdc8815c850 2024-09-13T17:11:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8de00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009eb0 tx=0x7fdc80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80031910 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc80031a70 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc8815e770 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.379+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc80031de0 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.380+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc880cd130 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.380+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc8815ec90 con 0x7fdc8815c850 2024-09-13T17:11:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.381+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdc8003c420 con 0x7fdc8815c850 2024-09-13T17:11:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.382+0000 7fdc85600640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 0x7fdc6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.382+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdc800aadf0 con 0x7fdc8815c850 2024-09-13T17:11:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.383+0000 7fdc8d400640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 0x7fdc6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.383+0000 7fdc8d400640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 0x7fdc6806d0c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c005880 tx=0x7fdc7c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.386+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdc8815ec90 con 0x7fdc8815c850 2024-09-13T17:11:58.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.565+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdc8815ec90 con 0x7fdc8815c850 2024-09-13T17:11:58.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.569+0000 7fdc85600640 1 -- 172.21.15.28:0/3398122395 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdc8815ec90 con 0x7fdc8815c850 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 msgr2=0x7fdc6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 0x7fdc6806d0c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdc7c005880 tx=0x7fdc7c009040 comp rx=0 tx=0).stop 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 msgr2=0x7fdc880ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fdc80009eb0 tx=0x7fdc80004a40 comp rx=0 tx=0).stop 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 shutdown_connections 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdc6806ac00 0x7fdc6806d0c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc8815c850 0x7fdc880ce7e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc88070d20 0x7fdc8814e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 --2- 172.21.15.28:0/3398122395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdc880708e0 0x7fdc88152430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 >> 172.21.15.28:0/3398122395 conn(0x7fdc880b1740 msgr2=0x7fdc8815b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 shutdown_connections 2024-09-13T17:11:58.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:58.572+0000 7fdc8fe00640 1 -- 172.21.15.28:0/3398122395 wait complete. 2024-09-13T17:11:58.589 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:11:58.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:58.691 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:11:58.787 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:11:58.787 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:58.787 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:58.822 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:11:58.823 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:58.823 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:58.859 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:58.860 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:11:58.860 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:58.898 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:11:58.899 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:11:58.899 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:11:58.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:11:58.935 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:11:58.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:11:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.094+0000 7f0009400640 1 Processor -- start 2024-09-13T17:11:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.094+0000 7f0009400640 1 -- start start 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00041592a0 0x7f000415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 0x7f0004159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 0x7f000406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000415b0b0 con 0x7f00041592a0 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000406cca0 con 0x7f000406c2d0 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0009400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00040b2040 con 0x7f000406f970 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0002800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00041592a0 0x7f000415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0002800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f00041592a0 0x7f000415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60942/0 (socket says 172.21.15.28:60942) 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.095+0000 7f0002800640 1 -- 172.21.15.28:0/2657136613 learned_addr learned my addr 172.21.15.28:0/2657136613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0003200640 1 --2- 172.21.15.28:0/2657136613 >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 0x7f000406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7efffbe00640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 0x7f0004159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0002800640 1 -- 172.21.15.28:0/2657136613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 msgr2=0x7f0004159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0002800640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 0x7f0004159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0002800640 1 -- 172.21.15.28:0/2657136613 >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 msgr2=0x7f000406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0002800640 1 --2- 172.21.15.28:0/2657136613 >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 0x7f000406c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0002800640 1 -- 172.21.15.28:0/2657136613 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004159ff0 con 0x7f00041592a0 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7efffbe00640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 0x7f0004159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:59.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.096+0000 7f0003200640 1 --2- 172.21.15.28:0/2657136613 >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 0x7f000406c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.097+0000 7f0002800640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f000415ab70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f000406f700 tx=0x7efff402a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da46bccb4b17051 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.097+0000 7f0001e00640 1 -- 172.21.15.28:0/2657136613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4032440 con 0x7f00041592a0 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.097+0000 7f0001e00640 1 -- 172.21.15.28:0/2657136613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff40325a0 con 0x7f00041592a0 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.097+0000 7f0001e00640 1 -- 172.21.15.28:0/2657136613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff4032930 con 0x7f00041592a0 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 -- 172.21.15.28:0/2657136613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 msgr2=0x7f000415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f000415ab70 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f000406f700 tx=0x7efff402a1d0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 -- 172.21.15.28:0/2657136613 shutdown_connections 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 --2- 172.21.15.28:0/2657136613 >> v2:172.21.15.189:3300/0 conn(0x7f000406c2d0 0x7f000406c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406f970 0x7f0004159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 --2- 172.21.15.28:0/2657136613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f000415ab70 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 -- 172.21.15.28:0/2657136613 >> 172.21.15.28:0/2657136613 conn(0x7f0004069e80 msgr2=0x7f000406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 -- 172.21.15.28:0/2657136613 shutdown_connections 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.098+0000 7f0009400640 1 -- 172.21.15.28:0/2657136613 wait complete. 2024-09-13T17:11:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.100+0000 7f0009400640 1 Processor -- start 2024-09-13T17:11:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.100+0000 7f0009400640 1 -- start start 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406c2d0 0x7f0004150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f000406f970 0x7f00040c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040ac360 con 0x7f00041592a0 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00040ac640 con 0x7f000406f970 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0009400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00040ac920 con 0x7f000406c2d0 2024-09-13T17:11:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0003200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0002800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406c2d0 0x7f0004150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7efffbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f000406f970 0x7f00040c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0003200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60958/0 (socket says 172.21.15.28:60958) 2024-09-13T17:11:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.101+0000 7f0003200640 1 -- 172.21.15.28:0/55901348 learned_addr learned my addr 172.21.15.28:0/55901348 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 -- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406c2d0 msgr2=0x7f0004150210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406c2d0 0x7f0004150210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 -- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f000406f970 msgr2=0x7f00040c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f000406f970 0x7f00040c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 -- 172.21.15.28:0/55901348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff40094d0 con 0x7f00041592a0 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0003200640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efffc009fa0 tx=0x7efffc00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc003b90 con 0x7f00041592a0 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efffc00f040 con 0x7f00041592a0 2024-09-13T17:11:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00040acc00 con 0x7f00041592a0 2024-09-13T17:11:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efffc004030 con 0x7f00041592a0 2024-09-13T17:11:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.102+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0004150750 con 0x7f00041592a0 2024-09-13T17:11:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.103+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00041f0e20 con 0x7f00041592a0 2024-09-13T17:11:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.104+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efffc004190 con 0x7f00041592a0 2024-09-13T17:11:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.105+0000 7efffaa00640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 0x7effe006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.106+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efffc089400 con 0x7f00041592a0 2024-09-13T17:11:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.106+0000 7f0002800640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 0x7effe006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.107+0000 7f0002800640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 0x7effe006d0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efff4034000 tx=0x7efff402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.110+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f00041f0e20 con 0x7f00041592a0 2024-09-13T17:11:59.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.324+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00041f0e20 con 0x7f00041592a0 2024-09-13T17:11:59.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.325+0000 7efffaa00640 1 -- 172.21.15.28:0/55901348 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3532 (secure 0 0 0) 0x7efffc01ade0 con 0x7f00041592a0 2024-09-13T17:11:59.326 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:11:59.326 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5090,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1287017369","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1287017369},{"type":"v1","addr":"172.21.15.189:6836","nonce":1287017369}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5443,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3900584251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3900584251},{"type":"v1","addr":"172.21.15.189:6837","nonce":3900584251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":5452,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/210793746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":210793746},{"type":"v1","addr":"172.21.15.28:6835","nonce":210793746}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":30},{"gid":6330,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/4035713209","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4035713209},{"type":"v1","addr":"172.21.15.28:6833","nonce":4035713209}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[]} 2024-09-13T17:11:59.327 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 43 2024-09-13T17:11:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 msgr2=0x7effe006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 0x7effe006d0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efff4034000 tx=0x7efff402e000 comp rx=0 tx=0).stop 2024-09-13T17:11:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 msgr2=0x7f00040c66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7efffc009fa0 tx=0x7efffc00d560 comp rx=0 tx=0).stop 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 shutdown_connections 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7effe006ac00 0x7effe006d0c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00041592a0 0x7f00040c66f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f000406f970 0x7f00040c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 --2- 172.21.15.28:0/55901348 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f000406c2d0 0x7f0004150210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 >> 172.21.15.28:0/55901348 conn(0x7f0004069e80 msgr2=0x7f00040b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 shutdown_connections 2024-09-13T17:11:59.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.327+0000 7f0009400640 1 -- 172.21.15.28:0/55901348 wait complete. 2024-09-13T17:11:59.345 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:11:59.345 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:11:59.345 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:11:59.345 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:59.346 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:11:59.346 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:11:59.346 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:11:59.346 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:59.346 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:11:59.346 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:11:59.346 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:11:59.347 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:59.347 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:11:59.347 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:11:59.347 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:11:59.347 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:11:59.362 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:11:59.362 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:11:59.363 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:11:59.363 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:11:59.365 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:11:59.365 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:11:59.365 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:11:59.366 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:11:59.407 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:11:59.408 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:59.408 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:11:59.409 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:59.409 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:11:59.410 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:59.411 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:11:59.411 DEBUG:teuthology.parallel:result is None 2024-09-13T17:11:59.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:11:59.537 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:11:59.537 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:11:59.538 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:11:59.538 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:11:59.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.578+0000 7f512a600640 1 Processor -- start 2024-09-13T17:11:59.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.579+0000 7f512a600640 1 -- start start 2024-09-13T17:11:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f512a600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f512415a780 0x7f5124070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f512a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f512415ab60 0x7f51240712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f512a600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512406c660 con 0x7f512406bb40 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f512a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512406c8b0 con 0x7f512415ab60 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f512a600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512406cb90 con 0x7f512415a780 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f511aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f512415ab60 0x7f51240712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f511aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f512415ab60 0x7f51240712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60988/0 (socket says 172.21.15.28:60988) 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f511aa00640 1 -- 172.21.15.28:0/2692212964 learned_addr learned my addr 172.21.15.28:0/2692212964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f5123400640 1 --2- 172.21.15.28:0/2692212964 >> v2:172.21.15.189:3300/0 conn(0x7f512415a780 0x7f5124070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f511aa00640 1 -- 172.21.15.28:0/2692212964 >> v2:172.21.15.189:3300/0 conn(0x7f512415a780 msgr2=0x7f5124070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.580+0000 7f511aa00640 1 --2- 172.21.15.28:0/2692212964 >> v2:172.21.15.189:3300/0 conn(0x7f512415a780 0x7f5124070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.581+0000 7f511aa00640 1 -- 172.21.15.28:0/2692212964 >> v1:172.21.15.189:6789/0 conn(0x7f512406bb40 legacy=0x7f512406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.581+0000 7f511aa00640 1 -- 172.21.15.28:0/2692212964 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51240cc980 con 0x7f512415ab60 2024-09-13T17:11:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.581+0000 7f511aa00640 1 --2- 172.21.15.28:0/2692212964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512415ab60 0x7f51240712c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f51240b1330 tx=0x7f510c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408170a4e21005d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f5122a00640 1 -- 172.21.15.28:0/2692212964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c032440 con 0x7f512415ab60 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f5122a00640 1 -- 172.21.15.28:0/2692212964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c0325a0 con 0x7f512415ab60 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f5122a00640 1 -- 172.21.15.28:0/2692212964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c032930 con 0x7f512415ab60 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 -- 172.21.15.28:0/2692212964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512415ab60 msgr2=0x7f51240712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 --2- 172.21.15.28:0/2692212964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512415ab60 0x7f51240712c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f51240b1330 tx=0x7f510c02a270 comp rx=0 tx=0).stop 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 -- 172.21.15.28:0/2692212964 shutdown_connections 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 --2- 172.21.15.28:0/2692212964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512415ab60 0x7f51240712c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 --2- 172.21.15.28:0/2692212964 >> v2:172.21.15.189:3300/0 conn(0x7f512415a780 0x7f5124070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 -- 172.21.15.28:0/2692212964 >> 172.21.15.28:0/2692212964 conn(0x7f51240b0270 msgr2=0x7f51240b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.582+0000 7f512a600640 1 -- 172.21.15.28:0/2692212964 shutdown_connections 2024-09-13T17:11:59.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.583+0000 7f512a600640 1 -- 172.21.15.28:0/2692212964 wait complete. 2024-09-13T17:11:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.584+0000 7f512a600640 1 Processor -- start 2024-09-13T17:11:59.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.584+0000 7f512a600640 1 -- start start 2024-09-13T17:11:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 0x7f512414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f512415ab60 0x7f5124150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51240ad620 con 0x7f512406bb40 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51240a9c60 con 0x7f512415a780 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f512a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51240ac9a0 con 0x7f512415ab60 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f5123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f5123400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60998/0 (socket says 172.21.15.28:60998) 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.585+0000 7f5123400640 1 -- 172.21.15.28:0/2128543358 learned_addr learned my addr 172.21.15.28:0/2128543358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f511aa00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 0x7f512414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123e00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f512415ab60 0x7f5124150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 -- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f512415ab60 msgr2=0x7f5124150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f512415ab60 0x7f5124150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 -- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 msgr2=0x7f512414fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 0x7f512414fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 -- 172.21.15.28:0/2128543358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510c0094d0 con 0x7f512406bb40 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f511aa00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 0x7f512414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f5123400640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f511400d830 tx=0x7f511400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511400b6c0 con 0x7f512406bb40 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f511400b840 con 0x7f512406bb40 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51240acc80 con 0x7f512406bb40 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511400bbf0 con 0x7f512406bb40 2024-09-13T17:11:59.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.586+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51240ad240 con 0x7f512406bb40 2024-09-13T17:11:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.587+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51240ad760 con 0x7f512406bb40 2024-09-13T17:11:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.588+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f511401e460 con 0x7f512406bb40 2024-09-13T17:11:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.589+0000 7f511be00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 0x7f510006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:11:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.589+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5114089890 con 0x7f512406bb40 2024-09-13T17:11:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.589+0000 7f511aa00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 0x7f510006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:11:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.590+0000 7f511aa00640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 0x7f510006d0c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f510c034000 tx=0x7f510c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:11:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.593+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5114054d60 con 0x7f512406bb40 2024-09-13T17:11:59.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:11:59.782+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f512406bf50 con 0x7f512406bb40 2024-09-13T17:12:00.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.574+0000 7f511be00640 1 -- 172.21.15.28:0/2128543358 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5114058380 con 0x7f512406bb40 2024-09-13T17:12:00.575 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:12:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 msgr2=0x7f510006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 0x7f510006d0c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f510c034000 tx=0x7f510c02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 msgr2=0x7f5124152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f511400d830 tx=0x7f511400dcf0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 shutdown_connections 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f510006ac00 0x7f510006d0c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f512415ab60 0x7f5124150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f512415a780 0x7f512414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 --2- 172.21.15.28:0/2128543358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f512406bb40 0x7f5124152e50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.577+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 >> 172.21.15.28:0/2128543358 conn(0x7f51240b0270 msgr2=0x7f51240cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.578+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 shutdown_connections 2024-09-13T17:12:00.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.578+0000 7f512a600640 1 -- 172.21.15.28:0/2128543358 wait complete. 2024-09-13T17:12:00.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:12:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.757+0000 7fdcd6e00640 1 Processor -- start 2024-09-13T17:12:00.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.758+0000 7fdcd6e00640 1 -- start start 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdcd0154a90 0x7fdcd015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcd00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd015bb10 con 0x7fdcd006b360 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd006b900 con 0x7fdcd0154a90 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd006bbe0 con 0x7fdcd00b0b50 2024-09-13T17:12:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdcd006b360 0x7fdcd00b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56642/0 (socket says 172.21.15.28:56642) 2024-09-13T17:12:00.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd5000640 1 -- 172.21.15.28:0/2817548646 learned_addr learned my addr 172.21.15.28:0/2817548646 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd6400640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcd00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.759+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> v2:172.21.15.189:3300/0 conn(0x7fdcd0154a90 0x7fdcd015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595109785 0 0) 0x7fdcd015bb10 con 0x7fdcd006b360 2024-09-13T17:12:00.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb8003520 con 0x7fdcd006b360 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3178140787 0 0) 0x7fdcb8003520 con 0x7fdcd006b360 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 msgr2=0x7fdcd00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcd00af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 >> v2:172.21.15.189:3300/0 conn(0x7fdcd0154a90 msgr2=0x7fdcd015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 --2- 172.21.15.28:0/2817548646 >> v2:172.21.15.189:3300/0 conn(0x7fdcd0154a90 0x7fdcd015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd006bec0 con 0x7fdcd006b360 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fdcbc001690 con 0x7fdcd006b360 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> v2:172.21.15.189:3300/0 conn(0x7fdcd0154a90 0x7fdcd015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 >> v1:172.21.15.28:6789/0 conn(0x7fdcd006b360 legacy=0x7fdcd00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb8003970 0x7fdcb8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcb80047c0 0x7fdcb8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcb8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd006b900 con 0x7fdcb8003970 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd006bbe0 con 0x7fdcb80047c0 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb800b980 con 0x7fdcd00b0b50 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.760+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb8003970 0x7fdcb8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd6400640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcb8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5000640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcb80047c0 0x7fdcb8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5a00640 1 -- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 msgr2=0x7fdcb8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcd00b0b50 0x7fdcb8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5a00640 1 -- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcb80047c0 msgr2=0x7fdcb8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcb80047c0 0x7fdcb8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5a00640 1 -- 172.21.15.28:0/2817548646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb8003520 con 0x7fdcb8003970 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.761+0000 7fdcd5000640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcb80047c0 0x7fdcb8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:00.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.762+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb8003970 0x7fdcb8003d90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009cc0 tx=0x7fdcb80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d08ec8915a032b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.762+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4031400 con 0x7fdcb8003970 2024-09-13T17:12:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.762+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc402d050 con 0x7fdcb8003970 2024-09-13T17:12:00.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.762+0000 7fdccfe00640 1 -- 172.21.15.28:0/2817548646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4031760 con 0x7fdcb8003970 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2817548646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb8003970 msgr2=0x7fdcb8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2817548646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcb8003970 0x7fdcb8003d90 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009cc0 tx=0x7fdcb80057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6400640 1 -- 172.21.15.28:0/2817548646 reap_dead start 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2817548646 shutdown_connections 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2817548646 >> 172.21.15.28:0/2817548646 conn(0x7fdcd0072e40 msgr2=0x7fdcd0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:00.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2817548646 shutdown_connections 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.763+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2817548646 wait complete. 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.765+0000 7fdcd6e00640 1 Processor -- start 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.765+0000 7fdcd6e00640 1 -- start start 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcb8003970 0x7fdcd015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcd00a9050 0x7fdcd01f3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00a9d90 con 0x7fdcd01f3c50 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00a8bc0 con 0x7fdcd00a9050 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01f4640 con 0x7fdcb8003970 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32792/0 (socket says 172.21.15.28:32792) 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 -- 172.21.15.28:0/2564511342 learned_addr learned my addr 172.21.15.28:0/2564511342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcb8003970 0x7fdcd015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd5000640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcd00a9050 0x7fdcd01f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 -- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcb8003970 msgr2=0x7fdcd015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcb8003970 0x7fdcd015c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 -- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcd00a9050 msgr2=0x7fdcd01f3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcd00a9050 0x7fdcd01f3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.766+0000 7fdcd6400640 1 -- 172.21.15.28:0/2564511342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd01f4090 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdcd6400640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdcd015e080 tx=0x7fdcc800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800e040 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc800f040 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8015430 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd01f4360 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.767+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd00a8850 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.768+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd00a8d50 con 0x7fdcd01f3c50 2024-09-13T17:12:00.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.769+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdcc8015590 con 0x7fdcd01f3c50 2024-09-13T17:12:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.770+0000 7fdccd600640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 0x7fdcb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.770+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdcc8089930 con 0x7fdcd01f3c50 2024-09-13T17:12:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.771+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 0x7fdcb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:00.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.772+0000 7fdcd5a00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 0x7fdcb006d0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4033000 tx=0x7fdcc4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:00.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:00.775+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdcc8054e80 con 0x7fdcd01f3c50 2024-09-13T17:12:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.025+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fdcd015b5d0 con 0x7fdcd01f3c50 2024-09-13T17:12:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.025+0000 7fdccd600640 1 -- 172.21.15.28:0/2564511342 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fdcc80584a0 con 0x7fdcd01f3c50 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:01.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:01.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:12:01.030 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:12:01.033 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:12:01.035 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:12:01.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:12:01.037 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:01.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:01.040 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:01.041 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:12:01.042 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:01.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.042 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:01.042 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:12:01.042 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 msgr2=0x7fdcb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 0x7fdcb006d0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4033000 tx=0x7fdcc4008030 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 msgr2=0x7fdcd013add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fdcd015e080 tx=0x7fdcc800f1b0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 shutdown_connections 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcb006ac00 0x7fdcb006d0c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcd01f3c50 0x7fdcd013add0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcd00a9050 0x7fdcd01f3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 --2- 172.21.15.28:0/2564511342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcb8003970 0x7fdcd015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 >> 172.21.15.28:0/2564511342 conn(0x7fdcd0072e40 msgr2=0x7fdcd00af870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 shutdown_connections 2024-09-13T17:12:01.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.028+0000 7fdcd6e00640 1 -- 172.21.15.28:0/2564511342 wait complete. 2024-09-13T17:12:01.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.201+0000 7f6294800640 1 Processor -- start 2024-09-13T17:12:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.201+0000 7f6294800640 1 -- start start 2024-09-13T17:12:01.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.202+0000 7f6294800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f628c1592a0 0x7f628c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.202+0000 7f6294800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 0x7f628c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.202+0000 7f6294800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0b2040 con 0x7f628c06c2d0 2024-09-13T17:12:01.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6294800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c15b0b0 con 0x7f628c1592a0 2024-09-13T17:12:01.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6294800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c159ff0 con 0x7f628c06f970 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f628c1592a0 0x7f628c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f628c1592a0 0x7f628c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32806/0 (socket says 172.21.15.28:32806) 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 -- 172.21.15.28:0/3336132471 learned_addr learned my addr 172.21.15.28:0/3336132471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 -- 172.21.15.28:0/3336132471 >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 msgr2=0x7f628c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291400640 1 --2- 172.21.15.28:0/3336132471 >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 0x7f628c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 --2- 172.21.15.28:0/3336132471 >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 0x7f628c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 -- 172.21.15.28:0/3336132471 >> v1:172.21.15.189:6789/0 conn(0x7f628c06c2d0 legacy=0x7f628c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291e00640 1 -- 172.21.15.28:0/3336132471 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c15a2d0 con 0x7f628c1592a0 2024-09-13T17:12:01.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.203+0000 7f6291400640 1 --2- 172.21.15.28:0/3336132471 >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 0x7f628c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.204+0000 7f6291e00640 1 --2- 172.21.15.28:0/3336132471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1592a0 0x7f628c15ab70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f628c06ac10 tx=0x7f628002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500b5efef4e74b6a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.204+0000 7f6290a00640 1 -- 172.21.15.28:0/3336132471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280032440 con 0x7f628c1592a0 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.204+0000 7f6290a00640 1 -- 172.21.15.28:0/3336132471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62800325a0 con 0x7f628c1592a0 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.204+0000 7f6290a00640 1 -- 172.21.15.28:0/3336132471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280032930 con 0x7f628c1592a0 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 -- 172.21.15.28:0/3336132471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1592a0 msgr2=0x7f628c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 --2- 172.21.15.28:0/3336132471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1592a0 0x7f628c15ab70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f628c06ac10 tx=0x7f628002a230 comp rx=0 tx=0).stop 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 -- 172.21.15.28:0/3336132471 shutdown_connections 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 --2- 172.21.15.28:0/3336132471 >> v2:172.21.15.189:3300/0 conn(0x7f628c06f970 0x7f628c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 --2- 172.21.15.28:0/3336132471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c1592a0 0x7f628c15ab70 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 -- 172.21.15.28:0/3336132471 >> 172.21.15.28:0/3336132471 conn(0x7f628c069e80 msgr2=0x7f628c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 -- 172.21.15.28:0/3336132471 shutdown_connections 2024-09-13T17:12:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.205+0000 7f6294800640 1 -- 172.21.15.28:0/3336132471 wait complete. 2024-09-13T17:12:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.207+0000 7f6294800640 1 Processor -- start 2024-09-13T17:12:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.207+0000 7f6294800640 1 -- start start 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 0x7f628c14f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f628c1592a0 0x7f628c0b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c14f890 con 0x7f628c06f970 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0ae140 con 0x7f628c06c2d0 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6294800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0ae3c0 con 0x7f628c1592a0 2024-09-13T17:12:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6291400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6291400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32818/0 (socket says 172.21.15.28:32818) 2024-09-13T17:12:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.208+0000 7f6291400640 1 -- 172.21.15.28:0/1766298650 learned_addr learned my addr 172.21.15.28:0/1766298650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 -- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f628c1592a0 msgr2=0x7f628c0b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f628c1592a0 0x7f628c0b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291e00640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 0x7f628c14f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 -- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 msgr2=0x7f628c14f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 0x7f628c14f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 -- 172.21.15.28:0/1766298650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800094d0 con 0x7f628c06f970 2024-09-13T17:12:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291e00640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 0x7f628c14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:01.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6291400640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c820 tx=0x7f627c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c007a30 con 0x7f628c06f970 2024-09-13T17:12:01.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c007b90 con 0x7f628c06f970 2024-09-13T17:12:01.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00ce70 con 0x7f628c06f970 2024-09-13T17:12:01.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c0ae6a0 con 0x7f628c06f970 2024-09-13T17:12:01.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.209+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c0aec60 con 0x7f628c06f970 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.210+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c15ab70 con 0x7f628c06f970 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.211+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f627c0113f0 con 0x7f628c06f970 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.212+0000 7f6286000640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 0x7f627006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.212+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f627c0891a0 con 0x7f628c06f970 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.213+0000 7f6291e00640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 0x7f627006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.213+0000 7f6291e00640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 0x7f627006d0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6280034000 tx=0x7f628002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.217+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f627c0546c0 con 0x7f628c06f970 2024-09-13T17:12:01.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.396+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f628c06ca70 con 0x7f628c06f970 2024-09-13T17:12:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.397+0000 7f6286000640 1 -- 172.21.15.28:0/1766298650 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+12857 (secure 0 0 0) 0x7f627c057ce0 con 0x7f628c06f970 2024-09-13T17:12:01.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:01.399 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":57,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:00.570074+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 msgr2=0x7f627006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 0x7f627006d0c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f6280034000 tx=0x7f628002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 msgr2=0x7f628c0adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c820 tx=0x7f627c00cc20 comp rx=0 tx=0).stop 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 shutdown_connections 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627006ac00 0x7f627006d0c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f628c1592a0 0x7f628c0b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f628c06f970 0x7f628c0adc00 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 --2- 172.21.15.28:0/1766298650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f628c06c2d0 0x7f628c14f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.399+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 >> 172.21.15.28:0/1766298650 conn(0x7f628c069e80 msgr2=0x7f628c15c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.400+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 shutdown_connections 2024-09-13T17:12:01.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.400+0000 7f6294800640 1 -- 172.21.15.28:0/1766298650 wait complete. 2024-09-13T17:12:01.416 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:12:01.416 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:12:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.574+0000 7ff278c00640 1 Processor -- start 2024-09-13T17:12:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.574+0000 7ff278c00640 1 -- start start 2024-09-13T17:12:01.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff278c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2740b2dc0 0x7ff274069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff278c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff27406a440 0x7ff27406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff278c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740ae200 con 0x7ff27415c900 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff278c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff27406ae10 con 0x7ff27406a440 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff278c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740ae4e0 con 0x7ff2740b2dc0 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff273e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff27406a440 0x7ff27406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff273e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff27406a440 0x7ff27406a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32826/0 (socket says 172.21.15.28:32826) 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.575+0000 7ff273e00640 1 -- 172.21.15.28:0/1901920519 learned_addr learned my addr 172.21.15.28:0/1901920519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.576+0000 7ff26be00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2740b2dc0 0x7ff274069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.576+0000 7ff273e00640 1 -- 172.21.15.28:0/1901920519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2740b2dc0 msgr2=0x7ff274069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.576+0000 7ff273e00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2740b2dc0 0x7ff274069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.576+0000 7ff273e00640 1 -- 172.21.15.28:0/1901920519 >> v1:172.21.15.189:6789/0 conn(0x7ff27415c900 legacy=0x7ff2740af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.576+0000 7ff273e00640 1 -- 172.21.15.28:0/1901920519 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2740ae7c0 con 0x7ff27406a440 2024-09-13T17:12:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.577+0000 7ff273e00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27406a440 0x7ff27406a8d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36519d457712c19b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.577+0000 7ff272a00640 1 -- 172.21.15.28:0/1901920519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c031600 con 0x7ff27406a440 2024-09-13T17:12:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.577+0000 7ff272a00640 1 -- 172.21.15.28:0/1901920519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c02d050 con 0x7ff27406a440 2024-09-13T17:12:01.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.577+0000 7ff272a00640 1 -- 172.21.15.28:0/1901920519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c031980 con 0x7ff27406a440 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 -- 172.21.15.28:0/1901920519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27406a440 msgr2=0x7ff27406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27406a440 0x7ff27406a8d0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c02a710 comp rx=0 tx=0).stop 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 -- 172.21.15.28:0/1901920519 shutdown_connections 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27406a440 0x7ff27406a8d0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 --2- 172.21.15.28:0/1901920519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2740b2dc0 0x7ff274069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 -- 172.21.15.28:0/1901920519 >> 172.21.15.28:0/1901920519 conn(0x7ff2740b2150 msgr2=0x7ff2740b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 -- 172.21.15.28:0/1901920519 shutdown_connections 2024-09-13T17:12:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.578+0000 7ff278c00640 1 -- 172.21.15.28:0/1901920519 wait complete. 2024-09-13T17:12:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.579+0000 7ff278c00640 1 Processor -- start 2024-09-13T17:12:01.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.580+0000 7ff278c00640 1 -- start start 2024-09-13T17:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.580+0000 7ff278c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff27406a440 0x7ff274155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff278c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2740b2dc0 0x7ff2740b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff278c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff278c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff274156170 con 0x7ff27415c900 2024-09-13T17:12:01.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff278c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b3f20 con 0x7ff2740b2dc0 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff278c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b43b0 con 0x7ff27406a440 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32834/0 (socket says 172.21.15.28:32834) 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 -- 172.21.15.28:0/31584944 learned_addr learned my addr 172.21.15.28:0/31584944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff26be00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2740b2dc0 0x7ff2740b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273400640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff27406a440 0x7ff274155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 -- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff27406a440 msgr2=0x7ff274155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff27406a440 0x7ff274155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 -- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2740b2dc0 msgr2=0x7ff2740b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2740b2dc0 0x7ff2740b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.581+0000 7ff273e00640 1 -- 172.21.15.28:0/31584944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26c009b30 con 0x7ff27415c900 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff273e00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c03d860 con 0x7ff27415c900 2024-09-13T17:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c03d9c0 con 0x7ff27415c900 2024-09-13T17:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2741f65e0 con 0x7ff27415c900 2024-09-13T17:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c03dd70 con 0x7ff27415c900 2024-09-13T17:12:01.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.582+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2740b38d0 con 0x7ff27415c900 2024-09-13T17:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.583+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27415cce0 con 0x7ff27415c900 2024-09-13T17:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.584+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff26c03b400 con 0x7ff27415c900 2024-09-13T17:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.585+0000 7ff26b400640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 0x7ff25006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.585+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff26c0a9e00 con 0x7ff27415c900 2024-09-13T17:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.585+0000 7ff273400640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 0x7ff25006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.586+0000 7ff273400640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 0x7ff25006d0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff264000f60 tx=0x7ff264009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.590+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff26c075340 con 0x7ff27415c900 2024-09-13T17:12:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.768+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2740b0580 con 0x7ff27415c900 2024-09-13T17:12:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.769+0000 7ff26b400640 1 -- 172.21.15.28:0/31584944 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v57) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff26c036020 con 0x7ff27415c900 2024-09-13T17:12:01.770 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:12:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 msgr2=0x7ff25006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 0x7ff25006d0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7ff264000f60 tx=0x7ff264009040 comp rx=0 tx=0).stop 2024-09-13T17:12:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 msgr2=0x7ff2740b5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff26c009e50 tx=0x7ff26c004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 shutdown_connections 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff25006ac00 0x7ff25006d0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff27415c900 0x7ff2740b5200 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2740b2dc0 0x7ff2740b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 --2- 172.21.15.28:0/31584944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff27406a440 0x7ff274155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 >> 172.21.15.28:0/31584944 conn(0x7ff2740b2150 msgr2=0x7ff2740d0500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 shutdown_connections 2024-09-13T17:12:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.771+0000 7ff278c00640 1 -- 172.21.15.28:0/31584944 wait complete. 2024-09-13T17:12:01.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:12:01.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.943+0000 7f7b11200640 1 Processor -- start 2024-09-13T17:12:01.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.943+0000 7f7b11200640 1 -- start start 2024-09-13T17:12:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b11200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b11200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c06c8b0 con 0x7f7b0c06c250 2024-09-13T17:12:01.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b11200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b0c06cb90 con 0x7f7b0c15ab60 2024-09-13T17:12:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b11200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c06c6a0 con 0x7f7b0c15a780 2024-09-13T17:12:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b0a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7b0c06c250 0x7f7b0c0cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56652/0 (socket says 172.21.15.28:56652) 2024-09-13T17:12:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.944+0000 7f7b0a800640 1 -- 172.21.15.28:0/2397952169 learned_addr learned my addr 172.21.15.28:0/2397952169 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09e00640 1 --2- 172.21.15.28:0/2397952169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70368581 0 0) 0x7f7b0c06c8b0 con 0x7f7b0c06c250 2024-09-13T17:12:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7af0003520 con 0x7f7b0c06c250 2024-09-13T17:12:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344148248 0 0) 0x7f7b0c06cb90 con 0x7f7b0c15ab60 2024-09-13T17:12:01.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b0c06c8b0 con 0x7f7b0c15ab60 2024-09-13T17:12:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09e00640 1 -- 172.21.15.28:0/2397952169 >> v1:172.21.15.189:6789/0 conn(0x7f7b0c15ab60 legacy=0x7f7b0c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09e00640 1 -- 172.21.15.28:0/2397952169 >> v1:172.21.15.28:6789/0 conn(0x7f7b0c06c250 legacy=0x7f7b0c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.945+0000 7f7b09e00640 1 -- 172.21.15.28:0/2397952169 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b0c0cc980 con 0x7f7b0c15a780 2024-09-13T17:12:01.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.946+0000 7f7b09e00640 1 --2- 172.21.15.28:0/2397952169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c070d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0097d0 tx=0x7f7afc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1d1b23c54e055e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc032440 con 0x7f7b0c15a780 2024-09-13T17:12:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7afc0325a0 con 0x7f7b0c15a780 2024-09-13T17:12:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b09400640 1 -- 172.21.15.28:0/2397952169 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc032930 con 0x7f7b0c15a780 2024-09-13T17:12:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b11200640 1 -- 172.21.15.28:0/2397952169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 msgr2=0x7f7b0c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b11200640 1 --2- 172.21.15.28:0/2397952169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c070d80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7afc0097d0 tx=0x7f7afc02a210 comp rx=0 tx=0).stop 2024-09-13T17:12:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b11200640 1 -- 172.21.15.28:0/2397952169 shutdown_connections 2024-09-13T17:12:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b11200640 1 --2- 172.21.15.28:0/2397952169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c070d80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.947+0000 7f7b11200640 1 -- 172.21.15.28:0/2397952169 >> 172.21.15.28:0/2397952169 conn(0x7f7b0c0b0270 msgr2=0x7f7b0c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.948+0000 7f7b11200640 1 -- 172.21.15.28:0/2397952169 shutdown_connections 2024-09-13T17:12:01.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.948+0000 7f7b11200640 1 -- 172.21.15.28:0/2397952169 wait complete. 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.949+0000 7f7b11200640 1 Processor -- start 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.949+0000 7f7b11200640 1 -- start start 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b0c06c250 0x7f7b0c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c1e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c1e4c70 con 0x7f7b0c15ab60 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c157230 con 0x7f7b0c06c250 2024-09-13T17:12:01.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b11200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c153ff0 con 0x7f7b0c15a780 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.950+0000 7f7b0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32838/0 (socket says 172.21.15.28:32838) 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 -- 172.21.15.28:0/2048409981 learned_addr learned my addr 172.21.15.28:0/2048409981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b09e00640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b0c06c250 0x7f7b0c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b03e00640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c1e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 -- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 msgr2=0x7f7b0c1e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c1e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 -- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b0c06c250 msgr2=0x7f7b0c152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b0c06c250 0x7f7b0c152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:01.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 -- 172.21.15.28:0/2048409981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc009400 con 0x7f7b0c15ab60 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b0a800640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400cbd0 tx=0x7f7b0400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.951+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0400ace0 con 0x7f7b0c15ab60 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.952+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0400ae40 con 0x7f7b0c15ab60 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.952+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c154290 con 0x7f7b0c15ab60 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.952+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b04012620 con 0x7f7b0c15ab60 2024-09-13T17:12:01.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.952+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c154850 con 0x7f7b0c15ab60 2024-09-13T17:12:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.953+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c0676b0 con 0x7f7b0c15ab60 2024-09-13T17:12:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.953+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7b04007070 con 0x7f7b0c15ab60 2024-09-13T17:12:01.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.955+0000 7f7b02000640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 0x7f7ae806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:01.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.955+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b04089280 con 0x7f7b0c15ab60 2024-09-13T17:12:01.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.955+0000 7f7b09e00640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 0x7f7ae806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:01.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.956+0000 7f7b09e00640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 0x7f7ae806d0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7afc034000 tx=0x7f7afc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:01.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:01.959+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7b04058250 con 0x7f7b0c15ab60 2024-09-13T17:12:02.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.139+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7b0c071270 con 0x7f7b0c15ab60 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.140+0000 7f7b02000640 1 -- 172.21.15.28:0/2048409981 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+65 (secure 0 0 0) 0x7f7b04017060 con 0x7f7b0c15ab60 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:12:02.141 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:02.142 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 msgr2=0x7f7ae806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 0x7f7ae806d0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7afc034000 tx=0x7f7afc02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 msgr2=0x7f7b0c154d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f7b0400cbd0 tx=0x7f7b0400a430 comp rx=0 tx=0).stop 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 shutdown_connections 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ae806ac00 0x7f7ae806d0c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b0c15ab60 0x7f7b0c154d70 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7b0c15a780 0x7f7b0c1e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 --2- 172.21.15.28:0/2048409981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b0c06c250 0x7f7b0c152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 >> 172.21.15.28:0/2048409981 conn(0x7f7b0c0b0270 msgr2=0x7f7b0c0aec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 shutdown_connections 2024-09-13T17:12:02.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.142+0000 7f7b11200640 1 -- 172.21.15.28:0/2048409981 wait complete. 2024-09-13T17:12:02.158 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:12:02.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:12:02.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.317+0000 7fc5f6600640 1 Processor -- start 2024-09-13T17:12:02.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.317+0000 7fc5f6600640 1 -- start start 2024-09-13T17:12:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5f00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5f6600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f015a4c0 con 0x7fc5f0159590 2024-09-13T17:12:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5f6600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f0071580 con 0x7fc5f00b2630 2024-09-13T17:12:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5f6600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00b2ae0 con 0x7fc5f0159970 2024-09-13T17:12:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5ef400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc5f0159590 0x7fc5f0070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56662/0 (socket says 172.21.15.28:56662) 2024-09-13T17:12:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.318+0000 7fc5ef400640 1 -- 172.21.15.28:0/1032663621 learned_addr learned my addr 172.21.15.28:0/1032663621 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5f00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743040355 0 0) 0x7fc5f015a4c0 con 0x7fc5f0159590 2024-09-13T17:12:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4003520 con 0x7fc5f0159590 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178781990 0 0) 0x7fc5f0071580 con 0x7fc5f00b2630 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f015a4c0 con 0x7fc5f00b2630 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023866284 0 0) 0x7fc5d4003520 con 0x7fc5f0159590 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 msgr2=0x7fc5f00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5f00b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 >> v1:172.21.15.189:6789/0 conn(0x7fc5f00b2630 legacy=0x7fc5f015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f0159e70 con 0x7fc5f0159590 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc5e00015f0 con 0x7fc5f0159590 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5f00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 >> v1:172.21.15.28:6789/0 conn(0x7fc5f0159590 legacy=0x7fc5f0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5d4003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d40040c0 0x7fc5d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d4004d00 0x7fc5d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f00b2ae0 con 0x7fc5d40040c0 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d400ba90 con 0x7fc5d4004d00 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.319+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d400bc50 con 0x7fc5f0159970 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d40040c0 0x7fc5d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5ef400640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5d4003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5efe00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d4004d00 0x7fc5d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 -- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 msgr2=0x7fc5d4003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f0159970 0x7fc5d4003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 -- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d4004d00 msgr2=0x7fc5d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d4004d00 0x7fc5d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.320+0000 7fc5e6a00640 1 -- 172.21.15.28:0/1032663621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009400 con 0x7fc5d40040c0 2024-09-13T17:12:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.321+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d40040c0 0x7fc5d40047c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00cf5e0 tx=0x7fc5d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425ecbaa31a027d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.321+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8031400 con 0x7fc5d40040c0 2024-09-13T17:12:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.321+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802d050 con 0x7fc5d40040c0 2024-09-13T17:12:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.321+0000 7fc5eea00640 1 -- 172.21.15.28:0/1032663621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8031780 con 0x7fc5d40040c0 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 -- 172.21.15.28:0/1032663621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d40040c0 msgr2=0x7fc5d40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 --2- 172.21.15.28:0/1032663621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5d40040c0 0x7fc5d40047c0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fc5f00cf5e0 tx=0x7fc5d40057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5efe00640 1 -- 172.21.15.28:0/1032663621 reap_dead start 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 -- 172.21.15.28:0/1032663621 shutdown_connections 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 -- 172.21.15.28:0/1032663621 >> 172.21.15.28:0/1032663621 conn(0x7fc5f0069e80 msgr2=0x7fc5f006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 -- 172.21.15.28:0/1032663621 shutdown_connections 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.322+0000 7fc5f6600640 1 -- 172.21.15.28:0/1032663621 wait complete. 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.324+0000 7fc5f6600640 1 Processor -- start 2024-09-13T17:12:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.324+0000 7fc5f6600640 1 -- start start 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d40040c0 0x7fc5f0140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f00707b0 0x7fc5f01ee5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0070c30 con 0x7fc5f014c710 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0151bf0 con 0x7fc5d40040c0 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5f6600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f0141470 con 0x7fc5f00707b0 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32856/0 (socket says 172.21.15.28:32856) 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 -- 172.21.15.28:0/3239997621 learned_addr learned my addr 172.21.15.28:0/3239997621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5ef400640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d40040c0 0x7fc5f0140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 -- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f00707b0 msgr2=0x7fc5f01ee5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f00707b0 0x7fc5f01ee5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 -- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d40040c0 msgr2=0x7fc5f0140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d40040c0 0x7fc5f0140f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.325+0000 7fc5e6a00640 1 -- 172.21.15.28:0/3239997621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f01ef320 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5e6a00640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8034040 tx=0x7fc5d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d803a600 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802ae90 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d803a960 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f01512c0 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.326+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f0151880 con 0x7fc5f014c710 2024-09-13T17:12:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.327+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f0070bd0 con 0x7fc5f014c710 2024-09-13T17:12:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.328+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc5d8038040 con 0x7fc5f014c710 2024-09-13T17:12:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.329+0000 7fc5e7e00640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 0x7fc5cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.329+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc5d80a90f0 con 0x7fc5f014c710 2024-09-13T17:12:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.330+0000 7fc5ef400640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 0x7fc5cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.330+0000 7fc5ef400640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 0x7fc5cc06d0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001af0 tx=0x7fc5e0000ef0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.333+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc5f0070bd0 con 0x7fc5f014c710 2024-09-13T17:12:02.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:02.511+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc5f0070bd0 con 0x7fc5f014c710 2024-09-13T17:12:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.201+0000 7fc5e7e00640 1 -- 172.21.15.28:0/3239997621 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v58) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc5f0070bd0 con 0x7fc5f014c710 2024-09-13T17:12:03.202 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:12:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 msgr2=0x7fc5cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 0x7fc5cc06d0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fc5e0001af0 tx=0x7fc5e0000ef0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 msgr2=0x7fc5f0070270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8034040 tx=0x7fc5d8004730 comp rx=0 tx=0).stop 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 shutdown_connections 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc5cc06ac00 0x7fc5cc06d0c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5f00707b0 0x7fc5f01ee5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5f014c710 0x7fc5f0070270 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 --2- 172.21.15.28:0/3239997621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d40040c0 0x7fc5f0140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 >> 172.21.15.28:0/3239997621 conn(0x7fc5f0069e80 msgr2=0x7fc5f00ae760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 shutdown_connections 2024-09-13T17:12:03.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.204+0000 7fc5f6600640 1 -- 172.21.15.28:0/3239997621 wait complete. 2024-09-13T17:12:03.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:12:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.391+0000 7fd83b200640 1 Processor -- start 2024-09-13T17:12:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.391+0000 7fd83b200640 1 -- start start 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd83b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8340b2400 0x7fd8340ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd83b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd8340ad940 0x7fd83406bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd83b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd834070470 con 0x7fd83415c9a0 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd83b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340ace30 con 0x7fd8340b2400 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd83b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8340ad110 con 0x7fd8340ad940 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd82be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8340b2400 0x7fd8340ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd82be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8340b2400 0x7fd8340ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32870/0 (socket says 172.21.15.28:32870) 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.392+0000 7fd82be00640 1 -- 172.21.15.28:0/3565738005 learned_addr learned my addr 172.21.15.28:0/3565738005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:03.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.393+0000 7fd839200640 1 --2- 172.21.15.28:0/3565738005 >> v2:172.21.15.189:3300/0 conn(0x7fd8340ad940 0x7fd83406bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.393+0000 7fd82be00640 1 -- 172.21.15.28:0/3565738005 >> v2:172.21.15.189:3300/0 conn(0x7fd8340ad940 msgr2=0x7fd83406bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.393+0000 7fd82be00640 1 --2- 172.21.15.28:0/3565738005 >> v2:172.21.15.189:3300/0 conn(0x7fd8340ad940 0x7fd83406bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.393+0000 7fd82be00640 1 -- 172.21.15.28:0/3565738005 >> v1:172.21.15.189:6789/0 conn(0x7fd83415c9a0 legacy=0x7fd8340b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.393+0000 7fd82be00640 1 -- 172.21.15.28:0/3565738005 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83406b170 con 0x7fd8340b2400 2024-09-13T17:12:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.394+0000 7fd82be00640 1 --2- 172.21.15.28:0/3565738005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340b2400 0x7fd8340ac850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd834160550 tx=0x7fd81c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20163d8da7edd79d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.394+0000 7fd833400640 1 -- 172.21.15.28:0/3565738005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c032440 con 0x7fd8340b2400 2024-09-13T17:12:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.394+0000 7fd833400640 1 -- 172.21.15.28:0/3565738005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81c0325a0 con 0x7fd8340b2400 2024-09-13T17:12:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.394+0000 7fd833400640 1 -- 172.21.15.28:0/3565738005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81c032930 con 0x7fd8340b2400 2024-09-13T17:12:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 -- 172.21.15.28:0/3565738005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340b2400 msgr2=0x7fd8340ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 --2- 172.21.15.28:0/3565738005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340b2400 0x7fd8340ac850 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd834160550 tx=0x7fd81c02a270 comp rx=0 tx=0).stop 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 -- 172.21.15.28:0/3565738005 shutdown_connections 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 --2- 172.21.15.28:0/3565738005 >> v2:172.21.15.189:3300/0 conn(0x7fd8340ad940 0x7fd83406bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 --2- 172.21.15.28:0/3565738005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340b2400 0x7fd8340ac850 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 -- 172.21.15.28:0/3565738005 >> 172.21.15.28:0/3565738005 conn(0x7fd8340b0270 msgr2=0x7fd8340b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 -- 172.21.15.28:0/3565738005 shutdown_connections 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.395+0000 7fd83b200640 1 -- 172.21.15.28:0/3565738005 wait complete. 2024-09-13T17:12:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.397+0000 7fd83b200640 1 Processor -- start 2024-09-13T17:12:03.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.397+0000 7fd83b200640 1 -- start start 2024-09-13T17:12:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 0x7fd8340cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd83415c9a0 0x7fd8340cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340ce4c0 con 0x7fd8340ad940 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834152800 con 0x7fd83415c9a0 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd83b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8340caf30 con 0x7fd8340b2400 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd833e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd833e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32878/0 (socket says 172.21.15.28:32878) 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd833e00640 1 -- 172.21.15.28:0/3859071434 learned_addr learned my addr 172.21.15.28:0/3859071434 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.398+0000 7fd839200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd83415c9a0 0x7fd8340cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd82be00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 0x7fd8340cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 -- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 msgr2=0x7fd8340cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 0x7fd8340cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 -- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd83415c9a0 msgr2=0x7fd8340cde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:03.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd83415c9a0 0x7fd8340cde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 -- 172.21.15.28:0/3859071434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81c0094d0 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd82be00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 0x7fd8340cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd833e00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c430 tx=0x7fd82400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240052d0 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824005430 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824003830 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd834070930 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.399+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8340cb4f0 con 0x7fd8340ad940 2024-09-13T17:12:03.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.400+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83415cd80 con 0x7fd8340ad940 2024-09-13T17:12:03.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.402+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd824003990 con 0x7fd8340ad940 2024-09-13T17:12:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.403+0000 7fd830c00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 0x7fd81006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.403+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fd824088b30 con 0x7fd8340ad940 2024-09-13T17:12:03.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.404+0000 7fd82be00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 0x7fd81006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:03.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.405+0000 7fd82be00640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 0x7fd81006d0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd81c034000 tx=0x7fd81c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:03.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.407+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd824053f60 con 0x7fd8340ad940 2024-09-13T17:12:03.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:03.587+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd83406c3b0 con 0x7fd8340ad940 2024-09-13T17:12:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.211+0000 7fd830c00640 1 -- 172.21.15.28:0/3859071434 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v59) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd8240170d0 con 0x7fd8340ad940 2024-09-13T17:12:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:12:04.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.216+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 msgr2=0x7fd81006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.216+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 0x7fd81006d0c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd81c034000 tx=0x7fd81c02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:04.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.216+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 msgr2=0x7fd83414c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.216+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fd82400c430 tx=0x7fd82400c8f0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 shutdown_connections 2024-09-13T17:12:04.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd81006ac00 0x7fd81006d0c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd83415c9a0 0x7fd8340cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8340b2400 0x7fd8340cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 --2- 172.21.15.28:0/3859071434 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8340ad940 0x7fd83414c0c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.220+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 >> 172.21.15.28:0/3859071434 conn(0x7fd8340b0270 msgr2=0x7fd8340af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.221+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 shutdown_connections 2024-09-13T17:12:04.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.221+0000 7fd83b200640 1 -- 172.21.15.28:0/3859071434 wait complete. 2024-09-13T17:12:04.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:12:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.407+0000 7feb34e00640 1 Processor -- start 2024-09-13T17:12:04.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.408+0000 7feb34e00640 1 -- start start 2024-09-13T17:12:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb34e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb300cd7c0 0x7feb300c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb34e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feb300c3ba0 0x7feb30070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb34e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3015ef70 con 0x7feb30143f70 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb34e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300c4320 con 0x7feb300cd7c0 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb34e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3015e1b0 con 0x7feb300c3ba0 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb26a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb300cd7c0 0x7feb300c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb26a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb300cd7c0 0x7feb300c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32994/0 (socket says 172.21.15.28:32994) 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb26a00640 1 -- 172.21.15.28:0/220058339 learned_addr learned my addr 172.21.15.28:0/220058339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.409+0000 7feb2fe00640 1 --2- 172.21.15.28:0/220058339 >> v2:172.21.15.189:3300/0 conn(0x7feb300c3ba0 0x7feb30070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.410+0000 7feb26a00640 1 -- 172.21.15.28:0/220058339 >> v2:172.21.15.189:3300/0 conn(0x7feb300c3ba0 msgr2=0x7feb30070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.410+0000 7feb26a00640 1 --2- 172.21.15.28:0/220058339 >> v2:172.21.15.189:3300/0 conn(0x7feb300c3ba0 0x7feb30070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.410+0000 7feb26a00640 1 -- 172.21.15.28:0/220058339 >> v1:172.21.15.189:6789/0 conn(0x7feb30143f70 legacy=0x7feb300cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.410+0000 7feb26a00640 1 -- 172.21.15.28:0/220058339 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3015e490 con 0x7feb300cd7c0 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.411+0000 7feb26a00640 1 --2- 172.21.15.28:0/220058339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300c3660 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feb300ce2b0 tx=0x7feb1802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6087224b8d9e3795 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.411+0000 7feb2ea00640 1 -- 172.21.15.28:0/220058339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18032440 con 0x7feb300cd7c0 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.411+0000 7feb2ea00640 1 -- 172.21.15.28:0/220058339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb180325a0 con 0x7feb300cd7c0 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.411+0000 7feb2ea00640 1 -- 172.21.15.28:0/220058339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18032930 con 0x7feb300cd7c0 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.411+0000 7feb34e00640 1 -- 172.21.15.28:0/220058339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 msgr2=0x7feb300c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 --2- 172.21.15.28:0/220058339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300c3660 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7feb300ce2b0 tx=0x7feb1802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 -- 172.21.15.28:0/220058339 shutdown_connections 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 --2- 172.21.15.28:0/220058339 >> v2:172.21.15.189:3300/0 conn(0x7feb300c3ba0 0x7feb30070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 --2- 172.21.15.28:0/220058339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300c3660 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 -- 172.21.15.28:0/220058339 >> 172.21.15.28:0/220058339 conn(0x7feb30069e80 msgr2=0x7feb3006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 -- 172.21.15.28:0/220058339 shutdown_connections 2024-09-13T17:12:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.412+0000 7feb34e00640 1 -- 172.21.15.28:0/220058339 wait complete. 2024-09-13T17:12:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.413+0000 7feb34e00640 1 Processor -- start 2024-09-13T17:12:04.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.413+0000 7feb34e00640 1 -- start start 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb300c3ba0 0x7feb3013d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb30143f70 0x7feb300ca5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3013dbf0 con 0x7feb300cd7c0 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb30141a00 con 0x7feb300c3ba0 2024-09-13T17:12:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.414+0000 7feb34e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb30141ce0 con 0x7feb30143f70 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb2f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb300c3ba0 0x7feb3013d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33006/0 (socket says 172.21.15.28:33006) 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 -- 172.21.15.28:0/1306028507 learned_addr learned my addr 172.21.15.28:0/1306028507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb2fe00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb30143f70 0x7feb300ca5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 -- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb30143f70 msgr2=0x7feb300ca5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb30143f70 0x7feb300ca5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 -- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb300c3ba0 msgr2=0x7feb3013d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb300c3ba0 0x7feb3013d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 -- 172.21.15.28:0/1306028507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb180094d0 con 0x7feb300cd7c0 2024-09-13T17:12:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.415+0000 7feb26a00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb1802a2b0 tx=0x7feb180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.416+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18032440 con 0x7feb300cd7c0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.416+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb18030980 con 0x7feb300cd7c0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.416+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb18030d10 con 0x7feb300cd7c0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.416+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb30141fc0 con 0x7feb300cd7c0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.416+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3013d8c0 con 0x7feb300cd7c0 2024-09-13T17:12:04.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.417+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb301424e0 con 0x7feb300cd7c0 2024-09-13T17:12:04.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.418+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feb18039040 con 0x7feb300cd7c0 2024-09-13T17:12:04.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.420+0000 7feb27e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 0x7feb0c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.420+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feb180a9d70 con 0x7feb300cd7c0 2024-09-13T17:12:04.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.420+0000 7feb2f400640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 0x7feb0c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.421+0000 7feb2f400640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 0x7feb0c06d0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f60 tx=0x7feb20009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.424+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feb180750d0 con 0x7feb300cd7c0 2024-09-13T17:12:04.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.638+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feb301424e0 con 0x7feb300cd7c0 2024-09-13T17:12:04.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.644+0000 7feb27e00640 1 -- 172.21.15.28:0/1306028507 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v45) v1 ==== 172+0+0 (secure 0 0 0) 0x7feb301424e0 con 0x7feb300cd7c0 2024-09-13T17:12:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 8 and data pool 9 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 msgr2=0x7feb0c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 0x7feb0c06d0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7feb20000f60 tx=0x7feb20009040 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 msgr2=0x7feb300ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7feb1802a2b0 tx=0x7feb180046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 shutdown_connections 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb0c06ac00 0x7feb0c06d0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb30143f70 0x7feb300ca5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb300cd7c0 0x7feb300ca0b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 --2- 172.21.15.28:0/1306028507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb300c3ba0 0x7feb3013d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.650+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 >> 172.21.15.28:0/1306028507 conn(0x7feb30069e80 msgr2=0x7feb3013f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.651+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 shutdown_connections 2024-09-13T17:12:04.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.651+0000 7feb34e00640 1 -- 172.21.15.28:0/1306028507 wait complete. 2024-09-13T17:12:04.682 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.837+0000 7fe24ac00640 1 Processor -- start 2024-09-13T17:12:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.837+0000 7fe24ac00640 1 -- start start 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe24ac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe24406f970 0x7fe24415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe24ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 0x7fe24406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe24ac00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe24406cc70 con 0x7fe2441592a0 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe24ac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24415b0b0 con 0x7fe24406f970 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe24ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe244159d90 con 0x7fe24406c2d0 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.838+0000 7fe243400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe24406f970 0x7fe24415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe24406f970 0x7fe24415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33008/0 (socket says 172.21.15.28:33008) 2024-09-13T17:12:04.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 -- 172.21.15.28:0/467440589 learned_addr learned my addr 172.21.15.28:0/467440589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe248c00640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 0x7fe24406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 -- 172.21.15.28:0/467440589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 msgr2=0x7fe24406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 0x7fe24406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 -- 172.21.15.28:0/467440589 >> v1:172.21.15.189:6789/0 conn(0x7fe2441592a0 legacy=0x7fe244159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe243400640 1 -- 172.21.15.28:0/467440589 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2440b2040 con 0x7fe24406f970 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.839+0000 7fe248c00640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 0x7fe24406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.840+0000 7fe243400640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24406f970 0x7fe24415ab70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe230009870 tx=0x7fe23002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7efd33574678684c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.840+0000 7fe242a00640 1 -- 172.21.15.28:0/467440589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230032440 con 0x7fe24406f970 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.840+0000 7fe242a00640 1 -- 172.21.15.28:0/467440589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2300325a0 con 0x7fe24406f970 2024-09-13T17:12:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.840+0000 7fe242a00640 1 -- 172.21.15.28:0/467440589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe230032930 con 0x7fe24406f970 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 -- 172.21.15.28:0/467440589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24406f970 msgr2=0x7fe24415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24406f970 0x7fe24415ab70 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe230009870 tx=0x7fe23002a250 comp rx=0 tx=0).stop 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 -- 172.21.15.28:0/467440589 shutdown_connections 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406c2d0 0x7fe24406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 --2- 172.21.15.28:0/467440589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe24406f970 0x7fe24415ab70 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 -- 172.21.15.28:0/467440589 >> 172.21.15.28:0/467440589 conn(0x7fe244069e80 msgr2=0x7fe24406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 -- 172.21.15.28:0/467440589 shutdown_connections 2024-09-13T17:12:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.841+0000 7fe24ac00640 1 -- 172.21.15.28:0/467440589 wait complete. 2024-09-13T17:12:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.843+0000 7fe24ac00640 1 Processor -- start 2024-09-13T17:12:04.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.843+0000 7fe24ac00640 1 -- start start 2024-09-13T17:12:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24406c2d0 0x7fe2440cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406f970 0x7fe2440afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe244141870 con 0x7fe2441592a0 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2441e2350 con 0x7fe24406c2d0 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe24ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2440cc500 con 0x7fe24406f970 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe243e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24406c2d0 0x7fe2440cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33022/0 (socket says 172.21.15.28:33022) 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 -- 172.21.15.28:0/2320043623 learned_addr learned my addr 172.21.15.28:0/2320043623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe243400640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406f970 0x7fe2440afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 -- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406f970 msgr2=0x7fe2440afab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406f970 0x7fe2440afab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 -- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24406c2d0 msgr2=0x7fe2440cbfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.844+0000 7fe248c00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24406c2d0 0x7fe2440cbfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:04.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe248c00640 1 -- 172.21.15.28:0/2320043623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2300094d0 con 0x7fe2441592a0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe248c00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009fa0 tx=0x7fe23c00d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c003b90 con 0x7fe2441592a0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23c00f040 con 0x7fe2441592a0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23c004030 con 0x7fe2441592a0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2441e1d90 con 0x7fe2441592a0 2024-09-13T17:12:04.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.845+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe244141500 con 0x7fe2441592a0 2024-09-13T17:12:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.846+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe244141360 con 0x7fe2441592a0 2024-09-13T17:12:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.847+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe23c004190 con 0x7fe2441592a0 2024-09-13T17:12:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.848+0000 7fe23be00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 0x7fe22406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.848+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe23c089680 con 0x7fe2441592a0 2024-09-13T17:12:04.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.849+0000 7fe243e00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 0x7fe22406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:04.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.849+0000 7fe243e00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 0x7fe22406d0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe234000f60 tx=0x7fe234009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:04.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:04.852+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe244141360 con 0x7fe2441592a0 2024-09-13T17:12:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.032+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe244141360 con 0x7fe2441592a0 2024-09-13T17:12:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.033+0000 7fe23be00640 1 -- 172.21.15.28:0/2320043623 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16578 (secure 0 0 0) 0x7fe244141360 con 0x7fe2441592a0 2024-09-13T17:12:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:05.034 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":60,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:04.638989+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 msgr2=0x7fe22406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 0x7fe22406d0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe234000f60 tx=0x7fe234009040 comp rx=0 tx=0).stop 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 msgr2=0x7fe2441406a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe23c009fa0 tx=0x7fe23c00d560 comp rx=0 tx=0).stop 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 shutdown_connections 2024-09-13T17:12:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe22406ac00 0x7fe22406d0c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2441592a0 0x7fe2441406a0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe24406f970 0x7fe2440afab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 --2- 172.21.15.28:0/2320043623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe24406c2d0 0x7fe2440cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 >> 172.21.15.28:0/2320043623 conn(0x7fe244069e80 msgr2=0x7fe244156f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 shutdown_connections 2024-09-13T17:12:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.035+0000 7fe24ac00640 1 -- 172.21.15.28:0/2320043623 wait complete. 2024-09-13T17:12:05.052 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:05.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.207+0000 7f3e5f000640 1 Processor -- start 2024-09-13T17:12:05.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.207+0000 7f3e5f000640 1 -- start start 2024-09-13T17:12:05.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.208+0000 7f3e5f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e580afbd0 0x7f3e5806c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.208+0000 7f3e5f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b39d0 con 0x7f3e58158b20 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.208+0000 7f3e5f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e580b2dc0 con 0x7f3e5806ba70 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.208+0000 7f3e5f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5806c930 con 0x7f3e580afbd0 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5dc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3e58158b20 0x7f3e5806b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45062/0 (socket says 172.21.15.28:45062) 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5dc00640 1 -- 172.21.15.28:0/3290444916 learned_addr learned my addr 172.21.15.28:0/3290444916 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5e600640 1 --2- 172.21.15.28:0/3290444916 >> v2:172.21.15.28:3300/0 conn(0x7f3e580afbd0 0x7f3e5806c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5e600640 1 -- 172.21.15.28:0/3290444916 >> v1:172.21.15.189:6789/0 conn(0x7f3e5806ba70 legacy=0x7f3e58158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5e600640 1 -- 172.21.15.28:0/3290444916 >> v1:172.21.15.28:6789/0 conn(0x7f3e58158b20 legacy=0x7f3e5806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.209+0000 7f3e5e600640 1 -- 172.21.15.28:0/3290444916 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e580b30a0 con 0x7f3e580afbd0 2024-09-13T17:12:05.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.210+0000 7f3e5e600640 1 --2- 172.21.15.28:0/3290444916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e580afbd0 0x7f3e5806c3f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009d90 tx=0x7f3e5002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca9a8e9527a6d661 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e57e00640 1 -- 172.21.15.28:0/3290444916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50007740 con 0x7f3e580afbd0 2024-09-13T17:12:05.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e57e00640 1 -- 172.21.15.28:0/3290444916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5002d050 con 0x7f3e580afbd0 2024-09-13T17:12:05.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e57e00640 1 -- 172.21.15.28:0/3290444916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e50007ac0 con 0x7f3e580afbd0 2024-09-13T17:12:05.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 -- 172.21.15.28:0/3290444916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e580afbd0 msgr2=0x7f3e5806c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 --2- 172.21.15.28:0/3290444916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e580afbd0 0x7f3e5806c3f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009d90 tx=0x7f3e5002a660 comp rx=0 tx=0).stop 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 -- 172.21.15.28:0/3290444916 shutdown_connections 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 --2- 172.21.15.28:0/3290444916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e580afbd0 0x7f3e5806c3f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 -- 172.21.15.28:0/3290444916 >> 172.21.15.28:0/3290444916 conn(0x7f3e58072e40 msgr2=0x7f3e58073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 -- 172.21.15.28:0/3290444916 shutdown_connections 2024-09-13T17:12:05.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.212+0000 7f3e5f000640 1 -- 172.21.15.28:0/3290444916 wait complete. 2024-09-13T17:12:05.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.214+0000 7f3e5f000640 1 Processor -- start 2024-09-13T17:12:05.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.214+0000 7f3e5f000640 1 -- start start 2024-09-13T17:12:05.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 0x7f3e5814c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e58158b20 0x7f3e5814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e58155470 con 0x7f3e5806ba70 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5814c720 con 0x7f3e580afbd0 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e580aeac0 con 0x7f3e58158b20 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33034/0 (socket says 172.21.15.28:33034) 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.215+0000 7f3e5dc00640 1 -- 172.21.15.28:0/1781346036 learned_addr learned my addr 172.21.15.28:0/1781346036 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5d200640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 0x7f3e5814c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5e600640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e58158b20 0x7f3e5814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 -- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e58158b20 msgr2=0x7f3e5814d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e58158b20 0x7f3e5814d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 -- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 msgr2=0x7f3e5814c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 0x7f3e5814c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 -- 172.21.15.28:0/1781346036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50009a70 con 0x7f3e5806ba70 2024-09-13T17:12:05.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5d200640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 0x7f3e5814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5dc00640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c480 tx=0x7f3e4c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c005ce0 con 0x7f3e5806ba70 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c005e40 con 0x7f3e5806ba70 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e580aeda0 con 0x7f3e5806ba70 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.216+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0072f0 con 0x7f3e5806ba70 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.217+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e580af360 con 0x7f3e5806ba70 2024-09-13T17:12:05.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.218+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e580b0990 con 0x7f3e5806ba70 2024-09-13T17:12:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.218+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e4c007450 con 0x7f3e5806ba70 2024-09-13T17:12:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.219+0000 7f3e55600640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 0x7f3e3806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.219+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e4c088850 con 0x7f3e5806ba70 2024-09-13T17:12:05.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.220+0000 7f3e5d200640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 0x7f3e3806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.220+0000 7f3e5d200640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 0x7f3e3806d0f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3e44007fd0 tx=0x7f3e44009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.223+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3e4c053ad0 con 0x7f3e5806ba70 2024-09-13T17:12:05.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.404+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e581572c0 con 0x7f3e5806ba70 2024-09-13T17:12:05.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.406+0000 7f3e55600640 1 -- 172.21.15.28:0/1781346036 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16578 (secure 0 0 0) 0x7f3e4c017090 con 0x7f3e5806ba70 2024-09-13T17:12:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:05.407 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":60,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:04.638989+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 msgr2=0x7f3e3806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 0x7f3e3806d0f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3e44007fd0 tx=0x7f3e44009040 comp rx=0 tx=0).stop 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 msgr2=0x7f3e58154f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00c480 tx=0x7f3e4c00c940 comp rx=0 tx=0).stop 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 shutdown_connections 2024-09-13T17:12:05.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e3806ac30 0x7f3e3806d0f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e58158b20 0x7f3e5814d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e580afbd0 0x7f3e5814c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 --2- 172.21.15.28:0/1781346036 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e5806ba70 0x7f3e58154f30 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 >> 172.21.15.28:0/1781346036 conn(0x7f3e58072e40 msgr2=0x7f3e5806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 shutdown_connections 2024-09-13T17:12:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.408+0000 7f3e5f000640 1 -- 172.21.15.28:0/1781346036 wait complete. 2024-09-13T17:12:05.426 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:12:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.583+0000 7f29b8400640 1 Processor -- start 2024-09-13T17:12:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.583+0000 7f29b8400640 1 -- start start 2024-09-13T17:12:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.584+0000 7f29b8400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f29b0159770 0x7f29b006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.584+0000 7f29b8400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 0x7f29b0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.584+0000 7f29b8400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b006f920 con 0x7f29b00add00 2024-09-13T17:12:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.584+0000 7f29b8400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b015b290 con 0x7f29b0159770 2024-09-13T17:12:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.584+0000 7f29b8400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b015b570 con 0x7f29b006f3f0 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29b5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 0x7f29b0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29b5a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f29b0159770 0x7f29b006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29b6400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f29b00add00 0x7f29b0159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:42486/0 (socket says 172.21.15.28:42486) 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29b6400640 1 -- 172.21.15.28:0/3986078088 learned_addr learned my addr 172.21.15.28:0/3986078088 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29afe00640 1 -- 172.21.15.28:0/3986078088 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367835017 0 0) 0x7f29b006f920 con 0x7f29b00add00 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.585+0000 7f29afe00640 1 -- 172.21.15.28:0/3986078088 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998003520 con 0x7f29b00add00 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.586+0000 7f29b5a00640 1 -- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 msgr2=0x7f29b0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.586+0000 7f29b5a00640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 0x7f29b0070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.586+0000 7f29b5a00640 1 -- 172.21.15.28:0/3986078088 >> v1:172.21.15.189:6789/0 conn(0x7f29b00add00 legacy=0x7f29b0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.586+0000 7f29b5a00640 1 -- 172.21.15.28:0/3986078088 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b015b850 con 0x7f29b0159770 2024-09-13T17:12:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.586+0000 7f29b5000640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 0x7f29b0070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29b5a00640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b0159770 0x7f29b006fd20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097a0 tx=0x7f29a402a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9641f189adabd590 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29afe00640 1 -- 172.21.15.28:0/3986078088 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a402ae40 con 0x7f29b0159770 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29afe00640 1 -- 172.21.15.28:0/3986078088 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a4032440 con 0x7f29b0159770 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29afe00640 1 -- 172.21.15.28:0/3986078088 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a40327d0 con 0x7f29b0159770 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29b8400640 1 -- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b0159770 msgr2=0x7f29b006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29b8400640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b0159770 0x7f29b006fd20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f29a40097a0 tx=0x7f29a402a0f0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.587+0000 7f29b8400640 1 -- 172.21.15.28:0/3986078088 shutdown_connections 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.588+0000 7f29b8400640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b006f3f0 0x7f29b0070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.588+0000 7f29b8400640 1 --2- 172.21.15.28:0/3986078088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b0159770 0x7f29b006fd20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.588+0000 7f29b8400640 1 -- 172.21.15.28:0/3986078088 >> 172.21.15.28:0/3986078088 conn(0x7f29b00b0270 msgr2=0x7f29b00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.588+0000 7f29b8400640 1 -- 172.21.15.28:0/3986078088 shutdown_connections 2024-09-13T17:12:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.588+0000 7f29b8400640 1 -- 172.21.15.28:0/3986078088 wait complete. 2024-09-13T17:12:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.589+0000 7f29b8400640 1 Processor -- start 2024-09-13T17:12:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.589+0000 7f29b8400640 1 -- start start 2024-09-13T17:12:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b00add00 0x7f29b0148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b0159770 0x7f29b0154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0149440 con 0x7f29b006f3f0 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00b2880 con 0x7f29b00add00 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b8400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b014f880 con 0x7f29b0159770 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.590+0000 7f29b5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33038/0 (socket says 172.21.15.28:33038) 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 -- 172.21.15.28:0/1907216431 learned_addr learned my addr 172.21.15.28:0/1907216431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b6400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b0159770 0x7f29b0154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5000640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b00add00 0x7f29b0148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 -- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b0159770 msgr2=0x7f29b0154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b0159770 0x7f29b0154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 -- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b00add00 msgr2=0x7f29b0148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b00add00 0x7f29b0148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 -- 172.21.15.28:0/1907216431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c009400 con 0x7f29b006f3f0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.591+0000 7f29b5a00640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c00 tx=0x7f29a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.592+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a403b8d0 con 0x7f29b006f3f0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.592+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a403ba30 con 0x7f29b006f3f0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.592+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a403bde0 con 0x7f29b006f3f0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.592+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a4009400 con 0x7f29b006f3f0 2024-09-13T17:12:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.592+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b00b1f50 con 0x7f29b006f3f0 2024-09-13T17:12:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.593+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b006fd20 con 0x7f29b006f3f0 2024-09-13T17:12:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.594+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f29a4039040 con 0x7f29b006f3f0 2024-09-13T17:12:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.594+0000 7f29ad600640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 0x7f299006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.595+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29a40a9b00 con 0x7f29b006f3f0 2024-09-13T17:12:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.595+0000 7f29b5000640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 0x7f299006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.596+0000 7f29b5000640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 0x7f299006d0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f29b01f5880 tx=0x7f299c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.599+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f29a40abe60 con 0x7f29b006f3f0 2024-09-13T17:12:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:05.815+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f29b00afb50 con 0x7f29b006f3f0 2024-09-13T17:12:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.652+0000 7f29ad600640 1 -- 172.21.15.28:0/1907216431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f29a4037070 con 0x7f29b006f3f0 2024-09-13T17:12:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 msgr2=0x7f299006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 0x7f299006d0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f29b01f5880 tx=0x7f299c009190 comp rx=0 tx=0).stop 2024-09-13T17:12:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 msgr2=0x7f29b00b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f29a4000c00 tx=0x7f29a40046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 shutdown_connections 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f299006ac00 0x7f299006d0c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29b0159770 0x7f29b0154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29b00add00 0x7f29b0148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 --2- 172.21.15.28:0/1907216431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29b006f3f0 0x7f29b00b1570 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 >> 172.21.15.28:0/1907216431 conn(0x7f29b00b0270 msgr2=0x7f29b00cebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 shutdown_connections 2024-09-13T17:12:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.655+0000 7f29b8400640 1 -- 172.21.15.28:0/1907216431 wait complete. 2024-09-13T17:12:06.676 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:12:06.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:12:06.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.838+0000 7f5f13000640 1 Processor -- start 2024-09-13T17:12:06.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.838+0000 7f5f13000640 1 -- start start 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f13000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 0x7f5f0c0ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f13000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f13000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f0c0703e0 con 0x7f5f0c14f950 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f13000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f0c0cd740 con 0x7f5f0c072210 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f13000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c06fd90 con 0x7f5f0c14fd30 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f11c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f0c14f950 0x7f5f0c15a250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45072/0 (socket says 172.21.15.28:45072) 2024-09-13T17:12:06.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f11c00640 1 -- 172.21.15.28:0/366830101 learned_addr learned my addr 172.21.15.28:0/366830101 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f12600640 1 --2- 172.21.15.28:0/366830101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f11200640 1 --2- 172.21.15.28:0/366830101 >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 0x7f5f0c0ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904622125 0 0) 0x7f5f0c0703e0 con 0x7f5f0c14f950 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.840+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ef4003520 con 0x7f5f0c14f950 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3770636175 0 0) 0x7f5ef4003520 con 0x7f5f0c14f950 2024-09-13T17:12:06.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f12600640 1 -- 172.21.15.28:0/366830101 >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 msgr2=0x7f5f0c0ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f12600640 1 --2- 172.21.15.28:0/366830101 >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 0x7f5f0c0ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f12600640 1 -- 172.21.15.28:0/366830101 >> v1:172.21.15.28:6789/0 conn(0x7f5f0c14f950 legacy=0x7f5f0c15a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f12600640 1 -- 172.21.15.28:0/366830101 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c070070 con 0x7f5f0c14fd30 2024-09-13T17:12:06.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.841+0000 7f5f11200640 1 --2- 172.21.15.28:0/366830101 >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 0x7f5f0c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.842+0000 7f5f12600640 1 --2- 172.21.15.28:0/366830101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15c680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009d90 tx=0x7f5f0402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fe52b949d8d5172 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.842+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f00001670 con 0x7f5f0c14fd30 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.842+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0402d050 con 0x7f5f0c14fd30 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.842+0000 7f5f0be00640 1 -- 172.21.15.28:0/366830101 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04007760 con 0x7f5f0c14fd30 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 -- 172.21.15.28:0/366830101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 msgr2=0x7f5f0c15c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 --2- 172.21.15.28:0/366830101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15c680 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f04009d90 tx=0x7f5f0402a5f0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 -- 172.21.15.28:0/366830101 shutdown_connections 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 --2- 172.21.15.28:0/366830101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15c680 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 --2- 172.21.15.28:0/366830101 >> v2:172.21.15.189:3300/0 conn(0x7f5f0c072210 0x7f5f0c0ccdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 -- 172.21.15.28:0/366830101 >> 172.21.15.28:0/366830101 conn(0x7f5f0c069e80 msgr2=0x7f5f0c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 -- 172.21.15.28:0/366830101 shutdown_connections 2024-09-13T17:12:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.843+0000 7f5f13000640 1 -- 172.21.15.28:0/366830101 wait complete. 2024-09-13T17:12:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.844+0000 7f5f13000640 1 Processor -- start 2024-09-13T17:12:06.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 -- start start 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f0c072210 0x7f5f0c1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c1f32a0 con 0x7f5f0c14f950 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c1f3580 con 0x7f5f0c072210 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.845+0000 7f5f13000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c1f3860 con 0x7f5f0c14fd30 2024-09-13T17:12:06.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33046/0 (socket says 172.21.15.28:33046) 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 -- 172.21.15.28:0/2315990403 learned_addr learned my addr 172.21.15.28:0/2315990403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11c00640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f0c072210 0x7f5f0c1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f12600640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 -- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 msgr2=0x7f5f0c15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 -- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f0c072210 msgr2=0x7f5f0c1f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f0c072210 0x7f5f0c1f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 -- 172.21.15.28:0/2315990403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8009400 con 0x7f5f0c14f950 2024-09-13T17:12:06.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.846+0000 7f5f11200640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800ebe0 tx=0x7f5ef800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.847+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef8009760 con 0x7f5f0c14f950 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.847+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ef800cc30 con 0x7f5f0c14f950 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.847+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800eee0 con 0x7f5f0c14f950 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.847+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f04009a70 con 0x7f5f0c14f950 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.847+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c1f3e20 con 0x7f5f0c14f950 2024-09-13T17:12:06.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.848+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c0ccdd0 con 0x7f5f0c14f950 2024-09-13T17:12:06.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.849+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5ef8013470 con 0x7f5f0c14f950 2024-09-13T17:12:06.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.849+0000 7f5f09600640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 0x7f5eec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:06.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.850+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ef80895d0 con 0x7f5f0c14f950 2024-09-13T17:12:06.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.850+0000 7f5f11c00640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 0x7f5eec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:06.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.851+0000 7f5f11c00640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 0x7f5eec06d0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5f00001eb0 tx=0x7f5f000016d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:06.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:06.854+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5ef8054850 con 0x7f5f0c14f950 2024-09-13T17:12:07.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.069+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5f0c0b2e70 con 0x7f5f0c14f950 2024-09-13T17:12:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.658+0000 7f5f09600640 1 -- 172.21.15.28:0/2315990403 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5ef8057e70 con 0x7f5f0c14f950 2024-09-13T17:12:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 msgr2=0x7f5eec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 0x7f5eec06d0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5f00001eb0 tx=0x7f5f000016d0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 msgr2=0x7f5f0c15d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f5ef800ebe0 tx=0x7f5ef800c520 comp rx=0 tx=0).stop 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 shutdown_connections 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5eec06ac00 0x7f5eec06d0c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f0c14fd30 0x7f5f0c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f0c14f950 0x7f5f0c15d7b0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 --2- 172.21.15.28:0/2315990403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f0c072210 0x7f5f0c1f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 >> 172.21.15.28:0/2315990403 conn(0x7f5f0c069e80 msgr2=0x7f5f0c157b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 shutdown_connections 2024-09-13T17:12:07.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.661+0000 7f5f13000640 1 -- 172.21.15.28:0/2315990403 wait complete. 2024-09-13T17:12:07.681 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:12:07.681 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.839+0000 7fb483e00640 1 Processor -- start 2024-09-13T17:12:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.839+0000 7fb483e00640 1 -- start start 2024-09-13T17:12:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb483e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb483e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb47c06c2d0 0x7fb47c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb483e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c159d90 con 0x7fb47c1592a0 2024-09-13T17:12:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb483e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c15b0b0 con 0x7fb47c06c2d0 2024-09-13T17:12:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb483e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c06cca0 con 0x7fb47c06f970 2024-09-13T17:12:07.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.840+0000 7fb480a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:44130/0 (socket says 172.21.15.28:44130) 2024-09-13T17:12:07.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb481400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb47c1592a0 0x7fb47c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:42502/0 (socket says 172.21.15.28:42502) 2024-09-13T17:12:07.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 -- 172.21.15.28:0/2382583237 learned_addr learned my addr 172.21.15.28:0/2382583237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:07.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb481e00640 1 --2- 172.21.15.28:0/2382583237 >> v2:172.21.15.189:3300/0 conn(0x7fb47c06c2d0 0x7fb47c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 -- 172.21.15.28:0/2382583237 >> v2:172.21.15.189:3300/0 conn(0x7fb47c06c2d0 msgr2=0x7fb47c06c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:07.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 --2- 172.21.15.28:0/2382583237 >> v2:172.21.15.189:3300/0 conn(0x7fb47c06c2d0 0x7fb47c06c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 -- 172.21.15.28:0/2382583237 >> v1:172.21.15.189:6789/0 conn(0x7fb47c1592a0 legacy=0x7fb47c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb480a00640 1 -- 172.21.15.28:0/2382583237 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0b2040 con 0x7fb47c06f970 2024-09-13T17:12:07.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.841+0000 7fb47be00640 1 -- 172.21.15.28:0/2382583237 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362300919 0 0) 0x7fb47c159d90 con 0x7fb47c1592a0 2024-09-13T17:12:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.842+0000 7fb480a00640 1 --2- 172.21.15.28:0/2382583237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb468009870 tx=0x7fb46802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adc319a690885d0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb47be00640 1 -- 172.21.15.28:0/2382583237 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468032440 con 0x7fb47c06f970 2024-09-13T17:12:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb47be00640 1 -- 172.21.15.28:0/2382583237 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4680325a0 con 0x7fb47c06f970 2024-09-13T17:12:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb47be00640 1 -- 172.21.15.28:0/2382583237 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb468032930 con 0x7fb47c06f970 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 -- 172.21.15.28:0/2382583237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 msgr2=0x7fb47c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 --2- 172.21.15.28:0/2382583237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb468009870 tx=0x7fb46802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 -- 172.21.15.28:0/2382583237 shutdown_connections 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 --2- 172.21.15.28:0/2382583237 >> v2:172.21.15.189:3300/0 conn(0x7fb47c06c2d0 0x7fb47c06c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 --2- 172.21.15.28:0/2382583237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c06f970 0x7fb47c15ab70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 -- 172.21.15.28:0/2382583237 >> 172.21.15.28:0/2382583237 conn(0x7fb47c069e80 msgr2=0x7fb47c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:07.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 -- 172.21.15.28:0/2382583237 shutdown_connections 2024-09-13T17:12:07.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.843+0000 7fb483e00640 1 -- 172.21.15.28:0/2382583237 wait complete. 2024-09-13T17:12:07.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.845+0000 7fb483e00640 1 Processor -- start 2024-09-13T17:12:07.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.845+0000 7fb483e00640 1 -- start start 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 0x7fb47c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 0x7fb47c0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c152670 con 0x7fb47c06c2d0 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c151b90 con 0x7fb47c06f970 2024-09-13T17:12:07.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb483e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0af6f0 con 0x7fb47c1592a0 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb481400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb481400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33052/0 (socket says 172.21.15.28:33052) 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.846+0000 7fb481400640 1 -- 172.21.15.28:0/4018791673 learned_addr learned my addr 172.21.15.28:0/4018791673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 -- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 msgr2=0x7fb47c0acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 0x7fb47c0acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 0x7fb47c0acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb480a00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 0x7fb47c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 -- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 msgr2=0x7fb47c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 0x7fb47c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 -- 172.21.15.28:0/4018791673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4680094d0 con 0x7fb47c06c2d0 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 0x7fb47c0acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb480a00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 0x7fb47c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:07.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb481400640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb47000c410 tx=0x7fb47000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4700074e0 con 0x7fb47c06c2d0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470007640 con 0x7fb47c06c2d0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0af9d0 con 0x7fb47c06c2d0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.847+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470004a20 con 0x7fb47c06c2d0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.848+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0aff90 con 0x7fb47c06c2d0 2024-09-13T17:12:07.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.848+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0b3740 con 0x7fb47c06c2d0 2024-09-13T17:12:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.850+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb470004b80 con 0x7fb47c06c2d0 2024-09-13T17:12:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.850+0000 7fb479600640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 0x7fb45c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.850+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb470088e50 con 0x7fb47c06c2d0 2024-09-13T17:12:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.851+0000 7fb480a00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 0x7fb45c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:07.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.852+0000 7fb480a00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 0x7fb45c06d0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb468034000 tx=0x7fb46802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:07.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:07.855+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb470057a20 con 0x7fb47c06c2d0 2024-09-13T17:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.066+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47c15be50 con 0x7fb47c06c2d0 2024-09-13T17:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.066+0000 7fb479600640 1 -- 172.21.15.28:0/4018791673 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fb4700575b0 con 0x7fb47c06c2d0 2024-09-13T17:12:08.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:08.069 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6145,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/918367694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":918367694},{"type":"v1","addr":"172.21.15.28:6834","nonce":918367694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3207667105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3207667105},{"type":"v1","addr":"172.21.15.28:6835","nonce":3207667105}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3369388937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3369388937},{"type":"v1","addr":"172.21.15.189:6836","nonce":3369388937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:07.650977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6420":{"gid":6420,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/693307152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":693307152},{"type":"v1","addr":"172.21.15.189:6837","nonce":693307152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 49 2024-09-13T17:12:08.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 msgr2=0x7fb45c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 0x7fb45c06d0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb468034000 tx=0x7fb46802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:08.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 msgr2=0x7fb47c149330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fb47000c410 tx=0x7fb47000c8d0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 shutdown_connections 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb45c06ac00 0x7fb45c06d0c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb47c1592a0 0x7fb47c0acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb47c06f970 0x7fb47c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 --2- 172.21.15.28:0/4018791673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb47c06c2d0 0x7fb47c149330 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 >> 172.21.15.28:0/4018791673 conn(0x7fb47c069e80 msgr2=0x7fb47c15c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 shutdown_connections 2024-09-13T17:12:08.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.069+0000 7fb483e00640 1 -- 172.21.15.28:0/4018791673 wait complete. 2024-09-13T17:12:08.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.241+0000 7f1e37200640 1 Processor -- start 2024-09-13T17:12:08.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.241+0000 7f1e37200640 1 -- start start 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.242+0000 7f1e37200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e3806b5a0 0x7f1e38160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.242+0000 7f1e37200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1e380c4380 0x7f1e380ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.242+0000 7f1e37200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380c4af0 con 0x7f1e380cb760 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.242+0000 7f1e37200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380cd460 con 0x7f1e380c4380 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.242+0000 7f1e37200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380cd740 con 0x7f1e3806b5a0 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e35e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1e380cb760 0x7f1e3806ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45080/0 (socket says 172.21.15.28:45080) 2024-09-13T17:12:08.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e35e00640 1 -- 172.21.15.28:0/275484143 learned_addr learned my addr 172.21.15.28:0/275484143 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e36800640 1 --2- 172.21.15.28:0/275484143 >> v2:172.21.15.28:3300/0 conn(0x7f1e380c4380 0x7f1e380ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e35400640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e3806b5a0 0x7f1e38160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e36800640 1 -- 172.21.15.28:0/275484143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e3806b5a0 msgr2=0x7f1e38160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e36800640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e3806b5a0 0x7f1e38160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e36800640 1 -- 172.21.15.28:0/275484143 >> v1:172.21.15.28:6789/0 conn(0x7f1e380cb760 legacy=0x7f1e3806ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e36800640 1 -- 172.21.15.28:0/275484143 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e380cda20 con 0x7f1e380c4380 2024-09-13T17:12:08.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.243+0000 7f1e34a00640 1 -- 172.21.15.28:0/275484143 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924955452 0 0) 0x7f1e380c4af0 con 0x7f1e380cb760 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.244+0000 7f1e36800640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380c4380 0x7f1e380ce220 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009d90 tx=0x7f1e3002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd95a5fde135c514 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.244+0000 7f1e34a00640 1 -- 172.21.15.28:0/275484143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30031600 con 0x7f1e380c4380 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e34a00640 1 -- 172.21.15.28:0/275484143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3002d050 con 0x7f1e380c4380 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e34a00640 1 -- 172.21.15.28:0/275484143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30031980 con 0x7f1e380c4380 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 -- 172.21.15.28:0/275484143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380c4380 msgr2=0x7f1e380ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380c4380 0x7f1e380ce220 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009d90 tx=0x7f1e3002a5d0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 -- 172.21.15.28:0/275484143 shutdown_connections 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380c4380 0x7f1e380ce220 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 --2- 172.21.15.28:0/275484143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e3806b5a0 0x7f1e38160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 -- 172.21.15.28:0/275484143 >> 172.21.15.28:0/275484143 conn(0x7f1e380703a0 msgr2=0x7f1e380707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 -- 172.21.15.28:0/275484143 shutdown_connections 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.245+0000 7f1e37200640 1 -- 172.21.15.28:0/275484143 wait complete. 2024-09-13T17:12:08.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.247+0000 7f1e37200640 1 Processor -- start 2024-09-13T17:12:08.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.247+0000 7f1e37200640 1 -- start start 2024-09-13T17:12:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e3806b5a0 0x7f1e380b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e380c4380 0x7f1e380b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3806e9b0 con 0x7f1e380cb760 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3806c3e0 con 0x7f1e3806b5a0 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e37200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380c8680 con 0x7f1e380c4380 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33080/0 (socket says 172.21.15.28:33080) 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e36800640 1 -- 172.21.15.28:0/161242505 learned_addr learned my addr 172.21.15.28:0/161242505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e35e00640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e3806b5a0 0x7f1e380b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.248+0000 7f1e35400640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e380c4380 0x7f1e380b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 -- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e380c4380 msgr2=0x7f1e380b4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e380c4380 0x7f1e380b4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 -- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e3806b5a0 msgr2=0x7f1e380b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e3806b5a0 0x7f1e380b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 -- 172.21.15.28:0/161242505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009a70 con 0x7f1e380cb760 2024-09-13T17:12:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e36800640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009eb0 tx=0x7f1e30004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3003c830 con 0x7f1e380cb760 2024-09-13T17:12:08.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e3003c990 con 0x7f1e380cb760 2024-09-13T17:12:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3003cd00 con 0x7f1e380cb760 2024-09-13T17:12:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e380c8960 con 0x7f1e380cb760 2024-09-13T17:12:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.249+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e380c8f20 con 0x7f1e380cb760 2024-09-13T17:12:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.250+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3806c5e0 con 0x7f1e380cb760 2024-09-13T17:12:08.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.251+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1e3003b3b0 con 0x7f1e380cb760 2024-09-13T17:12:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.252+0000 7f1e2e000640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 0x7f1e1806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.252+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1e300aac90 con 0x7f1e380cb760 2024-09-13T17:12:08.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.253+0000 7f1e35e00640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 0x7f1e1806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.253+0000 7f1e35e00640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 0x7f1e1806d0f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1e28007fd0 tx=0x7f1e28007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.256+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1e3806c5e0 con 0x7f1e380cb760 2024-09-13T17:12:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.437+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e380c63b0 con 0x7f1e380cb760 2024-09-13T17:12:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.438+0000 7f1e2e000640 1 -- 172.21.15.28:0/161242505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f1e30040090 con 0x7f1e380cb760 2024-09-13T17:12:08.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:08.440 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:05.641862+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 msgr2=0x7f1e1806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 0x7f1e1806d0f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f1e28007fd0 tx=0x7f1e28007de0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 msgr2=0x7f1e380b3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f1e30009eb0 tx=0x7f1e30004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 shutdown_connections 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e1806ac30 0x7f1e1806d0f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1e380cb760 0x7f1e380b3b40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1e380c4380 0x7f1e380b4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 --2- 172.21.15.28:0/161242505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e3806b5a0 0x7f1e380b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 >> 172.21.15.28:0/161242505 conn(0x7f1e380703a0 msgr2=0x7f1e380ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 shutdown_connections 2024-09-13T17:12:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.440+0000 7f1e37200640 1 -- 172.21.15.28:0/161242505 wait complete. 2024-09-13T17:12:08.458 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:12:08.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:12:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.612+0000 7faa35200640 1 Processor -- start 2024-09-13T17:12:08.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.612+0000 7faa35200640 1 -- start start 2024-09-13T17:12:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa35200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faa30154a90 0x7faa3015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa35200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3006b360 0x7faa300b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa35200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3015bb10 con 0x7faa300b0980 2024-09-13T17:12:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa35200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa300b0d80 con 0x7faa3006b360 2024-09-13T17:12:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa35200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3006b900 con 0x7faa30154a90 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa26a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3006b360 0x7faa300b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa26a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3006b360 0x7faa300b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33088/0 (socket says 172.21.15.28:33088) 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.613+0000 7faa26a00640 1 -- 172.21.15.28:0/457141985 learned_addr learned my addr 172.21.15.28:0/457141985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.614+0000 7faa2f400640 1 --2- 172.21.15.28:0/457141985 >> v2:172.21.15.189:3300/0 conn(0x7faa30154a90 0x7faa3015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.614+0000 7faa26a00640 1 -- 172.21.15.28:0/457141985 >> v2:172.21.15.189:3300/0 conn(0x7faa30154a90 msgr2=0x7faa3015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.614+0000 7faa26a00640 1 --2- 172.21.15.28:0/457141985 >> v2:172.21.15.189:3300/0 conn(0x7faa30154a90 0x7faa3015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.614+0000 7faa26a00640 1 -- 172.21.15.28:0/457141985 >> v1:172.21.15.189:6789/0 conn(0x7faa300b0980 legacy=0x7faa300af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.614+0000 7faa26a00640 1 -- 172.21.15.28:0/457141985 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3006bbe0 con 0x7faa3006b360 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.615+0000 7faa26a00640 1 --2- 172.21.15.28:0/457141985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3006b360 0x7faa300b0440 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faa300b3d70 tx=0x7faa1802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b29c5795a20f132 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.615+0000 7faa2ea00640 1 -- 172.21.15.28:0/457141985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18032440 con 0x7faa3006b360 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.615+0000 7faa2ea00640 1 -- 172.21.15.28:0/457141985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa180325a0 con 0x7faa3006b360 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.615+0000 7faa2ea00640 1 -- 172.21.15.28:0/457141985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18032930 con 0x7faa3006b360 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 -- 172.21.15.28:0/457141985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3006b360 msgr2=0x7faa300b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 --2- 172.21.15.28:0/457141985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3006b360 0x7faa300b0440 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7faa300b3d70 tx=0x7faa1802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 -- 172.21.15.28:0/457141985 shutdown_connections 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 --2- 172.21.15.28:0/457141985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3006b360 0x7faa300b0440 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 --2- 172.21.15.28:0/457141985 >> v2:172.21.15.189:3300/0 conn(0x7faa30154a90 0x7faa3015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 -- 172.21.15.28:0/457141985 >> 172.21.15.28:0/457141985 conn(0x7faa30072e40 msgr2=0x7faa30073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 -- 172.21.15.28:0/457141985 shutdown_connections 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.616+0000 7faa35200640 1 -- 172.21.15.28:0/457141985 wait complete. 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.618+0000 7faa35200640 1 Processor -- start 2024-09-13T17:12:08.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.618+0000 7faa35200640 1 -- start start 2024-09-13T17:12:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.618+0000 7faa35200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 0x7faa300c53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.618+0000 7faa35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa35200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa30154a90 0x7faa301f29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa35200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3013d340 con 0x7faa300b0980 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa35200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c65b0 con 0x7faa3006b360 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa35200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa300c5d10 con 0x7faa30154a90 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa2f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 0x7faa300c53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33094/0 (socket says 172.21.15.28:33094) 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 -- 172.21.15.28:0/2333832490 learned_addr learned my addr 172.21.15.28:0/2333832490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa2fe00640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa30154a90 0x7faa301f29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 -- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa30154a90 msgr2=0x7faa301f29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa30154a90 0x7faa301f29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 -- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 msgr2=0x7faa300c53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 0x7faa300c53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa26a00640 1 -- 172.21.15.28:0/2333832490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180094d0 con 0x7faa300b0980 2024-09-13T17:12:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.619+0000 7faa2f400640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 0x7faa300c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa26a00640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faa1802a2b0 tx=0x7faa180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18032440 con 0x7faa300b0980 2024-09-13T17:12:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa18030980 con 0x7faa300b0980 2024-09-13T17:12:08.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa18030d10 con 0x7faa300b0980 2024-09-13T17:12:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa300c5ff0 con 0x7faa300b0980 2024-09-13T17:12:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.620+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa30155340 con 0x7faa300b0980 2024-09-13T17:12:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.621+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa301551a0 con 0x7faa300b0980 2024-09-13T17:12:08.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.622+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faa18039040 con 0x7faa300b0980 2024-09-13T17:12:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.623+0000 7faa27e00640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 0x7faa0c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.623+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faa180a9bf0 con 0x7faa300b0980 2024-09-13T17:12:08.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.624+0000 7faa2f400640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 0x7faa0c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:08.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.624+0000 7faa2f400640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 0x7faa0c06d0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faa300afc30 tx=0x7faa20009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:08.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.627+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faa301551a0 con 0x7faa300b0980 2024-09-13T17:12:08.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.803+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa3006c1a0 con 0x7faa0c06ac00 2024-09-13T17:12:08.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.811+0000 7faa27e00640 1 -- 172.21.15.28:0/2333832490 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274598 (secure 0 0 0) 0x7faa20011dd0 con 0x7faa0c06ac00 2024-09-13T17:12:08.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:08.818 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:12:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 msgr2=0x7faa0c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 0x7faa0c06d0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7faa300afc30 tx=0x7faa20009190 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 msgr2=0x7faa301f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7faa1802a2b0 tx=0x7faa180046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 shutdown_connections 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa0c06ac00 0x7faa0c06d0c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa30154a90 0x7faa301f29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa300b0980 0x7faa301f2470 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 --2- 172.21.15.28:0/2333832490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3006b360 0x7faa300c53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 >> 172.21.15.28:0/2333832490 conn(0x7faa30072e40 msgr2=0x7faa30070480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 shutdown_connections 2024-09-13T17:12:08.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:08.815+0000 7faa35200640 1 -- 172.21.15.28:0/2333832490 wait complete. 2024-09-13T17:12:08.833 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-09-13T17:12:06.943088+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":368632,"kb_used_data":6496,"kb_used_omap":12,"kb_used_meta":361971,"kb_avail":754606088,"statfs":{"total":773094113280,"available":772716634112,"internally_reserved":0,"allocated":6651904,"data_stored":3435914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":370658873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000839"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:04.200072+0000","last_change":"2024-09-13T17:12:04.200072+0000","last_active":"2024-09-13T17:12:04.200072+0000","last_peered":"2024-09-13T17:12:04.200072+0000","last_clean":"2024-09-13T17:12:04.200072+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:04.200072+0000","last_undegraded":"2024-09-13T17:12:04.200072+0000","last_fullsized":"2024-09-13T17:12:04.200072+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:03.193631+0000","last_change":"2024-09-13T17:12:03.193631+0000","last_active":"2024-09-13T17:12:03.193631+0000","last_peered":"2024-09-13T17:12:03.193631+0000","last_clean":"2024-09-13T17:12:03.193631+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:03.193631+0000","last_undegraded":"2024-09-13T17:12:03.193631+0000","last_fullsized":"2024-09-13T17:12:03.193631+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":197,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-09-13T17:12:00.810447+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:12:00.810447+0000","last_peered":"2024-09-13T17:12:00.810447+0000","last_clean":"2024-09-13T17:12:00.810447+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:12:00.810447+0000","last_undegraded":"2024-09-13T17:12:00.810447+0000","last_fullsized":"2024-09-13T17:12:00.810447+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574900,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58620,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313220,"statfs":{"total":96636764160,"available":96576737280,"internally_reserved":0,"allocated":1277952,"data_stored":872520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.63900000000000001,"15min":0.63800000000000001},"min":{"1min":0.495,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.67900000000000005,"15min":0.68000000000000005},"min":{"1min":0.51200000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.056,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.96399999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.64000000000000001,"15min":0.63300000000000001},"min":{"1min":0.52100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.61099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":0.96099999999999997,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.878}]},{"osd":2,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.69599999999999995,"15min":0.70199999999999996},"min":{"1min":0.53900000000000003,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.60499999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":0.85399999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.59599999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.73799999999999999,"15min":0.73199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.83799999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.66800000000000004},"min":{"1min":0.38400000000000001,"5min":0.376,"15min":0.376},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.748,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.69899999999999995,"15min":0.70699999999999996},"min":{"1min":0.48299999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.78000000000000003,"15min":0.78700000000000003},"min":{"1min":0.45000000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.198,"5min":1.337,"15min":1.337},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:11:06 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.77100000000000002,"15min":0.77700000000000002},"min":{"1min":0.54300000000000004,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.032},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.54400000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.97099999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574901,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39980,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331860,"statfs":{"total":96636764160,"available":96595824640,"internally_reserved":0,"allocated":684032,"data_stored":282152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.42899999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.88400000000000001,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.72499999999999998,"15min":0.72099999999999997},"min":{"1min":0.47099999999999997,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.93700000000000006,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.73799999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.65900000000000003,"15min":0.64000000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.628},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65000000000000002,"15min":0.629},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.0129999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.48599999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.64600000000000002,"15min":0.66800000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82699999999999996,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84199999999999997,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.55000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68700000000000006,"15min":0.69699999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.84099999999999997,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95499999999999996,"5min":3.742,"15min":3.742},"last":0.72799999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.66900000000000004,"15min":0.65300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.0349999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.69499999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.71999999999999997,"15min":0.73199999999999998},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93600000000000005,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.72799999999999998,"15min":0.73299999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.0269999999999999,"5min":3.609,"15min":3.609},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.069,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.747,"5min":0.81100000000000005,"15min":0.80900000000000005},"min":{"1min":0.53800000000000003,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0069999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.79800000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39992,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331848,"statfs":{"total":96636764160,"available":96595812352,"internally_reserved":0,"allocated":684032,"data_stored":282152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.92400000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.004,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.876}]},{"osd":2,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":1.087},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.752}]},{"osd":3,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74199999999999999,"15min":0.75700000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.76700000000000002},"min":{"1min":0.56399999999999995,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93100000000000005,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.89300000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.622,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.96199999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.52600000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.622},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.83399999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.749,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.81200000000000006}]}]},{"osd":4,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":58024,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313816,"statfs":{"total":96636764160,"available":96577347584,"internally_reserved":0,"allocated":684032,"data_stored":282152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.64000000000000001,"15min":0.64100000000000001},"min":{"1min":0.46000000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.94299999999999995,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.746},"min":{"1min":0.48999999999999999,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.94199999999999995,"5min":1.341,"15min":1.341},"last":0.622}]},{"osd":1,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.51100000000000001,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85599999999999998,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.63300000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.64500000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.43099999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90100000000000002,"5min":1.377,"15min":1.377},"last":0.59799999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.65600000000000003,"15min":0.66000000000000003},"min":{"1min":0.47099999999999997,"5min":0.434,"15min":0.434},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.504},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.44600000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.625,"15min":0.64700000000000002},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.85299999999999998,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.436},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63700000000000001,"15min":0.65600000000000003},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.874,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.45600000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.64100000000000001,"15min":0.65100000000000002},"min":{"1min":0.39700000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.82799999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72499999999999998,"15min":0.75},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.92700000000000005,"5min":1.413,"15min":1.413},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:11:03 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67700000000000005,"15min":0.68500000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94199999999999995,"5min":0.998,"15min":0.998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.54600000000000004,"5min":0.371,"15min":0.371},"max":{"1min":0.95899999999999996,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.71699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45740,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326100,"statfs":{"total":96636764160,"available":96589926400,"internally_reserved":0,"allocated":684032,"data_stored":282152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.45900000000000002,"15min":0.46200000000000002},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":0.58899999999999997},"min":{"1min":0.36199999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.97099999999999997,"5min":2.96,"15min":2.96},"last":0.31900000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55100000000000005,"15min":0.52600000000000002},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":1.2809999999999999,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.73099999999999998,"15min":0.75700000000000001},"min":{"1min":0.45000000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.1830000000000001,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.39300000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.54700000000000004,"15min":0.52700000000000002},"min":{"1min":0.40600000000000003,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.60799999999999998,"15min":0.61199999999999999},"min":{"1min":0.42099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.41799999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.80600000000000005,"15min":0.83099999999999996},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.3560000000000001,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.77800000000000002,"15min":0.78100000000000003},"min":{"1min":0.56399999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.208,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":0.79500000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.391,"5min":1.391,"15min":1.391},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.83899999999999997,"15min":0.85499999999999998},"min":{"1min":0.61099999999999999,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.252,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.78300000000000003,"15min":0.76800000000000002},"min":{"1min":0.623,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.244,"5min":1.401,"15min":1.401},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80500000000000005,"15min":0.80300000000000005},"min":{"1min":0.58299999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.3300000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:11:04 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.86599999999999999,"15min":0.88100000000000001},"min":{"1min":0.65400000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.4139999999999999,"5min":2.899,"15min":2.899},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.83899999999999997,"15min":0.83899999999999997},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.278,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.68400000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574900,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":39972,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331868,"statfs":{"total":96636764160,"available":96595832832,"internally_reserved":0,"allocated":679936,"data_stored":281133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57799999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.443,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.45300000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":3.0699999999999998,"15min":1.3400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58599999999999997,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":3.1120000000000001,"15min":1.353},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59399999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.41599999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65600000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72799999999999998,"5min":1.121,"15min":1.121},"last":0.59699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.753,"15min":0.75800000000000001},"min":{"1min":0.56999999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2450000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.60599999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.397,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.99399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.048,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.76800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99299999999999999,"5min":1.569,"15min":1.569},"last":0.95799999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0940000000000001,"5min":1.413,"15min":1.413},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.024,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.024}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.073,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":1.034},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.56699999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.05}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45748,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326092,"statfs":{"total":96636764160,"available":96589918208,"internally_reserved":0,"allocated":684032,"data_stored":282152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.4939999999999998,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.63}]},{"osd":2,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.79500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.748,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":18.318000000000001,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.42699999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.84199999999999997,"15min":0.81999999999999995},"min":{"1min":0.51700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.1630000000000003,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.78,"15min":1.78},"last":0.80200000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6020000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":1.6020000000000001},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.84199999999999997,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.82799999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6599999999999999,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.82999999999999996},"min":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.69999999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.82899999999999996,"15min":0.83799999999999997},"min":{"1min":0.61699999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.472,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.746},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.544,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.544}]}]},{"osd":0,"up_from":13,"seq":55834574901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40556,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331284,"statfs":{"total":96636764160,"available":96595234816,"internally_reserved":0,"allocated":1273856,"data_stored":871501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.58199999999999996,"15min":0.72399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.99099999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.59199999999999997,"15min":0.70999999999999996},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1859999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.621,"15min":0.745},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66500000000000004,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.64200000000000002,"15min":0.76400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2190000000000001,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.60099999999999998,"15min":0.69899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.5069999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65700000000000003,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1000000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.92600000000000005},"min":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3160000000000001,"5min":4.734,"15min":4.734},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.84899999999999998,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.264,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.442}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.95999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.175,"5min":4.944,"15min":4.944},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.85399999999999998,"15min":0.93999999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2050000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94399999999999995,"15min":1.125},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1439999999999999,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.875,"15min":0.96899999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.57599999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89800000000000002,"15min":1.0069999999999999},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88500000000000001,"15min":0.98699999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.61399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:12:08.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:12:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.027+0000 7f3ea8c00640 1 Processor -- start 2024-09-13T17:12:09.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.028+0000 7f3ea8c00640 1 -- start start 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea406b360 0x7f3ea40b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40b0980 0x7f3ea40af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea8c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea40b0dd0 con 0x7f3ea4154a90 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea406b900 con 0x7f3ea406b360 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea406bbe0 con 0x7f3ea40b0980 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3e9be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea406b360 0x7f3ea40b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3e9be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea406b360 0x7f3ea40b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33098/0 (socket says 172.21.15.28:33098) 2024-09-13T17:12:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3e9be00640 1 -- 172.21.15.28:0/2613616871 learned_addr learned my addr 172.21.15.28:0/2613616871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.029+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40b0980 0x7f3ea40af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.030+0000 7f3e9be00640 1 -- 172.21.15.28:0/2613616871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40b0980 msgr2=0x7f3ea40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.030+0000 7f3e9be00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40b0980 0x7f3ea40af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.030+0000 7f3e9be00640 1 -- 172.21.15.28:0/2613616871 >> v1:172.21.15.189:6789/0 conn(0x7f3ea4154a90 legacy=0x7f3ea415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.030+0000 7f3e9be00640 1 -- 172.21.15.28:0/2613616871 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea406bec0 con 0x7f3ea406b360 2024-09-13T17:12:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3e9be00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea406b360 0x7f3ea40b0440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b78959cf56abedef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3ea2a00640 1 -- 172.21.15.28:0/2613616871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c032440 con 0x7f3ea406b360 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3ea2a00640 1 -- 172.21.15.28:0/2613616871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0325a0 con 0x7f3ea406b360 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3ea2a00640 1 -- 172.21.15.28:0/2613616871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c032930 con 0x7f3ea406b360 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3ea8c00640 1 -- 172.21.15.28:0/2613616871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea406b360 msgr2=0x7f3ea40b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.031+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea406b360 0x7f3ea40b0440 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c02a250 comp rx=0 tx=0).stop 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 -- 172.21.15.28:0/2613616871 shutdown_connections 2024-09-13T17:12:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40b0980 0x7f3ea40af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/2613616871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea406b360 0x7f3ea40b0440 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 -- 172.21.15.28:0/2613616871 >> 172.21.15.28:0/2613616871 conn(0x7f3ea4072e40 msgr2=0x7f3ea4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 -- 172.21.15.28:0/2613616871 shutdown_connections 2024-09-13T17:12:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.032+0000 7f3ea8c00640 1 -- 172.21.15.28:0/2613616871 wait complete. 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.033+0000 7f3ea8c00640 1 Processor -- start 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.033+0000 7f3ea8c00640 1 -- start start 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea406b360 0x7f3ea4153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4154a90 0x7f3ea4156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4154240 con 0x7f3ea40b0980 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea41ede60 con 0x7f3ea4154a90 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3ea8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea41f1480 con 0x7f3ea406b360 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.034+0000 7f3e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33104/0 (socket says 172.21.15.28:33104) 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 -- 172.21.15.28:0/3520828708 learned_addr learned my addr 172.21.15.28:0/3520828708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3ea3400640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea406b360 0x7f3ea4153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4154a90 0x7f3ea4156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 -- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea406b360 msgr2=0x7f3ea4153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea406b360 0x7f3ea4153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 -- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4154a90 msgr2=0x7f3ea4156760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4154a90 0x7f3ea4156760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 -- 172.21.15.28:0/3520828708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0094d0 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.035+0000 7f3e9be00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000c00 tx=0x7f3e8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.036+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c032440 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.036+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c030b80 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.036+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea41f1760 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.036+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c03d440 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.036+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea41f1d20 con 0x7f3ea40b0980 2024-09-13T17:12:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.037+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4158240 con 0x7f3ea40b0980 2024-09-13T17:12:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.038+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e8c039040 con 0x7f3ea40b0980 2024-09-13T17:12:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.039+0000 7f3e9b400640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 0x7f3e8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.039+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e8c0a9e10 con 0x7f3ea40b0980 2024-09-13T17:12:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.039+0000 7f3ea3400640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 0x7f3e8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.040+0000 7f3ea3400640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 0x7f3e8006d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e94000f60 tx=0x7f3e94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.043+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3e8c0ad030 con 0x7f3ea40b0980 2024-09-13T17:12:09.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.218+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ea40af9f0 con 0x7f3e8006ac00 2024-09-13T17:12:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.227+0000 7f3e9b400640 1 -- 172.21.15.28:0/3520828708 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289357 (secure 0 0 0) 0x7f3e94001d10 con 0x7f3e8006ac00 2024-09-13T17:12:09.232 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 msgr2=0x7f3e8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 0x7f3e8006d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3e94000f60 tx=0x7f3e94009040 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 msgr2=0x7f3ea40a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c000c00 tx=0x7f3e8c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 shutdown_connections 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8006ac00 0x7f3e8006d0c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea4154a90 0x7f3ea4156760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40b0980 0x7f3ea40a8440 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 --2- 172.21.15.28:0/3520828708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea406b360 0x7f3ea4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 >> 172.21.15.28:0/3520828708 conn(0x7f3ea4072e40 msgr2=0x7f3ea40cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:09.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.231+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 shutdown_connections 2024-09-13T17:12:09.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.232+0000 7f3ea8c00640 1 -- 172.21.15.28:0/3520828708 wait complete. 2024-09-13T17:12:09.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-13T17:12:08.943498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":373164,"kb_used_data":6944,"kb_used_omap":12,"kb_used_meta":366067,"kb_avail":754601556,"statfs":{"total":773094113280,"available":772711993344,"internally_reserved":0,"allocated":7110656,"data_stored":3641910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":374853177},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.742750"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650014+0000","last_change":"2024-09-13T17:12:04.668527+0000","last_active":"2024-09-13T17:12:05.650014+0000","last_peered":"2024-09-13T17:12:05.650014+0000","last_clean":"2024-09-13T17:12:05.650014+0000","last_became_active":"2024-09-13T17:12:04.668252+0000","last_became_peered":"2024-09-13T17:12:04.668252+0000","last_unstale":"2024-09-13T17:12:05.650014+0000","last_undegraded":"2024-09-13T17:12:05.650014+0000","last_fullsized":"2024-09-13T17:12:05.650014+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:32.050534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543741+0000","last_change":"2024-09-13T17:12:04.649728+0000","last_active":"2024-09-13T17:12:06.543741+0000","last_peered":"2024-09-13T17:12:06.543741+0000","last_clean":"2024-09-13T17:12:06.543741+0000","last_became_active":"2024-09-13T17:12:04.649455+0000","last_became_peered":"2024-09-13T17:12:04.649455+0000","last_unstale":"2024-09-13T17:12:06.543741+0000","last_undegraded":"2024-09-13T17:12:06.543741+0000","last_fullsized":"2024-09-13T17:12:06.543741+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:30.121661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543336+0000","last_change":"2024-09-13T17:12:04.673569+0000","last_active":"2024-09-13T17:12:06.543336+0000","last_peered":"2024-09-13T17:12:06.543336+0000","last_clean":"2024-09-13T17:12:06.543336+0000","last_became_active":"2024-09-13T17:12:04.670872+0000","last_became_peered":"2024-09-13T17:12:04.670872+0000","last_unstale":"2024-09-13T17:12:06.543336+0000","last_undegraded":"2024-09-13T17:12:06.543336+0000","last_fullsized":"2024-09-13T17:12:06.543336+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.182309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651640+0000","last_change":"2024-09-13T17:12:04.664327+0000","last_active":"2024-09-13T17:12:05.651640+0000","last_peered":"2024-09-13T17:12:05.651640+0000","last_clean":"2024-09-13T17:12:05.651640+0000","last_became_active":"2024-09-13T17:12:04.663924+0000","last_became_peered":"2024-09-13T17:12:04.663924+0000","last_unstale":"2024-09-13T17:12:05.651640+0000","last_undegraded":"2024-09-13T17:12:05.651640+0000","last_fullsized":"2024-09-13T17:12:05.651640+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:18.577596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.649705+0000","last_change":"2024-09-13T17:12:04.669283+0000","last_active":"2024-09-13T17:12:05.649705+0000","last_peered":"2024-09-13T17:12:05.649705+0000","last_clean":"2024-09-13T17:12:05.649705+0000","last_became_active":"2024-09-13T17:12:04.668378+0000","last_became_peered":"2024-09-13T17:12:04.668378+0000","last_unstale":"2024-09-13T17:12:05.649705+0000","last_undegraded":"2024-09-13T17:12:05.649705+0000","last_fullsized":"2024-09-13T17:12:05.649705+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:27.484514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543558+0000","last_change":"2024-09-13T17:12:04.661470+0000","last_active":"2024-09-13T17:12:06.543558+0000","last_peered":"2024-09-13T17:12:06.543558+0000","last_clean":"2024-09-13T17:12:06.543558+0000","last_became_active":"2024-09-13T17:12:04.661231+0000","last_became_peered":"2024-09-13T17:12:04.661231+0000","last_unstale":"2024-09-13T17:12:06.543558+0000","last_undegraded":"2024-09-13T17:12:06.543558+0000","last_fullsized":"2024-09-13T17:12:06.543558+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:15.146524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650573+0000","last_change":"2024-09-13T17:12:04.653419+0000","last_active":"2024-09-13T17:12:05.650573+0000","last_peered":"2024-09-13T17:12:05.650573+0000","last_clean":"2024-09-13T17:12:05.650573+0000","last_became_active":"2024-09-13T17:12:04.652761+0000","last_became_peered":"2024-09-13T17:12:04.652761+0000","last_unstale":"2024-09-13T17:12:05.650573+0000","last_undegraded":"2024-09-13T17:12:05.650573+0000","last_fullsized":"2024-09-13T17:12:05.650573+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:40.651699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542622+0000","last_change":"2024-09-13T17:12:04.655830+0000","last_active":"2024-09-13T17:12:06.542622+0000","last_peered":"2024-09-13T17:12:06.542622+0000","last_clean":"2024-09-13T17:12:06.542622+0000","last_became_active":"2024-09-13T17:12:04.655540+0000","last_became_peered":"2024-09-13T17:12:04.655540+0000","last_unstale":"2024-09-13T17:12:06.542622+0000","last_undegraded":"2024-09-13T17:12:06.542622+0000","last_fullsized":"2024-09-13T17:12:06.542622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:52.346780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651610+0000","last_change":"2024-09-13T17:12:04.661896+0000","last_active":"2024-09-13T17:12:05.651610+0000","last_peered":"2024-09-13T17:12:05.651610+0000","last_clean":"2024-09-13T17:12:05.651610+0000","last_became_active":"2024-09-13T17:12:04.657109+0000","last_became_peered":"2024-09-13T17:12:04.657109+0000","last_unstale":"2024-09-13T17:12:05.651610+0000","last_undegraded":"2024-09-13T17:12:05.651610+0000","last_fullsized":"2024-09-13T17:12:05.651610+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:58.789114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543982+0000","last_change":"2024-09-13T17:12:04.659543+0000","last_active":"2024-09-13T17:12:06.543982+0000","last_peered":"2024-09-13T17:12:06.543982+0000","last_clean":"2024-09-13T17:12:06.543982+0000","last_became_active":"2024-09-13T17:12:04.659250+0000","last_became_peered":"2024-09-13T17:12:04.659250+0000","last_unstale":"2024-09-13T17:12:06.543982+0000","last_undegraded":"2024-09-13T17:12:06.543982+0000","last_fullsized":"2024-09-13T17:12:06.543982+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:20.497733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543951+0000","last_change":"2024-09-13T17:12:04.657161+0000","last_active":"2024-09-13T17:12:06.543951+0000","last_peered":"2024-09-13T17:12:06.543951+0000","last_clean":"2024-09-13T17:12:06.543951+0000","last_became_active":"2024-09-13T17:12:04.654449+0000","last_became_peered":"2024-09-13T17:12:04.654449+0000","last_unstale":"2024-09-13T17:12:06.543951+0000","last_undegraded":"2024-09-13T17:12:06.543951+0000","last_fullsized":"2024-09-13T17:12:06.543951+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:22.164287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651754+0000","last_change":"2024-09-13T17:12:04.664165+0000","last_active":"2024-09-13T17:12:05.651754+0000","last_peered":"2024-09-13T17:12:05.651754+0000","last_clean":"2024-09-13T17:12:05.651754+0000","last_became_active":"2024-09-13T17:12:04.663570+0000","last_became_peered":"2024-09-13T17:12:04.663570+0000","last_unstale":"2024-09-13T17:12:05.651754+0000","last_undegraded":"2024-09-13T17:12:05.651754+0000","last_fullsized":"2024-09-13T17:12:05.651754+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:53.515375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.759967+0000","last_change":"2024-09-13T17:12:04.665382+0000","last_active":"2024-09-13T17:12:05.759967+0000","last_peered":"2024-09-13T17:12:05.759967+0000","last_clean":"2024-09-13T17:12:05.759967+0000","last_became_active":"2024-09-13T17:12:04.663870+0000","last_became_peered":"2024-09-13T17:12:04.663870+0000","last_unstale":"2024-09-13T17:12:05.759967+0000","last_undegraded":"2024-09-13T17:12:05.759967+0000","last_fullsized":"2024-09-13T17:12:05.759967+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:53.049771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650804+0000","last_change":"2024-09-13T17:12:04.656954+0000","last_active":"2024-09-13T17:12:05.650804+0000","last_peered":"2024-09-13T17:12:05.650804+0000","last_clean":"2024-09-13T17:12:05.650804+0000","last_became_active":"2024-09-13T17:12:04.656040+0000","last_became_peered":"2024-09-13T17:12:04.656040+0000","last_unstale":"2024-09-13T17:12:05.650804+0000","last_undegraded":"2024-09-13T17:12:05.650804+0000","last_fullsized":"2024-09-13T17:12:05.650804+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:24.341357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650772+0000","last_change":"2024-09-13T17:12:04.659559+0000","last_active":"2024-09-13T17:12:05.650772+0000","last_peered":"2024-09-13T17:12:05.650772+0000","last_clean":"2024-09-13T17:12:05.650772+0000","last_became_active":"2024-09-13T17:12:04.656433+0000","last_became_peered":"2024-09-13T17:12:04.656433+0000","last_unstale":"2024-09-13T17:12:05.650772+0000","last_undegraded":"2024-09-13T17:12:05.650772+0000","last_fullsized":"2024-09-13T17:12:05.650772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:03.936141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542766+0000","last_change":"2024-09-13T17:12:04.662284+0000","last_active":"2024-09-13T17:12:06.542766+0000","last_peered":"2024-09-13T17:12:06.542766+0000","last_clean":"2024-09-13T17:12:06.542766+0000","last_became_active":"2024-09-13T17:12:04.661967+0000","last_became_peered":"2024-09-13T17:12:04.661967+0000","last_unstale":"2024-09-13T17:12:06.542766+0000","last_undegraded":"2024-09-13T17:12:06.542766+0000","last_fullsized":"2024-09-13T17:12:06.542766+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:51.635576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543439+0000","last_change":"2024-09-13T17:12:04.661732+0000","last_active":"2024-09-13T17:12:06.543439+0000","last_peered":"2024-09-13T17:12:06.543439+0000","last_clean":"2024-09-13T17:12:06.543439+0000","last_became_active":"2024-09-13T17:12:04.661450+0000","last_became_peered":"2024-09-13T17:12:04.661450+0000","last_unstale":"2024-09-13T17:12:06.543439+0000","last_undegraded":"2024-09-13T17:12:06.543439+0000","last_fullsized":"2024-09-13T17:12:06.543439+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:29.220359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543463+0000","last_change":"2024-09-13T17:12:04.676253+0000","last_active":"2024-09-13T17:12:06.543463+0000","last_peered":"2024-09-13T17:12:06.543463+0000","last_clean":"2024-09-13T17:12:06.543463+0000","last_became_active":"2024-09-13T17:12:04.676010+0000","last_became_peered":"2024-09-13T17:12:04.676010+0000","last_unstale":"2024-09-13T17:12:06.543463+0000","last_undegraded":"2024-09-13T17:12:06.543463+0000","last_fullsized":"2024-09-13T17:12:06.543463+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:41.751614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.649896+0000","last_change":"2024-09-13T17:12:04.673958+0000","last_active":"2024-09-13T17:12:05.649896+0000","last_peered":"2024-09-13T17:12:05.649896+0000","last_clean":"2024-09-13T17:12:05.649896+0000","last_became_active":"2024-09-13T17:12:04.673746+0000","last_became_peered":"2024-09-13T17:12:04.673746+0000","last_unstale":"2024-09-13T17:12:05.649896+0000","last_undegraded":"2024-09-13T17:12:05.649896+0000","last_fullsized":"2024-09-13T17:12:05.649896+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:29.752316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542842+0000","last_change":"2024-09-13T17:12:04.659348+0000","last_active":"2024-09-13T17:12:06.542842+0000","last_peered":"2024-09-13T17:12:06.542842+0000","last_clean":"2024-09-13T17:12:06.542842+0000","last_became_active":"2024-09-13T17:12:04.659103+0000","last_became_peered":"2024-09-13T17:12:04.659103+0000","last_unstale":"2024-09-13T17:12:06.542842+0000","last_undegraded":"2024-09-13T17:12:06.542842+0000","last_fullsized":"2024-09-13T17:12:06.542842+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:36:16.159103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543168+0000","last_change":"2024-09-13T17:12:04.669343+0000","last_active":"2024-09-13T17:12:06.543168+0000","last_peered":"2024-09-13T17:12:06.543168+0000","last_clean":"2024-09-13T17:12:06.543168+0000","last_became_active":"2024-09-13T17:12:04.668792+0000","last_became_peered":"2024-09-13T17:12:04.668792+0000","last_unstale":"2024-09-13T17:12:06.543168+0000","last_undegraded":"2024-09-13T17:12:06.543168+0000","last_fullsized":"2024-09-13T17:12:06.543168+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:59.779047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542985+0000","last_change":"2024-09-13T17:12:04.665672+0000","last_active":"2024-09-13T17:12:06.542985+0000","last_peered":"2024-09-13T17:12:06.542985+0000","last_clean":"2024-09-13T17:12:06.542985+0000","last_became_active":"2024-09-13T17:12:04.665278+0000","last_became_peered":"2024-09-13T17:12:04.665278+0000","last_unstale":"2024-09-13T17:12:06.542985+0000","last_undegraded":"2024-09-13T17:12:06.542985+0000","last_fullsized":"2024-09-13T17:12:06.542985+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:34.542706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543161+0000","last_change":"2024-09-13T17:12:04.670157+0000","last_active":"2024-09-13T17:12:06.543161+0000","last_peered":"2024-09-13T17:12:06.543161+0000","last_clean":"2024-09-13T17:12:06.543161+0000","last_became_active":"2024-09-13T17:12:04.669458+0000","last_became_peered":"2024-09-13T17:12:04.669458+0000","last_unstale":"2024-09-13T17:12:06.543161+0000","last_undegraded":"2024-09-13T17:12:06.543161+0000","last_fullsized":"2024-09-13T17:12:06.543161+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:51.846384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.759800+0000","last_change":"2024-09-13T17:12:04.672391+0000","last_active":"2024-09-13T17:12:05.759800+0000","last_peered":"2024-09-13T17:12:05.759800+0000","last_clean":"2024-09-13T17:12:05.759800+0000","last_became_active":"2024-09-13T17:12:04.672136+0000","last_became_peered":"2024-09-13T17:12:04.672136+0000","last_unstale":"2024-09-13T17:12:05.759800+0000","last_undegraded":"2024-09-13T17:12:05.759800+0000","last_fullsized":"2024-09-13T17:12:05.759800+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:36.657381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650055+0000","last_change":"2024-09-13T17:12:04.671681+0000","last_active":"2024-09-13T17:12:05.650055+0000","last_peered":"2024-09-13T17:12:05.650055+0000","last_clean":"2024-09-13T17:12:05.650055+0000","last_became_active":"2024-09-13T17:12:04.671161+0000","last_became_peered":"2024-09-13T17:12:04.671161+0000","last_unstale":"2024-09-13T17:12:05.650055+0000","last_undegraded":"2024-09-13T17:12:05.650055+0000","last_fullsized":"2024-09-13T17:12:05.650055+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:52.507134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.652030+0000","last_change":"2024-09-13T17:12:04.658267+0000","last_active":"2024-09-13T17:12:05.652030+0000","last_peered":"2024-09-13T17:12:05.652030+0000","last_clean":"2024-09-13T17:12:05.652030+0000","last_became_active":"2024-09-13T17:12:04.658000+0000","last_became_peered":"2024-09-13T17:12:04.658000+0000","last_unstale":"2024-09-13T17:12:05.652030+0000","last_undegraded":"2024-09-13T17:12:05.652030+0000","last_fullsized":"2024-09-13T17:12:05.652030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:56.204290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544412+0000","last_change":"2024-09-13T17:12:04.662271+0000","last_active":"2024-09-13T17:12:06.544412+0000","last_peered":"2024-09-13T17:12:06.544412+0000","last_clean":"2024-09-13T17:12:06.544412+0000","last_became_active":"2024-09-13T17:12:04.661620+0000","last_became_peered":"2024-09-13T17:12:04.661620+0000","last_unstale":"2024-09-13T17:12:06.544412+0000","last_undegraded":"2024-09-13T17:12:06.544412+0000","last_fullsized":"2024-09-13T17:12:06.544412+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.182309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.649895+0000","last_change":"2024-09-13T17:12:04.670406+0000","last_active":"2024-09-13T17:12:05.649895+0000","last_peered":"2024-09-13T17:12:05.649895+0000","last_clean":"2024-09-13T17:12:05.649895+0000","last_became_active":"2024-09-13T17:12:04.670113+0000","last_became_peered":"2024-09-13T17:12:04.670113+0000","last_unstale":"2024-09-13T17:12:05.649895+0000","last_undegraded":"2024-09-13T17:12:05.649895+0000","last_fullsized":"2024-09-13T17:12:05.649895+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:11.333932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543070+0000","last_change":"2024-09-13T17:12:04.653500+0000","last_active":"2024-09-13T17:12:06.543070+0000","last_peered":"2024-09-13T17:12:06.543070+0000","last_clean":"2024-09-13T17:12:06.543070+0000","last_became_active":"2024-09-13T17:12:04.653156+0000","last_became_peered":"2024-09-13T17:12:04.653156+0000","last_unstale":"2024-09-13T17:12:06.543070+0000","last_undegraded":"2024-09-13T17:12:06.543070+0000","last_fullsized":"2024-09-13T17:12:06.543070+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:18:07.719115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543089+0000","last_change":"2024-09-13T17:12:04.667784+0000","last_active":"2024-09-13T17:12:06.543089+0000","last_peered":"2024-09-13T17:12:06.543089+0000","last_clean":"2024-09-13T17:12:06.543089+0000","last_became_active":"2024-09-13T17:12:04.667198+0000","last_became_peered":"2024-09-13T17:12:04.667198+0000","last_unstale":"2024-09-13T17:12:06.543089+0000","last_undegraded":"2024-09-13T17:12:06.543089+0000","last_fullsized":"2024-09-13T17:12:06.543089+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:40.884648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.649648+0000","last_change":"2024-09-13T17:12:04.669698+0000","last_active":"2024-09-13T17:12:05.649648+0000","last_peered":"2024-09-13T17:12:05.649648+0000","last_clean":"2024-09-13T17:12:05.649648+0000","last_became_active":"2024-09-13T17:12:04.668162+0000","last_became_peered":"2024-09-13T17:12:04.668162+0000","last_unstale":"2024-09-13T17:12:05.649648+0000","last_undegraded":"2024-09-13T17:12:05.649648+0000","last_fullsized":"2024-09-13T17:12:05.649648+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:19.666679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543094+0000","last_change":"2024-09-13T17:12:04.658707+0000","last_active":"2024-09-13T17:12:06.543094+0000","last_peered":"2024-09-13T17:12:06.543094+0000","last_clean":"2024-09-13T17:12:06.543094+0000","last_became_active":"2024-09-13T17:12:04.658478+0000","last_became_peered":"2024-09-13T17:12:04.658478+0000","last_unstale":"2024-09-13T17:12:06.543094+0000","last_undegraded":"2024-09-13T17:12:06.543094+0000","last_fullsized":"2024-09-13T17:12:06.543094+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:53.515375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544566+0000","last_change":"2024-09-13T17:12:04.669269+0000","last_active":"2024-09-13T17:12:06.544566+0000","last_peered":"2024-09-13T17:12:06.544566+0000","last_clean":"2024-09-13T17:12:06.544566+0000","last_became_active":"2024-09-13T17:12:04.669038+0000","last_became_peered":"2024-09-13T17:12:04.669038+0000","last_unstale":"2024-09-13T17:12:06.544566+0000","last_undegraded":"2024-09-13T17:12:06.544566+0000","last_fullsized":"2024-09-13T17:12:06.544566+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:17:51.846384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"60'4","reported_seq":15,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544580+0000","last_change":"2024-09-13T17:12:04.223046+0000","last_active":"2024-09-13T17:12:06.544580+0000","last_peered":"2024-09-13T17:12:06.544580+0000","last_clean":"2024-09-13T17:12:06.544580+0000","last_became_active":"2024-09-13T17:12:04.222720+0000","last_became_peered":"2024-09-13T17:12:04.222720+0000","last_unstale":"2024-09-13T17:12:06.544580+0000","last_undegraded":"2024-09-13T17:12:06.544580+0000","last_fullsized":"2024-09-13T17:12:06.544580+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:58.686370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544100+0000","last_change":"2024-09-13T17:12:04.670524+0000","last_active":"2024-09-13T17:12:06.544100+0000","last_peered":"2024-09-13T17:12:06.544100+0000","last_clean":"2024-09-13T17:12:06.544100+0000","last_became_active":"2024-09-13T17:12:04.669792+0000","last_became_peered":"2024-09-13T17:12:04.669792+0000","last_unstale":"2024-09-13T17:12:06.544100+0000","last_undegraded":"2024-09-13T17:12:06.544100+0000","last_fullsized":"2024-09-13T17:12:06.544100+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:04.410880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543712+0000","last_change":"2024-09-13T17:12:04.236191+0000","last_active":"2024-09-13T17:12:06.543712+0000","last_peered":"2024-09-13T17:12:06.543712+0000","last_clean":"2024-09-13T17:12:06.543712+0000","last_became_active":"2024-09-13T17:12:04.235852+0000","last_became_peered":"2024-09-13T17:12:04.235852+0000","last_unstale":"2024-09-13T17:12:06.543712+0000","last_undegraded":"2024-09-13T17:12:06.543712+0000","last_fullsized":"2024-09-13T17:12:06.543712+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:55.197849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544596+0000","last_change":"2024-09-13T17:12:04.657077+0000","last_active":"2024-09-13T17:12:06.544596+0000","last_peered":"2024-09-13T17:12:06.544596+0000","last_clean":"2024-09-13T17:12:06.544596+0000","last_became_active":"2024-09-13T17:12:04.656149+0000","last_became_peered":"2024-09-13T17:12:04.656149+0000","last_unstale":"2024-09-13T17:12:06.544596+0000","last_undegraded":"2024-09-13T17:12:06.544596+0000","last_fullsized":"2024-09-13T17:12:06.544596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:40.884648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544749+0000","last_change":"2024-09-13T17:12:04.227543+0000","last_active":"2024-09-13T17:12:06.544749+0000","last_peered":"2024-09-13T17:12:06.544749+0000","last_clean":"2024-09-13T17:12:06.544749+0000","last_became_active":"2024-09-13T17:12:04.227120+0000","last_became_peered":"2024-09-13T17:12:04.227120+0000","last_unstale":"2024-09-13T17:12:06.544749+0000","last_undegraded":"2024-09-13T17:12:06.544749+0000","last_fullsized":"2024-09-13T17:12:06.544749+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:11.633598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544554+0000","last_change":"2024-09-13T17:12:04.662818+0000","last_active":"2024-09-13T17:12:06.544554+0000","last_peered":"2024-09-13T17:12:06.544554+0000","last_clean":"2024-09-13T17:12:06.544554+0000","last_became_active":"2024-09-13T17:12:04.661999+0000","last_became_peered":"2024-09-13T17:12:04.661999+0000","last_unstale":"2024-09-13T17:12:06.544554+0000","last_undegraded":"2024-09-13T17:12:06.544554+0000","last_fullsized":"2024-09-13T17:12:06.544554+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:56.204290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651767+0000","last_change":"2024-09-13T17:12:04.225620+0000","last_active":"2024-09-13T17:12:05.651767+0000","last_peered":"2024-09-13T17:12:05.651767+0000","last_clean":"2024-09-13T17:12:05.651767+0000","last_became_active":"2024-09-13T17:12:04.225048+0000","last_became_peered":"2024-09-13T17:12:04.225048+0000","last_unstale":"2024-09-13T17:12:05.651767+0000","last_undegraded":"2024-09-13T17:12:05.651767+0000","last_fullsized":"2024-09-13T17:12:05.651767+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:53.624817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543093+0000","last_change":"2024-09-13T17:12:04.665515+0000","last_active":"2024-09-13T17:12:06.543093+0000","last_peered":"2024-09-13T17:12:06.543093+0000","last_clean":"2024-09-13T17:12:06.543093+0000","last_became_active":"2024-09-13T17:12:04.664910+0000","last_became_peered":"2024-09-13T17:12:04.664910+0000","last_unstale":"2024-09-13T17:12:06.543093+0000","last_undegraded":"2024-09-13T17:12:06.543093+0000","last_fullsized":"2024-09-13T17:12:06.543093+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:20:48.670638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651513+0000","last_change":"2024-09-13T17:12:04.229526+0000","last_active":"2024-09-13T17:12:05.651513+0000","last_peered":"2024-09-13T17:12:05.651513+0000","last_clean":"2024-09-13T17:12:05.651513+0000","last_became_active":"2024-09-13T17:12:04.228348+0000","last_became_peered":"2024-09-13T17:12:04.228348+0000","last_unstale":"2024-09-13T17:12:05.651513+0000","last_undegraded":"2024-09-13T17:12:05.651513+0000","last_fullsized":"2024-09-13T17:12:05.651513+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:03.404439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542995+0000","last_change":"2024-09-13T17:12:04.658866+0000","last_active":"2024-09-13T17:12:06.542995+0000","last_peered":"2024-09-13T17:12:06.542995+0000","last_clean":"2024-09-13T17:12:06.542995+0000","last_became_active":"2024-09-13T17:12:04.658541+0000","last_became_peered":"2024-09-13T17:12:04.658541+0000","last_unstale":"2024-09-13T17:12:06.542995+0000","last_undegraded":"2024-09-13T17:12:06.542995+0000","last_fullsized":"2024-09-13T17:12:06.542995+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:39:44.894998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544276+0000","last_change":"2024-09-13T17:12:04.228748+0000","last_active":"2024-09-13T17:12:06.544276+0000","last_peered":"2024-09-13T17:12:06.544276+0000","last_clean":"2024-09-13T17:12:06.544276+0000","last_became_active":"2024-09-13T17:12:04.228191+0000","last_became_peered":"2024-09-13T17:12:04.228191+0000","last_unstale":"2024-09-13T17:12:06.544276+0000","last_undegraded":"2024-09-13T17:12:06.544276+0000","last_fullsized":"2024-09-13T17:12:06.544276+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:59.383066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.759884+0000","last_change":"2024-09-13T17:12:04.670401+0000","last_active":"2024-09-13T17:12:05.759884+0000","last_peered":"2024-09-13T17:12:05.759884+0000","last_clean":"2024-09-13T17:12:05.759884+0000","last_became_active":"2024-09-13T17:12:04.670135+0000","last_became_peered":"2024-09-13T17:12:04.670135+0000","last_unstale":"2024-09-13T17:12:05.759884+0000","last_undegraded":"2024-09-13T17:12:05.759884+0000","last_fullsized":"2024-09-13T17:12:05.759884+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:46.172660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651282+0000","last_change":"2024-09-13T17:12:04.225871+0000","last_active":"2024-09-13T17:12:05.651282+0000","last_peered":"2024-09-13T17:12:05.651282+0000","last_clean":"2024-09-13T17:12:05.651282+0000","last_became_active":"2024-09-13T17:12:04.222171+0000","last_became_peered":"2024-09-13T17:12:04.222171+0000","last_unstale":"2024-09-13T17:12:05.651282+0000","last_undegraded":"2024-09-13T17:12:05.651282+0000","last_fullsized":"2024-09-13T17:12:05.651282+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:37.799876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542720+0000","last_change":"2024-09-13T17:12:04.664513+0000","last_active":"2024-09-13T17:12:06.542720+0000","last_peered":"2024-09-13T17:12:06.542720+0000","last_clean":"2024-09-13T17:12:06.542720+0000","last_became_active":"2024-09-13T17:12:04.662688+0000","last_became_peered":"2024-09-13T17:12:04.662688+0000","last_unstale":"2024-09-13T17:12:06.542720+0000","last_undegraded":"2024-09-13T17:12:06.542720+0000","last_fullsized":"2024-09-13T17:12:06.542720+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:19.933312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650681+0000","last_change":"2024-09-13T17:12:04.224980+0000","last_active":"2024-09-13T17:12:05.650681+0000","last_peered":"2024-09-13T17:12:05.650681+0000","last_clean":"2024-09-13T17:12:05.650681+0000","last_became_active":"2024-09-13T17:12:04.224451+0000","last_became_peered":"2024-09-13T17:12:04.224451+0000","last_unstale":"2024-09-13T17:12:05.650681+0000","last_undegraded":"2024-09-13T17:12:05.650681+0000","last_fullsized":"2024-09-13T17:12:05.650681+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:13:01.687412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651772+0000","last_change":"2024-09-13T17:12:04.655486+0000","last_active":"2024-09-13T17:12:05.651772+0000","last_peered":"2024-09-13T17:12:05.651772+0000","last_clean":"2024-09-13T17:12:05.651772+0000","last_became_active":"2024-09-13T17:12:04.654075+0000","last_became_peered":"2024-09-13T17:12:04.654075+0000","last_unstale":"2024-09-13T17:12:05.651772+0000","last_undegraded":"2024-09-13T17:12:05.651772+0000","last_fullsized":"2024-09-13T17:12:05.651772+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:58:50.072140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543616+0000","last_change":"2024-09-13T17:12:04.233054+0000","last_active":"2024-09-13T17:12:06.543616+0000","last_peered":"2024-09-13T17:12:06.543616+0000","last_clean":"2024-09-13T17:12:06.543616+0000","last_became_active":"2024-09-13T17:12:04.232001+0000","last_became_peered":"2024-09-13T17:12:04.232001+0000","last_unstale":"2024-09-13T17:12:06.543616+0000","last_undegraded":"2024-09-13T17:12:06.543616+0000","last_fullsized":"2024-09-13T17:12:06.543616+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:59.383066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543615+0000","last_change":"2024-09-13T17:12:04.674232+0000","last_active":"2024-09-13T17:12:06.543615+0000","last_peered":"2024-09-13T17:12:06.543615+0000","last_clean":"2024-09-13T17:12:06.543615+0000","last_became_active":"2024-09-13T17:12:04.673954+0000","last_became_peered":"2024-09-13T17:12:04.673954+0000","last_unstale":"2024-09-13T17:12:06.543615+0000","last_undegraded":"2024-09-13T17:12:06.543615+0000","last_fullsized":"2024-09-13T17:12:06.543615+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:15.146524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543801+0000","last_change":"2024-09-13T17:12:04.232697+0000","last_active":"2024-09-13T17:12:06.543801+0000","last_peered":"2024-09-13T17:12:06.543801+0000","last_clean":"2024-09-13T17:12:06.543801+0000","last_became_active":"2024-09-13T17:12:04.232385+0000","last_became_peered":"2024-09-13T17:12:04.232385+0000","last_unstale":"2024-09-13T17:12:06.543801+0000","last_undegraded":"2024-09-13T17:12:06.543801+0000","last_fullsized":"2024-09-13T17:12:06.543801+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:58.772605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543561+0000","last_change":"2024-09-13T17:12:04.657626+0000","last_active":"2024-09-13T17:12:06.543561+0000","last_peered":"2024-09-13T17:12:06.543561+0000","last_clean":"2024-09-13T17:12:06.543561+0000","last_became_active":"2024-09-13T17:12:04.657345+0000","last_became_peered":"2024-09-13T17:12:04.657345+0000","last_unstale":"2024-09-13T17:12:06.543561+0000","last_undegraded":"2024-09-13T17:12:06.543561+0000","last_fullsized":"2024-09-13T17:12:06.543561+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:51.127014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"60'3","reported_seq":14,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543606+0000","last_change":"2024-09-13T17:12:04.233431+0000","last_active":"2024-09-13T17:12:06.543606+0000","last_peered":"2024-09-13T17:12:06.543606+0000","last_clean":"2024-09-13T17:12:06.543606+0000","last_became_active":"2024-09-13T17:12:04.233140+0000","last_became_peered":"2024-09-13T17:12:04.233140+0000","last_unstale":"2024-09-13T17:12:06.543606+0000","last_undegraded":"2024-09-13T17:12:06.543606+0000","last_fullsized":"2024-09-13T17:12:06.543606+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:56.242264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543405+0000","last_change":"2024-09-13T17:12:04.670021+0000","last_active":"2024-09-13T17:12:06.543405+0000","last_peered":"2024-09-13T17:12:06.543405+0000","last_clean":"2024-09-13T17:12:06.543405+0000","last_became_active":"2024-09-13T17:12:04.667998+0000","last_became_peered":"2024-09-13T17:12:04.667998+0000","last_unstale":"2024-09-13T17:12:06.543405+0000","last_undegraded":"2024-09-13T17:12:06.543405+0000","last_fullsized":"2024-09-13T17:12:06.543405+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:51.635576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"60'6","reported_seq":17,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651627+0000","last_change":"2024-09-13T17:12:04.223038+0000","last_active":"2024-09-13T17:12:05.651627+0000","last_peered":"2024-09-13T17:12:05.651627+0000","last_clean":"2024-09-13T17:12:05.651627+0000","last_became_active":"2024-09-13T17:12:04.221409+0000","last_became_peered":"2024-09-13T17:12:04.221409+0000","last_unstale":"2024-09-13T17:12:05.651627+0000","last_undegraded":"2024-09-13T17:12:05.651627+0000","last_fullsized":"2024-09-13T17:12:05.651627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:56.242264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651855+0000","last_change":"2024-09-13T17:12:04.656617+0000","last_active":"2024-09-13T17:12:05.651855+0000","last_peered":"2024-09-13T17:12:05.651855+0000","last_clean":"2024-09-13T17:12:05.651855+0000","last_became_active":"2024-09-13T17:12:04.656009+0000","last_became_peered":"2024-09-13T17:12:04.656009+0000","last_unstale":"2024-09-13T17:12:05.651855+0000","last_undegraded":"2024-09-13T17:12:05.651855+0000","last_fullsized":"2024-09-13T17:12:05.651855+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:34.831511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542954+0000","last_change":"2024-09-13T17:12:04.234916+0000","last_active":"2024-09-13T17:12:06.542954+0000","last_peered":"2024-09-13T17:12:06.542954+0000","last_clean":"2024-09-13T17:12:06.542954+0000","last_became_active":"2024-09-13T17:12:04.234501+0000","last_became_peered":"2024-09-13T17:12:04.234501+0000","last_unstale":"2024-09-13T17:12:06.542954+0000","last_undegraded":"2024-09-13T17:12:06.542954+0000","last_fullsized":"2024-09-13T17:12:06.542954+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:44.975371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543195+0000","last_change":"2024-09-13T17:12:04.668071+0000","last_active":"2024-09-13T17:12:06.543195+0000","last_peered":"2024-09-13T17:12:06.543195+0000","last_clean":"2024-09-13T17:12:06.543195+0000","last_became_active":"2024-09-13T17:12:04.667782+0000","last_became_peered":"2024-09-13T17:12:04.667782+0000","last_unstale":"2024-09-13T17:12:06.543195+0000","last_undegraded":"2024-09-13T17:12:06.543195+0000","last_fullsized":"2024-09-13T17:12:06.543195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:28.661579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.759938+0000","last_change":"2024-09-13T17:12:04.232866+0000","last_active":"2024-09-13T17:12:05.759938+0000","last_peered":"2024-09-13T17:12:05.759938+0000","last_clean":"2024-09-13T17:12:05.759938+0000","last_became_active":"2024-09-13T17:12:04.232568+0000","last_became_peered":"2024-09-13T17:12:04.232568+0000","last_unstale":"2024-09-13T17:12:05.759938+0000","last_undegraded":"2024-09-13T17:12:05.759938+0000","last_fullsized":"2024-09-13T17:12:05.759938+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:26.478073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650867+0000","last_change":"2024-09-13T17:12:04.670962+0000","last_active":"2024-09-13T17:12:05.650867+0000","last_peered":"2024-09-13T17:12:05.650867+0000","last_clean":"2024-09-13T17:12:05.650867+0000","last_became_active":"2024-09-13T17:12:04.670708+0000","last_became_peered":"2024-09-13T17:12:04.670708+0000","last_unstale":"2024-09-13T17:12:05.650867+0000","last_undegraded":"2024-09-13T17:12:05.650867+0000","last_fullsized":"2024-09-13T17:12:05.650867+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:42.938301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651388+0000","last_change":"2024-09-13T17:12:04.231131+0000","last_active":"2024-09-13T17:12:05.651388+0000","last_peered":"2024-09-13T17:12:05.651388+0000","last_clean":"2024-09-13T17:12:05.651388+0000","last_became_active":"2024-09-13T17:12:04.230448+0000","last_became_peered":"2024-09-13T17:12:04.230448+0000","last_unstale":"2024-09-13T17:12:05.651388+0000","last_undegraded":"2024-09-13T17:12:05.651388+0000","last_fullsized":"2024-09-13T17:12:05.651388+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:56.242264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543135+0000","last_change":"2024-09-13T17:12:04.670498+0000","last_active":"2024-09-13T17:12:06.543135+0000","last_peered":"2024-09-13T17:12:06.543135+0000","last_clean":"2024-09-13T17:12:06.543135+0000","last_became_active":"2024-09-13T17:12:04.669863+0000","last_became_peered":"2024-09-13T17:12:04.669863+0000","last_unstale":"2024-09-13T17:12:06.543135+0000","last_undegraded":"2024-09-13T17:12:06.543135+0000","last_fullsized":"2024-09-13T17:12:06.543135+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:18.577596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"60'8","reported_seq":19,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543040+0000","last_change":"2024-09-13T17:12:04.232404+0000","last_active":"2024-09-13T17:12:06.543040+0000","last_peered":"2024-09-13T17:12:06.543040+0000","last_clean":"2024-09-13T17:12:06.543040+0000","last_became_active":"2024-09-13T17:12:04.232096+0000","last_became_peered":"2024-09-13T17:12:04.232096+0000","last_unstale":"2024-09-13T17:12:06.543040+0000","last_undegraded":"2024-09-13T17:12:06.543040+0000","last_fullsized":"2024-09-13T17:12:06.543040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:58.686370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543523+0000","last_change":"2024-09-13T17:12:04.674116+0000","last_active":"2024-09-13T17:12:06.543523+0000","last_peered":"2024-09-13T17:12:06.543523+0000","last_clean":"2024-09-13T17:12:06.543523+0000","last_became_active":"2024-09-13T17:12:04.670924+0000","last_became_peered":"2024-09-13T17:12:04.670924+0000","last_unstale":"2024-09-13T17:12:06.543523+0000","last_undegraded":"2024-09-13T17:12:06.543523+0000","last_fullsized":"2024-09-13T17:12:06.543523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:05.504518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543538+0000","last_change":"2024-09-13T17:12:04.234318+0000","last_active":"2024-09-13T17:12:06.543538+0000","last_peered":"2024-09-13T17:12:06.543538+0000","last_clean":"2024-09-13T17:12:06.543538+0000","last_became_active":"2024-09-13T17:12:04.233231+0000","last_became_peered":"2024-09-13T17:12:04.233231+0000","last_unstale":"2024-09-13T17:12:06.543538+0000","last_undegraded":"2024-09-13T17:12:06.543538+0000","last_fullsized":"2024-09-13T17:12:06.543538+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:40.745173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651948+0000","last_change":"2024-09-13T17:12:04.656426+0000","last_active":"2024-09-13T17:12:05.651948+0000","last_peered":"2024-09-13T17:12:05.651948+0000","last_clean":"2024-09-13T17:12:05.651948+0000","last_became_active":"2024-09-13T17:12:04.656142+0000","last_became_peered":"2024-09-13T17:12:04.656142+0000","last_unstale":"2024-09-13T17:12:05.651948+0000","last_undegraded":"2024-09-13T17:12:05.651948+0000","last_fullsized":"2024-09-13T17:12:05.651948+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:28.661579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544653+0000","last_change":"2024-09-13T17:12:04.231878+0000","last_active":"2024-09-13T17:12:06.544653+0000","last_peered":"2024-09-13T17:12:06.544653+0000","last_clean":"2024-09-13T17:12:06.544653+0000","last_became_active":"2024-09-13T17:12:04.231565+0000","last_became_peered":"2024-09-13T17:12:04.231565+0000","last_unstale":"2024-09-13T17:12:06.544653+0000","last_undegraded":"2024-09-13T17:12:06.544653+0000","last_fullsized":"2024-09-13T17:12:06.544653+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:37.799876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543015+0000","last_change":"2024-09-13T17:12:04.655125+0000","last_active":"2024-09-13T17:12:06.543015+0000","last_peered":"2024-09-13T17:12:06.543015+0000","last_clean":"2024-09-13T17:12:06.543015+0000","last_became_active":"2024-09-13T17:12:04.654872+0000","last_became_peered":"2024-09-13T17:12:04.654872+0000","last_unstale":"2024-09-13T17:12:06.543015+0000","last_undegraded":"2024-09-13T17:12:06.543015+0000","last_fullsized":"2024-09-13T17:12:06.543015+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:52.610711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"60'1","reported_seq":12,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651692+0000","last_change":"2024-09-13T17:12:04.226995+0000","last_active":"2024-09-13T17:12:05.651692+0000","last_peered":"2024-09-13T17:12:05.651692+0000","last_clean":"2024-09-13T17:12:05.651692+0000","last_became_active":"2024-09-13T17:12:04.226691+0000","last_became_peered":"2024-09-13T17:12:04.226691+0000","last_unstale":"2024-09-13T17:12:05.651692+0000","last_undegraded":"2024-09-13T17:12:05.651692+0000","last_fullsized":"2024-09-13T17:12:05.651692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:11.633598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543464+0000","last_change":"2024-09-13T17:12:04.671087+0000","last_active":"2024-09-13T17:12:06.543464+0000","last_peered":"2024-09-13T17:12:06.543464+0000","last_clean":"2024-09-13T17:12:06.543464+0000","last_became_active":"2024-09-13T17:12:04.670533+0000","last_became_peered":"2024-09-13T17:12:04.670533+0000","last_unstale":"2024-09-13T17:12:06.543464+0000","last_undegraded":"2024-09-13T17:12:06.543464+0000","last_fullsized":"2024-09-13T17:12:06.543464+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:53.515375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544586+0000","last_change":"2024-09-13T17:12:04.224866+0000","last_active":"2024-09-13T17:12:06.544586+0000","last_peered":"2024-09-13T17:12:06.544586+0000","last_clean":"2024-09-13T17:12:06.544586+0000","last_became_active":"2024-09-13T17:12:04.224580+0000","last_became_peered":"2024-09-13T17:12:04.224580+0000","last_unstale":"2024-09-13T17:12:06.544586+0000","last_undegraded":"2024-09-13T17:12:06.544586+0000","last_fullsized":"2024-09-13T17:12:06.544586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:53.624817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651835+0000","last_change":"2024-09-13T17:12:04.656798+0000","last_active":"2024-09-13T17:12:05.651835+0000","last_peered":"2024-09-13T17:12:05.651835+0000","last_clean":"2024-09-13T17:12:05.651835+0000","last_became_active":"2024-09-13T17:12:04.656379+0000","last_became_peered":"2024-09-13T17:12:04.656379+0000","last_unstale":"2024-09-13T17:12:05.651835+0000","last_undegraded":"2024-09-13T17:12:05.651835+0000","last_fullsized":"2024-09-13T17:12:05.651835+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:16:41.751614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543108+0000","last_change":"2024-09-13T17:12:04.229882+0000","last_active":"2024-09-13T17:12:06.543108+0000","last_peered":"2024-09-13T17:12:06.543108+0000","last_clean":"2024-09-13T17:12:06.543108+0000","last_became_active":"2024-09-13T17:12:04.229523+0000","last_became_peered":"2024-09-13T17:12:04.229523+0000","last_unstale":"2024-09-13T17:12:06.543108+0000","last_undegraded":"2024-09-13T17:12:06.543108+0000","last_fullsized":"2024-09-13T17:12:06.543108+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:30.780162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544426+0000","last_change":"2024-09-13T17:12:04.659597+0000","last_active":"2024-09-13T17:12:06.544426+0000","last_peered":"2024-09-13T17:12:06.544426+0000","last_clean":"2024-09-13T17:12:06.544426+0000","last_became_active":"2024-09-13T17:12:04.659308+0000","last_became_peered":"2024-09-13T17:12:04.659308+0000","last_unstale":"2024-09-13T17:12:06.544426+0000","last_undegraded":"2024-09-13T17:12:06.544426+0000","last_fullsized":"2024-09-13T17:12:06.544426+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:18.577596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543309+0000","last_change":"2024-09-13T17:12:04.230727+0000","last_active":"2024-09-13T17:12:06.543309+0000","last_peered":"2024-09-13T17:12:06.543309+0000","last_clean":"2024-09-13T17:12:06.543309+0000","last_became_active":"2024-09-13T17:12:04.230432+0000","last_became_peered":"2024-09-13T17:12:04.230432+0000","last_unstale":"2024-09-13T17:12:06.543309+0000","last_undegraded":"2024-09-13T17:12:06.543309+0000","last_fullsized":"2024-09-13T17:12:06.543309+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:10.327490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543596+0000","last_change":"2024-09-13T17:12:04.668385+0000","last_active":"2024-09-13T17:12:06.543596+0000","last_peered":"2024-09-13T17:12:06.543596+0000","last_clean":"2024-09-13T17:12:06.543596+0000","last_became_active":"2024-09-13T17:12:04.668094+0000","last_became_peered":"2024-09-13T17:12:04.668094+0000","last_unstale":"2024-09-13T17:12:06.543596+0000","last_undegraded":"2024-09-13T17:12:06.543596+0000","last_fullsized":"2024-09-13T17:12:06.543596+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:00:05.504518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651148+0000","last_change":"2024-09-13T17:12:04.224907+0000","last_active":"2024-09-13T17:12:05.651148+0000","last_peered":"2024-09-13T17:12:05.651148+0000","last_clean":"2024-09-13T17:12:05.651148+0000","last_became_active":"2024-09-13T17:12:04.224467+0000","last_became_peered":"2024-09-13T17:12:04.224467+0000","last_unstale":"2024-09-13T17:12:05.651148+0000","last_undegraded":"2024-09-13T17:12:05.651148+0000","last_fullsized":"2024-09-13T17:12:05.651148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:42.316065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.652107+0000","last_change":"2024-09-13T17:12:04.666696+0000","last_active":"2024-09-13T17:12:05.652107+0000","last_peered":"2024-09-13T17:12:05.652107+0000","last_clean":"2024-09-13T17:12:05.652107+0000","last_became_active":"2024-09-13T17:12:04.666409+0000","last_became_peered":"2024-09-13T17:12:04.666409+0000","last_unstale":"2024-09-13T17:12:05.652107+0000","last_undegraded":"2024-09-13T17:12:05.652107+0000","last_fullsized":"2024-09-13T17:12:05.652107+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:56:40.884648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"60'2","reported_seq":13,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544144+0000","last_change":"2024-09-13T17:12:04.229216+0000","last_active":"2024-09-13T17:12:06.544144+0000","last_peered":"2024-09-13T17:12:06.544144+0000","last_clean":"2024-09-13T17:12:06.544144+0000","last_became_active":"2024-09-13T17:12:04.228883+0000","last_became_peered":"2024-09-13T17:12:04.228883+0000","last_unstale":"2024-09-13T17:12:06.544144+0000","last_undegraded":"2024-09-13T17:12:06.544144+0000","last_fullsized":"2024-09-13T17:12:06.544144+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:12.900517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.652119+0000","last_change":"2024-09-13T17:12:04.670059+0000","last_active":"2024-09-13T17:12:05.652119+0000","last_peered":"2024-09-13T17:12:05.652119+0000","last_clean":"2024-09-13T17:12:05.652119+0000","last_became_active":"2024-09-13T17:12:04.663200+0000","last_became_peered":"2024-09-13T17:12:04.663200+0000","last_unstale":"2024-09-13T17:12:05.652119+0000","last_undegraded":"2024-09-13T17:12:05.652119+0000","last_fullsized":"2024-09-13T17:12:05.652119+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:57.182309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544701+0000","last_change":"2024-09-13T17:12:04.226579+0000","last_active":"2024-09-13T17:12:06.544701+0000","last_peered":"2024-09-13T17:12:06.544701+0000","last_clean":"2024-09-13T17:12:06.544701+0000","last_became_active":"2024-09-13T17:12:04.226303+0000","last_became_peered":"2024-09-13T17:12:04.226303+0000","last_unstale":"2024-09-13T17:12:06.544701+0000","last_undegraded":"2024-09-13T17:12:06.544701+0000","last_fullsized":"2024-09-13T17:12:06.544701+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:59.211475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543676+0000","last_change":"2024-09-13T17:12:04.656812+0000","last_active":"2024-09-13T17:12:06.543676+0000","last_peered":"2024-09-13T17:12:06.543676+0000","last_clean":"2024-09-13T17:12:06.543676+0000","last_became_active":"2024-09-13T17:12:04.656404+0000","last_became_peered":"2024-09-13T17:12:04.656404+0000","last_unstale":"2024-09-13T17:12:06.543676+0000","last_undegraded":"2024-09-13T17:12:06.543676+0000","last_fullsized":"2024-09-13T17:12:06.543676+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:02.320272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.760001+0000","last_change":"2024-09-13T17:12:04.229635+0000","last_active":"2024-09-13T17:12:05.760001+0000","last_peered":"2024-09-13T17:12:05.760001+0000","last_clean":"2024-09-13T17:12:05.760001+0000","last_became_active":"2024-09-13T17:12:04.228247+0000","last_became_peered":"2024-09-13T17:12:04.228247+0000","last_unstale":"2024-09-13T17:12:05.760001+0000","last_undegraded":"2024-09-13T17:12:05.760001+0000","last_fullsized":"2024-09-13T17:12:05.760001+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:28.745874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544661+0000","last_change":"2024-09-13T17:12:04.663019+0000","last_active":"2024-09-13T17:12:06.544661+0000","last_peered":"2024-09-13T17:12:06.544661+0000","last_clean":"2024-09-13T17:12:06.544661+0000","last_became_active":"2024-09-13T17:12:04.662529+0000","last_became_peered":"2024-09-13T17:12:04.662529+0000","last_unstale":"2024-09-13T17:12:06.544661+0000","last_undegraded":"2024-09-13T17:12:06.544661+0000","last_fullsized":"2024-09-13T17:12:06.544661+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:45.981812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.651842+0000","last_change":"2024-09-13T17:12:04.221234+0000","last_active":"2024-09-13T17:12:05.651842+0000","last_peered":"2024-09-13T17:12:05.651842+0000","last_clean":"2024-09-13T17:12:05.651842+0000","last_became_active":"2024-09-13T17:12:04.220945+0000","last_became_peered":"2024-09-13T17:12:04.220945+0000","last_unstale":"2024-09-13T17:12:05.651842+0000","last_undegraded":"2024-09-13T17:12:05.651842+0000","last_fullsized":"2024-09-13T17:12:05.651842+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:59.211475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543183+0000","last_change":"2024-09-13T17:12:04.650800+0000","last_active":"2024-09-13T17:12:06.543183+0000","last_peered":"2024-09-13T17:12:06.543183+0000","last_clean":"2024-09-13T17:12:06.543183+0000","last_became_active":"2024-09-13T17:12:04.649415+0000","last_became_peered":"2024-09-13T17:12:04.649415+0000","last_unstale":"2024-09-13T17:12:06.543183+0000","last_undegraded":"2024-09-13T17:12:06.543183+0000","last_fullsized":"2024-09-13T17:12:06.543183+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:09.932409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543235+0000","last_change":"2024-09-13T17:12:04.225020+0000","last_active":"2024-09-13T17:12:06.543235+0000","last_peered":"2024-09-13T17:12:06.543235+0000","last_clean":"2024-09-13T17:12:06.543235+0000","last_became_active":"2024-09-13T17:12:04.224769+0000","last_became_peered":"2024-09-13T17:12:04.224769+0000","last_unstale":"2024-09-13T17:12:06.543235+0000","last_undegraded":"2024-09-13T17:12:06.543235+0000","last_fullsized":"2024-09-13T17:12:06.543235+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:31.044093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543269+0000","last_change":"2024-09-13T17:12:04.667016+0000","last_active":"2024-09-13T17:12:06.543269+0000","last_peered":"2024-09-13T17:12:06.543269+0000","last_clean":"2024-09-13T17:12:06.543269+0000","last_became_active":"2024-09-13T17:12:04.666702+0000","last_became_peered":"2024-09-13T17:12:04.666702+0000","last_unstale":"2024-09-13T17:12:06.543269+0000","last_undegraded":"2024-09-13T17:12:06.543269+0000","last_fullsized":"2024-09-13T17:12:06.543269+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:45.981812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.650889+0000","last_change":"2024-09-13T17:12:04.219792+0000","last_active":"2024-09-13T17:12:05.650889+0000","last_peered":"2024-09-13T17:12:05.650889+0000","last_clean":"2024-09-13T17:12:05.650889+0000","last_became_active":"2024-09-13T17:12:04.219500+0000","last_became_peered":"2024-09-13T17:12:04.219500+0000","last_unstale":"2024-09-13T17:12:05.650889+0000","last_undegraded":"2024-09-13T17:12:05.650889+0000","last_fullsized":"2024-09-13T17:12:05.650889+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:59.576807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542622+0000","last_change":"2024-09-13T17:12:04.667042+0000","last_active":"2024-09-13T17:12:06.542622+0000","last_peered":"2024-09-13T17:12:06.542622+0000","last_clean":"2024-09-13T17:12:06.542622+0000","last_became_active":"2024-09-13T17:12:04.666783+0000","last_became_peered":"2024-09-13T17:12:04.666783+0000","last_unstale":"2024-09-13T17:12:06.542622+0000","last_undegraded":"2024-09-13T17:12:06.542622+0000","last_fullsized":"2024-09-13T17:12:06.542622+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:54.116089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.543036+0000","last_change":"2024-09-13T17:12:04.222167+0000","last_active":"2024-09-13T17:12:06.543036+0000","last_peered":"2024-09-13T17:12:06.543036+0000","last_clean":"2024-09-13T17:12:06.543036+0000","last_became_active":"2024-09-13T17:12:04.221909+0000","last_became_peered":"2024-09-13T17:12:04.221909+0000","last_unstale":"2024-09-13T17:12:06.543036+0000","last_undegraded":"2024-09-13T17:12:06.543036+0000","last_fullsized":"2024-09-13T17:12:06.543036+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:28.745874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542849+0000","last_change":"2024-09-13T17:12:04.662980+0000","last_active":"2024-09-13T17:12:06.542849+0000","last_peered":"2024-09-13T17:12:06.542849+0000","last_clean":"2024-09-13T17:12:06.542849+0000","last_became_active":"2024-09-13T17:12:04.661614+0000","last_became_peered":"2024-09-13T17:12:04.661614+0000","last_unstale":"2024-09-13T17:12:06.542849+0000","last_undegraded":"2024-09-13T17:12:06.542849+0000","last_fullsized":"2024-09-13T17:12:06.542849+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:49.361625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.542915+0000","last_change":"2024-09-13T17:12:04.220755+0000","last_active":"2024-09-13T17:12:06.542915+0000","last_peered":"2024-09-13T17:12:06.542915+0000","last_clean":"2024-09-13T17:12:06.542915+0000","last_became_active":"2024-09-13T17:12:04.220498+0000","last_became_peered":"2024-09-13T17:12:04.220498+0000","last_unstale":"2024-09-13T17:12:06.542915+0000","last_undegraded":"2024-09-13T17:12:06.542915+0000","last_fullsized":"2024-09-13T17:12:06.542915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:23.334916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:06.544078+0000","last_change":"2024-09-13T17:12:04.653743+0000","last_active":"2024-09-13T17:12:06.544078+0000","last_peered":"2024-09-13T17:12:06.544078+0000","last_clean":"2024-09-13T17:12:06.544078+0000","last_became_active":"2024-09-13T17:12:04.653480+0000","last_became_peered":"2024-09-13T17:12:04.653480+0000","last_unstale":"2024-09-13T17:12:06.544078+0000","last_undegraded":"2024-09-13T17:12:06.544078+0000","last_fullsized":"2024-09-13T17:12:06.544078+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:04.200072+0000","last_clean_scrub_stamp":"2024-09-13T17:12:04.200072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:54:19.501243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-13T17:12:05.759816+0000","last_change":"2024-09-13T17:12:04.225691+0000","last_active":"2024-09-13T17:12:05.759816+0000","last_peered":"2024-09-13T17:12:05.759816+0000","last_clean":"2024-09-13T17:12:05.759816+0000","last_became_active":"2024-09-13T17:12:04.225229+0000","last_became_peered":"2024-09-13T17:12:04.225229+0000","last_unstale":"2024-09-13T17:12:05.759816+0000","last_undegraded":"2024-09-13T17:12:05.759816+0000","last_fullsized":"2024-09-13T17:12:05.759816+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:03.193631+0000","last_clean_scrub_stamp":"2024-09-13T17:12:03.193631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:10.327490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":201,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-13T17:12:04.659476+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:12:04.659476+0000","last_peered":"2024-09-13T17:12:04.659476+0000","last_clean":"2024-09-13T17:12:04.659476+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:12:04.659476+0000","last_undegraded":"2024-09-13T17:12:04.659476+0000","last_fullsized":"2024-09-13T17:12:04.659476+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574901,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58660,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313180,"statfs":{"total":96636764160,"available":96576696320,"internally_reserved":0,"allocated":1335296,"data_stored":897839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.96199999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96399999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.76700000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.69099999999999995,"15min":0.68600000000000005},"min":{"1min":0.47699999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.86899999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.754},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.51600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90000000000000002,"5min":1.224,"15min":1.224},"last":0.82599999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.071,"15min":1.071},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97199999999999998,"5min":1.177,"15min":1.177},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.41699999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0449999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.67800000000000005,"15min":0.69999999999999996},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97099999999999997,"5min":1.143,"15min":1.143},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.58499999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.125,"5min":1.337,"15min":1.337},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.755,"15min":0.77100000000000002},"min":{"1min":0.434,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0449999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":0.78000000000000003},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.64700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574902,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40032,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331808,"statfs":{"total":96636764160,"available":96595771392,"internally_reserved":0,"allocated":733184,"data_stored":307311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68100000000000005,"15min":0.65600000000000003},"min":{"1min":0.39500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91000000000000003,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.52000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.002,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.84099999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84499999999999997,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.63200000000000001},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.622}]},{"osd":2,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64300000000000002,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.48099999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.94899999999999995,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.93899999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69799999999999995,"15min":0.70099999999999996},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.024,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89300000000000002,"5min":3.742,"15min":3.742},"last":0.79100000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66800000000000004,"15min":0.65300000000000002},"min":{"1min":0.46000000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86499999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":1.046}]},{"osd":5,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.71099999999999997,"15min":0.72699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89600000000000002,"5min":3.609,"15min":3.609},"last":0.91500000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76500000000000001,"15min":0.75700000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98599999999999999,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":1.004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.103,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":1.024}]}]},{"osd":5,"up_from":13,"seq":55834574901,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40032,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331808,"statfs":{"total":96636764160,"available":96595771392,"internally_reserved":0,"allocated":741376,"data_stored":307963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.65700000000000003,"15min":0.66700000000000004},"min":{"1min":0.46999999999999997,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81799999999999995,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.753},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69399999999999995,"15min":0.71199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.92300000000000004,"5min":1.026,"15min":1.026},"last":0.76600000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.67800000000000005,"15min":0.68700000000000006},"min":{"1min":0.50600000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.68700000000000006},"min":{"1min":0.41699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.004,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.70199999999999996,"15min":0.71499999999999997},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.97499999999999998,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.71999999999999997,"15min":0.73599999999999999},"min":{"1min":0.48099999999999998,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.91800000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.79100000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.74199999999999999,"15min":0.75700000000000001},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.877},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.752,"15min":0.76700000000000002},"min":{"1min":0.56399999999999995,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.93100000000000005,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.66700000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.622,"15min":0.624},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0169999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.81799999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.52600000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.70699999999999996,"15min":0.71499999999999997},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.749,"15min":0.748},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.505,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":1.0529999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574900,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62192,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94309648,"statfs":{"total":96636764160,"available":96573079552,"internally_reserved":0,"allocated":745472,"data_stored":308875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.438,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81200000000000006,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91200000000000003,"5min":1.341,"15min":1.341},"last":0.55800000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80500000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.73299999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.46700000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.86699999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.51600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.76100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.47399999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.96999999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":1.1020000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":1.1339999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90400000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71999999999999997,"15min":0.748},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.01,"5min":1.413,"15min":1.413},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.04,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":1.177}]}]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45792,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326048,"statfs":{"total":96636764160,"available":96589873152,"internally_reserved":0,"allocated":741376,"data_stored":307367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79400000000000004,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.5},{"interface":"front","average":{"1min":0.436,"5min":0.52700000000000002,"15min":0.57699999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92400000000000004,"5min":2.96,"15min":2.96},"last":0.35099999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95699999999999996,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.68799999999999994,"15min":0.74199999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93799999999999994,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.54000000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.39700000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80400000000000005,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.45500000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78300000000000003,"15min":0.82299999999999995},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.093,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.54200000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.123,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78600000000000003,"15min":0.79300000000000004},"min":{"1min":0.56599999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2529999999999999,"5min":1.391,"15min":1.391},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.81799999999999995,"15min":0.84799999999999998},"min":{"1min":0.52200000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.173,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.85299999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78900000000000003,"15min":0.77000000000000002},"min":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1579999999999999,"5min":1.401,"15min":1.401},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1850000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.75800000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.139,"5min":2.899,"15min":2.899},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.83499999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1220000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.72799999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574901,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40036,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331804,"statfs":{"total":96636764160,"available":96595767296,"internally_reserved":0,"allocated":737280,"data_stored":306348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57799999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.60899999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":3.0699999999999998,"15min":1.3400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58599999999999997,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":3.1120000000000001,"15min":1.353},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59399999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65600000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72799999999999998,"5min":1.121,"15min":1.121},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.753,"15min":0.75800000000000001},"min":{"1min":0.56999999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2450000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":1.379},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.60599999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.397,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":1.5229999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.048,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.76800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99299999999999999,"5min":1.569,"15min":1.569},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0940000000000001,"5min":1.413,"15min":1.413},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.024,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.073,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.56699999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":1.024}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45796,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94326044,"statfs":{"total":96636764160,"available":96589869056,"internally_reserved":0,"allocated":737280,"data_stored":307401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.4939999999999998,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.438}]},{"osd":3,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.79500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.501},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.748,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":18.318000000000001,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.46500000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.84199999999999997,"15min":0.81999999999999995},"min":{"1min":0.51700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.1630000000000003,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.78,"15min":1.78},"last":0.82299999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6020000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.84199999999999997,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.79800000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6599999999999999,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.745},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.82999999999999996},"min":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.83799999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.82899999999999996,"15min":0.83799999999999997},"min":{"1min":0.61699999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.472,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.544,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.89500000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40624,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331216,"statfs":{"total":96636764160,"available":96595165184,"internally_reserved":0,"allocated":1339392,"data_stored":898806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.58199999999999996,"15min":0.72399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.99099999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.374},{"interface":"front","average":{"1min":0.495,"5min":0.59199999999999997,"15min":0.70999999999999996},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1859999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.621,"15min":0.745},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66500000000000004,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.5},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.64200000000000002,"15min":0.76400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2190000000000001,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.60099999999999998,"15min":0.69899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.5069999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65700000000000003,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1000000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.56399999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.92600000000000005},"min":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3160000000000001,"5min":4.734,"15min":4.734},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.84899999999999998,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.264,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.78400000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.95999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.175,"5min":4.944,"15min":4.944},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.85399999999999998,"15min":0.93999999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2050000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.68400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94399999999999995,"15min":1.125},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1439999999999999,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.875,"15min":0.96899999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89800000000000002,"15min":1.0069999999999999},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88500000000000001,"15min":0.98699999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.83899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:12:09.254 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:12:09.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:12:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.450+0000 7fa0fc000640 1 Processor -- start 2024-09-13T17:12:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.450+0000 7fa0fc000640 1 -- start start 2024-09-13T17:12:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0fc000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0f4159770 0x7fa0f406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0fc000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f406f920 con 0x7fa0f406f3f0 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0fc000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f415b290 con 0x7fa0f415cd80 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0fc000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f415b570 con 0x7fa0f4159770 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0f9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0f4159770 0x7fa0f406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0f9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0f4159770 0x7fa0f406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33108/0 (socket says 172.21.15.28:33108) 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0f9600640 1 -- 172.21.15.28:0/361038995 learned_addr learned my addr 172.21.15.28:0/361038995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.451+0000 7fa0f8c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0f406f3f0 0x7fa0f4159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45094/0 (socket says 172.21.15.28:45094) 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.452+0000 7fa0f9600640 1 -- 172.21.15.28:0/361038995 >> v1:172.21.15.189:6789/0 conn(0x7fa0f415cd80 legacy=0x7fa0f415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.452+0000 7fa0f9600640 1 -- 172.21.15.28:0/361038995 >> v1:172.21.15.28:6789/0 conn(0x7fa0f406f3f0 legacy=0x7fa0f4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.452+0000 7fa0f9600640 1 -- 172.21.15.28:0/361038995 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f415b850 con 0x7fa0f4159770 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0f9600640 1 --2- 172.21.15.28:0/361038995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f4159770 0x7fa0f406fd20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009870 tx=0x7fa0e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7477261822f10aa5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0f3e00640 1 -- 172.21.15.28:0/361038995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8032440 con 0x7fa0f4159770 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0f3e00640 1 -- 172.21.15.28:0/361038995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e80325a0 con 0x7fa0f4159770 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0f3e00640 1 -- 172.21.15.28:0/361038995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8032930 con 0x7fa0f4159770 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0fc000640 1 -- 172.21.15.28:0/361038995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f4159770 msgr2=0x7fa0f406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.453+0000 7fa0fc000640 1 --2- 172.21.15.28:0/361038995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f4159770 0x7fa0f406fd20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009870 tx=0x7fa0e802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.454+0000 7fa0fc000640 1 -- 172.21.15.28:0/361038995 shutdown_connections 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.454+0000 7fa0fc000640 1 --2- 172.21.15.28:0/361038995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f4159770 0x7fa0f406fd20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.454+0000 7fa0fc000640 1 -- 172.21.15.28:0/361038995 >> 172.21.15.28:0/361038995 conn(0x7fa0f40b0270 msgr2=0x7fa0f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.454+0000 7fa0fc000640 1 -- 172.21.15.28:0/361038995 shutdown_connections 2024-09-13T17:12:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.454+0000 7fa0fc000640 1 -- 172.21.15.28:0/361038995 wait complete. 2024-09-13T17:12:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.455+0000 7fa0fc000640 1 Processor -- start 2024-09-13T17:12:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.455+0000 7fa0fc000640 1 -- start start 2024-09-13T17:12:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f406f3f0 0x7fa0f40b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0f4159770 0x7fa0f4153900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f40b1a80 con 0x7fa0f415cd80 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f4152490 con 0x7fa0f406f3f0 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fc000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f414d330 con 0x7fa0f4159770 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.456+0000 7fa0fa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33116/0 (socket says 172.21.15.28:33116) 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 -- 172.21.15.28:0/1366966985 learned_addr learned my addr 172.21.15.28:0/1366966985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0f9600640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f406f3f0 0x7fa0f40b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0f8c00640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0f4159770 0x7fa0f4153900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 -- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0f4159770 msgr2=0x7fa0f4153900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0f4159770 0x7fa0f4153900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 -- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f406f3f0 msgr2=0x7fa0f40b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f406f3f0 0x7fa0f40b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 -- 172.21.15.28:0/1366966985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e80094d0 con 0x7fa0f415cd80 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.457+0000 7fa0fa000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00c6b0 tx=0x7fa0ec00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.458+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec003b70 con 0x7fa0f415cd80 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.458+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec003cf0 con 0x7fa0f415cd80 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.458+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec003ea0 con 0x7fa0f415cd80 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.458+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f4153e40 con 0x7fa0f415cd80 2024-09-13T17:12:09.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.458+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f41521b0 con 0x7fa0f415cd80 2024-09-13T17:12:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.459+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f40b1bf0 con 0x7fa0f415cd80 2024-09-13T17:12:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.460+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0ec005310 con 0x7fa0f415cd80 2024-09-13T17:12:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.461+0000 7fa0f1600640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 0x7fa0d406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.461+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0ec088f90 con 0x7fa0f415cd80 2024-09-13T17:12:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.461+0000 7fa0f9600640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 0x7fa0d406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.462+0000 7fa0f9600640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 0x7fa0d406d0f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8034000 tx=0x7fa0e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.465+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0ec054210 con 0x7fa0f415cd80 2024-09-13T17:12:09.528 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:12:09.532 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:12:09.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.711+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa0f406fd20 con 0x7fa0f415cd80 2024-09-13T17:12:09.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.715+0000 7fa0f1600640 1 -- 172.21.15.28:0/1366966985 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7fa0ec057830 con 0x7fa0f415cd80 2024-09-13T17:12:09.717 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:12:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 msgr2=0x7fa0d406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 0x7fa0d406d0f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8034000 tx=0x7fa0e802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 msgr2=0x7fa0f4158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec00c6b0 tx=0x7fa0ec00cb70 comp rx=0 tx=0).stop 2024-09-13T17:12:09.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 shutdown_connections 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0d406ac30 0x7fa0d406d0f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0f415cd80 0x7fa0f4158320 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0f4159770 0x7fa0f4153900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 --2- 172.21.15.28:0/1366966985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0f406f3f0 0x7fa0f40b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 >> 172.21.15.28:0/1366966985 conn(0x7fa0f40b0270 msgr2=0x7fa0f40ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 shutdown_connections 2024-09-13T17:12:09.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.717+0000 7fa0fc000640 1 -- 172.21.15.28:0/1366966985 wait complete. 2024-09-13T17:12:09.735 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.888+0000 7fb266e00640 1 Processor -- start 2024-09-13T17:12:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.888+0000 7fb266e00640 1 -- start start 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb266e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26006f970 0x7fb26015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb266e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb26006c2d0 0x7fb26006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb266e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26006cc70 con 0x7fb2601592a0 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb266e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26015b0b0 con 0x7fb26006f970 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb266e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb260159d90 con 0x7fb26006c2d0 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb257400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26006f970 0x7fb26015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb257400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb26006f970 0x7fb26015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33124/0 (socket says 172.21.15.28:33124) 2024-09-13T17:12:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb257400640 1 -- 172.21.15.28:0/3007321495 learned_addr learned my addr 172.21.15.28:0/3007321495 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb264e00640 1 --2- 172.21.15.28:0/3007321495 >> v2:172.21.15.189:3300/0 conn(0x7fb26006c2d0 0x7fb26006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb257400640 1 -- 172.21.15.28:0/3007321495 >> v2:172.21.15.189:3300/0 conn(0x7fb26006c2d0 msgr2=0x7fb26006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.890+0000 7fb257400640 1 --2- 172.21.15.28:0/3007321495 >> v2:172.21.15.189:3300/0 conn(0x7fb26006c2d0 0x7fb26006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.891+0000 7fb257400640 1 -- 172.21.15.28:0/3007321495 >> v1:172.21.15.189:6789/0 conn(0x7fb2601592a0 legacy=0x7fb260159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.891+0000 7fb257400640 1 -- 172.21.15.28:0/3007321495 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2600b2040 con 0x7fb26006f970 2024-09-13T17:12:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.891+0000 7fb257400640 1 --2- 172.21.15.28:0/3007321495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb26015ab70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b1dd0 tx=0x7fb24802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ef0d224cd2c88c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb25f400640 1 -- 172.21.15.28:0/3007321495 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248032440 con 0x7fb26006f970 2024-09-13T17:12:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb25f400640 1 -- 172.21.15.28:0/3007321495 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2480325a0 con 0x7fb26006f970 2024-09-13T17:12:09.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb25f400640 1 -- 172.21.15.28:0/3007321495 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248032930 con 0x7fb26006f970 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 -- 172.21.15.28:0/3007321495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 msgr2=0x7fb26015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 --2- 172.21.15.28:0/3007321495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb26015ab70 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fb2600b1dd0 tx=0x7fb24802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 -- 172.21.15.28:0/3007321495 shutdown_connections 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 --2- 172.21.15.28:0/3007321495 >> v2:172.21.15.189:3300/0 conn(0x7fb26006c2d0 0x7fb26006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 --2- 172.21.15.28:0/3007321495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb26015ab70 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 -- 172.21.15.28:0/3007321495 >> 172.21.15.28:0/3007321495 conn(0x7fb260069e80 msgr2=0x7fb26006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 -- 172.21.15.28:0/3007321495 shutdown_connections 2024-09-13T17:12:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.892+0000 7fb266e00640 1 -- 172.21.15.28:0/3007321495 wait complete. 2024-09-13T17:12:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.894+0000 7fb266e00640 1 Processor -- start 2024-09-13T17:12:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.894+0000 7fb266e00640 1 -- start start 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 0x7fb26014a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 0x7fb26006b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb26014acc0 con 0x7fb26006f970 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb260142e70 con 0x7fb26006c2d0 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb266e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26006bdd0 con 0x7fb2601592a0 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb257400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb257400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33132/0 (socket says 172.21.15.28:33132) 2024-09-13T17:12:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.895+0000 7fb257400640 1 -- 172.21.15.28:0/2375675570 learned_addr learned my addr 172.21.15.28:0/2375675570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb25fe00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 0x7fb26014a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 -- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 msgr2=0x7fb26006b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb264e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 0x7fb26006b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 0x7fb26006b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 -- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 msgr2=0x7fb26014a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 0x7fb26014a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 -- 172.21.15.28:0/2375675570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480094d0 con 0x7fb26006f970 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb25fe00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 0x7fb26014a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb264e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 0x7fb26006b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb257400640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb24802a2b0 tx=0x7fb2480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248032440 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb248030980 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2601f3c70 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb248030d10 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.896+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2601f4230 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.897+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2600b1be0 con 0x7fb26006f970 2024-09-13T17:12:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.898+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb248039040 con 0x7fb26006f970 2024-09-13T17:12:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.899+0000 7fb25cc00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 0x7fb23c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.899+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2480a9d80 con 0x7fb26006f970 2024-09-13T17:12:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.900+0000 7fb25fe00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 0x7fb23c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.900+0000 7fb25fe00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 0x7fb23c06d0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb2601650b0 tx=0x7fb250009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:09.903+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb248074ec0 con 0x7fb26006f970 2024-09-13T17:12:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.119+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2600b3740 con 0x7fb26006f970 2024-09-13T17:12:10.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.119+0000 7fb25cc00640 1 -- 172.21.15.28:0/2375675570 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4734 (secure 0 0 0) 0x7fb248041090 con 0x7fb26006f970 2024-09-13T17:12:10.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:10.121 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6145,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/918367694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":918367694},{"type":"v1","addr":"172.21.15.28:6834","nonce":918367694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3207667105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3207667105},{"type":"v1","addr":"172.21.15.28:6835","nonce":3207667105}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3369388937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3369388937},{"type":"v1","addr":"172.21.15.189:6836","nonce":3369388937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:07.650977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6420":{"gid":6420,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/693307152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":693307152},{"type":"v1","addr":"172.21.15.189:6837","nonce":693307152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:10.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 49 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.121+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 msgr2=0x7fb23c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 0x7fb23c06d0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb2601650b0 tx=0x7fb250009040 comp rx=0 tx=0).stop 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 msgr2=0x7fb260142930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb24802a2b0 tx=0x7fb2480046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 shutdown_connections 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb23c06ac00 0x7fb23c06d0c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb2601592a0 0x7fb26006b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb26006f970 0x7fb260142930 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 --2- 172.21.15.28:0/2375675570 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb26006c2d0 0x7fb26014a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 >> 172.21.15.28:0/2375675570 conn(0x7fb260069e80 msgr2=0x7fb260156fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 shutdown_connections 2024-09-13T17:12:10.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.122+0000 7fb266e00640 1 -- 172.21.15.28:0/2375675570 wait complete. 2024-09-13T17:12:10.198 INFO:tasks.ceph.mds.d.smithi189.stderr:2024-09-13T17:12:10.196+0000 7fcb1d400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:12:10.541 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:04.639307+0000', 'modified': '2024-09-13T17:12:07.650977+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6420}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6420': {'gid': 6420, 'name': 'd', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6837/693307152', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6835', 'nonce': 693307152}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 693307152}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:12:10.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:12:10.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T17:12:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.698+0000 7f73aa000640 1 Processor -- start 2024-09-13T17:12:10.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.698+0000 7f73aa000640 1 -- start start 2024-09-13T17:12:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.699+0000 7f73aa000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f73a415a780 0x7f73a4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.699+0000 7f73aa000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a406c660 con 0x7f73a415ab60 2024-09-13T17:12:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.699+0000 7f73aa000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a406c8b0 con 0x7f73a406c250 2024-09-13T17:12:10.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.699+0000 7f73aa000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a406cb90 con 0x7f73a415a780 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f739be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73a415ab60 0x7f73a406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45106/0 (socket says 172.21.15.28:45106) 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f739be00640 1 -- 172.21.15.28:0/1894381818 learned_addr learned my addr 172.21.15.28:0/1894381818 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f73a3400640 1 --2- 172.21.15.28:0/1894381818 >> v2:172.21.15.189:3300/0 conn(0x7f73a415a780 0x7f73a4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955147686 0 0) 0x7f73a406c660 con 0x7f73a415ab60 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7388003520 con 0x7f73a415ab60 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396569237 0 0) 0x7f73a406c8b0 con 0x7f73a406c250 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.700+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a406c660 con 0x7f73a406c250 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.701+0000 7f73a3400640 1 -- 172.21.15.28:0/1894381818 >> v1:172.21.15.189:6789/0 conn(0x7f73a406c250 legacy=0x7f73a40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.701+0000 7f73a3400640 1 -- 172.21.15.28:0/1894381818 >> v1:172.21.15.28:6789/0 conn(0x7f73a415ab60 legacy=0x7f73a406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.701+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218617196 0 0) 0x7f7388003520 con 0x7f73a415ab60 2024-09-13T17:12:10.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.701+0000 7f73a3400640 1 -- 172.21.15.28:0/1894381818 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a40cc980 con 0x7f73a415a780 2024-09-13T17:12:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.702+0000 7f73a3400640 1 --2- 172.21.15.28:0/1894381818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a415a780 0x7f73a4070d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f73940097a0 tx=0x7f739402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdec7f006b5dce3e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.702+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c001670 con 0x7f73a415a780 2024-09-13T17:12:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.702+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739402ac50 con 0x7f73a415a780 2024-09-13T17:12:10.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.702+0000 7f73a2a00640 1 -- 172.21.15.28:0/1894381818 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7394032440 con 0x7f73a415a780 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 -- 172.21.15.28:0/1894381818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a415a780 msgr2=0x7f73a4070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 --2- 172.21.15.28:0/1894381818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a415a780 0x7f73a4070d80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f73940097a0 tx=0x7f739402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 -- 172.21.15.28:0/1894381818 shutdown_connections 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 --2- 172.21.15.28:0/1894381818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a415a780 0x7f73a4070d80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 -- 172.21.15.28:0/1894381818 >> 172.21.15.28:0/1894381818 conn(0x7f73a40b0270 msgr2=0x7f73a40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 -- 172.21.15.28:0/1894381818 shutdown_connections 2024-09-13T17:12:10.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.703+0000 7f73aa000640 1 -- 172.21.15.28:0/1894381818 wait complete. 2024-09-13T17:12:10.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.704+0000 7f73aa000640 1 Processor -- start 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 -- start start 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 0x7f73a4153750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a415ab60 0x7f73a415d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40ad360 con 0x7f73a415a780 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40acb50 con 0x7f73a406c250 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.705+0000 7f73aa000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a40a86f0 con 0x7f73a415ab60 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33142/0 (socket says 172.21.15.28:33142) 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 -- 172.21.15.28:0/4121586202 learned_addr learned my addr 172.21.15.28:0/4121586202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f73a3400640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 0x7f73a4153750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f73a3e00640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a415ab60 0x7f73a415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 -- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a415ab60 msgr2=0x7f73a415d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a415ab60 0x7f73a415d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 -- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 msgr2=0x7f73a4153750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 0x7f73a4153750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f739be00640 1 -- 172.21.15.28:0/4121586202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394009400 con 0x7f73a415a780 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.706+0000 7f73a3400640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 0x7f73a4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:10.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f739be00640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f738c0010e0 tx=0x7f738c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f739b400640 1 -- 172.21.15.28:0/4121586202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c005ad0 con 0x7f73a415a780 2024-09-13T17:12:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f739b400640 1 -- 172.21.15.28:0/4121586202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f738c005c30 con 0x7f73a415a780 2024-09-13T17:12:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a40acda0 con 0x7f73a415a780 2024-09-13T17:12:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f739b400640 1 -- 172.21.15.28:0/4121586202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c007070 con 0x7f73a415a780 2024-09-13T17:12:10.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.707+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a415d010 con 0x7f73a415a780 2024-09-13T17:12:10.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.709+0000 7f739b400640 1 -- 172.21.15.28:0/4121586202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f738c0071d0 con 0x7f73a415a780 2024-09-13T17:12:10.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.710+0000 7f739b400640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 0x7f738006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.710+0000 7f739b400640 1 -- 172.21.15.28:0/4121586202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f738c088900 con 0x7f73a415a780 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.710+0000 7f73a3400640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 0x7f738006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.711+0000 7f73ace03740 1 Processor -- start 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.711+0000 7f73ace03740 1 -- start start 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.711+0000 7f73a3400640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 0x7f738006d0f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7394034000 tx=0x7f739402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0515170 0x562ff0519860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff0482ab0 con 0x562ff055d690 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff0480240 con 0x562ff0515170 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ff047bb30 con 0x562ff0519da0 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33152/0 (socket says 172.21.15.28:33152) 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 -- 172.21.15.28:0/982348296 learned_addr learned my addr 172.21.15.28:0/982348296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f739be00640 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3400640 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0515170 0x562ff0519860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 -- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 msgr2=0x562ff05432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 -- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0515170 msgr2=0x562ff0519860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0515170 0x562ff0519860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.712+0000 7f73a3e00640 1 -- 172.21.15.28:0/982348296 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ff0446bd0 con 0x562ff055d690 2024-09-13T17:12:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f73a3e00640 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f739c001ca0 tx=0x7f739c002160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40016b783a68c0f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f7393e00640 1 -- 172.21.15.28:0/982348296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c003b80 con 0x562ff055d690 2024-09-13T17:12:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f7393e00640 1 -- 172.21.15.28:0/982348296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c003ce0 con 0x562ff055d690 2024-09-13T17:12:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f7393e00640 1 -- 172.21.15.28:0/982348296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c00eae0 con 0x562ff055d690 2024-09-13T17:12:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f73ace03740 1 -- 172.21.15.28:0/982348296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 msgr2=0x562ff05437f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.713+0000 7f73ace03740 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f739c001ca0 tx=0x7f739c002160 comp rx=0 tx=0).stop 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 -- 172.21.15.28:0/982348296 shutdown_connections 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05437f0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 --2- 172.21.15.28:0/982348296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0515170 0x562ff0519860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 -- 172.21.15.28:0/982348296 >> 172.21.15.28:0/982348296 conn(0x562ff0515e00 msgr2=0x562ff055e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 -- 172.21.15.28:0/982348296 shutdown_connections 2024-09-13T17:12:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 -- 172.21.15.28:0/982348296 wait complete. 2024-09-13T17:12:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.714+0000 7f73ace03740 1 -- start start 2024-09-13T17:12:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.715+0000 7f73ace03740 1 Processor -- start 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff055fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff0514070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff041d260 con 0x562ff0519da0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff041cf00 con 0x562ff055d690 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ff01e2f20 con 0x562ff0513bf0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73ace03740 10 client.0 _ll_register_callbacks cb 0x562ff04751a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33162/0 (socket says 172.21.15.28:33162) 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 -- 172.21.15.28:0/323773209 learned_addr learned my addr 172.21.15.28:0/323773209 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3e00640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff0514070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f739be00640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff055fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 -- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 msgr2=0x562ff0514070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff0514070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 -- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 msgr2=0x562ff055fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff055fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.716+0000 7f73a3400640 1 -- 172.21.15.28:0/323773209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f739c00f040 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f73a3400640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f739402a200 tx=0x7f7394004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6366aaa345006a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f7390c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739403cb10 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f739403cc70 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562ff04a5cd0 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f739403ce30 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f73ace03740 10 client.6540 fetch_fsmap learned FSMap version 49 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.717+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562ff047a1b0 con 0x562ff0519da0 2024-09-13T17:12:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562ff04a5cd0 con 0x562ff0519da0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7394044040 con 0x562ff0519da0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 10 client.6540 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 10 client.6540 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 20 client.6540 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05449c0 0x562ff0535430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 4 client.6540 mds_command: new command op to 6420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 --> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562ff016f3c0 con 0x562ff05449c0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.718+0000 7f739be00640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05449c0 0x562ff0535430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.719+0000 7f739be00640 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05449c0 0x562ff0535430 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9edd6229353c0aff server_cookie=5380ba11026dd82a in_seq=0 out_seq=0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.719+0000 7f7390c00640 10 client.6540 ms_handle_connect on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f7390c00640 1 -- 172.21.15.28:0/323773209 <== mds.0 v2:172.21.15.189:6835/693307152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562ff016f3c0 con 0x562ff05449c0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f7390c00640 10 client.6540 handle_command_reply: tid=0 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 client.6540 shutdown 2024-09-13T17:12:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 msgr2=0x562ff054c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f739402a200 tx=0x7f7394004730 comp rx=0 tx=0).stop 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 shutdown_connections 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05449c0 0x562ff0535430 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff0514070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f7390c00640 0 client.6540 ms_handle_reset on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff055fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 --2- 172.21.15.28:0/323773209 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff054c0a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 >> 172.21.15.28:0/323773209 conn(0x562ff0515170 msgr2=0x562ff055c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 shutdown_connections 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.720+0000 7f73ace03740 1 -- 172.21.15.28:0/323773209 wait complete. 2024-09-13T17:12:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.721+0000 7f73ace03740 20 client.6540 trim_cache size 0 max 16384 2024-09-13T17:12:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.736+0000 7f73ace03740 1 Processor -- start 2024-09-13T17:12:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.736+0000 7f73ace03740 1 -- start start 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0513bf0 0x562ff0547420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05423b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff055df90 con 0x562ff055d690 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff05432b0 con 0x562ff0513bf0 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ff0543800 con 0x562ff0519da0 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33168/0 (socket says 172.21.15.28:33168) 2024-09-13T17:12:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 -- 172.21.15.28:0/566610072 learned_addr learned my addr 172.21.15.28:0/566610072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f739be00640 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3400640 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0513bf0 0x562ff0547420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 -- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 msgr2=0x562ff05423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 -- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0513bf0 msgr2=0x562ff0547420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.737+0000 7f73a3e00640 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0513bf0 0x562ff0547420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f73a3e00640 1 -- 172.21.15.28:0/566610072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562ff04af170 con 0x562ff055d690 2024-09-13T17:12:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f73a3e00640 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f739c017040 tx=0x7f739c00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af631572e6a0cd56 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f7398c00640 1 -- 172.21.15.28:0/566610072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c003b80 con 0x562ff055d690 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f7398c00640 1 -- 172.21.15.28:0/566610072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c00e8c0 con 0x562ff055d690 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f7398c00640 1 -- 172.21.15.28:0/566610072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739c00ec70 con 0x562ff055d690 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.738+0000 7f73ace03740 1 -- 172.21.15.28:0/566610072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 msgr2=0x562ff05428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f739c017040 tx=0x7f739c00a000 comp rx=0 tx=0).stop 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 -- 172.21.15.28:0/566610072 shutdown_connections 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff055d690 0x562ff05428f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0519da0 0x562ff05423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 --2- 172.21.15.28:0/566610072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff0513bf0 0x562ff0547420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 -- 172.21.15.28:0/566610072 >> 172.21.15.28:0/566610072 conn(0x562ff0515170 msgr2=0x562ff0514070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 -- 172.21.15.28:0/566610072 shutdown_connections 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 -- 172.21.15.28:0/566610072 wait complete. 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 -- start start 2024-09-13T17:12:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.739+0000 7f73ace03740 1 Processor -- start 2024-09-13T17:12:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff053f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff053d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff0533be0 con 0x562ff0519da0 2024-09-13T17:12:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562ff053bfe0 con 0x562ff055d690 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562ff053ce40 con 0x562ff0513bf0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.740+0000 7f73ace03740 10 client.0 _ll_register_callbacks cb 0x7f7394035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33178/0 (socket says 172.21.15.28:33178) 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 -- 172.21.15.28:0/1028384922 learned_addr learned my addr 172.21.15.28:0/1028384922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f73a3400640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff053f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f73a3e00640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff053d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 -- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 msgr2=0x562ff053f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff053f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 -- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 msgr2=0x562ff053d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff053d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 -- 172.21.15.28:0/1028384922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562ff0446bd0 con 0x562ff0519da0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f739be00640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f738c057970 tx=0x7f738c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e65f48a359c52d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.741+0000 7f7390c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f738c05b120 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f738c084070 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f739c003ce0 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f738c076120 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 10 client.6546 fetch_fsmap learned FSMap version 49 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x562ff041c470 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739c003ce0 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f738c056ca0 con 0x562ff0519da0 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 10 client.6546 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 10 client.6546 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 20 client.6546 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05b0b10 0x562ff05b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 4 client.6546 mds_command: new command op to 6420 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:12:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.742+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 --> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562ff01806c0 con 0x562ff05b0b10 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.743+0000 7f73a3400640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05b0b10 0x562ff05b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.743+0000 7f73a3400640 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05b0b10 0x562ff05b0ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c52b085003b4b1dc server_cookie=67cfdc735fe644f8 in_seq=0 out_seq=0 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.743+0000 7f7390c00640 10 client.6546 ms_handle_connect on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f7390c00640 1 -- 172.21.15.28:0/1028384922 <== mds.0 v2:172.21.15.189:6835/693307152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x562ff01806c0 con 0x562ff05b0b10 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f7390c00640 10 client.6546 handle_command_reply: tid=0 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 client.6546 shutdown 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 msgr2=0x562ff0561960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f738c057970 tx=0x7f738c000ed0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 shutdown_connections 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x562ff05b0b10 0x562ff05b0ef0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562ff055d690 0x562ff053d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562ff0519da0 0x562ff0561960 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f7390c00640 0 client.6546 ms_handle_reset on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 --2- 172.21.15.28:0/1028384922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562ff0513bf0 0x562ff053f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 >> 172.21.15.28:0/1028384922 conn(0x562ff0515170 msgr2=0x562ff0542a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 shutdown_connections 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 1 -- 172.21.15.28:0/1028384922 wait complete. 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.744+0000 7f73ace03740 20 client.6546 trim_cache size 0 max 16384 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6420, 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:12:10.750 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1031432319999999, 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 49, 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 61, 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 61, 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.202848744000001 2024-09-13T17:12:10.751 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 msgr2=0x7f738006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 0x7f738006d0f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f7394034000 tx=0x7f739402e000 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 msgr2=0x7f73a415c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f738c0010e0 tx=0x7f738c00cb90 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 shutdown_connections 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac30 0x7f738006d0f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a415ab60 0x7f73a415d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a415a780 0x7f73a415c4e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 --2- 172.21.15.28:0/4121586202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a406c250 0x7f73a4153750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 >> 172.21.15.28:0/4121586202 conn(0x7f73a40b0270 msgr2=0x7f73a40aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 shutdown_connections 2024-09-13T17:12:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.745+0000 7f73aa000640 1 -- 172.21.15.28:0/4121586202 wait complete. 2024-09-13T17:12:10.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:12:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.915+0000 7ff40a800640 1 Processor -- start 2024-09-13T17:12:10.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.916+0000 7ff40a800640 1 -- start start 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff40a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff4040b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff40a800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4040b29f0 0x7ff4040b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff40a800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4040c3850 con 0x7ff404159820 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff40a800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff40415a930 con 0x7ff4040b29f0 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff40a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff404159d20 con 0x7ff404159440 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff408a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff404159820 0x7ff4040b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45114/0 (socket says 172.21.15.28:45114) 2024-09-13T17:12:10.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff408a00640 1 -- 172.21.15.28:0/3099257973 learned_addr learned my addr 172.21.15.28:0/3099257973 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff409400640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff4040b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.917+0000 7ff409e00640 1 --2- 172.21.15.28:0/3099257973 >> v2:172.21.15.189:3300/0 conn(0x7ff4040b29f0 0x7ff4040b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff403e00640 1 -- 172.21.15.28:0/3099257973 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2158867214 0 0) 0x7ff4040c3850 con 0x7ff404159820 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff403e00640 1 -- 172.21.15.28:0/3099257973 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3ec003520 con 0x7ff404159820 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff409e00640 1 -- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 msgr2=0x7ff4040b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:10.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff409e00640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff4040b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff409e00640 1 -- 172.21.15.28:0/3099257973 >> v1:172.21.15.28:6789/0 conn(0x7ff404159820 legacy=0x7ff4040b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.918+0000 7ff409e00640 1 -- 172.21.15.28:0/3099257973 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40415a000 con 0x7ff4040b29f0 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.919+0000 7ff409e00640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4040b29f0 0x7ff4040b0980 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009d90 tx=0x7ff3fc02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e673a88c4cb21afd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.919+0000 7ff403e00640 1 -- 172.21.15.28:0/3099257973 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc007740 con 0x7ff4040b29f0 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.919+0000 7ff403e00640 1 -- 172.21.15.28:0/3099257973 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3fc02d050 con 0x7ff4040b29f0 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.919+0000 7ff403e00640 1 -- 172.21.15.28:0/3099257973 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3fc007ac0 con 0x7ff4040b29f0 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 -- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4040b29f0 msgr2=0x7ff4040b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4040b29f0 0x7ff4040b0980 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff3fc009d90 tx=0x7ff3fc02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 -- 172.21.15.28:0/3099257973 shutdown_connections 2024-09-13T17:12:10.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4040b29f0 0x7ff4040b0980 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 --2- 172.21.15.28:0/3099257973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff4040b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 -- 172.21.15.28:0/3099257973 >> 172.21.15.28:0/3099257973 conn(0x7ff404072e40 msgr2=0x7ff404073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 -- 172.21.15.28:0/3099257973 shutdown_connections 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.920+0000 7ff40a800640 1 -- 172.21.15.28:0/3099257973 wait complete. 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.922+0000 7ff40a800640 1 Processor -- start 2024-09-13T17:12:10.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.922+0000 7ff40a800640 1 -- start start 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff40414c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404159820 0x7ff40413fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40414f020 con 0x7ff4040b29f0 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40414b150 con 0x7ff404159820 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff40a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40414e3a0 con 0x7ff404159440 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33186/0 (socket says 172.21.15.28:33186) 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 -- 172.21.15.28:0/2500756389 learned_addr learned my addr 172.21.15.28:0/2500756389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409e00640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404159820 0x7ff40413fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff408a00640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff40414c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 -- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 msgr2=0x7ff40414c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff40414c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 -- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404159820 msgr2=0x7ff40413fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404159820 0x7ff40413fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.923+0000 7ff409400640 1 -- 172.21.15.28:0/2500756389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3fc009a70 con 0x7ff4040b29f0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff409400640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800e9b0 tx=0x7ff3f800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f8009670 con 0x7ff4040b29f0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3f800c9a0 con 0x7ff4040b29f0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff40414e680 con 0x7ff4040b29f0 2024-09-13T17:12:10.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f800cd50 con 0x7ff4040b29f0 2024-09-13T17:12:10.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.924+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff40414ec40 con 0x7ff4040b29f0 2024-09-13T17:12:10.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.925+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4040b22e0 con 0x7ff4040b29f0 2024-09-13T17:12:10.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.926+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff3f8013470 con 0x7ff4040b29f0 2024-09-13T17:12:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.927+0000 7ff401600640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 0x7ff3e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.927+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3f8089700 con 0x7ff4040b29f0 2024-09-13T17:12:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.927+0000 7ff408a00640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 0x7ff3e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:10.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.928+0000 7ff408a00640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 0x7ff3e406d0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff4040cd570 tx=0x7ff3f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:10.931+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff3f80547c0 con 0x7ff4040b29f0 2024-09-13T17:12:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.146+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff4040cd1e0 con 0x7ff4040b29f0 2024-09-13T17:12:11.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.146+0000 7ff401600640 1 -- 172.21.15.28:0/2500756389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff3f8057de0 con 0x7ff4040b29f0 2024-09-13T17:12:11.148 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.148+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 msgr2=0x7ff3e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.148+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 0x7ff3e406d0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff4040cd570 tx=0x7ff3f0009040 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.148+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 msgr2=0x7ff40414bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.148+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7ff3f800e9b0 tx=0x7ff3f800ee70 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 shutdown_connections 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3e406ac00 0x7ff3e406d0c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff404159820 0x7ff40413fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff404159440 0x7ff40414c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 --2- 172.21.15.28:0/2500756389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4040b29f0 0x7ff40414bed0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 >> 172.21.15.28:0/2500756389 conn(0x7ff404072e40 msgr2=0x7ff4040b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 shutdown_connections 2024-09-13T17:12:11.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:11.149+0000 7ff40a800640 1 -- 172.21.15.28:0/2500756389 wait complete. 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:12:11.167 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:12:11.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.168 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.204 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42450sec preferred_lft 42450sec 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:12:11.205 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:12:11.206 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:12:11.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.207 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:12:11.299 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:12:11.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.302 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:12:11.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.339 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:12:11.339 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:12:11.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:12:11.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:12:11.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:12:11.340 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:12:11.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.484 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:12:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:12:11.520 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:12:11.520 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:12:11.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:12:11.618 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:12:11.618 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:12:11.653 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:12:11.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:12:11.653 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:12:11.727 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:12:11.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:12:11.856 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:12:11.856 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:12:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:12:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:12:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:12:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:12:11.884 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:12:11.886 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:12:11.886 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:12:11.886 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:12:11.886 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:12:11.886 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:12:11.889 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:12:11.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:12:11.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:11.946 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:12:11.993 INFO:teuthology.orchestra.run.smithi028.stdout:client6555 2024-09-13T17:12:11.993 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:11.993 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:12.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_size 20 2024-09-13T17:12:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.225+0000 7f300e600640 1 Processor -- start 2024-09-13T17:12:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.225+0000 7f300e600640 1 -- start start 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30080cc2e0 0x7f30081621b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3008162000 con 0x7f3008146ea0 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30081626f0 con 0x7f3008161b90 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3008147280 con 0x7f30080cc2e0 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3008146ea0 0x7f300815ec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45118/0 (socket says 172.21.15.28:45118) 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300d200640 1 -- 172.21.15.28:0/3447327211 learned_addr learned my addr 172.21.15.28:0/3447327211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.227+0000 7f300dc00640 1 --2- 172.21.15.28:0/3447327211 >> v2:172.21.15.28:3300/0 conn(0x7f30080cc2e0 0x7f30081621b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779525139 0 0) 0x7f3008162000 con 0x7f3008146ea0 2024-09-13T17:12:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff0003520 con 0x7f3008146ea0 2024-09-13T17:12:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f300dc00640 1 -- 172.21.15.28:0/3447327211 >> v1:172.21.15.189:6789/0 conn(0x7f3008161b90 legacy=0x7f30080cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131702372 0 0) 0x7f30081626f0 con 0x7f3008161b90 2024-09-13T17:12:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f300dc00640 1 -- 172.21.15.28:0/3447327211 >> v1:172.21.15.28:6789/0 conn(0x7f3008146ea0 legacy=0x7f300815ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.228+0000 7f300dc00640 1 -- 172.21.15.28:0/3447327211 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3008162920 con 0x7f30080cc2e0 2024-09-13T17:12:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.229+0000 7f300dc00640 1 --2- 172.21.15.28:0/3447327211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f30081621b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c43fcded633fb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.229+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000007740 con 0x7f30080cc2e0 2024-09-13T17:12:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.229+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300002d050 con 0x7f30080cc2e0 2024-09-13T17:12:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.229+0000 7f3007400640 1 -- 172.21.15.28:0/3447327211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000007aa0 con 0x7f30080cc2e0 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 -- 172.21.15.28:0/3447327211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 msgr2=0x7f30081621b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 --2- 172.21.15.28:0/3447327211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f30081621b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3000009d90 tx=0x7f300002a660 comp rx=0 tx=0).stop 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 -- 172.21.15.28:0/3447327211 shutdown_connections 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 --2- 172.21.15.28:0/3447327211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f30081621b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 -- 172.21.15.28:0/3447327211 >> 172.21.15.28:0/3447327211 conn(0x7f3008069e80 msgr2=0x7f300806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 -- 172.21.15.28:0/3447327211 shutdown_connections 2024-09-13T17:12:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.230+0000 7f300e600640 1 -- 172.21.15.28:0/3447327211 wait complete. 2024-09-13T17:12:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.232+0000 7f300e600640 1 Processor -- start 2024-09-13T17:12:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.232+0000 7f300e600640 1 -- start start 2024-09-13T17:12:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008146ea0 0x7f30081f76e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 0x7f30081f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008135ab0 con 0x7f30080cc2e0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008132de0 con 0x7f3008146ea0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30081330c0 con 0x7f3008161b90 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33206/0 (socket says 172.21.15.28:33206) 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.233+0000 7f300d200640 1 -- 172.21.15.28:0/1482027948 learned_addr learned my addr 172.21.15.28:0/1482027948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300dc00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 0x7f30081f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f3007e00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008146ea0 0x7f30081f76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 -- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 msgr2=0x7f30081f91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 0x7f30081f91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 -- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008146ea0 msgr2=0x7f30081f76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008146ea0 0x7f30081f76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 -- 172.21.15.28:0/1482027948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009a70 con 0x7f30080cc2e0 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300dc00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 0x7f30081f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300d200640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c430 tx=0x7f2ffc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc005ce0 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ffc005e40 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30081333a0 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.234+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ffc0072f0 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.235+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3008133960 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.236+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3008067580 con 0x7f30080cc2e0 2024-09-13T17:12:12.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.237+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ffc007450 con 0x7f30080cc2e0 2024-09-13T17:12:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.237+0000 7f3004c00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 0x7f2fe806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.238+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ffc088af0 con 0x7f30080cc2e0 2024-09-13T17:12:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.238+0000 7f3007e00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 0x7f2fe806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.239+0000 7f3007e00640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 0x7f2fe806d0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30080b2790 tx=0x7f2ff4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.242+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2ffc053c30 con 0x7f30080cc2e0 2024-09-13T17:12:12.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.421+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) v1 -- 0x7f3008163f80 con 0x7f30080cc2e0 2024-09-13T17:12:12.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.425+0000 7f3004c00640 1 -- 172.21.15.28:0/1482027948 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v15)=0 v15) v1 ==== 117+0+0 (secure 0 0 0) 0x7f2ffc017020 con 0x7f30080cc2e0 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 msgr2=0x7f2fe806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 0x7f2fe806d0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f30080b2790 tx=0x7f2ff4009040 comp rx=0 tx=0).stop 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 msgr2=0x7f3008135570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f2ffc00c430 tx=0x7f2ffc00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 shutdown_connections 2024-09-13T17:12:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe806ac00 0x7f2fe806d0c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3008161b90 0x7f30081f91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3008146ea0 0x7f30081f76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 --2- 172.21.15.28:0/1482027948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080cc2e0 0x7f3008135570 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.427+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 >> 172.21.15.28:0/1482027948 conn(0x7f3008069e80 msgr2=0x7f30080cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.428+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 shutdown_connections 2024-09-13T17:12:12.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.428+0000 7f300e600640 1 -- 172.21.15.28:0/1482027948 wait complete. 2024-09-13T17:12:12.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-09-13T17:12:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.601+0000 7f67f5200640 1 Processor -- start 2024-09-13T17:12:12.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.601+0000 7f67f5200640 1 -- start start 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.602+0000 7f67f5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67f0159770 0x7f67f006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.602+0000 7f67f5200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67f5200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f006f920 con 0x7f67f00add00 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67f5200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f015b290 con 0x7f67f0159770 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67f5200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f015b570 con 0x7f67f006f3f0 2024-09-13T17:12:12.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67f0159770 0x7f67f006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67f0159770 0x7f67f006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33208/0 (socket says 172.21.15.28:33208) 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 -- 172.21.15.28:0/3237688488 learned_addr learned my addr 172.21.15.28:0/3237688488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67e5400640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 -- 172.21.15.28:0/3237688488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 msgr2=0x7f67f0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f0070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 -- 172.21.15.28:0/3237688488 >> v1:172.21.15.189:6789/0 conn(0x7f67f00add00 legacy=0x7f67f0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.603+0000 7f67ede00640 1 -- 172.21.15.28:0/3237688488 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f015b850 con 0x7f67f0159770 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.604+0000 7f67ede00640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f0159770 0x7f67f006fd20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009870 tx=0x7f67e002c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a2b6b5448b7f10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67ed400640 1 -- 172.21.15.28:0/3237688488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031710 con 0x7f67f0159770 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67ed400640 1 -- 172.21.15.28:0/3237688488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e002f050 con 0x7f67f0159770 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67ed400640 1 -- 172.21.15.28:0/3237688488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031a90 con 0x7f67f0159770 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 -- 172.21.15.28:0/3237688488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f0159770 msgr2=0x7f67f006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f0159770 0x7f67f006fd20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009870 tx=0x7f67e002c3d0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 -- 172.21.15.28:0/3237688488 shutdown_connections 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f0070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 --2- 172.21.15.28:0/3237688488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f0159770 0x7f67f006fd20 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 -- 172.21.15.28:0/3237688488 >> 172.21.15.28:0/3237688488 conn(0x7f67f00b0270 msgr2=0x7f67f00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 -- 172.21.15.28:0/3237688488 shutdown_connections 2024-09-13T17:12:12.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.605+0000 7f67f5200640 1 -- 172.21.15.28:0/3237688488 wait complete. 2024-09-13T17:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.607+0000 7f67f5200640 1 Processor -- start 2024-09-13T17:12:12.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.607+0000 7f67f5200640 1 -- start start 2024-09-13T17:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f00ca730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 0x7f67f00ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f00c68f0 con 0x7f67f00add00 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f0155fe0 con 0x7f67f0159770 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67f5200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f01498d0 con 0x7f67f006f3f0 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33214/0 (socket says 172.21.15.28:33214) 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67e5400640 1 -- 172.21.15.28:0/4027764505 learned_addr learned my addr 172.21.15.28:0/4027764505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.608+0000 7f67ede00640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f00ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67ee800640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 0x7f67f00ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 -- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 msgr2=0x7f67f00ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f00ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 -- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 msgr2=0x7f67f00ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 0x7f67f00ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 -- 172.21.15.28:0/4027764505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e00094d0 con 0x7f67f00add00 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67ee800640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 0x7f67f00ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:12.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e5400640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f67d800d620 tx=0x7f67d800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800b620 con 0x7f67f00add00 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67d800b780 con 0x7f67f00add00 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d800bb30 con 0x7f67f00add00 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.609+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f014c120 con 0x7f67f00add00 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.610+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f014c670 con 0x7f67f00add00 2024-09-13T17:12:12.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.611+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f006fd20 con 0x7f67f00add00 2024-09-13T17:12:12.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.612+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f67d800bc90 con 0x7f67f00add00 2024-09-13T17:12:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.612+0000 7f67e6a00640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.613+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67d8089800 con 0x7f67f00add00 2024-09-13T17:12:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.613+0000 7f67ede00640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.614+0000 7f67ede00640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f67e0035000 tx=0x7f67e0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.617+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f67d8054940 con 0x7f67f00add00 2024-09-13T17:12:12.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.799+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f67f00cda70 con 0x7f67f00add00 2024-09-13T17:12:12.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.803+0000 7f67e6a00640 1 -- 172.21.15.28:0/4027764505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v16)=0 v16) v1 ==== 116+0+0 (secure 0 0 0) 0x7f67d8057f60 con 0x7f67f00add00 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 msgr2=0x7f67cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f67e0035000 tx=0x7f67e0008030 comp rx=0 tx=0).stop 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 msgr2=0x7f67f014bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f67d800d620 tx=0x7f67d800dae0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 shutdown_connections 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67cc06ac00 0x7f67cc06d0c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67f0159770 0x7f67f00ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67f00add00 0x7f67f014bbe0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 --2- 172.21.15.28:0/4027764505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67f006f3f0 0x7f67f00ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 >> 172.21.15.28:0/4027764505 conn(0x7f67f00b0270 msgr2=0x7f67f00af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 shutdown_connections 2024-09-13T17:12:12.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.806+0000 7f67f5200640 1 -- 172.21.15.28:0/4027764505 wait complete. 2024-09-13T17:12:12.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_bits 1 2024-09-13T17:12:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.981+0000 7fa1b9200640 1 Processor -- start 2024-09-13T17:12:12.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.981+0000 7fa1b9200640 1 -- start start 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1b415ab60 0x7fa1b406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b406c8b0 con 0x7fa1b415a780 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b9200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b406cb90 con 0x7fa1b4071490 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b40cc980 con 0x7fa1b415ab60 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1b415a780 0x7fa1b4070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45120/0 (socket says 172.21.15.28:45120) 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1b1e00640 1 -- 172.21.15.28:0/2254659987 learned_addr learned my addr 172.21.15.28:0/2254659987 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.982+0000 7fa1abe00640 1 --2- 172.21.15.28:0/2254659987 >> v2:172.21.15.28:3300/0 conn(0x7fa1b415ab60 0x7fa1b406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.983+0000 7fa1abe00640 1 -- 172.21.15.28:0/2254659987 >> v1:172.21.15.189:6789/0 conn(0x7fa1b4071490 legacy=0x7fa1b406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.983+0000 7fa1abe00640 1 -- 172.21.15.28:0/2254659987 >> v1:172.21.15.28:6789/0 conn(0x7fa1b415a780 legacy=0x7fa1b4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.983+0000 7fa1abe00640 1 -- 172.21.15.28:0/2254659987 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b40ccc60 con 0x7fa1b415ab60 2024-09-13T17:12:12.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.984+0000 7fa1abe00640 1 --2- 172.21.15.28:0/2254659987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b415ab60 0x7fa1b406bb40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0097a0 tx=0x7fa19c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb25c8a68e9c947f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.984+0000 7fa1b1400640 1 -- 172.21.15.28:0/2254659987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c032440 con 0x7fa1b415ab60 2024-09-13T17:12:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.984+0000 7fa1b1400640 1 -- 172.21.15.28:0/2254659987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c0325a0 con 0x7fa1b415ab60 2024-09-13T17:12:12.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.984+0000 7fa1b1400640 1 -- 172.21.15.28:0/2254659987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c032930 con 0x7fa1b415ab60 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 -- 172.21.15.28:0/2254659987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b415ab60 msgr2=0x7fa1b406bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 --2- 172.21.15.28:0/2254659987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b415ab60 0x7fa1b406bb40 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa19c0097a0 tx=0x7fa19c02a230 comp rx=0 tx=0).stop 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 -- 172.21.15.28:0/2254659987 shutdown_connections 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 --2- 172.21.15.28:0/2254659987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b415ab60 0x7fa1b406bb40 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 -- 172.21.15.28:0/2254659987 >> 172.21.15.28:0/2254659987 conn(0x7fa1b40b0270 msgr2=0x7fa1b40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 -- 172.21.15.28:0/2254659987 shutdown_connections 2024-09-13T17:12:12.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.985+0000 7fa1b9200640 1 -- 172.21.15.28:0/2254659987 wait complete. 2024-09-13T17:12:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.987+0000 7fa1b9200640 1 Processor -- start 2024-09-13T17:12:12.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.987+0000 7fa1b9200640 1 -- start start 2024-09-13T17:12:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.987+0000 7fa1b9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 0x7fa1b41548f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 0x7fa1b4147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4155640 con 0x7fa1b4071490 2024-09-13T17:12:12.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4154e30 con 0x7fa1b415ab60 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b4155080 con 0x7fa1b415a780 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33236/0 (socket says 172.21.15.28:33236) 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 -- 172.21.15.28:0/1252637084 learned_addr learned my addr 172.21.15.28:0/1252637084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b2800640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 0x7fa1b4147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1abe00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 0x7fa1b41548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 -- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 msgr2=0x7fa1b41548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 0x7fa1b41548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 -- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 msgr2=0x7fa1b4147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 0x7fa1b4147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b1e00640 1 -- 172.21.15.28:0/1252637084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19c009400 con 0x7fa1b4071490 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1abe00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 0x7fa1b41548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.988+0000 7fa1b2800640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 0x7fa1b4147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1b1e00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c4a0 tx=0x7fa1a400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4005ce0 con 0x7fa1b4071490 2024-09-13T17:12:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a4005e40 con 0x7fa1b4071490 2024-09-13T17:12:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4007270 con 0x7fa1b4071490 2024-09-13T17:12:12.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b4155360 con 0x7fa1b4071490 2024-09-13T17:12:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.989+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b414c2c0 con 0x7fa1b4071490 2024-09-13T17:12:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.990+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b414be40 con 0x7fa1b4071490 2024-09-13T17:12:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.991+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa1a4007490 con 0x7fa1b4071490 2024-09-13T17:12:12.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.992+0000 7fa1aa000640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 0x7fa19006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.992+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1a4088840 con 0x7fa1b4071490 2024-09-13T17:12:12.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.992+0000 7fa1abe00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 0x7fa19006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:12.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.993+0000 7fa1abe00640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 0x7fa19006d0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa19c034000 tx=0x7fa19c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:12.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:12.996+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa1b414be40 con 0x7fa1b4071490 2024-09-13T17:12:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.172+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) v1 -- 0x7fa1b414be40 con 0x7fa1b4071490 2024-09-13T17:12:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.177+0000 7fa1aa000640 1 -- 172.21.15.28:0/1252637084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v17)=0 v17) v1 ==== 116+0+0 (secure 0 0 0) 0x7fa1b414be40 con 0x7fa1b4071490 2024-09-13T17:12:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 msgr2=0x7fa19006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 0x7fa19006d0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa19c034000 tx=0x7fa19c02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 msgr2=0x7fa1b41510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa1a400c4a0 tx=0x7fa1a400c960 comp rx=0 tx=0).stop 2024-09-13T17:12:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 shutdown_connections 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa19006ac00 0x7fa19006d0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1b415ab60 0x7fa1b4147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1b415a780 0x7fa1b41548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 --2- 172.21.15.28:0/1252637084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1b4071490 0x7fa1b41510b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 >> 172.21.15.28:0/1252637084 conn(0x7fa1b40b0270 msgr2=0x7fa1b406bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.179+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 shutdown_connections 2024-09-13T17:12:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.180+0000 7fa1b9200640 1 -- 172.21.15.28:0/1252637084 wait complete. 2024-09-13T17:12:13.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:13.199 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> print(os.stat("/home/ubuntu/cephtest/mnt.0/testdir").st_ino) 2024-09-13T17:12:13.242 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:12:13.381 INFO:teuthology.orchestra.run.smithi028.stdout:1099511627776 2024-09-13T17:12:13.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.763+0000 7f1210a00640 1 Processor -- start 2024-09-13T17:12:13.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.763+0000 7f1210a00640 1 -- start start 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.764+0000 7f1210a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f120c06f970 0x7f120c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1210a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f120c06c2d0 0x7f120c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1210a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c06cc70 con 0x7f120c1592a0 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1210a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c15b0b0 con 0x7f120c06f970 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1210a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f120c159d90 con 0x7f120c06c2d0 2024-09-13T17:12:13.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f120c06f970 0x7f120c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f120c06f970 0x7f120c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33248/0 (socket says 172.21.15.28:33248) 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 -- 172.21.15.28:0/2308394197 learned_addr learned my addr 172.21.15.28:0/2308394197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f120a800640 1 --2- 172.21.15.28:0/2308394197 >> v2:172.21.15.189:3300/0 conn(0x7f120c06c2d0 0x7f120c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 -- 172.21.15.28:0/2308394197 >> v2:172.21.15.189:3300/0 conn(0x7f120c06c2d0 msgr2=0x7f120c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 --2- 172.21.15.28:0/2308394197 >> v2:172.21.15.189:3300/0 conn(0x7f120c06c2d0 0x7f120c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 -- 172.21.15.28:0/2308394197 >> v1:172.21.15.28:6789/0 conn(0x7f120c1592a0 legacy=0x7f120c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.765+0000 7f1203e00640 1 -- 172.21.15.28:0/2308394197 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c0b2040 con 0x7f120c06f970 2024-09-13T17:12:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.766+0000 7f1203e00640 1 --2- 172.21.15.28:0/2308394197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c15ab70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f120c0b1dd0 tx=0x7f11f402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ffdea5d0e54ae88 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1209400640 1 -- 172.21.15.28:0/2308394197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4032440 con 0x7f120c06f970 2024-09-13T17:12:13.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1209400640 1 -- 172.21.15.28:0/2308394197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f40325a0 con 0x7f120c06f970 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1209400640 1 -- 172.21.15.28:0/2308394197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4032930 con 0x7f120c06f970 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 -- 172.21.15.28:0/2308394197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 msgr2=0x7f120c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 --2- 172.21.15.28:0/2308394197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c15ab70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f120c0b1dd0 tx=0x7f11f402a270 comp rx=0 tx=0).stop 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 -- 172.21.15.28:0/2308394197 shutdown_connections 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 --2- 172.21.15.28:0/2308394197 >> v2:172.21.15.189:3300/0 conn(0x7f120c06c2d0 0x7f120c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 --2- 172.21.15.28:0/2308394197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c15ab70 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 -- 172.21.15.28:0/2308394197 >> 172.21.15.28:0/2308394197 conn(0x7f120c069e80 msgr2=0x7f120c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 -- 172.21.15.28:0/2308394197 shutdown_connections 2024-09-13T17:12:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.767+0000 7f1210a00640 1 -- 172.21.15.28:0/2308394197 wait complete. 2024-09-13T17:12:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.769+0000 7f1210a00640 1 Processor -- start 2024-09-13T17:12:13.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.769+0000 7f1210a00640 1 -- start start 2024-09-13T17:12:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f120c06c2d0 0x7f120c14cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f120c1592a0 0x7f120c0aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c14d820 con 0x7f120c06f970 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120c0af040 con 0x7f120c1592a0 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1210a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f120c14dc50 con 0x7f120c06c2d0 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33262/0 (socket says 172.21.15.28:33262) 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 -- 172.21.15.28:0/3148467355 learned_addr learned my addr 172.21.15.28:0/3148467355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 -- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f120c06c2d0 msgr2=0x7f120c14cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f120c06c2d0 0x7f120c14cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 -- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f120c1592a0 msgr2=0x7f120c0aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.770+0000 7f1203e00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f120c1592a0 0x7f120c0aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1203e00640 1 -- 172.21.15.28:0/3148467355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f40094d0 con 0x7f120c06f970 2024-09-13T17:12:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1203e00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f4032440 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4030ba0 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f403d440 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c14e7d0 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.771+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c14e180 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.772+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c0b3740 con 0x7f120c06f970 2024-09-13T17:12:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.773+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f11f4039040 con 0x7f120c06f970 2024-09-13T17:12:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.774+0000 7f1202000640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 0x7f11e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.774+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f11f40a9e30 con 0x7f120c06f970 2024-09-13T17:12:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.775+0000 7f1209e00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 0x7f11e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.775+0000 7f1209e00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 0x7f11e806d0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f11fc000f60 tx=0x7f11fc0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.778+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f11f40750a0 con 0x7f120c06f970 2024-09-13T17:12:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.991+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f120c14e960 con 0x7f120c06f970 2024-09-13T17:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.992+0000 7f1202000640 1 -- 172.21.15.28:0/3148467355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4734 (secure 0 0 0) 0x7f11f4037090 con 0x7f120c06f970 2024-09-13T17:12:13.994 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:13.994 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":49,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6145,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/918367694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":918367694},{"type":"v1","addr":"172.21.15.28:6834","nonce":918367694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6414,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3207667105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3207667105},{"type":"v1","addr":"172.21.15.28:6835","nonce":3207667105}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":6417,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3369388937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3369388937},{"type":"v1","addr":"172.21.15.189:6836","nonce":3369388937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:07.650977+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6420":{"gid":6420,"name":"d","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6837/693307152","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":693307152},{"type":"v1","addr":"172.21.15.189:6837","nonce":693307152}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:13.994 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 49 2024-09-13T17:12:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 msgr2=0x7f11e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 0x7f11e806d0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f11fc000f60 tx=0x7f11fc0079b0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 msgr2=0x7f120c140110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f11f4000c00 tx=0x7f11f40046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 shutdown_connections 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11e806ac00 0x7f11e806d0c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f120c1592a0 0x7f120c0aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f120c06f970 0x7f120c140110 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 --2- 172.21.15.28:0/3148467355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f120c06c2d0 0x7f120c14cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 >> 172.21.15.28:0/3148467355 conn(0x7f120c069e80 msgr2=0x7f120c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 shutdown_connections 2024-09-13T17:12:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:13.994+0000 7f1210a00640 1 -- 172.21.15.28:0/3148467355 wait complete. 2024-09-13T17:12:14.012 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json dump inode 1099511627776 2024-09-13T17:12:14.192 INFO:teuthology.orchestra.run.smithi189.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:12:13.235853+0000","btime":"2024-09-13T17:12:13.235853+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:12:13.235853+0000","atime":"2024-09-13T17:12:13.235853+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","type":"iauth","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","type":"ifile","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","type":"ixattr","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":6555,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":6555,"want_loner":6555,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:12:14.193 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:12:13.235853+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "iauth", "xlock_by": {} }, "backtrace_version": 2, "btime": "2024-09-13T17:12:13.235853+0000", "change_attr": 0, "client_caps": [ { "client_id": 6555, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-09-13T17:12:13.235853+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "ifile", "xlock_by": {} }, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 6555, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:12:13.235853+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": 6555, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "ixattr", "xlock_by": {} }, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:12:14.193 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:14.194 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:14.230 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:12:14.230 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:12:14.230 DEBUG:teuthology.orchestra.run.smithi028:> import uuid 2024-09-13T17:12:14.230 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:12:14.230 DEBUG:teuthology.orchestra.run.smithi028:> n = 40 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = False 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> if 0 > 0: 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = True 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> path = "/home/ubuntu/cephtest/mnt.0/testdir/file" 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> dpath = os.path.dirname(path) 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> fnameprefix = os.path.basename(path) 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> os.makedirs(dpath, exist_ok=True) 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> for i in range(n): 2024-09-13T17:12:14.231 DEBUG:teuthology.orchestra.run.smithi028:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> with open(fpath, '"'"'w'"'"') as f: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> f.write(f"{i}") 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> f.flush() 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(f.fileno()) 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> os.unlink(fpath) 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> if create_hard_links: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> for j in range(0): 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:12:14.232 DEBUG:teuthology.orchestra.run.smithi028:> finally: 2024-09-13T17:12:14.233 DEBUG:teuthology.orchestra.run.smithi028:> os.close(dirfd) 2024-09-13T17:12:14.233 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:12:14.555 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:14.555 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:14.631 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:14.631 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:14.707 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.900+0000 7f76fd400640 1 Processor -- start 2024-09-13T17:12:14.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.900+0000 7f76fd400640 1 -- start start 2024-09-13T17:12:14.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.901+0000 7f76fd400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76f81592a0 0x7f76f815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.901+0000 7f76fd400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f8159d90 con 0x7f76f806f970 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76fd400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f80b2040 con 0x7f76f806c2d0 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76fd400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f815b0b0 con 0x7f76f81592a0 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76f806f970 0x7f76f8159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54272/0 (socket says 172.21.15.28:54272) 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76efe00640 1 -- 172.21.15.28:0/4124829195 learned_addr learned my addr 172.21.15.28:0/4124829195 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:14.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f6800640 1 --2- 172.21.15.28:0/4124829195 >> v2:172.21.15.189:3300/0 conn(0x7f76f81592a0 0x7f76f815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033401474 0 0) 0x7f76f8159d90 con 0x7f76f806f970 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76dc003520 con 0x7f76f806f970 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395975233 0 0) 0x7f76f80b2040 con 0x7f76f806c2d0 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76f8159d90 con 0x7f76f806c2d0 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.902+0000 7f76f6800640 1 -- 172.21.15.28:0/4124829195 >> v1:172.21.15.189:6789/0 conn(0x7f76f806c2d0 legacy=0x7f76f80b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.903+0000 7f76f6800640 1 -- 172.21.15.28:0/4124829195 >> v1:172.21.15.28:6789/0 conn(0x7f76f806f970 legacy=0x7f76f8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:14.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.903+0000 7f76f6800640 1 -- 172.21.15.28:0/4124829195 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8159ff0 con 0x7f76f81592a0 2024-09-13T17:12:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.904+0000 7f76f6800640 1 --2- 172.21.15.28:0/4124829195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f81592a0 0x7f76f815ab70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f76e80097a0 tx=0x7f76e802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f12b0f72bf63ffb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.904+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8032440 con 0x7f76f81592a0 2024-09-13T17:12:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.904+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e80325a0 con 0x7f76f81592a0 2024-09-13T17:12:14.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.904+0000 7f76f5e00640 1 -- 172.21.15.28:0/4124829195 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8032930 con 0x7f76f81592a0 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 -- 172.21.15.28:0/4124829195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f81592a0 msgr2=0x7f76f815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 --2- 172.21.15.28:0/4124829195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f81592a0 0x7f76f815ab70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f76e80097a0 tx=0x7f76e802a230 comp rx=0 tx=0).stop 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 -- 172.21.15.28:0/4124829195 shutdown_connections 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 --2- 172.21.15.28:0/4124829195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f81592a0 0x7f76f815ab70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 -- 172.21.15.28:0/4124829195 >> 172.21.15.28:0/4124829195 conn(0x7f76f8069e80 msgr2=0x7f76f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:14.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 -- 172.21.15.28:0/4124829195 shutdown_connections 2024-09-13T17:12:14.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.905+0000 7f76fd400640 1 -- 172.21.15.28:0/4124829195 wait complete. 2024-09-13T17:12:14.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.907+0000 7f76fd400640 1 Processor -- start 2024-09-13T17:12:14.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.907+0000 7f76fd400640 1 -- start start 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f806c2d0 0x7f76f814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76f806f970 0x7f76f8154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f814ba00 con 0x7f76f81592a0 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8145cf0 con 0x7f76f806c2d0 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76fd400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f8145fd0 con 0x7f76f806f970 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60944/0 (socket says 172.21.15.28:60944) 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 -- 172.21.15.28:0/1188363134 learned_addr learned my addr 172.21.15.28:0/1188363134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f6800640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f806c2d0 0x7f76f814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:14.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76efe00640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76f806f970 0x7f76f8154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 -- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76f806f970 msgr2=0x7f76f8154470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76f806f970 0x7f76f8154470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 -- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f806c2d0 msgr2=0x7f76f814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f806c2d0 0x7f76f814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.908+0000 7f76f7200640 1 -- 172.21.15.28:0/1188363134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8009400 con 0x7f76f81592a0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76f7200640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c6e0 tx=0x7f76f000cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0003b70 con 0x7f76f81592a0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0003cd0 con 0x7f76f81592a0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f00052d0 con 0x7f76f81592a0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f81462b0 con 0x7f76f81592a0 2024-09-13T17:12:14.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.909+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f8146870 con 0x7f76f81592a0 2024-09-13T17:12:14.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.910+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f815ab70 con 0x7f76f81592a0 2024-09-13T17:12:14.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.911+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76f0005430 con 0x7f76f81592a0 2024-09-13T17:12:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.912+0000 7f76eea00640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 0x7f76d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.912+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76f00890d0 con 0x7f76f81592a0 2024-09-13T17:12:14.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.913+0000 7f76f6800640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 0x7f76d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:14.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.914+0000 7f76f6800640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 0x7f76d406d0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f76e8034000 tx=0x7f76e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:14.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:14.917+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f76f00542c0 con 0x7f76f81592a0 2024-09-13T17:12:15.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.096+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76f806a260 con 0x7f76f81592a0 2024-09-13T17:12:15.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.098+0000 7f76eea00640 1 -- 172.21.15.28:0/1188363134 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16565 (secure 0 0 0) 0x7f76f00578e0 con 0x7f76f81592a0 2024-09-13T17:12:15.100 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:15.100 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":61,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:05.641862+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 msgr2=0x7f76d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 0x7f76d406d0c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f76e8034000 tx=0x7f76e802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 msgr2=0x7f76f80cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f76f000c6e0 tx=0x7f76f000cba0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 shutdown_connections 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76d406ac00 0x7f76d406d0c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76f81592a0 0x7f76f80cb4e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76f806f970 0x7f76f8154470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 --2- 172.21.15.28:0/1188363134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76f806c2d0 0x7f76f814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 >> 172.21.15.28:0/1188363134 conn(0x7f76f8069e80 msgr2=0x7f76f815c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 shutdown_connections 2024-09-13T17:12:15.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.100+0000 7f76fd400640 1 -- 172.21.15.28:0/1188363134 wait complete. 2024-09-13T17:12:15.117 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:12:02.552870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:12:03.628750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:12:15.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:12:15.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.275+0000 7fad1ca00640 1 Processor -- start 2024-09-13T17:12:15.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.275+0000 7fad1ca00640 1 -- start start 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad1ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad180b3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad1ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad1815dbf0 0x7fad180b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad1ca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad180b4350 con 0x7fad180cb6e0 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad1ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad1815cf00 con 0x7fad1815dbf0 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad1ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad180b4070 con 0x7fad180ae580 2024-09-13T17:12:15.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad0fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad1815dbf0 0x7fad180b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad0fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fad1815dbf0 0x7fad180b3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60960/0 (socket says 172.21.15.28:60960) 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad16800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad180cb6e0 0x7fad181604f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54276/0 (socket says 172.21.15.28:54276) 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.276+0000 7fad0fe00640 1 -- 172.21.15.28:0/4209790311 learned_addr learned my addr 172.21.15.28:0/4209790311 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad15e00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad180b3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad0fe00640 1 -- 172.21.15.28:0/4209790311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 msgr2=0x7fad180b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad0fe00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad180b3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad0fe00640 1 -- 172.21.15.28:0/4209790311 >> v1:172.21.15.28:6789/0 conn(0x7fad180cb6e0 legacy=0x7fad181604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad15400640 1 -- 172.21.15.28:0/4209790311 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761852420 0 0) 0x7fad10001490 con 0x7fad180cb6e0 2024-09-13T17:12:15.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.277+0000 7fad0fe00640 1 -- 172.21.15.28:0/4209790311 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad1815d110 con 0x7fad1815dbf0 2024-09-13T17:12:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.278+0000 7fad0fe00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1815dbf0 0x7fad180b3a80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fad00009870 tx=0x7fad0002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=501bed34f2760917 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.278+0000 7fad15400640 1 -- 172.21.15.28:0/4209790311 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00032440 con 0x7fad1815dbf0 2024-09-13T17:12:15.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.278+0000 7fad15400640 1 -- 172.21.15.28:0/4209790311 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad000325a0 con 0x7fad1815dbf0 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.278+0000 7fad15400640 1 -- 172.21.15.28:0/4209790311 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00032930 con 0x7fad1815dbf0 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 -- 172.21.15.28:0/4209790311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1815dbf0 msgr2=0x7fad180b3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1815dbf0 0x7fad180b3a80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fad00009870 tx=0x7fad0002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 -- 172.21.15.28:0/4209790311 shutdown_connections 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad1815dbf0 0x7fad180b3a80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 --2- 172.21.15.28:0/4209790311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad180b3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 -- 172.21.15.28:0/4209790311 >> 172.21.15.28:0/4209790311 conn(0x7fad180b0af0 msgr2=0x7fad180b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 -- 172.21.15.28:0/4209790311 shutdown_connections 2024-09-13T17:12:15.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.279+0000 7fad1ca00640 1 -- 172.21.15.28:0/4209790311 wait complete. 2024-09-13T17:12:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.281+0000 7fad1ca00640 1 Processor -- start 2024-09-13T17:12:15.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.281+0000 7fad1ca00640 1 -- start start 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad181f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1815dbf0 0x7fad18070d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f54f0 con 0x7fad180cb6e0 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f5750 con 0x7fad1815dbf0 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad1ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad181f5a30 con 0x7fad180ae580 2024-09-13T17:12:15.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60962/0 (socket says 172.21.15.28:60962) 2024-09-13T17:12:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 -- 172.21.15.28:0/583775299 learned_addr learned my addr 172.21.15.28:0/583775299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad15e00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad181f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad16800640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1815dbf0 0x7fad18070d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 -- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 msgr2=0x7fad181f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad181f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 -- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1815dbf0 msgr2=0x7fad18070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1815dbf0 0x7fad18070d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.282+0000 7fad0fe00640 1 -- 172.21.15.28:0/583775299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000094d0 con 0x7fad180cb6e0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad0fe00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fad0002a200 tx=0x7fad000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad00032440 con 0x7fad180cb6e0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad00030ba0 con 0x7fad180cb6e0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0003d440 con 0x7fad180cb6e0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad181f5d10 con 0x7fad180cb6e0 2024-09-13T17:12:15.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.283+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad18071fa0 con 0x7fad180cb6e0 2024-09-13T17:12:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.284+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad180cbb70 con 0x7fad180cb6e0 2024-09-13T17:12:15.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.285+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad00039040 con 0x7fad180cb6e0 2024-09-13T17:12:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.286+0000 7fad0e000640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 0x7facf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.287+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad000a9dd0 con 0x7fad180cb6e0 2024-09-13T17:12:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.287+0000 7fad15e00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 0x7facf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.288+0000 7fad15e00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 0x7facf406d0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fad08006fd0 tx=0x7fad08008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.291+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fad00075050 con 0x7fad180cb6e0 2024-09-13T17:12:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.466+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fad180cbeb0 con 0x7fad180cb6e0 2024-09-13T17:12:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.467+0000 7fad0e000640 1 -- 172.21.15.28:0/583775299 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v61) v1 ==== 93+0+10 (secure 0 0 0) 0x7fad00078670 con 0x7fad180cb6e0 2024-09-13T17:12:15.469 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 msgr2=0x7facf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 0x7facf406d0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fad08006fd0 tx=0x7fad08008040 comp rx=0 tx=0).stop 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 msgr2=0x7fad18071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fad0002a200 tx=0x7fad000046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 shutdown_connections 2024-09-13T17:12:15.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facf406ac00 0x7facf406d0c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad1815dbf0 0x7fad18070d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad180cb6e0 0x7fad18071a60 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 --2- 172.21.15.28:0/583775299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad180ae580 0x7fad181f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 >> 172.21.15.28:0/583775299 conn(0x7fad180b0af0 msgr2=0x7fad1815f7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 shutdown_connections 2024-09-13T17:12:15.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.469+0000 7fad1ca00640 1 -- 172.21.15.28:0/583775299 wait complete. 2024-09-13T17:12:15.488 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:12:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.644+0000 7f7e2dc00640 1 Processor -- start 2024-09-13T17:12:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.644+0000 7f7e2dc00640 1 -- start start 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 0x7f7e2806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e2dc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2806f920 con 0x7f7e2806f3f0 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e2dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2815b290 con 0x7f7e2815cd80 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e2dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2815b570 con 0x7f7e28159770 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e1e000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e2806f3f0 0x7f7e28159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54280/0 (socket says 172.21.15.28:54280) 2024-09-13T17:12:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.645+0000 7f7e1e000640 1 -- 172.21.15.28:0/4006527881 learned_addr learned my addr 172.21.15.28:0/4006527881 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26a00640 1 --2- 172.21.15.28:0/4006527881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 0x7f7e2806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320387704 0 0) 0x7f7e2806f920 con 0x7f7e2806f3f0 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c005730 con 0x7f7e2806f3f0 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543380767 0 0) 0x7f7e2815b290 con 0x7f7e2815cd80 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e2806f920 con 0x7f7e2815cd80 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26a00640 1 -- 172.21.15.28:0/4006527881 >> v1:172.21.15.189:6789/0 conn(0x7f7e2815cd80 legacy=0x7f7e2815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26a00640 1 -- 172.21.15.28:0/4006527881 >> v1:172.21.15.28:6789/0 conn(0x7f7e2806f3f0 legacy=0x7f7e28159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.646+0000 7f7e26a00640 1 -- 172.21.15.28:0/4006527881 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2815b850 con 0x7f7e28159770 2024-09-13T17:12:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.647+0000 7f7e26a00640 1 --2- 172.21.15.28:0/4006527881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 0x7f7e2806fd20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097d0 tx=0x7f7e1802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40d218f0bf7f901b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.648+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18032440 con 0x7f7e28159770 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.648+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e180325a0 con 0x7f7e28159770 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.648+0000 7f7e26000640 1 -- 172.21.15.28:0/4006527881 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e18032930 con 0x7f7e28159770 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.648+0000 7f7e2dc00640 1 -- 172.21.15.28:0/4006527881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 msgr2=0x7f7e2806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.648+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/4006527881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 0x7f7e2806fd20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e180097d0 tx=0x7f7e1802a180 comp rx=0 tx=0).stop 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.649+0000 7f7e2dc00640 1 -- 172.21.15.28:0/4006527881 shutdown_connections 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.649+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/4006527881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e28159770 0x7f7e2806fd20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.649+0000 7f7e2dc00640 1 -- 172.21.15.28:0/4006527881 >> 172.21.15.28:0/4006527881 conn(0x7f7e280b0270 msgr2=0x7f7e280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.649+0000 7f7e2dc00640 1 -- 172.21.15.28:0/4006527881 shutdown_connections 2024-09-13T17:12:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.649+0000 7f7e2dc00640 1 -- 172.21.15.28:0/4006527881 wait complete. 2024-09-13T17:12:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.650+0000 7f7e2dc00640 1 Processor -- start 2024-09-13T17:12:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.650+0000 7f7e2dc00640 1 -- start start 2024-09-13T17:12:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 0x7f7e281f1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e2815cd80 0x7f7e280b2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281ee9e0 con 0x7f7e28159770 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281538e0 con 0x7f7e2806f3f0 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.651+0000 7f7e2dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e281ee080 con 0x7f7e2815cd80 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60964/0 (socket says 172.21.15.28:60964) 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 -- 172.21.15.28:0/2875748784 learned_addr learned my addr 172.21.15.28:0/2875748784 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e26a00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 0x7f7e281f1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e27400640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e2815cd80 0x7f7e280b2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 -- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e2815cd80 msgr2=0x7f7e280b2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e2815cd80 0x7f7e280b2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 -- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 msgr2=0x7f7e281f1da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 0x7f7e281f1da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 -- 172.21.15.28:0/2875748784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e18009400 con 0x7f7e28159770 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e26a00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 0x7f7e281f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.652+0000 7f7e1e000640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000c4d0 tx=0x7f7e1000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.653+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10005ce0 con 0x7f7e28159770 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.653+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10005e40 con 0x7f7e28159770 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.653+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e281ee320 con 0x7f7e28159770 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.653+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e10007270 con 0x7f7e28159770 2024-09-13T17:12:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.653+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e281eee00 con 0x7f7e28159770 2024-09-13T17:12:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.654+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e280afdc0 con 0x7f7e28159770 2024-09-13T17:12:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.655+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e100073d0 con 0x7f7e28159770 2024-09-13T17:12:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.655+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 0x7f7e0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.656+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e10088890 con 0x7f7e28159770 2024-09-13T17:12:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.656+0000 7f7e26a00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 0x7f7e0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.657+0000 7f7e26a00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 0x7f7e0406d0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7e18034000 tx=0x7f7e1802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:15.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.660+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e10053a80 con 0x7f7e28159770 2024-09-13T17:12:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.838+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7e28159da0 con 0x7f7e28159770 2024-09-13T17:12:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.839+0000 7f7e1f400640 1 -- 172.21.15.28:0/2875748784 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v61) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7e100170a0 con 0x7f7e28159770 2024-09-13T17:12:15.841 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:12:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 msgr2=0x7f7e0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 0x7f7e0406d0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7e18034000 tx=0x7f7e1802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 msgr2=0x7f7e280a8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7e1000c4d0 tx=0x7f7e1000c990 comp rx=0 tx=0).stop 2024-09-13T17:12:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 shutdown_connections 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e0406ac00 0x7f7e0406d0c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e2815cd80 0x7f7e280b2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28159770 0x7f7e280a8d30 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 --2- 172.21.15.28:0/2875748784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e2806f3f0 0x7f7e281f1da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 >> 172.21.15.28:0/2875748784 conn(0x7f7e280b0270 msgr2=0x7f7e280af2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 shutdown_connections 2024-09-13T17:12:15.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:15.841+0000 7f7e2dc00640 1 -- 172.21.15.28:0/2875748784 wait complete. 2024-09-13T17:12:15.859 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:12:16.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.015+0000 7f5283000640 1 Processor -- start 2024-09-13T17:12:16.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.015+0000 7f5283000640 1 -- start start 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5283000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f527c0cc530 0x7f527c072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5283000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5283000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0c78e0 con 0x7f527c0cc910 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5283000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c070c90 con 0x7f527c0cc530 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5283000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c0b1080 con 0x7f527c0b06f0 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f527c0cc530 0x7f527c072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f527c0cc530 0x7f527c072060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60980/0 (socket says 172.21.15.28:60980) 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 -- 172.21.15.28:0/3488712522 learned_addr learned my addr 172.21.15.28:0/3488712522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 -- 172.21.15.28:0/3488712522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 msgr2=0x7f527c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5282600640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c0b0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 -- 172.21.15.28:0/3488712522 >> v1:172.21.15.189:6789/0 conn(0x7f527c0cc910 legacy=0x7f527c15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5281c00640 1 -- 172.21.15.28:0/3488712522 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c0cbe40 con 0x7f527c0cc530 2024-09-13T17:12:16.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.017+0000 7f5282600640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.018+0000 7f5281c00640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c072060 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5270009870 tx=0x7f527002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282b4045be64d117 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f527be00640 1 -- 172.21.15.28:0/3488712522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270032440 con 0x7f527c0cc530 2024-09-13T17:12:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f527be00640 1 -- 172.21.15.28:0/3488712522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52700325a0 con 0x7f527c0cc530 2024-09-13T17:12:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f527be00640 1 -- 172.21.15.28:0/3488712522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270032930 con 0x7f527c0cc530 2024-09-13T17:12:16.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 -- 172.21.15.28:0/3488712522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 msgr2=0x7f527c072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c072060 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f5270009870 tx=0x7f527002a2a0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 -- 172.21.15.28:0/3488712522 shutdown_connections 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 --2- 172.21.15.28:0/3488712522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c072060 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 -- 172.21.15.28:0/3488712522 >> 172.21.15.28:0/3488712522 conn(0x7f527c069e80 msgr2=0x7f527c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 -- 172.21.15.28:0/3488712522 shutdown_connections 2024-09-13T17:12:16.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.019+0000 7f5283000640 1 -- 172.21.15.28:0/3488712522 wait complete. 2024-09-13T17:12:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.021+0000 7f5283000640 1 Processor -- start 2024-09-13T17:12:16.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.021+0000 7f5283000640 1 -- start start 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c152c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527c0cc910 0x7f527c06b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c1f1ec0 con 0x7f527c0cc530 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c15c460 con 0x7f527c0cc910 2024-09-13T17:12:16.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5283000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c1f2110 con 0x7f527c0b06f0 2024-09-13T17:12:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5281200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5281200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60982/0 (socket says 172.21.15.28:60982) 2024-09-13T17:12:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5281200640 1 -- 172.21.15.28:0/449330158 learned_addr learned my addr 172.21.15.28:0/449330158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5282600640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527c0cc910 0x7f527c06b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.022+0000 7f5281c00640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c152c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 -- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 msgr2=0x7f527c152c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c152c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 -- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527c0cc910 msgr2=0x7f527c06b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527c0cc910 0x7f527c06b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 -- 172.21.15.28:0/449330158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52700094d0 con 0x7f527c0cc530 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5281200640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f526800c430 tx=0x7f526800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268011790 con 0x7f527c0cc530 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52680118f0 con 0x7f527c0cc530 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5268011ce0 con 0x7f527c0cc530 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.023+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f527c1f23f0 con 0x7f527c0cc530 2024-09-13T17:12:16.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.024+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c1f29b0 con 0x7f527c0cc530 2024-09-13T17:12:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.024+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c1f2bd0 con 0x7f527c0cc530 2024-09-13T17:12:16.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.026+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f526801c460 con 0x7f527c0cc530 2024-09-13T17:12:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.026+0000 7f5279600640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 0x7f525c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.026+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5268089200 con 0x7f527c0cc530 2024-09-13T17:12:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.027+0000 7f5281c00640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 0x7f525c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.028+0000 7f5281c00640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 0x7f525c06d0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5270034000 tx=0x7f527002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.031+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f52680543f0 con 0x7f527c0cc530 2024-09-13T17:12:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.208+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f527c071b40 con 0x7f527c0cc530 2024-09-13T17:12:16.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.208+0000 7f5279600640 1 -- 172.21.15.28:0/449330158 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v61) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5268057a10 con 0x7f527c0cc530 2024-09-13T17:12:16.210 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 msgr2=0x7f525c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 0x7f525c06d0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5270034000 tx=0x7f527002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 msgr2=0x7f527c06b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f526800c430 tx=0x7f526800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 shutdown_connections 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f525c06ac00 0x7f525c06d0c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f527c0cc910 0x7f527c06b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f527c0cc530 0x7f527c06b330 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 --2- 172.21.15.28:0/449330158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f527c0b06f0 0x7f527c152c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 >> 172.21.15.28:0/449330158 conn(0x7f527c069e80 msgr2=0x7f527c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.210+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 shutdown_connections 2024-09-13T17:12:16.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.211+0000 7f5283000640 1 -- 172.21.15.28:0/449330158 wait complete. 2024-09-13T17:12:16.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:12:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.384+0000 7fcae0600640 1 Processor -- start 2024-09-13T17:12:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.384+0000 7fcae0600640 1 -- start start 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcae0600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcae0600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcad815ab60 0x7fcad80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcae0600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad806c8b0 con 0x7fcad806bb40 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcae0600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcad806c6a0 con 0x7fcad815ab60 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcae0600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad806cb90 con 0x7fcad815a780 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcadd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcad815ab60 0x7fcad80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcadd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcad815ab60 0x7fcad80712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60986/0 (socket says 172.21.15.28:60986) 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcadd200640 1 -- 172.21.15.28:0/3569519647 learned_addr learned my addr 172.21.15.28:0/3569519647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.385+0000 7fcaddc00640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.386+0000 7fcadd200640 1 -- 172.21.15.28:0/3569519647 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 msgr2=0x7fcad8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.386+0000 7fcadd200640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.386+0000 7fcadd200640 1 -- 172.21.15.28:0/3569519647 >> v1:172.21.15.189:6789/0 conn(0x7fcad806bb40 legacy=0x7fcad806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.386+0000 7fcadd200640 1 -- 172.21.15.28:0/3569519647 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad80cc980 con 0x7fcad815ab60 2024-09-13T17:12:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.386+0000 7fcad7e00640 1 -- 172.21.15.28:0/3569519647 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851583323 0 0) 0x7fcad806c8b0 con 0x7fcad806bb40 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.387+0000 7fcadd200640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80712c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009870 tx=0x7fcac402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a3420115f8f6c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.387+0000 7fcad7e00640 1 -- 172.21.15.28:0/3569519647 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4032440 con 0x7fcad815ab60 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.387+0000 7fcad7e00640 1 -- 172.21.15.28:0/3569519647 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac40325a0 con 0x7fcad815ab60 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.387+0000 7fcad7e00640 1 -- 172.21.15.28:0/3569519647 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac4032930 con 0x7fcad815ab60 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 -- 172.21.15.28:0/3569519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 msgr2=0x7fcad80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80712c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fcac4009870 tx=0x7fcac402a250 comp rx=0 tx=0).stop 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 -- 172.21.15.28:0/3569519647 shutdown_connections 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80712c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 --2- 172.21.15.28:0/3569519647 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 -- 172.21.15.28:0/3569519647 >> 172.21.15.28:0/3569519647 conn(0x7fcad80b0270 msgr2=0x7fcad80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 -- 172.21.15.28:0/3569519647 shutdown_connections 2024-09-13T17:12:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.388+0000 7fcae0600640 1 -- 172.21.15.28:0/3569519647 wait complete. 2024-09-13T17:12:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.390+0000 7fcae0600640 1 Processor -- start 2024-09-13T17:12:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.390+0000 7fcae0600640 1 -- start start 2024-09-13T17:12:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad806bb40 0x7fcad81f1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad80c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80c4480 con 0x7fcad815ab60 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80c4760 con 0x7fcad806bb40 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcae0600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad80c4a40 con 0x7fcad815a780 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcade600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcade600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60994/0 (socket says 172.21.15.28:60994) 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcade600640 1 -- 172.21.15.28:0/3620915783 learned_addr learned my addr 172.21.15.28:0/3620915783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcaddc00640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad806bb40 0x7fcad81f1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.391+0000 7fcadd200640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad80c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 -- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 msgr2=0x7fcad80c4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad80c4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 -- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad806bb40 msgr2=0x7fcad81f1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad806bb40 0x7fcad81f1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 -- 172.21.15.28:0/3620915783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac40094d0 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcadd200640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad80c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcade600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcad0001560 tx=0x7fcad000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0003da0 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad0005070 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad0005420 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad80b17f0 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.392+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad80c36b0 con 0x7fcad815ab60 2024-09-13T17:12:16.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.393+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad80b19e0 con 0x7fcad815ab60 2024-09-13T17:12:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.394+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcad0005580 con 0x7fcad815ab60 2024-09-13T17:12:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.395+0000 7fcad5600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 0x7fcab806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.395+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcad0088b90 con 0x7fcad815ab60 2024-09-13T17:12:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.396+0000 7fcaddc00640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 0x7fcab806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.396+0000 7fcaddc00640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 0x7fcab806d0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009b40 tx=0x7fcacc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.400+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcad0053e10 con 0x7fcad815ab60 2024-09-13T17:12:16.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.604+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcad806c8b0 con 0x7fcad815ab60 2024-09-13T17:12:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.605+0000 7fcad5600640 1 -- 172.21.15.28:0/3620915783 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v61) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcad0017070 con 0x7fcad815ab60 2024-09-13T17:12:16.607 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 msgr2=0x7fcab806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 0x7fcab806d0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009b40 tx=0x7fcacc009190 comp rx=0 tx=0).stop 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 msgr2=0x7fcad80c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fcad0001560 tx=0x7fcad000cc70 comp rx=0 tx=0).stop 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 shutdown_connections 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcab806ac00 0x7fcab806d0c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcad815ab60 0x7fcad80c54f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcad815a780 0x7fcad80c4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 --2- 172.21.15.28:0/3620915783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcad806bb40 0x7fcad81f1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 >> 172.21.15.28:0/3620915783 conn(0x7fcad80b0270 msgr2=0x7fcad80ce5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 shutdown_connections 2024-09-13T17:12:16.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.607+0000 7fcae0600640 1 -- 172.21.15.28:0/3620915783 wait complete. 2024-09-13T17:12:16.625 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:12:16.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:16.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:12:16.755 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:12:16.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:12:16.755 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:12:16.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 flush journal 2024-09-13T17:12:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.987+0000 7f5467200640 1 Processor -- start 2024-09-13T17:12:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.987+0000 7f5467200640 1 -- start start 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f5467200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f546006f3f0 0x7f546015c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f5467200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546015ab60 con 0x7f5460159770 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f5467200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f546006f920 con 0x7f5460070430 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f5467200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f546015b290 con 0x7f546006f3f0 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f545fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5460159770 0x7f546006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54284/0 (socket says 172.21.15.28:54284) 2024-09-13T17:12:16.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.988+0000 7f545fe00640 1 -- 172.21.15.28:0/4248861108 learned_addr learned my addr 172.21.15.28:0/4248861108 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f5465200640 1 --2- 172.21.15.28:0/4248861108 >> v2:172.21.15.189:3300/0 conn(0x7f546006f3f0 0x7f546015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635142732 0 0) 0x7f5450001110 con 0x7f5460159770 2024-09-13T17:12:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5444003520 con 0x7f5460159770 2024-09-13T17:12:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740024337 0 0) 0x7f546006f920 con 0x7f5460070430 2024-09-13T17:12:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5450001110 con 0x7f5460070430 2024-09-13T17:12:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f5465200640 1 -- 172.21.15.28:0/4248861108 >> v1:172.21.15.189:6789/0 conn(0x7f5460070430 legacy=0x7f546015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264440365 0 0) 0x7f5444003520 con 0x7f5460159770 2024-09-13T17:12:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f5465200640 1 -- 172.21.15.28:0/4248861108 >> v1:172.21.15.28:6789/0 conn(0x7f5460159770 legacy=0x7f546006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.989+0000 7f5465200640 1 -- 172.21.15.28:0/4248861108 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f546015b570 con 0x7f546006f3f0 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.990+0000 7f5465200640 1 --2- 172.21.15.28:0/4248861108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f546006f3f0 0x7f546015c480 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5458009d90 tx=0x7f545802a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df1f1b39e9f0021f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.990+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450001800 con 0x7f546006f3f0 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f545802d050 con 0x7f546006f3f0 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f545ea00640 1 -- 172.21.15.28:0/4248861108 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5458007760 con 0x7f546006f3f0 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 -- 172.21.15.28:0/4248861108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f546006f3f0 msgr2=0x7f546015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 --2- 172.21.15.28:0/4248861108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f546006f3f0 0x7f546015c480 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5458009d90 tx=0x7f545802a5f0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 -- 172.21.15.28:0/4248861108 shutdown_connections 2024-09-13T17:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 --2- 172.21.15.28:0/4248861108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f546006f3f0 0x7f546015c480 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 -- 172.21.15.28:0/4248861108 >> 172.21.15.28:0/4248861108 conn(0x7f54600b0270 msgr2=0x7f54600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 -- 172.21.15.28:0/4248861108 shutdown_connections 2024-09-13T17:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.991+0000 7f5467200640 1 -- 172.21.15.28:0/4248861108 wait complete. 2024-09-13T17:12:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.993+0000 7f5467200640 1 Processor -- start 2024-09-13T17:12:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.993+0000 7f5467200640 1 -- start start 2024-09-13T17:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 0x7f54601518c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 0x7f5460152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54601e0af0 con 0x7f546006f3f0 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54601507f0 con 0x7f5460070430 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5467200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5460152b30 con 0x7f5460159770 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32776/0 (socket says 172.21.15.28:32776) 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f545fe00640 1 -- 172.21.15.28:0/2969112433 learned_addr learned my addr 172.21.15.28:0/2969112433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f545f400640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 0x7f54601518c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.994+0000 7f5465200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 0x7f5460152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 -- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 msgr2=0x7f5460152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 0x7f5460152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 -- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 msgr2=0x7f54601518c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 0x7f54601518c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 -- 172.21.15.28:0/2969112433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5458009a70 con 0x7f546006f3f0 2024-09-13T17:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5465200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 0x7f5460152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545f400640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 0x7f54601518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f545fe00640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f545000c710 tx=0x7f545000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5457e00640 1 -- 172.21.15.28:0/2969112433 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450005ce0 con 0x7f546006f3f0 2024-09-13T17:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5457e00640 1 -- 172.21.15.28:0/2969112433 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5450005e60 con 0x7f546006f3f0 2024-09-13T17:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5460150b00 con 0x7f546006f3f0 2024-09-13T17:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5457e00640 1 -- 172.21.15.28:0/2969112433 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54500072b0 con 0x7f546006f3f0 2024-09-13T17:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.995+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54601510c0 con 0x7f546006f3f0 2024-09-13T17:12:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.997+0000 7f5457e00640 1 -- 172.21.15.28:0/2969112433 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5450007410 con 0x7f546006f3f0 2024-09-13T17:12:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.998+0000 7f5457e00640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 0x7f544006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.998+0000 7f5457e00640 1 -- 172.21.15.28:0/2969112433 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5450089140 con 0x7f546006f3f0 2024-09-13T17:12:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.999+0000 7f546a0aa740 1 Processor -- start 2024-09-13T17:12:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.999+0000 7f545f400640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 0x7f544006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.999+0000 7f546a0aa740 1 -- start start 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:16.999+0000 7f545f400640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 0x7f544006d0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f54600b30e0 tx=0x7f544c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7d4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb821670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb6da6a0 con 0x5612bb7d1420 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb73dcc0 con 0x5612bb7d3c60 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5612bb73b450 con 0x5612bb7fb110 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32794/0 (socket says 172.21.15.28:32794) 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 -- 172.21.15.28:0/4194531428 learned_addr learned my addr 172.21.15.28:0/4194531428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f5465200640 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb821670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545fe00640 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7d4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 -- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 msgr2=0x5612bb821670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb821670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 -- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 msgr2=0x5612bb7d4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7d4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.000+0000 7f545f400640 1 -- 172.21.15.28:0/4194531428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612bb7072e0 con 0x5612bb7d1420 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.001+0000 7f545f400640 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f544c005100 tx=0x7f544c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7083e68c851c45c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.001+0000 7f5454c00640 1 -- 172.21.15.28:0/4194531428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c0035f0 con 0x5612bb7d1420 2024-09-13T17:12:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.001+0000 7f5454c00640 1 -- 172.21.15.28:0/4194531428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c003750 con 0x5612bb7d1420 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.001+0000 7f5454c00640 1 -- 172.21.15.28:0/4194531428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c003ac0 con 0x5612bb7d1420 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- 172.21.15.28:0/4194531428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 msgr2=0x5612bb821130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f544c005100 tx=0x7f544c011000 comp rx=0 tx=0).stop 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- 172.21.15.28:0/4194531428 shutdown_connections 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb821670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d1420 0x5612bb821130 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 --2- 172.21.15.28:0/4194531428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7d4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- 172.21.15.28:0/4194531428 >> 172.21.15.28:0/4194531428 conn(0x5612bb7d0390 msgr2=0x5612bb818840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- 172.21.15.28:0/4194531428 shutdown_connections 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- 172.21.15.28:0/4194531428 wait complete. 2024-09-13T17:12:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.002+0000 7f546a0aa740 1 -- start start 2024-09-13T17:12:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.003+0000 7f546a0aa740 1 Processor -- start 2024-09-13T17:12:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.003+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.003+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb7cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 0x5612bb7cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f546a0aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb736d40 con 0x5612bb7d3c60 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb6d8280 con 0x5612bb7fa680 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5612bb6d7f20 con 0x5612bb7fb110 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f546a0aa740 10 client.0 _ll_register_callbacks cb 0x5612bb7303b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32804/0 (socket says 172.21.15.28:32804) 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 -- 172.21.15.28:0/1813156630 learned_addr learned my addr 172.21.15.28:0/1813156630 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545f400640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb7cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f5465200640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 0x5612bb7cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 -- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 msgr2=0x5612bb7cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb7cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 -- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 msgr2=0x5612bb7cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 0x5612bb7cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f545fe00640 1 -- 172.21.15.28:0/1813156630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c00f040 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.004+0000 7f5465200640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 0x5612bb7cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f545fe00640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f54500875d0 tx=0x7f5450058670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a46fe65ccf1935 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5450060330 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5450087220 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5612bb42a4e0 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54500573a0 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 10 client.6633 fetch_fsmap learned FSMap version 49 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x5612bb7353c0 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5612bb42a4e0 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5450019020 con 0x5612bb7d3c60 2024-09-13T17:12:17.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 10 client.6633 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 10 client.6633 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 20 client.6633 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb81d3e0 0x5612bb81d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 4 client.6633 mds_command: new command op to 6420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.005+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 --> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612bb43b7e0 con 0x5612bb81d3e0 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.006+0000 7f545f400640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb81d3e0 0x5612bb81d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.006+0000 7f5449600640 10 client.6633 ms_handle_connect on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:17.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.006+0000 7f545f400640 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb81d3e0 0x5612bb81d7f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d5ec6a5fd0cbdc3 server_cookie=f79c522d11409821 in_seq=0 out_seq=0 2024-09-13T17:12:17.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.007+0000 7f5449600640 1 -- 172.21.15.28:0/1813156630 <== mds.0 v2:172.21.15.189:6835/693307152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5612bb43b7e0 con 0x5612bb81d3e0 2024-09-13T17:12:17.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.007+0000 7f5449600640 10 client.6633 handle_command_reply: tid=0 2024-09-13T17:12:17.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.007+0000 7f546a0aa740 1 client.6633 shutdown 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 msgr2=0x5612bb7f8900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f54500875d0 tx=0x7f5450058670 comp rx=0 tx=0).stop 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 shutdown_connections 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb81d3e0 0x5612bb81d7f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fa680 0x5612bb7cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb7cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f5449600640 0 client.6633 ms_handle_reset on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1813156630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb7f8900 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 >> 172.21.15.28:0/1813156630 conn(0x5612bb7d1420 msgr2=0x5612bb816b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 shutdown_connections 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 1 -- 172.21.15.28:0/1813156630 wait complete. 2024-09-13T17:12:17.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.008+0000 7f546a0aa740 20 client.6633 trim_cache size 0 max 16384 2024-09-13T17:12:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.021+0000 7f546a0aa740 1 Processor -- start 2024-09-13T17:12:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.021+0000 7f546a0aa740 1 -- start start 2024-09-13T17:12:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7cf200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb81f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb49e040 con 0x5612bb7fa680 2024-09-13T17:12:17.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb81a920 con 0x5612bb7d3c60 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5612bb7d4ac0 con 0x5612bb7fb110 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32808/0 (socket says 172.21.15.28:32808) 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 -- 172.21.15.28:0/1815923621 learned_addr learned my addr 172.21.15.28:0/1815923621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545fe00640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7cf200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f5465200640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb81f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 -- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 msgr2=0x5612bb81f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb81f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 -- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 msgr2=0x5612bb7cf200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7cf200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545f400640 1 -- 172.21.15.28:0/1815923621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612bb6d7610 con 0x5612bb7fa680 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.022+0000 7f545fe00640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f545f400640 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f544c014000 tx=0x7f544c005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee5af0016f45697 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f5455600640 1 -- 172.21.15.28:0/1815923621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c0035f0 con 0x5612bb7fa680 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f5455600640 1 -- 172.21.15.28:0/1815923621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c003a80 con 0x5612bb7fa680 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f5455600640 1 -- 172.21.15.28:0/1815923621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f544c003df0 con 0x5612bb7fa680 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 -- 172.21.15.28:0/1815923621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 msgr2=0x5612bb820ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f544c014000 tx=0x7f544c005000 comp rx=0 tx=0).stop 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 -- 172.21.15.28:0/1815923621 shutdown_connections 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fb110 0x5612bb81f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7fa680 0x5612bb820ad0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 --2- 172.21.15.28:0/1815923621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7d3c60 0x5612bb7cf200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.023+0000 7f546a0aa740 1 -- 172.21.15.28:0/1815923621 >> 172.21.15.28:0/1815923621 conn(0x5612bb7d1420 msgr2=0x5612bb818880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.024+0000 7f546a0aa740 1 -- 172.21.15.28:0/1815923621 shutdown_connections 2024-09-13T17:12:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.024+0000 7f546a0aa740 1 -- 172.21.15.28:0/1815923621 wait complete. 2024-09-13T17:12:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.024+0000 7f546a0aa740 1 -- start start 2024-09-13T17:12:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.024+0000 7f546a0aa740 1 Processor -- start 2024-09-13T17:12:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 0x5612bb83a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 0x5612bb814450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb836ee0 con 0x5612bb7d3c60 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612bb837340 con 0x5612bb7fb110 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5612bb815960 con 0x5612bb7fa680 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f546a0aa740 10 client.0 _ll_register_callbacks cb 0x7f545000c660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.025+0000 7f545fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32812/0 (socket says 172.21.15.28:32812) 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 -- 172.21.15.28:0/367698994 learned_addr learned my addr 172.21.15.28:0/367698994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f5465200640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 0x5612bb814450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545f400640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 0x5612bb83a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 -- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 msgr2=0x5612bb83a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 0x5612bb83a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 -- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 msgr2=0x5612bb814450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 0x5612bb814450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 -- 172.21.15.28:0/367698994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612bb7072e0 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545f400640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 0x5612bb83a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f5465200640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 0x5612bb814450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f545fe00640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f54500564a0 tx=0x7f5450057db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f4694a088777667 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f5449600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54500541f0 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5450060330 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.026+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f544c003750 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54500600b0 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 10 client.6639 fetch_fsmap learned FSMap version 49 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x5612bb6fcb20 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f544c003750 con 0x5612bb7d3c60 2024-09-13T17:12:17.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5450019040 con 0x5612bb7d3c60 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 10 client.6639 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 10 client.6639 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 20 client.6639 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb7feb80 0x5612bb814990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 4 client.6639 mds_command: new command op to 6420 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 --> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5612bb48ee30 con 0x5612bb7feb80 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.027+0000 7f545f400640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb7feb80 0x5612bb814990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.028+0000 7f545f400640 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb7feb80 0x5612bb814990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=724b46320eaf9def server_cookie=17eda85087d432f7 in_seq=0 out_seq=0 2024-09-13T17:12:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.028+0000 7f5449600640 10 client.6639 ms_handle_connect on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.044+0000 7f5449600640 1 -- 172.21.15.28:0/367698994 <== mds.0 v2:172.21.15.189:6835/693307152 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5612bb48ee30 con 0x5612bb7feb80 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.044+0000 7f5449600640 10 client.6639 handle_command_reply: tid=0 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 client.6639 shutdown 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 msgr2=0x5612bb8369a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f54500564a0 tx=0x7f5450057db0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 shutdown_connections 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:6835/693307152,v1:172.21.15.189:6837/693307152] conn(0x5612bb7feb80 0x5612bb814990 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612bb7fb110 0x5612bb814450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5612bb7fa680 0x5612bb83a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f5449600640 0 client.6639 ms_handle_reset on v2:172.21.15.189:6835/693307152 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 --2- 172.21.15.28:0/367698994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5612bb7d3c60 0x5612bb8369a0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 >> 172.21.15.28:0/367698994 conn(0x5612bb7d1420 msgr2=0x5612bb7f7900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 shutdown_connections 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 1 -- 172.21.15.28:0/367698994 wait complete. 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.045+0000 7f546a0aa740 20 client.6639 trim_cache size 0 max 16384 2024-09-13T17:12:17.048 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:17.049 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-09-13T17:12:17.049 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:12:17.049 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:17.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 msgr2=0x7f544006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 0x7f544006d0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f54600b30e0 tx=0x7f544c009040 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 msgr2=0x7f54601502b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f545000c710 tx=0x7f545000cbd0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 shutdown_connections 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f544006ac00 0x7f544006d0c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5460159770 0x7f5460152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5460070430 0x7f54601518c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 --2- 172.21.15.28:0/2969112433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f546006f3f0 0x7f54601502b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 >> 172.21.15.28:0/2969112433 conn(0x7f54600b0270 msgr2=0x7f54600af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 shutdown_connections 2024-09-13T17:12:17.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.046+0000 7f5467200640 1 -- 172.21.15.28:0/2969112433 wait complete. 2024-09-13T17:12:17.063 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:12:17.063 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:17.063 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:12:17.064 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:17.064 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:12:17.064 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:17.064 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:12:17.064 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:17.077 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:12:17.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-13T17:12:17.078 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:12:17.079 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-13T17:12:17.080 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:12:17.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-13T17:12:17.082 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:12:17.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-13T17:12:17.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.282+0000 7f38ae600640 1 Processor -- start 2024-09-13T17:12:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.282+0000 7f2e4aa00640 1 Processor -- start 2024-09-13T17:12:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.282+0000 7f38ae600640 1 -- start start 2024-09-13T17:12:17.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.283+0000 7f2e4aa00640 1 -- start start 2024-09-13T17:12:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ae600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 0x7f38a80ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ae600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38a80b3ad0 0x7f38a80b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ae600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38a815b110 con 0x7f38a80b51b0 2024-09-13T17:12:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ae600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a80b3680 con 0x7f38a815abc0 2024-09-13T17:12:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ae600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38a815b3f0 con 0x7f38a80b3ad0 2024-09-13T17:12:17.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ad200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f38a80b51b0 0x7f38a80b5590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54288/0 (socket says 172.21.15.28:54288) 2024-09-13T17:12:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38ad200640 1 -- 172.21.15.28:0/1032151609 learned_addr learned my addr 172.21.15.28:0/1032151609 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1032151609 >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 0x7f38a80ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e4aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e440cd710 0x7f2e440cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e4aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e440cbc20 0x7f2e440b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e4aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e440b1a70 con 0x7f2e44143f10 2024-09-13T17:12:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e4aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e440cc0b0 con 0x7f2e440cbc20 2024-09-13T17:12:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e4aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440cdb50 con 0x7f2e440cd710 2024-09-13T17:12:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e49600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2e44143f10 0x7f2e4406d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54292/0 (socket says 172.21.15.28:54292) 2024-09-13T17:12:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f2e49600640 1 -- 172.21.15.28:0/1767172090 learned_addr learned my addr 172.21.15.28:0/1767172090 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.285+0000 7f38adc00640 1 --2- 172.21.15.28:0/1032151609 >> v2:172.21.15.189:3300/0 conn(0x7f38a80b3ad0 0x7f38a80b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1767172090 >> v2:172.21.15.189:3300/0 conn(0x7f2e440cbc20 0x7f2e440b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38a7400640 1 -- 172.21.15.28:0/1032151609 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971565561 0 0) 0x7f38a815b110 con 0x7f38a80b51b0 2024-09-13T17:12:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38a7400640 1 -- 172.21.15.28:0/1032151609 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3890003520 con 0x7f38a80b51b0 2024-09-13T17:12:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f2e48c00640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e440cd710 0x7f2e440cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38adc00640 1 -- 172.21.15.28:0/1032151609 >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 msgr2=0x7f38a80ccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38adc00640 1 --2- 172.21.15.28:0/1032151609 >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 0x7f38a80ccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38adc00640 1 -- 172.21.15.28:0/1032151609 >> v1:172.21.15.28:6789/0 conn(0x7f38a80b51b0 legacy=0x7f38a80b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38adc00640 1 -- 172.21.15.28:0/1032151609 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a815b6d0 con 0x7f38a80b3ad0 2024-09-13T17:12:17.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f2e43e00640 1 -- 172.21.15.28:0/1767172090 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3347881069 0 0) 0x7f2e440b1a70 con 0x7f2e44143f10 2024-09-13T17:12:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1032151609 >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 0x7f38a80ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f2e43e00640 1 -- 172.21.15.28:0/1767172090 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e2c003520 con 0x7f2e44143f10 2024-09-13T17:12:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.286+0000 7f2e4a000640 1 -- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e440cd710 msgr2=0x7f2e440cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.287+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e440cd710 0x7f2e440cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.287+0000 7f2e4a000640 1 -- 172.21.15.28:0/1767172090 >> v1:172.21.15.28:6789/0 conn(0x7f2e44143f10 legacy=0x7f2e4406d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.287+0000 7f2e4a000640 1 -- 172.21.15.28:0/1767172090 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e440ce370 con 0x7f2e440cbc20 2024-09-13T17:12:17.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.287+0000 7f575e600640 1 Processor -- start 2024-09-13T17:12:17.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.287+0000 7f575e600640 1 -- start start 2024-09-13T17:12:17.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f38adc00640 1 --2- 172.21.15.28:0/1032151609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b3ad0 0x7f38a80b3f60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f38a80b0140 tx=0x7f38a002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdb85a38ad780f46 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e440b1f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009d90 tx=0x7f2e3c02a530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cadab1fa8c63cea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f38a7400640 1 -- 172.21.15.28:0/1032151609 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a002aea0 con 0x7f38a80b3ad0 2024-09-13T17:12:17.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f38a7400640 1 -- 172.21.15.28:0/1032151609 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a002d050 con 0x7f38a80b3ad0 2024-09-13T17:12:17.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f38a7400640 1 -- 172.21.15.28:0/1032151609 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0007940 con 0x7f38a80b3ad0 2024-09-13T17:12:17.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.288+0000 7f2e43e00640 1 -- 172.21.15.28:0/1767172090 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c007740 con 0x7f2e440cbc20 2024-09-13T17:12:17.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f2e43e00640 1 -- 172.21.15.28:0/1767172090 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c02d050 con 0x7f2e440cbc20 2024-09-13T17:12:17.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f2e43e00640 1 -- 172.21.15.28:0/1767172090 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c007ac0 con 0x7f2e440cbc20 2024-09-13T17:12:17.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 -- 172.21.15.28:0/1032151609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b3ad0 msgr2=0x7f38a80b3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 --2- 172.21.15.28:0/1032151609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b3ad0 0x7f38a80b3f60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f38a80b0140 tx=0x7f38a002a680 comp rx=0 tx=0).stop 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 msgr2=0x7f2e440b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e440b1f00 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e3c009d90 tx=0x7f2e3c02a530 comp rx=0 tx=0).stop 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 -- 172.21.15.28:0/1032151609 shutdown_connections 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 --2- 172.21.15.28:0/1032151609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b3ad0 0x7f38a80b3f60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 --2- 172.21.15.28:0/1032151609 >> v2:172.21.15.28:3300/0 conn(0x7f38a815abc0 0x7f38a80ccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.289+0000 7f38ae600640 1 -- 172.21.15.28:0/1032151609 >> 172.21.15.28:0/1032151609 conn(0x7f38a806d570 msgr2=0x7f38a806d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 0x7f575806c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580cdf30 con 0x7f575806f8f0 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57580b3610 con 0x7f57580cdb50 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575806cba0 con 0x7f575806c1d0 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f38ae600640 1 -- 172.21.15.28:0/1032151609 shutdown_connections 2024-09-13T17:12:17.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f38ae600640 1 -- 172.21.15.28:0/1032151609 wait complete. 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1767172090 shutdown_connections 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e440b1f00 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1767172090 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e440cd710 0x7f2e440cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1767172090 >> 172.21.15.28:0/1767172090 conn(0x7f2e440b0310 msgr2=0x7f2e440b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575d200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f57580cdb50 0x7f57580b1610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:40936/0 (socket says 172.21.15.28:40936) 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575d200640 1 -- 172.21.15.28:0/2066350953 learned_addr learned my addr 172.21.15.28:0/2066350953 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:17.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575dc00640 1 --2- 172.21.15.28:0/2066350953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 0x7f575806c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1767172090 shutdown_connections 2024-09-13T17:12:17.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1767172090 wait complete. 2024-09-13T17:12:17.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f5757400640 1 -- 172.21.15.28:0/2066350953 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394527354 0 0) 0x7f57580b3610 con 0x7f57580cdb50 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f5757400640 1 -- 172.21.15.28:0/2066350953 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5740003520 con 0x7f57580cdb50 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575dc00640 1 -- 172.21.15.28:0/2066350953 >> v1:172.21.15.189:6789/0 conn(0x7f57580cdb50 legacy=0x7f57580b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.290+0000 7f575dc00640 1 -- 172.21.15.28:0/2066350953 >> v1:172.21.15.28:6789/0 conn(0x7f575806f8f0 legacy=0x7f57580b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.291+0000 7f575dc00640 1 -- 172.21.15.28:0/2066350953 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57580b1e60 con 0x7f575806c1d0 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.292+0000 7f575dc00640 1 --2- 172.21.15.28:0/2066350953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 0x7f575806c660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5750009dc0 tx=0x7f575002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b7629192731b4c2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f38ae600640 1 Processor -- start 2024-09-13T17:12:17.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f2e4aa00640 1 Processor -- start 2024-09-13T17:12:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f5757400640 1 -- 172.21.15.28:0/2066350953 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750007740 con 0x7f575806c1d0 2024-09-13T17:12:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f5757400640 1 -- 172.21.15.28:0/2066350953 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575002d050 con 0x7f575806c1d0 2024-09-13T17:12:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f5757400640 1 -- 172.21.15.28:0/2066350953 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5750007ac0 con 0x7f575806c1d0 2024-09-13T17:12:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f2e4aa00640 1 -- start start 2024-09-13T17:12:17.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.293+0000 7f38ae600640 1 -- start start 2024-09-13T17:12:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 -- 172.21.15.28:0/2066350953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 msgr2=0x7f575806c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 --2- 172.21.15.28:0/2066350953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 0x7f575806c660 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5750009dc0 tx=0x7f575002a680 comp rx=0 tx=0).stop 2024-09-13T17:12:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 -- 172.21.15.28:0/2066350953 shutdown_connections 2024-09-13T17:12:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 --2- 172.21.15.28:0/2066350953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f575806c1d0 0x7f575806c660 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 -- 172.21.15.28:0/2066350953 >> 172.21.15.28:0/2066350953 conn(0x7f57580b0310 msgr2=0x7f57580b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f32cc600640 1 Processor -- start 2024-09-13T17:12:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 -- 172.21.15.28:0/2066350953 shutdown_connections 2024-09-13T17:12:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f575e600640 1 -- 172.21.15.28:0/2066350953 wait complete. 2024-09-13T17:12:17.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f32cc600640 1 -- start start 2024-09-13T17:12:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 0x7f2e44164230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 0x7f2e440a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e44162980 con 0x7f2e440cd710 2024-09-13T17:12:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440a85e0 con 0x7f2e440cbc20 2024-09-13T17:12:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f2e4aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e440a8880 con 0x7f2e44143f10 2024-09-13T17:12:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e49600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.294+0000 7f38ae600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 0x7f38a80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ae600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 0x7f38a81554c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ae600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 0x7f38a814eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ae600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a8149d30 con 0x7f38a815abc0 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ae600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38a8149440 con 0x7f38a80b51b0 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ae600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38a80b1c20 con 0x7f38a80b3ad0 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e49600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:42618/0 (socket says 172.21.15.28:42618) 2024-09-13T17:12:17.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e49600640 1 -- 172.21.15.28:0/1895083535 learned_addr learned my addr 172.21.15.28:0/1895083535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ad200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 0x7f38a80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ad200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 0x7f38a80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:43344/0 (socket says 172.21.15.28:43344) 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38ad200640 1 -- 172.21.15.28:0/1634038376 learned_addr learned my addr 172.21.15.28:0/1634038376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 0x7f2e440a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e48c00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 0x7f2e44164230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 0x7f38a81554c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f38adc00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 0x7f38a814eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.295+0000 7f2e4a000640 1 -- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 msgr2=0x7f2e44163cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e4a000640 1 -- 172.21.15.28:0/1895083535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 msgr2=0x7f2e44164230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 -- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 msgr2=0x7f38a80b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 0x7f38a80b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 -- 172.21.15.28:0/1634038376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 msgr2=0x7f38a814eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 0x7f38a814eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 -- 172.21.15.28:0/1634038376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a0009a70 con 0x7f38a80b51b0 2024-09-13T17:12:17.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38adc00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 0x7f38a814eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 0x7f2e44164230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e4a000640 1 -- 172.21.15.28:0/1895083535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e3c009a70 con 0x7f2e44143f10 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e49600640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e48c00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 0x7f2e44164230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f38a7e00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 0x7f38a81554c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f389400d5c0 tx=0x7f389400da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f575e600640 1 Processor -- start 2024-09-13T17:12:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f2e4a000640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 0x7f2e440a9360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e44073a00 tx=0x7f2e3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.296+0000 7f575e600640 1 -- start start 2024-09-13T17:12:17.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894004030 con 0x7f38a80b51b0 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3894004190 con 0x7f38a80b51b0 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3894004520 con 0x7f38a80b51b0 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32cc600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 0x7f32c40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32cc600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c41610a0 con 0x7f32c40b17e0 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32cc600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c40b2620 con 0x7f32c406d070 2024-09-13T17:12:17.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32cc600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c40b2900 con 0x7f32c406c540 2024-09-13T17:12:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a81496e0 con 0x7f38a80b51b0 2024-09-13T17:12:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c03c830 con 0x7f2e44143f10 2024-09-13T17:12:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a8152cb0 con 0x7f38a80b51b0 2024-09-13T17:12:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32c9c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32c40b17e0 0x7f32c40b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54298/0 (socket says 172.21.15.28:54298) 2024-09-13T17:12:17.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32c9c00640 1 -- 172.21.15.28:0/3717136650 learned_addr learned my addr 172.21.15.28:0/3717136650 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f32ca600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 0x7f32c40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3c03c9b0 con 0x7f2e44143f10 2024-09-13T17:12:17.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3c03cd60 con 0x7f2e44143f10 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e440a8b60 con 0x7f2e44143f10 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.297+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e44165450 con 0x7f2e44143f10 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f32ca600640 1 -- 172.21.15.28:0/3717136650 >> v1:172.21.15.189:6789/0 conn(0x7f32c406d070 legacy=0x7f32c41687d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f32ca600640 1 -- 172.21.15.28:0/3717136650 >> v1:172.21.15.28:6789/0 conn(0x7f32c40b17e0 legacy=0x7f32c40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f32ca600640 1 -- 172.21.15.28:0/3717136650 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c40b2be0 con 0x7f32c406c540 2024-09-13T17:12:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e440a9080 con 0x7f2e44143f10 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575806f8f0 0x7f57581522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57580cdb50 0x7f5758152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57581f6aa0 con 0x7f575806c1d0 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5758164f60 con 0x7f575806f8f0 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57580a89b0 con 0x7f57580cdb50 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f5757e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575806f8f0 0x7f57581522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32854/0 (socket says 172.21.15.28:32854) 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.298+0000 7f575d200640 1 -- 172.21.15.28:0/2883561578 learned_addr learned my addr 172.21.15.28:0/2883561578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575dc00640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57580cdb50 0x7f5758152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575d200640 1 -- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57580cdb50 msgr2=0x7f5758152820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575d200640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57580cdb50 0x7f5758152820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575d200640 1 -- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575806f8f0 msgr2=0x7f57581522e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575d200640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575806f8f0 0x7f57581522e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f575d200640 1 -- 172.21.15.28:0/2883561578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750009a70 con 0x7f575806c1d0 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.299+0000 7f32ca600640 1 --2- 172.21.15.28:0/3717136650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 0x7f32c40b3480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009dc0 tx=0x7f32bc02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=464b14033cf2a091 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f575d200640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c460 tx=0x7f574c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8149ea0 con 0x7f38a80b51b0 2024-09-13T17:12:17.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c005ce0 con 0x7f575806c1d0 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574c005e40 con 0x7f575806c1d0 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574c007290 con 0x7f575806c1d0 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5758164c80 con 0x7f575806c1d0 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57580a8f30 con 0x7f575806c1d0 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2e3c02ad50 con 0x7f2e44143f10 2024-09-13T17:12:17.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.300+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3894014620 con 0x7f38a80b51b0 2024-09-13T17:12:17.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.301+0000 7f38a4c00640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 0x7f388806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.301+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57580a9120 con 0x7f575806c1d0 2024-09-13T17:12:17.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.302+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3894089800 con 0x7f38a80b51b0 2024-09-13T17:12:17.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.302+0000 7f2e41600640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 0x7f2e2406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.302+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e3c0aae50 con 0x7f2e44143f10 2024-09-13T17:12:17.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.302+0000 7f2e49600640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 0x7f2e2406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.302+0000 7f38ad200640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 0x7f388806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.303+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f574c0073f0 con 0x7f575806c1d0 2024-09-13T17:12:17.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.303+0000 7f2e49600640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 0x7f2e2406d110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2e380011e0 tx=0x7f2e38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.303+0000 7f38ad200640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 0x7f388806d1c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f389c005880 tx=0x7f389c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.303+0000 7f5754c00640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 0x7f573806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.303+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f574c088950 con 0x7f575806c1d0 2024-09-13T17:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.304+0000 7f5757e00640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 0x7f573806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.304+0000 7f5757e00640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 0x7f573806d0c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57580703f0 tx=0x7f5744007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.308+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f574c053ab0 con 0x7f575806c1d0 2024-09-13T17:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.308+0000 7f32c3e00640 1 -- 172.21.15.28:0/3717136650 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc007740 con 0x7f32c406c540 2024-09-13T17:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.308+0000 7f32c3e00640 1 -- 172.21.15.28:0/3717136650 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc02d050 con 0x7f32c406c540 2024-09-13T17:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.308+0000 7f32c3e00640 1 -- 172.21.15.28:0/3717136650 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc007ac0 con 0x7f32c406c540 2024-09-13T17:12:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.308+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2e3c0761e0 con 0x7f2e44143f10 2024-09-13T17:12:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.309+0000 7f32cc600640 1 -- 172.21.15.28:0/3717136650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 msgr2=0x7f32c40b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.309+0000 7f32cc600640 1 --2- 172.21.15.28:0/3717136650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 0x7f32c40b3480 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f32bc009dc0 tx=0x7f32bc02a680 comp rx=0 tx=0).stop 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.309+0000 7f32cc600640 1 -- 172.21.15.28:0/3717136650 shutdown_connections 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.309+0000 7f32cc600640 1 --2- 172.21.15.28:0/3717136650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406c540 0x7f32c40b3480 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.309+0000 7f32cc600640 1 -- 172.21.15.28:0/3717136650 >> 172.21.15.28:0/3717136650 conn(0x7f32c40b0270 msgr2=0x7f32c40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.310+0000 7f32cc600640 1 -- 172.21.15.28:0/3717136650 shutdown_connections 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.310+0000 7f32cc600640 1 -- 172.21.15.28:0/3717136650 wait complete. 2024-09-13T17:12:17.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.310+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3894054a50 con 0x7f38a80b51b0 2024-09-13T17:12:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.313+0000 7f32cc600640 1 Processor -- start 2024-09-13T17:12:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.313+0000 7f32cc600640 1 -- start start 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.314+0000 7f32cc600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c406c540 0x7f32c41e9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.314+0000 7f32cc600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32cc600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c40b17e0 0x7f32c415d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32cc600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c41ea260 con 0x7f32c40b17e0 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32cc600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c41e8cd0 con 0x7f32c406c540 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32cc600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c41e9650 con 0x7f32c406d070 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32c9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32ca600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c40b17e0 0x7f32c415d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32c9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:43372/0 (socket says 172.21.15.28:43372) 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32c9200640 1 -- 172.21.15.28:0/45900066 learned_addr learned my addr 172.21.15.28:0/45900066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.315+0000 7f32c9c00640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c406c540 0x7f32c41e9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 -- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c406c540 msgr2=0x7f32c41e9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c406c540 0x7f32c41e9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 -- 172.21.15.28:0/45900066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c40b17e0 msgr2=0x7f32c415d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c40b17e0 0x7f32c415d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 -- 172.21.15.28:0/45900066 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32bc009a70 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c9200640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b3f0 tx=0x7f32b000b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0011830 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0011990 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.316+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0011d40 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.317+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c41e9930 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.317+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c41e9ef0 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.318+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c41ea3d0 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.319+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f32b001a400 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.320+0000 7f32c1600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 0x7f32a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.320+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32b0088fe0 con 0x7f32c406d070 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.320+0000 7f32c9c00640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 0x7f32a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.321+0000 7f32c9c00640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 0x7f32a406d0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005880 tx=0x7f32b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.324+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32b00542a0 con 0x7f32c406d070 2024-09-13T17:12:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.559+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f575806c660 con 0x7f575806c1d0 2024-09-13T17:12:17.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.562+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7f38a80ce720 con 0x7f38a80b51b0 2024-09-13T17:12:17.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.563+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f2e440a9080 con 0x7f2e44143f10 2024-09-13T17:12:17.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.575+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7f32c40b3480 con 0x7f32c406d070 2024-09-13T17:12:17.579 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:12:17.577+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:12:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.585+0000 7f5754c00640 1 -- 172.21.15.28:0/2883561578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6417 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7f574c017070 con 0x7f575806c1d0 2024-09-13T17:12:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6417 2024-09-13T17:12:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.586+0000 7f2e41600640 1 -- 172.21.15.28:0/1895083535 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6145 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2e440a9080 con 0x7f2e44143f10 2024-09-13T17:12:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.586+0000 7f32c1600640 1 -- 172.21.15.28:0/45900066 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6420 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7f32b00578c0 con 0x7f32c406d070 2024-09-13T17:12:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6420 2024-09-13T17:12:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.586+0000 7f38a4c00640 1 -- 172.21.15.28:0/1634038376 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6414 v50) v1 ==== 95+0+0 (secure 0 0 0) 0x7f3894058070 con 0x7f38a80b51b0 2024-09-13T17:12:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6145 2024-09-13T17:12:17.589 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6414 2024-09-13T17:12:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 msgr2=0x7f32a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 msgr2=0x7f388806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 0x7f388806d1c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f389c005880 tx=0x7f389c009040 comp rx=0 tx=0).stop 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 msgr2=0x7f38a81554c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 0x7f38a81554c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f389400d5c0 tx=0x7f389400da80 comp rx=0 tx=0).stop 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 msgr2=0x7f573806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 0x7f573806d0c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f57580703f0 tx=0x7f5744007e30 comp rx=0 tx=0).stop 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 msgr2=0x7f57580ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c460 tx=0x7f574c00c920 comp rx=0 tx=0).stop 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 shutdown_connections 2024-09-13T17:12:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 0x7f32a406d0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005880 tx=0x7f32b8009040 comp rx=0 tx=0).stop 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 msgr2=0x7f32c41e8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f32b000b3f0 tx=0x7f32b000b8b0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 shutdown_connections 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f388806ad00 0x7f388806d1c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38a815abc0 0x7f38a814eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a80b51b0 0x7f38a81554c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 --2- 172.21.15.28:0/1634038376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38a80b3ad0 0x7f38a80b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 >> 172.21.15.28:0/1634038376 conn(0x7f38a806d570 msgr2=0x7f38a8158760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f573806ac00 0x7f573806d0c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57580cdb50 0x7f5758152820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f575806f8f0 0x7f57581522e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 --2- 172.21.15.28:0/2883561578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f575806c1d0 0x7f57580ab0c0 secure :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f574c00c460 tx=0x7f574c00c920 comp rx=0 tx=0).stop 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.588+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 >> 172.21.15.28:0/2883561578 conn(0x7f57580b0310 msgr2=0x7f57580af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 shutdown_connections 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 shutdown_connections 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32a406ac00 0x7f32a406d0c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 shutdown_connections 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c40b17e0 0x7f32c415d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c406d070 0x7f32c41e8790 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 --2- 172.21.15.28:0/45900066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c406c540 0x7f32c41e9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 >> 172.21.15.28:0/45900066 conn(0x7f32c40b0270 msgr2=0x7f32c40aea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f575e600640 1 -- 172.21.15.28:0/2883561578 wait complete. 2024-09-13T17:12:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 shutdown_connections 2024-09-13T17:12:17.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f32cc600640 1 -- 172.21.15.28:0/45900066 wait complete. 2024-09-13T17:12:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f38ae600640 1 -- 172.21.15.28:0/1634038376 wait complete. 2024-09-13T17:12:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 msgr2=0x7f2e2406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 0x7f2e2406d110 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2e380011e0 tx=0x7f2e38009040 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 msgr2=0x7f2e440a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 0x7f2e440a9360 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2e44073a00 tx=0x7f2e3c004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 shutdown_connections 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e2406ac50 0x7f2e2406d110 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e44143f10 0x7f2e440a9360 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e440cd710 0x7f2e44164230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 --2- 172.21.15.28:0/1895083535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e440cbc20 0x7f2e44163cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 >> 172.21.15.28:0/1895083535 conn(0x7f2e440b0310 msgr2=0x7f2e4415cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.589+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 shutdown_connections 2024-09-13T17:12:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.590+0000 7f2e4aa00640 1 -- 172.21.15.28:0/1895083535 wait complete. 2024-09-13T17:12:17.610 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:12:17.610 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:12:17.611 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:12:17.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:12:17.615 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:12:17.615 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:17.615 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:12:17.616 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:17.617 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:12:17.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:12:17.618 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:12:17.618 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:12:17.621 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:12:17.622 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:17.661 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:12:17.661 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:17.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:17.744 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:12:17.748 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:12:17.751 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:12:17.792 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:12:17.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.821+0000 7fc723e00640 1 Processor -- start 2024-09-13T17:12:17.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.821+0000 7fc723e00640 1 -- start start 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc723e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 0x7fc7240d14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc723e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240b5870 con 0x7fc7240cd710 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc723e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240d2560 con 0x7fc7240b48d0 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc723e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415cc10 con 0x7fc72415c830 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc722000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc7240cd710 0x7fc7240b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54300/0 (socket says 172.21.15.28:54300) 2024-09-13T17:12:17.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc722000640 1 -- 172.21.15.28:0/486586702 learned_addr learned my addr 172.21.15.28:0/486586702 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:17.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.822+0000 7fc722a00640 1 --2- 172.21.15.28:0/486586702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 0x7fc7240d14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.823+0000 7fc721600640 1 -- 172.21.15.28:0/486586702 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132871021 0 0) 0x7fc7240b5870 con 0x7fc7240cd710 2024-09-13T17:12:17.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.823+0000 7fc722a00640 1 -- 172.21.15.28:0/486586702 >> v1:172.21.15.189:6789/0 conn(0x7fc7240b48d0 legacy=0x7fc7240b5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.823+0000 7fc722a00640 1 -- 172.21.15.28:0/486586702 >> v1:172.21.15.28:6789/0 conn(0x7fc7240cd710 legacy=0x7fc7240b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.823+0000 7fc722a00640 1 -- 172.21.15.28:0/486586702 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240d19f0 con 0x7fc72415c830 2024-09-13T17:12:17.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.824+0000 7fc722a00640 1 --2- 172.21.15.28:0/486586702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 0x7fc7240d14b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc714009750 tx=0x7fc71402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6777d5bf3b063896 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.824+0000 7fc721600640 1 -- 172.21.15.28:0/486586702 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71402adb0 con 0x7fc72415c830 2024-09-13T17:12:17.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.824+0000 7fc721600640 1 -- 172.21.15.28:0/486586702 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc714032440 con 0x7fc72415c830 2024-09-13T17:12:17.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.824+0000 7fc721600640 1 -- 172.21.15.28:0/486586702 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7140327d0 con 0x7fc72415c830 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 -- 172.21.15.28:0/486586702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 msgr2=0x7fc7240d14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 --2- 172.21.15.28:0/486586702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 0x7fc7240d14b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc714009750 tx=0x7fc71402a0a0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 -- 172.21.15.28:0/486586702 shutdown_connections 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 --2- 172.21.15.28:0/486586702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc72415c830 0x7fc7240d14b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 -- 172.21.15.28:0/486586702 >> 172.21.15.28:0/486586702 conn(0x7fc7240cf3a0 msgr2=0x7fc7240cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 -- 172.21.15.28:0/486586702 shutdown_connections 2024-09-13T17:12:17.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.825+0000 7fc723e00640 1 -- 172.21.15.28:0/486586702 wait complete. 2024-09-13T17:12:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.826+0000 7fc723e00640 1 Processor -- start 2024-09-13T17:12:17.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.827+0000 7fc723e00640 1 -- start start 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.827+0000 7fc723e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.827+0000 7fc723e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7240cd710 0x7fc7240c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc723e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72415c830 0x7fc7240c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc723e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415f670 con 0x7fc7240b48d0 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc723e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7240c7b00 con 0x7fc72415c830 2024-09-13T17:12:17.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc723e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415cfb0 con 0x7fc7240cd710 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32886/0 (socket says 172.21.15.28:32886) 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 -- 172.21.15.28:0/444307262 learned_addr learned my addr 172.21.15.28:0/444307262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722000640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7240cd710 0x7fc7240c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc723400640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72415c830 0x7fc7240c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 -- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7240cd710 msgr2=0x7fc7240c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7240cd710 0x7fc7240c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 -- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72415c830 msgr2=0x7fc7240c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72415c830 0x7fc7240c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.828+0000 7fc722a00640 1 -- 172.21.15.28:0/444307262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714009400 con 0x7fc7240b48d0 2024-09-13T17:12:17.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc722a00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc714009870 tx=0x7fc7140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714004230 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc714004390 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72415d290 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714003850 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.829+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72415d850 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.830+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72415da70 con 0x7fc7240b48d0 2024-09-13T17:12:17.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.831+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7140039b0 con 0x7fc7240b48d0 2024-09-13T17:12:17.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.832+0000 7fc71b400640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 0x7fc70406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:17.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.832+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc7140aa080 con 0x7fc7240b48d0 2024-09-13T17:12:17.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.833+0000 7fc722000640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 0x7fc70406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:17.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.833+0000 7fc722000640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 0x7fc70406d0c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc710007fd0 tx=0x7fc710009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:17.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:17.837+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc7140dd9c0 con 0x7fc7240b48d0 2024-09-13T17:12:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.049+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7240d14b0 con 0x7fc7240b48d0 2024-09-13T17:12:18.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.050+0000 7fc71b400640 1 -- 172.21.15.28:0/444307262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+1699 (secure 0 0 0) 0x7fc714037090 con 0x7fc7240b48d0 2024-09-13T17:12:18.052 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:18.052 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":50,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":50,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:17.578735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:18.052 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 50 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 msgr2=0x7fc70406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 0x7fc70406d0c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc710007fd0 tx=0x7fc710009040 comp rx=0 tx=0).stop 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 msgr2=0x7fc72414cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc714009870 tx=0x7fc7140046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 shutdown_connections 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70406ac00 0x7fc70406d0c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:18.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc72415c830 0x7fc7240c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7240cd710 0x7fc7240c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 --2- 172.21.15.28:0/444307262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7240b48d0 0x7fc72414cb60 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 >> 172.21.15.28:0/444307262 conn(0x7fc7240cf3a0 msgr2=0x7fc7240b3740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 shutdown_connections 2024-09-13T17:12:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:18.052+0000 7fc723e00640 1 -- 172.21.15.28:0/444307262 wait complete. 2024-09-13T17:12:18.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 50, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:04.639307+0000', 'modified': '2024-09-13T17:12:17.578735+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 65, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {}, 'failed': [0], 'damaged': [], 'stopped': [], 'info': {}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:12:18.473 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 0/1 2024-09-13T17:12:19.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.641+0000 7f0d7f200640 1 Processor -- start 2024-09-13T17:12:19.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.641+0000 7f0d7f200640 1 -- start start 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d80159820 0x7f0d800b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 0x7f0d800b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d800c3850 con 0x7f0d80159820 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8015a930 con 0x7f0d800b22e0 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d7f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800b2cd0 con 0x7f0d80159440 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d77e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d80159820 0x7f0d800b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d77e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d80159820 0x7f0d800b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32900/0 (socket says 172.21.15.28:32900) 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.642+0000 7f0d77e00640 1 -- 172.21.15.28:0/1492397996 learned_addr learned my addr 172.21.15.28:0/1492397996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d7de00640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d7e800640 1 --2- 172.21.15.28:0/1492397996 >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 0x7f0d800b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d77e00640 1 -- 172.21.15.28:0/1492397996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 msgr2=0x7f0d800b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d77e00640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d77e00640 1 -- 172.21.15.28:0/1492397996 >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 msgr2=0x7f0d800b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d77e00640 1 --2- 172.21.15.28:0/1492397996 >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 0x7f0d800b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d77e00640 1 -- 172.21.15.28:0/1492397996 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80159d20 con 0x7f0d80159820 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d7de00640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:19.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.643+0000 7f0d7e800640 1 --2- 172.21.15.28:0/1492397996 >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 0x7f0d800b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.644+0000 7f0d77e00640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d800b0980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0d8006d2d0 tx=0x7f0d6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572eab36951bc147 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.644+0000 7f0d7d400640 1 -- 172.21.15.28:0/1492397996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68032440 con 0x7f0d80159820 2024-09-13T17:12:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.644+0000 7f0d7d400640 1 -- 172.21.15.28:0/1492397996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d680325a0 con 0x7f0d80159820 2024-09-13T17:12:19.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.644+0000 7f0d7d400640 1 -- 172.21.15.28:0/1492397996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d68032930 con 0x7f0d80159820 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 -- 172.21.15.28:0/1492397996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 msgr2=0x7f0d800b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d800b0980 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f0d8006d2d0 tx=0x7f0d6802a270 comp rx=0 tx=0).stop 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 -- 172.21.15.28:0/1492397996 shutdown_connections 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d800b0980 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 --2- 172.21.15.28:0/1492397996 >> v2:172.21.15.189:3300/0 conn(0x7f0d800b22e0 0x7f0d800b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 --2- 172.21.15.28:0/1492397996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 -- 172.21.15.28:0/1492397996 >> 172.21.15.28:0/1492397996 conn(0x7f0d80072e40 msgr2=0x7f0d80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 -- 172.21.15.28:0/1492397996 shutdown_connections 2024-09-13T17:12:19.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.645+0000 7f0d7f200640 1 -- 172.21.15.28:0/1492397996 wait complete. 2024-09-13T17:12:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.647+0000 7f0d7f200640 1 Processor -- start 2024-09-13T17:12:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.647+0000 7f0d7f200640 1 -- start start 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d800b22e0 0x7f0d800c44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800ade20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800ae360 con 0x7f0d80159820 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800af680 con 0x7f0d800b22e0 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d800ae5e0 con 0x7f0d80159440 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32904/0 (socket says 172.21.15.28:32904) 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7e800640 1 -- 172.21.15.28:0/4113741881 learned_addr learned my addr 172.21.15.28:0/4113741881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7de00640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d800b22e0 0x7f0d800c44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d77e00640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800ade20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7e800640 1 -- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 msgr2=0x7f0d800ade20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.648+0000 7f0d7e800640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800ade20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7e800640 1 -- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d800b22e0 msgr2=0x7f0d800c44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7e800640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d800b22e0 0x7f0d800c44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7e800640 1 -- 172.21.15.28:0/4113741881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d680094d0 con 0x7f0d80159820 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7e800640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0d78007d60 tx=0x7f0d7800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78003990 con 0x7f0d80159820 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d7800f040 con 0x7f0d80159820 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78003d30 con 0x7f0d80159820 2024-09-13T17:12:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d800ae8c0 con 0x7f0d80159820 2024-09-13T17:12:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.649+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d800aee80 con 0x7f0d80159820 2024-09-13T17:12:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.650+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8015aaa0 con 0x7f0d80159820 2024-09-13T17:12:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.651+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d78004030 con 0x7f0d80159820 2024-09-13T17:12:19.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.652+0000 7f0d76000640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 0x7f0d5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:19.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.652+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0d780895c0 con 0x7f0d80159820 2024-09-13T17:12:19.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.652+0000 7f0d7de00640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 0x7f0d5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:19.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.653+0000 7f0d7de00640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 0x7f0d5c06d0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0d80158db0 tx=0x7f0d70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:19.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.656+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d780547e0 con 0x7f0d80159820 2024-09-13T17:12:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.873+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d800b2790 con 0x7f0d80159820 2024-09-13T17:12:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.874+0000 7f0d76000640 1 -- 172.21.15.28:0/4113741881 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f0d78003e90 con 0x7f0d80159820 2024-09-13T17:12:19.876 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:19.876 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":53,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:19.591453+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:19.876 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 53 2024-09-13T17:12:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 msgr2=0x7f0d5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:19.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 0x7f0d5c06d0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0d80158db0 tx=0x7f0d70008040 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 msgr2=0x7f0d8014af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f0d78007d60 tx=0x7f0d7800d4a0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 shutdown_connections 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d5c06ac00 0x7f0d5c06d0c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d80159820 0x7f0d8014af10 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d80159440 0x7f0d800ade20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 --2- 172.21.15.28:0/4113741881 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d800b22e0 0x7f0d800c44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.876+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 >> 172.21.15.28:0/4113741881 conn(0x7f0d80072e40 msgr2=0x7f0d80070700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.877+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 shutdown_connections 2024-09-13T17:12:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:19.877+0000 7f0d7f200640 1 -- 172.21.15.28:0/4113741881 wait complete. 2024-09-13T17:12:20.296 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 53, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:04.639307+0000', 'modified': '2024-09-13T17:12:19.591453+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 65, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6684}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6684': {'gid': 6684, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.28:6835/2705534945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2705534945}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2705534945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:12:20.296 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_6684:up:reconnect 2024-09-13T17:12:21.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.463+0000 7f80a6e00640 1 Processor -- start 2024-09-13T17:12:21.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.463+0000 7f80a6e00640 1 -- start start 2024-09-13T17:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.464+0000 7f80a6e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80a01592a0 0x7f80a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.464+0000 7f80a6e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a0159d90 con 0x7f80a006f970 2024-09-13T17:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.464+0000 7f80a6e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80a00b2040 con 0x7f80a006c2d0 2024-09-13T17:12:21.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.464+0000 7f80a6e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80a015b0b0 con 0x7f80a01592a0 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80a01592a0 0x7f80a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80a006f970 0x7f80a0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54314/0 (socket says 172.21.15.28:54314) 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809f400640 1 -- 172.21.15.28:0/2228975568 learned_addr learned my addr 172.21.15.28:0/2228975568 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809fe00640 1 -- 172.21.15.28:0/2228975568 >> v1:172.21.15.189:6789/0 conn(0x7f80a006c2d0 legacy=0x7f80a00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809fe00640 1 -- 172.21.15.28:0/2228975568 >> v1:172.21.15.28:6789/0 conn(0x7f80a006f970 legacy=0x7f80a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809fe00640 1 -- 172.21.15.28:0/2228975568 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a0159ff0 con 0x7f80a01592a0 2024-09-13T17:12:21.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.465+0000 7f809ea00640 1 -- 172.21.15.28:0/2228975568 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697795575 0 0) 0x7f80a0159d90 con 0x7f80a006f970 2024-09-13T17:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.466+0000 7f809fe00640 1 --2- 172.21.15.28:0/2228975568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a01592a0 0x7f80a015ab70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f80900097a0 tx=0x7f809002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559eff019c74d95b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.466+0000 7f809ea00640 1 -- 172.21.15.28:0/2228975568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090032440 con 0x7f80a01592a0 2024-09-13T17:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.466+0000 7f809ea00640 1 -- 172.21.15.28:0/2228975568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80900325a0 con 0x7f80a01592a0 2024-09-13T17:12:21.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.466+0000 7f809ea00640 1 -- 172.21.15.28:0/2228975568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8090032930 con 0x7f80a01592a0 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 -- 172.21.15.28:0/2228975568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a01592a0 msgr2=0x7f80a015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 --2- 172.21.15.28:0/2228975568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a01592a0 0x7f80a015ab70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f80900097a0 tx=0x7f809002a230 comp rx=0 tx=0).stop 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 -- 172.21.15.28:0/2228975568 shutdown_connections 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 --2- 172.21.15.28:0/2228975568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a01592a0 0x7f80a015ab70 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 -- 172.21.15.28:0/2228975568 >> 172.21.15.28:0/2228975568 conn(0x7f80a0069e80 msgr2=0x7f80a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 -- 172.21.15.28:0/2228975568 shutdown_connections 2024-09-13T17:12:21.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.467+0000 7f80a6e00640 1 -- 172.21.15.28:0/2228975568 wait complete. 2024-09-13T17:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.469+0000 7f80a6e00640 1 Processor -- start 2024-09-13T17:12:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.469+0000 7f80a6e00640 1 -- start start 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80a006c2d0 0x7f80a014e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80a01592a0 0x7f80a0148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a0151320 con 0x7f80a006f970 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00af030 con 0x7f80a01592a0 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80a00afea0 con 0x7f80a006c2d0 2024-09-13T17:12:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f809f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f809f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32928/0 (socket says 172.21.15.28:32928) 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f809f400640 1 -- 172.21.15.28:0/4105642253 learned_addr learned my addr 172.21.15.28:0/4105642253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f809fe00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80a006c2d0 0x7f80a014e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.470+0000 7f80a4e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80a01592a0 0x7f80a0148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 -- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80a006c2d0 msgr2=0x7f80a014e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80a006c2d0 0x7f80a014e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 -- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80a01592a0 msgr2=0x7f80a0148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80a01592a0 0x7f80a0148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 -- 172.21.15.28:0/4105642253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8090009400 con 0x7f80a006f970 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f809f400640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f808c00c460 tx=0x7f808c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c005ce0 con 0x7f80a006f970 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c005e80 con 0x7f80a006f970 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0072b0 con 0x7f80a006f970 2024-09-13T17:12:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a00ae270 con 0x7f80a006f970 2024-09-13T17:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.471+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a00ae830 con 0x7f80a006f970 2024-09-13T17:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.472+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80a015ab70 con 0x7f80a006f970 2024-09-13T17:12:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.473+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f808c007410 con 0x7f80a006f970 2024-09-13T17:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.474+0000 7f8097e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 0x7f808006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.474+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f808c088f10 con 0x7f80a006f970 2024-09-13T17:12:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.474+0000 7f809fe00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 0x7f808006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.475+0000 7f809fe00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 0x7f808006d0f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8090034000 tx=0x7f809002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:21.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.478+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f808c054080 con 0x7f80a006f970 2024-09-13T17:12:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.694+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80a00b0e20 con 0x7f80a006f970 2024-09-13T17:12:21.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.694+0000 7f8097e00640 1 -- 172.21.15.28:0/4105642253 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f808c0576a0 con 0x7f80a006f970 2024-09-13T17:12:21.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:21.697 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:21.697 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:21.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 msgr2=0x7f808006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 0x7f808006d0f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8090034000 tx=0x7f809002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 msgr2=0x7f80a00af960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f808c00c460 tx=0x7f808c00c920 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 shutdown_connections 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f808006ac30 0x7f808006d0f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80a01592a0 0x7f80a0148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80a006f970 0x7f80a00af960 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 --2- 172.21.15.28:0/4105642253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80a006c2d0 0x7f80a014e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.696+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 >> 172.21.15.28:0/4105642253 conn(0x7f80a0069e80 msgr2=0x7f80a015b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.697+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 shutdown_connections 2024-09-13T17:12:21.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:21.697+0000 7f80a6e00640 1 -- 172.21.15.28:0/4105642253 wait complete. 2024-09-13T17:12:22.116 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:04.639307+0000', 'modified': '2024-09-13T17:12:21.595258+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 65, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6684}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6684': {'gid': 6684, 'name': 'c', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/2705534945', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 2705534945}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2705534945}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:12:22.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:12:22.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:12:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.275+0000 7f3d88000640 1 Processor -- start 2024-09-13T17:12:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.275+0000 7f3d88000640 1 -- start start 2024-09-13T17:12:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d88000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d8015a780 0x7f3d80070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d88000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3d8006c250 0x7f3d800712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d88000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d8006c660 con 0x7f3d8015ab60 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d88000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8006c8b0 con 0x7f3d8015a780 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d88000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d8006cb90 con 0x7f3d8006c250 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d85600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d8015a780 0x7f3d80070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d85600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d8015a780 0x7f3d80070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32938/0 (socket says 172.21.15.28:32938) 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.276+0000 7f3d85600640 1 -- 172.21.15.28:0/4117979496 learned_addr learned my addr 172.21.15.28:0/4117979496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.277+0000 7f3d86000640 1 --2- 172.21.15.28:0/4117979496 >> v2:172.21.15.189:3300/0 conn(0x7f3d8006c250 0x7f3d800712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.277+0000 7f3d85600640 1 -- 172.21.15.28:0/4117979496 >> v2:172.21.15.189:3300/0 conn(0x7f3d8006c250 msgr2=0x7f3d800712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.277+0000 7f3d85600640 1 --2- 172.21.15.28:0/4117979496 >> v2:172.21.15.189:3300/0 conn(0x7f3d8006c250 0x7f3d800712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.277+0000 7f3d85600640 1 -- 172.21.15.28:0/4117979496 >> v1:172.21.15.189:6789/0 conn(0x7f3d8015ab60 legacy=0x7f3d8006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.277+0000 7f3d85600640 1 -- 172.21.15.28:0/4117979496 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800cc980 con 0x7f3d8015a780 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.278+0000 7f3d85600640 1 --2- 172.21.15.28:0/4117979496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015a780 0x7f3d80070d80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3d800cc370 tx=0x7f3d7402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec4f68a8f1b881e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.278+0000 7f3d7fe00640 1 -- 172.21.15.28:0/4117979496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74032440 con 0x7f3d8015a780 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.278+0000 7f3d7fe00640 1 -- 172.21.15.28:0/4117979496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d740325a0 con 0x7f3d8015a780 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.278+0000 7f3d7fe00640 1 -- 172.21.15.28:0/4117979496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74032930 con 0x7f3d8015a780 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 -- 172.21.15.28:0/4117979496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015a780 msgr2=0x7f3d80070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 --2- 172.21.15.28:0/4117979496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015a780 0x7f3d80070d80 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f3d800cc370 tx=0x7f3d7402a230 comp rx=0 tx=0).stop 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 -- 172.21.15.28:0/4117979496 shutdown_connections 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 --2- 172.21.15.28:0/4117979496 >> v2:172.21.15.189:3300/0 conn(0x7f3d8006c250 0x7f3d800712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 --2- 172.21.15.28:0/4117979496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015a780 0x7f3d80070d80 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 -- 172.21.15.28:0/4117979496 >> 172.21.15.28:0/4117979496 conn(0x7f3d800b0270 msgr2=0x7f3d800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 -- 172.21.15.28:0/4117979496 shutdown_connections 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.279+0000 7f3d88000640 1 -- 172.21.15.28:0/4117979496 wait complete. 2024-09-13T17:12:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.281+0000 7f3d88000640 1 Processor -- start 2024-09-13T17:12:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.281+0000 7f3d88000640 1 -- start start 2024-09-13T17:12:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d8006c250 0x7f3d800acfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8015a780 0x7f3d800c4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d800c6020 con 0x7f3d8015ab60 2024-09-13T17:12:22.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d800c3660 con 0x7f3d8015a780 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d88000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d800c3940 con 0x7f3d8006c250 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32950/0 (socket says 172.21.15.28:32950) 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 -- 172.21.15.28:0/1815051578 learned_addr learned my addr 172.21.15.28:0/1815051578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d85600640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d8006c250 0x7f3d800acfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d84c00640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8015a780 0x7f3d800c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 -- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d8006c250 msgr2=0x7f3d800acfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d8006c250 0x7f3d800acfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 -- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8015a780 msgr2=0x7f3d800c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8015a780 0x7f3d800c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.282+0000 7f3d86000640 1 -- 172.21.15.28:0/1815051578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740094d0 con 0x7f3d8015ab60 2024-09-13T17:12:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d86000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3d78007e40 tx=0x7f3d7800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d7d600640 1 -- 172.21.15.28:0/1815051578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d78014620 con 0x7f3d8015ab60 2024-09-13T17:12:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d7d600640 1 -- 172.21.15.28:0/1815051578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7800ee90 con 0x7f3d8015ab60 2024-09-13T17:12:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d7d600640 1 -- 172.21.15.28:0/1815051578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d780149e0 con 0x7f3d8015ab60 2024-09-13T17:12:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d800c3c20 con 0x7f3d8015ab60 2024-09-13T17:12:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.283+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d800ad4e0 con 0x7f3d8015ab60 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.285+0000 7f3d7d600640 1 -- 172.21.15.28:0/1815051578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3d78014b40 con 0x7f3d8015ab60 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.286+0000 7f3d7d600640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 0x7f3d6006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.286+0000 7f3d7d600640 1 -- 172.21.15.28:0/1815051578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3d78089bf0 con 0x7f3d8015ab60 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.286+0000 7f3d8aec3740 1 Processor -- start 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.286+0000 7f3d85600640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 0x7f3d6006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 -- start start 2024-09-13T17:12:22.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d85600640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 0x7f3d6006d0f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3d74034000 tx=0x7f3d7402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cc7c6d0 0x559d7cc7cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7ccca0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cbe5ab0 con 0x559d7cc79e90 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cbe3240 con 0x559d7cc7c6d0 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.287+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d7cbdeb30 con 0x559d7cca3b80 2024-09-13T17:12:22.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32958/0 (socket says 172.21.15.28:32958) 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 -- 172.21.15.28:0/3749574895 learned_addr learned my addr 172.21.15.28:0/3749574895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d85600640 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cc7c6d0 0x559d7cc7cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d86000640 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7ccca0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 -- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 msgr2=0x559d7ccca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7ccca0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 -- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cc7c6d0 msgr2=0x559d7cc7cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cc7c6d0 0x559d7cc7cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 -- 172.21.15.28:0/3749574895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d7cba9bd0 con 0x559d7cc79e90 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.288+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009940 tx=0x7f3d6c009e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d6954b1c6d35de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d72000640 1 -- 172.21.15.28:0/3749574895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00ea80 con 0x559d7cc79e90 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d72000640 1 -- 172.21.15.28:0/3749574895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00ebe0 con 0x559d7cc79e90 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d72000640 1 -- 172.21.15.28:0/3749574895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c013470 con 0x559d7cc79e90 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3749574895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 msgr2=0x559d7ccc9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c009940 tx=0x7f3d6c009e00 comp rx=0 tx=0).stop 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3749574895 shutdown_connections 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7ccca0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc79e90 0x559d7ccc9ba0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3749574895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cc7c6d0 0x559d7cc7cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3749574895 >> 172.21.15.28:0/3749574895 conn(0x559d7cc78e00 msgr2=0x559d7ccc12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3749574895 shutdown_connections 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.289+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3749574895 wait complete. 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.290+0000 7f3d8aec3740 1 -- start start 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.290+0000 7f3d8aec3740 1 Processor -- start 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cca3b80 0x559d7ccc9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7ccc8650 0x559d7cca52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cb80260 con 0x559d7cc7c6d0 2024-09-13T17:12:22.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cb7ff00 con 0x559d7cca3b80 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d7c945f20 con 0x559d7ccc8650 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d8aec3740 10 client.0 _ll_register_callbacks cb 0x559d7cbd81a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32968/0 (socket says 172.21.15.28:32968) 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.291+0000 7f3d85600640 1 -- 172.21.15.28:0/318054200 learned_addr learned my addr 172.21.15.28:0/318054200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d84c00640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cca3b80 0x559d7ccc9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d86000640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7ccc8650 0x559d7cca52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 -- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7ccc8650 msgr2=0x559d7cca52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7ccc8650 0x559d7cca52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 -- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cca3b80 msgr2=0x559d7ccc9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cca3b80 0x559d7ccc9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 -- 172.21.15.28:0/318054200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d6c0095a0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d85600640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3d74035040 tx=0x7f3d74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527e6e430af98395 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d66a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7403cd10 con 0x559d7cc7c6d0 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3d74047410 con 0x559d7cc7c6d0 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.292+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d7cc08cd0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d7403ce70 con 0x559d7cc7c6d0 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 10 client.6717 fetch_fsmap learned FSMap version 55 2024-09-13T17:12:22.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x559d7cbdd1b0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559d7cc08cd0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3d7404f030 con 0x559d7cc7c6d0 2024-09-13T17:12:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 10 client.6717 fetch_fsmap finished waiting for FSMap version 55 2024-09-13T17:12:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 10 client.6717 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:12:22.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 20 client.6717 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccd1360 0x559d7cca2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 4 client.6717 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.293+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 --> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d7c8d23c0 con 0x559d7ccd1360 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.294+0000 7f3d84c00640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccd1360 0x559d7cca2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.294+0000 7f3d84c00640 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccd1360 0x559d7cca2bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0dd0cc85d736021 server_cookie=7fb5278b9d9bddb6 in_seq=0 out_seq=0 2024-09-13T17:12:22.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.294+0000 7f3d66a00640 10 client.6717 ms_handle_connect on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d66a00640 1 -- 172.21.15.28:0/318054200 <== mds.0 v2:172.21.15.28:6834/2705534945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x559d7c8d23c0 con 0x559d7ccd1360 2024-09-13T17:12:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d66a00640 10 client.6717 handle_command_reply: tid=0 2024-09-13T17:12:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 client.6717 shutdown 2024-09-13T17:12:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 msgr2=0x559d7cca7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3d74035040 tx=0x7f3d74004730 comp rx=0 tx=0).stop 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 shutdown_connections 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccd1360 0x559d7cca2bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7ccc8650 0x559d7cca52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7cca3b80 0x559d7ccc9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/318054200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7cca7660 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d66a00640 0 client.6717 ms_handle_reset on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:22.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.295+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 >> 172.21.15.28:0/318054200 conn(0x559d7cc79e90 msgr2=0x559d7ccc47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.296+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 shutdown_connections 2024-09-13T17:12:22.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.296+0000 7f3d8aec3740 1 -- 172.21.15.28:0/318054200 wait complete. 2024-09-13T17:12:22.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.296+0000 7f3d8aec3740 20 client.6717 trim_cache size 0 max 16384 2024-09-13T17:12:22.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 Processor -- start 2024-09-13T17:12:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 -- start start 2024-09-13T17:12:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7cc7b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cca74b0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cce3b90 con 0x559d7ccc8650 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.312+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d7cca6bc0 con 0x559d7cca3b80 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32978/0 (socket says 172.21.15.28:32978) 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 -- 172.21.15.28:0/3769419845 learned_addr learned my addr 172.21.15.28:0/3769419845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7cc7b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d86000640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 -- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 msgr2=0x559d7cc7b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7cc7b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 -- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 msgr2=0x559d7ccbd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d85600640 1 -- 172.21.15.28:0/3769419845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d7cc12170 con 0x559d7cc7c6d0 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.313+0000 7f3d86000640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d85600640 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3d7403f040 tx=0x7f3d74049070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905afaefc8b7b8fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d72a00640 1 -- 172.21.15.28:0/3769419845 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d74032cd0 con 0x559d7cc7c6d0 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d72a00640 1 -- 172.21.15.28:0/3769419845 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7404b070 con 0x559d7cc7c6d0 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d72a00640 1 -- 172.21.15.28:0/3769419845 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7403cd50 con 0x559d7cc7c6d0 2024-09-13T17:12:22.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3769419845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 msgr2=0x559d7ccc3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f3d7403f040 tx=0x7f3d74049070 comp rx=0 tx=0).stop 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3769419845 shutdown_connections 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cca3b80 0x559d7cc7b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.314+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3769419845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cc7c6d0 0x559d7ccc3870 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.315+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3769419845 >> 172.21.15.28:0/3769419845 conn(0x559d7cc79e90 msgr2=0x559d7cc7cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.315+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3769419845 shutdown_connections 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.315+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3769419845 wait complete. 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.315+0000 7f3d8aec3740 1 -- start start 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.315+0000 7f3d8aec3740 1 Processor -- start 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cc7c6d0 0x559d7ccc6970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbdaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cce1d50 con 0x559d7cca3b80 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d7cce21b0 con 0x559d7ccc8650 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559d7cd25170 con 0x559d7cc7c6d0 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.316+0000 7f3d8aec3740 10 client.0 _ll_register_callbacks cb 0x7f3d740096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32982/0 (socket says 172.21.15.28:32982) 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 -- 172.21.15.28:0/3011444975 learned_addr learned my addr 172.21.15.28:0/3011444975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d86000640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbdaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d85600640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cc7c6d0 0x559d7ccc6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 -- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cc7c6d0 msgr2=0x559d7ccc6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cc7c6d0 0x559d7ccc6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 -- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 msgr2=0x559d7ccbdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbdaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 -- 172.21.15.28:0/3011444975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d7cbdd1b0 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d84c00640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00e5a0 tx=0x7f3d6c011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36dfc818604fd5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d66a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c022040 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3d6c0287e0 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.317+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3d6c00ebe0 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3d6c005910 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 10 client.6723 fetch_fsmap learned FSMap version 55 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x559d7cb7f470 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00ebe0 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3d6c011070 con 0x559d7cca3b80 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 10 client.6723 fetch_fsmap finished waiting for FSMap version 55 2024-09-13T17:12:22.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 10 client.6723 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 20 client.6723 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccda1b0 0x559d7ccda590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 4 client.6723 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 --> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d7c8e36c0 con 0x559d7ccda1b0 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.318+0000 7f3d85600640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccda1b0 0x559d7ccda590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d85600640 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccda1b0 0x559d7ccda590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ede339da0a62d95 server_cookie=4e7ee4e61fd13399 in_seq=0 out_seq=0 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d66a00640 10 client.6723 ms_handle_connect on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d66a00640 1 -- 172.21.15.28:0/3011444975 <== mds.0 v2:172.21.15.28:6834/2705534945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x559d7c8e36c0 con 0x559d7ccda1b0 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d66a00640 10 client.6723 handle_command_reply: tid=0 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 client.6723 shutdown 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 msgr2=0x559d7ccbd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00e5a0 tx=0x7f3d6c011000 comp rx=0 tx=0).stop 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 shutdown_connections 2024-09-13T17:12:22.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x559d7ccda1b0 0x559d7ccda590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d7ccc8650 0x559d7ccbdaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559d7cca3b80 0x559d7ccbd560 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 --2- 172.21.15.28:0/3011444975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559d7cc7c6d0 0x559d7ccc6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 >> 172.21.15.28:0/3011444975 conn(0x559d7cc79e90 msgr2=0x559d7ccc0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.319+0000 7f3d66a00640 0 client.6723 ms_handle_reset on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 shutdown_connections 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d8aec3740 1 -- 172.21.15.28:0/3011444975 wait complete. 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d8aec3740 20 client.6723 trim_cache size 0 max 16384 2024-09-13T17:12:22.336 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6684, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 3.7257868940000001, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 55, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 65, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 65, 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 4.525734312 2024-09-13T17:12:22.337 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 msgr2=0x7f3d6006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 0x7f3d6006d0f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3d74034000 tx=0x7f3d7402e000 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 msgr2=0x7f3d8014eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f3d78007e40 tx=0x7f3d7800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 shutdown_connections 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d6006ac30 0x7f3d6006d0f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d8015ab60 0x7f3d8014eb50 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d8015a780 0x7f3d800c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 --2- 172.21.15.28:0/1815051578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d8006c250 0x7f3d800acfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.320+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 >> 172.21.15.28:0/1815051578 conn(0x7f3d800b0270 msgr2=0x7f3d800cd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.321+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 shutdown_connections 2024-09-13T17:12:22.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.321+0000 7f3d88000640 1 -- 172.21.15.28:0/1815051578 wait complete. 2024-09-13T17:12:22.340 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_size 100 2024-09-13T17:12:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.497+0000 7fbc86600640 1 Processor -- start 2024-09-13T17:12:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.497+0000 7fbc86600640 1 -- start start 2024-09-13T17:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.498+0000 7fbc86600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 0x7fbc8015c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.498+0000 7fbc86600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8015ab60 con 0x7fbc80159770 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.498+0000 7fbc86600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc8006f920 con 0x7fbc80070430 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.498+0000 7fbc86600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8015b290 con 0x7fbc8006f3f0 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc80159770 0x7fbc8006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54322/0 (socket says 172.21.15.28:54322) 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7f400640 1 -- 172.21.15.28:0/3418818512 learned_addr learned my addr 172.21.15.28:0/3418818512 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/3418818512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 0x7fbc8015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145207021 0 0) 0x7fbc70001110 con 0x7fbc80159770 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc64005730 con 0x7fbc80159770 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796882013 0 0) 0x7fbc8006f920 con 0x7fbc80070430 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc70001110 con 0x7fbc80070430 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7fe00640 1 -- 172.21.15.28:0/3418818512 >> v1:172.21.15.189:6789/0 conn(0x7fbc80070430 legacy=0x7fbc8015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808534688 0 0) 0x7fbc64005730 con 0x7fbc80159770 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7fe00640 1 -- 172.21.15.28:0/3418818512 >> v1:172.21.15.28:6789/0 conn(0x7fbc80159770 legacy=0x7fbc8006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.499+0000 7fbc7fe00640 1 -- 172.21.15.28:0/3418818512 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8015b570 con 0x7fbc8006f3f0 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/3418818512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 0x7fbc8015c480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009dc0 tx=0x7fbc7802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34168b1617b12f16 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70001800 con 0x7fbc8006f3f0 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7802d050 con 0x7fbc8006f3f0 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc7ea00640 1 -- 172.21.15.28:0/3418818512 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc78007760 con 0x7fbc8006f3f0 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc86600640 1 -- 172.21.15.28:0/3418818512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 msgr2=0x7fbc8015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.501+0000 7fbc86600640 1 --2- 172.21.15.28:0/3418818512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 0x7fbc8015c480 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbc78009dc0 tx=0x7fbc7802a5d0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.502+0000 7fbc86600640 1 -- 172.21.15.28:0/3418818512 shutdown_connections 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.502+0000 7fbc86600640 1 --2- 172.21.15.28:0/3418818512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc8006f3f0 0x7fbc8015c480 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.502+0000 7fbc86600640 1 -- 172.21.15.28:0/3418818512 >> 172.21.15.28:0/3418818512 conn(0x7fbc800b0270 msgr2=0x7fbc800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.502+0000 7fbc86600640 1 -- 172.21.15.28:0/3418818512 shutdown_connections 2024-09-13T17:12:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.502+0000 7fbc86600640 1 -- 172.21.15.28:0/3418818512 wait complete. 2024-09-13T17:12:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.503+0000 7fbc86600640 1 Processor -- start 2024-09-13T17:12:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.503+0000 7fbc86600640 1 -- start start 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc80070430 0x7fbc800cc230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80159770 0x7fbc800b2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc80153a50 con 0x7fbc8006f3f0 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8014e400 con 0x7fbc80159770 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc86600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc800cd8a0 con 0x7fbc80070430 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.504+0000 7fbc7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32998/0 (socket says 172.21.15.28:32998) 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 -- 172.21.15.28:0/4191940139 learned_addr learned my addr 172.21.15.28:0/4191940139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7fe00640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80159770 0x7fbc800b2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc76a00640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc80070430 0x7fbc800cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 -- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc80070430 msgr2=0x7fbc800cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc80070430 0x7fbc800cc230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 -- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80159770 msgr2=0x7fbc800b2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80159770 0x7fbc800b2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 -- 172.21.15.28:0/4191940139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc78009a70 con 0x7fbc8006f3f0 2024-09-13T17:12:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc7f400640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbc700010e0 tx=0x7fbc7000cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.505+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70005ce0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.506+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc70005e40 con 0x7fbc8006f3f0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.506+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc80154320 con 0x7fbc8006f3f0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.506+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc70007290 con 0x7fbc8006f3f0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.506+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc800cb7e0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.507+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc800cb5f0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.508+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbc700073f0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.508+0000 7fbc77e00640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 0x7fbc5c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.509+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbc70088ae0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.509+0000 7fbc76a00640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 0x7fbc5c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.510+0000 7fbc76a00640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 0x7fbc5c06d0f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbc800b3850 tx=0x7fbc68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.513+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbc800cb5f0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.693+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) v1 -- 0x7fbc800cb5f0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.697+0000 7fbc77e00640 1 -- 172.21.15.28:0/4191940139 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v18)=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbc800cb5f0 con 0x7fbc8006f3f0 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 msgr2=0x7fbc5c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 0x7fbc5c06d0f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbc800b3850 tx=0x7fbc68009040 comp rx=0 tx=0).stop 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 msgr2=0x7fbc80153de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fbc700010e0 tx=0x7fbc7000cf30 comp rx=0 tx=0).stop 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 shutdown_connections 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5c06ac30 0x7fbc5c06d0f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc80159770 0x7fbc800b2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc80070430 0x7fbc800cc230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 --2- 172.21.15.28:0/4191940139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8006f3f0 0x7fbc80153de0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 >> 172.21.15.28:0/4191940139 conn(0x7fbc800b0270 msgr2=0x7fbc8006c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 shutdown_connections 2024-09-13T17:12:22.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.700+0000 7fbc86600640 1 -- 172.21.15.28:0/4191940139 wait complete. 2024-09-13T17:12:22.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 30 2024-09-13T17:12:22.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.875+0000 7f57d5000640 1 Processor -- start 2024-09-13T17:12:22.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.875+0000 7f57d5000640 1 -- start start 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57d5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57d00ad940 0x7f57d00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57d5000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d0070470 con 0x7f57d00b2400 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57d5000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00ace30 con 0x7f57d015c9a0 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57d5000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57d00ad110 con 0x7f57d00ad940 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f57d00b2400 0x7f57d006bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54328/0 (socket says 172.21.15.28:54328) 2024-09-13T17:12:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57ce800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57d00ad940 0x7f57d00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.876+0000 7f57c7e00640 1 -- 172.21.15.28:0/2100142891 learned_addr learned my addr 172.21.15.28:0/2100142891 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.877+0000 7f57ce800640 1 -- 172.21.15.28:0/2100142891 >> v1:172.21.15.189:6789/0 conn(0x7f57d015c9a0 legacy=0x7f57d00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.877+0000 7f57ce800640 1 -- 172.21.15.28:0/2100142891 >> v1:172.21.15.28:6789/0 conn(0x7f57d00b2400 legacy=0x7f57d006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.877+0000 7f57ce800640 1 -- 172.21.15.28:0/2100142891 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d006b170 con 0x7f57d00ad940 2024-09-13T17:12:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.877+0000 7f57cd400640 1 -- 172.21.15.28:0/2100142891 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 479976536 0 0) 0x7f57d0070470 con 0x7f57d00b2400 2024-09-13T17:12:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.878+0000 7f57ce800640 1 --2- 172.21.15.28:0/2100142891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00ad940 0x7f57d00ac850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009d90 tx=0x7f57c802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1a2ab2016c2e41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.878+0000 7f57cd400640 1 -- 172.21.15.28:0/2100142891 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8007740 con 0x7f57d00ad940 2024-09-13T17:12:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.878+0000 7f57cd400640 1 -- 172.21.15.28:0/2100142891 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57c802d050 con 0x7f57d00ad940 2024-09-13T17:12:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.878+0000 7f57cd400640 1 -- 172.21.15.28:0/2100142891 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57c8007ac0 con 0x7f57d00ad940 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 -- 172.21.15.28:0/2100142891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00ad940 msgr2=0x7f57d00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 --2- 172.21.15.28:0/2100142891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00ad940 0x7f57d00ac850 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f57c8009d90 tx=0x7f57c802a6f0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 -- 172.21.15.28:0/2100142891 shutdown_connections 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 --2- 172.21.15.28:0/2100142891 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00ad940 0x7f57d00ac850 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 -- 172.21.15.28:0/2100142891 >> 172.21.15.28:0/2100142891 conn(0x7f57d00b0270 msgr2=0x7f57d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 -- 172.21.15.28:0/2100142891 shutdown_connections 2024-09-13T17:12:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.879+0000 7f57d5000640 1 -- 172.21.15.28:0/2100142891 wait complete. 2024-09-13T17:12:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.881+0000 7f57d5000640 1 Processor -- start 2024-09-13T17:12:22.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.881+0000 7f57d5000640 1 -- start start 2024-09-13T17:12:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57d00ad940 0x7f57d01510f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57d015c9a0 0x7f57d01e9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0151630 con 0x7f57d00b2400 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d00cb6f0 con 0x7f57d00ad940 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57d5000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57d0155410 con 0x7f57d015c9a0 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57d00ad940 0x7f57d01510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33018/0 (socket says 172.21.15.28:33018) 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 -- 172.21.15.28:0/1610902870 learned_addr learned my addr 172.21.15.28:0/1610902870 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57ce800640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57d015c9a0 0x7f57d01e9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 -- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57d015c9a0 msgr2=0x7f57d01e9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57d015c9a0 0x7f57d01e9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.882+0000 7f57c7e00640 1 -- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57d00ad940 msgr2=0x7f57d01510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c7e00640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57d00ad940 0x7f57d01510f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c7e00640 1 -- 172.21.15.28:0/1610902870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c8009a70 con 0x7f57d00b2400 2024-09-13T17:12:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c7e00640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c4a0 tx=0x7f57b800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b80074e0 con 0x7f57d00b2400 2024-09-13T17:12:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b8007640 con 0x7f57d00b2400 2024-09-13T17:12:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b8004a00 con 0x7f57d00b2400 2024-09-13T17:12:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d0070930 con 0x7f57d00b2400 2024-09-13T17:12:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.883+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d0155990 con 0x7f57d00b2400 2024-09-13T17:12:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.884+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00cb530 con 0x7f57d00b2400 2024-09-13T17:12:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.885+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f57b8004b60 con 0x7f57d00b2400 2024-09-13T17:12:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.886+0000 7f57c6000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 0x7f57ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.887+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f57b8088510 con 0x7f57d00b2400 2024-09-13T17:12:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.887+0000 7f57cde00640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 0x7f57ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:22.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.887+0000 7f57cde00640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 0x7f57ac06d0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57d00accf0 tx=0x7f57c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:22.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:22.891+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f57d00cb530 con 0x7f57d00b2400 2024-09-13T17:12:23.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.070+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f57d00cb530 con 0x7f57d00b2400 2024-09-13T17:12:23.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.074+0000 7f57c6000640 1 -- 172.21.15.28:0/1610902870 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v19)=0 v19) v1 ==== 117+0+0 (secure 0 0 0) 0x7f57d00cb530 con 0x7f57d00b2400 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 msgr2=0x7f57ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 0x7f57ac06d0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f57d00accf0 tx=0x7f57c0009040 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 msgr2=0x7f57d0154ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f57b800c4a0 tx=0x7f57b800c960 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 shutdown_connections 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f57ac06ac00 0x7f57ac06d0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.076+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f57d015c9a0 0x7f57d01e9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.077+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57d00b2400 0x7f57d0154ed0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.077+0000 7f57d5000640 1 --2- 172.21.15.28:0/1610902870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57d00ad940 0x7f57d01510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.077+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 >> 172.21.15.28:0/1610902870 conn(0x7f57d00b0270 msgr2=0x7f57d006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.077+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 shutdown_connections 2024-09-13T17:12:23.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.077+0000 7f57d5000640 1 -- 172.21.15.28:0/1610902870 wait complete. 2024-09-13T17:12:23.094 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.249+0000 7f3821400640 1 Processor -- start 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.249+0000 7f3821400640 1 -- start start 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.250+0000 7f3821400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 0x7f381c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.250+0000 7f3821400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 0x7f381c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3821400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c0b2040 con 0x7f381c06c2d0 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3821400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c15b0b0 con 0x7f381c06f970 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3821400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c159ff0 con 0x7f381c1592a0 2024-09-13T17:12:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f381b200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f381c06c2d0 0x7f381c0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54332/0 (socket says 172.21.15.28:54332) 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f381b200640 1 -- 172.21.15.28:0/841921266 learned_addr learned my addr 172.21.15.28:0/841921266 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f381a800640 1 --2- 172.21.15.28:0/841921266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 0x7f381c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3819e00640 1 --2- 172.21.15.28:0/841921266 >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 0x7f381c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3819400640 1 -- 172.21.15.28:0/841921266 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514697847 0 0) 0x7f381c0b2040 con 0x7f381c06c2d0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.251+0000 7f3819400640 1 -- 172.21.15.28:0/841921266 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3800003520 con 0x7f381c06c2d0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.252+0000 7f381a800640 1 -- 172.21.15.28:0/841921266 >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 msgr2=0x7f381c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.252+0000 7f381a800640 1 --2- 172.21.15.28:0/841921266 >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 0x7f381c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.252+0000 7f381a800640 1 -- 172.21.15.28:0/841921266 >> v1:172.21.15.28:6789/0 conn(0x7f381c06c2d0 legacy=0x7f381c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.252+0000 7f381a800640 1 -- 172.21.15.28:0/841921266 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c15a2d0 con 0x7f381c1592a0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.252+0000 7f3819e00640 1 --2- 172.21.15.28:0/841921266 >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 0x7f381c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f381a800640 1 --2- 172.21.15.28:0/841921266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 0x7f381c15ab70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f380c0097a0 tx=0x7f380c02c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63e6ca57fa433428 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f3819400640 1 -- 172.21.15.28:0/841921266 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c034640 con 0x7f381c1592a0 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f3819400640 1 -- 172.21.15.28:0/841921266 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c0347a0 con 0x7f381c1592a0 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f3819400640 1 -- 172.21.15.28:0/841921266 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c034b10 con 0x7f381c1592a0 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f3821400640 1 -- 172.21.15.28:0/841921266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 msgr2=0x7f381c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.253+0000 7f3821400640 1 --2- 172.21.15.28:0/841921266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 0x7f381c15ab70 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f380c0097a0 tx=0x7f380c02c390 comp rx=0 tx=0).stop 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 -- 172.21.15.28:0/841921266 shutdown_connections 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 --2- 172.21.15.28:0/841921266 >> v2:172.21.15.189:3300/0 conn(0x7f381c06f970 0x7f381c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 --2- 172.21.15.28:0/841921266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c1592a0 0x7f381c15ab70 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 -- 172.21.15.28:0/841921266 >> 172.21.15.28:0/841921266 conn(0x7f381c069e80 msgr2=0x7f381c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 -- 172.21.15.28:0/841921266 shutdown_connections 2024-09-13T17:12:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.254+0000 7f3821400640 1 -- 172.21.15.28:0/841921266 wait complete. 2024-09-13T17:12:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.255+0000 7f3821400640 1 Processor -- start 2024-09-13T17:12:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.255+0000 7f3821400640 1 -- start start 2024-09-13T17:12:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f381c06c2d0 0x7f381c1538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c06f970 0x7f381c1e7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c1536b0 con 0x7f381c1592a0 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c154120 con 0x7f381c06c2d0 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f3821400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c1e6280 con 0x7f381c06f970 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.256+0000 7f381b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33026/0 (socket says 172.21.15.28:33026) 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 -- 172.21.15.28:0/3561241658 learned_addr learned my addr 172.21.15.28:0/3561241658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f3819e00640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c06f970 0x7f381c1e7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381a800640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f381c06c2d0 0x7f381c1538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 -- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c06f970 msgr2=0x7f381c1e7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c06f970 0x7f381c1e7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 -- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f381c06c2d0 msgr2=0x7f381c1538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f381c06c2d0 0x7f381c1538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 -- 172.21.15.28:0/3561241658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3808009400 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f381b200640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f381400cb40 tx=0x7f381400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.257+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381400aca0 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.258+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381400ae00 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.258+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c009400 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.258+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814012620 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.258+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c1e6840 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.259+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c15ab70 con 0x7f381c1592a0 2024-09-13T17:12:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.260+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3814007070 con 0x7f381c1592a0 2024-09-13T17:12:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.260+0000 7f3812a00640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 0x7f37fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.261+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3814089390 con 0x7f381c1592a0 2024-09-13T17:12:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.261+0000 7f381a800640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 0x7f37fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.262+0000 7f381a800640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 0x7f37fc06d0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f380c036000 tx=0x7f380c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.265+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f38140545b0 con 0x7f381c1592a0 2024-09-13T17:12:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.478+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f381c1e7e30 con 0x7f381c1592a0 2024-09-13T17:12:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.479+0000 7f3812a00640 1 -- 172.21.15.28:0/3561241658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f3814057bd0 con 0x7f381c1592a0 2024-09-13T17:12:23.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:23.481 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:23.481 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 msgr2=0x7f37fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 0x7f37fc06d0c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f380c036000 tx=0x7f380c030000 comp rx=0 tx=0).stop 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 msgr2=0x7f381c153170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f381400cb40 tx=0x7f381400a060 comp rx=0 tx=0).stop 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 shutdown_connections 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37fc06ac00 0x7f37fc06d0c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f381c1592a0 0x7f381c153170 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f381c06f970 0x7f381c1e7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 --2- 172.21.15.28:0/3561241658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f381c06c2d0 0x7f381c1538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 >> 172.21.15.28:0/3561241658 conn(0x7f381c069e80 msgr2=0x7f381c0af320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 shutdown_connections 2024-09-13T17:12:23.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.481+0000 7f3821400640 1 -- 172.21.15.28:0/3561241658 wait complete. 2024-09-13T17:12:23.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 1099511627776 2024-09-13T17:12:23.670 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:12:14.384821+0000","btime":"2024-09-13T17:12:13.235853+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:12:14.384821+0000","atime":"2024-09-13T17:12:13.235853+0000","time_warp_seq":0,"change_attr":40,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:12:14.384821+0000","num_files":40,"num_subdirs":0,"change_attr":40},"rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"accounted_rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"version":244,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[{"frag":"*","children":1}]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{},"nref":0,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:12:23.670 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:12:13.235853+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:12:13.235853+0000", "change_attr": 40, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:12:14.384821+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [ { "children": 1, "frag": "*" } ] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 40, "mtime": "2024-09-13T17:12:14.384821+0000", "num_files": 40, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:12:14.384821+0000", "nestlock": {}, "nlink": 1, "nref": 0, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": {}, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 244, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:12:23.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start /testdir recursive 2024-09-13T17:12:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.829+0000 7f66f5a00640 1 Processor -- start 2024-09-13T17:12:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.829+0000 7f66f5a00640 1 -- start start 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66f5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66f015a7a0 0x7f66f00ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66f5a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66f00b1c80 0x7f66f00cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66f5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66f006c950 con 0x7f66f00ce9d0 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66f5a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f00cee40 con 0x7f66f015a7a0 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66f5a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66f00b2150 con 0x7f66f00b1c80 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66eea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66f015a7a0 0x7f66f00ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66e6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f66f00ce9d0 0x7f66f00b1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46666/0 (socket says 172.21.15.28:46666) 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.830+0000 7f66e6000640 1 -- 172.21.15.28:0/3990195980 learned_addr learned my addr 172.21.15.28:0/3990195980 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66ef400640 1 --2- 172.21.15.28:0/3990195980 >> v2:172.21.15.189:3300/0 conn(0x7f66f00b1c80 0x7f66f00cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66eea00640 1 -- 172.21.15.28:0/3990195980 >> v2:172.21.15.189:3300/0 conn(0x7f66f00b1c80 msgr2=0x7f66f00cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66eea00640 1 --2- 172.21.15.28:0/3990195980 >> v2:172.21.15.189:3300/0 conn(0x7f66f00b1c80 0x7f66f00cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66eea00640 1 -- 172.21.15.28:0/3990195980 >> v1:172.21.15.28:6789/0 conn(0x7f66f00ce9d0 legacy=0x7f66f00b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66eea00640 1 -- 172.21.15.28:0/3990195980 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f00b2430 con 0x7f66f015a7a0 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.831+0000 7f66ee000640 1 -- 172.21.15.28:0/3990195980 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301662016 0 0) 0x7f66d8001110 con 0x7f66f00ce9d0 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.832+0000 7f66eea00640 1 --2- 172.21.15.28:0/3990195980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f015a7a0 0x7f66f00ce490 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b3c30 tx=0x7f66e002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a478b3f9505959 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.832+0000 7f66ee000640 1 -- 172.21.15.28:0/3990195980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e002ae60 con 0x7f66f015a7a0 2024-09-13T17:12:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.832+0000 7f66ee000640 1 -- 172.21.15.28:0/3990195980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0032440 con 0x7f66f015a7a0 2024-09-13T17:12:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.832+0000 7f66ee000640 1 -- 172.21.15.28:0/3990195980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00327d0 con 0x7f66f015a7a0 2024-09-13T17:12:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 -- 172.21.15.28:0/3990195980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f015a7a0 msgr2=0x7f66f00ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 --2- 172.21.15.28:0/3990195980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f015a7a0 0x7f66f00ce490 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f66f00b3c30 tx=0x7f66e002a110 comp rx=0 tx=0).stop 2024-09-13T17:12:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 -- 172.21.15.28:0/3990195980 shutdown_connections 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 --2- 172.21.15.28:0/3990195980 >> v2:172.21.15.189:3300/0 conn(0x7f66f00b1c80 0x7f66f00cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 --2- 172.21.15.28:0/3990195980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f015a7a0 0x7f66f00ce490 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 -- 172.21.15.28:0/3990195980 >> 172.21.15.28:0/3990195980 conn(0x7f66f0069e80 msgr2=0x7f66f006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 -- 172.21.15.28:0/3990195980 shutdown_connections 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.833+0000 7f66f5a00640 1 -- 172.21.15.28:0/3990195980 wait complete. 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.835+0000 7f66f5a00640 1 Processor -- start 2024-09-13T17:12:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.835+0000 7f66f5a00640 1 -- start start 2024-09-13T17:12:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.835+0000 7f66f5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.835+0000 7f66f5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66f00ce9d0 0x7f66f01f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66f5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f015a7a0 0x7f66f01f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66f5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00aca20 con 0x7f66f00b1c80 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66f5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00af650 con 0x7f66f015a7a0 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66f5a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f00abe10 con 0x7f66f00ce9d0 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60406/0 (socket says 172.21.15.28:60406) 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 -- 172.21.15.28:0/2290946450 learned_addr learned my addr 172.21.15.28:0/2290946450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66ef400640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f015a7a0 0x7f66f01f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66e6000640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66f00ce9d0 0x7f66f01f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 -- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66f00ce9d0 msgr2=0x7f66f01f00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66f00ce9d0 0x7f66f01f00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 -- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f015a7a0 msgr2=0x7f66f01f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f015a7a0 0x7f66f01f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.836+0000 7f66eea00640 1 -- 172.21.15.28:0/2290946450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e0009400 con 0x7f66f00b1c80 2024-09-13T17:12:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66eea00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66e7400640 1 -- 172.21.15.28:0/2290946450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e00308d0 con 0x7f66f00b1c80 2024-09-13T17:12:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66e7400640 1 -- 172.21.15.28:0/2290946450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e0030a50 con 0x7f66f00b1c80 2024-09-13T17:12:23.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66e7400640 1 -- 172.21.15.28:0/2290946450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0030e00 con 0x7f66f00b1c80 2024-09-13T17:12:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f00ac0f0 con 0x7f66f00b1c80 2024-09-13T17:12:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.837+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f00ac6b0 con 0x7f66f00b1c80 2024-09-13T17:12:23.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.839+0000 7f66e7400640 1 -- 172.21.15.28:0/2290946450 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f66e0039040 con 0x7f66f00b1c80 2024-09-13T17:12:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.840+0000 7f66e7400640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 0x7f66cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.840+0000 7f66e7400640 1 -- 172.21.15.28:0/2290946450 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66e00a9f70 con 0x7f66f00b1c80 2024-09-13T17:12:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.840+0000 7f66e6000640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 0x7f66cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.841+0000 7f66f88be740 1 Processor -- start 2024-09-13T17:12:23.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.841+0000 7f66f88be740 1 -- start start 2024-09-13T17:12:23.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.841+0000 7f66e6000640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 0x7f66cc06d0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f66f01edcc0 tx=0x7f66d8007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b883900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b838750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b7f95e0 con 0x55fa2b83b920 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b7a7df0 con 0x55fa2b83e160 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b7a5580 con 0x55fa2b83f8f0 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60420/0 (socket says 172.21.15.28:60420) 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 -- 172.21.15.28:0/961176473 learned_addr learned my addr 172.21.15.28:0/961176473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66ef400640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b838750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66eea00640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b883900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 -- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 msgr2=0x55fa2b838750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b838750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 -- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 msgr2=0x55fa2b883900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b883900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66e6000640 1 -- 172.21.15.28:0/961176473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa2b7680e0 con 0x55fa2b83b920 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.842+0000 7f66eea00640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b883900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.843+0000 7f66e6000640 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f66d8004e30 tx=0x7f66d8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff0d98fbaf76c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.843+0000 7f66df400640 1 -- 172.21.15.28:0/961176473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8013b00 con 0x55fa2b83b920 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.843+0000 7f66df400640 1 -- 172.21.15.28:0/961176473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d8013c60 con 0x55fa2b83b920 2024-09-13T17:12:23.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.843+0000 7f66df400640 1 -- 172.21.15.28:0/961176473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8012350 con 0x55fa2b83b920 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- 172.21.15.28:0/961176473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 msgr2=0x55fa2b83e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f66d8004e30 tx=0x7f66d8013000 comp rx=0 tx=0).stop 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- 172.21.15.28:0/961176473 shutdown_connections 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b838750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83b920 0x55fa2b83e540 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 --2- 172.21.15.28:0/961176473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b883900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- 172.21.15.28:0/961176473 >> 172.21.15.28:0/961176473 conn(0x55fa2b83a920 msgr2=0x55fa2b882dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- 172.21.15.28:0/961176473 shutdown_connections 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- 172.21.15.28:0/961176473 wait complete. 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.844+0000 7f66f88be740 1 -- start start 2024-09-13T17:12:23.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.845+0000 7f66f88be740 1 Processor -- start 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 0x55fa2b866830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 0x55fa2b838a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b7a0e70 con 0x55fa2b83f8f0 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b7420a0 con 0x55fa2b838580 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b741d40 con 0x55fa2b83e160 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66f88be740 10 client.0 _ll_register_callbacks cb 0x55fa2b7678c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:23.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60422/0 (socket says 172.21.15.28:60422) 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 -- 172.21.15.28:0/2220108806 learned_addr learned my addr 172.21.15.28:0/2220108806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66eea00640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 0x55fa2b866830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66ef400640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 0x55fa2b838a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 -- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 msgr2=0x55fa2b866830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 0x55fa2b866830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 -- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 msgr2=0x55fa2b838a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 0x55fa2b838a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66e6000640 1 -- 172.21.15.28:0/2220108806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d8007650 con 0x55fa2b83f8f0 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66eea00640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 0x55fa2b866830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:23.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.846+0000 7f66ef400640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 0x55fa2b838a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66e6000640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f66d8007260 tx=0x7f66d8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38feb023cdc2142b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8013b00 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66d801b050 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa2b792d10 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d8021e70 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 10 client.6759 fetch_fsmap learned FSMap version 55 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x55fa2b79f4f0 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55fa2b792d10 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f66d8012070 con 0x55fa2b83f8f0 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 10 client.6759 fetch_fsmap finished waiting for FSMap version 55 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 10 client.6759 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-09-13T17:12:23.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 20 client.6759 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.847+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b85f0f0 0x55fa2b83c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.848+0000 7f66f88be740 4 client.6759 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.848+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 --> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa2b494700 con 0x55fa2b85f0f0 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.848+0000 7f66eea00640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b85f0f0 0x55fa2b83c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.848+0000 7f66eea00640 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b85f0f0 0x55fa2b83c4a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d35961edd51aec3 server_cookie=2883b7c01d1b0a5b in_seq=0 out_seq=0 2024-09-13T17:12:23.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.848+0000 7f66d3e00640 10 client.6759 ms_handle_connect on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:23.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66d3e00640 1 -- 172.21.15.28:0/2220108806 <== mds.0 v2:172.21.15.28:6834/2705534945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55fa2b494700 con 0x55fa2b85f0f0 2024-09-13T17:12:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66d3e00640 10 client.6759 handle_command_reply: tid=0 2024-09-13T17:12:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 client.6759 shutdown 2024-09-13T17:12:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 msgr2=0x55fa2b85c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f66d8007260 tx=0x7f66d8000ed0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 shutdown_connections 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b85f0f0 0x55fa2b83c4a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b838580 0x55fa2b838a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b83f8f0 0x55fa2b85c6e0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 --2- 172.21.15.28:0/2220108806 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83e160 0x55fa2b866830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66d3e00640 0 client.6759 ms_handle_reset on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 >> 172.21.15.28:0/2220108806 conn(0x55fa2b83b920 msgr2=0x55fa2b8825f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 shutdown_connections 2024-09-13T17:12:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.849+0000 7f66f88be740 1 -- 172.21.15.28:0/2220108806 wait complete. 2024-09-13T17:12:23.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.850+0000 7f66f88be740 20 client.6759 trim_cache size 0 max 16384 2024-09-13T17:12:23.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 Processor -- start 2024-09-13T17:12:23.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 -- start start 2024-09-13T17:12:23.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b87c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b87cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b508260 con 0x55fa2b838580 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b865310 con 0x55fa2b83e160 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.865+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b85d110 con 0x55fa2b83f8f0 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60434/0 (socket says 172.21.15.28:60434) 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 -- 172.21.15.28:0/2431012883 learned_addr learned my addr 172.21.15.28:0/2431012883 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66ef400640 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b87cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66e6000640 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b87c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 -- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 msgr2=0x55fa2b87cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b87cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 -- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 msgr2=0x55fa2b87c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b87c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 -- 172.21.15.28:0/2431012883 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa2b741240 con 0x55fa2b838580 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.866+0000 7f66eea00640 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f66e00a83d0 tx=0x7f66e00a8300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f1cffec3580d58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66dfe00640 1 -- 172.21.15.28:0/2431012883 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e0076f90 con 0x55fa2b838580 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66dfe00640 1 -- 172.21.15.28:0/2431012883 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66e00771f0 con 0x55fa2b838580 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66dfe00640 1 -- 172.21.15.28:0/2431012883 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e008cbe0 con 0x55fa2b838580 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 -- 172.21.15.28:0/2431012883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 msgr2=0x55fa2b865bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f66e00a83d0 tx=0x7f66e00a8300 comp rx=0 tx=0).stop 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 -- 172.21.15.28:0/2431012883 shutdown_connections 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b87cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b87c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 --2- 172.21.15.28:0/2431012883 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b865bf0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 -- 172.21.15.28:0/2431012883 >> 172.21.15.28:0/2431012883 conn(0x55fa2b83b920 msgr2=0x55fa2b883f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 -- 172.21.15.28:0/2431012883 shutdown_connections 2024-09-13T17:12:23.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.867+0000 7f66f88be740 1 -- 172.21.15.28:0/2431012883 wait complete. 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.868+0000 7f66f88be740 1 -- start start 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.868+0000 7f66f88be740 1 Processor -- start 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b8a0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b895740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b85e210 con 0x55fa2b838580 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b8a1120 con 0x55fa2b83e160 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55fa2b886ad0 con 0x55fa2b83f8f0 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66f88be740 10 client.0 _ll_register_callbacks cb 0x7f66d8007e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66eea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60436/0 (socket says 172.21.15.28:60436) 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66eea00640 1 -- 172.21.15.28:0/3023298510 learned_addr learned my addr 172.21.15.28:0/3023298510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66ef400640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b895740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.869+0000 7f66e6000640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b8a0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 -- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 msgr2=0x55fa2b895740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b895740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 -- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 msgr2=0x55fa2b8a0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b8a0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 -- 172.21.15.28:0/3023298510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f00ac6b0 con 0x55fa2b838580 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66e6000640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b8a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66eea00640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66e0080c60 tx=0x7f66d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15926da62aad45c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e008cde0 con 0x55fa2b838580 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66e00a8570 con 0x55fa2b838580 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66d8013c60 con 0x55fa2b838580 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66e00790b0 con 0x55fa2b838580 2024-09-13T17:12:23.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66f88be740 10 client.6765 fetch_fsmap learned FSMap version 55 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.870+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x55fa2b766940 con 0x55fa2b838580 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d8013c60 con 0x55fa2b838580 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 55) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f66e0078340 con 0x55fa2b838580 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 10 client.6765 fetch_fsmap finished waiting for FSMap version 55 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 10 client.6765 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.c 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 20 client.6765 populate_metadata read hostname 'smithi028' 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b8f50d0 0x55fa2b895c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 4 client.6765 mds_command: new command op to 6684 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/testdir", "scrubops": ["recursive"]}] 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 --> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] -- command(tid 0: {"prefix": "scrub start", "path": "/testdir", "scrubops": ["recursive"]}) v1 -- 0x55fa2b4a5a00 con 0x55fa2b8f50d0 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66e6000640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b8f50d0 0x55fa2b895c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66e6000640 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b8f50d0 0x55fa2b895c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3bff88891af6cb5 server_cookie=f0b05b68d1c4f060 in_seq=0 out_seq=0 2024-09-13T17:12:23.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.871+0000 7f66d3e00640 10 client.6765 ms_handle_connect on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:23.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66d3e00640 1 -- 172.21.15.28:0/3023298510 <== mds.0 v2:172.21.15.28:6834/2705534945 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55fa2b4a5a00 con 0x55fa2b8f50d0 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66d3e00640 10 client.6765 handle_command_reply: tid=0 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 client.6765 shutdown 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 msgr2=0x55fa2b8ef130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66e0080c60 tx=0x7f66d8014000 comp rx=0 tx=0).stop 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 shutdown_connections 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:6834/2705534945,v1:172.21.15.28:6835/2705534945] conn(0x55fa2b8f50d0 0x55fa2b895c80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55fa2b83f8f0 0x55fa2b895740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fa2b83e160 0x55fa2b8a0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 --2- 172.21.15.28:0/3023298510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55fa2b838580 0x55fa2b8ef130 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66d3e00640 0 client.6765 ms_handle_reset on v2:172.21.15.28:6834/2705534945 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 >> 172.21.15.28:0/3023298510 conn(0x55fa2b83b920 msgr2=0x55fa2b884930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.872+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 shutdown_connections 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f88be740 1 -- 172.21.15.28:0/3023298510 wait complete. 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f88be740 20 client.6765 trim_cache size 0 max 16384 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "66e6903d-9493-482f-8a54-fb4957bd3acb", 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:12:23.881 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 msgr2=0x7f66cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 0x7f66cc06d0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f66f01edcc0 tx=0x7f66d8007b90 comp rx=0 tx=0).stop 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 msgr2=0x7f66f00ab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f66e0000c00 tx=0x7f66e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 shutdown_connections 2024-09-13T17:12:23.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f66cc06ac00 0x7f66cc06d0c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66f015a7a0 0x7f66f01f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66f00ce9d0 0x7f66f01f00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 --2- 172.21.15.28:0/2290946450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66f00b1c80 0x7f66f00ab4a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 >> 172.21.15.28:0/2290946450 conn(0x7f66f0069e80 msgr2=0x7f66f00cd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.873+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 shutdown_connections 2024-09-13T17:12:23.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:23.874+0000 7f66f5a00640 1 -- 172.21.15.28:0/2290946450 wait complete. 2024-09-13T17:12:23.892 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.046+0000 7f5f5e200640 1 Processor -- start 2024-09-13T17:12:24.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.046+0000 7f5f5e200640 1 -- start start 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f5e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f5815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f5e200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f5806c2d0 0x7f5f5806c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f5e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f58159d90 con 0x7f5f581592a0 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f5e200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5815b0b0 con 0x7f5f5806c2d0 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f5e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5806cca0 con 0x7f5f5806f970 2024-09-13T17:12:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f57400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f581592a0 0x7f5f58159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46678/0 (socket says 172.21.15.28:46678) 2024-09-13T17:12:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.047+0000 7f5f57400640 1 -- 172.21.15.28:0/1421810479 learned_addr learned my addr 172.21.15.28:0/1421810479 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56a00640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f5815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f57e00640 1 --2- 172.21.15.28:0/1421810479 >> v2:172.21.15.189:3300/0 conn(0x7f5f5806c2d0 0x7f5f5806c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942889008 0 0) 0x7f5f58159d90 con 0x7f5f581592a0 2024-09-13T17:12:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f3c005730 con 0x7f5f581592a0 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000628742 0 0) 0x7f5f3c005730 con 0x7f5f581592a0 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 msgr2=0x7f5f5815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f5815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 >> v2:172.21.15.189:3300/0 conn(0x7f5f5806c2d0 msgr2=0x7f5f5806c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 --2- 172.21.15.28:0/1421810479 >> v2:172.21.15.189:3300/0 conn(0x7f5f5806c2d0 0x7f5f5806c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580b2040 con 0x7f5f581592a0 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f5f48001690 con 0x7f5f581592a0 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 >> v1:172.21.15.28:6789/0 conn(0x7f5f581592a0 legacy=0x7f5f58159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c005b80 0x7f5f3c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f3c0069d0 0x7f5f3c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f3c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5815b0b0 con 0x7f5f3c005b80 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5806cca0 con 0x7f5f3c0069d0 2024-09-13T17:12:24.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.048+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3c00db90 con 0x7f5f5806f970 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c005b80 0x7f5f3c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f56a00640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f3c0069d0 0x7f5f3c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57e00640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f3c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 -- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 msgr2=0x7f5f3c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5806f970 0x7f5f3c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 -- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f3c0069d0 msgr2=0x7f5f3c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f3c0069d0 0x7f5f3c006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.049+0000 7f5f57400640 1 -- 172.21.15.28:0/1421810479 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c005730 con 0x7f5f3c005b80 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.050+0000 7f5f57400640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c005b80 0x7f5f3c005fa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c210 tx=0x7f5f3c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e68914f0c9672cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.050+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4802e070 con 0x7f5f3c005b80 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.050+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f480058d0 con 0x7f5f3c005b80 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.050+0000 7f5f56000640 1 -- 172.21.15.28:0/1421810479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f48005c80 con 0x7f5f3c005b80 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 -- 172.21.15.28:0/1421810479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c005b80 msgr2=0x7f5f3c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1421810479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f3c005b80 0x7f5f3c005fa0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800c210 tx=0x7f5f3c007a00 comp rx=0 tx=0).stop 2024-09-13T17:12:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f57e00640 1 -- 172.21.15.28:0/1421810479 reap_dead start 2024-09-13T17:12:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 -- 172.21.15.28:0/1421810479 shutdown_connections 2024-09-13T17:12:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 -- 172.21.15.28:0/1421810479 >> 172.21.15.28:0/1421810479 conn(0x7f5f58069e80 msgr2=0x7f5f5806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 -- 172.21.15.28:0/1421810479 shutdown_connections 2024-09-13T17:12:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.051+0000 7f5f5e200640 1 -- 172.21.15.28:0/1421810479 wait complete. 2024-09-13T17:12:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.053+0000 7f5f5e200640 1 Processor -- start 2024-09-13T17:12:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.053+0000 7f5f5e200640 1 -- start start 2024-09-13T17:12:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f3c005b80 0x7f5f58152900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 0x7f5f5813ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f58152470 con 0x7f5f5814ef10 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f581404c0 con 0x7f5f58151360 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f5e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f58151860 con 0x7f5f3c005b80 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60456/0 (socket says 172.21.15.28:60456) 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 -- 172.21.15.28:0/1487842539 learned_addr learned my addr 172.21.15.28:0/1487842539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57400640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f3c005b80 0x7f5f58152900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f56a00640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 0x7f5f5813ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 -- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f3c005b80 msgr2=0x7f5f58152900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f3c005b80 0x7f5f58152900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 -- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 msgr2=0x7f5f5813ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 0x7f5f5813ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f57e00640 1 -- 172.21.15.28:0/1487842539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f58151b40 con 0x7f5f5814ef10 2024-09-13T17:12:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.054+0000 7f5f56a00640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 0x7f5f5813ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:24.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f57e00640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5f5806f6d0 tx=0x7f5f5000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5000e040 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5000f040 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f58151e20 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50015670 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.055+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5814f3c0 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.056+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5814f5e0 con 0x7f5f5814ef10 2024-09-13T17:12:24.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.057+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5f500157d0 con 0x7f5f5814ef10 2024-09-13T17:12:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.058+0000 7f5f4f400640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 0x7f5f3806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.058+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5f50089e20 con 0x7f5f5814ef10 2024-09-13T17:12:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.059+0000 7f5f57400640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 0x7f5f3806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:24.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.059+0000 7f5f57400640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 0x7f5f3806d0f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5f48033000 tx=0x7f5f48009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.062+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5f50054f90 con 0x7f5f5814ef10 2024-09-13T17:12:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.278+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f580b3740 con 0x7f5f5814ef10 2024-09-13T17:12:24.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.279+0000 7f5f4f400640 1 -- 172.21.15.28:0/1487842539 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f5f500585b0 con 0x7f5f5814ef10 2024-09-13T17:12:24.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:24.282 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:24.282 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 msgr2=0x7f5f3806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 0x7f5f3806d0f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5f48033000 tx=0x7f5f48009030 comp rx=0 tx=0).stop 2024-09-13T17:12:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 msgr2=0x7f5f58153d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5f5806f6d0 tx=0x7f5f5000f580 comp rx=0 tx=0).stop 2024-09-13T17:12:24.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 shutdown_connections 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3806ac30 0x7f5f3806d0f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5814ef10 0x7f5f58153d00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f58151360 0x7f5f5813ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 --2- 172.21.15.28:0/1487842539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f3c005b80 0x7f5f58152900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 >> 172.21.15.28:0/1487842539 conn(0x7f5f58069e80 msgr2=0x7f5f5806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 shutdown_connections 2024-09-13T17:12:24.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:24.281+0000 7f5f5e200640 1 -- 172.21.15.28:0/1487842539 wait complete. 2024-09-13T17:12:24.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 1099511627776 2024-09-13T17:12:24.472 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:12:14.384821+0000","btime":"2024-09-13T17:12:13.235853+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:12:14.384821+0000","atime":"2024-09-13T17:12:13.235853+0000","time_warp_seq":0,"change_attr":40,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:12:14.384821+0000","num_files":40,"num_subdirs":0,"change_attr":40},"rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"accounted_rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"version":244,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[{"frag":"*","children":1}]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirfrag":2,"authpin":0,"scrubqueue":0},"nref":2,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:12:24.472 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:12:13.235853+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:12:13.235853+0000", "change_attr": 40, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:12:14.384821+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [ { "children": 1, "frag": "*" } ] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 40, "mtime": "2024-09-13T17:12:14.384821+0000", "num_files": 40, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:12:14.384821+0000", "nestlock": {}, "nlink": 1, "nref": 2, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "authpin": 0, "dirfrag": 2, "scrubqueue": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 244, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:12:24.473 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-09-13T17:12:24.596 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:12:24.594+0000 7f8fd9400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:12:29.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:29.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.640+0000 7fb3e6a00640 1 Processor -- start 2024-09-13T17:12:29.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.640+0000 7fb3e6a00640 1 -- start start 2024-09-13T17:12:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3e6a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb3e00cabc0 0x7fb3e006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3e6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e015e100 con 0x7fb3e015a540 2024-09-13T17:12:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3e6a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e015e3e0 con 0x7fb3e0070cc0 2024-09-13T17:12:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3e6a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e00adb00 con 0x7fb3e00cabc0 2024-09-13T17:12:29.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3dfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb3e015a540 0x7fb3e015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46694/0 (socket says 172.21.15.28:46694) 2024-09-13T17:12:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.641+0000 7fb3dfe00640 1 -- 172.21.15.28:0/1479699645 learned_addr learned my addr 172.21.15.28:0/1479699645 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3df400640 1 --2- 172.21.15.28:0/1479699645 >> v2:172.21.15.189:3300/0 conn(0x7fb3e00cabc0 0x7fb3e006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313599430 0 0) 0x7fb3e015e100 con 0x7fb3e015a540 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3c4003520 con 0x7fb3e015a540 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1971273538 0 0) 0x7fb3c4003520 con 0x7fb3e015a540 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 >> v2:172.21.15.189:3300/0 conn(0x7fb3e00cabc0 msgr2=0x7fb3e006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 --2- 172.21.15.28:0/1479699645 >> v2:172.21.15.189:3300/0 conn(0x7fb3e00cabc0 0x7fb3e006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 >> v1:172.21.15.189:6789/0 conn(0x7fb3e0070cc0 legacy=0x7fb3e006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e015ef80 con 0x7fb3e015a540 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fb3d0001670 con 0x7fb3e015a540 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.642+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 >> v1:172.21.15.28:6789/0 conn(0x7fb3e015a540 legacy=0x7fb3e015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c40039c0 0x7fb3c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3c40051d0 0x7fb3c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c400c0f0 con 0x7fb3c40047c0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c400c260 con 0x7fb3c40039c0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c400c420 con 0x7fb3c40051d0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3dfe00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c40039c0 0x7fb3c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3e4a00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3c40051d0 0x7fb3c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 -- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3c40051d0 msgr2=0x7fb3c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3c40051d0 0x7fb3c400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 -- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c40039c0 msgr2=0x7fb3c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3c40039c0 0x7fb3c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3df400640 1 -- 172.21.15.28:0/1479699645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3c4003520 con 0x7fb3c40047c0 2024-09-13T17:12:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.643+0000 7fb3e4a00640 1 -- 172.21.15.28:0/1479699645 reap_dead start 2024-09-13T17:12:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.644+0000 7fb3df400640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3c4004c40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009a90 tx=0x7fb3c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ba08fda866595 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:29.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.644+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031400 con 0x7fb3c40047c0 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc02d050 con 0x7fb3c40047c0 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3dea00640 1 -- 172.21.15.28:0/1479699645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031780 con 0x7fb3c40047c0 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 -- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 msgr2=0x7fb3c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3c4004c40 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009a90 tx=0x7fb3c40057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 -- 172.21.15.28:0/1479699645 shutdown_connections 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/1479699645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3c4004c40 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 -- 172.21.15.28:0/1479699645 >> 172.21.15.28:0/1479699645 conn(0x7fb3e0069e80 msgr2=0x7fb3e006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 -- 172.21.15.28:0/1479699645 shutdown_connections 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.645+0000 7fb3e6a00640 1 -- 172.21.15.28:0/1479699645 wait complete. 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.647+0000 7fb3e6a00640 1 Processor -- start 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.647+0000 7fb3e6a00640 1 -- start start 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 0x7fb3e014b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 0x7fb3e0150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01525a0 con 0x7fb3c40047c0 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e0152d70 con 0x7fb3e014f030 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e01527b0 con 0x7fb3e0151520 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60480/0 (socket says 172.21.15.28:60480) 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3dfe00640 1 -- 172.21.15.28:0/979355768 learned_addr learned my addr 172.21.15.28:0/979355768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3e4a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 0x7fb3e0150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3df400640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 0x7fb3e014b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.648+0000 7fb3dfe00640 1 -- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 msgr2=0x7fb3e014b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3dfe00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 0x7fb3e014b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3dfe00640 1 -- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 msgr2=0x7fb3e0150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3dfe00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 0x7fb3e0150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3dfe00640 1 -- 172.21.15.28:0/979355768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e0152a90 con 0x7fb3c40047c0 2024-09-13T17:12:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3df400640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 0x7fb3e014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3e4a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 0x7fb3e0150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3dfe00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0001200 tx=0x7fb3d0007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0007790 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d000f940 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d000fcd0 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e0150120 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.649+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e014c950 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.650+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e014ff80 con 0x7fb3c40047c0 2024-09-13T17:12:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.651+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb3d001d460 con 0x7fb3c40047c0 2024-09-13T17:12:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.652+0000 7fb3d7e00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 0x7fb3c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.653+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3d0089560 con 0x7fb3c40047c0 2024-09-13T17:12:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.653+0000 7fb3df400640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 0x7fb3c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.654+0000 7fb3df400640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 0x7fb3c006d0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc033000 tx=0x7fb3cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:29.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.657+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb3e014ff80 con 0x7fb3c40047c0 2024-09-13T17:12:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.874+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3e0150640 con 0x7fb3c40047c0 2024-09-13T17:12:29.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.874+0000 7fb3d7e00640 1 -- 172.21.15.28:0/979355768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7fb3e0150640 con 0x7fb3c40047c0 2024-09-13T17:12:29.875 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:29.875 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 msgr2=0x7fb3c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 0x7fb3c006d0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc033000 tx=0x7fb3cc008030 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 msgr2=0x7fb3e014c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fb3d0001200 tx=0x7fb3d0007660 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 shutdown_connections 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3c006ac00 0x7fb3c006d0c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3e014f030 0x7fb3e0150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3e0151520 0x7fb3e014b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 --2- 172.21.15.28:0/979355768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3c40047c0 0x7fb3e014c410 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.876+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 >> 172.21.15.28:0/979355768 conn(0x7fb3e0069e80 msgr2=0x7fb3e015d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.877+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 shutdown_connections 2024-09-13T17:12:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:29.877+0000 7fb3e6a00640 1 -- 172.21.15.28:0/979355768 wait complete. 2024-09-13T17:12:29.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 1099511627776 2024-09-13T17:12:30.064 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:12:14.384821+0000","btime":"2024-09-13T17:12:13.235853+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:12:14.384821+0000","atime":"2024-09-13T17:12:13.235853+0000","time_warp_seq":0,"change_attr":40,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:12:14.384821+0000","num_files":40,"num_subdirs":0,"change_attr":40},"rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"accounted_rstat":{"version":0,"rbytes":70,"rfiles":40,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:12:14.384821+0000"},"version":247,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"2024-09-13T17:12:23.876052+0000","last_scrub_version":244,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"lock":0,"dirfrag":1,"dirtyrstat":1,"dirty":1,"authpin":0,"scrubqueue":0},"nref":3,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyrstat"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:12:30.065 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:12:13.235853+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:12:13.235853+0000", "change_attr": 40, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:12:14.384821+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 40, "mtime": "2024-09-13T17:12:14.384821+0000", "num_files": 40, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "2024-09-13T17:12:23.876052+0000", "last_scrub_version": 244, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:12:14.384821+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 3, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "authpin": 0, "dirfrag": 1, "dirty": 1, "dirtyrstat": 1, "lock": 0, "scrubqueue": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 70, "rctime": "2024-09-13T17:12:14.384821+0000", "rfiles": 40, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyrstat" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 247, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:12:30.065 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-13T17:12:30.065 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:12:30.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:12:30.162 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:12:30.265 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:12:30.265 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:30.265 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:30.302 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:12:30.303 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:12:30.303 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:30.342 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:12:30.342 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:30.342 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:30.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:12:30.378 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:12:30.378 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:30.419 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:12:30.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.574+0000 7f2c86000640 1 Processor -- start 2024-09-13T17:12:30.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.574+0000 7f2c86000640 1 -- start start 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c86000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 0x7f2c80164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c86000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c800c3850 con 0x7f2c80159440 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c86000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8015a930 con 0x7f2c800b0b50 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c86000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80159d20 con 0x7f2c80159820 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c84c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2c80159440 0x7f2c800b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46696/0 (socket says 172.21.15.28:46696) 2024-09-13T17:12:30.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c84c00640 1 -- 172.21.15.28:0/466389252 learned_addr learned my addr 172.21.15.28:0/466389252 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:30.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.575+0000 7f2c85600640 1 --2- 172.21.15.28:0/466389252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 0x7f2c80164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707566971 0 0) 0x7f2c800c3850 con 0x7f2c80159440 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c64003520 con 0x7f2c80159440 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1377111884 0 0) 0x7f2c8015a930 con 0x7f2c800b0b50 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c800c3850 con 0x7f2c800b0b50 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c85600640 1 -- 172.21.15.28:0/466389252 >> v1:172.21.15.189:6789/0 conn(0x7f2c800b0b50 legacy=0x7f2c800b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c85600640 1 -- 172.21.15.28:0/466389252 >> v1:172.21.15.28:6789/0 conn(0x7f2c80159440 legacy=0x7f2c800b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.576+0000 7f2c85600640 1 -- 172.21.15.28:0/466389252 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8015a000 con 0x7f2c80159820 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.577+0000 7f2c85600640 1 --2- 172.21.15.28:0/466389252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 0x7f2c80164260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009dc0 tx=0x7f2c7802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8818c2eb02e8f013 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.577+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78007740 con 0x7f2c80159820 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.577+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7802d050 con 0x7f2c80159820 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.577+0000 7f2c7fe00640 1 -- 172.21.15.28:0/466389252 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c78007ac0 con 0x7f2c80159820 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 -- 172.21.15.28:0/466389252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 msgr2=0x7f2c80164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 --2- 172.21.15.28:0/466389252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 0x7f2c80164260 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2c78009dc0 tx=0x7f2c7802a760 comp rx=0 tx=0).stop 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 -- 172.21.15.28:0/466389252 shutdown_connections 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 --2- 172.21.15.28:0/466389252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159820 0x7f2c80164260 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 -- 172.21.15.28:0/466389252 >> 172.21.15.28:0/466389252 conn(0x7f2c80072e40 msgr2=0x7f2c80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 -- 172.21.15.28:0/466389252 shutdown_connections 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.578+0000 7f2c86000640 1 -- 172.21.15.28:0/466389252 wait complete. 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.580+0000 7f2c86000640 1 Processor -- start 2024-09-13T17:12:30.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.580+0000 7f2c86000640 1 -- start start 2024-09-13T17:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 0x7f2c80150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159440 0x7f2c801eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80150660 con 0x7f2c80159820 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c800a9b20 con 0x7f2c800b0b50 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c86000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80157670 con 0x7f2c80159440 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c85600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60486/0 (socket says 172.21.15.28:60486) 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c85600640 1 -- 172.21.15.28:0/580153933 learned_addr learned my addr 172.21.15.28:0/580153933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 0x7f2c80150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c77e00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159440 0x7f2c801eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 -- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159440 msgr2=0x7f2c801eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159440 0x7f2c801eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 -- 172.21.15.28:0/580153933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 msgr2=0x7f2c8006b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c84c00640 1 -- 172.21.15.28:0/580153933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78009a70 con 0x7f2c800b0b50 2024-09-13T17:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.581+0000 7f2c85600640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:30.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c84c00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 0x7f2c80150120 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000c3e0 tx=0x7f2c7000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70005ce0 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c70005e40 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c80157950 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700072f0 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.582+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c800a9e00 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.583+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c80157e70 con 0x7f2c800b0b50 2024-09-13T17:12:30.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.584+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2c70007450 con 0x7f2c800b0b50 2024-09-13T17:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.585+0000 7f2c7d600640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 0x7f2c5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.585+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2c70088950 con 0x7f2c800b0b50 2024-09-13T17:12:30.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.585+0000 7f2c77e00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 0x7f2c5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.586+0000 7f2c77e00640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 0x7f2c5c06d0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2c801eef80 tx=0x7f2c6800d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:30.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.590+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2c70053b70 con 0x7f2c800b0b50 2024-09-13T17:12:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.808+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c800b22e0 con 0x7f2c800b0b50 2024-09-13T17:12:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.809+0000 7f2c7d600640 1 -- 172.21.15.28:0/580153933 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f2c70017090 con 0x7f2c800b0b50 2024-09-13T17:12:30.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:30.811 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 msgr2=0x7f2c5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 0x7f2c5c06d0c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2c801eef80 tx=0x7f2c6800d040 comp rx=0 tx=0).stop 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 msgr2=0x7f2c80150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 0x7f2c80150120 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000c3e0 tx=0x7f2c7000c8a0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 shutdown_connections 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c5c06ac00 0x7f2c5c06d0c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80159820 0x7f2c8006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c80159440 0x7f2c801eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 --2- 172.21.15.28:0/580153933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c800b0b50 0x7f2c80150120 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 >> 172.21.15.28:0/580153933 conn(0x7f2c80072e40 msgr2=0x7f2c80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 shutdown_connections 2024-09-13T17:12:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.812+0000 7f2c86000640 1 -- 172.21.15.28:0/580153933 wait complete. 2024-09-13T17:12:30.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.983+0000 7f7015000640 1 Processor -- start 2024-09-13T17:12:30.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.984+0000 7f7015000640 1 -- start start 2024-09-13T17:12:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f7015000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70101592a0 0x7f701015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f7015000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f701006c2d0 0x7f701006c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f7015000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7010159d90 con 0x7f701006f970 2024-09-13T17:12:30.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f7015000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f701015b0b0 con 0x7f701006c2d0 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f7015000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701006ccc0 con 0x7f70101592a0 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f701006f970 0x7f7010159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46712/0 (socket says 172.21.15.28:46712) 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700d400640 1 -- 172.21.15.28:0/3907082765 learned_addr learned my addr 172.21.15.28:0/3907082765 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f701006c2d0 0x7f701006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700de00640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70101592a0 0x7f701015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700e800640 1 -- 172.21.15.28:0/3907082765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70101592a0 msgr2=0x7f701015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700e800640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70101592a0 0x7f701015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.985+0000 7f700e800640 1 -- 172.21.15.28:0/3907082765 >> v1:172.21.15.28:6789/0 conn(0x7f701006f970 legacy=0x7f7010159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.986+0000 7f700e800640 1 -- 172.21.15.28:0/3907082765 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70100b2040 con 0x7f701006c2d0 2024-09-13T17:12:30.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.986+0000 7f700ca00640 1 -- 172.21.15.28:0/3907082765 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986956496 0 0) 0x7f7010159d90 con 0x7f701006f970 2024-09-13T17:12:30.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f700e800640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701006c2d0 0x7f701006c780 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7008009d90 tx=0x7f700802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9a3af626c6ce9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f700ca00640 1 -- 172.21.15.28:0/3907082765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008007740 con 0x7f701006c2d0 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f700ca00640 1 -- 172.21.15.28:0/3907082765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700802d050 con 0x7f701006c2d0 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f700ca00640 1 -- 172.21.15.28:0/3907082765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008007ac0 con 0x7f701006c2d0 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 -- 172.21.15.28:0/3907082765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701006c2d0 msgr2=0x7f701006c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701006c2d0 0x7f701006c780 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f7008009d90 tx=0x7f700802a660 comp rx=0 tx=0).stop 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 -- 172.21.15.28:0/3907082765 shutdown_connections 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f701006c2d0 0x7f701006c780 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 --2- 172.21.15.28:0/3907082765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70101592a0 0x7f701015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 -- 172.21.15.28:0/3907082765 >> 172.21.15.28:0/3907082765 conn(0x7f7010069e80 msgr2=0x7f701006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 -- 172.21.15.28:0/3907082765 shutdown_connections 2024-09-13T17:12:30.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.987+0000 7f7015000640 1 -- 172.21.15.28:0/3907082765 wait complete. 2024-09-13T17:12:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.989+0000 7f7015000640 1 Processor -- start 2024-09-13T17:12:30.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.989+0000 7f7015000640 1 -- start start 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f701006c2d0 0x7f70100adda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 0x7f7010166c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7010167160 con 0x7f70101592a0 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ae4a0 con 0x7f701006f970 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f7015000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100ae780 con 0x7f701006c2d0 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f700e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f700e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60506/0 (socket says 172.21.15.28:60506) 2024-09-13T17:12:30.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.990+0000 7f700e800640 1 -- 172.21.15.28:0/908863237 learned_addr learned my addr 172.21.15.28:0/908863237 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700d400640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 0x7f7010166c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700de00640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f701006c2d0 0x7f70100adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 -- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f701006c2d0 msgr2=0x7f70100adda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f701006c2d0 0x7f70100adda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 -- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 msgr2=0x7f7010166c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 0x7f7010166c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 -- 172.21.15.28:0/908863237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7008009a70 con 0x7f70101592a0 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700d400640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 0x7f7010166c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:30.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f700e800640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7008009d90 tx=0x7f7008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008003b90 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7008003cf0 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70100aea60 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.991+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7008004230 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.992+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70100af020 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.993+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100b3740 con 0x7f70101592a0 2024-09-13T17:12:30.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.993+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7008004390 con 0x7f70101592a0 2024-09-13T17:12:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.994+0000 7f7006000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 0x7f6ff006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.994+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f70080aaac0 con 0x7f70101592a0 2024-09-13T17:12:30.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.995+0000 7f700de00640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 0x7f6ff006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:30.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.995+0000 7f700de00640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 0x7f6ff006d0c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7000006fd0 tx=0x7f7000008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:30.999+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7008075ba0 con 0x7f70101592a0 2024-09-13T17:12:31.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.181+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f701006ca80 con 0x7f70101592a0 2024-09-13T17:12:31.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.183+0000 7f7006000640 1 -- 172.21.15.28:0/908863237 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16694 (secure 0 0 0) 0x7f7008040090 con 0x7f70101592a0 2024-09-13T17:12:31.184 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:31.184 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:17.578249+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/693307152":"2024-09-14T17:12:17.578224+0000","172.21.15.189:6835/693307152":"2024-09-14T17:12:17.578224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 msgr2=0x7f6ff006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 0x7f6ff006d0c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7000006fd0 tx=0x7f7000008040 comp rx=0 tx=0).stop 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 msgr2=0x7f70100af540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f7008009d90 tx=0x7f7008004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 shutdown_connections 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ff006ac00 0x7f6ff006d0c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70101592a0 0x7f70100af540 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f701006f970 0x7f7010166c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 --2- 172.21.15.28:0/908863237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f701006c2d0 0x7f70100adda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 >> 172.21.15.28:0/908863237 conn(0x7f7010069e80 msgr2=0x7f701006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 shutdown_connections 2024-09-13T17:12:31.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.185+0000 7f7015000640 1 -- 172.21.15.28:0/908863237 wait complete. 2024-09-13T17:12:31.202 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:12:02.552870+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:12:03.628750+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:12:31.203 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:12:31.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.361+0000 7fec6d000640 1 Processor -- start 2024-09-13T17:12:31.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.361+0000 7fec6d000640 1 -- start start 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680cd260 0x7fec68070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec6d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec68070480 con 0x7fec68156650 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec6d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec6815a010 con 0x7fec68159b30 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806fb50 con 0x7fec680cd260 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec67400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fec68156650 0x7fec6815a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46728/0 (socket says 172.21.15.28:46728) 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec67400640 1 -- 172.21.15.28:0/773618238 learned_addr learned my addr 172.21.15.28:0/773618238 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:31.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec67e00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680cd260 0x7fec68070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.363+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2651907463 0 0) 0x7fec68070480 con 0x7fec68156650 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec4c003440 con 0x7fec68156650 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824111652 0 0) 0x7fec6815a010 con 0x7fec68159b30 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec68070480 con 0x7fec68159b30 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4272530429 0 0) 0x7fec4c003440 con 0x7fec68156650 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680cd260 msgr2=0x7fec68070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680cd260 0x7fec68070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 >> v1:172.21.15.189:6789/0 conn(0x7fec68159b30 legacy=0x7fec680ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6806fe30 con 0x7fec68156650 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fec58001670 con 0x7fec68156650 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 >> v1:172.21.15.28:6789/0 conn(0x7fec68156650 legacy=0x7fec6815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c003c20 0x7fec4c008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec4c00ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c00b450 0x7fec4c00d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806fb50 con 0x7fec4c004100 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c00e170 con 0x7fec4c003c20 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c00e330 con 0x7fec4c00b450 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec67e00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680cd260 0x7fec68070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec5fe00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec4c00ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.364+0000 7fec67400640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c003c20 0x7fec4c008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec67e00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c00b450 0x7fec4c00d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec5fe00640 1 -- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c00b450 msgr2=0x7fec4c00d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec5fe00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c00b450 0x7fec4c00d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec5fe00640 1 -- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c003c20 msgr2=0x7fec4c008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec5fe00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c003c20 0x7fec4c008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.365+0000 7fec5fe00640 1 -- 172.21.15.28:0/773618238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c003440 con 0x7fec4c004100 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.366+0000 7fec5fe00640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec4c00ade0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fec500018a0 tx=0x7fec4c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657e177fdc7ab9b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.366+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec500052d0 con 0x7fec4c004100 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.366+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec50005430 con 0x7fec4c004100 2024-09-13T17:12:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.366+0000 7fec66a00640 1 -- 172.21.15.28:0/773618238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5002ac60 con 0x7fec4c004100 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 -- 172.21.15.28:0/773618238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 msgr2=0x7fec4c00ade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 --2- 172.21.15.28:0/773618238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec4c00ade0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fec500018a0 tx=0x7fec4c0036f0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec67e00640 1 -- 172.21.15.28:0/773618238 reap_dead start 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 -- 172.21.15.28:0/773618238 shutdown_connections 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 -- 172.21.15.28:0/773618238 >> 172.21.15.28:0/773618238 conn(0x7fec68069e80 msgr2=0x7fec6806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 -- 172.21.15.28:0/773618238 shutdown_connections 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.367+0000 7fec6d000640 1 -- 172.21.15.28:0/773618238 wait complete. 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.368+0000 7fec6d000640 1 Processor -- start 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.369+0000 7fec6d000640 1 -- start start 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.369+0000 7fec6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.369+0000 7fec6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680b0150 0x7fec6806b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec6d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 0x7fec681f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec6d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6806bcf0 con 0x7fec4c004100 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec6d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec681f3280 con 0x7fec6806b870 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec680ac8f0 con 0x7fec680b0150 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60532/0 (socket says 172.21.15.28:60532) 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 -- 172.21.15.28:0/385464730 learned_addr learned my addr 172.21.15.28:0/385464730 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec5fe00640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680b0150 0x7fec6806b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67e00640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 0x7fec681f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 -- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680b0150 msgr2=0x7fec6806b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680b0150 0x7fec6806b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 -- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 msgr2=0x7fec681f2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 0x7fec681f2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67400640 1 -- 172.21.15.28:0/385464730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec680acbd0 con 0x7fec4c004100 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.370+0000 7fec67e00640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 0x7fec681f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec67400640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fec580017a0 tx=0x7fec58007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800f750 con 0x7fec4c004100 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5800f8d0 con 0x7fec4c004100 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5800fc80 con 0x7fec4c004100 2024-09-13T17:12:31.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec680aceb0 con 0x7fec4c004100 2024-09-13T17:12:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.371+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec680b05b0 con 0x7fec4c004100 2024-09-13T17:12:31.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.372+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec680b0770 con 0x7fec4c004100 2024-09-13T17:12:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.373+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fec5801d460 con 0x7fec4c004100 2024-09-13T17:12:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.374+0000 7fec5f400640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 0x7fec4406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.374+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fec58089440 con 0x7fec4c004100 2024-09-13T17:12:31.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.374+0000 7fec5fe00640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 0x7fec4406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.375+0000 7fec5fe00640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 0x7fec4406d0f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fec50032000 tx=0x7fec50009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.378+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fec680b0770 con 0x7fec4c004100 2024-09-13T17:12:31.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.556+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec680ad3d0 con 0x7fec4c004100 2024-09-13T17:12:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.557+0000 7fec5f400640 1 -- 172.21.15.28:0/385464730 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v65) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec680ad3d0 con 0x7fec4c004100 2024-09-13T17:12:31.558 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:12:31.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 msgr2=0x7fec4406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 0x7fec4406d0f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fec50032000 tx=0x7fec50009030 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 msgr2=0x7fec6815a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fec580017a0 tx=0x7fec58007430 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 shutdown_connections 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec4406ac30 0x7fec4406d0f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec6806b870 0x7fec681f2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec680b0150 0x7fec6806b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 --2- 172.21.15.28:0/385464730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c004100 0x7fec6815a9d0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 >> 172.21.15.28:0/385464730 conn(0x7fec68069e80 msgr2=0x7fec681552a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:31.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 shutdown_connections 2024-09-13T17:12:31.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.559+0000 7fec6d000640 1 -- 172.21.15.28:0/385464730 wait complete. 2024-09-13T17:12:31.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:12:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.733+0000 7fcf02e00640 1 Processor -- start 2024-09-13T17:12:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.733+0000 7fcf02e00640 1 -- start start 2024-09-13T17:12:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.734+0000 7fcf02e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcefc070430 0x7fcefc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.734+0000 7fcf02e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc15ab60 con 0x7fcefc06f3f0 2024-09-13T17:12:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.734+0000 7fcf02e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcefc06f920 con 0x7fcefc159770 2024-09-13T17:12:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.734+0000 7fcf02e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcefc15b290 con 0x7fcefc070430 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcef3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcefc070430 0x7fcefc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcf00e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcefc06f3f0 0x7fcefc15c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46730/0 (socket says 172.21.15.28:46730) 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcf00e00640 1 -- 172.21.15.28:0/4043831030 learned_addr learned my addr 172.21.15.28:0/4043831030 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcef3e00640 1 -- 172.21.15.28:0/4043831030 >> v1:172.21.15.189:6789/0 conn(0x7fcefc159770 legacy=0x7fcefc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcef3e00640 1 -- 172.21.15.28:0/4043831030 >> v1:172.21.15.28:6789/0 conn(0x7fcefc06f3f0 legacy=0x7fcefc15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.735+0000 7fcef3e00640 1 -- 172.21.15.28:0/4043831030 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc15b570 con 0x7fcefc070430 2024-09-13T17:12:31.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.736+0000 7fcef3e00640 1 --2- 172.21.15.28:0/4043831030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc15bd70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcee40097a0 tx=0x7fcee402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a1ed9d6f664aba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.736+0000 7fcefb400640 1 -- 172.21.15.28:0/4043831030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4032440 con 0x7fcefc070430 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.736+0000 7fcefb400640 1 -- 172.21.15.28:0/4043831030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee40325a0 con 0x7fcefc070430 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.736+0000 7fcefb400640 1 -- 172.21.15.28:0/4043831030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4032930 con 0x7fcefc070430 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 -- 172.21.15.28:0/4043831030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 msgr2=0x7fcefc15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 --2- 172.21.15.28:0/4043831030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc15bd70 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fcee40097a0 tx=0x7fcee402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 -- 172.21.15.28:0/4043831030 shutdown_connections 2024-09-13T17:12:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 --2- 172.21.15.28:0/4043831030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc15bd70 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 -- 172.21.15.28:0/4043831030 >> 172.21.15.28:0/4043831030 conn(0x7fcefc0b0270 msgr2=0x7fcefc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 -- 172.21.15.28:0/4043831030 shutdown_connections 2024-09-13T17:12:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.737+0000 7fcf02e00640 1 -- 172.21.15.28:0/4043831030 wait complete. 2024-09-13T17:12:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.739+0000 7fcf02e00640 1 Processor -- start 2024-09-13T17:12:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.739+0000 7fcf02e00640 1 -- start start 2024-09-13T17:12:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcefc06f3f0 0x7fcefc14d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 0x7fcefc152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc14ebb0 con 0x7fcefc070430 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc150040 con 0x7fcefc159770 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf02e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc14dfa0 con 0x7fcefc06f3f0 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60550/0 (socket says 172.21.15.28:60550) 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 -- 172.21.15.28:0/3619231798 learned_addr learned my addr 172.21.15.28:0/3619231798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf00e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 0x7fcefc152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcefbe00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcefc06f3f0 0x7fcefc14d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 -- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcefc06f3f0 msgr2=0x7fcefc14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcefc06f3f0 0x7fcefc14d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 -- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 msgr2=0x7fcefc152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 0x7fcefc152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcef3e00640 1 -- 172.21.15.28:0/3619231798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee4009400 con 0x7fcefc070430 2024-09-13T17:12:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.740+0000 7fcf00e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 0x7fcefc152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcef3e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcee4035040 tx=0x7fcee40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4006e10 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcee4030980 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc14e280 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcee4030d50 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.741+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc14e840 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.742+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc14ed60 con 0x7fcefc070430 2024-09-13T17:12:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.743+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcee4039040 con 0x7fcefc070430 2024-09-13T17:12:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.744+0000 7fcef8c00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 0x7fced806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.744+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcee40a9de0 con 0x7fcefc070430 2024-09-13T17:12:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.744+0000 7fcefbe00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 0x7fced806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.745+0000 7fcefbe00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 0x7fced806d0f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fceec000f60 tx=0x7fceec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:31.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.748+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcefc14ed60 con 0x7fcefc070430 2024-09-13T17:12:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.926+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcefc14ecf0 con 0x7fcefc070430 2024-09-13T17:12:31.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.926+0000 7fcef8c00640 1 -- 172.21.15.28:0/3619231798 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v65) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcee4074f50 con 0x7fcefc070430 2024-09-13T17:12:31.928 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 msgr2=0x7fced806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 0x7fced806d0f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fceec000f60 tx=0x7fceec009040 comp rx=0 tx=0).stop 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 msgr2=0x7fcefc14d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fcee4035040 tx=0x7fcee40046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 shutdown_connections 2024-09-13T17:12:31.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fced806ac30 0x7fced806d0f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcefc159770 0x7fcefc152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcefc070430 0x7fcefc14d0d0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 --2- 172.21.15.28:0/3619231798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcefc06f3f0 0x7fcefc14d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 >> 172.21.15.28:0/3619231798 conn(0x7fcefc0b0270 msgr2=0x7fcefc157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 shutdown_connections 2024-09-13T17:12:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:31.929+0000 7fcf02e00640 1 -- 172.21.15.28:0/3619231798 wait complete. 2024-09-13T17:12:31.947 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.104+0000 7f9a74000640 1 Processor -- start 2024-09-13T17:12:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.104+0000 7f9a74000640 1 -- start start 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a74000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9a6c070430 0x7f9a6c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a74000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a74000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6c15ab60 con 0x7f9a6c159770 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a74000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c06f920 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a74000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c15b290 con 0x7f9a6c070430 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a72000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a72000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60552/0 (socket says 172.21.15.28:60552) 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.105+0000 7f9a72000640 1 -- 172.21.15.28:0/3475384826 learned_addr learned my addr 172.21.15.28:0/3475384826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.106+0000 7f9a70c00640 1 --2- 172.21.15.28:0/3475384826 >> v2:172.21.15.189:3300/0 conn(0x7f9a6c070430 0x7f9a6c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.106+0000 7f9a72000640 1 -- 172.21.15.28:0/3475384826 >> v2:172.21.15.189:3300/0 conn(0x7f9a6c070430 msgr2=0x7f9a6c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.106+0000 7f9a72000640 1 --2- 172.21.15.28:0/3475384826 >> v2:172.21.15.189:3300/0 conn(0x7f9a6c070430 0x7f9a6c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.106+0000 7f9a72000640 1 -- 172.21.15.28:0/3475384826 >> v1:172.21.15.189:6789/0 conn(0x7f9a6c159770 legacy=0x7f9a6c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.106+0000 7f9a72000640 1 -- 172.21.15.28:0/3475384826 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c15b570 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.107+0000 7f9a72000640 1 --2- 172.21.15.28:0/3475384826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c06b950 tx=0x7f9a6402c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d92feb4c76b85 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.107+0000 7f9a6be00640 1 -- 172.21.15.28:0/3475384826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64034420 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.107+0000 7f9a6be00640 1 -- 172.21.15.28:0/3475384826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a64034580 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.107+0000 7f9a6be00640 1 -- 172.21.15.28:0/3475384826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a64034910 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 -- 172.21.15.28:0/3475384826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 msgr2=0x7f9a6c15c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 --2- 172.21.15.28:0/3475384826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c06b950 tx=0x7f9a6402c870 comp rx=0 tx=0).stop 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 -- 172.21.15.28:0/3475384826 shutdown_connections 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 --2- 172.21.15.28:0/3475384826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c15c2b0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 --2- 172.21.15.28:0/3475384826 >> v2:172.21.15.189:3300/0 conn(0x7f9a6c070430 0x7f9a6c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 -- 172.21.15.28:0/3475384826 >> 172.21.15.28:0/3475384826 conn(0x7f9a6c0b0270 msgr2=0x7f9a6c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 -- 172.21.15.28:0/3475384826 shutdown_connections 2024-09-13T17:12:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.108+0000 7f9a74000640 1 -- 172.21.15.28:0/3475384826 wait complete. 2024-09-13T17:12:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.110+0000 7f9a74000640 1 Processor -- start 2024-09-13T17:12:32.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.110+0000 7f9a74000640 1 -- start start 2024-09-13T17:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a6c070430 0x7f9a6c14e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a6c159770 0x7f9a6c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c155020 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c155300 con 0x7f9a6c070430 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a74000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c1555e0 con 0x7f9a6c159770 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60566/0 (socket says 172.21.15.28:60566) 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 -- 172.21.15.28:0/1910832995 learned_addr learned my addr 172.21.15.28:0/1910832995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a70c00640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a6c070430 0x7f9a6c14e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a72000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a6c159770 0x7f9a6c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 -- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a6c159770 msgr2=0x7f9a6c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a6c159770 0x7f9a6c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 -- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a6c070430 msgr2=0x7f9a6c14e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a6c070430 0x7f9a6c14e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.111+0000 7f9a71600640 1 -- 172.21.15.28:0/1910832995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a64009b30 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a71600640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000b3f0 tx=0x7f9a6000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60011830 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a60011990 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a60011d40 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c1558c0 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.112+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c154d30 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.113+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c06fd20 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.114+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9a6001a400 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.115+0000 7f9a69600640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 0x7f9a4c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.115+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9a60089070 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.116+0000 7f9a70c00640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 0x7f9a4c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.116+0000 7f9a70c00640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 0x7f9a4c06d0f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0ad470 tx=0x7f9a58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.119+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9a600bc9c0 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.299+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9a6c155160 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.300+0000 7f9a69600640 1 -- 172.21.15.28:0/1910832995 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v65) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9a6c155160 con 0x7f9a6c06f3f0 2024-09-13T17:12:32.301 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:12:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 msgr2=0x7f9a4c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 0x7f9a4c06d0f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9a6c0ad470 tx=0x7f9a58008040 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 msgr2=0x7f9a6c1547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000b3f0 tx=0x7f9a6000b8b0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 shutdown_connections 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9a4c06ac30 0x7f9a4c06d0f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a6c159770 0x7f9a6c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a6c070430 0x7f9a6c14e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 --2- 172.21.15.28:0/1910832995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a6c06f3f0 0x7f9a6c1547f0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 >> 172.21.15.28:0/1910832995 conn(0x7f9a6c0b0270 msgr2=0x7f9a6c0b3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:32.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 shutdown_connections 2024-09-13T17:12:32.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.302+0000 7f9a74000640 1 -- 172.21.15.28:0/1910832995 wait complete. 2024-09-13T17:12:32.319 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.474+0000 7f3c01800640 1 Processor -- start 2024-09-13T17:12:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.474+0000 7f3c01800640 1 -- start start 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3c01800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3bfc15a540 0x7f3bfc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3c01800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc0cabc0 0x7f3bfc15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3c01800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bfc15e100 con 0x7f3bfc15edb0 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3c01800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bfc0adb00 con 0x7f3bfc15a540 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3c01800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc15e3e0 con 0x7f3bfc0cabc0 2024-09-13T17:12:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3bfb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3bfc15edb0 0x7f3bfc070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46742/0 (socket says 172.21.15.28:46742) 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.475+0000 7f3bfb400640 1 -- 172.21.15.28:0/3046173311 learned_addr learned my addr 172.21.15.28:0/3046173311 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/3046173311 >> v2:172.21.15.189:3300/0 conn(0x7f3bfc15a540 0x7f3bfc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bf2000640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc0cabc0 0x7f3bfc15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfa000640 1 -- 172.21.15.28:0/3046173311 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772065410 0 0) 0x7f3bfc15e100 con 0x7f3bfc15edb0 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfa000640 1 -- 172.21.15.28:0/3046173311 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be0003520 con 0x7f3bfc15edb0 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfaa00640 1 -- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc0cabc0 msgr2=0x7f3bfc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc0cabc0 0x7f3bfc15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfaa00640 1 -- 172.21.15.28:0/3046173311 >> v1:172.21.15.28:6789/0 conn(0x7f3bfc15edb0 legacy=0x7f3bfc070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.476+0000 7f3bfaa00640 1 -- 172.21.15.28:0/3046173311 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bfc06ce80 con 0x7f3bfc15a540 2024-09-13T17:12:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.477+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc15a540 0x7f3bfc06c430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514a5f658e51ddd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3bfa000640 1 -- 172.21.15.28:0/3046173311 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec02ae50 con 0x7f3bfc15a540 2024-09-13T17:12:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3bfa000640 1 -- 172.21.15.28:0/3046173311 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bec032440 con 0x7f3bfc15a540 2024-09-13T17:12:32.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3bfa000640 1 -- 172.21.15.28:0/3046173311 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bec0327d0 con 0x7f3bfc15a540 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 -- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc15a540 msgr2=0x7f3bfc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc15a540 0x7f3bfc06c430 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3bec0097a0 tx=0x7f3bec02a100 comp rx=0 tx=0).stop 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 -- 172.21.15.28:0/3046173311 shutdown_connections 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc0cabc0 0x7f3bfc15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 --2- 172.21.15.28:0/3046173311 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc15a540 0x7f3bfc06c430 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 -- 172.21.15.28:0/3046173311 >> 172.21.15.28:0/3046173311 conn(0x7f3bfc069e80 msgr2=0x7f3bfc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 -- 172.21.15.28:0/3046173311 shutdown_connections 2024-09-13T17:12:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.478+0000 7f3c01800640 1 -- 172.21.15.28:0/3046173311 wait complete. 2024-09-13T17:12:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.480+0000 7f3c01800640 1 Processor -- start 2024-09-13T17:12:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.480+0000 7f3c01800640 1 -- start start 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc0cabc0 0x7f3bfc1f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 0x7f3bfc151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc1f3840 con 0x7f3bfc15a540 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc15baf0 con 0x7f3bfc0cabc0 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3c01800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bfc14e5f0 con 0x7f3bfc14b260 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3bf2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3bf2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60568/0 (socket says 172.21.15.28:60568) 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3bf2000640 1 -- 172.21.15.28:0/1736042123 learned_addr learned my addr 172.21.15.28:0/1736042123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3bfb400640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 0x7f3bfc151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.481+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc0cabc0 0x7f3bfc1f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 -- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 msgr2=0x7f3bfc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 0x7f3bfc151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 -- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc0cabc0 msgr2=0x7f3bfc1f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc0cabc0 0x7f3bfc1f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 -- 172.21.15.28:0/1736042123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec009400 con 0x7f3bfc15a540 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bfb400640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 0x7f3bfc151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf2000640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3be400e980 tx=0x7f3be400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4009670 con 0x7f3bfc15a540 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be400ca00 con 0x7f3bfc15a540 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be400cd90 con 0x7f3bfc15a540 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bfc14e890 con 0x7f3bfc15a540 2024-09-13T17:12:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.482+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bfc14ee50 con 0x7f3bfc15a540 2024-09-13T17:12:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.483+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bfc15e850 con 0x7f3bfc15a540 2024-09-13T17:12:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.484+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3be4013470 con 0x7f3bfc15a540 2024-09-13T17:12:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.485+0000 7f3bf3400640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 0x7f3bd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.485+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3be4089550 con 0x7f3bfc15a540 2024-09-13T17:12:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.486+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 0x7f3bd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.486+0000 7f3bfaa00640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 0x7f3bd806d0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3bec034000 tx=0x7f3bec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.489+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3be4054770 con 0x7f3bfc15a540 2024-09-13T17:12:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.703+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bfc14efe0 con 0x7f3bfc15a540 2024-09-13T17:12:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.703+0000 7f3bf3400640 1 -- 172.21.15.28:0/1736042123 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f3be4057d90 con 0x7f3bfc15a540 2024-09-13T17:12:32.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:32.705 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.705+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 msgr2=0x7f3bd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.705+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 0x7f3bd806d0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3bec034000 tx=0x7f3bec02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.705+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 msgr2=0x7f3bfc14e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.705+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f3be400e980 tx=0x7f3be400ee40 comp rx=0 tx=0).stop 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 shutdown_connections 2024-09-13T17:12:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd806ac00 0x7f3bd806d0c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bfc14b260 0x7f3bfc151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bfc15a540 0x7f3bfc14e0b0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 --2- 172.21.15.28:0/1736042123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bfc0cabc0 0x7f3bfc1f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 >> 172.21.15.28:0/1736042123 conn(0x7f3bfc069e80 msgr2=0x7f3bfc0ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 shutdown_connections 2024-09-13T17:12:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.706+0000 7f3c01800640 1 -- 172.21.15.28:0/1736042123 wait complete. 2024-09-13T17:12:32.724 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:32.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.880+0000 7fca8e600640 1 Processor -- start 2024-09-13T17:12:32.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.880+0000 7fca8e600640 1 -- start start 2024-09-13T17:12:32.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.881+0000 7fca8e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca8806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.881+0000 7fca8e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca880cabc0 0x7fca8815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.881+0000 7fca8e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca8815e100 con 0x7fca8815edb0 2024-09-13T17:12:32.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.881+0000 7fca8e600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca880adb00 con 0x7fca880cabc0 2024-09-13T17:12:32.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.881+0000 7fca8e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8815e3e0 con 0x7fca8815a540 2024-09-13T17:12:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca880cabc0 0x7fca8815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca87400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca8806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca880cabc0 0x7fca8815e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:56304/0 (socket says 172.21.15.28:56304) 2024-09-13T17:12:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 -- 172.21.15.28:0/3855290126 learned_addr learned my addr 172.21.15.28:0/3855290126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:32.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca87e00640 1 --1- 172.21.15.28:0/3855290126 >> v1:172.21.15.189:6789/0 conn(0x7fca8815edb0 0x7fca88070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:44388/0 (socket says 172.21.15.28:44388) 2024-09-13T17:12:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 -- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 msgr2=0x7fca8806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 --2- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca8806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 -- 172.21.15.28:0/3855290126 >> v1:172.21.15.189:6789/0 conn(0x7fca8815edb0 legacy=0x7fca88070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.882+0000 7fca7fe00640 1 -- 172.21.15.28:0/3855290126 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8806ce80 con 0x7fca880cabc0 2024-09-13T17:12:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.883+0000 7fca7fe00640 1 --2- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca880cabc0 0x7fca8815e870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca7002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548cc6f07a7ff058 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.883+0000 7fca86a00640 1 -- 172.21.15.28:0/3855290126 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70032440 con 0x7fca880cabc0 2024-09-13T17:12:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.883+0000 7fca86a00640 1 -- 172.21.15.28:0/3855290126 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca700325a0 con 0x7fca880cabc0 2024-09-13T17:12:32.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.883+0000 7fca86a00640 1 -- 172.21.15.28:0/3855290126 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70032930 con 0x7fca880cabc0 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 -- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca880cabc0 msgr2=0x7fca8815e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 --2- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca880cabc0 0x7fca8815e870 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca7002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 -- 172.21.15.28:0/3855290126 shutdown_connections 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 --2- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca880cabc0 0x7fca8815e870 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 --2- 172.21.15.28:0/3855290126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca8806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 -- 172.21.15.28:0/3855290126 >> 172.21.15.28:0/3855290126 conn(0x7fca88069e80 msgr2=0x7fca8806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 -- 172.21.15.28:0/3855290126 shutdown_connections 2024-09-13T17:12:32.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.884+0000 7fca8e600640 1 -- 172.21.15.28:0/3855290126 wait complete. 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.886+0000 7fca8e600640 1 Processor -- start 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.886+0000 7fca8e600640 1 -- start start 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca88153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 0x7fca881544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88153090 con 0x7fca880cabc0 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca88152db0 con 0x7fca8814eca0 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca8e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca8814fe80 con 0x7fca8815a540 2024-09-13T17:12:32.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60570/0 (socket says 172.21.15.28:60570) 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 -- 172.21.15.28:0/1073530742 learned_addr learned my addr 172.21.15.28:0/1073530742 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca7fe00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca88153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 -- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 msgr2=0x7fca88153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca88153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.887+0000 7fca87400640 1 -- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 msgr2=0x7fca881544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca87e00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 0x7fca881544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca87400640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 0x7fca881544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca87400640 1 -- 172.21.15.28:0/1073530742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700094d0 con 0x7fca880cabc0 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca7fe00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca88153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca87e00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 0x7fca881544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca87400640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e9b0 tx=0x7fca7800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca78009670 con 0x7fca880cabc0 2024-09-13T17:12:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7800c9a0 con 0x7fca880cabc0 2024-09-13T17:12:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7800cd50 con 0x7fca880cabc0 2024-09-13T17:12:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca88150160 con 0x7fca880cabc0 2024-09-13T17:12:32.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.888+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca88154a20 con 0x7fca880cabc0 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.889+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca8814fce0 con 0x7fca880cabc0 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.890+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca78013470 con 0x7fca880cabc0 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.891+0000 7fca7f400640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 0x7fca6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.891+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fca78089660 con 0x7fca880cabc0 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.892+0000 7fca7fe00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 0x7fca6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:32.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.893+0000 7fca7fe00640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 0x7fca6406d0c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fca70034000 tx=0x7fca7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:32.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:32.895+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fca8814fce0 con 0x7fca880cabc0 2024-09-13T17:12:33.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.079+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca8814fce0 con 0x7fca880cabc0 2024-09-13T17:12:33.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.080+0000 7fca7f400640 1 -- 172.21.15.28:0/1073530742 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fca8814fce0 con 0x7fca880cabc0 2024-09-13T17:12:33.081 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:33.081 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:17.578249+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/693307152":"2024-09-14T17:12:17.578224+0000","172.21.15.189:6835/693307152":"2024-09-14T17:12:17.578224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 msgr2=0x7fca6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 0x7fca6406d0c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fca70034000 tx=0x7fca7002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 msgr2=0x7fca8813baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fca7800e9b0 tx=0x7fca7800ee70 comp rx=0 tx=0).stop 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 shutdown_connections 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca6406ac00 0x7fca6406d0c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca8814eca0 0x7fca881544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca8815a540 0x7fca88153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 --2- 172.21.15.28:0/1073530742 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca880cabc0 0x7fca8813baa0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 >> 172.21.15.28:0/1073530742 conn(0x7fca88069e80 msgr2=0x7fca8815b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 shutdown_connections 2024-09-13T17:12:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.082+0000 7fca8e600640 1 -- 172.21.15.28:0/1073530742 wait complete. 2024-09-13T17:12:33.102 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:12:33.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.257+0000 7f9eaac00640 1 Processor -- start 2024-09-13T17:12:33.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.257+0000 7f9eaac00640 1 -- start start 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.258+0000 7f9eaac00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ea40713a0 0x7f9ea4161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.258+0000 7f9eaac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea406cde0 0x7f9ea406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.258+0000 7f9eaac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea40b35c0 con 0x7f9ea415e860 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.258+0000 7f9eaac00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ea41618f0 con 0x7f9ea40713a0 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.258+0000 7f9eaac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea40b38a0 con 0x7f9ea406cde0 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ea40713a0 0x7f9ea4161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9ea415e860 0x7f9ea4070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:46748/0 (socket says 172.21.15.28:46748) 2024-09-13T17:12:33.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3e00640 1 -- 172.21.15.28:0/4210726621 learned_addr learned my addr 172.21.15.28:0/4210726621 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea8c00640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea406cde0 0x7f9ea406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3400640 1 -- 172.21.15.28:0/4210726621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea406cde0 msgr2=0x7f9ea406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3400640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea406cde0 0x7f9ea406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3400640 1 -- 172.21.15.28:0/4210726621 >> v1:172.21.15.28:6789/0 conn(0x7f9ea415e860 legacy=0x7f9ea4070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.259+0000 7f9ea3400640 1 -- 172.21.15.28:0/4210726621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea4161230 con 0x7f9ea40713a0 2024-09-13T17:12:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.260+0000 7f9ea3400640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea40713a0 0x7f9ea4161d10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9e900097a0 tx=0x7f9e9002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4458cc23b2227977 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.260+0000 7f9ea2a00640 1 -- 172.21.15.28:0/4210726621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9002adb0 con 0x7f9ea40713a0 2024-09-13T17:12:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.260+0000 7f9ea2a00640 1 -- 172.21.15.28:0/4210726621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e90032440 con 0x7f9ea40713a0 2024-09-13T17:12:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.260+0000 7f9ea2a00640 1 -- 172.21.15.28:0/4210726621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e900327d0 con 0x7f9ea40713a0 2024-09-13T17:12:33.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 -- 172.21.15.28:0/4210726621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea40713a0 msgr2=0x7f9ea4161d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea40713a0 0x7f9ea4161d10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f9e900097a0 tx=0x7f9e9002a060 comp rx=0 tx=0).stop 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 -- 172.21.15.28:0/4210726621 shutdown_connections 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea406cde0 0x7f9ea406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 --2- 172.21.15.28:0/4210726621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea40713a0 0x7f9ea4161d10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 -- 172.21.15.28:0/4210726621 >> 172.21.15.28:0/4210726621 conn(0x7f9ea4069e80 msgr2=0x7f9ea406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 -- 172.21.15.28:0/4210726621 shutdown_connections 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.261+0000 7f9eaac00640 1 -- 172.21.15.28:0/4210726621 wait complete. 2024-09-13T17:12:33.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.263+0000 7f9eaac00640 1 Processor -- start 2024-09-13T17:12:33.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.263+0000 7f9eaac00640 1 -- start start 2024-09-13T17:12:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ea40713a0 0x7f9ea40b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea415e860 0x7f9ea40b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea415a690 con 0x7f9ea406cde0 2024-09-13T17:12:33.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea4157f90 con 0x7f9ea40713a0 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9eaac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea4165430 con 0x7f9ea415e860 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60592/0 (socket says 172.21.15.28:60592) 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2391195725 learned_addr learned my addr 172.21.15.28:0/2391195725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea415e860 msgr2=0x7f9ea40b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea415e860 0x7f9ea40b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ea40713a0 msgr2=0x7f9ea40b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ea40713a0 0x7f9ea40b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.264+0000 7f9ea3e00640 1 -- 172.21.15.28:0/2391195725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90009400 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9ea3e00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c4d0 tx=0x7f9e9400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94005ce0 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e94005e40 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007270 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea4154900 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.265+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea4154ec0 con 0x7f9ea406cde0 2024-09-13T17:12:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.266+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea40b31c0 con 0x7f9ea406cde0 2024-09-13T17:12:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.267+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9e940073d0 con 0x7f9ea406cde0 2024-09-13T17:12:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.268+0000 7f9e9be00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 0x7f9e8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.269+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9e94088860 con 0x7f9ea406cde0 2024-09-13T17:12:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.269+0000 7f9ea3400640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 0x7f9e8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.270+0000 7f9ea3400640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 0x7f9e8406d0c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9e90034000 tx=0x7f9e9002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.273+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9e940539f0 con 0x7f9ea406cde0 2024-09-13T17:12:33.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.494+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ea415f970 con 0x7f9ea406cde0 2024-09-13T17:12:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.495+0000 7f9e9be00640 1 -- 172.21.15.28:0/2391195725 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f9e94017090 con 0x7f9ea406cde0 2024-09-13T17:12:33.496 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:33.496 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:33.496 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 msgr2=0x7f9e8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 0x7f9e8406d0c0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f9e90034000 tx=0x7f9e9002c000 comp rx=0 tx=0).stop 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 msgr2=0x7f9ea415a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f9e9400c4d0 tx=0x7f9e9400c990 comp rx=0 tx=0).stop 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 shutdown_connections 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9e8406ac00 0x7f9e8406d0c0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ea415e860 0x7f9ea40b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ea40713a0 0x7f9ea40b20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 --2- 172.21.15.28:0/2391195725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ea406cde0 0x7f9ea415a150 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 >> 172.21.15.28:0/2391195725 conn(0x7f9ea4069e80 msgr2=0x7f9ea406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 shutdown_connections 2024-09-13T17:12:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.497+0000 7f9eaac00640 1 -- 172.21.15.28:0/2391195725 wait complete. 2024-09-13T17:12:33.521 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:33.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.675+0000 7f057fc00640 1 Processor -- start 2024-09-13T17:12:33.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.675+0000 7f057fc00640 1 -- start start 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.676+0000 7f057fc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05781592a0 0x7f057815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.676+0000 7f057fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f057806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.676+0000 7f057fc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057806cc70 con 0x7f057806f970 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.676+0000 7f057fc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057815b0b0 con 0x7f05781592a0 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.676+0000 7f057fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0578159d90 con 0x7f057806c2d0 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05781592a0 0x7f057815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f0577e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f057806f970 0x7f0578159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:44398/0 (socket says 172.21.15.28:44398) 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05781592a0 0x7f057815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:56330/0 (socket says 172.21.15.28:56330) 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f0577e00640 1 -- 172.21.15.28:0/1556652629 learned_addr learned my addr 172.21.15.28:0/1556652629 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:33.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057dc00640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f057806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 -- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 msgr2=0x7f057806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f057806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 -- 172.21.15.28:0/1556652629 >> v1:172.21.15.189:6789/0 conn(0x7f057806f970 legacy=0x7f0578159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f057d200640 1 -- 172.21.15.28:0/1556652629 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05780b2040 con 0x7f05781592a0 2024-09-13T17:12:33.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.677+0000 7f0577400640 1 -- 172.21.15.28:0/1556652629 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35866288 0 0) 0x7f057806cc70 con 0x7f057806f970 2024-09-13T17:12:33.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.678+0000 7f057d200640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f057815ab70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f056c00ba40 tx=0x7f056c02c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dbf0d9d5d31e831 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f0577400640 1 -- 172.21.15.28:0/1556652629 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c032750 con 0x7f05781592a0 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f0577400640 1 -- 172.21.15.28:0/1556652629 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c0328b0 con 0x7f05781592a0 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f0577400640 1 -- 172.21.15.28:0/1556652629 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c032c40 con 0x7f05781592a0 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 -- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 msgr2=0x7f057815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f057815ab70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f056c00ba40 tx=0x7f056c02c850 comp rx=0 tx=0).stop 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 -- 172.21.15.28:0/1556652629 shutdown_connections 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f057806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 --2- 172.21.15.28:0/1556652629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f057815ab70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 -- 172.21.15.28:0/1556652629 >> 172.21.15.28:0/1556652629 conn(0x7f0578069e80 msgr2=0x7f057806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.679+0000 7f057fc00640 1 -- 172.21.15.28:0/1556652629 shutdown_connections 2024-09-13T17:12:33.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.680+0000 7f057fc00640 1 -- 172.21.15.28:0/1556652629 wait complete. 2024-09-13T17:12:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.681+0000 7f057fc00640 1 Processor -- start 2024-09-13T17:12:33.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.681+0000 7f057fc00640 1 -- start start 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f0578146ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f05781e2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05781e45e0 con 0x7f057806f970 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0578147530 con 0x7f05781592a0 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.682+0000 7f057fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05781e39d0 con 0x7f057806c2d0 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60604/0 (socket says 172.21.15.28:60604) 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 -- 172.21.15.28:0/3644678978 learned_addr learned my addr 172.21.15.28:0/3644678978 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:33.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f057d200640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f0578146ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f057dc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f05781e2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 -- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 msgr2=0x7f0578146ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f0578146ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 -- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 msgr2=0x7f05781e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f05781e2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 -- 172.21.15.28:0/3644678978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c00b6a0 con 0x7f057806f970 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.683+0000 7f0577e00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f056400c4d0 tx=0x7f056400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.684+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564005ce0 con 0x7f057806f970 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.684+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564005e60 con 0x7f057806f970 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.684+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05781e3cb0 con 0x7f057806f970 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.684+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05640072b0 con 0x7f057806f970 2024-09-13T17:12:33.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.684+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05781e4270 con 0x7f057806f970 2024-09-13T17:12:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.685+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05781e4790 con 0x7f057806f970 2024-09-13T17:12:33.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.686+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0564007410 con 0x7f057806f970 2024-09-13T17:12:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.687+0000 7f0574c00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 0x7f055806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.687+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05640888b0 con 0x7f057806f970 2024-09-13T17:12:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.687+0000 7f057d200640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 0x7f055806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:33.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.688+0000 7f057d200640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 0x7f055806d0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f056c037000 tx=0x7f056c036040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:33.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.691+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0564053ad0 con 0x7f057806f970 2024-09-13T17:12:33.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.904+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f057806d0a0 con 0x7f057806f970 2024-09-13T17:12:33.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.905+0000 7f0574c00640 1 -- 172.21.15.28:0/3644678978 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4738 (secure 0 0 0) 0x7f0564017020 con 0x7f057806f970 2024-09-13T17:12:33.906 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:33.906 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":55,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:04.639307+0000","modified":"2024-09-13T17:12:21.595258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6684},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6684":{"gid":6684,"name":"c","rank":0,"incarnation":52,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/2705534945","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2705534945},{"type":"v1","addr":"172.21.15.28:6835","nonce":2705534945}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T17:12:33.906 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 55 2024-09-13T17:12:33.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 msgr2=0x7f055806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 0x7f055806d0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f056c037000 tx=0x7f056c036040 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 msgr2=0x7f05781e4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f056400c4d0 tx=0x7f056400c990 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 shutdown_connections 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f055806ac00 0x7f055806d0c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05781592a0 0x7f05781e2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f057806f970 0x7f05781e4a70 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 --2- 172.21.15.28:0/3644678978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f057806c2d0 0x7f0578146ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.907+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 >> 172.21.15.28:0/3644678978 conn(0x7f0578069e80 msgr2=0x7f05780ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.908+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 shutdown_connections 2024-09-13T17:12:33.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:33.908+0000 7f057fc00640 1 -- 172.21.15.28:0/3644678978 wait complete. 2024-09-13T17:12:33.924 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.079+0000 7fc167e00640 1 Processor -- start 2024-09-13T17:12:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.079+0000 7fc167e00640 1 -- start start 2024-09-13T17:12:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc16806ba70 0x7fc1680afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1680b39d0 con 0x7fc16806c3f0 2024-09-13T17:12:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc16806c880 con 0x7fc168158b20 2024-09-13T17:12:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1680b2dc0 con 0x7fc16806ba70 2024-09-13T17:12:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc15fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc16806ba70 0x7fc1680afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc16806c3f0 0x7fc168158f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52096/0 (socket says 172.21.15.28:52096) 2024-09-13T17:12:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.080+0000 7fc167400640 1 -- 172.21.15.28:0/405561648 learned_addr learned my addr 172.21.15.28:0/405561648 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.081+0000 7fc15fe00640 1 -- 172.21.15.28:0/405561648 >> v1:172.21.15.189:6789/0 conn(0x7fc168158b20 legacy=0x7fc16806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.081+0000 7fc166000640 1 -- 172.21.15.28:0/405561648 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639972698 0 0) 0x7fc1680b39d0 con 0x7fc16806c3f0 2024-09-13T17:12:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.081+0000 7fc15fe00640 1 -- 172.21.15.28:0/405561648 >> v1:172.21.15.28:6789/0 conn(0x7fc16806c3f0 legacy=0x7fc168158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.081+0000 7fc15fe00640 1 -- 172.21.15.28:0/405561648 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1680b30a0 con 0x7fc16806ba70 2024-09-13T17:12:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.082+0000 7fc15fe00640 1 --2- 172.21.15.28:0/405561648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806ba70 0x7fc1680afbd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc15000a970 tx=0x7fc15002b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2444325b64f103 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.082+0000 7fc166000640 1 -- 172.21.15.28:0/405561648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150030710 con 0x7fc16806ba70 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.082+0000 7fc166000640 1 -- 172.21.15.28:0/405561648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc15002e050 con 0x7fc16806ba70 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.082+0000 7fc166000640 1 -- 172.21.15.28:0/405561648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150030a90 con 0x7fc16806ba70 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 -- 172.21.15.28:0/405561648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806ba70 msgr2=0x7fc1680afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 --2- 172.21.15.28:0/405561648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806ba70 0x7fc1680afbd0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fc15000a970 tx=0x7fc15002b580 comp rx=0 tx=0).stop 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 -- 172.21.15.28:0/405561648 shutdown_connections 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 --2- 172.21.15.28:0/405561648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806ba70 0x7fc1680afbd0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 -- 172.21.15.28:0/405561648 >> 172.21.15.28:0/405561648 conn(0x7fc168072e40 msgr2=0x7fc168073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 -- 172.21.15.28:0/405561648 shutdown_connections 2024-09-13T17:12:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.083+0000 7fc167e00640 1 -- 172.21.15.28:0/405561648 wait complete. 2024-09-13T17:12:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.084+0000 7fc167e00640 1 Processor -- start 2024-09-13T17:12:34.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 -- start start 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc16806ba70 0x7fc16814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc168158b20 0x7fc1681e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc168153030 con 0x7fc16806c3f0 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681e3a40 con 0x7fc16806ba70 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.085+0000 7fc167e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1681e3d20 con 0x7fc168158b20 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34438/0 (socket says 172.21.15.28:34438) 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 -- 172.21.15.28:0/372280728 learned_addr learned my addr 172.21.15.28:0/372280728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc166a00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc16806ba70 0x7fc16814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc167400640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc168158b20 0x7fc1681e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 -- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc168158b20 msgr2=0x7fc1681e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc168158b20 0x7fc1681e1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 -- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc16806ba70 msgr2=0x7fc16814d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc16806ba70 0x7fc16814d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.086+0000 7fc15fe00640 1 -- 172.21.15.28:0/372280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15000a5a0 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc15fe00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc150000c00 tx=0x7fc1500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150032a30 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc150032bd0 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc150043400 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1681e4000 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.087+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc16814d5e0 con 0x7fc16806c3f0 2024-09-13T17:12:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.088+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1681e4520 con 0x7fc16806c3f0 2024-09-13T17:12:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.089+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc15003b400 con 0x7fc16806c3f0 2024-09-13T17:12:34.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.090+0000 7fc15ea00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 0x7fc14406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.090+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc1500aa440 con 0x7fc16806c3f0 2024-09-13T17:12:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.090+0000 7fc166a00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 0x7fc14406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.091+0000 7fc166a00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 0x7fc14406d0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc158000f60 tx=0x7fc158009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.094+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc1681e4520 con 0x7fc16806c3f0 2024-09-13T17:12:34.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.273+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1681e4520 con 0x7fc16806c3f0 2024-09-13T17:12:34.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.274+0000 7fc15ea00640 1 -- 172.21.15.28:0/372280728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) v1 ==== 74+0+16694 (secure 0 0 0) 0x7fc1681e4520 con 0x7fc16806c3f0 2024-09-13T17:12:34.275 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:34.276 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":65,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:17.578249+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:02.552870+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:03.628750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/693307152":"2024-09-14T17:12:17.578224+0000","172.21.15.189:6835/693307152":"2024-09-14T17:12:17.578224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 msgr2=0x7fc14406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 0x7fc14406d0c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc158000f60 tx=0x7fc158009040 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 msgr2=0x7fc168149be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fc150000c00 tx=0x7fc1500046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 shutdown_connections 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc14406ac00 0x7fc14406d0c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc168158b20 0x7fc1681e1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc16806c3f0 0x7fc168149be0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 --2- 172.21.15.28:0/372280728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc16806ba70 0x7fc16814d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 >> 172.21.15.28:0/372280728 conn(0x7fc168072e40 msgr2=0x7fc168073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 shutdown_connections 2024-09-13T17:12:34.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.276+0000 7fc167e00640 1 -- 172.21.15.28:0/372280728 wait complete. 2024-09-13T17:12:34.293 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:12:34.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.449+0000 7f6dd6c00640 1 Processor -- start 2024-09-13T17:12:34.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.449+0000 7f6dd6c00640 1 -- start start 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6dd006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd6c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6dd006f3f0 0x7f6dd0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd6c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd006f920 con 0x7f6dd00add00 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd6c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd015b290 con 0x7f6dd006f3f0 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd015b570 con 0x7f6dd0159770 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd4c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6dd00add00 0x7f6dd0159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52102/0 (socket says 172.21.15.28:52102) 2024-09-13T17:12:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dd4c00640 1 -- 172.21.15.28:0/2488893630 learned_addr learned my addr 172.21.15.28:0/2488893630 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dcfe00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6dd006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2488893630 >> v2:172.21.15.189:3300/0 conn(0x7f6dd006f3f0 0x7f6dd0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837710124 0 0) 0x7f6dd006f920 con 0x7f6dd00add00 2024-09-13T17:12:34.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.451+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db4005730 con 0x7f6dd00add00 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168552894 0 0) 0x7f6db4005730 con 0x7f6dd00add00 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 msgr2=0x7f6dd006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6dd006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 >> v2:172.21.15.189:3300/0 conn(0x7f6dd006f3f0 msgr2=0x7f6dd0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 --2- 172.21.15.28:0/2488893630 >> v2:172.21.15.189:3300/0 conn(0x7f6dd006f3f0 0x7f6dd0070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd015b850 con 0x7f6dd00add00 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6dc80019f0 con 0x7f6dd00add00 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 >> v1:172.21.15.28:6789/0 conn(0x7f6dd00add00 legacy=0x7f6dd0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6db4005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db4006100 0x7f6db40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6db4006f10 0x7f6db4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd015b290 con 0x7f6db4006f10 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd015b570 con 0x7f6db4006100 2024-09-13T17:12:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db400dbe0 con 0x7f6dd0159770 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2488893630 >> v2:172.21.15.189:3300/0 conn(0x7f6dd006f3f0 0x7f6dd0070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dd4c00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6db4006f10 0x7f6db4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dcfe00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6db4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.452+0000 7f6dc7400640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db4006100 0x7f6db40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.453+0000 7f6dd4c00640 1 -- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 msgr2=0x7f6db4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.453+0000 7f6dd4c00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd0159770 0x7f6db4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.453+0000 7f6dd4c00640 1 -- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db4006100 msgr2=0x7f6db40069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.453+0000 7f6dd4c00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6db4006100 0x7f6db40069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.453+0000 7f6dd4c00640 1 -- 172.21.15.28:0/2488893630 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db4005730 con 0x7f6db4006f10 2024-09-13T17:12:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd4c00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6db4006f10 0x7f6db4007390 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800a8d0 tx=0x7f6db4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c62c0a7bd220755 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc802ba10 con 0x7f6db4006f10 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc802bb70 con 0x7f6db4006f10 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dcf400640 1 -- 172.21.15.28:0/2488893630 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8034470 con 0x7f6db4006f10 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd6c00640 1 -- 172.21.15.28:0/2488893630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6db4006f10 msgr2=0x7f6db4007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/2488893630 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6db4006f10 0x7f6db4007390 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800a8d0 tx=0x7f6db4007a00 comp rx=0 tx=0).stop 2024-09-13T17:12:34.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd4c00640 1 -- 172.21.15.28:0/2488893630 reap_dead start 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd6c00640 1 -- 172.21.15.28:0/2488893630 shutdown_connections 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.454+0000 7f6dd6c00640 1 -- 172.21.15.28:0/2488893630 >> 172.21.15.28:0/2488893630 conn(0x7f6dd00b0270 msgr2=0x7f6dd00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.455+0000 7f6dd6c00640 1 -- 172.21.15.28:0/2488893630 shutdown_connections 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.455+0000 7f6dd6c00640 1 -- 172.21.15.28:0/2488893630 wait complete. 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.456+0000 7f6dd6c00640 1 Processor -- start 2024-09-13T17:12:34.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.456+0000 7f6dd6c00640 1 -- start start 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd01559c0 0x7f6dd01e1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd01562b0 0x7f6dd006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd01e28d0 con 0x7f6dd0155da0 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd01e2bb0 con 0x7f6dd01559c0 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dd6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd01e2e90 con 0x7f6dd01562b0 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.457+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34450/0 (socket says 172.21.15.28:34450) 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 -- 172.21.15.28:0/4162896743 learned_addr learned my addr 172.21.15.28:0/4162896743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:34.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dcfe00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd01559c0 0x7f6dd01e1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dd4c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd01562b0 0x7f6dd006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 -- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd01562b0 msgr2=0x7f6dd006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd01562b0 0x7f6dd006bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 -- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd01559c0 msgr2=0x7f6dd01e1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd01559c0 0x7f6dd01e1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 -- 172.21.15.28:0/4162896743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd01e3170 con 0x7f6dd0155da0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.458+0000 7f6dc7400640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6db8011900 tx=0x7f6db8011d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.459+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db80096e0 con 0x7f6dd0155da0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.459+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800f9c0 con 0x7f6dd0155da0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.459+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd01e3450 con 0x7f6dd0155da0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.459+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db800fd70 con 0x7f6dd0155da0 2024-09-13T17:12:34.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.459+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd01e2680 con 0x7f6dd0155da0 2024-09-13T17:12:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.460+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd006fd20 con 0x7f6dd0155da0 2024-09-13T17:12:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.461+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6db8015460 con 0x7f6dd0155da0 2024-09-13T17:12:34.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.461+0000 7f6dccc00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 0x7f6dac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.462+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6db8089cc0 con 0x7f6dd0155da0 2024-09-13T17:12:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.462+0000 7f6dcfe00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 0x7f6dac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.463+0000 7f6dcfe00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 0x7f6dac06d0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6dd006c5c0 tx=0x7f6dc0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.465+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6db8090080 con 0x7f6dd0155da0 2024-09-13T17:12:34.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.681+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f6dd00afcc0 con 0x7f6dd0155da0 2024-09-13T17:12:34.682 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:12:34.682+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:12:34.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.693+0000 7f6dccc00640 1 -- 172.21.15.28:0/4162896743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v56) v1 ==== 161+0+0 (secure 0 0 0) 0x7f6db8054e30 con 0x7f6dd0155da0 2024-09-13T17:12:34.694 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:12:34.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 msgr2=0x7f6dac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 0x7f6dac06d0c0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6dd006c5c0 tx=0x7f6dc0009190 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 msgr2=0x7f6dd006ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f6db8011900 tx=0x7f6db8011d00 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 shutdown_connections 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6dac06ac00 0x7f6dac06d0c0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6dd01562b0 0x7f6dd006bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6dd0155da0 0x7f6dd006ba00 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 --2- 172.21.15.28:0/4162896743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6dd01559c0 0x7f6dd01e1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 >> 172.21.15.28:0/4162896743 conn(0x7f6dd00b0270 msgr2=0x7f6dd00b22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 shutdown_connections 2024-09-13T17:12:34.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.695+0000 7f6dd6c00640 1 -- 172.21.15.28:0/4162896743 wait complete. 2024-09-13T17:12:34.704 INFO:tasks.ceph.mds.c.smithi028.stderr: -600> 2024-09-13T17:12:24.594+0000 7f8fd9400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:12:34.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:12:34.748 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:12:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.874+0000 7f6cbe600640 1 Processor -- start 2024-09-13T17:12:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.874+0000 7f6cbe600640 1 -- start start 2024-09-13T17:12:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.875+0000 7f6cbe600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 0x7f6cb815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.875+0000 7f6cbe600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 0x7f6cb815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.875+0000 7f6cbe600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cb815ab60 con 0x7f6cb8159770 2024-09-13T17:12:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.875+0000 7f6cbe600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cb806f920 con 0x7f6cb806f3f0 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.875+0000 7f6cbe600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb815b290 con 0x7f6cb8070430 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb7400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f6cb8159770 0x7f6cb806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:44946/0 (socket says 172.21.15.28:44946) 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6caea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 0x7f6cb815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb7400640 1 -- 172.21.15.28:0/343541472 learned_addr learned my addr 172.21.15.28:0/343541472 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb7e00640 1 --2- 172.21.15.28:0/343541472 >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 0x7f6cb815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb6a00640 1 -- 172.21.15.28:0/343541472 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347585804 0 0) 0x7f6ca8001110 con 0x7f6cb8159770 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb6a00640 1 -- 172.21.15.28:0/343541472 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c9c003520 con 0x7f6cb8159770 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6caea00640 1 -- 172.21.15.28:0/343541472 >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 msgr2=0x7f6cb815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6caea00640 1 --2- 172.21.15.28:0/343541472 >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 0x7f6cb815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6caea00640 1 -- 172.21.15.28:0/343541472 >> v1:172.21.15.189:6789/0 conn(0x7f6cb8159770 legacy=0x7f6cb806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6caea00640 1 -- 172.21.15.28:0/343541472 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb815b570 con 0x7f6cb8070430 2024-09-13T17:12:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.876+0000 7f6cb7e00640 1 --2- 172.21.15.28:0/343541472 >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 0x7f6cb815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6caea00640 1 --2- 172.21.15.28:0/343541472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 0x7f6cb815bd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00097a0 tx=0x7f6ca002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f06cab579b1b168b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cb6a00640 1 -- 172.21.15.28:0/343541472 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca002ae50 con 0x7f6cb8070430 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cb6a00640 1 -- 172.21.15.28:0/343541472 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca0032440 con 0x7f6cb8070430 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cb6a00640 1 -- 172.21.15.28:0/343541472 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca00327d0 con 0x7f6cb8070430 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 -- 172.21.15.28:0/343541472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 msgr2=0x7f6cb815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 --2- 172.21.15.28:0/343541472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 0x7f6cb815bd70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ca00097a0 tx=0x7f6ca002a100 comp rx=0 tx=0).stop 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 -- 172.21.15.28:0/343541472 shutdown_connections 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 --2- 172.21.15.28:0/343541472 >> v2:172.21.15.189:3300/0 conn(0x7f6cb806f3f0 0x7f6cb815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 --2- 172.21.15.28:0/343541472 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8070430 0x7f6cb815bd70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.878+0000 7f6cbe600640 1 -- 172.21.15.28:0/343541472 >> 172.21.15.28:0/343541472 conn(0x7f6cb80b0270 msgr2=0x7f6cb80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.879+0000 7f6cbe600640 1 -- 172.21.15.28:0/343541472 shutdown_connections 2024-09-13T17:12:34.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.879+0000 7f6cbe600640 1 -- 172.21.15.28:0/343541472 wait complete. 2024-09-13T17:12:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.880+0000 7f6cbe600640 1 Processor -- start 2024-09-13T17:12:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.880+0000 7f6cbe600640 1 -- start start 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 0x7f6cb80cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8159770 0x7f6cb80af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80cc1a0 con 0x7f6cb806f3f0 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80cc480 con 0x7f6cb8070430 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cbe600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb80cc760 con 0x7f6cb8159770 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34480/0 (socket says 172.21.15.28:34480) 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.881+0000 7f6cb7400640 1 -- 172.21.15.28:0/1846598377 learned_addr learned my addr 172.21.15.28:0/1846598377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7e00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8159770 0x7f6cb80af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6caea00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 0x7f6cb80cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 -- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8159770 msgr2=0x7f6cb80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8159770 0x7f6cb80af3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 -- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 msgr2=0x7f6cb80cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 0x7f6cb80cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 -- 172.21.15.28:0/1846598377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cb0009a70 con 0x7f6cb806f3f0 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6caea00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 0x7f6cb80cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cb7400640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c420 tx=0x7f6ca800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca8005ce0 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ca8005e40 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ca80072f0 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.882+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca0009400 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.883+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb80ccd20 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.883+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb806fd20 con 0x7f6cb806f3f0 2024-09-13T17:12:34.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.884+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6ca8007510 con 0x7f6cb806f3f0 2024-09-13T17:12:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.885+0000 7f6cafe00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 0x7f6c9406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.885+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6ca8088960 con 0x7f6cb806f3f0 2024-09-13T17:12:34.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.886+0000 7f6caea00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 0x7f6c9406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:34.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.887+0000 7f6caea00640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 0x7f6c9406d0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0034000 tx=0x7f6ca002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:34.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:34.889+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6ca80539f0 con 0x7f6cb806f3f0 2024-09-13T17:12:35.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.103+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6cb814b050 con 0x7f6cb806f3f0 2024-09-13T17:12:35.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.698+0000 7f6cafe00640 1 -- 172.21.15.28:0/1846598377 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6ca80170a0 con 0x7f6cb806f3f0 2024-09-13T17:12:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 msgr2=0x7f6c9406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 0x7f6c9406d0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0034000 tx=0x7f6ca002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 msgr2=0x7f6cb8148180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f6ca800c420 tx=0x7f6ca800c8e0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 shutdown_connections 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c9406ac00 0x7f6c9406d0c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6cb8159770 0x7f6cb80af3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6cb8070430 0x7f6cb80cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 --2- 172.21.15.28:0/1846598377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6cb806f3f0 0x7f6cb8148180 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 >> 172.21.15.28:0/1846598377 conn(0x7f6cb80b0270 msgr2=0x7f6cb814e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 shutdown_connections 2024-09-13T17:12:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.701+0000 7f6cbe600640 1 -- 172.21.15.28:0/1846598377 wait complete. 2024-09-13T17:12:35.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:12:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.925+0000 7f5d13e00640 1 Processor -- start 2024-09-13T17:12:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.925+0000 7f5d13e00640 1 -- start start 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.926+0000 7f5d13e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 0x7f5d0c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.926+0000 7f5d13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d0c15cdd0 0x7f5d0c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d13e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d0c15d950 con 0x7f5d0c06b6b0 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d13e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c15d740 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d13e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0c15dc30 con 0x7f5d0c157000 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5d0c15cdd0 0x7f5d0c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d10a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d0c06b6b0 0x7f5d0c0b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52116/0 (socket says 172.21.15.28:52116) 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d10a00640 1 -- 172.21.15.28:0/1862171908 learned_addr learned my addr 172.21.15.28:0/1862171908 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:35.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11400640 1 --2- 172.21.15.28:0/1862171908 >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 0x7f5d0c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d0be00640 1 -- 172.21.15.28:0/1862171908 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174295223 0 0) 0x7f5d0c15d950 con 0x7f5d0c06b6b0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d0be00640 1 -- 172.21.15.28:0/1862171908 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4003520 con 0x7f5d0c06b6b0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11e00640 1 -- 172.21.15.28:0/1862171908 >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 msgr2=0x7f5d0c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11e00640 1 --2- 172.21.15.28:0/1862171908 >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 0x7f5d0c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11e00640 1 -- 172.21.15.28:0/1862171908 >> v1:172.21.15.28:6789/0 conn(0x7f5d0c06b6b0 legacy=0x7f5d0c0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11e00640 1 -- 172.21.15.28:0/1862171908 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0c0b1b80 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.927+0000 7f5d11400640 1 --2- 172.21.15.28:0/1862171908 >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 0x7f5d0c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d11e00640 1 --2- 172.21.15.28:0/1862171908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c15cdd0 0x7f5d0c15d200 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c06d570 tx=0x7f5d0402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c149506667e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d0be00640 1 -- 172.21.15.28:0/1862171908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0402aea0 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d0be00640 1 -- 172.21.15.28:0/1862171908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0402d050 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d0be00640 1 -- 172.21.15.28:0/1862171908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d04007960 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 -- 172.21.15.28:0/1862171908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c15cdd0 msgr2=0x7f5d0c15d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 --2- 172.21.15.28:0/1862171908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c15cdd0 0x7f5d0c15d200 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c06d570 tx=0x7f5d0402a680 comp rx=0 tx=0).stop 2024-09-13T17:12:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 -- 172.21.15.28:0/1862171908 shutdown_connections 2024-09-13T17:12:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 --2- 172.21.15.28:0/1862171908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c15cdd0 0x7f5d0c15d200 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 --2- 172.21.15.28:0/1862171908 >> v2:172.21.15.189:3300/0 conn(0x7f5d0c157000 0x7f5d0c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.929+0000 7f5d13e00640 1 -- 172.21.15.28:0/1862171908 >> 172.21.15.28:0/1862171908 conn(0x7f5d0c069e80 msgr2=0x7f5d0c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.930+0000 7f5d13e00640 1 -- 172.21.15.28:0/1862171908 shutdown_connections 2024-09-13T17:12:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.930+0000 7f5d13e00640 1 -- 172.21.15.28:0/1862171908 wait complete. 2024-09-13T17:12:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.931+0000 7f5d13e00640 1 Processor -- start 2024-09-13T17:12:35.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.931+0000 7f5d13e00640 1 -- start start 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d0c06b6b0 0x7f5d0c155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 0x7f5d0c15bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c1587f0 con 0x7f5d0c157000 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c158a00 con 0x7f5d0c06b6b0 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d13e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0c158ce0 con 0x7f5d0c15cdd0 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34494/0 (socket says 172.21.15.28:34494) 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.932+0000 7f5d10a00640 1 -- 172.21.15.28:0/4184168446 learned_addr learned my addr 172.21.15.28:0/4184168446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:35.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d11400640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d0c06b6b0 0x7f5d0c155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d11e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 0x7f5d0c15bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 -- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 msgr2=0x7f5d0c15bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 0x7f5d0c15bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 -- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d0c06b6b0 msgr2=0x7f5d0c155390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d0c06b6b0 0x7f5d0c155390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 -- 172.21.15.28:0/4184168446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d00009400 con 0x7f5d0c157000 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d11e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 0x7f5d0c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:35.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d10a00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c500 tx=0x7f5cf800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8005ce0 con 0x7f5d0c157000 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf8005e40 con 0x7f5d0c157000 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.933+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf8007330 con 0x7f5d0c157000 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.934+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d04009a70 con 0x7f5d0c157000 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.934+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0c1592a0 con 0x7f5d0c157000 2024-09-13T17:12:35.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.935+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0c1f2500 con 0x7f5d0c157000 2024-09-13T17:12:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.935+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5cf8007550 con 0x7f5d0c157000 2024-09-13T17:12:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.936+0000 7f5d09600640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 0x7f5cec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:35.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.936+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f5cf8088a50 con 0x7f5d0c157000 2024-09-13T17:12:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.937+0000 7f5d11400640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 0x7f5cec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.937+0000 7f5d11400640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 0x7f5cec06d0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009cd0 tx=0x7f5d00009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:35.941+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5d0c1f2500 con 0x7f5d0c157000 2024-09-13T17:12:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:36.122+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5d0c1f2500 con 0x7f5d0c157000 2024-09-13T17:12:37.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.035+0000 7f5d09600640 1 -- 172.21.15.28:0/4184168446 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v67) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5d0c1f2500 con 0x7f5d0c157000 2024-09-13T17:12:37.037 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.038+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 msgr2=0x7f5cec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.038+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 0x7f5cec06d0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5d00009cd0 tx=0x7f5d00009190 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 msgr2=0x7f5d0c1582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f5cf800c500 tx=0x7f5cf800c9c0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 shutdown_connections 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cec06ac00 0x7f5cec06d0c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d0c15cdd0 0x7f5d0c15bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d0c157000 0x7f5d0c1582b0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 --2- 172.21.15.28:0/4184168446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d0c06b6b0 0x7f5d0c155390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 >> 172.21.15.28:0/4184168446 conn(0x7f5d0c069e80 msgr2=0x7f5d0c15b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:37.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 shutdown_connections 2024-09-13T17:12:37.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.039+0000 7f5d13e00640 1 -- 172.21.15.28:0/4184168446 wait complete. 2024-09-13T17:12:37.059 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:12:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.220+0000 7f3312c00640 1 Processor -- start 2024-09-13T17:12:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.220+0000 7f3312c00640 1 -- start start 2024-09-13T17:12:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3312c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f330c159440 0x7f330c06b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3312c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f330c0b2fa0 0x7f330c159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3312c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c06bc40 con 0x7f330c06b7f0 2024-09-13T17:12:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3312c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c0b3410 con 0x7f330c159440 2024-09-13T17:12:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3312c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c0b46a0 con 0x7f330c0b2fa0 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3311800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f330c159440 0x7f330c06b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3311800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f330c159440 0x7f330c06b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34500/0 (socket says 172.21.15.28:34500) 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.221+0000 7f3311800640 1 -- 172.21.15.28:0/1274221319 learned_addr learned my addr 172.21.15.28:0/1274221319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.222+0000 7f3312200640 1 --2- 172.21.15.28:0/1274221319 >> v2:172.21.15.189:3300/0 conn(0x7f330c0b2fa0 0x7f330c159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.222+0000 7f3311800640 1 -- 172.21.15.28:0/1274221319 >> v2:172.21.15.189:3300/0 conn(0x7f330c0b2fa0 msgr2=0x7f330c159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.222+0000 7f3311800640 1 --2- 172.21.15.28:0/1274221319 >> v2:172.21.15.189:3300/0 conn(0x7f330c0b2fa0 0x7f330c159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.222+0000 7f3311800640 1 -- 172.21.15.28:0/1274221319 >> v1:172.21.15.189:6789/0 conn(0x7f330c06b7f0 legacy=0x7f330c0b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.222+0000 7f3311800640 1 -- 172.21.15.28:0/1274221319 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0b3600 con 0x7f330c159440 2024-09-13T17:12:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.223+0000 7f3311800640 1 --2- 172.21.15.28:0/1274221319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c06b2b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f330c06f030 tx=0x7f330002c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=176d6730c16e40d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.223+0000 7f330be00640 1 -- 172.21.15.28:0/1274221319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300032820 con 0x7f330c159440 2024-09-13T17:12:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.223+0000 7f330be00640 1 -- 172.21.15.28:0/1274221319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330002e040 con 0x7f330c159440 2024-09-13T17:12:37.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.223+0000 7f330be00640 1 -- 172.21.15.28:0/1274221319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300032ba0 con 0x7f330c159440 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 -- 172.21.15.28:0/1274221319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 msgr2=0x7f330c06b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 --2- 172.21.15.28:0/1274221319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c06b2b0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f330c06f030 tx=0x7f330002c960 comp rx=0 tx=0).stop 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 -- 172.21.15.28:0/1274221319 shutdown_connections 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 --2- 172.21.15.28:0/1274221319 >> v2:172.21.15.189:3300/0 conn(0x7f330c0b2fa0 0x7f330c159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 --2- 172.21.15.28:0/1274221319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c06b2b0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 -- 172.21.15.28:0/1274221319 >> 172.21.15.28:0/1274221319 conn(0x7f330c069e80 msgr2=0x7f330c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 -- 172.21.15.28:0/1274221319 shutdown_connections 2024-09-13T17:12:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.224+0000 7f3312c00640 1 -- 172.21.15.28:0/1274221319 wait complete. 2024-09-13T17:12:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.225+0000 7f3312c00640 1 Processor -- start 2024-09-13T17:12:37.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.226+0000 7f3312c00640 1 -- start start 2024-09-13T17:12:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.226+0000 7f3312c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f330c06b7f0 0x7f330c14d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330c0b2fa0 0x7f330c0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0ade10 con 0x7f330c159440 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c14dda0 con 0x7f330c06b7f0 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0ad190 con 0x7f330c0b2fa0 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34504/0 (socket says 172.21.15.28:34504) 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 -- 172.21.15.28:0/3677137762 learned_addr learned my addr 172.21.15.28:0/3677137762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3311800640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f330c06b7f0 0x7f330c14d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3310e00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330c0b2fa0 0x7f330c0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 -- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330c0b2fa0 msgr2=0x7f330c0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330c0b2fa0 0x7f330c0ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 -- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f330c06b7f0 msgr2=0x7f330c14d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f330c06b7f0 0x7f330c14d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.227+0000 7f3312200640 1 -- 172.21.15.28:0/3677137762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330000b770 con 0x7f330c159440 2024-09-13T17:12:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3312200640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3304007e40 tx=0x7f330400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304014620 con 0x7f330c159440 2024-09-13T17:12:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330400ee90 con 0x7f330c159440 2024-09-13T17:12:37.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33040149e0 con 0x7f330c159440 2024-09-13T17:12:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c0ad470 con 0x7f330c159440 2024-09-13T17:12:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.228+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c0ada30 con 0x7f330c159440 2024-09-13T17:12:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.229+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c067a10 con 0x7f330c159440 2024-09-13T17:12:37.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.230+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3304014b40 con 0x7f330c159440 2024-09-13T17:12:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.231+0000 7f3309600640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 0x7f32ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.231+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f3304089b20 con 0x7f330c159440 2024-09-13T17:12:37.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.231+0000 7f3311800640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 0x7f32ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:37.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.232+0000 7f3311800640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 0x7f32ec06d0f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3300037000 tx=0x7f3300032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:37.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.235+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3304054e10 con 0x7f330c159440 2024-09-13T17:12:37.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:37.414+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f330c0adf50 con 0x7f330c159440 2024-09-13T17:12:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.038+0000 7f3309600640 1 -- 172.21.15.28:0/3677137762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v68) v1 ==== 169+0+0 (secure 0 0 0) 0x7f330c0adf50 con 0x7f330c159440 2024-09-13T17:12:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:12:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.042+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 msgr2=0x7f32ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.042+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 0x7f32ec06d0f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3300037000 tx=0x7f3300032000 comp rx=0 tx=0).stop 2024-09-13T17:12:38.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.042+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 msgr2=0x7f330c0aeea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.042+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f3304007e40 tx=0x7f330400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 shutdown_connections 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32ec06ac30 0x7f32ec06d0f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330c159440 0x7f330c0aeea0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330c0b2fa0 0x7f330c0ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 --2- 172.21.15.28:0/3677137762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f330c06b7f0 0x7f330c14d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 >> 172.21.15.28:0/3677137762 conn(0x7f330c069e80 msgr2=0x7f330c0b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 shutdown_connections 2024-09-13T17:12:38.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.043+0000 7f3312c00640 1 -- 172.21.15.28:0/3677137762 wait complete. 2024-09-13T17:12:38.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_bits 2024-09-13T17:12:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.226+0000 7f499ba00640 1 Processor -- start 2024-09-13T17:12:38.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.226+0000 7f499ba00640 1 -- start start 2024-09-13T17:12:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f499ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 0x7f4994070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f499ba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f499406c250 0x7f49940712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f499ba00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499406c8b0 con 0x7f499415ab60 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f499ba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499406c6a0 con 0x7f499406c250 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f499ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499406cb90 con 0x7f499415a780 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f4999000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 0x7f4994070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f4999a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f499406c250 0x7f49940712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f4999000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 0x7f4994070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:45190/0 (socket says 172.21.15.28:45190) 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.228+0000 7f4999000640 1 -- 172.21.15.28:0/2073671199 learned_addr learned my addr 172.21.15.28:0/2073671199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.229+0000 7f4999a00640 1 -- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 msgr2=0x7f4994070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.229+0000 7f4999a00640 1 --2- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 0x7f4994070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.229+0000 7f4999a00640 1 -- 172.21.15.28:0/2073671199 >> v1:172.21.15.189:6789/0 conn(0x7f499415ab60 legacy=0x7f499406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.229+0000 7f4999a00640 1 -- 172.21.15.28:0/2073671199 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49940cc980 con 0x7f499406c250 2024-09-13T17:12:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f4999a00640 1 --2- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499406c250 0x7f49940712c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f498c009d90 tx=0x7f498c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e63f38a0e84b6c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f4993400640 1 -- 172.21.15.28:0/2073671199 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c007740 con 0x7f499406c250 2024-09-13T17:12:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f4993400640 1 -- 172.21.15.28:0/2073671199 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c02d050 con 0x7f499406c250 2024-09-13T17:12:38.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f4993400640 1 -- 172.21.15.28:0/2073671199 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c007ac0 con 0x7f499406c250 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f499ba00640 1 -- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499406c250 msgr2=0x7f49940712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.230+0000 7f499ba00640 1 --2- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499406c250 0x7f49940712c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f498c009d90 tx=0x7f498c02a660 comp rx=0 tx=0).stop 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 -- 172.21.15.28:0/2073671199 shutdown_connections 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 --2- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499406c250 0x7f49940712c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 --2- 172.21.15.28:0/2073671199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499415a780 0x7f4994070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 -- 172.21.15.28:0/2073671199 >> 172.21.15.28:0/2073671199 conn(0x7f49940b0270 msgr2=0x7f49940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 -- 172.21.15.28:0/2073671199 shutdown_connections 2024-09-13T17:12:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.231+0000 7f499ba00640 1 -- 172.21.15.28:0/2073671199 wait complete. 2024-09-13T17:12:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.232+0000 7f499ba00640 1 Processor -- start 2024-09-13T17:12:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.232+0000 7f499ba00640 1 -- start start 2024-09-13T17:12:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499406c250 0x7f4994156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499415ab60 0x7f49940b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b23b0 con 0x7f499415a780 2024-09-13T17:12:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b2690 con 0x7f499415ab60 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.233+0000 7f499ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49940b2970 con 0x7f499406c250 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34516/0 (socket says 172.21.15.28:34516) 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 -- 172.21.15.28:0/3293529708 learned_addr learned my addr 172.21.15.28:0/3293529708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4999000640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499406c250 0x7f4994156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4999a00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499415ab60 0x7f49940b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 -- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499406c250 msgr2=0x7f4994156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499406c250 0x7f4994156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 -- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499415ab60 msgr2=0x7f49940b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499415ab60 0x7f49940b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 -- 172.21.15.28:0/3293529708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c009a70 con 0x7f499415a780 2024-09-13T17:12:38.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.234+0000 7f4993e00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f498000c3c0 tx=0x7f498000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.235+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49800052d0 con 0x7f499415a780 2024-09-13T17:12:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.235+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4980005450 con 0x7f499415a780 2024-09-13T17:12:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.235+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980003830 con 0x7f499415a780 2024-09-13T17:12:38.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.235+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49940b2c50 con 0x7f499415a780 2024-09-13T17:12:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.235+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49940b2030 con 0x7f499415a780 2024-09-13T17:12:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.236+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49940b3170 con 0x7f499415a780 2024-09-13T17:12:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.237+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4980003990 con 0x7f499415a780 2024-09-13T17:12:38.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.238+0000 7f4990c00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 0x7f497406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.238+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f4980088490 con 0x7f499415a780 2024-09-13T17:12:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.238+0000 7f4999000640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 0x7f497406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.239+0000 7f4999000640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 0x7f497406d0f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4988009bd0 tx=0x7f4988009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.242+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f49940b3170 con 0x7f499415a780 2024-09-13T17:12:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.427+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"} v 0) v1 -- 0x7f49940b3170 con 0x7f499415a780 2024-09-13T17:12:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.431+0000 7f4990c00640 1 -- 172.21.15.28:0/3293529708 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"}]=0 v20) v1 ==== 101+0+0 (secure 0 0 0) 0x7f49940b3170 con 0x7f499415a780 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.433+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 msgr2=0x7f497406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.433+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 0x7f497406d0f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4988009bd0 tx=0x7f4988009240 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.433+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 msgr2=0x7f4994149090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.433+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f498000c3c0 tx=0x7f498000c880 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 shutdown_connections 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f497406ac30 0x7f497406d0f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f499415ab60 0x7f49940b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f499415a780 0x7f4994149090 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 --2- 172.21.15.28:0/3293529708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f499406c250 0x7f4994156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 >> 172.21.15.28:0/3293529708 conn(0x7f49940b0270 msgr2=0x7f49940cdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 shutdown_connections 2024-09-13T17:12:38.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.434+0000 7f499ba00640 1 -- 172.21.15.28:0/3293529708 wait complete. 2024-09-13T17:12:38.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_size 2024-09-13T17:12:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.616+0000 7f6426600640 1 Processor -- start 2024-09-13T17:12:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.616+0000 7f6426600640 1 -- start start 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f6426600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6420070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f6426600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642006c8b0 con 0x7f642006c250 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f6426600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642006cb90 con 0x7f642015ab60 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f6426600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642006c6a0 con 0x7f642015a780 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f641fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f642006c250 0x7f64200cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52122/0 (socket says 172.21.15.28:52122) 2024-09-13T17:12:38.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.617+0000 7f641fe00640 1 -- 172.21.15.28:0/1024646314 learned_addr learned my addr 172.21.15.28:0/1024646314 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641f400640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6420070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289171148 0 0) 0x7f642006c8b0 con 0x7f642006c250 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6404003520 con 0x7f642006c250 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378403859 0 0) 0x7f6404003520 con 0x7f642006c250 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 msgr2=0x7f6420070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:38.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6420070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 >> v1:172.21.15.189:6789/0 conn(0x7f642015ab60 legacy=0x7f642006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64200cc980 con 0x7f642006c250 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f64180019f0 con 0x7f642006c250 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 >> v1:172.21.15.28:6789/0 conn(0x7f642006c250 legacy=0x7f64200cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040039a0 0x7f6404003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040047c0 0x7f6404004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6404005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642006c6a0 con 0x7f64040047c0 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640400ba30 con 0x7f64040039a0 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.618+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640400bbf0 con 0x7f642015a780 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040047c0 0x7f6404004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641f400640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040039a0 0x7f6404003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641fe00640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6404005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 -- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 msgr2=0x7f6404005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:38.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f642015a780 0x7f6404005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 -- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040039a0 msgr2=0x7f6404003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:38.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040039a0 0x7f6404003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.619+0000 7f641ea00640 1 -- 172.21.15.28:0/1024646314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6404003520 con 0x7f64040047c0 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.620+0000 7f641ea00640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040047c0 0x7f6404004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f642015af40 tx=0x7f64040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef2e4c19d45714fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.620+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0052d0 con 0x7f64040047c0 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.620+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c005430 con 0x7f64040047c0 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.620+0000 7f641e000640 1 -- 172.21.15.28:0/1024646314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c02ab70 con 0x7f64040047c0 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 -- 172.21.15.28:0/1024646314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040047c0 msgr2=0x7f6404004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 --2- 172.21.15.28:0/1024646314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64040047c0 0x7f6404004c40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f642015af40 tx=0x7f64040057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f641fe00640 1 -- 172.21.15.28:0/1024646314 reap_dead start 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 -- 172.21.15.28:0/1024646314 shutdown_connections 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 -- 172.21.15.28:0/1024646314 >> 172.21.15.28:0/1024646314 conn(0x7f64200b0270 msgr2=0x7f64200b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 -- 172.21.15.28:0/1024646314 shutdown_connections 2024-09-13T17:12:38.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.621+0000 7f6426600640 1 -- 172.21.15.28:0/1024646314 wait complete. 2024-09-13T17:12:38.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.623+0000 7f6426600640 1 Processor -- start 2024-09-13T17:12:38.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.623+0000 7f6426600640 1 -- start start 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040047c0 0x7f642014f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6420153f10 0x7f64201514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64201531e0 con 0x7f64201f33b0 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642014fa80 con 0x7f64040047c0 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f6426600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6420151df0 con 0x7f6420153f10 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34536/0 (socket says 172.21.15.28:34536) 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 -- 172.21.15.28:0/4207150719 learned_addr learned my addr 172.21.15.28:0/4207150719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641f400640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040047c0 0x7f642014f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641ea00640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6420153f10 0x7f64201514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 -- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6420153f10 msgr2=0x7f64201514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6420153f10 0x7f64201514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 -- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040047c0 msgr2=0x7f642014f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040047c0 0x7f642014f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.624+0000 7f641fe00640 1 -- 172.21.15.28:0/4207150719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6420153430 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f641fe00640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f6404003970 tx=0x7f641800a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641800ac60 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641800ae00 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418013630 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6420153710 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.625+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64201519e0 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.626+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6420151b70 con 0x7f64201f33b0 2024-09-13T17:12:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.627+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6418013790 con 0x7f64201f33b0 2024-09-13T17:12:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.628+0000 7f6417400640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 0x7f640006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.628+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6418089750 con 0x7f64201f33b0 2024-09-13T17:12:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.628+0000 7f641f400640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 0x7f640006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.629+0000 7f641f400640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 0x7f640006d0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6420153190 tx=0x7f6410009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.632+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6418054b30 con 0x7f64201f33b0 2024-09-13T17:12:38.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.810+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"} v 0) v1 -- 0x7f642006c8b0 con 0x7f64201f33b0 2024-09-13T17:12:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.814+0000 7f6417400640 1 -- 172.21.15.28:0/4207150719 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"}]=0 v21) v1 ==== 101+0+0 (secure 0 0 0) 0x7f6418058150 con 0x7f64201f33b0 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 msgr2=0x7f640006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 0x7f640006d0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6420153190 tx=0x7f6410009190 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 msgr2=0x7f64200ae240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f6404003970 tx=0x7f641800a6e0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 shutdown_connections 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f640006ac00 0x7f640006d0c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64201f33b0 0x7f64200ae240 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6420153f10 0x7f64201514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 --2- 172.21.15.28:0/4207150719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64040047c0 0x7f642014f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 >> 172.21.15.28:0/4207150719 conn(0x7f64200b0270 msgr2=0x7f6420156cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.816+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 shutdown_connections 2024-09-13T17:12:38.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.817+0000 7f6426600640 1 -- 172.21.15.28:0/4207150719 wait complete. 2024-09-13T17:12:38.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-09-13T17:12:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.991+0000 7f03cde00640 1 Processor -- start 2024-09-13T17:12:38.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.991+0000 7f03cde00640 1 -- start start 2024-09-13T17:12:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03cde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f03c815a780 0x7f03c8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03cde00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f03c806c250 0x7f03c80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03cde00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c806c660 con 0x7f03c815ab60 2024-09-13T17:12:38.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03cde00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c806c8b0 con 0x7f03c815a780 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03cde00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03c806cb90 con 0x7f03c806c250 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03c6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f03c815a780 0x7f03c8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03c6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f03c815ab60 0x7f03c806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52138/0 (socket says 172.21.15.28:52138) 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.992+0000 7f03c6000640 1 -- 172.21.15.28:0/3968819112 learned_addr learned my addr 172.21.15.28:0/3968819112 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c7400640 1 --2- 172.21.15.28:0/3968819112 >> v2:172.21.15.189:3300/0 conn(0x7f03c806c250 0x7f03c80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c6a00640 1 -- 172.21.15.28:0/3968819112 >> v2:172.21.15.189:3300/0 conn(0x7f03c806c250 msgr2=0x7f03c80712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c6a00640 1 --2- 172.21.15.28:0/3968819112 >> v2:172.21.15.189:3300/0 conn(0x7f03c806c250 0x7f03c80712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c6a00640 1 -- 172.21.15.28:0/3968819112 >> v1:172.21.15.28:6789/0 conn(0x7f03c815ab60 legacy=0x7f03c806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c6a00640 1 -- 172.21.15.28:0/3968819112 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80cc980 con 0x7f03c815a780 2024-09-13T17:12:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.993+0000 7f03c5600640 1 -- 172.21.15.28:0/3968819112 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904951232 0 0) 0x7f03c806c660 con 0x7f03c815ab60 2024-09-13T17:12:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.994+0000 7f03c6a00640 1 --2- 172.21.15.28:0/3968819112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815a780 0x7f03c8070d80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f03c80cc370 tx=0x7f03b802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc5c630db991a8d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.994+0000 7f03c5600640 1 -- 172.21.15.28:0/3968819112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8032440 con 0x7f03c815a780 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.994+0000 7f03c5600640 1 -- 172.21.15.28:0/3968819112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b80325a0 con 0x7f03c815a780 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.994+0000 7f03c5600640 1 -- 172.21.15.28:0/3968819112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8032930 con 0x7f03c815a780 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 -- 172.21.15.28:0/3968819112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815a780 msgr2=0x7f03c8070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 --2- 172.21.15.28:0/3968819112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815a780 0x7f03c8070d80 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f03c80cc370 tx=0x7f03b802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 -- 172.21.15.28:0/3968819112 shutdown_connections 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 --2- 172.21.15.28:0/3968819112 >> v2:172.21.15.189:3300/0 conn(0x7f03c806c250 0x7f03c80712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 --2- 172.21.15.28:0/3968819112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815a780 0x7f03c8070d80 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 -- 172.21.15.28:0/3968819112 >> 172.21.15.28:0/3968819112 conn(0x7f03c80b0270 msgr2=0x7f03c80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 -- 172.21.15.28:0/3968819112 shutdown_connections 2024-09-13T17:12:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.995+0000 7f03cde00640 1 -- 172.21.15.28:0/3968819112 wait complete. 2024-09-13T17:12:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.997+0000 7f03cde00640 1 Processor -- start 2024-09-13T17:12:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.997+0000 7f03cde00640 1 -- start start 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c806c250 0x7f03c81559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 0x7f03c80a9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c8155ef0 con 0x7f03c815ab60 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80a8a00 con 0x7f03c806c250 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03cde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80a8ce0 con 0x7f03c815a780 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34558/0 (socket says 172.21.15.28:34558) 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 -- 172.21.15.28:0/645345556 learned_addr learned my addr 172.21.15.28:0/645345556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c6a00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c806c250 0x7f03c81559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c6000640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 0x7f03c80a9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 -- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 msgr2=0x7f03c80a9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 0x7f03c80a9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 -- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c806c250 msgr2=0x7f03c81559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c806c250 0x7f03c81559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c7400640 1 -- 172.21.15.28:0/645345556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b8009400 con 0x7f03c815ab60 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.998+0000 7f03c6000640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 0x7f03c80a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03c7400640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f03c000efc0 tx=0x7f03c000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0014480 con 0x7f03c815ab60 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c000f040 con 0x7f03c815ab60 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0014820 con 0x7f03c815ab60 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80a8fc0 con 0x7f03c815ab60 2024-09-13T17:12:39.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:38.999+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c80a9580 con 0x7f03c815ab60 2024-09-13T17:12:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.000+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c80cf2f0 con 0x7f03c815ab60 2024-09-13T17:12:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.001+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f03c0014980 con 0x7f03c815ab60 2024-09-13T17:12:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.002+0000 7f03bf400640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 0x7f03a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.002+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f03c0089b70 con 0x7f03c815ab60 2024-09-13T17:12:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.003+0000 7f03c6a00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 0x7f03a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.003+0000 7f03c6a00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 0x7f03a806d0f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03b8034000 tx=0x7f03b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.006+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f03c0054fe0 con 0x7f03c815ab60 2024-09-13T17:12:39.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.185+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7f03c806bb40 con 0x7f03c815ab60 2024-09-13T17:12:39.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.190+0000 7f03bf400640 1 -- 172.21.15.28:0/645345556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v22) v1 ==== 101+0+0 (secure 0 0 0) 0x7f03c00180a0 con 0x7f03c815ab60 2024-09-13T17:12:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 msgr2=0x7f03a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 0x7f03a806d0f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f03b8034000 tx=0x7f03b802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 msgr2=0x7f03c81520b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f03c000efc0 tx=0x7f03c000c490 comp rx=0 tx=0).stop 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 shutdown_connections 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f03a806ac30 0x7f03a806d0f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03c815ab60 0x7f03c81520b0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f03c815a780 0x7f03c80a9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 --2- 172.21.15.28:0/645345556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03c806c250 0x7f03c81559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 >> 172.21.15.28:0/645345556 conn(0x7f03c80b0270 msgr2=0x7f03c806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 shutdown_connections 2024-09-13T17:12:39.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.192+0000 7f03cde00640 1 -- 172.21.15.28:0/645345556 wait complete. 2024-09-13T17:12:39.210 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:12:39.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.367+0000 7f1a6dc00640 1 Processor -- start 2024-09-13T17:12:39.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.367+0000 7f1a6dc00640 1 -- start start 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a6dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 0x7f1a6815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a6dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 0x7f1a6815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a6dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6815ab60 con 0x7f1a68159770 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a6dc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6806f920 con 0x7f1a6806f3f0 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a6dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6815b290 con 0x7f1a68070430 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 0x7f1a6815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a67400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 0x7f1a6815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a67400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 0x7f1a6815c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:57988/0 (socket says 172.21.15.28:57988) 2024-09-13T17:12:39.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.369+0000 7f1a67400640 1 -- 172.21.15.28:0/274325800 learned_addr learned my addr 172.21.15.28:0/274325800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.370+0000 7f1a5fe00640 1 -- 172.21.15.28:0/274325800 >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 msgr2=0x7f1a6815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.370+0000 7f1a5fe00640 1 --2- 172.21.15.28:0/274325800 >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 0x7f1a6815c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.370+0000 7f1a5fe00640 1 -- 172.21.15.28:0/274325800 >> v1:172.21.15.189:6789/0 conn(0x7f1a68159770 legacy=0x7f1a6806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.370+0000 7f1a5fe00640 1 -- 172.21.15.28:0/274325800 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6815b570 con 0x7f1a68070430 2024-09-13T17:12:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.371+0000 7f1a5fe00640 1 --2- 172.21.15.28:0/274325800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 0x7f1a6815bd70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a500097a0 tx=0x7f1a5002c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f7bd0c1959e1070 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.371+0000 7f1a66000640 1 -- 172.21.15.28:0/274325800 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50034640 con 0x7f1a68070430 2024-09-13T17:12:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.371+0000 7f1a66000640 1 -- 172.21.15.28:0/274325800 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a500347a0 con 0x7f1a68070430 2024-09-13T17:12:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.371+0000 7f1a66000640 1 -- 172.21.15.28:0/274325800 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50034b30 con 0x7f1a68070430 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 -- 172.21.15.28:0/274325800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 msgr2=0x7f1a6815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/274325800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 0x7f1a6815bd70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a500097a0 tx=0x7f1a5002c310 comp rx=0 tx=0).stop 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 -- 172.21.15.28:0/274325800 shutdown_connections 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/274325800 >> v2:172.21.15.189:3300/0 conn(0x7f1a6806f3f0 0x7f1a6815c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/274325800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68070430 0x7f1a6815bd70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 -- 172.21.15.28:0/274325800 >> 172.21.15.28:0/274325800 conn(0x7f1a680b0270 msgr2=0x7f1a680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 -- 172.21.15.28:0/274325800 shutdown_connections 2024-09-13T17:12:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.372+0000 7f1a6dc00640 1 -- 172.21.15.28:0/274325800 wait complete. 2024-09-13T17:12:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.374+0000 7f1a6dc00640 1 Processor -- start 2024-09-13T17:12:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.374+0000 7f1a6dc00640 1 -- start start 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a6806f3f0 0x7f1a68156ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68159770 0x7f1a681587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a681570e0 con 0x7f1a68070430 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a681f3900 con 0x7f1a6806f3f0 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a6dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a681f2d80 con 0x7f1a68159770 2024-09-13T17:12:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34568/0 (socket says 172.21.15.28:34568) 2024-09-13T17:12:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a5fe00640 1 -- 172.21.15.28:0/2328121417 learned_addr learned my addr 172.21.15.28:0/2328121417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a66a00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a6806f3f0 0x7f1a68156ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.375+0000 7f1a67400640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68159770 0x7f1a681587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 -- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68159770 msgr2=0x7f1a681587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68159770 0x7f1a681587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 -- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a6806f3f0 msgr2=0x7f1a68156ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a6806f3f0 0x7f1a68156ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 -- 172.21.15.28:0/2328121417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a50009400 con 0x7f1a68070430 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5fe00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a50037040 tx=0x7f1a500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50043070 con 0x7f1a68070430 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a50032650 con 0x7f1a68070430 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a50032a20 con 0x7f1a68070430 2024-09-13T17:12:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.376+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a681f3060 con 0x7f1a68070430 2024-09-13T17:12:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.377+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a681f3620 con 0x7f1a68070430 2024-09-13T17:12:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.378+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6806fd20 con 0x7f1a68070430 2024-09-13T17:12:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.379+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a50032b80 con 0x7f1a68070430 2024-09-13T17:12:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.379+0000 7f1a5ea00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 0x7f1a4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.379+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1a500a9740 con 0x7f1a68070430 2024-09-13T17:12:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.380+0000 7f1a66a00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 0x7f1a4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.380+0000 7f1a66a00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 0x7f1a4406d0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1a58000f60 tx=0x7f1a58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.383+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1a50074ce0 con 0x7f1a68070430 2024-09-13T17:12:39.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.562+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1a681f6060 con 0x7f1a68070430 2024-09-13T17:12:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.566+0000 7f1a5ea00640 1 -- 172.21.15.28:0/2328121417 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1a5002ce60 con 0x7f1a68070430 2024-09-13T17:12:39.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 msgr2=0x7f1a4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 0x7f1a4406d0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1a58000f60 tx=0x7f1a58009040 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 msgr2=0x7f1a681f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f1a50037040 tx=0x7f1a500046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 shutdown_connections 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a4406ac00 0x7f1a4406d0c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a68159770 0x7f1a681587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a68070430 0x7f1a681f23c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 --2- 172.21.15.28:0/2328121417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a6806f3f0 0x7f1a68156ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 >> 172.21.15.28:0/2328121417 conn(0x7f1a680b0270 msgr2=0x7f1a680b16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 shutdown_connections 2024-09-13T17:12:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.569+0000 7f1a6dc00640 1 -- 172.21.15.28:0/2328121417 wait complete. 2024-09-13T17:12:39.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags' 2024-09-13T17:12:39.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:12:39.637 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:12:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.741+0000 7f7317200640 1 Processor -- start 2024-09-13T17:12:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.741+0000 7f7317200640 1 -- start start 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.742+0000 7f7317200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f73100cabc0 0x7f731006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.742+0000 7f7317200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7310070cc0 0x7f7310071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.742+0000 7f7317200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731015e100 con 0x7f731015a540 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.742+0000 7f7317200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73100adb00 con 0x7f73100cabc0 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.742+0000 7f7317200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7310071670 con 0x7f7310070cc0 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f731015a540 0x7f731015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52146/0 (socket says 172.21.15.28:52146) 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730fe00640 1 -- 172.21.15.28:0/3605557236 learned_addr learned my addr 172.21.15.28:0/3605557236 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730f400640 1 --2- 172.21.15.28:0/3605557236 >> v2:172.21.15.28:3300/0 conn(0x7f73100cabc0 0x7f731006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f7315200640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7310070cc0 0x7f7310071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730ea00640 1 -- 172.21.15.28:0/3605557236 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004592692 0 0) 0x7f731015e100 con 0x7f731015a540 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730ea00640 1 -- 172.21.15.28:0/3605557236 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f4003520 con 0x7f731015a540 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730f400640 1 -- 172.21.15.28:0/3605557236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7310070cc0 msgr2=0x7f7310071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730f400640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7310070cc0 0x7f7310071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730f400640 1 -- 172.21.15.28:0/3605557236 >> v1:172.21.15.28:6789/0 conn(0x7f731015a540 legacy=0x7f731015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.743+0000 7f730f400640 1 -- 172.21.15.28:0/3605557236 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731015e3e0 con 0x7f73100cabc0 2024-09-13T17:12:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.744+0000 7f730f400640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73100cabc0 0x7f731006c430 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0097a0 tx=0x7f72fc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50acc19bb903e2cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f730ea00640 1 -- 172.21.15.28:0/3605557236 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc02ae50 con 0x7f73100cabc0 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f730ea00640 1 -- 172.21.15.28:0/3605557236 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc032440 con 0x7f73100cabc0 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f730ea00640 1 -- 172.21.15.28:0/3605557236 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0327d0 con 0x7f73100cabc0 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 -- 172.21.15.28:0/3605557236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73100cabc0 msgr2=0x7f731006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73100cabc0 0x7f731006c430 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0097a0 tx=0x7f72fc02a100 comp rx=0 tx=0).stop 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 -- 172.21.15.28:0/3605557236 shutdown_connections 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7310070cc0 0x7f7310071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 --2- 172.21.15.28:0/3605557236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73100cabc0 0x7f731006c430 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 -- 172.21.15.28:0/3605557236 >> 172.21.15.28:0/3605557236 conn(0x7f7310069e80 msgr2=0x7f731006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 -- 172.21.15.28:0/3605557236 shutdown_connections 2024-09-13T17:12:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.745+0000 7f7317200640 1 -- 172.21.15.28:0/3605557236 wait complete. 2024-09-13T17:12:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.747+0000 7f7317200640 1 Processor -- start 2024-09-13T17:12:39.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.747+0000 7f7317200640 1 -- start start 2024-09-13T17:12:39.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73100cabc0 0x7f731014b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f731015a540 0x7f7310158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100b23f0 con 0x7f7310070cc0 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73100af650 con 0x7f73100cabc0 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f7317200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73100b17b0 con 0x7f731015a540 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f730fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f730fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34586/0 (socket says 172.21.15.28:34586) 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.748+0000 7f730fe00640 1 -- 172.21.15.28:0/3917241463 learned_addr learned my addr 172.21.15.28:0/3917241463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f7315200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f731015a540 0x7f7310158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730f400640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73100cabc0 0x7f731014b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 -- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f731015a540 msgr2=0x7f7310158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f731015a540 0x7f7310158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 -- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73100cabc0 msgr2=0x7f731014b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73100cabc0 0x7f731014b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 -- 172.21.15.28:0/3917241463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc009400 con 0x7f7310070cc0 2024-09-13T17:12:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f730fe00640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f730000c410 tx=0x7f730000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300005ce0 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300005e40 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73000072f0 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.749+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73100b1a50 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.750+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73100b2010 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.751+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7310071130 con 0x7f7310070cc0 2024-09-13T17:12:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.751+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7300007510 con 0x7f7310070cc0 2024-09-13T17:12:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.752+0000 7f7307e00640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 0x7f72f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.752+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7300088710 con 0x7f7310070cc0 2024-09-13T17:12:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.753+0000 7f730f400640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 0x7f72f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:39.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.753+0000 7f730f400640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 0x7f72f006d0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f72fc034000 tx=0x7f72fc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:39.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:39.756+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7300053c60 con 0x7f7310070cc0 2024-09-13T17:12:40.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.000+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags"]} v 0) v1 -- 0x7f73100713b0 con 0x7f7310070cc0 2024-09-13T17:12:40.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.051+0000 7f7307e00640 1 -- 172.21.15.28:0/3917241463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_merge_dirfrags"]}]=0 v244) v1 ==== 151+0+0 (secure 0 0 0) 0x7f7300057280 con 0x7f7310070cc0 2024-09-13T17:12:40.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 msgr2=0x7f72f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 0x7f72f006d0c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f72fc034000 tx=0x7f72fc02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:40.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 msgr2=0x7f73100af110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f730000c410 tx=0x7f730000c8d0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 shutdown_connections 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72f006ac00 0x7f72f006d0c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f731015a540 0x7f7310158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73100cabc0 0x7f731014b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 --2- 172.21.15.28:0/3917241463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7310070cc0 0x7f73100af110 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 >> 172.21.15.28:0/3917241463 conn(0x7f7310069e80 msgr2=0x7f731006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 shutdown_connections 2024-09-13T17:12:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.053+0000 7f7317200640 1 -- 172.21.15.28:0/3917241463 wait complete. 2024-09-13T17:12:40.069 INFO:tasks.cephfs_test_runner:That a directory is merged during scrub. ... ok 2024-09-13T17:12:40.070 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:12:40.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.263+0000 7fbb46c00640 1 Processor -- start 2024-09-13T17:12:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.263+0000 7fbb46c00640 1 -- start start 2024-09-13T17:12:40.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.264+0000 7fbb46c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb4015a540 0x7fbb4006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb46c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 0x7fbb4015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb46c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4015e100 con 0x7fbb4015edb0 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb46c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb400adb00 con 0x7fbb4015a540 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb46c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4015e3e0 con 0x7fbb400cabc0 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb4015a540 0x7fbb4006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb44c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbb4015edb0 0x7fbb40070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52154/0 (socket says 172.21.15.28:52154) 2024-09-13T17:12:40.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb44c00640 1 -- 172.21.15.28:0/1159052105 learned_addr learned my addr 172.21.15.28:0/1159052105 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb37e00640 1 --2- 172.21.15.28:0/1159052105 >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 0x7fbb4015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb3fe00640 1 -- 172.21.15.28:0/1159052105 >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 msgr2=0x7fbb4015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/1159052105 >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 0x7fbb4015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb3fe00640 1 -- 172.21.15.28:0/1159052105 >> v1:172.21.15.28:6789/0 conn(0x7fbb4015edb0 legacy=0x7fbb40070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb3fe00640 1 -- 172.21.15.28:0/1159052105 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4006ce80 con 0x7fbb4015a540 2024-09-13T17:12:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.265+0000 7fbb37e00640 1 --2- 172.21.15.28:0/1159052105 >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 0x7fbb4015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.266+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/1159052105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4015a540 0x7fbb4006c430 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae250 tx=0x7fbb3002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d6f106cd7aa644 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb3f400640 1 -- 172.21.15.28:0/1159052105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3002ae60 con 0x7fbb4015a540 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb3f400640 1 -- 172.21.15.28:0/1159052105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb30032440 con 0x7fbb4015a540 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb3f400640 1 -- 172.21.15.28:0/1159052105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb300327d0 con 0x7fbb4015a540 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 -- 172.21.15.28:0/1159052105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4015a540 msgr2=0x7fbb4006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 --2- 172.21.15.28:0/1159052105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4015a540 0x7fbb4006c430 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fbb400ae250 tx=0x7fbb3002a110 comp rx=0 tx=0).stop 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 -- 172.21.15.28:0/1159052105 shutdown_connections 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 --2- 172.21.15.28:0/1159052105 >> v2:172.21.15.189:3300/0 conn(0x7fbb400cabc0 0x7fbb4015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 --2- 172.21.15.28:0/1159052105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4015a540 0x7fbb4006c430 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 -- 172.21.15.28:0/1159052105 >> 172.21.15.28:0/1159052105 conn(0x7fbb40069e80 msgr2=0x7fbb4006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 -- 172.21.15.28:0/1159052105 shutdown_connections 2024-09-13T17:12:40.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.267+0000 7fbb46c00640 1 -- 172.21.15.28:0/1159052105 wait complete. 2024-09-13T17:12:40.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.269+0000 7fbb46c00640 1 Processor -- start 2024-09-13T17:12:40.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.269+0000 7fbb46c00640 1 -- start start 2024-09-13T17:12:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 0x7fbb4014e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb4015a540 0x7fbb4014c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4014d030 con 0x7fbb4014cba0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400af000 con 0x7fbb400cabc0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb46c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4014ef80 con 0x7fbb4015a540 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb44c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb44c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34606/0 (socket says 172.21.15.28:34606) 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb44c00640 1 -- 172.21.15.28:0/2088514856 learned_addr learned my addr 172.21.15.28:0/2088514856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 0x7fbb4014e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.270+0000 7fbb37e00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb4015a540 0x7fbb4014c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 -- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb4015a540 msgr2=0x7fbb4014c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb4015a540 0x7fbb4014c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 -- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 msgr2=0x7fbb4014e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 0x7fbb4014e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 -- 172.21.15.28:0/2088514856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb30009400 con 0x7fbb4014cba0 2024-09-13T17:12:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 0x7fbb4014e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:40.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb44c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800caa0 tx=0x7fbb3800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3800aba0 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3800ad00 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38012600 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4014f260 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.271+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4014f820 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.272+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb400af140 con 0x7fbb4014cba0 2024-09-13T17:12:40.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.273+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbb38007070 con 0x7fbb4014cba0 2024-09-13T17:12:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.274+0000 7fbb3cc00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 0x7fbb1c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.274+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbb38088ed0 con 0x7fbb4014cba0 2024-09-13T17:12:40.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.275+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 0x7fbb1c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.275+0000 7fbb3fe00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 0x7fbb1c06d0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbb30034000 tx=0x7fbb3002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.278+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbb38054360 con 0x7fbb4014cba0 2024-09-13T17:12:40.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.459+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb4015e850 con 0x7fbb4014cba0 2024-09-13T17:12:40.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.460+0000 7fbb3cc00640 1 -- 172.21.15.28:0/2088514856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+13116 (secure 0 0 0) 0x7fbb38017090 con 0x7fbb4014cba0 2024-09-13T17:12:40.461 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:40.461 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":68,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:38.034099+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2705534945":"2024-09-14T17:12:34.682216+0000","172.21.15.28:6834/2705534945":"2024-09-14T17:12:34.682216+0000","172.21.15.189:6837/693307152":"2024-09-14T17:12:17.578224+0000","172.21.15.189:6835/693307152":"2024-09-14T17:12:17.578224+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 msgr2=0x7fbb1c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 0x7fbb1c06d0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fbb30034000 tx=0x7fbb3002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 msgr2=0x7fbb4013e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fbb3800caa0 tx=0x7fbb3800cf60 comp rx=0 tx=0).stop 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 shutdown_connections 2024-09-13T17:12:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb1c06ac00 0x7fbb1c06d0c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb4014cba0 0x7fbb4013e380 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb4015a540 0x7fbb4014c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 --2- 172.21.15.28:0/2088514856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb400cabc0 0x7fbb4014e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 >> 172.21.15.28:0/2088514856 conn(0x7fbb40069e80 msgr2=0x7fbb4006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 shutdown_connections 2024-09-13T17:12:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.462+0000 7fbb46c00640 1 -- 172.21.15.28:0/2088514856 wait complete. 2024-09-13T17:12:40.480 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:12:40.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:12:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.638+0000 7f63d6400640 1 Processor -- start 2024-09-13T17:12:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.638+0000 7f63d6400640 1 -- start start 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63d6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63d006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63d6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d006c8b0 con 0x7f63d0071490 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63d6400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d006cb90 con 0x7f63d015a780 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63d6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00cc980 con 0x7f63d015ab60 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63cfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63d0071490 0x7f63d006c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52160/0 (socket says 172.21.15.28:52160) 2024-09-13T17:12:40.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.639+0000 7f63cfe00640 1 -- 172.21.15.28:0/3621253721 learned_addr learned my addr 172.21.15.28:0/3621253721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63c7e00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63d006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550895158 0 0) 0x7f63d006c8b0 con 0x7f63d0071490 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4003520 con 0x7f63d0071490 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928014934 0 0) 0x7f63d006cb90 con 0x7f63d015a780 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d006c8b0 con 0x7f63d015a780 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1200794558 0 0) 0x7f63b4003520 con 0x7f63d0071490 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 msgr2=0x7f63d006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:40.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63d006bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 >> v1:172.21.15.189:6789/0 conn(0x7f63d015a780 legacy=0x7f63d0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00ccc60 con 0x7f63d0071490 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f63c80019f0 con 0x7f63d0071490 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 >> v1:172.21.15.28:6789/0 conn(0x7f63d0071490 legacy=0x7f63d006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63b4003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004090 0x7f63b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b4004d00 0x7f63b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00cc980 con 0x7f63b4004d00 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b400ba60 con 0x7f63b4004090 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b400bc20 con 0x7f63d015ab60 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.640+0000 7f63cfe00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b4004d00 0x7f63b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cf400640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63b4003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63c7e00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004090 0x7f63b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cfe00640 1 -- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 msgr2=0x7f63b4003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cfe00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d015ab60 0x7f63b4003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cfe00640 1 -- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004090 msgr2=0x7f63b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cfe00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004090 0x7f63b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.641+0000 7f63cfe00640 1 -- 172.21.15.28:0/3621253721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b4003520 con 0x7f63b4004d00 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.642+0000 7f63cfe00640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b4004d00 0x7f63b4005180 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f63d006f3a0 tx=0x7f63b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424e8f56de0ffd1d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.642+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c802b940 con 0x7f63b4004d00 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.642+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c802bac0 con 0x7f63b4004d00 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.642+0000 7f63cea00640 1 -- 172.21.15.28:0/3621253721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8034470 con 0x7f63b4004d00 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 -- 172.21.15.28:0/3621253721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b4004d00 msgr2=0x7f63b4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 --2- 172.21.15.28:0/3621253721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63b4004d00 0x7f63b4005180 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f63d006f3a0 tx=0x7f63b40057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63cfe00640 1 -- 172.21.15.28:0/3621253721 reap_dead start 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 -- 172.21.15.28:0/3621253721 shutdown_connections 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 -- 172.21.15.28:0/3621253721 >> 172.21.15.28:0/3621253721 conn(0x7f63d00b0270 msgr2=0x7f63d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 -- 172.21.15.28:0/3621253721 shutdown_connections 2024-09-13T17:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.643+0000 7f63d6400640 1 -- 172.21.15.28:0/3621253721 wait complete. 2024-09-13T17:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.645+0000 7f63d6400640 1 Processor -- start 2024-09-13T17:12:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.645+0000 7f63d6400640 1 -- start start 2024-09-13T17:12:40.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 0x7f63d0154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d00a97c0 0x7f63d00acab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 0x7f63d00aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00a93a0 con 0x7f63d00a9c40 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d01f11a0 con 0x7f63b4004d00 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63d6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d00a8440 con 0x7f63d00a97c0 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63cf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 0x7f63d0154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 0x7f63d00aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63c7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d00a97c0 0x7f63d00acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63cf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 0x7f63d0154460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58042/0 (socket says 172.21.15.28:58042) 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63cf400640 1 -- 172.21.15.28:0/584304808 learned_addr learned my addr 172.21.15.28:0/584304808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.646+0000 7f63cfe00640 1 -- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d00a97c0 msgr2=0x7f63d00acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63cfe00640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d00a97c0 0x7f63d00acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63cfe00640 1 -- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 msgr2=0x7f63d0154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63cfe00640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 0x7f63d0154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63cfe00640 1 -- 172.21.15.28:0/584304808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00a8720 con 0x7f63d00a9c40 2024-09-13T17:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63cfe00640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 0x7f63d00aaa90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f63c8030000 tx=0x7f63c8001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8038070 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8034670 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c80349e0 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d00a8a00 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.647+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d00a8fc0 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.648+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d00a94e0 con 0x7f63d00a9c40 2024-09-13T17:12:40.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.649+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f63c8031070 con 0x7f63d00a9c40 2024-09-13T17:12:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.650+0000 7f63c7400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 0x7f63ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.650+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f63c803b040 con 0x7f63d00a9c40 2024-09-13T17:12:40.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.651+0000 7f63cf400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 0x7f63ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:40.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.651+0000 7f63cf400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 0x7f63ac06d0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63d00ac8a0 tx=0x7f63c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.655+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f63d00a94e0 con 0x7f63d00a9c40 2024-09-13T17:12:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.831+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63d00a94e0 con 0x7f63d00a9c40 2024-09-13T17:12:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.832+0000 7f63c7400640 1 -- 172.21.15.28:0/584304808 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63c8045020 con 0x7f63d00a9c40 2024-09-13T17:12:40.833 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:12:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 msgr2=0x7f63ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 0x7f63ac06d0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f63d00ac8a0 tx=0x7f63c0009040 comp rx=0 tx=0).stop 2024-09-13T17:12:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 msgr2=0x7f63d00aaa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 0x7f63d00aaa90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f63c8030000 tx=0x7f63c8001d00 comp rx=0 tx=0).stop 2024-09-13T17:12:40.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 shutdown_connections 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63ac06ac00 0x7f63ac06d0c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d00a9c40 0x7f63d00aaa90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d00a97c0 0x7f63d00acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 --2- 172.21.15.28:0/584304808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b4004d00 0x7f63d0154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 >> 172.21.15.28:0/584304808 conn(0x7f63d00b0270 msgr2=0x7f63d00ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 shutdown_connections 2024-09-13T17:12:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:40.834+0000 7f63d6400640 1 -- 172.21.15.28:0/584304808 wait complete. 2024-09-13T17:12:40.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair' 2024-09-13T17:12:41.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.004+0000 7f1654e00640 1 Processor -- start 2024-09-13T17:12:41.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.005+0000 7f1654e00640 1 -- start start 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f1654e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16500cabc0 0x7f165006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f1654e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165015e100 con 0x7f165015a540 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f1654e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165015e3e0 con 0x7f1650070cc0 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f1654e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16500adb00 con 0x7f16500cabc0 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f164de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f165015a540 0x7f165015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52176/0 (socket says 172.21.15.28:52176) 2024-09-13T17:12:41.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.006+0000 7f164de00640 1 -- 172.21.15.28:0/2288359859 learned_addr learned my addr 172.21.15.28:0/2288359859 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f1645400640 1 --2- 172.21.15.28:0/2288359859 >> v2:172.21.15.189:3300/0 conn(0x7f16500cabc0 0x7f165006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181195573 0 0) 0x7f165015e100 con 0x7f165015a540 2024-09-13T17:12:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1634003520 con 0x7f165015a540 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1194800151 0 0) 0x7f1634003520 con 0x7f165015a540 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 >> v2:172.21.15.189:3300/0 conn(0x7f16500cabc0 msgr2=0x7f165006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 --2- 172.21.15.28:0/2288359859 >> v2:172.21.15.189:3300/0 conn(0x7f16500cabc0 0x7f165006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 >> v1:172.21.15.189:6789/0 conn(0x7f1650070cc0 legacy=0x7f165006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165015ef80 con 0x7f165015a540 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f1640001670 con 0x7f165015a540 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 >> v1:172.21.15.28:6789/0 conn(0x7f165015a540 legacy=0x7f165015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f1634003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16340047c0 0x7f1634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340051d0 0x7f163400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163400c0f0 con 0x7f16340039c0 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163400c260 con 0x7f16340051d0 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163400c420 con 0x7f16340047c0 2024-09-13T17:12:41.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.007+0000 7f164de00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f1634003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f1645400640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16340047c0 0x7f1634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164e800640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340051d0 0x7f163400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164de00640 1 -- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16340047c0 msgr2=0x7f1634004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164de00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16340047c0 0x7f1634004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164de00640 1 -- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340051d0 msgr2=0x7f163400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164de00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340051d0 0x7f163400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164de00640 1 -- 172.21.15.28:0/2288359859 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634003520 con 0x7f16340039c0 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164e800640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16340051d0 0x7f163400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:41.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.008+0000 7f164e800640 1 -- 172.21.15.28:0/2288359859 reap_dead start 2024-09-13T17:12:41.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.009+0000 7f164de00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f1634003de0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1640001eb0 tx=0x7f16340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d04acbe9fbfb15c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.009+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640005ad0 con 0x7f16340039c0 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.009+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164002d070 con 0x7f16340039c0 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.009+0000 7f164d400640 1 -- 172.21.15.28:0/2288359859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640007070 con 0x7f16340039c0 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 -- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 msgr2=0x7f1634003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f1634003de0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f1640001eb0 tx=0x7f16340057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 -- 172.21.15.28:0/2288359859 shutdown_connections 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 --2- 172.21.15.28:0/2288359859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f1634003de0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 -- 172.21.15.28:0/2288359859 >> 172.21.15.28:0/2288359859 conn(0x7f1650069e80 msgr2=0x7f165006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 -- 172.21.15.28:0/2288359859 shutdown_connections 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.010+0000 7f1654e00640 1 -- 172.21.15.28:0/2288359859 wait complete. 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.012+0000 7f1654e00640 1 Processor -- start 2024-09-13T17:12:41.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.012+0000 7f1654e00640 1 -- start start 2024-09-13T17:12:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165013e150 0x7f1650159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165013f7b0 0x7f16501401e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165013de80 con 0x7f16340039c0 2024-09-13T17:12:41.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165015da90 con 0x7f165013e150 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1654e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165013e710 con 0x7f165013f7b0 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34650/0 (socket says 172.21.15.28:34650) 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 -- 172.21.15.28:0/100212673 learned_addr learned my addr 172.21.15.28:0/100212673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164e800640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165013f7b0 0x7f16501401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f1645400640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165013e150 0x7f1650159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 -- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165013f7b0 msgr2=0x7f16501401e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165013f7b0 0x7f16501401e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 -- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165013e150 msgr2=0x7f1650159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165013e150 0x7f1650159580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.013+0000 7f164de00640 1 -- 172.21.15.28:0/100212673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165013e9f0 con 0x7f16340039c0 2024-09-13T17:12:41.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f164de00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f164002ac00 tx=0x7f1640001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1640004ba0 con 0x7f16340039c0 2024-09-13T17:12:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1640004d00 con 0x7f16340039c0 2024-09-13T17:12:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16400072b0 con 0x7f16340039c0 2024-09-13T17:12:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165013ecd0 con 0x7f16340039c0 2024-09-13T17:12:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.014+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165013f290 con 0x7f16340039c0 2024-09-13T17:12:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.015+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165013f420 con 0x7f16340039c0 2024-09-13T17:12:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.016+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f164002f030 con 0x7f16340039c0 2024-09-13T17:12:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.017+0000 7f1646a00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 0x7f162c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.017+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f16400a89f0 con 0x7f16340039c0 2024-09-13T17:12:41.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.017+0000 7f1645400640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 0x7f162c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:41.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.018+0000 7f1645400640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 0x7f162c06d0f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1650144860 tx=0x7f1638008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:41.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.021+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1640073e60 con 0x7f16340039c0 2024-09-13T17:12:41.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:41.266+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]} v 0) v1 -- 0x7f165013f420 con 0x7f16340039c0 2024-09-13T17:12:42.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.058+0000 7f1646a00640 1 -- 172.21.15.28:0/100212673 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]}]=0 v246) v1 ==== 146+0+0 (secure 0 0 0) 0x7f1640077480 con 0x7f16340039c0 2024-09-13T17:12:42.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 msgr2=0x7f162c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 0x7f162c06d0f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1650144860 tx=0x7f1638008040 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 msgr2=0x7f165015d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f164002ac00 tx=0x7f1640001930 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 shutdown_connections 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f162c06ac30 0x7f162c06d0f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f165013f7b0 0x7f16501401e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f165013e150 0x7f1650159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 --2- 172.21.15.28:0/100212673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16340039c0 0x7f165015d550 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 >> 172.21.15.28:0/100212673 conn(0x7f1650069e80 msgr2=0x7f16500cb340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 shutdown_connections 2024-09-13T17:12:42.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.061+0000 7f1654e00640 1 -- 172.21.15.28:0/100212673 wait complete. 2024-09-13T17:12:42.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:12:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.243+0000 7fb3a1600640 1 Processor -- start 2024-09-13T17:12:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.243+0000 7fb3a1600640 1 -- start start 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.244+0000 7fb3a1600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb39c159770 0x7fb39c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb3a1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 0x7fb39c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb3a1600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c06f920 con 0x7fb39c06f3f0 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb3a1600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c15b290 con 0x7fb39c159770 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb3a1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c15b570 con 0x7fb39c15cd80 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb393e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb39c06f3f0 0x7fb39c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52178/0 (socket says 172.21.15.28:52178) 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb393e00640 1 -- 172.21.15.28:0/3253565076 learned_addr learned my addr 172.21.15.28:0/3253565076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3253565076 >> v2:172.21.15.189:3300/0 conn(0x7fb39c159770 0x7fb39c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb39b400640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 0x7fb39c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.245+0000 7fb39a000640 1 -- 172.21.15.28:0/3253565076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43144723 0 0) 0x7fb39c06f920 con 0x7fb39c06f3f0 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39a000640 1 -- 172.21.15.28:0/3253565076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb380003520 con 0x7fb39c06f3f0 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39aa00640 1 -- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 msgr2=0x7fb39c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 0x7fb39c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39aa00640 1 -- 172.21.15.28:0/3253565076 >> v1:172.21.15.28:6789/0 conn(0x7fb39c06f3f0 legacy=0x7fb39c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39aa00640 1 -- 172.21.15.28:0/3253565076 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c15b850 con 0x7fb39c159770 2024-09-13T17:12:42.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.246+0000 7fb39b400640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 0x7fb39c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:42.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb39aa00640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c159770 0x7fb39c06fd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0097a0 tx=0x7fb38c02a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63db83163189937 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb39a000640 1 -- 172.21.15.28:0/3253565076 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c02ae40 con 0x7fb39c159770 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb39a000640 1 -- 172.21.15.28:0/3253565076 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c032440 con 0x7fb39c159770 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb39a000640 1 -- 172.21.15.28:0/3253565076 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0327b0 con 0x7fb39c159770 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb3a1600640 1 -- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c159770 msgr2=0x7fb39c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.247+0000 7fb3a1600640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c159770 0x7fb39c06fd20 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0097a0 tx=0x7fb38c02a0f0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 -- 172.21.15.28:0/3253565076 shutdown_connections 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c15cd80 0x7fb39c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 --2- 172.21.15.28:0/3253565076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c159770 0x7fb39c06fd20 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 -- 172.21.15.28:0/3253565076 >> 172.21.15.28:0/3253565076 conn(0x7fb39c0b0270 msgr2=0x7fb39c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 -- 172.21.15.28:0/3253565076 shutdown_connections 2024-09-13T17:12:42.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.248+0000 7fb3a1600640 1 -- 172.21.15.28:0/3253565076 wait complete. 2024-09-13T17:12:42.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.249+0000 7fb3a1600640 1 Processor -- start 2024-09-13T17:12:42.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.250+0000 7fb3a1600640 1 -- start start 2024-09-13T17:12:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.250+0000 7fb3a1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c06f3f0 0x7fb39c1467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.250+0000 7fb3a1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.250+0000 7fb3a1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c15cd80 0x7fb39c0acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb3a1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c153260 con 0x7fb39c159770 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb3a1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c1511e0 con 0x7fb39c15cd80 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb3a1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c143aa0 con 0x7fb39c06f3f0 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34654/0 (socket says 172.21.15.28:34654) 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 -- 172.21.15.28:0/2720156992 learned_addr learned my addr 172.21.15.28:0/2720156992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb39aa00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c06f3f0 0x7fb39c1467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb39b400640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c15cd80 0x7fb39c0acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 -- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c06f3f0 msgr2=0x7fb39c1467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c06f3f0 0x7fb39c1467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 -- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c15cd80 msgr2=0x7fb39c0acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c15cd80 0x7fb39c0acef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.251+0000 7fb393e00640 1 -- 172.21.15.28:0/2720156992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb394009a70 con 0x7fb39c159770 2024-09-13T17:12:42.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb393e00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c410 tx=0x7fb38400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384005ce0 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb384005e40 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb384007310 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38c009400 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.252+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c152ba0 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.253+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c1533a0 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.254+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb384007530 con 0x7fb39c159770 2024-09-13T17:12:42.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.255+0000 7fb392a00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 0x7fb37806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.255+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fb384088690 con 0x7fb39c159770 2024-09-13T17:12:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.255+0000 7fb39aa00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 0x7fb37806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.256+0000 7fb39aa00640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 0x7fb37806d0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb38c034000 tx=0x7fb38c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.259+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb39c1533a0 con 0x7fb39c159770 2024-09-13T17:12:42.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.437+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fb39c1533a0 con 0x7fb39c159770 2024-09-13T17:12:42.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.442+0000 7fb392a00640 1 -- 172.21.15.28:0/2720156992 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb39c1533a0 con 0x7fb39c159770 2024-09-13T17:12:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.444+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 msgr2=0x7fb37806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.444+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 0x7fb37806d0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fb38c034000 tx=0x7fb38c02e000 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.444+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 msgr2=0x7fb39c1507b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.444+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb38400c410 tx=0x7fb38400c810 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 shutdown_connections 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb37806ac00 0x7fb37806d0c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb39c15cd80 0x7fb39c0acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb39c159770 0x7fb39c1507b0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 --2- 172.21.15.28:0/2720156992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb39c06f3f0 0x7fb39c1467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 >> 172.21.15.28:0/2720156992 conn(0x7fb39c0b0270 msgr2=0x7fb39c0ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 shutdown_connections 2024-09-13T17:12:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.445+0000 7fb3a1600640 1 -- 172.21.15.28:0/2720156992 wait complete. 2024-09-13T17:12:42.462 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:12:42.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:12:42.555 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:12:42.654 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:12:42.654 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:42.654 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:42.690 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:12:42.690 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:12:42.690 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:42.727 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:12:42.728 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:12:42.728 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:42.765 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:12:42.765 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:12:42.765 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:12:42.803 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:12:42.803 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:12:42.803 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:42.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.959+0000 7f0bdec00640 1 Processor -- start 2024-09-13T17:12:42.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.960+0000 7f0bdec00640 1 -- start start 2024-09-13T17:12:42.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0bd8071c00 0x7f0bd8146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bd80b27c0 0x7f0bd8070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd8143070 con 0x7f0bd80ad260 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd8071fe0 con 0x7f0bd80b27c0 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd80ada70 con 0x7f0bd8071c00 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bcf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0bd80ad260 0x7f0bd80b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52180/0 (socket says 172.21.15.28:52180) 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bcf400640 1 -- 172.21.15.28:0/2039832571 learned_addr learned my addr 172.21.15.28:0/2039832571 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/2039832571 >> v2:172.21.15.28:3300/0 conn(0x7f0bd80b27c0 0x7f0bd8070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bd7e00640 1 --2- 172.21.15.28:0/2039832571 >> v2:172.21.15.189:3300/0 conn(0x7f0bd8071c00 0x7f0bd8146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.961+0000 7f0bd7400640 1 -- 172.21.15.28:0/2039832571 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37600254 0 0) 0x7f0bd8143070 con 0x7f0bd80ad260 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.962+0000 7f0bd7400640 1 -- 172.21.15.28:0/2039832571 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bbc003520 con 0x7f0bd80ad260 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.962+0000 7f0bdcc00640 1 -- 172.21.15.28:0/2039832571 >> v2:172.21.15.189:3300/0 conn(0x7f0bd8071c00 msgr2=0x7f0bd8146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.962+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/2039832571 >> v2:172.21.15.189:3300/0 conn(0x7f0bd8071c00 0x7f0bd8146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.962+0000 7f0bdcc00640 1 -- 172.21.15.28:0/2039832571 >> v1:172.21.15.28:6789/0 conn(0x7f0bd80ad260 legacy=0x7f0bd80b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.962+0000 7f0bdcc00640 1 -- 172.21.15.28:0/2039832571 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd80add50 con 0x7f0bd80b27c0 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/2039832571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd8070290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b4fd0 tx=0x7f0bd002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d7e5273eae8fc0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bd7400640 1 -- 172.21.15.28:0/2039832571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd002aea0 con 0x7f0bd80b27c0 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bd7400640 1 -- 172.21.15.28:0/2039832571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd002d050 con 0x7f0bd80b27c0 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bd7400640 1 -- 172.21.15.28:0/2039832571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd0007960 con 0x7f0bd80b27c0 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bdec00640 1 -- 172.21.15.28:0/2039832571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 msgr2=0x7f0bd8070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.963+0000 7f0bdec00640 1 --2- 172.21.15.28:0/2039832571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd8070290 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80b4fd0 tx=0x7f0bd002a680 comp rx=0 tx=0).stop 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 -- 172.21.15.28:0/2039832571 shutdown_connections 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 --2- 172.21.15.28:0/2039832571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd8070290 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 --2- 172.21.15.28:0/2039832571 >> v2:172.21.15.189:3300/0 conn(0x7f0bd8071c00 0x7f0bd8146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 -- 172.21.15.28:0/2039832571 >> 172.21.15.28:0/2039832571 conn(0x7f0bd80b0b60 msgr2=0x7f0bd80b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 -- 172.21.15.28:0/2039832571 shutdown_connections 2024-09-13T17:12:42.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.964+0000 7f0bdec00640 1 -- 172.21.15.28:0/2039832571 wait complete. 2024-09-13T17:12:42.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.965+0000 7f0bdec00640 1 Processor -- start 2024-09-13T17:12:42.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.965+0000 7f0bdec00640 1 -- start start 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd8071c00 0x7f0bd8160cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd80ad260 0x7f0bd815cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd80cb690 con 0x7f0bd80b27c0 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd815d290 con 0x7f0bd80ad260 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.966+0000 7f0bdec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd815d530 con 0x7f0bd8071c00 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34662/0 (socket says 172.21.15.28:34662) 2024-09-13T17:12:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 -- 172.21.15.28:0/3174678282 learned_addr learned my addr 172.21.15.28:0/3174678282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bcf400640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd80ad260 0x7f0bd815cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bd7e00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd8071c00 0x7f0bd8160cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 -- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd8071c00 msgr2=0x7f0bd8160cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd8071c00 0x7f0bd8160cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 -- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd80ad260 msgr2=0x7f0bd815cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd80ad260 0x7f0bd815cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 -- 172.21.15.28:0/3174678282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd0009a70 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.967+0000 7f0bdcc00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0bd002ac70 tx=0x7f0bd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.968+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd003c830 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.968+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd003c990 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.968+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd003cd00 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.968+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd815d810 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.968+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd815ddd0 con 0x7f0bd80b27c0 2024-09-13T17:12:42.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.969+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd8146050 con 0x7f0bd80b27c0 2024-09-13T17:12:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.970+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0bd0031400 con 0x7f0bd80b27c0 2024-09-13T17:12:42.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.971+0000 7f0bd4c00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 0x7f0bb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.971+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0bd00aa9d0 con 0x7f0bd80b27c0 2024-09-13T17:12:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.971+0000 7f0bd7e00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 0x7f0bb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:42.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.972+0000 7f0bd7e00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 0x7f0bb406d0c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009aa0 tx=0x7f0bc8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:42.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:42.975+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0bd0075ef0 con 0x7f0bd80b27c0 2024-09-13T17:12:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.193+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bd806e420 con 0x7f0bd80b27c0 2024-09-13T17:12:43.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.194+0000 7f0bd4c00640 1 -- 172.21.15.28:0/3174678282 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f0bd004ade0 con 0x7f0bd80b27c0 2024-09-13T17:12:43.195 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:43.195 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":57,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5864,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/627091796","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":627091796},{"type":"v1","addr":"172.21.15.189:6835","nonce":627091796}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5867,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1046608134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":1046608134},{"type":"v1","addr":"172.21.15.189:6837","nonce":1046608134}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6672,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3778438142","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3778438142},{"type":"v1","addr":"172.21.15.28:6833","nonce":3778438142}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":6855,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3039283076","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3039283076},{"type":"v1","addr":"172.21.15.28:6835","nonce":3039283076}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":57}],"filesystems":[]} 2024-09-13T17:12:43.195 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 57 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 msgr2=0x7f0bb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 0x7f0bb406d0c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009aa0 tx=0x7f0bc8009190 comp rx=0 tx=0).stop 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 msgr2=0x7f0bd815ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f0bd002ac70 tx=0x7f0bd0004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 shutdown_connections 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb406ac00 0x7f0bb406d0c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd80b27c0 0x7f0bd815ed10 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:43.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd80ad260 0x7f0bd815cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 --2- 172.21.15.28:0/3174678282 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd8071c00 0x7f0bd8160cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 >> 172.21.15.28:0/3174678282 conn(0x7f0bd80b0b60 msgr2=0x7f0bd80b0340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 shutdown_connections 2024-09-13T17:12:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:43.196+0000 7f0bdec00640 1 -- 172.21.15.28:0/3174678282 wait complete. 2024-09-13T17:12:43.213 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:12:43.213 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:12:43.213 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:12:43.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:43.214 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:12:43.214 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:12:43.214 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:12:43.214 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:43.214 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:12:43.214 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:12:43.214 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:12:43.215 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:43.215 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:12:43.215 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:12:43.215 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:12:43.215 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:12:43.228 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:12:43.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:12:43.229 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:12:43.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:12:43.231 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:12:43.231 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:12:43.273 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:12:43.273 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:43.276 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:12:43.276 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:43.276 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:12:43.276 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:43.401 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:12:43.401 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:12:43.403 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:12:46.318 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-13T17:12:49.232 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:12:49.232 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:12:49.237 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:12:49.237 DEBUG:teuthology.parallel:result is None 2024-09-13T17:12:49.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:12:49.371 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:12:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.405+0000 7fed2ee00640 1 Processor -- start 2024-09-13T17:12:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.406+0000 7fed2ee00640 1 -- start start 2024-09-13T17:12:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed2815ab70 0x7fed2815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed2806c370 0x7fed2806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed280b2090 con 0x7fed280c40a0 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2806cd40 con 0x7fed2806c370 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280b2370 con 0x7fed2815ab70 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fed280c40a0 0x7fed280b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59802/0 (socket says 172.21.15.28:59802) 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2da00640 1 -- 172.21.15.28:0/1308985263 learned_addr learned my addr 172.21.15.28:0/1308985263 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2e400640 1 --2- 172.21.15.28:0/1308985263 >> v2:172.21.15.28:3300/0 conn(0x7fed2806c370 0x7fed2806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.407+0000 7fed2d000640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed2815ab70 0x7fed2815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed27e00640 1 -- 172.21.15.28:0/1308985263 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618723330 0 0) 0x7fed280b2090 con 0x7fed280c40a0 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed27e00640 1 -- 172.21.15.28:0/1308985263 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed280c40a0 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed2e400640 1 -- 172.21.15.28:0/1308985263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed2815ab70 msgr2=0x7fed2815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed2e400640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed2815ab70 0x7fed2815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed2e400640 1 -- 172.21.15.28:0/1308985263 >> v1:172.21.15.28:6789/0 conn(0x7fed280c40a0 legacy=0x7fed280b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.408+0000 7fed2e400640 1 -- 172.21.15.28:0/1308985263 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed280b2650 con 0x7fed2806c370 2024-09-13T17:12:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.409+0000 7fed2e400640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2806c800 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fed20009d90 tx=0x7fed2002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31d24f08053ca6b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.409+0000 7fed27e00640 1 -- 172.21.15.28:0/1308985263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20007740 con 0x7fed2806c370 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.409+0000 7fed27e00640 1 -- 172.21.15.28:0/1308985263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2002d050 con 0x7fed2806c370 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.409+0000 7fed27e00640 1 -- 172.21.15.28:0/1308985263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed20007aa0 con 0x7fed2806c370 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.409+0000 7fed2ee00640 1 -- 172.21.15.28:0/1308985263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 msgr2=0x7fed2806c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2806c800 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fed20009d90 tx=0x7fed2002a660 comp rx=0 tx=0).stop 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 -- 172.21.15.28:0/1308985263 shutdown_connections 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2806c800 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1308985263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed2815ab70 0x7fed2815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 -- 172.21.15.28:0/1308985263 >> 172.21.15.28:0/1308985263 conn(0x7fed28073470 msgr2=0x7fed28069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 -- 172.21.15.28:0/1308985263 shutdown_connections 2024-09-13T17:12:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.410+0000 7fed2ee00640 1 -- 172.21.15.28:0/1308985263 wait complete. 2024-09-13T17:12:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.411+0000 7fed2ee00640 1 Processor -- start 2024-09-13T17:12:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.412+0000 7fed2ee00640 1 -- start start 2024-09-13T17:12:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.412+0000 7fed2ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.412+0000 7fed2ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed280c40a0 0x7fed280c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2815ab70 0x7fed28142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2815a6b0 con 0x7fed2806c370 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280c97f0 con 0x7fed2815ab70 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed280c9ad0 con 0x7fed280c40a0 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52074/0 (socket says 172.21.15.28:52074) 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 -- 172.21.15.28:0/1903646897 learned_addr learned my addr 172.21.15.28:0/1903646897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2d000640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed280c40a0 0x7fed280c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2e400640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2815ab70 0x7fed28142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 -- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed280c40a0 msgr2=0x7fed280c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:49.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed280c40a0 0x7fed280c8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 -- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2815ab70 msgr2=0x7fed28142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2815ab70 0x7fed28142200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 -- 172.21.15.28:0/1903646897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed20009a70 con 0x7fed2806c370 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.413+0000 7fed2da00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00c430 tx=0x7fed1c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.414+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c005ce0 con 0x7fed2806c370 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.414+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c005e40 con 0x7fed2806c370 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.414+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1c0072f0 con 0x7fed2806c370 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.414+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2813cc60 con 0x7fed2806c370 2024-09-13T17:12:49.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.414+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2813d220 con 0x7fed2806c370 2024-09-13T17:12:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.415+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed280c4480 con 0x7fed2806c370 2024-09-13T17:12:49.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.416+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fed1c007450 con 0x7fed2806c370 2024-09-13T17:12:49.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.417+0000 7fed25600640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 0x7fed080714d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:49.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.417+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fed1c0887a0 con 0x7fed2806c370 2024-09-13T17:12:49.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.417+0000 7fed2d000640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 0x7fed080714d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.418+0000 7fed2d000640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 0x7fed080714d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed2813fd70 tx=0x7fed14009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:49.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.422+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fed1c0576c0 con 0x7fed2806c370 2024-09-13T17:12:49.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:49.609+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fed28143470 con 0x7fed2806c370 2024-09-13T17:12:50.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.385+0000 7fed25600640 1 -- 172.21.15.28:0/1903646897 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v73) v1 ==== 97+0+0 (secure 0 0 0) 0x7fed1c017080 con 0x7fed2806c370 2024-09-13T17:12:50.388 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.388+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 msgr2=0x7fed080714d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.388+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 0x7fed080714d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fed2813fd70 tx=0x7fed14009190 comp rx=0 tx=0).stop 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.388+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 msgr2=0x7fed2815a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.388+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fed1c00c430 tx=0x7fed1c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 shutdown_connections 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed0806f010 0x7fed080714d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed2815ab70 0x7fed28142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed280c40a0 0x7fed280c8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 --2- 172.21.15.28:0/1903646897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2806c370 0x7fed2815a170 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 >> 172.21.15.28:0/1903646897 conn(0x7fed28073470 msgr2=0x7fed281409b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 shutdown_connections 2024-09-13T17:12:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.389+0000 7fed2ee00640 1 -- 172.21.15.28:0/1903646897 wait complete. 2024-09-13T17:12:50.410 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:12:50.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.570+0000 7f0cc8800640 1 Processor -- start 2024-09-13T17:12:50.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.570+0000 7f0cc8800640 1 -- start start 2024-09-13T17:12:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc8800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cc0160310 0x7f0cc0160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc8800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0cc00ac850 0x7f0cc00acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc8800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc0146550 con 0x7f0cc0146170 2024-09-13T17:12:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc8800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc015cd20 con 0x7f0cc0160310 2024-09-13T17:12:50.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc8800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc015d280 con 0x7f0cc00ac850 2024-09-13T17:12:50.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0cc0146170 0x7f0cc015d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59812/0 (socket says 172.21.15.28:59812) 2024-09-13T17:12:50.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc5e00640 1 -- 172.21.15.28:0/764906149 learned_addr learned my addr 172.21.15.28:0/764906149 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:50.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.571+0000 7f0cc5400640 1 --2- 172.21.15.28:0/764906149 >> v2:172.21.15.28:3300/0 conn(0x7f0cc0160310 0x7f0cc0160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc6800640 1 --2- 172.21.15.28:0/764906149 >> v2:172.21.15.189:3300/0 conn(0x7f0cc00ac850 0x7f0cc00acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc4a00640 1 -- 172.21.15.28:0/764906149 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890641390 0 0) 0x7f0cc0146550 con 0x7f0cc0146170 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc4a00640 1 -- 172.21.15.28:0/764906149 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ca8003520 con 0x7f0cc0146170 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc5400640 1 -- 172.21.15.28:0/764906149 >> v2:172.21.15.189:3300/0 conn(0x7f0cc00ac850 msgr2=0x7f0cc00acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc5400640 1 --2- 172.21.15.28:0/764906149 >> v2:172.21.15.189:3300/0 conn(0x7f0cc00ac850 0x7f0cc00acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc5400640 1 -- 172.21.15.28:0/764906149 >> v1:172.21.15.28:6789/0 conn(0x7f0cc0146170 legacy=0x7f0cc015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.572+0000 7f0cc5400640 1 -- 172.21.15.28:0/764906149 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc00ad1c0 con 0x7f0cc0160310 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.573+0000 7f0cc5400640 1 --2- 172.21.15.28:0/764906149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0160310 0x7f0cc0160730 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0cc006d4b0 tx=0x7f0cb002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=179b8c7c6bdaab5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:50.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.573+0000 7f0cc4a00640 1 -- 172.21.15.28:0/764906149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0032440 con 0x7f0cc0160310 2024-09-13T17:12:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.573+0000 7f0cc4a00640 1 -- 172.21.15.28:0/764906149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb00325a0 con 0x7f0cc0160310 2024-09-13T17:12:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.573+0000 7f0cc4a00640 1 -- 172.21.15.28:0/764906149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0032930 con 0x7f0cc0160310 2024-09-13T17:12:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 -- 172.21.15.28:0/764906149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0160310 msgr2=0x7f0cc0160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 --2- 172.21.15.28:0/764906149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0160310 0x7f0cc0160730 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0cc006d4b0 tx=0x7f0cb002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 -- 172.21.15.28:0/764906149 shutdown_connections 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 --2- 172.21.15.28:0/764906149 >> v2:172.21.15.189:3300/0 conn(0x7f0cc00ac850 0x7f0cc00acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 --2- 172.21.15.28:0/764906149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0160310 0x7f0cc0160730 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 -- 172.21.15.28:0/764906149 >> 172.21.15.28:0/764906149 conn(0x7f0cc0069e80 msgr2=0x7f0cc006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 -- 172.21.15.28:0/764906149 shutdown_connections 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.574+0000 7f0cc8800640 1 -- 172.21.15.28:0/764906149 wait complete. 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.575+0000 7f0cc8800640 1 Processor -- start 2024-09-13T17:12:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 -- start start 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc00ac850 0x7f0cc0153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0160310 0x7f0cc015f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc01440c0 con 0x7f0cc0146170 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0142ab0 con 0x7f0cc0160310 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.576+0000 7f0cc8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0142180 con 0x7f0cc00ac850 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52092/0 (socket says 172.21.15.28:52092) 2024-09-13T17:12:50.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 -- 172.21.15.28:0/2928435819 learned_addr learned my addr 172.21.15.28:0/2928435819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5e00640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc00ac850 0x7f0cc0153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc6800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0160310 0x7f0cc015f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 -- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc00ac850 msgr2=0x7f0cc0153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc00ac850 0x7f0cc0153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 -- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0160310 msgr2=0x7f0cc015f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0160310 0x7f0cc015f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.577+0000 7f0cc5400640 1 -- 172.21.15.28:0/2928435819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009400 con 0x7f0cc0146170 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cc5400640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000c00 tx=0x7f0cb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0030b80 con 0x7f0cc0146170 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb0032e90 con 0x7f0cc0146170 2024-09-13T17:12:50.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0006e10 con 0x7f0cc0146170 2024-09-13T17:12:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc0142460 con 0x7f0cc0146170 2024-09-13T17:12:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.578+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc0153660 con 0x7f0cc0146170 2024-09-13T17:12:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.579+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc0142c20 con 0x7f0cc0146170 2024-09-13T17:12:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.580+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0cb0039040 con 0x7f0cc0146170 2024-09-13T17:12:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.581+0000 7f0cba000640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 0x7f0ca406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.581+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cb00a9980 con 0x7f0cc0146170 2024-09-13T17:12:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.581+0000 7f0cc5e00640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 0x7f0ca406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:50.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.582+0000 7f0cc5e00640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 0x7f0ca406d0c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40011e0 tx=0x7f0cb4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:50.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.585+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0cb0074ea0 con 0x7f0cc0146170 2024-09-13T17:12:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.830+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0cc00acc80 con 0x7f0cc0146170 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.831+0000 7f0cba000640 1 -- 172.21.15.28:0/2928435819 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f0cb00784c0 con 0x7f0cc0146170 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:50.834 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.835 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:12:50.836 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.837 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.840 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:12:50.841 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:12:50.842 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:50.843 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:12:50.845 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:12:50.846 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:12:50.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:50.847 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:12:50.847 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 msgr2=0x7f0ca406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 0x7f0ca406d0c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40011e0 tx=0x7f0cb4009040 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 msgr2=0x7f0cc015ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0000c00 tx=0x7f0cb00046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 shutdown_connections 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0ca406ac00 0x7f0ca406d0c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0160310 0x7f0cc015f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0146170 0x7f0cc015ef10 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 --2- 172.21.15.28:0/2928435819 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc00ac850 0x7f0cc0153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 >> 172.21.15.28:0/2928435819 conn(0x7f0cc0069e80 msgr2=0x7f0cc015bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 shutdown_connections 2024-09-13T17:12:50.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:50.833+0000 7f0cc8800640 1 -- 172.21.15.28:0/2928435819 wait complete. 2024-09-13T17:12:50.851 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:51.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.005+0000 7fb3bc200640 1 Processor -- start 2024-09-13T17:12:51.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.005+0000 7fb3bc200640 1 -- start start 2024-09-13T17:12:51.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.006+0000 7fb3bc200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 0x7fb3b406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.006+0000 7fb3bc200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb3b40cabc0 0x7fb3b415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.006+0000 7fb3bc200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b415e100 con 0x7fb3b415edb0 2024-09-13T17:12:51.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.006+0000 7fb3bc200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b40adb00 con 0x7fb3b40cabc0 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.006+0000 7fb3bc200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415e3e0 con 0x7fb3b415a540 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3ba200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb3b415edb0 0x7fb3b4070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59822/0 (socket says 172.21.15.28:59822) 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3ba200640 1 -- 172.21.15.28:0/847041395 learned_addr learned my addr 172.21.15.28:0/847041395 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b9800640 1 --2- 172.21.15.28:0/847041395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 0x7fb3b406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b8e00640 1 --2- 172.21.15.28:0/847041395 >> v2:172.21.15.189:3300/0 conn(0x7fb3b40cabc0 0x7fb3b415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b3e00640 1 -- 172.21.15.28:0/847041395 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311206158 0 0) 0x7fb3b415e100 con 0x7fb3b415edb0 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b3e00640 1 -- 172.21.15.28:0/847041395 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb39c003520 con 0x7fb3b415edb0 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b9800640 1 -- 172.21.15.28:0/847041395 >> v2:172.21.15.189:3300/0 conn(0x7fb3b40cabc0 msgr2=0x7fb3b415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b9800640 1 --2- 172.21.15.28:0/847041395 >> v2:172.21.15.189:3300/0 conn(0x7fb3b40cabc0 0x7fb3b415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b9800640 1 -- 172.21.15.28:0/847041395 >> v1:172.21.15.28:6789/0 conn(0x7fb3b415edb0 legacy=0x7fb3b4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.007+0000 7fb3b9800640 1 -- 172.21.15.28:0/847041395 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b406ce80 con 0x7fb3b415a540 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3b9800640 1 --2- 172.21.15.28:0/847041395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 0x7fb3b406c430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80097a0 tx=0x7fb3a802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=638983e3a43f389b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3b3e00640 1 -- 172.21.15.28:0/847041395 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802ae50 con 0x7fb3b415a540 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3b3e00640 1 -- 172.21.15.28:0/847041395 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a8032440 con 0x7fb3b415a540 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3b3e00640 1 -- 172.21.15.28:0/847041395 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80327d0 con 0x7fb3b415a540 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3bc200640 1 -- 172.21.15.28:0/847041395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 msgr2=0x7fb3b406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.009+0000 7fb3bc200640 1 --2- 172.21.15.28:0/847041395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 0x7fb3b406c430 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80097a0 tx=0x7fb3a802a100 comp rx=0 tx=0).stop 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 -- 172.21.15.28:0/847041395 shutdown_connections 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 --2- 172.21.15.28:0/847041395 >> v2:172.21.15.189:3300/0 conn(0x7fb3b40cabc0 0x7fb3b415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 --2- 172.21.15.28:0/847041395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b415a540 0x7fb3b406c430 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 -- 172.21.15.28:0/847041395 >> 172.21.15.28:0/847041395 conn(0x7fb3b4069e80 msgr2=0x7fb3b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 -- 172.21.15.28:0/847041395 shutdown_connections 2024-09-13T17:12:51.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.010+0000 7fb3bc200640 1 -- 172.21.15.28:0/847041395 wait complete. 2024-09-13T17:12:51.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.011+0000 7fb3bc200640 1 Processor -- start 2024-09-13T17:12:51.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.011+0000 7fb3bc200640 1 -- start start 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 0x7fb3b415b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3b415a540 0x7fb3b414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415c910 con 0x7fb3b40b0180 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415b9c0 con 0x7fb3b415a540 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.012+0000 7fb3bc200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b415bc90 con 0x7fb3b40cabc0 2024-09-13T17:12:51.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52108/0 (socket says 172.21.15.28:52108) 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3b9800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 0x7fb3b415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 -- 172.21.15.28:0/2965798420 learned_addr learned my addr 172.21.15.28:0/2965798420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3b8e00640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3b415a540 0x7fb3b414cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 -- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 msgr2=0x7fb3b415b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 0x7fb3b415b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 -- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3b415a540 msgr2=0x7fb3b414cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3b415a540 0x7fb3b414cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3ba200640 1 -- 172.21.15.28:0/2965798420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8009400 con 0x7fb3b40b0180 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.013+0000 7fb3b9800640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 0x7fb3b415b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3ba200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00caa0 tx=0x7fb3ac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac00ac00 con 0x7fb3b40b0180 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ac00ad60 con 0x7fb3b40b0180 2024-09-13T17:12:51.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b415bf70 con 0x7fb3b40b0180 2024-09-13T17:12:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ac012620 con 0x7fb3b40b0180 2024-09-13T17:12:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.014+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b415c530 con 0x7fb3b40b0180 2024-09-13T17:12:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.015+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b415e850 con 0x7fb3b40b0180 2024-09-13T17:12:51.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.016+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb3ac007070 con 0x7fb3b40b0180 2024-09-13T17:12:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.017+0000 7fb3b1600640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 0x7fb39406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.017+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb3ac088f40 con 0x7fb3b40b0180 2024-09-13T17:12:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.017+0000 7fb3b9800640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 0x7fb39406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.018+0000 7fb3b9800640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 0x7fb39406d0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8034000 tx=0x7fb3a802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.021+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb3ac054510 con 0x7fb3b40b0180 2024-09-13T17:12:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.204+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3b40cc940 con 0x7fb3b40b0180 2024-09-13T17:12:51.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.205+0000 7fb3b1600640 1 -- 172.21.15.28:0/2965798420 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+12857 (secure 0 0 0) 0x7fb3ac057b30 con 0x7fb3b40b0180 2024-09-13T17:12:51.207 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:51.207 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":73,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:50.382981+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 msgr2=0x7fb39406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 0x7fb39406d0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8034000 tx=0x7fb3a802e000 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 msgr2=0x7fb3b4153a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac00caa0 tx=0x7fb3ac00cf60 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 shutdown_connections 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb39406ac00 0x7fb39406d0c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb3b40b0180 0x7fb3b4153a70 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3b415a540 0x7fb3b414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 --2- 172.21.15.28:0/2965798420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb3b40cabc0 0x7fb3b415b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 >> 172.21.15.28:0/2965798420 conn(0x7fb3b4069e80 msgr2=0x7fb3b406c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 shutdown_connections 2024-09-13T17:12:51.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.207+0000 7fb3bc200640 1 -- 172.21.15.28:0/2965798420 wait complete. 2024-09-13T17:12:51.225 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:12:51.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:12:51.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.382+0000 7f8e53200640 1 Processor -- start 2024-09-13T17:12:51.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.382+0000 7f8e53200640 1 -- start start 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.383+0000 7f8e53200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c159770 0x7f8e4c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.383+0000 7f8e53200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4c15cd80 0x7f8e4c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.383+0000 7f8e53200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c06f920 con 0x7f8e4c06f3f0 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.383+0000 7f8e53200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e4c15b290 con 0x7f8e4c15cd80 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.383+0000 7f8e53200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c15b570 con 0x7f8e4c159770 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4c15cd80 0x7f8e4c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e4c15cd80 0x7f8e4c070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52118/0 (socket says 172.21.15.28:52118) 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 -- 172.21.15.28:0/4232176411 learned_addr learned my addr 172.21.15.28:0/4232176411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e4be00640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c159770 0x7f8e4c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 -- 172.21.15.28:0/4232176411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c159770 msgr2=0x7f8e4c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c159770 0x7f8e4c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 -- 172.21.15.28:0/4232176411 >> v1:172.21.15.189:6789/0 conn(0x7f8e4c06f3f0 legacy=0x7f8e4c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.384+0000 7f8e51200640 1 -- 172.21.15.28:0/4232176411 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c15b850 con 0x7f8e4c15cd80 2024-09-13T17:12:51.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.385+0000 7f8e51200640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c070260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e50 tx=0x7f8e4402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9242badb6ceab785 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.386+0000 7f8e4aa00640 1 -- 172.21.15.28:0/4232176411 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4402aea0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.386+0000 7f8e4aa00640 1 -- 172.21.15.28:0/4232176411 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4402d050 con 0x7f8e4c15cd80 2024-09-13T17:12:51.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.386+0000 7f8e4aa00640 1 -- 172.21.15.28:0/4232176411 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e44031820 con 0x7f8e4c15cd80 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 -- 172.21.15.28:0/4232176411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 msgr2=0x7f8e4c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c070260 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e50 tx=0x7f8e4402a680 comp rx=0 tx=0).stop 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 -- 172.21.15.28:0/4232176411 shutdown_connections 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c070260 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 --2- 172.21.15.28:0/4232176411 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c159770 0x7f8e4c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 -- 172.21.15.28:0/4232176411 >> 172.21.15.28:0/4232176411 conn(0x7f8e4c0b0270 msgr2=0x7f8e4c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 -- 172.21.15.28:0/4232176411 shutdown_connections 2024-09-13T17:12:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.387+0000 7f8e53200640 1 -- 172.21.15.28:0/4232176411 wait complete. 2024-09-13T17:12:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.389+0000 7f8e53200640 1 Processor -- start 2024-09-13T17:12:51.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.389+0000 7f8e53200640 1 -- start start 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c06f3f0 0x7f8e4c150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4c159770 0x7f8e4c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c1514c0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c155160 con 0x7f8e4c159770 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e53200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4c155440 con 0x7f8e4c06f3f0 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52124/0 (socket says 172.21.15.28:52124) 2024-09-13T17:12:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 -- 172.21.15.28:0/1283020717 learned_addr learned my addr 172.21.15.28:0/1283020717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e4be00640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c06f3f0 0x7f8e4c150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e4b400640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4c159770 0x7f8e4c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 -- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c06f3f0 msgr2=0x7f8e4c150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c06f3f0 0x7f8e4c150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 -- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4c159770 msgr2=0x7f8e4c1547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4c159770 0x7f8e4c1547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.390+0000 7f8e51200640 1 -- 172.21.15.28:0/1283020717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e44009b30 con 0x7f8e4c15cd80 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e51200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e50 tx=0x7f8e44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4403b820 con 0x7f8e4c15cd80 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4403b9a0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4403bd50 con 0x7f8e4c15cd80 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c155720 con 0x7f8e4c15cd80 2024-09-13T17:12:51.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.391+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c155ce0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.392+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c155e70 con 0x7f8e4c15cd80 2024-09-13T17:12:51.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.393+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e44038050 con 0x7f8e4c15cd80 2024-09-13T17:12:51.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.394+0000 7f8e43e00640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 0x7f8e2c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.394+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e440a99f0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.395+0000 7f8e4be00640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 0x7f8e2c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.395+0000 7f8e4be00640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 0x7f8e2c06d0c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.398+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e44074e80 con 0x7f8e4c15cd80 2024-09-13T17:12:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.573+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e4c06fd20 con 0x7f8e4c15cd80 2024-09-13T17:12:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.574+0000 7f8e43e00640 1 -- 172.21.15.28:0/1283020717 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e4404b0a0 con 0x7f8e4c15cd80 2024-09-13T17:12:51.576 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:12:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 msgr2=0x7f8e2c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 0x7f8e2c06d0c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c006fd0 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2024-09-13T17:12:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 msgr2=0x7f8e4c0cc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e50 tx=0x7f8e44004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 shutdown_connections 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2c06ac00 0x7f8e2c06d0c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e4c15cd80 0x7f8e4c0cc640 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4c159770 0x7f8e4c1547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 --2- 172.21.15.28:0/1283020717 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4c06f3f0 0x7f8e4c150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 >> 172.21.15.28:0/1283020717 conn(0x7f8e4c0b0270 msgr2=0x7f8e4c158780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 shutdown_connections 2024-09-13T17:12:51.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.576+0000 7f8e53200640 1 -- 172.21.15.28:0/1283020717 wait complete. 2024-09-13T17:12:51.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:12:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.746+0000 7f91a7e00640 1 Processor -- start 2024-09-13T17:12:51.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.747+0000 7f91a7e00640 1 -- start start 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f91a8159410 0x7f91a806f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a80b38d0 0x7f91a806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91a80ccb00 con 0x7f91a806d240 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91a806d6f0 con 0x7f91a8159410 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a806fc50 con 0x7f91a80b38d0 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91a806d240 0x7f91a80b15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59828/0 (socket says 172.21.15.28:59828) 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a7400640 1 -- 172.21.15.28:0/1851755000 learned_addr learned my addr 172.21.15.28:0/1851755000 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:51.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f91a6a00640 1 --2- 172.21.15.28:0/1851755000 >> v2:172.21.15.189:3300/0 conn(0x7f91a8159410 0x7f91a806f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.748+0000 7f919e000640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a80b38d0 0x7f91a806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968385159 0 0) 0x7f91a80ccb00 con 0x7f91a806d240 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c003520 con 0x7f91a806d240 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6a00640 1 -- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a80b38d0 msgr2=0x7f91a806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6a00640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a80b38d0 0x7f91a806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6a00640 1 -- 172.21.15.28:0/1851755000 >> v1:172.21.15.28:6789/0 conn(0x7f91a806d240 legacy=0x7f91a80b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1432075253 0 0) 0x7f918c003520 con 0x7f91a806d240 2024-09-13T17:12:51.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.749+0000 7f91a6a00640 1 -- 172.21.15.28:0/1851755000 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a806ff30 con 0x7f91a8159410 2024-09-13T17:12:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.750+0000 7f91a6a00640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a8159410 0x7f91a806f300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91980097a0 tx=0x7f919802a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9fa810192706e9a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91a00019f0 con 0x7f91a8159410 2024-09-13T17:12:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919802aba0 con 0x7f91a8159410 2024-09-13T17:12:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a6000640 1 -- 172.21.15.28:0/1851755000 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9198032440 con 0x7f91a8159410 2024-09-13T17:12:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 -- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a8159410 msgr2=0x7f91a806f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a8159410 0x7f91a806f300 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f91980097a0 tx=0x7f919802a110 comp rx=0 tx=0).stop 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 -- 172.21.15.28:0/1851755000 shutdown_connections 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a80b38d0 0x7f91a806cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 --2- 172.21.15.28:0/1851755000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a8159410 0x7f91a806f300 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 -- 172.21.15.28:0/1851755000 >> 172.21.15.28:0/1851755000 conn(0x7f91a8073670 msgr2=0x7f91a8069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 -- 172.21.15.28:0/1851755000 shutdown_connections 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.751+0000 7f91a7e00640 1 -- 172.21.15.28:0/1851755000 wait complete. 2024-09-13T17:12:51.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.753+0000 7f91a7e00640 1 Processor -- start 2024-09-13T17:12:51.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.753+0000 7f91a7e00640 1 -- start start 2024-09-13T17:12:51.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80b38d0 0x7f91a814fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a8159410 0x7f91a8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a815bb30 con 0x7f91a806d240 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a814fa60 con 0x7f91a80b38d0 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a815c5e0 con 0x7f91a8159410 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52136/0 (socket says 172.21.15.28:52136) 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 -- 172.21.15.28:0/4010708801 learned_addr learned my addr 172.21.15.28:0/4010708801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f919e000640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80b38d0 0x7f91a814fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a7400640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a8159410 0x7f91a8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 -- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a8159410 msgr2=0x7f91a8155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a8159410 0x7f91a8155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 -- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80b38d0 msgr2=0x7f91a814fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.754+0000 7f91a6a00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80b38d0 0x7f91a814fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f91a6a00640 1 -- 172.21.15.28:0/4010708801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9198009400 con 0x7f91a806d240 2024-09-13T17:12:51.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f91a6a00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9198035040 tx=0x7f91980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919803b810 con 0x7f91a806d240 2024-09-13T17:12:51.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f919803b970 con 0x7f91a806d240 2024-09-13T17:12:51.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a815bd40 con 0x7f91a806d240 2024-09-13T17:12:51.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f919803bd00 con 0x7f91a806d240 2024-09-13T17:12:51.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.755+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a8153e90 con 0x7f91a806d240 2024-09-13T17:12:51.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.756+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a806cd00 con 0x7f91a806d240 2024-09-13T17:12:51.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.757+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9198039040 con 0x7f91a806d240 2024-09-13T17:12:51.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.758+0000 7f919f400640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 0x7f918406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:51.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.758+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f91980a9530 con 0x7f91a806d240 2024-09-13T17:12:51.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.759+0000 7f919e000640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 0x7f918406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:51.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.759+0000 7f919e000640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 0x7f918406d0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f91a80afe70 tx=0x7f9190008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:51.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.762+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9198074a50 con 0x7f91a806d240 2024-09-13T17:12:51.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.941+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f91a80afbc0 con 0x7f91a806d240 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.942+0000 7f919f400640 1 -- 172.21.15.28:0/4010708801 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v73) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9198037090 con 0x7f91a806d240 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:12:51.944 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 msgr2=0x7f918406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 0x7f918406d0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f91a80afe70 tx=0x7f9190008040 comp rx=0 tx=0).stop 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 msgr2=0x7f91a815b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f9198035040 tx=0x7f91980046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 shutdown_connections 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f918406ac00 0x7f918406d0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a8159410 0x7f91a8155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a80b38d0 0x7f91a814fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 --2- 172.21.15.28:0/4010708801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a806d240 0x7f91a815b5f0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 >> 172.21.15.28:0/4010708801 conn(0x7f91a8073670 msgr2=0x7f91a80aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 shutdown_connections 2024-09-13T17:12:51.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:51.944+0000 7f91a7e00640 1 -- 172.21.15.28:0/4010708801 wait complete. 2024-09-13T17:12:51.962 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:12:51.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:12:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.120+0000 7f53b6400640 1 Processor -- start 2024-09-13T17:12:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.120+0000 7f53b6400640 1 -- start start 2024-09-13T17:12:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53b6400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53b0159970 0x7f53b00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53b6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b015a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53b6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b015a4c0 con 0x7f53b0159590 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53b6400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b00b2ae0 con 0x7f53b0159970 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53b6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0071390 con 0x7f53b00b2630 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53af400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f53b0159590 0x7f53b0070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59844/0 (socket says 172.21.15.28:59844) 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53af400640 1 -- 172.21.15.28:0/3260493878 learned_addr learned my addr 172.21.15.28:0/3260493878 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.121+0000 7f53aea00640 1 --2- 172.21.15.28:0/3260493878 >> v2:172.21.15.28:3300/0 conn(0x7f53b0159970 0x7f53b00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.122+0000 7f53afe00640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b015a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.122+0000 7f53aea00640 1 -- 172.21.15.28:0/3260493878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 msgr2=0x7f53b015a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.122+0000 7f53aea00640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b015a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.122+0000 7f53aea00640 1 -- 172.21.15.28:0/3260493878 >> v1:172.21.15.28:6789/0 conn(0x7f53b0159590 legacy=0x7f53b0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.122+0000 7f53aea00640 1 -- 172.21.15.28:0/3260493878 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b0071580 con 0x7f53b0159970 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.123+0000 7f53aea00640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159970 0x7f53b00b20f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f53980097d0 tx=0x7f539802c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebab512eca37eda server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.123+0000 7f53ae000640 1 -- 172.21.15.28:0/3260493878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398034640 con 0x7f53b0159970 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.123+0000 7f53ae000640 1 -- 172.21.15.28:0/3260493878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53980347a0 con 0x7f53b0159970 2024-09-13T17:12:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.123+0000 7f53ae000640 1 -- 172.21.15.28:0/3260493878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398034b30 con 0x7f53b0159970 2024-09-13T17:12:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 -- 172.21.15.28:0/3260493878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159970 msgr2=0x7f53b00b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159970 0x7f53b00b20f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f53980097d0 tx=0x7f539802c3d0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 -- 172.21.15.28:0/3260493878 shutdown_connections 2024-09-13T17:12:52.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b015a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 --2- 172.21.15.28:0/3260493878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159970 0x7f53b00b20f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 -- 172.21.15.28:0/3260493878 >> 172.21.15.28:0/3260493878 conn(0x7f53b0069e80 msgr2=0x7f53b006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 -- 172.21.15.28:0/3260493878 shutdown_connections 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.124+0000 7f53b6400640 1 -- 172.21.15.28:0/3260493878 wait complete. 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.126+0000 7f53b6400640 1 Processor -- start 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.126+0000 7f53b6400640 1 -- start start 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b0145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53b0159970 0x7f53b0157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01ef2a0 con 0x7f53b0159590 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b0146010 con 0x7f53b0159970 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53b6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b01ee900 con 0x7f53b00b2630 2024-09-13T17:12:52.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52152/0 (socket says 172.21.15.28:52152) 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 -- 172.21.15.28:0/239311732 learned_addr learned my addr 172.21.15.28:0/239311732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53af400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b0145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53afe00640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53b0159970 0x7f53b0157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 -- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 msgr2=0x7f53b0145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b0145690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 -- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53b0159970 msgr2=0x7f53b0157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53b0159970 0x7f53b0157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.127+0000 7f53aea00640 1 -- 172.21.15.28:0/239311732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5398009400 con 0x7f53b0159590 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53aea00640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f53980097d0 tx=0x7f53980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398032850 con 0x7f53b0159590 2024-09-13T17:12:52.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53980329d0 con 0x7f53b0159590 2024-09-13T17:12:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398032d80 con 0x7f53b0159590 2024-09-13T17:12:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b01eebe0 con 0x7f53b0159590 2024-09-13T17:12:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.128+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b01ee5b0 con 0x7f53b0159590 2024-09-13T17:12:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.129+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b01ef3e0 con 0x7f53b0159590 2024-09-13T17:12:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.130+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f539803b040 con 0x7f53b0159590 2024-09-13T17:12:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.131+0000 7f53a7400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 0x7f538c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.131+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f53980ab9c0 con 0x7f53b0159590 2024-09-13T17:12:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.132+0000 7f53af400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 0x7f538c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:52.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.132+0000 7f53af400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 0x7f538c06d0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005880 tx=0x7f53a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.135+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f53b01ef3e0 con 0x7f53b0159590 2024-09-13T17:12:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:52.312+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f53b01ef3e0 con 0x7f53b0159590 2024-09-13T17:12:53.033 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-13T17:12:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.093+0000 7f53a7400640 1 -- 172.21.15.28:0/239311732 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v74) v1 ==== 138+0+0 (secure 0 0 0) 0x7f53b01ef3e0 con 0x7f53b0159590 2024-09-13T17:12:53.096 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 msgr2=0x7f538c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 0x7f538c06d0c0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f53a0005880 tx=0x7f53a0009040 comp rx=0 tx=0).stop 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 msgr2=0x7f53b0157860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f53980097d0 tx=0x7f53980046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 shutdown_connections 2024-09-13T17:12:53.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f538c06ac00 0x7f538c06d0c0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53b0159970 0x7f53b0157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53b0159590 0x7f53b0157860 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 --2- 172.21.15.28:0/239311732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53b00b2630 0x7f53b0145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.098+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 >> 172.21.15.28:0/239311732 conn(0x7f53b0069e80 msgr2=0x7f53b006c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.099+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 shutdown_connections 2024-09-13T17:12:53.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.099+0000 7f53b6400640 1 -- 172.21.15.28:0/239311732 wait complete. 2024-09-13T17:12:53.120 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:12:53.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.283+0000 7fc6f2000640 1 Processor -- start 2024-09-13T17:12:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.283+0000 7fc6f2000640 1 -- start start 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6f2000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0b0b50 0x7fc6ec0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6f2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec06b900 con 0x7fc6ec06b360 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6f2000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ec06bbe0 con 0x7fc6ec154a90 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6f2000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ec06bec0 con 0x7fc6ec0b0b50 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6ebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc6ec06b360 0x7fc6ec0b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59852/0 (socket says 172.21.15.28:59852) 2024-09-13T17:12:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6ebe00640 1 -- 172.21.15.28:0/2418380981 learned_addr learned my addr 172.21.15.28:0/2418380981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6f1600640 1 --2- 172.21.15.28:0/2418380981 >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0b0b50 0x7fc6ec0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247097138 0 0) 0x7fc6ec06b900 con 0x7fc6ec06b360 2024-09-13T17:12:53.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.285+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d4003520 con 0x7fc6ec06b360 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 185832001 0 0) 0x7fc6d4003520 con 0x7fc6ec06b360 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0b0b50 msgr2=0x7fc6ec0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 --2- 172.21.15.28:0/2418380981 >> v2:172.21.15.189:3300/0 conn(0x7fc6ec0b0b50 0x7fc6ec0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 >> v1:172.21.15.189:6789/0 conn(0x7fc6ec154a90 legacy=0x7fc6ec15b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec06c1a0 con 0x7fc6ec06b360 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc6d8001670 con 0x7fc6ec06b360 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 >> v1:172.21.15.28:6789/0 conn(0x7fc6ec06b360 legacy=0x7fc6ec0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6d4003970 0x7fc6d4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d4005180 0x7fc6d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec06bec0 con 0x7fc6d40047c0 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d400c020 con 0x7fc6d4005180 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d400c1e0 con 0x7fc6d4003970 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6f0c00640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6d4003970 0x7fc6d4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.286+0000 7fc6f1600640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d4005180 0x7fc6d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6ebe00640 1 -- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6d4003970 msgr2=0x7fc6d4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:53.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6d4003970 0x7fc6d4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6ebe00640 1 -- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d4005180 msgr2=0x7fc6d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6d4005180 0x7fc6d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6ebe00640 1 -- 172.21.15.28:0/2418380981 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d4003520 con 0x7fc6d40047c0 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.287+0000 7fc6f1600640 1 -- 172.21.15.28:0/2418380981 reap_dead start 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6d4004c40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c190 tx=0x7fc6d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846327c502055fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8005ad0 con 0x7fc6d40047c0 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d802d070 con 0x7fc6d40047c0 2024-09-13T17:12:53.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6eb400640 1 -- 172.21.15.28:0/2418380981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8007070 con 0x7fc6d40047c0 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6f2000640 1 -- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 msgr2=0x7fc6d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.288+0000 7fc6f2000640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6d4004c40 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc6d800c190 tx=0x7fc6d40057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.289+0000 7fc6f2000640 1 -- 172.21.15.28:0/2418380981 shutdown_connections 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.289+0000 7fc6f2000640 1 --2- 172.21.15.28:0/2418380981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6d4004c40 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.289+0000 7fc6f2000640 1 -- 172.21.15.28:0/2418380981 >> 172.21.15.28:0/2418380981 conn(0x7fc6ec072e40 msgr2=0x7fc6ec073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.289+0000 7fc6f2000640 1 -- 172.21.15.28:0/2418380981 shutdown_connections 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.289+0000 7fc6f2000640 1 -- 172.21.15.28:0/2418380981 wait complete. 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.290+0000 7fc6f2000640 1 Processor -- start 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.290+0000 7fc6f2000640 1 -- start start 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6ec1f6770 0x7fc6ec0cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 0x7fc6ec1f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0b19e0 con 0x7fc6d40047c0 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0cc8e0 con 0x7fc6ec0cc430 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.291+0000 7fc6f2000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ec0af550 con 0x7fc6ec1f6770 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52168/0 (socket says 172.21.15.28:52168) 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 -- 172.21.15.28:0/58383619 learned_addr learned my addr 172.21.15.28:0/58383619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6ec1f6770 0x7fc6ec0cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f1600640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 0x7fc6ec1f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 -- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6ec1f6770 msgr2=0x7fc6ec0cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6ec1f6770 0x7fc6ec0cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 -- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 msgr2=0x7fc6ec1f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 0x7fc6ec1f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 -- 172.21.15.28:0/58383619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec155080 con 0x7fc6d40047c0 2024-09-13T17:12:53.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f1600640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 0x7fc6ec1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.292+0000 7fc6f0c00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000f620 tx=0x7fc6e000fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.293+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00052e0 con 0x7fc6d40047c0 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.293+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0005440 con 0x7fc6d40047c0 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.293+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0003810 con 0x7fc6d40047c0 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.293+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ec0c58e0 con 0x7fc6d40047c0 2024-09-13T17:12:53.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.293+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ec0c5ea0 con 0x7fc6d40047c0 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.294+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ec06b900 con 0x7fc6d40047c0 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.295+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6e0003970 con 0x7fc6d40047c0 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.296+0000 7fc6e8c00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 0x7fc6cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.296+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc6e0089010 con 0x7fc6d40047c0 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.296+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 0x7fc6cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:53.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.297+0000 7fc6ebe00640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 0x7fc6cc06d0f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8032000 tx=0x7fc6d8009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:53.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.300+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6e0054330 con 0x7fc6d40047c0 2024-09-13T17:12:53.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:53.482+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6ec158a20 con 0x7fc6d40047c0 2024-09-13T17:12:54.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.102+0000 7fc6e8c00640 1 -- 172.21.15.28:0/58383619 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v75) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc6e0057950 con 0x7fc6d40047c0 2024-09-13T17:12:54.104 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:12:54.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.110+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 msgr2=0x7fc6cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.110+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 0x7fc6cc06d0f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8032000 tx=0x7fc6d8009030 comp rx=0 tx=0).stop 2024-09-13T17:12:54.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.110+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 msgr2=0x7fc6ec13c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.110+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000f620 tx=0x7fc6e000fae0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 shutdown_connections 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6cc06ac30 0x7fc6cc06d0f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6ec0cc430 0x7fc6ec1f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6ec1f6770 0x7fc6ec0cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 --2- 172.21.15.28:0/58383619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6d40047c0 0x7fc6ec13c8e0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.111+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 >> 172.21.15.28:0/58383619 conn(0x7fc6ec072e40 msgr2=0x7fc6ec155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.112+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 shutdown_connections 2024-09-13T17:12:54.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.112+0000 7fc6f2000640 1 -- 172.21.15.28:0/58383619 wait complete. 2024-09-13T17:12:54.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:12:54.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.299+0000 7f1d9d200640 1 Processor -- start 2024-09-13T17:12:54.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.300+0000 7f1d9d200640 1 -- start start 2024-09-13T17:12:54.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d9d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1d98070430 0x7f1d9815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d9d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d9d200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d9815ab60 con 0x7f1d98159770 2024-09-13T17:12:54.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d9d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9806f920 con 0x7f1d98070430 2024-09-13T17:12:54.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d9d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9815b290 con 0x7f1d9806f3f0 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d96800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d95400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1d98070430 0x7f1d9815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d96800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47006/0 (socket says 172.21.15.28:47006) 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.301+0000 7f1d96800640 1 -- 172.21.15.28:0/3309822747 learned_addr learned my addr 172.21.15.28:0/3309822747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.302+0000 7f1d96800640 1 -- 172.21.15.28:0/3309822747 >> v2:172.21.15.189:3300/0 conn(0x7f1d98070430 msgr2=0x7f1d9815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.302+0000 7f1d96800640 1 --2- 172.21.15.28:0/3309822747 >> v2:172.21.15.189:3300/0 conn(0x7f1d98070430 0x7f1d9815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.302+0000 7f1d96800640 1 -- 172.21.15.28:0/3309822747 >> v1:172.21.15.189:6789/0 conn(0x7f1d98159770 legacy=0x7f1d9806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.302+0000 7f1d96800640 1 -- 172.21.15.28:0/3309822747 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9815b570 con 0x7f1d9806f3f0 2024-09-13T17:12:54.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.303+0000 7f1d96800640 1 --2- 172.21.15.28:0/3309822747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009e50 tx=0x7f1d9002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=298f5227da78e57e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.303+0000 7f1d94a00640 1 -- 172.21.15.28:0/3309822747 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d9002aea0 con 0x7f1d9806f3f0 2024-09-13T17:12:54.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.303+0000 7f1d94a00640 1 -- 172.21.15.28:0/3309822747 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d9002d050 con 0x7f1d9806f3f0 2024-09-13T17:12:54.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.303+0000 7f1d94a00640 1 -- 172.21.15.28:0/3309822747 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d90031820 con 0x7f1d9806f3f0 2024-09-13T17:12:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 -- 172.21.15.28:0/3309822747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 msgr2=0x7f1d9815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 --2- 172.21.15.28:0/3309822747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009e50 tx=0x7f1d9002a680 comp rx=0 tx=0).stop 2024-09-13T17:12:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 -- 172.21.15.28:0/3309822747 shutdown_connections 2024-09-13T17:12:54.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 --2- 172.21.15.28:0/3309822747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d9806f3f0 0x7f1d9815c2b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 --2- 172.21.15.28:0/3309822747 >> v2:172.21.15.189:3300/0 conn(0x7f1d98070430 0x7f1d9815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 -- 172.21.15.28:0/3309822747 >> 172.21.15.28:0/3309822747 conn(0x7f1d980b0270 msgr2=0x7f1d980b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 -- 172.21.15.28:0/3309822747 shutdown_connections 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.304+0000 7f1d9d200640 1 -- 172.21.15.28:0/3309822747 wait complete. 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.306+0000 7f1d9d200640 1 Processor -- start 2024-09-13T17:12:54.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.306+0000 7f1d9d200640 1 -- start start 2024-09-13T17:12:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d98159770 0x7f1d980af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d98153790 con 0x7f1d98159770 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980afb80 con 0x7f1d9806f3f0 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d9d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d980aa640 con 0x7f1d98070430 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d95400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d95e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d96800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d98159770 0x7f1d980af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d95e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58930/0 (socket says 172.21.15.28:58930) 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d95400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47014/0 (socket says 172.21.15.28:47014) 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.307+0000 7f1d95e00640 1 -- 172.21.15.28:0/1335204437 learned_addr learned my addr 172.21.15.28:0/1335204437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 -- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 msgr2=0x7f1d98153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 -- 172.21.15.28:0/1335204437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d98159770 msgr2=0x7f1d980af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d98159770 0x7f1d980af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 -- 172.21.15.28:0/1335204437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d90009b30 con 0x7f1d9806f3f0 2024-09-13T17:12:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95400640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d95e00640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800c3c0 tx=0x7f1d8800c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d880052d0 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d88005450 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d980aa920 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d88007260 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.308+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d980aaee0 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.309+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d980676b0 con 0x7f1d9806f3f0 2024-09-13T17:12:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.311+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1d880073c0 con 0x7f1d9806f3f0 2024-09-13T17:12:54.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.311+0000 7f1d8e000640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 0x7f1d7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.312+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f1d88088910 con 0x7f1d9806f3f0 2024-09-13T17:12:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.312+0000 7f1d95400640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 0x7f1d7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.313+0000 7f1d95400640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 0x7f1d7806d0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d980ad5d0 tx=0x7f1d84009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.315+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1d88053c70 con 0x7f1d9806f3f0 2024-09-13T17:12:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.531+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1d9815ab60 con 0x7f1d9806f3f0 2024-09-13T17:12:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.540+0000 7f1d8e000640 1 -- 172.21.15.28:0/1335204437 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v60) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1d9815ab60 con 0x7f1d9806f3f0 2024-09-13T17:12:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 10 and data pool 11 2024-09-13T17:12:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 msgr2=0x7f1d7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 0x7f1d7806d0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1d980ad5d0 tx=0x7f1d84009190 comp rx=0 tx=0).stop 2024-09-13T17:12:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 msgr2=0x7f1d98153e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800c3c0 tx=0x7f1d8800c880 comp rx=0 tx=0).stop 2024-09-13T17:12:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 shutdown_connections 2024-09-13T17:12:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1d7806ac00 0x7f1d7806d0c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d98159770 0x7f1d980af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d98070430 0x7f1d98153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 --2- 172.21.15.28:0/1335204437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d9806f3f0 0x7f1d98153e80 secure :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1d8800c3c0 tx=0x7f1d8800c880 comp rx=0 tx=0).stop 2024-09-13T17:12:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.544+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 >> 172.21.15.28:0/1335204437 conn(0x7f1d980b0270 msgr2=0x7f1d980ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.547+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 shutdown_connections 2024-09-13T17:12:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.547+0000 7f1d9d200640 1 -- 172.21.15.28:0/1335204437 wait complete. 2024-09-13T17:12:54.576 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:54.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.730+0000 7f330f200640 1 Processor -- start 2024-09-13T17:12:54.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.730+0000 7f330f200640 1 -- start start 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330815a540 0x7f330806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3308070cc0 0x7f3308071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330815e100 con 0x7f33080cabc0 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33080716b0 con 0x7f3308070cc0 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33080adb00 con 0x7f330815a540 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3308070cc0 0x7f3308071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3308070cc0 0x7f3308071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57164/0 (socket says 172.21.15.28:57164) 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 -- 172.21.15.28:0/2732069532 learned_addr learned my addr 172.21.15.28:0/2732069532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:54.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f3307e00640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330815a540 0x7f330806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 -- 172.21.15.28:0/2732069532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330815a540 msgr2=0x7f330806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330815a540 0x7f330806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 -- 172.21.15.28:0/2732069532 >> v1:172.21.15.189:6789/0 conn(0x7f33080cabc0 legacy=0x7f330815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.732+0000 7f330d200640 1 -- 172.21.15.28:0/2732069532 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330815e3e0 con 0x7f3308070cc0 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.733+0000 7f330d200640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3308070cc0 0x7f3308071170 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3300009e80 tx=0x7f330002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2467fd65bcbad87b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f3307400640 1 -- 172.21.15.28:0/2732069532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300031600 con 0x7f3308070cc0 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f3307400640 1 -- 172.21.15.28:0/2732069532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f330002d050 con 0x7f3308070cc0 2024-09-13T17:12:54.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f3307400640 1 -- 172.21.15.28:0/2732069532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300031980 con 0x7f3308070cc0 2024-09-13T17:12:54.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 -- 172.21.15.28:0/2732069532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3308070cc0 msgr2=0x7f3308071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3308070cc0 0x7f3308071170 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f3300009e80 tx=0x7f330002a660 comp rx=0 tx=0).stop 2024-09-13T17:12:54.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 -- 172.21.15.28:0/2732069532 shutdown_connections 2024-09-13T17:12:54.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3308070cc0 0x7f3308071170 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 --2- 172.21.15.28:0/2732069532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f330815a540 0x7f330806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 -- 172.21.15.28:0/2732069532 >> 172.21.15.28:0/2732069532 conn(0x7f3308069e80 msgr2=0x7f330806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 -- 172.21.15.28:0/2732069532 shutdown_connections 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.734+0000 7f330f200640 1 -- 172.21.15.28:0/2732069532 wait complete. 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.736+0000 7f330f200640 1 Processor -- start 2024-09-13T17:12:54.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.736+0000 7f330f200640 1 -- start start 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3308070cc0 0x7f3308144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33080cabc0 0x7f33080b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080b2c70 con 0x7f330815a540 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33080af290 con 0x7f3308070cc0 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33080b0070 con 0x7f33080cabc0 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57172/0 (socket says 172.21.15.28:57172) 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.737+0000 7f330d200640 1 -- 172.21.15.28:0/971654122 learned_addr learned my addr 172.21.15.28:0/971654122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f3307e00640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3308070cc0 0x7f3308144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f32ff400640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33080cabc0 0x7f33080b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 -- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33080cabc0 msgr2=0x7f33080b3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33080cabc0 0x7f33080b3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 -- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3308070cc0 msgr2=0x7f3308144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3308070cc0 0x7f3308144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 -- 172.21.15.28:0/971654122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3300009b30 con 0x7f330815a540 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.738+0000 7f330d200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3300009fa0 tx=0x7f3300004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.739+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3300003b90 con 0x7f330815a540 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.739+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3300003d10 con 0x7f330815a540 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.739+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308145510 con 0x7f330815a540 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.739+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330003d8a0 con 0x7f330815a540 2024-09-13T17:12:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.739+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33080b25b0 con 0x7f330815a540 2024-09-13T17:12:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.740+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33080b2db0 con 0x7f330815a540 2024-09-13T17:12:54.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.741+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f330003da00 con 0x7f330815a540 2024-09-13T17:12:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.742+0000 7f3304c00640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 0x7f32e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.742+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f330002ae40 con 0x7f330815a540 2024-09-13T17:12:54.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.742+0000 7f3307e00640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 0x7f32e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:54.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.743+0000 7f3307e00640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 0x7f32e406d0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32f80079a0 tx=0x7f32f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:54.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.745+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3300075d10 con 0x7f330815a540 2024-09-13T17:12:54.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.923+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33080b2db0 con 0x7f330815a540 2024-09-13T17:12:54.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.925+0000 7f3304c00640 1 -- 172.21.15.28:0/971654122 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16581 (secure 0 0 0) 0x7f33080b2db0 con 0x7f330815a540 2024-09-13T17:12:54.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:54.927 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:54.533880+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 msgr2=0x7f32e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 0x7f32e406d0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f32f80079a0 tx=0x7f32f8008040 comp rx=0 tx=0).stop 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 msgr2=0x7f33080ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f3300009fa0 tx=0x7f3300004a40 comp rx=0 tx=0).stop 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 shutdown_connections 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32e406ac00 0x7f32e406d0c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f330815a540 0x7f33080ac850 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33080cabc0 0x7f33080b3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 --2- 172.21.15.28:0/971654122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3308070cc0 0x7f3308144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 >> 172.21.15.28:0/971654122 conn(0x7f3308069e80 msgr2=0x7f33080ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 shutdown_connections 2024-09-13T17:12:54.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:54.927+0000 7f330f200640 1 -- 172.21.15.28:0/971654122 wait complete. 2024-09-13T17:12:54.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.100+0000 7fd963c00640 1 Processor -- start 2024-09-13T17:12:55.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.101+0000 7fd963c00640 1 -- start start 2024-09-13T17:12:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd963c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd95c15a540 0x7fd95c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd963c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd95c070cc0 0x7fd95c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd963c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95c15e100 con 0x7fd95c0cabc0 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd963c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c0adb00 con 0x7fd95c15a540 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd963c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95c071670 con 0x7fd95c070cc0 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd961200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd95c15a540 0x7fd95c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd961200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd95c15a540 0x7fd95c06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57174/0 (socket says 172.21.15.28:57174) 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd961200640 1 -- 172.21.15.28:0/4049800482 learned_addr learned my addr 172.21.15.28:0/4049800482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.102+0000 7fd961c00640 1 --2- 172.21.15.28:0/4049800482 >> v2:172.21.15.189:3300/0 conn(0x7fd95c070cc0 0x7fd95c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.103+0000 7fd961200640 1 -- 172.21.15.28:0/4049800482 >> v2:172.21.15.189:3300/0 conn(0x7fd95c070cc0 msgr2=0x7fd95c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.103+0000 7fd961200640 1 --2- 172.21.15.28:0/4049800482 >> v2:172.21.15.189:3300/0 conn(0x7fd95c070cc0 0x7fd95c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.103+0000 7fd961200640 1 -- 172.21.15.28:0/4049800482 >> v1:172.21.15.189:6789/0 conn(0x7fd95c0cabc0 legacy=0x7fd95c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.103+0000 7fd961200640 1 -- 172.21.15.28:0/4049800482 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c15e3e0 con 0x7fd95c15a540 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd961200640 1 --2- 172.21.15.28:0/4049800482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c06c430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd95c070900 tx=0x7fd95002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acc0f33e2c41c850 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd95b400640 1 -- 172.21.15.28:0/4049800482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950032440 con 0x7fd95c15a540 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd95b400640 1 -- 172.21.15.28:0/4049800482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9500325a0 con 0x7fd95c15a540 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd95b400640 1 -- 172.21.15.28:0/4049800482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd950032930 con 0x7fd95c15a540 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd963c00640 1 -- 172.21.15.28:0/4049800482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 msgr2=0x7fd95c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.104+0000 7fd963c00640 1 --2- 172.21.15.28:0/4049800482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c06c430 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd95c070900 tx=0x7fd95002a210 comp rx=0 tx=0).stop 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 -- 172.21.15.28:0/4049800482 shutdown_connections 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 --2- 172.21.15.28:0/4049800482 >> v2:172.21.15.189:3300/0 conn(0x7fd95c070cc0 0x7fd95c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 --2- 172.21.15.28:0/4049800482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c06c430 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 -- 172.21.15.28:0/4049800482 >> 172.21.15.28:0/4049800482 conn(0x7fd95c069e80 msgr2=0x7fd95c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 -- 172.21.15.28:0/4049800482 shutdown_connections 2024-09-13T17:12:55.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.105+0000 7fd963c00640 1 -- 172.21.15.28:0/4049800482 wait complete. 2024-09-13T17:12:55.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.106+0000 7fd963c00640 1 Processor -- start 2024-09-13T17:12:55.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.106+0000 7fd963c00640 1 -- start start 2024-09-13T17:12:55.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd95c070cc0 0x7fd95c13e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd95c0cabc0 0x7fd95c14c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c0b0b30 con 0x7fd95c15a540 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c13bc30 con 0x7fd95c0cabc0 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.107+0000 7fd963c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c13ee40 con 0x7fd95c070cc0 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57182/0 (socket says 172.21.15.28:57182) 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 -- 172.21.15.28:0/1632935607 learned_addr learned my addr 172.21.15.28:0/1632935607 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961200640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd95c070cc0 0x7fd95c13e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd95be00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd95c0cabc0 0x7fd95c14c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 -- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd95c070cc0 msgr2=0x7fd95c13e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd95c070cc0 0x7fd95c13e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 -- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd95c0cabc0 msgr2=0x7fd95c14c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd95c0cabc0 0x7fd95c14c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 -- 172.21.15.28:0/1632935607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9500094d0 con 0x7fd95c15a540 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.108+0000 7fd961c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fd954007e40 tx=0x7fd95400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.109+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954014620 con 0x7fd95c15a540 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.109+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95400ee90 con 0x7fd95c15a540 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.109+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95c0afeb0 con 0x7fd95c15a540 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.109+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9540149e0 con 0x7fd95c15a540 2024-09-13T17:12:55.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.109+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95c0b0470 con 0x7fd95c15a540 2024-09-13T17:12:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.110+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95c0b0c70 con 0x7fd95c15a540 2024-09-13T17:12:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.111+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd954014b40 con 0x7fd95c15a540 2024-09-13T17:12:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.112+0000 7fd958c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 0x7fd93c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.112+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd954089ca0 con 0x7fd95c15a540 2024-09-13T17:12:55.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.112+0000 7fd961200640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 0x7fd93c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.113+0000 7fd961200640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 0x7fd93c06d0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd950034000 tx=0x7fd95002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.116+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd954054e90 con 0x7fd95c15a540 2024-09-13T17:12:55.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.298+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd95c071130 con 0x7fd95c15a540 2024-09-13T17:12:55.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.299+0000 7fd958c00640 1 -- 172.21.15.28:0/1632935607 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v76) v1 ==== 74+0+16581 (secure 0 0 0) 0x7fd9540584b0 con 0x7fd95c15a540 2024-09-13T17:12:55.301 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:55.301 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":76,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:54.533880+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.301+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 msgr2=0x7fd93c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.301+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 0x7fd93c06d0c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fd950034000 tx=0x7fd95002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.301+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 msgr2=0x7fd95c14ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.301+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fd954007e40 tx=0x7fd95400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 shutdown_connections 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd93c06ac00 0x7fd93c06d0c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd95c15a540 0x7fd95c14ca80 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd95c0cabc0 0x7fd95c14c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 --2- 172.21.15.28:0/1632935607 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd95c070cc0 0x7fd95c13e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 >> 172.21.15.28:0/1632935607 conn(0x7fd95c069e80 msgr2=0x7fd95c155490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:55.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 shutdown_connections 2024-09-13T17:12:55.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.302+0000 7fd963c00640 1 -- 172.21.15.28:0/1632935607 wait complete. 2024-09-13T17:12:55.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:12:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.477+0000 7fa067800640 1 Processor -- start 2024-09-13T17:12:55.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.477+0000 7fa067800640 1 -- start start 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.478+0000 7fa067800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa060159770 0x7fa06006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.478+0000 7fa067800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa06006f3f0 0x7fa060070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa067800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06006f920 con 0x7fa0600add00 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa067800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06015b290 con 0x7fa060159770 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa067800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06015b570 con 0x7fa06006f3f0 2024-09-13T17:12:55.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa060159770 0x7fa06006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa060159770 0x7fa06006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57194/0 (socket says 172.21.15.28:57194) 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 -- 172.21.15.28:0/3048576674 learned_addr learned my addr 172.21.15.28:0/3048576674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa057e00640 1 --2- 172.21.15.28:0/3048576674 >> v2:172.21.15.189:3300/0 conn(0x7fa06006f3f0 0x7fa060070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 -- 172.21.15.28:0/3048576674 >> v2:172.21.15.189:3300/0 conn(0x7fa06006f3f0 msgr2=0x7fa060070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 --2- 172.21.15.28:0/3048576674 >> v2:172.21.15.189:3300/0 conn(0x7fa06006f3f0 0x7fa060070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 -- 172.21.15.28:0/3048576674 >> v1:172.21.15.189:6789/0 conn(0x7fa0600add00 legacy=0x7fa060159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.479+0000 7fa064e00640 1 -- 172.21.15.28:0/3048576674 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06015b850 con 0x7fa060159770 2024-09-13T17:12:55.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.480+0000 7fa064e00640 1 --2- 172.21.15.28:0/3048576674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa06006fd20 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa06006b950 tx=0x7fa05002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9feb27d10f5c26a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa05fe00640 1 -- 172.21.15.28:0/3048576674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050032440 con 0x7fa060159770 2024-09-13T17:12:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa05fe00640 1 -- 172.21.15.28:0/3048576674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0500325a0 con 0x7fa060159770 2024-09-13T17:12:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa05fe00640 1 -- 172.21.15.28:0/3048576674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050032930 con 0x7fa060159770 2024-09-13T17:12:55.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 -- 172.21.15.28:0/3048576674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 msgr2=0x7fa06006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 --2- 172.21.15.28:0/3048576674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa06006fd20 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fa06006b950 tx=0x7fa05002a160 comp rx=0 tx=0).stop 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 -- 172.21.15.28:0/3048576674 shutdown_connections 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 --2- 172.21.15.28:0/3048576674 >> v2:172.21.15.189:3300/0 conn(0x7fa06006f3f0 0x7fa060070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 --2- 172.21.15.28:0/3048576674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa06006fd20 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 -- 172.21.15.28:0/3048576674 >> 172.21.15.28:0/3048576674 conn(0x7fa0600b0270 msgr2=0x7fa0600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 -- 172.21.15.28:0/3048576674 shutdown_connections 2024-09-13T17:12:55.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.481+0000 7fa067800640 1 -- 172.21.15.28:0/3048576674 wait complete. 2024-09-13T17:12:55.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.483+0000 7fa067800640 1 Processor -- start 2024-09-13T17:12:55.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.483+0000 7fa067800640 1 -- start start 2024-09-13T17:12:55.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600add00 0x7fa0600ab220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa0601f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0601568b0 con 0x7fa060159770 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600ab760 con 0x7fa0600add00 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa067800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600abd50 con 0x7fa06006f3f0 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.484+0000 7fa065800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa0601f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa057e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600add00 0x7fa0600ab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47050/0 (socket says 172.21.15.28:47050) 2024-09-13T17:12:55.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 -- 172.21.15.28:0/4137704726 learned_addr learned my addr 172.21.15.28:0/4137704726 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 -- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600add00 msgr2=0x7fa0600ab220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600add00 0x7fa0600ab220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 -- 172.21.15.28:0/4137704726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 msgr2=0x7fa0601f15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa0601f15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:55.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 -- 172.21.15.28:0/4137704726 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500094d0 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa065800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa0601f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.485+0000 7fa064e00640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa050000c00 tx=0x7fa0500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.486+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa050032440 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.486+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa050030ba0 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.486+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa05003d440 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.486+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0600ac030 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.486+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0600ac5f0 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.487+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0600abaa0 con 0x7fa06006f3f0 2024-09-13T17:12:55.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.488+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa050039040 con 0x7fa06006f3f0 2024-09-13T17:12:55.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.489+0000 7fa05d600640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 0x7fa03c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:55.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.489+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0500a9e80 con 0x7fa06006f3f0 2024-09-13T17:12:55.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.489+0000 7fa057e00640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 0x7fa03c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:55.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.490+0000 7fa057e00640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 0x7fa03c06d0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0601f14c0 tx=0x7fa048008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:55.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.494+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa050075070 con 0x7fa06006f3f0 2024-09-13T17:12:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:55.715+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa06015a650 con 0x7fa06006f3f0 2024-09-13T17:12:56.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.546+0000 7fa05d600640 1 -- 172.21.15.28:0/4137704726 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa0500370b0 con 0x7fa06006f3f0 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.549+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 msgr2=0x7fa03c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.549+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 0x7fa03c06d0c0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa0601f14c0 tx=0x7fa048008040 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.549+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 msgr2=0x7fa060155a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.549+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa050000c00 tx=0x7fa0500046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 shutdown_connections 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03c06ac00 0x7fa03c06d0c0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa060159770 0x7fa0601f15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0600add00 0x7fa0600ab220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 --2- 172.21.15.28:0/4137704726 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa06006f3f0 0x7fa060155a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 >> 172.21.15.28:0/4137704726 conn(0x7fa0600b0270 msgr2=0x7fa0600cdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:56.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 shutdown_connections 2024-09-13T17:12:56.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.550+0000 7fa067800640 1 -- 172.21.15.28:0/4137704726 wait complete. 2024-09-13T17:12:56.570 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:12:56.570 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:12:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.732+0000 7f2d67c00640 1 Processor -- start 2024-09-13T17:12:56.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.732+0000 7f2d67c00640 1 -- start start 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.733+0000 7f2d67c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d6015a780 0x7f2d60070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.733+0000 7f2d67c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 0x7f2d600712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d67c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d6006c8b0 con 0x7f2d6006bb40 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d67c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d6006cb90 con 0x7f2d6015a780 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d67c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d600cc980 con 0x7f2d6015ab60 2024-09-13T17:12:56.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d6015a780 0x7f2d60070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d6015a780 0x7f2d60070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57208/0 (socket says 172.21.15.28:57208) 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 -- 172.21.15.28:0/1902499173 learned_addr learned my addr 172.21.15.28:0/1902499173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d57e00640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 0x7f2d600712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 -- 172.21.15.28:0/1902499173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 msgr2=0x7f2d600712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 0x7f2d600712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 -- 172.21.15.28:0/1902499173 >> v1:172.21.15.189:6789/0 conn(0x7f2d6006bb40 legacy=0x7f2d6006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d65200640 1 -- 172.21.15.28:0/1902499173 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d600ccc60 con 0x7f2d6015a780 2024-09-13T17:12:56.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.734+0000 7f2d57e00640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 0x7f2d600712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.735+0000 7f2d65200640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015a780 0x7f2d60070d80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009870 tx=0x7f2d5002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6292b91a13237cdf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d5fe00640 1 -- 172.21.15.28:0/1902499173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50032440 con 0x7f2d6015a780 2024-09-13T17:12:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d5fe00640 1 -- 172.21.15.28:0/1902499173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d500325a0 con 0x7f2d6015a780 2024-09-13T17:12:56.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d5fe00640 1 -- 172.21.15.28:0/1902499173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50032930 con 0x7f2d6015a780 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 -- 172.21.15.28:0/1902499173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015a780 msgr2=0x7f2d60070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015a780 0x7f2d60070d80 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f2d50009870 tx=0x7f2d5002a210 comp rx=0 tx=0).stop 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 -- 172.21.15.28:0/1902499173 shutdown_connections 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015ab60 0x7f2d600712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 --2- 172.21.15.28:0/1902499173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015a780 0x7f2d60070d80 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 -- 172.21.15.28:0/1902499173 >> 172.21.15.28:0/1902499173 conn(0x7f2d600b0270 msgr2=0x7f2d600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 -- 172.21.15.28:0/1902499173 shutdown_connections 2024-09-13T17:12:56.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.736+0000 7f2d67c00640 1 -- 172.21.15.28:0/1902499173 wait complete. 2024-09-13T17:12:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.738+0000 7f2d67c00640 1 Processor -- start 2024-09-13T17:12:56.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.738+0000 7f2d67c00640 1 -- start start 2024-09-13T17:12:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d6006bb40 0x7f2d600c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015a780 0x7f2d60149790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6014ace0 con 0x7f2d6015ab60 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d60149cd0 con 0x7f2d6006bb40 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d67c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d6014a060 con 0x7f2d6015a780 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d65c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d65200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d6006bb40 0x7f2d600c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d65c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57212/0 (socket says 172.21.15.28:57212) 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.739+0000 7f2d65c00640 1 -- 172.21.15.28:0/2269497933 learned_addr learned my addr 172.21.15.28:0/2269497933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 -- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015a780 msgr2=0x7f2d60149790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015a780 0x7f2d60149790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 -- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d6006bb40 msgr2=0x7f2d600c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d6006bb40 0x7f2d600c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:56.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 -- 172.21.15.28:0/2269497933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d500094d0 con 0x7f2d6015ab60 2024-09-13T17:12:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d65c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800c780 tx=0x7f2d5800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d58004b60 con 0x7f2d6015ab60 2024-09-13T17:12:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5800ce90 con 0x7f2d6015ab60 2024-09-13T17:12:56.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d5800a850 con 0x7f2d6015ab60 2024-09-13T17:12:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d6014a340 con 0x7f2d6015ab60 2024-09-13T17:12:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.740+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d6014a900 con 0x7f2d6015ab60 2024-09-13T17:12:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.741+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d6014ae20 con 0x7f2d6015ab60 2024-09-13T17:12:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.742+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2d5800a9b0 con 0x7f2d6015ab60 2024-09-13T17:12:56.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.743+0000 7f2d5d600640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 0x7f2d3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:56.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.743+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d58089440 con 0x7f2d6015ab60 2024-09-13T17:12:56.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.744+0000 7f2d65200640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 0x7f2d3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:56.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.745+0000 7f2d65200640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 0x7f2d3c06d0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2d50034000 tx=0x7f2d5002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:56.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.748+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2d6014ae20 con 0x7f2d6015ab60 2024-09-13T17:12:56.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:56.961+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2d6014ae20 con 0x7f2d6015ab60 2024-09-13T17:12:57.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.551+0000 7f2d5d600640 1 -- 172.21.15.28:0/2269497933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d6014ae20 con 0x7f2d6015ab60 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 msgr2=0x7f2d3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 0x7f2d3c06d0c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2d50034000 tx=0x7f2d5002e000 comp rx=0 tx=0).stop 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 msgr2=0x7f2d60157f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800c780 tx=0x7f2d5800cc40 comp rx=0 tx=0).stop 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 shutdown_connections 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3c06ac00 0x7f2d3c06d0c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d6015ab60 0x7f2d60157f90 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d6015a780 0x7f2d60149790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 --2- 172.21.15.28:0/2269497933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d6006bb40 0x7f2d600c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 >> 172.21.15.28:0/2269497933 conn(0x7f2d600b0270 msgr2=0x7f2d600cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:57.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 shutdown_connections 2024-09-13T17:12:57.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.554+0000 7f2d67c00640 1 -- 172.21.15.28:0/2269497933 wait complete. 2024-09-13T17:12:57.574 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:12:57.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:57.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.730+0000 7f0a88c00640 1 Processor -- start 2024-09-13T17:12:57.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.730+0000 7f0a88c00640 1 -- start start 2024-09-13T17:12:57.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a88c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8406f970 0x7f0a8415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a88c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a84159d90 con 0x7f0a841592a0 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a88c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a840b2040 con 0x7f0a8406c2d0 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a88c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8415b0b0 con 0x7f0a8406f970 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8406f970 0x7f0a8415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0a8406f970 0x7f0a8415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57216/0 (socket says 172.21.15.28:57216) 2024-09-13T17:12:57.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 -- 172.21.15.28:0/2611501398 learned_addr learned my addr 172.21.15.28:0/2611501398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 -- 172.21.15.28:0/2611501398 >> v1:172.21.15.189:6789/0 conn(0x7f0a8406c2d0 legacy=0x7f0a840b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 -- 172.21.15.28:0/2611501398 >> v1:172.21.15.28:6789/0 conn(0x7f0a841592a0 legacy=0x7f0a84159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.732+0000 7f0a79400640 1 -- 172.21.15.28:0/2611501398 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84159ff0 con 0x7f0a8406f970 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.733+0000 7f0a79400640 1 --2- 172.21.15.28:0/2611501398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406f970 0x7f0a8415ab70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009870 tx=0x7f0a6c02c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b670c293f02cc31c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a81400640 1 -- 172.21.15.28:0/2611501398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c031710 con 0x7f0a8406f970 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a81400640 1 -- 172.21.15.28:0/2611501398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c02f050 con 0x7f0a8406f970 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a81400640 1 -- 172.21.15.28:0/2611501398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c031a90 con 0x7f0a8406f970 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 -- 172.21.15.28:0/2611501398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406f970 msgr2=0x7f0a8415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2611501398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406f970 0x7f0a8415ab70 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009870 tx=0x7f0a6c02c4b0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 -- 172.21.15.28:0/2611501398 shutdown_connections 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2611501398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406f970 0x7f0a8415ab70 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 -- 172.21.15.28:0/2611501398 >> 172.21.15.28:0/2611501398 conn(0x7f0a84069e80 msgr2=0x7f0a8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 -- 172.21.15.28:0/2611501398 shutdown_connections 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.734+0000 7f0a88c00640 1 -- 172.21.15.28:0/2611501398 wait complete. 2024-09-13T17:12:57.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.736+0000 7f0a88c00640 1 Processor -- start 2024-09-13T17:12:57.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.736+0000 7f0a88c00640 1 -- start start 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a8406f970 0x7f0a840cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 0x7f0a840ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a841465b0 con 0x7f0a8406c2d0 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84150f50 con 0x7f0a841592a0 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a88c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a84151230 con 0x7f0a8406f970 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57220/0 (socket says 172.21.15.28:57220) 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 -- 172.21.15.28:0/2108300106 learned_addr learned my addr 172.21.15.28:0/2108300106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:57.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a79400640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a8406f970 0x7f0a840cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a82800640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 0x7f0a840ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 -- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a8406f970 msgr2=0x7f0a840cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a8406f970 0x7f0a840cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.737+0000 7f0a81e00640 1 -- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 msgr2=0x7f0a840ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a81e00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 0x7f0a840ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a81e00640 1 -- 172.21.15.28:0/2108300106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0094d0 con 0x7f0a8406c2d0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a82800640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 0x7f0a840ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a81e00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c390 tx=0x7f0a7400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a740052d0 con 0x7f0a8406c2d0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74005430 con 0x7f0a8406c2d0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74003830 con 0x7f0a8406c2d0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a84151510 con 0x7f0a8406c2d0 2024-09-13T17:12:57.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.738+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a84151ad0 con 0x7f0a8406c2d0 2024-09-13T17:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.739+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8415ab70 con 0x7f0a8406c2d0 2024-09-13T17:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.740+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0a74003990 con 0x7f0a8406c2d0 2024-09-13T17:12:57.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.741+0000 7f0a7aa00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 0x7f0a6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.741+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a740888b0 con 0x7f0a8406c2d0 2024-09-13T17:12:57.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.741+0000 7f0a79400640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 0x7f0a6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:57.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.742+0000 7f0a79400640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 0x7f0a6006d0c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c035000 tx=0x7f0a6c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:57.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.745+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0a74053a80 con 0x7f0a8406c2d0 2024-09-13T17:12:57.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.957+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a8415b850 con 0x7f0a8406c2d0 2024-09-13T17:12:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.958+0000 7f0a7aa00640 1 -- 172.21.15.28:0/2108300106 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f0a74017090 con 0x7f0a8406c2d0 2024-09-13T17:12:57.960 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:57.960 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1620029613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1620029613},{"type":"v1","addr":"172.21.15.189:6835","nonce":1620029613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6649,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3274497119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3274497119},{"type":"v1","addr":"172.21.15.28:6835","nonce":3274497119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6658,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2973808639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2973808639},{"type":"v1","addr":"172.21.15.189:6837","nonce":2973808639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:12:57.545680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3037911251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3037911251},{"type":"v1","addr":"172.21.15.28:6834","nonce":3037911251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:12:57.960 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 64 2024-09-13T17:12:57.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 msgr2=0x7f0a6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 0x7f0a6006d0c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c035000 tx=0x7f0a6c008030 comp rx=0 tx=0).stop 2024-09-13T17:12:57.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 msgr2=0x7f0a84146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c390 tx=0x7f0a7400c850 comp rx=0 tx=0).stop 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 shutdown_connections 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a6006ac00 0x7f0a6006d0c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a841592a0 0x7f0a840ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a8406f970 0x7f0a840cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 --2- 172.21.15.28:0/2108300106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a8406c2d0 0x7f0a84146070 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 >> 172.21.15.28:0/2108300106 conn(0x7f0a84069e80 msgr2=0x7f0a84159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 shutdown_connections 2024-09-13T17:12:57.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:57.960+0000 7f0a88c00640 1 -- 172.21.15.28:0/2108300106 wait complete. 2024-09-13T17:12:57.977 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:12:58.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.130+0000 7f23ed200640 1 Processor -- start 2024-09-13T17:12:58.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.131+0000 7f23ed200640 1 -- start start 2024-09-13T17:12:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23e8154a90 0x7f23e815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e806b360 0x7f23e80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 0x7f23e80af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e80b0df0 con 0x7f23e8154a90 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23e815bb10 con 0x7f23e80b0980 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23ed200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e806b900 con 0x7f23e806b360 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23e7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23e8154a90 0x7f23e815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23e7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23e8154a90 0x7f23e815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57230/0 (socket says 172.21.15.28:57230) 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23e7400640 1 -- 172.21.15.28:0/2688537437 learned_addr learned my addr 172.21.15.28:0/2688537437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e806b360 0x7f23e80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.132+0000 7f23e7e00640 1 --2- 172.21.15.28:0/2688537437 >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 0x7f23e80af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7400640 1 -- 172.21.15.28:0/2688537437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e806b360 msgr2=0x7f23e80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7400640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e806b360 0x7f23e80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7400640 1 -- 172.21.15.28:0/2688537437 >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 msgr2=0x7f23e80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7400640 1 --2- 172.21.15.28:0/2688537437 >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 0x7f23e80af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7400640 1 -- 172.21.15.28:0/2688537437 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e806bbe0 con 0x7f23e8154a90 2024-09-13T17:12:58.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.133+0000 7f23e7e00640 1 --2- 172.21.15.28:0/2688537437 >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 0x7f23e80af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23e7400640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e8154a90 0x7f23e815b5d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f23e815e080 tx=0x7f23d802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=533a2563cff256e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23e6a00640 1 -- 172.21.15.28:0/2688537437 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8032440 con 0x7f23e8154a90 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23e6a00640 1 -- 172.21.15.28:0/2688537437 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d80325a0 con 0x7f23e8154a90 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23e6a00640 1 -- 172.21.15.28:0/2688537437 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8032930 con 0x7f23e8154a90 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23ed200640 1 -- 172.21.15.28:0/2688537437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e8154a90 msgr2=0x7f23e815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23ed200640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e8154a90 0x7f23e815b5d0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f23e815e080 tx=0x7f23d802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.134+0000 7f23ed200640 1 -- 172.21.15.28:0/2688537437 shutdown_connections 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 --2- 172.21.15.28:0/2688537437 >> v2:172.21.15.189:3300/0 conn(0x7f23e80b0980 0x7f23e80af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e806b360 0x7f23e80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 --2- 172.21.15.28:0/2688537437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e8154a90 0x7f23e815b5d0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 -- 172.21.15.28:0/2688537437 >> 172.21.15.28:0/2688537437 conn(0x7f23e8072e40 msgr2=0x7f23e8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 -- 172.21.15.28:0/2688537437 shutdown_connections 2024-09-13T17:12:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.135+0000 7f23ed200640 1 -- 172.21.15.28:0/2688537437 wait complete. 2024-09-13T17:12:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.136+0000 7f23ed200640 1 Processor -- start 2024-09-13T17:12:58.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.136+0000 7f23ed200640 1 -- start start 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e80b0980 0x7f23e8153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 0x7f23e81f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8137c10 con 0x7f23e806b360 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8136f90 con 0x7f23e80b0980 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.137+0000 7f23ed200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23e8137270 con 0x7f23e8154a90 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57238/0 (socket says 172.21.15.28:57238) 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 -- 172.21.15.28:0/2685492693 learned_addr learned my addr 172.21.15.28:0/2685492693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e80b0980 0x7f23e8153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7e00640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 0x7f23e81f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 -- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 msgr2=0x7f23e81f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 0x7f23e81f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 -- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e80b0980 msgr2=0x7f23e8153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e80b0980 0x7f23e8153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 -- 172.21.15.28:0/2685492693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80094d0 con 0x7f23e806b360 2024-09-13T17:12:58.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7e00640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 0x7f23e81f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.138+0000 7f23e7400640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f23d802a220 tx=0x7f23d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.139+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8032440 con 0x7f23e806b360 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.139+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d8030980 con 0x7f23e806b360 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.139+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d8030d10 con 0x7f23e806b360 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.139+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23e8137550 con 0x7f23e806b360 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.139+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23e81f3020 con 0x7f23e806b360 2024-09-13T17:12:58.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.140+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23e80709b0 con 0x7f23e806b360 2024-09-13T17:12:58.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.141+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f23d8039040 con 0x7f23e806b360 2024-09-13T17:12:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.142+0000 7f23df400640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 0x7f23c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.142+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23d80a9d30 con 0x7f23e806b360 2024-09-13T17:12:58.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.142+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 0x7f23c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.143+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 0x7f23c406d0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b0060 tx=0x7f23d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.146+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f23d8074e70 con 0x7f23e806b360 2024-09-13T17:12:58.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.328+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23e80b03c0 con 0x7f23e806b360 2024-09-13T17:12:58.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.329+0000 7f23df400640 1 -- 172.21.15.28:0/2685492693 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f23d8041090 con 0x7f23e806b360 2024-09-13T17:12:58.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:58.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:55.537103+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 msgr2=0x7f23c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 0x7f23c406d0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f23e80b0060 tx=0x7f23d0009190 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 msgr2=0x7f23e81369b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f23d802a220 tx=0x7f23d80046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 shutdown_connections 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23c406ac00 0x7f23c406d0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23e8154a90 0x7f23e81f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e80b0980 0x7f23e8153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 --2- 172.21.15.28:0/2685492693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23e806b360 0x7f23e81369b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 >> 172.21.15.28:0/2685492693 conn(0x7f23e8072e40 msgr2=0x7f23e8158770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 shutdown_connections 2024-09-13T17:12:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.331+0000 7f23ed200640 1 -- 172.21.15.28:0/2685492693 wait complete. 2024-09-13T17:12:58.349 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:12:58.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:12:58.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.502+0000 7f24a6a00640 1 Processor -- start 2024-09-13T17:12:58.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.502+0000 7f24a6a00640 1 -- start start 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.503+0000 7f24a6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a01592a0 0x7f24a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f24a6a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 0x7f24a0159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f24a6a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a00b2040 con 0x7f24a006c2d0 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f24a6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a015b0b0 con 0x7f24a01592a0 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f24a6a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a0159ff0 con 0x7f24a006f970 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a01592a0 0x7f24a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f24a01592a0 0x7f24a015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57250/0 (socket says 172.21.15.28:57250) 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 -- 172.21.15.28:0/2721953927 learned_addr learned my addr 172.21.15.28:0/2721953927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f2497400640 1 --2- 172.21.15.28:0/2721953927 >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 0x7f24a0159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 -- 172.21.15.28:0/2721953927 >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 msgr2=0x7f24a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:58.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 --2- 172.21.15.28:0/2721953927 >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 0x7f24a0159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 -- 172.21.15.28:0/2721953927 >> v1:172.21.15.189:6789/0 conn(0x7f24a006c2d0 legacy=0x7f24a00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f249fe00640 1 -- 172.21.15.28:0/2721953927 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a015a2d0 con 0x7f24a01592a0 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.504+0000 7f2497400640 1 --2- 172.21.15.28:0/2721953927 >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 0x7f24a0159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.505+0000 7f249fe00640 1 --2- 172.21.15.28:0/2721953927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a01592a0 0x7f24a015ab70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f24a006ac10 tx=0x7f249002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a35fd1c1422d768 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f249f400640 1 -- 172.21.15.28:0/2721953927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490032440 con 0x7f24a01592a0 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f249f400640 1 -- 172.21.15.28:0/2721953927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24900325a0 con 0x7f24a01592a0 2024-09-13T17:12:58.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f249f400640 1 -- 172.21.15.28:0/2721953927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490032930 con 0x7f24a01592a0 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 -- 172.21.15.28:0/2721953927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a01592a0 msgr2=0x7f24a015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 --2- 172.21.15.28:0/2721953927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a01592a0 0x7f24a015ab70 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f24a006ac10 tx=0x7f249002a270 comp rx=0 tx=0).stop 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 -- 172.21.15.28:0/2721953927 shutdown_connections 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 --2- 172.21.15.28:0/2721953927 >> v2:172.21.15.189:3300/0 conn(0x7f24a006f970 0x7f24a0159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 --2- 172.21.15.28:0/2721953927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a01592a0 0x7f24a015ab70 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 -- 172.21.15.28:0/2721953927 >> 172.21.15.28:0/2721953927 conn(0x7f24a0069e80 msgr2=0x7f24a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 -- 172.21.15.28:0/2721953927 shutdown_connections 2024-09-13T17:12:58.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.506+0000 7f24a6a00640 1 -- 172.21.15.28:0/2721953927 wait complete. 2024-09-13T17:12:58.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.508+0000 7f24a6a00640 1 Processor -- start 2024-09-13T17:12:58.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.508+0000 7f24a6a00640 1 -- start start 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a006f970 0x7f24a014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24a01592a0 0x7f24a0154150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a01ef470 con 0x7f24a006c2d0 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a014e860 con 0x7f24a006f970 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f24a6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a014eb40 con 0x7f24a01592a0 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f249fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f249fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57252/0 (socket says 172.21.15.28:57252) 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.509+0000 7f249fe00640 1 -- 172.21.15.28:0/1597845930 learned_addr learned my addr 172.21.15.28:0/1597845930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 -- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24a01592a0 msgr2=0x7f24a0154150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24a01592a0 0x7f24a0154150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 -- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a006f970 msgr2=0x7f24a014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a006f970 0x7f24a014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 -- 172.21.15.28:0/1597845930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900094d0 con 0x7f24a006c2d0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249fe00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f249002a2b0 tx=0x7f24900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490032440 con 0x7f24a006c2d0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490030ba0 con 0x7f24a006c2d0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.510+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a014ee20 con 0x7f24a006c2d0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.511+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f249003d440 con 0x7f24a006c2d0 2024-09-13T17:12:58.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.511+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a0152730 con 0x7f24a006c2d0 2024-09-13T17:12:58.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.512+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a00adf60 con 0x7f24a006c2d0 2024-09-13T17:12:58.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.513+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2490039040 con 0x7f24a006c2d0 2024-09-13T17:12:58.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.514+0000 7f249cc00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 0x7f247c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.514+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24900a9f20 con 0x7f24a006c2d0 2024-09-13T17:12:58.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.514+0000 7f2497400640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 0x7f247c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.515+0000 7f2497400640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 0x7f247c06d0c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2488006e80 tx=0x7f2488008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.518+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f24900750f0 con 0x7f24a006c2d0 2024-09-13T17:12:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.692+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24a015a890 con 0x7f247c06ac00 2024-09-13T17:12:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.701+0000 7f249cc00640 1 -- 172.21.15.28:0/1597845930 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+274647 (secure 0 0 0) 0x7f2488009d80 con 0x7f247c06ac00 2024-09-13T17:12:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:58.708 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:12:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.705+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 msgr2=0x7f247c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.705+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 0x7f247c06d0c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2488006e80 tx=0x7f2488008040 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.705+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 msgr2=0x7f24a01eef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.705+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f249002a2b0 tx=0x7f24900046c0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 shutdown_connections 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f247c06ac00 0x7f247c06d0c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24a01592a0 0x7f24a0154150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24a006f970 0x7f24a014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 --2- 172.21.15.28:0/1597845930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24a006c2d0 0x7f24a01eef30 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 >> 172.21.15.28:0/1597845930 conn(0x7f24a0069e80 msgr2=0x7f24a00ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 shutdown_connections 2024-09-13T17:12:58.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.706+0000 7f24a6a00640 1 -- 172.21.15.28:0/1597845930 wait complete. 2024-09-13T17:12:58.723 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2024-09-13T17:12:56.955742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":378392,"kb_used_data":8096,"kb_used_omap":12,"kb_used_meta":370163,"kb_avail":754596328,"statfs":{"total":773094113280,"available":772706639872,"internally_reserved":0,"allocated":8290304,"data_stored":4179810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":379047481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000971"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:54.093653+0000","last_change":"2024-09-13T17:12:54.093653+0000","last_active":"2024-09-13T17:12:54.093653+0000","last_peered":"2024-09-13T17:12:54.093653+0000","last_clean":"2024-09-13T17:12:54.093653+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:54.093653+0000","last_undegraded":"2024-09-13T17:12:54.093653+0000","last_fullsized":"2024-09-13T17:12:54.093653+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:12:53.086682+0000","last_change":"2024-09-13T17:12:53.086682+0000","last_active":"2024-09-13T17:12:53.086682+0000","last_peered":"2024-09-13T17:12:53.086682+0000","last_clean":"2024-09-13T17:12:53.086682+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:12:53.086682+0000","last_undegraded":"2024-09-13T17:12:53.086682+0000","last_fullsized":"2024-09-13T17:12:53.086682+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":211,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-13T17:12:50.189184+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:12:50.189184+0000","last_peered":"2024-09-13T17:12:50.189184+0000","last_clean":"2024-09-13T17:12:50.189184+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:12:50.189184+0000","last_undegraded":"2024-09-13T17:12:50.189184+0000","last_fullsized":"2024-09-13T17:12:50.189184+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574910,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58804,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94313036,"statfs":{"total":96636764160,"available":96576548864,"internally_reserved":0,"allocated":1482752,"data_stored":965507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.96199999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96399999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.78800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.75},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.69099999999999995,"15min":0.68600000000000005},"min":{"1min":0.47699999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.68000000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.51600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90000000000000002,"5min":1.224,"15min":1.224},"last":0.76500000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.071,"15min":1.071},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97199999999999998,"5min":1.177,"15min":1.177},"last":0.80300000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.41699999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0449999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.67800000000000005,"15min":0.69999999999999996},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97099999999999997,"5min":1.143,"15min":1.143},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.58499999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.125,"5min":1.337,"15min":1.337},"last":0.62}]},{"osd":6,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.755,"15min":0.77100000000000002},"min":{"1min":0.434,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0449999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":0.78000000000000003},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.71299999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40180,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331660,"statfs":{"total":96636764160,"available":96595619840,"internally_reserved":0,"allocated":888832,"data_stored":375139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68100000000000005,"15min":0.65600000000000003},"min":{"1min":0.39500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91000000000000003,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.876},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.52000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.002,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84499999999999997,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.63200000000000001},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.84799999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64300000000000002,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.48099999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.94899999999999995,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.89100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69799999999999995,"15min":0.70099999999999996},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.024,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89300000000000002,"5min":3.742,"15min":3.742},"last":0.82499999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66800000000000004,"15min":0.65300000000000002},"min":{"1min":0.46000000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86499999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.55100000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.71099999999999997,"15min":0.72699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89600000000000002,"5min":3.609,"15min":3.609},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76500000000000001,"15min":0.75700000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98599999999999999,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.629},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.103,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.80900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40176,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":888832,"data_stored":375139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":2.391,"5min":1,"15min":0.78200000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":1.645,"5min":0.879,"15min":0.77300000000000002},"min":{"1min":0.45200000000000001,"5min":0.437,"15min":0.437},"max":{"1min":21.376999999999999,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.74199999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.83299999999999996,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":17.231000000000002,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.502},{"interface":"front","average":{"1min":1,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":0.85899999999999999,"15min":0.76700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":17.268000000000001,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":1.135,"5min":0.79800000000000004,"15min":0.76200000000000001},"min":{"1min":0.46500000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":9.6549999999999994,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.627}]},{"osd":3,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":6.0899999999999999,"5min":1.8069999999999999,"15min":1.1120000000000001},"min":{"1min":0.47299999999999998,"5min":0.435,"15min":0.435},"max":{"1min":113.363,"5min":113.363,"15min":113.363},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.77900000000000003,"15min":0.77600000000000002},"min":{"1min":0.52900000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.1180000000000003,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.626},"min":{"1min":0.434,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.48799999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72099999999999997,"15min":0.70699999999999996},"min":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.74099999999999999,"15min":0.72699999999999998},"min":{"1min":0.51200000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.78600000000000003,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77900000000000003,"15min":0.76600000000000001},"min":{"1min":0.56599999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.69299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":62324,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94309516,"statfs":{"total":96636764160,"available":96572944384,"internally_reserved":0,"allocated":888832,"data_stored":375139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.438,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81200000000000006,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91200000000000003,"5min":1.341,"15min":1.341},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80500000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.70099999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.46700000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.86699999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.51600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.71599999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.47399999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.96999999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.81699999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.623}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90400000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71999999999999997,"15min":0.748},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.01,"5min":1.413,"15min":1.413},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.04,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.78000000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":50036,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321804,"statfs":{"total":96636764160,"available":96585527296,"internally_reserved":0,"allocated":888832,"data_stored":375139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79400000000000004,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.752},{"interface":"front","average":{"1min":0.436,"5min":0.52700000000000002,"15min":0.57699999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92400000000000004,"5min":2.96,"15min":2.96},"last":0.46700000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95699999999999996,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.68799999999999994,"15min":0.74199999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93799999999999994,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.39700000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80400000000000005,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78300000000000003,"15min":0.82299999999999995},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.093,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.54200000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.123,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.63}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78600000000000003,"15min":0.79300000000000004},"min":{"1min":0.56599999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2529999999999999,"5min":1.391,"15min":1.391},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.81799999999999995,"15min":0.84799999999999998},"min":{"1min":0.52200000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.173,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.64600000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78900000000000003,"15min":0.77000000000000002},"min":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1579999999999999,"5min":1.401,"15min":1.401},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1850000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.80600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.139,"5min":2.899,"15min":2.899},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.83499999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1220000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.71599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574910,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":40176,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":884736,"data_stored":374120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57799999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.34499999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":3.0699999999999998,"15min":1.3400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58599999999999997,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":3.1120000000000001,"15min":1.353},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59399999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.39700000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65600000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.438},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72799999999999998,"5min":1.121,"15min":1.121},"last":0.45000000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.753,"15min":0.75800000000000001},"min":{"1min":0.56999999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2450000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.60599999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.397,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.71999999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.048,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.748},{"interface":"front","average":{"1min":0.746,"5min":0.76800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99299999999999999,"5min":1.569,"15min":1.569},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0940000000000001,"5min":1.413,"15min":1.413},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.024,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.073,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.56699999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.83099999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574909,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":45940,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325900,"statfs":{"total":96636764160,"available":96589721600,"internally_reserved":0,"allocated":888832,"data_stored":375139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.4939999999999998,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.40500000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.33100000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.79500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.748,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":18.318000000000001,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.46999999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.84199999999999997,"15min":0.81999999999999995},"min":{"1min":0.51700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.1630000000000003,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.78,"15min":1.78},"last":0.86799999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6020000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.84199999999999997,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6599999999999999,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.82999999999999996},"min":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.88}]},{"osd":7,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.82899999999999996,"15min":0.83799999999999997},"min":{"1min":0.61699999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.472,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.544,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.76800000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40756,"kb_used_data":1444,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331084,"statfs":{"total":96636764160,"available":96595030016,"internally_reserved":0,"allocated":1478656,"data_stored":964488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.58199999999999996,"15min":0.72399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.99099999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.436},{"interface":"front","average":{"1min":0.495,"5min":0.59199999999999997,"15min":0.70999999999999996},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1859999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.44800000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.621,"15min":0.745},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66500000000000004,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.64200000000000002,"15min":0.76400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2190000000000001,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.38200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.60099999999999998,"15min":0.69899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.5069999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65700000000000003,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1000000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.92600000000000005},"min":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3160000000000001,"5min":4.734,"15min":4.734},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.84899999999999998,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.264,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.95999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.175,"5min":4.944,"15min":4.944},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.85399999999999998,"15min":0.93999999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2050000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94399999999999995,"15min":1.125},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1439999999999999,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.875,"15min":0.96899999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.68899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89800000000000002,"15min":1.0069999999999999},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88500000000000001,"15min":0.98699999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.70099999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:12:58.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:12:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.916+0000 7f77a0a00640 1 Processor -- start 2024-09-13T17:12:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.916+0000 7f77a0a00640 1 -- start start 2024-09-13T17:12:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.917+0000 7f77a0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.917+0000 7f77a0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c159d90 con 0x7f779c06c2d0 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.917+0000 7f77a0a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779c0b2040 con 0x7f779c1592a0 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.917+0000 7f77a0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779c15b0b0 con 0x7f779c06f970 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.917+0000 7f7799400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f779a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f779c06c2d0 0x7f779c0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57082/0 (socket says 172.21.15.28:57082) 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f779a800640 1 -- 172.21.15.28:0/2325036160 learned_addr learned my addr 172.21.15.28:0/2325036160 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f7799400640 1 -- 172.21.15.28:0/2325036160 >> v1:172.21.15.189:6789/0 conn(0x7f779c1592a0 legacy=0x7f779c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f7799400640 1 -- 172.21.15.28:0/2325036160 >> v1:172.21.15.28:6789/0 conn(0x7f779c06c2d0 legacy=0x7f779c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f7799400640 1 -- 172.21.15.28:0/2325036160 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c159ff0 con 0x7f779c06f970 2024-09-13T17:12:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.918+0000 7f7798a00640 1 -- 172.21.15.28:0/2325036160 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606147226 0 0) 0x7f779c159d90 con 0x7f779c06c2d0 2024-09-13T17:12:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.919+0000 7f7799400640 1 --2- 172.21.15.28:0/2325036160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06f970 0x7f779c15ab70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f778800ba40 tx=0x7f778802c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c50987ccd6c32c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.919+0000 7f7798a00640 1 -- 172.21.15.28:0/2325036160 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77880327e0 con 0x7f779c06f970 2024-09-13T17:12:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.919+0000 7f7798a00640 1 -- 172.21.15.28:0/2325036160 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7788032940 con 0x7f779c06f970 2024-09-13T17:12:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.919+0000 7f7798a00640 1 -- 172.21.15.28:0/2325036160 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7788032cd0 con 0x7f779c06f970 2024-09-13T17:12:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 -- 172.21.15.28:0/2325036160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06f970 msgr2=0x7f779c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 --2- 172.21.15.28:0/2325036160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06f970 0x7f779c15ab70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f778800ba40 tx=0x7f778802c8e0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 -- 172.21.15.28:0/2325036160 shutdown_connections 2024-09-13T17:12:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 --2- 172.21.15.28:0/2325036160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c06f970 0x7f779c15ab70 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 -- 172.21.15.28:0/2325036160 >> 172.21.15.28:0/2325036160 conn(0x7f779c069e80 msgr2=0x7f779c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 -- 172.21.15.28:0/2325036160 shutdown_connections 2024-09-13T17:12:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.920+0000 7f77a0a00640 1 -- 172.21.15.28:0/2325036160 wait complete. 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.922+0000 7f77a0a00640 1 Processor -- start 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.922+0000 7f77a0a00640 1 -- start start 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06c2d0 0x7f779c14ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 0x7f779c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c14b190 con 0x7f779c1592a0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779c14da10 con 0x7f779c06f970 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f77a0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779c14dcf0 con 0x7f779c06c2d0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57272/0 (socket says 172.21.15.28:57272) 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 -- 172.21.15.28:0/554632519 learned_addr learned my addr 172.21.15.28:0/554632519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f7799400640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 0x7f779c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f7799e00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06c2d0 0x7f779c14ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 -- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06c2d0 msgr2=0x7f779c14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06c2d0 0x7f779c14ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 -- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 msgr2=0x7f779c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 0x7f779c152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f779a800640 1 -- 172.21.15.28:0/554632519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778800b6a0 con 0x7f779c1592a0 2024-09-13T17:12:58.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.923+0000 7f7799400640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 0x7f779c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:12:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f779a800640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f779400caf0 tx=0x7f779400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779400aca0 con 0x7f779c1592a0 2024-09-13T17:12:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779400ae00 con 0x7f779c1592a0 2024-09-13T17:12:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794012640 con 0x7f779c1592a0 2024-09-13T17:12:58.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779c14dfd0 con 0x7f779c1592a0 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.924+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779c14cbe0 con 0x7f779c1592a0 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.925+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779c15ab70 con 0x7f779c1592a0 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.926+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7794012860 con 0x7f779c1592a0 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.927+0000 7f7792000640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 0x7f777c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.927+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7794089980 con 0x7f779c1592a0 2024-09-13T17:12:58.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.928+0000 7f7799e00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 0x7f777c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:58.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.929+0000 7f7799e00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 0x7f777c06d0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f779c1f5530 tx=0x7f778c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:58.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:58.931+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7794054b50 con 0x7f779c1592a0 2024-09-13T17:12:59.069 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-13T17:12:59.067+0000 7fd090e00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:12:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.107+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f779c15a890 con 0x7f777c06ac00 2024-09-13T17:12:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.115+0000 7f7792000640 1 -- 172.21.15.28:0/554632519 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289458 (secure 0 0 0) 0x7f778c0050d0 con 0x7f777c06ac00 2024-09-13T17:12:59.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:12:59.122 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 msgr2=0x7f777c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 0x7f777c06d0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f779c1f5530 tx=0x7f778c009040 comp rx=0 tx=0).stop 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 msgr2=0x7f779c152e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f779400caf0 tx=0x7f779400a060 comp rx=0 tx=0).stop 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 shutdown_connections 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f777c06ac00 0x7f777c06d0c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f779c1592a0 0x7f779c152e60 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f779c06f970 0x7f779c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 --2- 172.21.15.28:0/554632519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f779c06c2d0 0x7f779c14ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 >> 172.21.15.28:0/554632519 conn(0x7f779c069e80 msgr2=0x7f779c06fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:59.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 shutdown_connections 2024-09-13T17:12:59.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.120+0000 7f77a0a00640 1 -- 172.21.15.28:0/554632519 wait complete. 2024-09-13T17:12:59.138 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-13T17:12:58.956166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":378852,"kb_used_data":8544,"kb_used_omap":12,"kb_used_meta":370163,"kb_avail":754595868,"statfs":{"total":773094113280,"available":772706168832,"internally_reserved":0,"allocated":8749056,"data_stored":4385870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":379047481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.861449"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616921+0000","last_change":"2024-09-13T17:12:54.566794+0000","last_active":"2024-09-13T17:12:55.616921+0000","last_peered":"2024-09-13T17:12:55.616921+0000","last_clean":"2024-09-13T17:12:55.616921+0000","last_became_active":"2024-09-13T17:12:54.566532+0000","last_became_peered":"2024-09-13T17:12:54.566532+0000","last_unstale":"2024-09-13T17:12:55.616921+0000","last_undegraded":"2024-09-13T17:12:55.616921+0000","last_fullsized":"2024-09-13T17:12:55.616921+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:49.477669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617611+0000","last_change":"2024-09-13T17:12:54.561453+0000","last_active":"2024-09-13T17:12:55.617611+0000","last_peered":"2024-09-13T17:12:55.617611+0000","last_clean":"2024-09-13T17:12:55.617611+0000","last_became_active":"2024-09-13T17:12:54.561170+0000","last_became_peered":"2024-09-13T17:12:54.561170+0000","last_unstale":"2024-09-13T17:12:55.617611+0000","last_undegraded":"2024-09-13T17:12:55.617611+0000","last_fullsized":"2024-09-13T17:12:55.617611+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:55.351131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617293+0000","last_change":"2024-09-13T17:12:54.553635+0000","last_active":"2024-09-13T17:12:55.617293+0000","last_peered":"2024-09-13T17:12:55.617293+0000","last_clean":"2024-09-13T17:12:55.617293+0000","last_became_active":"2024-09-13T17:12:54.553389+0000","last_became_peered":"2024-09-13T17:12:54.553389+0000","last_unstale":"2024-09-13T17:12:55.617293+0000","last_undegraded":"2024-09-13T17:12:55.617293+0000","last_fullsized":"2024-09-13T17:12:55.617293+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:57.148122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545553+0000","last_change":"2024-09-13T17:12:54.557646+0000","last_active":"2024-09-13T17:12:55.545553+0000","last_peered":"2024-09-13T17:12:55.545553+0000","last_clean":"2024-09-13T17:12:55.545553+0000","last_became_active":"2024-09-13T17:12:54.556986+0000","last_became_peered":"2024-09-13T17:12:54.556986+0000","last_unstale":"2024-09-13T17:12:55.545553+0000","last_undegraded":"2024-09-13T17:12:55.545553+0000","last_fullsized":"2024-09-13T17:12:55.545553+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:33:02.001821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616241+0000","last_change":"2024-09-13T17:12:54.564920+0000","last_active":"2024-09-13T17:12:55.616241+0000","last_peered":"2024-09-13T17:12:55.616241+0000","last_clean":"2024-09-13T17:12:55.616241+0000","last_became_active":"2024-09-13T17:12:54.564678+0000","last_became_peered":"2024-09-13T17:12:54.564678+0000","last_unstale":"2024-09-13T17:12:55.616241+0000","last_undegraded":"2024-09-13T17:12:55.616241+0000","last_fullsized":"2024-09-13T17:12:55.616241+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:49.398352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617415+0000","last_change":"2024-09-13T17:12:54.565455+0000","last_active":"2024-09-13T17:12:55.617415+0000","last_peered":"2024-09-13T17:12:55.617415+0000","last_clean":"2024-09-13T17:12:55.617415+0000","last_became_active":"2024-09-13T17:12:54.565125+0000","last_became_peered":"2024-09-13T17:12:54.565125+0000","last_unstale":"2024-09-13T17:12:55.617415+0000","last_undegraded":"2024-09-13T17:12:55.617415+0000","last_fullsized":"2024-09-13T17:12:55.617415+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:19.791133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547667+0000","last_change":"2024-09-13T17:12:54.548862+0000","last_active":"2024-09-13T17:12:55.547667+0000","last_peered":"2024-09-13T17:12:55.547667+0000","last_clean":"2024-09-13T17:12:55.547667+0000","last_became_active":"2024-09-13T17:12:54.548593+0000","last_became_peered":"2024-09-13T17:12:54.548593+0000","last_unstale":"2024-09-13T17:12:55.547667+0000","last_undegraded":"2024-09-13T17:12:55.547667+0000","last_fullsized":"2024-09-13T17:12:55.547667+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:37.493780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616948+0000","last_change":"2024-09-13T17:12:54.557109+0000","last_active":"2024-09-13T17:12:55.616948+0000","last_peered":"2024-09-13T17:12:55.616948+0000","last_clean":"2024-09-13T17:12:55.616948+0000","last_became_active":"2024-09-13T17:12:54.556476+0000","last_became_peered":"2024-09-13T17:12:54.556476+0000","last_unstale":"2024-09-13T17:12:55.616948+0000","last_undegraded":"2024-09-13T17:12:55.616948+0000","last_fullsized":"2024-09-13T17:12:55.616948+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:54.719099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617365+0000","last_change":"2024-09-13T17:12:54.568783+0000","last_active":"2024-09-13T17:12:55.617365+0000","last_peered":"2024-09-13T17:12:55.617365+0000","last_clean":"2024-09-13T17:12:55.617365+0000","last_became_active":"2024-09-13T17:12:54.567841+0000","last_became_peered":"2024-09-13T17:12:54.567841+0000","last_unstale":"2024-09-13T17:12:55.617365+0000","last_undegraded":"2024-09-13T17:12:55.617365+0000","last_fullsized":"2024-09-13T17:12:55.617365+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:37.545081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549492+0000","last_change":"2024-09-13T17:12:54.554614+0000","last_active":"2024-09-13T17:12:55.549492+0000","last_peered":"2024-09-13T17:12:55.549492+0000","last_clean":"2024-09-13T17:12:55.549492+0000","last_became_active":"2024-09-13T17:12:54.554351+0000","last_became_peered":"2024-09-13T17:12:54.554351+0000","last_unstale":"2024-09-13T17:12:55.549492+0000","last_undegraded":"2024-09-13T17:12:55.549492+0000","last_fullsized":"2024-09-13T17:12:55.549492+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:47.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616566+0000","last_change":"2024-09-13T17:12:54.567913+0000","last_active":"2024-09-13T17:12:55.616566+0000","last_peered":"2024-09-13T17:12:55.616566+0000","last_clean":"2024-09-13T17:12:55.616566+0000","last_became_active":"2024-09-13T17:12:54.567296+0000","last_became_peered":"2024-09-13T17:12:54.567296+0000","last_unstale":"2024-09-13T17:12:55.616566+0000","last_undegraded":"2024-09-13T17:12:55.616566+0000","last_fullsized":"2024-09-13T17:12:55.616566+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:37.493780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545649+0000","last_change":"2024-09-13T17:12:54.556716+0000","last_active":"2024-09-13T17:12:55.545649+0000","last_peered":"2024-09-13T17:12:55.545649+0000","last_clean":"2024-09-13T17:12:55.545649+0000","last_became_active":"2024-09-13T17:12:54.556450+0000","last_became_peered":"2024-09-13T17:12:54.556450+0000","last_unstale":"2024-09-13T17:12:55.545649+0000","last_undegraded":"2024-09-13T17:12:55.545649+0000","last_fullsized":"2024-09-13T17:12:55.545649+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:35.928893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616580+0000","last_change":"2024-09-13T17:12:54.556882+0000","last_active":"2024-09-13T17:12:55.616580+0000","last_peered":"2024-09-13T17:12:55.616580+0000","last_clean":"2024-09-13T17:12:55.616580+0000","last_became_active":"2024-09-13T17:12:54.556623+0000","last_became_peered":"2024-09-13T17:12:54.556623+0000","last_unstale":"2024-09-13T17:12:55.616580+0000","last_undegraded":"2024-09-13T17:12:55.616580+0000","last_fullsized":"2024-09-13T17:12:55.616580+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:00.053103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616426+0000","last_change":"2024-09-13T17:12:54.554781+0000","last_active":"2024-09-13T17:12:55.616426+0000","last_peered":"2024-09-13T17:12:55.616426+0000","last_clean":"2024-09-13T17:12:55.616426+0000","last_became_active":"2024-09-13T17:12:54.554592+0000","last_became_peered":"2024-09-13T17:12:54.554592+0000","last_unstale":"2024-09-13T17:12:55.616426+0000","last_undegraded":"2024-09-13T17:12:55.616426+0000","last_fullsized":"2024-09-13T17:12:55.616426+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:32.607080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.615944+0000","last_change":"2024-09-13T17:12:54.559362+0000","last_active":"2024-09-13T17:12:55.615944+0000","last_peered":"2024-09-13T17:12:55.615944+0000","last_clean":"2024-09-13T17:12:55.615944+0000","last_became_active":"2024-09-13T17:12:54.554991+0000","last_became_peered":"2024-09-13T17:12:54.554991+0000","last_unstale":"2024-09-13T17:12:55.615944+0000","last_undegraded":"2024-09-13T17:12:55.615944+0000","last_fullsized":"2024-09-13T17:12:55.615944+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:43.767771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548979+0000","last_change":"2024-09-13T17:12:54.567925+0000","last_active":"2024-09-13T17:12:55.548979+0000","last_peered":"2024-09-13T17:12:55.548979+0000","last_clean":"2024-09-13T17:12:55.548979+0000","last_became_active":"2024-09-13T17:12:54.567399+0000","last_became_peered":"2024-09-13T17:12:54.567399+0000","last_unstale":"2024-09-13T17:12:55.548979+0000","last_undegraded":"2024-09-13T17:12:55.548979+0000","last_fullsized":"2024-09-13T17:12:55.548979+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:09:30.571715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617378+0000","last_change":"2024-09-13T17:12:54.569077+0000","last_active":"2024-09-13T17:12:55.617378+0000","last_peered":"2024-09-13T17:12:55.617378+0000","last_clean":"2024-09-13T17:12:55.617378+0000","last_became_active":"2024-09-13T17:12:54.566890+0000","last_became_peered":"2024-09-13T17:12:54.566890+0000","last_unstale":"2024-09-13T17:12:55.617378+0000","last_undegraded":"2024-09-13T17:12:55.617378+0000","last_fullsized":"2024-09-13T17:12:55.617378+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:23.824048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545636+0000","last_change":"2024-09-13T17:12:54.566694+0000","last_active":"2024-09-13T17:12:55.545636+0000","last_peered":"2024-09-13T17:12:55.545636+0000","last_clean":"2024-09-13T17:12:55.545636+0000","last_became_active":"2024-09-13T17:12:54.566453+0000","last_became_peered":"2024-09-13T17:12:54.566453+0000","last_unstale":"2024-09-13T17:12:55.545636+0000","last_undegraded":"2024-09-13T17:12:55.545636+0000","last_fullsized":"2024-09-13T17:12:55.545636+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:32:01.015739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617274+0000","last_change":"2024-09-13T17:12:54.555629+0000","last_active":"2024-09-13T17:12:55.617274+0000","last_peered":"2024-09-13T17:12:55.617274+0000","last_clean":"2024-09-13T17:12:55.617274+0000","last_became_active":"2024-09-13T17:12:54.555311+0000","last_became_peered":"2024-09-13T17:12:54.555311+0000","last_unstale":"2024-09-13T17:12:55.617274+0000","last_undegraded":"2024-09-13T17:12:55.617274+0000","last_fullsized":"2024-09-13T17:12:55.617274+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:50.801854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616709+0000","last_change":"2024-09-13T17:12:54.548811+0000","last_active":"2024-09-13T17:12:55.616709+0000","last_peered":"2024-09-13T17:12:55.616709+0000","last_clean":"2024-09-13T17:12:55.616709+0000","last_became_active":"2024-09-13T17:12:54.548480+0000","last_became_peered":"2024-09-13T17:12:54.548480+0000","last_unstale":"2024-09-13T17:12:55.616709+0000","last_undegraded":"2024-09-13T17:12:55.616709+0000","last_fullsized":"2024-09-13T17:12:55.616709+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:52.144935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549218+0000","last_change":"2024-09-13T17:12:54.546911+0000","last_active":"2024-09-13T17:12:55.549218+0000","last_peered":"2024-09-13T17:12:55.549218+0000","last_clean":"2024-09-13T17:12:55.549218+0000","last_became_active":"2024-09-13T17:12:54.546484+0000","last_became_peered":"2024-09-13T17:12:54.546484+0000","last_unstale":"2024-09-13T17:12:55.549218+0000","last_undegraded":"2024-09-13T17:12:55.549218+0000","last_fullsized":"2024-09-13T17:12:55.549218+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:47.709940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549020+0000","last_change":"2024-09-13T17:12:54.556716+0000","last_active":"2024-09-13T17:12:55.549020+0000","last_peered":"2024-09-13T17:12:55.549020+0000","last_clean":"2024-09-13T17:12:55.549020+0000","last_became_active":"2024-09-13T17:12:54.556446+0000","last_became_peered":"2024-09-13T17:12:54.556446+0000","last_unstale":"2024-09-13T17:12:55.549020+0000","last_undegraded":"2024-09-13T17:12:55.549020+0000","last_fullsized":"2024-09-13T17:12:55.549020+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:17.310210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616877+0000","last_change":"2024-09-13T17:12:54.558865+0000","last_active":"2024-09-13T17:12:55.616877+0000","last_peered":"2024-09-13T17:12:55.616877+0000","last_clean":"2024-09-13T17:12:55.616877+0000","last_became_active":"2024-09-13T17:12:54.558497+0000","last_became_peered":"2024-09-13T17:12:54.558497+0000","last_unstale":"2024-09-13T17:12:55.616877+0000","last_undegraded":"2024-09-13T17:12:55.616877+0000","last_fullsized":"2024-09-13T17:12:55.616877+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:50.103115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616296+0000","last_change":"2024-09-13T17:12:54.568079+0000","last_active":"2024-09-13T17:12:55.616296+0000","last_peered":"2024-09-13T17:12:55.616296+0000","last_clean":"2024-09-13T17:12:55.616296+0000","last_became_active":"2024-09-13T17:12:54.567842+0000","last_became_peered":"2024-09-13T17:12:54.567842+0000","last_unstale":"2024-09-13T17:12:55.616296+0000","last_undegraded":"2024-09-13T17:12:55.616296+0000","last_fullsized":"2024-09-13T17:12:55.616296+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:48.118978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549331+0000","last_change":"2024-09-13T17:12:54.557629+0000","last_active":"2024-09-13T17:12:55.549331+0000","last_peered":"2024-09-13T17:12:55.549331+0000","last_clean":"2024-09-13T17:12:55.549331+0000","last_became_active":"2024-09-13T17:12:54.554903+0000","last_became_peered":"2024-09-13T17:12:54.554903+0000","last_unstale":"2024-09-13T17:12:55.549331+0000","last_undegraded":"2024-09-13T17:12:55.549331+0000","last_fullsized":"2024-09-13T17:12:55.549331+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:31.387068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548016+0000","last_change":"2024-09-13T17:12:54.555835+0000","last_active":"2024-09-13T17:12:55.548016+0000","last_peered":"2024-09-13T17:12:55.548016+0000","last_clean":"2024-09-13T17:12:55.548016+0000","last_became_active":"2024-09-13T17:12:54.555542+0000","last_became_peered":"2024-09-13T17:12:54.555542+0000","last_unstale":"2024-09-13T17:12:55.548016+0000","last_undegraded":"2024-09-13T17:12:55.548016+0000","last_fullsized":"2024-09-13T17:12:55.548016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:43.767771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617380+0000","last_change":"2024-09-13T17:12:54.556545+0000","last_active":"2024-09-13T17:12:55.617380+0000","last_peered":"2024-09-13T17:12:55.617380+0000","last_clean":"2024-09-13T17:12:55.617380+0000","last_became_active":"2024-09-13T17:12:54.556255+0000","last_became_peered":"2024-09-13T17:12:54.556255+0000","last_unstale":"2024-09-13T17:12:55.617380+0000","last_undegraded":"2024-09-13T17:12:55.617380+0000","last_fullsized":"2024-09-13T17:12:55.617380+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:11.325153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549386+0000","last_change":"2024-09-13T17:12:54.557382+0000","last_active":"2024-09-13T17:12:55.549386+0000","last_peered":"2024-09-13T17:12:55.549386+0000","last_clean":"2024-09-13T17:12:55.549386+0000","last_became_active":"2024-09-13T17:12:54.557107+0000","last_became_peered":"2024-09-13T17:12:54.557107+0000","last_unstale":"2024-09-13T17:12:55.549386+0000","last_undegraded":"2024-09-13T17:12:55.549386+0000","last_fullsized":"2024-09-13T17:12:55.549386+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:06:09.196533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616936+0000","last_change":"2024-09-13T17:12:54.551735+0000","last_active":"2024-09-13T17:12:55.616936+0000","last_peered":"2024-09-13T17:12:55.616936+0000","last_clean":"2024-09-13T17:12:55.616936+0000","last_became_active":"2024-09-13T17:12:54.549622+0000","last_became_peered":"2024-09-13T17:12:54.549622+0000","last_unstale":"2024-09-13T17:12:55.616936+0000","last_undegraded":"2024-09-13T17:12:55.616936+0000","last_fullsized":"2024-09-13T17:12:55.616936+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:37.964346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547707+0000","last_change":"2024-09-13T17:12:54.564698+0000","last_active":"2024-09-13T17:12:55.547707+0000","last_peered":"2024-09-13T17:12:55.547707+0000","last_clean":"2024-09-13T17:12:55.547707+0000","last_became_active":"2024-09-13T17:12:54.564437+0000","last_became_peered":"2024-09-13T17:12:54.564437+0000","last_unstale":"2024-09-13T17:12:55.547707+0000","last_undegraded":"2024-09-13T17:12:55.547707+0000","last_fullsized":"2024-09-13T17:12:55.547707+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:09:30.571715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547979+0000","last_change":"2024-09-13T17:12:54.558901+0000","last_active":"2024-09-13T17:12:55.547979+0000","last_peered":"2024-09-13T17:12:55.547979+0000","last_clean":"2024-09-13T17:12:55.547979+0000","last_became_active":"2024-09-13T17:12:54.558643+0000","last_became_peered":"2024-09-13T17:12:54.558643+0000","last_unstale":"2024-09-13T17:12:55.547979+0000","last_undegraded":"2024-09-13T17:12:55.547979+0000","last_fullsized":"2024-09-13T17:12:55.547979+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:01.932531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616995+0000","last_change":"2024-09-13T17:12:54.565756+0000","last_active":"2024-09-13T17:12:55.616995+0000","last_peered":"2024-09-13T17:12:55.616995+0000","last_clean":"2024-09-13T17:12:55.616995+0000","last_became_active":"2024-09-13T17:12:54.565160+0000","last_became_peered":"2024-09-13T17:12:54.565160+0000","last_unstale":"2024-09-13T17:12:55.616995+0000","last_undegraded":"2024-09-13T17:12:55.616995+0000","last_fullsized":"2024-09-13T17:12:55.616995+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:16.690910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616589+0000","last_change":"2024-09-13T17:12:54.568401+0000","last_active":"2024-09-13T17:12:55.616589+0000","last_peered":"2024-09-13T17:12:55.616589+0000","last_clean":"2024-09-13T17:12:55.616589+0000","last_became_active":"2024-09-13T17:12:54.567364+0000","last_became_peered":"2024-09-13T17:12:54.567364+0000","last_unstale":"2024-09-13T17:12:55.616589+0000","last_undegraded":"2024-09-13T17:12:55.616589+0000","last_fullsized":"2024-09-13T17:12:55.616589+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:42.312270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"76'4","reported_seq":15,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617084+0000","last_change":"2024-09-13T17:12:54.121257+0000","last_active":"2024-09-13T17:12:55.617084+0000","last_peered":"2024-09-13T17:12:55.617084+0000","last_clean":"2024-09-13T17:12:55.617084+0000","last_became_active":"2024-09-13T17:12:54.120971+0000","last_became_peered":"2024-09-13T17:12:54.120971+0000","last_unstale":"2024-09-13T17:12:55.617084+0000","last_undegraded":"2024-09-13T17:12:55.617084+0000","last_fullsized":"2024-09-13T17:12:55.617084+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:18.350656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617435+0000","last_change":"2024-09-13T17:12:54.568146+0000","last_active":"2024-09-13T17:12:55.617435+0000","last_peered":"2024-09-13T17:12:55.617435+0000","last_clean":"2024-09-13T17:12:55.617435+0000","last_became_active":"2024-09-13T17:12:54.566412+0000","last_became_peered":"2024-09-13T17:12:54.566412+0000","last_unstale":"2024-09-13T17:12:55.617435+0000","last_undegraded":"2024-09-13T17:12:55.617435+0000","last_fullsized":"2024-09-13T17:12:55.617435+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:02.264445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"76'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617105+0000","last_change":"2024-09-13T17:12:54.120628+0000","last_active":"2024-09-13T17:12:55.617105+0000","last_peered":"2024-09-13T17:12:55.617105+0000","last_clean":"2024-09-13T17:12:55.617105+0000","last_became_active":"2024-09-13T17:12:54.120147+0000","last_became_peered":"2024-09-13T17:12:54.120147+0000","last_unstale":"2024-09-13T17:12:55.617105+0000","last_undegraded":"2024-09-13T17:12:55.617105+0000","last_fullsized":"2024-09-13T17:12:55.617105+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:51.623991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617333+0000","last_change":"2024-09-13T17:12:54.560541+0000","last_active":"2024-09-13T17:12:55.617333+0000","last_peered":"2024-09-13T17:12:55.617333+0000","last_clean":"2024-09-13T17:12:55.617333+0000","last_became_active":"2024-09-13T17:12:54.559249+0000","last_became_peered":"2024-09-13T17:12:54.559249+0000","last_unstale":"2024-09-13T17:12:55.617333+0000","last_undegraded":"2024-09-13T17:12:55.617333+0000","last_fullsized":"2024-09-13T17:12:55.617333+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:07.672008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549520+0000","last_change":"2024-09-13T17:12:54.119907+0000","last_active":"2024-09-13T17:12:55.549520+0000","last_peered":"2024-09-13T17:12:55.549520+0000","last_clean":"2024-09-13T17:12:55.549520+0000","last_became_active":"2024-09-13T17:12:54.119546+0000","last_became_peered":"2024-09-13T17:12:54.119546+0000","last_unstale":"2024-09-13T17:12:55.549520+0000","last_undegraded":"2024-09-13T17:12:55.549520+0000","last_fullsized":"2024-09-13T17:12:55.549520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:40:15.683939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616692+0000","last_change":"2024-09-13T17:12:54.560886+0000","last_active":"2024-09-13T17:12:55.616692+0000","last_peered":"2024-09-13T17:12:55.616692+0000","last_clean":"2024-09-13T17:12:55.616692+0000","last_became_active":"2024-09-13T17:12:54.560616+0000","last_became_peered":"2024-09-13T17:12:54.560616+0000","last_unstale":"2024-09-13T17:12:55.616692+0000","last_undegraded":"2024-09-13T17:12:55.616692+0000","last_fullsized":"2024-09-13T17:12:55.616692+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:43.767771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617582+0000","last_change":"2024-09-13T17:12:54.125881+0000","last_active":"2024-09-13T17:12:55.617582+0000","last_peered":"2024-09-13T17:12:55.617582+0000","last_clean":"2024-09-13T17:12:55.617582+0000","last_became_active":"2024-09-13T17:12:54.124116+0000","last_became_peered":"2024-09-13T17:12:54.124116+0000","last_unstale":"2024-09-13T17:12:55.617582+0000","last_undegraded":"2024-09-13T17:12:55.617582+0000","last_fullsized":"2024-09-13T17:12:55.617582+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:48.391381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548053+0000","last_change":"2024-09-13T17:12:54.548662+0000","last_active":"2024-09-13T17:12:55.548053+0000","last_peered":"2024-09-13T17:12:55.548053+0000","last_clean":"2024-09-13T17:12:55.548053+0000","last_became_active":"2024-09-13T17:12:54.547538+0000","last_became_peered":"2024-09-13T17:12:54.547538+0000","last_unstale":"2024-09-13T17:12:55.548053+0000","last_undegraded":"2024-09-13T17:12:55.548053+0000","last_fullsized":"2024-09-13T17:12:55.548053+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:31.387068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"76'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617112+0000","last_change":"2024-09-13T17:12:54.119631+0000","last_active":"2024-09-13T17:12:55.617112+0000","last_peered":"2024-09-13T17:12:55.617112+0000","last_clean":"2024-09-13T17:12:55.617112+0000","last_became_active":"2024-09-13T17:12:54.119341+0000","last_became_peered":"2024-09-13T17:12:54.119341+0000","last_unstale":"2024-09-13T17:12:55.617112+0000","last_undegraded":"2024-09-13T17:12:55.617112+0000","last_fullsized":"2024-09-13T17:12:55.617112+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:22.170751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549240+0000","last_change":"2024-09-13T17:12:54.556514+0000","last_active":"2024-09-13T17:12:55.549240+0000","last_peered":"2024-09-13T17:12:55.549240+0000","last_clean":"2024-09-13T17:12:55.549240+0000","last_became_active":"2024-09-13T17:12:54.556218+0000","last_became_peered":"2024-09-13T17:12:54.556218+0000","last_unstale":"2024-09-13T17:12:55.549240+0000","last_undegraded":"2024-09-13T17:12:55.549240+0000","last_fullsized":"2024-09-13T17:12:55.549240+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:47.040584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"76'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617208+0000","last_change":"2024-09-13T17:12:54.127260+0000","last_active":"2024-09-13T17:12:55.617208+0000","last_peered":"2024-09-13T17:12:55.617208+0000","last_clean":"2024-09-13T17:12:55.617208+0000","last_became_active":"2024-09-13T17:12:54.126941+0000","last_became_peered":"2024-09-13T17:12:54.126941+0000","last_unstale":"2024-09-13T17:12:55.617208+0000","last_undegraded":"2024-09-13T17:12:55.617208+0000","last_fullsized":"2024-09-13T17:12:55.617208+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:59.980762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.615952+0000","last_change":"2024-09-13T17:12:54.552973+0000","last_active":"2024-09-13T17:12:55.615952+0000","last_peered":"2024-09-13T17:12:55.615952+0000","last_clean":"2024-09-13T17:12:55.615952+0000","last_became_active":"2024-09-13T17:12:54.552723+0000","last_became_peered":"2024-09-13T17:12:54.552723+0000","last_unstale":"2024-09-13T17:12:55.615952+0000","last_undegraded":"2024-09-13T17:12:55.615952+0000","last_fullsized":"2024-09-13T17:12:55.615952+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:41.772776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545287+0000","last_change":"2024-09-13T17:12:54.115804+0000","last_active":"2024-09-13T17:12:55.545287+0000","last_peered":"2024-09-13T17:12:55.545287+0000","last_clean":"2024-09-13T17:12:55.545287+0000","last_became_active":"2024-09-13T17:12:54.115505+0000","last_became_peered":"2024-09-13T17:12:54.115505+0000","last_unstale":"2024-09-13T17:12:55.545287+0000","last_undegraded":"2024-09-13T17:12:55.545287+0000","last_fullsized":"2024-09-13T17:12:55.545287+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:19:16.786908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616428+0000","last_change":"2024-09-13T17:12:54.553565+0000","last_active":"2024-09-13T17:12:55.616428+0000","last_peered":"2024-09-13T17:12:55.616428+0000","last_clean":"2024-09-13T17:12:55.616428+0000","last_became_active":"2024-09-13T17:12:54.553286+0000","last_became_peered":"2024-09-13T17:12:54.553286+0000","last_unstale":"2024-09-13T17:12:55.616428+0000","last_undegraded":"2024-09-13T17:12:55.616428+0000","last_fullsized":"2024-09-13T17:12:55.616428+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:09:30.571715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616032+0000","last_change":"2024-09-13T17:12:54.114923+0000","last_active":"2024-09-13T17:12:55.616032+0000","last_peered":"2024-09-13T17:12:55.616032+0000","last_clean":"2024-09-13T17:12:55.616032+0000","last_became_active":"2024-09-13T17:12:54.114581+0000","last_became_peered":"2024-09-13T17:12:54.114581+0000","last_unstale":"2024-09-13T17:12:55.616032+0000","last_undegraded":"2024-09-13T17:12:55.616032+0000","last_fullsized":"2024-09-13T17:12:55.616032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:39.976677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548134+0000","last_change":"2024-09-13T17:12:54.556368+0000","last_active":"2024-09-13T17:12:55.548134+0000","last_peered":"2024-09-13T17:12:55.548134+0000","last_clean":"2024-09-13T17:12:55.548134+0000","last_became_active":"2024-09-13T17:12:54.556099+0000","last_became_peered":"2024-09-13T17:12:54.556099+0000","last_unstale":"2024-09-13T17:12:55.548134+0000","last_undegraded":"2024-09-13T17:12:55.548134+0000","last_fullsized":"2024-09-13T17:12:55.548134+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:47.935661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549369+0000","last_change":"2024-09-13T17:12:54.124283+0000","last_active":"2024-09-13T17:12:55.549369+0000","last_peered":"2024-09-13T17:12:55.549369+0000","last_clean":"2024-09-13T17:12:55.549369+0000","last_became_active":"2024-09-13T17:12:54.121521+0000","last_became_peered":"2024-09-13T17:12:54.121521+0000","last_unstale":"2024-09-13T17:12:55.549369+0000","last_undegraded":"2024-09-13T17:12:55.549369+0000","last_fullsized":"2024-09-13T17:12:55.549369+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:14.292501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616989+0000","last_change":"2024-09-13T17:12:54.568830+0000","last_active":"2024-09-13T17:12:55.616989+0000","last_peered":"2024-09-13T17:12:55.616989+0000","last_clean":"2024-09-13T17:12:55.616989+0000","last_became_active":"2024-09-13T17:12:54.568578+0000","last_became_peered":"2024-09-13T17:12:54.568578+0000","last_unstale":"2024-09-13T17:12:55.616989+0000","last_undegraded":"2024-09-13T17:12:55.616989+0000","last_fullsized":"2024-09-13T17:12:55.616989+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:23.177722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617707+0000","last_change":"2024-09-13T17:12:54.125175+0000","last_active":"2024-09-13T17:12:55.617707+0000","last_peered":"2024-09-13T17:12:55.617707+0000","last_clean":"2024-09-13T17:12:55.617707+0000","last_became_active":"2024-09-13T17:12:54.124915+0000","last_became_peered":"2024-09-13T17:12:54.124915+0000","last_unstale":"2024-09-13T17:12:55.617707+0000","last_undegraded":"2024-09-13T17:12:55.617707+0000","last_fullsized":"2024-09-13T17:12:55.617707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:48.021731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617490+0000","last_change":"2024-09-13T17:12:54.555640+0000","last_active":"2024-09-13T17:12:55.617490+0000","last_peered":"2024-09-13T17:12:55.617490+0000","last_clean":"2024-09-13T17:12:55.617490+0000","last_became_active":"2024-09-13T17:12:54.555331+0000","last_became_peered":"2024-09-13T17:12:54.555331+0000","last_unstale":"2024-09-13T17:12:55.617490+0000","last_undegraded":"2024-09-13T17:12:55.617490+0000","last_fullsized":"2024-09-13T17:12:55.617490+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:59.385849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"76'3","reported_seq":14,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617631+0000","last_change":"2024-09-13T17:12:54.119911+0000","last_active":"2024-09-13T17:12:55.617631+0000","last_peered":"2024-09-13T17:12:55.617631+0000","last_clean":"2024-09-13T17:12:55.617631+0000","last_became_active":"2024-09-13T17:12:54.119654+0000","last_became_peered":"2024-09-13T17:12:54.119654+0000","last_unstale":"2024-09-13T17:12:55.617631+0000","last_undegraded":"2024-09-13T17:12:55.617631+0000","last_fullsized":"2024-09-13T17:12:55.617631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:59.046132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617062+0000","last_change":"2024-09-13T17:12:54.561803+0000","last_active":"2024-09-13T17:12:55.617062+0000","last_peered":"2024-09-13T17:12:55.617062+0000","last_clean":"2024-09-13T17:12:55.617062+0000","last_became_active":"2024-09-13T17:12:54.560626+0000","last_became_peered":"2024-09-13T17:12:54.560626+0000","last_unstale":"2024-09-13T17:12:55.617062+0000","last_undegraded":"2024-09-13T17:12:55.617062+0000","last_fullsized":"2024-09-13T17:12:55.617062+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:00.987733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"76'6","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545732+0000","last_change":"2024-09-13T17:12:54.118704+0000","last_active":"2024-09-13T17:12:55.545732+0000","last_peered":"2024-09-13T17:12:55.545732+0000","last_clean":"2024-09-13T17:12:55.545732+0000","last_became_active":"2024-09-13T17:12:54.118377+0000","last_became_peered":"2024-09-13T17:12:54.118377+0000","last_unstale":"2024-09-13T17:12:55.545732+0000","last_undegraded":"2024-09-13T17:12:55.545732+0000","last_fullsized":"2024-09-13T17:12:55.545732+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:27.365564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616739+0000","last_change":"2024-09-13T17:12:54.561458+0000","last_active":"2024-09-13T17:12:55.616739+0000","last_peered":"2024-09-13T17:12:55.616739+0000","last_clean":"2024-09-13T17:12:55.616739+0000","last_became_active":"2024-09-13T17:12:54.561177+0000","last_became_peered":"2024-09-13T17:12:54.561177+0000","last_unstale":"2024-09-13T17:12:55.616739+0000","last_undegraded":"2024-09-13T17:12:55.616739+0000","last_fullsized":"2024-09-13T17:12:55.616739+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:33:16.614811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545514+0000","last_change":"2024-09-13T17:12:54.114428+0000","last_active":"2024-09-13T17:12:55.545514+0000","last_peered":"2024-09-13T17:12:55.545514+0000","last_clean":"2024-09-13T17:12:55.545514+0000","last_became_active":"2024-09-13T17:12:54.114127+0000","last_became_peered":"2024-09-13T17:12:54.114127+0000","last_unstale":"2024-09-13T17:12:55.545514+0000","last_undegraded":"2024-09-13T17:12:55.545514+0000","last_fullsized":"2024-09-13T17:12:55.545514+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:57.796809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547846+0000","last_change":"2024-09-13T17:12:54.548557+0000","last_active":"2024-09-13T17:12:55.547846+0000","last_peered":"2024-09-13T17:12:55.547846+0000","last_clean":"2024-09-13T17:12:55.547846+0000","last_became_active":"2024-09-13T17:12:54.548187+0000","last_became_peered":"2024-09-13T17:12:54.548187+0000","last_unstale":"2024-09-13T17:12:55.547846+0000","last_undegraded":"2024-09-13T17:12:55.547846+0000","last_fullsized":"2024-09-13T17:12:55.547846+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:52.144935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617350+0000","last_change":"2024-09-13T17:12:54.125197+0000","last_active":"2024-09-13T17:12:55.617350+0000","last_peered":"2024-09-13T17:12:55.617350+0000","last_clean":"2024-09-13T17:12:55.617350+0000","last_became_active":"2024-09-13T17:12:54.124929+0000","last_became_peered":"2024-09-13T17:12:54.124929+0000","last_unstale":"2024-09-13T17:12:55.617350+0000","last_undegraded":"2024-09-13T17:12:55.617350+0000","last_fullsized":"2024-09-13T17:12:55.617350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:19:16.786908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616038+0000","last_change":"2024-09-13T17:12:54.567250+0000","last_active":"2024-09-13T17:12:55.616038+0000","last_peered":"2024-09-13T17:12:55.616038+0000","last_clean":"2024-09-13T17:12:55.616038+0000","last_became_active":"2024-09-13T17:12:54.564859+0000","last_became_peered":"2024-09-13T17:12:54.564859+0000","last_unstale":"2024-09-13T17:12:55.616038+0000","last_undegraded":"2024-09-13T17:12:55.616038+0000","last_fullsized":"2024-09-13T17:12:55.616038+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:23.177722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547803+0000","last_change":"2024-09-13T17:12:54.122713+0000","last_active":"2024-09-13T17:12:55.547803+0000","last_peered":"2024-09-13T17:12:55.547803+0000","last_clean":"2024-09-13T17:12:55.547803+0000","last_became_active":"2024-09-13T17:12:54.122231+0000","last_became_peered":"2024-09-13T17:12:54.122231+0000","last_unstale":"2024-09-13T17:12:55.547803+0000","last_undegraded":"2024-09-13T17:12:55.547803+0000","last_fullsized":"2024-09-13T17:12:55.547803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:18.350656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616114+0000","last_change":"2024-09-13T17:12:54.566379+0000","last_active":"2024-09-13T17:12:55.616114+0000","last_peered":"2024-09-13T17:12:55.616114+0000","last_clean":"2024-09-13T17:12:55.616114+0000","last_became_active":"2024-09-13T17:12:54.566109+0000","last_became_peered":"2024-09-13T17:12:54.566109+0000","last_unstale":"2024-09-13T17:12:55.616114+0000","last_undegraded":"2024-09-13T17:12:55.616114+0000","last_fullsized":"2024-09-13T17:12:55.616114+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:58.803780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"76'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549034+0000","last_change":"2024-09-13T17:12:54.126109+0000","last_active":"2024-09-13T17:12:55.549034+0000","last_peered":"2024-09-13T17:12:55.549034+0000","last_clean":"2024-09-13T17:12:55.549034+0000","last_became_active":"2024-09-13T17:12:54.124294+0000","last_became_peered":"2024-09-13T17:12:54.124294+0000","last_unstale":"2024-09-13T17:12:55.549034+0000","last_undegraded":"2024-09-13T17:12:55.549034+0000","last_fullsized":"2024-09-13T17:12:55.549034+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:06.665037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549316+0000","last_change":"2024-09-13T17:12:54.546006+0000","last_active":"2024-09-13T17:12:55.549316+0000","last_peered":"2024-09-13T17:12:55.549316+0000","last_clean":"2024-09-13T17:12:55.549316+0000","last_became_active":"2024-09-13T17:12:54.545706+0000","last_became_peered":"2024-09-13T17:12:54.545706+0000","last_unstale":"2024-09-13T17:12:55.549316+0000","last_undegraded":"2024-09-13T17:12:55.549316+0000","last_fullsized":"2024-09-13T17:12:55.549316+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:01.932531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"76'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617694+0000","last_change":"2024-09-13T17:12:54.124731+0000","last_active":"2024-09-13T17:12:55.617694+0000","last_peered":"2024-09-13T17:12:55.617694+0000","last_clean":"2024-09-13T17:12:55.617694+0000","last_became_active":"2024-09-13T17:12:54.124452+0000","last_became_peered":"2024-09-13T17:12:54.124452+0000","last_unstale":"2024-09-13T17:12:55.617694+0000","last_undegraded":"2024-09-13T17:12:55.617694+0000","last_fullsized":"2024-09-13T17:12:55.617694+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:36.538110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617016+0000","last_change":"2024-09-13T17:12:54.566179+0000","last_active":"2024-09-13T17:12:55.617016+0000","last_peered":"2024-09-13T17:12:55.617016+0000","last_clean":"2024-09-13T17:12:55.617016+0000","last_became_active":"2024-09-13T17:12:54.564185+0000","last_became_peered":"2024-09-13T17:12:54.564185+0000","last_unstale":"2024-09-13T17:12:55.617016+0000","last_undegraded":"2024-09-13T17:12:55.617016+0000","last_fullsized":"2024-09-13T17:12:55.617016+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:52.144935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549444+0000","last_change":"2024-09-13T17:12:54.125382+0000","last_active":"2024-09-13T17:12:55.549444+0000","last_peered":"2024-09-13T17:12:55.549444+0000","last_clean":"2024-09-13T17:12:55.549444+0000","last_became_active":"2024-09-13T17:12:54.125096+0000","last_became_peered":"2024-09-13T17:12:54.125096+0000","last_unstale":"2024-09-13T17:12:55.549444+0000","last_undegraded":"2024-09-13T17:12:55.549444+0000","last_fullsized":"2024-09-13T17:12:55.549444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:53.712128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545712+0000","last_change":"2024-09-13T17:12:54.562130+0000","last_active":"2024-09-13T17:12:55.545712+0000","last_peered":"2024-09-13T17:12:55.545712+0000","last_clean":"2024-09-13T17:12:55.545712+0000","last_became_active":"2024-09-13T17:12:54.561882+0000","last_became_peered":"2024-09-13T17:12:54.561882+0000","last_unstale":"2024-09-13T17:12:55.545712+0000","last_undegraded":"2024-09-13T17:12:55.545712+0000","last_fullsized":"2024-09-13T17:12:55.545712+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:58.528669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"76'1","reported_seq":12,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617469+0000","last_change":"2024-09-13T17:12:54.124230+0000","last_active":"2024-09-13T17:12:55.617469+0000","last_peered":"2024-09-13T17:12:55.617469+0000","last_clean":"2024-09-13T17:12:55.617469+0000","last_became_active":"2024-09-13T17:12:54.122552+0000","last_became_peered":"2024-09-13T17:12:54.122552+0000","last_unstale":"2024-09-13T17:12:55.617469+0000","last_undegraded":"2024-09-13T17:12:55.617469+0000","last_fullsized":"2024-09-13T17:12:55.617469+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:27.365564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617154+0000","last_change":"2024-09-13T17:12:54.555936+0000","last_active":"2024-09-13T17:12:55.617154+0000","last_peered":"2024-09-13T17:12:55.617154+0000","last_clean":"2024-09-13T17:12:55.617154+0000","last_became_active":"2024-09-13T17:12:54.555530+0000","last_became_peered":"2024-09-13T17:12:54.555530+0000","last_unstale":"2024-09-13T17:12:55.617154+0000","last_undegraded":"2024-09-13T17:12:55.617154+0000","last_fullsized":"2024-09-13T17:12:55.617154+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:28.372535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616573+0000","last_change":"2024-09-13T17:12:54.119557+0000","last_active":"2024-09-13T17:12:55.616573+0000","last_peered":"2024-09-13T17:12:55.616573+0000","last_clean":"2024-09-13T17:12:55.616573+0000","last_became_active":"2024-09-13T17:12:54.119235+0000","last_became_peered":"2024-09-13T17:12:54.119235+0000","last_unstale":"2024-09-13T17:12:55.616573+0000","last_undegraded":"2024-09-13T17:12:55.616573+0000","last_fullsized":"2024-09-13T17:12:55.616573+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:37.314250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617027+0000","last_change":"2024-09-13T17:12:54.552322+0000","last_active":"2024-09-13T17:12:55.617027+0000","last_peered":"2024-09-13T17:12:55.617027+0000","last_clean":"2024-09-13T17:12:55.617027+0000","last_became_active":"2024-09-13T17:12:54.552038+0000","last_became_peered":"2024-09-13T17:12:54.552038+0000","last_unstale":"2024-09-13T17:12:55.617027+0000","last_undegraded":"2024-09-13T17:12:55.617027+0000","last_fullsized":"2024-09-13T17:12:55.617027+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:24:14.616972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617750+0000","last_change":"2024-09-13T17:12:54.126420+0000","last_active":"2024-09-13T17:12:55.617750+0000","last_peered":"2024-09-13T17:12:55.617750+0000","last_clean":"2024-09-13T17:12:55.617750+0000","last_became_active":"2024-09-13T17:12:54.125959+0000","last_became_peered":"2024-09-13T17:12:54.125959+0000","last_unstale":"2024-09-13T17:12:55.617750+0000","last_undegraded":"2024-09-13T17:12:55.617750+0000","last_fullsized":"2024-09-13T17:12:55.617750+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:11.054570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616803+0000","last_change":"2024-09-13T17:12:54.565201+0000","last_active":"2024-09-13T17:12:55.616803+0000","last_peered":"2024-09-13T17:12:55.616803+0000","last_clean":"2024-09-13T17:12:55.616803+0000","last_became_active":"2024-09-13T17:12:54.564967+0000","last_became_peered":"2024-09-13T17:12:54.564967+0000","last_unstale":"2024-09-13T17:12:55.616803+0000","last_undegraded":"2024-09-13T17:12:55.616803+0000","last_fullsized":"2024-09-13T17:12:55.616803+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:42:21.196208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.547897+0000","last_change":"2024-09-13T17:12:54.124026+0000","last_active":"2024-09-13T17:12:55.547897+0000","last_peered":"2024-09-13T17:12:55.547897+0000","last_clean":"2024-09-13T17:12:55.547897+0000","last_became_active":"2024-09-13T17:12:54.121819+0000","last_became_peered":"2024-09-13T17:12:54.121819+0000","last_unstale":"2024-09-13T17:12:55.547897+0000","last_undegraded":"2024-09-13T17:12:55.547897+0000","last_fullsized":"2024-09-13T17:12:55.547897+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:49.794883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617768+0000","last_change":"2024-09-13T17:12:54.564990+0000","last_active":"2024-09-13T17:12:55.617768+0000","last_peered":"2024-09-13T17:12:55.617768+0000","last_clean":"2024-09-13T17:12:55.617768+0000","last_became_active":"2024-09-13T17:12:54.564613+0000","last_became_peered":"2024-09-13T17:12:54.564613+0000","last_unstale":"2024-09-13T17:12:55.617768+0000","last_undegraded":"2024-09-13T17:12:55.617768+0000","last_fullsized":"2024-09-13T17:12:55.617768+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:41.375670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.615932+0000","last_change":"2024-09-13T17:12:54.123790+0000","last_active":"2024-09-13T17:12:55.615932+0000","last_peered":"2024-09-13T17:12:55.615932+0000","last_clean":"2024-09-13T17:12:55.615932+0000","last_became_active":"2024-09-13T17:12:54.123522+0000","last_became_peered":"2024-09-13T17:12:54.123522+0000","last_unstale":"2024-09-13T17:12:55.615932+0000","last_undegraded":"2024-09-13T17:12:55.615932+0000","last_fullsized":"2024-09-13T17:12:55.615932+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:00.018770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617019+0000","last_change":"2024-09-13T17:12:54.555527+0000","last_active":"2024-09-13T17:12:55.617019+0000","last_peered":"2024-09-13T17:12:55.617019+0000","last_clean":"2024-09-13T17:12:55.617019+0000","last_became_active":"2024-09-13T17:12:54.554602+0000","last_became_peered":"2024-09-13T17:12:54.554602+0000","last_unstale":"2024-09-13T17:12:55.617019+0000","last_undegraded":"2024-09-13T17:12:55.617019+0000","last_fullsized":"2024-09-13T17:12:55.617019+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:37.493780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617692+0000","last_change":"2024-09-13T17:12:54.124375+0000","last_active":"2024-09-13T17:12:55.617692+0000","last_peered":"2024-09-13T17:12:55.617692+0000","last_clean":"2024-09-13T17:12:55.617692+0000","last_became_active":"2024-09-13T17:12:54.124075+0000","last_became_peered":"2024-09-13T17:12:54.124075+0000","last_unstale":"2024-09-13T17:12:55.617692+0000","last_undegraded":"2024-09-13T17:12:55.617692+0000","last_fullsized":"2024-09-13T17:12:55.617692+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:01.257474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548029+0000","last_change":"2024-09-13T17:12:54.549537+0000","last_active":"2024-09-13T17:12:55.548029+0000","last_peered":"2024-09-13T17:12:55.548029+0000","last_clean":"2024-09-13T17:12:55.548029+0000","last_became_active":"2024-09-13T17:12:54.549213+0000","last_became_peered":"2024-09-13T17:12:54.549213+0000","last_unstale":"2024-09-13T17:12:55.548029+0000","last_undegraded":"2024-09-13T17:12:55.548029+0000","last_fullsized":"2024-09-13T17:12:55.548029+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:47.709940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.548227+0000","last_change":"2024-09-13T17:12:54.116900+0000","last_active":"2024-09-13T17:12:55.548227+0000","last_peered":"2024-09-13T17:12:55.548227+0000","last_clean":"2024-09-13T17:12:55.548227+0000","last_became_active":"2024-09-13T17:12:54.115644+0000","last_became_peered":"2024-09-13T17:12:54.115644+0000","last_unstale":"2024-09-13T17:12:55.548227+0000","last_undegraded":"2024-09-13T17:12:55.548227+0000","last_fullsized":"2024-09-13T17:12:55.548227+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:54.344160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616777+0000","last_change":"2024-09-13T17:12:54.564431+0000","last_active":"2024-09-13T17:12:55.616777+0000","last_peered":"2024-09-13T17:12:55.616777+0000","last_clean":"2024-09-13T17:12:55.616777+0000","last_became_active":"2024-09-13T17:12:54.563979+0000","last_became_peered":"2024-09-13T17:12:54.563979+0000","last_unstale":"2024-09-13T17:12:55.616777+0000","last_undegraded":"2024-09-13T17:12:55.616777+0000","last_fullsized":"2024-09-13T17:12:55.616777+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:09.324777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617233+0000","last_change":"2024-09-13T17:12:54.123345+0000","last_active":"2024-09-13T17:12:55.617233+0000","last_peered":"2024-09-13T17:12:55.617233+0000","last_clean":"2024-09-13T17:12:55.617233+0000","last_became_active":"2024-09-13T17:12:54.122486+0000","last_became_peered":"2024-09-13T17:12:54.122486+0000","last_unstale":"2024-09-13T17:12:55.617233+0000","last_undegraded":"2024-09-13T17:12:55.617233+0000","last_fullsized":"2024-09-13T17:12:55.617233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:25.285627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616736+0000","last_change":"2024-09-13T17:12:54.563949+0000","last_active":"2024-09-13T17:12:55.616736+0000","last_peered":"2024-09-13T17:12:55.616736+0000","last_clean":"2024-09-13T17:12:55.616736+0000","last_became_active":"2024-09-13T17:12:54.563630+0000","last_became_peered":"2024-09-13T17:12:54.563630+0000","last_unstale":"2024-09-13T17:12:55.616736+0000","last_undegraded":"2024-09-13T17:12:55.616736+0000","last_fullsized":"2024-09-13T17:12:55.616736+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:00.053103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616787+0000","last_change":"2024-09-13T17:12:54.121174+0000","last_active":"2024-09-13T17:12:55.616787+0000","last_peered":"2024-09-13T17:12:55.616787+0000","last_clean":"2024-09-13T17:12:55.616787+0000","last_became_active":"2024-09-13T17:12:54.120943+0000","last_became_peered":"2024-09-13T17:12:54.120943+0000","last_unstale":"2024-09-13T17:12:55.616787+0000","last_undegraded":"2024-09-13T17:12:55.616787+0000","last_fullsized":"2024-09-13T17:12:55.616787+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:11.593132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616301+0000","last_change":"2024-09-13T17:12:54.566370+0000","last_active":"2024-09-13T17:12:55.616301+0000","last_peered":"2024-09-13T17:12:55.616301+0000","last_clean":"2024-09-13T17:12:55.616301+0000","last_became_active":"2024-09-13T17:12:54.566115+0000","last_became_peered":"2024-09-13T17:12:54.566115+0000","last_unstale":"2024-09-13T17:12:55.616301+0000","last_undegraded":"2024-09-13T17:12:55.616301+0000","last_fullsized":"2024-09-13T17:12:55.616301+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:19:17.793879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616231+0000","last_change":"2024-09-13T17:12:54.124240+0000","last_active":"2024-09-13T17:12:55.616231+0000","last_peered":"2024-09-13T17:12:55.616231+0000","last_clean":"2024-09-13T17:12:55.616231+0000","last_became_active":"2024-09-13T17:12:54.123930+0000","last_became_peered":"2024-09-13T17:12:54.123930+0000","last_unstale":"2024-09-13T17:12:55.616231+0000","last_undegraded":"2024-09-13T17:12:55.616231+0000","last_fullsized":"2024-09-13T17:12:55.616231+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:33.244410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617184+0000","last_change":"2024-09-13T17:12:54.559638+0000","last_active":"2024-09-13T17:12:55.617184+0000","last_peered":"2024-09-13T17:12:55.617184+0000","last_clean":"2024-09-13T17:12:55.617184+0000","last_became_active":"2024-09-13T17:12:54.559363+0000","last_became_peered":"2024-09-13T17:12:54.559363+0000","last_unstale":"2024-09-13T17:12:55.617184+0000","last_undegraded":"2024-09-13T17:12:55.617184+0000","last_fullsized":"2024-09-13T17:12:55.617184+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:33.214663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545313+0000","last_change":"2024-09-13T17:12:54.118948+0000","last_active":"2024-09-13T17:12:55.545313+0000","last_peered":"2024-09-13T17:12:55.545313+0000","last_clean":"2024-09-13T17:12:55.545313+0000","last_became_active":"2024-09-13T17:12:54.118714+0000","last_became_peered":"2024-09-13T17:12:54.118714+0000","last_unstale":"2024-09-13T17:12:55.545313+0000","last_undegraded":"2024-09-13T17:12:55.545313+0000","last_fullsized":"2024-09-13T17:12:55.545313+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:59.980762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.545388+0000","last_change":"2024-09-13T17:12:54.561238+0000","last_active":"2024-09-13T17:12:55.545388+0000","last_peered":"2024-09-13T17:12:55.545388+0000","last_clean":"2024-09-13T17:12:55.545388+0000","last_became_active":"2024-09-13T17:12:54.560986+0000","last_became_peered":"2024-09-13T17:12:54.560986+0000","last_unstale":"2024-09-13T17:12:55.545388+0000","last_undegraded":"2024-09-13T17:12:55.545388+0000","last_fullsized":"2024-09-13T17:12:55.545388+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:36:53.278301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.549041+0000","last_change":"2024-09-13T17:12:54.119961+0000","last_active":"2024-09-13T17:12:55.549041+0000","last_peered":"2024-09-13T17:12:55.549041+0000","last_clean":"2024-09-13T17:12:55.549041+0000","last_became_active":"2024-09-13T17:12:54.119322+0000","last_became_peered":"2024-09-13T17:12:54.119322+0000","last_unstale":"2024-09-13T17:12:55.549041+0000","last_undegraded":"2024-09-13T17:12:55.549041+0000","last_fullsized":"2024-09-13T17:12:55.549041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:58.378878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.616904+0000","last_change":"2024-09-13T17:12:54.569174+0000","last_active":"2024-09-13T17:12:55.616904+0000","last_peered":"2024-09-13T17:12:55.616904+0000","last_clean":"2024-09-13T17:12:55.616904+0000","last_became_active":"2024-09-13T17:12:54.568923+0000","last_became_peered":"2024-09-13T17:12:54.568923+0000","last_unstale":"2024-09-13T17:12:55.616904+0000","last_undegraded":"2024-09-13T17:12:55.616904+0000","last_fullsized":"2024-09-13T17:12:55.616904+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:15.299472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":11,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617070+0000","last_change":"2024-09-13T17:12:54.125404+0000","last_active":"2024-09-13T17:12:55.617070+0000","last_peered":"2024-09-13T17:12:55.617070+0000","last_clean":"2024-09-13T17:12:55.617070+0000","last_became_active":"2024-09-13T17:12:54.125114+0000","last_became_peered":"2024-09-13T17:12:54.125114+0000","last_unstale":"2024-09-13T17:12:55.617070+0000","last_undegraded":"2024-09-13T17:12:55.617070+0000","last_fullsized":"2024-09-13T17:12:55.617070+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:57.796809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617474+0000","last_change":"2024-09-13T17:12:54.565732+0000","last_active":"2024-09-13T17:12:55.617474+0000","last_peered":"2024-09-13T17:12:55.617474+0000","last_clean":"2024-09-13T17:12:55.617474+0000","last_became_active":"2024-09-13T17:12:54.565497+0000","last_became_peered":"2024-09-13T17:12:54.565497+0000","last_unstale":"2024-09-13T17:12:55.617474+0000","last_undegraded":"2024-09-13T17:12:55.617474+0000","last_fullsized":"2024-09-13T17:12:55.617474+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":75,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:54.093653+0000","last_clean_scrub_stamp":"2024-09-13T17:12:54.093653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:01:02.968401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"76'2","reported_seq":13,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-13T17:12:55.617239+0000","last_change":"2024-09-13T17:12:54.123181+0000","last_active":"2024-09-13T17:12:55.617239+0000","last_peered":"2024-09-13T17:12:55.617239+0000","last_clean":"2024-09-13T17:12:55.617239+0000","last_became_active":"2024-09-13T17:12:54.122915+0000","last_became_peered":"2024-09-13T17:12:54.122915+0000","last_unstale":"2024-09-13T17:12:55.617239+0000","last_undegraded":"2024-09-13T17:12:55.617239+0000","last_fullsized":"2024-09-13T17:12:55.617239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:12:53.086682+0000","last_clean_scrub_stamp":"2024-09-13T17:12:53.086682+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:22.817077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":215,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-13T17:12:54.548897+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:12:54.548897+0000","last_peered":"2024-09-13T17:12:54.548897+0000","last_clean":"2024-09-13T17:12:54.548897+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:12:54.548897+0000","last_undegraded":"2024-09-13T17:12:54.548897+0000","last_fullsized":"2024-09-13T17:12:54.548897+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574911,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58868,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312972,"statfs":{"total":96636764160,"available":96576483328,"internally_reserved":0,"allocated":1540096,"data_stored":991290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.65100000000000002,"15min":0.64200000000000002},"min":{"1min":0.46800000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.96199999999999997,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.96399999999999997,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.64700000000000002,"15min":0.63500000000000001},"min":{"1min":0.50600000000000001,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.88,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.69099999999999995,"15min":0.68600000000000005},"min":{"1min":0.47699999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.97999999999999998,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.69799999999999995,"15min":0.70199999999999996},"min":{"1min":0.47199999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0109999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71199999999999997,"15min":0.71399999999999997},"min":{"1min":0.51600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.90000000000000002,"5min":1.224,"15min":1.224},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.67400000000000004,"15min":0.67800000000000005},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.89700000000000002,"5min":1.071,"15min":1.071},"last":1.014},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.74399999999999999,"15min":0.73399999999999999},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.97199999999999998,"5min":1.177,"15min":1.177},"last":0.77000000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.63800000000000001,"15min":0.66300000000000003},"min":{"1min":0.41699999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.67900000000000005,"15min":0.69899999999999995},"min":{"1min":0.41299999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0449999999999999,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.67800000000000005,"15min":0.69999999999999996},"min":{"1min":0.38800000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.97099999999999997,"5min":1.143,"15min":1.143},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.58499999999999996,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.125,"5min":1.337,"15min":1.337},"last":0.749}]},{"osd":6,"last update":"Fri Sep 13 17:12:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.755,"15min":0.77100000000000002},"min":{"1min":0.434,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0449999999999999,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.76200000000000001,"15min":0.78000000000000003},"min":{"1min":0.50700000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.1519999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.67200000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574912,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40232,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331608,"statfs":{"total":96636764160,"available":96595566592,"internally_reserved":0,"allocated":946176,"data_stored":400398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68100000000000005,"15min":0.65600000000000003},"min":{"1min":0.39500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.91000000000000003,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.52000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.002,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.76900000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.66900000000000004,"15min":0.64300000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84499999999999997,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65900000000000003,"15min":0.63200000000000001},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86899999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.64100000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.64300000000000002,"15min":0.66700000000000004},"min":{"1min":0.46000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.48099999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.94899999999999995,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.83599999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.69799999999999995,"15min":0.70099999999999996},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.024,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.51700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.89300000000000002,"5min":3.742,"15min":3.742},"last":0.748}]},{"osd":4,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.61199999999999999,"15min":0.624},"min":{"1min":0.40300000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.66800000000000004,"15min":0.65300000000000002},"min":{"1min":0.46000000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.86499999999999999,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.60099999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.71499999999999997,"15min":0.72999999999999998},"min":{"1min":0.44400000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0269999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.626},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.71099999999999997,"15min":0.72699999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89600000000000002,"5min":3.609,"15min":3.609},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:12:05 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76500000000000001,"15min":0.75700000000000001},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.98599999999999999,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.81000000000000005,"15min":0.80900000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.103,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.68700000000000006}]}]},{"osd":5,"up_from":13,"seq":55834574911,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40236,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331604,"statfs":{"total":96636764160,"available":96595562496,"internally_reserved":0,"allocated":942080,"data_stored":400396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":2.391,"5min":1,"15min":0.78200000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":38.005000000000003,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.749},{"interface":"front","average":{"1min":1.645,"5min":0.879,"15min":0.77300000000000002},"min":{"1min":0.45200000000000001,"5min":0.437,"15min":0.437},"max":{"1min":21.376999999999999,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.92300000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":0.83299999999999996,"15min":0.73899999999999999},"min":{"1min":0.45200000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":17.231000000000002,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":1,"5min":0.72699999999999998,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":7.7370000000000001,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5069999999999999,"5min":0.85899999999999999,"15min":0.76700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":17.268000000000001,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.88},{"interface":"front","average":{"1min":1.135,"5min":0.79800000000000004,"15min":0.76200000000000001},"min":{"1min":0.46500000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":9.6549999999999994,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.79300000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":6.0899999999999999,"5min":1.8069999999999999,"15min":1.1120000000000001},"min":{"1min":0.47299999999999998,"5min":0.435,"15min":0.435},"max":{"1min":113.363,"5min":113.363,"15min":113.363},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.77900000000000003,"15min":0.77600000000000002},"min":{"1min":0.52900000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":4.1180000000000003,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.85199999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.628,"15min":0.626},"min":{"1min":0.434,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0600000000000001,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.72099999999999997,"15min":0.73099999999999998},"min":{"1min":0.48799999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.56699999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.72099999999999997,"15min":0.70699999999999996},"min":{"1min":0.47199999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":2.056,"5min":2.056,"15min":2.056},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.74099999999999999,"15min":0.72699999999999998},"min":{"1min":0.51200000000000001,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.1500000000000004,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.78600000000000003,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":4.1029999999999998,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.77900000000000003,"15min":0.76600000000000001},"min":{"1min":0.56599999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":2.02,"5min":2.02,"15min":2.02},"last":0.73199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574910,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":62376,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94309464,"statfs":{"total":96636764160,"available":96572891136,"internally_reserved":0,"allocated":942080,"data_stored":400376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.63200000000000001,"15min":0.63800000000000001},"min":{"1min":0.438,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81200000000000006,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.74199999999999999},"min":{"1min":0.50600000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.91200000000000003,"5min":1.341,"15min":1.341},"last":0.76400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.64400000000000002,"15min":0.66200000000000003},"min":{"1min":0.45900000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80500000000000005,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.629,"15min":0.61599999999999999},"min":{"1min":0.45400000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.80400000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.64000000000000001,"15min":0.64400000000000002},"min":{"1min":0.46700000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.86699999999999999,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.70499999999999996,"15min":0.72199999999999998},"min":{"1min":0.51600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.83499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65900000000000003,"15min":0.66100000000000003},"min":{"1min":0.47399999999999998,"5min":0.434,"15min":0.434},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.51100000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.96999999999999997,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.626,"15min":0.64700000000000002},"min":{"1min":0.436,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91600000000000004,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.375,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.095,"5min":1.095,"15min":1.095},"last":0.60799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.64400000000000002,"15min":0.65200000000000002},"min":{"1min":0.32500000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.90400000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71999999999999997,"15min":0.748},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.01,"5min":1.413,"15min":1.413},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69099999999999995,"15min":0.68899999999999995},"min":{"1min":0.53900000000000003,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70899999999999996,"15min":0.72299999999999998},"min":{"1min":0.40500000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.04,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.72299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321740,"statfs":{"total":96636764160,"available":96585461760,"internally_reserved":0,"allocated":946176,"data_stored":400418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.44900000000000001,"15min":0.45900000000000002},"min":{"1min":0.27800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.79400000000000004,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.52700000000000002,"15min":0.57699999999999996},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.92400000000000004,"5min":2.96,"15min":2.96},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.54400000000000004,"15min":0.52400000000000002},"min":{"1min":0.33700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.95699999999999996,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.68799999999999994,"15min":0.74199999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93799999999999994,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.44500000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83399999999999996,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.59199999999999997,"15min":0.60599999999999998},"min":{"1min":0.39700000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.80400000000000005,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.49199999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.78300000000000003,"15min":0.82299999999999995},"min":{"1min":0.56499999999999995,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.093,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77500000000000002,"15min":0.78000000000000003},"min":{"1min":0.54200000000000004,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.123,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.98199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78600000000000003,"15min":0.79300000000000004},"min":{"1min":0.56599999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.2529999999999999,"5min":1.391,"15min":1.391},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.81799999999999995,"15min":0.84799999999999998},"min":{"1min":0.52200000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.173,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.75700000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.78900000000000003,"15min":0.77000000000000002},"min":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.1579999999999999,"5min":1.401,"15min":1.401},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1850000000000001,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.89200000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.84299999999999997,"15min":0.873},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.139,"5min":2.899,"15min":2.899},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.82699999999999996,"15min":0.83499999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1220000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.879}]}]},{"osd":2,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40236,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331604,"statfs":{"total":96636764160,"available":96595562496,"internally_reserved":0,"allocated":946176,"data_stored":400851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.26400000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.57799999999999996,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.442,"15min":0.42899999999999999},"min":{"1min":0.28499999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.313}]},{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":3.0699999999999998,"15min":1.3400000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58599999999999997,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":3.1120000000000001,"15min":1.353},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.59399999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.38500000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49099999999999999,"15min":0.48299999999999998},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65600000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57299999999999995,"15min":0.57199999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.72799999999999998,"5min":1.121,"15min":1.121},"last":0.42799999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.753,"15min":0.75800000000000001},"min":{"1min":0.56999999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.2450000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.76900000000000002,"15min":0.78300000000000003},"min":{"1min":0.60599999999999998,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.397,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.76400000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.77600000000000002,"15min":0.79000000000000004},"min":{"1min":0.60099999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.048,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.76800000000000002,"15min":0.77600000000000002},"min":{"1min":0.57799999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.99299999999999999,"5min":1.569,"15min":1.569},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.76400000000000001},"min":{"1min":0.51900000000000002,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.0940000000000001,"5min":1.413,"15min":1.413},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.55400000000000005,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.024,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.78300000000000003,"15min":0.77900000000000003},"min":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.073,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.79000000000000004,"15min":0.80200000000000005},"min":{"1min":0.56699999999999995,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.75}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46004,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325836,"statfs":{"total":96636764160,"available":96589656064,"internally_reserved":0,"allocated":958464,"data_stored":402486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.51600000000000001,"15min":0.50800000000000001},"min":{"1min":0.28899999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":5.4939999999999998,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.434,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":6.7240000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.501,"15min":0.47299999999999998},"min":{"1min":0.34000000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.40699999999999997,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.5089999999999999,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.46700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":0.79500000000000004,"15min":0.70099999999999996},"min":{"1min":0.52700000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":16.382999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.65100000000000002},{"interface":"front","average":{"1min":1.3029999999999999,"5min":0.748,"15min":0.62},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":18.318000000000001,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.60999999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.84199999999999997,"15min":0.81999999999999995},"min":{"1min":0.51700000000000002,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":6.1630000000000003,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.57999999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.429,"5min":1.78,"15min":1.78},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.56599999999999995,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.6020000000000001,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.84199999999999997,"15min":0.80900000000000005},"min":{"1min":0.55900000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":6.1189999999999998,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.84499999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.82799999999999996,"15min":0.82099999999999995},"min":{"1min":0.55900000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.6599999999999999,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.86199999999999999,"15min":0.82999999999999996},"min":{"1min":0.51900000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":6.1970000000000001,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.92600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.82899999999999996,"15min":0.83799999999999997},"min":{"1min":0.61699999999999999,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.472,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.84899999999999998,"15min":0.83299999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.544,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.95799999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40800,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331040,"statfs":{"total":96636764160,"available":96594984960,"internally_reserved":0,"allocated":1527808,"data_stored":989655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.58199999999999996,"15min":0.72399999999999998},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.99099999999999999,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.312},{"interface":"front","average":{"1min":0.495,"5min":0.59199999999999997,"15min":0.70999999999999996},"min":{"1min":0.29999999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.1859999999999999,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.40699999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.621,"15min":0.745},"min":{"1min":0.34599999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.66500000000000004,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.375},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.64200000000000002,"15min":0.76400000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.2190000000000001,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.48099999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.60099999999999998,"15min":0.69899999999999995},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.5069999999999999,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65700000000000003,"15min":0.748},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.1000000000000001,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.44900000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.84699999999999998,"15min":0.92600000000000005},"min":{"1min":0.52800000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.3160000000000001,"5min":4.734,"15min":4.734},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.84899999999999998,"15min":0.94299999999999995},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.264,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.78100000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.85199999999999998,"15min":0.95999999999999996},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.175,"5min":4.944,"15min":4.944},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.85399999999999998,"15min":0.93999999999999995},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2050000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.79800000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.94399999999999995,"15min":1.125},"min":{"1min":0.57299999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1439999999999999,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.875,"15min":0.96899999999999997},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.622}]},{"osd":7,"last update":"Fri Sep 13 17:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89800000000000002,"15min":1.0069999999999999},"min":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3069999999999999,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.88500000000000001,"15min":0.98699999999999999},"min":{"1min":0.54400000000000004,"5min":0.505,"15min":0.505},"max":{"1min":1.4530000000000001,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.68000000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:12:59.143 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:12:59.143 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:12:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.340+0000 7f8d95400640 1 Processor -- start 2024-09-13T17:12:59.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.340+0000 7f8d95400640 1 -- start start 2024-09-13T17:12:59.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.341+0000 7f8d95400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d9015a780 0x7f8d90070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.341+0000 7f8d95400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9006c660 con 0x7f8d9015ab60 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.341+0000 7f8d95400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9006c8b0 con 0x7f8d9006c250 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.341+0000 7f8d95400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9006cb90 con 0x7f8d9015a780 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8d9015ab60 0x7f8d9006bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57088/0 (socket says 172.21.15.28:57088) 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8de00640 1 -- 172.21.15.28:0/1533524571 learned_addr learned my addr 172.21.15.28:0/1533524571 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8e800640 1 --2- 172.21.15.28:0/1533524571 >> v2:172.21.15.189:3300/0 conn(0x7f8d9015a780 0x7f8d90070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524024624 0 0) 0x7f8d9006c660 con 0x7f8d9015ab60 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d74003520 con 0x7f8d9015ab60 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3791884664 0 0) 0x7f8d74003520 con 0x7f8d9015ab60 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 >> v2:172.21.15.189:3300/0 conn(0x7f8d9015a780 msgr2=0x7f8d90070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 --2- 172.21.15.28:0/1533524571 >> v2:172.21.15.189:3300/0 conn(0x7f8d9015a780 0x7f8d90070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 >> v1:172.21.15.189:6789/0 conn(0x7f8d9006c250 legacy=0x7f8d900cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900cc980 con 0x7f8d9015ab60 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.342+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f8d7c001670 con 0x7f8d9015ab60 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 >> v1:172.21.15.28:6789/0 conn(0x7f8d9015ab60 legacy=0x7f8d9006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d74003970 0x7f8d74003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 0x7f8d74004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74005040 0x7f8d7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9006cb90 con 0x7f8d74004370 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7400bfa0 con 0x7f8d74005040 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7400c160 con 0x7f8d74003970 2024-09-13T17:12:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 0x7f8d74004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8e800640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d74003970 0x7f8d74003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8f200640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74005040 0x7f8d7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 -- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d74003970 msgr2=0x7f8d74003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d74003970 0x7f8d74003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 -- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74005040 msgr2=0x7f8d7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74005040 0x7f8d7400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8de00640 1 -- 172.21.15.28:0/1533524571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d74003520 con 0x7f8d74004370 2024-09-13T17:12:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.343+0000 7f8d8f200640 1 -- 172.21.15.28:0/1533524571 reap_dead start 2024-09-13T17:12:59.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.344+0000 7f8d8de00640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 0x7f8d74004840 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00c190 tx=0x7f8d740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7d2a03f4ae64f23 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c005ad0 con 0x7f8d74004370 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c02d070 con 0x7f8d74004370 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d8d400640 1 -- 172.21.15.28:0/1533524571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c007070 con 0x7f8d74004370 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 -- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 msgr2=0x7f8d74004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 0x7f8d74004840 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c00c190 tx=0x7f8d740057f0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 -- 172.21.15.28:0/1533524571 shutdown_connections 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 --2- 172.21.15.28:0/1533524571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d74004370 0x7f8d74004840 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 -- 172.21.15.28:0/1533524571 >> 172.21.15.28:0/1533524571 conn(0x7f8d900b0270 msgr2=0x7f8d900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.345+0000 7f8d95400640 1 -- 172.21.15.28:0/1533524571 shutdown_connections 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.346+0000 7f8d95400640 1 -- 172.21.15.28:0/1533524571 wait complete. 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.347+0000 7f8d95400640 1 Processor -- start 2024-09-13T17:12:59.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.347+0000 7f8d95400640 1 -- start start 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74004370 0x7f8d901438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d900c68d0 0x7f8d900c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90143e00 con 0x7f8d900c97c0 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900c9c70 con 0x7f8d74004370 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d95400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90144220 con 0x7f8d900c68d0 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.348+0000 7f8d8f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57296/0 (socket says 172.21.15.28:57296) 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 -- 172.21.15.28:0/2738509488 learned_addr learned my addr 172.21.15.28:0/2738509488 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8e800640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74004370 0x7f8d901438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8de00640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d900c68d0 0x7f8d900c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 -- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d900c68d0 msgr2=0x7f8d900c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d900c68d0 0x7f8d900c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 -- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74004370 msgr2=0x7f8d901438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74004370 0x7f8d901438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 -- 172.21.15.28:0/2738509488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90144500 con 0x7f8d900c97c0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d8f200640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b12e0 tx=0x7f8d8800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.349+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88006e20 con 0x7f8d900c97c0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.350+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d88003970 con 0x7f8d900c97c0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.350+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88003d20 con 0x7f8d900c97c0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.350+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d901447e0 con 0x7f8d900c97c0 2024-09-13T17:12:59.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.350+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900c5860 con 0x7f8d900c97c0 2024-09-13T17:12:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.351+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d90144d00 con 0x7f8d900c97c0 2024-09-13T17:12:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.352+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d88005070 con 0x7f8d900c97c0 2024-09-13T17:12:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.352+0000 7f8d86a00640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 0x7f8d7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.353+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d88088870 con 0x7f8d900c97c0 2024-09-13T17:12:59.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.353+0000 7f8d8e800640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 0x7f8d7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.354+0000 7f8d8e800640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 0x7f8d7006d0f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8d90067520 tx=0x7f8d80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.357+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8d90144d00 con 0x7f8d900c97c0 2024-09-13T17:12:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.601+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8d90144d00 con 0x7f8d900c97c0 2024-09-13T17:12:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.605+0000 7f8d86a00640 1 -- 172.21.15.28:0/2738509488 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8d90144d00 con 0x7f8d900c97c0 2024-09-13T17:12:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:12:59.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 msgr2=0x7f8d7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 0x7f8d7006d0f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8d90067520 tx=0x7f8d80008040 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 msgr2=0x7f8d900c5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b12e0 tx=0x7f8d8800ff70 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 shutdown_connections 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d7006ac30 0x7f8d7006d0f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d900c97c0 0x7f8d900c5320 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d900c68d0 0x7f8d900c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 --2- 172.21.15.28:0/2738509488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d74004370 0x7f8d901438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 >> 172.21.15.28:0/2738509488 conn(0x7f8d900b0270 msgr2=0x7f8d900b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.607+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 shutdown_connections 2024-09-13T17:12:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.608+0000 7f8d95400640 1 -- 172.21.15.28:0/2738509488 wait complete. 2024-09-13T17:12:59.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:12:59.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.777+0000 7f71efe00640 1 Processor -- start 2024-09-13T17:12:59.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.778+0000 7f71efe00640 1 -- start start 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f006b360 0x7f71f00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f00b0d80 con 0x7f71f00b0980 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71efe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f006b900 con 0x7f71f0154a90 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71efe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f006bbe0 con 0x7f71f006b360 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f006b360 0x7f71f00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71e6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f71f006b360 0x7f71f00b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57304/0 (socket says 172.21.15.28:57304) 2024-09-13T17:12:59.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.779+0000 7f71e6000640 1 -- 172.21.15.28:0/105007278 learned_addr learned my addr 172.21.15.28:0/105007278 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.780+0000 7f71e6000640 1 -- 172.21.15.28:0/105007278 >> v1:172.21.15.189:6789/0 conn(0x7f71f0154a90 legacy=0x7f71f015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:12:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.780+0000 7f71e6000640 1 -- 172.21.15.28:0/105007278 >> v1:172.21.15.28:6789/0 conn(0x7f71f00b0980 legacy=0x7f71f00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.780+0000 7f71e6000640 1 -- 172.21.15.28:0/105007278 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f006bec0 con 0x7f71f006b360 2024-09-13T17:12:59.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71e6000640 1 --2- 172.21.15.28:0/105007278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f006b360 0x7f71f00b0440 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009870 tx=0x7f71d802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58475380048f5b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71ee000640 1 -- 172.21.15.28:0/105007278 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8032440 con 0x7f71f006b360 2024-09-13T17:12:59.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71ee000640 1 -- 172.21.15.28:0/105007278 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d80325a0 con 0x7f71f006b360 2024-09-13T17:12:59.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71ee000640 1 -- 172.21.15.28:0/105007278 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8032930 con 0x7f71f006b360 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71efe00640 1 -- 172.21.15.28:0/105007278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f006b360 msgr2=0x7f71f00b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.781+0000 7f71efe00640 1 --2- 172.21.15.28:0/105007278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f006b360 0x7f71f00b0440 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009870 tx=0x7f71d802a2a0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.782+0000 7f71efe00640 1 -- 172.21.15.28:0/105007278 shutdown_connections 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.782+0000 7f71efe00640 1 --2- 172.21.15.28:0/105007278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f006b360 0x7f71f00b0440 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.782+0000 7f71efe00640 1 -- 172.21.15.28:0/105007278 >> 172.21.15.28:0/105007278 conn(0x7f71f0072e40 msgr2=0x7f71f0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.782+0000 7f71efe00640 1 -- 172.21.15.28:0/105007278 shutdown_connections 2024-09-13T17:12:59.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.782+0000 7f71efe00640 1 -- 172.21.15.28:0/105007278 wait complete. 2024-09-13T17:12:59.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.783+0000 7f71efe00640 1 Processor -- start 2024-09-13T17:12:59.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.783+0000 7f71efe00640 1 -- start start 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71f006b360 0x7f71f013cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 0x7f71f00a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f12c0 con 0x7f71f00b0980 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f0990 con 0x7f71f006b360 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71efe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f01f0c70 con 0x7f71f0154a90 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.784+0000 7f71e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57308/0 (socket says 172.21.15.28:57308) 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 -- 172.21.15.28:0/3323314526 learned_addr learned my addr 172.21.15.28:0/3323314526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:12:59.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71eea00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71f006b360 0x7f71f013cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71ef400640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 0x7f71f00a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 -- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 msgr2=0x7f71f00a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 0x7f71f00a99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 -- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71f006b360 msgr2=0x7f71f013cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71f006b360 0x7f71f013cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 -- 172.21.15.28:0/3323314526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d80094d0 con 0x7f71f00b0980 2024-09-13T17:12:59.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71ef400640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 0x7f71f00a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.785+0000 7f71e6000640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009870 tx=0x7f71d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.786+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d8030b00 con 0x7f71f00b0980 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.786+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d8030c60 con 0x7f71f00b0980 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.786+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f01f0f50 con 0x7f71f00b0980 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.786+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d803d420 con 0x7f71f00b0980 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.786+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f00a9ee0 con 0x7f71f00b0980 2024-09-13T17:12:59.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.787+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f004e210 con 0x7f71f00b0980 2024-09-13T17:12:59.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.788+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f71d8039040 con 0x7f71f00b0980 2024-09-13T17:12:59.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.789+0000 7f71e7400640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 0x7f71cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:12:59.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.789+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71d80aa100 con 0x7f71f00b0980 2024-09-13T17:12:59.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.789+0000 7f71eea00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 0x7f71cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:12:59.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.790+0000 7f71eea00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 0x7f71cc06d0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:12:59.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:12:59.793+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f71d8075360 con 0x7f71f00b0980 2024-09-13T17:13:00.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.005+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71f00b0440 con 0x7f71f00b0980 2024-09-13T17:13:00.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.006+0000 7f71e7400640 1 -- 172.21.15.28:0/3323314526 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f71d8078980 con 0x7f71f00b0980 2024-09-13T17:13:00.008 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:00.008 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1620029613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1620029613},{"type":"v1","addr":"172.21.15.189:6835","nonce":1620029613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6649,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3274497119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3274497119},{"type":"v1","addr":"172.21.15.28:6835","nonce":3274497119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6658,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2973808639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2973808639},{"type":"v1","addr":"172.21.15.189:6837","nonce":2973808639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:12:57.545680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3037911251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3037911251},{"type":"v1","addr":"172.21.15.28:6834","nonce":3037911251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:00.008 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 64 2024-09-13T17:13:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 msgr2=0x7f71cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 0x7f71cc06d0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f71e0006fd0 tx=0x7f71e0008040 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 msgr2=0x7f71f00a9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f71d8009870 tx=0x7f71d80046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 shutdown_connections 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f71cc06ac00 0x7f71cc06d0c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f71f0154a90 0x7f71f00a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f71f00b0980 0x7f71f00a9460 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 --2- 172.21.15.28:0/3323314526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f71f006b360 0x7f71f013cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 >> 172.21.15.28:0/3323314526 conn(0x7f71f0072e40 msgr2=0x7f71f0158fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 shutdown_connections 2024-09-13T17:13:00.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.008+0000 7f71efe00640 1 -- 172.21.15.28:0/3323314526 wait complete. 2024-09-13T17:13:00.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:54.534325+0000', 'modified': '2024-09-13T17:12:57.545680+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6954}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6954': {'gid': 6954, 'name': 'a', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6834/3037911251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3037911251}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 3037911251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:13:00.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:13:00.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:13:00.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.586+0000 7f42f3200640 1 Processor -- start 2024-09-13T17:13:00.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.587+0000 7f42f3200640 1 -- start start 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f3200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 0x7f42ec15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42ec06c370 0x7f42ec06c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0b2090 con 0x7f42ec0c40a0 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec06cd20 con 0x7f42ec06c370 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f3200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec0b2370 con 0x7f42ec15ab70 2024-09-13T17:13:00.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42ec06c370 0x7f42ec06c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42ec06c370 0x7f42ec06c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57314/0 (socket says 172.21.15.28:57314) 2024-09-13T17:13:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f2800640 1 -- 172.21.15.28:0/4195988406 learned_addr learned my addr 172.21.15.28:0/4195988406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.588+0000 7f42f1400640 1 --2- 172.21.15.28:0/4195988406 >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 0x7f42ec15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.589+0000 7f42f2800640 1 -- 172.21.15.28:0/4195988406 >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 msgr2=0x7f42ec15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.589+0000 7f42f2800640 1 --2- 172.21.15.28:0/4195988406 >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 0x7f42ec15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.589+0000 7f42f2800640 1 -- 172.21.15.28:0/4195988406 >> v1:172.21.15.189:6789/0 conn(0x7f42ec0c40a0 legacy=0x7f42ec0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.589+0000 7f42f2800640 1 -- 172.21.15.28:0/4195988406 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0b2650 con 0x7f42ec06c370 2024-09-13T17:13:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.589+0000 7f42f1400640 1 --2- 172.21.15.28:0/4195988406 >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 0x7f42ec15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.590+0000 7f42f2800640 1 --2- 172.21.15.28:0/4195988406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec06c370 0x7f42ec06c7e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f42ec144500 tx=0x7f42e802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80820f7cb3877ed9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.590+0000 7f42f0a00640 1 -- 172.21.15.28:0/4195988406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8031600 con 0x7f42ec06c370 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.590+0000 7f42f0a00640 1 -- 172.21.15.28:0/4195988406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e802d050 con 0x7f42ec06c370 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.590+0000 7f42f0a00640 1 -- 172.21.15.28:0/4195988406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8031980 con 0x7f42ec06c370 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 -- 172.21.15.28:0/4195988406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec06c370 msgr2=0x7f42ec06c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 --2- 172.21.15.28:0/4195988406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec06c370 0x7f42ec06c7e0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f42ec144500 tx=0x7f42e802a730 comp rx=0 tx=0).stop 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 -- 172.21.15.28:0/4195988406 shutdown_connections 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 --2- 172.21.15.28:0/4195988406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec06c370 0x7f42ec06c7e0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 --2- 172.21.15.28:0/4195988406 >> v2:172.21.15.189:3300/0 conn(0x7f42ec15ab70 0x7f42ec15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 -- 172.21.15.28:0/4195988406 >> 172.21.15.28:0/4195988406 conn(0x7f42ec073470 msgr2=0x7f42ec069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 -- 172.21.15.28:0/4195988406 shutdown_connections 2024-09-13T17:13:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.591+0000 7f42f3200640 1 -- 172.21.15.28:0/4195988406 wait complete. 2024-09-13T17:13:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.592+0000 7f42f3200640 1 Processor -- start 2024-09-13T17:13:00.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 -- start start 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42ec06c370 0x7f42ec13c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42ec15ab70 0x7f42ec0cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0cba40 con 0x7f42ec0c40a0 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec13cde0 con 0x7f42ec15ab70 2024-09-13T17:13:00.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.593+0000 7f42f3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0ca9a0 con 0x7f42ec06c370 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57324/0 (socket says 172.21.15.28:57324) 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 -- 172.21.15.28:0/328600731 learned_addr learned my addr 172.21.15.28:0/328600731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1e00640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42ec06c370 0x7f42ec13c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f2800640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42ec15ab70 0x7f42ec0cc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 -- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42ec06c370 msgr2=0x7f42ec13c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42ec06c370 0x7f42ec13c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 -- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42ec15ab70 msgr2=0x7f42ec0cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42ec15ab70 0x7f42ec0cc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 -- 172.21.15.28:0/328600731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e8009b30 con 0x7f42ec0c40a0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.594+0000 7f42f1400640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00d7d0 tx=0x7f42dc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.595+0000 7f42e6000640 1 -- 172.21.15.28:0/328600731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc004030 con 0x7f42ec0c40a0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.595+0000 7f42e6000640 1 -- 172.21.15.28:0/328600731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc004190 con 0x7f42ec0c40a0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.595+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec0cac80 con 0x7f42ec0c40a0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.595+0000 7f42e6000640 1 -- 172.21.15.28:0/328600731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc004520 con 0x7f42ec0c40a0 2024-09-13T17:13:00.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.595+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec0cb240 con 0x7f42ec0c40a0 2024-09-13T17:13:00.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.597+0000 7f42e6000640 1 -- 172.21.15.28:0/328600731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f42dc0096a0 con 0x7f42ec0c40a0 2024-09-13T17:13:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.598+0000 7f42e6000640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 0x7f42d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.598+0000 7f42e6000640 1 -- 172.21.15.28:0/328600731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42dc089480 con 0x7f42ec0c40a0 2024-09-13T17:13:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.599+0000 7f42f1e00640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 0x7f42d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.599+0000 7f42f763d740 1 Processor -- start 2024-09-13T17:13:00.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.599+0000 7f42f763d740 1 -- start start 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.599+0000 7f42f1e00640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 0x7f42d006d0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f42e0000f60 tx=0x7f42e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 0x563c84977d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c84951cf0 0x563c84978250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c84893ab0 con 0x563c8492a840 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c84891240 con 0x563c84951cf0 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c8488cb30 con 0x563c84928000 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57328/0 (socket says 172.21.15.28:57328) 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1e00640 1 -- 172.21.15.28:0/2013709123 learned_addr learned my addr 172.21.15.28:0/2013709123 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 0x563c84977d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f2800640 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c84951cf0 0x563c84978250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 -- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c84951cf0 msgr2=0x563c84978250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c84951cf0 0x563c84978250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 -- 172.21.15.28:0/2013709123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 msgr2=0x563c8492ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.600+0000 7f42f1400640 1 -- 172.21.15.28:0/2013709123 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c84857bd0 con 0x563c84928000 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42f1400640 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 0x563c84977d10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f42dc087910 tx=0x7f42dc057da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f235c4d207462549 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42daa00640 1 -- 172.21.15.28:0/2013709123 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc054a40 con 0x563c84928000 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42daa00640 1 -- 172.21.15.28:0/2013709123 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc054ba0 con 0x563c84928000 2024-09-13T17:13:00.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42daa00640 1 -- 172.21.15.28:0/2013709123 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc057a40 con 0x563c84928000 2024-09-13T17:13:00.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42f763d740 1 -- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 msgr2=0x563c84977d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.601+0000 7f42f763d740 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 0x563c84977d10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f42dc087910 tx=0x7f42dc057da0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 -- 172.21.15.28:0/2013709123 shutdown_connections 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c84951cf0 0x563c84978250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84928000 0x563c84977d10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 --2- 172.21.15.28:0/2013709123 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 -- 172.21.15.28:0/2013709123 >> 172.21.15.28:0/2013709123 conn(0x563c84926f70 msgr2=0x563c8496f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 -- 172.21.15.28:0/2013709123 shutdown_connections 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 -- 172.21.15.28:0/2013709123 wait complete. 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.602+0000 7f42f763d740 1 -- start start 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.603+0000 7f42f763d740 1 Processor -- start 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.603+0000 7f42f763d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.603+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c84948950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8494a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f763d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8482e260 con 0x563c8492a840 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8482df00 con 0x563c8494bb70 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c845f3f20 con 0x563c84951cf0 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f763d740 10 client.0 _ll_register_callbacks cb 0x563c848861a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57338/0 (socket says 172.21.15.28:57338) 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 -- 172.21.15.28:0/2963171500 learned_addr learned my addr 172.21.15.28:0/2963171500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1400640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c84948950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f2800640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8494a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 -- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 msgr2=0x563c84948950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c84948950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 -- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 msgr2=0x563c8494a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8494a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 -- 172.21.15.28:0/2963171500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec0cb240 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1400640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c84948950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f2800640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8494a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014700 tx=0x7f42e0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ce78ddeb8b50d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.604+0000 7f42cf400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0013680 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563c848b6cd0 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42e001c040 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42e00137e0 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 10 client.7101 fetch_fsmap learned FSMap version 64 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x563c8488b1b0 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563c848b6cd0 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f42e0012070 con 0x563c8492a840 2024-09-13T17:13:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 10 client.7101 fetch_fsmap finished waiting for FSMap version 64 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 10 client.7101 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 20 client.7101 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8494cb50 0x563c8494d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 4 client.7101 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.605+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 --> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c845803c0 con 0x563c8494cb50 2024-09-13T17:13:00.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.606+0000 7f42f1400640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8494cb50 0x563c8494d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.606+0000 7f42f1400640 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8494cb50 0x563c8494d0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce5083dad6d9238 server_cookie=5ed6cbecd472a163 in_seq=0 out_seq=0 2024-09-13T17:13:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.606+0000 7f42cf400640 10 client.7101 ms_handle_connect on v2:172.21.15.28:6832/3037911251 2024-09-13T17:13:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42cf400640 1 -- 172.21.15.28:0/2963171500 <== mds.0 v2:172.21.15.28:6832/3037911251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x563c845803c0 con 0x563c8494cb50 2024-09-13T17:13:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42cf400640 10 client.7101 handle_command_reply: tid=0 2024-09-13T17:13:00.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 client.7101 shutdown 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 msgr2=0x563c84952210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014700 tx=0x7f42e0001120 comp rx=0 tx=0).stop 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 shutdown_connections 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8494cb50 0x563c8494d0a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8494a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c84948950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 --2- 172.21.15.28:0/2963171500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c84952210 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42cf400640 0 client.7101 ms_handle_reset on v2:172.21.15.28:6832/3037911251 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 >> 172.21.15.28:0/2963171500 conn(0x563c84928000 msgr2=0x563c8496d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.607+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 shutdown_connections 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.608+0000 7f42f763d740 1 -- 172.21.15.28:0/2963171500 wait complete. 2024-09-13T17:13:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.608+0000 7f42f763d740 20 client.7101 trim_cache size 0 max 16384 2024-09-13T17:13:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.623+0000 7f42f763d740 1 Processor -- start 2024-09-13T17:13:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.623+0000 7f42f763d740 1 -- start start 2024-09-13T17:13:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8496d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c8496ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8494ef00 con 0x563c8492a840 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8494c6c0 con 0x563c8494bb70 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c8494aa90 con 0x563c84951cf0 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57350/0 (socket says 172.21.15.28:57350) 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.624+0000 7f42f1e00640 1 -- 172.21.15.28:0/3575520239 learned_addr learned my addr 172.21.15.28:0/3575520239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 -- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 msgr2=0x563c8496ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c8496ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 -- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 msgr2=0x563c8496d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8496d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 -- 172.21.15.28:0/3575520239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c848c0170 con 0x563c8492a840 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42f1e00640 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f42e00146d0 tx=0x7f42e0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=300422f4ff162e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42db400640 1 -- 172.21.15.28:0/3575520239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0013a80 con 0x563c8492a840 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42db400640 1 -- 172.21.15.28:0/3575520239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e0013c90 con 0x563c8492a840 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.625+0000 7f42db400640 1 -- 172.21.15.28:0/3575520239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e003b620 con 0x563c8492a840 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 -- 172.21.15.28:0/3575520239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 msgr2=0x563c8492ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f42e00146d0 tx=0x7f42e0021000 comp rx=0 tx=0).stop 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 -- 172.21.15.28:0/3575520239 shutdown_connections 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c8496ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c8496d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 --2- 172.21.15.28:0/3575520239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8492ac20 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 -- 172.21.15.28:0/3575520239 >> 172.21.15.28:0/3575520239 conn(0x563c84928000 msgr2=0x563c8494b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 -- 172.21.15.28:0/3575520239 shutdown_connections 2024-09-13T17:13:00.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.626+0000 7f42f763d740 1 -- 172.21.15.28:0/3575520239 wait complete. 2024-09-13T17:13:00.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.627+0000 7f42f763d740 1 -- start start 2024-09-13T17:13:00.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.627+0000 7f42f763d740 1 Processor -- start 2024-09-13T17:13:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c84924060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c849245a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c849da170 con 0x563c8492a840 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8496b670 con 0x563c8494bb70 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563c8498cc80 con 0x563c84951cf0 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f763d740 10 client.0 _ll_register_callbacks cb 0x7f42e0007af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57354/0 (socket says 172.21.15.28:57354) 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 -- 172.21.15.28:0/2538402811 learned_addr learned my addr 172.21.15.28:0/2538402811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 -- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 msgr2=0x563c849245a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c849245a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1400640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c84924060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.628+0000 7f42f1e00640 1 -- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 msgr2=0x563c84924060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c84924060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f1e00640 1 -- 172.21.15.28:0/2538402811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c8488b1b0 con 0x563c8492a840 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f1400640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c84924060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f1e00640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014fd0 tx=0x7f42e0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13fc08b6bf50800d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42cf400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e0013680 con 0x563c8492a840 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42e003bbd0 con 0x563c8492a840 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f42dc054ba0 con 0x563c8492a840 2024-09-13T17:13:00.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42e00137e0 con 0x563c8492a840 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f763d740 10 client.7107 fetch_fsmap learned FSMap version 64 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.629+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x7f42ec1f63a0 con 0x563c8492a840 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc054ba0 con 0x563c8492a840 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 64) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f42e0039020 con 0x563c8492a840 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 10 client.7107 fetch_fsmap finished waiting for FSMap version 64 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 10 client.7107 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 20 client.7107 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8496d990 0x563c849d92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 4 client.7107 mds_command: new command op to 6954 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 --> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563c845916c0 con 0x563c8496d990 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.630+0000 7f42f1400640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8496d990 0x563c849d92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f1400640 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8496d990 0x563c849d92c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b706e9a3743538 server_cookie=e499a1acb227bdfc in_seq=0 out_seq=0 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42cf400640 10 client.7107 ms_handle_connect on v2:172.21.15.28:6832/3037911251 2024-09-13T17:13:00.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42cf400640 1 -- 172.21.15.28:0/2538402811 <== mds.0 v2:172.21.15.28:6832/3037911251 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x563c845916c0 con 0x563c8496d990 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42cf400640 10 client.7107 handle_command_reply: tid=0 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 client.7107 shutdown 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 msgr2=0x563c8496c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f42e0014fd0 tx=0x7f42e0014000 comp rx=0 tx=0).stop 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 shutdown_connections 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:6832/3037911251,v1:172.21.15.28:6834/3037911251] conn(0x563c8496d990 0x563c849d92c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563c84951cf0 0x563c849245a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8494bb70 0x563c84924060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 --2- 172.21.15.28:0/2538402811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563c8492a840 0x563c8496c010 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42cf400640 0 client.7107 ms_handle_reset on v2:172.21.15.28:6832/3037911251 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 >> 172.21.15.28:0/2538402811 conn(0x563c84928000 msgr2=0x563c849917d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 shutdown_connections 2024-09-13T17:13:00.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 1 -- 172.21.15.28:0/2538402811 wait complete. 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.631+0000 7f42f763d740 20 client.7107 trim_cache size 0 max 16384 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 6954, 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.0953392060000002, 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 64, 2024-09-13T17:13:00.638 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 77, 2024-09-13T17:13:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 77, 2024-09-13T17:13:00.639 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.228219545000002 2024-09-13T17:13:00.639 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:00.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 msgr2=0x7f42d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 0x7f42d006d0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f42e0000f60 tx=0x7f42e0009040 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 msgr2=0x7f42ec0ca030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00d7d0 tx=0x7f42dc00dc90 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 shutdown_connections 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f42d006ac00 0x7f42d006d0c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42ec15ab70 0x7f42ec0cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42ec0c40a0 0x7f42ec0ca030 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 --2- 172.21.15.28:0/328600731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42ec06c370 0x7f42ec13c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 >> 172.21.15.28:0/328600731 conn(0x7f42ec073470 msgr2=0x7f42ec06fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 shutdown_connections 2024-09-13T17:13:00.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.632+0000 7f42f3200640 1 -- 172.21.15.28:0/328600731 wait complete. 2024-09-13T17:13:00.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:13:00.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.802+0000 7f0dc3c00640 1 Processor -- start 2024-09-13T17:13:00.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.802+0000 7f0dc3c00640 1 -- start start 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc3c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dbc1592a0 0x7f0dbc15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc3c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0dbc06c2d0 0x7f0dbc06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc3c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dbc06cc70 con 0x7f0dbc06f970 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc3c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc15b0b0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc3c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dbc159d90 con 0x7f0dbc06c2d0 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.803+0000 7f0dc1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dbc1592a0 0x7f0dbc15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dbc1592a0 0x7f0dbc15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57364/0 (socket says 172.21.15.28:57364) 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 -- 172.21.15.28:0/2902420438 learned_addr learned my addr 172.21.15.28:0/2902420438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1c00640 1 --2- 172.21.15.28:0/2902420438 >> v2:172.21.15.189:3300/0 conn(0x7f0dbc06c2d0 0x7f0dbc06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 -- 172.21.15.28:0/2902420438 >> v2:172.21.15.189:3300/0 conn(0x7f0dbc06c2d0 msgr2=0x7f0dbc06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 --2- 172.21.15.28:0/2902420438 >> v2:172.21.15.189:3300/0 conn(0x7f0dbc06c2d0 0x7f0dbc06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 -- 172.21.15.28:0/2902420438 >> v1:172.21.15.189:6789/0 conn(0x7f0dbc06f970 legacy=0x7f0dbc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:00.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.804+0000 7f0dc1200640 1 -- 172.21.15.28:0/2902420438 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0b2040 con 0x7f0dbc1592a0 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.805+0000 7f0dc1200640 1 --2- 172.21.15.28:0/2902420438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc15ab70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0b1dd0 tx=0x7f0db002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab2d575faed3dce1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.805+0000 7f0dbb400640 1 -- 172.21.15.28:0/2902420438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0032440 con 0x7f0dbc1592a0 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.805+0000 7f0dbb400640 1 -- 172.21.15.28:0/2902420438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db00325a0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.805+0000 7f0dbb400640 1 -- 172.21.15.28:0/2902420438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0032930 con 0x7f0dbc1592a0 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 -- 172.21.15.28:0/2902420438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 msgr2=0x7f0dbc15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/2902420438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc15ab70 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc0b1dd0 tx=0x7f0db002a230 comp rx=0 tx=0).stop 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 -- 172.21.15.28:0/2902420438 shutdown_connections 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/2902420438 >> v2:172.21.15.189:3300/0 conn(0x7f0dbc06c2d0 0x7f0dbc06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/2902420438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc15ab70 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 -- 172.21.15.28:0/2902420438 >> 172.21.15.28:0/2902420438 conn(0x7f0dbc069e80 msgr2=0x7f0dbc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 -- 172.21.15.28:0/2902420438 shutdown_connections 2024-09-13T17:13:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.806+0000 7f0dc3c00640 1 -- 172.21.15.28:0/2902420438 wait complete. 2024-09-13T17:13:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.808+0000 7f0dc3c00640 1 Processor -- start 2024-09-13T17:13:00.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.808+0000 7f0dc3c00640 1 -- start start 2024-09-13T17:13:00.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dbc06c2d0 0x7f0dbc150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc06f970 0x7f0dbc0b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc151b50 con 0x7f0dbc1592a0 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc151e30 con 0x7f0dbc06f970 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc3c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dbc152110 con 0x7f0dbc06c2d0 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57366/0 (socket says 172.21.15.28:57366) 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 -- 172.21.15.28:0/3627591263 learned_addr learned my addr 172.21.15.28:0/3627591263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1200640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dbc06c2d0 0x7f0dbc150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dbbe00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc06f970 0x7f0dbc0b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 -- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dbc06c2d0 msgr2=0x7f0dbc150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dbc06c2d0 0x7f0dbc150d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 -- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc06f970 msgr2=0x7f0dbc0b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc06f970 0x7f0dbc0b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:00.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.809+0000 7f0dc1c00640 1 -- 172.21.15.28:0/3627591263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00094d0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0dc1c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007e60 tx=0x7f0db400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4014420 con 0x7f0dbc1592a0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db40145a0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db4014950 con 0x7f0dbc1592a0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dbc1523f0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.810+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dbc151790 con 0x7f0dbc1592a0 2024-09-13T17:13:00.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.811+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dbc152910 con 0x7f0dbc1592a0 2024-09-13T17:13:00.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.812+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0db4014ab0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.813+0000 7f0db8c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 0x7f0d9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:00.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.813+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0db4089ba0 con 0x7f0dbc1592a0 2024-09-13T17:13:00.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.813+0000 7f0dc1200640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 0x7f0d9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:00.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.814+0000 7f0dc1200640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 0x7f0d9c06d0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:00.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:00.817+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0dbc152910 con 0x7f0dbc1592a0 2024-09-13T17:13:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.031+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0dbc152910 con 0x7f0dbc1592a0 2024-09-13T17:13:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.031+0000 7f0db8c00640 1 -- 172.21.15.28:0/3627591263 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0dbc152910 con 0x7f0dbc1592a0 2024-09-13T17:13:01.033 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 msgr2=0x7f0d9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 0x7f0d9c06d0c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 msgr2=0x7f0dbc13da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f0db4007e60 tx=0x7f0db400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 shutdown_connections 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d9c06ac00 0x7f0d9c06d0c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dbc1592a0 0x7f0dbc13da30 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dbc06f970 0x7f0dbc0b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 --2- 172.21.15.28:0/3627591263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dbc06c2d0 0x7f0dbc150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.033+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 >> 172.21.15.28:0/3627591263 conn(0x7f0dbc069e80 msgr2=0x7f0dbc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.034+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 shutdown_connections 2024-09-13T17:13:01.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:01.034+0000 7f0dc3c00640 1 -- 172.21.15.28:0/3627591263 wait complete. 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:13:01.051 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:01.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.052 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:13:01.088 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42400sec preferred_lft 42400sec 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:13:01.089 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:13:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:01.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.091 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:13:01.182 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:13:01.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.186 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:13:01.222 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:13:01.223 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:13:01.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.223 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:13:01.223 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:13:01.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:13:01.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:13:01.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:13:01.224 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:13:01.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.369 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:13:01.404 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:13:01.404 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:13:01.404 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:01.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:13:01.500 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:01.500 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:01.535 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:01.535 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:01.536 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:01.611 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:13:01.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:01.740 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:01.741 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:01.767 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:01.767 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:01.767 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:13:01.767 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:13:01.767 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:13:01.769 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:13:01.769 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:13:01.769 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:13:01.769 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:13:01.769 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:13:01.772 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:13:01.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.772 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:01.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.828 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:13:01.873 INFO:teuthology.orchestra.run.smithi028.stdout:client7116 2024-09-13T17:13:01.873 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:01.873 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:01.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:01.950 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir scrub_repair_path) 2024-09-13T17:13:02.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:02.031 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch scrub_repair_path/file) 2024-09-13T17:13:02.108 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:13:02.109 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:13:02.109 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:13:02.109 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:02.109 DEBUG:teuthology.orchestra.run.smithi028:> print(os.stat("/home/ubuntu/cephtest/mnt.0/scrub_repair_path").st_ino) 2024-09-13T17:13:02.109 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:13:02.247 INFO:teuthology.orchestra.run.smithi028.stdout:1099511627776 2024-09-13T17:13:02.434 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:02.434 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:02.510 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:02.510 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:02.587 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.781+0000 7f33aa400640 1 Processor -- start 2024-09-13T17:13:02.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.781+0000 7f33aa400640 1 -- start start 2024-09-13T17:13:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33aa400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a40b21c0 0x7f33a40b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33aa400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a40b0440 0x7f33a40b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33aa400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40cad20 con 0x7f33a415fb90 2024-09-13T17:13:02.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33aa400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a40b0df0 con 0x7f33a40b0440 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33aa400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a406bb40 con 0x7f33a40b21c0 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f33a415fb90 0x7f33a406b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57106/0 (socket says 172.21.15.28:57106) 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9000640 1 -- 172.21.15.28:0/564463515 learned_addr learned my addr 172.21.15.28:0/564463515 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9a00640 1 --2- 172.21.15.28:0/564463515 >> v2:172.21.15.28:3300/0 conn(0x7f33a40b0440 0x7f33a40b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a3e00640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a40b21c0 0x7f33a40b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a3400640 1 -- 172.21.15.28:0/564463515 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970704040 0 0) 0x7f33a40cad20 con 0x7f33a415fb90 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a3400640 1 -- 172.21.15.28:0/564463515 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f338c003520 con 0x7f33a415fb90 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9a00640 1 -- 172.21.15.28:0/564463515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a40b21c0 msgr2=0x7f33a40b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9a00640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a40b21c0 0x7f33a40b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9a00640 1 -- 172.21.15.28:0/564463515 >> v1:172.21.15.28:6789/0 conn(0x7f33a415fb90 legacy=0x7f33a406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.783+0000 7f33a9a00640 1 -- 172.21.15.28:0/564463515 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a406be20 con 0x7f33a40b0440 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33a9a00640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a40b08b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f339c009d90 tx=0x7f339c02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6251b374ba742a45 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33a3400640 1 -- 172.21.15.28:0/564463515 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c007740 con 0x7f33a40b0440 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33a3400640 1 -- 172.21.15.28:0/564463515 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339c02d050 con 0x7f33a40b0440 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33a3400640 1 -- 172.21.15.28:0/564463515 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339c007ac0 con 0x7f33a40b0440 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 -- 172.21.15.28:0/564463515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 msgr2=0x7f33a40b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a40b08b0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f339c009d90 tx=0x7f339c02a530 comp rx=0 tx=0).stop 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 -- 172.21.15.28:0/564463515 shutdown_connections 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a40b08b0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 --2- 172.21.15.28:0/564463515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a40b21c0 0x7f33a40b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.785+0000 7f33aa400640 1 -- 172.21.15.28:0/564463515 >> 172.21.15.28:0/564463515 conn(0x7f33a4072e40 msgr2=0x7f33a4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.786+0000 7f33aa400640 1 -- 172.21.15.28:0/564463515 shutdown_connections 2024-09-13T17:13:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.786+0000 7f33aa400640 1 -- 172.21.15.28:0/564463515 wait complete. 2024-09-13T17:13:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.787+0000 7f33aa400640 1 Processor -- start 2024-09-13T17:13:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.787+0000 7f33aa400640 1 -- start start 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a40b21c0 0x7f33a40aa910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415fb90 0x7f33a415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40a8440 con 0x7f33a40b0440 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40c9080 con 0x7f33a40b21c0 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33aa400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40c9360 con 0x7f33a415fb90 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33a9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33a9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57378/0 (socket says 172.21.15.28:57378) 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.788+0000 7f33a9000640 1 -- 172.21.15.28:0/3308072848 learned_addr learned my addr 172.21.15.28:0/3308072848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9a00640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415fb90 0x7f33a415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a3e00640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a40b21c0 0x7f33a40aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 -- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415fb90 msgr2=0x7f33a415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415fb90 0x7f33a415e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 -- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a40b21c0 msgr2=0x7f33a40aa910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a40b21c0 0x7f33a40aa910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 -- 172.21.15.28:0/3308072848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339c009a70 con 0x7f33a40b0440 2024-09-13T17:13:02.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a9000640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f339800c410 tx=0x7f339800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398005ce0 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3398005e40 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a40c9640 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33980072f0 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.789+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a40aae50 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.791+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a40b08b0 con 0x7f33a40b0440 2024-09-13T17:13:02.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.791+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3398007510 con 0x7f33a40b0440 2024-09-13T17:13:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.792+0000 7f33a0c00640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 0x7f338406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.792+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3398088a60 con 0x7f33a40b0440 2024-09-13T17:13:02.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.793+0000 7f33a3e00640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 0x7f338406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:02.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.793+0000 7f33a3e00640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 0x7f338406d0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f33a41f6700 tx=0x7f3390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:02.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.797+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3398053c60 con 0x7f33a40b0440 2024-09-13T17:13:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.976+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33a40b2fa0 con 0x7f33a40b0440 2024-09-13T17:13:02.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.977+0000 7f33a0c00640 1 -- 172.21.15.28:0/3308072848 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+16568 (secure 0 0 0) 0x7f3398017090 con 0x7f33a40b0440 2024-09-13T17:13:02.979 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:02.980 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":77,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:12:55.537103+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 msgr2=0x7f338406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 0x7f338406d0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f33a41f6700 tx=0x7f3390008040 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 msgr2=0x7f33a41f3210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f339800c410 tx=0x7f339800c8d0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 shutdown_connections 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338406ac00 0x7f338406d0c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415fb90 0x7f33a415e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a40b21c0 0x7f33a40aa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 --2- 172.21.15.28:0/3308072848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a40b0440 0x7f33a41f3210 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 >> 172.21.15.28:0/3308072848 conn(0x7f33a4072e40 msgr2=0x7f33a40cd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 shutdown_connections 2024-09-13T17:13:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:02.979+0000 7f33aa400640 1 -- 172.21.15.28:0/3308072848 wait complete. 2024-09-13T17:13:02.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:12:52.355786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:12:53.525437+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:02.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:03.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.156+0000 7f7a56a00640 1 Processor -- start 2024-09-13T17:13:03.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.157+0000 7f7a56a00640 1 -- start start 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143910 0x7f7a50070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a500cd120 0x7f7a500cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a50071630 con 0x7f7a50143320 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a500cdb10 con 0x7f7a500cd120 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50143700 con 0x7f7a50143910 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a56000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a500cd120 0x7f7a500cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a55600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7a50143320 0x7f7a50071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57122/0 (socket says 172.21.15.28:57122) 2024-09-13T17:13:03.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a55600640 1 -- 172.21.15.28:0/546334808 learned_addr learned my addr 172.21.15.28:0/546334808 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.158+0000 7f7a54c00640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143910 0x7f7a50070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.159+0000 7f7a56000640 1 -- 172.21.15.28:0/546334808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143910 msgr2=0x7f7a50070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.159+0000 7f7a56000640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143910 0x7f7a50070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.159+0000 7f7a56000640 1 -- 172.21.15.28:0/546334808 >> v1:172.21.15.28:6789/0 conn(0x7f7a50143320 legacy=0x7f7a50071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.159+0000 7f7a56000640 1 -- 172.21.15.28:0/546334808 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a50070a20 con 0x7f7a500cd120 2024-09-13T17:13:03.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.159+0000 7f7a4fe00640 1 -- 172.21.15.28:0/546334808 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141225862 0 0) 0x7f7a50071630 con 0x7f7a50143320 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.160+0000 7f7a56000640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a500cd120 0x7f7a500cd5d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009d90 tx=0x7f7a4802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a8cee9aaad861d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.160+0000 7f7a4fe00640 1 -- 172.21.15.28:0/546334808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48007740 con 0x7f7a500cd120 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.160+0000 7f7a4fe00640 1 -- 172.21.15.28:0/546334808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a4802d050 con 0x7f7a500cd120 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.160+0000 7f7a4fe00640 1 -- 172.21.15.28:0/546334808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48007ac0 con 0x7f7a500cd120 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 -- 172.21.15.28:0/546334808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a500cd120 msgr2=0x7f7a500cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a500cd120 0x7f7a500cd5d0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009d90 tx=0x7f7a4802a660 comp rx=0 tx=0).stop 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 -- 172.21.15.28:0/546334808 shutdown_connections 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a500cd120 0x7f7a500cd5d0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 --2- 172.21.15.28:0/546334808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143910 0x7f7a50070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 -- 172.21.15.28:0/546334808 >> 172.21.15.28:0/546334808 conn(0x7f7a50069e80 msgr2=0x7f7a5006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 -- 172.21.15.28:0/546334808 shutdown_connections 2024-09-13T17:13:03.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.161+0000 7f7a56a00640 1 -- 172.21.15.28:0/546334808 wait complete. 2024-09-13T17:13:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.162+0000 7f7a56a00640 1 Processor -- start 2024-09-13T17:13:03.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 -- start start 2024-09-13T17:13:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a500cd120 0x7f7a50159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143320 0x7f7a5015cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500c9ff0 con 0x7f7a50143910 2024-09-13T17:13:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500c9470 con 0x7f7a500cd120 2024-09-13T17:13:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.163+0000 7f7a56a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a500c9750 con 0x7f7a50143320 2024-09-13T17:13:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57388/0 (socket says 172.21.15.28:57388) 2024-09-13T17:13:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 -- 172.21.15.28:0/1722995779 learned_addr learned my addr 172.21.15.28:0/1722995779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a55600640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a500cd120 0x7f7a50159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a54c00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143320 0x7f7a5015cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 -- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143320 msgr2=0x7f7a5015cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143320 0x7f7a5015cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 -- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a500cd120 msgr2=0x7f7a50159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a500cd120 0x7f7a50159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 -- 172.21.15.28:0/1722995779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48009a70 con 0x7f7a50143910 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.164+0000 7f7a56000640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009eb0 tx=0x7f7a48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.165+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48031890 con 0x7f7a50143910 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.165+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a480319f0 con 0x7f7a50143910 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.165+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a48031d60 con 0x7f7a50143910 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.165+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a500c9a30 con 0x7f7a50143910 2024-09-13T17:13:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.165+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a5015a4f0 con 0x7f7a50143910 2024-09-13T17:13:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.166+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a500b3800 con 0x7f7a50143910 2024-09-13T17:13:03.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.167+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7a4803b410 con 0x7f7a50143910 2024-09-13T17:13:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.168+0000 7f7a4d600640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 0x7f7a3006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.168+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7a480aab60 con 0x7f7a50143910 2024-09-13T17:13:03.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.168+0000 7f7a55600640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 0x7f7a3006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.169+0000 7f7a55600640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 0x7f7a3006d0f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000f60 tx=0x7f7a44007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.172+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7a48075f00 con 0x7f7a50143910 2024-09-13T17:13:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.352+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7a50143ff0 con 0x7f7a50143910 2024-09-13T17:13:03.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.352+0000 7f7a4d600640 1 -- 172.21.15.28:0/1722995779 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7a480490a0 con 0x7f7a50143910 2024-09-13T17:13:03.354 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 msgr2=0x7f7a3006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 0x7f7a3006d0f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f7a44000f60 tx=0x7f7a44007d50 comp rx=0 tx=0).stop 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 msgr2=0x7f7a5015d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f7a48009eb0 tx=0x7f7a48004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 shutdown_connections 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a3006ac30 0x7f7a3006d0f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a50143910 0x7f7a5015d080 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a50143320 0x7f7a5015cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 --2- 172.21.15.28:0/1722995779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a500cd120 0x7f7a50159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 >> 172.21.15.28:0/1722995779 conn(0x7f7a50069e80 msgr2=0x7f7a5006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 shutdown_connections 2024-09-13T17:13:03.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.355+0000 7f7a56a00640 1 -- 172.21.15.28:0/1722995779 wait complete. 2024-09-13T17:13:03.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:13:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.530+0000 7f9695200640 1 Processor -- start 2024-09-13T17:13:03.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.530+0000 7f9695200640 1 -- start start 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.531+0000 7f9695200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9690159770 0x7f969006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.531+0000 7f9695200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 0x7f9690070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.531+0000 7f9695200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f969006f920 con 0x7f969006f3f0 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.531+0000 7f9695200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f969015b290 con 0x7f9690159770 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.531+0000 7f9695200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969015b570 con 0x7f969015cd80 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9690159770 0x7f969006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9690159770 0x7f969006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57398/0 (socket says 172.21.15.28:57398) 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 -- 172.21.15.28:0/2720945617 learned_addr learned my addr 172.21.15.28:0/2720945617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968e800640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 0x7f9690070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 -- 172.21.15.28:0/2720945617 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 msgr2=0x7f9690070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 0x7f9690070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 -- 172.21.15.28:0/2720945617 >> v1:172.21.15.189:6789/0 conn(0x7f969006f3f0 legacy=0x7f9690159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968de00640 1 -- 172.21.15.28:0/2720945617 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f969015b850 con 0x7f9690159770 2024-09-13T17:13:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.532+0000 7f968e800640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 0x7f9690070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.533+0000 7f968de00640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f969006fd20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f9680009870 tx=0x7f968002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a35e3a0270ee7f57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.533+0000 7f968ca00640 1 -- 172.21.15.28:0/2720945617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680032440 con 0x7f9690159770 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.533+0000 7f968ca00640 1 -- 172.21.15.28:0/2720945617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96800325a0 con 0x7f9690159770 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.533+0000 7f968ca00640 1 -- 172.21.15.28:0/2720945617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680032930 con 0x7f9690159770 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 -- 172.21.15.28:0/2720945617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 msgr2=0x7f969006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f969006fd20 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f9680009870 tx=0x7f968002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 -- 172.21.15.28:0/2720945617 shutdown_connections 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969015cd80 0x7f9690070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 --2- 172.21.15.28:0/2720945617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f969006fd20 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 -- 172.21.15.28:0/2720945617 >> 172.21.15.28:0/2720945617 conn(0x7f96900b0270 msgr2=0x7f96900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 -- 172.21.15.28:0/2720945617 shutdown_connections 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.534+0000 7f9695200640 1 -- 172.21.15.28:0/2720945617 wait complete. 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.536+0000 7f9695200640 1 Processor -- start 2024-09-13T17:13:03.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.536+0000 7f9695200640 1 -- start start 2024-09-13T17:13:03.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 0x7f96900c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969015cd80 0x7f9690155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96900c6210 con 0x7f9690159770 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96900c3de0 con 0x7f969015cd80 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f9695200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96900c40c0 con 0x7f969006f3f0 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 0x7f96900c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969015cd80 0x7f9690155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57402/0 (socket says 172.21.15.28:57402) 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 0x7f96900c34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47222/0 (socket says 172.21.15.28:47222) 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968d400640 1 -- 172.21.15.28:0/112517970 learned_addr learned my addr 172.21.15.28:0/112517970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.537+0000 7f968d400640 1 -- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 msgr2=0x7f96900c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f968d400640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 0x7f96900c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f968d400640 1 -- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969015cd80 msgr2=0x7f9690155c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f968d400640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969015cd80 0x7f9690155c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f968d400640 1 -- 172.21.15.28:0/112517970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800094d0 con 0x7f9690159770 2024-09-13T17:13:03.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f968d400640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c400 tx=0x7f967c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c0052d0 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c005430 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c003830 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96900c43a0 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.538+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96900c4960 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.540+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96900704e0 con 0x7f9690159770 2024-09-13T17:13:03.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.540+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f967c003990 con 0x7f9690159770 2024-09-13T17:13:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.541+0000 7f9686000640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 0x7f967006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.541+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f967c088860 con 0x7f9690159770 2024-09-13T17:13:03.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.542+0000 7f968de00640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 0x7f967006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.543+0000 7f968de00640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 0x7f967006d0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9680034000 tx=0x7f968002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.545+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f967c053ae0 con 0x7f9690159770 2024-09-13T17:13:03.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.721+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f969006c780 con 0x7f9690159770 2024-09-13T17:13:03.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.722+0000 7f9686000640 1 -- 172.21.15.28:0/112517970 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) v1 ==== 104+0+11 (secure 0 0 0) 0x7f967c057100 con 0x7f9690159770 2024-09-13T17:13:03.723 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:13:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 msgr2=0x7f967006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 0x7f967006d0c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f9680034000 tx=0x7f968002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 msgr2=0x7f96900acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c400 tx=0x7f967c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 shutdown_connections 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f967006ac00 0x7f967006d0c0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f969015cd80 0x7f9690155c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9690159770 0x7f96900acac0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 --2- 172.21.15.28:0/112517970 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f969006f3f0 0x7f96900c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 >> 172.21.15.28:0/112517970 conn(0x7f96900b0270 msgr2=0x7f96900cecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 shutdown_connections 2024-09-13T17:13:03.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.724+0000 7f9695200640 1 -- 172.21.15.28:0/112517970 wait complete. 2024-09-13T17:13:03.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:13:03.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.899+0000 7fa5d0a00640 1 Processor -- start 2024-09-13T17:13:03.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.899+0000 7fa5d0a00640 1 -- start start 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5d0a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5cc071490 0x7fa5cc06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5d0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc06c8b0 con 0x7fa5cc15ab60 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5d0a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc06cb90 con 0x7fa5cc15a780 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5d0a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cc980 con 0x7fa5cc071490 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5c1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5cc15ab60 0x7fa5cc06bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53242/0 (socket says 172.21.15.28:53242) 2024-09-13T17:13:03.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.900+0000 7fa5c1400640 1 -- 172.21.15.28:0/1937908094 learned_addr learned my addr 172.21.15.28:0/1937908094 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1937908094 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc071490 0x7fa5cc06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316041621 0 0) 0x7fa5cc06c8b0 con 0x7fa5cc15ab60 2024-09-13T17:13:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b0003520 con 0x7fa5cc15ab60 2024-09-13T17:13:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 955121919 0 0) 0x7fa5b0003520 con 0x7fa5cc15ab60 2024-09-13T17:13:03.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc071490 msgr2=0x7fa5cc06c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 --2- 172.21.15.28:0/1937908094 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc071490 0x7fa5cc06c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 >> v1:172.21.15.189:6789/0 conn(0x7fa5cc15a780 legacy=0x7fa5cc070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc0ccc60 con 0x7fa5cc15ab60 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa5b4001670 con 0x7fa5cc15ab60 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 >> v1:172.21.15.28:6789/0 conn(0x7fa5cc15ab60 legacy=0x7fa5cc06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b0003970 0x7fa5b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b00047c0 0x7fa5b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 0x7fa5b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0cc980 con 0x7fa5b0005180 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b000c020 con 0x7fa5b0003970 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.901+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b000c1e0 con 0x7fa5b00047c0 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 0x7fa5b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5c9e00640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b0003970 0x7fa5b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5c1400640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b00047c0 0x7fa5b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 -- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b00047c0 msgr2=0x7fa5b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b00047c0 0x7fa5b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 -- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b0003970 msgr2=0x7fa5b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b0003970 0x7fa5b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5ca800640 1 -- 172.21.15.28:0/1937908094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b0003520 con 0x7fa5b0005180 2024-09-13T17:13:03.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.902+0000 7fa5c9e00640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5b0003970 0x7fa5b0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.903+0000 7fa5ca800640 1 -- 172.21.15.28:0/1937908094 reap_dead start 2024-09-13T17:13:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.903+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 0x7fa5b000b980 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009ac0 tx=0x7fa5b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e6ad63216aff4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.903+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004020 con 0x7fa5b0005180 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.903+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c402d050 con 0x7fa5b0005180 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.903+0000 7fa5c9400640 1 -- 172.21.15.28:0/1937908094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c40043c0 con 0x7fa5b0005180 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 msgr2=0x7fa5b000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 0x7fa5b000b980 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009ac0 tx=0x7fa5b00057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1937908094 shutdown_connections 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1937908094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5b0005180 0x7fa5b000b980 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1937908094 >> 172.21.15.28:0/1937908094 conn(0x7fa5cc0b0270 msgr2=0x7fa5cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1937908094 shutdown_connections 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.904+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1937908094 wait complete. 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.906+0000 7fa5d0a00640 1 Processor -- start 2024-09-13T17:13:03.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.906+0000 7fa5d0a00640 1 -- start start 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b0005180 0x7fa5cc06f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 0x7fa5cc0a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1538e0 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc1528d0 con 0x7fa5cc0a9080 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5d0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc152b20 con 0x7fa5b0005180 2024-09-13T17:13:03.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41600/0 (socket says 172.21.15.28:41600) 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 -- 172.21.15.28:0/1586180254 learned_addr learned my addr 172.21.15.28:0/1586180254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 -- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b0005180 msgr2=0x7fa5cc06f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b0005180 0x7fa5cc06f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 -- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 msgr2=0x7fa5cc0a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5c1400640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 0x7fa5cc0a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 0x7fa5cc0a9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5ca800640 1 -- 172.21.15.28:0/1586180254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc152e00 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.907+0000 7fa5c1400640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 0x7fa5cc0a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5ca800640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5c402abe0 tx=0x7fa5c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4003c50 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c4003db0 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c403a6b0 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc1530e0 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.908+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc0a9a40 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.909+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc153600 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.910+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa5c403a810 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.911+0000 7fa5c2a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 0x7fa5a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.911+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5c40a94b0 con 0x7fa5cc1f07b0 2024-09-13T17:13:03.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.912+0000 7fa5c9e00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 0x7fa5a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:03.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.912+0000 7fa5c9e00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 0x7fa5a806d0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc000f80 tx=0x7fa5bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:03.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:03.915+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa5cc153600 con 0x7fa5cc1f07b0 2024-09-13T17:13:04.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.093+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa5cc153600 con 0x7fa5cc1f07b0 2024-09-13T17:13:04.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.094+0000 7fa5c2a00640 1 -- 172.21.15.28:0/1586180254 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa5cc153600 con 0x7fa5cc1f07b0 2024-09-13T17:13:04.095 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 msgr2=0x7fa5a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 0x7fa5a806d0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc000f80 tx=0x7fa5bc009040 comp rx=0 tx=0).stop 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 msgr2=0x7fa5cc1f0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7fa5c402abe0 tx=0x7fa5c4004730 comp rx=0 tx=0).stop 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 shutdown_connections 2024-09-13T17:13:04.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac00 0x7fa5a806d0c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc1f07b0 0x7fa5cc1f0c40 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc0a9080 0x7fa5cc0a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/1586180254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5b0005180 0x7fa5cc06f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 >> 172.21.15.28:0/1586180254 conn(0x7fa5cc0b0270 msgr2=0x7fa5cc0af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 shutdown_connections 2024-09-13T17:13:04.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.096+0000 7fa5d0a00640 1 -- 172.21.15.28:0/1586180254 wait complete. 2024-09-13T17:13:04.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:13:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.269+0000 7f86db200640 1 Processor -- start 2024-09-13T17:13:04.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.269+0000 7f86db200640 1 -- start start 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86db200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 0x7f86dc15cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86db200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86dc0b29e0 0x7f86dc0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86db200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc15d6c0 con 0x7f86dc0b2440 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86db200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc15d9a0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86db200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc15dc80 con 0x7f86dc156cf0 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.270+0000 7f86d1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86dc0b29e0 0x7f86dc0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f86dc0b29e0 0x7f86dc0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41614/0 (socket says 172.21.15.28:41614) 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 -- 172.21.15.28:0/1911871832 learned_addr learned my addr 172.21.15.28:0/1911871832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d9e00640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 0x7f86dc15cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 -- 172.21.15.28:0/1911871832 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 msgr2=0x7f86dc15cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 0x7f86dc15cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 -- 172.21.15.28:0/1911871832 >> v1:172.21.15.189:6789/0 conn(0x7f86dc0b2440 legacy=0x7f86dc06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d1400640 1 -- 172.21.15.28:0/1911871832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0b1700 con 0x7f86dc0b29e0 2024-09-13T17:13:04.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.271+0000 7f86d9e00640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 0x7f86dc15cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.272+0000 7f86d1400640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc0b1f00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009870 tx=0x7f86c402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88db06afa3e594 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.272+0000 7f86d9400640 1 -- 172.21.15.28:0/1911871832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4032440 con 0x7f86dc0b29e0 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.272+0000 7f86d9400640 1 -- 172.21.15.28:0/1911871832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c40325a0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.272+0000 7f86d9400640 1 -- 172.21.15.28:0/1911871832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c4032930 con 0x7f86dc0b29e0 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 -- 172.21.15.28:0/1911871832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 msgr2=0x7f86dc0b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc0b1f00 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f86c4009870 tx=0x7f86c402a250 comp rx=0 tx=0).stop 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 -- 172.21.15.28:0/1911871832 shutdown_connections 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc0b1f00 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 --2- 172.21.15.28:0/1911871832 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc156cf0 0x7f86dc15cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 -- 172.21.15.28:0/1911871832 >> 172.21.15.28:0/1911871832 conn(0x7f86dc073470 msgr2=0x7f86dc069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 -- 172.21.15.28:0/1911871832 shutdown_connections 2024-09-13T17:13:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.273+0000 7f86db200640 1 -- 172.21.15.28:0/1911871832 wait complete. 2024-09-13T17:13:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.275+0000 7f86db200640 1 Processor -- start 2024-09-13T17:13:04.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.275+0000 7f86db200640 1 -- start start 2024-09-13T17:13:04.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86dc156cf0 0x7f86dc0a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc158150 0x7f86dc0c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc157cc0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc150580 con 0x7f86dc156cf0 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86db200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0a8e70 con 0x7f86dc158150 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41624/0 (socket says 172.21.15.28:41624) 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86d9e00640 1 -- 172.21.15.28:0/2223706289 learned_addr learned my addr 172.21.15.28:0/2223706289 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86d1400640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86dc156cf0 0x7f86dc0a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.276+0000 7f86da800640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc158150 0x7f86dc0c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 -- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc158150 msgr2=0x7f86dc0c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc158150 0x7f86dc0c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 -- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86dc156cf0 msgr2=0x7f86dc0a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86dc156cf0 0x7f86dc0a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 -- 172.21.15.28:0/2223706289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c40094d0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d9e00640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00d7d0 tx=0x7f86cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc004030 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc004190 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc004520 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc157390 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.277+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc157950 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.279+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0cb6c0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.279+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f86cc0096a0 con 0x7f86dc0b29e0 2024-09-13T17:13:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.281+0000 7f86d2a00640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 0x7f86b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.281+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f86cc089480 con 0x7f86dc0b29e0 2024-09-13T17:13:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.281+0000 7f86d1400640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 0x7f86b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.282+0000 7f86d1400640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 0x7f86b806d0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f86c4034000 tx=0x7f86c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.285+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f86cc054670 con 0x7f86dc0b29e0 2024-09-13T17:13:04.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.473+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f86dc0cc820 con 0x7f86dc0b29e0 2024-09-13T17:13:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.474+0000 7f86d2a00640 1 -- 172.21.15.28:0/2223706289 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v77) v1 ==== 80+0+0 (secure 0 0 0) 0x7f86cc057c90 con 0x7f86dc0b29e0 2024-09-13T17:13:04.475 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 msgr2=0x7f86b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 0x7f86b806d0c0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f86c4034000 tx=0x7f86c402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 msgr2=0x7f86dc150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00d7d0 tx=0x7f86cc00dc90 comp rx=0 tx=0).stop 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 shutdown_connections 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86b806ac00 0x7f86b806d0c0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f86dc158150 0x7f86dc0c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86dc156cf0 0x7f86dc0a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 --2- 172.21.15.28:0/2223706289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f86dc0b29e0 0x7f86dc150040 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 >> 172.21.15.28:0/2223706289 conn(0x7f86dc073470 msgr2=0x7f86dc0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 shutdown_connections 2024-09-13T17:13:04.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.476+0000 7f86db200640 1 -- 172.21.15.28:0/2223706289 wait complete. 2024-09-13T17:13:04.493 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:13:04.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:04.493 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:04.583 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:13:04.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:04.584 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:13:04.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.822+0000 7efe8b800640 1 Processor -- start 2024-09-13T17:13:04.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.822+0000 7efe8b800640 1 -- start start 2024-09-13T17:13:04.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe8b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe840cabc0 0x7efe8406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe8b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe84070cc0 0x7efe84071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe8b800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8415e100 con 0x7efe8415a540 2024-09-13T17:13:04.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe8b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe840716b0 con 0x7efe84070cc0 2024-09-13T17:13:04.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe8b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840adb00 con 0x7efe840cabc0 2024-09-13T17:13:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe89800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe84070cc0 0x7efe84071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe89800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efe84070cc0 0x7efe84071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41628/0 (socket says 172.21.15.28:41628) 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.823+0000 7efe89800640 1 -- 172.21.15.28:0/1342008820 learned_addr learned my addr 172.21.15.28:0/1342008820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.824+0000 7efe83e00640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe840cabc0 0x7efe8406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.824+0000 7efe89800640 1 -- 172.21.15.28:0/1342008820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe840cabc0 msgr2=0x7efe8406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.824+0000 7efe89800640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe840cabc0 0x7efe8406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.824+0000 7efe89800640 1 -- 172.21.15.28:0/1342008820 >> v1:172.21.15.189:6789/0 conn(0x7efe8415a540 legacy=0x7efe8415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.824+0000 7efe89800640 1 -- 172.21.15.28:0/1342008820 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe8415e3e0 con 0x7efe84070cc0 2024-09-13T17:13:04.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.825+0000 7efe89800640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe84070cc0 0x7efe84071170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009e80 tx=0x7efe7c02a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f59ab57d2b592e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.825+0000 7efe83400640 1 -- 172.21.15.28:0/1342008820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c031600 con 0x7efe84070cc0 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.825+0000 7efe83400640 1 -- 172.21.15.28:0/1342008820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c02d050 con 0x7efe84070cc0 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.825+0000 7efe83400640 1 -- 172.21.15.28:0/1342008820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c031980 con 0x7efe84070cc0 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 -- 172.21.15.28:0/1342008820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe84070cc0 msgr2=0x7efe84071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe84070cc0 0x7efe84071170 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7efe7c009e80 tx=0x7efe7c02a6b0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 -- 172.21.15.28:0/1342008820 shutdown_connections 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe84070cc0 0x7efe84071170 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 --2- 172.21.15.28:0/1342008820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe840cabc0 0x7efe8406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 -- 172.21.15.28:0/1342008820 >> 172.21.15.28:0/1342008820 conn(0x7efe84069e80 msgr2=0x7efe8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 -- 172.21.15.28:0/1342008820 shutdown_connections 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.826+0000 7efe8b800640 1 -- 172.21.15.28:0/1342008820 wait complete. 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.828+0000 7efe8b800640 1 Processor -- start 2024-09-13T17:13:04.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.828+0000 7efe8b800640 1 -- start start 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe84070cc0 0x7efe84148190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe840cabc0 0x7efe8415b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8415bc10 con 0x7efe8415a540 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe8415b240 con 0x7efe840cabc0 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe8b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe841505e0 con 0x7efe84070cc0 2024-09-13T17:13:04.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41638/0 (socket says 172.21.15.28:41638) 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 -- 172.21.15.28:0/52030213 learned_addr learned my addr 172.21.15.28:0/52030213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe88e00640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe84070cc0 0x7efe84148190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe83e00640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe840cabc0 0x7efe8415b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 -- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe84070cc0 msgr2=0x7efe84148190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe84070cc0 0x7efe84148190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 -- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe840cabc0 msgr2=0x7efe8415b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe840cabc0 0x7efe8415b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.829+0000 7efe89800640 1 -- 172.21.15.28:0/52030213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c009b30 con 0x7efe8415a540 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe89800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7efe7c034040 tx=0x7efe7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c03d860 con 0x7efe8415a540 2024-09-13T17:13:04.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe7c02ae90 con 0x7efe8415a540 2024-09-13T17:13:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7c03dc20 con 0x7efe8415a540 2024-09-13T17:13:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe841528e0 con 0x7efe8415a540 2024-09-13T17:13:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.830+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe8415aed0 con 0x7efe8415a540 2024-09-13T17:13:04.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.831+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe8415b3f0 con 0x7efe8415a540 2024-09-13T17:13:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.832+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efe7c046400 con 0x7efe8415a540 2024-09-13T17:13:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.833+0000 7efe80c00640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 0x7efe6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.833+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efe7c0aa350 con 0x7efe8415a540 2024-09-13T17:13:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.833+0000 7efe88e00640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 0x7efe6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:04.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.834+0000 7efe88e00640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 0x7efe6406d0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7efe78000f60 tx=0x7efe78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:04.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:04.837+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efe8415b3f0 con 0x7efe8415a540 2024-09-13T17:13:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.054+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe8415b3b0 con 0x7efe8415a540 2024-09-13T17:13:05.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.054+0000 7efe80c00640 1 -- 172.21.15.28:0/52030213 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4742 (secure 0 0 0) 0x7efe7c075540 con 0x7efe8415a540 2024-09-13T17:13:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:05.056 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":64,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1620029613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1620029613},{"type":"v1","addr":"172.21.15.189:6835","nonce":1620029613}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6649,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3274497119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3274497119},{"type":"v1","addr":"172.21.15.28:6835","nonce":3274497119}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":6658,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2973808639","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2973808639},{"type":"v1","addr":"172.21.15.189:6837","nonce":2973808639}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":59}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:12:57.545680+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6954":{"gid":6954,"name":"a","rank":0,"incarnation":60,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3037911251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3037911251},{"type":"v1","addr":"172.21.15.28:6834","nonce":3037911251}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:05.056 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 64 2024-09-13T17:13:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 msgr2=0x7efe6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 0x7efe6406d0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7efe78000f60 tx=0x7efe78009040 comp rx=0 tx=0).stop 2024-09-13T17:13:05.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 msgr2=0x7efe84150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7efe7c034040 tx=0x7efe7c004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 shutdown_connections 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efe6406ac00 0x7efe6406d0c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe8415a540 0x7efe84150dd0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe840cabc0 0x7efe8415b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 --2- 172.21.15.28:0/52030213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe84070cc0 0x7efe84148190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 >> 172.21.15.28:0/52030213 conn(0x7efe84069e80 msgr2=0x7efe8406fed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 shutdown_connections 2024-09-13T17:13:05.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.056+0000 7efe8b800640 1 -- 172.21.15.28:0/52030213 wait complete. 2024-09-13T17:13:05.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-13T17:13:05.253 INFO:teuthology.orchestra.run.smithi028.stdout:{"message":"","return_code":0} 2024-09-13T17:13:05.254 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T17:13:05.254 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:13:05.254 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:05.255 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:13:05.255 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:05.255 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:13:05.255 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:05.255 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:13:05.255 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:05.271 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:13:05.271 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.272 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:13:05.272 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.272 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:13:05.273 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.286 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:13:05.286 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.287 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000000.00000000 file_head 2024-09-13T17:13:05.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.385+0000 7f473ed22f40 1 Processor -- start 2024-09-13T17:13:05.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.385+0000 7f473ed22f40 1 -- start start 2024-09-13T17:13:05.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd97208a0 0x558cd97ed8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97d8c40 0x558cd97f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd96c7400 con 0x558cd9767100 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd9662fe0 con 0x558cd97d8c40 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ed22f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd9662c80 con 0x558cd97208a0 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41658/0 (socket says 172.21.15.28:41658) 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473b600640 1 -- 172.21.15.28:0/3035567546 learned_addr learned my addr 172.21.15.28:0/3035567546 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473c000640 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd97208a0 0x558cd97ed8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.386+0000 7f473ca00640 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97d8c40 0x558cd97f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 -- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd97208a0 msgr2=0x558cd97ed8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd97208a0 0x558cd97ed8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 -- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97d8c40 msgr2=0x558cd97f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97d8c40 0x558cd97f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 -- 172.21.15.28:0/3035567546 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473400a1c0 con 0x558cd9767100 2024-09-13T17:13:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473b600640 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4728001dd0 tx=0x7f472800b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8e3a9053174c6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473a200640 1 -- 172.21.15.28:0/3035567546 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728014070 con 0x558cd9767100 2024-09-13T17:13:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473a200640 1 -- 172.21.15.28:0/3035567546 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4728008030 con 0x558cd9767100 2024-09-13T17:13:05.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.387+0000 7f473a200640 1 -- 172.21.15.28:0/3035567546 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472800f400 con 0x558cd9767100 2024-09-13T17:13:05.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 -- 172.21.15.28:0/3035567546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 msgr2=0x558cd97f19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4728001dd0 tx=0x7f472800b5b0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 -- 172.21.15.28:0/3035567546 shutdown_connections 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97d8c40 0x558cd97f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd9767100 0x558cd97f19e0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 --2- 172.21.15.28:0/3035567546 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd97208a0 0x558cd97ed8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 -- 172.21.15.28:0/3035567546 >> 172.21.15.28:0/3035567546 conn(0x558cd974b270 msgr2=0x558cd974dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 -- 172.21.15.28:0/3035567546 shutdown_connections 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.388+0000 7f473ed22f40 1 -- 172.21.15.28:0/3035567546 wait complete. 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.389+0000 7f473ed22f40 1 Processor -- start 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.389+0000 7f473ed22f40 1 -- start start 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97208a0 0x558cd97d3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd9767100 0x558cd97efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 0x558cd97493a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd96598f0 con 0x558cd97d8c40 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558cd97d5960 con 0x558cd97208a0 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ed22f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558cd97d5c40 con 0x558cd9767100 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 0x558cd97493a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 0x558cd97493a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41668/0 (socket says 172.21.15.28:41668) 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 -- 172.21.15.28:0/801656476 learned_addr learned my addr 172.21.15.28:0/801656476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473c000640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97208a0 0x558cd97d3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473b600640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd9767100 0x558cd97efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 -- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd9767100 msgr2=0x558cd97efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd9767100 0x558cd97efcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 -- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97208a0 msgr2=0x558cd97d3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558cd97208a0 0x558cd97d3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473ca00640 1 -- 172.21.15.28:0/801656476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472800a040 con 0x558cd97d8c40 2024-09-13T17:13:05.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.390+0000 7f473b600640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558cd9767100 0x558cd97efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f473ca00640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 0x558cd97493a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4734045b80 tx=0x7f4734003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f4738e00640 1 -- 172.21.15.28:0/801656476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734066440 con 0x558cd97d8c40 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f4738e00640 1 -- 172.21.15.28:0/801656476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47340665a0 con 0x558cd97d8c40 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f4738e00640 1 -- 172.21.15.28:0/801656476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4734066910 con 0x558cd97d8c40 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558cd9671b20 con 0x558cd97d8c40 2024-09-13T17:13:05.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.391+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558cd9664f60 con 0x558cd97d8c40 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.393+0000 7f4738e00640 1 -- 172.21.15.28:0/801656476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f473406f440 con 0x558cd97d8c40 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.394+0000 7f4738e00640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471c069610 0x7f471c06bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.394+0000 7f4738e00640 1 -- 172.21.15.28:0/801656476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47340cc860 con 0x558cd97d8c40 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.394+0000 7f473ed22f40 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x558cd97f0200 0x558cd97d4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.394+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 --> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] -- osd_op(unknown.0.0:1 10.10 10:0cf6ad0f:::10000000000.00000000:head [omap-rm-keys in=17b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) v8 -- 0x558cd97d4b80 con 0x558cd97f0200 2024-09-13T17:13:05.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.394+0000 7f473c000640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471c069610 0x7f471c06bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.395+0000 7f473b600640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x558cd97f0200 0x558cd97d4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.395+0000 7f473c000640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471c069610 0x7f471c06bad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f472c005400 tx=0x7f472c005310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.395+0000 7f473b600640 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x558cd97f0200 0x558cd97d4640 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473b600640 1 -- 172.21.15.28:0/801656476 <== osd.4 v2:172.21.15.189:6800/531106775 1 ==== osd_op_reply(1 10000000000.00000000 [omap-rm-keys] v77'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f4728016070 con 0x558cd97f0200 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x558cd97f0200 msgr2=0x558cd97d4640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x558cd97f0200 0x558cd97d4640 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471c069610 msgr2=0x7f471c06bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471c069610 0x7f471c06bad0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f472c005400 tx=0x7f472c005310 comp rx=0 tx=0).stop 2024-09-13T17:13:05.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 msgr2=0x558cd97493a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 --2- 172.21.15.28:0/801656476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558cd97d8c40 0x558cd97493a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f4734045b80 tx=0x7f4734003330 comp rx=0 tx=0).stop 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ca00640 1 -- 172.21.15.28:0/801656476 reap_dead start 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 shutdown_connections 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 >> 172.21.15.28:0/801656476 conn(0x558cd974b270 msgr2=0x558cd974a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 shutdown_connections 2024-09-13T17:13:05.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.397+0000 7f473ed22f40 1 -- 172.21.15.28:0/801656476 wait complete. 2024-09-13T17:13:05.402 ERROR:tasks.ceph.mds.a:tried to stop a non-running daemon 2024-09-13T17:13:05.402 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-13T17:13:05.403 ERROR:tasks.ceph.mds.c:tried to stop a non-running daemon 2024-09-13T17:13:05.403 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-13T17:13:05.403 ERROR:tasks.ceph.mds.b:tried to stop a non-running daemon 2024-09-13T17:13:05.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-13T17:13:05.404 ERROR:tasks.ceph.mds.d:tried to stop a non-running daemon 2024-09-13T17:13:05.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-13T17:13:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7f2c01400640 1 Processor -- start 2024-09-13T17:13:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7f28ef200640 1 Processor -- start 2024-09-13T17:13:05.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7fc6c1600640 1 Processor -- start 2024-09-13T17:13:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7f28ef200640 1 -- start start 2024-09-13T17:13:05.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7f2c01400640 1 -- start start 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.570+0000 7fc6c1600640 1 -- start start 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.571+0000 7fc6c1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 0x7fc6bc06c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.571+0000 7fc6c1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 0x7fc6bc0b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.571+0000 7fc6c1600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc06cbb0 con 0x7fc6bc06c710 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.571+0000 7fc6c1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc06ce90 con 0x7fc6bc0b3420 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.571+0000 7fc6c1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc06d170 con 0x7fc6bc146030 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6c0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 0x7fc6bc0b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6c0c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 0x7fc6bc0b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41672/0 (socket says 172.21.15.28:41672) 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6c0c00640 1 -- 172.21.15.28:0/1884527506 learned_addr learned my addr 172.21.15.28:0/1884527506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bb400640 1 --1- 172.21.15.28:0/1884527506 >> v1:172.21.15.28:6789/0 conn(0x7fc6bc06c710 0x7fc6bc0b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53250/0 (socket says 172.21.15.28:53250) 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/1884527506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 0x7fc6bc06c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ef200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f28f00cbdf0 0x7f28f0165200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bbe00640 1 -- 172.21.15.28:0/1884527506 >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 msgr2=0x7fc6bc0b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/1884527506 >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 0x7fc6bc0b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bbe00640 1 -- 172.21.15.28:0/1884527506 >> v1:172.21.15.28:6789/0 conn(0x7fc6bc06c710 legacy=0x7fc6bc0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7fc6bbe00640 1 -- 172.21.15.28:0/1884527506 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc06d450 con 0x7fc6bc146030 2024-09-13T17:13:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f2c01400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2bfc06f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f2c01400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc160390 con 0x7f2bfc0d0b10 2024-09-13T17:13:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f2c01400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bfc160c30 con 0x7f2bfc1610c0 2024-09-13T17:13:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f2c01400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc15dd60 con 0x7f2bfc0cf480 2024-09-13T17:13:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ef200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00cdb80 con 0x7f28f00cd710 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ef200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28f00b1a70 con 0x7f28f0143f10 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ef200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00ce370 con 0x7f28f00cbdf0 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ed400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f28f00cd710 0x7f28f00cb6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53260/0 (socket says 172.21.15.28:53260) 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.572+0000 7f28ed400640 1 -- 172.21.15.28:0/4113690622 learned_addr learned my addr 172.21.15.28:0/4113690622 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f28ee800640 1 --2- 172.21.15.28:0/4113690622 >> v2:172.21.15.189:3300/0 conn(0x7f28f00cbdf0 0x7f28f0165200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f2c00a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2bfc0d0b10 0x7f2bfc159a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53264/0 (socket says 172.21.15.28:53264) 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f2c00a00640 1 -- 172.21.15.28:0/3888643510 learned_addr learned my addr 172.21.15.28:0/3888643510 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:05.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2bfc06f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f28eca00640 1 -- 172.21.15.28:0/4113690622 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716371081 0 0) 0x7f28f00cdb80 con 0x7f28f00cd710 2024-09-13T17:13:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f28ee800640 1 -- 172.21.15.28:0/4113690622 >> v1:172.21.15.189:6789/0 conn(0x7f28f0143f10 legacy=0x7f28f006d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f28ee800640 1 -- 172.21.15.28:0/4113690622 >> v1:172.21.15.28:6789/0 conn(0x7f28f00cd710 legacy=0x7f28f00cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.573+0000 7f28ee800640 1 -- 172.21.15.28:0/4113690622 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00b1740 con 0x7f28f00cbdf0 2024-09-13T17:13:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000998429 0 0) 0x7f2bfc160c30 con 0x7f2bfc1610c0 2024-09-13T17:13:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0003520 con 0x7f2bfc1610c0 2024-09-13T17:13:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785112983 0 0) 0x7f2bfc160390 con 0x7f2bfc0d0b10 2024-09-13T17:13:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bfc160c30 con 0x7f2bfc0d0b10 2024-09-13T17:13:05.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/1884527506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 0x7fc6bc06c1d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097a0 tx=0x7fc6ac02a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9af04988bf18c713 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v1:172.21.15.189:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881779523 0 0) 0x7f2be0003520 con 0x7f2bfc1610c0 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 msgr2=0x7f2bfc06f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2bfc06f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 >> v1:172.21.15.28:6789/0 conn(0x7f2bfc0d0b10 legacy=0x7f2bfc159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> v1:172.21.15.189:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0b39a0 con 0x7f2bfc1610c0 2024-09-13T17:13:05.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v1:172.21.15.189:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f2be8001670 con 0x7f2bfc1610c0 2024-09-13T17:13:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7fc6baa00640 1 -- 172.21.15.28:0/1884527506 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac02ae40 con 0x7fc6bc146030 2024-09-13T17:13:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7fc6baa00640 1 -- 172.21.15.28:0/1884527506 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ac032440 con 0x7fc6bc146030 2024-09-13T17:13:05.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7fc6baa00640 1 -- 172.21.15.28:0/1884527506 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ac0327d0 con 0x7fc6bc146030 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2bfc06f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.574+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 >> v1:172.21.15.189:6789/0 conn(0x7f2bfc1610c0 legacy=0x7f2bfc0d0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2be0003b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be00040e0 0x7f2be00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2be0004d00 0x7f2be0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc15dd60 con 0x7f2be00040e0 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be000bac0 con 0x7f2be0004d00 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2be000bc80 con 0x7f2bfc0cf480 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfb400640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be00040e0 0x7f2be00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f28ee800640 1 --2- 172.21.15.28:0/4113690622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cbdf0 0x7f28f0165200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff52f109bf55a2ed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2be0003b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7f2c00a00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2be0004d00 0x7f2be0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7fc6c1600640 1 -- 172.21.15.28:0/1884527506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 msgr2=0x7fc6bc06c1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.575+0000 7fc6c1600640 1 --2- 172.21.15.28:0/1884527506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 0x7fc6bc06c1d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac0097a0 tx=0x7fc6ac02a0f0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2c00a00640 1 -- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 msgr2=0x7f2be0003b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 -- 172.21.15.28:0/1884527506 shutdown_connections 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 --2- 172.21.15.28:0/1884527506 >> v2:172.21.15.28:3300/0 conn(0x7fc6bc0b3420 0x7fc6bc0b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 --2- 172.21.15.28:0/1884527506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc146030 0x7fc6bc06c1d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 -- 172.21.15.28:0/1884527506 >> 172.21.15.28:0/1884527506 conn(0x7fc6bc0b0310 msgr2=0x7fc6bc0b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fe067200640 1 Processor -- start 2024-09-13T17:13:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 -- 172.21.15.28:0/1884527506 shutdown_connections 2024-09-13T17:13:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2c00a00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2be0003b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fc6c1600640 1 -- 172.21.15.28:0/1884527506 wait complete. 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f28eca00640 1 -- 172.21.15.28:0/4113690622 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8031600 con 0x7f28f00cbdf0 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f28eca00640 1 -- 172.21.15.28:0/4113690622 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e802d050 con 0x7f28f00cbdf0 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f28eca00640 1 -- 172.21.15.28:0/4113690622 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8031980 con 0x7f28f00cbdf0 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2c00a00640 1 -- 172.21.15.28:0/3888643510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be00040e0 msgr2=0x7f2be00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2c00a00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be00040e0 0x7f2be00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2c00a00640 1 -- 172.21.15.28:0/3888643510 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bec009400 con 0x7f2be0004d00 2024-09-13T17:13:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc0cf480 0x7f2be0003b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f2bfb400640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be00040e0 0x7f2be00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:05.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7fe067200640 1 -- start start 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f28ef200640 1 -- 172.21.15.28:0/4113690622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cbdf0 msgr2=0x7f28f0165200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.576+0000 7f28ef200640 1 --2- 172.21.15.28:0/4113690622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cbdf0 0x7f28f0165200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009d90 tx=0x7f28e802a5d0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f28ef200640 1 -- 172.21.15.28:0/4113690622 shutdown_connections 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f28ef200640 1 --2- 172.21.15.28:0/4113690622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cbdf0 0x7f28f0165200 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f28ef200640 1 -- 172.21.15.28:0/4113690622 >> 172.21.15.28:0/4113690622 conn(0x7f28f00b0310 msgr2=0x7f28f00b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f28ef200640 1 -- 172.21.15.28:0/4113690622 shutdown_connections 2024-09-13T17:13:05.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f2c00a00640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2be0004d00 0x7f2be0005180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c210 tx=0x7f2be00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76928bb3441368f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7fe067200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 0x7fe0680cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7fe067200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680d0420 con 0x7fe06815cab0 2024-09-13T17:13:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7fe067200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680cdb60 con 0x7fe0680cd5b0 2024-09-13T17:13:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7f28ef200640 1 -- 172.21.15.28:0/4113690622 wait complete. 2024-09-13T17:13:05.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.577+0000 7fe067200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cde40 con 0x7fe06815c6d0 2024-09-13T17:13:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fc6c1600640 1 Processor -- start 2024-09-13T17:13:05.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4007270 con 0x7f2be0004d00 2024-09-13T17:13:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 0x7fe0680cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe06815cab0 0x7fe0680ccea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53266/0 (socket says 172.21.15.28:53266) 2024-09-13T17:13:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065400640 1 -- 172.21.15.28:0/266793107 learned_addr learned my addr 172.21.15.28:0/266793107 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf40073f0 con 0x7f2be0004d00 2024-09-13T17:13:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7f2bfaa00640 1 -- 172.21.15.28:0/3888643510 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40077a0 con 0x7f2be0004d00 2024-09-13T17:13:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fc6c1600640 1 -- start start 2024-09-13T17:13:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe064a00640 1 -- 172.21.15.28:0/266793107 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421421575 0 0) 0x7fe0680d0420 con 0x7fe06815cab0 2024-09-13T17:13:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065e00640 1 -- 172.21.15.28:0/266793107 >> v1:172.21.15.189:6789/0 conn(0x7fe0680cd5b0 legacy=0x7fe0680d0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065e00640 1 -- 172.21.15.28:0/266793107 >> v1:172.21.15.28:6789/0 conn(0x7fe06815cab0 legacy=0x7fe0680ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.578+0000 7fe065e00640 1 -- 172.21.15.28:0/266793107 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680d0670 con 0x7fe06815c6d0 2024-09-13T17:13:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f28ef200640 1 Processor -- start 2024-09-13T17:13:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f28ef200640 1 -- start start 2024-09-13T17:13:05.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 -- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2be0004d00 msgr2=0x7f2be0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 --2- 172.21.15.28:0/3888643510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2be0004d00 0x7f2be0005180 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400c210 tx=0x7f2be00057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c00a00640 1 -- 172.21.15.28:0/3888643510 reap_dead start 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 -- 172.21.15.28:0/3888643510 shutdown_connections 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 -- 172.21.15.28:0/3888643510 >> 172.21.15.28:0/3888643510 conn(0x7f2bfc06d570 msgr2=0x7f2bfc06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 -- 172.21.15.28:0/3888643510 shutdown_connections 2024-09-13T17:13:05.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.579+0000 7f2c01400640 1 -- 172.21.15.28:0/3888643510 wait complete. 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 0x7fc6bc13cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc146030 0x7fc6bc0ab3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc0a9210 con 0x7fc6bc0b3420 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc0a8590 con 0x7fc6bc146030 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc0a8870 con 0x7fc6bc06c710 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6bb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41684/0 (socket says 172.21.15.28:41684) 2024-09-13T17:13:05.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6bb400640 1 -- 172.21.15.28:0/144515260 learned_addr learned my addr 172.21.15.28:0/144515260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 0x7fc6bc13cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fe065e00640 1 --2- 172.21.15.28:0/266793107 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 0x7fe0680cf170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe058009750 tx=0x7fe05802a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c7e2b496c9fd0af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7fc6c0c00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc146030 0x7fc6bc0ab3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.580+0000 7f28ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 -- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 msgr2=0x7fc6bc13cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 0x7fc6bc13cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 -- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc146030 msgr2=0x7fc6bc0ab3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc146030 0x7fc6bc0ab3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 -- 172.21.15.28:0/144515260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac009400 con 0x7fc6bc0b3420 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 0x7f28f00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28f0143f10 0x7f28f00c82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f01437d0 con 0x7f28f00cbdf0 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00c8c90 con 0x7f28f00cd710 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00c8f70 con 0x7f28f0143f10 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fe064a00640 1 -- 172.21.15.28:0/266793107 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05802ad20 con 0x7fe06815c6d0 2024-09-13T17:13:05.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fe064a00640 1 -- 172.21.15.28:0/266793107 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05802ae80 con 0x7fe06815c6d0 2024-09-13T17:13:05.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 0x7fc6bc13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:05.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fe064a00640 1 -- 172.21.15.28:0/266793107 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058032660 con 0x7fe06815c6d0 2024-09-13T17:13:05.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6bb400640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c430 tx=0x7fc6a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41690/0 (socket says 172.21.15.28:41690) 2024-09-13T17:13:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ede00640 1 -- 172.21.15.28:0/258168603 learned_addr learned my addr 172.21.15.28:0/258168603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a80074e0 con 0x7fc6bc0b3420 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a8007640 con 0x7fc6bc0b3420 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8004a00 con 0x7fc6bc0b3420 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc0a8b50 con 0x7fc6bc0b3420 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc1e3800 con 0x7fc6bc0b3420 2024-09-13T17:13:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.581+0000 7f28ee800640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28f0143f10 0x7f28f00c82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fe067200640 1 -- 172.21.15.28:0/266793107 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 msgr2=0x7fe0680cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fe067200640 1 --2- 172.21.15.28:0/266793107 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 0x7fe0680cf170 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe058009750 tx=0x7fe05802a010 comp rx=0 tx=0).stop 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ed400640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 0x7f28f00c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ede00640 1 -- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28f0143f10 msgr2=0x7f28f00c82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ede00640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28f0143f10 0x7f28f00c82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ede00640 1 -- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 msgr2=0x7f28f00c9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ede00640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 0x7f28f00c9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ede00640 1 -- 172.21.15.28:0/258168603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009a70 con 0x7f28f00cbdf0 2024-09-13T17:13:05.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f28ed400640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 0x7f28f00c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7f2c01400640 1 Processor -- start 2024-09-13T17:13:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fe067200640 1 -- 172.21.15.28:0/266793107 shutdown_connections 2024-09-13T17:13:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fe067200640 1 --2- 172.21.15.28:0/266793107 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe06815c6d0 0x7fe0680cf170 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.582+0000 7fe067200640 1 -- 172.21.15.28:0/266793107 >> 172.21.15.28:0/266793107 conn(0x7fe068069e80 msgr2=0x7fe06806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7f28ede00640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c400 tx=0x7f28e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e00052d0 con 0x7f28f00cbdf0 2024-09-13T17:13:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc0b19f0 con 0x7fc6bc0b3420 2024-09-13T17:13:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0005430 con 0x7f28f00cbdf0 2024-09-13T17:13:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7fe067200640 1 -- 172.21.15.28:0/266793107 shutdown_connections 2024-09-13T17:13:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7fe067200640 1 -- 172.21.15.28:0/266793107 wait complete. 2024-09-13T17:13:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0003830 con 0x7f28f00cbdf0 2024-09-13T17:13:05.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6a8004b60 con 0x7fc6bc0b3420 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f00c9250 con 0x7f28f00cbdf0 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f00c9810 con 0x7f28f00cbdf0 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.583+0000 7f2c01400640 1 -- start start 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 0x7f2bfc0cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bfc1532b0 0x7f2bfc156220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c9a00 con 0x7f2be0004d00 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0cc0d0 con 0x7f2bfc1532b0 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2c01400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bfc0c9c50 con 0x7f2bfc152e30 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2bfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2bfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41704/0 (socket says 172.21.15.28:41704) 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.584+0000 7f2bfbe00640 1 -- 172.21.15.28:0/4264768648 learned_addr learned my addr 172.21.15.28:0/4264768648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfb400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 0x7f2bfc0cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 -- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 msgr2=0x7f2bfc0cacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 0x7f2bfc0cacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 -- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bfc1532b0 msgr2=0x7f2bfc156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bfc1532b0 0x7f2bfc156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 -- 172.21.15.28:0/4264768648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bfc0c9f30 con 0x7f2be0004d00 2024-09-13T17:13:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7fe067200640 1 Processor -- start 2024-09-13T17:13:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f00cb6e0 con 0x7f28f00cbdf0 2024-09-13T17:13:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7fe067200640 1 -- start start 2024-09-13T17:13:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7f2bfb400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 0x7f2bfc0cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe06815cab0 0x7fe0680b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fc6b3e00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 0x7fc69c06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6a8088c40 con 0x7fc6bc0b3420 2024-09-13T17:13:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.585+0000 7f2bfbe00640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2bec011ca0 tx=0x7f2bec00f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 0x7fc69c06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 0x7fe0681595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681f6800 con 0x7fe068159120 2024-09-13T17:13:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680b32c0 con 0x7fe06815cab0 2024-09-13T17:13:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe067200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680b3510 con 0x7fe0680cd5b0 2024-09-13T17:13:05.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe065e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe065e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47800/0 (socket says 172.21.15.28:47800) 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.586+0000 7fe065e00640 1 -- 172.21.15.28:0/1709659772 learned_addr learned my addr 172.21.15.28:0/1709659772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065400640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe06815cab0 0x7fe0680b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe066800640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 0x7fe0681595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 -- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe06815cab0 msgr2=0x7fe0680b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe06815cab0 0x7fe0680b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 -- 172.21.15.28:0/1709659772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 msgr2=0x7fe0681595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 0x7fe0681595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 -- 172.21.15.28:0/1709659772 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe058009400 con 0x7fe0680cd5b0 2024-09-13T17:13:05.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe066800640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 0x7fe0681595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 0x7fc69c06d110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac034000 tx=0x7fc6ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.587+0000 7fe065e00640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe05802a580 tx=0x7fe0580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05803d680 con 0x7fe0680cd5b0 2024-09-13T17:13:05.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe05803d7e0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05803db90 con 0x7fe0680cd5b0 2024-09-13T17:13:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0680b37f0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0680b3db0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.588+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28e0003990 con 0x7f28f00cbdf0 2024-09-13T17:13:05.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.589+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0680cf170 con 0x7fe0680cd5b0 2024-09-13T17:13:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.590+0000 7f28e6000640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 0x7f28d006d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.590+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28e0088cf0 con 0x7f28f00cbdf0 2024-09-13T17:13:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.590+0000 7f28ed400640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 0x7f28d006d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.591+0000 7f28ed400640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 0x7f28d006d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f28f015a640 tx=0x7f28dc007c70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.591+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe058039040 con 0x7fe0680cd5b0 2024-09-13T17:13:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.591+0000 7fe05e000640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 0x7fe04806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.592+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0580a9dd0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.592+0000 7fe065400640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 0x7fe04806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.593+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6a8057920 con 0x7fc6bc0b3420 2024-09-13T17:13:05.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.593+0000 7fe065400640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 0x7fe04806d0c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe054007f70 tx=0x7fe054007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.595+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe0580750d0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.597+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec00feb0 con 0x7f2be0004d00 2024-09-13T17:13:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.597+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bec014440 con 0x7f2be0004d00 2024-09-13T17:13:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.597+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bec0147b0 con 0x7f2be0004d00 2024-09-13T17:13:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.597+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bfc0ca210 con 0x7f2be0004d00 2024-09-13T17:13:05.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.597+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28e0053ff0 con 0x7f28f00cbdf0 2024-09-13T17:13:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.598+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bfc0ca7d0 con 0x7f2be0004d00 2024-09-13T17:13:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.599+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bfc0d0400 con 0x7f2be0004d00 2024-09-13T17:13:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.601+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2bec014910 con 0x7f2be0004d00 2024-09-13T17:13:05.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.602+0000 7f2bf3e00640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 0x7f2bdc06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.602+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2bec089d10 con 0x7f2be0004d00 2024-09-13T17:13:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.603+0000 7f2bfb400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 0x7f2bdc06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.604+0000 7f2bfb400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 0x7f2bdc06d110 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2be8001bd0 tx=0x7f2be8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:05.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.607+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2bec055060 con 0x7f2be0004d00 2024-09-13T17:13:05.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.845+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fe0680b87d0 con 0x7fe0680cd5b0 2024-09-13T17:13:05.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.857+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f2bfc0d13d0 con 0x7f2be0004d00 2024-09-13T17:13:05.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.879+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fc6bc0ae690 con 0x7fc6bc0b3420 2024-09-13T17:13:05.881 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:13:05.880+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:13:05.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.883+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f28f00b3510 con 0x7f28f00cbdf0 2024-09-13T17:13:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.887+0000 7f2bf3e00640 1 -- 172.21.15.28:0/4264768648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 6065 v65) v1 ==== 95+0+0 (secure 0 0 0) 0x7f2bec058680 con 0x7f2be0004d00 2024-09-13T17:13:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.887+0000 7fc6b3e00640 1 -- 172.21.15.28:0/144515260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6954 v65) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc6a80574b0 con 0x7fc6bc0b3420 2024-09-13T17:13:05.888 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6065 2024-09-13T17:13:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6954 2024-09-13T17:13:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.888+0000 7fe05e000640 1 -- 172.21.15.28:0/1709659772 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6658 v65) v1 ==== 95+0+0 (secure 0 0 0) 0x7fe058037090 con 0x7fe0680cd5b0 2024-09-13T17:13:05.889 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6658 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 msgr2=0x7fc69c06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 0x7fc69c06d110 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc6ac034000 tx=0x7fc6ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 msgr2=0x7fc6bc0aaea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c430 tx=0x7fc6a800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 shutdown_connections 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac50 0x7fc69c06d110 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc146030 0x7fc6bc0ab3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0b3420 0x7fc6bc0aaea0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 --2- 172.21.15.28:0/144515260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc06c710 0x7fc6bc13cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 >> 172.21.15.28:0/144515260 conn(0x7fc6bc0b0310 msgr2=0x7fc6bc0afa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 msgr2=0x7f2bdc06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 0x7f2bdc06d110 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f2be8001bd0 tx=0x7f2be8009040 comp rx=0 tx=0).stop 2024-09-13T17:13:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 msgr2=0x7f2bfc0b2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f2bec011ca0 tx=0x7f2bec00f610 comp rx=0 tx=0).stop 2024-09-13T17:13:05.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 shutdown_connections 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 shutdown_connections 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2bdc06ac50 0x7f2bdc06d110 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2bfc1532b0 0x7f2bfc156220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2bfc152e30 0x7f2bfc0cacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 --2- 172.21.15.28:0/4264768648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2be0004d00 0x7f2bfc0b2960 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 >> 172.21.15.28:0/4264768648 conn(0x7f2bfc06d570 msgr2=0x7f2bfc0cce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7fc6c1600640 1 -- 172.21.15.28:0/144515260 wait complete. 2024-09-13T17:13:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.890+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 shutdown_connections 2024-09-13T17:13:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7f2c01400640 1 -- 172.21.15.28:0/4264768648 wait complete. 2024-09-13T17:13:05.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 msgr2=0x7fe04806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 0x7fe04806d0c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fe054007f70 tx=0x7fe054007f00 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 msgr2=0x7fe0680cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe05802a580 tx=0x7fe0580046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 shutdown_connections 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe04806ac00 0x7fe04806d0c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe068159120 0x7fe0681595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe06815cab0 0x7fe0680b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 --2- 172.21.15.28:0/1709659772 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0680cd5b0 0x7fe0680cc410 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe05802a580 tx=0x7fe0580046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.891+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 >> 172.21.15.28:0/1709659772 conn(0x7fe068069e80 msgr2=0x7fe06806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.892+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 shutdown_connections 2024-09-13T17:13:05.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:05.892+0000 7fe067200640 1 -- 172.21.15.28:0/1709659772 wait complete. 2024-09-13T17:13:05.910 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:13:05.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:13:05.911 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:13:05.912 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:13:05.913 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:13:05.913 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:13:05.914 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:13:05.915 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.917 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:13:05.917 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:05.919 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:13:05.919 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:06.043 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:13:06.050 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:13:06.050 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:13:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.890+0000 7f28e6000640 1 -- 172.21.15.28:0/258168603 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 6649 v66) v1 ==== 95+0+0 (secure 0 0 0) 0x7f28e0017090 con 0x7f28f00cbdf0 2024-09-13T17:13:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6649 2024-09-13T17:13:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 msgr2=0x7f28d006d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 0x7f28d006d190 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f28f015a640 tx=0x7f28dc007c70 comp rx=0 tx=0).stop 2024-09-13T17:13:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 msgr2=0x7f28f0143290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c400 tx=0x7f28e000c8c0 comp rx=0 tx=0).stop 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 shutdown_connections 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28d006acd0 0x7f28d006d190 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28f0143f10 0x7f28f00c82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28f00cd710 0x7f28f00c9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 --2- 172.21.15.28:0/258168603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28f00cbdf0 0x7f28f0143290 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 >> 172.21.15.28:0/258168603 conn(0x7f28f00b0310 msgr2=0x7f28f015cf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.894+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 shutdown_connections 2024-09-13T17:13:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:06.895+0000 7f28ef200640 1 -- 172.21.15.28:0/258168603 wait complete. 2024-09-13T17:13:06.915 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:13:06.915 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:13:06.919 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:13:06.919 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:06.919 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:07.050 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:13:07.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.076+0000 7fd509000640 1 Processor -- start 2024-09-13T17:13:07.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.077+0000 7fd509000640 1 -- start start 2024-09-13T17:13:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd504159440 0x7fd5040b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd504159820 0x7fd5040b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5040b22e0 0x7fd5040b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5040c3850 con 0x7fd5040b22e0 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd50415a930 con 0x7fd504159820 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd509000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5040b2cd0 con 0x7fd504159440 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd503e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5040b22e0 0x7fd5040b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd503e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd5040b22e0 0x7fd5040b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41752/0 (socket says 172.21.15.28:41752) 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd503e00640 1 -- 172.21.15.28:0/2040461256 learned_addr learned my addr 172.21.15.28:0/2040461256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd503400640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd504159440 0x7fd5040b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.078+0000 7fd4fbe00640 1 --2- 172.21.15.28:0/2040461256 >> v2:172.21.15.189:3300/0 conn(0x7fd504159820 0x7fd5040b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.079+0000 7fd503e00640 1 -- 172.21.15.28:0/2040461256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd504159440 msgr2=0x7fd5040b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.079+0000 7fd503e00640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd504159440 0x7fd5040b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.079+0000 7fd503e00640 1 -- 172.21.15.28:0/2040461256 >> v2:172.21.15.189:3300/0 conn(0x7fd504159820 msgr2=0x7fd5040b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.079+0000 7fd503e00640 1 --2- 172.21.15.28:0/2040461256 >> v2:172.21.15.189:3300/0 conn(0x7fd504159820 0x7fd5040b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.079+0000 7fd503e00640 1 -- 172.21.15.28:0/2040461256 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd504159d20 con 0x7fd5040b22e0 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd503e00640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040b22e0 0x7fd5040b2790 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd5040b10b0 tx=0x7fd4fc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187bdd1c75f3da81 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd502a00640 1 -- 172.21.15.28:0/2040461256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc031600 con 0x7fd5040b22e0 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd502a00640 1 -- 172.21.15.28:0/2040461256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc02d050 con 0x7fd5040b22e0 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd502a00640 1 -- 172.21.15.28:0/2040461256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc031980 con 0x7fd5040b22e0 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd509000640 1 -- 172.21.15.28:0/2040461256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040b22e0 msgr2=0x7fd5040b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.080+0000 7fd509000640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040b22e0 0x7fd5040b2790 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fd5040b10b0 tx=0x7fd4fc02a730 comp rx=0 tx=0).stop 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 -- 172.21.15.28:0/2040461256 shutdown_connections 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5040b22e0 0x7fd5040b2790 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 --2- 172.21.15.28:0/2040461256 >> v2:172.21.15.189:3300/0 conn(0x7fd504159820 0x7fd5040b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 --2- 172.21.15.28:0/2040461256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd504159440 0x7fd5040b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 -- 172.21.15.28:0/2040461256 >> 172.21.15.28:0/2040461256 conn(0x7fd504072e40 msgr2=0x7fd504073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 -- 172.21.15.28:0/2040461256 shutdown_connections 2024-09-13T17:13:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.081+0000 7fd509000640 1 -- 172.21.15.28:0/2040461256 wait complete. 2024-09-13T17:13:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.082+0000 7fd509000640 1 Processor -- start 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.082+0000 7fd509000640 1 -- start start 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5040b22e0 0x7fd504153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504159440 0x7fd5041e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd504158970 con 0x7fd504159820 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5041e3730 con 0x7fd504159440 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.083+0000 7fd509000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd504158690 con 0x7fd5040b22e0 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41762/0 (socket says 172.21.15.28:41762) 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 -- 172.21.15.28:0/2170583820 learned_addr learned my addr 172.21.15.28:0/2170583820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd4fbe00640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504159440 0x7fd5041e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503400640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5040b22e0 0x7fd504153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 -- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5040b22e0 msgr2=0x7fd504153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5040b22e0 0x7fd504153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 -- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504159440 msgr2=0x7fd5041e3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504159440 0x7fd5041e3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 -- 172.21.15.28:0/2170583820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc009b30 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.084+0000 7fd503e00640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc034040 tx=0x7fd4fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.085+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc03b820 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.085+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4fc03b9a0 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.085+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc03bd50 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.085+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5041e30e0 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.085+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5041e4100 con 0x7fd504159820 2024-09-13T17:13:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.086+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5041e2f40 con 0x7fd504159820 2024-09-13T17:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.087+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd4fc038050 con 0x7fd504159820 2024-09-13T17:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.087+0000 7fd4fb400640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 0x7fd4e006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.088+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd4fc0a9c30 con 0x7fd504159820 2024-09-13T17:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.088+0000 7fd503400640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 0x7fd4e006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.089+0000 7fd503400640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 0x7fd4e006d0f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd5041f55e0 tx=0x7fd4f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.092+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd5041e2f40 con 0x7fd504159820 2024-09-13T17:13:07.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.305+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd5041e2f40 con 0x7fd504159820 2024-09-13T17:13:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.306+0000 7fd4fb400640 1 -- 172.21.15.28:0/2170583820 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3986 (secure 0 0 0) 0x7fd5041e2f40 con 0x7fd504159820 2024-09-13T17:13:07.307 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:07.307 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":67,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:06.891511+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:replay","state_seq":1,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:07.307 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 67 2024-09-13T17:13:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 msgr2=0x7fd4e006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 0x7fd4e006d0f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd5041f55e0 tx=0x7fd4f4008040 comp rx=0 tx=0).stop 2024-09-13T17:13:07.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 msgr2=0x7fd50415c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc034040 tx=0x7fd4fc004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 shutdown_connections 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd4e006ac30 0x7fd4e006d0f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd504159820 0x7fd50415c070 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd504159440 0x7fd5041e3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 --2- 172.21.15.28:0/2170583820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd5040b22e0 0x7fd504153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 >> 172.21.15.28:0/2170583820 conn(0x7fd504072e40 msgr2=0x7fd50415afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 shutdown_connections 2024-09-13T17:13:07.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:07.308+0000 7fd509000640 1 -- 172.21.15.28:0/2170583820 wait complete. 2024-09-13T17:13:07.728 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 67, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:54.534325+0000', 'modified': '2024-09-13T17:13:06.891511+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7212}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7212': {'gid': 7212, 'name': 'd', 'rank': 0, 'incarnation': 67, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.189:6836/1743829849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1743829849}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 1743829849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:13:07.728 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7212:up:replay 2024-09-13T17:13:08.729 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.895+0000 7fa84b800640 1 Processor -- start 2024-09-13T17:13:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.895+0000 7fa84b800640 1 -- start start 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 0x7fa84415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa84406f970 0x7fa844159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa84406c2d0 0x7fa84406c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84415b0b0 con 0x7fa84406f970 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84406ccc0 con 0x7fa84406c2d0 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa84b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440b2040 con 0x7fa8441592a0 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa843e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa84406f970 0x7fa844159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa843e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa84406f970 0x7fa844159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41766/0 (socket says 172.21.15.28:41766) 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.896+0000 7fa843e00640 1 -- 172.21.15.28:0/2812150151 learned_addr learned my addr 172.21.15.28:0/2812150151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa848e00640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 0x7fa84415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa849800640 1 --2- 172.21.15.28:0/2812150151 >> v2:172.21.15.189:3300/0 conn(0x7fa84406c2d0 0x7fa84406c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa843e00640 1 -- 172.21.15.28:0/2812150151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 msgr2=0x7fa84415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa843e00640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 0x7fa84415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa843e00640 1 -- 172.21.15.28:0/2812150151 >> v2:172.21.15.189:3300/0 conn(0x7fa84406c2d0 msgr2=0x7fa84406c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa843e00640 1 --2- 172.21.15.28:0/2812150151 >> v2:172.21.15.189:3300/0 conn(0x7fa84406c2d0 0x7fa84406c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa843e00640 1 -- 172.21.15.28:0/2812150151 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa844159ff0 con 0x7fa84406f970 2024-09-13T17:13:08.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.897+0000 7fa848e00640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 0x7fa84415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.898+0000 7fa843e00640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa84406f970 0x7fa844159680 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa84406f700 tx=0x7fa83002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcc9e2c86183949 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.898+0000 7fa843400640 1 -- 172.21.15.28:0/2812150151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830032440 con 0x7fa84406f970 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.898+0000 7fa843400640 1 -- 172.21.15.28:0/2812150151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8300325a0 con 0x7fa84406f970 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.898+0000 7fa843400640 1 -- 172.21.15.28:0/2812150151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa830032930 con 0x7fa84406f970 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 -- 172.21.15.28:0/2812150151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa84406f970 msgr2=0x7fa844159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa84406f970 0x7fa844159680 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fa84406f700 tx=0x7fa83002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 -- 172.21.15.28:0/2812150151 shutdown_connections 2024-09-13T17:13:08.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa84406f970 0x7fa844159680 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 --2- 172.21.15.28:0/2812150151 >> v2:172.21.15.189:3300/0 conn(0x7fa84406c2d0 0x7fa84406c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 --2- 172.21.15.28:0/2812150151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8441592a0 0x7fa84415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 -- 172.21.15.28:0/2812150151 >> 172.21.15.28:0/2812150151 conn(0x7fa844069e80 msgr2=0x7fa84406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 -- 172.21.15.28:0/2812150151 shutdown_connections 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.899+0000 7fa84b800640 1 -- 172.21.15.28:0/2812150151 wait complete. 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.901+0000 7fa84b800640 1 Processor -- start 2024-09-13T17:13:08.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.901+0000 7fa84b800640 1 -- start start 2024-09-13T17:13:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84406c2d0 0x7fa84414e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa84406f970 0x7fa8440af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440ac1f0 con 0x7fa8441592a0 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440ac4d0 con 0x7fa84406c2d0 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa84b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8440ac7b0 con 0x7fa84406f970 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa849800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa849800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41768/0 (socket says 172.21.15.28:41768) 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa849800640 1 -- 172.21.15.28:0/3426326919 learned_addr learned my addr 172.21.15.28:0/3426326919 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa848e00640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84406c2d0 0x7fa84414e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.902+0000 7fa843e00640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa84406f970 0x7fa8440af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 -- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa84406f970 msgr2=0x7fa8440af3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa84406f970 0x7fa8440af3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 -- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84406c2d0 msgr2=0x7fa84414e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84406c2d0 0x7fa84414e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:08.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 -- 172.21.15.28:0/3426326919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300094d0 con 0x7fa8441592a0 2024-09-13T17:13:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.903+0000 7fa849800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa83c007e40 tx=0x7fa83c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:08.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.904+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c014620 con 0x7fa8441592a0 2024-09-13T17:13:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.904+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c00ee90 con 0x7fa8441592a0 2024-09-13T17:13:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.904+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0149e0 con 0x7fa8441592a0 2024-09-13T17:13:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.904+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8440aca90 con 0x7fa8441592a0 2024-09-13T17:13:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.904+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8440ad290 con 0x7fa8441592a0 2024-09-13T17:13:08.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.906+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8440acfb0 con 0x7fa8441592a0 2024-09-13T17:13:08.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.906+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa83c014b40 con 0x7fa8441592a0 2024-09-13T17:13:08.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.907+0000 7fa840c00640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 0x7fa82406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:08.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.907+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa83c089c80 con 0x7fa8441592a0 2024-09-13T17:13:08.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.908+0000 7fa848e00640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 0x7fa82406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:08.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.908+0000 7fa848e00640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 0x7fa82406d0f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa838006e80 tx=0x7fa838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:08.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:08.912+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa8440acfb0 con 0x7fa8441592a0 2024-09-13T17:13:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.126+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8440acfb0 con 0x7fa8441592a0 2024-09-13T17:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.127+0000 7fa840c00640 1 -- 172.21.15.28:0/3426326919 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fa8440acfb0 con 0x7fa8441592a0 2024-09-13T17:13:09.128 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:09.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":69,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:08.900749+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:09.128 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 69 2024-09-13T17:13:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 msgr2=0x7fa82406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 0x7fa82406d0f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa838006e80 tx=0x7fa838008040 comp rx=0 tx=0).stop 2024-09-13T17:13:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 msgr2=0x7fa8440af8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fa83c007e40 tx=0x7fa83c00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 shutdown_connections 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa82406ac30 0x7fa82406d0f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8441592a0 0x7fa8440af8e0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa84406f970 0x7fa8440af3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 --2- 172.21.15.28:0/3426326919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa84406c2d0 0x7fa84414e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 >> 172.21.15.28:0/3426326919 conn(0x7fa844069e80 msgr2=0x7fa844159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 shutdown_connections 2024-09-13T17:13:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:09.129+0000 7fa84b800640 1 -- 172.21.15.28:0/3426326919 wait complete. 2024-09-13T17:13:09.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:54.534325+0000', 'modified': '2024-09-13T17:13:08.900749+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7212}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7212': {'gid': 7212, 'name': 'd', 'rank': 0, 'incarnation': 67, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.189:6836/1743829849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1743829849}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 1743829849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:13:09.550 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7212:up:rejoin 2024-09-13T17:13:09.731 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:13:09.735 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:13:10.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.718+0000 7fc8f8a00640 1 Processor -- start 2024-09-13T17:13:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.718+0000 7fc8f8a00640 1 -- start start 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f8a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc8f40b0b50 0x7fc8f40b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8f40b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8f40c3850 con 0x7fc8f4159440 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f8a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8f40b2d60 con 0x7fc8f40b0b50 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f415a930 con 0x7fc8f4159820 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc8f4159440 0x7fc8f40b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53288/0 (socket says 172.21.15.28:53288) 2024-09-13T17:13:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.719+0000 7fc8f3400640 1 -- 172.21.15.28:0/2853202721 learned_addr learned my addr 172.21.15.28:0/2853202721 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f3e00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8f40b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8ebe00640 1 --2- 172.21.15.28:0/2853202721 >> v2:172.21.15.189:3300/0 conn(0x7fc8f40b0b50 0x7fc8f40b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400715708 0 0) 0x7fc8f40c3850 con 0x7fc8f4159440 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d8003520 con 0x7fc8f4159440 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339586604 0 0) 0x7fc8d8003520 con 0x7fc8f4159440 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 msgr2=0x7fc8f40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8f40b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 >> v2:172.21.15.189:3300/0 conn(0x7fc8f40b0b50 msgr2=0x7fc8f40b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 --2- 172.21.15.28:0/2853202721 >> v2:172.21.15.189:3300/0 conn(0x7fc8f40b0b50 0x7fc8f40b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4159d20 con 0x7fc8f4159440 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc8e4001670 con 0x7fc8f4159440 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 >> v1:172.21.15.28:6789/0 conn(0x7fc8f4159440 legacy=0x7fc8f40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8d8004d00 0x7fc8d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f40b2d60 con 0x7fc8d8003970 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f415a930 con 0x7fc8d8004d00 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.720+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d800b980 con 0x7fc8f4159820 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8ebe00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3e00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8d8004d00 0x7fc8d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 -- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 msgr2=0x7fc8d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4159820 0x7fc8d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 -- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8d8004d00 msgr2=0x7fc8d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8d8004d00 0x7fc8d8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.721+0000 7fc8f3400640 1 -- 172.21.15.28:0/2853202721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8003520 con 0x7fc8d8003970 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.722+0000 7fc8f3400640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8d8003d90 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c240 tx=0x7fc8d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a27d6237a1837e68 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.722+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4005ad0 con 0x7fc8d8003970 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.722+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4005c30 con 0x7fc8d8003970 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.722+0000 7fc8f2a00640 1 -- 172.21.15.28:0/2853202721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4007070 con 0x7fc8d8003970 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 -- 172.21.15.28:0/2853202721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 msgr2=0x7fc8d8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/2853202721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8d8003d90 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fc8e400c240 tx=0x7fc8d80057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f3e00640 1 -- 172.21.15.28:0/2853202721 reap_dead start 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 -- 172.21.15.28:0/2853202721 shutdown_connections 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 -- 172.21.15.28:0/2853202721 >> 172.21.15.28:0/2853202721 conn(0x7fc8f4072e40 msgr2=0x7fc8f4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 -- 172.21.15.28:0/2853202721 shutdown_connections 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.723+0000 7fc8f8a00640 1 -- 172.21.15.28:0/2853202721 wait complete. 2024-09-13T17:13:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.725+0000 7fc8f8a00640 1 Processor -- start 2024-09-13T17:13:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.725+0000 7fc8f8a00640 1 -- start start 2024-09-13T17:13:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4157880 0x7fc8f4157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8f414ba10 0x7fc8f414be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f414d210 con 0x7fc8d8003970 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4158240 con 0x7fc8f414ba10 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f414aca0 con 0x7fc8f4157880 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41776/0 (socket says 172.21.15.28:41776) 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f3400640 1 -- 172.21.15.28:0/1573884923 learned_addr learned my addr 172.21.15.28:0/1573884923 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8ebe00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4157880 0x7fc8f4157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.726+0000 7fc8f3e00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8f414ba10 0x7fc8f414be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 -- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4157880 msgr2=0x7fc8f4157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4157880 0x7fc8f4157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 -- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8f414ba10 msgr2=0x7fc8f414be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8f414ba10 0x7fc8f414be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 -- 172.21.15.28:0/1573884923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4149c00 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f3400640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc8e402ad50 tx=0x7fc8e4001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4007500 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4007660 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f4149ee0 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.727+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e403e650 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.728+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f414a4a0 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.729+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f40b22e0 con 0x7fc8d8003970 2024-09-13T17:13:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.729+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc8e403e7b0 con 0x7fc8d8003970 2024-09-13T17:13:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.730+0000 7fc8eb400640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 0x7fc8d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.730+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc8e40aa2f0 con 0x7fc8d8003970 2024-09-13T17:13:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.731+0000 7fc8ebe00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 0x7fc8d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:10.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.732+0000 7fc8ebe00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 0x7fc8d006d0f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8f415b6b0 tx=0x7fc8dc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:10.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.734+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc8e4075630 con 0x7fc8d8003970 2024-09-13T17:13:10.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.949+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8f406beb0 con 0x7fc8d8003970 2024-09-13T17:13:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.949+0000 7fc8eb400640 1 -- 172.21.15.28:0/1573884923 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fc8e4078c50 con 0x7fc8d8003970 2024-09-13T17:13:10.950 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:10.950 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:09.912894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:10.950 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 70 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 msgr2=0x7fc8d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 0x7fc8d006d0f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc8f415b6b0 tx=0x7fc8dc009190 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 msgr2=0x7fc8f4148f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7fc8e402ad50 tx=0x7fc8e4001a00 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 shutdown_connections 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc8d006ac30 0x7fc8d006d0f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8f414ba10 0x7fc8f414be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc8f4157880 0x7fc8f4157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 --2- 172.21.15.28:0/1573884923 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc8d8003970 0x7fc8f4148f70 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 >> 172.21.15.28:0/1573884923 conn(0x7fc8f4072e40 msgr2=0x7fc8f406b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 shutdown_connections 2024-09-13T17:13:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:10.952+0000 7fc8f8a00640 1 -- 172.21.15.28:0/1573884923 wait complete. 2024-09-13T17:13:11.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 70, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:12:54.534325+0000', 'modified': '2024-09-13T17:13:09.912894+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 80, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7212}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7212': {'gid': 7212, 'name': 'd', 'rank': 0, 'incarnation': 67, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.189:6836/1743829849', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 1743829849}, {'type': 'v1', 'addr': '172.21.15.189:6836', 'nonce': 1743829849}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:13:11.371 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:13:11.371 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T17:13:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.530+0000 7f0023600640 1 Processor -- start 2024-09-13T17:13:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.530+0000 7f0023600640 1 -- start start 2024-09-13T17:13:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0023600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 0x7f001c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0023600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f001c071490 0x7f001c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0023600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c06c8b0 con 0x7f001c15a780 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0023600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f001c06cb90 con 0x7f001c071490 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0023600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c0cc980 con 0x7f001c15ab60 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0021600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f001c071490 0x7f001c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0021600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f001c071490 0x7f001c06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41792/0 (socket says 172.21.15.28:41792) 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.531+0000 7f0021600640 1 -- 172.21.15.28:0/567278215 learned_addr learned my addr 172.21.15.28:0/567278215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f001be00640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 0x7f001c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f0021600640 1 -- 172.21.15.28:0/567278215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 msgr2=0x7f001c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f0021600640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 0x7f001c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f0021600640 1 -- 172.21.15.28:0/567278215 >> v1:172.21.15.189:6789/0 conn(0x7f001c15a780 legacy=0x7f001c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f0021600640 1 -- 172.21.15.28:0/567278215 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0ccc60 con 0x7f001c071490 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.532+0000 7f001be00640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 0x7f001c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.533+0000 7f0021600640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c071490 0x7f001c06c080 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84acb823bc29b865 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.533+0000 7f001b400640 1 -- 172.21.15.28:0/567278215 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014031600 con 0x7f001c071490 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.533+0000 7f001b400640 1 -- 172.21.15.28:0/567278215 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001402d050 con 0x7f001c071490 2024-09-13T17:13:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.533+0000 7f001b400640 1 -- 172.21.15.28:0/567278215 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014031980 con 0x7f001c071490 2024-09-13T17:13:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 -- 172.21.15.28:0/567278215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c071490 msgr2=0x7f001c06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c071490 0x7f001c06c080 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f001402a710 comp rx=0 tx=0).stop 2024-09-13T17:13:11.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 -- 172.21.15.28:0/567278215 shutdown_connections 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c071490 0x7f001c06c080 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 --2- 172.21.15.28:0/567278215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c15ab60 0x7f001c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 -- 172.21.15.28:0/567278215 >> 172.21.15.28:0/567278215 conn(0x7f001c0b0270 msgr2=0x7f001c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 -- 172.21.15.28:0/567278215 shutdown_connections 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.534+0000 7f0023600640 1 -- 172.21.15.28:0/567278215 wait complete. 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.536+0000 7f0023600640 1 Processor -- start 2024-09-13T17:13:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.536+0000 7f0023600640 1 -- start start 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.536+0000 7f0023600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c071490 0x7f001c1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0023600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f001c15a780 0x7f001c145b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0023600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0023600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c14bea0 con 0x7f001c15ab60 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0023600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c1461e0 con 0x7f001c15a780 2024-09-13T17:13:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0023600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f001c1464c0 con 0x7f001c071490 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41800/0 (socket says 172.21.15.28:41800) 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 -- 172.21.15.28:0/4049437765 learned_addr learned my addr 172.21.15.28:0/4049437765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0020c00640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c071490 0x7f001c1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f001be00640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f001c15a780 0x7f001c145b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 -- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c071490 msgr2=0x7f001c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c071490 0x7f001c1454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 -- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f001c15a780 msgr2=0x7f001c145b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f001c15a780 0x7f001c145b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.537+0000 7f0021600640 1 -- 172.21.15.28:0/4049437765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0014009b30 con 0x7f001c15ab60 2024-09-13T17:13:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0021600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f0014004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0018c00640 1 -- 172.21.15.28:0/4049437765 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001403d860 con 0x7f001c15ab60 2024-09-13T17:13:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0018c00640 1 -- 172.21.15.28:0/4049437765 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f001403d9e0 con 0x7f001c15ab60 2024-09-13T17:13:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0018c00640 1 -- 172.21.15.28:0/4049437765 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001403dd90 con 0x7f001c15ab60 2024-09-13T17:13:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c1467a0 con 0x7f001c15ab60 2024-09-13T17:13:11.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.538+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c146d60 con 0x7f001c15ab60 2024-09-13T17:13:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.540+0000 7f0018c00640 1 -- 172.21.15.28:0/4049437765 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f001403b400 con 0x7f001c15ab60 2024-09-13T17:13:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.541+0000 7f0018c00640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 0x7efffc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.541+0000 7f0018c00640 1 -- 172.21.15.28:0/4049437765 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00140aa220 con 0x7f001c15ab60 2024-09-13T17:13:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.541+0000 7f0020c00640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 0x7efffc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 Processor -- start 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 -- start start 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0020c00640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 0x7efffc06d0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0010000f60 tx=0x7f0010009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 0x55c780788ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c780785e90 0x55c7807d5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7806f1ab0 con 0x55c7807afb80 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7806ef240 con 0x55c7807886d0 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.542+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c7806eab30 con 0x55c780785e90 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0020c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 0x55c780788ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41810/0 (socket says 172.21.15.28:41810) 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 -- 172.21.15.28:0/2824696200 learned_addr learned my addr 172.21.15.28:0/2824696200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f001be00640 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c780785e90 0x55c7807d5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 -- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c780785e90 msgr2=0x55c7807d5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c780785e90 0x55c7807d5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 -- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 msgr2=0x55c780788ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 0x55c780788ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0021600640 1 -- 172.21.15.28:0/2824696200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7806b5bd0 con 0x55c7807afb80 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.543+0000 7f0020c00640 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 0x55c780788ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0021600640 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0014079000 tx=0x7f0014009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d120097ad820158e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f000d600640 1 -- 172.21.15.28:0/2824696200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001407cdf0 con 0x55c7807afb80 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f000d600640 1 -- 172.21.15.28:0/2824696200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00140813b0 con 0x55c7807afb80 2024-09-13T17:13:11.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f000d600640 1 -- 172.21.15.28:0/2824696200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014077410 con 0x55c7807afb80 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 -- 172.21.15.28:0/2824696200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 msgr2=0x55c7807d60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f0014079000 tx=0x7f0014009510 comp rx=0 tx=0).stop 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 -- 172.21.15.28:0/2824696200 shutdown_connections 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807afb80 0x55c7807d60e0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c780785e90 0x55c7807d5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 --2- 172.21.15.28:0/2824696200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807886d0 0x55c780788ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 -- 172.21.15.28:0/2824696200 >> 172.21.15.28:0/2824696200 conn(0x55c780784e00 msgr2=0x55c7807cd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.544+0000 7f0026404740 1 -- 172.21.15.28:0/2824696200 shutdown_connections 2024-09-13T17:13:11.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.545+0000 7f0026404740 1 -- 172.21.15.28:0/2824696200 wait complete. 2024-09-13T17:13:11.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.545+0000 7f0026404740 1 -- start start 2024-09-13T17:13:11.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.545+0000 7f0026404740 1 Processor -- start 2024-09-13T17:13:11.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807a6ff0 0x55c7807ae970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c78068c260 con 0x55c7807886d0 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c78068bf00 con 0x55c7807afb80 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c780451f20 con 0x55c7807a6ff0 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0026404740 10 client.0 _ll_register_callbacks cb 0x55c7806e41a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.546+0000 7f0020c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41814/0 (socket says 172.21.15.28:41814) 2024-09-13T17:13:11.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 -- 172.21.15.28:0/1606388203 learned_addr learned my addr 172.21.15.28:0/1606388203 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f001be00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0021600640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807a6ff0 0x55c7807ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 -- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807a6ff0 msgr2=0x55c7807ae970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807a6ff0 0x55c7807ae970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 -- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 msgr2=0x55c7807ca7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807ca7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 -- 172.21.15.28:0/1606388203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c146d60 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f001be00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0020c00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0010014700 tx=0x7f0010001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=400cbf3609892fe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.547+0000 7f0002000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010013700 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c780714cd0 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f001001c040 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0010013860 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 10 client.7251 fetch_fsmap learned FSMap version 70 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55c7806e91b0 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c780714cd0 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0010012070 con 0x55c7807886d0 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 10 client.7251 fetch_fsmap finished waiting for FSMap version 70 2024-09-13T17:13:11.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 10 client.7251 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 20 client.7251 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c780792490 0x55c7807938a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 4 client.7251 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.548+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 --> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c7803de3c0 con 0x55c780792490 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.549+0000 7f001be00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c780792490 0x55c7807938a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.549+0000 7f001be00640 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c780792490 0x55c7807938a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47763cff29656e9c server_cookie=14aa994fc5c497fa in_seq=0 out_seq=0 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.549+0000 7f0002000640 10 client.7251 ms_handle_connect on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:11.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.550+0000 7f0002000640 1 -- 172.21.15.28:0/1606388203 <== mds.0 v2:172.21.15.189:6834/1743829849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55c7803de3c0 con 0x55c780792490 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.550+0000 7f0002000640 10 client.7251 handle_command_reply: tid=0 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.550+0000 7f0026404740 1 client.7251 shutdown 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 msgr2=0x55c780787f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f0010014700 tx=0x7f0010001120 comp rx=0 tx=0).stop 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 shutdown_connections 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c780792490 0x55c7807938a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807a6ff0 0x55c7807ae970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807ca7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 --2- 172.21.15.28:0/1606388203 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c780787f40 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0002000640 0 client.7251 ms_handle_reset on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:11.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 >> 172.21.15.28:0/1606388203 conn(0x55c780785e90 msgr2=0x55c7807ad380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 shutdown_connections 2024-09-13T17:13:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 1 -- 172.21.15.28:0/1606388203 wait complete. 2024-09-13T17:13:11.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.551+0000 7f0026404740 20 client.7251 trim_cache size 0 max 16384 2024-09-13T17:13:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.566+0000 7f0026404740 1 Processor -- start 2024-09-13T17:13:11.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.566+0000 7f0026404740 1 -- start start 2024-09-13T17:13:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 0x55c7807cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c780671860 con 0x55c7807ce4a0 2024-09-13T17:13:11.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c78077eb90 con 0x55c7807afb80 2024-09-13T17:13:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c7807884e0 con 0x55c7807886d0 2024-09-13T17:13:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.567+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41820/0 (socket says 172.21.15.28:41820) 2024-09-13T17:13:11.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 -- 172.21.15.28:0/2713064869 learned_addr learned my addr 172.21.15.28:0/2713064869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f001be00640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0020c00640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 0x55c7807cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 -- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 msgr2=0x55c7807cdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 0x55c7807cdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 -- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 msgr2=0x55c7807e5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807e5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 -- 172.21.15.28:0/2713064869 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c78071e170 con 0x55c7807ce4a0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f001be00640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807e5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0020c00640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 0x55c7807cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f0021600640 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0014009680 tx=0x7f001407f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efafd2e4c1c1ed31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f000e000640 1 -- 172.21.15.28:0/2713064869 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001407cdf0 con 0x55c7807ce4a0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f000e000640 1 -- 172.21.15.28:0/2713064869 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00140a6070 con 0x55c7807ce4a0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.568+0000 7f000e000640 1 -- 172.21.15.28:0/2713064869 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014077410 con 0x55c7807ce4a0 2024-09-13T17:13:11.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 -- 172.21.15.28:0/2713064869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 msgr2=0x55c7807e6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f0014009680 tx=0x7f001407f000 comp rx=0 tx=0).stop 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 -- 172.21.15.28:0/2713064869 shutdown_connections 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807ce4a0 0x55c7807e6470 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c7807e5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 --2- 172.21.15.28:0/2713064869 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807886d0 0x55c7807cdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 -- 172.21.15.28:0/2713064869 >> 172.21.15.28:0/2713064869 conn(0x55c780785e90 msgr2=0x55c780784420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 -- 172.21.15.28:0/2713064869 shutdown_connections 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.569+0000 7f0026404740 1 -- 172.21.15.28:0/2713064869 wait complete. 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.570+0000 7f0026404740 1 -- start start 2024-09-13T17:13:11.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.570+0000 7f0026404740 1 Processor -- start 2024-09-13T17:13:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c78082a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807ce4a0 0x55c78082a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7807ae0c0 con 0x55c7807886d0 2024-09-13T17:13:11.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7807ac390 con 0x55c7807afb80 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c78082aae0 con 0x55c7807ce4a0 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0026404740 10 client.0 _ll_register_callbacks cb 0x7f0010001df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41832/0 (socket says 172.21.15.28:41832) 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 -- 172.21.15.28:0/2453909770 learned_addr learned my addr 172.21.15.28:0/2453909770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f001be00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c78082a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0021600640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807ce4a0 0x55c78082a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 -- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807ce4a0 msgr2=0x55c78082a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807ce4a0 0x55c78082a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 -- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 msgr2=0x55c78082a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c78082a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f0020c00640 1 -- 172.21.15.28:0/2453909770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7806b5bd0 con 0x55c7807886d0 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.571+0000 7f001be00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c78082a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0020c00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0010000fd0 tx=0x7f0010014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f1ddd1906fb0a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0002000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:11.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010013700 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00100333f0 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00140813b0 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0010013860 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0026404740 10 client.7257 fetch_fsmap learned FSMap version 70 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.572+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x55c78068b470 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00140813b0 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f00100335b0 con 0x55c7807886d0 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 10 client.7257 fetch_fsmap finished waiting for FSMap version 70 2024-09-13T17:13:11.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 10 client.7257 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 20 client.7257 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c7807a7a50 0x55c7807a7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 4 client.7257 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 --> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c7803ef6c0 con 0x55c7807a7a50 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.573+0000 7f001be00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c7807a7a50 0x55c7807a7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.574+0000 7f001be00640 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c7807a7a50 0x55c7807a7e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5ec9b1da9c8ff11 server_cookie=7bc72604b00753a8 in_seq=0 out_seq=0 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.574+0000 7f0002000640 10 client.7257 ms_handle_connect on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0002000640 1 -- 172.21.15.28:0/2453909770 <== mds.0 v2:172.21.15.189:6834/1743829849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c7803ef6c0 con 0x55c7807a7a50 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0002000640 10 client.7257 handle_command_reply: tid=0 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 client.7257 shutdown 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 msgr2=0x55c7807a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0010000fd0 tx=0x7f0010014000 comp rx=0 tx=0).stop 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 shutdown_connections 2024-09-13T17:13:11.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x55c7807a7a50 0x55c7807a7e30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c7807ce4a0 0x55c78082a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c7807afb80 0x55c78082a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0002000640 0 client.7257 ms_handle_reset on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 --2- 172.21.15.28:0/2453909770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c7807886d0 0x55c7807a6410 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 >> 172.21.15.28:0/2453909770 conn(0x55c780785e90 msgr2=0x55c7807b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.575+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 shutdown_connections 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0026404740 1 -- 172.21.15.28:0/2453909770 wait complete. 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0026404740 20 client.7257 trim_cache size 0 max 16384 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:13:11.582 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7212, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.6774917790000003, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 70, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 81, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 81, 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 5.5223240669999996 2024-09-13T17:13:11.583 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 msgr2=0x7efffc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 0x7efffc06d0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0010000f60 tx=0x7f0010009040 comp rx=0 tx=0).stop 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 msgr2=0x7f001c14b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f0014009e50 tx=0x7f0014004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 shutdown_connections 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efffc06ac00 0x7efffc06d0c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f001c15ab60 0x7f001c14b960 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f001c15a780 0x7f001c145b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 --2- 172.21.15.28:0/4049437765 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f001c071490 0x7f001c1454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 >> 172.21.15.28:0/4049437765 conn(0x7f001c0b0270 msgr2=0x7f001c0cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 shutdown_connections 2024-09-13T17:13:11.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.576+0000 7f0023600640 1 -- 172.21.15.28:0/4049437765 wait complete. 2024-09-13T17:13:11.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:13:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.746+0000 7efd6d000640 1 Processor -- start 2024-09-13T17:13:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.747+0000 7efd6d000640 1 -- start start 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd6d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd680b0de0 0x7efd68071780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd6d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd68159d70 con 0x7efd6813c520 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd6d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6815a050 con 0x7efd68071cc0 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd6d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6815a330 con 0x7efd680b0de0 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd67400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd6813c520 0x7efd680b06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53304/0 (socket says 172.21.15.28:53304) 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd67400640 1 -- 172.21.15.28:0/2349247098 learned_addr learned my addr 172.21.15.28:0/2349247098 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.748+0000 7efd5ea00640 1 --2- 172.21.15.28:0/2349247098 >> v2:172.21.15.28:3300/0 conn(0x7efd680b0de0 0x7efd68071780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.749+0000 7efd5ea00640 1 -- 172.21.15.28:0/2349247098 >> v1:172.21.15.189:6789/0 conn(0x7efd68071cc0 legacy=0x7efd6813ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.749+0000 7efd5ea00640 1 -- 172.21.15.28:0/2349247098 >> v1:172.21.15.28:6789/0 conn(0x7efd6813c520 legacy=0x7efd680b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.749+0000 7efd5ea00640 1 -- 172.21.15.28:0/2349247098 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6815a610 con 0x7efd680b0de0 2024-09-13T17:13:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.749+0000 7efd66a00640 1 -- 172.21.15.28:0/2349247098 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467700181 0 0) 0x7efd68159d70 con 0x7efd6813c520 2024-09-13T17:13:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.750+0000 7efd5ea00640 1 --2- 172.21.15.28:0/2349247098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd680b0de0 0x7efd68071780 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7efd500097a0 tx=0x7efd5002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96c5211974ee0411 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.750+0000 7efd66a00640 1 -- 172.21.15.28:0/2349247098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50032440 con 0x7efd680b0de0 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.750+0000 7efd66a00640 1 -- 172.21.15.28:0/2349247098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd500325a0 con 0x7efd680b0de0 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.750+0000 7efd66a00640 1 -- 172.21.15.28:0/2349247098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50032930 con 0x7efd680b0de0 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 -- 172.21.15.28:0/2349247098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd680b0de0 msgr2=0x7efd68071780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 --2- 172.21.15.28:0/2349247098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd680b0de0 0x7efd68071780 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7efd500097a0 tx=0x7efd5002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 -- 172.21.15.28:0/2349247098 shutdown_connections 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 --2- 172.21.15.28:0/2349247098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd680b0de0 0x7efd68071780 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 -- 172.21.15.28:0/2349247098 >> 172.21.15.28:0/2349247098 conn(0x7efd68073550 msgr2=0x7efd6806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 -- 172.21.15.28:0/2349247098 shutdown_connections 2024-09-13T17:13:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.751+0000 7efd6d000640 1 -- 172.21.15.28:0/2349247098 wait complete. 2024-09-13T17:13:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.752+0000 7efd6d000640 1 Processor -- start 2024-09-13T17:13:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 -- start start 2024-09-13T17:13:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 0x7efd68158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 0x7efd68070400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681421e0 con 0x7efd6813c520 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd681424c0 con 0x7efd680b0de0 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.753+0000 7efd6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd681427a0 con 0x7efd68071cc0 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41858/0 (socket says 172.21.15.28:41858) 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 -- 172.21.15.28:0/2021371061 learned_addr learned my addr 172.21.15.28:0/2021371061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd5ea00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 0x7efd68070400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67400640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 0x7efd68158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 -- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 msgr2=0x7efd68158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 0x7efd68158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 -- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 msgr2=0x7efd68070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 0x7efd68070400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67e00640 1 -- 172.21.15.28:0/2021371061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd50009400 con 0x7efd6813c520 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd5ea00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 0x7efd68070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.754+0000 7efd67400640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 0x7efd68158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd67e00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c7a0 tx=0x7efd6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60003b70 con 0x7efd6813c520 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd60003cf0 con 0x7efd6813c520 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd60003ea0 con 0x7efd6813c520 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd68142a80 con 0x7efd6813c520 2024-09-13T17:13:11.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.755+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd68159190 con 0x7efd6813c520 2024-09-13T17:13:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.756+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd68071780 con 0x7efd6813c520 2024-09-13T17:13:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.757+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efd6000aa90 con 0x7efd6813c520 2024-09-13T17:13:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.758+0000 7efd5fe00640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 0x7efd4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.758+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efd60089230 con 0x7efd6813c520 2024-09-13T17:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.758+0000 7efd67400640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 0x7efd4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:11.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.759+0000 7efd67400640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 0x7efd4406d0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd58000f60 tx=0x7efd58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:11.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.762+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efd600543a0 con 0x7efd6813c520 2024-09-13T17:13:11.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.979+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efd680b2350 con 0x7efd6813c520 2024-09-13T17:13:11.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.979+0000 7efd5fe00640 1 -- 172.21.15.28:0/2021371061 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v70) v1 ==== 53+0+73 (secure 0 0 0) 0x7efd600579c0 con 0x7efd6813c520 2024-09-13T17:13:11.980 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 msgr2=0x7efd4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 0x7efd4406d0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7efd58000f60 tx=0x7efd58009040 comp rx=0 tx=0).stop 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 msgr2=0x7efd68070940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7efd6000c7a0 tx=0x7efd6000cc60 comp rx=0 tx=0).stop 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 shutdown_connections 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd4406ac00 0x7efd4406d0c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd6813c520 0x7efd68070940 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd680b0de0 0x7efd68070400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 --2- 172.21.15.28:0/2021371061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd68071cc0 0x7efd68158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 >> 172.21.15.28:0/2021371061 conn(0x7efd68073550 msgr2=0x7efd680b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 shutdown_connections 2024-09-13T17:13:11.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:11.982+0000 7efd6d000640 1 -- 172.21.15.28:0/2021371061 wait complete. 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:13:11.999 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:11.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.000 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.035 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42389sec preferred_lft 42389sec 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:12.036 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:13:12.037 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:13:12.038 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:12.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.038 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:13:12.129 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:13:12.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.132 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:13:12.168 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:13:12.168 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:13:12.168 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:13:12.168 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:13:12.169 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:13:12.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:13:12.169 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:13:12.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.315 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:13:12.350 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:13:12.351 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:13:12.351 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:12.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:13:12.448 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:12.448 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:12.483 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:12.483 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:12.483 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:12.558 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:13:12.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:12.687 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:12.687 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:12.715 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:12.715 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:12.715 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:13:12.715 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:13:12.716 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:13:12.717 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:13:12.718 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:13:12.718 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:13:12.718 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:13:12.718 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:13:12.720 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:13:12.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:12.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.777 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:13:12.815 INFO:teuthology.orchestra.run.smithi028.stdout:client7269 2024-09-13T17:13:12.816 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:12.816 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:12.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:12.891 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir scrub_repair_path) 2024-09-13T17:13:12.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:13:12.970 INFO:teuthology.orchestra.run.smithi028.stderr:rmdir: failed to remove 'scrub_repair_path': Directory not empty 2024-09-13T17:13:12.971 INFO:tasks.cephfs.test_scrub_checks:Running command 'scrub start /scrub_repair_path repair' 2024-09-13T17:13:12.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub start /scrub_repair_path repair 2024-09-13T17:13:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.169+0000 7f2d93600640 1 Processor -- start 2024-09-13T17:13:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.170+0000 7f2d93600640 1 -- start start 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 0x7f2d8c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c0cee40 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c06c950 con 0x7f2d8c15a7a0 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d93600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0b1ed0 con 0x7f2d8c0b1ab0 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d83e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d83e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41866/0 (socket says 172.21.15.28:41866) 2024-09-13T17:13:13.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d83e00640 1 -- 172.21.15.28:0/3480260329 learned_addr learned my addr 172.21.15.28:0/3480260329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d90c00640 1 --2- 172.21.15.28:0/3480260329 >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 0x7f2d8c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d91600640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d83e00640 1 -- 172.21.15.28:0/3480260329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 msgr2=0x7f2d8c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.171+0000 7f2d83e00640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.172+0000 7f2d83e00640 1 -- 172.21.15.28:0/3480260329 >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 msgr2=0x7f2d8c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.172+0000 7f2d83e00640 1 --2- 172.21.15.28:0/3480260329 >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 0x7f2d8c0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.172+0000 7f2d83e00640 1 -- 172.21.15.28:0/3480260329 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0b2150 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.172+0000 7f2d90c00640 1 --2- 172.21.15.28:0/3480260329 >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 0x7f2d8c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d83e00640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0716c0 tx=0x7f2d7402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc628839e033dd31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d8be00640 1 -- 172.21.15.28:0/3480260329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7402ae90 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d8be00640 1 -- 172.21.15.28:0/3480260329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d74032440 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d8be00640 1 -- 172.21.15.28:0/3480260329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740327d0 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 -- 172.21.15.28:0/3480260329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 msgr2=0x7f2d8c0b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0716c0 tx=0x7f2d7402a140 comp rx=0 tx=0).stop 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 -- 172.21.15.28:0/3480260329 shutdown_connections 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0b1570 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 --2- 172.21.15.28:0/3480260329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 --2- 172.21.15.28:0/3480260329 >> v2:172.21.15.189:3300/0 conn(0x7f2d8c15a7a0 0x7f2d8c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.173+0000 7f2d93600640 1 -- 172.21.15.28:0/3480260329 >> 172.21.15.28:0/3480260329 conn(0x7f2d8c069e80 msgr2=0x7f2d8c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.174+0000 7f2d93600640 1 -- 172.21.15.28:0/3480260329 shutdown_connections 2024-09-13T17:13:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.174+0000 7f2d93600640 1 -- 172.21.15.28:0/3480260329 wait complete. 2024-09-13T17:13:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.175+0000 7f2d93600640 1 Processor -- start 2024-09-13T17:13:13.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.175+0000 7f2d93600640 1 -- start start 2024-09-13T17:13:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c1f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8c15a7a0 0x7f2d8c14eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0c9140 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c1f2180 con 0x7f2d8c15a7a0 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d93600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0c88a0 con 0x7f2d8c0b1ab0 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41880/0 (socket says 172.21.15.28:41880) 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.176+0000 7f2d83e00640 1 -- 172.21.15.28:0/2039725880 learned_addr learned my addr 172.21.15.28:0/2039725880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d91600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8c15a7a0 0x7f2d8c14eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c1f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 -- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 msgr2=0x7f2d8c1f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c1f62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 -- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8c15a7a0 msgr2=0x7f2d8c14eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8c15a7a0 0x7f2d8c14eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 -- 172.21.15.28:0/2039725880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d740094d0 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c1f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.177+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2d7402a180 tx=0x7f2d740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.178+0000 7f2d89600640 1 -- 172.21.15.28:0/2039725880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74030850 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.178+0000 7f2d89600640 1 -- 172.21.15.28:0/2039725880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d740309b0 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.178+0000 7f2d89600640 1 -- 172.21.15.28:0/2039725880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74030d60 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.178+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c0c8b80 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.178+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c15c270 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.180+0000 7f2d89600640 1 -- 172.21.15.28:0/2039725880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2d74039040 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.181+0000 7f2d89600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 0x7f2d6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.181+0000 7f2d89600640 1 -- 172.21.15.28:0/2039725880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d740a9e70 con 0x7f2d8c0ce9d0 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.182+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 0x7f2d6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.182+0000 7f2d96453740 1 Processor -- start 2024-09-13T17:13:13.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.182+0000 7f2d96453740 1 -- start start 2024-09-13T17:13:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.182+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 0x7f2d6806d0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00a990 tx=0x7f2d7c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c07258260 0x558c07258640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c07255a20 0x558c072a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c071c1830 con 0x558c0727f710 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c071befc0 con 0x558c07258260 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c071ba8b0 con 0x558c07255a20 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41884/0 (socket says 172.21.15.28:41884) 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d91600640 1 -- 172.21.15.28:0/409857800 learned_addr learned my addr 172.21.15.28:0/409857800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d90c00640 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c07258260 0x558c07258640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.183+0000 7f2d83e00640 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c07255a20 0x558c072a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 -- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c07255a20 msgr2=0x558c072a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c07255a20 0x558c072a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 -- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c07258260 msgr2=0x558c07258640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c07258260 0x558c07258640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 -- 172.21.15.28:0/409857800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c07181c80 con 0x558c0727f710 2024-09-13T17:13:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d91600640 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f90 tx=0x7f2d84003e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba68110722ce8f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d81600640 1 -- 172.21.15.28:0/409857800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d84015420 con 0x558c0727f710 2024-09-13T17:13:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d81600640 1 -- 172.21.15.28:0/409857800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d84015580 con 0x558c0727f710 2024-09-13T17:13:13.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.184+0000 7f2d81600640 1 -- 172.21.15.28:0/409857800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d840158f0 con 0x558c0727f710 2024-09-13T17:13:13.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 -- 172.21.15.28:0/409857800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 msgr2=0x558c072a5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f2d84009f90 tx=0x7f2d84003e20 comp rx=0 tx=0).stop 2024-09-13T17:13:13.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 -- 172.21.15.28:0/409857800 shutdown_connections 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c0727f710 0x558c072a5c70 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c07255a20 0x558c072a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 --2- 172.21.15.28:0/409857800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c07258260 0x558c07258640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 -- 172.21.15.28:0/409857800 >> 172.21.15.28:0/409857800 conn(0x558c07254990 msgr2=0x558c0729ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 -- 172.21.15.28:0/409857800 shutdown_connections 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.185+0000 7f2d96453740 1 -- 172.21.15.28:0/409857800 wait complete. 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.186+0000 7f2d96453740 1 -- start start 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.186+0000 7f2d96453740 1 Processor -- start 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c07251cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c0715ba10 con 0x558c07258260 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c0715b6b0 con 0x558c072537a0 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c06eb8280 con 0x558c0727f710 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d96453740 10 client.0 _ll_register_callbacks cb 0x558c07143320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d90c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d90c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41890/0 (socket says 172.21.15.28:41890) 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d90c00640 1 -- 172.21.15.28:0/2625727389 learned_addr learned my addr 172.21.15.28:0/2625727389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.187+0000 7f2d91600640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c07251cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 -- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 msgr2=0x558c07251790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 -- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 msgr2=0x558c07251cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c07251cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 -- 172.21.15.28:0/2625727389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d84011040 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c014710 tx=0x7f2d7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5f9dec6d75c3c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d7a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c0158e0 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d7c015a40 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.188+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558c071e38f0 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d7c015c00 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 10 client.7281 fetch_fsmap learned FSMap version 70 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x558c071b8f30 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558c071e38f0 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d7c021420 con 0x558c07258260 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 10 client.7281 fetch_fsmap finished waiting for FSMap version 70 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 10 client.7281 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-09-13T17:13:13.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 20 client.7281 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c0727ad90 0x558c0727b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 4 client.7281 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:13:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 --> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558c071af250 con 0x558c0727ad90 2024-09-13T17:13:13.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.189+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c0727ad90 0x558c0727b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.190+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c0727ad90 0x558c0727b170 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37b2e0d8a2c4c629 server_cookie=ad89e7ba3b73e8f6 in_seq=0 out_seq=0 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.190+0000 7f2d7a000640 10 client.7281 ms_handle_connect on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.191+0000 7f2d7a000640 1 -- 172.21.15.28:0/2625727389 <== mds.0 v2:172.21.15.189:6834/1743829849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x558c071af250 con 0x558c0727ad90 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.191+0000 7f2d7a000640 10 client.7281 handle_command_reply: tid=0 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.191+0000 7f2d96453740 1 client.7281 shutdown 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.191+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 msgr2=0x558c07274880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.191+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c014710 tx=0x7f2d7c004730 comp rx=0 tx=0).stop 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 shutdown_connections 2024-09-13T17:13:13.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c0727ad90 0x558c0727b170 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c07251cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 --2- 172.21.15.28:0/2625727389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07274880 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d7a000640 0 client.7281 ms_handle_reset on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 >> 172.21.15.28:0/2625727389 conn(0x558c07255a20 msgr2=0x558c072a0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 shutdown_connections 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 1 -- 172.21.15.28:0/2625727389 wait complete. 2024-09-13T17:13:13.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.192+0000 7f2d96453740 20 client.7281 trim_cache size 0 max 16384 2024-09-13T17:13:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.207+0000 7f2d96453740 1 Processor -- start 2024-09-13T17:13:13.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.207+0000 7f2d96453740 1 -- start start 2024-09-13T17:13:13.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072fef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c07253010 con 0x558c07258260 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c072747b0 con 0x558c072537a0 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c0727b860 con 0x558c0727f710 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41906/0 (socket says 172.21.15.28:41906) 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 -- 172.21.15.28:0/918239343 learned_addr learned my addr 172.21.15.28:0/918239343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d91600640 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d90c00640 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072fef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 -- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 msgr2=0x558c07251cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 -- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 msgr2=0x558c072fef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072fef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.208+0000 7f2d83e00640 1 -- 172.21.15.28:0/918239343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558c071ecc40 con 0x558c07258260 2024-09-13T17:13:13.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d83e00640 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2d74078ce0 tx=0x7f2d74078e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59e0bd63fc1ab86 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d82000640 1 -- 172.21.15.28:0/918239343 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d740a83b0 con 0x558c07258260 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d82000640 1 -- 172.21.15.28:0/918239343 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d740770f0 con 0x558c07258260 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d82000640 1 -- 172.21.15.28:0/918239343 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74078fb0 con 0x558c07258260 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 -- 172.21.15.28:0/918239343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 msgr2=0x558c07251790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f2d74078ce0 tx=0x7f2d74078e90 comp rx=0 tx=0).stop 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 -- 172.21.15.28:0/918239343 shutdown_connections 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c07251cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c07251790 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 --2- 172.21.15.28:0/918239343 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.209+0000 7f2d96453740 1 -- 172.21.15.28:0/918239343 >> 172.21.15.28:0/918239343 conn(0x558c07255a20 msgr2=0x558c072540d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.210+0000 7f2d96453740 1 -- 172.21.15.28:0/918239343 shutdown_connections 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.210+0000 7f2d96453740 1 -- 172.21.15.28:0/918239343 wait complete. 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.210+0000 7f2d96453740 1 -- start start 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.210+0000 7f2d96453740 1 Processor -- start 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072750b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c0727d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c07274760 con 0x558c07258260 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558c07275680 con 0x558c072537a0 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558c07274510 con 0x558c0727f710 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d96453740 10 client.0 _ll_register_callbacks cb 0x7f2d84009bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.211+0000 7f2d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41920/0 (socket says 172.21.15.28:41920) 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 -- 172.21.15.28:0/2857083922 learned_addr learned my addr 172.21.15.28:0/2857083922 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d91600640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c0727d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072750b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 -- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 msgr2=0x558c0727d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c0727d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 -- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 msgr2=0x558c072750b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072750b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 -- 172.21.15.28:0/2857083922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c15c270 con 0x558c07258260 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d83e00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2d74080b60 tx=0x7f2d740a8200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=557509ed17ea83c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d7a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:13.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74078240 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.212+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2d740783a0 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d84015580 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d74078fb0 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 10 client.7287 fetch_fsmap learned FSMap version 70 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=70,monmap=2+}) v3 -- 0x558c0715ac00 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d84015580 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 70) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d74080c00 con 0x558c07258260 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 10 client.7287 fetch_fsmap finished waiting for FSMap version 70 2024-09-13T17:13:13.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 10 client.7287 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.d 2024-09-13T17:13:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 20 client.7287 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c072fe340 0x558c0727d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 4 client.7287 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_repair_path", "scrubops": ["repair"]}] 2024-09-13T17:13:13.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.213+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 --> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_repair_path", "scrubops": ["repair"]}) v1 -- 0x558c06e6b9f0 con 0x558c072fe340 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.214+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c072fe340 0x558c0727d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.214+0000 7f2d7a000640 10 client.7287 ms_handle_connect on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.214+0000 7f2d90c00640 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c072fe340 0x558c0727d5c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=367ec8ff218151e8 server_cookie=6d8b8b5b12a791ea in_seq=0 out_seq=0 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d7a000640 1 -- 172.21.15.28:0/2857083922 <== mds.0 v2:172.21.15.189:6834/1743829849 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558c06e6b9f0 con 0x558c072fe340 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d7a000640 10 client.7287 handle_command_reply: tid=0 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 client.7287 shutdown 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 msgr2=0x558c072fbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f2d74080b60 tx=0x7f2d740a8200 comp rx=0 tx=0).stop 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 shutdown_connections 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:6834/1743829849,v1:172.21.15.189:6836/1743829849] conn(0x558c072fe340 0x558c0727d5c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558c0727f710 0x558c0727d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558c07258260 0x558c072fbc00 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 --2- 172.21.15.28:0/2857083922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558c072537a0 0x558c072750b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d7a000640 0 client.7287 ms_handle_reset on v2:172.21.15.189:6834/1743829849 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 >> 172.21.15.28:0/2857083922 conn(0x558c07255a20 msgr2=0x558c0724df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 shutdown_connections 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 1 -- 172.21.15.28:0/2857083922 wait complete. 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.215+0000 7f2d96453740 20 client.7287 trim_cache size 0 max 16384 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "c8493436-dd0f-4d70-a2a1-f6dc6101773d", 2024-09-13T17:13:13.223 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:13:13.224 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 msgr2=0x7f2d6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 0x7f2d6806d0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00a990 tx=0x7f2d7c008040 comp rx=0 tx=0).stop 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 msgr2=0x7f2d8c0c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f2d7402a180 tx=0x7f2d740046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 shutdown_connections 2024-09-13T17:13:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6806ac00 0x7f2d6806d0c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8c15a7a0 0x7f2d8c14eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8c0ce9d0 0x7f2d8c0c8050 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 --2- 172.21.15.28:0/2039725880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d8c0b1ab0 0x7f2d8c1f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 >> 172.21.15.28:0/2039725880 conn(0x7f2d8c069e80 msgr2=0x7f2d8c157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 shutdown_connections 2024-09-13T17:13:13.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:13.216+0000 7f2d93600640 1 -- 172.21.15.28:0/2039725880 wait complete. 2024-09-13T17:13:13.232 INFO:tasks.cephfs.test_scrub_checks:command 'scrub start /scrub_repair_path repair' returned '{'return_code': 0, 'scrub_tag': 'c8493436-dd0f-4d70-a2a1-f6dc6101773d', 'mode': 'asynchronous'}' 2024-09-13T17:13:23.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:23.234 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir scrub_repair_path) 2024-09-13T17:13:23.278 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:13:23.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:13:23.410 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:13:23.513 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:13:23.513 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:23.513 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:23.550 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:23.551 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:23.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.821+0000 7effa1400640 1 Processor -- start 2024-09-13T17:13:23.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.821+0000 7effa1400640 1 -- start start 2024-09-13T17:13:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.822+0000 7effa1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06f970 0x7eff9c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.822+0000 7effa1400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff9c06c2d0 0x7eff9c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.822+0000 7effa1400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9c159d90 con 0x7eff9c1592a0 2024-09-13T17:13:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.822+0000 7effa1400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9c15b0b0 con 0x7eff9c06c2d0 2024-09-13T17:13:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7effa1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c06cca0 con 0x7eff9c06f970 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff9c06c2d0 0x7eff9c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff9c06c2d0 0x7eff9c06c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43262/0 (socket says 172.21.15.28:43262) 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 -- 172.21.15.28:0/1015569622 learned_addr learned my addr 172.21.15.28:0/1015569622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff91e00640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06f970 0x7eff9c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 -- 172.21.15.28:0/1015569622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06f970 msgr2=0x7eff9c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06f970 0x7eff9c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 -- 172.21.15.28:0/1015569622 >> v1:172.21.15.189:6789/0 conn(0x7eff9c1592a0 legacy=0x7eff9c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.823+0000 7eff9b200640 1 -- 172.21.15.28:0/1015569622 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c0b2040 con 0x7eff9c06c2d0 2024-09-13T17:13:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.824+0000 7eff9b200640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06c2d0 0x7eff9c06c760 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7eff94009e50 tx=0x7eff9402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c3f8ea7f060fc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7eff99e00640 1 -- 172.21.15.28:0/1015569622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94031600 con 0x7eff9c06c2d0 2024-09-13T17:13:23.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7eff99e00640 1 -- 172.21.15.28:0/1015569622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9402d050 con 0x7eff9c06c2d0 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7eff99e00640 1 -- 172.21.15.28:0/1015569622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff94031980 con 0x7eff9c06c2d0 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 -- 172.21.15.28:0/1015569622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06c2d0 msgr2=0x7eff9c06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06c2d0 0x7eff9c06c760 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7eff94009e50 tx=0x7eff9402a710 comp rx=0 tx=0).stop 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 -- 172.21.15.28:0/1015569622 shutdown_connections 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06c2d0 0x7eff9c06c760 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 --2- 172.21.15.28:0/1015569622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06f970 0x7eff9c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 -- 172.21.15.28:0/1015569622 >> 172.21.15.28:0/1015569622 conn(0x7eff9c069e80 msgr2=0x7eff9c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 -- 172.21.15.28:0/1015569622 shutdown_connections 2024-09-13T17:13:23.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.825+0000 7effa1400640 1 -- 172.21.15.28:0/1015569622 wait complete. 2024-09-13T17:13:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.827+0000 7effa1400640 1 Processor -- start 2024-09-13T17:13:23.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.827+0000 7effa1400640 1 -- start start 2024-09-13T17:13:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06c2d0 0x7eff9c153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9c1592a0 0x7eff9c0aedf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c1f30e0 con 0x7eff9c06f970 2024-09-13T17:13:23.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c156c30 con 0x7eff9c1592a0 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7effa1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c156f10 con 0x7eff9c06c2d0 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7eff91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7eff91e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46648/0 (socket says 172.21.15.28:46648) 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7eff91e00640 1 -- 172.21.15.28:0/1904270150 learned_addr learned my addr 172.21.15.28:0/1904270150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff9b200640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9c1592a0 0x7eff9c0aedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.828+0000 7eff9a800640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06c2d0 0x7eff9c153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 -- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06c2d0 msgr2=0x7eff9c153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06c2d0 0x7eff9c153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 -- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9c1592a0 msgr2=0x7eff9c0aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9c1592a0 0x7eff9c0aedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 -- 172.21.15.28:0/1904270150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff94009b30 con 0x7eff9c06f970 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff91e00640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e9b0 tx=0x7eff8400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:23.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84009670 con 0x7eff9c06f970 2024-09-13T17:13:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8400c9a0 con 0x7eff9c06f970 2024-09-13T17:13:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8400cd50 con 0x7eff9c06f970 2024-09-13T17:13:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.829+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9c1571f0 con 0x7eff9c06f970 2024-09-13T17:13:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.830+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9c1577b0 con 0x7eff9c06f970 2024-09-13T17:13:23.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.831+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9c0b3740 con 0x7eff9c06f970 2024-09-13T17:13:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.831+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff84013470 con 0x7eff9c06f970 2024-09-13T17:13:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.833+0000 7eff93400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 0x7eff7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:23.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.833+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7eff840896a0 con 0x7eff9c06f970 2024-09-13T17:13:23.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.833+0000 7eff9a800640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 0x7eff7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:23.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.834+0000 7eff9a800640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 0x7eff7806d0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7eff8c000f60 tx=0x7eff8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:23.837+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7eff84054830 con 0x7eff9c06f970 2024-09-13T17:13:24.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.015+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff9c157b50 con 0x7eff9c06f970 2024-09-13T17:13:24.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.016+0000 7eff93400640 1 -- 172.21.15.28:0/1904270150 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16697 (secure 0 0 0) 0x7eff84057e50 con 0x7eff9c06f970 2024-09-13T17:13:24.019 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:24.019 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:05.887982+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3037911251":"2024-09-14T17:13:05.880422+0000","172.21.15.28:6832/3037911251":"2024-09-14T17:13:05.880422+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.018+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 msgr2=0x7eff7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.018+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 0x7eff7806d0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7eff8c000f60 tx=0x7eff8c009040 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.018+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 msgr2=0x7eff9c1f2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.018+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7eff8400e9b0 tx=0x7eff8400ee70 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 shutdown_connections 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7806ac00 0x7eff7806d0c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9c1592a0 0x7eff9c0aedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff9c06f970 0x7eff9c1f2ba0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 --2- 172.21.15.28:0/1904270150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9c06c2d0 0x7eff9c153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 >> 172.21.15.28:0/1904270150 conn(0x7eff9c069e80 msgr2=0x7eff9c0cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 shutdown_connections 2024-09-13T17:13:24.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.019+0000 7effa1400640 1 -- 172.21.15.28:0/1904270150 wait complete. 2024-09-13T17:13:24.037 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:12:52.355786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:12:53.525437+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:24.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.194+0000 7f5f2fc00640 1 Processor -- start 2024-09-13T17:13:24.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.194+0000 7f5f2fc00640 1 -- start start 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f2815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2fc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2815ab60 con 0x7f5f2806f3f0 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2fc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2806f920 con 0x7f5f28159770 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2815b290 con 0x7f5f28070430 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2dc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f2806f3f0 0x7f5f2815c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34026/0 (socket says 172.21.15.28:34026) 2024-09-13T17:13:24.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.195+0000 7f5f2dc00640 1 -- 172.21.15.28:0/2716905303 learned_addr learned my addr 172.21.15.28:0/2716905303 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27e00640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f2815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186249988 0 0) 0x7f5f20001490 con 0x7f5f2806f3f0 2024-09-13T17:13:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f10003520 con 0x7f5f2806f3f0 2024-09-13T17:13:24.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572211465 0 0) 0x7f5f10003520 con 0x7f5f2806f3f0 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 msgr2=0x7f5f2815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f2815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 >> v1:172.21.15.189:6789/0 conn(0x7f5f28159770 legacy=0x7f5f2806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2815b570 con 0x7f5f2806f3f0 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f5f20001b80 con 0x7f5f2806f3f0 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 >> v1:172.21.15.28:6789/0 conn(0x7f5f2806f3f0 legacy=0x7f5f2815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f100039a0 0x7f5f10003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f10004350 0x7f5f10004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f10005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2815b290 con 0x7f5f10004350 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1000ba30 con 0x7f5f100039a0 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.196+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1000bbf0 con 0x7f5f28070430 2024-09-13T17:13:24.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f100039a0 0x7f5f10003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2dc00640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f10005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f27e00640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f10004350 0x7f5f10004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 -- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 msgr2=0x7f5f10005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f28070430 0x7f5f10005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 -- 172.21.15.28:0/2716905303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f10004350 msgr2=0x7f5f10004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f10004350 0x7f5f10004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.197+0000 7f5f2d200640 1 -- 172.21.15.28:0/2716905303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f10003520 con 0x7f5f100039a0 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.198+0000 7f5f2d200640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f100039a0 0x7f5f10003d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0018d0 tx=0x7f5f100057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26d816face397f0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c033600 con 0x7f5f100039a0 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c02f050 con 0x7f5f100039a0 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f27400640 1 -- 172.21.15.28:0/2716905303 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0339a0 con 0x7f5f100039a0 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2fc00640 1 -- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f100039a0 msgr2=0x7f5f10003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/2716905303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f100039a0 0x7f5f10003d80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c0018d0 tx=0x7f5f100057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2dc00640 1 -- 172.21.15.28:0/2716905303 reap_dead start 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2fc00640 1 -- 172.21.15.28:0/2716905303 shutdown_connections 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2fc00640 1 -- 172.21.15.28:0/2716905303 >> 172.21.15.28:0/2716905303 conn(0x7f5f280b0270 msgr2=0x7f5f280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.199+0000 7f5f2fc00640 1 -- 172.21.15.28:0/2716905303 shutdown_connections 2024-09-13T17:13:24.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.200+0000 7f5f2fc00640 1 -- 172.21.15.28:0/2716905303 wait complete. 2024-09-13T17:13:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.201+0000 7f5f2fc00640 1 Processor -- start 2024-09-13T17:13:24.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.201+0000 7f5f2fc00640 1 -- start start 2024-09-13T17:13:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f281f2650 0x7f5f281f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f281f0e10 0x7f5f281f1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280a9b70 con 0x7f5f100039a0 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280a8ef0 con 0x7f5f281f2650 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2fc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f280a91d0 con 0x7f5f281f0e10 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.202+0000 7f5f2d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46670/0 (socket says 172.21.15.28:46670) 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 -- 172.21.15.28:0/720478039 learned_addr learned my addr 172.21.15.28:0/720478039 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f27e00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f281f2650 0x7f5f281f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2dc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f281f0e10 0x7f5f281f1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 -- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f281f0e10 msgr2=0x7f5f281f1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f281f0e10 0x7f5f281f1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 -- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f281f2650 msgr2=0x7f5f281f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f281f2650 0x7f5f281f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 -- 172.21.15.28:0/720478039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f280a94b0 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.203+0000 7f5f2d200640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c02cc80 tx=0x7f5f1c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.204+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c02cee0 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.204+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c0054e0 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.204+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c003830 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.204+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f280a9790 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.204+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f281f17d0 con 0x7f5f100039a0 2024-09-13T17:13:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.205+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2806f920 con 0x7f5f100039a0 2024-09-13T17:13:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.206+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5f1c003990 con 0x7f5f100039a0 2024-09-13T17:13:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.206+0000 7f5f24c00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 0x7f5f0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.207+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5f1c0ac9f0 con 0x7f5f100039a0 2024-09-13T17:13:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.207+0000 7f5f27e00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 0x7f5f0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.208+0000 7f5f27e00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 0x7f5f0806d0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5f280aad80 tx=0x7f5f14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.211+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5f1c077ad0 con 0x7f5f100039a0 2024-09-13T17:13:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.390+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f28156b80 con 0x7f5f100039a0 2024-09-13T17:13:24.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.390+0000 7f5f24c00640 1 -- 172.21.15.28:0/720478039 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f1c07b0f0 con 0x7f5f100039a0 2024-09-13T17:13:24.393 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 msgr2=0x7f5f0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 0x7f5f0806d0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5f280aad80 tx=0x7f5f14008040 comp rx=0 tx=0).stop 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 msgr2=0x7f5f280c34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c02cc80 tx=0x7f5f1c001230 comp rx=0 tx=0).stop 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 shutdown_connections 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f0806ac00 0x7f5f0806d0c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f281f0e10 0x7f5f281f1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f281f2650 0x7f5f281f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 --2- 172.21.15.28:0/720478039 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f100039a0 0x7f5f280c34e0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 >> 172.21.15.28:0/720478039 conn(0x7f5f280b0270 msgr2=0x7f5f280ae780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 shutdown_connections 2024-09-13T17:13:24.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.393+0000 7f5f2fc00640 1 -- 172.21.15.28:0/720478039 wait complete. 2024-09-13T17:13:24.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:13:24.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.568+0000 7fa9d3200640 1 Processor -- start 2024-09-13T17:13:24.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.569+0000 7fa9d3200640 1 -- start start 2024-09-13T17:13:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9d414f950 0x7fa9d415a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d3200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa9d414fd30 0x7fa9d415c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40703e0 con 0x7fa9d4072210 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d40cd4e0 con 0x7fa9d414f950 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d3200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9d415a790 con 0x7fa9d414fd30 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9d414f950 0x7fa9d415a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9d414f950 0x7fa9d415a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46672/0 (socket says 172.21.15.28:46672) 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d1e00640 1 -- 172.21.15.28:0/3809616450 learned_addr learned my addr 172.21.15.28:0/3809616450 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.570+0000 7fa9d2800640 1 --2- 172.21.15.28:0/3809616450 >> v2:172.21.15.189:3300/0 conn(0x7fa9d414fd30 0x7fa9d415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.571+0000 7fa9d1e00640 1 -- 172.21.15.28:0/3809616450 >> v2:172.21.15.189:3300/0 conn(0x7fa9d414fd30 msgr2=0x7fa9d415c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.571+0000 7fa9d1e00640 1 --2- 172.21.15.28:0/3809616450 >> v2:172.21.15.189:3300/0 conn(0x7fa9d414fd30 0x7fa9d415c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.571+0000 7fa9d1e00640 1 -- 172.21.15.28:0/3809616450 >> v1:172.21.15.189:6789/0 conn(0x7fa9d4072210 legacy=0x7fa9d40ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.571+0000 7fa9d1e00640 1 -- 172.21.15.28:0/3809616450 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40cd740 con 0x7fa9d414f950 2024-09-13T17:13:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d1e00640 1 --2- 172.21.15.28:0/3809616450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d415a250 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b3af0 tx=0x7fa9c402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8066c6dc357a5ed8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d1400640 1 -- 172.21.15.28:0/3809616450 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4032440 con 0x7fa9d414f950 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d1400640 1 -- 172.21.15.28:0/3809616450 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c40325a0 con 0x7fa9d414f950 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d1400640 1 -- 172.21.15.28:0/3809616450 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4032930 con 0x7fa9d414f950 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d3200640 1 -- 172.21.15.28:0/3809616450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 msgr2=0x7fa9d415a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.572+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3809616450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d415a250 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b3af0 tx=0x7fa9c402a250 comp rx=0 tx=0).stop 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 -- 172.21.15.28:0/3809616450 shutdown_connections 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3809616450 >> v2:172.21.15.189:3300/0 conn(0x7fa9d414fd30 0x7fa9d415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3809616450 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d415a250 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 -- 172.21.15.28:0/3809616450 >> 172.21.15.28:0/3809616450 conn(0x7fa9d4069e80 msgr2=0x7fa9d406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 -- 172.21.15.28:0/3809616450 shutdown_connections 2024-09-13T17:13:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.573+0000 7fa9d3200640 1 -- 172.21.15.28:0/3809616450 wait complete. 2024-09-13T17:13:24.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.574+0000 7fa9d3200640 1 Processor -- start 2024-09-13T17:13:24.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.574+0000 7fa9d3200640 1 -- start start 2024-09-13T17:13:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9d4072210 0x7fa9d4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9d414fd30 0x7fa9d41501b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41581d0 con 0x7fa9d414f950 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41575c0 con 0x7fa9d4072210 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9d3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d41578a0 con 0x7fa9d414fd30 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46684/0 (socket says 172.21.15.28:46684) 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.575+0000 7fa9cbe00640 1 -- 172.21.15.28:0/3930088669 learned_addr learned my addr 172.21.15.28:0/3930088669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 -- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9d414fd30 msgr2=0x7fa9d41501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9d414fd30 0x7fa9d41501b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 -- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9d4072210 msgr2=0x7fa9d4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9d4072210 0x7fa9d4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 -- 172.21.15.28:0/3930088669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c40094d0 con 0x7fa9d414f950 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9cbe00640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00c430 tx=0x7fa9bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc011790 con 0x7fa9d414f950 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9bc0118f0 con 0x7fa9d414f950 2024-09-13T17:13:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc011ce0 con 0x7fa9d414f950 2024-09-13T17:13:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d4157b80 con 0x7fa9d414f950 2024-09-13T17:13:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.576+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d41571b0 con 0x7fa9d414f950 2024-09-13T17:13:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.577+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d406c220 con 0x7fa9d414f950 2024-09-13T17:13:24.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.578+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa9bc01c460 con 0x7fa9d414f950 2024-09-13T17:13:24.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.579+0000 7fa9ca000640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 0x7fa9b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.579+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa9bc0893b0 con 0x7fa9d414f950 2024-09-13T17:13:24.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.580+0000 7fa9d1e00640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 0x7fa9b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.580+0000 7fa9d1e00640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 0x7fa9b006d0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4034000 tx=0x7fa9c402c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.583+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa9bc054490 con 0x7fa9d414f950 2024-09-13T17:13:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.761+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa9d4157340 con 0x7fa9d414f950 2024-09-13T17:13:24.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.761+0000 7fa9ca000640 1 -- 172.21.15.28:0/3930088669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa9bc057ab0 con 0x7fa9d414f950 2024-09-13T17:13:24.763 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.763+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 msgr2=0x7fa9b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.763+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 0x7fa9b006d0c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4034000 tx=0x7fa9c402c000 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.763+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 msgr2=0x7fa9d4070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.763+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc00c430 tx=0x7fa9bc00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 shutdown_connections 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9b006ac00 0x7fa9b006d0c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9d414fd30 0x7fa9d41501b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9d414f950 0x7fa9d4070ae0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 --2- 172.21.15.28:0/3930088669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9d4072210 0x7fa9d4156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 >> 172.21.15.28:0/3930088669 conn(0x7fa9d4069e80 msgr2=0x7fa9d415bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 shutdown_connections 2024-09-13T17:13:24.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.764+0000 7fa9d3200640 1 -- 172.21.15.28:0/3930088669 wait complete. 2024-09-13T17:13:24.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:13:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.939+0000 7f4759800640 1 Processor -- start 2024-09-13T17:13:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.939+0000 7f4759800640 1 -- start start 2024-09-13T17:13:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4759800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f475415ff20 0x7f47540b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4759800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f475406b1a0 con 0x7f47540b11d0 2024-09-13T17:13:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4759800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540cf820 con 0x7f4754160300 2024-09-13T17:13:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4759800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47540cb5f0 con 0x7f475415ff20 2024-09-13T17:13:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4753e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f475415ff20 0x7f47540b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4758e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47540b11d0 0x7f47540cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34038/0 (socket says 172.21.15.28:34038) 2024-09-13T17:13:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.940+0000 7f4758e00640 1 -- 172.21.15.28:0/2945395143 learned_addr learned my addr 172.21.15.28:0/2945395143 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.941+0000 7f4753e00640 1 -- 172.21.15.28:0/2945395143 >> v1:172.21.15.189:6789/0 conn(0x7f4754160300 legacy=0x7f47540ce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.941+0000 7f4753e00640 1 -- 172.21.15.28:0/2945395143 >> v1:172.21.15.28:6789/0 conn(0x7f47540b11d0 legacy=0x7f47540cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.941+0000 7f4753e00640 1 -- 172.21.15.28:0/2945395143 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540cfb00 con 0x7f475415ff20 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.941+0000 7f4753400640 1 -- 172.21.15.28:0/2945395143 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906366782 0 0) 0x7f475406b1a0 con 0x7f47540b11d0 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4753e00640 1 --2- 172.21.15.28:0/2945395143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f475415ff20 0x7f47540b2c70 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f47440097a0 tx=0x7f474402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe8a862676ccf02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4753400640 1 -- 172.21.15.28:0/2945395143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474402ae50 con 0x7f475415ff20 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4753400640 1 -- 172.21.15.28:0/2945395143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4744032440 con 0x7f475415ff20 2024-09-13T17:13:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4753400640 1 -- 172.21.15.28:0/2945395143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47440327d0 con 0x7f475415ff20 2024-09-13T17:13:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4759800640 1 -- 172.21.15.28:0/2945395143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f475415ff20 msgr2=0x7f47540b2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.942+0000 7f4759800640 1 --2- 172.21.15.28:0/2945395143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f475415ff20 0x7f47540b2c70 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f47440097a0 tx=0x7f474402a140 comp rx=0 tx=0).stop 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.943+0000 7f4759800640 1 -- 172.21.15.28:0/2945395143 shutdown_connections 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.943+0000 7f4759800640 1 --2- 172.21.15.28:0/2945395143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f475415ff20 0x7f47540b2c70 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.943+0000 7f4759800640 1 -- 172.21.15.28:0/2945395143 >> 172.21.15.28:0/2945395143 conn(0x7f47540b2150 msgr2=0x7f47540b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.943+0000 7f4759800640 1 -- 172.21.15.28:0/2945395143 shutdown_connections 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.943+0000 7f4759800640 1 -- 172.21.15.28:0/2945395143 wait complete. 2024-09-13T17:13:24.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.944+0000 7f4759800640 1 Processor -- start 2024-09-13T17:13:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.944+0000 7f4759800640 1 -- start start 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47540b11d0 0x7f47540b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f475415ff20 0x7f4754152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b4720 con 0x7f47540b48b0 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b3990 con 0x7f47540b11d0 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.945+0000 7f4759800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4754153070 con 0x7f475415ff20 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46692/0 (socket says 172.21.15.28:46692) 2024-09-13T17:13:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 -- 172.21.15.28:0/1040604076 learned_addr learned my addr 172.21.15.28:0/1040604076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4753e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47540b11d0 0x7f47540b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f474b400640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f475415ff20 0x7f4754152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 -- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f475415ff20 msgr2=0x7f4754152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f475415ff20 0x7f4754152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 -- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47540b11d0 msgr2=0x7f47540b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47540b11d0 0x7f47540b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 -- 172.21.15.28:0/1040604076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4744009400 con 0x7f47540b48b0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.946+0000 7f4758e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f474c00c7e0 tx=0x7f474c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.947+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00a640 con 0x7f47540b48b0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.947+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c00a7c0 con 0x7f47540b48b0 2024-09-13T17:13:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.947+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474c00ab70 con 0x7f47540b48b0 2024-09-13T17:13:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.947+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47540adb30 con 0x7f47540b48b0 2024-09-13T17:13:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.947+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540b3e80 con 0x7f47540b48b0 2024-09-13T17:13:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.948+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475406e510 con 0x7f47540b48b0 2024-09-13T17:13:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.949+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f474c00acd0 con 0x7f47540b48b0 2024-09-13T17:13:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.949+0000 7f4750c00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 0x7f473006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.950+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f474c088850 con 0x7f47540b48b0 2024-09-13T17:13:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.950+0000 7f4753e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 0x7f473006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:24.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.951+0000 7f4753e00640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 0x7f473006d0f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4744034000 tx=0x7f474402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:24.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:24.954+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f474c0539e0 con 0x7f47540b48b0 2024-09-13T17:13:25.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.132+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f475406e230 con 0x7f47540b48b0 2024-09-13T17:13:25.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.133+0000 7f4750c00640 1 -- 172.21.15.28:0/1040604076 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f474c0180a0 con 0x7f47540b48b0 2024-09-13T17:13:25.135 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 msgr2=0x7f473006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 0x7f473006d0f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4744034000 tx=0x7f474402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 msgr2=0x7f475415b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f474c00c7e0 tx=0x7f474c00cca0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 shutdown_connections 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f473006ac30 0x7f473006d0f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47540b48b0 0x7f475415b6b0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f475415ff20 0x7f4754152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 --2- 172.21.15.28:0/1040604076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47540b11d0 0x7f47540b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 >> 172.21.15.28:0/1040604076 conn(0x7f47540b2150 msgr2=0x7f475406ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 shutdown_connections 2024-09-13T17:13:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.135+0000 7f4759800640 1 -- 172.21.15.28:0/1040604076 wait complete. 2024-09-13T17:13:25.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:13:25.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.307+0000 7fc598400640 1 Processor -- start 2024-09-13T17:13:25.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.307+0000 7fc598400640 1 -- start start 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc59015c9a0 0x7fc5900ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc5900ad940 0x7fc5900b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc59006f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900b26c0 con 0x7fc5900ad940 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5900ace30 con 0x7fc59015c9a0 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc598400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5900ad110 con 0x7fc5900b2230 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc595000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc5900ad940 0x7fc5900b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc595000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc5900ad940 0x7fc5900b1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46706/0 (socket says 172.21.15.28:46706) 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.308+0000 7fc595000640 1 -- 172.21.15.28:0/452256303 learned_addr learned my addr 172.21.15.28:0/452256303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595a00640 1 --2- 172.21.15.28:0/452256303 >> v2:172.21.15.189:3300/0 conn(0x7fc59015c9a0 0x7fc5900ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc596400640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc59006f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595000640 1 -- 172.21.15.28:0/452256303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 msgr2=0x7fc59006f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595000640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc59006f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595000640 1 -- 172.21.15.28:0/452256303 >> v2:172.21.15.189:3300/0 conn(0x7fc59015c9a0 msgr2=0x7fc5900ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595000640 1 --2- 172.21.15.28:0/452256303 >> v2:172.21.15.189:3300/0 conn(0x7fc59015c9a0 0x7fc5900ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.309+0000 7fc595000640 1 -- 172.21.15.28:0/452256303 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59006b170 con 0x7fc5900ad940 2024-09-13T17:13:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.310+0000 7fc595000640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5900b1cf0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009870 tx=0x7fc57c02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ab28d1075e6ad1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.310+0000 7fc58fe00640 1 -- 172.21.15.28:0/452256303 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c032440 con 0x7fc5900ad940 2024-09-13T17:13:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.310+0000 7fc58fe00640 1 -- 172.21.15.28:0/452256303 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c0325a0 con 0x7fc5900ad940 2024-09-13T17:13:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.310+0000 7fc58fe00640 1 -- 172.21.15.28:0/452256303 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c032930 con 0x7fc5900ad940 2024-09-13T17:13:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 -- 172.21.15.28:0/452256303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 msgr2=0x7fc5900b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5900b1cf0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009870 tx=0x7fc57c02a2a0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 -- 172.21.15.28:0/452256303 shutdown_connections 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5900b1cf0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 --2- 172.21.15.28:0/452256303 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc59006f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 --2- 172.21.15.28:0/452256303 >> v2:172.21.15.189:3300/0 conn(0x7fc59015c9a0 0x7fc5900ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 -- 172.21.15.28:0/452256303 >> 172.21.15.28:0/452256303 conn(0x7fc5900b0270 msgr2=0x7fc5900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 -- 172.21.15.28:0/452256303 shutdown_connections 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.311+0000 7fc598400640 1 -- 172.21.15.28:0/452256303 wait complete. 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.313+0000 7fc598400640 1 Processor -- start 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.313+0000 7fc598400640 1 -- start start 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.313+0000 7fc598400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc598400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc5901550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc598400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 0x7fc5900c51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc598400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590155c70 con 0x7fc5900ad940 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc598400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc590155f50 con 0x7fc59015c9a0 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc598400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc590156230 con 0x7fc5900b2230 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46720/0 (socket says 172.21.15.28:46720) 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 -- 172.21.15.28:0/2119558528 learned_addr learned my addr 172.21.15.28:0/2119558528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc596400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 0x7fc5900c51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595000640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc5901550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 -- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 msgr2=0x7fc5901550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc5901550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 -- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 msgr2=0x7fc5900c51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 0x7fc5900c51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595a00640 1 -- 172.21.15.28:0/2119558528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0094d0 con 0x7fc5900ad940 2024-09-13T17:13:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc595000640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc5901550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.314+0000 7fc596400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 0x7fc5900c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc595a00640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc58400efc0 tx=0x7fc58400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584014480 con 0x7fc5900ad940 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400f040 con 0x7fc5900ad940 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc590070930 con 0x7fc5900ad940 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584014800 con 0x7fc5900ad940 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.315+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5901567f0 con 0x7fc5900ad940 2024-09-13T17:13:25.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.316+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5900b1790 con 0x7fc5900ad940 2024-09-13T17:13:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.318+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc584014960 con 0x7fc5900ad940 2024-09-13T17:13:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.318+0000 7fc58d600640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 0x7fc57006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.318+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc584089f20 con 0x7fc5900ad940 2024-09-13T17:13:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.319+0000 7fc595000640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 0x7fc57006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.320+0000 7fc595000640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 0x7fc57006d0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc57c034000 tx=0x7fc57c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.322+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc584055090 con 0x7fc5900ad940 2024-09-13T17:13:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.513+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc590070760 con 0x7fc5900ad940 2024-09-13T17:13:25.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.513+0000 7fc58d600640 1 -- 172.21.15.28:0/2119558528 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v81) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc5840586b0 con 0x7fc5900ad940 2024-09-13T17:13:25.515 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6834/3037911251 2024-09-14T17:13:05.880422+0000 2024-09-13T17:13:25.516 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6832/3037911251 2024-09-14T17:13:05.880422+0000 2024-09-13T17:13:25.516 INFO:teuthology.orchestra.run.smithi028.stderr:listed 2 entries 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 msgr2=0x7fc57006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 0x7fc57006d0c0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc57c034000 tx=0x7fc57c02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 msgr2=0x7fc5901f6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fc58400efc0 tx=0x7fc58400c490 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 shutdown_connections 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc57006ac00 0x7fc57006d0c0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc59015c9a0 0x7fc5900c51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5900b2230 0x7fc5901550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 --2- 172.21.15.28:0/2119558528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc5900ad940 0x7fc5901f6160 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 >> 172.21.15.28:0/2119558528 conn(0x7fc5900b0270 msgr2=0x7fc590070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 shutdown_connections 2024-09-13T17:13:25.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.516+0000 7fc598400640 1 -- 172.21.15.28:0/2119558528 wait complete. 2024-09-13T17:13:25.534 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:13:25.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:25.534 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:25.598 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:13:25.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:25.599 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:13:25.635 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:13:25.635 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:25.675 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:13:25.675 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:25.675 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:25.710 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:13:25.710 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:13:25.710 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:25.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:13:25.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.907+0000 7fee20c00640 1 Processor -- start 2024-09-13T17:13:25.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.908+0000 7fee20c00640 1 -- start start 2024-09-13T17:13:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee20c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee1c156e00 0x7fee1c0b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee20c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee1c15d370 0x7fee1c15d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee20c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee1c06bde0 con 0x7fee1c156a20 2024-09-13T17:13:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee20c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c06b820 con 0x7fee1c15d370 2024-09-13T17:13:25.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee20c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1c06bb00 con 0x7fee1c156e00 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee1be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee1c15d370 0x7fee1c15d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee1be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee1c15d370 0x7fee1c15d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46728/0 (socket says 172.21.15.28:46728) 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.909+0000 7fee1be00640 1 -- 172.21.15.28:0/3867352832 learned_addr learned my addr 172.21.15.28:0/3867352832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.910+0000 7fee12a00640 1 --2- 172.21.15.28:0/3867352832 >> v2:172.21.15.189:3300/0 conn(0x7fee1c156e00 0x7fee1c0b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.910+0000 7fee1be00640 1 -- 172.21.15.28:0/3867352832 >> v2:172.21.15.189:3300/0 conn(0x7fee1c156e00 msgr2=0x7fee1c0b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.910+0000 7fee1be00640 1 --2- 172.21.15.28:0/3867352832 >> v2:172.21.15.189:3300/0 conn(0x7fee1c156e00 0x7fee1c0b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.910+0000 7fee1be00640 1 -- 172.21.15.28:0/3867352832 >> v1:172.21.15.189:6789/0 conn(0x7fee1c156a20 legacy=0x7fee1c071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.910+0000 7fee1be00640 1 -- 172.21.15.28:0/3867352832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1c15ce50 con 0x7fee1c15d370 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.911+0000 7fee1be00640 1 --2- 172.21.15.28:0/3867352832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c15d800 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fee1c15a810 tx=0x7fee1402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4110bff66113a872 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.911+0000 7fee1aa00640 1 -- 172.21.15.28:0/3867352832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14032440 con 0x7fee1c15d370 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.911+0000 7fee1aa00640 1 -- 172.21.15.28:0/3867352832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee140325a0 con 0x7fee1c15d370 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.911+0000 7fee1aa00640 1 -- 172.21.15.28:0/3867352832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14032930 con 0x7fee1c15d370 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 -- 172.21.15.28:0/3867352832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 msgr2=0x7fee1c15d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 --2- 172.21.15.28:0/3867352832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c15d800 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fee1c15a810 tx=0x7fee1402a270 comp rx=0 tx=0).stop 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 -- 172.21.15.28:0/3867352832 shutdown_connections 2024-09-13T17:13:25.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 --2- 172.21.15.28:0/3867352832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c15d800 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 --2- 172.21.15.28:0/3867352832 >> v2:172.21.15.189:3300/0 conn(0x7fee1c156e00 0x7fee1c0b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 -- 172.21.15.28:0/3867352832 >> 172.21.15.28:0/3867352832 conn(0x7fee1c06d4e0 msgr2=0x7fee1c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:25.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 -- 172.21.15.28:0/3867352832 shutdown_connections 2024-09-13T17:13:25.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.912+0000 7fee20c00640 1 -- 172.21.15.28:0/3867352832 wait complete. 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.913+0000 7fee20c00640 1 Processor -- start 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.914+0000 7fee20c00640 1 -- start start 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1c156a20 0x7fee1c06ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee1c156e00 0x7fee1c06efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c15a360 con 0x7fee1c15d370 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c142f60 con 0x7fee1c156a20 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee20c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1c159da0 con 0x7fee1c156e00 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46734/0 (socket says 172.21.15.28:46734) 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee1be00640 1 -- 172.21.15.28:0/3431615191 learned_addr learned my addr 172.21.15.28:0/3431615191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee1b400640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1c156a20 0x7fee1c06ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.915+0000 7fee12a00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee1c156e00 0x7fee1c06efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 -- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee1c156e00 msgr2=0x7fee1c06efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee1c156e00 0x7fee1c06efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 -- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1c156a20 msgr2=0x7fee1c06ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1c156a20 0x7fee1c06ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:25.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 -- 172.21.15.28:0/3431615191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee140094d0 con 0x7fee1c15d370 2024-09-13T17:13:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee1be00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fee1402a2b0 tx=0x7fee140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14032440 con 0x7fee1c15d370 2024-09-13T17:13:25.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee14032e90 con 0x7fee1c15d370 2024-09-13T17:13:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14030de0 con 0x7fee1c15d370 2024-09-13T17:13:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.916+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1c15a080 con 0x7fee1c15d370 2024-09-13T17:13:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.917+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1c148610 con 0x7fee1c15d370 2024-09-13T17:13:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.918+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1c15d800 con 0x7fee1c15d370 2024-09-13T17:13:25.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.918+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee14039040 con 0x7fee1c15d370 2024-09-13T17:13:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.919+0000 7fee13e00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 0x7fedf806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.919+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fee140a9c40 con 0x7fee1c15d370 2024-09-13T17:13:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.920+0000 7fee1b400640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 0x7fedf806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:25.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.920+0000 7fee1b400640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 0x7fedf806d0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0012e0 tx=0x7fee0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:25.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:25.923+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee14074e60 con 0x7fee1c15d370 2024-09-13T17:13:26.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.142+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee1c1430a0 con 0x7fee1c15d370 2024-09-13T17:13:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.143+0000 7fee13e00640 1 -- 172.21.15.28:0/3431615191 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fee14078480 con 0x7fee1c15d370 2024-09-13T17:13:26.145 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:26.145 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:09.912894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:26.145 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 70 2024-09-13T17:13:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 msgr2=0x7fedf806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 0x7fedf806d0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fee0c0012e0 tx=0x7fee0c009040 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 msgr2=0x7fee1c144f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fee1402a2b0 tx=0x7fee140046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 shutdown_connections 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf806ac00 0x7fedf806d0c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee1c15d370 0x7fee1c144f90 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee1c156e00 0x7fee1c06efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 --2- 172.21.15.28:0/3431615191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1c156a20 0x7fee1c06ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 >> 172.21.15.28:0/3431615191 conn(0x7fee1c06d4e0 msgr2=0x7fee1c0b3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 shutdown_connections 2024-09-13T17:13:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.145+0000 7fee20c00640 1 -- 172.21.15.28:0/3431615191 wait complete. 2024-09-13T17:13:26.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:26.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.318+0000 7fafcec00640 1 Processor -- start 2024-09-13T17:13:26.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.318+0000 7fafcec00640 1 -- start start 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafcec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fafc815a540 0x7fafc806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafcec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc815e100 con 0x7fafc80cabc0 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafcec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc815e3e0 con 0x7fafc8070cc0 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafcec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafc80adb00 con 0x7fafc815a540 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafc7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fafc80cabc0 0x7fafc815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34042/0 (socket says 172.21.15.28:34042) 2024-09-13T17:13:26.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.319+0000 7fafc7400640 1 -- 172.21.15.28:0/2068224279 learned_addr learned my addr 172.21.15.28:0/2068224279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2068224279 >> v2:172.21.15.189:3300/0 conn(0x7fafc815a540 0x7fafc806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983392314 0 0) 0x7fafc815e100 con 0x7fafc80cabc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac003520 con 0x7fafc80cabc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715458399 0 0) 0x7fafc815e3e0 con 0x7fafc8070cc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc815e100 con 0x7fafc8070cc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3648554686 0 0) 0x7fafac003520 con 0x7fafc80cabc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 >> v2:172.21.15.189:3300/0 conn(0x7fafc815a540 msgr2=0x7fafc806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 --2- 172.21.15.28:0/2068224279 >> v2:172.21.15.189:3300/0 conn(0x7fafc815a540 0x7fafc806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 >> v1:172.21.15.189:6789/0 conn(0x7fafc8070cc0 legacy=0x7fafc806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc815ef80 con 0x7fafc80cabc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fafb4001670 con 0x7fafc80cabc0 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 >> v1:172.21.15.28:6789/0 conn(0x7fafc80cabc0 legacy=0x7fafc815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafac003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafac0047c0 0x7fafac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafac0051d0 0x7fafac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00c0d0 con 0x7fafac0051d0 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00c240 con 0x7fafac003b50 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.320+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00c400 con 0x7fafac0047c0 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafccc00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafac0051d0 0x7fafac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafac003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7400640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafac0047c0 0x7fafac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 -- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafac0047c0 msgr2=0x7fafac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafac0047c0 0x7fafac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 -- 172.21.15.28:0/2068224279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafac0051d0 msgr2=0x7fafac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafac0051d0 0x7fafac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafc7e00640 1 -- 172.21.15.28:0/2068224279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac003520 con 0x7fafac003b50 2024-09-13T17:13:26.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafccc00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafac0051d0 0x7fafac00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:26.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.321+0000 7fafccc00640 1 -- 172.21.15.28:0/2068224279 reap_dead start 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.322+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafac003f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009ac0 tx=0x7fafac0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fbd711cbda4fc01 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031400 con 0x7fafac003b50 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb802d050 con 0x7fafac003b50 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafc6a00640 1 -- 172.21.15.28:0/2068224279 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031780 con 0x7fafac003b50 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 -- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 msgr2=0x7fafac003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafac003f50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fafb8009ac0 tx=0x7fafac0057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 -- 172.21.15.28:0/2068224279 shutdown_connections 2024-09-13T17:13:26.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 --2- 172.21.15.28:0/2068224279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafac003f50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 -- 172.21.15.28:0/2068224279 >> 172.21.15.28:0/2068224279 conn(0x7fafc8069e80 msgr2=0x7fafc806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 -- 172.21.15.28:0/2068224279 shutdown_connections 2024-09-13T17:13:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.323+0000 7fafcec00640 1 -- 172.21.15.28:0/2068224279 wait complete. 2024-09-13T17:13:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.325+0000 7fafcec00640 1 Processor -- start 2024-09-13T17:13:26.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.325+0000 7fafcec00640 1 -- start start 2024-09-13T17:13:26.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafc814abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafc806fe40 0x7fafc8157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8152080 con 0x7fafc81507b0 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8157f80 con 0x7fafac003b50 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafcec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc81587e0 con 0x7fafc806fe40 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafccc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46752/0 (socket says 172.21.15.28:46752) 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.326+0000 7fafccc00640 1 -- 172.21.15.28:0/2468171833 learned_addr learned my addr 172.21.15.28:0/2468171833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 -- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafc806fe40 msgr2=0x7fafc8157080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafc806fe40 0x7fafc8157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafc814abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 -- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 msgr2=0x7fafc814abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafc814abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 -- 172.21.15.28:0/2468171833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc814b120 con 0x7fafc81507b0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafc814abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafccc00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fafc806d950 tx=0x7fafc00021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0018070 con 0x7fafc81507b0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafc0003b80 con 0x7fafc81507b0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafc0003d10 con 0x7fafc81507b0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc81579e0 con 0x7fafc81507b0 2024-09-13T17:13:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.327+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc8158470 con 0x7fafc81507b0 2024-09-13T17:13:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.328+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc806c7c0 con 0x7fafc81507b0 2024-09-13T17:13:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.329+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fafc0006b50 con 0x7fafc81507b0 2024-09-13T17:13:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.330+0000 7fafbfe00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 0x7fafa806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.330+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fafc0088d70 con 0x7fafc81507b0 2024-09-13T17:13:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.331+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 0x7fafa806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.331+0000 7fafc7e00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 0x7fafa806d0f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fafb8033000 tx=0x7fafb8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.335+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fafc0053ee0 con 0x7fafc81507b0 2024-09-13T17:13:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.514+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafc8158920 con 0x7fafc81507b0 2024-09-13T17:13:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.515+0000 7fafbfe00640 1 -- 172.21.15.28:0/2468171833 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16697 (secure 0 0 0) 0x7fafc0001bf0 con 0x7fafc81507b0 2024-09-13T17:13:26.517 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:26.517 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:05.887982+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3037911251":"2024-09-14T17:13:05.880422+0000","172.21.15.28:6832/3037911251":"2024-09-14T17:13:05.880422+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:26.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 msgr2=0x7fafa806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 0x7fafa806d0f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fafb8033000 tx=0x7fafb8008030 comp rx=0 tx=0).stop 2024-09-13T17:13:26.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 msgr2=0x7fafc8150c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fafc806d950 tx=0x7fafc00021d0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 shutdown_connections 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafa806ac30 0x7fafa806d0f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafc81507b0 0x7fafc8150c60 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafc806fe40 0x7fafc8157080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 --2- 172.21.15.28:0/2468171833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafac003b50 0x7fafc814abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 >> 172.21.15.28:0/2468171833 conn(0x7fafc8069e80 msgr2=0x7fafc80cd010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 shutdown_connections 2024-09-13T17:13:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.517+0000 7fafcec00640 1 -- 172.21.15.28:0/2468171833 wait complete. 2024-09-13T17:13:26.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:12:52.355786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:12:53.525437+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '77', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:26.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:26.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.694+0000 7f6d5ce00640 1 Processor -- start 2024-09-13T17:13:26.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.694+0000 7f6d5ce00640 1 -- start start 2024-09-13T17:13:26.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.695+0000 7f6d5ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d580b2400 0x7f6d580ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d5ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d580ad940 0x7f6d5806bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d5ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d58070470 con 0x7f6d5815c9a0 2024-09-13T17:13:26.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d5ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d580ace30 con 0x7f6d580ad940 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d5ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d580ad110 con 0x7f6d580b2400 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d580ad940 0x7f6d5806bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d580ad940 0x7f6d5806bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46766/0 (socket says 172.21.15.28:46766) 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 -- 172.21.15.28:0/1869847641 learned_addr learned my addr 172.21.15.28:0/1869847641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d55400640 1 --2- 172.21.15.28:0/1869847641 >> v2:172.21.15.189:3300/0 conn(0x7f6d580b2400 0x7f6d580ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 -- 172.21.15.28:0/1869847641 >> v2:172.21.15.189:3300/0 conn(0x7f6d580b2400 msgr2=0x7f6d580ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 --2- 172.21.15.28:0/1869847641 >> v2:172.21.15.189:3300/0 conn(0x7f6d580b2400 0x7f6d580ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 -- 172.21.15.28:0/1869847641 >> v1:172.21.15.189:6789/0 conn(0x7f6d5815c9a0 legacy=0x7f6d580b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.696+0000 7f6d56800640 1 -- 172.21.15.28:0/1869847641 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5806b170 con 0x7f6d580ad940 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.697+0000 7f6d56800640 1 --2- 172.21.15.28:0/1869847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d580ad940 0x7f6d5806bc50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6d580ce450 tx=0x7f6d5002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc2131ca9e326a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d54a00640 1 -- 172.21.15.28:0/1869847641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50031600 con 0x7f6d580ad940 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d54a00640 1 -- 172.21.15.28:0/1869847641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5002d050 con 0x7f6d580ad940 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d54a00640 1 -- 172.21.15.28:0/1869847641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d50031980 con 0x7f6d580ad940 2024-09-13T17:13:26.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 -- 172.21.15.28:0/1869847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d580ad940 msgr2=0x7f6d5806bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/1869847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d580ad940 0x7f6d5806bc50 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f6d580ce450 tx=0x7f6d5002a730 comp rx=0 tx=0).stop 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 -- 172.21.15.28:0/1869847641 shutdown_connections 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/1869847641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d580ad940 0x7f6d5806bc50 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/1869847641 >> v2:172.21.15.189:3300/0 conn(0x7f6d580b2400 0x7f6d580ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 -- 172.21.15.28:0/1869847641 >> 172.21.15.28:0/1869847641 conn(0x7f6d580b0270 msgr2=0x7f6d580b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 -- 172.21.15.28:0/1869847641 shutdown_connections 2024-09-13T17:13:26.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.698+0000 7f6d5ce00640 1 -- 172.21.15.28:0/1869847641 wait complete. 2024-09-13T17:13:26.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.700+0000 7f6d5ce00640 1 Processor -- start 2024-09-13T17:13:26.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.700+0000 7f6d5ce00640 1 -- start start 2024-09-13T17:13:26.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 0x7f6d581e28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 0x7f6d581e7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d581e7c80 con 0x7f6d5815c9a0 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d58154560 con 0x7f6d580ad940 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d5ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d581e2ed0 con 0x7f6d580b2400 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d56800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d56800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46772/0 (socket says 172.21.15.28:46772) 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d56800640 1 -- 172.21.15.28:0/4175117459 learned_addr learned my addr 172.21.15.28:0/4175117459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d55e00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 0x7f6d581e28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.701+0000 7f6d55400640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 0x7f6d581e7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 -- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 msgr2=0x7f6d581e7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 0x7f6d581e7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 -- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 msgr2=0x7f6d581e28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 0x7f6d581e28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 -- 172.21.15.28:0/4175117459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d50009b30 con 0x7f6d5815c9a0 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d55400640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 0x7f6d581e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d55e00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 0x7f6d581e28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d56800640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f6d50034040 tx=0x7f6d50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5003d860 con 0x7f6d5815c9a0 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5003d9c0 con 0x7f6d5815c9a0 2024-09-13T17:13:26.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5003dd70 con 0x7f6d5815c9a0 2024-09-13T17:13:26.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d58070930 con 0x7f6d5815c9a0 2024-09-13T17:13:26.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.702+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d581e3490 con 0x7f6d5815c9a0 2024-09-13T17:13:26.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.703+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5815cd80 con 0x7f6d5815c9a0 2024-09-13T17:13:26.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.704+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6d5003b400 con 0x7f6d5815c9a0 2024-09-13T17:13:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.705+0000 7f6d4e000640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 0x7f6d3806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.706+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6d500aa300 con 0x7f6d5815c9a0 2024-09-13T17:13:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.706+0000 7f6d55e00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 0x7f6d3806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:26.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.707+0000 7f6d55e00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 0x7f6d3806d0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6d48000f60 tx=0x7f6d48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:26.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.710+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6d500753e0 con 0x7f6d5815c9a0 2024-09-13T17:13:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.886+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6d5815b240 con 0x7f6d5815c9a0 2024-09-13T17:13:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.887+0000 7f6d4e000640 1 -- 172.21.15.28:0/4175117459 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6d50009510 con 0x7f6d5815c9a0 2024-09-13T17:13:26.889 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 msgr2=0x7f6d3806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 0x7f6d3806d0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6d48000f60 tx=0x7f6d48009040 comp rx=0 tx=0).stop 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 msgr2=0x7f6d58154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f6d50034040 tx=0x7f6d50004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 shutdown_connections 2024-09-13T17:13:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d3806ac00 0x7f6d3806d0c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d5815c9a0 0x7f6d58154020 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d580b2400 0x7f6d581e7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 --2- 172.21.15.28:0/4175117459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d580ad940 0x7f6d581e28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 >> 172.21.15.28:0/4175117459 conn(0x7f6d580b0270 msgr2=0x7f6d58159e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 shutdown_connections 2024-09-13T17:13:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:26.889+0000 7f6d5ce00640 1 -- 172.21.15.28:0/4175117459 wait complete. 2024-09-13T17:13:26.907 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:13:27.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.063+0000 7f4134a00640 1 Processor -- start 2024-09-13T17:13:27.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.063+0000 7f4134a00640 1 -- start start 2024-09-13T17:13:27.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 0x7f4130070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41300cf290 0x7f41300711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f413015b9d0 con 0x7f41300cf290 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41300b2090 con 0x7f41300cee50 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4134a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413015ae50 con 0x7f4130158080 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41300cf290 0x7f41300711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f41300cf290 0x7f41300711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46786/0 (socket says 172.21.15.28:46786) 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 -- 172.21.15.28:0/1064660662 learned_addr learned my addr 172.21.15.28:0/1064660662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f4127e00640 1 --2- 172.21.15.28:0/1064660662 >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 0x7f4130070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412e800640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 -- 172.21.15.28:0/1064660662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 msgr2=0x7f41300b1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300b1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 -- 172.21.15.28:0/1064660662 >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 msgr2=0x7f4130070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:27.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.065+0000 7f412f200640 1 --2- 172.21.15.28:0/1064660662 >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 0x7f4130070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.066+0000 7f412f200640 1 -- 172.21.15.28:0/1064660662 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f413015b130 con 0x7f41300cf290 2024-09-13T17:13:27.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.066+0000 7f4127e00640 1 --2- 172.21.15.28:0/1064660662 >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 0x7f4130070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f412f200640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cf290 0x7f41300711f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f41300cfad0 tx=0x7f412802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8496296b2ad4368 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f412de00640 1 -- 172.21.15.28:0/1064660662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128031600 con 0x7f41300cf290 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f412de00640 1 -- 172.21.15.28:0/1064660662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412802d050 con 0x7f41300cf290 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f412de00640 1 -- 172.21.15.28:0/1064660662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128031980 con 0x7f41300cf290 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 -- 172.21.15.28:0/1064660662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cf290 msgr2=0x7f41300711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cf290 0x7f41300711f0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f41300cfad0 tx=0x7f412802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 -- 172.21.15.28:0/1064660662 shutdown_connections 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cf290 0x7f41300711f0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 --2- 172.21.15.28:0/1064660662 >> v2:172.21.15.189:3300/0 conn(0x7f41300cee50 0x7f4130070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 --2- 172.21.15.28:0/1064660662 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300b1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.067+0000 7f4134a00640 1 -- 172.21.15.28:0/1064660662 >> 172.21.15.28:0/1064660662 conn(0x7f4130069e80 msgr2=0x7f413006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.068+0000 7f4134a00640 1 -- 172.21.15.28:0/1064660662 shutdown_connections 2024-09-13T17:13:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.068+0000 7f4134a00640 1 -- 172.21.15.28:0/1064660662 wait complete. 2024-09-13T17:13:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.069+0000 7f4134a00640 1 Processor -- start 2024-09-13T17:13:27.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.069+0000 7f4134a00640 1 -- start start 2024-09-13T17:13:27.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41300cf290 0x7f41300ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41300c8660 con 0x7f41300cee50 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4130156ee0 con 0x7f41300cf290 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f4134a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41300c8fc0 con 0x7f4130158080 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f412e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f412e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46788/0 (socket says 172.21.15.28:46788) 2024-09-13T17:13:27.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.070+0000 7f412e800640 1 -- 172.21.15.28:0/4209954561 learned_addr learned my addr 172.21.15.28:0/4209954561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4127e00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41300cf290 0x7f41300ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412f200640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 -- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 msgr2=0x7f41300adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300adf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 -- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41300cf290 msgr2=0x7f41300ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41300cf290 0x7f41300ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 -- 172.21.15.28:0/4209954561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4128009b30 con 0x7f41300cee50 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f412e800640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f412000d830 tx=0x7f412000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000b660 con 0x7f41300cee50 2024-09-13T17:13:27.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412000b7c0 con 0x7f41300cee50 2024-09-13T17:13:27.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412000bb50 con 0x7f41300cee50 2024-09-13T17:13:27.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4130156640 con 0x7f41300cee50 2024-09-13T17:13:27.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.071+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4130156c00 con 0x7f41300cee50 2024-09-13T17:13:27.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.072+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41300b1b50 con 0x7f41300cee50 2024-09-13T17:13:27.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.074+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f412001e460 con 0x7f41300cee50 2024-09-13T17:13:27.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.074+0000 7f4126a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 0x7f410c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.075+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f412008a580 con 0x7f41300cee50 2024-09-13T17:13:27.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.075+0000 7f4127e00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 0x7f410c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.076+0000 7f4127e00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 0x7f410c06d0f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f413006d630 tx=0x7f4118008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.078+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4120055770 con 0x7f41300cee50 2024-09-13T17:13:27.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.256+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4130154eb0 con 0x7f41300cee50 2024-09-13T17:13:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.257+0000 7f4126a00640 1 -- 172.21.15.28:0/4209954561 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4120058d90 con 0x7f41300cee50 2024-09-13T17:13:27.259 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:13:27.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 msgr2=0x7f410c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 0x7f410c06d0f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f413006d630 tx=0x7f4118008040 comp rx=0 tx=0).stop 2024-09-13T17:13:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 msgr2=0x7f41300c88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f412000d830 tx=0x7f412000dcf0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 shutdown_connections 2024-09-13T17:13:27.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f410c06ac30 0x7f410c06d0f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4130158080 0x7f41300adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41300cf290 0x7f41300ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 --2- 172.21.15.28:0/4209954561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41300cee50 0x7f41300c88e0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 >> 172.21.15.28:0/4209954561 conn(0x7f4130069e80 msgr2=0x7f41300608f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 shutdown_connections 2024-09-13T17:13:27.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.259+0000 7f4134a00640 1 -- 172.21.15.28:0/4209954561 wait complete. 2024-09-13T17:13:27.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:13:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.434+0000 7fc3fc800640 1 Processor -- start 2024-09-13T17:13:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.434+0000 7fc3fc800640 1 -- start start 2024-09-13T17:13:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.435+0000 7fc3fc800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3fc800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 0x7fc3f406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3fc800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f406c8b0 con 0x7fc3f415a780 2024-09-13T17:13:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3fc800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f406cb90 con 0x7fc3f415ab60 2024-09-13T17:13:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3fc800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40cc980 con 0x7fc3f4071490 2024-09-13T17:13:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3f9400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3fa800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 0x7fc3f406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3f9400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48170/0 (socket says 172.21.15.28:48170) 2024-09-13T17:13:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3f9400640 1 -- 172.21.15.28:0/2870248207 learned_addr learned my addr 172.21.15.28:0/2870248207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.436+0000 7fc3f9e00640 1 --1- 172.21.15.28:0/2870248207 >> v1:172.21.15.189:6789/0 conn(0x7fc3f415a780 0x7fc3f4070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60554/0 (socket says 172.21.15.28:60554) 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3f8a00640 1 -- 172.21.15.28:0/2870248207 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422886490 0 0) 0x7fc3f406c8b0 con 0x7fc3f415a780 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3f8a00640 1 -- 172.21.15.28:0/2870248207 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3dc003520 con 0x7fc3f415a780 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3fa800640 1 -- 172.21.15.28:0/2870248207 >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 msgr2=0x7fc3f406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3fa800640 1 --2- 172.21.15.28:0/2870248207 >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3fa800640 1 -- 172.21.15.28:0/2870248207 >> v1:172.21.15.189:6789/0 conn(0x7fc3f415a780 legacy=0x7fc3f4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3fa800640 1 -- 172.21.15.28:0/2870248207 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40ccc60 con 0x7fc3f4071490 2024-09-13T17:13:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.437+0000 7fc3f9400640 1 --2- 172.21.15.28:0/2870248207 >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.438+0000 7fc3fa800640 1 --2- 172.21.15.28:0/2870248207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 0x7fc3f406c080 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8c575f9747697ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.438+0000 7fc3f8a00640 1 -- 172.21.15.28:0/2870248207 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0007740 con 0x7fc3f4071490 2024-09-13T17:13:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.438+0000 7fc3f8a00640 1 -- 172.21.15.28:0/2870248207 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f002d050 con 0x7fc3f4071490 2024-09-13T17:13:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.438+0000 7fc3f8a00640 1 -- 172.21.15.28:0/2870248207 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0007ac0 con 0x7fc3f4071490 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 -- 172.21.15.28:0/2870248207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 msgr2=0x7fc3f406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 --2- 172.21.15.28:0/2870248207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 0x7fc3f406c080 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002a660 comp rx=0 tx=0).stop 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 -- 172.21.15.28:0/2870248207 shutdown_connections 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 --2- 172.21.15.28:0/2870248207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f4071490 0x7fc3f406c080 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 --2- 172.21.15.28:0/2870248207 >> v2:172.21.15.189:3300/0 conn(0x7fc3f415ab60 0x7fc3f406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 -- 172.21.15.28:0/2870248207 >> 172.21.15.28:0/2870248207 conn(0x7fc3f40b0270 msgr2=0x7fc3f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 -- 172.21.15.28:0/2870248207 shutdown_connections 2024-09-13T17:13:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.439+0000 7fc3fc800640 1 -- 172.21.15.28:0/2870248207 wait complete. 2024-09-13T17:13:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.441+0000 7fc3fc800640 1 Processor -- start 2024-09-13T17:13:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.441+0000 7fc3fc800640 1 -- start start 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f4071490 0x7fc3f41536b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f415a780 0x7fc3f4149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40a9180 con 0x7fc3f415ab60 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4153480 con 0x7fc3f4071490 2024-09-13T17:13:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fc800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40a8570 con 0x7fc3f415a780 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46790/0 (socket says 172.21.15.28:46790) 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 -- 172.21.15.28:0/1294824518 learned_addr learned my addr 172.21.15.28:0/1294824518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3f9e00640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f4071490 0x7fc3f41536b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3f9400640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f415a780 0x7fc3f4149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 -- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f415a780 msgr2=0x7fc3f4149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f415a780 0x7fc3f4149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 -- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f4071490 msgr2=0x7fc3f41536b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f4071490 0x7fc3f41536b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.442+0000 7fc3fa800640 1 -- 172.21.15.28:0/1294824518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40094d0 con 0x7fc3f415ab60 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3fa800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009eb0 tx=0x7fc3f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003c830 con 0x7fc3f415ab60 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f003c990 con 0x7fc3f415ab60 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003cd00 con 0x7fc3f415ab60 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f0009a70 con 0x7fc3f415ab60 2024-09-13T17:13:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.443+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f40a8b30 con 0x7fc3f415ab60 2024-09-13T17:13:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.444+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f40a92f0 con 0x7fc3f415ab60 2024-09-13T17:13:27.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.445+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc3f0031400 con 0x7fc3f415ab60 2024-09-13T17:13:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.446+0000 7fc3ee000640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 0x7fc3d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.446+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc3f00aaf80 con 0x7fc3f415ab60 2024-09-13T17:13:27.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.446+0000 7fc3f9e00640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 0x7fc3d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.447+0000 7fc3f9e00640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 0x7fc3d806d0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80011e0 tx=0x7fc3e8007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.450+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc3f0076210 con 0x7fc3f415ab60 2024-09-13T17:13:27.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.627+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3f406c8b0 con 0x7fc3f415ab60 2024-09-13T17:13:27.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.628+0000 7fc3ee000640 1 -- 172.21.15.28:0/1294824518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3f003ce60 con 0x7fc3f415ab60 2024-09-13T17:13:27.630 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:13:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 msgr2=0x7fc3d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 0x7fc3d806d0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80011e0 tx=0x7fc3e8007b50 comp rx=0 tx=0).stop 2024-09-13T17:13:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 msgr2=0x7fc3f40a9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009eb0 tx=0x7fc3f0004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 shutdown_connections 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d806ac00 0x7fc3d806d0c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f415ab60 0x7fc3f40a9ff0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f415a780 0x7fc3f4149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 --2- 172.21.15.28:0/1294824518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f4071490 0x7fc3f41536b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 >> 172.21.15.28:0/1294824518 conn(0x7fc3f40b0270 msgr2=0x7fc3f4159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 shutdown_connections 2024-09-13T17:13:27.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.630+0000 7fc3fc800640 1 -- 172.21.15.28:0/1294824518 wait complete. 2024-09-13T17:13:27.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.802+0000 7fe38e800640 1 Processor -- start 2024-09-13T17:13:27.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.802+0000 7fe38e800640 1 -- start start 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe38e800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe38806f970 0x7fe38815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe38e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe38806c2d0 0x7fe38806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe38e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38806cc70 con 0x7fe3881592a0 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe38e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38815b0b0 con 0x7fe38806c2d0 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe38e800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe388159d90 con 0x7fe38806f970 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe387e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe38806c2d0 0x7fe38806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.803+0000 7fe387400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3881592a0 0x7fe388159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34050/0 (socket says 172.21.15.28:34050) 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe387400640 1 -- 172.21.15.28:0/2407702332 learned_addr learned my addr 172.21.15.28:0/2407702332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:27.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe386a00640 1 --2- 172.21.15.28:0/2407702332 >> v2:172.21.15.189:3300/0 conn(0x7fe38806f970 0x7fe38815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe387e00640 1 -- 172.21.15.28:0/2407702332 >> v2:172.21.15.189:3300/0 conn(0x7fe38806f970 msgr2=0x7fe38815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe387e00640 1 --2- 172.21.15.28:0/2407702332 >> v2:172.21.15.189:3300/0 conn(0x7fe38806f970 0x7fe38815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe387e00640 1 -- 172.21.15.28:0/2407702332 >> v1:172.21.15.28:6789/0 conn(0x7fe3881592a0 legacy=0x7fe388159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.804+0000 7fe387e00640 1 -- 172.21.15.28:0/2407702332 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3880b2040 con 0x7fe38806c2d0 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.805+0000 7fe387e00640 1 --2- 172.21.15.28:0/2407702332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38806c2d0 0x7fe38806c730 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe38806ac10 tx=0x7fe38002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1707325b56d9ba39 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.805+0000 7fe386000640 1 -- 172.21.15.28:0/2407702332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38002aea0 con 0x7fe38806c2d0 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.805+0000 7fe386000640 1 -- 172.21.15.28:0/2407702332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38002d050 con 0x7fe38806c2d0 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.805+0000 7fe386000640 1 -- 172.21.15.28:0/2407702332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380007960 con 0x7fe38806c2d0 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 -- 172.21.15.28:0/2407702332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38806c2d0 msgr2=0x7fe38806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 --2- 172.21.15.28:0/2407702332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38806c2d0 0x7fe38806c730 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fe38806ac10 tx=0x7fe38002a680 comp rx=0 tx=0).stop 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 -- 172.21.15.28:0/2407702332 shutdown_connections 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 --2- 172.21.15.28:0/2407702332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe38806c2d0 0x7fe38806c730 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 --2- 172.21.15.28:0/2407702332 >> v2:172.21.15.189:3300/0 conn(0x7fe38806f970 0x7fe38815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 -- 172.21.15.28:0/2407702332 >> 172.21.15.28:0/2407702332 conn(0x7fe388069e80 msgr2=0x7fe38806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 -- 172.21.15.28:0/2407702332 shutdown_connections 2024-09-13T17:13:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.806+0000 7fe38e800640 1 -- 172.21.15.28:0/2407702332 wait complete. 2024-09-13T17:13:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.808+0000 7fe38e800640 1 Processor -- start 2024-09-13T17:13:27.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.808+0000 7fe38e800640 1 -- start start 2024-09-13T17:13:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe38806c2d0 0x7fe3880cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe38806f970 0x7fe388150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe38814e580 con 0x7fe3881592a0 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880b00b0 con 0x7fe38806c2d0 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe38e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3880b0390 con 0x7fe38806f970 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46814/0 (socket says 172.21.15.28:46814) 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 -- 172.21.15.28:0/3240635790 learned_addr learned my addr 172.21.15.28:0/3240635790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387400640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe38806c2d0 0x7fe3880cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe386a00640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe38806f970 0x7fe388150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 -- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe38806f970 msgr2=0x7fe388150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe38806f970 0x7fe388150f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 -- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe38806c2d0 msgr2=0x7fe3880cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe38806c2d0 0x7fe3880cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:27.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.809+0000 7fe387e00640 1 -- 172.21.15.28:0/3240635790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe380009a70 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe387e00640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe38002ac70 tx=0x7fe380004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38003c830 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38003c990 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38003cd00 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe38814e210 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.810+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3880b0c90 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.811+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe38814e6f0 con 0x7fe3881592a0 2024-09-13T17:13:27.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.812+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe380031400 con 0x7fe3881592a0 2024-09-13T17:13:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.813+0000 7fe37f400640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 0x7fe36806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.813+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe3800aaf80 con 0x7fe3881592a0 2024-09-13T17:13:27.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.814+0000 7fe387400640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 0x7fe36806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.814+0000 7fe387400640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 0x7fe36806d0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe378005880 tx=0x7fe378009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:27.817+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe380076210 con 0x7fe3881592a0 2024-09-13T17:13:28.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.033+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3880b3740 con 0x7fe3881592a0 2024-09-13T17:13:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.034+0000 7fe37f400640 1 -- 172.21.15.28:0/3240635790 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fe38003ce60 con 0x7fe3881592a0 2024-09-13T17:13:28.036 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:28.036 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:09.912894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:28.036 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 70 2024-09-13T17:13:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 msgr2=0x7fe36806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 0x7fe36806d0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fe378005880 tx=0x7fe378009040 comp rx=0 tx=0).stop 2024-09-13T17:13:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 msgr2=0x7fe3880af660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fe38002ac70 tx=0x7fe380004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 shutdown_connections 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe36806ac00 0x7fe36806d0c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3881592a0 0x7fe3880af660 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe38806f970 0x7fe388150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 --2- 172.21.15.28:0/3240635790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe38806c2d0 0x7fe3880cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 >> 172.21.15.28:0/3240635790 conn(0x7fe388069e80 msgr2=0x7fe38806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 shutdown_connections 2024-09-13T17:13:28.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.036+0000 7fe38e800640 1 -- 172.21.15.28:0/3240635790 wait complete. 2024-09-13T17:13:28.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.210+0000 7f654e600640 1 Processor -- start 2024-09-13T17:13:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.210+0000 7f654e600640 1 -- start start 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654806b610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f654806f480 0x7f654806f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654806b3f0 con 0x7f654815c8d0 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654806fe50 con 0x7f654806f480 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654806bb50 con 0x7f65480ccb60 2024-09-13T17:13:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f654806f480 0x7f654806f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f654806f480 0x7f654806f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46824/0 (socket says 172.21.15.28:46824) 2024-09-13T17:13:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.211+0000 7f654dc00640 1 -- 172.21.15.28:0/3978138733 learned_addr learned my addr 172.21.15.28:0/3978138733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f653fe00640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654806b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f654dc00640 1 -- 172.21.15.28:0/3978138733 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 msgr2=0x7f654806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f654dc00640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654806b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f654dc00640 1 -- 172.21.15.28:0/3978138733 >> v1:172.21.15.189:6789/0 conn(0x7f654815c8d0 legacy=0x7f654806ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f654dc00640 1 -- 172.21.15.28:0/3978138733 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65480b38a0 con 0x7f654806f480 2024-09-13T17:13:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.212+0000 7f653fe00640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654806b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:28.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.213+0000 7f654dc00640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654806f480 0x7f654806f910 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6540009e50 tx=0x7f654002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32614ad9596da659 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.213+0000 7f6547e00640 1 -- 172.21.15.28:0/3978138733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540031600 con 0x7f654806f480 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.213+0000 7f6547e00640 1 -- 172.21.15.28:0/3978138733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654002d050 con 0x7f654806f480 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.213+0000 7f6547e00640 1 -- 172.21.15.28:0/3978138733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540031980 con 0x7f654806f480 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 -- 172.21.15.28:0/3978138733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654806f480 msgr2=0x7f654806f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654806f480 0x7f654806f910 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6540009e50 tx=0x7f654002a6d0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 -- 172.21.15.28:0/3978138733 shutdown_connections 2024-09-13T17:13:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654806f480 0x7f654806f910 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 --2- 172.21.15.28:0/3978138733 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654806b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 -- 172.21.15.28:0/3978138733 >> 172.21.15.28:0/3978138733 conn(0x7f654806d4e0 msgr2=0x7f654806d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 -- 172.21.15.28:0/3978138733 shutdown_connections 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.214+0000 7f654e600640 1 -- 172.21.15.28:0/3978138733 wait complete. 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.215+0000 7f654e600640 1 Processor -- start 2024-09-13T17:13:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.216+0000 7f654e600640 1 -- start start 2024-09-13T17:13:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.216+0000 7f654e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654806f480 0x7f654813dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.216+0000 7f654e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654815eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654815ea90 con 0x7f654815c8d0 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654813e4e0 con 0x7f654806f480 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654815de10 con 0x7f65480ccb60 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46840/0 (socket says 172.21.15.28:46840) 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 -- 172.21.15.28:0/3856384581 learned_addr learned my addr 172.21.15.28:0/3856384581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f653fe00640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654815eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654d200640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654806f480 0x7f654813dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 -- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 msgr2=0x7f654815eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654815eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 -- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654806f480 msgr2=0x7f654813dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654806f480 0x7f654813dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.217+0000 7f654dc00640 1 -- 172.21.15.28:0/3856384581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540009b30 con 0x7f654815c8d0 2024-09-13T17:13:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f654dc00640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f6540034040 tx=0x7f6540004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654003d860 con 0x7f654815c8d0 2024-09-13T17:13:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654003d9c0 con 0x7f654815c8d0 2024-09-13T17:13:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654003dd70 con 0x7f654815c8d0 2024-09-13T17:13:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654815e0f0 con 0x7f654815c8d0 2024-09-13T17:13:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.218+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654815e6b0 con 0x7f654815c8d0 2024-09-13T17:13:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.219+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f654815ebd0 con 0x7f654815c8d0 2024-09-13T17:13:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.220+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f654003b400 con 0x7f654815c8d0 2024-09-13T17:13:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.221+0000 7f6545600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 0x7f652406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.222+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f65400aa120 con 0x7f654815c8d0 2024-09-13T17:13:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.222+0000 7f654d200640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 0x7f652406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.223+0000 7f654d200640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 0x7f652406d0c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6538006fd0 tx=0x7f6538009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.226+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f654815ebd0 con 0x7f654815c8d0 2024-09-13T17:13:28.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.407+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f654815ebd0 con 0x7f654815c8d0 2024-09-13T17:13:28.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.409+0000 7f6545600640 1 -- 172.21.15.28:0/3856384581 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16697 (secure 0 0 0) 0x7f654815ebd0 con 0x7f654815c8d0 2024-09-13T17:13:28.411 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:28.411 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:05.887982+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3037911251":"2024-09-14T17:13:05.880422+0000","172.21.15.28:6832/3037911251":"2024-09-14T17:13:05.880422+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 msgr2=0x7f652406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 0x7f652406d0c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f6538006fd0 tx=0x7f6538009060 comp rx=0 tx=0).stop 2024-09-13T17:13:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 msgr2=0x7f654814bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f6540034040 tx=0x7f6540004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 shutdown_connections 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f652406ac00 0x7f652406d0c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f654815c8d0 0x7f654814bc60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65480ccb60 0x7f654815eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 --2- 172.21.15.28:0/3856384581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f654806f480 0x7f654813dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 >> 172.21.15.28:0/3856384581 conn(0x7f654806d4e0 msgr2=0x7f65480b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 shutdown_connections 2024-09-13T17:13:28.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.411+0000 7f654e600640 1 -- 172.21.15.28:0/3856384581 wait complete. 2024-09-13T17:13:28.429 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:13:28.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:28.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.583+0000 7f63d7a00640 1 Processor -- start 2024-09-13T17:13:28.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.583+0000 7f63d7a00640 1 -- start start 2024-09-13T17:13:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.584+0000 7f63d7a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63d01592a0 0x7f63d015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.584+0000 7f63d7a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d0159d90 con 0x7f63d006c2d0 2024-09-13T17:13:28.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.584+0000 7f63d7a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00b2040 con 0x7f63d006f970 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.584+0000 7f63d7a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d015b0b0 con 0x7f63d01592a0 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63d01592a0 0x7f63d015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63d006c2d0 0x7f63d00b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34056/0 (socket says 172.21.15.28:34056) 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5a00640 1 -- 172.21.15.28:0/488265586 learned_addr learned my addr 172.21.15.28:0/488265586 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63d01592a0 0x7f63d015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46856/0 (socket says 172.21.15.28:46856) 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5000640 1 -- 172.21.15.28:0/488265586 >> v1:172.21.15.189:6789/0 conn(0x7f63d006f970 legacy=0x7f63d0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:28.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5000640 1 -- 172.21.15.28:0/488265586 >> v1:172.21.15.28:6789/0 conn(0x7f63d006c2d0 legacy=0x7f63d00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63cf400640 1 -- 172.21.15.28:0/488265586 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929301364 0 0) 0x7f63d0159d90 con 0x7f63d006c2d0 2024-09-13T17:13:28.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.585+0000 7f63d5000640 1 -- 172.21.15.28:0/488265586 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0159ff0 con 0x7f63d01592a0 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.586+0000 7f63d5000640 1 --2- 172.21.15.28:0/488265586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d015ab70 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f63c40097a0 tx=0x7f63c402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c092388a7f91f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.586+0000 7f63cf400640 1 -- 172.21.15.28:0/488265586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032440 con 0x7f63d01592a0 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.586+0000 7f63cf400640 1 -- 172.21.15.28:0/488265586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c40325a0 con 0x7f63d01592a0 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.586+0000 7f63cf400640 1 -- 172.21.15.28:0/488265586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c4032930 con 0x7f63d01592a0 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 -- 172.21.15.28:0/488265586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 msgr2=0x7f63d015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 --2- 172.21.15.28:0/488265586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d015ab70 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f63c40097a0 tx=0x7f63c402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 -- 172.21.15.28:0/488265586 shutdown_connections 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 --2- 172.21.15.28:0/488265586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d015ab70 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 -- 172.21.15.28:0/488265586 >> 172.21.15.28:0/488265586 conn(0x7f63d0069e80 msgr2=0x7f63d006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 -- 172.21.15.28:0/488265586 shutdown_connections 2024-09-13T17:13:28.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.587+0000 7f63d7a00640 1 -- 172.21.15.28:0/488265586 wait complete. 2024-09-13T17:13:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.589+0000 7f63d7a00640 1 Processor -- start 2024-09-13T17:13:28.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.589+0000 7f63d7a00640 1 -- start start 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63d006c2d0 0x7f63d014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 0x7f63d01e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0152620 con 0x7f63d01592a0 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0152870 con 0x7f63d006c2d0 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d7a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0152b50 con 0x7f63d006f970 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46870/0 (socket says 172.21.15.28:46870) 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 -- 172.21.15.28:0/342439338 learned_addr learned my addr 172.21.15.28:0/342439338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:28.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5000640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63d006c2d0 0x7f63d014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63cfe00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 0x7f63d01e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 -- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 msgr2=0x7f63d01e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 0x7f63d01e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 -- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63d006c2d0 msgr2=0x7f63d014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63d006c2d0 0x7f63d014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63d5a00640 1 -- 172.21.15.28:0/342439338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4009400 con 0x7f63d01592a0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.590+0000 7f63cfe00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 0x7f63d01e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63d5a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f63c8002230 tx=0x7f63c800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c800ac30 con 0x7f63d01592a0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c800ad90 con 0x7f63d01592a0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8012620 con 0x7f63d01592a0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0152e30 con 0x7f63d01592a0 2024-09-13T17:13:28.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.591+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d01533f0 con 0x7f63d01592a0 2024-09-13T17:13:28.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.592+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0153610 con 0x7f63d01592a0 2024-09-13T17:13:28.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.593+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f63c8007070 con 0x7f63d01592a0 2024-09-13T17:13:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.594+0000 7f63ccc00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 0x7f63b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.594+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63c80891e0 con 0x7f63d01592a0 2024-09-13T17:13:28.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.595+0000 7f63d5000640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 0x7f63b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.595+0000 7f63d5000640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 0x7f63b006d0c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63c4034000 tx=0x7f63c402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.598+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f63c80544a0 con 0x7f63d01592a0 2024-09-13T17:13:28.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.812+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63d015ab70 con 0x7f63d01592a0 2024-09-13T17:13:28.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.813+0000 7f63ccc00640 1 -- 172.21.15.28:0/342439338 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f63c8057ac0 con 0x7f63d01592a0 2024-09-13T17:13:28.815 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:28.815 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:09.912894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:28.816 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 70 2024-09-13T17:13:28.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 msgr2=0x7f63b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 0x7f63b006d0c0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63c4034000 tx=0x7f63c402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:28.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 msgr2=0x7f63d01e57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f63c8002230 tx=0x7f63c800cf50 comp rx=0 tx=0).stop 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 shutdown_connections 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63b006ac00 0x7f63b006d0c0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63d01592a0 0x7f63d01e57a0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63d006f970 0x7f63d01e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 --2- 172.21.15.28:0/342439338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63d006c2d0 0x7f63d014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 >> 172.21.15.28:0/342439338 conn(0x7f63d0069e80 msgr2=0x7f63d0158120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 shutdown_connections 2024-09-13T17:13:28.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.815+0000 7f63d7a00640 1 -- 172.21.15.28:0/342439338 wait complete. 2024-09-13T17:13:28.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.987+0000 7f4889800640 1 Processor -- start 2024-09-13T17:13:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.987+0000 7f4889800640 1 -- start start 2024-09-13T17:13:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.988+0000 7f4889800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.988+0000 7f4889800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.988+0000 7f4889800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48840b2040 con 0x7f488406c2d0 2024-09-13T17:13:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.988+0000 7f4889800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488415b0b0 con 0x7f488406f970 2024-09-13T17:13:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.988+0000 7f4889800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4884159ff0 con 0x7f48841592a0 2024-09-13T17:13:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:34408/0 (socket says 172.21.15.28:34408) 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48236/0 (socket says 172.21.15.28:48236) 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 -- 172.21.15.28:0/4227699799 learned_addr learned my addr 172.21.15.28:0/4227699799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 -- 172.21.15.28:0/4227699799 >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 msgr2=0x7f4884159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 --2- 172.21.15.28:0/4227699799 >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 -- 172.21.15.28:0/4227699799 >> v1:172.21.15.189:6789/0 conn(0x7f488406c2d0 legacy=0x7f48840b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882a00640 1 -- 172.21.15.28:0/4227699799 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488415a2d0 con 0x7f48841592a0 2024-09-13T17:13:28.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.989+0000 7f4882000640 1 --2- 172.21.15.28:0/4227699799 >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.990+0000 7f4882a00640 1 --2- 172.21.15.28:0/4227699799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee1c629b5aadd9d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.991+0000 7f4881600640 1 -- 172.21.15.28:0/4227699799 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874032440 con 0x7f48841592a0 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.991+0000 7f4881600640 1 -- 172.21.15.28:0/4227699799 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48740325a0 con 0x7f48841592a0 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.991+0000 7f4881600640 1 -- 172.21.15.28:0/4227699799 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4874032930 con 0x7f48841592a0 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.991+0000 7f4889800640 1 -- 172.21.15.28:0/4227699799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 msgr2=0x7f488415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.991+0000 7f4889800640 1 --2- 172.21.15.28:0/4227699799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4874009870 tx=0x7f487402a270 comp rx=0 tx=0).stop 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 -- 172.21.15.28:0/4227699799 shutdown_connections 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 --2- 172.21.15.28:0/4227699799 >> v2:172.21.15.189:3300/0 conn(0x7f488406f970 0x7f4884159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 --2- 172.21.15.28:0/4227699799 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48841592a0 0x7f488415ab70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 -- 172.21.15.28:0/4227699799 >> 172.21.15.28:0/4227699799 conn(0x7f4884069e80 msgr2=0x7f488406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 -- 172.21.15.28:0/4227699799 shutdown_connections 2024-09-13T17:13:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.992+0000 7f4889800640 1 -- 172.21.15.28:0/4227699799 wait complete. 2024-09-13T17:13:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.993+0000 7f4889800640 1 Processor -- start 2024-09-13T17:13:28.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.993+0000 7f4889800640 1 -- start start 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.994+0000 7f4889800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 0x7f488414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4889800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488406f970 0x7f48841e3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4889800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 0x7f48841e41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4889800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488414a5f0 con 0x7f48841592a0 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4889800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48841e53f0 con 0x7f488406c2d0 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4889800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48841e47e0 con 0x7f488406f970 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 0x7f48841e41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4882a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 0x7f488414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4882000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488406f970 0x7f48841e3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:28.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4882a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 0x7f488414a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48250/0 (socket says 172.21.15.28:48250) 2024-09-13T17:13:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4882a00640 1 -- 172.21.15.28:0/3747038837 learned_addr learned my addr 172.21.15.28:0/3747038837 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:28.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 -- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488406f970 msgr2=0x7f48841e3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488406f970 0x7f48841e3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 -- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 msgr2=0x7f488414a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 0x7f488414a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.995+0000 7f4883400640 1 -- 172.21.15.28:0/3747038837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4870009400 con 0x7f48841592a0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f4883400640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 0x7f48841e41d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c690 tx=0x7f487c00cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c003b70 con 0x7f48841592a0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c003cd0 con 0x7f48841592a0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c005310 con 0x7f48841592a0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48740094d0 con 0x7f48841592a0 2024-09-13T17:13:28.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.996+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48841e4da0 con 0x7f48841592a0 2024-09-13T17:13:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.997+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48841e5580 con 0x7f48841592a0 2024-09-13T17:13:29.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.998+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f487c005470 con 0x7f48841592a0 2024-09-13T17:13:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.999+0000 7f487aa00640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 0x7f486406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:28.999+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f487c0890d0 con 0x7f48841592a0 2024-09-13T17:13:29.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.000+0000 7f4882a00640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 0x7f486406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.000+0000 7f4882a00640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 0x7f486406d0c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4874035040 tx=0x7f487402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.003+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f487c054260 con 0x7f48841592a0 2024-09-13T17:13:29.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.220+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f488415ab70 con 0x7f48841592a0 2024-09-13T17:13:29.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.220+0000 7f487aa00640 1 -- 172.21.15.28:0/3747038837 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f487c057880 con 0x7f48841592a0 2024-09-13T17:13:29.222 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:29.223 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":70,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:12:54.534325+0000","modified":"2024-09-13T17:13:09.912894+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"d","rank":0,"incarnation":67,"state":"up:active","state_seq":4,"addr":"172.21.15.189:6836/1743829849","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1743829849},{"type":"v1","addr":"172.21.15.189:6836","nonce":1743829849}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T17:13:29.223 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 70 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 msgr2=0x7f486406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 0x7f486406d0c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f4874035040 tx=0x7f487402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 msgr2=0x7f48841e41d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 0x7f48841e41d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f487c00c690 tx=0x7f487c00cb50 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 shutdown_connections 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486406ac00 0x7f486406d0c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48841592a0 0x7f48841e41d0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488406f970 0x7f48841e3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 --2- 172.21.15.28:0/3747038837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488406c2d0 0x7f488414a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 >> 172.21.15.28:0/3747038837 conn(0x7f4884069e80 msgr2=0x7f48840af290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 shutdown_connections 2024-09-13T17:13:29.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.223+0000 7f4889800640 1 -- 172.21.15.28:0/3747038837 wait complete. 2024-09-13T17:13:29.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.395+0000 7f288ce00640 1 Processor -- start 2024-09-13T17:13:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.395+0000 7f288ce00640 1 -- start start 2024-09-13T17:13:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.396+0000 7f288ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.396+0000 7f288ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 0x7f2888071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.396+0000 7f288ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288815e100 con 0x7f28880cabc0 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.396+0000 7f288ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28880716b0 con 0x7f2888070cc0 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.396+0000 7f288ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28880adb00 con 0x7f288815a540 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2886800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 0x7f2888071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f287fe00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f28880cabc0 0x7f288815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60604/0 (socket says 172.21.15.28:60604) 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f287fe00640 1 -- 172.21.15.28:0/1496556870 learned_addr learned my addr 172.21.15.28:0/1496556870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885400640 1 -- 172.21.15.28:0/1496556870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000462945 0 0) 0x7f288815e100 con 0x7f28880cabc0 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885400640 1 -- 172.21.15.28:0/1496556870 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f286c003520 con 0x7f28880cabc0 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885e00640 1 -- 172.21.15.28:0/1496556870 >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 msgr2=0x7f2888071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885e00640 1 --2- 172.21.15.28:0/1496556870 >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 0x7f2888071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885e00640 1 -- 172.21.15.28:0/1496556870 >> v1:172.21.15.189:6789/0 conn(0x7f28880cabc0 legacy=0x7f288815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2885e00640 1 -- 172.21.15.28:0/1496556870 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288815e3e0 con 0x7f288815a540 2024-09-13T17:13:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.397+0000 7f2886800640 1 --2- 172.21.15.28:0/1496556870 >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 0x7f2888071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.398+0000 7f2885e00640 1 --2- 172.21.15.28:0/1496556870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288806c430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f28780097a0 tx=0x7f287802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74522c192c5349c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f2885400640 1 -- 172.21.15.28:0/1496556870 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287802ae50 con 0x7f288815a540 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f2885400640 1 -- 172.21.15.28:0/1496556870 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2878032440 con 0x7f288815a540 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f2885400640 1 -- 172.21.15.28:0/1496556870 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28780327d0 con 0x7f288815a540 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 -- 172.21.15.28:0/1496556870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 msgr2=0x7f288806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 --2- 172.21.15.28:0/1496556870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288806c430 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f28780097a0 tx=0x7f287802a100 comp rx=0 tx=0).stop 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 -- 172.21.15.28:0/1496556870 shutdown_connections 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 --2- 172.21.15.28:0/1496556870 >> v2:172.21.15.189:3300/0 conn(0x7f2888070cc0 0x7f2888071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 --2- 172.21.15.28:0/1496556870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288806c430 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.399+0000 7f288ce00640 1 -- 172.21.15.28:0/1496556870 >> 172.21.15.28:0/1496556870 conn(0x7f2888069e80 msgr2=0x7f288806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.400+0000 7f288ce00640 1 -- 172.21.15.28:0/1496556870 shutdown_connections 2024-09-13T17:13:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.400+0000 7f288ce00640 1 -- 172.21.15.28:0/1496556870 wait complete. 2024-09-13T17:13:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.401+0000 7f288ce00640 1 Processor -- start 2024-09-13T17:13:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.401+0000 7f288ce00640 1 -- start start 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888070cc0 0x7f288814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288815d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880af2d0 con 0x7f28880cabc0 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28880af520 con 0x7f2888070cc0 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.402+0000 7f288ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28880af800 con 0x7f288815a540 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46888/0 (socket says 172.21.15.28:46888) 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 -- 172.21.15.28:0/3447700440 learned_addr learned my addr 172.21.15.28:0/3447700440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f2885e00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888070cc0 0x7f288814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f2886800640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288815d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 -- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 msgr2=0x7f288815d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288815d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 -- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888070cc0 msgr2=0x7f288814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888070cc0 0x7f288814a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 -- 172.21.15.28:0/3447700440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2880009a70 con 0x7f28880cabc0 2024-09-13T17:13:29.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.403+0000 7f287fe00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f287000c4d0 tx=0x7f287000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.404+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870005ce0 con 0x7f28880cabc0 2024-09-13T17:13:29.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.404+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2870005e40 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.404+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2878009400 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.404+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28700072d0 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.404+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28880afdc0 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.405+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2888071170 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.406+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28700074f0 con 0x7f28880cabc0 2024-09-13T17:13:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.407+0000 7f287e000640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 0x7f286406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.407+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2870088900 con 0x7f28880cabc0 2024-09-13T17:13:29.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.407+0000 7f2885e00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 0x7f286406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.408+0000 7f2885e00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 0x7f286406d0c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f28780097a0 tx=0x7f287802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.411+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2870053a90 con 0x7f28880cabc0 2024-09-13T17:13:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.591+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f288806c270 con 0x7f28880cabc0 2024-09-13T17:13:29.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.592+0000 7f287e000640 1 -- 172.21.15.28:0/3447700440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16697 (secure 0 0 0) 0x7f2870017020 con 0x7f28880cabc0 2024-09-13T17:13:29.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:29.595 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:05.887982+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:12:52.355786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-09-13T17:12:53.525437+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"77","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3037911251":"2024-09-14T17:13:05.880422+0000","172.21.15.28:6832/3037911251":"2024-09-14T17:13:05.880422+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:29.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 msgr2=0x7f286406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 0x7f286406d0c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f28780097a0 tx=0x7f287802e000 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 msgr2=0x7f288814f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f287000c4d0 tx=0x7f287000c8d0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 shutdown_connections 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286406ac00 0x7f286406d0c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288815a540 0x7f288815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880cabc0 0x7f288814f9b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 --2- 172.21.15.28:0/3447700440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888070cc0 0x7f288814a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 >> 172.21.15.28:0/3447700440 conn(0x7f2888069e80 msgr2=0x7f28880ccb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 shutdown_connections 2024-09-13T17:13:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.594+0000 7f288ce00640 1 -- 172.21.15.28:0/3447700440 wait complete. 2024-09-13T17:13:29.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:13:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.766+0000 7fe7be800640 1 Processor -- start 2024-09-13T17:13:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.766+0000 7fe7be800640 1 -- start start 2024-09-13T17:13:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7b8159770 0x7fe7b806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b806f3f0 0x7fe7b8070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe7b80add00 0x7fe7b8159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b806f920 con 0x7fe7b8159770 2024-09-13T17:13:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b815b290 con 0x7fe7b80add00 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7be800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815b570 con 0x7fe7b806f3f0 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7b8159770 0x7fe7b806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe7b8159770 0x7fe7b806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46898/0 (socket says 172.21.15.28:46898) 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 -- 172.21.15.28:0/146265693 learned_addr learned my addr 172.21.15.28:0/146265693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7aea00640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b806f3f0 0x7fe7b8070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7e00640 1 --2- 172.21.15.28:0/146265693 >> v2:172.21.15.189:3300/0 conn(0x7fe7b80add00 0x7fe7b8159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 -- 172.21.15.28:0/146265693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b806f3f0 msgr2=0x7fe7b8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b806f3f0 0x7fe7b8070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 -- 172.21.15.28:0/146265693 >> v2:172.21.15.189:3300/0 conn(0x7fe7b80add00 msgr2=0x7fe7b8159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 --2- 172.21.15.28:0/146265693 >> v2:172.21.15.189:3300/0 conn(0x7fe7b80add00 0x7fe7b8159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.768+0000 7fe7b7400640 1 -- 172.21.15.28:0/146265693 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b815b850 con 0x7fe7b8159770 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.769+0000 7fe7b7400640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b8159770 0x7fe7b806fd20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80b0e90 tx=0x7fe7a802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb996c085cbdd53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7b6a00640 1 -- 172.21.15.28:0/146265693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a802ae90 con 0x7fe7b8159770 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7b6a00640 1 -- 172.21.15.28:0/146265693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a8032440 con 0x7fe7b8159770 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7b6a00640 1 -- 172.21.15.28:0/146265693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80327d0 con 0x7fe7b8159770 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 -- 172.21.15.28:0/146265693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b8159770 msgr2=0x7fe7b806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b8159770 0x7fe7b806fd20 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe7b80b0e90 tx=0x7fe7a802a140 comp rx=0 tx=0).stop 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 -- 172.21.15.28:0/146265693 shutdown_connections 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 --2- 172.21.15.28:0/146265693 >> v2:172.21.15.189:3300/0 conn(0x7fe7b80add00 0x7fe7b8159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b806f3f0 0x7fe7b8070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 --2- 172.21.15.28:0/146265693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b8159770 0x7fe7b806fd20 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 -- 172.21.15.28:0/146265693 >> 172.21.15.28:0/146265693 conn(0x7fe7b80b0270 msgr2=0x7fe7b80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 -- 172.21.15.28:0/146265693 shutdown_connections 2024-09-13T17:13:29.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.770+0000 7fe7be800640 1 -- 172.21.15.28:0/146265693 wait complete. 2024-09-13T17:13:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.772+0000 7fe7be800640 1 Processor -- start 2024-09-13T17:13:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.772+0000 7fe7be800640 1 -- start start 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b806f3f0 0x7fe7b81e2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b8159770 0x7fe7b81f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80ccde0 con 0x7fe7b80add00 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80cd0c0 con 0x7fe7b806f3f0 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7be800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80cd3a0 con 0x7fe7b8159770 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7aea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46914/0 (socket says 172.21.15.28:46914) 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.773+0000 7fe7aea00640 1 -- 172.21.15.28:0/1184794403 learned_addr learned my addr 172.21.15.28:0/1184794403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7b7e00640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b8159770 0x7fe7b81f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7b7400640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b806f3f0 0x7fe7b81e2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 -- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b8159770 msgr2=0x7fe7b81f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b8159770 0x7fe7b81f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 -- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b806f3f0 msgr2=0x7fe7b81e2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b806f3f0 0x7fe7b81e2460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 -- 172.21.15.28:0/1184794403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a80094d0 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7aea00640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e9b0 tx=0x7fe7a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0009670 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a000c9a0 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b80cd680 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a000cd50 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.774+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b80ad000 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.776+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b80ad220 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.776+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe7a0013470 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.777+0000 7fe7afe00640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 0x7fe79406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.777+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe7a00896f0 con 0x7fe7b80add00 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.778+0000 7fe7b7400640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 0x7fe79406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.778+0000 7fe7b7400640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 0x7fe79406d0c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8034000 tx=0x7fe7a802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:29.781+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe7a00547d0 con 0x7fe7b80add00 2024-09-13T17:13:30.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.004+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe7b8159b50 con 0x7fe7b80add00 2024-09-13T17:13:30.007 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:13:30.005+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:13:30.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.016+0000 7fe7afe00640 1 -- 172.21.15.28:0/1184794403 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v71) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe7a0057df0 con 0x7fe7b80add00 2024-09-13T17:13:30.018 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:13:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 msgr2=0x7fe79406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 0x7fe79406d0c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8034000 tx=0x7fe7a802e000 comp rx=0 tx=0).stop 2024-09-13T17:13:30.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 msgr2=0x7fe7b80acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000e9b0 tx=0x7fe7a000ee70 comp rx=0 tx=0).stop 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 shutdown_connections 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe79406ac00 0x7fe79406d0c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe7b8159770 0x7fe7b81f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7b80add00 0x7fe7b80acac0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 --2- 172.21.15.28:0/1184794403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7b806f3f0 0x7fe7b81e2460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 >> 172.21.15.28:0/1184794403 conn(0x7fe7b80b0270 msgr2=0x7fe7b80b1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 shutdown_connections 2024-09-13T17:13:30.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.019+0000 7fe7be800640 1 -- 172.21.15.28:0/1184794403 wait complete. 2024-09-13T17:13:30.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:13:30.074 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:13:30.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.195+0000 7fbf73800640 1 Processor -- start 2024-09-13T17:13:30.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.195+0000 7fbf73800640 1 -- start start 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 0x7fbf6c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf6c15ab60 0x7fbf6c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf6c06bb40 0x7fbf6c06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c06c8b0 con 0x7fbf6c06bb40 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c06cb90 con 0x7fbf6c15ab60 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf73800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0cc980 con 0x7fbf6c15a780 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf71800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf6c06bb40 0x7fbf6c06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.196+0000 7fbf71800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf6c06bb40 0x7fbf6c06bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46924/0 (socket says 172.21.15.28:46924) 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 -- 172.21.15.28:0/2480635162 learned_addr learned my addr 172.21.15.28:0/2480635162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf70e00640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 0x7fbf6c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf6be00640 1 --2- 172.21.15.28:0/2480635162 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c15ab60 0x7fbf6c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 -- 172.21.15.28:0/2480635162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 msgr2=0x7fbf6c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 0x7fbf6c070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 -- 172.21.15.28:0/2480635162 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c15ab60 msgr2=0x7fbf6c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 --2- 172.21.15.28:0/2480635162 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c15ab60 0x7fbf6c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf71800640 1 -- 172.21.15.28:0/2480635162 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c0ccc60 con 0x7fbf6c06bb40 2024-09-13T17:13:30.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.197+0000 7fbf70e00640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 0x7fbf6c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.198+0000 7fbf71800640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c06bf90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0cf5c0 tx=0x7fbf6402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5428f6d04a2566b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:30.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.198+0000 7fbf6b400640 1 -- 172.21.15.28:0/2480635162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64032440 con 0x7fbf6c06bb40 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.198+0000 7fbf6b400640 1 -- 172.21.15.28:0/2480635162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf640325a0 con 0x7fbf6c06bb40 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.198+0000 7fbf6b400640 1 -- 172.21.15.28:0/2480635162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf64032930 con 0x7fbf6c06bb40 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 -- 172.21.15.28:0/2480635162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c06bb40 msgr2=0x7fbf6c06bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c06bf90 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0cf5c0 tx=0x7fbf6402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 -- 172.21.15.28:0/2480635162 shutdown_connections 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c06bf90 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 --2- 172.21.15.28:0/2480635162 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c15ab60 0x7fbf6c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 --2- 172.21.15.28:0/2480635162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15a780 0x7fbf6c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 -- 172.21.15.28:0/2480635162 >> 172.21.15.28:0/2480635162 conn(0x7fbf6c0b0270 msgr2=0x7fbf6c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 -- 172.21.15.28:0/2480635162 shutdown_connections 2024-09-13T17:13:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.199+0000 7fbf73800640 1 -- 172.21.15.28:0/2480635162 wait complete. 2024-09-13T17:13:30.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.201+0000 7fbf73800640 1 Processor -- start 2024-09-13T17:13:30.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.201+0000 7fbf73800640 1 -- start start 2024-09-13T17:13:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15ab60 0x7fbf6c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0c34e0 con 0x7fbf6c15a780 2024-09-13T17:13:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0c85f0 con 0x7fbf6c06bb40 2024-09-13T17:13:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf73800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0c88d0 con 0x7fbf6c15ab60 2024-09-13T17:13:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46932/0 (socket says 172.21.15.28:46932) 2024-09-13T17:13:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 -- 172.21.15.28:0/1116004095 learned_addr learned my addr 172.21.15.28:0/1116004095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf70e00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf71800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15ab60 0x7fbf6c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 -- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15ab60 msgr2=0x7fbf6c14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15ab60 0x7fbf6c14fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 -- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c06bb40 msgr2=0x7fbf6c0ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c0ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.202+0000 7fbf6be00640 1 -- 172.21.15.28:0/1116004095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf640094d0 con 0x7fbf6c15a780 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf6be00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e840 tx=0x7fbf5800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf580094f0 con 0x7fbf6c15a780 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf5800c940 con 0x7fbf6c15a780 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf5800ccd0 con 0x7fbf6c15a780 2024-09-13T17:13:30.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c0c8bb0 con 0x7fbf6c15a780 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.203+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c0c9170 con 0x7fbf6c15a780 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.204+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c06bf90 con 0x7fbf6c15a780 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.205+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf58013470 con 0x7fbf6c15a780 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.206+0000 7fbf68c00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 0x7fbf4c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.206+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fbf580894a0 con 0x7fbf6c15a780 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.206+0000 7fbf70e00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 0x7fbf4c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.207+0000 7fbf70e00640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 0x7fbf4c06d0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0adc00 tx=0x7fbf60009910 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.210+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf580545e0 con 0x7fbf6c15a780 2024-09-13T17:13:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:30.424+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbf6c0b3660 con 0x7fbf6c15a780 2024-09-13T17:13:31.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.062+0000 7fbf68c00640 1 -- 172.21.15.28:0/1116004095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v72) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbf58057c00 con 0x7fbf6c15a780 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.064+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 msgr2=0x7fbf4c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.064+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 0x7fbf4c06d0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c0adc00 tx=0x7fbf60009910 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 msgr2=0x7fbf6c0c64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fbf5800e840 tx=0x7fbf5800ed00 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 shutdown_connections 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf4c06ac00 0x7fbf4c06d0c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c15ab60 0x7fbf6c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf6c15a780 0x7fbf6c0c64c0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 --2- 172.21.15.28:0/1116004095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c06bb40 0x7fbf6c0ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 >> 172.21.15.28:0/1116004095 conn(0x7fbf6c0b0270 msgr2=0x7fbf6c0b2ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 shutdown_connections 2024-09-13T17:13:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.065+0000 7fbf73800640 1 -- 172.21.15.28:0/1116004095 wait complete. 2024-09-13T17:13:31.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:13:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.245+0000 7f6d3d000640 1 Processor -- start 2024-09-13T17:13:31.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.245+0000 7f6d3d000640 1 -- start start 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d3815c8c0 0x7f6d380cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d38069e80 0x7f6d3806a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d380b4250 0x7f6d380b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d380d2630 con 0x7f6d38069e80 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3806a830 con 0x7f6d3815c8c0 2024-09-13T17:13:31.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d3d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d380d1590 con 0x7f6d380b4250 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d2fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d38069e80 0x7f6d3806a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d2fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d38069e80 0x7f6d3806a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46944/0 (socket says 172.21.15.28:46944) 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.246+0000 7f6d2fe00640 1 -- 172.21.15.28:0/2983410396 learned_addr learned my addr 172.21.15.28:0/2983410396 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d37e00640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d380b4250 0x7f6d380b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d37400640 1 --2- 172.21.15.28:0/2983410396 >> v2:172.21.15.189:3300/0 conn(0x7f6d3815c8c0 0x7f6d380cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d2fe00640 1 -- 172.21.15.28:0/2983410396 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d380b4250 msgr2=0x7f6d380b4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d380b4250 0x7f6d380b4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d2fe00640 1 -- 172.21.15.28:0/2983410396 >> v2:172.21.15.189:3300/0 conn(0x7f6d3815c8c0 msgr2=0x7f6d380cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/2983410396 >> v2:172.21.15.189:3300/0 conn(0x7f6d3815c8c0 0x7f6d380cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.247+0000 7f6d2fe00640 1 -- 172.21.15.28:0/2983410396 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d380d1870 con 0x7f6d38069e80 2024-09-13T17:13:31.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.248+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d3806a2f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f6d380d3840 tx=0x7f6d2002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c30f91db5cfdd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.248+0000 7f6d36a00640 1 -- 172.21.15.28:0/2983410396 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20032440 con 0x7f6d38069e80 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.248+0000 7f6d36a00640 1 -- 172.21.15.28:0/2983410396 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d200325a0 con 0x7f6d38069e80 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.248+0000 7f6d36a00640 1 -- 172.21.15.28:0/2983410396 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d20032930 con 0x7f6d38069e80 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 -- 172.21.15.28:0/2983410396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 msgr2=0x7f6d3806a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d3806a2f0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f6d380d3840 tx=0x7f6d2002a270 comp rx=0 tx=0).stop 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 -- 172.21.15.28:0/2983410396 shutdown_connections 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d3806a2f0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 --2- 172.21.15.28:0/2983410396 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d380b4250 0x7f6d380b4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 --2- 172.21.15.28:0/2983410396 >> v2:172.21.15.189:3300/0 conn(0x7f6d3815c8c0 0x7f6d380cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 -- 172.21.15.28:0/2983410396 >> 172.21.15.28:0/2983410396 conn(0x7f6d380708c0 msgr2=0x7f6d38070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 -- 172.21.15.28:0/2983410396 shutdown_connections 2024-09-13T17:13:31.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.249+0000 7f6d3d000640 1 -- 172.21.15.28:0/2983410396 wait complete. 2024-09-13T17:13:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.251+0000 7f6d3d000640 1 Processor -- start 2024-09-13T17:13:31.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.251+0000 7f6d3d000640 1 -- start start 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d380b4250 0x7f6d380cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d3815c8c0 0x7f6d380cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3815e600 con 0x7f6d38069e80 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3815d9f0 con 0x7f6d380b4250 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d3d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3815dcd0 con 0x7f6d3815c8c0 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46952/0 (socket says 172.21.15.28:46952) 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 -- 172.21.15.28:0/376908812 learned_addr learned my addr 172.21.15.28:0/376908812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:31.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 -- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d3815c8c0 msgr2=0x7f6d380cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d3815c8c0 0x7f6d380cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 -- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d380b4250 msgr2=0x7f6d380cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d380b4250 0x7f6d380cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d380b4250 0x7f6d380cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.252+0000 7f6d37400640 1 -- 172.21.15.28:0/376908812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d200094d0 con 0x7f6d38069e80 2024-09-13T17:13:31.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d37400640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e7f0 tx=0x7f6d2800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d280094f0 con 0x7f6d38069e80 2024-09-13T17:13:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2800eea0 con 0x7f6d38069e80 2024-09-13T17:13:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800cb80 con 0x7f6d38069e80 2024-09-13T17:13:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3815dfb0 con 0x7f6d38069e80 2024-09-13T17:13:31.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.253+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3815cd30 con 0x7f6d38069e80 2024-09-13T17:13:31.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.254+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3806a2f0 con 0x7f6d38069e80 2024-09-13T17:13:31.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.255+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6d2800cce0 con 0x7f6d38069e80 2024-09-13T17:13:31.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.256+0000 7f6d2f400640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 0x7f6d1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:31.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.256+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f6d28089890 con 0x7f6d38069e80 2024-09-13T17:13:31.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.256+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 0x7f6d1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:31.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.257+0000 7f6d2fe00640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 0x7f6d1406d0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6d20034000 tx=0x7f6d2002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:31.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.260+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6d28054920 con 0x7f6d38069e80 2024-09-13T17:13:31.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:31.438+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6d38073a20 con 0x7f6d38069e80 2024-09-13T17:13:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.068+0000 7f6d2f400640 1 -- 172.21.15.28:0/376908812 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6d280170d0 con 0x7f6d38069e80 2024-09-13T17:13:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:13:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 msgr2=0x7f6d1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 0x7f6d1406d0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6d20034000 tx=0x7f6d2002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 msgr2=0x7f6d380c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e7f0 tx=0x7f6d2800ecb0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 shutdown_connections 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d1406ac00 0x7f6d1406d0c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d3815c8c0 0x7f6d380cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d380b4250 0x7f6d380cc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 --2- 172.21.15.28:0/376908812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d38069e80 0x7f6d380c8c80 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.072+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 >> 172.21.15.28:0/376908812 conn(0x7f6d380708c0 msgr2=0x7f6d380b5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.073+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 shutdown_connections 2024-09-13T17:13:32.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.073+0000 7f6d3d000640 1 -- 172.21.15.28:0/376908812 wait complete. 2024-09-13T17:13:32.092 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:13:32.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.254+0000 7f242f800640 1 Processor -- start 2024-09-13T17:13:32.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.254+0000 7f242f800640 1 -- start start 2024-09-13T17:13:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242f800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f242815a4c0 con 0x7f24280b2630 2024-09-13T17:13:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242f800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2428071580 con 0x7f2428159590 2024-09-13T17:13:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280b2ae0 con 0x7f2428159970 2024-09-13T17:13:32.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242d800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f24280b2630 0x7f242815a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34066/0 (socket says 172.21.15.28:34066) 2024-09-13T17:13:32.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.255+0000 7f242d800640 1 -- 172.21.15.28:0/3199177914 learned_addr learned my addr 172.21.15.28:0/3199177914 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:32.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427e00640 1 --2- 172.21.15.28:0/3199177914 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:32.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508852335 0 0) 0x7f242815a4c0 con 0x7f24280b2630 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2410003520 con 0x7f24280b2630 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444976172 0 0) 0x7f2428071580 con 0x7f2428159590 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f242815a4c0 con 0x7f2428159590 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427e00640 1 -- 172.21.15.28:0/3199177914 >> v1:172.21.15.189:6789/0 conn(0x7f2428159590 legacy=0x7f2428070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427e00640 1 -- 172.21.15.28:0/3199177914 >> v1:172.21.15.28:6789/0 conn(0x7f24280b2630 legacy=0x7f242815a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.256+0000 7f2427e00640 1 -- 172.21.15.28:0/3199177914 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2428159e70 con 0x7f2428159970 2024-09-13T17:13:32.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.257+0000 7f2427e00640 1 --2- 172.21.15.28:0/3199177914 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280b20f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24140097d0 tx=0x7f241402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9461644b3a7c0412 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:32.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414032440 con 0x7f2428159970 2024-09-13T17:13:32.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24140325a0 con 0x7f2428159970 2024-09-13T17:13:32.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f2427400640 1 -- 172.21.15.28:0/3199177914 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2414032910 con 0x7f2428159970 2024-09-13T17:13:32.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f242f800640 1 -- 172.21.15.28:0/3199177914 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 msgr2=0x7f24280b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f242f800640 1 --2- 172.21.15.28:0/3199177914 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280b20f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f24140097d0 tx=0x7f241402a210 comp rx=0 tx=0).stop 2024-09-13T17:13:32.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f242f800640 1 -- 172.21.15.28:0/3199177914 shutdown_connections 2024-09-13T17:13:32.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f242f800640 1 --2- 172.21.15.28:0/3199177914 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280b20f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.258+0000 7f242f800640 1 -- 172.21.15.28:0/3199177914 >> 172.21.15.28:0/3199177914 conn(0x7f2428069e80 msgr2=0x7f242806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:32.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.259+0000 7f242f800640 1 -- 172.21.15.28:0/3199177914 shutdown_connections 2024-09-13T17:13:32.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.259+0000 7f242f800640 1 -- 172.21.15.28:0/3199177914 wait complete. 2024-09-13T17:13:32.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.260+0000 7f242f800640 1 Processor -- start 2024-09-13T17:13:32.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.260+0000 7f242f800640 1 -- start start 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2428159590 0x7f24280ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280cca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280cc5f0 con 0x7f24280b2630 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24280cb9e0 con 0x7f2428159590 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24280cbcc0 con 0x7f2428159970 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46954/0 (socket says 172.21.15.28:46954) 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.261+0000 7f242ce00640 1 -- 172.21.15.28:0/2529664974 learned_addr learned my addr 172.21.15.28:0/2529664974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242d800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280cca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f2427e00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2428159590 0x7f24280ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 -- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 msgr2=0x7f24280cca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280cca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 -- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2428159590 msgr2=0x7f24280ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2428159590 0x7f24280ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 -- 172.21.15.28:0/2529664974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2414009400 con 0x7f24280b2630 2024-09-13T17:13:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242ce00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f241c00c490 tx=0x7f241c00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c0052d0 con 0x7f24280b2630 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c005430 con 0x7f24280b2630 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c012870 con 0x7f24280b2630 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24280cbfa0 con 0x7f24280b2630 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.262+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2428152680 con 0x7f24280b2630 2024-09-13T17:13:32.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.264+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2428071580 con 0x7f24280b2630 2024-09-13T17:13:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.264+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f241c0129d0 con 0x7f24280b2630 2024-09-13T17:13:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.265+0000 7f2424c00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 0x7f240806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.265+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f241c088e70 con 0x7f24280b2630 2024-09-13T17:13:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.266+0000 7f2427e00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 0x7f240806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:32.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.266+0000 7f2427e00640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 0x7f240806d0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2414034000 tx=0x7f241402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.269+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f241c054150 con 0x7f24280b2630 2024-09-13T17:13:32.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:32.448+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2428073bb0 con 0x7f24280b2630 2024-09-13T17:13:33.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.078+0000 7f2424c00640 1 -- 172.21.15.28:0/2529664974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f241c057770 con 0x7f24280b2630 2024-09-13T17:13:33.081 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:13:33.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.082+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 msgr2=0x7f240806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.082+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 0x7f240806d0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f2414034000 tx=0x7f241402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.082+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 msgr2=0x7f2428152140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.082+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f241c00c490 tx=0x7f241c00c950 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 shutdown_connections 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f240806ac00 0x7f240806d0c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2428159970 0x7f24280cca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2428159590 0x7f24280ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 --2- 172.21.15.28:0/2529664974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24280b2630 0x7f2428152140 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 >> 172.21.15.28:0/2529664974 conn(0x7f2428069e80 msgr2=0x7f242806c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 shutdown_connections 2024-09-13T17:13:33.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.083+0000 7f242f800640 1 -- 172.21.15.28:0/2529664974 wait complete. 2024-09-13T17:13:33.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:13:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.264+0000 7f8b76e00640 1 Processor -- start 2024-09-13T17:13:33.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.264+0000 7f8b76e00640 1 -- start start 2024-09-13T17:13:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b76e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b70159770 0x7f8b7006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b76e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b70070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b76e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b7006f920 con 0x7f8b700add00 2024-09-13T17:13:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b76e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b7015b290 con 0x7f8b70159770 2024-09-13T17:13:33.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b76e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b7015b570 con 0x7f8b7006f3f0 2024-09-13T17:13:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b6fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8b70159770 0x7f8b7006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b74e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8b700add00 0x7f8b70159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34080/0 (socket says 172.21.15.28:34080) 2024-09-13T17:13:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.265+0000 7f8b74e00640 1 -- 172.21.15.28:0/698376696 learned_addr learned my addr 172.21.15.28:0/698376696 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:33.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b67400640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b70070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b6fe00640 1 -- 172.21.15.28:0/698376696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 msgr2=0x7f8b70070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b6fe00640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b70070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b6fe00640 1 -- 172.21.15.28:0/698376696 >> v1:172.21.15.28:6789/0 conn(0x7f8b700add00 legacy=0x7f8b70159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b6fe00640 1 -- 172.21.15.28:0/698376696 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b7015b850 con 0x7f8b70159770 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b6f400640 1 -- 172.21.15.28:0/698376696 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658720450 0 0) 0x7f8b7006f920 con 0x7f8b700add00 2024-09-13T17:13:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.266+0000 7f8b67400640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b70070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.267+0000 7f8b6fe00640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b7006fd20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8b600097a0 tx=0x7f8b6002a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634c66744c84a7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.267+0000 7f8b6f400640 1 -- 172.21.15.28:0/698376696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6002ae40 con 0x7f8b70159770 2024-09-13T17:13:33.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.267+0000 7f8b6f400640 1 -- 172.21.15.28:0/698376696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b60032440 con 0x7f8b70159770 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.267+0000 7f8b6f400640 1 -- 172.21.15.28:0/698376696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b600327d0 con 0x7f8b70159770 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 -- 172.21.15.28:0/698376696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 msgr2=0x7f8b7006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b7006fd20 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f8b600097a0 tx=0x7f8b6002a0f0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 -- 172.21.15.28:0/698376696 shutdown_connections 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b70070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 --2- 172.21.15.28:0/698376696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b7006fd20 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 -- 172.21.15.28:0/698376696 >> 172.21.15.28:0/698376696 conn(0x7f8b700b0270 msgr2=0x7f8b700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 -- 172.21.15.28:0/698376696 shutdown_connections 2024-09-13T17:13:33.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.268+0000 7f8b76e00640 1 -- 172.21.15.28:0/698376696 wait complete. 2024-09-13T17:13:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.270+0000 7f8b76e00640 1 Processor -- start 2024-09-13T17:13:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.270+0000 7f8b76e00640 1 -- start start 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b7014f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b700add00 0x7f8b70149b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700cb800 con 0x7f8b70159770 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b700cbae0 con 0x7f8b700add00 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b76e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b701f6020 con 0x7f8b7006f3f0 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b74e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b74e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46978/0 (socket says 172.21.15.28:46978) 2024-09-13T17:13:33.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b74e00640 1 -- 172.21.15.28:0/1800728503 learned_addr learned my addr 172.21.15.28:0/1800728503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b6fe00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b7014f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.271+0000 7f8b67400640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b700add00 0x7f8b70149b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 -- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 msgr2=0x7f8b7014f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b7014f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 -- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b700add00 msgr2=0x7f8b70149b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b700add00 0x7f8b70149b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 -- 172.21.15.28:0/1800728503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b60009400 con 0x7f8b70159770 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b74e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c730 tx=0x7f8b6800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800ce40 con 0x7f8b70159770 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b6800a640 con 0x7f8b70159770 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b7014af50 con 0x7f8b70159770 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b6800aa30 con 0x7f8b70159770 2024-09-13T17:13:33.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.272+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b7014b510 con 0x7f8b70159770 2024-09-13T17:13:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.273+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b7006fd20 con 0x7f8b70159770 2024-09-13T17:13:33.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.274+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8b6800ab90 con 0x7f8b70159770 2024-09-13T17:13:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.275+0000 7f8b6cc00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 0x7f8b4c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.275+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8b680886e0 con 0x7f8b70159770 2024-09-13T17:13:33.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.276+0000 7f8b6fe00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 0x7f8b4c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.276+0000 7f8b6fe00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 0x7f8b4c06d0f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8b60034000 tx=0x7f8b6002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.279+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8b68053c00 con 0x7f8b70159770 2024-09-13T17:13:33.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.459+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8b700cbc20 con 0x7f8b70159770 2024-09-13T17:13:33.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.463+0000 7f8b6cc00640 1 -- 172.21.15.28:0/1800728503 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8b68057220 con 0x7f8b70159770 2024-09-13T17:13:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 msgr2=0x7f8b4c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 0x7f8b4c06d0f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8b60034000 tx=0x7f8b6002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:33.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 msgr2=0x7f8b700ac880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f8b6800c730 tx=0x7f8b6800cbf0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 shutdown_connections 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b4c06ac30 0x7f8b4c06d0f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b70159770 0x7f8b700ac880 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b700add00 0x7f8b70149b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 --2- 172.21.15.28:0/1800728503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8b7006f3f0 0x7f8b7014f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 >> 172.21.15.28:0/1800728503 conn(0x7f8b700b0270 msgr2=0x7f8b70159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 shutdown_connections 2024-09-13T17:13:33.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.465+0000 7f8b76e00640 1 -- 172.21.15.28:0/1800728503 wait complete. 2024-09-13T17:13:33.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair' 2024-09-13T17:13:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.643+0000 7fee05600640 1 Processor -- start 2024-09-13T17:13:33.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.643+0000 7fee05600640 1 -- start start 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee05600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee0015a150 0x7fee000ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee05600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0006f3a0 con 0x7fee0015d560 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee05600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0006f680 con 0x7fee00070160 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee05600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0006f960 con 0x7fee0015a150 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee04c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee0015d560 0x7fee000b28d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34086/0 (socket says 172.21.15.28:34086) 2024-09-13T17:13:33.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.644+0000 7fee04c00640 1 -- 172.21.15.28:0/4212628805 learned_addr learned my addr 172.21.15.28:0/4212628805 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedffe00640 1 --2- 172.21.15.28:0/4212628805 >> v2:172.21.15.189:3300/0 conn(0x7fee0015a150 0x7fee000ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503032088 0 0) 0x7fee0006f3a0 con 0x7fee0015d560 2024-09-13T17:13:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede4003520 con 0x7fee0015d560 2024-09-13T17:13:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163344204 0 0) 0x7fee0006f680 con 0x7fee00070160 2024-09-13T17:13:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee0006f3a0 con 0x7fee00070160 2024-09-13T17:13:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedffe00640 1 -- 172.21.15.28:0/4212628805 >> v1:172.21.15.189:6789/0 conn(0x7fee00070160 legacy=0x7fee0015ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedffe00640 1 -- 172.21.15.28:0/4212628805 >> v1:172.21.15.28:6789/0 conn(0x7fee0015d560 legacy=0x7fee000b28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.645+0000 7fedffe00640 1 -- 172.21.15.28:0/4212628805 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0006fc40 con 0x7fee0015a150 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.646+0000 7fedffe00640 1 --2- 172.21.15.28:0/4212628805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015a150 0x7fee000ac850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fedf00097a0 tx=0x7fedf002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d34a07eaf576f426 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0032440 con 0x7fee0015a150 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf00325a0 con 0x7fee0015a150 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fedff400640 1 -- 172.21.15.28:0/4212628805 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0032930 con 0x7fee0015a150 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 -- 172.21.15.28:0/4212628805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015a150 msgr2=0x7fee000ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 --2- 172.21.15.28:0/4212628805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015a150 0x7fee000ac850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fedf00097a0 tx=0x7fedf002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 -- 172.21.15.28:0/4212628805 shutdown_connections 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 --2- 172.21.15.28:0/4212628805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015a150 0x7fee000ac850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 -- 172.21.15.28:0/4212628805 >> 172.21.15.28:0/4212628805 conn(0x7fee00073670 msgr2=0x7fee00069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 -- 172.21.15.28:0/4212628805 shutdown_connections 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.647+0000 7fee05600640 1 -- 172.21.15.28:0/4212628805 wait complete. 2024-09-13T17:13:33.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.649+0000 7fee05600640 1 Processor -- start 2024-09-13T17:13:33.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.649+0000 7fee05600640 1 -- start start 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 0x7fee0014be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee0014c360 0x7fee000cc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee0014c7f0 con 0x7fee0015a150 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee000ce090 con 0x7fee0015d560 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee05600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee000cd480 con 0x7fee0014c360 2024-09-13T17:13:33.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46986/0 (socket says 172.21.15.28:46986) 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 -- 172.21.15.28:0/859890355 learned_addr learned my addr 172.21.15.28:0/859890355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fee04c00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee0014c360 0x7fee000cc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedf7e00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 0x7fee0014be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 -- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee0014c360 msgr2=0x7fee000cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee0014c360 0x7fee000cc730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 -- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 msgr2=0x7fee0014be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 0x7fee0014be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedffe00640 1 -- 172.21.15.28:0/859890355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf0009400 con 0x7fee0015a150 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.650+0000 7fedf7e00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 0x7fee0014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fedffe00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fedf0035040 tx=0x7fedf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf003d8f0 con 0x7fee0015a150 2024-09-13T17:13:33.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf003da70 con 0x7fee0015a150 2024-09-13T17:13:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf003de20 con 0x7fee0015a150 2024-09-13T17:13:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee000cd760 con 0x7fee0015a150 2024-09-13T17:13:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.651+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee000cdd20 con 0x7fee0015a150 2024-09-13T17:13:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.652+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0006f680 con 0x7fee0015a150 2024-09-13T17:13:33.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.653+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fedf0039040 con 0x7fee0015a150 2024-09-13T17:13:33.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.654+0000 7fedfcc00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 0x7feddc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.654+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fedf00a9b40 con 0x7fee0015a150 2024-09-13T17:13:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.654+0000 7fedf7e00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 0x7feddc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.655+0000 7fedf7e00640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 0x7feddc06d0c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fee0014cf60 tx=0x7fede8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:33.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.658+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fedf00750d0 con 0x7fee0015a150 2024-09-13T17:13:33.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:33.914+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]} v 0) v1 -- 0x7fee0006bf90 con 0x7fee0015a150 2024-09-13T17:13:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.078+0000 7fedfcc00640 1 -- 172.21.15.28:0/859890355 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_repair"]}]=0 v291) v1 ==== 143+0+0 (secure 0 0 0) 0x7fedf00786f0 con 0x7fee0015a150 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 msgr2=0x7feddc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 0x7feddc06d0c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fee0014cf60 tx=0x7fede8009040 comp rx=0 tx=0).stop 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 msgr2=0x7fee00158bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fedf0035040 tx=0x7fedf00046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 shutdown_connections 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feddc06ac00 0x7feddc06d0c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee0014c360 0x7fee000cc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee0015d560 0x7fee0014be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 --2- 172.21.15.28:0/859890355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee0015a150 0x7fee00158bf0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 >> 172.21.15.28:0/859890355 conn(0x7fee00073670 msgr2=0x7fee0015dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 shutdown_connections 2024-09-13T17:13:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.080+0000 7fee05600640 1 -- 172.21.15.28:0/859890355 wait complete. 2024-09-13T17:13:34.098 INFO:tasks.cephfs_test_runner:test_scrub_repair (tasks.cephfs.test_scrub_checks.TestScrubChecks) ... ok 2024-09-13T17:13:34.098 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_split_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) That a directory is split during scrub. 2024-09-13T17:13:34.098 INFO:tasks.cephfs_test_runner:test_scrub_split_dirfrags (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:13:34.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.253+0000 7f8716800640 1 Processor -- start 2024-09-13T17:13:34.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.253+0000 7f8716800640 1 -- start start 2024-09-13T17:13:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8716800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f871015d9e0 0x7f87100b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8716800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871006b8c0 con 0x7f87100ac850 2024-09-13T17:13:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8716800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8710160aa0 con 0x7f871015c5f0 2024-09-13T17:13:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8716800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8710161340 con 0x7f871015d9e0 2024-09-13T17:13:34.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8714a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f871015d9e0 0x7f87100b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8715e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f87100ac850 0x7f871006bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58644/0 (socket says 172.21.15.28:58644) 2024-09-13T17:13:34.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.254+0000 7f8715e00640 1 -- 172.21.15.28:0/506989426 learned_addr learned my addr 172.21.15.28:0/506989426 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.255+0000 7f8714a00640 1 -- 172.21.15.28:0/506989426 >> v1:172.21.15.189:6789/0 conn(0x7f871015c5f0 legacy=0x7f87101602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.255+0000 7f8714a00640 1 -- 172.21.15.28:0/506989426 >> v1:172.21.15.28:6789/0 conn(0x7f87100ac850 legacy=0x7f871006bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.255+0000 7f8714a00640 1 -- 172.21.15.28:0/506989426 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8710160d80 con 0x7f871015d9e0 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.255+0000 7f870fe00640 1 -- 172.21.15.28:0/506989426 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111214110 0 0) 0x7f871006b8c0 con 0x7f87100ac850 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.256+0000 7f8714a00640 1 --2- 172.21.15.28:0/506989426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871015d9e0 0x7f87100b1ab0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097a0 tx=0x7f86fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed88d65c2f42d061 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.256+0000 7f870fe00640 1 -- 172.21.15.28:0/506989426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc032440 con 0x7f871015d9e0 2024-09-13T17:13:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.256+0000 7f870fe00640 1 -- 172.21.15.28:0/506989426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86fc0325a0 con 0x7f871015d9e0 2024-09-13T17:13:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.256+0000 7f870fe00640 1 -- 172.21.15.28:0/506989426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc032930 con 0x7f871015d9e0 2024-09-13T17:13:34.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 -- 172.21.15.28:0/506989426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871015d9e0 msgr2=0x7f87100b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 --2- 172.21.15.28:0/506989426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871015d9e0 0x7f87100b1ab0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f86fc0097a0 tx=0x7f86fc02a230 comp rx=0 tx=0).stop 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 -- 172.21.15.28:0/506989426 shutdown_connections 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 --2- 172.21.15.28:0/506989426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f871015d9e0 0x7f87100b1ab0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 -- 172.21.15.28:0/506989426 >> 172.21.15.28:0/506989426 conn(0x7f87100ce880 msgr2=0x7f87100cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 -- 172.21.15.28:0/506989426 shutdown_connections 2024-09-13T17:13:34.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.257+0000 7f8716800640 1 -- 172.21.15.28:0/506989426 wait complete. 2024-09-13T17:13:34.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.258+0000 7f8716800640 1 Processor -- start 2024-09-13T17:13:34.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.258+0000 7f8716800640 1 -- start start 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f871015c5f0 0x7f87101f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f871015d9e0 0x7f871015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101524e0 con 0x7f87100ac850 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101527c0 con 0x7f871015d9e0 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.259+0000 7f8716800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8710152aa0 con 0x7f871015c5f0 2024-09-13T17:13:34.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43626/0 (socket says 172.21.15.28:43626) 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 -- 172.21.15.28:0/1483047871 learned_addr learned my addr 172.21.15.28:0/1483047871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715e00640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f871015d9e0 0x7f871015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8714a00640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f871015c5f0 0x7f87101f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 -- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f871015c5f0 msgr2=0x7f87101f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f871015c5f0 0x7f87101f1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 -- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f871015d9e0 msgr2=0x7f871015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f871015d9e0 0x7f871015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.260+0000 7f8715400640 1 -- 172.21.15.28:0/1483047871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc009400 con 0x7f87100ac850 2024-09-13T17:13:34.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f8715400640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f870400c450 tx=0x7f870400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87040052d0 con 0x7f87100ac850 2024-09-13T17:13:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8704005430 con 0x7f87100ac850 2024-09-13T17:13:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8704012870 con 0x7f87100ac850 2024-09-13T17:13:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8710152d80 con 0x7f87100ac850 2024-09-13T17:13:34.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.261+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8710147600 con 0x7f87100ac850 2024-09-13T17:13:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.262+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8710147790 con 0x7f87100ac850 2024-09-13T17:13:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.263+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f87040129d0 con 0x7f87100ac850 2024-09-13T17:13:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.264+0000 7f870d600640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 0x7f86f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.264+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f8704088b50 con 0x7f87100ac850 2024-09-13T17:13:34.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.264+0000 7f8714a00640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 0x7f86f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.265+0000 7f8714a00640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 0x7f86f006d0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f86fc034000 tx=0x7f86fc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.268+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8704053fe0 con 0x7f87100ac850 2024-09-13T17:13:34.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.450+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87100b1ab0 con 0x7f87100ac850 2024-09-13T17:13:34.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.451+0000 7f870d600640 1 -- 172.21.15.28:0/1483047871 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+13119 (secure 0 0 0) 0x7f8704057600 con 0x7f87100ac850 2024-09-13T17:13:34.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:34.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":84,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:33.069017+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6836/1743829849":"2024-09-14T17:13:30.006178+0000","172.21.15.189:6834/1743829849":"2024-09-14T17:13:30.006178+0000","172.21.15.28:6834/3037911251":"2024-09-14T17:13:05.880422+0000","172.21.15.28:6832/3037911251":"2024-09-14T17:13:05.880422+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 msgr2=0x7f86f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 0x7f86f006d0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f86fc034000 tx=0x7f86fc02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 msgr2=0x7f8710153580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f870400c450 tx=0x7f870400c910 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 shutdown_connections 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f86f006ac00 0x7f86f006d0c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f871015d9e0 0x7f871015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f871015c5f0 0x7f87101f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 --2- 172.21.15.28:0/1483047871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87100ac850 0x7f8710153580 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.453+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 >> 172.21.15.28:0/1483047871 conn(0x7f87100ce880 msgr2=0x7f871006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.454+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 shutdown_connections 2024-09-13T17:13:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.454+0000 7f8716800640 1 -- 172.21.15.28:0/1483047871 wait complete. 2024-09-13T17:13:34.473 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:34.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:34.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.629+0000 7f53e6e00640 1 Processor -- start 2024-09-13T17:13:34.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.629+0000 7f53e6e00640 1 -- start start 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e6e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53e015a780 0x7f53e0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53e00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e6e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e006c8b0 con 0x7f53e006bb40 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e6e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e006cb90 con 0x7f53e015a780 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00cc980 con 0x7f53e015ab60 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f53e006bb40 0x7f53e006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58648/0 (socket says 172.21.15.28:58648) 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53e4e00640 1 -- 172.21.15.28:0/2803451791 learned_addr learned my addr 172.21.15.28:0/2803451791 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:34.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53d7400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53e00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53dfe00640 1 --2- 172.21.15.28:0/2803451791 >> v2:172.21.15.189:3300/0 conn(0x7f53e015a780 0x7f53e0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332611087 0 0) 0x7f53e006c8b0 con 0x7f53e006bb40 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.631+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c4003520 con 0x7f53e006bb40 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 292621296 0 0) 0x7f53c4003520 con 0x7f53e006bb40 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 msgr2=0x7f53e00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53e00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 >> v2:172.21.15.189:3300/0 conn(0x7f53e015a780 msgr2=0x7f53e0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 --2- 172.21.15.28:0/2803451791 >> v2:172.21.15.189:3300/0 conn(0x7f53e015a780 0x7f53e0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00ccc60 con 0x7f53e006bb40 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f53d80019f0 con 0x7f53e006bb40 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 >> v1:172.21.15.28:6789/0 conn(0x7f53e006bb40 legacy=0x7f53e006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53c40047c0 0x7f53c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e006cb90 con 0x7f53c4003970 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00cc980 con 0x7f53c40047c0 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c400b980 con 0x7f53e015ab60 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53dfe00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53d7400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53c40047c0 0x7f53c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.632+0000 7f53e4e00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53dfe00640 1 -- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 msgr2=0x7f53c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53dfe00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ab60 0x7f53c4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53dfe00640 1 -- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53c40047c0 msgr2=0x7f53c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53dfe00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53c40047c0 0x7f53c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53dfe00640 1 -- 172.21.15.28:0/2803451791 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4003520 con 0x7f53c4003970 2024-09-13T17:13:34.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.633+0000 7f53d7400640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53c40047c0 0x7f53c4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53dfe00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53c4003d90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884b672b7b996c6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0031400 con 0x7f53c4003970 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d002d050 con 0x7f53c4003970 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53df400640 1 -- 172.21.15.28:0/2803451791 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0031780 con 0x7f53c4003970 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e6e00640 1 -- 172.21.15.28:0/2803451791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 msgr2=0x7f53c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e6e00640 1 --2- 172.21.15.28:0/2803451791 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53c4003d90 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53c40057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e4e00640 1 -- 172.21.15.28:0/2803451791 reap_dead start 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e6e00640 1 -- 172.21.15.28:0/2803451791 shutdown_connections 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e6e00640 1 -- 172.21.15.28:0/2803451791 >> 172.21.15.28:0/2803451791 conn(0x7f53e00b0270 msgr2=0x7f53e00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.634+0000 7f53e6e00640 1 -- 172.21.15.28:0/2803451791 shutdown_connections 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.635+0000 7f53e6e00640 1 -- 172.21.15.28:0/2803451791 wait complete. 2024-09-13T17:13:34.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.636+0000 7f53e6e00640 1 Processor -- start 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.636+0000 7f53e6e00640 1 -- start start 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e006fed0 0x7f53e00c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 0x7f53e00c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00c6bc0 con 0x7f53c4003970 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0070350 con 0x7f53e00c4a70 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53e6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0157670 con 0x7f53e006fed0 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43644/0 (socket says 172.21.15.28:43644) 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.637+0000 7f53dfe00640 1 -- 172.21.15.28:0/3285611300 learned_addr learned my addr 172.21.15.28:0/3285611300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53e4e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 0x7f53e00c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53d7400640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e006fed0 0x7f53e00c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 -- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e006fed0 msgr2=0x7f53e00c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e006fed0 0x7f53e00c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 -- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 msgr2=0x7f53e00c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 0x7f53e00c4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 -- 172.21.15.28:0/3285611300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e014f980 con 0x7f53c4003970 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53e4e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 0x7f53e00c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dfe00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d003b610 con 0x7f53c4003970 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d003b770 con 0x7f53c4003970 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d003bb00 con 0x7f53c4003970 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.638+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e014fc60 con 0x7f53c4003970 2024-09-13T17:13:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.639+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e00c67e0 con 0x7f53c4003970 2024-09-13T17:13:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.640+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e00c6e30 con 0x7f53c4003970 2024-09-13T17:13:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.641+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f53d0038050 con 0x7f53c4003970 2024-09-13T17:13:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.641+0000 7f53dcc00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 0x7f53bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.641+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f53d00a9530 con 0x7f53c4003970 2024-09-13T17:13:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.642+0000 7f53d7400640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 0x7f53bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:34.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.642+0000 7f53d7400640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 0x7f53bc06d0c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f53e01533f0 tx=0x7f53c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:34.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.646+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f53d0074af0 con 0x7f53c4003970 2024-09-13T17:13:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.821+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53e0070d80 con 0x7f53c4003970 2024-09-13T17:13:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.821+0000 7f53dcc00640 1 -- 172.21.15.28:0/3285611300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v84) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53d0036070 con 0x7f53c4003970 2024-09-13T17:13:34.823 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 msgr2=0x7f53bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 0x7f53bc06d0c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f53e01533f0 tx=0x7f53c8008040 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 msgr2=0x7f53e006f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009ac0 tx=0x7f53d0004730 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 shutdown_connections 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53bc06ac00 0x7f53bc06d0c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e00c4a70 0x7f53e00c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e006fed0 0x7f53e00c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 --2- 172.21.15.28:0/3285611300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53c4003970 0x7f53e006f990 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.823+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 >> 172.21.15.28:0/3285611300 conn(0x7f53e00b0270 msgr2=0x7f53e006b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.824+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 shutdown_connections 2024-09-13T17:13:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.824+0000 7f53e6e00640 1 -- 172.21.15.28:0/3285611300 wait complete. 2024-09-13T17:13:34.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags' 2024-09-13T17:13:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.996+0000 7f834e800640 1 Processor -- start 2024-09-13T17:13:34.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.996+0000 7f834e800640 1 -- start start 2024-09-13T17:13:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83481592a0 0x7f834815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f834806f970 0x7f8348159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f834806c2d0 0x7f834806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834815b0b0 con 0x7f83481592a0 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f834806cc70 con 0x7f834806f970 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f834e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83480b2040 con 0x7f834806c2d0 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83481592a0 0x7f834815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f83481592a0 0x7f834815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43646/0 (socket says 172.21.15.28:43646) 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 -- 172.21.15.28:0/2141944443 learned_addr learned my addr 172.21.15.28:0/2141944443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f833ea00640 1 --2- 172.21.15.28:0/2141944443 >> v2:172.21.15.189:3300/0 conn(0x7f834806f970 0x7f8348159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347e00640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f834806c2d0 0x7f834806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 -- 172.21.15.28:0/2141944443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f834806c2d0 msgr2=0x7f834806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f834806c2d0 0x7f834806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 -- 172.21.15.28:0/2141944443 >> v2:172.21.15.189:3300/0 conn(0x7f834806f970 msgr2=0x7f8348159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 --2- 172.21.15.28:0/2141944443 >> v2:172.21.15.189:3300/0 conn(0x7f834806f970 0x7f8348159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.998+0000 7f8347400640 1 -- 172.21.15.28:0/2141944443 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348159ff0 con 0x7f83481592a0 2024-09-13T17:13:35.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:34.999+0000 7f8347400640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83481592a0 0x7f834815ab70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f834806ac10 tx=0x7f833802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b959ce12886bd429 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f8346a00640 1 -- 172.21.15.28:0/2141944443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338032440 con 0x7f83481592a0 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f8346a00640 1 -- 172.21.15.28:0/2141944443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83380325a0 con 0x7f83481592a0 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f8346a00640 1 -- 172.21.15.28:0/2141944443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338032930 con 0x7f83481592a0 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 -- 172.21.15.28:0/2141944443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83481592a0 msgr2=0x7f834815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83481592a0 0x7f834815ab70 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f834806ac10 tx=0x7f833802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 -- 172.21.15.28:0/2141944443 shutdown_connections 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f834806c2d0 0x7f834806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 --2- 172.21.15.28:0/2141944443 >> v2:172.21.15.189:3300/0 conn(0x7f834806f970 0x7f8348159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 --2- 172.21.15.28:0/2141944443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83481592a0 0x7f834815ab70 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 -- 172.21.15.28:0/2141944443 >> 172.21.15.28:0/2141944443 conn(0x7f8348069e80 msgr2=0x7f834806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 -- 172.21.15.28:0/2141944443 shutdown_connections 2024-09-13T17:13:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.000+0000 7f834e800640 1 -- 172.21.15.28:0/2141944443 wait complete. 2024-09-13T17:13:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.002+0000 7f834e800640 1 Processor -- start 2024-09-13T17:13:35.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.002+0000 7f834e800640 1 -- start start 2024-09-13T17:13:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 0x7f8348150cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83481592a0 0x7f83480ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83481511f0 con 0x7f834806c2d0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83481f2440 con 0x7f834806f970 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f834e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8348151ae0 con 0x7f83481592a0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f8347400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f8347400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43660/0 (socket says 172.21.15.28:43660) 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f8347400640 1 -- 172.21.15.28:0/3367290895 learned_addr learned my addr 172.21.15.28:0/3367290895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f8347e00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83481592a0 0x7f83480ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.003+0000 7f833ea00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 0x7f8348150cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 -- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83481592a0 msgr2=0x7f83480ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83481592a0 0x7f83480ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 -- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 msgr2=0x7f8348150cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 0x7f8348150cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 -- 172.21.15.28:0/3367290895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380094d0 con 0x7f834806c2d0 2024-09-13T17:13:35.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f833ea00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 0x7f8348150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f8347400640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f833802a220 tx=0x7f83380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8338032440 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8338030ba0 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f833803d440 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83480ccbe0 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.004+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8348152060 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.005+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f834806c730 con 0x7f834806c2d0 2024-09-13T17:13:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.006+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8338039040 con 0x7f834806c2d0 2024-09-13T17:13:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.007+0000 7f833fe00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 0x7f832406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.007+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f83380a9d10 con 0x7f834806c2d0 2024-09-13T17:13:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.007+0000 7f833ea00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 0x7f832406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.008+0000 7f833ea00640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 0x7f832406d0c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8330006fd0 tx=0x7f8330008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.011+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f83380af020 con 0x7f834806c2d0 2024-09-13T17:13:35.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:35.262+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags"]} v 0) v1 -- 0x7f83480afe50 con 0x7f834806c2d0 2024-09-13T17:13:36.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.088+0000 7f833fe00640 1 -- 172.21.15.28:0/3367290895 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags"]}]=0 v293) v1 ==== 154+0+0 (secure 0 0 0) 0x7f83380370e0 con 0x7f834806c2d0 2024-09-13T17:13:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 msgr2=0x7f832406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 0x7f832406d0c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f8330006fd0 tx=0x7f8330008040 comp rx=0 tx=0).stop 2024-09-13T17:13:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 msgr2=0x7f8348152580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f833802a220 tx=0x7f83380046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 shutdown_connections 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f832406ac00 0x7f832406d0c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83481592a0 0x7f83480ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f834806f970 0x7f8348150cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 --2- 172.21.15.28:0/3367290895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f834806c2d0 0x7f8348152580 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 >> 172.21.15.28:0/3367290895 conn(0x7f8348069e80 msgr2=0x7f83480b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 shutdown_connections 2024-09-13T17:13:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.091+0000 7f834e800640 1 -- 172.21.15.28:0/3367290895 wait complete. 2024-09-13T17:13:36.114 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:13:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.276+0000 7f87bd200640 1 Processor -- start 2024-09-13T17:13:36.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.276+0000 7f87bd200640 1 -- start start 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.277+0000 7f87bd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f87b8158cd0 0x7f87b81590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87bd200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b81595f0 con 0x7f87b806ccd0 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87bd200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87b81598c0 con 0x7f87b806c110 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87bd200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87b8159ba0 con 0x7f87b8158cd0 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87ad400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f87b806ccd0 0x7f87b806ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58660/0 (socket says 172.21.15.28:58660) 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87b5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f87b8158cd0 0x7f87b81590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87ad400640 1 -- 172.21.15.28:0/236171491 learned_addr learned my addr 172.21.15.28:0/236171491 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87b5e00640 1 -- 172.21.15.28:0/236171491 >> v1:172.21.15.189:6789/0 conn(0x7f87b806c110 legacy=0x7f87b80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87b5e00640 1 -- 172.21.15.28:0/236171491 >> v1:172.21.15.28:6789/0 conn(0x7f87b806ccd0 legacy=0x7f87b806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.278+0000 7f87b5e00640 1 -- 172.21.15.28:0/236171491 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b8159e80 con 0x7f87b8158cd0 2024-09-13T17:13:36.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.279+0000 7f87b5e00640 1 --2- 172.21.15.28:0/236171491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b8158cd0 0x7f87b81590b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f87a80097a0 tx=0x7f87a802c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=869adf19a8f0df7c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87b5400640 1 -- 172.21.15.28:0/236171491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8034640 con 0x7f87b8158cd0 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87b5400640 1 -- 172.21.15.28:0/236171491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a80347a0 con 0x7f87b8158cd0 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87b5400640 1 -- 172.21.15.28:0/236171491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a8034b30 con 0x7f87b8158cd0 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 -- 172.21.15.28:0/236171491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b8158cd0 msgr2=0x7f87b81590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 --2- 172.21.15.28:0/236171491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b8158cd0 0x7f87b81590b0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f87a80097a0 tx=0x7f87a802c350 comp rx=0 tx=0).stop 2024-09-13T17:13:36.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 -- 172.21.15.28:0/236171491 shutdown_connections 2024-09-13T17:13:36.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 --2- 172.21.15.28:0/236171491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b8158cd0 0x7f87b81590b0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 -- 172.21.15.28:0/236171491 >> 172.21.15.28:0/236171491 conn(0x7f87b80b0270 msgr2=0x7f87b80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:36.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.280+0000 7f87bd200640 1 -- 172.21.15.28:0/236171491 shutdown_connections 2024-09-13T17:13:36.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.281+0000 7f87bd200640 1 -- 172.21.15.28:0/236171491 wait complete. 2024-09-13T17:13:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.282+0000 7f87bd200640 1 Processor -- start 2024-09-13T17:13:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.282+0000 7f87bd200640 1 -- start start 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b806c110 0x7f87b81f66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87b8158cd0 0x7f87b80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b81f6c00 con 0x7f87b806ccd0 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b81f2600 con 0x7f87b806c110 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87bd200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87b80cc980 con 0x7f87b8158cd0 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87ad400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.283+0000 7f87ad400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43672/0 (socket says 172.21.15.28:43672) 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 -- 172.21.15.28:0/3854155625 learned_addr learned my addr 172.21.15.28:0/3854155625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87b5e00640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b806c110 0x7f87b81f66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87b6800640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87b8158cd0 0x7f87b80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 -- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87b8158cd0 msgr2=0x7f87b80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87b8158cd0 0x7f87b80ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 -- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b806c110 msgr2=0x7f87b81f66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b806c110 0x7f87b81f66c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 -- 172.21.15.28:0/3854155625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a8009400 con 0x7f87b806ccd0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.284+0000 7f87ad400640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c4a0 tx=0x7f87a000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.285+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0005ce0 con 0x7f87b806ccd0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.285+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87a0005e40 con 0x7f87b806ccd0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.285+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a0007270 con 0x7f87b806ccd0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.285+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b80ccc60 con 0x7f87b806ccd0 2024-09-13T17:13:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.285+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b80cd220 con 0x7f87b806ccd0 2024-09-13T17:13:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.286+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b81590b0 con 0x7f87b806ccd0 2024-09-13T17:13:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.287+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f87a0007490 con 0x7f87b806ccd0 2024-09-13T17:13:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.288+0000 7f87aea00640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 0x7f879406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.288+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f87a00884d0 con 0x7f87b806ccd0 2024-09-13T17:13:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.288+0000 7f87b5e00640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 0x7f879406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.289+0000 7f87b5e00640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 0x7f879406d0c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.293+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f87a00539f0 con 0x7f87b806ccd0 2024-09-13T17:13:36.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.470+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f87b8070630 con 0x7f87b806ccd0 2024-09-13T17:13:36.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.474+0000 7f87aea00640 1 -- 172.21.15.28:0/3854155625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f87a00170c0 con 0x7f87b806ccd0 2024-09-13T17:13:36.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 msgr2=0x7f879406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 0x7f879406d0c0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f87a8036000 tx=0x7f87a8030000 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 msgr2=0x7f87b81f20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c4a0 tx=0x7f87a000c960 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 shutdown_connections 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f879406ac00 0x7f879406d0c0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f87b8158cd0 0x7f87b80ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f87b806ccd0 0x7f87b81f20c0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 --2- 172.21.15.28:0/3854155625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87b806c110 0x7f87b81f66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 >> 172.21.15.28:0/3854155625 conn(0x7f87b80b0270 msgr2=0x7f87b80b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 shutdown_connections 2024-09-13T17:13:36.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.476+0000 7f87bd200640 1 -- 172.21.15.28:0/3854155625 wait complete. 2024-09-13T17:13:36.494 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:13:36.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:13:36.572 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:13:36.670 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:13:36.670 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:36.670 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:36.705 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:13:36.705 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:13:36.706 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:36.742 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:13:36.742 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:36.742 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:36.780 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:13:36.781 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:13:36.781 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:36.818 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:13:36.818 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:13:36.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.973+0000 7fc7dec00640 1 Processor -- start 2024-09-13T17:13:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.974+0000 7fc7dec00640 1 -- start start 2024-09-13T17:13:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7d8154a90 0x7fc7d815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dec00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d815bb10 con 0x7fc7d80b0980 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80b0df0 con 0x7fc7d8154a90 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d806b900 con 0x7fc7d806b360 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7de200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc7d80b0980 0x7fc7d80af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58664/0 (socket says 172.21.15.28:58664) 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dd800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7d8154a90 0x7fc7d815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7de200640 1 -- 172.21.15.28:0/3829131606 learned_addr learned my addr 172.21.15.28:0/3829131606 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.975+0000 7fc7dce00640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.976+0000 7fc7dd800640 1 -- 172.21.15.28:0/3829131606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 msgr2=0x7fc7d80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.976+0000 7fc7dd800640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.976+0000 7fc7dd800640 1 -- 172.21.15.28:0/3829131606 >> v1:172.21.15.28:6789/0 conn(0x7fc7d80b0980 legacy=0x7fc7d80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.976+0000 7fc7dd800640 1 -- 172.21.15.28:0/3829131606 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d806bbe0 con 0x7fc7d8154a90 2024-09-13T17:13:36.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.976+0000 7fc7d7e00640 1 -- 172.21.15.28:0/3829131606 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 928906025 0 0) 0x7fc7d815bb10 con 0x7fc7d80b0980 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.977+0000 7fc7dd800640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d815b5d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f79a8203a3557f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.977+0000 7fc7d7e00640 1 -- 172.21.15.28:0/3829131606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc032440 con 0x7fc7d8154a90 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.977+0000 7fc7d7e00640 1 -- 172.21.15.28:0/3829131606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7cc0325a0 con 0x7fc7d8154a90 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.977+0000 7fc7d7e00640 1 -- 172.21.15.28:0/3829131606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7cc032930 con 0x7fc7d8154a90 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 -- 172.21.15.28:0/3829131606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 msgr2=0x7fc7d815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d815b5d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc0097a0 tx=0x7fc7cc02a180 comp rx=0 tx=0).stop 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 -- 172.21.15.28:0/3829131606 shutdown_connections 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 --2- 172.21.15.28:0/3829131606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d815b5d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 -- 172.21.15.28:0/3829131606 >> 172.21.15.28:0/3829131606 conn(0x7fc7d8072e40 msgr2=0x7fc7d8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:36.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 -- 172.21.15.28:0/3829131606 shutdown_connections 2024-09-13T17:13:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.978+0000 7fc7dec00640 1 -- 172.21.15.28:0/3829131606 wait complete. 2024-09-13T17:13:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.979+0000 7fc7dec00640 1 Processor -- start 2024-09-13T17:13:36.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 -- start start 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d81f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 0x7fc7d813d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d813d590 con 0x7fc7d8154a90 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80b1df0 con 0x7fc7d80b0980 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.980+0000 7fc7dec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d80af050 con 0x7fc7d806b360 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43678/0 (socket says 172.21.15.28:43678) 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 -- 172.21.15.28:0/1169809934 learned_addr learned my addr 172.21.15.28:0/1169809934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7dd800640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d81f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7dce00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 0x7fc7d813d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 -- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 msgr2=0x7fc7d81f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d81f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 -- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 msgr2=0x7fc7d813d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 0x7fc7d813d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 -- 172.21.15.28:0/1169809934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc009400 con 0x7fc7d8154a90 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7dce00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 0x7fc7d813d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.981+0000 7fc7de200640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c780 tx=0x7fc7d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.982+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d000ce30 con 0x7fc7d8154a90 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.982+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d000a640 con 0x7fc7d8154a90 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.982+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d000a9f0 con 0x7fc7d8154a90 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.982+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d80b2000 con 0x7fc7d8154a90 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.982+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d80b25c0 con 0x7fc7d8154a90 2024-09-13T17:13:36.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.983+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d80b27e0 con 0x7fc7d8154a90 2024-09-13T17:13:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.984+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7d000ab50 con 0x7fc7d8154a90 2024-09-13T17:13:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.985+0000 7fc7d5600640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 0x7fc7b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.985+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fc7d0088570 con 0x7fc7d8154a90 2024-09-13T17:13:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.985+0000 7fc7dd800640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 0x7fc7b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:36.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.986+0000 7fc7dd800640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 0x7fc7b806d0c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc034000 tx=0x7fc7cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:36.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:36.989+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc7d00539e0 con 0x7fc7d8154a90 2024-09-13T17:13:37.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.203+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7d815d840 con 0x7fc7d8154a90 2024-09-13T17:13:37.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.204+0000 7fc7d5600640 1 -- 172.21.15.28:0/1169809934 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+3535 (secure 0 0 0) 0x7fc7d0018020 con 0x7fc7d8154a90 2024-09-13T17:13:37.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:37.206 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":72,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6203,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2815498668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2815498668},{"type":"v1","addr":"172.21.15.189:6837","nonce":2815498668}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":6326,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2601715441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2601715441},{"type":"v1","addr":"172.21.15.189:6836","nonce":2601715441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7209,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3837627777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3837627777},{"type":"v1","addr":"172.21.15.28:6833","nonce":3837627777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":66},{"gid":7221,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3507464735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3507464735},{"type":"v1","addr":"172.21.15.28:6835","nonce":3507464735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":68}],"filesystems":[]} 2024-09-13T17:13:37.207 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 72 2024-09-13T17:13:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 msgr2=0x7fc7b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 0x7fc7b806d0c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc034000 tx=0x7fc7cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 msgr2=0x7fc7d80c5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c780 tx=0x7fc7d000cc40 comp rx=0 tx=0).stop 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 shutdown_connections 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc7b806ac00 0x7fc7b806d0c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7d8154a90 0x7fc7d80c5400 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7d80b0980 0x7fc7d813d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 --2- 172.21.15.28:0/1169809934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7d806b360 0x7fc7d81f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 >> 172.21.15.28:0/1169809934 conn(0x7fc7d8072e40 msgr2=0x7fc7d8159e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 shutdown_connections 2024-09-13T17:13:37.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.206+0000 7fc7dec00640 1 -- 172.21.15.28:0/1169809934 wait complete. 2024-09-13T17:13:37.224 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:13:37.224 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:13:37.224 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:13:37.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:37.225 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:13:37.225 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:13:37.225 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:13:37.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:37.225 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:13:37.225 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:13:37.226 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:13:37.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:37.226 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:13:37.226 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:13:37.226 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:13:37.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:37.239 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:13:37.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:13:37.240 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:13:37.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:13:37.241 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:13:37.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:13:37.242 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:13:37.242 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:13:37.284 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:13:37.284 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:37.286 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:13:37.286 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:37.286 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:13:37.286 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:37.288 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:13:37.288 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:37.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:13:37.415 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:13:37.415 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:13:37.417 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:13:37.417 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:13:37.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.456+0000 7fee16800640 1 Processor -- start 2024-09-13T17:13:37.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.456+0000 7fee16800640 1 -- start start 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee1015c9a0 0x7fee100ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100ad940 0x7fee100b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee100b2230 0x7fee1006f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee100b26c0 con 0x7fee100b2230 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee100ace30 con 0x7fee1015c9a0 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee16800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee100ad110 con 0x7fee100ad940 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.457+0000 7fee0fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee100b2230 0x7fee1006f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee100b2230 0x7fee1006f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43704/0 (socket says 172.21.15.28:43704) 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 -- 172.21.15.28:0/4139336895 learned_addr learned my addr 172.21.15.28:0/4139336895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0ea00640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100ad940 0x7fee100b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0f400640 1 --2- 172.21.15.28:0/4139336895 >> v2:172.21.15.189:3300/0 conn(0x7fee1015c9a0 0x7fee100ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 -- 172.21.15.28:0/4139336895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100ad940 msgr2=0x7fee100b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100ad940 0x7fee100b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 -- 172.21.15.28:0/4139336895 >> v2:172.21.15.189:3300/0 conn(0x7fee1015c9a0 msgr2=0x7fee100ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 --2- 172.21.15.28:0/4139336895 >> v2:172.21.15.189:3300/0 conn(0x7fee1015c9a0 0x7fee100ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.458+0000 7fee0fe00640 1 -- 172.21.15.28:0/4139336895 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1006b170 con 0x7fee100b2230 2024-09-13T17:13:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.459+0000 7fee0fe00640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100b2230 0x7fee1006f460 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fee10074440 tx=0x7fee0802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f6bec2faf45c68 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.459+0000 7fee0e000640 1 -- 172.21.15.28:0/4139336895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08031600 con 0x7fee100b2230 2024-09-13T17:13:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.459+0000 7fee0e000640 1 -- 172.21.15.28:0/4139336895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0802d050 con 0x7fee100b2230 2024-09-13T17:13:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.459+0000 7fee0e000640 1 -- 172.21.15.28:0/4139336895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee08031980 con 0x7fee100b2230 2024-09-13T17:13:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 -- 172.21.15.28:0/4139336895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100b2230 msgr2=0x7fee1006f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100b2230 0x7fee1006f460 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fee10074440 tx=0x7fee0802a730 comp rx=0 tx=0).stop 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 -- 172.21.15.28:0/4139336895 shutdown_connections 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100b2230 0x7fee1006f460 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 --2- 172.21.15.28:0/4139336895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100ad940 0x7fee100b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 --2- 172.21.15.28:0/4139336895 >> v2:172.21.15.189:3300/0 conn(0x7fee1015c9a0 0x7fee100ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 -- 172.21.15.28:0/4139336895 >> 172.21.15.28:0/4139336895 conn(0x7fee100b0270 msgr2=0x7fee100b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 -- 172.21.15.28:0/4139336895 shutdown_connections 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.460+0000 7fee16800640 1 -- 172.21.15.28:0/4139336895 wait complete. 2024-09-13T17:13:37.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.462+0000 7fee16800640 1 Processor -- start 2024-09-13T17:13:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.462+0000 7fee16800640 1 -- start start 2024-09-13T17:13:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 0x7fee100af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1015c9a0 0x7fee100cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1014e920 con 0x7fee100ad940 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee1015feb0 con 0x7fee1015c9a0 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee16800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1015f2a0 con 0x7fee100b2230 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43710/0 (socket says 172.21.15.28:43710) 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 -- 172.21.15.28:0/2047512828 learned_addr learned my addr 172.21.15.28:0/2047512828 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0fe00640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1015c9a0 0x7fee100cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0ea00640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 0x7fee100af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 -- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 msgr2=0x7fee100af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 0x7fee100af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 -- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1015c9a0 msgr2=0x7fee100cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1015c9a0 0x7fee100cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0f400640 1 -- 172.21.15.28:0/2047512828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee08009b30 con 0x7fee100ad940 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.463+0000 7fee0ea00640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 0x7fee100af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:37.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee0f400640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fee0000e9b0 tx=0x7fee0000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00009670 con 0x7fee100ad940 2024-09-13T17:13:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0000c9a0 con 0x7fee100ad940 2024-09-13T17:13:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000cd50 con 0x7fee100ad940 2024-09-13T17:13:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee10070930 con 0x7fee100ad940 2024-09-13T17:13:37.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.464+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1015f860 con 0x7fee100ad940 2024-09-13T17:13:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.465+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee100afcd0 con 0x7fee100ad940 2024-09-13T17:13:37.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.466+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee00013470 con 0x7fee100ad940 2024-09-13T17:13:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.467+0000 7fee07400640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 0x7fedf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.467+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fee00089220 con 0x7fee100ad940 2024-09-13T17:13:37.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.467+0000 7fee0ea00640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 0x7fedf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:37.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.468+0000 7fee0ea00640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 0x7fedf006d0c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee10156fa0 tx=0x7fedfc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:37.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.471+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee100afcd0 con 0x7fee100ad940 2024-09-13T17:13:37.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:37.665+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fee100afcd0 con 0x7fee100ad940 2024-09-13T17:13:38.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.478+0000 7fee07400640 1 -- 172.21.15.28:0/2047512828 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7fee100afcd0 con 0x7fee100ad940 2024-09-13T17:13:38.480 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 msgr2=0x7fedf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 0x7fedf006d0c0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fee10156fa0 tx=0x7fedfc008040 comp rx=0 tx=0).stop 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 msgr2=0x7fee100af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fee0000e9b0 tx=0x7fee0000ee70 comp rx=0 tx=0).stop 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 shutdown_connections 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fedf006ac00 0x7fedf006d0c0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee1015c9a0 0x7fee100cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee100b2230 0x7fee100af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 --2- 172.21.15.28:0/2047512828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee100ad940 0x7fee100af250 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 >> 172.21.15.28:0/2047512828 conn(0x7fee100b0270 msgr2=0x7fee1006f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 shutdown_connections 2024-09-13T17:13:38.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.481+0000 7fee16800640 1 -- 172.21.15.28:0/2047512828 wait complete. 2024-09-13T17:13:38.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:13:38.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.661+0000 7f3ebe200640 1 Processor -- start 2024-09-13T17:13:38.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.661+0000 7f3ebe200640 1 -- start start 2024-09-13T17:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebe200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3eb815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebe200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3eb80b0b50 0x7f3eb80af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebe200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb815bb10 con 0x7f3eb806b360 2024-09-13T17:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebe200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb806b900 con 0x7f3eb80b0b50 2024-09-13T17:13:38.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebe200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb806bbe0 con 0x7f3eb8154a90 2024-09-13T17:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eafe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3eb806b360 0x7f3eb80b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58692/0 (socket says 172.21.15.28:58692) 2024-09-13T17:13:38.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eafe00640 1 -- 172.21.15.28:0/1335965730 learned_addr learned my addr 172.21.15.28:0/1335965730 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:38.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebd800640 1 --2- 172.21.15.28:0/1335965730 >> v2:172.21.15.189:3300/0 conn(0x7f3eb80b0b50 0x7f3eb80af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3ebce00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3eb815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206714542 0 0) 0x7f3eb815bb10 con 0x7f3eb806b360 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e9c003520 con 0x7f3eb806b360 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1208639867 0 0) 0x7f3e9c003520 con 0x7f3eb806b360 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 msgr2=0x7f3eb815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3eb815b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 >> v2:172.21.15.189:3300/0 conn(0x7f3eb80b0b50 msgr2=0x7f3eb80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 --2- 172.21.15.28:0/1335965730 >> v2:172.21.15.189:3300/0 conn(0x7f3eb80b0b50 0x7f3eb80af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb806bec0 con 0x7f3eb806b360 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3ea0001670 con 0x7f3eb806b360 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 >> v1:172.21.15.28:6789/0 conn(0x7f3eb806b360 legacy=0x7f3eb80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9c003970 0x7f3e9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c0047c0 0x7f3e9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3e9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb806b900 con 0x7f3e9c003970 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb806bbe0 con 0x7f3e9c0047c0 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.662+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c00b980 con 0x7f3eb8154a90 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9c003970 0x7f3e9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3eafe00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c0047c0 0x7f3e9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebd800640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3e9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 -- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 msgr2=0x7f3e9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb8154a90 0x7f3e9c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 -- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c0047c0 msgr2=0x7f3e9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c0047c0 0x7f3e9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.664+0000 7f3ebce00640 1 -- 172.21.15.28:0/1335965730 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e9c003520 con 0x7f3e9c003970 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebce00640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9c003970 0x7f3e9c003d90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009ac0 tx=0x7f3e9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ba88e3fdfbf89c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8031400 con 0x7f3e9c003970 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea802d050 con 0x7f3e9c003970 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3eb7e00640 1 -- 172.21.15.28:0/1335965730 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8031780 con 0x7f3e9c003970 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 -- 172.21.15.28:0/1335965730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9c003970 msgr2=0x7f3e9c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 --2- 172.21.15.28:0/1335965730 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e9c003970 0x7f3e9c003d90 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8009ac0 tx=0x7f3e9c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebd800640 1 -- 172.21.15.28:0/1335965730 reap_dead start 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 -- 172.21.15.28:0/1335965730 shutdown_connections 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 -- 172.21.15.28:0/1335965730 >> 172.21.15.28:0/1335965730 conn(0x7f3eb8072e40 msgr2=0x7f3eb8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 -- 172.21.15.28:0/1335965730 shutdown_connections 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.665+0000 7f3ebe200640 1 -- 172.21.15.28:0/1335965730 wait complete. 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.668+0000 7f3ebe200640 1 Processor -- start 2024-09-13T17:13:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.668+0000 7f3ebe200640 1 -- start start 2024-09-13T17:13:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c003970 0x7f3eb814cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb80ad7c0 0x7f3eb80adc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80abb00 con 0x7f3eb80af6f0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80ae180 con 0x7f3e9c003970 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebe200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb80aaef0 con 0x7f3eb80ad7c0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43730/0 (socket says 172.21.15.28:43730) 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebd800640 1 -- 172.21.15.28:0/648796790 learned_addr learned my addr 172.21.15.28:0/648796790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.669+0000 7f3ebce00640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c003970 0x7f3eb814cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3eafe00640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb80ad7c0 0x7f3eb80adc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 -- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb80ad7c0 msgr2=0x7f3eb80adc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb80ad7c0 0x7f3eb80adc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 -- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c003970 msgr2=0x7f3eb814cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c003970 0x7f3eb814cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 -- 172.21.15.28:0/648796790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80ab1d0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebd800640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f3eb815ab50 tx=0x7f3eb000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb000e040 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb000f040 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb0015430 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eb80ab4b0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.670+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb814d0a0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.671+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb80abcb0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.672+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3eb00157a0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.673+0000 7f3eb5600640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 0x7f3e9406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.673+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3eb00898e0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.674+0000 7f3ebce00640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 0x7f3e9406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.674+0000 7f3ebce00640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 0x7f3e9406d0c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8033000 tx=0x7f3ea8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:38.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.677+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3eb80abcb0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.922+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f3eb80abcb0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.923+0000 7f3eb5600640 1 -- 172.21.15.28:0/648796790 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3eb80abcb0 con 0x7f3eb80af6f0 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:13:38.924 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.925 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.926 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.927 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.930 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.931 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:13:38.932 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:13:38.933 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:13:38.934 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.935 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:13:38.936 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.937 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:38.937 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:13:38.937 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 msgr2=0x7f3e9406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 0x7f3e9406d0c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f3ea8033000 tx=0x7f3ea8008030 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 msgr2=0x7f3eb80afb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f3eb815ab50 tx=0x7f3eb000f580 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 shutdown_connections 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e9406ac00 0x7f3e9406d0c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3eb80af6f0 0x7f3eb80afb70 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3eb80ad7c0 0x7f3eb80adc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 --2- 172.21.15.28:0/648796790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e9c003970 0x7f3eb814cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 >> 172.21.15.28:0/648796790 conn(0x7f3eb8072e40 msgr2=0x7f3eb8155170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:38.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 shutdown_connections 2024-09-13T17:13:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:38.925+0000 7f3ebe200640 1 -- 172.21.15.28:0/648796790 wait complete. 2024-09-13T17:13:38.942 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.097+0000 7f4739a00640 1 Processor -- start 2024-09-13T17:13:39.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.097+0000 7f4739a00640 1 -- start start 2024-09-13T17:13:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f473415c750 0x7f47340d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47340b4ad0 0x7f4734069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340d0d80 con 0x7f47340d1d10 2024-09-13T17:13:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340d0960 con 0x7f47340b4ad0 2024-09-13T17:13:39.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473406aa60 con 0x7f473415c750 2024-09-13T17:13:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47340b4ad0 0x7f4734069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f47340b4ad0 0x7f4734069e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43746/0 (socket says 172.21.15.28:43746) 2024-09-13T17:13:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4739000640 1 -- 172.21.15.28:0/140379323 learned_addr learned my addr 172.21.15.28:0/140379323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.098+0000 7f4733e00640 1 --2- 172.21.15.28:0/140379323 >> v2:172.21.15.189:3300/0 conn(0x7f473415c750 0x7f47340d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.099+0000 7f4739000640 1 -- 172.21.15.28:0/140379323 >> v2:172.21.15.189:3300/0 conn(0x7f473415c750 msgr2=0x7f47340d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.099+0000 7f4739000640 1 --2- 172.21.15.28:0/140379323 >> v2:172.21.15.189:3300/0 conn(0x7f473415c750 0x7f47340d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.099+0000 7f4739000640 1 -- 172.21.15.28:0/140379323 >> v1:172.21.15.28:6789/0 conn(0x7f47340d1d10 legacy=0x7f47340b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.099+0000 7f4739000640 1 -- 172.21.15.28:0/140379323 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473406ad40 con 0x7f47340b4ad0 2024-09-13T17:13:39.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.100+0000 7f4739000640 1 --2- 172.21.15.28:0/140379323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47340b4ad0 0x7f4734069e80 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f473408b150 tx=0x7f472c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a689be8ca05b981 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.100+0000 7f4732a00640 1 -- 172.21.15.28:0/140379323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c031600 con 0x7f47340b4ad0 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.100+0000 7f4732a00640 1 -- 172.21.15.28:0/140379323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c02d050 con 0x7f47340b4ad0 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.100+0000 7f4732a00640 1 -- 172.21.15.28:0/140379323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c031980 con 0x7f47340b4ad0 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 -- 172.21.15.28:0/140379323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47340b4ad0 msgr2=0x7f4734069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 --2- 172.21.15.28:0/140379323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47340b4ad0 0x7f4734069e80 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f473408b150 tx=0x7f472c02a6a0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 -- 172.21.15.28:0/140379323 shutdown_connections 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 --2- 172.21.15.28:0/140379323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f47340b4ad0 0x7f4734069e80 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 --2- 172.21.15.28:0/140379323 >> v2:172.21.15.189:3300/0 conn(0x7f473415c750 0x7f47340d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 -- 172.21.15.28:0/140379323 >> 172.21.15.28:0/140379323 conn(0x7f47340cf5a0 msgr2=0x7f47340cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:39.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 -- 172.21.15.28:0/140379323 shutdown_connections 2024-09-13T17:13:39.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.101+0000 7f4739a00640 1 -- 172.21.15.28:0/140379323 wait complete. 2024-09-13T17:13:39.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.103+0000 7f4739a00640 1 Processor -- start 2024-09-13T17:13:39.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.103+0000 7f4739a00640 1 -- start start 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47340b4ad0 0x7f47340cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47341492a0 0x7f47340b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47341f6160 con 0x7f473415c750 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47340c9a10 con 0x7f47341492a0 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47340c9cf0 con 0x7f47340b4ad0 2024-09-13T17:13:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43760/0 (socket says 172.21.15.28:43760) 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 -- 172.21.15.28:0/3832158241 learned_addr learned my addr 172.21.15.28:0/3832158241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733e00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47340b4ad0 0x7f47340cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4739000640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47341492a0 0x7f47340b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 -- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47340b4ad0 msgr2=0x7f47340cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47340b4ad0 0x7f47340cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 -- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47341492a0 msgr2=0x7f47340b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47341492a0 0x7f47340b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.104+0000 7f4733400640 1 -- 172.21.15.28:0/3832158241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c009b30 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f4733400640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f472000c450 tx=0x7f472000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47200052d0 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4720005430 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47341f63b0 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720012870 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.105+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47341f6970 con 0x7f473415c750 2024-09-13T17:13:39.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.106+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47340d17d0 con 0x7f473415c750 2024-09-13T17:13:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.107+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f47200129d0 con 0x7f473415c750 2024-09-13T17:13:39.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.108+0000 7f472be00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.108+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4720088a10 con 0x7f473415c750 2024-09-13T17:13:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.108+0000 7f4733e00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.109+0000 7f4733e00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4724009aa0 tx=0x7f4724009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.112+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4720053fe0 con 0x7f473415c750 2024-09-13T17:13:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.292+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47340d0b60 con 0x7f473415c750 2024-09-13T17:13:39.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.293+0000 7f472be00640 1 -- 172.21.15.28:0/3832158241 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12858 (secure 0 0 0) 0x7f4720057600 con 0x7f473415c750 2024-09-13T17:13:39.294 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:39.294 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":89,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:38.475306+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 msgr2=0x7f471406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4724009aa0 tx=0x7f4724009190 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 msgr2=0x7f47341f5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f472000c450 tx=0x7f472000c910 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 shutdown_connections 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47341492a0 0x7f47340b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473415c750 0x7f47341f5890 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 --2- 172.21.15.28:0/3832158241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47340b4ad0 0x7f47340cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 >> 172.21.15.28:0/3832158241 conn(0x7f47340cf5a0 msgr2=0x7f47340b42b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 shutdown_connections 2024-09-13T17:13:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.295+0000 7f4739a00640 1 -- 172.21.15.28:0/3832158241 wait complete. 2024-09-13T17:13:39.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:39.313 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:39.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.471+0000 7f4115c00640 1 Processor -- start 2024-09-13T17:13:39.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.471+0000 7f4115c00640 1 -- start start 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4115c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f41100ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4115c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4110070470 con 0x7f41100ad940 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4115c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41100ace30 con 0x7f41100b2400 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4115c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100ad110 con 0x7f411015c9a0 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4106000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f41100ad940 0x7f41100b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58714/0 (socket says 172.21.15.28:58714) 2024-09-13T17:13:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f4106000640 1 -- 172.21.15.28:0/163519781 learned_addr learned my addr 172.21.15.28:0/163519781 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f410ea00640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f41100ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964054177 0 0) 0x7f4110070470 con 0x7f41100ad940 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.473+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40f4003520 con 0x7f41100ad940 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1291031562 0 0) 0x7f40f4003520 con 0x7f41100ad940 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 msgr2=0x7f41100ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f41100ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 >> v1:172.21.15.189:6789/0 conn(0x7f41100b2400 legacy=0x7f411006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411006b170 con 0x7f41100ad940 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f40f8001670 con 0x7f41100ad940 2024-09-13T17:13:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 >> v1:172.21.15.28:6789/0 conn(0x7f41100ad940 legacy=0x7f41100b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f40f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f4003f20 0x7f40f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f40f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100ad110 con 0x7f40f4004d00 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40f400baf0 con 0x7f40f4003f20 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40f400bcb0 con 0x7f411015c9a0 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410f400640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f40f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f410ea00640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f40f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.474+0000 7f4106000640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f4003f20 0x7f40f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.475+0000 7f410f400640 1 -- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 msgr2=0x7f40f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.475+0000 7f410f400640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f411015c9a0 0x7f40f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.475+0000 7f410f400640 1 -- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f4003f20 msgr2=0x7f40f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.475+0000 7f410f400640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f40f4003f20 0x7f40f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.475+0000 7f410f400640 1 -- 172.21.15.28:0/163519781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4003520 con 0x7f40f4004d00 2024-09-13T17:13:39.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f410f400640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f40f4005180 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f411006f3a0 tx=0x7f40f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3995056866d8c812 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410802ae10 con 0x7f40f4004d00 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4108003970 con 0x7f40f4004d00 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f410e000640 1 -- 172.21.15.28:0/163519781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4108003d20 con 0x7f40f4004d00 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f4115c00640 1 -- 172.21.15.28:0/163519781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 msgr2=0x7f40f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f4115c00640 1 --2- 172.21.15.28:0/163519781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f40f4005180 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f411006f3a0 tx=0x7f40f40057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.476+0000 7f410f400640 1 -- 172.21.15.28:0/163519781 reap_dead start 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.477+0000 7f4115c00640 1 -- 172.21.15.28:0/163519781 shutdown_connections 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.477+0000 7f4115c00640 1 -- 172.21.15.28:0/163519781 >> 172.21.15.28:0/163519781 conn(0x7f41100b0270 msgr2=0x7f41100b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.477+0000 7f4115c00640 1 -- 172.21.15.28:0/163519781 shutdown_connections 2024-09-13T17:13:39.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.477+0000 7f4115c00640 1 -- 172.21.15.28:0/163519781 wait complete. 2024-09-13T17:13:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.479+0000 7f4115c00640 1 Processor -- start 2024-09-13T17:13:39.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.479+0000 7f4115c00640 1 -- start start 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41101f2aa0 0x7f4110152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f411015f220 0x7f411015f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411015fbe0 con 0x7f40f4004d00 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100cd620 con 0x7f411015f220 2024-09-13T17:13:39.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.480+0000 7f4115c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100cd340 con 0x7f41101f2aa0 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43774/0 (socket says 172.21.15.28:43774) 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 -- 172.21.15.28:0/540549670 learned_addr learned my addr 172.21.15.28:0/540549670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410f400640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f411015f220 0x7f411015f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f4106000640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41101f2aa0 0x7f4110152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 -- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41101f2aa0 msgr2=0x7f4110152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41101f2aa0 0x7f4110152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 -- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f411015f220 msgr2=0x7f411015f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f411015f220 0x7f411015f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.481+0000 7f410ea00640 1 -- 172.21.15.28:0/540549670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110070930 con 0x7f40f4004d00 2024-09-13T17:13:39.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f410ea00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f41000119d0 tx=0x7f4100011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410000fb50 con 0x7f40f4004d00 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410000fcb0 con 0x7f40f4004d00 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4100014440 con 0x7f40f4004d00 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4110154ad0 con 0x7f40f4004d00 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.482+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4110155090 con 0x7f40f4004d00 2024-09-13T17:13:39.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.483+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110070730 con 0x7f40f4004d00 2024-09-13T17:13:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.484+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f41000145a0 con 0x7f40f4004d00 2024-09-13T17:13:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.485+0000 7f4107400640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 0x7f40ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.485+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f41000897a0 con 0x7f40f4004d00 2024-09-13T17:13:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.485+0000 7f4106000640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 0x7f40ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.486+0000 7f4106000640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 0x7f40ec06d0f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f411015b970 tx=0x7f40f8000f80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.489+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4100054ce0 con 0x7f40f4004d00 2024-09-13T17:13:39.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.668+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f411015cd80 con 0x7f40f4004d00 2024-09-13T17:13:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.669+0000 7f4107400640 1 -- 172.21.15.28:0/540549670 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4100058300 con 0x7f40f4004d00 2024-09-13T17:13:39.670 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 msgr2=0x7f40ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 0x7f40ec06d0f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f411015b970 tx=0x7f40f8000f80 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 msgr2=0x7f411015cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f41000119d0 tx=0x7f4100011e90 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 shutdown_connections 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f40ec06ac30 0x7f40ec06d0f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f411015f220 0x7f411015f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41101f2aa0 0x7f4110152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 --2- 172.21.15.28:0/540549670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f40f4004d00 0x7f411015cf90 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 >> 172.21.15.28:0/540549670 conn(0x7f41100b0270 msgr2=0x7f41100af880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 shutdown_connections 2024-09-13T17:13:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.671+0000 7f4115c00640 1 -- 172.21.15.28:0/540549670 wait complete. 2024-09-13T17:13:39.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:13:39.820 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:13:39.824 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:13:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.842+0000 7f81f6a00640 1 Processor -- start 2024-09-13T17:13:39.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.842+0000 7f81f6a00640 1 -- start start 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.843+0000 7f81f6a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 0x7f81f0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.843+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.843+0000 7f81f6a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81f006c680 con 0x7f81f015ab60 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.843+0000 7f81f6a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81f006c8b0 con 0x7f81f015a780 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.843+0000 7f81f6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f006cb90 con 0x7f81f006c250 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:39744/0 (socket says 172.21.15.28:39744) 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81efe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 0x7f81f0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 -- 172.21.15.28:0/4043823628 learned_addr learned my addr 172.21.15.28:0/4043823628 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81ef400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f81f015ab60 0x7f81f006bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:42758/0 (socket says 172.21.15.28:42758) 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81eea00640 1 -- 172.21.15.28:0/4043823628 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513150271 0 0) 0x7f81f006c680 con 0x7f81f015ab60 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81eea00640 1 -- 172.21.15.28:0/4043823628 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d4003520 con 0x7f81f015ab60 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 -- 172.21.15.28:0/4043823628 >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 msgr2=0x7f81f0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 --2- 172.21.15.28:0/4043823628 >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 0x7f81f0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 -- 172.21.15.28:0/4043823628 >> v1:172.21.15.189:6789/0 conn(0x7f81f015ab60 legacy=0x7f81f006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81f4a00640 1 -- 172.21.15.28:0/4043823628 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f00cc980 con 0x7f81f006c250 2024-09-13T17:13:39.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.844+0000 7f81efe00640 1 --2- 172.21.15.28:0/4043823628 >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 0x7f81f0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81f4a00640 1 --2- 172.21.15.28:0/4043823628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009870 tx=0x7f81e802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43e8542b843d2ba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81eea00640 1 -- 172.21.15.28:0/4043823628 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8032440 con 0x7f81f006c250 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81eea00640 1 -- 172.21.15.28:0/4043823628 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e80325a0 con 0x7f81f006c250 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81eea00640 1 -- 172.21.15.28:0/4043823628 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8032930 con 0x7f81f006c250 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81f6a00640 1 -- 172.21.15.28:0/4043823628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 msgr2=0x7f81f00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.846+0000 7f81f6a00640 1 --2- 172.21.15.28:0/4043823628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009870 tx=0x7f81e802a250 comp rx=0 tx=0).stop 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 -- 172.21.15.28:0/4043823628 shutdown_connections 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 --2- 172.21.15.28:0/4043823628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f00712c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 --2- 172.21.15.28:0/4043823628 >> v2:172.21.15.189:3300/0 conn(0x7f81f015a780 0x7f81f0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 -- 172.21.15.28:0/4043823628 >> 172.21.15.28:0/4043823628 conn(0x7f81f00b0270 msgr2=0x7f81f00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 -- 172.21.15.28:0/4043823628 shutdown_connections 2024-09-13T17:13:39.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.847+0000 7f81f6a00640 1 -- 172.21.15.28:0/4043823628 wait complete. 2024-09-13T17:13:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.848+0000 7f81f6a00640 1 Processor -- start 2024-09-13T17:13:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.848+0000 7f81f6a00640 1 -- start start 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f0144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f015a780 0x7f81f00aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f015df10 con 0x7f81f015ab60 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f01450b0 con 0x7f81f015a780 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.849+0000 7f81f6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81f015d390 con 0x7f81f006c250 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43778/0 (socket says 172.21.15.28:43778) 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 -- 172.21.15.28:0/670888722 learned_addr learned my addr 172.21.15.28:0/670888722 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81ef400640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f015a780 0x7f81f00aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81efe00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f0144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 -- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 msgr2=0x7f81f0144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f0144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 -- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f015a780 msgr2=0x7f81f00aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f015a780 0x7f81f00aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:39.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 -- 172.21.15.28:0/670888722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0009a70 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.850+0000 7f81f4a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009870 tx=0x7f81e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.851+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e803d830 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.851+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e803d9b0 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.851+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e803dd60 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.851+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e80094d0 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.851+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f015d950 con 0x7f81f015ab60 2024-09-13T17:13:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.852+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f006bb40 con 0x7f81f015ab60 2024-09-13T17:13:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.853+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f81e8039040 con 0x7f81f015ab60 2024-09-13T17:13:39.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.854+0000 7f81e7e00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 0x7f81cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.854+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f81e80a9a00 con 0x7f81f015ab60 2024-09-13T17:13:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.854+0000 7f81efe00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 0x7f81cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:39.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.855+0000 7f81efe00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 0x7f81cc06d0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009fd0 tx=0x7f81e000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:39.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:39.858+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f81e8074f40 con 0x7f81f015ab60 2024-09-13T17:13:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.043+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81f0159a10 con 0x7f81f015ab60 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.044+0000 7f81e7e00640 1 -- 172.21.15.28:0/670888722 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7f81e8078560 con 0x7f81f015ab60 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:13:40.045 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:13:40.046 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:13:40.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 msgr2=0x7f81cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 0x7f81cc06d0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009fd0 tx=0x7f81e000b040 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 msgr2=0x7f81f00ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009870 tx=0x7f81e80046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 shutdown_connections 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f81cc06ac00 0x7f81cc06d0c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81f015ab60 0x7f81f00ab330 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f015a780 0x7f81f00aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 --2- 172.21.15.28:0/670888722 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81f006c250 0x7f81f0144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.046+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 >> 172.21.15.28:0/670888722 conn(0x7f81f00b0270 msgr2=0x7f81f00af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.047+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 shutdown_connections 2024-09-13T17:13:40.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.047+0000 7f81f6a00640 1 -- 172.21.15.28:0/670888722 wait complete. 2024-09-13T17:13:40.064 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:13:40.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:13:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.225+0000 7f3bdc000640 1 Processor -- start 2024-09-13T17:13:40.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.225+0000 7f3bdc000640 1 -- start start 2024-09-13T17:13:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bdc000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd415a7a0 0x7f3bd40ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bdc000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bd40ce9d0 0x7f3bd40b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bdc000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd406c950 con 0x7f3bd40b1ab0 2024-09-13T17:13:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bdc000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd40cee10 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bdc000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd40b1ec0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bd8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bd40ce9d0 0x7f3bd40b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bd8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bd40ce9d0 0x7f3bd40b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43790/0 (socket says 172.21.15.28:43790) 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.226+0000 7f3bd8c00640 1 -- 172.21.15.28:0/3141094305 learned_addr learned my addr 172.21.15.28:0/3141094305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.227+0000 7f3bd9600640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd415a7a0 0x7f3bd40ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.227+0000 7f3bd8c00640 1 -- 172.21.15.28:0/3141094305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd415a7a0 msgr2=0x7f3bd40ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.227+0000 7f3bd8c00640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd415a7a0 0x7f3bd40ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.227+0000 7f3bd8c00640 1 -- 172.21.15.28:0/3141094305 >> v1:172.21.15.28:6789/0 conn(0x7f3bd40b1ab0 legacy=0x7f3bd406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.227+0000 7f3bd8c00640 1 -- 172.21.15.28:0/3141094305 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd40b2150 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.228+0000 7f3bd8c00640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40ce9d0 0x7f3bd40b1570 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009870 tx=0x7f3bc002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e1b4445e7d357d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.228+0000 7f3bd3e00640 1 -- 172.21.15.28:0/3141094305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0032440 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.228+0000 7f3bd3e00640 1 -- 172.21.15.28:0/3141094305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc00325a0 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.228+0000 7f3bd3e00640 1 -- 172.21.15.28:0/3141094305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0032930 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 -- 172.21.15.28:0/3141094305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40ce9d0 msgr2=0x7f3bd40b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40ce9d0 0x7f3bd40b1570 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009870 tx=0x7f3bc002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 -- 172.21.15.28:0/3141094305 shutdown_connections 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd40ce9d0 0x7f3bd40b1570 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 --2- 172.21.15.28:0/3141094305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd415a7a0 0x7f3bd40ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 -- 172.21.15.28:0/3141094305 >> 172.21.15.28:0/3141094305 conn(0x7f3bd4069e80 msgr2=0x7f3bd406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 -- 172.21.15.28:0/3141094305 shutdown_connections 2024-09-13T17:13:40.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.229+0000 7f3bdc000640 1 -- 172.21.15.28:0/3141094305 wait complete. 2024-09-13T17:13:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.230+0000 7f3bdc000640 1 Processor -- start 2024-09-13T17:13:40.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.231+0000 7f3bdc000640 1 -- start start 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.231+0000 7f3bdc000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd40b1ab0 0x7f3bd415db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bdc000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd40ce9d0 0x7f3bd41563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bdc000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bdc000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd4155f80 con 0x7f3bd415a7a0 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bdc000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd415bec0 con 0x7f3bd40b1ab0 2024-09-13T17:13:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bdc000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd415c0f0 con 0x7f3bd40ce9d0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43802/0 (socket says 172.21.15.28:43802) 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 -- 172.21.15.28:0/2234304566 learned_addr learned my addr 172.21.15.28:0/2234304566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bd9600640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd40b1ab0 0x7f3bd415db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bd8c00640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd40ce9d0 0x7f3bd41563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 -- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd40ce9d0 msgr2=0x7f3bd41563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd40ce9d0 0x7f3bd41563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 -- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd40b1ab0 msgr2=0x7f3bd415db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd40b1ab0 0x7f3bd415db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.232+0000 7f3bda000640 1 -- 172.21.15.28:0/2234304566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc00094d0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bda000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00c6b0 tx=0x7f3bcc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc003b70 con 0x7f3bd415a7a0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bcc003cf0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bcc003ea0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd415c3d0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.233+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd4155ba0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.234+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd41560c0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.235+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3bcc005310 con 0x7f3bd415a7a0 2024-09-13T17:13:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.236+0000 7f3bd1600640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 0x7f3bb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.236+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3bcc088b00 con 0x7f3bd415a7a0 2024-09-13T17:13:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.237+0000 7f3bd9600640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 0x7f3bb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.237+0000 7f3bd9600640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 0x7f3bb406d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:40.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.240+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3bd41560c0 con 0x7f3bd415a7a0 2024-09-13T17:13:40.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:40.421+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3bd41560c0 con 0x7f3bd415a7a0 2024-09-13T17:13:41.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.106+0000 7f3bd1600640 1 -- 172.21.15.28:0/2234304566 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3bd41560c0 con 0x7f3bd415a7a0 2024-09-13T17:13:41.107 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 msgr2=0x7f3bb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 0x7f3bb406d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8006fd0 tx=0x7f3bc8008040 comp rx=0 tx=0).stop 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 msgr2=0x7f3bd415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3bcc00c6b0 tx=0x7f3bcc00cb70 comp rx=0 tx=0).stop 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 shutdown_connections 2024-09-13T17:13:41.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bb406ac00 0x7f3bb406d0c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd415a7a0 0x7f3bd415b980 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd40ce9d0 0x7f3bd41563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 --2- 172.21.15.28:0/2234304566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd40b1ab0 0x7f3bd415db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.111+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 >> 172.21.15.28:0/2234304566 conn(0x7f3bd4069e80 msgr2=0x7f3bd40ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.112+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 shutdown_connections 2024-09-13T17:13:41.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.112+0000 7f3bdc000640 1 -- 172.21.15.28:0/2234304566 wait complete. 2024-09-13T17:13:41.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:13:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.299+0000 7fbc73e00640 1 Processor -- start 2024-09-13T17:13:41.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.299+0000 7fbc73e00640 1 -- start start 2024-09-13T17:13:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 0x7fbc740b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:41.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc73e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc740c3850 con 0x7fbc74159440 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc73e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc7415a930 con 0x7fbc74159820 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc73e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc74159d20 con 0x7fbc740b0b50 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc72a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbc74159440 0x7fbc740b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58736/0 (socket says 172.21.15.28:58736) 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.300+0000 7fbc72a00640 1 -- 172.21.15.28:0/1071040871 learned_addr learned my addr 172.21.15.28:0/1071040871 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc6a000640 1 --2- 172.21.15.28:0/1071040871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 0x7fbc740b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088149616 0 0) 0x7fbc740c3850 con 0x7fbc74159440 2024-09-13T17:13:41.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc58005730 con 0x7fbc74159440 2024-09-13T17:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706336059 0 0) 0x7fbc7415a930 con 0x7fbc74159820 2024-09-13T17:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc740c3850 con 0x7fbc74159820 2024-09-13T17:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339820692 0 0) 0x7fbc58005730 con 0x7fbc74159440 2024-09-13T17:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc6a000640 1 -- 172.21.15.28:0/1071040871 >> v1:172.21.15.189:6789/0 conn(0x7fbc74159820 legacy=0x7fbc7415adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc6a000640 1 -- 172.21.15.28:0/1071040871 >> v1:172.21.15.28:6789/0 conn(0x7fbc74159440 legacy=0x7fbc740b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.301+0000 7fbc6a000640 1 -- 172.21.15.28:0/1071040871 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7415a000 con 0x7fbc740b0b50 2024-09-13T17:13:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.302+0000 7fbc6a000640 1 --2- 172.21.15.28:0/1071040871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 0x7fbc740b22e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0097d0 tx=0x7fbc5c02a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ee562276f8a4fea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64001670 con 0x7fbc740b0b50 2024-09-13T17:13:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c02acc0 con 0x7fbc740b0b50 2024-09-13T17:13:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc72000640 1 -- 172.21.15.28:0/1071040871 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c032660 con 0x7fbc740b0b50 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 -- 172.21.15.28:0/1071040871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 msgr2=0x7fbc740b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1071040871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 0x7fbc740b22e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c0097d0 tx=0x7fbc5c02a230 comp rx=0 tx=0).stop 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 -- 172.21.15.28:0/1071040871 shutdown_connections 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1071040871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc740b0b50 0x7fbc740b22e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 -- 172.21.15.28:0/1071040871 >> 172.21.15.28:0/1071040871 conn(0x7fbc74072e40 msgr2=0x7fbc74073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 -- 172.21.15.28:0/1071040871 shutdown_connections 2024-09-13T17:13:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.303+0000 7fbc73e00640 1 -- 172.21.15.28:0/1071040871 wait complete. 2024-09-13T17:13:41.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.305+0000 7fbc73e00640 1 Processor -- start 2024-09-13T17:13:41.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.305+0000 7fbc73e00640 1 -- start start 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc740b0b50 0x7fbc74140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc74159820 0x7fbc74157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740af090 con 0x7fbc74159440 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740a9480 con 0x7fbc740b0b50 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc740aea40 con 0x7fbc74159820 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc6a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc6a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43816/0 (socket says 172.21.15.28:43816) 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc6a000640 1 -- 172.21.15.28:0/1846730286 learned_addr learned my addr 172.21.15.28:0/1846730286 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc72a00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc740b0b50 0x7fbc74140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.306+0000 7fbc73400640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc74159820 0x7fbc74157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 -- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc74159820 msgr2=0x7fbc74157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc74159820 0x7fbc74157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 -- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc740b0b50 msgr2=0x7fbc74140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc740b0b50 0x7fbc74140920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 -- 172.21.15.28:0/1846730286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c009400 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6a000640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c035040 tx=0x7fbc5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c03b810 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5c03b990 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c03bd40 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.307+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc740aed20 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.308+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc740afa60 con 0x7fbc74159440 2024-09-13T17:13:41.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.308+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc740afcb0 con 0x7fbc74159440 2024-09-13T17:13:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.310+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbc5c039040 con 0x7fbc74159440 2024-09-13T17:13:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.310+0000 7fbc6b400640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 0x7fbc5006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.310+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fbc5c0a96e0 con 0x7fbc74159440 2024-09-13T17:13:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.311+0000 7fbc72a00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 0x7fbc5006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.312+0000 7fbc72a00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 0x7fbc5006d0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000f60 tx=0x7fbc64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:41.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.316+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbc5c074a90 con 0x7fbc74159440 2024-09-13T17:13:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:41.495+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbc740aeb80 con 0x7fbc74159440 2024-09-13T17:13:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.114+0000 7fbc6b400640 1 -- 172.21.15.28:0/1846730286 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbc5c04c0d0 con 0x7fbc74159440 2024-09-13T17:13:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:13:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.121+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 msgr2=0x7fbc5006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.121+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 0x7fbc5006d0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000f60 tx=0x7fbc64009040 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.121+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 msgr2=0x7fbc740af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.121+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c035040 tx=0x7fbc5c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 shutdown_connections 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc5006ac00 0x7fbc5006d0c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc74159820 0x7fbc74157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc74159440 0x7fbc740af520 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 --2- 172.21.15.28:0/1846730286 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc740b0b50 0x7fbc74140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.122+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 >> 172.21.15.28:0/1846730286 conn(0x7fbc74072e40 msgr2=0x7fbc740b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.124+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 shutdown_connections 2024-09-13T17:13:42.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.124+0000 7fbc73e00640 1 -- 172.21.15.28:0/1846730286 wait complete. 2024-09-13T17:13:42.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:13:42.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.311+0000 7f6155a00640 1 Processor -- start 2024-09-13T17:13:42.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.311+0000 7f6155a00640 1 -- start start 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6155a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6150070e30 0x7f6150071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6155a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6150072140 0x7f61500cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6155a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6150159e10 con 0x7f6150071210 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6155a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615015a0f0 con 0x7f6150072140 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6155a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f615015a3d0 con 0x7f6150070e30 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6147400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6150072140 0x7f61500cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6147400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6150072140 0x7f61500cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43830/0 (socket says 172.21.15.28:43830) 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.312+0000 7f6147400640 1 -- 172.21.15.28:0/823759936 learned_addr learned my addr 172.21.15.28:0/823759936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.313+0000 7f614fe00640 1 --2- 172.21.15.28:0/823759936 >> v2:172.21.15.189:3300/0 conn(0x7f6150070e30 0x7f6150071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.313+0000 7f6147400640 1 -- 172.21.15.28:0/823759936 >> v2:172.21.15.189:3300/0 conn(0x7f6150070e30 msgr2=0x7f6150071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.313+0000 7f6147400640 1 --2- 172.21.15.28:0/823759936 >> v2:172.21.15.189:3300/0 conn(0x7f6150070e30 0x7f6150071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.313+0000 7f6147400640 1 -- 172.21.15.28:0/823759936 >> v1:172.21.15.189:6789/0 conn(0x7f6150071210 legacy=0x7f615015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.313+0000 7f6147400640 1 -- 172.21.15.28:0/823759936 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615015a6b0 con 0x7f6150072140 2024-09-13T17:13:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.314+0000 7f6147400640 1 --2- 172.21.15.28:0/823759936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150072140 0x7f61500cd870 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f615006d430 tx=0x7f613802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0efac1d2a942a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.314+0000 7f614f400640 1 -- 172.21.15.28:0/823759936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138032440 con 0x7f6150072140 2024-09-13T17:13:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.314+0000 7f614f400640 1 -- 172.21.15.28:0/823759936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61380325a0 con 0x7f6150072140 2024-09-13T17:13:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.314+0000 7f614f400640 1 -- 172.21.15.28:0/823759936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138032930 con 0x7f6150072140 2024-09-13T17:13:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 -- 172.21.15.28:0/823759936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150072140 msgr2=0x7f61500cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 --2- 172.21.15.28:0/823759936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150072140 0x7f61500cd870 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f615006d430 tx=0x7f613802a270 comp rx=0 tx=0).stop 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 -- 172.21.15.28:0/823759936 shutdown_connections 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 --2- 172.21.15.28:0/823759936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150072140 0x7f61500cd870 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 --2- 172.21.15.28:0/823759936 >> v2:172.21.15.189:3300/0 conn(0x7f6150070e30 0x7f6150071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 -- 172.21.15.28:0/823759936 >> 172.21.15.28:0/823759936 conn(0x7f6150069e80 msgr2=0x7f615006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 -- 172.21.15.28:0/823759936 shutdown_connections 2024-09-13T17:13:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.315+0000 7f6155a00640 1 -- 172.21.15.28:0/823759936 wait complete. 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.316+0000 7f6155a00640 1 Processor -- start 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.316+0000 7f6155a00640 1 -- start start 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 0x7f6150139930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6150072140 0x7f615013b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61500cac10 con 0x7f6150071210 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6150139fc0 con 0x7f6150070e30 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6155a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f615013a2a0 con 0x7f6150072140 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.317+0000 7f6147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43846/0 (socket says 172.21.15.28:43846) 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 -- 172.21.15.28:0/1209898077 learned_addr learned my addr 172.21.15.28:0/1209898077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f614fe00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 0x7f6150139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6155000640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6150072140 0x7f615013b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 -- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6150072140 msgr2=0x7f615013b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6150072140 0x7f615013b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 -- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 msgr2=0x7f6150139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 0x7f6150139930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 -- 172.21.15.28:0/1209898077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61380094d0 con 0x7f6150071210 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f614fe00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 0x7f6150139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f6147400640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f613802a2b0 tx=0x7f61380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.318+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138032440 con 0x7f6150071210 2024-09-13T17:13:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.319+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138030ba0 con 0x7f6150071210 2024-09-13T17:13:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.319+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613803d440 con 0x7f6150071210 2024-09-13T17:13:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.319+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615013a580 con 0x7f6150071210 2024-09-13T17:13:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.319+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f615013ab40 con 0x7f6150071210 2024-09-13T17:13:42.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.320+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f615013ad30 con 0x7f6150071210 2024-09-13T17:13:42.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.321+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6138039040 con 0x7f6150071210 2024-09-13T17:13:42.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.322+0000 7f614cc00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 0x7f612c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.322+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f61380a9f20 con 0x7f6150071210 2024-09-13T17:13:42.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.322+0000 7f614fe00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 0x7f612c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.323+0000 7f614fe00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 0x7f612c06d0f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6140006fd0 tx=0x7f6140008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.327+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f61380750c0 con 0x7f6150071210 2024-09-13T17:13:42.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.544+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f615013a3e0 con 0x7f6150071210 2024-09-13T17:13:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 12 and data pool 13 2024-09-13T17:13:42.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.551+0000 7f614cc00640 1 -- 172.21.15.28:0/1209898077 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v74) v1 ==== 174+0+0 (secure 0 0 0) 0x7f615013a3e0 con 0x7f6150071210 2024-09-13T17:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.556+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 msgr2=0x7f612c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.556+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 0x7f612c06d0f0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f6140006fd0 tx=0x7f6140008040 comp rx=0 tx=0).stop 2024-09-13T17:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.556+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 msgr2=0x7f61500ca6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.556+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f613802a2b0 tx=0x7f61380046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.558+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 shutdown_connections 2024-09-13T17:13:42.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.559+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612c06ac30 0x7f612c06d0f0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.559+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6150072140 0x7f615013b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.559+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6150071210 0x7f61500ca6d0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.559+0000 7f6155a00640 1 --2- 172.21.15.28:0/1209898077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6150070e30 0x7f6150139930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.559+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 >> 172.21.15.28:0/1209898077 conn(0x7f6150069e80 msgr2=0x7f615006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:42.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.561+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 shutdown_connections 2024-09-13T17:13:42.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.561+0000 7f6155a00640 1 -- 172.21.15.28:0/1209898077 wait complete. 2024-09-13T17:13:42.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:42.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.747+0000 7fe0f0400640 1 Processor -- start 2024-09-13T17:13:42.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.747+0000 7fe0f0400640 1 -- start start 2024-09-13T17:13:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.748+0000 7fe0f0400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.748+0000 7fe0f0400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0e8160310 0x7fe0e8160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.748+0000 7fe0f0400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e8146550 con 0x7fe0e80ac850 2024-09-13T17:13:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.748+0000 7fe0f0400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e815cd20 con 0x7fe0e8160310 2024-09-13T17:13:42.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.748+0000 7fe0f0400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e815d280 con 0x7fe0e8146170 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ed000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe0e8160310 0x7fe0e8160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ee400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fe0e80ac850 0x7fe0e806b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:42782/0 (socket says 172.21.15.28:42782) 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ee400640 1 -- 172.21.15.28:0/1368114813 learned_addr learned my addr 172.21.15.28:0/1368114813 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0eda00640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ed000640 1 -- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 msgr2=0x7fe0e815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:42.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ed000640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ed000640 1 -- 172.21.15.28:0/1368114813 >> v1:172.21.15.189:6789/0 conn(0x7fe0e80ac850 legacy=0x7fe0e806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0ed000640 1 -- 172.21.15.28:0/1368114813 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e815dc50 con 0x7fe0e8160310 2024-09-13T17:13:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.749+0000 7fe0e7e00640 1 -- 172.21.15.28:0/1368114813 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330564367 0 0) 0x7fe0e8146550 con 0x7fe0e80ac850 2024-09-13T17:13:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.750+0000 7fe0ed000640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e8160750 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40097a0 tx=0x7fe0d402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f05cc1ce720079e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0e7e00640 1 -- 172.21.15.28:0/1368114813 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4032440 con 0x7fe0e8160310 2024-09-13T17:13:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0e7e00640 1 -- 172.21.15.28:0/1368114813 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d40325a0 con 0x7fe0e8160310 2024-09-13T17:13:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0e7e00640 1 -- 172.21.15.28:0/1368114813 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4032930 con 0x7fe0e8160310 2024-09-13T17:13:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 -- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 msgr2=0x7fe0e8160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e8160750 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe0d40097a0 tx=0x7fe0d402a180 comp rx=0 tx=0).stop 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 -- 172.21.15.28:0/1368114813 shutdown_connections 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e8160750 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 --2- 172.21.15.28:0/1368114813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 -- 172.21.15.28:0/1368114813 >> 172.21.15.28:0/1368114813 conn(0x7fe0e8069e80 msgr2=0x7fe0e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 -- 172.21.15.28:0/1368114813 shutdown_connections 2024-09-13T17:13:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.751+0000 7fe0f0400640 1 -- 172.21.15.28:0/1368114813 wait complete. 2024-09-13T17:13:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.753+0000 7fe0f0400640 1 Processor -- start 2024-09-13T17:13:42.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.753+0000 7fe0f0400640 1 -- start start 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e806c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e813e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e813da80 con 0x7fe0e80ac850 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e813dd60 con 0x7fe0e8160310 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0f0400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e813e040 con 0x7fe0e8146170 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0eda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0eda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43860/0 (socket says 172.21.15.28:43860) 2024-09-13T17:13:42.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0eda00640 1 -- 172.21.15.28:0/3657934405 learned_addr learned my addr 172.21.15.28:0/3657934405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0ee400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.754+0000 7fe0ed000640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e806c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 -- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 msgr2=0x7fe0e806c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e806c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 -- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 msgr2=0x7fe0e813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e813e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 -- 172.21.15.28:0/3657934405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d4009400 con 0x7fe0e80ac850 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0ee400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e813e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0eda00640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9b0 tx=0x7fe0dc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc009670 con 0x7fe0e80ac850 2024-09-13T17:13:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc00c9a0 con 0x7fe0e80ac850 2024-09-13T17:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc00cd50 con 0x7fe0e80ac850 2024-09-13T17:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e813e320 con 0x7fe0e80ac850 2024-09-13T17:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.755+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e813d420 con 0x7fe0e80ac850 2024-09-13T17:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.756+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e8066e70 con 0x7fe0e80ac850 2024-09-13T17:13:42.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.757+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe0dc013470 con 0x7fe0e80ac850 2024-09-13T17:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.758+0000 7fe0e5600640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 0x7fe0c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.758+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0dc089690 con 0x7fe0e80ac850 2024-09-13T17:13:42.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.759+0000 7fe0ed000640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 0x7fe0c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:42.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.759+0000 7fe0ed000640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 0x7fe0c806d0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4034000 tx=0x7fe0d402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:42.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.762+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe0dc0547d0 con 0x7fe0e80ac850 2024-09-13T17:13:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.943+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0e813dbc0 con 0x7fe0e80ac850 2024-09-13T17:13:42.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.944+0000 7fe0e5600640 1 -- 172.21.15.28:0/3657934405 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16573 (secure 0 0 0) 0x7fe0e813dbc0 con 0x7fe0e80ac850 2024-09-13T17:13:42.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:42.946 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:42.545191+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 msgr2=0x7fe0c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 0x7fe0c806d0c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4034000 tx=0x7fe0d402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 msgr2=0x7fe0e81307b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e9b0 tx=0x7fe0dc00ee70 comp rx=0 tx=0).stop 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 shutdown_connections 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe0c806ac00 0x7fe0c806d0c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0e8160310 0x7fe0e813e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0e8146170 0x7fe0e806c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 --2- 172.21.15.28:0/3657934405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0e80ac850 0x7fe0e81307b0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:42.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 >> 172.21.15.28:0/3657934405 conn(0x7fe0e8069e80 msgr2=0x7fe0e80ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:42.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 shutdown_connections 2024-09-13T17:13:42.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:42.947+0000 7fe0f0400640 1 -- 172.21.15.28:0/3657934405 wait complete. 2024-09-13T17:13:42.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:43.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.120+0000 7f0900e00640 1 Processor -- start 2024-09-13T17:13:43.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.120+0000 7f0900e00640 1 -- start start 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f0900e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc0b43c0 0x7f08fc0b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f0900e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08fc069e80 0x7f08fc06a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f0900e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc0d0d80 con 0x7f08fc15c750 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f0900e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc06a850 con 0x7f08fc069e80 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f0900e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0b4d40 con 0x7f08fc0b43c0 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f08fbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08fc069e80 0x7f08fc06a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f08fb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08fc15c750 0x7f08fc0d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58744/0 (socket says 172.21.15.28:58744) 2024-09-13T17:13:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f08fb400640 1 -- 172.21.15.28:0/1708489114 learned_addr learned my addr 172.21.15.28:0/1708489114 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.121+0000 7f08faa00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc0b43c0 0x7f08fc0b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.122+0000 7f08fbe00640 1 -- 172.21.15.28:0/1708489114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc0b43c0 msgr2=0x7f08fc0b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.122+0000 7f08fbe00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc0b43c0 0x7f08fc0b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.122+0000 7f08fbe00640 1 -- 172.21.15.28:0/1708489114 >> v1:172.21.15.28:6789/0 conn(0x7f08fc15c750 legacy=0x7f08fc0d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.122+0000 7f08fbe00640 1 -- 172.21.15.28:0/1708489114 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc0d0960 con 0x7f08fc069e80 2024-09-13T17:13:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.122+0000 7f08fa000640 1 -- 172.21.15.28:0/1708489114 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935901955 0 0) 0x7f08fc0d0d80 con 0x7f08fc15c750 2024-09-13T17:13:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.123+0000 7f08fbe00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc069e80 0x7f08fc06a310 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009d90 tx=0x7f08f402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b268c2632dca48b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.123+0000 7f08fa000640 1 -- 172.21.15.28:0/1708489114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4031600 con 0x7f08fc069e80 2024-09-13T17:13:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.123+0000 7f08fa000640 1 -- 172.21.15.28:0/1708489114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f402d050 con 0x7f08fc069e80 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.123+0000 7f08fa000640 1 -- 172.21.15.28:0/1708489114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f4031980 con 0x7f08fc069e80 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 -- 172.21.15.28:0/1708489114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc069e80 msgr2=0x7f08fc06a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc069e80 0x7f08fc06a310 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f08f4009d90 tx=0x7f08f402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 -- 172.21.15.28:0/1708489114 shutdown_connections 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc069e80 0x7f08fc06a310 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 --2- 172.21.15.28:0/1708489114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc0b43c0 0x7f08fc0b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 -- 172.21.15.28:0/1708489114 >> 172.21.15.28:0/1708489114 conn(0x7f08fc0cf5a0 msgr2=0x7f08fc0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 -- 172.21.15.28:0/1708489114 shutdown_connections 2024-09-13T17:13:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.124+0000 7f0900e00640 1 -- 172.21.15.28:0/1708489114 wait complete. 2024-09-13T17:13:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.125+0000 7f0900e00640 1 Processor -- start 2024-09-13T17:13:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 -- start start 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 0x7f08fc153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 0x7f08fc154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c7e60 con 0x7f08fc0b43c0 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1535b0 con 0x7f08fc15c750 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.126+0000 7f0900e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c71e0 con 0x7f08fc069e80 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43876/0 (socket says 172.21.15.28:43876) 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 -- 172.21.15.28:0/394838254 learned_addr learned my addr 172.21.15.28:0/394838254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08fbe00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 0x7f08fc154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08fb400640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 0x7f08fc153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 -- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 msgr2=0x7f08fc153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 0x7f08fc153070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 -- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 msgr2=0x7f08fc154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 0x7f08fc154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08faa00640 1 -- 172.21.15.28:0/394838254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4009a70 con 0x7f08fc0b43c0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08fbe00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 0x7f08fc154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.127+0000 7f08fb400640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 0x7f08fc153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f08faa00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f08e800e930 tx=0x7f08e800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e8009690 con 0x7f08fc0b43c0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08e800ca00 con 0x7f08fc0b43c0 2024-09-13T17:13:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08e800cdb0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08fc0c74c0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.128+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc0c7a80 con 0x7f08fc0b43c0 2024-09-13T17:13:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.129+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc0c7fa0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.130+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f08e8013470 con 0x7f08fc0b43c0 2024-09-13T17:13:43.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.131+0000 7f08f3400640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 0x7f08dc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.131+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08e80895e0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.131+0000 7f08fb400640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 0x7f08dc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.132+0000 7f08fb400640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 0x7f08dc06d0f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08ec000f60 tx=0x7f08ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.135+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f08fc0c7fa0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.316+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08fc0c7fa0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.317+0000 7f08f3400640 1 -- 172.21.15.28:0/394838254 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16573 (secure 0 0 0) 0x7f08fc0c7fa0 con 0x7f08fc0b43c0 2024-09-13T17:13:43.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:43.319 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":92,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:42.545191+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 msgr2=0x7f08dc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 0x7f08dc06d0f0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f08ec000f60 tx=0x7f08ec009040 comp rx=0 tx=0).stop 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 msgr2=0x7f08fc0cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f08e800e930 tx=0x7f08e800edf0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 shutdown_connections 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08dc06ac30 0x7f08dc06d0f0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08fc15c750 0x7f08fc154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08fc0b43c0 0x7f08fc0cbbd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 --2- 172.21.15.28:0/394838254 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08fc069e80 0x7f08fc153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 >> 172.21.15.28:0/394838254 conn(0x7f08fc0cf5a0 msgr2=0x7f08fc0d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 shutdown_connections 2024-09-13T17:13:43.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.320+0000 7f0900e00640 1 -- 172.21.15.28:0/394838254 wait complete. 2024-09-13T17:13:43.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:13:43.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.494+0000 7f6d96400640 1 Processor -- start 2024-09-13T17:13:43.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.494+0000 7f6d96400640 1 -- start start 2024-09-13T17:13:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.495+0000 7f6d96400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d9015a780 0x7f6d90070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.495+0000 7f6d96400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.495+0000 7f6d96400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9006c8b0 con 0x7f6d9006bb40 2024-09-13T17:13:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.495+0000 7f6d96400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9006cb90 con 0x7f6d9015a780 2024-09-13T17:13:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.495+0000 7f6d96400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900cc980 con 0x7f6d9015ab60 2024-09-13T17:13:43.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:39846/0 (socket says 172.21.15.28:39846) 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 -- 172.21.15.28:0/3090785328 learned_addr learned my addr 172.21.15.28:0/3090785328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8f400640 1 --2- 172.21.15.28:0/3090785328 >> v2:172.21.15.189:3300/0 conn(0x7f6d9015a780 0x7f6d90070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 -- 172.21.15.28:0/3090785328 >> v2:172.21.15.189:3300/0 conn(0x7f6d9015a780 msgr2=0x7f6d90070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 --2- 172.21.15.28:0/3090785328 >> v2:172.21.15.189:3300/0 conn(0x7f6d9015a780 0x7f6d90070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 -- 172.21.15.28:0/3090785328 >> v1:172.21.15.189:6789/0 conn(0x7f6d9006bb40 legacy=0x7f6d9006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8ea00640 1 -- 172.21.15.28:0/3090785328 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d900ccc60 con 0x7f6d9015ab60 2024-09-13T17:13:43.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.496+0000 7f6d8e000640 1 -- 172.21.15.28:0/3090785328 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458570271 0 0) 0x7f6d9006c8b0 con 0x7f6d9006bb40 2024-09-13T17:13:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.497+0000 7f6d8ea00640 1 --2- 172.21.15.28:0/3090785328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009870 tx=0x7f6d7c02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fcee7d8327340c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.497+0000 7f6d8e000640 1 -- 172.21.15.28:0/3090785328 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c032440 con 0x7f6d9015ab60 2024-09-13T17:13:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d8e000640 1 -- 172.21.15.28:0/3090785328 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c0325a0 con 0x7f6d9015ab60 2024-09-13T17:13:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d8e000640 1 -- 172.21.15.28:0/3090785328 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c032930 con 0x7f6d9015ab60 2024-09-13T17:13:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 -- 172.21.15.28:0/3090785328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 msgr2=0x7f6d900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 --2- 172.21.15.28:0/3090785328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009870 tx=0x7f6d7c02a250 comp rx=0 tx=0).stop 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 -- 172.21.15.28:0/3090785328 shutdown_connections 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 --2- 172.21.15.28:0/3090785328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9015ab60 0x7f6d900712c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 --2- 172.21.15.28:0/3090785328 >> v2:172.21.15.189:3300/0 conn(0x7f6d9015a780 0x7f6d90070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 -- 172.21.15.28:0/3090785328 >> 172.21.15.28:0/3090785328 conn(0x7f6d900b0270 msgr2=0x7f6d900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 -- 172.21.15.28:0/3090785328 shutdown_connections 2024-09-13T17:13:43.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.498+0000 7f6d96400640 1 -- 172.21.15.28:0/3090785328 wait complete. 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.500+0000 7f6d96400640 1 Processor -- start 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.500+0000 7f6d96400640 1 -- start start 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 0x7f6d900ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 0x7f6d900b1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900b2220 con 0x7f6d9015ab60 2024-09-13T17:13:43.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900b2410 con 0x7f6d9015a780 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d96400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d900b26f0 con 0x7f6d9006bb40 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43890/0 (socket says 172.21.15.28:43890) 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d8fe00640 1 -- 172.21.15.28:0/1238454186 learned_addr learned my addr 172.21.15.28:0/1238454186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.501+0000 7f6d8f400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 0x7f6d900ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8ea00640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 0x7f6d900b1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 -- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 msgr2=0x7f6d900ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 0x7f6d900ad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 -- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 msgr2=0x7f6d900b1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 0x7f6d900b1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:43.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 -- 172.21.15.28:0/1238454186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c0094d0 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8f400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 0x7f6d900ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8ea00640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 0x7f6d900b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d8fe00640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6d88001560 tx=0x7f6d8800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88003da0 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d88005070 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d88005420 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.502+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d900b29d0 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.503+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d900b2f90 con 0x7f6d9015ab60 2024-09-13T17:13:43.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.504+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9006c8b0 con 0x7f6d9015ab60 2024-09-13T17:13:43.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.505+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6d88005580 con 0x7f6d9015ab60 2024-09-13T17:13:43.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.506+0000 7f6d87400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 0x7f6d7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:43.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.506+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d88088cd0 con 0x7f6d9015ab60 2024-09-13T17:13:43.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.506+0000 7f6d8f400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 0x7f6d7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:43.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.507+0000 7f6d8f400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 0x7f6d7006d0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6d9006b5d0 tx=0x7f6d80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:43.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.510+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6d88053e10 con 0x7f6d9015ab60 2024-09-13T17:13:43.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:43.727+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6d9006b7c0 con 0x7f6d9015ab60 2024-09-13T17:13:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.562+0000 7f6d87400640 1 -- 172.21.15.28:0/1238454186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6d88017070 con 0x7f6d9015ab60 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 msgr2=0x7f6d7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 0x7f6d7006d0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6d9006b5d0 tx=0x7f6d80008040 comp rx=0 tx=0).stop 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 msgr2=0x7f6d90151dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f6d88001560 tx=0x7f6d8800cc70 comp rx=0 tx=0).stop 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 shutdown_connections 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d7006ac00 0x7f6d7006d0c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d9015ab60 0x7f6d90151dd0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d9015a780 0x7f6d900b1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 --2- 172.21.15.28:0/1238454186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d9006bb40 0x7f6d900ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 >> 172.21.15.28:0/1238454186 conn(0x7f6d900b0270 msgr2=0x7f6d900ce5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 shutdown_connections 2024-09-13T17:13:44.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.565+0000 7f6d96400640 1 -- 172.21.15.28:0/1238454186 wait complete. 2024-09-13T17:13:44.585 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:13:44.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:13:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.747+0000 7f1576000640 1 Processor -- start 2024-09-13T17:13:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.747+0000 7f1576000640 1 -- start start 2024-09-13T17:13:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f1576000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f157006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f1576000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1570071490 0x7f157006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f1576000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157006c8b0 con 0x7f157015a780 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f1576000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157006cb90 con 0x7f1570071490 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f1576000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15700cc980 con 0x7f157015ab60 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f156f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f157015a780 0x7f1570070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54830/0 (socket says 172.21.15.28:54830) 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f156f400640 1 -- 172.21.15.28:0/2296516309 learned_addr learned my addr 172.21.15.28:0/2296516309 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.748+0000 7f156fe00640 1 --2- 172.21.15.28:0/2296516309 >> v2:172.21.15.28:3300/0 conn(0x7f1570071490 0x7f157006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f1566a00640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f157006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f156fe00640 1 -- 172.21.15.28:0/2296516309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 msgr2=0x7f157006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f156fe00640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f157006bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f156fe00640 1 -- 172.21.15.28:0/2296516309 >> v1:172.21.15.28:6789/0 conn(0x7f157015a780 legacy=0x7f1570070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f156fe00640 1 -- 172.21.15.28:0/2296516309 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15700ccc60 con 0x7f1570071490 2024-09-13T17:13:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f156ea00640 1 -- 172.21.15.28:0/2296516309 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166651789 0 0) 0x7f157006c8b0 con 0x7f157015a780 2024-09-13T17:13:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.749+0000 7f1566a00640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f157006bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.750+0000 7f156fe00640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157006c080 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de1eb633ec35c29 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.750+0000 7f156ea00640 1 -- 172.21.15.28:0/2296516309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568007740 con 0x7f1570071490 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.750+0000 7f156ea00640 1 -- 172.21.15.28:0/2296516309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156802d050 con 0x7f1570071490 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.750+0000 7f156ea00640 1 -- 172.21.15.28:0/2296516309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568007ac0 con 0x7f1570071490 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 -- 172.21.15.28:0/2296516309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 msgr2=0x7f157006c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157006c080 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f1568009d90 tx=0x7f156802a660 comp rx=0 tx=0).stop 2024-09-13T17:13:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 -- 172.21.15.28:0/2296516309 shutdown_connections 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157006c080 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 --2- 172.21.15.28:0/2296516309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f157006bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 -- 172.21.15.28:0/2296516309 >> 172.21.15.28:0/2296516309 conn(0x7f15700b0270 msgr2=0x7f15700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 -- 172.21.15.28:0/2296516309 shutdown_connections 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.751+0000 7f1576000640 1 -- 172.21.15.28:0/2296516309 wait complete. 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.753+0000 7f1576000640 1 Processor -- start 2024-09-13T17:13:44.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.753+0000 7f1576000640 1 -- start start 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.753+0000 7f1576000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1576000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 0x7f1570153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1576000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f15700aae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1576000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15700a8700 con 0x7f1570071490 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1576000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570153960 con 0x7f157015a780 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1576000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1570153c00 con 0x7f157015ab60 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44054/0 (socket says 172.21.15.28:44054) 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 -- 172.21.15.28:0/1345211707 learned_addr learned my addr 172.21.15.28:0/1345211707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1566a00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 0x7f1570153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156fe00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f15700aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 -- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 msgr2=0x7f15700aae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f15700aae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 -- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 msgr2=0x7f1570153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 0x7f1570153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f156f400640 1 -- 172.21.15.28:0/1345211707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009a70 con 0x7f1570071490 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.754+0000 7f1566a00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 0x7f1570153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f156f400640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f156000c4d0 tx=0x7f156000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15600074e0 con 0x7f1570071490 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1560007660 con 0x7f1570071490 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560004a20 con 0x7f1570071490 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1570153ee0 con 0x7f1570071490 2024-09-13T17:13:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.755+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15701544a0 con 0x7f1570071490 2024-09-13T17:13:44.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.756+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157006c8b0 con 0x7f1570071490 2024-09-13T17:13:44.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.757+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1560004b80 con 0x7f1570071490 2024-09-13T17:13:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.758+0000 7f1567e00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 0x7f154c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.758+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15600885c0 con 0x7f1570071490 2024-09-13T17:13:44.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.758+0000 7f1566a00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 0x7f154c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:44.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.759+0000 7f1566a00640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 0x7f154c06d0f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f15700b0110 tx=0x7f1558008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:44.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.762+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1560053700 con 0x7f1570071490 2024-09-13T17:13:44.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:44.979+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f15700ad7e0 con 0x7f1570071490 2024-09-13T17:13:45.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.570+0000 7f1567e00640 1 -- 172.21.15.28:0/1345211707 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v78) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1560056d20 con 0x7f1570071490 2024-09-13T17:13:45.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 msgr2=0x7f154c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 0x7f154c06d0f0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f15700b0110 tx=0x7f1558008040 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 msgr2=0x7f157014afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f156000c4d0 tx=0x7f156000c990 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 shutdown_connections 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f154c06ac30 0x7f154c06d0f0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f157015ab60 0x7f15700aae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f157015a780 0x7f1570153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 --2- 172.21.15.28:0/1345211707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1570071490 0x7f157014afb0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.572+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 >> 172.21.15.28:0/1345211707 conn(0x7f15700b0270 msgr2=0x7f157006f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.573+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 shutdown_connections 2024-09-13T17:13:45.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.573+0000 7f1576000640 1 -- 172.21.15.28:0/1345211707 wait complete. 2024-09-13T17:13:45.593 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:13:45.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.751+0000 7fb071600640 1 Processor -- start 2024-09-13T17:13:45.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.751+0000 7fb071600640 1 -- start start 2024-09-13T17:13:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.752+0000 7fb071600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 0x7fb06c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.752+0000 7fb071600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.752+0000 7fb071600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06c15e100 con 0x7fb06c0cabc0 2024-09-13T17:13:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.752+0000 7fb071600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06c0adb00 con 0x7fb06c15a540 2024-09-13T17:13:45.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.752+0000 7fb071600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c071670 con 0x7fb06c070cc0 2024-09-13T17:13:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 0x7fb06c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06a000640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fb06c0cabc0 0x7fb06c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:33086/0 (socket says 172.21.15.28:33086) 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06a000640 1 -- 172.21.15.28:0/4249532594 learned_addr learned my addr 172.21.15.28:0/4249532594 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:51256/0 (socket says 172.21.15.28:51256) 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 -- 172.21.15.28:0/4249532594 >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 msgr2=0x7fb06c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 --2- 172.21.15.28:0/4249532594 >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 0x7fb06c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 -- 172.21.15.28:0/4249532594 >> v1:172.21.15.189:6789/0 conn(0x7fb06c0cabc0 legacy=0x7fb06c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06b400640 1 -- 172.21.15.28:0/4249532594 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c15e3e0 con 0x7fb06c070cc0 2024-09-13T17:13:45.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.753+0000 7fb06aa00640 1 --2- 172.21.15.28:0/4249532594 >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 0x7fb06c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.754+0000 7fb06b400640 1 --2- 172.21.15.28:0/4249532594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb064009d90 tx=0x7fb06402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90e4c5b8f385415 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:45.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb069600640 1 -- 172.21.15.28:0/4249532594 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064031600 con 0x7fb06c070cc0 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb069600640 1 -- 172.21.15.28:0/4249532594 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06402d050 con 0x7fb06c070cc0 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb069600640 1 -- 172.21.15.28:0/4249532594 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb064031980 con 0x7fb06c070cc0 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 -- 172.21.15.28:0/4249532594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 msgr2=0x7fb06c071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 --2- 172.21.15.28:0/4249532594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb064009d90 tx=0x7fb06402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 -- 172.21.15.28:0/4249532594 shutdown_connections 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 --2- 172.21.15.28:0/4249532594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c071130 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 --2- 172.21.15.28:0/4249532594 >> v2:172.21.15.189:3300/0 conn(0x7fb06c15a540 0x7fb06c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 -- 172.21.15.28:0/4249532594 >> 172.21.15.28:0/4249532594 conn(0x7fb06c069e80 msgr2=0x7fb06c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 -- 172.21.15.28:0/4249532594 shutdown_connections 2024-09-13T17:13:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.755+0000 7fb071600640 1 -- 172.21.15.28:0/4249532594 wait complete. 2024-09-13T17:13:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.757+0000 7fb071600640 1 Processor -- start 2024-09-13T17:13:45.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.757+0000 7fb071600640 1 -- start start 2024-09-13T17:13:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c13aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 0x7fb06c15cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c13bf50 con 0x7fb06c0cabc0 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0b0d00 con 0x7fb06c15a540 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb071600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06c0b0a20 con 0x7fb06c070cc0 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb06a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb06a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44060/0 (socket says 172.21.15.28:44060) 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.758+0000 7fb06a000640 1 -- 172.21.15.28:0/3389648480 learned_addr learned my addr 172.21.15.28:0/3389648480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06aa00640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c13aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 -- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 msgr2=0x7fb06c13aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06b400640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 0x7fb06c15cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c13aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 -- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 msgr2=0x7fb06c15cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 0x7fb06c15cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 -- 172.21.15.28:0/3389648480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c009400 con 0x7fb06c0cabc0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06aa00640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c13aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06b400640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 0x7fb06c15cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb06a000640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb05800c4a0 tx=0x7fb05800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058005ce0 con 0x7fb06c0cabc0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb058005e40 con 0x7fb06c0cabc0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058007270 con 0x7fb06c0cabc0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.759+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb064009a70 con 0x7fb06c0cabc0 2024-09-13T17:13:45.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.760+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06c13bbe0 con 0x7fb06c0cabc0 2024-09-13T17:13:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.760+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06c13c100 con 0x7fb06c0cabc0 2024-09-13T17:13:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.761+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb058007490 con 0x7fb06c0cabc0 2024-09-13T17:13:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.762+0000 7fb062a00640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 0x7fb04c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.762+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb0580887b0 con 0x7fb06c0cabc0 2024-09-13T17:13:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.763+0000 7fb06aa00640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 0x7fb04c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.763+0000 7fb06aa00640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 0x7fb04c06d0c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009bb0 tx=0x7fb05c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.766+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb06c13c100 con 0x7fb06c0cabc0 2024-09-13T17:13:45.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.984+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb06c13bd70 con 0x7fb06c0cabc0 2024-09-13T17:13:45.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.985+0000 7fb062a00640 1 -- 172.21.15.28:0/3389648480 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fb058017090 con 0x7fb06c0cabc0 2024-09-13T17:13:45.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:45.986 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3444244429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3444244429},{"type":"v1","addr":"172.21.15.189:6837","nonce":3444244429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6380,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3281700882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3281700882},{"type":"v1","addr":"172.21.15.28:6835","nonce":3281700882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":7075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3898110859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3898110859},{"type":"v1","addr":"172.21.15.189:6836","nonce":3898110859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:13:45.560696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7491":{"gid":7491,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2861751170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2861751170},{"type":"v1","addr":"172.21.15.28:6834","nonce":2861751170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:13:45.986 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 78 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 msgr2=0x7fb04c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 0x7fb04c06d0c0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb05c009bb0 tx=0x7fb05c008040 comp rx=0 tx=0).stop 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 msgr2=0x7fb06c0b00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb05800c4a0 tx=0x7fb05800c960 comp rx=0 tx=0).stop 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 shutdown_connections 2024-09-13T17:13:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04c06ac00 0x7fb04c06d0c0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06c15a540 0x7fb06c15cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06c0cabc0 0x7fb06c0b00f0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 --2- 172.21.15.28:0/3389648480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06c070cc0 0x7fb06c13aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 >> 172.21.15.28:0/3389648480 conn(0x7fb06c069e80 msgr2=0x7fb06c15c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 shutdown_connections 2024-09-13T17:13:45.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:45.987+0000 7fb071600640 1 -- 172.21.15.28:0/3389648480 wait complete. 2024-09-13T17:13:46.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.157+0000 7f33ef200640 1 Processor -- start 2024-09-13T17:13:46.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.158+0000 7f33ef200640 1 -- start start 2024-09-13T17:13:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ef200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33f0154a90 0x7f33f015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ef200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f015bb10 con 0x7f33f006b360 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ef200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f006b900 con 0x7f33f0154a90 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f006bbe0 con 0x7f33f00b0b50 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33f0154a90 0x7f33f015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ede00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33f0154a90 0x7f33f015b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44064/0 (socket says 172.21.15.28:44064) 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ede00640 1 -- 172.21.15.28:0/4211679777 learned_addr learned my addr 172.21.15.28:0/4211679777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.159+0000 7f33ede00640 1 -- 172.21.15.28:0/4211679777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 msgr2=0x7f33f00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.160+0000 7f33ede00640 1 --2- 172.21.15.28:0/4211679777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.160+0000 7f33ede00640 1 -- 172.21.15.28:0/4211679777 >> v1:172.21.15.189:6789/0 conn(0x7f33f006b360 legacy=0x7f33f00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:46.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.160+0000 7f33ede00640 1 -- 172.21.15.28:0/4211679777 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f006bec0 con 0x7f33f0154a90 2024-09-13T17:13:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.160+0000 7f33ede00640 1 --2- 172.21.15.28:0/4211679777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f0154a90 0x7f33f015b5d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f33f00b3d70 tx=0x7f33e002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b4daeb12505081 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ed400640 1 -- 172.21.15.28:0/4211679777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0032440 con 0x7f33f0154a90 2024-09-13T17:13:46.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ed400640 1 -- 172.21.15.28:0/4211679777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e00325a0 con 0x7f33f0154a90 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ed400640 1 -- 172.21.15.28:0/4211679777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0032930 con 0x7f33f0154a90 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 -- 172.21.15.28:0/4211679777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f0154a90 msgr2=0x7f33f015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 --2- 172.21.15.28:0/4211679777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f0154a90 0x7f33f015b5d0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f33f00b3d70 tx=0x7f33e002a200 comp rx=0 tx=0).stop 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 -- 172.21.15.28:0/4211679777 shutdown_connections 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 --2- 172.21.15.28:0/4211679777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f00af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 --2- 172.21.15.28:0/4211679777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f0154a90 0x7f33f015b5d0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.161+0000 7f33ef200640 1 -- 172.21.15.28:0/4211679777 >> 172.21.15.28:0/4211679777 conn(0x7f33f0072e40 msgr2=0x7f33f0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.162+0000 7f33ef200640 1 -- 172.21.15.28:0/4211679777 shutdown_connections 2024-09-13T17:13:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.162+0000 7f33ef200640 1 -- 172.21.15.28:0/4211679777 wait complete. 2024-09-13T17:13:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.163+0000 7f33ef200640 1 Processor -- start 2024-09-13T17:13:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.163+0000 7f33ef200640 1 -- start start 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f0153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33f0154a90 0x7f33f013aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f00aeaa0 con 0x7f33f006b360 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f01e33a0 con 0x7f33f0154a90 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f01e3680 con 0x7f33f00b0b50 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44066/0 (socket says 172.21.15.28:44066) 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.164+0000 7f33ede00640 1 -- 172.21.15.28:0/566471985 learned_addr learned my addr 172.21.15.28:0/566471985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ee800640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33f0154a90 0x7f33f013aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33e5400640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f0153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 -- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 msgr2=0x7f33f0153c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f0153c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 -- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33f0154a90 msgr2=0x7f33f013aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33f0154a90 0x7f33f013aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 -- 172.21.15.28:0/566471985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e00094d0 con 0x7f33f006b360 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ede00640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f33e0000c00 tx=0x7f33e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e0032440 con 0x7f33f006b360 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e0030ba0 con 0x7f33f006b360 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e003b400 con 0x7f33f006b360 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.165+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f01e3960 con 0x7f33f006b360 2024-09-13T17:13:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.166+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f01e3f20 con 0x7f33f006b360 2024-09-13T17:13:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.167+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f00aec10 con 0x7f33f006b360 2024-09-13T17:13:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.168+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f33e0039040 con 0x7f33f006b360 2024-09-13T17:13:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.169+0000 7f33e6a00640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 0x7f33cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.169+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33e00a9ce0 con 0x7f33f006b360 2024-09-13T17:13:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.169+0000 7f33e5400640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 0x7f33cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.170+0000 7f33e5400640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 0x7f33cc06d0c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f33d8007fd0 tx=0x7f33d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.173+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f33e0074e30 con 0x7f33f006b360 2024-09-13T17:13:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.351+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33f00aebe0 con 0x7f33f006b360 2024-09-13T17:13:46.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.352+0000 7f33e6a00640 1 -- 172.21.15.28:0/566471985 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16560 (secure 0 0 0) 0x7f33e0037020 con 0x7f33f006b360 2024-09-13T17:13:46.354 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:46.354 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":93,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:43.548330+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:46.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.354+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 msgr2=0x7f33cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.354+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 0x7f33cc06d0c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f33d8007fd0 tx=0x7f33d8009040 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.354+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 msgr2=0x7f33f00ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.354+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f33e0000c00 tx=0x7f33e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 shutdown_connections 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33cc06ac00 0x7f33cc06d0c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33f0154a90 0x7f33f013aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33f00b0b50 0x7f33f0153c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 --2- 172.21.15.28:0/566471985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33f006b360 0x7f33f00ae560 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 >> 172.21.15.28:0/566471985 conn(0x7f33f0072e40 msgr2=0x7f33f0159fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 shutdown_connections 2024-09-13T17:13:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.355+0000 7f33ef200640 1 -- 172.21.15.28:0/566471985 wait complete. 2024-09-13T17:13:46.371 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:13:46.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:13:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.526+0000 7f4267e00640 1 Processor -- start 2024-09-13T17:13:46.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.527+0000 7f4267e00640 1 -- start start 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 0x7f42680b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42680b29f0 0x7f42680b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680c3850 con 0x7f4268159820 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426815a930 con 0x7f42680b29f0 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4268159d20 con 0x7f4268159440 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f425fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4268159820 0x7f42680b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54832/0 (socket says 172.21.15.28:54832) 2024-09-13T17:13:46.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f425fe00640 1 -- 172.21.15.28:0/3523411854 learned_addr learned my addr 172.21.15.28:0/3523411854 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4266a00640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 0x7f42680b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.528+0000 7f4267400640 1 --2- 172.21.15.28:0/3523411854 >> v2:172.21.15.189:3300/0 conn(0x7f42680b29f0 0x7f42680b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489492719 0 0) 0x7f42680c3850 con 0x7f4268159820 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f424c003520 con 0x7f4268159820 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4267400640 1 -- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 msgr2=0x7f42680b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:46.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4267400640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 0x7f42680b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4267400640 1 -- 172.21.15.28:0/3523411854 >> v1:172.21.15.28:6789/0 conn(0x7f4268159820 legacy=0x7f42680b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3536176119 0 0) 0x7f424c003520 con 0x7f4268159820 2024-09-13T17:13:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4267400640 1 -- 172.21.15.28:0/3523411854 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426815a000 con 0x7f42680b29f0 2024-09-13T17:13:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.529+0000 7f4266a00640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 0x7f42680b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.530+0000 7f4267400640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42680b29f0 0x7f42680b0980 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4260009d90 tx=0x7f426002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d8f60b80a25e48 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.530+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250001670 con 0x7f42680b29f0 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.530+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426002d050 con 0x7f42680b29f0 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.530+0000 7f4266000640 1 -- 172.21.15.28:0/3523411854 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260007740 con 0x7f42680b29f0 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 -- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42680b29f0 msgr2=0x7f42680b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42680b29f0 0x7f42680b0980 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f4260009d90 tx=0x7f426002a680 comp rx=0 tx=0).stop 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 -- 172.21.15.28:0/3523411854 shutdown_connections 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42680b29f0 0x7f42680b0980 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 --2- 172.21.15.28:0/3523411854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4268159440 0x7f42680b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 -- 172.21.15.28:0/3523411854 >> 172.21.15.28:0/3523411854 conn(0x7f4268072e40 msgr2=0x7f4268073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 -- 172.21.15.28:0/3523411854 shutdown_connections 2024-09-13T17:13:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.531+0000 7f4267e00640 1 -- 172.21.15.28:0/3523411854 wait complete. 2024-09-13T17:13:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.533+0000 7f4267e00640 1 Processor -- start 2024-09-13T17:13:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.533+0000 7f4267e00640 1 -- start start 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42680b29f0 0x7f42680cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 0x7f426814ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680af0d0 con 0x7f4268159440 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42680cd610 con 0x7f4268159820 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42680af560 con 0x7f42680b29f0 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44072/0 (socket says 172.21.15.28:44072) 2024-09-13T17:13:46.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 -- 172.21.15.28:0/4081622115 learned_addr learned my addr 172.21.15.28:0/4081622115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4266a00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42680b29f0 0x7f42680cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f4267400640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 0x7f426814ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 -- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42680b29f0 msgr2=0x7f42680cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42680b29f0 0x7f42680cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 -- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 msgr2=0x7f426814ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 0x7f426814ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.534+0000 7f425fe00640 1 -- 172.21.15.28:0/4081622115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258009400 con 0x7f4268159440 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f4267400640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 0x7f426814ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f425fe00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f425000c750 tx=0x7f425000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250005ad0 con 0x7f4268159440 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4250005c30 con 0x7f4268159440 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250007070 con 0x7f4268159440 2024-09-13T17:13:46.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260009a70 con 0x7f4268159440 2024-09-13T17:13:46.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.535+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426814f420 con 0x7f4268159440 2024-09-13T17:13:46.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.536+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42680af240 con 0x7f4268159440 2024-09-13T17:13:46.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.537+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f42500071d0 con 0x7f4268159440 2024-09-13T17:13:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.538+0000 7f425ea00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.538+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4250088bb0 con 0x7f4268159440 2024-09-13T17:13:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.539+0000 7f4266a00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.539+0000 7f4266a00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 0x7f424406d0c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4258009c60 tx=0x7f4258008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.542+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4250053cf0 con 0x7f4268159440 2024-09-13T17:13:46.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.720+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f426815a5c0 con 0x7f424406ac00 2024-09-13T17:13:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.729+0000 7f425ea00640 1 -- 172.21.15.28:0/4081622115 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289460 (secure 0 0 0) 0x7f42580098b0 con 0x7f424406ac00 2024-09-13T17:13:46.734 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:46.735 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:13:46.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 msgr2=0x7f424406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 0x7f424406d0c0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4258009c60 tx=0x7f4258008040 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 msgr2=0x7f426814e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f425000c750 tx=0x7f425000cc10 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 shutdown_connections 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4268159820 0x7f426814ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4268159440 0x7f426814e8f0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 --2- 172.21.15.28:0/4081622115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42680b29f0 0x7f42680cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 >> 172.21.15.28:0/4081622115 conn(0x7f4268072e40 msgr2=0x7f42681531f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 shutdown_connections 2024-09-13T17:13:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.734+0000 7f4267e00640 1 -- 172.21.15.28:0/4081622115 wait complete. 2024-09-13T17:13:46.750 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-13T17:13:44.968937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":367872,"kb_used_data":9868,"kb_used_omap":12,"kb_used_meta":357875,"kb_avail":754606848,"statfs":{"total":773094113280,"available":772717412352,"internally_reserved":0,"allocated":10104832,"data_stored":5009726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366464569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001171"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.579045+0000","last_change":"2024-09-13T17:13:42.568583+0000","last_active":"2024-09-13T17:13:42.579045+0000","last_peered":"2024-09-13T17:13:42.579045+0000","last_clean":"2024-09-13T17:13:42.579045+0000","last_became_active":"2024-09-13T17:13:42.568311+0000","last_became_peered":"2024-09-13T17:13:42.568311+0000","last_unstale":"2024-09-13T17:13:42.579045+0000","last_undegraded":"2024-09-13T17:13:42.579045+0000","last_fullsized":"2024-09-13T17:13:42.579045+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:40.379101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587618+0000","last_change":"2024-09-13T17:13:42.586645+0000","last_active":"2024-09-13T17:13:42.587618+0000","last_peered":"2024-09-13T17:13:42.587618+0000","last_clean":"2024-09-13T17:13:42.587618+0000","last_became_active":"2024-09-13T17:13:42.586364+0000","last_became_peered":"2024-09-13T17:13:42.586364+0000","last_unstale":"2024-09-13T17:13:42.587618+0000","last_undegraded":"2024-09-13T17:13:42.587618+0000","last_fullsized":"2024-09-13T17:13:42.587618+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:26.995166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.564003+0000","last_change":"2024-09-13T17:13:42.561894+0000","last_active":"2024-09-13T17:13:42.564003+0000","last_peered":"2024-09-13T17:13:42.564003+0000","last_clean":"2024-09-13T17:13:42.564003+0000","last_became_active":"2024-09-13T17:13:42.561644+0000","last_became_peered":"2024-09-13T17:13:42.561644+0000","last_unstale":"2024-09-13T17:13:42.564003+0000","last_undegraded":"2024-09-13T17:13:42.564003+0000","last_fullsized":"2024-09-13T17:13:42.564003+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586962+0000","last_change":"2024-09-13T17:13:42.582338+0000","last_active":"2024-09-13T17:13:42.586962+0000","last_peered":"2024-09-13T17:13:42.586962+0000","last_clean":"2024-09-13T17:13:42.586962+0000","last_became_active":"2024-09-13T17:13:42.579692+0000","last_became_peered":"2024-09-13T17:13:42.579692+0000","last_unstale":"2024-09-13T17:13:42.586962+0000","last_undegraded":"2024-09-13T17:13:42.586962+0000","last_fullsized":"2024-09-13T17:13:42.586962+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:43.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578457+0000","last_change":"2024-09-13T17:13:42.577390+0000","last_active":"2024-09-13T17:13:42.578457+0000","last_peered":"2024-09-13T17:13:42.578457+0000","last_clean":"2024-09-13T17:13:42.578457+0000","last_became_active":"2024-09-13T17:13:42.577043+0000","last_became_peered":"2024-09-13T17:13:42.577043+0000","last_unstale":"2024-09-13T17:13:42.578457+0000","last_undegraded":"2024-09-13T17:13:42.578457+0000","last_fullsized":"2024-09-13T17:13:42.578457+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:43.292409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568419+0000","last_change":"2024-09-13T17:13:42.563964+0000","last_active":"2024-09-13T17:13:42.568419+0000","last_peered":"2024-09-13T17:13:42.568419+0000","last_clean":"2024-09-13T17:13:42.568419+0000","last_became_active":"2024-09-13T17:13:42.563105+0000","last_became_peered":"2024-09-13T17:13:42.563105+0000","last_unstale":"2024-09-13T17:13:42.568419+0000","last_undegraded":"2024-09-13T17:13:42.568419+0000","last_fullsized":"2024-09-13T17:13:42.568419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586810+0000","last_change":"2024-09-13T17:13:42.582234+0000","last_active":"2024-09-13T17:13:42.586810+0000","last_peered":"2024-09-13T17:13:42.586810+0000","last_clean":"2024-09-13T17:13:42.586810+0000","last_became_active":"2024-09-13T17:13:42.581589+0000","last_became_peered":"2024-09-13T17:13:42.581589+0000","last_unstale":"2024-09-13T17:13:42.586810+0000","last_undegraded":"2024-09-13T17:13:42.586810+0000","last_fullsized":"2024-09-13T17:13:42.586810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:56.122473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.570255+0000","last_change":"2024-09-13T17:13:42.568298+0000","last_active":"2024-09-13T17:13:42.570255+0000","last_peered":"2024-09-13T17:13:42.570255+0000","last_clean":"2024-09-13T17:13:42.570255+0000","last_became_active":"2024-09-13T17:13:42.568019+0000","last_became_peered":"2024-09-13T17:13:42.568019+0000","last_unstale":"2024-09-13T17:13:42.570255+0000","last_undegraded":"2024-09-13T17:13:42.570255+0000","last_fullsized":"2024-09-13T17:13:42.570255+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.810829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568053+0000","last_change":"2024-09-13T17:13:42.564787+0000","last_active":"2024-09-13T17:13:42.568053+0000","last_peered":"2024-09-13T17:13:42.568053+0000","last_clean":"2024-09-13T17:13:42.568053+0000","last_became_active":"2024-09-13T17:13:42.564481+0000","last_became_peered":"2024-09-13T17:13:42.564481+0000","last_unstale":"2024-09-13T17:13:42.568053+0000","last_undegraded":"2024-09-13T17:13:42.568053+0000","last_fullsized":"2024-09-13T17:13:42.568053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:24.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577923+0000","last_change":"2024-09-13T17:13:42.570151+0000","last_active":"2024-09-13T17:13:42.577923+0000","last_peered":"2024-09-13T17:13:42.577923+0000","last_clean":"2024-09-13T17:13:42.577923+0000","last_became_active":"2024-09-13T17:13:42.569875+0000","last_became_peered":"2024-09-13T17:13:42.569875+0000","last_unstale":"2024-09-13T17:13:42.577923+0000","last_undegraded":"2024-09-13T17:13:42.577923+0000","last_fullsized":"2024-09-13T17:13:42.577923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584361+0000","last_change":"2024-09-13T17:13:42.582009+0000","last_active":"2024-09-13T17:13:42.584361+0000","last_peered":"2024-09-13T17:13:42.584361+0000","last_clean":"2024-09-13T17:13:42.584361+0000","last_became_active":"2024-09-13T17:13:42.581777+0000","last_became_peered":"2024-09-13T17:13:42.581777+0000","last_unstale":"2024-09-13T17:13:42.584361+0000","last_undegraded":"2024-09-13T17:13:42.584361+0000","last_fullsized":"2024-09-13T17:13:42.584361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:49.192916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.570821+0000","last_change":"2024-09-13T17:13:42.565044+0000","last_active":"2024-09-13T17:13:42.570821+0000","last_peered":"2024-09-13T17:13:42.570821+0000","last_clean":"2024-09-13T17:13:42.570821+0000","last_became_active":"2024-09-13T17:13:42.564729+0000","last_became_peered":"2024-09-13T17:13:42.564729+0000","last_unstale":"2024-09-13T17:13:42.570821+0000","last_undegraded":"2024-09-13T17:13:42.570821+0000","last_fullsized":"2024-09-13T17:13:42.570821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572843+0000","last_change":"2024-09-13T17:13:42.566886+0000","last_active":"2024-09-13T17:13:42.572843+0000","last_peered":"2024-09-13T17:13:42.572843+0000","last_clean":"2024-09-13T17:13:42.572843+0000","last_became_active":"2024-09-13T17:13:42.566621+0000","last_became_peered":"2024-09-13T17:13:42.566621+0000","last_unstale":"2024-09-13T17:13:42.572843+0000","last_undegraded":"2024-09-13T17:13:42.572843+0000","last_fullsized":"2024-09-13T17:13:42.572843+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:16.283948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586878+0000","last_change":"2024-09-13T17:13:42.581392+0000","last_active":"2024-09-13T17:13:42.586878+0000","last_peered":"2024-09-13T17:13:42.586878+0000","last_clean":"2024-09-13T17:13:42.586878+0000","last_became_active":"2024-09-13T17:13:42.581128+0000","last_became_peered":"2024-09-13T17:13:42.581128+0000","last_unstale":"2024-09-13T17:13:42.586878+0000","last_undegraded":"2024-09-13T17:13:42.586878+0000","last_fullsized":"2024-09-13T17:13:42.586878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:15.881460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.565104+0000","last_change":"2024-09-13T17:13:42.561267+0000","last_active":"2024-09-13T17:13:42.565104+0000","last_peered":"2024-09-13T17:13:42.565104+0000","last_clean":"2024-09-13T17:13:42.565104+0000","last_became_active":"2024-09-13T17:13:42.560983+0000","last_became_peered":"2024-09-13T17:13:42.560983+0000","last_unstale":"2024-09-13T17:13:42.565104+0000","last_undegraded":"2024-09-13T17:13:42.565104+0000","last_fullsized":"2024-09-13T17:13:42.565104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:14.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575831+0000","last_change":"2024-09-13T17:13:42.574786+0000","last_active":"2024-09-13T17:13:42.575831+0000","last_peered":"2024-09-13T17:13:42.575831+0000","last_clean":"2024-09-13T17:13:42.575831+0000","last_became_active":"2024-09-13T17:13:42.573757+0000","last_became_peered":"2024-09-13T17:13:42.573757+0000","last_unstale":"2024-09-13T17:13:42.575831+0000","last_undegraded":"2024-09-13T17:13:42.575831+0000","last_fullsized":"2024-09-13T17:13:42.575831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:30.869322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580964+0000","last_change":"2024-09-13T17:13:42.579770+0000","last_active":"2024-09-13T17:13:42.580964+0000","last_peered":"2024-09-13T17:13:42.580964+0000","last_clean":"2024-09-13T17:13:42.580964+0000","last_became_active":"2024-09-13T17:13:42.579496+0000","last_became_peered":"2024-09-13T17:13:42.579496+0000","last_unstale":"2024-09-13T17:13:42.580964+0000","last_undegraded":"2024-09-13T17:13:42.580964+0000","last_fullsized":"2024-09-13T17:13:42.580964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:18.169178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577079+0000","last_change":"2024-09-13T17:13:42.573857+0000","last_active":"2024-09-13T17:13:42.577079+0000","last_peered":"2024-09-13T17:13:42.577079+0000","last_clean":"2024-09-13T17:13:42.577079+0000","last_became_active":"2024-09-13T17:13:42.573622+0000","last_became_peered":"2024-09-13T17:13:42.573622+0000","last_unstale":"2024-09-13T17:13:42.577079+0000","last_undegraded":"2024-09-13T17:13:42.577079+0000","last_fullsized":"2024-09-13T17:13:42.577079+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:24.525557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.562666+0000","last_change":"2024-09-13T17:13:42.559351+0000","last_active":"2024-09-13T17:13:42.562666+0000","last_peered":"2024-09-13T17:13:42.562666+0000","last_clean":"2024-09-13T17:13:42.562666+0000","last_became_active":"2024-09-13T17:13:42.559100+0000","last_became_peered":"2024-09-13T17:13:42.559100+0000","last_unstale":"2024-09-13T17:13:42.562666+0000","last_undegraded":"2024-09-13T17:13:42.562666+0000","last_fullsized":"2024-09-13T17:13:42.562666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:15.878385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578013+0000","last_change":"2024-09-13T17:13:42.571537+0000","last_active":"2024-09-13T17:13:42.578013+0000","last_peered":"2024-09-13T17:13:42.578013+0000","last_clean":"2024-09-13T17:13:42.578013+0000","last_became_active":"2024-09-13T17:13:42.571241+0000","last_became_peered":"2024-09-13T17:13:42.571241+0000","last_unstale":"2024-09-13T17:13:42.578013+0000","last_undegraded":"2024-09-13T17:13:42.578013+0000","last_fullsized":"2024-09-13T17:13:42.578013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:46:40.654864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573393+0000","last_change":"2024-09-13T17:13:42.563308+0000","last_active":"2024-09-13T17:13:42.573393+0000","last_peered":"2024-09-13T17:13:42.573393+0000","last_clean":"2024-09-13T17:13:42.573393+0000","last_became_active":"2024-09-13T17:13:42.563040+0000","last_became_peered":"2024-09-13T17:13:42.563040+0000","last_unstale":"2024-09-13T17:13:42.573393+0000","last_undegraded":"2024-09-13T17:13:42.573393+0000","last_fullsized":"2024-09-13T17:13:42.573393+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:58.649961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.574798+0000","last_change":"2024-09-13T17:13:42.569096+0000","last_active":"2024-09-13T17:13:42.574798+0000","last_peered":"2024-09-13T17:13:42.574798+0000","last_clean":"2024-09-13T17:13:42.574798+0000","last_became_active":"2024-09-13T17:13:42.568809+0000","last_became_peered":"2024-09-13T17:13:42.568809+0000","last_unstale":"2024-09-13T17:13:42.574798+0000","last_undegraded":"2024-09-13T17:13:42.574798+0000","last_fullsized":"2024-09-13T17:13:42.574798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:18.197898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577313+0000","last_change":"2024-09-13T17:13:42.572008+0000","last_active":"2024-09-13T17:13:42.577313+0000","last_peered":"2024-09-13T17:13:42.577313+0000","last_clean":"2024-09-13T17:13:42.577313+0000","last_became_active":"2024-09-13T17:13:42.571758+0000","last_became_peered":"2024-09-13T17:13:42.571758+0000","last_unstale":"2024-09-13T17:13:42.577313+0000","last_undegraded":"2024-09-13T17:13:42.577313+0000","last_fullsized":"2024-09-13T17:13:42.577313+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585569+0000","last_change":"2024-09-13T17:13:42.571312+0000","last_active":"2024-09-13T17:13:42.585569+0000","last_peered":"2024-09-13T17:13:42.585569+0000","last_clean":"2024-09-13T17:13:42.585569+0000","last_became_active":"2024-09-13T17:13:42.565718+0000","last_became_peered":"2024-09-13T17:13:42.565718+0000","last_unstale":"2024-09-13T17:13:42.585569+0000","last_undegraded":"2024-09-13T17:13:42.585569+0000","last_fullsized":"2024-09-13T17:13:42.585569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:51.271074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587489+0000","last_change":"2024-09-13T17:13:42.586526+0000","last_active":"2024-09-13T17:13:42.587489+0000","last_peered":"2024-09-13T17:13:42.587489+0000","last_clean":"2024-09-13T17:13:42.587489+0000","last_became_active":"2024-09-13T17:13:42.586262+0000","last_became_peered":"2024-09-13T17:13:42.586262+0000","last_unstale":"2024-09-13T17:13:42.587489+0000","last_undegraded":"2024-09-13T17:13:42.587489+0000","last_fullsized":"2024-09-13T17:13:42.587489+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.553985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580227+0000","last_change":"2024-09-13T17:13:42.578735+0000","last_active":"2024-09-13T17:13:42.580227+0000","last_peered":"2024-09-13T17:13:42.580227+0000","last_clean":"2024-09-13T17:13:42.580227+0000","last_became_active":"2024-09-13T17:13:42.578424+0000","last_became_peered":"2024-09-13T17:13:42.578424+0000","last_unstale":"2024-09-13T17:13:42.580227+0000","last_undegraded":"2024-09-13T17:13:42.580227+0000","last_fullsized":"2024-09-13T17:13:42.580227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.571086+0000","last_change":"2024-09-13T17:13:42.567591+0000","last_active":"2024-09-13T17:13:42.571086+0000","last_peered":"2024-09-13T17:13:42.571086+0000","last_clean":"2024-09-13T17:13:42.571086+0000","last_became_active":"2024-09-13T17:13:42.564990+0000","last_became_peered":"2024-09-13T17:13:42.564990+0000","last_unstale":"2024-09-13T17:13:42.571086+0000","last_undegraded":"2024-09-13T17:13:42.571086+0000","last_fullsized":"2024-09-13T17:13:42.571086+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:54.017990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576836+0000","last_change":"2024-09-13T17:13:42.570056+0000","last_active":"2024-09-13T17:13:42.576836+0000","last_peered":"2024-09-13T17:13:42.576836+0000","last_clean":"2024-09-13T17:13:42.576836+0000","last_became_active":"2024-09-13T17:13:42.569789+0000","last_became_peered":"2024-09-13T17:13:42.569789+0000","last_unstale":"2024-09-13T17:13:42.576836+0000","last_undegraded":"2024-09-13T17:13:42.576836+0000","last_fullsized":"2024-09-13T17:13:42.576836+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:20.512031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584516+0000","last_change":"2024-09-13T17:13:42.582648+0000","last_active":"2024-09-13T17:13:42.584516+0000","last_peered":"2024-09-13T17:13:42.584516+0000","last_clean":"2024-09-13T17:13:42.584516+0000","last_became_active":"2024-09-13T17:13:42.582367+0000","last_became_peered":"2024-09-13T17:13:42.582367+0000","last_unstale":"2024-09-13T17:13:42.584516+0000","last_undegraded":"2024-09-13T17:13:42.584516+0000","last_fullsized":"2024-09-13T17:13:42.584516+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:56.122473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.588006+0000","last_change":"2024-09-13T17:13:42.586867+0000","last_active":"2024-09-13T17:13:42.588006+0000","last_peered":"2024-09-13T17:13:42.588006+0000","last_clean":"2024-09-13T17:13:42.588006+0000","last_became_active":"2024-09-13T17:13:42.586639+0000","last_became_peered":"2024-09-13T17:13:42.586639+0000","last_unstale":"2024-09-13T17:13:42.588006+0000","last_undegraded":"2024-09-13T17:13:42.588006+0000","last_fullsized":"2024-09-13T17:13:42.588006+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:40.149830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572477+0000","last_change":"2024-09-13T17:13:42.570072+0000","last_active":"2024-09-13T17:13:42.572477+0000","last_peered":"2024-09-13T17:13:42.572477+0000","last_clean":"2024-09-13T17:13:42.572477+0000","last_became_active":"2024-09-13T17:13:42.569722+0000","last_became_peered":"2024-09-13T17:13:42.569722+0000","last_unstale":"2024-09-13T17:13:42.572477+0000","last_undegraded":"2024-09-13T17:13:42.572477+0000","last_fullsized":"2024-09-13T17:13:42.572477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:46.147942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.581550+0000","last_change":"2024-09-13T17:13:42.579436+0000","last_active":"2024-09-13T17:13:42.581550+0000","last_peered":"2024-09-13T17:13:42.581550+0000","last_clean":"2024-09-13T17:13:42.581550+0000","last_became_active":"2024-09-13T17:13:42.579150+0000","last_became_peered":"2024-09-13T17:13:42.579150+0000","last_unstale":"2024-09-13T17:13:42.581550+0000","last_undegraded":"2024-09-13T17:13:42.581550+0000","last_fullsized":"2024-09-13T17:13:42.581550+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:18.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575397+0000","last_change":"2024-09-13T17:13:42.573033+0000","last_active":"2024-09-13T17:13:42.575397+0000","last_peered":"2024-09-13T17:13:42.575397+0000","last_clean":"2024-09-13T17:13:42.575397+0000","last_became_active":"2024-09-13T17:13:42.571818+0000","last_became_peered":"2024-09-13T17:13:42.571818+0000","last_unstale":"2024-09-13T17:13:42.575397+0000","last_undegraded":"2024-09-13T17:13:42.575397+0000","last_fullsized":"2024-09-13T17:13:42.575397+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:31.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"92'4","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587235+0000","last_change":"2024-09-13T17:13:42.136454+0000","last_active":"2024-09-13T17:13:42.587235+0000","last_peered":"2024-09-13T17:13:42.587235+0000","last_clean":"2024-09-13T17:13:42.587235+0000","last_became_active":"2024-09-13T17:13:42.136178+0000","last_became_peered":"2024-09-13T17:13:42.136178+0000","last_unstale":"2024-09-13T17:13:42.587235+0000","last_undegraded":"2024-09-13T17:13:42.587235+0000","last_fullsized":"2024-09-13T17:13:42.587235+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:39.142667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.579070+0000","last_change":"2024-09-13T17:13:42.573094+0000","last_active":"2024-09-13T17:13:42.579070+0000","last_peered":"2024-09-13T17:13:42.579070+0000","last_clean":"2024-09-13T17:13:42.579070+0000","last_became_active":"2024-09-13T17:13:42.572771+0000","last_became_peered":"2024-09-13T17:13:42.572771+0000","last_unstale":"2024-09-13T17:13:42.579070+0000","last_undegraded":"2024-09-13T17:13:42.579070+0000","last_fullsized":"2024-09-13T17:13:42.579070+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:46.367537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.574532+0000","last_change":"2024-09-13T17:13:42.140396+0000","last_active":"2024-09-13T17:13:42.574532+0000","last_peered":"2024-09-13T17:13:42.574532+0000","last_clean":"2024-09-13T17:13:42.574532+0000","last_became_active":"2024-09-13T17:13:42.139805+0000","last_became_peered":"2024-09-13T17:13:42.139805+0000","last_unstale":"2024-09-13T17:13:42.574532+0000","last_undegraded":"2024-09-13T17:13:42.574532+0000","last_fullsized":"2024-09-13T17:13:42.574532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.581560+0000","last_change":"2024-09-13T17:13:42.579528+0000","last_active":"2024-09-13T17:13:42.581560+0000","last_peered":"2024-09-13T17:13:42.581560+0000","last_clean":"2024-09-13T17:13:42.581560+0000","last_became_active":"2024-09-13T17:13:42.579206+0000","last_became_peered":"2024-09-13T17:13:42.579206+0000","last_unstale":"2024-09-13T17:13:42.581560+0000","last_undegraded":"2024-09-13T17:13:42.581560+0000","last_fullsized":"2024-09-13T17:13:42.581560+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:22.798945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586023+0000","last_change":"2024-09-13T17:13:42.131756+0000","last_active":"2024-09-13T17:13:42.586023+0000","last_peered":"2024-09-13T17:13:42.586023+0000","last_clean":"2024-09-13T17:13:42.586023+0000","last_became_active":"2024-09-13T17:13:42.131508+0000","last_became_peered":"2024-09-13T17:13:42.131508+0000","last_unstale":"2024-09-13T17:13:42.586023+0000","last_undegraded":"2024-09-13T17:13:42.586023+0000","last_fullsized":"2024-09-13T17:13:42.586023+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:39.371938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573522+0000","last_change":"2024-09-13T17:13:42.568985+0000","last_active":"2024-09-13T17:13:42.573522+0000","last_peered":"2024-09-13T17:13:42.573522+0000","last_clean":"2024-09-13T17:13:42.573522+0000","last_became_active":"2024-09-13T17:13:42.567956+0000","last_became_peered":"2024-09-13T17:13:42.567956+0000","last_unstale":"2024-09-13T17:13:42.573522+0000","last_undegraded":"2024-09-13T17:13:42.573522+0000","last_fullsized":"2024-09-13T17:13:42.573522+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:15.878385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585621+0000","last_change":"2024-09-13T17:13:42.136542+0000","last_active":"2024-09-13T17:13:42.585621+0000","last_peered":"2024-09-13T17:13:42.585621+0000","last_clean":"2024-09-13T17:13:42.585621+0000","last_became_active":"2024-09-13T17:13:42.136253+0000","last_became_peered":"2024-09-13T17:13:42.136253+0000","last_unstale":"2024-09-13T17:13:42.585621+0000","last_undegraded":"2024-09-13T17:13:42.585621+0000","last_fullsized":"2024-09-13T17:13:42.585621+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:51.247223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584658+0000","last_change":"2024-09-13T17:13:42.582758+0000","last_active":"2024-09-13T17:13:42.584658+0000","last_peered":"2024-09-13T17:13:42.584658+0000","last_clean":"2024-09-13T17:13:42.584658+0000","last_became_active":"2024-09-13T17:13:42.580619+0000","last_became_peered":"2024-09-13T17:13:42.580619+0000","last_unstale":"2024-09-13T17:13:42.584658+0000","last_undegraded":"2024-09-13T17:13:42.584658+0000","last_fullsized":"2024-09-13T17:13:42.584658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:14.800216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578350+0000","last_change":"2024-09-13T17:13:42.133793+0000","last_active":"2024-09-13T17:13:42.578350+0000","last_peered":"2024-09-13T17:13:42.578350+0000","last_clean":"2024-09-13T17:13:42.578350+0000","last_became_active":"2024-09-13T17:13:42.133310+0000","last_became_peered":"2024-09-13T17:13:42.133310+0000","last_unstale":"2024-09-13T17:13:42.578350+0000","last_undegraded":"2024-09-13T17:13:42.578350+0000","last_fullsized":"2024-09-13T17:13:42.578350+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:40.822787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.571032+0000","last_change":"2024-09-13T17:13:42.568607+0000","last_active":"2024-09-13T17:13:42.571032+0000","last_peered":"2024-09-13T17:13:42.571032+0000","last_clean":"2024-09-13T17:13:42.571032+0000","last_became_active":"2024-09-13T17:13:42.565679+0000","last_became_peered":"2024-09-13T17:13:42.565679+0000","last_unstale":"2024-09-13T17:13:42.571032+0000","last_undegraded":"2024-09-13T17:13:42.571032+0000","last_fullsized":"2024-09-13T17:13:42.571032+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572930+0000","last_change":"2024-09-13T17:13:42.140282+0000","last_active":"2024-09-13T17:13:42.572930+0000","last_peered":"2024-09-13T17:13:42.572930+0000","last_clean":"2024-09-13T17:13:42.572930+0000","last_became_active":"2024-09-13T17:13:42.139519+0000","last_became_peered":"2024-09-13T17:13:42.139519+0000","last_unstale":"2024-09-13T17:13:42.572930+0000","last_undegraded":"2024-09-13T17:13:42.572930+0000","last_fullsized":"2024-09-13T17:13:42.572930+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:00.927188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563776+0000","last_change":"2024-09-13T17:13:42.560328+0000","last_active":"2024-09-13T17:13:42.563776+0000","last_peered":"2024-09-13T17:13:42.563776+0000","last_clean":"2024-09-13T17:13:42.563776+0000","last_became_active":"2024-09-13T17:13:42.560032+0000","last_became_peered":"2024-09-13T17:13:42.560032+0000","last_unstale":"2024-09-13T17:13:42.563776+0000","last_undegraded":"2024-09-13T17:13:42.563776+0000","last_fullsized":"2024-09-13T17:13:42.563776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554170+0000","last_change":"2024-09-13T17:13:42.130235+0000","last_active":"2024-09-13T17:13:42.554170+0000","last_peered":"2024-09-13T17:13:42.554170+0000","last_clean":"2024-09-13T17:13:42.554170+0000","last_became_active":"2024-09-13T17:13:42.129992+0000","last_became_peered":"2024-09-13T17:13:42.129992+0000","last_unstale":"2024-09-13T17:13:42.554170+0000","last_undegraded":"2024-09-13T17:13:42.554170+0000","last_fullsized":"2024-09-13T17:13:42.554170+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:21.791782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.556197+0000","last_change":"2024-09-13T17:13:42.554766+0000","last_active":"2024-09-13T17:13:42.556197+0000","last_peered":"2024-09-13T17:13:42.556197+0000","last_clean":"2024-09-13T17:13:42.556197+0000","last_became_active":"2024-09-13T17:13:42.554484+0000","last_became_peered":"2024-09-13T17:13:42.554484+0000","last_unstale":"2024-09-13T17:13:42.556197+0000","last_undegraded":"2024-09-13T17:13:42.556197+0000","last_fullsized":"2024-09-13T17:13:42.556197+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:18.169178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563157+0000","last_change":"2024-09-13T17:13:42.132000+0000","last_active":"2024-09-13T17:13:42.563157+0000","last_peered":"2024-09-13T17:13:42.563157+0000","last_clean":"2024-09-13T17:13:42.563157+0000","last_became_active":"2024-09-13T17:13:42.131745+0000","last_became_peered":"2024-09-13T17:13:42.131745+0000","last_unstale":"2024-09-13T17:13:42.563157+0000","last_undegraded":"2024-09-13T17:13:42.563157+0000","last_fullsized":"2024-09-13T17:13:42.563157+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:21.888375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580947+0000","last_change":"2024-09-13T17:13:42.578419+0000","last_active":"2024-09-13T17:13:42.580947+0000","last_peered":"2024-09-13T17:13:42.580947+0000","last_clean":"2024-09-13T17:13:42.580947+0000","last_became_active":"2024-09-13T17:13:42.578093+0000","last_became_peered":"2024-09-13T17:13:42.578093+0000","last_unstale":"2024-09-13T17:13:42.580947+0000","last_undegraded":"2024-09-13T17:13:42.580947+0000","last_fullsized":"2024-09-13T17:13:42.580947+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:00.274545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561771+0000","last_change":"2024-09-13T17:13:42.140479+0000","last_active":"2024-09-13T17:13:42.561771+0000","last_peered":"2024-09-13T17:13:42.561771+0000","last_clean":"2024-09-13T17:13:42.561771+0000","last_became_active":"2024-09-13T17:13:42.139612+0000","last_became_peered":"2024-09-13T17:13:42.139612+0000","last_unstale":"2024-09-13T17:13:42.561771+0000","last_undegraded":"2024-09-13T17:13:42.561771+0000","last_fullsized":"2024-09-13T17:13:42.561771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:24.655571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576002+0000","last_change":"2024-09-13T17:13:42.572110+0000","last_active":"2024-09-13T17:13:42.576002+0000","last_peered":"2024-09-13T17:13:42.576002+0000","last_clean":"2024-09-13T17:13:42.576002+0000","last_became_active":"2024-09-13T17:13:42.571804+0000","last_became_peered":"2024-09-13T17:13:42.571804+0000","last_unstale":"2024-09-13T17:13:42.576002+0000","last_undegraded":"2024-09-13T17:13:42.576002+0000","last_fullsized":"2024-09-13T17:13:42.576002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561845+0000","last_change":"2024-09-13T17:13:42.138368+0000","last_active":"2024-09-13T17:13:42.561845+0000","last_peered":"2024-09-13T17:13:42.561845+0000","last_clean":"2024-09-13T17:13:42.561845+0000","last_became_active":"2024-09-13T17:13:42.138078+0000","last_became_peered":"2024-09-13T17:13:42.138078+0000","last_unstale":"2024-09-13T17:13:42.561845+0000","last_undegraded":"2024-09-13T17:13:42.561845+0000","last_fullsized":"2024-09-13T17:13:42.561845+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:50.620928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.567696+0000","last_change":"2024-09-13T17:13:42.565567+0000","last_active":"2024-09-13T17:13:42.567696+0000","last_peered":"2024-09-13T17:13:42.567696+0000","last_clean":"2024-09-13T17:13:42.567696+0000","last_became_active":"2024-09-13T17:13:42.564955+0000","last_became_peered":"2024-09-13T17:13:42.564955+0000","last_unstale":"2024-09-13T17:13:42.567696+0000","last_undegraded":"2024-09-13T17:13:42.567696+0000","last_fullsized":"2024-09-13T17:13:42.567696+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"92'3","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585384+0000","last_change":"2024-09-13T17:13:42.137728+0000","last_active":"2024-09-13T17:13:42.585384+0000","last_peered":"2024-09-13T17:13:42.585384+0000","last_clean":"2024-09-13T17:13:42.585384+0000","last_became_active":"2024-09-13T17:13:42.137438+0000","last_became_peered":"2024-09-13T17:13:42.137438+0000","last_unstale":"2024-09-13T17:13:42.585384+0000","last_undegraded":"2024-09-13T17:13:42.585384+0000","last_fullsized":"2024-09-13T17:13:42.585384+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:10.962523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572474+0000","last_change":"2024-09-13T17:13:42.569958+0000","last_active":"2024-09-13T17:13:42.572474+0000","last_peered":"2024-09-13T17:13:42.572474+0000","last_clean":"2024-09-13T17:13:42.572474+0000","last_became_active":"2024-09-13T17:13:42.569540+0000","last_became_peered":"2024-09-13T17:13:42.569540+0000","last_unstale":"2024-09-13T17:13:42.572474+0000","last_undegraded":"2024-09-13T17:13:42.572474+0000","last_fullsized":"2024-09-13T17:13:42.572474+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:14.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"92'6","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584456+0000","last_change":"2024-09-13T17:13:42.139208+0000","last_active":"2024-09-13T17:13:42.584456+0000","last_peered":"2024-09-13T17:13:42.584456+0000","last_clean":"2024-09-13T17:13:42.584456+0000","last_became_active":"2024-09-13T17:13:42.138134+0000","last_became_peered":"2024-09-13T17:13:42.138134+0000","last_unstale":"2024-09-13T17:13:42.584456+0000","last_undegraded":"2024-09-13T17:13:42.584456+0000","last_fullsized":"2024-09-13T17:13:42.584456+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:45.360374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576151+0000","last_change":"2024-09-13T17:13:42.573037+0000","last_active":"2024-09-13T17:13:42.576151+0000","last_peered":"2024-09-13T17:13:42.576151+0000","last_clean":"2024-09-13T17:13:42.576151+0000","last_became_active":"2024-09-13T17:13:42.572779+0000","last_became_peered":"2024-09-13T17:13:42.572779+0000","last_unstale":"2024-09-13T17:13:42.576151+0000","last_undegraded":"2024-09-13T17:13:42.576151+0000","last_fullsized":"2024-09-13T17:13:42.576151+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:16.283948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584581+0000","last_change":"2024-09-13T17:13:42.126098+0000","last_active":"2024-09-13T17:13:42.584581+0000","last_peered":"2024-09-13T17:13:42.584581+0000","last_clean":"2024-09-13T17:13:42.584581+0000","last_became_active":"2024-09-13T17:13:42.125525+0000","last_became_peered":"2024-09-13T17:13:42.125525+0000","last_unstale":"2024-09-13T17:13:42.584581+0000","last_undegraded":"2024-09-13T17:13:42.584581+0000","last_fullsized":"2024-09-13T17:13:42.584581+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:50.263911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572742+0000","last_change":"2024-09-13T17:13:42.565918+0000","last_active":"2024-09-13T17:13:42.572742+0000","last_peered":"2024-09-13T17:13:42.572742+0000","last_clean":"2024-09-13T17:13:42.572742+0000","last_became_active":"2024-09-13T17:13:42.564373+0000","last_became_peered":"2024-09-13T17:13:42.564373+0000","last_unstale":"2024-09-13T17:13:42.572742+0000","last_undegraded":"2024-09-13T17:13:42.572742+0000","last_fullsized":"2024-09-13T17:13:42.572742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.550015+0000","last_change":"2024-09-13T17:13:42.135482+0000","last_active":"2024-09-13T17:13:42.550015+0000","last_peered":"2024-09-13T17:13:42.550015+0000","last_clean":"2024-09-13T17:13:42.550015+0000","last_became_active":"2024-09-13T17:13:42.132794+0000","last_became_peered":"2024-09-13T17:13:42.132794+0000","last_unstale":"2024-09-13T17:13:42.550015+0000","last_undegraded":"2024-09-13T17:13:42.550015+0000","last_fullsized":"2024-09-13T17:13:42.550015+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:15.276785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584150+0000","last_change":"2024-09-13T17:13:42.581645+0000","last_active":"2024-09-13T17:13:42.584150+0000","last_peered":"2024-09-13T17:13:42.584150+0000","last_clean":"2024-09-13T17:13:42.584150+0000","last_became_active":"2024-09-13T17:13:42.581320+0000","last_became_peered":"2024-09-13T17:13:42.581320+0000","last_unstale":"2024-09-13T17:13:42.584150+0000","last_undegraded":"2024-09-13T17:13:42.584150+0000","last_fullsized":"2024-09-13T17:13:42.584150+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:33.422248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.553044+0000","last_change":"2024-09-13T17:13:42.135228+0000","last_active":"2024-09-13T17:13:42.553044+0000","last_peered":"2024-09-13T17:13:42.553044+0000","last_clean":"2024-09-13T17:13:42.553044+0000","last_became_active":"2024-09-13T17:13:42.134935+0000","last_became_peered":"2024-09-13T17:13:42.134935+0000","last_unstale":"2024-09-13T17:13:42.553044+0000","last_undegraded":"2024-09-13T17:13:42.553044+0000","last_fullsized":"2024-09-13T17:13:42.553044+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:16.947362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587581+0000","last_change":"2024-09-13T17:13:42.586505+0000","last_active":"2024-09-13T17:13:42.587581+0000","last_peered":"2024-09-13T17:13:42.587581+0000","last_clean":"2024-09-13T17:13:42.587581+0000","last_became_active":"2024-09-13T17:13:42.586229+0000","last_became_peered":"2024-09-13T17:13:42.586229+0000","last_unstale":"2024-09-13T17:13:42.587581+0000","last_undegraded":"2024-09-13T17:13:42.587581+0000","last_fullsized":"2024-09-13T17:13:42.587581+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.940621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"92'8","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.583826+0000","last_change":"2024-09-13T17:13:42.126042+0000","last_active":"2024-09-13T17:13:42.583826+0000","last_peered":"2024-09-13T17:13:42.583826+0000","last_clean":"2024-09-13T17:13:42.583826+0000","last_became_active":"2024-09-13T17:13:42.125753+0000","last_became_peered":"2024-09-13T17:13:42.125753+0000","last_unstale":"2024-09-13T17:13:42.583826+0000","last_undegraded":"2024-09-13T17:13:42.583826+0000","last_fullsized":"2024-09-13T17:13:42.583826+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:40.822787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573025+0000","last_change":"2024-09-13T17:13:42.565041+0000","last_active":"2024-09-13T17:13:42.573025+0000","last_peered":"2024-09-13T17:13:42.573025+0000","last_clean":"2024-09-13T17:13:42.573025+0000","last_became_active":"2024-09-13T17:13:42.564780+0000","last_became_peered":"2024-09-13T17:13:42.564780+0000","last_unstale":"2024-09-13T17:13:42.573025+0000","last_undegraded":"2024-09-13T17:13:42.573025+0000","last_fullsized":"2024-09-13T17:13:42.573025+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:04.990277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580524+0000","last_change":"2024-09-13T17:13:42.134017+0000","last_active":"2024-09-13T17:13:42.580524+0000","last_peered":"2024-09-13T17:13:42.580524+0000","last_clean":"2024-09-13T17:13:42.580524+0000","last_became_active":"2024-09-13T17:13:42.133741+0000","last_became_peered":"2024-09-13T17:13:42.133741+0000","last_unstale":"2024-09-13T17:13:42.580524+0000","last_undegraded":"2024-09-13T17:13:42.580524+0000","last_fullsized":"2024-09-13T17:13:42.580524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:17.190735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573413+0000","last_change":"2024-09-13T17:13:42.565995+0000","last_active":"2024-09-13T17:13:42.573413+0000","last_peered":"2024-09-13T17:13:42.573413+0000","last_clean":"2024-09-13T17:13:42.573413+0000","last_became_active":"2024-09-13T17:13:42.565717+0000","last_became_peered":"2024-09-13T17:13:42.565717+0000","last_unstale":"2024-09-13T17:13:42.573413+0000","last_undegraded":"2024-09-13T17:13:42.573413+0000","last_fullsized":"2024-09-13T17:13:42.573413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:06.495250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.583830+0000","last_change":"2024-09-13T17:13:42.135305+0000","last_active":"2024-09-13T17:13:42.583830+0000","last_peered":"2024-09-13T17:13:42.583830+0000","last_clean":"2024-09-13T17:13:42.583830+0000","last_became_active":"2024-09-13T17:13:42.135047+0000","last_became_peered":"2024-09-13T17:13:42.135047+0000","last_unstale":"2024-09-13T17:13:42.583830+0000","last_undegraded":"2024-09-13T17:13:42.583830+0000","last_fullsized":"2024-09-13T17:13:42.583830+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:17.471476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580916+0000","last_change":"2024-09-13T17:13:42.579748+0000","last_active":"2024-09-13T17:13:42.580916+0000","last_peered":"2024-09-13T17:13:42.580916+0000","last_clean":"2024-09-13T17:13:42.580916+0000","last_became_active":"2024-09-13T17:13:42.579430+0000","last_became_peered":"2024-09-13T17:13:42.579430+0000","last_unstale":"2024-09-13T17:13:42.580916+0000","last_undegraded":"2024-09-13T17:13:42.580916+0000","last_fullsized":"2024-09-13T17:13:42.580916+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554543+0000","last_change":"2024-09-13T17:13:42.127749+0000","last_active":"2024-09-13T17:13:42.554543+0000","last_peered":"2024-09-13T17:13:42.554543+0000","last_clean":"2024-09-13T17:13:42.554543+0000","last_became_active":"2024-09-13T17:13:42.127277+0000","last_became_peered":"2024-09-13T17:13:42.127277+0000","last_unstale":"2024-09-13T17:13:42.554543+0000","last_undegraded":"2024-09-13T17:13:42.554543+0000","last_fullsized":"2024-09-13T17:13:42.554543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:19.504868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586857+0000","last_change":"2024-09-13T17:13:42.581085+0000","last_active":"2024-09-13T17:13:42.586857+0000","last_peered":"2024-09-13T17:13:42.586857+0000","last_clean":"2024-09-13T17:13:42.586857+0000","last_became_active":"2024-09-13T17:13:42.579417+0000","last_became_peered":"2024-09-13T17:13:42.579417+0000","last_unstale":"2024-09-13T17:13:42.586857+0000","last_undegraded":"2024-09-13T17:13:42.586857+0000","last_fullsized":"2024-09-13T17:13:42.586857+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:22.895538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580193+0000","last_change":"2024-09-13T17:13:42.132935+0000","last_active":"2024-09-13T17:13:42.580193+0000","last_peered":"2024-09-13T17:13:42.580193+0000","last_clean":"2024-09-13T17:13:42.580193+0000","last_became_active":"2024-09-13T17:13:42.129217+0000","last_became_peered":"2024-09-13T17:13:42.129217+0000","last_unstale":"2024-09-13T17:13:42.580193+0000","last_undegraded":"2024-09-13T17:13:42.580193+0000","last_fullsized":"2024-09-13T17:13:42.580193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:02.027252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.566173+0000","last_change":"2024-09-13T17:13:42.561180+0000","last_active":"2024-09-13T17:13:42.566173+0000","last_peered":"2024-09-13T17:13:42.566173+0000","last_clean":"2024-09-13T17:13:42.566173+0000","last_became_active":"2024-09-13T17:13:42.560880+0000","last_became_peered":"2024-09-13T17:13:42.560880+0000","last_unstale":"2024-09-13T17:13:42.566173+0000","last_undegraded":"2024-09-13T17:13:42.566173+0000","last_fullsized":"2024-09-13T17:13:42.566173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:22.798945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.559119+0000","last_change":"2024-09-13T17:13:42.126581+0000","last_active":"2024-09-13T17:13:42.559119+0000","last_peered":"2024-09-13T17:13:42.559119+0000","last_clean":"2024-09-13T17:13:42.559119+0000","last_became_active":"2024-09-13T17:13:42.126297+0000","last_became_peered":"2024-09-13T17:13:42.126297+0000","last_unstale":"2024-09-13T17:13:42.559119+0000","last_undegraded":"2024-09-13T17:13:42.559119+0000","last_fullsized":"2024-09-13T17:13:42.559119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:10.889340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576136+0000","last_change":"2024-09-13T17:13:42.562810+0000","last_active":"2024-09-13T17:13:42.576136+0000","last_peered":"2024-09-13T17:13:42.576136+0000","last_clean":"2024-09-13T17:13:42.576136+0000","last_became_active":"2024-09-13T17:13:42.562293+0000","last_became_peered":"2024-09-13T17:13:42.562293+0000","last_unstale":"2024-09-13T17:13:42.576136+0000","last_undegraded":"2024-09-13T17:13:42.576136+0000","last_fullsized":"2024-09-13T17:13:42.576136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:38.126662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558124+0000","last_change":"2024-09-13T17:13:42.136062+0000","last_active":"2024-09-13T17:13:42.558124+0000","last_peered":"2024-09-13T17:13:42.558124+0000","last_clean":"2024-09-13T17:13:42.558124+0000","last_became_active":"2024-09-13T17:13:42.135770+0000","last_became_peered":"2024-09-13T17:13:42.135770+0000","last_unstale":"2024-09-13T17:13:42.558124+0000","last_undegraded":"2024-09-13T17:13:42.558124+0000","last_fullsized":"2024-09-13T17:13:42.558124+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:23.852299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568087+0000","last_change":"2024-09-13T17:13:42.565573+0000","last_active":"2024-09-13T17:13:42.568087+0000","last_peered":"2024-09-13T17:13:42.568087+0000","last_clean":"2024-09-13T17:13:42.568087+0000","last_became_active":"2024-09-13T17:13:42.565261+0000","last_became_peered":"2024-09-13T17:13:42.565261+0000","last_unstale":"2024-09-13T17:13:42.568087+0000","last_undegraded":"2024-09-13T17:13:42.568087+0000","last_fullsized":"2024-09-13T17:13:42.568087+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.549848+0000","last_change":"2024-09-13T17:13:42.130872+0000","last_active":"2024-09-13T17:13:42.549848+0000","last_peered":"2024-09-13T17:13:42.549848+0000","last_clean":"2024-09-13T17:13:42.549848+0000","last_became_active":"2024-09-13T17:13:42.130446+0000","last_became_peered":"2024-09-13T17:13:42.130446+0000","last_unstale":"2024-09-13T17:13:42.549848+0000","last_undegraded":"2024-09-13T17:13:42.549848+0000","last_fullsized":"2024-09-13T17:13:42.549848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:47.565746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573438+0000","last_change":"2024-09-13T17:13:42.570453+0000","last_active":"2024-09-13T17:13:42.573438+0000","last_peered":"2024-09-13T17:13:42.573438+0000","last_clean":"2024-09-13T17:13:42.573438+0000","last_became_active":"2024-09-13T17:13:42.569615+0000","last_became_peered":"2024-09-13T17:13:42.569615+0000","last_unstale":"2024-09-13T17:13:42.573438+0000","last_undegraded":"2024-09-13T17:13:42.573438+0000","last_fullsized":"2024-09-13T17:13:42.573438+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:04.021181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587095+0000","last_change":"2024-09-13T17:13:42.128559+0000","last_active":"2024-09-13T17:13:42.587095+0000","last_peered":"2024-09-13T17:13:42.587095+0000","last_clean":"2024-09-13T17:13:42.587095+0000","last_became_active":"2024-09-13T17:13:42.128286+0000","last_became_peered":"2024-09-13T17:13:42.128286+0000","last_unstale":"2024-09-13T17:13:42.587095+0000","last_undegraded":"2024-09-13T17:13:42.587095+0000","last_fullsized":"2024-09-13T17:13:42.587095+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:35:25.031522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575732+0000","last_change":"2024-09-13T17:13:42.570898+0000","last_active":"2024-09-13T17:13:42.575732+0000","last_peered":"2024-09-13T17:13:42.575732+0000","last_clean":"2024-09-13T17:13:42.575732+0000","last_became_active":"2024-09-13T17:13:42.570607+0000","last_became_peered":"2024-09-13T17:13:42.570607+0000","last_unstale":"2024-09-13T17:13:42.575732+0000","last_undegraded":"2024-09-13T17:13:42.575732+0000","last_fullsized":"2024-09-13T17:13:42.575732+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563516+0000","last_change":"2024-09-13T17:13:42.126651+0000","last_active":"2024-09-13T17:13:42.563516+0000","last_peered":"2024-09-13T17:13:42.563516+0000","last_clean":"2024-09-13T17:13:42.563516+0000","last_became_active":"2024-09-13T17:13:42.126377+0000","last_became_peered":"2024-09-13T17:13:42.126377+0000","last_unstale":"2024-09-13T17:13:42.563516+0000","last_undegraded":"2024-09-13T17:13:42.563516+0000","last_fullsized":"2024-09-13T17:13:42.563516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:37.119499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578431+0000","last_change":"2024-09-13T17:13:42.575233+0000","last_active":"2024-09-13T17:13:42.578431+0000","last_peered":"2024-09-13T17:13:42.578431+0000","last_clean":"2024-09-13T17:13:42.578431+0000","last_became_active":"2024-09-13T17:13:42.574862+0000","last_became_peered":"2024-09-13T17:13:42.574862+0000","last_unstale":"2024-09-13T17:13:42.578431+0000","last_undegraded":"2024-09-13T17:13:42.578431+0000","last_fullsized":"2024-09-13T17:13:42.578431+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.810829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558605+0000","last_change":"2024-09-13T17:13:42.137207+0000","last_active":"2024-09-13T17:13:42.558605+0000","last_peered":"2024-09-13T17:13:42.558605+0000","last_clean":"2024-09-13T17:13:42.558605+0000","last_became_active":"2024-09-13T17:13:42.136326+0000","last_became_peered":"2024-09-13T17:13:42.136326+0000","last_unstale":"2024-09-13T17:13:42.558605+0000","last_undegraded":"2024-09-13T17:13:42.558605+0000","last_fullsized":"2024-09-13T17:13:42.558605+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:13.793053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572351+0000","last_change":"2024-09-13T17:13:42.569496+0000","last_active":"2024-09-13T17:13:42.572351+0000","last_peered":"2024-09-13T17:13:42.572351+0000","last_clean":"2024-09-13T17:13:42.572351+0000","last_became_active":"2024-09-13T17:13:42.569173+0000","last_became_peered":"2024-09-13T17:13:42.569173+0000","last_unstale":"2024-09-13T17:13:42.572351+0000","last_undegraded":"2024-09-13T17:13:42.572351+0000","last_fullsized":"2024-09-13T17:13:42.572351+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:00.274545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558758+0000","last_change":"2024-09-13T17:13:42.140084+0000","last_active":"2024-09-13T17:13:42.558758+0000","last_peered":"2024-09-13T17:13:42.558758+0000","last_clean":"2024-09-13T17:13:42.558758+0000","last_became_active":"2024-09-13T17:13:42.139742+0000","last_became_peered":"2024-09-13T17:13:42.139742+0000","last_unstale":"2024-09-13T17:13:42.558758+0000","last_undegraded":"2024-09-13T17:13:42.558758+0000","last_fullsized":"2024-09-13T17:13:42.558758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:48.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.566026+0000","last_change":"2024-09-13T17:13:42.560682+0000","last_active":"2024-09-13T17:13:42.566026+0000","last_peered":"2024-09-13T17:13:42.566026+0000","last_clean":"2024-09-13T17:13:42.566026+0000","last_became_active":"2024-09-13T17:13:42.560408+0000","last_became_peered":"2024-09-13T17:13:42.560408+0000","last_unstale":"2024-09-13T17:13:42.566026+0000","last_undegraded":"2024-09-13T17:13:42.566026+0000","last_fullsized":"2024-09-13T17:13:42.566026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:40.379101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554535+0000","last_change":"2024-09-13T17:13:42.128098+0000","last_active":"2024-09-13T17:13:42.554535+0000","last_peered":"2024-09-13T17:13:42.554535+0000","last_clean":"2024-09-13T17:13:42.554535+0000","last_became_active":"2024-09-13T17:13:42.127802+0000","last_became_peered":"2024-09-13T17:13:42.127802+0000","last_unstale":"2024-09-13T17:13:42.554535+0000","last_undegraded":"2024-09-13T17:13:42.554535+0000","last_fullsized":"2024-09-13T17:13:42.554535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:05.488087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576155+0000","last_change":"2024-09-13T17:13:42.573548+0000","last_active":"2024-09-13T17:13:42.576155+0000","last_peered":"2024-09-13T17:13:42.576155+0000","last_clean":"2024-09-13T17:13:42.576155+0000","last_became_active":"2024-09-13T17:13:42.572378+0000","last_became_peered":"2024-09-13T17:13:42.572378+0000","last_unstale":"2024-09-13T17:13:42.576155+0000","last_undegraded":"2024-09-13T17:13:42.576155+0000","last_fullsized":"2024-09-13T17:13:42.576155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561697+0000","last_change":"2024-09-13T17:13:42.138432+0000","last_active":"2024-09-13T17:13:42.561697+0000","last_peered":"2024-09-13T17:13:42.561697+0000","last_clean":"2024-09-13T17:13:42.561697+0000","last_became_active":"2024-09-13T17:13:42.138149+0000","last_became_peered":"2024-09-13T17:13:42.138149+0000","last_unstale":"2024-09-13T17:13:42.561697+0000","last_undegraded":"2024-09-13T17:13:42.561697+0000","last_fullsized":"2024-09-13T17:13:42.561697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:54.525735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580102+0000","last_change":"2024-09-13T17:13:42.578744+0000","last_active":"2024-09-13T17:13:42.580102+0000","last_peered":"2024-09-13T17:13:42.580102+0000","last_clean":"2024-09-13T17:13:42.580102+0000","last_became_active":"2024-09-13T17:13:42.578486+0000","last_became_peered":"2024-09-13T17:13:42.578486+0000","last_unstale":"2024-09-13T17:13:42.580102+0000","last_undegraded":"2024-09-13T17:13:42.580102+0000","last_fullsized":"2024-09-13T17:13:42.580102+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:24.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561128+0000","last_change":"2024-09-13T17:13:42.139980+0000","last_active":"2024-09-13T17:13:42.561128+0000","last_peered":"2024-09-13T17:13:42.561128+0000","last_clean":"2024-09-13T17:13:42.561128+0000","last_became_active":"2024-09-13T17:13:42.139627+0000","last_became_peered":"2024-09-13T17:13:42.139627+0000","last_unstale":"2024-09-13T17:13:42.561128+0000","last_undegraded":"2024-09-13T17:13:42.561128+0000","last_fullsized":"2024-09-13T17:13:42.561128+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:32.415085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573003+0000","last_change":"2024-09-13T17:13:42.566787+0000","last_active":"2024-09-13T17:13:42.573003+0000","last_peered":"2024-09-13T17:13:42.573003+0000","last_clean":"2024-09-13T17:13:42.573003+0000","last_became_active":"2024-09-13T17:13:42.566517+0000","last_became_peered":"2024-09-13T17:13:42.566517+0000","last_unstale":"2024-09-13T17:13:42.573003+0000","last_undegraded":"2024-09-13T17:13:42.573003+0000","last_fullsized":"2024-09-13T17:13:42.573003+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:52.254386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558579+0000","last_change":"2024-09-13T17:13:42.137717+0000","last_active":"2024-09-13T17:13:42.558579+0000","last_peered":"2024-09-13T17:13:42.558579+0000","last_clean":"2024-09-13T17:13:42.558579+0000","last_became_active":"2024-09-13T17:13:42.137325+0000","last_became_peered":"2024-09-13T17:13:42.137325+0000","last_unstale":"2024-09-13T17:13:42.558579+0000","last_undegraded":"2024-09-13T17:13:42.558579+0000","last_fullsized":"2024-09-13T17:13:42.558579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:05.488087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576401+0000","last_change":"2024-09-13T17:13:42.570333+0000","last_active":"2024-09-13T17:13:42.576401+0000","last_peered":"2024-09-13T17:13:42.576401+0000","last_clean":"2024-09-13T17:13:42.576401+0000","last_became_active":"2024-09-13T17:13:42.570055+0000","last_became_peered":"2024-09-13T17:13:42.570055+0000","last_unstale":"2024-09-13T17:13:42.576401+0000","last_undegraded":"2024-09-13T17:13:42.576401+0000","last_fullsized":"2024-09-13T17:13:42.576401+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:11.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585372+0000","last_change":"2024-09-13T17:13:42.135670+0000","last_active":"2024-09-13T17:13:42.585372+0000","last_peered":"2024-09-13T17:13:42.585372+0000","last_clean":"2024-09-13T17:13:42.585372+0000","last_became_active":"2024-09-13T17:13:42.135394+0000","last_became_peered":"2024-09-13T17:13:42.135394+0000","last_unstale":"2024-09-13T17:13:42.585372+0000","last_undegraded":"2024-09-13T17:13:42.585372+0000","last_fullsized":"2024-09-13T17:13:42.585372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.124978+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:13:42.124978+0000","last_peered":"2024-09-13T17:13:42.124978+0000","last_clean":"2024-09-13T17:13:42.124978+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:13:42.124978+0000","last_undegraded":"2024-09-13T17:13:42.124978+0000","last_fullsized":"2024-09-13T17:13:42.124978+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574920,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":1699840,"data_stored":1064405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66000000000000003,"15min":0.64500000000000002},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.877,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":2.0880000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97999999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":2.6739999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.66100000000000003,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":2.5030000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":6.4909999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.48499999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84899999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":1.544},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.54300000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93400000000000005,"5min":1.224,"15min":1.224},"last":1.605}]},{"osd":3,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68000000000000005},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.014,"5min":1.071,"15min":1.071},"last":2.4249999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.63900000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.998,"5min":1.177,"15min":1.177},"last":2.8239999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":1.054,"15min":1.054},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67300000000000004,"15min":0.69699999999999995},"min":{"1min":0.42699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.92900000000000005,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77200000000000002,"5min":1.143,"15min":1.143},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.62,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93600000000000005,"5min":1.337,"15min":1.337},"last":0.71799999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.74299999999999999,"15min":0.76700000000000002},"min":{"1min":0.495,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85699999999999998,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.77800000000000002},"min":{"1min":0.60299999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90600000000000003,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.75}]}]},{"osd":6,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40388,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331452,"statfs":{"total":96636764160,"available":96595406848,"internally_reserved":0,"allocated":1105920,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69599999999999995,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93500000000000005,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.73699999999999999,"15min":0.72499999999999998},"min":{"1min":0.64800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89800000000000002,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.80900000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.69399999999999995,"15min":0.65200000000000002},"min":{"1min":0.55400000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.437},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.63900000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96399999999999997,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.746}]},{"osd":2,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.65000000000000002,"15min":0.66900000000000004},"min":{"1min":0.54000000000000004,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90400000000000003,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.71099999999999997,"15min":0.68899999999999995},"min":{"1min":0.56899999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1.012,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.78400000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97799999999999998,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.58399999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.089,"5min":3.742,"15min":3.742},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.62,"15min":0.627},"min":{"1min":0.49199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.026,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.075,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.53200000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0589999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70099999999999996,"15min":0.72399999999999998},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.127,"5min":3.609,"15min":3.609},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.497,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1499999999999999,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60099999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.202,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.72599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574920,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40400,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331440,"statfs":{"total":96636764160,"available":96595394560,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.997,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.877,"15min":0.77300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.92300000000000004,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.64400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.83199999999999996,"15min":0.73799999999999999},"min":{"1min":0.502,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.628},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72399999999999998,"15min":0.70599999999999996},"min":{"1min":0.499,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91200000000000003,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.84999999999999998,"15min":0.76400000000000001},"min":{"1min":0.52500000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78900000000000003,"15min":0.75900000000000001},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84499999999999997,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.8029999999999999,"15min":1.111},"min":{"1min":0.54000000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.86099999999999999,"5min":113.363,"15min":113.363},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89100000000000001,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.628,"15min":0.626},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94799999999999995,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.70699999999999996},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0800000000000001,"5min":2.056,"15min":2.056},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.86099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79200000000000004,"15min":0.76200000000000001},"min":{"1min":0.59399999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.96599999999999997,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78600000000000003,"15min":0.76800000000000002},"min":{"1min":0.63200000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0549999999999999,"5min":2.02,"15min":2.02},"last":0.77300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46164,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325676,"statfs":{"total":96636764160,"available":96589492224,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.89900000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.63},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94999999999999996,"5min":1.341,"15min":1.341},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.64300000000000002,"15min":0.66100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64400000000000002,"15min":0.621},"min":{"1min":0.49199999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92000000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.48299999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70399999999999996,"15min":0.72199999999999998},"min":{"1min":0.47599999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.995,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66600000000000004,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.002,"15min":1.002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68999999999999995,"15min":0.69299999999999995},"min":{"1min":0.53500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.62,"15min":0.64500000000000002},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.502},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.47299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71099999999999997,"15min":0.745},"min":{"1min":0.55300000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.998,"5min":1.413,"15min":1.413},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.54700000000000004,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0369999999999999,"5min":1.054,"15min":1.054},"last":0.754},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.63,"5min":0.371,"15min":0.371},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.72199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50252,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321588,"statfs":{"total":96636764160,"available":96585306112,"internally_reserved":0,"allocated":1105920,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46100000000000002,"15min":0.46300000000000002},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.177,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.501,"15min":0.56899999999999995},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73499999999999999,"5min":2.96,"15min":2.96},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53700000000000003,"15min":0.52100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91900000000000004,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.65300000000000002,"15min":0.73099999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0309999999999999,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.39800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.97699999999999998,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.51500000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1339999999999999,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.61499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.63400000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0580000000000001,"5min":1.391,"15min":1.391},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78600000000000003,"15min":0.76900000000000002},"min":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.042,"5min":1.401,"15min":1.401},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79600000000000004,"15min":0.80000000000000004},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81899999999999995,"15min":0.86499999999999999},"min":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0960000000000001,"5min":2.899,"15min":2.899},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.83299999999999996},"min":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0740000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.81200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574920,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40408,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331432,"statfs":{"total":96636764160,"available":96595386368,"internally_reserved":0,"allocated":1130496,"data_stored":486650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.751,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.434},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":3.073,"15min":1.341},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.97399999999999998,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":3.1219999999999999,"15min":1.3560000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95099999999999996,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.38}]},{"osd":3,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.90800000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60499999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.379,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.77900000000000003,"15min":0.78600000000000003},"min":{"1min":0.58999999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5229999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.59899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.302,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.61699999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2789999999999999,"5min":1.569,"15min":1.569},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.63600000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":1.413,"15min":1.413},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2490000000000001,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.72599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.66400000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.80100000000000005,"15min":0.80600000000000005},"min":{"1min":0.61699999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.369,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50260,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321580,"statfs":{"total":96636764160,"available":96585297920,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57699999999999996,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57599999999999996,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64900000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.54700000000000004,"15min":0.56699999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66600000000000004,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.77900000000000003,"15min":0.69499999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73699999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.74199999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78300000000000003,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83599999999999997,"15min":0.81799999999999995},"min":{"1min":0.55400000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0720000000000001,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82699999999999996,"15min":0.82299999999999995},"min":{"1min":0.53000000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0649999999999999,"5min":1.78,"15min":1.78},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.55600000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":0.80900000000000005},"min":{"1min":0.63900000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0900000000000001,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0740000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.628},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.85999999999999999,"15min":0.82999999999999996},"min":{"1min":0.57999999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.83299999999999996},"min":{"1min":0.56699999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1080000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84799999999999998,"15min":0.83199999999999996},"min":{"1min":0.60699999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.103,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.66500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40980,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330860,"statfs":{"total":96636764160,"available":96594800640,"internally_reserved":0,"allocated":1708032,"data_stored":1069661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51600000000000001,"15min":0.70199999999999996},"min":{"1min":0.29299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53400000000000003,"15min":0.69099999999999995},"min":{"1min":0.309,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.95399999999999996,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56200000000000006,"15min":0.72599999999999998},"min":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.57499999999999996,"15min":0.74199999999999999},"min":{"1min":0.375,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66600000000000004,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.34599999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.54700000000000004,"15min":0.68100000000000005},"min":{"1min":0.34300000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.77400000000000002,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":0.73299999999999998},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.80300000000000005,"15min":0.91100000000000003},"min":{"1min":0.55600000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.139,"5min":4.734,"15min":4.734},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79800000000000004,"15min":0.92600000000000005},"min":{"1min":0.54900000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.1180000000000001,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.80700000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.94299999999999995},"min":{"1min":0.54400000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90500000000000003,"5min":4.944,"15min":4.944},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.92600000000000005},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85999999999999999,"15min":1.097},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.194,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82599999999999996,"15min":0.95199999999999996},"min":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.044,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.629}]},{"osd":7,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.84099999999999997,"15min":0.98799999999999999},"min":{"1min":0.57499999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.96799999999999997},"min":{"1min":0.57599999999999996,"5min":0.505,"15min":0.505},"max":{"1min":0.97799999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:13:46.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:13:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.945+0000 7f07a2e00640 1 Processor -- start 2024-09-13T17:13:46.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.945+0000 7f07a2e00640 1 -- start start 2024-09-13T17:13:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.946+0000 7f07a2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.946+0000 7f07a2e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f079c06f970 0x7f079c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.946+0000 7f07a2e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0b2040 con 0x7f079c06c2d0 2024-09-13T17:13:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.946+0000 7f07a2e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079c15b0b0 con 0x7f079c06f970 2024-09-13T17:13:46.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.946+0000 7f07a2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c159ff0 con 0x7f079c1592a0 2024-09-13T17:13:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:51300/0 (socket says 172.21.15.28:51300) 2024-09-13T17:13:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 -- 172.21.15.28:0/4283551302 learned_addr learned my addr 172.21.15.28:0/4283551302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:46.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f07a0e00640 1 --1- 172.21.15.28:0/4283551302 >> v1:172.21.15.189:6789/0 conn(0x7f079c06c2d0 0x7f079c0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:33100/0 (socket says 172.21.15.28:33100) 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f0793400640 1 --2- 172.21.15.28:0/4283551302 >> v2:172.21.15.189:3300/0 conn(0x7f079c06f970 0x7f079c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 -- 172.21.15.28:0/4283551302 >> v2:172.21.15.189:3300/0 conn(0x7f079c06f970 msgr2=0x7f079c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 --2- 172.21.15.28:0/4283551302 >> v2:172.21.15.189:3300/0 conn(0x7f079c06f970 0x7f079c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 -- 172.21.15.28:0/4283551302 >> v1:172.21.15.189:6789/0 conn(0x7f079c06c2d0 legacy=0x7f079c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.947+0000 7f079be00640 1 -- 172.21.15.28:0/4283551302 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c15a2d0 con 0x7f079c1592a0 2024-09-13T17:13:46.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.948+0000 7f079be00640 1 --2- 172.21.15.28:0/4283551302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bb10 tx=0x7f078c02c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d13bdc131ceebd9a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f079b400640 1 -- 172.21.15.28:0/4283551302 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c032820 con 0x7f079c1592a0 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f079b400640 1 -- 172.21.15.28:0/4283551302 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c02e040 con 0x7f079c1592a0 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f079b400640 1 -- 172.21.15.28:0/4283551302 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c032ba0 con 0x7f079c1592a0 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 -- 172.21.15.28:0/4283551302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 msgr2=0x7f079c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 --2- 172.21.15.28:0/4283551302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f078c00bb10 tx=0x7f078c02c920 comp rx=0 tx=0).stop 2024-09-13T17:13:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 -- 172.21.15.28:0/4283551302 shutdown_connections 2024-09-13T17:13:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 --2- 172.21.15.28:0/4283551302 >> v2:172.21.15.189:3300/0 conn(0x7f079c06f970 0x7f079c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 --2- 172.21.15.28:0/4283551302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c1592a0 0x7f079c15ab70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 -- 172.21.15.28:0/4283551302 >> 172.21.15.28:0/4283551302 conn(0x7f079c069e80 msgr2=0x7f079c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 -- 172.21.15.28:0/4283551302 shutdown_connections 2024-09-13T17:13:46.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.949+0000 7f07a2e00640 1 -- 172.21.15.28:0/4283551302 wait complete. 2024-09-13T17:13:46.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.951+0000 7f07a2e00640 1 Processor -- start 2024-09-13T17:13:46.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.951+0000 7f07a2e00640 1 -- start start 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 0x7f079c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c06f970 0x7f079c14c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c1538f0 con 0x7f079c1592a0 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c14cda0 con 0x7f079c06f970 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c153b40 con 0x7f079c06c2d0 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44074/0 (socket says 172.21.15.28:44074) 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f07a0e00640 1 -- 172.21.15.28:0/3196168669 learned_addr learned my addr 172.21.15.28:0/3196168669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.952+0000 7f079be00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 0x7f079c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f0793400640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c06f970 0x7f079c14c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 -- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 msgr2=0x7f079c14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 0x7f079c14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 -- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c06f970 msgr2=0x7f079c14c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c06f970 0x7f079c14c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 -- 172.21.15.28:0/3196168669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c00b770 con 0x7f079c1592a0 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f079be00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 0x7f079c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a0e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f079400c7b0 tx=0x7f079400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400a640 con 0x7f079c1592a0 2024-09-13T17:13:46.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079400a7a0 con 0x7f079c1592a0 2024-09-13T17:13:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400ab10 con 0x7f079c1592a0 2024-09-13T17:13:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c153e20 con 0x7f079c1592a0 2024-09-13T17:13:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.953+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c1543e0 con 0x7f079c1592a0 2024-09-13T17:13:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.954+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c154600 con 0x7f079c1592a0 2024-09-13T17:13:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.955+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f079400ac70 con 0x7f079c1592a0 2024-09-13T17:13:46.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.956+0000 7f0798c00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 0x7f077806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:46.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.956+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0794088bf0 con 0x7f079c1592a0 2024-09-13T17:13:46.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.957+0000 7f079be00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 0x7f077806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:46.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.957+0000 7f079be00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 0x7f077806d0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f078c037000 tx=0x7f078c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:46.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:46.960+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0794053d50 con 0x7f079c1592a0 2024-09-13T17:13:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.137+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f079c156e70 con 0x7f077806ac00 2024-09-13T17:13:47.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.145+0000 7f0798c00640 1 -- 172.21.15.28:0/3196168669 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289460 (secure 0 0 0) 0x7f079c156e70 con 0x7f077806ac00 2024-09-13T17:13:47.149 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:47.150 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 msgr2=0x7f077806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 0x7f077806d0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f078c037000 tx=0x7f078c032000 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 msgr2=0x7f079c0c5440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f079400c7b0 tx=0x7f079400cc70 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 shutdown_connections 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f077806ac00 0x7f077806d0c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f079c1592a0 0x7f079c0c5440 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c06f970 0x7f079c14c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 --2- 172.21.15.28:0/3196168669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f079c06c2d0 0x7f079c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.148+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 >> 172.21.15.28:0/3196168669 conn(0x7f079c069e80 msgr2=0x7f079c06fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.149+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 shutdown_connections 2024-09-13T17:13:47.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.149+0000 7f07a2e00640 1 -- 172.21.15.28:0/3196168669 wait complete. 2024-09-13T17:13:47.165 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-13T17:13:46.969420+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":98,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":367872,"kb_used_data":9868,"kb_used_omap":12,"kb_used_meta":357875,"kb_avail":754606848,"statfs":{"total":773094113280,"available":772717412352,"internally_reserved":0,"allocated":10104832,"data_stored":5009726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":366464569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.863345"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.579045+0000","last_change":"2024-09-13T17:13:42.568583+0000","last_active":"2024-09-13T17:13:42.579045+0000","last_peered":"2024-09-13T17:13:42.579045+0000","last_clean":"2024-09-13T17:13:42.579045+0000","last_became_active":"2024-09-13T17:13:42.568311+0000","last_became_peered":"2024-09-13T17:13:42.568311+0000","last_unstale":"2024-09-13T17:13:42.579045+0000","last_undegraded":"2024-09-13T17:13:42.579045+0000","last_fullsized":"2024-09-13T17:13:42.579045+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:40.379101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587618+0000","last_change":"2024-09-13T17:13:42.586645+0000","last_active":"2024-09-13T17:13:42.587618+0000","last_peered":"2024-09-13T17:13:42.587618+0000","last_clean":"2024-09-13T17:13:42.587618+0000","last_became_active":"2024-09-13T17:13:42.586364+0000","last_became_peered":"2024-09-13T17:13:42.586364+0000","last_unstale":"2024-09-13T17:13:42.587618+0000","last_undegraded":"2024-09-13T17:13:42.587618+0000","last_fullsized":"2024-09-13T17:13:42.587618+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:26.995166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.564003+0000","last_change":"2024-09-13T17:13:42.561894+0000","last_active":"2024-09-13T17:13:42.564003+0000","last_peered":"2024-09-13T17:13:42.564003+0000","last_clean":"2024-09-13T17:13:42.564003+0000","last_became_active":"2024-09-13T17:13:42.561644+0000","last_became_peered":"2024-09-13T17:13:42.561644+0000","last_unstale":"2024-09-13T17:13:42.564003+0000","last_undegraded":"2024-09-13T17:13:42.564003+0000","last_fullsized":"2024-09-13T17:13:42.564003+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586962+0000","last_change":"2024-09-13T17:13:42.582338+0000","last_active":"2024-09-13T17:13:42.586962+0000","last_peered":"2024-09-13T17:13:42.586962+0000","last_clean":"2024-09-13T17:13:42.586962+0000","last_became_active":"2024-09-13T17:13:42.579692+0000","last_became_peered":"2024-09-13T17:13:42.579692+0000","last_unstale":"2024-09-13T17:13:42.586962+0000","last_undegraded":"2024-09-13T17:13:42.586962+0000","last_fullsized":"2024-09-13T17:13:42.586962+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:43.404490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578457+0000","last_change":"2024-09-13T17:13:42.577390+0000","last_active":"2024-09-13T17:13:42.578457+0000","last_peered":"2024-09-13T17:13:42.578457+0000","last_clean":"2024-09-13T17:13:42.578457+0000","last_became_active":"2024-09-13T17:13:42.577043+0000","last_became_peered":"2024-09-13T17:13:42.577043+0000","last_unstale":"2024-09-13T17:13:42.578457+0000","last_undegraded":"2024-09-13T17:13:42.578457+0000","last_fullsized":"2024-09-13T17:13:42.578457+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:38:43.292409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568419+0000","last_change":"2024-09-13T17:13:42.563964+0000","last_active":"2024-09-13T17:13:42.568419+0000","last_peered":"2024-09-13T17:13:42.568419+0000","last_clean":"2024-09-13T17:13:42.568419+0000","last_became_active":"2024-09-13T17:13:42.563105+0000","last_became_peered":"2024-09-13T17:13:42.563105+0000","last_unstale":"2024-09-13T17:13:42.568419+0000","last_undegraded":"2024-09-13T17:13:42.568419+0000","last_fullsized":"2024-09-13T17:13:42.568419+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586810+0000","last_change":"2024-09-13T17:13:42.582234+0000","last_active":"2024-09-13T17:13:42.586810+0000","last_peered":"2024-09-13T17:13:42.586810+0000","last_clean":"2024-09-13T17:13:42.586810+0000","last_became_active":"2024-09-13T17:13:42.581589+0000","last_became_peered":"2024-09-13T17:13:42.581589+0000","last_unstale":"2024-09-13T17:13:42.586810+0000","last_undegraded":"2024-09-13T17:13:42.586810+0000","last_fullsized":"2024-09-13T17:13:42.586810+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:56.122473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.570255+0000","last_change":"2024-09-13T17:13:42.568298+0000","last_active":"2024-09-13T17:13:42.570255+0000","last_peered":"2024-09-13T17:13:42.570255+0000","last_clean":"2024-09-13T17:13:42.570255+0000","last_became_active":"2024-09-13T17:13:42.568019+0000","last_became_peered":"2024-09-13T17:13:42.568019+0000","last_unstale":"2024-09-13T17:13:42.570255+0000","last_undegraded":"2024-09-13T17:13:42.570255+0000","last_fullsized":"2024-09-13T17:13:42.570255+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.810829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568053+0000","last_change":"2024-09-13T17:13:42.564787+0000","last_active":"2024-09-13T17:13:42.568053+0000","last_peered":"2024-09-13T17:13:42.568053+0000","last_clean":"2024-09-13T17:13:42.568053+0000","last_became_active":"2024-09-13T17:13:42.564481+0000","last_became_peered":"2024-09-13T17:13:42.564481+0000","last_unstale":"2024-09-13T17:13:42.568053+0000","last_undegraded":"2024-09-13T17:13:42.568053+0000","last_fullsized":"2024-09-13T17:13:42.568053+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:24.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577923+0000","last_change":"2024-09-13T17:13:42.570151+0000","last_active":"2024-09-13T17:13:42.577923+0000","last_peered":"2024-09-13T17:13:42.577923+0000","last_clean":"2024-09-13T17:13:42.577923+0000","last_became_active":"2024-09-13T17:13:42.569875+0000","last_became_peered":"2024-09-13T17:13:42.569875+0000","last_unstale":"2024-09-13T17:13:42.577923+0000","last_undegraded":"2024-09-13T17:13:42.577923+0000","last_fullsized":"2024-09-13T17:13:42.577923+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584361+0000","last_change":"2024-09-13T17:13:42.582009+0000","last_active":"2024-09-13T17:13:42.584361+0000","last_peered":"2024-09-13T17:13:42.584361+0000","last_clean":"2024-09-13T17:13:42.584361+0000","last_became_active":"2024-09-13T17:13:42.581777+0000","last_became_peered":"2024-09-13T17:13:42.581777+0000","last_unstale":"2024-09-13T17:13:42.584361+0000","last_undegraded":"2024-09-13T17:13:42.584361+0000","last_fullsized":"2024-09-13T17:13:42.584361+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:49.192916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.570821+0000","last_change":"2024-09-13T17:13:42.565044+0000","last_active":"2024-09-13T17:13:42.570821+0000","last_peered":"2024-09-13T17:13:42.570821+0000","last_clean":"2024-09-13T17:13:42.570821+0000","last_became_active":"2024-09-13T17:13:42.564729+0000","last_became_peered":"2024-09-13T17:13:42.564729+0000","last_unstale":"2024-09-13T17:13:42.570821+0000","last_undegraded":"2024-09-13T17:13:42.570821+0000","last_fullsized":"2024-09-13T17:13:42.570821+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572843+0000","last_change":"2024-09-13T17:13:42.566886+0000","last_active":"2024-09-13T17:13:42.572843+0000","last_peered":"2024-09-13T17:13:42.572843+0000","last_clean":"2024-09-13T17:13:42.572843+0000","last_became_active":"2024-09-13T17:13:42.566621+0000","last_became_peered":"2024-09-13T17:13:42.566621+0000","last_unstale":"2024-09-13T17:13:42.572843+0000","last_undegraded":"2024-09-13T17:13:42.572843+0000","last_fullsized":"2024-09-13T17:13:42.572843+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:16.283948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586878+0000","last_change":"2024-09-13T17:13:42.581392+0000","last_active":"2024-09-13T17:13:42.586878+0000","last_peered":"2024-09-13T17:13:42.586878+0000","last_clean":"2024-09-13T17:13:42.586878+0000","last_became_active":"2024-09-13T17:13:42.581128+0000","last_became_peered":"2024-09-13T17:13:42.581128+0000","last_unstale":"2024-09-13T17:13:42.586878+0000","last_undegraded":"2024-09-13T17:13:42.586878+0000","last_fullsized":"2024-09-13T17:13:42.586878+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:15.881460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.565104+0000","last_change":"2024-09-13T17:13:42.561267+0000","last_active":"2024-09-13T17:13:42.565104+0000","last_peered":"2024-09-13T17:13:42.565104+0000","last_clean":"2024-09-13T17:13:42.565104+0000","last_became_active":"2024-09-13T17:13:42.560983+0000","last_became_peered":"2024-09-13T17:13:42.560983+0000","last_unstale":"2024-09-13T17:13:42.565104+0000","last_undegraded":"2024-09-13T17:13:42.565104+0000","last_fullsized":"2024-09-13T17:13:42.565104+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:14.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575831+0000","last_change":"2024-09-13T17:13:42.574786+0000","last_active":"2024-09-13T17:13:42.575831+0000","last_peered":"2024-09-13T17:13:42.575831+0000","last_clean":"2024-09-13T17:13:42.575831+0000","last_became_active":"2024-09-13T17:13:42.573757+0000","last_became_peered":"2024-09-13T17:13:42.573757+0000","last_unstale":"2024-09-13T17:13:42.575831+0000","last_undegraded":"2024-09-13T17:13:42.575831+0000","last_fullsized":"2024-09-13T17:13:42.575831+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:30.869322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580964+0000","last_change":"2024-09-13T17:13:42.579770+0000","last_active":"2024-09-13T17:13:42.580964+0000","last_peered":"2024-09-13T17:13:42.580964+0000","last_clean":"2024-09-13T17:13:42.580964+0000","last_became_active":"2024-09-13T17:13:42.579496+0000","last_became_peered":"2024-09-13T17:13:42.579496+0000","last_unstale":"2024-09-13T17:13:42.580964+0000","last_undegraded":"2024-09-13T17:13:42.580964+0000","last_fullsized":"2024-09-13T17:13:42.580964+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:18.169178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577079+0000","last_change":"2024-09-13T17:13:42.573857+0000","last_active":"2024-09-13T17:13:42.577079+0000","last_peered":"2024-09-13T17:13:42.577079+0000","last_clean":"2024-09-13T17:13:42.577079+0000","last_became_active":"2024-09-13T17:13:42.573622+0000","last_became_peered":"2024-09-13T17:13:42.573622+0000","last_unstale":"2024-09-13T17:13:42.577079+0000","last_undegraded":"2024-09-13T17:13:42.577079+0000","last_fullsized":"2024-09-13T17:13:42.577079+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:44:24.525557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.562666+0000","last_change":"2024-09-13T17:13:42.559351+0000","last_active":"2024-09-13T17:13:42.562666+0000","last_peered":"2024-09-13T17:13:42.562666+0000","last_clean":"2024-09-13T17:13:42.562666+0000","last_became_active":"2024-09-13T17:13:42.559100+0000","last_became_peered":"2024-09-13T17:13:42.559100+0000","last_unstale":"2024-09-13T17:13:42.562666+0000","last_undegraded":"2024-09-13T17:13:42.562666+0000","last_fullsized":"2024-09-13T17:13:42.562666+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:15.878385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578013+0000","last_change":"2024-09-13T17:13:42.571537+0000","last_active":"2024-09-13T17:13:42.578013+0000","last_peered":"2024-09-13T17:13:42.578013+0000","last_clean":"2024-09-13T17:13:42.578013+0000","last_became_active":"2024-09-13T17:13:42.571241+0000","last_became_peered":"2024-09-13T17:13:42.571241+0000","last_unstale":"2024-09-13T17:13:42.578013+0000","last_undegraded":"2024-09-13T17:13:42.578013+0000","last_fullsized":"2024-09-13T17:13:42.578013+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:46:40.654864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573393+0000","last_change":"2024-09-13T17:13:42.563308+0000","last_active":"2024-09-13T17:13:42.573393+0000","last_peered":"2024-09-13T17:13:42.573393+0000","last_clean":"2024-09-13T17:13:42.573393+0000","last_became_active":"2024-09-13T17:13:42.563040+0000","last_became_peered":"2024-09-13T17:13:42.563040+0000","last_unstale":"2024-09-13T17:13:42.573393+0000","last_undegraded":"2024-09-13T17:13:42.573393+0000","last_fullsized":"2024-09-13T17:13:42.573393+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:58.649961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.574798+0000","last_change":"2024-09-13T17:13:42.569096+0000","last_active":"2024-09-13T17:13:42.574798+0000","last_peered":"2024-09-13T17:13:42.574798+0000","last_clean":"2024-09-13T17:13:42.574798+0000","last_became_active":"2024-09-13T17:13:42.568809+0000","last_became_peered":"2024-09-13T17:13:42.568809+0000","last_unstale":"2024-09-13T17:13:42.574798+0000","last_undegraded":"2024-09-13T17:13:42.574798+0000","last_fullsized":"2024-09-13T17:13:42.574798+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:18.197898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.577313+0000","last_change":"2024-09-13T17:13:42.572008+0000","last_active":"2024-09-13T17:13:42.577313+0000","last_peered":"2024-09-13T17:13:42.577313+0000","last_clean":"2024-09-13T17:13:42.577313+0000","last_became_active":"2024-09-13T17:13:42.571758+0000","last_became_peered":"2024-09-13T17:13:42.571758+0000","last_unstale":"2024-09-13T17:13:42.577313+0000","last_undegraded":"2024-09-13T17:13:42.577313+0000","last_fullsized":"2024-09-13T17:13:42.577313+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585569+0000","last_change":"2024-09-13T17:13:42.571312+0000","last_active":"2024-09-13T17:13:42.585569+0000","last_peered":"2024-09-13T17:13:42.585569+0000","last_clean":"2024-09-13T17:13:42.585569+0000","last_became_active":"2024-09-13T17:13:42.565718+0000","last_became_peered":"2024-09-13T17:13:42.565718+0000","last_unstale":"2024-09-13T17:13:42.585569+0000","last_undegraded":"2024-09-13T17:13:42.585569+0000","last_fullsized":"2024-09-13T17:13:42.585569+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:51.271074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587489+0000","last_change":"2024-09-13T17:13:42.586526+0000","last_active":"2024-09-13T17:13:42.587489+0000","last_peered":"2024-09-13T17:13:42.587489+0000","last_clean":"2024-09-13T17:13:42.587489+0000","last_became_active":"2024-09-13T17:13:42.586262+0000","last_became_peered":"2024-09-13T17:13:42.586262+0000","last_unstale":"2024-09-13T17:13:42.587489+0000","last_undegraded":"2024-09-13T17:13:42.587489+0000","last_fullsized":"2024-09-13T17:13:42.587489+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:26:57.553985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580227+0000","last_change":"2024-09-13T17:13:42.578735+0000","last_active":"2024-09-13T17:13:42.580227+0000","last_peered":"2024-09-13T17:13:42.580227+0000","last_clean":"2024-09-13T17:13:42.580227+0000","last_became_active":"2024-09-13T17:13:42.578424+0000","last_became_peered":"2024-09-13T17:13:42.578424+0000","last_unstale":"2024-09-13T17:13:42.580227+0000","last_undegraded":"2024-09-13T17:13:42.580227+0000","last_fullsized":"2024-09-13T17:13:42.580227+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.571086+0000","last_change":"2024-09-13T17:13:42.567591+0000","last_active":"2024-09-13T17:13:42.571086+0000","last_peered":"2024-09-13T17:13:42.571086+0000","last_clean":"2024-09-13T17:13:42.571086+0000","last_became_active":"2024-09-13T17:13:42.564990+0000","last_became_peered":"2024-09-13T17:13:42.564990+0000","last_unstale":"2024-09-13T17:13:42.571086+0000","last_undegraded":"2024-09-13T17:13:42.571086+0000","last_fullsized":"2024-09-13T17:13:42.571086+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:54.017990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576836+0000","last_change":"2024-09-13T17:13:42.570056+0000","last_active":"2024-09-13T17:13:42.576836+0000","last_peered":"2024-09-13T17:13:42.576836+0000","last_clean":"2024-09-13T17:13:42.576836+0000","last_became_active":"2024-09-13T17:13:42.569789+0000","last_became_peered":"2024-09-13T17:13:42.569789+0000","last_unstale":"2024-09-13T17:13:42.576836+0000","last_undegraded":"2024-09-13T17:13:42.576836+0000","last_fullsized":"2024-09-13T17:13:42.576836+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:20.512031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584516+0000","last_change":"2024-09-13T17:13:42.582648+0000","last_active":"2024-09-13T17:13:42.584516+0000","last_peered":"2024-09-13T17:13:42.584516+0000","last_clean":"2024-09-13T17:13:42.584516+0000","last_became_active":"2024-09-13T17:13:42.582367+0000","last_became_peered":"2024-09-13T17:13:42.582367+0000","last_unstale":"2024-09-13T17:13:42.584516+0000","last_undegraded":"2024-09-13T17:13:42.584516+0000","last_fullsized":"2024-09-13T17:13:42.584516+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:56.122473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.588006+0000","last_change":"2024-09-13T17:13:42.586867+0000","last_active":"2024-09-13T17:13:42.588006+0000","last_peered":"2024-09-13T17:13:42.588006+0000","last_clean":"2024-09-13T17:13:42.588006+0000","last_became_active":"2024-09-13T17:13:42.586639+0000","last_became_peered":"2024-09-13T17:13:42.586639+0000","last_unstale":"2024-09-13T17:13:42.588006+0000","last_undegraded":"2024-09-13T17:13:42.588006+0000","last_fullsized":"2024-09-13T17:13:42.588006+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:40.149830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572477+0000","last_change":"2024-09-13T17:13:42.570072+0000","last_active":"2024-09-13T17:13:42.572477+0000","last_peered":"2024-09-13T17:13:42.572477+0000","last_clean":"2024-09-13T17:13:42.572477+0000","last_became_active":"2024-09-13T17:13:42.569722+0000","last_became_peered":"2024-09-13T17:13:42.569722+0000","last_unstale":"2024-09-13T17:13:42.572477+0000","last_undegraded":"2024-09-13T17:13:42.572477+0000","last_fullsized":"2024-09-13T17:13:42.572477+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:46.147942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.581550+0000","last_change":"2024-09-13T17:13:42.579436+0000","last_active":"2024-09-13T17:13:42.581550+0000","last_peered":"2024-09-13T17:13:42.581550+0000","last_clean":"2024-09-13T17:13:42.581550+0000","last_became_active":"2024-09-13T17:13:42.579150+0000","last_became_peered":"2024-09-13T17:13:42.579150+0000","last_unstale":"2024-09-13T17:13:42.581550+0000","last_undegraded":"2024-09-13T17:13:42.581550+0000","last_fullsized":"2024-09-13T17:13:42.581550+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:18.478639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575397+0000","last_change":"2024-09-13T17:13:42.573033+0000","last_active":"2024-09-13T17:13:42.575397+0000","last_peered":"2024-09-13T17:13:42.575397+0000","last_clean":"2024-09-13T17:13:42.575397+0000","last_became_active":"2024-09-13T17:13:42.571818+0000","last_became_peered":"2024-09-13T17:13:42.571818+0000","last_unstale":"2024-09-13T17:13:42.575397+0000","last_undegraded":"2024-09-13T17:13:42.575397+0000","last_fullsized":"2024-09-13T17:13:42.575397+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:31.037415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"92'4","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587235+0000","last_change":"2024-09-13T17:13:42.136454+0000","last_active":"2024-09-13T17:13:42.587235+0000","last_peered":"2024-09-13T17:13:42.587235+0000","last_clean":"2024-09-13T17:13:42.587235+0000","last_became_active":"2024-09-13T17:13:42.136178+0000","last_became_peered":"2024-09-13T17:13:42.136178+0000","last_unstale":"2024-09-13T17:13:42.587235+0000","last_undegraded":"2024-09-13T17:13:42.587235+0000","last_fullsized":"2024-09-13T17:13:42.587235+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:07:39.142667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.579070+0000","last_change":"2024-09-13T17:13:42.573094+0000","last_active":"2024-09-13T17:13:42.579070+0000","last_peered":"2024-09-13T17:13:42.579070+0000","last_clean":"2024-09-13T17:13:42.579070+0000","last_became_active":"2024-09-13T17:13:42.572771+0000","last_became_peered":"2024-09-13T17:13:42.572771+0000","last_unstale":"2024-09-13T17:13:42.579070+0000","last_undegraded":"2024-09-13T17:13:42.579070+0000","last_fullsized":"2024-09-13T17:13:42.579070+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:46.367537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.574532+0000","last_change":"2024-09-13T17:13:42.140396+0000","last_active":"2024-09-13T17:13:42.574532+0000","last_peered":"2024-09-13T17:13:42.574532+0000","last_clean":"2024-09-13T17:13:42.574532+0000","last_became_active":"2024-09-13T17:13:42.139805+0000","last_became_peered":"2024-09-13T17:13:42.139805+0000","last_unstale":"2024-09-13T17:13:42.574532+0000","last_undegraded":"2024-09-13T17:13:42.574532+0000","last_fullsized":"2024-09-13T17:13:42.574532+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.581560+0000","last_change":"2024-09-13T17:13:42.579528+0000","last_active":"2024-09-13T17:13:42.581560+0000","last_peered":"2024-09-13T17:13:42.581560+0000","last_clean":"2024-09-13T17:13:42.581560+0000","last_became_active":"2024-09-13T17:13:42.579206+0000","last_became_peered":"2024-09-13T17:13:42.579206+0000","last_unstale":"2024-09-13T17:13:42.581560+0000","last_undegraded":"2024-09-13T17:13:42.581560+0000","last_fullsized":"2024-09-13T17:13:42.581560+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:22.798945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586023+0000","last_change":"2024-09-13T17:13:42.131756+0000","last_active":"2024-09-13T17:13:42.586023+0000","last_peered":"2024-09-13T17:13:42.586023+0000","last_clean":"2024-09-13T17:13:42.586023+0000","last_became_active":"2024-09-13T17:13:42.131508+0000","last_became_peered":"2024-09-13T17:13:42.131508+0000","last_unstale":"2024-09-13T17:13:42.586023+0000","last_undegraded":"2024-09-13T17:13:42.586023+0000","last_fullsized":"2024-09-13T17:13:42.586023+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:39.371938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573522+0000","last_change":"2024-09-13T17:13:42.568985+0000","last_active":"2024-09-13T17:13:42.573522+0000","last_peered":"2024-09-13T17:13:42.573522+0000","last_clean":"2024-09-13T17:13:42.573522+0000","last_became_active":"2024-09-13T17:13:42.567956+0000","last_became_peered":"2024-09-13T17:13:42.567956+0000","last_unstale":"2024-09-13T17:13:42.573522+0000","last_undegraded":"2024-09-13T17:13:42.573522+0000","last_fullsized":"2024-09-13T17:13:42.573522+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:15.878385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585621+0000","last_change":"2024-09-13T17:13:42.136542+0000","last_active":"2024-09-13T17:13:42.585621+0000","last_peered":"2024-09-13T17:13:42.585621+0000","last_clean":"2024-09-13T17:13:42.585621+0000","last_became_active":"2024-09-13T17:13:42.136253+0000","last_became_peered":"2024-09-13T17:13:42.136253+0000","last_unstale":"2024-09-13T17:13:42.585621+0000","last_undegraded":"2024-09-13T17:13:42.585621+0000","last_fullsized":"2024-09-13T17:13:42.585621+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:51.247223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584658+0000","last_change":"2024-09-13T17:13:42.582758+0000","last_active":"2024-09-13T17:13:42.584658+0000","last_peered":"2024-09-13T17:13:42.584658+0000","last_clean":"2024-09-13T17:13:42.584658+0000","last_became_active":"2024-09-13T17:13:42.580619+0000","last_became_peered":"2024-09-13T17:13:42.580619+0000","last_unstale":"2024-09-13T17:13:42.584658+0000","last_undegraded":"2024-09-13T17:13:42.584658+0000","last_fullsized":"2024-09-13T17:13:42.584658+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:14.800216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578350+0000","last_change":"2024-09-13T17:13:42.133793+0000","last_active":"2024-09-13T17:13:42.578350+0000","last_peered":"2024-09-13T17:13:42.578350+0000","last_clean":"2024-09-13T17:13:42.578350+0000","last_became_active":"2024-09-13T17:13:42.133310+0000","last_became_peered":"2024-09-13T17:13:42.133310+0000","last_unstale":"2024-09-13T17:13:42.578350+0000","last_undegraded":"2024-09-13T17:13:42.578350+0000","last_fullsized":"2024-09-13T17:13:42.578350+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:40.822787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.571032+0000","last_change":"2024-09-13T17:13:42.568607+0000","last_active":"2024-09-13T17:13:42.571032+0000","last_peered":"2024-09-13T17:13:42.571032+0000","last_clean":"2024-09-13T17:13:42.571032+0000","last_became_active":"2024-09-13T17:13:42.565679+0000","last_became_peered":"2024-09-13T17:13:42.565679+0000","last_unstale":"2024-09-13T17:13:42.571032+0000","last_undegraded":"2024-09-13T17:13:42.571032+0000","last_fullsized":"2024-09-13T17:13:42.571032+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572930+0000","last_change":"2024-09-13T17:13:42.140282+0000","last_active":"2024-09-13T17:13:42.572930+0000","last_peered":"2024-09-13T17:13:42.572930+0000","last_clean":"2024-09-13T17:13:42.572930+0000","last_became_active":"2024-09-13T17:13:42.139519+0000","last_became_peered":"2024-09-13T17:13:42.139519+0000","last_unstale":"2024-09-13T17:13:42.572930+0000","last_undegraded":"2024-09-13T17:13:42.572930+0000","last_fullsized":"2024-09-13T17:13:42.572930+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:00.927188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563776+0000","last_change":"2024-09-13T17:13:42.560328+0000","last_active":"2024-09-13T17:13:42.563776+0000","last_peered":"2024-09-13T17:13:42.563776+0000","last_clean":"2024-09-13T17:13:42.563776+0000","last_became_active":"2024-09-13T17:13:42.560032+0000","last_became_peered":"2024-09-13T17:13:42.560032+0000","last_unstale":"2024-09-13T17:13:42.563776+0000","last_undegraded":"2024-09-13T17:13:42.563776+0000","last_fullsized":"2024-09-13T17:13:42.563776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554170+0000","last_change":"2024-09-13T17:13:42.130235+0000","last_active":"2024-09-13T17:13:42.554170+0000","last_peered":"2024-09-13T17:13:42.554170+0000","last_clean":"2024-09-13T17:13:42.554170+0000","last_became_active":"2024-09-13T17:13:42.129992+0000","last_became_peered":"2024-09-13T17:13:42.129992+0000","last_unstale":"2024-09-13T17:13:42.554170+0000","last_undegraded":"2024-09-13T17:13:42.554170+0000","last_fullsized":"2024-09-13T17:13:42.554170+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:21.791782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.556197+0000","last_change":"2024-09-13T17:13:42.554766+0000","last_active":"2024-09-13T17:13:42.556197+0000","last_peered":"2024-09-13T17:13:42.556197+0000","last_clean":"2024-09-13T17:13:42.556197+0000","last_became_active":"2024-09-13T17:13:42.554484+0000","last_became_peered":"2024-09-13T17:13:42.554484+0000","last_unstale":"2024-09-13T17:13:42.556197+0000","last_undegraded":"2024-09-13T17:13:42.556197+0000","last_fullsized":"2024-09-13T17:13:42.556197+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:18.169178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563157+0000","last_change":"2024-09-13T17:13:42.132000+0000","last_active":"2024-09-13T17:13:42.563157+0000","last_peered":"2024-09-13T17:13:42.563157+0000","last_clean":"2024-09-13T17:13:42.563157+0000","last_became_active":"2024-09-13T17:13:42.131745+0000","last_became_peered":"2024-09-13T17:13:42.131745+0000","last_unstale":"2024-09-13T17:13:42.563157+0000","last_undegraded":"2024-09-13T17:13:42.563157+0000","last_fullsized":"2024-09-13T17:13:42.563157+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:21.888375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580947+0000","last_change":"2024-09-13T17:13:42.578419+0000","last_active":"2024-09-13T17:13:42.580947+0000","last_peered":"2024-09-13T17:13:42.580947+0000","last_clean":"2024-09-13T17:13:42.580947+0000","last_became_active":"2024-09-13T17:13:42.578093+0000","last_became_peered":"2024-09-13T17:13:42.578093+0000","last_unstale":"2024-09-13T17:13:42.580947+0000","last_undegraded":"2024-09-13T17:13:42.580947+0000","last_fullsized":"2024-09-13T17:13:42.580947+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:00.274545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561771+0000","last_change":"2024-09-13T17:13:42.140479+0000","last_active":"2024-09-13T17:13:42.561771+0000","last_peered":"2024-09-13T17:13:42.561771+0000","last_clean":"2024-09-13T17:13:42.561771+0000","last_became_active":"2024-09-13T17:13:42.139612+0000","last_became_peered":"2024-09-13T17:13:42.139612+0000","last_unstale":"2024-09-13T17:13:42.561771+0000","last_undegraded":"2024-09-13T17:13:42.561771+0000","last_fullsized":"2024-09-13T17:13:42.561771+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:24.655571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576002+0000","last_change":"2024-09-13T17:13:42.572110+0000","last_active":"2024-09-13T17:13:42.576002+0000","last_peered":"2024-09-13T17:13:42.576002+0000","last_clean":"2024-09-13T17:13:42.576002+0000","last_became_active":"2024-09-13T17:13:42.571804+0000","last_became_peered":"2024-09-13T17:13:42.571804+0000","last_unstale":"2024-09-13T17:13:42.576002+0000","last_undegraded":"2024-09-13T17:13:42.576002+0000","last_fullsized":"2024-09-13T17:13:42.576002+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:32.321344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561845+0000","last_change":"2024-09-13T17:13:42.138368+0000","last_active":"2024-09-13T17:13:42.561845+0000","last_peered":"2024-09-13T17:13:42.561845+0000","last_clean":"2024-09-13T17:13:42.561845+0000","last_became_active":"2024-09-13T17:13:42.138078+0000","last_became_peered":"2024-09-13T17:13:42.138078+0000","last_unstale":"2024-09-13T17:13:42.561845+0000","last_undegraded":"2024-09-13T17:13:42.561845+0000","last_fullsized":"2024-09-13T17:13:42.561845+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:50.620928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.567696+0000","last_change":"2024-09-13T17:13:42.565567+0000","last_active":"2024-09-13T17:13:42.567696+0000","last_peered":"2024-09-13T17:13:42.567696+0000","last_clean":"2024-09-13T17:13:42.567696+0000","last_became_active":"2024-09-13T17:13:42.564955+0000","last_became_peered":"2024-09-13T17:13:42.564955+0000","last_unstale":"2024-09-13T17:13:42.567696+0000","last_undegraded":"2024-09-13T17:13:42.567696+0000","last_fullsized":"2024-09-13T17:13:42.567696+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"92'3","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585384+0000","last_change":"2024-09-13T17:13:42.137728+0000","last_active":"2024-09-13T17:13:42.585384+0000","last_peered":"2024-09-13T17:13:42.585384+0000","last_clean":"2024-09-13T17:13:42.585384+0000","last_became_active":"2024-09-13T17:13:42.137438+0000","last_became_peered":"2024-09-13T17:13:42.137438+0000","last_unstale":"2024-09-13T17:13:42.585384+0000","last_undegraded":"2024-09-13T17:13:42.585384+0000","last_fullsized":"2024-09-13T17:13:42.585384+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:10.962523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572474+0000","last_change":"2024-09-13T17:13:42.569958+0000","last_active":"2024-09-13T17:13:42.572474+0000","last_peered":"2024-09-13T17:13:42.572474+0000","last_clean":"2024-09-13T17:13:42.572474+0000","last_became_active":"2024-09-13T17:13:42.569540+0000","last_became_peered":"2024-09-13T17:13:42.569540+0000","last_unstale":"2024-09-13T17:13:42.572474+0000","last_undegraded":"2024-09-13T17:13:42.572474+0000","last_fullsized":"2024-09-13T17:13:42.572474+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:14.352475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"92'6","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584456+0000","last_change":"2024-09-13T17:13:42.139208+0000","last_active":"2024-09-13T17:13:42.584456+0000","last_peered":"2024-09-13T17:13:42.584456+0000","last_clean":"2024-09-13T17:13:42.584456+0000","last_became_active":"2024-09-13T17:13:42.138134+0000","last_became_peered":"2024-09-13T17:13:42.138134+0000","last_unstale":"2024-09-13T17:13:42.584456+0000","last_undegraded":"2024-09-13T17:13:42.584456+0000","last_fullsized":"2024-09-13T17:13:42.584456+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:27:45.360374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576151+0000","last_change":"2024-09-13T17:13:42.573037+0000","last_active":"2024-09-13T17:13:42.576151+0000","last_peered":"2024-09-13T17:13:42.576151+0000","last_clean":"2024-09-13T17:13:42.576151+0000","last_became_active":"2024-09-13T17:13:42.572779+0000","last_became_peered":"2024-09-13T17:13:42.572779+0000","last_unstale":"2024-09-13T17:13:42.576151+0000","last_undegraded":"2024-09-13T17:13:42.576151+0000","last_fullsized":"2024-09-13T17:13:42.576151+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:16.283948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584581+0000","last_change":"2024-09-13T17:13:42.126098+0000","last_active":"2024-09-13T17:13:42.584581+0000","last_peered":"2024-09-13T17:13:42.584581+0000","last_clean":"2024-09-13T17:13:42.584581+0000","last_became_active":"2024-09-13T17:13:42.125525+0000","last_became_peered":"2024-09-13T17:13:42.125525+0000","last_unstale":"2024-09-13T17:13:42.584581+0000","last_undegraded":"2024-09-13T17:13:42.584581+0000","last_fullsized":"2024-09-13T17:13:42.584581+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:50.263911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572742+0000","last_change":"2024-09-13T17:13:42.565918+0000","last_active":"2024-09-13T17:13:42.572742+0000","last_peered":"2024-09-13T17:13:42.572742+0000","last_clean":"2024-09-13T17:13:42.572742+0000","last_became_active":"2024-09-13T17:13:42.564373+0000","last_became_peered":"2024-09-13T17:13:42.564373+0000","last_unstale":"2024-09-13T17:13:42.572742+0000","last_undegraded":"2024-09-13T17:13:42.572742+0000","last_fullsized":"2024-09-13T17:13:42.572742+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:28.898052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.550015+0000","last_change":"2024-09-13T17:13:42.135482+0000","last_active":"2024-09-13T17:13:42.550015+0000","last_peered":"2024-09-13T17:13:42.550015+0000","last_clean":"2024-09-13T17:13:42.550015+0000","last_became_active":"2024-09-13T17:13:42.132794+0000","last_became_peered":"2024-09-13T17:13:42.132794+0000","last_unstale":"2024-09-13T17:13:42.550015+0000","last_undegraded":"2024-09-13T17:13:42.550015+0000","last_fullsized":"2024-09-13T17:13:42.550015+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:15.276785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.584150+0000","last_change":"2024-09-13T17:13:42.581645+0000","last_active":"2024-09-13T17:13:42.584150+0000","last_peered":"2024-09-13T17:13:42.584150+0000","last_clean":"2024-09-13T17:13:42.584150+0000","last_became_active":"2024-09-13T17:13:42.581320+0000","last_became_peered":"2024-09-13T17:13:42.581320+0000","last_unstale":"2024-09-13T17:13:42.584150+0000","last_undegraded":"2024-09-13T17:13:42.584150+0000","last_fullsized":"2024-09-13T17:13:42.584150+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:33.422248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.553044+0000","last_change":"2024-09-13T17:13:42.135228+0000","last_active":"2024-09-13T17:13:42.553044+0000","last_peered":"2024-09-13T17:13:42.553044+0000","last_clean":"2024-09-13T17:13:42.553044+0000","last_became_active":"2024-09-13T17:13:42.134935+0000","last_became_peered":"2024-09-13T17:13:42.134935+0000","last_unstale":"2024-09-13T17:13:42.553044+0000","last_undegraded":"2024-09-13T17:13:42.553044+0000","last_fullsized":"2024-09-13T17:13:42.553044+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:16.947362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587581+0000","last_change":"2024-09-13T17:13:42.586505+0000","last_active":"2024-09-13T17:13:42.587581+0000","last_peered":"2024-09-13T17:13:42.587581+0000","last_clean":"2024-09-13T17:13:42.587581+0000","last_became_active":"2024-09-13T17:13:42.586229+0000","last_became_peered":"2024-09-13T17:13:42.586229+0000","last_unstale":"2024-09-13T17:13:42.587581+0000","last_undegraded":"2024-09-13T17:13:42.587581+0000","last_fullsized":"2024-09-13T17:13:42.587581+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:09.940621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"92'8","reported_seq":18,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.583826+0000","last_change":"2024-09-13T17:13:42.126042+0000","last_active":"2024-09-13T17:13:42.583826+0000","last_peered":"2024-09-13T17:13:42.583826+0000","last_clean":"2024-09-13T17:13:42.583826+0000","last_became_active":"2024-09-13T17:13:42.125753+0000","last_became_peered":"2024-09-13T17:13:42.125753+0000","last_unstale":"2024-09-13T17:13:42.583826+0000","last_undegraded":"2024-09-13T17:13:42.583826+0000","last_fullsized":"2024-09-13T17:13:42.583826+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:40.822787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573025+0000","last_change":"2024-09-13T17:13:42.565041+0000","last_active":"2024-09-13T17:13:42.573025+0000","last_peered":"2024-09-13T17:13:42.573025+0000","last_clean":"2024-09-13T17:13:42.573025+0000","last_became_active":"2024-09-13T17:13:42.564780+0000","last_became_peered":"2024-09-13T17:13:42.564780+0000","last_unstale":"2024-09-13T17:13:42.573025+0000","last_undegraded":"2024-09-13T17:13:42.573025+0000","last_fullsized":"2024-09-13T17:13:42.573025+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:04.990277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580524+0000","last_change":"2024-09-13T17:13:42.134017+0000","last_active":"2024-09-13T17:13:42.580524+0000","last_peered":"2024-09-13T17:13:42.580524+0000","last_clean":"2024-09-13T17:13:42.580524+0000","last_became_active":"2024-09-13T17:13:42.133741+0000","last_became_peered":"2024-09-13T17:13:42.133741+0000","last_unstale":"2024-09-13T17:13:42.580524+0000","last_undegraded":"2024-09-13T17:13:42.580524+0000","last_fullsized":"2024-09-13T17:13:42.580524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:17.190735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573413+0000","last_change":"2024-09-13T17:13:42.565995+0000","last_active":"2024-09-13T17:13:42.573413+0000","last_peered":"2024-09-13T17:13:42.573413+0000","last_clean":"2024-09-13T17:13:42.573413+0000","last_became_active":"2024-09-13T17:13:42.565717+0000","last_became_peered":"2024-09-13T17:13:42.565717+0000","last_unstale":"2024-09-13T17:13:42.573413+0000","last_undegraded":"2024-09-13T17:13:42.573413+0000","last_fullsized":"2024-09-13T17:13:42.573413+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:06.495250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.583830+0000","last_change":"2024-09-13T17:13:42.135305+0000","last_active":"2024-09-13T17:13:42.583830+0000","last_peered":"2024-09-13T17:13:42.583830+0000","last_clean":"2024-09-13T17:13:42.583830+0000","last_became_active":"2024-09-13T17:13:42.135047+0000","last_became_peered":"2024-09-13T17:13:42.135047+0000","last_unstale":"2024-09-13T17:13:42.583830+0000","last_undegraded":"2024-09-13T17:13:42.583830+0000","last_fullsized":"2024-09-13T17:13:42.583830+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:16:17.471476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580916+0000","last_change":"2024-09-13T17:13:42.579748+0000","last_active":"2024-09-13T17:13:42.580916+0000","last_peered":"2024-09-13T17:13:42.580916+0000","last_clean":"2024-09-13T17:13:42.580916+0000","last_became_active":"2024-09-13T17:13:42.579430+0000","last_became_peered":"2024-09-13T17:13:42.579430+0000","last_unstale":"2024-09-13T17:13:42.580916+0000","last_undegraded":"2024-09-13T17:13:42.580916+0000","last_fullsized":"2024-09-13T17:13:42.580916+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554543+0000","last_change":"2024-09-13T17:13:42.127749+0000","last_active":"2024-09-13T17:13:42.554543+0000","last_peered":"2024-09-13T17:13:42.554543+0000","last_clean":"2024-09-13T17:13:42.554543+0000","last_became_active":"2024-09-13T17:13:42.127277+0000","last_became_peered":"2024-09-13T17:13:42.127277+0000","last_unstale":"2024-09-13T17:13:42.554543+0000","last_undegraded":"2024-09-13T17:13:42.554543+0000","last_fullsized":"2024-09-13T17:13:42.554543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:19.504868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.586857+0000","last_change":"2024-09-13T17:13:42.581085+0000","last_active":"2024-09-13T17:13:42.586857+0000","last_peered":"2024-09-13T17:13:42.586857+0000","last_clean":"2024-09-13T17:13:42.586857+0000","last_became_active":"2024-09-13T17:13:42.579417+0000","last_became_peered":"2024-09-13T17:13:42.579417+0000","last_unstale":"2024-09-13T17:13:42.586857+0000","last_undegraded":"2024-09-13T17:13:42.586857+0000","last_fullsized":"2024-09-13T17:13:42.586857+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:22.895538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580193+0000","last_change":"2024-09-13T17:13:42.132935+0000","last_active":"2024-09-13T17:13:42.580193+0000","last_peered":"2024-09-13T17:13:42.580193+0000","last_clean":"2024-09-13T17:13:42.580193+0000","last_became_active":"2024-09-13T17:13:42.129217+0000","last_became_peered":"2024-09-13T17:13:42.129217+0000","last_unstale":"2024-09-13T17:13:42.580193+0000","last_undegraded":"2024-09-13T17:13:42.580193+0000","last_fullsized":"2024-09-13T17:13:42.580193+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:02.027252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.566173+0000","last_change":"2024-09-13T17:13:42.561180+0000","last_active":"2024-09-13T17:13:42.566173+0000","last_peered":"2024-09-13T17:13:42.566173+0000","last_clean":"2024-09-13T17:13:42.566173+0000","last_became_active":"2024-09-13T17:13:42.560880+0000","last_became_peered":"2024-09-13T17:13:42.560880+0000","last_unstale":"2024-09-13T17:13:42.566173+0000","last_undegraded":"2024-09-13T17:13:42.566173+0000","last_fullsized":"2024-09-13T17:13:42.566173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:00:22.798945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.559119+0000","last_change":"2024-09-13T17:13:42.126581+0000","last_active":"2024-09-13T17:13:42.559119+0000","last_peered":"2024-09-13T17:13:42.559119+0000","last_clean":"2024-09-13T17:13:42.559119+0000","last_became_active":"2024-09-13T17:13:42.126297+0000","last_became_peered":"2024-09-13T17:13:42.126297+0000","last_unstale":"2024-09-13T17:13:42.559119+0000","last_undegraded":"2024-09-13T17:13:42.559119+0000","last_fullsized":"2024-09-13T17:13:42.559119+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:10.889340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576136+0000","last_change":"2024-09-13T17:13:42.562810+0000","last_active":"2024-09-13T17:13:42.576136+0000","last_peered":"2024-09-13T17:13:42.576136+0000","last_clean":"2024-09-13T17:13:42.576136+0000","last_became_active":"2024-09-13T17:13:42.562293+0000","last_became_peered":"2024-09-13T17:13:42.562293+0000","last_unstale":"2024-09-13T17:13:42.576136+0000","last_undegraded":"2024-09-13T17:13:42.576136+0000","last_fullsized":"2024-09-13T17:13:42.576136+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:38.126662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558124+0000","last_change":"2024-09-13T17:13:42.136062+0000","last_active":"2024-09-13T17:13:42.558124+0000","last_peered":"2024-09-13T17:13:42.558124+0000","last_clean":"2024-09-13T17:13:42.558124+0000","last_became_active":"2024-09-13T17:13:42.135770+0000","last_became_peered":"2024-09-13T17:13:42.135770+0000","last_unstale":"2024-09-13T17:13:42.558124+0000","last_undegraded":"2024-09-13T17:13:42.558124+0000","last_fullsized":"2024-09-13T17:13:42.558124+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:23.852299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.568087+0000","last_change":"2024-09-13T17:13:42.565573+0000","last_active":"2024-09-13T17:13:42.568087+0000","last_peered":"2024-09-13T17:13:42.568087+0000","last_clean":"2024-09-13T17:13:42.568087+0000","last_became_active":"2024-09-13T17:13:42.565261+0000","last_became_peered":"2024-09-13T17:13:42.565261+0000","last_unstale":"2024-09-13T17:13:42.568087+0000","last_undegraded":"2024-09-13T17:13:42.568087+0000","last_fullsized":"2024-09-13T17:13:42.568087+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:34.904902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.549848+0000","last_change":"2024-09-13T17:13:42.130872+0000","last_active":"2024-09-13T17:13:42.549848+0000","last_peered":"2024-09-13T17:13:42.549848+0000","last_clean":"2024-09-13T17:13:42.549848+0000","last_became_active":"2024-09-13T17:13:42.130446+0000","last_became_peered":"2024-09-13T17:13:42.130446+0000","last_unstale":"2024-09-13T17:13:42.549848+0000","last_undegraded":"2024-09-13T17:13:42.549848+0000","last_fullsized":"2024-09-13T17:13:42.549848+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:47.565746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573438+0000","last_change":"2024-09-13T17:13:42.570453+0000","last_active":"2024-09-13T17:13:42.573438+0000","last_peered":"2024-09-13T17:13:42.573438+0000","last_clean":"2024-09-13T17:13:42.573438+0000","last_became_active":"2024-09-13T17:13:42.569615+0000","last_became_peered":"2024-09-13T17:13:42.569615+0000","last_unstale":"2024-09-13T17:13:42.573438+0000","last_undegraded":"2024-09-13T17:13:42.573438+0000","last_fullsized":"2024-09-13T17:13:42.573438+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:04.021181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.587095+0000","last_change":"2024-09-13T17:13:42.128559+0000","last_active":"2024-09-13T17:13:42.587095+0000","last_peered":"2024-09-13T17:13:42.587095+0000","last_clean":"2024-09-13T17:13:42.587095+0000","last_became_active":"2024-09-13T17:13:42.128286+0000","last_became_peered":"2024-09-13T17:13:42.128286+0000","last_unstale":"2024-09-13T17:13:42.587095+0000","last_undegraded":"2024-09-13T17:13:42.587095+0000","last_fullsized":"2024-09-13T17:13:42.587095+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:35:25.031522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.575732+0000","last_change":"2024-09-13T17:13:42.570898+0000","last_active":"2024-09-13T17:13:42.575732+0000","last_peered":"2024-09-13T17:13:42.575732+0000","last_clean":"2024-09-13T17:13:42.575732+0000","last_became_active":"2024-09-13T17:13:42.570607+0000","last_became_peered":"2024-09-13T17:13:42.570607+0000","last_unstale":"2024-09-13T17:13:42.575732+0000","last_undegraded":"2024-09-13T17:13:42.575732+0000","last_fullsized":"2024-09-13T17:13:42.575732+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:48.572909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.563516+0000","last_change":"2024-09-13T17:13:42.126651+0000","last_active":"2024-09-13T17:13:42.563516+0000","last_peered":"2024-09-13T17:13:42.563516+0000","last_clean":"2024-09-13T17:13:42.563516+0000","last_became_active":"2024-09-13T17:13:42.126377+0000","last_became_peered":"2024-09-13T17:13:42.126377+0000","last_unstale":"2024-09-13T17:13:42.563516+0000","last_undegraded":"2024-09-13T17:13:42.563516+0000","last_fullsized":"2024-09-13T17:13:42.563516+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:37.119499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.578431+0000","last_change":"2024-09-13T17:13:42.575233+0000","last_active":"2024-09-13T17:13:42.578431+0000","last_peered":"2024-09-13T17:13:42.578431+0000","last_clean":"2024-09-13T17:13:42.578431+0000","last_became_active":"2024-09-13T17:13:42.574862+0000","last_became_peered":"2024-09-13T17:13:42.574862+0000","last_unstale":"2024-09-13T17:13:42.578431+0000","last_undegraded":"2024-09-13T17:13:42.578431+0000","last_fullsized":"2024-09-13T17:13:42.578431+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:13:21.810829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558605+0000","last_change":"2024-09-13T17:13:42.137207+0000","last_active":"2024-09-13T17:13:42.558605+0000","last_peered":"2024-09-13T17:13:42.558605+0000","last_clean":"2024-09-13T17:13:42.558605+0000","last_became_active":"2024-09-13T17:13:42.136326+0000","last_became_peered":"2024-09-13T17:13:42.136326+0000","last_unstale":"2024-09-13T17:13:42.558605+0000","last_undegraded":"2024-09-13T17:13:42.558605+0000","last_fullsized":"2024-09-13T17:13:42.558605+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:13.793053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.572351+0000","last_change":"2024-09-13T17:13:42.569496+0000","last_active":"2024-09-13T17:13:42.572351+0000","last_peered":"2024-09-13T17:13:42.572351+0000","last_clean":"2024-09-13T17:13:42.572351+0000","last_became_active":"2024-09-13T17:13:42.569173+0000","last_became_peered":"2024-09-13T17:13:42.569173+0000","last_unstale":"2024-09-13T17:13:42.572351+0000","last_undegraded":"2024-09-13T17:13:42.572351+0000","last_fullsized":"2024-09-13T17:13:42.572351+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:00.274545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558758+0000","last_change":"2024-09-13T17:13:42.140084+0000","last_active":"2024-09-13T17:13:42.558758+0000","last_peered":"2024-09-13T17:13:42.558758+0000","last_clean":"2024-09-13T17:13:42.558758+0000","last_became_active":"2024-09-13T17:13:42.139742+0000","last_became_peered":"2024-09-13T17:13:42.139742+0000","last_unstale":"2024-09-13T17:13:42.558758+0000","last_undegraded":"2024-09-13T17:13:42.558758+0000","last_fullsized":"2024-09-13T17:13:42.558758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:45:48.185753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.566026+0000","last_change":"2024-09-13T17:13:42.560682+0000","last_active":"2024-09-13T17:13:42.566026+0000","last_peered":"2024-09-13T17:13:42.566026+0000","last_clean":"2024-09-13T17:13:42.566026+0000","last_became_active":"2024-09-13T17:13:42.560408+0000","last_became_peered":"2024-09-13T17:13:42.560408+0000","last_unstale":"2024-09-13T17:13:42.566026+0000","last_undegraded":"2024-09-13T17:13:42.566026+0000","last_fullsized":"2024-09-13T17:13:42.566026+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:40.379101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.554535+0000","last_change":"2024-09-13T17:13:42.128098+0000","last_active":"2024-09-13T17:13:42.554535+0000","last_peered":"2024-09-13T17:13:42.554535+0000","last_clean":"2024-09-13T17:13:42.554535+0000","last_became_active":"2024-09-13T17:13:42.127802+0000","last_became_peered":"2024-09-13T17:13:42.127802+0000","last_unstale":"2024-09-13T17:13:42.554535+0000","last_undegraded":"2024-09-13T17:13:42.554535+0000","last_fullsized":"2024-09-13T17:13:42.554535+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:05.488087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576155+0000","last_change":"2024-09-13T17:13:42.573548+0000","last_active":"2024-09-13T17:13:42.576155+0000","last_peered":"2024-09-13T17:13:42.576155+0000","last_clean":"2024-09-13T17:13:42.576155+0000","last_became_active":"2024-09-13T17:13:42.572378+0000","last_became_peered":"2024-09-13T17:13:42.572378+0000","last_unstale":"2024-09-13T17:13:42.576155+0000","last_undegraded":"2024-09-13T17:13:42.576155+0000","last_fullsized":"2024-09-13T17:13:42.576155+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:03.034415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561697+0000","last_change":"2024-09-13T17:13:42.138432+0000","last_active":"2024-09-13T17:13:42.561697+0000","last_peered":"2024-09-13T17:13:42.561697+0000","last_clean":"2024-09-13T17:13:42.561697+0000","last_became_active":"2024-09-13T17:13:42.138149+0000","last_became_peered":"2024-09-13T17:13:42.138149+0000","last_unstale":"2024-09-13T17:13:42.561697+0000","last_undegraded":"2024-09-13T17:13:42.561697+0000","last_fullsized":"2024-09-13T17:13:42.561697+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:54.525735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.580102+0000","last_change":"2024-09-13T17:13:42.578744+0000","last_active":"2024-09-13T17:13:42.580102+0000","last_peered":"2024-09-13T17:13:42.580102+0000","last_clean":"2024-09-13T17:13:42.580102+0000","last_became_active":"2024-09-13T17:13:42.578486+0000","last_became_peered":"2024-09-13T17:13:42.578486+0000","last_unstale":"2024-09-13T17:13:42.580102+0000","last_undegraded":"2024-09-13T17:13:42.580102+0000","last_fullsized":"2024-09-13T17:13:42.580102+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:24.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.561128+0000","last_change":"2024-09-13T17:13:42.139980+0000","last_active":"2024-09-13T17:13:42.561128+0000","last_peered":"2024-09-13T17:13:42.561128+0000","last_clean":"2024-09-13T17:13:42.561128+0000","last_became_active":"2024-09-13T17:13:42.139627+0000","last_became_peered":"2024-09-13T17:13:42.139627+0000","last_unstale":"2024-09-13T17:13:42.561128+0000","last_undegraded":"2024-09-13T17:13:42.561128+0000","last_fullsized":"2024-09-13T17:13:42.561128+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:32.415085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.573003+0000","last_change":"2024-09-13T17:13:42.566787+0000","last_active":"2024-09-13T17:13:42.573003+0000","last_peered":"2024-09-13T17:13:42.573003+0000","last_clean":"2024-09-13T17:13:42.573003+0000","last_became_active":"2024-09-13T17:13:42.566517+0000","last_became_peered":"2024-09-13T17:13:42.566517+0000","last_unstale":"2024-09-13T17:13:42.573003+0000","last_undegraded":"2024-09-13T17:13:42.573003+0000","last_fullsized":"2024-09-13T17:13:42.573003+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:52.254386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.558579+0000","last_change":"2024-09-13T17:13:42.137717+0000","last_active":"2024-09-13T17:13:42.558579+0000","last_peered":"2024-09-13T17:13:42.558579+0000","last_clean":"2024-09-13T17:13:42.558579+0000","last_became_active":"2024-09-13T17:13:42.137325+0000","last_became_peered":"2024-09-13T17:13:42.137325+0000","last_unstale":"2024-09-13T17:13:42.558579+0000","last_undegraded":"2024-09-13T17:13:42.558579+0000","last_fullsized":"2024-09-13T17:13:42.558579+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:05.488087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.576401+0000","last_change":"2024-09-13T17:13:42.570333+0000","last_active":"2024-09-13T17:13:42.576401+0000","last_peered":"2024-09-13T17:13:42.576401+0000","last_clean":"2024-09-13T17:13:42.576401+0000","last_became_active":"2024-09-13T17:13:42.570055+0000","last_became_peered":"2024-09-13T17:13:42.570055+0000","last_unstale":"2024-09-13T17:13:42.576401+0000","last_undegraded":"2024-09-13T17:13:42.576401+0000","last_fullsized":"2024-09-13T17:13:42.576401+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:42.105693+0000","last_clean_scrub_stamp":"2024-09-13T17:13:42.105693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:11.969686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.585372+0000","last_change":"2024-09-13T17:13:42.135670+0000","last_active":"2024-09-13T17:13:42.585372+0000","last_peered":"2024-09-13T17:13:42.585372+0000","last_clean":"2024-09-13T17:13:42.585372+0000","last_became_active":"2024-09-13T17:13:42.135394+0000","last_became_peered":"2024-09-13T17:13:42.135394+0000","last_unstale":"2024-09-13T17:13:42.585372+0000","last_undegraded":"2024-09-13T17:13:42.585372+0000","last_fullsized":"2024-09-13T17:13:42.585372+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:13:41.098530+0000","last_clean_scrub_stamp":"2024-09-13T17:13:41.098530+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:00.880334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":227,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-13T17:13:42.124978+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:13:42.124978+0000","last_peered":"2024-09-13T17:13:42.124978+0000","last_clean":"2024-09-13T17:13:42.124978+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:13:42.124978+0000","last_undegraded":"2024-09-13T17:13:42.124978+0000","last_fullsized":"2024-09-13T17:13:42.124978+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574920,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":1660,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":1699840,"data_stored":1064405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":58718667},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66000000000000003,"15min":0.64500000000000002},"min":{"1min":0.41399999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.877,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":2.0880000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.67500000000000004,"15min":0.67900000000000005},"min":{"1min":0.41199999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.97999999999999998,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":2.6739999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.66100000000000003,"15min":0.64000000000000001},"min":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.95999999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":2.5030000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.68799999999999994},"min":{"1min":0.51000000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.88,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":6.4909999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.69199999999999995,"15min":0.69999999999999996},"min":{"1min":0.48499999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.84899999999999998,"5min":1.0389999999999999,"15min":1.0389999999999999},"last":1.544},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71299999999999997,"15min":0.71399999999999997},"min":{"1min":0.54300000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93400000000000005,"5min":1.224,"15min":1.224},"last":1.605}]},{"osd":3,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68000000000000005},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.014,"5min":1.071,"15min":1.071},"last":2.4249999999999998},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.75600000000000001,"15min":0.73799999999999999},"min":{"1min":0.63900000000000001,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.998,"5min":1.177,"15min":1.177},"last":2.8239999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.48699999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.78100000000000003,"5min":1.054,"15min":1.054},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67300000000000004,"15min":0.69699999999999995},"min":{"1min":0.42699999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.92900000000000005,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.66500000000000004,"15min":0.69499999999999995},"min":{"1min":0.46700000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77200000000000002,"5min":1.143,"15min":1.143},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.76100000000000001,"15min":0.78100000000000003},"min":{"1min":0.62,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.93600000000000005,"5min":1.337,"15min":1.337},"last":0.71799999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.74299999999999999,"15min":0.76700000000000002},"min":{"1min":0.495,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85699999999999998,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.755,"15min":0.77800000000000002},"min":{"1min":0.60299999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.90600000000000003,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.75}]}]},{"osd":6,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40388,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331452,"statfs":{"total":96636764160,"available":96595406848,"internally_reserved":0,"allocated":1105920,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.69599999999999995,"15min":0.66100000000000003},"min":{"1min":0.44600000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93500000000000005,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.73699999999999999,"15min":0.72499999999999998},"min":{"1min":0.64800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.89800000000000002,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.80900000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.69399999999999995,"15min":0.65200000000000002},"min":{"1min":0.55400000000000005,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94499999999999995,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.437},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.68000000000000005,"15min":0.63900000000000001},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96399999999999997,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.746}]},{"osd":2,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.65000000000000002,"15min":0.66900000000000004},"min":{"1min":0.54000000000000004,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.90400000000000003,"5min":1.2909999999999999,"15min":1.2909999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.71099999999999997,"15min":0.68899999999999995},"min":{"1min":0.56899999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1.012,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.78400000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97799999999999998,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.70999999999999996,"15min":0.70499999999999996},"min":{"1min":0.58399999999999996,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.089,"5min":3.742,"15min":3.742},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.62,"15min":0.627},"min":{"1min":0.49199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.026,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.67700000000000005,"15min":0.65600000000000003},"min":{"1min":0.45900000000000002,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.075,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.67500000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70999999999999996,"15min":0.72799999999999998},"min":{"1min":0.53200000000000003,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0589999999999999,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70099999999999996,"15min":0.72399999999999998},"min":{"1min":0.45900000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.127,"5min":3.609,"15min":3.609},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.76100000000000001,"15min":0.75600000000000001},"min":{"1min":0.497,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1499999999999999,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60099999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.202,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.72599999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574920,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40400,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331440,"statfs":{"total":96636764160,"available":96595394560,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.997,"15min":0.78100000000000003},"min":{"1min":0.51100000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.85999999999999999,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.877,"15min":0.77300000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.92300000000000004,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.64400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.83199999999999996,"15min":0.73799999999999999},"min":{"1min":0.502,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.83499999999999996,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.628},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72399999999999998,"15min":0.70599999999999996},"min":{"1min":0.499,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.91200000000000003,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.55300000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.84999999999999998,"15min":0.76400000000000001},"min":{"1min":0.52500000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78900000000000003,"15min":0.75900000000000001},"min":{"1min":0.55500000000000005,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84499999999999997,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":1.8029999999999999,"15min":1.111},"min":{"1min":0.54000000000000004,"5min":0.435,"15min":0.435},"max":{"1min":0.86099999999999999,"5min":113.363,"15min":113.363},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.77300000000000002},"min":{"1min":0.60399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89100000000000001,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.628,"15min":0.626},"min":{"1min":0.39300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94799999999999995,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.70599999999999996,"15min":0.72599999999999998},"min":{"1min":0.47299999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0089999999999999,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.72099999999999997,"15min":0.70699999999999996},"min":{"1min":0.47399999999999998,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0800000000000001,"5min":2.056,"15min":2.056},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.74199999999999999,"15min":0.72699999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.012,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.86099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:13:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.79200000000000004,"15min":0.76200000000000001},"min":{"1min":0.59399999999999997,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.96599999999999997,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.78600000000000003,"15min":0.76800000000000002},"min":{"1min":0.63200000000000001,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0549999999999999,"5min":2.02,"15min":2.02},"last":0.77300000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574919,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46164,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325676,"statfs":{"total":96636764160,"available":96589492224,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.89900000000000002,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.63},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.94999999999999996,"5min":1.341,"15min":1.341},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.64300000000000002,"15min":0.66100000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94299999999999995,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.64400000000000002,"15min":0.621},"min":{"1min":0.49199999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.92000000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.48299999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.65000000000000002,"15min":0.64700000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.014,"5min":1.014,"15min":1.014},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.70399999999999996,"15min":0.72199999999999998},"min":{"1min":0.47599999999999998,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.995,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.66600000000000004,"15min":0.66400000000000003},"min":{"1min":0.50700000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.94999999999999996,"5min":1.002,"15min":1.002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68999999999999995,"15min":0.69299999999999995},"min":{"1min":0.53500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.62,"15min":0.64500000000000002},"min":{"1min":0.42199999999999999,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.502},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.64300000000000002,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59299999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.65700000000000003,"15min":0.65600000000000003},"min":{"1min":0.47299999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71099999999999997,"15min":0.745},"min":{"1min":0.55300000000000005,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.998,"5min":1.413,"15min":1.413},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:13:08 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.69899999999999995,"15min":0.69199999999999995},"min":{"1min":0.54700000000000004,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0369999999999999,"5min":1.054,"15min":1.054},"last":0.754},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.72399999999999998,"15min":0.72799999999999998},"min":{"1min":0.63,"5min":0.371,"15min":0.371},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.72199999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574920,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50252,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321588,"statfs":{"total":96636764160,"available":96585306112,"internally_reserved":0,"allocated":1105920,"data_stored":474037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.46100000000000002,"15min":0.46300000000000002},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":1.177,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.501,"15min":0.56899999999999995},"min":{"1min":0.33500000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73499999999999999,"5min":2.96,"15min":2.96},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.53700000000000003,"15min":0.52100000000000002},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.91900000000000004,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.65300000000000002,"15min":0.73099999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0309999999999999,"5min":3.6829999999999998,"15min":3.6829999999999998},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.54100000000000004,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.88,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.499},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.58599999999999997,"15min":0.60399999999999998},"min":{"1min":0.39800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.97699999999999998,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.53100000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.76500000000000001,"15min":0.81699999999999995},"min":{"1min":0.51500000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1339999999999999,"5min":2.8610000000000002,"15min":2.8610000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.77700000000000002,"15min":0.78100000000000003},"min":{"1min":0.61499999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.999,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.63400000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0580000000000001,"5min":1.391,"15min":1.391},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.748,"5min":0.79500000000000004,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0489999999999999,"5min":2.9300000000000002,"15min":2.9300000000000002},"last":0.71099999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78600000000000003,"15min":0.76900000000000002},"min":{"1min":0.56299999999999994,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.042,"5min":1.401,"15min":1.401},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.79600000000000004,"15min":0.80000000000000004},"min":{"1min":0.57499999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.71899999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81899999999999995,"15min":0.86499999999999999},"min":{"1min":0.58499999999999996,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0960000000000001,"5min":2.899,"15min":2.899},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.83299999999999996},"min":{"1min":0.64000000000000001,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.0740000000000001,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.81200000000000006}]}]},{"osd":2,"up_from":13,"seq":55834574920,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40408,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331432,"statfs":{"total":96636764160,"available":96595386368,"internally_reserved":0,"allocated":1130496,"data_stored":486650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.27700000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.751,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.45600000000000002,"15min":0.434},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.42399999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":3.073,"15min":1.341},"min":{"1min":0.34100000000000003,"5min":0.307,"15min":0.307},"max":{"1min":0.97399999999999998,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":3.1219999999999999,"15min":1.3560000000000001},"min":{"1min":0.373,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95099999999999996,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.38}]},{"osd":3,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.499,"15min":0.48499999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.90800000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57599999999999996,"15min":0.57299999999999995},"min":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.214,"5min":1.214,"15min":1.214},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.60499999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.379,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.77900000000000003,"15min":0.78600000000000003},"min":{"1min":0.58999999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.5229999999999999,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.78400000000000003,"15min":0.79200000000000004},"min":{"1min":0.59899999999999998,"5min":0.376,"15min":0.376},"max":{"1min":1.302,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.77300000000000002,"15min":0.77700000000000002},"min":{"1min":0.61699999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.2789999999999999,"5min":1.569,"15min":1.569},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.77400000000000002,"15min":0.76600000000000001},"min":{"1min":0.63600000000000001,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.1499999999999999,"5min":1.413,"15min":1.413},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.79500000000000004,"15min":0.81000000000000005},"min":{"1min":0.57399999999999995,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.2490000000000001,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.72599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:13:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78800000000000003,"15min":0.78100000000000003},"min":{"1min":0.66400000000000003,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.121,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.80100000000000005,"15min":0.80600000000000005},"min":{"1min":0.61699999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.369,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50260,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321580,"statfs":{"total":96636764160,"available":96585297920,"internally_reserved":0,"allocated":1118208,"data_stored":480312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.50700000000000001,"15min":0.505},"min":{"1min":0.32200000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.57699999999999996,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57599999999999996,"15min":0.51700000000000002},"min":{"1min":0.39000000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.64900000000000002,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.42299999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.497,"15min":0.47099999999999997},"min":{"1min":0.30599999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67800000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.54700000000000004,"15min":0.56699999999999995},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66600000000000004,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.77900000000000003,"15min":0.69499999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.73699999999999999,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.74199999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78300000000000003,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83599999999999997,"15min":0.81799999999999995},"min":{"1min":0.55400000000000005,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.0720000000000001,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.82699999999999996,"15min":0.82299999999999995},"min":{"1min":0.53000000000000003,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.0649999999999999,"5min":1.78,"15min":1.78},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.79100000000000004,"15min":0.77800000000000002},"min":{"1min":0.55600000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.95299999999999996,"5min":1.6870000000000001,"15min":1.6870000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.84099999999999997,"15min":0.80900000000000005},"min":{"1min":0.63900000000000001,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.0900000000000001,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.63700000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.58599999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0740000000000001,"5min":1.7310000000000001,"15min":1.7310000000000001},"last":0.628},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.85999999999999999,"15min":0.82999999999999996},"min":{"1min":0.57999999999999996,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.083,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:13:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.81499999999999995,"15min":0.83299999999999996},"min":{"1min":0.56699999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1080000000000001,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.84799999999999998,"15min":0.83199999999999996},"min":{"1min":0.60699999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.103,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.66500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834574921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40980,"kb_used_data":1668,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330860,"statfs":{"total":96636764160,"available":96594800640,"internally_reserved":0,"allocated":1708032,"data_stored":1069661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51600000000000001,"15min":0.70199999999999996},"min":{"1min":0.29299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.70599999999999996,"5min":5.5259999999999998,"15min":5.5259999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.53400000000000003,"15min":0.69099999999999995},"min":{"1min":0.309,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.95399999999999996,"5min":5.4889999999999999,"15min":5.4889999999999999},"last":0.44400000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56200000000000006,"15min":0.72599999999999998},"min":{"1min":0.33900000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81100000000000005,"5min":6.0430000000000001,"15min":6.0430000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.57499999999999996,"15min":0.74199999999999999},"min":{"1min":0.375,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.66600000000000004,"5min":4.6760000000000002,"15min":4.6760000000000002},"last":0.34599999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.54700000000000004,"15min":0.68100000000000005},"min":{"1min":0.34300000000000003,"5min":0.314,"15min":0.314},"max":{"1min":0.77400000000000002,"5min":4.7549999999999999,"15min":4.7549999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.61299999999999999,"15min":0.73299999999999998},"min":{"1min":0.39400000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83099999999999996,"5min":4.5940000000000003,"15min":4.5940000000000003},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.80300000000000005,"15min":0.91100000000000003},"min":{"1min":0.55600000000000005,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.139,"5min":4.734,"15min":4.734},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79800000000000004,"15min":0.92600000000000005},"min":{"1min":0.54900000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.1180000000000001,"5min":4.8079999999999998,"15min":4.8079999999999998},"last":0.80700000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.80000000000000004,"15min":0.94299999999999995},"min":{"1min":0.54400000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90500000000000003,"5min":4.944,"15min":4.944},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.81100000000000005,"15min":0.92600000000000005},"min":{"1min":0.56000000000000005,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0880000000000001,"5min":4.9710000000000001,"15min":4.9710000000000001},"last":0.77700000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.85999999999999999,"15min":1.097},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.194,"5min":4.8470000000000004,"15min":4.8470000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.82599999999999996,"15min":0.95199999999999996},"min":{"1min":0.53600000000000003,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.044,"5min":5.0170000000000003,"15min":5.0170000000000003},"last":0.629}]},{"osd":7,"last update":"Fri Sep 13 17:13:05 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.84099999999999997,"15min":0.98799999999999999},"min":{"1min":0.57499999999999996,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.98099999999999998,"5min":5.0540000000000003,"15min":5.0540000000000003},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.82899999999999996,"15min":0.96799999999999997},"min":{"1min":0.57599999999999996,"5min":0.505,"15min":0.505},"max":{"1min":0.97799999999999998,"5min":4.9950000000000001,"15min":4.9950000000000001},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:13:47.170 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:13:47.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:13:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.368+0000 7f9c6b200640 1 Processor -- start 2024-09-13T17:13:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.368+0000 7f9c6b200640 1 -- start start 2024-09-13T17:13:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c6b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c070230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c6b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0722e0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c6b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6c0cb3e0 con 0x7f9c6c0cc4a0 2024-09-13T17:13:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c6b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c072000 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c61400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9c6c0cc880 0x7f9c6c0b06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54836/0 (socket says 172.21.15.28:54836) 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.369+0000 7f9c61400640 1 -- 172.21.15.28:0/1649104042 learned_addr learned my addr 172.21.15.28:0/1649104042 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c6a800640 1 --2- 172.21.15.28:0/1649104042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325332775 0 0) 0x7f9c6c0722e0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c50003520 con 0x7f9c6c0cc880 2024-09-13T17:13:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611688408 0 0) 0x7f9c6c0cb3e0 con 0x7f9c6c0cc4a0 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c6c0722e0 con 0x7f9c6c0cc4a0 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c6a800640 1 -- 172.21.15.28:0/1649104042 >> v1:172.21.15.189:6789/0 conn(0x7f9c6c0cc4a0 legacy=0x7f9c6c0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c6a800640 1 -- 172.21.15.28:0/1649104042 >> v1:172.21.15.28:6789/0 conn(0x7f9c6c0cc880 legacy=0x7f9c6c0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.370+0000 7f9c6a800640 1 -- 172.21.15.28:0/1649104042 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c0cad90 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.371+0000 7f9c6a800640 1 --2- 172.21.15.28:0/1649104042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c070230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009dc0 tx=0x7f9c6402a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d558ca021b2e2c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64007740 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6402d050 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c69400640 1 -- 172.21.15.28:0/1649104042 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c64007aa0 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 -- 172.21.15.28:0/1649104042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 msgr2=0x7f9c6c070230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 --2- 172.21.15.28:0/1649104042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c070230 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9c64009dc0 tx=0x7f9c6402a5b0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 -- 172.21.15.28:0/1649104042 shutdown_connections 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 --2- 172.21.15.28:0/1649104042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c070230 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 -- 172.21.15.28:0/1649104042 >> 172.21.15.28:0/1649104042 conn(0x7f9c6c069e80 msgr2=0x7f9c6c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 -- 172.21.15.28:0/1649104042 shutdown_connections 2024-09-13T17:13:47.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.372+0000 7f9c6b200640 1 -- 172.21.15.28:0/1649104042 wait complete. 2024-09-13T17:13:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.374+0000 7f9c6b200640 1 Processor -- start 2024-09-13T17:13:47.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.374+0000 7f9c6b200640 1 -- start start 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c1f4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c6c0cc4a0 0x7f9c6c1f2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c1f54f0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c1558e0 con 0x7f9c6c0cc4a0 2024-09-13T17:13:47.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6c14ad30 con 0x7f9c6c0b0e00 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.375+0000 7f9c6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44090/0 (socket says 172.21.15.28:44090) 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 -- 172.21.15.28:0/3146017283 learned_addr learned my addr 172.21.15.28:0/3146017283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c61400640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c6c0cc4a0 0x7f9c6c1f2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c69e00640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c1f4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 -- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 msgr2=0x7f9c6c1f4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c1f4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 -- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c6c0cc4a0 msgr2=0x7f9c6c1f2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c6c0cc4a0 0x7f9c6c1f2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 -- 172.21.15.28:0/3146017283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64009a70 con 0x7f9c6c0cc880 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.376+0000 7f9c6a800640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f9c6402aba0 tx=0x7f9c64004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.377+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6403b820 con 0x7f9c6c0cc880 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.377+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6403b980 con 0x7f9c6c0cc880 2024-09-13T17:13:47.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.377+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6403bd10 con 0x7f9c6c0cc880 2024-09-13T17:13:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.377+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6c14b010 con 0x7f9c6c0cc880 2024-09-13T17:13:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.377+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6c14b5d0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.378+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6c0cb870 con 0x7f9c6c0cc880 2024-09-13T17:13:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.379+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9c64031400 con 0x7f9c6c0cc880 2024-09-13T17:13:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.379+0000 7f9c62a00640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 0x7f9c4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.380+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c640aaba0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.380+0000 7f9c69e00640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 0x7f9c4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.381+0000 7f9c69e00640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 0x7f9c4806d0c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c007fd0 tx=0x7f9c5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.384+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9c64075eb0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.630+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9c6c06c190 con 0x7f9c6c0cc880 2024-09-13T17:13:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.634+0000 7f9c62a00640 1 -- 172.21.15.28:0/3146017283 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9c640490e0 con 0x7f9c6c0cc880 2024-09-13T17:13:47.635 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:13:47.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 msgr2=0x7f9c4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 0x7f9c4806d0c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c007fd0 tx=0x7f9c5c009040 comp rx=0 tx=0).stop 2024-09-13T17:13:47.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 msgr2=0x7f9c6c0c8e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f9c6402aba0 tx=0x7f9c64004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 shutdown_connections 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9c4806ac00 0x7f9c4806d0c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9c6c0cc880 0x7f9c6c0c8e90 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9c6c0cc4a0 0x7f9c6c1f2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 --2- 172.21.15.28:0/3146017283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9c6c0b0e00 0x7f9c6c1f4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 >> 172.21.15.28:0/3146017283 conn(0x7f9c6c069e80 msgr2=0x7f9c6c15de70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 shutdown_connections 2024-09-13T17:13:47.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.636+0000 7f9c6b200640 1 -- 172.21.15.28:0/3146017283 wait complete. 2024-09-13T17:13:47.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.807+0000 7fe1a7000640 1 Processor -- start 2024-09-13T17:13:47.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.807+0000 7fe1a7000640 1 -- start start 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 0x7fe1a006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a015e100 con 0x7fe1a0070cc0 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a7000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a015e3e0 con 0x7fe1a015a540 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00adb00 con 0x7fe1a00cabc0 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1a0070cc0 0x7fe1a006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54840/0 (socket says 172.21.15.28:54840) 2024-09-13T17:13:47.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.808+0000 7fe1a5000640 1 -- 172.21.15.28:0/2953950338 learned_addr learned my addr 172.21.15.28:0/2953950338 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:47.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe197e00640 1 --2- 172.21.15.28:0/2953950338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 0x7fe1a006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078953223 0 0) 0x7fe1a015e100 con 0x7fe1a0070cc0 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184003520 con 0x7fe1a0070cc0 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892829613 0 0) 0x7fe1a015e3e0 con 0x7fe1a015a540 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1a015e100 con 0x7fe1a015a540 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe197e00640 1 -- 172.21.15.28:0/2953950338 >> v1:172.21.15.189:6789/0 conn(0x7fe1a015a540 legacy=0x7fe1a015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562948220 0 0) 0x7fe184003520 con 0x7fe1a0070cc0 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe197e00640 1 -- 172.21.15.28:0/2953950338 >> v1:172.21.15.28:6789/0 conn(0x7fe1a0070cc0 legacy=0x7fe1a006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.809+0000 7fe197e00640 1 -- 172.21.15.28:0/2953950338 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a015ef80 con 0x7fe1a00cabc0 2024-09-13T17:13:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.810+0000 7fe197e00640 1 --2- 172.21.15.28:0/2953950338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 0x7fe1a006c430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097d0 tx=0x7fe18802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e63df3c341a8fdac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980019f0 con 0x7fe1a00cabc0 2024-09-13T17:13:47.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18802ab90 con 0x7fe1a00cabc0 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe19f400640 1 -- 172.21.15.28:0/2953950338 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188032440 con 0x7fe1a00cabc0 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 -- 172.21.15.28:0/2953950338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 msgr2=0x7fe1a006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 --2- 172.21.15.28:0/2953950338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 0x7fe1a006c430 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097d0 tx=0x7fe18802a100 comp rx=0 tx=0).stop 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 -- 172.21.15.28:0/2953950338 shutdown_connections 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 --2- 172.21.15.28:0/2953950338 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a00cabc0 0x7fe1a006c430 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 -- 172.21.15.28:0/2953950338 >> 172.21.15.28:0/2953950338 conn(0x7fe1a0069e80 msgr2=0x7fe1a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 -- 172.21.15.28:0/2953950338 shutdown_connections 2024-09-13T17:13:47.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.811+0000 7fe1a7000640 1 -- 172.21.15.28:0/2953950338 wait complete. 2024-09-13T17:13:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.813+0000 7fe1a7000640 1 Processor -- start 2024-09-13T17:13:47.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.813+0000 7fe1a7000640 1 -- start start 2024-09-13T17:13:47.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00cabc0 0x7fe1a0143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015a540 0x7fe1a00aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a01534a0 con 0x7fe1a0070cc0 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0152890 con 0x7fe1a00cabc0 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe1a7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0152b70 con 0x7fe1a015a540 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe19fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe19fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44104/0 (socket says 172.21.15.28:44104) 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.814+0000 7fe19fe00640 1 -- 172.21.15.28:0/3346933738 learned_addr learned my addr 172.21.15.28:0/3346933738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe197e00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00cabc0 0x7fe1a0143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe1a5000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015a540 0x7fe1a00aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 -- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015a540 msgr2=0x7fe1a00aff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015a540 0x7fe1a00aff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 -- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00cabc0 msgr2=0x7fe1a0143e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00cabc0 0x7fe1a0143e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 -- 172.21.15.28:0/3346933738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe188009400 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19fe00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c430 tx=0x7fe19000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.815+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1900052d0 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.816+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190005450 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.816+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190003830 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.816+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a0152e50 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.816+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a0152580 con 0x7fe1a0070cc0 2024-09-13T17:13:47.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.817+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a006c430 con 0x7fe1a0070cc0 2024-09-13T17:13:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.818+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe190003990 con 0x7fe1a0070cc0 2024-09-13T17:13:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.818+0000 7fe19cc00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 0x7fe17c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:47.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.819+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe190088950 con 0x7fe1a0070cc0 2024-09-13T17:13:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.819+0000 7fe197e00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 0x7fe17c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:47.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.820+0000 7fe197e00640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 0x7fe17c06d0f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe188034000 tx=0x7fe18802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:47.823+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe190053a90 con 0x7fe1a0070cc0 2024-09-13T17:13:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.035+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1a0155800 con 0x7fe1a0070cc0 2024-09-13T17:13:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.036+0000 7fe19cc00640 1 -- 172.21.15.28:0/3346933738 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fe190017090 con 0x7fe1a0070cc0 2024-09-13T17:13:48.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:48.037 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3444244429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3444244429},{"type":"v1","addr":"172.21.15.189:6837","nonce":3444244429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6380,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3281700882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3281700882},{"type":"v1","addr":"172.21.15.28:6835","nonce":3281700882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":7075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3898110859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3898110859},{"type":"v1","addr":"172.21.15.189:6836","nonce":3898110859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:13:45.560696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7491":{"gid":7491,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2861751170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2861751170},{"type":"v1","addr":"172.21.15.28:6834","nonce":2861751170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:13:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 78 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.037+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 msgr2=0x7fe17c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.037+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 0x7fe17c06d0f0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fe188034000 tx=0x7fe18802e000 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 msgr2=0x7fe1a0152040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c430 tx=0x7fe19000c8f0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 shutdown_connections 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac30 0x7fe17c06d0f0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015a540 0x7fe1a00aff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00cabc0 0x7fe1a0143e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 --2- 172.21.15.28:0/3346933738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a0070cc0 0x7fe1a0152040 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 >> 172.21.15.28:0/3346933738 conn(0x7fe1a0069e80 msgr2=0x7fe1a00cd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 shutdown_connections 2024-09-13T17:13:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.038+0000 7fe1a7000640 1 -- 172.21.15.28:0/3346933738 wait complete. 2024-09-13T17:13:48.099 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:13:48.098+0000 7fd9ff400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:13:48.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 78, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:13:42.545599+0000', 'modified': '2024-09-13T17:13:45.560696+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7491}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7491': {'gid': 7491, 'name': 'c', 'rank': 0, 'incarnation': 74, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/2861751170', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 2861751170}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 2861751170}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:13:48.457 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:13:48.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:13:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.615+0000 7fc709400640 1 Processor -- start 2024-09-13T17:13:48.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.616+0000 7fc709400640 1 -- start start 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc709400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 0x7fc7040b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc709400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 0x7fc70415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc709400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7040b2090 con 0x7fc70415b0b0 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc709400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7040b2370 con 0x7fc7040b1ba0 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc709400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040b2650 con 0x7fc7040c40a0 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc708a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc70415b0b0 0x7fc70406c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54852/0 (socket says 172.21.15.28:54852) 2024-09-13T17:13:48.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc708a00640 1 -- 172.21.15.28:0/3157365045 learned_addr learned my addr 172.21.15.28:0/3157365045 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc703e00640 1 --2- 172.21.15.28:0/3157365045 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 0x7fc7040b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.617+0000 7fc6fb400640 1 --2- 172.21.15.28:0/3157365045 >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 0x7fc70415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703400640 1 -- 172.21.15.28:0/3157365045 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268838172 0 0) 0x7fc7040b2090 con 0x7fc70415b0b0 2024-09-13T17:13:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703400640 1 -- 172.21.15.28:0/3157365045 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e8003520 con 0x7fc70415b0b0 2024-09-13T17:13:48.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703e00640 1 -- 172.21.15.28:0/3157365045 >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 msgr2=0x7fc70415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703e00640 1 --2- 172.21.15.28:0/3157365045 >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 0x7fc70415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703e00640 1 -- 172.21.15.28:0/3157365045 >> v1:172.21.15.28:6789/0 conn(0x7fc70415b0b0 legacy=0x7fc70406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc703e00640 1 -- 172.21.15.28:0/3157365045 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040b2930 con 0x7fc7040c40a0 2024-09-13T17:13:48.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.618+0000 7fc6fb400640 1 --2- 172.21.15.28:0/3157365045 >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 0x7fc70415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.619+0000 7fc703e00640 1 --2- 172.21.15.28:0/3157365045 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 0x7fc7040b1660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a4ebe6c86364627 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc703400640 1 -- 172.21.15.28:0/3157365045 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4032440 con 0x7fc7040c40a0 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc703400640 1 -- 172.21.15.28:0/3157365045 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f40325a0 con 0x7fc7040c40a0 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc703400640 1 -- 172.21.15.28:0/3157365045 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4032930 con 0x7fc7040c40a0 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 -- 172.21.15.28:0/3157365045 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 msgr2=0x7fc7040b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 --2- 172.21.15.28:0/3157365045 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 0x7fc7040b1660 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc6f40097a0 tx=0x7fc6f402a180 comp rx=0 tx=0).stop 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 -- 172.21.15.28:0/3157365045 shutdown_connections 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 --2- 172.21.15.28:0/3157365045 >> v2:172.21.15.189:3300/0 conn(0x7fc7040b1ba0 0x7fc70415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 --2- 172.21.15.28:0/3157365045 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7040c40a0 0x7fc7040b1660 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 -- 172.21.15.28:0/3157365045 >> 172.21.15.28:0/3157365045 conn(0x7fc704073470 msgr2=0x7fc704069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 -- 172.21.15.28:0/3157365045 shutdown_connections 2024-09-13T17:13:48.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.620+0000 7fc709400640 1 -- 172.21.15.28:0/3157365045 wait complete. 2024-09-13T17:13:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.622+0000 7fc709400640 1 Processor -- start 2024-09-13T17:13:48.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.622+0000 7fc709400640 1 -- start start 2024-09-13T17:13:48.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 0x7fc7040c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 0x7fc7040ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70413fe10 con 0x7fc7040c40a0 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040acd90 con 0x7fc7040b1ba0 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc709400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7040c75d0 con 0x7fc70415b0b0 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc6fb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44114/0 (socket says 172.21.15.28:44114) 2024-09-13T17:13:48.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc6fb400640 1 -- 172.21.15.28:0/3968001358 learned_addr learned my addr 172.21.15.28:0/3968001358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.623+0000 7fc708a00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 0x7fc7040ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc703e00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 0x7fc7040c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 -- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 msgr2=0x7fc7040ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 0x7fc7040ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 -- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 msgr2=0x7fc7040c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 0x7fc7040c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 -- 172.21.15.28:0/3968001358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec009400 con 0x7fc7040c40a0 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc708a00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 0x7fc7040ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc703e00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 0x7fc7040c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc6fb400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e980 tx=0x7fc6ec00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc700c00640 1 -- 172.21.15.28:0/3968001358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec00ca00 con 0x7fc7040c40a0 2024-09-13T17:13:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc700c00640 1 -- 172.21.15.28:0/3968001358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec00cb60 con 0x7fc7040c40a0 2024-09-13T17:13:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc700c00640 1 -- 172.21.15.28:0/3968001358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec013470 con 0x7fc7040c40a0 2024-09-13T17:13:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f4009400 con 0x7fc7040c40a0 2024-09-13T17:13:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.624+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7040c7b80 con 0x7fc7040c40a0 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.626+0000 7fc700c00640 1 -- 172.21.15.28:0/3968001358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6ec0135d0 con 0x7fc7040c40a0 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.627+0000 7fc700c00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 0x7fc6e006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.627+0000 7fc700c00640 1 -- 172.21.15.28:0/3968001358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6ec0896a0 con 0x7fc7040c40a0 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.628+0000 7fc70d9e6740 1 Processor -- start 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.628+0000 7fc70d9e6740 1 -- start start 2024-09-13T17:13:48.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.628+0000 7fc703e00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 0x7fc6e006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 0x7fc6e006d0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4034000 tx=0x7fc6f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d64dd000 0x5634d652cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6506cf0 0x5634d652d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d6448ab0 con 0x5634d64df840 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d6446240 con 0x5634d64dd000 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634d6441b30 con 0x5634d6506cf0 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44130/0 (socket says 172.21.15.28:44130) 2024-09-13T17:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 -- 172.21.15.28:0/2143252783 learned_addr learned my addr 172.21.15.28:0/2143252783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc6fb400640 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d64dd000 0x5634d652cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc708a00640 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6506cf0 0x5634d652d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 -- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6506cf0 msgr2=0x5634d652d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6506cf0 0x5634d652d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 -- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d64dd000 msgr2=0x5634d652cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d64dd000 0x5634d652cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.629+0000 7fc703e00640 1 -- 172.21.15.28:0/2143252783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634d640cbd0 con 0x5634d64df840 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.630+0000 7fc703e00640 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc6f402a1c0 tx=0x7fc6f402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9037bf383a3d8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.630+0000 7fc6f8c00640 1 -- 172.21.15.28:0/2143252783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f403c790 con 0x5634d64df840 2024-09-13T17:13:48.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.630+0000 7fc6f8c00640 1 -- 172.21.15.28:0/2143252783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4042070 con 0x5634d64df840 2024-09-13T17:13:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.630+0000 7fc6f8c00640 1 -- 172.21.15.28:0/2143252783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f403caf0 con 0x5634d64df840 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2143252783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 msgr2=0x5634d64dfc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fc6f402a1c0 tx=0x7fc6f402c000 comp rx=0 tx=0).stop 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2143252783 shutdown_connections 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6506cf0 0x5634d652d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d64dd000 0x5634d652cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2143252783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64dfc20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2143252783 >> 172.21.15.28:0/2143252783 conn(0x5634d64dbf70 msgr2=0x5634d6524420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2143252783 shutdown_connections 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.631+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2143252783 wait complete. 2024-09-13T17:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.632+0000 7fc70d9e6740 1 -- start start 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.632+0000 7fc70d9e6740 1 Processor -- start 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d650a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6508450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d63e3260 con 0x5634d6508990 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d63e2f00 con 0x5634d6506cf0 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634d61a8f20 con 0x5634d64df840 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc70d9e6740 10 client.0 _ll_register_callbacks cb 0x5634d643b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44136/0 (socket says 172.21.15.28:44136) 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 -- 172.21.15.28:0/2909581323 learned_addr learned my addr 172.21.15.28:0/2909581323 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc6fb400640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6508450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc703e00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d650a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 -- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 msgr2=0x5634d650a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d650a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 -- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 msgr2=0x5634d6508450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6508450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc708a00640 1 -- 172.21.15.28:0/2909581323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7040b2930 con 0x5634d6508990 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.633+0000 7fc6fb400640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6508450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc708a00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00f800 tx=0x7fc6fc00fcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e810e66dbd3756 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc6f1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc005de0 con 0x5634d6508990 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6fc00d730 con 0x5634d6508990 2024-09-13T17:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5634d646bcd0 con 0x5634d6508990 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6fc001c00 con 0x5634d6508990 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc70d9e6740 10 client.7611 fetch_fsmap learned FSMap version 78 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5634d64401b0 con 0x5634d6508990 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.634+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5634d646bcd0 con 0x5634d6508990 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc6fc012070 con 0x5634d6508990 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 10 client.7611 fetch_fsmap finished waiting for FSMap version 78 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 10 client.7611 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 20 client.7611 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d64f1ba0 0x5634d64f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 4 client.7611 mds_command: new command op to 7491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 --> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5634d61353c0 con 0x5634d64f1ba0 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc703e00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d64f1ba0 0x5634d64f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc703e00640 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d64f1ba0 0x5634d64f1f80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93c3076c032ecc54 server_cookie=58d46593dd617a7f in_seq=0 out_seq=0 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.635+0000 7fc6f1600640 10 client.7611 ms_handle_connect on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.636+0000 7fc6f1600640 1 -- 172.21.15.28:0/2909581323 <== mds.0 v2:172.21.15.28:6832/2861751170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5634d61353c0 con 0x5634d64f1ba0 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.636+0000 7fc6f1600640 10 client.7611 handle_command_reply: tid=0 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.636+0000 7fc70d9e6740 1 client.7611 shutdown 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.636+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 msgr2=0x5634d65216d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.636+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc00f800 tx=0x7fc6fc00fcc0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 shutdown_connections 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65216d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6508450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d64f1ba0 0x5634d64f1f80 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/2909581323 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d650a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 >> 172.21.15.28:0/2909581323 conn(0x5634d64dd000 msgr2=0x5634d6527910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc6f1600640 0 client.7611 ms_handle_reset on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 shutdown_connections 2024-09-13T17:13:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 1 -- 172.21.15.28:0/2909581323 wait complete. 2024-09-13T17:13:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.637+0000 7fc70d9e6740 20 client.7611 trim_cache size 0 max 16384 2024-09-13T17:13:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.652+0000 7fc70d9e6740 1 Processor -- start 2024-09-13T17:13:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 -- start start 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d64d55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6527ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d6523ff0 con 0x5634d6508990 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d64d5b30 con 0x5634d6506cf0 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.653+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634d6521440 con 0x5634d64df840 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44150/0 (socket says 172.21.15.28:44150) 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 -- 172.21.15.28:0/1993682782 learned_addr learned my addr 172.21.15.28:0/1993682782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc6fb400640 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6527ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc703e00640 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d64d55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 -- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 msgr2=0x5634d64d55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d64d55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 -- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 msgr2=0x5634d6527ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6527ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 -- 172.21.15.28:0/1993682782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634d63e2470 con 0x5634d6508990 2024-09-13T17:13:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.654+0000 7fc708a00640 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009010 tx=0x7fc6fc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9230ebff4b25530 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc6f9600640 1 -- 172.21.15.28:0/1993682782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc00d520 con 0x5634d6508990 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc6f9600640 1 -- 172.21.15.28:0/1993682782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc00d730 con 0x5634d6508990 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc6f9600640 1 -- 172.21.15.28:0/1993682782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc001c10 con 0x5634d6508990 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1993682782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 msgr2=0x5634d65261d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc009010 tx=0x7fc6fc013000 comp rx=0 tx=0).stop 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1993682782 shutdown_connections 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d6508990 0x5634d65261d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6527ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1993682782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d64df840 0x5634d64d55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1993682782 >> 172.21.15.28:0/1993682782 conn(0x5634d64dd000 msgr2=0x5634d65095b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1993682782 shutdown_connections 2024-09-13T17:13:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.655+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1993682782 wait complete. 2024-09-13T17:13:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.656+0000 7fc70d9e6740 1 -- start start 2024-09-13T17:13:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.656+0000 7fc70d9e6740 1 Processor -- start 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6576c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6508990 0x5634d64deb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d64ef7d0 con 0x5634d64df840 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634d65771a0 con 0x5634d6506cf0 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5634d64efc00 con 0x5634d6508990 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.657+0000 7fc70d9e6740 10 client.0 _ll_register_callbacks cb 0x7fc6fc001940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44164/0 (socket says 172.21.15.28:44164) 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 -- 172.21.15.28:0/1151355430 learned_addr learned my addr 172.21.15.28:0/1151355430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc708a00640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6508990 0x5634d64deb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc6fb400640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6576c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 -- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6508990 msgr2=0x5634d64deb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6508990 0x5634d64deb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 -- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 msgr2=0x5634d6576c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6576c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 -- 172.21.15.28:0/1151355430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634d64401b0 con 0x5634d64df840 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc703e00640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4035040 tx=0x7fc6f402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbf65cb3e6a9b606 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc6f1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f403c910 con 0x5634d64df840 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc6f403ca70 con 0x5634d64df840 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.658+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc6f4042070 con 0x5634d64df840 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6f403cc30 con 0x5634d64df840 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 10 client.7617 fetch_fsmap learned FSMap version 78 2024-09-13T17:13:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x5634d6407980 con 0x5634d64df840 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f4042070 con 0x5634d64df840 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc6f405f450 con 0x5634d64df840 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 10 client.7617 fetch_fsmap finished waiting for FSMap version 78 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 10 client.7617 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 20 client.7617 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d65261d0 0x5634d65265b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 4 client.7617 mds_command: new command op to 7491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 --> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5634d61466c0 con 0x5634d65261d0 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.659+0000 7fc6fb400640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d65261d0 0x5634d65265b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc6fb400640 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d65261d0 0x5634d65265b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24ef9feb4c1a241d server_cookie=61a3a7f178408b0f in_seq=0 out_seq=0 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc6f1600640 10 client.7617 ms_handle_connect on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc6f1600640 1 -- 172.21.15.28:0/1151355430 <== mds.0 v2:172.21.15.28:6832/2861751170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x5634d61466c0 con 0x5634d65261d0 2024-09-13T17:13:48.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc6f1600640 10 client.7617 handle_command_reply: tid=0 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 client.7617 shutdown 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 msgr2=0x5634d64ef290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4035040 tx=0x7fc6f402c070 comp rx=0 tx=0).stop 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 shutdown_connections 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x5634d65261d0 0x5634d65265b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5634d6508990 0x5634d64deb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634d6506cf0 0x5634d6576c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc6f1600640 0 client.7617 ms_handle_reset on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 --2- 172.21.15.28:0/1151355430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5634d64df840 0x5634d64ef290 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.660+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 >> 172.21.15.28:0/1151355430 conn(0x5634d64dd000 msgr2=0x5634d6524180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 shutdown_connections 2024-09-13T17:13:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc70d9e6740 1 -- 172.21.15.28:0/1151355430 wait complete. 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc70d9e6740 20 client.7617 trim_cache size 0 max 16384 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7491, 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1122918110000004, 2024-09-13T17:13:48.667 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 78, 2024-09-13T17:13:48.668 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 93, 2024-09-13T17:13:48.668 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 93, 2024-09-13T17:13:48.668 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.240093089 2024-09-13T17:13:48.668 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 msgr2=0x7fc6e006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 0x7fc6e006d0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4034000 tx=0x7fc6f402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 msgr2=0x7fc70413f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec00e980 tx=0x7fc6ec00ee40 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 shutdown_connections 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6e006ac30 0x7fc6e006d0f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc70415b0b0 0x7fc7040ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7040c40a0 0x7fc70413f8d0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 --2- 172.21.15.28:0/3968001358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7040b1ba0 0x7fc7040c95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 >> 172.21.15.28:0/3968001358 conn(0x7fc704073470 msgr2=0x7fc70415b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 shutdown_connections 2024-09-13T17:13:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.661+0000 7fc709400640 1 -- 172.21.15.28:0/3968001358 wait complete. 2024-09-13T17:13:48.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:13:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.832+0000 7f9336200640 1 Processor -- start 2024-09-13T17:13:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.832+0000 7f9336200640 1 -- start start 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f9336200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f9336200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f933006f970 0x7f9330159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f9336200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93300b2040 con 0x7f933006c2d0 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f9336200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933015b0b0 con 0x7f933006f970 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f9336200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9330159ff0 con 0x7f93301592a0 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.833+0000 7f932ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f933006f970 0x7f9330159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f933006f970 0x7f9330159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44178/0 (socket says 172.21.15.28:44178) 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 -- 172.21.15.28:0/3808756001 learned_addr learned my addr 172.21.15.28:0/3808756001 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f933006c2d0 0x7f93300b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54854/0 (socket says 172.21.15.28:54854) 2024-09-13T17:13:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932f400640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 -- 172.21.15.28:0/3808756001 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 msgr2=0x7f933015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 -- 172.21.15.28:0/3808756001 >> v1:172.21.15.28:6789/0 conn(0x7f933006c2d0 legacy=0x7f93300b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.834+0000 7f932ea00640 1 -- 172.21.15.28:0/3808756001 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933015a2d0 con 0x7f933006f970 2024-09-13T17:13:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.835+0000 7f932ea00640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006f970 0x7f9330159680 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f931c009870 tx=0x7f931c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f155b8c02055712f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.835+0000 7f932e000640 1 -- 172.21.15.28:0/3808756001 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c032440 con 0x7f933006f970 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.835+0000 7f932e000640 1 -- 172.21.15.28:0/3808756001 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c0325a0 con 0x7f933006f970 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.835+0000 7f932e000640 1 -- 172.21.15.28:0/3808756001 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c032930 con 0x7f933006f970 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 -- 172.21.15.28:0/3808756001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006f970 msgr2=0x7f9330159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006f970 0x7f9330159680 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f931c009870 tx=0x7f931c02a250 comp rx=0 tx=0).stop 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 -- 172.21.15.28:0/3808756001 shutdown_connections 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006f970 0x7f9330159680 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 --2- 172.21.15.28:0/3808756001 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 -- 172.21.15.28:0/3808756001 >> 172.21.15.28:0/3808756001 conn(0x7f9330069e80 msgr2=0x7f933006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 -- 172.21.15.28:0/3808756001 shutdown_connections 2024-09-13T17:13:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.836+0000 7f9336200640 1 -- 172.21.15.28:0/3808756001 wait complete. 2024-09-13T17:13:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.838+0000 7f9336200640 1 Processor -- start 2024-09-13T17:13:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.838+0000 7f9336200640 1 -- start start 2024-09-13T17:13:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 0x7f9330148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330148f90 con 0x7f933006c2d0 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9330149260 con 0x7f933006f970 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f9336200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9330149540 con 0x7f93301592a0 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44192/0 (socket says 172.21.15.28:44192) 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 -- 172.21.15.28:0/66690826 learned_addr learned my addr 172.21.15.28:0/66690826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932fe00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932ea00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 0x7f9330148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 -- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 msgr2=0x7f933015b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 -- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 msgr2=0x7f9330148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 0x7f9330148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932f400640 1 -- 172.21.15.28:0/66690826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0094d0 con 0x7f933006c2d0 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932ea00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 0x7f9330148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.839+0000 7f932fe00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f932f400640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f932000d5f0 tx=0x7f932000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93200125f0 con 0x7f933006c2d0 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9320012750 con 0x7f933006c2d0 2024-09-13T17:13:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9330149820 con 0x7f933006c2d0 2024-09-13T17:13:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9320012b00 con 0x7f933006c2d0 2024-09-13T17:13:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.840+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9330149b00 con 0x7f933006c2d0 2024-09-13T17:13:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.841+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933015ab70 con 0x7f933006c2d0 2024-09-13T17:13:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.842+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9320012c60 con 0x7f933006c2d0 2024-09-13T17:13:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.843+0000 7f9327400640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 0x7f931006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.843+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9320089840 con 0x7f933006c2d0 2024-09-13T17:13:48.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.843+0000 7f932ea00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 0x7f931006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:48.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.844+0000 7f932ea00640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 0x7f931006d0c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f931c034000 tx=0x7f931c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:48.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:48.847+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9320054a20 con 0x7f933006c2d0 2024-09-13T17:13:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.063+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9330067040 con 0x7f933006c2d0 2024-09-13T17:13:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.063+0000 7f9327400640 1 -- 172.21.15.28:0/66690826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v78) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9320058040 con 0x7f933006c2d0 2024-09-13T17:13:49.064 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:13:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 msgr2=0x7f931006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 0x7f931006d0c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f931c034000 tx=0x7f931c02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:49.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 msgr2=0x7f93300af1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f932000d5f0 tx=0x7f932000dab0 comp rx=0 tx=0).stop 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 shutdown_connections 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f931006ac00 0x7f931006d0c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93301592a0 0x7f933015b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f933006f970 0x7f9330148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 --2- 172.21.15.28:0/66690826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f933006c2d0 0x7f93300af1b0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 >> 172.21.15.28:0/66690826 conn(0x7f9330069e80 msgr2=0x7f933014e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 shutdown_connections 2024-09-13T17:13:49.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:49.065+0000 7f9336200640 1 -- 172.21.15.28:0/66690826 wait complete. 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:13:49.083 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:49.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.083 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:13:49.119 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42352sec preferred_lft 42352sec 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:13:49.120 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:13:49.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.122 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:13:49.215 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:13:49.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.218 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:13:49.253 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:13:49.254 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:13:49.254 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:13:49.254 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:13:49.254 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:13:49.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:13:49.255 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:13:49.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.401 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:13:49.437 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:13:49.437 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:13:49.437 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:49.438 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:13:49.533 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:49.533 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:49.568 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:13:49.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:13:49.568 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:49.643 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:13:49.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:49.773 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:49.773 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:49.800 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:13:49.800 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:13:49.800 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:13:49.800 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:13:49.800 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:13:49.803 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:13:49.803 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:13:49.803 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:13:49.803 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:13:49.803 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:13:49.806 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:13:49.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:49.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.862 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:13:49.908 INFO:teuthology.orchestra.run.smithi028.stdout:client6494 2024-09-13T17:13:49.908 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:49.908 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:49.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:49.984 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-09-13T17:13:50.066 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:13:50.066 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:13:50.066 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:13:50.066 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:50.067 DEBUG:teuthology.orchestra.run.smithi028:> print(os.stat("/home/ubuntu/cephtest/mnt.0/testdir").st_ino) 2024-09-13T17:13:50.067 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:13:50.207 INFO:teuthology.orchestra.run.smithi028.stdout:1099511627776 2024-09-13T17:13:50.393 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:13:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.593+0000 7f4eb6000640 1 Processor -- start 2024-09-13T17:13:50.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.594+0000 7f4eb6000640 1 -- start start 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb6000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb015fb90 0x7f4eb00b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4eb00b21c0 0x7f4eb006b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb6000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00cad20 con 0x7f4eb00b0440 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb6000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb006b8a0 con 0x7f4eb00b21c0 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb6000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb006bb40 con 0x7f4eb015fb90 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eafe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4eb00b21c0 0x7f4eb006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eafe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4eb00b21c0 0x7f4eb006b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44200/0 (socket says 172.21.15.28:44200) 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eafe00640 1 -- 172.21.15.28:0/2446520744 learned_addr learned my addr 172.21.15.28:0/2446520744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.595+0000 7f4eb4c00640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb015fb90 0x7f4eb00b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.596+0000 7f4eafe00640 1 -- 172.21.15.28:0/2446520744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb015fb90 msgr2=0x7f4eb00b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.596+0000 7f4eafe00640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb015fb90 0x7f4eb00b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.596+0000 7f4eafe00640 1 -- 172.21.15.28:0/2446520744 >> v1:172.21.15.28:6789/0 conn(0x7f4eb00b0440 legacy=0x7f4eb0160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.596+0000 7f4eafe00640 1 -- 172.21.15.28:0/2446520744 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb006be20 con 0x7f4eb00b21c0 2024-09-13T17:13:50.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.596+0000 7f4eaf400640 1 -- 172.21.15.28:0/2446520744 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225520557 0 0) 0x7f4eb00cad20 con 0x7f4eb00b0440 2024-09-13T17:13:50.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.597+0000 7f4eafe00640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b21c0 0x7f4eb006b360 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009870 tx=0x7f4e9c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc431496bf24da9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:50.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.597+0000 7f4eaf400640 1 -- 172.21.15.28:0/2446520744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c02ae80 con 0x7f4eb00b21c0 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.597+0000 7f4eaf400640 1 -- 172.21.15.28:0/2446520744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c032440 con 0x7f4eb00b21c0 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.597+0000 7f4eaf400640 1 -- 172.21.15.28:0/2446520744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c0327b0 con 0x7f4eb00b21c0 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 -- 172.21.15.28:0/2446520744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b21c0 msgr2=0x7f4eb006b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b21c0 0x7f4eb006b360 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c009870 tx=0x7f4e9c02a130 comp rx=0 tx=0).stop 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 -- 172.21.15.28:0/2446520744 shutdown_connections 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b21c0 0x7f4eb006b360 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 --2- 172.21.15.28:0/2446520744 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb015fb90 0x7f4eb00b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 -- 172.21.15.28:0/2446520744 >> 172.21.15.28:0/2446520744 conn(0x7f4eb0072e40 msgr2=0x7f4eb0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 -- 172.21.15.28:0/2446520744 shutdown_connections 2024-09-13T17:13:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.598+0000 7f4eb6000640 1 -- 172.21.15.28:0/2446520744 wait complete. 2024-09-13T17:13:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.600+0000 7f4eb6000640 1 Processor -- start 2024-09-13T17:13:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.600+0000 7f4eb6000640 1 -- start start 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.600+0000 7f4eb6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb6000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb00b21c0 0x7f4eb015dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 0x7f4eb00b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb6000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01485c0 con 0x7f4eb00b0440 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb6000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0156aa0 con 0x7f4eb015fb90 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb6000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb015ea00 con 0x7f4eb00b21c0 2024-09-13T17:13:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44216/0 (socket says 172.21.15.28:44216) 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 -- 172.21.15.28:0/4208399104 learned_addr learned my addr 172.21.15.28:0/4208399104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eafe00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb00b21c0 0x7f4eb015dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb5600640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 0x7f4eb00b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 -- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb00b21c0 msgr2=0x7f4eb015dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb00b21c0 0x7f4eb015dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 -- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 msgr2=0x7f4eb00b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 0x7f4eb00b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb4c00640 1 -- 172.21.15.28:0/4208399104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c0094d0 con 0x7f4eb00b0440 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.601+0000 7f4eb5600640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 0x7f4eb00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eb4c00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e870 tx=0x7f4ea400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea40094f0 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400c940 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400ccf0 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb015ece0 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.602+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb0156d40 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.603+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb00008d0 con 0x7f4eb00b0440 2024-09-13T17:13:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.604+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4ea4013470 con 0x7f4eb00b0440 2024-09-13T17:13:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.605+0000 7f4eacc00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 0x7f4e9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.605+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ea4089430 con 0x7f4eb00b0440 2024-09-13T17:13:50.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.606+0000 7f4eafe00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 0x7f4e9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.606+0000 7f4eafe00640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 0x7f4e9006d0c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c034000 tx=0x7f4e9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.609+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4ea4054620 con 0x7f4eb00b0440 2024-09-13T17:13:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.822+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4eb0073a20 con 0x7f4eb00b0440 2024-09-13T17:13:50.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.822+0000 7f4eacc00640 1 -- 172.21.15.28:0/4208399104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f4ea4057c40 con 0x7f4eb00b0440 2024-09-13T17:13:50.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:50.824 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":78,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6374,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3444244429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":3444244429},{"type":"v1","addr":"172.21.15.189:6837","nonce":3444244429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":6380,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3281700882","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3281700882},{"type":"v1","addr":"172.21.15.28:6835","nonce":3281700882}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73},{"gid":7075,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3898110859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3898110859},{"type":"v1","addr":"172.21.15.189:6836","nonce":3898110859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":73}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:13:45.560696+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7491":{"gid":7491,"name":"c","rank":0,"incarnation":74,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/2861751170","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2861751170},{"type":"v1","addr":"172.21.15.28:6834","nonce":2861751170}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:13:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 78 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 msgr2=0x7f4e9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 0x7f4e9006d0c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c034000 tx=0x7f4e9c02e000 comp rx=0 tx=0).stop 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 msgr2=0x7f4eb0148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f4ea400e870 tx=0x7f4ea400ed30 comp rx=0 tx=0).stop 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 shutdown_connections 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e9006ac00 0x7f4e9006d0c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4eb015fb90 0x7f4eb00b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4eb00b21c0 0x7f4eb015dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 --2- 172.21.15.28:0/4208399104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4eb00b0440 0x7f4eb0148080 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 >> 172.21.15.28:0/4208399104 conn(0x7f4eb0072e40 msgr2=0x7f4eb00b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 shutdown_connections 2024-09-13T17:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:50.825+0000 7f4eb6000640 1 -- 172.21.15.28:0/4208399104 wait complete. 2024-09-13T17:13:50.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json dump inode 1099511627776 2024-09-13T17:13:51.015 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:13:50.061139+0000","btime":"2024-09-13T17:13:50.061139+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:13:50.061139+0000","atime":"2024-09-13T17:13:50.061139+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","type":"iauth","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","type":"ifile","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","type":"ixattr","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":6494,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":6494,"want_loner":6494,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:13:51.015 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:13:50.061139+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "iauth", "xlock_by": {} }, "backtrace_version": 2, "btime": "2024-09-13T17:13:50.061139+0000", "change_attr": 0, "client_caps": [ { "client_id": 6494, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-09-13T17:13:50.061139+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "ifile", "xlock_by": {} }, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 6494, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:13:50.061139+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": 6494, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "type": "ixattr", "xlock_by": {} }, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:13:51.016 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:51.016 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:51.053 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:13:51.053 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:13:51.053 DEBUG:teuthology.orchestra.run.smithi028:> import uuid 2024-09-13T17:13:51.053 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:51.053 DEBUG:teuthology.orchestra.run.smithi028:> n = 21 2024-09-13T17:13:51.054 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = False 2024-09-13T17:13:51.054 DEBUG:teuthology.orchestra.run.smithi028:> if 0 > 0: 2024-09-13T17:13:51.054 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = True 2024-09-13T17:13:51.054 DEBUG:teuthology.orchestra.run.smithi028:> path = "/home/ubuntu/cephtest/mnt.0/testdir/file" 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> dpath = os.path.dirname(path) 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> fnameprefix = os.path.basename(path) 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> os.makedirs(dpath, exist_ok=True) 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:51.055 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:13:51.056 DEBUG:teuthology.orchestra.run.smithi028:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-13T17:13:51.056 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:13:51.056 DEBUG:teuthology.orchestra.run.smithi028:> for i in range(n): 2024-09-13T17:13:51.056 DEBUG:teuthology.orchestra.run.smithi028:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-13T17:13:51.056 DEBUG:teuthology.orchestra.run.smithi028:> with open(fpath, '"'"'w'"'"') as f: 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> f.write(f"{i}") 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> f.flush() 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(f.fileno()) 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:13:51.057 DEBUG:teuthology.orchestra.run.smithi028:> os.unlink(fpath) 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> if create_hard_links: 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> for j in range(0): 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-13T17:13:51.058 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:13:51.059 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:13:51.059 DEBUG:teuthology.orchestra.run.smithi028:> finally: 2024-09-13T17:13:51.059 DEBUG:teuthology.orchestra.run.smithi028:> os.close(dirfd) 2024-09-13T17:13:51.059 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:13:51.377 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:51.378 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:51.454 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:13:51.454 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:13:51.530 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:13:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.723+0000 7f07f4200640 1 Processor -- start 2024-09-13T17:13:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.723+0000 7f07f4200640 1 -- start start 2024-09-13T17:13:51.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.724+0000 7f07f4200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f07ec1592a0 0x7f07ec15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.724+0000 7f07f4200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.724+0000 7f07f4200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07ec0b2040 con 0x7f07ec06c2d0 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.724+0000 7f07f4200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07ec15b0b0 con 0x7f07ec1592a0 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.724+0000 7f07f4200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec159ff0 con 0x7f07ec06f970 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f2200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f07ec06c2d0 0x7f07ec0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54864/0 (socket says 172.21.15.28:54864) 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f2200640 1 -- 172.21.15.28:0/2026219611 learned_addr learned my addr 172.21.15.28:0/2026219611 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f1800640 1 --2- 172.21.15.28:0/2026219611 >> v2:172.21.15.189:3300/0 conn(0x7f07ec1592a0 0x7f07ec15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f0e00640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07ebe00640 1 -- 172.21.15.28:0/2026219611 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580163451 0 0) 0x7f07ec0b2040 con 0x7f07ec06c2d0 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07ebe00640 1 -- 172.21.15.28:0/2026219611 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07d4005730 con 0x7f07ec06c2d0 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f1800640 1 -- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 msgr2=0x7f07ec159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f1800640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f1800640 1 -- 172.21.15.28:0/2026219611 >> v1:172.21.15.28:6789/0 conn(0x7f07ec06c2d0 legacy=0x7f07ec0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f1800640 1 -- 172.21.15.28:0/2026219611 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07ec15a2d0 con 0x7f07ec1592a0 2024-09-13T17:13:51.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.725+0000 7f07f0e00640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.727+0000 7f07f1800640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec15ab70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f07e00097a0 tx=0x7f07e002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb2d091ac6b0920e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.727+0000 7f07ebe00640 1 -- 172.21.15.28:0/2026219611 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0032440 con 0x7f07ec1592a0 2024-09-13T17:13:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.727+0000 7f07ebe00640 1 -- 172.21.15.28:0/2026219611 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e00325a0 con 0x7f07ec1592a0 2024-09-13T17:13:51.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.727+0000 7f07ebe00640 1 -- 172.21.15.28:0/2026219611 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e0032930 con 0x7f07ec1592a0 2024-09-13T17:13:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 -- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 msgr2=0x7f07ec15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec15ab70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f07e00097a0 tx=0x7f07e002a180 comp rx=0 tx=0).stop 2024-09-13T17:13:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 -- 172.21.15.28:0/2026219611 shutdown_connections 2024-09-13T17:13:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 --2- 172.21.15.28:0/2026219611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec15ab70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 -- 172.21.15.28:0/2026219611 >> 172.21.15.28:0/2026219611 conn(0x7f07ec069e80 msgr2=0x7f07ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 -- 172.21.15.28:0/2026219611 shutdown_connections 2024-09-13T17:13:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.728+0000 7f07f4200640 1 -- 172.21.15.28:0/2026219611 wait complete. 2024-09-13T17:13:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.729+0000 7f07f4200640 1 Processor -- start 2024-09-13T17:13:51.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.730+0000 7f07f4200640 1 -- start start 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.730+0000 7f07f4200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.730+0000 7f07f4200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec0ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f4200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec1e1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f4200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec143380 con 0x7f07ec06c2d0 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f4200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0ceac0 con 0x7f07ec1592a0 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f4200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec0cf290 con 0x7f07ec06f970 2024-09-13T17:13:51.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44218/0 (socket says 172.21.15.28:44218) 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 -- 172.21.15.28:0/1017731068 learned_addr learned my addr 172.21.15.28:0/1017731068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f0e00640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec0ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f2200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec1e1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 -- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 msgr2=0x7f07ec0ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec0ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 -- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 msgr2=0x7f07ec1e1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec1e1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 -- 172.21.15.28:0/1017731068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8009400 con 0x7f07ec06c2d0 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f0e00640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec0ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f2200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec1e1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.731+0000 7f07f1800640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f07e0035040 tx=0x7f07e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:51.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.732+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e003d8f0 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.732+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07e003da70 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.732+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e0009400 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.732+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07e003de20 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.732+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07ec0cefb0 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.733+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07ec06d010 con 0x7f07ec06c2d0 2024-09-13T17:13:51.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.734+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f07e0039040 con 0x7f07ec06c2d0 2024-09-13T17:13:51.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.735+0000 7f07e9600640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 0x7f07cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:51.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.735+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07e00a9ae0 con 0x7f07ec06c2d0 2024-09-13T17:13:51.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.736+0000 7f07f0e00640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 0x7f07cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:51.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.736+0000 7f07f0e00640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 0x7f07cc06d0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f07ec156af0 tx=0x7f07d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:51.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.740+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f07e0074dd0 con 0x7f07ec06c2d0 2024-09-13T17:13:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.923+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07ec15ab70 con 0x7f07ec06c2d0 2024-09-13T17:13:51.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.924+0000 7f07e9600640 1 -- 172.21.15.28:0/1017731068 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16706 (secure 0 0 0) 0x7f07e00783f0 con 0x7f07ec06c2d0 2024-09-13T17:13:51.925 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:13:51.925 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:13:51.875570+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.16","mappings":[{"from":1,"to":5}]},{"pgid":"13.2a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:13:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 msgr2=0x7f07cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 0x7f07cc06d0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f07ec156af0 tx=0x7f07d8008040 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 msgr2=0x7f07ec145ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f07e0035040 tx=0x7f07e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 shutdown_connections 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f07cc06ac00 0x7f07cc06d0c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07ec1592a0 0x7f07ec1e1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f07ec06f970 0x7f07ec0ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 --2- 172.21.15.28:0/1017731068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f07ec06c2d0 0x7f07ec145ed0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 >> 172.21.15.28:0/1017731068 conn(0x7f07ec069e80 msgr2=0x7f07ec06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.926+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 shutdown_connections 2024-09-13T17:13:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:51.927+0000 7f07f4200640 1 -- 172.21.15.28:0/1017731068 wait complete. 2024-09-13T17:13:51.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:13:40.462897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:13:41.537996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:13:51.944 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:13:52.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.101+0000 7f637b000640 1 Processor -- start 2024-09-13T17:13:52.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.101+0000 7f637b000640 1 -- start start 2024-09-13T17:13:52.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.102+0000 7f637b000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 0x7f637406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.102+0000 7f637b000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 0x7f6374070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.102+0000 7f637b000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637406f920 con 0x7f637406f3f0 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f637b000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637415b290 con 0x7f6374159770 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f637b000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637415b570 con 0x7f637415cd80 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f636b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f637406f3f0 0x7f6374159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54876/0 (socket says 172.21.15.28:54876) 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f636b400640 1 -- 172.21.15.28:0/11072359 learned_addr learned my addr 172.21.15.28:0/11072359 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6373e00640 1 --2- 172.21.15.28:0/11072359 >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 0x7f637406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6379000640 1 --2- 172.21.15.28:0/11072359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 0x7f6374070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6373400640 1 -- 172.21.15.28:0/11072359 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2082819602 0 0) 0x7f637406f920 con 0x7f637406f3f0 2024-09-13T17:13:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6373400640 1 -- 172.21.15.28:0/11072359 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6358003520 con 0x7f637406f3f0 2024-09-13T17:13:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6379000640 1 -- 172.21.15.28:0/11072359 >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 msgr2=0x7f637406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6379000640 1 --2- 172.21.15.28:0/11072359 >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 0x7f637406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6379000640 1 -- 172.21.15.28:0/11072359 >> v1:172.21.15.28:6789/0 conn(0x7f637406f3f0 legacy=0x7f6374159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.103+0000 7f6379000640 1 -- 172.21.15.28:0/11072359 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637415b850 con 0x7f637415cd80 2024-09-13T17:13:52.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.104+0000 7f6373e00640 1 --2- 172.21.15.28:0/11072359 >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 0x7f637406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:52.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f6379000640 1 --2- 172.21.15.28:0/11072359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 0x7f6374070260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f636c009d90 tx=0x7f636c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac583eaff998ecd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f6373400640 1 -- 172.21.15.28:0/11072359 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c031600 con 0x7f637415cd80 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f6373400640 1 -- 172.21.15.28:0/11072359 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c02d050 con 0x7f637415cd80 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f6373400640 1 -- 172.21.15.28:0/11072359 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c031980 con 0x7f637415cd80 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f637b000640 1 -- 172.21.15.28:0/11072359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 msgr2=0x7f6374070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f637b000640 1 --2- 172.21.15.28:0/11072359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 0x7f6374070260 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f636c009d90 tx=0x7f636c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.105+0000 7f637b000640 1 -- 172.21.15.28:0/11072359 shutdown_connections 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.106+0000 7f637b000640 1 --2- 172.21.15.28:0/11072359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f637415cd80 0x7f6374070260 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.106+0000 7f637b000640 1 --2- 172.21.15.28:0/11072359 >> v2:172.21.15.189:3300/0 conn(0x7f6374159770 0x7f637406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.106+0000 7f637b000640 1 -- 172.21.15.28:0/11072359 >> 172.21.15.28:0/11072359 conn(0x7f63740b0270 msgr2=0x7f63740b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.106+0000 7f637b000640 1 -- 172.21.15.28:0/11072359 shutdown_connections 2024-09-13T17:13:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.106+0000 7f637b000640 1 -- 172.21.15.28:0/11072359 wait complete. 2024-09-13T17:13:52.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.107+0000 7f637b000640 1 Processor -- start 2024-09-13T17:13:52.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.107+0000 7f637b000640 1 -- start start 2024-09-13T17:13:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 0x7f637413fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 0x7f63740cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740cbfc0 con 0x7f637415cd80 2024-09-13T17:13:52.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63740b3690 con 0x7f637406f3f0 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.108+0000 7f637b000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63740cb690 con 0x7f6374159770 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44226/0 (socket says 172.21.15.28:44226) 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 -- 172.21.15.28:0/3889478505 learned_addr learned my addr 172.21.15.28:0/3889478505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f636b400640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 0x7f63740cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6373e00640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 0x7f637413fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 -- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 msgr2=0x7f63740cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 0x7f63740cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 -- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 msgr2=0x7f637413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 0x7f637413fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 -- 172.21.15.28:0/3889478505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6364009400 con 0x7f637415cd80 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6373e00640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 0x7f637413fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f636b400640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 0x7f63740cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:52.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.109+0000 7f6379000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f636c009eb0 tx=0x7f636c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.110+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c03c830 con 0x7f637415cd80 2024-09-13T17:13:52.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.110+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c03c990 con 0x7f637415cd80 2024-09-13T17:13:52.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.110+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c03cd20 con 0x7f637415cd80 2024-09-13T17:13:52.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.110+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c009a70 con 0x7f637415cd80 2024-09-13T17:13:52.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.110+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63740cbc50 con 0x7f637415cd80 2024-09-13T17:13:52.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.111+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63740cbde0 con 0x7f637415cd80 2024-09-13T17:13:52.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.112+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f636c03b3b0 con 0x7f637415cd80 2024-09-13T17:13:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.113+0000 7f6370c00640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 0x7f635006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.113+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f636c0aae10 con 0x7f637415cd80 2024-09-13T17:13:52.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.114+0000 7f6373e00640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 0x7f635006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.114+0000 7f6373e00640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 0x7f635006d0c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6364009c80 tx=0x7f6364008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.117+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f636c0ae030 con 0x7f637415cd80 2024-09-13T17:13:52.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.293+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6374156800 con 0x7f637415cd80 2024-09-13T17:13:52.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.293+0000 7f6370c00640 1 -- 172.21.15.28:0/3889478505 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v94) v1 ==== 93+0+10 (secure 0 0 0) 0x7f636c079980 con 0x7f637415cd80 2024-09-13T17:13:52.294 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 msgr2=0x7f635006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 0x7f635006d0c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6364009c80 tx=0x7f6364008040 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 msgr2=0x7f63740cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f636c009eb0 tx=0x7f636c004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 shutdown_connections 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f635006ac00 0x7f635006d0c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f637415cd80 0x7f63740cc990 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6374159770 0x7f63740cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 --2- 172.21.15.28:0/3889478505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f637406f3f0 0x7f637413fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.295+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 >> 172.21.15.28:0/3889478505 conn(0x7f63740b0270 msgr2=0x7f63740ae270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.296+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 shutdown_connections 2024-09-13T17:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.296+0000 7f637b000640 1 -- 172.21.15.28:0/3889478505 wait complete. 2024-09-13T17:13:52.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.469+0000 7f90db400640 1 Processor -- start 2024-09-13T17:13:52.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.470+0000 7f90db400640 1 -- start start 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90db400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90db400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406c250 0x7f90d40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90db400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d406c8b0 con 0x7f90d415ab60 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90db400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d406c6a0 con 0x7f90d406c250 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90db400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d406cb90 con 0x7f90d415a780 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406c250 0x7f90d40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f90d406c250 0x7f90d40712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44228/0 (socket says 172.21.15.28:44228) 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.471+0000 7f90d9400640 1 -- 172.21.15.28:0/2995237571 learned_addr learned my addr 172.21.15.28:0/2995237571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d8a00640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d9400640 1 -- 172.21.15.28:0/2995237571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 msgr2=0x7f90d4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d9400640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d9400640 1 -- 172.21.15.28:0/2995237571 >> v1:172.21.15.189:6789/0 conn(0x7f90d415ab60 legacy=0x7f90d406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d9400640 1 -- 172.21.15.28:0/2995237571 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d40cc980 con 0x7f90d406c250 2024-09-13T17:13:52.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.472+0000 7f90d8a00640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90d9400640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406c250 0x7f90d40712c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a44b5121084c0c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90d3400640 1 -- 172.21.15.28:0/2995237571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc02aea0 con 0x7f90d406c250 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90d3400640 1 -- 172.21.15.28:0/2995237571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc02d050 con 0x7f90d406c250 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90d3400640 1 -- 172.21.15.28:0/2995237571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc031820 con 0x7f90d406c250 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90db400640 1 -- 172.21.15.28:0/2995237571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406c250 msgr2=0x7f90d40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.473+0000 7f90db400640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406c250 0x7f90d40712c0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f90cc009e50 tx=0x7f90cc02a6c0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 -- 172.21.15.28:0/2995237571 shutdown_connections 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d406c250 0x7f90d40712c0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 --2- 172.21.15.28:0/2995237571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 -- 172.21.15.28:0/2995237571 >> 172.21.15.28:0/2995237571 conn(0x7f90d40b0270 msgr2=0x7f90d40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 -- 172.21.15.28:0/2995237571 shutdown_connections 2024-09-13T17:13:52.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.474+0000 7f90db400640 1 -- 172.21.15.28:0/2995237571 wait complete. 2024-09-13T17:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.475+0000 7f90db400640 1 Processor -- start 2024-09-13T17:13:52.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.475+0000 7f90db400640 1 -- start start 2024-09-13T17:13:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 0x7f90d40b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d41516a0 con 0x7f90d415ab60 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d40b1e60 con 0x7f90d406c250 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.476+0000 7f90db400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d40b3470 con 0x7f90d415a780 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44238/0 (socket says 172.21.15.28:44238) 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 -- 172.21.15.28:0/3477436916 learned_addr learned my addr 172.21.15.28:0/3477436916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d8a00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 0x7f90d40b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d3e00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 -- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 msgr2=0x7f90d40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 -- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 msgr2=0x7f90d40b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 0x7f90d40b22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 -- 172.21.15.28:0/3477436916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90cc009b30 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d8a00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 0x7f90d40b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d3e00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.477+0000 7f90d9400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f90cc034040 tx=0x7f90cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.478+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc03d860 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.478+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90cc03d9c0 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.478+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90cc03dd70 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.478+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d40b1810 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.478+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d40c3570 con 0x7f90d415ab60 2024-09-13T17:13:52.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.479+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d40b1670 con 0x7f90d415ab60 2024-09-13T17:13:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.480+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f90cc03b400 con 0x7f90d415ab60 2024-09-13T17:13:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.481+0000 7f90d0c00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 0x7f90b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.481+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f90cc0aa320 con 0x7f90d415ab60 2024-09-13T17:13:52.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.481+0000 7f90d8a00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 0x7f90b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.482+0000 7f90d8a00640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 0x7f90b406d0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f90c800aa30 tx=0x7f90c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.485+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f90d40b1670 con 0x7f90d415ab60 2024-09-13T17:13:52.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.665+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90d40b1670 con 0x7f90d415ab60 2024-09-13T17:13:52.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.666+0000 7f90d0c00640 1 -- 172.21.15.28:0/3477436916 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v94) v1 ==== 104+0+11 (secure 0 0 0) 0x7f90d40b1670 con 0x7f90d415ab60 2024-09-13T17:13:52.667 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:13:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 msgr2=0x7f90b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 0x7f90b406d0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f90c800aa30 tx=0x7f90c8008040 comp rx=0 tx=0).stop 2024-09-13T17:13:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 msgr2=0x7f90d40acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f90cc034040 tx=0x7f90cc004a40 comp rx=0 tx=0).stop 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 shutdown_connections 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f90b406ac00 0x7f90b406d0c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f90d415ab60 0x7f90d40acd90 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f90d415a780 0x7f90d40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 --2- 172.21.15.28:0/3477436916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f90d406c250 0x7f90d40b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 >> 172.21.15.28:0/3477436916 conn(0x7f90d40b0270 msgr2=0x7f90d40af880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 shutdown_connections 2024-09-13T17:13:52.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.668+0000 7f90db400640 1 -- 172.21.15.28:0/3477436916 wait complete. 2024-09-13T17:13:52.685 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:13:52.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.842+0000 7fdfaa800640 1 Processor -- start 2024-09-13T17:13:52.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.842+0000 7fdfaa800640 1 -- start start 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfaa800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdfa406f3f0 0x7fdfa415c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfaa800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa415ab60 con 0x7fdfa4159770 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfaa800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa406f920 con 0x7fdfa4070430 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfaa800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa415b290 con 0x7fdfa406f3f0 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfa3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdfa4159770 0x7fdfa406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54886/0 (socket says 172.21.15.28:54886) 2024-09-13T17:13:52.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.843+0000 7fdfa3400640 1 -- 172.21.15.28:0/2254152777 learned_addr learned my addr 172.21.15.28:0/2254152777 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa3e00640 1 --2- 172.21.15.28:0/2254152777 >> v2:172.21.15.189:3300/0 conn(0x7fdfa406f3f0 0x7fdfa415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478383469 0 0) 0x7fdf94001110 con 0x7fdfa4159770 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf88003520 con 0x7fdfa4159770 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225824072 0 0) 0x7fdfa406f920 con 0x7fdfa4070430 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf94001110 con 0x7fdfa4070430 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859997887 0 0) 0x7fdf88003520 con 0x7fdfa4159770 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 >> v2:172.21.15.189:3300/0 conn(0x7fdfa406f3f0 msgr2=0x7fdfa415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 --2- 172.21.15.28:0/2254152777 >> v2:172.21.15.189:3300/0 conn(0x7fdfa406f3f0 0x7fdfa415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 >> v1:172.21.15.189:6789/0 conn(0x7fdfa4070430 legacy=0x7fdfa415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa415b570 con 0x7fdfa4159770 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fdf94001800 con 0x7fdfa4159770 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 >> v1:172.21.15.28:6789/0 conn(0x7fdfa4159770 legacy=0x7fdfa406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdf88003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdf88005180 0x7fdf8800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa415b290 con 0x7fdf88003b00 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800e2c0 con 0x7fdf880047c0 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8800e480 con 0x7fdf88005180 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.844+0000 7fdfa3400640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdf88003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3e00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdf88005180 0x7fdf8800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3400640 1 -- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdf88005180 msgr2=0x7fdf8800db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3400640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdf88005180 0x7fdf8800db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3400640 1 -- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf880047c0 msgr2=0x7fdf88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3400640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3400640 1 -- 172.21.15.28:0/2254152777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88003520 con 0x7fdf88003b00 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdfa3e00640 1 -- 172.21.15.28:0/2254152777 reap_dead start 2024-09-13T17:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.845+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdf880047c0 0x7fdf88004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:52.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.846+0000 7fdfa3400640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdf88003f00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400c160 tx=0x7fdf880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6eb843b94bed39f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.846+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94005ad0 con 0x7fdf88003b00 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.846+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9402d070 con 0x7fdf88003b00 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.846+0000 7fdfa2a00640 1 -- 172.21.15.28:0/2254152777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94007070 con 0x7fdf88003b00 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 -- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 msgr2=0x7fdf88003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdf88003f00 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fdf9400c160 tx=0x7fdf880057f0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 -- 172.21.15.28:0/2254152777 shutdown_connections 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 --2- 172.21.15.28:0/2254152777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdf88003f00 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 -- 172.21.15.28:0/2254152777 >> 172.21.15.28:0/2254152777 conn(0x7fdfa40b0270 msgr2=0x7fdfa40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 -- 172.21.15.28:0/2254152777 shutdown_connections 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.847+0000 7fdfaa800640 1 -- 172.21.15.28:0/2254152777 wait complete. 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.849+0000 7fdfaa800640 1 Processor -- start 2024-09-13T17:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.849+0000 7fdfaa800640 1 -- start start 2024-09-13T17:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 0x7fdfa40c3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 0x7fdfa414a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa4156320 con 0x7fdf88003b00 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40c3ea0 con 0x7fdfa414a210 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfaa800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa40c40f0 con 0x7fdfa40c34e0 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44264/0 (socket says 172.21.15.28:44264) 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 -- 172.21.15.28:0/674423034 learned_addr learned my addr 172.21.15.28:0/674423034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 0x7fdfa40c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 -- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 msgr2=0x7fdfa40c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3e00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 0x7fdfa414a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 0x7fdfa40c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 -- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 msgr2=0x7fdfa414a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 0x7fdfa414a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3400640 1 -- 172.21.15.28:0/674423034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa40c43d0 con 0x7fdf88003b00 2024-09-13T17:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 0x7fdfa40c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.850+0000 7fdfa3e00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 0x7fdfa414a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdfa3400640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402ac90 tx=0x7fdf94001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94004ae0 con 0x7fdf88003b00 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf94004c40 con 0x7fdf88003b00 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa40c46b0 con 0x7fdf88003b00 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf94007070 con 0x7fdf88003b00 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.851+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa414ab90 con 0x7fdf88003b00 2024-09-13T17:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.852+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa406f920 con 0x7fdf88003b00 2024-09-13T17:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.853+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdf9402f030 con 0x7fdf88003b00 2024-09-13T17:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.854+0000 7fdf9be00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 0x7fdf8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.854+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fdf940a87b0 con 0x7fdf88003b00 2024-09-13T17:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.855+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 0x7fdf8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.855+0000 7fdf9aa00640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 0x7fdf8006d0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4071b00 tx=0x7fdf8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:52.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:52.858+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdf94073890 con 0x7fdf88003b00 2024-09-13T17:13:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.044+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdfa415aae0 con 0x7fdf88003b00 2024-09-13T17:13:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.044+0000 7fdf9be00640 1 -- 172.21.15.28:0/674423034 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdf94076eb0 con 0x7fdf88003b00 2024-09-13T17:13:53.045 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.046+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 msgr2=0x7fdf8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.046+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 0x7fdf8006d0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4071b00 tx=0x7fdf8c009040 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.046+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 msgr2=0x7fdfa40acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.046+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fdf9402ac90 tx=0x7fdf94001b10 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 shutdown_connections 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdf8006ac00 0x7fdf8006d0c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfa414a210 0x7fdfa414a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdfa40c34e0 0x7fdfa40c3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 --2- 172.21.15.28:0/674423034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdf88003b00 0x7fdfa40acac0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 >> 172.21.15.28:0/674423034 conn(0x7fdfa40b0270 msgr2=0x7fdfa4159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 shutdown_connections 2024-09-13T17:13:53.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.047+0000 7fdfaa800640 1 -- 172.21.15.28:0/674423034 wait complete. 2024-09-13T17:13:53.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:13:53.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.218+0000 7f1eacc00640 1 Processor -- start 2024-09-13T17:13:53.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.219+0000 7f1eacc00640 1 -- start start 2024-09-13T17:13:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1eacc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ea815ab70 0x7f1ea815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1eacc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ea806c370 0x7f1ea806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1eacc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ea80b2090 con 0x7f1ea80c40a0 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1eacc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea806cd20 con 0x7f1ea815ab70 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1eacc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ea80b2370 con 0x7f1ea806c370 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1e9ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ea815ab70 0x7f1ea815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1e9ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1ea815ab70 0x7f1ea815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44266/0 (socket says 172.21.15.28:44266) 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1e9ea00640 1 -- 172.21.15.28:0/3384017584 learned_addr learned my addr 172.21.15.28:0/3384017584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.220+0000 7f1ea7e00640 1 --2- 172.21.15.28:0/3384017584 >> v2:172.21.15.189:3300/0 conn(0x7f1ea806c370 0x7f1ea806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.221+0000 7f1e9ea00640 1 -- 172.21.15.28:0/3384017584 >> v2:172.21.15.189:3300/0 conn(0x7f1ea806c370 msgr2=0x7f1ea806c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.221+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/3384017584 >> v2:172.21.15.189:3300/0 conn(0x7f1ea806c370 0x7f1ea806c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.221+0000 7f1e9ea00640 1 -- 172.21.15.28:0/3384017584 >> v1:172.21.15.189:6789/0 conn(0x7f1ea80c40a0 legacy=0x7f1ea80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.221+0000 7f1e9ea00640 1 -- 172.21.15.28:0/3384017584 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea80b2650 con 0x7f1ea815ab70 2024-09-13T17:13:53.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/3384017584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea815ab70 0x7f1ea815afb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806f220 tx=0x7f1e9002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d7910df18a2b5ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1ea6a00640 1 -- 172.21.15.28:0/3384017584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90032440 con 0x7f1ea815ab70 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1ea6a00640 1 -- 172.21.15.28:0/3384017584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e900325a0 con 0x7f1ea815ab70 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1ea6a00640 1 -- 172.21.15.28:0/3384017584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e90032930 con 0x7f1ea815ab70 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1eacc00640 1 -- 172.21.15.28:0/3384017584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea815ab70 msgr2=0x7f1ea815afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.222+0000 7f1eacc00640 1 --2- 172.21.15.28:0/3384017584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea815ab70 0x7f1ea815afb0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f1ea806f220 tx=0x7f1e9002a230 comp rx=0 tx=0).stop 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 -- 172.21.15.28:0/3384017584 shutdown_connections 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 --2- 172.21.15.28:0/3384017584 >> v2:172.21.15.189:3300/0 conn(0x7f1ea806c370 0x7f1ea806c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 --2- 172.21.15.28:0/3384017584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea815ab70 0x7f1ea815afb0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 -- 172.21.15.28:0/3384017584 >> 172.21.15.28:0/3384017584 conn(0x7f1ea8073470 msgr2=0x7f1ea8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 -- 172.21.15.28:0/3384017584 shutdown_connections 2024-09-13T17:13:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.223+0000 7f1eacc00640 1 -- 172.21.15.28:0/3384017584 wait complete. 2024-09-13T17:13:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.224+0000 7f1eacc00640 1 Processor -- start 2024-09-13T17:13:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.224+0000 7f1eacc00640 1 -- start start 2024-09-13T17:13:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 0x7f1ea8140b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 0x7f1ea81f2170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81409d0 con 0x7f1ea806c370 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea80ccfb0 con 0x7f1ea80c40a0 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1eacc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea81413e0 con 0x7f1ea815ab70 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.225+0000 7f1ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44274/0 (socket says 172.21.15.28:44274) 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 -- 172.21.15.28:0/1267820098 learned_addr learned my addr 172.21.15.28:0/1267820098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 0x7f1ea8140b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7e00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 0x7f1ea81f2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 -- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 msgr2=0x7f1ea81f2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 0x7f1ea81f2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 -- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 msgr2=0x7f1ea8140b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 0x7f1ea8140b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 -- 172.21.15.28:0/1267820098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e900094d0 con 0x7f1ea806c370 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 0x7f1ea8140b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7e00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 0x7f1ea81f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1ea7400640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800c430 tx=0x7f1e9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.226+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98011790 con 0x7f1ea806c370 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.227+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e980118f0 con 0x7f1ea806c370 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.227+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e98011ce0 con 0x7f1ea806c370 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.227+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea80cc160 con 0x7f1ea806c370 2024-09-13T17:13:53.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.227+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea80cc720 con 0x7f1ea806c370 2024-09-13T17:13:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.228+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea80c4480 con 0x7f1ea806c370 2024-09-13T17:13:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.229+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1e9801c460 con 0x7f1ea806c370 2024-09-13T17:13:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.229+0000 7f1e9fe00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 0x7f1e8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.230+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f1e98089410 con 0x7f1ea806c370 2024-09-13T17:13:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.230+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 0x7f1e8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.231+0000 7f1e9ea00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 0x7f1e8406d0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1e90034000 tx=0x7f1e9002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.234+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1e980544f0 con 0x7f1ea806c370 2024-09-13T17:13:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.425+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1ea806cb00 con 0x7f1ea806c370 2024-09-13T17:13:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.426+0000 7f1e9fe00640 1 -- 172.21.15.28:0/1267820098 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v95) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1e98057b10 con 0x7f1ea806c370 2024-09-13T17:13:53.427 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:13:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 msgr2=0x7f1e8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 0x7f1e8406d0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f1e90034000 tx=0x7f1e9002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:53.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 msgr2=0x7f1ea8140490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f1e9800c430 tx=0x7f1e9800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 shutdown_connections 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1e8406ac00 0x7f1e8406d0c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ea815ab70 0x7f1ea81f2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea80c40a0 0x7f1ea8140b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 --2- 172.21.15.28:0/1267820098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ea806c370 0x7f1ea8140490 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 >> 172.21.15.28:0/1267820098 conn(0x7f1ea8073470 msgr2=0x7f1ea8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 shutdown_connections 2024-09-13T17:13:53.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.428+0000 7f1eacc00640 1 -- 172.21.15.28:0/1267820098 wait complete. 2024-09-13T17:13:53.445 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:13:53.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:53.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:13:53.549 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:13:53.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:13:53.550 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:13:53.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 flush journal 2024-09-13T17:13:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.788+0000 7fccaf600640 1 Processor -- start 2024-09-13T17:13:53.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.788+0000 7fccaf600640 1 -- start start 2024-09-13T17:13:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccaf600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcca8158cd0 0x7fcca81590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccaf600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 0x7fcca80b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccaf600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca81595f0 con 0x7fcca806ccd0 2024-09-13T17:13:53.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccaf600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca81598c0 con 0x7fcca8158cd0 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccaf600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca8159ba0 con 0x7fcca806c110 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccacc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcca8158cd0 0x7fcca81590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccacc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcca8158cd0 0x7fcca81590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.789+0000 7fccacc00640 1 -- 172.21.15.28:0/3786236150 learned_addr learned my addr 172.21.15.28:0/3786236150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccad600640 1 --2- 172.21.15.28:0/3786236150 >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 0x7fcca80b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccacc00640 1 -- 172.21.15.28:0/3786236150 >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 msgr2=0x7fcca80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccacc00640 1 --2- 172.21.15.28:0/3786236150 >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 0x7fcca80b1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccacc00640 1 -- 172.21.15.28:0/3786236150 >> v1:172.21.15.28:6789/0 conn(0x7fcca806ccd0 legacy=0x7fcca806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccacc00640 1 -- 172.21.15.28:0/3786236150 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8159e80 con 0x7fcca8158cd0 2024-09-13T17:13:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.790+0000 7fccad600640 1 --2- 172.21.15.28:0/3786236150 >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 0x7fcca80b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.791+0000 7fccacc00640 1 --2- 172.21.15.28:0/3786236150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8158cd0 0x7fcca81590b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fcca80ce470 tx=0x7fcc9c02a1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ebf6d86389ed58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.791+0000 7fcca7400640 1 -- 172.21.15.28:0/3786236150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c032440 con 0x7fcca8158cd0 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.791+0000 7fcca7400640 1 -- 172.21.15.28:0/3786236150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c0325a0 con 0x7fcca8158cd0 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.791+0000 7fcca7400640 1 -- 172.21.15.28:0/3786236150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c032930 con 0x7fcca8158cd0 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 -- 172.21.15.28:0/3786236150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8158cd0 msgr2=0x7fcca81590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 --2- 172.21.15.28:0/3786236150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8158cd0 0x7fcca81590b0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7fcca80ce470 tx=0x7fcc9c02a1f0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 -- 172.21.15.28:0/3786236150 shutdown_connections 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 --2- 172.21.15.28:0/3786236150 >> v2:172.21.15.189:3300/0 conn(0x7fcca806c110 0x7fcca80b1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 --2- 172.21.15.28:0/3786236150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca8158cd0 0x7fcca81590b0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 -- 172.21.15.28:0/3786236150 >> 172.21.15.28:0/3786236150 conn(0x7fcca80b0270 msgr2=0x7fcca80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 -- 172.21.15.28:0/3786236150 shutdown_connections 2024-09-13T17:13:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.792+0000 7fccaf600640 1 -- 172.21.15.28:0/3786236150 wait complete. 2024-09-13T17:13:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.793+0000 7fccaf600640 1 Processor -- start 2024-09-13T17:13:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 -- start start 2024-09-13T17:13:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 0x7fcca80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 0x7fcca80cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca814e620 con 0x7fcca806c110 2024-09-13T17:13:53.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca80cb100 con 0x7fcca8158cd0 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.794+0000 7fccaf600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca80cb330 con 0x7fcca806ccd0 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44296/0 (socket says 172.21.15.28:44296) 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 -- 172.21.15.28:0/1108147758 learned_addr learned my addr 172.21.15.28:0/1108147758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccad600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 0x7fcca80cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 0x7fcca80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 -- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 msgr2=0x7fcca80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 0x7fcca80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 -- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 msgr2=0x7fcca80cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 0x7fcca80cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 -- 172.21.15.28:0/1108147758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c0094d0 con 0x7fcca806c110 2024-09-13T17:13:53.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccad600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 0x7fcca80cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 0x7fcca80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.795+0000 7fccacc00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c02a230 tx=0x7fcc9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.796+0000 7fcca4c00640 1 -- 172.21.15.28:0/1108147758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c032440 con 0x7fcca806c110 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.796+0000 7fcca4c00640 1 -- 172.21.15.28:0/1108147758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c030ba0 con 0x7fcca806c110 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.796+0000 7fcca4c00640 1 -- 172.21.15.28:0/1108147758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c03d440 con 0x7fcca806c110 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.796+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca80cb610 con 0x7fcca806c110 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.796+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca80cbbd0 con 0x7fcca806c110 2024-09-13T17:13:53.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.798+0000 7fcca4c00640 1 -- 172.21.15.28:0/1108147758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcc9c039040 con 0x7fcca806c110 2024-09-13T17:13:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.799+0000 7fcca4c00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 0x7fcc8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.799+0000 7fcca4c00640 1 -- 172.21.15.28:0/1108147758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcc9c0a9ff0 con 0x7fcca806c110 2024-09-13T17:13:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.800+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 0x7fcc8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.800+0000 7fccb2483740 1 Processor -- start 2024-09-13T17:13:53.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.800+0000 7fccb2483740 1 -- start start 2024-09-13T17:13:53.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.800+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 0x7fcc8806d0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcca814ed70 tx=0x7fcc94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f84e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f89ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f7506a0 con 0x55656f84b830 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f7b3cc0 con 0x55656f84e070 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55656f7b1450 con 0x55656f875520 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44302/0 (socket says 172.21.15.28:44302) 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 -- 172.21.15.28:0/4027736897 learned_addr learned my addr 172.21.15.28:0/4027736897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccacc00640 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f84e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fccad600640 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f89ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 -- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 msgr2=0x55656f89ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f89ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 -- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 msgr2=0x55656f84e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f84e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.801+0000 7fcca7e00640 1 -- 172.21.15.28:0/4027736897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55656f77d2e0 con 0x55656f84b830 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.802+0000 7fcca7e00640 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400d7d0 tx=0x7fcc94010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70cb0d45027de0e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.802+0000 7fcc99600640 1 -- 172.21.15.28:0/4027736897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94014ba0 con 0x55656f84b830 2024-09-13T17:13:53.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.802+0000 7fcc99600640 1 -- 172.21.15.28:0/4027736897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94014d00 con 0x55656f84b830 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.802+0000 7fcc99600640 1 -- 172.21.15.28:0/4027736897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94007230 con 0x55656f84b830 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 -- 172.21.15.28:0/4027736897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 msgr2=0x55656f89b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400d7d0 tx=0x7fcc94010000 comp rx=0 tx=0).stop 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 -- 172.21.15.28:0/4027736897 shutdown_connections 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f89ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84b830 0x55656f89b540 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 --2- 172.21.15.28:0/4027736897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f84e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 -- 172.21.15.28:0/4027736897 >> 172.21.15.28:0/4027736897 conn(0x55656f84a7a0 msgr2=0x55656f892c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 -- 172.21.15.28:0/4027736897 shutdown_connections 2024-09-13T17:13:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.803+0000 7fccb2483740 1 -- 172.21.15.28:0/4027736897 wait complete. 2024-09-13T17:13:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.804+0000 7fccb2483740 1 -- start start 2024-09-13T17:13:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.804+0000 7fccb2483740 1 Processor -- start 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f86ccb0 0x55656f86faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f7acd40 con 0x55656f84e070 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f74e280 con 0x55656f875520 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55656f74df20 con 0x55656f86ccb0 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccb2483740 10 client.0 _ll_register_callbacks cb 0x55656f7a63b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44314/0 (socket says 172.21.15.28:44314) 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 -- 172.21.15.28:0/420979494 learned_addr learned my addr 172.21.15.28:0/420979494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccad600640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f86ccb0 0x55656f86faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fcca7e00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 -- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f86ccb0 msgr2=0x55656f86faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f86ccb0 0x55656f86faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 -- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 msgr2=0x55656f894150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fccacc00640 1 -- 172.21.15.28:0/420979494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca81595f0 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.805+0000 7fcca7e00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccacc00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0752b0 tx=0x7fcc9c078ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdeeeb8cc9d0f1d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c079210 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcc9c077210 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55656f4a04e0 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc9c0a8170 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 10 client.7680 fetch_fsmap learned FSMap version 78 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55656f7ab3c0 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55656f4a04e0 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcc9c04d020 con 0x55656f84e070 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 10 client.7680 fetch_fsmap finished waiting for FSMap version 78 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 10 client.7680 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T17:13:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 20 client.7680 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f875fa0 0x55656f86c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.806+0000 7fccb2483740 4 client.7680 mds_command: new command op to 7491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.807+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 --> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55656f4b17e0 con 0x55656f875fa0 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.807+0000 7fcca7e00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f875fa0 0x55656f86c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.807+0000 7fcca7e00640 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f875fa0 0x55656f86c000 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c60f02b6eed667d server_cookie=8bf698842555fd44 in_seq=0 out_seq=0 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.807+0000 7fcc8e000640 10 client.7680 ms_handle_connect on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fcc8e000640 1 -- 172.21.15.28:0/420979494 <== mds.0 v2:172.21.15.28:6832/2861751170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55656f4b17e0 con 0x55656f875fa0 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fcc8e000640 10 client.7680 handle_command_reply: tid=0 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 client.7680 shutdown 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 msgr2=0x55656f871e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c0752b0 tx=0x7fcc9c078ec0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 shutdown_connections 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f875fa0 0x55656f86c000 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f86ccb0 0x55656f86faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 --2- 172.21.15.28:0/420979494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f84e070 0x55656f871e10 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fcc8e000640 0 client.7680 ms_handle_reset on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 >> 172.21.15.28:0/420979494 conn(0x55656f84b830 msgr2=0x55656f896140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 shutdown_connections 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.808+0000 7fccb2483740 1 -- 172.21.15.28:0/420979494 wait complete. 2024-09-13T17:13:53.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.809+0000 7fccb2483740 20 client.7680 trim_cache size 0 max 16384 2024-09-13T17:13:53.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.822+0000 7fccb2483740 1 Processor -- start 2024-09-13T17:13:53.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.822+0000 7fccb2483740 1 -- start start 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f8b5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f894770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f514040 con 0x55656f86ccb0 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f896d50 con 0x55656f84e070 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55656f892c00 con 0x55656f875520 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44316/0 (socket says 172.21.15.28:44316) 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 -- 172.21.15.28:0/1385811648 learned_addr learned my addr 172.21.15.28:0/1385811648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccad600640 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f894770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fccacc00640 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f8b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 -- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 msgr2=0x55656f894770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f894770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 -- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 msgr2=0x55656f8b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f8b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.823+0000 7fcca7e00640 1 -- 172.21.15.28:0/1385811648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55656f74d610 con 0x55656f86ccb0 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fcca7e00640 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000f60 tx=0x7fcc9401c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758fae6a81cc351 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fcc9a000640 1 -- 172.21.15.28:0/1385811648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc94030d00 con 0x55656f86ccb0 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fcc9a000640 1 -- 172.21.15.28:0/1385811648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94018070 con 0x55656f86ccb0 2024-09-13T17:13:53.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fcc9a000640 1 -- 172.21.15.28:0/1385811648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc940304c0 con 0x55656f86ccb0 2024-09-13T17:13:53.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 -- 172.21.15.28:0/1385811648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 msgr2=0x55656f890040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fcc94000f60 tx=0x7fcc9401c000 comp rx=0 tx=0).stop 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 -- 172.21.15.28:0/1385811648 shutdown_connections 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f875520 0x55656f894770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f890040 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 --2- 172.21.15.28:0/1385811648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f84e070 0x55656f8b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.824+0000 7fccb2483740 1 -- 172.21.15.28:0/1385811648 >> 172.21.15.28:0/1385811648 conn(0x55656f84b830 msgr2=0x55656f86c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.825+0000 7fccb2483740 1 -- 172.21.15.28:0/1385811648 shutdown_connections 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.825+0000 7fccb2483740 1 -- 172.21.15.28:0/1385811648 wait complete. 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.825+0000 7fccb2483740 1 -- start start 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.825+0000 7fccb2483740 1 Processor -- start 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f84e070 0x55656f86d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f86de20 con 0x55656f86ccb0 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656f8b49c0 con 0x55656f875520 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55656f894660 con 0x55656f84e070 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.826+0000 7fccb2483740 10 client.0 _ll_register_callbacks cb 0x7fcc9c030f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35626/0 (socket says 172.21.15.28:35626) 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fccacc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f84e070 0x55656f86d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 -- 172.21.15.28:0/595839378 learned_addr learned my addr 172.21.15.28:0/595839378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fccad600640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 -- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f84e070 msgr2=0x55656f86d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f84e070 0x55656f86d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 -- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 msgr2=0x55656f894af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:53.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 -- 172.21.15.28:0/595839378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55656f77d2e0 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcca7e00640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400d7a0 tx=0x7fcc94000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=babafe832a4da93e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.827+0000 7fcc8e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc940302c0 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fcc94030420 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc94014d00 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=78) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc940305e0 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 10 client.7686 fetch_fsmap learned FSMap version 78 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=78,monmap=2+}) v3 -- 0x55656f772b20 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc94014d00 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 78) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fcc94030740 con 0x55656f86ccb0 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 10 client.7686 fetch_fsmap finished waiting for FSMap version 78 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 10 client.7686 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T17:13:53.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 20 client.7686 populate_metadata read hostname 'smithi028' 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f84e730 0x55656f873cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 4 client.7686 mds_command: new command op to 7491 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 --> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55656f504e30 con 0x55656f84e730 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.828+0000 7fccacc00640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f84e730 0x55656f873cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.829+0000 7fccacc00640 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f84e730 0x55656f873cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca9fedbb68e50b08 server_cookie=d5209ec9ccb2af7f in_seq=0 out_seq=0 2024-09-13T17:13:53.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.829+0000 7fcc8e000640 10 client.7686 ms_handle_connect on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fcc8e000640 1 -- 172.21.15.28:0/595839378 <== mds.0 v2:172.21.15.28:6832/2861751170 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55656f504e30 con 0x55656f84e730 2024-09-13T17:13:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fcc8e000640 10 client.7686 handle_command_reply: tid=0 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 client.7686 shutdown 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 msgr2=0x55656f8fcb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7fcc9400d7a0 tx=0x7fcc94000ed0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 shutdown_connections 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55656f875520 0x55656f894af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55656f86ccb0 0x55656f8fcb10 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.28:6832/2861751170,v1:172.21.15.28:6834/2861751170] conn(0x55656f84e730 0x55656f873cb0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 --2- 172.21.15.28:0/595839378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55656f84e070 0x55656f86d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fcc8e000640 0 client.7686 ms_handle_reset on v2:172.21.15.28:6832/2861751170 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 >> 172.21.15.28:0/595839378 conn(0x55656f84b830 msgr2=0x55656f849cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 shutdown_connections 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.840+0000 7fccb2483740 1 -- 172.21.15.28:0/595839378 wait complete. 2024-09-13T17:13:53.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccb2483740 20 client.7686 trim_cache size 0 max 16384 2024-09-13T17:13:53.843 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:13:53.843 INFO:teuthology.orchestra.run.smithi028.stdout: "message": "", 2024-09-13T17:13:53.843 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:13:53.843 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 msgr2=0x7fcc8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 0x7fcc8806d0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcca814ed70 tx=0x7fcc94009040 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 msgr2=0x7fcca80cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c02a230 tx=0x7fcc9c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 shutdown_connections 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc8806ac00 0x7fcc8806d0c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8158cd0 0x7fcca80cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcca806ccd0 0x7fcca80ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 --2- 172.21.15.28:0/1108147758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcca806c110 0x7fcca80cabc0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 >> 172.21.15.28:0/1108147758 conn(0x7fcca80b0270 msgr2=0x7fcca806f690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:53.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 shutdown_connections 2024-09-13T17:13:53.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:53.841+0000 7fccaf600640 1 -- 172.21.15.28:0/1108147758 wait complete. 2024-09-13T17:13:53.858 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:13:53.858 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:53.859 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:13:53.859 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:53.859 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:13:53.859 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:53.859 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:13:53.859 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:13:53.874 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:13:53.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2024-09-13T17:13:53.875 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:13:53.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2024-09-13T17:13:53.877 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:13:53.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail c 2024-09-13T17:13:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.076+0000 7f7e6d000640 1 Processor -- start 2024-09-13T17:13:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.076+0000 7f7e6d000640 1 -- start start 2024-09-13T17:13:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.076+0000 7fda06c00640 1 Processor -- start 2024-09-13T17:13:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7fda06c00640 1 -- start start 2024-09-13T17:13:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7f7e6d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e680b17e0 0x7f7e680b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7f7e6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6816a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7f7e6d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e681610a0 con 0x7f7e6806d070 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7f7e6d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e680b2620 con 0x7f7e680b17e0 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.077+0000 7f7e6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680b2900 con 0x7f7e6806c540 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e6806d070 0x7f7e680b1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34518/0 (socket says 172.21.15.28:34518) 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65400640 1 -- 172.21.15.28:0/3240414539 learned_addr learned my addr 172.21.15.28:0/3240414539 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3240414539 >> v2:172.21.15.189:3300/0 conn(0x7f7e680b17e0 0x7f7e680b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e66800640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6816a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda0015cab0 0x7fda000cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e64a00640 1 -- 172.21.15.28:0/3240414539 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882382572 0 0) 0x7f7e681610a0 con 0x7f7e6806d070 2024-09-13T17:13:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e64a00640 1 -- 172.21.15.28:0/3240414539 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e4c003520 con 0x7f7e6806d070 2024-09-13T17:13:54.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 0x7fda000cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda000d0420 con 0x7fda000ccea0 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda000cdb60 con 0x7fda0015cab0 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda06c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda000cde40 con 0x7fda0015c6d0 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65e00640 1 -- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 msgr2=0x7f7e6816a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6816a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65e00640 1 -- 172.21.15.28:0/3240414539 >> v1:172.21.15.28:6789/0 conn(0x7f7e6806d070 legacy=0x7f7e680b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e65e00640 1 -- 172.21.15.28:0/3240414539 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e680b2be0 con 0x7f7e680b17e0 2024-09-13T17:13:54.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7f7e66800640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6816a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.078+0000 7fda05800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:36564/0 (socket says 172.21.15.28:36564) 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 -- 172.21.15.28:0/1510373645 learned_addr learned my addr 172.21.15.28:0/1510373645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda04e00640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.189:3300/0 conn(0x7fda0015cab0 0x7fda000cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda06200640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 0x7fda000cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 -- 172.21.15.28:0/1510373645 >> v2:172.21.15.189:3300/0 conn(0x7fda0015cab0 msgr2=0x7fda000cf6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.189:3300/0 conn(0x7fda0015cab0 0x7fda000cf6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 -- 172.21.15.28:0/1510373645 >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 msgr2=0x7fda000cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 0x7fda000cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda05800640 1 -- 172.21.15.28:0/1510373645 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda000d0670 con 0x7fda0015c6d0 2024-09-13T17:13:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.079+0000 7fda06200640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 0x7fda000cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:13:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.080+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e680b17e0 0x7f7e680b3480 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7e580097a0 tx=0x7f7e5802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=202e2d5f6e915ec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.080+0000 7f7e64a00640 1 -- 172.21.15.28:0/3240414539 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58032440 con 0x7f7e680b17e0 2024-09-13T17:13:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.080+0000 7f7e64a00640 1 -- 172.21.15.28:0/3240414539 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e580325a0 con 0x7f7e680b17e0 2024-09-13T17:13:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.080+0000 7f7e64a00640 1 -- 172.21.15.28:0/3240414539 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e58032930 con 0x7f7e680b17e0 2024-09-13T17:13:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7fda05800640 1 --2- 172.21.15.28:0/1510373645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda0006bd50 tx=0x7fd9f402a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eba98a9860f1dd73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1510373645 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f402ae90 con 0x7fda0015c6d0 2024-09-13T17:13:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1510373645 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9f4032440 con 0x7fda0015c6d0 2024-09-13T17:13:54.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7fd9ffe00640 1 -- 172.21.15.28:0/1510373645 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9f40327d0 con 0x7fda0015c6d0 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 -- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e680b17e0 msgr2=0x7f7e680b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e680b17e0 0x7f7e680b3480 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f7e580097a0 tx=0x7f7e5802a180 comp rx=0 tx=0).stop 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 -- 172.21.15.28:0/3240414539 shutdown_connections 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6816a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3240414539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e680b17e0 0x7f7e680b3480 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 -- 172.21.15.28:0/3240414539 >> 172.21.15.28:0/3240414539 conn(0x7f7e680b0270 msgr2=0x7f7e680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 -- 172.21.15.28:0/3240414539 shutdown_connections 2024-09-13T17:13:54.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.081+0000 7f7e6d000640 1 -- 172.21.15.28:0/3240414539 wait complete. 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 -- 172.21.15.28:0/1510373645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 msgr2=0x7fda000cf170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 --2- 172.21.15.28:0/1510373645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fda0006bd50 tx=0x7fd9f402a140 comp rx=0 tx=0).stop 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 -- 172.21.15.28:0/1510373645 shutdown_connections 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.28:3300/0 conn(0x7fda000ccea0 0x7fda000cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 --2- 172.21.15.28:0/1510373645 >> v2:172.21.15.189:3300/0 conn(0x7fda0015cab0 0x7fda000cf6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 --2- 172.21.15.28:0/1510373645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015c6d0 0x7fda000cf170 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 -- 172.21.15.28:0/1510373645 >> 172.21.15.28:0/1510373645 conn(0x7fda00069e80 msgr2=0x7fda0006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 -- 172.21.15.28:0/1510373645 shutdown_connections 2024-09-13T17:13:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.082+0000 7fda06c00640 1 -- 172.21.15.28:0/1510373645 wait complete. 2024-09-13T17:13:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.083+0000 7f7e6d000640 1 Processor -- start 2024-09-13T17:13:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.083+0000 7f7e6d000640 1 -- start start 2024-09-13T17:13:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7fda06c00640 1 Processor -- start 2024-09-13T17:13:54.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7fda06c00640 1 -- start start 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6814a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6806d070 0x7f7e680a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680a8db0 con 0x7f7e680b17e0 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680a9090 con 0x7f7e6806d070 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.084+0000 7f7e6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e680a9370 con 0x7f7e6806c540 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35644/0 (socket says 172.21.15.28:35644) 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 -- 172.21.15.28:0/3215486989 learned_addr learned my addr 172.21.15.28:0/3215486989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6814a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e65400640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6806d070 0x7f7e680a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000ccea0 0x7fda000a9d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015cab0 0x7fda001e7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000ca850 con 0x7fda0015c6d0 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000cab30 con 0x7fda000ccea0 2024-09-13T17:13:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda000cae10 con 0x7fda0015cab0 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 -- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 msgr2=0x7f7e6814a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6814a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 -- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6806d070 msgr2=0x7f7e680a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6806d070 0x7f7e680a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda04e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7f7e66800640 1 -- 172.21.15.28:0/3215486989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e60009a70 con 0x7f7e680b17e0 2024-09-13T17:13:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda04e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35650/0 (socket says 172.21.15.28:35650) 2024-09-13T17:13:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda05800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000ccea0 0x7fda000a9d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda04e00640 1 -- 172.21.15.28:0/705159902 learned_addr learned my addr 172.21.15.28:0/705159902 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.085+0000 7fda06200640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015cab0 0x7fda001e7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e66800640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000ef90 tx=0x7f7e6000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 -- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015cab0 msgr2=0x7fda001e7130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015cab0 0x7fda001e7130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 -- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000ccea0 msgr2=0x7fda000a9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000ccea0 0x7fda000a9d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 -- 172.21.15.28:0/705159902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f8009a70 con 0x7fda0015c6d0 2024-09-13T17:13:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e60009cd0 con 0x7f7e680b17e0 2024-09-13T17:13:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6000f040 con 0x7f7e680b17e0 2024-09-13T17:13:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e600146a0 con 0x7f7e680b17e0 2024-09-13T17:13:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e58009400 con 0x7f7e680b17e0 2024-09-13T17:13:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e6814aea0 con 0x7f7e680b17e0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.086+0000 7fda04e00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e9b0 tx=0x7fd9ec00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.087+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec009670 con 0x7fda0015c6d0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.087+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9ec00c940 con 0x7fda0015c6d0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.087+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f40094d0 con 0x7fda0015c6d0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.087+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec00ccf0 con 0x7fda0015c6d0 2024-09-13T17:13:54.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.088+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e6814b0c0 con 0x7f7e680b17e0 2024-09-13T17:13:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.087+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda000cb3d0 con 0x7fda0015c6d0 2024-09-13T17:13:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.089+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda000cd2f0 con 0x7fda0015c6d0 2024-09-13T17:13:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.089+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd9ec013470 con 0x7fda0015c6d0 2024-09-13T17:13:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.089+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e60014800 con 0x7f7e680b17e0 2024-09-13T17:13:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.090+0000 7f7e5e000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 0x7f7e4806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.090+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f7e6008a200 con 0x7f7e680b17e0 2024-09-13T17:13:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.090+0000 7fd9fd600640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 0x7fd9e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.091+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7fd9ec0895b0 con 0x7fda0015c6d0 2024-09-13T17:13:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.091+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 0x7f7e4806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.091+0000 7fda05800640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 0x7fd9e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.092+0000 7fda05800640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 0x7fd9e006d0c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4034000 tx=0x7fd9f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.092+0000 7f7e65e00640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 0x7f7e4806d110 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7e58034000 tx=0x7f7e5802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.095+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd9ec054740 con 0x7fda0015c6d0 2024-09-13T17:13:54.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.100+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e60055410 con 0x7f7e680b17e0 2024-09-13T17:13:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.130+0000 7f4b23e00640 1 Processor -- start 2024-09-13T17:13:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.130+0000 7f4b23e00640 1 -- start start 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b23e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4b1c15c9a0 0x7f4b1c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b23e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0b2400 0x7f4b1c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b23e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1c070470 con 0x7f4b1c0ad940 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b23e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b1c0ace30 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b23e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c0ad110 con 0x7f4b1c0b2400 2024-09-13T17:13:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b20a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4b1c0ad940 0x7f4b1c0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34520/0 (socket says 172.21.15.28:34520) 2024-09-13T17:13:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.131+0000 7f4b20a00640 1 -- 172.21.15.28:0/97816755 learned_addr learned my addr 172.21.15.28:0/97816755 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:13:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21e00640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0b2400 0x7f4b1c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21400640 1 --2- 172.21.15.28:0/97816755 >> v2:172.21.15.189:3300/0 conn(0x7f4b1c15c9a0 0x7f4b1c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054609835 0 0) 0x7f4b1c070470 con 0x7f4b1c0ad940 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b04003520 con 0x7f4b1c0ad940 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21400640 1 -- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0b2400 msgr2=0x7f4b1c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21400640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0b2400 0x7f4b1c06bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456707481 0 0) 0x7f4b04003520 con 0x7f4b1c0ad940 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21400640 1 -- 172.21.15.28:0/97816755 >> v1:172.21.15.28:6789/0 conn(0x7f4b1c0ad940 legacy=0x7f4b1c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.132+0000 7f4b21400640 1 -- 172.21.15.28:0/97816755 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c06b170 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.134+0000 7f4b21400640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c0ac850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097d0 tx=0x7f4b1002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b75bc362f584ece3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.135+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b08001690 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.135+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1002ac70 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.135+0000 7f4b1be00640 1 -- 172.21.15.28:0/97816755 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10032440 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.135+0000 7f4b23e00640 1 -- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c15c9a0 msgr2=0x7f4b1c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.135+0000 7f4b23e00640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c0ac850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f4b100097d0 tx=0x7f4b1002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 -- 172.21.15.28:0/97816755 shutdown_connections 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0b2400 0x7f4b1c06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 --2- 172.21.15.28:0/97816755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c0ac850 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 -- 172.21.15.28:0/97816755 >> 172.21.15.28:0/97816755 conn(0x7f4b1c0b0270 msgr2=0x7f4b1c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 -- 172.21.15.28:0/97816755 shutdown_connections 2024-09-13T17:13:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.136+0000 7f4b23e00640 1 -- 172.21.15.28:0/97816755 wait complete. 2024-09-13T17:13:54.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.137+0000 7f4b23e00640 1 Processor -- start 2024-09-13T17:13:54.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.138+0000 7f4b23e00640 1 -- start start 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.138+0000 7f4b23e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0ad940 0x7f4b1c1e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.138+0000 7f4b23e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c0b2400 0x7f4b1c155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.138+0000 7f4b23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b23e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c155980 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b23e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c156430 con 0x7f4b1c0b2400 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b23e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1c155b90 con 0x7f4b1c0ad940 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35658/0 (socket says 172.21.15.28:35658) 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 -- 172.21.15.28:0/309700851 learned_addr learned my addr 172.21.15.28:0/309700851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b20a00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c0b2400 0x7f4b1c155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21400640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0ad940 0x7f4b1c1e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 -- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0ad940 msgr2=0x7f4b1c1e7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0ad940 0x7f4b1c1e7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 -- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c0b2400 msgr2=0x7f4b1c155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c0b2400 0x7f4b1c155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.139+0000 7f4b21e00640 1 -- 172.21.15.28:0/309700851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b10009400 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b21e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400ef90 tx=0x7f4b1400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b14009ca0 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1400f040 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b140146c0 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1c070930 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.140+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b1c156150 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.141+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b1c15b830 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.144+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4b14014820 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.144+0000 7f4b19600640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 0x7f4afc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:13:54.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.145+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5796+0+0 (secure 0 0 0) 0x7f4b1408a170 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.145+0000 7f4b21400640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 0x7f4afc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:13:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.146+0000 7f4b21400640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 0x7f4afc06d0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4b10034000 tx=0x7f4b1002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:13:54.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.153+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4b14055390 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.351+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fda000b87d0 con 0x7fda0015c6d0 2024-09-13T17:13:54.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.352+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f7e680a9530 con 0x7f7e680b17e0 2024-09-13T17:13:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.408+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "c"} v 0) v1 -- 0x7f4b1c15ce10 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.410 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:13:54.409+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:13:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.417+0000 7fd9fd600640 1 -- 172.21.15.28:0/705159902 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 6380 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7fd9ec057d60 con 0x7fda0015c6d0 2024-09-13T17:13:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.417+0000 7f7e5e000640 1 -- 172.21.15.28:0/3215486989 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7075 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f7e60058a30 con 0x7f7e680b17e0 2024-09-13T17:13:54.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.417+0000 7f4b19600640 1 -- 172.21.15.28:0/309700851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "c"}]=0 failed mds gid 7491 v79) v1 ==== 95+0+0 (secure 0 0 0) 0x7f4b140589b0 con 0x7f4b1c15c9a0 2024-09-13T17:13:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6380 2024-09-13T17:13:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 7075 2024-09-13T17:13:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 7491 2024-09-13T17:13:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 msgr2=0x7f7e4806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 0x7f7e4806d110 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7e58034000 tx=0x7f7e5802e000 comp rx=0 tx=0).stop 2024-09-13T17:13:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 msgr2=0x7f7e68070d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f7e6000ef90 tx=0x7f7e6000c490 comp rx=0 tx=0).stop 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 shutdown_connections 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e4806ac50 0x7f7e4806d110 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e680b17e0 0x7f7e68070d10 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e6806d070 0x7f7e680a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 --2- 172.21.15.28:0/3215486989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e6806c540 0x7f7e6814a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 >> 172.21.15.28:0/3215486989 conn(0x7f7e680b0270 msgr2=0x7f7e6815ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 shutdown_connections 2024-09-13T17:13:54.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.420+0000 7f7e6d000640 1 -- 172.21.15.28:0/3215486989 wait complete. 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 msgr2=0x7fd9e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 0x7fd9e006d0c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4034000 tx=0x7fd9f402e000 comp rx=0 tx=0).stop 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 msgr2=0x7fda000a91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00e9b0 tx=0x7fd9ec00edb0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 shutdown_connections 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9e006ac00 0x7fd9e006d0c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda0015cab0 0x7fda001e7130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda0015c6d0 0x7fda000a91b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 --2- 172.21.15.28:0/705159902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda000ccea0 0x7fda000a9d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 >> 172.21.15.28:0/705159902 conn(0x7fda00069e80 msgr2=0x7fda000b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 shutdown_connections 2024-09-13T17:13:54.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7fda06c00640 1 -- 172.21.15.28:0/705159902 wait complete. 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 msgr2=0x7f4afc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 0x7f4afc06d0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4b10034000 tx=0x7f4b1002e000 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 msgr2=0x7f4b1c1e6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f4b1400ef90 tx=0x7f4b1400c490 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 shutdown_connections 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4afc06ac00 0x7f4afc06d0c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b1c15c9a0 0x7f4b1c1e6bd0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4b1c0b2400 0x7f4b1c155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 --2- 172.21.15.28:0/309700851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4b1c0ad940 0x7f4b1c1e7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 >> 172.21.15.28:0/309700851 conn(0x7f4b1c0b0270 msgr2=0x7f4b1c15b120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:13:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 shutdown_connections 2024-09-13T17:13:54.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:13:54.421+0000 7f4b23e00640 1 -- 172.21.15.28:0/309700851 wait complete. 2024-09-13T17:13:54.440 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:13:54.440 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:13:54.441 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:13:54.441 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:13:54.442 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:13:54.442 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:13:54.447 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:13:54.447 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:54.487 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:13:54.488 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:54.489 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:13:54.490 DEBUG:teuthology.parallel:result is None 2024-09-13T17:13:54.576 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:13:54.618 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:13:54.619 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:13:59.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-13T17:13:59.876 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:13:59.876 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail d 2024-09-13T17:14:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.042+0000 7fcc4da00640 1 Processor -- start 2024-09-13T17:14:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.042+0000 7fcc4da00640 1 -- start start 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 0x7fcc4806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc4806f3f0 0x7fcc48070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc480add00 0x7fcc48159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4806f920 con 0x7fcc4806f3f0 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4815b290 con 0x7fcc48159770 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.043+0000 7fcc4da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4815b570 con 0x7fcc480add00 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc4806f3f0 0x7fcc48070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc4806f3f0 0x7fcc48070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35700/0 (socket says 172.21.15.28:35700) 2024-09-13T17:14:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 -- 172.21.15.28:0/3927313175 learned_addr learned my addr 172.21.15.28:0/3927313175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc47400640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc480add00 0x7fcc48159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc46a00640 1 --2- 172.21.15.28:0/3927313175 >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 0x7fcc4806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 -- 172.21.15.28:0/3927313175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc480add00 msgr2=0x7fcc48159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc480add00 0x7fcc48159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 -- 172.21.15.28:0/3927313175 >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 msgr2=0x7fcc4806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3927313175 >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 0x7fcc4806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc3e000640 1 -- 172.21.15.28:0/3927313175 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4815b850 con 0x7fcc4806f3f0 2024-09-13T17:14:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.044+0000 7fcc46a00640 1 --2- 172.21.15.28:0/3927313175 >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 0x7fcc4806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.045+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc4806f3f0 0x7fcc48070260 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcc48068040 tx=0x7fcc3002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9001a0a275adf5bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.045+0000 7fcc46000640 1 -- 172.21.15.28:0/3927313175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3002ae90 con 0x7fcc4806f3f0 2024-09-13T17:14:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc46000640 1 -- 172.21.15.28:0/3927313175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30032440 con 0x7fcc4806f3f0 2024-09-13T17:14:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc46000640 1 -- 172.21.15.28:0/3927313175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc300327d0 con 0x7fcc4806f3f0 2024-09-13T17:14:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 -- 172.21.15.28:0/3927313175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc4806f3f0 msgr2=0x7fcc48070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc4806f3f0 0x7fcc48070260 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fcc48068040 tx=0x7fcc3002a140 comp rx=0 tx=0).stop 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 -- 172.21.15.28:0/3927313175 shutdown_connections 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc4806f3f0 0x7fcc48070260 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3927313175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc480add00 0x7fcc48159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3927313175 >> v2:172.21.15.189:3300/0 conn(0x7fcc48159770 0x7fcc4806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 -- 172.21.15.28:0/3927313175 >> 172.21.15.28:0/3927313175 conn(0x7fcc480b0270 msgr2=0x7fcc480b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 -- 172.21.15.28:0/3927313175 shutdown_connections 2024-09-13T17:14:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.046+0000 7fcc4da00640 1 -- 172.21.15.28:0/3927313175 wait complete. 2024-09-13T17:14:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.048+0000 7fcc4da00640 1 Processor -- start 2024-09-13T17:14:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.048+0000 7fcc4da00640 1 -- start start 2024-09-13T17:14:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4806f3f0 0x7fcc480cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc48159770 0x7fcc481652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4813fdc0 con 0x7fcc480add00 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4813eea0 con 0x7fcc4806f3f0 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc4da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4813f140 con 0x7fcc48159770 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc3e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35704/0 (socket says 172.21.15.28:35704) 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc3e000640 1 -- 172.21.15.28:0/3628942317 learned_addr learned my addr 172.21.15.28:0/3628942317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc46a00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4806f3f0 0x7fcc480cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.049+0000 7fcc47400640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc48159770 0x7fcc481652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 -- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc48159770 msgr2=0x7fcc481652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc48159770 0x7fcc481652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 -- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4806f3f0 msgr2=0x7fcc480cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4806f3f0 0x7fcc480cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 -- 172.21.15.28:0/3628942317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc300094d0 con 0x7fcc480add00 2024-09-13T17:14:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3e000640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002a180 tx=0x7fcc300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3003d830 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3003d990 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3003dd40 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4813f420 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.050+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4813f9e0 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.052+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4813ff00 con 0x7fcc480add00 2024-09-13T17:14:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.052+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcc30039040 con 0x7fcc480add00 2024-09-13T17:14:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.054+0000 7fcc3f400640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 0x7fcc2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.054+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5882+0+0 (secure 0 0 0) 0x7fcc300a9e40 con 0x7fcc480add00 2024-09-13T17:14:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.054+0000 7fcc46a00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 0x7fcc2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.055+0000 7fcc46a00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 0x7fcc2406d0c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc481547a0 tx=0x7fcc38008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.058+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcc4813ff00 con 0x7fcc480add00 2024-09-13T17:14:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.306+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "d"} v 0) v1 -- 0x7fcc4813ff00 con 0x7fcc480add00 2024-09-13T17:14:00.309 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:14:00.307+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:14:00.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.319+0000 7fcc3f400640 1 -- 172.21.15.28:0/3628942317 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "d"}]=0 failed mds gid 6374 v82) v1 ==== 95+0+0 (secure 0 0 0) 0x7fcc4813ff00 con 0x7fcc480add00 2024-09-13T17:14:00.321 INFO:teuthology.orchestra.run.smithi028.stderr:failed mds gid 6374 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 msgr2=0x7fcc2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 0x7fcc2406d0c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fcc481547a0 tx=0x7fcc38008040 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 msgr2=0x7fcc4813e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fcc3002a180 tx=0x7fcc300046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 shutdown_connections 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc2406ac00 0x7fcc2406d0c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc48159770 0x7fcc481652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc480add00 0x7fcc4813e530 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 --2- 172.21.15.28:0/3628942317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc4806f3f0 0x7fcc480cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 >> 172.21.15.28:0/3628942317 conn(0x7fcc480b0270 msgr2=0x7fcc4806fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 shutdown_connections 2024-09-13T17:14:00.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.322+0000 7fcc4da00640 1 -- 172.21.15.28:0/3628942317 wait complete. 2024-09-13T17:14:00.343 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:14:00.343 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:14:00.348 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:14:00.348 DEBUG:teuthology.parallel:result is None 2024-09-13T17:14:00.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:00.476 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:14:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.501+0000 7fdbb5400640 1 Processor -- start 2024-09-13T17:14:00.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.501+0000 7fdbb5400640 1 -- start start 2024-09-13T17:14:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbb0158730 0x7fdbb00cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb5400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdbb00afdd0 0x7fdbb00cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb5400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbb00b28d0 con 0x7fdbb00ce0f0 2024-09-13T17:14:00.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb5400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00b2bb0 con 0x7fdbb0158730 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb5400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbb00b2e90 con 0x7fdbb00afdd0 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbafe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdbb0158730 0x7fdbb00cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdba7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdbb00ce0f0 0x7fdbb00af6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34548/0 (socket says 172.21.15.28:34548) 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdba7400640 1 -- 172.21.15.28:0/578991335 learned_addr learned my addr 172.21.15.28:0/578991335 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/578991335 >> v2:172.21.15.189:3300/0 conn(0x7fdbb00afdd0 0x7fdbb00cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbafe00640 1 -- 172.21.15.28:0/578991335 >> v2:172.21.15.189:3300/0 conn(0x7fdbb00afdd0 msgr2=0x7fdbb00cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbafe00640 1 --2- 172.21.15.28:0/578991335 >> v2:172.21.15.189:3300/0 conn(0x7fdbb00afdd0 0x7fdbb00cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbafe00640 1 -- 172.21.15.28:0/578991335 >> v1:172.21.15.28:6789/0 conn(0x7fdbb00ce0f0 legacy=0x7fdbb00af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.503+0000 7fdbafe00640 1 -- 172.21.15.28:0/578991335 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00b3170 con 0x7fdbb0158730 2024-09-13T17:14:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.504+0000 7fdbafe00640 1 --2- 172.21.15.28:0/578991335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb00cdbb0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00cd410 tx=0x7fdba002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc661301b13f599d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbaf400640 1 -- 172.21.15.28:0/578991335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0032440 con 0x7fdbb0158730 2024-09-13T17:14:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbaf400640 1 -- 172.21.15.28:0/578991335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba00325a0 con 0x7fdbb0158730 2024-09-13T17:14:00.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbaf400640 1 -- 172.21.15.28:0/578991335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba0032930 con 0x7fdbb0158730 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 -- 172.21.15.28:0/578991335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 msgr2=0x7fdbb00cdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 --2- 172.21.15.28:0/578991335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb00cdbb0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00cd410 tx=0x7fdba002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 -- 172.21.15.28:0/578991335 shutdown_connections 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 --2- 172.21.15.28:0/578991335 >> v2:172.21.15.189:3300/0 conn(0x7fdbb00afdd0 0x7fdbb00cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 --2- 172.21.15.28:0/578991335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb00cdbb0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 -- 172.21.15.28:0/578991335 >> 172.21.15.28:0/578991335 conn(0x7fdbb0072e40 msgr2=0x7fdbb0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 -- 172.21.15.28:0/578991335 shutdown_connections 2024-09-13T17:14:00.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.505+0000 7fdbb5400640 1 -- 172.21.15.28:0/578991335 wait complete. 2024-09-13T17:14:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.507+0000 7fdbb5400640 1 Processor -- start 2024-09-13T17:14:00.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.507+0000 7fdbb5400640 1 -- start start 2024-09-13T17:14:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb00afdd0 0x7fdbb0148a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdbb00ce0f0 0x7fdbb0145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00c37e0 con 0x7fdbb0158730 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb00c3a80 con 0x7fdbb00afdd0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb5400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbb01460b0 con 0x7fdbb00ce0f0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35726/0 (socket says 172.21.15.28:35726) 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbb4a00640 1 -- 172.21.15.28:0/876040681 learned_addr learned my addr 172.21.15.28:0/876040681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdba7400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdbb00ce0f0 0x7fdbb0145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.508+0000 7fdbafe00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb00afdd0 0x7fdbb0148a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 -- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdbb00ce0f0 msgr2=0x7fdbb0145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdbb00ce0f0 0x7fdbb0145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 -- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb00afdd0 msgr2=0x7fdbb0148a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb00afdd0 0x7fdbb0148a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 -- 172.21.15.28:0/876040681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba0009400 con 0x7fdbb0158730 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb4a00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fdba8007d80 tx=0x7fdba800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba8014420 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba8014580 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba8014930 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.509+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbb0146390 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.510+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbb00c3d60 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.511+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbb00cdbb0 con 0x7fdbb0158730 2024-09-13T17:14:00.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.511+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdba8014a90 con 0x7fdbb0158730 2024-09-13T17:14:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.512+0000 7fdbacc00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 0x7fdb8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.512+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fdba8089c00 con 0x7fdbb0158730 2024-09-13T17:14:00.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.513+0000 7fdbafe00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 0x7fdb8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:00.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.513+0000 7fdbafe00640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 0x7fdb8c06d0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdba0034000 tx=0x7fdba002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:00.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.516+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdba8054da0 con 0x7fdbb0158730 2024-09-13T17:14:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.728+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbb00cbf30 con 0x7fdbb0158730 2024-09-13T17:14:00.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.729+0000 7fdbacc00640 1 -- 172.21.15.28:0/876040681 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3983 (secure 0 0 0) 0x7fdba80583c0 con 0x7fdbb0158730 2024-09-13T17:14:00.731 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:00.731 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":83,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:00.320973+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:00.731 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 83 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 msgr2=0x7fdb8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 0x7fdb8c06d0c0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdba0034000 tx=0x7fdba002e000 comp rx=0 tx=0).stop 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 msgr2=0x7fdbb0154350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7fdba8007d80 tx=0x7fdba800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 shutdown_connections 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb8c06ac00 0x7fdb8c06d0c0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdbb0158730 0x7fdbb0154350 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdbb00ce0f0 0x7fdbb0145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 --2- 172.21.15.28:0/876040681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdbb00afdd0 0x7fdbb0148a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 >> 172.21.15.28:0/876040681 conn(0x7fdbb0072e40 msgr2=0x7fdbb00caff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 shutdown_connections 2024-09-13T17:14:00.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:00.731+0000 7fdbb5400640 1 -- 172.21.15.28:0/876040681 wait complete. 2024-09-13T17:14:01.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:13:42.545599+0000', 'modified': '2024-09-13T17:14:00.320973+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'a', 'rank': 0, 'incarnation': 83, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/2200414779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 2200414779}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2200414779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:14:01.152 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7725:up:replay 2024-09-13T17:14:02.152 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.317+0000 7f9164800640 1 Processor -- start 2024-09-13T17:14:02.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.317+0000 7f9164800640 1 -- start start 2024-09-13T17:14:02.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.318+0000 7f9164800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f915c15a540 0x7f915c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.318+0000 7f9164800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f915c070cc0 0x7f915c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.318+0000 7f9164800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f915c15e100 con 0x7f915c0cabc0 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.318+0000 7f9164800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c0adb00 con 0x7f915c15a540 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.318+0000 7f9164800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f915c071670 con 0x7f915c070cc0 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f915c0cabc0 0x7f915c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34564/0 (socket says 172.21.15.28:34564) 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f915c15a540 0x7f915c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161400640 1 -- 172.21.15.28:0/3595012649 learned_addr learned my addr 172.21.15.28:0/3595012649 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9162800640 1 --2- 172.21.15.28:0/3595012649 >> v2:172.21.15.189:3300/0 conn(0x7f915c070cc0 0x7f915c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161e00640 1 -- 172.21.15.28:0/3595012649 >> v2:172.21.15.189:3300/0 conn(0x7f915c070cc0 msgr2=0x7f915c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161e00640 1 --2- 172.21.15.28:0/3595012649 >> v2:172.21.15.189:3300/0 conn(0x7f915c070cc0 0x7f915c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161e00640 1 -- 172.21.15.28:0/3595012649 >> v1:172.21.15.28:6789/0 conn(0x7f915c0cabc0 legacy=0x7f915c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9161e00640 1 -- 172.21.15.28:0/3595012649 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c15e3e0 con 0x7f915c15a540 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.319+0000 7f9160a00640 1 -- 172.21.15.28:0/3595012649 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079408964 0 0) 0x7f915c15e100 con 0x7f915c0cabc0 2024-09-13T17:14:02.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.320+0000 7f9161e00640 1 --2- 172.21.15.28:0/3595012649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c15a540 0x7f915c06c430 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f915c070900 tx=0x7f915002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c064a3241c95ea3e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9160a00640 1 -- 172.21.15.28:0/3595012649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915002ae50 con 0x7f915c15a540 2024-09-13T17:14:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9160a00640 1 -- 172.21.15.28:0/3595012649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9150032440 con 0x7f915c15a540 2024-09-13T17:14:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9160a00640 1 -- 172.21.15.28:0/3595012649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91500327d0 con 0x7f915c15a540 2024-09-13T17:14:02.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 -- 172.21.15.28:0/3595012649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c15a540 msgr2=0x7f915c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 --2- 172.21.15.28:0/3595012649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c15a540 0x7f915c06c430 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f915c070900 tx=0x7f915002a140 comp rx=0 tx=0).stop 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 -- 172.21.15.28:0/3595012649 shutdown_connections 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 --2- 172.21.15.28:0/3595012649 >> v2:172.21.15.189:3300/0 conn(0x7f915c070cc0 0x7f915c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 --2- 172.21.15.28:0/3595012649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c15a540 0x7f915c06c430 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 -- 172.21.15.28:0/3595012649 >> 172.21.15.28:0/3595012649 conn(0x7f915c069e80 msgr2=0x7f915c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 -- 172.21.15.28:0/3595012649 shutdown_connections 2024-09-13T17:14:02.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.321+0000 7f9164800640 1 -- 172.21.15.28:0/3595012649 wait complete. 2024-09-13T17:14:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.323+0000 7f9164800640 1 Processor -- start 2024-09-13T17:14:02.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.323+0000 7f9164800640 1 -- start start 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f915c070cc0 0x7f915c15bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f915c15a540 0x7f915c1552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c15cd70 con 0x7f915c0cabc0 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f915c15c220 con 0x7f915c15a540 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9164800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f915c15c4d0 con 0x7f915c070cc0 2024-09-13T17:14:02.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9161400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9161400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35752/0 (socket says 172.21.15.28:35752) 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9161400640 1 -- 172.21.15.28:0/1589489471 learned_addr learned my addr 172.21.15.28:0/1589489471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9161e00640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f915c070cc0 0x7f915c15bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.324+0000 7f9162800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f915c15a540 0x7f915c1552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 -- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f915c070cc0 msgr2=0x7f915c15bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f915c070cc0 0x7f915c15bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 -- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f915c15a540 msgr2=0x7f915c1552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f915c15a540 0x7f915c1552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 -- 172.21.15.28:0/1589489471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9150009400 con 0x7f915c0cabc0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9161400640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f914c00c4d0 tx=0x7f914c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c0074e0 con 0x7f915c0cabc0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914c007660 con 0x7f915c0cabc0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914c004a20 con 0x7f915c0cabc0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f915c15c7b0 con 0x7f915c0cabc0 2024-09-13T17:14:02.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.325+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f915c158210 con 0x7f915c0cabc0 2024-09-13T17:14:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.326+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f915c158070 con 0x7f915c0cabc0 2024-09-13T17:14:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.328+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f914c004b80 con 0x7f915c0cabc0 2024-09-13T17:14:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.328+0000 7f9156000640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 0x7f914006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:02.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.329+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f914c0885f0 con 0x7f915c0cabc0 2024-09-13T17:14:02.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.329+0000 7f9161e00640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 0x7f914006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:02.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.330+0000 7f9161e00640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 0x7f914006d0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9150034000 tx=0x7f915002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.333+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f915c158070 con 0x7f915c0cabc0 2024-09-13T17:14:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.547+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f915c158070 con 0x7f915c0cabc0 2024-09-13T17:14:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.548+0000 7f9156000640 1 -- 172.21.15.28:0/1589489471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f915c158070 con 0x7f915c0cabc0 2024-09-13T17:14:02.550 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:02.550 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:02.334870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:rejoin","state_seq":4,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 85 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 msgr2=0x7f914006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 0x7f914006d0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9150034000 tx=0x7f915002e000 comp rx=0 tx=0).stop 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 msgr2=0x7f915c1f3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f914c00c4d0 tx=0x7f914c00c990 comp rx=0 tx=0).stop 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 shutdown_connections 2024-09-13T17:14:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914006ac00 0x7f914006d0c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f915c15a540 0x7f915c1552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f915c0cabc0 0x7f915c1f3230 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 --2- 172.21.15.28:0/1589489471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f915c070cc0 0x7f915c15bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 >> 172.21.15.28:0/1589489471 conn(0x7f915c069e80 msgr2=0x7f915c06f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 shutdown_connections 2024-09-13T17:14:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:02.550+0000 7f9164800640 1 -- 172.21.15.28:0/1589489471 wait complete. 2024-09-13T17:14:02.971 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 85, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:13:42.545599+0000', 'modified': '2024-09-13T17:14:02.334870+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'a', 'rank': 0, 'incarnation': 83, 'state': 'up:rejoin', 'state_seq': 4, 'addr': '172.21.15.28:6835/2200414779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 2200414779}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2200414779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:14:02.972 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7725:up:rejoin 2024-09-13T17:14:03.973 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:04.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.139+0000 7f7ddbe00640 1 Processor -- start 2024-09-13T17:14:04.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.140+0000 7f7ddbe00640 1 -- start start 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7ddbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 0x7f7ddc164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7ddbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ddc0c3850 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7ddbe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ddc15a930 con 0x7f7ddc159440 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7ddbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc159d20 con 0x7f7ddc159820 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7dd3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ddc0b0b50 0x7f7ddc0b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48588/0 (socket says 172.21.15.28:48588) 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7dd3e00640 1 -- 172.21.15.28:0/922700932 learned_addr learned my addr 172.21.15.28:0/922700932 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:04.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.141+0000 7f7ddb400640 1 --2- 172.21.15.28:0/922700932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 0x7f7ddc164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438568739 0 0) 0x7f7ddc0c3850 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc0005730 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968992107 0 0) 0x7f7ddc15a930 con 0x7f7ddc159440 2024-09-13T17:14:04.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ddc0c3850 con 0x7f7ddc159440 2024-09-13T17:14:04.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7ddb400640 1 -- 172.21.15.28:0/922700932 >> v1:172.21.15.189:6789/0 conn(0x7f7ddc159440 legacy=0x7f7ddc0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7ddb400640 1 -- 172.21.15.28:0/922700932 >> v1:172.21.15.28:6789/0 conn(0x7f7ddc0b0b50 legacy=0x7f7ddc0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.142+0000 7f7ddb400640 1 -- 172.21.15.28:0/922700932 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc15a000 con 0x7f7ddc159820 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.143+0000 7f7ddb400640 1 --2- 172.21.15.28:0/922700932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 0x7f7ddc164260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009dc0 tx=0x7f7dd402a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23473da7a729093 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.143+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4007740 con 0x7f7ddc159820 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.143+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd402d050 con 0x7f7ddc159820 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.143+0000 7f7dda000640 1 -- 172.21.15.28:0/922700932 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4007ac0 con 0x7f7ddc159820 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 -- 172.21.15.28:0/922700932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 msgr2=0x7f7ddc164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/922700932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 0x7f7ddc164260 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4009dc0 tx=0x7f7dd402a760 comp rx=0 tx=0).stop 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 -- 172.21.15.28:0/922700932 shutdown_connections 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/922700932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159820 0x7f7ddc164260 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 -- 172.21.15.28:0/922700932 >> 172.21.15.28:0/922700932 conn(0x7f7ddc072e40 msgr2=0x7f7ddc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 -- 172.21.15.28:0/922700932 shutdown_connections 2024-09-13T17:14:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.144+0000 7f7ddbe00640 1 -- 172.21.15.28:0/922700932 wait complete. 2024-09-13T17:14:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.146+0000 7f7ddbe00640 1 Processor -- start 2024-09-13T17:14:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.146+0000 7f7ddbe00640 1 -- start start 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159440 0x7f7ddc0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ddc159820 0x7f7ddc157390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc149560 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc0ad0f0 con 0x7f7ddc159820 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc0ad3d0 con 0x7f7ddc159440 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57278/0 (socket says 172.21.15.28:57278) 2024-09-13T17:14:04.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 -- 172.21.15.28:0/409414013 learned_addr learned my addr 172.21.15.28:0/409414013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7dd3e00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159440 0x7f7ddc0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddb400640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ddc159820 0x7f7ddc157390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 -- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159440 msgr2=0x7f7ddc0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159440 0x7f7ddc0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 -- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ddc159820 msgr2=0x7f7ddc157390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ddc159820 0x7f7ddc157390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.147+0000 7f7ddaa00640 1 -- 172.21.15.28:0/409414013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd4009a70 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7ddaa00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c490 tx=0x7f7dcc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc0052d0 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc005430 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc012870 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ddc0ad6b0 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.148+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ddc0adc70 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.149+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ddc0b22e0 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.150+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7dcc0129d0 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.151+0000 7f7dd2a00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 0x7f7db806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.151+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f7dcc0890d0 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.151+0000 7f7dd3e00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 0x7f7db806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.152+0000 7f7dd3e00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 0x7f7db806d0f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc157150 tx=0x7f7dc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.155+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7dcc054130 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.369+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ddc0b1860 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.370+0000 7f7dd2a00640 1 -- 172.21.15.28:0/409414013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f7dcc057750 con 0x7f7ddc0b0b50 2024-09-13T17:14:04.372 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:04.372 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:04.372 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 msgr2=0x7f7db806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 0x7f7db806d0f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc157150 tx=0x7f7dc4009040 comp rx=0 tx=0).stop 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 msgr2=0x7f7ddc149020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00c490 tx=0x7f7dcc00c950 comp rx=0 tx=0).stop 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 shutdown_connections 2024-09-13T17:14:04.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7db806ac30 0x7f7db806d0f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ddc159820 0x7f7ddc157390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ddc159440 0x7f7ddc0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 --2- 172.21.15.28:0/409414013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ddc0b0b50 0x7f7ddc149020 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 >> 172.21.15.28:0/409414013 conn(0x7f7ddc072e40 msgr2=0x7f7ddc152eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 shutdown_connections 2024-09-13T17:14:04.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.372+0000 7f7ddbe00640 1 -- 172.21.15.28:0/409414013 wait complete. 2024-09-13T17:14:04.792 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 86, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:13:42.545599+0000', 'modified': '2024-09-13T17:14:03.336862+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7725}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7725': {'gid': 7725, 'name': 'a', 'rank': 0, 'incarnation': 83, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6835/2200414779', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 2200414779}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 2200414779}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:14:04.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:14:04.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:14:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.950+0000 7fd915a00640 1 Processor -- start 2024-09-13T17:14:04.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.950+0000 7fd915a00640 1 -- start start 2024-09-13T17:14:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd915a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd910159680 0x7fd91006c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd915a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91006cf20 con 0x7fd9100710a0 2024-09-13T17:14:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd915a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9100aef70 con 0x7fd910159a60 2024-09-13T17:14:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd915a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91006ca20 con 0x7fd910159680 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd90ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd910159680 0x7fd91006c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd90f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd9100710a0 0x7fd91006d3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48600/0 (socket says 172.21.15.28:48600) 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.951+0000 7fd90f400640 1 -- 172.21.15.28:0/4157155165 learned_addr learned my addr 172.21.15.28:0/4157155165 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.952+0000 7fd90ea00640 1 -- 172.21.15.28:0/4157155165 >> v1:172.21.15.189:6789/0 conn(0x7fd910159a60 legacy=0x7fd910070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.952+0000 7fd90ea00640 1 -- 172.21.15.28:0/4157155165 >> v1:172.21.15.28:6789/0 conn(0x7fd9100710a0 legacy=0x7fd91006d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.952+0000 7fd90ea00640 1 -- 172.21.15.28:0/4157155165 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91015a020 con 0x7fd910159680 2024-09-13T17:14:04.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.953+0000 7fd90ea00640 1 --2- 172.21.15.28:0/4157155165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd91006c4e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd9000097a0 tx=0x7fd90002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc0d8c474af0aac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.953+0000 7fd90d600640 1 -- 172.21.15.28:0/4157155165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90002ae50 con 0x7fd910159680 2024-09-13T17:14:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.953+0000 7fd90d600640 1 -- 172.21.15.28:0/4157155165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900032440 con 0x7fd910159680 2024-09-13T17:14:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.953+0000 7fd90d600640 1 -- 172.21.15.28:0/4157155165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000327d0 con 0x7fd910159680 2024-09-13T17:14:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 -- 172.21.15.28:0/4157155165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 msgr2=0x7fd91006c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 --2- 172.21.15.28:0/4157155165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd91006c4e0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fd9000097a0 tx=0x7fd90002a140 comp rx=0 tx=0).stop 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 -- 172.21.15.28:0/4157155165 shutdown_connections 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 --2- 172.21.15.28:0/4157155165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd91006c4e0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 -- 172.21.15.28:0/4157155165 >> 172.21.15.28:0/4157155165 conn(0x7fd9100b1740 msgr2=0x7fd9100b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 -- 172.21.15.28:0/4157155165 shutdown_connections 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.954+0000 7fd915a00640 1 -- 172.21.15.28:0/4157155165 wait complete. 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.956+0000 7fd915a00640 1 Processor -- start 2024-09-13T17:14:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.956+0000 7fd915a00640 1 -- start start 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9100710a0 0x7fd91014f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd910159a60 0x7fd9100cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101517a0 con 0x7fd910159680 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101f3cb0 con 0x7fd9100710a0 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd915a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd910152b60 con 0x7fd910159a60 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57286/0 (socket says 172.21.15.28:57286) 2024-09-13T17:14:04.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 -- 172.21.15.28:0/1760354504 learned_addr learned my addr 172.21.15.28:0/1760354504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90ea00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9100710a0 0x7fd91014f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90f400640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd910159a60 0x7fd9100cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 -- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd910159a60 msgr2=0x7fd9100cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd910159a60 0x7fd9100cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 -- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9100710a0 msgr2=0x7fd91014f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9100710a0 0x7fd91014f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.957+0000 7fd90e000640 1 -- 172.21.15.28:0/1760354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009400 con 0x7fd910159680 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd90e000640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b3f0 tx=0x7fd8fc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd906a00640 1 -- 172.21.15.28:0/1760354504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011830 con 0x7fd910159680 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd906a00640 1 -- 172.21.15.28:0/1760354504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc011990 con 0x7fd910159680 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd906a00640 1 -- 172.21.15.28:0/1760354504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc011d40 con 0x7fd910159680 2024-09-13T17:14:04.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9101f3410 con 0x7fd910159680 2024-09-13T17:14:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.958+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9101f39d0 con 0x7fd910159680 2024-09-13T17:14:04.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.960+0000 7fd906a00640 1 -- 172.21.15.28:0/1760354504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd8fc01a400 con 0x7fd910159680 2024-09-13T17:14:04.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.962+0000 7fd906a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 0x7fd8f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.962+0000 7fd906a00640 1 -- 172.21.15.28:0/1760354504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd8fc0891a0 con 0x7fd910159680 2024-09-13T17:14:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.962+0000 7fd90ea00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 0x7fd8f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.962+0000 7fd918867740 1 Processor -- start 2024-09-13T17:14:04.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 -- start start 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd90ea00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 0x7fd8f006d0c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd900034000 tx=0x7fd90002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x56040263bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402638e90 0x560402688ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604025a4ab0 con 0x560402662b80 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604025a2240 con 0x56040263b6d0 2024-09-13T17:14:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.963+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56040259db30 con 0x560402638e90 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57298/0 (socket says 172.21.15.28:57298) 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 -- 172.21.15.28:0/117859980 learned_addr learned my addr 172.21.15.28:0/117859980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90ea00640 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x56040263bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90e000640 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402638e90 0x560402688ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 -- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402638e90 msgr2=0x560402688ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402638e90 0x560402688ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 -- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 msgr2=0x56040263bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x56040263bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 -- 172.21.15.28:0/117859980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560402568bd0 con 0x560402662b80 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.964+0000 7fd90f400640 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd908002150 tx=0x7fd9080019f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b272c2654ea8e613 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd8fb400640 1 -- 172.21.15.28:0/117859980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908001e30 con 0x560402662b80 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd8fb400640 1 -- 172.21.15.28:0/117859980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080070d0 con 0x560402662b80 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd8fb400640 1 -- 172.21.15.28:0/117859980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908007440 con 0x560402662b80 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 -- 172.21.15.28:0/117859980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 msgr2=0x5604026890e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fd908002150 tx=0x7fd9080019f0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 -- 172.21.15.28:0/117859980 shutdown_connections 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x5604026890e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402638e90 0x560402688ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 --2- 172.21.15.28:0/117859980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x56040263bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 -- 172.21.15.28:0/117859980 >> 172.21.15.28:0/117859980 conn(0x560402637e00 msgr2=0x5604026802b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 -- 172.21.15.28:0/117859980 shutdown_connections 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.965+0000 7fd918867740 1 -- 172.21.15.28:0/117859980 wait complete. 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.966+0000 7fd918867740 1 -- start start 2024-09-13T17:14:04.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.966+0000 7fd918867740 1 Processor -- start 2024-09-13T17:14:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x5604026642e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x560402685f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040253f260 con 0x560402662b80 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040253ef00 con 0x56040263b6d0 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560402304f20 con 0x560402664820 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd918867740 10 client.0 _ll_register_callbacks cb 0x5604025971a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.967+0000 7fd90e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57310/0 (socket says 172.21.15.28:57310) 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 -- 172.21.15.28:0/872296801 learned_addr learned my addr 172.21.15.28:0/872296801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90f400640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x560402685f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90ea00640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x5604026642e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 -- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 msgr2=0x560402685f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x560402685f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 -- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 msgr2=0x5604026642e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x5604026642e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 -- 172.21.15.28:0/872296801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91006cf20 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd90e000640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc057f00 tx=0x7fd8fc058030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4898fd5746190c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd8efe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc06f070 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604025c7cd0 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.968+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd8fc082c00 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8fc087280 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 10 client.7779 fetch_fsmap learned FSMap version 86 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x56040259c1b0 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5604025c7cd0 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd8fc021020 con 0x560402662b80 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 10 client.7779 fetch_fsmap finished waiting for FSMap version 86 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 10 client.7779 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:14:04.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 20 client.7779 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026621c0 0x56040269d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 4 client.7779 mds_command: new command op to 7725 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 --> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604022913c0 con 0x5604026621c0 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.969+0000 7fd90ea00640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026621c0 0x56040269d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.970+0000 7fd90ea00640 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026621c0 0x56040269d200 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a07b1967619b55b server_cookie=ecc25adfe4b2a9be in_seq=0 out_seq=0 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.970+0000 7fd8efe00640 10 client.7779 ms_handle_connect on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd8efe00640 1 -- 172.21.15.28:0/872296801 <== mds.0 v2:172.21.15.28:6833/2200414779 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5604022913c0 con 0x5604026621c0 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd8efe00640 10 client.7779 handle_command_reply: tid=0 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 client.7779 shutdown 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 msgr2=0x560402685a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc057f00 tx=0x7fd8fc058030 comp rx=0 tx=0).stop 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 shutdown_connections 2024-09-13T17:14:04.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026621c0 0x56040269d200 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x560402685f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402662b80 0x560402685a20 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 --2- 172.21.15.28:0/872296801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x5604026642e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 >> 172.21.15.28:0/872296801 conn(0x560402638e90 msgr2=0x560402683770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd8efe00640 0 client.7779 ms_handle_reset on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 shutdown_connections 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 1 -- 172.21.15.28:0/872296801 wait complete. 2024-09-13T17:14:04.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.971+0000 7fd918867740 20 client.7779 trim_cache size 0 max 16384 2024-09-13T17:14:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.987+0000 7fd918867740 1 Processor -- start 2024-09-13T17:14:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.987+0000 7fd918867740 1 -- start start 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x560402664c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402662b80 0x56040265f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604026617b0 con 0x560402664820 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560402692090 con 0x56040263b6d0 2024-09-13T17:14:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560402685aa0 con 0x560402662b80 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd90f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd90f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57314/0 (socket says 172.21.15.28:57314) 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.988+0000 7fd90f400640 1 -- 172.21.15.28:0/220086623 learned_addr learned my addr 172.21.15.28:0/220086623 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90e000640 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402662b80 0x56040265f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90ea00640 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x560402664c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 -- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402662b80 msgr2=0x56040265f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402662b80 0x56040265f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 -- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 msgr2=0x560402664c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x560402664c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 -- 172.21.15.28:0/220086623 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604025d1170 con 0x560402664820 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd90f400640 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd908005c00 tx=0x7fd90800a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa16e27580836921 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd8fbe00640 1 -- 172.21.15.28:0/220086623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908024070 con 0x560402664820 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd8fbe00640 1 -- 172.21.15.28:0/220086623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd908007240 con 0x560402664820 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.989+0000 7fd8fbe00640 1 -- 172.21.15.28:0/220086623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908001e30 con 0x560402664820 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 -- 172.21.15.28:0/220086623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 msgr2=0x560402683770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7fd908005c00 tx=0x7fd90800a000 comp rx=0 tx=0).stop 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 -- 172.21.15.28:0/220086623 shutdown_connections 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560402664820 0x560402683770 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402662b80 0x56040265f7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 --2- 172.21.15.28:0/220086623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56040263b6d0 0x560402664c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 -- 172.21.15.28:0/220086623 >> 172.21.15.28:0/220086623 conn(0x560402638e90 msgr2=0x560402699620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 -- 172.21.15.28:0/220086623 shutdown_connections 2024-09-13T17:14:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.990+0000 7fd918867740 1 -- 172.21.15.28:0/220086623 wait complete. 2024-09-13T17:14:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.991+0000 7fd918867740 1 -- start start 2024-09-13T17:14:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.991+0000 7fd918867740 1 Processor -- start 2024-09-13T17:14:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560402662b80 0x560402690d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x5604026912a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604026671e0 con 0x56040263b6d0 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56040269d780 con 0x560402662b80 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5604026eba20 con 0x560402664820 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd918867740 10 client.0 _ll_register_callbacks cb 0x7fd8fc0542b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57322/0 (socket says 172.21.15.28:57322) 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 -- 172.21.15.28:0/272040211 learned_addr learned my addr 172.21.15.28:0/272040211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90e000640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560402662b80 0x560402690d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90f400640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x5604026912a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 -- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 msgr2=0x5604026912a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x5604026912a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 -- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560402662b80 msgr2=0x560402690d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560402662b80 0x560402690d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.992+0000 7fd90ea00640 1 -- 172.21.15.28:0/272040211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560402568bd0 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd90ea00640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd900035040 tx=0x7fd90002c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a1d30d23c7444c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd8efe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90004d040 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd900053700 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd9080070d0 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9000538c0 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd918867740 10 client.7785 fetch_fsmap learned FSMap version 86 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.993+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x56040253e470 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080070d0 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd900044040 con 0x56040263b6d0 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 10 client.7785 fetch_fsmap finished waiting for FSMap version 86 2024-09-13T17:14:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 10 client.7785 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 20 client.7785 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026eaa80 0x5604026eae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 4 client.7785 mds_command: new command op to 7725 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 --> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5604022a26c0 con 0x5604026eaa80 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd90e000640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026eaa80 0x5604026eae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd90e000640 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026eaa80 0x5604026eae60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7525359ee7339d5e server_cookie=82c540473ceb3b34 in_seq=0 out_seq=0 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.994+0000 7fd8efe00640 10 client.7785 ms_handle_connect on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd8efe00640 1 -- 172.21.15.28:0/272040211 <== mds.0 v2:172.21.15.28:6833/2200414779 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5604022a26c0 con 0x5604026eaa80 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd8efe00640 10 client.7785 handle_command_reply: tid=0 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 client.7785 shutdown 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 msgr2=0x5604026e4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fd900035040 tx=0x7fd90002c070 comp rx=0 tx=0).stop 2024-09-13T17:14:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 shutdown_connections 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x5604026eaa80 0x5604026eae60 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560402664820 0x5604026912a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560402662b80 0x560402690d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 --2- 172.21.15.28:0/272040211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56040263b6d0 0x5604026e4a20 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd8efe00640 0 client.7785 ms_handle_reset on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 >> 172.21.15.28:0/272040211 conn(0x560402638e90 msgr2=0x56040269f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 shutdown_connections 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 1 -- 172.21.15.28:0/272040211 wait complete. 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.995+0000 7fd918867740 20 client.7785 trim_cache size 0 max 16384 2024-09-13T17:14:05.001 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7725, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.670256674, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 86, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 99, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 99, 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 10.373998783999999 2024-09-13T17:14:05.002 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:14:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 msgr2=0x7fd8f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 0x7fd8f006d0c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fd900034000 tx=0x7fd90002e000 comp rx=0 tx=0).stop 2024-09-13T17:14:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 msgr2=0x7fd910156fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc00b3f0 tx=0x7fd8fc00b8b0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 shutdown_connections 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f006ac00 0x7fd8f006d0c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd910159a60 0x7fd9100cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd910159680 0x7fd910156fa0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 --2- 172.21.15.28:0/1760354504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9100710a0 0x7fd91014f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 >> 172.21.15.28:0/1760354504 conn(0x7fd9100b1740 msgr2=0x7fd91006da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 shutdown_connections 2024-09-13T17:14:05.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:04.996+0000 7fd915a00640 1 -- 172.21.15.28:0/1760354504 wait complete. 2024-09-13T17:14:05.014 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_size 20 2024-09-13T17:14:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.171+0000 7f28c8c00640 1 Processor -- start 2024-09-13T17:14:05.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.171+0000 7f28c8c00640 1 -- start start 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.172+0000 7f28c8c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f28c415a780 0x7f28c4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.172+0000 7f28c8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c406bb40 0x7f28c406bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c406c8b0 con 0x7f28c415ab60 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c8c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c406cb90 con 0x7f28c415a780 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40cc980 con 0x7f28c406bb40 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28b9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f28c415a780 0x7f28c4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28b9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57326/0 (socket says 172.21.15.28:57326) 2024-09-13T17:14:05.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28b9400640 1 -- 172.21.15.28:0/2466198270 learned_addr learned my addr 172.21.15.28:0/2466198270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c2800640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c406bb40 0x7f28c406bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 -- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c406bb40 msgr2=0x7f28c406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c406bb40 0x7f28c406bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 -- 172.21.15.28:0/2466198270 >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 msgr2=0x7f28c40712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 --2- 172.21.15.28:0/2466198270 >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28c1e00640 1 -- 172.21.15.28:0/2466198270 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40ccc60 con 0x7f28c415a780 2024-09-13T17:14:05.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.173+0000 7f28b9400640 1 --2- 172.21.15.28:0/2466198270 >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:14:05.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.174+0000 7f28c1e00640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415a780 0x7f28c4070d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f28c40b1330 tx=0x7f28b402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f25cdeb394d181e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.175+0000 7f28c1400640 1 -- 172.21.15.28:0/2466198270 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b402ae60 con 0x7f28c415a780 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.175+0000 7f28c1400640 1 -- 172.21.15.28:0/2466198270 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b4032440 con 0x7f28c415a780 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.175+0000 7f28c1400640 1 -- 172.21.15.28:0/2466198270 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b40327d0 con 0x7f28c415a780 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.175+0000 7f28c8c00640 1 -- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415a780 msgr2=0x7f28c4070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.175+0000 7f28c8c00640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415a780 0x7f28c4070d80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f28c40b1330 tx=0x7f28b402a110 comp rx=0 tx=0).stop 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 -- 172.21.15.28:0/2466198270 shutdown_connections 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 --2- 172.21.15.28:0/2466198270 >> v2:172.21.15.28:3300/0 conn(0x7f28c415ab60 0x7f28c40712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c406bb40 0x7f28c406bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 --2- 172.21.15.28:0/2466198270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415a780 0x7f28c4070d80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 -- 172.21.15.28:0/2466198270 >> 172.21.15.28:0/2466198270 conn(0x7f28c40b0270 msgr2=0x7f28c40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 -- 172.21.15.28:0/2466198270 shutdown_connections 2024-09-13T17:14:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.176+0000 7f28c8c00640 1 -- 172.21.15.28:0/2466198270 wait complete. 2024-09-13T17:14:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.177+0000 7f28c8c00640 1 Processor -- start 2024-09-13T17:14:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.177+0000 7f28c8c00640 1 -- start start 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c415a780 0x7f28c40a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415ab60 0x7f28c40ab2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41494d0 con 0x7f28c406bb40 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40a8980 con 0x7f28c415ab60 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.178+0000 7f28c8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41544b0 con 0x7f28c415a780 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57328/0 (socket says 172.21.15.28:57328) 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 -- 172.21.15.28:0/4056538435 learned_addr learned my addr 172.21.15.28:0/4056538435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28b9400640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c415a780 0x7f28c40a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c2800640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415ab60 0x7f28c40ab2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 -- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c415a780 msgr2=0x7f28c40a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c415a780 0x7f28c40a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 -- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415ab60 msgr2=0x7f28c40ab2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415ab60 0x7f28c40ab2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 -- 172.21.15.28:0/4056538435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac0094d0 con 0x7f28c406bb40 2024-09-13T17:14:05.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28c1e00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.179+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b403d8f0 con 0x7f28c406bb40 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.180+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28b403da70 con 0x7f28c406bb40 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.180+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b4009400 con 0x7f28c406bb40 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.180+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b403de20 con 0x7f28c406bb40 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.180+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c4154a50 con 0x7f28c406bb40 2024-09-13T17:14:05.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.181+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c40cf2f0 con 0x7f28c406bb40 2024-09-13T17:14:05.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.182+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28b4039040 con 0x7f28c406bb40 2024-09-13T17:14:05.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.182+0000 7f28baa00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 0x7f28a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.183+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f28b40a9c10 con 0x7f28c406bb40 2024-09-13T17:14:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.183+0000 7f28b9400640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 0x7f28a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.184+0000 7f28b9400640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 0x7f28a006d0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28c40a8e00 tx=0x7f28ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.187+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28b4074e20 con 0x7f28c406bb40 2024-09-13T17:14:05.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.367+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_size}] v 0) v1 -- 0x7f28c406d6c0 con 0x7f28c406bb40 2024-09-13T17:14:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.371+0000 7f28baa00640 1 -- 172.21.15.28:0/4056538435 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_size}]=0 v27)=0 v27) v1 ==== 117+0+0 (secure 0 0 0) 0x7f28b4078440 con 0x7f28c406bb40 2024-09-13T17:14:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 msgr2=0x7f28a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 0x7f28a006d0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f28c40a8e00 tx=0x7f28ac008040 comp rx=0 tx=0).stop 2024-09-13T17:14:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 msgr2=0x7f28c4148f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f28b4000c00 tx=0x7f28b40046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 shutdown_connections 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28a006ac00 0x7f28a006d0c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28c415ab60 0x7f28c40ab2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28c415a780 0x7f28c40a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 --2- 172.21.15.28:0/4056538435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28c406bb40 0x7f28c4148f90 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 >> 172.21.15.28:0/4056538435 conn(0x7f28c40b0270 msgr2=0x7f28c40cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 shutdown_connections 2024-09-13T17:14:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.373+0000 7f28c8c00640 1 -- 172.21.15.28:0/4056538435 wait complete. 2024-09-13T17:14:05.391 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_merge_size 5 2024-09-13T17:14:05.431 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-13T17:14:05.429+0000 7f8b64600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:14:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.548+0000 7f742be00640 1 Processor -- start 2024-09-13T17:14:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.548+0000 7f742be00640 1 -- start start 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f742c070e30 0x7f742c071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 0x7f742c0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 0x7f742c0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c159e10 con 0x7f742c070e30 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c15a0f0 con 0x7f742c071210 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c15a3d0 con 0x7f742c072140 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f742c070e30 0x7f742c071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f742c070e30 0x7f742c071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57344/0 (socket says 172.21.15.28:57344) 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.549+0000 7f742aa00640 1 -- 172.21.15.28:0/3825128693 learned_addr learned my addr 172.21.15.28:0/3825128693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742b400640 1 --2- 172.21.15.28:0/3825128693 >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 0x7f742c0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f7422000640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 0x7f742c0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742aa00640 1 -- 172.21.15.28:0/3825128693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 msgr2=0x7f742c0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742aa00640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 0x7f742c0cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742aa00640 1 -- 172.21.15.28:0/3825128693 >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 msgr2=0x7f742c0cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742aa00640 1 --2- 172.21.15.28:0/3825128693 >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 0x7f742c0cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742aa00640 1 -- 172.21.15.28:0/3825128693 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c15a6b0 con 0x7f742c070e30 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f742b400640 1 --2- 172.21.15.28:0/3825128693 >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 0x7f742c0cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.550+0000 7f7422000640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 0x7f742c0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742aa00640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c070e30 0x7f742c071c00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f742c0b3840 tx=0x7f741c02a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9c9cefdf945a1c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742a000640 1 -- 172.21.15.28:0/3825128693 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c032440 con 0x7f742c070e30 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742a000640 1 -- 172.21.15.28:0/3825128693 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c0325a0 con 0x7f742c070e30 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742a000640 1 -- 172.21.15.28:0/3825128693 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c032930 con 0x7f742c070e30 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742be00640 1 -- 172.21.15.28:0/3825128693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c070e30 msgr2=0x7f742c071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.551+0000 7f742be00640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c070e30 0x7f742c071c00 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f742c0b3840 tx=0x7f741c02a1d0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 -- 172.21.15.28:0/3825128693 shutdown_connections 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 --2- 172.21.15.28:0/3825128693 >> v2:172.21.15.189:3300/0 conn(0x7f742c071210 0x7f742c0cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c072140 0x7f742c0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 --2- 172.21.15.28:0/3825128693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c070e30 0x7f742c071c00 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 -- 172.21.15.28:0/3825128693 >> 172.21.15.28:0/3825128693 conn(0x7f742c069e80 msgr2=0x7f742c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 -- 172.21.15.28:0/3825128693 shutdown_connections 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.552+0000 7f742be00640 1 -- 172.21.15.28:0/3825128693 wait complete. 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.553+0000 7f742be00640 1 Processor -- start 2024-09-13T17:14:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.553+0000 7f742be00640 1 -- start start 2024-09-13T17:14:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c070e30 0x7f742c1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c071210 0x7f742c145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c142600 con 0x7f742c072140 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c0cb700 con 0x7f742c071210 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c143540 con 0x7f742c070e30 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c070e30 0x7f742c1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57348/0 (socket says 172.21.15.28:57348) 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 -- 172.21.15.28:0/2577136705 learned_addr learned my addr 172.21.15.28:0/2577136705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f7422000640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c071210 0x7f742c145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 -- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c070e30 msgr2=0x7f742c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c070e30 0x7f742c1454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 -- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c071210 msgr2=0x7f742c145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c071210 0x7f742c145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.555+0000 7f742b400640 1 -- 172.21.15.28:0/2577136705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741c0094d0 con 0x7f742c072140 2024-09-13T17:14:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f742b400640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f742400cef0 tx=0x7f742400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742400d840 con 0x7f742c072140 2024-09-13T17:14:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742400d9a0 con 0x7f742c072140 2024-09-13T17:14:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742400dd50 con 0x7f742c072140 2024-09-13T17:14:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c142ca0 con 0x7f742c072140 2024-09-13T17:14:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.556+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c143260 con 0x7f742c072140 2024-09-13T17:14:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.557+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c0cb870 con 0x7f742c072140 2024-09-13T17:14:05.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.558+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f742401e3b0 con 0x7f742c072140 2024-09-13T17:14:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.559+0000 7f7423400640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 0x7f740806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.559+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f7424089cf0 con 0x7f742c072140 2024-09-13T17:14:05.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.560+0000 7f742aa00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 0x7f740806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.560+0000 7f742aa00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 0x7f740806d0f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f741c034000 tx=0x7f741c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.563+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f742408e070 con 0x7f742c072140 2024-09-13T17:14:05.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.740+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_merge_size}] v 0) v1 -- 0x7f742c06f720 con 0x7f742c072140 2024-09-13T17:14:05.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.744+0000 7f7423400640 1 -- 172.21.15.28:0/2577136705 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_merge_size}]=0 v28)=0 v28) v1 ==== 116+0+0 (secure 0 0 0) 0x7f742401a020 con 0x7f742c072140 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 msgr2=0x7f740806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 0x7f740806d0f0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f741c034000 tx=0x7f741c02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 msgr2=0x7f742c158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f742400cef0 tx=0x7f742400a710 comp rx=0 tx=0).stop 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 shutdown_connections 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f740806ac30 0x7f740806d0f0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f742c072140 0x7f742c158500 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f742c071210 0x7f742c145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 --2- 172.21.15.28:0/2577136705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f742c070e30 0x7f742c1454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 >> 172.21.15.28:0/2577136705 conn(0x7f742c069e80 msgr2=0x7f742c0b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 shutdown_connections 2024-09-13T17:14:05.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.747+0000 7f742be00640 1 -- 172.21.15.28:0/2577136705 wait complete. 2024-09-13T17:14:05.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_bal_split_bits 1 2024-09-13T17:14:05.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.961+0000 7f36dce00640 1 Processor -- start 2024-09-13T17:14:05.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.962+0000 7f36dce00640 1 -- start start 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36dce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36d80cd7c0 0x7f36d80c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36dce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36dce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d815ef70 con 0x7f36d8143f70 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36dce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d80c4320 con 0x7f36d80cd7c0 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36dce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d815e1b0 con 0x7f36d80c3ba0 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36d80cd7c0 0x7f36d80c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36d80cd7c0 0x7f36d80c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57354/0 (socket says 172.21.15.28:57354) 2024-09-13T17:14:05.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36cfe00640 1 -- 172.21.15.28:0/3967927357 learned_addr learned my addr 172.21.15.28:0/3967927357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.963+0000 7f36d7e00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.964+0000 7f36cfe00640 1 -- 172.21.15.28:0/3967927357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 msgr2=0x7f36d8070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.964+0000 7f36cfe00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d8070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.964+0000 7f36cfe00640 1 -- 172.21.15.28:0/3967927357 >> v1:172.21.15.189:6789/0 conn(0x7f36d8143f70 legacy=0x7f36d80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:05.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.964+0000 7f36cfe00640 1 -- 172.21.15.28:0/3967927357 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d815e490 con 0x7f36d80cd7c0 2024-09-13T17:14:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.964+0000 7f36d7e00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d8070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:05.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36cfe00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d80c3660 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009870 tx=0x7f36c002a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4675878f8258ecda server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36d6a00640 1 -- 172.21.15.28:0/3967927357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0032440 con 0x7f36d80cd7c0 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36d6a00640 1 -- 172.21.15.28:0/3967927357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c00325a0 con 0x7f36d80cd7c0 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36d6a00640 1 -- 172.21.15.28:0/3967927357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0032930 con 0x7f36d80cd7c0 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36dce00640 1 -- 172.21.15.28:0/3967927357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 msgr2=0x7f36d80c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.965+0000 7f36dce00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d80c3660 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009870 tx=0x7f36c002a200 comp rx=0 tx=0).stop 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 -- 172.21.15.28:0/3967927357 shutdown_connections 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d8070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 --2- 172.21.15.28:0/3967927357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d80c3660 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 -- 172.21.15.28:0/3967927357 >> 172.21.15.28:0/3967927357 conn(0x7f36d8069e80 msgr2=0x7f36d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 -- 172.21.15.28:0/3967927357 shutdown_connections 2024-09-13T17:14:05.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.966+0000 7f36dce00640 1 -- 172.21.15.28:0/3967927357 wait complete. 2024-09-13T17:14:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.967+0000 7f36dce00640 1 Processor -- start 2024-09-13T17:14:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.967+0000 7f36dce00640 1 -- start start 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d815ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36d8143f70 0x7f36d80c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f3040 con 0x7f36d80cd7c0 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d815b370 con 0x7f36d8143f70 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36dce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f5da0 con 0x7f36d80c3ba0 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.968+0000 7f36cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57356/0 (socket says 172.21.15.28:57356) 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 -- 172.21.15.28:0/4249703151 learned_addr learned my addr 172.21.15.28:0/4249703151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36d7400640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d815ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36d7e00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36d8143f70 0x7f36d80c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 -- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 msgr2=0x7f36d815ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d815ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 -- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36d8143f70 msgr2=0x7f36d80c9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36d8143f70 0x7f36d80c9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:05.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 -- 172.21.15.28:0/4249703151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c00094d0 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.969+0000 7f36cfe00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009870 tx=0x7f36c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.970+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0030b00 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.970+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0030c60 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.970+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c003d420 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.970+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d81f6080 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.970+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d80cb8d0 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.971+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d8070440 con 0x7f36d80cd7c0 2024-09-13T17:14:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.972+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f36c0039040 con 0x7f36d80cd7c0 2024-09-13T17:14:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.973+0000 7f36cf400640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 0x7f36b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.973+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f36c00aa1b0 con 0x7f36d80cd7c0 2024-09-13T17:14:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.973+0000 7f36d7400640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 0x7f36b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:05.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.974+0000 7f36d7400640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 0x7f36b406d0c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006fd0 tx=0x7f36c8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:05.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:05.977+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f36c00752c0 con 0x7f36d80cd7c0 2024-09-13T17:14:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.156+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_bal_split_bits}] v 0) v1 -- 0x7f36d80cbac0 con 0x7f36d80cd7c0 2024-09-13T17:14:06.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.160+0000 7f36cf400640 1 -- 172.21.15.28:0/4249703151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_bal_split_bits}]=0 v29)=0 v29) v1 ==== 116+0+0 (secure 0 0 0) 0x7f36c00788e0 con 0x7f36d80cd7c0 2024-09-13T17:14:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 msgr2=0x7f36b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 0x7f36b406d0c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f36c8006fd0 tx=0x7f36c8009060 comp rx=0 tx=0).stop 2024-09-13T17:14:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 msgr2=0x7f36d81f5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009870 tx=0x7f36c00046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 shutdown_connections 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36b406ac00 0x7f36b406d0c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36d8143f70 0x7f36d80c9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36d80cd7c0 0x7f36d81f5570 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 --2- 172.21.15.28:0/4249703151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36d80c3ba0 0x7f36d815ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 >> 172.21.15.28:0/4249703151 conn(0x7f36d8069e80 msgr2=0x7f36d80cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 shutdown_connections 2024-09-13T17:14:06.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.162+0000 7f36dce00640 1 -- 172.21.15.28:0/4249703151 wait complete. 2024-09-13T17:14:06.180 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.333+0000 7fcc17200640 1 Processor -- start 2024-09-13T17:14:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.333+0000 7fcc17200640 1 -- start start 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc17200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc1006f970 0x7fcc1015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc17200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc1006c2d0 0x7fcc1006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc17200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1006cc70 con 0x7fcc101592a0 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc17200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1015b0b0 con 0x7fcc1006c2d0 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc17200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc10159d90 con 0x7fcc1006f970 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc1006c2d0 0x7fcc1006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc1006c2d0 0x7fcc1006c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57372/0 (socket says 172.21.15.28:57372) 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 -- 172.21.15.28:0/3657569320 learned_addr learned my addr 172.21.15.28:0/3657569320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc07400640 1 --2- 172.21.15.28:0/3657569320 >> v2:172.21.15.189:3300/0 conn(0x7fcc1006f970 0x7fcc1015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 -- 172.21.15.28:0/3657569320 >> v2:172.21.15.189:3300/0 conn(0x7fcc1006f970 msgr2=0x7fcc1015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 --2- 172.21.15.28:0/3657569320 >> v2:172.21.15.189:3300/0 conn(0x7fcc1006f970 0x7fcc1015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 -- 172.21.15.28:0/3657569320 >> v1:172.21.15.189:6789/0 conn(0x7fcc101592a0 legacy=0x7fcc10159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.335+0000 7fcc15200640 1 -- 172.21.15.28:0/3657569320 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100b2040 con 0x7fcc1006c2d0 2024-09-13T17:14:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.336+0000 7fcc15200640 1 --2- 172.21.15.28:0/3657569320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc1006c730 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006ac10 tx=0x7fcc0802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3163c86f5669b4b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc0f400640 1 -- 172.21.15.28:0/3657569320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08031600 con 0x7fcc1006c2d0 2024-09-13T17:14:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc0f400640 1 -- 172.21.15.28:0/3657569320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0802d050 con 0x7fcc1006c2d0 2024-09-13T17:14:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc0f400640 1 -- 172.21.15.28:0/3657569320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08031980 con 0x7fcc1006c2d0 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 -- 172.21.15.28:0/3657569320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 msgr2=0x7fcc1006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 --2- 172.21.15.28:0/3657569320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc1006c730 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fcc1006ac10 tx=0x7fcc0802a730 comp rx=0 tx=0).stop 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 -- 172.21.15.28:0/3657569320 shutdown_connections 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 --2- 172.21.15.28:0/3657569320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc1006c730 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 --2- 172.21.15.28:0/3657569320 >> v2:172.21.15.189:3300/0 conn(0x7fcc1006f970 0x7fcc1015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.337+0000 7fcc17200640 1 -- 172.21.15.28:0/3657569320 >> 172.21.15.28:0/3657569320 conn(0x7fcc10069e80 msgr2=0x7fcc1006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.338+0000 7fcc17200640 1 -- 172.21.15.28:0/3657569320 shutdown_connections 2024-09-13T17:14:06.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.338+0000 7fcc17200640 1 -- 172.21.15.28:0/3657569320 wait complete. 2024-09-13T17:14:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.339+0000 7fcc17200640 1 Processor -- start 2024-09-13T17:14:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.339+0000 7fcc17200640 1 -- start start 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 0x7fcc1014fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc101592a0 0x7fcc100cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc101510c0 con 0x7fcc1006c2d0 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10150440 con 0x7fcc101592a0 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.340+0000 7fcc17200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc10150720 con 0x7fcc1006f970 2024-09-13T17:14:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57380/0 (socket says 172.21.15.28:57380) 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 -- 172.21.15.28:0/857848267 learned_addr learned my addr 172.21.15.28:0/857848267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc07400640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 0x7fcc1014fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 -- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 msgr2=0x7fcc1014fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 0x7fcc1014fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc15200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc101592a0 0x7fcc100cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 -- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc101592a0 msgr2=0x7fcc100cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc101592a0 0x7fcc100cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 -- 172.21.15.28:0/857848267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc08009b30 con 0x7fcc1006c2d0 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc07400640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 0x7fcc1014fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.341+0000 7fcc0fe00640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c400 tx=0x7fcc0000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.342+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc000052d0 con 0x7fcc1006c2d0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.342+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00005430 con 0x7fcc1006c2d0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.342+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc10150a00 con 0x7fcc1006c2d0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.342+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00003830 con 0x7fcc1006c2d0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.342+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc10152140 con 0x7fcc1006c2d0 2024-09-13T17:14:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.343+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc10151200 con 0x7fcc1006c2d0 2024-09-13T17:14:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.344+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcc00003990 con 0x7fcc1006c2d0 2024-09-13T17:14:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.345+0000 7fcc0cc00640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 0x7fcbec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.345+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fcc000889a0 con 0x7fcc1006c2d0 2024-09-13T17:14:06.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.346+0000 7fcc07400640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 0x7fcbec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.347+0000 7fcc07400640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 0x7fcbec06d0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcc101f5910 tx=0x7fcbf8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.349+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcc10151200 con 0x7fcc1006c2d0 2024-09-13T17:14:06.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-13T17:14:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.564+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc101522d0 con 0x7fcc1006c2d0 2024-09-13T17:14:06.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.565+0000 7fcc0cc00640 1 -- 172.21.15.28:0/857848267 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fcc00017090 con 0x7fcc1006c2d0 2024-09-13T17:14:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:06.567 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 msgr2=0x7fcbec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 0x7fcbec06d0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcc101f5910 tx=0x7fcbf8008040 comp rx=0 tx=0).stop 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 msgr2=0x7fcc100ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000c400 tx=0x7fcc0000c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 shutdown_connections 2024-09-13T17:14:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbec06ac00 0x7fcbec06d0c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc101592a0 0x7fcc100cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc1006f970 0x7fcc1014fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 --2- 172.21.15.28:0/857848267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc1006c2d0 0x7fcc100ce610 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 >> 172.21.15.28:0/857848267 conn(0x7fcc10069e80 msgr2=0x7fcc10156950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 shutdown_connections 2024-09-13T17:14:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.567+0000 7fcc17200640 1 -- 172.21.15.28:0/857848267 wait complete. 2024-09-13T17:14:06.585 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 1099511627776 2024-09-13T17:14:06.756 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:13:51.205107+0000","btime":"2024-09-13T17:13:50.061139+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:13:51.205107+0000","atime":"2024-09-13T17:13:50.061139+0000","time_warp_seq":0,"change_attr":21,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:13:51.205107+0000","num_files":21,"num_subdirs":0,"change_attr":21},"rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"accounted_rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"version":128,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{},"nref":0,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:14:06.757 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:13:50.061139+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:13:50.061139+0000", "change_attr": 21, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:13:51.205107+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 21, "mtime": "2024-09-13T17:13:51.205107+0000", "num_files": 21, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:13:51.205107+0000", "nestlock": {}, "nlink": 1, "nref": 0, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": {}, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 128, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:14:06.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /testdir recursive 2024-09-13T17:14:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.914+0000 7fd7b3000640 1 Processor -- start 2024-09-13T17:14:06.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.914+0000 7fd7b3000640 1 -- start start 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7b3000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7ac0b2800 0x7fd7ac1641e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7b3000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac15a4c0 con 0x7fd7ac159970 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7b3000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7ac071580 con 0x7fd7ac159590 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7b3000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7ac071390 con 0x7fd7ac0b2800 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7a3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd7ac159970 0x7fd7ac0b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48610/0 (socket says 172.21.15.28:48610) 2024-09-13T17:14:06.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.915+0000 7fd7a3400640 1 -- 172.21.15.28:0/2867745775 learned_addr learned my addr 172.21.15.28:0/2867745775 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:06.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2867745775 >> v2:172.21.15.189:3300/0 conn(0x7fd7ac0b2800 0x7fd7ac1641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411077878 0 0) 0x7fd7ac15a4c0 con 0x7fd7ac159970 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd790005730 con 0x7fd7ac159970 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621218369 0 0) 0x7fd790005730 con 0x7fd7ac159970 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 >> v2:172.21.15.189:3300/0 conn(0x7fd7ac0b2800 msgr2=0x7fd7ac1641e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 --2- 172.21.15.28:0/2867745775 >> v2:172.21.15.189:3300/0 conn(0x7fd7ac0b2800 0x7fd7ac1641e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 >> v1:172.21.15.189:6789/0 conn(0x7fd7ac159590 legacy=0x7fd7ac070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac159e70 con 0x7fd7ac159970 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fd794001670 con 0x7fd7ac159970 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 >> v1:172.21.15.28:6789/0 conn(0x7fd7ac159970 legacy=0x7fd7ac0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd790005b80 0x7fd790005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 0x7fd790006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd790007390 0x7fd79000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79000e200 con 0x7fd7900069d0 2024-09-13T17:14:06.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd79000e370 con 0x7fd790005b80 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.916+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79000e530 con 0x7fd790007390 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 0x7fd790006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd790005b80 0x7fd790005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd790007390 0x7fd79000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 -- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd790007390 msgr2=0x7fd79000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd790007390 0x7fd79000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 -- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd790005b80 msgr2=0x7fd790005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd790005b80 0x7fd790005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7a3400640 1 -- 172.21.15.28:0/2867745775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790005730 con 0x7fd7900069d0 2024-09-13T17:14:06.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.917+0000 7fd7b1000640 1 -- 172.21.15.28:0/2867745775 reap_dead start 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.918+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 0x7fd790006e50 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd790007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c849bedd21ffa0d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.918+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794005ad0 con 0x7fd7900069d0 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.918+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79402d070 con 0x7fd7900069d0 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.918+0000 7fd7ab400640 1 -- 172.21.15.28:0/2867745775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794007070 con 0x7fd7900069d0 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 -- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 msgr2=0x7fd790006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 0x7fd790006e50 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c240 tx=0x7fd790007a00 comp rx=0 tx=0).stop 2024-09-13T17:14:06.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 -- 172.21.15.28:0/2867745775 shutdown_connections 2024-09-13T17:14:06.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 --2- 172.21.15.28:0/2867745775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7900069d0 0x7fd790006e50 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 -- 172.21.15.28:0/2867745775 >> 172.21.15.28:0/2867745775 conn(0x7fd7ac069e80 msgr2=0x7fd7ac06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 -- 172.21.15.28:0/2867745775 shutdown_connections 2024-09-13T17:14:06.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.919+0000 7fd7b3000640 1 -- 172.21.15.28:0/2867745775 wait complete. 2024-09-13T17:14:06.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.921+0000 7fd7b3000640 1 Processor -- start 2024-09-13T17:14:06.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.921+0000 7fd7b3000640 1 -- start start 2024-09-13T17:14:06.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7900069d0 0x7fd7ac14ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7ac152b00 0x7fd7ac150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac158090 con 0x7fd7ac158770 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac152f80 con 0x7fd7ac152b00 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b3000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7ac1571c0 con 0x7fd7900069d0 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57400/0 (socket says 172.21.15.28:57400) 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7a3400640 1 -- 172.21.15.28:0/4253438656 learned_addr learned my addr 172.21.15.28:0/4253438656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7b1000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7ac152b00 0x7fd7ac150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.922+0000 7fd7abe00640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7900069d0 0x7fd7ac14ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 -- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7900069d0 msgr2=0x7fd7ac14ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7900069d0 0x7fd7ac14ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 -- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7ac152b00 msgr2=0x7fd7ac150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7ac152b00 0x7fd7ac150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 -- 172.21.15.28:0/4253438656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ac14f170 con 0x7fd7ac158770 2024-09-13T17:14:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a3400640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd79402ad00 tx=0x7fd794001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a8c00640 1 -- 172.21.15.28:0/4253438656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd794004bf0 con 0x7fd7ac158770 2024-09-13T17:14:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a8c00640 1 -- 172.21.15.28:0/4253438656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd794004d90 con 0x7fd7ac158770 2024-09-13T17:14:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7a8c00640 1 -- 172.21.15.28:0/4253438656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7940072b0 con 0x7fd7ac158770 2024-09-13T17:14:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7ac157410 con 0x7fd7ac158770 2024-09-13T17:14:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.923+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac1579d0 con 0x7fd7ac158770 2024-09-13T17:14:06.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.925+0000 7fd7a8c00640 1 -- 172.21.15.28:0/4253438656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd79402f030 con 0x7fd7ac158770 2024-09-13T17:14:06.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.926+0000 7fd7a8c00640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 0x7fd78806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.926+0000 7fd7a8c00640 1 -- 172.21.15.28:0/4253438656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd7940a90b0 con 0x7fd7ac158770 2024-09-13T17:14:06.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.927+0000 7fd7abe00640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 0x7fd78806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.927+0000 7fd7b5e59740 1 Processor -- start 2024-09-13T17:14:06.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.927+0000 7fd7b5e59740 1 -- start start 2024-09-13T17:14:06.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7abe00640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 0x7fd78806d0c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac068da0 tx=0x7fd79c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293c920 0x55653293f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532939750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565328fa5e0 con 0x55653293f160 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565328a8df0 con 0x55653293c920 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565328a6580 con 0x5565329408f0 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57412/0 (socket says 172.21.15.28:57412) 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.928+0000 7fd7abe00640 1 -- 172.21.15.28:0/661447813 learned_addr learned my addr 172.21.15.28:0/661447813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7a3400640 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293c920 0x55653293f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7b1000640 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532939750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 -- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 msgr2=0x556532939750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532939750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 -- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293c920 msgr2=0x55653293f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293c920 0x55653293f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 -- 172.21.15.28:0/661447813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565328690e0 con 0x55653293f160 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7abe00640 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd79c012a30 tx=0x7fd79c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d72c3ea9d6ef753 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7a0c00640 1 -- 172.21.15.28:0/661447813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c015790 con 0x55653293f160 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7a0c00640 1 -- 172.21.15.28:0/661447813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c0158f0 con 0x55653293f160 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.929+0000 7fd7a0c00640 1 -- 172.21.15.28:0/661447813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd79c015c60 con 0x55653293f160 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 -- 172.21.15.28:0/661447813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 msgr2=0x556532984900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fd79c012a30 tx=0x7fd79c015000 comp rx=0 tx=0).stop 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 -- 172.21.15.28:0/661447813 shutdown_connections 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532939750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293c920 0x55653293f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/661447813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55653293f160 0x556532984900 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 -- 172.21.15.28:0/661447813 >> 172.21.15.28:0/661447813 conn(0x55653293b920 msgr2=0x556532983dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 -- 172.21.15.28:0/661447813 shutdown_connections 2024-09-13T17:14:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.930+0000 7fd7b5e59740 1 -- 172.21.15.28:0/661447813 wait complete. 2024-09-13T17:14:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.931+0000 7fd7b5e59740 1 -- start start 2024-09-13T17:14:06.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.931+0000 7fd7b5e59740 1 Processor -- start 2024-09-13T17:14:06.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x556532968670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532981440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565328a1e70 con 0x5565329632f0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565328430a0 con 0x55653293f160 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556532842d40 con 0x5565329408f0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b5e59740 10 client.0 _ll_register_callbacks cb 0x5565328688c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57414/0 (socket says 172.21.15.28:57414) 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.932+0000 7fd7b1000640 1 -- 172.21.15.28:0/2842051072 learned_addr learned my addr 172.21.15.28:0/2842051072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x556532968670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532981440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 -- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 msgr2=0x556532981440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532981440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 -- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 msgr2=0x556532968670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x556532968670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 -- 172.21.15.28:0/2842051072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7ac071580 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x556532968670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0b39d0 tx=0x7fd7a4015760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36152f247941512 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd799600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a400e040 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556532893d10 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.933+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd7a4015e80 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7a4003990 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 10 client.7821 fetch_fsmap learned FSMap version 86 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x5565328a04f0 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556532893d10 con 0x5565329632f0 2024-09-13T17:14:06.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd7a4003af0 con 0x5565329632f0 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 10 client.7821 fetch_fsmap finished waiting for FSMap version 86 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 10 client.7821 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 20 client.7821 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532962c20 0x5565329394e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 4 client.7821 mds_command: new command op to 7725 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 --> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556532595700 con 0x556532962c20 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.934+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532962c20 0x5565329394e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.935+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532962c20 0x5565329394e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a7faf123f7fd227 server_cookie=2876a3f66833358f in_seq=0 out_seq=0 2024-09-13T17:14:06.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.935+0000 7fd799600640 10 client.7821 ms_handle_connect on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.935+0000 7fd799600640 1 -- 172.21.15.28:0/2842051072 <== mds.0 v2:172.21.15.28:6833/2200414779 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x556532595700 con 0x556532962c20 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.935+0000 7fd799600640 10 client.7821 handle_command_reply: tid=0 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 client.7821 shutdown 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 msgr2=0x55653297db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0b39d0 tx=0x7fd7a4015760 comp rx=0 tx=0).stop 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 shutdown_connections 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653297db20 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532962c20 0x5565329394e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x556532981440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2842051072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x556532968670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd799600640 0 client.7821 ms_handle_reset on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:06.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 >> 172.21.15.28:0/2842051072 conn(0x55653293c920 msgr2=0x556532981e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 shutdown_connections 2024-09-13T17:14:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2842051072 wait complete. 2024-09-13T17:14:06.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.936+0000 7fd7b5e59740 20 client.7821 trim_cache size 0 max 16384 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.951+0000 7fd7b5e59740 1 Processor -- start 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.951+0000 7fd7b5e59740 1 -- start start 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x5565329e4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x55653297cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556532609260 con 0x5565329632f0 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565329666f0 con 0x55653293f160 2024-09-13T17:14:06.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556532938900 con 0x5565329408f0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57428/0 (socket says 172.21.15.28:57428) 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7b1000640 1 -- 172.21.15.28:0/3477247934 learned_addr learned my addr 172.21.15.28:0/3477247934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.952+0000 7fd7a3400640 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x55653297cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7abe00640 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x5565329e4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 -- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 msgr2=0x55653297cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x55653297cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 -- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 msgr2=0x5565329e4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x5565329e4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 -- 172.21.15.28:0/3477247934 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556532842240 con 0x5565329632f0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7b1000640 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000f80 tx=0x7fd7a401a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f8d726f2089264 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7a1600640 1 -- 172.21.15.28:0/3477247934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a400e050 con 0x5565329632f0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7a1600640 1 -- 172.21.15.28:0/3477247934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a400f040 con 0x5565329632f0 2024-09-13T17:14:06.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.953+0000 7fd7a1600640 1 -- 172.21.15.28:0/3477247934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a4009410 con 0x5565329632f0 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 -- 172.21.15.28:0/3477247934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 msgr2=0x556532995ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4000f80 tx=0x7fd7a401a000 comp rx=0 tx=0).stop 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 -- 172.21.15.28:0/3477247934 shutdown_connections 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x556532995ea0 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5565329408f0 0x55653297cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/3477247934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55653293f160 0x5565329e4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 -- 172.21.15.28:0/3477247934 >> 172.21.15.28:0/3477247934 conn(0x55653293c920 msgr2=0x556532934c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 -- 172.21.15.28:0/3477247934 shutdown_connections 2024-09-13T17:14:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.954+0000 7fd7b5e59740 1 -- 172.21.15.28:0/3477247934 wait complete. 2024-09-13T17:14:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.955+0000 7fd7b5e59740 1 -- start start 2024-09-13T17:14:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.955+0000 7fd7b5e59740 1 Processor -- start 2024-09-13T17:14:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55653293f160 0x5565329e2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5565329408f0 0x556532982cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55653299d880 con 0x5565329632f0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556532995190 con 0x5565329408f0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565329398c0 con 0x55653293f160 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b5e59740 10 client.0 _ll_register_callbacks cb 0x7fd79c001560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57438/0 (socket says 172.21.15.28:57438) 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 -- 172.21.15.28:0/2026385177 learned_addr learned my addr 172.21.15.28:0/2026385177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55653293f160 0x5565329e2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7a3400640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5565329408f0 0x556532982cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 -- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55653293f160 msgr2=0x5565329e2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55653293f160 0x5565329e2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 -- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5565329408f0 msgr2=0x556532982cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.956+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5565329408f0 0x556532982cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd7b1000640 1 -- 172.21.15.28:0/2026385177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565328a04f0 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd7b1000640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4004340 tx=0x7fd7a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d642bde7167b860a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd799600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a400e040 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fd7a4015e80 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd79c0158f0 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7a4009410 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd7b5e59740 10 client.7827 fetch_fsmap learned FSMap version 86 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.957+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=2+}) v3 -- 0x556532867940 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79c0158f0 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 86) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd7a4027070 con 0x5565329632f0 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 10 client.7827 fetch_fsmap finished waiting for FSMap version 86 2024-09-13T17:14:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 10 client.7827 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.a 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 20 client.7827 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532994c70 0x5565329ded30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 4 client.7827 mds_command: new command op to 7725 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/testdir", "scrubops": ["recursive"]}] 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 --> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] -- command(tid 0: {"prefix": "scrub start", "path": "/testdir", "scrubops": ["recursive"]}) v1 -- 0x5565325a6a00 con 0x556532994c70 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.958+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532994c70 0x5565329ded30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7abe00640 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532994c70 0x5565329ded30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8300b65d3b889fcd server_cookie=273ad206707ab4eb in_seq=0 out_seq=0 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd799600640 10 client.7827 ms_handle_connect on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd799600640 1 -- 172.21.15.28:0/2026385177 <== mds.0 v2:172.21.15.28:6833/2200414779 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5565325a6a00 con 0x556532994c70 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd799600640 10 client.7827 handle_command_reply: tid=0 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 client.7827 shutdown 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 msgr2=0x55653299cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fd7a4004340 tx=0x7fd7a4004ab0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 shutdown_connections 2024-09-13T17:14:06.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565329632f0 0x55653299cb60 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5565329408f0 0x556532982cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.28:6833/2200414779,v1:172.21.15.28:6835/2200414779] conn(0x556532994c70 0x5565329ded30 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 --2- 172.21.15.28:0/2026385177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55653293f160 0x5565329e2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 >> 172.21.15.28:0/2026385177 conn(0x55653293c920 msgr2=0x556532960dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.959+0000 7fd799600640 0 client.7827 ms_handle_reset on v2:172.21.15.28:6833/2200414779 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.960+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 shutdown_connections 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.960+0000 7fd7b5e59740 1 -- 172.21.15.28:0/2026385177 wait complete. 2024-09-13T17:14:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.960+0000 7fd7b5e59740 20 client.7827 trim_cache size 0 max 16384 2024-09-13T17:14:06.966 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:14:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:14:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "33cd7e15-f036-4a3c-9d28-147a0c1caf35", 2024-09-13T17:14:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:14:06.966 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 msgr2=0x7fd78806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 0x7fd78806d0c0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac068da0 tx=0x7fd79c009040 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 msgr2=0x7fd7ac1525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fd79402ad00 tx=0x7fd794001a00 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 shutdown_connections 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78806ac00 0x7fd78806d0c0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7ac152b00 0x7fd7ac150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7ac158770 0x7fd7ac1525c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 --2- 172.21.15.28:0/4253438656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7900069d0 0x7fd7ac14ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 >> 172.21.15.28:0/4253438656 conn(0x7fd7ac069e80 msgr2=0x7fd7ac06c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 shutdown_connections 2024-09-13T17:14:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:06.961+0000 7fd7b3000640 1 -- 172.21.15.28:0/4253438656 wait complete. 2024-09-13T17:14:06.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:07.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.133+0000 7f09c9c00640 1 Processor -- start 2024-09-13T17:14:07.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.133+0000 7f09c9c00640 1 -- start start 2024-09-13T17:14:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c9c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09c40cabc0 0x7f09c406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c9c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c9c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09c415e100 con 0x7f09c415a540 2024-09-13T17:14:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c9c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c40adb00 con 0x7f09c40cabc0 2024-09-13T17:14:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c9c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4071670 con 0x7f09c4070cc0 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.134+0000 7f09c3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09ba000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09c40cabc0 0x7f09c406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:40202/0 (socket says 172.21.15.28:40202) 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 -- 172.21.15.28:0/1141784267 learned_addr learned my addr 172.21.15.28:0/1141784267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 -- 172.21.15.28:0/1141784267 >> v2:172.21.15.189:3300/0 conn(0x7f09c40cabc0 msgr2=0x7f09c406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 --2- 172.21.15.28:0/1141784267 >> v2:172.21.15.189:3300/0 conn(0x7f09c40cabc0 0x7f09c406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 -- 172.21.15.28:0/1141784267 >> v1:172.21.15.189:6789/0 conn(0x7f09c415a540 legacy=0x7f09c415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.135+0000 7f09c3400640 1 -- 172.21.15.28:0/1141784267 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c415e3e0 con 0x7f09c4070cc0 2024-09-13T17:14:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.136+0000 7f09c3400640 1 --2- 172.21.15.28:0/1141784267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009e50 tx=0x7f09bc02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfdd2f05a5027bbb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:07.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.137+0000 7f09c2000640 1 -- 172.21.15.28:0/1141784267 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc02aea0 con 0x7f09c4070cc0 2024-09-13T17:14:07.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.137+0000 7f09c2000640 1 -- 172.21.15.28:0/1141784267 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09bc02d050 con 0x7f09c4070cc0 2024-09-13T17:14:07.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.137+0000 7f09c2000640 1 -- 172.21.15.28:0/1141784267 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc031820 con 0x7f09c4070cc0 2024-09-13T17:14:07.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 -- 172.21.15.28:0/1141784267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 msgr2=0x7f09c4071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 --2- 172.21.15.28:0/1141784267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f09bc009e50 tx=0x7f09bc02a680 comp rx=0 tx=0).stop 2024-09-13T17:14:07.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 -- 172.21.15.28:0/1141784267 shutdown_connections 2024-09-13T17:14:07.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 --2- 172.21.15.28:0/1141784267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c4070cc0 0x7f09c4071130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 --2- 172.21.15.28:0/1141784267 >> v2:172.21.15.189:3300/0 conn(0x7f09c40cabc0 0x7f09c406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 -- 172.21.15.28:0/1141784267 >> 172.21.15.28:0/1141784267 conn(0x7f09c4069e80 msgr2=0x7f09c406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:07.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 -- 172.21.15.28:0/1141784267 shutdown_connections 2024-09-13T17:14:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.138+0000 7f09c9c00640 1 -- 172.21.15.28:0/1141784267 wait complete. 2024-09-13T17:14:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.139+0000 7f09c9c00640 1 Processor -- start 2024-09-13T17:14:07.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 -- start start 2024-09-13T17:14:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c40cabc0 0x7f09c41508e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c415a540 0x7f09c4150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c414e370 con 0x7f09c4070cc0 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c4151720 con 0x7f09c415a540 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.140+0000 7f09c9c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c40afc80 con 0x7f09c40cabc0 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57444/0 (socket says 172.21.15.28:57444) 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 -- 172.21.15.28:0/784433552 learned_addr learned my addr 172.21.15.28:0/784433552 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09ba000640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c40cabc0 0x7f09c41508e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c3400640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c415a540 0x7f09c4150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 -- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c40cabc0 msgr2=0x7f09c41508e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c40cabc0 0x7f09c41508e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 -- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c415a540 msgr2=0x7f09c4150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c415a540 0x7f09c4150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 -- 172.21.15.28:0/784433552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09bc009b30 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.141+0000 7f09c2a00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c400 tx=0x7f09b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.142+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b40052d0 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.142+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b4005430 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.142+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b4003830 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.142+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c40aff60 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.142+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c40b0520 con 0x7f09c4070cc0 2024-09-13T17:14:07.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.143+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c4071130 con 0x7f09c4070cc0 2024-09-13T17:14:07.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.144+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09b4003990 con 0x7f09c4070cc0 2024-09-13T17:14:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.144+0000 7f09bb400640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.145+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f09b40889d0 con 0x7f09c4070cc0 2024-09-13T17:14:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.145+0000 7f09ba000640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:07.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.146+0000 7f09ba000640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 0x7f09a006d0c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f09c40cd250 tx=0x7f09ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.149+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f09b4053ab0 con 0x7f09c4070cc0 2024-09-13T17:14:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.366+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09c40b11c0 con 0x7f09c4070cc0 2024-09-13T17:14:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.366+0000 7f09bb400640 1 -- 172.21.15.28:0/784433552 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f09b4017090 con 0x7f09c4070cc0 2024-09-13T17:14:07.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:07.369 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:07.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 msgr2=0x7f09a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 0x7f09a006d0c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f09c40cd250 tx=0x7f09ac008040 comp rx=0 tx=0).stop 2024-09-13T17:14:07.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 msgr2=0x7f09c414de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:07.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f09b400c400 tx=0x7f09b400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 shutdown_connections 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09c415a540 0x7f09c4150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09c40cabc0 0x7f09c41508e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 --2- 172.21.15.28:0/784433552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09c4070cc0 0x7f09c414de30 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 >> 172.21.15.28:0/784433552 conn(0x7f09c4069e80 msgr2=0x7f09c406ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 shutdown_connections 2024-09-13T17:14:07.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:07.369+0000 7f09c9c00640 1 -- 172.21.15.28:0/784433552 wait complete. 2024-09-13T17:14:07.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 1099511627776 2024-09-13T17:14:07.560 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:13:51.205107+0000","btime":"2024-09-13T17:13:50.061139+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:13:51.205107+0000","atime":"2024-09-13T17:13:50.061139+0000","time_warp_seq":0,"change_attr":21,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:13:51.205107+0000","num_files":21,"num_subdirs":0,"change_attr":21},"rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"accounted_rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"version":128,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"dirfrag":1,"authpin":0,"scrubqueue":0},"nref":1,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:14:07.560 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:13:50.061139+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:13:50.061139+0000", "change_attr": 21, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:13:51.205107+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 21, "mtime": "2024-09-13T17:13:51.205107+0000", "num_files": 21, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:13:51.205107+0000", "nestlock": {}, "nlink": 1, "nref": 1, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "authpin": 0, "dirfrag": 1, "scrubqueue": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 128, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:14:07.561 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30 retry_count=0)... 2024-09-13T17:14:09.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:14:09.931 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:14:12.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.727+0000 7f1a7ba00640 1 Processor -- start 2024-09-13T17:14:12.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.727+0000 7f1a7ba00640 1 -- start start 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a7ba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1a740cdfa0 0x7f1a7415d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a7ba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a7406fb50 0x7f1a7406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a7ba00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a740b3970 con 0x7f1a740b3160 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a7ba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a74159230 con 0x7f1a7406fb50 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a7ba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a74159430 con 0x7f1a740cdfa0 2024-09-13T17:14:12.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a79a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a7406fb50 0x7f1a7406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a79a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1a7406fb50 0x7f1a7406b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57448/0 (socket says 172.21.15.28:57448) 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a79a00640 1 -- 172.21.15.28:0/3853882865 learned_addr learned my addr 172.21.15.28:0/3853882865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.728+0000 7f1a79000640 1 --2- 172.21.15.28:0/3853882865 >> v2:172.21.15.189:3300/0 conn(0x7f1a740cdfa0 0x7f1a7415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.729+0000 7f1a79a00640 1 -- 172.21.15.28:0/3853882865 >> v2:172.21.15.189:3300/0 conn(0x7f1a740cdfa0 msgr2=0x7f1a7415d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.729+0000 7f1a79a00640 1 --2- 172.21.15.28:0/3853882865 >> v2:172.21.15.189:3300/0 conn(0x7f1a740cdfa0 0x7f1a7415d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.729+0000 7f1a79a00640 1 -- 172.21.15.28:0/3853882865 >> v1:172.21.15.189:6789/0 conn(0x7f1a740b3160 legacy=0x7f1a7406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.729+0000 7f1a79a00640 1 -- 172.21.15.28:0/3853882865 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74159710 con 0x7f1a7406fb50 2024-09-13T17:14:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.730+0000 7f1a79a00640 1 --2- 172.21.15.28:0/3853882865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7406fb50 0x7f1a7406b1c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1a7406ae20 tx=0x7f1a6c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567820b3f776ef53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.730+0000 7f1a73e00640 1 -- 172.21.15.28:0/3853882865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031600 con 0x7f1a7406fb50 2024-09-13T17:14:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.730+0000 7f1a73e00640 1 -- 172.21.15.28:0/3853882865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c02d050 con 0x7f1a7406fb50 2024-09-13T17:14:12.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.730+0000 7f1a73e00640 1 -- 172.21.15.28:0/3853882865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c031980 con 0x7f1a7406fb50 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 -- 172.21.15.28:0/3853882865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7406fb50 msgr2=0x7f1a7406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/3853882865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7406fb50 0x7f1a7406b1c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1a7406ae20 tx=0x7f1a6c02a730 comp rx=0 tx=0).stop 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 -- 172.21.15.28:0/3853882865 shutdown_connections 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/3853882865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a7406fb50 0x7f1a7406b1c0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/3853882865 >> v2:172.21.15.189:3300/0 conn(0x7f1a740cdfa0 0x7f1a7415d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 -- 172.21.15.28:0/3853882865 >> 172.21.15.28:0/3853882865 conn(0x7f1a74069e80 msgr2=0x7f1a7406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 -- 172.21.15.28:0/3853882865 shutdown_connections 2024-09-13T17:14:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.731+0000 7f1a7ba00640 1 -- 172.21.15.28:0/3853882865 wait complete. 2024-09-13T17:14:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.733+0000 7f1a7ba00640 1 Processor -- start 2024-09-13T17:14:12.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.733+0000 7f1a7ba00640 1 -- start start 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7406fb50 0x7f1a7414b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a740b3160 0x7f1a7414b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414cd00 con 0x7f1a740cdfa0 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414cfe0 con 0x7f1a7406fb50 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a7ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7414a880 con 0x7f1a740b3160 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57454/0 (socket says 172.21.15.28:57454) 2024-09-13T17:14:12.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 -- 172.21.15.28:0/1906582055 learned_addr learned my addr 172.21.15.28:0/1906582055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7406fb50 0x7f1a7414b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a6be00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a740b3160 0x7f1a7414b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 -- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a740b3160 msgr2=0x7f1a7414b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a740b3160 0x7f1a7414b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.734+0000 7f1a79a00640 1 -- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7406fb50 msgr2=0x7f1a7414b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a79a00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7406fb50 0x7f1a7414b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a79a00640 1 -- 172.21.15.28:0/1906582055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c009b30 con 0x7f1a740cdfa0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a79a00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c034040 tx=0x7f1a6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c03d860 con 0x7f1a740cdfa0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6c03d9c0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7414ab60 con 0x7f1a740cdfa0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c03dd70 con 0x7f1a740cdfa0 2024-09-13T17:14:12.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.735+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7414c940 con 0x7f1a740cdfa0 2024-09-13T17:14:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.736+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a7414cad0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.737+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a6c03b400 con 0x7f1a740cdfa0 2024-09-13T17:14:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.738+0000 7f1a71600640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 0x7f1a5006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.738+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f1a6c0aa1b0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.739+0000 7f1a79000640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 0x7f1a5006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:12.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.739+0000 7f1a79000640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 0x7f1a5006d0c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1a7415bce0 tx=0x7f1a64009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:12.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.742+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1a6c0752c0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.957+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a740715e0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.958+0000 7f1a71600640 1 -- 172.21.15.28:0/1906582055 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f1a740715e0 con 0x7f1a740cdfa0 2024-09-13T17:14:12.960 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:12.960 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:12.960 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 msgr2=0x7f1a5006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 0x7f1a5006d0c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1a7415bce0 tx=0x7f1a64009190 comp rx=0 tx=0).stop 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 msgr2=0x7f1a7414bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c034040 tx=0x7f1a6c004a40 comp rx=0 tx=0).stop 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 shutdown_connections 2024-09-13T17:14:12.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a5006ac00 0x7f1a5006d0c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a740cdfa0 0x7f1a7414bde0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1a740b3160 0x7f1a7414b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 --2- 172.21.15.28:0/1906582055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a7406fb50 0x7f1a7414b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 >> 172.21.15.28:0/1906582055 conn(0x7f1a74069e80 msgr2=0x7f1a740b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 shutdown_connections 2024-09-13T17:14:12.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:12.960+0000 7f1a7ba00640 1 -- 172.21.15.28:0/1906582055 wait complete. 2024-09-13T17:14:12.979 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 1099511627776 2024-09-13T17:14:13.150 INFO:teuthology.orchestra.run.smithi028.stdout:{"path":"/testdir","ino":1099511627776,"rdev":0,"ctime":"2024-09-13T17:13:51.205107+0000","btime":"2024-09-13T17:13:50.061139+0000","mode":16877,"uid":1000,"gid":1333,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-09-13T17:13:51.205107+0000","atime":"2024-09-13T17:13:50.061139+0000","time_warp_seq":0,"change_attr":21,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-09-13T17:13:51.205107+0000","num_files":21,"num_subdirs":0,"change_attr":21},"rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"accounted_rstat":{"version":0,"rbytes":32,"rfiles":21,"rsubdirs":1,"rsnaps":0,"rctime":"2024-09-13T17:13:51.205107+0000"},"version":131,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"2024-09-13T17:14:06.961969+0000","last_scrub_version":128,"symlink":"","xattrs":[{"key":"security.selinux","val":"unconfined_u:object_r:unlabeled_t:s0\u0000"}],"dirfragtree":{"splits":[{"frag":"*","children":1}]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"lock":0,"dirfrag":2,"dirtyrstat":1,"dirty":1,"authpin":0,"scrubqueue":0},"nref":4,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyrstat"],"client_caps":[],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"is_subvolume":false} 2024-09-13T17:14:13.150 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-09-13T17:13:50.061139+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-09-13T17:13:50.061139+0000", "change_attr": 21, "client_caps": [], "client_ranges": [], "ctime": "2024-09-13T17:13:51.205107+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [ { "children": 1, "frag": "*" } ] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 21, "mtime": "2024-09-13T17:13:51.205107+0000", "num_files": 21, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 1333, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "2024-09-13T17:14:06.961969+0000", "last_scrub_version": 128, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2024-09-13T17:13:51.205107+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/testdir", "pins": { "authpin": 0, "dirfrag": 2, "dirty": 1, "dirtyrstat": 1, "lock": 0, "scrubqueue": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 32, "rctime": "2024-09-13T17:13:51.205107+0000", "rfiles": 21, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyrstat" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 1000, "version": 131, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [ { "key": "security.selinux", "val": "unconfined_u:object_r:unlabeled_t:s0\u0000" } ] } 2024-09-13T17:14:13.150 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-13T17:14:13.151 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:14:13.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:14:13.244 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:14:13.335 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:14:13.335 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:13.335 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:13.371 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:14:13.371 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:14:13.371 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:13.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:14:13.408 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:13.408 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:13.447 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:14:13.447 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:14:13.447 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:13.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:14:13.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:13.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.640+0000 7f80c4a00640 1 Processor -- start 2024-09-13T17:14:13.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.641+0000 7f80c4a00640 1 -- start start 2024-09-13T17:14:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80c4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c0154a90 0x7f80c015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80c4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80c006b360 0x7f80c00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80c4a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80c015bb10 con 0x7f80c00b0980 2024-09-13T17:14:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80c4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80c00b0df0 con 0x7f80c006b360 2024-09-13T17:14:13.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80c4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c006b900 con 0x7f80c0154a90 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80bfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80c00b0980 0x7f80c00af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48622/0 (socket says 172.21.15.28:48622) 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80bfe00640 1 -- 172.21.15.28:0/3522933289 learned_addr learned my addr 172.21.15.28:0/3522933289 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80b6a00640 1 --2- 172.21.15.28:0/3522933289 >> v2:172.21.15.28:3300/0 conn(0x7f80c006b360 0x7f80c00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.642+0000 7f80bf400640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c0154a90 0x7f80c015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80bea00640 1 -- 172.21.15.28:0/3522933289 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187245561 0 0) 0x7f80c015bb10 con 0x7f80c00b0980 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80bea00640 1 -- 172.21.15.28:0/3522933289 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80a4003520 con 0x7f80c00b0980 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80b6a00640 1 -- 172.21.15.28:0/3522933289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c0154a90 msgr2=0x7f80c015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:13.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80b6a00640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c0154a90 0x7f80c015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80b6a00640 1 -- 172.21.15.28:0/3522933289 >> v1:172.21.15.28:6789/0 conn(0x7f80c00b0980 legacy=0x7f80c00af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.643+0000 7f80b6a00640 1 -- 172.21.15.28:0/3522933289 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c006bbe0 con 0x7f80c006b360 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.644+0000 7f80b6a00640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c006b360 0x7f80c00b0440 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f80a80097a0 tx=0x7f80a802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2992b9f499928c1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.644+0000 7f80bea00640 1 -- 172.21.15.28:0/3522933289 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8032440 con 0x7f80c006b360 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.644+0000 7f80bea00640 1 -- 172.21.15.28:0/3522933289 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a80325a0 con 0x7f80c006b360 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.644+0000 7f80bea00640 1 -- 172.21.15.28:0/3522933289 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8032930 con 0x7f80c006b360 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 -- 172.21.15.28:0/3522933289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c006b360 msgr2=0x7f80c00b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c006b360 0x7f80c00b0440 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f80a80097a0 tx=0x7f80a802a180 comp rx=0 tx=0).stop 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 -- 172.21.15.28:0/3522933289 shutdown_connections 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c006b360 0x7f80c00b0440 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3522933289 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c0154a90 0x7f80c015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 -- 172.21.15.28:0/3522933289 >> 172.21.15.28:0/3522933289 conn(0x7f80c0072e40 msgr2=0x7f80c0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:13.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 -- 172.21.15.28:0/3522933289 shutdown_connections 2024-09-13T17:14:13.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.645+0000 7f80c4a00640 1 -- 172.21.15.28:0/3522933289 wait complete. 2024-09-13T17:14:13.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.646+0000 7f80c4a00640 1 Processor -- start 2024-09-13T17:14:13.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.647+0000 7f80c4a00640 1 -- start start 2024-09-13T17:14:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.647+0000 7f80c4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c006b360 0x7f80c0153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.647+0000 7f80c4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80c00b0980 0x7f80c00ad040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80c4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80c4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00b0130 con 0x7f80c0154a90 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80c4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80c013af70 con 0x7f80c00b0980 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80c4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80c00afb70 con 0x7f80c006b360 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57476/0 (socket says 172.21.15.28:57476) 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 -- 172.21.15.28:0/3997778179 learned_addr learned my addr 172.21.15.28:0/3997778179 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bf400640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c006b360 0x7f80c0153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80b6a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80c00b0980 0x7f80c00ad040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 -- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c006b360 msgr2=0x7f80c0153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c006b360 0x7f80c0153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 -- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80c00b0980 msgr2=0x7f80c00ad040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80c00b0980 0x7f80c00ad040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.648+0000 7f80bfe00640 1 -- 172.21.15.28:0/3997778179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8009400 con 0x7f80c0154a90 2024-09-13T17:14:13.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80bfe00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f80b800caa0 tx=0x7f80b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b800ac00 con 0x7f80c0154a90 2024-09-13T17:14:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80b800ad60 con 0x7f80c0154a90 2024-09-13T17:14:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80b8012620 con 0x7f80c0154a90 2024-09-13T17:14:13.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c00afe50 con 0x7f80c0154a90 2024-09-13T17:14:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.649+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c013b7d0 con 0x7f80c0154a90 2024-09-13T17:14:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.650+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80c013b350 con 0x7f80c0154a90 2024-09-13T17:14:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.651+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f80b8007070 con 0x7f80c0154a90 2024-09-13T17:14:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.652+0000 7f80b7e00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 0x7f809c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:13.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.652+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f80b8089400 con 0x7f80c0154a90 2024-09-13T17:14:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.652+0000 7f80bf400640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 0x7f809c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:13.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.653+0000 7f80bf400640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 0x7f809c06d0c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009b40 tx=0x7f80b0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.656+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f80c013b350 con 0x7f80c0154a90 2024-09-13T17:14:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.871+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80c013b350 con 0x7f80c0154a90 2024-09-13T17:14:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.872+0000 7f80b7e00640 1 -- 172.21.15.28:0/3997778179 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f80c013b350 con 0x7f80c0154a90 2024-09-13T17:14:13.874 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:13.874 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 msgr2=0x7f809c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 0x7f809c06d0c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f80b0009b40 tx=0x7f80b0009190 comp rx=0 tx=0).stop 2024-09-13T17:14:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 msgr2=0x7f80c00ad580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f80b800caa0 tx=0x7f80b800cf60 comp rx=0 tx=0).stop 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 shutdown_connections 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809c06ac00 0x7f809c06d0c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80c0154a90 0x7f80c00ad580 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80c00b0980 0x7f80c00ad040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 --2- 172.21.15.28:0/3997778179 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80c006b360 0x7f80c0153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 >> 172.21.15.28:0/3997778179 conn(0x7f80c0072e40 msgr2=0x7f80c00b1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 shutdown_connections 2024-09-13T17:14:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:13.874+0000 7f80c4a00640 1 -- 172.21.15.28:0/3997778179 wait complete. 2024-09-13T17:14:13.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 Processor -- start 2024-09-13T17:14:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 -- start start 2024-09-13T17:14:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 0x7fc3f406c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f4159d90 con 0x7fc3f406f970 2024-09-13T17:14:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f415b0b0 con 0x7fc3f406c2d0 2024-09-13T17:14:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.047+0000 7fc3fb600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f406ccc0 con 0x7fc3f41592a0 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 0x7fc3f406c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:42068/0 (socket says 172.21.15.28:42068) 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 0x7fc3f406c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46250/0 (socket says 172.21.15.28:46250) 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 -- 172.21.15.28:0/1188801422 learned_addr learned my addr 172.21.15.28:0/1188801422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 -- 172.21.15.28:0/1188801422 >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 msgr2=0x7fc3f406c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 --2- 172.21.15.28:0/1188801422 >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 0x7fc3f406c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f3400640 1 -- 172.21.15.28:0/1188801422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689098653 0 0) 0x7fc3f4159d90 con 0x7fc3f406f970 2024-09-13T17:14:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 -- 172.21.15.28:0/1188801422 >> v1:172.21.15.189:6789/0 conn(0x7fc3f406f970 legacy=0x7fc3f4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.048+0000 7fc3f8c00640 1 -- 172.21.15.28:0/1188801422 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40b2040 con 0x7fc3f41592a0 2024-09-13T17:14:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.049+0000 7fc3f8c00640 1 --2- 172.21.15.28:0/1188801422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009870 tx=0x7fc3e802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27d43a4e1d2f70b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.049+0000 7fc3f3400640 1 -- 172.21.15.28:0/1188801422 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8032440 con 0x7fc3f41592a0 2024-09-13T17:14:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3f3400640 1 -- 172.21.15.28:0/1188801422 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e80325a0 con 0x7fc3f41592a0 2024-09-13T17:14:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3f3400640 1 -- 172.21.15.28:0/1188801422 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8032930 con 0x7fc3f41592a0 2024-09-13T17:14:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 -- 172.21.15.28:0/1188801422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 msgr2=0x7fc3f415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 --2- 172.21.15.28:0/1188801422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009870 tx=0x7fc3e802a270 comp rx=0 tx=0).stop 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 -- 172.21.15.28:0/1188801422 shutdown_connections 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 --2- 172.21.15.28:0/1188801422 >> v2:172.21.15.189:3300/0 conn(0x7fc3f406c2d0 0x7fc3f406c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 --2- 172.21.15.28:0/1188801422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f41592a0 0x7fc3f415ab70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 -- 172.21.15.28:0/1188801422 >> 172.21.15.28:0/1188801422 conn(0x7fc3f4069e80 msgr2=0x7fc3f406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 -- 172.21.15.28:0/1188801422 shutdown_connections 2024-09-13T17:14:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.050+0000 7fc3fb600640 1 -- 172.21.15.28:0/1188801422 wait complete. 2024-09-13T17:14:14.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.052+0000 7fc3fb600640 1 Processor -- start 2024-09-13T17:14:14.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.052+0000 7fc3fb600640 1 -- start start 2024-09-13T17:14:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f406c2d0 0x7fc3f4141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 0x7fc3f41f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4156970 con 0x7fc3f406f970 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4156c50 con 0x7fc3f41592a0 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3fb600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4156f30 con 0x7fc3f406c2d0 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36592/0 (socket says 172.21.15.28:36592) 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.053+0000 7fc3f3e00640 1 -- 172.21.15.28:0/2115299357 learned_addr learned my addr 172.21.15.28:0/2115299357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f8c00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f406c2d0 0x7fc3f4141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f9600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 0x7fc3f41f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 -- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f406c2d0 msgr2=0x7fc3f4141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f406c2d0 0x7fc3f4141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 -- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 msgr2=0x7fc3f41f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 0x7fc3f41f5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 -- 172.21.15.28:0/2115299357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80094d0 con 0x7fc3f406f970 2024-09-13T17:14:14.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f9600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 0x7fc3f41f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:14.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f3e00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c450 tx=0x7fc3e000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.054+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e00052d0 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.055+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e0005430 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.055+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0012870 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.055+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f4157210 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.055+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f41f6af0 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.056+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f40b3740 con 0x7fc3f406f970 2024-09-13T17:14:14.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.057+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc3e00129d0 con 0x7fc3f406f970 2024-09-13T17:14:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.058+0000 7fc3f0c00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.058+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7fc3e0088ff0 con 0x7fc3f406f970 2024-09-13T17:14:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.059+0000 7fc3f8c00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.059+0000 7fc3f8c00640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8034000 tx=0x7fc3e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.062+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc3e00bc9c0 con 0x7fc3f406f970 2024-09-13T17:14:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.245+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3f40ad4f0 con 0x7fc3f406f970 2024-09-13T17:14:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.246+0000 7fc3f0c00640 1 -- 172.21.15.28:0/2115299357 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16967 (secure 0 0 0) 0x7fc3e00540b0 con 0x7fc3f406f970 2024-09-13T17:14:14.248 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:14.248 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:00.308433+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.16","mappings":[{"from":1,"to":5}]},{"pgid":"13.2a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.189:6835/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.28:6834/2861751170":"2024-09-14T17:13:54.409423+0000","172.21.15.28:6832/2861751170":"2024-09-14T17:13:54.409423+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 msgr2=0x7fc3d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8034000 tx=0x7fc3e802e000 comp rx=0 tx=0).stop 2024-09-13T17:14:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 msgr2=0x7fc3f4155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7fc3e000c450 tx=0x7fc3e000c910 comp rx=0 tx=0).stop 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 shutdown_connections 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f41592a0 0x7fc3f41f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f406f970 0x7fc3f4155ed0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 --2- 172.21.15.28:0/2115299357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f406c2d0 0x7fc3f4141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 >> 172.21.15.28:0/2115299357 conn(0x7fc3f4069e80 msgr2=0x7fc3f40b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 shutdown_connections 2024-09-13T17:14:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.249+0000 7fc3fb600640 1 -- 172.21.15.28:0/2115299357 wait complete. 2024-09-13T17:14:14.266 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:13:40.462897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:13:41.537996+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:14:14.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:14:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.425+0000 7f1b02a00640 1 Processor -- start 2024-09-13T17:14:14.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.425+0000 7f1b02a00640 1 -- start start 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc15a780 0x7f1afc070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1afc15ab60 0x7f1afc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1afc06bb40 0x7f1afc06bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc06c8b0 con 0x7f1afc15ab60 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc06cb90 con 0x7f1afc06bb40 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b02a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0cc980 con 0x7f1afc15a780 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1afc15ab60 0x7f1afc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1afc15ab60 0x7f1afc0712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36594/0 (socket says 172.21.15.28:36594) 2024-09-13T17:14:14.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 -- 172.21.15.28:0/3111596041 learned_addr learned my addr 172.21.15.28:0/3111596041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:14.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1b00a00640 1 --2- 172.21.15.28:0/3111596041 >> v2:172.21.15.189:3300/0 conn(0x7f1afc06bb40 0x7f1afc06bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1afbe00640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc15a780 0x7f1afc070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 -- 172.21.15.28:0/3111596041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc15a780 msgr2=0x7f1afc070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc15a780 0x7f1afc070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 -- 172.21.15.28:0/3111596041 >> v2:172.21.15.189:3300/0 conn(0x7f1afc06bb40 msgr2=0x7f1afc06bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 --2- 172.21.15.28:0/3111596041 >> v2:172.21.15.189:3300/0 conn(0x7f1afc06bb40 0x7f1afc06bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.427+0000 7f1af3400640 1 -- 172.21.15.28:0/3111596041 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0ccc60 con 0x7f1afc15ab60 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.428+0000 7f1af3400640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15ab60 0x7f1afc0712c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0b1600 tx=0x7f1ae402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524a25843aa300cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1afb400640 1 -- 172.21.15.28:0/3111596041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4032440 con 0x7f1afc15ab60 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1afb400640 1 -- 172.21.15.28:0/3111596041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae40325a0 con 0x7f1afc15ab60 2024-09-13T17:14:14.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1afb400640 1 -- 172.21.15.28:0/3111596041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4032930 con 0x7f1afc15ab60 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 -- 172.21.15.28:0/3111596041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15ab60 msgr2=0x7f1afc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15ab60 0x7f1afc0712c0 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1afc0b1600 tx=0x7f1ae402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 -- 172.21.15.28:0/3111596041 shutdown_connections 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15ab60 0x7f1afc0712c0 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3111596041 >> v2:172.21.15.189:3300/0 conn(0x7f1afc06bb40 0x7f1afc06bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3111596041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc15a780 0x7f1afc070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 -- 172.21.15.28:0/3111596041 >> 172.21.15.28:0/3111596041 conn(0x7f1afc0b0270 msgr2=0x7f1afc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 -- 172.21.15.28:0/3111596041 shutdown_connections 2024-09-13T17:14:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.429+0000 7f1b02a00640 1 -- 172.21.15.28:0/3111596041 wait complete. 2024-09-13T17:14:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.431+0000 7f1b02a00640 1 Processor -- start 2024-09-13T17:14:14.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.431+0000 7f1b02a00640 1 -- start start 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc06bb40 0x7f1afc14bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1afc15ab60 0x7f1afc148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc14c170 con 0x7f1afc15a780 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0c7150 con 0x7f1afc15ab60 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b02a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0ad1d0 con 0x7f1afc06bb40 2024-09-13T17:14:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1af3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1af3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36608/0 (socket says 172.21.15.28:36608) 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1af3400640 1 -- 172.21.15.28:0/3841711400 learned_addr learned my addr 172.21.15.28:0/3841711400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1afbe00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc06bb40 0x7f1afc14bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.432+0000 7f1b00a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1afc15ab60 0x7f1afc148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 -- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc06bb40 msgr2=0x7f1afc14bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc06bb40 0x7f1afc14bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 -- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1afc15ab60 msgr2=0x7f1afc148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1afc15ab60 0x7f1afc148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 -- 172.21.15.28:0/3841711400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae40094d0 con 0x7f1afc15a780 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af3400640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1ae402a220 tx=0x7f1ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4032440 con 0x7f1afc15a780 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae4030b00 con 0x7f1afc15a780 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae403d440 con 0x7f1afc15a780 2024-09-13T17:14:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc148e50 con 0x7f1afc15a780 2024-09-13T17:14:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.433+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc0aab00 con 0x7f1afc15a780 2024-09-13T17:14:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.434+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc06bf90 con 0x7f1afc15a780 2024-09-13T17:14:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.435+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ae4039040 con 0x7f1afc15a780 2024-09-13T17:14:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.436+0000 7f1af8c00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.436+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f1ae40a9f00 con 0x7f1afc15a780 2024-09-13T17:14:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.437+0000 7f1afbe00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.437+0000 7f1afbe00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.440+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1ae4074f60 con 0x7f1afc15a780 2024-09-13T17:14:14.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.620+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1afc148e50 con 0x7f1afc15a780 2024-09-13T17:14:14.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.620+0000 7f1af8c00640 1 -- 172.21.15.28:0/3841711400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1afc148e50 con 0x7f1afc15a780 2024-09-13T17:14:14.622 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 msgr2=0x7f1ad806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1aec006fd0 tx=0x7f1aec008040 comp rx=0 tx=0).stop 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 msgr2=0x7f1afc0acc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f1ae402a220 tx=0x7f1ae40046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 shutdown_connections 2024-09-13T17:14:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ad806ac00 0x7f1ad806d0c0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1afc15ab60 0x7f1afc148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1afc15a780 0x7f1afc0acc90 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 --2- 172.21.15.28:0/3841711400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1afc06bb40 0x7f1afc14bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 >> 172.21.15.28:0/3841711400 conn(0x7f1afc0b0270 msgr2=0x7f1afc157810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 shutdown_connections 2024-09-13T17:14:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.623+0000 7f1b02a00640 1 -- 172.21.15.28:0/3841711400 wait complete. 2024-09-13T17:14:14.641 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:14:14.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.797+0000 7f7acb000640 1 Processor -- start 2024-09-13T17:14:14.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.798+0000 7f7acb000640 1 -- start start 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7acb000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ac4070e30 0x7f7ac4071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7acb000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac4159e10 con 0x7f7ac40cdf80 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7acb000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ac415a0f0 con 0x7f7ac4072140 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7acb000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ac415a3d0 con 0x7f7ac4070e30 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7aca600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ac40cdf80 0x7f7ac415aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60194/0 (socket says 172.21.15.28:60194) 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7ac9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ac4070e30 0x7f7ac4071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7aca600640 1 -- 172.21.15.28:0/774343372 learned_addr learned my addr 172.21.15.28:0/774343372 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:14.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.799+0000 7f7ac9c00640 1 -- 172.21.15.28:0/774343372 >> v1:172.21.15.189:6789/0 conn(0x7f7ac4072140 legacy=0x7f7ac40cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:14.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.800+0000 7f7ac9c00640 1 -- 172.21.15.28:0/774343372 >> v1:172.21.15.28:6789/0 conn(0x7f7ac40cdf80 legacy=0x7f7ac415aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.800+0000 7f7ac3e00640 1 -- 172.21.15.28:0/774343372 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702944571 0 0) 0x7f7ac4159e10 con 0x7f7ac40cdf80 2024-09-13T17:14:14.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.800+0000 7f7ac9c00640 1 -- 172.21.15.28:0/774343372 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac415a6b0 con 0x7f7ac4070e30 2024-09-13T17:14:14.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.801+0000 7f7ac9c00640 1 --2- 172.21.15.28:0/774343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4070e30 0x7f7ac4071c00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80097a0 tx=0x7f7ab802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea686df53292543 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.801+0000 7f7ac3e00640 1 -- 172.21.15.28:0/774343372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8032440 con 0x7f7ac4070e30 2024-09-13T17:14:14.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.801+0000 7f7ac3e00640 1 -- 172.21.15.28:0/774343372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab80325a0 con 0x7f7ac4070e30 2024-09-13T17:14:14.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.801+0000 7f7ac3e00640 1 -- 172.21.15.28:0/774343372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8032930 con 0x7f7ac4070e30 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 -- 172.21.15.28:0/774343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4070e30 msgr2=0x7f7ac4071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 --2- 172.21.15.28:0/774343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4070e30 0x7f7ac4071c00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7ab80097a0 tx=0x7f7ab802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 -- 172.21.15.28:0/774343372 shutdown_connections 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 --2- 172.21.15.28:0/774343372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4070e30 0x7f7ac4071c00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 -- 172.21.15.28:0/774343372 >> 172.21.15.28:0/774343372 conn(0x7f7ac4069e80 msgr2=0x7f7ac406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 -- 172.21.15.28:0/774343372 shutdown_connections 2024-09-13T17:14:14.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.802+0000 7f7acb000640 1 -- 172.21.15.28:0/774343372 wait complete. 2024-09-13T17:14:14.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.803+0000 7f7acb000640 1 Processor -- start 2024-09-13T17:14:14.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 -- start start 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 0x7f7ac413eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ac40cdf80 0x7f7ac4151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac413ca60 con 0x7f7ac4072140 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac413e440 con 0x7f7ac4070e30 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.804+0000 7f7acb000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ac413efe0 con 0x7f7ac40cdf80 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36624/0 (socket says 172.21.15.28:36624) 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 -- 172.21.15.28:0/572904156 learned_addr learned my addr 172.21.15.28:0/572904156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9c00640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 0x7f7ac413eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7aca600640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ac40cdf80 0x7f7ac4151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 -- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ac40cdf80 msgr2=0x7f7ac4151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ac40cdf80 0x7f7ac4151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 -- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 msgr2=0x7f7ac413eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 0x7f7ac413eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9200640 1 -- 172.21.15.28:0/572904156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ab8009400 con 0x7f7ac4072140 2024-09-13T17:14:14.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.805+0000 7f7ac9c00640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 0x7f7ac413eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:14.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7ac9200640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c400 tx=0x7f7ab000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0011790 con 0x7f7ac4072140 2024-09-13T17:14:14.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ab0011930 con 0x7f7ac4072140 2024-09-13T17:14:14.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0011ce0 con 0x7f7ac4072140 2024-09-13T17:14:14.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ac413f2b0 con 0x7f7ac4072140 2024-09-13T17:14:14.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.806+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ac413f870 con 0x7f7ac4072140 2024-09-13T17:14:14.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.807+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ac406a390 con 0x7f7ac4072140 2024-09-13T17:14:14.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.808+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7ab001c460 con 0x7f7ac4072140 2024-09-13T17:14:14.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.809+0000 7f7ac1600640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 0x7f7aa406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:14.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.809+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f7ab0089470 con 0x7f7ac4072140 2024-09-13T17:14:14.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.809+0000 7f7ac9c00640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 0x7f7aa406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:14.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.810+0000 7f7ac9c00640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 0x7f7aa406d0c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8034000 tx=0x7f7ab802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:14.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.813+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7ab00544d0 con 0x7f7ac4072140 2024-09-13T17:14:14.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.989+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ac4073a20 con 0x7f7ac4072140 2024-09-13T17:14:14.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.990+0000 7f7ac1600640 1 -- 172.21.15.28:0/572904156 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7ab0057af0 con 0x7f7ac4072140 2024-09-13T17:14:14.991 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 msgr2=0x7f7aa406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 0x7f7aa406d0c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8034000 tx=0x7f7ab802e000 comp rx=0 tx=0).stop 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 msgr2=0x7f7ac41e6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000c400 tx=0x7f7ab000c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 shutdown_connections 2024-09-13T17:14:14.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7aa406ac00 0x7f7aa406d0c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ac40cdf80 0x7f7ac4151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ac4072140 0x7f7ac41e6e90 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 --2- 172.21.15.28:0/572904156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ac4070e30 0x7f7ac413eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 >> 172.21.15.28:0/572904156 conn(0x7f7ac4069e80 msgr2=0x7f7ac415cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 shutdown_connections 2024-09-13T17:14:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:14.992+0000 7f7acb000640 1 -- 172.21.15.28:0/572904156 wait complete. 2024-09-13T17:14:15.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:14:15.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.166+0000 7f7f37200640 1 Processor -- start 2024-09-13T17:14:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.167+0000 7f7f37200640 1 -- start start 2024-09-13T17:14:15.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f37200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7f38070e30 0x7f7f38071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f37200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 0x7f7f3815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f37200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f38159e10 con 0x7f7f38072140 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f37200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3815a0f0 con 0x7f7f38070e30 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f37200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3815a3d0 con 0x7f7f380cdf80 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f35e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7f38070e30 0x7f7f38071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f35e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7f38070e30 0x7f7f38071c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36632/0 (socket says 172.21.15.28:36632) 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.168+0000 7f7f35e00640 1 -- 172.21.15.28:0/356102414 learned_addr learned my addr 172.21.15.28:0/356102414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f36800640 1 --2- 172.21.15.28:0/356102414 >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 0x7f7f3815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f35e00640 1 -- 172.21.15.28:0/356102414 >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 msgr2=0x7f7f3815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f35e00640 1 --2- 172.21.15.28:0/356102414 >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 0x7f7f3815aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f35e00640 1 -- 172.21.15.28:0/356102414 >> v1:172.21.15.189:6789/0 conn(0x7f7f38072140 legacy=0x7f7f380cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f35e00640 1 -- 172.21.15.28:0/356102414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f3815a6b0 con 0x7f7f38070e30 2024-09-13T17:14:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.169+0000 7f7f36800640 1 --2- 172.21.15.28:0/356102414 >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 0x7f7f3815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.170+0000 7f7f35e00640 1 --2- 172.21.15.28:0/356102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38070e30 0x7f7f38071c00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7f380c33f0 tx=0x7f7f2802c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dab877ac9e938fda server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.170+0000 7f7f35400640 1 -- 172.21.15.28:0/356102414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2802f070 con 0x7f7f38070e30 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.170+0000 7f7f35400640 1 -- 172.21.15.28:0/356102414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f280325a0 con 0x7f7f38070e30 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.170+0000 7f7f35400640 1 -- 172.21.15.28:0/356102414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f28032930 con 0x7f7f38070e30 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 -- 172.21.15.28:0/356102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38070e30 msgr2=0x7f7f38071c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 --2- 172.21.15.28:0/356102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38070e30 0x7f7f38071c00 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7f380c33f0 tx=0x7f7f2802c8e0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 -- 172.21.15.28:0/356102414 shutdown_connections 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 --2- 172.21.15.28:0/356102414 >> v2:172.21.15.189:3300/0 conn(0x7f7f380cdf80 0x7f7f3815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 --2- 172.21.15.28:0/356102414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f38070e30 0x7f7f38071c00 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 -- 172.21.15.28:0/356102414 >> 172.21.15.28:0/356102414 conn(0x7f7f38069e80 msgr2=0x7f7f3806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 -- 172.21.15.28:0/356102414 shutdown_connections 2024-09-13T17:14:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.171+0000 7f7f37200640 1 -- 172.21.15.28:0/356102414 wait complete. 2024-09-13T17:14:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.172+0000 7f7f37200640 1 Processor -- start 2024-09-13T17:14:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.173+0000 7f7f37200640 1 -- start start 2024-09-13T17:14:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.173+0000 7f7f37200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f38070e30 0x7f7f3814c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.173+0000 7f7f37200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 0x7f7f38136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f37200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f37200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3814f220 con 0x7f7f380cdf80 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f37200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3814e6a0 con 0x7f7f38070e30 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f37200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f3814e980 con 0x7f7f38072140 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36648/0 (socket says 172.21.15.28:36648) 2024-09-13T17:14:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 -- 172.21.15.28:0/3669718613 learned_addr learned my addr 172.21.15.28:0/3669718613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f35e00640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f38070e30 0x7f7f3814c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f2d400640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 0x7f7f38136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 -- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 msgr2=0x7f7f38136320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 0x7f7f38136320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 -- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f38070e30 msgr2=0x7f7f3814c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f38070e30 0x7f7f3814c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f36800640 1 -- 172.21.15.28:0/3669718613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2800b770 con 0x7f7f380cdf80 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.174+0000 7f7f2d400640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 0x7f7f38136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f36800640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000cd50 tx=0x7f7f3000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000f680 con 0x7f7f380cdf80 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3000f800 con 0x7f7f380cdf80 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3000fbb0 con 0x7f7f380cdf80 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f3814ec60 con 0x7f7f380cdf80 2024-09-13T17:14:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.175+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f3813d060 con 0x7f7f380cdf80 2024-09-13T17:14:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.176+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3813d280 con 0x7f7f380cdf80 2024-09-13T17:14:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.177+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7f3000b040 con 0x7f7f380cdf80 2024-09-13T17:14:15.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.178+0000 7f7f2ea00640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 0x7f7f1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.179+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f7f30089d60 con 0x7f7f380cdf80 2024-09-13T17:14:15.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.179+0000 7f7f35e00640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 0x7f7f1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.180+0000 7f7f35e00640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 0x7f7f1406d0c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7f28037000 tx=0x7f7f2803e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.183+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7f30054e50 con 0x7f7f380cdf80 2024-09-13T17:14:15.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.361+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f3806f720 con 0x7f7f380cdf80 2024-09-13T17:14:15.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.361+0000 7f7f2ea00640 1 -- 172.21.15.28:0/3669718613 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f3001d020 con 0x7f7f380cdf80 2024-09-13T17:14:15.363 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 msgr2=0x7f7f1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 0x7f7f1406d0c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f7f28037000 tx=0x7f7f2803e040 comp rx=0 tx=0).stop 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 msgr2=0x7f7f38136860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f7f3000cd50 tx=0x7f7f3000a650 comp rx=0 tx=0).stop 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 shutdown_connections 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f1406ac00 0x7f7f1406d0c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f380cdf80 0x7f7f38136860 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f38072140 0x7f7f38136320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 --2- 172.21.15.28:0/3669718613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f38070e30 0x7f7f3814c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 >> 172.21.15.28:0/3669718613 conn(0x7f7f38069e80 msgr2=0x7f7f38153400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 shutdown_connections 2024-09-13T17:14:15.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.364+0000 7f7f37200640 1 -- 172.21.15.28:0/3669718613 wait complete. 2024-09-13T17:14:15.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.535+0000 7f6a2b400640 1 Processor -- start 2024-09-13T17:14:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.535+0000 7f6a2b400640 1 -- start start 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a2406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a2b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2415e100 con 0x7f6a2415a540 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a2b400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a2415e3e0 con 0x7f6a240cabc0 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a2b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240adb00 con 0x7f6a240713d0 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a28a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a2415a540 0x7f6a2415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60200/0 (socket says 172.21.15.28:60200) 2024-09-13T17:14:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.536+0000 7f6a28a00640 1 -- 172.21.15.28:0/3703870876 learned_addr learned my addr 172.21.15.28:0/3703870876 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622255973 0 0) 0x7f6a2415e100 con 0x7f6a2415a540 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c003520 con 0x7f6a2415a540 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a29400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a2406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1310997872 0 0) 0x7f6a0c003520 con 0x7f6a2415a540 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 msgr2=0x7f6a2406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a2406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 >> v1:172.21.15.189:6789/0 conn(0x7f6a240cabc0 legacy=0x7f6a24070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2415ef80 con 0x7f6a2415a540 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6a18001670 con 0x7f6a2415a540 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 >> v1:172.21.15.28:6789/0 conn(0x7f6a2415a540 legacy=0x7f6a2415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0c0039a0 0x7f6a0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0c004d00 0x7f6a0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c00bb40 con 0x7f6a0c004d00 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c00bcb0 con 0x7f6a0c0039a0 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.537+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c00be70 con 0x7f6a240713d0 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0c004d00 0x7f6a0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a28a00640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0c0039a0 0x7f6a0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a23e00640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 -- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 msgr2=0x7f6a0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a240713d0 0x7f6a0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 -- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0c0039a0 msgr2=0x7f6a0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a0c0039a0 0x7f6a0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.538+0000 7f6a29400640 1 -- 172.21.15.28:0/3703870876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c003520 con 0x7f6a0c004d00 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.539+0000 7f6a29400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0c004d00 0x7f6a0c005180 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009fc0 tx=0x7f6a0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec6bc8d97f19975 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.539+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c031640 con 0x7f6a0c004d00 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.539+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c02d070 con 0x7f6a0c004d00 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.539+0000 7f6a23400640 1 -- 172.21.15.28:0/3703870876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c0319e0 con 0x7f6a0c004d00 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 -- 172.21.15.28:0/3703870876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0c004d00 msgr2=0x7f6a0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 --2- 172.21.15.28:0/3703870876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a0c004d00 0x7f6a0c005180 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c009fc0 tx=0x7f6a0c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a29400640 1 -- 172.21.15.28:0/3703870876 reap_dead start 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 -- 172.21.15.28:0/3703870876 shutdown_connections 2024-09-13T17:14:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 -- 172.21.15.28:0/3703870876 >> 172.21.15.28:0/3703870876 conn(0x7f6a24069e80 msgr2=0x7f6a2406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 -- 172.21.15.28:0/3703870876 shutdown_connections 2024-09-13T17:14:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.540+0000 7f6a2b400640 1 -- 172.21.15.28:0/3703870876 wait complete. 2024-09-13T17:14:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.542+0000 7f6a2b400640 1 Processor -- start 2024-09-13T17:14:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.542+0000 7f6a2b400640 1 -- start start 2024-09-13T17:14:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0c004d00 0x7f6a2414d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 0x7f6a2415d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2415aaa0 con 0x7f6a240ac850 2024-09-13T17:14:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2415b5e0 con 0x7f6a2414e960 2024-09-13T17:14:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a2b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a240ad1f0 con 0x7f6a0c004d00 2024-09-13T17:14:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36674/0 (socket says 172.21.15.28:36674) 2024-09-13T17:14:15.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 -- 172.21.15.28:0/681013065 learned_addr learned my addr 172.21.15.28:0/681013065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a28a00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0c004d00 0x7f6a2414d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a23e00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 0x7f6a2415d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 -- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0c004d00 msgr2=0x7f6a2414d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0c004d00 0x7f6a2414d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 -- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 msgr2=0x7f6a2415d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 0x7f6a2415d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a29400640 1 -- 172.21.15.28:0/681013065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2415acb0 con 0x7f6a240ac850 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.543+0000 7f6a23e00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 0x7f6a2415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a29400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c02aee0 tx=0x7f6a1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c031df0 con 0x7f6a240ac850 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c040070 con 0x7f6a240ac850 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a1c003bd0 con 0x7f6a240ac850 2024-09-13T17:14:15.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2415af90 con 0x7f6a240ac850 2024-09-13T17:14:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.544+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2415d9b0 con 0x7f6a240ac850 2024-09-13T17:14:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.545+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a24061090 con 0x7f6a240ac850 2024-09-13T17:14:15.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.546+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a1c042070 con 0x7f6a240ac850 2024-09-13T17:14:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.547+0000 7f6a20c00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 0x7f6a0406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.547+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f6a1c0aa430 con 0x7f6a240ac850 2024-09-13T17:14:15.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.548+0000 7f6a28a00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 0x7f6a0406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.548+0000 7f6a28a00640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 0x7f6a0406d0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6a18001eb0 tx=0x7f6a18001690 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.551+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6a1c0754b0 con 0x7f6a240ac850 2024-09-13T17:14:15.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.770+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a2415b770 con 0x7f6a240ac850 2024-09-13T17:14:15.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.770+0000 7f6a20c00640 1 -- 172.21.15.28:0/681013065 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f6a1c04a020 con 0x7f6a240ac850 2024-09-13T17:14:15.771 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:15.771 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:15.772 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 msgr2=0x7f6a0406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 0x7f6a0406d0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6a18001eb0 tx=0x7f6a18001690 comp rx=0 tx=0).stop 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 msgr2=0x7f6a240accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c02aee0 tx=0x7f6a1c009750 comp rx=0 tx=0).stop 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 shutdown_connections 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a0406ac30 0x7f6a0406d0f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a240ac850 0x7f6a240accb0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a2414e960 0x7f6a2415d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 --2- 172.21.15.28:0/681013065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a0c004d00 0x7f6a2414d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 >> 172.21.15.28:0/681013065 conn(0x7f6a24069e80 msgr2=0x7f6a240aecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 shutdown_connections 2024-09-13T17:14:15.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.773+0000 7f6a2b400640 1 -- 172.21.15.28:0/681013065 wait complete. 2024-09-13T17:14:15.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:15.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.945+0000 7f20a2800640 1 Processor -- start 2024-09-13T17:14:15.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.945+0000 7f20a2800640 1 -- start start 2024-09-13T17:14:15.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a2800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f209c159440 0x7f209c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f209c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0c3850 con 0x7f209c0b22e0 2024-09-13T17:14:15.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a2800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209c15a930 con 0x7f209c159440 2024-09-13T17:14:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a2800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209c159d20 con 0x7f209c159820 2024-09-13T17:14:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f209c0b22e0 0x7f209c0b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60216/0 (socket says 172.21.15.28:60216) 2024-09-13T17:14:15.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.947+0000 7f20a1e00640 1 -- 172.21.15.28:0/3828260983 learned_addr learned my addr 172.21.15.28:0/3828260983 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f20a1400640 1 --2- 172.21.15.28:0/3828260983 >> v2:172.21.15.189:3300/0 conn(0x7f209c159440 0x7f209c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f20a0a00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f209c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 267893915 0 0) 0x7f209c0c3850 con 0x7f209c0b22e0 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2084005730 con 0x7f209c0b22e0 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582264415 0 0) 0x7f2084005730 con 0x7f209c0b22e0 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 msgr2=0x7f209c0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f209c0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 >> v2:172.21.15.189:3300/0 conn(0x7f209c159440 msgr2=0x7f209c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 --2- 172.21.15.28:0/3828260983 >> v2:172.21.15.189:3300/0 conn(0x7f209c159440 0x7f209c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c15a000 con 0x7f209c0b22e0 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f20940019f0 con 0x7f209c0b22e0 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 >> v1:172.21.15.28:6789/0 conn(0x7f209c0b22e0 legacy=0x7f209c0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084005b80 0x7f2084005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f20840069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2084006f10 0x7f2084007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c15a930 con 0x7f2084006f10 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c159d20 con 0x7f2084005b80 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.948+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f208400db90 con 0x7f209c159820 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2084006f10 0x7f2084007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a0a00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f20840069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1400640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084005b80 0x7f2084005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 -- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 msgr2=0x7f20840069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c159820 0x7f20840069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 -- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084005b80 msgr2=0x7f2084005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084005b80 0x7f2084005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1e00640 1 -- 172.21.15.28:0/3828260983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084005730 con 0x7f2084006f10 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.949+0000 7f20a1400640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084005b80 0x7f2084005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.950+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2084006f10 0x7f2084007390 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f209400a860 tx=0x7f2084007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7679a91a80b6bce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.950+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209402ba80 con 0x7f2084006f10 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.950+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209402bbe0 con 0x7f2084006f10 2024-09-13T17:14:15.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.950+0000 7f209be00640 1 -- 172.21.15.28:0/3828260983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094034470 con 0x7f2084006f10 2024-09-13T17:14:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 -- 172.21.15.28:0/3828260983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2084006f10 msgr2=0x7f2084007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 --2- 172.21.15.28:0/3828260983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2084006f10 0x7f2084007390 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f209400a860 tx=0x7f2084007a00 comp rx=0 tx=0).stop 2024-09-13T17:14:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a1e00640 1 -- 172.21.15.28:0/3828260983 reap_dead start 2024-09-13T17:14:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 -- 172.21.15.28:0/3828260983 shutdown_connections 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 -- 172.21.15.28:0/3828260983 >> 172.21.15.28:0/3828260983 conn(0x7f209c072e40 msgr2=0x7f209c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 -- 172.21.15.28:0/3828260983 shutdown_connections 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.951+0000 7f20a2800640 1 -- 172.21.15.28:0/3828260983 wait complete. 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.953+0000 7f20a2800640 1 Processor -- start 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.953+0000 7f20a2800640 1 -- start start 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 0x7f209c1540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 0x7f209c0c6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c153f50 con 0x7f209c0b1740 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c0b2100 con 0x7f2084006f10 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a2800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209c0c6840 con 0x7f209c153ad0 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36692/0 (socket says 172.21.15.28:36692) 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 -- 172.21.15.28:0/3090499025 learned_addr learned my addr 172.21.15.28:0/3090499025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1400640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 0x7f209c1540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 -- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 msgr2=0x7f209c0c6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a0a00640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 0x7f209c0c6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 0x7f209c0c6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 -- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 msgr2=0x7f209c1540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 0x7f209c1540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1e00640 1 -- 172.21.15.28:0/3090499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c0710a0 con 0x7f209c0b1740 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a1400640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 0x7f209c1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.954+0000 7f20a0a00640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 0x7f209c0c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f20a1e00640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f209400a860 tx=0x7f2094001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2094038070 con 0x7f209c0b1740 2024-09-13T17:14:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2094034670 con 0x7f209c0b1740 2024-09-13T17:14:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20940349e0 con 0x7f209c0b1740 2024-09-13T17:14:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209c1f2050 con 0x7f209c0b1740 2024-09-13T17:14:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.955+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209c1f25d0 con 0x7f209c0b1740 2024-09-13T17:14:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.956+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c0b0440 con 0x7f209c0b1740 2024-09-13T17:14:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.957+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2094031070 con 0x7f209c0b1740 2024-09-13T17:14:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.958+0000 7f2099600640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 0x7f207c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.958+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f209403b040 con 0x7f209c0b1740 2024-09-13T17:14:15.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.959+0000 7f20a1400640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 0x7f207c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:15.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.959+0000 7f20a1400640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 0x7f207c06d0f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f209c1f30b0 tx=0x7f2090008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:15.963+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2094074420 con 0x7f209c0b1740 2024-09-13T17:14:16.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.143+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f209c06fe80 con 0x7f209c0b1740 2024-09-13T17:14:16.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.144+0000 7f2099600640 1 -- 172.21.15.28:0/3090499025 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f2094001fd0 con 0x7f209c0b1740 2024-09-13T17:14:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:16.145 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:00.308433+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.16","mappings":[{"from":1,"to":5}]},{"pgid":"13.2a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.189:6835/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.28:6834/2861751170":"2024-09-14T17:13:54.409423+0000","172.21.15.28:6832/2861751170":"2024-09-14T17:13:54.409423+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 msgr2=0x7f207c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 0x7f207c06d0f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f209c1f30b0 tx=0x7f2090008040 comp rx=0 tx=0).stop 2024-09-13T17:14:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 msgr2=0x7f209c0b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f209400a860 tx=0x7f2094001d80 comp rx=0 tx=0).stop 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 shutdown_connections 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f207c06ac30 0x7f207c06d0f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f209c0b1740 0x7f209c0b1bc0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f209c153ad0 0x7f209c0c6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 --2- 172.21.15.28:0/3090499025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2084006f10 0x7f209c1540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 >> 172.21.15.28:0/3090499025 conn(0x7f209c072e40 msgr2=0x7f209c157460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 shutdown_connections 2024-09-13T17:14:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.146+0000 7f20a2800640 1 -- 172.21.15.28:0/3090499025 wait complete. 2024-09-13T17:14:16.164 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:14:16.164 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:16.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.320+0000 7f76f2000640 1 Processor -- start 2024-09-13T17:14:16.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.320+0000 7f76f2000640 1 -- start start 2024-09-13T17:14:16.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f2000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 0x7f76ec0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f2000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec0c3850 con 0x7f76ec159820 2024-09-13T17:14:16.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f2000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ec15a930 con 0x7f76ec159440 2024-09-13T17:14:16.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f2000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec159d20 con 0x7f76ec0b29f0 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f0c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 0x7f76ec0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:42174/0 (socket says 172.21.15.28:42174) 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f1600640 1 -- 172.21.15.28:0/2956990173 learned_addr learned my addr 172.21.15.28:0/2956990173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.322+0000 7f76f0c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 0x7f76ec0b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46376/0 (socket says 172.21.15.28:46376) 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.323+0000 7f76f1600640 1 -- 172.21.15.28:0/2956990173 >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 msgr2=0x7f76ec0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.323+0000 7f76f1600640 1 --2- 172.21.15.28:0/2956990173 >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 0x7f76ec0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.323+0000 7f76f1600640 1 -- 172.21.15.28:0/2956990173 >> v1:172.21.15.189:6789/0 conn(0x7f76ec159820 legacy=0x7f76ec0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.323+0000 7f76f1600640 1 -- 172.21.15.28:0/2956990173 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec15a000 con 0x7f76ec0b29f0 2024-09-13T17:14:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.324+0000 7f76f1600640 1 --2- 172.21.15.28:0/2956990173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009e80 tx=0x7f76e402a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b824ac79ff060df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.324+0000 7f76eb400640 1 -- 172.21.15.28:0/2956990173 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4031600 con 0x7f76ec0b29f0 2024-09-13T17:14:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.324+0000 7f76eb400640 1 -- 172.21.15.28:0/2956990173 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e402d050 con 0x7f76ec0b29f0 2024-09-13T17:14:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.324+0000 7f76eb400640 1 -- 172.21.15.28:0/2956990173 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e4031980 con 0x7f76ec0b29f0 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 -- 172.21.15.28:0/2956990173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 msgr2=0x7f76ec0b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 --2- 172.21.15.28:0/2956990173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009e80 tx=0x7f76e402a7d0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 -- 172.21.15.28:0/2956990173 shutdown_connections 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 --2- 172.21.15.28:0/2956990173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec0b0980 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 --2- 172.21.15.28:0/2956990173 >> v2:172.21.15.189:3300/0 conn(0x7f76ec159440 0x7f76ec0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 -- 172.21.15.28:0/2956990173 >> 172.21.15.28:0/2956990173 conn(0x7f76ec072e40 msgr2=0x7f76ec073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 -- 172.21.15.28:0/2956990173 shutdown_connections 2024-09-13T17:14:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.325+0000 7f76f2000640 1 -- 172.21.15.28:0/2956990173 wait complete. 2024-09-13T17:14:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.327+0000 7f76f2000640 1 Processor -- start 2024-09-13T17:14:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.327+0000 7f76f2000640 1 -- start start 2024-09-13T17:14:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec163050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76ec159820 0x7f76ec14b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec153980 con 0x7f76ec159440 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec164560 con 0x7f76ec159820 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f2000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ec163950 con 0x7f76ec0b29f0 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36706/0 (socket says 172.21.15.28:36706) 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 -- 172.21.15.28:0/3905709115 learned_addr learned my addr 172.21.15.28:0/3905709115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f0c00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec163050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76f1600640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76ec159820 0x7f76ec14b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 -- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 msgr2=0x7f76ec163050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec163050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 -- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76ec159820 msgr2=0x7f76ec14b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76ec159820 0x7f76ec14b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.328+0000 7f76ebe00640 1 -- 172.21.15.28:0/3905709115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4009b30 con 0x7f76ec159440 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76ebe00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f76d800c3c0 tx=0x7f76d800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d80052d0 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d8005450 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec163c30 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d8003830 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.329+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec1641f0 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.330+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ec0b22e0 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.331+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76d8003990 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.332+0000 7f76e8c00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 0x7f76cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.332+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f76d80888a0 con 0x7f76ec159440 2024-09-13T17:14:16.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.332+0000 7f76f0c00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 0x7f76cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.333+0000 7f76f0c00640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 0x7f76cc06d0f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b40 tx=0x7f76e0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.336+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f76d80539b0 con 0x7f76ec159440 2024-09-13T17:14:16.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.549+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76ec073220 con 0x7f76ec159440 2024-09-13T17:14:16.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.549+0000 7f76e8c00640 1 -- 172.21.15.28:0/3905709115 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f76d8017090 con 0x7f76ec159440 2024-09-13T17:14:16.550 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:16.550 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:16.551 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:16.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 msgr2=0x7f76cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 0x7f76cc06d0f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009b40 tx=0x7f76e0009190 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 msgr2=0x7f76ec15b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f76d800c3c0 tx=0x7f76d800c880 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 shutdown_connections 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76cc06ac30 0x7f76cc06d0f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76ec159820 0x7f76ec14b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76ec159440 0x7f76ec15b8a0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 --2- 172.21.15.28:0/3905709115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76ec0b29f0 0x7f76ec163050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.551+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 >> 172.21.15.28:0/3905709115 conn(0x7f76ec072e40 msgr2=0x7f76ec0b0440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.552+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 shutdown_connections 2024-09-13T17:14:16.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.552+0000 7f76f2000640 1 -- 172.21.15.28:0/3905709115 wait complete. 2024-09-13T17:14:16.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.722+0000 7f3aa3200640 1 Processor -- start 2024-09-13T17:14:16.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.722+0000 7f3aa3200640 1 -- start start 2024-09-13T17:14:16.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa3200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a9c146170 0x7f3a9c15d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c0acf60 0x7f3a9c160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c146550 con 0x7f3a9c160310 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa3200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a9c15cd20 con 0x7f3a9c146170 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c15d280 con 0x7f3a9c0acf60 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3a93e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3a9c160310 0x7f3a9c0ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60222/0 (socket says 172.21.15.28:60222) 2024-09-13T17:14:16.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3a93e00640 1 -- 172.21.15.28:0/769499563 learned_addr learned my addr 172.21.15.28:0/769499563 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3a9be00640 1 --2- 172.21.15.28:0/769499563 >> v2:172.21.15.189:3300/0 conn(0x7f3a9c146170 0x7f3a9c15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3aa1200640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c0acf60 0x7f3a9c160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3a9b400640 1 -- 172.21.15.28:0/769499563 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496119897 0 0) 0x7f3a9c146550 con 0x7f3a9c160310 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.724+0000 7f3a9b400640 1 -- 172.21.15.28:0/769499563 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a80003520 con 0x7f3a9c160310 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.725+0000 7f3a9be00640 1 -- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c0acf60 msgr2=0x7f3a9c160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.725+0000 7f3a9be00640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c0acf60 0x7f3a9c160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.725+0000 7f3a9be00640 1 -- 172.21.15.28:0/769499563 >> v1:172.21.15.28:6789/0 conn(0x7f3a9c160310 legacy=0x7f3a9c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.725+0000 7f3a9be00640 1 -- 172.21.15.28:0/769499563 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c15dc50 con 0x7f3a9c146170 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3a9be00640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c146170 0x7f3a9c15d710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0097a0 tx=0x7f3a8c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23eeaff207085a1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3a9b400640 1 -- 172.21.15.28:0/769499563 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c032440 con 0x7f3a9c146170 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3a9b400640 1 -- 172.21.15.28:0/769499563 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c0325a0 con 0x7f3a9c146170 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3a9b400640 1 -- 172.21.15.28:0/769499563 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c032930 con 0x7f3a9c146170 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3aa3200640 1 -- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c146170 msgr2=0x7f3a9c15d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.726+0000 7f3aa3200640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c146170 0x7f3a9c15d710 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c0097a0 tx=0x7f3a8c02a180 comp rx=0 tx=0).stop 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 -- 172.21.15.28:0/769499563 shutdown_connections 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c0acf60 0x7f3a9c160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 --2- 172.21.15.28:0/769499563 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c146170 0x7f3a9c15d710 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 -- 172.21.15.28:0/769499563 >> 172.21.15.28:0/769499563 conn(0x7f3a9c069e80 msgr2=0x7f3a9c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:16.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 -- 172.21.15.28:0/769499563 shutdown_connections 2024-09-13T17:14:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.727+0000 7f3aa3200640 1 -- 172.21.15.28:0/769499563 wait complete. 2024-09-13T17:14:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.728+0000 7f3aa3200640 1 Processor -- start 2024-09-13T17:14:16.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.729+0000 7f3aa3200640 1 -- start start 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.729+0000 7f3aa3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c0acf60 0x7f3a9c0ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.729+0000 7f3aa3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 0x7f3a9c156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1f2e00 con 0x7f3a9c160310 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1f24d0 con 0x7f3a9c0acf60 2024-09-13T17:14:16.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1f27b0 con 0x7f3a9c146170 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36722/0 (socket says 172.21.15.28:36722) 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 -- 172.21.15.28:0/3641961199 learned_addr learned my addr 172.21.15.28:0/3641961199 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c0acf60 0x7f3a9c0ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3a93e00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 0x7f3a9c156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 -- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 msgr2=0x7f3a9c156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 0x7f3a9c156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 -- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c0acf60 msgr2=0x7f3a9c0ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c0acf60 0x7f3a9c0ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3aa1200640 1 -- 172.21.15.28:0/3641961199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c009400 con 0x7f3a9c160310 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.730+0000 7f3a93e00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 0x7f3a9c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:16.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3aa1200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400ef90 tx=0x7f3a9400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a94009b20 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9400f040 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c1f2a90 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a940146c0 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.731+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c0cee10 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.732+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9c1f2f70 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.733+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a94014820 con 0x7f3a9c160310 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.734+0000 7f3a98c00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 0x7f3a7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:16.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.734+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f3a9408a2f0 con 0x7f3a9c160310 2024-09-13T17:14:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.734+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 0x7f3a7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:16.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.735+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 0x7f3a7806d0c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c034000 tx=0x7f3a8c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:16.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.738+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3a940552d0 con 0x7f3a9c160310 2024-09-13T17:14:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.953+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a9c160750 con 0x7f3a9c160310 2024-09-13T17:14:16.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.953+0000 7f3a98c00640 1 -- 172.21.15.28:0/3641961199 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f3a94009e60 con 0x7f3a9c160310 2024-09-13T17:14:16.954 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:16.955 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:13:42.545599+0000","modified":"2024-09-13T17:14:03.336862+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7725},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7725":{"gid":7725,"name":"a","rank":0,"incarnation":83,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6835/2200414779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2200414779},{"type":"v1","addr":"172.21.15.28:6835","nonce":2200414779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T17:14:16.955 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 86 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.955+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 msgr2=0x7f3a7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.955+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 0x7f3a7806d0c0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c034000 tx=0x7f3a8c02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.955+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 msgr2=0x7f3a9c156550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.955+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3a9400ef90 tx=0x7f3a9400c490 comp rx=0 tx=0).stop 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 shutdown_connections 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7806ac00 0x7f3a7806d0c0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c160310 0x7f3a9c156550 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c146170 0x7f3a9c156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 --2- 172.21.15.28:0/3641961199 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a9c0acf60 0x7f3a9c0ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:16.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 >> 172.21.15.28:0/3641961199 conn(0x7f3a9c069e80 msgr2=0x7f3a9c06b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:16.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 shutdown_connections 2024-09-13T17:14:16.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:16.956+0000 7f3aa3200640 1 -- 172.21.15.28:0/3641961199 wait complete. 2024-09-13T17:14:16.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.128+0000 7f3cdc600640 1 Processor -- start 2024-09-13T17:14:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.128+0000 7f3cdc600640 1 -- start start 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.129+0000 7f3cdc600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cd40cdfa0 0x7f3cd415d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cdc600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3cd406fb50 0x7f3cd406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cdc600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cd40b3970 con 0x7f3cd40b3160 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cdc600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd4159230 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cdc600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cd4159430 con 0x7f3cd406fb50 2024-09-13T17:14:17.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3cd40cdfa0 0x7f3cd415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3cd40b3160 0x7f3cd406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60228/0 (socket says 172.21.15.28:60228) 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9200640 1 -- 172.21.15.28:0/4222858822 learned_addr learned my addr 172.21.15.28:0/4222858822 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cda600640 1 --2- 172.21.15.28:0/4222858822 >> v2:172.21.15.189:3300/0 conn(0x7f3cd406fb50 0x7f3cd406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9c00640 1 -- 172.21.15.28:0/4222858822 >> v2:172.21.15.189:3300/0 conn(0x7f3cd406fb50 msgr2=0x7f3cd406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/4222858822 >> v2:172.21.15.189:3300/0 conn(0x7f3cd406fb50 0x7f3cd406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9c00640 1 -- 172.21.15.28:0/4222858822 >> v1:172.21.15.28:6789/0 conn(0x7f3cd40b3160 legacy=0x7f3cd406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd9c00640 1 -- 172.21.15.28:0/4222858822 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4159710 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.130+0000 7f3cd3e00640 1 -- 172.21.15.28:0/4222858822 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157502469 0 0) 0x7f3cc0001110 con 0x7f3cd40b3160 2024-09-13T17:14:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.131+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/4222858822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40cdfa0 0x7f3cd415d340 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b1b50 tx=0x7f3cc802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d2a8ccf9481a8f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cd3e00640 1 -- 172.21.15.28:0/4222858822 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8032440 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cd3e00640 1 -- 172.21.15.28:0/4222858822 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc80325a0 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cd3e00640 1 -- 172.21.15.28:0/4222858822 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8032930 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 -- 172.21.15.28:0/4222858822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40cdfa0 msgr2=0x7f3cd415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 --2- 172.21.15.28:0/4222858822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40cdfa0 0x7f3cd415d340 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b1b50 tx=0x7f3cc802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 -- 172.21.15.28:0/4222858822 shutdown_connections 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 --2- 172.21.15.28:0/4222858822 >> v2:172.21.15.189:3300/0 conn(0x7f3cd406fb50 0x7f3cd406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 --2- 172.21.15.28:0/4222858822 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40cdfa0 0x7f3cd415d340 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.132+0000 7f3cdc600640 1 -- 172.21.15.28:0/4222858822 >> 172.21.15.28:0/4222858822 conn(0x7f3cd4069e80 msgr2=0x7f3cd406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.133+0000 7f3cdc600640 1 -- 172.21.15.28:0/4222858822 shutdown_connections 2024-09-13T17:14:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.133+0000 7f3cdc600640 1 -- 172.21.15.28:0/4222858822 wait complete. 2024-09-13T17:14:17.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.134+0000 7f3cdc600640 1 Processor -- start 2024-09-13T17:14:17.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.134+0000 7f3cdc600640 1 -- start start 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 0x7f3cd4154360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3cd40cdfa0 0x7f3cd4152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4153350 con 0x7f3cd40b3160 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd4153ed0 con 0x7f3cd406fb50 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cdc600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cd41538f0 con 0x7f3cd40cdfa0 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cd9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cd9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36732/0 (socket says 172.21.15.28:36732) 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.135+0000 7f3cd9200640 1 -- 172.21.15.28:0/3252648544 learned_addr learned my addr 172.21.15.28:0/3252648544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 -- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3cd40cdfa0 msgr2=0x7f3cd4152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3cd40cdfa0 0x7f3cd4152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 0x7f3cd4154360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 -- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 msgr2=0x7f3cd4154360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 0x7f3cd4154360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 -- 172.21.15.28:0/3252648544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc8009400 con 0x7f3cd40b3160 2024-09-13T17:14:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 0x7f3cd4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:17.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd9200640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c7b0 tx=0x7f3cc000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00072d0 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc0007450 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc00047e0 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cd414d8e0 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.136+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cd414dbc0 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.137+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cd415d340 con 0x7f3cd40b3160 2024-09-13T17:14:17.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.138+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3cc0004960 con 0x7f3cd40b3160 2024-09-13T17:14:17.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.140+0000 7f3cd1600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 0x7f3cb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.140+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f3cc00884f0 con 0x7f3cd40b3160 2024-09-13T17:14:17.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.140+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 0x7f3cb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.141+0000 7f3cd9c00640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 0x7f3cb406d0c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8034000 tx=0x7f3cc802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.144+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3cc0053550 con 0x7f3cd40b3160 2024-09-13T17:14:17.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.328+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cd40b3970 con 0x7f3cd40b3160 2024-09-13T17:14:17.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.329+0000 7f3cd1600640 1 -- 172.21.15.28:0/3252648544 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16967 (secure 0 0 0) 0x7f3cd40b3970 con 0x7f3cd40b3160 2024-09-13T17:14:17.331 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:17.331 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:00.308433+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:13:40.462897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-09-13T17:13:41.537996+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.14","mappings":[{"from":6,"to":5}]},{"pgid":"13.16","mappings":[{"from":1,"to":5}]},{"pgid":"13.2a","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.189:6837/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.189:6835/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.28:6834/2861751170":"2024-09-14T17:13:54.409423+0000","172.21.15.28:6832/2861751170":"2024-09-14T17:13:54.409423+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.331+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 msgr2=0x7f3cb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.331+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 0x7f3cb406d0c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8034000 tx=0x7f3cc802e000 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.331+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 msgr2=0x7f3cd414d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.331+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f3cc000c7b0 tx=0x7f3cc000cc70 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 shutdown_connections 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3cb406ac00 0x7f3cb406d0c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3cd40cdfa0 0x7f3cd4152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3cd40b3160 0x7f3cd414d200 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 --2- 172.21.15.28:0/3252648544 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cd406fb50 0x7f3cd4154360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 >> 172.21.15.28:0/3252648544 conn(0x7f3cd4069e80 msgr2=0x7f3cd4165420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 shutdown_connections 2024-09-13T17:14:17.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.332+0000 7f3cdc600640 1 -- 172.21.15.28:0/3252648544 wait complete. 2024-09-13T17:14:17.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:14:17.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.504+0000 7f5a5b800640 1 Processor -- start 2024-09-13T17:14:17.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.504+0000 7f5a5b800640 1 -- start start 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415a780 0x7f5a54070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5a5415ab60 0x7f5a540712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 0x7f5a5406bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5406c8b0 con 0x7f5a5415ab60 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a5406cb90 con 0x7f5a5406bb40 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.505+0000 7f5a5b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540cc980 con 0x7f5a5415a780 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5a5415ab60 0x7f5a540712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5a5415ab60 0x7f5a540712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36748/0 (socket says 172.21.15.28:36748) 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 -- 172.21.15.28:0/1611904068 learned_addr learned my addr 172.21.15.28:0/1611904068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a59800640 1 --2- 172.21.15.28:0/1611904068 >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 0x7f5a5406bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a58e00640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415a780 0x7f5a54070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 -- 172.21.15.28:0/1611904068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415a780 msgr2=0x7f5a54070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415a780 0x7f5a54070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 -- 172.21.15.28:0/1611904068 >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 msgr2=0x7f5a5406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 --2- 172.21.15.28:0/1611904068 >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 0x7f5a5406bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a4be00640 1 -- 172.21.15.28:0/1611904068 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a540ccc60 con 0x7f5a5415ab60 2024-09-13T17:14:17.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.506+0000 7f5a59800640 1 --2- 172.21.15.28:0/1611904068 >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 0x7f5a5406bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.507+0000 7f5a4be00640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415ab60 0x7f5a540712c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5a540b1600 tx=0x7f5a3c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ede553c91e3f07c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.507+0000 7f5a53e00640 1 -- 172.21.15.28:0/1611904068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c032440 con 0x7f5a5415ab60 2024-09-13T17:14:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.507+0000 7f5a53e00640 1 -- 172.21.15.28:0/1611904068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c0325a0 con 0x7f5a5415ab60 2024-09-13T17:14:17.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.507+0000 7f5a53e00640 1 -- 172.21.15.28:0/1611904068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c032930 con 0x7f5a5415ab60 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 -- 172.21.15.28:0/1611904068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415ab60 msgr2=0x7f5a540712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415ab60 0x7f5a540712c0 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f5a540b1600 tx=0x7f5a3c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 -- 172.21.15.28:0/1611904068 shutdown_connections 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415ab60 0x7f5a540712c0 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 --2- 172.21.15.28:0/1611904068 >> v2:172.21.15.189:3300/0 conn(0x7f5a5406bb40 0x7f5a5406bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 --2- 172.21.15.28:0/1611904068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415a780 0x7f5a54070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 -- 172.21.15.28:0/1611904068 >> 172.21.15.28:0/1611904068 conn(0x7f5a540b0270 msgr2=0x7f5a540b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 -- 172.21.15.28:0/1611904068 shutdown_connections 2024-09-13T17:14:17.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.508+0000 7f5a5b800640 1 -- 172.21.15.28:0/1611904068 wait complete. 2024-09-13T17:14:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.510+0000 7f5a5b800640 1 Processor -- start 2024-09-13T17:14:17.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.510+0000 7f5a5b800640 1 -- start start 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 0x7f5a540afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 0x7f5a541e4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540b21b0 con 0x7f5a5415a780 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540aa810 con 0x7f5a5406bb40 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a5b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a540ad500 con 0x7f5a5415ab60 2024-09-13T17:14:17.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36764/0 (socket says 172.21.15.28:36764) 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a4be00640 1 -- 172.21.15.28:0/4065228268 learned_addr learned my addr 172.21.15.28:0/4065228268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a58e00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 0x7f5a540afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.511+0000 7f5a59800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 0x7f5a541e4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 -- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 msgr2=0x7f5a541e4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 0x7f5a541e4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 -- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 msgr2=0x7f5a540afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 0x7f5a540afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 -- 172.21.15.28:0/4065228268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c0094d0 con 0x7f5a5415a780 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a58e00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 0x7f5a540afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a59800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 0x7f5a541e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a4be00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c02a220 tx=0x7f5a3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c032440 con 0x7f5a5415a780 2024-09-13T17:14:17.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a3c030b00 con 0x7f5a5415a780 2024-09-13T17:14:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a540b1810 con 0x7f5a5415a780 2024-09-13T17:14:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c03d440 con 0x7f5a5415a780 2024-09-13T17:14:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.512+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a540b1dd0 con 0x7f5a5415a780 2024-09-13T17:14:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.513+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a5406bf90 con 0x7f5a5415a780 2024-09-13T17:14:17.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.514+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5a3c039040 con 0x7f5a5415a780 2024-09-13T17:14:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.515+0000 7f5a51600640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 0x7f5a3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.515+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5968+0+0 (secure 0 0 0) 0x7f5a3c0aa050 con 0x7f5a5415a780 2024-09-13T17:14:17.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.516+0000 7f5a58e00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 0x7f5a3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.516+0000 7f5a58e00640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 0x7f5a3006d0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5a44006fd0 tx=0x7f5a44008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.519+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5a3c0dd9c0 con 0x7f5a5415a780 2024-09-13T17:14:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.734+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5a541e4310 con 0x7f5a5415a780 2024-09-13T17:14:17.736 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:14:17.735+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:14:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.743+0000 7f5a51600640 1 -- 172.21.15.28:0/4065228268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v87) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5a3c075160 con 0x7f5a5415a780 2024-09-13T17:14:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:14:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 msgr2=0x7f5a3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 0x7f5a3006d0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f5a44006fd0 tx=0x7f5a44008040 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 msgr2=0x7f5a541e4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c02a220 tx=0x7f5a3c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 shutdown_connections 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5a3006ac00 0x7f5a3006d0c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5a5415ab60 0x7f5a541e4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a5415a780 0x7f5a541e4530 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 --2- 172.21.15.28:0/4065228268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5a5406bb40 0x7f5a540afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.745+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 >> 172.21.15.28:0/4065228268 conn(0x7f5a540b0270 msgr2=0x7f5a54070e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.746+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 shutdown_connections 2024-09-13T17:14:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.746+0000 7f5a5b800640 1 -- 172.21.15.28:0/4065228268 wait complete. 2024-09-13T17:14:17.750 INFO:tasks.ceph.mds.a.smithi028.stderr: -1446> 2024-09-13T17:14:05.429+0000 7f8b64600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:14:17.764 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:14:17.794 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:14:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.923+0000 7fe463200640 1 Processor -- start 2024-09-13T17:14:17.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.923+0000 7fe463200640 1 -- start start 2024-09-13T17:14:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.924+0000 7fe463200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 0x7fe4640ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.924+0000 7fe463200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe464070480 con 0x7fe464156650 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.924+0000 7fe463200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe46415a010 con 0x7fe4640b0a90 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.924+0000 7fe463200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46406fb50 con 0x7fe464159b30 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe464156650 0x7fe46415a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60252/0 (socket says 172.21.15.28:60252) 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461e00640 1 -- 172.21.15.28:0/2613415340 learned_addr learned my addr 172.21.15.28:0/2613415340 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461400640 1 --2- 172.21.15.28:0/2613415340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 0x7fe4640ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352809073 0 0) 0x7fe464070480 con 0x7fe464156650 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe448003520 con 0x7fe464156650 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36918619 0 0) 0x7fe46415a010 con 0x7fe4640b0a90 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461400640 1 -- 172.21.15.28:0/2613415340 >> v1:172.21.15.189:6789/0 conn(0x7fe4640b0a90 legacy=0x7fe464070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461400640 1 -- 172.21.15.28:0/2613415340 >> v1:172.21.15.28:6789/0 conn(0x7fe464156650 legacy=0x7fe46415a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.925+0000 7fe461400640 1 -- 172.21.15.28:0/2613415340 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46406fe30 con 0x7fe464159b30 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.926+0000 7fe461400640 1 --2- 172.21.15.28:0/2613415340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 0x7fe4640ccb50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097d0 tx=0x7fe45002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af35f927c41c3e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450032440 con 0x7fe464159b30 2024-09-13T17:14:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4500325a0 con 0x7fe464159b30 2024-09-13T17:14:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe460a00640 1 -- 172.21.15.28:0/2613415340 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe450032910 con 0x7fe464159b30 2024-09-13T17:14:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 -- 172.21.15.28:0/2613415340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 msgr2=0x7fe4640ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 --2- 172.21.15.28:0/2613415340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 0x7fe4640ccb50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097d0 tx=0x7fe45002a210 comp rx=0 tx=0).stop 2024-09-13T17:14:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 -- 172.21.15.28:0/2613415340 shutdown_connections 2024-09-13T17:14:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 --2- 172.21.15.28:0/2613415340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464159b30 0x7fe4640ccb50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 -- 172.21.15.28:0/2613415340 >> 172.21.15.28:0/2613415340 conn(0x7fe464069e80 msgr2=0x7fe46406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 -- 172.21.15.28:0/2613415340 shutdown_connections 2024-09-13T17:14:17.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.927+0000 7fe463200640 1 -- 172.21.15.28:0/2613415340 wait complete. 2024-09-13T17:14:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.929+0000 7fe463200640 1 Processor -- start 2024-09-13T17:14:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.929+0000 7fe463200640 1 -- start start 2024-09-13T17:14:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4640b0a90 0x7fe46414a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464156650 0x7fe46406b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46414b5b0 con 0x7fe464159b30 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4641327b0 con 0x7fe4640b0a90 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe463200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe464131ba0 con 0x7fe464156650 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe462800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe462800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36792/0 (socket says 172.21.15.28:36792) 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.930+0000 7fe462800640 1 -- 172.21.15.28:0/2519028998 learned_addr learned my addr 172.21.15.28:0/2519028998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe461e00640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4640b0a90 0x7fe46414a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe461400640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464156650 0x7fe46406b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 -- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464156650 msgr2=0x7fe46406b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464156650 0x7fe46406b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 -- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4640b0a90 msgr2=0x7fe46414a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4640b0a90 0x7fe46414a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 -- 172.21.15.28:0/2519028998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe450009400 con 0x7fe464159b30 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe462800640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00c790 tx=0x7fe45c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c00cea0 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45c003da0 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45c0052b0 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.931+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe464131e80 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.932+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe464132440 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.932+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe46412f800 con 0x7fe464159b30 2024-09-13T17:14:17.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.933+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe45c005410 con 0x7fe464159b30 2024-09-13T17:14:17.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.935+0000 7fe45a000640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 0x7fe44406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.935+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6054+0+0 (secure 0 0 0) 0x7fe45c0892c0 con 0x7fe464159b30 2024-09-13T17:14:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.935+0000 7fe461e00640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 0x7fe44406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.936+0000 7fe461e00640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 0x7fe44406d0c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe454007fd0 tx=0x7fe454009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:17.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:17.939+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe46412f800 con 0x7fe464159b30 2024-09-13T17:14:18.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.155+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe46412f800 con 0x7fe464159b30 2024-09-13T17:14:18.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.745+0000 7fe45a000640 1 -- 172.21.15.28:0/2519028998 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v88) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe46412f800 con 0x7fe464159b30 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 msgr2=0x7fe44406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 0x7fe44406d0c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fe454007fd0 tx=0x7fe454009040 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 msgr2=0x7fe46406baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fe45c00c790 tx=0x7fe45c00cc50 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 shutdown_connections 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe44406ac00 0x7fe44406d0c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe464159b30 0x7fe46406baf0 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe464156650 0x7fe46406b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 --2- 172.21.15.28:0/2519028998 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4640b0a90 0x7fe46414a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.748+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 >> 172.21.15.28:0/2519028998 conn(0x7fe464069e80 msgr2=0x7fe464155630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.749+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 shutdown_connections 2024-09-13T17:14:18.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.749+0000 7fe463200640 1 -- 172.21.15.28:0/2519028998 wait complete. 2024-09-13T17:14:18.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:14:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.970+0000 7f0bd9600640 1 Processor -- start 2024-09-13T17:14:18.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.970+0000 7f0bd9600640 1 -- start start 2024-09-13T17:14:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.971+0000 7f0bd9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0bd40ceba0 0x7f0bd40b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.971+0000 7f0bd9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bd40b1ab0 0x7f0bd40cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.971+0000 7f0bd9600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bd40b1f20 con 0x7f0bd415a7a0 2024-09-13T17:14:18.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.971+0000 7f0bd9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd406c950 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.971+0000 7f0bd9600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bd40b2150 con 0x7f0bd40ceba0 2024-09-13T17:14:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bd40b1ab0 0x7f0bd40cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0bd40b1ab0 0x7f0bd40cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36794/0 (socket says 172.21.15.28:36794) 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 -- 172.21.15.28:0/1349703827 learned_addr learned my addr 172.21.15.28:0/1349703827 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bca000640 1 --2- 172.21.15.28:0/1349703827 >> v2:172.21.15.189:3300/0 conn(0x7f0bd40ceba0 0x7f0bd40b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 -- 172.21.15.28:0/1349703827 >> v2:172.21.15.189:3300/0 conn(0x7f0bd40ceba0 msgr2=0x7f0bd40b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 --2- 172.21.15.28:0/1349703827 >> v2:172.21.15.189:3300/0 conn(0x7f0bd40ceba0 0x7f0bd40b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 -- 172.21.15.28:0/1349703827 >> v1:172.21.15.189:6789/0 conn(0x7f0bd415a7a0 legacy=0x7f0bd40ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.972+0000 7f0bd3400640 1 -- 172.21.15.28:0/1349703827 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40b2430 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.973+0000 7f0bd3400640 1 --2- 172.21.15.28:0/1349703827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd40cd160 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40cad50 tx=0x7f0bcc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cd761e48d71ad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd2000640 1 -- 172.21.15.28:0/1349703827 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc031600 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd2000640 1 -- 172.21.15.28:0/1349703827 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc02d050 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd2000640 1 -- 172.21.15.28:0/1349703827 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc031980 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 -- 172.21.15.28:0/1349703827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40b1ab0 msgr2=0x7f0bd40cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1349703827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd40cd160 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40cad50 tx=0x7f0bcc02a6a0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 -- 172.21.15.28:0/1349703827 shutdown_connections 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1349703827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd40cd160 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1349703827 >> v2:172.21.15.189:3300/0 conn(0x7f0bd40ceba0 0x7f0bd40b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 -- 172.21.15.28:0/1349703827 >> 172.21.15.28:0/1349703827 conn(0x7f0bd4069e80 msgr2=0x7f0bd406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 -- 172.21.15.28:0/1349703827 shutdown_connections 2024-09-13T17:14:18.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.974+0000 7f0bd9600640 1 -- 172.21.15.28:0/1349703827 wait complete. 2024-09-13T17:14:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.976+0000 7f0bd9600640 1 Processor -- start 2024-09-13T17:14:18.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.976+0000 7f0bd9600640 1 -- start start 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd41e3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 0x7f0bd4153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41f6b40 con 0x7f0bd40ceba0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41f5c00 con 0x7f0bd40b1ab0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bd41f5ec0 con 0x7f0bd415a7a0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36806/0 (socket says 172.21.15.28:36806) 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bca000640 1 -- 172.21.15.28:0/1099836255 learned_addr learned my addr 172.21.15.28:0/1099836255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd3400640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 0x7f0bd4153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.977+0000 7f0bd2a00640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd41e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 -- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 msgr2=0x7f0bd4153d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 0x7f0bd4153d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 -- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd40b1ab0 msgr2=0x7f0bd41e3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd41e3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 -- 172.21.15.28:0/1099836255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc009b30 con 0x7f0bd40ceba0 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bd3400640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 0x7f0bd4153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:18.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bca000640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00e8b0 tx=0x7f0bbc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00c940 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00caa0 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00ce30 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.978+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bd41f61a0 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.979+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bd41f6760 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.979+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bd40b1570 con 0x7f0bd40ceba0 2024-09-13T17:14:18.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.980+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0bbc013470 con 0x7f0bd40ceba0 2024-09-13T17:14:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.981+0000 7f0bcb400640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 0x7f0bb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.981+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6054+0+0 (secure 0 0 0) 0x7f0bbc0896d0 con 0x7f0bd40ceba0 2024-09-13T17:14:18.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.982+0000 7f0bd2a00640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 0x7f0bb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:18.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.982+0000 7f0bd2a00640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 0x7f0bb006d0c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4000f60 tx=0x7f0bc4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:18.986+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0bbc054740 con 0x7f0bd40ceba0 2024-09-13T17:14:19.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:19.166+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0bd406f490 con 0x7f0bd40ceba0 2024-09-13T17:14:20.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.055+0000 7f0bcb400640 1 -- 172.21.15.28:0/1099836255 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0bbc057d60 con 0x7f0bd40ceba0 2024-09-13T17:14:20.058 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.060+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 msgr2=0x7f0bb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.060+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 0x7f0bb006d0c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0bc4000f60 tx=0x7f0bc4009040 comp rx=0 tx=0).stop 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.060+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 msgr2=0x7f0bd40b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.060+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00e8b0 tx=0x7f0bbc00ed70 comp rx=0 tx=0).stop 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 shutdown_connections 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0bb006ac00 0x7f0bb006d0c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0bd415a7a0 0x7f0bd4153d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0bd40ceba0 0x7f0bd40b04c0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 --2- 172.21.15.28:0/1099836255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0bd40b1ab0 0x7f0bd41e3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 >> 172.21.15.28:0/1099836255 conn(0x7f0bd4069e80 msgr2=0x7f0bd40cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 shutdown_connections 2024-09-13T17:14:20.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.061+0000 7f0bd9600640 1 -- 172.21.15.28:0/1099836255 wait complete. 2024-09-13T17:14:20.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:14:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.245+0000 7f66bda00640 1 Processor -- start 2024-09-13T17:14:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.245+0000 7f66bda00640 1 -- start start 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66bda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 0x7f66b8070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66bda00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f66b80b2800 0x7f66b815a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66bda00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b815a4c0 con 0x7f66b8159970 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66bda00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b8071580 con 0x7f66b80b2800 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66bda00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8159e70 con 0x7f66b8159590 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66afe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f66b8159970 0x7f66b80b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60268/0 (socket says 172.21.15.28:60268) 2024-09-13T17:14:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.246+0000 7f66afe00640 1 -- 172.21.15.28:0/2983394573 learned_addr learned my addr 172.21.15.28:0/2983394573 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:20.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b6a00640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 0x7f66b8070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b7400640 1 --2- 172.21.15.28:0/2983394573 >> v2:172.21.15.189:3300/0 conn(0x7f66b80b2800 0x7f66b815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585355121 0 0) 0x7f66b815a4c0 con 0x7f66b8159970 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f669c003520 con 0x7f66b8159970 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 598905117 0 0) 0x7f669c003520 con 0x7f66b8159970 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b7400640 1 -- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 msgr2=0x7f66b8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b7400640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 0x7f66b8070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b7400640 1 -- 172.21.15.28:0/2983394573 >> v1:172.21.15.28:6789/0 conn(0x7f66b8159970 legacy=0x7f66b80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b7400640 1 -- 172.21.15.28:0/2983394573 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b815a150 con 0x7f66b80b2800 2024-09-13T17:14:20.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.247+0000 7f66b6a00640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 0x7f66b8070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:14:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66b7400640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b80b2800 0x7f66b815a950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009d90 tx=0x7f66b002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=343f634b0a22cd92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a0001670 con 0x7f66b80b2800 2024-09-13T17:14:20.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b002d050 con 0x7f66b80b2800 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66b6000640 1 -- 172.21.15.28:0/2983394573 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b0007740 con 0x7f66b80b2800 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66bda00640 1 -- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b80b2800 msgr2=0x7f66b815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66bda00640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b80b2800 0x7f66b815a950 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f66b0009d90 tx=0x7f66b002a5f0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66bda00640 1 -- 172.21.15.28:0/2983394573 shutdown_connections 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.249+0000 7f66bda00640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b80b2800 0x7f66b815a950 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.250+0000 7f66bda00640 1 --2- 172.21.15.28:0/2983394573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159590 0x7f66b8070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.250+0000 7f66bda00640 1 -- 172.21.15.28:0/2983394573 >> 172.21.15.28:0/2983394573 conn(0x7f66b8069e80 msgr2=0x7f66b806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.250+0000 7f66bda00640 1 -- 172.21.15.28:0/2983394573 shutdown_connections 2024-09-13T17:14:20.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.250+0000 7f66bda00640 1 -- 172.21.15.28:0/2983394573 wait complete. 2024-09-13T17:14:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.251+0000 7f66bda00640 1 Processor -- start 2024-09-13T17:14:20.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.251+0000 7f66bda00640 1 -- start start 2024-09-13T17:14:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 0x7f66b80cef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 0x7f66b80c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b81483f0 con 0x7f66b80b2800 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b81477e0 con 0x7f66b8159590 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66bda00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b8147ac0 con 0x7f66b8159970 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.252+0000 7f66b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36808/0 (socket says 172.21.15.28:36808) 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 -- 172.21.15.28:0/1072270191 learned_addr learned my addr 172.21.15.28:0/1072270191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66afe00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 0x7f66b80cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b7400640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 0x7f66b80c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 -- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 msgr2=0x7f66b80c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 0x7f66b80c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 -- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 msgr2=0x7f66b80cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 0x7f66b80cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 -- 172.21.15.28:0/1072270191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8009400 con 0x7f66b80b2800 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66afe00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 0x7f66b80cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b7400640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 0x7f66b80c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.253+0000 7f66b6a00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f66a800ebe0 tx=0x7f66a800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.254+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a8009760 con 0x7f66b80b2800 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.254+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a800cc30 con 0x7f66b80b2800 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.254+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0009a70 con 0x7f66b80b2800 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.254+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a800eee0 con 0x7f66b80b2800 2024-09-13T17:14:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.254+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b8148080 con 0x7f66b80b2800 2024-09-13T17:14:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.255+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b806a260 con 0x7f66b80b2800 2024-09-13T17:14:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.256+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f66a8013470 con 0x7f66b80b2800 2024-09-13T17:14:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.256+0000 7f66aea00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 0x7f669406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:20.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.257+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5616+0+0 (secure 0 0 0) 0x7f66a8089650 con 0x7f66b80b2800 2024-09-13T17:14:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.257+0000 7f66afe00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 0x7f669406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.258+0000 7f66afe00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 0x7f669406d0f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ca730 tx=0x7f66a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:20.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.261+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f66a8054810 con 0x7f66b80b2800 2024-09-13T17:14:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:20.449+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66b815a950 con 0x7f66b80b2800 2024-09-13T17:14:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.063+0000 7f66aea00640 1 -- 172.21.15.28:0/1072270191 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f66a8057e30 con 0x7f66b80b2800 2024-09-13T17:14:21.064 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:14:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.066+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 msgr2=0x7f669406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.066+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 0x7f669406d0f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ca730 tx=0x7f66a0009040 comp rx=0 tx=0).stop 2024-09-13T17:14:21.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.066+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 msgr2=0x7f66b80cc5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.066+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f66a800ebe0 tx=0x7f66a800c520 comp rx=0 tx=0).stop 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 shutdown_connections 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f669406ac30 0x7f669406d0f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66b8159970 0x7f66b80c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f66b8159590 0x7f66b80cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 --2- 172.21.15.28:0/1072270191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66b80b2800 0x7f66b80cc5c0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 >> 172.21.15.28:0/1072270191 conn(0x7f66b8069e80 msgr2=0x7f66b80ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:21.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 shutdown_connections 2024-09-13T17:14:21.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.067+0000 7f66bda00640 1 -- 172.21.15.28:0/1072270191 wait complete. 2024-09-13T17:14:21.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_bits 2024-09-13T17:14:21.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.247+0000 7f3abf400640 1 Processor -- start 2024-09-13T17:14:21.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.247+0000 7f3abf400640 1 -- start start 2024-09-13T17:14:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.248+0000 7f3abf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3ab80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.248+0000 7f3abf400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab8070470 con 0x7f3ab815c9a0 2024-09-13T17:14:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.248+0000 7f3abf400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab80ace30 con 0x7f3ab80ad940 2024-09-13T17:14:21.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.248+0000 7f3abf400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80ad110 con 0x7f3ab80b2400 2024-09-13T17:14:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3abca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3ab815c9a0 0x7f3ab80b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60280/0 (socket says 172.21.15.28:60280) 2024-09-13T17:14:21.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3abca00640 1 -- 172.21.15.28:0/4049320053 learned_addr learned my addr 172.21.15.28:0/4049320053 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3ab80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335582587 0 0) 0x7f3ab8070470 con 0x7f3ab815c9a0 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa0003520 con 0x7f3ab815c9a0 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345380620 0 0) 0x7f3ab80ace30 con 0x7f3ab80ad940 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab8070470 con 0x7f3ab80ad940 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3860211316 0 0) 0x7f3aa0003520 con 0x7f3ab815c9a0 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 msgr2=0x7f3ab80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:21.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3ab80ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 >> v1:172.21.15.189:6789/0 conn(0x7f3ab80ad940 legacy=0x7f3ab806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab806b170 con 0x7f3ab815c9a0 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3aac001670 con 0x7f3ab815c9a0 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 >> v1:172.21.15.28:6789/0 conn(0x7f3ab815c9a0 legacy=0x7f3ab80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003b30 0x7f3aa0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.249+0000 7f3ab7400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa00047c0 0x7f3aa0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3aa0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab80ad110 con 0x7f3aa00047c0 2024-09-13T17:14:21.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa000ba30 con 0x7f3aa0003b30 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa000bbf0 con 0x7f3ab80b2400 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa00047c0 0x7f3aa0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3abca00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003b30 0x7f3aa0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3abd400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3aa0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 -- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 msgr2=0x7f3aa0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3aa0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 -- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003b30 msgr2=0x7f3aa0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:21.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3aa0003b30 0x7f3aa0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3ab7e00640 1 -- 172.21.15.28:0/4049320053 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa0003520 con 0x7f3aa00047c0 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.250+0000 7f3abd400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ab80b2400 0x7f3aa0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.251+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa00047c0 0x7f3aa0004c40 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009a90 tx=0x7f3aa00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ee61dc52fe84a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.251+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4031400 con 0x7f3aa00047c0 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.251+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa402d050 con 0x7f3aa00047c0 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.251+0000 7f3ab7400640 1 -- 172.21.15.28:0/4049320053 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4031780 con 0x7f3aa00047c0 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 -- 172.21.15.28:0/4049320053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa00047c0 msgr2=0x7f3aa0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 --2- 172.21.15.28:0/4049320053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3aa00047c0 0x7f3aa0004c40 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009a90 tx=0x7f3aa00057f0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abd400640 1 -- 172.21.15.28:0/4049320053 reap_dead start 2024-09-13T17:14:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 -- 172.21.15.28:0/4049320053 shutdown_connections 2024-09-13T17:14:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 -- 172.21.15.28:0/4049320053 >> 172.21.15.28:0/4049320053 conn(0x7f3ab80b0270 msgr2=0x7f3ab80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 -- 172.21.15.28:0/4049320053 shutdown_connections 2024-09-13T17:14:21.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.252+0000 7f3abf400640 1 -- 172.21.15.28:0/4049320053 wait complete. 2024-09-13T17:14:21.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.254+0000 7f3abf400640 1 Processor -- start 2024-09-13T17:14:21.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.254+0000 7f3abf400640 1 -- start start 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 0x7f3ab8143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8153e20 0x7f3ab8154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab815fe80 con 0x7f3ab815f4c0 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81f27b0 con 0x7f3ab8153e20 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abf400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab81547d0 con 0x7f3aa00047c0 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36818/0 (socket says 172.21.15.28:36818) 2024-09-13T17:14:21.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 -- 172.21.15.28:0/2160831290 learned_addr learned my addr 172.21.15.28:0/2160831290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abca00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 0x7f3ab8143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abd400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8153e20 0x7f3ab8154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 -- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 msgr2=0x7f3ab8143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 0x7f3ab8143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 -- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8153e20 msgr2=0x7f3ab8154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8153e20 0x7f3ab8154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3ab7e00640 1 -- 172.21.15.28:0/2160831290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab8070930 con 0x7f3ab815f4c0 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.255+0000 7f3abca00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 0x7f3ab8143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3ab7e00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402aa80 tx=0x7f3aa4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa403b610 con 0x7f3ab815f4c0 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa403b770 con 0x7f3ab815f4c0 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab8154a30 con 0x7f3ab815f4c0 2024-09-13T17:14:21.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa403bb20 con 0x7f3ab815f4c0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.256+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab8154ff0 con 0x7f3ab815f4c0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.257+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab8155920 con 0x7f3ab815f4c0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.258+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3aa4038050 con 0x7f3ab815f4c0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.259+0000 7f3ab4c00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 0x7f3a9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.259+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3aa40a9520 con 0x7f3ab815f4c0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.259+0000 7f3abca00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 0x7f3a9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.260+0000 7f3abca00640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 0x7f3a9806d0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3aac001eb0 tx=0x7f3aac009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.263+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3aa4074b10 con 0x7f3ab815f4c0 2024-09-13T17:14:21.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.451+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"} v 0) v1 -- 0x7f3ab815cd80 con 0x7f3ab815f4c0 2024-09-13T17:14:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.455+0000 7f3ab4c00640 1 -- 172.21.15.28:0/2160831290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_bits"}]=0 v30) v1 ==== 101+0+0 (secure 0 0 0) 0x7f3aa40360a0 con 0x7f3ab815f4c0 2024-09-13T17:14:21.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 msgr2=0x7f3a9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 0x7f3a9806d0c0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f3aac001eb0 tx=0x7f3aac009040 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 msgr2=0x7f3ab815f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f3aa402aa80 tx=0x7f3aa4004730 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 shutdown_connections 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a9806ac00 0x7f3a9806d0c0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8153e20 0x7f3ab8154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ab815f4c0 0x7f3ab815f940 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 --2- 172.21.15.28:0/2160831290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3aa00047c0 0x7f3ab8143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 >> 172.21.15.28:0/2160831290 conn(0x7f3ab80b0270 msgr2=0x7f3ab806f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 shutdown_connections 2024-09-13T17:14:21.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.457+0000 7f3abf400640 1 -- 172.21.15.28:0/2160831290 wait complete. 2024-09-13T17:14:21.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_split_size 2024-09-13T17:14:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.632+0000 7f958dc00640 1 Processor -- start 2024-09-13T17:14:21.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.632+0000 7f958dc00640 1 -- start start 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f958dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9588070430 0x7f958815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f958dc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958815ab60 con 0x7f9588159770 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f958dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f958806f920 con 0x7f958806f3f0 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f958dc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f958815b290 con 0x7f9588070430 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f9586000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9588070430 0x7f958815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f9586a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9588159770 0x7f958806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60288/0 (socket says 172.21.15.28:60288) 2024-09-13T17:14:21.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.633+0000 7f9586a00640 1 -- 172.21.15.28:0/4103184817 learned_addr learned my addr 172.21.15.28:0/4103184817 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:21.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.634+0000 7f9586000640 1 -- 172.21.15.28:0/4103184817 >> v1:172.21.15.189:6789/0 conn(0x7f958806f3f0 legacy=0x7f958815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:21.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.634+0000 7f9586000640 1 -- 172.21.15.28:0/4103184817 >> v1:172.21.15.28:6789/0 conn(0x7f9588159770 legacy=0x7f958806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.634+0000 7f9586000640 1 -- 172.21.15.28:0/4103184817 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f958815b570 con 0x7f9588070430 2024-09-13T17:14:21.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.635+0000 7f9586000640 1 --2- 172.21.15.28:0/4103184817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588070430 0x7f958815bd70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f95740097a0 tx=0x7f957402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e959fe3081213b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.635+0000 7f9585600640 1 -- 172.21.15.28:0/4103184817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f957402ae50 con 0x7f9588070430 2024-09-13T17:14:21.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.635+0000 7f9585600640 1 -- 172.21.15.28:0/4103184817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9574032440 con 0x7f9588070430 2024-09-13T17:14:21.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.635+0000 7f9585600640 1 -- 172.21.15.28:0/4103184817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95740327d0 con 0x7f9588070430 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 -- 172.21.15.28:0/4103184817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588070430 msgr2=0x7f958815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 --2- 172.21.15.28:0/4103184817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588070430 0x7f958815bd70 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f95740097a0 tx=0x7f957402a140 comp rx=0 tx=0).stop 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 -- 172.21.15.28:0/4103184817 shutdown_connections 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 --2- 172.21.15.28:0/4103184817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588070430 0x7f958815bd70 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 -- 172.21.15.28:0/4103184817 >> 172.21.15.28:0/4103184817 conn(0x7f95880b0270 msgr2=0x7f95880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 -- 172.21.15.28:0/4103184817 shutdown_connections 2024-09-13T17:14:21.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.636+0000 7f958dc00640 1 -- 172.21.15.28:0/4103184817 wait complete. 2024-09-13T17:14:21.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.637+0000 7f958dc00640 1 Processor -- start 2024-09-13T17:14:21.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 -- start start 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f958806f3f0 0x7f958814d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9588070430 0x7f95881f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95881f3120 con 0x7f9588159770 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95881f3400 con 0x7f958806f3f0 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.638+0000 7f958dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95881f36e0 con 0x7f9588070430 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36840/0 (socket says 172.21.15.28:36840) 2024-09-13T17:14:21.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 -- 172.21.15.28:0/1401516898 learned_addr learned my addr 172.21.15.28:0/1401516898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9586000640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9588070430 0x7f95881f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9586a00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f958806f3f0 0x7f958814d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 -- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9588070430 msgr2=0x7f95881f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9588070430 0x7f95881f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 -- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f958806f3f0 msgr2=0x7f958814d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f958806f3f0 0x7f958814d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 -- 172.21.15.28:0/1401516898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9574009400 con 0x7f9588159770 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.639+0000 7f9587400640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f958000c780 tx=0x7f958000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.640+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9580004b60 con 0x7f9588159770 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.640+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f958000ce90 con 0x7f9588159770 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.640+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f958000a850 con 0x7f9588159770 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.640+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95880b1610 con 0x7f9588159770 2024-09-13T17:14:21.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.640+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958814d580 con 0x7f9588159770 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.641+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95880b1b30 con 0x7f9588159770 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.642+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f958000a9b0 con 0x7f9588159770 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.642+0000 7f957ea00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 0x7f956806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.643+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f9580089370 con 0x7f9588159770 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.643+0000 7f9586a00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 0x7f956806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:21.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.644+0000 7f9586a00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 0x7f956806d0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9578005880 tx=0x7f9578009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:21.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.647+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9580054700 con 0x7f9588159770 2024-09-13T17:14:21.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.826+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"} v 0) v1 -- 0x7f958806fd20 con 0x7f9588159770 2024-09-13T17:14:21.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.830+0000 7f957ea00640 1 -- 172.21.15.28:0/1401516898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_split_size"}]=0 v31) v1 ==== 101+0+0 (secure 0 0 0) 0x7f9580057d20 con 0x7f9588159770 2024-09-13T17:14:21.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 msgr2=0x7f956806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 0x7f956806d0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f9578005880 tx=0x7f9578009040 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 msgr2=0x7f958814d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f958000c780 tx=0x7f958000cc40 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 shutdown_connections 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f956806ac00 0x7f956806d0c0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9588159770 0x7f958814d8a0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9588070430 0x7f95881f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 --2- 172.21.15.28:0/1401516898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f958806f3f0 0x7f958814d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.832+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 >> 172.21.15.28:0/1401516898 conn(0x7f95880b0270 msgr2=0x7f95881586b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.833+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 shutdown_connections 2024-09-13T17:14:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:21.833+0000 7f958dc00640 1 -- 172.21.15.28:0/1401516898 wait complete. 2024-09-13T17:14:21.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_bal_merge_size 2024-09-13T17:14:22.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.005+0000 7fc735c00640 1 Processor -- start 2024-09-13T17:14:22.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.006+0000 7fc735c00640 1 -- start start 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc735c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc73015a780 0x7fc730070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc735c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc7300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc735c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73006c8b0 con 0x7fc73006bb40 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc735c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73006cb90 con 0x7fc73015a780 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc735c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300cc980 con 0x7fc73015ab60 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc72f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc73006bb40 0x7fc73006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60300/0 (socket says 172.21.15.28:60300) 2024-09-13T17:14:22.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc72f400640 1 -- 172.21.15.28:0/3860456925 learned_addr learned my addr 172.21.15.28:0/3860456925 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666636255 0 0) 0x7fc73006c8b0 con 0x7fc73006bb40 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc72ea00640 1 --2- 172.21.15.28:0/3860456925 >> v2:172.21.15.189:3300/0 conn(0x7fc73015a780 0x7fc730070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.007+0000 7fc726000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc7300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc714003520 con 0x7fc73006bb40 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2608226926 0 0) 0x7fc714003520 con 0x7fc73006bb40 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 msgr2=0x7fc7300712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc7300712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 >> v2:172.21.15.189:3300/0 conn(0x7fc73015a780 msgr2=0x7fc730070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 --2- 172.21.15.28:0/3860456925 >> v2:172.21.15.189:3300/0 conn(0x7fc73015a780 0x7fc730070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300ccc60 con 0x7fc73006bb40 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc7280019f0 con 0x7fc73006bb40 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 >> v1:172.21.15.28:6789/0 conn(0x7fc73006bb40 legacy=0x7fc73006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc714003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7140047c0 0x7fc714004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc714005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73006cb90 con 0x7fc714003970 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300cc980 con 0x7fc7140047c0 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71400b980 con 0x7fc73015ab60 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72ea00640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc714003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc72f400640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc714005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.008+0000 7fc726000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7140047c0 0x7fc714004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc72ea00640 1 -- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 msgr2=0x7fc714005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc72ea00640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc73015ab60 0x7fc714005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc72ea00640 1 -- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7140047c0 msgr2=0x7fc714004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc72ea00640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7140047c0 0x7fc714004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc72ea00640 1 -- 172.21.15.28:0/3860456925 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714003520 con 0x7fc714003970 2024-09-13T17:14:22.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.009+0000 7fc726000640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7140047c0 0x7fc714004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc72ea00640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc714003d90 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ac0 tx=0x7fc7140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ce62e9e84a1694 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720031400 con 0x7fc714003970 2024-09-13T17:14:22.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72002d050 con 0x7fc714003970 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc72e000640 1 -- 172.21.15.28:0/3860456925 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc720031780 con 0x7fc714003970 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc735c00640 1 -- 172.21.15.28:0/3860456925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 msgr2=0x7fc714003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc735c00640 1 --2- 172.21.15.28:0/3860456925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc714003d90 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ac0 tx=0x7fc7140057f0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.010+0000 7fc72f400640 1 -- 172.21.15.28:0/3860456925 reap_dead start 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.011+0000 7fc735c00640 1 -- 172.21.15.28:0/3860456925 shutdown_connections 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.011+0000 7fc735c00640 1 -- 172.21.15.28:0/3860456925 >> 172.21.15.28:0/3860456925 conn(0x7fc7300b0270 msgr2=0x7fc7300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.011+0000 7fc735c00640 1 -- 172.21.15.28:0/3860456925 shutdown_connections 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.011+0000 7fc735c00640 1 -- 172.21.15.28:0/3860456925 wait complete. 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.012+0000 7fc735c00640 1 Processor -- start 2024-09-13T17:14:22.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.012+0000 7fc735c00640 1 -- start start 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7300ae270 0x7fc7300ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7300c4660 0x7fc7300c4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300aec30 con 0x7fc714003970 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300c6b30 con 0x7fc7300c4660 2024-09-13T17:14:22.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc735c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7300c6520 con 0x7fc7300ae270 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc72ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc72ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36870/0 (socket says 172.21.15.28:36870) 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.013+0000 7fc72ea00640 1 -- 172.21.15.28:0/1170350824 learned_addr learned my addr 172.21.15.28:0/1170350824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc726000640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7300ae270 0x7fc7300ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72f400640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7300c4660 0x7fc7300c4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 -- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7300ae270 msgr2=0x7fc7300ae6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7300ae270 0x7fc7300ae6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 -- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7300c4660 msgr2=0x7fc7300c4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7300c4660 0x7fc7300c4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 -- 172.21.15.28:0/1170350824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7300c53d0 con 0x7fc714003970 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc72ea00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ac0 tx=0x7fc720004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72003b610 con 0x7fc714003970 2024-09-13T17:14:22.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.014+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72003b770 con 0x7fc714003970 2024-09-13T17:14:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.015+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72003bb00 con 0x7fc714003970 2024-09-13T17:14:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.015+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7300c67c0 con 0x7fc714003970 2024-09-13T17:14:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.015+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7300c96b0 con 0x7fc714003970 2024-09-13T17:14:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.016+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc730070d80 con 0x7fc714003970 2024-09-13T17:14:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.016+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc720038050 con 0x7fc714003970 2024-09-13T17:14:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.017+0000 7fc727400640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 0x7fc70c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.017+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7fc7200a9590 con 0x7fc714003970 2024-09-13T17:14:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.018+0000 7fc726000640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 0x7fc70c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.018+0000 7fc726000640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 0x7fc70c06d0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc7300c61c0 tx=0x7fc718008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.021+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc720074af0 con 0x7fc714003970 2024-09-13T17:14:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.201+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"} v 0) v1 -- 0x7fc7300b3610 con 0x7fc714003970 2024-09-13T17:14:22.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.205+0000 7fc727400640 1 -- 172.21.15.28:0/1170350824 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_bal_merge_size"}]=0 v32) v1 ==== 101+0+0 (secure 0 0 0) 0x7fc720036020 con 0x7fc714003970 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 msgr2=0x7fc70c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 0x7fc70c06d0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fc7300c61c0 tx=0x7fc718008040 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 msgr2=0x7fc7300c9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7fc720009ac0 tx=0x7fc720004730 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 shutdown_connections 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70c06ac00 0x7fc70c06d0c0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7300c4660 0x7fc7300c4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7300ae270 0x7fc7300ae6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 --2- 172.21.15.28:0/1170350824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc714003970 0x7fc7300c9170 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 >> 172.21.15.28:0/1170350824 conn(0x7fc7300b0270 msgr2=0x7fc730158510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 shutdown_connections 2024-09-13T17:14:22.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.207+0000 7fc735c00640 1 -- 172.21.15.28:0/1170350824 wait complete. 2024-09-13T17:14:22.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:14:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.381+0000 7f6563400640 1 Processor -- start 2024-09-13T17:14:22.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.381+0000 7f6563400640 1 -- start start 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6563400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f655c159770 0x7f655c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6563400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 0x7f655c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6563400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655c06f920 con 0x7f655c06f3f0 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6563400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655c15b290 con 0x7f655c159770 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6563400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c15b570 con 0x7f655c15cd80 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6560a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f655c159770 0x7f655c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.382+0000 7f6560a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f655c159770 0x7f655c06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36876/0 (socket says 172.21.15.28:36876) 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6560a00640 1 -- 172.21.15.28:0/3613036764 learned_addr learned my addr 172.21.15.28:0/3613036764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:22.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6561400640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 0x7f655c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6560a00640 1 -- 172.21.15.28:0/3613036764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 msgr2=0x7f655c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6560a00640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 0x7f655c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6560a00640 1 -- 172.21.15.28:0/3613036764 >> v1:172.21.15.189:6789/0 conn(0x7f655c06f3f0 legacy=0x7f655c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6560a00640 1 -- 172.21.15.28:0/3613036764 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c15b850 con 0x7f655c159770 2024-09-13T17:14:22.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.383+0000 7f6561400640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 0x7f655c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.384+0000 7f6560a00640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c06fd20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6550009870 tx=0x7f655002a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c81a8525702ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.384+0000 7f655b400640 1 -- 172.21.15.28:0/3613036764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550032440 con 0x7f655c159770 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.384+0000 7f655b400640 1 -- 172.21.15.28:0/3613036764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65500325a0 con 0x7f655c159770 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.384+0000 7f655b400640 1 -- 172.21.15.28:0/3613036764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550032930 con 0x7f655c159770 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 -- 172.21.15.28:0/3613036764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 msgr2=0x7f655c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c06fd20 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f6550009870 tx=0x7f655002a1d0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 -- 172.21.15.28:0/3613036764 shutdown_connections 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c15cd80 0x7f655c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 --2- 172.21.15.28:0/3613036764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c06fd20 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 -- 172.21.15.28:0/3613036764 >> 172.21.15.28:0/3613036764 conn(0x7f655c0b0270 msgr2=0x7f655c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 -- 172.21.15.28:0/3613036764 shutdown_connections 2024-09-13T17:14:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.385+0000 7f6563400640 1 -- 172.21.15.28:0/3613036764 wait complete. 2024-09-13T17:14:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.387+0000 7f6563400640 1 Processor -- start 2024-09-13T17:14:22.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.387+0000 7f6563400640 1 -- start start 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.387+0000 7f6563400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c06f3f0 0x7f655c0aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.387+0000 7f6563400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6563400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c15cd80 0x7f655c156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6563400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c0a9d50 con 0x7f655c159770 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6563400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c0aa030 con 0x7f655c15cd80 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6563400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655c0aa310 con 0x7f655c06f3f0 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36888/0 (socket says 172.21.15.28:36888) 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 -- 172.21.15.28:0/1928234728 learned_addr learned my addr 172.21.15.28:0/1928234728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6560a00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c06f3f0 0x7f655c0aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f6561400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c15cd80 0x7f655c156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 -- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c06f3f0 msgr2=0x7f655c0aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c06f3f0 0x7f655c0aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 -- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c15cd80 msgr2=0x7f655c156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c15cd80 0x7f655c156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.388+0000 7f655be00640 1 -- 172.21.15.28:0/1928234728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65500094d0 con 0x7f655c159770 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f655be00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f654800c400 tx=0x7f654800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65480052d0 con 0x7f655c159770 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6548005430 con 0x7f655c159770 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6548003830 con 0x7f655c159770 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655c0aa5f0 con 0x7f655c159770 2024-09-13T17:14:22.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.389+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655c0ab330 con 0x7f655c159770 2024-09-13T17:14:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.390+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655c06fd20 con 0x7f655c159770 2024-09-13T17:14:22.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.391+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6548003990 con 0x7f655c159770 2024-09-13T17:14:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.392+0000 7f6558c00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 0x7f653c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.392+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f6548088730 con 0x7f655c159770 2024-09-13T17:14:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.392+0000 7f6560a00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 0x7f653c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.393+0000 7f6560a00640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 0x7f653c06d0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6550034000 tx=0x7f655002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.396+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6548053ac0 con 0x7f655c159770 2024-09-13T17:14:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.576+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f655c0cf130 con 0x7f655c159770 2024-09-13T17:14:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.580+0000 7f6558c00640 1 -- 172.21.15.28:0/1928234728 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f65480570e0 con 0x7f655c159770 2024-09-13T17:14:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 msgr2=0x7f653c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 0x7f653c06d0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f6550034000 tx=0x7f655002e000 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 msgr2=0x7f655c14bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f654800c400 tx=0x7f654800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 shutdown_connections 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f653c06ac00 0x7f653c06d0c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f655c15cd80 0x7f655c156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f655c159770 0x7f655c14bf30 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 --2- 172.21.15.28:0/1928234728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f655c06f3f0 0x7f655c0aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 >> 172.21.15.28:0/1928234728 conn(0x7f655c0b0270 msgr2=0x7f655c0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 shutdown_connections 2024-09-13T17:14:22.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.582+0000 7f6563400640 1 -- 172.21.15.28:0/1928234728 wait complete. 2024-09-13T17:14:22.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags' 2024-09-13T17:14:22.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.755+0000 7f839aa00640 1 Processor -- start 2024-09-13T17:14:22.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.755+0000 7f839aa00640 1 -- start start 2024-09-13T17:14:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.756+0000 7f839aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8394154a90 0x7f839415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.756+0000 7f839aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83940af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.756+0000 7f839aa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839415bb10 con 0x7f839406b360 2024-09-13T17:14:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.756+0000 7f839aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839406b900 con 0x7f8394154a90 2024-09-13T17:14:22.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.756+0000 7f839aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839406bbe0 con 0x7f83940b0b50 2024-09-13T17:14:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8394154a90 0x7f839415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f839a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83940af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8394154a90 0x7f839415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46604/0 (socket says 172.21.15.28:46604) 2024-09-13T17:14:22.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f839a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83940af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:42342/0 (socket says 172.21.15.28:42342) 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 -- 172.21.15.28:0/3597545677 learned_addr learned my addr 172.21.15.28:0/3597545677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 -- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 msgr2=0x7f83940af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 --2- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83940af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8399600640 1 -- 172.21.15.28:0/3597545677 >> v1:172.21.15.189:6789/0 conn(0x7f839406b360 legacy=0x7f83940b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.757+0000 7f8393e00640 1 -- 172.21.15.28:0/3597545677 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576766419 0 0) 0x7f839415bb10 con 0x7f839406b360 2024-09-13T17:14:22.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.758+0000 7f8399600640 1 -- 172.21.15.28:0/3597545677 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839406bec0 con 0x7f8394154a90 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f8399600640 1 --2- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839415b5d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8388009870 tx=0x7f838802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1d8e72326fd20db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f8393e00640 1 -- 172.21.15.28:0/3597545677 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388032440 con 0x7f8394154a90 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f8393e00640 1 -- 172.21.15.28:0/3597545677 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83880325a0 con 0x7f8394154a90 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f8393e00640 1 -- 172.21.15.28:0/3597545677 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388032930 con 0x7f8394154a90 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f839aa00640 1 -- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 msgr2=0x7f839415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.759+0000 7f839aa00640 1 --2- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839415b5d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8388009870 tx=0x7f838802a270 comp rx=0 tx=0).stop 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 -- 172.21.15.28:0/3597545677 shutdown_connections 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 --2- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83940af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 --2- 172.21.15.28:0/3597545677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839415b5d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 -- 172.21.15.28:0/3597545677 >> 172.21.15.28:0/3597545677 conn(0x7f8394072e40 msgr2=0x7f8394073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:22.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 -- 172.21.15.28:0/3597545677 shutdown_connections 2024-09-13T17:14:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.760+0000 7f839aa00640 1 -- 172.21.15.28:0/3597545677 wait complete. 2024-09-13T17:14:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.761+0000 7f839aa00640 1 Processor -- start 2024-09-13T17:14:22.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.761+0000 7f839aa00640 1 -- start start 2024-09-13T17:14:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83941e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83941f3d20 con 0x7f839406b360 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83941e3b10 con 0x7f8394154a90 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.762+0000 7f839aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83941f33f0 con 0x7f83940b0b50 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36902/0 (socket says 172.21.15.28:36902) 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 -- 172.21.15.28:0/1175564381 learned_addr learned my addr 172.21.15.28:0/1175564381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8398c00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83941e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f839a000640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 -- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 msgr2=0x7f83941e35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83941e35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 -- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 msgr2=0x7f839413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839413c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 -- 172.21.15.28:0/1175564381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83880094d0 con 0x7f839406b360 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.763+0000 7f8399600640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8388009870 tx=0x7f83880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.764+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388032440 con 0x7f839406b360 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.764+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8388030980 con 0x7f839406b360 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.764+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8388030d30 con 0x7f839406b360 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.764+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83941f36d0 con 0x7f839406b360 2024-09-13T17:14:22.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.764+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839413cd70 con 0x7f839406b360 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.765+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83940af4b0 con 0x7f839406b360 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.766+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8388039040 con 0x7f839406b360 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.767+0000 7f8391600640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 0x7f837406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.767+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f83880a9a90 con 0x7f839406b360 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.767+0000 7f8398c00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 0x7f837406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:22.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.768+0000 7f8398c00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 0x7f837406d0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f83940af1c0 tx=0x7f8380009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:22.771+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8388074f60 con 0x7f839406b360 2024-09-13T17:14:23.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.016+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags"]} v 0) v1 -- 0x7f83940b0570 con 0x7f839406b360 2024-09-13T17:14:23.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.066+0000 7f8391600640 1 -- 172.21.15.28:0/1175564381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_scrub_split_dirfrags"]}]=0 v336) v1 ==== 151+0+0 (secure 0 0 0) 0x7f8388078580 con 0x7f839406b360 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 msgr2=0x7f837406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 0x7f837406d0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f83940af1c0 tx=0x7f8380009040 comp rx=0 tx=0).stop 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 msgr2=0x7f83941f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f8388009870 tx=0x7f83880046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 shutdown_connections 2024-09-13T17:14:23.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f837406ac30 0x7f837406d0f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8394154a90 0x7f839413c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f83940b0b50 0x7f83941e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 --2- 172.21.15.28:0/1175564381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f839406b360 0x7f83941f2cf0 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 >> 172.21.15.28:0/1175564381 conn(0x7f8394072e40 msgr2=0x7f83940707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 shutdown_connections 2024-09-13T17:14:23.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.068+0000 7f839aa00640 1 -- 172.21.15.28:0/1175564381 wait complete. 2024-09-13T17:14:23.086 INFO:tasks.cephfs_test_runner:That a directory is split during scrub. ... ok 2024-09-13T17:14:23.086 INFO:tasks.cephfs_test_runner:Starting test: test_stray_evaluation_with_scrub (tasks.cephfs.test_scrub_checks.TestScrubChecks) test that scrub can iterate over ~mdsdir and evaluate strays 2024-09-13T17:14:23.086 INFO:tasks.cephfs_test_runner:test_stray_evaluation_with_scrub (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2024-09-13T17:14:23.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:23.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.241+0000 7f9364a00640 1 Processor -- start 2024-09-13T17:14:23.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.242+0000 7f9364a00640 1 -- start start 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f9364a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f93600b0b50 0x7f93600b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f9364a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93600c3850 con 0x7f9360159440 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f9364a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936015a930 con 0x7f9360159820 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f9364a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9360159d20 con 0x7f93600b0b50 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f935f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9360159440 0x7f93600b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60310/0 (socket says 172.21.15.28:60310) 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f935f400640 1 -- 172.21.15.28:0/544488332 learned_addr learned my addr 172.21.15.28:0/544488332 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:23.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.243+0000 7f9357e00640 1 --2- 172.21.15.28:0/544488332 >> v2:172.21.15.189:3300/0 conn(0x7f93600b0b50 0x7f93600b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.244+0000 7f935ea00640 1 -- 172.21.15.28:0/544488332 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2719943560 0 0) 0x7f93600c3850 con 0x7f9360159440 2024-09-13T17:14:23.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.244+0000 7f935ea00640 1 -- 172.21.15.28:0/544488332 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9344003520 con 0x7f9360159440 2024-09-13T17:14:23.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.244+0000 7f9357e00640 1 -- 172.21.15.28:0/544488332 >> v1:172.21.15.189:6789/0 conn(0x7f9360159820 legacy=0x7f936015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.244+0000 7f9357e00640 1 -- 172.21.15.28:0/544488332 >> v1:172.21.15.28:6789/0 conn(0x7f9360159440 legacy=0x7f93600b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.244+0000 7f9357e00640 1 -- 172.21.15.28:0/544488332 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936015a000 con 0x7f93600b0b50 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.245+0000 7f9357e00640 1 --2- 172.21.15.28:0/544488332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93600b0b50 0x7f93600b22e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f93480097a0 tx=0x7f934802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a50da48931266118 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.245+0000 7f935ea00640 1 -- 172.21.15.28:0/544488332 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348032640 con 0x7f93600b0b50 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.245+0000 7f935ea00640 1 -- 172.21.15.28:0/544488332 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93480327a0 con 0x7f93600b0b50 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.245+0000 7f935ea00640 1 -- 172.21.15.28:0/544488332 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348032b30 con 0x7f93600b0b50 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 -- 172.21.15.28:0/544488332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93600b0b50 msgr2=0x7f93600b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 --2- 172.21.15.28:0/544488332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93600b0b50 0x7f93600b22e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f93480097a0 tx=0x7f934802a2c0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 -- 172.21.15.28:0/544488332 shutdown_connections 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 --2- 172.21.15.28:0/544488332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93600b0b50 0x7f93600b22e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 -- 172.21.15.28:0/544488332 >> 172.21.15.28:0/544488332 conn(0x7f9360072e40 msgr2=0x7f9360073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 -- 172.21.15.28:0/544488332 shutdown_connections 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.246+0000 7f9364a00640 1 -- 172.21.15.28:0/544488332 wait complete. 2024-09-13T17:14:23.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.248+0000 7f9364a00640 1 Processor -- start 2024-09-13T17:14:23.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.248+0000 7f9364a00640 1 -- start start 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 0x7f9360154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 0x7f936015c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936013a7e0 con 0x7f9360159440 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936014e4b0 con 0x7f9360159820 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9364a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936014e790 con 0x7f93600b0b50 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2024-09-13T17:14:23.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 -- 172.21.15.28:0/4292098884 learned_addr learned my addr 172.21.15.28:0/4292098884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f935fe00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 0x7f936015c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f935f400640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 0x7f9360154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 -- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 msgr2=0x7f9360154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 0x7f9360154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 -- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 msgr2=0x7f936015c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 0x7f936015c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f9357e00640 1 -- 172.21.15.28:0/4292098884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9348009400 con 0x7f9360159440 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f935fe00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 0x7f936015c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.249+0000 7f935f400640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 0x7f9360154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9357e00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9348035040 tx=0x7f93480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348030910 con 0x7f9360159440 2024-09-13T17:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9348030a90 con 0x7f9360159440 2024-09-13T17:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9348030e40 con 0x7f9360159440 2024-09-13T17:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9360157140 con 0x7f9360159440 2024-09-13T17:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.250+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9360157700 con 0x7f9360159440 2024-09-13T17:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.252+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9360156fa0 con 0x7f9360159440 2024-09-13T17:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.252+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9348039040 con 0x7f9360159440 2024-09-13T17:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.253+0000 7f9357400640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 0x7f933c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.253+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f93480a9d60 con 0x7f9360159440 2024-09-13T17:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.254+0000 7f935f400640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 0x7f933c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.254+0000 7f935f400640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 0x7f933c06d0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93500011e0 tx=0x7f9350009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.257+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9360156fa0 con 0x7f9360159440 2024-09-13T17:14:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.435+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9360156fa0 con 0x7f9360159440 2024-09-13T17:14:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.436+0000 7f9357400640 1 -- 172.21.15.28:0/4292098884 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13250 (secure 0 0 0) 0x7f9360156fa0 con 0x7f9360159440 2024-09-13T17:14:23.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:23.437 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:21.054707+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/2200414779":"2024-09-14T17:14:17.735277+0000","172.21.15.189:6837/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.28:6833/2200414779":"2024-09-14T17:14:17.735277+0000","172.21.15.189:6835/3444244429":"2024-09-14T17:14:00.308410+0000","172.21.15.28:6834/2861751170":"2024-09-14T17:13:54.409423+0000","172.21.15.28:6832/2861751170":"2024-09-14T17:13:54.409423+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 msgr2=0x7f933c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 0x7f933c06d0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f93500011e0 tx=0x7f9350009040 comp rx=0 tx=0).stop 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 msgr2=0x7f936015be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9348035040 tx=0x7f93480046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 shutdown_connections 2024-09-13T17:14:23.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f933c06ac00 0x7f933c06d0c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9360159820 0x7f936015c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9360159440 0x7f936015be10 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 --2- 172.21.15.28:0/4292098884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f93600b0b50 0x7f9360154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 >> 172.21.15.28:0/4292098884 conn(0x7f9360072e40 msgr2=0x7f93600ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 shutdown_connections 2024-09-13T17:14:23.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.438+0000 7f9364a00640 1 -- 172.21.15.28:0/4292098884 wait complete. 2024-09-13T17:14:23.455 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:14:23.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:14:23.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.613+0000 7f33a5600640 1 Processor -- start 2024-09-13T17:14:23.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.613+0000 7f33a5600640 1 -- start start 2024-09-13T17:14:23.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.614+0000 7f33a5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a0159770 0x7f33a006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.614+0000 7f33a5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a006f3f0 0x7f33a0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.614+0000 7f33a5600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a006f920 con 0x7f33a00add00 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.614+0000 7f33a5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a015b290 con 0x7f33a006f3f0 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.614+0000 7f33a5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a015b570 con 0x7f33a0159770 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a006f3f0 0x7f33a0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a006f3f0 0x7f33a0070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36918/0 (socket says 172.21.15.28:36918) 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f33a00add00 0x7f33a0159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60318/0 (socket says 172.21.15.28:60318) 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 -- 172.21.15.28:0/2926679572 learned_addr learned my addr 172.21.15.28:0/2926679572 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:23.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339ea00640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a0159770 0x7f33a006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 -- 172.21.15.28:0/2926679572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a0159770 msgr2=0x7f33a006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a0159770 0x7f33a006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 -- 172.21.15.28:0/2926679572 >> v1:172.21.15.28:6789/0 conn(0x7f33a00add00 legacy=0x7f33a0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339e000640 1 -- 172.21.15.28:0/2926679572 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a015b850 con 0x7f33a006f3f0 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.615+0000 7f339d600640 1 -- 172.21.15.28:0/2926679572 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314832115 0 0) 0x7f33a006f920 con 0x7f33a00add00 2024-09-13T17:14:23.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.616+0000 7f339e000640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a006f3f0 0x7f33a0070260 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f338c009870 tx=0x7f338c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=706fac59257b184d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.616+0000 7f339d600640 1 -- 172.21.15.28:0/2926679572 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c032440 con 0x7f33a006f3f0 2024-09-13T17:14:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f339d600640 1 -- 172.21.15.28:0/2926679572 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c0325a0 con 0x7f33a006f3f0 2024-09-13T17:14:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f339d600640 1 -- 172.21.15.28:0/2926679572 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c032930 con 0x7f33a006f3f0 2024-09-13T17:14:23.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 -- 172.21.15.28:0/2926679572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a006f3f0 msgr2=0x7f33a0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a006f3f0 0x7f33a0070260 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f338c009870 tx=0x7f338c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 -- 172.21.15.28:0/2926679572 shutdown_connections 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a006f3f0 0x7f33a0070260 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 --2- 172.21.15.28:0/2926679572 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a0159770 0x7f33a006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 -- 172.21.15.28:0/2926679572 >> 172.21.15.28:0/2926679572 conn(0x7f33a00b0270 msgr2=0x7f33a00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 -- 172.21.15.28:0/2926679572 shutdown_connections 2024-09-13T17:14:23.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.617+0000 7f33a5600640 1 -- 172.21.15.28:0/2926679572 wait complete. 2024-09-13T17:14:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.619+0000 7f33a5600640 1 Processor -- start 2024-09-13T17:14:23.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.619+0000 7f33a5600640 1 -- start start 2024-09-13T17:14:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a006f3f0 0x7f33a0148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 0x7f33a0155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01547f0 con 0x7f33a0159770 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a01549d0 con 0x7f33a00add00 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f33a5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a0154cb0 con 0x7f33a006f3f0 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f339f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f339f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36930/0 (socket says 172.21.15.28:36930) 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f339f400640 1 -- 172.21.15.28:0/3953075024 learned_addr learned my addr 172.21.15.28:0/3953075024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.620+0000 7f339ea00640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a006f3f0 0x7f33a0148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339e000640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 0x7f33a0155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 -- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a006f3f0 msgr2=0x7f33a0148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a006f3f0 0x7f33a0148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 -- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 msgr2=0x7f33a0155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 0x7f33a0155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 -- 172.21.15.28:0/3953075024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c0094d0 con 0x7f33a0159770 2024-09-13T17:14:23.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339e000640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 0x7f33a0155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f339f400640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f339800c6d0 tx=0x7f339800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398004d60 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339800a640 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339800aa70 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a0154f90 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.621+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a0155550 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.622+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a006cb10 con 0x7f33a0159770 2024-09-13T17:14:23.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.623+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f33980070d0 con 0x7f33a0159770 2024-09-13T17:14:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.624+0000 7f3396a00640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 0x7f338006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.624+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f3398088880 con 0x7f33a0159770 2024-09-13T17:14:23.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.625+0000 7f339ea00640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 0x7f338006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.625+0000 7f339ea00640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 0x7f338006d0c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3390006fd0 tx=0x7f3390008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.628+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3398053c10 con 0x7f33a0159770 2024-09-13T17:14:23.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.804+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33a0061700 con 0x7f33a0159770 2024-09-13T17:14:23.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.805+0000 7f3396a00640 1 -- 172.21.15.28:0/3953075024 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33a0061700 con 0x7f33a0159770 2024-09-13T17:14:23.806 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:14:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 msgr2=0x7f338006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 0x7f338006d0c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3390006fd0 tx=0x7f3390008040 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 msgr2=0x7f33a00cc880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f339800c6d0 tx=0x7f339800cb90 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 shutdown_connections 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338006ac00 0x7f338006d0c0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a0159770 0x7f33a00cc880 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a00add00 0x7f33a0155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 --2- 172.21.15.28:0/3953075024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a006f3f0 0x7f33a0148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 >> 172.21.15.28:0/3953075024 conn(0x7f33a00b0270 msgr2=0x7f33a0156bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 shutdown_connections 2024-09-13T17:14:23.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.807+0000 7f33a5600640 1 -- 172.21.15.28:0/3953075024 wait complete. 2024-09-13T17:14:23.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub' 2024-09-13T17:14:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.978+0000 7f7be2800640 1 Processor -- start 2024-09-13T17:14:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.978+0000 7f7be2800640 1 -- start start 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.979+0000 7f7be2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.979+0000 7f7be2800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 0x7f7bdc06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.979+0000 7f7be2800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bdc0b3970 con 0x7f7bdc06f980 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.979+0000 7f7be2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc159230 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.979+0000 7f7be2800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bdc06fdb0 con 0x7f7bdc0b3160 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38946/0 (socket says 172.21.15.28:38946) 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 -- 172.21.15.28:0/4167949872 learned_addr learned my addr 172.21.15.28:0/4167949872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdaa00640 1 --2- 172.21.15.28:0/4167949872 >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 0x7f7bdc06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 -- 172.21.15.28:0/4167949872 >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 msgr2=0x7f7bdc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 --2- 172.21.15.28:0/4167949872 >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 0x7f7bdc06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 -- 172.21.15.28:0/4167949872 >> v1:172.21.15.189:6789/0 conn(0x7f7bdc06f980 legacy=0x7f7bdc0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdb400640 1 -- 172.21.15.28:0/4167949872 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc159430 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.980+0000 7f7bdaa00640 1 --2- 172.21.15.28:0/4167949872 >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 0x7f7bdc06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.981+0000 7f7bdb400640 1 --2- 172.21.15.28:0/4167949872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06d220 tx=0x7f7bcc02c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82766b294f134637 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.981+0000 7f7bda000640 1 -- 172.21.15.28:0/4167949872 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc032480 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.981+0000 7f7bda000640 1 -- 172.21.15.28:0/4167949872 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bcc02f030 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.981+0000 7f7bda000640 1 -- 172.21.15.28:0/4167949872 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc032800 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 -- 172.21.15.28:0/4167949872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0cdfa0 msgr2=0x7f7bdc15d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 --2- 172.21.15.28:0/4167949872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc06d220 tx=0x7f7bcc02c6b0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 -- 172.21.15.28:0/4167949872 shutdown_connections 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 --2- 172.21.15.28:0/4167949872 >> v2:172.21.15.189:3300/0 conn(0x7f7bdc0b3160 0x7f7bdc06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 --2- 172.21.15.28:0/4167949872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc15d340 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 -- 172.21.15.28:0/4167949872 >> 172.21.15.28:0/4167949872 conn(0x7f7bdc069e80 msgr2=0x7f7bdc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 -- 172.21.15.28:0/4167949872 shutdown_connections 2024-09-13T17:14:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.982+0000 7f7be2800640 1 -- 172.21.15.28:0/4167949872 wait complete. 2024-09-13T17:14:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.984+0000 7f7be2800640 1 Processor -- start 2024-09-13T17:14:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.984+0000 7f7be2800640 1 -- start start 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7bdc06f980 0x7f7bdc148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc1551d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc154130 con 0x7f7bdc0b3160 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc154410 con 0x7f7bdc0cdfa0 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7be2800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bdc1546f0 con 0x7f7bdc06f980 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38960/0 (socket says 172.21.15.28:38960) 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 -- 172.21.15.28:0/1414099802 learned_addr learned my addr 172.21.15.28:0/1414099802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdb400640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7bdc06f980 0x7f7bdc148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 -- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7bdc06f980 msgr2=0x7f7bdc148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdbe00640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc1551d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7bdc06f980 0x7f7bdc148000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 -- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 msgr2=0x7f7bdc1551d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc1551d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdaa00640 1 -- 172.21.15.28:0/1414099802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bcc00b760 con 0x7f7bdc0b3160 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.985+0000 7f7bdbe00640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7bdaa00640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c7f0 tx=0x7f7bc800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc8007a30 con 0x7f7bdc0b3160 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bc8007b90 con 0x7f7bdc0b3160 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bc800ce40 con 0x7f7bdc0b3160 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bdc1549d0 con 0x7f7bdc0b3160 2024-09-13T17:14:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.986+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bdc148540 con 0x7f7bdc0b3160 2024-09-13T17:14:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.987+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bdc06f440 con 0x7f7bdc0b3160 2024-09-13T17:14:23.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.988+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7bc80113f0 con 0x7f7bdc0b3160 2024-09-13T17:14:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.989+0000 7f7bd3400640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 0x7f7bbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.989+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f7bc80893f0 con 0x7f7bdc0b3160 2024-09-13T17:14:23.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.990+0000 7f7bdb400640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 0x7f7bbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:23.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.990+0000 7f7bdb400640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 0x7f7bbc06d0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc037000 tx=0x7f7bcc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:23.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:23.993+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7bc80546f0 con 0x7f7bdc0b3160 2024-09-13T17:14:24.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:24.238+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]} v 0) v1 -- 0x7f7bdc0cf280 con 0x7f7bdc0b3160 2024-09-13T17:14:25.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.073+0000 7f7bd3400640 1 -- 172.21.15.28:0/1414099802 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]}]=0 v338) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7bc801f0e0 con 0x7f7bdc0b3160 2024-09-13T17:14:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 msgr2=0x7f7bbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 0x7f7bbc06d0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc037000 tx=0x7f7bcc030000 comp rx=0 tx=0).stop 2024-09-13T17:14:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 msgr2=0x7f7bdc153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f7bc800c7f0 tx=0x7f7bc800cbf0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 shutdown_connections 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7bbc06ac00 0x7f7bbc06d0c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7bdc0cdfa0 0x7f7bdc1551d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7bdc0b3160 0x7f7bdc153830 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 --2- 172.21.15.28:0/1414099802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7bdc06f980 0x7f7bdc148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 >> 172.21.15.28:0/1414099802 conn(0x7f7bdc069e80 msgr2=0x7f7bdc0b0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 shutdown_connections 2024-09-13T17:14:25.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.076+0000 7f7be2800640 1 -- 172.21.15.28:0/1414099802 wait complete. 2024-09-13T17:14:25.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:14:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.258+0000 7f4ac2200640 1 Processor -- start 2024-09-13T17:14:25.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.258+0000 7f4ac2200640 1 -- start start 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4ac2200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4abc0b17e0 0x7f4abc0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4ac2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc06c540 0x7f4abc16a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4ac2200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4abc1610a0 con 0x7f4abc06d070 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4ac2200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4abc0b2620 con 0x7f4abc0b17e0 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4ac2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc0b2900 con 0x7f4abc06c540 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4abc0b17e0 0x7f4abc0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4abc0b17e0 0x7f4abc0b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38976/0 (socket says 172.21.15.28:38976) 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 -- 172.21.15.28:0/999438533 learned_addr learned my addr 172.21.15.28:0/999438533 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abbe00640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc06c540 0x7f4abc16a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 -- 172.21.15.28:0/999438533 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc06c540 msgr2=0x7f4abc16a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc06c540 0x7f4abc16a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 -- 172.21.15.28:0/999438533 >> v1:172.21.15.189:6789/0 conn(0x7f4abc06d070 legacy=0x7f4abc0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.260+0000 7f4abb400640 1 -- 172.21.15.28:0/999438533 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4abc0b2be0 con 0x7f4abc0b17e0 2024-09-13T17:14:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.261+0000 7f4abb400640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc0b17e0 0x7f4abc0b3480 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009870 tx=0x7f4aac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737d277c6d63de32 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4abaa00640 1 -- 172.21.15.28:0/999438533 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac032440 con 0x7f4abc0b17e0 2024-09-13T17:14:25.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4abaa00640 1 -- 172.21.15.28:0/999438533 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac0325a0 con 0x7f4abc0b17e0 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4abaa00640 1 -- 172.21.15.28:0/999438533 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac032930 con 0x7f4abc0b17e0 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 -- 172.21.15.28:0/999438533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc0b17e0 msgr2=0x7f4abc0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc0b17e0 0x7f4abc0b3480 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f4aac009870 tx=0x7f4aac02a270 comp rx=0 tx=0).stop 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 -- 172.21.15.28:0/999438533 shutdown_connections 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc06c540 0x7f4abc16a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 --2- 172.21.15.28:0/999438533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc0b17e0 0x7f4abc0b3480 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 -- 172.21.15.28:0/999438533 >> 172.21.15.28:0/999438533 conn(0x7f4abc0b0270 msgr2=0x7f4abc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:25.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.262+0000 7f4ac2200640 1 -- 172.21.15.28:0/999438533 shutdown_connections 2024-09-13T17:14:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.263+0000 7f4ac2200640 1 -- 172.21.15.28:0/999438533 wait complete. 2024-09-13T17:14:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.264+0000 7f4ac2200640 1 Processor -- start 2024-09-13T17:14:25.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.264+0000 7f4ac2200640 1 -- start start 2024-09-13T17:14:25.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4abc06c540 0x7f4abc155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc0b17e0 0x7f4abc0aae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc15ca60 con 0x7f4abc06d070 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc156140 con 0x7f4abc06c540 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ac2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4abc15a480 con 0x7f4abc0b17e0 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ab2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4abb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4abc06c540 0x7f4abc155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ab2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38978/0 (socket says 172.21.15.28:38978) 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4ab2a00640 1 -- 172.21.15.28:0/3070768058 learned_addr learned my addr 172.21.15.28:0/3070768058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.265+0000 7f4abbe00640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc0b17e0 0x7f4abc0aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 -- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc0b17e0 msgr2=0x7f4abc0aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc0b17e0 0x7f4abc0aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 -- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4abc06c540 msgr2=0x7f4abc155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4abc06c540 0x7f4abc155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 -- 172.21.15.28:0/3070768058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0094d0 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab2a00640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400c400 tx=0x7f4aa400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40052d0 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4005430 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003830 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.266+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4abc15a760 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.267+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4abc15ad20 con 0x7f4abc06d070 2024-09-13T17:14:25.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.268+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4abc072550 con 0x7f4abc06d070 2024-09-13T17:14:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.269+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4aa4003990 con 0x7f4abc06d070 2024-09-13T17:14:25.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.270+0000 7f4ab3e00640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 0x7f4a9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.270+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f4aa4088670 con 0x7f4abc06d070 2024-09-13T17:14:25.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.270+0000 7f4abb400640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 0x7f4a9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.271+0000 7f4abb400640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 0x7f4a9806d0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4aac034000 tx=0x7f4aac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.274+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4aa4053ab0 con 0x7f4abc06d070 2024-09-13T17:14:25.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.453+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4abc0b3480 con 0x7f4abc06d070 2024-09-13T17:14:25.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.457+0000 7f4ab3e00640 1 -- 172.21.15.28:0/3070768058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4aa4088070 con 0x7f4abc06d070 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 msgr2=0x7f4a9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 0x7f4a9806d0c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f4aac034000 tx=0x7f4aac02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 msgr2=0x7f4abc0aa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400c400 tx=0x7f4aa400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 shutdown_connections 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4a9806ac00 0x7f4a9806d0c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4abc0b17e0 0x7f4abc0aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4abc06d070 0x7f4abc0aa930 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 --2- 172.21.15.28:0/3070768058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4abc06c540 0x7f4abc155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.459+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 >> 172.21.15.28:0/3070768058 conn(0x7f4abc0b0270 msgr2=0x7f4abc15e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.460+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 shutdown_connections 2024-09-13T17:14:25.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.460+0000 7f4ac2200640 1 -- 172.21.15.28:0/3070768058 wait complete. 2024-09-13T17:14:25.476 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:14:25.477 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:14:25.560 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:14:25.666 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:14:25.666 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:25.667 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:25.704 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:14:25.704 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:14:25.704 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:25.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:14:25.744 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:25.745 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:25.781 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:14:25.781 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:14:25.781 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:25.818 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:14:25.818 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:14:25.819 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.975+0000 7faee1000640 1 Processor -- start 2024-09-13T17:14:25.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.976+0000 7faee1000640 1 -- start start 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faee1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faedc159440 0x7faedc0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faee1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faee1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faedc0c3850 con 0x7faedc159820 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faee1000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc15a930 con 0x7faedc159440 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faee1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc159d20 con 0x7faedc0b29f0 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faedb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faedc159440 0x7faedc0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faed2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faedc159820 0x7faedc0b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57278/0 (socket says 172.21.15.28:57278) 2024-09-13T17:14:25.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.977+0000 7faed2a00640 1 -- 172.21.15.28:0/1752068432 learned_addr learned my addr 172.21.15.28:0/1752068432 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:25.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.978+0000 7faedb400640 1 -- 172.21.15.28:0/1752068432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 msgr2=0x7faedc0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:25.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.978+0000 7faedb400640 1 --2- 172.21.15.28:0/1752068432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.978+0000 7faedb400640 1 -- 172.21.15.28:0/1752068432 >> v1:172.21.15.28:6789/0 conn(0x7faedc159820 legacy=0x7faedc0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.978+0000 7faedb400640 1 -- 172.21.15.28:0/1752068432 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc15a000 con 0x7faedc159440 2024-09-13T17:14:25.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.978+0000 7faedaa00640 1 -- 172.21.15.28:0/1752068432 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723741666 0 0) 0x7faedc0c3850 con 0x7faedc159820 2024-09-13T17:14:25.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.979+0000 7faedb400640 1 --2- 172.21.15.28:0/1752068432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159440 0x7faedc0b0440 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7faecc0097d0 tx=0x7faecc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f3e36e4de2ac4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.979+0000 7faedaa00640 1 -- 172.21.15.28:0/1752068432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc032440 con 0x7faedc159440 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.979+0000 7faedaa00640 1 -- 172.21.15.28:0/1752068432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faecc0325a0 con 0x7faedc159440 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.979+0000 7faedaa00640 1 -- 172.21.15.28:0/1752068432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc032930 con 0x7faedc159440 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 -- 172.21.15.28:0/1752068432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159440 msgr2=0x7faedc0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 --2- 172.21.15.28:0/1752068432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159440 0x7faedc0b0440 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7faecc0097d0 tx=0x7faecc02a250 comp rx=0 tx=0).stop 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 -- 172.21.15.28:0/1752068432 shutdown_connections 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 --2- 172.21.15.28:0/1752068432 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 --2- 172.21.15.28:0/1752068432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159440 0x7faedc0b0440 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 -- 172.21.15.28:0/1752068432 >> 172.21.15.28:0/1752068432 conn(0x7faedc072e40 msgr2=0x7faedc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 -- 172.21.15.28:0/1752068432 shutdown_connections 2024-09-13T17:14:25.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.980+0000 7faee1000640 1 -- 172.21.15.28:0/1752068432 wait complete. 2024-09-13T17:14:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.981+0000 7faee1000640 1 Processor -- start 2024-09-13T17:14:25.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.982+0000 7faee1000640 1 -- start start 2024-09-13T17:14:25.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.982+0000 7faee1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc1f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.982+0000 7faee1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faedc159440 0x7faedc1f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faee1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faee1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc152720 con 0x7faedc159820 2024-09-13T17:14:25.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faee1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1f5c30 con 0x7faedc159440 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faee1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1f5f10 con 0x7faedc0b29f0 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38994/0 (socket says 172.21.15.28:38994) 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 -- 172.21.15.28:0/3822240461 learned_addr learned my addr 172.21.15.28:0/3822240461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedb400640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc1f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faed2a00640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faedc159440 0x7faedc1f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 -- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 msgr2=0x7faedc1f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc1f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 -- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faedc159440 msgr2=0x7faedc1f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faedc159440 0x7faedc1f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.983+0000 7faedbe00640 1 -- 172.21.15.28:0/3822240461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc009400 con 0x7faedc159820 2024-09-13T17:14:25.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faedbe00640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7faed400b9e0 tx=0x7faed400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed400caa0 con 0x7faedc159820 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed400cc20 con 0x7faedc159820 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4014440 con 0x7faedc159820 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc1f61f0 con 0x7faedc159820 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.984+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc0c3880 con 0x7faedc159820 2024-09-13T17:14:25.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.985+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc0c3a10 con 0x7faedc159820 2024-09-13T17:14:25.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.986+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faed401d3b0 con 0x7faedc159820 2024-09-13T17:14:25.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.987+0000 7faed3e00640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 0x7faeb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.987+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5115+0+0 (secure 0 0 0) 0x7faed4088fe0 con 0x7faedc159820 2024-09-13T17:14:25.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.987+0000 7faedb400640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 0x7faeb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:25.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.988+0000 7faedb400640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 0x7faeb806d0c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7faecc034000 tx=0x7faecc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:25.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:25.991+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faed4054440 con 0x7faedc159820 2024-09-13T17:14:26.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.205+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faedc070400 con 0x7faedc159820 2024-09-13T17:14:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.205+0000 7faed3e00640 1 -- 172.21.15.28:0/3822240461 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+3535 (secure 0 0 0) 0x7faed4019090 con 0x7faedc159820 2024-09-13T17:14:26.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:26.206 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6548,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2640369005","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2640369005},{"type":"v1","addr":"172.21.15.189:6835","nonce":2640369005}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":6572,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3001162997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3001162997},{"type":"v1","addr":"172.21.15.189:6837","nonce":3001162997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84},{"gid":7722,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1806106184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1806106184},{"type":"v1","addr":"172.21.15.28:6834","nonce":1806106184}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":81},{"gid":7914,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3504273483","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3504273483},{"type":"v1","addr":"172.21.15.28:6835","nonce":3504273483}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":88}],"filesystems":[]} 2024-09-13T17:14:26.207 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 88 2024-09-13T17:14:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 msgr2=0x7faeb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 0x7faeb806d0c0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7faecc034000 tx=0x7faecc02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:26.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 msgr2=0x7faedc1521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7faed400b9e0 tx=0x7faed400bea0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 shutdown_connections 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faeb806ac00 0x7faeb806d0c0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faedc159820 0x7faedc1521e0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faedc159440 0x7faedc1f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 --2- 172.21.15.28:0/3822240461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faedc0b29f0 0x7faedc1f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 >> 172.21.15.28:0/3822240461 conn(0x7faedc072e40 msgr2=0x7faedc158770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 shutdown_connections 2024-09-13T17:14:26.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.207+0000 7faee1000640 1 -- 172.21.15.28:0/3822240461 wait complete. 2024-09-13T17:14:26.224 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:14:26.224 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:14:26.224 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:14:26.224 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:14:26.225 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:14:26.225 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:14:26.225 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:14:26.225 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:14:26.225 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:14:26.225 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:14:26.225 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:14:26.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:14:26.226 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:14:26.226 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:14:26.227 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:14:26.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:14:26.239 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:14:26.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:14:26.240 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:14:26.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:14:26.241 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:14:26.241 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:14:26.242 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:14:26.242 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:14:26.284 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:14:26.284 DEBUG:teuthology.parallel:result is None 2024-09-13T17:14:26.286 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:14:26.286 DEBUG:teuthology.parallel:result is None 2024-09-13T17:14:26.287 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:14:26.287 DEBUG:teuthology.parallel:result is None 2024-09-13T17:14:26.288 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:14:26.288 DEBUG:teuthology.parallel:result is None 2024-09-13T17:14:26.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:14:26.417 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:14:26.417 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:14:26.418 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:14:26.418 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:14:26.444 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-13T17:14:26.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.457+0000 7f0acf200640 1 Processor -- start 2024-09-13T17:14:26.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.457+0000 7f0acf200640 1 -- start start 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0acf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ac80c40a0 0x7f0ac80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0acf200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac80b2090 con 0x7f0ac806c370 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0acf200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac80b2370 con 0x7f0ac80b1ba0 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0acf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac80b2650 con 0x7f0ac80c40a0 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0acde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ac80c40a0 0x7f0ac80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:26.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0ace800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ac806c370 0x7f0ac80b3130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57294/0 (socket says 172.21.15.28:57294) 2024-09-13T17:14:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.458+0000 7f0ace800640 1 -- 172.21.15.28:0/1041931623 learned_addr learned my addr 172.21.15.28:0/1041931623 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.459+0000 7f0acde00640 1 -- 172.21.15.28:0/1041931623 >> v1:172.21.15.189:6789/0 conn(0x7f0ac80b1ba0 legacy=0x7f0ac815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.459+0000 7f0acde00640 1 -- 172.21.15.28:0/1041931623 >> v1:172.21.15.28:6789/0 conn(0x7f0ac806c370 legacy=0x7f0ac80b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.459+0000 7f0acde00640 1 -- 172.21.15.28:0/1041931623 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80b2930 con 0x7f0ac80c40a0 2024-09-13T17:14:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.460+0000 7f0acde00640 1 --2- 172.21.15.28:0/1041931623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac80c40a0 0x7f0ac80b1660 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af0a736ef22dec7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.460+0000 7f0acca00640 1 -- 172.21.15.28:0/1041931623 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc032440 con 0x7f0ac80c40a0 2024-09-13T17:14:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.460+0000 7f0acca00640 1 -- 172.21.15.28:0/1041931623 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc0325a0 con 0x7f0ac80c40a0 2024-09-13T17:14:26.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.460+0000 7f0acca00640 1 -- 172.21.15.28:0/1041931623 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc032930 con 0x7f0ac80c40a0 2024-09-13T17:14:26.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 -- 172.21.15.28:0/1041931623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac80c40a0 msgr2=0x7f0ac80b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 --2- 172.21.15.28:0/1041931623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac80c40a0 0x7f0ac80b1660 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f0abc0097a0 tx=0x7f0abc02a230 comp rx=0 tx=0).stop 2024-09-13T17:14:26.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 -- 172.21.15.28:0/1041931623 shutdown_connections 2024-09-13T17:14:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 --2- 172.21.15.28:0/1041931623 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac80c40a0 0x7f0ac80b1660 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 -- 172.21.15.28:0/1041931623 >> 172.21.15.28:0/1041931623 conn(0x7f0ac8073470 msgr2=0x7f0ac8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 -- 172.21.15.28:0/1041931623 shutdown_connections 2024-09-13T17:14:26.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.461+0000 7f0acf200640 1 -- 172.21.15.28:0/1041931623 wait complete. 2024-09-13T17:14:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.463+0000 7f0acf200640 1 Processor -- start 2024-09-13T17:14:26.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.463+0000 7f0acf200640 1 -- start start 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ac80b1ba0 0x7f0ac813cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac80c40a0 0x7f0ac813d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac81e3ef0 con 0x7f0ac806c370 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac81e41d0 con 0x7f0ac80c40a0 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acf200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac81e44b0 con 0x7f0ac80b1ba0 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39050/0 (socket says 172.21.15.28:39050) 2024-09-13T17:14:26.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acde00640 1 -- 172.21.15.28:0/4180084590 learned_addr learned my addr 172.21.15.28:0/4180084590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.464+0000 7f0acd400640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ac80b1ba0 0x7f0ac813cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0ace800640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac80c40a0 0x7f0ac813d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 -- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ac80b1ba0 msgr2=0x7f0ac813cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ac80b1ba0 0x7f0ac813cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 -- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac80c40a0 msgr2=0x7f0ac813d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac80c40a0 0x7f0ac813d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:26.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 -- 172.21.15.28:0/4180084590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0abc009400 con 0x7f0ac806c370 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.465+0000 7f0acde00640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0abc035040 tx=0x7f0abc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.466+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc006e10 con 0x7f0ac806c370 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.466+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0abc030980 con 0x7f0ac806c370 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.466+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0abc030d50 con 0x7f0ac806c370 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.466+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac815b400 con 0x7f0ac806c370 2024-09-13T17:14:26.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.466+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac81e4a70 con 0x7f0ac806c370 2024-09-13T17:14:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.467+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac81e45f0 con 0x7f0ac806c370 2024-09-13T17:14:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.468+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0abc039040 con 0x7f0ac806c370 2024-09-13T17:14:26.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.469+0000 7f0ac2000640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 0x7f0aac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.469+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5115+0+0 (secure 0 0 0) 0x7f0abc0a9ae0 con 0x7f0ac806c370 2024-09-13T17:14:26.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.469+0000 7f0acd400640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 0x7f0aac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:26.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.470+0000 7f0acd400640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 0x7f0aac06d0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0ac815a7a0 tx=0x7f0ab8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:26.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.473+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0ac81e45f0 con 0x7f0ac806c370 2024-09-13T17:14:26.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:26.665+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0ac81e45f0 con 0x7f0ac806c370 2024-09-13T17:14:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.465+0000 7f0ac2000640 1 -- 172.21.15.28:0/4180084590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0ac81e45f0 con 0x7f0ac806c370 2024-09-13T17:14:27.466 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 msgr2=0x7f0aac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 0x7f0aac06d0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f0ac815a7a0 tx=0x7f0ab8009040 comp rx=0 tx=0).stop 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 msgr2=0x7f0ac813c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f0abc035040 tx=0x7f0abc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 shutdown_connections 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0aac06ac00 0x7f0aac06d0c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ac80c40a0 0x7f0ac813d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ac80b1ba0 0x7f0ac813cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 --2- 172.21.15.28:0/4180084590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ac806c370 0x7f0ac813c3c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 >> 172.21.15.28:0/4180084590 conn(0x7f0ac8073470 msgr2=0x7f0ac8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 shutdown_connections 2024-09-13T17:14:27.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.468+0000 7f0acf200640 1 -- 172.21.15.28:0/4180084590 wait complete. 2024-09-13T17:14:27.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:14:27.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.648+0000 7fcbe1000640 1 Processor -- start 2024-09-13T17:14:27.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.648+0000 7fcbe1000640 1 -- start start 2024-09-13T17:14:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.649+0000 7fcbe1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 0x7fcbdc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.649+0000 7fcbe1000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc15e100 con 0x7fcbdc070cc0 2024-09-13T17:14:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbe1000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbdc15e3e0 con 0x7fcbdc15a540 2024-09-13T17:14:27.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbe1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc0adb00 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbda800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcbdc070cc0 0x7fcbdc06f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57310/0 (socket says 172.21.15.28:57310) 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbda800640 1 -- 172.21.15.28:0/2211068361 learned_addr learned my addr 172.21.15.28:0/2211068361 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbd1400640 1 --2- 172.21.15.28:0/2211068361 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 0x7fcbdc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1068154323 0 0) 0x7fcbdc15e100 con 0x7fcbdc070cc0 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.650+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbc0003520 con 0x7fcbdc070cc0 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.651+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721257117 0 0) 0x7fcbdc15e3e0 con 0x7fcbdc15a540 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.651+0000 7fcbd1400640 1 -- 172.21.15.28:0/2211068361 >> v1:172.21.15.189:6789/0 conn(0x7fcbdc15a540 legacy=0x7fcbdc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.651+0000 7fcbd1400640 1 -- 172.21.15.28:0/2211068361 >> v1:172.21.15.28:6789/0 conn(0x7fcbdc070cc0 legacy=0x7fcbdc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.651+0000 7fcbd1400640 1 -- 172.21.15.28:0/2211068361 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc15ef80 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbd1400640 1 --2- 172.21.15.28:0/2211068361 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 0x7fcbdc06c430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40097d0 tx=0x7fcbc402a0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aef582ba34e0bf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:27.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4001970 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc402ab70 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbd9400640 1 -- 172.21.15.28:0/2211068361 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc4032440 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbe1000640 1 -- 172.21.15.28:0/2211068361 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 msgr2=0x7fcbdc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.652+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2211068361 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 0x7fcbdc06c430 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40097d0 tx=0x7fcbc402a0e0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.653+0000 7fcbe1000640 1 -- 172.21.15.28:0/2211068361 shutdown_connections 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.653+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2211068361 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc0cabc0 0x7fcbdc06c430 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.653+0000 7fcbe1000640 1 -- 172.21.15.28:0/2211068361 >> 172.21.15.28:0/2211068361 conn(0x7fcbdc069e80 msgr2=0x7fcbdc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.653+0000 7fcbe1000640 1 -- 172.21.15.28:0/2211068361 shutdown_connections 2024-09-13T17:14:27.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.653+0000 7fcbe1000640 1 -- 172.21.15.28:0/2211068361 wait complete. 2024-09-13T17:14:27.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.654+0000 7fcbe1000640 1 Processor -- start 2024-09-13T17:14:27.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.654+0000 7fcbe1000640 1 -- start start 2024-09-13T17:14:27.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc070cc0 0x7fcbdc13e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbdc0cabc0 0x7fcbdc0b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc159620 con 0x7fcbdc15a540 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc158a10 con 0x7fcbdc0cabc0 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.655+0000 7fcbe1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc158cf0 con 0x7fcbdc070cc0 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39060/0 (socket says 172.21.15.28:39060) 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 -- 172.21.15.28:0/2189170151 learned_addr learned my addr 172.21.15.28:0/2189170151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbd9e00640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc070cc0 0x7fcbdc13e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbd1400640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbdc0cabc0 0x7fcbdc0b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 -- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc070cc0 msgr2=0x7fcbdc13e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc070cc0 0x7fcbdc13e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 -- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbdc0cabc0 msgr2=0x7fcbdc0b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbdc0cabc0 0x7fcbdc0b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 -- 172.21.15.28:0/2189170151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc4009400 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.656+0000 7fcbda800640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400caf0 tx=0x7fcbd4002120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.657+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400a760 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.657+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd400a8c0 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.657+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc158fd0 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.657+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd400acb0 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.657+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc13ebb0 con 0x7fcbdc15a540 2024-09-13T17:14:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.658+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc159790 con 0x7fcbdc15a540 2024-09-13T17:14:27.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.659+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcbd40070d0 con 0x7fcbdc15a540 2024-09-13T17:14:27.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.660+0000 7fcbd2a00640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 0x7fcbb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:27.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.660+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcbd40889e0 con 0x7fcbdc15a540 2024-09-13T17:14:27.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.660+0000 7fcbd9e00640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 0x7fcbb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:27.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.661+0000 7fcbd9e00640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 0x7fcbb806d0c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc000f60 tx=0x7fcbcc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:27.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.664+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcbd4053fb0 con 0x7fcbdc15a540 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.906+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcbdc15bed0 con 0x7fcbdc15a540 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.906+0000 7fcbd2a00640 1 -- 172.21.15.28:0/2189170151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcbd4018020 con 0x7fcbdc15a540 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:14:27.907 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.908 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.910 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.911 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:14:27.914 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:14:27.918 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:14:27.920 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:14:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.908+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 msgr2=0x7fcbb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.908+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 0x7fcbb806d0c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc000f60 tx=0x7fcbcc009040 comp rx=0 tx=0).stop 2024-09-13T17:14:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.908+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 msgr2=0x7fcbdc0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.908+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400caf0 tx=0x7fcbd4002120 comp rx=0 tx=0).stop 2024-09-13T17:14:27.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 shutdown_connections 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcbb806ac00 0x7fcbb806d0c0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcbdc15a540 0x7fcbdc0ac850 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcbdc0cabc0 0x7fcbdc0b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 --2- 172.21.15.28:0/2189170151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcbdc070cc0 0x7fcbdc13e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 >> 172.21.15.28:0/2189170151 conn(0x7fcbdc069e80 msgr2=0x7fcbdc15e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 shutdown_connections 2024-09-13T17:14:27.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:27.909+0000 7fcbe1000640 1 -- 172.21.15.28:0/2189170151 wait complete. 2024-09-13T17:14:27.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.081+0000 7fbf8f000640 1 Processor -- start 2024-09-13T17:14:28.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.081+0000 7fbf8f000640 1 -- start start 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.082+0000 7fbf8f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 0x7fbf8815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf8f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf88159d90 con 0x7fbf8806f970 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf8f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf880b2040 con 0x7fbf8806c2d0 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf8f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf8815b0b0 con 0x7fbf881592a0 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf87400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbf8806f970 0x7fbf88159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57320/0 (socket says 172.21.15.28:57320) 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf87400640 1 -- 172.21.15.28:0/1025950437 learned_addr learned my addr 172.21.15.28:0/1025950437 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf87e00640 1 --2- 172.21.15.28:0/1025950437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 0x7fbf8815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf86a00640 1 -- 172.21.15.28:0/1025950437 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108969635 0 0) 0x7fbf88159d90 con 0x7fbf8806f970 2024-09-13T17:14:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.083+0000 7fbf86a00640 1 -- 172.21.15.28:0/1025950437 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf6c003520 con 0x7fbf8806f970 2024-09-13T17:14:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.084+0000 7fbf87e00640 1 -- 172.21.15.28:0/1025950437 >> v1:172.21.15.189:6789/0 conn(0x7fbf8806c2d0 legacy=0x7fbf880b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.084+0000 7fbf87e00640 1 -- 172.21.15.28:0/1025950437 >> v1:172.21.15.28:6789/0 conn(0x7fbf8806f970 legacy=0x7fbf88159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.084+0000 7fbf87e00640 1 -- 172.21.15.28:0/1025950437 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88159ff0 con 0x7fbf881592a0 2024-09-13T17:14:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf87e00640 1 --2- 172.21.15.28:0/1025950437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 0x7fbf8815ab70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf780097d0 tx=0x7fbf7802a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5c878bc41be75b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf86a00640 1 -- 172.21.15.28:0/1025950437 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78032440 con 0x7fbf881592a0 2024-09-13T17:14:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf86a00640 1 -- 172.21.15.28:0/1025950437 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf780325a0 con 0x7fbf881592a0 2024-09-13T17:14:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf86a00640 1 -- 172.21.15.28:0/1025950437 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78032930 con 0x7fbf881592a0 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf8f000640 1 -- 172.21.15.28:0/1025950437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 msgr2=0x7fbf8815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.085+0000 7fbf8f000640 1 --2- 172.21.15.28:0/1025950437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 0x7fbf8815ab70 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbf780097d0 tx=0x7fbf7802a200 comp rx=0 tx=0).stop 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.086+0000 7fbf8f000640 1 -- 172.21.15.28:0/1025950437 shutdown_connections 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.086+0000 7fbf8f000640 1 --2- 172.21.15.28:0/1025950437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf881592a0 0x7fbf8815ab70 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.086+0000 7fbf8f000640 1 -- 172.21.15.28:0/1025950437 >> 172.21.15.28:0/1025950437 conn(0x7fbf88069e80 msgr2=0x7fbf8806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.086+0000 7fbf8f000640 1 -- 172.21.15.28:0/1025950437 shutdown_connections 2024-09-13T17:14:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.086+0000 7fbf8f000640 1 -- 172.21.15.28:0/1025950437 wait complete. 2024-09-13T17:14:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.087+0000 7fbf8f000640 1 Processor -- start 2024-09-13T17:14:28.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.087+0000 7fbf8f000640 1 -- start start 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf8806f970 0x7fbf880acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 0x7fbf880c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880acaf0 con 0x7fbf881592a0 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880abee0 con 0x7fbf8806c2d0 2024-09-13T17:14:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.088+0000 7fbf8f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf880ac1c0 con 0x7fbf8806f970 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf8d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 0x7fbf880c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:34346/0 (socket says 172.21.15.28:34346) 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 -- 172.21.15.28:0/3757913455 learned_addr learned my addr 172.21.15.28:0/3757913455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87400640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf8806f970 0x7fbf880acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 -- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf8806f970 msgr2=0x7fbf880acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf8806f970 0x7fbf880acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 -- 172.21.15.28:0/3757913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 msgr2=0x7fbf880c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 0x7fbf880c3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf87e00640 1 -- 172.21.15.28:0/3757913455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf78009400 con 0x7fbf8806c2d0 2024-09-13T17:14:28.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.089+0000 7fbf8d000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 0x7fbf880c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf87e00640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbf780097d0 tx=0x7fbf780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78030b80 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf78032e90 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf880ac4a0 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf78006e10 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.090+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf881465b0 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.091+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf8815ab70 con 0x7fbf8806c2d0 2024-09-13T17:14:28.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.092+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf78039040 con 0x7fbf8806c2d0 2024-09-13T17:14:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.093+0000 7fbf7fe00640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 0x7fbf6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.093+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf780a9960 con 0x7fbf8806c2d0 2024-09-13T17:14:28.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.093+0000 7fbf87400640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 0x7fbf6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.094+0000 7fbf87400640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 0x7fbf6806d0c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbf74005880 tx=0x7fbf74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.098+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf78074ea0 con 0x7fbf8806c2d0 2024-09-13T17:14:28.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.279+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf880adc20 con 0x7fbf8806c2d0 2024-09-13T17:14:28.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.281+0000 7fbf7fe00640 1 -- 172.21.15.28:0/3757913455 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+12859 (secure 0 0 0) 0x7fbf780784c0 con 0x7fbf8806c2d0 2024-09-13T17:14:28.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:28.282 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":107,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:27.458824+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 msgr2=0x7fbf6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 0x7fbf6806d0c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fbf74005880 tx=0x7fbf74009040 comp rx=0 tx=0).stop 2024-09-13T17:14:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 msgr2=0x7fbf88146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbf780097d0 tx=0x7fbf780046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 shutdown_connections 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf6806ac00 0x7fbf6806d0c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf881592a0 0x7fbf880c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf8806f970 0x7fbf880acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 --2- 172.21.15.28:0/3757913455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf8806c2d0 0x7fbf88146070 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 >> 172.21.15.28:0/3757913455 conn(0x7fbf88069e80 msgr2=0x7fbf880b0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 shutdown_connections 2024-09-13T17:14:28.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.283+0000 7fbf8f000640 1 -- 172.21.15.28:0/3757913455 wait complete. 2024-09-13T17:14:28.300 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:14:28.300 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:14:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.458+0000 7f501a200640 1 Processor -- start 2024-09-13T17:14:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.458+0000 7f501a200640 1 -- start start 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f501a200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50140cdc70 0x7f50140b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f501a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5014071240 0x7f501406bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f501a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501406f5c0 con 0x7f501415ccf0 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f501a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501406f8a0 con 0x7f5014071240 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f501a200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501406fb80 con 0x7f50140cdc70 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f5013400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f501415ccf0 0x7f50140cd560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57330/0 (socket says 172.21.15.28:57330) 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f5013400640 1 -- 172.21.15.28:0/3617688779 learned_addr learned my addr 172.21.15.28:0/3617688779 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.459+0000 7f5013e00640 1 --2- 172.21.15.28:0/3617688779 >> v2:172.21.15.28:3300/0 conn(0x7f5014071240 0x7f501406bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f500aa00640 1 --2- 172.21.15.28:0/3617688779 >> v2:172.21.15.189:3300/0 conn(0x7f50140cdc70 0x7f50140b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5012a00640 1 -- 172.21.15.28:0/3617688779 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319129469 0 0) 0x7f501406f5c0 con 0x7f501415ccf0 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5012a00640 1 -- 172.21.15.28:0/3617688779 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff8003520 con 0x7f501415ccf0 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5013e00640 1 -- 172.21.15.28:0/3617688779 >> v2:172.21.15.189:3300/0 conn(0x7f50140cdc70 msgr2=0x7f50140b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5013e00640 1 --2- 172.21.15.28:0/3617688779 >> v2:172.21.15.189:3300/0 conn(0x7f50140cdc70 0x7f50140b1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5013e00640 1 -- 172.21.15.28:0/3617688779 >> v1:172.21.15.28:6789/0 conn(0x7f501415ccf0 legacy=0x7f50140cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.460+0000 7f5013e00640 1 -- 172.21.15.28:0/3617688779 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501406b610 con 0x7f5014071240 2024-09-13T17:14:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.461+0000 7f5013e00640 1 --2- 172.21.15.28:0/3617688779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f501406bb30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5014071cc0 tx=0x7f500c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d711e94a559d865 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.461+0000 7f5012a00640 1 -- 172.21.15.28:0/3617688779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c007740 con 0x7f5014071240 2024-09-13T17:14:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.461+0000 7f5012a00640 1 -- 172.21.15.28:0/3617688779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c02d050 con 0x7f5014071240 2024-09-13T17:14:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.461+0000 7f5012a00640 1 -- 172.21.15.28:0/3617688779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c007aa0 con 0x7f5014071240 2024-09-13T17:14:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 -- 172.21.15.28:0/3617688779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 msgr2=0x7f501406bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 --2- 172.21.15.28:0/3617688779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f501406bb30 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f5014071cc0 tx=0x7f500c02a710 comp rx=0 tx=0).stop 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 -- 172.21.15.28:0/3617688779 shutdown_connections 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 --2- 172.21.15.28:0/3617688779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f501406bb30 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 --2- 172.21.15.28:0/3617688779 >> v2:172.21.15.189:3300/0 conn(0x7f50140cdc70 0x7f50140b1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 -- 172.21.15.28:0/3617688779 >> 172.21.15.28:0/3617688779 conn(0x7f50140b0200 msgr2=0x7f50140b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 -- 172.21.15.28:0/3617688779 shutdown_connections 2024-09-13T17:14:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.462+0000 7f501a200640 1 -- 172.21.15.28:0/3617688779 wait complete. 2024-09-13T17:14:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.463+0000 7f501a200640 1 Processor -- start 2024-09-13T17:14:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.464+0000 7f501a200640 1 -- start start 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.464+0000 7f501a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.464+0000 7f501a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50140cdc70 0x7f50140a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.464+0000 7f501a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f501415ccf0 0x7f50140a9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f501a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5014152410 con 0x7f5014071240 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f501a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50140a8980 con 0x7f501415ccf0 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f501a200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50140a8c50 con 0x7f50140cdc70 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39086/0 (socket says 172.21.15.28:39086) 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 -- 172.21.15.28:0/2937685055 learned_addr learned my addr 172.21.15.28:0/2937685055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:28.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f500aa00640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50140cdc70 0x7f50140a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013e00640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f501415ccf0 0x7f50140a9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 -- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50140cdc70 msgr2=0x7f50140a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50140cdc70 0x7f50140a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 -- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f501415ccf0 msgr2=0x7f50140a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f501415ccf0 0x7f50140a9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.465+0000 7f5013400640 1 -- 172.21.15.28:0/2937685055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500c009a70 con 0x7f5014071240 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f5013400640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f500400c390 tx=0x7f500400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5004005ce0 con 0x7f5014071240 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5004005e40 con 0x7f5014071240 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50040072f0 con 0x7f5014071240 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50140a8f30 con 0x7f5014071240 2024-09-13T17:14:28.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.466+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50140a94f0 con 0x7f5014071240 2024-09-13T17:14:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.467+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5014070690 con 0x7f5014071240 2024-09-13T17:14:28.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.468+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5004007450 con 0x7f5014071240 2024-09-13T17:14:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.469+0000 7f500be00640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 0x7f4ff006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.469+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5004088580 con 0x7f5014071240 2024-09-13T17:14:28.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.469+0000 7f500aa00640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 0x7f4ff006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.470+0000 7f500aa00640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 0x7f4ff006d0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5014157260 tx=0x7f4ffc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.473+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5004053b40 con 0x7f5014071240 2024-09-13T17:14:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.650+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f501406d630 con 0x7f5014071240 2024-09-13T17:14:28.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.650+0000 7f500be00640 1 -- 172.21.15.28:0/2937685055 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f50040170a0 con 0x7f5014071240 2024-09-13T17:14:28.652 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:14:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 msgr2=0x7f4ff006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 0x7f4ff006d0c0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f5014157260 tx=0x7f4ffc008040 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 msgr2=0x7f5014151ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f500400c390 tx=0x7f500400c850 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 shutdown_connections 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ff006ac00 0x7f4ff006d0c0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f501415ccf0 0x7f50140a9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50140cdc70 0x7f50140a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 --2- 172.21.15.28:0/2937685055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5014071240 0x7f5014151ed0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 >> 172.21.15.28:0/2937685055 conn(0x7f50140b0200 msgr2=0x7f501415a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 shutdown_connections 2024-09-13T17:14:28.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.652+0000 7f501a200640 1 -- 172.21.15.28:0/2937685055 wait complete. 2024-09-13T17:14:28.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:14:28.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.823+0000 7f6795c00640 1 Processor -- start 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.823+0000 7f6795c00640 1 -- start start 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6795c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f679006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6795c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 0x7f6790070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6795c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679006f920 con 0x7f679006f3f0 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6795c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679015b290 con 0x7f679015cd80 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6795c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679015b570 con 0x7f6790159770 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.824+0000 7f6787e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f679006f3f0 0x7f6790159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57334/0 (socket says 172.21.15.28:57334) 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f6787e00640 1 -- 172.21.15.28:0/4088071293 learned_addr learned my addr 172.21.15.28:0/4088071293 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678ea00640 1 --2- 172.21.15.28:0/4088071293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f679006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678f400640 1 --2- 172.21.15.28:0/4088071293 >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 0x7f6790070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678e000640 1 -- 172.21.15.28:0/4088071293 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659208156 0 0) 0x7f679006f920 con 0x7f679006f3f0 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678e000640 1 -- 172.21.15.28:0/4088071293 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6774005730 con 0x7f679006f3f0 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678ea00640 1 -- 172.21.15.28:0/4088071293 >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 msgr2=0x7f6790070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678ea00640 1 --2- 172.21.15.28:0/4088071293 >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 0x7f6790070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678ea00640 1 -- 172.21.15.28:0/4088071293 >> v1:172.21.15.28:6789/0 conn(0x7f679006f3f0 legacy=0x7f6790159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678ea00640 1 -- 172.21.15.28:0/4088071293 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679015b850 con 0x7f6790159770 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.825+0000 7f678f400640 1 --2- 172.21.15.28:0/4088071293 >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 0x7f6790070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.826+0000 7f678ea00640 1 --2- 172.21.15.28:0/4088071293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f679006fd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f67800097a0 tx=0x7f678002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=653e98c5cc46e9af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f678e000640 1 -- 172.21.15.28:0/4088071293 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678002ae50 con 0x7f6790159770 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f678e000640 1 -- 172.21.15.28:0/4088071293 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780032440 con 0x7f6790159770 2024-09-13T17:14:28.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f678e000640 1 -- 172.21.15.28:0/4088071293 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67800327b0 con 0x7f6790159770 2024-09-13T17:14:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 -- 172.21.15.28:0/4088071293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 msgr2=0x7f679006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 --2- 172.21.15.28:0/4088071293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f679006fd20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f67800097a0 tx=0x7f678002a100 comp rx=0 tx=0).stop 2024-09-13T17:14:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 -- 172.21.15.28:0/4088071293 shutdown_connections 2024-09-13T17:14:28.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 --2- 172.21.15.28:0/4088071293 >> v2:172.21.15.189:3300/0 conn(0x7f679015cd80 0x7f6790070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 --2- 172.21.15.28:0/4088071293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f679006fd20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 -- 172.21.15.28:0/4088071293 >> 172.21.15.28:0/4088071293 conn(0x7f67900b0270 msgr2=0x7f67900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:28.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 -- 172.21.15.28:0/4088071293 shutdown_connections 2024-09-13T17:14:28.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.827+0000 7f6795c00640 1 -- 172.21.15.28:0/4088071293 wait complete. 2024-09-13T17:14:28.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.829+0000 7f6795c00640 1 Processor -- start 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.829+0000 7f6795c00640 1 -- start start 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f6790157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679015cd80 0x7f67900c5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6790149e60 con 0x7f679006f3f0 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679014a0e0 con 0x7f679015cd80 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6795c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6790157b70 con 0x7f6790159770 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f678ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f678ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39090/0 (socket says 172.21.15.28:39090) 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f678ea00640 1 -- 172.21.15.28:0/363893802 learned_addr learned my addr 172.21.15.28:0/363893802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:28.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f678f400640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679015cd80 0x7f67900c5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.830+0000 7f6787e00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f6790157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 -- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 msgr2=0x7f6790157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f6790157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 -- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679015cd80 msgr2=0x7f67900c5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679015cd80 0x7f67900c5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 -- 172.21.15.28:0/363893802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788009a70 con 0x7f679006f3f0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f6787e00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f6790157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f678ea00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6780000c00 tx=0x7f67800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678003c810 con 0x7f679006f3f0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678003c970 con 0x7f679006f3f0 2024-09-13T17:14:28.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678003cd00 con 0x7f679006f3f0 2024-09-13T17:14:28.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.831+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6780009400 con 0x7f679006f3f0 2024-09-13T17:14:28.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.832+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6790158130 con 0x7f679006f3f0 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.833+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67900b38e0 con 0x7f679006f3f0 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.833+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f678003b3a0 con 0x7f679006f3f0 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.834+0000 7f6786a00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 0x7f676c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.834+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f67800a9c70 con 0x7f679006f3f0 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.835+0000 7f6787e00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 0x7f676c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:28.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.836+0000 7f6787e00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 0x7f676c06d0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6778007fd0 tx=0x7f6778009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:28.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:28.838+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6780075100 con 0x7f679006f3f0 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.020+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f679006fd20 con 0x7f679006f3f0 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.020+0000 7f6786a00640 1 -- 172.21.15.28:0/363893802 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7f678003ce60 con 0x7f679006f3f0 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:14:29.021 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:14:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:14:29.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:14:29.022 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:14:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 msgr2=0x7f676c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 0x7f676c06d0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6778007fd0 tx=0x7f6778009040 comp rx=0 tx=0).stop 2024-09-13T17:14:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 msgr2=0x7f6790149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6780000c00 tx=0x7f67800046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 shutdown_connections 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f676c06ac00 0x7f676c06d0c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679015cd80 0x7f67900c5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6790159770 0x7f6790157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 --2- 172.21.15.28:0/363893802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f679006f3f0 0x7f6790149920 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 >> 172.21.15.28:0/363893802 conn(0x7f67900b0270 msgr2=0x7f67900ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 shutdown_connections 2024-09-13T17:14:29.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.022+0000 7f6795c00640 1 -- 172.21.15.28:0/363893802 wait complete. 2024-09-13T17:14:29.039 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:14:29.039 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:14:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.200+0000 7fa912e00640 1 Processor -- start 2024-09-13T17:14:29.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.200+0000 7fa912e00640 1 -- start start 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa912e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c1650f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa912e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c144280 con 0x7fa90c143ea0 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa912e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa90c0b4490 con 0x7fa90c15ebd0 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa912e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0b3060 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa911a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa90c143ea0 0x7fa90c0b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57344/0 (socket says 172.21.15.28:57344) 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa911a00640 1 -- 172.21.15.28:0/1851615076 learned_addr learned my addr 172.21.15.28:0/1851615076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.201+0000 7fa912400640 1 --2- 172.21.15.28:0/1851615076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c1650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:29.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 626562045 0 0) 0x7fa900001110 con 0x7fa90c143ea0 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8f4003520 con 0x7fa90c143ea0 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612539113 0 0) 0x7fa90c0b4490 con 0x7fa90c15ebd0 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa900001110 con 0x7fa90c15ebd0 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa912400640 1 -- 172.21.15.28:0/1851615076 >> v1:172.21.15.189:6789/0 conn(0x7fa90c15ebd0 legacy=0x7fa90c0cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa912400640 1 -- 172.21.15.28:0/1851615076 >> v1:172.21.15.28:6789/0 conn(0x7fa90c143ea0 legacy=0x7fa90c0b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.202+0000 7fa912400640 1 -- 172.21.15.28:0/1851615076 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0b23e0 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.203+0000 7fa912400640 1 --2- 172.21.15.28:0/1851615076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c1650f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa904009dc0 tx=0x7fa90402a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39ce7ce9df9c983c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.203+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904007740 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.203+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90402d050 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.203+0000 7fa90be00640 1 -- 172.21.15.28:0/1851615076 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa904007aa0 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 -- 172.21.15.28:0/1851615076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 msgr2=0x7fa90c1650f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 --2- 172.21.15.28:0/1851615076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c1650f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa904009dc0 tx=0x7fa90402a640 comp rx=0 tx=0).stop 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 -- 172.21.15.28:0/1851615076 shutdown_connections 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 --2- 172.21.15.28:0/1851615076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c1650f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 -- 172.21.15.28:0/1851615076 >> 172.21.15.28:0/1851615076 conn(0x7fa90c069e80 msgr2=0x7fa90c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 -- 172.21.15.28:0/1851615076 shutdown_connections 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.204+0000 7fa912e00640 1 -- 172.21.15.28:0/1851615076 wait complete. 2024-09-13T17:14:29.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.206+0000 7fa912e00640 1 Processor -- start 2024-09-13T17:14:29.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.206+0000 7fa912e00640 1 -- start start 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c0c7590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa90c143ea0 0x7fa90c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0c7ad0 con 0x7fa90c15ebd0 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c13de50 con 0x7fa90c143ea0 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0c5c70 con 0x7fa90c0cd8a0 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39098/0 (socket says 172.21.15.28:39098) 2024-09-13T17:14:29.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 -- 172.21.15.28:0/3898423646 learned_addr learned my addr 172.21.15.28:0/3898423646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:29.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 -- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 msgr2=0x7fa90c0c7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c0c7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 -- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa90c143ea0 msgr2=0x7fa90c0b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa90c143ea0 0x7fa90c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.207+0000 7fa912400640 1 -- 172.21.15.28:0/3898423646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa904009a70 con 0x7fa90c15ebd0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa912400640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa90402ac30 tx=0x7fa904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90403c830 con 0x7fa90c15ebd0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90403c990 con 0x7fa90c15ebd0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90403cd20 con 0x7fa90c15ebd0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c0c5f50 con 0x7fa90c15ebd0 2024-09-13T17:14:29.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.208+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c13e3d0 con 0x7fa90c15ebd0 2024-09-13T17:14:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.209+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c067a10 con 0x7fa90c15ebd0 2024-09-13T17:14:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.210+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa904031400 con 0x7fa90c15ebd0 2024-09-13T17:14:29.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.211+0000 7fa909600640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 0x7fa8ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.211+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa9040aa9a0 con 0x7fa90c15ebd0 2024-09-13T17:14:29.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.211+0000 7fa911a00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 0x7fa8ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:29.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.212+0000 7fa911a00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 0x7fa8ec06d0c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa900001390 tx=0x7fa900007950 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:29.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.215+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa904075f60 con 0x7fa90c15ebd0 2024-09-13T17:14:29.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:29.393+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa90c13e5c0 con 0x7fa90c15ebd0 2024-09-13T17:14:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.092+0000 7fa909600640 1 -- 172.21.15.28:0/3898423646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa9040400b0 con 0x7fa90c15ebd0 2024-09-13T17:14:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:14:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.097+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 msgr2=0x7fa8ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.097+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 0x7fa8ec06d0c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa900001390 tx=0x7fa900007950 comp rx=0 tx=0).stop 2024-09-13T17:14:30.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.097+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 msgr2=0x7fa90c1599c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.097+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fa90402ac30 tx=0x7fa904004a40 comp rx=0 tx=0).stop 2024-09-13T17:14:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 shutdown_connections 2024-09-13T17:14:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa8ec06ac00 0x7fa8ec06d0c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa90c15ebd0 0x7fa90c1599c0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa90c143ea0 0x7fa90c0b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 --2- 172.21.15.28:0/3898423646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa90c0cd8a0 0x7fa90c0c7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 >> 172.21.15.28:0/3898423646 conn(0x7fa90c069e80 msgr2=0x7fa90c142710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 shutdown_connections 2024-09-13T17:14:30.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.098+0000 7fa912e00640 1 -- 172.21.15.28:0/3898423646 wait complete. 2024-09-13T17:14:30.122 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:14:30.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.287+0000 7f84a8800640 1 Processor -- start 2024-09-13T17:14:30.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.287+0000 7f84a8800640 1 -- start start 2024-09-13T17:14:30.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.288+0000 7f84a8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015a780 0x7f84a0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.288+0000 7f84a8800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84a015ab60 0x7f84a00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a8800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a006c8b0 con 0x7f84a006bb40 2024-09-13T17:14:30.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a8800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a006c6a0 con 0x7f84a015ab60 2024-09-13T17:14:30.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a006cb90 con 0x7f84a015a780 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f84a015ab60 0x7f84a00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f84a006bb40 0x7f84a006bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57352/0 (socket says 172.21.15.28:57352) 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a6800640 1 -- 172.21.15.28:0/752455467 learned_addr learned my addr 172.21.15.28:0/752455467 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5e00640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015a780 0x7f84a0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a4a00640 1 -- 172.21.15.28:0/752455467 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317442827 0 0) 0x7f84a006c8b0 con 0x7f84a006bb40 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a4a00640 1 -- 172.21.15.28:0/752455467 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8488003520 con 0x7f84a006bb40 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5400640 1 -- 172.21.15.28:0/752455467 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015a780 msgr2=0x7f84a0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5400640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015a780 0x7f84a0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5400640 1 -- 172.21.15.28:0/752455467 >> v1:172.21.15.28:6789/0 conn(0x7f84a006bb40 legacy=0x7f84a006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.289+0000 7f84a5400640 1 -- 172.21.15.28:0/752455467 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a00cc980 con 0x7f84a015ab60 2024-09-13T17:14:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.290+0000 7f84a5400640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015ab60 0x7f84a00712c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f84900097a0 tx=0x7f849002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f2f9ea501f9260 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a4a00640 1 -- 172.21.15.28:0/752455467 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490032440 con 0x7f84a015ab60 2024-09-13T17:14:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a4a00640 1 -- 172.21.15.28:0/752455467 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84900325a0 con 0x7f84a015ab60 2024-09-13T17:14:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a4a00640 1 -- 172.21.15.28:0/752455467 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8490032930 con 0x7f84a015ab60 2024-09-13T17:14:30.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 -- 172.21.15.28:0/752455467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015ab60 msgr2=0x7f84a00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015ab60 0x7f84a00712c0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f84900097a0 tx=0x7f849002a180 comp rx=0 tx=0).stop 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 -- 172.21.15.28:0/752455467 shutdown_connections 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015ab60 0x7f84a00712c0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 --2- 172.21.15.28:0/752455467 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015a780 0x7f84a0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 -- 172.21.15.28:0/752455467 >> 172.21.15.28:0/752455467 conn(0x7f84a00b0270 msgr2=0x7f84a00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 -- 172.21.15.28:0/752455467 shutdown_connections 2024-09-13T17:14:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.291+0000 7f84a8800640 1 -- 172.21.15.28:0/752455467 wait complete. 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.293+0000 7f84a8800640 1 Processor -- start 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.293+0000 7f84a8800640 1 -- start start 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84a006bb40 0x7f84a014af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015ab60 0x7f84a0151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00a8e80 con 0x7f84a015a780 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00a84e0 con 0x7f84a006bb40 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a00a87c0 con 0x7f84a015ab60 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39128/0 (socket says 172.21.15.28:39128) 2024-09-13T17:14:30.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a5400640 1 -- 172.21.15.28:0/2246949205 learned_addr learned my addr 172.21.15.28:0/2246949205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a5e00640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84a006bb40 0x7f84a014af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.294+0000 7f84a6800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015ab60 0x7f84a0151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 -- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015ab60 msgr2=0x7f84a0151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015ab60 0x7f84a0151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 -- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84a006bb40 msgr2=0x7f84a014af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84a006bb40 0x7f84a014af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 -- 172.21.15.28:0/2246949205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8490009400 con 0x7f84a015a780 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a5400640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8490000c00 tx=0x7f84900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849003d830 con 0x7f84a015a780 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849003d990 con 0x7f84a015a780 2024-09-13T17:14:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f849003dd40 con 0x7f84a015a780 2024-09-13T17:14:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.295+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a00a8aa0 con 0x7f84a015a780 2024-09-13T17:14:30.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.296+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a00ab210 con 0x7f84a015a780 2024-09-13T17:14:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.296+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a006d2d0 con 0x7f84a015a780 2024-09-13T17:14:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.298+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8490039040 con 0x7f84a015a780 2024-09-13T17:14:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.298+0000 7f849a000640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 0x7f848406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.299+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f84900a9fe0 con 0x7f84a015a780 2024-09-13T17:14:30.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.299+0000 7f84a5e00640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 0x7f848406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:30.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.300+0000 7f84a5e00640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 0x7f848406d0c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f84a00ad410 tx=0x7f8494009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:30.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.302+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8490075430 con 0x7f84a015a780 2024-09-13T17:14:30.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:30.482+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f84a00676b0 con 0x7f84a015a780 2024-09-13T17:14:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.094+0000 7f849a000640 1 -- 172.21.15.28:0/2246949205 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8490078a50 con 0x7f84a015a780 2024-09-13T17:14:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:14:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 msgr2=0x7f848406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 0x7f848406d0c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f84a00ad410 tx=0x7f8494009190 comp rx=0 tx=0).stop 2024-09-13T17:14:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 msgr2=0x7f84a01f0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8490000c00 tx=0x7f84900046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 shutdown_connections 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f848406ac00 0x7f848406d0c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f84a015ab60 0x7f84a0151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f84a015a780 0x7f84a01f0330 secure :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f8490000c00 tx=0x7f84900046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 --2- 172.21.15.28:0/2246949205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84a006bb40 0x7f84a014af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.101+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 >> 172.21.15.28:0/2246949205 conn(0x7f84a00b0270 msgr2=0x7f84a00b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:31.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.103+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 shutdown_connections 2024-09-13T17:14:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.103+0000 7f84a8800640 1 -- 172.21.15.28:0/2246949205 wait complete. 2024-09-13T17:14:31.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:14:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.292+0000 7ff1ebe00640 1 Processor -- start 2024-09-13T17:14:31.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.292+0000 7ff1ebe00640 1 -- start start 2024-09-13T17:14:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1ebe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff1ec146250 0x7ff1ec0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1ebe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec06ff70 0x7ff1ec0b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1ebe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec0ad940 con 0x7ff1ec06bbd0 2024-09-13T17:14:31.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1ebe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0ad0a0 con 0x7ff1ec146250 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1ebe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec0ad380 con 0x7ff1ec06ff70 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1e2000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff1ec06bbd0 0x7ff1ec06f860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57362/0 (socket says 172.21.15.28:57362) 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1e2000640 1 -- 172.21.15.28:0/1988188978 learned_addr learned my addr 172.21.15.28:0/1988188978 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/1988188978 >> v2:172.21.15.28:3300/0 conn(0x7ff1ec146250 0x7ff1ec0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.293+0000 7ff1eb400640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec06ff70 0x7ff1ec0b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.294+0000 7ff1eaa00640 1 -- 172.21.15.28:0/1988188978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec06ff70 msgr2=0x7ff1ec0b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.294+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec06ff70 0x7ff1ec0b1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.294+0000 7ff1eaa00640 1 -- 172.21.15.28:0/1988188978 >> v1:172.21.15.28:6789/0 conn(0x7ff1ec06bbd0 legacy=0x7ff1ec06f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.294+0000 7ff1eaa00640 1 -- 172.21.15.28:0/1988188978 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0ad660 con 0x7ff1ec146250 2024-09-13T17:14:31.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.295+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec146250 0x7ff1ec0ac850 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097a0 tx=0x7ff1dc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3ae5ec2aae8b7d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.295+0000 7ff1ea000640 1 -- 172.21.15.28:0/1988188978 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc032440 con 0x7ff1ec146250 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.295+0000 7ff1ea000640 1 -- 172.21.15.28:0/1988188978 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0325a0 con 0x7ff1ec146250 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.295+0000 7ff1ea000640 1 -- 172.21.15.28:0/1988188978 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc032930 con 0x7ff1ec146250 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 -- 172.21.15.28:0/1988188978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec146250 msgr2=0x7ff1ec0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec146250 0x7ff1ec0ac850 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097a0 tx=0x7ff1dc02a180 comp rx=0 tx=0).stop 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 -- 172.21.15.28:0/1988188978 shutdown_connections 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec06ff70 0x7ff1ec0b1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/1988188978 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec146250 0x7ff1ec0ac850 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 -- 172.21.15.28:0/1988188978 >> 172.21.15.28:0/1988188978 conn(0x7ff1ec073470 msgr2=0x7ff1ec069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 -- 172.21.15.28:0/1988188978 shutdown_connections 2024-09-13T17:14:31.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.296+0000 7ff1ebe00640 1 -- 172.21.15.28:0/1988188978 wait complete. 2024-09-13T17:14:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.298+0000 7ff1ebe00640 1 Processor -- start 2024-09-13T17:14:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.298+0000 7ff1ebe00640 1 -- start start 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.298+0000 7ff1ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ec06bbd0 0x7ff1ec144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1ebe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec146250 0x7ff1ec0c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1ebe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15f260 con 0x7ff1ec06ff70 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1ebe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15f540 con 0x7ff1ec06bbd0 2024-09-13T17:14:31.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1ebe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15f820 con 0x7ff1ec146250 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39158/0 (socket says 172.21.15.28:39158) 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 -- 172.21.15.28:0/2232283192 learned_addr learned my addr 172.21.15.28:0/2232283192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ec06bbd0 0x7ff1ec144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1eb400640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec146250 0x7ff1ec0c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 -- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec146250 msgr2=0x7ff1ec0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec146250 0x7ff1ec0c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 -- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ec06bbd0 msgr2=0x7ff1ec144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ec06bbd0 0x7ff1ec144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.299+0000 7ff1e2000640 1 -- 172.21.15.28:0/2232283192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc009400 con 0x7ff1ec06ff70 2024-09-13T17:14:31.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1e2000640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400c4a0 tx=0x7ff1d400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4005ce0 con 0x7ff1ec06ff70 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1d4005e40 con 0x7ff1ec06ff70 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4007270 con 0x7ff1ec06ff70 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec15fb00 con 0x7ff1ec06ff70 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.300+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec0c4930 con 0x7ff1ec06ff70 2024-09-13T17:14:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.301+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec160020 con 0x7ff1ec06ff70 2024-09-13T17:14:31.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.302+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff1d4007490 con 0x7ff1ec06ff70 2024-09-13T17:14:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.303+0000 7ff1e3400640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 0x7ff1c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.303+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff1d40887e0 con 0x7ff1ec06ff70 2024-09-13T17:14:31.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.303+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 0x7ff1c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.304+0000 7ff1eaa00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 0x7ff1c806d0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc034000 tx=0x7ff1dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.308+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff1ec160020 con 0x7ff1ec06ff70 2024-09-13T17:14:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.524+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff1ec0b1d40 con 0x7ff1ec06ff70 2024-09-13T17:14:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.533+0000 7ff1e3400640 1 -- 172.21.15.28:0/2232283192 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v90) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff1d40170c0 con 0x7ff1ec06ff70 2024-09-13T17:14:31.536 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 14 and data pool 15 2024-09-13T17:14:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.539+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 msgr2=0x7ff1c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.540+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 0x7ff1c806d0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc034000 tx=0x7ff1dc02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.540+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 msgr2=0x7ff1ec0c36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.540+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400c4a0 tx=0x7ff1d400c960 comp rx=0 tx=0).stop 2024-09-13T17:14:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 shutdown_connections 2024-09-13T17:14:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1c806ac00 0x7ff1c806d0c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1ec146250 0x7ff1ec0c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1ec06ff70 0x7ff1ec0c36f0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 --2- 172.21.15.28:0/2232283192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1ec06bbd0 0x7ff1ec144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.541+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 >> 172.21.15.28:0/2232283192 conn(0x7ff1ec073470 msgr2=0x7ff1ec0cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.543+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 shutdown_connections 2024-09-13T17:14:31.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.543+0000 7ff1ebe00640 1 -- 172.21.15.28:0/2232283192 wait complete. 2024-09-13T17:14:31.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:31.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.725+0000 7fd030800640 1 Processor -- start 2024-09-13T17:14:31.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.725+0000 7fd030800640 1 -- start start 2024-09-13T17:14:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.726+0000 7fd030800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0281592a0 0x7fd02815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.726+0000 7fd030800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd02806f970 0x7fd028159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.726+0000 7fd030800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0280b2040 con 0x7fd02806c2d0 2024-09-13T17:14:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.726+0000 7fd030800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02815b0b0 con 0x7fd0281592a0 2024-09-13T17:14:31.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.726+0000 7fd030800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd028159ff0 con 0x7fd02806f970 2024-09-13T17:14:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd0281592a0 0x7fd02815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd02806c2d0 0x7fd0280b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57372/0 (socket says 172.21.15.28:57372) 2024-09-13T17:14:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02e800640 1 -- 172.21.15.28:0/1293288414 learned_addr learned my addr 172.21.15.28:0/1293288414 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02d400640 1 --2- 172.21.15.28:0/1293288414 >> v2:172.21.15.189:3300/0 conn(0x7fd02806f970 0x7fd028159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02de00640 1 -- 172.21.15.28:0/1293288414 >> v2:172.21.15.189:3300/0 conn(0x7fd02806f970 msgr2=0x7fd028159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02de00640 1 --2- 172.21.15.28:0/1293288414 >> v2:172.21.15.189:3300/0 conn(0x7fd02806f970 0x7fd028159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02de00640 1 -- 172.21.15.28:0/1293288414 >> v1:172.21.15.28:6789/0 conn(0x7fd02806c2d0 legacy=0x7fd0280b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02de00640 1 -- 172.21.15.28:0/1293288414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02815a2d0 con 0x7fd0281592a0 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.727+0000 7fd02ca00640 1 -- 172.21.15.28:0/1293288414 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184819550 0 0) 0x7fd0280b2040 con 0x7fd02806c2d0 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.728+0000 7fd02de00640 1 --2- 172.21.15.28:0/1293288414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0281592a0 0x7fd02815ab70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fd02806ac10 tx=0x7fd01c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8f0186a520f122 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.728+0000 7fd02ca00640 1 -- 172.21.15.28:0/1293288414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c032440 con 0x7fd0281592a0 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd02ca00640 1 -- 172.21.15.28:0/1293288414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c0325a0 con 0x7fd0281592a0 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd02ca00640 1 -- 172.21.15.28:0/1293288414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c032930 con 0x7fd0281592a0 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 -- 172.21.15.28:0/1293288414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0281592a0 msgr2=0x7fd02815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 --2- 172.21.15.28:0/1293288414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0281592a0 0x7fd02815ab70 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fd02806ac10 tx=0x7fd01c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 -- 172.21.15.28:0/1293288414 shutdown_connections 2024-09-13T17:14:31.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 --2- 172.21.15.28:0/1293288414 >> v2:172.21.15.189:3300/0 conn(0x7fd02806f970 0x7fd028159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 --2- 172.21.15.28:0/1293288414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0281592a0 0x7fd02815ab70 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 -- 172.21.15.28:0/1293288414 >> 172.21.15.28:0/1293288414 conn(0x7fd028069e80 msgr2=0x7fd02806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 -- 172.21.15.28:0/1293288414 shutdown_connections 2024-09-13T17:14:31.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.729+0000 7fd030800640 1 -- 172.21.15.28:0/1293288414 wait complete. 2024-09-13T17:14:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.731+0000 7fd030800640 1 Processor -- start 2024-09-13T17:14:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.731+0000 7fd030800640 1 -- start start 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 0x7fd02814b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0281592a0 0x7fd028150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02814ef90 con 0x7fd02806c2d0 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02814b700 con 0x7fd0281592a0 2024-09-13T17:14:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd030800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd028151210 con 0x7fd02806f970 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39174/0 (socket says 172.21.15.28:39174) 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 -- 172.21.15.28:0/3044044530 learned_addr learned my addr 172.21.15.28:0/3044044530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02e800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0281592a0 0x7fd028150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02d400640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 0x7fd02814b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 -- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 msgr2=0x7fd02814b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 0x7fd02814b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 -- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0281592a0 msgr2=0x7fd028150cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.732+0000 7fd02de00640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0281592a0 0x7fd028150cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd02de00640 1 -- 172.21.15.28:0/3044044530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd01c009400 con 0x7fd02806c2d0 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd02d400640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 0x7fd02814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd02de00640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fd01c000c00 tx=0x7fd01c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c030b80 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01c032e90 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01c006e10 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0281f2400 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.733+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0281f29c0 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.734+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd02815ab70 con 0x7fd02806c2d0 2024-09-13T17:14:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.735+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd01c039040 con 0x7fd02806c2d0 2024-09-13T17:14:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.736+0000 7fd022000640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 0x7fd00c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.736+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd01c0a9d60 con 0x7fd02806c2d0 2024-09-13T17:14:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.736+0000 7fd02d400640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 0x7fd00c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.737+0000 7fd02d400640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 0x7fd00c06d0c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd0281f2330 tx=0x7fd018008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.740+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd01c074f30 con 0x7fd02806c2d0 2024-09-13T17:14:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.920+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0280b1990 con 0x7fd02806c2d0 2024-09-13T17:14:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.921+0000 7fd022000640 1 -- 172.21.15.28:0/3044044530 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16532 (secure 0 0 0) 0x7fd01c041090 con 0x7fd02806c2d0 2024-09-13T17:14:31.923 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:31.923 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":110,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:31.526005+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.923+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 msgr2=0x7fd00c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.923+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 0x7fd00c06d0c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd0281f2330 tx=0x7fd018008040 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.923+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 msgr2=0x7fd0281f1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.923+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fd01c000c00 tx=0x7fd01c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 shutdown_connections 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd00c06ac00 0x7fd00c06d0c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0281592a0 0x7fd028150cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd02806f970 0x7fd02814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 --2- 172.21.15.28:0/3044044530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd02806c2d0 0x7fd0281f1df0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 >> 172.21.15.28:0/3044044530 conn(0x7fd028069e80 msgr2=0x7fd02815c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 shutdown_connections 2024-09-13T17:14:31.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:31.924+0000 7fd030800640 1 -- 172.21.15.28:0/3044044530 wait complete. 2024-09-13T17:14:31.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.095+0000 7f0de4a00640 1 Processor -- start 2024-09-13T17:14:32.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.095+0000 7f0de4a00640 1 -- start start 2024-09-13T17:14:32.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.096+0000 7f0de4a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0de00cabc0 0x7f0de006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0de4a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de015e100 con 0x7f0de015a540 2024-09-13T17:14:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0de4a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0de015e3e0 con 0x7f0de0070cc0 2024-09-13T17:14:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0de4a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0de00adb00 con 0x7f0de00cabc0 2024-09-13T17:14:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0ddde00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0de015a540 0x7f0de015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57388/0 (socket says 172.21.15.28:57388) 2024-09-13T17:14:32.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0ddde00640 1 -- 172.21.15.28:0/4018162920 learned_addr learned my addr 172.21.15.28:0/4018162920 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0dd5400640 1 --2- 172.21.15.28:0/4018162920 >> v2:172.21.15.189:3300/0 conn(0x7f0de00cabc0 0x7f0de006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083234975 0 0) 0x7f0de015e100 con 0x7f0de015a540 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.097+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dc4003520 con 0x7f0de015a540 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4214459512 0 0) 0x7f0dc4003520 con 0x7f0de015a540 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 >> v2:172.21.15.189:3300/0 conn(0x7f0de00cabc0 msgr2=0x7f0de006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 --2- 172.21.15.28:0/4018162920 >> v2:172.21.15.189:3300/0 conn(0x7f0de00cabc0 0x7f0de006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 >> v1:172.21.15.189:6789/0 conn(0x7f0de0070cc0 legacy=0x7f0de006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de015ef80 con 0x7f0de015a540 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f0dd0001670 con 0x7f0de015a540 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 >> v1:172.21.15.28:6789/0 conn(0x7f0de015a540 legacy=0x7f0de015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dc4003970 0x7f0dc4003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc4004370 0x7f0dc4004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 0x7f0dc400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc400bff0 con 0x7f0dc40050e0 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc400c160 con 0x7f0dc4004370 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dc400c320 con 0x7f0dc4003970 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0dde800640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 0x7f0dc400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0dd5400640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc4004370 0x7f0dc4004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0ddde00640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dc4003970 0x7f0dc4003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0dde800640 1 -- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dc4003970 msgr2=0x7f0dc4003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:32.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.098+0000 7f0dde800640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dc4003970 0x7f0dc4003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.099+0000 7f0dde800640 1 -- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc4004370 msgr2=0x7f0dc4004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.099+0000 7f0dde800640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc4004370 0x7f0dc4004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.099+0000 7f0dde800640 1 -- 172.21.15.28:0/4018162920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dc4003520 con 0x7f0dc40050e0 2024-09-13T17:14:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.099+0000 7f0ddde00640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dc4003970 0x7f0dc4003d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.099+0000 7f0dde800640 1 -- 172.21.15.28:0/4018162920 reap_dead start 2024-09-13T17:14:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0dde800640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 0x7f0dc400b980 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8002010 tx=0x7f0dc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a55e32d30156753 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8003b70 con 0x7f0dc40050e0 2024-09-13T17:14:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dd802d050 con 0x7f0dc40050e0 2024-09-13T17:14:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0ddd400640 1 -- 172.21.15.28:0/4018162920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dd8005110 con 0x7f0dc40050e0 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 -- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 msgr2=0x7f0dc400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 0x7f0dc400b980 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0dd8002010 tx=0x7f0dc40057f0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 -- 172.21.15.28:0/4018162920 shutdown_connections 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 --2- 172.21.15.28:0/4018162920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dc40050e0 0x7f0dc400b980 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 -- 172.21.15.28:0/4018162920 >> 172.21.15.28:0/4018162920 conn(0x7f0de0069e80 msgr2=0x7f0de006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.100+0000 7f0de4a00640 1 -- 172.21.15.28:0/4018162920 shutdown_connections 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.101+0000 7f0de4a00640 1 -- 172.21.15.28:0/4018162920 wait complete. 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.102+0000 7f0de4a00640 1 Processor -- start 2024-09-13T17:14:32.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.102+0000 7f0de4a00640 1 -- start start 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc40050e0 0x7f0de013c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0de00b0240 0x7f0de00b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de00af930 con 0x7f0de00afdc0 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0de013ce10 con 0x7f0dc40050e0 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0de4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0de015afb0 con 0x7f0de00b0240 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0dd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0dd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.103+0000 7f0dd5400640 1 -- 172.21.15.28:0/3241622852 learned_addr learned my addr 172.21.15.28:0/3241622852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0ddde00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc40050e0 0x7f0de013c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dde800640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0de00b0240 0x7f0de00b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 -- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0de00b0240 msgr2=0x7f0de00b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0de00b0240 0x7f0de00b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 -- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc40050e0 msgr2=0x7f0de013c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc40050e0 0x7f0de013c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 -- 172.21.15.28:0/3241622852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de00aed20 con 0x7f0de00afdc0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd5400640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80118e0 tx=0x7f0dc8011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc800faf0 con 0x7f0de00afdc0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc800fc50 con 0x7f0de00afdc0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.104+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0de00af000 con 0x7f0de00afdc0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.105+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dc8014440 con 0x7f0de00afdc0 2024-09-13T17:14:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.105+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0de00af5c0 con 0x7f0de00afdc0 2024-09-13T17:14:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.106+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0de0061090 con 0x7f0de00afdc0 2024-09-13T17:14:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.107+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0dc80145a0 con 0x7f0de00afdc0 2024-09-13T17:14:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.107+0000 7f0dd6a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 0x7f0dbc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.108+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0dc8089b50 con 0x7f0de00afdc0 2024-09-13T17:14:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.108+0000 7f0ddde00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 0x7f0dbc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.109+0000 7f0ddde00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 0x7f0dbc06d0f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0de00cdff0 tx=0x7f0dd0007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.112+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0dc8054d40 con 0x7f0de00afdc0 2024-09-13T17:14:32.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.294+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0de006c7c0 con 0x7f0de00afdc0 2024-09-13T17:14:32.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.295+0000 7f0dd6a00640 1 -- 172.21.15.28:0/3241622852 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f0dc8058360 con 0x7f0de00afdc0 2024-09-13T17:14:32.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:32.298 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":110,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:31.526005+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 msgr2=0x7f0dbc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 0x7f0dbc06d0f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0de00cdff0 tx=0x7f0dd0007040 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 msgr2=0x7f0de00b1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f0dc80118e0 tx=0x7f0dc8011da0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 shutdown_connections 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0dbc06ac30 0x7f0dbc06d0f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0de00b0240 0x7f0de00b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0de00afdc0 0x7f0de00b1a00 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 --2- 172.21.15.28:0/3241622852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dc40050e0 0x7f0de013c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.297+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 >> 172.21.15.28:0/3241622852 conn(0x7f0de0069e80 msgr2=0x7f0de014c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.298+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 shutdown_connections 2024-09-13T17:14:32.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.298+0000 7f0de4a00640 1 -- 172.21.15.28:0/3241622852 wait complete. 2024-09-13T17:14:32.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:14:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.471+0000 7fbf98e00640 1 Processor -- start 2024-09-13T17:14:32.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.471+0000 7fbf98e00640 1 -- start start 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf94159770 0x7fbf9406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf9406f3f0 0x7fbf94070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf940add00 0x7fbf94159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9406f920 con 0x7fbf9406f3f0 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9415b290 con 0x7fbf940add00 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf98e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9415b570 con 0x7fbf94159770 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.472+0000 7fbf89400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf9406f3f0 0x7fbf94070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf9406f3f0 0x7fbf94070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39194/0 (socket says 172.21.15.28:39194) 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 -- 172.21.15.28:0/3557487746 learned_addr learned my addr 172.21.15.28:0/3557487746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf91e00640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf94159770 0x7fbf9406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf92800640 1 --2- 172.21.15.28:0/3557487746 >> v2:172.21.15.189:3300/0 conn(0x7fbf940add00 0x7fbf94159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 -- 172.21.15.28:0/3557487746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf94159770 msgr2=0x7fbf9406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf94159770 0x7fbf9406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 -- 172.21.15.28:0/3557487746 >> v2:172.21.15.189:3300/0 conn(0x7fbf940add00 msgr2=0x7fbf94159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 --2- 172.21.15.28:0/3557487746 >> v2:172.21.15.189:3300/0 conn(0x7fbf940add00 0x7fbf94159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.473+0000 7fbf89400640 1 -- 172.21.15.28:0/3557487746 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9415b850 con 0x7fbf9406f3f0 2024-09-13T17:14:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.474+0000 7fbf89400640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406f3f0 0x7fbf94070260 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406ce40 tx=0x7fbf7c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b34ed5ff2c51b4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.474+0000 7fbf91400640 1 -- 172.21.15.28:0/3557487746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c02ae90 con 0x7fbf9406f3f0 2024-09-13T17:14:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.474+0000 7fbf91400640 1 -- 172.21.15.28:0/3557487746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf7c032440 con 0x7fbf9406f3f0 2024-09-13T17:14:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.474+0000 7fbf91400640 1 -- 172.21.15.28:0/3557487746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0327d0 con 0x7fbf9406f3f0 2024-09-13T17:14:32.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 -- 172.21.15.28:0/3557487746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406f3f0 msgr2=0x7fbf94070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406f3f0 0x7fbf94070260 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fbf9406ce40 tx=0x7fbf7c02a140 comp rx=0 tx=0).stop 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 -- 172.21.15.28:0/3557487746 shutdown_connections 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf9406f3f0 0x7fbf94070260 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 --2- 172.21.15.28:0/3557487746 >> v2:172.21.15.189:3300/0 conn(0x7fbf940add00 0x7fbf94159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 --2- 172.21.15.28:0/3557487746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf94159770 0x7fbf9406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 -- 172.21.15.28:0/3557487746 >> 172.21.15.28:0/3557487746 conn(0x7fbf940b0270 msgr2=0x7fbf940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 -- 172.21.15.28:0/3557487746 shutdown_connections 2024-09-13T17:14:32.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.475+0000 7fbf98e00640 1 -- 172.21.15.28:0/3557487746 wait complete. 2024-09-13T17:14:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.477+0000 7fbf98e00640 1 Processor -- start 2024-09-13T17:14:32.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.477+0000 7fbf98e00640 1 -- start start 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf9406f3f0 0x7fbf9406bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 0x7fbf94152730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94154a20 con 0x7fbf94159770 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94152c70 con 0x7fbf940add00 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf98e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf94154410 con 0x7fbf9406f3f0 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf92800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39204/0 (socket says 172.21.15.28:39204) 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf92800640 1 -- 172.21.15.28:0/926697986 learned_addr learned my addr 172.21.15.28:0/926697986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf91e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf9406f3f0 0x7fbf9406bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.478+0000 7fbf89400640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 0x7fbf94152730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 -- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf9406f3f0 msgr2=0x7fbf9406bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf9406f3f0 0x7fbf9406bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 -- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 msgr2=0x7fbf94152730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 0x7fbf94152730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 -- 172.21.15.28:0/926697986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf7c0094d0 con 0x7fbf94159770 2024-09-13T17:14:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf89400640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 0x7fbf94152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf92800640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009e30 tx=0x7fbf8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c009ca0 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c00f040 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9406c2b0 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0146c0 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.479+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf941546b0 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.480+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf94154bd0 con 0x7fbf94159770 2024-09-13T17:14:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.481+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf8c014820 con 0x7fbf94159770 2024-09-13T17:14:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.482+0000 7fbf8aa00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 0x7fbf7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.482+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf8c08a1d0 con 0x7fbf94159770 2024-09-13T17:14:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.483+0000 7fbf91e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 0x7fbf7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.483+0000 7fbf91e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 0x7fbf7006d0c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009a20 tx=0x7fbf84009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.486+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf94154bd0 con 0x7fbf94159770 2024-09-13T17:14:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:32.704+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbf94154bd0 con 0x7fbf94159770 2024-09-13T17:14:33.156 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-13T17:14:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.542+0000 7fbf8aa00640 1 -- 172.21.15.28:0/926697986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v93) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbf94154bd0 con 0x7fbf94159770 2024-09-13T17:14:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 msgr2=0x7fbf7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 0x7fbf7006d0c0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fbf84009a20 tx=0x7fbf84009190 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 msgr2=0x7fbf9414df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009e30 tx=0x7fbf8c00c490 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 shutdown_connections 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf7006ac00 0x7fbf7006d0c0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf94159770 0x7fbf9414df50 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf940add00 0x7fbf94152730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 --2- 172.21.15.28:0/926697986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf9406f3f0 0x7fbf9406bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 >> 172.21.15.28:0/926697986 conn(0x7fbf940b0270 msgr2=0x7fbf94156ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 shutdown_connections 2024-09-13T17:14:33.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.545+0000 7fbf98e00640 1 -- 172.21.15.28:0/926697986 wait complete. 2024-09-13T17:14:33.566 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:14:33.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:14:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.727+0000 7f3a69600640 1 Processor -- start 2024-09-13T17:14:33.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.727+0000 7f3a69600640 1 -- start start 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64070e30 0x7f3a64071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 0x7f3a640cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a64071210 0x7f3a640cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a64159e10 con 0x7f3a64071210 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a6415a0f0 con 0x7f3a64072140 2024-09-13T17:14:33.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a69600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6415a3d0 con 0x7f3a64070e30 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a64071210 0x7f3a640cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a64071210 0x7f3a640cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39208/0 (socket says 172.21.15.28:39208) 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 -- 172.21.15.28:0/4025555522 learned_addr learned my addr 172.21.15.28:0/4025555522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a63e00640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64070e30 0x7f3a64071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a5b400640 1 --2- 172.21.15.28:0/4025555522 >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 0x7f3a640cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 -- 172.21.15.28:0/4025555522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64070e30 msgr2=0x7f3a64071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64070e30 0x7f3a64071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 -- 172.21.15.28:0/4025555522 >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 msgr2=0x7f3a640cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.729+0000 7f3a68c00640 1 --2- 172.21.15.28:0/4025555522 >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 0x7f3a640cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.730+0000 7f3a68c00640 1 -- 172.21.15.28:0/4025555522 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a6415a6b0 con 0x7f3a64071210 2024-09-13T17:14:33.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.730+0000 7f3a5b400640 1 --2- 172.21.15.28:0/4025555522 >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 0x7f3a640cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a68c00640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64071210 0x7f3a640cddb0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3a6406aa10 tx=0x7f3a5c02c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a4e58e0524c97b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:33.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a63400640 1 -- 172.21.15.28:0/4025555522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c02f070 con 0x7f3a64071210 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a63400640 1 -- 172.21.15.28:0/4025555522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5c00b2e0 con 0x7f3a64071210 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a63400640 1 -- 172.21.15.28:0/4025555522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c009820 con 0x7f3a64071210 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 -- 172.21.15.28:0/4025555522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64071210 msgr2=0x7f3a640cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64071210 0x7f3a640cddb0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f3a6406aa10 tx=0x7f3a5c02c8a0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 -- 172.21.15.28:0/4025555522 shutdown_connections 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64071210 0x7f3a640cddb0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 --2- 172.21.15.28:0/4025555522 >> v2:172.21.15.189:3300/0 conn(0x7f3a64072140 0x7f3a640cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 --2- 172.21.15.28:0/4025555522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64070e30 0x7f3a64071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.731+0000 7f3a69600640 1 -- 172.21.15.28:0/4025555522 >> 172.21.15.28:0/4025555522 conn(0x7f3a64069e80 msgr2=0x7f3a6406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.732+0000 7f3a69600640 1 -- 172.21.15.28:0/4025555522 shutdown_connections 2024-09-13T17:14:33.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.732+0000 7f3a69600640 1 -- 172.21.15.28:0/4025555522 wait complete. 2024-09-13T17:14:33.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.733+0000 7f3a69600640 1 Processor -- start 2024-09-13T17:14:33.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.733+0000 7f3a69600640 1 -- start start 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a64071210 0x7f3a6414c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64072140 0x7f3a6414d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a64143e40 con 0x7f3a64070e30 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6414b610 con 0x7f3a64071210 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.734+0000 7f3a69600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a6414b8f0 con 0x7f3a64072140 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39216/0 (socket says 172.21.15.28:39216) 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 -- 172.21.15.28:0/364991894 learned_addr learned my addr 172.21.15.28:0/364991894 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a68c00640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64072140 0x7f3a6414d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a5b400640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a64071210 0x7f3a6414c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 -- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64072140 msgr2=0x7f3a6414d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64072140 0x7f3a6414d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 -- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a64071210 msgr2=0x7f3a6414c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:33.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a64071210 0x7f3a6414c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:33.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 -- 172.21.15.28:0/364991894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c00bd40 con 0x7f3a64070e30 2024-09-13T17:14:33.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.735+0000 7f3a63e00640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400d800 tx=0x7f3a5400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.736+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a540096a0 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.736+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a54009800 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.736+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54009b90 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.736+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a6414bbd0 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.736+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a641442b0 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.737+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a640cd080 con 0x7f3a64070e30 2024-09-13T17:14:33.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.738+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a54009cf0 con 0x7f3a64070e30 2024-09-13T17:14:33.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.739+0000 7f3a60c00640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 0x7f3a4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:33.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.739+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a540899b0 con 0x7f3a64070e30 2024-09-13T17:14:33.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.740+0000 7f3a5b400640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 0x7f3a4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:33.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.740+0000 7f3a5b400640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 0x7f3a4006d0c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a641501b0 tx=0x7f3a4c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:33.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.743+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3a54054ba0 con 0x7f3a64070e30 2024-09-13T17:14:33.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:33.956+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f3a6406f720 con 0x7f3a64070e30 2024-09-13T17:14:34.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.544+0000 7f3a60c00640 1 -- 172.21.15.28:0/364991894 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v94) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3a5400b440 con 0x7f3a64070e30 2024-09-13T17:14:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 msgr2=0x7f3a4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 0x7f3a4006d0c0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f3a641501b0 tx=0x7f3a4c008040 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 msgr2=0x7f3a64143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400d800 tx=0x7f3a5400dcc0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 shutdown_connections 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a4006ac00 0x7f3a4006d0c0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a64072140 0x7f3a6414d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a64071210 0x7f3a6414c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 --2- 172.21.15.28:0/364991894 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a64070e30 0x7f3a64143900 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 >> 172.21.15.28:0/364991894 conn(0x7f3a64069e80 msgr2=0x7f3a6415b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.546+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 shutdown_connections 2024-09-13T17:14:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.547+0000 7f3a69600640 1 -- 172.21.15.28:0/364991894 wait complete. 2024-09-13T17:14:34.565 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:14:34.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:34.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.719+0000 7f33ca200640 1 Processor -- start 2024-09-13T17:14:34.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.719+0000 7f33ca200640 1 -- start start 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33ca200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33c40cabc0 0x7f33c406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33ca200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33ca200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c415e100 con 0x7f33c415a540 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33ca200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c40adb00 con 0x7f33c40cabc0 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33ca200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c4071670 con 0x7f33c4070cc0 2024-09-13T17:14:34.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33c3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33baa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f33c40cabc0 0x7f33c406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33c3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47224/0 (socket says 172.21.15.28:47224) 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.721+0000 7f33c3e00640 1 -- 172.21.15.28:0/3272450673 learned_addr learned my addr 172.21.15.28:0/3272450673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.722+0000 7f33baa00640 1 -- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 msgr2=0x7f33c4071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.722+0000 7f33baa00640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.722+0000 7f33baa00640 1 -- 172.21.15.28:0/3272450673 >> v1:172.21.15.189:6789/0 conn(0x7f33c415a540 legacy=0x7f33c415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.722+0000 7f33baa00640 1 -- 172.21.15.28:0/3272450673 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c415e3e0 con 0x7f33c40cabc0 2024-09-13T17:14:34.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.722+0000 7f33c3e00640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:14:34.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.723+0000 7f33baa00640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c40cabc0 0x7f33c406c430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0097a0 tx=0x7f33ac02a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=667458b32708366 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:34.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33c2a00640 1 -- 172.21.15.28:0/3272450673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac02ae40 con 0x7f33c40cabc0 2024-09-13T17:14:34.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33c2a00640 1 -- 172.21.15.28:0/3272450673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ac032440 con 0x7f33c40cabc0 2024-09-13T17:14:34.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33c2a00640 1 -- 172.21.15.28:0/3272450673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ac0327d0 con 0x7f33c40cabc0 2024-09-13T17:14:34.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 -- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c40cabc0 msgr2=0x7f33c406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c40cabc0 0x7f33c406c430 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f33ac0097a0 tx=0x7f33ac02a0f0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 -- 172.21.15.28:0/3272450673 shutdown_connections 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c4070cc0 0x7f33c4071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 --2- 172.21.15.28:0/3272450673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c40cabc0 0x7f33c406c430 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.724+0000 7f33ca200640 1 -- 172.21.15.28:0/3272450673 >> 172.21.15.28:0/3272450673 conn(0x7f33c4069e80 msgr2=0x7f33c406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.725+0000 7f33ca200640 1 -- 172.21.15.28:0/3272450673 shutdown_connections 2024-09-13T17:14:34.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.725+0000 7f33ca200640 1 -- 172.21.15.28:0/3272450673 wait complete. 2024-09-13T17:14:34.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.726+0000 7f33ca200640 1 Processor -- start 2024-09-13T17:14:34.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.726+0000 7f33ca200640 1 -- start start 2024-09-13T17:14:34.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c40cabc0 0x7f33c40a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c415a540 0x7f33c40a8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40a8ec0 con 0x7f33c4070cc0 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40b3030 con 0x7f33c415a540 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33ca200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c40b2790 con 0x7f33c40cabc0 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.727+0000 7f33c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40742/0 (socket says 172.21.15.28:40742) 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 -- 172.21.15.28:0/3925298993 learned_addr learned my addr 172.21.15.28:0/3925298993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33baa00640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c40cabc0 0x7f33c40a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3e00640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c415a540 0x7f33c40a8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 -- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c40cabc0 msgr2=0x7f33c40a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c40cabc0 0x7f33c40a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 -- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c415a540 msgr2=0x7f33c40a8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c415a540 0x7f33c40a8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 -- 172.21.15.28:0/3925298993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33bc009b30 con 0x7f33c4070cc0 2024-09-13T17:14:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.728+0000 7f33c3400640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f33b400c400 tx=0x7f33b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.729+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b40052d0 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.729+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b4005430 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.729+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b4003830 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.729+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ac009400 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.729+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c40b2d50 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.730+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c4071130 con 0x7f33c4070cc0 2024-09-13T17:14:34.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.731+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f33b4003990 con 0x7f33c4070cc0 2024-09-13T17:14:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.732+0000 7f33bbe00640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 0x7f33a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:34.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.732+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33b40888c0 con 0x7f33c4070cc0 2024-09-13T17:14:34.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.732+0000 7f33baa00640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 0x7f33a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:34.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.733+0000 7f33baa00640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 0x7f33a006d0c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f33ac034000 tx=0x7f33ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:34.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.736+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f33b4053ab0 con 0x7f33c4070cc0 2024-09-13T17:14:34.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.947+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33c4070130 con 0x7f33c4070cc0 2024-09-13T17:14:34.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.948+0000 7f33bbe00640 1 -- 172.21.15.28:0/3925298993 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f33b4017090 con 0x7f33c4070cc0 2024-09-13T17:14:34.950 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:34.950 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:14:34.542201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:14:34.950 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 94 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 msgr2=0x7f33a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 0x7f33a006d0c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f33ac034000 tx=0x7f33ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 msgr2=0x7f33c4141920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f33b400c400 tx=0x7f33b400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 shutdown_connections 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f33a006ac00 0x7f33a006d0c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33c415a540 0x7f33c40a8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33c40cabc0 0x7f33c40a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 --2- 172.21.15.28:0/3925298993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33c4070cc0 0x7f33c4141920 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 >> 172.21.15.28:0/3925298993 conn(0x7f33c4069e80 msgr2=0x7f33c406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 shutdown_connections 2024-09-13T17:14:34.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:34.950+0000 7f33ca200640 1 -- 172.21.15.28:0/3925298993 wait complete. 2024-09-13T17:14:34.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:14:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.122+0000 7f414da00640 1 Processor -- start 2024-09-13T17:14:35.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.123+0000 7f414da00640 1 -- start start 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f414da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4148159820 0x7f4148164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f414da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41480c3850 con 0x7f41480b0b50 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f414da00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f414815a930 con 0x7f4148159440 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f414da00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4148159d20 con 0x7f4148159820 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f413f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f41480b0b50 0x7f41480b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40018/0 (socket says 172.21.15.28:40018) 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f413f400640 1 -- 172.21.15.28:0/2337180733 learned_addr learned my addr 172.21.15.28:0/2337180733 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:35.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.124+0000 7f414d000640 1 --2- 172.21.15.28:0/2337180733 >> v2:172.21.15.28:3300/0 conn(0x7f4148159820 0x7f4148164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.125+0000 7f414d000640 1 -- 172.21.15.28:0/2337180733 >> v1:172.21.15.189:6789/0 conn(0x7f4148159440 legacy=0x7f41480b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.125+0000 7f414d000640 1 -- 172.21.15.28:0/2337180733 >> v1:172.21.15.28:6789/0 conn(0x7f41480b0b50 legacy=0x7f41480b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.125+0000 7f414d000640 1 -- 172.21.15.28:0/2337180733 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414815a000 con 0x7f4148159820 2024-09-13T17:14:35.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.125+0000 7f4147400640 1 -- 172.21.15.28:0/2337180733 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385888507 0 0) 0x7f41480c3850 con 0x7f41480b0b50 2024-09-13T17:14:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f414d000640 1 --2- 172.21.15.28:0/2337180733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159820 0x7f4148164260 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4140009d90 tx=0x7f414002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4cb4100252aa8b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f4147400640 1 -- 172.21.15.28:0/2337180733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140007740 con 0x7f4148159820 2024-09-13T17:14:35.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f4147400640 1 -- 172.21.15.28:0/2337180733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41400078a0 con 0x7f4148159820 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f4147400640 1 -- 172.21.15.28:0/2337180733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4140007c30 con 0x7f4148159820 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f414da00640 1 -- 172.21.15.28:0/2337180733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159820 msgr2=0x7f4148164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.126+0000 7f414da00640 1 --2- 172.21.15.28:0/2337180733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159820 0x7f4148164260 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f4140009d90 tx=0x7f414002a780 comp rx=0 tx=0).stop 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.127+0000 7f414da00640 1 -- 172.21.15.28:0/2337180733 shutdown_connections 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.127+0000 7f414da00640 1 --2- 172.21.15.28:0/2337180733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159820 0x7f4148164260 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.127+0000 7f414da00640 1 -- 172.21.15.28:0/2337180733 >> 172.21.15.28:0/2337180733 conn(0x7f4148072e40 msgr2=0x7f4148073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.127+0000 7f414da00640 1 -- 172.21.15.28:0/2337180733 shutdown_connections 2024-09-13T17:14:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.127+0000 7f414da00640 1 -- 172.21.15.28:0/2337180733 wait complete. 2024-09-13T17:14:35.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.128+0000 7f414da00640 1 Processor -- start 2024-09-13T17:14:35.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.128+0000 7f414da00640 1 -- start start 2024-09-13T17:14:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 0x7f414814b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4148159820 0x7f4148157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41481f2570 con 0x7f4148159440 2024-09-13T17:14:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414815b750 con 0x7f4148159820 2024-09-13T17:14:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f414da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f414815ae60 con 0x7f41480b0b50 2024-09-13T17:14:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.129+0000 7f413f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40754/0 (socket says 172.21.15.28:40754) 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 -- 172.21.15.28:0/494856911 learned_addr learned my addr 172.21.15.28:0/494856911 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f414d000640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4148159820 0x7f4148157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f4147e00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 0x7f414814b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 -- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 msgr2=0x7f414814b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 0x7f414814b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 -- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4148159820 msgr2=0x7f4148157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4148159820 0x7f4148157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 -- 172.21.15.28:0/494856911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140009a70 con 0x7f4148159440 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f4147e00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 0x7f414814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f413f400640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f413000c4a0 tx=0x7f413000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.130+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41300074e0 con 0x7f4148159440 2024-09-13T17:14:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.131+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4130007640 con 0x7f4148159440 2024-09-13T17:14:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.131+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4130004a00 con 0x7f4148159440 2024-09-13T17:14:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.131+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414815b100 con 0x7f4148159440 2024-09-13T17:14:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.131+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41480ae7f0 con 0x7f4148159440 2024-09-13T17:14:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.132+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414815b8c0 con 0x7f4148159440 2024-09-13T17:14:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.133+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4130004b60 con 0x7f4148159440 2024-09-13T17:14:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.134+0000 7f4144c00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 0x7f412406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.134+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4130088520 con 0x7f4148159440 2024-09-13T17:14:35.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.135+0000 7f4147e00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 0x7f412406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.135+0000 7f4147e00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 0x7f412406d0c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4138007fd0 tx=0x7f4138009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.138+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f41300537a0 con 0x7f4148159440 2024-09-13T17:14:35.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.319+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41480b22e0 con 0x7f4148159440 2024-09-13T17:14:35.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.320+0000 7f4144c00640 1 -- 172.21.15.28:0/494856911 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f4130056dc0 con 0x7f4148159440 2024-09-13T17:14:35.323 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:35.323 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":111,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:14:32.528886+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 msgr2=0x7f412406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 0x7f412406d0c0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4138007fd0 tx=0x7f4138009040 comp rx=0 tx=0).stop 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 msgr2=0x7f41480ae2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f413000c4a0 tx=0x7f413000c960 comp rx=0 tx=0).stop 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 shutdown_connections 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f412406ac00 0x7f412406d0c0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4148159820 0x7f4148157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4148159440 0x7f41480ae2b0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 --2- 172.21.15.28:0/494856911 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f41480b0b50 0x7f414814b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 >> 172.21.15.28:0/494856911 conn(0x7f4148072e40 msgr2=0x7f4148070530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 shutdown_connections 2024-09-13T17:14:35.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.323+0000 7f414da00640 1 -- 172.21.15.28:0/494856911 wait complete. 2024-09-13T17:14:35.342 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:14:35.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:14:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.494+0000 7f37f4c00640 1 Processor -- start 2024-09-13T17:14:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.495+0000 7f37f4c00640 1 -- start start 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37f4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37f4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37f006c3f0 0x7f37f0158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37f4c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f00b39d0 con 0x7f37f0158b20 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37f4c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f00b2dc0 con 0x7f37f006c3f0 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37f4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00b30a0 con 0x7f37f006ba70 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37f006c3f0 0x7f37f0158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37f006c3f0 0x7f37f0158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40760/0 (socket says 172.21.15.28:40760) 2024-09-13T17:14:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37efe00640 1 -- 172.21.15.28:0/2815102019 learned_addr learned my addr 172.21.15.28:0/2815102019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.496+0000 7f37e7e00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.497+0000 7f37efe00640 1 -- 172.21.15.28:0/2815102019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 msgr2=0x7f37f00afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.497+0000 7f37efe00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.497+0000 7f37efe00640 1 -- 172.21.15.28:0/2815102019 >> v1:172.21.15.189:6789/0 conn(0x7f37f0158b20 legacy=0x7f37f006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.497+0000 7f37efe00640 1 -- 172.21.15.28:0/2815102019 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f00b3380 con 0x7f37f006c3f0 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.497+0000 7f37e7e00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37efe00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f006c3f0 0x7f37f0158f00 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009e80 tx=0x7f37e802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d851782db80fb5ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37eea00640 1 -- 172.21.15.28:0/2815102019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8031600 con 0x7f37f006c3f0 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37eea00640 1 -- 172.21.15.28:0/2815102019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e802d050 con 0x7f37f006c3f0 2024-09-13T17:14:35.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37eea00640 1 -- 172.21.15.28:0/2815102019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e8031980 con 0x7f37f006c3f0 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37f4c00640 1 -- 172.21.15.28:0/2815102019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f006c3f0 msgr2=0x7f37f0158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.498+0000 7f37f4c00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f006c3f0 0x7f37f0158f00 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f37e8009e80 tx=0x7f37e802a750 comp rx=0 tx=0).stop 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 -- 172.21.15.28:0/2815102019 shutdown_connections 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f006c3f0 0x7f37f0158f00 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 --2- 172.21.15.28:0/2815102019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00afbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 -- 172.21.15.28:0/2815102019 >> 172.21.15.28:0/2815102019 conn(0x7f37f0072e40 msgr2=0x7f37f0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 -- 172.21.15.28:0/2815102019 shutdown_connections 2024-09-13T17:14:35.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.499+0000 7f37f4c00640 1 -- 172.21.15.28:0/2815102019 wait complete. 2024-09-13T17:14:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.500+0000 7f37f4c00640 1 Processor -- start 2024-09-13T17:14:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.500+0000 7f37f4c00640 1 -- start start 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f006c3f0 0x7f37f014bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f014c640 con 0x7f37f0158b20 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00af990 con 0x7f37f006c3f0 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37f4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00ac9f0 con 0x7f37f006ba70 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.501+0000 7f37efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40776/0 (socket says 172.21.15.28:40776) 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 -- 172.21.15.28:0/1694678771 learned_addr learned my addr 172.21.15.28:0/1694678771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37ef400640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37e7e00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f006c3f0 0x7f37f014bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 -- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 msgr2=0x7f37f00af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00af450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 -- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f006c3f0 msgr2=0x7f37f014bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f006c3f0 0x7f37f014bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 -- 172.21.15.28:0/1694678771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8009b30 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.502+0000 7f37efe00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f37e8034040 tx=0x7f37e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.503+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e803d860 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.503+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e803d9e0 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.503+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e803dd90 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.503+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f00accd0 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.503+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f00ad290 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.504+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f00ad480 con 0x7f37f0158b20 2024-09-13T17:14:35.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.505+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f37e803b400 con 0x7f37f0158b20 2024-09-13T17:14:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.506+0000 7f37e7400640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 0x7f37cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.506+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f37e80aa090 con 0x7f37f0158b20 2024-09-13T17:14:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.506+0000 7f37ef400640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 0x7f37cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.507+0000 7f37ef400640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 0x7f37cc06d0c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f37f01535f0 tx=0x7f37e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.510+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f37e8075310 con 0x7f37f0158b20 2024-09-13T17:14:35.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.684+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37f006c860 con 0x7f37cc06ac00 2024-09-13T17:14:35.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.693+0000 7f37e7400640 1 -- 172.21.15.28:0/1694678771 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289779 (secure 0 0 0) 0x7f37e0001d10 con 0x7f37cc06ac00 2024-09-13T17:14:35.698 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:35.700 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 msgr2=0x7f37cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 0x7f37cc06d0c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f37f01535f0 tx=0x7f37e0009040 comp rx=0 tx=0).stop 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 msgr2=0x7f37f00705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7f37e8034040 tx=0x7f37e8004a40 comp rx=0 tx=0).stop 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 shutdown_connections 2024-09-13T17:14:35.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37cc06ac00 0x7f37cc06d0c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37f0158b20 0x7f37f00705c0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f006c3f0 0x7f37f014bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 --2- 172.21.15.28:0/1694678771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37f006ba70 0x7f37f00af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 >> 172.21.15.28:0/1694678771 conn(0x7f37f0072e40 msgr2=0x7f37f0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 shutdown_connections 2024-09-13T17:14:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.697+0000 7f37f4c00640 1 -- 172.21.15.28:0/1694678771 wait complete. 2024-09-13T17:14:35.715 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-13T17:14:34.982326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":373952,"kb_used_data":11840,"kb_used_omap":12,"kb_used_meta":361971,"kb_avail":754600768,"statfs":{"total":773094113280,"available":772711186432,"internally_reserved":0,"allocated":12124160,"data_stored":5932862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":370658873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892061"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561027+0000","last_change":"2024-09-13T17:14:31.554424+0000","last_active":"2024-09-13T17:14:31.561027+0000","last_peered":"2024-09-13T17:14:31.561027+0000","last_clean":"2024-09-13T17:14:31.561027+0000","last_became_active":"2024-09-13T17:14:31.554085+0000","last_became_peered":"2024-09-13T17:14:31.554085+0000","last_unstale":"2024-09-13T17:14:31.561027+0000","last_undegraded":"2024-09-13T17:14:31.561027+0000","last_fullsized":"2024-09-13T17:14:31.561027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:26.708148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553633+0000","last_change":"2024-09-13T17:14:31.550028+0000","last_active":"2024-09-13T17:14:31.553633+0000","last_peered":"2024-09-13T17:14:31.553633+0000","last_clean":"2024-09-13T17:14:31.553633+0000","last_became_active":"2024-09-13T17:14:31.545311+0000","last_became_peered":"2024-09-13T17:14:31.545311+0000","last_unstale":"2024-09-13T17:14:31.553633+0000","last_undegraded":"2024-09-13T17:14:31.553633+0000","last_fullsized":"2024-09-13T17:14:31.553633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:59.641934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.548580+0000","last_change":"2024-09-13T17:14:31.544920+0000","last_active":"2024-09-13T17:14:31.548580+0000","last_peered":"2024-09-13T17:14:31.548580+0000","last_clean":"2024-09-13T17:14:31.548580+0000","last_became_active":"2024-09-13T17:14:31.544688+0000","last_became_peered":"2024-09-13T17:14:31.544688+0000","last_unstale":"2024-09-13T17:14:31.548580+0000","last_undegraded":"2024-09-13T17:14:31.548580+0000","last_fullsized":"2024-09-13T17:14:31.548580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:29.033773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536839+0000","last_change":"2024-09-13T17:14:31.567127+0000","last_active":"2024-09-13T17:14:32.536839+0000","last_peered":"2024-09-13T17:14:32.536839+0000","last_clean":"2024-09-13T17:14:32.536839+0000","last_became_active":"2024-09-13T17:14:31.566097+0000","last_became_peered":"2024-09-13T17:14:31.566097+0000","last_unstale":"2024-09-13T17:14:32.536839+0000","last_undegraded":"2024-09-13T17:14:32.536839+0000","last_fullsized":"2024-09-13T17:14:32.536839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:39.388937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.560773+0000","last_change":"2024-09-13T17:14:31.556891+0000","last_active":"2024-09-13T17:14:31.560773+0000","last_peered":"2024-09-13T17:14:31.560773+0000","last_clean":"2024-09-13T17:14:31.560773+0000","last_became_active":"2024-09-13T17:14:31.556642+0000","last_became_peered":"2024-09-13T17:14:31.556642+0000","last_unstale":"2024-09-13T17:14:31.560773+0000","last_undegraded":"2024-09-13T17:14:31.560773+0000","last_fullsized":"2024-09-13T17:14:31.560773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:07.826659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546944+0000","last_change":"2024-09-13T17:14:31.545197+0000","last_active":"2024-09-13T17:14:31.546944+0000","last_peered":"2024-09-13T17:14:31.546944+0000","last_clean":"2024-09-13T17:14:31.546944+0000","last_became_active":"2024-09-13T17:14:31.544897+0000","last_became_peered":"2024-09-13T17:14:31.544897+0000","last_unstale":"2024-09-13T17:14:31.546944+0000","last_undegraded":"2024-09-13T17:14:31.546944+0000","last_fullsized":"2024-09-13T17:14:31.546944+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:32.296423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550110+0000","last_change":"2024-09-13T17:14:31.543232+0000","last_active":"2024-09-13T17:14:31.550110+0000","last_peered":"2024-09-13T17:14:31.550110+0000","last_clean":"2024-09-13T17:14:31.550110+0000","last_became_active":"2024-09-13T17:14:31.542934+0000","last_became_peered":"2024-09-13T17:14:31.542934+0000","last_unstale":"2024-09-13T17:14:31.550110+0000","last_undegraded":"2024-09-13T17:14:31.550110+0000","last_fullsized":"2024-09-13T17:14:31.550110+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:07.826659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555729+0000","last_change":"2024-09-13T17:14:31.553530+0000","last_active":"2024-09-13T17:14:31.555729+0000","last_peered":"2024-09-13T17:14:31.555729+0000","last_clean":"2024-09-13T17:14:31.555729+0000","last_became_active":"2024-09-13T17:14:31.553280+0000","last_became_peered":"2024-09-13T17:14:31.553280+0000","last_unstale":"2024-09-13T17:14:31.555729+0000","last_undegraded":"2024-09-13T17:14:31.555729+0000","last_fullsized":"2024-09-13T17:14:31.555729+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:15.793328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541598+0000","last_change":"2024-09-13T17:14:31.560095+0000","last_active":"2024-09-13T17:14:32.541598+0000","last_peered":"2024-09-13T17:14:32.541598+0000","last_clean":"2024-09-13T17:14:32.541598+0000","last_became_active":"2024-09-13T17:14:31.559826+0000","last_became_peered":"2024-09-13T17:14:31.559826+0000","last_unstale":"2024-09-13T17:14:32.541598+0000","last_undegraded":"2024-09-13T17:14:32.541598+0000","last_fullsized":"2024-09-13T17:14:32.541598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:39.388937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.567848+0000","last_change":"2024-09-13T17:14:31.564723+0000","last_active":"2024-09-13T17:14:31.567848+0000","last_peered":"2024-09-13T17:14:31.567848+0000","last_clean":"2024-09-13T17:14:31.567848+0000","last_became_active":"2024-09-13T17:14:31.564423+0000","last_became_peered":"2024-09-13T17:14:31.564423+0000","last_unstale":"2024-09-13T17:14:31.567848+0000","last_undegraded":"2024-09-13T17:14:31.567848+0000","last_fullsized":"2024-09-13T17:14:31.567848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.910100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.557637+0000","last_change":"2024-09-13T17:14:31.553009+0000","last_active":"2024-09-13T17:14:31.557637+0000","last_peered":"2024-09-13T17:14:31.557637+0000","last_clean":"2024-09-13T17:14:31.557637+0000","last_became_active":"2024-09-13T17:14:31.552493+0000","last_became_peered":"2024-09-13T17:14:31.552493+0000","last_unstale":"2024-09-13T17:14:31.557637+0000","last_undegraded":"2024-09-13T17:14:31.557637+0000","last_fullsized":"2024-09-13T17:14:31.557637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:16.523848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541470+0000","last_change":"2024-09-13T17:14:31.555997+0000","last_active":"2024-09-13T17:14:32.541470+0000","last_peered":"2024-09-13T17:14:32.541470+0000","last_clean":"2024-09-13T17:14:32.541470+0000","last_became_active":"2024-09-13T17:14:31.555707+0000","last_became_peered":"2024-09-13T17:14:31.555707+0000","last_unstale":"2024-09-13T17:14:32.541470+0000","last_undegraded":"2024-09-13T17:14:32.541470+0000","last_fullsized":"2024-09-13T17:14:32.541470+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536596+0000","last_change":"2024-09-13T17:14:31.554798+0000","last_active":"2024-09-13T17:14:32.536596+0000","last_peered":"2024-09-13T17:14:32.536596+0000","last_clean":"2024-09-13T17:14:32.536596+0000","last_became_active":"2024-09-13T17:14:31.554427+0000","last_became_peered":"2024-09-13T17:14:31.554427+0000","last_unstale":"2024-09-13T17:14:32.536596+0000","last_undegraded":"2024-09-13T17:14:32.536596+0000","last_fullsized":"2024-09-13T17:14:32.536596+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565769+0000","last_change":"2024-09-13T17:14:31.564011+0000","last_active":"2024-09-13T17:14:31.565769+0000","last_peered":"2024-09-13T17:14:31.565769+0000","last_clean":"2024-09-13T17:14:31.565769+0000","last_became_active":"2024-09-13T17:14:31.558339+0000","last_became_peered":"2024-09-13T17:14:31.558339+0000","last_unstale":"2024-09-13T17:14:31.565769+0000","last_undegraded":"2024-09-13T17:14:31.565769+0000","last_fullsized":"2024-09-13T17:14:31.565769+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:10.995098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536471+0000","last_change":"2024-09-13T17:14:31.557150+0000","last_active":"2024-09-13T17:14:32.536471+0000","last_peered":"2024-09-13T17:14:32.536471+0000","last_clean":"2024-09-13T17:14:32.536471+0000","last_became_active":"2024-09-13T17:14:31.554479+0000","last_became_peered":"2024-09-13T17:14:31.554479+0000","last_unstale":"2024-09-13T17:14:32.536471+0000","last_undegraded":"2024-09-13T17:14:32.536471+0000","last_fullsized":"2024-09-13T17:14:32.536471+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550238+0000","last_change":"2024-09-13T17:14:31.548205+0000","last_active":"2024-09-13T17:14:31.550238+0000","last_peered":"2024-09-13T17:14:31.550238+0000","last_clean":"2024-09-13T17:14:31.550238+0000","last_became_active":"2024-09-13T17:14:31.547780+0000","last_became_peered":"2024-09-13T17:14:31.547780+0000","last_unstale":"2024-09-13T17:14:31.550238+0000","last_undegraded":"2024-09-13T17:14:31.550238+0000","last_fullsized":"2024-09-13T17:14:31.550238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.910100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563137+0000","last_change":"2024-09-13T17:14:31.562103+0000","last_active":"2024-09-13T17:14:31.563137+0000","last_peered":"2024-09-13T17:14:31.563137+0000","last_clean":"2024-09-13T17:14:31.563137+0000","last_became_active":"2024-09-13T17:14:31.560557+0000","last_became_peered":"2024-09-13T17:14:31.560557+0000","last_unstale":"2024-09-13T17:14:31.563137+0000","last_undegraded":"2024-09-13T17:14:31.563137+0000","last_fullsized":"2024-09-13T17:14:31.563137+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:26.090024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.537580+0000","last_change":"2024-09-13T17:14:31.566397+0000","last_active":"2024-09-13T17:14:32.537580+0000","last_peered":"2024-09-13T17:14:32.537580+0000","last_clean":"2024-09-13T17:14:32.537580+0000","last_became_active":"2024-09-13T17:14:31.566114+0000","last_became_peered":"2024-09-13T17:14:31.566114+0000","last_unstale":"2024-09-13T17:14:32.537580+0000","last_undegraded":"2024-09-13T17:14:32.537580+0000","last_fullsized":"2024-09-13T17:14:32.537580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:59.045988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563756+0000","last_change":"2024-09-13T17:14:31.547194+0000","last_active":"2024-09-13T17:14:31.563756+0000","last_peered":"2024-09-13T17:14:31.563756+0000","last_clean":"2024-09-13T17:14:31.563756+0000","last_became_active":"2024-09-13T17:14:31.546940+0000","last_became_peered":"2024-09-13T17:14:31.546940+0000","last_unstale":"2024-09-13T17:14:31.563756+0000","last_undegraded":"2024-09-13T17:14:31.563756+0000","last_fullsized":"2024-09-13T17:14:31.563756+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:19.702514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541528+0000","last_change":"2024-09-13T17:14:31.560456+0000","last_active":"2024-09-13T17:14:32.541528+0000","last_peered":"2024-09-13T17:14:32.541528+0000","last_clean":"2024-09-13T17:14:32.541528+0000","last_became_active":"2024-09-13T17:14:31.560088+0000","last_became_peered":"2024-09-13T17:14:31.560088+0000","last_unstale":"2024-09-13T17:14:32.541528+0000","last_undegraded":"2024-09-13T17:14:32.541528+0000","last_fullsized":"2024-09-13T17:14:32.541528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:11.549156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.545008+0000","last_change":"2024-09-13T17:14:31.543284+0000","last_active":"2024-09-13T17:14:31.545008+0000","last_peered":"2024-09-13T17:14:31.545008+0000","last_clean":"2024-09-13T17:14:31.545008+0000","last_became_active":"2024-09-13T17:14:31.542952+0000","last_became_peered":"2024-09-13T17:14:31.542952+0000","last_unstale":"2024-09-13T17:14:31.545008+0000","last_undegraded":"2024-09-13T17:14:31.545008+0000","last_fullsized":"2024-09-13T17:14:31.545008+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:24.213895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553337+0000","last_change":"2024-09-13T17:14:31.549440+0000","last_active":"2024-09-13T17:14:31.553337+0000","last_peered":"2024-09-13T17:14:31.553337+0000","last_clean":"2024-09-13T17:14:31.553337+0000","last_became_active":"2024-09-13T17:14:31.549202+0000","last_became_peered":"2024-09-13T17:14:31.549202+0000","last_unstale":"2024-09-13T17:14:31.553337+0000","last_undegraded":"2024-09-13T17:14:31.553337+0000","last_fullsized":"2024-09-13T17:14:31.553337+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:37.773761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546513+0000","last_change":"2024-09-13T17:14:31.545065+0000","last_active":"2024-09-13T17:14:31.546513+0000","last_peered":"2024-09-13T17:14:31.546513+0000","last_clean":"2024-09-13T17:14:31.546513+0000","last_became_active":"2024-09-13T17:14:31.544773+0000","last_became_peered":"2024-09-13T17:14:31.544773+0000","last_unstale":"2024-09-13T17:14:31.546513+0000","last_undegraded":"2024-09-13T17:14:31.546513+0000","last_fullsized":"2024-09-13T17:14:31.546513+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:59.646095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550217+0000","last_change":"2024-09-13T17:14:31.548038+0000","last_active":"2024-09-13T17:14:31.550217+0000","last_peered":"2024-09-13T17:14:31.550217+0000","last_clean":"2024-09-13T17:14:31.550217+0000","last_became_active":"2024-09-13T17:14:31.547461+0000","last_became_peered":"2024-09-13T17:14:31.547461+0000","last_unstale":"2024-09-13T17:14:31.550217+0000","last_undegraded":"2024-09-13T17:14:31.550217+0000","last_fullsized":"2024-09-13T17:14:31.550217+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:40.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561966+0000","last_change":"2024-09-13T17:14:31.556084+0000","last_active":"2024-09-13T17:14:31.561966+0000","last_peered":"2024-09-13T17:14:31.561966+0000","last_clean":"2024-09-13T17:14:31.561966+0000","last_became_active":"2024-09-13T17:14:31.554457+0000","last_became_peered":"2024-09-13T17:14:31.554457+0000","last_unstale":"2024-09-13T17:14:31.561966+0000","last_undegraded":"2024-09-13T17:14:31.561966+0000","last_fullsized":"2024-09-13T17:14:31.561966+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.866955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553612+0000","last_change":"2024-09-13T17:14:31.548061+0000","last_active":"2024-09-13T17:14:31.553612+0000","last_peered":"2024-09-13T17:14:31.553612+0000","last_clean":"2024-09-13T17:14:31.553612+0000","last_became_active":"2024-09-13T17:14:31.547834+0000","last_became_peered":"2024-09-13T17:14:31.547834+0000","last_unstale":"2024-09-13T17:14:31.553612+0000","last_undegraded":"2024-09-13T17:14:31.553612+0000","last_fullsized":"2024-09-13T17:14:31.553612+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:18.000332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549712+0000","last_change":"2024-09-13T17:14:31.544968+0000","last_active":"2024-09-13T17:14:31.549712+0000","last_peered":"2024-09-13T17:14:31.549712+0000","last_clean":"2024-09-13T17:14:31.549712+0000","last_became_active":"2024-09-13T17:14:31.544692+0000","last_became_peered":"2024-09-13T17:14:31.544692+0000","last_unstale":"2024-09-13T17:14:31.549712+0000","last_undegraded":"2024-09-13T17:14:31.549712+0000","last_fullsized":"2024-09-13T17:14:31.549712+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:36.617587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565179+0000","last_change":"2024-09-13T17:14:31.561345+0000","last_active":"2024-09-13T17:14:31.565179+0000","last_peered":"2024-09-13T17:14:31.565179+0000","last_clean":"2024-09-13T17:14:31.565179+0000","last_became_active":"2024-09-13T17:14:31.561065+0000","last_became_peered":"2024-09-13T17:14:31.561065+0000","last_unstale":"2024-09-13T17:14:31.565179+0000","last_undegraded":"2024-09-13T17:14:31.565179+0000","last_fullsized":"2024-09-13T17:14:31.565179+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:32.296423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553281+0000","last_change":"2024-09-13T17:14:31.547279+0000","last_active":"2024-09-13T17:14:31.553281+0000","last_peered":"2024-09-13T17:14:31.553281+0000","last_clean":"2024-09-13T17:14:31.553281+0000","last_became_active":"2024-09-13T17:14:31.547020+0000","last_became_peered":"2024-09-13T17:14:31.547020+0000","last_unstale":"2024-09-13T17:14:31.553281+0000","last_undegraded":"2024-09-13T17:14:31.553281+0000","last_fullsized":"2024-09-13T17:14:31.553281+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563732+0000","last_change":"2024-09-13T17:14:31.548647+0000","last_active":"2024-09-13T17:14:31.563732+0000","last_peered":"2024-09-13T17:14:31.563732+0000","last_clean":"2024-09-13T17:14:31.563732+0000","last_became_active":"2024-09-13T17:14:31.547941+0000","last_became_peered":"2024-09-13T17:14:31.547941+0000","last_unstale":"2024-09-13T17:14:31.563732+0000","last_undegraded":"2024-09-13T17:14:31.563732+0000","last_fullsized":"2024-09-13T17:14:31.563732+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:43.431963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536634+0000","last_change":"2024-09-13T17:14:31.559545+0000","last_active":"2024-09-13T17:14:32.536634+0000","last_peered":"2024-09-13T17:14:32.536634+0000","last_clean":"2024-09-13T17:14:32.536634+0000","last_became_active":"2024-09-13T17:14:31.559058+0000","last_became_peered":"2024-09-13T17:14:31.559058+0000","last_unstale":"2024-09-13T17:14:32.536634+0000","last_undegraded":"2024-09-13T17:14:32.536634+0000","last_fullsized":"2024-09-13T17:14:32.536634+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:17.504567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564231+0000","last_change":"2024-09-13T17:14:31.560844+0000","last_active":"2024-09-13T17:14:31.564231+0000","last_peered":"2024-09-13T17:14:31.564231+0000","last_clean":"2024-09-13T17:14:31.564231+0000","last_became_active":"2024-09-13T17:14:31.556568+0000","last_became_peered":"2024-09-13T17:14:31.556568+0000","last_unstale":"2024-09-13T17:14:31.564231+0000","last_undegraded":"2024-09-13T17:14:31.564231+0000","last_fullsized":"2024-09-13T17:14:31.564231+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:07.892038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.544068+0000","last_change":"2024-09-13T17:14:31.541796+0000","last_active":"2024-09-13T17:14:31.544068+0000","last_peered":"2024-09-13T17:14:31.544068+0000","last_clean":"2024-09-13T17:14:31.544068+0000","last_became_active":"2024-09-13T17:14:31.541535+0000","last_became_peered":"2024-09-13T17:14:31.541535+0000","last_unstale":"2024-09-13T17:14:31.544068+0000","last_undegraded":"2024-09-13T17:14:31.544068+0000","last_fullsized":"2024-09-13T17:14:31.544068+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:13.983156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"110'4","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.554721+0000","last_change":"2024-09-13T17:14:31.118456+0000","last_active":"2024-09-13T17:14:31.554721+0000","last_peered":"2024-09-13T17:14:31.554721+0000","last_clean":"2024-09-13T17:14:31.554721+0000","last_became_active":"2024-09-13T17:14:31.118201+0000","last_became_peered":"2024-09-13T17:14:31.118201+0000","last_unstale":"2024-09-13T17:14:31.554721+0000","last_undegraded":"2024-09-13T17:14:31.554721+0000","last_fullsized":"2024-09-13T17:14:31.554721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:06.824037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536872+0000","last_change":"2024-09-13T17:14:31.559003+0000","last_active":"2024-09-13T17:14:32.536872+0000","last_peered":"2024-09-13T17:14:32.536872+0000","last_clean":"2024-09-13T17:14:32.536872+0000","last_became_active":"2024-09-13T17:14:31.558693+0000","last_became_peered":"2024-09-13T17:14:31.558693+0000","last_unstale":"2024-09-13T17:14:32.536872+0000","last_undegraded":"2024-09-13T17:14:32.536872+0000","last_fullsized":"2024-09-13T17:14:32.536872+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:11.549156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552723+0000","last_change":"2024-09-13T17:14:31.122718+0000","last_active":"2024-09-13T17:14:31.552723+0000","last_peered":"2024-09-13T17:14:31.552723+0000","last_clean":"2024-09-13T17:14:31.552723+0000","last_became_active":"2024-09-13T17:14:31.122312+0000","last_became_peered":"2024-09-13T17:14:31.122312+0000","last_unstale":"2024-09-13T17:14:31.552723+0000","last_undegraded":"2024-09-13T17:14:31.552723+0000","last_fullsized":"2024-09-13T17:14:31.552723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:38.621021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561156+0000","last_change":"2024-09-13T17:14:31.552961+0000","last_active":"2024-09-13T17:14:31.561156+0000","last_peered":"2024-09-13T17:14:31.561156+0000","last_clean":"2024-09-13T17:14:31.561156+0000","last_became_active":"2024-09-13T17:14:31.552704+0000","last_became_peered":"2024-09-13T17:14:31.552704+0000","last_unstale":"2024-09-13T17:14:31.561156+0000","last_undegraded":"2024-09-13T17:14:31.561156+0000","last_fullsized":"2024-09-13T17:14:31.561156+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:19.702514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555597+0000","last_change":"2024-09-13T17:14:31.114818+0000","last_active":"2024-09-13T17:14:31.555597+0000","last_peered":"2024-09-13T17:14:31.555597+0000","last_clean":"2024-09-13T17:14:31.555597+0000","last_became_active":"2024-09-13T17:14:31.114530+0000","last_became_peered":"2024-09-13T17:14:31.114530+0000","last_unstale":"2024-09-13T17:14:31.555597+0000","last_undegraded":"2024-09-13T17:14:31.555597+0000","last_fullsized":"2024-09-13T17:14:31.555597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:59.436683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541406+0000","last_change":"2024-09-13T17:14:31.550576+0000","last_active":"2024-09-13T17:14:32.541406+0000","last_peered":"2024-09-13T17:14:32.541406+0000","last_clean":"2024-09-13T17:14:32.541406+0000","last_became_active":"2024-09-13T17:14:31.549293+0000","last_became_peered":"2024-09-13T17:14:31.549293+0000","last_unstale":"2024-09-13T17:14:32.541406+0000","last_undegraded":"2024-09-13T17:14:32.541406+0000","last_fullsized":"2024-09-13T17:14:32.541406+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:15.361955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552370+0000","last_change":"2024-09-13T17:14:31.115204+0000","last_active":"2024-09-13T17:14:31.552370+0000","last_peered":"2024-09-13T17:14:31.552370+0000","last_clean":"2024-09-13T17:14:31.552370+0000","last_became_active":"2024-09-13T17:14:31.112326+0000","last_became_peered":"2024-09-13T17:14:31.112326+0000","last_unstale":"2024-09-13T17:14:31.552370+0000","last_undegraded":"2024-09-13T17:14:31.552370+0000","last_fullsized":"2024-09-13T17:14:31.552370+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:16.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541162+0000","last_change":"2024-09-13T17:14:31.554469+0000","last_active":"2024-09-13T17:14:32.541162+0000","last_peered":"2024-09-13T17:14:32.541162+0000","last_clean":"2024-09-13T17:14:32.541162+0000","last_became_active":"2024-09-13T17:14:31.552118+0000","last_became_peered":"2024-09-13T17:14:31.552118+0000","last_unstale":"2024-09-13T17:14:32.541162+0000","last_undegraded":"2024-09-13T17:14:32.541162+0000","last_fullsized":"2024-09-13T17:14:32.541162+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:07.493248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561417+0000","last_change":"2024-09-13T17:14:31.121466+0000","last_active":"2024-09-13T17:14:31.561417+0000","last_peered":"2024-09-13T17:14:31.561417+0000","last_clean":"2024-09-13T17:14:31.561417+0000","last_became_active":"2024-09-13T17:14:31.121160+0000","last_became_peered":"2024-09-13T17:14:31.121160+0000","last_unstale":"2024-09-13T17:14:31.561417+0000","last_undegraded":"2024-09-13T17:14:31.561417+0000","last_fullsized":"2024-09-13T17:14:31.561417+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:02.935337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.557269+0000","last_change":"2024-09-13T17:14:31.555644+0000","last_active":"2024-09-13T17:14:31.557269+0000","last_peered":"2024-09-13T17:14:31.557269+0000","last_clean":"2024-09-13T17:14:31.557269+0000","last_became_active":"2024-09-13T17:14:31.555398+0000","last_became_peered":"2024-09-13T17:14:31.555398+0000","last_unstale":"2024-09-13T17:14:31.557269+0000","last_undegraded":"2024-09-13T17:14:31.557269+0000","last_fullsized":"2024-09-13T17:14:31.557269+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.866955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.551398+0000","last_change":"2024-09-13T17:14:31.123721+0000","last_active":"2024-09-13T17:14:31.551398+0000","last_peered":"2024-09-13T17:14:31.551398+0000","last_clean":"2024-09-13T17:14:31.551398+0000","last_became_active":"2024-09-13T17:14:31.123334+0000","last_became_peered":"2024-09-13T17:14:31.123334+0000","last_unstale":"2024-09-13T17:14:31.551398+0000","last_undegraded":"2024-09-13T17:14:31.551398+0000","last_fullsized":"2024-09-13T17:14:31.551398+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:15.591329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536451+0000","last_change":"2024-09-13T17:14:31.555508+0000","last_active":"2024-09-13T17:14:32.536451+0000","last_peered":"2024-09-13T17:14:32.536451+0000","last_clean":"2024-09-13T17:14:32.536451+0000","last_became_active":"2024-09-13T17:14:31.555217+0000","last_became_peered":"2024-09-13T17:14:31.555217+0000","last_unstale":"2024-09-13T17:14:32.536451+0000","last_undegraded":"2024-09-13T17:14:32.536451+0000","last_fullsized":"2024-09-13T17:14:32.536451+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:15.793328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550630+0000","last_change":"2024-09-13T17:14:31.112196+0000","last_active":"2024-09-13T17:14:31.550630+0000","last_peered":"2024-09-13T17:14:31.550630+0000","last_clean":"2024-09-13T17:14:31.550630+0000","last_became_active":"2024-09-13T17:14:31.111867+0000","last_became_peered":"2024-09-13T17:14:31.111867+0000","last_unstale":"2024-09-13T17:14:31.550630+0000","last_undegraded":"2024-09-13T17:14:31.550630+0000","last_fullsized":"2024-09-13T17:14:31.550630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:38.621021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541720+0000","last_change":"2024-09-13T17:14:31.556238+0000","last_active":"2024-09-13T17:14:32.541720+0000","last_peered":"2024-09-13T17:14:32.541720+0000","last_clean":"2024-09-13T17:14:32.541720+0000","last_became_active":"2024-09-13T17:14:31.555892+0000","last_became_peered":"2024-09-13T17:14:31.555892+0000","last_unstale":"2024-09-13T17:14:32.541720+0000","last_undegraded":"2024-09-13T17:14:32.541720+0000","last_fullsized":"2024-09-13T17:14:32.541720+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.539453+0000","last_change":"2024-09-13T17:14:31.122267+0000","last_active":"2024-09-13T17:14:31.539453+0000","last_peered":"2024-09-13T17:14:31.539453+0000","last_clean":"2024-09-13T17:14:31.539453+0000","last_became_active":"2024-09-13T17:14:31.122018+0000","last_became_peered":"2024-09-13T17:14:31.122018+0000","last_unstale":"2024-09-13T17:14:31.539453+0000","last_undegraded":"2024-09-13T17:14:31.539453+0000","last_fullsized":"2024-09-13T17:14:31.539453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:41.562217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555699+0000","last_change":"2024-09-13T17:14:31.553095+0000","last_active":"2024-09-13T17:14:31.555699+0000","last_peered":"2024-09-13T17:14:31.555699+0000","last_clean":"2024-09-13T17:14:31.555699+0000","last_became_active":"2024-09-13T17:14:31.552671+0000","last_became_peered":"2024-09-13T17:14:31.552671+0000","last_unstale":"2024-09-13T17:14:31.555699+0000","last_undegraded":"2024-09-13T17:14:31.555699+0000","last_fullsized":"2024-09-13T17:14:31.555699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:14.140980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.543026+0000","last_change":"2024-09-13T17:14:31.115979+0000","last_active":"2024-09-13T17:14:31.543026+0000","last_peered":"2024-09-13T17:14:31.543026+0000","last_clean":"2024-09-13T17:14:31.543026+0000","last_became_active":"2024-09-13T17:14:31.115730+0000","last_became_peered":"2024-09-13T17:14:31.115730+0000","last_unstale":"2024-09-13T17:14:31.543026+0000","last_undegraded":"2024-09-13T17:14:31.543026+0000","last_fullsized":"2024-09-13T17:14:31.543026+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:17.192176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541641+0000","last_change":"2024-09-13T17:14:31.556403+0000","last_active":"2024-09-13T17:14:32.541641+0000","last_peered":"2024-09-13T17:14:32.541641+0000","last_clean":"2024-09-13T17:14:32.541641+0000","last_became_active":"2024-09-13T17:14:31.556104+0000","last_became_peered":"2024-09-13T17:14:31.556104+0000","last_unstale":"2024-09-13T17:14:32.541641+0000","last_undegraded":"2024-09-13T17:14:32.541641+0000","last_fullsized":"2024-09-13T17:14:32.541641+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:59.045988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.537556+0000","last_change":"2024-09-13T17:14:31.121706+0000","last_active":"2024-09-13T17:14:32.537556+0000","last_peered":"2024-09-13T17:14:32.537556+0000","last_clean":"2024-09-13T17:14:32.537556+0000","last_became_active":"2024-09-13T17:14:31.121396+0000","last_became_peered":"2024-09-13T17:14:31.121396+0000","last_unstale":"2024-09-13T17:14:32.537556+0000","last_undegraded":"2024-09-13T17:14:32.537556+0000","last_fullsized":"2024-09-13T17:14:32.537556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:31.293801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561669+0000","last_change":"2024-09-13T17:14:31.555509+0000","last_active":"2024-09-13T17:14:31.561669+0000","last_peered":"2024-09-13T17:14:31.561669+0000","last_clean":"2024-09-13T17:14:31.561669+0000","last_became_active":"2024-09-13T17:14:31.555152+0000","last_became_peered":"2024-09-13T17:14:31.555152+0000","last_unstale":"2024-09-13T17:14:31.561669+0000","last_undegraded":"2024-09-13T17:14:31.561669+0000","last_fullsized":"2024-09-13T17:14:31.561669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:17.202561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"110'3","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552852+0000","last_change":"2024-09-13T17:14:31.114295+0000","last_active":"2024-09-13T17:14:31.552852+0000","last_peered":"2024-09-13T17:14:31.552852+0000","last_clean":"2024-09-13T17:14:31.552852+0000","last_became_active":"2024-09-13T17:14:31.113507+0000","last_became_peered":"2024-09-13T17:14:31.113507+0000","last_unstale":"2024-09-13T17:14:31.552852+0000","last_undegraded":"2024-09-13T17:14:31.552852+0000","last_fullsized":"2024-09-13T17:14:31.552852+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:11.627636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549934+0000","last_change":"2024-09-13T17:14:31.548810+0000","last_active":"2024-09-13T17:14:31.549934+0000","last_peered":"2024-09-13T17:14:31.549934+0000","last_clean":"2024-09-13T17:14:31.549934+0000","last_became_active":"2024-09-13T17:14:31.547135+0000","last_became_peered":"2024-09-13T17:14:31.547135+0000","last_unstale":"2024-09-13T17:14:31.549934+0000","last_undegraded":"2024-09-13T17:14:31.549934+0000","last_fullsized":"2024-09-13T17:14:31.549934+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:17.202561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"110'6","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541559+0000","last_change":"2024-09-13T17:14:31.109566+0000","last_active":"2024-09-13T17:14:32.541559+0000","last_peered":"2024-09-13T17:14:32.541559+0000","last_clean":"2024-09-13T17:14:32.541559+0000","last_became_active":"2024-09-13T17:14:31.109298+0000","last_became_peered":"2024-09-13T17:14:31.109298+0000","last_unstale":"2024-09-13T17:14:32.541559+0000","last_undegraded":"2024-09-13T17:14:32.541559+0000","last_fullsized":"2024-09-13T17:14:32.541559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:06.824037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536633+0000","last_change":"2024-09-13T17:14:31.550987+0000","last_active":"2024-09-13T17:14:32.536633+0000","last_peered":"2024-09-13T17:14:32.536633+0000","last_clean":"2024-09-13T17:14:32.536633+0000","last_became_active":"2024-09-13T17:14:31.548999+0000","last_became_peered":"2024-09-13T17:14:31.548999+0000","last_unstale":"2024-09-13T17:14:32.536633+0000","last_undegraded":"2024-09-13T17:14:32.536633+0000","last_fullsized":"2024-09-13T17:14:32.536633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:15.361955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564946+0000","last_change":"2024-09-13T17:14:31.104085+0000","last_active":"2024-09-13T17:14:31.564946+0000","last_peered":"2024-09-13T17:14:31.564946+0000","last_clean":"2024-09-13T17:14:31.564946+0000","last_became_active":"2024-09-13T17:14:31.102170+0000","last_became_peered":"2024-09-13T17:14:31.102170+0000","last_unstale":"2024-09-13T17:14:31.564946+0000","last_undegraded":"2024-09-13T17:14:31.564946+0000","last_fullsized":"2024-09-13T17:14:31.564946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:37.091849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.548791+0000","last_change":"2024-09-13T17:14:31.544359+0000","last_active":"2024-09-13T17:14:31.548791+0000","last_peered":"2024-09-13T17:14:31.548791+0000","last_clean":"2024-09-13T17:14:31.548791+0000","last_became_active":"2024-09-13T17:14:31.543530+0000","last_became_peered":"2024-09-13T17:14:31.543530+0000","last_unstale":"2024-09-13T17:14:31.548791+0000","last_undegraded":"2024-09-13T17:14:31.548791+0000","last_fullsized":"2024-09-13T17:14:31.548791+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:02.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.531892+0000","last_change":"2024-09-13T17:14:31.117012+0000","last_active":"2024-09-13T17:14:31.531892+0000","last_peered":"2024-09-13T17:14:31.531892+0000","last_clean":"2024-09-13T17:14:31.531892+0000","last_became_active":"2024-09-13T17:14:31.116735+0000","last_became_peered":"2024-09-13T17:14:31.116735+0000","last_unstale":"2024-09-13T17:14:31.531892+0000","last_undegraded":"2024-09-13T17:14:31.531892+0000","last_fullsized":"2024-09-13T17:14:31.531892+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:04.459517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540716+0000","last_change":"2024-09-13T17:14:31.555616+0000","last_active":"2024-09-13T17:14:32.540716+0000","last_peered":"2024-09-13T17:14:32.540716+0000","last_clean":"2024-09-13T17:14:32.540716+0000","last_became_active":"2024-09-13T17:14:31.555338+0000","last_became_peered":"2024-09-13T17:14:31.555338+0000","last_unstale":"2024-09-13T17:14:32.540716+0000","last_undegraded":"2024-09-13T17:14:32.540716+0000","last_fullsized":"2024-09-13T17:14:32.540716+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:17.504567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540993+0000","last_change":"2024-09-13T17:14:31.102542+0000","last_active":"2024-09-13T17:14:32.540993+0000","last_peered":"2024-09-13T17:14:32.540993+0000","last_clean":"2024-09-13T17:14:32.540993+0000","last_became_active":"2024-09-13T17:14:31.102250+0000","last_became_peered":"2024-09-13T17:14:31.102250+0000","last_unstale":"2024-09-13T17:14:32.540993+0000","last_undegraded":"2024-09-13T17:14:32.540993+0000","last_fullsized":"2024-09-13T17:14:32.540993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:53.228895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564623+0000","last_change":"2024-09-13T17:14:31.555418+0000","last_active":"2024-09-13T17:14:31.564623+0000","last_peered":"2024-09-13T17:14:31.564623+0000","last_clean":"2024-09-13T17:14:31.564623+0000","last_became_active":"2024-09-13T17:14:31.555110+0000","last_became_peered":"2024-09-13T17:14:31.555110+0000","last_unstale":"2024-09-13T17:14:31.564623+0000","last_undegraded":"2024-09-13T17:14:31.564623+0000","last_fullsized":"2024-09-13T17:14:31.564623+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:10.870738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"110'8","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.568077+0000","last_change":"2024-09-13T17:14:31.122012+0000","last_active":"2024-09-13T17:14:31.568077+0000","last_peered":"2024-09-13T17:14:31.568077+0000","last_clean":"2024-09-13T17:14:31.568077+0000","last_became_active":"2024-09-13T17:14:31.121741+0000","last_became_peered":"2024-09-13T17:14:31.121741+0000","last_unstale":"2024-09-13T17:14:31.568077+0000","last_undegraded":"2024-09-13T17:14:31.568077+0000","last_fullsized":"2024-09-13T17:14:31.568077+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:02.015241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561983+0000","last_change":"2024-09-13T17:14:31.556640+0000","last_active":"2024-09-13T17:14:31.561983+0000","last_peered":"2024-09-13T17:14:31.561983+0000","last_clean":"2024-09-13T17:14:31.561983+0000","last_became_active":"2024-09-13T17:14:31.556369+0000","last_became_peered":"2024-09-13T17:14:31.556369+0000","last_unstale":"2024-09-13T17:14:31.561983+0000","last_undegraded":"2024-09-13T17:14:31.561983+0000","last_fullsized":"2024-09-13T17:14:31.561983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:41.645450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541095+0000","last_change":"2024-09-13T17:14:31.108877+0000","last_active":"2024-09-13T17:14:32.541095+0000","last_peered":"2024-09-13T17:14:32.541095+0000","last_clean":"2024-09-13T17:14:32.541095+0000","last_became_active":"2024-09-13T17:14:31.108618+0000","last_became_peered":"2024-09-13T17:14:31.108618+0000","last_unstale":"2024-09-13T17:14:32.541095+0000","last_undegraded":"2024-09-13T17:14:32.541095+0000","last_fullsized":"2024-09-13T17:14:32.541095+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:35.614965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541002+0000","last_change":"2024-09-13T17:14:31.559173+0000","last_active":"2024-09-13T17:14:32.541002+0000","last_peered":"2024-09-13T17:14:32.541002+0000","last_clean":"2024-09-13T17:14:32.541002+0000","last_became_active":"2024-09-13T17:14:31.558893+0000","last_became_peered":"2024-09-13T17:14:31.558893+0000","last_unstale":"2024-09-13T17:14:32.541002+0000","last_undegraded":"2024-09-13T17:14:32.541002+0000","last_fullsized":"2024-09-13T17:14:32.541002+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:33.680681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.559630+0000","last_change":"2024-09-13T17:14:31.111472+0000","last_active":"2024-09-13T17:14:31.559630+0000","last_peered":"2024-09-13T17:14:31.559630+0000","last_clean":"2024-09-13T17:14:31.559630+0000","last_became_active":"2024-09-13T17:14:31.111120+0000","last_became_peered":"2024-09-13T17:14:31.111120+0000","last_unstale":"2024-09-13T17:14:31.559630+0000","last_undegraded":"2024-09-13T17:14:31.559630+0000","last_fullsized":"2024-09-13T17:14:31.559630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549917+0000","last_change":"2024-09-13T17:14:31.547183+0000","last_active":"2024-09-13T17:14:31.549917+0000","last_peered":"2024-09-13T17:14:31.549917+0000","last_clean":"2024-09-13T17:14:31.549917+0000","last_became_active":"2024-09-13T17:14:31.546907+0000","last_became_peered":"2024-09-13T17:14:31.546907+0000","last_unstale":"2024-09-13T17:14:31.549917+0000","last_undegraded":"2024-09-13T17:14:31.549917+0000","last_fullsized":"2024-09-13T17:14:31.549917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:54.231517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.535382+0000","last_change":"2024-09-13T17:14:31.113994+0000","last_active":"2024-09-13T17:14:31.535382+0000","last_peered":"2024-09-13T17:14:31.535382+0000","last_clean":"2024-09-13T17:14:31.535382+0000","last_became_active":"2024-09-13T17:14:31.113246+0000","last_became_peered":"2024-09-13T17:14:31.113246+0000","last_unstale":"2024-09-13T17:14:31.535382+0000","last_undegraded":"2024-09-13T17:14:31.535382+0000","last_fullsized":"2024-09-13T17:14:31.535382+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:31.293801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565753+0000","last_change":"2024-09-13T17:14:31.562193+0000","last_active":"2024-09-13T17:14:31.565753+0000","last_peered":"2024-09-13T17:14:31.565753+0000","last_clean":"2024-09-13T17:14:31.565753+0000","last_became_active":"2024-09-13T17:14:31.557845+0000","last_became_peered":"2024-09-13T17:14:31.557845+0000","last_unstale":"2024-09-13T17:14:31.565753+0000","last_undegraded":"2024-09-13T17:14:31.565753+0000","last_fullsized":"2024-09-13T17:14:31.565753+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.059861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546772+0000","last_change":"2024-09-13T17:14:31.114167+0000","last_active":"2024-09-13T17:14:31.546772+0000","last_peered":"2024-09-13T17:14:31.546772+0000","last_clean":"2024-09-13T17:14:31.546772+0000","last_became_active":"2024-09-13T17:14:31.113810+0000","last_became_peered":"2024-09-13T17:14:31.113810+0000","last_unstale":"2024-09-13T17:14:31.546772+0000","last_undegraded":"2024-09-13T17:14:31.546772+0000","last_fullsized":"2024-09-13T17:14:31.546772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:27.864333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564773+0000","last_change":"2024-09-13T17:14:31.550030+0000","last_active":"2024-09-13T17:14:31.564773+0000","last_peered":"2024-09-13T17:14:31.564773+0000","last_clean":"2024-09-13T17:14:31.564773+0000","last_became_active":"2024-09-13T17:14:31.549705+0000","last_became_peered":"2024-09-13T17:14:31.549705+0000","last_unstale":"2024-09-13T17:14:31.564773+0000","last_undegraded":"2024-09-13T17:14:31.564773+0000","last_fullsized":"2024-09-13T17:14:31.564773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:18.263370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541248+0000","last_change":"2024-09-13T17:14:31.120749+0000","last_active":"2024-09-13T17:14:32.541248+0000","last_peered":"2024-09-13T17:14:32.541248+0000","last_clean":"2024-09-13T17:14:32.541248+0000","last_became_active":"2024-09-13T17:14:31.120496+0000","last_became_peered":"2024-09-13T17:14:31.120496+0000","last_unstale":"2024-09-13T17:14:32.541248+0000","last_undegraded":"2024-09-13T17:14:32.541248+0000","last_fullsized":"2024-09-13T17:14:32.541248+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.115575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565582+0000","last_change":"2024-09-13T17:14:31.562716+0000","last_active":"2024-09-13T17:14:31.565582+0000","last_peered":"2024-09-13T17:14:31.565582+0000","last_clean":"2024-09-13T17:14:31.565582+0000","last_became_active":"2024-09-13T17:14:31.556196+0000","last_became_peered":"2024-09-13T17:14:31.556196+0000","last_unstale":"2024-09-13T17:14:31.565582+0000","last_undegraded":"2024-09-13T17:14:31.565582+0000","last_fullsized":"2024-09-13T17:14:31.565582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:03.937959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541537+0000","last_change":"2024-09-13T17:14:31.113229+0000","last_active":"2024-09-13T17:14:32.541537+0000","last_peered":"2024-09-13T17:14:32.541537+0000","last_clean":"2024-09-13T17:14:32.541537+0000","last_became_active":"2024-09-13T17:14:31.112956+0000","last_became_peered":"2024-09-13T17:14:31.112956+0000","last_unstale":"2024-09-13T17:14:32.541537+0000","last_undegraded":"2024-09-13T17:14:32.541537+0000","last_fullsized":"2024-09-13T17:14:32.541537+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.562161+0000","last_change":"2024-09-13T17:14:31.558135+0000","last_active":"2024-09-13T17:14:31.562161+0000","last_peered":"2024-09-13T17:14:31.562161+0000","last_clean":"2024-09-13T17:14:31.562161+0000","last_became_active":"2024-09-13T17:14:31.557891+0000","last_became_peered":"2024-09-13T17:14:31.557891+0000","last_unstale":"2024-09-13T17:14:31.562161+0000","last_undegraded":"2024-09-13T17:14:31.562161+0000","last_fullsized":"2024-09-13T17:14:31.562161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:40.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.538433+0000","last_change":"2024-09-13T17:14:31.124389+0000","last_active":"2024-09-13T17:14:31.538433+0000","last_peered":"2024-09-13T17:14:31.538433+0000","last_clean":"2024-09-13T17:14:31.538433+0000","last_became_active":"2024-09-13T17:14:31.124070+0000","last_became_peered":"2024-09-13T17:14:31.124070+0000","last_unstale":"2024-09-13T17:14:31.538433+0000","last_undegraded":"2024-09-13T17:14:31.538433+0000","last_fullsized":"2024-09-13T17:14:31.538433+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:15.521226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553591+0000","last_change":"2024-09-13T17:14:31.547532+0000","last_active":"2024-09-13T17:14:31.553591+0000","last_peered":"2024-09-13T17:14:31.553591+0000","last_clean":"2024-09-13T17:14:31.553591+0000","last_became_active":"2024-09-13T17:14:31.546114+0000","last_became_peered":"2024-09-13T17:14:31.546114+0000","last_unstale":"2024-09-13T17:14:31.553591+0000","last_undegraded":"2024-09-13T17:14:31.553591+0000","last_fullsized":"2024-09-13T17:14:31.553591+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:31.979640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541313+0000","last_change":"2024-09-13T17:14:31.112592+0000","last_active":"2024-09-13T17:14:32.541313+0000","last_peered":"2024-09-13T17:14:32.541313+0000","last_clean":"2024-09-13T17:14:32.541313+0000","last_became_active":"2024-09-13T17:14:31.108144+0000","last_became_peered":"2024-09-13T17:14:31.108144+0000","last_unstale":"2024-09-13T17:14:32.541313+0000","last_undegraded":"2024-09-13T17:14:32.541313+0000","last_fullsized":"2024-09-13T17:14:32.541313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:08.871649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.566636+0000","last_change":"2024-09-13T17:14:31.564944+0000","last_active":"2024-09-13T17:14:31.566636+0000","last_peered":"2024-09-13T17:14:31.566636+0000","last_clean":"2024-09-13T17:14:31.566636+0000","last_became_active":"2024-09-13T17:14:31.564690+0000","last_became_peered":"2024-09-13T17:14:31.564690+0000","last_unstale":"2024-09-13T17:14:31.566636+0000","last_undegraded":"2024-09-13T17:14:31.566636+0000","last_fullsized":"2024-09-13T17:14:31.566636+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:54.660491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.536048+0000","last_change":"2024-09-13T17:14:31.120433+0000","last_active":"2024-09-13T17:14:31.536048+0000","last_peered":"2024-09-13T17:14:31.536048+0000","last_clean":"2024-09-13T17:14:31.536048+0000","last_became_active":"2024-09-13T17:14:31.119944+0000","last_became_peered":"2024-09-13T17:14:31.119944+0000","last_unstale":"2024-09-13T17:14:31.536048+0000","last_undegraded":"2024-09-13T17:14:31.536048+0000","last_fullsized":"2024-09-13T17:14:31.536048+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:19.907478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565170+0000","last_change":"2024-09-13T17:14:31.562704+0000","last_active":"2024-09-13T17:14:31.565170+0000","last_peered":"2024-09-13T17:14:31.565170+0000","last_clean":"2024-09-13T17:14:31.565170+0000","last_became_active":"2024-09-13T17:14:31.562417+0000","last_became_peered":"2024-09-13T17:14:31.562417+0000","last_unstale":"2024-09-13T17:14:31.565170+0000","last_undegraded":"2024-09-13T17:14:31.565170+0000","last_fullsized":"2024-09-13T17:14:31.565170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:09.874271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.535399+0000","last_change":"2024-09-13T17:14:31.114224+0000","last_active":"2024-09-13T17:14:31.535399+0000","last_peered":"2024-09-13T17:14:31.535399+0000","last_clean":"2024-09-13T17:14:31.535399+0000","last_became_active":"2024-09-13T17:14:31.112368+0000","last_became_peered":"2024-09-13T17:14:31.112368+0000","last_unstale":"2024-09-13T17:14:31.535399+0000","last_undegraded":"2024-09-13T17:14:31.535399+0000","last_fullsized":"2024-09-13T17:14:31.535399+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.771574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561850+0000","last_change":"2024-09-13T17:14:31.556231+0000","last_active":"2024-09-13T17:14:31.561850+0000","last_peered":"2024-09-13T17:14:31.561850+0000","last_clean":"2024-09-13T17:14:31.561850+0000","last_became_active":"2024-09-13T17:14:31.555741+0000","last_became_peered":"2024-09-13T17:14:31.555741+0000","last_unstale":"2024-09-13T17:14:31.561850+0000","last_undegraded":"2024-09-13T17:14:31.561850+0000","last_fullsized":"2024-09-13T17:14:31.561850+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:54.231517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550977+0000","last_change":"2024-09-13T17:14:31.109595+0000","last_active":"2024-09-13T17:14:31.550977+0000","last_peered":"2024-09-13T17:14:31.550977+0000","last_clean":"2024-09-13T17:14:31.550977+0000","last_became_active":"2024-09-13T17:14:31.109310+0000","last_became_peered":"2024-09-13T17:14:31.109310+0000","last_unstale":"2024-09-13T17:14:31.550977+0000","last_undegraded":"2024-09-13T17:14:31.550977+0000","last_fullsized":"2024-09-13T17:14:31.550977+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.115575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565146+0000","last_change":"2024-09-13T17:14:31.558116+0000","last_active":"2024-09-13T17:14:31.565146+0000","last_peered":"2024-09-13T17:14:31.565146+0000","last_clean":"2024-09-13T17:14:31.565146+0000","last_became_active":"2024-09-13T17:14:31.557480+0000","last_became_peered":"2024-09-13T17:14:31.557480+0000","last_unstale":"2024-09-13T17:14:31.565146+0000","last_undegraded":"2024-09-13T17:14:31.565146+0000","last_fullsized":"2024-09-13T17:14:31.565146+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:43.674903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.536566+0000","last_change":"2024-09-13T17:14:31.114083+0000","last_active":"2024-09-13T17:14:31.536566+0000","last_peered":"2024-09-13T17:14:31.536566+0000","last_clean":"2024-09-13T17:14:31.536566+0000","last_became_active":"2024-09-13T17:14:31.113479+0000","last_became_peered":"2024-09-13T17:14:31.113479+0000","last_unstale":"2024-09-13T17:14:31.536566+0000","last_undegraded":"2024-09-13T17:14:31.536566+0000","last_fullsized":"2024-09-13T17:14:31.536566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:13.698999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540907+0000","last_change":"2024-09-13T17:14:31.559346+0000","last_active":"2024-09-13T17:14:32.540907+0000","last_peered":"2024-09-13T17:14:32.540907+0000","last_clean":"2024-09-13T17:14:32.540907+0000","last_became_active":"2024-09-13T17:14:31.558649+0000","last_became_peered":"2024-09-13T17:14:31.558649+0000","last_unstale":"2024-09-13T17:14:32.540907+0000","last_undegraded":"2024-09-13T17:14:32.540907+0000","last_fullsized":"2024-09-13T17:14:32.540907+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:14.125462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.551533+0000","last_change":"2024-09-13T17:14:31.116820+0000","last_active":"2024-09-13T17:14:31.551533+0000","last_peered":"2024-09-13T17:14:31.551533+0000","last_clean":"2024-09-13T17:14:31.551533+0000","last_became_active":"2024-09-13T17:14:31.116495+0000","last_became_peered":"2024-09-13T17:14:31.116495+0000","last_unstale":"2024-09-13T17:14:31.551533+0000","last_undegraded":"2024-09-13T17:14:31.551533+0000","last_fullsized":"2024-09-13T17:14:31.551533+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:11.627636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565564+0000","last_change":"2024-09-13T17:14:31.562555+0000","last_active":"2024-09-13T17:14:31.565564+0000","last_peered":"2024-09-13T17:14:31.565564+0000","last_clean":"2024-09-13T17:14:31.565564+0000","last_became_active":"2024-09-13T17:14:31.555855+0000","last_became_peered":"2024-09-13T17:14:31.555855+0000","last_unstale":"2024-09-13T17:14:31.565564+0000","last_undegraded":"2024-09-13T17:14:31.565564+0000","last_fullsized":"2024-09-13T17:14:31.565564+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:53.200392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536533+0000","last_change":"2024-09-13T17:14:31.122578+0000","last_active":"2024-09-13T17:14:32.536533+0000","last_peered":"2024-09-13T17:14:32.536533+0000","last_clean":"2024-09-13T17:14:32.536533+0000","last_became_active":"2024-09-13T17:14:31.122299+0000","last_became_peered":"2024-09-13T17:14:31.122299+0000","last_unstale":"2024-09-13T17:14:32.536533+0000","last_undegraded":"2024-09-13T17:14:32.536533+0000","last_fullsized":"2024-09-13T17:14:32.536533+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:15.521226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561199+0000","last_change":"2024-09-13T17:14:31.555934+0000","last_active":"2024-09-13T17:14:31.561199+0000","last_peered":"2024-09-13T17:14:31.561199+0000","last_clean":"2024-09-13T17:14:31.561199+0000","last_became_active":"2024-09-13T17:14:31.553557+0000","last_became_peered":"2024-09-13T17:14:31.553557+0000","last_unstale":"2024-09-13T17:14:31.561199+0000","last_undegraded":"2024-09-13T17:14:31.561199+0000","last_fullsized":"2024-09-13T17:14:31.561199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:16.523848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.531428+0000","last_change":"2024-09-13T17:14:31.118072+0000","last_active":"2024-09-13T17:14:31.531428+0000","last_peered":"2024-09-13T17:14:31.531428+0000","last_clean":"2024-09-13T17:14:31.531428+0000","last_became_active":"2024-09-13T17:14:31.117799+0000","last_became_peered":"2024-09-13T17:14:31.117799+0000","last_unstale":"2024-09-13T17:14:31.531428+0000","last_undegraded":"2024-09-13T17:14:31.531428+0000","last_fullsized":"2024-09-13T17:14:31.531428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:12.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565572+0000","last_change":"2024-09-13T17:14:31.561149+0000","last_active":"2024-09-13T17:14:31.565572+0000","last_peered":"2024-09-13T17:14:31.565572+0000","last_clean":"2024-09-13T17:14:31.565572+0000","last_became_active":"2024-09-13T17:14:31.560912+0000","last_became_peered":"2024-09-13T17:14:31.560912+0000","last_unstale":"2024-09-13T17:14:31.565572+0000","last_undegraded":"2024-09-13T17:14:31.565572+0000","last_fullsized":"2024-09-13T17:14:31.565572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:55.558600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.559380+0000","last_change":"2024-09-13T17:14:31.115877+0000","last_active":"2024-09-13T17:14:31.559380+0000","last_peered":"2024-09-13T17:14:31.559380+0000","last_clean":"2024-09-13T17:14:31.559380+0000","last_became_active":"2024-09-13T17:14:31.115620+0000","last_became_peered":"2024-09-13T17:14:31.115620+0000","last_unstale":"2024-09-13T17:14:31.559380+0000","last_undegraded":"2024-09-13T17:14:31.559380+0000","last_fullsized":"2024-09-13T17:14:31.559380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:22:26.273416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":240,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.107852+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:14:31.107852+0000","last_peered":"2024-09-13T17:14:31.107852+0000","last_clean":"2024-09-13T17:14:31.107852+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:14:31.107852+0000","last_undegraded":"2024-09-13T17:14:31.107852+0000","last_fullsized":"2024-09-13T17:14:31.107852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574930,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63376,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94308464,"statfs":{"total":96636764160,"available":96571867136,"internally_reserved":0,"allocated":1961984,"data_stored":1184160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":62912971},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.69099999999999995,"15min":0.65500000000000003},"min":{"1min":0.51800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.47599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.69599999999999995,"15min":0.65200000000000002},"min":{"1min":0.53200000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.109,"5min":0.78300000000000003,"15min":0.71699999999999997},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.64600000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72599999999999998,"15min":0.71799999999999997},"min":{"1min":0.57199999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.70499999999999996,"15min":0.68899999999999995},"min":{"1min":0.55800000000000005,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.5},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.78900000000000003,"15min":0.749},"min":{"1min":0.57999999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61499999999999999,"15min":0.65500000000000003},"min":{"1min":0.49199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.879,"5min":1.054,"15min":1.054},"last":0.621},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89400000000000002,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84599999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65400000000000003,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94899999999999995,"5min":1.143,"15min":1.143},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76200000000000001,"15min":0.78100000000000003},"min":{"1min":0.67700000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.92000000000000004,"5min":1.198,"15min":1.337},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.53200000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.878},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.77600000000000002},"min":{"1min":0.64000000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97199999999999998,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.77300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":1368064,"data_stored":594332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.66600000000000004},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97099999999999997,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.878},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.746,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94799999999999995,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.41599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91500000000000004,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69999999999999996,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86699999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.2909999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72699999999999998,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1.004,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71699999999999997,"15min":0.70699999999999996},"min":{"1min":0.51700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.2609999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71299999999999997,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.92700000000000005,"5min":3.742,"15min":3.742},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.624,"15min":0.628},"min":{"1min":0.51800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90300000000000002,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.624}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70199999999999996,"15min":0.72599999999999998},"min":{"1min":0.51300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68799999999999994,"15min":0.71999999999999997},"min":{"1min":0.54900000000000004,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89800000000000002,"5min":3.609,"15min":3.609},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.751,"15min":0.753},"min":{"1min":0.54500000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95199999999999996,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80700000000000005,"15min":0.80800000000000005},"min":{"1min":0.64500000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0069999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.75600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574930,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40648,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331192,"statfs":{"total":96636764160,"available":96595140608,"internally_reserved":0,"allocated":1363968,"data_stored":593750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98999999999999999,"15min":0.77800000000000002},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79000000000000004,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.86099999999999999,"15min":0.76700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.83199999999999996,"15min":0.73799999999999999},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91400000000000003,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.71799999999999997,"15min":0.70399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.76300000000000001},"min":{"1min":0.52100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84299999999999997,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.77800000000000002,"15min":0.75600000000000001},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89800000000000002,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.792,"15min":1.107},"min":{"1min":0.51500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.876,"5min":113.363,"15min":113.363},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92800000000000005,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.627}]},{"osd":4,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.628,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68899999999999995,"15min":0.71999999999999997},"min":{"1min":0.442,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88100000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71899999999999997,"15min":0.70699999999999996},"min":{"1min":0.50800000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94999999999999996,"5min":2.056,"15min":2.056},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86099999999999999,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.78600000000000003,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.78400000000000003,"15min":0.76800000000000002},"min":{"1min":0.58399999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95699999999999996,"5min":2.02,"15min":2.02},"last":0.76400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46420,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325420,"statfs":{"total":96636764160,"available":96589230080,"internally_reserved":0,"allocated":1380352,"data_stored":595368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.0620000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.998,"5min":0.998,"15min":1.341},"last":0.84099999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0109999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66100000000000003,"15min":0.627},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91800000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.86799999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.66500000000000004,"15min":0.65200000000000002},"min":{"1min":0.497,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.008,"5min":1.014,"15min":1.014},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70099999999999996,"15min":0.72099999999999997},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.034,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.68400000000000005,"15min":0.67000000000000004},"min":{"1min":0.503,"5min":0.434,"15min":0.434},"max":{"1min":0.97199999999999998,"5min":1.002,"15min":1.002},"last":1.006},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.69699999999999995},"min":{"1min":0.51800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.075,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":1.232}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.621,"15min":0.64500000000000002},"min":{"1min":0.48199999999999998,"5min":0.38700000000000001,"15min":0.38400000000000001},"max":{"1min":1.008,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.51800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.03,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":1.274},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.74399999999999999},"min":{"1min":0.56999999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.413},"last":1.292}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.71699999999999997,"15min":0.69799999999999995},"min":{"1min":0.59199999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73599999999999999,"15min":0.73199999999999998},"min":{"1min":0.54900000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.1459999999999999,"5min":1.177,"15min":1.177},"last":1.3100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50516,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321324,"statfs":{"total":96636764160,"available":96585035776,"internally_reserved":0,"allocated":1376256,"data_stored":595266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.30099999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.59899999999999998,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46999999999999997,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.625,"5min":0.97099999999999997,"15min":2.96},"last":0.36199999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.40300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61499999999999999,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61499999999999999,"15min":0.71799999999999997},"min":{"1min":0.47199999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73699999999999999,"5min":3.621,"15min":3.6829999999999998},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.502},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57099999999999995,"15min":0.59899999999999998},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85999999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73999999999999999,"15min":0.80900000000000005},"min":{"1min":0.56299999999999994,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91100000000000003,"5min":1.3560000000000001,"15min":2.8610000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.60399999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.208,"15min":1.3640000000000001},"last":0.85799999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.89900000000000002,"5min":1.391,"15min":1.391},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.82899999999999996},"min":{"1min":0.52300000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89000000000000001,"5min":1.252,"15min":2.9300000000000002},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76800000000000002},"min":{"1min":0.59499999999999997,"5min":0.53800000000000003,"15min":0.53000000000000003},"max":{"1min":0.89400000000000002,"5min":1.244,"15min":1.401},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.56499999999999995,"5min":0.55200000000000005,"15min":0.55000000000000004},"max":{"1min":0.93799999999999994,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.79900000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.85599999999999998},"min":{"1min":0.61199999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.877,"5min":1.4139999999999999,"15min":2.899},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.64700000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.93000000000000005,"5min":1.278,"15min":1.4359999999999999},"last":0.90600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574930,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40640,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331200,"statfs":{"total":96636764160,"available":96595148800,"internally_reserved":0,"allocated":1359872,"data_stored":592695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40600000000000003,"15min":0.42099999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.441},"min":{"1min":0.36399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91800000000000004,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":3.0819999999999999,"15min":1.3440000000000001},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.97699999999999998,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.495},{"interface":"front","average":{"1min":0.55200000000000005,"5min":3.1379999999999999,"15min":1.3620000000000001},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.96999999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.503,"15min":0.48699999999999999},"min":{"1min":0.36299999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83099999999999996,"5min":1.214,"15min":1.214},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77200000000000002,"15min":0.76400000000000001},"min":{"1min":0.63400000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0089999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78200000000000003,"15min":0.78700000000000003},"min":{"1min":0.55600000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.135,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.66300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.0369999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.52800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0700000000000001,"5min":1.569,"15min":1.569},"last":0.90700000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.61899999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.091,"5min":1.413,"15min":1.413},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.627,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0780000000000001,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80000000000000004,"15min":0.78500000000000003},"min":{"1min":0.70599999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0429999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80100000000000005,"15min":0.80600000000000005},"min":{"1min":0.61799999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.82899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50492,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321348,"statfs":{"total":96636764160,"available":96585060352,"internally_reserved":0,"allocated":1359872,"data_stored":593680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68400000000000005,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49399999999999999,"15min":0.46999999999999997},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63700000000000001,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.433},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.41899999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.76700000000000002,"15min":0.69099999999999995},"min":{"1min":0.46300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.72999999999999998,"15min":0.61399999999999999},"min":{"1min":0.36299999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.57599999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.329,"5min":1.78,"15min":1.78},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77600000000000002,"15min":0.77300000000000002},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92300000000000004,"5min":1.6020000000000001,"15min":1.6870000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.98399999999999999,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.79700000000000004,"15min":0.81100000000000005},"min":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0069999999999999,"5min":1.6599999999999999,"15min":1.7310000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.84099999999999997,"15min":0.82299999999999995},"min":{"1min":0.55200000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.165,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79700000000000004,"15min":0.82699999999999996},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0189999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56699999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.056,"5min":1.544,"15min":1.6359999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41216,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330624,"statfs":{"total":96636764160,"available":96594558976,"internally_reserved":0,"allocated":1953792,"data_stored":1183611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.67800000000000005},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71199999999999997,"5min":0.99099999999999999,"15min":5.5259999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47299999999999998,"15min":0.67100000000000004},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82699999999999996,"5min":1.1859999999999999,"15min":5.4889999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.499,"15min":0.70499999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"max":{"1min":0.66800000000000004,"5min":0.81100000000000005,"15min":6.0430000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.71999999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68999999999999995,"5min":1.2190000000000001,"15min":4.6760000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49399999999999999,"15min":0.66300000000000003},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":1.5069999999999999,"15min":4.7549999999999999},"last":0.5},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56699999999999995,"15min":0.71799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36199999999999999},"max":{"1min":0.81399999999999995,"5min":1.1000000000000001,"15min":4.5940000000000003},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75800000000000001,"15min":0.89600000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.083,"5min":1.3160000000000001,"15min":4.734},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.754,"15min":0.91200000000000003},"min":{"1min":0.52300000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.337,"5min":1.337,"15min":4.8079999999999998},"last":0.874}]},{"osd":5,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.747,"15min":0.92500000000000004},"min":{"1min":0.55000000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1319999999999999,"5min":1.175,"15min":4.944},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77800000000000002,"15min":0.91500000000000004},"min":{"1min":0.66900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9710000000000001},"last":0.85699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.77400000000000002,"15min":1.0680000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.54200000000000004},"max":{"1min":1.1859999999999999,"5min":1.242,"15min":4.8470000000000004},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77700000000000002,"15min":0.93600000000000005},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.163,"5min":1.274,"15min":5.0170000000000003},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78300000000000003,"15min":0.96899999999999997},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2609999999999999,"5min":1.3069999999999999,"15min":5.0540000000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77900000000000003,"15min":0.95099999999999996},"min":{"1min":0.59999999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.242,"5min":1.4530000000000001,"15min":4.9950000000000001},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:14:35.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:14:35.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.909+0000 7f5ba3200640 1 Processor -- start 2024-09-13T17:14:35.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.910+0000 7f5ba3200640 1 -- start start 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ba415c5f0 0x7f5ba41602e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 0x7f5ba40b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba406b8c0 con 0x7f5ba40b1ff0 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba4161340 con 0x7f5ba415c5f0 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba40b24a0 con 0x7f5ba40ac850 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ba415c5f0 0x7f5ba41602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ba415c5f0 0x7f5ba41602e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40792/0 (socket says 172.21.15.28:40792) 2024-09-13T17:14:35.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5ba1e00640 1 -- 172.21.15.28:0/1351799498 learned_addr learned my addr 172.21.15.28:0/1351799498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.912+0000 7f5ba1e00640 1 -- 172.21.15.28:0/1351799498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 msgr2=0x7f5ba40b1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.911+0000 7f5b99400640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 0x7f5ba40b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.912+0000 7f5ba1e00640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 0x7f5ba40b1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.912+0000 7f5ba1e00640 1 -- 172.21.15.28:0/1351799498 >> v1:172.21.15.189:6789/0 conn(0x7f5ba40b1ff0 legacy=0x7f5ba415c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.912+0000 7f5ba1e00640 1 -- 172.21.15.28:0/1351799498 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba4160820 con 0x7f5ba415c5f0 2024-09-13T17:14:35.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.912+0000 7f5b99400640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 0x7f5ba40b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:35.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.913+0000 7f5ba1e00640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba415c5f0 0x7f5ba41602e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098a0 tx=0x7f5b9402a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c10efb8879175be server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.913+0000 7f5ba1400640 1 -- 172.21.15.28:0/1351799498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94032440 con 0x7f5ba415c5f0 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.913+0000 7f5ba1400640 1 -- 172.21.15.28:0/1351799498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b940325a0 con 0x7f5ba415c5f0 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.913+0000 7f5ba1400640 1 -- 172.21.15.28:0/1351799498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94032930 con 0x7f5ba415c5f0 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 -- 172.21.15.28:0/1351799498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba415c5f0 msgr2=0x7f5ba41602e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba415c5f0 0x7f5ba41602e0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f5b940098a0 tx=0x7f5b9402a290 comp rx=0 tx=0).stop 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 -- 172.21.15.28:0/1351799498 shutdown_connections 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba40ac850 0x7f5ba40b1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 --2- 172.21.15.28:0/1351799498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba415c5f0 0x7f5ba41602e0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 -- 172.21.15.28:0/1351799498 >> 172.21.15.28:0/1351799498 conn(0x7f5ba40ce880 msgr2=0x7f5ba40cec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 -- 172.21.15.28:0/1351799498 shutdown_connections 2024-09-13T17:14:35.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.914+0000 7f5ba3200640 1 -- 172.21.15.28:0/1351799498 wait complete. 2024-09-13T17:14:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.915+0000 7f5ba3200640 1 Processor -- start 2024-09-13T17:14:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.916+0000 7f5ba3200640 1 -- start start 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.916+0000 7f5ba3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 0x7f5ba4156980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 0x7f5ba4144290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba4156ec0 con 0x7f5ba40ac850 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba415e610 con 0x7f5ba40b1ff0 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba415dd20 con 0x7f5ba415c5f0 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40796/0 (socket says 172.21.15.28:40796) 2024-09-13T17:14:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 -- 172.21.15.28:0/3116745174 learned_addr learned my addr 172.21.15.28:0/3116745174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5b99400640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 0x7f5ba4156980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba2800640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 0x7f5ba4144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 -- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 msgr2=0x7f5ba4144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 0x7f5ba4144290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 -- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 msgr2=0x7f5ba4156980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 0x7f5ba4156980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba1e00640 1 -- 172.21.15.28:0/3116745174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940094d0 con 0x7f5ba40ac850 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5b99400640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 0x7f5ba4156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.917+0000 7f5ba2800640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 0x7f5ba4144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5ba1e00640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5b94000c00 tx=0x7f5b940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b94030b00 con 0x7f5ba40ac850 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b94030c60 con 0x7f5ba40ac850 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba415dfc0 con 0x7f5ba40ac850 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b9403d440 con 0x7f5ba40ac850 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.918+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba41447d0 con 0x7f5ba40ac850 2024-09-13T17:14:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.919+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba41603b0 con 0x7f5ba40ac850 2024-09-13T17:14:35.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.920+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b94039040 con 0x7f5ba40ac850 2024-09-13T17:14:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.921+0000 7f5b9aa00640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 0x7f5b8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.921+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b940aa0e0 con 0x7f5ba40ac850 2024-09-13T17:14:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.922+0000 7f5b99400640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 0x7f5b8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.922+0000 7f5b99400640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 0x7f5b8006d0c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4153790 tx=0x7f5b8c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:35.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:35.925+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5b940752d0 con 0x7f5ba40ac850 2024-09-13T17:14:36.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.100+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ba4161060 con 0x7f5b8006ac00 2024-09-13T17:14:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.107+0000 7f5b9aa00640 1 -- 172.21.15.28:0/3116745174 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289779 (secure 0 0 0) 0x7f5b8c00c040 con 0x7f5b8006ac00 2024-09-13T17:14:36.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:14:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 msgr2=0x7f5b8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 0x7f5b8006d0c0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4153790 tx=0x7f5b8c008040 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 msgr2=0x7f5ba4144c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f5b94000c00 tx=0x7f5b940046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 shutdown_connections 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b8006ac00 0x7f5b8006d0c0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ba415c5f0 0x7f5ba4144290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ba40b1ff0 0x7f5ba4156980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 --2- 172.21.15.28:0/3116745174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ba40ac850 0x7f5ba4144c30 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 >> 172.21.15.28:0/3116745174 conn(0x7f5ba40ce880 msgr2=0x7f5ba415fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 shutdown_connections 2024-09-13T17:14:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.111+0000 7f5ba3200640 1 -- 172.21.15.28:0/3116745174 wait complete. 2024-09-13T17:14:36.128 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":303,"stamp":"2024-09-13T17:14:34.982326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":373952,"kb_used_data":11840,"kb_used_omap":12,"kb_used_meta":361971,"kb_avail":754600768,"statfs":{"total":773094113280,"available":772711186432,"internally_reserved":0,"allocated":12124160,"data_stored":5932862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":370658873},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.892061"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561027+0000","last_change":"2024-09-13T17:14:31.554424+0000","last_active":"2024-09-13T17:14:31.561027+0000","last_peered":"2024-09-13T17:14:31.561027+0000","last_clean":"2024-09-13T17:14:31.561027+0000","last_became_active":"2024-09-13T17:14:31.554085+0000","last_became_peered":"2024-09-13T17:14:31.554085+0000","last_unstale":"2024-09-13T17:14:31.561027+0000","last_undegraded":"2024-09-13T17:14:31.561027+0000","last_fullsized":"2024-09-13T17:14:31.561027+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:26.708148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553633+0000","last_change":"2024-09-13T17:14:31.550028+0000","last_active":"2024-09-13T17:14:31.553633+0000","last_peered":"2024-09-13T17:14:31.553633+0000","last_clean":"2024-09-13T17:14:31.553633+0000","last_became_active":"2024-09-13T17:14:31.545311+0000","last_became_peered":"2024-09-13T17:14:31.545311+0000","last_unstale":"2024-09-13T17:14:31.553633+0000","last_undegraded":"2024-09-13T17:14:31.553633+0000","last_fullsized":"2024-09-13T17:14:31.553633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:59.641934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.548580+0000","last_change":"2024-09-13T17:14:31.544920+0000","last_active":"2024-09-13T17:14:31.548580+0000","last_peered":"2024-09-13T17:14:31.548580+0000","last_clean":"2024-09-13T17:14:31.548580+0000","last_became_active":"2024-09-13T17:14:31.544688+0000","last_became_peered":"2024-09-13T17:14:31.544688+0000","last_unstale":"2024-09-13T17:14:31.548580+0000","last_undegraded":"2024-09-13T17:14:31.548580+0000","last_fullsized":"2024-09-13T17:14:31.548580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:29.033773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536839+0000","last_change":"2024-09-13T17:14:31.567127+0000","last_active":"2024-09-13T17:14:32.536839+0000","last_peered":"2024-09-13T17:14:32.536839+0000","last_clean":"2024-09-13T17:14:32.536839+0000","last_became_active":"2024-09-13T17:14:31.566097+0000","last_became_peered":"2024-09-13T17:14:31.566097+0000","last_unstale":"2024-09-13T17:14:32.536839+0000","last_undegraded":"2024-09-13T17:14:32.536839+0000","last_fullsized":"2024-09-13T17:14:32.536839+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:39.388937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.560773+0000","last_change":"2024-09-13T17:14:31.556891+0000","last_active":"2024-09-13T17:14:31.560773+0000","last_peered":"2024-09-13T17:14:31.560773+0000","last_clean":"2024-09-13T17:14:31.560773+0000","last_became_active":"2024-09-13T17:14:31.556642+0000","last_became_peered":"2024-09-13T17:14:31.556642+0000","last_unstale":"2024-09-13T17:14:31.560773+0000","last_undegraded":"2024-09-13T17:14:31.560773+0000","last_fullsized":"2024-09-13T17:14:31.560773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:07.826659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546944+0000","last_change":"2024-09-13T17:14:31.545197+0000","last_active":"2024-09-13T17:14:31.546944+0000","last_peered":"2024-09-13T17:14:31.546944+0000","last_clean":"2024-09-13T17:14:31.546944+0000","last_became_active":"2024-09-13T17:14:31.544897+0000","last_became_peered":"2024-09-13T17:14:31.544897+0000","last_unstale":"2024-09-13T17:14:31.546944+0000","last_undegraded":"2024-09-13T17:14:31.546944+0000","last_fullsized":"2024-09-13T17:14:31.546944+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:32.296423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550110+0000","last_change":"2024-09-13T17:14:31.543232+0000","last_active":"2024-09-13T17:14:31.550110+0000","last_peered":"2024-09-13T17:14:31.550110+0000","last_clean":"2024-09-13T17:14:31.550110+0000","last_became_active":"2024-09-13T17:14:31.542934+0000","last_became_peered":"2024-09-13T17:14:31.542934+0000","last_unstale":"2024-09-13T17:14:31.550110+0000","last_undegraded":"2024-09-13T17:14:31.550110+0000","last_fullsized":"2024-09-13T17:14:31.550110+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:07.826659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555729+0000","last_change":"2024-09-13T17:14:31.553530+0000","last_active":"2024-09-13T17:14:31.555729+0000","last_peered":"2024-09-13T17:14:31.555729+0000","last_clean":"2024-09-13T17:14:31.555729+0000","last_became_active":"2024-09-13T17:14:31.553280+0000","last_became_peered":"2024-09-13T17:14:31.553280+0000","last_unstale":"2024-09-13T17:14:31.555729+0000","last_undegraded":"2024-09-13T17:14:31.555729+0000","last_fullsized":"2024-09-13T17:14:31.555729+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:15.793328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541598+0000","last_change":"2024-09-13T17:14:31.560095+0000","last_active":"2024-09-13T17:14:32.541598+0000","last_peered":"2024-09-13T17:14:32.541598+0000","last_clean":"2024-09-13T17:14:32.541598+0000","last_became_active":"2024-09-13T17:14:31.559826+0000","last_became_peered":"2024-09-13T17:14:31.559826+0000","last_unstale":"2024-09-13T17:14:32.541598+0000","last_undegraded":"2024-09-13T17:14:32.541598+0000","last_fullsized":"2024-09-13T17:14:32.541598+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:39.388937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.567848+0000","last_change":"2024-09-13T17:14:31.564723+0000","last_active":"2024-09-13T17:14:31.567848+0000","last_peered":"2024-09-13T17:14:31.567848+0000","last_clean":"2024-09-13T17:14:31.567848+0000","last_became_active":"2024-09-13T17:14:31.564423+0000","last_became_peered":"2024-09-13T17:14:31.564423+0000","last_unstale":"2024-09-13T17:14:31.567848+0000","last_undegraded":"2024-09-13T17:14:31.567848+0000","last_fullsized":"2024-09-13T17:14:31.567848+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.910100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.557637+0000","last_change":"2024-09-13T17:14:31.553009+0000","last_active":"2024-09-13T17:14:31.557637+0000","last_peered":"2024-09-13T17:14:31.557637+0000","last_clean":"2024-09-13T17:14:31.557637+0000","last_became_active":"2024-09-13T17:14:31.552493+0000","last_became_peered":"2024-09-13T17:14:31.552493+0000","last_unstale":"2024-09-13T17:14:31.557637+0000","last_undegraded":"2024-09-13T17:14:31.557637+0000","last_fullsized":"2024-09-13T17:14:31.557637+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:16.523848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541470+0000","last_change":"2024-09-13T17:14:31.555997+0000","last_active":"2024-09-13T17:14:32.541470+0000","last_peered":"2024-09-13T17:14:32.541470+0000","last_clean":"2024-09-13T17:14:32.541470+0000","last_became_active":"2024-09-13T17:14:31.555707+0000","last_became_peered":"2024-09-13T17:14:31.555707+0000","last_unstale":"2024-09-13T17:14:32.541470+0000","last_undegraded":"2024-09-13T17:14:32.541470+0000","last_fullsized":"2024-09-13T17:14:32.541470+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536596+0000","last_change":"2024-09-13T17:14:31.554798+0000","last_active":"2024-09-13T17:14:32.536596+0000","last_peered":"2024-09-13T17:14:32.536596+0000","last_clean":"2024-09-13T17:14:32.536596+0000","last_became_active":"2024-09-13T17:14:31.554427+0000","last_became_peered":"2024-09-13T17:14:31.554427+0000","last_unstale":"2024-09-13T17:14:32.536596+0000","last_undegraded":"2024-09-13T17:14:32.536596+0000","last_fullsized":"2024-09-13T17:14:32.536596+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565769+0000","last_change":"2024-09-13T17:14:31.564011+0000","last_active":"2024-09-13T17:14:31.565769+0000","last_peered":"2024-09-13T17:14:31.565769+0000","last_clean":"2024-09-13T17:14:31.565769+0000","last_became_active":"2024-09-13T17:14:31.558339+0000","last_became_peered":"2024-09-13T17:14:31.558339+0000","last_unstale":"2024-09-13T17:14:31.565769+0000","last_undegraded":"2024-09-13T17:14:31.565769+0000","last_fullsized":"2024-09-13T17:14:31.565769+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:10.995098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536471+0000","last_change":"2024-09-13T17:14:31.557150+0000","last_active":"2024-09-13T17:14:32.536471+0000","last_peered":"2024-09-13T17:14:32.536471+0000","last_clean":"2024-09-13T17:14:32.536471+0000","last_became_active":"2024-09-13T17:14:31.554479+0000","last_became_peered":"2024-09-13T17:14:31.554479+0000","last_unstale":"2024-09-13T17:14:32.536471+0000","last_undegraded":"2024-09-13T17:14:32.536471+0000","last_fullsized":"2024-09-13T17:14:32.536471+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550238+0000","last_change":"2024-09-13T17:14:31.548205+0000","last_active":"2024-09-13T17:14:31.550238+0000","last_peered":"2024-09-13T17:14:31.550238+0000","last_clean":"2024-09-13T17:14:31.550238+0000","last_became_active":"2024-09-13T17:14:31.547780+0000","last_became_peered":"2024-09-13T17:14:31.547780+0000","last_unstale":"2024-09-13T17:14:31.550238+0000","last_undegraded":"2024-09-13T17:14:31.550238+0000","last_fullsized":"2024-09-13T17:14:31.550238+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:20.910100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563137+0000","last_change":"2024-09-13T17:14:31.562103+0000","last_active":"2024-09-13T17:14:31.563137+0000","last_peered":"2024-09-13T17:14:31.563137+0000","last_clean":"2024-09-13T17:14:31.563137+0000","last_became_active":"2024-09-13T17:14:31.560557+0000","last_became_peered":"2024-09-13T17:14:31.560557+0000","last_unstale":"2024-09-13T17:14:31.563137+0000","last_undegraded":"2024-09-13T17:14:31.563137+0000","last_fullsized":"2024-09-13T17:14:31.563137+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:26.090024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.537580+0000","last_change":"2024-09-13T17:14:31.566397+0000","last_active":"2024-09-13T17:14:32.537580+0000","last_peered":"2024-09-13T17:14:32.537580+0000","last_clean":"2024-09-13T17:14:32.537580+0000","last_became_active":"2024-09-13T17:14:31.566114+0000","last_became_peered":"2024-09-13T17:14:31.566114+0000","last_unstale":"2024-09-13T17:14:32.537580+0000","last_undegraded":"2024-09-13T17:14:32.537580+0000","last_fullsized":"2024-09-13T17:14:32.537580+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:59.045988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563756+0000","last_change":"2024-09-13T17:14:31.547194+0000","last_active":"2024-09-13T17:14:31.563756+0000","last_peered":"2024-09-13T17:14:31.563756+0000","last_clean":"2024-09-13T17:14:31.563756+0000","last_became_active":"2024-09-13T17:14:31.546940+0000","last_became_peered":"2024-09-13T17:14:31.546940+0000","last_unstale":"2024-09-13T17:14:31.563756+0000","last_undegraded":"2024-09-13T17:14:31.563756+0000","last_fullsized":"2024-09-13T17:14:31.563756+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:19.702514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541528+0000","last_change":"2024-09-13T17:14:31.560456+0000","last_active":"2024-09-13T17:14:32.541528+0000","last_peered":"2024-09-13T17:14:32.541528+0000","last_clean":"2024-09-13T17:14:32.541528+0000","last_became_active":"2024-09-13T17:14:31.560088+0000","last_became_peered":"2024-09-13T17:14:31.560088+0000","last_unstale":"2024-09-13T17:14:32.541528+0000","last_undegraded":"2024-09-13T17:14:32.541528+0000","last_fullsized":"2024-09-13T17:14:32.541528+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:11.549156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.545008+0000","last_change":"2024-09-13T17:14:31.543284+0000","last_active":"2024-09-13T17:14:31.545008+0000","last_peered":"2024-09-13T17:14:31.545008+0000","last_clean":"2024-09-13T17:14:31.545008+0000","last_became_active":"2024-09-13T17:14:31.542952+0000","last_became_peered":"2024-09-13T17:14:31.542952+0000","last_unstale":"2024-09-13T17:14:31.545008+0000","last_undegraded":"2024-09-13T17:14:31.545008+0000","last_fullsized":"2024-09-13T17:14:31.545008+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:24.213895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553337+0000","last_change":"2024-09-13T17:14:31.549440+0000","last_active":"2024-09-13T17:14:31.553337+0000","last_peered":"2024-09-13T17:14:31.553337+0000","last_clean":"2024-09-13T17:14:31.553337+0000","last_became_active":"2024-09-13T17:14:31.549202+0000","last_became_peered":"2024-09-13T17:14:31.549202+0000","last_unstale":"2024-09-13T17:14:31.553337+0000","last_undegraded":"2024-09-13T17:14:31.553337+0000","last_fullsized":"2024-09-13T17:14:31.553337+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:40:37.773761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546513+0000","last_change":"2024-09-13T17:14:31.545065+0000","last_active":"2024-09-13T17:14:31.546513+0000","last_peered":"2024-09-13T17:14:31.546513+0000","last_clean":"2024-09-13T17:14:31.546513+0000","last_became_active":"2024-09-13T17:14:31.544773+0000","last_became_peered":"2024-09-13T17:14:31.544773+0000","last_unstale":"2024-09-13T17:14:31.546513+0000","last_undegraded":"2024-09-13T17:14:31.546513+0000","last_fullsized":"2024-09-13T17:14:31.546513+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:59.646095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550217+0000","last_change":"2024-09-13T17:14:31.548038+0000","last_active":"2024-09-13T17:14:31.550217+0000","last_peered":"2024-09-13T17:14:31.550217+0000","last_clean":"2024-09-13T17:14:31.550217+0000","last_became_active":"2024-09-13T17:14:31.547461+0000","last_became_peered":"2024-09-13T17:14:31.547461+0000","last_unstale":"2024-09-13T17:14:31.550217+0000","last_undegraded":"2024-09-13T17:14:31.550217+0000","last_fullsized":"2024-09-13T17:14:31.550217+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:40.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561966+0000","last_change":"2024-09-13T17:14:31.556084+0000","last_active":"2024-09-13T17:14:31.561966+0000","last_peered":"2024-09-13T17:14:31.561966+0000","last_clean":"2024-09-13T17:14:31.561966+0000","last_became_active":"2024-09-13T17:14:31.554457+0000","last_became_peered":"2024-09-13T17:14:31.554457+0000","last_unstale":"2024-09-13T17:14:31.561966+0000","last_undegraded":"2024-09-13T17:14:31.561966+0000","last_fullsized":"2024-09-13T17:14:31.561966+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.866955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553612+0000","last_change":"2024-09-13T17:14:31.548061+0000","last_active":"2024-09-13T17:14:31.553612+0000","last_peered":"2024-09-13T17:14:31.553612+0000","last_clean":"2024-09-13T17:14:31.553612+0000","last_became_active":"2024-09-13T17:14:31.547834+0000","last_became_peered":"2024-09-13T17:14:31.547834+0000","last_unstale":"2024-09-13T17:14:31.553612+0000","last_undegraded":"2024-09-13T17:14:31.553612+0000","last_fullsized":"2024-09-13T17:14:31.553612+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:18.000332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549712+0000","last_change":"2024-09-13T17:14:31.544968+0000","last_active":"2024-09-13T17:14:31.549712+0000","last_peered":"2024-09-13T17:14:31.549712+0000","last_clean":"2024-09-13T17:14:31.549712+0000","last_became_active":"2024-09-13T17:14:31.544692+0000","last_became_peered":"2024-09-13T17:14:31.544692+0000","last_unstale":"2024-09-13T17:14:31.549712+0000","last_undegraded":"2024-09-13T17:14:31.549712+0000","last_fullsized":"2024-09-13T17:14:31.549712+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:36.617587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565179+0000","last_change":"2024-09-13T17:14:31.561345+0000","last_active":"2024-09-13T17:14:31.565179+0000","last_peered":"2024-09-13T17:14:31.565179+0000","last_clean":"2024-09-13T17:14:31.565179+0000","last_became_active":"2024-09-13T17:14:31.561065+0000","last_became_peered":"2024-09-13T17:14:31.561065+0000","last_unstale":"2024-09-13T17:14:31.565179+0000","last_undegraded":"2024-09-13T17:14:31.565179+0000","last_fullsized":"2024-09-13T17:14:31.565179+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:32.296423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553281+0000","last_change":"2024-09-13T17:14:31.547279+0000","last_active":"2024-09-13T17:14:31.553281+0000","last_peered":"2024-09-13T17:14:31.553281+0000","last_clean":"2024-09-13T17:14:31.553281+0000","last_became_active":"2024-09-13T17:14:31.547020+0000","last_became_peered":"2024-09-13T17:14:31.547020+0000","last_unstale":"2024-09-13T17:14:31.553281+0000","last_undegraded":"2024-09-13T17:14:31.553281+0000","last_fullsized":"2024-09-13T17:14:31.553281+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:01.512228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.563732+0000","last_change":"2024-09-13T17:14:31.548647+0000","last_active":"2024-09-13T17:14:31.563732+0000","last_peered":"2024-09-13T17:14:31.563732+0000","last_clean":"2024-09-13T17:14:31.563732+0000","last_became_active":"2024-09-13T17:14:31.547941+0000","last_became_peered":"2024-09-13T17:14:31.547941+0000","last_unstale":"2024-09-13T17:14:31.563732+0000","last_undegraded":"2024-09-13T17:14:31.563732+0000","last_fullsized":"2024-09-13T17:14:31.563732+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:43.431963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536634+0000","last_change":"2024-09-13T17:14:31.559545+0000","last_active":"2024-09-13T17:14:32.536634+0000","last_peered":"2024-09-13T17:14:32.536634+0000","last_clean":"2024-09-13T17:14:32.536634+0000","last_became_active":"2024-09-13T17:14:31.559058+0000","last_became_peered":"2024-09-13T17:14:31.559058+0000","last_unstale":"2024-09-13T17:14:32.536634+0000","last_undegraded":"2024-09-13T17:14:32.536634+0000","last_fullsized":"2024-09-13T17:14:32.536634+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:17.504567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564231+0000","last_change":"2024-09-13T17:14:31.560844+0000","last_active":"2024-09-13T17:14:31.564231+0000","last_peered":"2024-09-13T17:14:31.564231+0000","last_clean":"2024-09-13T17:14:31.564231+0000","last_became_active":"2024-09-13T17:14:31.556568+0000","last_became_peered":"2024-09-13T17:14:31.556568+0000","last_unstale":"2024-09-13T17:14:31.564231+0000","last_undegraded":"2024-09-13T17:14:31.564231+0000","last_fullsized":"2024-09-13T17:14:31.564231+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:07.892038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.544068+0000","last_change":"2024-09-13T17:14:31.541796+0000","last_active":"2024-09-13T17:14:31.544068+0000","last_peered":"2024-09-13T17:14:31.544068+0000","last_clean":"2024-09-13T17:14:31.544068+0000","last_became_active":"2024-09-13T17:14:31.541535+0000","last_became_peered":"2024-09-13T17:14:31.541535+0000","last_unstale":"2024-09-13T17:14:31.544068+0000","last_undegraded":"2024-09-13T17:14:31.544068+0000","last_fullsized":"2024-09-13T17:14:31.544068+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:13.983156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"110'4","reported_seq":14,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.554721+0000","last_change":"2024-09-13T17:14:31.118456+0000","last_active":"2024-09-13T17:14:31.554721+0000","last_peered":"2024-09-13T17:14:31.554721+0000","last_clean":"2024-09-13T17:14:31.554721+0000","last_became_active":"2024-09-13T17:14:31.118201+0000","last_became_peered":"2024-09-13T17:14:31.118201+0000","last_unstale":"2024-09-13T17:14:31.554721+0000","last_undegraded":"2024-09-13T17:14:31.554721+0000","last_fullsized":"2024-09-13T17:14:31.554721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:06.824037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536872+0000","last_change":"2024-09-13T17:14:31.559003+0000","last_active":"2024-09-13T17:14:32.536872+0000","last_peered":"2024-09-13T17:14:32.536872+0000","last_clean":"2024-09-13T17:14:32.536872+0000","last_became_active":"2024-09-13T17:14:31.558693+0000","last_became_peered":"2024-09-13T17:14:31.558693+0000","last_unstale":"2024-09-13T17:14:32.536872+0000","last_undegraded":"2024-09-13T17:14:32.536872+0000","last_fullsized":"2024-09-13T17:14:32.536872+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:11.549156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552723+0000","last_change":"2024-09-13T17:14:31.122718+0000","last_active":"2024-09-13T17:14:31.552723+0000","last_peered":"2024-09-13T17:14:31.552723+0000","last_clean":"2024-09-13T17:14:31.552723+0000","last_became_active":"2024-09-13T17:14:31.122312+0000","last_became_peered":"2024-09-13T17:14:31.122312+0000","last_unstale":"2024-09-13T17:14:31.552723+0000","last_undegraded":"2024-09-13T17:14:31.552723+0000","last_fullsized":"2024-09-13T17:14:31.552723+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:38.621021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561156+0000","last_change":"2024-09-13T17:14:31.552961+0000","last_active":"2024-09-13T17:14:31.561156+0000","last_peered":"2024-09-13T17:14:31.561156+0000","last_clean":"2024-09-13T17:14:31.561156+0000","last_became_active":"2024-09-13T17:14:31.552704+0000","last_became_peered":"2024-09-13T17:14:31.552704+0000","last_unstale":"2024-09-13T17:14:31.561156+0000","last_undegraded":"2024-09-13T17:14:31.561156+0000","last_fullsized":"2024-09-13T17:14:31.561156+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:19.702514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555597+0000","last_change":"2024-09-13T17:14:31.114818+0000","last_active":"2024-09-13T17:14:31.555597+0000","last_peered":"2024-09-13T17:14:31.555597+0000","last_clean":"2024-09-13T17:14:31.555597+0000","last_became_active":"2024-09-13T17:14:31.114530+0000","last_became_peered":"2024-09-13T17:14:31.114530+0000","last_unstale":"2024-09-13T17:14:31.555597+0000","last_undegraded":"2024-09-13T17:14:31.555597+0000","last_fullsized":"2024-09-13T17:14:31.555597+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:59.436683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541406+0000","last_change":"2024-09-13T17:14:31.550576+0000","last_active":"2024-09-13T17:14:32.541406+0000","last_peered":"2024-09-13T17:14:32.541406+0000","last_clean":"2024-09-13T17:14:32.541406+0000","last_became_active":"2024-09-13T17:14:31.549293+0000","last_became_peered":"2024-09-13T17:14:31.549293+0000","last_unstale":"2024-09-13T17:14:32.541406+0000","last_undegraded":"2024-09-13T17:14:32.541406+0000","last_fullsized":"2024-09-13T17:14:32.541406+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:15.361955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552370+0000","last_change":"2024-09-13T17:14:31.115204+0000","last_active":"2024-09-13T17:14:31.552370+0000","last_peered":"2024-09-13T17:14:31.552370+0000","last_clean":"2024-09-13T17:14:31.552370+0000","last_became_active":"2024-09-13T17:14:31.112326+0000","last_became_peered":"2024-09-13T17:14:31.112326+0000","last_unstale":"2024-09-13T17:14:31.552370+0000","last_undegraded":"2024-09-13T17:14:31.552370+0000","last_fullsized":"2024-09-13T17:14:31.552370+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:16.199940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541162+0000","last_change":"2024-09-13T17:14:31.554469+0000","last_active":"2024-09-13T17:14:32.541162+0000","last_peered":"2024-09-13T17:14:32.541162+0000","last_clean":"2024-09-13T17:14:32.541162+0000","last_became_active":"2024-09-13T17:14:31.552118+0000","last_became_peered":"2024-09-13T17:14:31.552118+0000","last_unstale":"2024-09-13T17:14:32.541162+0000","last_undegraded":"2024-09-13T17:14:32.541162+0000","last_fullsized":"2024-09-13T17:14:32.541162+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:57:07.493248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561417+0000","last_change":"2024-09-13T17:14:31.121466+0000","last_active":"2024-09-13T17:14:31.561417+0000","last_peered":"2024-09-13T17:14:31.561417+0000","last_clean":"2024-09-13T17:14:31.561417+0000","last_became_active":"2024-09-13T17:14:31.121160+0000","last_became_peered":"2024-09-13T17:14:31.121160+0000","last_unstale":"2024-09-13T17:14:31.561417+0000","last_undegraded":"2024-09-13T17:14:31.561417+0000","last_fullsized":"2024-09-13T17:14:31.561417+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:02.935337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.557269+0000","last_change":"2024-09-13T17:14:31.555644+0000","last_active":"2024-09-13T17:14:31.557269+0000","last_peered":"2024-09-13T17:14:31.557269+0000","last_clean":"2024-09-13T17:14:31.557269+0000","last_became_active":"2024-09-13T17:14:31.555398+0000","last_became_peered":"2024-09-13T17:14:31.555398+0000","last_unstale":"2024-09-13T17:14:31.557269+0000","last_undegraded":"2024-09-13T17:14:31.557269+0000","last_fullsized":"2024-09-13T17:14:31.557269+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:28.866955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.551398+0000","last_change":"2024-09-13T17:14:31.123721+0000","last_active":"2024-09-13T17:14:31.551398+0000","last_peered":"2024-09-13T17:14:31.551398+0000","last_clean":"2024-09-13T17:14:31.551398+0000","last_became_active":"2024-09-13T17:14:31.123334+0000","last_became_peered":"2024-09-13T17:14:31.123334+0000","last_unstale":"2024-09-13T17:14:31.551398+0000","last_undegraded":"2024-09-13T17:14:31.551398+0000","last_fullsized":"2024-09-13T17:14:31.551398+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:15.591329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536451+0000","last_change":"2024-09-13T17:14:31.555508+0000","last_active":"2024-09-13T17:14:32.536451+0000","last_peered":"2024-09-13T17:14:32.536451+0000","last_clean":"2024-09-13T17:14:32.536451+0000","last_became_active":"2024-09-13T17:14:31.555217+0000","last_became_peered":"2024-09-13T17:14:31.555217+0000","last_unstale":"2024-09-13T17:14:32.536451+0000","last_undegraded":"2024-09-13T17:14:32.536451+0000","last_fullsized":"2024-09-13T17:14:32.536451+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:15.793328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550630+0000","last_change":"2024-09-13T17:14:31.112196+0000","last_active":"2024-09-13T17:14:31.550630+0000","last_peered":"2024-09-13T17:14:31.550630+0000","last_clean":"2024-09-13T17:14:31.550630+0000","last_became_active":"2024-09-13T17:14:31.111867+0000","last_became_peered":"2024-09-13T17:14:31.111867+0000","last_unstale":"2024-09-13T17:14:31.550630+0000","last_undegraded":"2024-09-13T17:14:31.550630+0000","last_fullsized":"2024-09-13T17:14:31.550630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:38.621021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541720+0000","last_change":"2024-09-13T17:14:31.556238+0000","last_active":"2024-09-13T17:14:32.541720+0000","last_peered":"2024-09-13T17:14:32.541720+0000","last_clean":"2024-09-13T17:14:32.541720+0000","last_became_active":"2024-09-13T17:14:31.555892+0000","last_became_peered":"2024-09-13T17:14:31.555892+0000","last_unstale":"2024-09-13T17:14:32.541720+0000","last_undegraded":"2024-09-13T17:14:32.541720+0000","last_fullsized":"2024-09-13T17:14:32.541720+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:30.854130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.539453+0000","last_change":"2024-09-13T17:14:31.122267+0000","last_active":"2024-09-13T17:14:31.539453+0000","last_peered":"2024-09-13T17:14:31.539453+0000","last_clean":"2024-09-13T17:14:31.539453+0000","last_became_active":"2024-09-13T17:14:31.122018+0000","last_became_peered":"2024-09-13T17:14:31.122018+0000","last_unstale":"2024-09-13T17:14:31.539453+0000","last_undegraded":"2024-09-13T17:14:31.539453+0000","last_fullsized":"2024-09-13T17:14:31.539453+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:41.562217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.555699+0000","last_change":"2024-09-13T17:14:31.553095+0000","last_active":"2024-09-13T17:14:31.555699+0000","last_peered":"2024-09-13T17:14:31.555699+0000","last_clean":"2024-09-13T17:14:31.555699+0000","last_became_active":"2024-09-13T17:14:31.552671+0000","last_became_peered":"2024-09-13T17:14:31.552671+0000","last_unstale":"2024-09-13T17:14:31.555699+0000","last_undegraded":"2024-09-13T17:14:31.555699+0000","last_fullsized":"2024-09-13T17:14:31.555699+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:14.140980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.543026+0000","last_change":"2024-09-13T17:14:31.115979+0000","last_active":"2024-09-13T17:14:31.543026+0000","last_peered":"2024-09-13T17:14:31.543026+0000","last_clean":"2024-09-13T17:14:31.543026+0000","last_became_active":"2024-09-13T17:14:31.115730+0000","last_became_peered":"2024-09-13T17:14:31.115730+0000","last_unstale":"2024-09-13T17:14:31.543026+0000","last_undegraded":"2024-09-13T17:14:31.543026+0000","last_fullsized":"2024-09-13T17:14:31.543026+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:17.192176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541641+0000","last_change":"2024-09-13T17:14:31.556403+0000","last_active":"2024-09-13T17:14:32.541641+0000","last_peered":"2024-09-13T17:14:32.541641+0000","last_clean":"2024-09-13T17:14:32.541641+0000","last_became_active":"2024-09-13T17:14:31.556104+0000","last_became_peered":"2024-09-13T17:14:31.556104+0000","last_unstale":"2024-09-13T17:14:32.541641+0000","last_undegraded":"2024-09-13T17:14:32.541641+0000","last_fullsized":"2024-09-13T17:14:32.541641+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:59.045988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.537556+0000","last_change":"2024-09-13T17:14:31.121706+0000","last_active":"2024-09-13T17:14:32.537556+0000","last_peered":"2024-09-13T17:14:32.537556+0000","last_clean":"2024-09-13T17:14:32.537556+0000","last_became_active":"2024-09-13T17:14:31.121396+0000","last_became_peered":"2024-09-13T17:14:31.121396+0000","last_unstale":"2024-09-13T17:14:32.537556+0000","last_undegraded":"2024-09-13T17:14:32.537556+0000","last_fullsized":"2024-09-13T17:14:32.537556+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:31.293801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561669+0000","last_change":"2024-09-13T17:14:31.555509+0000","last_active":"2024-09-13T17:14:31.561669+0000","last_peered":"2024-09-13T17:14:31.561669+0000","last_clean":"2024-09-13T17:14:31.561669+0000","last_became_active":"2024-09-13T17:14:31.555152+0000","last_became_peered":"2024-09-13T17:14:31.555152+0000","last_unstale":"2024-09-13T17:14:31.561669+0000","last_undegraded":"2024-09-13T17:14:31.561669+0000","last_fullsized":"2024-09-13T17:14:31.561669+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:17.202561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"110'3","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.552852+0000","last_change":"2024-09-13T17:14:31.114295+0000","last_active":"2024-09-13T17:14:31.552852+0000","last_peered":"2024-09-13T17:14:31.552852+0000","last_clean":"2024-09-13T17:14:31.552852+0000","last_became_active":"2024-09-13T17:14:31.113507+0000","last_became_peered":"2024-09-13T17:14:31.113507+0000","last_unstale":"2024-09-13T17:14:31.552852+0000","last_undegraded":"2024-09-13T17:14:31.552852+0000","last_fullsized":"2024-09-13T17:14:31.552852+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:11.627636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549934+0000","last_change":"2024-09-13T17:14:31.548810+0000","last_active":"2024-09-13T17:14:31.549934+0000","last_peered":"2024-09-13T17:14:31.549934+0000","last_clean":"2024-09-13T17:14:31.549934+0000","last_became_active":"2024-09-13T17:14:31.547135+0000","last_became_peered":"2024-09-13T17:14:31.547135+0000","last_unstale":"2024-09-13T17:14:31.549934+0000","last_undegraded":"2024-09-13T17:14:31.549934+0000","last_fullsized":"2024-09-13T17:14:31.549934+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:17.202561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"110'6","reported_seq":17,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541559+0000","last_change":"2024-09-13T17:14:31.109566+0000","last_active":"2024-09-13T17:14:32.541559+0000","last_peered":"2024-09-13T17:14:32.541559+0000","last_clean":"2024-09-13T17:14:32.541559+0000","last_became_active":"2024-09-13T17:14:31.109298+0000","last_became_peered":"2024-09-13T17:14:31.109298+0000","last_unstale":"2024-09-13T17:14:32.541559+0000","last_undegraded":"2024-09-13T17:14:32.541559+0000","last_fullsized":"2024-09-13T17:14:32.541559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:06.824037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536633+0000","last_change":"2024-09-13T17:14:31.550987+0000","last_active":"2024-09-13T17:14:32.536633+0000","last_peered":"2024-09-13T17:14:32.536633+0000","last_clean":"2024-09-13T17:14:32.536633+0000","last_became_active":"2024-09-13T17:14:31.548999+0000","last_became_peered":"2024-09-13T17:14:31.548999+0000","last_unstale":"2024-09-13T17:14:32.536633+0000","last_undegraded":"2024-09-13T17:14:32.536633+0000","last_fullsized":"2024-09-13T17:14:32.536633+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:10:15.361955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564946+0000","last_change":"2024-09-13T17:14:31.104085+0000","last_active":"2024-09-13T17:14:31.564946+0000","last_peered":"2024-09-13T17:14:31.564946+0000","last_clean":"2024-09-13T17:14:31.564946+0000","last_became_active":"2024-09-13T17:14:31.102170+0000","last_became_peered":"2024-09-13T17:14:31.102170+0000","last_unstale":"2024-09-13T17:14:31.564946+0000","last_undegraded":"2024-09-13T17:14:31.564946+0000","last_fullsized":"2024-09-13T17:14:31.564946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:08:37.091849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.548791+0000","last_change":"2024-09-13T17:14:31.544359+0000","last_active":"2024-09-13T17:14:31.548791+0000","last_peered":"2024-09-13T17:14:31.548791+0000","last_clean":"2024-09-13T17:14:31.548791+0000","last_became_active":"2024-09-13T17:14:31.543530+0000","last_became_peered":"2024-09-13T17:14:31.543530+0000","last_unstale":"2024-09-13T17:14:31.548791+0000","last_undegraded":"2024-09-13T17:14:31.548791+0000","last_fullsized":"2024-09-13T17:14:31.548791+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:02.661186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.531892+0000","last_change":"2024-09-13T17:14:31.117012+0000","last_active":"2024-09-13T17:14:31.531892+0000","last_peered":"2024-09-13T17:14:31.531892+0000","last_clean":"2024-09-13T17:14:31.531892+0000","last_became_active":"2024-09-13T17:14:31.116735+0000","last_became_peered":"2024-09-13T17:14:31.116735+0000","last_unstale":"2024-09-13T17:14:31.531892+0000","last_undegraded":"2024-09-13T17:14:31.531892+0000","last_fullsized":"2024-09-13T17:14:31.531892+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:29:04.459517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540716+0000","last_change":"2024-09-13T17:14:31.555616+0000","last_active":"2024-09-13T17:14:32.540716+0000","last_peered":"2024-09-13T17:14:32.540716+0000","last_clean":"2024-09-13T17:14:32.540716+0000","last_became_active":"2024-09-13T17:14:31.555338+0000","last_became_peered":"2024-09-13T17:14:31.555338+0000","last_unstale":"2024-09-13T17:14:32.540716+0000","last_undegraded":"2024-09-13T17:14:32.540716+0000","last_fullsized":"2024-09-13T17:14:32.540716+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:17.504567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540993+0000","last_change":"2024-09-13T17:14:31.102542+0000","last_active":"2024-09-13T17:14:32.540993+0000","last_peered":"2024-09-13T17:14:32.540993+0000","last_clean":"2024-09-13T17:14:32.540993+0000","last_became_active":"2024-09-13T17:14:31.102250+0000","last_became_peered":"2024-09-13T17:14:31.102250+0000","last_unstale":"2024-09-13T17:14:32.540993+0000","last_undegraded":"2024-09-13T17:14:32.540993+0000","last_fullsized":"2024-09-13T17:14:32.540993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:53.228895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564623+0000","last_change":"2024-09-13T17:14:31.555418+0000","last_active":"2024-09-13T17:14:31.564623+0000","last_peered":"2024-09-13T17:14:31.564623+0000","last_clean":"2024-09-13T17:14:31.564623+0000","last_became_active":"2024-09-13T17:14:31.555110+0000","last_became_peered":"2024-09-13T17:14:31.555110+0000","last_unstale":"2024-09-13T17:14:31.564623+0000","last_undegraded":"2024-09-13T17:14:31.564623+0000","last_fullsized":"2024-09-13T17:14:31.564623+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:10.870738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"110'8","reported_seq":18,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.568077+0000","last_change":"2024-09-13T17:14:31.122012+0000","last_active":"2024-09-13T17:14:31.568077+0000","last_peered":"2024-09-13T17:14:31.568077+0000","last_clean":"2024-09-13T17:14:31.568077+0000","last_became_active":"2024-09-13T17:14:31.121741+0000","last_became_peered":"2024-09-13T17:14:31.121741+0000","last_unstale":"2024-09-13T17:14:31.568077+0000","last_undegraded":"2024-09-13T17:14:31.568077+0000","last_fullsized":"2024-09-13T17:14:31.568077+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:02.015241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561983+0000","last_change":"2024-09-13T17:14:31.556640+0000","last_active":"2024-09-13T17:14:31.561983+0000","last_peered":"2024-09-13T17:14:31.561983+0000","last_clean":"2024-09-13T17:14:31.561983+0000","last_became_active":"2024-09-13T17:14:31.556369+0000","last_became_peered":"2024-09-13T17:14:31.556369+0000","last_unstale":"2024-09-13T17:14:31.561983+0000","last_undegraded":"2024-09-13T17:14:31.561983+0000","last_fullsized":"2024-09-13T17:14:31.561983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:41.645450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541095+0000","last_change":"2024-09-13T17:14:31.108877+0000","last_active":"2024-09-13T17:14:32.541095+0000","last_peered":"2024-09-13T17:14:32.541095+0000","last_clean":"2024-09-13T17:14:32.541095+0000","last_became_active":"2024-09-13T17:14:31.108618+0000","last_became_peered":"2024-09-13T17:14:31.108618+0000","last_unstale":"2024-09-13T17:14:32.541095+0000","last_undegraded":"2024-09-13T17:14:32.541095+0000","last_fullsized":"2024-09-13T17:14:32.541095+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:35.614965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541002+0000","last_change":"2024-09-13T17:14:31.559173+0000","last_active":"2024-09-13T17:14:32.541002+0000","last_peered":"2024-09-13T17:14:32.541002+0000","last_clean":"2024-09-13T17:14:32.541002+0000","last_became_active":"2024-09-13T17:14:31.558893+0000","last_became_peered":"2024-09-13T17:14:31.558893+0000","last_unstale":"2024-09-13T17:14:32.541002+0000","last_undegraded":"2024-09-13T17:14:32.541002+0000","last_fullsized":"2024-09-13T17:14:32.541002+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:33.680681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.559630+0000","last_change":"2024-09-13T17:14:31.111472+0000","last_active":"2024-09-13T17:14:31.559630+0000","last_peered":"2024-09-13T17:14:31.559630+0000","last_clean":"2024-09-13T17:14:31.559630+0000","last_became_active":"2024-09-13T17:14:31.111120+0000","last_became_peered":"2024-09-13T17:14:31.111120+0000","last_unstale":"2024-09-13T17:14:31.559630+0000","last_undegraded":"2024-09-13T17:14:31.559630+0000","last_fullsized":"2024-09-13T17:14:31.559630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.549917+0000","last_change":"2024-09-13T17:14:31.547183+0000","last_active":"2024-09-13T17:14:31.549917+0000","last_peered":"2024-09-13T17:14:31.549917+0000","last_clean":"2024-09-13T17:14:31.549917+0000","last_became_active":"2024-09-13T17:14:31.546907+0000","last_became_peered":"2024-09-13T17:14:31.546907+0000","last_unstale":"2024-09-13T17:14:31.549917+0000","last_undegraded":"2024-09-13T17:14:31.549917+0000","last_fullsized":"2024-09-13T17:14:31.549917+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:54.231517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.535382+0000","last_change":"2024-09-13T17:14:31.113994+0000","last_active":"2024-09-13T17:14:31.535382+0000","last_peered":"2024-09-13T17:14:31.535382+0000","last_clean":"2024-09-13T17:14:31.535382+0000","last_became_active":"2024-09-13T17:14:31.113246+0000","last_became_peered":"2024-09-13T17:14:31.113246+0000","last_unstale":"2024-09-13T17:14:31.535382+0000","last_undegraded":"2024-09-13T17:14:31.535382+0000","last_fullsized":"2024-09-13T17:14:31.535382+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:31.293801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565753+0000","last_change":"2024-09-13T17:14:31.562193+0000","last_active":"2024-09-13T17:14:31.565753+0000","last_peered":"2024-09-13T17:14:31.565753+0000","last_clean":"2024-09-13T17:14:31.565753+0000","last_became_active":"2024-09-13T17:14:31.557845+0000","last_became_peered":"2024-09-13T17:14:31.557845+0000","last_unstale":"2024-09-13T17:14:31.565753+0000","last_undegraded":"2024-09-13T17:14:31.565753+0000","last_fullsized":"2024-09-13T17:14:31.565753+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:54.059861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.546772+0000","last_change":"2024-09-13T17:14:31.114167+0000","last_active":"2024-09-13T17:14:31.546772+0000","last_peered":"2024-09-13T17:14:31.546772+0000","last_clean":"2024-09-13T17:14:31.546772+0000","last_became_active":"2024-09-13T17:14:31.113810+0000","last_became_peered":"2024-09-13T17:14:31.113810+0000","last_unstale":"2024-09-13T17:14:31.546772+0000","last_undegraded":"2024-09-13T17:14:31.546772+0000","last_fullsized":"2024-09-13T17:14:31.546772+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:27.864333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.564773+0000","last_change":"2024-09-13T17:14:31.550030+0000","last_active":"2024-09-13T17:14:31.564773+0000","last_peered":"2024-09-13T17:14:31.564773+0000","last_clean":"2024-09-13T17:14:31.564773+0000","last_became_active":"2024-09-13T17:14:31.549705+0000","last_became_peered":"2024-09-13T17:14:31.549705+0000","last_unstale":"2024-09-13T17:14:31.564773+0000","last_undegraded":"2024-09-13T17:14:31.564773+0000","last_fullsized":"2024-09-13T17:14:31.564773+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:18.263370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541248+0000","last_change":"2024-09-13T17:14:31.120749+0000","last_active":"2024-09-13T17:14:32.541248+0000","last_peered":"2024-09-13T17:14:32.541248+0000","last_clean":"2024-09-13T17:14:32.541248+0000","last_became_active":"2024-09-13T17:14:31.120496+0000","last_became_peered":"2024-09-13T17:14:31.120496+0000","last_unstale":"2024-09-13T17:14:32.541248+0000","last_undegraded":"2024-09-13T17:14:32.541248+0000","last_fullsized":"2024-09-13T17:14:32.541248+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.115575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565582+0000","last_change":"2024-09-13T17:14:31.562716+0000","last_active":"2024-09-13T17:14:31.565582+0000","last_peered":"2024-09-13T17:14:31.565582+0000","last_clean":"2024-09-13T17:14:31.565582+0000","last_became_active":"2024-09-13T17:14:31.556196+0000","last_became_peered":"2024-09-13T17:14:31.556196+0000","last_unstale":"2024-09-13T17:14:31.565582+0000","last_undegraded":"2024-09-13T17:14:31.565582+0000","last_fullsized":"2024-09-13T17:14:31.565582+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:03.937959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541537+0000","last_change":"2024-09-13T17:14:31.113229+0000","last_active":"2024-09-13T17:14:32.541537+0000","last_peered":"2024-09-13T17:14:32.541537+0000","last_clean":"2024-09-13T17:14:32.541537+0000","last_became_active":"2024-09-13T17:14:31.112956+0000","last_became_peered":"2024-09-13T17:14:31.112956+0000","last_unstale":"2024-09-13T17:14:32.541537+0000","last_undegraded":"2024-09-13T17:14:32.541537+0000","last_fullsized":"2024-09-13T17:14:32.541537+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:46.814882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.562161+0000","last_change":"2024-09-13T17:14:31.558135+0000","last_active":"2024-09-13T17:14:31.562161+0000","last_peered":"2024-09-13T17:14:31.562161+0000","last_clean":"2024-09-13T17:14:31.562161+0000","last_became_active":"2024-09-13T17:14:31.557891+0000","last_became_peered":"2024-09-13T17:14:31.557891+0000","last_unstale":"2024-09-13T17:14:31.562161+0000","last_undegraded":"2024-09-13T17:14:31.562161+0000","last_fullsized":"2024-09-13T17:14:31.562161+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:40.774196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.538433+0000","last_change":"2024-09-13T17:14:31.124389+0000","last_active":"2024-09-13T17:14:31.538433+0000","last_peered":"2024-09-13T17:14:31.538433+0000","last_clean":"2024-09-13T17:14:31.538433+0000","last_became_active":"2024-09-13T17:14:31.124070+0000","last_became_peered":"2024-09-13T17:14:31.124070+0000","last_unstale":"2024-09-13T17:14:31.538433+0000","last_undegraded":"2024-09-13T17:14:31.538433+0000","last_fullsized":"2024-09-13T17:14:31.538433+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:15.521226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.553591+0000","last_change":"2024-09-13T17:14:31.547532+0000","last_active":"2024-09-13T17:14:31.553591+0000","last_peered":"2024-09-13T17:14:31.553591+0000","last_clean":"2024-09-13T17:14:31.553591+0000","last_became_active":"2024-09-13T17:14:31.546114+0000","last_became_peered":"2024-09-13T17:14:31.546114+0000","last_unstale":"2024-09-13T17:14:31.553591+0000","last_undegraded":"2024-09-13T17:14:31.553591+0000","last_fullsized":"2024-09-13T17:14:31.553591+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:31.979640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.541313+0000","last_change":"2024-09-13T17:14:31.112592+0000","last_active":"2024-09-13T17:14:32.541313+0000","last_peered":"2024-09-13T17:14:32.541313+0000","last_clean":"2024-09-13T17:14:32.541313+0000","last_became_active":"2024-09-13T17:14:31.108144+0000","last_became_peered":"2024-09-13T17:14:31.108144+0000","last_unstale":"2024-09-13T17:14:32.541313+0000","last_undegraded":"2024-09-13T17:14:32.541313+0000","last_fullsized":"2024-09-13T17:14:32.541313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:08.871649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.566636+0000","last_change":"2024-09-13T17:14:31.564944+0000","last_active":"2024-09-13T17:14:31.566636+0000","last_peered":"2024-09-13T17:14:31.566636+0000","last_clean":"2024-09-13T17:14:31.566636+0000","last_became_active":"2024-09-13T17:14:31.564690+0000","last_became_peered":"2024-09-13T17:14:31.564690+0000","last_unstale":"2024-09-13T17:14:31.566636+0000","last_undegraded":"2024-09-13T17:14:31.566636+0000","last_fullsized":"2024-09-13T17:14:31.566636+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:54.660491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.536048+0000","last_change":"2024-09-13T17:14:31.120433+0000","last_active":"2024-09-13T17:14:31.536048+0000","last_peered":"2024-09-13T17:14:31.536048+0000","last_clean":"2024-09-13T17:14:31.536048+0000","last_became_active":"2024-09-13T17:14:31.119944+0000","last_became_peered":"2024-09-13T17:14:31.119944+0000","last_unstale":"2024-09-13T17:14:31.536048+0000","last_undegraded":"2024-09-13T17:14:31.536048+0000","last_fullsized":"2024-09-13T17:14:31.536048+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:19.907478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565170+0000","last_change":"2024-09-13T17:14:31.562704+0000","last_active":"2024-09-13T17:14:31.565170+0000","last_peered":"2024-09-13T17:14:31.565170+0000","last_clean":"2024-09-13T17:14:31.565170+0000","last_became_active":"2024-09-13T17:14:31.562417+0000","last_became_peered":"2024-09-13T17:14:31.562417+0000","last_unstale":"2024-09-13T17:14:31.565170+0000","last_undegraded":"2024-09-13T17:14:31.565170+0000","last_fullsized":"2024-09-13T17:14:31.565170+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:09.874271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.535399+0000","last_change":"2024-09-13T17:14:31.114224+0000","last_active":"2024-09-13T17:14:31.535399+0000","last_peered":"2024-09-13T17:14:31.535399+0000","last_clean":"2024-09-13T17:14:31.535399+0000","last_became_active":"2024-09-13T17:14:31.112368+0000","last_became_peered":"2024-09-13T17:14:31.112368+0000","last_unstale":"2024-09-13T17:14:31.535399+0000","last_undegraded":"2024-09-13T17:14:31.535399+0000","last_fullsized":"2024-09-13T17:14:31.535399+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:39.771574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561850+0000","last_change":"2024-09-13T17:14:31.556231+0000","last_active":"2024-09-13T17:14:31.561850+0000","last_peered":"2024-09-13T17:14:31.561850+0000","last_clean":"2024-09-13T17:14:31.561850+0000","last_became_active":"2024-09-13T17:14:31.555741+0000","last_became_peered":"2024-09-13T17:14:31.555741+0000","last_unstale":"2024-09-13T17:14:31.561850+0000","last_undegraded":"2024-09-13T17:14:31.561850+0000","last_fullsized":"2024-09-13T17:14:31.561850+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:54.231517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.550977+0000","last_change":"2024-09-13T17:14:31.109595+0000","last_active":"2024-09-13T17:14:31.550977+0000","last_peered":"2024-09-13T17:14:31.550977+0000","last_clean":"2024-09-13T17:14:31.550977+0000","last_became_active":"2024-09-13T17:14:31.109310+0000","last_became_peered":"2024-09-13T17:14:31.109310+0000","last_unstale":"2024-09-13T17:14:31.550977+0000","last_undegraded":"2024-09-13T17:14:31.550977+0000","last_fullsized":"2024-09-13T17:14:31.550977+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:01:52.115575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565146+0000","last_change":"2024-09-13T17:14:31.558116+0000","last_active":"2024-09-13T17:14:31.565146+0000","last_peered":"2024-09-13T17:14:31.565146+0000","last_clean":"2024-09-13T17:14:31.565146+0000","last_became_active":"2024-09-13T17:14:31.557480+0000","last_became_peered":"2024-09-13T17:14:31.557480+0000","last_unstale":"2024-09-13T17:14:31.565146+0000","last_undegraded":"2024-09-13T17:14:31.565146+0000","last_fullsized":"2024-09-13T17:14:31.565146+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:43.674903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.536566+0000","last_change":"2024-09-13T17:14:31.114083+0000","last_active":"2024-09-13T17:14:31.536566+0000","last_peered":"2024-09-13T17:14:31.536566+0000","last_clean":"2024-09-13T17:14:31.536566+0000","last_became_active":"2024-09-13T17:14:31.113479+0000","last_became_peered":"2024-09-13T17:14:31.113479+0000","last_unstale":"2024-09-13T17:14:31.536566+0000","last_undegraded":"2024-09-13T17:14:31.536566+0000","last_fullsized":"2024-09-13T17:14:31.536566+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:13.698999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.540907+0000","last_change":"2024-09-13T17:14:31.559346+0000","last_active":"2024-09-13T17:14:32.540907+0000","last_peered":"2024-09-13T17:14:32.540907+0000","last_clean":"2024-09-13T17:14:32.540907+0000","last_became_active":"2024-09-13T17:14:31.558649+0000","last_became_peered":"2024-09-13T17:14:31.558649+0000","last_unstale":"2024-09-13T17:14:32.540907+0000","last_undegraded":"2024-09-13T17:14:32.540907+0000","last_fullsized":"2024-09-13T17:14:32.540907+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:14.125462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.551533+0000","last_change":"2024-09-13T17:14:31.116820+0000","last_active":"2024-09-13T17:14:31.551533+0000","last_peered":"2024-09-13T17:14:31.551533+0000","last_clean":"2024-09-13T17:14:31.551533+0000","last_became_active":"2024-09-13T17:14:31.116495+0000","last_became_peered":"2024-09-13T17:14:31.116495+0000","last_unstale":"2024-09-13T17:14:31.551533+0000","last_undegraded":"2024-09-13T17:14:31.551533+0000","last_fullsized":"2024-09-13T17:14:31.551533+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:11.627636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565564+0000","last_change":"2024-09-13T17:14:31.562555+0000","last_active":"2024-09-13T17:14:31.565564+0000","last_peered":"2024-09-13T17:14:31.565564+0000","last_clean":"2024-09-13T17:14:31.565564+0000","last_became_active":"2024-09-13T17:14:31.555855+0000","last_became_peered":"2024-09-13T17:14:31.555855+0000","last_unstale":"2024-09-13T17:14:31.565564+0000","last_undegraded":"2024-09-13T17:14:31.565564+0000","last_fullsized":"2024-09-13T17:14:31.565564+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:34:53.200392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T17:14:32.536533+0000","last_change":"2024-09-13T17:14:31.122578+0000","last_active":"2024-09-13T17:14:32.536533+0000","last_peered":"2024-09-13T17:14:32.536533+0000","last_clean":"2024-09-13T17:14:32.536533+0000","last_became_active":"2024-09-13T17:14:31.122299+0000","last_became_peered":"2024-09-13T17:14:31.122299+0000","last_unstale":"2024-09-13T17:14:32.536533+0000","last_undegraded":"2024-09-13T17:14:32.536533+0000","last_fullsized":"2024-09-13T17:14:32.536533+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:15.521226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.561199+0000","last_change":"2024-09-13T17:14:31.555934+0000","last_active":"2024-09-13T17:14:31.561199+0000","last_peered":"2024-09-13T17:14:31.561199+0000","last_clean":"2024-09-13T17:14:31.561199+0000","last_became_active":"2024-09-13T17:14:31.553557+0000","last_became_peered":"2024-09-13T17:14:31.553557+0000","last_unstale":"2024-09-13T17:14:31.561199+0000","last_undegraded":"2024-09-13T17:14:31.561199+0000","last_fullsized":"2024-09-13T17:14:31.561199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:38:16.523848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.531428+0000","last_change":"2024-09-13T17:14:31.118072+0000","last_active":"2024-09-13T17:14:31.531428+0000","last_peered":"2024-09-13T17:14:31.531428+0000","last_clean":"2024-09-13T17:14:31.531428+0000","last_became_active":"2024-09-13T17:14:31.117799+0000","last_became_peered":"2024-09-13T17:14:31.117799+0000","last_unstale":"2024-09-13T17:14:31.531428+0000","last_undegraded":"2024-09-13T17:14:31.531428+0000","last_fullsized":"2024-09-13T17:14:31.531428+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:12.200022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.565572+0000","last_change":"2024-09-13T17:14:31.561149+0000","last_active":"2024-09-13T17:14:31.565572+0000","last_peered":"2024-09-13T17:14:31.565572+0000","last_clean":"2024-09-13T17:14:31.565572+0000","last_became_active":"2024-09-13T17:14:31.560912+0000","last_became_peered":"2024-09-13T17:14:31.560912+0000","last_unstale":"2024-09-13T17:14:31.565572+0000","last_undegraded":"2024-09-13T17:14:31.565572+0000","last_fullsized":"2024-09-13T17:14:31.565572+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:31.088837+0000","last_clean_scrub_stamp":"2024-09-13T17:14:31.088837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:55.558600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.559380+0000","last_change":"2024-09-13T17:14:31.115877+0000","last_active":"2024-09-13T17:14:31.559380+0000","last_peered":"2024-09-13T17:14:31.559380+0000","last_clean":"2024-09-13T17:14:31.559380+0000","last_became_active":"2024-09-13T17:14:31.115620+0000","last_became_peered":"2024-09-13T17:14:31.115620+0000","last_unstale":"2024-09-13T17:14:31.559380+0000","last_undegraded":"2024-09-13T17:14:31.559380+0000","last_fullsized":"2024-09-13T17:14:31.559380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:14:30.086215+0000","last_clean_scrub_stamp":"2024-09-13T17:14:30.086215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:22:26.273416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":240,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-13T17:14:31.107852+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:14:31.107852+0000","last_peered":"2024-09-13T17:14:31.107852+0000","last_clean":"2024-09-13T17:14:31.107852+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:14:31.107852+0000","last_undegraded":"2024-09-13T17:14:31.107852+0000","last_fullsized":"2024-09-13T17:14:31.107852+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574930,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63376,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94308464,"statfs":{"total":96636764160,"available":96571867136,"internally_reserved":0,"allocated":1961984,"data_stored":1184160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":62912971},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.69099999999999995,"15min":0.65500000000000003},"min":{"1min":0.51800000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.70399999999999996,"15min":0.68799999999999994},"min":{"1min":0.47599999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":2.6739999999999999,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.69599999999999995,"15min":0.65200000000000002},"min":{"1min":0.53200000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.109,"5min":0.78300000000000003,"15min":0.71699999999999997},"min":{"1min":0.49099999999999999,"5min":0.437,"15min":0.437},"max":{"1min":6.4909999999999997,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.64600000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.70699999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.72599999999999998,"15min":0.71799999999999997},"min":{"1min":0.57199999999999995,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.70499999999999996,"15min":0.68899999999999995},"min":{"1min":0.55800000000000005,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.4249999999999998,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.5},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.78900000000000003,"15min":0.749},"min":{"1min":0.57999999999999996,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.71799999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61499999999999999,"15min":0.65500000000000003},"min":{"1min":0.49199999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.879,"5min":1.054,"15min":1.054},"last":0.621},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66300000000000003,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89400000000000002,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84599999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65400000000000003,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94899999999999995,"5min":1.143,"15min":1.143},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76200000000000001,"15min":0.78100000000000003},"min":{"1min":0.67700000000000005,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.92000000000000004,"5min":1.198,"15min":1.337},"last":0.74199999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.73799999999999999,"15min":0.76600000000000001},"min":{"1min":0.53200000000000003,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.878},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.749,"15min":0.77600000000000002},"min":{"1min":0.64000000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.97199999999999998,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.77300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574931,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40644,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331196,"statfs":{"total":96636764160,"available":96595144704,"internally_reserved":0,"allocated":1368064,"data_stored":594332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.71099999999999997,"15min":0.66600000000000004},"min":{"1min":0.36299999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.97099999999999997,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":0.878},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.746,"15min":0.72799999999999998},"min":{"1min":0.47199999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94799999999999995,"5min":3.1709999999999998,"15min":3.1709999999999998},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.69699999999999995,"15min":0.65300000000000002},"min":{"1min":0.41599999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91500000000000004,"5min":3.7040000000000002,"15min":3.7040000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69999999999999996,"15min":0.64600000000000002},"min":{"1min":0.54300000000000004,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.86699999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.70899999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.65400000000000003,"15min":0.67000000000000004},"min":{"1min":0.47099999999999997,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":1.2909999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72699999999999998,"15min":0.69399999999999995},"min":{"1min":0.56499999999999995,"5min":0.439,"15min":0.439},"max":{"1min":1.004,"5min":3.2629999999999999,"15min":3.2629999999999999},"last":0.55800000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71699999999999997,"15min":0.70699999999999996},"min":{"1min":0.51700000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.2609999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71299999999999997,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.92700000000000005,"5min":3.742,"15min":3.742},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.624,"15min":0.628},"min":{"1min":0.51800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.96499999999999997,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.65600000000000003},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.90300000000000002,"5min":4.7750000000000004,"15min":4.7750000000000004},"last":0.624}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70199999999999996,"15min":0.72599999999999998},"min":{"1min":0.51300000000000001,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91400000000000003,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68799999999999994,"15min":0.71999999999999997},"min":{"1min":0.54900000000000004,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.89800000000000002,"5min":3.609,"15min":3.609},"last":0.72399999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.751,"15min":0.753},"min":{"1min":0.54500000000000004,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95199999999999996,"5min":4.9669999999999996,"15min":4.9669999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.80700000000000005,"15min":0.80800000000000005},"min":{"1min":0.64500000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0069999999999999,"5min":3.8100000000000001,"15min":3.8100000000000001},"last":0.75600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574930,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40648,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331192,"statfs":{"total":96636764160,"available":96595140608,"internally_reserved":0,"allocated":1363968,"data_stored":593750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.98999999999999999,"15min":0.77800000000000002},"min":{"1min":0.435,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.79000000000000004,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.86099999999999999,"15min":0.76700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.83199999999999996,"15min":0.73799999999999999},"min":{"1min":0.437,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.91400000000000003,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.71799999999999997,"15min":0.70399999999999996},"min":{"1min":0.41899999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85199999999999998,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.84499999999999997,"15min":0.76300000000000001},"min":{"1min":0.52100000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.84299999999999997,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.77800000000000002,"15min":0.75600000000000001},"min":{"1min":0.52000000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.89800000000000002,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.72499999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":1.792,"15min":1.107},"min":{"1min":0.51500000000000001,"5min":0.435,"15min":0.435},"max":{"1min":0.876,"5min":113.363,"15min":113.363},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75800000000000001,"15min":0.76900000000000002},"min":{"1min":0.51500000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92800000000000005,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.627}]},{"osd":4,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.628,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86699999999999999,"5min":1.0600000000000001,"15min":1.0600000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.68899999999999995,"15min":0.71999999999999997},"min":{"1min":0.442,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.88100000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.71899999999999997,"15min":0.70699999999999996},"min":{"1min":0.50800000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.94999999999999996,"5min":2.056,"15min":2.056},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86099999999999999,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.70699999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:14:14 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.78600000000000003,"15min":0.76000000000000001},"min":{"1min":0.54300000000000004,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.91900000000000004,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.749,"5min":0.78400000000000003,"15min":0.76800000000000002},"min":{"1min":0.58399999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.95699999999999996,"5min":2.02,"15min":2.02},"last":0.76400000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46420,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325420,"statfs":{"total":96636764160,"available":96589230080,"internally_reserved":0,"allocated":1380352,"data_stored":595368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.65700000000000003,"15min":0.64700000000000002},"min":{"1min":0.441,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.056,"5min":1.056,"15min":1.0620000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.70499999999999996,"15min":0.73999999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.998,"5min":0.998,"15min":1.341},"last":0.84099999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.65300000000000002,"15min":0.66500000000000004},"min":{"1min":0.437,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":1.0109999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66100000000000003,"15min":0.627},"min":{"1min":0.48299999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.91800000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.86799999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.66500000000000004,"15min":0.65200000000000002},"min":{"1min":0.497,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.008,"5min":1.014,"15min":1.014},"last":1.1910000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.70099999999999996,"15min":0.72099999999999997},"min":{"1min":0.50600000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.034,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.1719999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.68400000000000005,"15min":0.67000000000000004},"min":{"1min":0.503,"5min":0.434,"15min":0.434},"max":{"1min":0.97199999999999998,"5min":1.002,"15min":1.002},"last":1.006},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.69699999999999995},"min":{"1min":0.51800000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.075,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":1.232}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.621,"15min":0.64500000000000002},"min":{"1min":0.48199999999999998,"5min":0.38700000000000001,"15min":0.38400000000000001},"max":{"1min":1.008,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65000000000000002,"15min":0.66000000000000003},"min":{"1min":0.51800000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.03,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.3240000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67100000000000004,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1200000000000001,"5min":1.1200000000000001,"15min":1.1200000000000001},"last":1.274},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.74399999999999999},"min":{"1min":0.56999999999999995,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.413},"last":1.292}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.71699999999999997,"15min":0.69799999999999995},"min":{"1min":0.59199999999999997,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.73599999999999999,"15min":0.73199999999999998},"min":{"1min":0.54900000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.1459999999999999,"5min":1.177,"15min":1.177},"last":1.3100000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50516,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321324,"statfs":{"total":96636764160,"available":96585035776,"internally_reserved":0,"allocated":1376256,"data_stored":595266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.45400000000000001,"15min":0.46000000000000002},"min":{"1min":0.30099999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.59899999999999998,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46999999999999997,"15min":0.55800000000000005},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.625,"5min":0.97099999999999997,"15min":2.96},"last":0.36199999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.53000000000000003,"15min":0.51900000000000002},"min":{"1min":0.40300000000000002,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.61499999999999999,"5min":3.5600000000000001,"15min":3.5600000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.61499999999999999,"15min":0.71799999999999997},"min":{"1min":0.47199999999999998,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.73699999999999999,"5min":3.621,"15min":3.6829999999999998},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.39700000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.71099999999999997,"5min":1.9590000000000001,"15min":1.9590000000000001},"last":0.502},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57099999999999995,"15min":0.59899999999999998},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85999999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.48999999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.73999999999999999,"15min":0.80900000000000005},"min":{"1min":0.56299999999999994,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.91100000000000003,"5min":1.3560000000000001,"15min":2.8610000000000002},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.77200000000000002,"15min":0.77900000000000003},"min":{"1min":0.60399999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88400000000000001,"5min":1.208,"15min":1.3640000000000001},"last":0.85799999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.78000000000000003,"15min":0.79100000000000004},"min":{"1min":0.63800000000000001,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.89900000000000002,"5min":1.391,"15min":1.391},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.76200000000000001,"15min":0.82899999999999996},"min":{"1min":0.52300000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.89000000000000001,"5min":1.252,"15min":2.9300000000000002},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.78200000000000003,"15min":0.76800000000000002},"min":{"1min":0.59499999999999997,"5min":0.53800000000000003,"15min":0.53000000000000003},"max":{"1min":0.89400000000000002,"5min":1.244,"15min":1.401},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79900000000000004},"min":{"1min":0.56499999999999995,"5min":0.55200000000000005,"15min":0.55000000000000004},"max":{"1min":0.93799999999999994,"5min":1.3300000000000001,"15min":1.3300000000000001},"last":0.79900000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:14:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.85599999999999998},"min":{"1min":0.61199999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.877,"5min":1.4139999999999999,"15min":2.899},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.80400000000000005,"15min":0.82799999999999996},"min":{"1min":0.64700000000000002,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.93000000000000005,"5min":1.278,"15min":1.4359999999999999},"last":0.90600000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574930,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40640,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94331200,"statfs":{"total":96636764160,"available":96595148800,"internally_reserved":0,"allocated":1359872,"data_stored":592695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.40600000000000003,"15min":0.42099999999999999},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.47699999999999998,"15min":0.441},"min":{"1min":0.36399999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.91800000000000004,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.46100000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":3.0819999999999999,"15min":1.3440000000000001},"min":{"1min":0.39800000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.97699999999999998,"5min":260.53399999999999,"15min":260.53399999999999},"last":0.495},{"interface":"front","average":{"1min":0.55200000000000005,"5min":3.1379999999999999,"15min":1.3620000000000001},"min":{"1min":0.38,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.96999999999999997,"5min":262.61399999999998,"15min":262.61399999999998},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.503,"15min":0.48699999999999999},"min":{"1min":0.36299999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.753,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57899999999999996,"15min":0.57399999999999995},"min":{"1min":0.47099999999999997,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.83099999999999996,"5min":1.214,"15min":1.214},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77200000000000002,"15min":0.76400000000000001},"min":{"1min":0.63400000000000001,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0089999999999999,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.78200000000000003,"15min":0.78700000000000003},"min":{"1min":0.55600000000000005,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.135,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.91800000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.78500000000000003,"15min":0.79300000000000004},"min":{"1min":0.66300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.0369999999999999,"5min":1.6240000000000001,"15min":1.6240000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.52800000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0700000000000001,"5min":1.569,"15min":1.569},"last":0.90700000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.79000000000000004,"15min":0.77100000000000002},"min":{"1min":0.61899999999999999,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.091,"5min":1.413,"15min":1.413},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79100000000000004,"15min":0.80900000000000005},"min":{"1min":0.627,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.0780000000000001,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.93600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:14:01 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80000000000000004,"15min":0.78500000000000003},"min":{"1min":0.70599999999999996,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0429999999999999,"5min":1.4730000000000001,"15min":1.4730000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.81100000000000005,"5min":0.80100000000000005,"15min":0.80600000000000005},"min":{"1min":0.61799999999999999,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.82899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50492,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321348,"statfs":{"total":96636764160,"available":96585060352,"internally_reserved":0,"allocated":1359872,"data_stored":593680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.499,"15min":0.503},"min":{"1min":0.34200000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.64000000000000001,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.57199999999999995,"15min":0.51600000000000001},"min":{"1min":0.38400000000000001,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.68400000000000005,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.40200000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49399999999999999,"15min":0.46999999999999997},"min":{"1min":0.33200000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.63700000000000001,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.433},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53300000000000003,"15min":0.56299999999999994},"min":{"1min":0.41899999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.72999999999999998,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.47699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.76700000000000002,"15min":0.69099999999999995},"min":{"1min":0.46300000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84499999999999997,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.72999999999999998,"15min":0.61399999999999999},"min":{"1min":0.36299999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.81499999999999995,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.54400000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81699999999999995,"15min":0.81899999999999995},"min":{"1min":0.57599999999999996,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.329,"5min":1.78,"15min":1.78},"last":0.72399999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.77600000000000002,"15min":0.77300000000000002},"min":{"1min":0.55800000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.92300000000000004,"5min":1.6020000000000001,"15min":1.6870000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.82399999999999995,"15min":0.80300000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.98399999999999999,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.79700000000000004,"15min":0.81100000000000005},"min":{"1min":0.59099999999999997,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0069999999999999,"5min":1.6599999999999999,"15min":1.7310000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.84099999999999997,"15min":0.82299999999999995},"min":{"1min":0.55200000000000005,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.165,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:14:06 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79700000000000004,"15min":0.82699999999999996},"min":{"1min":0.58399999999999996,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.0189999999999999,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.83499999999999996,"15min":0.82799999999999996},"min":{"1min":0.56699999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.056,"5min":1.544,"15min":1.6359999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41216,"kb_used_data":1908,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330624,"statfs":{"total":96636764160,"available":96594558976,"internally_reserved":0,"allocated":1953792,"data_stored":1183611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44400000000000001,"15min":0.67800000000000005},"min":{"1min":0.27800000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.71199999999999997,"5min":0.99099999999999999,"15min":5.5259999999999998},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.47299999999999998,"15min":0.67100000000000004},"min":{"1min":0.312,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.82699999999999996,"5min":1.1859999999999999,"15min":5.4889999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.499,"15min":0.70499999999999996},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33500000000000002},"max":{"1min":0.66800000000000004,"5min":0.81100000000000005,"15min":6.0430000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.71999999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68999999999999995,"5min":1.2190000000000001,"15min":4.6760000000000002},"last":0.47399999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.49399999999999999,"15min":0.66300000000000003},"min":{"1min":0.371,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":1.5069999999999999,"15min":4.7549999999999999},"last":0.5},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56699999999999995,"15min":0.71799999999999997},"min":{"1min":0.432,"5min":0.36299999999999999,"15min":0.36199999999999999},"max":{"1min":0.81399999999999995,"5min":1.1000000000000001,"15min":4.5940000000000003},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75800000000000001,"15min":0.89600000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.083,"5min":1.3160000000000001,"15min":4.734},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.754,"15min":0.91200000000000003},"min":{"1min":0.52300000000000002,"5min":0.442,"15min":0.442},"max":{"1min":1.337,"5min":1.337,"15min":4.8079999999999998},"last":0.874}]},{"osd":5,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.747,"15min":0.92500000000000004},"min":{"1min":0.55000000000000004,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.1319999999999999,"5min":1.175,"15min":4.944},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77800000000000002,"15min":0.91500000000000004},"min":{"1min":0.66900000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":4.9710000000000001},"last":0.85699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.77400000000000002,"15min":1.0680000000000001},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.54200000000000004},"max":{"1min":1.1859999999999999,"5min":1.242,"15min":4.8470000000000004},"last":0.95199999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77700000000000002,"15min":0.93600000000000005},"min":{"1min":0.55700000000000005,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.163,"5min":1.274,"15min":5.0170000000000003},"last":0.89500000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:14:10 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.78300000000000003,"15min":0.96899999999999997},"min":{"1min":0.51100000000000001,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.2609999999999999,"5min":1.3069999999999999,"15min":5.0540000000000003},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77900000000000003,"15min":0.95099999999999996},"min":{"1min":0.59999999999999998,"5min":0.505,"15min":0.505},"max":{"1min":1.242,"5min":1.4530000000000001,"15min":4.9950000000000001},"last":0.79100000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:14:36.133 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:14:36.134 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:14:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.330+0000 7f3145800640 1 Processor -- start 2024-09-13T17:14:36.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.330+0000 7f3145800640 1 -- start start 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.331+0000 7f3145800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314015a780 0x7f3140070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.331+0000 7f3145800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f314015ab60 0x7f31400712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.331+0000 7f3145800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314006c8b0 con 0x7f314006bb40 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.331+0000 7f3145800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314006c6a0 con 0x7f314015ab60 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.331+0000 7f3145800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314006cb90 con 0x7f314015a780 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f314006bb40 0x7f314006bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40024/0 (socket says 172.21.15.28:40024) 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f314015ab60 0x7f31400712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313f400640 1 -- 172.21.15.28:0/3494234103 learned_addr learned my addr 172.21.15.28:0/3494234103 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313ea00640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314015a780 0x7f3140070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313e000640 1 -- 172.21.15.28:0/3494234103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314015a780 msgr2=0x7f3140070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313e000640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314015a780 0x7f3140070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313e000640 1 -- 172.21.15.28:0/3494234103 >> v1:172.21.15.28:6789/0 conn(0x7f314006bb40 legacy=0x7f314006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313e000640 1 -- 172.21.15.28:0/3494234103 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31400cc980 con 0x7f314015ab60 2024-09-13T17:14:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.332+0000 7f313d600640 1 -- 172.21.15.28:0/3494234103 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799958357 0 0) 0x7f314006c8b0 con 0x7f314006bb40 2024-09-13T17:14:36.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.333+0000 7f313e000640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015ab60 0x7f31400712c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f312c0097a0 tx=0x7f312c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19845b541b3d0eb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.333+0000 7f313d600640 1 -- 172.21.15.28:0/3494234103 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c032440 con 0x7f314015ab60 2024-09-13T17:14:36.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.333+0000 7f313d600640 1 -- 172.21.15.28:0/3494234103 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c0325a0 con 0x7f314015ab60 2024-09-13T17:14:36.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.333+0000 7f313d600640 1 -- 172.21.15.28:0/3494234103 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c032930 con 0x7f314015ab60 2024-09-13T17:14:36.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 -- 172.21.15.28:0/3494234103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015ab60 msgr2=0x7f31400712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015ab60 0x7f31400712c0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f312c0097a0 tx=0x7f312c02a180 comp rx=0 tx=0).stop 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 -- 172.21.15.28:0/3494234103 shutdown_connections 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015ab60 0x7f31400712c0 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 --2- 172.21.15.28:0/3494234103 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314015a780 0x7f3140070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 -- 172.21.15.28:0/3494234103 >> 172.21.15.28:0/3494234103 conn(0x7f31400b0270 msgr2=0x7f31400b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 -- 172.21.15.28:0/3494234103 shutdown_connections 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.334+0000 7f3145800640 1 -- 172.21.15.28:0/3494234103 wait complete. 2024-09-13T17:14:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.336+0000 7f3145800640 1 Processor -- start 2024-09-13T17:14:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.336+0000 7f3145800640 1 -- start start 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 0x7f3140150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 0x7f31400accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400a8e60 con 0x7f314015a780 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31400ad5a0 con 0x7f314015ab60 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f3145800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3140150f70 con 0x7f314006bb40 2024-09-13T17:14:36.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f313e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f313e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40824/0 (socket says 172.21.15.28:40824) 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f313e000640 1 -- 172.21.15.28:0/4150979953 learned_addr learned my addr 172.21.15.28:0/4150979953 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f313ea00640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 0x7f3140150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.337+0000 7f313f400640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 0x7f31400accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 -- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 msgr2=0x7f3140150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 0x7f3140150a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 -- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 msgr2=0x7f31400accc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 0x7f31400accc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 -- 172.21.15.28:0/4150979953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312c009400 con 0x7f314015a780 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313ea00640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 0x7f3140150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313f400640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 0x7f31400accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:14:36.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f313e000640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c032440 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f312c030ba0 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312c006e10 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.338+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31400a8570 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.339+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31400a8af0 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.340+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314006d2d0 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.340+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f312c039040 con 0x7f314015a780 2024-09-13T17:14:36.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.341+0000 7f3136a00640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 0x7f312006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.341+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f312c0a9e00 con 0x7f314015a780 2024-09-13T17:14:36.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.342+0000 7f313ea00640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 0x7f312006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.342+0000 7f313ea00640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 0x7f312006d0c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3130006fd0 tx=0x7f3130008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.346+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f312c0750c0 con 0x7f314015a780 2024-09-13T17:14:36.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.597+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f314006c8b0 con 0x7f314015a780 2024-09-13T17:14:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.600+0000 7f3136a00640 1 -- 172.21.15.28:0/4150979953 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f312c0410e0 con 0x7f314015a780 2024-09-13T17:14:36.602 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 msgr2=0x7f312006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 0x7f312006d0c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f3130006fd0 tx=0x7f3130008040 comp rx=0 tx=0).stop 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 msgr2=0x7f31401f0220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f312c000c00 tx=0x7f312c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 shutdown_connections 2024-09-13T17:14:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f312006ac00 0x7f312006d0c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f314015ab60 0x7f31400accc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f314015a780 0x7f31401f0220 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 --2- 172.21.15.28:0/4150979953 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f314006bb40 0x7f3140150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 >> 172.21.15.28:0/4150979953 conn(0x7f31400b0270 msgr2=0x7f3140070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 shutdown_connections 2024-09-13T17:14:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.603+0000 7f3145800640 1 -- 172.21.15.28:0/4150979953 wait complete. 2024-09-13T17:14:36.621 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:14:36.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.774+0000 7f08e9600640 1 Processor -- start 2024-09-13T17:14:36.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.775+0000 7f08e9600640 1 -- start start 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08e40b0b50 0x7f08e40b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 0x7f08e40b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e9600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e40c3850 con 0x7f08e4159440 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e40b2d60 con 0x7f08e40b0b50 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e415a930 con 0x7f08e4159820 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08e4159440 0x7f08e40b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40034/0 (socket says 172.21.15.28:40034) 2024-09-13T17:14:36.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08e3e00640 1 -- 172.21.15.28:0/191980309 learned_addr learned my addr 172.21.15.28:0/191980309 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.776+0000 7f08dbe00640 1 --2- 172.21.15.28:0/191980309 >> v2:172.21.15.28:3300/0 conn(0x7f08e40b0b50 0x7f08e40b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08e8c00640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 0x7f08e40b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08dbe00640 1 -- 172.21.15.28:0/191980309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 msgr2=0x7f08e40b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08dbe00640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 0x7f08e40b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08dbe00640 1 -- 172.21.15.28:0/191980309 >> v1:172.21.15.28:6789/0 conn(0x7f08e4159440 legacy=0x7f08e40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08dbe00640 1 -- 172.21.15.28:0/191980309 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e4159d20 con 0x7f08e40b0b50 2024-09-13T17:14:36.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.777+0000 7f08e8c00640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 0x7f08e40b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.778+0000 7f08dbe00640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e40b0b50 0x7f08e40b22e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097a0 tx=0x7f08cc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3223f954e73ad3f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.778+0000 7f08e3400640 1 -- 172.21.15.28:0/191980309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc032440 con 0x7f08e40b0b50 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.778+0000 7f08e3400640 1 -- 172.21.15.28:0/191980309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc0325a0 con 0x7f08e40b0b50 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.778+0000 7f08e3400640 1 -- 172.21.15.28:0/191980309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc032930 con 0x7f08e40b0b50 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 -- 172.21.15.28:0/191980309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e40b0b50 msgr2=0x7f08e40b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e40b0b50 0x7f08e40b22e0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0097a0 tx=0x7f08cc02a230 comp rx=0 tx=0).stop 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 -- 172.21.15.28:0/191980309 shutdown_connections 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159820 0x7f08e40b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 --2- 172.21.15.28:0/191980309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e40b0b50 0x7f08e40b22e0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 -- 172.21.15.28:0/191980309 >> 172.21.15.28:0/191980309 conn(0x7f08e4072e40 msgr2=0x7f08e4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 -- 172.21.15.28:0/191980309 shutdown_connections 2024-09-13T17:14:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.779+0000 7f08e9600640 1 -- 172.21.15.28:0/191980309 wait complete. 2024-09-13T17:14:36.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.780+0000 7f08e9600640 1 Processor -- start 2024-09-13T17:14:36.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 -- start start 2024-09-13T17:14:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e40b0b50 0x7f08e40ab1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159440 0x7f08e40ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41f1e20 con 0x7f08e4159820 2024-09-13T17:14:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e41edcb0 con 0x7f08e40b0b50 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.781+0000 7f08e9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e40abf10 con 0x7f08e4159440 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40846/0 (socket says 172.21.15.28:40846) 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 -- 172.21.15.28:0/3706205443 learned_addr learned my addr 172.21.15.28:0/3706205443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08dbe00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159440 0x7f08e40ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e3e00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e40b0b50 0x7f08e40ab1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 -- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159440 msgr2=0x7f08e40ab6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159440 0x7f08e40ab6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 -- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e40b0b50 msgr2=0x7f08e40ab1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e40b0b50 0x7f08e40ab1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.782+0000 7f08e8c00640 1 -- 172.21.15.28:0/3706205443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc009400 con 0x7f08e4159820 2024-09-13T17:14:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e8c00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00cd10 tx=0x7f08dc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc00f680 con 0x7f08e4159820 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08dc00f800 con 0x7f08e4159820 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08dc00fbb0 con 0x7f08e4159820 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e41f1480 con 0x7f08e4159820 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.783+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e41f1a40 con 0x7f08e4159820 2024-09-13T17:14:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.784+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e41f1f60 con 0x7f08e4159820 2024-09-13T17:14:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.785+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f08dc00b040 con 0x7f08e4159820 2024-09-13T17:14:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.786+0000 7f08e0c00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 0x7f08c006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:36.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.786+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f08dc089ab0 con 0x7f08e4159820 2024-09-13T17:14:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.786+0000 7f08e3e00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 0x7f08c006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:36.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.787+0000 7f08e3e00640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 0x7f08c006d0f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08d4005880 tx=0x7f08d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:36.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:36.790+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f08dc054d30 con 0x7f08e4159820 2024-09-13T17:14:37.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.004+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08e40abc30 con 0x7f08e4159820 2024-09-13T17:14:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.005+0000 7f08e0c00640 1 -- 172.21.15.28:0/3706205443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f08dc058350 con 0x7f08e4159820 2024-09-13T17:14:37.007 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:14:37.007 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":94,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:14:34.542201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:14:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 94 2024-09-13T17:14:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 msgr2=0x7f08c006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 0x7f08c006d0f0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08d4005880 tx=0x7f08d4009040 comp rx=0 tx=0).stop 2024-09-13T17:14:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 msgr2=0x7f08e41f01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f08dc00cd10 tx=0x7f08dc00a650 comp rx=0 tx=0).stop 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 shutdown_connections 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f08c006ac30 0x7f08c006d0f0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08e4159820 0x7f08e41f01d0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08e4159440 0x7f08e40ab6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 --2- 172.21.15.28:0/3706205443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08e40b0b50 0x7f08e40ab1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 >> 172.21.15.28:0/3706205443 conn(0x7f08e4072e40 msgr2=0x7f08e415d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 shutdown_connections 2024-09-13T17:14:37.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.007+0000 7f08e9600640 1 -- 172.21.15.28:0/3706205443 wait complete. 2024-09-13T17:14:37.084 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:14:37.082+0000 7f46d7200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:14:37.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 94, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:14:31.526446+0000', 'modified': '2024-09-13T17:14:34.542201+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8022}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8022': {'gid': 8022, 'name': 'c', 'rank': 0, 'incarnation': 90, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4107617548', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 4107617548}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4107617548}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:14:37.428 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:14:37.428 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.585+0000 7f48fde00640 1 Processor -- start 2024-09-13T17:14:37.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.585+0000 7f48fde00640 1 -- start start 2024-09-13T17:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48fde00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f48f815a780 0x7f48f8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48fde00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f806c660 con 0x7f48f815ab60 2024-09-13T17:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48fde00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f806c8b0 con 0x7f48f806c250 2024-09-13T17:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48fde00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48f806cb90 con 0x7f48f815a780 2024-09-13T17:14:37.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f48f815ab60 0x7f48f806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40040/0 (socket says 172.21.15.28:40040) 2024-09-13T17:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.586+0000 7f48efe00640 1 -- 172.21.15.28:0/2230760673 learned_addr learned my addr 172.21.15.28:0/2230760673 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6a00640 1 --2- 172.21.15.28:0/2230760673 >> v2:172.21.15.189:3300/0 conn(0x7f48f815a780 0x7f48f8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6000640 1 -- 172.21.15.28:0/2230760673 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33873730 0 0) 0x7f48f806c660 con 0x7f48f815ab60 2024-09-13T17:14:37.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6000640 1 -- 172.21.15.28:0/2230760673 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc003520 con 0x7f48f815ab60 2024-09-13T17:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6a00640 1 -- 172.21.15.28:0/2230760673 >> v1:172.21.15.189:6789/0 conn(0x7f48f806c250 legacy=0x7f48f80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6a00640 1 -- 172.21.15.28:0/2230760673 >> v1:172.21.15.28:6789/0 conn(0x7f48f815ab60 legacy=0x7f48f806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.587+0000 7f48f6a00640 1 -- 172.21.15.28:0/2230760673 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f80cc980 con 0x7f48f815a780 2024-09-13T17:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.588+0000 7f48f6a00640 1 --2- 172.21.15.28:0/2230760673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f815a780 0x7f48f8070d80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f48e80097a0 tx=0x7f48e802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=691417c127f854d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48f6000640 1 -- 172.21.15.28:0/2230760673 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8032440 con 0x7f48f815a780 2024-09-13T17:14:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48f6000640 1 -- 172.21.15.28:0/2230760673 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e80325a0 con 0x7f48f815a780 2024-09-13T17:14:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48f6000640 1 -- 172.21.15.28:0/2230760673 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8032930 con 0x7f48f815a780 2024-09-13T17:14:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 -- 172.21.15.28:0/2230760673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f815a780 msgr2=0x7f48f8070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 --2- 172.21.15.28:0/2230760673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f815a780 0x7f48f8070d80 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f48e80097a0 tx=0x7f48e802a230 comp rx=0 tx=0).stop 2024-09-13T17:14:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 -- 172.21.15.28:0/2230760673 shutdown_connections 2024-09-13T17:14:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 --2- 172.21.15.28:0/2230760673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f815a780 0x7f48f8070d80 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 -- 172.21.15.28:0/2230760673 >> 172.21.15.28:0/2230760673 conn(0x7f48f80b0270 msgr2=0x7f48f80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 -- 172.21.15.28:0/2230760673 shutdown_connections 2024-09-13T17:14:37.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.589+0000 7f48fde00640 1 -- 172.21.15.28:0/2230760673 wait complete. 2024-09-13T17:14:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.591+0000 7f48fde00640 1 Processor -- start 2024-09-13T17:14:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.591+0000 7f48fde00640 1 -- start start 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f806c250 0x7f48f814f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48f815a780 0x7f48f815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f815c060 con 0x7f48f815ab60 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8148450 con 0x7f48f806c250 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48fde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48f814f3c0 con 0x7f48f815a780 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.592+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40852/0 (socket says 172.21.15.28:40852) 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 -- 172.21.15.28:0/1183185148 learned_addr learned my addr 172.21.15.28:0/1183185148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f6a00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f806c250 0x7f48f814f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48efe00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48f815a780 0x7f48f815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 -- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48f815a780 msgr2=0x7f48f815b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48f815a780 0x7f48f815b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 -- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f806c250 msgr2=0x7f48f814f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f806c250 0x7f48f814f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 -- 172.21.15.28:0/1183185148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e8009400 con 0x7f48f815ab60 2024-09-13T17:14:37.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48f7400640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f48f000c7d0 tx=0x7f48f000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.593+0000 7f48ef400640 1 -- 172.21.15.28:0/1183185148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0003b70 con 0x7f48f815ab60 2024-09-13T17:14:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.594+0000 7f48ef400640 1 -- 172.21.15.28:0/1183185148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f0003cf0 con 0x7f48f815ab60 2024-09-13T17:14:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.594+0000 7f48ef400640 1 -- 172.21.15.28:0/1183185148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f000aa90 con 0x7f48f815ab60 2024-09-13T17:14:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.594+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f8150a50 con 0x7f48f815ab60 2024-09-13T17:14:37.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.594+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f8151010 con 0x7f48f815ab60 2024-09-13T17:14:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.596+0000 7f48ef400640 1 -- 172.21.15.28:0/1183185148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f48f000abf0 con 0x7f48f815ab60 2024-09-13T17:14:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.596+0000 7f48ef400640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 0x7f48d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.597+0000 7f48ef400640 1 -- 172.21.15.28:0/1183185148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48f00891d0 con 0x7f48f815ab60 2024-09-13T17:14:37.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.597+0000 7f48f6a00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 0x7f48d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.597+0000 7f4900bc3740 1 Processor -- start 2024-09-13T17:14:37.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.597+0000 7f4900bc3740 1 -- start start 2024-09-13T17:14:37.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f48f6a00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 0x7f48d406d0c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f48e8034000 tx=0x7f48e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e086d0 0x556262e08ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e05e90 0x556262e55ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262d71ab0 con 0x556262e2fb80 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262d6f240 con 0x556262e086d0 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.598+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556262d6ab30 con 0x556262e05e90 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40868/0 (socket says 172.21.15.28:40868) 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 -- 172.21.15.28:0/1921534558 learned_addr learned my addr 172.21.15.28:0/1921534558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f6a00640 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e086d0 0x556262e08ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48efe00640 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e05e90 0x556262e55ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 -- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e05e90 msgr2=0x556262e55ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e05e90 0x556262e55ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 -- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e086d0 msgr2=0x556262e08ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e086d0 0x556262e08ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 -- 172.21.15.28:0/1921534558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556262d35bd0 con 0x556262e2fb80 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.599+0000 7f48f7400640 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f48f005fe10 tx=0x7f48f00578d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ff12608e257737 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f48e7e00640 1 -- 172.21.15.28:0/1921534558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0060340 con 0x556262e2fb80 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f48e7e00640 1 -- 172.21.15.28:0/1921534558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f0056310 con 0x556262e2fb80 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f48e7e00640 1 -- 172.21.15.28:0/1921534558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f005feb0 con 0x556262e2fb80 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 -- 172.21.15.28:0/1921534558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 msgr2=0x556262e560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f48f005fe10 tx=0x7f48f00578d0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 -- 172.21.15.28:0/1921534558 shutdown_connections 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e560e0 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e05e90 0x556262e55ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 --2- 172.21.15.28:0/1921534558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e086d0 0x556262e08ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 -- 172.21.15.28:0/1921534558 >> 172.21.15.28:0/1921534558 conn(0x556262e04e00 msgr2=0x556262e4d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.600+0000 7f4900bc3740 1 -- 172.21.15.28:0/1921534558 shutdown_connections 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.601+0000 7f4900bc3740 1 -- 172.21.15.28:0/1921534558 wait complete. 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.601+0000 7f4900bc3740 1 -- start start 2024-09-13T17:14:37.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.601+0000 7f4900bc3740 1 Processor -- start 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.602+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.602+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e27310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.602+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e2ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.602+0000 7f4900bc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262d0c260 con 0x556262e086d0 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.602+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262d0bf00 con 0x556262e2fb80 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556262ad1f20 con 0x556262e2eac0 2024-09-13T17:14:37.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f4900bc3740 10 client.0 _ll_register_callbacks cb 0x556262d641a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40878/0 (socket says 172.21.15.28:40878) 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 -- 172.21.15.28:0/3714916936 learned_addr learned my addr 172.21.15.28:0/3714916936 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f7400640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e2ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48efe00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e27310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 -- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 msgr2=0x556262e2ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e2ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 -- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 msgr2=0x556262e27310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e27310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.603+0000 7f48f6a00640 1 -- 172.21.15.28:0/3714916936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f8151010 con 0x556262e086d0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48f6a00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f48e803f040 tx=0x7f48e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9840f2133ee5643 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8054680 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48e804f070 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556262d94cd0 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48e80547e0 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f4900bc3740 10 client.8160 fetch_fsmap learned FSMap version 94 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556262d691b0 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556262d94cd0 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f48e8044040 con 0x556262e086d0 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f4900bc3740 10 client.8160 fetch_fsmap finished waiting for FSMap version 94 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.604+0000 7f4900bc3740 10 client.8160 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f4900bc3740 20 client.8160 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262e68180 0x556262e6a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f4900bc3740 4 client.8160 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556262a5e3c0 con 0x556262e68180 2024-09-13T17:14:37.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f48efe00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262e68180 0x556262e6a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f48efe00640 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262e68180 0x556262e6a570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=202faf29346b412a server_cookie=c2113426e6ac1e2e in_seq=0 out_seq=0 2024-09-13T17:14:37.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.605+0000 7f48e4c00640 10 client.8160 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f48e4c00640 1 -- 172.21.15.28:0/3714916936 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x556262a5e3c0 con 0x556262e68180 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f48e4c00640 10 client.8160 handle_command_reply: tid=0 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 client.8160 shutdown 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 msgr2=0x556262e2b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f48e803f040 tx=0x7f48e8004730 comp rx=0 tx=0).stop 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 shutdown_connections 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262e68180 0x556262e6a570 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e2ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e27310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f48e4c00640 0 client.8160 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:14:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 --2- 172.21.15.28:0/3714916936 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e2b060 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.606+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 >> 172.21.15.28:0/3714916936 conn(0x556262e05e90 msgr2=0x556262e4dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.607+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 shutdown_connections 2024-09-13T17:14:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.607+0000 7f4900bc3740 1 -- 172.21.15.28:0/3714916936 wait complete. 2024-09-13T17:14:37.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.607+0000 7f4900bc3740 20 client.8160 trim_cache size 0 max 16384 2024-09-13T17:14:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.622+0000 7f4900bc3740 1 Processor -- start 2024-09-13T17:14:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 -- start start 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e4aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e4dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262e4edc0 con 0x556262e086d0 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262e4c870 con 0x556262e2fb80 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.623+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556262e28070 con 0x556262e2eac0 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40882/0 (socket says 172.21.15.28:40882) 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 -- 172.21.15.28:0/112460618 learned_addr learned my addr 172.21.15.28:0/112460618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f7400640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e4dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48efe00640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e4aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 -- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 msgr2=0x556262e4aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e4aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 -- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 msgr2=0x556262e4dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e4dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 -- 172.21.15.28:0/112460618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556262d9e170 con 0x556262e086d0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f7400640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e4dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48efe00640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e4aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.624+0000 7f48f6a00640 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f48e8000c00 tx=0x7f48e804b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99f4c6eac2bb0ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f48ecc00640 1 -- 172.21.15.28:0/112460618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8054cd0 con 0x556262e086d0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f48ecc00640 1 -- 172.21.15.28:0/112460618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48e804d070 con 0x556262e086d0 2024-09-13T17:14:37.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f48ecc00640 1 -- 172.21.15.28:0/112460618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8054680 con 0x556262e086d0 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 -- 172.21.15.28:0/112460618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 msgr2=0x556262e290c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f48e8000c00 tx=0x7f48e804b070 comp rx=0 tx=0).stop 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 -- 172.21.15.28:0/112460618 shutdown_connections 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2fb80 0x556262e4dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e2eac0 0x556262e4aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 --2- 172.21.15.28:0/112460618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e086d0 0x556262e290c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 -- 172.21.15.28:0/112460618 >> 172.21.15.28:0/112460618 conn(0x556262e05e90 msgr2=0x556262e2af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 -- 172.21.15.28:0/112460618 shutdown_connections 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.625+0000 7f4900bc3740 1 -- 172.21.15.28:0/112460618 wait complete. 2024-09-13T17:14:37.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.626+0000 7f4900bc3740 1 -- start start 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.626+0000 7f4900bc3740 1 Processor -- start 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 0x556262e4bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 0x556262e2d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262eb9090 con 0x556262e2fb80 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556262e517f0 con 0x556262e2eac0 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556262e4b3a0 con 0x556262e086d0 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f4900bc3740 10 client.0 _ll_register_callbacks cb 0x7f48dc0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f48f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40888/0 (socket says 172.21.15.28:40888) 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.627+0000 7f48f7400640 1 -- 172.21.15.28:0/508434007 learned_addr learned my addr 172.21.15.28:0/508434007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f6a00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 0x556262e4bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 -- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 msgr2=0x556262e4bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48efe00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 0x556262e2d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 0x556262e4bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 -- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 msgr2=0x556262e2d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 0x556262e2d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 -- 172.21.15.28:0/508434007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556262d691b0 con 0x556262e2fb80 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f6a00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 0x556262e4bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48efe00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 0x556262e2d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48f7400640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f48f000afd0 tx=0x7f48f0059000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28890321d435b90f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0086070 con 0x556262e2fb80 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f48f0060340 con 0x556262e2fb80 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f0056310 con 0x556262e2fb80 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=94) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f0087260 con 0x556262e2fb80 2024-09-13T17:14:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f4900bc3740 10 client.8166 fetch_fsmap learned FSMap version 94 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.628+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=94,monmap=2+}) v3 -- 0x556262d0b470 con 0x556262e2fb80 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f0056310 con 0x556262e2fb80 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 94) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f48f0076d20 con 0x556262e2fb80 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 10 client.8166 fetch_fsmap finished waiting for FSMap version 94 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 10 client.8166 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 20 client.8166 populate_metadata read hostname 'smithi028' 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262ebbb30 0x556262ebbf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 4 client.8166 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556262a6f6c0 con 0x556262ebbb30 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f48f6a00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262ebbb30 0x556262ebbf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f48f6a00640 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262ebbb30 0x556262ebbf10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=31dbe738f0e3b7eb server_cookie=3d5e54a882977474 in_seq=0 out_seq=0 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.629+0000 7f48e4c00640 10 client.8166 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:14:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f48e4c00640 1 -- 172.21.15.28:0/508434007 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556262a6f6c0 con 0x556262ebbb30 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f48e4c00640 10 client.8166 handle_command_reply: tid=0 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 client.8166 shutdown 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 msgr2=0x556262e2d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f48f000afd0 tx=0x7f48f0059000 comp rx=0 tx=0).stop 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 shutdown_connections 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556262ebbb30 0x556262ebbf10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556262e2fb80 0x556262e2d790 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556262e2eac0 0x556262e2d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 --2- 172.21.15.28:0/508434007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556262e086d0 0x556262e4bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f48e4c00640 0 client.8166 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 >> 172.21.15.28:0/508434007 conn(0x556262e05e90 msgr2=0x556262e29030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 shutdown_connections 2024-09-13T17:14:37.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 1 -- 172.21.15.28:0/508434007 wait complete. 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.630+0000 7f4900bc3740 20 client.8166 trim_cache size 0 max 16384 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8022, 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1020220480000003, 2024-09-13T17:14:37.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 94, 2024-09-13T17:14:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 111, 2024-09-13T17:14:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 111, 2024-09-13T17:14:37.639 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.207551364 2024-09-13T17:14:37.639 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 msgr2=0x7f48d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 0x7f48d406d0c0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f48e8034000 tx=0x7f48e802e000 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 msgr2=0x7f48f8147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f48f000c7d0 tx=0x7f48f000cc90 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 shutdown_connections 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f48d406ac00 0x7f48d406d0c0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48f815ab60 0x7f48f8147f10 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48f815a780 0x7f48f815b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 --2- 172.21.15.28:0/1183185148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f48f806c250 0x7f48f814f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 >> 172.21.15.28:0/1183185148 conn(0x7f48f80b0270 msgr2=0x7f48f80b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 shutdown_connections 2024-09-13T17:14:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.631+0000 7f48fde00640 1 -- 172.21.15.28:0/1183185148 wait complete. 2024-09-13T17:14:37.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:14:37.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.801+0000 7f5ea2e00640 1 Processor -- start 2024-09-13T17:14:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.801+0000 7f5ea2e00640 1 -- start start 2024-09-13T17:14:37.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.802+0000 7f5ea2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e9c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.802+0000 7f5ea2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c159d90 con 0x7f5e9c06f970 2024-09-13T17:14:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.802+0000 7f5ea2e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e9c0b2040 con 0x7f5e9c1592a0 2024-09-13T17:14:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.802+0000 7f5ea2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c15b0b0 con 0x7f5e9c06c9e0 2024-09-13T17:14:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5e9c06f970 0x7f5e9c06c2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40044/0 (socket says 172.21.15.28:40044) 2024-09-13T17:14:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9b400640 1 -- 172.21.15.28:0/2594456416 learned_addr learned my addr 172.21.15.28:0/2594456416 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:14:37.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e9c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796960925 0 0) 0x7f5e9c159d90 con 0x7f5e9c06f970 2024-09-13T17:14:37.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e80003520 con 0x7f5e9c06f970 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147917154 0 0) 0x7f5e9c0b2040 con 0x7f5e9c1592a0 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e9c159d90 con 0x7f5e9c1592a0 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 373841265 0 0) 0x7f5e80003520 con 0x7f5e9c06f970 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 msgr2=0x7f5e9c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e9c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 >> v1:172.21.15.189:6789/0 conn(0x7f5e9c1592a0 legacy=0x7f5e9c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c159ff0 con 0x7f5e9c06f970 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.803+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f5e88001690 con 0x7f5e9c06f970 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 >> v1:172.21.15.28:6789/0 conn(0x7f5e9c06f970 legacy=0x7f5e9c06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80003b30 0x7f5e80003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e80004d00 0x7f5e80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c15b0b0 con 0x7f5e80004d00 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8000dc40 con 0x7f5e80003b30 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8000ddb0 con 0x7f5e9c06c9e0 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e80004d00 0x7f5e80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9be00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80003b30 0x7f5e80003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5e9b400640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 -- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 msgr2=0x7f5e800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c06c9e0 0x7f5e800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 -- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80003b30 msgr2=0x7f5e80003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80003b30 0x7f5e80003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.804+0000 7f5ea0e00640 1 -- 172.21.15.28:0/2594456416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e80003520 con 0x7f5e80004d00 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.805+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e80004d00 0x7f5e80005180 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009fc0 tx=0x7f5e800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86114da19bbdf43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94031640 con 0x7f5e80004d00 2024-09-13T17:14:37.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e9402d070 con 0x7f5e80004d00 2024-09-13T17:14:37.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5e9aa00640 1 -- 172.21.15.28:0/2594456416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e940319e0 con 0x7f5e80004d00 2024-09-13T17:14:37.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 -- 172.21.15.28:0/2594456416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e80004d00 msgr2=0x7f5e80005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/2594456416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e80004d00 0x7f5e80005180 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009fc0 tx=0x7f5e800057f0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea0e00640 1 -- 172.21.15.28:0/2594456416 reap_dead start 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 -- 172.21.15.28:0/2594456416 shutdown_connections 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 -- 172.21.15.28:0/2594456416 >> 172.21.15.28:0/2594456416 conn(0x7f5e9c069e80 msgr2=0x7f5e9c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 -- 172.21.15.28:0/2594456416 shutdown_connections 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.806+0000 7f5ea2e00640 1 -- 172.21.15.28:0/2594456416 wait complete. 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.808+0000 7f5ea2e00640 1 Processor -- start 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.808+0000 7f5ea2e00640 1 -- start start 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80004d00 0x7f5e9c1edd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c0ce280 0x7f5e9c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0cb870 con 0x7f5e9c0ce700 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0cbb50 con 0x7f5e80004d00 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9c0cbe30 con 0x7f5e9c0ce280 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40900/0 (socket says 172.21.15.28:40900) 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.809+0000 7f5ea0e00640 1 -- 172.21.15.28:0/1190085306 learned_addr learned my addr 172.21.15.28:0/1190085306 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5e9b400640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c0ce280 0x7f5e9c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80004d00 0x7f5e9c1edd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 -- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c0ce280 msgr2=0x7f5e9c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c0ce280 0x7f5e9c150e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 -- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80004d00 msgr2=0x7f5e9c1edd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80004d00 0x7f5e9c1edd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 -- 172.21.15.28:0/1190085306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c0cc110 con 0x7f5e9c0ce700 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5ea0e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e230 tx=0x7f5e94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.810+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94031df0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.811+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e94040070 con 0x7f5e9c0ce700 2024-09-13T17:14:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.811+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e94003bd0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.811+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9c0cc3f0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.811+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e9c1534e0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.812+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9c0cc5e0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.812+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e94042070 con 0x7f5e9c0ce700 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.813+0000 7f5e93e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 0x7f5e7c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.813+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e940aa2e0 con 0x7f5e9c0ce700 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.814+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 0x7f5e7c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.814+0000 7f5e9be00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 0x7f5e7c06d0c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0055b0 tx=0x7f5e8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:37.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:37.817+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5e940754b0 con 0x7f5e9c0ce700 2024-09-13T17:14:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.033+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5e9c15ab70 con 0x7f5e9c0ce700 2024-09-13T17:14:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.034+0000 7f5e93e00640 1 -- 172.21.15.28:0/1190085306 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v94) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5e94078ad0 con 0x7f5e9c0ce700 2024-09-13T17:14:38.036 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 msgr2=0x7f5e7c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 0x7f5e7c06d0c0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c0055b0 tx=0x7f5e8c009040 comp rx=0 tx=0).stop 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 msgr2=0x7f5e9c151360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f5e9400e230 tx=0x7f5e94009750 comp rx=0 tx=0).stop 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 shutdown_connections 2024-09-13T17:14:38.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e7c06ac00 0x7f5e7c06d0c0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e9c0ce700 0x7f5e9c151360 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e9c0ce280 0x7f5e9c150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 --2- 172.21.15.28:0/1190085306 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e80004d00 0x7f5e9c1edd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 >> 172.21.15.28:0/1190085306 conn(0x7f5e9c069e80 msgr2=0x7f5e9c0af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 shutdown_connections 2024-09-13T17:14:38.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:38.036+0000 7f5ea2e00640 1 -- 172.21.15.28:0/1190085306 wait complete. 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:14:38.054 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:14:38.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.055 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:14:38.091 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42303sec preferred_lft 42303sec 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:14:38.092 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:14:38.093 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:14:38.094 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.094 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:14:38.184 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:14:38.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.187 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:14:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:14:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:14:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:14:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:14:38.224 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:14:38.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:14:38.225 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:14:38.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.373 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:14:38.408 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:14:38.408 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:14:38.409 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:14:38.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.409 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:14:38.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:14:38.507 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:14:38.543 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:14:38.543 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:14:38.543 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:14:38.618 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:14:38.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.618 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:14:38.747 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:14:38.747 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:14:38.774 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:14:38.774 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:14:38.774 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:14:38.774 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:14:38.774 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:14:38.776 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:14:38.776 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:14:38.776 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:14:38.776 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:14:38.776 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:14:38.779 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:14:38.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:14:38.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:38.836 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:14:38.880 INFO:teuthology.orchestra.run.smithi028.stdout:client8175 2024-09-13T17:14:38.880 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:38.880 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:38.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2024-09-13T17:14:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.154+0000 7f21afc00640 1 Processor -- start 2024-09-13T17:14:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.154+0000 7f21afc00640 1 -- start start 2024-09-13T17:14:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.155+0000 7f21afc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21a815a7a0 0x7f21a80ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.155+0000 7f21afc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.155+0000 7f21afc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a806c950 con 0x7f21a80ce9d0 2024-09-13T17:14:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.155+0000 7f21afc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80cee40 con 0x7f21a815a7a0 2024-09-13T17:14:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.155+0000 7f21afc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80b2150 con 0x7f21a80b1c80 2024-09-13T17:14:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47392/0 (socket says 172.21.15.28:47392) 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 -- 172.21.15.28:0/551435031 learned_addr learned my addr 172.21.15.28:0/551435031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21ad200640 1 --2- 172.21.15.28:0/551435031 >> v2:172.21.15.189:3300/0 conn(0x7f21a815a7a0 0x7f21a80ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 -- 172.21.15.28:0/551435031 >> v2:172.21.15.189:3300/0 conn(0x7f21a815a7a0 msgr2=0x7f21a80ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 --2- 172.21.15.28:0/551435031 >> v2:172.21.15.189:3300/0 conn(0x7f21a815a7a0 0x7f21a80ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 -- 172.21.15.28:0/551435031 >> v1:172.21.15.189:6789/0 conn(0x7f21a80ce9d0 legacy=0x7f21a80b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21a7400640 1 -- 172.21.15.28:0/551435031 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817254161 0 0) 0x7f2194001110 con 0x7f21a80ce9d0 2024-09-13T17:14:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.156+0000 7f21adc00640 1 -- 172.21.15.28:0/551435031 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80b2430 con 0x7f21a80b1c80 2024-09-13T17:14:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.157+0000 7f21adc00640 1 --2- 172.21.15.28:0/551435031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009e50 tx=0x7f21a002a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d5312551c657d5c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21a7400640 1 -- 172.21.15.28:0/551435031 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a002aea0 con 0x7f21a80b1c80 2024-09-13T17:14:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21a7400640 1 -- 172.21.15.28:0/551435031 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a002d050 con 0x7f21a80b1c80 2024-09-13T17:14:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21a7400640 1 -- 172.21.15.28:0/551435031 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0031820 con 0x7f21a80b1c80 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 -- 172.21.15.28:0/551435031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 msgr2=0x7f21a80cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 --2- 172.21.15.28:0/551435031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f21a0009e50 tx=0x7f21a002a680 comp rx=0 tx=0).stop 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 -- 172.21.15.28:0/551435031 shutdown_connections 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 --2- 172.21.15.28:0/551435031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a80cd160 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 --2- 172.21.15.28:0/551435031 >> v2:172.21.15.189:3300/0 conn(0x7f21a815a7a0 0x7f21a80ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.158+0000 7f21afc00640 1 -- 172.21.15.28:0/551435031 >> 172.21.15.28:0/551435031 conn(0x7f21a8069e80 msgr2=0x7f21a806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.159+0000 7f21afc00640 1 -- 172.21.15.28:0/551435031 shutdown_connections 2024-09-13T17:14:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.159+0000 7f21afc00640 1 -- 172.21.15.28:0/551435031 wait complete. 2024-09-13T17:14:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.160+0000 7f21afc00640 1 Processor -- start 2024-09-13T17:14:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.160+0000 7f21afc00640 1 -- start start 2024-09-13T17:14:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a8154b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 0x7f21a80c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8158b90 con 0x7f21a80ce9d0 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8157af0 con 0x7f21a815a7a0 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21afc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a8157dd0 con 0x7f21a80b1c80 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.161+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40908/0 (socket says 172.21.15.28:40908) 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 -- 172.21.15.28:0/1053032624 learned_addr learned my addr 172.21.15.28:0/1053032624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21ad200640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a8154b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21adc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 0x7f21a80c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 -- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 msgr2=0x7f21a8154b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a8154b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 -- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 msgr2=0x7f21a80c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 0x7f21a80c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 -- 172.21.15.28:0/1053032624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009b30 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21adc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 0x7f21a80c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a7e00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f219400c750 tx=0x7f219400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2194005270 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21940053f0 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219400a880 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.162+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a81580b0 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.163+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a8158670 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.164+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8158860 con 0x7f21a80ce9d0 2024-09-13T17:14:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.165+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f219400a9e0 con 0x7f21a80ce9d0 2024-09-13T17:14:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.166+0000 7f21a4c00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 0x7f218806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:14:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.166+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2194088720 con 0x7f21a80ce9d0 2024-09-13T17:14:39.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.167+0000 7f21ad200640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 0x7f218806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:14:39.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.167+0000 7f21ad200640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 0x7f218806d0c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21a80cd540 tx=0x7f219c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:14:39.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.170+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2194053990 con 0x7f21a80ce9d0 2024-09-13T17:14:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.387+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f21a80ce490 con 0x7f21a80ce9d0 2024-09-13T17:14:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.610+0000 7f21a4c00640 1 -- 172.21.15.28:0/1053032624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_new_snaps", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v95) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2194017020 con 0x7f21a80ce9d0 2024-09-13T17:14:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:enabled new snapshots 2024-09-13T17:14:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 msgr2=0x7f218806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 0x7f218806d0c0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f21a80cd540 tx=0x7f219c008040 comp rx=0 tx=0).stop 2024-09-13T17:14:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 msgr2=0x7f21a80c8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:14:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f219400c750 tx=0x7f219400cc10 comp rx=0 tx=0).stop 2024-09-13T17:14:39.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 shutdown_connections 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f218806ac00 0x7f218806d0c0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21a815a7a0 0x7f21a80c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21a80ce9d0 0x7f21a80c8260 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 --2- 172.21.15.28:0/1053032624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21a80b1c80 0x7f21a8154b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 >> 172.21.15.28:0/1053032624 conn(0x7f21a8069e80 msgr2=0x7f21a80ca010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:14:39.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 shutdown_connections 2024-09-13T17:14:39.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:14:39.612+0000 7f21afc00640 1 -- 172.21.15.28:0/1053032624 wait complete. 2024-09-13T17:14:39.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:14:39.636 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir stray_eval_dir) 2024-09-13T17:14:39.681 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:14:39.681 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:14:39.757 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:14:39.757 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:14:39.757 DEBUG:teuthology.orchestra.run.smithi028:> import uuid 2024-09-13T17:14:39.757 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> n = 2000 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = False 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> if 3 > 0: 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = True 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> path = "/home/ubuntu/cephtest/mnt.0/stray_eval_dir/file" 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> dpath = os.path.dirname(path) 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> fnameprefix = os.path.basename(path) 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> os.makedirs(dpath, exist_ok=True) 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> for i in range(n): 2024-09-13T17:14:39.758 DEBUG:teuthology.orchestra.run.smithi028:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> with open(fpath, '"'"'w'"'"') as f: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> f.write(f"{i}") 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> f.flush() 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(f.fileno()) 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> os.unlink(fpath) 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> if create_hard_links: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> for j in range(3): 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:14:39.759 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:14:39.760 DEBUG:teuthology.orchestra.run.smithi028:> finally: 2024-09-13T17:14:39.760 DEBUG:teuthology.orchestra.run.smithi028:> os.close(dirfd) 2024-09-13T17:14:39.760 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:14:40.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:14:40.044 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:15:10.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:15:10.151 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:15:10.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:15:10.722 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-09-13T17:15:10.788 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:15:10.788 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec find /home/ubuntu/cephtest/mnt.0/stray_eval_dir/ -type f -delete) 2024-09-13T17:15:38.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:15:38.837 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir /home/ubuntu/cephtest/mnt.0/stray_eval_dir/.snap/snap1-stray_eval_dir) 2024-09-13T17:15:38.956 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 perf dump 2024-09-13T17:15:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.113+0000 7f97ae600640 1 Processor -- start 2024-09-13T17:15:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.113+0000 7f97ae600640 1 -- start start 2024-09-13T17:15:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.114+0000 7f97ae600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f97a8071490 0x7f97a806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.114+0000 7f97ae600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a806c8b0 con 0x7f97a815a780 2024-09-13T17:15:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.114+0000 7f97ae600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a806cb90 con 0x7f97a815ab60 2024-09-13T17:15:39.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.114+0000 7f97ae600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a80cc980 con 0x7f97a8071490 2024-09-13T17:15:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97a815a780 0x7f97a8070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47132/0 (socket says 172.21.15.28:47132) 2024-09-13T17:15:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7400640 1 -- 172.21.15.28:0/1746924227 learned_addr learned my addr 172.21.15.28:0/1746924227 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1746924227 >> v2:172.21.15.189:3300/0 conn(0x7f97a8071490 0x7f97a806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 366908201 0 0) 0x7f97a806c8b0 con 0x7f97a815a780 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f978c003520 con 0x7f97a815a780 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104288826 0 0) 0x7f97a806cb90 con 0x7f97a815ab60 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a806c8b0 con 0x7f97a815ab60 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7e00640 1 -- 172.21.15.28:0/1746924227 >> v1:172.21.15.189:6789/0 conn(0x7f97a815ab60 legacy=0x7f97a806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7e00640 1 -- 172.21.15.28:0/1746924227 >> v1:172.21.15.28:6789/0 conn(0x7f97a815a780 legacy=0x7f97a8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.115+0000 7f97a7e00640 1 -- 172.21.15.28:0/1746924227 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80ccc60 con 0x7f97a8071490 2024-09-13T17:15:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.116+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1746924227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a8071490 0x7f97a806c250 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009d90 tx=0x7f97a002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93a4cbac81de885a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0007740 con 0x7f97a8071490 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a002d050 con 0x7f97a8071490 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97a6000640 1 -- 172.21.15.28:0/1746924227 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0007ac0 con 0x7f97a8071490 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 -- 172.21.15.28:0/1746924227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a8071490 msgr2=0x7f97a806c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 --2- 172.21.15.28:0/1746924227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a8071490 0x7f97a806c250 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009d90 tx=0x7f97a002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 -- 172.21.15.28:0/1746924227 shutdown_connections 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 --2- 172.21.15.28:0/1746924227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a8071490 0x7f97a806c250 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 -- 172.21.15.28:0/1746924227 >> 172.21.15.28:0/1746924227 conn(0x7f97a80b0270 msgr2=0x7f97a80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 -- 172.21.15.28:0/1746924227 shutdown_connections 2024-09-13T17:15:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.117+0000 7f97ae600640 1 -- 172.21.15.28:0/1746924227 wait complete. 2024-09-13T17:15:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.119+0000 7f97ae600640 1 Processor -- start 2024-09-13T17:15:39.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.119+0000 7f97ae600640 1 -- start start 2024-09-13T17:15:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97a8071490 0x7f97a815cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a815ab60 0x7f97a8155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8153730 con 0x7f97a815a780 2024-09-13T17:15:39.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a815d7c0 con 0x7f97a815ab60 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97ae600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8152e00 con 0x7f97a8071490 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56876/0 (socket says 172.21.15.28:56876) 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97a6a00640 1 -- 172.21.15.28:0/1748416451 learned_addr learned my addr 172.21.15.28:0/1748416451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97a7e00640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a815ab60 0x7f97a8155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.120+0000 7f97a7400640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97a8071490 0x7f97a815cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 -- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97a8071490 msgr2=0x7f97a815cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97a8071490 0x7f97a815cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 -- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a815ab60 msgr2=0x7f97a8155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a815ab60 0x7f97a8155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 -- 172.21.15.28:0/1748416451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a0009a70 con 0x7f97a815a780 2024-09-13T17:15:39.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97a6a00640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f979400c430 tx=0x7f979400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f979f400640 1 -- 172.21.15.28:0/1748416451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97940052d0 con 0x7f97a815a780 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f979f400640 1 -- 172.21.15.28:0/1748416451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9794005450 con 0x7f97a815a780 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f979f400640 1 -- 172.21.15.28:0/1748416451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9794003830 con 0x7f97a815a780 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.121+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a81530e0 con 0x7f97a815a780 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.122+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a815d420 con 0x7f97a815a780 2024-09-13T17:15:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.124+0000 7f979f400640 1 -- 172.21.15.28:0/1748416451 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9794003990 con 0x7f97a815a780 2024-09-13T17:15:39.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.124+0000 7f979f400640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 0x7f978806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.125+0000 7f979f400640 1 -- 172.21.15.28:0/1748416451 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97940888b0 con 0x7f97a815a780 2024-09-13T17:15:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.125+0000 7f97a7400640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 0x7f978806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.125+0000 7f97b1413740 1 Processor -- start 2024-09-13T17:15:39.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.125+0000 7f97b1413740 1 -- start start 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a7400640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 0x7f978806d0c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f97980011e0 tx=0x7f9798007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff6b40 0x562e37046850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e37046d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e36f04000 con 0x562e37020830 2024-09-13T17:15:39.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e36f61d40 con 0x562e36ff6b40 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e36f5f4d0 con 0x562e36ff9380 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e37046d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:52192/0 (socket says 172.21.15.28:52192) 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff6b40 0x562e37046850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.126+0000 7f97a7400640 1 -- 172.21.15.28:0/714782433 learned_addr learned my addr 172.21.15.28:0/714782433 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 -- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff6b40 msgr2=0x562e37046850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff6b40 0x562e37046850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 -- 172.21.15.28:0/714782433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 msgr2=0x562e37046d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e37046d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 -- 172.21.15.28:0/714782433 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e36f163a0 con 0x562e36ff9380 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7e00640 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e37046d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f97a7400640 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9798001690 tx=0x7f9798012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c22f710e68942fb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f9793e00640 1 -- 172.21.15.28:0/714782433 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97980149d0 con 0x562e36ff9380 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f9793e00640 1 -- 172.21.15.28:0/714782433 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9798014b30 con 0x562e36ff9380 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.127+0000 7f9793e00640 1 -- 172.21.15.28:0/714782433 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9798014ea0 con 0x562e36ff9380 2024-09-13T17:15:39.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 -- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 msgr2=0x562e36ff9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9798001690 tx=0x7f9798012000 comp rx=0 tx=0).stop 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 -- 172.21.15.28:0/714782433 shutdown_connections 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e37046d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff6b40 0x562e37046850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 --2- 172.21.15.28:0/714782433 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e36ff9760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 -- 172.21.15.28:0/714782433 >> 172.21.15.28:0/714782433 conn(0x562e36ff5ab0 msgr2=0x562e3703df60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 -- 172.21.15.28:0/714782433 shutdown_connections 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.128+0000 7f97b1413740 1 -- 172.21.15.28:0/714782433 wait complete. 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.129+0000 7f97b1413740 1 -- start start 2024-09-13T17:15:39.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.129+0000 7f97b1413740 1 Processor -- start 2024-09-13T17:15:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e37045d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37020830 0x562e37043860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e36f5adc0 con 0x562e37024310 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e36f193e0 con 0x562e37020830 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e36f19080 con 0x562e36ff9380 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97b1413740 10 client.0 _ll_register_callbacks cb 0x562e36f54430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56904/0 (socket says 172.21.15.28:56904) 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.130+0000 7f97a7e00640 1 -- 172.21.15.28:0/2112169399 learned_addr learned my addr 172.21.15.28:0/2112169399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7400640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e37045d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a6a00640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37020830 0x562e37043860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 -- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 msgr2=0x562e37045d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e37045d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 -- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37020830 msgr2=0x562e37043860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37020830 0x562e37043860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 -- 172.21.15.28:0/2112169399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f978c003520 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97a7e00640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009eb0 tx=0x7f97a0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6527ac6e8e915e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f9790c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0042070 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97a003c420 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.131+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e36f84600 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97a0009450 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 10 client.8190 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x562e36f59440 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562e36f84600 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f97a0038040 con 0x562e37024310 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 10 client.8190 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 10 client.8190 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 20 client.8190 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37021760 0x562e37021b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 4 client.8190 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e36f4cc90 con 0x562e37021760 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.132+0000 7f97a7400640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37021760 0x562e37021b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.133+0000 7f97a7400640 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37021760 0x562e37021b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=817ed7f2d72dd3f8 server_cookie=12b1f055756b8e0b in_seq=0 out_seq=0 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.133+0000 7f9790c00640 10 client.8190 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f9790c00640 1 -- 172.21.15.28:0/2112169399 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562e36f4cc90 con 0x562e37021760 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f9790c00640 10 client.8190 handle_command_reply: tid=0 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 client.8190 shutdown 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 msgr2=0x562e37043da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009eb0 tx=0x7f97a0004ab0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 shutdown_connections 2024-09-13T17:15:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37021760 0x562e37021b70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37024310 0x562e37043da0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37020830 0x562e37043860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 --2- 172.21.15.28:0/2112169399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e37045d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f9790c00640 0 client.8190 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 >> 172.21.15.28:0/2112169399 conn(0x562e36ff6b40 msgr2=0x562e3703f700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 shutdown_connections 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 1 -- 172.21.15.28:0/2112169399 wait complete. 2024-09-13T17:15:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.134+0000 7f97b1413740 20 client.8190 trim_cache size 0 max 16384 2024-09-13T17:15:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.148+0000 7f97b1413740 1 Processor -- start 2024-09-13T17:15:39.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.148+0000 7f97b1413740 1 -- start start 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e370405e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37024310 0x562e3703dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e36cc2040 con 0x562e37020830 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e37024f00 con 0x562e37024310 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.149+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e37021100 con 0x562e36ff9380 2024-09-13T17:15:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56906/0 (socket says 172.21.15.28:56906) 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 -- 172.21.15.28:0/3565345162 learned_addr learned my addr 172.21.15.28:0/3565345162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a7400640 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e370405e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a7e00640 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37024310 0x562e3703dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 -- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 msgr2=0x562e370405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e370405e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 -- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37024310 msgr2=0x562e3703dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37024310 0x562e3703dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 -- 172.21.15.28:0/3565345162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e36efb5d0 con 0x562e37020830 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f97a6a00640 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9794007fd0 tx=0x7f9794001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322f40022f6a3d5a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f979cc00640 1 -- 172.21.15.28:0/3565345162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979406f170 con 0x562e37020830 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f979cc00640 1 -- 172.21.15.28:0/3565345162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9794053f00 con 0x562e37020830 2024-09-13T17:15:39.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.150+0000 7f979cc00640 1 -- 172.21.15.28:0/3565345162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979405b350 con 0x562e37020830 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 -- 172.21.15.28:0/3565345162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 msgr2=0x562e3703d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f9794007fd0 tx=0x7f9794001150 comp rx=0 tx=0).stop 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 -- 172.21.15.28:0/3565345162 shutdown_connections 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e37024310 0x562e3703dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703d790 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 --2- 172.21.15.28:0/3565345162 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e36ff9380 0x562e370405e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 -- 172.21.15.28:0/3565345162 >> 172.21.15.28:0/3565345162 conn(0x562e36ff6b40 msgr2=0x562e3701f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 -- 172.21.15.28:0/3565345162 shutdown_connections 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.151+0000 7f97b1413740 1 -- 172.21.15.28:0/3565345162 wait complete. 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.152+0000 7f97b1413740 1 -- start start 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.152+0000 7f97b1413740 1 Processor -- start 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 0x562e370a7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e37024310 0x562e3703ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e3701f930 con 0x562e37020830 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e370a9a60 con 0x562e36ff9380 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562e3701f6e0 con 0x562e37024310 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97b1413740 10 client.0 _ll_register_callbacks cb 0x7f97a0009c90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56918/0 (socket says 172.21.15.28:56918) 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97a6a00640 1 -- 172.21.15.28:0/2681163367 learned_addr learned my addr 172.21.15.28:0/2681163367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.153+0000 7f97a7e00640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e37024310 0x562e3703ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a7400640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 0x562e370a7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 -- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e37024310 msgr2=0x562e3703ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e37024310 0x562e3703ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 -- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 msgr2=0x562e370a7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 0x562e370a7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 -- 172.21.15.28:0/2681163367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9794088040 con 0x562e37020830 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a7400640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 0x562e370a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97a6a00640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9794057780 tx=0x7f9794086ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07b965001a7c977 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f9790c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979405f8a0 con 0x562e37020830 2024-09-13T17:15:39.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9794054070 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9798014b30 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f979406f170 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97b1413740 10 client.8196 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.154+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x562e36f209b0 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9798014b30 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f97940535e0 con 0x562e37020830 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 10 client.8196 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 10 client.8196 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 20 client.8196 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37023140 0x562e37041450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 4 client.8196 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x562e36c4e4e0 con 0x562e37023140 2024-09-13T17:15:39.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.155+0000 7f97a7400640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37023140 0x562e37041450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97a7400640 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37023140 0x562e37041450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=831ee2b234f0ef7e server_cookie=93955040913c9712 in_seq=0 out_seq=0 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f9790c00640 10 client.8196 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f9790c00640 1 -- 172.21.15.28:0/2681163367 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17910 (crc 0 0 0) 0x562e36c4e4e0 con 0x562e37023140 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f9790c00640 10 client.8196 handle_command_reply: tid=0 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 client.8196 shutdown 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 msgr2=0x562e3703ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f9794057780 tx=0x7f9794086ca0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 shutdown_connections 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x562e37023140 0x562e37041450 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562e37024310 0x562e3703ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e37020830 0x562e3703ea20 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f9790c00640 0 client.8196 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 --2- 172.21.15.28:0/2681163367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e36ff9380 0x562e370a7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.156+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 >> 172.21.15.28:0/2681163367 conn(0x562e36ff6b40 msgr2=0x562e3701d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 shutdown_connections 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97b1413740 1 -- 172.21.15.28:0/2681163367 wait complete. 2024-09-13T17:15:39.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97b1413740 20 client.8196 trim_cache size 0 max 16384 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3416, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 3417, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 960209, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 40401155, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 9, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.505995825, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.271682145, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.073214587, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.128251125, 2024-09-13T17:15:39.163 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3416, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5.309022502, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001554163 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000201, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000100 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.164 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 29592, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 50478, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 10694438, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 41472885, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 4.904178223, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 2.882572605, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 2.503722607, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.059112546, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 50477, 2024-09-13T17:15:39.165 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.964632155, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000038921 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 71236, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017437805, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000244 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 121040, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 19040 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4904, 2024-09-13T17:15:39.166 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4919, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1567433, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 54077860, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.615888685, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.312889114, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.112226506, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.160577017, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4918, 2024-09-13T17:15:39.167 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.138328896, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000434796 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 110928 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-13T17:15:39.168 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4746, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.373290830, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003449913 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:15:39.169 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1789, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.924716435, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000516890 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 25948, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 25948, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 25948, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 254.608344355, 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.009812253 2024-09-13T17:15:39.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 43, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 18218, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 8834, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 9384, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 5598, 2024-09-13T17:15:39.171 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 60, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 60, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 2, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 48767, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 48165, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 161, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-13T17:15:39.172 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-13T17:15:39.173 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 1198, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 4034, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 4260, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 2000, 2024-09-13T17:15:39.174 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 21, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 14092 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 2000, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 58, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 4128, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 1942, 2024-09-13T17:15:39.175 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 2233, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.176 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 30179, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 23201, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 20073, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 10106, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 3128, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 46, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 34, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 30, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 16, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 4, 2024-09-13T17:15:39.177 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 67113615, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 101885958, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4274, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 76.241633257, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.017838472 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 2014, 2024-09-13T17:15:39.178 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 2014, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 0, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 13, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 13, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 18218, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 20013, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 1795, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 2, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 2770, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 2768, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 111768, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 207200 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.179 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 29763, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 25948, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 4, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2000, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.336601444, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000668300 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 29, 2024-09-13T17:15:39.180 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006855157, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000236384 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6000, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.886349622, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001147724 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-13T17:15:39.181 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7673, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.645871337, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000866137 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.182 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000050139, 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000050139 2024-09-13T17:15:39.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.003285389, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003285389 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-13T17:15:39.184 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006367092, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006367092 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.571651706, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.071456463 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2233, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 222.311504972, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.099557324 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.186 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-13T17:15:39.187 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.034863823, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.034863823 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000251929, 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000251929 2024-09-13T17:15:39.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.189 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-13T17:15:39.190 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8000, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.804691745, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002100586 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-13T17:15:39.191 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 18299, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 18299, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 57, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-13T17:15:39.192 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-13T17:15:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-13T17:15:39.194 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 18842693, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 6782, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 440, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 5, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-13T17:15:39.195 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 5312, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 75, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 20268502, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 868499, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-13T17:15:39.196 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 9725, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 107498475, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 30, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 9725, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9725, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 102.684123810, 2024-09-13T17:15:39.197 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.010558778 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 9725, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16391 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 9725, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 37, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 9688, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-13T17:15:39.198 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 30, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 3279, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 37, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 4311, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 39, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 1942, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 49, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-13T17:15:39.199 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 6558, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-13T17:15:39.200 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-13T17:15:39.201 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 111, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 1, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 8, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 8, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 113, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 33 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-13T17:15:39.202 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 32, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 32, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 1942, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 1942, 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-13T17:15:39.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 37908, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 9805551, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 37908, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 37908, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 9805551, 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:15:39.204 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 9725, 2024-09-13T17:15:39.205 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 107710153, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 9725, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 107710153, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:15:39.206 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 9725, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 9725, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 9725, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 9725, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:15:39.207 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 1942, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 211678, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 1942, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 37, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 211678, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x564e0163c0a0": { 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 84, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 30179, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 97691654, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 30179, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4274, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 97691570, 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:15:39.209 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:15:39.210 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:15:39.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:15:39.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.210 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.210 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:15:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 msgr2=0x7f978806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 0x7f978806d0c0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f97980011e0 tx=0x7f9798007d30 comp rx=0 tx=0).stop 2024-09-13T17:15:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 msgr2=0x7f97a8139d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f979400c430 tx=0x7f979400c8f0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 shutdown_connections 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f978806ac00 0x7f978806d0c0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97a815ab60 0x7f97a8155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97a815a780 0x7f97a8139d50 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 --2- 172.21.15.28:0/1748416451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97a8071490 0x7f97a815cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.157+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 >> 172.21.15.28:0/1748416451 conn(0x7f97a80b0270 msgr2=0x7f97a806b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.158+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 shutdown_connections 2024-09-13T17:15:39.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.158+0000 7f97ae600640 1 -- 172.21.15.28:0/1748416451 wait complete. 2024-09-13T17:15:39.213 INFO:tasks.cephfs.test_scrub_checks:num of strays: 2000 2024-09-13T17:15:39.213 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub start '~mdsdir' recursive 2024-09-13T17:15:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.371+0000 7ff111200640 1 Processor -- start 2024-09-13T17:15:39.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.371+0000 7ff111200640 1 -- start start 2024-09-13T17:15:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff111200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff10c159970 0x7ff10c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff111200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10c15a4c0 con 0x7ff10c159590 2024-09-13T17:15:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff111200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10c071580 con 0x7ff10c0b2630 2024-09-13T17:15:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff111200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10c0b2ae0 con 0x7ff10c159970 2024-09-13T17:15:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff109400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff10c159970 0x7ff10c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff109e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff10c159590 0x7ff10c070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47148/0 (socket says 172.21.15.28:47148) 2024-09-13T17:15:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.372+0000 7ff109e00640 1 -- 172.21.15.28:0/1387502273 learned_addr learned my addr 172.21.15.28:0/1387502273 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:15:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.373+0000 7ff109400640 1 -- 172.21.15.28:0/1387502273 >> v1:172.21.15.189:6789/0 conn(0x7ff10c0b2630 legacy=0x7ff10c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.373+0000 7ff109400640 1 -- 172.21.15.28:0/1387502273 >> v1:172.21.15.28:6789/0 conn(0x7ff10c159590 legacy=0x7ff10c070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.373+0000 7ff108a00640 1 -- 172.21.15.28:0/1387502273 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278462863 0 0) 0x7ff10c15a4c0 con 0x7ff10c159590 2024-09-13T17:15:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.373+0000 7ff109400640 1 -- 172.21.15.28:0/1387502273 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10c159e70 con 0x7ff10c159970 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.374+0000 7ff109400640 1 --2- 172.21.15.28:0/1387502273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c159970 0x7ff10c0b20f0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e0a8e96b61ab9b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.374+0000 7ff108a00640 1 -- 172.21.15.28:0/1387502273 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8032440 con 0x7ff10c159970 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.374+0000 7ff108a00640 1 -- 172.21.15.28:0/1387502273 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f80325a0 con 0x7ff10c159970 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.374+0000 7ff108a00640 1 -- 172.21.15.28:0/1387502273 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8032930 con 0x7ff10c159970 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 -- 172.21.15.28:0/1387502273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c159970 msgr2=0x7ff10c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 --2- 172.21.15.28:0/1387502273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c159970 0x7ff10c0b20f0 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7ff0f80097a0 tx=0x7ff0f802a230 comp rx=0 tx=0).stop 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 -- 172.21.15.28:0/1387502273 shutdown_connections 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 --2- 172.21.15.28:0/1387502273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c159970 0x7ff10c0b20f0 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 -- 172.21.15.28:0/1387502273 >> 172.21.15.28:0/1387502273 conn(0x7ff10c069e80 msgr2=0x7ff10c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 -- 172.21.15.28:0/1387502273 shutdown_connections 2024-09-13T17:15:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.375+0000 7ff111200640 1 -- 172.21.15.28:0/1387502273 wait complete. 2024-09-13T17:15:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.376+0000 7ff111200640 1 Processor -- start 2024-09-13T17:15:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 -- start start 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff10c159590 0x7ff10c15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff10c159970 0x7ff10c1ed2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c152b80 con 0x7ff10c0b2630 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c14e0f0 con 0x7ff10c159970 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.377+0000 7ff111200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10c14e3d0 con 0x7ff10c159590 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56942/0 (socket says 172.21.15.28:56942) 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 -- 172.21.15.28:0/1128063094 learned_addr learned my addr 172.21.15.28:0/1128063094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109400640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff10c159590 0x7ff10c15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff10a800640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff10c159970 0x7ff10c1ed2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 -- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff10c159590 msgr2=0x7ff10c15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff10c159590 0x7ff10c15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 -- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff10c159970 msgr2=0x7ff10c1ed2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff10c159970 0x7ff10c1ed2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.378+0000 7ff109e00640 1 -- 172.21.15.28:0/1128063094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f8009400 con 0x7ff10c0b2630 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff109e00640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c4a0 tx=0x7ff0fc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff102000640 1 -- 172.21.15.28:0/1128063094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc0074e0 con 0x7ff10c0b2630 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff102000640 1 -- 172.21.15.28:0/1128063094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc007640 con 0x7ff10c0b2630 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff10c14e6b0 con 0x7ff10c0b2630 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff102000640 1 -- 172.21.15.28:0/1128063094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc004a00 con 0x7ff10c0b2630 2024-09-13T17:15:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.379+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c15c410 con 0x7ff10c0b2630 2024-09-13T17:15:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.381+0000 7ff102000640 1 -- 172.21.15.28:0/1128063094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff0fc004b60 con 0x7ff10c0b2630 2024-09-13T17:15:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.382+0000 7ff102000640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 0x7ff0ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.382+0000 7ff102000640 1 -- 172.21.15.28:0/1128063094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc0887a0 con 0x7ff10c0b2630 2024-09-13T17:15:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.382+0000 7ff109400640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 0x7ff0ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.382+0000 7ff114038740 1 Processor -- start 2024-09-13T17:15:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 -- start start 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff109400640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 0x7ff0ec06d0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8034000 tx=0x7ff0f802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40b920 0x5611ee40e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee408750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee3c95e0 con 0x5611ee40e160 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee377df0 con 0x5611ee40f8f0 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.383+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611ee375580 con 0x5611ee40b920 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56946/0 (socket says 172.21.15.28:56946) 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 -- 172.21.15.28:0/2735452751 learned_addr learned my addr 172.21.15.28:0/2735452751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109400640 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40b920 0x5611ee40e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff10a800640 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee408750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 -- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40b920 msgr2=0x5611ee40e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40b920 0x5611ee40e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 -- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 msgr2=0x5611ee408750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee408750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 -- 172.21.15.28:0/2735452751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611ee3380e0 con 0x5611ee40e160 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.384+0000 7ff109e00640 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc05f4f0 tx=0x7ff0fc057110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4723e92b70b3f8a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff0f6a00640 1 -- 172.21.15.28:0/2735452751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc05fa20 con 0x5611ee40e160 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff0f6a00640 1 -- 172.21.15.28:0/2735452751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc004e10 con 0x5611ee40e160 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff0f6a00640 1 -- 172.21.15.28:0/2735452751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc05b150 con 0x5611ee40e160 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 -- 172.21.15.28:0/2735452751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 msgr2=0x5611ee453900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc05f4f0 tx=0x7ff0fc057110 comp rx=0 tx=0).stop 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 -- 172.21.15.28:0/2735452751 shutdown_connections 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee408750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40b920 0x5611ee40e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 --2- 172.21.15.28:0/2735452751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee453900 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 -- 172.21.15.28:0/2735452751 >> 172.21.15.28:0/2735452751 conn(0x5611ee40a920 msgr2=0x5611ee452dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.385+0000 7ff114038740 1 -- 172.21.15.28:0/2735452751 shutdown_connections 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.386+0000 7ff114038740 1 -- 172.21.15.28:0/2735452751 wait complete. 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.386+0000 7ff114038740 1 -- start start 2024-09-13T17:15:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.386+0000 7ff114038740 1 Processor -- start 2024-09-13T17:15:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40e160 0x5611ee408780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee456c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee370e70 con 0x5611ee40f8f0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee3120a0 con 0x5611ee40e160 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.387+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611ee311d40 con 0x5611ee4567c0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff114038740 10 client.0 _ll_register_callbacks cb 0x5611ee3378c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56962/0 (socket says 172.21.15.28:56962) 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 -- 172.21.15.28:0/4250745774 learned_addr learned my addr 172.21.15.28:0/4250745774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff10a800640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee456c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109e00640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40e160 0x5611ee408780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 -- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 msgr2=0x5611ee456c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee456c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 -- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40e160 msgr2=0x5611ee408780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40e160 0x5611ee408780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 -- 172.21.15.28:0/4250745774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff10c15c410 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff109400640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009890 tx=0x7ff0f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e179b102774e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.388+0000 7ff0eb400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f8049840 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0f80499a0 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611ee362d10 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0f8049b60 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 10 client.8211 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x5611ee36f4f0 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5611ee362d10 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff0f8044030 con 0x5611ee40f8f0 2024-09-13T17:15:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 10 client.8211 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 10 client.8211 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 20 client.8211 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee40ca30 0x5611ee40ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 4 client.8211 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.389+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611ee064700 con 0x5611ee40ca30 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.390+0000 7ff109e00640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee40ca30 0x5611ee40ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.390+0000 7ff109e00640 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee40ca30 0x5611ee40ce40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e94394ce02b22ef4 server_cookie=fb9e7c93d5939304 in_seq=0 out_seq=0 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.390+0000 7ff0eb400640 10 client.8211 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff0eb400640 1 -- 172.21.15.28:0/4250745774 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5611ee064700 con 0x5611ee40ca30 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff0eb400640 10 client.8211 handle_command_reply: tid=0 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 client.8211 shutdown 2024-09-13T17:15:39.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 msgr2=0x5611ee42ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009890 tx=0x7ff0f8004730 comp rx=0 tx=0).stop 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 shutdown_connections 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee40ca30 0x5611ee40ce40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee456c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40f8f0 0x5611ee42ca10 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 --2- 172.21.15.28:0/4250745774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40e160 0x5611ee408780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff0eb400640 0 client.8211 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 >> 172.21.15.28:0/4250745774 conn(0x5611ee40b920 msgr2=0x5611ee454f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 shutdown_connections 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.391+0000 7ff114038740 1 -- 172.21.15.28:0/4250745774 wait complete. 2024-09-13T17:15:39.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.392+0000 7ff114038740 20 client.8211 trim_cache size 0 max 16384 2024-09-13T17:15:39.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.406+0000 7ff114038740 1 Processor -- start 2024-09-13T17:15:39.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 -- start start 2024-09-13T17:15:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee454ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee4530f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee0d8260 con 0x5611ee40e160 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee40be30 con 0x5611ee40f8f0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.407+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611ee4082c0 con 0x5611ee4567c0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56972/0 (socket says 172.21.15.28:56972) 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 -- 172.21.15.28:0/2287460989 learned_addr learned my addr 172.21.15.28:0/2287460989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff10a800640 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee4530f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109400640 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee454ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 -- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 msgr2=0x5611ee4530f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee4530f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 -- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 msgr2=0x5611ee454ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee454ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 -- 172.21.15.28:0/2287460989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611ee311240 con 0x5611ee40e160 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff109e00640 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc014040 tx=0x7ff0fc01e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db90a6a27d18c42e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff0f7400640 1 -- 172.21.15.28:0/2287460989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc053a20 con 0x5611ee40e160 2024-09-13T17:15:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff0f7400640 1 -- 172.21.15.28:0/2287460989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc086040 con 0x5611ee40e160 2024-09-13T17:15:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.408+0000 7ff0f7400640 1 -- 172.21.15.28:0/2287460989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc05b150 con 0x5611ee40e160 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 -- 172.21.15.28:0/2287460989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 msgr2=0x5611ee4549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc014040 tx=0x7ff0fc01e000 comp rx=0 tx=0).stop 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 -- 172.21.15.28:0/2287460989 shutdown_connections 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee4567c0 0x5611ee4530f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee40f8f0 0x5611ee454ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 --2- 172.21.15.28:0/2287460989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee4549a0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 -- 172.21.15.28:0/2287460989 >> 172.21.15.28:0/2287460989 conn(0x5611ee40b920 msgr2=0x5611ee407730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 -- 172.21.15.28:0/2287460989 shutdown_connections 2024-09-13T17:15:39.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.409+0000 7ff114038740 1 -- 172.21.15.28:0/2287460989 wait complete. 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.410+0000 7ff114038740 1 -- start start 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.410+0000 7ff114038740 1 Processor -- start 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40f8f0 0x5611ee4c51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 0x5611ee454930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee479050 con 0x5611ee40e160 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ee453890 con 0x5611ee4567c0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5611ee478c90 con 0x5611ee40f8f0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff114038740 10 client.0 _ll_register_callbacks cb 0x7ff0f8035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56986/0 (socket says 172.21.15.28:56986) 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 -- 172.21.15.28:0/2575555944 learned_addr learned my addr 172.21.15.28:0/2575555944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109400640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40f8f0 0x5611ee4c51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff10a800640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 0x5611ee454930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 -- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40f8f0 msgr2=0x5611ee4c51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40f8f0 0x5611ee4c51c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 -- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 msgr2=0x5611ee454930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 0x5611ee454930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff109e00640 1 -- 172.21.15.28:0/2575555944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611ee3380e0 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.411+0000 7ff10a800640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 0x5611ee454930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff109e00640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0578a0 tx=0x7ff0fc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a249205854ccb9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0fc05b150 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0fc07be40 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0fc004e10 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0fc053df0 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 10 client.8217 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x5611ee336940 con 0x5611ee40e160 2024-09-13T17:15:39.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0fc068020 con 0x5611ee40e160 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff0fc071040 con 0x5611ee40e160 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 10 client.8217 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 10 client.8217 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 20 client.8217 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee42e4b0 0x5611ee454e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 4 client.8217 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}] 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.412+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}) v1 -- 0x5611ee075a00 con 0x5611ee42e4b0 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.413+0000 7ff109400640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee42e4b0 0x5611ee454e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.413+0000 7ff0eb400640 10 client.8217 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.413+0000 7ff109400640 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee42e4b0 0x5611ee454e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=560dc007e93ef92e server_cookie=e4e92de87de8fba6 in_seq=0 out_seq=0 2024-09-13T17:15:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.419+0000 7ff0eb400640 1 -- 172.21.15.28:0/2575555944 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5611ee075a00 con 0x5611ee42e4b0 2024-09-13T17:15:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.419+0000 7ff0eb400640 10 client.8217 handle_command_reply: tid=0 2024-09-13T17:15:39.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.419+0000 7ff114038740 1 client.8217 shutdown 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 msgr2=0x5611ee479530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc0578a0 tx=0x7ff0fc000ed0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 shutdown_connections 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x5611ee42e4b0 0x5611ee454e70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5611ee4567c0 0x5611ee454930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5611ee40f8f0 0x5611ee4c51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff0eb400640 0 client.8217 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 --2- 172.21.15.28:0/2575555944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5611ee40e160 0x5611ee479530 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 >> 172.21.15.28:0/2575555944 conn(0x5611ee40b920 msgr2=0x5611ee42d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 shutdown_connections 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 1 -- 172.21.15.28:0/2575555944 wait complete. 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.420+0000 7ff114038740 20 client.8217 trim_cache size 0 max 16384 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "8699b72d-3835-49ca-944d-885298e77d0d", 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:15:39.422 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 msgr2=0x7ff0ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 0x7ff0ec06d0f0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8034000 tx=0x7ff0f802e000 comp rx=0 tx=0).stop 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 msgr2=0x7ff10c152640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7ff0fc00c4a0 tx=0x7ff0fc00c960 comp rx=0 tx=0).stop 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 shutdown_connections 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff0ec06ac30 0x7ff0ec06d0f0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff10c159970 0x7ff10c1ed2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff10c159590 0x7ff10c15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 --2- 172.21.15.28:0/1128063094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff10c0b2630 0x7ff10c152640 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 >> 172.21.15.28:0/1128063094 conn(0x7ff10c069e80 msgr2=0x7ff10c070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 shutdown_connections 2024-09-13T17:15:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.421+0000 7ff111200640 1 -- 172.21.15.28:0/1128063094 wait complete. 2024-09-13T17:15:39.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T17:15:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.690+0000 7f49ce000640 1 Processor -- start 2024-09-13T17:15:39.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.690+0000 7f49ce000640 1 -- start start 2024-09-13T17:15:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49ce000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49c80cd830 0x7f49c806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49ce000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80cf530 con 0x7f49c806daf0 2024-09-13T17:15:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49ce000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49c80cf780 con 0x7f49c806e6a0 2024-09-13T17:15:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49ce000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49c80b2c60 con 0x7f49c80cd830 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49c7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f49c80cd830 0x7f49c806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.692+0000 7f49c6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f49c806daf0 0x7f49c806df90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47152/0 (socket says 172.21.15.28:47152) 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.693+0000 7f49c6a00640 1 -- 172.21.15.28:0/2782903575 learned_addr learned my addr 172.21.15.28:0/2782903575 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.693+0000 7f49c7400640 1 -- 172.21.15.28:0/2782903575 >> v1:172.21.15.189:6789/0 conn(0x7f49c806e6a0 legacy=0x7f49c806a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.693+0000 7f49c7400640 1 -- 172.21.15.28:0/2782903575 >> v1:172.21.15.28:6789/0 conn(0x7f49c806daf0 legacy=0x7f49c806df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.693+0000 7f49c7400640 1 -- 172.21.15.28:0/2782903575 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c80cfa60 con 0x7f49c80cd830 2024-09-13T17:15:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.695+0000 7f49c7400640 1 --2- 172.21.15.28:0/2782903575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c806d5b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bd7e009640cf3a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.695+0000 7f49c6000640 1 -- 172.21.15.28:0/2782903575 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0007740 con 0x7f49c80cd830 2024-09-13T17:15:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.695+0000 7f49c6000640 1 -- 172.21.15.28:0/2782903575 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c002d050 con 0x7f49c80cd830 2024-09-13T17:15:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.695+0000 7f49c6000640 1 -- 172.21.15.28:0/2782903575 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0007ac0 con 0x7f49c80cd830 2024-09-13T17:15:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.696+0000 7f49ce000640 1 -- 172.21.15.28:0/2782903575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 msgr2=0x7f49c806d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.696+0000 7f49ce000640 1 --2- 172.21.15.28:0/2782903575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c806d5b0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009d90 tx=0x7f49c002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.696+0000 7f49ce000640 1 -- 172.21.15.28:0/2782903575 shutdown_connections 2024-09-13T17:15:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.696+0000 7f49ce000640 1 --2- 172.21.15.28:0/2782903575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c806d5b0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.696+0000 7f49ce000640 1 -- 172.21.15.28:0/2782903575 >> 172.21.15.28:0/2782903575 conn(0x7f49c80999e0 msgr2=0x7f49c80b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.697+0000 7f49ce000640 1 -- 172.21.15.28:0/2782903575 shutdown_connections 2024-09-13T17:15:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.697+0000 7f49ce000640 1 -- 172.21.15.28:0/2782903575 wait complete. 2024-09-13T17:15:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.699+0000 7f49ce000640 1 Processor -- start 2024-09-13T17:15:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.699+0000 7f49ce000640 1 -- start start 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.700+0000 7f49ce000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49c806daf0 0x7f49c80b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.700+0000 7f49ce000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.700+0000 7f49ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c80ae730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49ce000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c8157f60 con 0x7f49c806e6a0 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49ce000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c80b56c0 con 0x7f49c80cd830 2024-09-13T17:15:39.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49ce000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49c81e7ca0 con 0x7f49c806daf0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57000/0 (socket says 172.21.15.28:57000) 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 -- 172.21.15.28:0/1668079110 learned_addr learned my addr 172.21.15.28:0/1668079110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c7400640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49c806daf0 0x7f49c80b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c7e00640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c80ae730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 -- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49c806daf0 msgr2=0x7f49c80b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49c806daf0 0x7f49c80b3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 -- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 msgr2=0x7f49c80ae730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c80ae730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.701+0000 7f49c6a00640 1 -- 172.21.15.28:0/1668079110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0009a70 con 0x7f49c806e6a0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49c6a00640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c500 tx=0x7f49b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49bf400640 1 -- 172.21.15.28:0/1668079110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b8005ce0 con 0x7f49c806e6a0 2024-09-13T17:15:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49bf400640 1 -- 172.21.15.28:0/1668079110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b8005e40 con 0x7f49c806e6a0 2024-09-13T17:15:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49bf400640 1 -- 172.21.15.28:0/1668079110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b80072f0 con 0x7f49c806e6a0 2024-09-13T17:15:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c81e7f80 con 0x7f49c806e6a0 2024-09-13T17:15:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.702+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c81e8540 con 0x7f49c806e6a0 2024-09-13T17:15:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.704+0000 7f49bf400640 1 -- 172.21.15.28:0/1668079110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f49b8007510 con 0x7f49c806e6a0 2024-09-13T17:15:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.705+0000 7f49bf400640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 0x7f49a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.705+0000 7f49bf400640 1 -- 172.21.15.28:0/1668079110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49b8088800 con 0x7f49c806e6a0 2024-09-13T17:15:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.706+0000 7f49c7400640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 0x7f49a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.706+0000 7f49d0e09740 1 Processor -- start 2024-09-13T17:15:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.706+0000 7f49d0e09740 1 -- start start 2024-09-13T17:15:39.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.706+0000 7f49c7400640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 0x7f49a806d0c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49c0034000 tx=0x7f49c002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4a0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d3596a0 con 0x56249d47a110 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d3bccc0 con 0x56249d452c60 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249d3ba450 con 0x56249d450420 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:48676/0 (socket says 172.21.15.28:48676) 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c7400640 1 -- 172.21.15.28:0/3737764556 learned_addr learned my addr 172.21.15.28:0/3737764556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:52290/0 (socket says 172.21.15.28:52290) 2024-09-13T17:15:39.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.707+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57012/0 (socket says 172.21.15.28:57012) 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 -- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 msgr2=0x56249d453040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 -- 172.21.15.28:0/3737764556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 msgr2=0x56249d4a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 -- 172.21.15.28:0/3737764556 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56249d3862e0 con 0x56249d450420 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c7400640 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49b8086d20 tx=0x7f49b8057630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa0666b16c693f67 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49b3e00640 1 -- 172.21.15.28:0/3737764556 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b805b4c0 con 0x56249d450420 2024-09-13T17:15:39.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49b3e00640 1 -- 172.21.15.28:0/3737764556 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b805fa80 con 0x56249d450420 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.708+0000 7f49b3e00640 1 -- 172.21.15.28:0/3737764556 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b805b170 con 0x56249d450420 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 -- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 msgr2=0x56249d4a0130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f49b8086d20 tx=0x7f49b8057630 comp rx=0 tx=0).stop 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 -- 172.21.15.28:0/3737764556 shutdown_connections 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d450420 0x56249d4a0130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3737764556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d452c60 0x56249d453040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 -- 172.21.15.28:0/3737764556 >> 172.21.15.28:0/3737764556 conn(0x56249d44f390 msgr2=0x56249d497840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 -- 172.21.15.28:0/3737764556 shutdown_connections 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.709+0000 7f49d0e09740 1 -- 172.21.15.28:0/3737764556 wait complete. 2024-09-13T17:15:39.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.710+0000 7f49d0e09740 1 -- start start 2024-09-13T17:15:39.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.710+0000 7f49d0e09740 1 Processor -- start 2024-09-13T17:15:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4b2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d3b5d40 con 0x56249d47a110 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d357280 con 0x56249d479c70 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249d356f20 con 0x56249d452c60 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49d0e09740 10 client.0 _ll_register_callbacks cb 0x56249d3af3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57026/0 (socket says 172.21.15.28:57026) 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 -- 172.21.15.28:0/625938847 learned_addr learned my addr 172.21.15.28:0/625938847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c7400640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c7e00640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 -- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 msgr2=0x56249d4b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 -- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 msgr2=0x56249d4b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4b2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.711+0000 7f49c6a00640 1 -- 172.21.15.28:0/625938847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c81e8540 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49c6a00640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f49b8057900 tx=0x7f49b8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff24e27da0bd13b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b805b540 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49b805b170 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56249d0a94e0 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49b8056c30 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49d0e09740 10 client.8229 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x56249d3b43c0 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.712+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56249d0a94e0 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f49b805f5f0 con 0x56249d47a110 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 10 client.8229 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 10 client.8229 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 20 client.8229 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d494bc0 0x56249d46bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 4 client.8229 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56249d0ba7e0 con 0x56249d494bc0 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49c7400640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d494bc0 0x56249d46bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49c7400640 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d494bc0 0x56249d46bbc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b270236ef697876 server_cookie=2ac13eb3bef2a98a in_seq=0 out_seq=0 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.713+0000 7f49b0c00640 10 client.8229 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49b0c00640 1 -- 172.21.15.28:0/625938847 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56249d0ba7e0 con 0x56249d494bc0 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49b0c00640 10 client.8229 handle_command_reply: tid=0 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 client.8229 shutdown 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 msgr2=0x56249d46f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f49b8057900 tx=0x7f49b8000ed0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 shutdown_connections 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d494bc0 0x56249d46bbc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4b2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d46f500 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49b0c00640 0 client.8229 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 --2- 172.21.15.28:0/625938847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.714+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 >> 172.21.15.28:0/625938847 conn(0x56249d450420 msgr2=0x56249d49b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.715+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 shutdown_connections 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.715+0000 7f49d0e09740 1 -- 172.21.15.28:0/625938847 wait complete. 2024-09-13T17:15:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.715+0000 7f49d0e09740 20 client.8229 trim_cache size 0 max 16384 2024-09-13T17:15:39.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.730+0000 7f49d0e09740 1 Processor -- start 2024-09-13T17:15:39.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.730+0000 7f49d0e09740 1 -- start start 2024-09-13T17:15:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4966d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d11d040 con 0x56249d47a110 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d4501d0 con 0x56249d479c70 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249d4b2170 con 0x56249d452c60 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57030/0 (socket says 172.21.15.28:57030) 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 -- 172.21.15.28:0/3400214156 learned_addr learned my addr 172.21.15.28:0/3400214156 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7400640 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 -- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 msgr2=0x56249d4c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 -- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 msgr2=0x56249d4966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.731+0000 7f49c7e00640 1 -- 172.21.15.28:0/3400214156 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56249d356610 con 0x56249d47a110 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.732+0000 7f49c7e00640 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f49b4001570 tx=0x7f49b4007db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8434afbdad2b186e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.732+0000 7f49bcc00640 1 -- 172.21.15.28:0/3400214156 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b40052e0 con 0x56249d47a110 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.732+0000 7f49bcc00640 1 -- 172.21.15.28:0/3400214156 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49b4005440 con 0x56249d47a110 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.732+0000 7f49bcc00640 1 -- 172.21.15.28:0/3400214156 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49b4015860 con 0x56249d47a110 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- 172.21.15.28:0/3400214156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 msgr2=0x56249d4714f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f49b4001570 tx=0x7f49b4007db0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- 172.21.15.28:0/3400214156 shutdown_connections 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d47a110 0x56249d4714f0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3400214156 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d452c60 0x56249d4c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- 172.21.15.28:0/3400214156 >> 172.21.15.28:0/3400214156 conn(0x56249d450420 msgr2=0x56249d47c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- 172.21.15.28:0/3400214156 shutdown_connections 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- 172.21.15.28:0/3400214156 wait complete. 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 -- start start 2024-09-13T17:15:39.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.733+0000 7f49d0e09740 1 Processor -- start 2024-09-13T17:15:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d47a110 0x56249d492fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d493a50 con 0x56249d452c60 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56249d472560 con 0x56249d479c70 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56249d50a5e0 con 0x56249d47a110 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.734+0000 7f49d0e09740 10 client.0 _ll_register_callbacks cb 0x7f49b806e070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57042/0 (socket says 172.21.15.28:57042) 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 -- 172.21.15.28:0/3524946200 learned_addr learned my addr 172.21.15.28:0/3524946200 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7e00640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d47a110 0x56249d492fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 -- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d47a110 msgr2=0x56249d492fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d47a110 0x56249d492fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 -- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 msgr2=0x56249d4f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4f34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 -- 172.21.15.28:0/3524946200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c80cf780 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49c7400640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009790 tx=0x7f49c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d160e1f719bfc28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.735+0000 7f49b0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0046ba0 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f49c0046d00 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56249d46c9d0 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=100) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49c0054050 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 10 client.8235 fetch_fsmap learned FSMap version 100 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=100,monmap=2+}) v3 -- 0x56249d37bb20 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56249d46c9d0 con 0x56249d452c60 2024-09-13T17:15:39.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 100) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f49c0038040 con 0x56249d452c60 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 10 client.8235 fetch_fsmap finished waiting for FSMap version 100 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 10 client.8235 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 20 client.8235 populate_metadata read hostname 'smithi028' 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d50aa70 0x56249d4957f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 4 client.8235 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f49b805fa80 con 0x56249d50aa70 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.736+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d50aa70 0x56249d4957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49b0c00640 10 client.8235 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49c6a00640 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d50aa70 0x56249d4957f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd8a5b6d96ca8769 server_cookie=3745360a64a9ba87 in_seq=0 out_seq=0 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49b0c00640 1 -- 172.21.15.28:0/3524946200 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+257 (crc 0 0 0) 0x7f49b805fa80 con 0x56249d50aa70 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49b0c00640 10 client.8235 handle_command_reply: tid=0 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 client.8235 shutdown 2024-09-13T17:15:39.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 msgr2=0x56249d472020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009790 tx=0x7f49c0004ab0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 shutdown_connections 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x56249d50aa70 0x56249d4957f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56249d47a110 0x56249d492fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56249d479c70 0x56249d4f34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49b0c00640 0 client.8235 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 --2- 172.21.15.28:0/3524946200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56249d452c60 0x56249d472020 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.737+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 >> 172.21.15.28:0/3524946200 conn(0x56249d450420 msgr2=0x56249d479200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 shutdown_connections 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49d0e09740 1 -- 172.21.15.28:0/3524946200 wait complete. 2024-09-13T17:15:39.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49d0e09740 20 client.8235 trim_cache size 0 max 16384 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "8699b72d-3835-49ca-944d-885298e77d0d": { 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "~mds0", 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "8699b72d-3835-49ca-944d-885298e77d0d", 2024-09-13T17:15:39.745 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:15:39.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.746 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:15:39.746 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 msgr2=0x7f49a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 0x7f49a806d0c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f49c0034000 tx=0x7f49c002f000 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 msgr2=0x7f49c8157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f49b800c500 tx=0x7f49b800c9c0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 shutdown_connections 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f49a806ac00 0x7f49a806d0c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49c80cd830 0x7f49c80ae730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f49c806e6a0 0x7f49c8157a20 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 --2- 172.21.15.28:0/1668079110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f49c806daf0 0x7f49c80b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.738+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 >> 172.21.15.28:0/1668079110 conn(0x7f49c80999e0 msgr2=0x7f49c80b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.739+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 shutdown_connections 2024-09-13T17:15:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:15:39.739+0000 7f49ce000640 1 -- 172.21.15.28:0/1668079110 wait complete. 2024-09-13T17:15:39.754 INFO:tasks.cephfs.filesystem:scrub status for tag:8699b72d-3835-49ca-944d-885298e77d0d - {'path': '~mds0', 'tag': '8699b72d-3835-49ca-944d-885298e77d0d', 'options': 'recursive'} 2024-09-13T17:15:40.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:15:40.267 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:16:09.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T17:16:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.926+0000 7f3927200640 1 Processor -- start 2024-09-13T17:16:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.926+0000 7f3927200640 1 -- start start 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f3927200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f392015c9a0 0x7f39200ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f3927200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f392006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f3927200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3920070470 con 0x7f39200ad940 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f3927200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39200ace30 con 0x7f392015c9a0 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f3927200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39200ad110 con 0x7f39200b2400 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f391f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f39200ad940 0x7f39200b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42422/0 (socket says 172.21.15.28:42422) 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.927+0000 7f391f400640 1 -- 172.21.15.28:0/2402115042 learned_addr learned my addr 172.21.15.28:0/2402115042 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f3925200640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f392006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391fe00640 1 --2- 172.21.15.28:0/2402115042 >> v2:172.21.15.189:3300/0 conn(0x7f392015c9a0 0x7f39200ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815813396 0 0) 0x7f3920070470 con 0x7f39200ad940 2024-09-13T17:16:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3904003520 con 0x7f39200ad940 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2304313578 0 0) 0x7f3904003520 con 0x7f39200ad940 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 msgr2=0x7f392006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f392006bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 >> v2:172.21.15.189:3300/0 conn(0x7f392015c9a0 msgr2=0x7f39200ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 --2- 172.21.15.28:0/2402115042 >> v2:172.21.15.189:3300/0 conn(0x7f392015c9a0 0x7f39200ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392006b170 con 0x7f39200ad940 2024-09-13T17:16:09.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f390c001690 con 0x7f39200ad940 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391fe00640 1 --2- 172.21.15.28:0/2402115042 >> v2:172.21.15.189:3300/0 conn(0x7f392015c9a0 0x7f39200ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 >> v1:172.21.15.28:6789/0 conn(0x7f39200ad940 legacy=0x7f39200b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3904003970 0x7f3904003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3904004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.928+0000 7f391ea00640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f3904005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39200ace30 con 0x7f39040047c0 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39200ad110 con 0x7f3904003970 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390400b9b0 con 0x7f39200b2400 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3904004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391fe00640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3904003970 0x7f3904003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f3925200640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f3904005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 -- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 msgr2=0x7f3904005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200b2400 0x7f3904005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 -- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3904003970 msgr2=0x7f3904003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3904003970 0x7f3904003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.929+0000 7f391f400640 1 -- 172.21.15.28:0/2402115042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904003520 con 0x7f39040047c0 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.930+0000 7f391f400640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3904004c40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f39200ad420 tx=0x7f39040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af0c7a9ffc81f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.930+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c005ad0 con 0x7f39040047c0 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.930+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f390c02d070 con 0x7f39040047c0 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.930+0000 7f391ea00640 1 -- 172.21.15.28:0/2402115042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390c005e50 con 0x7f39040047c0 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 -- 172.21.15.28:0/2402115042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 msgr2=0x7f3904004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 --2- 172.21.15.28:0/2402115042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3904004c40 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f39200ad420 tx=0x7f39040057f0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3925200640 1 -- 172.21.15.28:0/2402115042 reap_dead start 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 -- 172.21.15.28:0/2402115042 shutdown_connections 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 -- 172.21.15.28:0/2402115042 >> 172.21.15.28:0/2402115042 conn(0x7f39200b0270 msgr2=0x7f39200b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 -- 172.21.15.28:0/2402115042 shutdown_connections 2024-09-13T17:16:09.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.931+0000 7f3927200640 1 -- 172.21.15.28:0/2402115042 wait complete. 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.933+0000 7f3927200640 1 Processor -- start 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.933+0000 7f3927200640 1 -- start start 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 0x7f39201f2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 0x7f39201f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39201548a0 con 0x7f39040047c0 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3920154b80 con 0x7f39200c34e0 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3927200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3920154e60 con 0x7f39201f2bc0 2024-09-13T17:16:09.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39236/0 (socket says 172.21.15.28:39236) 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 -- 172.21.15.28:0/4232064330 learned_addr learned my addr 172.21.15.28:0/4232064330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391f400640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 0x7f39201f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3925200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 0x7f39201f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 -- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 msgr2=0x7f39201f3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 0x7f39201f3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 -- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 msgr2=0x7f39201f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 0x7f39201f2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391fe00640 1 -- 172.21.15.28:0/4232064330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920070930 con 0x7f39040047c0 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f391f400640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 0x7f39201f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.934+0000 7f3925200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 0x7f39201f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f391fe00640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f3910011cd0 tx=0x7f391000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f3917e00640 1 -- 172.21.15.28:0/4232064330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391000fd20 con 0x7f39040047c0 2024-09-13T17:16:09.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f3917e00640 1 -- 172.21.15.28:0/4232064330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391000fe80 con 0x7f39040047c0 2024-09-13T17:16:09.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f3917e00640 1 -- 172.21.15.28:0/4232064330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910004270 con 0x7f39040047c0 2024-09-13T17:16:09.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3920155140 con 0x7f39040047c0 2024-09-13T17:16:09.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.935+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3920155700 con 0x7f39040047c0 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.937+0000 7f3917e00640 1 -- 172.21.15.28:0/4232064330 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f39100043d0 con 0x7f39040047c0 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.938+0000 7f3917e00640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.938+0000 7f3917e00640 1 -- 172.21.15.28:0/4232064330 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3910089b50 con 0x7f39040047c0 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.939+0000 7f391f400640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.939+0000 7f392a067740 1 Processor -- start 2024-09-13T17:16:09.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.939+0000 7f392a067740 1 -- start start 2024-09-13T17:16:09.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.939+0000 7f391f400640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 0x7f390006d0f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f390c032000 tx=0x7f390c02d000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b40464040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b40461420 0x556b404b1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b4036a6a0 con 0x556b4048b110 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b403cdcc0 con 0x556b40463c60 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b403cb450 con 0x556b40461420 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39240/0 (socket says 172.21.15.28:39240) 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 -- 172.21.15.28:0/479302481 learned_addr learned my addr 172.21.15.28:0/479302481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f391f400640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b40461420 0x556b404b1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f391fe00640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b40464040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 -- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b40461420 msgr2=0x556b404b1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b40461420 0x556b404b1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 -- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 msgr2=0x556b40464040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b40464040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f3925200640 1 -- 172.21.15.28:0/479302481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b403972e0 con 0x556b4048b110 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.940+0000 7f391fe00640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b40464040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f3925200640 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f3920073b40 tx=0x7f3918009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8123f73cbc7c5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f3914c00640 1 -- 172.21.15.28:0/479302481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391800e040 con 0x556b4048b110 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f3914c00640 1 -- 172.21.15.28:0/479302481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391800f040 con 0x556b4048b110 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f3914c00640 1 -- 172.21.15.28:0/479302481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918003dd0 con 0x556b4048b110 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f392a067740 1 -- 172.21.15.28:0/479302481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 msgr2=0x556b404b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.941+0000 7f392a067740 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f3920073b40 tx=0x7f3918009750 comp rx=0 tx=0).stop 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 -- 172.21.15.28:0/479302481 shutdown_connections 2024-09-13T17:16:09.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048b110 0x556b404b1670 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b40461420 0x556b404b1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 --2- 172.21.15.28:0/479302481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b40464040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 -- 172.21.15.28:0/479302481 >> 172.21.15.28:0/479302481 conn(0x556b40460390 msgr2=0x556b404a8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 -- 172.21.15.28:0/479302481 shutdown_connections 2024-09-13T17:16:09.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 -- 172.21.15.28:0/479302481 wait complete. 2024-09-13T17:16:09.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.942+0000 7f392a067740 1 -- start start 2024-09-13T17:16:09.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.943+0000 7f392a067740 1 Processor -- start 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.943+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b404b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.943+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f392a067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f392a067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b403c6d40 con 0x556b4048ecd0 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b40368280 con 0x556b40463c60 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b40367f20 con 0x556b4048b110 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f392a067740 10 client.0 _ll_register_callbacks cb 0x556b403c03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39244/0 (socket says 172.21.15.28:39244) 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 -- 172.21.15.28:0/61913190 learned_addr learned my addr 172.21.15.28:0/61913190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f391fe00640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b404b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f391f400640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 -- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 msgr2=0x556b404ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 -- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 msgr2=0x556b404b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b404b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.944+0000 7f3925200640 1 -- 172.21.15.28:0/61913190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3910009400 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3925200640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f392006f3a0 tx=0x7f3918004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bf9727c9bfbbc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3909600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391800e040 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b400ba4e0 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3918004440 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3918004600 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f392a067740 10 client.8247 fetch_fsmap learned FSMap version 106 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x556b403c53c0 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.945+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556b400ba4e0 con 0x556b4048ecd0 2024-09-13T17:16:09.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3918014040 con 0x556b4048ecd0 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 10 client.8247 fetch_fsmap finished waiting for FSMap version 106 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 10 client.8247 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 20 client.8247 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b4048c290 0x556b4048c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 4 client.8247 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b400cb7e0 con 0x556b4048c290 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f391fe00640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b4048c290 0x556b4048c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f391fe00640 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b4048c290 0x556b4048c6a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26b0e50fb48ee54 server_cookie=b5f641d91ce3db90 in_seq=0 out_seq=0 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.946+0000 7f3909600640 10 client.8247 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f3909600640 1 -- 172.21.15.28:0/61913190 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x556b400cb7e0 con 0x556b4048c290 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f3909600640 10 client.8247 handle_command_reply: tid=0 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 client.8247 shutdown 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 msgr2=0x556b404ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f392006f3a0 tx=0x7f3918004a40 comp rx=0 tx=0).stop 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 shutdown_connections 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b4048c290 0x556b4048c6a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b4048ecd0 0x556b404ae620 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 --2- 172.21.15.28:0/61913190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b40463c60 0x556b404b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f3909600640 0 client.8247 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.947+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 >> 172.21.15.28:0/61913190 conn(0x556b40461420 msgr2=0x556b404a9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.948+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 shutdown_connections 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.948+0000 7f392a067740 1 -- 172.21.15.28:0/61913190 wait complete. 2024-09-13T17:16:09.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.948+0000 7f392a067740 20 client.8247 trim_cache size 0 max 16384 2024-09-13T17:16:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 Processor -- start 2024-09-13T17:16:09.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 -- start start 2024-09-13T17:16:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404aa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 0x556b404a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b4012e040 con 0x556b40463c60 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b404aa8b0 con 0x556b4048ecd0 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.963+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b40100b10 con 0x556b4048b110 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39256/0 (socket says 172.21.15.28:39256) 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 -- 172.21.15.28:0/118547800 learned_addr learned my addr 172.21.15.28:0/118547800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 -- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 msgr2=0x556b404aa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f3925200640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 0x556b404a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391f400640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404aa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404aa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 -- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 msgr2=0x556b404a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 0x556b404a8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 -- 172.21.15.28:0/118547800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b40367610 con 0x556b40463c60 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391f400640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f3925200640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 0x556b404a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.964+0000 7f391fe00640 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3910088000 tx=0x7f391001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a30faa88276faa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f3915600640 1 -- 172.21.15.28:0/118547800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910060dc0 con 0x556b40463c60 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f3915600640 1 -- 172.21.15.28:0/118547800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3910070070 con 0x556b40463c60 2024-09-13T17:16:09.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f3915600640 1 -- 172.21.15.28:0/118547800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910056ce0 con 0x556b40463c60 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 -- 172.21.15.28:0/118547800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 msgr2=0x556b4048c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3910088000 tx=0x7f391001f040 comp rx=0 tx=0).stop 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 -- 172.21.15.28:0/118547800 shutdown_connections 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048ecd0 0x556b404a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048b110 0x556b404aa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 --2- 172.21.15.28:0/118547800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b4048c670 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 -- 172.21.15.28:0/118547800 >> 172.21.15.28:0/118547800 conn(0x556b40461420 msgr2=0x556b404a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 -- 172.21.15.28:0/118547800 shutdown_connections 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.965+0000 7f392a067740 1 -- 172.21.15.28:0/118547800 wait complete. 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.966+0000 7f392a067740 1 -- start start 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.966+0000 7f392a067740 1 Processor -- start 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 0x556b404c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048ecd0 0x556b404ac430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b404880c0 con 0x556b40463c60 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b40487e50 con 0x556b4048b110 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556b404a8130 con 0x556b4048ecd0 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f392a067740 10 client.0 _ll_register_callbacks cb 0x7f39040036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:09.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f391fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39262/0 (socket says 172.21.15.28:39262) 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.967+0000 7f391fe00640 1 -- 172.21.15.28:0/3991972476 learned_addr learned my addr 172.21.15.28:0/3991972476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 -- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048ecd0 msgr2=0x556b404ac430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048ecd0 0x556b404ac430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 -- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 msgr2=0x556b404c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391f400640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 0x556b404c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 0x556b404c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 -- 172.21.15.28:0/3991972476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3920155700 con 0x556b40463c60 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391f400640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 0x556b404c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f391fe00640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f39100588d0 tx=0x7f3910011fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c6eaac579680b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f3909600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3910058710 con 0x556b40463c60 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3910055050 con 0x556b40463c60 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f391800f040 con 0x556b40463c60 2024-09-13T17:16:09.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3910055210 con 0x556b40463c60 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f392a067740 10 client.8253 fetch_fsmap learned FSMap version 106 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.968+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x556b4038cb20 con 0x556b40463c60 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391007b020 con 0x556b40463c60 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f39100701e0 con 0x556b40463c60 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 10 client.8253 fetch_fsmap finished waiting for FSMap version 106 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 10 client.8253 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 20 client.8253 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b404ac970 0x556b404a6930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 4 client.8253 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556b4011ee30 con 0x556b404ac970 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.969+0000 7f391f400640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b404ac970 0x556b404a6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f391f400640 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b404ac970 0x556b404a6930 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d9dc160898f4b8b server_cookie=70771d81f689c92f in_seq=0 out_seq=0 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f3909600640 10 client.8253 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:09.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f3909600640 1 -- 172.21.15.28:0/3991972476 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556b4011ee30 con 0x556b404ac970 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f3909600640 10 client.8253 handle_command_reply: tid=0 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 client.8253 shutdown 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 msgr2=0x556b405121f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f39100588d0 tx=0x7f3910011fd0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 shutdown_connections 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x556b404ac970 0x556b404a6930 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556b4048ecd0 0x556b404ac430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556b4048b110 0x556b404c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 --2- 172.21.15.28:0/3991972476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556b40463c60 0x556b405121f0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f3909600640 0 client.8253 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 >> 172.21.15.28:0/3991972476 conn(0x556b40461420 msgr2=0x556b404898f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 shutdown_connections 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 1 -- 172.21.15.28:0/3991972476 wait complete. 2024-09-13T17:16:09.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.970+0000 7f392a067740 20 client.8253 trim_cache size 0 max 16384 2024-09-13T17:16:09.976 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:09.976 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:16:09.976 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:16:09.976 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 msgr2=0x7f390006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 0x7f390006d0f0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f390c032000 tx=0x7f390c02d000 comp rx=0 tx=0).stop 2024-09-13T17:16:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 msgr2=0x7f3920149fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:09.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f3910011cd0 tx=0x7f391000f610 comp rx=0 tx=0).stop 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 shutdown_connections 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39201f2bc0 0x7f39201f3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200c34e0 0x7f39201f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 --2- 172.21.15.28:0/4232064330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39040047c0 0x7f3920149fa0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 >> 172.21.15.28:0/4232064330 conn(0x7f39200b0270 msgr2=0x7f392006bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 shutdown_connections 2024-09-13T17:16:09.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:09.971+0000 7f3927200640 1 -- 172.21.15.28:0/4232064330 wait complete. 2024-09-13T17:16:09.987 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:16:09.988 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 perf dump 2024-09-13T17:16:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.145+0000 7eff03c00640 1 Processor -- start 2024-09-13T17:16:10.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.145+0000 7eff03c00640 1 -- start start 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.146+0000 7eff03c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0b2400 0x7efefc0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.146+0000 7eff03c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efefc0ad940 0x7efefc06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff03c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc070470 con 0x7efefc15c9a0 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff03c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc0ace30 con 0x7efefc0ad940 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff03c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0ad110 con 0x7efefc0b2400 2024-09-13T17:16:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efefc0ad940 0x7efefc06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efefc0ad940 0x7efefc06bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39270/0 (socket says 172.21.15.28:39270) 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 -- 172.21.15.28:0/847502447 learned_addr learned my addr 172.21.15.28:0/847502447 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7efefbe00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0b2400 0x7efefc0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 -- 172.21.15.28:0/847502447 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0b2400 msgr2=0x7efefc0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0b2400 0x7efefc0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 -- 172.21.15.28:0/847502447 >> v1:172.21.15.189:6789/0 conn(0x7efefc15c9a0 legacy=0x7efefc0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.147+0000 7eff01c00640 1 -- 172.21.15.28:0/847502447 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc06b170 con 0x7efefc0ad940 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.148+0000 7eff01c00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc0ad940 0x7efefc06bc50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e50 tx=0x7efef402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1764480813d943f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7efefb400640 1 -- 172.21.15.28:0/847502447 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031600 con 0x7efefc0ad940 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7efefb400640 1 -- 172.21.15.28:0/847502447 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef402d050 con 0x7efefc0ad940 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7efefb400640 1 -- 172.21.15.28:0/847502447 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4031980 con 0x7efefc0ad940 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 -- 172.21.15.28:0/847502447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc0ad940 msgr2=0x7efefc06bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc0ad940 0x7efefc06bc50 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e50 tx=0x7efef402a7a0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 -- 172.21.15.28:0/847502447 shutdown_connections 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc0ad940 0x7efefc06bc50 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 --2- 172.21.15.28:0/847502447 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0b2400 0x7efefc0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 -- 172.21.15.28:0/847502447 >> 172.21.15.28:0/847502447 conn(0x7efefc0b0270 msgr2=0x7efefc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 -- 172.21.15.28:0/847502447 shutdown_connections 2024-09-13T17:16:10.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.149+0000 7eff03c00640 1 -- 172.21.15.28:0/847502447 wait complete. 2024-09-13T17:16:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.151+0000 7eff03c00640 1 Processor -- start 2024-09-13T17:16:10.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.151+0000 7eff03c00640 1 -- start start 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 0x7efefc1f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efefc0b2400 0x7efefc0c3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc15e410 con 0x7efefc15c9a0 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc15e6d0 con 0x7efefc0b2400 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff03c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc15e9b0 con 0x7efefc0ad940 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39276/0 (socket says 172.21.15.28:39276) 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff01c00640 1 -- 172.21.15.28:0/2705181094 learned_addr learned my addr 172.21.15.28:0/2705181094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7eff01200640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 0x7efefc1f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.152+0000 7efefbe00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efefc0b2400 0x7efefc0c3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 -- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 msgr2=0x7efefc1f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 0x7efefc1f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 -- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efefc0b2400 msgr2=0x7efefc0c3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efefc0b2400 0x7efefc0c3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 -- 172.21.15.28:0/2705181094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef4009b30 con 0x7efefc15c9a0 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01200640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 0x7efefc1f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff01c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e50 tx=0x7efef4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7efef8c00640 1 -- 172.21.15.28:0/2705181094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef403d860 con 0x7efefc15c9a0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7efef8c00640 1 -- 172.21.15.28:0/2705181094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef403d9e0 con 0x7efefc15c9a0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7efef8c00640 1 -- 172.21.15.28:0/2705181094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef403dd90 con 0x7efefc15c9a0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.153+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc070930 con 0x7efefc15c9a0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.154+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc15ef70 con 0x7efefc15c9a0 2024-09-13T17:16:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.156+0000 7efef8c00640 1 -- 172.21.15.28:0/2705181094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efef403b400 con 0x7efefc15c9a0 2024-09-13T17:16:10.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.156+0000 7efef8c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 0x7efedc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.157+0000 7efef8c00640 1 -- 172.21.15.28:0/2705181094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efef40aa150 con 0x7efefc15c9a0 2024-09-13T17:16:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.157+0000 7eff01200640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 0x7efedc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.157+0000 7eff06a99740 1 Processor -- start 2024-09-13T17:16:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.157+0000 7eff06a99740 1 -- start start 2024-09-13T17:16:10.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff01200640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 0x7efedc06d0c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efef0000f60 tx=0x7efef0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb4045760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 0x55feb4092850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3f50000 con 0x55feb406c830 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3fadd40 con 0x55feb4045380 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb3fab4d0 con 0x55feb4042b40 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.158+0000 7eff01200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb4045760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7efefbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 0x55feb4092850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb4045760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:39412/0 (socket says 172.21.15.28:39412) 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01200640 1 -- 172.21.15.28:0/2283634562 learned_addr learned my addr 172.21.15.28:0/2283634562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39286/0 (socket says 172.21.15.28:39286) 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 -- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 msgr2=0x55feb4092850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 0x55feb4092850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 -- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 msgr2=0x55feb4045760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb4045760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 -- 172.21.15.28:0/2283634562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feb3f623a0 con 0x55feb406c830 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7efefbe00640 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 0x55feb4092850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:10.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7eff01c00640 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7efef40754e0 tx=0x7efef4078a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6e8b867f5d99f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7efeed600640 1 -- 172.21.15.28:0/2283634562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef407cd60 con 0x55feb406c830 2024-09-13T17:16:10.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7efeed600640 1 -- 172.21.15.28:0/2283634562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4081320 con 0x55feb406c830 2024-09-13T17:16:10.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.159+0000 7efeed600640 1 -- 172.21.15.28:0/2283634562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef40756f0 con 0x55feb406c830 2024-09-13T17:16:10.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 -- 172.21.15.28:0/2283634562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 msgr2=0x55feb4092d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7efef40754e0 tx=0x7efef4078a00 comp rx=0 tx=0).stop 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 -- 172.21.15.28:0/2283634562 shutdown_connections 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb406c830 0x55feb4092d90 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb4042b40 0x55feb4092850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 --2- 172.21.15.28:0/2283634562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb4045760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 -- 172.21.15.28:0/2283634562 >> 172.21.15.28:0/2283634562 conn(0x55feb4041ab0 msgr2=0x55feb4089f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 -- 172.21.15.28:0/2283634562 shutdown_connections 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.160+0000 7eff06a99740 1 -- 172.21.15.28:0/2283634562 wait complete. 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.161+0000 7eff06a99740 1 -- start start 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.161+0000 7eff06a99740 1 Processor -- start 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb4069b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb406a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3fa6dc0 con 0x55feb4045380 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3f653e0 con 0x55feb4092100 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb3f65080 con 0x55feb406c830 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff06a99740 10 client.0 _ll_register_callbacks cb 0x55feb3fa0430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39302/0 (socket says 172.21.15.28:39302) 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.162+0000 7eff01200640 1 -- 172.21.15.28:0/2137281955 learned_addr learned my addr 172.21.15.28:0/2137281955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01c00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb406a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 -- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 msgr2=0x55feb4069b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7efefbe00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb4069b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb4069b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 -- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 msgr2=0x55feb406a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb406a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 -- 172.21.15.28:0/2137281955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc15ef70 con 0x55feb4045380 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01c00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb406a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7efefbe00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb4069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff01200640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7efef00149d0 tx=0x7efef0001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5f74eb0c6d7247 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7efee2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef0013ac0 con 0x55feb4045380 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efef001c040 con 0x55feb4045380 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.163+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55feb3fd0600 con 0x55feb4045380 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef0013c20 con 0x55feb4045380 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 10 client.8265 fetch_fsmap learned FSMap version 106 2024-09-13T17:16:10.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55feb3fa5440 con 0x55feb4045380 2024-09-13T17:16:10.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55feb3fd0600 con 0x55feb4045380 2024-09-13T17:16:10.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efef0012070 con 0x55feb4045380 2024-09-13T17:16:10.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 10 client.8265 fetch_fsmap finished waiting for FSMap version 106 2024-09-13T17:16:10.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 10 client.8265 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 20 client.8265 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406ab70 0x55feb406af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 4 client.8265 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55feb3f98c90 con 0x55feb406ab70 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.164+0000 7efefbe00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406ab70 0x55feb406af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.165+0000 7efefbe00640 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406ab70 0x55feb406af80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e728d944390058e7 server_cookie=bb7f4399d2642608 in_seq=0 out_seq=0 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.165+0000 7efee2000640 10 client.8265 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.165+0000 7efee2000640 1 -- 172.21.15.28:0/2137281955 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55feb3f98c90 con 0x55feb406ab70 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.165+0000 7efee2000640 10 client.8265 handle_command_reply: tid=0 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 client.8265 shutdown 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 msgr2=0x55feb4091bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7efef00149d0 tx=0x7efef0001080 comp rx=0 tx=0).stop 2024-09-13T17:16:10.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 shutdown_connections 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406ab70 0x55feb406af80 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb406a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb4069b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 --2- 172.21.15.28:0/2137281955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb4091bc0 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7efee2000640 0 client.8265 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 >> 172.21.15.28:0/2137281955 conn(0x55feb4042b40 msgr2=0x55feb408a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 shutdown_connections 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 1 -- 172.21.15.28:0/2137281955 wait complete. 2024-09-13T17:16:10.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.166+0000 7eff06a99740 20 client.8265 trim_cache size 0 max 16384 2024-09-13T17:16:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.180+0000 7eff06a99740 1 Processor -- start 2024-09-13T17:16:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.180+0000 7eff06a99740 1 -- start start 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb408c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb40704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3d0e040 con 0x55feb4092100 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb3f47440 con 0x55feb4045380 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb408bee0 con 0x55feb406c830 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff01c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39306/0 (socket says 172.21.15.28:39306) 2024-09-13T17:16:10.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff01c00640 1 -- 172.21.15.28:0/1922697258 learned_addr learned my addr 172.21.15.28:0/1922697258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7eff01200640 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb408c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.181+0000 7efefbe00640 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb40704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 -- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 msgr2=0x55feb40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb40704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 -- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 msgr2=0x55feb408c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb408c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 -- 172.21.15.28:0/1922697258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55feb3f475d0 con 0x55feb4092100 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7eff01c00640 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7efef4080df0 tx=0x7efef407f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c882beba5d67aded server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7efeee000640 1 -- 172.21.15.28:0/1922697258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef407cd60 con 0x55feb4092100 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7efeee000640 1 -- 172.21.15.28:0/1922697258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4090070 con 0x55feb4092100 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.182+0000 7efeee000640 1 -- 172.21.15.28:0/1922697258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef40a82a0 con 0x55feb4092100 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- 172.21.15.28:0/1922697258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 msgr2=0x55feb40a2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7efef4080df0 tx=0x7efef407f000 comp rx=0 tx=0).stop 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- 172.21.15.28:0/1922697258 shutdown_connections 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4092100 0x55feb40a2cb0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb40704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 --2- 172.21.15.28:0/1922697258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4045380 0x55feb408c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- 172.21.15.28:0/1922697258 >> 172.21.15.28:0/1922697258 conn(0x55feb4042b40 msgr2=0x55feb406ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- 172.21.15.28:0/1922697258 shutdown_connections 2024-09-13T17:16:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- 172.21.15.28:0/1922697258 wait complete. 2024-09-13T17:16:10.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.183+0000 7eff06a99740 1 -- start start 2024-09-13T17:16:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.184+0000 7eff06a99740 1 Processor -- start 2024-09-13T17:16:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb408d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb408d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb40e8190 con 0x55feb4045380 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55feb40891f0 con 0x55feb4092100 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55feb40a36a0 con 0x55feb406c830 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff06a99740 10 client.0 _ll_register_callbacks cb 0x7efef0001890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39312/0 (socket says 172.21.15.28:39312) 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 -- 172.21.15.28:0/1151314133 learned_addr learned my addr 172.21.15.28:0/1151314133 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7efefbe00640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb408d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01c00640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb408d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 -- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 msgr2=0x55feb408d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb408d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 -- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 msgr2=0x55feb408d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb408d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.185+0000 7eff01200640 1 -- 172.21.15.28:0/1151314133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55feb3f623a0 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7eff01200640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7efef0021cf0 tx=0x7efef0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ecff11350b3834 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef003d630 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efef003d790 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efef4081320 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=106) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef003d950 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7eff06a99740 10 client.8271 fetch_fsmap learned FSMap version 106 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=106,monmap=2+}) v3 -- 0x55feb3f6c9b0 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4081320 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.186+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 106) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efef003dab0 con 0x55feb4045380 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 10 client.8271 fetch_fsmap finished waiting for FSMap version 106 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 10 client.8271 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T17:16:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 20 client.8271 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406f040 0x55feb4043a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 4 client.8271 mds_command: new command op to 8022 tid=0 multi_id=0 [{"prefix": "perf dump"}] 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 --> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] -- command(tid 0: {"prefix": "perf dump"}) v1 -- 0x55feb3c9a4e0 con 0x55feb406f040 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7efefbe00640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406f040 0x55feb4043a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7efefbe00640 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406f040 0x55feb4043a60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6054af372e2575f4 server_cookie=53ecf8c685ced990 in_seq=0 out_seq=0 2024-09-13T17:16:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.187+0000 7efee2000640 10 client.8271 ms_handle_connect on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7efee2000640 1 -- 172.21.15.28:0/1151314133 <== mds.0 v2:172.21.15.28:6832/4107617548 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17933 (crc 0 0 0) 0x55feb3c9a4e0 con 0x55feb406f040 2024-09-13T17:16:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7efee2000640 10 client.8271 handle_command_reply: tid=0 2024-09-13T17:16:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 client.8271 shutdown 2024-09-13T17:16:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 msgr2=0x55feb40e76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7efef0021cf0 tx=0x7efef0015000 comp rx=0 tx=0).stop 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 shutdown_connections 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:6832/4107617548,v1:172.21.15.28:6835/4107617548] conn(0x55feb406f040 0x55feb4043a60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feb4092100 0x55feb408d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55feb406c830 0x55feb408d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 --2- 172.21.15.28:0/1151314133 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55feb4045380 0x55feb40e76b0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7efee2000640 0 client.8271 ms_handle_reset on v2:172.21.15.28:6832/4107617548 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 >> 172.21.15.28:0/1151314133 conn(0x55feb4042b40 msgr2=0x55feb40a2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 shutdown_connections 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.188+0000 7eff06a99740 1 -- 172.21.15.28:0/1151314133 wait complete. 2024-09-13T17:16:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff06a99740 20 client.8271 trim_cache size 0 max 16384 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4920, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4921, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1427612, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 41458392, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 17, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 6, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.643074471, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.321555756, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.108679274, 2024-09-13T17:16:10.194 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.167812485, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4920, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 5.490901079, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001116036 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 10, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000001012, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000101 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-09-13T17:16:10.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 30310, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 51159, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 10926872, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 41966609, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 7, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 4.975998762, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 2.906511356, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 2.522230290, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.080739209, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 51158, 2024-09-13T17:16:10.196 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.043822232, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000039951 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 71271, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.017444087, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000244 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 132912, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 26480 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6808, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 6829, 2024-09-13T17:16:10.197 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2362417, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 56271142, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 6, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 4, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.814733769, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.407065348, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.151710799, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.212517028, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6828, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 2.413955730, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000353537 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-09-13T17:16:10.198 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 121584 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:16:10.199 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6205, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.825169991, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002711550 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3715, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.152578648, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000310249 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.200 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 25948, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 25948, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 25948, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 254.608344355, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.009812253 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 0, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 61, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 0, 2024-09-13T17:16:10.201 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 15037, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 10179, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 4858, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 435, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 2, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 2, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 2, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 48768, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 48166, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-09-13T17:16:10.202 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 161, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 2011, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 0, 2024-09-13T17:16:10.203 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 2011, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 11, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 1, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 10, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 2000, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 1198, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 4034, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-09-13T17:16:10.204 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 4260, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 2000, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 21, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 14092 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-09-13T17:16:10.205 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 4128, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 3942, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 2233, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 1, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-09-13T17:16:10.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 32188, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 31723, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 31723, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 1492, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 1027, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 49, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 48, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 48, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 3, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 2, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 102819991, 2024-09-13T17:16:10.207 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 103100545, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4278, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 76.303202486, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.017836185 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 14, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 2015, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 2001, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 13, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 13, 2024-09-13T17:16:10.208 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 0, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 15037, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 20013, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 4976, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 2, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 2770, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 2768, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 95356, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 207200 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 29763, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 0, 2024-09-13T17:16:10.209 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 25948, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 6, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 0, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2000, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.336601444, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000668300 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-09-13T17:16:10.210 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 29, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006855157, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000236384 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 6000, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.886349622, 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001147724 2024-09-13T17:16:10.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7673, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 6.645871337, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000866137 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.212 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:16:10.213 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000050139, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000050139 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.003285389, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003285389 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.214 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.006367092, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.006367092 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-09-13T17:16:10.215 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.571651706, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.071456463 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2233, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 222.311504972, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.099557324 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-09-13T17:16:10.216 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.034863823, 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.034863823 2024-09-13T17:16:10.217 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 1, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000251929, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000251929 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.218 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.219 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8000, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 16.804691745, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002100586 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-09-13T17:16:10.220 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 1, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 1, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 0, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 1, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 18299, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 18299, 2024-09-13T17:16:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 87, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-09-13T17:16:10.222 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-09-13T17:16:10.223 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-09-13T17:16:10.224 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 250696, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 92, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 440, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 5, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 5416, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 77, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-09-13T17:16:10.225 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 11367669, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 605442, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 13788, 2024-09-13T17:16:10.226 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 109831684, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 30, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 13788, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 13788, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 115.253771951, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.008358991 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 13788, 2024-09-13T17:16:10.227 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 20498 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 13788, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 2053, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 11735, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 48, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 3279, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 41, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 4318, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 49, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-09-13T17:16:10.228 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 1942, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 2058, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 2012, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 6558, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-09-13T17:16:10.229 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-09-13T17:16:10.230 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 113, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 3, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 8, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 8, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-09-13T17:16:10.231 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 143, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 50 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 0 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-09-13T17:16:10.232 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 32, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 32, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 3942, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 3942, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 42034, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 10928025, 2024-09-13T17:16:10.233 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 42034, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 42034, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 10928025, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-09-13T17:16:10.234 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 13788, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 110245362, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 13788, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 110245362, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.235 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 13788, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 13788, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 13788, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 13788, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.236 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 3942, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 413678, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 3942, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 40, 2024-09-13T17:16:10.237 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 413678, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x564e0163c0a0": { 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 32188, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 98906241, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-09-13T17:16:10.238 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 32188, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 4278, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 98906241, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:10.239 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 msgr2=0x7efedc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 0x7efedc06d0c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7efef0000f60 tx=0x7efef0009040 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 msgr2=0x7efefc0c42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7efef4009e50 tx=0x7efef4004a40 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 shutdown_connections 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efedc06ac00 0x7efedc06d0c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efefc15c9a0 0x7efefc0c42f0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efefc0b2400 0x7efefc0c3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 --2- 172.21.15.28:0/2705181094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efefc0ad940 0x7efefc1f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 >> 172.21.15.28:0/2705181094 conn(0x7efefc0b0270 msgr2=0x7efefc0af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 shutdown_connections 2024-09-13T17:16:10.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.189+0000 7eff03c00640 1 -- 172.21.15.28:0/2705181094 wait complete. 2024-09-13T17:16:10.242 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:16:10.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:16:10.331 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:16:10.357 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:16:10.361 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:16:10.429 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:16:10.429 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:10.429 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:10.464 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:10.465 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:10.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.735+0000 7f8e61200640 1 Processor -- start 2024-09-13T17:16:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.735+0000 7f8e61200640 1 -- start start 2024-09-13T17:16:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 0x7f8e5c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c06f980 0x7f8e5c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c159230 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e5c06fdf0 con 0x7f8e5c0b3160 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.736+0000 7f8e61200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0b3970 con 0x7f8e5c06f980 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39314/0 (socket says 172.21.15.28:39314) 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 -- 172.21.15.28:0/642106942 learned_addr learned my addr 172.21.15.28:0/642106942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59400640 1 --2- 172.21.15.28:0/642106942 >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 0x7f8e5c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e5a800640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c06f980 0x7f8e5c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 -- 172.21.15.28:0/642106942 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c06f980 msgr2=0x7f8e5c0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c06f980 0x7f8e5c0b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 -- 172.21.15.28:0/642106942 >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 msgr2=0x7f8e5c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 --2- 172.21.15.28:0/642106942 >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 0x7f8e5c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59e00640 1 -- 172.21.15.28:0/642106942 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c159430 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.737+0000 7f8e59400640 1 --2- 172.21.15.28:0/642106942 >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 0x7f8e5c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.738+0000 7f8e59e00640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06d220 tx=0x7f8e4c02c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da45426189008510 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.738+0000 7f8e58a00640 1 -- 172.21.15.28:0/642106942 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c032790 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.738+0000 7f8e58a00640 1 -- 172.21.15.28:0/642106942 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0328f0 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.738+0000 7f8e58a00640 1 -- 172.21.15.28:0/642106942 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c032c80 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 -- 172.21.15.28:0/642106942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c0cdfa0 msgr2=0x7f8e5c15d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c06d220 tx=0x7f8e4c02c8d0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 -- 172.21.15.28:0/642106942 shutdown_connections 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c06f980 0x7f8e5c0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 --2- 172.21.15.28:0/642106942 >> v2:172.21.15.189:3300/0 conn(0x7f8e5c0b3160 0x7f8e5c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 --2- 172.21.15.28:0/642106942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c15d340 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 -- 172.21.15.28:0/642106942 >> 172.21.15.28:0/642106942 conn(0x7f8e5c069e80 msgr2=0x7f8e5c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 -- 172.21.15.28:0/642106942 shutdown_connections 2024-09-13T17:16:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.739+0000 7f8e61200640 1 -- 172.21.15.28:0/642106942 wait complete. 2024-09-13T17:16:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.741+0000 7f8e61200640 1 Processor -- start 2024-09-13T17:16:10.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.741+0000 7f8e61200640 1 -- start start 2024-09-13T17:16:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c0b3160 0x7f8e5c155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c0b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c148590 con 0x7f8e5c06f980 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1557d0 con 0x7f8e5c0cdfa0 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e61200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c148b70 con 0x7f8e5c0b3160 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39316/0 (socket says 172.21.15.28:39316) 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 -- 172.21.15.28:0/893165395 learned_addr learned my addr 172.21.15.28:0/893165395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 -- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c0b3160 msgr2=0x7f8e5c155290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c0b3160 0x7f8e5c155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 -- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c0cdfa0 msgr2=0x7f8e5c0b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c0b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.742+0000 7f8e59e00640 1 -- 172.21.15.28:0/893165395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c00b770 con 0x7f8e5c06f980 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e59e00640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c000c00 tx=0x7f8e4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c03c830 con 0x7f8e5c06f980 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c03c9b0 con 0x7f8e5c06f980 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c03cd60 con 0x7f8e5c06f980 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c148e50 con 0x7f8e5c06f980 2024-09-13T17:16:10.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.743+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c149410 con 0x7f8e5c06f980 2024-09-13T17:16:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.744+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c06f440 con 0x7f8e5c06f980 2024-09-13T17:16:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.745+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e4c033420 con 0x7f8e5c06f980 2024-09-13T17:16:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.746+0000 7f8e52000640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 0x7f8e3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.746+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8e4c0aa0d0 con 0x7f8e5c06f980 2024-09-13T17:16:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.747+0000 7f8e59400640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 0x7f8e3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.747+0000 7f8e59400640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 0x7f8e3c06d0c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.751+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e4c0753a0 con 0x7f8e5c06f980 2024-09-13T17:16:10.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.930+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e5c0cf280 con 0x7f8e5c06f980 2024-09-13T17:16:10.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.932+0000 7f8e52000640 1 -- 172.21.15.28:0/893165395 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16586 (secure 0 0 0) 0x7f8e4c0789c0 con 0x7f8e5c06f980 2024-09-13T17:16:10.933 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:10.933 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":113,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:15:45.960852+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 msgr2=0x7f8e3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 0x7f8e3c06d0c0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8e48006fd0 tx=0x7f8e48008040 comp rx=0 tx=0).stop 2024-09-13T17:16:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 msgr2=0x7f8e5c148050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c000c00 tx=0x7f8e4c004730 comp rx=0 tx=0).stop 2024-09-13T17:16:10.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 shutdown_connections 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3c06ac00 0x7f8e3c06d0c0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e5c0cdfa0 0x7f8e5c0b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e5c0b3160 0x7f8e5c155290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 --2- 172.21.15.28:0/893165395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c06f980 0x7f8e5c148050 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 >> 172.21.15.28:0/893165395 conn(0x7f8e5c069e80 msgr2=0x7f8e5c0b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 shutdown_connections 2024-09-13T17:16:10.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:10.934+0000 7f8e61200640 1 -- 172.21.15.28:0/893165395 wait complete. 2024-09-13T17:16:10.951 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:14:29.435536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:14:30.524696+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:16:10.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:16:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.110+0000 7f9ed2800640 1 Processor -- start 2024-09-13T17:16:11.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.110+0000 7f9ed2800640 1 -- start start 2024-09-13T17:16:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed2800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9ecc143320 0x7f9ecc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9ecc0cd120 0x7f9ecc0cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ecc071630 con 0x7f9ecc143910 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed2800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc0cdad0 con 0x7f9ecc0cd120 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed2800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ecc143700 con 0x7f9ecc143320 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.111+0000 7f9ed0a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9ecc143910 0x7f9ecc071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42436/0 (socket says 172.21.15.28:42436) 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed0a00640 1 -- 172.21.15.28:0/657102157 learned_addr learned my addr 172.21.15.28:0/657102157 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1e00640 1 --2- 172.21.15.28:0/657102157 >> v2:172.21.15.28:3300/0 conn(0x7f9ecc0cd120 0x7f9ecc0cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1400640 1 --2- 172.21.15.28:0/657102157 >> v2:172.21.15.189:3300/0 conn(0x7f9ecc143320 0x7f9ecc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ecbe00640 1 -- 172.21.15.28:0/657102157 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091352811 0 0) 0x7f9ecc071630 con 0x7f9ecc143910 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ecbe00640 1 -- 172.21.15.28:0/657102157 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb4003520 con 0x7f9ecc143910 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1e00640 1 -- 172.21.15.28:0/657102157 >> v2:172.21.15.189:3300/0 conn(0x7f9ecc143320 msgr2=0x7f9ecc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1e00640 1 --2- 172.21.15.28:0/657102157 >> v2:172.21.15.189:3300/0 conn(0x7f9ecc143320 0x7f9ecc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1e00640 1 -- 172.21.15.28:0/657102157 >> v1:172.21.15.28:6789/0 conn(0x7f9ecc143910 legacy=0x7f9ecc071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.112+0000 7f9ed1e00640 1 -- 172.21.15.28:0/657102157 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc070a20 con 0x7f9ecc0cd120 2024-09-13T17:16:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.113+0000 7f9ed1e00640 1 --2- 172.21.15.28:0/657102157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc0cd590 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc06c690 tx=0x7f9ec402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65583d4573155ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ecbe00640 1 -- 172.21.15.28:0/657102157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec402aea0 con 0x7f9ecc0cd120 2024-09-13T17:16:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ecbe00640 1 -- 172.21.15.28:0/657102157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec402d050 con 0x7f9ecc0cd120 2024-09-13T17:16:11.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ecbe00640 1 -- 172.21.15.28:0/657102157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4007940 con 0x7f9ecc0cd120 2024-09-13T17:16:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 -- 172.21.15.28:0/657102157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 msgr2=0x7f9ecc0cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 --2- 172.21.15.28:0/657102157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc0cd590 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc06c690 tx=0x7f9ec402a680 comp rx=0 tx=0).stop 2024-09-13T17:16:11.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 -- 172.21.15.28:0/657102157 shutdown_connections 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 --2- 172.21.15.28:0/657102157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc0cd590 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 --2- 172.21.15.28:0/657102157 >> v2:172.21.15.189:3300/0 conn(0x7f9ecc143320 0x7f9ecc070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 -- 172.21.15.28:0/657102157 >> 172.21.15.28:0/657102157 conn(0x7f9ecc069e80 msgr2=0x7f9ecc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 -- 172.21.15.28:0/657102157 shutdown_connections 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.114+0000 7f9ed2800640 1 -- 172.21.15.28:0/657102157 wait complete. 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.116+0000 7f9ed2800640 1 Processor -- start 2024-09-13T17:16:11.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.116+0000 7f9ed2800640 1 -- start start 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ecc143320 0x7f9ecc1f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 0x7f9ecc0b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1f3280 con 0x7f9ecc0cd120 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1f3560 con 0x7f9ecc143910 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed2800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc1f3840 con 0x7f9ecc143320 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39344/0 (socket says 172.21.15.28:39344) 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.117+0000 7f9ed1400640 1 -- 172.21.15.28:0/3840173943 learned_addr learned my addr 172.21.15.28:0/3840173943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1e00640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 0x7f9ecc0b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed0a00640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ecc143320 0x7f9ecc1f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 -- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ecc143320 msgr2=0x7f9ecc1f29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ecc143320 0x7f9ecc1f29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 -- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 msgr2=0x7f9ecc0b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 0x7f9ecc0b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 -- 172.21.15.28:0/3840173943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec4009a70 con 0x7f9ecc0cd120 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1e00640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 0x7f9ecc0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed1400640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000e9b0 tx=0x7f9ec000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec0009670 con 0x7f9ecc0cd120 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec000c9a0 con 0x7f9ecc0cd120 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec000cd50 con 0x7f9ecc0cd120 2024-09-13T17:16:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.118+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc1f3b20 con 0x7f9ecc0cd120 2024-09-13T17:16:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.119+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc159b50 con 0x7f9ecc0cd120 2024-09-13T17:16:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.120+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc0b3800 con 0x7f9ecc0cd120 2024-09-13T17:16:11.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.120+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9ec0013470 con 0x7f9ecc0cd120 2024-09-13T17:16:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.122+0000 7f9ec9600640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 0x7f9eac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.122+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9ec0089660 con 0x7f9ecc0cd120 2024-09-13T17:16:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.122+0000 7f9ed0a00640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 0x7f9eac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.123+0000 7f9ed0a00640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 0x7f9eac06d0c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0b3760 tx=0x7f9eb8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.126+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9ec0054780 con 0x7f9ecc0cd120 2024-09-13T17:16:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.307+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ecc144010 con 0x7f9ecc0cd120 2024-09-13T17:16:11.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.308+0000 7f9ec9600640 1 -- 172.21.15.28:0/3840173943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ec0057da0 con 0x7f9ecc0cd120 2024-09-13T17:16:11.309 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:16:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 msgr2=0x7f9eac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 0x7f9eac06d0c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc0b3760 tx=0x7f9eb8009040 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 msgr2=0x7f9ecc159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f9ec000e9b0 tx=0x7f9ec000ee70 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 shutdown_connections 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9eac06ac00 0x7f9eac06d0c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9ecc143910 0x7f9ecc0b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9ecc143320 0x7f9ecc1f29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 --2- 172.21.15.28:0/3840173943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ecc0cd120 0x7f9ecc159610 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 >> 172.21.15.28:0/3840173943 conn(0x7f9ecc069e80 msgr2=0x7f9ecc06b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 shutdown_connections 2024-09-13T17:16:11.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.310+0000 7f9ed2800640 1 -- 172.21.15.28:0/3840173943 wait complete. 2024-09-13T17:16:11.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:16:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.484+0000 7fd893a00640 1 Processor -- start 2024-09-13T17:16:11.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.484+0000 7fd893a00640 1 -- start start 2024-09-13T17:16:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.485+0000 7fd893a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd88c15ab60 0x7fd88c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.485+0000 7fd893a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c071490 0x7fd88c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.485+0000 7fd893a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88c06c8b0 con 0x7fd88c15a780 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.485+0000 7fd893a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c06cb90 con 0x7fd88c071490 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.485+0000 7fd893a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c0cc980 con 0x7fd88c15ab60 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c071490 0x7fd88c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c071490 0x7fd88c06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39346/0 (socket says 172.21.15.28:39346) 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 -- 172.21.15.28:0/4142145839 learned_addr learned my addr 172.21.15.28:0/4142145839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd883e00640 1 --2- 172.21.15.28:0/4142145839 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15ab60 0x7fd88c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 -- 172.21.15.28:0/4142145839 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15ab60 msgr2=0x7fd88c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 --2- 172.21.15.28:0/4142145839 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15ab60 0x7fd88c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 -- 172.21.15.28:0/4142145839 >> v1:172.21.15.189:6789/0 conn(0x7fd88c15a780 legacy=0x7fd88c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.486+0000 7fd891a00640 1 -- 172.21.15.28:0/4142145839 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c0ccc60 con 0x7fd88c071490 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.487+0000 7fd891a00640 1 --2- 172.21.15.28:0/4142145839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c071490 0x7fd88c06c080 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fd88c06f380 tx=0x7fd88402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30fbdacd09c4dbcd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd88be00640 1 -- 172.21.15.28:0/4142145839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031600 con 0x7fd88c071490 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd88be00640 1 -- 172.21.15.28:0/4142145839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88402d050 con 0x7fd88c071490 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd88be00640 1 -- 172.21.15.28:0/4142145839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031980 con 0x7fd88c071490 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 -- 172.21.15.28:0/4142145839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c071490 msgr2=0x7fd88c06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 --2- 172.21.15.28:0/4142145839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c071490 0x7fd88c06c080 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fd88c06f380 tx=0x7fd88402a730 comp rx=0 tx=0).stop 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 -- 172.21.15.28:0/4142145839 shutdown_connections 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 --2- 172.21.15.28:0/4142145839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c071490 0x7fd88c06c080 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 --2- 172.21.15.28:0/4142145839 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15ab60 0x7fd88c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 -- 172.21.15.28:0/4142145839 >> 172.21.15.28:0/4142145839 conn(0x7fd88c0b0270 msgr2=0x7fd88c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 -- 172.21.15.28:0/4142145839 shutdown_connections 2024-09-13T17:16:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.488+0000 7fd893a00640 1 -- 172.21.15.28:0/4142145839 wait complete. 2024-09-13T17:16:11.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.490+0000 7fd893a00640 1 Processor -- start 2024-09-13T17:16:11.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.490+0000 7fd893a00640 1 -- start start 2024-09-13T17:16:11.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c071490 0x7fd88c14ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 0x7fd88c0af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c14cfa0 con 0x7fd88c15a780 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c0b2280 con 0x7fd88c15ab60 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd893a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c0b1670 con 0x7fd88c071490 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd883e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd883e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39358/0 (socket says 172.21.15.28:39358) 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd883e00640 1 -- 172.21.15.28:0/449114033 learned_addr learned my addr 172.21.15.28:0/449114033 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd891000640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c071490 0x7fd88c14ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.491+0000 7fd891a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 0x7fd88c0af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 -- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c071490 msgr2=0x7fd88c14ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c071490 0x7fd88c14ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 -- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 msgr2=0x7fd88c0af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 0x7fd88c0af250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 -- 172.21.15.28:0/449114033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd884009b30 con 0x7fd88c15a780 2024-09-13T17:16:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd891a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 0x7fd88c0af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd883e00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e870 tx=0x7fd87400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740094f0 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87400c940 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87400ccf0 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c0b1950 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.492+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c0b1f10 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.493+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c0b23f0 con 0x7fd88c15a780 2024-09-13T17:16:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.494+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd874013470 con 0x7fd88c15a780 2024-09-13T17:16:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.495+0000 7fd889600640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.495+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd874089410 con 0x7fd88c15a780 2024-09-13T17:16:11.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.496+0000 7fd891000640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.496+0000 7fd891000640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd88c153bc0 tx=0x7fd87c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.499+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd874054670 con 0x7fd88c15a780 2024-09-13T17:16:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.679+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd88c070d80 con 0x7fd88c15a780 2024-09-13T17:16:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.679+0000 7fd889600640 1 -- 172.21.15.28:0/449114033 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd874057c90 con 0x7fd88c15a780 2024-09-13T17:16:11.680 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 msgr2=0x7fd86806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fd88c153bc0 tx=0x7fd87c009040 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 msgr2=0x7fd88c157950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e870 tx=0x7fd87400ed30 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 shutdown_connections 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15ab60 0x7fd88c0af250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c15a780 0x7fd88c157950 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.681+0000 7fd893a00640 1 --2- 172.21.15.28:0/449114033 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c071490 0x7fd88c14ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.682+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 >> 172.21.15.28:0/449114033 conn(0x7fd88c0b0270 msgr2=0x7fd88c0cdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.682+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 shutdown_connections 2024-09-13T17:16:11.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.682+0000 7fd893a00640 1 -- 172.21.15.28:0/449114033 wait complete. 2024-09-13T17:16:11.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:16:11.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.855+0000 7fb000e00640 1 Processor -- start 2024-09-13T17:16:11.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.855+0000 7fb000e00640 1 -- start start 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faffc159770 0x7faffc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faffc06f3f0 0x7faffc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc0add00 0x7faffc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc06f920 con 0x7faffc159770 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faffc15b290 con 0x7faffc06f3f0 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7fb000e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc15b570 con 0x7faffc0add00 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7faff9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faffc159770 0x7faffc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7faff9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faffc159770 0x7faffc06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39364/0 (socket says 172.21.15.28:39364) 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.856+0000 7faff9e00640 1 -- 172.21.15.28:0/831851483 learned_addr learned my addr 172.21.15.28:0/831851483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff3e00640 1 --2- 172.21.15.28:0/831851483 >> v2:172.21.15.189:3300/0 conn(0x7faffc06f3f0 0x7faffc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faffa800640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc0add00 0x7faffc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff9e00640 1 -- 172.21.15.28:0/831851483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc0add00 msgr2=0x7faffc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff9e00640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc0add00 0x7faffc159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff9e00640 1 -- 172.21.15.28:0/831851483 >> v2:172.21.15.189:3300/0 conn(0x7faffc06f3f0 msgr2=0x7faffc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff9e00640 1 --2- 172.21.15.28:0/831851483 >> v2:172.21.15.189:3300/0 conn(0x7faffc06f3f0 0x7faffc070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.857+0000 7faff9e00640 1 -- 172.21.15.28:0/831851483 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc15b850 con 0x7faffc159770 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.858+0000 7faff9e00640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc06fd20 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7faffc06b950 tx=0x7fafec02c350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bd2717ffaeb1641 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.858+0000 7faff9400640 1 -- 172.21.15.28:0/831851483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec02cea0 con 0x7faffc159770 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.858+0000 7faff9400640 1 -- 172.21.15.28:0/831851483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec034640 con 0x7faffc159770 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.858+0000 7faff9400640 1 -- 172.21.15.28:0/831851483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec0349d0 con 0x7faffc159770 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 -- 172.21.15.28:0/831851483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 msgr2=0x7faffc06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc06fd20 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7faffc06b950 tx=0x7fafec02c350 comp rx=0 tx=0).stop 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 -- 172.21.15.28:0/831851483 shutdown_connections 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc0add00 0x7faffc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 --2- 172.21.15.28:0/831851483 >> v2:172.21.15.189:3300/0 conn(0x7faffc06f3f0 0x7faffc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 --2- 172.21.15.28:0/831851483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc06fd20 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 -- 172.21.15.28:0/831851483 >> 172.21.15.28:0/831851483 conn(0x7faffc0b0270 msgr2=0x7faffc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 -- 172.21.15.28:0/831851483 shutdown_connections 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.859+0000 7fb000e00640 1 -- 172.21.15.28:0/831851483 wait complete. 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.860+0000 7fb000e00640 1 Processor -- start 2024-09-13T17:16:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.861+0000 7fb000e00640 1 -- start start 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.861+0000 7fb000e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc06f3f0 0x7faffc145f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.861+0000 7fb000e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc0add00 0x7faffc157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7fb000e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7fb000e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1587d0 con 0x7faffc159770 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7fb000e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1464a0 con 0x7faffc0add00 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7fb000e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc157bc0 con 0x7faffc06f3f0 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39380/0 (socket says 172.21.15.28:39380) 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 -- 172.21.15.28:0/3213819614 learned_addr learned my addr 172.21.15.28:0/3213819614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faff9e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc06f3f0 0x7faffc145f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faff3e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc0add00 0x7faffc157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 -- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc06f3f0 msgr2=0x7faffc145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc06f3f0 0x7faffc145f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 -- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc0add00 msgr2=0x7faffc157350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc0add00 0x7faffc157350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.862+0000 7faffa800640 1 -- 172.21.15.28:0/3213819614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec0094d0 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7faffa800640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7faff4007e70 tx=0x7faff400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4014620 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff40147a0 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc157ea0 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff4014b50 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.863+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc158460 con 0x7faffc159770 2024-09-13T17:16:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.864+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc158620 con 0x7faffc159770 2024-09-13T17:16:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.865+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faff4014cb0 con 0x7faffc159770 2024-09-13T17:16:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.866+0000 7faff2000640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 0x7fafd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.866+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faff4089d20 con 0x7faffc159770 2024-09-13T17:16:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.866+0000 7faff9e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 0x7fafd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.867+0000 7faff9e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 0x7fafd806d0c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fafec036000 tx=0x7fafec030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:11.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:11.870+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faff4054e40 con 0x7faffc159770 2024-09-13T17:16:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.048+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faffc159b50 con 0x7faffc159770 2024-09-13T17:16:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.048+0000 7faff2000640 1 -- 172.21.15.28:0/3213819614 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7faff4058460 con 0x7faffc159770 2024-09-13T17:16:12.050 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.050+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 msgr2=0x7fafd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.050+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 0x7fafd806d0c0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fafec036000 tx=0x7fafec030000 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.050+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 msgr2=0x7faffc155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.050+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7faff4007e70 tx=0x7faff400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 shutdown_connections 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafd806ac00 0x7fafd806d0c0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faffc159770 0x7faffc155850 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faffc0add00 0x7faffc157350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 --2- 172.21.15.28:0/3213819614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faffc06f3f0 0x7faffc145f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 >> 172.21.15.28:0/3213819614 conn(0x7faffc0b0270 msgr2=0x7faffc0b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 shutdown_connections 2024-09-13T17:16:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.051+0000 7fb000e00640 1 -- 172.21.15.28:0/3213819614 wait complete. 2024-09-13T17:16:12.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:16:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.223+0000 7fc3a7200640 1 Processor -- start 2024-09-13T17:16:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.224+0000 7fc3a7200640 1 -- start start 2024-09-13T17:16:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a815ab70 0x7fc3a815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a806c370 0x7fc3a806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a7200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a80b2090 con 0x7fc3a80c40a0 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a806cd40 con 0x7fc3a806c370 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80b2370 con 0x7fc3a815ab70 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a806c370 0x7fc3a806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3a806c370 0x7fc3a806c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39388/0 (socket says 172.21.15.28:39388) 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc3a6800640 1 -- 172.21.15.28:0/2104119253 learned_addr learned my addr 172.21.15.28:0/2104119253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.225+0000 7fc39d400640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a815ab70 0x7fc3a815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.226+0000 7fc3a6800640 1 -- 172.21.15.28:0/2104119253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a815ab70 msgr2=0x7fc3a815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.226+0000 7fc3a6800640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a815ab70 0x7fc3a815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.226+0000 7fc3a6800640 1 -- 172.21.15.28:0/2104119253 >> v1:172.21.15.189:6789/0 conn(0x7fc3a80c40a0 legacy=0x7fc3a80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.226+0000 7fc3a6800640 1 -- 172.21.15.28:0/2104119253 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a80b2650 con 0x7fc3a806c370 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.227+0000 7fc3a6800640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806c370 0x7fc3a806c800 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009e50 tx=0x7fc3a002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c4dcda1fee288d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.227+0000 7fc3a5400640 1 -- 172.21.15.28:0/2104119253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031600 con 0x7fc3a806c370 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.227+0000 7fc3a5400640 1 -- 172.21.15.28:0/2104119253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3a002d050 con 0x7fc3a806c370 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.227+0000 7fc3a5400640 1 -- 172.21.15.28:0/2104119253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0031980 con 0x7fc3a806c370 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 -- 172.21.15.28:0/2104119253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806c370 msgr2=0x7fc3a806c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806c370 0x7fc3a806c800 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0009e50 tx=0x7fc3a002a710 comp rx=0 tx=0).stop 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 -- 172.21.15.28:0/2104119253 shutdown_connections 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a806c370 0x7fc3a806c800 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 --2- 172.21.15.28:0/2104119253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a815ab70 0x7fc3a815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 -- 172.21.15.28:0/2104119253 >> 172.21.15.28:0/2104119253 conn(0x7fc3a8073470 msgr2=0x7fc3a8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 -- 172.21.15.28:0/2104119253 shutdown_connections 2024-09-13T17:16:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.228+0000 7fc3a7200640 1 -- 172.21.15.28:0/2104119253 wait complete. 2024-09-13T17:16:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.229+0000 7fc3a7200640 1 Processor -- start 2024-09-13T17:16:12.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 -- start start 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a806c370 0x7fc3a81f1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 0x7fc3a80a9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80c6a60 con 0x7fc3a80c40a0 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80a9390 con 0x7fc3a815ab70 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.230+0000 7fc3a7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a80c92c0 con 0x7fc3a806c370 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39396/0 (socket says 172.21.15.28:39396) 2024-09-13T17:16:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 -- 172.21.15.28:0/3791936014 learned_addr learned my addr 172.21.15.28:0/3791936014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc3a5e00640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a806c370 0x7fc3a81f1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc3a6800640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 0x7fc3a80a9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 -- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a806c370 msgr2=0x7fc3a81f1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a806c370 0x7fc3a81f1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 -- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 msgr2=0x7fc3a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 0x7fc3a80a9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc39d400640 1 -- 172.21.15.28:0/3791936014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a0009b30 con 0x7fc3a80c40a0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.231+0000 7fc3a6800640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 0x7fc3a80a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc39d400640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc39000d620 tx=0x7fc39000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39000b620 con 0x7fc3a80c40a0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39000b780 con 0x7fc3a80c40a0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39000bb30 con 0x7fc3a80c40a0 2024-09-13T17:16:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a80c6cb0 con 0x7fc3a80c40a0 2024-09-13T17:16:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.232+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a80c7270 con 0x7fc3a80c40a0 2024-09-13T17:16:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.233+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a80c7430 con 0x7fc3a80c40a0 2024-09-13T17:16:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.234+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc39000bc90 con 0x7fc3a80c40a0 2024-09-13T17:16:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.235+0000 7fc39ea00640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 0x7fc38406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.235+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc390089860 con 0x7fc3a80c40a0 2024-09-13T17:16:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.235+0000 7fc3a5e00640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 0x7fc38406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.236+0000 7fc3a5e00640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 0x7fc38406d0f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc398000f60 tx=0x7fc398009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.239+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc390054900 con 0x7fc3a80c40a0 2024-09-13T17:16:12.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.429+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc3a80c3b00 con 0x7fc3a80c40a0 2024-09-13T17:16:12.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.430+0000 7fc39ea00640 1 -- 172.21.15.28:0/3791936014 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc390057f20 con 0x7fc3a80c40a0 2024-09-13T17:16:12.431 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:16:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 msgr2=0x7fc38406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 0x7fc38406d0f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fc398000f60 tx=0x7fc398009040 comp rx=0 tx=0).stop 2024-09-13T17:16:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 msgr2=0x7fc3a80a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7fc39000d620 tx=0x7fc39000dae0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 shutdown_connections 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc38406ac30 0x7fc38406d0f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3a815ab70 0x7fc3a80a9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a80c40a0 0x7fc3a80a8e50 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 --2- 172.21.15.28:0/3791936014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3a806c370 0x7fc3a81f1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 >> 172.21.15.28:0/3791936014 conn(0x7fc3a8073470 msgr2=0x7fc3a806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 shutdown_connections 2024-09-13T17:16:12.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.432+0000 7fc3a7200640 1 -- 172.21.15.28:0/3791936014 wait complete. 2024-09-13T17:16:12.448 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:16:12.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:12.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:16:12.522 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:16:12.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:12.522 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:16:12.567 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:16:12.567 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:12.606 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:16:12.606 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:12.606 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:12.643 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:16:12.643 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:16:12.643 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:12.683 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:16:12.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:12.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.837+0000 7fdcff400640 1 Processor -- start 2024-09-13T17:16:12.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.837+0000 7fdcff400640 1 -- start start 2024-09-13T17:16:12.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcf8160310 0x7fdcf8160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcff400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 0x7fdcf80acc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcff400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcf8146550 con 0x7fdcf8146170 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcff400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf815cd20 con 0x7fdcf8160310 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcff400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcf815d280 con 0x7fdcf80ac850 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcf8160310 0x7fdcf8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdcf8160310 0x7fdcf8160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39410/0 (socket says 172.21.15.28:39410) 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 -- 172.21.15.28:0/2834364192 learned_addr learned my addr 172.21.15.28:0/2834364192 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcfd400640 1 --2- 172.21.15.28:0/2834364192 >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 0x7fdcf80acc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 -- 172.21.15.28:0/2834364192 >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 msgr2=0x7fdcf80acc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 --2- 172.21.15.28:0/2834364192 >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 0x7fdcf80acc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 -- 172.21.15.28:0/2834364192 >> v1:172.21.15.189:6789/0 conn(0x7fdcf8146170 legacy=0x7fdcf815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcefe00640 1 -- 172.21.15.28:0/2834364192 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf80ad1c0 con 0x7fdcf8160310 2024-09-13T17:16:12.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.839+0000 7fdcfd400640 1 --2- 172.21.15.28:0/2834364192 >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 0x7fdcf80acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:12.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.840+0000 7fdcefe00640 1 --2- 172.21.15.28:0/2834364192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8160310 0x7fdcf8160730 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fdcf806d4b0 tx=0x7fdce002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa99df552dc9b2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcf7e00640 1 -- 172.21.15.28:0/2834364192 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0032440 con 0x7fdcf8160310 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcf7e00640 1 -- 172.21.15.28:0/2834364192 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce00325a0 con 0x7fdcf8160310 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcf7e00640 1 -- 172.21.15.28:0/2834364192 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0032930 con 0x7fdcf8160310 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 -- 172.21.15.28:0/2834364192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8160310 msgr2=0x7fdcf8160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 --2- 172.21.15.28:0/2834364192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8160310 0x7fdcf8160730 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fdcf806d4b0 tx=0x7fdce002a270 comp rx=0 tx=0).stop 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 -- 172.21.15.28:0/2834364192 shutdown_connections 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 --2- 172.21.15.28:0/2834364192 >> v2:172.21.15.189:3300/0 conn(0x7fdcf80ac850 0x7fdcf80acc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 --2- 172.21.15.28:0/2834364192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8160310 0x7fdcf8160730 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 -- 172.21.15.28:0/2834364192 >> 172.21.15.28:0/2834364192 conn(0x7fdcf8069e80 msgr2=0x7fdcf806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.841+0000 7fdcff400640 1 -- 172.21.15.28:0/2834364192 shutdown_connections 2024-09-13T17:16:12.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.842+0000 7fdcff400640 1 -- 172.21.15.28:0/2834364192 wait complete. 2024-09-13T17:16:12.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.843+0000 7fdcff400640 1 Processor -- start 2024-09-13T17:16:12.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.843+0000 7fdcff400640 1 -- start start 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 0x7fdcf8144ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcf8160310 0x7fdcf81576a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf8159aa0 con 0x7fdcf8146170 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf8159d80 con 0x7fdcf80ac850 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcff400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf815a060 con 0x7fdcf8160310 2024-09-13T17:16:12.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39424/0 (socket says 172.21.15.28:39424) 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.844+0000 7fdcefe00640 1 -- 172.21.15.28:0/3203624124 learned_addr learned my addr 172.21.15.28:0/3203624124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcfca00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 0x7fdcf8144ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcfd400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcf8160310 0x7fdcf81576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 -- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcf8160310 msgr2=0x7fdcf81576a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcf8160310 0x7fdcf81576a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 -- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 msgr2=0x7fdcf8144ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 0x7fdcf8144ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 -- 172.21.15.28:0/3203624124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce00094d0 con 0x7fdcf8146170 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcfca00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 0x7fdcf8144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:12.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcefe00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fdce002a2b0 tx=0x7fdce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.845+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce0032440 con 0x7fdcf8146170 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.846+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0030ba0 con 0x7fdcf8146170 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.846+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdce003d440 con 0x7fdcf8146170 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.846+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf815a340 con 0x7fdcf8146170 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.846+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf8144690 con 0x7fdcf8146170 2024-09-13T17:16:12.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.847+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf815a860 con 0x7fdcf8146170 2024-09-13T17:16:12.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.848+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdce0039040 con 0x7fdcf8146170 2024-09-13T17:16:12.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.848+0000 7fdcf5600640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 0x7fdcd406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:12.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.849+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdce00a9f00 con 0x7fdcf8146170 2024-09-13T17:16:12.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.849+0000 7fdcfca00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 0x7fdcd406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:12.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.849+0000 7fdcfca00640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 0x7fdcd406d110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdcf81e8ec0 tx=0x7fdce8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:12.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:12.852+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdcf815a860 con 0x7fdcf8146170 2024-09-13T17:16:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.067+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcf815a860 con 0x7fdcf8146170 2024-09-13T17:16:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.067+0000 7fdcf5600640 1 -- 172.21.15.28:0/3203624124 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4745 (secure 0 0 0) 0x7fdcf815a860 con 0x7fdcf8146170 2024-09-13T17:16:13.069 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:13.069 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:16:04.081667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:16:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-13T17:16:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 msgr2=0x7fdcd406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 0x7fdcd406d110 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fdcf81e8ec0 tx=0x7fdce8009040 comp rx=0 tx=0).stop 2024-09-13T17:16:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 msgr2=0x7fdcf8144150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fdce002a2b0 tx=0x7fdce00046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 shutdown_connections 2024-09-13T17:16:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdcd406ac50 0x7fdcd406d110 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdcf8160310 0x7fdcf81576a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdcf8146170 0x7fdcf8144150 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 --2- 172.21.15.28:0/3203624124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdcf80ac850 0x7fdcf8144ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 >> 172.21.15.28:0/3203624124 conn(0x7fdcf8069e80 msgr2=0x7fdcf80aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 shutdown_connections 2024-09-13T17:16:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.069+0000 7fdcff400640 1 -- 172.21.15.28:0/3203624124 wait complete. 2024-09-13T17:16:13.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:13.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.242+0000 7f3924e00640 1 Processor -- start 2024-09-13T17:16:13.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.242+0000 7f3924e00640 1 -- start start 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f3924e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3920071c00 0x7f3920146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f3924e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39200b27c0 0x7f3920070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f3924e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3920143070 con 0x7f39200ad260 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f3924e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3920071fe0 con 0x7f39200b27c0 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f3924e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39200ada70 con 0x7f3920071c00 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f391e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39200b27c0 0x7f3920070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f391e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39200b27c0 0x7f3920070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39440/0 (socket says 172.21.15.28:39440) 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.243+0000 7f391e800640 1 -- 172.21.15.28:0/2383404971 learned_addr learned my addr 172.21.15.28:0/2383404971 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:13.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.244+0000 7f391de00640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3920071c00 0x7f3920146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.244+0000 7f391e800640 1 -- 172.21.15.28:0/2383404971 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3920071c00 msgr2=0x7f3920146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.244+0000 7f391e800640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3920071c00 0x7f3920146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.244+0000 7f391e800640 1 -- 172.21.15.28:0/2383404971 >> v1:172.21.15.189:6789/0 conn(0x7f39200ad260 legacy=0x7f39200b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.244+0000 7f391e800640 1 -- 172.21.15.28:0/2383404971 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39200add50 con 0x7f39200b27c0 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.245+0000 7f391e800640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39200b27c0 0x7f3920070290 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f3918009e50 tx=0x7f391802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de79870a7f3edd3a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.245+0000 7f391d400640 1 -- 172.21.15.28:0/2383404971 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918031600 con 0x7f39200b27c0 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.245+0000 7f391d400640 1 -- 172.21.15.28:0/2383404971 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391802d050 con 0x7f39200b27c0 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.245+0000 7f391d400640 1 -- 172.21.15.28:0/2383404971 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3918031980 con 0x7f39200b27c0 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 -- 172.21.15.28:0/2383404971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39200b27c0 msgr2=0x7f3920070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39200b27c0 0x7f3920070290 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f3918009e50 tx=0x7f391802a710 comp rx=0 tx=0).stop 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 -- 172.21.15.28:0/2383404971 shutdown_connections 2024-09-13T17:16:13.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39200b27c0 0x7f3920070290 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 --2- 172.21.15.28:0/2383404971 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3920071c00 0x7f3920146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 -- 172.21.15.28:0/2383404971 >> 172.21.15.28:0/2383404971 conn(0x7f39200b0b60 msgr2=0x7f39200b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 -- 172.21.15.28:0/2383404971 shutdown_connections 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.246+0000 7f3924e00640 1 -- 172.21.15.28:0/2383404971 wait complete. 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.247+0000 7f3924e00640 1 Processor -- start 2024-09-13T17:16:13.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 -- start start 2024-09-13T17:16:13.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200ad260 0x7f392015e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200b27c0 0x7f3920141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39200ce770 con 0x7f3920071c00 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392015d0c0 con 0x7f39200b27c0 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.248+0000 7f3924e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392015d3a0 con 0x7f39200ad260 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39450/0 (socket says 172.21.15.28:39450) 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 -- 172.21.15.28:0/3269518482 learned_addr learned my addr 172.21.15.28:0/3269518482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391e800640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200b27c0 0x7f3920141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f3915400640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200ad260 0x7f392015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 -- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200ad260 msgr2=0x7f392015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200ad260 0x7f392015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 -- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200b27c0 msgr2=0x7f3920141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200b27c0 0x7f3920141d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.249+0000 7f391de00640 1 -- 172.21.15.28:0/3269518482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918009b30 con 0x7f3920071c00 2024-09-13T17:16:13.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f391de00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f391000e870 tx=0x7f391000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39100094f0 con 0x7f3920071c00 2024-09-13T17:16:13.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391000c940 con 0x7f3920071c00 2024-09-13T17:16:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391000ccf0 con 0x7f3920071c00 2024-09-13T17:16:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392015d680 con 0x7f3920071c00 2024-09-13T17:16:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.250+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392015dc40 con 0x7f3920071c00 2024-09-13T17:16:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.251+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39200b1220 con 0x7f3920071c00 2024-09-13T17:16:13.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.252+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3910013470 con 0x7f3920071c00 2024-09-13T17:16:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.253+0000 7f3916a00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 0x7f38fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.253+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f39100894a0 con 0x7f3920071c00 2024-09-13T17:16:13.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.254+0000 7f3915400640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 0x7f38fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.254+0000 7f3915400640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 0x7f38fc06d0c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3920142660 tx=0x7f3908009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.257+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3910054670 con 0x7f3920071c00 2024-09-13T17:16:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.437+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39200b0f40 con 0x7f3920071c00 2024-09-13T17:16:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.438+0000 7f3916a00640 1 -- 172.21.15.28:0/3269518482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16586 (secure 0 0 0) 0x7f3910057c90 con 0x7f3920071c00 2024-09-13T17:16:13.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:13.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":113,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:15:45.960852+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:13.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 msgr2=0x7f38fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 0x7f38fc06d0c0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f3920142660 tx=0x7f3908009040 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 msgr2=0x7f39200ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f391000e870 tx=0x7f391000ed30 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 shutdown_connections 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38fc06ac00 0x7f38fc06d0c0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39200b27c0 0x7f3920141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39200ad260 0x7f392015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 --2- 172.21.15.28:0/3269518482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3920071c00 0x7f39200ce230 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.440+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 >> 172.21.15.28:0/3269518482 conn(0x7f39200b0b60 msgr2=0x7f39200700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.441+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 shutdown_connections 2024-09-13T17:16:13.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.441+0000 7f3924e00640 1 -- 172.21.15.28:0/3269518482 wait complete. 2024-09-13T17:16:13.458 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:14:29.435536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:14:30.524696+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '112', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 112, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:16:13.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:16:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.616+0000 7f0d95c00640 1 Processor -- start 2024-09-13T17:16:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.616+0000 7f0d95c00640 1 -- start start 2024-09-13T17:16:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d95c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d9015a780 0x7f0d90070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d95c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 0x7f0d900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d95c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9006c8b0 con 0x7f0d9006bb40 2024-09-13T17:16:13.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d95c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9006cb90 con 0x7f0d9015a780 2024-09-13T17:16:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d95c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900cc980 con 0x7f0d9015ab60 2024-09-13T17:16:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d8f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d9006bb40 0x7f0d9006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42446/0 (socket says 172.21.15.28:42446) 2024-09-13T17:16:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.617+0000 7f0d8f400640 1 -- 172.21.15.28:0/60351172 learned_addr learned my addr 172.21.15.28:0/60351172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d86000640 1 --2- 172.21.15.28:0/60351172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 0x7f0d900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d8ea00640 1 --2- 172.21.15.28:0/60351172 >> v2:172.21.15.189:3300/0 conn(0x7f0d9015a780 0x7f0d90070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d8e000640 1 -- 172.21.15.28:0/60351172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679475064 0 0) 0x7f0d9006c8b0 con 0x7f0d9006bb40 2024-09-13T17:16:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d8e000640 1 -- 172.21.15.28:0/60351172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d74003520 con 0x7f0d9006bb40 2024-09-13T17:16:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d86000640 1 -- 172.21.15.28:0/60351172 >> v2:172.21.15.189:3300/0 conn(0x7f0d9015a780 msgr2=0x7f0d90070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d86000640 1 --2- 172.21.15.28:0/60351172 >> v2:172.21.15.189:3300/0 conn(0x7f0d9015a780 0x7f0d90070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d86000640 1 -- 172.21.15.28:0/60351172 >> v1:172.21.15.28:6789/0 conn(0x7f0d9006bb40 legacy=0x7f0d9006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.618+0000 7f0d86000640 1 -- 172.21.15.28:0/60351172 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900ccc60 con 0x7f0d9015ab60 2024-09-13T17:16:13.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.619+0000 7f0d86000640 1 --2- 172.21.15.28:0/60351172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 0x7f0d900712c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0d780097a0 tx=0x7f0d7802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a8482dc1983257 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d8e000640 1 -- 172.21.15.28:0/60351172 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78032440 con 0x7f0d9015ab60 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d8e000640 1 -- 172.21.15.28:0/60351172 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d780325a0 con 0x7f0d9015ab60 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d8e000640 1 -- 172.21.15.28:0/60351172 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d78032930 con 0x7f0d9015ab60 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 -- 172.21.15.28:0/60351172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 msgr2=0x7f0d900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 --2- 172.21.15.28:0/60351172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 0x7f0d900712c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0d780097a0 tx=0x7f0d7802a180 comp rx=0 tx=0).stop 2024-09-13T17:16:13.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 -- 172.21.15.28:0/60351172 shutdown_connections 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 --2- 172.21.15.28:0/60351172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9015ab60 0x7f0d900712c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 --2- 172.21.15.28:0/60351172 >> v2:172.21.15.189:3300/0 conn(0x7f0d9015a780 0x7f0d90070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 -- 172.21.15.28:0/60351172 >> 172.21.15.28:0/60351172 conn(0x7f0d900b0270 msgr2=0x7f0d900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 -- 172.21.15.28:0/60351172 shutdown_connections 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.620+0000 7f0d95c00640 1 -- 172.21.15.28:0/60351172 wait complete. 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.622+0000 7f0d95c00640 1 Processor -- start 2024-09-13T17:16:13.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.622+0000 7f0d95c00640 1 -- start start 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9006bb40 0x7f0d90144c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9015a780 0x7f0d90140df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90140710 con 0x7f0d9015ab60 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90140b30 con 0x7f0d9015a780 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d95c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9013fa90 con 0x7f0d9006bb40 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39456/0 (socket says 172.21.15.28:39456) 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d8f400640 1 -- 172.21.15.28:0/2545698751 learned_addr learned my addr 172.21.15.28:0/2545698751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:13.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d8ea00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9006bb40 0x7f0d90144c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.623+0000 7f0d86000640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9015a780 0x7f0d90140df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 -- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9006bb40 msgr2=0x7f0d90144c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9006bb40 0x7f0d90144c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 -- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9015a780 msgr2=0x7f0d90140df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9015a780 0x7f0d90140df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 -- 172.21.15.28:0/2545698751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d78009400 con 0x7f0d9015ab60 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d8f400640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cb40 tx=0x7f0d8800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800aca0 con 0x7f0d9015ab60 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8800ae00 con 0x7f0d9015ab60 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88012620 con 0x7f0d9015ab60 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.624+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9013fd70 con 0x7f0d9015ab60 2024-09-13T17:16:13.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.625+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d90140330 con 0x7f0d9015ab60 2024-09-13T17:16:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.626+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9006c8b0 con 0x7f0d9015ab60 2024-09-13T17:16:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.626+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d88007070 con 0x7f0d9015ab60 2024-09-13T17:16:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.627+0000 7f0d87400640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 0x7f0d6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.627+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0d88089350 con 0x7f0d9015ab60 2024-09-13T17:16:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.628+0000 7f0d8ea00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 0x7f0d6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.629+0000 7f0d8ea00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 0x7f0d6c06d0c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009b40 tx=0x7f0d80009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.631+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d880545b0 con 0x7f0d9015ab60 2024-09-13T17:16:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.810+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d900aef60 con 0x7f0d9015ab60 2024-09-13T17:16:13.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.811+0000 7f0d87400640 1 -- 172.21.15.28:0/2545698751 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d88057bd0 con 0x7f0d9015ab60 2024-09-13T17:16:13.812 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:16:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 msgr2=0x7f0d6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 0x7f0d6c06d0c0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009b40 tx=0x7f0d80009190 comp rx=0 tx=0).stop 2024-09-13T17:16:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 msgr2=0x7f0d90141330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cb40 tx=0x7f0d8800a060 comp rx=0 tx=0).stop 2024-09-13T17:16:13.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 shutdown_connections 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d6c06ac00 0x7f0d6c06d0c0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9015ab60 0x7f0d90141330 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9015a780 0x7f0d90140df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 --2- 172.21.15.28:0/2545698751 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9006bb40 0x7f0d90144c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 >> 172.21.15.28:0/2545698751 conn(0x7f0d900b0270 msgr2=0x7f0d900ce9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 shutdown_connections 2024-09-13T17:16:13.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.813+0000 7f0d95c00640 1 -- 172.21.15.28:0/2545698751 wait complete. 2024-09-13T17:16:13.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:16:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.986+0000 7f8454e00640 1 Processor -- start 2024-09-13T17:16:13.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.986+0000 7f8454e00640 1 -- start start 2024-09-13T17:16:13.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f8454e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8450158cd0 0x7f84501590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f8454e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f845006ccd0 0x7f845006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f8454e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84501595f0 con 0x7f845006bf40 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f8454e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845006c3d0 con 0x7f8450158cd0 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f8454e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84501598c0 con 0x7f845006ccd0 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f844e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f845006bf40 0x7f845015c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39930/0 (socket says 172.21.15.28:39930) 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.987+0000 7f844e800640 1 -- 172.21.15.28:0/2027317429 learned_addr learned my addr 172.21.15.28:0/2027317429 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844de00640 1 --2- 172.21.15.28:0/2027317429 >> v2:172.21.15.189:3300/0 conn(0x7f8450158cd0 0x7f84501590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f8445400640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f845006ccd0 0x7f845006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844de00640 1 -- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f845006ccd0 msgr2=0x7f845006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844de00640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f845006ccd0 0x7f845006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844d400640 1 -- 172.21.15.28:0/2027317429 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223102331 0 0) 0x7f84501595f0 con 0x7f845006bf40 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844de00640 1 -- 172.21.15.28:0/2027317429 >> v1:172.21.15.28:6789/0 conn(0x7f845006bf40 legacy=0x7f845015c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.988+0000 7f844de00640 1 -- 172.21.15.28:0/2027317429 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8450159ba0 con 0x7f8450158cd0 2024-09-13T17:16:13.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.989+0000 7f844de00640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8450158cd0 0x7f84501590b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8440009d90 tx=0x7f844002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df212aadf870d5a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.989+0000 7f844d400640 1 -- 172.21.15.28:0/2027317429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440007740 con 0x7f8450158cd0 2024-09-13T17:16:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f844d400640 1 -- 172.21.15.28:0/2027317429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844002d050 con 0x7f8450158cd0 2024-09-13T17:16:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f844d400640 1 -- 172.21.15.28:0/2027317429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440007ac0 con 0x7f8450158cd0 2024-09-13T17:16:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 -- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8450158cd0 msgr2=0x7f84501590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8450158cd0 0x7f84501590b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f8440009d90 tx=0x7f844002a660 comp rx=0 tx=0).stop 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 -- 172.21.15.28:0/2027317429 shutdown_connections 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f845006ccd0 0x7f845006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 --2- 172.21.15.28:0/2027317429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8450158cd0 0x7f84501590b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 -- 172.21.15.28:0/2027317429 >> 172.21.15.28:0/2027317429 conn(0x7f84500b0270 msgr2=0x7f84500b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 -- 172.21.15.28:0/2027317429 shutdown_connections 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.990+0000 7f8454e00640 1 -- 172.21.15.28:0/2027317429 wait complete. 2024-09-13T17:16:13.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.992+0000 7f8454e00640 1 Processor -- start 2024-09-13T17:16:13.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.992+0000 7f8454e00640 1 -- start start 2024-09-13T17:16:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f845006bf40 0x7f84501571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8450158cd0 0x7f84500af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84500af930 con 0x7f845006ccd0 2024-09-13T17:16:13.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84501f33e0 con 0x7f845006bf40 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8454e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84501f27f0 con 0x7f8450158cd0 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8445400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58592/0 (socket says 172.21.15.28:58592) 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8445400640 1 -- 172.21.15.28:0/2905813366 learned_addr learned my addr 172.21.15.28:0/2905813366 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f844de00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f845006bf40 0x7f84501571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f844e800640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8450158cd0 0x7f84500af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.993+0000 7f8445400640 1 -- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8450158cd0 msgr2=0x7f84500af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8445400640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8450158cd0 0x7f84500af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8445400640 1 -- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f845006bf40 msgr2=0x7f84501571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8445400640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f845006bf40 0x7f84501571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:13.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8445400640 1 -- 172.21.15.28:0/2905813366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440009a70 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8445400640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f843800e910 tx=0x7f843800edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f843800cc40 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f843800cda0 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8438011610 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84501f2ab0 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.994+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84501f3070 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.995+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84500706b0 con 0x7f845006ccd0 2024-09-13T17:16:13.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.996+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8438011770 con 0x7f845006ccd0 2024-09-13T17:16:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.997+0000 7f8446a00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 0x7f842c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:13.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.998+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8438089130 con 0x7f845006ccd0 2024-09-13T17:16:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.998+0000 7f844de00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 0x7f842c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:13.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:13.999+0000 7f844de00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 0x7f842c06d0c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8440034000 tx=0x7f844002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.002+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8438054300 con 0x7f845006ccd0 2024-09-13T17:16:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.185+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84501590b0 con 0x7f845006ccd0 2024-09-13T17:16:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.185+0000 7f8446a00640 1 -- 172.21.15.28:0/2905813366 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8438057920 con 0x7f845006ccd0 2024-09-13T17:16:14.187 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 msgr2=0x7f842c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 0x7f842c06d0c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f8440034000 tx=0x7f844002f000 comp rx=0 tx=0).stop 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 msgr2=0x7f84500aeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f843800e910 tx=0x7f843800edd0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 shutdown_connections 2024-09-13T17:16:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f842c06ac00 0x7f842c06d0c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8450158cd0 0x7f84500af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f845006ccd0 0x7f84500aeeb0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 --2- 172.21.15.28:0/2905813366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f845006bf40 0x7f84501571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 >> 172.21.15.28:0/2905813366 conn(0x7f84500b0270 msgr2=0x7f84500ae390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 shutdown_connections 2024-09-13T17:16:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.188+0000 7f8454e00640 1 -- 172.21.15.28:0/2905813366 wait complete. 2024-09-13T17:16:14.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:16:14.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.361+0000 7fccf5a00640 1 Processor -- start 2024-09-13T17:16:14.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.361+0000 7fccf5a00640 1 -- start start 2024-09-13T17:16:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fccf00cd7c0 0x7fccf00c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf015ef70 con 0x7fccf00c3ba0 2024-09-13T17:16:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccf00c4320 con 0x7fccf0143f70 2024-09-13T17:16:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccf015e1b0 con 0x7fccf00cd7c0 2024-09-13T17:16:14.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fccf00c3ba0 0x7fccf0070440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39936/0 (socket says 172.21.15.28:39936) 2024-09-13T17:16:14.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.362+0000 7fccf5000640 1 -- 172.21.15.28:0/3867085378 learned_addr learned my addr 172.21.15.28:0/3867085378 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:14.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fcce7400640 1 --2- 172.21.15.28:0/3867085378 >> v2:172.21.15.189:3300/0 conn(0x7fccf00cd7c0 0x7fccf00c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970653854 0 0) 0x7fccf015ef70 con 0x7fccf00c3ba0 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd4003520 con 0x7fccf00c3ba0 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22036933 0 0) 0x7fccf00c4320 con 0x7fccf0143f70 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccf015ef70 con 0x7fccf0143f70 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fcce7400640 1 -- 172.21.15.28:0/3867085378 >> v1:172.21.15.189:6789/0 conn(0x7fccf0143f70 legacy=0x7fccf00cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fcce7400640 1 -- 172.21.15.28:0/3867085378 >> v1:172.21.15.28:6789/0 conn(0x7fccf00c3ba0 legacy=0x7fccf0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030937603 0 0) 0x7fccd4003520 con 0x7fccf00c3ba0 2024-09-13T17:16:14.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.363+0000 7fcce7400640 1 -- 172.21.15.28:0/3867085378 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf015e490 con 0x7fccf00cd7c0 2024-09-13T17:16:14.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.364+0000 7fcce7400640 1 --2- 172.21.15.28:0/3867085378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00cd7c0 0x7fccf00c3660 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fccd80097a0 tx=0x7fccd802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ee1e905078d9a9c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.364+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8032440 con 0x7fccf00cd7c0 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccd80325a0 con 0x7fccf00cd7c0 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccef400640 1 -- 172.21.15.28:0/3867085378 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccd8032930 con 0x7fccf00cd7c0 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 -- 172.21.15.28:0/3867085378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00cd7c0 msgr2=0x7fccf00c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 --2- 172.21.15.28:0/3867085378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00cd7c0 0x7fccf00c3660 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fccd80097a0 tx=0x7fccd802a230 comp rx=0 tx=0).stop 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 -- 172.21.15.28:0/3867085378 shutdown_connections 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 --2- 172.21.15.28:0/3867085378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00cd7c0 0x7fccf00c3660 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 -- 172.21.15.28:0/3867085378 >> 172.21.15.28:0/3867085378 conn(0x7fccf0069e80 msgr2=0x7fccf006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 -- 172.21.15.28:0/3867085378 shutdown_connections 2024-09-13T17:16:14.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.365+0000 7fccf5a00640 1 -- 172.21.15.28:0/3867085378 wait complete. 2024-09-13T17:16:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.367+0000 7fccf5a00640 1 Processor -- start 2024-09-13T17:16:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.367+0000 7fccf5a00640 1 -- start start 2024-09-13T17:16:14.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00c3ba0 0x7fccf015d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 0x7fccf013d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00cbae0 con 0x7fccf0143f70 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00cbdc0 con 0x7fccf00c3ba0 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccf00cc0a0 con 0x7fccf00cd7c0 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58608/0 (socket says 172.21.15.28:58608) 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fccf5000640 1 -- 172.21.15.28:0/1525045408 learned_addr learned my addr 172.21.15.28:0/1525045408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.368+0000 7fcce7400640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 0x7fccf013d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccefe00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00c3ba0 0x7fccf015d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 -- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 msgr2=0x7fccf013d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 0x7fccf013d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 -- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00c3ba0 msgr2=0x7fccf015d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00c3ba0 0x7fccf015d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 -- 172.21.15.28:0/1525045408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd8009400 con 0x7fccf0143f70 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fcce7400640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 0x7fccf013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5000640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c7a0 tx=0x7fcce800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce800a640 con 0x7fccf0143f70 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce800a7a0 con 0x7fccf0143f70 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce800ab90 con 0x7fccf0143f70 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf00cc380 con 0x7fccf0143f70 2024-09-13T17:16:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.369+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf015d980 con 0x7fccf0143f70 2024-09-13T17:16:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.370+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf015db70 con 0x7fccf0143f70 2024-09-13T17:16:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.371+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcce800acf0 con 0x7fccf0143f70 2024-09-13T17:16:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.372+0000 7fccecc00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 0x7fcccc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.372+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcce8088900 con 0x7fccf0143f70 2024-09-13T17:16:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.373+0000 7fccefe00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 0x7fcccc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.373+0000 7fccefe00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 0x7fcccc06d0c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcce00011e0 tx=0x7fcce0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.376+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcce8053a20 con 0x7fccf0143f70 2024-09-13T17:16:14.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.555+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fccf00cc8a0 con 0x7fccf0143f70 2024-09-13T17:16:14.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.555+0000 7fccecc00640 1 -- 172.21.15.28:0/1525045408 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcce8018020 con 0x7fccf0143f70 2024-09-13T17:16:14.556 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 msgr2=0x7fcccc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 0x7fcccc06d0c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7fcce00011e0 tx=0x7fcce0009040 comp rx=0 tx=0).stop 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 msgr2=0x7fccf01f22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fcce800c7a0 tx=0x7fcce800cc60 comp rx=0 tx=0).stop 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 shutdown_connections 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcccc06ac00 0x7fcccc06d0c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fccf0143f70 0x7fccf01f22d0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fccf00cd7c0 0x7fccf013d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 --2- 172.21.15.28:0/1525045408 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccf00c3ba0 0x7fccf015d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 >> 172.21.15.28:0/1525045408 conn(0x7fccf0069e80 msgr2=0x7fccf00b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 shutdown_connections 2024-09-13T17:16:14.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.558+0000 7fccf5a00640 1 -- 172.21.15.28:0/1525045408 wait complete. 2024-09-13T17:16:14.575 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.728+0000 7efcf5800640 1 Processor -- start 2024-09-13T17:16:14.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.728+0000 7efcf5800640 1 -- start start 2024-09-13T17:16:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.729+0000 7efcf5800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efcf015a540 0x7efcf006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.729+0000 7efcf5800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf0070cc0 0x7efcf0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.729+0000 7efcf5800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcf015e100 con 0x7efcf00cabc0 2024-09-13T17:16:14.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efcf5800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcf00adb00 con 0x7efcf015a540 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efcf5800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf0071670 con 0x7efcf0070cc0 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efcf015a540 0x7efcf006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efcf015a540 0x7efcf006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58624/0 (socket says 172.21.15.28:58624) 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 -- 172.21.15.28:0/2331501545 learned_addr learned my addr 172.21.15.28:0/2331501545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efcef400640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf0070cc0 0x7efcf0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 -- 172.21.15.28:0/2331501545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf0070cc0 msgr2=0x7efcf0071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf0070cc0 0x7efcf0071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 -- 172.21.15.28:0/2331501545 >> v1:172.21.15.189:6789/0 conn(0x7efcf00cabc0 legacy=0x7efcf015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.730+0000 7efceea00640 1 -- 172.21.15.28:0/2331501545 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcf015e3e0 con 0x7efcf015a540 2024-09-13T17:16:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.731+0000 7efceea00640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf015a540 0x7efcf006c430 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7efce0009870 tx=0x7efce002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0443cda7ee0fe1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcee000640 1 -- 172.21.15.28:0/2331501545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0032440 con 0x7efcf015a540 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcee000640 1 -- 172.21.15.28:0/2331501545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce00325a0 con 0x7efcf015a540 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcee000640 1 -- 172.21.15.28:0/2331501545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0032930 con 0x7efcf015a540 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 -- 172.21.15.28:0/2331501545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf015a540 msgr2=0x7efcf006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf015a540 0x7efcf006c430 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7efce0009870 tx=0x7efce002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 -- 172.21.15.28:0/2331501545 shutdown_connections 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf0070cc0 0x7efcf0071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 --2- 172.21.15.28:0/2331501545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf015a540 0x7efcf006c430 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.732+0000 7efcf5800640 1 -- 172.21.15.28:0/2331501545 >> 172.21.15.28:0/2331501545 conn(0x7efcf0069e80 msgr2=0x7efcf006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.733+0000 7efcf5800640 1 -- 172.21.15.28:0/2331501545 shutdown_connections 2024-09-13T17:16:14.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.733+0000 7efcf5800640 1 -- 172.21.15.28:0/2331501545 wait complete. 2024-09-13T17:16:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.734+0000 7efcf5800640 1 Processor -- start 2024-09-13T17:16:14.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.734+0000 7efcf5800640 1 -- start start 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 0x7efcf00b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 0x7efcf00aef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00b3280 con 0x7efcf00cabc0 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00aea70 con 0x7efcf0070cc0 2024-09-13T17:16:14.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efcf5800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcf00b1190 con 0x7efcf015a540 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.735+0000 7efce7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58632/0 (socket says 172.21.15.28:58632) 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 -- 172.21.15.28:0/1185173768 learned_addr learned my addr 172.21.15.28:0/1185173768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efcef400640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 0x7efcf00aef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efceea00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 0x7efcf00b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 -- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 msgr2=0x7efcf00aef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 0x7efcf00aef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 -- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 msgr2=0x7efcf00b0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 0x7efcf00b0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 -- 172.21.15.28:0/1185173768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00094d0 con 0x7efcf00cabc0 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efcef400640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 0x7efcf00aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efceea00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 0x7efcf00b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:14.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.736+0000 7efce7e00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c480 tx=0x7efcd800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.737+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80052d0 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.737+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd8005430 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.737+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8012870 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.737+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcf00b2fa0 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.737+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcf00ae700 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.738+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf00ae280 con 0x7efcf00cabc0 2024-09-13T17:16:14.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.739+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efcd80129d0 con 0x7efcf00cabc0 2024-09-13T17:16:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.739+0000 7efce6a00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 0x7efccc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.740+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efcd8088f40 con 0x7efcf00cabc0 2024-09-13T17:16:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.740+0000 7efceea00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 0x7efccc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:14.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.741+0000 7efceea00640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 0x7efccc06d0f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efce0034000 tx=0x7efce002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:14.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.744+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efcf00ae280 con 0x7efcf00cabc0 2024-09-13T17:16:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.960+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcf00ae280 con 0x7efcf00cabc0 2024-09-13T17:16:14.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.961+0000 7efce6a00640 1 -- 172.21.15.28:0/1185173768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4745 (secure 0 0 0) 0x7efcf00ae280 con 0x7efcf00cabc0 2024-09-13T17:16:14.962 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:14.962 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:16:04.081667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:16:14.963 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-13T17:16:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 msgr2=0x7efccc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 0x7efccc06d0f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7efce0034000 tx=0x7efce002e000 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 msgr2=0x7efcf00b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c480 tx=0x7efcd800c940 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 shutdown_connections 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efccc06ac30 0x7efccc06d0f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efcf015a540 0x7efcf00aef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf00cabc0 0x7efcf00b2600 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 --2- 172.21.15.28:0/1185173768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcf0070cc0 0x7efcf00b0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 >> 172.21.15.28:0/1185173768 conn(0x7efcf0069e80 msgr2=0x7efcf00ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 shutdown_connections 2024-09-13T17:16:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:14.963+0000 7efcf5800640 1 -- 172.21.15.28:0/1185173768 wait complete. 2024-09-13T17:16:14.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.137+0000 7f4696e00640 1 Processor -- start 2024-09-13T17:16:15.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.137+0000 7f4696e00640 1 -- start start 2024-09-13T17:16:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.138+0000 7f4696e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.138+0000 7f4696e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f469006c2d0 0x7f469006c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.138+0000 7f4696e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4690159d90 con 0x7f469006f970 2024-09-13T17:16:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.138+0000 7f4696e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f469015b0b0 con 0x7f469006c2d0 2024-09-13T17:16:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.138+0000 7f4696e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469006ccc0 con 0x7f46901592a0 2024-09-13T17:16:15.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f468fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f468fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:37370/0 (socket says 172.21.15.28:37370) 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f468fe00640 1 -- 172.21.15.28:0/884750183 learned_addr learned my addr 172.21.15.28:0/884750183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f4694e00640 1 --2- 172.21.15.28:0/884750183 >> v2:172.21.15.189:3300/0 conn(0x7f469006c2d0 0x7f469006c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f4694e00640 1 -- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 msgr2=0x7f469015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f4694e00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.139+0000 7f4694e00640 1 -- 172.21.15.28:0/884750183 >> v1:172.21.15.189:6789/0 conn(0x7f469006f970 legacy=0x7f4690159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.140+0000 7f4694e00640 1 -- 172.21.15.28:0/884750183 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900b2040 con 0x7f469006c2d0 2024-09-13T17:16:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.140+0000 7f468fe00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.141+0000 7f4694e00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006c2d0 0x7f469006c780 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f468802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d5411f7bf7fdbe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.141+0000 7f468ea00640 1 -- 172.21.15.28:0/884750183 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688007740 con 0x7f469006c2d0 2024-09-13T17:16:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.141+0000 7f468ea00640 1 -- 172.21.15.28:0/884750183 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468802d050 con 0x7f469006c2d0 2024-09-13T17:16:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.141+0000 7f468ea00640 1 -- 172.21.15.28:0/884750183 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688007ac0 con 0x7f469006c2d0 2024-09-13T17:16:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 -- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006c2d0 msgr2=0x7f469006c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006c2d0 0x7f469006c780 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f4688009d90 tx=0x7f468802a660 comp rx=0 tx=0).stop 2024-09-13T17:16:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 -- 172.21.15.28:0/884750183 shutdown_connections 2024-09-13T17:16:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006c2d0 0x7f469006c780 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 --2- 172.21.15.28:0/884750183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 -- 172.21.15.28:0/884750183 >> 172.21.15.28:0/884750183 conn(0x7f4690069e80 msgr2=0x7f469006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 -- 172.21.15.28:0/884750183 shutdown_connections 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.142+0000 7f4696e00640 1 -- 172.21.15.28:0/884750183 wait complete. 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.143+0000 7f4696e00640 1 Processor -- start 2024-09-13T17:16:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 -- start start 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006f970 0x7f4690149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469014a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469014a5e0 con 0x7f469006c2d0 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469014fe80 con 0x7f469006f970 2024-09-13T17:16:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.144+0000 7f4696e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4690150160 con 0x7f46901592a0 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58640/0 (socket says 172.21.15.28:58640) 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 -- 172.21.15.28:0/3241315399 learned_addr learned my addr 172.21.15.28:0/3241315399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468f400640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006f970 0x7f4690149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f4694e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469014a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 -- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 msgr2=0x7f469014a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469014a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 -- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006f970 msgr2=0x7f4690149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006f970 0x7f4690149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 -- 172.21.15.28:0/3241315399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468000b770 con 0x7f469006c2d0 2024-09-13T17:16:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.145+0000 7f468fe00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f468000ed40 tx=0x7f468000d720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.146+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680019680 con 0x7f469006c2d0 2024-09-13T17:16:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.146+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46800197e0 con 0x7f469006c2d0 2024-09-13T17:16:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.146+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4680019b50 con 0x7f469006c2d0 2024-09-13T17:16:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.146+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4688009a70 con 0x7f469006c2d0 2024-09-13T17:16:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.146+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4690150720 con 0x7f469006c2d0 2024-09-13T17:16:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.147+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46900b3740 con 0x7f469006c2d0 2024-09-13T17:16:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.148+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4680010440 con 0x7f469006c2d0 2024-09-13T17:16:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.149+0000 7f4687e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 0x7f467006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.149+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4680014070 con 0x7f469006c2d0 2024-09-13T17:16:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.149+0000 7f468f400640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 0x7f467006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.150+0000 7f468f400640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 0x7f467006d0c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f46900c6100 tx=0x7f467c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.152+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46800550b0 con 0x7f469006c2d0 2024-09-13T17:16:15.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.335+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46900ca820 con 0x7f469006c2d0 2024-09-13T17:16:15.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.336+0000 7f4687e00640 1 -- 172.21.15.28:0/3241315399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16586 (secure 0 0 0) 0x7f468001e090 con 0x7f469006c2d0 2024-09-13T17:16:15.337 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:15.337 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":113,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:15:45.960852+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:15.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 msgr2=0x7f467006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 0x7f467006d0c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f46900c6100 tx=0x7f467c009040 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 msgr2=0x7f46900b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f468000ed40 tx=0x7f468000d720 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 shutdown_connections 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f467006ac00 0x7f467006d0c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46901592a0 0x7f469014a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f469006f970 0x7f4690149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 --2- 172.21.15.28:0/3241315399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f469006c2d0 0x7f46900b04c0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 >> 172.21.15.28:0/3241315399 conn(0x7f4690069e80 msgr2=0x7f4690157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 shutdown_connections 2024-09-13T17:16:15.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.338+0000 7f4696e00640 1 -- 172.21.15.28:0/3241315399 wait complete. 2024-09-13T17:16:15.355 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:16:15.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.508+0000 7faef6200640 1 Processor -- start 2024-09-13T17:16:15.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.508+0000 7faef6200640 1 -- start start 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.509+0000 7faef6200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faef015a540 0x7faef006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.509+0000 7faef6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faef6200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef015e100 con 0x7faef00cabc0 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faef6200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef00adb00 con 0x7faef015a540 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faef6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef0071670 con 0x7faef0070cc0 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faee6a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faef00cabc0 0x7faef015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39940/0 (socket says 172.21.15.28:39940) 2024-09-13T17:16:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faee6a00640 1 -- 172.21.15.28:0/2463091686 learned_addr learned my addr 172.21.15.28:0/2463091686 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faeef400640 1 --2- 172.21.15.28:0/2463091686 >> v2:172.21.15.189:3300/0 conn(0x7faef015a540 0x7faef006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faeefe00640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727694396 0 0) 0x7faef015e100 con 0x7faef00cabc0 2024-09-13T17:16:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.510+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faed4003520 con 0x7faef00cabc0 2024-09-13T17:16:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.511+0000 7faeef400640 1 -- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 msgr2=0x7faef0071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.511+0000 7faeef400640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef0071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.511+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729260815 0 0) 0x7faed4003520 con 0x7faef00cabc0 2024-09-13T17:16:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.511+0000 7faeef400640 1 -- 172.21.15.28:0/2463091686 >> v1:172.21.15.28:6789/0 conn(0x7faef00cabc0 legacy=0x7faef015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.511+0000 7faeef400640 1 -- 172.21.15.28:0/2463091686 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef015e3e0 con 0x7faef015a540 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.512+0000 7faeef400640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef006c430 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faee00097a0 tx=0x7faee002a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8f0d0f5908cc1f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.512+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8001670 con 0x7faef015a540 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.512+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee002ab10 con 0x7faef015a540 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.512+0000 7faeeea00640 1 -- 172.21.15.28:0/2463091686 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee002ae80 con 0x7faef015a540 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 -- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 msgr2=0x7faef006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef006c430 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7faee00097a0 tx=0x7faee002a080 comp rx=0 tx=0).stop 2024-09-13T17:16:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 -- 172.21.15.28:0/2463091686 shutdown_connections 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef0071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 --2- 172.21.15.28:0/2463091686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef006c430 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 -- 172.21.15.28:0/2463091686 >> 172.21.15.28:0/2463091686 conn(0x7faef0069e80 msgr2=0x7faef006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 -- 172.21.15.28:0/2463091686 shutdown_connections 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.513+0000 7faef6200640 1 -- 172.21.15.28:0/2463091686 wait complete. 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.515+0000 7faef6200640 1 Processor -- start 2024-09-13T17:16:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.515+0000 7faef6200640 1 -- start start 2024-09-13T17:16:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef01f3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef00b0e30 con 0x7faef00cabc0 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef00b0bd0 con 0x7faef015a540 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faef6200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef015ac80 con 0x7faef0070cc0 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58652/0 (socket says 172.21.15.28:58652) 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 -- 172.21.15.28:0/1469478111 learned_addr learned my addr 172.21.15.28:0/1469478111 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faeef400640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef01f3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faeefe00640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 -- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 msgr2=0x7faef01f3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef01f3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 -- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 msgr2=0x7faef015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.516+0000 7faee6a00640 1 -- 172.21.15.28:0/1469478111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee0009400 con 0x7faef00cabc0 2024-09-13T17:16:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faee6a00640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7faed80011e0 tx=0x7faed800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8005ad0 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed8005c50 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef015af60 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed8007070 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.517+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef015b520 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.518+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef015b740 con 0x7faef00cabc0 2024-09-13T17:16:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.519+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faed80071d0 con 0x7faef00cabc0 2024-09-13T17:16:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.520+0000 7faee7e00640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 0x7faecc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.520+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faed8088a00 con 0x7faef00cabc0 2024-09-13T17:16:15.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.521+0000 7faeef400640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 0x7faecc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.521+0000 7faeef400640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 0x7faecc06d0c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faee0034000 tx=0x7faee002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.524+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faed8053c60 con 0x7faef00cabc0 2024-09-13T17:16:15.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.740+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faef0071130 con 0x7faef00cabc0 2024-09-13T17:16:15.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.740+0000 7faee7e00640 1 -- 172.21.15.28:0/1469478111 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4745 (secure 0 0 0) 0x7faed8017020 con 0x7faef00cabc0 2024-09-13T17:16:15.742 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:15.742 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:16:04.081667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:16:15.742 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 msgr2=0x7faecc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 0x7faecc06d0c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faee0034000 tx=0x7faee002e000 comp rx=0 tx=0).stop 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 msgr2=0x7faef00b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7faed80011e0 tx=0x7faed800cb90 comp rx=0 tx=0).stop 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 shutdown_connections 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faecc06ac00 0x7faecc06d0c0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faef015a540 0x7faef015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faef00cabc0 0x7faef00b0690 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 --2- 172.21.15.28:0/1469478111 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faef0070cc0 0x7faef01f3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 >> 172.21.15.28:0/1469478111 conn(0x7faef0069e80 msgr2=0x7faef00cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.742+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 shutdown_connections 2024-09-13T17:16:15.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.743+0000 7faef6200640 1 -- 172.21.15.28:0/1469478111 wait complete. 2024-09-13T17:16:15.759 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.914+0000 7f614d800640 1 Processor -- start 2024-09-13T17:16:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.914+0000 7f614d800640 1 -- start start 2024-09-13T17:16:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.915+0000 7f614d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f614806f970 0x7f614815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.915+0000 7f614d800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f614806c2d0 0x7f614806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.915+0000 7f614d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f614806cc70 con 0x7f61481592a0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.915+0000 7f614d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f614815b0b0 con 0x7f614806f970 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.915+0000 7f614d800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6148159d90 con 0x7f614806c2d0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f614806f970 0x7f614815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f61481592a0 0x7f6148159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39954/0 (socket says 172.21.15.28:39954) 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146a00640 1 -- 172.21.15.28:0/3669618692 learned_addr learned my addr 172.21.15.28:0/3669618692 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6147400640 1 --2- 172.21.15.28:0/3669618692 >> v2:172.21.15.189:3300/0 conn(0x7f614806c2d0 0x7f614806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146000640 1 -- 172.21.15.28:0/3669618692 >> v2:172.21.15.189:3300/0 conn(0x7f614806c2d0 msgr2=0x7f614806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146000640 1 --2- 172.21.15.28:0/3669618692 >> v2:172.21.15.189:3300/0 conn(0x7f614806c2d0 0x7f614806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146000640 1 -- 172.21.15.28:0/3669618692 >> v1:172.21.15.28:6789/0 conn(0x7f61481592a0 legacy=0x7f6148159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6146000640 1 -- 172.21.15.28:0/3669618692 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61480b2040 con 0x7f614806f970 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.916+0000 7f6145600640 1 -- 172.21.15.28:0/3669618692 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123680786 0 0) 0x7f614806cc70 con 0x7f61481592a0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.917+0000 7f6146000640 1 --2- 172.21.15.28:0/3669618692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f614815ab70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f61480b1dd0 tx=0x7f613402c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72b70394cfbe47b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.917+0000 7f6145600640 1 -- 172.21.15.28:0/3669618692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134032770 con 0x7f614806f970 2024-09-13T17:16:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.917+0000 7f6145600640 1 -- 172.21.15.28:0/3669618692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61340328d0 con 0x7f614806f970 2024-09-13T17:16:15.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.917+0000 7f6145600640 1 -- 172.21.15.28:0/3669618692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134032c60 con 0x7f614806f970 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 -- 172.21.15.28:0/3669618692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 msgr2=0x7f614815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 --2- 172.21.15.28:0/3669618692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f614815ab70 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f61480b1dd0 tx=0x7f613402c870 comp rx=0 tx=0).stop 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 -- 172.21.15.28:0/3669618692 shutdown_connections 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 --2- 172.21.15.28:0/3669618692 >> v2:172.21.15.189:3300/0 conn(0x7f614806c2d0 0x7f614806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 --2- 172.21.15.28:0/3669618692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f614815ab70 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 -- 172.21.15.28:0/3669618692 >> 172.21.15.28:0/3669618692 conn(0x7f6148069e80 msgr2=0x7f614806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 -- 172.21.15.28:0/3669618692 shutdown_connections 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.918+0000 7f614d800640 1 -- 172.21.15.28:0/3669618692 wait complete. 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.920+0000 7f614d800640 1 Processor -- start 2024-09-13T17:16:15.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.920+0000 7f614d800640 1 -- start start 2024-09-13T17:16:15.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614806c2d0 0x7f614814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 0x7f61480acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61481577f0 con 0x7f614806f970 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614814fb00 con 0x7f614806c2d0 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f614d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61480cc130 con 0x7f61481592a0 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58670/0 (socket says 172.21.15.28:58670) 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 -- 172.21.15.28:0/1744485044 learned_addr learned my addr 172.21.15.28:0/1744485044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146a00640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614806c2d0 0x7f614814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6147400640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 0x7f61480acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 -- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 msgr2=0x7f61480acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 0x7f61480acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 -- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614806c2d0 msgr2=0x7f614814f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.921+0000 7f6146000640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614806c2d0 0x7f614814f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f6146000640 1 -- 172.21.15.28:0/1744485044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613400b6a0 con 0x7f614806f970 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f6147400640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 0x7f61480acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f6146000640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6134035040 tx=0x7f6134004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134033c40 con 0x7f614806f970 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6134033da0 con 0x7f614806f970 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6134004250 con 0x7f614806f970 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61480cc410 con 0x7f614806f970 2024-09-13T17:16:15.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.922+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61480cc9d0 con 0x7f614806f970 2024-09-13T17:16:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.923+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61480b3740 con 0x7f614806f970 2024-09-13T17:16:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.924+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f61340043b0 con 0x7f614806f970 2024-09-13T17:16:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.925+0000 7f613ea00640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 0x7f612806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:15.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.925+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f61340aa5f0 con 0x7f614806f970 2024-09-13T17:16:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.926+0000 7f6146a00640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 0x7f612806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:15.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.926+0000 7f6146a00640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 0x7f612806d0c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6138007fd0 tx=0x7f6138007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:15.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:15.929+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6134075970 con 0x7f614806f970 2024-09-13T17:16:16.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.143+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6148158870 con 0x7f614806f970 2024-09-13T17:16:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.144+0000 7f613ea00640 1 -- 172.21.15.28:0/1744485044 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+4745 (secure 0 0 0) 0x7f6134006e10 con 0x7f614806f970 2024-09-13T17:16:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:16.145 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":106,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2024-09-13T17:14:31.526446+0000","modified":"2024-09-13T17:16:04.081667+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":90,"state":"up:active","state_seq":26,"addr":"172.21.15.28:6835/4107617548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":4107617548},{"type":"v1","addr":"172.21.15.28:6835","nonce":4107617548}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T17:16:16.145 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 106 2024-09-13T17:16:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 msgr2=0x7f612806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 0x7f612806d0c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f6138007fd0 tx=0x7f6138007de0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 msgr2=0x7f61480ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f6134035040 tx=0x7f6134004730 comp rx=0 tx=0).stop 2024-09-13T17:16:16.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 shutdown_connections 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f612806ac00 0x7f612806d0c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61481592a0 0x7f61480acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f614806f970 0x7f61480ac850 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 --2- 172.21.15.28:0/1744485044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614806c2d0 0x7f614814f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 >> 172.21.15.28:0/1744485044 conn(0x7f6148069e80 msgr2=0x7f614806b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 shutdown_connections 2024-09-13T17:16:16.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.146+0000 7f614d800640 1 -- 172.21.15.28:0/1744485044 wait complete. 2024-09-13T17:16:16.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.317+0000 7f1ac8a00640 1 Processor -- start 2024-09-13T17:16:16.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.317+0000 7f1ac8a00640 1 -- start start 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac8a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1ac406f970 0x7f1ac415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1ac406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac8a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac406cc70 con 0x7f1ac41592a0 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac8a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac415b0b0 con 0x7f1ac406f970 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4159d90 con 0x7f1ac406c2d0 2024-09-13T17:16:16.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1ac41592a0 0x7f1ac4159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39966/0 (socket says 172.21.15.28:39966) 2024-09-13T17:16:16.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.318+0000 7f1ac1e00640 1 -- 172.21.15.28:0/1303285614 learned_addr learned my addr 172.21.15.28:0/1303285614 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831374837 0 0) 0x7f1ac406cc70 con 0x7f1ac41592a0 2024-09-13T17:16:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa8003520 con 0x7f1ac41592a0 2024-09-13T17:16:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac2800640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1ac406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac1400640 1 --2- 172.21.15.28:0/1303285614 >> v2:172.21.15.189:3300/0 conn(0x7f1ac406f970 0x7f1ac415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262435582 0 0) 0x7f1aa8003520 con 0x7f1ac41592a0 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 msgr2=0x7f1ac406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1ac406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 >> v2:172.21.15.189:3300/0 conn(0x7f1ac406f970 msgr2=0x7f1ac415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 --2- 172.21.15.28:0/1303285614 >> v2:172.21.15.189:3300/0 conn(0x7f1ac406f970 0x7f1ac415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40b2040 con 0x7f1ac41592a0 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f1ab4001690 con 0x7f1ac41592a0 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac1400640 1 --2- 172.21.15.28:0/1303285614 >> v2:172.21.15.189:3300/0 conn(0x7f1ac406f970 0x7f1ac415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 >> v1:172.21.15.28:6789/0 conn(0x7f1ac41592a0 legacy=0x7f1ac4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1aa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1aa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.319+0000 7f1ac0a00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa8004d00 0x7f1aa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac415b0b0 con 0x7f1aa8003970 2024-09-13T17:16:16.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4159d90 con 0x7f1aa8004d00 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa800dbc0 con 0x7f1ac406c2d0 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1aa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1400640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1aa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac2800640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa8004d00 0x7f1aa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 -- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 msgr2=0x7f1aa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac406c2d0 0x7f1aa80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 -- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa8004d00 msgr2=0x7f1aa8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa8004d00 0x7f1aa8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.320+0000 7f1ac1e00640 1 -- 172.21.15.28:0/1303285614 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa8003520 con 0x7f1aa8003970 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.321+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1aa8003d90 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400c210 tx=0x7f1aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614c33db9e91fbff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab402e070 con 0x7f1aa8003970 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab40058d0 con 0x7f1aa8003970 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac0a00640 1 -- 172.21.15.28:0/1303285614 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4005c80 con 0x7f1aa8003970 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 -- 172.21.15.28:0/1303285614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 msgr2=0x7f1aa8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/1303285614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1aa8003d90 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400c210 tx=0x7f1aa80057f0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac2800640 1 -- 172.21.15.28:0/1303285614 reap_dead start 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 -- 172.21.15.28:0/1303285614 shutdown_connections 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 -- 172.21.15.28:0/1303285614 >> 172.21.15.28:0/1303285614 conn(0x7f1ac4069e80 msgr2=0x7f1ac406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 -- 172.21.15.28:0/1303285614 shutdown_connections 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.322+0000 7f1ac8a00640 1 -- 172.21.15.28:0/1303285614 wait complete. 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.324+0000 7f1ac8a00640 1 Processor -- start 2024-09-13T17:16:16.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.324+0000 7f1ac8a00640 1 -- start start 2024-09-13T17:16:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac40c9ff0 0x7f1ac4151400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac4155050 0x7f1ac4151c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4151940 con 0x7f1aa8003970 2024-09-13T17:16:16.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4155bf0 con 0x7f1ac4155050 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4155910 con 0x7f1ac40c9ff0 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58684/0 (socket says 172.21.15.28:58684) 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac1e00640 1 -- 172.21.15.28:0/3288873402 learned_addr learned my addr 172.21.15.28:0/3288873402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac2800640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac4155050 0x7f1ac4151c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.325+0000 7f1ac1400640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac40c9ff0 0x7f1ac4151400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 -- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac40c9ff0 msgr2=0x7f1ac4151400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac40c9ff0 0x7f1ac4151400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 -- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac4155050 msgr2=0x7f1ac4151c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac4155050 0x7f1ac4151c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 -- 172.21.15.28:0/3288873402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac41f2a40 con 0x7f1aa8003970 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1ac1e00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40058a0 tx=0x7f1ab4001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4005e70 con 0x7f1aa8003970 2024-09-13T17:16:16.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.326+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4001c70 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.327+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac41f2d20 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.327+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab40074a0 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.327+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac4155500 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.328+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac41556c0 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.328+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ab4007600 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.329+0000 7f1aba000640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 0x7f1aa406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.329+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1ab40a92b0 con 0x7f1aa8003970 2024-09-13T17:16:16.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.330+0000 7f1ac1400640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 0x7f1aa406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.331+0000 7f1ac1400640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 0x7f1aa406d0c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ca430 tx=0x7f1ab0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.334+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1ab4074480 con 0x7f1aa8003970 2024-09-13T17:16:16.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.515+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac41f2870 con 0x7f1aa8003970 2024-09-13T17:16:16.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.517+0000 7f1aba000640 1 -- 172.21.15.28:0/3288873402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16586 (secure 0 0 0) 0x7f1ab4077aa0 con 0x7f1aa8003970 2024-09-13T17:16:16.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:16.518 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":113,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:15:45.960852+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:14:29.435536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-09-13T17:14:30.524696+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"112","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":112,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 msgr2=0x7f1aa406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 0x7f1aa406d0c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40ca430 tx=0x7f1ab0008040 comp rx=0 tx=0).stop 2024-09-13T17:16:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 msgr2=0x7f1ac41f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40058a0 tx=0x7f1ab4001a20 comp rx=0 tx=0).stop 2024-09-13T17:16:16.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 shutdown_connections 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1aa406ac00 0x7f1aa406d0c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ac4155050 0x7f1ac4151c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1ac40c9ff0 0x7f1ac4151400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 --2- 172.21.15.28:0/3288873402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa8003970 0x7f1ac41f3520 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 >> 172.21.15.28:0/3288873402 conn(0x7f1ac4069e80 msgr2=0x7f1ac40b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 shutdown_connections 2024-09-13T17:16:16.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.519+0000 7f1ac8a00640 1 -- 172.21.15.28:0/3288873402 wait complete. 2024-09-13T17:16:16.536 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:16:16.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.692+0000 7f7935600640 1 Processor -- start 2024-09-13T17:16:16.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.692+0000 7f7935600640 1 -- start start 2024-09-13T17:16:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7935600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7930070430 0x7f793015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7935600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f793006f3f0 0x7f793015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7935600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793015ab60 con 0x7f7930159770 2024-09-13T17:16:16.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7935600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793006f920 con 0x7f7930070430 2024-09-13T17:16:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7935600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793015b290 con 0x7f793006f3f0 2024-09-13T17:16:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.693+0000 7f7927e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7930070430 0x7f793015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7930070430 0x7f793015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58694/0 (socket says 172.21.15.28:58694) 2024-09-13T17:16:16.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 -- 172.21.15.28:0/248313020 learned_addr learned my addr 172.21.15.28:0/248313020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f792f400640 1 --2- 172.21.15.28:0/248313020 >> v2:172.21.15.189:3300/0 conn(0x7f793006f3f0 0x7f793015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 -- 172.21.15.28:0/248313020 >> v2:172.21.15.189:3300/0 conn(0x7f793006f3f0 msgr2=0x7f793015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 --2- 172.21.15.28:0/248313020 >> v2:172.21.15.189:3300/0 conn(0x7f793006f3f0 0x7f793015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 -- 172.21.15.28:0/248313020 >> v1:172.21.15.189:6789/0 conn(0x7f7930159770 legacy=0x7f793006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.694+0000 7f7927e00640 1 -- 172.21.15.28:0/248313020 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f793015b570 con 0x7f7930070430 2024-09-13T17:16:16.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.695+0000 7f7927e00640 1 --2- 172.21.15.28:0/248313020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930070430 0x7f793015bd70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f793006f380 tx=0x7f791802c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace0167f244c3c3a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.695+0000 7f792e000640 1 -- 172.21.15.28:0/248313020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7918033640 con 0x7f7930070430 2024-09-13T17:16:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.695+0000 7f792e000640 1 -- 172.21.15.28:0/248313020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791802f070 con 0x7f7930070430 2024-09-13T17:16:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.695+0000 7f792e000640 1 -- 172.21.15.28:0/248313020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180339c0 con 0x7f7930070430 2024-09-13T17:16:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 -- 172.21.15.28:0/248313020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930070430 msgr2=0x7f793015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 --2- 172.21.15.28:0/248313020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930070430 0x7f793015bd70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f793006f380 tx=0x7f791802c8b0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 -- 172.21.15.28:0/248313020 shutdown_connections 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 --2- 172.21.15.28:0/248313020 >> v2:172.21.15.189:3300/0 conn(0x7f793006f3f0 0x7f793015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 --2- 172.21.15.28:0/248313020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930070430 0x7f793015bd70 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 -- 172.21.15.28:0/248313020 >> 172.21.15.28:0/248313020 conn(0x7f79300b0270 msgr2=0x7f79300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 -- 172.21.15.28:0/248313020 shutdown_connections 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.696+0000 7f7935600640 1 -- 172.21.15.28:0/248313020 wait complete. 2024-09-13T17:16:16.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.698+0000 7f7935600640 1 Processor -- start 2024-09-13T17:16:16.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.698+0000 7f7935600640 1 -- start start 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7930070430 0x7f7930153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79300a8650 con 0x7f7930159770 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79301e3b30 con 0x7f793006f3f0 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7935600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7930154180 con 0x7f7930070430 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f792f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f792ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f792f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58706/0 (socket says 172.21.15.28:58706) 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f792ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:34858/0 (socket says 172.21.15.28:34858) 2024-09-13T17:16:16.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f792f400640 1 -- 172.21.15.28:0/2620212236 learned_addr learned my addr 172.21.15.28:0/2620212236 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.699+0000 7f7927e00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7930070430 0x7f7930153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 -- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7930070430 msgr2=0x7f7930153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7930070430 0x7f7930153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 -- 172.21.15.28:0/2620212236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 msgr2=0x7f7930156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 -- 172.21.15.28:0/2620212236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791800b770 con 0x7f793006f3f0 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792f400640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:16.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f792ea00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f792000c390 tx=0x7f792000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79200052d0 con 0x7f793006f3f0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7920005430 con 0x7f793006f3f0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920003830 con 0x7f793006f3f0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79300cc7e0 con 0x7f793006f3f0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.700+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79300ccda0 con 0x7f793006f3f0 2024-09-13T17:16:16.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.701+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793006fd20 con 0x7f793006f3f0 2024-09-13T17:16:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.703+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7920003990 con 0x7f793006f3f0 2024-09-13T17:16:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.703+0000 7f7926a00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 0x7f790c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.703+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f79200888b0 con 0x7f793006f3f0 2024-09-13T17:16:16.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.704+0000 7f7927e00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 0x7f790c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:16.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.704+0000 7f7927e00640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 0x7f790c06d0c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7918037000 tx=0x7f791803e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:16.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.707+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7920053b00 con 0x7f793006f3f0 2024-09-13T17:16:16.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.923+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f793015ab60 con 0x7f793006f3f0 2024-09-13T17:16:16.925 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:16:16.924+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:16:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.936+0000 7f7926a00640 1 -- 172.21.15.28:0/2620212236 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v107) v1 ==== 161+0+0 (secure 0 0 0) 0x7f793015ab60 con 0x7f793006f3f0 2024-09-13T17:16:16.937 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 msgr2=0x7f790c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 0x7f790c06d0c0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f7918037000 tx=0x7f791803e040 comp rx=0 tx=0).stop 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 msgr2=0x7f79301e3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f792000c390 tx=0x7f792000c850 comp rx=0 tx=0).stop 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 shutdown_connections 2024-09-13T17:16:16.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac00 0x7f790c06d0c0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7930159770 0x7f7930156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7930070430 0x7f7930153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 --2- 172.21.15.28:0/2620212236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793006f3f0 0x7f79301e3430 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 >> 172.21.15.28:0/2620212236 conn(0x7f79300b0270 msgr2=0x7f79300b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 shutdown_connections 2024-09-13T17:16:16.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:16.940+0000 7f7935600640 1 -- 172.21.15.28:0/2620212236 wait complete. 2024-09-13T17:16:16.964 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:16:17.015 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:16:17.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.123+0000 7ff22f000640 1 Processor -- start 2024-09-13T17:16:17.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.123+0000 7ff22f000640 1 -- start start 2024-09-13T17:16:17.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff22815c8c0 0x7ff2280cd660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 0x7ff2280b5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2280d2630 con 0x7ff228069e80 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2280d1590 con 0x7ff22815c8c0 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280d1870 con 0x7ff2280b57d0 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 0x7ff2280b5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.124+0000 7ff22dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff22815c8c0 0x7ff2280cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22d200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff228069e80 0x7ff22806a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:41142/0 (socket says 172.21.15.28:41142) 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 0x7ff2280b5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:37450/0 (socket says 172.21.15.28:37450) 2024-09-13T17:16:17.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22d200640 1 -- 172.21.15.28:0/885038283 learned_addr learned my addr 172.21.15.28:0/885038283 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22dc00640 1 -- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 msgr2=0x7ff2280b5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22dc00640 1 --2- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 0x7ff2280b5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22dc00640 1 -- 172.21.15.28:0/885038283 >> v1:172.21.15.189:6789/0 conn(0x7ff228069e80 legacy=0x7ff22806a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff22dc00640 1 -- 172.21.15.28:0/885038283 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280d1b50 con 0x7ff22815c8c0 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.125+0000 7ff227e00640 1 -- 172.21.15.28:0/885038283 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487128884 0 0) 0x7ff214001110 con 0x7ff228069e80 2024-09-13T17:16:17.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.126+0000 7ff22dc00640 1 --2- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22815c8c0 0x7ff2280cd660 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=524da60c066b7e7d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:17.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff227e00640 1 -- 172.21.15.28:0/885038283 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c032440 con 0x7ff22815c8c0 2024-09-13T17:16:17.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff227e00640 1 -- 172.21.15.28:0/885038283 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff21c0325a0 con 0x7ff22815c8c0 2024-09-13T17:16:17.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff227e00640 1 -- 172.21.15.28:0/885038283 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff21c032930 con 0x7ff22815c8c0 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 -- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22815c8c0 msgr2=0x7ff2280cd660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 --2- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22815c8c0 0x7ff2280cd660 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7ff21c0097a0 tx=0x7ff21c02a210 comp rx=0 tx=0).stop 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 -- 172.21.15.28:0/885038283 shutdown_connections 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 --2- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2280b57d0 0x7ff2280b5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 --2- 172.21.15.28:0/885038283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff22815c8c0 0x7ff2280cd660 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.127+0000 7ff22f000640 1 -- 172.21.15.28:0/885038283 >> 172.21.15.28:0/885038283 conn(0x7ff2280708c0 msgr2=0x7ff228070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.128+0000 7ff22f000640 1 -- 172.21.15.28:0/885038283 shutdown_connections 2024-09-13T17:16:17.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.128+0000 7ff22f000640 1 -- 172.21.15.28:0/885038283 wait complete. 2024-09-13T17:16:17.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.129+0000 7ff22f000640 1 Processor -- start 2024-09-13T17:16:17.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 -- start start 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff228069e80 0x7ff2280cca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2280b57d0 0x7ff2281f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280a94d0 con 0x7ff22815c8c0 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280a97b0 con 0x7ff2280b57d0 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.130+0000 7ff22f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2280a9a90 con 0x7ff228069e80 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58740/0 (socket says 172.21.15.28:58740) 2024-09-13T17:16:17.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2280b57d0 0x7ff2281f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff228069e80 0x7ff2280cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 -- 172.21.15.28:0/912896972 learned_addr learned my addr 172.21.15.28:0/912896972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 -- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff228069e80 msgr2=0x7ff2280cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff228069e80 0x7ff2280cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 -- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2280b57d0 msgr2=0x7ff2281f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2280b57d0 0x7ff2281f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.131+0000 7ff22e600640 1 -- 172.21.15.28:0/912896972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c009400 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff22e600640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff22000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220014480 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff22000f040 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220014820 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2280a9d70 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.132+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2280a9180 con 0x7ff22815c8c0 2024-09-13T17:16:17.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.133+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2280aa290 con 0x7ff22815c8c0 2024-09-13T17:16:17.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.134+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff220014980 con 0x7ff22815c8c0 2024-09-13T17:16:17.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.135+0000 7ff225600640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 0x7ff20806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:17.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.135+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff22008a1b0 con 0x7ff22815c8c0 2024-09-13T17:16:17.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.136+0000 7ff22dc00640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 0x7ff20806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:17.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.136+0000 7ff22dc00640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 0x7ff20806d0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff21c034000 tx=0x7ff21c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:17.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.140+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff2280aa290 con 0x7ff22815c8c0 2024-09-13T17:16:17.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.360+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff2280aa290 con 0x7ff22815c8c0 2024-09-13T17:16:17.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.934+0000 7ff225600640 1 -- 172.21.15.28:0/912896972 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff2280aa290 con 0x7ff22815c8c0 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.936+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 msgr2=0x7ff20806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.936+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 0x7ff20806d0c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff21c034000 tx=0x7ff21c02e000 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.936+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 msgr2=0x7ff2280aa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.936+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7ff22000efc0 tx=0x7ff22000c490 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 shutdown_connections 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff20806ac00 0x7ff20806d0c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff22815c8c0 0x7ff2280aa700 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2280b57d0 0x7ff2281f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 --2- 172.21.15.28:0/912896972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff228069e80 0x7ff2280cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 >> 172.21.15.28:0/912896972 conn(0x7ff2280708c0 msgr2=0x7ff2280b72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 shutdown_connections 2024-09-13T17:16:17.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:17.937+0000 7ff22f000640 1 -- 172.21.15.28:0/912896972 wait complete. 2024-09-13T17:16:17.957 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:16:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.117+0000 7fc585600640 1 Processor -- start 2024-09-13T17:16:18.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.117+0000 7fc585600640 1 -- start start 2024-09-13T17:16:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.118+0000 7fc585600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc58006b9c0 0x7fc5800b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.118+0000 7fc585600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.118+0000 7fc585600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5800b3600 con 0x7fc580159440 2024-09-13T17:16:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.118+0000 7fc585600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5800b46a0 con 0x7fc58006b9c0 2024-09-13T17:16:18.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.118+0000 7fc585600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5800b38e0 con 0x7fc5800b2dd0 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc584c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc584c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:37462/0 (socket says 172.21.15.28:37462) 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc577400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc58006b9c0 0x7fc5800b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc584c00640 1 -- 172.21.15.28:0/3292050708 learned_addr learned my addr 172.21.15.28:0/3292050708 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc57fe00640 1 --1- 172.21.15.28:0/3292050708 >> v1:172.21.15.189:6789/0 conn(0x7fc580159440 0x7fc58006b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:41150/0 (socket says 172.21.15.28:41150) 2024-09-13T17:16:18.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc57f400640 1 -- 172.21.15.28:0/3292050708 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623736116 0 0) 0x7fc5800b3600 con 0x7fc580159440 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc57f400640 1 -- 172.21.15.28:0/3292050708 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc564003520 con 0x7fc580159440 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc577400640 1 -- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 msgr2=0x7fc580159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc577400640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc577400640 1 -- 172.21.15.28:0/3292050708 >> v1:172.21.15.189:6789/0 conn(0x7fc580159440 legacy=0x7fc58006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc577400640 1 -- 172.21.15.28:0/3292050708 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5800b3bc0 con 0x7fc58006b9c0 2024-09-13T17:16:18.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.119+0000 7fc584c00640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.120+0000 7fc577400640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc58006b9c0 0x7fc5800b2890 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc5680097a0 tx=0x7fc56802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92e47b256111a85 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc57f400640 1 -- 172.21.15.28:0/3292050708 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568032440 con 0x7fc58006b9c0 2024-09-13T17:16:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc57f400640 1 -- 172.21.15.28:0/3292050708 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5680325a0 con 0x7fc58006b9c0 2024-09-13T17:16:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc57f400640 1 -- 172.21.15.28:0/3292050708 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc568032930 con 0x7fc58006b9c0 2024-09-13T17:16:18.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 -- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc58006b9c0 msgr2=0x7fc5800b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc58006b9c0 0x7fc5800b2890 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fc5680097a0 tx=0x7fc56802a180 comp rx=0 tx=0).stop 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 -- 172.21.15.28:0/3292050708 shutdown_connections 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc5800b2dd0 0x7fc580159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 --2- 172.21.15.28:0/3292050708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc58006b9c0 0x7fc5800b2890 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 -- 172.21.15.28:0/3292050708 >> 172.21.15.28:0/3292050708 conn(0x7fc580069e80 msgr2=0x7fc58006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 -- 172.21.15.28:0/3292050708 shutdown_connections 2024-09-13T17:16:18.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.121+0000 7fc585600640 1 -- 172.21.15.28:0/3292050708 wait complete. 2024-09-13T17:16:18.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.123+0000 7fc585600640 1 Processor -- start 2024-09-13T17:16:18.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.123+0000 7fc585600640 1 -- start start 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58006b9c0 0x7fc580150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5800b2dd0 0x7fc58013ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58014dbf0 con 0x7fc580159440 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580152960 con 0x7fc5800b2dd0 2024-09-13T17:16:18.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc585600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc580151e90 con 0x7fc58006b9c0 2024-09-13T17:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc584c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc584c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58750/0 (socket says 172.21.15.28:58750) 2024-09-13T17:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc584c00640 1 -- 172.21.15.28:0/2990510813 learned_addr learned my addr 172.21.15.28:0/2990510813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc577400640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5800b2dd0 0x7fc58013ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.124+0000 7fc57fe00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58006b9c0 0x7fc580150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 -- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58006b9c0 msgr2=0x7fc580150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58006b9c0 0x7fc580150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 -- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5800b2dd0 msgr2=0x7fc58013ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5800b2dd0 0x7fc58013ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 -- 172.21.15.28:0/2990510813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57800bd40 con 0x7fc580159440 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc584c00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc57800e370 tx=0x7fc57800e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578010040 con 0x7fc580159440 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57800b2d0 con 0x7fc580159440 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc568009400 con 0x7fc580159440 2024-09-13T17:16:18.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc578007e90 con 0x7fc580159440 2024-09-13T17:16:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.125+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5801523a0 con 0x7fc580159440 2024-09-13T17:16:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.126+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580067a10 con 0x7fc580159440 2024-09-13T17:16:18.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.127+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc578009600 con 0x7fc580159440 2024-09-13T17:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.128+0000 7fc57cc00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 0x7fc55c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.128+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc578089c60 con 0x7fc580159440 2024-09-13T17:16:18.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.129+0000 7fc57fe00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 0x7fc55c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:18.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.129+0000 7fc57fe00640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 0x7fc55c06d0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc5700011e0 tx=0x7fc570007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:18.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.135+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc578054df0 con 0x7fc580159440 2024-09-13T17:16:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:18.312+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc58014dd30 con 0x7fc580159440 2024-09-13T17:16:19.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.136+0000 7fc57cc00640 1 -- 172.21.15.28:0/2990510813 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc57801a020 con 0x7fc580159440 2024-09-13T17:16:19.140 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.140+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 msgr2=0x7fc55c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.140+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 0x7fc55c06d0c0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc5700011e0 tx=0x7fc570007d30 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.140+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 msgr2=0x7fc580151950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.140+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc57800e370 tx=0x7fc57800e830 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 shutdown_connections 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc55c06ac00 0x7fc55c06d0c0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc580159440 0x7fc580151950 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5800b2dd0 0x7fc58013ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 --2- 172.21.15.28:0/2990510813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc58006b9c0 0x7fc580150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 >> 172.21.15.28:0/2990510813 conn(0x7fc580069e80 msgr2=0x7fc5800ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 shutdown_connections 2024-09-13T17:16:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.141+0000 7fc585600640 1 -- 172.21.15.28:0/2990510813 wait complete. 2024-09-13T17:16:19.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:16:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.327+0000 7f3279400640 1 Processor -- start 2024-09-13T17:16:19.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.327+0000 7f3279400640 1 -- start start 2024-09-13T17:16:19.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3279400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3274157000 0x7f327406b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:19.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3279400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327415d950 con 0x7f327415cdd0 2024-09-13T17:16:19.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3279400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327415dc30 con 0x7f327406b6b0 2024-09-13T17:16:19.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3279400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32740b1b80 con 0x7f3274157000 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3272800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3274157000 0x7f327406b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.328+0000 7f3273200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f327415cdd0 0x7f32740ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39986/0 (socket says 172.21.15.28:39986) 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.329+0000 7f3273200640 1 -- 172.21.15.28:0/2277376331 learned_addr learned my addr 172.21.15.28:0/2277376331 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.329+0000 7f3272800640 1 -- 172.21.15.28:0/2277376331 >> v1:172.21.15.189:6789/0 conn(0x7f327406b6b0 legacy=0x7f32740b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.329+0000 7f3272800640 1 -- 172.21.15.28:0/2277376331 >> v1:172.21.15.28:6789/0 conn(0x7f327415cdd0 legacy=0x7f32740ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.329+0000 7f3272800640 1 -- 172.21.15.28:0/2277376331 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32740b1e60 con 0x7f3274157000 2024-09-13T17:16:19.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.329+0000 7f3271400640 1 -- 172.21.15.28:0/2277376331 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281654240 0 0) 0x7f327415d950 con 0x7f327415cdd0 2024-09-13T17:16:19.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.330+0000 7f3272800640 1 --2- 172.21.15.28:0/2277376331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3274157000 0x7f327406b170 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f32640097a0 tx=0x7f326402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c1f870ba350d83 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:19.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.330+0000 7f3271400640 1 -- 172.21.15.28:0/2277376331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264032440 con 0x7f3274157000 2024-09-13T17:16:19.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.330+0000 7f3271400640 1 -- 172.21.15.28:0/2277376331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32640325a0 con 0x7f3274157000 2024-09-13T17:16:19.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.330+0000 7f3271400640 1 -- 172.21.15.28:0/2277376331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264032930 con 0x7f3274157000 2024-09-13T17:16:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 -- 172.21.15.28:0/2277376331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3274157000 msgr2=0x7f327406b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 --2- 172.21.15.28:0/2277376331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3274157000 0x7f327406b170 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f32640097a0 tx=0x7f326402a230 comp rx=0 tx=0).stop 2024-09-13T17:16:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 -- 172.21.15.28:0/2277376331 shutdown_connections 2024-09-13T17:16:19.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 --2- 172.21.15.28:0/2277376331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3274157000 0x7f327406b170 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 -- 172.21.15.28:0/2277376331 >> 172.21.15.28:0/2277376331 conn(0x7f3274069e80 msgr2=0x7f327406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 -- 172.21.15.28:0/2277376331 shutdown_connections 2024-09-13T17:16:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.331+0000 7f3279400640 1 -- 172.21.15.28:0/2277376331 wait complete. 2024-09-13T17:16:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.333+0000 7f3279400640 1 Processor -- start 2024-09-13T17:16:19.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.333+0000 7f3279400640 1 -- start start 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f327406b6b0 0x7f3274146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3274157000 0x7f32740cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32740cd390 con 0x7f327415cdd0 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3274147460 con 0x7f327406b6b0 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3279400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32740cc780 con 0x7f3274157000 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58760/0 (socket says 172.21.15.28:58760) 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 -- 172.21.15.28:0/2546017589 learned_addr learned my addr 172.21.15.28:0/2546017589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3271e00640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3274157000 0x7f32740cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3272800640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f327406b6b0 0x7f3274146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 -- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3274157000 msgr2=0x7f32740cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3274157000 0x7f32740cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 -- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f327406b6b0 msgr2=0x7f3274146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f327406b6b0 0x7f3274146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.334+0000 7f3273200640 1 -- 172.21.15.28:0/2546017589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3264009400 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f3273200640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c780 tx=0x7f326c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00ce90 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c00a640 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00a9d0 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32740cca60 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.335+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32740cd020 con 0x7f327415cdd0 2024-09-13T17:16:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.336+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32740cd540 con 0x7f327415cdd0 2024-09-13T17:16:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.337+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f326c00ab30 con 0x7f327415cdd0 2024-09-13T17:16:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.338+0000 7f326aa00640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 0x7f325406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.338+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f326c088720 con 0x7f327415cdd0 2024-09-13T17:16:19.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.338+0000 7f3272800640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 0x7f325406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:19.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.339+0000 7f3272800640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 0x7f325406d0c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3264034000 tx=0x7f326402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:19.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.342+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32740cd540 con 0x7f327415cdd0 2024-09-13T17:16:19.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:19.534+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f327406b170 con 0x7f327415cdd0 2024-09-13T17:16:20.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.144+0000 7f326aa00640 1 -- 172.21.15.28:0/2546017589 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) v1 ==== 169+0+0 (secure 0 0 0) 0x7f326c018020 con 0x7f327415cdd0 2024-09-13T17:16:20.147 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:16:20.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.148+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 msgr2=0x7f325406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.148+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 0x7f325406d0c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3264034000 tx=0x7f326402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:20.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.148+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 msgr2=0x7f32741573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.148+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f326c00c780 tx=0x7f326c00cc40 comp rx=0 tx=0).stop 2024-09-13T17:16:20.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.148+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 shutdown_connections 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f325406ac00 0x7f325406d0c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f327415cdd0 0x7f32741573e0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3274157000 0x7f32740cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 --2- 172.21.15.28:0/2546017589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f327406b6b0 0x7f3274146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 >> 172.21.15.28:0/2546017589 conn(0x7f3274069e80 msgr2=0x7f327415c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 shutdown_connections 2024-09-13T17:16:20.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.149+0000 7f3279400640 1 -- 172.21.15.28:0/2546017589 wait complete. 2024-09-13T17:16:20.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:16:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.349+0000 7f77ffe00640 1 Processor -- start 2024-09-13T17:16:20.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.349+0000 7f77ffe00640 1 -- start start 2024-09-13T17:16:20.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7800159290 0x7f78000b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 0x7f7800070880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 0x7f7800070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f780006c7e0 con 0x7f780015cfe0 2024-09-13T17:16:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f780006cdb0 con 0x7f7800159290 2024-09-13T17:16:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ffe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780015d4e0 con 0x7f78000cc510 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77ff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 0x7f7800070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77fe000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 0x7f7800070880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77fea00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7800159290 0x7f78000b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77fe000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 0x7f7800070880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:37490/0 (socket says 172.21.15.28:37490) 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.352+0000 7f77fe000640 1 -- 172.21.15.28:0/3577925507 learned_addr learned my addr 172.21.15.28:0/3577925507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77fea00640 1 -- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 msgr2=0x7f7800070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77fea00640 1 --2- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 0x7f7800070880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77fea00640 1 -- 172.21.15.28:0/3577925507 >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 msgr2=0x7f7800070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77fea00640 1 --2- 172.21.15.28:0/3577925507 >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 0x7f7800070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77fea00640 1 -- 172.21.15.28:0/3577925507 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780015d7c0 con 0x7f7800159290 2024-09-13T17:16:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.353+0000 7f77ff400640 1 --2- 172.21.15.28:0/3577925507 >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 0x7f7800070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.354+0000 7f77fea00640 1 --2- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7800159290 0x7f78000b24f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f780006ba00 tx=0x7f77f802e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=978fe748f2f59d1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.354+0000 7f77fd600640 1 -- 172.21.15.28:0/3577925507 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f800d4e0 con 0x7f7800159290 2024-09-13T17:16:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.354+0000 7f77fd600640 1 -- 172.21.15.28:0/3577925507 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f800b740 con 0x7f7800159290 2024-09-13T17:16:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.354+0000 7f77fd600640 1 -- 172.21.15.28:0/3577925507 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f800bad0 con 0x7f7800159290 2024-09-13T17:16:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 -- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7800159290 msgr2=0x7f78000b24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 --2- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7800159290 0x7f78000b24f0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f780006ba00 tx=0x7f77f802e9e0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 -- 172.21.15.28:0/3577925507 shutdown_connections 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 --2- 172.21.15.28:0/3577925507 >> v2:172.21.15.28:3300/0 conn(0x7f780015cfe0 0x7f7800070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 --2- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78000cc510 0x7f7800070880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 --2- 172.21.15.28:0/3577925507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7800159290 0x7f78000b24f0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 -- 172.21.15.28:0/3577925507 >> 172.21.15.28:0/3577925507 conn(0x7f78000b1610 msgr2=0x7f78000b1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 -- 172.21.15.28:0/3577925507 shutdown_connections 2024-09-13T17:16:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.355+0000 7f77ffe00640 1 -- 172.21.15.28:0/3577925507 wait complete. 2024-09-13T17:16:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.357+0000 7f77ffe00640 1 Processor -- start 2024-09-13T17:16:20.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.357+0000 7f77ffe00640 1 -- start start 2024-09-13T17:16:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78000cc510 0x7f7800144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7800159290 0x7f780015bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800156a50 con 0x7f780015cfe0 2024-09-13T17:16:20.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7800144dd0 con 0x7f78000cc510 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ffe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78000ae010 con 0x7f7800159290 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58784/0 (socket says 172.21.15.28:58784) 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77ff400640 1 -- 172.21.15.28:0/2517887602 learned_addr learned my addr 172.21.15.28:0/2517887602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77fe000640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7800159290 0x7f780015bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.358+0000 7f77fea00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78000cc510 0x7f7800144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 -- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7800159290 msgr2=0x7f780015bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7800159290 0x7f780015bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 -- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78000cc510 msgr2=0x7f7800144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78000cc510 0x7f7800144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 -- 172.21.15.28:0/2517887602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec009400 con 0x7f780015cfe0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ff400640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00e850 tx=0x7f77ec00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec009640 con 0x7f780015cfe0 2024-09-13T17:16:20.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ec00c940 con 0x7f780015cfe0 2024-09-13T17:16:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f800f050 con 0x7f780015cfe0 2024-09-13T17:16:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.359+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ec00ccb0 con 0x7f780015cfe0 2024-09-13T17:16:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.360+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78000ae5d0 con 0x7f780015cfe0 2024-09-13T17:16:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.361+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78000ae7f0 con 0x7f780015cfe0 2024-09-13T17:16:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.361+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f77ec013470 con 0x7f780015cfe0 2024-09-13T17:16:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.362+0000 7f77f7400640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 0x7f77e006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.362+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f77ec0892e0 con 0x7f780015cfe0 2024-09-13T17:16:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.363+0000 7f77fea00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 0x7f77e006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.364+0000 7f77fea00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 0x7f77e006d0f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f77f8038000 tx=0x7f77f8040040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.367+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f77ec054740 con 0x7f780015cfe0 2024-09-13T17:16:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.548+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f78000cbc10 con 0x7f780015cfe0 2024-09-13T17:16:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.552+0000 7f77f7400640 1 -- 172.21.15.28:0/2517887602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f77ec057d60 con 0x7f780015cfe0 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.554+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 msgr2=0x7f77e006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.554+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 0x7f77e006d0f0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f77f8038000 tx=0x7f77f8040040 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.554+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 msgr2=0x7f780015c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.554+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00e850 tx=0x7f77ec00ed10 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 shutdown_connections 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f77e006ac30 0x7f77e006d0f0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f780015cfe0 0x7f780015c000 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7800159290 0x7f780015bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 --2- 172.21.15.28:0/2517887602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78000cc510 0x7f7800144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 >> 172.21.15.28:0/2517887602 conn(0x7f78000b1610 msgr2=0x7f78000af350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 shutdown_connections 2024-09-13T17:16:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.555+0000 7f77ffe00640 1 -- 172.21.15.28:0/2517887602 wait complete. 2024-09-13T17:16:20.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub' 2024-09-13T17:16:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.728+0000 7f18bd200640 1 Processor -- start 2024-09-13T17:16:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.728+0000 7f18bd200640 1 -- start start 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18bd200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 0x7f18b80b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18bd200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b80c3850 con 0x7f18b8159820 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18bd200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b815a930 con 0x7f18b8159440 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18bd200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8159d20 con 0x7f18b80b0b50 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18b7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18b8159820 0x7f18b815adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-09-13T17:16:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.729+0000 7f18b7e00640 1 -- 172.21.15.28:0/3453803680 learned_addr learned my addr 172.21.15.28:0/3453803680 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6a00640 1 --2- 172.21.15.28:0/3453803680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 0x7f18b80b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289380017 0 0) 0x7f18b80c3850 con 0x7f18b8159820 2024-09-13T17:16:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f189c003520 con 0x7f18b8159820 2024-09-13T17:16:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749927022 0 0) 0x7f18b815a930 con 0x7f18b8159440 2024-09-13T17:16:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6a00640 1 -- 172.21.15.28:0/3453803680 >> v1:172.21.15.189:6789/0 conn(0x7f18b8159440 legacy=0x7f18b80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6a00640 1 -- 172.21.15.28:0/3453803680 >> v1:172.21.15.28:6789/0 conn(0x7f18b8159820 legacy=0x7f18b815adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.730+0000 7f18b6a00640 1 -- 172.21.15.28:0/3453803680 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b815a000 con 0x7f18b80b0b50 2024-09-13T17:16:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.732+0000 7f18b6a00640 1 --2- 172.21.15.28:0/3453803680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 0x7f18b80b22e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097d0 tx=0x7f18a402a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2b397d1ee9ccb23 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.732+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032440 con 0x7f18b80b0b50 2024-09-13T17:16:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.732+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a40325a0 con 0x7f18b80b0b50 2024-09-13T17:16:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.732+0000 7f18b6000640 1 -- 172.21.15.28:0/3453803680 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032910 con 0x7f18b80b0b50 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 -- 172.21.15.28:0/3453803680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 msgr2=0x7f18b80b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 --2- 172.21.15.28:0/3453803680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 0x7f18b80b22e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097d0 tx=0x7f18a402a290 comp rx=0 tx=0).stop 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 -- 172.21.15.28:0/3453803680 shutdown_connections 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 --2- 172.21.15.28:0/3453803680 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b80b0b50 0x7f18b80b22e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 -- 172.21.15.28:0/3453803680 >> 172.21.15.28:0/3453803680 conn(0x7f18b8072e40 msgr2=0x7f18b8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 -- 172.21.15.28:0/3453803680 shutdown_connections 2024-09-13T17:16:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.733+0000 7f18bd200640 1 -- 172.21.15.28:0/3453803680 wait complete. 2024-09-13T17:16:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.734+0000 7f18bd200640 1 Processor -- start 2024-09-13T17:16:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 -- start start 2024-09-13T17:16:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b8159440 0x7f18b81586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8159820 0x7f18b81f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80ad8a0 con 0x7f18b80b0b50 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8156a70 con 0x7f18b8159820 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.735+0000 7f18bd200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8156d50 con 0x7f18b8159440 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58794/0 (socket says 172.21.15.28:58794) 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 -- 172.21.15.28:0/291100392 learned_addr learned my addr 172.21.15.28:0/291100392 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7e00640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8159820 0x7f18b81f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b6a00640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b8159440 0x7f18b81586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 -- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b8159440 msgr2=0x7f18b81586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b8159440 0x7f18b81586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 -- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8159820 msgr2=0x7f18b81f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8159820 0x7f18b81f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 -- 172.21.15.28:0/291100392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a4009400 con 0x7f18b80b0b50 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.736+0000 7f18b7400640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f18a800c4b0 tx=0x7f18a800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.737+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8005480 con 0x7f18b80b0b50 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.737+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a80035f0 con 0x7f18b80b0b50 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.737+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80039e0 con 0x7f18b80b0b50 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.737+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b8157030 con 0x7f18b80b0b50 2024-09-13T17:16:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.737+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8158c30 con 0x7f18b80b0b50 2024-09-13T17:16:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.738+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b806bfe0 con 0x7f18b80b0b50 2024-09-13T17:16:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.739+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f18a8003b40 con 0x7f18b80b0b50 2024-09-13T17:16:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.740+0000 7f18af400640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 0x7f189806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:20.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.740+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f18a80885a0 con 0x7f18b80b0b50 2024-09-13T17:16:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.740+0000 7f18b6a00640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 0x7f189806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.741+0000 7f18b6a00640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 0x7f189806d0c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:20.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.744+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f18a8053b20 con 0x7f18b80b0b50 2024-09-13T17:16:20.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:20.987+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]} v 0) v1 -- 0x7f18b806bd00 con 0x7f18b80b0b50 2024-09-13T17:16:21.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.146+0000 7f18af400640 1 -- 172.21.15.28:0/291100392 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubChecks.test_stray_evaluation_with_scrub"]}]=0 v421) v1 ==== 158+0+0 (secure 0 0 0) 0x7f18a8017020 con 0x7f18b80b0b50 2024-09-13T17:16:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 msgr2=0x7f189806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 0x7f189806d0c0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 msgr2=0x7f18b8156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f18a800c4b0 tx=0x7f18a800c970 comp rx=0 tx=0).stop 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 shutdown_connections 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f189806ac00 0x7f189806d0c0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18b8159820 0x7f18b81f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18b8159440 0x7f18b81586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 --2- 172.21.15.28:0/291100392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18b80b0b50 0x7f18b8156490 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 >> 172.21.15.28:0/291100392 conn(0x7f18b8072e40 msgr2=0x7f18b815adc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 shutdown_connections 2024-09-13T17:16:21.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.148+0000 7f18bd200640 1 -- 172.21.15.28:0/291100392 wait complete. 2024-09-13T17:16:21.166 INFO:tasks.cephfs_test_runner:test that scrub can iterate over ~mdsdir and evaluate strays ... ok 2024-09-13T17:16:21.166 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-09-13T17:16:21.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.320+0000 7f9f43e00640 1 Processor -- start 2024-09-13T17:16:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.320+0000 7f9f43e00640 1 -- start start 2024-09-13T17:16:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.321+0000 7f9f43e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f44154a90 0x7f9f4415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 0x7f9f440af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f4415bb10 con 0x7f9f4406b360 2024-09-13T17:16:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f4406b900 con 0x7f9f44154a90 2024-09-13T17:16:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4406bbe0 con 0x7f9f440b0b50 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f44154a90 0x7f9f4415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f44154a90 0x7f9f4415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58808/0 (socket says 172.21.15.28:58808) 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 -- 172.21.15.28:0/124605279 learned_addr learned my addr 172.21.15.28:0/124605279 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 -- 172.21.15.28:0/124605279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 msgr2=0x7f9f440af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43400640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 0x7f9f440af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 0x7f9f440af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 -- 172.21.15.28:0/124605279 >> v1:172.21.15.189:6789/0 conn(0x7f9f4406b360 legacy=0x7f9f440b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f42a00640 1 -- 172.21.15.28:0/124605279 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4406bec0 con 0x7f9f44154a90 2024-09-13T17:16:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.322+0000 7f9f43400640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 0x7f9f440af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:21.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.323+0000 7f9f42a00640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415b5d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9f440b3d70 tx=0x7f9f3402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd4199dd46b5a1d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f42000640 1 -- 172.21.15.28:0/124605279 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34032440 con 0x7f9f44154a90 2024-09-13T17:16:21.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f42000640 1 -- 172.21.15.28:0/124605279 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f340325a0 con 0x7f9f44154a90 2024-09-13T17:16:21.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f42000640 1 -- 172.21.15.28:0/124605279 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f34032930 con 0x7f9f44154a90 2024-09-13T17:16:21.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 -- 172.21.15.28:0/124605279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 msgr2=0x7f9f4415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415b5d0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f9f440b3d70 tx=0x7f9f3402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 -- 172.21.15.28:0/124605279 shutdown_connections 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f440b0b50 0x7f9f440af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 --2- 172.21.15.28:0/124605279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415b5d0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 -- 172.21.15.28:0/124605279 >> 172.21.15.28:0/124605279 conn(0x7f9f44072e40 msgr2=0x7f9f44073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 -- 172.21.15.28:0/124605279 shutdown_connections 2024-09-13T17:16:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.324+0000 7f9f43e00640 1 -- 172.21.15.28:0/124605279 wait complete. 2024-09-13T17:16:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.326+0000 7f9f43e00640 1 Processor -- start 2024-09-13T17:16:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.326+0000 7f9f43e00640 1 -- start start 2024-09-13T17:16:21.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f4406b360 0x7f9f44153ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f440b0b50 0x7f9f440ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f440b3360 con 0x7f9f44154a90 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f440b37f0 con 0x7f9f440b0b50 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f440b2750 con 0x7f9f4406b360 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58812/0 (socket says 172.21.15.28:58812) 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f43400640 1 -- 172.21.15.28:0/2642246035 learned_addr learned my addr 172.21.15.28:0/2642246035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f3be00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f440b0b50 0x7f9f440ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.327+0000 7f9f42a00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f4406b360 0x7f9f44153ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 -- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f4406b360 msgr2=0x7f9f44153ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f4406b360 0x7f9f44153ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 -- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f440b0b50 msgr2=0x7f9f440ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f440b0b50 0x7f9f440ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 -- 172.21.15.28:0/2642246035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340094d0 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43400640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00cdd0 tx=0x7f9f3c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00d840 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c00d9a0 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f3c00dd90 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.328+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f440b2a30 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.329+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f440b2ff0 con 0x7f9f44154a90 2024-09-13T17:16:21.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.330+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f440b3510 con 0x7f9f44154a90 2024-09-13T17:16:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.330+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9f3c01d360 con 0x7f9f44154a90 2024-09-13T17:16:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.331+0000 7f9f3aa00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 0x7f9f2006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.331+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9f3c0899c0 con 0x7f9f44154a90 2024-09-13T17:16:21.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.332+0000 7f9f42a00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 0x7f9f2006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.332+0000 7f9f42a00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 0x7f9f2006d0f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9f34034000 tx=0x7f9f3402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.336+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9f440b3510 con 0x7f9f44154a90 2024-09-13T17:16:21.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.517+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f440af4b0 con 0x7f9f44154a90 2024-09-13T17:16:21.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.518+0000 7f9f3aa00640 1 -- 172.21.15.28:0/2642246035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+12996 (secure 0 0 0) 0x7f9f3c01a020 con 0x7f9f44154a90 2024-09-13T17:16:21.521 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:21.521 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:20.136913+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/4107617548":"2024-09-14T17:16:16.924675+0000","172.21.15.28:6832/4107617548":"2024-09-14T17:16:16.924675+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 msgr2=0x7f9f2006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 0x7f9f2006d0f0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9f34034000 tx=0x7f9f3402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 msgr2=0x7f9f4415d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9f3c00cdd0 tx=0x7f9f3c00a710 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 shutdown_connections 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f2006ac30 0x7f9f2006d0f0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f44154a90 0x7f9f4415d230 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f440b0b50 0x7f9f440ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 --2- 172.21.15.28:0/2642246035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f4406b360 0x7f9f44153ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 >> 172.21.15.28:0/2642246035 conn(0x7f9f44072e40 msgr2=0x7f9f44158740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 shutdown_connections 2024-09-13T17:16:21.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.521+0000 7f9f43e00640 1 -- 172.21.15.28:0/2642246035 wait complete. 2024-09-13T17:16:21.539 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:16:21.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:16:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.698+0000 7f3005e00640 1 Processor -- start 2024-09-13T17:16:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.698+0000 7f3005e00640 1 -- start start 2024-09-13T17:16:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f3005e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30000b2ce0 0x7f30000b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f3005e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30000b1980 con 0x7f30000cec20 2024-09-13T17:16:21.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f3005e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30000b31e0 con 0x7f300006c610 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f3005e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3000071720 con 0x7f30000b2ce0 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f2ffe000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30000b2ce0 0x7f30000b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f2ffea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f30000cec20 0x7f3000071010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40004/0 (socket says 172.21.15.28:40004) 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.699+0000 7f2ffea00640 1 -- 172.21.15.28:0/3239309170 learned_addr learned my addr 172.21.15.28:0/3239309170 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.700+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629316637 0 0) 0x7f30000b1980 con 0x7f30000cec20 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fe4003500 con 0x7f30000cec20 2024-09-13T17:16:21.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26598703 0 0) 0x7f30000b31e0 con 0x7f300006c610 2024-09-13T17:16:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30000b1980 con 0x7f300006c610 2024-09-13T17:16:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffe000640 1 -- 172.21.15.28:0/3239309170 >> v1:172.21.15.189:6789/0 conn(0x7f300006c610 legacy=0x7f3000169af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 529070291 0 0) 0x7f2fe4003500 con 0x7f30000cec20 2024-09-13T17:16:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffe000640 1 -- 172.21.15.28:0/3239309170 >> v1:172.21.15.28:6789/0 conn(0x7f30000cec20 legacy=0x7f3000071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.905+0000 7f2ffe000640 1 -- 172.21.15.28:0/3239309170 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30000b34c0 con 0x7f30000b2ce0 2024-09-13T17:16:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.907+0000 7f2ffe000640 1 --2- 172.21.15.28:0/3239309170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000b2ce0 0x7f30000b1e10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007090 tx=0x7f2fec029ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd61db5e71917d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.907+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0001670 con 0x7f30000b2ce0 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.907+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fec007ae0 con 0x7f30000b2ce0 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.907+0000 7f2ffd600640 1 -- 172.21.15.28:0/3239309170 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fec007e70 con 0x7f30000b2ce0 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 -- 172.21.15.28:0/3239309170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000b2ce0 msgr2=0x7f30000b1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 --2- 172.21.15.28:0/3239309170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000b2ce0 0x7f30000b1e10 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f2fec007090 tx=0x7f2fec029ff0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 -- 172.21.15.28:0/3239309170 shutdown_connections 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 --2- 172.21.15.28:0/3239309170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000b2ce0 0x7f30000b1e10 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 -- 172.21.15.28:0/3239309170 >> 172.21.15.28:0/3239309170 conn(0x7f3000069e80 msgr2=0x7f300006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 -- 172.21.15.28:0/3239309170 shutdown_connections 2024-09-13T17:16:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.908+0000 7f3005e00640 1 -- 172.21.15.28:0/3239309170 wait complete. 2024-09-13T17:16:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.910+0000 7f3005e00640 1 Processor -- start 2024-09-13T17:16:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.910+0000 7f3005e00640 1 -- start start 2024-09-13T17:16:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300006c610 0x7f3000158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30000b2ce0 0x7f300016a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30000a91a0 con 0x7f30000cec20 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3000159560 con 0x7f30000b2ce0 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f3005e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3000159840 con 0x7f300006c610 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f2fff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f2fff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58832/0 (socket says 172.21.15.28:58832) 2024-09-13T17:16:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.911+0000 7f2fff400640 1 -- 172.21.15.28:0/4070418169 learned_addr learned my addr 172.21.15.28:0/4070418169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2ffea00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300006c610 0x7f3000158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2ffe000640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30000b2ce0 0x7f300016a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 -- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300006c610 msgr2=0x7f3000158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300006c610 0x7f3000158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 -- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30000b2ce0 msgr2=0x7f300016a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30000b2ce0 0x7f300016a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 -- 172.21.15.28:0/4070418169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fec009d20 con 0x7f30000cec20 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.912+0000 7f2fff400640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c6e0 tx=0x7f2ff800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.913+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8003b70 con 0x7f30000cec20 2024-09-13T17:16:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.913+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff8003cd0 con 0x7f30000cec20 2024-09-13T17:16:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.913+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3000159b20 con 0x7f30000cec20 2024-09-13T17:16:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.913+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff80052d0 con 0x7f30000cec20 2024-09-13T17:16:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.913+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30001591e0 con 0x7f30000cec20 2024-09-13T17:16:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.914+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30000a9680 con 0x7f30000cec20 2024-09-13T17:16:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.915+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ff8005430 con 0x7f30000cec20 2024-09-13T17:16:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.916+0000 7f2ff7400640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 0x7f2fe006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.916+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2ff8088cc0 con 0x7f30000cec20 2024-09-13T17:16:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.917+0000 7f2ffea00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 0x7f2fe006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:21.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.917+0000 7f2ffea00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 0x7f2fe006d0f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00011e0 tx=0x7f2ff00016d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:21.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:21.925+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f30000a9680 con 0x7f30000cec20 2024-09-13T17:16:22.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.107+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30000a9680 con 0x7f30000cec20 2024-09-13T17:16:22.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.108+0000 7f2ff7400640 1 -- 172.21.15.28:0/4070418169 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30000a9680 con 0x7f30000cec20 2024-09-13T17:16:22.110 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 msgr2=0x7f2fe006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 0x7f2fe006d0f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00011e0 tx=0x7f2ff00016d0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 msgr2=0x7f300016ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f2ff800c6e0 tx=0x7f2ff800cba0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 shutdown_connections 2024-09-13T17:16:22.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe006ac30 0x7f2fe006d0f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30000cec20 0x7f300016ae70 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30000b2ce0 0x7f300016a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 --2- 172.21.15.28:0/4070418169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300006c610 0x7f3000158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 >> 172.21.15.28:0/4070418169 conn(0x7f3000069e80 msgr2=0x7f3000169e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 shutdown_connections 2024-09-13T17:16:22.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.110+0000 7f3005e00640 1 -- 172.21.15.28:0/4070418169 wait complete. 2024-09-13T17:16:22.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort' 2024-09-13T17:16:22.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.283+0000 7fd23e000640 1 Processor -- start 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.283+0000 7fd23e000640 1 -- start start 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd23e000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2380ad930 0x7fd2380ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd23e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380b3750 con 0x7fd23806cd90 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd23e000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2380cef90 con 0x7fd2380add10 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd23e000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2380b2e20 con 0x7fd2380ad930 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd237e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd23806cd90 0x7fd23816d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40020/0 (socket says 172.21.15.28:40020) 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.284+0000 7fd237e00640 1 -- 172.21.15.28:0/1668412186 learned_addr learned my addr 172.21.15.28:0/1668412186 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.285+0000 7fd237400640 1 --2- 172.21.15.28:0/1668412186 >> v2:172.21.15.189:3300/0 conn(0x7fd2380ad930 0x7fd2380ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.285+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134738635 0 0) 0x7fd2380b3750 con 0x7fd23806cd90 2024-09-13T17:16:22.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd21c003440 con 0x7fd23806cd90 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146050015 0 0) 0x7fd2380cef90 con 0x7fd2380add10 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2380b3750 con 0x7fd2380add10 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2337627972 0 0) 0x7fd21c003440 con 0x7fd23806cd90 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 >> v2:172.21.15.189:3300/0 conn(0x7fd2380ad930 msgr2=0x7fd2380ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 --2- 172.21.15.28:0/1668412186 >> v2:172.21.15.189:3300/0 conn(0x7fd2380ad930 0x7fd2380ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 >> v1:172.21.15.189:6789/0 conn(0x7fd2380add10 legacy=0x7fd23806c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380b3100 con 0x7fd23806cd90 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fd2300019f0 con 0x7fd23806cd90 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 >> v1:172.21.15.28:6789/0 conn(0x7fd23806cd90 legacy=0x7fd23816d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 0x7fd21c004160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd21c004790 0x7fd21c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c00d800 0x7fd21c00fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380b2e20 con 0x7fd21c004790 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c0105a0 con 0x7fd21c003d60 2024-09-13T17:16:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.582+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21c010760 con 0x7fd21c00d800 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 0x7fd21c004160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237e00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c00d800 0x7fd21c00fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd22ea00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd21c004790 0x7fd21c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 -- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c00d800 msgr2=0x7fd21c00fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c00d800 0x7fd21c00fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 -- 172.21.15.28:0/1668412186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd21c004790 msgr2=0x7fd21c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd21c004790 0x7fd21c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237400640 1 -- 172.21.15.28:0/1668412186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c003440 con 0x7fd21c003d60 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd22ea00640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd21c004790 0x7fd21c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:22.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.583+0000 7fd237e00640 1 -- 172.21.15.28:0/1668412186 reap_dead start 2024-09-13T17:16:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.585+0000 7fd237400640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 0x7fd21c004160 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd22800bb20 tx=0x7fd21c0036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8693001e1f8fb841 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.585+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228031600 con 0x7fd21c003d60 2024-09-13T17:16:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.585+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22802d050 con 0x7fd21c003d60 2024-09-13T17:16:22.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.585+0000 7fd236a00640 1 -- 172.21.15.28:0/1668412186 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228031980 con 0x7fd21c003d60 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 -- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 msgr2=0x7fd21c004160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 0x7fd21c004160 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd22800bb20 tx=0x7fd21c0036f0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 -- 172.21.15.28:0/1668412186 shutdown_connections 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 --2- 172.21.15.28:0/1668412186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd21c003d60 0x7fd21c004160 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 -- 172.21.15.28:0/1668412186 >> 172.21.15.28:0/1668412186 conn(0x7fd238069e80 msgr2=0x7fd23806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 -- 172.21.15.28:0/1668412186 shutdown_connections 2024-09-13T17:16:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.586+0000 7fd23e000640 1 -- 172.21.15.28:0/1668412186 wait complete. 2024-09-13T17:16:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.588+0000 7fd23e000640 1 Processor -- start 2024-09-13T17:16:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.589+0000 7fd23e000640 1 -- start start 2024-09-13T17:16:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.589+0000 7fd23e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 0x7fd2381ffac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd23e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd23e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2380aadd0 0x7fd2380ab280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd23e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380ab7c0 con 0x7fd238162ff0 2024-09-13T17:16:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd23e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd238200000 con 0x7fd2380aadd0 2024-09-13T17:16:22.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd23e000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2380a8890 con 0x7fd21c003d60 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd22ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd22ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58852/0 (socket says 172.21.15.28:58852) 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd22ea00640 1 -- 172.21.15.28:0/2340400141 learned_addr learned my addr 172.21.15.28:0/2340400141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd237400640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 0x7fd2381ffac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.590+0000 7fd237e00640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2380aadd0 0x7fd2380ab280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 -- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2380aadd0 msgr2=0x7fd2380ab280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2380aadd0 0x7fd2380ab280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 -- 172.21.15.28:0/2340400141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 msgr2=0x7fd2380a9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 -- 172.21.15.28:0/2340400141 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2380a8b70 con 0x7fd21c003d60 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd22ea00640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.591+0000 7fd237400640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 0x7fd2381ffac0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd228006fd0 tx=0x7fd228004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:22.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.592+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22803b8e0 con 0x7fd21c003d60 2024-09-13T17:16:22.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.592+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22803ba40 con 0x7fd21c003d60 2024-09-13T17:16:22.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.592+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22803bdf0 con 0x7fd21c003d60 2024-09-13T17:16:22.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.592+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2380a8e50 con 0x7fd21c003d60 2024-09-13T17:16:22.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.592+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2380a9410 con 0x7fd21c003d60 2024-09-13T17:16:22.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.593+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2380a95a0 con 0x7fd21c003d60 2024-09-13T17:16:22.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.594+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd22804c3a0 con 0x7fd21c003d60 2024-09-13T17:16:22.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.595+0000 7fd22fe00640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 0x7fd21406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:22.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.595+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd2280a9fc0 con 0x7fd21c003d60 2024-09-13T17:16:22.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.595+0000 7fd22ea00640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 0x7fd21406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:22.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.596+0000 7fd22ea00640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 0x7fd21406d0f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd238166b70 tx=0x7fd220009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:22.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.600+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd228075540 con 0x7fd21c003d60 2024-09-13T17:16:22.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:22.848+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]} v 0) v1 -- 0x7fd23808a7d0 con 0x7fd21c003d60 2024-09-13T17:16:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.158+0000 7fd22fe00640 1 -- 172.21.15.28:0/2340400141 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]}]=0 v423) v1 ==== 147+0+0 (secure 0 0 0) 0x7fd228078b60 con 0x7fd21c003d60 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.160+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 msgr2=0x7fd21406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.160+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 0x7fd21406d0f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd238166b70 tx=0x7fd220009040 comp rx=0 tx=0).stop 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.160+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 msgr2=0x7fd2381ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.160+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 0x7fd2381ffac0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd228006fd0 tx=0x7fd228004730 comp rx=0 tx=0).stop 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 shutdown_connections 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd21406ac30 0x7fd21406d0f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2380aadd0 0x7fd2380ab280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd238162ff0 0x7fd2380a9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 --2- 172.21.15.28:0/2340400141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd21c003d60 0x7fd2381ffac0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 >> 172.21.15.28:0/2340400141 conn(0x7fd238069e80 msgr2=0x7fd238169940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 shutdown_connections 2024-09-13T17:16:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.161+0000 7fd23e000640 1 -- 172.21.15.28:0/2340400141 wait complete. 2024-09-13T17:16:23.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:16:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.343+0000 7f4c5ce00640 1 Processor -- start 2024-09-13T17:16:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.343+0000 7f4c5ce00640 1 -- start start 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c5ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c5806af70 0x7f4c58072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c5ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5806d2f0 con 0x7f4c581460f0 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c5ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5806a8e0 con 0x7f4c5806b3e0 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c5ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c5806c400 con 0x7f4c5806af70 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c4ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c5806af70 0x7f4c58072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c57400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4c581460f0 0x7f4c580ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40022/0 (socket says 172.21.15.28:40022) 2024-09-13T17:16:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.344+0000 7f4c57400640 1 -- 172.21.15.28:0/3688873439 learned_addr learned my addr 172.21.15.28:0/3688873439 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.345+0000 7f4c56a00640 1 -- 172.21.15.28:0/3688873439 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225109208 0 0) 0x7f4c5806d2f0 con 0x7f4c581460f0 2024-09-13T17:16:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.345+0000 7f4c56a00640 1 -- 172.21.15.28:0/3688873439 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3c003520 con 0x7f4c581460f0 2024-09-13T17:16:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.345+0000 7f4c4ea00640 1 -- 172.21.15.28:0/3688873439 >> v1:172.21.15.189:6789/0 conn(0x7f4c5806b3e0 legacy=0x7f4c5806d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.345+0000 7f4c4ea00640 1 -- 172.21.15.28:0/3688873439 >> v1:172.21.15.28:6789/0 conn(0x7f4c581460f0 legacy=0x7f4c580ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.345+0000 7f4c4ea00640 1 -- 172.21.15.28:0/3688873439 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5806c6e0 con 0x7f4c5806af70 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.346+0000 7f4c4ea00640 1 --2- 172.21.15.28:0/3688873439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806af70 0x7f4c58072e40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4c400097d0 tx=0x7f4c4002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774f642e1d3841ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.346+0000 7f4c56a00640 1 -- 172.21.15.28:0/3688873439 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40032440 con 0x7f4c5806af70 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.346+0000 7f4c56a00640 1 -- 172.21.15.28:0/3688873439 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c400325a0 con 0x7f4c5806af70 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.346+0000 7f4c56a00640 1 -- 172.21.15.28:0/3688873439 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40032930 con 0x7f4c5806af70 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.346+0000 7f4c5ce00640 1 -- 172.21.15.28:0/3688873439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806af70 msgr2=0x7f4c58072e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/3688873439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806af70 0x7f4c58072e40 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f4c400097d0 tx=0x7f4c4002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 -- 172.21.15.28:0/3688873439 shutdown_connections 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/3688873439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806af70 0x7f4c58072e40 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 -- 172.21.15.28:0/3688873439 >> 172.21.15.28:0/3688873439 conn(0x7f4c580b1280 msgr2=0x7f4c580b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 -- 172.21.15.28:0/3688873439 shutdown_connections 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.347+0000 7f4c5ce00640 1 -- 172.21.15.28:0/3688873439 wait complete. 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.348+0000 7f4c5ce00640 1 Processor -- start 2024-09-13T17:16:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.348+0000 7f4c5ce00640 1 -- start start 2024-09-13T17:16:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c5806af70 0x7f4c581f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c581460f0 0x7f4c5815ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581f6520 con 0x7f4c5806b3e0 2024-09-13T17:16:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581f58a0 con 0x7f4c5806af70 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.349+0000 7f4c5ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581f5b80 con 0x7f4c581460f0 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58868/0 (socket says 172.21.15.28:58868) 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 -- 172.21.15.28:0/1886915110 learned_addr learned my addr 172.21.15.28:0/1886915110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c57400640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c5806af70 0x7f4c581f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c57e00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c581460f0 0x7f4c5815ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 -- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c581460f0 msgr2=0x7f4c5815ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c581460f0 0x7f4c5815ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 -- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c5806af70 msgr2=0x7f4c581f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c5806af70 0x7f4c581f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 -- 172.21.15.28:0/1886915110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c40009400 con 0x7f4c5806b3e0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.350+0000 7f4c4ea00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009720 tx=0x7f4c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.351+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40032440 con 0x7f4c5806b3e0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.351+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c40004230 con 0x7f4c5806b3e0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.351+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c40006e10 con 0x7f4c5806b3e0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.351+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c581f5e60 con 0x7f4c5806b3e0 2024-09-13T17:16:23.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.351+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c581f2f10 con 0x7f4c5806b3e0 2024-09-13T17:16:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.352+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c581f6660 con 0x7f4c5806b3e0 2024-09-13T17:16:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.353+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4c40039050 con 0x7f4c5806b3e0 2024-09-13T17:16:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.354+0000 7f4c4fe00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.354+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4c400a99c0 con 0x7f4c5806b3e0 2024-09-13T17:16:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.354+0000 7f4c57400640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:23.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.355+0000 7f4c57400640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 0x7f4c3406d0c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4c480011e0 tx=0x7f4c48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:23.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.358+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4c581f6660 con 0x7f4c5806b3e0 2024-09-13T17:16:23.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.537+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4c581f6660 con 0x7f4c5806b3e0 2024-09-13T17:16:23.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.541+0000 7f4c4fe00640 1 -- 172.21.15.28:0/1886915110 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c581f6660 con 0x7f4c5806b3e0 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 msgr2=0x7f4c3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 0x7f4c3406d0c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4c480011e0 tx=0x7f4c48009040 comp rx=0 tx=0).stop 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 msgr2=0x7f4c5813e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009720 tx=0x7f4c400046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 shutdown_connections 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c581460f0 0x7f4c5815ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c5806b3e0 0x7f4c5813e2e0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 --2- 172.21.15.28:0/1886915110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c5806af70 0x7f4c581f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 >> 172.21.15.28:0/1886915110 conn(0x7f4c580b1280 msgr2=0x7f4c580b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 shutdown_connections 2024-09-13T17:16:23.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:23.544+0000 7f4c5ce00640 1 -- 172.21.15.28:0/1886915110 wait complete. 2024-09-13T17:16:23.562 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:16:23.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:16:23.658 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:16:23.760 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:16:23.760 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:23.760 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:23.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:16:23.796 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:16:23.796 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:23.832 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:16:23.833 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:23.833 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:23.872 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:16:23.872 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:16:23.872 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:23.910 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:16:23.910 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:16:23.910 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:24.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.065+0000 7f3fdfe00640 1 Processor -- start 2024-09-13T17:16:24.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.066+0000 7f3fdfe00640 1 -- start start 2024-09-13T17:16:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdfe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fe0159440 0x7f3fe00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdfe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00c3850 con 0x7f3fe0159820 2024-09-13T17:16:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdfe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe015a930 con 0x7f3fe0159440 2024-09-13T17:16:24.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0159d20 con 0x7f3fe00b29f0 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fd7e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f3fe0159820 0x7f3fe00b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:47782/0 (socket says 172.21.15.28:47782) 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fdea00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3fe0159440 0x7f3fe00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.067+0000 7f3fd7e00640 1 -- 172.21.15.28:0/1555498138 learned_addr learned my addr 172.21.15.28:0/1555498138 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fde000640 1 -- 172.21.15.28:0/1555498138 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002805040 0 0) 0x7f3fe00c3850 con 0x7f3fe0159820 2024-09-13T17:16:24.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fde000640 1 -- 172.21.15.28:0/1555498138 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fc4003520 con 0x7f3fe0159820 2024-09-13T17:16:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fdea00640 1 -- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 msgr2=0x7f3fe00b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fdea00640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fdea00640 1 -- 172.21.15.28:0/1555498138 >> v1:172.21.15.189:6789/0 conn(0x7f3fe0159820 legacy=0x7f3fe00b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fdea00640 1 -- 172.21.15.28:0/1555498138 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe015a000 con 0x7f3fe0159440 2024-09-13T17:16:24.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.068+0000 7f3fdf400640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.069+0000 7f3fdea00640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe00b0440 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097d0 tx=0x7f3fd002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=771b1b0b4cae763e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.069+0000 7f3fde000640 1 -- 172.21.15.28:0/1555498138 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0032440 con 0x7f3fe0159440 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fde000640 1 -- 172.21.15.28:0/1555498138 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd00325a0 con 0x7f3fe0159440 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fde000640 1 -- 172.21.15.28:0/1555498138 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0032930 con 0x7f3fe0159440 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 -- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 msgr2=0x7f3fe00b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe00b0440 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3fd00097d0 tx=0x7f3fd002a250 comp rx=0 tx=0).stop 2024-09-13T17:16:24.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 -- 172.21.15.28:0/1555498138 shutdown_connections 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/1555498138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe00b0440 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 -- 172.21.15.28:0/1555498138 >> 172.21.15.28:0/1555498138 conn(0x7f3fe0072e40 msgr2=0x7f3fe0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 -- 172.21.15.28:0/1555498138 shutdown_connections 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.070+0000 7f3fdfe00640 1 -- 172.21.15.28:0/1555498138 wait complete. 2024-09-13T17:16:24.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.072+0000 7f3fdfe00640 1 Processor -- start 2024-09-13T17:16:24.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.072+0000 7f3fdfe00640 1 -- start start 2024-09-13T17:16:24.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe015c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00ac650 con 0x7f3fe0159820 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00a9af0 con 0x7f3fe0159440 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00a9dd0 con 0x7f3fe00b29f0 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32876/0 (socket says 172.21.15.28:32876) 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdf400640 1 -- 172.21.15.28:0/3109614312 learned_addr learned my addr 172.21.15.28:0/3109614312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fdea00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.073+0000 7f3fd7e00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe015c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 -- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 msgr2=0x7f3fe015c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe015c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 -- 172.21.15.28:0/3109614312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 msgr2=0x7f3fe0154280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 -- 172.21.15.28:0/3109614312 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd8009a70 con 0x7f3fe00b29f0 2024-09-13T17:16:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdf400640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:24.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdea00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00a92d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0031000 tx=0x7f3fd00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0006e10 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd0030980 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd0009400 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.074+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0030d50 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.075+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe00aa390 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.076+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe00a9810 con 0x7f3fe00b29f0 2024-09-13T17:16:24.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.077+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3fd0039040 con 0x7f3fe00b29f0 2024-09-13T17:16:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.078+0000 7f3fd6a00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 0x7f3fbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.078+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3fd00a9c90 con 0x7f3fe00b29f0 2024-09-13T17:16:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.078+0000 7f3fd7e00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 0x7f3fbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.079+0000 7f3fd7e00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 0x7f3fbc06d0c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00a9000 tx=0x7f3fc8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.081+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3fd0075180 con 0x7f3fe00b29f0 2024-09-13T17:16:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.313+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe006fed0 con 0x7f3fe00b29f0 2024-09-13T17:16:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.314+0000 7f3fd6a00640 1 -- 172.21.15.28:0/3109614312 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+3534 (secure 0 0 0) 0x7f3fd0037de0 con 0x7f3fe00b29f0 2024-09-13T17:16:24.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:24.316 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":108,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6749,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1539335522","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1539335522},{"type":"v1","addr":"172.21.15.189:6837","nonce":1539335522}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":7525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1007330580","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1007330580},{"type":"v1","addr":"172.21.15.189:6836","nonce":1007330580}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8019,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3434672926","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3434672926},{"type":"v1","addr":"172.21.15.28:6834","nonce":3434672926}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":89},{"gid":8376,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/476134146","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":476134146},{"type":"v1","addr":"172.21.15.28:6835","nonce":476134146}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":108}],"filesystems":[]} 2024-09-13T17:16:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 108 2024-09-13T17:16:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 msgr2=0x7f3fbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 0x7f3fbc06d0c0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f3fe00a9000 tx=0x7f3fc8007d30 comp rx=0 tx=0).stop 2024-09-13T17:16:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 msgr2=0x7f3fe00a92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00a92d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0031000 tx=0x7f3fd00046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 shutdown_connections 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3fbc06ac00 0x7f3fbc06d0c0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fe0159820 0x7f3fe0154280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3fe0159440 0x7f3fe015c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 --2- 172.21.15.28:0/3109614312 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3fe00b29f0 0x7f3fe00a92d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 >> 172.21.15.28:0/3109614312 conn(0x7f3fe0072e40 msgr2=0x7f3fe00b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 shutdown_connections 2024-09-13T17:16:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.316+0000 7f3fdfe00640 1 -- 172.21.15.28:0/3109614312 wait complete. 2024-09-13T17:16:24.334 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:16:24.334 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:16:24.334 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:16:24.334 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:16:24.334 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:16:24.335 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:16:24.335 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:16:24.335 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:16:24.335 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:16:24.335 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:16:24.335 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:16:24.336 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:16:24.336 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:16:24.336 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:16:24.336 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:16:24.336 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:16:24.349 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:16:24.349 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:16:24.350 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:16:24.350 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:16:24.351 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:16:24.351 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:16:24.352 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:16:24.352 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:16:24.393 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:16:24.394 DEBUG:teuthology.parallel:result is None 2024-09-13T17:16:24.395 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:16:24.395 DEBUG:teuthology.parallel:result is None 2024-09-13T17:16:24.396 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:16:24.396 DEBUG:teuthology.parallel:result is None 2024-09-13T17:16:24.398 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:16:24.398 DEBUG:teuthology.parallel:result is None 2024-09-13T17:16:24.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:16:24.525 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:16:24.525 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:16:24.527 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:16:24.527 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:16:24.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.562+0000 7faf24a00640 1 Processor -- start 2024-09-13T17:16:24.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.562+0000 7faf24a00640 1 -- start start 2024-09-13T17:16:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.563+0000 7faf24a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf200b17e0 0x7faf200b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.563+0000 7faf24a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf200b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.563+0000 7faf24a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf201610a0 con 0x7faf200b2100 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.563+0000 7faf24a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf200b2620 con 0x7faf200b17e0 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.563+0000 7faf24a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf200b2900 con 0x7faf2006d070 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faf200b2100 0x7faf2006b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1e800640 1 -- 172.21.15.28:0/3169604273 learned_addr learned my addr 172.21.15.28:0/3169604273 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1d400640 1 --2- 172.21.15.28:0/3169604273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf200b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1de00640 1 --2- 172.21.15.28:0/3169604273 >> v2:172.21.15.189:3300/0 conn(0x7faf200b17e0 0x7faf200b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1ca00640 1 -- 172.21.15.28:0/3169604273 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177910851 0 0) 0x7faf201610a0 con 0x7faf200b2100 2024-09-13T17:16:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1ca00640 1 -- 172.21.15.28:0/3169604273 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf04003520 con 0x7faf200b2100 2024-09-13T17:16:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1d400640 1 -- 172.21.15.28:0/3169604273 >> v2:172.21.15.189:3300/0 conn(0x7faf200b17e0 msgr2=0x7faf200b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:24.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1d400640 1 --2- 172.21.15.28:0/3169604273 >> v2:172.21.15.189:3300/0 conn(0x7faf200b17e0 0x7faf200b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1d400640 1 -- 172.21.15.28:0/3169604273 >> v1:172.21.15.28:6789/0 conn(0x7faf200b2100 legacy=0x7faf2006b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.564+0000 7faf1d400640 1 -- 172.21.15.28:0/3169604273 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf200b2be0 con 0x7faf2006d070 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.565+0000 7faf1d400640 1 --2- 172.21.15.28:0/3169604273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf200b1bc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0097a0 tx=0x7faf0c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f51ad8be96aff01e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.565+0000 7faf1ca00640 1 -- 172.21.15.28:0/3169604273 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c032440 con 0x7faf2006d070 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf1ca00640 1 -- 172.21.15.28:0/3169604273 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c0325a0 con 0x7faf2006d070 2024-09-13T17:16:24.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf1ca00640 1 -- 172.21.15.28:0/3169604273 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c032930 con 0x7faf2006d070 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 -- 172.21.15.28:0/3169604273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 msgr2=0x7faf200b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 --2- 172.21.15.28:0/3169604273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf200b1bc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0097a0 tx=0x7faf0c02a180 comp rx=0 tx=0).stop 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 -- 172.21.15.28:0/3169604273 shutdown_connections 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 --2- 172.21.15.28:0/3169604273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf200b1bc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 --2- 172.21.15.28:0/3169604273 >> v2:172.21.15.189:3300/0 conn(0x7faf200b17e0 0x7faf200b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 -- 172.21.15.28:0/3169604273 >> 172.21.15.28:0/3169604273 conn(0x7faf200b0270 msgr2=0x7faf200b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 -- 172.21.15.28:0/3169604273 shutdown_connections 2024-09-13T17:16:24.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.566+0000 7faf24a00640 1 -- 172.21.15.28:0/3169604273 wait complete. 2024-09-13T17:16:24.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.568+0000 7faf24a00640 1 Processor -- start 2024-09-13T17:16:24.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.568+0000 7faf24a00640 1 -- start start 2024-09-13T17:16:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf201548d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf200b2100 0x7faf200ab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20156fe0 con 0x7faf200b17e0 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf20154e10 con 0x7faf200b2100 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf24a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf200a9940 con 0x7faf2006d070 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf1d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf1d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32910/0 (socket says 172.21.15.28:32910) 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf1d400640 1 -- 172.21.15.28:0/2952501906 learned_addr learned my addr 172.21.15.28:0/2952501906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf1de00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf201548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.569+0000 7faf1e800640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf200b2100 0x7faf200ab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 -- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 msgr2=0x7faf201548d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf201548d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 -- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf200b2100 msgr2=0x7faf200ab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf200b2100 0x7faf200ab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 -- 172.21.15.28:0/2952501906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c009400 con 0x7faf200b17e0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf1d400640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0098c0 tx=0x7faf0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c03d8f0 con 0x7faf200b17e0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0c03da70 con 0x7faf200b17e0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0c03de20 con 0x7faf200b17e0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.570+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf200a9c20 con 0x7faf200b17e0 2024-09-13T17:16:24.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.571+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf20157230 con 0x7faf200b17e0 2024-09-13T17:16:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.572+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf20157420 con 0x7faf200b17e0 2024-09-13T17:16:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.573+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faf0c039040 con 0x7faf200b17e0 2024-09-13T17:16:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.573+0000 7faf16000640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 0x7faf0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:24.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.574+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faf0c0a96c0 con 0x7faf200b17e0 2024-09-13T17:16:24.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.574+0000 7faf1de00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 0x7faf0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:24.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.574+0000 7faf1de00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 0x7faf0006d0c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faf10009b40 tx=0x7faf10009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:24.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.578+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faf0c074d60 con 0x7faf200b17e0 2024-09-13T17:16:24.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:24.768+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7faf200b1bc0 con 0x7faf200b17e0 2024-09-13T17:16:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.564+0000 7faf16000640 1 -- 172.21.15.28:0/2952501906 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7faf0c078380 con 0x7faf200b17e0 2024-09-13T17:16:25.566 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:16:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 msgr2=0x7faf0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 0x7faf0006d0c0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7faf10009b40 tx=0x7faf10009190 comp rx=0 tx=0).stop 2024-09-13T17:16:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 msgr2=0x7faf200a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7faf0c0098c0 tx=0x7faf0c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 shutdown_connections 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf0006ac00 0x7faf0006d0c0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf200b2100 0x7faf200ab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf200b17e0 0x7faf200a8e50 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 --2- 172.21.15.28:0/2952501906 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf2006d070 0x7faf201548d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 >> 172.21.15.28:0/2952501906 conn(0x7faf200b0270 msgr2=0x7faf2006c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 shutdown_connections 2024-09-13T17:16:25.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.567+0000 7faf24a00640 1 -- 172.21.15.28:0/2952501906 wait complete. 2024-09-13T17:16:25.590 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:16:25.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.749+0000 7f8f68c00640 1 Processor -- start 2024-09-13T17:16:25.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.749+0000 7f8f68c00640 1 -- start start 2024-09-13T17:16:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.750+0000 7f8f68c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 0x7f8f640d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.750+0000 7f8f68c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 0x7f8f64069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.750+0000 7f8f68c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f640d0d80 con 0x7f8f640d1d10 2024-09-13T17:16:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.750+0000 7f8f68c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f640d0960 con 0x7f8f6415c750 2024-09-13T17:16:25.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.750+0000 7f8f68c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6406aa60 con 0x7f8f640b4ad0 2024-09-13T17:16:25.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 0x7f8f64069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 0x7f8f640d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f5be00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f8f640d1d10 0x7f8f640b43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:47800/0 (socket says 172.21.15.28:47800) 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f5be00640 1 -- 172.21.15.28:0/3658488858 learned_addr learned my addr 172.21.15.28:0/3658488858 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f62a00640 1 -- 172.21.15.28:0/3658488858 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717399268 0 0) 0x7f8f640d0d80 con 0x7f8f640d1d10 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f62a00640 1 -- 172.21.15.28:0/3658488858 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f48003520 con 0x7f8f640d1d10 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63e00640 1 -- 172.21.15.28:0/3658488858 >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 msgr2=0x7f8f640d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63e00640 1 --2- 172.21.15.28:0/3658488858 >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 0x7f8f640d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63e00640 1 -- 172.21.15.28:0/3658488858 >> v1:172.21.15.189:6789/0 conn(0x7f8f640d1d10 legacy=0x7f8f640b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63e00640 1 -- 172.21.15.28:0/3658488858 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6406ad40 con 0x7f8f640b4ad0 2024-09-13T17:16:25.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.751+0000 7f8f63400640 1 --2- 172.21.15.28:0/3658488858 >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 0x7f8f640d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.752+0000 7f8f63e00640 1 --2- 172.21.15.28:0/3658488858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 0x7f8f64069e80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009d90 tx=0x7f8f5c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3117e1e769817913 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f62a00640 1 -- 172.21.15.28:0/3658488858 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031600 con 0x7f8f640b4ad0 2024-09-13T17:16:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f62a00640 1 -- 172.21.15.28:0/3658488858 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5c02d050 con 0x7f8f640b4ad0 2024-09-13T17:16:25.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f62a00640 1 -- 172.21.15.28:0/3658488858 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5c031980 con 0x7f8f640b4ad0 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 -- 172.21.15.28:0/3658488858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 msgr2=0x7f8f64069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 --2- 172.21.15.28:0/3658488858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 0x7f8f64069e80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009d90 tx=0x7f8f5c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 -- 172.21.15.28:0/3658488858 shutdown_connections 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 --2- 172.21.15.28:0/3658488858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f640b4ad0 0x7f8f64069e80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 --2- 172.21.15.28:0/3658488858 >> v2:172.21.15.189:3300/0 conn(0x7f8f6415c750 0x7f8f640d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.753+0000 7f8f68c00640 1 -- 172.21.15.28:0/3658488858 >> 172.21.15.28:0/3658488858 conn(0x7f8f640cf5a0 msgr2=0x7f8f640cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.754+0000 7f8f68c00640 1 -- 172.21.15.28:0/3658488858 shutdown_connections 2024-09-13T17:16:25.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.754+0000 7f8f68c00640 1 -- 172.21.15.28:0/3658488858 wait complete. 2024-09-13T17:16:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.755+0000 7f8f68c00640 1 Processor -- start 2024-09-13T17:16:25.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.755+0000 7f8f68c00640 1 -- start start 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6415c750 0x7f8f640cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f640c7af0 0x7f8f640c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640cc250 con 0x7f8f640b4ad0 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641f1f30 con 0x7f8f640c7af0 2024-09-13T17:16:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f68c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f640cb5d0 con 0x7f8f6415c750 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32918/0 (socket says 172.21.15.28:32918) 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.756+0000 7f8f63400640 1 -- 172.21.15.28:0/2638753398 learned_addr learned my addr 172.21.15.28:0/2638753398 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f5be00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6415c750 0x7f8f640cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63e00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f640c7af0 0x7f8f640c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 -- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6415c750 msgr2=0x7f8f640cafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6415c750 0x7f8f640cafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 -- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f640c7af0 msgr2=0x7f8f640c7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f640c7af0 0x7f8f640c7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 -- 172.21.15.28:0/2638753398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54009400 con 0x7f8f640b4ad0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f63400640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400e960 tx=0x7f8f5400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.757+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5400ca00 con 0x7f8f640b4ad0 2024-09-13T17:16:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.758+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f5400cb60 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.758+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54013470 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.758+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c009a70 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.758+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f640cbb90 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.759+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f640d0b60 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.760+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8f540135d0 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.760+0000 7f8f5b400640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 0x7f8f4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.760+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8f54089260 con 0x7f8f640b4ad0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.761+0000 7f8f5be00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 0x7f8f4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.761+0000 7f8f5be00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 0x7f8f4006d0c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8f640d0830 tx=0x7f8f4c007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:25.764+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8f54054830 con 0x7f8f640b4ad0 2024-09-13T17:16:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.011+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8f64152fd0 con 0x7f8f640b4ad0 2024-09-13T17:16:26.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.011+0000 7f8f5b400640 1 -- 172.21.15.28:0/2638753398 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8f54087e60 con 0x7f8f640b4ad0 2024-09-13T17:16:26.013 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.014 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:16:26.015 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:16:26.016 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.017 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.018 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.019 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.020 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:16:26.021 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:16:26.022 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:16:26.023 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.024 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:16:26.025 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:26.026 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:16:26.027 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 msgr2=0x7f8f4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 0x7f8f4006d0c0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8f640d0830 tx=0x7f8f4c007d30 comp rx=0 tx=0).stop 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 msgr2=0x7f8f640caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400e960 tx=0x7f8f5400ee20 comp rx=0 tx=0).stop 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 shutdown_connections 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f4006ac00 0x7f8f4006d0c0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f640c7af0 0x7f8f640c7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f6415c750 0x7f8f640cafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 --2- 172.21.15.28:0/2638753398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f640b4ad0 0x7f8f640caa60 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 >> 172.21.15.28:0/2638753398 conn(0x7f8f640cf5a0 msgr2=0x7f8f6415ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 shutdown_connections 2024-09-13T17:16:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.014+0000 7f8f68c00640 1 -- 172.21.15.28:0/2638753398 wait complete. 2024-09-13T17:16:26.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.188+0000 7f7f9c200640 1 Processor -- start 2024-09-13T17:16:26.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.188+0000 7f7f9c200640 1 -- start start 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9c200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7f9415a540 0x7f7f9406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f9415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9415e100 con 0x7f7f9415edb0 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9c200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f940adb00 con 0x7f7f9415a540 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415e3e0 con 0x7f7f940cabc0 2024-09-13T17:16:26.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9a200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f9415edb0 0x7f7f94070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53526/0 (socket says 172.21.15.28:53526) 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.189+0000 7f7f9a200640 1 -- 172.21.15.28:0/302519132 learned_addr learned my addr 172.21.15.28:0/302519132 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f98e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f9415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f99800640 1 --2- 172.21.15.28:0/302519132 >> v2:172.21.15.189:3300/0 conn(0x7f7f9415a540 0x7f7f9406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414220360 0 0) 0x7f7f9415e100 con 0x7f7f9415edb0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c003520 con 0x7f7f9415edb0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1264884337 0 0) 0x7f7f7c003520 con 0x7f7f9415edb0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 msgr2=0x7f7f9415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f9415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 >> v2:172.21.15.189:3300/0 conn(0x7f7f9415a540 msgr2=0x7f7f9406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 --2- 172.21.15.28:0/302519132 >> v2:172.21.15.189:3300/0 conn(0x7f7f9415a540 0x7f7f9406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9406ce80 con 0x7f7f9415edb0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7f8c0019f0 con 0x7f7f9415edb0 2024-09-13T17:16:26.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 >> v1:172.21.15.28:6789/0 conn(0x7f7f9415edb0 legacy=0x7f7f94070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f7c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c003f40 0x7f7f7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f7c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415e3e0 con 0x7f7f7c004d00 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c00b9b0 con 0x7f7f7c003f40 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c00bb70 con 0x7f7f940cabc0 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.190+0000 7f7f9a200640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f7c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f98e00640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c003f40 0x7f7f7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f99800640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f7c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f9a200640 1 -- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 msgr2=0x7f7f7c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f9a200640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940cabc0 0x7f7f7c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f9a200640 1 -- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c003f40 msgr2=0x7f7f7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f9a200640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f7c003f40 0x7f7f7c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.191+0000 7f7f9a200640 1 -- 172.21.15.28:0/302519132 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c003520 con 0x7f7f7c004d00 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.192+0000 7f7f9a200640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f7c005180 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a790 tx=0x7f7f7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8954d17919b7d063 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.192+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c02b9a0 con 0x7f7f7c004d00 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.192+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8c02bb20 con 0x7f7f7c004d00 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.192+0000 7f7f93e00640 1 -- 172.21.15.28:0/302519132 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8c034470 con 0x7f7f7c004d00 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 -- 172.21.15.28:0/302519132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 msgr2=0x7f7f7c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 --2- 172.21.15.28:0/302519132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f7c005180 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c00a790 tx=0x7f7f7c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9a200640 1 -- 172.21.15.28:0/302519132 reap_dead start 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 -- 172.21.15.28:0/302519132 shutdown_connections 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 -- 172.21.15.28:0/302519132 >> 172.21.15.28:0/302519132 conn(0x7f7f94069e80 msgr2=0x7f7f9406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 -- 172.21.15.28:0/302519132 shutdown_connections 2024-09-13T17:16:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.193+0000 7f7f9c200640 1 -- 172.21.15.28:0/302519132 wait complete. 2024-09-13T17:16:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.195+0000 7f7f9c200640 1 Processor -- start 2024-09-13T17:16:26.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.195+0000 7f7f9c200640 1 -- start start 2024-09-13T17:16:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 0x7f7f940ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940b31e0 0x7f7f940acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9414c510 con 0x7f7f7c004d00 2024-09-13T17:16:26.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f940b37b0 con 0x7f7f940b2d60 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9415ca50 con 0x7f7f940b31e0 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f99800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f99800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32930/0 (socket says 172.21.15.28:32930) 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f99800640 1 -- 172.21.15.28:0/3589282389 learned_addr learned my addr 172.21.15.28:0/3589282389 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f9a200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940b31e0 0x7f7f940acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.196+0000 7f7f98e00640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 0x7f7f940ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 -- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940b31e0 msgr2=0x7f7f940acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940b31e0 0x7f7f940acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 -- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 msgr2=0x7f7f940ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 0x7f7f940ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 -- 172.21.15.28:0/3589282389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9415bc90 con 0x7f7f7c004d00 2024-09-13T17:16:26.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f98e00640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 0x7f7f940ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f99800640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7f88011a70 tx=0x7f7f88011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f88009680 con 0x7f7f7c004d00 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f8800fb50 con 0x7f7f7c004d00 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f8800fee0 con 0x7f7f7c004d00 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9415bf70 con 0x7f7f7c004d00 2024-09-13T17:16:26.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.197+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9415c530 con 0x7f7f7c004d00 2024-09-13T17:16:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.198+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9415e870 con 0x7f7f7c004d00 2024-09-13T17:16:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.199+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7f88014440 con 0x7f7f7c004d00 2024-09-13T17:16:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.200+0000 7f7f91600640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 0x7f7f7406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.200+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7f880899a0 con 0x7f7f7c004d00 2024-09-13T17:16:26.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.200+0000 7f7f98e00640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 0x7f7f7406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.201+0000 7f7f98e00640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 0x7f7f7406d0f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ad410 tx=0x7f7f80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.204+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7f88054e30 con 0x7f7f7c004d00 2024-09-13T17:16:26.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.384+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f940afae0 con 0x7f7f7c004d00 2024-09-13T17:16:26.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.385+0000 7f7f91600640 1 -- 172.21.15.28:0/3589282389 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f7f88058450 con 0x7f7f7c004d00 2024-09-13T17:16:26.387 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:26.387 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":121,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:25.558947+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.387+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 msgr2=0x7f7f7406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.387+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 0x7f7f7406d0f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7f940ad410 tx=0x7f7f80008040 comp rx=0 tx=0).stop 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.387+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 msgr2=0x7f7f9414b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.387+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f7f88011a70 tx=0x7f7f88011f30 comp rx=0 tx=0).stop 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 shutdown_connections 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f7406ac30 0x7f7f7406d0f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f940b31e0 0x7f7f940acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f940b2d60 0x7f7f940ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 --2- 172.21.15.28:0/3589282389 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f7c004d00 0x7f7f9414b9c0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 >> 172.21.15.28:0/3589282389 conn(0x7f7f94069e80 msgr2=0x7f7f940ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 shutdown_connections 2024-09-13T17:16:26.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.388+0000 7f7f9c200640 1 -- 172.21.15.28:0/3589282389 wait complete. 2024-09-13T17:16:26.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:16:26.405 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:16:26.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.568+0000 7f3e6ea00640 1 Processor -- start 2024-09-13T17:16:26.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.568+0000 7f3e6ea00640 1 -- start start 2024-09-13T17:16:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6ea00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3e6815c8e0 0x7f3e680ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e680cd300 0x7f3e680d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6ea00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e680b2020 con 0x7f3e680cce70 2024-09-13T17:16:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6ea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e680cd790 con 0x7f3e680cd300 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6ea00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e680b1a60 con 0x7f3e6815c8e0 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e680cd300 0x7f3e680d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e680cd300 0x7f3e680d0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32936/0 (socket says 172.21.15.28:32936) 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6e000640 1 -- 172.21.15.28:0/3655894774 learned_addr learned my addr 172.21.15.28:0/3655894774 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.569+0000 7f3e6d600640 1 --2- 172.21.15.28:0/3655894774 >> v2:172.21.15.189:3300/0 conn(0x7f3e6815c8e0 0x7f3e680ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.570+0000 7f3e6e000640 1 -- 172.21.15.28:0/3655894774 >> v2:172.21.15.189:3300/0 conn(0x7f3e6815c8e0 msgr2=0x7f3e680ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.570+0000 7f3e6e000640 1 --2- 172.21.15.28:0/3655894774 >> v2:172.21.15.189:3300/0 conn(0x7f3e6815c8e0 0x7f3e680ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.570+0000 7f3e6e000640 1 -- 172.21.15.28:0/3655894774 >> v1:172.21.15.189:6789/0 conn(0x7f3e680cce70 legacy=0x7f3e6806a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.570+0000 7f3e6e000640 1 -- 172.21.15.28:0/3655894774 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e680b1d40 con 0x7f3e680cd300 2024-09-13T17:16:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6e000640 1 --2- 172.21.15.28:0/3655894774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e680d0e40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3e680738f0 tx=0x7f3e6002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff35d72452abb5f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e67e00640 1 -- 172.21.15.28:0/3655894774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60031600 con 0x7f3e680cd300 2024-09-13T17:16:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e67e00640 1 -- 172.21.15.28:0/3655894774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6002d050 con 0x7f3e680cd300 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e67e00640 1 -- 172.21.15.28:0/3655894774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60031980 con 0x7f3e680cd300 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 -- 172.21.15.28:0/3655894774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 msgr2=0x7f3e680d0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/3655894774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e680d0e40 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f3e680738f0 tx=0x7f3e6002a660 comp rx=0 tx=0).stop 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 -- 172.21.15.28:0/3655894774 shutdown_connections 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/3655894774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e680d0e40 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/3655894774 >> v2:172.21.15.189:3300/0 conn(0x7f3e6815c8e0 0x7f3e680ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.571+0000 7f3e6ea00640 1 -- 172.21.15.28:0/3655894774 >> 172.21.15.28:0/3655894774 conn(0x7f3e68071a50 msgr2=0x7f3e680999e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.572+0000 7f3e6ea00640 1 -- 172.21.15.28:0/3655894774 shutdown_connections 2024-09-13T17:16:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.572+0000 7f3e6ea00640 1 -- 172.21.15.28:0/3655894774 wait complete. 2024-09-13T17:16:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.573+0000 7f3e6ea00640 1 Processor -- start 2024-09-13T17:16:26.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.573+0000 7f3e6ea00640 1 -- start start 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 0x7f3e6815da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e6815c8e0 0x7f3e680caa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6815ebf0 con 0x7f3e680cd300 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6815dfe0 con 0x7f3e6815c8e0 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6ea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6815e2c0 con 0x7f3e680cce70 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.574+0000 7f3e6cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32938/0 (socket says 172.21.15.28:32938) 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 -- 172.21.15.28:0/660887332 learned_addr learned my addr 172.21.15.28:0/660887332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6d600640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 0x7f3e6815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6e000640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e6815c8e0 0x7f3e680caa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 -- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 msgr2=0x7f3e6815da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 0x7f3e6815da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 -- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e6815c8e0 msgr2=0x7f3e680caa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e6815c8e0 0x7f3e680caa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 -- 172.21.15.28:0/660887332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e60009b30 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6d600640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 0x7f3e6815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e6cc00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c430 tx=0x7f3e5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.575+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54011790 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.576+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e540118f0 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.576+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6815e5a0 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.576+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54011ca0 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.576+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6815ff90 con 0x7f3e680cd300 2024-09-13T17:16:26.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.577+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e680ce580 con 0x7f3e680cd300 2024-09-13T17:16:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.578+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e5401c460 con 0x7f3e680cd300 2024-09-13T17:16:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.578+0000 7f3e65600640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 0x7f3e4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.579+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3e54088eb0 con 0x7f3e680cd300 2024-09-13T17:16:26.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.579+0000 7f3e6d600640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 0x7f3e4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.580+0000 7f3e6d600640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 0x7f3e4806d0c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3e6815f7e0 tx=0x7f3e5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.583+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3e540543d0 con 0x7f3e680cd300 2024-09-13T17:16:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.763+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e680b1500 con 0x7f3e680cd300 2024-09-13T17:16:26.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.764+0000 7f3e65600640 1 -- 172.21.15.28:0/660887332 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e540579f0 con 0x7f3e680cd300 2024-09-13T17:16:26.766 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:16:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 msgr2=0x7f3e4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 0x7f3e4806d0c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f3e6815f7e0 tx=0x7f3e5c008040 comp rx=0 tx=0).stop 2024-09-13T17:16:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 msgr2=0x7f3e6815fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f3e5400c430 tx=0x7f3e5400c8f0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 shutdown_connections 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e4806ac00 0x7f3e4806d0c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e6815c8e0 0x7f3e680caa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e680cd300 0x7f3e6815fa50 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 --2- 172.21.15.28:0/660887332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e680cce70 0x7f3e6815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 >> 172.21.15.28:0/660887332 conn(0x7f3e68071a50 msgr2=0x7f3e6806b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 shutdown_connections 2024-09-13T17:16:26.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.766+0000 7f3e6ea00640 1 -- 172.21.15.28:0/660887332 wait complete. 2024-09-13T17:16:26.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:16:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.938+0000 7f97d2600640 1 Processor -- start 2024-09-13T17:16:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.938+0000 7f97d2600640 1 -- start start 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97d2600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97cc070430 0x7f97cc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97d2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97d2600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97cc15ab60 con 0x7f97cc159770 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97d2600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97cc06f920 con 0x7f97cc070430 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97d2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc15b290 con 0x7f97cc06f3f0 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97caa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f97cc070430 0x7f97cc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97cb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f97cc159770 0x7f97cc06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53540/0 (socket says 172.21.15.28:53540) 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.939+0000 7f97cb400640 1 -- 172.21.15.28:0/838589712 learned_addr learned my addr 172.21.15.28:0/838589712 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.940+0000 7f97caa00640 1 -- 172.21.15.28:0/838589712 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 msgr2=0x7f97cc15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.940+0000 7f97caa00640 1 --2- 172.21.15.28:0/838589712 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.940+0000 7f97caa00640 1 -- 172.21.15.28:0/838589712 >> v1:172.21.15.28:6789/0 conn(0x7f97cc159770 legacy=0x7f97cc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.940+0000 7f97caa00640 1 -- 172.21.15.28:0/838589712 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97cc15b570 con 0x7f97cc070430 2024-09-13T17:16:26.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.940+0000 7f97ca000640 1 -- 172.21.15.28:0/838589712 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1130955118 0 0) 0x7f97bc001110 con 0x7f97cc159770 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.941+0000 7f97caa00640 1 --2- 172.21.15.28:0/838589712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc070430 0x7f97cc15bd70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f97b80097a0 tx=0x7f97b802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5571963cd2b802e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.941+0000 7f97ca000640 1 -- 172.21.15.28:0/838589712 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b802ae50 con 0x7f97cc070430 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.941+0000 7f97ca000640 1 -- 172.21.15.28:0/838589712 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b8032440 con 0x7f97cc070430 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.941+0000 7f97ca000640 1 -- 172.21.15.28:0/838589712 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b80327d0 con 0x7f97cc070430 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 -- 172.21.15.28:0/838589712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc070430 msgr2=0x7f97cc15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 --2- 172.21.15.28:0/838589712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc070430 0x7f97cc15bd70 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f97b80097a0 tx=0x7f97b802a100 comp rx=0 tx=0).stop 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 -- 172.21.15.28:0/838589712 shutdown_connections 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 --2- 172.21.15.28:0/838589712 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 --2- 172.21.15.28:0/838589712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc070430 0x7f97cc15bd70 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 -- 172.21.15.28:0/838589712 >> 172.21.15.28:0/838589712 conn(0x7f97cc0b0270 msgr2=0x7f97cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 -- 172.21.15.28:0/838589712 shutdown_connections 2024-09-13T17:16:26.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.942+0000 7f97d2600640 1 -- 172.21.15.28:0/838589712 wait complete. 2024-09-13T17:16:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.943+0000 7f97d2600640 1 Processor -- start 2024-09-13T17:16:26.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 -- start start 2024-09-13T17:16:26.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc147c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 0x7f97cc1507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc1492d0 con 0x7f97cc159770 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc151260 con 0x7f97cc070430 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.944+0000 7f97d2600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97cc0cd0c0 con 0x7f97cc06f3f0 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32954/0 (socket says 172.21.15.28:32954) 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 -- 172.21.15.28:0/4180561423 learned_addr learned my addr 172.21.15.28:0/4180561423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97caa00640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 0x7f97cc1507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cb400640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc147c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 -- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 msgr2=0x7f97cc147c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc147c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 -- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 msgr2=0x7f97cc1507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 0x7f97cc1507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 -- 172.21.15.28:0/4180561423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b8009400 con 0x7f97cc159770 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cb400640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97caa00640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 0x7f97cc1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.945+0000 7f97cbe00640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ba10 tx=0x7f97c400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.946+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c400cb40 con 0x7f97cc159770 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.946+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97c400cca0 con 0x7f97cc159770 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.946+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97cc148930 con 0x7f97cc159770 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.946+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97c4014680 con 0x7f97cc159770 2024-09-13T17:16:26.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.946+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97cc148ef0 con 0x7f97cc159770 2024-09-13T17:16:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.947+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97cc149410 con 0x7f97cc159770 2024-09-13T17:16:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.948+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f97c401d3b0 con 0x7f97cc159770 2024-09-13T17:16:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.949+0000 7f97c3400640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 0x7f97ac06d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.949+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f97c4088f80 con 0x7f97cc159770 2024-09-13T17:16:26.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.949+0000 7f97cb400640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 0x7f97ac06d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:26.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.950+0000 7f97cb400640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 0x7f97ac06d140 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000f60 tx=0x7f97bc007cc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:26.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:26.953+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f97cc149410 con 0x7f97cc159770 2024-09-13T17:16:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.136+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f97cc149410 con 0x7f97cc159770 2024-09-13T17:16:27.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.137+0000 7f97c3400640 1 -- 172.21.15.28:0/4180561423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7f97cc149410 con 0x7f97cc159770 2024-09-13T17:16:27.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:27.140 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:16:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:16:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:16:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:16:27.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:27.141 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:16:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 msgr2=0x7f97ac06d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 0x7f97ac06d140 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f97bc000f60 tx=0x7f97bc007cc0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 msgr2=0x7f97cc150d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f97c400ba10 tx=0x7f97c400bed0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 shutdown_connections 2024-09-13T17:16:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f97ac06ac80 0x7f97ac06d140 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97cc159770 0x7f97cc150d20 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97cc070430 0x7f97cc1507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 --2- 172.21.15.28:0/4180561423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f97cc06f3f0 0x7f97cc147c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 >> 172.21.15.28:0/4180561423 conn(0x7f97cc0b0270 msgr2=0x7f97cc158560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 shutdown_connections 2024-09-13T17:16:27.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.139+0000 7f97d2600640 1 -- 172.21.15.28:0/4180561423 wait complete. 2024-09-13T17:16:27.156 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:16:27.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:16:27.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.315+0000 7fba84400640 1 Processor -- start 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.316+0000 7fba84400640 1 -- start start 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba84400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba7c1695f0 0x7fba7c1699d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba84400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c0b1570 con 0x7fba7c0ceba0 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba84400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7c0b1850 con 0x7fba7c071170 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba84400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7c06f520 con 0x7fba7c1695f0 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba81a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fba7c1695f0 0x7fba7c1699d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba82400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fba7c0ceba0 0x7fba7c070000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53544/0 (socket says 172.21.15.28:53544) 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba82400640 1 -- 172.21.15.28:0/4020587354 learned_addr learned my addr 172.21.15.28:0/4020587354 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.317+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255190529 0 0) 0x7fba7c0b1570 con 0x7fba7c0ceba0 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.585+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba64003500 con 0x7fba7c0ceba0 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.585+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950115584 0 0) 0x7fba7c0b1850 con 0x7fba7c071170 2024-09-13T17:16:27.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba7c0b1570 con 0x7fba7c071170 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1621933961 0 0) 0x7fba64003500 con 0x7fba7c0ceba0 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 >> v2:172.21.15.28:3300/0 conn(0x7fba7c1695f0 msgr2=0x7fba7c1699d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 --2- 172.21.15.28:0/4020587354 >> v2:172.21.15.28:3300/0 conn(0x7fba7c1695f0 0x7fba7c1699d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 >> v1:172.21.15.189:6789/0 conn(0x7fba7c071170 legacy=0x7fba7c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c06f800 con 0x7fba7c0ceba0 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fba740019f0 con 0x7fba7c0ceba0 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 >> v1:172.21.15.28:6789/0 conn(0x7fba7c0ceba0 legacy=0x7fba7c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64004d90 0x7fba64003890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba64003ee0 0x7fba64004360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 0x7fba64005810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c06f520 con 0x7fba64005360 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.586+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba640060f0 con 0x7fba64003ee0 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.587+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba640062b0 con 0x7fba64004d90 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.587+0000 7fba81a00640 1 --2- 172.21.15.28:0/4020587354 >> v2:172.21.15.28:3300/0 conn(0x7fba7c1695f0 0x7fba7c1699d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:27.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.587+0000 7fba82400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 0x7fba64005810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.587+0000 7fba81000640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba64003ee0 0x7fba64004360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.587+0000 7fba81a00640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64004d90 0x7fba64003890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.588+0000 7fba82400640 1 -- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64004d90 msgr2=0x7fba64003890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.588+0000 7fba82400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64004d90 0x7fba64003890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.588+0000 7fba82400640 1 -- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba64003ee0 msgr2=0x7fba64004360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.588+0000 7fba82400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba64003ee0 0x7fba64004360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.588+0000 7fba82400640 1 -- 172.21.15.28:0/4020587354 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64003500 con 0x7fba64005360 2024-09-13T17:16:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.589+0000 7fba82400640 1 -- 172.21.15.28:0/4020587354 reap_dead start 2024-09-13T17:16:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.589+0000 7fba82400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 0x7fba64005810 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a710 tx=0x7fba64004910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2acfc84cb38fbad3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.589+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7402b900 con 0x7fba64005360 2024-09-13T17:16:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.589+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7402ba80 con 0x7fba64005360 2024-09-13T17:16:27.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.589+0000 7fba7be00640 1 -- 172.21.15.28:0/4020587354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7402be30 con 0x7fba64005360 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.590+0000 7fba84400640 1 -- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 msgr2=0x7fba64005810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.590+0000 7fba84400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 0x7fba64005810 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a710 tx=0x7fba64004910 comp rx=0 tx=0).stop 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.591+0000 7fba84400640 1 -- 172.21.15.28:0/4020587354 shutdown_connections 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.591+0000 7fba84400640 1 --2- 172.21.15.28:0/4020587354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba64005360 0x7fba64005810 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.591+0000 7fba84400640 1 -- 172.21.15.28:0/4020587354 >> 172.21.15.28:0/4020587354 conn(0x7fba7c069e80 msgr2=0x7fba7c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.591+0000 7fba84400640 1 -- 172.21.15.28:0/4020587354 shutdown_connections 2024-09-13T17:16:27.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.591+0000 7fba84400640 1 -- 172.21.15.28:0/4020587354 wait complete. 2024-09-13T17:16:27.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.594+0000 7fba84400640 1 Processor -- start 2024-09-13T17:16:27.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.594+0000 7fba84400640 1 -- start start 2024-09-13T17:16:27.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 0x7fba7c15c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 0x7fba7c0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0ad290 con 0x7fba7c0ac8a0 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c159ca0 con 0x7fba7c15bce0 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba84400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c15c830 con 0x7fba64005360 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba81a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba81a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:53678/0 (socket says 172.21.15.28:53678) 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba82400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 0x7fba7c0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba81a00640 1 -- 172.21.15.28:0/3170529605 learned_addr learned my addr 172.21.15.28:0/3170529605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.596+0000 7fba81000640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 0x7fba7c15c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 -- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 msgr2=0x7fba7c15c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 0x7fba7c15c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 -- 172.21.15.28:0/3170529605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 msgr2=0x7fba7c0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:27.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 0x7fba7c0acd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 -- 172.21.15.28:0/3170529605 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7c15cb10 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81000640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 0x7fba7c15c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba82400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 0x7fba7c0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba81a00640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba70011920 tx=0x7fba70011de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.597+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7000e040 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.598+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7000f040 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.598+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7000fe80 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.598+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c159ef0 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.598+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c1664c0 con 0x7fba64005360 2024-09-13T17:16:27.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.599+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c1699d0 con 0x7fba64005360 2024-09-13T17:16:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.600+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fba70004030 con 0x7fba64005360 2024-09-13T17:16:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.601+0000 7fba79600640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 0x7fba5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.601+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba70089490 con 0x7fba64005360 2024-09-13T17:16:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.602+0000 7fba81000640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 0x7fba5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:27.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.602+0000 7fba81000640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 0x7fba5c06d0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0c47a0 tx=0x7fba68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:27.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.606+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fba70054940 con 0x7fba64005360 2024-09-13T17:16:27.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:27.789+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fba7c15cca0 con 0x7fba64005360 2024-09-13T17:16:28.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.173+0000 7fba79600640 1 -- 172.21.15.28:0/3170529605 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7fba700220b0 con 0x7fba64005360 2024-09-13T17:16:28.176 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:16:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 msgr2=0x7fba5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 0x7fba5c06d0c0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fba7c0c47a0 tx=0x7fba68009040 comp rx=0 tx=0).stop 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 msgr2=0x7fba7c201a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fba70011920 tx=0x7fba70011de0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 shutdown_connections 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba5c06ac00 0x7fba5c06d0c0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fba7c0ac8a0 0x7fba7c0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fba7c15bce0 0x7fba7c15c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 --2- 172.21.15.28:0/3170529605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fba64005360 0x7fba7c201a20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 >> 172.21.15.28:0/3170529605 conn(0x7fba7c069e80 msgr2=0x7fba7c0ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 shutdown_connections 2024-09-13T17:16:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.178+0000 7fba84400640 1 -- 172.21.15.28:0/3170529605 wait complete. 2024-09-13T17:16:28.256 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:16:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.420+0000 7f2372200640 1 Processor -- start 2024-09-13T17:16:28.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.420+0000 7f2372200640 1 -- start start 2024-09-13T17:16:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f2372200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f236c15ab60 0x7f236c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f2372200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c06c8b0 con 0x7f236c15a780 2024-09-13T17:16:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f2372200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236c06cb90 con 0x7f236c071490 2024-09-13T17:16:28.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f2372200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236c0cc980 con 0x7f236c15ab60 2024-09-13T17:16:28.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f236b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f236c15a780 0x7f236c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53558/0 (socket says 172.21.15.28:53558) 2024-09-13T17:16:28.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f236b400640 1 -- 172.21.15.28:0/496810435 learned_addr learned my addr 172.21.15.28:0/496810435 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:28.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.421+0000 7f236aa00640 1 --2- 172.21.15.28:0/496810435 >> v2:172.21.15.28:3300/0 conn(0x7f236c15ab60 0x7f236c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.422+0000 7f236aa00640 1 -- 172.21.15.28:0/496810435 >> v1:172.21.15.189:6789/0 conn(0x7f236c071490 legacy=0x7f236c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.422+0000 7f236aa00640 1 -- 172.21.15.28:0/496810435 >> v1:172.21.15.28:6789/0 conn(0x7f236c15a780 legacy=0x7f236c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.422+0000 7f236aa00640 1 -- 172.21.15.28:0/496810435 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236c0ccc60 con 0x7f236c15ab60 2024-09-13T17:16:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.422+0000 7f236a000640 1 -- 172.21.15.28:0/496810435 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768957107 0 0) 0x7f236c06c8b0 con 0x7f236c15a780 2024-09-13T17:16:28.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.423+0000 7f236aa00640 1 --2- 172.21.15.28:0/496810435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c06bb40 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f23580097a0 tx=0x7f235802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330ff965ef3e0129 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.423+0000 7f236a000640 1 -- 172.21.15.28:0/496810435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358032440 con 0x7f236c15ab60 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.423+0000 7f236a000640 1 -- 172.21.15.28:0/496810435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23580325a0 con 0x7f236c15ab60 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.423+0000 7f236a000640 1 -- 172.21.15.28:0/496810435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358032930 con 0x7f236c15ab60 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 -- 172.21.15.28:0/496810435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 msgr2=0x7f236c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 --2- 172.21.15.28:0/496810435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c06bb40 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f23580097a0 tx=0x7f235802a230 comp rx=0 tx=0).stop 2024-09-13T17:16:28.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 -- 172.21.15.28:0/496810435 shutdown_connections 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 --2- 172.21.15.28:0/496810435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c06bb40 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 -- 172.21.15.28:0/496810435 >> 172.21.15.28:0/496810435 conn(0x7f236c0b0270 msgr2=0x7f236c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 -- 172.21.15.28:0/496810435 shutdown_connections 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.424+0000 7f2372200640 1 -- 172.21.15.28:0/496810435 wait complete. 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.426+0000 7f2372200640 1 Processor -- start 2024-09-13T17:16:28.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.426+0000 7f2372200640 1 -- start start 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f236c071490 0x7f236c154090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f236c15a780 0x7f236c0ce3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0b23b0 con 0x7f236c15ab60 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0b17a0 con 0x7f236c15a780 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f2372200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f236c0b1a80 con 0x7f236c071490 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33010/0 (socket says 172.21.15.28:33010) 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 -- 172.21.15.28:0/2640449481 learned_addr learned my addr 172.21.15.28:0/2640449481 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236b400640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f236c071490 0x7f236c154090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236aa00640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f236c15a780 0x7f236c0ce3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:28.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 -- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f236c071490 msgr2=0x7f236c154090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f236c071490 0x7f236c154090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 -- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f236c15a780 msgr2=0x7f236c0ce3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f236c15a780 0x7f236c0ce3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.427+0000 7f236be00640 1 -- 172.21.15.28:0/2640449481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2358009400 con 0x7f236c15ab60 2024-09-13T17:16:28.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f236be00640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f236400c7a0 tx=0x7f236400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364003b70 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2364003cf0 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f236c0b1d60 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2364003ea0 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.428+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f236c1545d0 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.429+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236c154760 con 0x7f236c15ab60 2024-09-13T17:16:28.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.431+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f236400aa90 con 0x7f236c15ab60 2024-09-13T17:16:28.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.431+0000 7f2363400640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 0x7f234c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:28.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.432+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f23640898c0 con 0x7f236c15ab60 2024-09-13T17:16:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.432+0000 7f236b400640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 0x7f234c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:28.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.433+0000 7f236b400640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 0x7f234c06d0f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f235c007f70 tx=0x7f235c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:28.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.435+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2364054da0 con 0x7f236c15ab60 2024-09-13T17:16:28.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:28.618+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f236c0b24f0 con 0x7f236c15ab60 2024-09-13T17:16:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.187+0000 7f2363400640 1 -- 172.21.15.28:0/2640449481 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7f236c0b24f0 con 0x7f236c15ab60 2024-09-13T17:16:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:16:29.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.192+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 msgr2=0x7f234c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.192+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 0x7f234c06d0f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f235c007f70 tx=0x7f235c007f00 comp rx=0 tx=0).stop 2024-09-13T17:16:29.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.193+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 msgr2=0x7f236c0ce8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.193+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f236400c7a0 tx=0x7f236400cc60 comp rx=0 tx=0).stop 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 shutdown_connections 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f234c06ac30 0x7f234c06d0f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f236c15ab60 0x7f236c0ce8e0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f236c15a780 0x7f236c0ce3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 --2- 172.21.15.28:0/2640449481 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f236c071490 0x7f236c154090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.194+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 >> 172.21.15.28:0/2640449481 conn(0x7f236c0b0270 msgr2=0x7f236c06f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:29.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.195+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 shutdown_connections 2024-09-13T17:16:29.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.198+0000 7f2372200640 1 -- 172.21.15.28:0/2640449481 wait complete. 2024-09-13T17:16:29.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:16:29.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.390+0000 7ff085400640 1 Processor -- start 2024-09-13T17:16:29.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.390+0000 7ff085400640 1 -- start start 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 0x7ff080070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015ab60 0x7ff0800712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff08006bb40 0x7ff08006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08006c8b0 con 0x7ff08006bb40 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08006cb90 con 0x7ff08015a780 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.391+0000 7ff085400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800cc980 con 0x7ff08015ab60 2024-09-13T17:16:29.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff08006bb40 0x7ff08006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff08006bb40 0x7ff08006bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33026/0 (socket says 172.21.15.28:33026) 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 -- 172.21.15.28:0/1655725426 learned_addr learned my addr 172.21.15.28:0/1655725426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff075e00640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015ab60 0x7ff0800712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07e800640 1 --2- 172.21.15.28:0/1655725426 >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 0x7ff080070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 -- 172.21.15.28:0/1655725426 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015ab60 msgr2=0x7ff0800712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015ab60 0x7ff0800712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 -- 172.21.15.28:0/1655725426 >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 msgr2=0x7ff080070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 --2- 172.21.15.28:0/1655725426 >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 0x7ff080070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07f200640 1 -- 172.21.15.28:0/1655725426 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0800ccc60 con 0x7ff08006bb40 2024-09-13T17:16:29.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.392+0000 7ff07e800640 1 --2- 172.21.15.28:0/1655725426 >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 0x7ff080070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:29.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.393+0000 7ff07f200640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08006bb40 0x7ff08006bfd0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff0800b1330 tx=0x7ff07802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deffee43eceb20e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.393+0000 7ff07de00640 1 -- 172.21.15.28:0/1655725426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078031600 con 0x7ff08006bb40 2024-09-13T17:16:29.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff07de00640 1 -- 172.21.15.28:0/1655725426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07802d050 con 0x7ff08006bb40 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff07de00640 1 -- 172.21.15.28:0/1655725426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078031980 con 0x7ff08006bb40 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 -- 172.21.15.28:0/1655725426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08006bb40 msgr2=0x7ff08006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08006bb40 0x7ff08006bfd0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7ff0800b1330 tx=0x7ff07802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 -- 172.21.15.28:0/1655725426 shutdown_connections 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08006bb40 0x7ff08006bfd0 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 --2- 172.21.15.28:0/1655725426 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015ab60 0x7ff0800712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 --2- 172.21.15.28:0/1655725426 >> v2:172.21.15.189:3300/0 conn(0x7ff08015a780 0x7ff080070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 -- 172.21.15.28:0/1655725426 >> 172.21.15.28:0/1655725426 conn(0x7ff0800b0270 msgr2=0x7ff0800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 -- 172.21.15.28:0/1655725426 shutdown_connections 2024-09-13T17:16:29.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.394+0000 7ff085400640 1 -- 172.21.15.28:0/1655725426 wait complete. 2024-09-13T17:16:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.396+0000 7ff085400640 1 Processor -- start 2024-09-13T17:16:29.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.396+0000 7ff085400640 1 -- start start 2024-09-13T17:16:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff08006bb40 0x7ff080154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015a780 0x7ff0800abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff080154d60 con 0x7ff08015ab60 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800ac900 con 0x7ff08006bb40 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff085400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0800acbe0 con 0x7ff08015a780 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33028/0 (socket says 172.21.15.28:33028) 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 -- 172.21.15.28:0/908783777 learned_addr learned my addr 172.21.15.28:0/908783777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff075e00640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015a780 0x7ff0800abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07e800640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff08006bb40 0x7ff080154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 -- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015a780 msgr2=0x7ff0800abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015a780 0x7ff0800abeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 -- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff08006bb40 msgr2=0x7ff080154820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff08006bb40 0x7ff080154820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.397+0000 7ff07f200640 1 -- 172.21.15.28:0/908783777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff078009b30 con 0x7ff08015ab60 2024-09-13T17:16:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff07f200640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff078034040 tx=0x7ff078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07803d860 con 0x7ff08015ab60 2024-09-13T17:16:29.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07803d9c0 con 0x7ff08015ab60 2024-09-13T17:16:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07803dd70 con 0x7ff08015ab60 2024-09-13T17:16:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0800acec0 con 0x7ff08015ab60 2024-09-13T17:16:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.398+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0800ad480 con 0x7ff08015ab60 2024-09-13T17:16:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.399+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08006bfd0 con 0x7ff08015ab60 2024-09-13T17:16:29.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.400+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff07803b400 con 0x7ff08015ab60 2024-09-13T17:16:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.401+0000 7ff077400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 0x7ff05c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.401+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff0780aa100 con 0x7ff08015ab60 2024-09-13T17:16:29.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.402+0000 7ff07e800640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 0x7ff05c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.402+0000 7ff07e800640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 0x7ff05c06d0c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff0800afed0 tx=0x7ff070008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.406+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff078075350 con 0x7ff08015ab60 2024-09-13T17:16:29.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.634+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff080070f70 con 0x7ff08015ab60 2024-09-13T17:16:29.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.642+0000 7ff077400640 1 -- 172.21.15.28:0/908783777 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v110) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff078009510 con 0x7ff08015ab60 2024-09-13T17:16:29.646 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 16 and data pool 17 2024-09-13T17:16:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.650+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 msgr2=0x7ff05c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.650+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 0x7ff05c06d0c0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff0800afed0 tx=0x7ff070008040 comp rx=0 tx=0).stop 2024-09-13T17:16:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.650+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 msgr2=0x7ff080157af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.650+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7ff078034040 tx=0x7ff078004a40 comp rx=0 tx=0).stop 2024-09-13T17:16:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 shutdown_connections 2024-09-13T17:16:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff05c06ac00 0x7ff05c06d0c0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff08015ab60 0x7ff080157af0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff08015a780 0x7ff0800abeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 --2- 172.21.15.28:0/908783777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff08006bb40 0x7ff080154820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.651+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 >> 172.21.15.28:0/908783777 conn(0x7ff0800b0270 msgr2=0x7ff0800b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:29.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.653+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 shutdown_connections 2024-09-13T17:16:29.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.654+0000 7ff085400640 1 -- 172.21.15.28:0/908783777 wait complete. 2024-09-13T17:16:29.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.841+0000 7f4dfec00640 1 Processor -- start 2024-09-13T17:16:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.842+0000 7f4dfec00640 1 -- start start 2024-09-13T17:16:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4df806b360 0x7f4df80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfec00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df80b0980 0x7f4df80af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4df80b0d80 con 0x7f4df8154a90 2024-09-13T17:16:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfec00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df806b900 con 0x7f4df80b0980 2024-09-13T17:16:29.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4df806bbe0 con 0x7f4df806b360 2024-09-13T17:16:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfe200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df80b0980 0x7f4df80af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfe200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4df80b0980 0x7f4df80af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33038/0 (socket says 172.21.15.28:33038) 2024-09-13T17:16:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfe200640 1 -- 172.21.15.28:0/3552263556 learned_addr learned my addr 172.21.15.28:0/3552263556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:29.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.843+0000 7f4dfce00640 1 --2- 172.21.15.28:0/3552263556 >> v2:172.21.15.189:3300/0 conn(0x7f4df806b360 0x7f4df80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.844+0000 7f4dfe200640 1 -- 172.21.15.28:0/3552263556 >> v2:172.21.15.189:3300/0 conn(0x7f4df806b360 msgr2=0x7f4df80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.844+0000 7f4dfe200640 1 --2- 172.21.15.28:0/3552263556 >> v2:172.21.15.189:3300/0 conn(0x7f4df806b360 0x7f4df80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.844+0000 7f4dfe200640 1 -- 172.21.15.28:0/3552263556 >> v1:172.21.15.189:6789/0 conn(0x7f4df8154a90 legacy=0x7f4df815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.844+0000 7f4dfe200640 1 -- 172.21.15.28:0/3552263556 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df806bec0 con 0x7f4df80b0980 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4dfe200640 1 --2- 172.21.15.28:0/3552263556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df80af4b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4df813dab0 tx=0x7f4df002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd119bcca9d3bf63 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4df7e00640 1 -- 172.21.15.28:0/3552263556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0031600 con 0x7f4df80b0980 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4df7e00640 1 -- 172.21.15.28:0/3552263556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df002d050 con 0x7f4df80b0980 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4df7e00640 1 -- 172.21.15.28:0/3552263556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df0031980 con 0x7f4df80b0980 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4dfec00640 1 -- 172.21.15.28:0/3552263556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 msgr2=0x7f4df80af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.845+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3552263556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df80af4b0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f4df813dab0 tx=0x7f4df002a730 comp rx=0 tx=0).stop 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 -- 172.21.15.28:0/3552263556 shutdown_connections 2024-09-13T17:16:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3552263556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df80af4b0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3552263556 >> v2:172.21.15.189:3300/0 conn(0x7f4df806b360 0x7f4df80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 -- 172.21.15.28:0/3552263556 >> 172.21.15.28:0/3552263556 conn(0x7f4df8072e40 msgr2=0x7f4df8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 -- 172.21.15.28:0/3552263556 shutdown_connections 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.846+0000 7f4dfec00640 1 -- 172.21.15.28:0/3552263556 wait complete. 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.847+0000 7f4dfec00640 1 Processor -- start 2024-09-13T17:16:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.847+0000 7f4dfec00640 1 -- start start 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4df806b360 0x7f4df813c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df8154a90 0x7f4df8156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df80cd2d0 con 0x7f4df80b0980 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df80ada00 con 0x7f4df8154a90 2024-09-13T17:16:29.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4df80adce0 con 0x7f4df806b360 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.848+0000 7f4dfce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33044/0 (socket says 172.21.15.28:33044) 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 -- 172.21.15.28:0/3668357089 learned_addr learned my addr 172.21.15.28:0/3668357089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfe200640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df8154a90 0x7f4df8156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfd800640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4df806b360 0x7f4df813c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 -- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4df806b360 msgr2=0x7f4df813c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4df806b360 0x7f4df813c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 -- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df8154a90 msgr2=0x7f4df8156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df8154a90 0x7f4df8156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 -- 172.21.15.28:0/3668357089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df0009b30 con 0x7f4df80b0980 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.849+0000 7f4dfce00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f4de400e9b0 tx=0x7f4de400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.850+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de4009670 con 0x7f4df80b0980 2024-09-13T17:16:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.850+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de400c9a0 con 0x7f4df80b0980 2024-09-13T17:16:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.850+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de400cd50 con 0x7f4df80b0980 2024-09-13T17:16:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.850+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df80adfc0 con 0x7f4df80b0980 2024-09-13T17:16:29.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.850+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df813c950 con 0x7f4df80b0980 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.851+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df80008d0 con 0x7f4df80b0980 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.852+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4de4013470 con 0x7f4df80b0980 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.852+0000 7f4df5600640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 0x7f4dd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.853+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4de40895e0 con 0x7f4df80b0980 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.853+0000 7f4dfd800640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 0x7f4dd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:29.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.854+0000 7f4dfd800640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 0x7f4dd806d0c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4dec000f60 tx=0x7f4dec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:29.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:29.858+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4de40547d0 con 0x7f4df80b0980 2024-09-13T17:16:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.047+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4df8073a20 con 0x7f4df80b0980 2024-09-13T17:16:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.048+0000 7f4df5600640 1 -- 172.21.15.28:0/3668357089 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16540 (secure 0 0 0) 0x7f4de4057df0 con 0x7f4df80b0980 2024-09-13T17:16:30.051 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:30.051 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:29.636155+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.050+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 msgr2=0x7f4dd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.050+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 0x7f4dd806d0c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f4dec000f60 tx=0x7f4dec009040 comp rx=0 tx=0).stop 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.050+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 msgr2=0x7f4df8156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.050+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f4de400e9b0 tx=0x7f4de400ee70 comp rx=0 tx=0).stop 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 shutdown_connections 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dd806ac00 0x7f4dd806d0c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4df8154a90 0x7f4df8156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4df80b0980 0x7f4df8156760 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 --2- 172.21.15.28:0/3668357089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4df806b360 0x7f4df813c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 >> 172.21.15.28:0/3668357089 conn(0x7f4df8072e40 msgr2=0x7f4df8159600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 shutdown_connections 2024-09-13T17:16:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.051+0000 7f4dfec00640 1 -- 172.21.15.28:0/3668357089 wait complete. 2024-09-13T17:16:30.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.224+0000 7f92e9600640 1 Processor -- start 2024-09-13T17:16:30.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.224+0000 7f92e9600640 1 -- start start 2024-09-13T17:16:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f92e41592a0 0x7f92e415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e406f970 0x7f92e4159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92e406c2d0 0x7f92e406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e415b0b0 con 0x7f92e406c2d0 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92e406cca0 con 0x7f92e41592a0 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e40b2040 con 0x7f92e406f970 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92e406c2d0 0x7f92e406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92e406c2d0 0x7f92e406c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33058/0 (socket says 172.21.15.28:33058) 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.225+0000 7f92e3400640 1 -- 172.21.15.28:0/78229164 learned_addr learned my addr 172.21.15.28:0/78229164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92da000640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e406f970 0x7f92e4159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e2a00640 1 --2- 172.21.15.28:0/78229164 >> v2:172.21.15.189:3300/0 conn(0x7f92e41592a0 0x7f92e415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e3400640 1 -- 172.21.15.28:0/78229164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e406f970 msgr2=0x7f92e4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e3400640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e406f970 0x7f92e4159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e3400640 1 -- 172.21.15.28:0/78229164 >> v2:172.21.15.189:3300/0 conn(0x7f92e41592a0 msgr2=0x7f92e415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e3400640 1 --2- 172.21.15.28:0/78229164 >> v2:172.21.15.189:3300/0 conn(0x7f92e41592a0 0x7f92e415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.226+0000 7f92e3400640 1 -- 172.21.15.28:0/78229164 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e4159ff0 con 0x7f92e406c2d0 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.227+0000 7f92e3400640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e406c760 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f92e406d370 tx=0x7f92dc02a690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b0908aeb01f390 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.227+0000 7f92e2000640 1 -- 172.21.15.28:0/78229164 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc031600 con 0x7f92e406c2d0 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.227+0000 7f92e2000640 1 -- 172.21.15.28:0/78229164 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92dc02d050 con 0x7f92e406c2d0 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.227+0000 7f92e2000640 1 -- 172.21.15.28:0/78229164 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92dc031980 con 0x7f92e406c2d0 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 -- 172.21.15.28:0/78229164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 msgr2=0x7f92e406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e406c760 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f92e406d370 tx=0x7f92dc02a690 comp rx=0 tx=0).stop 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 -- 172.21.15.28:0/78229164 shutdown_connections 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e406c760 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 --2- 172.21.15.28:0/78229164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e406f970 0x7f92e4159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 --2- 172.21.15.28:0/78229164 >> v2:172.21.15.189:3300/0 conn(0x7f92e41592a0 0x7f92e415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 -- 172.21.15.28:0/78229164 >> 172.21.15.28:0/78229164 conn(0x7f92e4069e80 msgr2=0x7f92e406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 -- 172.21.15.28:0/78229164 shutdown_connections 2024-09-13T17:16:30.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.228+0000 7f92e9600640 1 -- 172.21.15.28:0/78229164 wait complete. 2024-09-13T17:16:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.229+0000 7f92e9600640 1 Processor -- start 2024-09-13T17:16:30.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.229+0000 7f92e9600640 1 -- start start 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e406f970 0x7f92e41f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e41592a0 0x7f92e4156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41f67a0 con 0x7f92e406c2d0 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41f61e0 con 0x7f92e406f970 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.230+0000 7f92e9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92e41f64c0 con 0x7f92e41592a0 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33068/0 (socket says 172.21.15.28:33068) 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 -- 172.21.15.28:0/1817627823 learned_addr learned my addr 172.21.15.28:0/1817627823 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e3400640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e41592a0 0x7f92e4156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92da000640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e406f970 0x7f92e41f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 -- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e41592a0 msgr2=0x7f92e4156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e41592a0 0x7f92e4156c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 -- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e406f970 msgr2=0x7f92e41f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e406f970 0x7f92e41f5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 -- 172.21.15.28:0/1817627823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92dc009b30 con 0x7f92e406c2d0 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.231+0000 7f92e2a00640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f92d400d5f0 tx=0x7f92d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.232+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d40125f0 con 0x7f92e406c2d0 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.232+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92d4012750 con 0x7f92e406c2d0 2024-09-13T17:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.232+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92d4012b00 con 0x7f92e406c2d0 2024-09-13T17:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.232+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92e4148db0 con 0x7f92e406c2d0 2024-09-13T17:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.232+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92e414c680 con 0x7f92e406c2d0 2024-09-13T17:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.233+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92e4148c10 con 0x7f92e406c2d0 2024-09-13T17:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.234+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f92d4012c60 con 0x7f92e406c2d0 2024-09-13T17:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.235+0000 7f92db400640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 0x7f92c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.235+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92d4089710 con 0x7f92e406c2d0 2024-09-13T17:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.235+0000 7f92da000640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 0x7f92c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.236+0000 7f92da000640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 0x7f92c006d0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f92e41f6900 tx=0x7f92cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.239+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f92e4148c10 con 0x7f92e406c2d0 2024-09-13T17:16:30.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.425+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92e4148c10 con 0x7f92e406c2d0 2024-09-13T17:16:30.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.426+0000 7f92db400640 1 -- 172.21.15.28:0/1817627823 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16540 (secure 0 0 0) 0x7f92e4148c10 con 0x7f92e406c2d0 2024-09-13T17:16:30.428 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:30.428 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:29.636155+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 msgr2=0x7f92c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 0x7f92c006d0c0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f92e41f6900 tx=0x7f92cc008040 comp rx=0 tx=0).stop 2024-09-13T17:16:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 msgr2=0x7f92e414c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f92d400d5f0 tx=0x7f92d400dab0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 shutdown_connections 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f92c006ac00 0x7f92c006d0c0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92e41592a0 0x7f92e4156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92e406f970 0x7f92e41f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 --2- 172.21.15.28:0/1817627823 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92e406c2d0 0x7f92e414c140 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 >> 172.21.15.28:0/1817627823 conn(0x7f92e4069e80 msgr2=0x7f92e415c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 shutdown_connections 2024-09-13T17:16:30.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.428+0000 7f92e9600640 1 -- 172.21.15.28:0/1817627823 wait complete. 2024-09-13T17:16:30.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:16:30.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.604+0000 7ff737e00640 1 Processor -- start 2024-09-13T17:16:30.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.604+0000 7ff737e00640 1 -- start start 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff737e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 0x7ff73006a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff737e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff73015a7e0 0x7ff73006c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff737e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7300ce1d0 con 0x7ff73006aa90 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff737e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7300ce4b0 con 0x7ff73015a7e0 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff737e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7300ce790 con 0x7ff73015a400 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff734a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff73006aa90 0x7ff7300708e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53568/0 (socket says 172.21.15.28:53568) 2024-09-13T17:16:30.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff734a00640 1 -- 172.21.15.28:0/3367813189 learned_addr learned my addr 172.21.15.28:0/3367813189 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff735400640 1 --2- 172.21.15.28:0/3367813189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 0x7ff73006a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.605+0000 7ff735e00640 1 --2- 172.21.15.28:0/3367813189 >> v2:172.21.15.189:3300/0 conn(0x7ff73015a7e0 0x7ff73006c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff72fe00640 1 -- 172.21.15.28:0/3367813189 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209050199 0 0) 0x7ff7300ce1d0 con 0x7ff73006aa90 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff72fe00640 1 -- 172.21.15.28:0/3367813189 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718003520 con 0x7ff73006aa90 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff735400640 1 -- 172.21.15.28:0/3367813189 >> v2:172.21.15.189:3300/0 conn(0x7ff73015a7e0 msgr2=0x7ff73006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff735400640 1 --2- 172.21.15.28:0/3367813189 >> v2:172.21.15.189:3300/0 conn(0x7ff73015a7e0 0x7ff73006c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff735400640 1 -- 172.21.15.28:0/3367813189 >> v1:172.21.15.28:6789/0 conn(0x7ff73006aa90 legacy=0x7ff7300708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.606+0000 7ff735400640 1 -- 172.21.15.28:0/3367813189 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300cea70 con 0x7ff73015a400 2024-09-13T17:16:30.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.607+0000 7ff735400640 1 --2- 172.21.15.28:0/3367813189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 0x7ff73006a550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7240097a0 tx=0x7ff72402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad77fbc00a84c4a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.607+0000 7ff72fe00640 1 -- 172.21.15.28:0/3367813189 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724032440 con 0x7ff73015a400 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff72fe00640 1 -- 172.21.15.28:0/3367813189 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7240325a0 con 0x7ff73015a400 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff72fe00640 1 -- 172.21.15.28:0/3367813189 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724032910 con 0x7ff73015a400 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 -- 172.21.15.28:0/3367813189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 msgr2=0x7ff73006a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 --2- 172.21.15.28:0/3367813189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 0x7ff73006a550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7240097a0 tx=0x7ff72402a180 comp rx=0 tx=0).stop 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 -- 172.21.15.28:0/3367813189 shutdown_connections 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 --2- 172.21.15.28:0/3367813189 >> v2:172.21.15.189:3300/0 conn(0x7ff73015a7e0 0x7ff73006c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 --2- 172.21.15.28:0/3367813189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73015a400 0x7ff73006a550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 -- 172.21.15.28:0/3367813189 >> 172.21.15.28:0/3367813189 conn(0x7ff7300b1740 msgr2=0x7ff7300b1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 -- 172.21.15.28:0/3367813189 shutdown_connections 2024-09-13T17:16:30.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.608+0000 7ff737e00640 1 -- 172.21.15.28:0/3367813189 wait complete. 2024-09-13T17:16:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.610+0000 7ff737e00640 1 Processor -- start 2024-09-13T17:16:30.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.610+0000 7ff737e00640 1 -- start start 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73006aa90 0x7ff7301f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73015a400 0x7ff73015d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7301566e0 con 0x7ff73015a7e0 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff730157a20 con 0x7ff73015a400 2024-09-13T17:16:30.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff737e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff730157d00 con 0x7ff73006aa90 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33074/0 (socket says 172.21.15.28:33074) 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735e00640 1 -- 172.21.15.28:0/2036021880 learned_addr learned my addr 172.21.15.28:0/2036021880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735400640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73006aa90 0x7ff7301f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff734a00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73015a400 0x7ff73015d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735e00640 1 -- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73006aa90 msgr2=0x7ff7301f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.611+0000 7ff735e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73006aa90 0x7ff7301f22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff735e00640 1 -- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73015a400 msgr2=0x7ff73015d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff735e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73015a400 0x7ff73015d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff735e00640 1 -- 172.21.15.28:0/2036021880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff724009400 con 0x7ff73015a7e0 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff735e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ff72800efc0 tx=0x7ff72800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff728009ba0 con 0x7ff73015a7e0 2024-09-13T17:16:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72800f040 con 0x7ff73015a7e0 2024-09-13T17:16:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7280146c0 con 0x7ff73015a7e0 2024-09-13T17:16:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7300cf3f0 con 0x7ff73015a7e0 2024-09-13T17:16:30.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.612+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7301582c0 con 0x7ff73015a7e0 2024-09-13T17:16:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.613+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7301584b0 con 0x7ff73015a7e0 2024-09-13T17:16:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.614+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff728014820 con 0x7ff73015a7e0 2024-09-13T17:16:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.615+0000 7ff72d600640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 0x7ff71006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.615+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff728089df0 con 0x7ff73015a7e0 2024-09-13T17:16:30.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.616+0000 7ff735400640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 0x7ff71006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.616+0000 7ff735400640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 0x7ff71006d0c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff724034000 tx=0x7ff72402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.619+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff728054fc0 con 0x7ff73015a7e0 2024-09-13T17:16:30.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:30.833+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff73006a550 con 0x7ff73015a7e0 2024-09-13T17:16:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.652+0000 7ff72d600640 1 -- 172.21.15.28:0/2036021880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v113) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff7280180b0 con 0x7ff73015a7e0 2024-09-13T17:16:31.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 msgr2=0x7ff71006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:31.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 0x7ff71006d0c0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7ff724034000 tx=0x7ff72402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:31.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 msgr2=0x7ff730145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:31.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7ff72800efc0 tx=0x7ff72800c490 comp rx=0 tx=0).stop 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 shutdown_connections 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff71006ac00 0x7ff71006d0c0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff73015a7e0 0x7ff730145be0 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff73015a400 0x7ff73015d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 --2- 172.21.15.28:0/2036021880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff73006aa90 0x7ff7301f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 >> 172.21.15.28:0/2036021880 conn(0x7ff7300b1740 msgr2=0x7ff7300b1b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 shutdown_connections 2024-09-13T17:16:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.655+0000 7ff737e00640 1 -- 172.21.15.28:0/2036021880 wait complete. 2024-09-13T17:16:31.676 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:16:31.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:16:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.837+0000 7f8bfc400640 1 Processor -- start 2024-09-13T17:16:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.837+0000 7f8bfc400640 1 -- start start 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bfc400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bf4070430 0x7f8bf415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bfc400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 0x7f8bf415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bfc400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf415ab60 con 0x7f8bf4159770 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bfc400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf406f920 con 0x7f8bf4070430 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bfc400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf415b290 con 0x7f8bf406f3f0 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bf9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bf4070430 0x7f8bf415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bf9000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8bf4070430 0x7f8bf415bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33082/0 (socket says 172.21.15.28:33082) 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.838+0000 7f8bf9000640 1 -- 172.21.15.28:0/977472474 learned_addr learned my addr 172.21.15.28:0/977472474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bfa400640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 0x7f8bf415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bf9000640 1 -- 172.21.15.28:0/977472474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 msgr2=0x7f8bf415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bf9000640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 0x7f8bf415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bf9000640 1 -- 172.21.15.28:0/977472474 >> v1:172.21.15.189:6789/0 conn(0x7f8bf4159770 legacy=0x7f8bf406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bf9000640 1 -- 172.21.15.28:0/977472474 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf415b570 con 0x7f8bf4070430 2024-09-13T17:16:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.839+0000 7f8bfa400640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 0x7f8bf415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.840+0000 7f8bf9000640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf415bd70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009870 tx=0x7f8be002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690ec3ecec889d84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.840+0000 7f8bf3e00640 1 -- 172.21.15.28:0/977472474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be002ae90 con 0x7f8bf4070430 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.840+0000 7f8bf3e00640 1 -- 172.21.15.28:0/977472474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be0032440 con 0x7f8bf4070430 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.840+0000 7f8bf3e00640 1 -- 172.21.15.28:0/977472474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be00327d0 con 0x7f8bf4070430 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 -- 172.21.15.28:0/977472474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 msgr2=0x7f8bf415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf415bd70 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009870 tx=0x7f8be002a180 comp rx=0 tx=0).stop 2024-09-13T17:16:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 -- 172.21.15.28:0/977472474 shutdown_connections 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf406f3f0 0x7f8bf415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 --2- 172.21.15.28:0/977472474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf415bd70 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 -- 172.21.15.28:0/977472474 >> 172.21.15.28:0/977472474 conn(0x7f8bf40b0270 msgr2=0x7f8bf40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 -- 172.21.15.28:0/977472474 shutdown_connections 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.841+0000 7f8bfc400640 1 -- 172.21.15.28:0/977472474 wait complete. 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.843+0000 7f8bfc400640 1 Processor -- start 2024-09-13T17:16:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.843+0000 7f8bfc400640 1 -- start start 2024-09-13T17:16:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bf406f3f0 0x7f8bf40ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf4159770 0x7f8bf40b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4151080 con 0x7f8bf4070430 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4155280 con 0x7f8bf406f3f0 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfc400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf4155560 con 0x7f8bf4159770 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33098/0 (socket says 172.21.15.28:33098) 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 -- 172.21.15.28:0/257514020 learned_addr learned my addr 172.21.15.28:0/257514020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9a00640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bf406f3f0 0x7f8bf40ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bfa400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf4159770 0x7f8bf40b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 -- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf4159770 msgr2=0x7f8bf40b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf4159770 0x7f8bf40b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 -- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bf406f3f0 msgr2=0x7f8bf40ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bf406f3f0 0x7f8bf40ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.844+0000 7f8bf9000640 1 -- 172.21.15.28:0/257514020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be00094d0 con 0x7f8bf4070430 2024-09-13T17:16:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bf9000640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009870 tx=0x7f8be00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be0030b80 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be0030ce0 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf4155840 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be003d660 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.845+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf40caf30 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.846+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf4155d60 con 0x7f8bf4070430 2024-09-13T17:16:31.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.847+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8be0039040 con 0x7f8bf4070430 2024-09-13T17:16:31.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.848+0000 7f8bf1600640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 0x7f8bd406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:31.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.848+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8be00a9f50 con 0x7f8bf4070430 2024-09-13T17:16:31.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.848+0000 7f8bf9a00640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 0x7f8bd406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:31.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.849+0000 7f8bf9a00640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 0x7f8bd406d0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8be8006fd0 tx=0x7f8be8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:31.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:31.853+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8be0075120 con 0x7f8bf4070430 2024-09-13T17:16:32.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.072+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8bf415ab60 con 0x7f8bf4070430 2024-09-13T17:16:32.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.655+0000 7f8bf1600640 1 -- 172.21.15.28:0/257514020 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v114) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8bf415ab60 con 0x7f8bf4070430 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 msgr2=0x7f8bd406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 0x7f8bd406d0c0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f8be8006fd0 tx=0x7f8be8008040 comp rx=0 tx=0).stop 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 msgr2=0x7f8bf414e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8be0009870 tx=0x7f8be00046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 shutdown_connections 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8bd406ac00 0x7f8bd406d0c0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bf4159770 0x7f8bf40b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bf4070430 0x7f8bf414e950 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 --2- 172.21.15.28:0/257514020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bf406f3f0 0x7f8bf40ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 >> 172.21.15.28:0/257514020 conn(0x7f8bf40b0270 msgr2=0x7f8bf4158850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 shutdown_connections 2024-09-13T17:16:32.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.657+0000 7f8bfc400640 1 -- 172.21.15.28:0/257514020 wait complete. 2024-09-13T17:16:32.676 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:16:32.677 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:32.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.832+0000 7f18a5200640 1 Processor -- start 2024-09-13T17:16:32.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.832+0000 7f18a5200640 1 -- start start 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f18a5200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f18a006f970 0x7f18a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f18a5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f18a006c2d0 0x7f18a006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f18a5200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a006cc70 con 0x7f18a01592a0 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f18a5200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a015b0b0 con 0x7f18a006c2d0 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f18a5200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a0159d90 con 0x7f18a006f970 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f18a01592a0 0x7f18a0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53570/0 (socket says 172.21.15.28:53570) 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189de00640 1 -- 172.21.15.28:0/3756855349 learned_addr learned my addr 172.21.15.28:0/3756855349 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189e800640 1 --2- 172.21.15.28:0/3756855349 >> v2:172.21.15.28:3300/0 conn(0x7f18a006c2d0 0x7f18a006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f1897e00640 1 --2- 172.21.15.28:0/3756855349 >> v2:172.21.15.189:3300/0 conn(0x7f18a006f970 0x7f18a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189e800640 1 -- 172.21.15.28:0/3756855349 >> v2:172.21.15.189:3300/0 conn(0x7f18a006f970 msgr2=0x7f18a015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189e800640 1 --2- 172.21.15.28:0/3756855349 >> v2:172.21.15.189:3300/0 conn(0x7f18a006f970 0x7f18a015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189e800640 1 -- 172.21.15.28:0/3756855349 >> v1:172.21.15.28:6789/0 conn(0x7f18a01592a0 legacy=0x7f18a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189d400640 1 -- 172.21.15.28:0/3756855349 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331099622 0 0) 0x7f18a006cc70 con 0x7f18a01592a0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.834+0000 7f189e800640 1 -- 172.21.15.28:0/3756855349 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a00b2040 con 0x7f18a006c2d0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.835+0000 7f189e800640 1 --2- 172.21.15.28:0/3756855349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006c2d0 0x7f18a006c730 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f18a006ac10 tx=0x7f189802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cde21d2038d95b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f189d400640 1 -- 172.21.15.28:0/3756855349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189802aea0 con 0x7f18a006c2d0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f189d400640 1 -- 172.21.15.28:0/3756855349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189802d050 con 0x7f18a006c2d0 2024-09-13T17:16:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f189d400640 1 -- 172.21.15.28:0/3756855349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898007960 con 0x7f18a006c2d0 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 -- 172.21.15.28:0/3756855349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006c2d0 msgr2=0x7f18a006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 --2- 172.21.15.28:0/3756855349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006c2d0 0x7f18a006c730 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f18a006ac10 tx=0x7f189802a680 comp rx=0 tx=0).stop 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 -- 172.21.15.28:0/3756855349 shutdown_connections 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 --2- 172.21.15.28:0/3756855349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006c2d0 0x7f18a006c730 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 --2- 172.21.15.28:0/3756855349 >> v2:172.21.15.189:3300/0 conn(0x7f18a006f970 0x7f18a015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 -- 172.21.15.28:0/3756855349 >> 172.21.15.28:0/3756855349 conn(0x7f18a0069e80 msgr2=0x7f18a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 -- 172.21.15.28:0/3756855349 shutdown_connections 2024-09-13T17:16:32.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.836+0000 7f18a5200640 1 -- 172.21.15.28:0/3756855349 wait complete. 2024-09-13T17:16:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.838+0000 7f18a5200640 1 Processor -- start 2024-09-13T17:16:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.838+0000 7f18a5200640 1 -- start start 2024-09-13T17:16:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a006c2d0 0x7f18a014b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 0x7f18a01f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a014bd70 con 0x7f18a006f970 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a014cc00 con 0x7f18a006c2d0 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f18a5200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a014e040 con 0x7f18a01592a0 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f1897e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.839+0000 7f1897e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33128/0 (socket says 172.21.15.28:33128) 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 -- 172.21.15.28:0/2907725367 learned_addr learned my addr 172.21.15.28:0/2907725367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f189de00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a006c2d0 0x7f18a014b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f189e800640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 0x7f18a01f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 -- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 msgr2=0x7f18a01f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 0x7f18a01f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 -- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a006c2d0 msgr2=0x7f18a014b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a006c2d0 0x7f18a014b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:32.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 -- 172.21.15.28:0/2907725367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1898009a70 con 0x7f18a006f970 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f189e800640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 0x7f18a01f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1897e00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f188800d800 tx=0x7f188800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.840+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188800b6c0 con 0x7f18a006f970 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.841+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188800b820 con 0x7f18a006f970 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.841+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188800bbb0 con 0x7f18a006f970 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.841+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a014e320 con 0x7f18a006f970 2024-09-13T17:16:32.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.841+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a014e600 con 0x7f18a006f970 2024-09-13T17:16:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.842+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a00712f0 con 0x7f18a006f970 2024-09-13T17:16:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.843+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f188800bd10 con 0x7f18a006f970 2024-09-13T17:16:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.843+0000 7f1896000640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 0x7f187c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:32.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.844+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18880898c0 con 0x7f18a006f970 2024-09-13T17:16:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.844+0000 7f189de00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 0x7f187c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:32.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.845+0000 7f189de00640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 0x7f187c06d0c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1890007fd0 tx=0x7f1890007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:32.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:32.848+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1888054a90 con 0x7f18a006f970 2024-09-13T17:16:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.062+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18a0067720 con 0x7f18a006f970 2024-09-13T17:16:33.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.063+0000 7f1896000640 1 -- 172.21.15.28:0/2907725367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7f1888089070 con 0x7f18a006f970 2024-09-13T17:16:33.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:33.065 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:16:33.065 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 msgr2=0x7f187c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 0x7f187c06d0c0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f1890007fd0 tx=0x7f1890007de0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 msgr2=0x7f18a014bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f188800d800 tx=0x7f188800dcc0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 shutdown_connections 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f187c06ac00 0x7f187c06d0c0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a01592a0 0x7f18a01f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a006f970 0x7f18a014bf30 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 --2- 172.21.15.28:0/2907725367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a006c2d0 0x7f18a014b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 >> 172.21.15.28:0/2907725367 conn(0x7f18a0069e80 msgr2=0x7f18a0159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:33.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 shutdown_connections 2024-09-13T17:16:33.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.065+0000 7f18a5200640 1 -- 172.21.15.28:0/2907725367 wait complete. 2024-09-13T17:16:33.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:16:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.237+0000 7fb484a00640 1 Processor -- start 2024-09-13T17:16:33.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.237+0000 7fb484a00640 1 -- start start 2024-09-13T17:16:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.238+0000 7fb484a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4800d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.238+0000 7fb484a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4800b4930 0x7fb4800b8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.238+0000 7fb484a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4800b8750 con 0x7fb4800d0b00 2024-09-13T17:16:33.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.238+0000 7fb484a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4800d0f40 con 0x7fb4800b4930 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.238+0000 7fb484a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4800d1190 con 0x7fb48015c870 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4800b4930 0x7fb4800b8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4800b4930 0x7fb4800b8210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33144/0 (socket says 172.21.15.28:33144) 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 -- 172.21.15.28:0/4219730197 learned_addr learned my addr 172.21.15.28:0/4219730197 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47e800640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4800d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 -- 172.21.15.28:0/4219730197 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 msgr2=0x7fb4800d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4800d2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 -- 172.21.15.28:0/4219730197 >> v1:172.21.15.189:6789/0 conn(0x7fb4800d0b00 legacy=0x7fb4800b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:33.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.239+0000 7fb47f200640 1 -- 172.21.15.28:0/4219730197 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800d1470 con 0x7fb4800b4930 2024-09-13T17:16:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.240+0000 7fb47f200640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb4800b8210 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb478009e50 tx=0x7fb47802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfa49bf5e1a1619 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.240+0000 7fb47d400640 1 -- 172.21.15.28:0/4219730197 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478031600 con 0x7fb4800b4930 2024-09-13T17:16:33.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb47d400640 1 -- 172.21.15.28:0/4219730197 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb47802d050 con 0x7fb4800b4930 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb47d400640 1 -- 172.21.15.28:0/4219730197 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780319a0 con 0x7fb4800b4930 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 -- 172.21.15.28:0/4219730197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 msgr2=0x7fb4800b8210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb4800b8210 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7fb478009e50 tx=0x7fb47802a7c0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 -- 172.21.15.28:0/4219730197 shutdown_connections 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb4800b8210 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 --2- 172.21.15.28:0/4219730197 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4800d2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 -- 172.21.15.28:0/4219730197 >> 172.21.15.28:0/4219730197 conn(0x7fb4800cf400 msgr2=0x7fb4800cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 -- 172.21.15.28:0/4219730197 shutdown_connections 2024-09-13T17:16:33.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.241+0000 7fb484a00640 1 -- 172.21.15.28:0/4219730197 wait complete. 2024-09-13T17:16:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.243+0000 7fb484a00640 1 Processor -- start 2024-09-13T17:16:33.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.243+0000 7fb484a00640 1 -- start start 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 0x7fb4800c63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4801f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb480148a50 con 0x7fb4800b4930 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48015e430 con 0x7fb4800d0b00 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb484a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb48015e710 con 0x7fb48015c870 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb47e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb47e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33146/0 (socket says 172.21.15.28:33146) 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb47e800640 1 -- 172.21.15.28:0/1650636443 learned_addr learned my addr 172.21.15.28:0/1650636443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb47f200640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4801f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.244+0000 7fb47de00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 0x7fb4800c63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 -- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 msgr2=0x7fb4801f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4801f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 -- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 msgr2=0x7fb4800c63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 0x7fb4800c63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 -- 172.21.15.28:0/1650636443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478009b30 con 0x7fb4800b4930 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47de00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 0x7fb4800c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:33.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb47e800640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb47000d5f0 tx=0x7fb47000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4700125f0 con 0x7fb4800b4930 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb470012750 con 0x7fb4800b4930 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470012b00 con 0x7fb4800b4930 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.245+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48015e9f0 con 0x7fb4800b4930 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.246+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48015efb0 con 0x7fb4800b4930 2024-09-13T17:16:33.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.246+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48015f140 con 0x7fb4800b4930 2024-09-13T17:16:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.248+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb470012c60 con 0x7fb4800b4930 2024-09-13T17:16:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.248+0000 7fb476a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 0x7fb46006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.248+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb470089830 con 0x7fb4800b4930 2024-09-13T17:16:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.249+0000 7fb47de00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 0x7fb46006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.250+0000 7fb47de00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 0x7fb46006d0c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb46c007fd0 tx=0x7fb46c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.253+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb470054970 con 0x7fb4800b4930 2024-09-13T17:16:33.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.434+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4800b8210 con 0x7fb4800b4930 2024-09-13T17:16:33.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.435+0000 7fb476a00640 1 -- 172.21.15.28:0/1650636443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16527 (secure 0 0 0) 0x7fb470057f90 con 0x7fb4800b4930 2024-09-13T17:16:33.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:33.438 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":125,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:30.639092+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:16:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 msgr2=0x7fb46006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 0x7fb46006d0c0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fb46c007fd0 tx=0x7fb46c009040 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 msgr2=0x7fb48015f4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7fb47000d5f0 tx=0x7fb47000dab0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 shutdown_connections 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb46006ac00 0x7fb46006d0c0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb48015c870 0x7fb4801f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4800d0b00 0x7fb4800c63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 --2- 172.21.15.28:0/1650636443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4800b4930 0x7fb48015f4d0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 >> 172.21.15.28:0/1650636443 conn(0x7fb4800cf400 msgr2=0x7fb48015a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.437+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 shutdown_connections 2024-09-13T17:16:33.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.438+0000 7fb484a00640 1 -- 172.21.15.28:0/1650636443 wait complete. 2024-09-13T17:16:33.456 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:16:33.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:16:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.609+0000 7f98b9200640 1 Processor -- start 2024-09-13T17:16:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.609+0000 7f98b9200640 1 -- start start 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98b4159440 0x7f98b40b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b40b29f0 0x7f98b40b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b40c3850 con 0x7f98b4159820 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b415a930 con 0x7f98b4159440 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4159d20 con 0x7f98b40b29f0 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98b3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f98b4159440 0x7f98b40b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98aaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98b4159820 0x7f98b40b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53576/0 (socket says 172.21.15.28:53576) 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.610+0000 7f98aaa00640 1 -- 172.21.15.28:0/4134400464 learned_addr learned my addr 172.21.15.28:0/4134400464 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b3e00640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b40b29f0 0x7f98b40b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b3400640 1 -- 172.21.15.28:0/4134400464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b40b29f0 msgr2=0x7f98b40b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b3400640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b40b29f0 0x7f98b40b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b3400640 1 -- 172.21.15.28:0/4134400464 >> v1:172.21.15.28:6789/0 conn(0x7f98b4159820 legacy=0x7f98b40b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b3400640 1 -- 172.21.15.28:0/4134400464 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b415a000 con 0x7f98b4159440 2024-09-13T17:16:33.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.611+0000 7f98b2a00640 1 -- 172.21.15.28:0/4134400464 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471543960 0 0) 0x7f98b40c3850 con 0x7f98b4159820 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.612+0000 7f98b3400640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4159440 0x7f98b40b0440 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f98a40097d0 tx=0x7f98a402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9647611cf7564f13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.612+0000 7f98b2a00640 1 -- 172.21.15.28:0/4134400464 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4032440 con 0x7f98b4159440 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.612+0000 7f98b2a00640 1 -- 172.21.15.28:0/4134400464 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a40325a0 con 0x7f98b4159440 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.612+0000 7f98b2a00640 1 -- 172.21.15.28:0/4134400464 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4032930 con 0x7f98b4159440 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 -- 172.21.15.28:0/4134400464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4159440 msgr2=0x7f98b40b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4159440 0x7f98b40b0440 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f98a40097d0 tx=0x7f98a402a250 comp rx=0 tx=0).stop 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 -- 172.21.15.28:0/4134400464 shutdown_connections 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b40b29f0 0x7f98b40b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 --2- 172.21.15.28:0/4134400464 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b4159440 0x7f98b40b0440 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 -- 172.21.15.28:0/4134400464 >> 172.21.15.28:0/4134400464 conn(0x7f98b4072e40 msgr2=0x7f98b4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 -- 172.21.15.28:0/4134400464 shutdown_connections 2024-09-13T17:16:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.613+0000 7f98b9200640 1 -- 172.21.15.28:0/4134400464 wait complete. 2024-09-13T17:16:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.615+0000 7f98b9200640 1 Processor -- start 2024-09-13T17:16:33.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.615+0000 7f98b9200640 1 -- start start 2024-09-13T17:16:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 0x7f98b40c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b4159820 0x7f98b4153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40c6b40 con 0x7f98b40b29f0 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b40c51c0 con 0x7f98b4159440 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b4156820 con 0x7f98b4159820 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33152/0 (socket says 172.21.15.28:33152) 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 -- 172.21.15.28:0/1420761406 learned_addr learned my addr 172.21.15.28:0/1420761406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98aaa00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 0x7f98b40c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3e00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b4159820 0x7f98b4153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 -- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b4159820 msgr2=0x7f98b4153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b4159820 0x7f98b4153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 -- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 msgr2=0x7f98b40c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 0x7f98b40c46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98b3400640 1 -- 172.21.15.28:0/1420761406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a4009400 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.616+0000 7f98aaa00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 0x7f98b40c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98b3400640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f98a40098f0 tx=0x7f98a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4032440 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a4030b80 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b40c4ee0 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4006e10 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.617+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b4156da0 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.618+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b40b22e0 con 0x7f98b40b29f0 2024-09-13T17:16:33.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.619+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98a4039040 con 0x7f98b40b29f0 2024-09-13T17:16:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.620+0000 7f98abe00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 0x7f989006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.620+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a40a9e70 con 0x7f98b40b29f0 2024-09-13T17:16:33.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.621+0000 7f98aaa00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 0x7f989006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:33.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.621+0000 7f98aaa00640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 0x7f989006d0c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f98b415dbf0 tx=0x7f989c007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:33.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.624+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98a4074fb0 con 0x7f98b40b29f0 2024-09-13T17:16:33.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.802+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98b40ce190 con 0x7f989006ac00 2024-09-13T17:16:33.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.811+0000 7f98abe00640 1 -- 172.21.15.28:0/1420761406 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289157 (secure 0 0 0) 0x7f98b40ce190 con 0x7f989006ac00 2024-09-13T17:16:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:33.819 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:16:33.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 msgr2=0x7f989006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 0x7f989006d0c0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f98b415dbf0 tx=0x7f989c007ca0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 msgr2=0x7f98b41f20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f98a40098f0 tx=0x7f98a40046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 shutdown_connections 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f989006ac00 0x7f989006d0c0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98b4159820 0x7f98b4153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98b4159440 0x7f98b40c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 --2- 172.21.15.28:0/1420761406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98b40b29f0 0x7f98b41f20c0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 >> 172.21.15.28:0/1420761406 conn(0x7f98b4072e40 msgr2=0x7f98b40b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 shutdown_connections 2024-09-13T17:16:33.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:33.816+0000 7f98b9200640 1 -- 172.21.15.28:0/1420761406 wait complete. 2024-09-13T17:16:33.833 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-13T17:16:33.013199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485284,"kb_used_data":13344,"kb_used_omap":12,"kb_used_meta":471795,"kb_avail":754489436,"statfs":{"total":773094113280,"available":772597182464,"internally_reserved":0,"allocated":13664256,"data_stored":6630174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":483118649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.841094"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648379+0000","last_change":"2024-09-13T17:16:29.665764+0000","last_active":"2024-09-13T17:16:30.648379+0000","last_peered":"2024-09-13T17:16:30.648379+0000","last_clean":"2024-09-13T17:16:30.648379+0000","last_became_active":"2024-09-13T17:16:29.665522+0000","last_became_peered":"2024-09-13T17:16:29.665522+0000","last_unstale":"2024-09-13T17:16:30.648379+0000","last_undegraded":"2024-09-13T17:16:30.648379+0000","last_fullsized":"2024-09-13T17:16:30.648379+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:48.512922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649813+0000","last_change":"2024-09-13T17:16:29.661453+0000","last_active":"2024-09-13T17:16:30.649813+0000","last_peered":"2024-09-13T17:16:30.649813+0000","last_clean":"2024-09-13T17:16:30.649813+0000","last_became_active":"2024-09-13T17:16:29.661189+0000","last_became_peered":"2024-09-13T17:16:29.661189+0000","last_unstale":"2024-09-13T17:16:30.649813+0000","last_undegraded":"2024-09-13T17:16:30.649813+0000","last_fullsized":"2024-09-13T17:16:30.649813+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:30.228156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412057+0000","last_change":"2024-09-13T17:16:29.665392+0000","last_active":"2024-09-13T17:16:31.412057+0000","last_peered":"2024-09-13T17:16:31.412057+0000","last_clean":"2024-09-13T17:16:31.412057+0000","last_became_active":"2024-09-13T17:16:29.664898+0000","last_became_peered":"2024-09-13T17:16:29.664898+0000","last_unstale":"2024-09-13T17:16:31.412057+0000","last_undegraded":"2024-09-13T17:16:31.412057+0000","last_fullsized":"2024-09-13T17:16:31.412057+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:49.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647709+0000","last_change":"2024-09-13T17:16:29.659417+0000","last_active":"2024-09-13T17:16:30.647709+0000","last_peered":"2024-09-13T17:16:30.647709+0000","last_clean":"2024-09-13T17:16:30.647709+0000","last_became_active":"2024-09-13T17:16:29.659095+0000","last_became_peered":"2024-09-13T17:16:29.659095+0000","last_unstale":"2024-09-13T17:16:30.647709+0000","last_undegraded":"2024-09-13T17:16:30.647709+0000","last_fullsized":"2024-09-13T17:16:30.647709+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:21.799379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485700+0000","last_change":"2024-09-13T17:16:29.658880+0000","last_active":"2024-09-13T17:16:31.485700+0000","last_peered":"2024-09-13T17:16:31.485700+0000","last_clean":"2024-09-13T17:16:31.485700+0000","last_became_active":"2024-09-13T17:16:29.658628+0000","last_became_peered":"2024-09-13T17:16:29.658628+0000","last_unstale":"2024-09-13T17:16:31.485700+0000","last_undegraded":"2024-09-13T17:16:31.485700+0000","last_fullsized":"2024-09-13T17:16:31.485700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:45.435256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249562+0000","last_change":"2024-09-13T17:16:29.654544+0000","last_active":"2024-09-13T17:16:32.249562+0000","last_peered":"2024-09-13T17:16:32.249562+0000","last_clean":"2024-09-13T17:16:32.249562+0000","last_became_active":"2024-09-13T17:16:29.654278+0000","last_became_peered":"2024-09-13T17:16:29.654278+0000","last_unstale":"2024-09-13T17:16:32.249562+0000","last_undegraded":"2024-09-13T17:16:32.249562+0000","last_fullsized":"2024-09-13T17:16:32.249562+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:40.943816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649421+0000","last_change":"2024-09-13T17:16:29.660123+0000","last_active":"2024-09-13T17:16:30.649421+0000","last_peered":"2024-09-13T17:16:30.649421+0000","last_clean":"2024-09-13T17:16:30.649421+0000","last_became_active":"2024-09-13T17:16:29.658010+0000","last_became_peered":"2024-09-13T17:16:29.658010+0000","last_unstale":"2024-09-13T17:16:30.649421+0000","last_undegraded":"2024-09-13T17:16:30.649421+0000","last_fullsized":"2024-09-13T17:16:30.649421+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:56.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411893+0000","last_change":"2024-09-13T17:16:29.651350+0000","last_active":"2024-09-13T17:16:31.411893+0000","last_peered":"2024-09-13T17:16:31.411893+0000","last_clean":"2024-09-13T17:16:31.411893+0000","last_became_active":"2024-09-13T17:16:29.651046+0000","last_became_peered":"2024-09-13T17:16:29.651046+0000","last_unstale":"2024-09-13T17:16:31.411893+0000","last_undegraded":"2024-09-13T17:16:31.411893+0000","last_fullsized":"2024-09-13T17:16:31.411893+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.611849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648446+0000","last_change":"2024-09-13T17:16:29.658358+0000","last_active":"2024-09-13T17:16:30.648446+0000","last_peered":"2024-09-13T17:16:30.648446+0000","last_clean":"2024-09-13T17:16:30.648446+0000","last_became_active":"2024-09-13T17:16:29.657946+0000","last_became_peered":"2024-09-13T17:16:29.657946+0000","last_unstale":"2024-09-13T17:16:30.648446+0000","last_undegraded":"2024-09-13T17:16:30.648446+0000","last_fullsized":"2024-09-13T17:16:30.648446+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:08.676017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648242+0000","last_change":"2024-09-13T17:16:29.665917+0000","last_active":"2024-09-13T17:16:30.648242+0000","last_peered":"2024-09-13T17:16:30.648242+0000","last_clean":"2024-09-13T17:16:30.648242+0000","last_became_active":"2024-09-13T17:16:29.665468+0000","last_became_peered":"2024-09-13T17:16:29.665468+0000","last_unstale":"2024-09-13T17:16:30.648242+0000","last_undegraded":"2024-09-13T17:16:30.648242+0000","last_fullsized":"2024-09-13T17:16:30.648242+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:34.333738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649733+0000","last_change":"2024-09-13T17:16:29.662854+0000","last_active":"2024-09-13T17:16:30.649733+0000","last_peered":"2024-09-13T17:16:30.649733+0000","last_clean":"2024-09-13T17:16:30.649733+0000","last_became_active":"2024-09-13T17:16:29.662573+0000","last_became_peered":"2024-09-13T17:16:29.662573+0000","last_unstale":"2024-09-13T17:16:30.649733+0000","last_undegraded":"2024-09-13T17:16:30.649733+0000","last_fullsized":"2024-09-13T17:16:30.649733+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:13.331922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647801+0000","last_change":"2024-09-13T17:16:29.658722+0000","last_active":"2024-09-13T17:16:30.647801+0000","last_peered":"2024-09-13T17:16:30.647801+0000","last_clean":"2024-09-13T17:16:30.647801+0000","last_became_active":"2024-09-13T17:16:29.658502+0000","last_became_peered":"2024-09-13T17:16:29.658502+0000","last_unstale":"2024-09-13T17:16:30.647801+0000","last_undegraded":"2024-09-13T17:16:30.647801+0000","last_fullsized":"2024-09-13T17:16:30.647801+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:52.038064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412226+0000","last_change":"2024-09-13T17:16:29.666592+0000","last_active":"2024-09-13T17:16:31.412226+0000","last_peered":"2024-09-13T17:16:31.412226+0000","last_clean":"2024-09-13T17:16:31.412226+0000","last_became_active":"2024-09-13T17:16:29.663240+0000","last_became_peered":"2024-09-13T17:16:29.663240+0000","last_unstale":"2024-09-13T17:16:31.412226+0000","last_undegraded":"2024-09-13T17:16:31.412226+0000","last_fullsized":"2024-09-13T17:16:31.412226+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:32.791598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411904+0000","last_change":"2024-09-13T17:16:29.657258+0000","last_active":"2024-09-13T17:16:31.411904+0000","last_peered":"2024-09-13T17:16:31.411904+0000","last_clean":"2024-09-13T17:16:31.411904+0000","last_became_active":"2024-09-13T17:16:29.656990+0000","last_became_peered":"2024-09-13T17:16:29.656990+0000","last_unstale":"2024-09-13T17:16:31.411904+0000","last_undegraded":"2024-09-13T17:16:31.411904+0000","last_fullsized":"2024-09-13T17:16:31.411904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:01:21.413522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249404+0000","last_change":"2024-09-13T17:16:29.652040+0000","last_active":"2024-09-13T17:16:32.249404+0000","last_peered":"2024-09-13T17:16:32.249404+0000","last_clean":"2024-09-13T17:16:32.249404+0000","last_became_active":"2024-09-13T17:16:29.651764+0000","last_became_peered":"2024-09-13T17:16:29.651764+0000","last_unstale":"2024-09-13T17:16:32.249404+0000","last_undegraded":"2024-09-13T17:16:32.249404+0000","last_fullsized":"2024-09-13T17:16:32.249404+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:50.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647370+0000","last_change":"2024-09-13T17:16:29.651488+0000","last_active":"2024-09-13T17:16:30.647370+0000","last_peered":"2024-09-13T17:16:30.647370+0000","last_clean":"2024-09-13T17:16:30.647370+0000","last_became_active":"2024-09-13T17:16:29.651248+0000","last_became_peered":"2024-09-13T17:16:29.651248+0000","last_unstale":"2024-09-13T17:16:30.647370+0000","last_undegraded":"2024-09-13T17:16:30.647370+0000","last_fullsized":"2024-09-13T17:16:30.647370+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:49.880733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649800+0000","last_change":"2024-09-13T17:16:29.666742+0000","last_active":"2024-09-13T17:16:30.649800+0000","last_peered":"2024-09-13T17:16:30.649800+0000","last_clean":"2024-09-13T17:16:30.649800+0000","last_became_active":"2024-09-13T17:16:29.665315+0000","last_became_peered":"2024-09-13T17:16:29.665315+0000","last_unstale":"2024-09-13T17:16:30.649800+0000","last_undegraded":"2024-09-13T17:16:30.649800+0000","last_fullsized":"2024-09-13T17:16:30.649800+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:19.954724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002896+0000","last_change":"2024-09-13T17:16:29.666635+0000","last_active":"2024-09-13T17:16:32.002896+0000","last_peered":"2024-09-13T17:16:32.002896+0000","last_clean":"2024-09-13T17:16:32.002896+0000","last_became_active":"2024-09-13T17:16:29.666335+0000","last_became_peered":"2024-09-13T17:16:29.666335+0000","last_unstale":"2024-09-13T17:16:32.002896+0000","last_undegraded":"2024-09-13T17:16:32.002896+0000","last_fullsized":"2024-09-13T17:16:32.002896+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:57.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412148+0000","last_change":"2024-09-13T17:16:29.665084+0000","last_active":"2024-09-13T17:16:31.412148+0000","last_peered":"2024-09-13T17:16:31.412148+0000","last_clean":"2024-09-13T17:16:31.412148+0000","last_became_active":"2024-09-13T17:16:29.662135+0000","last_became_peered":"2024-09-13T17:16:29.662135+0000","last_unstale":"2024-09-13T17:16:31.412148+0000","last_undegraded":"2024-09-13T17:16:31.412148+0000","last_fullsized":"2024-09-13T17:16:31.412148+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:31.353568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649741+0000","last_change":"2024-09-13T17:16:29.659670+0000","last_active":"2024-09-13T17:16:30.649741+0000","last_peered":"2024-09-13T17:16:30.649741+0000","last_clean":"2024-09-13T17:16:30.649741+0000","last_became_active":"2024-09-13T17:16:29.659241+0000","last_became_peered":"2024-09-13T17:16:29.659241+0000","last_unstale":"2024-09-13T17:16:30.649741+0000","last_undegraded":"2024-09-13T17:16:30.649741+0000","last_fullsized":"2024-09-13T17:16:30.649741+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:09.032346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649835+0000","last_change":"2024-09-13T17:16:29.666308+0000","last_active":"2024-09-13T17:16:30.649835+0000","last_peered":"2024-09-13T17:16:30.649835+0000","last_clean":"2024-09-13T17:16:30.649835+0000","last_became_active":"2024-09-13T17:16:29.665879+0000","last_became_peered":"2024-09-13T17:16:29.665879+0000","last_unstale":"2024-09-13T17:16:30.649835+0000","last_undegraded":"2024-09-13T17:16:30.649835+0000","last_fullsized":"2024-09-13T17:16:30.649835+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:53.523884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863367+0000","last_change":"2024-09-13T17:16:29.659207+0000","last_active":"2024-09-13T17:16:31.863367+0000","last_peered":"2024-09-13T17:16:31.863367+0000","last_clean":"2024-09-13T17:16:31.863367+0000","last_became_active":"2024-09-13T17:16:29.656586+0000","last_became_peered":"2024-09-13T17:16:29.656586+0000","last_unstale":"2024-09-13T17:16:31.863367+0000","last_undegraded":"2024-09-13T17:16:31.863367+0000","last_fullsized":"2024-09-13T17:16:31.863367+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:13.845396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648855+0000","last_change":"2024-09-13T17:16:29.659836+0000","last_active":"2024-09-13T17:16:30.648855+0000","last_peered":"2024-09-13T17:16:30.648855+0000","last_clean":"2024-09-13T17:16:30.648855+0000","last_became_active":"2024-09-13T17:16:29.659520+0000","last_became_peered":"2024-09-13T17:16:29.659520+0000","last_unstale":"2024-09-13T17:16:30.648855+0000","last_undegraded":"2024-09-13T17:16:30.648855+0000","last_fullsized":"2024-09-13T17:16:30.648855+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:45.563984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863085+0000","last_change":"2024-09-13T17:16:29.657067+0000","last_active":"2024-09-13T17:16:31.863085+0000","last_peered":"2024-09-13T17:16:31.863085+0000","last_clean":"2024-09-13T17:16:31.863085+0000","last_became_active":"2024-09-13T17:16:29.656801+0000","last_became_peered":"2024-09-13T17:16:29.656801+0000","last_unstale":"2024-09-13T17:16:31.863085+0000","last_undegraded":"2024-09-13T17:16:31.863085+0000","last_fullsized":"2024-09-13T17:16:31.863085+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:56.649937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649891+0000","last_change":"2024-09-13T17:16:29.664265+0000","last_active":"2024-09-13T17:16:30.649891+0000","last_peered":"2024-09-13T17:16:30.649891+0000","last_clean":"2024-09-13T17:16:30.649891+0000","last_became_active":"2024-09-13T17:16:29.662613+0000","last_became_peered":"2024-09-13T17:16:29.662613+0000","last_unstale":"2024-09-13T17:16:30.649891+0000","last_undegraded":"2024-09-13T17:16:30.649891+0000","last_fullsized":"2024-09-13T17:16:30.649891+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:52.806265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648271+0000","last_change":"2024-09-13T17:16:29.654771+0000","last_active":"2024-09-13T17:16:30.648271+0000","last_peered":"2024-09-13T17:16:30.648271+0000","last_clean":"2024-09-13T17:16:30.648271+0000","last_became_active":"2024-09-13T17:16:29.654475+0000","last_became_peered":"2024-09-13T17:16:29.654475+0000","last_unstale":"2024-09-13T17:16:30.648271+0000","last_undegraded":"2024-09-13T17:16:30.648271+0000","last_fullsized":"2024-09-13T17:16:30.648271+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:23.975549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249614+0000","last_change":"2024-09-13T17:16:29.652225+0000","last_active":"2024-09-13T17:16:32.249614+0000","last_peered":"2024-09-13T17:16:32.249614+0000","last_clean":"2024-09-13T17:16:32.249614+0000","last_became_active":"2024-09-13T17:16:29.651960+0000","last_became_peered":"2024-09-13T17:16:29.651960+0000","last_unstale":"2024-09-13T17:16:32.249614+0000","last_undegraded":"2024-09-13T17:16:32.249614+0000","last_fullsized":"2024-09-13T17:16:32.249614+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:33.186295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863265+0000","last_change":"2024-09-13T17:16:29.661386+0000","last_active":"2024-09-13T17:16:31.863265+0000","last_peered":"2024-09-13T17:16:31.863265+0000","last_clean":"2024-09-13T17:16:31.863265+0000","last_became_active":"2024-09-13T17:16:29.661078+0000","last_became_peered":"2024-09-13T17:16:29.661078+0000","last_unstale":"2024-09-13T17:16:31.863265+0000","last_undegraded":"2024-09-13T17:16:31.863265+0000","last_fullsized":"2024-09-13T17:16:31.863265+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:57.138084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648791+0000","last_change":"2024-09-13T17:16:29.666139+0000","last_active":"2024-09-13T17:16:30.648791+0000","last_peered":"2024-09-13T17:16:30.648791+0000","last_clean":"2024-09-13T17:16:30.648791+0000","last_became_active":"2024-09-13T17:16:29.665543+0000","last_became_peered":"2024-09-13T17:16:29.665543+0000","last_unstale":"2024-09-13T17:16:30.648791+0000","last_undegraded":"2024-09-13T17:16:30.648791+0000","last_fullsized":"2024-09-13T17:16:30.648791+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:13.845396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649041+0000","last_change":"2024-09-13T17:16:29.659480+0000","last_active":"2024-09-13T17:16:30.649041+0000","last_peered":"2024-09-13T17:16:30.649041+0000","last_clean":"2024-09-13T17:16:30.649041+0000","last_became_active":"2024-09-13T17:16:29.655998+0000","last_became_peered":"2024-09-13T17:16:29.655998+0000","last_unstale":"2024-09-13T17:16:30.649041+0000","last_undegraded":"2024-09-13T17:16:30.649041+0000","last_fullsized":"2024-09-13T17:16:30.649041+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:38:44.513205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647595+0000","last_change":"2024-09-13T17:16:29.655894+0000","last_active":"2024-09-13T17:16:30.647595+0000","last_peered":"2024-09-13T17:16:30.647595+0000","last_clean":"2024-09-13T17:16:30.647595+0000","last_became_active":"2024-09-13T17:16:29.655634+0000","last_became_peered":"2024-09-13T17:16:29.655634+0000","last_unstale":"2024-09-13T17:16:30.647595+0000","last_undegraded":"2024-09-13T17:16:30.647595+0000","last_fullsized":"2024-09-13T17:16:30.647595+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:29.829550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249520+0000","last_change":"2024-09-13T17:16:29.655005+0000","last_active":"2024-09-13T17:16:32.249520+0000","last_peered":"2024-09-13T17:16:32.249520+0000","last_clean":"2024-09-13T17:16:32.249520+0000","last_became_active":"2024-09-13T17:16:29.652689+0000","last_became_peered":"2024-09-13T17:16:29.652689+0000","last_unstale":"2024-09-13T17:16:32.249520+0000","last_undegraded":"2024-09-13T17:16:32.249520+0000","last_fullsized":"2024-09-13T17:16:32.249520+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:19.689858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249755+0000","last_change":"2024-09-13T17:16:29.661725+0000","last_active":"2024-09-13T17:16:32.249755+0000","last_peered":"2024-09-13T17:16:32.249755+0000","last_clean":"2024-09-13T17:16:32.249755+0000","last_became_active":"2024-09-13T17:16:29.660371+0000","last_became_peered":"2024-09-13T17:16:29.660371+0000","last_unstale":"2024-09-13T17:16:32.249755+0000","last_undegraded":"2024-09-13T17:16:32.249755+0000","last_fullsized":"2024-09-13T17:16:32.249755+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:16.779035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"124'4","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649531+0000","last_change":"2024-09-13T17:16:29.203458+0000","last_active":"2024-09-13T17:16:30.649531+0000","last_peered":"2024-09-13T17:16:30.649531+0000","last_clean":"2024-09-13T17:16:30.649531+0000","last_became_active":"2024-09-13T17:16:29.203136+0000","last_became_peered":"2024-09-13T17:16:29.203136+0000","last_unstale":"2024-09-13T17:16:30.649531+0000","last_undegraded":"2024-09-13T17:16:30.649531+0000","last_fullsized":"2024-09-13T17:16:30.649531+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:18.680008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003122+0000","last_change":"2024-09-13T17:16:29.661702+0000","last_active":"2024-09-13T17:16:32.003122+0000","last_peered":"2024-09-13T17:16:32.003122+0000","last_clean":"2024-09-13T17:16:32.003122+0000","last_became_active":"2024-09-13T17:16:29.661388+0000","last_became_peered":"2024-09-13T17:16:29.661388+0000","last_unstale":"2024-09-13T17:16:32.003122+0000","last_undegraded":"2024-09-13T17:16:32.003122+0000","last_fullsized":"2024-09-13T17:16:32.003122+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:32.791598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412132+0000","last_change":"2024-09-13T17:16:29.207258+0000","last_active":"2024-09-13T17:16:31.412132+0000","last_peered":"2024-09-13T17:16:31.412132+0000","last_clean":"2024-09-13T17:16:31.412132+0000","last_became_active":"2024-09-13T17:16:29.206951+0000","last_became_peered":"2024-09-13T17:16:29.206951+0000","last_unstale":"2024-09-13T17:16:31.412132+0000","last_undegraded":"2024-09-13T17:16:31.412132+0000","last_fullsized":"2024-09-13T17:16:31.412132+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:31.573685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249686+0000","last_change":"2024-09-13T17:16:29.651826+0000","last_active":"2024-09-13T17:16:32.249686+0000","last_peered":"2024-09-13T17:16:32.249686+0000","last_clean":"2024-09-13T17:16:32.249686+0000","last_became_active":"2024-09-13T17:16:29.650291+0000","last_became_peered":"2024-09-13T17:16:29.650291+0000","last_unstale":"2024-09-13T17:16:32.249686+0000","last_undegraded":"2024-09-13T17:16:32.249686+0000","last_fullsized":"2024-09-13T17:16:32.249686+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:51.538560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485910+0000","last_change":"2024-09-13T17:16:29.210221+0000","last_active":"2024-09-13T17:16:31.485910+0000","last_peered":"2024-09-13T17:16:31.485910+0000","last_clean":"2024-09-13T17:16:31.485910+0000","last_became_active":"2024-09-13T17:16:29.209963+0000","last_became_peered":"2024-09-13T17:16:29.209963+0000","last_unstale":"2024-09-13T17:16:31.485910+0000","last_undegraded":"2024-09-13T17:16:31.485910+0000","last_fullsized":"2024-09-13T17:16:31.485910+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.290230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412240+0000","last_change":"2024-09-13T17:16:29.664882+0000","last_active":"2024-09-13T17:16:31.412240+0000","last_peered":"2024-09-13T17:16:31.412240+0000","last_clean":"2024-09-13T17:16:31.412240+0000","last_became_active":"2024-09-13T17:16:29.661267+0000","last_became_peered":"2024-09-13T17:16:29.661267+0000","last_unstale":"2024-09-13T17:16:31.412240+0000","last_undegraded":"2024-09-13T17:16:31.412240+0000","last_fullsized":"2024-09-13T17:16:31.412240+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:01.626883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485806+0000","last_change":"2024-09-13T17:16:29.207591+0000","last_active":"2024-09-13T17:16:31.485806+0000","last_peered":"2024-09-13T17:16:31.485806+0000","last_clean":"2024-09-13T17:16:31.485806+0000","last_became_active":"2024-09-13T17:16:29.207260+0000","last_became_peered":"2024-09-13T17:16:29.207260+0000","last_unstale":"2024-09-13T17:16:31.485806+0000","last_undegraded":"2024-09-13T17:16:31.485806+0000","last_fullsized":"2024-09-13T17:16:31.485806+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:13:41.113112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648120+0000","last_change":"2024-09-13T17:16:29.655951+0000","last_active":"2024-09-13T17:16:30.648120+0000","last_peered":"2024-09-13T17:16:30.648120+0000","last_clean":"2024-09-13T17:16:30.648120+0000","last_became_active":"2024-09-13T17:16:29.654996+0000","last_became_peered":"2024-09-13T17:16:29.654996+0000","last_unstale":"2024-09-13T17:16:30.648120+0000","last_undegraded":"2024-09-13T17:16:30.648120+0000","last_fullsized":"2024-09-13T17:16:30.648120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:28.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649118+0000","last_change":"2024-09-13T17:16:29.204745+0000","last_active":"2024-09-13T17:16:30.649118+0000","last_peered":"2024-09-13T17:16:30.649118+0000","last_clean":"2024-09-13T17:16:30.649118+0000","last_became_active":"2024-09-13T17:16:29.204482+0000","last_became_peered":"2024-09-13T17:16:29.204482+0000","last_unstale":"2024-09-13T17:16:30.649118+0000","last_undegraded":"2024-09-13T17:16:30.649118+0000","last_fullsized":"2024-09-13T17:16:30.649118+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:15.769185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648911+0000","last_change":"2024-09-13T17:16:29.669165+0000","last_active":"2024-09-13T17:16:30.648911+0000","last_peered":"2024-09-13T17:16:30.648911+0000","last_clean":"2024-09-13T17:16:30.648911+0000","last_became_active":"2024-09-13T17:16:29.667601+0000","last_became_peered":"2024-09-13T17:16:29.667601+0000","last_unstale":"2024-09-13T17:16:30.648911+0000","last_undegraded":"2024-09-13T17:16:30.648911+0000","last_fullsized":"2024-09-13T17:16:30.648911+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:19.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863480+0000","last_change":"2024-09-13T17:16:29.204954+0000","last_active":"2024-09-13T17:16:31.863480+0000","last_peered":"2024-09-13T17:16:31.863480+0000","last_clean":"2024-09-13T17:16:31.863480+0000","last_became_active":"2024-09-13T17:16:29.204621+0000","last_became_peered":"2024-09-13T17:16:29.204621+0000","last_unstale":"2024-09-13T17:16:31.863480+0000","last_undegraded":"2024-09-13T17:16:31.863480+0000","last_fullsized":"2024-09-13T17:16:31.863480+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.218691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647431+0000","last_change":"2024-09-13T17:16:29.659876+0000","last_active":"2024-09-13T17:16:30.647431+0000","last_peered":"2024-09-13T17:16:30.647431+0000","last_clean":"2024-09-13T17:16:30.647431+0000","last_became_active":"2024-09-13T17:16:29.659640+0000","last_became_peered":"2024-09-13T17:16:29.659640+0000","last_unstale":"2024-09-13T17:16:30.647431+0000","last_undegraded":"2024-09-13T17:16:30.647431+0000","last_fullsized":"2024-09-13T17:16:30.647431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:51.623446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648910+0000","last_change":"2024-09-13T17:16:29.202965+0000","last_active":"2024-09-13T17:16:30.648910+0000","last_peered":"2024-09-13T17:16:30.648910+0000","last_clean":"2024-09-13T17:16:30.648910+0000","last_became_active":"2024-09-13T17:16:29.202345+0000","last_became_peered":"2024-09-13T17:16:29.202345+0000","last_unstale":"2024-09-13T17:16:30.648910+0000","last_undegraded":"2024-09-13T17:16:30.648910+0000","last_fullsized":"2024-09-13T17:16:30.648910+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:41.248992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649230+0000","last_change":"2024-09-13T17:16:29.660341+0000","last_active":"2024-09-13T17:16:30.649230+0000","last_peered":"2024-09-13T17:16:30.649230+0000","last_clean":"2024-09-13T17:16:30.649230+0000","last_became_active":"2024-09-13T17:16:29.657065+0000","last_became_peered":"2024-09-13T17:16:29.657065+0000","last_unstale":"2024-09-13T17:16:30.649230+0000","last_undegraded":"2024-09-13T17:16:30.649230+0000","last_fullsized":"2024-09-13T17:16:30.649230+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:46.073611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647452+0000","last_change":"2024-09-13T17:16:29.209359+0000","last_active":"2024-09-13T17:16:30.647452+0000","last_peered":"2024-09-13T17:16:30.647452+0000","last_clean":"2024-09-13T17:16:30.647452+0000","last_became_active":"2024-09-13T17:16:29.208676+0000","last_became_peered":"2024-09-13T17:16:29.208676+0000","last_unstale":"2024-09-13T17:16:30.647452+0000","last_undegraded":"2024-09-13T17:16:30.647452+0000","last_fullsized":"2024-09-13T17:16:30.647452+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:12.322072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649512+0000","last_change":"2024-09-13T17:16:29.661135+0000","last_active":"2024-09-13T17:16:30.649512+0000","last_peered":"2024-09-13T17:16:30.649512+0000","last_clean":"2024-09-13T17:16:30.649512+0000","last_became_active":"2024-09-13T17:16:29.660850+0000","last_became_peered":"2024-09-13T17:16:29.660850+0000","last_unstale":"2024-09-13T17:16:30.649512+0000","last_undegraded":"2024-09-13T17:16:30.649512+0000","last_fullsized":"2024-09-13T17:16:30.649512+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:48.950157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485957+0000","last_change":"2024-09-13T17:16:29.213989+0000","last_active":"2024-09-13T17:16:31.485957+0000","last_peered":"2024-09-13T17:16:31.485957+0000","last_clean":"2024-09-13T17:16:31.485957+0000","last_became_active":"2024-09-13T17:16:29.213715+0000","last_became_peered":"2024-09-13T17:16:29.213715+0000","last_unstale":"2024-09-13T17:16:31.485957+0000","last_undegraded":"2024-09-13T17:16:31.485957+0000","last_fullsized":"2024-09-13T17:16:31.485957+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:00.489585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003100+0000","last_change":"2024-09-13T17:16:29.672374+0000","last_active":"2024-09-13T17:16:32.003100+0000","last_peered":"2024-09-13T17:16:32.003100+0000","last_clean":"2024-09-13T17:16:32.003100+0000","last_became_active":"2024-09-13T17:16:29.672044+0000","last_became_peered":"2024-09-13T17:16:29.672044+0000","last_unstale":"2024-09-13T17:16:32.003100+0000","last_undegraded":"2024-09-13T17:16:32.003100+0000","last_fullsized":"2024-09-13T17:16:32.003100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:01:21.413522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863548+0000","last_change":"2024-09-13T17:16:29.207160+0000","last_active":"2024-09-13T17:16:31.863548+0000","last_peered":"2024-09-13T17:16:31.863548+0000","last_clean":"2024-09-13T17:16:31.863548+0000","last_became_active":"2024-09-13T17:16:29.206877+0000","last_became_peered":"2024-09-13T17:16:29.206877+0000","last_unstale":"2024-09-13T17:16:31.863548+0000","last_undegraded":"2024-09-13T17:16:31.863548+0000","last_fullsized":"2024-09-13T17:16:31.863548+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:13.601999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863316+0000","last_change":"2024-09-13T17:16:29.664255+0000","last_active":"2024-09-13T17:16:31.863316+0000","last_peered":"2024-09-13T17:16:31.863316+0000","last_clean":"2024-09-13T17:16:31.863316+0000","last_became_active":"2024-09-13T17:16:29.662911+0000","last_became_peered":"2024-09-13T17:16:29.662911+0000","last_unstale":"2024-09-13T17:16:31.863316+0000","last_undegraded":"2024-09-13T17:16:31.863316+0000","last_fullsized":"2024-09-13T17:16:31.863316+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:19.954724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"124'3","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648177+0000","last_change":"2024-09-13T17:16:29.210846+0000","last_active":"2024-09-13T17:16:30.648177+0000","last_peered":"2024-09-13T17:16:30.648177+0000","last_clean":"2024-09-13T17:16:30.648177+0000","last_became_active":"2024-09-13T17:16:29.210596+0000","last_became_peered":"2024-09-13T17:16:29.210596+0000","last_unstale":"2024-09-13T17:16:30.648177+0000","last_undegraded":"2024-09-13T17:16:30.648177+0000","last_fullsized":"2024-09-13T17:16:30.648177+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:13.988363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649100+0000","last_change":"2024-09-13T17:16:29.660476+0000","last_active":"2024-09-13T17:16:30.649100+0000","last_peered":"2024-09-13T17:16:30.649100+0000","last_clean":"2024-09-13T17:16:30.649100+0000","last_became_active":"2024-09-13T17:16:29.655669+0000","last_became_peered":"2024-09-13T17:16:29.655669+0000","last_unstale":"2024-09-13T17:16:30.649100+0000","last_undegraded":"2024-09-13T17:16:30.649100+0000","last_fullsized":"2024-09-13T17:16:30.649100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:28.647894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"124'6","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648015+0000","last_change":"2024-09-13T17:16:29.204587+0000","last_active":"2024-09-13T17:16:30.648015+0000","last_peered":"2024-09-13T17:16:30.648015+0000","last_clean":"2024-09-13T17:16:30.648015+0000","last_became_active":"2024-09-13T17:16:29.204304+0000","last_became_peered":"2024-09-13T17:16:29.204304+0000","last_unstale":"2024-09-13T17:16:30.648015+0000","last_undegraded":"2024-09-13T17:16:30.648015+0000","last_fullsized":"2024-09-13T17:16:30.648015+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:11.529554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863403+0000","last_change":"2024-09-13T17:16:29.654760+0000","last_active":"2024-09-13T17:16:31.863403+0000","last_peered":"2024-09-13T17:16:31.863403+0000","last_clean":"2024-09-13T17:16:31.863403+0000","last_became_active":"2024-09-13T17:16:29.654034+0000","last_became_peered":"2024-09-13T17:16:29.654034+0000","last_unstale":"2024-09-13T17:16:31.863403+0000","last_undegraded":"2024-09-13T17:16:31.863403+0000","last_fullsized":"2024-09-13T17:16:31.863403+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:53.523884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249597+0000","last_change":"2024-09-13T17:16:29.210084+0000","last_active":"2024-09-13T17:16:32.249597+0000","last_peered":"2024-09-13T17:16:32.249597+0000","last_clean":"2024-09-13T17:16:32.249597+0000","last_became_active":"2024-09-13T17:16:29.209426+0000","last_became_peered":"2024-09-13T17:16:29.209426+0000","last_unstale":"2024-09-13T17:16:32.249597+0000","last_undegraded":"2024-09-13T17:16:32.249597+0000","last_fullsized":"2024-09-13T17:16:32.249597+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:54.716094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863311+0000","last_change":"2024-09-13T17:16:29.654859+0000","last_active":"2024-09-13T17:16:31.863311+0000","last_peered":"2024-09-13T17:16:31.863311+0000","last_clean":"2024-09-13T17:16:31.863311+0000","last_became_active":"2024-09-13T17:16:29.654613+0000","last_became_peered":"2024-09-13T17:16:29.654613+0000","last_unstale":"2024-09-13T17:16:31.863311+0000","last_undegraded":"2024-09-13T17:16:31.863311+0000","last_fullsized":"2024-09-13T17:16:31.863311+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:19.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002843+0000","last_change":"2024-09-13T17:16:29.214047+0000","last_active":"2024-09-13T17:16:32.002843+0000","last_peered":"2024-09-13T17:16:32.002843+0000","last_clean":"2024-09-13T17:16:32.002843+0000","last_became_active":"2024-09-13T17:16:29.213794+0000","last_became_peered":"2024-09-13T17:16:29.213794+0000","last_unstale":"2024-09-13T17:16:32.002843+0000","last_undegraded":"2024-09-13T17:16:32.002843+0000","last_fullsized":"2024-09-13T17:16:32.002843+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:35.567486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863396+0000","last_change":"2024-09-13T17:16:29.656281+0000","last_active":"2024-09-13T17:16:31.863396+0000","last_peered":"2024-09-13T17:16:31.863396+0000","last_clean":"2024-09-13T17:16:31.863396+0000","last_became_active":"2024-09-13T17:16:29.655999+0000","last_became_peered":"2024-09-13T17:16:29.655999+0000","last_unstale":"2024-09-13T17:16:31.863396+0000","last_undegraded":"2024-09-13T17:16:31.863396+0000","last_fullsized":"2024-09-13T17:16:31.863396+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:38.893591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"124'8","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485524+0000","last_change":"2024-09-13T17:16:29.217569+0000","last_active":"2024-09-13T17:16:31.485524+0000","last_peered":"2024-09-13T17:16:31.485524+0000","last_clean":"2024-09-13T17:16:31.485524+0000","last_became_active":"2024-09-13T17:16:29.217240+0000","last_became_peered":"2024-09-13T17:16:29.217240+0000","last_unstale":"2024-09-13T17:16:31.485524+0000","last_undegraded":"2024-09-13T17:16:31.485524+0000","last_fullsized":"2024-09-13T17:16:31.485524+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.886265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485992+0000","last_change":"2024-09-13T17:16:29.675256+0000","last_active":"2024-09-13T17:16:31.485992+0000","last_peered":"2024-09-13T17:16:31.485992+0000","last_clean":"2024-09-13T17:16:31.485992+0000","last_became_active":"2024-09-13T17:16:29.674732+0000","last_became_peered":"2024-09-13T17:16:29.674732+0000","last_unstale":"2024-09-13T17:16:31.485992+0000","last_undegraded":"2024-09-13T17:16:31.485992+0000","last_fullsized":"2024-09-13T17:16:31.485992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:30.679012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648482+0000","last_change":"2024-09-13T17:16:29.206006+0000","last_active":"2024-09-13T17:16:30.648482+0000","last_peered":"2024-09-13T17:16:30.648482+0000","last_clean":"2024-09-13T17:16:30.648482+0000","last_became_active":"2024-09-13T17:16:29.203992+0000","last_became_peered":"2024-09-13T17:16:29.203992+0000","last_unstale":"2024-09-13T17:16:30.648482+0000","last_undegraded":"2024-09-13T17:16:30.648482+0000","last_fullsized":"2024-09-13T17:16:30.648482+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:04.922113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249759+0000","last_change":"2024-09-13T17:16:29.656030+0000","last_active":"2024-09-13T17:16:32.249759+0000","last_peered":"2024-09-13T17:16:32.249759+0000","last_clean":"2024-09-13T17:16:32.249759+0000","last_became_active":"2024-09-13T17:16:29.655037+0000","last_became_peered":"2024-09-13T17:16:29.655037+0000","last_unstale":"2024-09-13T17:16:32.249759+0000","last_undegraded":"2024-09-13T17:16:32.249759+0000","last_fullsized":"2024-09-13T17:16:32.249759+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:34.114361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.486034+0000","last_change":"2024-09-13T17:16:29.200192+0000","last_active":"2024-09-13T17:16:31.486034+0000","last_peered":"2024-09-13T17:16:31.486034+0000","last_clean":"2024-09-13T17:16:31.486034+0000","last_became_active":"2024-09-13T17:16:29.199877+0000","last_became_peered":"2024-09-13T17:16:29.199877+0000","last_unstale":"2024-09-13T17:16:31.486034+0000","last_undegraded":"2024-09-13T17:16:31.486034+0000","last_fullsized":"2024-09-13T17:16:31.486034+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:11.220144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411949+0000","last_change":"2024-09-13T17:16:29.656519+0000","last_active":"2024-09-13T17:16:31.411949+0000","last_peered":"2024-09-13T17:16:31.411949+0000","last_clean":"2024-09-13T17:16:31.411949+0000","last_became_active":"2024-09-13T17:16:29.655247+0000","last_became_peered":"2024-09-13T17:16:29.655247+0000","last_unstale":"2024-09-13T17:16:31.411949+0000","last_undegraded":"2024-09-13T17:16:31.411949+0000","last_fullsized":"2024-09-13T17:16:31.411949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:51.670675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002933+0000","last_change":"2024-09-13T17:16:29.204169+0000","last_active":"2024-09-13T17:16:32.002933+0000","last_peered":"2024-09-13T17:16:32.002933+0000","last_clean":"2024-09-13T17:16:32.002933+0000","last_became_active":"2024-09-13T17:16:29.203876+0000","last_became_peered":"2024-09-13T17:16:29.203876+0000","last_unstale":"2024-09-13T17:16:32.002933+0000","last_undegraded":"2024-09-13T17:16:32.002933+0000","last_fullsized":"2024-09-13T17:16:32.002933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:04.887710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412254+0000","last_change":"2024-09-13T17:16:29.655915+0000","last_active":"2024-09-13T17:16:31.412254+0000","last_peered":"2024-09-13T17:16:31.412254+0000","last_clean":"2024-09-13T17:16:31.412254+0000","last_became_active":"2024-09-13T17:16:29.655625+0000","last_became_peered":"2024-09-13T17:16:29.655625+0000","last_unstale":"2024-09-13T17:16:31.412254+0000","last_undegraded":"2024-09-13T17:16:31.412254+0000","last_fullsized":"2024-09-13T17:16:31.412254+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:42.258842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648797+0000","last_change":"2024-09-13T17:16:29.202384+0000","last_active":"2024-09-13T17:16:30.648797+0000","last_peered":"2024-09-13T17:16:30.648797+0000","last_clean":"2024-09-13T17:16:30.648797+0000","last_became_active":"2024-09-13T17:16:29.202101+0000","last_became_peered":"2024-09-13T17:16:29.202101+0000","last_unstale":"2024-09-13T17:16:30.648797+0000","last_undegraded":"2024-09-13T17:16:30.648797+0000","last_fullsized":"2024-09-13T17:16:30.648797+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:33.104511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411828+0000","last_change":"2024-09-13T17:16:29.663532+0000","last_active":"2024-09-13T17:16:31.411828+0000","last_peered":"2024-09-13T17:16:31.411828+0000","last_clean":"2024-09-13T17:16:31.411828+0000","last_became_active":"2024-09-13T17:16:29.658714+0000","last_became_peered":"2024-09-13T17:16:29.658714+0000","last_unstale":"2024-09-13T17:16:31.411828+0000","last_undegraded":"2024-09-13T17:16:31.411828+0000","last_fullsized":"2024-09-13T17:16:31.411828+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:24.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249693+0000","last_change":"2024-09-13T17:16:29.210025+0000","last_active":"2024-09-13T17:16:32.249693+0000","last_peered":"2024-09-13T17:16:32.249693+0000","last_clean":"2024-09-13T17:16:32.249693+0000","last_became_active":"2024-09-13T17:16:29.209766+0000","last_became_peered":"2024-09-13T17:16:29.209766+0000","last_unstale":"2024-09-13T17:16:32.249693+0000","last_undegraded":"2024-09-13T17:16:32.249693+0000","last_fullsized":"2024-09-13T17:16:32.249693+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.290230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647516+0000","last_change":"2024-09-13T17:16:29.651378+0000","last_active":"2024-09-13T17:16:30.647516+0000","last_peered":"2024-09-13T17:16:30.647516+0000","last_clean":"2024-09-13T17:16:30.647516+0000","last_became_active":"2024-09-13T17:16:29.651128+0000","last_became_peered":"2024-09-13T17:16:29.651128+0000","last_unstale":"2024-09-13T17:16:30.647516+0000","last_undegraded":"2024-09-13T17:16:30.647516+0000","last_fullsized":"2024-09-13T17:16:30.647516+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:52.773195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485611+0000","last_change":"2024-09-13T17:16:29.204244+0000","last_active":"2024-09-13T17:16:31.485611+0000","last_peered":"2024-09-13T17:16:31.485611+0000","last_clean":"2024-09-13T17:16:31.485611+0000","last_became_active":"2024-09-13T17:16:29.203970+0000","last_became_peered":"2024-09-13T17:16:29.203970+0000","last_unstale":"2024-09-13T17:16:31.485611+0000","last_undegraded":"2024-09-13T17:16:31.485611+0000","last_fullsized":"2024-09-13T17:16:31.485611+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:26.607101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412011+0000","last_change":"2024-09-13T17:16:29.658856+0000","last_active":"2024-09-13T17:16:31.412011+0000","last_peered":"2024-09-13T17:16:31.412011+0000","last_clean":"2024-09-13T17:16:31.412011+0000","last_became_active":"2024-09-13T17:16:29.658347+0000","last_became_peered":"2024-09-13T17:16:29.658347+0000","last_unstale":"2024-09-13T17:16:31.412011+0000","last_undegraded":"2024-09-13T17:16:31.412011+0000","last_fullsized":"2024-09-13T17:16:31.412011+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:57.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249543+0000","last_change":"2024-09-13T17:16:29.207781+0000","last_active":"2024-09-13T17:16:32.249543+0000","last_peered":"2024-09-13T17:16:32.249543+0000","last_clean":"2024-09-13T17:16:32.249543+0000","last_became_active":"2024-09-13T17:16:29.207515+0000","last_became_peered":"2024-09-13T17:16:29.207515+0000","last_unstale":"2024-09-13T17:16:32.249543+0000","last_undegraded":"2024-09-13T17:16:32.249543+0000","last_fullsized":"2024-09-13T17:16:32.249543+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:43.178935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863229+0000","last_change":"2024-09-13T17:16:29.660547+0000","last_active":"2024-09-13T17:16:31.863229+0000","last_peered":"2024-09-13T17:16:31.863229+0000","last_clean":"2024-09-13T17:16:31.863229+0000","last_became_active":"2024-09-13T17:16:29.660291+0000","last_became_peered":"2024-09-13T17:16:29.660291+0000","last_unstale":"2024-09-13T17:16:31.863229+0000","last_undegraded":"2024-09-13T17:16:31.863229+0000","last_fullsized":"2024-09-13T17:16:31.863229+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:08.675355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647559+0000","last_change":"2024-09-13T17:16:29.203550+0000","last_active":"2024-09-13T17:16:30.647559+0000","last_peered":"2024-09-13T17:16:30.647559+0000","last_clean":"2024-09-13T17:16:30.647559+0000","last_became_active":"2024-09-13T17:16:29.203281+0000","last_became_peered":"2024-09-13T17:16:29.203281+0000","last_unstale":"2024-09-13T17:16:30.647559+0000","last_undegraded":"2024-09-13T17:16:30.647559+0000","last_fullsized":"2024-09-13T17:16:30.647559+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:37.883741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003019+0000","last_change":"2024-09-13T17:16:29.669011+0000","last_active":"2024-09-13T17:16:32.003019+0000","last_peered":"2024-09-13T17:16:32.003019+0000","last_clean":"2024-09-13T17:16:32.003019+0000","last_became_active":"2024-09-13T17:16:29.668700+0000","last_became_peered":"2024-09-13T17:16:29.668700+0000","last_unstale":"2024-09-13T17:16:32.003019+0000","last_undegraded":"2024-09-13T17:16:32.003019+0000","last_fullsized":"2024-09-13T17:16:32.003019+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:06.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863599+0000","last_change":"2024-09-13T17:16:29.212961+0000","last_active":"2024-09-13T17:16:31.863599+0000","last_peered":"2024-09-13T17:16:31.863599+0000","last_clean":"2024-09-13T17:16:31.863599+0000","last_became_active":"2024-09-13T17:16:29.212313+0000","last_became_peered":"2024-09-13T17:16:29.212313+0000","last_unstale":"2024-09-13T17:16:31.863599+0000","last_undegraded":"2024-09-13T17:16:31.863599+0000","last_fullsized":"2024-09-13T17:16:31.863599+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:52.704901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648425+0000","last_change":"2024-09-13T17:16:29.667626+0000","last_active":"2024-09-13T17:16:30.648425+0000","last_peered":"2024-09-13T17:16:30.648425+0000","last_clean":"2024-09-13T17:16:30.648425+0000","last_became_active":"2024-09-13T17:16:29.667191+0000","last_became_peered":"2024-09-13T17:16:29.667191+0000","last_unstale":"2024-09-13T17:16:30.648425+0000","last_undegraded":"2024-09-13T17:16:30.648425+0000","last_fullsized":"2024-09-13T17:16:30.648425+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:57.380064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003026+0000","last_change":"2024-09-13T17:16:29.199642+0000","last_active":"2024-09-13T17:16:32.003026+0000","last_peered":"2024-09-13T17:16:32.003026+0000","last_clean":"2024-09-13T17:16:32.003026+0000","last_became_active":"2024-09-13T17:16:29.197660+0000","last_became_peered":"2024-09-13T17:16:29.197660+0000","last_unstale":"2024-09-13T17:16:32.003026+0000","last_undegraded":"2024-09-13T17:16:32.003026+0000","last_fullsized":"2024-09-13T17:16:32.003026+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:32:42.080221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647888+0000","last_change":"2024-09-13T17:16:29.659508+0000","last_active":"2024-09-13T17:16:30.647888+0000","last_peered":"2024-09-13T17:16:30.647888+0000","last_clean":"2024-09-13T17:16:30.647888+0000","last_became_active":"2024-09-13T17:16:29.658736+0000","last_became_peered":"2024-09-13T17:16:29.658736+0000","last_unstale":"2024-09-13T17:16:30.647888+0000","last_undegraded":"2024-09-13T17:16:30.647888+0000","last_fullsized":"2024-09-13T17:16:30.647888+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:44.049952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648207+0000","last_change":"2024-09-13T17:16:29.204371+0000","last_active":"2024-09-13T17:16:30.648207+0000","last_peered":"2024-09-13T17:16:30.648207+0000","last_clean":"2024-09-13T17:16:30.648207+0000","last_became_active":"2024-09-13T17:16:29.204095+0000","last_became_peered":"2024-09-13T17:16:29.204095+0000","last_unstale":"2024-09-13T17:16:30.648207+0000","last_undegraded":"2024-09-13T17:16:30.648207+0000","last_fullsized":"2024-09-13T17:16:30.648207+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:52.704901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002814+0000","last_change":"2024-09-13T17:16:29.669265+0000","last_active":"2024-09-13T17:16:32.002814+0000","last_peered":"2024-09-13T17:16:32.002814+0000","last_clean":"2024-09-13T17:16:32.002814+0000","last_became_active":"2024-09-13T17:16:29.668408+0000","last_became_peered":"2024-09-13T17:16:29.668408+0000","last_unstale":"2024-09-13T17:16:32.002814+0000","last_undegraded":"2024-09-13T17:16:32.002814+0000","last_fullsized":"2024-09-13T17:16:32.002814+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:24.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863494+0000","last_change":"2024-09-13T17:16:29.216236+0000","last_active":"2024-09-13T17:16:31.863494+0000","last_peered":"2024-09-13T17:16:31.863494+0000","last_clean":"2024-09-13T17:16:31.863494+0000","last_became_active":"2024-09-13T17:16:29.215681+0000","last_became_peered":"2024-09-13T17:16:29.215681+0000","last_unstale":"2024-09-13T17:16:31.863494+0000","last_undegraded":"2024-09-13T17:16:31.863494+0000","last_fullsized":"2024-09-13T17:16:31.863494+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:30.343718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648787+0000","last_change":"2024-09-13T17:16:29.664568+0000","last_active":"2024-09-13T17:16:30.648787+0000","last_peered":"2024-09-13T17:16:30.648787+0000","last_clean":"2024-09-13T17:16:30.648787+0000","last_became_active":"2024-09-13T17:16:29.664203+0000","last_became_peered":"2024-09-13T17:16:29.664203+0000","last_unstale":"2024-09-13T17:16:30.648787+0000","last_undegraded":"2024-09-13T17:16:30.648787+0000","last_fullsized":"2024-09-13T17:16:30.648787+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:05.931963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649170+0000","last_change":"2024-09-13T17:16:29.198464+0000","last_active":"2024-09-13T17:16:30.649170+0000","last_peered":"2024-09-13T17:16:30.649170+0000","last_clean":"2024-09-13T17:16:30.649170+0000","last_became_active":"2024-09-13T17:16:29.198179+0000","last_became_peered":"2024-09-13T17:16:29.198179+0000","last_unstale":"2024-09-13T17:16:30.649170+0000","last_undegraded":"2024-09-13T17:16:30.649170+0000","last_fullsized":"2024-09-13T17:16:30.649170+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:50.528710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649012+0000","last_change":"2024-09-13T17:16:29.664914+0000","last_active":"2024-09-13T17:16:30.649012+0000","last_peered":"2024-09-13T17:16:30.649012+0000","last_clean":"2024-09-13T17:16:30.649012+0000","last_became_active":"2024-09-13T17:16:29.664103+0000","last_became_peered":"2024-09-13T17:16:29.664103+0000","last_unstale":"2024-09-13T17:16:30.649012+0000","last_undegraded":"2024-09-13T17:16:30.649012+0000","last_fullsized":"2024-09-13T17:16:30.649012+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:49.259587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648891+0000","last_change":"2024-09-13T17:16:29.203661+0000","last_active":"2024-09-13T17:16:30.648891+0000","last_peered":"2024-09-13T17:16:30.648891+0000","last_clean":"2024-09-13T17:16:30.648891+0000","last_became_active":"2024-09-13T17:16:29.203378+0000","last_became_peered":"2024-09-13T17:16:29.203378+0000","last_unstale":"2024-09-13T17:16:30.648891+0000","last_undegraded":"2024-09-13T17:16:30.648891+0000","last_fullsized":"2024-09-13T17:16:30.648891+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:36.661530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412060+0000","last_change":"2024-09-13T17:16:29.656308+0000","last_active":"2024-09-13T17:16:31.412060+0000","last_peered":"2024-09-13T17:16:31.412060+0000","last_clean":"2024-09-13T17:16:31.412060+0000","last_became_active":"2024-09-13T17:16:29.655207+0000","last_became_peered":"2024-09-13T17:16:29.655207+0000","last_unstale":"2024-09-13T17:16:31.412060+0000","last_undegraded":"2024-09-13T17:16:31.412060+0000","last_fullsized":"2024-09-13T17:16:31.412060+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:05.347447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485516+0000","last_change":"2024-09-13T17:16:29.211105+0000","last_active":"2024-09-13T17:16:31.485516+0000","last_peered":"2024-09-13T17:16:31.485516+0000","last_clean":"2024-09-13T17:16:31.485516+0000","last_became_active":"2024-09-13T17:16:29.210778+0000","last_became_peered":"2024-09-13T17:16:29.210778+0000","last_unstale":"2024-09-13T17:16:31.485516+0000","last_undegraded":"2024-09-13T17:16:31.485516+0000","last_fullsized":"2024-09-13T17:16:31.485516+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:05.921728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647681+0000","last_change":"2024-09-13T17:16:29.665332+0000","last_active":"2024-09-13T17:16:30.647681+0000","last_peered":"2024-09-13T17:16:30.647681+0000","last_clean":"2024-09-13T17:16:30.647681+0000","last_became_active":"2024-09-13T17:16:29.664916+0000","last_became_peered":"2024-09-13T17:16:29.664916+0000","last_unstale":"2024-09-13T17:16:30.647681+0000","last_undegraded":"2024-09-13T17:16:30.647681+0000","last_fullsized":"2024-09-13T17:16:30.647681+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:14.868860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647808+0000","last_change":"2024-09-13T17:16:29.205289+0000","last_active":"2024-09-13T17:16:30.647808+0000","last_peered":"2024-09-13T17:16:30.647808+0000","last_clean":"2024-09-13T17:16:30.647808+0000","last_became_active":"2024-09-13T17:16:29.204985+0000","last_became_peered":"2024-09-13T17:16:29.204985+0000","last_unstale":"2024-09-13T17:16:30.647808+0000","last_undegraded":"2024-09-13T17:16:30.647808+0000","last_fullsized":"2024-09-13T17:16:30.647808+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:55:08.123417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863163+0000","last_change":"2024-09-13T17:16:29.653022+0000","last_active":"2024-09-13T17:16:31.863163+0000","last_peered":"2024-09-13T17:16:31.863163+0000","last_clean":"2024-09-13T17:16:31.863163+0000","last_became_active":"2024-09-13T17:16:29.651275+0000","last_became_peered":"2024-09-13T17:16:29.651275+0000","last_unstale":"2024-09-13T17:16:31.863163+0000","last_undegraded":"2024-09-13T17:16:31.863163+0000","last_fullsized":"2024-09-13T17:16:31.863163+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:45.563984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649373+0000","last_change":"2024-09-13T17:16:29.202273+0000","last_active":"2024-09-13T17:16:30.649373+0000","last_peered":"2024-09-13T17:16:30.649373+0000","last_clean":"2024-09-13T17:16:30.649373+0000","last_became_active":"2024-09-13T17:16:29.201980+0000","last_became_peered":"2024-09-13T17:16:29.201980+0000","last_unstale":"2024-09-13T17:16:30.649373+0000","last_undegraded":"2024-09-13T17:16:30.649373+0000","last_fullsized":"2024-09-13T17:16:30.649373+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:49.450970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":254,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T17:16:29.645270+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:16:29.645270+0000","last_peered":"2024-09-13T17:16:29.645270+0000","last_clean":"2024-09-13T17:16:29.645270+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:16:29.645270+0000","last_undegraded":"2024-09-13T17:16:29.645270+0000","last_fullsized":"2024-09-13T17:16:29.645270+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51276,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320564,"statfs":{"total":96636764160,"available":96584257536,"internally_reserved":0,"allocated":2154496,"data_stored":1271828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66300000000000003,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.876,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.004}]},{"osd":1,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68400000000000005,"15min":0.65100000000000002},"min":{"1min":0.495,"5min":0.433,"15min":0.40400000000000003},"max":{"1min":0.85499999999999998,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76700000000000002,"15min":0.71099999999999997},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.437},"max":{"1min":0.89500000000000002,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.86399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.53200000000000003,"5min":0.47199999999999998,"15min":0.42399999999999999},"max":{"1min":0.80700000000000005,"5min":1.544,"15min":1.544},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.51600000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.90800000000000003,"5min":1.605,"15min":1.605},"last":0.95499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.76600000000000001,"15min":0.745},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89900000000000002,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60699999999999998,"15min":0.64500000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77200000000000002,"5min":1.054,"15min":1.054},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65800000000000003,"15min":0.68600000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.0449999999999999,"15min":1.1220000000000001},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.437,"5min":0.38800000000000001,"15min":0.33800000000000002},"max":{"1min":0.86899999999999999,"5min":0.97099999999999997,"15min":1.143},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":0.83399999999999996,"5min":1.125,"15min":1.337},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72599999999999998,"15min":0.75900000000000001},"min":{"1min":0.53900000000000003,"5min":0.434,"15min":0.38200000000000001},"max":{"1min":0.91400000000000003,"5min":1.0449999999999999,"15min":1.1930000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":1.1519999999999999,"15min":1.2030000000000001},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":1556480,"data_stored":680914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69399999999999995,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.298,"5min":1.298,"15min":3.6629999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":3.1709999999999998},"last":0.70099999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69899999999999995,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40000000000000002},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70199999999999996,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.46,"5min":1.46,"15min":3.3079999999999998},"last":0.57799999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.68899999999999995,"15min":0.67100000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38100000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71199999999999997,"15min":0.69399999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.012,"15min":3.2629999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.75700000000000001,"15min":0.71499999999999997},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.742},"last":0.82299999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.64700000000000002,"15min":0.65400000000000003},"min":{"1min":0.36699999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.98999999999999999,"5min":1.075,"15min":4.7750000000000004},"last":0.753}]},{"osd":5,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69199999999999995,"15min":0.71699999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":1.304},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.70599999999999996},"min":{"1min":0.42199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92100000000000004,"5min":1.127,"15min":3.609},"last":0.86099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73099999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.601,"5min":1.601,"15min":4.9669999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":1.238}]}]},{"osd":5,"up_from":13,"seq":55834574954,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50688,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321152,"statfs":{"total":96636764160,"available":96584859648,"internally_reserved":0,"allocated":1560576,"data_stored":681448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.998,"15min":0.77600000000000002},"min":{"1min":0.41399999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.498,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.86199999999999999,"15min":0.75900000000000001},"min":{"1min":0.39800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.577,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.44500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.4770000000000001,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":1.5569999999999999,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.84399999999999997,"15min":0.75600000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.524,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.79800000000000004,"15min":0.754},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.8680000000000001,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.623}]},{"osd":3,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.8200000000000001,"15min":1.109},"min":{"1min":0.44,"5min":0.44,"15min":0.435},"max":{"1min":2.8690000000000002,"5min":113.363,"15min":113.363},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.52800000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.359,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.71399999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.827,"5min":2.827,"15min":2.827},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.41199999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.75,"15min":0.72499999999999998},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.9929999999999999,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78200000000000003,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.9319999999999999,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.9379999999999999,"5min":2.02,"15min":2.02},"last":0.68600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54788,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317052,"statfs":{"total":96636764160,"available":96580661248,"internally_reserved":0,"allocated":1552384,"data_stored":680844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.65000000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93200000000000005,"5min":1.056,"15min":1.0620000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.73499999999999999},"min":{"1min":0.40699999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.341},"last":0.69399999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66600000000000004,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66400000000000003,"15min":0.63},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85499999999999998,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.77900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.5,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.874,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.754},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72099999999999997},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.42199999999999999},"max":{"1min":1.22,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70199999999999996,"15min":0.67300000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.006,"15min":1.006},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.69999999999999996},"min":{"1min":0.47299999999999998,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97499999999999998,"5min":1.232,"15min":1.232},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.371,"15min":0.371},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.67200000000000004,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.30299999999999999},"max":{"1min":1.3069999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69299999999999995,"15min":0.66300000000000003},"min":{"1min":0.34999999999999998,"5min":0.32500000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.274,"15min":1.274},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72799999999999998,"15min":0.73899999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.254,"5min":1.292,"15min":1.413},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73299999999999998,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.76800000000000002,"15min":0.73499999999999999},"min":{"1min":0.44500000000000001,"5min":0.40500000000000003,"15min":0.371},"max":{"1min":1.0589999999999999,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.82099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574954,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76872,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94294968,"statfs":{"total":96636764160,"available":96558047232,"internally_reserved":0,"allocated":1568768,"data_stored":682478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77134279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42299999999999999,"15min":0.44900000000000001},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.55900000000000005,"5min":1.177,"15min":1.9810000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45000000000000001,"15min":0.53800000000000003},"min":{"1min":0.32200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68300000000000005,"5min":0.92400000000000004,"15min":2.96},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.95699999999999996,"15min":3.5600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55200000000000005,"15min":0.68400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.746,"5min":1.0309999999999999,"15min":3.6829999999999998},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85999999999999999,"5min":0.88,"15min":1.9590000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.58599999999999997},"min":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.97699999999999998,"15min":1.3819999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72799999999999998,"15min":0.79200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.96899999999999997,"5min":1.1339999999999999,"15min":2.8610000000000002},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.52000000000000002,"5min":0.505,"15min":0.48199999999999998},"max":{"1min":1.012,"5min":1.123,"15min":1.3640000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.755,"15min":0.78200000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.2529999999999999,"15min":1.391},"last":1.032},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72999999999999998,"15min":0.81000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98899999999999999,"5min":1.173,"15min":2.9300000000000002},"last":0.92000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.752,"15min":0.76500000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95999999999999996,"5min":1.1579999999999999,"15min":1.401},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.76200000000000001,"15min":0.78900000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0589999999999999,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":1.0580000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75900000000000001,"15min":0.83799999999999997},"min":{"1min":0.50900000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.022,"5min":1.139,"15min":2.899},"last":0.878},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77300000000000002,"15min":0.81699999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0449999999999999,"5min":1.1220000000000001,"15min":1.4359999999999999},"last":0.93799999999999994}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":1548288,"data_stored":679825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25700000000000001},"max":{"1min":0.63900000000000001,"5min":1.097,"15min":1.097},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.81499999999999995,"15min":0.55800000000000005},"min":{"1min":0.30099999999999999,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.70199999999999996,"5min":30.023,"15min":30.023},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48499999999999999,"15min":1.345},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":1.046,"15min":260.53399999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.745,"15min":1.4450000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.86199999999999999,"5min":21.279,"15min":262.61399999999998},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.82099999999999995,"15min":0.59599999999999997},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80300000000000005,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.875,"5min":1.214,"15min":1.214},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.89100000000000001,"15min":0.80100000000000005},"min":{"1min":0.52900000000000003,"5min":0.495,"15min":0.49099999999999999},"max":{"1min":1.008,"5min":10.598000000000001,"15min":10.598000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.016,"15min":0.85899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0529999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.006}]},{"osd":5,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.0900000000000001,"15min":0.88900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":0.97599999999999998,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.999,"5min":1.2789999999999999,"15min":1.569},"last":0.89500000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.89600000000000002,"15min":0.80700000000000005},"min":{"1min":0.495,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98799999999999999,"5min":10.664,"15min":10.664},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.0920000000000001,"15min":0.90300000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.083,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.93899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.82199999999999995},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1040000000000001,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.1200000000000001,"15min":0.90700000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.46100000000000002},"max":{"1min":1.1379999999999999,"5min":27.199999999999999,"15min":27.199999999999999},"last":1.05}]}]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82624,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289216,"statfs":{"total":96636764160,"available":96552157184,"internally_reserved":0,"allocated":1560576,"data_stored":680968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73399999999999999,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70299999999999996,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.72999999999999998,"15min":0.68400000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.67700000000000005,"15min":0.61099999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.998,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":1.159},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.429,"15min":1.78},"last":1.131}]},{"osd":5,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.751,"15min":0.76800000000000002},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.49199999999999999},"max":{"1min":0.92100000000000004,"5min":1.6020000000000001,"15min":1.6870000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.997,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":1.073}]},{"osd":6,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.755,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.53200000000000003},"max":{"1min":0.91100000000000003,"5min":1.6599999999999999,"15min":1.7310000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.81699999999999995},"min":{"1min":0.55300000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94899999999999995,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":1.204}]},{"osd":7,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76300000000000001,"15min":0.81299999999999994},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.92600000000000005,"5min":1.472,"15min":1.8320000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0089999999999999,"5min":1.544,"15min":1.6359999999999999},"last":1.145}]}]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59476,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312364,"statfs":{"total":96636764160,"available":96575860736,"internally_reserved":0,"allocated":2162688,"data_stored":1271869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.45300000000000001,"15min":0.63500000000000001},"min":{"1min":0.27700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.018,"5min":1.018,"15min":5.5259999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.52300000000000002,"15min":0.64600000000000002},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.27000000000000002},"max":{"1min":2.738,"5min":3.004,"15min":5.4889999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.54400000000000004,"15min":0.67900000000000005},"min":{"1min":0.34799999999999998,"5min":0.313,"15min":0.313},"max":{"1min":3.605,"5min":3.605,"15min":6.0430000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52200000000000002,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.29,"5min":1.29,"15min":4.6760000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52100000000000002,"15min":0.63900000000000001},"min":{"1min":0.378,"5min":0.314,"15min":0.314},"max":{"1min":0.98999999999999999,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.59999999999999998,"15min":0.69799999999999995},"min":{"1min":0.435,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.59099999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.78700000000000003,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.443,"15min":0.443},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":4.734},"last":0.751},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.88800000000000001},"min":{"1min":0.58499999999999996,"5min":0.442,"15min":0.442},"max":{"1min":1.079,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.69199999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.79200000000000004,"15min":0.90400000000000003},"min":{"1min":0.60799999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2190000000000001,"5min":2.7320000000000002,"15min":4.944},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.80100000000000005,"15min":0.89300000000000002},"min":{"1min":0.622,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.78300000000000003,"15min":1.0109999999999999},"min":{"1min":0.56599999999999995,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.0680000000000001,"5min":1.194,"15min":4.8470000000000004},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.79600000000000004,"15min":0.91200000000000003},"min":{"1min":0.70799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.627}]},{"osd":7,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.80500000000000005,"15min":0.93999999999999995},"min":{"1min":0.60599999999999998,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":1.405,"5min":1.405,"15min":5.0540000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.81899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.505},"max":{"1min":1.3149999999999999,"5min":2.706,"15min":4.9950000000000001},"last":0.81200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:16:33.839 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:16:34.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.027+0000 7fa82ce00640 1 Processor -- start 2024-09-13T17:16:34.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.027+0000 7fa82ce00640 1 -- start start 2024-09-13T17:16:34.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.028+0000 7fa82ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8281592a0 0x7fa82815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa82ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 0x7fa82806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa82ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82806cc70 con 0x7fa82806f970 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa82ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82815b0b0 con 0x7fa8281592a0 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa82ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa828159d90 con 0x7fa82806c2d0 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8281592a0 0x7fa82815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa8281592a0 0x7fa82815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44940/0 (socket says 172.21.15.28:44940) 2024-09-13T17:16:34.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 -- 172.21.15.28:0/503036272 learned_addr learned my addr 172.21.15.28:0/503036272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 -- 172.21.15.28:0/503036272 >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 msgr2=0x7fa82806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa826800640 1 --2- 172.21.15.28:0/503036272 >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 0x7fa82806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 --2- 172.21.15.28:0/503036272 >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 0x7fa82806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 -- 172.21.15.28:0/503036272 >> v1:172.21.15.189:6789/0 conn(0x7fa82806f970 legacy=0x7fa828159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa825e00640 1 -- 172.21.15.28:0/503036272 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8280b2040 con 0x7fa8281592a0 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.029+0000 7fa826800640 1 --2- 172.21.15.28:0/503036272 >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 0x7fa82806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:34.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.030+0000 7fa825e00640 1 --2- 172.21.15.28:0/503036272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82815ab70 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fa8280b1dd0 tx=0x7fa81802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5590cf65d97aed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa825400640 1 -- 172.21.15.28:0/503036272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818032440 con 0x7fa8281592a0 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa825400640 1 -- 172.21.15.28:0/503036272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8180325a0 con 0x7fa8281592a0 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa825400640 1 -- 172.21.15.28:0/503036272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818032930 con 0x7fa8281592a0 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 -- 172.21.15.28:0/503036272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 msgr2=0x7fa82815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 --2- 172.21.15.28:0/503036272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82815ab70 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fa8280b1dd0 tx=0x7fa81802a230 comp rx=0 tx=0).stop 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 -- 172.21.15.28:0/503036272 shutdown_connections 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 --2- 172.21.15.28:0/503036272 >> v2:172.21.15.189:3300/0 conn(0x7fa82806c2d0 0x7fa82806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 --2- 172.21.15.28:0/503036272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82815ab70 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.031+0000 7fa82ce00640 1 -- 172.21.15.28:0/503036272 >> 172.21.15.28:0/503036272 conn(0x7fa828069e80 msgr2=0x7fa82806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.032+0000 7fa82ce00640 1 -- 172.21.15.28:0/503036272 shutdown_connections 2024-09-13T17:16:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.032+0000 7fa82ce00640 1 -- 172.21.15.28:0/503036272 wait complete. 2024-09-13T17:16:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.033+0000 7fa82ce00640 1 Processor -- start 2024-09-13T17:16:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.033+0000 7fa82ce00640 1 -- start start 2024-09-13T17:16:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa82806c2d0 0x7fa8280ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82806f970 0x7fa828152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8280cb4e0 con 0x7fa8281592a0 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa828153d40 con 0x7fa82806f970 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.034+0000 7fa82ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa828153220 con 0x7fa82806c2d0 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44950/0 (socket says 172.21.15.28:44950) 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 -- 172.21.15.28:0/1722011017 learned_addr learned my addr 172.21.15.28:0/1722011017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa825e00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa82806c2d0 0x7fa8280ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa81fe00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82806f970 0x7fa828152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 -- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa82806c2d0 msgr2=0x7fa8280ced80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa82806c2d0 0x7fa8280ced80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 -- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82806f970 msgr2=0x7fa828152ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82806f970 0x7fa828152ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 -- 172.21.15.28:0/1722011017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8180094d0 con 0x7fa8281592a0 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.035+0000 7fa826800640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fa82000cef0 tx=0x7fa82000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.036+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000d840 con 0x7fa8281592a0 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.036+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000d9c0 con 0x7fa8281592a0 2024-09-13T17:16:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.036+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8281534a0 con 0x7fa8281592a0 2024-09-13T17:16:34.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.036+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000dd70 con 0x7fa8281592a0 2024-09-13T17:16:34.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.036+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa828153a60 con 0x7fa8281592a0 2024-09-13T17:16:34.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.037+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82815a2b0 con 0x7fa8281592a0 2024-09-13T17:16:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.038+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa82001e3b0 con 0x7fa8281592a0 2024-09-13T17:16:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.038+0000 7fa81e000640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 0x7fa80406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.039+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa820089e80 con 0x7fa8281592a0 2024-09-13T17:16:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.039+0000 7fa825e00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 0x7fa80406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.040+0000 7fa825e00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 0x7fa80406d0c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa818034000 tx=0x7fa81802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.043+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa82815a2b0 con 0x7fa8281592a0 2024-09-13T17:16:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.219+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa82815a5b0 con 0x7fa80406ac00 2024-09-13T17:16:34.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.227+0000 7fa81e000640 1 -- 172.21.15.28:0/1722011017 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289157 (secure 0 0 0) 0x7fa818046450 con 0x7fa80406ac00 2024-09-13T17:16:34.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:34.237 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:16:34.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 msgr2=0x7fa80406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 0x7fa80406d0c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fa818034000 tx=0x7fa81802e000 comp rx=0 tx=0).stop 2024-09-13T17:16:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 msgr2=0x7fa82814b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fa82000cef0 tx=0x7fa82000a710 comp rx=0 tx=0).stop 2024-09-13T17:16:34.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 shutdown_connections 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa80406ac00 0x7fa80406d0c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8281592a0 0x7fa82814b0a0 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa82806f970 0x7fa828152ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 --2- 172.21.15.28:0/1722011017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa82806c2d0 0x7fa8280ced80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.231+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 >> 172.21.15.28:0/1722011017 conn(0x7fa828069e80 msgr2=0x7fa8280b07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.232+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 shutdown_connections 2024-09-13T17:16:34.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.232+0000 7fa82ce00640 1 -- 172.21.15.28:0/1722011017 wait complete. 2024-09-13T17:16:34.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-09-13T17:16:33.013199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":181,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":485284,"kb_used_data":13344,"kb_used_omap":12,"kb_used_meta":471795,"kb_avail":754489436,"statfs":{"total":773094113280,"available":772597182464,"internally_reserved":0,"allocated":13664256,"data_stored":6630174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":483118649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.841094"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648379+0000","last_change":"2024-09-13T17:16:29.665764+0000","last_active":"2024-09-13T17:16:30.648379+0000","last_peered":"2024-09-13T17:16:30.648379+0000","last_clean":"2024-09-13T17:16:30.648379+0000","last_became_active":"2024-09-13T17:16:29.665522+0000","last_became_peered":"2024-09-13T17:16:29.665522+0000","last_unstale":"2024-09-13T17:16:30.648379+0000","last_undegraded":"2024-09-13T17:16:30.648379+0000","last_fullsized":"2024-09-13T17:16:30.648379+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:48.512922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649813+0000","last_change":"2024-09-13T17:16:29.661453+0000","last_active":"2024-09-13T17:16:30.649813+0000","last_peered":"2024-09-13T17:16:30.649813+0000","last_clean":"2024-09-13T17:16:30.649813+0000","last_became_active":"2024-09-13T17:16:29.661189+0000","last_became_peered":"2024-09-13T17:16:29.661189+0000","last_unstale":"2024-09-13T17:16:30.649813+0000","last_undegraded":"2024-09-13T17:16:30.649813+0000","last_fullsized":"2024-09-13T17:16:30.649813+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:30.228156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412057+0000","last_change":"2024-09-13T17:16:29.665392+0000","last_active":"2024-09-13T17:16:31.412057+0000","last_peered":"2024-09-13T17:16:31.412057+0000","last_clean":"2024-09-13T17:16:31.412057+0000","last_became_active":"2024-09-13T17:16:29.664898+0000","last_became_peered":"2024-09-13T17:16:29.664898+0000","last_unstale":"2024-09-13T17:16:31.412057+0000","last_undegraded":"2024-09-13T17:16:31.412057+0000","last_fullsized":"2024-09-13T17:16:31.412057+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:49.228541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647709+0000","last_change":"2024-09-13T17:16:29.659417+0000","last_active":"2024-09-13T17:16:30.647709+0000","last_peered":"2024-09-13T17:16:30.647709+0000","last_clean":"2024-09-13T17:16:30.647709+0000","last_became_active":"2024-09-13T17:16:29.659095+0000","last_became_peered":"2024-09-13T17:16:29.659095+0000","last_unstale":"2024-09-13T17:16:30.647709+0000","last_undegraded":"2024-09-13T17:16:30.647709+0000","last_fullsized":"2024-09-13T17:16:30.647709+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:21.799379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485700+0000","last_change":"2024-09-13T17:16:29.658880+0000","last_active":"2024-09-13T17:16:31.485700+0000","last_peered":"2024-09-13T17:16:31.485700+0000","last_clean":"2024-09-13T17:16:31.485700+0000","last_became_active":"2024-09-13T17:16:29.658628+0000","last_became_peered":"2024-09-13T17:16:29.658628+0000","last_unstale":"2024-09-13T17:16:31.485700+0000","last_undegraded":"2024-09-13T17:16:31.485700+0000","last_fullsized":"2024-09-13T17:16:31.485700+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:45.435256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249562+0000","last_change":"2024-09-13T17:16:29.654544+0000","last_active":"2024-09-13T17:16:32.249562+0000","last_peered":"2024-09-13T17:16:32.249562+0000","last_clean":"2024-09-13T17:16:32.249562+0000","last_became_active":"2024-09-13T17:16:29.654278+0000","last_became_peered":"2024-09-13T17:16:29.654278+0000","last_unstale":"2024-09-13T17:16:32.249562+0000","last_undegraded":"2024-09-13T17:16:32.249562+0000","last_fullsized":"2024-09-13T17:16:32.249562+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:40.943816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649421+0000","last_change":"2024-09-13T17:16:29.660123+0000","last_active":"2024-09-13T17:16:30.649421+0000","last_peered":"2024-09-13T17:16:30.649421+0000","last_clean":"2024-09-13T17:16:30.649421+0000","last_became_active":"2024-09-13T17:16:29.658010+0000","last_became_peered":"2024-09-13T17:16:29.658010+0000","last_unstale":"2024-09-13T17:16:30.649421+0000","last_undegraded":"2024-09-13T17:16:30.649421+0000","last_fullsized":"2024-09-13T17:16:30.649421+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:56.325836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411893+0000","last_change":"2024-09-13T17:16:29.651350+0000","last_active":"2024-09-13T17:16:31.411893+0000","last_peered":"2024-09-13T17:16:31.411893+0000","last_clean":"2024-09-13T17:16:31.411893+0000","last_became_active":"2024-09-13T17:16:29.651046+0000","last_became_peered":"2024-09-13T17:16:29.651046+0000","last_unstale":"2024-09-13T17:16:31.411893+0000","last_undegraded":"2024-09-13T17:16:31.411893+0000","last_fullsized":"2024-09-13T17:16:31.411893+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:14.611849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648446+0000","last_change":"2024-09-13T17:16:29.658358+0000","last_active":"2024-09-13T17:16:30.648446+0000","last_peered":"2024-09-13T17:16:30.648446+0000","last_clean":"2024-09-13T17:16:30.648446+0000","last_became_active":"2024-09-13T17:16:29.657946+0000","last_became_peered":"2024-09-13T17:16:29.657946+0000","last_unstale":"2024-09-13T17:16:30.648446+0000","last_undegraded":"2024-09-13T17:16:30.648446+0000","last_fullsized":"2024-09-13T17:16:30.648446+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:08.676017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648242+0000","last_change":"2024-09-13T17:16:29.665917+0000","last_active":"2024-09-13T17:16:30.648242+0000","last_peered":"2024-09-13T17:16:30.648242+0000","last_clean":"2024-09-13T17:16:30.648242+0000","last_became_active":"2024-09-13T17:16:29.665468+0000","last_became_peered":"2024-09-13T17:16:29.665468+0000","last_unstale":"2024-09-13T17:16:30.648242+0000","last_undegraded":"2024-09-13T17:16:30.648242+0000","last_fullsized":"2024-09-13T17:16:30.648242+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:34.333738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649733+0000","last_change":"2024-09-13T17:16:29.662854+0000","last_active":"2024-09-13T17:16:30.649733+0000","last_peered":"2024-09-13T17:16:30.649733+0000","last_clean":"2024-09-13T17:16:30.649733+0000","last_became_active":"2024-09-13T17:16:29.662573+0000","last_became_peered":"2024-09-13T17:16:29.662573+0000","last_unstale":"2024-09-13T17:16:30.649733+0000","last_undegraded":"2024-09-13T17:16:30.649733+0000","last_fullsized":"2024-09-13T17:16:30.649733+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:13.331922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647801+0000","last_change":"2024-09-13T17:16:29.658722+0000","last_active":"2024-09-13T17:16:30.647801+0000","last_peered":"2024-09-13T17:16:30.647801+0000","last_clean":"2024-09-13T17:16:30.647801+0000","last_became_active":"2024-09-13T17:16:29.658502+0000","last_became_peered":"2024-09-13T17:16:29.658502+0000","last_unstale":"2024-09-13T17:16:30.647801+0000","last_undegraded":"2024-09-13T17:16:30.647801+0000","last_fullsized":"2024-09-13T17:16:30.647801+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:52.038064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412226+0000","last_change":"2024-09-13T17:16:29.666592+0000","last_active":"2024-09-13T17:16:31.412226+0000","last_peered":"2024-09-13T17:16:31.412226+0000","last_clean":"2024-09-13T17:16:31.412226+0000","last_became_active":"2024-09-13T17:16:29.663240+0000","last_became_peered":"2024-09-13T17:16:29.663240+0000","last_unstale":"2024-09-13T17:16:31.412226+0000","last_undegraded":"2024-09-13T17:16:31.412226+0000","last_fullsized":"2024-09-13T17:16:31.412226+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:32.791598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411904+0000","last_change":"2024-09-13T17:16:29.657258+0000","last_active":"2024-09-13T17:16:31.411904+0000","last_peered":"2024-09-13T17:16:31.411904+0000","last_clean":"2024-09-13T17:16:31.411904+0000","last_became_active":"2024-09-13T17:16:29.656990+0000","last_became_peered":"2024-09-13T17:16:29.656990+0000","last_unstale":"2024-09-13T17:16:31.411904+0000","last_undegraded":"2024-09-13T17:16:31.411904+0000","last_fullsized":"2024-09-13T17:16:31.411904+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:01:21.413522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249404+0000","last_change":"2024-09-13T17:16:29.652040+0000","last_active":"2024-09-13T17:16:32.249404+0000","last_peered":"2024-09-13T17:16:32.249404+0000","last_clean":"2024-09-13T17:16:32.249404+0000","last_became_active":"2024-09-13T17:16:29.651764+0000","last_became_peered":"2024-09-13T17:16:29.651764+0000","last_unstale":"2024-09-13T17:16:32.249404+0000","last_undegraded":"2024-09-13T17:16:32.249404+0000","last_fullsized":"2024-09-13T17:16:32.249404+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:50.460820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647370+0000","last_change":"2024-09-13T17:16:29.651488+0000","last_active":"2024-09-13T17:16:30.647370+0000","last_peered":"2024-09-13T17:16:30.647370+0000","last_clean":"2024-09-13T17:16:30.647370+0000","last_became_active":"2024-09-13T17:16:29.651248+0000","last_became_peered":"2024-09-13T17:16:29.651248+0000","last_unstale":"2024-09-13T17:16:30.647370+0000","last_undegraded":"2024-09-13T17:16:30.647370+0000","last_fullsized":"2024-09-13T17:16:30.647370+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:49.880733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649800+0000","last_change":"2024-09-13T17:16:29.666742+0000","last_active":"2024-09-13T17:16:30.649800+0000","last_peered":"2024-09-13T17:16:30.649800+0000","last_clean":"2024-09-13T17:16:30.649800+0000","last_became_active":"2024-09-13T17:16:29.665315+0000","last_became_peered":"2024-09-13T17:16:29.665315+0000","last_unstale":"2024-09-13T17:16:30.649800+0000","last_undegraded":"2024-09-13T17:16:30.649800+0000","last_fullsized":"2024-09-13T17:16:30.649800+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:19.954724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002896+0000","last_change":"2024-09-13T17:16:29.666635+0000","last_active":"2024-09-13T17:16:32.002896+0000","last_peered":"2024-09-13T17:16:32.002896+0000","last_clean":"2024-09-13T17:16:32.002896+0000","last_became_active":"2024-09-13T17:16:29.666335+0000","last_became_peered":"2024-09-13T17:16:29.666335+0000","last_unstale":"2024-09-13T17:16:32.002896+0000","last_undegraded":"2024-09-13T17:16:32.002896+0000","last_fullsized":"2024-09-13T17:16:32.002896+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:57.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412148+0000","last_change":"2024-09-13T17:16:29.665084+0000","last_active":"2024-09-13T17:16:31.412148+0000","last_peered":"2024-09-13T17:16:31.412148+0000","last_clean":"2024-09-13T17:16:31.412148+0000","last_became_active":"2024-09-13T17:16:29.662135+0000","last_became_peered":"2024-09-13T17:16:29.662135+0000","last_unstale":"2024-09-13T17:16:31.412148+0000","last_undegraded":"2024-09-13T17:16:31.412148+0000","last_fullsized":"2024-09-13T17:16:31.412148+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:31.353568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649741+0000","last_change":"2024-09-13T17:16:29.659670+0000","last_active":"2024-09-13T17:16:30.649741+0000","last_peered":"2024-09-13T17:16:30.649741+0000","last_clean":"2024-09-13T17:16:30.649741+0000","last_became_active":"2024-09-13T17:16:29.659241+0000","last_became_peered":"2024-09-13T17:16:29.659241+0000","last_unstale":"2024-09-13T17:16:30.649741+0000","last_undegraded":"2024-09-13T17:16:30.649741+0000","last_fullsized":"2024-09-13T17:16:30.649741+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:09.032346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649835+0000","last_change":"2024-09-13T17:16:29.666308+0000","last_active":"2024-09-13T17:16:30.649835+0000","last_peered":"2024-09-13T17:16:30.649835+0000","last_clean":"2024-09-13T17:16:30.649835+0000","last_became_active":"2024-09-13T17:16:29.665879+0000","last_became_peered":"2024-09-13T17:16:29.665879+0000","last_unstale":"2024-09-13T17:16:30.649835+0000","last_undegraded":"2024-09-13T17:16:30.649835+0000","last_fullsized":"2024-09-13T17:16:30.649835+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:53.523884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863367+0000","last_change":"2024-09-13T17:16:29.659207+0000","last_active":"2024-09-13T17:16:31.863367+0000","last_peered":"2024-09-13T17:16:31.863367+0000","last_clean":"2024-09-13T17:16:31.863367+0000","last_became_active":"2024-09-13T17:16:29.656586+0000","last_became_peered":"2024-09-13T17:16:29.656586+0000","last_unstale":"2024-09-13T17:16:31.863367+0000","last_undegraded":"2024-09-13T17:16:31.863367+0000","last_fullsized":"2024-09-13T17:16:31.863367+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:13.845396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648855+0000","last_change":"2024-09-13T17:16:29.659836+0000","last_active":"2024-09-13T17:16:30.648855+0000","last_peered":"2024-09-13T17:16:30.648855+0000","last_clean":"2024-09-13T17:16:30.648855+0000","last_became_active":"2024-09-13T17:16:29.659520+0000","last_became_peered":"2024-09-13T17:16:29.659520+0000","last_unstale":"2024-09-13T17:16:30.648855+0000","last_undegraded":"2024-09-13T17:16:30.648855+0000","last_fullsized":"2024-09-13T17:16:30.648855+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:45.563984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863085+0000","last_change":"2024-09-13T17:16:29.657067+0000","last_active":"2024-09-13T17:16:31.863085+0000","last_peered":"2024-09-13T17:16:31.863085+0000","last_clean":"2024-09-13T17:16:31.863085+0000","last_became_active":"2024-09-13T17:16:29.656801+0000","last_became_peered":"2024-09-13T17:16:29.656801+0000","last_unstale":"2024-09-13T17:16:31.863085+0000","last_undegraded":"2024-09-13T17:16:31.863085+0000","last_fullsized":"2024-09-13T17:16:31.863085+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:56.649937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649891+0000","last_change":"2024-09-13T17:16:29.664265+0000","last_active":"2024-09-13T17:16:30.649891+0000","last_peered":"2024-09-13T17:16:30.649891+0000","last_clean":"2024-09-13T17:16:30.649891+0000","last_became_active":"2024-09-13T17:16:29.662613+0000","last_became_peered":"2024-09-13T17:16:29.662613+0000","last_unstale":"2024-09-13T17:16:30.649891+0000","last_undegraded":"2024-09-13T17:16:30.649891+0000","last_fullsized":"2024-09-13T17:16:30.649891+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:52.806265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648271+0000","last_change":"2024-09-13T17:16:29.654771+0000","last_active":"2024-09-13T17:16:30.648271+0000","last_peered":"2024-09-13T17:16:30.648271+0000","last_clean":"2024-09-13T17:16:30.648271+0000","last_became_active":"2024-09-13T17:16:29.654475+0000","last_became_peered":"2024-09-13T17:16:29.654475+0000","last_unstale":"2024-09-13T17:16:30.648271+0000","last_undegraded":"2024-09-13T17:16:30.648271+0000","last_fullsized":"2024-09-13T17:16:30.648271+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:23.975549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249614+0000","last_change":"2024-09-13T17:16:29.652225+0000","last_active":"2024-09-13T17:16:32.249614+0000","last_peered":"2024-09-13T17:16:32.249614+0000","last_clean":"2024-09-13T17:16:32.249614+0000","last_became_active":"2024-09-13T17:16:29.651960+0000","last_became_peered":"2024-09-13T17:16:29.651960+0000","last_unstale":"2024-09-13T17:16:32.249614+0000","last_undegraded":"2024-09-13T17:16:32.249614+0000","last_fullsized":"2024-09-13T17:16:32.249614+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:33.186295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863265+0000","last_change":"2024-09-13T17:16:29.661386+0000","last_active":"2024-09-13T17:16:31.863265+0000","last_peered":"2024-09-13T17:16:31.863265+0000","last_clean":"2024-09-13T17:16:31.863265+0000","last_became_active":"2024-09-13T17:16:29.661078+0000","last_became_peered":"2024-09-13T17:16:29.661078+0000","last_unstale":"2024-09-13T17:16:31.863265+0000","last_undegraded":"2024-09-13T17:16:31.863265+0000","last_fullsized":"2024-09-13T17:16:31.863265+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:57.138084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648791+0000","last_change":"2024-09-13T17:16:29.666139+0000","last_active":"2024-09-13T17:16:30.648791+0000","last_peered":"2024-09-13T17:16:30.648791+0000","last_clean":"2024-09-13T17:16:30.648791+0000","last_became_active":"2024-09-13T17:16:29.665543+0000","last_became_peered":"2024-09-13T17:16:29.665543+0000","last_unstale":"2024-09-13T17:16:30.648791+0000","last_undegraded":"2024-09-13T17:16:30.648791+0000","last_fullsized":"2024-09-13T17:16:30.648791+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:41:13.845396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649041+0000","last_change":"2024-09-13T17:16:29.659480+0000","last_active":"2024-09-13T17:16:30.649041+0000","last_peered":"2024-09-13T17:16:30.649041+0000","last_clean":"2024-09-13T17:16:30.649041+0000","last_became_active":"2024-09-13T17:16:29.655998+0000","last_became_peered":"2024-09-13T17:16:29.655998+0000","last_unstale":"2024-09-13T17:16:30.649041+0000","last_undegraded":"2024-09-13T17:16:30.649041+0000","last_fullsized":"2024-09-13T17:16:30.649041+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:38:44.513205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647595+0000","last_change":"2024-09-13T17:16:29.655894+0000","last_active":"2024-09-13T17:16:30.647595+0000","last_peered":"2024-09-13T17:16:30.647595+0000","last_clean":"2024-09-13T17:16:30.647595+0000","last_became_active":"2024-09-13T17:16:29.655634+0000","last_became_peered":"2024-09-13T17:16:29.655634+0000","last_unstale":"2024-09-13T17:16:30.647595+0000","last_undegraded":"2024-09-13T17:16:30.647595+0000","last_fullsized":"2024-09-13T17:16:30.647595+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:29.829550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249520+0000","last_change":"2024-09-13T17:16:29.655005+0000","last_active":"2024-09-13T17:16:32.249520+0000","last_peered":"2024-09-13T17:16:32.249520+0000","last_clean":"2024-09-13T17:16:32.249520+0000","last_became_active":"2024-09-13T17:16:29.652689+0000","last_became_peered":"2024-09-13T17:16:29.652689+0000","last_unstale":"2024-09-13T17:16:32.249520+0000","last_undegraded":"2024-09-13T17:16:32.249520+0000","last_fullsized":"2024-09-13T17:16:32.249520+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:19.689858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249755+0000","last_change":"2024-09-13T17:16:29.661725+0000","last_active":"2024-09-13T17:16:32.249755+0000","last_peered":"2024-09-13T17:16:32.249755+0000","last_clean":"2024-09-13T17:16:32.249755+0000","last_became_active":"2024-09-13T17:16:29.660371+0000","last_became_peered":"2024-09-13T17:16:29.660371+0000","last_unstale":"2024-09-13T17:16:32.249755+0000","last_undegraded":"2024-09-13T17:16:32.249755+0000","last_fullsized":"2024-09-13T17:16:32.249755+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:16.779035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"124'4","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649531+0000","last_change":"2024-09-13T17:16:29.203458+0000","last_active":"2024-09-13T17:16:30.649531+0000","last_peered":"2024-09-13T17:16:30.649531+0000","last_clean":"2024-09-13T17:16:30.649531+0000","last_became_active":"2024-09-13T17:16:29.203136+0000","last_became_peered":"2024-09-13T17:16:29.203136+0000","last_unstale":"2024-09-13T17:16:30.649531+0000","last_undegraded":"2024-09-13T17:16:30.649531+0000","last_fullsized":"2024-09-13T17:16:30.649531+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:18.680008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003122+0000","last_change":"2024-09-13T17:16:29.661702+0000","last_active":"2024-09-13T17:16:32.003122+0000","last_peered":"2024-09-13T17:16:32.003122+0000","last_clean":"2024-09-13T17:16:32.003122+0000","last_became_active":"2024-09-13T17:16:29.661388+0000","last_became_peered":"2024-09-13T17:16:29.661388+0000","last_unstale":"2024-09-13T17:16:32.003122+0000","last_undegraded":"2024-09-13T17:16:32.003122+0000","last_fullsized":"2024-09-13T17:16:32.003122+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:32.791598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412132+0000","last_change":"2024-09-13T17:16:29.207258+0000","last_active":"2024-09-13T17:16:31.412132+0000","last_peered":"2024-09-13T17:16:31.412132+0000","last_clean":"2024-09-13T17:16:31.412132+0000","last_became_active":"2024-09-13T17:16:29.206951+0000","last_became_peered":"2024-09-13T17:16:29.206951+0000","last_unstale":"2024-09-13T17:16:31.412132+0000","last_undegraded":"2024-09-13T17:16:31.412132+0000","last_fullsized":"2024-09-13T17:16:31.412132+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:51:31.573685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249686+0000","last_change":"2024-09-13T17:16:29.651826+0000","last_active":"2024-09-13T17:16:32.249686+0000","last_peered":"2024-09-13T17:16:32.249686+0000","last_clean":"2024-09-13T17:16:32.249686+0000","last_became_active":"2024-09-13T17:16:29.650291+0000","last_became_peered":"2024-09-13T17:16:29.650291+0000","last_unstale":"2024-09-13T17:16:32.249686+0000","last_undegraded":"2024-09-13T17:16:32.249686+0000","last_fullsized":"2024-09-13T17:16:32.249686+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:51.538560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485910+0000","last_change":"2024-09-13T17:16:29.210221+0000","last_active":"2024-09-13T17:16:31.485910+0000","last_peered":"2024-09-13T17:16:31.485910+0000","last_clean":"2024-09-13T17:16:31.485910+0000","last_became_active":"2024-09-13T17:16:29.209963+0000","last_became_peered":"2024-09-13T17:16:29.209963+0000","last_unstale":"2024-09-13T17:16:31.485910+0000","last_undegraded":"2024-09-13T17:16:31.485910+0000","last_fullsized":"2024-09-13T17:16:31.485910+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.290230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412240+0000","last_change":"2024-09-13T17:16:29.664882+0000","last_active":"2024-09-13T17:16:31.412240+0000","last_peered":"2024-09-13T17:16:31.412240+0000","last_clean":"2024-09-13T17:16:31.412240+0000","last_became_active":"2024-09-13T17:16:29.661267+0000","last_became_peered":"2024-09-13T17:16:29.661267+0000","last_unstale":"2024-09-13T17:16:31.412240+0000","last_undegraded":"2024-09-13T17:16:31.412240+0000","last_fullsized":"2024-09-13T17:16:31.412240+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:01.626883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485806+0000","last_change":"2024-09-13T17:16:29.207591+0000","last_active":"2024-09-13T17:16:31.485806+0000","last_peered":"2024-09-13T17:16:31.485806+0000","last_clean":"2024-09-13T17:16:31.485806+0000","last_became_active":"2024-09-13T17:16:29.207260+0000","last_became_peered":"2024-09-13T17:16:29.207260+0000","last_unstale":"2024-09-13T17:16:31.485806+0000","last_undegraded":"2024-09-13T17:16:31.485806+0000","last_fullsized":"2024-09-13T17:16:31.485806+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:13:41.113112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648120+0000","last_change":"2024-09-13T17:16:29.655951+0000","last_active":"2024-09-13T17:16:30.648120+0000","last_peered":"2024-09-13T17:16:30.648120+0000","last_clean":"2024-09-13T17:16:30.648120+0000","last_became_active":"2024-09-13T17:16:29.654996+0000","last_became_peered":"2024-09-13T17:16:29.654996+0000","last_unstale":"2024-09-13T17:16:30.648120+0000","last_undegraded":"2024-09-13T17:16:30.648120+0000","last_fullsized":"2024-09-13T17:16:30.648120+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:28.720147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649118+0000","last_change":"2024-09-13T17:16:29.204745+0000","last_active":"2024-09-13T17:16:30.649118+0000","last_peered":"2024-09-13T17:16:30.649118+0000","last_clean":"2024-09-13T17:16:30.649118+0000","last_became_active":"2024-09-13T17:16:29.204482+0000","last_became_peered":"2024-09-13T17:16:29.204482+0000","last_unstale":"2024-09-13T17:16:30.649118+0000","last_undegraded":"2024-09-13T17:16:30.649118+0000","last_fullsized":"2024-09-13T17:16:30.649118+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:15.769185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648911+0000","last_change":"2024-09-13T17:16:29.669165+0000","last_active":"2024-09-13T17:16:30.648911+0000","last_peered":"2024-09-13T17:16:30.648911+0000","last_clean":"2024-09-13T17:16:30.648911+0000","last_became_active":"2024-09-13T17:16:29.667601+0000","last_became_peered":"2024-09-13T17:16:29.667601+0000","last_unstale":"2024-09-13T17:16:30.648911+0000","last_undegraded":"2024-09-13T17:16:30.648911+0000","last_fullsized":"2024-09-13T17:16:30.648911+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:19.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863480+0000","last_change":"2024-09-13T17:16:29.204954+0000","last_active":"2024-09-13T17:16:31.863480+0000","last_peered":"2024-09-13T17:16:31.863480+0000","last_clean":"2024-09-13T17:16:31.863480+0000","last_became_active":"2024-09-13T17:16:29.204621+0000","last_became_peered":"2024-09-13T17:16:29.204621+0000","last_unstale":"2024-09-13T17:16:31.863480+0000","last_undegraded":"2024-09-13T17:16:31.863480+0000","last_fullsized":"2024-09-13T17:16:31.863480+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:25:48.218691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647431+0000","last_change":"2024-09-13T17:16:29.659876+0000","last_active":"2024-09-13T17:16:30.647431+0000","last_peered":"2024-09-13T17:16:30.647431+0000","last_clean":"2024-09-13T17:16:30.647431+0000","last_became_active":"2024-09-13T17:16:29.659640+0000","last_became_peered":"2024-09-13T17:16:29.659640+0000","last_unstale":"2024-09-13T17:16:30.647431+0000","last_undegraded":"2024-09-13T17:16:30.647431+0000","last_fullsized":"2024-09-13T17:16:30.647431+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:51.623446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648910+0000","last_change":"2024-09-13T17:16:29.202965+0000","last_active":"2024-09-13T17:16:30.648910+0000","last_peered":"2024-09-13T17:16:30.648910+0000","last_clean":"2024-09-13T17:16:30.648910+0000","last_became_active":"2024-09-13T17:16:29.202345+0000","last_became_peered":"2024-09-13T17:16:29.202345+0000","last_unstale":"2024-09-13T17:16:30.648910+0000","last_undegraded":"2024-09-13T17:16:30.648910+0000","last_fullsized":"2024-09-13T17:16:30.648910+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:41.248992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649230+0000","last_change":"2024-09-13T17:16:29.660341+0000","last_active":"2024-09-13T17:16:30.649230+0000","last_peered":"2024-09-13T17:16:30.649230+0000","last_clean":"2024-09-13T17:16:30.649230+0000","last_became_active":"2024-09-13T17:16:29.657065+0000","last_became_peered":"2024-09-13T17:16:29.657065+0000","last_unstale":"2024-09-13T17:16:30.649230+0000","last_undegraded":"2024-09-13T17:16:30.649230+0000","last_fullsized":"2024-09-13T17:16:30.649230+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:46.073611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647452+0000","last_change":"2024-09-13T17:16:29.209359+0000","last_active":"2024-09-13T17:16:30.647452+0000","last_peered":"2024-09-13T17:16:30.647452+0000","last_clean":"2024-09-13T17:16:30.647452+0000","last_became_active":"2024-09-13T17:16:29.208676+0000","last_became_peered":"2024-09-13T17:16:29.208676+0000","last_unstale":"2024-09-13T17:16:30.647452+0000","last_undegraded":"2024-09-13T17:16:30.647452+0000","last_fullsized":"2024-09-13T17:16:30.647452+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:12.322072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649512+0000","last_change":"2024-09-13T17:16:29.661135+0000","last_active":"2024-09-13T17:16:30.649512+0000","last_peered":"2024-09-13T17:16:30.649512+0000","last_clean":"2024-09-13T17:16:30.649512+0000","last_became_active":"2024-09-13T17:16:29.660850+0000","last_became_peered":"2024-09-13T17:16:29.660850+0000","last_unstale":"2024-09-13T17:16:30.649512+0000","last_undegraded":"2024-09-13T17:16:30.649512+0000","last_fullsized":"2024-09-13T17:16:30.649512+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:48.950157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485957+0000","last_change":"2024-09-13T17:16:29.213989+0000","last_active":"2024-09-13T17:16:31.485957+0000","last_peered":"2024-09-13T17:16:31.485957+0000","last_clean":"2024-09-13T17:16:31.485957+0000","last_became_active":"2024-09-13T17:16:29.213715+0000","last_became_peered":"2024-09-13T17:16:29.213715+0000","last_unstale":"2024-09-13T17:16:31.485957+0000","last_undegraded":"2024-09-13T17:16:31.485957+0000","last_fullsized":"2024-09-13T17:16:31.485957+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:57:00.489585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003100+0000","last_change":"2024-09-13T17:16:29.672374+0000","last_active":"2024-09-13T17:16:32.003100+0000","last_peered":"2024-09-13T17:16:32.003100+0000","last_clean":"2024-09-13T17:16:32.003100+0000","last_became_active":"2024-09-13T17:16:29.672044+0000","last_became_peered":"2024-09-13T17:16:29.672044+0000","last_unstale":"2024-09-13T17:16:32.003100+0000","last_undegraded":"2024-09-13T17:16:32.003100+0000","last_fullsized":"2024-09-13T17:16:32.003100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:01:21.413522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863548+0000","last_change":"2024-09-13T17:16:29.207160+0000","last_active":"2024-09-13T17:16:31.863548+0000","last_peered":"2024-09-13T17:16:31.863548+0000","last_clean":"2024-09-13T17:16:31.863548+0000","last_became_active":"2024-09-13T17:16:29.206877+0000","last_became_peered":"2024-09-13T17:16:29.206877+0000","last_unstale":"2024-09-13T17:16:31.863548+0000","last_undegraded":"2024-09-13T17:16:31.863548+0000","last_fullsized":"2024-09-13T17:16:31.863548+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:13.601999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863316+0000","last_change":"2024-09-13T17:16:29.664255+0000","last_active":"2024-09-13T17:16:31.863316+0000","last_peered":"2024-09-13T17:16:31.863316+0000","last_clean":"2024-09-13T17:16:31.863316+0000","last_became_active":"2024-09-13T17:16:29.662911+0000","last_became_peered":"2024-09-13T17:16:29.662911+0000","last_unstale":"2024-09-13T17:16:31.863316+0000","last_undegraded":"2024-09-13T17:16:31.863316+0000","last_fullsized":"2024-09-13T17:16:31.863316+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:19.954724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"124'3","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648177+0000","last_change":"2024-09-13T17:16:29.210846+0000","last_active":"2024-09-13T17:16:30.648177+0000","last_peered":"2024-09-13T17:16:30.648177+0000","last_clean":"2024-09-13T17:16:30.648177+0000","last_became_active":"2024-09-13T17:16:29.210596+0000","last_became_peered":"2024-09-13T17:16:29.210596+0000","last_unstale":"2024-09-13T17:16:30.648177+0000","last_undegraded":"2024-09-13T17:16:30.648177+0000","last_fullsized":"2024-09-13T17:16:30.648177+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:13.988363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649100+0000","last_change":"2024-09-13T17:16:29.660476+0000","last_active":"2024-09-13T17:16:30.649100+0000","last_peered":"2024-09-13T17:16:30.649100+0000","last_clean":"2024-09-13T17:16:30.649100+0000","last_became_active":"2024-09-13T17:16:29.655669+0000","last_became_peered":"2024-09-13T17:16:29.655669+0000","last_unstale":"2024-09-13T17:16:30.649100+0000","last_undegraded":"2024-09-13T17:16:30.649100+0000","last_fullsized":"2024-09-13T17:16:30.649100+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:01:28.647894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"124'6","reported_seq":17,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648015+0000","last_change":"2024-09-13T17:16:29.204587+0000","last_active":"2024-09-13T17:16:30.648015+0000","last_peered":"2024-09-13T17:16:30.648015+0000","last_clean":"2024-09-13T17:16:30.648015+0000","last_became_active":"2024-09-13T17:16:29.204304+0000","last_became_peered":"2024-09-13T17:16:29.204304+0000","last_unstale":"2024-09-13T17:16:30.648015+0000","last_undegraded":"2024-09-13T17:16:30.648015+0000","last_fullsized":"2024-09-13T17:16:30.648015+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:11.529554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863403+0000","last_change":"2024-09-13T17:16:29.654760+0000","last_active":"2024-09-13T17:16:31.863403+0000","last_peered":"2024-09-13T17:16:31.863403+0000","last_clean":"2024-09-13T17:16:31.863403+0000","last_became_active":"2024-09-13T17:16:29.654034+0000","last_became_peered":"2024-09-13T17:16:29.654034+0000","last_unstale":"2024-09-13T17:16:31.863403+0000","last_undegraded":"2024-09-13T17:16:31.863403+0000","last_fullsized":"2024-09-13T17:16:31.863403+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:53.523884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249597+0000","last_change":"2024-09-13T17:16:29.210084+0000","last_active":"2024-09-13T17:16:32.249597+0000","last_peered":"2024-09-13T17:16:32.249597+0000","last_clean":"2024-09-13T17:16:32.249597+0000","last_became_active":"2024-09-13T17:16:29.209426+0000","last_became_peered":"2024-09-13T17:16:29.209426+0000","last_unstale":"2024-09-13T17:16:32.249597+0000","last_undegraded":"2024-09-13T17:16:32.249597+0000","last_fullsized":"2024-09-13T17:16:32.249597+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:54.716094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863311+0000","last_change":"2024-09-13T17:16:29.654859+0000","last_active":"2024-09-13T17:16:31.863311+0000","last_peered":"2024-09-13T17:16:31.863311+0000","last_clean":"2024-09-13T17:16:31.863311+0000","last_became_active":"2024-09-13T17:16:29.654613+0000","last_became_peered":"2024-09-13T17:16:29.654613+0000","last_unstale":"2024-09-13T17:16:31.863311+0000","last_undegraded":"2024-09-13T17:16:31.863311+0000","last_fullsized":"2024-09-13T17:16:31.863311+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:19.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002843+0000","last_change":"2024-09-13T17:16:29.214047+0000","last_active":"2024-09-13T17:16:32.002843+0000","last_peered":"2024-09-13T17:16:32.002843+0000","last_clean":"2024-09-13T17:16:32.002843+0000","last_became_active":"2024-09-13T17:16:29.213794+0000","last_became_peered":"2024-09-13T17:16:29.213794+0000","last_unstale":"2024-09-13T17:16:32.002843+0000","last_undegraded":"2024-09-13T17:16:32.002843+0000","last_fullsized":"2024-09-13T17:16:32.002843+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:35.567486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863396+0000","last_change":"2024-09-13T17:16:29.656281+0000","last_active":"2024-09-13T17:16:31.863396+0000","last_peered":"2024-09-13T17:16:31.863396+0000","last_clean":"2024-09-13T17:16:31.863396+0000","last_became_active":"2024-09-13T17:16:29.655999+0000","last_became_peered":"2024-09-13T17:16:29.655999+0000","last_unstale":"2024-09-13T17:16:31.863396+0000","last_undegraded":"2024-09-13T17:16:31.863396+0000","last_fullsized":"2024-09-13T17:16:31.863396+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:38.893591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"124'8","reported_seq":19,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485524+0000","last_change":"2024-09-13T17:16:29.217569+0000","last_active":"2024-09-13T17:16:31.485524+0000","last_peered":"2024-09-13T17:16:31.485524+0000","last_clean":"2024-09-13T17:16:31.485524+0000","last_became_active":"2024-09-13T17:16:29.217240+0000","last_became_peered":"2024-09-13T17:16:29.217240+0000","last_unstale":"2024-09-13T17:16:31.485524+0000","last_undegraded":"2024-09-13T17:16:31.485524+0000","last_fullsized":"2024-09-13T17:16:31.485524+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.886265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485992+0000","last_change":"2024-09-13T17:16:29.675256+0000","last_active":"2024-09-13T17:16:31.485992+0000","last_peered":"2024-09-13T17:16:31.485992+0000","last_clean":"2024-09-13T17:16:31.485992+0000","last_became_active":"2024-09-13T17:16:29.674732+0000","last_became_peered":"2024-09-13T17:16:29.674732+0000","last_unstale":"2024-09-13T17:16:31.485992+0000","last_undegraded":"2024-09-13T17:16:31.485992+0000","last_fullsized":"2024-09-13T17:16:31.485992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:30.679012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648482+0000","last_change":"2024-09-13T17:16:29.206006+0000","last_active":"2024-09-13T17:16:30.648482+0000","last_peered":"2024-09-13T17:16:30.648482+0000","last_clean":"2024-09-13T17:16:30.648482+0000","last_became_active":"2024-09-13T17:16:29.203992+0000","last_became_peered":"2024-09-13T17:16:29.203992+0000","last_unstale":"2024-09-13T17:16:30.648482+0000","last_undegraded":"2024-09-13T17:16:30.648482+0000","last_fullsized":"2024-09-13T17:16:30.648482+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:04.922113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249759+0000","last_change":"2024-09-13T17:16:29.656030+0000","last_active":"2024-09-13T17:16:32.249759+0000","last_peered":"2024-09-13T17:16:32.249759+0000","last_clean":"2024-09-13T17:16:32.249759+0000","last_became_active":"2024-09-13T17:16:29.655037+0000","last_became_peered":"2024-09-13T17:16:29.655037+0000","last_unstale":"2024-09-13T17:16:32.249759+0000","last_undegraded":"2024-09-13T17:16:32.249759+0000","last_fullsized":"2024-09-13T17:16:32.249759+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:34.114361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.486034+0000","last_change":"2024-09-13T17:16:29.200192+0000","last_active":"2024-09-13T17:16:31.486034+0000","last_peered":"2024-09-13T17:16:31.486034+0000","last_clean":"2024-09-13T17:16:31.486034+0000","last_became_active":"2024-09-13T17:16:29.199877+0000","last_became_peered":"2024-09-13T17:16:29.199877+0000","last_unstale":"2024-09-13T17:16:31.486034+0000","last_undegraded":"2024-09-13T17:16:31.486034+0000","last_fullsized":"2024-09-13T17:16:31.486034+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:11.220144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411949+0000","last_change":"2024-09-13T17:16:29.656519+0000","last_active":"2024-09-13T17:16:31.411949+0000","last_peered":"2024-09-13T17:16:31.411949+0000","last_clean":"2024-09-13T17:16:31.411949+0000","last_became_active":"2024-09-13T17:16:29.655247+0000","last_became_peered":"2024-09-13T17:16:29.655247+0000","last_unstale":"2024-09-13T17:16:31.411949+0000","last_undegraded":"2024-09-13T17:16:31.411949+0000","last_fullsized":"2024-09-13T17:16:31.411949+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:51.670675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"124'1","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002933+0000","last_change":"2024-09-13T17:16:29.204169+0000","last_active":"2024-09-13T17:16:32.002933+0000","last_peered":"2024-09-13T17:16:32.002933+0000","last_clean":"2024-09-13T17:16:32.002933+0000","last_became_active":"2024-09-13T17:16:29.203876+0000","last_became_peered":"2024-09-13T17:16:29.203876+0000","last_unstale":"2024-09-13T17:16:32.002933+0000","last_undegraded":"2024-09-13T17:16:32.002933+0000","last_fullsized":"2024-09-13T17:16:32.002933+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:59:04.887710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412254+0000","last_change":"2024-09-13T17:16:29.655915+0000","last_active":"2024-09-13T17:16:31.412254+0000","last_peered":"2024-09-13T17:16:31.412254+0000","last_clean":"2024-09-13T17:16:31.412254+0000","last_became_active":"2024-09-13T17:16:29.655625+0000","last_became_peered":"2024-09-13T17:16:29.655625+0000","last_unstale":"2024-09-13T17:16:31.412254+0000","last_undegraded":"2024-09-13T17:16:31.412254+0000","last_fullsized":"2024-09-13T17:16:31.412254+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:53:42.258842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648797+0000","last_change":"2024-09-13T17:16:29.202384+0000","last_active":"2024-09-13T17:16:30.648797+0000","last_peered":"2024-09-13T17:16:30.648797+0000","last_clean":"2024-09-13T17:16:30.648797+0000","last_became_active":"2024-09-13T17:16:29.202101+0000","last_became_peered":"2024-09-13T17:16:29.202101+0000","last_unstale":"2024-09-13T17:16:30.648797+0000","last_undegraded":"2024-09-13T17:16:30.648797+0000","last_fullsized":"2024-09-13T17:16:30.648797+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:33.104511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.411828+0000","last_change":"2024-09-13T17:16:29.663532+0000","last_active":"2024-09-13T17:16:31.411828+0000","last_peered":"2024-09-13T17:16:31.411828+0000","last_clean":"2024-09-13T17:16:31.411828+0000","last_became_active":"2024-09-13T17:16:29.658714+0000","last_became_peered":"2024-09-13T17:16:29.658714+0000","last_unstale":"2024-09-13T17:16:31.411828+0000","last_undegraded":"2024-09-13T17:16:31.411828+0000","last_fullsized":"2024-09-13T17:16:31.411828+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:24.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249693+0000","last_change":"2024-09-13T17:16:29.210025+0000","last_active":"2024-09-13T17:16:32.249693+0000","last_peered":"2024-09-13T17:16:32.249693+0000","last_clean":"2024-09-13T17:16:32.249693+0000","last_became_active":"2024-09-13T17:16:29.209766+0000","last_became_peered":"2024-09-13T17:16:29.209766+0000","last_unstale":"2024-09-13T17:16:32.249693+0000","last_undegraded":"2024-09-13T17:16:32.249693+0000","last_fullsized":"2024-09-13T17:16:32.249693+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:47.290230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647516+0000","last_change":"2024-09-13T17:16:29.651378+0000","last_active":"2024-09-13T17:16:30.647516+0000","last_peered":"2024-09-13T17:16:30.647516+0000","last_clean":"2024-09-13T17:16:30.647516+0000","last_became_active":"2024-09-13T17:16:29.651128+0000","last_became_peered":"2024-09-13T17:16:29.651128+0000","last_unstale":"2024-09-13T17:16:30.647516+0000","last_undegraded":"2024-09-13T17:16:30.647516+0000","last_fullsized":"2024-09-13T17:16:30.647516+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:52.773195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485611+0000","last_change":"2024-09-13T17:16:29.204244+0000","last_active":"2024-09-13T17:16:31.485611+0000","last_peered":"2024-09-13T17:16:31.485611+0000","last_clean":"2024-09-13T17:16:31.485611+0000","last_became_active":"2024-09-13T17:16:29.203970+0000","last_became_peered":"2024-09-13T17:16:29.203970+0000","last_unstale":"2024-09-13T17:16:31.485611+0000","last_undegraded":"2024-09-13T17:16:31.485611+0000","last_fullsized":"2024-09-13T17:16:31.485611+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:26.607101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412011+0000","last_change":"2024-09-13T17:16:29.658856+0000","last_active":"2024-09-13T17:16:31.412011+0000","last_peered":"2024-09-13T17:16:31.412011+0000","last_clean":"2024-09-13T17:16:31.412011+0000","last_became_active":"2024-09-13T17:16:29.658347+0000","last_became_peered":"2024-09-13T17:16:29.658347+0000","last_unstale":"2024-09-13T17:16:31.412011+0000","last_undegraded":"2024-09-13T17:16:31.412011+0000","last_fullsized":"2024-09-13T17:16:31.412011+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:35:57.458770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.249543+0000","last_change":"2024-09-13T17:16:29.207781+0000","last_active":"2024-09-13T17:16:32.249543+0000","last_peered":"2024-09-13T17:16:32.249543+0000","last_clean":"2024-09-13T17:16:32.249543+0000","last_became_active":"2024-09-13T17:16:29.207515+0000","last_became_peered":"2024-09-13T17:16:29.207515+0000","last_unstale":"2024-09-13T17:16:32.249543+0000","last_undegraded":"2024-09-13T17:16:32.249543+0000","last_fullsized":"2024-09-13T17:16:32.249543+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:21:43.178935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863229+0000","last_change":"2024-09-13T17:16:29.660547+0000","last_active":"2024-09-13T17:16:31.863229+0000","last_peered":"2024-09-13T17:16:31.863229+0000","last_clean":"2024-09-13T17:16:31.863229+0000","last_became_active":"2024-09-13T17:16:29.660291+0000","last_became_peered":"2024-09-13T17:16:29.660291+0000","last_unstale":"2024-09-13T17:16:31.863229+0000","last_undegraded":"2024-09-13T17:16:31.863229+0000","last_fullsized":"2024-09-13T17:16:31.863229+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:08.675355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647559+0000","last_change":"2024-09-13T17:16:29.203550+0000","last_active":"2024-09-13T17:16:30.647559+0000","last_peered":"2024-09-13T17:16:30.647559+0000","last_clean":"2024-09-13T17:16:30.647559+0000","last_became_active":"2024-09-13T17:16:29.203281+0000","last_became_peered":"2024-09-13T17:16:29.203281+0000","last_unstale":"2024-09-13T17:16:30.647559+0000","last_undegraded":"2024-09-13T17:16:30.647559+0000","last_fullsized":"2024-09-13T17:16:30.647559+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:37.883741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003019+0000","last_change":"2024-09-13T17:16:29.669011+0000","last_active":"2024-09-13T17:16:32.003019+0000","last_peered":"2024-09-13T17:16:32.003019+0000","last_clean":"2024-09-13T17:16:32.003019+0000","last_became_active":"2024-09-13T17:16:29.668700+0000","last_became_peered":"2024-09-13T17:16:29.668700+0000","last_unstale":"2024-09-13T17:16:32.003019+0000","last_undegraded":"2024-09-13T17:16:32.003019+0000","last_fullsized":"2024-09-13T17:16:32.003019+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:06.402094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"124'2","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863599+0000","last_change":"2024-09-13T17:16:29.212961+0000","last_active":"2024-09-13T17:16:31.863599+0000","last_peered":"2024-09-13T17:16:31.863599+0000","last_clean":"2024-09-13T17:16:31.863599+0000","last_became_active":"2024-09-13T17:16:29.212313+0000","last_became_peered":"2024-09-13T17:16:29.212313+0000","last_unstale":"2024-09-13T17:16:31.863599+0000","last_undegraded":"2024-09-13T17:16:31.863599+0000","last_fullsized":"2024-09-13T17:16:31.863599+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:52.704901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648425+0000","last_change":"2024-09-13T17:16:29.667626+0000","last_active":"2024-09-13T17:16:30.648425+0000","last_peered":"2024-09-13T17:16:30.648425+0000","last_clean":"2024-09-13T17:16:30.648425+0000","last_became_active":"2024-09-13T17:16:29.667191+0000","last_became_peered":"2024-09-13T17:16:29.667191+0000","last_unstale":"2024-09-13T17:16:30.648425+0000","last_undegraded":"2024-09-13T17:16:30.648425+0000","last_fullsized":"2024-09-13T17:16:30.648425+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:42:57.380064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.003026+0000","last_change":"2024-09-13T17:16:29.199642+0000","last_active":"2024-09-13T17:16:32.003026+0000","last_peered":"2024-09-13T17:16:32.003026+0000","last_clean":"2024-09-13T17:16:32.003026+0000","last_became_active":"2024-09-13T17:16:29.197660+0000","last_became_peered":"2024-09-13T17:16:29.197660+0000","last_unstale":"2024-09-13T17:16:32.003026+0000","last_undegraded":"2024-09-13T17:16:32.003026+0000","last_fullsized":"2024-09-13T17:16:32.003026+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:32:42.080221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647888+0000","last_change":"2024-09-13T17:16:29.659508+0000","last_active":"2024-09-13T17:16:30.647888+0000","last_peered":"2024-09-13T17:16:30.647888+0000","last_clean":"2024-09-13T17:16:30.647888+0000","last_became_active":"2024-09-13T17:16:29.658736+0000","last_became_peered":"2024-09-13T17:16:29.658736+0000","last_unstale":"2024-09-13T17:16:30.647888+0000","last_undegraded":"2024-09-13T17:16:30.647888+0000","last_fullsized":"2024-09-13T17:16:30.647888+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:44.049952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648207+0000","last_change":"2024-09-13T17:16:29.204371+0000","last_active":"2024-09-13T17:16:30.648207+0000","last_peered":"2024-09-13T17:16:30.648207+0000","last_clean":"2024-09-13T17:16:30.648207+0000","last_became_active":"2024-09-13T17:16:29.204095+0000","last_became_peered":"2024-09-13T17:16:29.204095+0000","last_unstale":"2024-09-13T17:16:30.648207+0000","last_undegraded":"2024-09-13T17:16:30.648207+0000","last_fullsized":"2024-09-13T17:16:30.648207+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:52.704901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:32.002814+0000","last_change":"2024-09-13T17:16:29.669265+0000","last_active":"2024-09-13T17:16:32.002814+0000","last_peered":"2024-09-13T17:16:32.002814+0000","last_clean":"2024-09-13T17:16:32.002814+0000","last_became_active":"2024-09-13T17:16:29.668408+0000","last_became_peered":"2024-09-13T17:16:29.668408+0000","last_unstale":"2024-09-13T17:16:32.002814+0000","last_undegraded":"2024-09-13T17:16:32.002814+0000","last_fullsized":"2024-09-13T17:16:32.002814+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:24.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863494+0000","last_change":"2024-09-13T17:16:29.216236+0000","last_active":"2024-09-13T17:16:31.863494+0000","last_peered":"2024-09-13T17:16:31.863494+0000","last_clean":"2024-09-13T17:16:31.863494+0000","last_became_active":"2024-09-13T17:16:29.215681+0000","last_became_peered":"2024-09-13T17:16:29.215681+0000","last_unstale":"2024-09-13T17:16:31.863494+0000","last_undegraded":"2024-09-13T17:16:31.863494+0000","last_fullsized":"2024-09-13T17:16:31.863494+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:30.343718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648787+0000","last_change":"2024-09-13T17:16:29.664568+0000","last_active":"2024-09-13T17:16:30.648787+0000","last_peered":"2024-09-13T17:16:30.648787+0000","last_clean":"2024-09-13T17:16:30.648787+0000","last_became_active":"2024-09-13T17:16:29.664203+0000","last_became_peered":"2024-09-13T17:16:29.664203+0000","last_unstale":"2024-09-13T17:16:30.648787+0000","last_undegraded":"2024-09-13T17:16:30.648787+0000","last_fullsized":"2024-09-13T17:16:30.648787+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:05.931963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649170+0000","last_change":"2024-09-13T17:16:29.198464+0000","last_active":"2024-09-13T17:16:30.649170+0000","last_peered":"2024-09-13T17:16:30.649170+0000","last_clean":"2024-09-13T17:16:30.649170+0000","last_became_active":"2024-09-13T17:16:29.198179+0000","last_became_peered":"2024-09-13T17:16:29.198179+0000","last_unstale":"2024-09-13T17:16:30.649170+0000","last_undegraded":"2024-09-13T17:16:30.649170+0000","last_fullsized":"2024-09-13T17:16:30.649170+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:50.528710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649012+0000","last_change":"2024-09-13T17:16:29.664914+0000","last_active":"2024-09-13T17:16:30.649012+0000","last_peered":"2024-09-13T17:16:30.649012+0000","last_clean":"2024-09-13T17:16:30.649012+0000","last_became_active":"2024-09-13T17:16:29.664103+0000","last_became_peered":"2024-09-13T17:16:29.664103+0000","last_unstale":"2024-09-13T17:16:30.649012+0000","last_undegraded":"2024-09-13T17:16:30.649012+0000","last_fullsized":"2024-09-13T17:16:30.649012+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:49.259587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.648891+0000","last_change":"2024-09-13T17:16:29.203661+0000","last_active":"2024-09-13T17:16:30.648891+0000","last_peered":"2024-09-13T17:16:30.648891+0000","last_clean":"2024-09-13T17:16:30.648891+0000","last_became_active":"2024-09-13T17:16:29.203378+0000","last_became_peered":"2024-09-13T17:16:29.203378+0000","last_unstale":"2024-09-13T17:16:30.648891+0000","last_undegraded":"2024-09-13T17:16:30.648891+0000","last_fullsized":"2024-09-13T17:16:30.648891+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:36.661530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.412060+0000","last_change":"2024-09-13T17:16:29.656308+0000","last_active":"2024-09-13T17:16:31.412060+0000","last_peered":"2024-09-13T17:16:31.412060+0000","last_clean":"2024-09-13T17:16:31.412060+0000","last_became_active":"2024-09-13T17:16:29.655207+0000","last_became_peered":"2024-09-13T17:16:29.655207+0000","last_unstale":"2024-09-13T17:16:31.412060+0000","last_undegraded":"2024-09-13T17:16:31.412060+0000","last_fullsized":"2024-09-13T17:16:31.412060+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:42:05.347447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.485516+0000","last_change":"2024-09-13T17:16:29.211105+0000","last_active":"2024-09-13T17:16:31.485516+0000","last_peered":"2024-09-13T17:16:31.485516+0000","last_clean":"2024-09-13T17:16:31.485516+0000","last_became_active":"2024-09-13T17:16:29.210778+0000","last_became_peered":"2024-09-13T17:16:29.210778+0000","last_unstale":"2024-09-13T17:16:31.485516+0000","last_undegraded":"2024-09-13T17:16:31.485516+0000","last_fullsized":"2024-09-13T17:16:31.485516+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:05.921728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647681+0000","last_change":"2024-09-13T17:16:29.665332+0000","last_active":"2024-09-13T17:16:30.647681+0000","last_peered":"2024-09-13T17:16:30.647681+0000","last_clean":"2024-09-13T17:16:30.647681+0000","last_became_active":"2024-09-13T17:16:29.664916+0000","last_became_peered":"2024-09-13T17:16:29.664916+0000","last_unstale":"2024-09-13T17:16:30.647681+0000","last_undegraded":"2024-09-13T17:16:30.647681+0000","last_fullsized":"2024-09-13T17:16:30.647681+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:23:14.868860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.647808+0000","last_change":"2024-09-13T17:16:29.205289+0000","last_active":"2024-09-13T17:16:30.647808+0000","last_peered":"2024-09-13T17:16:30.647808+0000","last_clean":"2024-09-13T17:16:30.647808+0000","last_became_active":"2024-09-13T17:16:29.204985+0000","last_became_peered":"2024-09-13T17:16:29.204985+0000","last_unstale":"2024-09-13T17:16:30.647808+0000","last_undegraded":"2024-09-13T17:16:30.647808+0000","last_fullsized":"2024-09-13T17:16:30.647808+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:55:08.123417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:31.863163+0000","last_change":"2024-09-13T17:16:29.653022+0000","last_active":"2024-09-13T17:16:31.863163+0000","last_peered":"2024-09-13T17:16:31.863163+0000","last_clean":"2024-09-13T17:16:31.863163+0000","last_became_active":"2024-09-13T17:16:29.651275+0000","last_became_peered":"2024-09-13T17:16:29.651275+0000","last_unstale":"2024-09-13T17:16:31.863163+0000","last_undegraded":"2024-09-13T17:16:31.863163+0000","last_fullsized":"2024-09-13T17:16:31.863163+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:29.177377+0000","last_clean_scrub_stamp":"2024-09-13T17:16:29.177377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:45.563984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T17:16:30.649373+0000","last_change":"2024-09-13T17:16:29.202273+0000","last_active":"2024-09-13T17:16:30.649373+0000","last_peered":"2024-09-13T17:16:30.649373+0000","last_clean":"2024-09-13T17:16:30.649373+0000","last_became_active":"2024-09-13T17:16:29.201980+0000","last_became_peered":"2024-09-13T17:16:29.201980+0000","last_unstale":"2024-09-13T17:16:30.649373+0000","last_undegraded":"2024-09-13T17:16:30.649373+0000","last_fullsized":"2024-09-13T17:16:30.649373+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:16:28.167527+0000","last_clean_scrub_stamp":"2024-09-13T17:16:28.167527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:49.450970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":254,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T17:16:29.645270+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:16:29.645270+0000","last_peered":"2024-09-13T17:16:29.645270+0000","last_clean":"2024-09-13T17:16:29.645270+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:16:29.645270+0000","last_undegraded":"2024-09-13T17:16:29.645270+0000","last_fullsized":"2024-09-13T17:16:29.645270+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574954,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51276,"kb_used_data":2104,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320564,"statfs":{"total":96636764160,"available":96584257536,"internally_reserved":0,"allocated":2154496,"data_stored":1271828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50330059},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.66300000000000003,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.78900000000000003,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68200000000000005},"min":{"1min":0.43099999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.876,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":1.004}]},{"osd":1,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.68400000000000005,"15min":0.65100000000000002},"min":{"1min":0.495,"5min":0.433,"15min":0.40400000000000003},"max":{"1min":0.85499999999999998,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.76700000000000002,"15min":0.71099999999999997},"min":{"1min":0.48399999999999999,"5min":0.47299999999999998,"15min":0.437},"max":{"1min":0.89500000000000002,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.86399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.69799999999999995,"15min":0.69899999999999995},"min":{"1min":0.53200000000000003,"5min":0.47199999999999998,"15min":0.42399999999999999},"max":{"1min":0.80700000000000005,"5min":1.544,"15min":1.544},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.70499999999999996,"15min":0.70999999999999996},"min":{"1min":0.51600000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.90800000000000003,"5min":1.605,"15min":1.605},"last":0.95499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.68600000000000005,"15min":0.68200000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.38800000000000001},"max":{"1min":0.80700000000000005,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.76600000000000001,"15min":0.745},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89900000000000002,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60699999999999998,"15min":0.64500000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.77200000000000002,"5min":1.054,"15min":1.054},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65800000000000003,"15min":0.68600000000000005},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.35399999999999998},"max":{"1min":0.85299999999999998,"5min":1.0449999999999999,"15min":1.1220000000000001},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.437,"5min":0.38800000000000001,"15min":0.33800000000000002},"max":{"1min":0.86899999999999999,"5min":0.97099999999999997,"15min":1.143},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":0.83399999999999996,"5min":1.125,"15min":1.337},"last":0.85199999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72599999999999998,"15min":0.75900000000000001},"min":{"1min":0.53900000000000003,"5min":0.434,"15min":0.38200000000000001},"max":{"1min":0.91400000000000003,"5min":1.0449999999999999,"15min":1.1930000000000001},"last":0.622},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.76700000000000002},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.46400000000000002},"max":{"1min":0.92800000000000005,"5min":1.1519999999999999,"15min":1.2030000000000001},"last":0.65100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574955,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":1556480,"data_stored":680914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.69399999999999995,"15min":0.67000000000000004},"min":{"1min":0.45400000000000001,"5min":0.36299999999999999,"15min":0.32900000000000001},"max":{"1min":1.298,"5min":1.298,"15min":3.6629999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.74199999999999999,"15min":0.72799999999999998},"min":{"1min":0.47699999999999998,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":1.5680000000000001,"5min":1.5680000000000001,"15min":3.1709999999999998},"last":0.70099999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.69899999999999995,"15min":0.66300000000000003},"min":{"1min":0.441,"5min":0.40699999999999997,"15min":0.40000000000000002},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.70199999999999996,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.46,"5min":1.46,"15min":3.3079999999999998},"last":0.57799999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.68899999999999995,"15min":0.67100000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.38100000000000001},"max":{"1min":2.3359999999999999,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":1.004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.71199999999999997,"15min":0.69399999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96899999999999997,"5min":1.012,"15min":3.2629999999999999},"last":0.79700000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.75700000000000001,"15min":0.71499999999999997},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":3.6520000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.55000000000000004,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":3.6720000000000002,"5min":3.6720000000000002,"15min":3.742},"last":0.82299999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.63900000000000001,"15min":0.627},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.64700000000000002,"15min":0.65400000000000003},"min":{"1min":0.36699999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.98999999999999999,"5min":1.075,"15min":4.7750000000000004},"last":0.753}]},{"osd":5,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69199999999999995,"15min":0.71699999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":1.304},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65400000000000003,"15min":0.70599999999999996},"min":{"1min":0.42199999999999999,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.92100000000000004,"5min":1.127,"15min":3.609},"last":0.86099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:16:13 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.73099999999999998,"15min":0.751},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.601,"5min":1.601,"15min":4.9669999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.79100000000000004,"15min":0.80300000000000005},"min":{"1min":0.53300000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":1.238}]}]},{"osd":5,"up_from":13,"seq":55834574954,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50688,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94321152,"statfs":{"total":96636764160,"available":96584859648,"internally_reserved":0,"allocated":1560576,"data_stored":681448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.998,"15min":0.77600000000000002},"min":{"1min":0.41399999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.498,"5min":38.005000000000003,"15min":38.005000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.86199999999999999,"15min":0.75900000000000001},"min":{"1min":0.39800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.577,"5min":21.376999999999999,"15min":21.376999999999999},"last":0.41699999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.83099999999999996,"15min":0.73299999999999998},"min":{"1min":0.44500000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.4770000000000001,"5min":17.231000000000002,"15min":17.231000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.73899999999999999,"15min":0.70099999999999996},"min":{"1min":0.41999999999999998,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":1.5569999999999999,"5min":7.7370000000000001,"15min":7.7370000000000001},"last":0.71699999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.84399999999999997,"15min":0.75600000000000001},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.524,"5min":17.268000000000001,"15min":17.268000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.79800000000000004,"15min":0.754},"min":{"1min":0.47499999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.8680000000000001,"5min":9.6549999999999994,"15min":9.6549999999999994},"last":0.623}]},{"osd":3,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":1.8200000000000001,"15min":1.109},"min":{"1min":0.44,"5min":0.44,"15min":0.435},"max":{"1min":2.8690000000000002,"5min":113.363,"15min":113.363},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.79400000000000004,"5min":0.76900000000000002,"15min":0.76500000000000001},"min":{"1min":0.52800000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":2.359,"5min":4.1180000000000003,"15min":4.1180000000000003},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.63100000000000001,"15min":0.626},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.71399999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.827,"5min":2.827,"15min":2.827},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.70899999999999996,"15min":0.70499999999999996},"min":{"1min":0.41199999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.75,"15min":0.72499999999999998},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":2.9929999999999999,"5min":4.1500000000000004,"15min":4.1500000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.78200000000000003,"15min":0.75900000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":2.9319999999999999,"5min":4.1029999999999998,"15min":4.1029999999999998},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.77400000000000002,"15min":0.76500000000000001},"min":{"1min":0.46600000000000003,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.9379999999999999,"5min":2.02,"15min":2.02},"last":0.68600000000000005}]}]},{"osd":4,"up_from":13,"seq":55834574953,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54788,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317052,"statfs":{"total":96636764160,"available":96580661248,"internally_reserved":0,"allocated":1552384,"data_stored":680844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66800000000000004,"15min":0.65000000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93200000000000005,"5min":1.056,"15min":1.0620000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.72199999999999998,"15min":0.73499999999999999},"min":{"1min":0.40699999999999997,"5min":0.373,"15min":0.373},"max":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.341},"last":0.69399999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66600000000000004,"15min":0.66300000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.66400000000000003,"15min":0.63},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.85499999999999998,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.77900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.67300000000000004,"15min":0.65400000000000003},"min":{"1min":0.5,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.874,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.754},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72099999999999997},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.42199999999999999},"max":{"1min":1.22,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":0.63300000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70199999999999996,"15min":0.67300000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.006,"15min":1.006},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.72299999999999998,"15min":0.69999999999999996},"min":{"1min":0.47299999999999998,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.97499999999999998,"5min":1.232,"15min":1.232},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.63900000000000001,"15min":0.64000000000000001},"min":{"1min":0.40899999999999997,"5min":0.371,"15min":0.371},"max":{"1min":1.284,"5min":1.284,"15min":1.284},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.67200000000000004,"15min":0.65800000000000003},"min":{"1min":0.39700000000000002,"5min":0.375,"15min":0.30299999999999999},"max":{"1min":1.3069999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.61099999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.69299999999999995,"15min":0.66300000000000003},"min":{"1min":0.34999999999999998,"5min":0.32500000000000001,"15min":0.32200000000000001},"max":{"1min":1.0129999999999999,"5min":1.274,"15min":1.274},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.72799999999999998,"15min":0.73899999999999999},"min":{"1min":0.46600000000000003,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.254,"5min":1.292,"15min":1.413},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.73299999999999998,"15min":0.69999999999999996},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.39200000000000002},"max":{"1min":0.93799999999999994,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.76800000000000002,"15min":0.73499999999999999},"min":{"1min":0.44500000000000001,"5min":0.40500000000000003,"15min":0.371},"max":{"1min":1.0589999999999999,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.82099999999999995}]}]},{"osd":3,"up_from":13,"seq":55834574954,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76872,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94294968,"statfs":{"total":96636764160,"available":96558047232,"internally_reserved":0,"allocated":1568768,"data_stored":682478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77134279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42299999999999999,"15min":0.44900000000000001},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.55900000000000005,"5min":1.177,"15min":1.9810000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.45000000000000001,"15min":0.53800000000000003},"min":{"1min":0.32200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.68300000000000005,"5min":0.92400000000000004,"15min":2.96},"last":0.46400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48799999999999999,"15min":0.51700000000000002},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.81499999999999995,"5min":0.95699999999999996,"15min":3.5600000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.55200000000000005,"15min":0.68400000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.746,"5min":1.0309999999999999,"15min":3.6829999999999998},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85999999999999999,"5min":0.88,"15min":1.9590000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53600000000000003,"15min":0.58599999999999997},"min":{"1min":0.34799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70399999999999996,"5min":0.97699999999999998,"15min":1.3819999999999999},"last":0.73099999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72799999999999998,"15min":0.79200000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.96899999999999997,"5min":1.1339999999999999,"15min":2.8610000000000002},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.752,"15min":0.77100000000000002},"min":{"1min":0.52000000000000002,"5min":0.505,"15min":0.48199999999999998},"max":{"1min":1.012,"5min":1.123,"15min":1.3640000000000001},"last":0.85899999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.755,"15min":0.78200000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.2529999999999999,"15min":1.391},"last":1.032},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72999999999999998,"15min":0.81000000000000005},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.98899999999999999,"5min":1.173,"15min":2.9300000000000002},"last":0.92000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.752,"15min":0.76500000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.95999999999999996,"5min":1.1579999999999999,"15min":1.401},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.76200000000000001,"15min":0.78900000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0589999999999999,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":1.0580000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:16:12 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.75900000000000001,"15min":0.83799999999999997},"min":{"1min":0.50900000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.022,"5min":1.139,"15min":2.899},"last":0.878},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.77300000000000002,"15min":0.81699999999999995},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0449999999999999,"5min":1.1220000000000001,"15min":1.4359999999999999},"last":0.93799999999999994}]}]},{"osd":2,"up_from":13,"seq":55834574954,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54780,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94317060,"statfs":{"total":96636764160,"available":96580669440,"internally_reserved":0,"allocated":1548288,"data_stored":679825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.28499999999999998,"5min":0.25900000000000001,"15min":0.25700000000000001},"max":{"1min":0.63900000000000001,"5min":1.097,"15min":1.097},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.81499999999999995,"15min":0.55800000000000005},"min":{"1min":0.30099999999999999,"5min":0.28499999999999998,"15min":0.28100000000000003},"max":{"1min":0.70199999999999996,"5min":30.023,"15min":30.023},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48499999999999999,"15min":1.345},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":1.046,"15min":260.53399999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.745,"15min":1.4450000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.86199999999999999,"5min":21.279,"15min":262.61399999999998},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.82099999999999995,"15min":0.59599999999999997},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80300000000000005,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.875,"5min":1.214,"15min":1.214},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.89100000000000001,"15min":0.80100000000000005},"min":{"1min":0.52900000000000003,"5min":0.495,"15min":0.49099999999999999},"max":{"1min":1.008,"5min":10.598000000000001,"15min":10.598000000000001},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":1.016,"15min":0.85899999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":1.0529999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.006}]},{"osd":5,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":1.0900000000000001,"15min":0.88900000000000001},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":0.97599999999999998,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.98999999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.999,"5min":1.2789999999999999,"15min":1.569},"last":0.89500000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.89600000000000002,"15min":0.80700000000000005},"min":{"1min":0.495,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.98799999999999999,"5min":10.664,"15min":10.664},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":1.0920000000000001,"15min":0.90300000000000002},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.083,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.93899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:16:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.90900000000000003,"15min":0.82199999999999995},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.1040000000000001,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":1.1200000000000001,"15min":0.90700000000000003},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.46100000000000002},"max":{"1min":1.1379999999999999,"5min":27.199999999999999,"15min":27.199999999999999},"last":1.05}]}]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82624,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94289216,"statfs":{"total":96636764160,"available":96552157184,"internally_reserved":0,"allocated":1560576,"data_stored":680968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.48599999999999999,"15min":0.49299999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":0.59399999999999997,"5min":5.4939999999999998,"15min":5.4939999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.375,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73399999999999999,"5min":6.7240000000000002,"15min":6.7240000000000002},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.46600000000000003,"15min":0.46999999999999997},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68300000000000005,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.70299999999999996,"5min":1.5089999999999999,"15min":1.5089999999999999},"last":0.60699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.72999999999999998,"15min":0.68400000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":16.382999999999999,"15min":16.382999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.67700000000000005,"15min":0.61099999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.79300000000000004,"5min":18.318000000000001,"15min":18.318000000000001},"last":0.76400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.79900000000000004,"15min":0.80500000000000005},"min":{"1min":0.45500000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.998,"5min":6.1630000000000003,"15min":6.1630000000000003},"last":1.159},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78300000000000003,"15min":0.81200000000000006},"min":{"1min":0.55600000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":0.97499999999999998,"5min":1.429,"15min":1.78},"last":1.131}]},{"osd":5,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.751,"15min":0.76800000000000002},"min":{"1min":0.52800000000000002,"5min":0.502,"15min":0.49199999999999999},"max":{"1min":0.92100000000000004,"5min":1.6020000000000001,"15min":1.6870000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.79000000000000004,"15min":0.79500000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.997,"5min":6.1189999999999998,"15min":6.1189999999999998},"last":1.073}]},{"osd":6,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.755,"15min":0.79900000000000004},"min":{"1min":0.57499999999999996,"5min":0.55900000000000005,"15min":0.53200000000000003},"max":{"1min":0.91100000000000003,"5min":1.6599999999999999,"15min":1.7310000000000001},"last":1.1850000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.80800000000000005,"15min":0.81699999999999995},"min":{"1min":0.55300000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.94899999999999995,"5min":6.1970000000000001,"15min":6.1970000000000001},"last":1.204}]},{"osd":7,"last update":"Fri Sep 13 17:16:07 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76300000000000001,"15min":0.81299999999999994},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.92600000000000005,"5min":1.472,"15min":1.8320000000000001},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.79100000000000004,"15min":0.81999999999999995},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.0089999999999999,"5min":1.544,"15min":1.6359999999999999},"last":1.145}]}]},{"osd":0,"up_from":13,"seq":55834574955,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59476,"kb_used_data":2112,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312364,"statfs":{"total":96636764160,"available":96575860736,"internally_reserved":0,"allocated":2162688,"data_stored":1271869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.45300000000000001,"15min":0.63500000000000001},"min":{"1min":0.27700000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.018,"5min":1.018,"15min":5.5259999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.52300000000000002,"15min":0.64600000000000002},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.27000000000000002},"max":{"1min":2.738,"5min":3.004,"15min":5.4889999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.54400000000000004,"15min":0.67900000000000005},"min":{"1min":0.34799999999999998,"5min":0.313,"15min":0.313},"max":{"1min":3.605,"5min":3.605,"15min":6.0430000000000001},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.52200000000000002,"15min":0.68300000000000005},"min":{"1min":0.41099999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.29,"5min":1.29,"15min":4.6760000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.52100000000000002,"15min":0.63900000000000001},"min":{"1min":0.378,"5min":0.314,"15min":0.314},"max":{"1min":0.98999999999999999,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.59999999999999998,"15min":0.69799999999999995},"min":{"1min":0.435,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.85199999999999998,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.59099999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.78700000000000003,"15min":0.878},"min":{"1min":0.56200000000000006,"5min":0.443,"15min":0.443},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":4.734},"last":0.751},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.78300000000000003,"15min":0.88800000000000001},"min":{"1min":0.58499999999999996,"5min":0.442,"15min":0.442},"max":{"1min":1.079,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.69199999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.79200000000000004,"15min":0.90400000000000003},"min":{"1min":0.60799999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.2190000000000001,"5min":2.7320000000000002,"15min":4.944},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.80100000000000005,"15min":0.89300000000000002},"min":{"1min":0.622,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.8740000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.78300000000000003,"15min":1.0109999999999999},"min":{"1min":0.56599999999999995,"5min":0.54500000000000004,"15min":0.54200000000000004},"max":{"1min":1.0680000000000001,"5min":1.194,"15min":4.8470000000000004},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.79600000000000004,"15min":0.91200000000000003},"min":{"1min":0.70799999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.627}]},{"osd":7,"last update":"Fri Sep 13 17:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.80500000000000005,"15min":0.93999999999999995},"min":{"1min":0.60599999999999998,"5min":0.50600000000000001,"15min":0.47399999999999998},"max":{"1min":1.405,"5min":1.405,"15min":5.0540000000000003},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.81899999999999995,"15min":0.92900000000000005},"min":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.505},"max":{"1min":1.3149999999999999,"5min":2.706,"15min":4.9950000000000001},"last":0.81200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:16:34.303 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:16:34.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:16:34.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.463+0000 7fda23200640 1 Processor -- start 2024-09-13T17:16:34.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.464+0000 7fda23200640 1 -- start start 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda23200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda24143320 0x7fda24070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda23200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24071630 con 0x7fda24143910 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda23200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda24143700 con 0x7fda240cd120 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda23200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda24070a20 con 0x7fda24143320 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda21e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda24143320 0x7fda24070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda21e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda24143320 0x7fda24070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44954/0 (socket says 172.21.15.28:44954) 2024-09-13T17:16:34.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.465+0000 7fda21e00640 1 -- 172.21.15.28:0/3465547333 learned_addr learned my addr 172.21.15.28:0/3465547333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.466+0000 7fda21e00640 1 -- 172.21.15.28:0/3465547333 >> v1:172.21.15.189:6789/0 conn(0x7fda240cd120 legacy=0x7fda2415ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.466+0000 7fda21400640 1 -- 172.21.15.28:0/3465547333 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282536927 0 0) 0x7fda24071630 con 0x7fda24143910 2024-09-13T17:16:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.466+0000 7fda21e00640 1 -- 172.21.15.28:0/3465547333 >> v1:172.21.15.28:6789/0 conn(0x7fda24143910 legacy=0x7fda24071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.466+0000 7fda21e00640 1 -- 172.21.15.28:0/3465547333 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda24070d00 con 0x7fda24143320 2024-09-13T17:16:34.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.467+0000 7fda21e00640 1 --2- 172.21.15.28:0/3465547333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda24070440 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fda14009870 tx=0x7fda1402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d20b99bea83723 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.467+0000 7fda21400640 1 -- 172.21.15.28:0/3465547333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14032640 con 0x7fda24143320 2024-09-13T17:16:34.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.467+0000 7fda21400640 1 -- 172.21.15.28:0/3465547333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda140327a0 con 0x7fda24143320 2024-09-13T17:16:34.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.467+0000 7fda21400640 1 -- 172.21.15.28:0/3465547333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda14032b30 con 0x7fda24143320 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 -- 172.21.15.28:0/3465547333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 msgr2=0x7fda24070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 --2- 172.21.15.28:0/3465547333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda24070440 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fda14009870 tx=0x7fda1402a300 comp rx=0 tx=0).stop 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 -- 172.21.15.28:0/3465547333 shutdown_connections 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 --2- 172.21.15.28:0/3465547333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda24070440 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 -- 172.21.15.28:0/3465547333 >> 172.21.15.28:0/3465547333 conn(0x7fda24069e80 msgr2=0x7fda2406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 -- 172.21.15.28:0/3465547333 shutdown_connections 2024-09-13T17:16:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.468+0000 7fda23200640 1 -- 172.21.15.28:0/3465547333 wait complete. 2024-09-13T17:16:34.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.469+0000 7fda23200640 1 Processor -- start 2024-09-13T17:16:34.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.469+0000 7fda23200640 1 -- start start 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda240cd120 0x7fda24139e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda24143910 0x7fda2413caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2414a990 con 0x7fda24143320 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2414ac70 con 0x7fda24143910 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.470+0000 7fda23200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda2414af50 con 0x7fda240cd120 2024-09-13T17:16:34.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44968/0 (socket says 172.21.15.28:44968) 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 -- 172.21.15.28:0/3339613240 learned_addr learned my addr 172.21.15.28:0/3339613240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda21e00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda240cd120 0x7fda24139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda22800640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda24143910 0x7fda2413caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 -- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda240cd120 msgr2=0x7fda24139e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda240cd120 0x7fda24139e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 -- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda24143910 msgr2=0x7fda2413caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda24143910 0x7fda2413caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 -- 172.21.15.28:0/3339613240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda140094d0 con 0x7fda24143320 2024-09-13T17:16:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.471+0000 7fda1be00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c460 tx=0x7fda0c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.472+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c005480 con 0x7fda24143320 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.472+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda0c0035f0 con 0x7fda24143320 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.472+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0c0039a0 con 0x7fda24143320 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.472+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2414b230 con 0x7fda24143320 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.472+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2414a5d0 con 0x7fda24143320 2024-09-13T17:16:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.473+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda240b3800 con 0x7fda24143320 2024-09-13T17:16:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.474+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fda0c003b00 con 0x7fda24143320 2024-09-13T17:16:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.475+0000 7fda1a000640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 0x7fda0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.475+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda0c088e50 con 0x7fda24143320 2024-09-13T17:16:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.475+0000 7fda21e00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 0x7fda0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.476+0000 7fda21e00640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 0x7fda0006d0c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fda14034000 tx=0x7fda1402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.479+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fda0c054040 con 0x7fda24143320 2024-09-13T17:16:34.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.727+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fda24070440 con 0x7fda24143320 2024-09-13T17:16:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.732+0000 7fda1a000640 1 -- 172.21.15.28:0/3339613240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v20) v1 ==== 190+0+0 (secure 0 0 0) 0x7fda0c057660 con 0x7fda24143320 2024-09-13T17:16:34.733 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:16:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 msgr2=0x7fda0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 0x7fda0006d0c0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fda14034000 tx=0x7fda1402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 msgr2=0x7fda240c6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00c460 tx=0x7fda0c00c920 comp rx=0 tx=0).stop 2024-09-13T17:16:34.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 shutdown_connections 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda0006ac00 0x7fda0006d0c0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda24143910 0x7fda2413caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda24143320 0x7fda240c6020 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 --2- 172.21.15.28:0/3339613240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda240cd120 0x7fda24139e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 >> 172.21.15.28:0/3339613240 conn(0x7fda24069e80 msgr2=0x7fda240b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 shutdown_connections 2024-09-13T17:16:34.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.734+0000 7fda23200640 1 -- 172.21.15.28:0/3339613240 wait complete. 2024-09-13T17:16:34.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:16:34.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.906+0000 7f3217600640 1 Processor -- start 2024-09-13T17:16:34.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.906+0000 7f3217600640 1 -- start start 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3217600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f321006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3217600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3210070cc0 0x7f3210071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3217600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321015e100 con 0x7f32100cabc0 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3217600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32100716b0 con 0x7f3210070cc0 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3217600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32100adb00 con 0x7f321015a540 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3207e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32100cabc0 0x7f321015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52960/0 (socket says 172.21.15.28:52960) 2024-09-13T17:16:34.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3207e00640 1 -- 172.21.15.28:0/1656433749 learned_addr learned my addr 172.21.15.28:0/1656433749 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3215600640 1 --2- 172.21.15.28:0/1656433749 >> v2:172.21.15.189:3300/0 conn(0x7f3210070cc0 0x7f3210071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f321006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.908+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385577516 0 0) 0x7f321015e100 con 0x7f32100cabc0 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31f4003520 con 0x7f32100cabc0 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4065617043 0 0) 0x7f31f4003520 con 0x7f32100cabc0 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 msgr2=0x7f321006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f321006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 >> v2:172.21.15.189:3300/0 conn(0x7f3210070cc0 msgr2=0x7f3210071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 --2- 172.21.15.28:0/1656433749 >> v2:172.21.15.189:3300/0 conn(0x7f3210070cc0 0x7f3210071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321015e3e0 con 0x7f32100cabc0 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f31f8001670 con 0x7f32100cabc0 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f321006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 >> v1:172.21.15.28:6789/0 conn(0x7f32100cabc0 legacy=0x7f321015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31f40039a0 0x7f31f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f31f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f4004d00 0x7f31f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32100716b0 con 0x7f31f40039a0 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31f400b980 con 0x7f31f4004d00 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31f400baf0 con 0x7f321015a540 2024-09-13T17:16:34.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f3215600640 1 --2- 172.21.15.28:0/1656433749 >> v2:172.21.15.189:3300/0 conn(0x7f3210070cc0 0x7f3210071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31f40039a0 0x7f31f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.909+0000 7f3207e00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f31f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3215600640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f4004d00 0x7f31f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3214c00640 1 -- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 msgr2=0x7f31f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321015a540 0x7f31f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3214c00640 1 -- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f4004d00 msgr2=0x7f31f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f4004d00 0x7f31f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3214c00640 1 -- 172.21.15.28:0/1656433749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3200009400 con 0x7f31f40039a0 2024-09-13T17:16:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.910+0000 7f3215600640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f4004d00 0x7f31f4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f3214c00640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31f40039a0 0x7f31f4003dc0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3200009c90 tx=0x7f31f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c032821de7320aa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200031400 con 0x7f31f40039a0 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320002d050 con 0x7f31f40039a0 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f320fe00640 1 -- 172.21.15.28:0/1656433749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200031760 con 0x7f31f40039a0 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f3217600640 1 -- 172.21.15.28:0/1656433749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31f40039a0 msgr2=0x7f31f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f3217600640 1 --2- 172.21.15.28:0/1656433749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31f40039a0 0x7f31f4003dc0 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f3200009c90 tx=0x7f31f40057f0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.911+0000 7f3215600640 1 -- 172.21.15.28:0/1656433749 reap_dead start 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.912+0000 7f3217600640 1 -- 172.21.15.28:0/1656433749 shutdown_connections 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.912+0000 7f3217600640 1 -- 172.21.15.28:0/1656433749 >> 172.21.15.28:0/1656433749 conn(0x7f3210069e80 msgr2=0x7f321006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.912+0000 7f3217600640 1 -- 172.21.15.28:0/1656433749 shutdown_connections 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.912+0000 7f3217600640 1 -- 172.21.15.28:0/1656433749 wait complete. 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.913+0000 7f3217600640 1 Processor -- start 2024-09-13T17:16:34.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.913+0000 7f3217600640 1 -- start start 2024-09-13T17:16:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f40039a0 0x7f32101f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321014f320 0x7f321014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321014a1f0 con 0x7f32101543c0 2024-09-13T17:16:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32101503a0 con 0x7f31f40039a0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.914+0000 7f3217600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32101f6b30 con 0x7f321014f320 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44986/0 (socket says 172.21.15.28:44986) 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 -- 172.21.15.28:0/732411821 learned_addr learned my addr 172.21.15.28:0/732411821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3214c00640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f40039a0 0x7f32101f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3207e00640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321014f320 0x7f321014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 -- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321014f320 msgr2=0x7f321014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321014f320 0x7f321014e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 -- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f40039a0 msgr2=0x7f32101f65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f40039a0 0x7f32101f65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 -- 172.21.15.28:0/732411821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210154d90 con 0x7f32101543c0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.915+0000 7f3215600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f32100ad6f0 tx=0x7f320800f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.916+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208017400 con 0x7f32101543c0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.916+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3208011040 con 0x7f32101543c0 2024-09-13T17:16:34.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.916+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208017780 con 0x7f32101543c0 2024-09-13T17:16:34.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.916+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f321014f820 con 0x7f32101543c0 2024-09-13T17:16:34.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.916+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f321014fde0 con 0x7f32101543c0 2024-09-13T17:16:34.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.917+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3210071170 con 0x7f32101543c0 2024-09-13T17:16:34.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.918+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f32080203b0 con 0x7f32101543c0 2024-09-13T17:16:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.919+0000 7f320d600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 0x7f31ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.919+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3208089520 con 0x7f32101543c0 2024-09-13T17:16:34.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.919+0000 7f3214c00640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 0x7f31ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:34.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.920+0000 7f3214c00640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 0x7f31ec06d0f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3200033000 tx=0x7f3200008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:34.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:34.923+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3208054710 con 0x7f32101543c0 2024-09-13T17:16:35.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.139+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3210159d60 con 0x7f32101543c0 2024-09-13T17:16:35.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.140+0000 7f320d600640 1 -- 172.21.15.28:0/732411821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7f3208057d30 con 0x7f32101543c0 2024-09-13T17:16:35.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:16:35.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:16:35.143 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:16:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 msgr2=0x7f31ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 0x7f31ec06d0f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3200033000 tx=0x7f3200008030 comp rx=0 tx=0).stop 2024-09-13T17:16:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 msgr2=0x7f3210154850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f32100ad6f0 tx=0x7f320800f790 comp rx=0 tx=0).stop 2024-09-13T17:16:35.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 shutdown_connections 2024-09-13T17:16:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31ec06ac30 0x7f31ec06d0f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32101543c0 0x7f3210154850 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f321014f320 0x7f321014e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 --2- 172.21.15.28:0/732411821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31f40039a0 0x7f32101f65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 >> 172.21.15.28:0/732411821 conn(0x7f3210069e80 msgr2=0x7f32100ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 shutdown_connections 2024-09-13T17:16:35.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.142+0000 7f3217600640 1 -- 172.21.15.28:0/732411821 wait complete. 2024-09-13T17:16:35.165 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:16:35.164+0000 7f8349a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:16:35.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 114, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:16:29.636609+0000', 'modified': '2024-09-13T17:16:32.652735+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7939}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7939': {'gid': 7939, 'name': 'c', 'rank': 0, 'incarnation': 110, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/199464366', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 199464366}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 199464366}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:16:35.562 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:16:35.562 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:16:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.721+0000 7f0e66200640 1 Processor -- start 2024-09-13T17:16:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.721+0000 7f0e66200640 1 -- start start 2024-09-13T17:16:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.722+0000 7f0e66200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e600c40a0 0x7f0e600b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.722+0000 7f0e66200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 0x7f0e6015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.722+0000 7f0e66200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e600b2090 con 0x7f0e6015b0b0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e66200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e600b2370 con 0x7f0e600c40a0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e66200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e600b2650 con 0x7f0e600b1ba0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e65800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e6015b0b0 0x7f0e6006c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52974/0 (socket says 172.21.15.28:52974) 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e65800640 1 -- 172.21.15.28:0/1247773554 learned_addr learned my addr 172.21.15.28:0/1247773554 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1247773554 >> v2:172.21.15.189:3300/0 conn(0x7f0e600c40a0 0x7f0e600b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 0x7f0e6015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541093435 0 0) 0x7f0e600b2090 con 0x7f0e6015b0b0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.723+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e48003520 con 0x7f0e6015b0b0 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e64e00640 1 -- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 msgr2=0x7f0e6015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 0x7f0e6015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e64e00640 1 -- 172.21.15.28:0/1247773554 >> v1:172.21.15.28:6789/0 conn(0x7f0e6015b0b0 legacy=0x7f0e6006c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790154912 0 0) 0x7f0e48003520 con 0x7f0e6015b0b0 2024-09-13T17:16:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e64e00640 1 -- 172.21.15.28:0/1247773554 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e600b2930 con 0x7f0e600c40a0 2024-09-13T17:16:35.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.724+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 0x7f0e6015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600c40a0 0x7f0e600b1660 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0e540097a0 tx=0x7f0e5402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=206a60a74e3609d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e580019f0 con 0x7f0e600c40a0 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5402aba0 con 0x7f0e600c40a0 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e5f400640 1 -- 172.21.15.28:0/1247773554 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54032440 con 0x7f0e600c40a0 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e66200640 1 -- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600c40a0 msgr2=0x7f0e600b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.725+0000 7f0e66200640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600c40a0 0x7f0e600b1660 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0e540097a0 tx=0x7f0e5402a110 comp rx=0 tx=0).stop 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 -- 172.21.15.28:0/1247773554 shutdown_connections 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e600b1ba0 0x7f0e6015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 --2- 172.21.15.28:0/1247773554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600c40a0 0x7f0e600b1660 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 -- 172.21.15.28:0/1247773554 >> 172.21.15.28:0/1247773554 conn(0x7f0e60073470 msgr2=0x7f0e60069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 -- 172.21.15.28:0/1247773554 shutdown_connections 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.726+0000 7f0e66200640 1 -- 172.21.15.28:0/1247773554 wait complete. 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.727+0000 7f0e66200640 1 Processor -- start 2024-09-13T17:16:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.727+0000 7f0e66200640 1 -- start start 2024-09-13T17:16:35.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 0x7f0e60140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e6015b0b0 0x7f0e600aef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e600ad1a0 con 0x7f0e600c40a0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e601506b0 con 0x7f0e600b1ba0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.728+0000 7f0e66200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e60140a00 con 0x7f0e6015b0b0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44988/0 (socket says 172.21.15.28:44988) 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1542187854 learned_addr learned my addr 172.21.15.28:0/1542187854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 0x7f0e60140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e65800640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e6015b0b0 0x7f0e600aef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e6015b0b0 msgr2=0x7f0e600aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e6015b0b0 0x7f0e600aef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 msgr2=0x7f0e60140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 0x7f0e60140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 -- 172.21.15.28:0/1542187854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c009400 con 0x7f0e600c40a0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 0x7f0e60140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e9b0 tx=0x7f0e4c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5cc00640 1 -- 172.21.15.28:0/1542187854 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c00ca00 con 0x7f0e600c40a0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e5cc00640 1 -- 172.21.15.28:0/1542187854 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c00cb60 con 0x7f0e600c40a0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.729+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e54009400 con 0x7f0e600c40a0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.730+0000 7f0e5cc00640 1 -- 172.21.15.28:0/1542187854 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c013470 con 0x7f0e600c40a0 2024-09-13T17:16:35.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.730+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e60140fc0 con 0x7f0e600c40a0 2024-09-13T17:16:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.733+0000 7f0e5cc00640 1 -- 172.21.15.28:0/1542187854 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e4c0135d0 con 0x7f0e600c40a0 2024-09-13T17:16:35.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.734+0000 7f0e5cc00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 0x7f0e4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.734+0000 7f0e5cc00640 1 -- 172.21.15.28:0/1542187854 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e4c0892e0 con 0x7f0e600c40a0 2024-09-13T17:16:35.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.734+0000 7f0e6a61e740 1 Processor -- start 2024-09-13T17:16:35.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.734+0000 7f0e6a61e740 1 -- start start 2024-09-13T17:16:35.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.734+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 0x7f0e4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d641000 0x555a6d690d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d691250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d5acab0 con 0x555a6d643840 2024-09-13T17:16:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d5aa240 con 0x555a6d641000 2024-09-13T17:16:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6d5a5b30 con 0x555a6d66acf0 2024-09-13T17:16:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e64e00640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 0x7f0e4006d0c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0e54034000 tx=0x7f0e5402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.735+0000 7f0e64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45004/0 (socket says 172.21.15.28:45004) 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 -- 172.21.15.28:0/2229304581 learned_addr learned my addr 172.21.15.28:0/2229304581 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e65800640 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d691250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d641000 0x555a6d690d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 -- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 msgr2=0x555a6d691250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d691250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 -- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d641000 msgr2=0x555a6d690d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d641000 0x555a6d690d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 -- 172.21.15.28:0/2229304581 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a6d570bd0 con 0x555a6d643840 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e64e00640 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402a150 tx=0x7f0e5402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47394c28e8bf222 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e51600640 1 -- 172.21.15.28:0/2229304581 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54032440 con 0x555a6d643840 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e51600640 1 -- 172.21.15.28:0/2229304581 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54043040 con 0x555a6d643840 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.736+0000 7f0e51600640 1 -- 172.21.15.28:0/2229304581 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5403c870 con 0x555a6d643840 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 -- 172.21.15.28:0/2229304581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 msgr2=0x555a6d643c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f0e5402a150 tx=0x7f0e5402c000 comp rx=0 tx=0).stop 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 -- 172.21.15.28:0/2229304581 shutdown_connections 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d691250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d641000 0x555a6d690d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/2229304581 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d643c20 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 -- 172.21.15.28:0/2229304581 >> 172.21.15.28:0/2229304581 conn(0x555a6d63ff70 msgr2=0x555a6d688420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 -- 172.21.15.28:0/2229304581 shutdown_connections 2024-09-13T17:16:35.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.737+0000 7f0e6a61e740 1 -- 172.21.15.28:0/2229304581 wait complete. 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.738+0000 7f0e6a61e740 1 -- start start 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.738+0000 7f0e6a61e740 1 Processor -- start 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d68f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 0x555a6d669be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d547260 con 0x555a6d643840 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d546f00 con 0x555a6d66a490 2024-09-13T17:16:35.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6d30cf20 con 0x555a6d66acf0 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e6a61e740 10 client.0 _ll_register_callbacks cb 0x555a6d59f1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e64e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45014/0 (socket says 172.21.15.28:45014) 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.739+0000 7f0e64e00640 1 -- 172.21.15.28:0/3476548480 learned_addr learned my addr 172.21.15.28:0/3476548480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e65800640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 0x555a6d669be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d68f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 -- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 msgr2=0x555a6d68f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d68f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 -- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 msgr2=0x555a6d669be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 0x555a6d669be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 -- 172.21.15.28:0/3476548480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e600b2930 con 0x555a6d643840 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e65800640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 0x555a6d669be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:35.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000c00 tx=0x7f0e54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42fc7ece9218c0f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e46000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e54052440 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e540525a0 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.740+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a6d5cfcd0 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e54052760 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 10 client.8601 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x555a6d5a41b0 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555a6d5cfcd0 con 0x555a6d643840 2024-09-13T17:16:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0e54046070 con 0x555a6d643840 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 10 client.8601 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 10 client.8601 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 20 client.8601 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d698360 0x555a6d66c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 4 client.8601 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a6d2993c0 con 0x555a6d698360 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.741+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d698360 0x555a6d66c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.742+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d698360 0x555a6d66c240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=795d0348ad9312d6 server_cookie=e11a076563528b3a in_seq=0 out_seq=0 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.742+0000 7f0e46000640 10 client.8601 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e46000640 1 -- 172.21.15.28:0/3476548480 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x555a6d2993c0 con 0x555a6d698360 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e46000640 10 client.8601 handle_command_reply: tid=0 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 client.8601 shutdown 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 msgr2=0x555a6d6906a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f0e54000c00 tx=0x7f0e54004730 comp rx=0 tx=0).stop 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 shutdown_connections 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d698360 0x555a6d66c240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66a490 0x555a6d669be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d68f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3476548480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d643840 0x555a6d6906a0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e46000640 0 client.8601 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 >> 172.21.15.28:0/3476548480 conn(0x555a6d641000 msgr2=0x555a6d68c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 shutdown_connections 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3476548480 wait complete. 2024-09-13T17:16:35.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.743+0000 7f0e6a61e740 20 client.8601 trim_cache size 0 max 16384 2024-09-13T17:16:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.759+0000 7f0e6a61e740 1 Processor -- start 2024-09-13T17:16:35.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.759+0000 7f0e6a61e740 1 -- start start 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 0x555a6d669760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d6ebe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d642e40 con 0x555a6d66a490 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d643fe0 con 0x555a6d643840 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6d68e5c0 con 0x555a6d66acf0 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45020/0 (socket says 172.21.15.28:45020) 2024-09-13T17:16:35.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3628614674 learned_addr learned my addr 172.21.15.28:0/3628614674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 0x555a6d669760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.760+0000 7f0e65800640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d6ebe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 msgr2=0x555a6d6ebe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d6ebe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 msgr2=0x555a6d669760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 0x555a6d669760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3628614674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a6d5d9170 con 0x555a6d66a490 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 0x555a6d669760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c087800 tx=0x7f0e4c05e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d0b67d5bb8b9df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e52000640 1 -- 172.21.15.28:0/3628614674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0562b0 con 0x555a6d66a490 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e52000640 1 -- 172.21.15.28:0/3628614674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c087880 con 0x555a6d66a490 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.761+0000 7f0e52000640 1 -- 172.21.15.28:0/3628614674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c054880 con 0x555a6d66a490 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3628614674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 msgr2=0x555a6d68e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c087800 tx=0x7f0e4c05e040 comp rx=0 tx=0).stop 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3628614674 shutdown_connections 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d66acf0 0x555a6d6ebe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d68e080 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3628614674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d643840 0x555a6d669760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3628614674 >> 172.21.15.28:0/3628614674 conn(0x555a6d641000 msgr2=0x555a6d66d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3628614674 shutdown_connections 2024-09-13T17:16:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3628614674 wait complete. 2024-09-13T17:16:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.762+0000 7f0e6a61e740 1 -- start start 2024-09-13T17:16:35.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.763+0000 7f0e6a61e740 1 Processor -- start 2024-09-13T17:16:35.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.763+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d643840 0x555a6d6e8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.763+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.763+0000 7f0e6a61e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66acf0 0x555a6d6d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.763+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d6e87e0 con 0x555a6d66a490 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a6d6d3df0 con 0x555a6d66acf0 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e6a61e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555a6d6e7910 con 0x555a6d643840 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e6a61e740 10 client.0 _ll_register_callbacks cb 0x7f0e4c00cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45030/0 (socket says 172.21.15.28:45030) 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3870048127 learned_addr learned my addr 172.21.15.28:0/3870048127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d643840 0x555a6d6e8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e65800640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66acf0 0x555a6d6d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d643840 msgr2=0x555a6d6e8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d643840 0x555a6d6e8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66acf0 msgr2=0x555a6d6d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66acf0 0x555a6d6d3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.764+0000 7f0e5fe00640 1 -- 172.21.15.28:0/3870048127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555a6d5a41b0 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e5fe00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e980 tx=0x7f0e5403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa9e3735682ed6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c056370 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0e4c054880 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e54043040 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e4c054a40 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 10 client.8607 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x7f0e60140fc0 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e54043040 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0e4c011070 con 0x555a6d66a490 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 10 client.8607 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 10 client.8607 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 20 client.8607 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d6878a0 0x555a6d687c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 4 client.8607 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.765+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555a6d2aa6c0 con 0x555a6d6878a0 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d6878a0 0x555a6d687c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e46000640 10 client.8607 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e64e00640 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d6878a0 0x555a6d687c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fee63a0d7e5ea99 server_cookie=3343e0058d4249f9 in_seq=0 out_seq=0 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e46000640 1 -- 172.21.15.28:0/3870048127 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x555a6d2aa6c0 con 0x555a6d6878a0 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e46000640 10 client.8607 handle_command_reply: tid=0 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.766+0000 7f0e6a61e740 1 client.8607 shutdown 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 msgr2=0x555a6d6d2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e980 tx=0x7f0e5403c000 comp rx=0 tx=0).stop 2024-09-13T17:16:35.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 shutdown_connections 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x555a6d6878a0 0x555a6d687c80 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555a6d66acf0 0x555a6d6d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a6d66a490 0x555a6d6d2630 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 --2- 172.21.15.28:0/3870048127 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555a6d643840 0x555a6d6e8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e46000640 0 client.8607 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 >> 172.21.15.28:0/3870048127 conn(0x555a6d641000 msgr2=0x555a6d68cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 shutdown_connections 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 1 -- 172.21.15.28:0/3870048127 wait complete. 2024-09-13T17:16:35.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e6a61e740 20 client.8607 trim_cache size 0 max 16384 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 7939, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1275947039999998, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 114, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 125, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 125, 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.236167799 2024-09-13T17:16:35.774 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 msgr2=0x7f0e4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 0x7f0e4006d0c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0e54034000 tx=0x7f0e5402e000 comp rx=0 tx=0).stop 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 msgr2=0x7f0e600aea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.767+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00e9b0 tx=0x7f0e4c00ee70 comp rx=0 tx=0).stop 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 shutdown_connections 2024-09-13T17:16:35.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e4006ac00 0x7f0e4006d0c0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e6015b0b0 0x7f0e600aef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e600c40a0 0x7f0e600aea10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 --2- 172.21.15.28:0/1542187854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e600b1ba0 0x7f0e60140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 >> 172.21.15.28:0/1542187854 conn(0x7f0e60073470 msgr2=0x7f0e6015ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 shutdown_connections 2024-09-13T17:16:35.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.768+0000 7f0e66200640 1 -- 172.21.15.28:0/1542187854 wait complete. 2024-09-13T17:16:35.786 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:16:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.939+0000 7ff3b0e00640 1 Processor -- start 2024-09-13T17:16:35.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.939+0000 7ff3b0e00640 1 -- start start 2024-09-13T17:16:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3b0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3ac1592a0 0x7ff3ac15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3b0e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06c2d0 0x7ff3ac06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3b0e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3ac06cc70 con 0x7ff3ac06f970 2024-09-13T17:16:35.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3b0e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3ac15b0b0 con 0x7ff3ac1592a0 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3b0e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac159d90 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3ac1592a0 0x7ff3ac15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3a1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3ac06f970 0x7ff3ac159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52984/0 (socket says 172.21.15.28:52984) 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.940+0000 7ff3a1400640 1 -- 172.21.15.28:0/2611688095 learned_addr learned my addr 172.21.15.28:0/2611688095 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3aa800640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06c2d0 0x7ff3ac06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3a9e00640 1 -- 172.21.15.28:0/2611688095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06c2d0 msgr2=0x7ff3ac06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:35.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3a9e00640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06c2d0 0x7ff3ac06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3a9e00640 1 -- 172.21.15.28:0/2611688095 >> v1:172.21.15.28:6789/0 conn(0x7ff3ac06f970 legacy=0x7ff3ac159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3a9e00640 1 -- 172.21.15.28:0/2611688095 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac0b2040 con 0x7ff3ac1592a0 2024-09-13T17:16:35.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.941+0000 7ff3a9400640 1 -- 172.21.15.28:0/2611688095 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862997119 0 0) 0x7ff3ac06cc70 con 0x7ff3ac06f970 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.942+0000 7ff3a9e00640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15ab70 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0097a0 tx=0x7ff39c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ecf86bb19f04c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.942+0000 7ff3a9400640 1 -- 172.21.15.28:0/2611688095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c032440 con 0x7ff3ac1592a0 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.942+0000 7ff3a9400640 1 -- 172.21.15.28:0/2611688095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c0325a0 con 0x7ff3ac1592a0 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.942+0000 7ff3a9400640 1 -- 172.21.15.28:0/2611688095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c032930 con 0x7ff3ac1592a0 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 -- 172.21.15.28:0/2611688095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac1592a0 msgr2=0x7ff3ac15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15ab70 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7ff39c0097a0 tx=0x7ff39c02a180 comp rx=0 tx=0).stop 2024-09-13T17:16:35.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 -- 172.21.15.28:0/2611688095 shutdown_connections 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06c2d0 0x7ff3ac06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/2611688095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15ab70 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 -- 172.21.15.28:0/2611688095 >> 172.21.15.28:0/2611688095 conn(0x7ff3ac069e80 msgr2=0x7ff3ac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 -- 172.21.15.28:0/2611688095 shutdown_connections 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.943+0000 7ff3b0e00640 1 -- 172.21.15.28:0/2611688095 wait complete. 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.945+0000 7ff3b0e00640 1 Processor -- start 2024-09-13T17:16:35.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.945+0000 7ff3b0e00640 1 -- start start 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 0x7ff3ac151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0cb990 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0cbc70 con 0x7ff3ac1592a0 2024-09-13T17:16:35.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3b0e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ac0cbf50 con 0x7ff3ac06f970 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45052/0 (socket says 172.21.15.28:45052) 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 -- 172.21.15.28:0/442747340 learned_addr learned my addr 172.21.15.28:0/442747340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3aa800640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a1400640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 0x7ff3ac151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 -- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 msgr2=0x7ff3ac151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 0x7ff3ac151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 -- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3ac1592a0 msgr2=0x7ff3ac15b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a9e00640 1 -- 172.21.15.28:0/442747340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff39c009400 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.946+0000 7ff3a1400640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 0x7ff3ac151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:16:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3a9e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7ff39c035040 tx=0x7ff39c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c006e10 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c032e90 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39c030be0 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ac0cc230 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.947+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ac1f2b20 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.948+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ac1f2cb0 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.949+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff39c039040 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.950+0000 7ff3a2a00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 0x7ff38806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.950+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff39c0a9ca0 con 0x7ff3ac06c2d0 2024-09-13T17:16:35.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.951+0000 7ff3a1400640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 0x7ff38806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:35.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.951+0000 7ff3a1400640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 0x7ff38806d0c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff394007fd0 tx=0x7ff394009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:35.954+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff39c074f00 con 0x7ff3ac06c2d0 2024-09-13T17:16:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.165+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff3ac0b3740 con 0x7ff3ac06c2d0 2024-09-13T17:16:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.166+0000 7ff3a2a00640 1 -- 172.21.15.28:0/442747340 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v114) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff39c030d40 con 0x7ff3ac06c2d0 2024-09-13T17:16:36.168 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 msgr2=0x7ff38806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 0x7ff38806d0c0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ff394007fd0 tx=0x7ff394009040 comp rx=0 tx=0).stop 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 msgr2=0x7ff3ac1f25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7ff39c035040 tx=0x7ff39c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 shutdown_connections 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38806ac00 0x7ff38806d0c0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3ac1592a0 0x7ff3ac15b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3ac06f970 0x7ff3ac151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 --2- 172.21.15.28:0/442747340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3ac06c2d0 0x7ff3ac1f25e0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 >> 172.21.15.28:0/442747340 conn(0x7ff3ac069e80 msgr2=0x7ff3ac15c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 shutdown_connections 2024-09-13T17:16:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:36.168+0000 7ff3b0e00640 1 -- 172.21.15.28:0/442747340 wait complete. 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:16:36.186 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:16:36.187 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:16:36.187 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:16:36.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.187 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:16:36.223 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:16:36.224 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42185sec preferred_lft 42185sec 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:16:36.225 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:16:36.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.227 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:16:36.318 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:16:36.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.321 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:16:36.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.358 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:16:36.358 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:36.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:16:36.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:16:36.359 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:16:36.359 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:36.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.505 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:16:36.541 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:16:36.541 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:16:36.541 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:16:36.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:16:36.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:16:36.637 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:16:36.672 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:16:36.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:16:36.672 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:16:36.748 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:16:36.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:16:36.877 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:16:36.877 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:16:36.904 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:16:36.904 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:16:36.904 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:16:36.905 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:16:36.905 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:16:36.906 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:16:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:16:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:16:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:16:36.906 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:16:36.909 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:16:36.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:16:36.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:36.966 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:16:37.009 INFO:teuthology.orchestra.run.smithi028.stdout:client7163 2024-09-13T17:16:37.009 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:16:37.010 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:16:37.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.0 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:37.086 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.196 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215767 s, 48.6 MB/s 2024-09-13T17:16:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209382 s, 50.1 MB/s 2024-09-13T17:16:37.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170366 s, 61.5 MB/s 2024-09-13T17:16:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168447 s, 62.2 MB/s 2024-09-13T17:16:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158967 s, 66.0 MB/s 2024-09-13T17:16:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178586 s, 58.7 MB/s 2024-09-13T17:16:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160177 s, 65.5 MB/s 2024-09-13T17:16:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157464 s, 66.6 MB/s 2024-09-13T17:16:37.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.351 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161396 s, 65.0 MB/s 2024-09-13T17:16:37.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152686 s, 68.7 MB/s 2024-09-13T17:16:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174207 s, 60.2 MB/s 2024-09-13T17:16:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.405 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145816 s, 71.9 MB/s 2024-09-13T17:16:37.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127055 s, 82.5 MB/s 2024-09-13T17:16:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171485 s, 61.1 MB/s 2024-09-13T17:16:37.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.458 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160872 s, 65.2 MB/s 2024-09-13T17:16:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.474 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142756 s, 73.5 MB/s 2024-09-13T17:16:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.490 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013959 s, 75.1 MB/s 2024-09-13T17:16:37.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.506 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137371 s, 76.3 MB/s 2024-09-13T17:16:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158477 s, 66.2 MB/s 2024-09-13T17:16:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.545 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186488 s, 56.2 MB/s 2024-09-13T17:16:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161618 s, 64.9 MB/s 2024-09-13T17:16:37.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.580 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147165 s, 71.3 MB/s 2024-09-13T17:16:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.598 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151324 s, 69.3 MB/s 2024-09-13T17:16:37.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.614 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139461 s, 75.2 MB/s 2024-09-13T17:16:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189116 s, 55.4 MB/s 2024-09-13T17:16:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.649 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120012 s, 87.4 MB/s 2024-09-13T17:16:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175442 s, 59.8 MB/s 2024-09-13T17:16:37.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.685 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128848 s, 81.4 MB/s 2024-09-13T17:16:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.699 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118513 s, 88.5 MB/s 2024-09-13T17:16:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161942 s, 64.8 MB/s 2024-09-13T17:16:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.732 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126753 s, 82.7 MB/s 2024-09-13T17:16:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157365 s, 66.6 MB/s 2024-09-13T17:16:37.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.1 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:37.752 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.808 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136479 s, 76.8 MB/s 2024-09-13T17:16:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137197 s, 76.4 MB/s 2024-09-13T17:16:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.842 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161345 s, 65.0 MB/s 2024-09-13T17:16:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.857 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.858 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131635 s, 79.7 MB/s 2024-09-13T17:16:37.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.874 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140562 s, 74.6 MB/s 2024-09-13T17:16:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.893 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016545 s, 63.4 MB/s 2024-09-13T17:16:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140309 s, 74.7 MB/s 2024-09-13T17:16:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139906 s, 74.9 MB/s 2024-09-13T17:16:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130787 s, 80.2 MB/s 2024-09-13T17:16:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137358 s, 76.3 MB/s 2024-09-13T17:16:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135642 s, 77.3 MB/s 2024-09-13T17:16:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:37.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133508 s, 78.5 MB/s 2024-09-13T17:16:38.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130725 s, 80.2 MB/s 2024-09-13T17:16:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137841 s, 76.1 MB/s 2024-09-13T17:16:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139136 s, 75.4 MB/s 2024-09-13T17:16:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.053 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162117 s, 64.7 MB/s 2024-09-13T17:16:38.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.070 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139101 s, 75.4 MB/s 2024-09-13T17:16:38.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.085 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130609 s, 80.3 MB/s 2024-09-13T17:16:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142066 s, 73.8 MB/s 2024-09-13T17:16:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.117 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135737 s, 77.3 MB/s 2024-09-13T17:16:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150091 s, 69.9 MB/s 2024-09-13T17:16:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168567 s, 62.2 MB/s 2024-09-13T17:16:38.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.168 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123285 s, 85.1 MB/s 2024-09-13T17:16:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159957 s, 65.6 MB/s 2024-09-13T17:16:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.203 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140739 s, 74.5 MB/s 2024-09-13T17:16:38.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126497 s, 82.9 MB/s 2024-09-13T17:16:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134018 s, 78.2 MB/s 2024-09-13T17:16:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178057 s, 58.9 MB/s 2024-09-13T17:16:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143467 s, 73.1 MB/s 2024-09-13T17:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016322 s, 64.2 MB/s 2024-09-13T17:16:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.304 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132056 s, 79.4 MB/s 2024-09-13T17:16:38.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133452 s, 78.6 MB/s 2024-09-13T17:16:38.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:38.321 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:38.321 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:38.321 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.2 2024-09-13T17:16:38.321 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:38.321 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:38.322 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:38.322 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136711 s, 76.7 MB/s 2024-09-13T17:16:38.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.392 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013679 s, 76.7 MB/s 2024-09-13T17:16:38.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172804 s, 60.7 MB/s 2024-09-13T17:16:38.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.428 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141503 s, 74.1 MB/s 2024-09-13T17:16:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131218 s, 79.9 MB/s 2024-09-13T17:16:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.458 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012766 s, 82.1 MB/s 2024-09-13T17:16:38.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.474 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132912 s, 78.9 MB/s 2024-09-13T17:16:38.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.490 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137694 s, 76.2 MB/s 2024-09-13T17:16:38.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.506 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140167 s, 74.8 MB/s 2024-09-13T17:16:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.521 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131962 s, 79.5 MB/s 2024-09-13T17:16:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.540 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166557 s, 63.0 MB/s 2024-09-13T17:16:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.555 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129387 s, 81.0 MB/s 2024-09-13T17:16:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.571 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132412 s, 79.2 MB/s 2024-09-13T17:16:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132502 s, 79.1 MB/s 2024-09-13T17:16:38.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130867 s, 80.1 MB/s 2024-09-13T17:16:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013433 s, 78.1 MB/s 2024-09-13T17:16:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139846 s, 75.0 MB/s 2024-09-13T17:16:38.653 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.653 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.653 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170224 s, 61.6 MB/s 2024-09-13T17:16:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.669 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138704 s, 75.6 MB/s 2024-09-13T17:16:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.685 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137826 s, 76.1 MB/s 2024-09-13T17:16:38.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135655 s, 77.3 MB/s 2024-09-13T17:16:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135554 s, 77.4 MB/s 2024-09-13T17:16:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129528 s, 81.0 MB/s 2024-09-13T17:16:38.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.746 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123629 s, 84.8 MB/s 2024-09-13T17:16:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.761 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130791 s, 80.2 MB/s 2024-09-13T17:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134108 s, 78.2 MB/s 2024-09-13T17:16:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.792 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012683 s, 82.7 MB/s 2024-09-13T17:16:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.807 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133653 s, 78.5 MB/s 2024-09-13T17:16:38.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143013 s, 73.3 MB/s 2024-09-13T17:16:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.843 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171259 s, 61.2 MB/s 2024-09-13T17:16:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163356 s, 64.2 MB/s 2024-09-13T17:16:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133238 s, 78.7 MB/s 2024-09-13T17:16:38.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:38.879 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:38.879 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:38.879 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.3 2024-09-13T17:16:38.879 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:38.879 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:38.880 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:38.880 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141269 s, 74.2 MB/s 2024-09-13T17:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125808 s, 83.3 MB/s 2024-09-13T17:16:38.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178522 s, 58.7 MB/s 2024-09-13T17:16:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:38.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159395 s, 65.8 MB/s 2024-09-13T17:16:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131641 s, 79.7 MB/s 2024-09-13T17:16:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129169 s, 81.2 MB/s 2024-09-13T17:16:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.033 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012728 s, 82.4 MB/s 2024-09-13T17:16:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015528 s, 67.5 MB/s 2024-09-13T17:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128154 s, 81.8 MB/s 2024-09-13T17:16:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156084 s, 67.2 MB/s 2024-09-13T17:16:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.105 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181589 s, 57.7 MB/s 2024-09-13T17:16:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170141 s, 61.6 MB/s 2024-09-13T17:16:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140309 s, 74.7 MB/s 2024-09-13T17:16:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.160 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170696 s, 61.4 MB/s 2024-09-13T17:16:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012548 s, 83.6 MB/s 2024-09-13T17:16:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162352 s, 64.6 MB/s 2024-09-13T17:16:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135489 s, 77.4 MB/s 2024-09-13T17:16:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171412 s, 61.2 MB/s 2024-09-13T17:16:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129518 s, 81.0 MB/s 2024-09-13T17:16:39.260 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146686 s, 71.5 MB/s 2024-09-13T17:16:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185029 s, 56.7 MB/s 2024-09-13T17:16:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126318 s, 83.0 MB/s 2024-09-13T17:16:39.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.311 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127626 s, 82.2 MB/s 2024-09-13T17:16:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158443 s, 66.2 MB/s 2024-09-13T17:16:39.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.345 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136233 s, 77.0 MB/s 2024-09-13T17:16:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.362 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014344 s, 73.1 MB/s 2024-09-13T17:16:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164124 s, 63.9 MB/s 2024-09-13T17:16:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.398 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155189 s, 67.6 MB/s 2024-09-13T17:16:39.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118179 s, 88.7 MB/s 2024-09-13T17:16:39.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168711 s, 62.2 MB/s 2024-09-13T17:16:39.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012974 s, 80.8 MB/s 2024-09-13T17:16:39.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136351 s, 76.9 MB/s 2024-09-13T17:16:39.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:39.463 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:39.463 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:39.463 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.4 2024-09-13T17:16:39.464 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:39.464 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:39.464 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:39.464 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144432 s, 72.6 MB/s 2024-09-13T17:16:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138839 s, 75.5 MB/s 2024-09-13T17:16:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.554 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154419 s, 67.9 MB/s 2024-09-13T17:16:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015112 s, 69.4 MB/s 2024-09-13T17:16:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161513 s, 64.9 MB/s 2024-09-13T17:16:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.605 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131835 s, 79.5 MB/s 2024-09-13T17:16:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.620 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125665 s, 83.4 MB/s 2024-09-13T17:16:39.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161028 s, 65.1 MB/s 2024-09-13T17:16:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146516 s, 71.6 MB/s 2024-09-13T17:16:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136753 s, 76.7 MB/s 2024-09-13T17:16:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136886 s, 76.6 MB/s 2024-09-13T17:16:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132783 s, 79.0 MB/s 2024-09-13T17:16:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139207 s, 75.3 MB/s 2024-09-13T17:16:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141149 s, 74.3 MB/s 2024-09-13T17:16:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013154 s, 79.7 MB/s 2024-09-13T17:16:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130687 s, 80.2 MB/s 2024-09-13T17:16:39.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.780 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125695 s, 83.4 MB/s 2024-09-13T17:16:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138649 s, 75.6 MB/s 2024-09-13T17:16:39.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119055 s, 88.1 MB/s 2024-09-13T17:16:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156943 s, 66.8 MB/s 2024-09-13T17:16:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134187 s, 78.1 MB/s 2024-09-13T17:16:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138647 s, 75.6 MB/s 2024-09-13T17:16:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155538 s, 67.4 MB/s 2024-09-13T17:16:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137142 s, 76.5 MB/s 2024-09-13T17:16:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139483 s, 75.2 MB/s 2024-09-13T17:16:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130739 s, 80.2 MB/s 2024-09-13T17:16:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132201 s, 79.3 MB/s 2024-09-13T17:16:39.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132237 s, 79.3 MB/s 2024-09-13T17:16:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125665 s, 83.4 MB/s 2024-09-13T17:16:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152549 s, 68.7 MB/s 2024-09-13T17:16:40.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127999 s, 81.9 MB/s 2024-09-13T17:16:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012626 s, 83.0 MB/s 2024-09-13T17:16:40.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.5 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:40.020 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127361 s, 82.3 MB/s 2024-09-13T17:16:40.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128813 s, 81.4 MB/s 2024-09-13T17:16:40.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.108 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165078 s, 63.5 MB/s 2024-09-13T17:16:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132267 s, 79.3 MB/s 2024-09-13T17:16:40.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.139 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134296 s, 78.1 MB/s 2024-09-13T17:16:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164153 s, 63.9 MB/s 2024-09-13T17:16:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.173 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130182 s, 80.5 MB/s 2024-09-13T17:16:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139075 s, 75.4 MB/s 2024-09-13T17:16:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125302 s, 83.7 MB/s 2024-09-13T17:16:40.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.219 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012555 s, 83.5 MB/s 2024-09-13T17:16:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165153 s, 63.5 MB/s 2024-09-13T17:16:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011878 s, 88.3 MB/s 2024-09-13T17:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.268 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131729 s, 79.6 MB/s 2024-09-13T17:16:40.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.282 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125613 s, 83.5 MB/s 2024-09-13T17:16:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.297 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126472 s, 82.9 MB/s 2024-09-13T17:16:40.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128946 s, 81.3 MB/s 2024-09-13T17:16:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129941 s, 80.7 MB/s 2024-09-13T17:16:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121909 s, 86.0 MB/s 2024-09-13T17:16:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136243 s, 77.0 MB/s 2024-09-13T17:16:40.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.376 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160238 s, 65.4 MB/s 2024-09-13T17:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129258 s, 81.1 MB/s 2024-09-13T17:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128557 s, 81.6 MB/s 2024-09-13T17:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139364 s, 75.2 MB/s 2024-09-13T17:16:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134588 s, 77.9 MB/s 2024-09-13T17:16:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136855 s, 76.6 MB/s 2024-09-13T17:16:40.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:16:40.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:16:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014827 s, 70.7 MB/s 2024-09-13T17:16:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.489 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157235 s, 66.7 MB/s 2024-09-13T17:16:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168027 s, 62.4 MB/s 2024-09-13T17:16:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135597 s, 77.3 MB/s 2024-09-13T17:16:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.540 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135998 s, 77.1 MB/s 2024-09-13T17:16:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.555 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012985 s, 80.8 MB/s 2024-09-13T17:16:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172409 s, 60.8 MB/s 2024-09-13T17:16:40.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.6 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:40.577 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128255 s, 81.8 MB/s 2024-09-13T17:16:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129506 s, 81.0 MB/s 2024-09-13T17:16:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127765 s, 82.1 MB/s 2024-09-13T17:16:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166147 s, 63.1 MB/s 2024-09-13T17:16:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.733 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114624 s, 91.5 MB/s 2024-09-13T17:16:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133484 s, 78.6 MB/s 2024-09-13T17:16:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127568 s, 82.2 MB/s 2024-09-13T17:16:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120379 s, 87.1 MB/s 2024-09-13T17:16:40.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132622 s, 79.1 MB/s 2024-09-13T17:16:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139459 s, 75.2 MB/s 2024-09-13T17:16:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137867 s, 76.1 MB/s 2024-09-13T17:16:40.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125843 s, 83.3 MB/s 2024-09-13T17:16:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.856 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130065 s, 80.6 MB/s 2024-09-13T17:16:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.872 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135914 s, 77.1 MB/s 2024-09-13T17:16:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167842 s, 62.5 MB/s 2024-09-13T17:16:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128728 s, 81.5 MB/s 2024-09-13T17:16:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145846 s, 71.9 MB/s 2024-09-13T17:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137377 s, 76.3 MB/s 2024-09-13T17:16:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130953 s, 80.1 MB/s 2024-09-13T17:16:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163604 s, 64.1 MB/s 2024-09-13T17:16:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:40.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136836 s, 76.6 MB/s 2024-09-13T17:16:41.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139584 s, 75.1 MB/s 2024-09-13T17:16:41.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170546 s, 61.5 MB/s 2024-09-13T17:16:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140667 s, 74.5 MB/s 2024-09-13T17:16:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137932 s, 76.0 MB/s 2024-09-13T17:16:41.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.071 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012473 s, 84.1 MB/s 2024-09-13T17:16:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133804 s, 78.4 MB/s 2024-09-13T17:16:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130358 s, 80.4 MB/s 2024-09-13T17:16:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.118 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140304 s, 74.7 MB/s 2024-09-13T17:16:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158046 s, 66.3 MB/s 2024-09-13T17:16:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.152 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132735 s, 79.0 MB/s 2024-09-13T17:16:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129103 s, 81.2 MB/s 2024-09-13T17:16:41.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:41.168 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:41.168 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:41.168 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.7 2024-09-13T17:16:41.168 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:41.169 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:41.169 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:41.169 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170921 s, 61.3 MB/s 2024-09-13T17:16:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158032 s, 66.4 MB/s 2024-09-13T17:16:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.260 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130443 s, 80.4 MB/s 2024-09-13T17:16:41.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123879 s, 84.6 MB/s 2024-09-13T17:16:41.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117677 s, 89.1 MB/s 2024-09-13T17:16:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140354 s, 74.7 MB/s 2024-09-13T17:16:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168027 s, 62.4 MB/s 2024-09-13T17:16:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139609 s, 75.1 MB/s 2024-09-13T17:16:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158659 s, 66.1 MB/s 2024-09-13T17:16:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.374 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137407 s, 76.3 MB/s 2024-09-13T17:16:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168151 s, 62.4 MB/s 2024-09-13T17:16:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118682 s, 88.4 MB/s 2024-09-13T17:16:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125624 s, 83.5 MB/s 2024-09-13T17:16:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141994 s, 73.8 MB/s 2024-09-13T17:16:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129113 s, 81.2 MB/s 2024-09-13T17:16:41.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.469 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136287 s, 76.9 MB/s 2024-09-13T17:16:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143296 s, 73.2 MB/s 2024-09-13T17:16:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125758 s, 83.4 MB/s 2024-09-13T17:16:41.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138734 s, 75.6 MB/s 2024-09-13T17:16:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013404 s, 78.2 MB/s 2024-09-13T17:16:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156652 s, 66.9 MB/s 2024-09-13T17:16:41.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.566 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013095 s, 80.1 MB/s 2024-09-13T17:16:41.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.581 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130454 s, 80.4 MB/s 2024-09-13T17:16:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169398 s, 61.9 MB/s 2024-09-13T17:16:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132358 s, 79.2 MB/s 2024-09-13T17:16:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.634 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156581 s, 67.0 MB/s 2024-09-13T17:16:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.653 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170632 s, 61.5 MB/s 2024-09-13T17:16:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.669 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139394 s, 75.2 MB/s 2024-09-13T17:16:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.685 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135194 s, 77.6 MB/s 2024-09-13T17:16:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132358 s, 79.2 MB/s 2024-09-13T17:16:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135423 s, 77.4 MB/s 2024-09-13T17:16:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161675 s, 64.9 MB/s 2024-09-13T17:16:41.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.8 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:41.736 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:41.790 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.791 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132336 s, 79.2 MB/s 2024-09-13T17:16:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111467 s, 94.1 MB/s 2024-09-13T17:16:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127461 s, 82.3 MB/s 2024-09-13T17:16:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167452 s, 62.6 MB/s 2024-09-13T17:16:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.853 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130462 s, 80.4 MB/s 2024-09-13T17:16:41.868 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129736 s, 80.8 MB/s 2024-09-13T17:16:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.884 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133905 s, 78.3 MB/s 2024-09-13T17:16:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173961 s, 60.3 MB/s 2024-09-13T17:16:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.922 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016143 s, 65.0 MB/s 2024-09-13T17:16:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155124 s, 67.6 MB/s 2024-09-13T17:16:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123539 s, 84.9 MB/s 2024-09-13T17:16:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138812 s, 75.5 MB/s 2024-09-13T17:16:41.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:41.986 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01303 s, 80.5 MB/s 2024-09-13T17:16:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135674 s, 77.3 MB/s 2024-09-13T17:16:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134138 s, 78.2 MB/s 2024-09-13T17:16:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127844 s, 82.0 MB/s 2024-09-13T17:16:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.049 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142919 s, 73.4 MB/s 2024-09-13T17:16:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136623 s, 76.7 MB/s 2024-09-13T17:16:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169796 s, 61.8 MB/s 2024-09-13T17:16:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132652 s, 79.0 MB/s 2024-09-13T17:16:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133557 s, 78.5 MB/s 2024-09-13T17:16:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133884 s, 78.3 MB/s 2024-09-13T17:16:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.146 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128119 s, 81.8 MB/s 2024-09-13T17:16:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168974 s, 62.1 MB/s 2024-09-13T17:16:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123175 s, 85.1 MB/s 2024-09-13T17:16:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125011 s, 83.9 MB/s 2024-09-13T17:16:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131287 s, 79.9 MB/s 2024-09-13T17:16:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128656 s, 81.5 MB/s 2024-09-13T17:16:42.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165981 s, 63.2 MB/s 2024-09-13T17:16:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134063 s, 78.2 MB/s 2024-09-13T17:16:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137186 s, 76.4 MB/s 2024-09-13T17:16:42.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.294 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171821 s, 61.0 MB/s 2024-09-13T17:16:42.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.9 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:42.296 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:42.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133018 s, 78.8 MB/s 2024-09-13T17:16:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132073 s, 79.4 MB/s 2024-09-13T17:16:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012725 s, 82.4 MB/s 2024-09-13T17:16:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134062 s, 78.2 MB/s 2024-09-13T17:16:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123868 s, 84.7 MB/s 2024-09-13T17:16:42.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.427 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135183 s, 77.6 MB/s 2024-09-13T17:16:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165656 s, 63.3 MB/s 2024-09-13T17:16:42.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137563 s, 76.2 MB/s 2024-09-13T17:16:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.478 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138156 s, 75.9 MB/s 2024-09-13T17:16:42.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127072 s, 82.5 MB/s 2024-09-13T17:16:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012898 s, 81.3 MB/s 2024-09-13T17:16:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.526 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162398 s, 64.6 MB/s 2024-09-13T17:16:42.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.542 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130157 s, 80.6 MB/s 2024-09-13T17:16:42.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133685 s, 78.4 MB/s 2024-09-13T17:16:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.571 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112648 s, 93.1 MB/s 2024-09-13T17:16:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161078 s, 65.1 MB/s 2024-09-13T17:16:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154418 s, 67.9 MB/s 2024-09-13T17:16:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150565 s, 69.6 MB/s 2024-09-13T17:16:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127119 s, 82.5 MB/s 2024-09-13T17:16:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.658 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168685 s, 62.2 MB/s 2024-09-13T17:16:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125682 s, 83.4 MB/s 2024-09-13T17:16:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133299 s, 78.7 MB/s 2024-09-13T17:16:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128421 s, 81.7 MB/s 2024-09-13T17:16:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016275 s, 64.4 MB/s 2024-09-13T17:16:42.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153478 s, 68.3 MB/s 2024-09-13T17:16:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.757 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154728 s, 67.8 MB/s 2024-09-13T17:16:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133349 s, 78.6 MB/s 2024-09-13T17:16:42.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.790 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.790 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151115 s, 69.4 MB/s 2024-09-13T17:16:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161895 s, 64.8 MB/s 2024-09-13T17:16:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136202 s, 77.0 MB/s 2024-09-13T17:16:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015519 s, 67.6 MB/s 2024-09-13T17:16:42.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168795 s, 62.1 MB/s 2024-09-13T17:16:42.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:42.862 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:42.862 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:42.863 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.10 2024-09-13T17:16:42.863 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:42.863 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:42.863 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:42.863 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130673 s, 80.2 MB/s 2024-09-13T17:16:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.936 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166018 s, 63.2 MB/s 2024-09-13T17:16:42.951 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.951 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.951 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012923 s, 81.1 MB/s 2024-09-13T17:16:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135987 s, 77.1 MB/s 2024-09-13T17:16:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132567 s, 79.1 MB/s 2024-09-13T17:16:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:42.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137973 s, 76.0 MB/s 2024-09-13T17:16:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168996 s, 62.0 MB/s 2024-09-13T17:16:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139533 s, 75.1 MB/s 2024-09-13T17:16:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136179 s, 77.0 MB/s 2024-09-13T17:16:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139366 s, 75.2 MB/s 2024-09-13T17:16:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014365 s, 73.0 MB/s 2024-09-13T17:16:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165852 s, 63.2 MB/s 2024-09-13T17:16:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.115 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120738 s, 86.8 MB/s 2024-09-13T17:16:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.131 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013113 s, 80.0 MB/s 2024-09-13T17:16:43.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.146 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132578 s, 79.1 MB/s 2024-09-13T17:16:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166164 s, 63.1 MB/s 2024-09-13T17:16:43.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.184 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162504 s, 64.5 MB/s 2024-09-13T17:16:43.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.200 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134903 s, 77.7 MB/s 2024-09-13T17:16:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129303 s, 81.1 MB/s 2024-09-13T17:16:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135387 s, 77.5 MB/s 2024-09-13T17:16:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135126 s, 77.6 MB/s 2024-09-13T17:16:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167602 s, 62.6 MB/s 2024-09-13T17:16:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166716 s, 62.9 MB/s 2024-09-13T17:16:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162601 s, 64.5 MB/s 2024-09-13T17:16:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.319 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139599 s, 75.1 MB/s 2024-09-13T17:16:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.334 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130297 s, 80.5 MB/s 2024-09-13T17:16:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137895 s, 76.0 MB/s 2024-09-13T17:16:43.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128127 s, 81.8 MB/s 2024-09-13T17:16:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012731 s, 82.4 MB/s 2024-09-13T17:16:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.395 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128259 s, 81.8 MB/s 2024-09-13T17:16:43.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161778 s, 64.8 MB/s 2024-09-13T17:16:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133339 s, 78.6 MB/s 2024-09-13T17:16:43.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.11 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:43.431 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:43.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132258 s, 79.3 MB/s 2024-09-13T17:16:43.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.501 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013161 s, 79.7 MB/s 2024-09-13T17:16:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132532 s, 79.1 MB/s 2024-09-13T17:16:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153968 s, 68.1 MB/s 2024-09-13T17:16:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.549 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128163 s, 81.8 MB/s 2024-09-13T17:16:43.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.564 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125874 s, 83.3 MB/s 2024-09-13T17:16:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.580 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139094 s, 75.4 MB/s 2024-09-13T17:16:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167927 s, 62.4 MB/s 2024-09-13T17:16:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.614 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129646 s, 80.9 MB/s 2024-09-13T17:16:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.630 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139932 s, 74.9 MB/s 2024-09-13T17:16:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141944 s, 73.9 MB/s 2024-09-13T17:16:43.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128365 s, 81.7 MB/s 2024-09-13T17:16:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134423 s, 78.0 MB/s 2024-09-13T17:16:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136582 s, 76.8 MB/s 2024-09-13T17:16:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132342 s, 79.2 MB/s 2024-09-13T17:16:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160743 s, 65.2 MB/s 2024-09-13T17:16:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012915 s, 81.2 MB/s 2024-09-13T17:16:43.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.757 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129042 s, 81.3 MB/s 2024-09-13T17:16:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129056 s, 81.2 MB/s 2024-09-13T17:16:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.787 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012273 s, 85.4 MB/s 2024-09-13T17:16:43.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.802 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132794 s, 79.0 MB/s 2024-09-13T17:16:43.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139701 s, 75.1 MB/s 2024-09-13T17:16:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127189 s, 82.4 MB/s 2024-09-13T17:16:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131629 s, 79.7 MB/s 2024-09-13T17:16:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130003 s, 80.7 MB/s 2024-09-13T17:16:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137808 s, 76.1 MB/s 2024-09-13T17:16:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130564 s, 80.3 MB/s 2024-09-13T17:16:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.911 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126706 s, 82.8 MB/s 2024-09-13T17:16:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.929 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162084 s, 64.7 MB/s 2024-09-13T17:16:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015501 s, 67.6 MB/s 2024-09-13T17:16:43.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149895 s, 70.0 MB/s 2024-09-13T17:16:43.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:43.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:43.981 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129549 s, 80.9 MB/s 2024-09-13T17:16:43.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.12 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:43.981 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:44.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134975 s, 77.7 MB/s 2024-09-13T17:16:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156254 s, 67.1 MB/s 2024-09-13T17:16:44.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.071 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139 s, 75.4 MB/s 2024-09-13T17:16:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.087 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140128 s, 74.8 MB/s 2024-09-13T17:16:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129349 s, 81.1 MB/s 2024-09-13T17:16:44.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.118 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130078 s, 80.6 MB/s 2024-09-13T17:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.133 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126802 s, 82.7 MB/s 2024-09-13T17:16:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124722 s, 84.1 MB/s 2024-09-13T17:16:44.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129321 s, 81.1 MB/s 2024-09-13T17:16:44.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131088 s, 80.0 MB/s 2024-09-13T17:16:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.196 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016717 s, 62.7 MB/s 2024-09-13T17:16:44.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164141 s, 63.9 MB/s 2024-09-13T17:16:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132166 s, 79.3 MB/s 2024-09-13T17:16:44.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169995 s, 61.7 MB/s 2024-09-13T17:16:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.264 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125576 s, 83.5 MB/s 2024-09-13T17:16:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172849 s, 60.7 MB/s 2024-09-13T17:16:44.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.300 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013606 s, 77.1 MB/s 2024-09-13T17:16:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159018 s, 65.9 MB/s 2024-09-13T17:16:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132651 s, 79.0 MB/s 2024-09-13T17:16:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012797 s, 81.9 MB/s 2024-09-13T17:16:44.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130535 s, 80.3 MB/s 2024-09-13T17:16:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127073 s, 82.5 MB/s 2024-09-13T17:16:44.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128603 s, 81.5 MB/s 2024-09-13T17:16:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125008 s, 83.9 MB/s 2024-09-13T17:16:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138913 s, 75.5 MB/s 2024-09-13T17:16:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129635 s, 80.9 MB/s 2024-09-13T17:16:44.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140001 s, 74.9 MB/s 2024-09-13T17:16:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133226 s, 78.7 MB/s 2024-09-13T17:16:44.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.536 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159542 s, 65.7 MB/s 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172436 s, 60.8 MB/s 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132631 s, 79.1 MB/s 2024-09-13T17:16:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142149 s, 73.8 MB/s 2024-09-13T17:16:44.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:44.543 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:44.543 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:44.543 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.13 2024-09-13T17:16:44.543 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:44.543 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:44.544 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:44.544 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:44.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135422 s, 77.4 MB/s 2024-09-13T17:16:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131755 s, 79.6 MB/s 2024-09-13T17:16:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.663 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137248 s, 76.4 MB/s 2024-09-13T17:16:44.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136597 s, 76.8 MB/s 2024-09-13T17:16:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135211 s, 77.6 MB/s 2024-09-13T17:16:44.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.710 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130859 s, 80.1 MB/s 2024-09-13T17:16:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136662 s, 76.7 MB/s 2024-09-13T17:16:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129393 s, 81.0 MB/s 2024-09-13T17:16:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.757 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136366 s, 76.9 MB/s 2024-09-13T17:16:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.772 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124733 s, 84.1 MB/s 2024-09-13T17:16:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131732 s, 79.6 MB/s 2024-09-13T17:16:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013471 s, 77.8 MB/s 2024-09-13T17:16:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.819 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135443 s, 77.4 MB/s 2024-09-13T17:16:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144371 s, 72.6 MB/s 2024-09-13T17:16:44.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.854 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154799 s, 67.7 MB/s 2024-09-13T17:16:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130338 s, 80.5 MB/s 2024-09-13T17:16:44.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135508 s, 77.4 MB/s 2024-09-13T17:16:44.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125964 s, 83.2 MB/s 2024-09-13T17:16:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163807 s, 64.0 MB/s 2024-09-13T17:16:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132829 s, 78.9 MB/s 2024-09-13T17:16:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014365 s, 73.0 MB/s 2024-09-13T17:16:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131617 s, 79.7 MB/s 2024-09-13T17:16:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.983 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161552 s, 64.9 MB/s 2024-09-13T17:16:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124573 s, 84.2 MB/s 2024-09-13T17:16:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130976 s, 80.1 MB/s 2024-09-13T17:16:45.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135969 s, 77.1 MB/s 2024-09-13T17:16:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133072 s, 78.8 MB/s 2024-09-13T17:16:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.061 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138881 s, 75.5 MB/s 2024-09-13T17:16:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.077 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01329 s, 78.9 MB/s 2024-09-13T17:16:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.096 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172525 s, 60.8 MB/s 2024-09-13T17:16:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129326 s, 81.1 MB/s 2024-09-13T17:16:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132847 s, 78.9 MB/s 2024-09-13T17:16:45.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.14 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:45.129 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:45.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.185 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134249 s, 78.1 MB/s 2024-09-13T17:16:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146009 s, 71.8 MB/s 2024-09-13T17:16:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135501 s, 77.4 MB/s 2024-09-13T17:16:45.231 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119043 s, 88.1 MB/s 2024-09-13T17:16:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135306 s, 77.5 MB/s 2024-09-13T17:16:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135827 s, 77.2 MB/s 2024-09-13T17:16:45.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.278 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126546 s, 82.9 MB/s 2024-09-13T17:16:45.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126307 s, 83.0 MB/s 2024-09-13T17:16:45.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.308 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132994 s, 78.8 MB/s 2024-09-13T17:16:45.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143006 s, 73.3 MB/s 2024-09-13T17:16:45.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013081 s, 80.2 MB/s 2024-09-13T17:16:45.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.356 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133527 s, 78.5 MB/s 2024-09-13T17:16:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134022 s, 78.2 MB/s 2024-09-13T17:16:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128776 s, 81.4 MB/s 2024-09-13T17:16:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163745 s, 64.0 MB/s 2024-09-13T17:16:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137394 s, 76.3 MB/s 2024-09-13T17:16:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.436 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135114 s, 77.6 MB/s 2024-09-13T17:16:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136831 s, 76.6 MB/s 2024-09-13T17:16:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114231 s, 91.8 MB/s 2024-09-13T17:16:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164948 s, 63.6 MB/s 2024-09-13T17:16:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154314 s, 68.0 MB/s 2024-09-13T17:16:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137686 s, 76.2 MB/s 2024-09-13T17:16:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140912 s, 74.4 MB/s 2024-09-13T17:16:45.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.548 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011214 s, 93.5 MB/s 2024-09-13T17:16:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129643 s, 80.9 MB/s 2024-09-13T17:16:45.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128631 s, 81.5 MB/s 2024-09-13T17:16:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136669 s, 76.7 MB/s 2024-09-13T17:16:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127137 s, 82.5 MB/s 2024-09-13T17:16:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124925 s, 83.9 MB/s 2024-09-13T17:16:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132033 s, 79.4 MB/s 2024-09-13T17:16:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.659 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173373 s, 60.5 MB/s 2024-09-13T17:16:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169509 s, 61.9 MB/s 2024-09-13T17:16:45.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:45.680 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.15 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:45.681 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131412 s, 79.8 MB/s 2024-09-13T17:16:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126969 s, 82.6 MB/s 2024-09-13T17:16:45.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.768 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153715 s, 68.2 MB/s 2024-09-13T17:16:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.784 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139972 s, 74.9 MB/s 2024-09-13T17:16:45.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.800 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136458 s, 76.8 MB/s 2024-09-13T17:16:45.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133308 s, 78.7 MB/s 2024-09-13T17:16:45.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.834 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168432 s, 62.3 MB/s 2024-09-13T17:16:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127262 s, 82.4 MB/s 2024-09-13T17:16:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126091 s, 83.2 MB/s 2024-09-13T17:16:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184511 s, 56.8 MB/s 2024-09-13T17:16:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128662 s, 81.5 MB/s 2024-09-13T17:16:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163536 s, 64.1 MB/s 2024-09-13T17:16:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127592 s, 82.2 MB/s 2024-09-13T17:16:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133201 s, 78.7 MB/s 2024-09-13T17:16:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.963 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118963 s, 88.1 MB/s 2024-09-13T17:16:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134489 s, 78.0 MB/s 2024-09-13T17:16:45.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:45.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:45.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168415 s, 62.3 MB/s 2024-09-13T17:16:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135017 s, 77.7 MB/s 2024-09-13T17:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128733 s, 81.5 MB/s 2024-09-13T17:16:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.044 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133052 s, 78.8 MB/s 2024-09-13T17:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123652 s, 84.8 MB/s 2024-09-13T17:16:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.077 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162889 s, 64.4 MB/s 2024-09-13T17:16:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137082 s, 76.5 MB/s 2024-09-13T17:16:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130804 s, 80.2 MB/s 2024-09-13T17:16:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012851 s, 81.6 MB/s 2024-09-13T17:16:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122889 s, 85.3 MB/s 2024-09-13T17:16:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133046 s, 78.8 MB/s 2024-09-13T17:16:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125263 s, 83.7 MB/s 2024-09-13T17:16:46.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.185 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137245 s, 76.4 MB/s 2024-09-13T17:16:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.200 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132954 s, 78.9 MB/s 2024-09-13T17:16:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144922 s, 72.4 MB/s 2024-09-13T17:16:46.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135458 s, 77.4 MB/s 2024-09-13T17:16:46.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:46.234 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:46.234 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:46.235 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.16 2024-09-13T17:16:46.235 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:46.235 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:46.235 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:46.235 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171692 s, 61.1 MB/s 2024-09-13T17:16:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129479 s, 81.0 MB/s 2024-09-13T17:16:46.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130232 s, 80.5 MB/s 2024-09-13T17:16:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161752 s, 64.8 MB/s 2024-09-13T17:16:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137636 s, 76.2 MB/s 2024-09-13T17:16:46.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129437 s, 81.0 MB/s 2024-09-13T17:16:46.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.392 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164285 s, 63.8 MB/s 2024-09-13T17:16:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158025 s, 66.4 MB/s 2024-09-13T17:16:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131936 s, 79.5 MB/s 2024-09-13T17:16:46.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130894 s, 80.1 MB/s 2024-09-13T17:16:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127701 s, 82.1 MB/s 2024-09-13T17:16:46.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.469 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111012 s, 94.5 MB/s 2024-09-13T17:16:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.485 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132001 s, 79.4 MB/s 2024-09-13T17:16:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130943 s, 80.1 MB/s 2024-09-13T17:16:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159818 s, 65.6 MB/s 2024-09-13T17:16:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128912 s, 81.3 MB/s 2024-09-13T17:16:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130555 s, 80.3 MB/s 2024-09-13T17:16:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127249 s, 82.4 MB/s 2024-09-13T17:16:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012631 s, 83.0 MB/s 2024-09-13T17:16:46.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138881 s, 75.5 MB/s 2024-09-13T17:16:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126325 s, 83.0 MB/s 2024-09-13T17:16:46.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.625 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132481 s, 79.1 MB/s 2024-09-13T17:16:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165748 s, 63.3 MB/s 2024-09-13T17:16:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.659 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132902 s, 78.9 MB/s 2024-09-13T17:16:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.674 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129272 s, 81.1 MB/s 2024-09-13T17:16:46.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161258 s, 65.0 MB/s 2024-09-13T17:16:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130303 s, 80.5 MB/s 2024-09-13T17:16:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160689 s, 65.3 MB/s 2024-09-13T17:16:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132965 s, 78.9 MB/s 2024-09-13T17:16:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131377 s, 79.8 MB/s 2024-09-13T17:16:46.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013788 s, 76.0 MB/s 2024-09-13T17:16:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127409 s, 82.3 MB/s 2024-09-13T17:16:46.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:46.789 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:46.789 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:46.789 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.17 2024-09-13T17:16:46.789 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:46.790 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:46.790 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:46.790 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134314 s, 78.1 MB/s 2024-09-13T17:16:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143766 s, 72.9 MB/s 2024-09-13T17:16:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126323 s, 83.0 MB/s 2024-09-13T17:16:46.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128317 s, 81.7 MB/s 2024-09-13T17:16:46.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129047 s, 81.3 MB/s 2024-09-13T17:16:46.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122837 s, 85.4 MB/s 2024-09-13T17:16:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123391 s, 85.0 MB/s 2024-09-13T17:16:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167846 s, 62.5 MB/s 2024-09-13T17:16:46.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016973 s, 61.8 MB/s 2024-09-13T17:16:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141467 s, 74.1 MB/s 2024-09-13T17:16:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136885 s, 76.6 MB/s 2024-09-13T17:16:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158954 s, 66.0 MB/s 2024-09-13T17:16:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170139 s, 61.6 MB/s 2024-09-13T17:16:47.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126657 s, 82.8 MB/s 2024-09-13T17:16:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134062 s, 78.2 MB/s 2024-09-13T17:16:47.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.091 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138742 s, 75.6 MB/s 2024-09-13T17:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.106 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126793 s, 82.7 MB/s 2024-09-13T17:16:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141194 s, 74.3 MB/s 2024-09-13T17:16:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013628 s, 76.9 MB/s 2024-09-13T17:16:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01373 s, 76.4 MB/s 2024-09-13T17:16:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172795 s, 60.7 MB/s 2024-09-13T17:16:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134512 s, 78.0 MB/s 2024-09-13T17:16:47.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127989 s, 81.9 MB/s 2024-09-13T17:16:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165387 s, 63.4 MB/s 2024-09-13T17:16:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132952 s, 78.9 MB/s 2024-09-13T17:16:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013209 s, 79.4 MB/s 2024-09-13T17:16:47.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141597 s, 74.1 MB/s 2024-09-13T17:16:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012982 s, 80.8 MB/s 2024-09-13T17:16:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172193 s, 60.9 MB/s 2024-09-13T17:16:47.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.321 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128692 s, 81.5 MB/s 2024-09-13T17:16:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.336 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130207 s, 80.5 MB/s 2024-09-13T17:16:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014071 s, 74.5 MB/s 2024-09-13T17:16:47.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.18 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:47.354 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.408 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126761 s, 82.7 MB/s 2024-09-13T17:16:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.424 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133131 s, 78.8 MB/s 2024-09-13T17:16:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129622 s, 80.9 MB/s 2024-09-13T17:16:47.454 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013537 s, 77.5 MB/s 2024-09-13T17:16:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130576 s, 80.3 MB/s 2024-09-13T17:16:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.485 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132497 s, 79.1 MB/s 2024-09-13T17:16:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158156 s, 66.3 MB/s 2024-09-13T17:16:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120834 s, 86.8 MB/s 2024-09-13T17:16:47.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174556 s, 60.1 MB/s 2024-09-13T17:16:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.553 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133088 s, 78.8 MB/s 2024-09-13T17:16:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011836 s, 88.6 MB/s 2024-09-13T17:16:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.582 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131956 s, 79.5 MB/s 2024-09-13T17:16:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.601 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168202 s, 62.3 MB/s 2024-09-13T17:16:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130143 s, 80.6 MB/s 2024-09-13T17:16:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134798 s, 77.8 MB/s 2024-09-13T17:16:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.651 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016279 s, 64.4 MB/s 2024-09-13T17:16:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.668 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131826 s, 79.5 MB/s 2024-09-13T17:16:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.682 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129005 s, 81.3 MB/s 2024-09-13T17:16:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163896 s, 64.0 MB/s 2024-09-13T17:16:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152626 s, 68.7 MB/s 2024-09-13T17:16:47.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.733 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128977 s, 81.3 MB/s 2024-09-13T17:16:47.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125801 s, 83.4 MB/s 2024-09-13T17:16:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133679 s, 78.4 MB/s 2024-09-13T17:16:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01257 s, 83.4 MB/s 2024-09-13T17:16:47.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157234 s, 66.7 MB/s 2024-09-13T17:16:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130095 s, 80.6 MB/s 2024-09-13T17:16:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158728 s, 66.1 MB/s 2024-09-13T17:16:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.845 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132714 s, 79.0 MB/s 2024-09-13T17:16:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013338 s, 78.6 MB/s 2024-09-13T17:16:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128078 s, 81.9 MB/s 2024-09-13T17:16:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137596 s, 76.2 MB/s 2024-09-13T17:16:47.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.911 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173589 s, 60.4 MB/s 2024-09-13T17:16:47.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.19 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:47.913 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:47.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150305 s, 69.8 MB/s 2024-09-13T17:16:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:47.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124454 s, 84.3 MB/s 2024-09-13T17:16:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.000 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128503 s, 81.6 MB/s 2024-09-13T17:16:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.015 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130023 s, 80.6 MB/s 2024-09-13T17:16:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165278 s, 63.4 MB/s 2024-09-13T17:16:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.049 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134666 s, 77.9 MB/s 2024-09-13T17:16:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116467 s, 90.0 MB/s 2024-09-13T17:16:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.079 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125928 s, 83.3 MB/s 2024-09-13T17:16:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174978 s, 59.9 MB/s 2024-09-13T17:16:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132892 s, 78.9 MB/s 2024-09-13T17:16:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133311 s, 78.7 MB/s 2024-09-13T17:16:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122096 s, 85.9 MB/s 2024-09-13T17:16:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017159 s, 61.1 MB/s 2024-09-13T17:16:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134668 s, 77.9 MB/s 2024-09-13T17:16:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132699 s, 79.0 MB/s 2024-09-13T17:16:48.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133465 s, 78.6 MB/s 2024-09-13T17:16:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130848 s, 80.1 MB/s 2024-09-13T17:16:48.240 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.240 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.240 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131054 s, 80.0 MB/s 2024-09-13T17:16:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124165 s, 84.5 MB/s 2024-09-13T17:16:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129938 s, 80.7 MB/s 2024-09-13T17:16:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.285 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127463 s, 82.3 MB/s 2024-09-13T17:16:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013074 s, 80.2 MB/s 2024-09-13T17:16:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.319 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016475 s, 63.6 MB/s 2024-09-13T17:16:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119317 s, 87.9 MB/s 2024-09-13T17:16:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.353 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173803 s, 60.3 MB/s 2024-09-13T17:16:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110793 s, 94.6 MB/s 2024-09-13T17:16:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133387 s, 78.6 MB/s 2024-09-13T17:16:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129809 s, 80.8 MB/s 2024-09-13T17:16:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126134 s, 83.1 MB/s 2024-09-13T17:16:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135451 s, 77.4 MB/s 2024-09-13T17:16:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013052 s, 80.3 MB/s 2024-09-13T17:16:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156428 s, 67.0 MB/s 2024-09-13T17:16:48.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:48.462 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:48.462 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:48.462 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.20 2024-09-13T17:16:48.462 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:48.462 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:48.463 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:48.463 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128698 s, 81.5 MB/s 2024-09-13T17:16:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153124 s, 68.5 MB/s 2024-09-13T17:16:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131342 s, 79.8 MB/s 2024-09-13T17:16:48.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.566 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131959 s, 79.5 MB/s 2024-09-13T17:16:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127346 s, 82.3 MB/s 2024-09-13T17:16:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012993 s, 80.7 MB/s 2024-09-13T17:16:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141569 s, 74.1 MB/s 2024-09-13T17:16:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132798 s, 79.0 MB/s 2024-09-13T17:16:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016528 s, 63.4 MB/s 2024-09-13T17:16:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124215 s, 84.4 MB/s 2024-09-13T17:16:48.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160197 s, 65.5 MB/s 2024-09-13T17:16:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133005 s, 78.8 MB/s 2024-09-13T17:16:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141313 s, 74.2 MB/s 2024-09-13T17:16:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158729 s, 66.1 MB/s 2024-09-13T17:16:48.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162486 s, 64.5 MB/s 2024-09-13T17:16:48.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015883 s, 66.0 MB/s 2024-09-13T17:16:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.782 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139709 s, 75.1 MB/s 2024-09-13T17:16:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013572 s, 77.3 MB/s 2024-09-13T17:16:48.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137773 s, 76.1 MB/s 2024-09-13T17:16:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129426 s, 81.0 MB/s 2024-09-13T17:16:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159966 s, 65.6 MB/s 2024-09-13T17:16:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129539 s, 80.9 MB/s 2024-09-13T17:16:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129697 s, 80.8 MB/s 2024-09-13T17:16:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.892 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126038 s, 83.2 MB/s 2024-09-13T17:16:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144438 s, 72.6 MB/s 2024-09-13T17:16:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.927 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158491 s, 66.2 MB/s 2024-09-13T17:16:48.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.947 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176222 s, 59.5 MB/s 2024-09-13T17:16:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.962 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131179 s, 79.9 MB/s 2024-09-13T17:16:48.977 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.977 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.977 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124384 s, 84.3 MB/s 2024-09-13T17:16:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136328 s, 76.9 MB/s 2024-09-13T17:16:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.009 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136799 s, 76.7 MB/s 2024-09-13T17:16:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169147 s, 62.0 MB/s 2024-09-13T17:16:49.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:49.029 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.21 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:49.030 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:49.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.085 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129575 s, 80.9 MB/s 2024-09-13T17:16:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146886 s, 71.4 MB/s 2024-09-13T17:16:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.118 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143754 s, 72.9 MB/s 2024-09-13T17:16:49.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177662 s, 59.0 MB/s 2024-09-13T17:16:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154725 s, 67.8 MB/s 2024-09-13T17:16:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124779 s, 84.0 MB/s 2024-09-13T17:16:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136468 s, 76.8 MB/s 2024-09-13T17:16:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160019 s, 65.5 MB/s 2024-09-13T17:16:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134711 s, 77.8 MB/s 2024-09-13T17:16:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.235 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127451 s, 82.3 MB/s 2024-09-13T17:16:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.251 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132273 s, 79.3 MB/s 2024-09-13T17:16:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162497 s, 64.5 MB/s 2024-09-13T17:16:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153304 s, 68.4 MB/s 2024-09-13T17:16:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129389 s, 81.0 MB/s 2024-09-13T17:16:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119277 s, 87.9 MB/s 2024-09-13T17:16:49.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013099 s, 80.1 MB/s 2024-09-13T17:16:49.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.346 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013025 s, 80.5 MB/s 2024-09-13T17:16:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121114 s, 86.6 MB/s 2024-09-13T17:16:49.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160095 s, 65.5 MB/s 2024-09-13T17:16:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127366 s, 82.3 MB/s 2024-09-13T17:16:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013834 s, 75.8 MB/s 2024-09-13T17:16:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170795 s, 61.4 MB/s 2024-09-13T17:16:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166586 s, 62.9 MB/s 2024-09-13T17:16:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.465 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144016 s, 72.8 MB/s 2024-09-13T17:16:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012924 s, 81.1 MB/s 2024-09-13T17:16:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.498 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165656 s, 63.3 MB/s 2024-09-13T17:16:49.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134117 s, 78.2 MB/s 2024-09-13T17:16:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.530 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142334 s, 73.7 MB/s 2024-09-13T17:16:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187532 s, 55.9 MB/s 2024-09-13T17:16:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213299 s, 49.2 MB/s 2024-09-13T17:16:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142857 s, 73.4 MB/s 2024-09-13T17:16:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.614 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190147 s, 55.1 MB/s 2024-09-13T17:16:49.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:49.615 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.22 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:49.616 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136359 s, 76.9 MB/s 2024-09-13T17:16:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128599 s, 81.5 MB/s 2024-09-13T17:16:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130326 s, 80.5 MB/s 2024-09-13T17:16:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134982 s, 77.7 MB/s 2024-09-13T17:16:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014692 s, 71.4 MB/s 2024-09-13T17:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130437 s, 80.4 MB/s 2024-09-13T17:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154089 s, 68.1 MB/s 2024-09-13T17:16:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139422 s, 75.2 MB/s 2024-09-13T17:16:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.802 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169269 s, 61.9 MB/s 2024-09-13T17:16:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132452 s, 79.2 MB/s 2024-09-13T17:16:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134911 s, 77.7 MB/s 2024-09-13T17:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.852 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016405 s, 63.9 MB/s 2024-09-13T17:16:49.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166372 s, 63.0 MB/s 2024-09-13T17:16:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135104 s, 77.6 MB/s 2024-09-13T17:16:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.902 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132706 s, 79.0 MB/s 2024-09-13T17:16:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164249 s, 63.8 MB/s 2024-09-13T17:16:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164095 s, 63.9 MB/s 2024-09-13T17:16:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.955 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132035 s, 79.4 MB/s 2024-09-13T17:16:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170923 s, 61.3 MB/s 2024-09-13T17:16:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:49.989 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126444 s, 82.9 MB/s 2024-09-13T17:16:50.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134092 s, 78.2 MB/s 2024-09-13T17:16:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134557 s, 77.9 MB/s 2024-09-13T17:16:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166016 s, 63.2 MB/s 2024-09-13T17:16:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137104 s, 76.5 MB/s 2024-09-13T17:16:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.070 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130288 s, 80.5 MB/s 2024-09-13T17:16:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134291 s, 78.1 MB/s 2024-09-13T17:16:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.103 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144545 s, 72.5 MB/s 2024-09-13T17:16:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147286 s, 71.2 MB/s 2024-09-13T17:16:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133505 s, 78.5 MB/s 2024-09-13T17:16:50.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135506 s, 77.4 MB/s 2024-09-13T17:16:50.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160886 s, 65.2 MB/s 2024-09-13T17:16:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129064 s, 81.2 MB/s 2024-09-13T17:16:50.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:50.186 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.23 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:50.187 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170926 s, 61.3 MB/s 2024-09-13T17:16:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131616 s, 79.7 MB/s 2024-09-13T17:16:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013187 s, 79.5 MB/s 2024-09-13T17:16:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146504 s, 71.6 MB/s 2024-09-13T17:16:50.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.310 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138178 s, 75.9 MB/s 2024-09-13T17:16:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.326 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136882 s, 76.6 MB/s 2024-09-13T17:16:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138334 s, 75.8 MB/s 2024-09-13T17:16:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138029 s, 76.0 MB/s 2024-09-13T17:16:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129561 s, 80.9 MB/s 2024-09-13T17:16:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.392 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168206 s, 62.3 MB/s 2024-09-13T17:16:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012597 s, 83.2 MB/s 2024-09-13T17:16:50.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013647 s, 76.8 MB/s 2024-09-13T17:16:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131317 s, 79.9 MB/s 2024-09-13T17:16:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157742 s, 66.5 MB/s 2024-09-13T17:16:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126288 s, 83.0 MB/s 2024-09-13T17:16:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.490 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165607 s, 63.3 MB/s 2024-09-13T17:16:50.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.506 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138465 s, 75.7 MB/s 2024-09-13T17:16:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.522 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013736 s, 76.3 MB/s 2024-09-13T17:16:50.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.538 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138025 s, 76.0 MB/s 2024-09-13T17:16:50.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.552 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123157 s, 85.1 MB/s 2024-09-13T17:16:50.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.568 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133284 s, 78.7 MB/s 2024-09-13T17:16:50.582 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.583 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125587 s, 83.5 MB/s 2024-09-13T17:16:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.597 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122901 s, 85.3 MB/s 2024-09-13T17:16:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166338 s, 63.0 MB/s 2024-09-13T17:16:50.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133988 s, 78.3 MB/s 2024-09-13T17:16:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134587 s, 77.9 MB/s 2024-09-13T17:16:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163635 s, 64.1 MB/s 2024-09-13T17:16:50.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.682 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131312 s, 79.9 MB/s 2024-09-13T17:16:50.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129662 s, 80.9 MB/s 2024-09-13T17:16:50.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.713 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135048 s, 77.6 MB/s 2024-09-13T17:16:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136902 s, 76.6 MB/s 2024-09-13T17:16:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.745 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135549 s, 77.4 MB/s 2024-09-13T17:16:50.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:50.746 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:50.746 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:50.746 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.24 2024-09-13T17:16:50.747 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:50.747 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:50.747 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:50.747 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128749 s, 81.4 MB/s 2024-09-13T17:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.816 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131628 s, 79.7 MB/s 2024-09-13T17:16:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134057 s, 78.2 MB/s 2024-09-13T17:16:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130012 s, 80.7 MB/s 2024-09-13T17:16:50.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013172 s, 79.6 MB/s 2024-09-13T17:16:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133122 s, 78.8 MB/s 2024-09-13T17:16:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138383 s, 75.8 MB/s 2024-09-13T17:16:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137382 s, 76.3 MB/s 2024-09-13T17:16:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.930 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171774 s, 61.0 MB/s 2024-09-13T17:16:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.945 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129556 s, 80.9 MB/s 2024-09-13T17:16:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.961 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013238 s, 79.2 MB/s 2024-09-13T17:16:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131828 s, 79.5 MB/s 2024-09-13T17:16:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012618 s, 83.1 MB/s 2024-09-13T17:16:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129133 s, 81.2 MB/s 2024-09-13T17:16:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154872 s, 67.7 MB/s 2024-09-13T17:16:51.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016757 s, 62.6 MB/s 2024-09-13T17:16:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01373 s, 76.4 MB/s 2024-09-13T17:16:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132604 s, 79.1 MB/s 2024-09-13T17:16:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.090 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139109 s, 75.4 MB/s 2024-09-13T17:16:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141252 s, 74.2 MB/s 2024-09-13T17:16:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.123 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136684 s, 76.7 MB/s 2024-09-13T17:16:51.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134791 s, 77.8 MB/s 2024-09-13T17:16:51.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131848 s, 79.5 MB/s 2024-09-13T17:16:51.172 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.172 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.172 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158062 s, 66.3 MB/s 2024-09-13T17:16:51.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.191 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165363 s, 63.4 MB/s 2024-09-13T17:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163877 s, 64.0 MB/s 2024-09-13T17:16:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.228 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162426 s, 64.6 MB/s 2024-09-13T17:16:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132451 s, 79.2 MB/s 2024-09-13T17:16:51.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.262 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162022 s, 64.7 MB/s 2024-09-13T17:16:51.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.277 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132987 s, 78.8 MB/s 2024-09-13T17:16:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162461 s, 64.5 MB/s 2024-09-13T17:16:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135768 s, 77.2 MB/s 2024-09-13T17:16:51.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.25 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:51.313 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:51.367 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.367 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.367 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128512 s, 81.6 MB/s 2024-09-13T17:16:51.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.385 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157685 s, 66.5 MB/s 2024-09-13T17:16:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162901 s, 64.4 MB/s 2024-09-13T17:16:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138972 s, 75.5 MB/s 2024-09-13T17:16:51.435 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.436 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137221 s, 76.4 MB/s 2024-09-13T17:16:51.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136765 s, 76.7 MB/s 2024-09-13T17:16:51.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159974 s, 65.5 MB/s 2024-09-13T17:16:51.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137041 s, 76.5 MB/s 2024-09-13T17:16:51.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.501 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135707 s, 77.3 MB/s 2024-09-13T17:16:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161279 s, 65.0 MB/s 2024-09-13T17:16:51.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.539 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017066 s, 61.4 MB/s 2024-09-13T17:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015622 s, 67.1 MB/s 2024-09-13T17:16:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132579 s, 79.1 MB/s 2024-09-13T17:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127017 s, 82.6 MB/s 2024-09-13T17:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012619 s, 83.1 MB/s 2024-09-13T17:16:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.622 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172706 s, 60.7 MB/s 2024-09-13T17:16:51.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.640 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160777 s, 65.2 MB/s 2024-09-13T17:16:51.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132045 s, 79.4 MB/s 2024-09-13T17:16:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132643 s, 79.1 MB/s 2024-09-13T17:16:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158604 s, 66.1 MB/s 2024-09-13T17:16:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170909 s, 61.4 MB/s 2024-09-13T17:16:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013191 s, 79.5 MB/s 2024-09-13T17:16:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.738 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118297 s, 88.6 MB/s 2024-09-13T17:16:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131427 s, 79.8 MB/s 2024-09-13T17:16:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138395 s, 75.8 MB/s 2024-09-13T17:16:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165619 s, 63.3 MB/s 2024-09-13T17:16:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123885 s, 84.6 MB/s 2024-09-13T17:16:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160562 s, 65.3 MB/s 2024-09-13T17:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016093 s, 65.2 MB/s 2024-09-13T17:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.856 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140481 s, 74.6 MB/s 2024-09-13T17:16:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129189 s, 81.2 MB/s 2024-09-13T17:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131356 s, 79.8 MB/s 2024-09-13T17:16:51.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.26 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:51.888 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.945 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153497 s, 68.3 MB/s 2024-09-13T17:16:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135994 s, 77.1 MB/s 2024-09-13T17:16:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119786 s, 87.5 MB/s 2024-09-13T17:16:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:51.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135357 s, 77.5 MB/s 2024-09-13T17:16:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143106 s, 73.3 MB/s 2024-09-13T17:16:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170808 s, 61.4 MB/s 2024-09-13T17:16:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.041 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124686 s, 84.1 MB/s 2024-09-13T17:16:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127684 s, 82.1 MB/s 2024-09-13T17:16:52.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.071 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012368 s, 84.8 MB/s 2024-09-13T17:16:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.088 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152005 s, 69.0 MB/s 2024-09-13T17:16:52.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185639 s, 56.5 MB/s 2024-09-13T17:16:52.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128948 s, 81.3 MB/s 2024-09-13T17:16:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.141 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141383 s, 74.2 MB/s 2024-09-13T17:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.160 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163161 s, 64.3 MB/s 2024-09-13T17:16:52.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152701 s, 68.7 MB/s 2024-09-13T17:16:52.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131311 s, 79.9 MB/s 2024-09-13T17:16:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161862 s, 64.8 MB/s 2024-09-13T17:16:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116469 s, 90.0 MB/s 2024-09-13T17:16:52.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.240 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.240 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127026 s, 82.5 MB/s 2024-09-13T17:16:52.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129086 s, 81.2 MB/s 2024-09-13T17:16:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139898 s, 75.0 MB/s 2024-09-13T17:16:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122976 s, 85.3 MB/s 2024-09-13T17:16:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01673 s, 62.7 MB/s 2024-09-13T17:16:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165635 s, 63.3 MB/s 2024-09-13T17:16:52.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135471 s, 77.4 MB/s 2024-09-13T17:16:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.356 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013965 s, 75.1 MB/s 2024-09-13T17:16:52.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137322 s, 76.4 MB/s 2024-09-13T17:16:52.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013158 s, 79.7 MB/s 2024-09-13T17:16:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165165 s, 63.5 MB/s 2024-09-13T17:16:52.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135161 s, 77.6 MB/s 2024-09-13T17:16:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.435 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113043 s, 92.8 MB/s 2024-09-13T17:16:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.450 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130461 s, 80.4 MB/s 2024-09-13T17:16:52.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.27 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:52.452 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012948 s, 81.0 MB/s 2024-09-13T17:16:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.522 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129544 s, 80.9 MB/s 2024-09-13T17:16:52.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134293 s, 78.1 MB/s 2024-09-13T17:16:52.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.553 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131147 s, 80.0 MB/s 2024-09-13T17:16:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.569 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142613 s, 73.5 MB/s 2024-09-13T17:16:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122328 s, 85.7 MB/s 2024-09-13T17:16:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128754 s, 81.4 MB/s 2024-09-13T17:16:52.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.614 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.614 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013053 s, 80.3 MB/s 2024-09-13T17:16:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130554 s, 80.3 MB/s 2024-09-13T17:16:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133701 s, 78.4 MB/s 2024-09-13T17:16:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134628 s, 77.9 MB/s 2024-09-13T17:16:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133986 s, 78.3 MB/s 2024-09-13T17:16:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132835 s, 78.9 MB/s 2024-09-13T17:16:52.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.710 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155158 s, 67.6 MB/s 2024-09-13T17:16:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129031 s, 81.3 MB/s 2024-09-13T17:16:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.743 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159662 s, 65.7 MB/s 2024-09-13T17:16:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128167 s, 81.8 MB/s 2024-09-13T17:16:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129819 s, 80.8 MB/s 2024-09-13T17:16:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.789 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132781 s, 79.0 MB/s 2024-09-13T17:16:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.806 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155642 s, 67.4 MB/s 2024-09-13T17:16:52.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.822 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131481 s, 79.8 MB/s 2024-09-13T17:16:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.837 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132959 s, 78.9 MB/s 2024-09-13T17:16:52.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125999 s, 83.2 MB/s 2024-09-13T17:16:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014499 s, 72.3 MB/s 2024-09-13T17:16:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133327 s, 78.6 MB/s 2024-09-13T17:16:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134851 s, 77.8 MB/s 2024-09-13T17:16:52.919 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174441 s, 60.1 MB/s 2024-09-13T17:16:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135117 s, 77.6 MB/s 2024-09-13T17:16:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136825 s, 76.6 MB/s 2024-09-13T17:16:52.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.968 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.968 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139673 s, 75.1 MB/s 2024-09-13T17:16:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.983 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130273 s, 80.5 MB/s 2024-09-13T17:16:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:52.999 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014256 s, 73.6 MB/s 2024-09-13T17:16:53.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.28 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:53.001 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:53.002 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128999 s, 81.3 MB/s 2024-09-13T17:16:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139743 s, 75.0 MB/s 2024-09-13T17:16:53.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.087 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125971 s, 83.2 MB/s 2024-09-13T17:16:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152895 s, 68.6 MB/s 2024-09-13T17:16:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.122 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152849 s, 68.6 MB/s 2024-09-13T17:16:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.142 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175876 s, 59.6 MB/s 2024-09-13T17:16:53.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.161 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168034 s, 62.4 MB/s 2024-09-13T17:16:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183516 s, 57.1 MB/s 2024-09-13T17:16:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171304 s, 61.2 MB/s 2024-09-13T17:16:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136106 s, 77.0 MB/s 2024-09-13T17:16:53.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165235 s, 63.5 MB/s 2024-09-13T17:16:53.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158678 s, 66.1 MB/s 2024-09-13T17:16:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131172 s, 79.9 MB/s 2024-09-13T17:16:53.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133603 s, 78.5 MB/s 2024-09-13T17:16:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172849 s, 60.7 MB/s 2024-09-13T17:16:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.321 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140251 s, 74.8 MB/s 2024-09-13T17:16:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.335 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112798 s, 93.0 MB/s 2024-09-13T17:16:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133392 s, 78.6 MB/s 2024-09-13T17:16:53.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128926 s, 81.3 MB/s 2024-09-13T17:16:53.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130732 s, 80.2 MB/s 2024-09-13T17:16:53.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141548 s, 74.1 MB/s 2024-09-13T17:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.413 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136631 s, 76.7 MB/s 2024-09-13T17:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164163 s, 63.9 MB/s 2024-09-13T17:16:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.447 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131634 s, 79.7 MB/s 2024-09-13T17:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.464 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142687 s, 73.5 MB/s 2024-09-13T17:16:53.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137887 s, 76.0 MB/s 2024-09-13T17:16:53.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.498 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162862 s, 64.4 MB/s 2024-09-13T17:16:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.514 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134561 s, 77.9 MB/s 2024-09-13T17:16:53.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.529 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012909 s, 81.2 MB/s 2024-09-13T17:16:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153646 s, 68.2 MB/s 2024-09-13T17:16:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.562 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129403 s, 81.0 MB/s 2024-09-13T17:16:53.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.580 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159449 s, 65.8 MB/s 2024-09-13T17:16:53.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.29 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:53.582 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:53.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.637 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013093 s, 80.1 MB/s 2024-09-13T17:16:53.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.653 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143553 s, 73.0 MB/s 2024-09-13T17:16:53.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175432 s, 59.8 MB/s 2024-09-13T17:16:53.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.689 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136366 s, 76.9 MB/s 2024-09-13T17:16:53.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128962 s, 81.3 MB/s 2024-09-13T17:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129291 s, 81.1 MB/s 2024-09-13T17:16:53.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135594 s, 77.3 MB/s 2024-09-13T17:16:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132371 s, 79.2 MB/s 2024-09-13T17:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.765 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124998 s, 83.9 MB/s 2024-09-13T17:16:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.781 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135948 s, 77.1 MB/s 2024-09-13T17:16:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118218 s, 88.7 MB/s 2024-09-13T17:16:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136904 s, 76.6 MB/s 2024-09-13T17:16:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114515 s, 91.6 MB/s 2024-09-13T17:16:53.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.840 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134038 s, 78.2 MB/s 2024-09-13T17:16:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126019 s, 83.2 MB/s 2024-09-13T17:16:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136016 s, 77.1 MB/s 2024-09-13T17:16:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134507 s, 78.0 MB/s 2024-09-13T17:16:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118179 s, 88.7 MB/s 2024-09-13T17:16:53.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132174 s, 79.3 MB/s 2024-09-13T17:16:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136345 s, 76.9 MB/s 2024-09-13T17:16:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158135 s, 66.3 MB/s 2024-09-13T17:16:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.963 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011774 s, 89.1 MB/s 2024-09-13T17:16:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142289 s, 73.7 MB/s 2024-09-13T17:16:53.995 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:53.995 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:53.995 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129302 s, 81.1 MB/s 2024-09-13T17:16:54.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.010 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127268 s, 82.4 MB/s 2024-09-13T17:16:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129933 s, 80.7 MB/s 2024-09-13T17:16:54.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132113 s, 79.4 MB/s 2024-09-13T17:16:54.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162472 s, 64.5 MB/s 2024-09-13T17:16:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125003 s, 83.9 MB/s 2024-09-13T17:16:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.091 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155764 s, 67.3 MB/s 2024-09-13T17:16:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.108 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145014 s, 72.3 MB/s 2024-09-13T17:16:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156288 s, 67.1 MB/s 2024-09-13T17:16:54.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.30 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:54.128 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012916 s, 81.2 MB/s 2024-09-13T17:16:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169762 s, 61.8 MB/s 2024-09-13T17:16:54.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140969 s, 74.4 MB/s 2024-09-13T17:16:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013266 s, 79.0 MB/s 2024-09-13T17:16:54.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.249 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142132 s, 73.8 MB/s 2024-09-13T17:16:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140603 s, 74.6 MB/s 2024-09-13T17:16:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122955 s, 85.3 MB/s 2024-09-13T17:16:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129798 s, 80.8 MB/s 2024-09-13T17:16:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.311 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134858 s, 77.8 MB/s 2024-09-13T17:16:54.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.327 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133538 s, 78.5 MB/s 2024-09-13T17:16:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132522 s, 79.1 MB/s 2024-09-13T17:16:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.360 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156765 s, 66.9 MB/s 2024-09-13T17:16:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.377 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149103 s, 70.3 MB/s 2024-09-13T17:16:54.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140424 s, 74.7 MB/s 2024-09-13T17:16:54.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167715 s, 62.5 MB/s 2024-09-13T17:16:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.428 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138011 s, 76.0 MB/s 2024-09-13T17:16:54.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132184 s, 79.3 MB/s 2024-09-13T17:16:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.459 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132684 s, 79.0 MB/s 2024-09-13T17:16:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.475 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133724 s, 78.4 MB/s 2024-09-13T17:16:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135499 s, 77.4 MB/s 2024-09-13T17:16:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013959 s, 75.1 MB/s 2024-09-13T17:16:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114115 s, 91.9 MB/s 2024-09-13T17:16:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.536 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133258 s, 78.7 MB/s 2024-09-13T17:16:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.556 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166794 s, 62.9 MB/s 2024-09-13T17:16:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.569 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106169 s, 98.8 MB/s 2024-09-13T17:16:54.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160164 s, 65.5 MB/s 2024-09-13T17:16:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143965 s, 72.8 MB/s 2024-09-13T17:16:54.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.622 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155983 s, 67.2 MB/s 2024-09-13T17:16:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160186 s, 65.5 MB/s 2024-09-13T17:16:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132545 s, 79.1 MB/s 2024-09-13T17:16:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132029 s, 79.4 MB/s 2024-09-13T17:16:54.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166884 s, 62.8 MB/s 2024-09-13T17:16:54.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.31 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:16:54.692 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:16:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153742 s, 68.2 MB/s 2024-09-13T17:16:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.767 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163303 s, 64.2 MB/s 2024-09-13T17:16:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.784 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014011 s, 74.8 MB/s 2024-09-13T17:16:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01336 s, 78.5 MB/s 2024-09-13T17:16:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128883 s, 81.4 MB/s 2024-09-13T17:16:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134116 s, 78.2 MB/s 2024-09-13T17:16:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.845 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136699 s, 76.7 MB/s 2024-09-13T17:16:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133013 s, 78.8 MB/s 2024-09-13T17:16:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134853 s, 77.8 MB/s 2024-09-13T17:16:54.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.892 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128191 s, 81.8 MB/s 2024-09-13T17:16:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135657 s, 77.3 MB/s 2024-09-13T17:16:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.924 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138658 s, 75.6 MB/s 2024-09-13T17:16:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131068 s, 80.0 MB/s 2024-09-13T17:16:54.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143662 s, 73.0 MB/s 2024-09-13T17:16:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133385 s, 78.6 MB/s 2024-09-13T17:16:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131836 s, 79.5 MB/s 2024-09-13T17:16:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164962 s, 63.6 MB/s 2024-09-13T17:16:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161241 s, 65.0 MB/s 2024-09-13T17:16:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139175 s, 75.3 MB/s 2024-09-13T17:16:55.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136422 s, 76.9 MB/s 2024-09-13T17:16:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158743 s, 66.1 MB/s 2024-09-13T17:16:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123224 s, 85.1 MB/s 2024-09-13T17:16:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132805 s, 79.0 MB/s 2024-09-13T17:16:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.119 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129593 s, 80.9 MB/s 2024-09-13T17:16:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167665 s, 62.5 MB/s 2024-09-13T17:16:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156535 s, 67.0 MB/s 2024-09-13T17:16:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129798 s, 80.8 MB/s 2024-09-13T17:16:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135242 s, 77.5 MB/s 2024-09-13T17:16:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138277 s, 75.8 MB/s 2024-09-13T17:16:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170887 s, 61.4 MB/s 2024-09-13T17:16:55.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118296 s, 88.6 MB/s 2024-09-13T17:16:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:16:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:16:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129401 s, 81.0 MB/s 2024-09-13T17:16:55.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 scrub start /scrub_control_test_path recursive 2024-09-13T17:16:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.412+0000 7f2eaf200640 1 Processor -- start 2024-09-13T17:16:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.412+0000 7f2eaf200640 1 -- start start 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.413+0000 7f2eaf200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea8070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.413+0000 7f2eaf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ea8159970 0x7f2ea80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.413+0000 7f2eaf200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea815a4c0 con 0x7f2ea80b2630 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.413+0000 7f2eaf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea80b2ae0 con 0x7f2ea8159970 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.413+0000 7f2eaf200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8071580 con 0x7f2ea8159590 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ea8159970 0x7f2ea80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2ea8159970 0x7f2ea80b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49340/0 (socket says 172.21.15.28:49340) 2024-09-13T17:16:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 -- 172.21.15.28:0/2568850654 learned_addr learned my addr 172.21.15.28:0/2568850654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea8070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 -- 172.21.15.28:0/2568850654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 msgr2=0x7f2ea8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea8070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 -- 172.21.15.28:0/2568850654 >> v1:172.21.15.189:6789/0 conn(0x7f2ea80b2630 legacy=0x7f2ea806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.414+0000 7f2ea7400640 1 -- 172.21.15.28:0/2568850654 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8159e70 con 0x7f2ea8159970 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.415+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea80b20f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400bb10 tx=0x7f2e9402c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a4e1c75934690c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2ea6a00640 1 -- 172.21.15.28:0/2568850654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94032820 con 0x7f2ea8159970 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2ea6a00640 1 -- 172.21.15.28:0/2568850654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9402e040 con 0x7f2ea8159970 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2ea6a00640 1 -- 172.21.15.28:0/2568850654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94032ba0 con 0x7f2ea8159970 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 -- 172.21.15.28:0/2568850654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 msgr2=0x7f2ea80b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea80b20f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400bb10 tx=0x7f2e9402c920 comp rx=0 tx=0).stop 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 -- 172.21.15.28:0/2568850654 shutdown_connections 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea80b20f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 --2- 172.21.15.28:0/2568850654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea8070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 -- 172.21.15.28:0/2568850654 >> 172.21.15.28:0/2568850654 conn(0x7f2ea8069e80 msgr2=0x7f2ea806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 -- 172.21.15.28:0/2568850654 shutdown_connections 2024-09-13T17:16:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.416+0000 7f2eaf200640 1 -- 172.21.15.28:0/2568850654 wait complete. 2024-09-13T17:16:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.418+0000 7f2eaf200640 1 Processor -- start 2024-09-13T17:16:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.418+0000 7f2eaf200640 1 -- start start 2024-09-13T17:16:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea80b2630 0x7f2ea8154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea81524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8152020 con 0x7f2ea8159970 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8151190 con 0x7f2ea80b2630 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2eaf200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8151410 con 0x7f2ea8159590 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2ead200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2ea7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea81524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea80b2630 0x7f2ea8154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2ead200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49350/0 (socket says 172.21.15.28:49350) 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.419+0000 7f2ead200640 1 -- 172.21.15.28:0/1760455248 learned_addr learned my addr 172.21.15.28:0/1760455248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 -- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 msgr2=0x7f2ea81524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea81524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 -- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea80b2630 msgr2=0x7f2ea8154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea80b2630 0x7f2ea8154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 -- 172.21.15.28:0/1760455248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9400b770 con 0x7f2ea8159970 2024-09-13T17:16:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2ead200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c7d0 tx=0x7f2ea000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2e9fe00640 1 -- 172.21.15.28:0/1760455248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0003b70 con 0x7f2ea8159970 2024-09-13T17:16:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2e9fe00640 1 -- 172.21.15.28:0/1760455248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea0003cf0 con 0x7f2ea8159970 2024-09-13T17:16:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2e9fe00640 1 -- 172.21.15.28:0/1760455248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea000aa90 con 0x7f2ea8159970 2024-09-13T17:16:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.420+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea81516f0 con 0x7f2ea8159970 2024-09-13T17:16:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.421+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea8151cb0 con 0x7f2ea8159970 2024-09-13T17:16:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.423+0000 7f2e9fe00640 1 -- 172.21.15.28:0/1760455248 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ea000abf0 con 0x7f2ea8159970 2024-09-13T17:16:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.423+0000 7f2e9fe00640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 0x7f2e8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.423+0000 7f2e9fe00640 1 -- 172.21.15.28:0/1760455248 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2ea0089060 con 0x7f2ea8159970 2024-09-13T17:16:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.424+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 0x7f2e8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.424+0000 7f2eb1fba740 1 Processor -- start 2024-09-13T17:16:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.424+0000 7f2eb1fba740 1 -- start start 2024-09-13T17:16:55.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.424+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 0x7f2e8806d0c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006fd0 tx=0x7f2e98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 0x5573c38af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 0x5573c38d8f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c3818f00 con 0x5573c38b0cb0 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c3816690 con 0x5573c38f7dc0 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573c3811f80 con 0x5573c38af3f0 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2ead200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2ead200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49364/0 (socket says 172.21.15.28:49364) 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2ead200640 1 -- 172.21.15.28:0/306607024 learned_addr learned my addr 172.21.15.28:0/306607024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.425+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 0x5573c38af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ea7400640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 0x5573c38d8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 -- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 msgr2=0x5573c38af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 0x5573c38af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 -- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 msgr2=0x5573c38d8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 0x5573c38d8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 -- 172.21.15.28:0/306607024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573c37dc390 con 0x5573c38b0cb0 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ea7400640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 0x5573c38d8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 0x5573c38af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2ead200640 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00874f0 tx=0x7f2ea0057ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49301a48c55c0420 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2e9cc00640 1 -- 172.21.15.28:0/306607024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea005bcf0 con 0x5573c38b0cb0 2024-09-13T17:16:55.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2e9cc00640 1 -- 172.21.15.28:0/306607024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00602b0 con 0x5573c38b0cb0 2024-09-13T17:16:55.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.426+0000 7f2e9cc00640 1 -- 172.21.15.28:0/306607024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0054680 con 0x5573c38b0cb0 2024-09-13T17:16:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 -- 172.21.15.28:0/306607024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 msgr2=0x5573c38d9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00874f0 tx=0x7f2ea0057ab0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 -- 172.21.15.28:0/306607024 shutdown_connections 2024-09-13T17:16:55.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38d9460 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38f7dc0 0x5573c38d8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/306607024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38af3f0 0x5573c38af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 -- 172.21.15.28:0/306607024 >> 172.21.15.28:0/306607024 conn(0x5573c38aba90 msgr2=0x5573c38f74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 -- 172.21.15.28:0/306607024 shutdown_connections 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.427+0000 7f2eb1fba740 1 -- 172.21.15.28:0/306607024 wait complete. 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.428+0000 7f2eb1fba740 1 -- start start 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.428+0000 7f2eb1fba740 1 Processor -- start 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38d8b00 0x5573c38ec510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c37d02c0 con 0x5573c38b0cb0 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c37cff60 con 0x5573c38ef6d0 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573c35793e0 con 0x5573c38d8b00 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2eb1fba740 10 client.0 _ll_register_callbacks cb 0x5573c37e6db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49368/0 (socket says 172.21.15.28:49368) 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2912701634 learned_addr learned my addr 172.21.15.28:0/2912701634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38d8b00 msgr2=0x5573c38ec510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38d8b00 0x5573c38ec510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 msgr2=0x5573c38a9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.429+0000 7f2ea7e00640 1 -- 172.21.15.28:0/2912701634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea8151cb0 con 0x5573c38b0cb0 2024-09-13T17:16:55.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f2e98016a00 tx=0x7f2e98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d074fa0a9dc70cd9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2e91600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98010e60 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e98014bc0 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5573c383aab0 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e98014d80 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2eb1fba740 10 client.8628 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x5573c3810600 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.430+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5573c383aab0 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2e98022410 con 0x5573c38b0cb0 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 10 client.8628 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 10 client.8628 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 20 client.8628 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c38aa4d0 0x5573c38d5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 4 client.8628 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573c3803320 con 0x5573c38aa4d0 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c38aa4d0 0x5573c38d5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c38aa4d0 0x5573c38d5130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af683a310a1746 server_cookie=f463e068c0b8662c in_seq=0 out_seq=0 2024-09-13T17:16:55.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.431+0000 7f2e91600640 10 client.8628 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2e91600640 1 -- 172.21.15.28:0/2912701634 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5573c3803320 con 0x5573c38aa4d0 2024-09-13T17:16:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2e91600640 10 client.8628 handle_command_reply: tid=0 2024-09-13T17:16:55.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 client.8628 shutdown 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 msgr2=0x5573c38eab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f2e98016a00 tx=0x7f2e98004730 comp rx=0 tx=0).stop 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 shutdown_connections 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c38aa4d0 0x5573c38d5130 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38a9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38d8b00 0x5573c38ec510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2912701634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38b0cb0 0x5573c38eab60 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2e91600640 0 client.8628 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.432+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 >> 172.21.15.28:0/2912701634 conn(0x5573c38af3f0 msgr2=0x5573c38b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.433+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 shutdown_connections 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.433+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2912701634 wait complete. 2024-09-13T17:16:55.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.433+0000 7f2eb1fba740 20 client.8628 trim_cache size 0 max 16384 2024-09-13T17:16:55.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.448+0000 7f2eb1fba740 1 Processor -- start 2024-09-13T17:16:55.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.448+0000 7f2eb1fba740 1 -- start start 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38f5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c38ab560 con 0x5573c38d8b00 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c38480a0 con 0x5573c38ef6d0 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573c38a5c70 con 0x5573c38b0cb0 2024-09-13T17:16:55.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49370/0 (socket says 172.21.15.28:49370) 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 -- 172.21.15.28:0/3278959704 learned_addr learned my addr 172.21.15.28:0/3278959704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 -- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 msgr2=0x5573c38ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ead200640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 -- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 msgr2=0x5573c38f5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.449+0000 7f2ea7400640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38f5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2ea7400640 1 -- 172.21.15.28:0/3278959704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573c37b2400 con 0x5573c38d8b00 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2ead200640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2ea7400640 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f2e94035040 tx=0x7f2e94031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a0f9256e06ea85 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2e9d600640 1 -- 172.21.15.28:0/3278959704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e940042c0 con 0x5573c38d8b00 2024-09-13T17:16:55.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2e9d600640 1 -- 172.21.15.28:0/3278959704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e94004420 con 0x5573c38d8b00 2024-09-13T17:16:55.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.450+0000 7f2e9d600640 1 -- 172.21.15.28:0/3278959704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94044610 con 0x5573c38d8b00 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- 172.21.15.28:0/3278959704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 msgr2=0x5573c38d9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f2e94035040 tx=0x7f2e94031000 comp rx=0 tx=0).stop 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- 172.21.15.28:0/3278959704 shutdown_connections 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38f5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38d9380 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/3278959704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- 172.21.15.28:0/3278959704 >> 172.21.15.28:0/3278959704 conn(0x5573c38af3f0 msgr2=0x5573c38f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- 172.21.15.28:0/3278959704 shutdown_connections 2024-09-13T17:16:55.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- 172.21.15.28:0/3278959704 wait complete. 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.451+0000 7f2eb1fba740 1 -- start start 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.452+0000 7f2eb1fba740 1 Processor -- start 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.452+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.452+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.452+0000 7f2eb1fba740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38efb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c38f0410 con 0x5573c38d8b00 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573c395f000 con 0x5573c38ef6d0 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2eb1fba740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5573c395ed10 con 0x5573c38b0cb0 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2eb1fba740 10 client.0 _ll_register_callbacks cb 0x7f2ea005e0d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49378/0 (socket says 172.21.15.28:49378) 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 -- 172.21.15.28:0/2778397686 learned_addr learned my addr 172.21.15.28:0/2778397686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ead200640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38efb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 -- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 msgr2=0x5573c38ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 -- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 msgr2=0x5573c38efb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38efb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 -- 172.21.15.28:0/2778397686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea8159e70 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2ea7400640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400bb10 tx=0x7f2e94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ed82154c39f767 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.453+0000 7f2e91600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94044610 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e94040040 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ea00602b0 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e94044770 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 10 client.8634 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x5573c37d79d0 con 0x5573c38d8b00 2024-09-13T17:16:55.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea00602b0 con 0x5573c38d8b00 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2e94032810 con 0x5573c38d8b00 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 10 client.8634 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 10 client.8634 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 20 client.8634 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c3948260 0x5573c395f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 4 client.8634 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.454+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x5573c3505880 con 0x5573c3948260 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.455+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c3948260 0x5573c395f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.455+0000 7f2ea7e00640 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c3948260 0x5573c395f490 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4564f92cbadffaa server_cookie=5b1a94cb6fd4895f in_seq=0 out_seq=0 2024-09-13T17:16:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.455+0000 7f2e91600640 10 client.8634 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2e91600640 1 -- 172.21.15.28:0/2778397686 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5573c3505880 con 0x5573c3948260 2024-09-13T17:16:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2e91600640 10 client.8634 handle_command_reply: tid=0 2024-09-13T17:16:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 client.8634 shutdown 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 msgr2=0x5573c38f5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400bb10 tx=0x7f2e94004730 comp rx=0 tx=0).stop 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 shutdown_connections 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x5573c3948260 0x5573c395f490 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5573c38ef6d0 0x5573c38efb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5573c38d8b00 0x5573c38f5110 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 --2- 172.21.15.28:0/2778397686 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5573c38b0cb0 0x5573c38ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2e91600640 0 client.8634 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.459+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 >> 172.21.15.28:0/2778397686 conn(0x5573c38af3f0 msgr2=0x5573c38f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 shutdown_connections 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eb1fba740 1 -- 172.21.15.28:0/2778397686 wait complete. 2024-09-13T17:16:55.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eb1fba740 20 client.8634 trim_cache size 0 max 16384 2024-09-13T17:16:55.463 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:16:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "623cff98-4878-4926-9158-b986833eef1f", 2024-09-13T17:16:55.463 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:16:55.463 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 msgr2=0x7f2e8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 0x7f2e8806d0c0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f2e98006fd0 tx=0x7f2e98008040 comp rx=0 tx=0).stop 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 msgr2=0x7f2ea813f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.460+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c7d0 tx=0x7f2ea000cc90 comp rx=0 tx=0).stop 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 shutdown_connections 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e8806ac00 0x7f2e8806d0c0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ea8159970 0x7f2ea813f960 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ea8159590 0x7f2ea81524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 --2- 172.21.15.28:0/1760455248 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ea80b2630 0x7f2ea8154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 >> 172.21.15.28:0/1760455248 conn(0x7f2ea8069e80 msgr2=0x7f2ea806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 shutdown_connections 2024-09-13T17:16:55.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.461+0000 7f2eaf200640 1 -- 172.21.15.28:0/1760455248 wait complete. 2024-09-13T17:16:55.479 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 scrub abort 2024-09-13T17:16:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.673+0000 7ff662e00640 1 Processor -- start 2024-09-13T17:16:55.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.673+0000 7ff662e00640 1 -- start start 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff65c159770 0x7ff65c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c06f3f0 0x7ff65c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff65c0add00 0x7ff65c159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65c06f920 con 0x7ff65c0add00 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65c15b290 con 0x7ff65c159770 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.674+0000 7ff662e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c15b570 con 0x7ff65c06f3f0 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff65c0add00 0x7ff65c159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff65c0add00 0x7ff65c159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49386/0 (socket says 172.21.15.28:49386) 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 -- 172.21.15.28:0/2307968330 learned_addr learned my addr 172.21.15.28:0/2307968330 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff65b400640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c06f3f0 0x7ff65c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff65be00640 1 --2- 172.21.15.28:0/2307968330 >> v2:172.21.15.189:3300/0 conn(0x7ff65c159770 0x7ff65c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 -- 172.21.15.28:0/2307968330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c06f3f0 msgr2=0x7ff65c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c06f3f0 0x7ff65c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 -- 172.21.15.28:0/2307968330 >> v2:172.21.15.189:3300/0 conn(0x7ff65c159770 msgr2=0x7ff65c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 --2- 172.21.15.28:0/2307968330 >> v2:172.21.15.189:3300/0 conn(0x7ff65c159770 0x7ff65c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.675+0000 7ff660e00640 1 -- 172.21.15.28:0/2307968330 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c15b850 con 0x7ff65c0add00 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.676+0000 7ff660e00640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c0add00 0x7ff65c159b50 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff65c067f80 tx=0x7ff65402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0eb81f567979841 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.676+0000 7ff65aa00640 1 -- 172.21.15.28:0/2307968330 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654031600 con 0x7ff65c0add00 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.676+0000 7ff65aa00640 1 -- 172.21.15.28:0/2307968330 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65402d050 con 0x7ff65c0add00 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.676+0000 7ff65aa00640 1 -- 172.21.15.28:0/2307968330 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654031980 con 0x7ff65c0add00 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 -- 172.21.15.28:0/2307968330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c0add00 msgr2=0x7ff65c159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c0add00 0x7ff65c159b50 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7ff65c067f80 tx=0x7ff65402a610 comp rx=0 tx=0).stop 2024-09-13T17:16:55.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 -- 172.21.15.28:0/2307968330 shutdown_connections 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c0add00 0x7ff65c159b50 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 --2- 172.21.15.28:0/2307968330 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c06f3f0 0x7ff65c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 --2- 172.21.15.28:0/2307968330 >> v2:172.21.15.189:3300/0 conn(0x7ff65c159770 0x7ff65c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 -- 172.21.15.28:0/2307968330 >> 172.21.15.28:0/2307968330 conn(0x7ff65c0b0270 msgr2=0x7ff65c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 -- 172.21.15.28:0/2307968330 shutdown_connections 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.677+0000 7ff662e00640 1 -- 172.21.15.28:0/2307968330 wait complete. 2024-09-13T17:16:55.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.679+0000 7ff662e00640 1 Processor -- start 2024-09-13T17:16:55.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.679+0000 7ff662e00640 1 -- start start 2024-09-13T17:16:55.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff65c06f3f0 0x7ff65c0aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c0add00 0x7ff65c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c1e58a0 con 0x7ff65c159770 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c1e6420 con 0x7ff65c06f3f0 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff662e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c0cbee0 con 0x7ff65c0add00 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49400/0 (socket says 172.21.15.28:49400) 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 -- 172.21.15.28:0/679346833 learned_addr learned my addr 172.21.15.28:0/679346833 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff65b400640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c0add00 0x7ff65c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff65be00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff65c06f3f0 0x7ff65c0aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 -- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c0add00 msgr2=0x7ff65c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c0add00 0x7ff65c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 -- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff65c06f3f0 msgr2=0x7ff65c0aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff65c06f3f0 0x7ff65c0aa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.680+0000 7ff660e00640 1 -- 172.21.15.28:0/679346833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff654009b30 con 0x7ff65c159770 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff660e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff654009f40 tx=0x7ff654004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff653e00640 1 -- 172.21.15.28:0/679346833 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65403d860 con 0x7ff65c159770 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff653e00640 1 -- 172.21.15.28:0/679346833 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff65403d9e0 con 0x7ff65c159770 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff653e00640 1 -- 172.21.15.28:0/679346833 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff65403dd90 con 0x7ff65c159770 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c1e5af0 con 0x7ff65c159770 2024-09-13T17:16:55.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.681+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c1e60b0 con 0x7ff65c159770 2024-09-13T17:16:55.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.683+0000 7ff653e00640 1 -- 172.21.15.28:0/679346833 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff65403b400 con 0x7ff65c159770 2024-09-13T17:16:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.684+0000 7ff653e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 0x7ff63c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.684+0000 7ff653e00640 1 -- 172.21.15.28:0/679346833 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6540aa1c0 con 0x7ff65c159770 2024-09-13T17:16:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.684+0000 7ff65be00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 0x7ff63c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.685+0000 7ff665dc8740 1 Processor -- start 2024-09-13T17:16:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.685+0000 7ff665dc8740 1 -- start start 2024-09-13T17:16:55.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.685+0000 7ff65be00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 0x7ff63c06d0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff64c007920 tx=0x7ff64c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf03556760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf035a3d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf03461000 con 0x55cf03553b40 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf034bed40 con 0x55cf0357d830 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf034bc4d0 con 0x55cf03556380 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49406/0 (socket says 172.21.15.28:49406) 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 -- 172.21.15.28:0/1630904945 learned_addr learned my addr 172.21.15.28:0/1630904945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65be00640 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf03556760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff660e00640 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf035a3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 -- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 msgr2=0x55cf03556760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf03556760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 -- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 msgr2=0x55cf035a3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf035a3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.686+0000 7ff65b400640 1 -- 172.21.15.28:0/1630904945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf034733a0 con 0x55cf03553b40 2024-09-13T17:16:55.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff65b400640 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff648006fd0 tx=0x7ff648007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed725982a6df6ceb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff650c00640 1 -- 172.21.15.28:0/1630904945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648004240 con 0x55cf03553b40 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff650c00640 1 -- 172.21.15.28:0/1630904945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6480043a0 con 0x55cf03553b40 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff650c00640 1 -- 172.21.15.28:0/1630904945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648010a40 con 0x55cf03553b40 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff665dc8740 1 -- 172.21.15.28:0/1630904945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 msgr2=0x55cf035a3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.687+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7ff648006fd0 tx=0x7ff648007d60 comp rx=0 tx=0).stop 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 -- 172.21.15.28:0/1630904945 shutdown_connections 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf035a3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03553b40 0x55cf035a3850 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1630904945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf03556760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 -- 172.21.15.28:0/1630904945 >> 172.21.15.28:0/1630904945 conn(0x55cf03552ab0 msgr2=0x55cf0359af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 -- 172.21.15.28:0/1630904945 shutdown_connections 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 -- 172.21.15.28:0/1630904945 wait complete. 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 -- start start 2024-09-13T17:16:55.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.688+0000 7ff665dc8740 1 Processor -- start 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf034b7dc0 con 0x55cf03556380 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf034763e0 con 0x55cf0357d830 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf03476080 con 0x55cf0357e9f0 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.689+0000 7ff665dc8740 10 client.0 _ll_register_callbacks cb 0x55cf034b1430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49414/0 (socket says 172.21.15.28:49414) 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 -- 172.21.15.28:0/1066716436 learned_addr learned my addr 172.21.15.28:0/1066716436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65b400640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff660e00640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 -- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 msgr2=0x55cf0357ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 -- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 msgr2=0x55cf0357d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 -- 172.21.15.28:0/1066716436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64800e040 con 0x55cf03556380 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff660e00640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff65be00640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff64c008010 tx=0x7ff64c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e6e83f4999b20d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.690+0000 7ff645600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c0149b0 con 0x55cf03556380 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff64c014b10 con 0x55cf03556380 2024-09-13T17:16:55.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf034e1600 con 0x55cf03556380 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff64c014cd0 con 0x55cf03556380 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 10 client.8646 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x55cf034b6440 con 0x55cf03556380 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cf034e1600 con 0x55cf03556380 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff64c02b440 con 0x55cf03556380 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 10 client.8646 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 10 client.8646 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 20 client.8646 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf035a0cc0 0x55cf035a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 4 client.8646 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.691+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf034a9c90 con 0x55cf035a0cc0 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.692+0000 7ff65b400640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf035a0cc0 0x55cf035a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.692+0000 7ff65b400640 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf035a0cc0 0x55cf035a10a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ec17d0946eb58a1 server_cookie=fd8940410591abfd in_seq=0 out_seq=0 2024-09-13T17:16:55.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.692+0000 7ff645600640 10 client.8646 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff645600640 1 -- 172.21.15.28:0/1066716436 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55cf034a9c90 con 0x55cf035a0cc0 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff645600640 10 client.8646 handle_command_reply: tid=0 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 client.8646 shutdown 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 msgr2=0x55cf035a2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7ff64c008010 tx=0x7ff64c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 shutdown_connections 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf035a0cc0 0x55cf035a10a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff645600640 0 client.8646 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1066716436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035a2d60 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 >> 172.21.15.28:0/1066716436 conn(0x55cf03553b40 msgr2=0x55cf0359a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 shutdown_connections 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.693+0000 7ff665dc8740 1 -- 172.21.15.28:0/1066716436 wait complete. 2024-09-13T17:16:55.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.694+0000 7ff665dc8740 20 client.8646 trim_cache size 0 max 16384 2024-09-13T17:16:55.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.708+0000 7ff665dc8740 1 Processor -- start 2024-09-13T17:16:55.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 -- start start 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf0321f040 con 0x55cf03556380 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf034583a0 con 0x55cf0357d830 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.709+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf035bc5f0 con 0x55cf0357e9f0 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49422/0 (socket says 172.21.15.28:49422) 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 -- 172.21.15.28:0/239601843 learned_addr learned my addr 172.21.15.28:0/239601843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff660e00640 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65b400640 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 -- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 msgr2=0x55cf0357dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 -- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 msgr2=0x55cf0357a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 -- 172.21.15.28:0/239601843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf034585d0 con 0x55cf03556380 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.710+0000 7ff65be00640 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff64c000c00 tx=0x7ff64c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29ea802f1d93057 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff651600640 1 -- 172.21.15.28:0/239601843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c02b440 con 0x55cf03556380 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff651600640 1 -- 172.21.15.28:0/239601843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c02b5a0 con 0x55cf03556380 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff651600640 1 -- 172.21.15.28:0/239601843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c014800 con 0x55cf03556380 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 -- 172.21.15.28:0/239601843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 msgr2=0x55cf035984a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7ff64c000c00 tx=0x7ff64c023000 comp rx=0 tx=0).stop 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 -- 172.21.15.28:0/239601843 shutdown_connections 2024-09-13T17:16:55.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf0357e9f0 0x55cf0357dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357d830 0x55cf0357a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 --2- 172.21.15.28:0/239601843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf03556380 0x55cf035984a0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 -- 172.21.15.28:0/239601843 >> 172.21.15.28:0/239601843 conn(0x55cf03553b40 msgr2=0x55cf03582170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 -- 172.21.15.28:0/239601843 shutdown_connections 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.711+0000 7ff665dc8740 1 -- 172.21.15.28:0/239601843 wait complete. 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.712+0000 7ff665dc8740 1 -- start start 2024-09-13T17:16:55.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.712+0000 7ff665dc8740 1 Processor -- start 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf035eaf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357e9f0 0x55cf0357c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf035ec990 con 0x55cf0357d830 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf035b5fc0 con 0x55cf0357e9f0 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf03555300 con 0x55cf03556380 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff665dc8740 10 client.0 _ll_register_callbacks cb 0x7ff64c007650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff65b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49424/0 (socket says 172.21.15.28:49424) 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff65b400640 1 -- 172.21.15.28:0/1950859663 learned_addr learned my addr 172.21.15.28:0/1950859663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff65be00640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf035eaf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.713+0000 7ff660e00640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357e9f0 0x55cf0357c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 -- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 msgr2=0x55cf035eaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf035eaf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 -- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357e9f0 msgr2=0x55cf0357c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357e9f0 0x55cf0357c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 -- 172.21.15.28:0/1950859663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf034b6440 con 0x55cf0357d830 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff65b400640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ff648016000 tx=0x7ff648004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1fe53bd18658ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff645600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff648024040 con 0x55cf0357d830 2024-09-13T17:16:55.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6480295b0 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.714+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf031ab4e0 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff648029770 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 10 client.8652 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x55cf0347d9b0 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cf031ab4e0 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff6480108a0 con 0x55cf0357d830 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 10 client.8652 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 10 client.8652 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 20 client.8652 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf0357c920 0x55cf035eb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 4 client.8652 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "scrub abort"}) v1 -- 0x55cf031bc7e0 con 0x55cf0357c920 2024-09-13T17:16:55.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.715+0000 7ff65be00640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf0357c920 0x55cf035eb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff65be00640 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf0357c920 0x55cf035eb760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53d3c9b19b6f7e64 server_cookie=27f1a90c355e37b in_seq=0 out_seq=0 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff645600640 10 client.8652 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff645600640 1 -- 172.21.15.28:0/1950859663 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x55cf031bc7e0 con 0x55cf0357c920 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff645600640 10 client.8652 handle_command_reply: tid=0 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 client.8652 shutdown 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 msgr2=0x55cf0359ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7ff648016000 tx=0x7ff648004730 comp rx=0 tx=0).stop 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 shutdown_connections 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x55cf0357c920 0x55cf035eb760 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf0357e9f0 0x55cf0357c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf0357d830 0x55cf0359ad50 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff645600640 0 client.8652 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 --2- 172.21.15.28:0/1950859663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf03556380 0x55cf035eaf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.716+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 >> 172.21.15.28:0/1950859663 conn(0x55cf03553b40 msgr2=0x55cf035984a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 shutdown_connections 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff665dc8740 1 -- 172.21.15.28:0/1950859663 wait complete. 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff665dc8740 20 client.8652 trim_cache size 0 max 16384 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:16:55.722 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 msgr2=0x7ff63c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 0x7ff63c06d0c0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff64c007920 tx=0x7ff64c008040 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 msgr2=0x7ff65c0ab140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7ff654009f40 tx=0x7ff654004a40 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 shutdown_connections 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff63c06ac00 0x7ff63c06d0c0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff65c159770 0x7ff65c0ab140 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff65c0add00 0x7ff65c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 --2- 172.21.15.28:0/679346833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff65c06f3f0 0x7ff65c0aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 >> 172.21.15.28:0/679346833 conn(0x7ff65c0b0270 msgr2=0x7ff65c0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 shutdown_connections 2024-09-13T17:16:55.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.717+0000 7ff662e00640 1 -- 172.21.15.28:0/679346833 wait complete. 2024-09-13T17:16:55.734 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 scrub status 2024-09-13T17:16:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.890+0000 7f99b6a00640 1 Processor -- start 2024-09-13T17:16:55.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.890+0000 7f99b6a00640 1 -- start start 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.891+0000 7f99b6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99b015c9a0 0x7f99b00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99b6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b0070470 con 0x7f99b00b2400 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99b6a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00ace30 con 0x7f99b00ad940 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99b6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00ad110 con 0x7f99b015c9a0 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99b015c9a0 0x7f99b00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99b015c9a0 0x7f99b00ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49428/0 (socket says 172.21.15.28:49428) 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 -- 172.21.15.28:0/1901486213 learned_addr learned my addr 172.21.15.28:0/1901486213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99b4a00640 1 --1- 172.21.15.28:0/1901486213 >> v1:172.21.15.28:6789/0 conn(0x7f99b00b2400 0x7f99b006bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45442/0 (socket says 172.21.15.28:45442) 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 -- 172.21.15.28:0/1901486213 >> v1:172.21.15.189:6789/0 conn(0x7f99b00ad940 legacy=0x7f99b00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 -- 172.21.15.28:0/1901486213 >> v1:172.21.15.28:6789/0 conn(0x7f99b00b2400 legacy=0x7f99b006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.892+0000 7f99afe00640 1 -- 172.21.15.28:0/1901486213 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b006b170 con 0x7f99b015c9a0 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.893+0000 7f99afe00640 1 --2- 172.21.15.28:0/1901486213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b015c9a0 0x7f99b00ac850 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f99a00098a0 tx=0x7f99a002a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=544acd8a24ce64f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99af400640 1 -- 172.21.15.28:0/1901486213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0032640 con 0x7f99b015c9a0 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99af400640 1 -- 172.21.15.28:0/1901486213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a00327a0 con 0x7f99b015c9a0 2024-09-13T17:16:55.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99af400640 1 -- 172.21.15.28:0/1901486213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0032b30 con 0x7f99b015c9a0 2024-09-13T17:16:55.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 -- 172.21.15.28:0/1901486213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b015c9a0 msgr2=0x7f99b00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 --2- 172.21.15.28:0/1901486213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b015c9a0 0x7f99b00ac850 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f99a00098a0 tx=0x7f99a002a3c0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 -- 172.21.15.28:0/1901486213 shutdown_connections 2024-09-13T17:16:55.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 --2- 172.21.15.28:0/1901486213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b015c9a0 0x7f99b00ac850 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 -- 172.21.15.28:0/1901486213 >> 172.21.15.28:0/1901486213 conn(0x7f99b00b0270 msgr2=0x7f99b00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 -- 172.21.15.28:0/1901486213 shutdown_connections 2024-09-13T17:16:55.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.894+0000 7f99b6a00640 1 -- 172.21.15.28:0/1901486213 wait complete. 2024-09-13T17:16:55.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.896+0000 7f99b6a00640 1 Processor -- start 2024-09-13T17:16:55.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.896+0000 7f99b6a00640 1 -- start start 2024-09-13T17:16:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f99b00ad940 0x7f99b0149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b015c9a0 0x7f99b015f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015dd80 con 0x7f99b00b2400 2024-09-13T17:16:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015e060 con 0x7f99b015c9a0 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b015e340 con 0x7f99b00ad940 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49436/0 (socket says 172.21.15.28:49436) 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99a7e00640 1 -- 172.21.15.28:0/717348292 learned_addr learned my addr 172.21.15.28:0/717348292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99b4a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b015c9a0 0x7f99b015f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.897+0000 7f99afe00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f99b00ad940 0x7f99b0149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 -- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f99b00ad940 msgr2=0x7f99b0149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f99b00ad940 0x7f99b0149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 -- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b015c9a0 msgr2=0x7f99b015f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b015c9a0 0x7f99b015f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 -- 172.21.15.28:0/717348292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a00094d0 con 0x7f99b00b2400 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99a7e00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f999800c450 tx=0x7f999800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99acc00640 1 -- 172.21.15.28:0/717348292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99980052d0 con 0x7f99b00b2400 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99acc00640 1 -- 172.21.15.28:0/717348292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9998005430 con 0x7f99b00b2400 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99acc00640 1 -- 172.21.15.28:0/717348292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998012870 con 0x7f99b00b2400 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b0070930 con 0x7f99b00b2400 2024-09-13T17:16:55.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.898+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b015e900 con 0x7f99b00b2400 2024-09-13T17:16:55.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.900+0000 7f99acc00640 1 -- 172.21.15.28:0/717348292 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f99980129d0 con 0x7f99b00b2400 2024-09-13T17:16:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.901+0000 7f99acc00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 0x7f998c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.901+0000 7f99acc00640 1 -- 172.21.15.28:0/717348292 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9998088e20 con 0x7f99b00b2400 2024-09-13T17:16:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.902+0000 7f99afe00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 0x7f998c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.902+0000 7f99b99d4740 1 Processor -- start 2024-09-13T17:16:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.902+0000 7f99b99d4740 1 -- start start 2024-09-13T17:16:55.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99afe00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 0x7f998c06d0c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f99a0034000 tx=0x7f99a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a909420 0x56377a959130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a959670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a8126a0 con 0x56377a90bc60 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a875cc0 con 0x56377a909420 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56377a873450 con 0x56377a933110 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49442/0 (socket says 172.21.15.28:49442) 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.903+0000 7f99afe00640 1 -- 172.21.15.28:0/3027094328 learned_addr learned my addr 172.21.15.28:0/3027094328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99b4a00640 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a959670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99a7e00640 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a909420 0x56377a959130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 -- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 msgr2=0x56377a959670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a959670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 -- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a909420 msgr2=0x56377a959130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a909420 0x56377a959130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 -- 172.21.15.28:0/3027094328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56377a83f2e0 con 0x56377a90bc60 2024-09-13T17:16:55.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99afe00640 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f99a0000c00 tx=0x7f99a002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd0af257df426bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99a4c00640 1 -- 172.21.15.28:0/3027094328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0004230 con 0x56377a90bc60 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99a4c00640 1 -- 172.21.15.28:0/3027094328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0004390 con 0x56377a90bc60 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.904+0000 7f99a4c00640 1 -- 172.21.15.28:0/3027094328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0003810 con 0x56377a90bc60 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 -- 172.21.15.28:0/3027094328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 msgr2=0x56377a90c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f99a0000c00 tx=0x7f99a002c000 comp rx=0 tx=0).stop 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 -- 172.21.15.28:0/3027094328 shutdown_connections 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a959670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a909420 0x56377a959130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3027094328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a90c040 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 -- 172.21.15.28:0/3027094328 >> 172.21.15.28:0/3027094328 conn(0x56377a908390 msgr2=0x56377a950840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 -- 172.21.15.28:0/3027094328 shutdown_connections 2024-09-13T17:16:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.905+0000 7f99b99d4740 1 -- 172.21.15.28:0/3027094328 wait complete. 2024-09-13T17:16:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.906+0000 7f99b99d4740 1 -- start start 2024-09-13T17:16:55.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.906+0000 7f99b99d4740 1 Processor -- start 2024-09-13T17:16:55.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9345e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 0x56377a934b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a86ed40 con 0x56377a90bc60 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a810280 con 0x56377a933110 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56377a80ff20 con 0x56377a91f950 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99b99d4740 10 client.0 _ll_register_callbacks cb 0x56377a8683b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49452/0 (socket says 172.21.15.28:49452) 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99afe00640 1 -- 172.21.15.28:0/3771800825 learned_addr learned my addr 172.21.15.28:0/3771800825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.907+0000 7f99afe00640 1 -- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 msgr2=0x56377a934b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99b4a00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 0x56377a934b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99a7e00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99afe00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 0x56377a934b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99afe00640 1 -- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 msgr2=0x56377a9345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99afe00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9345e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99afe00640 1 -- 172.21.15.28:0/3771800825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b006b170 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99b4a00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 0x56377a934b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99a7e00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99afe00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f99a00099c0 tx=0x7f99a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ef3d3b037fabdd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f999d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0003d10 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f99a003c600 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56377a5624e0 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99a0003e70 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.908+0000 7f99b99d4740 10 client.8667 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x56377a86d3c0 con 0x56377a90bc60 2024-09-13T17:16:55.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56377a5624e0 con 0x56377a90bc60 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f99a004f030 con 0x56377a90bc60 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 10 client.8667 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 10 client.8667 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 20 client.8667 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a91e510 0x56377a91e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 4 client.8667 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56377a5737e0 con 0x56377a91e510 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.909+0000 7f99a7e00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a91e510 0x56377a91e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f99a7e00640 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a91e510 0x56377a91e8f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2af0ddf22b33ad4c server_cookie=e320d9e5770327f4 in_seq=0 out_seq=0 2024-09-13T17:16:55.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f999d600640 10 client.8667 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f999d600640 1 -- 172.21.15.28:0/3771800825 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56377a5737e0 con 0x56377a91e510 2024-09-13T17:16:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f999d600640 10 client.8667 handle_command_reply: tid=0 2024-09-13T17:16:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f99b99d4740 1 client.8667 shutdown 2024-09-13T17:16:55.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 msgr2=0x56377a94dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.910+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f99a00099c0 tx=0x7f99a0004730 comp rx=0 tx=0).stop 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 shutdown_connections 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a91e510 0x56377a91e8f0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a91f950 0x56377a934b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9345e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 --2- 172.21.15.28:0/3771800825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a94dd00 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f999d600640 0 client.8667 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 >> 172.21.15.28:0/3771800825 conn(0x56377a909420 msgr2=0x56377a954550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 shutdown_connections 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 1 -- 172.21.15.28:0/3771800825 wait complete. 2024-09-13T17:16:55.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.911+0000 7f99b99d4740 20 client.8667 trim_cache size 0 max 16384 2024-09-13T17:16:55.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.926+0000 7f99b99d4740 1 Processor -- start 2024-09-13T17:16:55.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.926+0000 7f99b99d4740 1 -- start start 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a91f950 0x56377a933f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a91f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a5d6040 con 0x56377a90bc60 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a973070 con 0x56377a91f950 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56377a932080 con 0x56377a933110 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49458/0 (socket says 172.21.15.28:49458) 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 -- 172.21.15.28:0/2878443884 learned_addr learned my addr 172.21.15.28:0/2878443884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99a7e00640 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a91f950 0x56377a933f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99b4a00640 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a91f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 -- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 msgr2=0x56377a91f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a91f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 -- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a91f950 msgr2=0x56377a933f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a91f950 0x56377a933f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.927+0000 7f99afe00640 1 -- 172.21.15.28:0/2878443884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56377a80f610 con 0x56377a90bc60 2024-09-13T17:16:55.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.928+0000 7f99afe00640 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f99a0035040 tx=0x7f99a0054040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6553f3853fe323a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.928+0000 7f99a5600640 1 -- 172.21.15.28:0/2878443884 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0004500 con 0x56377a90bc60 2024-09-13T17:16:55.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.928+0000 7f99a5600640 1 -- 172.21.15.28:0/2878443884 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a003c860 con 0x56377a90bc60 2024-09-13T17:16:55.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.928+0000 7f99a5600640 1 -- 172.21.15.28:0/2878443884 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0003d10 con 0x56377a90bc60 2024-09-13T17:16:55.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- 172.21.15.28:0/2878443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 msgr2=0x56377a931740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f99a0035040 tx=0x7f99a0054040 comp rx=0 tx=0).stop 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- 172.21.15.28:0/2878443884 shutdown_connections 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a933110 0x56377a91f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a91f950 0x56377a933f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2878443884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a90bc60 0x56377a931740 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- 172.21.15.28:0/2878443884 >> 172.21.15.28:0/2878443884 conn(0x56377a909420 msgr2=0x56377a9349c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- 172.21.15.28:0/2878443884 shutdown_connections 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- 172.21.15.28:0/2878443884 wait complete. 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 -- start start 2024-09-13T17:16:55.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.929+0000 7f99b99d4740 1 Processor -- start 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a90bc60 0x56377a94c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9be3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a96dc20 con 0x56377a91f950 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56377a96f380 con 0x56377a933110 2024-09-13T17:16:55.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56377a94d6e0 con 0x56377a90bc60 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.930+0000 7f99b99d4740 10 client.0 _ll_register_callbacks cb 0x7f99a004b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49468/0 (socket says 172.21.15.28:49468) 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 -- 172.21.15.28:0/2996306235 learned_addr learned my addr 172.21.15.28:0/2996306235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99afe00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a90bc60 0x56377a94c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99b4a00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9be3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 -- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a90bc60 msgr2=0x56377a94c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a90bc60 0x56377a94c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 -- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 msgr2=0x56377a9be3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9be3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 -- 172.21.15.28:0/2996306235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56377a86d3c0 con 0x56377a91f950 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f99a7e00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f9998054190 tx=0x7f999800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9991f394140e2d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.931+0000 7f999d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9998012c80 con 0x56377a91f950 2024-09-13T17:16:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9998012de0 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99a0004390 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f999805fb40 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 10 client.8673 fetch_fsmap learned FSMap version 114 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x56377a834b20 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0004390 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9998084070 con 0x56377a91f950 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 10 client.8673 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 10 client.8673 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 20 client.8673 populate_metadata read hostname 'smithi028' 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a9be930 0x56377a96e040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 4 client.8673 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56377a5c6e30 con 0x56377a9be930 2024-09-13T17:16:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.932+0000 7f99afe00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a9be930 0x56377a96e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:16:55.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99afe00640 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a9be930 0x56377a96e040 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4546f41f557352c server_cookie=2d72524da32a24d1 in_seq=0 out_seq=0 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f999d600640 10 client.8673 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f999d600640 1 -- 172.21.15.28:0/2996306235 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x56377a5c6e30 con 0x56377a9be930 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f999d600640 10 client.8673 handle_command_reply: tid=0 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 client.8673 shutdown 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 msgr2=0x56377a9bdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f9998054190 tx=0x7f999800cf40 comp rx=0 tx=0).stop 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 shutdown_connections 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x56377a9be930 0x56377a96e040 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56377a933110 0x56377a9be3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56377a91f950 0x56377a9bdeb0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f999d600640 0 client.8673 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 --2- 172.21.15.28:0/2996306235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56377a90bc60 0x56377a94c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.933+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 >> 172.21.15.28:0/2996306235 conn(0x56377a909420 msgr2=0x56377a907100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 shutdown_connections 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b99d4740 1 -- 172.21.15.28:0/2996306235 wait complete. 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b99d4740 20 client.8673 trim_cache size 0 max 16384 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "623cff98-4878-4926-9158-b986833eef1f": { 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_control_test_path", 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "623cff98-4878-4926-9158-b986833eef1f", 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:16:55.939 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:16:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 msgr2=0x7f998c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 0x7f998c06d0c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f99a0034000 tx=0x7f99a002e000 comp rx=0 tx=0).stop 2024-09-13T17:16:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 msgr2=0x7f99b0155fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:16:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f999800c450 tx=0x7f999800c910 comp rx=0 tx=0).stop 2024-09-13T17:16:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 shutdown_connections 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f998c06ac00 0x7f998c06d0c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b015c9a0 0x7f99b015f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99b00b2400 0x7f99b0155fb0 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 --2- 172.21.15.28:0/717348292 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f99b00ad940 0x7f99b0149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 >> 172.21.15.28:0/717348292 conn(0x7f99b00b0270 msgr2=0x7f99b00af640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 shutdown_connections 2024-09-13T17:16:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:16:55.934+0000 7f99b6a00640 1 -- 172.21.15.28:0/717348292 wait complete. 2024-09-13T17:17:00.952 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 scrub status 2024-09-13T17:17:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.117+0000 7f5e13000640 1 Processor -- start 2024-09-13T17:17:01.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.117+0000 7f5e13000640 1 -- start start 2024-09-13T17:17:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.118+0000 7f5e13000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c160e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.118+0000 7f5e13000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e0c15d400 0x7f5e0c0732a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.118+0000 7f5e13000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e0c0737e0 con 0x7f5e0c072e40 2024-09-13T17:17:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.118+0000 7f5e13000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e0c160990 con 0x7f5e0c15d400 2024-09-13T17:17:01.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.118+0000 7f5e13000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c15d930 con 0x7f5e0c15c910 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c160e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e11200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5e0c072e40 0x7f5e0c15ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:57048/0 (socket says 172.21.15.28:57048) 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e12600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e0c15d400 0x7f5e0c0732a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c160e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:39412/0 (socket says 172.21.15.28:39412) 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e11c00640 1 -- 172.21.15.28:0/3936343912 learned_addr learned my addr 172.21.15.28:0/3936343912 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e12600640 1 -- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 msgr2=0x7f5e0c160e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e12600640 1 --2- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c160e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e12600640 1 -- 172.21.15.28:0/3936343912 >> v1:172.21.15.189:6789/0 conn(0x7f5e0c072e40 legacy=0x7f5e0c15ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.119+0000 7f5e12600640 1 -- 172.21.15.28:0/3936343912 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e0c15dc10 con 0x7f5e0c15d400 2024-09-13T17:17:01.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.120+0000 7f5e12600640 1 --2- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c0732a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79d14177caff19e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.123+0000 7f5e0be00640 1 -- 172.21.15.28:0/3936343912 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04007740 con 0x7f5e0c15d400 2024-09-13T17:17:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.123+0000 7f5e0be00640 1 -- 172.21.15.28:0/3936343912 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0402d050 con 0x7f5e0c15d400 2024-09-13T17:17:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.123+0000 7f5e0be00640 1 -- 172.21.15.28:0/3936343912 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04007ac0 con 0x7f5e0c15d400 2024-09-13T17:17:01.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.123+0000 7f5e13000640 1 -- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 msgr2=0x7f5e0c0732a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.123+0000 7f5e13000640 1 --2- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c0732a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402a660 comp rx=0 tx=0).stop 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 -- 172.21.15.28:0/3936343912 shutdown_connections 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 --2- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c0732a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 --2- 172.21.15.28:0/3936343912 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c160e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 -- 172.21.15.28:0/3936343912 >> 172.21.15.28:0/3936343912 conn(0x7f5e0c0cbec0 msgr2=0x7f5e0c0cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 -- 172.21.15.28:0/3936343912 shutdown_connections 2024-09-13T17:17:01.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.124+0000 7f5e13000640 1 -- 172.21.15.28:0/3936343912 wait complete. 2024-09-13T17:17:01.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.125+0000 7f5e13000640 1 Processor -- start 2024-09-13T17:17:01.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.126+0000 7f5e13000640 1 -- start start 2024-09-13T17:17:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.126+0000 7f5e13000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e13000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c06c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e13000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c06c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e13000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1f6170 con 0x7f5e0c072e40 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e13000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1f3350 con 0x7f5e0c15d400 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e13000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e0c1586a0 con 0x7f5e0c15c910 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49484/0 (socket says 172.21.15.28:49484) 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 -- 172.21.15.28:0/3881646202 learned_addr learned my addr 172.21.15.28:0/3881646202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e12600640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c06c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11200640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c06c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 -- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 msgr2=0x7f5e0c06c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c06c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 -- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 msgr2=0x7f5e0c06c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c06c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.127+0000 7f5e11c00640 1 -- 172.21.15.28:0/3881646202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e04009a70 con 0x7f5e0c072e40 2024-09-13T17:17:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e11c00640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f5e0000e8b0 tx=0x7f5e0000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e09600640 1 -- 172.21.15.28:0/3881646202 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0000c940 con 0x7f5e0c072e40 2024-09-13T17:17:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e09600640 1 -- 172.21.15.28:0/3881646202 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e0000caa0 con 0x7f5e0c072e40 2024-09-13T17:17:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e0c0b4860 con 0x7f5e0c072e40 2024-09-13T17:17:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e09600640 1 -- 172.21.15.28:0/3881646202 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e0000ce30 con 0x7f5e0c072e40 2024-09-13T17:17:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.128+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1f5b20 con 0x7f5e0c072e40 2024-09-13T17:17:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.130+0000 7f5e09600640 1 -- 172.21.15.28:0/3881646202 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e00013470 con 0x7f5e0c072e40 2024-09-13T17:17:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.131+0000 7f5e09600640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 0x7f5dec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.131+0000 7f5e09600640 1 -- 172.21.15.28:0/3881646202 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e000894c0 con 0x7f5e0c072e40 2024-09-13T17:17:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.132+0000 7f5e11200640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 0x7f5dec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.132+0000 7f5e174b0740 1 Processor -- start 2024-09-13T17:17:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.132+0000 7f5e174b0740 1 -- start start 2024-09-13T17:17:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.132+0000 7f5e11200640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 0x7f5dec06d0c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c070d70 tx=0x7f5df8000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c0d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c5a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91b136a0 con 0x564a91c0a590 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91b76cc0 con 0x564a91c34280 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a91b74450 con 0x564a91c0cdd0 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c0d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49500/0 (socket says 172.21.15.28:49500) 2024-09-13T17:17:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 -- 172.21.15.28:0/1823347046 learned_addr learned my addr 172.21.15.28:0/1823347046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e12600640 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c5a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 -- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 msgr2=0x564a91c0d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c0d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 -- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 msgr2=0x564a91c5a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.133+0000 7f5e11200640 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c5a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.134+0000 7f5e11200640 1 -- 172.21.15.28:0/1823347046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a91b402e0 con 0x564a91c0a590 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.134+0000 7f5e11200640 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5df8007840 tx=0x7f5df8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d251470de2de1c45 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.134+0000 7f5dfe000640 1 -- 172.21.15.28:0/1823347046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df80150d0 con 0x564a91c0a590 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.134+0000 7f5dfe000640 1 -- 172.21.15.28:0/1823347046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8015230 con 0x564a91c0a590 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.134+0000 7f5dfe000640 1 -- 172.21.15.28:0/1823347046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8015b90 con 0x564a91c0a590 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- 172.21.15.28:0/1823347046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 msgr2=0x564a91c5a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5df8007840 tx=0x7f5df8012000 comp rx=0 tx=0).stop 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- 172.21.15.28:0/1823347046 shutdown_connections 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c5a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0a590 0x564a91c5a2a0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1823347046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c0d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- 172.21.15.28:0/1823347046 >> 172.21.15.28:0/1823347046 conn(0x564a91c09500 msgr2=0x564a91c519b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- 172.21.15.28:0/1823347046 shutdown_connections 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- 172.21.15.28:0/1823347046 wait complete. 2024-09-13T17:17:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.135+0000 7f5e174b0740 1 -- start start 2024-09-13T17:17:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.136+0000 7f5e174b0740 1 Processor -- start 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c0cdd0 0x564a91c35c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91c330e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91b6fd40 con 0x564a91c33750 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91b11280 con 0x564a91c0cdd0 2024-09-13T17:17:01.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a91b10f20 con 0x564a91c34280 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e174b0740 10 client.0 _ll_register_callbacks cb 0x564a91b693b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49502/0 (socket says 172.21.15.28:49502) 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 -- 172.21.15.28:0/1835096788 learned_addr learned my addr 172.21.15.28:0/1835096788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e11c00640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c0cdd0 0x564a91c35c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e11200640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91c330e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 -- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 msgr2=0x564a91c330e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91c330e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 -- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c0cdd0 msgr2=0x564a91c35c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c0cdd0 0x564a91c35c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.137+0000 7f5e12600640 1 -- 172.21.15.28:0/1835096788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c0737e0 con 0x564a91c33750 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5e12600640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496f959053cb3d8c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5df2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:01.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e04041070 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5e04046b00 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564a918634e0 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e04046cc0 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5e174b0740 10 client.8682 fetch_fsmap learned FSMap version 114 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x564a91b6e3c0 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.138+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564a918634e0 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e04038040 con 0x564a91c33750 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 10 client.8682 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 10 client.8682 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 20 client.8682 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c64b20 0x564a91c64f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 4 client.8682 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a918747e0 con 0x564a91c64b20 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.139+0000 7f5e11c00640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c64b20 0x564a91c64f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e11c00640 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c64b20 0x564a91c64f00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b925b161cec9f01 server_cookie=fda8f70e4947ac05 in_seq=0 out_seq=0 2024-09-13T17:17:01.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5df2a00640 10 client.8682 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5df2a00640 1 -- 172.21.15.28:0/1835096788 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x564a918747e0 con 0x564a91c64b20 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5df2a00640 10 client.8682 handle_command_reply: tid=0 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 client.8682 shutdown 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 msgr2=0x564a91c594b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009d90 tx=0x7f5e0402c000 comp rx=0 tx=0).stop 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 shutdown_connections 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c64b20 0x564a91c64f00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91c594b0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91c330e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1835096788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c0cdd0 0x564a91c35c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5df2a00640 0 client.8682 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:17:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.140+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 >> 172.21.15.28:0/1835096788 conn(0x564a91c0a590 msgr2=0x564a91c51030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.141+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 shutdown_connections 2024-09-13T17:17:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.141+0000 7f5e174b0740 1 -- 172.21.15.28:0/1835096788 wait complete. 2024-09-13T17:17:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.141+0000 7f5e174b0740 20 client.8682 trim_cache size 0 max 16384 2024-09-13T17:17:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.157+0000 7f5e174b0740 1 Processor -- start 2024-09-13T17:17:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.157+0000 7f5e174b0740 1 -- start start 2024-09-13T17:17:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c33750 0x564a91c32cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91cb2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a918d7040 con 0x564a91c0cdd0 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91c64210 con 0x564a91c33750 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a91c34ea0 con 0x564a91c34280 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49506/0 (socket says 172.21.15.28:49506) 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.158+0000 7f5e11c00640 1 -- 172.21.15.28:0/380826321 learned_addr learned my addr 172.21.15.28:0/380826321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 -- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 msgr2=0x564a91cb2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91cb2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 -- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c33750 msgr2=0x564a91c32cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c33750 0x564a91c32cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 -- 172.21.15.28:0/380826321 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a91b10610 con 0x564a91c0cdd0 2024-09-13T17:17:01.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5e11c00640 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5e00058350 tx=0x7f5e00089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77aab67c72bc387b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5dfea00640 1 -- 172.21.15.28:0/380826321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00073030 con 0x564a91c0cdd0 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5dfea00640 1 -- 172.21.15.28:0/380826321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e00060700 con 0x564a91c0cdd0 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.159+0000 7f5dfea00640 1 -- 172.21.15.28:0/380826321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e00086070 con 0x564a91c0cdd0 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- 172.21.15.28:0/380826321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 msgr2=0x564a91c56650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5e00058350 tx=0x7f5e00089040 comp rx=0 tx=0).stop 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- 172.21.15.28:0/380826321 shutdown_connections 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c34280 0x564a91cb2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c33750 0x564a91c32cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 --2- 172.21.15.28:0/380826321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c0cdd0 0x564a91c56650 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- 172.21.15.28:0/380826321 >> 172.21.15.28:0/380826321 conn(0x564a91c0a590 msgr2=0x564a91c53980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- 172.21.15.28:0/380826321 shutdown_connections 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- 172.21.15.28:0/380826321 wait complete. 2024-09-13T17:17:01.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.160+0000 7f5e174b0740 1 -- start start 2024-09-13T17:17:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 Processor -- start 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c56ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91c33540 con 0x564a91c33750 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a91c57420 con 0x564a91c34280 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.161+0000 7f5e174b0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x564a91c56670 con 0x564a91c0cdd0 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e174b0740 10 client.0 _ll_register_callbacks cb 0x7f5e04009450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c56ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49516/0 (socket says 172.21.15.28:49516) 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 -- 172.21.15.28:0/1948788781 learned_addr learned my addr 172.21.15.28:0/1948788781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e12600640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 -- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 msgr2=0x564a91c56ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c56ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 -- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 msgr2=0x564a91c539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 -- 172.21.15.28:0/1948788781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e0c1f5b20 con 0x564a91c33750 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e12600640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5e11200640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f5df8014000 tx=0x7f5df8007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bfb928fd2f4af6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.162+0000 7f5df2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df8027070 con 0x564a91c33750 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5df8021660 con 0x564a91c33750 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5df8015230 con 0x564a91c33750 2024-09-13T17:17:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=114) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5df80150d0 con 0x564a91c33750 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 10 client.8688 fetch_fsmap learned FSMap version 114 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=114,monmap=2+}) v3 -- 0x564a91b35b20 con 0x564a91c33750 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5df8015230 con 0x564a91c33750 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 114) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5df8012070 con 0x564a91c33750 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 10 client.8688 fetch_fsmap finished waiting for FSMap version 114 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 10 client.8688 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.c 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 20 client.8688 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c6aff0 0x564a91c50cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 4 client.8688 mds_command: new command op to 7939 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.163+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 --> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564a918c7e30 con 0x564a91c6aff0 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5e11c00640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c6aff0 0x564a91c50cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5df2a00640 10 client.8688 ms_handle_connect on v2:172.21.15.28:6833/199464366 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5e11c00640 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c6aff0 0x564a91c50cb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b68ec987d68c820 server_cookie=2931975dc8cb4714 in_seq=0 out_seq=0 2024-09-13T17:17:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5df2a00640 1 -- 172.21.15.28:0/1948788781 <== mds.0 v2:172.21.15.28:6833/199464366 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564a918c7e30 con 0x564a91c6aff0 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5df2a00640 10 client.8688 handle_command_reply: tid=0 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.164+0000 7f5e174b0740 1 client.8688 shutdown 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 msgr2=0x564a91cb1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f5df8014000 tx=0x7f5df8007e40 comp rx=0 tx=0).stop 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 shutdown_connections 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:6833/199464366,v1:172.21.15.28:6835/199464366] conn(0x564a91c6aff0 0x564a91c50cb0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564a91c34280 0x564a91c539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564a91c33750 0x564a91cb1260 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 --2- 172.21.15.28:0/1948788781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x564a91c0cdd0 0x564a91c56ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 >> 172.21.15.28:0/1948788781 conn(0x564a91c0a590 msgr2=0x564a91c32a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5df2a00640 0 client.8688 ms_handle_reset on v2:172.21.15.28:6833/199464366 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 shutdown_connections 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 1 -- 172.21.15.28:0/1948788781 wait complete. 2024-09-13T17:17:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.165+0000 7f5e174b0740 20 client.8688 trim_cache size 0 max 16384 2024-09-13T17:17:01.171 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:01.171 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:17:01.171 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:17:01.171 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 msgr2=0x7f5dec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 0x7f5dec06d0c0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c070d70 tx=0x7f5df8000ed0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 msgr2=0x7f5e0c158160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f5e0000e8b0 tx=0x7f5e0000ed70 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 shutdown_connections 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dec06ac00 0x7f5dec06d0c0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e0c15d400 0x7f5e0c06c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e0c15c910 0x7f5e0c06c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 --2- 172.21.15.28:0/3881646202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e0c072e40 0x7f5e0c158160 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 >> 172.21.15.28:0/3881646202 conn(0x7f5e0c0cbec0 msgr2=0x7f5e0c15fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 shutdown_connections 2024-09-13T17:17:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.166+0000 7f5e13000640 1 -- 172.21.15.28:0/3881646202 wait complete. 2024-09-13T17:17:01.183 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:17:01.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.339+0000 7f083dc00640 1 Processor -- start 2024-09-13T17:17:01.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.339+0000 7f083dc00640 1 -- start start 2024-09-13T17:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f083dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08381592a0 0x7f083815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f083dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f083806c2d0 0x7f083806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f083dc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083806cc70 con 0x7f083806f970 2024-09-13T17:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f083dc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f083815b0b0 con 0x7f08381592a0 2024-09-13T17:17:01.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f083dc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0838159d90 con 0x7f083806c2d0 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08381592a0 0x7f083815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08381592a0 0x7f083815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49518/0 (socket says 172.21.15.28:49518) 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 -- 172.21.15.28:0/370199861 learned_addr learned my addr 172.21.15.28:0/370199861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f082fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f083806f970 0x7f0838159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45446/0 (socket says 172.21.15.28:45446) 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0837400640 1 --2- 172.21.15.28:0/370199861 >> v2:172.21.15.189:3300/0 conn(0x7f083806c2d0 0x7f083806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 -- 172.21.15.28:0/370199861 >> v2:172.21.15.189:3300/0 conn(0x7f083806c2d0 msgr2=0x7f083806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 --2- 172.21.15.28:0/370199861 >> v2:172.21.15.189:3300/0 conn(0x7f083806c2d0 0x7f083806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 -- 172.21.15.28:0/370199861 >> v1:172.21.15.28:6789/0 conn(0x7f083806f970 legacy=0x7f0838159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.341+0000 7f0836a00640 1 -- 172.21.15.28:0/370199861 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08380b2040 con 0x7f08381592a0 2024-09-13T17:17:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f0836a00640 1 --2- 172.21.15.28:0/370199861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08381592a0 0x7f083815ab70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f08380b1dd0 tx=0x7f082802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa17a715456dcdf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f0836000640 1 -- 172.21.15.28:0/370199861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828032440 con 0x7f08381592a0 2024-09-13T17:17:01.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f0836000640 1 -- 172.21.15.28:0/370199861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08280325a0 con 0x7f08381592a0 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f0836000640 1 -- 172.21.15.28:0/370199861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828032930 con 0x7f08381592a0 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 -- 172.21.15.28:0/370199861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08381592a0 msgr2=0x7f083815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 --2- 172.21.15.28:0/370199861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08381592a0 0x7f083815ab70 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f08380b1dd0 tx=0x7f082802a270 comp rx=0 tx=0).stop 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 -- 172.21.15.28:0/370199861 shutdown_connections 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 --2- 172.21.15.28:0/370199861 >> v2:172.21.15.189:3300/0 conn(0x7f083806c2d0 0x7f083806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 --2- 172.21.15.28:0/370199861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08381592a0 0x7f083815ab70 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.343+0000 7f083dc00640 1 -- 172.21.15.28:0/370199861 >> 172.21.15.28:0/370199861 conn(0x7f0838069e80 msgr2=0x7f083806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.344+0000 7f083dc00640 1 -- 172.21.15.28:0/370199861 shutdown_connections 2024-09-13T17:17:01.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.344+0000 7f083dc00640 1 -- 172.21.15.28:0/370199861 wait complete. 2024-09-13T17:17:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.345+0000 7f083dc00640 1 Processor -- start 2024-09-13T17:17:01.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.345+0000 7f083dc00640 1 -- start start 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f083806f970 0x7f0838143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 0x7f0838150680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0838151a70 con 0x7f083806c2d0 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08381442c0 con 0x7f08381592a0 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f083dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0838150bc0 con 0x7f083806f970 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f0836a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f0836a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49530/0 (socket says 172.21.15.28:49530) 2024-09-13T17:17:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.346+0000 7f0836a00640 1 -- 172.21.15.28:0/3107438080 learned_addr learned my addr 172.21.15.28:0/3107438080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0837400640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 0x7f0838150680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f082fe00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f083806f970 0x7f0838143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 -- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f083806f970 msgr2=0x7f0838143d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f083806f970 0x7f0838143d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 -- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 msgr2=0x7f0838150680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 0x7f0838150680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 -- 172.21.15.28:0/3107438080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280094d0 con 0x7f083806c2d0 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0837400640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 0x7f0838150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:01.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f0836a00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f082802a2b0 tx=0x7f08280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.347+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0828032440 con 0x7f083806c2d0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.348+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0828030ba0 con 0x7f083806c2d0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.348+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f082803d440 con 0x7f083806c2d0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.348+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0838150e60 con 0x7f083806c2d0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.348+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0838151420 con 0x7f083806c2d0 2024-09-13T17:17:01.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.349+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08380b3740 con 0x7f083806c2d0 2024-09-13T17:17:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.350+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0828039040 con 0x7f083806c2d0 2024-09-13T17:17:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.350+0000 7f082ea00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 0x7f081406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.350+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f08280a9ff0 con 0x7f083806c2d0 2024-09-13T17:17:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.351+0000 7f082fe00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 0x7f081406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.352+0000 7f082fe00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 0x7f081406d0c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0838145400 tx=0x7f0820007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.355+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f08280750f0 con 0x7f083806c2d0 2024-09-13T17:17:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.569+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08380af800 con 0x7f083806c2d0 2024-09-13T17:17:01.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.569+0000 7f082ea00640 1 -- 172.21.15.28:0/3107438080 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7f0828041090 con 0x7f083806c2d0 2024-09-13T17:17:01.572 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:01.572 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:17:01.572 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 msgr2=0x7f081406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 0x7f081406d0c0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f0838145400 tx=0x7f0820007c80 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 msgr2=0x7f0838152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f082802a2b0 tx=0x7f08280046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 shutdown_connections 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f081406ac00 0x7f081406d0c0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08381592a0 0x7f0838150680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f083806f970 0x7f0838143d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 --2- 172.21.15.28:0/3107438080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f083806c2d0 0x7f0838152fe0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 >> 172.21.15.28:0/3107438080 conn(0x7f0838069e80 msgr2=0x7f0838159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 shutdown_connections 2024-09-13T17:17:01.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.572+0000 7f083dc00640 1 -- 172.21.15.28:0/3107438080 wait complete. 2024-09-13T17:17:01.589 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:01.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.745+0000 7fa063400640 1 Processor -- start 2024-09-13T17:17:01.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.745+0000 7fa063400640 1 -- start start 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa063400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa05c0cabc0 0x7fa05c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa063400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c15e100 con 0x7fa05c15a540 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa063400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa05c15e3e0 con 0x7fa05c070cc0 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa063400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa05c0adb00 con 0x7fa05c0cabc0 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa060a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa05c15a540 0x7fa05c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45456/0 (socket says 172.21.15.28:45456) 2024-09-13T17:17:01.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.746+0000 7fa060a00640 1 -- 172.21.15.28:0/3033009276 learned_addr learned my addr 172.21.15.28:0/3033009276 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573029894 0 0) 0x7fa05c15e100 con 0x7fa05c15a540 2024-09-13T17:17:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa040003520 con 0x7fa05c15a540 2024-09-13T17:17:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa053e00640 1 --2- 172.21.15.28:0/3033009276 >> v2:172.21.15.189:3300/0 conn(0x7fa05c0cabc0 0x7fa05c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3989443997 0 0) 0x7fa040003520 con 0x7fa05c15a540 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 >> v2:172.21.15.189:3300/0 conn(0x7fa05c0cabc0 msgr2=0x7fa05c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 --2- 172.21.15.28:0/3033009276 >> v2:172.21.15.189:3300/0 conn(0x7fa05c0cabc0 0x7fa05c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 >> v1:172.21.15.189:6789/0 conn(0x7fa05c070cc0 legacy=0x7fa05c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c15ef80 con 0x7fa05c15a540 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa04c001670 con 0x7fa05c15a540 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 >> v1:172.21.15.28:6789/0 conn(0x7fa05c15a540 legacy=0x7fa05c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 0x7fa040003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0400047c0 0x7fa040004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040005180 0x7fa04000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.747+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04000c0f0 con 0x7fa040003970 2024-09-13T17:17:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04000c260 con 0x7fa040005180 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04000c420 con 0x7fa0400047c0 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 0x7fa040003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa053e00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0400047c0 0x7fa040004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa061400640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040005180 0x7fa04000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 -- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0400047c0 msgr2=0x7fa040004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0400047c0 0x7fa040004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 -- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040005180 msgr2=0x7fa04000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040005180 0x7fa04000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa060a00640 1 -- 172.21.15.28:0/3033009276 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040003520 con 0x7fa040003970 2024-09-13T17:17:01.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.748+0000 7fa061400640 1 -- 172.21.15.28:0/3033009276 reap_dead start 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.749+0000 7fa060a00640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 0x7fa040003d90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00c240 tx=0x7fa0400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74d0a46288a7937 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.749+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c02e070 con 0x7fa040003970 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.749+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04c0058d0 con 0x7fa040003970 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.749+0000 7fa05be00640 1 -- 172.21.15.28:0/3033009276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04c005d00 con 0x7fa040003970 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 -- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 msgr2=0x7fa040003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 0x7fa040003d90 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fa04c00c240 tx=0x7fa0400057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 -- 172.21.15.28:0/3033009276 shutdown_connections 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 --2- 172.21.15.28:0/3033009276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa040003970 0x7fa040003d90 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 -- 172.21.15.28:0/3033009276 >> 172.21.15.28:0/3033009276 conn(0x7fa05c069e80 msgr2=0x7fa05c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:01.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 -- 172.21.15.28:0/3033009276 shutdown_connections 2024-09-13T17:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.750+0000 7fa063400640 1 -- 172.21.15.28:0/3033009276 wait complete. 2024-09-13T17:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.752+0000 7fa063400640 1 Processor -- start 2024-09-13T17:17:01.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.752+0000 7fa063400640 1 -- start start 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040003970 0x7fa05c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa05c0b3400 0x7fa05c0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0b2210 con 0x7fa05c14ae00 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c14be90 con 0x7fa040003970 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa063400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05c0b18b0 con 0x7fa05c0b3400 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49536/0 (socket says 172.21.15.28:49536) 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 -- 172.21.15.28:0/4033886171 learned_addr learned my addr 172.21.15.28:0/4033886171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa060a00640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040003970 0x7fa05c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa053e00640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa05c0b3400 0x7fa05c0acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 -- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa05c0b3400 msgr2=0x7fa05c0acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa05c0b3400 0x7fa05c0acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 -- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040003970 msgr2=0x7fa05c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040003970 0x7fa05c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.753+0000 7fa061400640 1 -- 172.21.15.28:0/4033886171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa05c0b1b50 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa061400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0ce3b0 tx=0x7fa05400ea00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054006d60 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa054003970 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa054003d20 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa05c0b1e30 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.754+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05c14b7c0 con 0x7fa05c14ae00 2024-09-13T17:17:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.755+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa05c061090 con 0x7fa05c14ae00 2024-09-13T17:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.756+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa054005070 con 0x7fa05c14ae00 2024-09-13T17:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.757+0000 7fa059600640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 0x7fa03806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.757+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa054088b30 con 0x7fa05c14ae00 2024-09-13T17:17:01.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.758+0000 7fa060a00640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 0x7fa03806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:01.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.758+0000 7fa060a00640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 0x7fa03806d0f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa04c033000 tx=0x7fa04c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:01.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:01.761+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa054053ce0 con 0x7fa05c14ae00 2024-09-13T17:17:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.010+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa05c0b2350 con 0x7fa05c14ae00 2024-09-13T17:17:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.010+0000 7fa059600640 1 -- 172.21.15.28:0/4033886171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1202 (secure 0 0 0) 0x7fa05c0b2350 con 0x7fa05c14ae00 2024-09-13T17:17:02.012 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:02.012 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":555,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":127,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076497100,"bytes_used":2675691520,"bytes_avail":770418421760,"bytes_total":773094113280,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":2586381,"recovering_keys_per_sec":0,"num_objects_recovered":22,"num_bytes_recovered":23068672,"num_keys_recovered":0,"write_bytes_sec":49055034,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":114,"id":8,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":8,"rank":0,"name":"c","status":"up:active","gid":7939}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-09-13T17:16:17.009496+0000","services":{}},"progress_events":{}} 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 msgr2=0x7fa03806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 0x7fa03806d0f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa04c033000 tx=0x7fa04c02e000 comp rx=0 tx=0).stop 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 msgr2=0x7fa05c14b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0ce3b0 tx=0x7fa05400ea00 comp rx=0 tx=0).stop 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 shutdown_connections 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa03806ac30 0x7fa03806d0f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa05c14ae00 0x7fa05c14b280 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa05c0b3400 0x7fa05c0acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 --2- 172.21.15.28:0/4033886171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa040003970 0x7fa05c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 >> 172.21.15.28:0/4033886171 conn(0x7fa05c069e80 msgr2=0x7fa05c0ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 shutdown_connections 2024-09-13T17:17:02.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.012+0000 7fa063400640 1 -- 172.21.15.28:0/4033886171 wait complete. 2024-09-13T17:17:02.029 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:17:02.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:02.121 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:02.225 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:17:02.225 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:02.226 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:02.261 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:02.261 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:02.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:02.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.531+0000 7f1bcb200640 1 Processor -- start 2024-09-13T17:17:02.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.531+0000 7f1bcb200640 1 -- start start 2024-09-13T17:17:02.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.532+0000 7f1bcb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 0x7f1bc406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.532+0000 7f1bcb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc415e100 con 0x7f1bc40cabc0 2024-09-13T17:17:02.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.532+0000 7f1bcb200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc415e3e0 con 0x7f1bc4070cc0 2024-09-13T17:17:02.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.532+0000 7f1bcb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40adb00 con 0x7f1bc415a540 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1bc40cabc0 0x7f1bc415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45464/0 (socket says 172.21.15.28:45464) 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3400640 1 -- 172.21.15.28:0/659339870 learned_addr learned my addr 172.21.15.28:0/659339870 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3e00640 1 --2- 172.21.15.28:0/659339870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 0x7f1bc406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105325674 0 0) 0x7f1bc415e100 con 0x7f1bc40cabc0 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba8003520 con 0x7f1bc40cabc0 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3e00640 1 -- 172.21.15.28:0/659339870 >> v1:172.21.15.189:6789/0 conn(0x7f1bc4070cc0 legacy=0x7f1bc406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3e00640 1 -- 172.21.15.28:0/659339870 >> v1:172.21.15.28:6789/0 conn(0x7f1bc40cabc0 legacy=0x7f1bc415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954861047 0 0) 0x7f1bc415e3e0 con 0x7f1bc4070cc0 2024-09-13T17:17:02.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.533+0000 7f1bc3e00640 1 -- 172.21.15.28:0/659339870 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc415ef80 con 0x7f1bc415a540 2024-09-13T17:17:02.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.534+0000 7f1bc3e00640 1 --2- 172.21.15.28:0/659339870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 0x7f1bc406c430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40097d0 tx=0x7f1bb402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68adb7d9a74d24fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb402ad90 con 0x7f1bc415a540 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb4032440 con 0x7f1bc415a540 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bc2a00640 1 -- 172.21.15.28:0/659339870 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb40327d0 con 0x7f1bc415a540 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 -- 172.21.15.28:0/659339870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 msgr2=0x7f1bc406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 --2- 172.21.15.28:0/659339870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 0x7f1bc406c430 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1bb40097d0 tx=0x7f1bb402a100 comp rx=0 tx=0).stop 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 -- 172.21.15.28:0/659339870 shutdown_connections 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 --2- 172.21.15.28:0/659339870 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc415a540 0x7f1bc406c430 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 -- 172.21.15.28:0/659339870 >> 172.21.15.28:0/659339870 conn(0x7f1bc4069e80 msgr2=0x7f1bc406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 -- 172.21.15.28:0/659339870 shutdown_connections 2024-09-13T17:17:02.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.535+0000 7f1bcb200640 1 -- 172.21.15.28:0/659339870 wait complete. 2024-09-13T17:17:02.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.537+0000 7f1bcb200640 1 Processor -- start 2024-09-13T17:17:02.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.537+0000 7f1bcb200640 1 -- start start 2024-09-13T17:17:02.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc4070cc0 0x7f1bc414bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bc415a540 0x7f1bc415cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc4159c10 con 0x7f1bc40cabc0 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc415b500 con 0x7f1bc415a540 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bcb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc415abd0 con 0x7f1bc4070cc0 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bc3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49538/0 (socket says 172.21.15.28:49538) 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bc3400640 1 -- 172.21.15.28:0/2988787443 learned_addr learned my addr 172.21.15.28:0/2988787443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bc3e00640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc4070cc0 0x7f1bc414bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.538+0000 7f1bc9200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bc415a540 0x7f1bc415cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 -- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc4070cc0 msgr2=0x7f1bc414bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc4070cc0 0x7f1bc414bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 -- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bc415a540 msgr2=0x7f1bc415cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bc415a540 0x7f1bc415cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 -- 172.21.15.28:0/2988787443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb4009400 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bc3400640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000c780 tx=0x7f1bb000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0005ab0 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bb0005c10 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0007070 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc415aeb0 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.539+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc40b1c50 con 0x7f1bc40cabc0 2024-09-13T17:17:02.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.541+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc40b1e70 con 0x7f1bc40cabc0 2024-09-13T17:17:02.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.541+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1bb00071d0 con 0x7f1bc40cabc0 2024-09-13T17:17:02.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.542+0000 7f1bbbe00640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 0x7f1ba406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.542+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1bb0088a40 con 0x7f1bc40cabc0 2024-09-13T17:17:02.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.543+0000 7f1bc3e00640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 0x7f1ba406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.544+0000 7f1bc3e00640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 0x7f1ba406d0c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4034000 tx=0x7f1bb402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.546+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1bc40b1e70 con 0x7f1bc40cabc0 2024-09-13T17:17:02.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.726+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bc40b1e70 con 0x7f1bc40cabc0 2024-09-13T17:17:02.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.727+0000 7f1bbbe00640 1 -- 172.21.15.28:0/2988787443 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f1bc40b1e70 con 0x7f1bc40cabc0 2024-09-13T17:17:02.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:02.730 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:53.097532+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:02.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 msgr2=0x7f1ba406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 0x7f1ba406d0c0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4034000 tx=0x7f1bb402e000 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 msgr2=0x7f1bc40b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f1bb000c780 tx=0x7f1bb000cb80 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 shutdown_connections 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1ba406ac00 0x7f1ba406d0c0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bc415a540 0x7f1bc415cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bc40cabc0 0x7f1bc40b1710 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 --2- 172.21.15.28:0/2988787443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bc4070cc0 0x7f1bc414bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 >> 172.21.15.28:0/2988787443 conn(0x7f1bc4069e80 msgr2=0x7f1bc40ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 shutdown_connections 2024-09-13T17:17:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.729+0000 7f1bcb200640 1 -- 172.21.15.28:0/2988787443 wait complete. 2024-09-13T17:17:02.747 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:16:27.833325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:16:28.661422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:17:02.747 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:17:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.903+0000 7f8f37000640 1 Processor -- start 2024-09-13T17:17:02.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.903+0000 7f8f37000640 1 -- start start 2024-09-13T17:17:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.904+0000 7f8f37000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 0x7f8f3006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.904+0000 7f8f37000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f3015cd80 0x7f8f30070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.904+0000 7f8f37000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3006f920 con 0x7f8f3006f3f0 2024-09-13T17:17:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.904+0000 7f8f37000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3015b290 con 0x7f8f3015cd80 2024-09-13T17:17:02.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.904+0000 7f8f37000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3015b570 con 0x7f8f30159770 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f3015cd80 0x7f8f30070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f2fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 0x7f8f3006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8f3015cd80 0x7f8f30070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44228/0 (socket says 172.21.15.28:44228) 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 -- 172.21.15.28:0/1216700412 learned_addr learned my addr 172.21.15.28:0/1216700412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 -- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 msgr2=0x7f8f3006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 0x7f8f3006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 -- 172.21.15.28:0/1216700412 >> v1:172.21.15.189:6789/0 conn(0x7f8f3006f3f0 legacy=0x7f8f30159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f2ea00640 1 -- 172.21.15.28:0/1216700412 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755426345 0 0) 0x7f8f3006f920 con 0x7f8f3006f3f0 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f35000640 1 -- 172.21.15.28:0/1216700412 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3015b850 con 0x7f8f3015cd80 2024-09-13T17:17:02.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.905+0000 7f8f2fe00640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 0x7f8f3006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f35000640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3015cd80 0x7f8f30070260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009e50 tx=0x7f8f2802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408d96649ce42f5d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f2ea00640 1 -- 172.21.15.28:0/1216700412 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2802aea0 con 0x7f8f3015cd80 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f2ea00640 1 -- 172.21.15.28:0/1216700412 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2802d050 con 0x7f8f3015cd80 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f2ea00640 1 -- 172.21.15.28:0/1216700412 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f28031820 con 0x7f8f3015cd80 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 -- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3015cd80 msgr2=0x7f8f30070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3015cd80 0x7f8f30070260 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8f28009e50 tx=0x7f8f2802a680 comp rx=0 tx=0).stop 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 -- 172.21.15.28:0/1216700412 shutdown_connections 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3015cd80 0x7f8f30070260 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 --2- 172.21.15.28:0/1216700412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f30159770 0x7f8f3006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.907+0000 7f8f37000640 1 -- 172.21.15.28:0/1216700412 >> 172.21.15.28:0/1216700412 conn(0x7f8f300b0270 msgr2=0x7f8f300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.908+0000 7f8f37000640 1 -- 172.21.15.28:0/1216700412 shutdown_connections 2024-09-13T17:17:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.908+0000 7f8f37000640 1 -- 172.21.15.28:0/1216700412 wait complete. 2024-09-13T17:17:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.909+0000 7f8f37000640 1 Processor -- start 2024-09-13T17:17:02.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.909+0000 7f8f37000640 1 -- start start 2024-09-13T17:17:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3006f3f0 0x7f8f30146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f3015cd80 0x7f8f300af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f30148f90 con 0x7f8f30159770 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3014c140 con 0x7f8f3006f3f0 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f37000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f30148580 con 0x7f8f3015cd80 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49552/0 (socket says 172.21.15.28:49552) 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.910+0000 7f8f2f400640 1 -- 172.21.15.28:0/3867074272 learned_addr learned my addr 172.21.15.28:0/3867074272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2fe00640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3006f3f0 0x7f8f30146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f35000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f3015cd80 0x7f8f300af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 -- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f3015cd80 msgr2=0x7f8f300af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f3015cd80 0x7f8f300af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 -- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3006f3f0 msgr2=0x7f8f30146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3006f3f0 0x7f8f30146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 -- 172.21.15.28:0/3867074272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f20009400 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f2f400640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c3e0 tx=0x7f8f1c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0035f0 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c003750 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c003b20 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.911+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f28009b30 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.912+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f301482a0 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.913+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f301490d0 con 0x7f8f30159770 2024-09-13T17:17:02.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.913+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8f1c003c80 con 0x7f8f30159770 2024-09-13T17:17:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.914+0000 7f8f27e00640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:02.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.914+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8f1c0889a0 con 0x7f8f30159770 2024-09-13T17:17:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.915+0000 7f8f2fe00640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:02.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.916+0000 7f8f2fe00640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 0x7f8f1006d0c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8f300b3930 tx=0x7f8f20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:02.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:02.918+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8f301490d0 con 0x7f8f30159770 2024-09-13T17:17:03.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.095+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f301490d0 con 0x7f8f30159770 2024-09-13T17:17:03.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.096+0000 7f8f27e00640 1 -- 172.21.15.28:0/3867074272 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f301490d0 con 0x7f8f30159770 2024-09-13T17:17:03.098 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 msgr2=0x7f8f1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 0x7f8f1006d0c0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f8f300b3930 tx=0x7f8f20008040 comp rx=0 tx=0).stop 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 msgr2=0x7f8f300b1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00c3e0 tx=0x7f8f1c00c8a0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 shutdown_connections 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8f1006ac00 0x7f8f1006d0c0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8f3015cd80 0x7f8f300af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f30159770 0x7f8f300b1d60 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 --2- 172.21.15.28:0/3867074272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f3006f3f0 0x7f8f30146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 >> 172.21.15.28:0/3867074272 conn(0x7f8f300b0270 msgr2=0x7f8f30158610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 shutdown_connections 2024-09-13T17:17:03.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.098+0000 7f8f37000640 1 -- 172.21.15.28:0/3867074272 wait complete. 2024-09-13T17:17:03.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:17:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.272+0000 7f7283c00640 1 Processor -- start 2024-09-13T17:17:03.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.272+0000 7f7283c00640 1 -- start start 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7283c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f727c0b2400 0x7f727c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7283c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c070470 con 0x7f727c0ad940 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7283c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c0ace30 con 0x7f727c15c9a0 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7283c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c0ad110 con 0x7f727c0b2400 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f727be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f727c0b2400 0x7f727c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7281c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f727c0ad940 0x7f727c06bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45474/0 (socket says 172.21.15.28:45474) 2024-09-13T17:17:03.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f7281c00640 1 -- 172.21.15.28:0/3208358200 learned_addr learned my addr 172.21.15.28:0/3208358200 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f727be00640 1 -- 172.21.15.28:0/3208358200 >> v1:172.21.15.189:6789/0 conn(0x7f727c15c9a0 legacy=0x7f727c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.274+0000 7f727be00640 1 -- 172.21.15.28:0/3208358200 >> v1:172.21.15.28:6789/0 conn(0x7f727c0ad940 legacy=0x7f727c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.275+0000 7f727be00640 1 -- 172.21.15.28:0/3208358200 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c06b170 con 0x7f727c0b2400 2024-09-13T17:17:03.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.275+0000 7f727b400640 1 -- 172.21.15.28:0/3208358200 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167785032 0 0) 0x7f727c070470 con 0x7f727c0ad940 2024-09-13T17:17:03.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f727be00640 1 --2- 172.21.15.28:0/3208358200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c0ac850 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f72680097a0 tx=0x7f726802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38895fc0a3b62e15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f727b400640 1 -- 172.21.15.28:0/3208358200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268032440 con 0x7f727c0b2400 2024-09-13T17:17:03.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f727b400640 1 -- 172.21.15.28:0/3208358200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72680325a0 con 0x7f727c0b2400 2024-09-13T17:17:03.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f727b400640 1 -- 172.21.15.28:0/3208358200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268032930 con 0x7f727c0b2400 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f7283c00640 1 -- 172.21.15.28:0/3208358200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 msgr2=0x7f727c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f7283c00640 1 --2- 172.21.15.28:0/3208358200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c0ac850 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f72680097a0 tx=0x7f726802a230 comp rx=0 tx=0).stop 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f7283c00640 1 -- 172.21.15.28:0/3208358200 shutdown_connections 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f7283c00640 1 --2- 172.21.15.28:0/3208358200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c0ac850 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.276+0000 7f7283c00640 1 -- 172.21.15.28:0/3208358200 >> 172.21.15.28:0/3208358200 conn(0x7f727c0b0270 msgr2=0x7f727c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.277+0000 7f7283c00640 1 -- 172.21.15.28:0/3208358200 shutdown_connections 2024-09-13T17:17:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.277+0000 7f7283c00640 1 -- 172.21.15.28:0/3208358200 wait complete. 2024-09-13T17:17:03.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.278+0000 7f7283c00640 1 Processor -- start 2024-09-13T17:17:03.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.278+0000 7f7283c00640 1 -- start start 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f727c0ad940 0x7f727c150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727c15c9a0 0x7f727c0cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c150d80 con 0x7f727c0b2400 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c152bc0 con 0x7f727c15c9a0 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f7283c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0cb300 con 0x7f727c0ad940 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f727be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f727be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49580/0 (socket says 172.21.15.28:49580) 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.279+0000 7f727be00640 1 -- 172.21.15.28:0/250006418 learned_addr learned my addr 172.21.15.28:0/250006418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f7281200640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f727c0ad940 0x7f727c150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f7281c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727c15c9a0 0x7f727c0cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 -- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f727c0ad940 msgr2=0x7f727c150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f727c0ad940 0x7f727c150840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 -- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727c15c9a0 msgr2=0x7f727c0cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727c15c9a0 0x7f727c0cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 -- 172.21.15.28:0/250006418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7268009400 con 0x7f727c0b2400 2024-09-13T17:17:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.280+0000 7f727be00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7268000c00 tx=0x7f72680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.281+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7268030b80 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.281+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7268030ce0 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.281+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726803d680 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.281+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c070930 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.281+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c0cb8c0 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.282+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c15cd80 con 0x7f727c0b2400 2024-09-13T17:17:03.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.283+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7268039040 con 0x7f727c0b2400 2024-09-13T17:17:03.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.284+0000 7f7278c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 0x7f725c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.284+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f72680a9e50 con 0x7f727c0b2400 2024-09-13T17:17:03.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.285+0000 7f7281200640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 0x7f725c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.285+0000 7f7281200640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 0x7f725c06d0c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7270000f60 tx=0x7f7270009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.288+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7268075000 con 0x7f727c0b2400 2024-09-13T17:17:03.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.465+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f727c070cd0 con 0x7f727c0b2400 2024-09-13T17:17:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.465+0000 7f7278c00640 1 -- 172.21.15.28:0/250006418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7268041020 con 0x7f727c0b2400 2024-09-13T17:17:03.467 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:17:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 msgr2=0x7f725c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 0x7f725c06d0c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7270000f60 tx=0x7f7270009040 comp rx=0 tx=0).stop 2024-09-13T17:17:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 msgr2=0x7f727c152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f7268000c00 tx=0x7f72680046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 shutdown_connections 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f725c06ac00 0x7f725c06d0c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f727c15c9a0 0x7f727c0cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f727c0b2400 0x7f727c152680 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 --2- 172.21.15.28:0/250006418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f727c0ad940 0x7f727c150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 >> 172.21.15.28:0/250006418 conn(0x7f727c0b0270 msgr2=0x7f727c159e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 shutdown_connections 2024-09-13T17:17:03.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.467+0000 7f7283c00640 1 -- 172.21.15.28:0/250006418 wait complete. 2024-09-13T17:17:03.484 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:17:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.640+0000 7f7d8b800640 1 Processor -- start 2024-09-13T17:17:03.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.640+0000 7f7d8b800640 1 -- start start 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.641+0000 7f7d8b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d84070430 0x7f7d8415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.641+0000 7f7d8b800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d8406f3f0 0x7f7d8415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.641+0000 7f7d8b800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8415ab60 con 0x7f7d84159770 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.641+0000 7f7d8b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8406f920 con 0x7f7d84070430 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.641+0000 7f7d8b800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8415b290 con 0x7f7d8406f3f0 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d84070430 0x7f7d8415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7d84070430 0x7f7d8415bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49590/0 (socket says 172.21.15.28:49590) 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 -- 172.21.15.28:0/1939177637 learned_addr learned my addr 172.21.15.28:0/1939177637 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d89800640 1 --2- 172.21.15.28:0/1939177637 >> v2:172.21.15.189:3300/0 conn(0x7f7d8406f3f0 0x7f7d8415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 -- 172.21.15.28:0/1939177637 >> v2:172.21.15.189:3300/0 conn(0x7f7d8406f3f0 msgr2=0x7f7d8415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 --2- 172.21.15.28:0/1939177637 >> v2:172.21.15.189:3300/0 conn(0x7f7d8406f3f0 0x7f7d8415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 -- 172.21.15.28:0/1939177637 >> v1:172.21.15.189:6789/0 conn(0x7f7d84159770 legacy=0x7f7d8406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.642+0000 7f7d83e00640 1 -- 172.21.15.28:0/1939177637 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8415b570 con 0x7f7d84070430 2024-09-13T17:17:03.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.643+0000 7f7d83e00640 1 --2- 172.21.15.28:0/1939177637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d8415bd70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406f380 tx=0x7f7d7002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d089104e6b6953 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d83400640 1 -- 172.21.15.28:0/1939177637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70032440 con 0x7f7d84070430 2024-09-13T17:17:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d83400640 1 -- 172.21.15.28:0/1939177637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d700325a0 con 0x7f7d84070430 2024-09-13T17:17:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d83400640 1 -- 172.21.15.28:0/1939177637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70032930 con 0x7f7d84070430 2024-09-13T17:17:03.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 -- 172.21.15.28:0/1939177637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 msgr2=0x7f7d8415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 --2- 172.21.15.28:0/1939177637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d8415bd70 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f7d8406f380 tx=0x7f7d7002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 -- 172.21.15.28:0/1939177637 shutdown_connections 2024-09-13T17:17:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 --2- 172.21.15.28:0/1939177637 >> v2:172.21.15.189:3300/0 conn(0x7f7d8406f3f0 0x7f7d8415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 --2- 172.21.15.28:0/1939177637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d8415bd70 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 -- 172.21.15.28:0/1939177637 >> 172.21.15.28:0/1939177637 conn(0x7f7d840b0270 msgr2=0x7f7d840b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:03.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 -- 172.21.15.28:0/1939177637 shutdown_connections 2024-09-13T17:17:03.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.644+0000 7f7d8b800640 1 -- 172.21.15.28:0/1939177637 wait complete. 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.646+0000 7f7d8b800640 1 Processor -- start 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.646+0000 7f7d8b800640 1 -- start start 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 0x7f7d840abb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d84159770 0x7f7d84157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840ab690 con 0x7f7d84070430 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d841542c0 con 0x7f7d8406f3f0 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d8b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d840aaa80 con 0x7f7d84159770 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49596/0 (socket says 172.21.15.28:49596) 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.647+0000 7f7d83e00640 1 -- 172.21.15.28:0/2001395113 learned_addr learned my addr 172.21.15.28:0/2001395113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 -- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d84159770 msgr2=0x7f7d84157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d84159770 0x7f7d84157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 -- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 msgr2=0x7f7d840abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d88e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 0x7f7d840abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 0x7f7d840abb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 -- 172.21.15.28:0/2001395113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d700094d0 con 0x7f7d84070430 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d88e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 0x7f7d840abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:03.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d83e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f7d7002a220 tx=0x7f7d700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d70032440 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d70030ba0 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d840aad60 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7003d440 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.648+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d840ab320 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.649+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d840ab800 con 0x7f7d84070430 2024-09-13T17:17:03.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.650+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d70039040 con 0x7f7d84070430 2024-09-13T17:17:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.651+0000 7f7d80c00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 0x7f7d6406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:03.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.651+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7d700a9fc0 con 0x7f7d84070430 2024-09-13T17:17:03.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.652+0000 7f7d88e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 0x7f7d6406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:03.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.652+0000 7f7d88e00640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 0x7f7d6406d0f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f60 tx=0x7f7d78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:03.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.655+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7d700750c0 con 0x7f7d84070430 2024-09-13T17:17:03.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.832+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d8406fd20 con 0x7f7d84070430 2024-09-13T17:17:03.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.833+0000 7f7d80c00640 1 -- 172.21.15.28:0/2001395113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d700410a0 con 0x7f7d84070430 2024-09-13T17:17:03.835 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:17:03.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 msgr2=0x7f7d6406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 0x7f7d6406d0f0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f7d78000f60 tx=0x7f7d78009040 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 msgr2=0x7f7d841e09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f7d7002a220 tx=0x7f7d700046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 shutdown_connections 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d6406ac30 0x7f7d6406d0f0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d84159770 0x7f7d84157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d84070430 0x7f7d841e09f0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 --2- 172.21.15.28:0/2001395113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d8406f3f0 0x7f7d840abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 >> 172.21.15.28:0/2001395113 conn(0x7f7d840b0270 msgr2=0x7f7d840cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:03.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 shutdown_connections 2024-09-13T17:17:03.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:03.835+0000 7f7d8b800640 1 -- 172.21.15.28:0/2001395113 wait complete. 2024-09-13T17:17:03.853 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:17:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.008+0000 7f2e1b200640 1 Processor -- start 2024-09-13T17:17:04.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.008+0000 7f2e1b200640 1 -- start start 2024-09-13T17:17:04.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14159a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e140b2870 0x7f2e140b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e140b32f0 con 0x7f2e1406c370 2024-09-13T17:17:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e140cbac0 con 0x7f2e140b2870 2024-09-13T17:17:04.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e140b35d0 con 0x7f2e140b0e10 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1a800640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f2e1406c370 0x7f2e140cbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60414/0 (socket says 172.21.15.28:60414) 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e1a800640 1 -- 172.21.15.28:0/496086231 learned_addr learned my addr 172.21.15.28:0/496086231 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e19e00640 1 --2- 172.21.15.28:0/496086231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.010+0000 7f2e19400640 1 --2- 172.21.15.28:0/496086231 >> v2:172.21.15.189:3300/0 conn(0x7f2e140b2870 0x7f2e140b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e18a00640 1 -- 172.21.15.28:0/496086231 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213755407 0 0) 0x7f2e140b32f0 con 0x7f2e1406c370 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e18a00640 1 -- 172.21.15.28:0/496086231 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dfc003520 con 0x7f2e1406c370 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e19e00640 1 -- 172.21.15.28:0/496086231 >> v2:172.21.15.189:3300/0 conn(0x7f2e140b2870 msgr2=0x7f2e140b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e19e00640 1 --2- 172.21.15.28:0/496086231 >> v2:172.21.15.189:3300/0 conn(0x7f2e140b2870 0x7f2e140b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e19e00640 1 -- 172.21.15.28:0/496086231 >> v1:172.21.15.189:6789/0 conn(0x7f2e1406c370 legacy=0x7f2e140cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.011+0000 7f2e19e00640 1 -- 172.21.15.28:0/496086231 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140b38b0 con 0x7f2e140b0e10 2024-09-13T17:17:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.012+0000 7f2e19e00640 1 --2- 172.21.15.28:0/496086231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14159a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e080097a0 tx=0x7f2e0802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a50df261d91a1b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.012+0000 7f2e18a00640 1 -- 172.21.15.28:0/496086231 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0802ae50 con 0x7f2e140b0e10 2024-09-13T17:17:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.012+0000 7f2e18a00640 1 -- 172.21.15.28:0/496086231 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08032440 con 0x7f2e140b0e10 2024-09-13T17:17:04.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.012+0000 7f2e18a00640 1 -- 172.21.15.28:0/496086231 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e080327d0 con 0x7f2e140b0e10 2024-09-13T17:17:04.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 -- 172.21.15.28:0/496086231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 msgr2=0x7f2e14159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 --2- 172.21.15.28:0/496086231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14159a60 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2e080097a0 tx=0x7f2e0802a100 comp rx=0 tx=0).stop 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 -- 172.21.15.28:0/496086231 shutdown_connections 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 --2- 172.21.15.28:0/496086231 >> v2:172.21.15.189:3300/0 conn(0x7f2e140b2870 0x7f2e140b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 --2- 172.21.15.28:0/496086231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14159a60 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 -- 172.21.15.28:0/496086231 >> 172.21.15.28:0/496086231 conn(0x7f2e14073470 msgr2=0x7f2e14069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 -- 172.21.15.28:0/496086231 shutdown_connections 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.013+0000 7f2e1b200640 1 -- 172.21.15.28:0/496086231 wait complete. 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.015+0000 7f2e1b200640 1 Processor -- start 2024-09-13T17:17:04.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.015+0000 7f2e1b200640 1 -- start start 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.015+0000 7f2e1b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1406c370 0x7f2e14132830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14130540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14159550 con 0x7f2e140b2870 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14131820 con 0x7f2e1406c370 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e14131a70 con 0x7f2e140b0e10 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e19e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1406c370 0x7f2e14132830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34612/0 (socket says 172.21.15.28:34612) 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 -- 172.21.15.28:0/1932872801 learned_addr learned my addr 172.21.15.28:0/1932872801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e19400640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14130540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 -- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 msgr2=0x7f2e14130540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14130540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 -- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1406c370 msgr2=0x7f2e14132830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1406c370 0x7f2e14132830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.016+0000 7f2e1a800640 1 -- 172.21.15.28:0/1932872801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08009400 con 0x7f2e140b2870 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e1a800640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e1000cb40 tx=0x7f2e1000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1000aca0 con 0x7f2e140b2870 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1000ae00 con 0x7f2e140b2870 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e14131d50 con 0x7f2e140b2870 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e10012620 con 0x7f2e140b2870 2024-09-13T17:17:04.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.017+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e14132310 con 0x7f2e140b2870 2024-09-13T17:17:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.018+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e14132530 con 0x7f2e140b2870 2024-09-13T17:17:04.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.019+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2e10007070 con 0x7f2e140b2870 2024-09-13T17:17:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.020+0000 7f2e0e000640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 0x7f2df806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.020+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2e100892b0 con 0x7f2e140b2870 2024-09-13T17:17:04.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.020+0000 7f2e19e00640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 0x7f2df806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:04.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.021+0000 7f2e19e00640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 0x7f2df806d0f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2e08034000 tx=0x7f2e0802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:04.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.024+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2e100544f0 con 0x7f2e140b2870 2024-09-13T17:17:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.216+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2e14159a60 con 0x7f2e140b2870 2024-09-13T17:17:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.216+0000 7f2e0e000640 1 -- 172.21.15.28:0/1932872801 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v127) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2e10057b10 con 0x7f2e140b2870 2024-09-13T17:17:04.218 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:17:04.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 msgr2=0x7f2df806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 0x7f2df806d0f0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2e08034000 tx=0x7f2e0802e000 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 msgr2=0x7f2e14159010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f2e1000cb40 tx=0x7f2e1000a060 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 shutdown_connections 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2df806ac30 0x7f2df806d0f0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e140b2870 0x7f2e14159010 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e140b0e10 0x7f2e14130540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 --2- 172.21.15.28:0/1932872801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e1406c370 0x7f2e14132830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 >> 172.21.15.28:0/1932872801 conn(0x7f2e14073470 msgr2=0x7f2e1406f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 shutdown_connections 2024-09-13T17:17:04.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:04.218+0000 7f2e1b200640 1 -- 172.21.15.28:0/1932872801 wait complete. 2024-09-13T17:17:04.236 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:17:04.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:04.236 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:05.062 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:17:05.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:05.063 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:17:05.098 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:17:05.098 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:05.136 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:17:05.137 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:05.137 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:05.174 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:17:05.174 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:17:05.174 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:05.213 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:17:05.214 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.368+0000 7f6a6d000640 1 Processor -- start 2024-09-13T17:17:05.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.368+0000 7f6a6d000640 1 -- start start 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.369+0000 7f6a6d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a681592a0 0x7f6a6815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a6d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6a6806c2d0 0x7f6a6806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a6d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6806cc70 con 0x7f6a6806f970 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a6d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6815b0b0 con 0x7f6a681592a0 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a6d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a68159d90 con 0x7f6a6806c2d0 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a681592a0 0x7f6a6815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a5d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a6806f970 0x7f6a68159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41598/0 (socket says 172.21.15.28:41598) 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a681592a0 0x7f6a6815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34628/0 (socket says 172.21.15.28:34628) 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a5d400640 1 -- 172.21.15.28:0/171714481 learned_addr learned my addr 172.21.15.28:0/171714481 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 -- 172.21.15.28:0/171714481 >> v2:172.21.15.189:3300/0 conn(0x7f6a6806c2d0 msgr2=0x7f6a6806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 --2- 172.21.15.28:0/171714481 >> v2:172.21.15.189:3300/0 conn(0x7f6a6806c2d0 0x7f6a6806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 -- 172.21.15.28:0/171714481 >> v1:172.21.15.28:6789/0 conn(0x7f6a6806f970 legacy=0x7f6a68159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65e00640 1 -- 172.21.15.28:0/171714481 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680b2040 con 0x7f6a681592a0 2024-09-13T17:17:05.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.370+0000 7f6a65400640 1 -- 172.21.15.28:0/171714481 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768426851 0 0) 0x7f6a6806cc70 con 0x7f6a6806f970 2024-09-13T17:17:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.371+0000 7f6a65e00640 1 --2- 172.21.15.28:0/171714481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a681592a0 0x7f6a6815ab70 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b1dd0 tx=0x7f6a5802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde7a5d0eed3b7fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a65400640 1 -- 172.21.15.28:0/171714481 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58032440 con 0x7f6a681592a0 2024-09-13T17:17:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a65400640 1 -- 172.21.15.28:0/171714481 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580325a0 con 0x7f6a681592a0 2024-09-13T17:17:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a65400640 1 -- 172.21.15.28:0/171714481 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58032930 con 0x7f6a681592a0 2024-09-13T17:17:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 -- 172.21.15.28:0/171714481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a681592a0 msgr2=0x7f6a6815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 --2- 172.21.15.28:0/171714481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a681592a0 0x7f6a6815ab70 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f6a680b1dd0 tx=0x7f6a5802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 -- 172.21.15.28:0/171714481 shutdown_connections 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 --2- 172.21.15.28:0/171714481 >> v2:172.21.15.189:3300/0 conn(0x7f6a6806c2d0 0x7f6a6806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 --2- 172.21.15.28:0/171714481 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a681592a0 0x7f6a6815ab70 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 -- 172.21.15.28:0/171714481 >> 172.21.15.28:0/171714481 conn(0x7f6a68069e80 msgr2=0x7f6a6806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 -- 172.21.15.28:0/171714481 shutdown_connections 2024-09-13T17:17:05.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.372+0000 7f6a6d000640 1 -- 172.21.15.28:0/171714481 wait complete. 2024-09-13T17:17:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.374+0000 7f6a6d000640 1 Processor -- start 2024-09-13T17:17:05.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.374+0000 7f6a6d000640 1 -- start start 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 0x7f6a68156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 0x7f6a68147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a68158150 con 0x7f6a6806c2d0 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681574d0 con 0x7f6a681592a0 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a6d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a681577b0 con 0x7f6a6806f970 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34632/0 (socket says 172.21.15.28:34632) 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.375+0000 7f6a65e00640 1 -- 172.21.15.28:0/2117317073 learned_addr learned my addr 172.21.15.28:0/2117317073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a5d400640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 0x7f6a68156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a66800640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 0x7f6a68147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 -- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 msgr2=0x7f6a68156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 0x7f6a68156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 -- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 msgr2=0x7f6a68147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 0x7f6a68147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 -- 172.21.15.28:0/2117317073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a58009400 con 0x7f6a6806c2d0 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a5d400640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 0x7f6a68156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a66800640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 0x7f6a68147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:05.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a65e00640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000c00 tx=0x7f6a580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a58030b80 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a58030d00 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a5803b640 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a68157a90 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.376+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6815c6b0 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.377+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a680b3740 con 0x7f6a6806c2d0 2024-09-13T17:17:05.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.378+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a58039040 con 0x7f6a6806c2d0 2024-09-13T17:17:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.379+0000 7f6a5ea00640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 0x7f6a4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.379+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6a580a9d70 con 0x7f6a6806c2d0 2024-09-13T17:17:05.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.380+0000 7f6a5d400640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 0x7f6a4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.380+0000 7f6a5d400640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 0x7f6a4406d0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6a680ca730 tx=0x7f6a50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.384+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6a58074f00 con 0x7f6a6806c2d0 2024-09-13T17:17:05.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.597+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a6815b440 con 0x7f6a6806c2d0 2024-09-13T17:17:05.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.597+0000 7f6a5ea00640 1 -- 172.21.15.28:0/2117317073 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7f6a58037090 con 0x7f6a6806c2d0 2024-09-13T17:17:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:05.599 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:17:05.600 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 msgr2=0x7f6a4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 0x7f6a4406d0c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f6a680ca730 tx=0x7f6a50009040 comp rx=0 tx=0).stop 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 msgr2=0x7f6a6815c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7f6a58000c00 tx=0x7f6a580046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 shutdown_connections 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a4406ac00 0x7f6a4406d0c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a681592a0 0x7f6a68147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a6806f970 0x7f6a68156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 --2- 172.21.15.28:0/2117317073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a6806c2d0 0x7f6a6815c170 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 >> 172.21.15.28:0/2117317073 conn(0x7f6a68069e80 msgr2=0x7f6a680acc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 shutdown_connections 2024-09-13T17:17:05.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.600+0000 7f6a6d000640 1 -- 172.21.15.28:0/2117317073 wait complete. 2024-09-13T17:17:05.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.771+0000 7f3a50a00640 1 Processor -- start 2024-09-13T17:17:05.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.771+0000 7f3a50a00640 1 -- start start 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a50a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 0x7f3a4c0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a50a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0c3850 con 0x7f3a4c159440 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a50a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c15a930 con 0x7f3a4c159820 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a50a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c159d20 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a4b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3a4c159440 0x7f3a4c0b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41612/0 (socket says 172.21.15.28:41612) 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a4b400640 1 -- 172.21.15.28:0/215380367 learned_addr learned my addr 172.21.15.28:0/215380367 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:05.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.772+0000 7f3a43e00640 1 --2- 172.21.15.28:0/215380367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 0x7f3a4c0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3083270272 0 0) 0x7f3a4c0c3850 con 0x7f3a4c159440 2024-09-13T17:17:05.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30003520 con 0x7f3a4c159440 2024-09-13T17:17:05.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3548428459 0 0) 0x7f3a30003520 con 0x7f3a4c159440 2024-09-13T17:17:05.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a43e00640 1 -- 172.21.15.28:0/215380367 >> v1:172.21.15.189:6789/0 conn(0x7f3a4c159820 legacy=0x7f3a4c15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a43e00640 1 -- 172.21.15.28:0/215380367 >> v1:172.21.15.28:6789/0 conn(0x7f3a4c159440 legacy=0x7f3a4c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.773+0000 7f3a43e00640 1 -- 172.21.15.28:0/215380367 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c15a000 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.774+0000 7f3a43e00640 1 --2- 172.21.15.28:0/215380367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 0x7f3a4c0b22e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a340097d0 tx=0x7f3a3402a140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0b7eeca74918ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.774+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c001670 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.774+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3402abd0 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.774+0000 7f3a4aa00640 1 -- 172.21.15.28:0/215380367 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a34032440 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 -- 172.21.15.28:0/215380367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 msgr2=0x7f3a4c0b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 --2- 172.21.15.28:0/215380367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 0x7f3a4c0b22e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3a340097d0 tx=0x7f3a3402a140 comp rx=0 tx=0).stop 2024-09-13T17:17:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 -- 172.21.15.28:0/215380367 shutdown_connections 2024-09-13T17:17:05.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 --2- 172.21.15.28:0/215380367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c0b0b50 0x7f3a4c0b22e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 -- 172.21.15.28:0/215380367 >> 172.21.15.28:0/215380367 conn(0x7f3a4c072e40 msgr2=0x7f3a4c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 -- 172.21.15.28:0/215380367 shutdown_connections 2024-09-13T17:17:05.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.775+0000 7f3a50a00640 1 -- 172.21.15.28:0/215380367 wait complete. 2024-09-13T17:17:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.777+0000 7f3a50a00640 1 Processor -- start 2024-09-13T17:17:05.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.777+0000 7f3a50a00640 1 -- start start 2024-09-13T17:17:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4c0b0b50 0x7f3a4c144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c159440 0x7f3a4c1e2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c142580 con 0x7f3a4c159820 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c1445b0 con 0x7f3a4c0b0b50 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a50a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c141900 con 0x7f3a4c159440 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a4be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34638/0 (socket says 172.21.15.28:34638) 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a4be00640 1 -- 172.21.15.28:0/3915921171 learned_addr learned my addr 172.21.15.28:0/3915921171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a4b400640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4c0b0b50 0x7f3a4c144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.778+0000 7f3a43e00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c159440 0x7f3a4c1e2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 -- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c159440 msgr2=0x7f3a4c1e2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c159440 0x7f3a4c1e2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 -- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4c0b0b50 msgr2=0x7f3a4c144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4c0b0b50 0x7f3a4c144070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 -- 172.21.15.28:0/3915921171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a34009400 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a4be00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c7b0 tx=0x7f3a4400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44003da0 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4400ce60 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a44005290 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.779+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c141be0 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.780+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c1421a0 con 0x7f3a4c159820 2024-09-13T17:17:05.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.781+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c1426c0 con 0x7f3a4c159820 2024-09-13T17:17:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.781+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a440053f0 con 0x7f3a4c159820 2024-09-13T17:17:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.782+0000 7f3a43400640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 0x7f3a2806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.782+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f3a44088ae0 con 0x7f3a4c159820 2024-09-13T17:17:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.783+0000 7f3a4b400640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 0x7f3a2806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:05.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.783+0000 7f3a4b400640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 0x7f3a2806d0c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000f60 tx=0x7f3a3c001e40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:05.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.787+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3a44053d20 con 0x7f3a4c159820 2024-09-13T17:17:05.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.967+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a4c0b22e0 con 0x7f3a4c159820 2024-09-13T17:17:05.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.968+0000 7f3a43400640 1 -- 172.21.15.28:0/3915921171 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f3a44017070 con 0x7f3a4c159820 2024-09-13T17:17:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:05.971 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:53.097532+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.970+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 msgr2=0x7f3a2806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.970+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 0x7f3a2806d0c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c000f60 tx=0x7f3a3c001e40 comp rx=0 tx=0).stop 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.970+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 msgr2=0x7f3a4c06b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.970+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f3a4400c7b0 tx=0x7f3a4400cc70 comp rx=0 tx=0).stop 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 shutdown_connections 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a2806ac00 0x7f3a2806d0c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a4c159820 0x7f3a4c06b5d0 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a4c159440 0x7f3a4c1e2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 --2- 172.21.15.28:0/3915921171 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a4c0b0b50 0x7f3a4c144070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 >> 172.21.15.28:0/3915921171 conn(0x7f3a4c072e40 msgr2=0x7f3a4c1578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 shutdown_connections 2024-09-13T17:17:05.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:05.971+0000 7f3a50a00640 1 -- 172.21.15.28:0/3915921171 wait complete. 2024-09-13T17:17:05.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:16:27.833325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:16:28.661422+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:17:05.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:17:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.146+0000 7fa107e00640 1 Processor -- start 2024-09-13T17:17:06.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.146+0000 7fa107e00640 1 -- start start 2024-09-13T17:17:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.147+0000 7fa107e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1000b2400 0x7fa1000ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.147+0000 7fa107e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa100070470 con 0x7fa10015c9a0 2024-09-13T17:17:06.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.147+0000 7fa107e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1000ace30 con 0x7fa1000ad940 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.147+0000 7fa107e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1000ad110 con 0x7fa1000b2400 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa105400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa10015c9a0 0x7fa1000b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41614/0 (socket says 172.21.15.28:41614) 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa105400640 1 -- 172.21.15.28:0/2816486222 learned_addr learned my addr 172.21.15.28:0/2816486222 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa104a00640 1 --2- 172.21.15.28:0/2816486222 >> v2:172.21.15.28:3300/0 conn(0x7fa1000b2400 0x7fa1000ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa104a00640 1 -- 172.21.15.28:0/2816486222 >> v1:172.21.15.189:6789/0 conn(0x7fa1000ad940 legacy=0x7fa10006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa104a00640 1 -- 172.21.15.28:0/2816486222 >> v1:172.21.15.28:6789/0 conn(0x7fa10015c9a0 legacy=0x7fa1000b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.148+0000 7fa104a00640 1 -- 172.21.15.28:0/2816486222 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10006b170 con 0x7fa1000b2400 2024-09-13T17:17:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.149+0000 7fa104a00640 1 --2- 172.21.15.28:0/2816486222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1000b2400 0x7fa1000ac850 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0097a0 tx=0x7fa0ec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b090340f5455bd48 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.149+0000 7fa0ffe00640 1 -- 172.21.15.28:0/2816486222 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec032440 con 0x7fa1000b2400 2024-09-13T17:17:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.149+0000 7fa0ffe00640 1 -- 172.21.15.28:0/2816486222 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec0325a0 con 0x7fa1000b2400 2024-09-13T17:17:06.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.149+0000 7fa0ffe00640 1 -- 172.21.15.28:0/2816486222 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec032930 con 0x7fa1000b2400 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 -- 172.21.15.28:0/2816486222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1000b2400 msgr2=0x7fa1000ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 --2- 172.21.15.28:0/2816486222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1000b2400 0x7fa1000ac850 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0097a0 tx=0x7fa0ec02a230 comp rx=0 tx=0).stop 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 -- 172.21.15.28:0/2816486222 shutdown_connections 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 --2- 172.21.15.28:0/2816486222 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1000b2400 0x7fa1000ac850 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 -- 172.21.15.28:0/2816486222 >> 172.21.15.28:0/2816486222 conn(0x7fa1000b0270 msgr2=0x7fa1000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 -- 172.21.15.28:0/2816486222 shutdown_connections 2024-09-13T17:17:06.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.150+0000 7fa107e00640 1 -- 172.21.15.28:0/2816486222 wait complete. 2024-09-13T17:17:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.152+0000 7fa107e00640 1 Processor -- start 2024-09-13T17:17:06.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.152+0000 7fa107e00640 1 -- start start 2024-09-13T17:17:06.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1000ad940 0x7fa1000ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1000b2400 0x7fa1001546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1000cd060 con 0x7fa10015c9a0 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10013d020 con 0x7fa1000b2400 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa107e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10013d2c0 con 0x7fa1000ad940 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34646/0 (socket says 172.21.15.28:34646) 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 -- 172.21.15.28:0/354158745 learned_addr learned my addr 172.21.15.28:0/354158745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105400640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1000ad940 0x7fa1000ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa104a00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1000b2400 0x7fa1001546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 -- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1000ad940 msgr2=0x7fa1000ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1000ad940 0x7fa1000ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 -- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1000b2400 msgr2=0x7fa1001546b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.153+0000 7fa105e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1000b2400 0x7fa1001546b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa105e00640 1 -- 172.21.15.28:0/354158745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec009400 con 0x7fa10015c9a0 2024-09-13T17:17:06.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa105e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800c7a0 tx=0x7fa0f800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8003b70 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f8003cf0 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8003ea0 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100070930 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.154+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10013d880 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.155+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10015cd80 con 0x7fa10015c9a0 2024-09-13T17:17:06.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.156+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0f800aa90 con 0x7fa10015c9a0 2024-09-13T17:17:06.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.158+0000 7fa0fd600640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 0x7fa0e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.158+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa0f8089160 con 0x7fa10015c9a0 2024-09-13T17:17:06.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.158+0000 7fa105400640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 0x7fa0e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.159+0000 7fa105400640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 0x7fa0e006d0c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4005880 tx=0x7fa0f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.162+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0f8054310 con 0x7fa10015c9a0 2024-09-13T17:17:06.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.340+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa100071300 con 0x7fa10015c9a0 2024-09-13T17:17:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.341+0000 7fa0fd600640 1 -- 172.21.15.28:0/354158745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0f8057930 con 0x7fa10015c9a0 2024-09-13T17:17:06.343 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 msgr2=0x7fa0e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 0x7fa0e006d0c0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4005880 tx=0x7fa0f4009040 comp rx=0 tx=0).stop 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 msgr2=0x7fa10013f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fa0f800c7a0 tx=0x7fa0f800cc60 comp rx=0 tx=0).stop 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 shutdown_connections 2024-09-13T17:17:06.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0e006ac00 0x7fa0e006d0c0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa10015c9a0 0x7fa10013f010 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1000b2400 0x7fa1001546b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 --2- 172.21.15.28:0/354158745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1000ad940 0x7fa1000ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 >> 172.21.15.28:0/354158745 conn(0x7fa1000b0270 msgr2=0x7fa1000b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 shutdown_connections 2024-09-13T17:17:06.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.343+0000 7fa107e00640 1 -- 172.21.15.28:0/354158745 wait complete. 2024-09-13T17:17:06.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:17:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.517+0000 7f50c1c00640 1 Processor -- start 2024-09-13T17:17:06.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.517+0000 7f50c1c00640 1 -- start start 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50c1c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 0x7f50bc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50c1c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50bc06f3f0 0x7f50bc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50c1c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50bc06f920 con 0x7f50bc0add00 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50c1c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc15b290 con 0x7f50bc06f3f0 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50c1c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50bc15b570 con 0x7f50bc159770 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.518+0000 7f50b3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50bc06f3f0 0x7f50bc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f50bc06f3f0 0x7f50bc070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34662/0 (socket says 172.21.15.28:34662) 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 -- 172.21.15.28:0/3319445023 learned_addr learned my addr 172.21.15.28:0/3319445023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50baa00640 1 --2- 172.21.15.28:0/3319445023 >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 0x7f50bc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 -- 172.21.15.28:0/3319445023 >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 msgr2=0x7f50bc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 --2- 172.21.15.28:0/3319445023 >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 0x7f50bc06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 -- 172.21.15.28:0/3319445023 >> v1:172.21.15.189:6789/0 conn(0x7f50bc0add00 legacy=0x7f50bc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50b3e00640 1 -- 172.21.15.28:0/3319445023 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50bc15b850 con 0x7f50bc06f3f0 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.519+0000 7f50baa00640 1 --2- 172.21.15.28:0/3319445023 >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 0x7f50bc06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.520+0000 7f50b3e00640 1 --2- 172.21.15.28:0/3319445023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc070260 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f50bc067f80 tx=0x7f50a402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f8c58226faf5c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.520+0000 7f50ba000640 1 -- 172.21.15.28:0/3319445023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4032440 con 0x7f50bc06f3f0 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.520+0000 7f50ba000640 1 -- 172.21.15.28:0/3319445023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a40325a0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.520+0000 7f50ba000640 1 -- 172.21.15.28:0/3319445023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4032930 con 0x7f50bc06f3f0 2024-09-13T17:17:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 -- 172.21.15.28:0/3319445023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 msgr2=0x7f50bc070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 --2- 172.21.15.28:0/3319445023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc070260 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f50bc067f80 tx=0x7f50a402a160 comp rx=0 tx=0).stop 2024-09-13T17:17:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 -- 172.21.15.28:0/3319445023 shutdown_connections 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 --2- 172.21.15.28:0/3319445023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc070260 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 --2- 172.21.15.28:0/3319445023 >> v2:172.21.15.189:3300/0 conn(0x7f50bc159770 0x7f50bc06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 -- 172.21.15.28:0/3319445023 >> 172.21.15.28:0/3319445023 conn(0x7f50bc0b0270 msgr2=0x7f50bc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 -- 172.21.15.28:0/3319445023 shutdown_connections 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.521+0000 7f50c1c00640 1 -- 172.21.15.28:0/3319445023 wait complete. 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.523+0000 7f50c1c00640 1 Processor -- start 2024-09-13T17:17:06.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.523+0000 7f50c1c00640 1 -- start start 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 0x7f50bc153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 0x7f50bc14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0cc110 con 0x7f50bc06f3f0 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc147520 con 0x7f50bc0add00 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50c1c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50bc0cb770 con 0x7f50bc159770 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34666/0 (socket says 172.21.15.28:34666) 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 -- 172.21.15.28:0/1147056866 learned_addr learned my addr 172.21.15.28:0/1147056866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50b3e00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 0x7f50bc153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50bb400640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 0x7f50bc14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 -- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 msgr2=0x7f50bc14c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 0x7f50bc14c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 -- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 msgr2=0x7f50bc153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 0x7f50bc153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50baa00640 1 -- 172.21.15.28:0/1147056866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a40094d0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50b3e00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 0x7f50bc153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.524+0000 7f50bb400640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 0x7f50bc14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50baa00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00d7d0 tx=0x7f50ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac004030 con 0x7f50bc06f3f0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ac004190 con 0x7f50bc06f3f0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac004520 con 0x7f50bc06f3f0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50bc0cba50 con 0x7f50bc06f3f0 2024-09-13T17:17:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.525+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50bc1524f0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.526+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50bc0cc380 con 0x7f50bc06f3f0 2024-09-13T17:17:06.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.527+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f50ac00b6c0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.528+0000 7f50b2a00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 0x7f509806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.528+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50ac089560 con 0x7f50bc06f3f0 2024-09-13T17:17:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.528+0000 7f50b3e00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 0x7f509806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.529+0000 7f50b3e00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 0x7f509806d0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50a4034000 tx=0x7f50a402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.532+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f50ac0546f0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.712+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50bc0cf2f0 con 0x7f50bc06f3f0 2024-09-13T17:17:06.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.713+0000 7f50b2a00640 1 -- 172.21.15.28:0/1147056866 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50ac057d10 con 0x7f50bc06f3f0 2024-09-13T17:17:06.715 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 msgr2=0x7f509806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 0x7f509806d0f0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f50a4034000 tx=0x7f50a402e000 comp rx=0 tx=0).stop 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 msgr2=0x7f50bc146fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00d7d0 tx=0x7f50ac00dc90 comp rx=0 tx=0).stop 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 shutdown_connections 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f509806ac30 0x7f509806d0f0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f50bc159770 0x7f50bc14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50bc0add00 0x7f50bc153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 --2- 172.21.15.28:0/1147056866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f50bc06f3f0 0x7f50bc146fe0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 >> 172.21.15.28:0/1147056866 conn(0x7f50bc0b0270 msgr2=0x7f50bc070fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:06.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 shutdown_connections 2024-09-13T17:17:06.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.715+0000 7f50c1c00640 1 -- 172.21.15.28:0/1147056866 wait complete. 2024-09-13T17:17:06.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:17:06.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.888+0000 7fbd1f200640 1 Processor -- start 2024-09-13T17:17:06.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.888+0000 7fbd1f200640 1 -- start start 2024-09-13T17:17:06.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.889+0000 7fbd1f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbd180ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd1f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd18070470 con 0x7fbd180ad940 2024-09-13T17:17:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd1f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180ace30 con 0x7fbd1815c9a0 2024-09-13T17:17:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd1f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180ad110 con 0x7fbd180b2400 2024-09-13T17:17:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd1d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd180ad940 0x7fbd1806bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41620/0 (socket says 172.21.15.28:41620) 2024-09-13T17:17:06.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd1d200640 1 -- 172.21.15.28:0/1729164157 learned_addr learned my addr 172.21.15.28:0/1729164157 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbd180ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530962782 0 0) 0x7fbd18070470 con 0x7fbd180ad940 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.890+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcfc003520 con 0x7fbd180ad940 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366673056 0 0) 0x7fbcfc003520 con 0x7fbd180ad940 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 msgr2=0x7fbd180ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbd180ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 >> v1:172.21.15.189:6789/0 conn(0x7fbd1815c9a0 legacy=0x7fbd180b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1806b170 con 0x7fbd180ad940 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fbd100019f0 con 0x7fbd180ad940 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 >> v1:172.21.15.28:6789/0 conn(0x7fbd180ad940 legacy=0x7fbd1806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0039a0 0x7fbcfc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc0047c0 0x7fbcfc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbcfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180ad110 con 0x7fbcfc0047c0 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc00baf0 con 0x7fbcfc0039a0 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcfc00bcb0 con 0x7fbd180b2400 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc0047c0 0x7fbcfc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd17e00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0039a0 0x7fbcfc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd1d200640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbcfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd0fe00640 1 -- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 msgr2=0x7fbcfc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:06.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbcfc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd0fe00640 1 -- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0039a0 msgr2=0x7fbcfc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.891+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0039a0 0x7fbcfc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.892+0000 7fbd0fe00640 1 -- 172.21.15.28:0/1729164157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc003520 con 0x7fbcfc0047c0 2024-09-13T17:17:06.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.892+0000 7fbd1d200640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd180b2400 0x7fbcfc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.892+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc0047c0 0x7fbcfc004c40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009a90 tx=0x7fbcfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0d668343580a9c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00031400 con 0x7fbcfc0047c0 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0002d050 con 0x7fbcfc0047c0 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd17400640 1 -- 172.21.15.28:0/1729164157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd00031780 con 0x7fbcfc0047c0 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 -- 172.21.15.28:0/1729164157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc0047c0 msgr2=0x7fbcfc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 --2- 172.21.15.28:0/1729164157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc0047c0 0x7fbcfc004c40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7fbd00009a90 tx=0x7fbcfc0057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1d200640 1 -- 172.21.15.28:0/1729164157 reap_dead start 2024-09-13T17:17:06.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 -- 172.21.15.28:0/1729164157 shutdown_connections 2024-09-13T17:17:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 -- 172.21.15.28:0/1729164157 >> 172.21.15.28:0/1729164157 conn(0x7fbd180b0270 msgr2=0x7fbd180b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 -- 172.21.15.28:0/1729164157 shutdown_connections 2024-09-13T17:17:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.893+0000 7fbd1f200640 1 -- 172.21.15.28:0/1729164157 wait complete. 2024-09-13T17:17:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.895+0000 7fbd1f200640 1 Processor -- start 2024-09-13T17:17:06.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.895+0000 7fbd1f200640 1 -- start start 2024-09-13T17:17:06.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0047c0 0x7fbd181532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd181f3390 0x7fbd1815f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18153800 con 0x7fbd181f3810 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1815e580 con 0x7fbcfc0047c0 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd18154a00 con 0x7fbd181f3390 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34684/0 (socket says 172.21.15.28:34684) 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.896+0000 7fbd1d200640 1 -- 172.21.15.28:0/2495649420 learned_addr learned my addr 172.21.15.28:0/2495649420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd0fe00640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd181f3390 0x7fbd1815f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd17e00640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0047c0 0x7fbd181532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 -- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd181f3390 msgr2=0x7fbd1815f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd181f3390 0x7fbd1815f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 -- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0047c0 msgr2=0x7fbd181532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0047c0 0x7fbd181532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 -- 172.21.15.28:0/2495649420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18070930 con 0x7fbd181f3810 2024-09-13T17:17:06.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1d200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806f380 tx=0x7fbd1000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10007ae0 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1000eda0 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18156e90 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.897+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10014690 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.898+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd181543b0 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.899+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1815cd80 con 0x7fbd181f3810 2024-09-13T17:17:06.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.899+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbd100147f0 con 0x7fbd181f3810 2024-09-13T17:17:06.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.900+0000 7fbd14c00640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 0x7fbcf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:06.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.900+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbd10089140 con 0x7fbd181f3810 2024-09-13T17:17:06.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.901+0000 7fbd17e00640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 0x7fbcf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:06.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.902+0000 7fbd17e00640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 0x7fbcf406d0c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbd080017e0 tx=0x7fbd08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:06.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:06.904+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbd10054380 con 0x7fbd181f3810 2024-09-13T17:17:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.083+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbd18154b40 con 0x7fbd181f3810 2024-09-13T17:17:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.084+0000 7fbd14c00640 1 -- 172.21.15.28:0/2495649420 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbd100579a0 con 0x7fbd181f3810 2024-09-13T17:17:07.086 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 msgr2=0x7fbcf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 0x7fbcf406d0c0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fbd080017e0 tx=0x7fbd08009040 comp rx=0 tx=0).stop 2024-09-13T17:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 msgr2=0x7fbd1815f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7fbd1806f380 tx=0x7fbd1000e530 comp rx=0 tx=0).stop 2024-09-13T17:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 shutdown_connections 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcf406ac00 0x7fbcf406d0c0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd181f3810 0x7fbd1815f610 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd181f3390 0x7fbd1815f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 --2- 172.21.15.28:0/2495649420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbcfc0047c0 0x7fbd181532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 >> 172.21.15.28:0/2495649420 conn(0x7fbd180b0270 msgr2=0x7fbd1815b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 shutdown_connections 2024-09-13T17:17:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.086+0000 7fbd1f200640 1 -- 172.21.15.28:0/2495649420 wait complete. 2024-09-13T17:17:07.104 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:07.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.260+0000 7f3562400640 1 Processor -- start 2024-09-13T17:17:07.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.260+0000 7f3562400640 1 -- start start 2024-09-13T17:17:07.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f3562400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f355c15e860 0x7f355c161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f3562400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06cde0 0x7f355c070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f3562400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355c0b35c0 con 0x7f355c0711d0 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f3562400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c1618f0 con 0x7f355c06cde0 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f3562400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355c0b38a0 con 0x7f355c15e860 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f355aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06cde0 0x7f355c070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f355aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f355c06cde0 0x7f355c070c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34700/0 (socket says 172.21.15.28:34700) 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.261+0000 7f355aa00640 1 -- 172.21.15.28:0/240867724 learned_addr learned my addr 172.21.15.28:0/240867724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.262+0000 7f355b400640 1 --2- 172.21.15.28:0/240867724 >> v2:172.21.15.189:3300/0 conn(0x7f355c15e860 0x7f355c161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.262+0000 7f355aa00640 1 -- 172.21.15.28:0/240867724 >> v2:172.21.15.189:3300/0 conn(0x7f355c15e860 msgr2=0x7f355c161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.262+0000 7f355aa00640 1 --2- 172.21.15.28:0/240867724 >> v2:172.21.15.189:3300/0 conn(0x7f355c15e860 0x7f355c161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.262+0000 7f355aa00640 1 -- 172.21.15.28:0/240867724 >> v1:172.21.15.189:6789/0 conn(0x7f355c0711d0 legacy=0x7f355c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:07.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.262+0000 7f355aa00640 1 -- 172.21.15.28:0/240867724 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c161230 con 0x7f355c06cde0 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.263+0000 7f355aa00640 1 --2- 172.21.15.28:0/240867724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06cde0 0x7f355c070c90 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f355c06ac30 tx=0x7f354c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3af9c996e160a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.263+0000 7f355a000640 1 -- 172.21.15.28:0/240867724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c032440 con 0x7f355c06cde0 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.263+0000 7f355a000640 1 -- 172.21.15.28:0/240867724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354c0325a0 con 0x7f355c06cde0 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.263+0000 7f355a000640 1 -- 172.21.15.28:0/240867724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354c032930 con 0x7f355c06cde0 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 -- 172.21.15.28:0/240867724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06cde0 msgr2=0x7f355c070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 --2- 172.21.15.28:0/240867724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06cde0 0x7f355c070c90 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f355c06ac30 tx=0x7f354c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 -- 172.21.15.28:0/240867724 shutdown_connections 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 --2- 172.21.15.28:0/240867724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c06cde0 0x7f355c070c90 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 --2- 172.21.15.28:0/240867724 >> v2:172.21.15.189:3300/0 conn(0x7f355c15e860 0x7f355c161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 -- 172.21.15.28:0/240867724 >> 172.21.15.28:0/240867724 conn(0x7f355c069e80 msgr2=0x7f355c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 -- 172.21.15.28:0/240867724 shutdown_connections 2024-09-13T17:17:07.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.264+0000 7f3562400640 1 -- 172.21.15.28:0/240867724 wait complete. 2024-09-13T17:17:07.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.266+0000 7f3562400640 1 Processor -- start 2024-09-13T17:17:07.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.266+0000 7f3562400640 1 -- start start 2024-09-13T17:17:07.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 0x7f355c145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c0711d0 0x7f355c15d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c156cd0 con 0x7f355c15e860 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c15a5e0 con 0x7f355c0711d0 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f3562400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355c159d40 con 0x7f355c06cde0 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f355be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f355be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34702/0 (socket says 172.21.15.28:34702) 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f355be00640 1 -- 172.21.15.28:0/4150201927 learned_addr learned my addr 172.21.15.28:0/4150201927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.267+0000 7f355b400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 0x7f355c145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355aa00640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c0711d0 0x7f355c15d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 -- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 msgr2=0x7f355c145a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 0x7f355c145a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 -- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c0711d0 msgr2=0x7f355c15d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c0711d0 0x7f355c15d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 -- 172.21.15.28:0/4150201927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c0094d0 con 0x7f355c15e860 2024-09-13T17:17:07.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355b400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 0x7f355c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:07.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f355be00640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f355400b970 tx=0x7f355400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355400cb40 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355400cca0 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3554014680 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c15a020 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.268+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c1464b0 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.269+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c0ce0c0 con 0x7f355c15e860 2024-09-13T17:17:07.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.270+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f355401d3b0 con 0x7f355c15e860 2024-09-13T17:17:07.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.271+0000 7f3553400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 0x7f353806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.271+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f35540891a0 con 0x7f355c15e860 2024-09-13T17:17:07.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.272+0000 7f355b400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 0x7f353806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.272+0000 7f355b400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 0x7f353806d0c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3544006fd0 tx=0x7f3544008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.275+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3554054330 con 0x7f355c15e860 2024-09-13T17:17:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.491+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f355c06f5c0 con 0x7f355c15e860 2024-09-13T17:17:07.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.492+0000 7f3553400640 1 -- 172.21.15.28:0/4150201927 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7f3554057950 con 0x7f355c15e860 2024-09-13T17:17:07.494 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:07.494 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:17:07.494 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 msgr2=0x7f353806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 0x7f353806d0c0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f3544006fd0 tx=0x7f3544008040 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 msgr2=0x7f355c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f355400b970 tx=0x7f355400be30 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 shutdown_connections 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f353806ac00 0x7f353806d0c0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f355c15e860 0x7f355c15d6f0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f355c0711d0 0x7f355c15d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 --2- 172.21.15.28:0/4150201927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f355c06cde0 0x7f355c145a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 >> 172.21.15.28:0/4150201927 conn(0x7f355c069e80 msgr2=0x7f355c162390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:07.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 shutdown_connections 2024-09-13T17:17:07.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.494+0000 7f3562400640 1 -- 172.21.15.28:0/4150201927 wait complete. 2024-09-13T17:17:07.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.665+0000 7fc10de00640 1 Processor -- start 2024-09-13T17:17:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.665+0000 7fc10de00640 1 -- start start 2024-09-13T17:17:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc10de00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc1081592a0 0x7fc10815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc10de00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc108159d90 con 0x7fc10806f970 2024-09-13T17:17:07.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc10de00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1080b2040 con 0x7fc10806c2d0 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc10de00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc10815b0b0 con 0x7fc1081592a0 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc0fe000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc10806f970 0x7fc108159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41626/0 (socket says 172.21.15.28:41626) 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.666+0000 7fc0fe000640 1 -- 172.21.15.28:0/4280820046 learned_addr learned my addr 172.21.15.28:0/4280820046 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106a00640 1 --2- 172.21.15.28:0/4280820046 >> v2:172.21.15.189:3300/0 conn(0x7fc1081592a0 0x7fc10815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106000640 1 -- 172.21.15.28:0/4280820046 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 277501288 0 0) 0x7fc108159d90 con 0x7fc10806f970 2024-09-13T17:17:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106000640 1 -- 172.21.15.28:0/4280820046 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ec005730 con 0x7fc10806f970 2024-09-13T17:17:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106a00640 1 -- 172.21.15.28:0/4280820046 >> v1:172.21.15.189:6789/0 conn(0x7fc10806c2d0 legacy=0x7fc1080b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106a00640 1 -- 172.21.15.28:0/4280820046 >> v1:172.21.15.28:6789/0 conn(0x7fc10806f970 legacy=0x7fc108159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.667+0000 7fc106a00640 1 -- 172.21.15.28:0/4280820046 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc108159ff0 con 0x7fc1081592a0 2024-09-13T17:17:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.668+0000 7fc106a00640 1 --2- 172.21.15.28:0/4280820046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc10815ab70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80097a0 tx=0x7fc0f802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf0a96faa454e6b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc106000640 1 -- 172.21.15.28:0/4280820046 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8032440 con 0x7fc1081592a0 2024-09-13T17:17:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc106000640 1 -- 172.21.15.28:0/4280820046 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f80325a0 con 0x7fc1081592a0 2024-09-13T17:17:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc106000640 1 -- 172.21.15.28:0/4280820046 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8032930 con 0x7fc1081592a0 2024-09-13T17:17:07.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 -- 172.21.15.28:0/4280820046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 msgr2=0x7fc10815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 --2- 172.21.15.28:0/4280820046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc10815ab70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc0f80097a0 tx=0x7fc0f802a230 comp rx=0 tx=0).stop 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 -- 172.21.15.28:0/4280820046 shutdown_connections 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 --2- 172.21.15.28:0/4280820046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc10815ab70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 -- 172.21.15.28:0/4280820046 >> 172.21.15.28:0/4280820046 conn(0x7fc108069e80 msgr2=0x7fc10806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 -- 172.21.15.28:0/4280820046 shutdown_connections 2024-09-13T17:17:07.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.669+0000 7fc10de00640 1 -- 172.21.15.28:0/4280820046 wait complete. 2024-09-13T17:17:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.671+0000 7fc10de00640 1 Processor -- start 2024-09-13T17:17:07.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.671+0000 7fc10de00640 1 -- start start 2024-09-13T17:17:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 0x7fc10814d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc1081f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10814b7e0 con 0x7fc10806c2d0 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc10814bd60 con 0x7fc1081592a0 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc10de00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1081f6760 con 0x7fc10806f970 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc106a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc106a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34708/0 (socket says 172.21.15.28:34708) 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.672+0000 7fc106a00640 1 -- 172.21.15.28:0/154499387 learned_addr learned my addr 172.21.15.28:0/154499387 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc0fe000640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 0x7fc10814d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc107400640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc1081f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 -- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 msgr2=0x7fc10814d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 0x7fc10814d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 -- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 msgr2=0x7fc1081f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc1081f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 -- 172.21.15.28:0/154499387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8009400 con 0x7fc10806c2d0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc0fe000640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 0x7fc10814d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc106a00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8035040 tx=0x7fc0f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8006e10 con 0x7fc10806c2d0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f8032e90 con 0x7fc10806c2d0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.673+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1081f6a40 con 0x7fc10806c2d0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.674+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0f8030be0 con 0x7fc10806c2d0 2024-09-13T17:17:07.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.674+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1081f29b0 con 0x7fc10806c2d0 2024-09-13T17:17:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.675+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc10815ab70 con 0x7fc10806c2d0 2024-09-13T17:17:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.676+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc0f8039040 con 0x7fc10806c2d0 2024-09-13T17:17:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.676+0000 7fc0ffe00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 0x7fc0e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:07.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.677+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fc0f80a9d60 con 0x7fc10806c2d0 2024-09-13T17:17:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.677+0000 7fc0fe000640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 0x7fc0e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:07.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.678+0000 7fc0fe000640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 0x7fc0e406d0c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc10806c2a0 tx=0x7fc0f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:07.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.681+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc0f8074f80 con 0x7fc10806c2d0 2024-09-13T17:17:07.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.861+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc108154da0 con 0x7fc10806c2d0 2024-09-13T17:17:07.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.861+0000 7fc0ffe00640 1 -- 172.21.15.28:0/154499387 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fc0f8030d40 con 0x7fc10806c2d0 2024-09-13T17:17:07.863 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:07.863 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:53.097532+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 msgr2=0x7fc0e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 0x7fc0e406d0c0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fc10806c2a0 tx=0x7fc0f0009040 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 msgr2=0x7fc108154340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fc0f8035040 tx=0x7fc0f80046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 shutdown_connections 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc0e406ac00 0x7fc0e406d0c0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1081592a0 0x7fc1081f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc10806f970 0x7fc10814d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 --2- 172.21.15.28:0/154499387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc10806c2d0 0x7fc108154340 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 >> 172.21.15.28:0/154499387 conn(0x7fc108069e80 msgr2=0x7fc1080ade10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 shutdown_connections 2024-09-13T17:17:07.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:07.863+0000 7fc10de00640 1 -- 172.21.15.28:0/154499387 wait complete. 2024-09-13T17:17:07.881 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:17:07.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.035+0000 7fd2efa00640 1 Processor -- start 2024-09-13T17:17:08.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.035+0000 7fd2efa00640 1 -- start start 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.036+0000 7fd2efa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2e806f970 0x7fd2e815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.036+0000 7fd2efa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.036+0000 7fd2efa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2e806cc70 con 0x7fd2e81592a0 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.036+0000 7fd2efa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2e815b0b0 con 0x7fd2e806f970 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.036+0000 7fd2efa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e8159d90 con 0x7fd2e806c2d0 2024-09-13T17:17:08.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2e806f970 0x7fd2e815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2eda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2eda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:32846/0 (socket says 172.21.15.28:32846) 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2eda00640 1 -- 172.21.15.28:0/1249675482 learned_addr learned my addr 172.21.15.28:0/1249675482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2e806f970 0x7fd2e815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:56224/0 (socket says 172.21.15.28:56224) 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2ed000640 1 --1- 172.21.15.28:0/1249675482 >> v1:172.21.15.189:6789/0 conn(0x7fd2e81592a0 0x7fd2e8159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60462/0 (socket says 172.21.15.28:60462) 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 -- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 msgr2=0x7fd2e806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 -- 172.21.15.28:0/1249675482 >> v1:172.21.15.189:6789/0 conn(0x7fd2e81592a0 legacy=0x7fd2e8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7e00640 1 -- 172.21.15.28:0/1249675482 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e80b2040 con 0x7fd2e806f970 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2e7400640 1 -- 172.21.15.28:0/1249675482 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558301768 0 0) 0x7fd2e806cc70 con 0x7fd2e81592a0 2024-09-13T17:17:08.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.037+0000 7fd2eda00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.038+0000 7fd2e7e00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806f970 0x7fd2e815ab70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40097a0 tx=0x7fd2d402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9468018c1edd7a3d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.038+0000 7fd2e7400640 1 -- 172.21.15.28:0/1249675482 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4032440 con 0x7fd2e806f970 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.038+0000 7fd2e7400640 1 -- 172.21.15.28:0/1249675482 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d40325a0 con 0x7fd2e806f970 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.038+0000 7fd2e7400640 1 -- 172.21.15.28:0/1249675482 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4032930 con 0x7fd2e806f970 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 -- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806f970 msgr2=0x7fd2e815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806f970 0x7fd2e815ab70 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd2d40097a0 tx=0x7fd2d402a180 comp rx=0 tx=0).stop 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 -- 172.21.15.28:0/1249675482 shutdown_connections 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e806c2d0 0x7fd2e806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 --2- 172.21.15.28:0/1249675482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806f970 0x7fd2e815ab70 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 -- 172.21.15.28:0/1249675482 >> 172.21.15.28:0/1249675482 conn(0x7fd2e8069e80 msgr2=0x7fd2e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 -- 172.21.15.28:0/1249675482 shutdown_connections 2024-09-13T17:17:08.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.039+0000 7fd2efa00640 1 -- 172.21.15.28:0/1249675482 wait complete. 2024-09-13T17:17:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.041+0000 7fd2efa00640 1 Processor -- start 2024-09-13T17:17:08.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.041+0000 7fd2efa00640 1 -- start start 2024-09-13T17:17:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 0x7fd2e814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e81592a0 0x7fd2e80acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80cede0 con 0x7fd2e806f970 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80ce160 con 0x7fd2e806c2d0 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2efa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2e80ce440 con 0x7fd2e81592a0 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34722/0 (socket says 172.21.15.28:34722) 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 -- 172.21.15.28:0/988353858 learned_addr learned my addr 172.21.15.28:0/988353858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2eda00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e81592a0 0x7fd2e80acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2ed000640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 0x7fd2e814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 -- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e81592a0 msgr2=0x7fd2e80acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e81592a0 0x7fd2e80acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 -- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 msgr2=0x7fd2e814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 0x7fd2e814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.042+0000 7fd2e7e00640 1 -- 172.21.15.28:0/988353858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e000bd40 con 0x7fd2e806f970 2024-09-13T17:17:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2ed000640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 0x7fd2e814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:08.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2e7e00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4035040 tx=0x7fd2d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4006e10 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d4032e90 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d4030be0 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d4009400 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.043+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2e80cea00 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.044+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2e80b3740 con 0x7fd2e806f970 2024-09-13T17:17:08.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.045+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd2d4039040 con 0x7fd2e806f970 2024-09-13T17:17:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.046+0000 7fd2e4c00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 0x7fd2c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.046+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd2d40a9e00 con 0x7fd2e806f970 2024-09-13T17:17:08.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.046+0000 7fd2ed000640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 0x7fd2c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.047+0000 7fd2ed000640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 0x7fd2c806d0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc000f60 tx=0x7fd2dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.050+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd2d40750b0 con 0x7fd2e806f970 2024-09-13T17:17:08.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.267+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2e80b1990 con 0x7fd2e806f970 2024-09-13T17:17:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.268+0000 7fd2e4c00640 1 -- 172.21.15.28:0/988353858 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7fd2d4030d40 con 0x7fd2e806f970 2024-09-13T17:17:08.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:08.270 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:17:08.270 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:17:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 msgr2=0x7fd2c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 0x7fd2c806d0c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd2dc000f60 tx=0x7fd2dc009040 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 msgr2=0x7fd2e80ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fd2d4035040 tx=0x7fd2d40046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 shutdown_connections 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd2c806ac00 0x7fd2c806d0c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2e81592a0 0x7fd2e80acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2e806f970 0x7fd2e80ac850 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 --2- 172.21.15.28:0/988353858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2e806c2d0 0x7fd2e814dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 >> 172.21.15.28:0/988353858 conn(0x7fd2e8069e80 msgr2=0x7fd2e806fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 shutdown_connections 2024-09-13T17:17:08.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.270+0000 7fd2efa00640 1 -- 172.21.15.28:0/988353858 wait complete. 2024-09-13T17:17:08.290 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.443+0000 7fa92e400640 1 Processor -- start 2024-09-13T17:17:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.443+0000 7fa92e400640 1 -- start start 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa92e400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa92815a540 0x7fa92806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa92e400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa928070cc0 0x7fa928071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa92e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92815e100 con 0x7fa9280cabc0 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa92e400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9280adb00 con 0x7fa92815a540 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa92e400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa928071670 con 0x7fa928070cc0 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa91ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9280cabc0 0x7fa92815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41638/0 (socket says 172.21.15.28:41638) 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.444+0000 7fa91ea00640 1 -- 172.21.15.28:0/1564256725 learned_addr learned my addr 172.21.15.28:0/1564256725 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927400640 1 --2- 172.21.15.28:0/1564256725 >> v2:172.21.15.189:3300/0 conn(0x7fa92815a540 0x7fa92806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927e00640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa928070cc0 0x7fa928071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa926a00640 1 -- 172.21.15.28:0/1564256725 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032872880 0 0) 0x7fa92815e100 con 0x7fa9280cabc0 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa926a00640 1 -- 172.21.15.28:0/1564256725 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa90c003520 con 0x7fa9280cabc0 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927400640 1 -- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa928070cc0 msgr2=0x7fa928071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927400640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa928070cc0 0x7fa928071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927400640 1 -- 172.21.15.28:0/1564256725 >> v1:172.21.15.28:6789/0 conn(0x7fa9280cabc0 legacy=0x7fa92815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.445+0000 7fa927400640 1 -- 172.21.15.28:0/1564256725 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92815e3e0 con 0x7fa92815a540 2024-09-13T17:17:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.446+0000 7fa927400640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806c430 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa9180097a0 tx=0x7fa91802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e080487b0a2f9cf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa926a00640 1 -- 172.21.15.28:0/1564256725 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91802ae50 con 0x7fa92815a540 2024-09-13T17:17:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa926a00640 1 -- 172.21.15.28:0/1564256725 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa918032440 con 0x7fa92815a540 2024-09-13T17:17:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa926a00640 1 -- 172.21.15.28:0/1564256725 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9180327d0 con 0x7fa92815a540 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 -- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 msgr2=0x7fa92806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806c430 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fa9180097a0 tx=0x7fa91802a100 comp rx=0 tx=0).stop 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 -- 172.21.15.28:0/1564256725 shutdown_connections 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa928070cc0 0x7fa928071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 --2- 172.21.15.28:0/1564256725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806c430 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 -- 172.21.15.28:0/1564256725 >> 172.21.15.28:0/1564256725 conn(0x7fa928069e80 msgr2=0x7fa92806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 -- 172.21.15.28:0/1564256725 shutdown_connections 2024-09-13T17:17:08.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.447+0000 7fa92e400640 1 -- 172.21.15.28:0/1564256725 wait complete. 2024-09-13T17:17:08.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.449+0000 7fa92e400640 1 Processor -- start 2024-09-13T17:17:08.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.449+0000 7fa92e400640 1 -- start start 2024-09-13T17:17:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9280cabc0 0x7fa9280b0740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa928156990 con 0x7fa928070cc0 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9280b0c80 con 0x7fa92815a540 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa92e400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92815d8e0 con 0x7fa9280cabc0 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa927400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa927400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34730/0 (socket says 172.21.15.28:34730) 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.450+0000 7fa927400640 1 -- 172.21.15.28:0/907433232 learned_addr learned my addr 172.21.15.28:0/907433232 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927e00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa91ea00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9280cabc0 0x7fa9280b0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 -- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9280cabc0 msgr2=0x7fa9280b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9280cabc0 0x7fa9280b0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 -- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 msgr2=0x7fa92806f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 -- 172.21.15.28:0/907433232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918009400 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927e00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa927400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa918035040 tx=0x7fa9180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91803d710 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91803d890 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa928156be0 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.451+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91803dc40 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.452+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9281571a0 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.453+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92806d120 con 0x7fa928070cc0 2024-09-13T17:17:08.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.453+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa918039040 con 0x7fa928070cc0 2024-09-13T17:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.454+0000 7fa91fe00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 0x7fa90406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.454+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9180a9ac0 con 0x7fa928070cc0 2024-09-13T17:17:08.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.455+0000 7fa91ea00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 0x7fa90406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.456+0000 7fa91ea00640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 0x7fa90406d0c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa9280b1090 tx=0x7fa910007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.458+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa918074cf0 con 0x7fa928070cc0 2024-09-13T17:17:08.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.672+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa928071130 con 0x7fa928070cc0 2024-09-13T17:17:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.673+0000 7fa91fe00640 1 -- 172.21.15.28:0/907433232 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+4742 (secure 0 0 0) 0x7fa918078310 con 0x7fa928070cc0 2024-09-13T17:17:08.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:08.675 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":114,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:16:29.636609+0000","modified":"2024-09-13T17:16:32.652735+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7939},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7939":{"gid":7939,"name":"c","rank":0,"incarnation":110,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/199464366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":199464366},{"type":"v1","addr":"172.21.15.28:6835","nonce":199464366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T17:17:08.675 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 114 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 msgr2=0x7fa90406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 0x7fa90406d0c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa9280b1090 tx=0x7fa910007d30 comp rx=0 tx=0).stop 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 msgr2=0x7fa92815d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7fa918035040 tx=0x7fa9180046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 shutdown_connections 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa90406ac00 0x7fa90406d0c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa92815a540 0x7fa92806f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9280cabc0 0x7fa9280b0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 --2- 172.21.15.28:0/907433232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa928070cc0 0x7fa92815d270 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 >> 172.21.15.28:0/907433232 conn(0x7fa928069e80 msgr2=0x7fa92815e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 shutdown_connections 2024-09-13T17:17:08.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.675+0000 7fa92e400640 1 -- 172.21.15.28:0/907433232 wait complete. 2024-09-13T17:17:08.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.846+0000 7f6e95a00640 1 Processor -- start 2024-09-13T17:17:08.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.846+0000 7f6e95a00640 1 -- start start 2024-09-13T17:17:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.847+0000 7f6e95a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6e90071c00 0x7f6e90146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.847+0000 7f6e95a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e90070290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.847+0000 7f6e95a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e90143070 con 0x7f6e900ad260 2024-09-13T17:17:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.847+0000 7f6e95a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e90071fe0 con 0x7f6e90071c00 2024-09-13T17:17:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.847+0000 7f6e95a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900ada70 con 0x7f6e900b27c0 2024-09-13T17:17:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8e000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6e900ad260 0x7f6e900b20b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41648/0 (socket says 172.21.15.28:41648) 2024-09-13T17:17:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8e000640 1 -- 172.21.15.28:0/3499143029 learned_addr learned my addr 172.21.15.28:0/3499143029 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/3499143029 >> v2:172.21.15.189:3300/0 conn(0x7f6e90071c00 0x7f6e90146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8f400640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e90070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21969777 0 0) 0x7f6e90143070 con 0x7f6e900ad260 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e74003520 con 0x7f6e900ad260 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202504324 0 0) 0x7f6e74003520 con 0x7f6e900ad260 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 msgr2=0x7f6e90070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e90070290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 >> v2:172.21.15.189:3300/0 conn(0x7f6e90071c00 msgr2=0x7f6e90146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 --2- 172.21.15.28:0/3499143029 >> v2:172.21.15.189:3300/0 conn(0x7f6e90071c00 0x7f6e90146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900add50 con 0x7f6e900ad260 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6e7c001670 con 0x7f6e900ad260 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 >> v1:172.21.15.28:6789/0 conn(0x7f6e900ad260 legacy=0x7f6e900b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.848+0000 7f6e8d600640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e74003970 0x7f6e74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8d600640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e740047c0 0x7f6e74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8d600640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90071fe0 con 0x7f6e74003970 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e900ada70 con 0x7f6e740047c0 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e7400b980 con 0x7f6e900b27c0 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e74003970 0x7f6e74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8e000640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e740047c0 0x7f6e74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8f400640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 -- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 msgr2=0x7f6e74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e900b27c0 0x7f6e74005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 -- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e740047c0 msgr2=0x7f6e74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e740047c0 0x7f6e74004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.849+0000 7f6e8ea00640 1 -- 172.21.15.28:0/3499143029 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e74003520 con 0x7f6e74003970 2024-09-13T17:17:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.850+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e74003970 0x7f6e74003d90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009ac0 tx=0x7f6e740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6971da6fbdec5f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80031400 con 0x7f6e74003970 2024-09-13T17:17:08.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8002d050 con 0x7f6e74003970 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e8d600640 1 -- 172.21.15.28:0/3499143029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e80031780 con 0x7f6e74003970 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 -- 172.21.15.28:0/3499143029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e74003970 msgr2=0x7f6e74003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 --2- 172.21.15.28:0/3499143029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e74003970 0x7f6e74003d90 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f6e80009ac0 tx=0x7f6e740057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e8f400640 1 -- 172.21.15.28:0/3499143029 reap_dead start 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 -- 172.21.15.28:0/3499143029 shutdown_connections 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 -- 172.21.15.28:0/3499143029 >> 172.21.15.28:0/3499143029 conn(0x7f6e900b0b60 msgr2=0x7f6e900b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 -- 172.21.15.28:0/3499143029 shutdown_connections 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.851+0000 7f6e95a00640 1 -- 172.21.15.28:0/3499143029 wait complete. 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.853+0000 7f6e95a00640 1 Processor -- start 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.853+0000 7f6e95a00640 1 -- start start 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e74003970 0x7f6e9015c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e90141350 0x7f6e901417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9015c850 con 0x7f6e90135390 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e90141d10 con 0x7f6e74003970 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e95a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9015e420 con 0x7f6e90141350 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e8e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e8e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34748/0 (socket says 172.21.15.28:34748) 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e8e000640 1 -- 172.21.15.28:0/2696326753 learned_addr learned my addr 172.21.15.28:0/2696326753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e74003970 0x7f6e9015c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.854+0000 7f6e8f400640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e90141350 0x7f6e901417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 -- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e90141350 msgr2=0x7f6e901417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e90141350 0x7f6e901417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 -- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e74003970 msgr2=0x7f6e9015c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e74003970 0x7f6e9015c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 -- 172.21.15.28:0/2696326753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9015d630 con 0x7f6e90135390 2024-09-13T17:17:08.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e8e000640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c007340 tx=0x7f6e7c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00f860 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7c00f9c0 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c00fd70 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.855+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e9015d910 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.856+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e9015ded0 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.857+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e90161400 con 0x7f6e90135390 2024-09-13T17:17:08.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.857+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6e7c01d460 con 0x7f6e90135390 2024-09-13T17:17:08.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.858+0000 7f6e86a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 0x7f6e7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:08.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.858+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e7c089420 con 0x7f6e90135390 2024-09-13T17:17:08.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.859+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 0x7f6e7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:08.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.859+0000 7f6e8ea00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 0x7f6e7006d0f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6e80033000 tx=0x7f6e80008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:08.862+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6e7c054690 con 0x7f6e90135390 2024-09-13T17:17:09.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.041+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e90146050 con 0x7f6e90135390 2024-09-13T17:17:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.043+0000 7f6e86a00640 1 -- 172.21.15.28:0/2696326753 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f6e7c057cb0 con 0x7f6e90135390 2024-09-13T17:17:09.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:09.045 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":127,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:16:53.097532+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:16:27.833325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-09-13T17:16:28.661422+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.7","mappings":[{"from":0,"to":5}]},{"pgid":"17.3f","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 msgr2=0x7f6e7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 0x7f6e7006d0f0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f6e80033000 tx=0x7f6e80008030 comp rx=0 tx=0).stop 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 msgr2=0x7f6e9015cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c007340 tx=0x7f6e7c007800 comp rx=0 tx=0).stop 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 shutdown_connections 2024-09-13T17:17:09.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e7006ac30 0x7f6e7006d0f0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e90141350 0x7f6e901417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e90135390 0x7f6e9015cd50 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 --2- 172.21.15.28:0/2696326753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e74003970 0x7f6e9015c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 >> 172.21.15.28:0/2696326753 conn(0x7f6e900b0b60 msgr2=0x7f6e900afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 shutdown_connections 2024-09-13T17:17:09.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.045+0000 7f6e95a00640 1 -- 172.21.15.28:0/2696326753 wait complete. 2024-09-13T17:17:09.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:17:09.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.221+0000 7f12eb200640 1 Processor -- start 2024-09-13T17:17:09.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.221+0000 7f12eb200640 1 -- start start 2024-09-13T17:17:09.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.222+0000 7f12eb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec159a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.222+0000 7f12eb200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12ec0b2870 0x7f12ec0b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.222+0000 7f12eb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ec0b32f0 con 0x7f12ec06c370 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.222+0000 7f12eb200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ec0cbac0 con 0x7f12ec0b2870 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.222+0000 7f12eb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec0b35d0 con 0x7f12ec0b0e10 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12ec0b2870 0x7f12ec0b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12ea800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f12ec06c370 0x7f12ec0cbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41654/0 (socket says 172.21.15.28:41654) 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f12ec0b2870 0x7f12ec0b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34764/0 (socket says 172.21.15.28:34764) 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12ea800640 1 -- 172.21.15.28:0/2351583632 learned_addr learned my addr 172.21.15.28:0/2351583632 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:09.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9e00640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 -- 172.21.15.28:0/2351583632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 msgr2=0x7f12ec159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:09.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 -- 172.21.15.28:0/2351583632 >> v1:172.21.15.28:6789/0 conn(0x7f12ec06c370 legacy=0x7f12ec0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e9400640 1 -- 172.21.15.28:0/2351583632 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec0b38b0 con 0x7f12ec0b2870 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.223+0000 7f12e8a00640 1 -- 172.21.15.28:0/2351583632 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663539655 0 0) 0x7f12ec0b32f0 con 0x7f12ec06c370 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.224+0000 7f12e9400640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec0b2870 0x7f12ec0b2cb0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4976b168791913a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.224+0000 7f12e8a00640 1 -- 172.21.15.28:0/2351583632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d802ae50 con 0x7f12ec0b2870 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12e8a00640 1 -- 172.21.15.28:0/2351583632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d8032440 con 0x7f12ec0b2870 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12e8a00640 1 -- 172.21.15.28:0/2351583632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d80327d0 con 0x7f12ec0b2870 2024-09-13T17:17:09.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 -- 172.21.15.28:0/2351583632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec0b2870 msgr2=0x7f12ec0b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec0b2870 0x7f12ec0b2cb0 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f12d80097a0 tx=0x7f12d802a100 comp rx=0 tx=0).stop 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 -- 172.21.15.28:0/2351583632 shutdown_connections 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec0b2870 0x7f12ec0b2cb0 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 --2- 172.21.15.28:0/2351583632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 -- 172.21.15.28:0/2351583632 >> 172.21.15.28:0/2351583632 conn(0x7f12ec073470 msgr2=0x7f12ec069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 -- 172.21.15.28:0/2351583632 shutdown_connections 2024-09-13T17:17:09.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.225+0000 7f12eb200640 1 -- 172.21.15.28:0/2351583632 wait complete. 2024-09-13T17:17:09.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.227+0000 7f12eb200640 1 Processor -- start 2024-09-13T17:17:09.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.227+0000 7f12eb200640 1 -- start start 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec138980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 0x7f12ec138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec13a650 con 0x7f12ec06c370 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec1593f0 con 0x7f12ec0b2870 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12eb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec141680 con 0x7f12ec0b0e10 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34768/0 (socket says 172.21.15.28:34768) 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12e9e00640 1 -- 172.21.15.28:0/652567813 learned_addr learned my addr 172.21.15.28:0/652567813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12e9400640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec138980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.228+0000 7f12ea800640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 0x7f12ec138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 -- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 msgr2=0x7f12ec138980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec138980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 -- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 msgr2=0x7f12ec138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 0x7f12ec138ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 -- 172.21.15.28:0/652567813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d8009400 con 0x7f12ec06c370 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12ea800640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 0x7f12ec138ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:09.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e9e00640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00d800 tx=0x7f12dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00b6c0 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12dc00b820 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12dc00bbb0 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ec141960 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.229+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ec15be90 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.231+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ec15ba10 con 0x7f12ec06c370 2024-09-13T17:17:09.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.232+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f12dc00bd10 con 0x7f12ec06c370 2024-09-13T17:17:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.232+0000 7f12e2000640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 0x7f12cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.232+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f12dc089920 con 0x7f12ec06c370 2024-09-13T17:17:09.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.233+0000 7f12e9400640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 0x7f12cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.233+0000 7f12e9400640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 0x7f12cc06d0c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f12d8034000 tx=0x7f12d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.236+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f12ec15ba10 con 0x7f12ec06c370 2024-09-13T17:17:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.455+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f12ec15ba10 con 0x7f12ec06c370 2024-09-13T17:17:09.458 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:17:09.456+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:17:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.469+0000 7f12e2000640 1 -- 172.21.15.28:0/652567813 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v115) v1 ==== 161+0+0 (secure 0 0 0) 0x7f12ec15ba10 con 0x7f12ec06c370 2024-09-13T17:17:09.471 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 msgr2=0x7f12cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 0x7f12cc06d0c0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f12d8034000 tx=0x7f12d802e000 comp rx=0 tx=0).stop 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 msgr2=0x7f12ec15a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f12dc00d800 tx=0x7f12dc00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 shutdown_connections 2024-09-13T17:17:09.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f12cc06ac00 0x7f12cc06d0c0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12ec0b2870 0x7f12ec138ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f12ec0b0e10 0x7f12ec138980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 --2- 172.21.15.28:0/652567813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ec06c370 0x7f12ec15a120 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 >> 172.21.15.28:0/652567813 conn(0x7f12ec073470 msgr2=0x7f12ec0cd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 shutdown_connections 2024-09-13T17:17:09.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.472+0000 7f12eb200640 1 -- 172.21.15.28:0/652567813 wait complete. 2024-09-13T17:17:09.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:17:09.544 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:17:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.649+0000 7fbb6e600640 1 Processor -- start 2024-09-13T17:17:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.649+0000 7fbb6e600640 1 -- start start 2024-09-13T17:17:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.650+0000 7fbb6e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb68070430 0x7fbb6815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.650+0000 7fbb6e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6815ab60 con 0x7fbb68159770 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.650+0000 7fbb6e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6806f920 con 0x7fbb6806f3f0 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.650+0000 7fbb6e600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6815b290 con 0x7fbb68070430 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb67e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fbb6806f3f0 0x7fbb6815c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60478/0 (socket says 172.21.15.28:60478) 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb5fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb68070430 0x7fbb6815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb67e00640 1 -- 172.21.15.28:0/1220170513 learned_addr learned my addr 172.21.15.28:0/1220170513 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb66a00640 1 -- 172.21.15.28:0/1220170513 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669400897 0 0) 0x7fbb6806f920 con 0x7fbb6806f3f0 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb66a00640 1 -- 172.21.15.28:0/1220170513 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb4c005730 con 0x7fbb6806f3f0 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb5fe00640 1 -- 172.21.15.28:0/1220170513 >> v1:172.21.15.189:6789/0 conn(0x7fbb6806f3f0 legacy=0x7fbb6815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb5fe00640 1 -- 172.21.15.28:0/1220170513 >> v1:172.21.15.28:6789/0 conn(0x7fbb68159770 legacy=0x7fbb6806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.651+0000 7fbb5fe00640 1 -- 172.21.15.28:0/1220170513 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6815b570 con 0x7fbb68070430 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.653+0000 7fbb5fe00640 1 --2- 172.21.15.28:0/1220170513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6815bd70 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fbb500097a0 tx=0x7fbb5002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba84cce3df512a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.653+0000 7fbb66a00640 1 -- 172.21.15.28:0/1220170513 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5002ad30 con 0x7fbb68070430 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.653+0000 7fbb66a00640 1 -- 172.21.15.28:0/1220170513 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5002ae90 con 0x7fbb68070430 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.653+0000 7fbb66a00640 1 -- 172.21.15.28:0/1220170513 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb50032640 con 0x7fbb68070430 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 -- 172.21.15.28:0/1220170513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 msgr2=0x7fbb6815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 --2- 172.21.15.28:0/1220170513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6815bd70 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fbb500097a0 tx=0x7fbb5002a0a0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 -- 172.21.15.28:0/1220170513 shutdown_connections 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 --2- 172.21.15.28:0/1220170513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6815bd70 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 -- 172.21.15.28:0/1220170513 >> 172.21.15.28:0/1220170513 conn(0x7fbb680b0270 msgr2=0x7fbb680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:09.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 -- 172.21.15.28:0/1220170513 shutdown_connections 2024-09-13T17:17:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.654+0000 7fbb6e600640 1 -- 172.21.15.28:0/1220170513 wait complete. 2024-09-13T17:17:09.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.656+0000 7fbb6e600640 1 Processor -- start 2024-09-13T17:17:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.656+0000 7fbb6e600640 1 -- start start 2024-09-13T17:17:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6814eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb68159770 0x7fbb680af230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb681517c0 con 0x7fbb68070430 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6814e0a0 con 0x7fbb6806f3f0 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb6e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb681508a0 con 0x7fbb68159770 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb67400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb67400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:56330/0 (socket says 172.21.15.28:56330) 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6814eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb67400640 1 -- 172.21.15.28:0/2736246681 learned_addr learned my addr 172.21.15.28:0/2736246681 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.657+0000 7fbb67e00640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb68159770 0x7fbb680af230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 -- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb68159770 msgr2=0x7fbb680af230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb68159770 0x7fbb680af230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 -- 172.21.15.28:0/2736246681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 msgr2=0x7fbb6814eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6814eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 -- 172.21.15.28:0/2736246681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50009400 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.658+0000 7fbb67400640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c7b0 tx=0x7fbb5800cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.659+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58005480 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.659+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5800ce60 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.659+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb68150b40 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.659+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5800a880 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.659+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb68151100 con 0x7fbb6806f3f0 2024-09-13T17:17:09.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.660+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb68151900 con 0x7fbb6806f3f0 2024-09-13T17:17:09.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.661+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbb5800a9e0 con 0x7fbb6806f3f0 2024-09-13T17:17:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.662+0000 7fbb5f400640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 0x7fbb4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.662+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5853+0+0 (secure 0 0 0) 0x7fbb58088970 con 0x7fbb6806f3f0 2024-09-13T17:17:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.663+0000 7fbb5fe00640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 0x7fbb4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:09.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.664+0000 7fbb5fe00640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 0x7fbb4406d0c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbb50034000 tx=0x7fbb5002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.667+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbb68151900 con 0x7fbb6806f3f0 2024-09-13T17:17:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:09.883+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbb6815ab60 con 0x7fbb6806f3f0 2024-09-13T17:17:10.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.476+0000 7fbb5f400640 1 -- 172.21.15.28:0/2736246681 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v116) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbb6815ab60 con 0x7fbb6806f3f0 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 msgr2=0x7fbb4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 0x7fbb4406d0c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fbb50034000 tx=0x7fbb5002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 msgr2=0x7fbb68149f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbb5800c7b0 tx=0x7fbb5800cc70 comp rx=0 tx=0).stop 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 shutdown_connections 2024-09-13T17:17:10.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4406ac00 0x7fbb4406d0c0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb68159770 0x7fbb680af230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb68070430 0x7fbb6814eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 --2- 172.21.15.28:0/2736246681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6806f3f0 0x7fbb68149f10 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 >> 172.21.15.28:0/2736246681 conn(0x7fbb680b0270 msgr2=0x7fbb680ae090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 shutdown_connections 2024-09-13T17:17:10.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.479+0000 7fbb6e600640 1 -- 172.21.15.28:0/2736246681 wait complete. 2024-09-13T17:17:10.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:17:10.551 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:17:10.555 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:17:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.659+0000 7f72f1400640 1 Processor -- start 2024-09-13T17:17:10.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.659+0000 7f72f1400640 1 -- start start 2024-09-13T17:17:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.660+0000 7f72f1400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f72ec159970 0x7f72ec0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.660+0000 7f72f1400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec15a4c0 con 0x7f72ec0b2630 2024-09-13T17:17:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.660+0000 7f72f1400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec071580 con 0x7f72ec159590 2024-09-13T17:17:10.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.660+0000 7f72f1400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec0b2ae0 con 0x7f72ec159970 2024-09-13T17:17:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72eb200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72ec0b2630 0x7f72ec15a950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41694/0 (socket says 172.21.15.28:41694) 2024-09-13T17:17:10.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72eb200640 1 -- 172.21.15.28:0/954134174 learned_addr learned my addr 172.21.15.28:0/954134174 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9e00640 1 --2- 172.21.15.28:0/954134174 >> v2:172.21.15.189:3300/0 conn(0x7f72ec159970 0x7f72ec0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581502273 0 0) 0x7f72ec15a4c0 con 0x7f72ec0b2630 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d0003520 con 0x7f72ec0b2630 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246536422 0 0) 0x7f72ec071580 con 0x7f72ec159590 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ec15a4c0 con 0x7f72ec159590 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4125361239 0 0) 0x7f72d0003520 con 0x7f72ec0b2630 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 >> v2:172.21.15.189:3300/0 conn(0x7f72ec159970 msgr2=0x7f72ec0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 --2- 172.21.15.28:0/954134174 >> v2:172.21.15.189:3300/0 conn(0x7f72ec159970 0x7f72ec0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 >> v1:172.21.15.189:6789/0 conn(0x7f72ec159590 legacy=0x7f72ec070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec159e70 con 0x7f72ec0b2630 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f72e40019f0 con 0x7f72ec0b2630 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 >> v1:172.21.15.28:6789/0 conn(0x7f72ec0b2630 legacy=0x7f72ec15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.661+0000 7f72e9400640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d0003b50 0x7f72d0003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9400640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9400640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d0005180 0x7f72d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0b2ae0 con 0x7f72d00047c0 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d000c050 con 0x7f72d0005180 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d000c210 con 0x7f72d0003b50 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72ea800640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d0003b50 0x7f72d0003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72eb200640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d0005180 0x7f72d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 -- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d0003b50 msgr2=0x7f72d0003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d0003b50 0x7f72d0003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 -- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d0005180 msgr2=0x7f72d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:10.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d0005180 0x7f72d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72e9e00640 1 -- 172.21.15.28:0/954134174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d0003520 con 0x7f72d00047c0 2024-09-13T17:17:10.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.662+0000 7f72eb200640 1 -- 172.21.15.28:0/954134174 reap_dead start 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.663+0000 7f72e9e00640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72d0004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009650 tx=0x7f72d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9091a5a151dde505 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80333f0 con 0x7f72d00047c0 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d8033550 con 0x7f72d00047c0 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72e9400640 1 -- 172.21.15.28:0/954134174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d80338e0 con 0x7f72d00047c0 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 -- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 msgr2=0x7f72d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72d0004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7f72d8009650 tx=0x7f72d00057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 -- 172.21.15.28:0/954134174 shutdown_connections 2024-09-13T17:17:10.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 --2- 172.21.15.28:0/954134174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72d0004c40 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 -- 172.21.15.28:0/954134174 >> 172.21.15.28:0/954134174 conn(0x7f72ec069e80 msgr2=0x7f72ec06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 -- 172.21.15.28:0/954134174 shutdown_connections 2024-09-13T17:17:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.664+0000 7f72f1400640 1 -- 172.21.15.28:0/954134174 wait complete. 2024-09-13T17:17:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.666+0000 7f72f1400640 1 Processor -- start 2024-09-13T17:17:10.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.666+0000 7f72f1400640 1 -- start start 2024-09-13T17:17:10.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ec13a380 0x7f72ec152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72ec15ba90 0x7f72ec13cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec15b600 con 0x7f72d00047c0 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec13a800 con 0x7f72ec13a380 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72f1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec152b90 con 0x7f72ec15ba90 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72ea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72ea800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34824/0 (socket says 172.21.15.28:34824) 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72ea800640 1 -- 172.21.15.28:0/1494103007 learned_addr learned my addr 172.21.15.28:0/1494103007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72e9e00640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ec13a380 0x7f72ec152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.667+0000 7f72eb200640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72ec15ba90 0x7f72ec13cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 -- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72ec15ba90 msgr2=0x7f72ec13cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72ec15ba90 0x7f72ec13cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 -- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ec13a380 msgr2=0x7f72ec152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:10.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ec13a380 0x7f72ec152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 -- 172.21.15.28:0/1494103007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec15a9f0 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72ea800640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00f7a0 tx=0x7f72dc00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00fe50 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc005320 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc007290 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec15acd0 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.668+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec15b290 con 0x7f72d00047c0 2024-09-13T17:17:10.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.669+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec0b2070 con 0x7f72d00047c0 2024-09-13T17:17:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.670+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72dc0073f0 con 0x7f72d00047c0 2024-09-13T17:17:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.671+0000 7f72e2a00640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 0x7f72cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.671+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f72dc088a10 con 0x7f72d00047c0 2024-09-13T17:17:10.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.672+0000 7f72e9e00640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 0x7f72cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:10.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.672+0000 7f72e9e00640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 0x7f72cc06d0f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f72d8039000 tx=0x7f72d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:10.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.675+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f72dc053b50 con 0x7f72d00047c0 2024-09-13T17:17:10.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:10.853+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f72ec071580 con 0x7f72d00047c0 2024-09-13T17:17:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.486+0000 7f72e2a00640 1 -- 172.21.15.28:0/1494103007 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v129) v1 ==== 181+0+0 (secure 0 0 0) 0x7f72dc01a0d0 con 0x7f72d00047c0 2024-09-13T17:17:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 msgr2=0x7f72cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 0x7f72cc06d0f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f72d8039000 tx=0x7f72d8008030 comp rx=0 tx=0).stop 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 msgr2=0x7f72ec14c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00f7a0 tx=0x7f72dc00fc60 comp rx=0 tx=0).stop 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 shutdown_connections 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72cc06ac30 0x7f72cc06d0f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72ec15ba90 0x7f72ec13cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72ec13a380 0x7f72ec152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 --2- 172.21.15.28:0/1494103007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d00047c0 0x7f72ec14c3b0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.490+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 >> 172.21.15.28:0/1494103007 conn(0x7f72ec069e80 msgr2=0x7f72ec06c640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.491+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 shutdown_connections 2024-09-13T17:17:11.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.491+0000 7f72f1400640 1 -- 172.21.15.28:0/1494103007 wait complete. 2024-09-13T17:17:11.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:17:11.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.715+0000 7eff33000640 1 Processor -- start 2024-09-13T17:17:11.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.715+0000 7eff33000640 1 -- start start 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff33000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff2c0ceba0 0x7eff2c0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff33000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c06c950 con 0x7eff2c15a7a0 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff33000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2c0b1ee0 con 0x7eff2c0b1ab0 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff33000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0b2150 con 0x7eff2c0ceba0 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff2be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7eff2c15a7a0 0x7eff2c0ce490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41704/0 (socket says 172.21.15.28:41704) 2024-09-13T17:17:11.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff2be00640 1 -- 172.21.15.28:0/3211866871 learned_addr learned my addr 172.21.15.28:0/3211866871 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff23e00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff2c0ceba0 0x7eff2c0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750143370 0 0) 0x7eff1c001110 con 0x7eff2c15a7a0 2024-09-13T17:17:11.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.717+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff10003520 con 0x7eff2c15a7a0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4086686877 0 0) 0x7eff2c0b1ee0 con 0x7eff2c0b1ab0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff1c001110 con 0x7eff2c0b1ab0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1185022902 0 0) 0x7eff10003520 con 0x7eff2c15a7a0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff2c0ceba0 msgr2=0x7eff2c0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff2c0ceba0 0x7eff2c0b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 >> v1:172.21.15.189:6789/0 conn(0x7eff2c0b1ab0 legacy=0x7eff2c0cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c0b2430 con 0x7eff2c15a7a0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7eff1c001800 con 0x7eff2c15a7a0 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 >> v1:172.21.15.28:6789/0 conn(0x7eff2c15a7a0 legacy=0x7eff2c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10003b30 0x7eff10003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff100047c0 0x7eff10004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10005180 0x7eff1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c0b2150 con 0x7eff10003b30 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000bff0 con 0x7eff10005180 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000c1b0 con 0x7eff100047c0 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff23e00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff2c0ceba0 0x7eff2c0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff2be00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10003b30 0x7eff10003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.718+0000 7eff31000640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10005180 0x7eff1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff23e00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff100047c0 0x7eff10004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff2be00640 1 -- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff100047c0 msgr2=0x7eff10004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff2be00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff100047c0 0x7eff10004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff2be00640 1 -- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10005180 msgr2=0x7eff1000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff2be00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10005180 0x7eff1000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff2be00640 1 -- 172.21.15.28:0/3211866871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10003520 con 0x7eff10003b30 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.719+0000 7eff31000640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff10005180 0x7eff1000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff2be00640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10003b30 0x7eff10003f30 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00c160 tx=0x7eff100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d149c1753d5745d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c005ad0 con 0x7eff10003b30 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c02d070 con 0x7eff10003b30 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff2b400640 1 -- 172.21.15.28:0/3211866871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c005e90 con 0x7eff10003b30 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff33000640 1 -- 172.21.15.28:0/3211866871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10003b30 msgr2=0x7eff10003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff33000640 1 --2- 172.21.15.28:0/3211866871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff10003b30 0x7eff10003f30 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00c160 tx=0x7eff100057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff31000640 1 -- 172.21.15.28:0/3211866871 reap_dead start 2024-09-13T17:17:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff33000640 1 -- 172.21.15.28:0/3211866871 shutdown_connections 2024-09-13T17:17:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.720+0000 7eff33000640 1 -- 172.21.15.28:0/3211866871 >> 172.21.15.28:0/3211866871 conn(0x7eff2c069e80 msgr2=0x7eff2c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.721+0000 7eff33000640 1 -- 172.21.15.28:0/3211866871 shutdown_connections 2024-09-13T17:17:11.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.721+0000 7eff33000640 1 -- 172.21.15.28:0/3211866871 wait complete. 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.722+0000 7eff33000640 1 Processor -- start 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.722+0000 7eff33000640 1 -- start start 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff10003b30 0x7eff2c1f23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff2c14eff0 0x7eff2c0ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1f3360 con 0x7eff2c06f3f0 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1f2ac0 con 0x7eff2c14eff0 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.723+0000 7eff33000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2c1f2da0 con 0x7eff10003b30 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34842/0 (socket says 172.21.15.28:34842) 2024-09-13T17:17:11.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 -- 172.21.15.28:0/807188188 learned_addr learned my addr 172.21.15.28:0/807188188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff2be00640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff10003b30 0x7eff2c1f23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff23e00640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff2c14eff0 0x7eff2c0ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 -- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff10003b30 msgr2=0x7eff2c1f23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff10003b30 0x7eff2c1f23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 -- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff2c14eff0 msgr2=0x7eff2c0ca040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff2c14eff0 0x7eff2c0ca040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 -- 172.21.15.28:0/807188188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c1f3080 con 0x7eff2c06f3f0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.724+0000 7eff31000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7eff24001330 tx=0x7eff2400eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.725+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24017070 con 0x7eff2c06f3f0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.725+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff24006b50 con 0x7eff2c06f3f0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.725+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24006d20 con 0x7eff2c06f3f0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.725+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2c152ce0 con 0x7eff2c06f3f0 2024-09-13T17:17:11.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.725+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2c14f470 con 0x7eff2c06f3f0 2024-09-13T17:17:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.726+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2c06af60 con 0x7eff2c06f3f0 2024-09-13T17:17:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.727+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff24003970 con 0x7eff2c06f3f0 2024-09-13T17:17:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.728+0000 7eff28c00640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 0x7eff0806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.728+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 5415+0+0 (secure 0 0 0) 0x7eff24088910 con 0x7eff2c06f3f0 2024-09-13T17:17:11.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.728+0000 7eff2be00640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 0x7eff0806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:11.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.729+0000 7eff2be00640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 0x7eff0806d0f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7eff1c032000 tx=0x7eff1c034040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:11.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.733+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7eff24053b70 con 0x7eff2c06f3f0 2024-09-13T17:17:11.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:11.917+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7eff2c159230 con 0x7eff2c06f3f0 2024-09-13T17:17:12.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.488+0000 7eff28c00640 1 -- 172.21.15.28:0/807188188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v130) v1 ==== 169+0+0 (secure 0 0 0) 0x7eff24057190 con 0x7eff2c06f3f0 2024-09-13T17:17:12.491 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:17:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.492+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 msgr2=0x7eff0806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.492+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 0x7eff0806d0f0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7eff1c032000 tx=0x7eff1c034040 comp rx=0 tx=0).stop 2024-09-13T17:17:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.492+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 msgr2=0x7eff2c0cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.492+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7eff24001330 tx=0x7eff2400eb40 comp rx=0 tx=0).stop 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.492+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 shutdown_connections 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff0806ac30 0x7eff0806d0f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff2c06f3f0 0x7eff2c0cd060 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff2c14eff0 0x7eff2c0ca040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 --2- 172.21.15.28:0/807188188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff10003b30 0x7eff2c1f23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 >> 172.21.15.28:0/807188188 conn(0x7eff2c069e80 msgr2=0x7eff2c15aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 shutdown_connections 2024-09-13T17:17:12.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.493+0000 7eff33000640 1 -- 172.21.15.28:0/807188188 wait complete. 2024-09-13T17:17:12.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:17:12.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.686+0000 7ff194a00640 1 Processor -- start 2024-09-13T17:17:12.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.686+0000 7ff194a00640 1 -- start start 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff19015a780 0x7ff190070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19015ab60 0x7ff1900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff19006bb40 0x7ff19006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19006c8b0 con 0x7ff19015a780 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19006cb90 con 0x7ff19006bb40 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff194a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900cc980 con 0x7ff19015ab60 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff19015a780 0x7ff190070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff19015a780 0x7ff190070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34846/0 (socket says 172.21.15.28:34846) 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 -- 172.21.15.28:0/3277129789 learned_addr learned my addr 172.21.15.28:0/3277129789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18d400640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19015ab60 0x7ff1900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18e800640 1 --2- 172.21.15.28:0/3277129789 >> v2:172.21.15.189:3300/0 conn(0x7ff19006bb40 0x7ff19006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 -- 172.21.15.28:0/3277129789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19015ab60 msgr2=0x7ff1900712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:12.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19015ab60 0x7ff1900712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 -- 172.21.15.28:0/3277129789 >> v2:172.21.15.189:3300/0 conn(0x7ff19006bb40 msgr2=0x7ff19006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:12.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 --2- 172.21.15.28:0/3277129789 >> v2:172.21.15.189:3300/0 conn(0x7ff19006bb40 0x7ff19006bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.687+0000 7ff18de00640 1 -- 172.21.15.28:0/3277129789 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1900ccc60 con 0x7ff19015a780 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.688+0000 7ff18de00640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015a780 0x7ff190070d80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff19006f1b0 tx=0x7ff18002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef9cc546fc482371 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.688+0000 7ff18ca00640 1 -- 172.21.15.28:0/3277129789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180032440 con 0x7ff19015a780 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.688+0000 7ff18ca00640 1 -- 172.21.15.28:0/3277129789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1800325a0 con 0x7ff19015a780 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.688+0000 7ff18ca00640 1 -- 172.21.15.28:0/3277129789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180032930 con 0x7ff19015a780 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 -- 172.21.15.28:0/3277129789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015a780 msgr2=0x7ff190070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015a780 0x7ff190070d80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7ff19006f1b0 tx=0x7ff18002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 -- 172.21.15.28:0/3277129789 shutdown_connections 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 --2- 172.21.15.28:0/3277129789 >> v2:172.21.15.189:3300/0 conn(0x7ff19006bb40 0x7ff19006bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19015ab60 0x7ff1900712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 --2- 172.21.15.28:0/3277129789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015a780 0x7ff190070d80 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 -- 172.21.15.28:0/3277129789 >> 172.21.15.28:0/3277129789 conn(0x7ff1900b0270 msgr2=0x7ff1900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 -- 172.21.15.28:0/3277129789 shutdown_connections 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.689+0000 7ff194a00640 1 -- 172.21.15.28:0/3277129789 wait complete. 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.691+0000 7ff194a00640 1 Processor -- start 2024-09-13T17:17:12.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.691+0000 7ff194a00640 1 -- start start 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19006bb40 0x7ff19014f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19015a780 0x7ff19013ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1901404d0 con 0x7ff19015ab60 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff190152b00 con 0x7ff19015a780 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1900cdc40 con 0x7ff19006bb40 2024-09-13T17:17:12.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19006bb40 0x7ff19014f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34862/0 (socket says 172.21.15.28:34862) 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 -- 172.21.15.28:0/3920167608 learned_addr learned my addr 172.21.15.28:0/3920167608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18d400640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19015a780 0x7ff19013ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 -- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19006bb40 msgr2=0x7ff19014f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19006bb40 0x7ff19014f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 -- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19015a780 msgr2=0x7ff19013ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19015a780 0x7ff19013ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 -- 172.21.15.28:0/3920167608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1800094d0 con 0x7ff19015ab60 2024-09-13T17:17:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff18e800640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff188007e70 tx=0x7ff18800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188014620 con 0x7ff19015ab60 2024-09-13T17:17:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1880147a0 con 0x7ff19015ab60 2024-09-13T17:17:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff188014b50 con 0x7ff19015ab60 2024-09-13T17:17:12.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19014f540 con 0x7ff19015ab60 2024-09-13T17:17:12.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.693+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1900cd8d0 con 0x7ff19015ab60 2024-09-13T17:17:12.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.694+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1900cddf0 con 0x7ff19015ab60 2024-09-13T17:17:12.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.695+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff188014cb0 con 0x7ff19015ab60 2024-09-13T17:17:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.696+0000 7ff186000640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 0x7ff17006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.696+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff188089890 con 0x7ff19015ab60 2024-09-13T17:17:12.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.697+0000 7ff18de00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 0x7ff17006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:12.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.697+0000 7ff18de00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 0x7ff17006d0c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff180034000 tx=0x7ff18002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:12.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.701+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff1900cddf0 con 0x7ff19015ab60 2024-09-13T17:17:12.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.878+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff1900cda60 con 0x7ff19015ab60 2024-09-13T17:17:12.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.882+0000 7ff186000640 1 -- 172.21.15.28:0/3920167608 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff188054e10 con 0x7ff19015ab60 2024-09-13T17:17:12.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 msgr2=0x7ff17006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 0x7ff17006d0c0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff180034000 tx=0x7ff18002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 msgr2=0x7ff190150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7ff188007e70 tx=0x7ff18800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 shutdown_connections 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff17006ac00 0x7ff17006d0c0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff19015ab60 0x7ff190150350 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff19015a780 0x7ff19013ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 --2- 172.21.15.28:0/3920167608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff19006bb40 0x7ff19014f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.885+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 >> 172.21.15.28:0/3920167608 conn(0x7ff1900b0270 msgr2=0x7ff1901583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.886+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 shutdown_connections 2024-09-13T17:17:12.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:12.886+0000 7ff194a00640 1 -- 172.21.15.28:0/3920167608 wait complete. 2024-09-13T17:17:12.902 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort' 2024-09-13T17:17:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.058+0000 7f7a91600640 1 Processor -- start 2024-09-13T17:17:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.058+0000 7f7a91600640 1 -- start start 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a91600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15a780 0x7f7a8c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a91600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a8c06c250 0x7f7a8c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a91600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8c06c8b0 con 0x7f7a8c15ab60 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a91600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8c06c6a0 con 0x7f7a8c06c250 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a91600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c06cb90 con 0x7f7a8c15a780 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a8c06c250 0x7f7a8c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7a8c06c250 0x7f7a8c0712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34876/0 (socket says 172.21.15.28:34876) 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a8b400640 1 -- 172.21.15.28:0/2677346798 learned_addr learned my addr 172.21.15.28:0/2677346798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.060+0000 7f7a8aa00640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15a780 0x7f7a8c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.061+0000 7f7a8b400640 1 -- 172.21.15.28:0/2677346798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15a780 msgr2=0x7f7a8c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.061+0000 7f7a8b400640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15a780 0x7f7a8c070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.061+0000 7f7a8b400640 1 -- 172.21.15.28:0/2677346798 >> v1:172.21.15.189:6789/0 conn(0x7f7a8c15ab60 legacy=0x7f7a8c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.061+0000 7f7a8b400640 1 -- 172.21.15.28:0/2677346798 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c0cc980 con 0x7f7a8c06c250 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a8b400640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c0712c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009e50 tx=0x7f7a8402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a554d0a432ea61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a89600640 1 -- 172.21.15.28:0/2677346798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84031600 con 0x7f7a8c06c250 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a89600640 1 -- 172.21.15.28:0/2677346798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8402d050 con 0x7f7a8c06c250 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a89600640 1 -- 172.21.15.28:0/2677346798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a84031980 con 0x7f7a8c06c250 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 -- 172.21.15.28:0/2677346798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 msgr2=0x7f7a8c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c0712c0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f7a84009e50 tx=0x7f7a8402a710 comp rx=0 tx=0).stop 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 -- 172.21.15.28:0/2677346798 shutdown_connections 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c0712c0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 --2- 172.21.15.28:0/2677346798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15a780 0x7f7a8c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 -- 172.21.15.28:0/2677346798 >> 172.21.15.28:0/2677346798 conn(0x7f7a8c0b0270 msgr2=0x7f7a8c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:13.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.062+0000 7f7a91600640 1 -- 172.21.15.28:0/2677346798 shutdown_connections 2024-09-13T17:17:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.063+0000 7f7a91600640 1 -- 172.21.15.28:0/2677346798 wait complete. 2024-09-13T17:17:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.064+0000 7f7a91600640 1 Processor -- start 2024-09-13T17:17:13.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.064+0000 7f7a91600640 1 -- start start 2024-09-13T17:17:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a8c15a780 0x7f7a8c157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15ab60 0x7f7a8c157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1e2930 con 0x7f7a8c06c250 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1e1cb0 con 0x7f7a8c15a780 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a91600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8c1e1f90 con 0x7f7a8c15ab60 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34882/0 (socket says 172.21.15.28:34882) 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.065+0000 7f7a8aa00640 1 -- 172.21.15.28:0/2903732968 learned_addr learned my addr 172.21.15.28:0/2903732968 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8b400640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15ab60 0x7f7a8c157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8a000640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a8c15a780 0x7f7a8c157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 -- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15ab60 msgr2=0x7f7a8c157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15ab60 0x7f7a8c157d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 -- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a8c15a780 msgr2=0x7f7a8c157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a8c15a780 0x7f7a8c157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 -- 172.21.15.28:0/2903732968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a84009b30 con 0x7f7a8c06c250 2024-09-13T17:17:13.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a8aa00640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9b0 tx=0x7f7a7c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c009670 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7c00c9a0 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8c1e2270 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.066+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c00cd50 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.067+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8c14c470 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.068+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8c06bb40 con 0x7f7a8c06c250 2024-09-13T17:17:13.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.069+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7a7c013470 con 0x7f7a8c06c250 2024-09-13T17:17:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.069+0000 7f7a82a00640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 0x7f7a6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.069+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7a7c0893f0 con 0x7f7a8c06c250 2024-09-13T17:17:13.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.070+0000 7f7a8a000640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 0x7f7a6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.071+0000 7f7a8a000640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 0x7f7a6c06d0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7a78007fd0 tx=0x7f7a78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.073+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7a7c054830 con 0x7f7a8c06c250 2024-09-13T17:17:13.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.330+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]} v 0) v1 -- 0x7f7a8c0ae430 con 0x7f7a8c06c250 2024-09-13T17:17:13.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.491+0000 7f7a82a00640 1 -- 172.21.15.28:0/2903732968 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_abort"]}]=0 v463) v1 ==== 144+0+0 (secure 0 0 0) 0x7f7a7c057e50 con 0x7f7a8c06c250 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 msgr2=0x7f7a6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 0x7f7a6c06d0c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f7a78007fd0 tx=0x7f7a78009040 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 msgr2=0x7f7a8c14bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00e9b0 tx=0x7f7a7c00ee70 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 shutdown_connections 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7a6c06ac00 0x7f7a6c06d0c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7a8c15ab60 0x7f7a8c157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a8c15a780 0x7f7a8c157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 --2- 172.21.15.28:0/2903732968 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7a8c06c250 0x7f7a8c14bf30 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 >> 172.21.15.28:0/2903732968 conn(0x7f7a8c0b0270 msgr2=0x7f7a8c06fd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 shutdown_connections 2024-09-13T17:17:13.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.493+0000 7f7a91600640 1 -- 172.21.15.28:0/2903732968 wait complete. 2024-09-13T17:17:13.510 INFO:tasks.cephfs_test_runner:test_scrub_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-09-13T17:17:13.510 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-09-13T17:17:13.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:13.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.666+0000 7fde1e200640 1 Processor -- start 2024-09-13T17:17:13.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.666+0000 7fde1e200640 1 -- start start 2024-09-13T17:17:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.667+0000 7fde1e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde181592a0 0x7fde1815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.667+0000 7fde1e200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 0x7fde18159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.667+0000 7fde1e200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde180b2040 con 0x7fde1806c2d0 2024-09-13T17:17:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.667+0000 7fde1e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1815b0b0 con 0x7fde181592a0 2024-09-13T17:17:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.667+0000 7fde1e200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde18159ff0 con 0x7fde1806f970 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde181592a0 0x7fde1815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde181592a0 0x7fde1815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34898/0 (socket says 172.21.15.28:34898) 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 -- 172.21.15.28:0/4190831490 learned_addr learned my addr 172.21.15.28:0/4190831490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde0fe00640 1 --2- 172.21.15.28:0/4190831490 >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 0x7fde18159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 -- 172.21.15.28:0/4190831490 >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 msgr2=0x7fde18159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 --2- 172.21.15.28:0/4190831490 >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 0x7fde18159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 -- 172.21.15.28:0/4190831490 >> v1:172.21.15.189:6789/0 conn(0x7fde1806c2d0 legacy=0x7fde180b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde17400640 1 -- 172.21.15.28:0/4190831490 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1815a2d0 con 0x7fde181592a0 2024-09-13T17:17:13.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.668+0000 7fde0fe00640 1 --2- 172.21.15.28:0/4190831490 >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 0x7fde18159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:13.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.669+0000 7fde17400640 1 --2- 172.21.15.28:0/4190831490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde1815ab70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fde1806ac10 tx=0x7fde0802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52413b28a8222e2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.669+0000 7fde16a00640 1 -- 172.21.15.28:0/4190831490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08032440 con 0x7fde181592a0 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde16a00640 1 -- 172.21.15.28:0/4190831490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde080325a0 con 0x7fde181592a0 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde16a00640 1 -- 172.21.15.28:0/4190831490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08032930 con 0x7fde181592a0 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 -- 172.21.15.28:0/4190831490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 msgr2=0x7fde1815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 --2- 172.21.15.28:0/4190831490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde1815ab70 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fde1806ac10 tx=0x7fde0802a230 comp rx=0 tx=0).stop 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 -- 172.21.15.28:0/4190831490 shutdown_connections 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 --2- 172.21.15.28:0/4190831490 >> v2:172.21.15.189:3300/0 conn(0x7fde1806f970 0x7fde18159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 --2- 172.21.15.28:0/4190831490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde1815ab70 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 -- 172.21.15.28:0/4190831490 >> 172.21.15.28:0/4190831490 conn(0x7fde18069e80 msgr2=0x7fde1806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 -- 172.21.15.28:0/4190831490 shutdown_connections 2024-09-13T17:17:13.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.670+0000 7fde1e200640 1 -- 172.21.15.28:0/4190831490 wait complete. 2024-09-13T17:17:13.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.672+0000 7fde1e200640 1 Processor -- start 2024-09-13T17:17:13.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.672+0000 7fde1e200640 1 -- start start 2024-09-13T17:17:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde1806c2d0 0x7fde18151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 0x7fde180cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180ccd60 con 0x7fde181592a0 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde180cc4c0 con 0x7fde1806f970 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde1e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde180cc7a0 con 0x7fde1806c2d0 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34902/0 (socket says 172.21.15.28:34902) 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde17e00640 1 -- 172.21.15.28:0/4294377176 learned_addr learned my addr 172.21.15.28:0/4294377176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.673+0000 7fde0fe00640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 0x7fde180cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17400640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde1806c2d0 0x7fde18151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 -- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde1806c2d0 msgr2=0x7fde18151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde1806c2d0 0x7fde18151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 -- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 msgr2=0x7fde180cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 0x7fde180cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 -- 172.21.15.28:0/4294377176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde080094d0 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde0fe00640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 0x7fde180cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde17e00640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c780 tx=0x7fde1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10004b60 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1000ce90 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000a850 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde180cca80 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.674+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde180cdc10 con 0x7fde181592a0 2024-09-13T17:17:13.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.675+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde18151e70 con 0x7fde181592a0 2024-09-13T17:17:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.676+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fde1000a9b0 con 0x7fde181592a0 2024-09-13T17:17:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.677+0000 7fde0f400640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 0x7fddf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.677+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fde10089230 con 0x7fde181592a0 2024-09-13T17:17:13.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.678+0000 7fde17400640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 0x7fddf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:13.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.678+0000 7fde17400640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 0x7fddf406d0c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fde08034000 tx=0x7fde0802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:13.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.681+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fde18151e70 con 0x7fde181592a0 2024-09-13T17:17:13.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.860+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde18151e70 con 0x7fde181592a0 2024-09-13T17:17:13.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.861+0000 7fde0f400640 1 -- 172.21.15.28:0/4294377176 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+12994 (secure 0 0 0) 0x7fde18151e70 con 0x7fde181592a0 2024-09-13T17:17:13.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:13.862 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:12.485398+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/199464366":"2024-09-14T17:17:09.457262+0000","172.21.15.28:6833/199464366":"2024-09-14T17:17:09.457262+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 msgr2=0x7fddf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 0x7fddf406d0c0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fde08034000 tx=0x7fde0802e000 comp rx=0 tx=0).stop 2024-09-13T17:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 msgr2=0x7fde181544c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:13.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fde1000c780 tx=0x7fde1000cc40 comp rx=0 tx=0).stop 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 shutdown_connections 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddf406ac00 0x7fddf406d0c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde181592a0 0x7fde181544c0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde1806f970 0x7fde180cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 --2- 172.21.15.28:0/4294377176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde1806c2d0 0x7fde18151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 >> 172.21.15.28:0/4294377176 conn(0x7fde18069e80 msgr2=0x7fde180b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 shutdown_connections 2024-09-13T17:17:13.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:13.863+0000 7fde1e200640 1 -- 172.21.15.28:0/4294377176 wait complete. 2024-09-13T17:17:13.880 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:17:13.881 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:17:14.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.038+0000 7f33ac800640 1 Processor -- start 2024-09-13T17:17:14.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.038+0000 7f33ac800640 1 -- start start 2024-09-13T17:17:14.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33ac800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a4159770 0x7f33a406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33ac800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415cd80 0x7f33a4070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33ac800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a406f920 con 0x7f33a406f3f0 2024-09-13T17:17:14.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33ac800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a415b290 con 0x7f33a4159770 2024-09-13T17:17:14.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33ac800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a415b570 con 0x7f33a415cd80 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.039+0000 7f33a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a4159770 0x7f33a406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f33a4159770 0x7f33a406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54114/0 (socket says 172.21.15.28:54114) 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 -- 172.21.15.28:0/1158609585 learned_addr learned my addr 172.21.15.28:0/1158609585 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33aa800640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415cd80 0x7f33a4070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 -- 172.21.15.28:0/1158609585 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415cd80 msgr2=0x7f33a4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415cd80 0x7f33a4070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 -- 172.21.15.28:0/1158609585 >> v1:172.21.15.28:6789/0 conn(0x7f33a406f3f0 legacy=0x7f33a4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.040+0000 7f33a9e00640 1 -- 172.21.15.28:0/1158609585 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a415b850 con 0x7f33a4159770 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.041+0000 7f33a9e00640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a4159770 0x7f33a406fd20 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f339800bb10 tx=0x7f339802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6056558706ba7916 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.041+0000 7f33a8a00640 1 -- 172.21.15.28:0/1158609585 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398032790 con 0x7f33a4159770 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.041+0000 7f33a8a00640 1 -- 172.21.15.28:0/1158609585 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33980328f0 con 0x7f33a4159770 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.041+0000 7f33a8a00640 1 -- 172.21.15.28:0/1158609585 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3398032c80 con 0x7f33a4159770 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 -- 172.21.15.28:0/1158609585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a4159770 msgr2=0x7f33a406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a4159770 0x7f33a406fd20 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7f339800bb10 tx=0x7f339802c890 comp rx=0 tx=0).stop 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 -- 172.21.15.28:0/1158609585 shutdown_connections 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a415cd80 0x7f33a4070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 --2- 172.21.15.28:0/1158609585 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a4159770 0x7f33a406fd20 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 -- 172.21.15.28:0/1158609585 >> 172.21.15.28:0/1158609585 conn(0x7f33a40b0270 msgr2=0x7f33a40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:14.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 -- 172.21.15.28:0/1158609585 shutdown_connections 2024-09-13T17:17:14.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.042+0000 7f33ac800640 1 -- 172.21.15.28:0/1158609585 wait complete. 2024-09-13T17:17:14.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.044+0000 7f33ac800640 1 Processor -- start 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.044+0000 7f33ac800640 1 -- start start 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a406f3f0 0x7f33a414fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a4159770 0x7f33a40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40c5220 con 0x7f33a415cd80 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40c45a0 con 0x7f33a406f3f0 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33ac800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a40c4880 con 0x7f33a4159770 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54118/0 (socket says 172.21.15.28:54118) 2024-09-13T17:17:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33aa800640 1 -- 172.21.15.28:0/3184592590 learned_addr learned my addr 172.21.15.28:0/3184592590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33a9400640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a4159770 0x7f33a40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.045+0000 7f33a9e00640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a406f3f0 0x7f33a414fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 -- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a4159770 msgr2=0x7f33a40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a4159770 0x7f33a40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 -- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a406f3f0 msgr2=0x7f33a414fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a406f3f0 0x7f33a414fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 -- 172.21.15.28:0/3184592590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f339800b770 con 0x7f33a415cd80 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33aa800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009fd0 tx=0x7f33a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a0014420 con 0x7f33a415cd80 2024-09-13T17:17:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a0014580 con 0x7f33a415cd80 2024-09-13T17:17:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a40c4b60 con 0x7f33a415cd80 2024-09-13T17:17:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a0014930 con 0x7f33a415cd80 2024-09-13T17:17:14.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.046+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a4150280 con 0x7f33a415cd80 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.047+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a406fd20 con 0x7f33a415cd80 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.048+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f33a0014a90 con 0x7f33a415cd80 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.049+0000 7f339e000640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 0x7f338806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.049+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f33a00898a0 con 0x7f33a415cd80 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.050+0000 7f33a9e00640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 0x7f338806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.050+0000 7f33a9e00640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 0x7f338806d0f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3398037000 tx=0x7f3398032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.054+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f33a0054d90 con 0x7f33a415cd80 2024-09-13T17:17:14.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.229+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f33a4158990 con 0x7f33a415cd80 2024-09-13T17:17:14.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.230+0000 7f339e000640 1 -- 172.21.15.28:0/3184592590 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v130) v1 ==== 93+0+10 (secure 0 0 0) 0x7f33a00583b0 con 0x7f33a415cd80 2024-09-13T17:17:14.231 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:17:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 msgr2=0x7f338806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 0x7f338806d0f0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f3398037000 tx=0x7f3398032000 comp rx=0 tx=0).stop 2024-09-13T17:17:14.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 msgr2=0x7f33a40acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f33a0009fd0 tx=0x7f33a000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 shutdown_connections 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f338806ac30 0x7f338806d0f0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f33a415cd80 0x7f33a40acd90 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f33a4159770 0x7f33a40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 --2- 172.21.15.28:0/3184592590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f33a406f3f0 0x7f33a414fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 >> 172.21.15.28:0/3184592590 conn(0x7f33a40b0270 msgr2=0x7f33a40af3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 shutdown_connections 2024-09-13T17:17:14.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.232+0000 7f33ac800640 1 -- 172.21.15.28:0/3184592590 wait complete. 2024-09-13T17:17:14.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume' 2024-09-13T17:17:14.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.403+0000 7f26a8800640 1 Processor -- start 2024-09-13T17:17:14.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.403+0000 7f26a8800640 1 -- start start 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.404+0000 7f26a8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a0160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a8800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a0146550 con 0x7f26a00ac850 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a8800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a015d280 con 0x7f26a0146170 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a015cd20 con 0x7f26a0160310 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a6800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26a00ac850 0x7f26a006b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48422/0 (socket says 172.21.15.28:48422) 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a6800640 1 -- 172.21.15.28:0/682628353 learned_addr learned my addr 172.21.15.28:0/682628353 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:14.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a5400640 1 --2- 172.21.15.28:0/682628353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a0160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a4a00640 1 -- 172.21.15.28:0/682628353 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014503807 0 0) 0x7f26a0146550 con 0x7f26a00ac850 2024-09-13T17:17:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.405+0000 7f26a4a00640 1 -- 172.21.15.28:0/682628353 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003520 con 0x7f26a00ac850 2024-09-13T17:17:14.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.406+0000 7f26a5400640 1 -- 172.21.15.28:0/682628353 >> v1:172.21.15.189:6789/0 conn(0x7f26a0146170 legacy=0x7f26a015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.406+0000 7f26a5400640 1 -- 172.21.15.28:0/682628353 >> v1:172.21.15.28:6789/0 conn(0x7f26a00ac850 legacy=0x7f26a006b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.406+0000 7f26a5400640 1 -- 172.21.15.28:0/682628353 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0160d60 con 0x7f26a0160310 2024-09-13T17:17:14.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.407+0000 7f26a5400640 1 --2- 172.21.15.28:0/682628353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a0160730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26900097d0 tx=0x7f269002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=544c52a26c5a678a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.407+0000 7f26a4a00640 1 -- 172.21.15.28:0/682628353 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690032440 con 0x7f26a0160310 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.407+0000 7f26a4a00640 1 -- 172.21.15.28:0/682628353 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26900325a0 con 0x7f26a0160310 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.407+0000 7f26a4a00640 1 -- 172.21.15.28:0/682628353 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690032910 con 0x7f26a0160310 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 -- 172.21.15.28:0/682628353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 msgr2=0x7f26a0160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 --2- 172.21.15.28:0/682628353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a0160730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f26900097d0 tx=0x7f269002a200 comp rx=0 tx=0).stop 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 -- 172.21.15.28:0/682628353 shutdown_connections 2024-09-13T17:17:14.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 --2- 172.21.15.28:0/682628353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a0160730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 -- 172.21.15.28:0/682628353 >> 172.21.15.28:0/682628353 conn(0x7f26a0069e80 msgr2=0x7f26a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 -- 172.21.15.28:0/682628353 shutdown_connections 2024-09-13T17:17:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.408+0000 7f26a8800640 1 -- 172.21.15.28:0/682628353 wait complete. 2024-09-13T17:17:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.409+0000 7f26a8800640 1 Processor -- start 2024-09-13T17:17:14.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.409+0000 7f26a8800640 1 -- start start 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a00ac850 0x7f26a013bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a013ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01426c0 con 0x7f26a0146170 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0142020 con 0x7f26a00ac850 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.410+0000 7f26a8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a013b0b0 con 0x7f26a0160310 2024-09-13T17:17:14.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54124/0 (socket says 172.21.15.28:54124) 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 -- 172.21.15.28:0/3833603393 learned_addr learned my addr 172.21.15.28:0/3833603393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a6800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a013ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5e00640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a00ac850 0x7f26a013bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 -- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 msgr2=0x7f26a013ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a013ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 -- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a00ac850 msgr2=0x7f26a013bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a00ac850 0x7f26a013bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.411+0000 7f26a5400640 1 -- 172.21.15.28:0/3833603393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2690009400 con 0x7f26a0146170 2024-09-13T17:17:14.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f26a5400640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2690000c00 tx=0x7f26900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690003810 con 0x7f26a0146170 2024-09-13T17:17:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2690003970 con 0x7f26a0146170 2024-09-13T17:17:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2690003ce0 con 0x7f26a0146170 2024-09-13T17:17:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a013c3b0 con 0x7f26a0146170 2024-09-13T17:17:14.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.412+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a013c970 con 0x7f26a0146170 2024-09-13T17:17:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.413+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0160730 con 0x7f26a0146170 2024-09-13T17:17:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.414+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2690039040 con 0x7f26a0146170 2024-09-13T17:17:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.415+0000 7f269a000640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 0x7f268406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.415+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f26900a9b80 con 0x7f26a0146170 2024-09-13T17:17:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.415+0000 7f26a5e00640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 0x7f268406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.416+0000 7f26a5e00640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 0x7f268406d0c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2694000f60 tx=0x7f2694009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.419+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2690075220 con 0x7f26a0146170 2024-09-13T17:17:14.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:14.662+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]} v 0) v1 -- 0x7f26a006b730 con 0x7f26a0146170 2024-09-13T17:17:15.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.498+0000 7f269a000640 1 -- 172.21.15.28:0/3833603393 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]}]=0 v465) v1 ==== 158+0+0 (secure 0 0 0) 0x7f26900044e0 con 0x7f26a0146170 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 msgr2=0x7f268406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 0x7f268406d0c0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f2694000f60 tx=0x7f2694009040 comp rx=0 tx=0).stop 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 msgr2=0x7f26a0141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f2690000c00 tx=0x7f26900046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 shutdown_connections 2024-09-13T17:17:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268406ac00 0x7f268406d0c0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0160310 0x7f26a013ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0146170 0x7f26a0141ae0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 --2- 172.21.15.28:0/3833603393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a00ac850 0x7f26a013bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 >> 172.21.15.28:0/3833603393 conn(0x7f26a0069e80 msgr2=0x7f26a00cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 shutdown_connections 2024-09-13T17:17:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.501+0000 7f26a8800640 1 -- 172.21.15.28:0/3833603393 wait complete. 2024-09-13T17:17:15.522 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:17:15.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.684+0000 7f4958c00640 1 Processor -- start 2024-09-13T17:17:15.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.684+0000 7f4958c00640 1 -- start start 2024-09-13T17:17:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.685+0000 7f4958c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 0x7f495406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.685+0000 7f4958c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 0x7f4954070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.685+0000 7f4958c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f495406f920 con 0x7f495406f3f0 2024-09-13T17:17:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.685+0000 7f4958c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f495415b290 con 0x7f4954159770 2024-09-13T17:17:15.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.685+0000 7f4958c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f495415b570 con 0x7f495415cd80 2024-09-13T17:17:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4952800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 0x7f4954070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4951400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f495406f3f0 0x7f4954159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:50764/0 (socket says 172.21.15.28:50764) 2024-09-13T17:17:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4951400640 1 -- 172.21.15.28:0/2776201437 learned_addr learned my addr 172.21.15.28:0/2776201437 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:15.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4951e00640 1 --2- 172.21.15.28:0/2776201437 >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 0x7f495406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4950a00640 1 -- 172.21.15.28:0/2776201437 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580795949 0 0) 0x7f495406f920 con 0x7f495406f3f0 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4950a00640 1 -- 172.21.15.28:0/2776201437 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4938003520 con 0x7f495406f3f0 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4952800640 1 -- 172.21.15.28:0/2776201437 >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 msgr2=0x7f495406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4952800640 1 --2- 172.21.15.28:0/2776201437 >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 0x7f495406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4952800640 1 -- 172.21.15.28:0/2776201437 >> v1:172.21.15.189:6789/0 conn(0x7f495406f3f0 legacy=0x7f4954159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4952800640 1 -- 172.21.15.28:0/2776201437 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f495415b850 con 0x7f495415cd80 2024-09-13T17:17:15.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.686+0000 7f4951e00640 1 --2- 172.21.15.28:0/2776201437 >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 0x7f495406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.688+0000 7f4952800640 1 --2- 172.21.15.28:0/2776201437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 0x7f4954070260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f494c009d90 tx=0x7f494c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903448e6706d6597 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.688+0000 7f4950a00640 1 -- 172.21.15.28:0/2776201437 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c031600 con 0x7f495415cd80 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.688+0000 7f4950a00640 1 -- 172.21.15.28:0/2776201437 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c02d050 con 0x7f495415cd80 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.688+0000 7f4950a00640 1 -- 172.21.15.28:0/2776201437 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c031980 con 0x7f495415cd80 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 -- 172.21.15.28:0/2776201437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 msgr2=0x7f4954070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 --2- 172.21.15.28:0/2776201437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 0x7f4954070260 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f494c009d90 tx=0x7f494c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 -- 172.21.15.28:0/2776201437 shutdown_connections 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 --2- 172.21.15.28:0/2776201437 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f495415cd80 0x7f4954070260 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 --2- 172.21.15.28:0/2776201437 >> v2:172.21.15.189:3300/0 conn(0x7f4954159770 0x7f495406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 -- 172.21.15.28:0/2776201437 >> 172.21.15.28:0/2776201437 conn(0x7f49540b0270 msgr2=0x7f49540b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 -- 172.21.15.28:0/2776201437 shutdown_connections 2024-09-13T17:17:15.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.689+0000 7f4958c00640 1 -- 172.21.15.28:0/2776201437 wait complete. 2024-09-13T17:17:15.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.690+0000 7f4958c00640 1 Processor -- start 2024-09-13T17:17:15.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.691+0000 7f4958c00640 1 -- start start 2024-09-13T17:17:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.691+0000 7f4958c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.691+0000 7f4958c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4954159770 0x7f495414d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4958c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f495415cd80 0x7f495414b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4958c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4954148480 con 0x7f495406f3f0 2024-09-13T17:17:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4958c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49541556c0 con 0x7f495415cd80 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4958c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49541559a0 con 0x7f4954159770 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54130/0 (socket says 172.21.15.28:54130) 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 -- 172.21.15.28:0/493373081 learned_addr learned my addr 172.21.15.28:0/493373081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951400640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4954159770 0x7f495414d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4952800640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f495415cd80 0x7f495414b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 -- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4954159770 msgr2=0x7f495414d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4954159770 0x7f495414d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 -- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f495415cd80 msgr2=0x7f495414b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f495415cd80 0x7f495414b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.692+0000 7f4951e00640 1 -- 172.21.15.28:0/493373081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944009400 con 0x7f495406f3f0 2024-09-13T17:17:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f4951e00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f494400ebe0 tx=0x7f494400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944009760 con 0x7f495406f3f0 2024-09-13T17:17:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494400cc30 con 0x7f495406f3f0 2024-09-13T17:17:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494400eee0 con 0x7f495406f3f0 2024-09-13T17:17:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c009a70 con 0x7f495406f3f0 2024-09-13T17:17:15.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.693+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4954148110 con 0x7f495406f3f0 2024-09-13T17:17:15.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.694+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f495406fd20 con 0x7f495406f3f0 2024-09-13T17:17:15.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.695+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4944013470 con 0x7f495406f3f0 2024-09-13T17:17:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.696+0000 7f494a000640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 0x7f493406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.696+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4944089600 con 0x7f495406f3f0 2024-09-13T17:17:15.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.696+0000 7f4951400640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 0x7f493406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.697+0000 7f4951400640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 0x7f493406d0c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f49540b0030 tx=0x7f4940007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.700+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4944054b80 con 0x7f495406f3f0 2024-09-13T17:17:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.876+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f495406d070 con 0x7f495406f3f0 2024-09-13T17:17:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.880+0000 7f494a000640 1 -- 172.21.15.28:0/493373081 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v38)=0 v38) v1 ==== 122+0+0 (secure 0 0 0) 0x7f49440581a0 con 0x7f495406f3f0 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.882+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 msgr2=0x7f493406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.882+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 0x7f493406d0c0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f49540b0030 tx=0x7f4940007d30 comp rx=0 tx=0).stop 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 msgr2=0x7f4954147850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f494400ebe0 tx=0x7f494400c520 comp rx=0 tx=0).stop 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 shutdown_connections 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f493406ac00 0x7f493406d0c0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f495415cd80 0x7f495414b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4954159770 0x7f495414d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 --2- 172.21.15.28:0/493373081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f495406f3f0 0x7f4954147850 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:15.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 >> 172.21.15.28:0/493373081 conn(0x7f49540b0270 msgr2=0x7f49541589c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:15.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 shutdown_connections 2024-09-13T17:17:15.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:15.883+0000 7f4958c00640 1 -- 172.21.15.28:0/493373081 wait complete. 2024-09-13T17:17:15.900 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:17:15.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:15.985 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:16.076 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:17:16.076 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:16.076 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:16.111 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:17:16.111 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:17:16.112 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:16.149 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:17:16.149 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:16.149 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:16.187 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:17:16.187 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:17:16.187 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:16.226 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:17:16.226 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:17:16.226 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.382+0000 7fc300000640 1 Processor -- start 2024-09-13T17:17:16.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.382+0000 7fc300000640 1 -- start start 2024-09-13T17:17:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc300000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 0x7fc2f815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc300000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 0x7fc2f806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc300000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2f806cc70 con 0x7fc2f806f970 2024-09-13T17:17:16.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc300000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2f815b0b0 con 0x7fc2f81592a0 2024-09-13T17:17:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc300000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f8159d90 con 0x7fc2f806c2d0 2024-09-13T17:17:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc2fcc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc2f806f970 0x7fc2f8159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48424/0 (socket says 172.21.15.28:48424) 2024-09-13T17:17:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.383+0000 7fc2fcc00640 1 -- 172.21.15.28:0/1451027560 learned_addr learned my addr 172.21.15.28:0/1451027560 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fd600640 1 --2- 172.21.15.28:0/1451027560 >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 0x7fc2f815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fe000640 1 --2- 172.21.15.28:0/1451027560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 0x7fc2f806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2f7e00640 1 -- 172.21.15.28:0/1451027560 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566548114 0 0) 0x7fc2f806cc70 con 0x7fc2f806f970 2024-09-13T17:17:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2f7e00640 1 -- 172.21.15.28:0/1451027560 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2e0005730 con 0x7fc2f806f970 2024-09-13T17:17:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fe000640 1 -- 172.21.15.28:0/1451027560 >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 msgr2=0x7fc2f815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fe000640 1 --2- 172.21.15.28:0/1451027560 >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 0x7fc2f815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fe000640 1 -- 172.21.15.28:0/1451027560 >> v1:172.21.15.28:6789/0 conn(0x7fc2f806f970 legacy=0x7fc2f8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fe000640 1 -- 172.21.15.28:0/1451027560 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f80b2040 con 0x7fc2f806c2d0 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.384+0000 7fc2fd600640 1 --2- 172.21.15.28:0/1451027560 >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 0x7fc2f815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.385+0000 7fc2fe000640 1 --2- 172.21.15.28:0/1451027560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 0x7fc2f806c730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d90 tx=0x7fc2f002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37dc8ce801dc2fa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.385+0000 7fc2f7e00640 1 -- 172.21.15.28:0/1451027560 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007740 con 0x7fc2f806c2d0 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.385+0000 7fc2f7e00640 1 -- 172.21.15.28:0/1451027560 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f002d050 con 0x7fc2f806c2d0 2024-09-13T17:17:16.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.385+0000 7fc2f7e00640 1 -- 172.21.15.28:0/1451027560 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0007ac0 con 0x7fc2f806c2d0 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 -- 172.21.15.28:0/1451027560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 msgr2=0x7fc2f806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 --2- 172.21.15.28:0/1451027560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 0x7fc2f806c730 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc2f0009d90 tx=0x7fc2f002a660 comp rx=0 tx=0).stop 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 -- 172.21.15.28:0/1451027560 shutdown_connections 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 --2- 172.21.15.28:0/1451027560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f806c2d0 0x7fc2f806c730 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 --2- 172.21.15.28:0/1451027560 >> v2:172.21.15.189:3300/0 conn(0x7fc2f81592a0 0x7fc2f815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 -- 172.21.15.28:0/1451027560 >> 172.21.15.28:0/1451027560 conn(0x7fc2f8069e80 msgr2=0x7fc2f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 -- 172.21.15.28:0/1451027560 shutdown_connections 2024-09-13T17:17:16.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.386+0000 7fc300000640 1 -- 172.21.15.28:0/1451027560 wait complete. 2024-09-13T17:17:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.388+0000 7fc300000640 1 Processor -- start 2024-09-13T17:17:16.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.388+0000 7fc300000640 1 -- start start 2024-09-13T17:17:16.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2f806c2d0 0x7fc2f80ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f81592a0 0x7fc2f80cb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80cf170 con 0x7fc2f806f970 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80af890 con 0x7fc2f806c2d0 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc300000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2f80aec80 con 0x7fc2f81592a0 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc2fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc2fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54144/0 (socket says 172.21.15.28:54144) 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc2fcc00640 1 -- 172.21.15.28:0/2501396946 learned_addr learned my addr 172.21.15.28:0/2501396946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc2fd600640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2f806c2d0 0x7fc2f80ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.389+0000 7fc2fe000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f81592a0 0x7fc2f80cb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 -- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f81592a0 msgr2=0x7fc2f80cb7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f81592a0 0x7fc2f80cb7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 -- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2f806c2d0 msgr2=0x7fc2f80ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2f806c2d0 0x7fc2f80ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 -- 172.21.15.28:0/2501396946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec009400 con 0x7fc2f806f970 2024-09-13T17:17:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2fcc00640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4001eb0 tx=0x7fc2e400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4005ce0 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e4005e40 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e4007270 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2f0009a70 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.390+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2f80af240 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.391+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2f80cb4e0 con 0x7fc2f806f970 2024-09-13T17:17:16.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.392+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc2e4007490 con 0x7fc2f806f970 2024-09-13T17:17:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.393+0000 7fc2f5600640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 0x7fc2d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.393+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc2e4088550 con 0x7fc2f806f970 2024-09-13T17:17:16.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.393+0000 7fc2fd600640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 0x7fc2d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.394+0000 7fc2fd600640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 0x7fc2d806d0c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009c30 tx=0x7fc2ec009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.398+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc2f80cb4e0 con 0x7fc2f806f970 2024-09-13T17:17:16.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.613+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2f80cb4e0 con 0x7fc2f806f970 2024-09-13T17:17:16.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.614+0000 7fc2f5600640 1 -- 172.21.15.28:0/2501396946 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+3537 (secure 0 0 0) 0x7fc2f80cb4e0 con 0x7fc2f806f970 2024-09-13T17:17:16.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:16.615 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":116,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7046,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/1309824366","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1309824366},{"type":"v1","addr":"172.21.15.189:6836","nonce":1309824366}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7052,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3627238404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3627238404},{"type":"v1","addr":"172.21.15.28:6834","nonce":3627238404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":7930,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/450618547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":450618547},{"type":"v1","addr":"172.21.15.189:6837","nonce":450618547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":109},{"gid":8808,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2924786445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2924786445},{"type":"v1","addr":"172.21.15.28:6835","nonce":2924786445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":116}],"filesystems":[]} 2024-09-13T17:17:16.615 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 116 2024-09-13T17:17:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 msgr2=0x7fc2d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 0x7fc2d806d0c0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009c30 tx=0x7fc2ec009190 comp rx=0 tx=0).stop 2024-09-13T17:17:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 msgr2=0x7fc2f80cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7fc2e4001eb0 tx=0x7fc2e400c960 comp rx=0 tx=0).stop 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 shutdown_connections 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2d806ac00 0x7fc2d806d0c0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2f81592a0 0x7fc2f80cb7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2f806f970 0x7fc2f80cec30 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 --2- 172.21.15.28:0/2501396946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2f806c2d0 0x7fc2f80ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 >> 172.21.15.28:0/2501396946 conn(0x7fc2f8069e80 msgr2=0x7fc2f80ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 shutdown_connections 2024-09-13T17:17:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.616+0000 7fc300000640 1 -- 172.21.15.28:0/2501396946 wait complete. 2024-09-13T17:17:16.634 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:17:16.634 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:17:16.634 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:17:16.634 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:17:16.634 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:17:16.635 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:17:16.635 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:17:16.635 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:17:16.635 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:17:16.635 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:17:16.635 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:17:16.635 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:17:16.636 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:17:16.636 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:17:16.636 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:17:16.636 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:17:16.650 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:17:16.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:17:16.651 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:17:16.651 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:17:16.652 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:17:16.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:17:16.653 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:17:16.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:17:16.696 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:17:16.696 DEBUG:teuthology.parallel:result is None 2024-09-13T17:17:16.697 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:17:16.697 DEBUG:teuthology.parallel:result is None 2024-09-13T17:17:16.698 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:17:16.698 DEBUG:teuthology.parallel:result is None 2024-09-13T17:17:16.698 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:17:16.699 DEBUG:teuthology.parallel:result is None 2024-09-13T17:17:16.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:17:16.825 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:17:16.826 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:17:16.827 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:17:16.829 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.868+0000 7fcabc200640 1 Processor -- start 2024-09-13T17:17:16.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.868+0000 7fcabc200640 1 -- start start 2024-09-13T17:17:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.869+0000 7fcabc200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcab406c540 0x7fcab40b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcabc200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab41610a0 con 0x7fcab406d070 2024-09-13T17:17:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcabc200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcab40b2620 con 0x7fcab40b17e0 2024-09-13T17:17:16.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcabc200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcab40b2900 con 0x7fcab406c540 2024-09-13T17:17:16.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcab8e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcab406d070 0x7fcab41687d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48432/0 (socket says 172.21.15.28:48432) 2024-09-13T17:17:16.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcab8e00640 1 -- 172.21.15.28:0/4088465534 learned_addr learned my addr 172.21.15.28:0/4088465534 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> v2:172.21.15.189:3300/0 conn(0x7fcab406c540 0x7fcab40b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288856244 0 0) 0x7fcab41610a0 con 0x7fcab406d070 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca9c003520 con 0x7fcab406d070 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.870+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794491615 0 0) 0x7fca9c003520 con 0x7fcab406d070 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 >> v2:172.21.15.189:3300/0 conn(0x7fcab406c540 msgr2=0x7fcab40b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 --2- 172.21.15.28:0/4088465534 >> v2:172.21.15.189:3300/0 conn(0x7fcab406c540 0x7fcab40b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 >> v1:172.21.15.189:6789/0 conn(0x7fcab40b17e0 legacy=0x7fcab40b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab40b2be0 con 0x7fcab406d070 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> v2:172.21.15.189:3300/0 conn(0x7fcab406c540 0x7fcab40b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fcaa0001670 con 0x7fcab406d070 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 >> v1:172.21.15.28:6789/0 conn(0x7fcab406d070 legacy=0x7fcab41687d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0039c0 0x7fca9c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c0047c0 0x7fca9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 0x7fca9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab40b2900 con 0x7fca9c005180 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c00c080 con 0x7fca9c0047c0 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c00c240 con 0x7fca9c0039c0 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 0x7fca9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab9800640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0039c0 0x7fca9c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.871+0000 7fcab8e00640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c0047c0 0x7fca9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 -- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0039c0 msgr2=0x7fca9c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0039c0 0x7fca9c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 -- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c0047c0 msgr2=0x7fca9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c0047c0 0x7fca9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 -- 172.21.15.28:0/4088465534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c003520 con 0x7fca9c005180 2024-09-13T17:17:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 -- 172.21.15.28:0/4088465534 reap_dead start 2024-09-13T17:17:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.872+0000 7fcaba200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 0x7fca9c00b980 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00e320 tx=0x7fca9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d7db9a6fc8e69d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac003b90 con 0x7fca9c005180 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaac00f040 con 0x7fca9c005180 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcab3e00640 1 -- 172.21.15.28:0/4088465534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaac004020 con 0x7fca9c005180 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 -- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 msgr2=0x7fca9c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 0x7fca9c00b980 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7fcaac00e320 tx=0x7fca9c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 -- 172.21.15.28:0/4088465534 shutdown_connections 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 --2- 172.21.15.28:0/4088465534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c005180 0x7fca9c00b980 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 -- 172.21.15.28:0/4088465534 >> 172.21.15.28:0/4088465534 conn(0x7fcab40b0270 msgr2=0x7fcab40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.873+0000 7fcabc200640 1 -- 172.21.15.28:0/4088465534 shutdown_connections 2024-09-13T17:17:16.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.874+0000 7fcabc200640 1 -- 172.21.15.28:0/4088465534 wait complete. 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.875+0000 7fcabc200640 1 Processor -- start 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.875+0000 7fcabc200640 1 -- start start 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c005180 0x7fcab4154cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 0x7fcab41e6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab41e8a30 con 0x7fcab4159a40 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab415a400 con 0x7fcab41e85b0 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcabc200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcab4157a50 con 0x7fca9c005180 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcaba200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcaba200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54196/0 (socket says 172.21.15.28:54196) 2024-09-13T17:17:16.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcaba200640 1 -- 172.21.15.28:0/2317000190 learned_addr learned my addr 172.21.15.28:0/2317000190 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcab8e00640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 0x7fcab41e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.876+0000 7fcab9800640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c005180 0x7fcab4154cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 -- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c005180 msgr2=0x7fcab4154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c005180 0x7fcab4154cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 -- 172.21.15.28:0/2317000190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 msgr2=0x7fcab4159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 -- 172.21.15.28:0/2317000190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab41e74f0 con 0x7fcab41e85b0 2024-09-13T17:17:16.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcaba200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.877+0000 7fcab8e00640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 0x7fcab41e6540 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0004ec0 tx=0x7fcaa0007430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.878+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000f750 con 0x7fcab41e85b0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.878+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaa000f8d0 con 0x7fcab41e85b0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.878+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa000fc80 con 0x7fcab41e85b0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.878+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab41e6c50 con 0x7fcab41e85b0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.878+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab41e7210 con 0x7fcab41e85b0 2024-09-13T17:17:16.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.879+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab40ce2c0 con 0x7fcab41e85b0 2024-09-13T17:17:16.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.880+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcaa001d460 con 0x7fcab41e85b0 2024-09-13T17:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.881+0000 7fcab1600640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 0x7fca9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.881+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(134..134 src has 1..134) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcaa0089200 con 0x7fcab41e85b0 2024-09-13T17:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.881+0000 7fcab9800640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 0x7fca9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:16.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.882+0000 7fcab9800640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 0x7fca9406d0f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000c00 tx=0x7fcaa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:16.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:16.885+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcaa0054800 con 0x7fcab41e85b0 2024-09-13T17:17:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.075+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcab40b3810 con 0x7fcab41e85b0 2024-09-13T17:17:17.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.873+0000 7fcab1600640 1 -- 172.21.15.28:0/2317000190 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v135) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcaa0057e20 con 0x7fcab41e85b0 2024-09-13T17:17:17.874 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:17:17.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 msgr2=0x7fca9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 0x7fca9406d0f0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8000c00 tx=0x7fcaa8009040 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 msgr2=0x7fcab41e6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 0x7fcab41e6540 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fcaa0004ec0 tx=0x7fcaa0007430 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 shutdown_connections 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca9406ac30 0x7fca9406d0f0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4159a40 0x7fcab4159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcab41e85b0 0x7fcab41e6540 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 --2- 172.21.15.28:0/2317000190 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c005180 0x7fcab4154cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.875+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 >> 172.21.15.28:0/2317000190 conn(0x7fcab40b0270 msgr2=0x7fcab415ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.876+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 shutdown_connections 2024-09-13T17:17:17.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:17.876+0000 7fcabc200640 1 -- 172.21.15.28:0/2317000190 wait complete. 2024-09-13T17:17:17.894 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:17:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.052+0000 7f791c800640 1 Processor -- start 2024-09-13T17:17:18.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.052+0000 7f791c800640 1 -- start start 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f791c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 0x7f791415d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f791c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7914160310 0x7f7914160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f791c800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7914146550 con 0x7f79140ac850 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f791c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791415cd20 con 0x7f7914160310 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f791c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791415d280 con 0x7f7914146170 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f7919400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7914160310 0x7f7914160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f7919400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7914160310 0x7f7914160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54204/0 (socket says 172.21.15.28:54204) 2024-09-13T17:17:18.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.053+0000 7f7919400640 1 -- 172.21.15.28:0/4234470316 learned_addr learned my addr 172.21.15.28:0/4234470316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919e00640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 0x7f791415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919400640 1 -- 172.21.15.28:0/4234470316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 msgr2=0x7f791415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919400640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 0x7f791415d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919400640 1 -- 172.21.15.28:0/4234470316 >> v1:172.21.15.189:6789/0 conn(0x7f79140ac850 legacy=0x7f791406b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919400640 1 -- 172.21.15.28:0/4234470316 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791415dc50 con 0x7f7914160310 2024-09-13T17:17:18.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.054+0000 7f7919e00640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 0x7f791415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:18.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.055+0000 7f7919400640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914160310 0x7f7914160750 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f790402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9f93fb08da72a98 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.055+0000 7f7918a00640 1 -- 172.21.15.28:0/4234470316 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904032440 con 0x7f7914160310 2024-09-13T17:17:18.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.055+0000 7f7918a00640 1 -- 172.21.15.28:0/4234470316 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79040325a0 con 0x7f7914160310 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.055+0000 7f7918a00640 1 -- 172.21.15.28:0/4234470316 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904032930 con 0x7f7914160310 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 -- 172.21.15.28:0/4234470316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914160310 msgr2=0x7f7914160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914160310 0x7f7914160750 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f790402a250 comp rx=0 tx=0).stop 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 -- 172.21.15.28:0/4234470316 shutdown_connections 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914160310 0x7f7914160750 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 --2- 172.21.15.28:0/4234470316 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914146170 0x7f791415d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 -- 172.21.15.28:0/4234470316 >> 172.21.15.28:0/4234470316 conn(0x7f7914069e80 msgr2=0x7f791406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 -- 172.21.15.28:0/4234470316 shutdown_connections 2024-09-13T17:17:18.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.056+0000 7f791c800640 1 -- 172.21.15.28:0/4234470316 wait complete. 2024-09-13T17:17:18.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.057+0000 7f791c800640 1 Processor -- start 2024-09-13T17:17:18.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 -- start start 2024-09-13T17:17:18.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79140ac850 0x7f7914143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914160310 0x7f7914159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914156c70 con 0x7f7914146170 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7914155150 con 0x7f79140ac850 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.058+0000 7f791c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7914155430 con 0x7f7914160310 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54218/0 (socket says 172.21.15.28:54218) 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 -- 172.21.15.28:0/814369399 learned_addr learned my addr 172.21.15.28:0/814369399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919e00640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79140ac850 0x7f7914143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f791a800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914160310 0x7f7914159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 -- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914160310 msgr2=0x7f7914159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914160310 0x7f7914159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 -- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79140ac850 msgr2=0x7f7914143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79140ac850 0x7f7914143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.059+0000 7f7919400640 1 -- 172.21.15.28:0/814369399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040094d0 con 0x7f7914146170 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f7919400640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f79040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7904032440 con 0x7f7914146170 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7904030ba0 con 0x7f7914146170 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7914155710 con 0x7f7914146170 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790403b400 con 0x7f7914146170 2024-09-13T17:17:18.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.060+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7914143ab0 con 0x7f7914146170 2024-09-13T17:17:18.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.061+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79141558f0 con 0x7f7914146170 2024-09-13T17:17:18.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.062+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7904039040 con 0x7f7914146170 2024-09-13T17:17:18.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.063+0000 7f790e000640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 0x7f78f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.063+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f79040a9770 con 0x7f7914146170 2024-09-13T17:17:18.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.063+0000 7f7919e00640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 0x7f78f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.064+0000 7f7919e00640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 0x7f78f806d0c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7908006fd0 tx=0x7f7908008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.067+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f79141558f0 con 0x7f7914146170 2024-09-13T17:17:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.316+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f7914155570 con 0x7f7914146170 2024-09-13T17:17:18.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.316+0000 7f790e000640 1 -- 172.21.15.28:0/814369399 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v20) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f7914155570 con 0x7f7914146170 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:17:18.318 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.321 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.322 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:17:18.323 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.324 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.325 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.326 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:17:18.327 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:17:18.328 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:17:18.329 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:17:18.331 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.332 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:17:18.333 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:17:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:18.334 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:17:18.334 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:18.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 msgr2=0x7f78f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 0x7f78f806d0c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f7908006fd0 tx=0x7f7908008040 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 msgr2=0x7f7914154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f7904009870 tx=0x7f79040046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 shutdown_connections 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f78f806ac00 0x7f78f806d0c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7914160310 0x7f7914159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7914146170 0x7f7914154800 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 --2- 172.21.15.28:0/814369399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79140ac850 0x7f7914143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.318+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 >> 172.21.15.28:0/814369399 conn(0x7f7914069e80 msgr2=0x7f79140af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.319+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 shutdown_connections 2024-09-13T17:17:18.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.319+0000 7f791c800640 1 -- 172.21.15.28:0/814369399 wait complete. 2024-09-13T17:17:18.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.492+0000 7f268ec00640 1 Processor -- start 2024-09-13T17:17:18.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.492+0000 7f268ec00640 1 -- start start 2024-09-13T17:17:18.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.493+0000 7f268ec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268ec00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2688070cc0 0x7f2688071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268ec00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f268815e100 con 0x7f26880cabc0 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268ec00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26880716b0 con 0x7f2688070cc0 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268ec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26880adb00 con 0x7f268815a540 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268cc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2688070cc0 0x7f2688071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f2687e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268cc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2688070cc0 0x7f2688071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f268cc00640 1 -- 172.21.15.28:0/1587531438 learned_addr learned my addr 172.21.15.28:0/1587531438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.494+0000 7f2687e00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:34630/0 (socket says 172.21.15.28:34630) 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.495+0000 7f268cc00640 1 -- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 msgr2=0x7f268806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.495+0000 7f268cc00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.495+0000 7f268cc00640 1 -- 172.21.15.28:0/1587531438 >> v1:172.21.15.189:6789/0 conn(0x7f26880cabc0 legacy=0x7f268815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.495+0000 7f268cc00640 1 -- 172.21.15.28:0/1587531438 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268815e3e0 con 0x7f2688070cc0 2024-09-13T17:17:18.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.495+0000 7f2687e00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268cc00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688070cc0 0x7f2688071170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e80 tx=0x7f268002a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b2b4b4ac20e339 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f2687400640 1 -- 172.21.15.28:0/1587531438 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680031600 con 0x7f2688070cc0 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f2687400640 1 -- 172.21.15.28:0/1587531438 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268002d050 con 0x7f2688070cc0 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f2687400640 1 -- 172.21.15.28:0/1587531438 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680031980 con 0x7f2688070cc0 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 -- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688070cc0 msgr2=0x7f2688071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688070cc0 0x7f2688071170 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2680009e80 tx=0x7f268002a6b0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 -- 172.21.15.28:0/1587531438 shutdown_connections 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2688070cc0 0x7f2688071170 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 --2- 172.21.15.28:0/1587531438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f268815a540 0x7f268806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.496+0000 7f268ec00640 1 -- 172.21.15.28:0/1587531438 >> 172.21.15.28:0/1587531438 conn(0x7f2688069e80 msgr2=0x7f268806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.497+0000 7f268ec00640 1 -- 172.21.15.28:0/1587531438 shutdown_connections 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.497+0000 7f268ec00640 1 -- 172.21.15.28:0/1587531438 wait complete. 2024-09-13T17:17:18.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.498+0000 7f268ec00640 1 Processor -- start 2024-09-13T17:17:18.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.498+0000 7f268ec00640 1 -- start start 2024-09-13T17:17:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 0x7f268815d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 0x7f268814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688157ff0 con 0x7f2688070cc0 2024-09-13T17:17:18.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26881507c0 con 0x7f268815a540 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f268ec00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2688150aa0 con 0x7f26880cabc0 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f2687e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f2687e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54222/0 (socket says 172.21.15.28:54222) 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.499+0000 7f2687e00640 1 -- 172.21.15.28:0/892273359 learned_addr learned my addr 172.21.15.28:0/892273359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f267fe00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 0x7f268815d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f268cc00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 0x7f268814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 -- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 msgr2=0x7f268815d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 0x7f268815d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 -- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 msgr2=0x7f268814d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 0x7f268814d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 -- 172.21.15.28:0/892273359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2678009400 con 0x7f2688070cc0 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f267fe00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 0x7f268815d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:18.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f268cc00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 0x7f268814d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2687e00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f267800e960 tx=0x7f267800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.500+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267800ca00 con 0x7f2688070cc0 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.501+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267800cb60 con 0x7f2688070cc0 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.501+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2678013470 con 0x7f2688070cc0 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.501+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2680009b30 con 0x7f2688070cc0 2024-09-13T17:17:18.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.501+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268814ccf0 con 0x7f2688070cc0 2024-09-13T17:17:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.502+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2688071170 con 0x7f2688070cc0 2024-09-13T17:17:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.503+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f26780135d0 con 0x7f2688070cc0 2024-09-13T17:17:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.504+0000 7f2684c00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 0x7f266406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.504+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2678089260 con 0x7f2688070cc0 2024-09-13T17:17:18.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.505+0000 7f267fe00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 0x7f266406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.505+0000 7f267fe00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 0x7f266406d0c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26880ca890 tx=0x7f2670009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.509+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2678054790 con 0x7f2688070cc0 2024-09-13T17:17:18.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.690+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f268815da40 con 0x7f2688070cc0 2024-09-13T17:17:18.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.691+0000 7f2684c00640 1 -- 172.21.15.28:0/892273359 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v135) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f2678057db0 con 0x7f2688070cc0 2024-09-13T17:17:18.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:18.692 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":135,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:17.859380+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":126,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 msgr2=0x7f266406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 0x7f266406d0c0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f26880ca890 tx=0x7f2670009040 comp rx=0 tx=0).stop 2024-09-13T17:17:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 msgr2=0x7f2688157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f267800e960 tx=0x7f267800ee20 comp rx=0 tx=0).stop 2024-09-13T17:17:18.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 shutdown_connections 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f266406ac00 0x7f266406d0c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f268815a540 0x7f268814d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26880cabc0 0x7f268815d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 --2- 172.21.15.28:0/892273359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2688070cc0 0x7f2688157ab0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 >> 172.21.15.28:0/892273359 conn(0x7f2688069e80 msgr2=0x7f2688155f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 shutdown_connections 2024-09-13T17:17:18.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.693+0000 7f268ec00640 1 -- 172.21.15.28:0/892273359 wait complete. 2024-09-13T17:17:18.710 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:17:18.710 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:17:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.868+0000 7f5f8e400640 1 Processor -- start 2024-09-13T17:17:18.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.868+0000 7f5f8e400640 1 -- start start 2024-09-13T17:17:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f8e400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f88071490 0x7f5f8806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f8e400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8806c8b0 con 0x7f5f8815a780 2024-09-13T17:17:18.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f8e400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f8806cb90 con 0x7f5f8815ab60 2024-09-13T17:17:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f8e400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f880cc980 con 0x7f5f88071490 2024-09-13T17:17:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f87400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5f8815a780 0x7f5f88070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48436/0 (socket says 172.21.15.28:48436) 2024-09-13T17:17:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.869+0000 7f5f87400640 1 -- 172.21.15.28:0/3760812604 learned_addr learned my addr 172.21.15.28:0/3760812604 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:18.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f87e00640 1 --2- 172.21.15.28:0/3760812604 >> v2:172.21.15.189:3300/0 conn(0x7f5f88071490 0x7f5f8806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859830073 0 0) 0x7f5f8806c8b0 con 0x7f5f8815a780 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6c003520 con 0x7f5f8815a780 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2964488272 0 0) 0x7f5f8806cb90 con 0x7f5f8815ab60 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8806c8b0 con 0x7f5f8815ab60 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 919622036 0 0) 0x7f5f6c003520 con 0x7f5f8815a780 2024-09-13T17:17:18.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 >> v2:172.21.15.189:3300/0 conn(0x7f5f88071490 msgr2=0x7f5f8806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 --2- 172.21.15.28:0/3760812604 >> v2:172.21.15.189:3300/0 conn(0x7f5f88071490 0x7f5f8806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 >> v1:172.21.15.189:6789/0 conn(0x7f5f8815ab60 legacy=0x7f5f8806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f880ccc60 con 0x7f5f8815a780 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f5f78001670 con 0x7f5f8815a780 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 >> v1:172.21.15.28:6789/0 conn(0x7f5f8815a780 legacy=0x7f5f88070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 0x7f5f6c003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c0047c0 0x7f5f6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f6c005180 0x7f5f6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880cc980 con 0x7f5f6c003b00 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c00c000 con 0x7f5f6c0047c0 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6c00c1c0 con 0x7f5f6c005180 2024-09-13T17:17:18.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.870+0000 7f5f87e00640 1 --2- 172.21.15.28:0/3760812604 >> v2:172.21.15.189:3300/0 conn(0x7f5f88071490 0x7f5f8806c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 0x7f5f6c003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f7ea00640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c0047c0 0x7f5f6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87e00640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f6c005180 0x7f5f6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 -- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f6c005180 msgr2=0x7f5f6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f6c005180 0x7f5f6c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 -- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c0047c0 msgr2=0x7f5f6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c0047c0 0x7f5f6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87400640 1 -- 172.21.15.28:0/3760812604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6c003520 con 0x7f5f6c003b00 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.871+0000 7f5f87e00640 1 -- 172.21.15.28:0/3760812604 reap_dead start 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.872+0000 7f5f87400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 0x7f5f6c003f00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c160 tx=0x7f5f6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38984edccc62a03c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.872+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78005ad0 con 0x7f5f6c003b00 2024-09-13T17:17:18.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.872+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7802d070 con 0x7f5f6c003b00 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.872+0000 7f5f86a00640 1 -- 172.21.15.28:0/3760812604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f78007070 con 0x7f5f6c003b00 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 -- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 msgr2=0x7f5f6c003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 0x7f5f6c003f00 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f5f7800c160 tx=0x7f5f6c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 -- 172.21.15.28:0/3760812604 shutdown_connections 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 --2- 172.21.15.28:0/3760812604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f6c003b00 0x7f5f6c003f00 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 -- 172.21.15.28:0/3760812604 >> 172.21.15.28:0/3760812604 conn(0x7f5f880b0270 msgr2=0x7f5f880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 -- 172.21.15.28:0/3760812604 shutdown_connections 2024-09-13T17:17:18.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.873+0000 7f5f8e400640 1 -- 172.21.15.28:0/3760812604 wait complete. 2024-09-13T17:17:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.875+0000 7f5f8e400640 1 Processor -- start 2024-09-13T17:17:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.875+0000 7f5f8e400640 1 -- start start 2024-09-13T17:17:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 0x7f5f8814f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 0x7f5f880ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f880cea20 con 0x7f5f880ce5a0 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88151800 con 0x7f5f6c003b00 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f8e400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f88152660 con 0x7f5f880b2080 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54236/0 (socket says 172.21.15.28:54236) 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 -- 172.21.15.28:0/2873635105 learned_addr learned my addr 172.21.15.28:0/2873635105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f7ea00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 0x7f5f880ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 0x7f5f8814f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 -- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 msgr2=0x7f5f880ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 0x7f5f880ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 -- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 msgr2=0x7f5f8814f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 0x7f5f8814f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87e00640 1 -- 172.21.15.28:0/2873635105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f88151a50 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f7ea00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 0x7f5f880ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.876+0000 7f5f87400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 0x7f5f8814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f87e00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f5f8806c790 tx=0x7f5f80009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8000e070 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80011ec0 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f80003990 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f88151d30 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.877+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f881522f0 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.878+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f880cbeb0 con 0x7f5f880ce5a0 2024-09-13T17:17:18.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.879+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5f80003af0 con 0x7f5f880ce5a0 2024-09-13T17:17:18.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.880+0000 7f5f7fe00640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 0x7f5f6406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:18.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.880+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f80089940 con 0x7f5f880ce5a0 2024-09-13T17:17:18.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.880+0000 7f5f87400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 0x7f5f6406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:18.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.881+0000 7f5f87400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 0x7f5f6406d0f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5f78032000 tx=0x7f5f78007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:18.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:18.884+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5f80054dd0 con 0x7f5f880ce5a0 2024-09-13T17:17:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.063+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5f88152480 con 0x7f5f880ce5a0 2024-09-13T17:17:19.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.064+0000 7f5f7fe00640 1 -- 172.21.15.28:0/2873635105 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v135) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5f800583f0 con 0x7f5f880ce5a0 2024-09-13T17:17:19.065 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:17:19.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 msgr2=0x7f5f6406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 0x7f5f6406d0f0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5f78032000 tx=0x7f5f78007000 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 msgr2=0x7f5f8814ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f5f8806c790 tx=0x7f5f80009f90 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 shutdown_connections 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f6406ac30 0x7f5f6406d0f0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f880ce5a0 0x7f5f8814ffc0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f880b2080 0x7f5f880ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 --2- 172.21.15.28:0/2873635105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f6c003b00 0x7f5f8814f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 >> 172.21.15.28:0/2873635105 conn(0x7f5f880b0270 msgr2=0x7f5f88070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 shutdown_connections 2024-09-13T17:17:19.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.066+0000 7f5f8e400640 1 -- 172.21.15.28:0/2873635105 wait complete. 2024-09-13T17:17:19.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:17:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.237+0000 7fa65aa00640 1 Processor -- start 2024-09-13T17:17:19.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.237+0000 7fa65aa00640 1 -- start start 2024-09-13T17:17:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa65aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa654159410 0x7fa65406f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa65aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6540b38d0 0x7fa65406cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa65aa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6540ccb00 con 0x7fa65406d240 2024-09-13T17:17:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa65aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65406d6f0 con 0x7fa654159410 2024-09-13T17:17:19.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa65aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65406fc50 con 0x7fa6540b38d0 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa659600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa654159410 0x7fa65406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa659600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa654159410 0x7fa65406f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54238/0 (socket says 172.21.15.28:54238) 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.238+0000 7fa659600640 1 -- 172.21.15.28:0/3683349851 learned_addr learned my addr 172.21.15.28:0/3683349851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.239+0000 7fa658c00640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6540b38d0 0x7fa65406cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.239+0000 7fa659600640 1 -- 172.21.15.28:0/3683349851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6540b38d0 msgr2=0x7fa65406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.239+0000 7fa659600640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6540b38d0 0x7fa65406cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.239+0000 7fa659600640 1 -- 172.21.15.28:0/3683349851 >> v1:172.21.15.189:6789/0 conn(0x7fa65406d240 legacy=0x7fa6540b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.239+0000 7fa659600640 1 -- 172.21.15.28:0/3683349851 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65406ff30 con 0x7fa654159410 2024-09-13T17:17:19.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.240+0000 7fa659600640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa654159410 0x7fa65406f300 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa648009870 tx=0x7fa64802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51439c9896aea789 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.240+0000 7fa653e00640 1 -- 172.21.15.28:0/3683349851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648032440 con 0x7fa654159410 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.240+0000 7fa653e00640 1 -- 172.21.15.28:0/3683349851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6480325a0 con 0x7fa654159410 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.240+0000 7fa653e00640 1 -- 172.21.15.28:0/3683349851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa648032930 con 0x7fa654159410 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 -- 172.21.15.28:0/3683349851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa654159410 msgr2=0x7fa65406f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa654159410 0x7fa65406f300 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7fa648009870 tx=0x7fa64802a270 comp rx=0 tx=0).stop 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 -- 172.21.15.28:0/3683349851 shutdown_connections 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6540b38d0 0x7fa65406cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3683349851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa654159410 0x7fa65406f300 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 -- 172.21.15.28:0/3683349851 >> 172.21.15.28:0/3683349851 conn(0x7fa654073670 msgr2=0x7fa654069eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 -- 172.21.15.28:0/3683349851 shutdown_connections 2024-09-13T17:17:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.241+0000 7fa65aa00640 1 -- 172.21.15.28:0/3683349851 wait complete. 2024-09-13T17:17:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.243+0000 7fa65aa00640 1 Processor -- start 2024-09-13T17:17:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.243+0000 7fa65aa00640 1 -- start start 2024-09-13T17:17:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65406d240 0x7fa6540acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa654159410 0x7fa654153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa654153550 con 0x7fa6540b38d0 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6541f22b0 con 0x7fa65406d240 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6541f30d0 con 0x7fa654159410 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54252/0 (socket says 172.21.15.28:54252) 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 -- 172.21.15.28:0/3506079260 learned_addr learned my addr 172.21.15.28:0/3506079260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa659600640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65406d240 0x7fa6540acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa65a000640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa654159410 0x7fa654153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 -- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa654159410 msgr2=0x7fa654153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa654159410 0x7fa654153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 -- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65406d240 msgr2=0x7fa6540acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65406d240 0x7fa6540acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.244+0000 7fa658c00640 1 -- 172.21.15.28:0/3506079260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480094d0 con 0x7fa6540b38d0 2024-09-13T17:17:19.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa658c00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa640009fd0 tx=0x7fa64000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000c940 con 0x7fa6540b38d0 2024-09-13T17:17:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000caa0 con 0x7fa6540b38d0 2024-09-13T17:17:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000ce30 con 0x7fa6540b38d0 2024-09-13T17:17:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6541f24c0 con 0x7fa6540b38d0 2024-09-13T17:17:19.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.245+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6541f2d60 con 0x7fa6540b38d0 2024-09-13T17:17:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.246+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65406cd00 con 0x7fa6540b38d0 2024-09-13T17:17:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.247+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa640013470 con 0x7fa6540b38d0 2024-09-13T17:17:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.248+0000 7fa651600640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 0x7fa63406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.248+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa6400891e0 con 0x7fa6540b38d0 2024-09-13T17:17:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.248+0000 7fa659600640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 0x7fa63406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.249+0000 7fa659600640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 0x7fa63406d0c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa648034000 tx=0x7fa64802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.252+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa640054720 con 0x7fa6540b38d0 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.432+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa654158550 con 0x7fa6540b38d0 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.433+0000 7fa651600640 1 -- 172.21.15.28:0/3506079260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v135) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa640057d40 con 0x7fa6540b38d0 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:19.434 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:17:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 msgr2=0x7fa63406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 0x7fa63406d0c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa648034000 tx=0x7fa64802e000 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 msgr2=0x7fa654152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7fa640009fd0 tx=0x7fa64000ed40 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 shutdown_connections 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa63406ac00 0x7fa63406d0c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa654159410 0x7fa654153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6540b38d0 0x7fa654152ad0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 --2- 172.21.15.28:0/3506079260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65406d240 0x7fa6540acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 >> 172.21.15.28:0/3506079260 conn(0x7fa654073670 msgr2=0x7fa6540adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 shutdown_connections 2024-09-13T17:17:19.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.435+0000 7fa65aa00640 1 -- 172.21.15.28:0/3506079260 wait complete. 2024-09-13T17:17:19.451 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:17:19.451 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:17:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.611+0000 7f0da1c00640 1 Processor -- start 2024-09-13T17:17:19.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.611+0000 7f0da1c00640 1 -- start start 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0da1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0da1c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d9c15cb60 0x7f0d9c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0da1c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c15a5e0 con 0x7f0d9c15a200 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0da1c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9c069e80 con 0x7f0d9c15cb60 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0da1c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c06a720 con 0x7f0d9c071b40 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0d9be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0d9c15a200 0x7f0d9c071430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48438/0 (socket says 172.21.15.28:48438) 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.612+0000 7f0d9be00640 1 -- 172.21.15.28:0/4247477774 learned_addr learned my addr 172.21.15.28:0/4247477774 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:19.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0da1200640 1 --2- 172.21.15.28:0/4247477774 >> v2:172.21.15.189:3300/0 conn(0x7f0d9c15cb60 0x7f0d9c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d93400640 1 --2- 172.21.15.28:0/4247477774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d9b400640 1 -- 172.21.15.28:0/4247477774 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544531724 0 0) 0x7f0d8c001110 con 0x7f0d9c15a200 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d9b400640 1 -- 172.21.15.28:0/4247477774 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d80003520 con 0x7f0d9c15a200 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d93400640 1 -- 172.21.15.28:0/4247477774 >> v2:172.21.15.189:3300/0 conn(0x7f0d9c15cb60 msgr2=0x7f0d9c15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d93400640 1 --2- 172.21.15.28:0/4247477774 >> v2:172.21.15.189:3300/0 conn(0x7f0d9c15cb60 0x7f0d9c15cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d93400640 1 -- 172.21.15.28:0/4247477774 >> v1:172.21.15.28:6789/0 conn(0x7f0d9c15a200 legacy=0x7f0d9c071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.613+0000 7f0d93400640 1 -- 172.21.15.28:0/4247477774 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c15d940 con 0x7f0d9c071b40 2024-09-13T17:17:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.614+0000 7f0d93400640 1 --2- 172.21.15.28:0/4247477774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c06abb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d840097a0 tx=0x7f0d8402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f095f5c0c81e95b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0d9b400640 1 -- 172.21.15.28:0/4247477774 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8402ae50 con 0x7f0d9c071b40 2024-09-13T17:17:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0d9b400640 1 -- 172.21.15.28:0/4247477774 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84032440 con 0x7f0d9c071b40 2024-09-13T17:17:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0d9b400640 1 -- 172.21.15.28:0/4247477774 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840327d0 con 0x7f0d9c071b40 2024-09-13T17:17:19.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 -- 172.21.15.28:0/4247477774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 msgr2=0x7f0d9c06abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4247477774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c06abb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f0d840097a0 tx=0x7f0d8402a100 comp rx=0 tx=0).stop 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 -- 172.21.15.28:0/4247477774 shutdown_connections 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4247477774 >> v2:172.21.15.189:3300/0 conn(0x7f0d9c15cb60 0x7f0d9c15cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4247477774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c06abb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 -- 172.21.15.28:0/4247477774 >> 172.21.15.28:0/4247477774 conn(0x7f0d9c0b2190 msgr2=0x7f0d9c0b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 -- 172.21.15.28:0/4247477774 shutdown_connections 2024-09-13T17:17:19.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.615+0000 7f0da1c00640 1 -- 172.21.15.28:0/4247477774 wait complete. 2024-09-13T17:17:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.617+0000 7f0da1c00640 1 Processor -- start 2024-09-13T17:17:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.617+0000 7f0da1c00640 1 -- start start 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c15cb60 0x7f0d9c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c151740 con 0x7f0d9c15a200 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c0c9ce0 con 0x7f0d9c15cb60 2024-09-13T17:17:19.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0da1c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c1f64e0 con 0x7f0d9c071b40 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.618+0000 7f0d93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54254/0 (socket says 172.21.15.28:54254) 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 -- 172.21.15.28:0/4116922084 learned_addr learned my addr 172.21.15.28:0/4116922084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0da1200640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c15cb60 0x7f0d9c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d9be00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 -- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 msgr2=0x7f0d9c151200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c151200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 -- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c15cb60 msgr2=0x7f0d9c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c15cb60 0x7f0d9c15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 -- 172.21.15.28:0/4116922084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84009400 con 0x7f0d9c15a200 2024-09-13T17:17:19.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.619+0000 7f0d93400640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0d84035040 tx=0x7f0d840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.620+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8403d710 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.620+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8403d890 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.620+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8403dc40 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.620+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c0c9f30 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.620+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c0ca4f0 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.621+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c06abb0 con 0x7f0d9c15a200 2024-09-13T17:17:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.622+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d84039040 con 0x7f0d9c15a200 2024-09-13T17:17:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.622+0000 7f0d98c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 0x7f0d7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.622+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0d840a97a0 con 0x7f0d9c15a200 2024-09-13T17:17:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.623+0000 7f0d9be00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 0x7f0d7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.624+0000 7f0d9be00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 0x7f0d7806d0c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c007fd0 tx=0x7f0d8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:19.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.626+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d84074e60 con 0x7f0d9c15a200 2024-09-13T17:17:19.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:19.811+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0d9c0c9ac0 con 0x7f0d9c15a200 2024-09-13T17:17:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.524+0000 7f0d98c00640 1 -- 172.21.15.28:0/4116922084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v136) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0d84078480 con 0x7f0d9c15a200 2024-09-13T17:17:20.525 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 msgr2=0x7f0d7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 0x7f0d7806d0c0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c007fd0 tx=0x7f0d8c009040 comp rx=0 tx=0).stop 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 msgr2=0x7f0d9c1f5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f0d84035040 tx=0x7f0d840046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 shutdown_connections 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d7806ac00 0x7f0d7806d0c0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d9c15cb60 0x7f0d9c15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d9c15a200 0x7f0d9c1f5fa0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 --2- 172.21.15.28:0/4116922084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d9c071b40 0x7f0d9c151200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.530+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 >> 172.21.15.28:0/4116922084 conn(0x7f0d9c0b2190 msgr2=0x7f0d9c0b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.531+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 shutdown_connections 2024-09-13T17:17:20.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.531+0000 7f0da1c00640 1 -- 172.21.15.28:0/4116922084 wait complete. 2024-09-13T17:17:20.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:17:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.718+0000 7fc6c8c00640 1 Processor -- start 2024-09-13T17:17:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.718+0000 7fc6c8c00640 1 -- start start 2024-09-13T17:17:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6c4159770 0x7fc6c406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c406f3f0 0x7fc6c4070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6c40add00 0x7fc6c4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c406f920 con 0x7fc6c4159770 2024-09-13T17:17:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6c415b290 con 0x7fc6c40add00 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c415b570 con 0x7fc6c406f3f0 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6c4159770 0x7fc6c406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6c4159770 0x7fc6c406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54264/0 (socket says 172.21.15.28:54264) 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.719+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1812790721 learned_addr learned my addr 172.21.15.28:0/1812790721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1400640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c406f3f0 0x7fc6c4070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c2800640 1 --2- 172.21.15.28:0/1812790721 >> v2:172.21.15.189:3300/0 conn(0x7fc6c40add00 0x7fc6c4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1812790721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c406f3f0 msgr2=0x7fc6c4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c406f3f0 0x7fc6c4070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1812790721 >> v2:172.21.15.189:3300/0 conn(0x7fc6c40add00 msgr2=0x7fc6c4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1812790721 >> v2:172.21.15.189:3300/0 conn(0x7fc6c40add00 0x7fc6c4159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.720+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1812790721 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6c415b850 con 0x7fc6c4159770 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.721+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c4159770 0x7fc6c406fd20 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40b0e90 tx=0x7fc6b402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945cc7955cf59571 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.721+0000 7fc6c0a00640 1 -- 172.21.15.28:0/1812790721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b402ae90 con 0x7fc6c4159770 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.721+0000 7fc6c0a00640 1 -- 172.21.15.28:0/1812790721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b4032440 con 0x7fc6c4159770 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.721+0000 7fc6c0a00640 1 -- 172.21.15.28:0/1812790721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b40327d0 con 0x7fc6c4159770 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1812790721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c4159770 msgr2=0x7fc6c406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c4159770 0x7fc6c406fd20 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc6c40b0e90 tx=0x7fc6b402a140 comp rx=0 tx=0).stop 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1812790721 shutdown_connections 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1812790721 >> v2:172.21.15.189:3300/0 conn(0x7fc6c40add00 0x7fc6c4159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c406f3f0 0x7fc6c4070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1812790721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c4159770 0x7fc6c406fd20 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1812790721 >> 172.21.15.28:0/1812790721 conn(0x7fc6c40b0270 msgr2=0x7fc6c40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1812790721 shutdown_connections 2024-09-13T17:17:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.722+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1812790721 wait complete. 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 Processor -- start 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 -- start start 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c40add00 0x7fc6c4155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c4159770 0x7fc6c4155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.724+0000 7fc6c8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c4146a50 con 0x7fc6c406f3f0 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c41e3f60 con 0x7fc6c40add00 2024-09-13T17:17:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6c41e4240 con 0x7fc6c4159770 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54272/0 (socket says 172.21.15.28:54272) 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1512287526 learned_addr learned my addr 172.21.15.28:0/1512287526 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1400640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c40add00 0x7fc6c4155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c4159770 msgr2=0x7fc6c4155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c4159770 0x7fc6c4155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c40add00 msgr2=0x7fc6c4155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c40add00 0x7fc6c4155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.725+0000 7fc6c1e00640 1 -- 172.21.15.28:0/1512287526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b40094d0 con 0x7fc6c406f3f0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6c1e00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4000c00 tx=0x7fc6b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4030850 con 0x7fc6c406f3f0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b40309b0 con 0x7fc6c406f3f0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6c4146cc0 con 0x7fc6c406f3f0 2024-09-13T17:17:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4030d40 con 0x7fc6c406f3f0 2024-09-13T17:17:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.726+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6c4146370 con 0x7fc6c406f3f0 2024-09-13T17:17:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.727+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6c4146f00 con 0x7fc6c406f3f0 2024-09-13T17:17:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.728+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6b4039040 con 0x7fc6c406f3f0 2024-09-13T17:17:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.729+0000 7fc6ba000640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 0x7fc6a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.729+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc6b40a9d70 con 0x7fc6c406f3f0 2024-09-13T17:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.730+0000 7fc6c1400640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 0x7fc6a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.730+0000 7fc6c1400640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 0x7fc6a406d0c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc6c41e4e50 tx=0x7fc6b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.734+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6c4146f00 con 0x7fc6c406f3f0 2024-09-13T17:17:20.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:20.913+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc6c40ce2e0 con 0x7fc6c406f3f0 2024-09-13T17:17:21.533 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:17:21.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.532+0000 7fc6ba000640 1 -- 172.21.15.28:0/1512287526 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v137) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc6b404b0d0 con 0x7fc6c406f3f0 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 msgr2=0x7fc6a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 0x7fc6a406d0c0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fc6c41e4e50 tx=0x7fc6b0008040 comp rx=0 tx=0).stop 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 msgr2=0x7fc6c4145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4000c00 tx=0x7fc6b40046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 shutdown_connections 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6a406ac00 0x7fc6a406d0c0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6c4159770 0x7fc6c4155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6c40add00 0x7fc6c4155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 --2- 172.21.15.28:0/1512287526 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6c406f3f0 0x7fc6c4145e30 secure :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4000c00 tx=0x7fc6b40046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.537+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 >> 172.21.15.28:0/1512287526 conn(0x7fc6c40b0270 msgr2=0x7fc6c40af7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:21.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.538+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 shutdown_connections 2024-09-13T17:17:21.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.539+0000 7fc6c8c00640 1 -- 172.21.15.28:0/1512287526 wait complete. 2024-09-13T17:17:21.571 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:17:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.731+0000 7f82b4e00640 1 Processor -- start 2024-09-13T17:17:21.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.731+0000 7f82b4e00640 1 -- start start 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82b0143f70 0x7f82b00cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 0x7f82b00c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b00c3ba0 0x7f82b006f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b00cda80 con 0x7f82b0143f70 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b015ef70 con 0x7f82b00cd5f0 2024-09-13T17:17:21.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82b4e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b00c4320 con 0x7f82b00c3ba0 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82b0143f70 0x7f82b00cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82af400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f82b0143f70 0x7f82b00cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54274/0 (socket says 172.21.15.28:54274) 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.732+0000 7f82af400640 1 -- 172.21.15.28:0/1212434195 learned_addr learned my addr 172.21.15.28:0/1212434195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82a6a00640 1 --2- 172.21.15.28:0/1212434195 >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 0x7f82b00c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82afe00640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b00c3ba0 0x7f82b006f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82af400640 1 -- 172.21.15.28:0/1212434195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b00c3ba0 msgr2=0x7f82b006f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:21.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82af400640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b00c3ba0 0x7f82b006f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82af400640 1 -- 172.21.15.28:0/1212434195 >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 msgr2=0x7f82b00c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82af400640 1 --2- 172.21.15.28:0/1212434195 >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 0x7f82b00c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82af400640 1 -- 172.21.15.28:0/1212434195 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b015e1b0 con 0x7f82b0143f70 2024-09-13T17:17:21.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.733+0000 7f82a6a00640 1 --2- 172.21.15.28:0/1212434195 >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 0x7f82b00c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.734+0000 7f82af400640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b0143f70 0x7f82b00cd0b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f82b006f1b0 tx=0x7f82a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a127e83e8163d010 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.734+0000 7f82aea00640 1 -- 172.21.15.28:0/1212434195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0032440 con 0x7f82b0143f70 2024-09-13T17:17:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.734+0000 7f82aea00640 1 -- 172.21.15.28:0/1212434195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a00325a0 con 0x7f82b0143f70 2024-09-13T17:17:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.734+0000 7f82aea00640 1 -- 172.21.15.28:0/1212434195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a0032930 con 0x7f82b0143f70 2024-09-13T17:17:21.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 -- 172.21.15.28:0/1212434195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b0143f70 msgr2=0x7f82b00cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b0143f70 0x7f82b00cd0b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f82b006f1b0 tx=0x7f82a002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 -- 172.21.15.28:0/1212434195 shutdown_connections 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b00c3ba0 0x7f82b006f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 --2- 172.21.15.28:0/1212434195 >> v2:172.21.15.189:3300/0 conn(0x7f82b00cd5f0 0x7f82b00c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 --2- 172.21.15.28:0/1212434195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b0143f70 0x7f82b00cd0b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 -- 172.21.15.28:0/1212434195 >> 172.21.15.28:0/1212434195 conn(0x7f82b0069e80 msgr2=0x7f82b006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 -- 172.21.15.28:0/1212434195 shutdown_connections 2024-09-13T17:17:21.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.735+0000 7f82b4e00640 1 -- 172.21.15.28:0/1212434195 wait complete. 2024-09-13T17:17:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.737+0000 7f82b4e00640 1 Processor -- start 2024-09-13T17:17:21.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.737+0000 7f82b4e00640 1 -- start start 2024-09-13T17:17:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82b00c3ba0 0x7f82b014eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b0143f70 0x7f82b015aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b015aff0 con 0x7f82b00cd5f0 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b014ea10 con 0x7f82b00c3ba0 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82b4e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b014d970 con 0x7f82b0143f70 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82a6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54282/0 (socket says 172.21.15.28:54282) 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82a6a00640 1 -- 172.21.15.28:0/2481322768 learned_addr learned my addr 172.21.15.28:0/2481322768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82afe00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b0143f70 0x7f82b015aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.738+0000 7f82af400640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82b00c3ba0 0x7f82b014eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 -- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b0143f70 msgr2=0x7f82b015aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b0143f70 0x7f82b015aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 -- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82b00c3ba0 msgr2=0x7f82b014eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82b00c3ba0 0x7f82b014eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 -- 172.21.15.28:0/2481322768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a00094d0 con 0x7f82b00cd5f0 2024-09-13T17:17:21.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a6a00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f829800d7d0 tx=0x7f829800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:21.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298004030 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8298004190 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8298004520 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.739+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b014dc50 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.740+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b014e210 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.740+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b00b3470 con 0x7f82b00cd5f0 2024-09-13T17:17:21.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.741+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f829800b6c0 con 0x7f82b00cd5f0 2024-09-13T17:17:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.742+0000 7f82a7e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 0x7f828c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.742+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f82980893b0 con 0x7f82b00cd5f0 2024-09-13T17:17:21.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.743+0000 7f82af400640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 0x7f828c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:21.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.744+0000 7f82af400640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 0x7f828c06d0c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f82a0034000 tx=0x7f82a002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:21.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.747+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8298054600 con 0x7f82b00cd5f0 2024-09-13T17:17:21.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.962+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f82b00c3660 con 0x7f82b00cd5f0 2024-09-13T17:17:21.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.969+0000 7f82a7e00640 1 -- 172.21.15.28:0/2481322768 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v118) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8298057c20 con 0x7f82b00cd5f0 2024-09-13T17:17:21.972 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 18 and data pool 19 2024-09-13T17:17:21.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.974+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 msgr2=0x7f828c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.974+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 0x7f828c06d0c0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f82a0034000 tx=0x7f82a002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.974+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 msgr2=0x7f82b015a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:21.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.974+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f829800d7d0 tx=0x7f829800dc90 comp rx=0 tx=0).stop 2024-09-13T17:17:21.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 shutdown_connections 2024-09-13T17:17:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f828c06ac00 0x7f828c06d0c0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f82b0143f70 0x7f82b015aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f82b00cd5f0 0x7f82b015a570 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 --2- 172.21.15.28:0/2481322768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82b00c3ba0 0x7f82b014eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:21.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.976+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 >> 172.21.15.28:0/2481322768 conn(0x7f82b0069e80 msgr2=0x7f82b00cceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.980+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 shutdown_connections 2024-09-13T17:17:21.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:21.980+0000 7f82b4e00640 1 -- 172.21.15.28:0/2481322768 wait complete. 2024-09-13T17:17:22.008 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.162+0000 7f9b41a00640 1 Processor -- start 2024-09-13T17:17:22.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.162+0000 7f9b41a00640 1 -- start start 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b41a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b3c1592a0 0x7f9b3c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b41a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 0x7f9b3c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b41a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0b2040 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b41a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3c15b0b0 con 0x7f9b3c1592a0 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b41a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c159ff0 con 0x7f9b3c06f970 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.163+0000 7f9b3aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b3c1592a0 0x7f9b3c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b3c1592a0 0x7f9b3c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54290/0 (socket says 172.21.15.28:54290) 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 -- 172.21.15.28:0/3506542615 learned_addr learned my addr 172.21.15.28:0/3506542615 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3a000640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 0x7f9b3c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 -- 172.21.15.28:0/3506542615 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 msgr2=0x7f9b3c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 0x7f9b3c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 -- 172.21.15.28:0/3506542615 >> v1:172.21.15.189:6789/0 conn(0x7f9b3c06c2d0 legacy=0x7f9b3c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3aa00640 1 -- 172.21.15.28:0/3506542615 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c15a2d0 con 0x7f9b3c1592a0 2024-09-13T17:17:22.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.164+0000 7f9b3a000640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 0x7f9b3c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.165+0000 7f9b3aa00640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c1592a0 0x7f9b3c15ab70 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6501a4590c04a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.165+0000 7f9b39600640 1 -- 172.21.15.28:0/3506542615 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c032440 con 0x7f9b3c1592a0 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.165+0000 7f9b39600640 1 -- 172.21.15.28:0/3506542615 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0325a0 con 0x7f9b3c1592a0 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.165+0000 7f9b39600640 1 -- 172.21.15.28:0/3506542615 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c032930 con 0x7f9b3c1592a0 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 -- 172.21.15.28:0/3506542615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c1592a0 msgr2=0x7f9b3c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c1592a0 0x7f9b3c15ab70 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c02a250 comp rx=0 tx=0).stop 2024-09-13T17:17:22.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 -- 172.21.15.28:0/3506542615 shutdown_connections 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c06f970 0x7f9b3c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 --2- 172.21.15.28:0/3506542615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c1592a0 0x7f9b3c15ab70 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 -- 172.21.15.28:0/3506542615 >> 172.21.15.28:0/3506542615 conn(0x7f9b3c069e80 msgr2=0x7f9b3c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 -- 172.21.15.28:0/3506542615 shutdown_connections 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.166+0000 7f9b41a00640 1 -- 172.21.15.28:0/3506542615 wait complete. 2024-09-13T17:17:22.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.168+0000 7f9b41a00640 1 Processor -- start 2024-09-13T17:17:22.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.168+0000 7f9b41a00640 1 -- start start 2024-09-13T17:17:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 0x7f9b3c151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c1592a0 0x7f9b3c13b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c152250 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0cbc00 con 0x7f9b3c06f970 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b41a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0cbee0 con 0x7f9b3c1592a0 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54300/0 (socket says 172.21.15.28:54300) 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 -- 172.21.15.28:0/2905216329 learned_addr learned my addr 172.21.15.28:0/2905216329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3a000640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 0x7f9b3c151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3b400640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c1592a0 0x7f9b3c13b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 -- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c1592a0 msgr2=0x7f9b3c13b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c1592a0 0x7f9b3c13b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.169+0000 7f9b3aa00640 1 -- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 msgr2=0x7f9b3c151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b3aa00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 0x7f9b3c151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b3aa00640 1 -- 172.21.15.28:0/2905216329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0094d0 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b3a000640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 0x7f9b3c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b3aa00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c032440 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c030ba0 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c0cc1c0 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c03b400 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.170+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c0cc780 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.171+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c15ab70 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.172+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9b2c039040 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.173+0000 7f9b32a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 0x7f9b1c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.173+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b2c0a9d30 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.174+0000 7f9b3a000640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 0x7f9b1c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.175+0000 7f9b3a000640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 0x7f9b1c06d0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c14c8f0 tx=0x7f9b28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.177+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9b2c074de0 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.357+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3c0b0b10 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.358+0000 7f9b32a00640 1 -- 172.21.15.28:0/2905216329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f9b2c037020 con 0x7f9b3c06c2d0 2024-09-13T17:17:22.359 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:22.359 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:21.963694+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:22.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 msgr2=0x7f9b1c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 0x7f9b1c06d0c0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c14c8f0 tx=0x7f9b28008040 comp rx=0 tx=0).stop 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 msgr2=0x7f9b3c0cb5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009870 tx=0x7f9b2c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 shutdown_connections 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b1c06ac00 0x7f9b1c06d0c0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b3c1592a0 0x7f9b3c13b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b3c06f970 0x7f9b3c151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 --2- 172.21.15.28:0/2905216329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b3c06c2d0 0x7f9b3c0cb5b0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 >> 172.21.15.28:0/2905216329 conn(0x7f9b3c069e80 msgr2=0x7f9b3c0ad050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 shutdown_connections 2024-09-13T17:17:22.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.360+0000 7f9b41a00640 1 -- 172.21.15.28:0/2905216329 wait complete. 2024-09-13T17:17:22.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.534+0000 7f786be00640 1 Processor -- start 2024-09-13T17:17:22.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.534+0000 7f786be00640 1 -- start start 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f786c0b48d0 0x7f786c167350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0b5870 con 0x7f786c0cd710 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786be00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786c0d2560 con 0x7f786c15c830 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786c15cc10 con 0x7f786c0b48d0 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f786c0cd710 0x7f786c0b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48452/0 (socket says 172.21.15.28:48452) 2024-09-13T17:17:22.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786a000640 1 -- 172.21.15.28:0/3016235154 learned_addr learned my addr 172.21.15.28:0/3016235154 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.535+0000 7f786b400640 1 --2- 172.21.15.28:0/3016235154 >> v2:172.21.15.28:3300/0 conn(0x7f786c0b48d0 0x7f786c167350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.536+0000 7f7869600640 1 -- 172.21.15.28:0/3016235154 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231459472 0 0) 0x7f786c0b5870 con 0x7f786c0cd710 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.536+0000 7f7869600640 1 -- 172.21.15.28:0/3016235154 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7850003520 con 0x7f786c0cd710 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.536+0000 7f786b400640 1 -- 172.21.15.28:0/3016235154 >> v1:172.21.15.189:6789/0 conn(0x7f786c15c830 legacy=0x7f786c0d14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.536+0000 7f786b400640 1 -- 172.21.15.28:0/3016235154 >> v1:172.21.15.28:6789/0 conn(0x7f786c0cd710 legacy=0x7f786c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.536+0000 7f786b400640 1 -- 172.21.15.28:0/3016235154 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0d1cc0 con 0x7f786c0b48d0 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.537+0000 7f786b400640 1 --2- 172.21.15.28:0/3016235154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c167350 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7864009d90 tx=0x7f786402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c511926f7506bd5d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.537+0000 7f7869600640 1 -- 172.21.15.28:0/3016235154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864007740 con 0x7f786c0b48d0 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.537+0000 7f7869600640 1 -- 172.21.15.28:0/3016235154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786402d050 con 0x7f786c0b48d0 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.537+0000 7f7869600640 1 -- 172.21.15.28:0/3016235154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7864007ac0 con 0x7f786c0b48d0 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 -- 172.21.15.28:0/3016235154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 msgr2=0x7f786c167350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 --2- 172.21.15.28:0/3016235154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c167350 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f7864009d90 tx=0x7f786402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 -- 172.21.15.28:0/3016235154 shutdown_connections 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 --2- 172.21.15.28:0/3016235154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c167350 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 -- 172.21.15.28:0/3016235154 >> 172.21.15.28:0/3016235154 conn(0x7f786c0cf3a0 msgr2=0x7f786c0cf7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 -- 172.21.15.28:0/3016235154 shutdown_connections 2024-09-13T17:17:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.538+0000 7f786be00640 1 -- 172.21.15.28:0/3016235154 wait complete. 2024-09-13T17:17:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.539+0000 7f786be00640 1 Processor -- start 2024-09-13T17:17:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.540+0000 7f786be00640 1 -- start start 2024-09-13T17:17:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.540+0000 7f786be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.540+0000 7f786be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c0cd710 0x7f786c14b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.540+0000 7f786be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f786c15c830 0x7f786c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.540+0000 7f786be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c14ec50 con 0x7f786c0b48d0 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786c0c91d0 con 0x7f786c0cd710 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786c14e0d0 con 0x7f786c15c830 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54320/0 (socket says 172.21.15.28:54320) 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 -- 172.21.15.28:0/187560586 learned_addr learned my addr 172.21.15.28:0/187560586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 -- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f786c15c830 msgr2=0x7f786c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f786c15c830 0x7f786c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 -- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c0cd710 msgr2=0x7f786c14b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c0cd710 0x7f786c14b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 -- 172.21.15.28:0/187560586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7864009a70 con 0x7f786c0b48d0 2024-09-13T17:17:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.541+0000 7f786aa00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c400 tx=0x7f785c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.542+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c0052d0 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.542+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c005430 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.542+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c003830 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.542+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f786c14e3b0 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.542+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786c14e970 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.543+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f786c0b41c0 con 0x7f786c0b48d0 2024-09-13T17:17:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.544+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f785c003990 con 0x7f786c0b48d0 2024-09-13T17:17:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.545+0000 7f7862a00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 0x7f784c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.545+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f785c088900 con 0x7f786c0b48d0 2024-09-13T17:17:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.545+0000 7f786a000640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 0x7f784c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.546+0000 7f786a000640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 0x7f784c06d0c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7858001380 tx=0x7f78580077c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.549+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f785c053a40 con 0x7f786c0b48d0 2024-09-13T17:17:22.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.731+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f786c0a93c0 con 0x7f786c0b48d0 2024-09-13T17:17:22.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.732+0000 7f7862a00640 1 -- 172.21.15.28:0/187560586 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v138) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f785c017090 con 0x7f786c0b48d0 2024-09-13T17:17:22.733 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:22.733 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:21.963694+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:22.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 msgr2=0x7f784c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 0x7f784c06d0c0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7858001380 tx=0x7f78580077c0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 msgr2=0x7f786c0c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f785c00c400 tx=0x7f785c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 shutdown_connections 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f784c06ac00 0x7f784c06d0c0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f786c15c830 0x7f786c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f786c0cd710 0x7f786c14b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 --2- 172.21.15.28:0/187560586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f786c0b48d0 0x7f786c0c8c90 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.734+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 >> 172.21.15.28:0/187560586 conn(0x7f786c0cf3a0 msgr2=0x7f786c0cf780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.735+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 shutdown_connections 2024-09-13T17:17:22.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.735+0000 7f786be00640 1 -- 172.21.15.28:0/187560586 wait complete. 2024-09-13T17:17:22.752 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:17:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.909+0000 7f1117c00640 1 Processor -- start 2024-09-13T17:17:22.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.909+0000 7f1117c00640 1 -- start start 2024-09-13T17:17:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.910+0000 7f1117c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f1110070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.910+0000 7f1117c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f111015ab60 0x7f11100712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.910+0000 7f1117c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f111006c8b0 con 0x7f111006bb40 2024-09-13T17:17:22.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.910+0000 7f1117c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111006c6a0 con 0x7f111015ab60 2024-09-13T17:17:22.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.910+0000 7f1117c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111006cb90 con 0x7f111015a780 2024-09-13T17:17:22.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f1115c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f111006bb40 0x7f111006bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:48464/0 (socket says 172.21.15.28:48464) 2024-09-13T17:17:22.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f1115c00640 1 -- 172.21.15.28:0/3335036219 learned_addr learned my addr 172.21.15.28:0/3335036219 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f1107e00640 1 --2- 172.21.15.28:0/3335036219 >> v2:172.21.15.189:3300/0 conn(0x7f111015ab60 0x7f11100712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f1115200640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f1110070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070451120 0 0) 0x7f111006c8b0 con 0x7f111006bb40 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10f4003520 con 0x7f111006bb40 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1679736288 0 0) 0x7f10f4003520 con 0x7f111006bb40 2024-09-13T17:17:22.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 msgr2=0x7f1110070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f1110070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 >> v2:172.21.15.189:3300/0 conn(0x7f111015ab60 msgr2=0x7f11100712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 --2- 172.21.15.28:0/3335036219 >> v2:172.21.15.189:3300/0 conn(0x7f111015ab60 0x7f11100712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.911+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100cc980 con 0x7f111006bb40 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f11080019f0 con 0x7f111006bb40 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 >> v1:172.21.15.28:6789/0 conn(0x7f111006bb40 legacy=0x7f111006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f10f4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f10f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f4004d00 0x7f10f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111006c6a0 con 0x7f10f4003f40 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111006cb90 con 0x7f10f4004d00 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f400b980 con 0x7f111015a780 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f10f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1115200640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f10f4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1115c00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f4004d00 0x7f10f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 -- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 msgr2=0x7f10f4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f111015a780 0x7f10f4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 -- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f4004d00 msgr2=0x7f10f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10f4004d00 0x7f10f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.912+0000 7f1107e00640 1 -- 172.21.15.28:0/3335036219 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f4003520 con 0x7f10f4003f40 2024-09-13T17:17:22.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.913+0000 7f1107e00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f10f40047c0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009ac0 tx=0x7f10f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddefe0a3e773c04 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8031400 con 0x7f10f4003f40 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f802d050 con 0x7f10f4003f40 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f110fe00640 1 -- 172.21.15.28:0/3335036219 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8031780 con 0x7f10f4003f40 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 -- 172.21.15.28:0/3335036219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 msgr2=0x7f10f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 --2- 172.21.15.28:0/3335036219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f10f40047c0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009ac0 tx=0x7f10f40057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1115c00640 1 -- 172.21.15.28:0/3335036219 reap_dead start 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 -- 172.21.15.28:0/3335036219 shutdown_connections 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 -- 172.21.15.28:0/3335036219 >> 172.21.15.28:0/3335036219 conn(0x7f11100b0270 msgr2=0x7f11100b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 -- 172.21.15.28:0/3335036219 shutdown_connections 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.914+0000 7f1117c00640 1 -- 172.21.15.28:0/3335036219 wait complete. 2024-09-13T17:17:22.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.916+0000 7f1117c00640 1 Processor -- start 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.916+0000 7f1117c00640 1 -- start start 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11100c6d50 0x7f11100c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f11101f3520 0x7f11101f39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1110151200 con 0x7f10f4003f40 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1110158890 con 0x7f11100c6d50 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1117c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100c5a30 con 0x7f11101f3520 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1115200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1115200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54334/0 (socket says 172.21.15.28:54334) 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.917+0000 7f1115200640 1 -- 172.21.15.28:0/3069536115 learned_addr learned my addr 172.21.15.28:0/3069536115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f11101f3520 0x7f11101f39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1107e00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11100c6d50 0x7f11100c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 -- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f11101f3520 msgr2=0x7f11101f39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f11101f3520 0x7f11101f39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 -- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11100c6d50 msgr2=0x7f11100c54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11100c6d50 0x7f11100c54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 -- 172.21.15.28:0/3069536115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100c5cb0 con 0x7f10f4003f40 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1115200640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1100011a00 tx=0x7f1100011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110000fb50 con 0x7f10f4003f40 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110000fcb0 con 0x7f10f4003f40 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.918+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11100c5f90 con 0x7f10f4003f40 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.919+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1100014440 con 0x7f10f4003f40 2024-09-13T17:17:22.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.919+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11100c6550 con 0x7f10f4003f40 2024-09-13T17:17:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.920+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111006c8b0 con 0x7f10f4003f40 2024-09-13T17:17:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.921+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f11000145a0 con 0x7f10f4003f40 2024-09-13T17:17:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.922+0000 7f110d600640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:22.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.922+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1100089c30 con 0x7f10f4003f40 2024-09-13T17:17:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.922+0000 7f1107e00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:22.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.923+0000 7f1107e00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 0x7f10ec06d0c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f10f8033000 tx=0x7f10f8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:22.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:22.926+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1100054d70 con 0x7f10f4003f40 2024-09-13T17:17:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.146+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1110070d50 con 0x7f10f4003f40 2024-09-13T17:17:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.983+0000 7f110d600640 1 -- 172.21.15.28:0/3069536115 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v121) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1100058390 con 0x7f10f4003f40 2024-09-13T17:17:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 msgr2=0x7f10ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:23.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 0x7f10ec06d0c0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f10f8033000 tx=0x7f10f8008030 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 msgr2=0x7f1110158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f1100011a00 tx=0x7f1100011ec0 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 shutdown_connections 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f11101f3520 0x7f11101f39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11100c6d50 0x7f11100c54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 --2- 172.21.15.28:0/3069536115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10f4003f40 0x7f1110158350 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.986+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 >> 172.21.15.28:0/3069536115 conn(0x7f11100b0270 msgr2=0x7f11100b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.987+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 shutdown_connections 2024-09-13T17:17:23.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:23.987+0000 7f1117c00640 1 -- 172.21.15.28:0/3069536115 wait complete. 2024-09-13T17:17:24.005 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:17:24.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:17:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.167+0000 7f9f3fe00640 1 Processor -- start 2024-09-13T17:17:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.167+0000 7f9f3fe00640 1 -- start start 2024-09-13T17:17:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.168+0000 7f9f3fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f3815c9a0 0x7f9f380ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.168+0000 7f9f3fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 0x7f9f3806bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.168+0000 7f9f3fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f38070470 con 0x7f9f380ad940 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.168+0000 7f9f3fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f380ace30 con 0x7f9f3815c9a0 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.168+0000 7f9f3fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f380ad110 con 0x7f9f380b2400 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f3815c9a0 0x7f9f380ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9f380ad940 0x7f9f380b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58442/0 (socket says 172.21.15.28:58442) 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3ca00640 1 -- 172.21.15.28:0/2041570561 learned_addr learned my addr 172.21.15.28:0/2041570561 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3de00640 1 --2- 172.21.15.28:0/2041570561 >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 0x7f9f3806bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3d400640 1 -- 172.21.15.28:0/2041570561 >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 msgr2=0x7f9f3806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3d400640 1 --2- 172.21.15.28:0/2041570561 >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 0x7f9f3806bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3d400640 1 -- 172.21.15.28:0/2041570561 >> v1:172.21.15.28:6789/0 conn(0x7f9f380ad940 legacy=0x7f9f380b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f37e00640 1 -- 172.21.15.28:0/2041570561 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751862246 0 0) 0x7f9f38070470 con 0x7f9f380ad940 2024-09-13T17:17:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3d400640 1 -- 172.21.15.28:0/2041570561 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3806b170 con 0x7f9f3815c9a0 2024-09-13T17:17:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.169+0000 7f9f3de00640 1 --2- 172.21.15.28:0/2041570561 >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 0x7f9f3806bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.170+0000 7f9f3d400640 1 --2- 172.21.15.28:0/2041570561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f3815c9a0 0x7f9f380ac850 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0097a0 tx=0x7f9f2c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54421bf6af6cd76d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f37e00640 1 -- 172.21.15.28:0/2041570561 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c032440 con 0x7f9f3815c9a0 2024-09-13T17:17:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f37e00640 1 -- 172.21.15.28:0/2041570561 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c0325a0 con 0x7f9f3815c9a0 2024-09-13T17:17:24.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f37e00640 1 -- 172.21.15.28:0/2041570561 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c032930 con 0x7f9f3815c9a0 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 -- 172.21.15.28:0/2041570561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f3815c9a0 msgr2=0x7f9f380ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/2041570561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f3815c9a0 0x7f9f380ac850 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0097a0 tx=0x7f9f2c02a270 comp rx=0 tx=0).stop 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 -- 172.21.15.28:0/2041570561 shutdown_connections 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/2041570561 >> v2:172.21.15.189:3300/0 conn(0x7f9f380b2400 0x7f9f3806bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/2041570561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f3815c9a0 0x7f9f380ac850 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 -- 172.21.15.28:0/2041570561 >> 172.21.15.28:0/2041570561 conn(0x7f9f380b0270 msgr2=0x7f9f380b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 -- 172.21.15.28:0/2041570561 shutdown_connections 2024-09-13T17:17:24.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.171+0000 7f9f3fe00640 1 -- 172.21.15.28:0/2041570561 wait complete. 2024-09-13T17:17:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.173+0000 7f9f3fe00640 1 Processor -- start 2024-09-13T17:17:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.173+0000 7f9f3fe00640 1 -- start start 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f380b2400 0x7f9f38143650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f3815c9a0 0x7f9f380c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38144da0 con 0x7f9f380ad940 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f381547b0 con 0x7f9f380b2400 2024-09-13T17:17:24.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f38146e20 con 0x7f9f3815c9a0 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45564/0 (socket says 172.21.15.28:45564) 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 -- 172.21.15.28:0/465233424 learned_addr learned my addr 172.21.15.28:0/465233424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 -- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f3815c9a0 msgr2=0x7f9f380c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f3815c9a0 0x7f9f380c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 -- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f380b2400 msgr2=0x7f9f38143650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f380b2400 0x7f9f38143650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.174+0000 7f9f3d400640 1 -- 172.21.15.28:0/465233424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f2c009400 con 0x7f9f380ad940 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f3d400640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0098c0 tx=0x7f9f2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c032440 con 0x7f9f380ad940 2024-09-13T17:17:24.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2c030b80 con 0x7f9f380ad940 2024-09-13T17:17:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f2c006e10 con 0x7f9f380ad940 2024-09-13T17:17:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f38070930 con 0x7f9f380ad940 2024-09-13T17:17:24.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.175+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f38144400 con 0x7f9f380ad940 2024-09-13T17:17:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.176+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f38144ee0 con 0x7f9f380ad940 2024-09-13T17:17:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.177+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9f2c039040 con 0x7f9f380ad940 2024-09-13T17:17:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.178+0000 7f9f35600640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 0x7f9f1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.178+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f2c0a9db0 con 0x7f9f380ad940 2024-09-13T17:17:24.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.178+0000 7f9f3ca00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 0x7f9f1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:24.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.179+0000 7f9f3ca00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 0x7f9f1806d0c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f24007fd0 tx=0x7f9f24007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:24.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.182+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9f38144ee0 con 0x7f9f380ad940 2024-09-13T17:17:24.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.394+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9f38144ee0 con 0x7f9f380ad940 2024-09-13T17:17:24.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.986+0000 7f9f35600640 1 -- 172.21.15.28:0/465233424 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v122) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9f38144ee0 con 0x7f9f380ad940 2024-09-13T17:17:24.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 msgr2=0x7f9f1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 0x7f9f1806d0c0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f9f24007fd0 tx=0x7f9f24007920 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 msgr2=0x7f9f38154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f9f2c0098c0 tx=0x7f9f2c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 shutdown_connections 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f1806ac00 0x7f9f1806d0c0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f3815c9a0 0x7f9f380c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f380b2400 0x7f9f38143650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 --2- 172.21.15.28:0/465233424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f380ad940 0x7f9f38154270 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.989+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 >> 172.21.15.28:0/465233424 conn(0x7f9f380b0270 msgr2=0x7f9f380ae690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.990+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 shutdown_connections 2024-09-13T17:17:24.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:24.990+0000 7f9f3fe00640 1 -- 172.21.15.28:0/465233424 wait complete. 2024-09-13T17:17:25.010 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:17:25.010 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.169+0000 7f01a1200640 1 Processor -- start 2024-09-13T17:17:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.169+0000 7f01a1200640 1 -- start start 2024-09-13T17:17:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.170+0000 7f01a1200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f019c15a540 0x7f019c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.170+0000 7f01a1200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f019c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.170+0000 7f01a1200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019c15e100 con 0x7f019c15edb0 2024-09-13T17:17:25.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f01a1200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f019c0adb00 con 0x7f019c15a540 2024-09-13T17:17:25.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f01a1200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15e3e0 con 0x7f019c0cabc0 2024-09-13T17:17:25.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f019a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f019c15edb0 0x7f019c070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58452/0 (socket says 172.21.15.28:58452) 2024-09-13T17:17:25.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f019a800640 1 -- 172.21.15.28:0/2880116569 learned_addr learned my addr 172.21.15.28:0/2880116569 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f0199e00640 1 --2- 172.21.15.28:0/2880116569 >> v2:172.21.15.189:3300/0 conn(0x7f019c15a540 0x7f019c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f0191400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f019c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3717878172 0 0) 0x7f019c15e100 con 0x7f019c15edb0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.171+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0180003520 con 0x7f019c15edb0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3006911556 0 0) 0x7f0180003520 con 0x7f019c15edb0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 msgr2=0x7f019c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f019c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 >> v2:172.21.15.189:3300/0 conn(0x7f019c15a540 msgr2=0x7f019c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 --2- 172.21.15.28:0/2880116569 >> v2:172.21.15.189:3300/0 conn(0x7f019c15a540 0x7f019c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c06ce80 con 0x7f019c15edb0 2024-09-13T17:17:25.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f01940019f0 con 0x7f019c15edb0 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 >> v1:172.21.15.28:6789/0 conn(0x7f019c15edb0 legacy=0x7f019c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0180003970 0x7f0180003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f01800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0180004d00 0x7f0180005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c15e3e0 con 0x7f0180004d00 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018000fd90 con 0x7f0180003970 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018000fed0 con 0x7f019c0cabc0 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0199e00640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0180003970 0x7f0180003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f019a800640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0180004d00 0x7f0180005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.172+0000 7f0191400640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f01800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.173+0000 7f0199e00640 1 -- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 msgr2=0x7f01800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.173+0000 7f0199e00640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f019c0cabc0 0x7f01800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.173+0000 7f0199e00640 1 -- 172.21.15.28:0/2880116569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0180004d00 msgr2=0x7f0180005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.173+0000 7f0199e00640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0180004d00 0x7f0180005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.173+0000 7f0199e00640 1 -- 172.21.15.28:0/2880116569 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0180003520 con 0x7f0180003970 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.174+0000 7f0199e00640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0180003970 0x7f0180003d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f018c009ac0 tx=0x7f01800057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ad29d52476370fc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.174+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c031400 con 0x7f0180003970 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.174+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018c02d050 con 0x7f0180003970 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.174+0000 7f0199400640 1 -- 172.21.15.28:0/2880116569 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018c031780 con 0x7f0180003970 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 -- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0180003970 msgr2=0x7f0180003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 --2- 172.21.15.28:0/2880116569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0180003970 0x7f0180003d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f018c009ac0 tx=0x7f01800057f0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f019a800640 1 -- 172.21.15.28:0/2880116569 reap_dead start 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 -- 172.21.15.28:0/2880116569 shutdown_connections 2024-09-13T17:17:25.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 -- 172.21.15.28:0/2880116569 >> 172.21.15.28:0/2880116569 conn(0x7f019c069e80 msgr2=0x7f019c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 -- 172.21.15.28:0/2880116569 shutdown_connections 2024-09-13T17:17:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.175+0000 7f01a1200640 1 -- 172.21.15.28:0/2880116569 wait complete. 2024-09-13T17:17:25.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.176+0000 7f01a1200640 1 Processor -- start 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 -- start start 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0180003970 0x7f019c0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c1525d0 0x7f019c0ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0aeed0 con 0x7f019c152150 2024-09-13T17:17:25.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0af1b0 con 0x7f019c1525d0 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.177+0000 7f01a1200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019c0af490 con 0x7f0180003970 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45588/0 (socket says 172.21.15.28:45588) 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 -- 172.21.15.28:0/165468252 learned_addr learned my addr 172.21.15.28:0/165468252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0199e00640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0180003970 0x7f019c0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f019a800640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c1525d0 0x7f019c0ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 -- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0180003970 msgr2=0x7f019c0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0180003970 0x7f019c0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 -- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c1525d0 msgr2=0x7f019c0ae7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c1525d0 0x7f019c0ae7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 -- 172.21.15.28:0/165468252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019c0af770 con 0x7f019c152150 2024-09-13T17:17:25.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.178+0000 7f0191400640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0184011910 tx=0x7f0184011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.179+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0184009590 con 0x7f019c152150 2024-09-13T17:17:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.179+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f018400fa30 con 0x7f019c152150 2024-09-13T17:17:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.179+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f018400fda0 con 0x7f019c152150 2024-09-13T17:17:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.179+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019c0afa50 con 0x7f019c152150 2024-09-13T17:17:25.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.179+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019c14a680 con 0x7f019c152150 2024-09-13T17:17:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.180+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019c0afbe0 con 0x7f019c152150 2024-09-13T17:17:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.181+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0184014440 con 0x7f019c152150 2024-09-13T17:17:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.182+0000 7f0192a00640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 0x7f017806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.182+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0184089a70 con 0x7f019c152150 2024-09-13T17:17:25.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.182+0000 7f0199e00640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 0x7f017806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.183+0000 7f0199e00640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 0x7f017806d0c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f018c033000 tx=0x7f018c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.186+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0184054ce0 con 0x7f019c152150 2024-09-13T17:17:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.401+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f019c15e870 con 0x7f019c152150 2024-09-13T17:17:25.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.402+0000 7f0192a00640 1 -- 172.21.15.28:0/165468252 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f0184058300 con 0x7f019c152150 2024-09-13T17:17:25.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:25.403 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:25.403 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 msgr2=0x7f017806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 0x7f017806d0c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f018c033000 tx=0x7f018c008030 comp rx=0 tx=0).stop 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 msgr2=0x7f019c0ae280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f0184011910 tx=0x7f0184011d10 comp rx=0 tx=0).stop 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 shutdown_connections 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f017806ac00 0x7f017806d0c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f019c1525d0 0x7f019c0ae7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f019c152150 0x7f019c0ae280 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 --2- 172.21.15.28:0/165468252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0180003970 0x7f019c0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 >> 172.21.15.28:0/165468252 conn(0x7f019c069e80 msgr2=0x7f019c152e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 shutdown_connections 2024-09-13T17:17:25.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.404+0000 7f01a1200640 1 -- 172.21.15.28:0/165468252 wait complete. 2024-09-13T17:17:25.422 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:17:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.576+0000 7fd75f400640 1 Processor -- start 2024-09-13T17:17:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.576+0000 7fd75f400640 1 -- start start 2024-09-13T17:17:25.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7581592a0 0x7fd75815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806f970 0x7fd758159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75f400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7580b2040 con 0x7fd75806c2d0 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75815b0b0 con 0x7fd7581592a0 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd758159ff0 con 0x7fd75806f970 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7581592a0 0x7fd75815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.577+0000 7fd75ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd7581592a0 0x7fd75815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45604/0 (socket says 172.21.15.28:45604) 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd75ca00640 1 -- 172.21.15.28:0/2943182032 learned_addr learned my addr 172.21.15.28:0/2943182032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd74fe00640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806f970 0x7fd758159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd75ca00640 1 -- 172.21.15.28:0/2943182032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806f970 msgr2=0x7fd758159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd75ca00640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806f970 0x7fd758159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd75ca00640 1 -- 172.21.15.28:0/2943182032 >> v1:172.21.15.189:6789/0 conn(0x7fd75806c2d0 legacy=0x7fd7580b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.578+0000 7fd75ca00640 1 -- 172.21.15.28:0/2943182032 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75815a2d0 con 0x7fd7581592a0 2024-09-13T17:17:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.579+0000 7fd75ca00640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7581592a0 0x7fd75815ab70 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fd74800bb10 tx=0x7fd74802c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1e5cb4a839674 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.579+0000 7fd757e00640 1 -- 172.21.15.28:0/2943182032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748032790 con 0x7fd7581592a0 2024-09-13T17:17:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.579+0000 7fd757e00640 1 -- 172.21.15.28:0/2943182032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480328f0 con 0x7fd7581592a0 2024-09-13T17:17:25.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.579+0000 7fd757e00640 1 -- 172.21.15.28:0/2943182032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748032c80 con 0x7fd7581592a0 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 -- 172.21.15.28:0/2943182032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7581592a0 msgr2=0x7fd75815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7581592a0 0x7fd75815ab70 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7fd74800bb10 tx=0x7fd74802c8d0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 -- 172.21.15.28:0/2943182032 shutdown_connections 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806f970 0x7fd758159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 --2- 172.21.15.28:0/2943182032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7581592a0 0x7fd75815ab70 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 -- 172.21.15.28:0/2943182032 >> 172.21.15.28:0/2943182032 conn(0x7fd758069e80 msgr2=0x7fd75806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 -- 172.21.15.28:0/2943182032 shutdown_connections 2024-09-13T17:17:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.580+0000 7fd75f400640 1 -- 172.21.15.28:0/2943182032 wait complete. 2024-09-13T17:17:25.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.581+0000 7fd75f400640 1 Processor -- start 2024-09-13T17:17:25.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.582+0000 7fd75f400640 1 -- start start 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.582+0000 7fd75f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806c2d0 0x7fd7580aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.582+0000 7fd75f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.582+0000 7fd75f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 0x7fd7580a8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.582+0000 7fd75f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7581eec70 con 0x7fd75806f970 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd75f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580708e0 con 0x7fd7581592a0 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd75f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7581ee0f0 con 0x7fd75806c2d0 2024-09-13T17:17:25.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45616/0 (socket says 172.21.15.28:45616) 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 -- 172.21.15.28:0/2989506758 learned_addr learned my addr 172.21.15.28:0/2989506758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd75ca00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806c2d0 0x7fd7580aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd75d400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 0x7fd7580a8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 -- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806c2d0 msgr2=0x7fd7580aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806c2d0 0x7fd7580aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 -- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 msgr2=0x7fd7580a8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 0x7fd7580a8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd74fe00640 1 -- 172.21.15.28:0/2989506758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74800b770 con 0x7fd75806f970 2024-09-13T17:17:25.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.583+0000 7fd75d400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 0x7fd7580a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd74fe00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fd74000d800 tx=0x7fd74000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74000b6c0 con 0x7fd75806f970 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74000b820 con 0x7fd75806f970 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74000bbb0 con 0x7fd75806f970 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7581ee3d0 con 0x7fd75806f970 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.584+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7581ee990 con 0x7fd75806f970 2024-09-13T17:17:25.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.585+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75815ab70 con 0x7fd75806f970 2024-09-13T17:17:25.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.586+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd74000bd10 con 0x7fd75806f970 2024-09-13T17:17:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.587+0000 7fd755600640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 0x7fd73406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.587+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd740089800 con 0x7fd75806f970 2024-09-13T17:17:25.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.588+0000 7fd75ca00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 0x7fd73406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.588+0000 7fd75ca00640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 0x7fd73406d0f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd748037000 tx=0x7fd74803e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.592+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd7400549d0 con 0x7fd75806f970 2024-09-13T17:17:25.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.771+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7580aef90 con 0x7fd75806f970 2024-09-13T17:17:25.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.772+0000 7fd755600640 1 -- 172.21.15.28:0/2989506758 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16579 (secure 0 0 0) 0x7fd740057ff0 con 0x7fd75806f970 2024-09-13T17:17:25.774 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:25.774 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":139,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:22.966365+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:17:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 msgr2=0x7fd73406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 0x7fd73406d0f0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd748037000 tx=0x7fd74803e040 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 msgr2=0x7fd7580aabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7fd74000d800 tx=0x7fd74000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 shutdown_connections 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73406ac30 0x7fd73406d0f0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7581592a0 0x7fd7580a8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd75806f970 0x7fd7580aabc0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 --2- 172.21.15.28:0/2989506758 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806c2d0 0x7fd7580aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.774+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 >> 172.21.15.28:0/2989506758 conn(0x7fd758069e80 msgr2=0x7fd7580adf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.775+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 shutdown_connections 2024-09-13T17:17:25.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.775+0000 7fd75f400640 1 -- 172.21.15.28:0/2989506758 wait complete. 2024-09-13T17:17:25.792 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:17:25.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:17:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.946+0000 7f72d3600640 1 Processor -- start 2024-09-13T17:17:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.946+0000 7f72d3600640 1 -- start start 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc160310 0x7f72cc160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d3600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72cc0ac850 0x7f72cc0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d3600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72cc146550 con 0x7f72cc146170 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d3600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72cc0ad240 con 0x7f72cc0ac850 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc15cd20 con 0x7f72cc160310 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f72cc0ac850 0x7f72cc0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d0c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72cc146170 0x7f72cc15d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58466/0 (socket says 172.21.15.28:58466) 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72d0c00640 1 -- 172.21.15.28:0/1356433317 learned_addr learned my addr 172.21.15.28:0/1356433317 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72c3e00640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc160310 0x7f72cc160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.948+0000 7f72cbe00640 1 -- 172.21.15.28:0/1356433317 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874652117 0 0) 0x7f72cc146550 con 0x7f72cc146170 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.949+0000 7f72cbe00640 1 -- 172.21.15.28:0/1356433317 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72b0003520 con 0x7f72cc146170 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.949+0000 7f72d1600640 1 -- 172.21.15.28:0/1356433317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc160310 msgr2=0x7f72cc160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.949+0000 7f72d1600640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc160310 0x7f72cc160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.949+0000 7f72d1600640 1 -- 172.21.15.28:0/1356433317 >> v1:172.21.15.28:6789/0 conn(0x7f72cc146170 legacy=0x7f72cc15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.949+0000 7f72d1600640 1 -- 172.21.15.28:0/1356433317 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc15d280 con 0x7f72cc0ac850 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72d1600640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc0acd00 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009d90 tx=0x7f72c402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed2d248ea8dde06 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72cbe00640 1 -- 172.21.15.28:0/1356433317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4007740 con 0x7f72cc0ac850 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72cbe00640 1 -- 172.21.15.28:0/1356433317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c402d050 con 0x7f72cc0ac850 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72cbe00640 1 -- 172.21.15.28:0/1356433317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c4007ac0 con 0x7f72cc0ac850 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72d3600640 1 -- 172.21.15.28:0/1356433317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 msgr2=0x7f72cc0acd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72d3600640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc0acd00 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f72c4009d90 tx=0x7f72c402a660 comp rx=0 tx=0).stop 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72d3600640 1 -- 172.21.15.28:0/1356433317 shutdown_connections 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.950+0000 7f72d3600640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc0acd00 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.951+0000 7f72d3600640 1 --2- 172.21.15.28:0/1356433317 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc160310 0x7f72cc160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.951+0000 7f72d3600640 1 -- 172.21.15.28:0/1356433317 >> 172.21.15.28:0/1356433317 conn(0x7f72cc069e80 msgr2=0x7f72cc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.951+0000 7f72d3600640 1 -- 172.21.15.28:0/1356433317 shutdown_connections 2024-09-13T17:17:25.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.951+0000 7f72d3600640 1 -- 172.21.15.28:0/1356433317 wait complete. 2024-09-13T17:17:25.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.952+0000 7f72d3600640 1 Processor -- start 2024-09-13T17:17:25.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.952+0000 7f72d3600640 1 -- start start 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc146170 0x7f72cc13ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc160310 0x7f72cc15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc1579a0 con 0x7f72cc0ac850 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc145530 con 0x7f72cc160310 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d3600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc13b340 con 0x7f72cc146170 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.953+0000 7f72d0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 -- 172.21.15.28:0/1658409973 learned_addr learned my addr 172.21.15.28:0/1658409973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72c3e00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc146170 0x7f72cc13ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d1600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc160310 0x7f72cc15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 -- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc146170 msgr2=0x7f72cc13ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc146170 0x7f72cc13ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 -- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc160310 msgr2=0x7f72cc15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc160310 0x7f72cc15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 -- 172.21.15.28:0/1658409973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c4009a70 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.954+0000 7f72d0c00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00c500 tx=0x7f72bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.955+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc005ce0 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.955+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72bc005e40 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.955+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc007330 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.955+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc156440 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.955+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc157ef0 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.956+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc145cb0 con 0x7f72cc0ac850 2024-09-13T17:17:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.957+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72bc007550 con 0x7f72cc0ac850 2024-09-13T17:17:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.957+0000 7f72c9600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 0x7f72a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:25.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.958+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72bc0889c0 con 0x7f72cc0ac850 2024-09-13T17:17:25.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.958+0000 7f72c3e00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 0x7f72a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:25.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.959+0000 7f72c3e00640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 0x7f72a806d0c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f72cc157890 tx=0x7f72b4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:25.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:25.961+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f72bc053d00 con 0x7f72cc0ac850 2024-09-13T17:17:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.139+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72cc06c6a0 con 0x7f72a806ac00 2024-09-13T17:17:26.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.147+0000 7f72c9600640 1 -- 172.21.15.28:0/1658409973 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289092 (secure 0 0 0) 0x7f72cc06c6a0 con 0x7f72a806ac00 2024-09-13T17:17:26.155 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:26.157 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 msgr2=0x7f72a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 0x7f72a806d0c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f72cc157890 tx=0x7f72b4009190 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 msgr2=0x7f72cc155db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00c500 tx=0x7f72bc00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 shutdown_connections 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72a806ac00 0x7f72a806d0c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72cc160310 0x7f72cc15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72cc146170 0x7f72cc13ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 --2- 172.21.15.28:0/1658409973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72cc0ac850 0x7f72cc155db0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 >> 172.21.15.28:0/1658409973 conn(0x7f72cc069e80 msgr2=0x7f72cc0af4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 shutdown_connections 2024-09-13T17:17:26.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.151+0000 7f72d3600640 1 -- 172.21.15.28:0/1658409973 wait complete. 2024-09-13T17:17:26.168 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-13T17:17:25.025657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":520984,"kb_used_data":14656,"kb_used_omap":12,"kb_used_meta":506163,"kb_avail":754453736,"statfs":{"total":773094113280,"available":772560625664,"internally_reserved":0,"allocated":15007744,"data_stored":7225158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":518311481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.501692"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998994+0000","last_change":"2024-09-13T17:17:21.995206+0000","last_active":"2024-09-13T17:17:21.998994+0000","last_peered":"2024-09-13T17:17:21.998994+0000","last_clean":"2024-09-13T17:17:21.998994+0000","last_became_active":"2024-09-13T17:17:21.992692+0000","last_became_peered":"2024-09-13T17:17:21.992692+0000","last_unstale":"2024-09-13T17:17:21.998994+0000","last_undegraded":"2024-09-13T17:17:21.998994+0000","last_fullsized":"2024-09-13T17:17:21.998994+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:16.168759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986353+0000","last_change":"2024-09-13T17:17:21.980448+0000","last_active":"2024-09-13T17:17:21.986353+0000","last_peered":"2024-09-13T17:17:21.986353+0000","last_clean":"2024-09-13T17:17:21.986353+0000","last_became_active":"2024-09-13T17:17:21.979916+0000","last_became_peered":"2024-09-13T17:17:21.979916+0000","last_unstale":"2024-09-13T17:17:21.986353+0000","last_undegraded":"2024-09-13T17:17:21.986353+0000","last_fullsized":"2024-09-13T17:17:21.986353+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:29.054388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993373+0000","last_change":"2024-09-13T17:17:21.989525+0000","last_active":"2024-09-13T17:17:21.993373+0000","last_peered":"2024-09-13T17:17:21.993373+0000","last_clean":"2024-09-13T17:17:21.993373+0000","last_became_active":"2024-09-13T17:17:21.988738+0000","last_became_peered":"2024-09-13T17:17:21.988738+0000","last_unstale":"2024-09-13T17:17:21.993373+0000","last_undegraded":"2024-09-13T17:17:21.993373+0000","last_fullsized":"2024-09-13T17:17:21.993373+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003316+0000","last_change":"2024-09-13T17:17:22.002101+0000","last_active":"2024-09-13T17:17:22.003316+0000","last_peered":"2024-09-13T17:17:22.003316+0000","last_clean":"2024-09-13T17:17:22.003316+0000","last_became_active":"2024-09-13T17:17:22.001760+0000","last_became_peered":"2024-09-13T17:17:22.001760+0000","last_unstale":"2024-09-13T17:17:22.003316+0000","last_undegraded":"2024-09-13T17:17:22.003316+0000","last_fullsized":"2024-09-13T17:17:22.003316+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:03.588912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998814+0000","last_change":"2024-09-13T17:17:21.991237+0000","last_active":"2024-09-13T17:17:21.998814+0000","last_peered":"2024-09-13T17:17:21.998814+0000","last_clean":"2024-09-13T17:17:21.998814+0000","last_became_active":"2024-09-13T17:17:21.990966+0000","last_became_peered":"2024-09-13T17:17:21.990966+0000","last_unstale":"2024-09-13T17:17:21.998814+0000","last_undegraded":"2024-09-13T17:17:21.998814+0000","last_fullsized":"2024-09-13T17:17:21.998814+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:38.877759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.994825+0000","last_change":"2024-09-13T17:17:21.993207+0000","last_active":"2024-09-13T17:17:21.994825+0000","last_peered":"2024-09-13T17:17:21.994825+0000","last_clean":"2024-09-13T17:17:21.994825+0000","last_became_active":"2024-09-13T17:17:21.992913+0000","last_became_peered":"2024-09-13T17:17:21.992913+0000","last_unstale":"2024-09-13T17:17:21.994825+0000","last_undegraded":"2024-09-13T17:17:21.994825+0000","last_fullsized":"2024-09-13T17:17:21.994825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:08.071748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.988935+0000","last_change":"2024-09-13T17:17:21.985479+0000","last_active":"2024-09-13T17:17:21.988935+0000","last_peered":"2024-09-13T17:17:21.988935+0000","last_clean":"2024-09-13T17:17:21.988935+0000","last_became_active":"2024-09-13T17:17:21.984330+0000","last_became_peered":"2024-09-13T17:17:21.984330+0000","last_unstale":"2024-09-13T17:17:21.988935+0000","last_undegraded":"2024-09-13T17:17:21.988935+0000","last_fullsized":"2024-09-13T17:17:21.988935+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:30.344049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999313+0000","last_change":"2024-09-13T17:17:21.992490+0000","last_active":"2024-09-13T17:17:21.999313+0000","last_peered":"2024-09-13T17:17:21.999313+0000","last_clean":"2024-09-13T17:17:21.999313+0000","last_became_active":"2024-09-13T17:17:21.984503+0000","last_became_peered":"2024-09-13T17:17:21.984503+0000","last_unstale":"2024-09-13T17:17:21.999313+0000","last_undegraded":"2024-09-13T17:17:21.999313+0000","last_fullsized":"2024-09-13T17:17:21.999313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:21.939438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989127+0000","last_change":"2024-09-13T17:17:21.982123+0000","last_active":"2024-09-13T17:17:21.989127+0000","last_peered":"2024-09-13T17:17:21.989127+0000","last_clean":"2024-09-13T17:17:21.989127+0000","last_became_active":"2024-09-13T17:17:21.981844+0000","last_became_peered":"2024-09-13T17:17:21.981844+0000","last_unstale":"2024-09-13T17:17:21.989127+0000","last_undegraded":"2024-09-13T17:17:21.989127+0000","last_fullsized":"2024-09-13T17:17:21.989127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.260215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986335+0000","last_change":"2024-09-13T17:17:21.980215+0000","last_active":"2024-09-13T17:17:21.986335+0000","last_peered":"2024-09-13T17:17:21.986335+0000","last_clean":"2024-09-13T17:17:21.986335+0000","last_became_active":"2024-09-13T17:17:21.979596+0000","last_became_peered":"2024-09-13T17:17:21.979596+0000","last_unstale":"2024-09-13T17:17:21.986335+0000","last_undegraded":"2024-09-13T17:17:21.986335+0000","last_fullsized":"2024-09-13T17:17:21.986335+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990415+0000","last_change":"2024-09-13T17:17:21.981341+0000","last_active":"2024-09-13T17:17:21.990415+0000","last_peered":"2024-09-13T17:17:21.990415+0000","last_clean":"2024-09-13T17:17:21.990415+0000","last_became_active":"2024-09-13T17:17:21.980804+0000","last_became_peered":"2024-09-13T17:17:21.980804+0000","last_unstale":"2024-09-13T17:17:21.990415+0000","last_undegraded":"2024-09-13T17:17:21.990415+0000","last_fullsized":"2024-09-13T17:17:21.990415+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:15:05.926821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999072+0000","last_change":"2024-09-13T17:17:21.992131+0000","last_active":"2024-09-13T17:17:21.999072+0000","last_peered":"2024-09-13T17:17:21.999072+0000","last_clean":"2024-09-13T17:17:21.999072+0000","last_became_active":"2024-09-13T17:17:21.991875+0000","last_became_peered":"2024-09-13T17:17:21.991875+0000","last_unstale":"2024-09-13T17:17:21.999072+0000","last_undegraded":"2024-09-13T17:17:21.999072+0000","last_fullsized":"2024-09-13T17:17:21.999072+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:29.054388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995122+0000","last_change":"2024-09-13T17:17:21.990558+0000","last_active":"2024-09-13T17:17:21.995122+0000","last_peered":"2024-09-13T17:17:21.995122+0000","last_clean":"2024-09-13T17:17:21.995122+0000","last_became_active":"2024-09-13T17:17:21.990182+0000","last_became_peered":"2024-09-13T17:17:21.990182+0000","last_unstale":"2024-09-13T17:17:21.995122+0000","last_undegraded":"2024-09-13T17:17:21.995122+0000","last_fullsized":"2024-09-13T17:17:21.995122+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:09.727162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.992960+0000","last_change":"2024-09-13T17:17:21.991486+0000","last_active":"2024-09-13T17:17:21.992960+0000","last_peered":"2024-09-13T17:17:21.992960+0000","last_clean":"2024-09-13T17:17:21.992960+0000","last_became_active":"2024-09-13T17:17:21.989663+0000","last_became_peered":"2024-09-13T17:17:21.989663+0000","last_unstale":"2024-09-13T17:17:21.992960+0000","last_undegraded":"2024-09-13T17:17:21.992960+0000","last_fullsized":"2024-09-13T17:17:21.992960+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:28.080818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995497+0000","last_change":"2024-09-13T17:17:21.992219+0000","last_active":"2024-09-13T17:17:21.995497+0000","last_peered":"2024-09-13T17:17:21.995497+0000","last_clean":"2024-09-13T17:17:21.995497+0000","last_became_active":"2024-09-13T17:17:21.992000+0000","last_became_peered":"2024-09-13T17:17:21.992000+0000","last_unstale":"2024-09-13T17:17:21.995497+0000","last_undegraded":"2024-09-13T17:17:21.995497+0000","last_fullsized":"2024-09-13T17:17:21.995497+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:25.395189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996927+0000","last_change":"2024-09-13T17:17:21.990853+0000","last_active":"2024-09-13T17:17:21.996927+0000","last_peered":"2024-09-13T17:17:21.996927+0000","last_clean":"2024-09-13T17:17:21.996927+0000","last_became_active":"2024-09-13T17:17:21.990625+0000","last_became_peered":"2024-09-13T17:17:21.990625+0000","last_unstale":"2024-09-13T17:17:21.996927+0000","last_undegraded":"2024-09-13T17:17:21.996927+0000","last_fullsized":"2024-09-13T17:17:21.996927+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:09.718296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979551+0000","last_change":"2024-09-13T17:17:21.978654+0000","last_active":"2024-09-13T17:17:21.979551+0000","last_peered":"2024-09-13T17:17:21.979551+0000","last_clean":"2024-09-13T17:17:21.979551+0000","last_became_active":"2024-09-13T17:17:21.976431+0000","last_became_peered":"2024-09-13T17:17:21.976431+0000","last_unstale":"2024-09-13T17:17:21.979551+0000","last_undegraded":"2024-09-13T17:17:21.979551+0000","last_fullsized":"2024-09-13T17:17:21.979551+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:40.508854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986571+0000","last_change":"2024-09-13T17:17:21.984409+0000","last_active":"2024-09-13T17:17:21.986571+0000","last_peered":"2024-09-13T17:17:21.986571+0000","last_clean":"2024-09-13T17:17:21.986571+0000","last_became_active":"2024-09-13T17:17:21.984156+0000","last_became_peered":"2024-09-13T17:17:21.984156+0000","last_unstale":"2024-09-13T17:17:21.986571+0000","last_undegraded":"2024-09-13T17:17:21.986571+0000","last_fullsized":"2024-09-13T17:17:21.986571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:54.197007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.991645+0000","last_change":"2024-09-13T17:17:21.981523+0000","last_active":"2024-09-13T17:17:21.991645+0000","last_peered":"2024-09-13T17:17:21.991645+0000","last_clean":"2024-09-13T17:17:21.991645+0000","last_became_active":"2024-09-13T17:17:21.981112+0000","last_became_peered":"2024-09-13T17:17:21.981112+0000","last_unstale":"2024-09-13T17:17:21.991645+0000","last_undegraded":"2024-09-13T17:17:21.991645+0000","last_fullsized":"2024-09-13T17:17:21.991645+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:56.819736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993701+0000","last_change":"2024-09-13T17:17:21.989435+0000","last_active":"2024-09-13T17:17:21.993701+0000","last_peered":"2024-09-13T17:17:21.993701+0000","last_clean":"2024-09-13T17:17:21.993701+0000","last_became_active":"2024-09-13T17:17:21.986372+0000","last_became_peered":"2024-09-13T17:17:21.986372+0000","last_unstale":"2024-09-13T17:17:21.993701+0000","last_undegraded":"2024-09-13T17:17:21.993701+0000","last_fullsized":"2024-09-13T17:17:21.993701+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:00.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999599+0000","last_change":"2024-09-13T17:17:21.995595+0000","last_active":"2024-09-13T17:17:21.999599+0000","last_peered":"2024-09-13T17:17:21.999599+0000","last_clean":"2024-09-13T17:17:21.999599+0000","last_became_active":"2024-09-13T17:17:21.994943+0000","last_became_peered":"2024-09-13T17:17:21.994943+0000","last_unstale":"2024-09-13T17:17:21.999599+0000","last_undegraded":"2024-09-13T17:17:21.999599+0000","last_fullsized":"2024-09-13T17:17:21.999599+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:18.957602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989173+0000","last_change":"2024-09-13T17:17:21.980473+0000","last_active":"2024-09-13T17:17:21.989173+0000","last_peered":"2024-09-13T17:17:21.989173+0000","last_clean":"2024-09-13T17:17:21.989173+0000","last_became_active":"2024-09-13T17:17:21.980201+0000","last_became_peered":"2024-09-13T17:17:21.980201+0000","last_unstale":"2024-09-13T17:17:21.989173+0000","last_undegraded":"2024-09-13T17:17:21.989173+0000","last_fullsized":"2024-09-13T17:17:21.989173+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:20.546092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000533+0000","last_change":"2024-09-13T17:17:21.995917+0000","last_active":"2024-09-13T17:17:22.000533+0000","last_peered":"2024-09-13T17:17:22.000533+0000","last_clean":"2024-09-13T17:17:22.000533+0000","last_became_active":"2024-09-13T17:17:21.990973+0000","last_became_peered":"2024-09-13T17:17:21.990973+0000","last_unstale":"2024-09-13T17:17:22.000533+0000","last_undegraded":"2024-09-13T17:17:22.000533+0000","last_fullsized":"2024-09-13T17:17:22.000533+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:40.178733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989995+0000","last_change":"2024-09-13T17:17:21.980281+0000","last_active":"2024-09-13T17:17:21.989995+0000","last_peered":"2024-09-13T17:17:21.989995+0000","last_clean":"2024-09-13T17:17:21.989995+0000","last_became_active":"2024-09-13T17:17:21.979062+0000","last_became_peered":"2024-09-13T17:17:21.979062+0000","last_unstale":"2024-09-13T17:17:21.989995+0000","last_undegraded":"2024-09-13T17:17:21.989995+0000","last_fullsized":"2024-09-13T17:17:21.989995+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:16.743527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990281+0000","last_change":"2024-09-13T17:17:21.988901+0000","last_active":"2024-09-13T17:17:21.990281+0000","last_peered":"2024-09-13T17:17:21.990281+0000","last_clean":"2024-09-13T17:17:21.990281+0000","last_became_active":"2024-09-13T17:17:21.988626+0000","last_became_peered":"2024-09-13T17:17:21.988626+0000","last_unstale":"2024-09-13T17:17:21.990281+0000","last_undegraded":"2024-09-13T17:17:21.990281+0000","last_fullsized":"2024-09-13T17:17:21.990281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:40.178733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998287+0000","last_change":"2024-09-13T17:17:21.992354+0000","last_active":"2024-09-13T17:17:21.998287+0000","last_peered":"2024-09-13T17:17:21.998287+0000","last_clean":"2024-09-13T17:17:21.998287+0000","last_became_active":"2024-09-13T17:17:21.992100+0000","last_became_peered":"2024-09-13T17:17:21.992100+0000","last_unstale":"2024-09-13T17:17:21.998287+0000","last_undegraded":"2024-09-13T17:17:21.998287+0000","last_fullsized":"2024-09-13T17:17:21.998287+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:40.423842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.991100+0000","last_change":"2024-09-13T17:17:21.986059+0000","last_active":"2024-09-13T17:17:21.991100+0000","last_peered":"2024-09-13T17:17:21.991100+0000","last_clean":"2024-09-13T17:17:21.991100+0000","last_became_active":"2024-09-13T17:17:21.985807+0000","last_became_peered":"2024-09-13T17:17:21.985807+0000","last_unstale":"2024-09-13T17:17:21.991100+0000","last_undegraded":"2024-09-13T17:17:21.991100+0000","last_fullsized":"2024-09-13T17:17:21.991100+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:08.522356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986294+0000","last_change":"2024-09-13T17:17:21.978805+0000","last_active":"2024-09-13T17:17:21.986294+0000","last_peered":"2024-09-13T17:17:21.986294+0000","last_clean":"2024-09-13T17:17:21.986294+0000","last_became_active":"2024-09-13T17:17:21.978423+0000","last_became_peered":"2024-09-13T17:17:21.978423+0000","last_unstale":"2024-09-13T17:17:21.986294+0000","last_undegraded":"2024-09-13T17:17:21.986294+0000","last_fullsized":"2024-09-13T17:17:21.986294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:11.997263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000505+0000","last_change":"2024-09-13T17:17:21.999177+0000","last_active":"2024-09-13T17:17:22.000505+0000","last_peered":"2024-09-13T17:17:22.000505+0000","last_clean":"2024-09-13T17:17:22.000505+0000","last_became_active":"2024-09-13T17:17:21.995626+0000","last_became_peered":"2024-09-13T17:17:21.995626+0000","last_unstale":"2024-09-13T17:17:22.000505+0000","last_undegraded":"2024-09-13T17:17:22.000505+0000","last_fullsized":"2024-09-13T17:17:22.000505+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:57.085899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998332+0000","last_change":"2024-09-13T17:17:21.995291+0000","last_active":"2024-09-13T17:17:21.998332+0000","last_peered":"2024-09-13T17:17:21.998332+0000","last_clean":"2024-09-13T17:17:21.998332+0000","last_became_active":"2024-09-13T17:17:21.995012+0000","last_became_peered":"2024-09-13T17:17:21.995012+0000","last_unstale":"2024-09-13T17:17:21.998332+0000","last_undegraded":"2024-09-13T17:17:21.998332+0000","last_fullsized":"2024-09-13T17:17:21.998332+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:56.433707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999172+0000","last_change":"2024-09-13T17:17:21.995333+0000","last_active":"2024-09-13T17:17:21.999172+0000","last_peered":"2024-09-13T17:17:21.999172+0000","last_clean":"2024-09-13T17:17:21.999172+0000","last_became_active":"2024-09-13T17:17:21.994969+0000","last_became_peered":"2024-09-13T17:17:21.994969+0000","last_unstale":"2024-09-13T17:17:21.999172+0000","last_undegraded":"2024-09-13T17:17:21.999172+0000","last_fullsized":"2024-09-13T17:17:21.999172+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:52.663993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989407+0000","last_change":"2024-09-13T17:17:21.986032+0000","last_active":"2024-09-13T17:17:21.989407+0000","last_peered":"2024-09-13T17:17:21.989407+0000","last_clean":"2024-09-13T17:17:21.989407+0000","last_became_active":"2024-09-13T17:17:21.984686+0000","last_became_peered":"2024-09-13T17:17:21.984686+0000","last_unstale":"2024-09-13T17:17:21.989407+0000","last_undegraded":"2024-09-13T17:17:21.989407+0000","last_fullsized":"2024-09-13T17:17:21.989407+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:42.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995690+0000","last_change":"2024-09-13T17:17:21.990012+0000","last_active":"2024-09-13T17:17:21.995690+0000","last_peered":"2024-09-13T17:17:21.995690+0000","last_clean":"2024-09-13T17:17:21.995690+0000","last_became_active":"2024-09-13T17:17:21.989769+0000","last_became_peered":"2024-09-13T17:17:21.989769+0000","last_unstale":"2024-09-13T17:17:21.995690+0000","last_undegraded":"2024-09-13T17:17:21.995690+0000","last_fullsized":"2024-09-13T17:17:21.995690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:09.420840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"138'4","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003235+0000","last_change":"2024-09-13T17:17:21.559595+0000","last_active":"2024-09-13T17:17:22.003235+0000","last_peered":"2024-09-13T17:17:22.003235+0000","last_clean":"2024-09-13T17:17:22.003235+0000","last_became_active":"2024-09-13T17:17:21.559290+0000","last_became_peered":"2024-09-13T17:17:21.559290+0000","last_unstale":"2024-09-13T17:17:22.003235+0000","last_undegraded":"2024-09-13T17:17:22.003235+0000","last_fullsized":"2024-09-13T17:17:22.003235+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:49.726171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998981+0000","last_change":"2024-09-13T17:17:21.992307+0000","last_active":"2024-09-13T17:17:21.998981+0000","last_peered":"2024-09-13T17:17:21.998981+0000","last_clean":"2024-09-13T17:17:21.998981+0000","last_became_active":"2024-09-13T17:17:21.992062+0000","last_became_peered":"2024-09-13T17:17:21.992062+0000","last_unstale":"2024-09-13T17:17:21.998981+0000","last_undegraded":"2024-09-13T17:17:21.998981+0000","last_fullsized":"2024-09-13T17:17:21.998981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:12.922709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997347+0000","last_change":"2024-09-13T17:17:21.545483+0000","last_active":"2024-09-13T17:17:21.997347+0000","last_peered":"2024-09-13T17:17:21.997347+0000","last_clean":"2024-09-13T17:17:21.997347+0000","last_became_active":"2024-09-13T17:17:21.544889+0000","last_became_peered":"2024-09-13T17:17:21.544889+0000","last_unstale":"2024-09-13T17:17:21.997347+0000","last_undegraded":"2024-09-13T17:17:21.997347+0000","last_fullsized":"2024-09-13T17:17:21.997347+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:02.473285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000566+0000","last_change":"2024-09-13T17:17:21.998528+0000","last_active":"2024-09-13T17:17:22.000566+0000","last_peered":"2024-09-13T17:17:22.000566+0000","last_clean":"2024-09-13T17:17:22.000566+0000","last_became_active":"2024-09-13T17:17:21.991185+0000","last_became_peered":"2024-09-13T17:17:21.991185+0000","last_unstale":"2024-09-13T17:17:22.000566+0000","last_undegraded":"2024-09-13T17:17:22.000566+0000","last_fullsized":"2024-09-13T17:17:22.000566+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:24:52.661143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999228+0000","last_change":"2024-09-13T17:17:21.552752+0000","last_active":"2024-09-13T17:17:21.999228+0000","last_peered":"2024-09-13T17:17:21.999228+0000","last_clean":"2024-09-13T17:17:21.999228+0000","last_became_active":"2024-09-13T17:17:21.552419+0000","last_became_peered":"2024-09-13T17:17:21.552419+0000","last_unstale":"2024-09-13T17:17:21.999228+0000","last_undegraded":"2024-09-13T17:17:21.999228+0000","last_fullsized":"2024-09-13T17:17:21.999228+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:08.710863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999726+0000","last_change":"2024-09-13T17:17:21.989609+0000","last_active":"2024-09-13T17:17:21.999726+0000","last_peered":"2024-09-13T17:17:21.999726+0000","last_clean":"2024-09-13T17:17:21.999726+0000","last_became_active":"2024-09-13T17:17:21.989276+0000","last_became_peered":"2024-09-13T17:17:21.989276+0000","last_unstale":"2024-09-13T17:17:21.999726+0000","last_undegraded":"2024-09-13T17:17:21.999726+0000","last_fullsized":"2024-09-13T17:17:21.999726+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:02.424600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996090+0000","last_change":"2024-09-13T17:17:21.557156+0000","last_active":"2024-09-13T17:17:21.996090+0000","last_peered":"2024-09-13T17:17:21.996090+0000","last_clean":"2024-09-13T17:17:21.996090+0000","last_became_active":"2024-09-13T17:17:21.556412+0000","last_became_peered":"2024-09-13T17:17:21.556412+0000","last_unstale":"2024-09-13T17:17:21.996090+0000","last_undegraded":"2024-09-13T17:17:21.996090+0000","last_fullsized":"2024-09-13T17:17:21.996090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:15.161326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.982382+0000","last_change":"2024-09-13T17:17:21.981075+0000","last_active":"2024-09-13T17:17:21.982382+0000","last_peered":"2024-09-13T17:17:21.982382+0000","last_clean":"2024-09-13T17:17:21.982382+0000","last_became_active":"2024-09-13T17:17:21.980813+0000","last_became_peered":"2024-09-13T17:17:21.980813+0000","last_unstale":"2024-09-13T17:17:21.982382+0000","last_undegraded":"2024-09-13T17:17:21.982382+0000","last_fullsized":"2024-09-13T17:17:21.982382+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:48.291495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995429+0000","last_change":"2024-09-13T17:17:21.554549+0000","last_active":"2024-09-13T17:17:21.995429+0000","last_peered":"2024-09-13T17:17:21.995429+0000","last_clean":"2024-09-13T17:17:21.995429+0000","last_became_active":"2024-09-13T17:17:21.554075+0000","last_became_peered":"2024-09-13T17:17:21.554075+0000","last_unstale":"2024-09-13T17:17:21.995429+0000","last_undegraded":"2024-09-13T17:17:21.995429+0000","last_fullsized":"2024-09-13T17:17:21.995429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:07.514923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979202+0000","last_change":"2024-09-13T17:17:21.977138+0000","last_active":"2024-09-13T17:17:21.979202+0000","last_peered":"2024-09-13T17:17:21.979202+0000","last_clean":"2024-09-13T17:17:21.979202+0000","last_became_active":"2024-09-13T17:17:21.976844+0000","last_became_peered":"2024-09-13T17:17:21.976844+0000","last_unstale":"2024-09-13T17:17:21.979202+0000","last_undegraded":"2024-09-13T17:17:21.979202+0000","last_fullsized":"2024-09-13T17:17:21.979202+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:50.034777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997518+0000","last_change":"2024-09-13T17:17:21.545304+0000","last_active":"2024-09-13T17:17:21.997518+0000","last_peered":"2024-09-13T17:17:21.997518+0000","last_clean":"2024-09-13T17:17:21.997518+0000","last_became_active":"2024-09-13T17:17:21.545005+0000","last_became_peered":"2024-09-13T17:17:21.545005+0000","last_unstale":"2024-09-13T17:17:21.997518+0000","last_undegraded":"2024-09-13T17:17:21.997518+0000","last_fullsized":"2024-09-13T17:17:21.997518+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:41.496143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993727+0000","last_change":"2024-09-13T17:17:21.985724+0000","last_active":"2024-09-13T17:17:21.993727+0000","last_peered":"2024-09-13T17:17:21.993727+0000","last_clean":"2024-09-13T17:17:21.993727+0000","last_became_active":"2024-09-13T17:17:21.984151+0000","last_became_peered":"2024-09-13T17:17:21.984151+0000","last_unstale":"2024-09-13T17:17:21.993727+0000","last_undegraded":"2024-09-13T17:17:21.993727+0000","last_fullsized":"2024-09-13T17:17:21.993727+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:04.216711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.972069+0000","last_change":"2024-09-13T17:17:21.550873+0000","last_active":"2024-09-13T17:17:21.972069+0000","last_peered":"2024-09-13T17:17:21.972069+0000","last_clean":"2024-09-13T17:17:21.972069+0000","last_became_active":"2024-09-13T17:17:21.550463+0000","last_became_peered":"2024-09-13T17:17:21.550463+0000","last_unstale":"2024-09-13T17:17:21.972069+0000","last_undegraded":"2024-09-13T17:17:21.972069+0000","last_fullsized":"2024-09-13T17:17:21.972069+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:37.870327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.983624+0000","last_change":"2024-09-13T17:17:21.978063+0000","last_active":"2024-09-13T17:17:21.983624+0000","last_peered":"2024-09-13T17:17:21.983624+0000","last_clean":"2024-09-13T17:17:21.983624+0000","last_became_active":"2024-09-13T17:17:21.977839+0000","last_became_peered":"2024-09-13T17:17:21.977839+0000","last_unstale":"2024-09-13T17:17:21.983624+0000","last_undegraded":"2024-09-13T17:17:21.983624+0000","last_fullsized":"2024-09-13T17:17:21.983624+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:25.926767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.968718+0000","last_change":"2024-09-13T17:17:21.550860+0000","last_active":"2024-09-13T17:17:21.968718+0000","last_peered":"2024-09-13T17:17:21.968718+0000","last_clean":"2024-09-13T17:17:21.968718+0000","last_became_active":"2024-09-13T17:17:21.550574+0000","last_became_peered":"2024-09-13T17:17:21.550574+0000","last_unstale":"2024-09-13T17:17:21.968718+0000","last_undegraded":"2024-09-13T17:17:21.968718+0000","last_fullsized":"2024-09-13T17:17:21.968718+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:35.620810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000521+0000","last_change":"2024-09-13T17:17:21.994502+0000","last_active":"2024-09-13T17:17:22.000521+0000","last_peered":"2024-09-13T17:17:22.000521+0000","last_clean":"2024-09-13T17:17:22.000521+0000","last_became_active":"2024-09-13T17:17:21.994234+0000","last_became_peered":"2024-09-13T17:17:21.994234+0000","last_unstale":"2024-09-13T17:17:22.000521+0000","last_undegraded":"2024-09-13T17:17:22.000521+0000","last_fullsized":"2024-09-13T17:17:22.000521+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:28.231040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.971688+0000","last_change":"2024-09-13T17:17:21.551427+0000","last_active":"2024-09-13T17:17:21.971688+0000","last_peered":"2024-09-13T17:17:21.971688+0000","last_clean":"2024-09-13T17:17:21.971688+0000","last_became_active":"2024-09-13T17:17:21.551124+0000","last_became_peered":"2024-09-13T17:17:21.551124+0000","last_unstale":"2024-09-13T17:17:21.971688+0000","last_undegraded":"2024-09-13T17:17:21.971688+0000","last_fullsized":"2024-09-13T17:17:21.971688+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:05.186289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989485+0000","last_change":"2024-09-13T17:17:21.987309+0000","last_active":"2024-09-13T17:17:21.989485+0000","last_peered":"2024-09-13T17:17:21.989485+0000","last_clean":"2024-09-13T17:17:21.989485+0000","last_became_active":"2024-09-13T17:17:21.986955+0000","last_became_peered":"2024-09-13T17:17:21.986955+0000","last_unstale":"2024-09-13T17:17:21.989485+0000","last_undegraded":"2024-09-13T17:17:21.989485+0000","last_fullsized":"2024-09-13T17:17:21.989485+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:34.795107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.982601+0000","last_change":"2024-09-13T17:17:21.556146+0000","last_active":"2024-09-13T17:17:21.982601+0000","last_peered":"2024-09-13T17:17:21.982601+0000","last_clean":"2024-09-13T17:17:21.982601+0000","last_became_active":"2024-09-13T17:17:21.555157+0000","last_became_peered":"2024-09-13T17:17:21.555157+0000","last_unstale":"2024-09-13T17:17:21.982601+0000","last_undegraded":"2024-09-13T17:17:21.982601+0000","last_fullsized":"2024-09-13T17:17:21.982601+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:51.656560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999604+0000","last_change":"2024-09-13T17:17:21.994983+0000","last_active":"2024-09-13T17:17:21.999604+0000","last_peered":"2024-09-13T17:17:21.999604+0000","last_clean":"2024-09-13T17:17:21.999604+0000","last_became_active":"2024-09-13T17:17:21.990951+0000","last_became_peered":"2024-09-13T17:17:21.990951+0000","last_unstale":"2024-09-13T17:17:21.999604+0000","last_undegraded":"2024-09-13T17:17:21.999604+0000","last_fullsized":"2024-09-13T17:17:21.999604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:01.023328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"138'3","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999692+0000","last_change":"2024-09-13T17:17:21.547608+0000","last_active":"2024-09-13T17:17:21.999692+0000","last_peered":"2024-09-13T17:17:21.999692+0000","last_clean":"2024-09-13T17:17:21.999692+0000","last_became_active":"2024-09-13T17:17:21.547135+0000","last_became_peered":"2024-09-13T17:17:21.547135+0000","last_unstale":"2024-09-13T17:17:21.999692+0000","last_undegraded":"2024-09-13T17:17:21.999692+0000","last_fullsized":"2024-09-13T17:17:21.999692+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:16.781802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999744+0000","last_change":"2024-09-13T17:17:21.995654+0000","last_active":"2024-09-13T17:17:21.999744+0000","last_peered":"2024-09-13T17:17:21.999744+0000","last_clean":"2024-09-13T17:17:21.999744+0000","last_became_active":"2024-09-13T17:17:21.995413+0000","last_became_peered":"2024-09-13T17:17:21.995413+0000","last_unstale":"2024-09-13T17:17:21.999744+0000","last_undegraded":"2024-09-13T17:17:21.999744+0000","last_fullsized":"2024-09-13T17:17:21.999744+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:10.706481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"138'6","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003483+0000","last_change":"2024-09-13T17:17:21.553589+0000","last_active":"2024-09-13T17:17:22.003483+0000","last_peered":"2024-09-13T17:17:22.003483+0000","last_clean":"2024-09-13T17:17:22.003483+0000","last_became_active":"2024-09-13T17:17:21.553037+0000","last_became_peered":"2024-09-13T17:17:21.553037+0000","last_unstale":"2024-09-13T17:17:22.003483+0000","last_undegraded":"2024-09-13T17:17:22.003483+0000","last_fullsized":"2024-09-13T17:17:22.003483+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.144900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.992433+0000","last_change":"2024-09-13T17:17:21.987385+0000","last_active":"2024-09-13T17:17:21.992433+0000","last_peered":"2024-09-13T17:17:21.992433+0000","last_clean":"2024-09-13T17:17:21.992433+0000","last_became_active":"2024-09-13T17:17:21.987057+0000","last_became_peered":"2024-09-13T17:17:21.987057+0000","last_unstale":"2024-09-13T17:17:21.992433+0000","last_undegraded":"2024-09-13T17:17:21.992433+0000","last_fullsized":"2024-09-13T17:17:21.992433+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:06:03.556535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.975872+0000","last_change":"2024-09-13T17:17:21.548772+0000","last_active":"2024-09-13T17:17:21.975872+0000","last_peered":"2024-09-13T17:17:21.975872+0000","last_clean":"2024-09-13T17:17:21.975872+0000","last_became_active":"2024-09-13T17:17:21.548511+0000","last_became_peered":"2024-09-13T17:17:21.548511+0000","last_unstale":"2024-09-13T17:17:21.975872+0000","last_undegraded":"2024-09-13T17:17:21.975872+0000","last_fullsized":"2024-09-13T17:17:21.975872+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:00.014392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990276+0000","last_change":"2024-09-13T17:17:21.988770+0000","last_active":"2024-09-13T17:17:21.990276+0000","last_peered":"2024-09-13T17:17:21.990276+0000","last_clean":"2024-09-13T17:17:21.990276+0000","last_became_active":"2024-09-13T17:17:21.988500+0000","last_became_peered":"2024-09-13T17:17:21.988500+0000","last_unstale":"2024-09-13T17:17:21.990276+0000","last_undegraded":"2024-09-13T17:17:21.990276+0000","last_fullsized":"2024-09-13T17:17:21.990276+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:32.527360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.969717+0000","last_change":"2024-09-13T17:17:21.554545+0000","last_active":"2024-09-13T17:17:21.969717+0000","last_peered":"2024-09-13T17:17:21.969717+0000","last_clean":"2024-09-13T17:17:21.969717+0000","last_became_active":"2024-09-13T17:17:21.554207+0000","last_became_peered":"2024-09-13T17:17:21.554207+0000","last_unstale":"2024-09-13T17:17:21.969717+0000","last_undegraded":"2024-09-13T17:17:21.969717+0000","last_fullsized":"2024-09-13T17:17:21.969717+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:29.221751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000650+0000","last_change":"2024-09-13T17:17:21.992018+0000","last_active":"2024-09-13T17:17:22.000650+0000","last_peered":"2024-09-13T17:17:22.000650+0000","last_clean":"2024-09-13T17:17:22.000650+0000","last_became_active":"2024-09-13T17:17:21.991668+0000","last_became_peered":"2024-09-13T17:17:21.991668+0000","last_unstale":"2024-09-13T17:17:22.000650+0000","last_undegraded":"2024-09-13T17:17:22.000650+0000","last_fullsized":"2024-09-13T17:17:22.000650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:34.795107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"138'8","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.001812+0000","last_change":"2024-09-13T17:17:21.550775+0000","last_active":"2024-09-13T17:17:22.001812+0000","last_peered":"2024-09-13T17:17:22.001812+0000","last_clean":"2024-09-13T17:17:22.001812+0000","last_became_active":"2024-09-13T17:17:21.550268+0000","last_became_peered":"2024-09-13T17:17:21.550268+0000","last_unstale":"2024-09-13T17:17:22.001812+0000","last_undegraded":"2024-09-13T17:17:22.001812+0000","last_fullsized":"2024-09-13T17:17:22.001812+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:16.781802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000912+0000","last_change":"2024-09-13T17:17:21.997552+0000","last_active":"2024-09-13T17:17:22.000912+0000","last_peered":"2024-09-13T17:17:22.000912+0000","last_clean":"2024-09-13T17:17:22.000912+0000","last_became_active":"2024-09-13T17:17:21.993602+0000","last_became_peered":"2024-09-13T17:17:21.993602+0000","last_unstale":"2024-09-13T17:17:22.000912+0000","last_undegraded":"2024-09-13T17:17:22.000912+0000","last_fullsized":"2024-09-13T17:17:22.000912+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:57.259385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000248+0000","last_change":"2024-09-13T17:17:21.548233+0000","last_active":"2024-09-13T17:17:22.000248+0000","last_peered":"2024-09-13T17:17:22.000248+0000","last_clean":"2024-09-13T17:17:22.000248+0000","last_became_active":"2024-09-13T17:17:21.548018+0000","last_became_peered":"2024-09-13T17:17:21.548018+0000","last_unstale":"2024-09-13T17:17:22.000248+0000","last_undegraded":"2024-09-13T17:17:22.000248+0000","last_fullsized":"2024-09-13T17:17:22.000248+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:21.917024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.987158+0000","last_change":"2024-09-13T17:17:21.978340+0000","last_active":"2024-09-13T17:17:21.987158+0000","last_peered":"2024-09-13T17:17:21.987158+0000","last_clean":"2024-09-13T17:17:21.987158+0000","last_became_active":"2024-09-13T17:17:21.978045+0000","last_became_peered":"2024-09-13T17:17:21.978045+0000","last_unstale":"2024-09-13T17:17:21.987158+0000","last_undegraded":"2024-09-13T17:17:21.987158+0000","last_fullsized":"2024-09-13T17:17:21.987158+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998192+0000","last_change":"2024-09-13T17:17:21.548112+0000","last_active":"2024-09-13T17:17:21.998192+0000","last_peered":"2024-09-13T17:17:21.998192+0000","last_clean":"2024-09-13T17:17:21.998192+0000","last_became_active":"2024-09-13T17:17:21.547791+0000","last_became_peered":"2024-09-13T17:17:21.547791+0000","last_unstale":"2024-09-13T17:17:21.998192+0000","last_undegraded":"2024-09-13T17:17:21.998192+0000","last_fullsized":"2024-09-13T17:17:21.998192+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:23.020261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002939+0000","last_change":"2024-09-13T17:17:22.002028+0000","last_active":"2024-09-13T17:17:22.002939+0000","last_peered":"2024-09-13T17:17:22.002939+0000","last_clean":"2024-09-13T17:17:22.002939+0000","last_became_active":"2024-09-13T17:17:22.001559+0000","last_became_peered":"2024-09-13T17:17:22.001559+0000","last_unstale":"2024-09-13T17:17:22.002939+0000","last_undegraded":"2024-09-13T17:17:22.002939+0000","last_fullsized":"2024-09-13T17:17:22.002939+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:24.027694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986881+0000","last_change":"2024-09-13T17:17:21.546597+0000","last_active":"2024-09-13T17:17:21.986881+0000","last_peered":"2024-09-13T17:17:21.986881+0000","last_clean":"2024-09-13T17:17:21.986881+0000","last_became_active":"2024-09-13T17:17:21.546290+0000","last_became_peered":"2024-09-13T17:17:21.546290+0000","last_unstale":"2024-09-13T17:17:21.986881+0000","last_undegraded":"2024-09-13T17:17:21.986881+0000","last_fullsized":"2024-09-13T17:17:21.986881+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:17.950169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.984821+0000","last_change":"2024-09-13T17:17:21.979588+0000","last_active":"2024-09-13T17:17:21.984821+0000","last_peered":"2024-09-13T17:17:21.984821+0000","last_clean":"2024-09-13T17:17:21.984821+0000","last_became_active":"2024-09-13T17:17:21.979317+0000","last_became_peered":"2024-09-13T17:17:21.979317+0000","last_unstale":"2024-09-13T17:17:21.984821+0000","last_undegraded":"2024-09-13T17:17:21.984821+0000","last_fullsized":"2024-09-13T17:17:21.984821+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:42.953373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.970867+0000","last_change":"2024-09-13T17:17:21.545670+0000","last_active":"2024-09-13T17:17:21.970867+0000","last_peered":"2024-09-13T17:17:21.970867+0000","last_clean":"2024-09-13T17:17:21.970867+0000","last_became_active":"2024-09-13T17:17:21.543851+0000","last_became_peered":"2024-09-13T17:17:21.543851+0000","last_unstale":"2024-09-13T17:17:21.970867+0000","last_undegraded":"2024-09-13T17:17:21.970867+0000","last_fullsized":"2024-09-13T17:17:21.970867+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:25.687649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998312+0000","last_change":"2024-09-13T17:17:21.991030+0000","last_active":"2024-09-13T17:17:21.998312+0000","last_peered":"2024-09-13T17:17:21.998312+0000","last_clean":"2024-09-13T17:17:21.998312+0000","last_became_active":"2024-09-13T17:17:21.990795+0000","last_became_peered":"2024-09-13T17:17:21.990795+0000","last_unstale":"2024-09-13T17:17:21.998312+0000","last_undegraded":"2024-09-13T17:17:21.998312+0000","last_fullsized":"2024-09-13T17:17:21.998312+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:36.628243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.001223+0000","last_change":"2024-09-13T17:17:21.552671+0000","last_active":"2024-09-13T17:17:22.001223+0000","last_peered":"2024-09-13T17:17:22.001223+0000","last_clean":"2024-09-13T17:17:22.001223+0000","last_became_active":"2024-09-13T17:17:21.551524+0000","last_became_peered":"2024-09-13T17:17:21.551524+0000","last_unstale":"2024-09-13T17:17:22.001223+0000","last_undegraded":"2024-09-13T17:17:22.001223+0000","last_fullsized":"2024-09-13T17:17:22.001223+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:27.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990082+0000","last_change":"2024-09-13T17:17:21.985804+0000","last_active":"2024-09-13T17:17:21.990082+0000","last_peered":"2024-09-13T17:17:21.990082+0000","last_clean":"2024-09-13T17:17:21.990082+0000","last_became_active":"2024-09-13T17:17:21.985500+0000","last_became_peered":"2024-09-13T17:17:21.985500+0000","last_unstale":"2024-09-13T17:17:21.990082+0000","last_undegraded":"2024-09-13T17:17:21.990082+0000","last_fullsized":"2024-09-13T17:17:21.990082+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:57.067573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.977121+0000","last_change":"2024-09-13T17:17:21.555641+0000","last_active":"2024-09-13T17:17:21.977121+0000","last_peered":"2024-09-13T17:17:21.977121+0000","last_clean":"2024-09-13T17:17:21.977121+0000","last_became_active":"2024-09-13T17:17:21.554741+0000","last_became_peered":"2024-09-13T17:17:21.554741+0000","last_unstale":"2024-09-13T17:17:21.977121+0000","last_undegraded":"2024-09-13T17:17:21.977121+0000","last_fullsized":"2024-09-13T17:17:21.977121+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:01.417167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986432+0000","last_change":"2024-09-13T17:17:21.981864+0000","last_active":"2024-09-13T17:17:21.986432+0000","last_peered":"2024-09-13T17:17:21.986432+0000","last_clean":"2024-09-13T17:17:21.986432+0000","last_became_active":"2024-09-13T17:17:21.981602+0000","last_became_peered":"2024-09-13T17:17:21.981602+0000","last_unstale":"2024-09-13T17:17:21.986432+0000","last_undegraded":"2024-09-13T17:17:21.986432+0000","last_fullsized":"2024-09-13T17:17:21.986432+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.980913+0000","last_change":"2024-09-13T17:17:21.541881+0000","last_active":"2024-09-13T17:17:21.980913+0000","last_peered":"2024-09-13T17:17:21.980913+0000","last_clean":"2024-09-13T17:17:21.980913+0000","last_became_active":"2024-09-13T17:17:21.541253+0000","last_became_peered":"2024-09-13T17:17:21.541253+0000","last_unstale":"2024-09-13T17:17:21.980913+0000","last_undegraded":"2024-09-13T17:17:21.980913+0000","last_fullsized":"2024-09-13T17:17:21.980913+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:07.974799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002671+0000","last_change":"2024-09-13T17:17:22.001879+0000","last_active":"2024-09-13T17:17:22.002671+0000","last_peered":"2024-09-13T17:17:22.002671+0000","last_clean":"2024-09-13T17:17:22.002671+0000","last_became_active":"2024-09-13T17:17:22.001581+0000","last_became_peered":"2024-09-13T17:17:22.001581+0000","last_unstale":"2024-09-13T17:17:22.002671+0000","last_undegraded":"2024-09-13T17:17:22.002671+0000","last_fullsized":"2024-09-13T17:17:22.002671+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:08.982232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.973394+0000","last_change":"2024-09-13T17:17:21.549737+0000","last_active":"2024-09-13T17:17:21.973394+0000","last_peered":"2024-09-13T17:17:21.973394+0000","last_clean":"2024-09-13T17:17:21.973394+0000","last_became_active":"2024-09-13T17:17:21.549415+0000","last_became_peered":"2024-09-13T17:17:21.549415+0000","last_unstale":"2024-09-13T17:17:21.973394+0000","last_undegraded":"2024-09-13T17:17:21.973394+0000","last_fullsized":"2024-09-13T17:17:21.973394+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:05.464294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.976673+0000","last_change":"2024-09-13T17:17:21.974701+0000","last_active":"2024-09-13T17:17:21.976673+0000","last_peered":"2024-09-13T17:17:21.976673+0000","last_clean":"2024-09-13T17:17:21.976673+0000","last_became_active":"2024-09-13T17:17:21.974399+0000","last_became_peered":"2024-09-13T17:17:21.974399+0000","last_unstale":"2024-09-13T17:17:21.976673+0000","last_undegraded":"2024-09-13T17:17:21.976673+0000","last_fullsized":"2024-09-13T17:17:21.976673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:30.412399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002938+0000","last_change":"2024-09-13T17:17:21.551133+0000","last_active":"2024-09-13T17:17:22.002938+0000","last_peered":"2024-09-13T17:17:22.002938+0000","last_clean":"2024-09-13T17:17:22.002938+0000","last_became_active":"2024-09-13T17:17:21.550782+0000","last_became_peered":"2024-09-13T17:17:21.550782+0000","last_unstale":"2024-09-13T17:17:22.002938+0000","last_undegraded":"2024-09-13T17:17:22.002938+0000","last_fullsized":"2024-09-13T17:17:22.002938+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:25.672114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000258+0000","last_change":"2024-09-13T17:17:21.998885+0000","last_active":"2024-09-13T17:17:22.000258+0000","last_peered":"2024-09-13T17:17:22.000258+0000","last_clean":"2024-09-13T17:17:22.000258+0000","last_became_active":"2024-09-13T17:17:21.995692+0000","last_became_peered":"2024-09-13T17:17:21.995692+0000","last_unstale":"2024-09-13T17:17:22.000258+0000","last_undegraded":"2024-09-13T17:17:22.000258+0000","last_fullsized":"2024-09-13T17:17:22.000258+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:08.790760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.980646+0000","last_change":"2024-09-13T17:17:21.554984+0000","last_active":"2024-09-13T17:17:21.980646+0000","last_peered":"2024-09-13T17:17:21.980646+0000","last_clean":"2024-09-13T17:17:21.980646+0000","last_became_active":"2024-09-13T17:17:21.554700+0000","last_became_peered":"2024-09-13T17:17:21.554700+0000","last_unstale":"2024-09-13T17:17:21.980646+0000","last_undegraded":"2024-09-13T17:17:21.980646+0000","last_fullsized":"2024-09-13T17:17:21.980646+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:10.989830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993323+0000","last_change":"2024-09-13T17:17:21.990869+0000","last_active":"2024-09-13T17:17:21.993323+0000","last_peered":"2024-09-13T17:17:21.993323+0000","last_clean":"2024-09-13T17:17:21.993323+0000","last_became_active":"2024-09-13T17:17:21.990638+0000","last_became_peered":"2024-09-13T17:17:21.990638+0000","last_unstale":"2024-09-13T17:17:21.993323+0000","last_undegraded":"2024-09-13T17:17:21.993323+0000","last_fullsized":"2024-09-13T17:17:21.993323+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:11.997263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.969156+0000","last_change":"2024-09-13T17:17:21.549987+0000","last_active":"2024-09-13T17:17:21.969156+0000","last_peered":"2024-09-13T17:17:21.969156+0000","last_clean":"2024-09-13T17:17:21.969156+0000","last_became_active":"2024-09-13T17:17:21.549725+0000","last_became_peered":"2024-09-13T17:17:21.549725+0000","last_unstale":"2024-09-13T17:17:21.969156+0000","last_undegraded":"2024-09-13T17:17:21.969156+0000","last_fullsized":"2024-09-13T17:17:21.969156+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:27.223607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996064+0000","last_change":"2024-09-13T17:17:21.993795+0000","last_active":"2024-09-13T17:17:21.996064+0000","last_peered":"2024-09-13T17:17:21.996064+0000","last_clean":"2024-09-13T17:17:21.996064+0000","last_became_active":"2024-09-13T17:17:21.993520+0000","last_became_peered":"2024-09-13T17:17:21.993520+0000","last_unstale":"2024-09-13T17:17:21.996064+0000","last_undegraded":"2024-09-13T17:17:21.996064+0000","last_fullsized":"2024-09-13T17:17:21.996064+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:26.727294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.974149+0000","last_change":"2024-09-13T17:17:21.552928+0000","last_active":"2024-09-13T17:17:21.974149+0000","last_peered":"2024-09-13T17:17:21.974149+0000","last_clean":"2024-09-13T17:17:21.974149+0000","last_became_active":"2024-09-13T17:17:21.552628+0000","last_became_peered":"2024-09-13T17:17:21.552628+0000","last_unstale":"2024-09-13T17:17:21.974149+0000","last_undegraded":"2024-09-13T17:17:21.974149+0000","last_fullsized":"2024-09-13T17:17:21.974149+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:53.189574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999210+0000","last_change":"2024-09-13T17:17:21.996620+0000","last_active":"2024-09-13T17:17:21.999210+0000","last_peered":"2024-09-13T17:17:21.999210+0000","last_clean":"2024-09-13T17:17:21.999210+0000","last_became_active":"2024-09-13T17:17:21.996322+0000","last_became_peered":"2024-09-13T17:17:21.996322+0000","last_unstale":"2024-09-13T17:17:21.999210+0000","last_undegraded":"2024-09-13T17:17:21.999210+0000","last_fullsized":"2024-09-13T17:17:21.999210+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:03.967219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.968628+0000","last_change":"2024-09-13T17:17:21.554349+0000","last_active":"2024-09-13T17:17:21.968628+0000","last_peered":"2024-09-13T17:17:21.968628+0000","last_clean":"2024-09-13T17:17:21.968628+0000","last_became_active":"2024-09-13T17:17:21.553680+0000","last_became_peered":"2024-09-13T17:17:21.553680+0000","last_unstale":"2024-09-13T17:17:21.968628+0000","last_undegraded":"2024-09-13T17:17:21.968628+0000","last_fullsized":"2024-09-13T17:17:21.968628+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:08.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989352+0000","last_change":"2024-09-13T17:17:21.987327+0000","last_active":"2024-09-13T17:17:21.989352+0000","last_peered":"2024-09-13T17:17:21.989352+0000","last_clean":"2024-09-13T17:17:21.989352+0000","last_became_active":"2024-09-13T17:17:21.986322+0000","last_became_peered":"2024-09-13T17:17:21.986322+0000","last_unstale":"2024-09-13T17:17:21.989352+0000","last_undegraded":"2024-09-13T17:17:21.989352+0000","last_fullsized":"2024-09-13T17:17:21.989352+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:04.216711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979465+0000","last_change":"2024-09-13T17:17:21.556477+0000","last_active":"2024-09-13T17:17:21.979465+0000","last_peered":"2024-09-13T17:17:21.979465+0000","last_clean":"2024-09-13T17:17:21.979465+0000","last_became_active":"2024-09-13T17:17:21.556142+0000","last_became_peered":"2024-09-13T17:17:21.556142+0000","last_unstale":"2024-09-13T17:17:21.979465+0000","last_undegraded":"2024-09-13T17:17:21.979465+0000","last_fullsized":"2024-09-13T17:17:21.979465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:39.171300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998819+0000","last_change":"2024-09-13T17:17:21.997075+0000","last_active":"2024-09-13T17:17:21.998819+0000","last_peered":"2024-09-13T17:17:21.998819+0000","last_clean":"2024-09-13T17:17:21.998819+0000","last_became_active":"2024-09-13T17:17:21.996300+0000","last_became_peered":"2024-09-13T17:17:21.996300+0000","last_unstale":"2024-09-13T17:17:21.998819+0000","last_undegraded":"2024-09-13T17:17:21.998819+0000","last_fullsized":"2024-09-13T17:17:21.998819+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:06.471727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.981025+0000","last_change":"2024-09-13T17:17:21.553019+0000","last_active":"2024-09-13T17:17:21.981025+0000","last_peered":"2024-09-13T17:17:21.981025+0000","last_clean":"2024-09-13T17:17:21.981025+0000","last_became_active":"2024-09-13T17:17:21.552745+0000","last_became_peered":"2024-09-13T17:17:21.552745+0000","last_unstale":"2024-09-13T17:17:21.981025+0000","last_undegraded":"2024-09-13T17:17:21.981025+0000","last_fullsized":"2024-09-13T17:17:21.981025+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:56.825607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997988+0000","last_change":"2024-09-13T17:17:21.997306+0000","last_active":"2024-09-13T17:17:21.997988+0000","last_peered":"2024-09-13T17:17:21.997988+0000","last_clean":"2024-09-13T17:17:21.997988+0000","last_became_active":"2024-09-13T17:17:21.997057+0000","last_became_peered":"2024-09-13T17:17:21.997057+0000","last_unstale":"2024-09-13T17:17:21.997988+0000","last_undegraded":"2024-09-13T17:17:21.997988+0000","last_fullsized":"2024-09-13T17:17:21.997988+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:21.454835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.973562+0000","last_change":"2024-09-13T17:17:21.543552+0000","last_active":"2024-09-13T17:17:21.973562+0000","last_peered":"2024-09-13T17:17:21.973562+0000","last_clean":"2024-09-13T17:17:21.973562+0000","last_became_active":"2024-09-13T17:17:21.543275+0000","last_became_peered":"2024-09-13T17:17:21.543275+0000","last_unstale":"2024-09-13T17:17:21.973562+0000","last_undegraded":"2024-09-13T17:17:21.973562+0000","last_fullsized":"2024-09-13T17:17:21.973562+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:39.416409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002714+0000","last_change":"2024-09-13T17:17:22.001979+0000","last_active":"2024-09-13T17:17:22.002714+0000","last_peered":"2024-09-13T17:17:22.002714+0000","last_clean":"2024-09-13T17:17:22.002714+0000","last_became_active":"2024-09-13T17:17:22.001642+0000","last_became_peered":"2024-09-13T17:17:22.001642+0000","last_unstale":"2024-09-13T17:17:22.002714+0000","last_undegraded":"2024-09-13T17:17:22.002714+0000","last_fullsized":"2024-09-13T17:17:22.002714+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002015+0000","last_change":"2024-09-13T17:17:21.559463+0000","last_active":"2024-09-13T17:17:22.002015+0000","last_peered":"2024-09-13T17:17:22.002015+0000","last_clean":"2024-09-13T17:17:22.002015+0000","last_became_active":"2024-09-13T17:17:21.559131+0000","last_became_peered":"2024-09-13T17:17:21.559131+0000","last_unstale":"2024-09-13T17:17:22.002015+0000","last_undegraded":"2024-09-13T17:17:22.002015+0000","last_fullsized":"2024-09-13T17:17:22.002015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:33.787674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":263,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.550175+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:17:21.550175+0000","last_peered":"2024-09-13T17:17:21.550175+0000","last_clean":"2024-09-13T17:17:21.550175+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:17:21.550175+0000","last_undegraded":"2024-09-13T17:17:21.550175+0000","last_fullsized":"2024-09-13T17:17:21.550175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73536,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298304,"statfs":{"total":96636764160,"available":96561463296,"internally_reserved":0,"allocated":2338816,"data_stored":1347787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":72939979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.64700000000000002},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.96899999999999997,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.004,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68400000000000005,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.748,"15min":0.70799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86399999999999999,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.51800000000000002,"5min":0.48299999999999998,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":1.544,"15min":1.544},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":1.605,"15min":1.605},"last":0.63}]},{"osd":3,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.38800000000000001},"max":{"1min":0.92200000000000004,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.496,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93600000000000005,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.60699999999999998,"15min":0.64000000000000001},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.879,"5min":0.90400000000000003,"15min":1.054},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64700000000000002,"15min":0.67800000000000005},"min":{"1min":0.432,"5min":0.39800000000000002,"15min":0.35399999999999998},"max":{"1min":0.90400000000000003,"5min":0.92900000000000005,"15min":1.1220000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.80400000000000005,"5min":0.94899999999999995,"15min":1.143},"last":0.375},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74399999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":0.91100000000000003,"5min":1.026,"15min":1.337},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.71699999999999997,"15min":0.751},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.89300000000000002,"5min":0.99199999999999999,"15min":1.1930000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":1.2030000000000001},"last":0.46000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54960,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316880,"statfs":{"total":96636764160,"available":96580485120,"internally_reserved":0,"allocated":1732608,"data_stored":756781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32900000000000001},"max":{"1min":0.871,"5min":1.298,"15min":3.6629999999999998},"last":0.627},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":0.88900000000000001,"5min":1.5680000000000001,"15min":3.1709999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68899999999999995,"15min":0.65500000000000003},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.46,"15min":3.3079999999999998},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68500000000000005,"15min":0.66900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.004,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.62,"5min":0.70099999999999996,"15min":0.68999999999999995},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":3.2629999999999999},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.73499999999999999,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":1.0660000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":3.6720000000000002,"15min":3.742},"last":0.68300000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64400000000000002,"15min":0.629},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":1.1599999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40600000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83399999999999996,"5min":1.075,"15min":4.7750000000000004},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.304,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65600000000000003,"15min":0.70099999999999996},"min":{"1min":0.39200000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90000000000000002,"5min":1.127,"15min":3.609},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88500000000000001,"5min":1.601,"15min":4.9669999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.77900000000000003,"15min":0.79900000000000004},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.238,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.78300000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574964,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50856,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320984,"statfs":{"total":96636764160,"available":96584687616,"internally_reserved":0,"allocated":1728512,"data_stored":755329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.77200000000000002},"min":{"1min":0.40899999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95399999999999996,"5min":1.498,"15min":38.005000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65500000000000003,"15min":0.751},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.749,"5min":1.577,"15min":21.376999999999999},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.66000000000000003,"15min":0.72799999999999998},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.877,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66800000000000004,"15min":0.69699999999999995},"min":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":0.98799999999999999,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.66300000000000003,"15min":0.748},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81699999999999995,"5min":1.524,"15min":17.268000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.69699999999999995,"15min":0.746},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91800000000000004,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.73599999999999999,"15min":1.103},"min":{"1min":0.47699999999999998,"5min":0.44,"15min":0.435},"max":{"1min":0.89600000000000002,"5min":2.8690000000000002,"15min":113.363},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71999999999999997,"15min":0.75800000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97599999999999998,"5min":2.359,"15min":4.1180000000000003},"last":0.86399999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.37,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90500000000000003,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.70599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88800000000000001,"5min":2.827,"15min":2.827},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68300000000000005,"15min":0.70099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86499999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.73599999999999999,"15min":0.75600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96199999999999997,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74299999999999999,"15min":0.76100000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94999999999999996,"5min":1.9379999999999999,"15min":2.02},"last":0.89000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574963,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59052,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312788,"statfs":{"total":96636764160,"available":96576294912,"internally_reserved":0,"allocated":1724416,"data_stored":755287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68200000000000005,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86899999999999999,"5min":1.056,"15min":1.0620000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71799999999999997,"15min":0.72899999999999998},"min":{"1min":0.55200000000000005,"5min":0.373,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":1.3400000000000001,"15min":1.341},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":0.86499999999999999,"5min":1.173,"15min":1.173},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65900000000000003,"15min":0.63200000000000001},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88700000000000001,"5min":0.92000000000000004,"15min":0.93999999999999995},"last":0.59499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.68000000000000005,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.495,"5min":0.44900000000000001,"15min":0.42199999999999999},"max":{"1min":0.91100000000000003,"5min":1.22,"15min":2.1259999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70199999999999996,"15min":0.67400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.97799999999999998,"5min":1.006,"15min":1.006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72599999999999998,"15min":0.69899999999999995},"min":{"1min":0.55700000000000005,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94099999999999995,"5min":1.232,"15min":1.232},"last":0.624}]},{"osd":5,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.97899999999999998,"5min":1.284,"15min":1.284},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.39700000000000002,"15min":0.30299999999999999},"max":{"1min":0.83699999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68899999999999995,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1.03,"5min":1.274,"15min":1.274},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":1.0169999999999999,"5min":1.292,"15min":1.413},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71799999999999997,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.018,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77200000000000002,"15min":0.73699999999999999},"min":{"1min":0.58199999999999996,"5min":0.44500000000000001,"15min":0.371},"max":{"1min":1.0700000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.875}]}]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77036,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94294804,"statfs":{"total":96636764160,"available":96557879296,"internally_reserved":0,"allocated":1728512,"data_stored":755377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77134279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.44700000000000001},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.622,"5min":1.177,"15min":1.9810000000000001},"last":0.436},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45800000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":2.96},"last":0.46000000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.29799999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68000000000000005,"5min":0.91900000000000004,"15min":3.5600000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.67100000000000004},"min":{"1min":0.41999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":1.0309999999999999,"15min":3.6829999999999998},"last":0.62}]},{"osd":2,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.38600000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69499999999999995,"5min":0.88,"15min":1.9590000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53800000000000003,"15min":0.58099999999999996},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73499999999999999,"5min":0.97699999999999998,"15min":1.3819999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72399999999999998,"15min":0.78300000000000003},"min":{"1min":0.42699999999999999,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.1040000000000001,"5min":1.1339999999999999,"15min":2.8610000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.74299999999999999,"15min":0.76700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.48199999999999998},"max":{"1min":0.91000000000000003,"5min":1.012,"15min":1.3640000000000001},"last":0.71399999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.746,"15min":0.77600000000000002},"min":{"1min":0.496,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.032,"5min":1.0580000000000001,"15min":1.391},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.746,"15min":0.76500000000000001},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.401},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76300000000000001,"15min":0.78600000000000003},"min":{"1min":0.53000000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":0.87}]},{"osd":7,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.755,"15min":0.82899999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.45600000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":2.899},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0489999999999999,"5min":1.075,"15min":1.4359999999999999},"last":0.96599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574964,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54952,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316888,"statfs":{"total":96636764160,"available":96580493312,"internally_reserved":0,"allocated":1724416,"data_stored":754814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83899999999999997,"5min":1.097,"15min":1.097},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.81699999999999995,"15min":0.56000000000000005},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28100000000000003},"max":{"1min":0.56000000000000005,"5min":30.023,"15min":30.023},"last":0.45400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":1.3460000000000001},"min":{"1min":0.33100000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.046,"15min":260.53399999999999},"last":0.441},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.75900000000000001,"15min":1.448},"min":{"1min":0.40799999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.65800000000000003,"5min":21.279,"15min":262.61399999999998},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.81799999999999995,"15min":0.59599999999999997},"min":{"1min":0.36399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59199999999999997,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.375},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.745,"5min":1.214,"15min":1.214},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.89400000000000002,"15min":0.80300000000000005},"min":{"1min":0.58599999999999997,"5min":0.495,"15min":0.49099999999999999},"max":{"1min":1.0209999999999999,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.018,"15min":0.85899999999999999},"min":{"1min":0.48799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0569999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.748}]},{"osd":5,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.0900000000000001,"15min":0.88700000000000001},"min":{"1min":0.54600000000000004,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":1.099,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78200000000000003,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0469999999999999,"5min":1.2789999999999999,"15min":1.569},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.90200000000000002,"15min":0.81100000000000005},"min":{"1min":0.53700000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0289999999999999,"5min":10.664,"15min":10.664},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.091,"15min":0.90000000000000002},"min":{"1min":0.57999999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0760000000000001,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.91000000000000003,"15min":0.82299999999999995},"min":{"1min":0.59999999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.99199999999999999,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.1240000000000001,"15min":0.90700000000000003},"min":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.59499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86876,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284964,"statfs":{"total":96636764160,"available":96547803136,"internally_reserved":0,"allocated":1720320,"data_stored":755217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.48999999999999999},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51900000000000002},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.67000000000000004,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.436}]},{"osd":2,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.62,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.73499999999999999,"5min":0.84499999999999997,"15min":16.382999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52700000000000002,"15min":0.60999999999999999},"min":{"1min":0.42299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76400000000000001,"5min":0.81499999999999995,"15min":18.318000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.76500000000000001,"15min":0.80500000000000005},"min":{"1min":0.496,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.159,"5min":1.165,"15min":6.1630000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78900000000000003,"15min":0.81299999999999994},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":1.149,"5min":1.329,"15min":1.78},"last":0.91100000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.6870000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.754,"15min":0.79600000000000004},"min":{"1min":0.53800000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.95199999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.749,"15min":0.79800000000000004},"min":{"1min":0.58199999999999996,"5min":0.56200000000000006,"15min":0.53200000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77500000000000002,"15min":0.81699999999999995},"min":{"1min":0.503,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.204,"5min":1.204,"15min":6.1970000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.76500000000000001,"15min":0.81200000000000006},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78900000000000003,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.53800000000000003},"max":{"1min":1.145,"5min":1.145,"15min":1.6359999999999999},"last":0.93200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63716,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94308124,"statfs":{"total":96636764160,"available":96571518976,"internally_reserved":0,"allocated":2310144,"data_stored":1344566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45500000000000002,"15min":0.61099999999999999},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.872,"5min":1.018,"15min":5.5259999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.629},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.27000000000000002},"max":{"1min":0.84699999999999998,"5min":3.004,"15min":5.4889999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.66200000000000003},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.79400000000000004,"5min":3.605,"15min":6.0430000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52400000000000002,"15min":0.66400000000000003},"min":{"1min":0.41999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.90500000000000003,"5min":1.29,"15min":4.6760000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51600000000000001,"15min":0.621},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.314},"max":{"1min":0.75700000000000001,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60199999999999998,"15min":0.68400000000000005},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88300000000000001,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78000000000000003,"15min":0.86399999999999999},"min":{"1min":0.56000000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.137,"5min":1.9490000000000001,"15min":4.734},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3109999999999999,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.79500000000000004,"15min":0.88700000000000001},"min":{"1min":0.47799999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1839999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79700000000000004,"15min":0.878},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.234,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78900000000000003,"15min":0.98299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.099,"5min":1.194,"15min":4.8470000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80000000000000004,"15min":0.89700000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.48599999999999999},"max":{"1min":1.2889999999999999,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.92500000000000004},"min":{"1min":0.59299999999999997,"5min":0.51100000000000001,"15min":0.47399999999999998},"max":{"1min":1.375,"5min":1.405,"15min":5.0540000000000003},"last":0.748},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.505},"max":{"1min":1.2649999999999999,"5min":2.706,"15min":4.9950000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:17:26.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:17:26.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.363+0000 7ff70de00640 1 Processor -- start 2024-09-13T17:17:26.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.363+0000 7ff70de00640 1 -- start start 2024-09-13T17:17:26.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.364+0000 7ff70de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.364+0000 7ff70de00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7080713a0 0x7ff70806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.364+0000 7ff70de00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7080b35c0 con 0x7ff70806cde0 2024-09-13T17:17:26.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.364+0000 7ff70de00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7081618f0 con 0x7ff7080713a0 2024-09-13T17:17:26.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.364+0000 7ff70de00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080b38a0 con 0x7ff70815e860 2024-09-13T17:17:26.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.365+0000 7ff706a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.365+0000 7ff707400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7080713a0 0x7ff70806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.365+0000 7ff706a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:37320/0 (socket says 172.21.15.28:37320) 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.365+0000 7ff706a00640 1 -- 172.21.15.28:0/1695049041 learned_addr learned my addr 172.21.15.28:0/1695049041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.365+0000 7ff707400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff7080713a0 0x7ff70806b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:51954/0 (socket says 172.21.15.28:51954) 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.366+0000 7ff707400640 1 -- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 msgr2=0x7ff708161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.366+0000 7ff707400640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.366+0000 7ff707400640 1 -- 172.21.15.28:0/1695049041 >> v1:172.21.15.189:6789/0 conn(0x7ff70806cde0 legacy=0x7ff708070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.366+0000 7ff707400640 1 -- 172.21.15.28:0/1695049041 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708161230 con 0x7ff7080713a0 2024-09-13T17:17:26.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.366+0000 7ff706a00640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.367+0000 7ff707400640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff70806b1c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff700009d90 tx=0x7ff70002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97775c0ddc9e2cf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.367+0000 7ff705600640 1 -- 172.21.15.28:0/1695049041 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700031600 con 0x7ff7080713a0 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.367+0000 7ff705600640 1 -- 172.21.15.28:0/1695049041 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70002d050 con 0x7ff7080713a0 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.367+0000 7ff705600640 1 -- 172.21.15.28:0/1695049041 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff700031980 con 0x7ff7080713a0 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 -- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 msgr2=0x7ff70806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff70806b1c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff700009d90 tx=0x7ff70002a5d0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 -- 172.21.15.28:0/1695049041 shutdown_connections 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff70806b1c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 --2- 172.21.15.28:0/1695049041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 -- 172.21.15.28:0/1695049041 >> 172.21.15.28:0/1695049041 conn(0x7ff708069e80 msgr2=0x7ff70806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 -- 172.21.15.28:0/1695049041 shutdown_connections 2024-09-13T17:17:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.368+0000 7ff70de00640 1 -- 172.21.15.28:0/1695049041 wait complete. 2024-09-13T17:17:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.369+0000 7ff70de00640 1 Processor -- start 2024-09-13T17:17:26.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.369+0000 7ff70de00640 1 -- start start 2024-09-13T17:17:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff7081466c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708157820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081582e0 con 0x7ff70806cde0 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081585c0 con 0x7ff7080713a0 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.370+0000 7ff70de00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081588a0 con 0x7ff70815e860 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45660/0 (socket says 172.21.15.28:45660) 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 -- 172.21.15.28:0/392372270 learned_addr learned my addr 172.21.15.28:0/392372270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706000640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff7081466c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff707400640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708157820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 -- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 msgr2=0x7ff708157820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708157820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 -- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 msgr2=0x7ff7081466c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff7081466c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 -- 172.21.15.28:0/392372270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80094d0 con 0x7ff70806cde0 2024-09-13T17:17:26.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.371+0000 7ff706a00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e9d0 tx=0x7ff6f800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.372+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f800ca60 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.372+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f800cbc0 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.372+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff700009a70 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.372+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f8013470 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.372+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff708158e60 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.373+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7080b31c0 con 0x7ff70806cde0 2024-09-13T17:17:26.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.374+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff6f80135d0 con 0x7ff70806cde0 2024-09-13T17:17:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.375+0000 7ff6ff400640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 0x7ff6e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.375+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6f8089680 con 0x7ff70806cde0 2024-09-13T17:17:26.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.375+0000 7ff706000640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 0x7ff6e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.376+0000 7ff706000640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 0x7ff6e806d0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff70814a140 tx=0x7ff6f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.379+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff6f8054870 con 0x7ff70806cde0 2024-09-13T17:17:26.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.557+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff708158730 con 0x7ff6e806ac00 2024-09-13T17:17:26.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.566+0000 7ff6ff400640 1 -- 172.21.15.28:0/392372270 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289092 (secure 0 0 0) 0x7ff708158730 con 0x7ff6e806ac00 2024-09-13T17:17:26.570 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:26.571 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 msgr2=0x7ff6e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 0x7ff6e806d0c0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7ff70814a140 tx=0x7ff6f4009040 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 msgr2=0x7ff708159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7ff6f800e9d0 tx=0x7ff6f800ee90 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 shutdown_connections 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff6e806ac00 0x7ff6e806d0c0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff70815e860 0x7ff708157820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff7080713a0 0x7ff7081466c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 --2- 172.21.15.28:0/392372270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff70806cde0 0x7ff708159380 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 >> 172.21.15.28:0/392372270 conn(0x7ff708069e80 msgr2=0x7ff7080a8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 shutdown_connections 2024-09-13T17:17:26.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.570+0000 7ff70de00640 1 -- 172.21.15.28:0/392372270 wait complete. 2024-09-13T17:17:26.587 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-13T17:17:25.025657+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":65,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":520984,"kb_used_data":14656,"kb_used_omap":12,"kb_used_meta":506163,"kb_avail":754453736,"statfs":{"total":773094113280,"available":772560625664,"internally_reserved":0,"allocated":15007744,"data_stored":7225158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":518311481},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.501692"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998994+0000","last_change":"2024-09-13T17:17:21.995206+0000","last_active":"2024-09-13T17:17:21.998994+0000","last_peered":"2024-09-13T17:17:21.998994+0000","last_clean":"2024-09-13T17:17:21.998994+0000","last_became_active":"2024-09-13T17:17:21.992692+0000","last_became_peered":"2024-09-13T17:17:21.992692+0000","last_unstale":"2024-09-13T17:17:21.998994+0000","last_undegraded":"2024-09-13T17:17:21.998994+0000","last_fullsized":"2024-09-13T17:17:21.998994+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:16.168759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986353+0000","last_change":"2024-09-13T17:17:21.980448+0000","last_active":"2024-09-13T17:17:21.986353+0000","last_peered":"2024-09-13T17:17:21.986353+0000","last_clean":"2024-09-13T17:17:21.986353+0000","last_became_active":"2024-09-13T17:17:21.979916+0000","last_became_peered":"2024-09-13T17:17:21.979916+0000","last_unstale":"2024-09-13T17:17:21.986353+0000","last_undegraded":"2024-09-13T17:17:21.986353+0000","last_fullsized":"2024-09-13T17:17:21.986353+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:29.054388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993373+0000","last_change":"2024-09-13T17:17:21.989525+0000","last_active":"2024-09-13T17:17:21.993373+0000","last_peered":"2024-09-13T17:17:21.993373+0000","last_clean":"2024-09-13T17:17:21.993373+0000","last_became_active":"2024-09-13T17:17:21.988738+0000","last_became_peered":"2024-09-13T17:17:21.988738+0000","last_unstale":"2024-09-13T17:17:21.993373+0000","last_undegraded":"2024-09-13T17:17:21.993373+0000","last_fullsized":"2024-09-13T17:17:21.993373+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003316+0000","last_change":"2024-09-13T17:17:22.002101+0000","last_active":"2024-09-13T17:17:22.003316+0000","last_peered":"2024-09-13T17:17:22.003316+0000","last_clean":"2024-09-13T17:17:22.003316+0000","last_became_active":"2024-09-13T17:17:22.001760+0000","last_became_peered":"2024-09-13T17:17:22.001760+0000","last_unstale":"2024-09-13T17:17:22.003316+0000","last_undegraded":"2024-09-13T17:17:22.003316+0000","last_fullsized":"2024-09-13T17:17:22.003316+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:03.588912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998814+0000","last_change":"2024-09-13T17:17:21.991237+0000","last_active":"2024-09-13T17:17:21.998814+0000","last_peered":"2024-09-13T17:17:21.998814+0000","last_clean":"2024-09-13T17:17:21.998814+0000","last_became_active":"2024-09-13T17:17:21.990966+0000","last_became_peered":"2024-09-13T17:17:21.990966+0000","last_unstale":"2024-09-13T17:17:21.998814+0000","last_undegraded":"2024-09-13T17:17:21.998814+0000","last_fullsized":"2024-09-13T17:17:21.998814+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:38.877759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.994825+0000","last_change":"2024-09-13T17:17:21.993207+0000","last_active":"2024-09-13T17:17:21.994825+0000","last_peered":"2024-09-13T17:17:21.994825+0000","last_clean":"2024-09-13T17:17:21.994825+0000","last_became_active":"2024-09-13T17:17:21.992913+0000","last_became_peered":"2024-09-13T17:17:21.992913+0000","last_unstale":"2024-09-13T17:17:21.994825+0000","last_undegraded":"2024-09-13T17:17:21.994825+0000","last_fullsized":"2024-09-13T17:17:21.994825+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:08.071748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.988935+0000","last_change":"2024-09-13T17:17:21.985479+0000","last_active":"2024-09-13T17:17:21.988935+0000","last_peered":"2024-09-13T17:17:21.988935+0000","last_clean":"2024-09-13T17:17:21.988935+0000","last_became_active":"2024-09-13T17:17:21.984330+0000","last_became_peered":"2024-09-13T17:17:21.984330+0000","last_unstale":"2024-09-13T17:17:21.988935+0000","last_undegraded":"2024-09-13T17:17:21.988935+0000","last_fullsized":"2024-09-13T17:17:21.988935+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:30.344049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999313+0000","last_change":"2024-09-13T17:17:21.992490+0000","last_active":"2024-09-13T17:17:21.999313+0000","last_peered":"2024-09-13T17:17:21.999313+0000","last_clean":"2024-09-13T17:17:21.999313+0000","last_became_active":"2024-09-13T17:17:21.984503+0000","last_became_peered":"2024-09-13T17:17:21.984503+0000","last_unstale":"2024-09-13T17:17:21.999313+0000","last_undegraded":"2024-09-13T17:17:21.999313+0000","last_fullsized":"2024-09-13T17:17:21.999313+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:21.939438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989127+0000","last_change":"2024-09-13T17:17:21.982123+0000","last_active":"2024-09-13T17:17:21.989127+0000","last_peered":"2024-09-13T17:17:21.989127+0000","last_clean":"2024-09-13T17:17:21.989127+0000","last_became_active":"2024-09-13T17:17:21.981844+0000","last_became_peered":"2024-09-13T17:17:21.981844+0000","last_unstale":"2024-09-13T17:17:21.989127+0000","last_undegraded":"2024-09-13T17:17:21.989127+0000","last_fullsized":"2024-09-13T17:17:21.989127+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.260215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986335+0000","last_change":"2024-09-13T17:17:21.980215+0000","last_active":"2024-09-13T17:17:21.986335+0000","last_peered":"2024-09-13T17:17:21.986335+0000","last_clean":"2024-09-13T17:17:21.986335+0000","last_became_active":"2024-09-13T17:17:21.979596+0000","last_became_peered":"2024-09-13T17:17:21.979596+0000","last_unstale":"2024-09-13T17:17:21.986335+0000","last_undegraded":"2024-09-13T17:17:21.986335+0000","last_fullsized":"2024-09-13T17:17:21.986335+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:06.193722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990415+0000","last_change":"2024-09-13T17:17:21.981341+0000","last_active":"2024-09-13T17:17:21.990415+0000","last_peered":"2024-09-13T17:17:21.990415+0000","last_clean":"2024-09-13T17:17:21.990415+0000","last_became_active":"2024-09-13T17:17:21.980804+0000","last_became_peered":"2024-09-13T17:17:21.980804+0000","last_unstale":"2024-09-13T17:17:21.990415+0000","last_undegraded":"2024-09-13T17:17:21.990415+0000","last_fullsized":"2024-09-13T17:17:21.990415+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:15:05.926821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999072+0000","last_change":"2024-09-13T17:17:21.992131+0000","last_active":"2024-09-13T17:17:21.999072+0000","last_peered":"2024-09-13T17:17:21.999072+0000","last_clean":"2024-09-13T17:17:21.999072+0000","last_became_active":"2024-09-13T17:17:21.991875+0000","last_became_peered":"2024-09-13T17:17:21.991875+0000","last_unstale":"2024-09-13T17:17:21.999072+0000","last_undegraded":"2024-09-13T17:17:21.999072+0000","last_fullsized":"2024-09-13T17:17:21.999072+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:29.054388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995122+0000","last_change":"2024-09-13T17:17:21.990558+0000","last_active":"2024-09-13T17:17:21.995122+0000","last_peered":"2024-09-13T17:17:21.995122+0000","last_clean":"2024-09-13T17:17:21.995122+0000","last_became_active":"2024-09-13T17:17:21.990182+0000","last_became_peered":"2024-09-13T17:17:21.990182+0000","last_unstale":"2024-09-13T17:17:21.995122+0000","last_undegraded":"2024-09-13T17:17:21.995122+0000","last_fullsized":"2024-09-13T17:17:21.995122+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:09.727162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.992960+0000","last_change":"2024-09-13T17:17:21.991486+0000","last_active":"2024-09-13T17:17:21.992960+0000","last_peered":"2024-09-13T17:17:21.992960+0000","last_clean":"2024-09-13T17:17:21.992960+0000","last_became_active":"2024-09-13T17:17:21.989663+0000","last_became_peered":"2024-09-13T17:17:21.989663+0000","last_unstale":"2024-09-13T17:17:21.992960+0000","last_undegraded":"2024-09-13T17:17:21.992960+0000","last_fullsized":"2024-09-13T17:17:21.992960+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:28.080818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995497+0000","last_change":"2024-09-13T17:17:21.992219+0000","last_active":"2024-09-13T17:17:21.995497+0000","last_peered":"2024-09-13T17:17:21.995497+0000","last_clean":"2024-09-13T17:17:21.995497+0000","last_became_active":"2024-09-13T17:17:21.992000+0000","last_became_peered":"2024-09-13T17:17:21.992000+0000","last_unstale":"2024-09-13T17:17:21.995497+0000","last_undegraded":"2024-09-13T17:17:21.995497+0000","last_fullsized":"2024-09-13T17:17:21.995497+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:25.395189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996927+0000","last_change":"2024-09-13T17:17:21.990853+0000","last_active":"2024-09-13T17:17:21.996927+0000","last_peered":"2024-09-13T17:17:21.996927+0000","last_clean":"2024-09-13T17:17:21.996927+0000","last_became_active":"2024-09-13T17:17:21.990625+0000","last_became_peered":"2024-09-13T17:17:21.990625+0000","last_unstale":"2024-09-13T17:17:21.996927+0000","last_undegraded":"2024-09-13T17:17:21.996927+0000","last_fullsized":"2024-09-13T17:17:21.996927+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:09.718296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979551+0000","last_change":"2024-09-13T17:17:21.978654+0000","last_active":"2024-09-13T17:17:21.979551+0000","last_peered":"2024-09-13T17:17:21.979551+0000","last_clean":"2024-09-13T17:17:21.979551+0000","last_became_active":"2024-09-13T17:17:21.976431+0000","last_became_peered":"2024-09-13T17:17:21.976431+0000","last_unstale":"2024-09-13T17:17:21.979551+0000","last_undegraded":"2024-09-13T17:17:21.979551+0000","last_fullsized":"2024-09-13T17:17:21.979551+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:40.508854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986571+0000","last_change":"2024-09-13T17:17:21.984409+0000","last_active":"2024-09-13T17:17:21.986571+0000","last_peered":"2024-09-13T17:17:21.986571+0000","last_clean":"2024-09-13T17:17:21.986571+0000","last_became_active":"2024-09-13T17:17:21.984156+0000","last_became_peered":"2024-09-13T17:17:21.984156+0000","last_unstale":"2024-09-13T17:17:21.986571+0000","last_undegraded":"2024-09-13T17:17:21.986571+0000","last_fullsized":"2024-09-13T17:17:21.986571+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:54.197007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.991645+0000","last_change":"2024-09-13T17:17:21.981523+0000","last_active":"2024-09-13T17:17:21.991645+0000","last_peered":"2024-09-13T17:17:21.991645+0000","last_clean":"2024-09-13T17:17:21.991645+0000","last_became_active":"2024-09-13T17:17:21.981112+0000","last_became_peered":"2024-09-13T17:17:21.981112+0000","last_unstale":"2024-09-13T17:17:21.991645+0000","last_undegraded":"2024-09-13T17:17:21.991645+0000","last_fullsized":"2024-09-13T17:17:21.991645+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:56.819736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993701+0000","last_change":"2024-09-13T17:17:21.989435+0000","last_active":"2024-09-13T17:17:21.993701+0000","last_peered":"2024-09-13T17:17:21.993701+0000","last_clean":"2024-09-13T17:17:21.993701+0000","last_became_active":"2024-09-13T17:17:21.986372+0000","last_became_peered":"2024-09-13T17:17:21.986372+0000","last_unstale":"2024-09-13T17:17:21.993701+0000","last_undegraded":"2024-09-13T17:17:21.993701+0000","last_fullsized":"2024-09-13T17:17:21.993701+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:00.820343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999599+0000","last_change":"2024-09-13T17:17:21.995595+0000","last_active":"2024-09-13T17:17:21.999599+0000","last_peered":"2024-09-13T17:17:21.999599+0000","last_clean":"2024-09-13T17:17:21.999599+0000","last_became_active":"2024-09-13T17:17:21.994943+0000","last_became_peered":"2024-09-13T17:17:21.994943+0000","last_unstale":"2024-09-13T17:17:21.999599+0000","last_undegraded":"2024-09-13T17:17:21.999599+0000","last_fullsized":"2024-09-13T17:17:21.999599+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:18.957602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989173+0000","last_change":"2024-09-13T17:17:21.980473+0000","last_active":"2024-09-13T17:17:21.989173+0000","last_peered":"2024-09-13T17:17:21.989173+0000","last_clean":"2024-09-13T17:17:21.989173+0000","last_became_active":"2024-09-13T17:17:21.980201+0000","last_became_peered":"2024-09-13T17:17:21.980201+0000","last_unstale":"2024-09-13T17:17:21.989173+0000","last_undegraded":"2024-09-13T17:17:21.989173+0000","last_fullsized":"2024-09-13T17:17:21.989173+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:52:20.546092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000533+0000","last_change":"2024-09-13T17:17:21.995917+0000","last_active":"2024-09-13T17:17:22.000533+0000","last_peered":"2024-09-13T17:17:22.000533+0000","last_clean":"2024-09-13T17:17:22.000533+0000","last_became_active":"2024-09-13T17:17:21.990973+0000","last_became_peered":"2024-09-13T17:17:21.990973+0000","last_unstale":"2024-09-13T17:17:22.000533+0000","last_undegraded":"2024-09-13T17:17:22.000533+0000","last_fullsized":"2024-09-13T17:17:22.000533+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:40.178733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989995+0000","last_change":"2024-09-13T17:17:21.980281+0000","last_active":"2024-09-13T17:17:21.989995+0000","last_peered":"2024-09-13T17:17:21.989995+0000","last_clean":"2024-09-13T17:17:21.989995+0000","last_became_active":"2024-09-13T17:17:21.979062+0000","last_became_peered":"2024-09-13T17:17:21.979062+0000","last_unstale":"2024-09-13T17:17:21.989995+0000","last_undegraded":"2024-09-13T17:17:21.989995+0000","last_fullsized":"2024-09-13T17:17:21.989995+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:16:16.743527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990281+0000","last_change":"2024-09-13T17:17:21.988901+0000","last_active":"2024-09-13T17:17:21.990281+0000","last_peered":"2024-09-13T17:17:21.990281+0000","last_clean":"2024-09-13T17:17:21.990281+0000","last_became_active":"2024-09-13T17:17:21.988626+0000","last_became_peered":"2024-09-13T17:17:21.988626+0000","last_unstale":"2024-09-13T17:17:21.990281+0000","last_undegraded":"2024-09-13T17:17:21.990281+0000","last_fullsized":"2024-09-13T17:17:21.990281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:40.178733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998287+0000","last_change":"2024-09-13T17:17:21.992354+0000","last_active":"2024-09-13T17:17:21.998287+0000","last_peered":"2024-09-13T17:17:21.998287+0000","last_clean":"2024-09-13T17:17:21.998287+0000","last_became_active":"2024-09-13T17:17:21.992100+0000","last_became_peered":"2024-09-13T17:17:21.992100+0000","last_unstale":"2024-09-13T17:17:21.998287+0000","last_undegraded":"2024-09-13T17:17:21.998287+0000","last_fullsized":"2024-09-13T17:17:21.998287+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:40.423842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.991100+0000","last_change":"2024-09-13T17:17:21.986059+0000","last_active":"2024-09-13T17:17:21.991100+0000","last_peered":"2024-09-13T17:17:21.991100+0000","last_clean":"2024-09-13T17:17:21.991100+0000","last_became_active":"2024-09-13T17:17:21.985807+0000","last_became_peered":"2024-09-13T17:17:21.985807+0000","last_unstale":"2024-09-13T17:17:21.991100+0000","last_undegraded":"2024-09-13T17:17:21.991100+0000","last_fullsized":"2024-09-13T17:17:21.991100+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:08.522356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986294+0000","last_change":"2024-09-13T17:17:21.978805+0000","last_active":"2024-09-13T17:17:21.986294+0000","last_peered":"2024-09-13T17:17:21.986294+0000","last_clean":"2024-09-13T17:17:21.986294+0000","last_became_active":"2024-09-13T17:17:21.978423+0000","last_became_peered":"2024-09-13T17:17:21.978423+0000","last_unstale":"2024-09-13T17:17:21.986294+0000","last_undegraded":"2024-09-13T17:17:21.986294+0000","last_fullsized":"2024-09-13T17:17:21.986294+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:11.997263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000505+0000","last_change":"2024-09-13T17:17:21.999177+0000","last_active":"2024-09-13T17:17:22.000505+0000","last_peered":"2024-09-13T17:17:22.000505+0000","last_clean":"2024-09-13T17:17:22.000505+0000","last_became_active":"2024-09-13T17:17:21.995626+0000","last_became_peered":"2024-09-13T17:17:21.995626+0000","last_unstale":"2024-09-13T17:17:22.000505+0000","last_undegraded":"2024-09-13T17:17:22.000505+0000","last_fullsized":"2024-09-13T17:17:22.000505+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:57.085899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998332+0000","last_change":"2024-09-13T17:17:21.995291+0000","last_active":"2024-09-13T17:17:21.998332+0000","last_peered":"2024-09-13T17:17:21.998332+0000","last_clean":"2024-09-13T17:17:21.998332+0000","last_became_active":"2024-09-13T17:17:21.995012+0000","last_became_peered":"2024-09-13T17:17:21.995012+0000","last_unstale":"2024-09-13T17:17:21.998332+0000","last_undegraded":"2024-09-13T17:17:21.998332+0000","last_fullsized":"2024-09-13T17:17:21.998332+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:56.433707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999172+0000","last_change":"2024-09-13T17:17:21.995333+0000","last_active":"2024-09-13T17:17:21.999172+0000","last_peered":"2024-09-13T17:17:21.999172+0000","last_clean":"2024-09-13T17:17:21.999172+0000","last_became_active":"2024-09-13T17:17:21.994969+0000","last_became_peered":"2024-09-13T17:17:21.994969+0000","last_unstale":"2024-09-13T17:17:21.999172+0000","last_undegraded":"2024-09-13T17:17:21.999172+0000","last_fullsized":"2024-09-13T17:17:21.999172+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:52.663993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989407+0000","last_change":"2024-09-13T17:17:21.986032+0000","last_active":"2024-09-13T17:17:21.989407+0000","last_peered":"2024-09-13T17:17:21.989407+0000","last_clean":"2024-09-13T17:17:21.989407+0000","last_became_active":"2024-09-13T17:17:21.984686+0000","last_became_peered":"2024-09-13T17:17:21.984686+0000","last_unstale":"2024-09-13T17:17:21.989407+0000","last_undegraded":"2024-09-13T17:17:21.989407+0000","last_fullsized":"2024-09-13T17:17:21.989407+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:42.503575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995690+0000","last_change":"2024-09-13T17:17:21.990012+0000","last_active":"2024-09-13T17:17:21.995690+0000","last_peered":"2024-09-13T17:17:21.995690+0000","last_clean":"2024-09-13T17:17:21.995690+0000","last_became_active":"2024-09-13T17:17:21.989769+0000","last_became_peered":"2024-09-13T17:17:21.989769+0000","last_unstale":"2024-09-13T17:17:21.995690+0000","last_undegraded":"2024-09-13T17:17:21.995690+0000","last_fullsized":"2024-09-13T17:17:21.995690+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:14:09.420840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"138'4","reported_seq":14,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003235+0000","last_change":"2024-09-13T17:17:21.559595+0000","last_active":"2024-09-13T17:17:22.003235+0000","last_peered":"2024-09-13T17:17:22.003235+0000","last_clean":"2024-09-13T17:17:22.003235+0000","last_became_active":"2024-09-13T17:17:21.559290+0000","last_became_peered":"2024-09-13T17:17:21.559290+0000","last_unstale":"2024-09-13T17:17:22.003235+0000","last_undegraded":"2024-09-13T17:17:22.003235+0000","last_fullsized":"2024-09-13T17:17:22.003235+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:49.726171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998981+0000","last_change":"2024-09-13T17:17:21.992307+0000","last_active":"2024-09-13T17:17:21.998981+0000","last_peered":"2024-09-13T17:17:21.998981+0000","last_clean":"2024-09-13T17:17:21.998981+0000","last_became_active":"2024-09-13T17:17:21.992062+0000","last_became_peered":"2024-09-13T17:17:21.992062+0000","last_unstale":"2024-09-13T17:17:21.998981+0000","last_undegraded":"2024-09-13T17:17:21.998981+0000","last_fullsized":"2024-09-13T17:17:21.998981+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:12.922709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997347+0000","last_change":"2024-09-13T17:17:21.545483+0000","last_active":"2024-09-13T17:17:21.997347+0000","last_peered":"2024-09-13T17:17:21.997347+0000","last_clean":"2024-09-13T17:17:21.997347+0000","last_became_active":"2024-09-13T17:17:21.544889+0000","last_became_peered":"2024-09-13T17:17:21.544889+0000","last_unstale":"2024-09-13T17:17:21.997347+0000","last_undegraded":"2024-09-13T17:17:21.997347+0000","last_fullsized":"2024-09-13T17:17:21.997347+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:02.473285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000566+0000","last_change":"2024-09-13T17:17:21.998528+0000","last_active":"2024-09-13T17:17:22.000566+0000","last_peered":"2024-09-13T17:17:22.000566+0000","last_clean":"2024-09-13T17:17:22.000566+0000","last_became_active":"2024-09-13T17:17:21.991185+0000","last_became_peered":"2024-09-13T17:17:21.991185+0000","last_unstale":"2024-09-13T17:17:22.000566+0000","last_undegraded":"2024-09-13T17:17:22.000566+0000","last_fullsized":"2024-09-13T17:17:22.000566+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:24:52.661143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999228+0000","last_change":"2024-09-13T17:17:21.552752+0000","last_active":"2024-09-13T17:17:21.999228+0000","last_peered":"2024-09-13T17:17:21.999228+0000","last_clean":"2024-09-13T17:17:21.999228+0000","last_became_active":"2024-09-13T17:17:21.552419+0000","last_became_peered":"2024-09-13T17:17:21.552419+0000","last_unstale":"2024-09-13T17:17:21.999228+0000","last_undegraded":"2024-09-13T17:17:21.999228+0000","last_fullsized":"2024-09-13T17:17:21.999228+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:08.710863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999726+0000","last_change":"2024-09-13T17:17:21.989609+0000","last_active":"2024-09-13T17:17:21.999726+0000","last_peered":"2024-09-13T17:17:21.999726+0000","last_clean":"2024-09-13T17:17:21.999726+0000","last_became_active":"2024-09-13T17:17:21.989276+0000","last_became_peered":"2024-09-13T17:17:21.989276+0000","last_unstale":"2024-09-13T17:17:21.999726+0000","last_undegraded":"2024-09-13T17:17:21.999726+0000","last_fullsized":"2024-09-13T17:17:21.999726+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:02.424600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996090+0000","last_change":"2024-09-13T17:17:21.557156+0000","last_active":"2024-09-13T17:17:21.996090+0000","last_peered":"2024-09-13T17:17:21.996090+0000","last_clean":"2024-09-13T17:17:21.996090+0000","last_became_active":"2024-09-13T17:17:21.556412+0000","last_became_peered":"2024-09-13T17:17:21.556412+0000","last_unstale":"2024-09-13T17:17:21.996090+0000","last_undegraded":"2024-09-13T17:17:21.996090+0000","last_fullsized":"2024-09-13T17:17:21.996090+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:15.161326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.982382+0000","last_change":"2024-09-13T17:17:21.981075+0000","last_active":"2024-09-13T17:17:21.982382+0000","last_peered":"2024-09-13T17:17:21.982382+0000","last_clean":"2024-09-13T17:17:21.982382+0000","last_became_active":"2024-09-13T17:17:21.980813+0000","last_became_peered":"2024-09-13T17:17:21.980813+0000","last_unstale":"2024-09-13T17:17:21.982382+0000","last_undegraded":"2024-09-13T17:17:21.982382+0000","last_fullsized":"2024-09-13T17:17:21.982382+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:48.291495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.995429+0000","last_change":"2024-09-13T17:17:21.554549+0000","last_active":"2024-09-13T17:17:21.995429+0000","last_peered":"2024-09-13T17:17:21.995429+0000","last_clean":"2024-09-13T17:17:21.995429+0000","last_became_active":"2024-09-13T17:17:21.554075+0000","last_became_peered":"2024-09-13T17:17:21.554075+0000","last_unstale":"2024-09-13T17:17:21.995429+0000","last_undegraded":"2024-09-13T17:17:21.995429+0000","last_fullsized":"2024-09-13T17:17:21.995429+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:07.514923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979202+0000","last_change":"2024-09-13T17:17:21.977138+0000","last_active":"2024-09-13T17:17:21.979202+0000","last_peered":"2024-09-13T17:17:21.979202+0000","last_clean":"2024-09-13T17:17:21.979202+0000","last_became_active":"2024-09-13T17:17:21.976844+0000","last_became_peered":"2024-09-13T17:17:21.976844+0000","last_unstale":"2024-09-13T17:17:21.979202+0000","last_undegraded":"2024-09-13T17:17:21.979202+0000","last_fullsized":"2024-09-13T17:17:21.979202+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:50.034777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997518+0000","last_change":"2024-09-13T17:17:21.545304+0000","last_active":"2024-09-13T17:17:21.997518+0000","last_peered":"2024-09-13T17:17:21.997518+0000","last_clean":"2024-09-13T17:17:21.997518+0000","last_became_active":"2024-09-13T17:17:21.545005+0000","last_became_peered":"2024-09-13T17:17:21.545005+0000","last_unstale":"2024-09-13T17:17:21.997518+0000","last_undegraded":"2024-09-13T17:17:21.997518+0000","last_fullsized":"2024-09-13T17:17:21.997518+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:44:41.496143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993727+0000","last_change":"2024-09-13T17:17:21.985724+0000","last_active":"2024-09-13T17:17:21.993727+0000","last_peered":"2024-09-13T17:17:21.993727+0000","last_clean":"2024-09-13T17:17:21.993727+0000","last_became_active":"2024-09-13T17:17:21.984151+0000","last_became_peered":"2024-09-13T17:17:21.984151+0000","last_unstale":"2024-09-13T17:17:21.993727+0000","last_undegraded":"2024-09-13T17:17:21.993727+0000","last_fullsized":"2024-09-13T17:17:21.993727+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:04.216711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.972069+0000","last_change":"2024-09-13T17:17:21.550873+0000","last_active":"2024-09-13T17:17:21.972069+0000","last_peered":"2024-09-13T17:17:21.972069+0000","last_clean":"2024-09-13T17:17:21.972069+0000","last_became_active":"2024-09-13T17:17:21.550463+0000","last_became_peered":"2024-09-13T17:17:21.550463+0000","last_unstale":"2024-09-13T17:17:21.972069+0000","last_undegraded":"2024-09-13T17:17:21.972069+0000","last_fullsized":"2024-09-13T17:17:21.972069+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:37.870327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.983624+0000","last_change":"2024-09-13T17:17:21.978063+0000","last_active":"2024-09-13T17:17:21.983624+0000","last_peered":"2024-09-13T17:17:21.983624+0000","last_clean":"2024-09-13T17:17:21.983624+0000","last_became_active":"2024-09-13T17:17:21.977839+0000","last_became_peered":"2024-09-13T17:17:21.977839+0000","last_unstale":"2024-09-13T17:17:21.983624+0000","last_undegraded":"2024-09-13T17:17:21.983624+0000","last_fullsized":"2024-09-13T17:17:21.983624+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:41:25.926767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.968718+0000","last_change":"2024-09-13T17:17:21.550860+0000","last_active":"2024-09-13T17:17:21.968718+0000","last_peered":"2024-09-13T17:17:21.968718+0000","last_clean":"2024-09-13T17:17:21.968718+0000","last_became_active":"2024-09-13T17:17:21.550574+0000","last_became_peered":"2024-09-13T17:17:21.550574+0000","last_unstale":"2024-09-13T17:17:21.968718+0000","last_undegraded":"2024-09-13T17:17:21.968718+0000","last_fullsized":"2024-09-13T17:17:21.968718+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:35.620810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000521+0000","last_change":"2024-09-13T17:17:21.994502+0000","last_active":"2024-09-13T17:17:22.000521+0000","last_peered":"2024-09-13T17:17:22.000521+0000","last_clean":"2024-09-13T17:17:22.000521+0000","last_became_active":"2024-09-13T17:17:21.994234+0000","last_became_peered":"2024-09-13T17:17:21.994234+0000","last_unstale":"2024-09-13T17:17:22.000521+0000","last_undegraded":"2024-09-13T17:17:22.000521+0000","last_fullsized":"2024-09-13T17:17:22.000521+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:28.231040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.971688+0000","last_change":"2024-09-13T17:17:21.551427+0000","last_active":"2024-09-13T17:17:21.971688+0000","last_peered":"2024-09-13T17:17:21.971688+0000","last_clean":"2024-09-13T17:17:21.971688+0000","last_became_active":"2024-09-13T17:17:21.551124+0000","last_became_peered":"2024-09-13T17:17:21.551124+0000","last_unstale":"2024-09-13T17:17:21.971688+0000","last_undegraded":"2024-09-13T17:17:21.971688+0000","last_fullsized":"2024-09-13T17:17:21.971688+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:05.186289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989485+0000","last_change":"2024-09-13T17:17:21.987309+0000","last_active":"2024-09-13T17:17:21.989485+0000","last_peered":"2024-09-13T17:17:21.989485+0000","last_clean":"2024-09-13T17:17:21.989485+0000","last_became_active":"2024-09-13T17:17:21.986955+0000","last_became_peered":"2024-09-13T17:17:21.986955+0000","last_unstale":"2024-09-13T17:17:21.989485+0000","last_undegraded":"2024-09-13T17:17:21.989485+0000","last_fullsized":"2024-09-13T17:17:21.989485+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:34.795107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.982601+0000","last_change":"2024-09-13T17:17:21.556146+0000","last_active":"2024-09-13T17:17:21.982601+0000","last_peered":"2024-09-13T17:17:21.982601+0000","last_clean":"2024-09-13T17:17:21.982601+0000","last_became_active":"2024-09-13T17:17:21.555157+0000","last_became_peered":"2024-09-13T17:17:21.555157+0000","last_unstale":"2024-09-13T17:17:21.982601+0000","last_undegraded":"2024-09-13T17:17:21.982601+0000","last_fullsized":"2024-09-13T17:17:21.982601+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:51.656560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999604+0000","last_change":"2024-09-13T17:17:21.994983+0000","last_active":"2024-09-13T17:17:21.999604+0000","last_peered":"2024-09-13T17:17:21.999604+0000","last_clean":"2024-09-13T17:17:21.999604+0000","last_became_active":"2024-09-13T17:17:21.990951+0000","last_became_peered":"2024-09-13T17:17:21.990951+0000","last_unstale":"2024-09-13T17:17:21.999604+0000","last_undegraded":"2024-09-13T17:17:21.999604+0000","last_fullsized":"2024-09-13T17:17:21.999604+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:01.023328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"138'3","reported_seq":13,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999692+0000","last_change":"2024-09-13T17:17:21.547608+0000","last_active":"2024-09-13T17:17:21.999692+0000","last_peered":"2024-09-13T17:17:21.999692+0000","last_clean":"2024-09-13T17:17:21.999692+0000","last_became_active":"2024-09-13T17:17:21.547135+0000","last_became_peered":"2024-09-13T17:17:21.547135+0000","last_unstale":"2024-09-13T17:17:21.999692+0000","last_undegraded":"2024-09-13T17:17:21.999692+0000","last_fullsized":"2024-09-13T17:17:21.999692+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:16.781802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999744+0000","last_change":"2024-09-13T17:17:21.995654+0000","last_active":"2024-09-13T17:17:21.999744+0000","last_peered":"2024-09-13T17:17:21.999744+0000","last_clean":"2024-09-13T17:17:21.999744+0000","last_became_active":"2024-09-13T17:17:21.995413+0000","last_became_peered":"2024-09-13T17:17:21.995413+0000","last_unstale":"2024-09-13T17:17:21.999744+0000","last_undegraded":"2024-09-13T17:17:21.999744+0000","last_fullsized":"2024-09-13T17:17:21.999744+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:10.706481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"138'6","reported_seq":16,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.003483+0000","last_change":"2024-09-13T17:17:21.553589+0000","last_active":"2024-09-13T17:17:22.003483+0000","last_peered":"2024-09-13T17:17:22.003483+0000","last_clean":"2024-09-13T17:17:22.003483+0000","last_became_active":"2024-09-13T17:17:21.553037+0000","last_became_peered":"2024-09-13T17:17:21.553037+0000","last_unstale":"2024-09-13T17:17:22.003483+0000","last_undegraded":"2024-09-13T17:17:22.003483+0000","last_fullsized":"2024-09-13T17:17:22.003483+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:45:26.144900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.992433+0000","last_change":"2024-09-13T17:17:21.987385+0000","last_active":"2024-09-13T17:17:21.992433+0000","last_peered":"2024-09-13T17:17:21.992433+0000","last_clean":"2024-09-13T17:17:21.992433+0000","last_became_active":"2024-09-13T17:17:21.987057+0000","last_became_peered":"2024-09-13T17:17:21.987057+0000","last_unstale":"2024-09-13T17:17:21.992433+0000","last_undegraded":"2024-09-13T17:17:21.992433+0000","last_fullsized":"2024-09-13T17:17:21.992433+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:06:03.556535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.975872+0000","last_change":"2024-09-13T17:17:21.548772+0000","last_active":"2024-09-13T17:17:21.975872+0000","last_peered":"2024-09-13T17:17:21.975872+0000","last_clean":"2024-09-13T17:17:21.975872+0000","last_became_active":"2024-09-13T17:17:21.548511+0000","last_became_peered":"2024-09-13T17:17:21.548511+0000","last_unstale":"2024-09-13T17:17:21.975872+0000","last_undegraded":"2024-09-13T17:17:21.975872+0000","last_fullsized":"2024-09-13T17:17:21.975872+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:00.014392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990276+0000","last_change":"2024-09-13T17:17:21.988770+0000","last_active":"2024-09-13T17:17:21.990276+0000","last_peered":"2024-09-13T17:17:21.990276+0000","last_clean":"2024-09-13T17:17:21.990276+0000","last_became_active":"2024-09-13T17:17:21.988500+0000","last_became_peered":"2024-09-13T17:17:21.988500+0000","last_unstale":"2024-09-13T17:17:21.990276+0000","last_undegraded":"2024-09-13T17:17:21.990276+0000","last_fullsized":"2024-09-13T17:17:21.990276+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:32.527360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.969717+0000","last_change":"2024-09-13T17:17:21.554545+0000","last_active":"2024-09-13T17:17:21.969717+0000","last_peered":"2024-09-13T17:17:21.969717+0000","last_clean":"2024-09-13T17:17:21.969717+0000","last_became_active":"2024-09-13T17:17:21.554207+0000","last_became_peered":"2024-09-13T17:17:21.554207+0000","last_unstale":"2024-09-13T17:17:21.969717+0000","last_undegraded":"2024-09-13T17:17:21.969717+0000","last_fullsized":"2024-09-13T17:17:21.969717+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:29.221751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000650+0000","last_change":"2024-09-13T17:17:21.992018+0000","last_active":"2024-09-13T17:17:22.000650+0000","last_peered":"2024-09-13T17:17:22.000650+0000","last_clean":"2024-09-13T17:17:22.000650+0000","last_became_active":"2024-09-13T17:17:21.991668+0000","last_became_peered":"2024-09-13T17:17:21.991668+0000","last_unstale":"2024-09-13T17:17:22.000650+0000","last_undegraded":"2024-09-13T17:17:22.000650+0000","last_fullsized":"2024-09-13T17:17:22.000650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:34.795107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"138'8","reported_seq":18,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.001812+0000","last_change":"2024-09-13T17:17:21.550775+0000","last_active":"2024-09-13T17:17:22.001812+0000","last_peered":"2024-09-13T17:17:22.001812+0000","last_clean":"2024-09-13T17:17:22.001812+0000","last_became_active":"2024-09-13T17:17:21.550268+0000","last_became_peered":"2024-09-13T17:17:21.550268+0000","last_unstale":"2024-09-13T17:17:22.001812+0000","last_undegraded":"2024-09-13T17:17:22.001812+0000","last_fullsized":"2024-09-13T17:17:22.001812+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:16.781802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000912+0000","last_change":"2024-09-13T17:17:21.997552+0000","last_active":"2024-09-13T17:17:22.000912+0000","last_peered":"2024-09-13T17:17:22.000912+0000","last_clean":"2024-09-13T17:17:22.000912+0000","last_became_active":"2024-09-13T17:17:21.993602+0000","last_became_peered":"2024-09-13T17:17:21.993602+0000","last_unstale":"2024-09-13T17:17:22.000912+0000","last_undegraded":"2024-09-13T17:17:22.000912+0000","last_fullsized":"2024-09-13T17:17:22.000912+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:57.259385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000248+0000","last_change":"2024-09-13T17:17:21.548233+0000","last_active":"2024-09-13T17:17:22.000248+0000","last_peered":"2024-09-13T17:17:22.000248+0000","last_clean":"2024-09-13T17:17:22.000248+0000","last_became_active":"2024-09-13T17:17:21.548018+0000","last_became_peered":"2024-09-13T17:17:21.548018+0000","last_unstale":"2024-09-13T17:17:22.000248+0000","last_undegraded":"2024-09-13T17:17:22.000248+0000","last_fullsized":"2024-09-13T17:17:22.000248+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:21.917024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.987158+0000","last_change":"2024-09-13T17:17:21.978340+0000","last_active":"2024-09-13T17:17:21.987158+0000","last_peered":"2024-09-13T17:17:21.987158+0000","last_clean":"2024-09-13T17:17:21.987158+0000","last_became_active":"2024-09-13T17:17:21.978045+0000","last_became_peered":"2024-09-13T17:17:21.978045+0000","last_unstale":"2024-09-13T17:17:21.987158+0000","last_undegraded":"2024-09-13T17:17:21.987158+0000","last_fullsized":"2024-09-13T17:17:21.987158+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998192+0000","last_change":"2024-09-13T17:17:21.548112+0000","last_active":"2024-09-13T17:17:21.998192+0000","last_peered":"2024-09-13T17:17:21.998192+0000","last_clean":"2024-09-13T17:17:21.998192+0000","last_became_active":"2024-09-13T17:17:21.547791+0000","last_became_peered":"2024-09-13T17:17:21.547791+0000","last_unstale":"2024-09-13T17:17:21.998192+0000","last_undegraded":"2024-09-13T17:17:21.998192+0000","last_fullsized":"2024-09-13T17:17:21.998192+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:23.020261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002939+0000","last_change":"2024-09-13T17:17:22.002028+0000","last_active":"2024-09-13T17:17:22.002939+0000","last_peered":"2024-09-13T17:17:22.002939+0000","last_clean":"2024-09-13T17:17:22.002939+0000","last_became_active":"2024-09-13T17:17:22.001559+0000","last_became_peered":"2024-09-13T17:17:22.001559+0000","last_unstale":"2024-09-13T17:17:22.002939+0000","last_undegraded":"2024-09-13T17:17:22.002939+0000","last_fullsized":"2024-09-13T17:17:22.002939+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:24.027694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"138'1","reported_seq":11,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986881+0000","last_change":"2024-09-13T17:17:21.546597+0000","last_active":"2024-09-13T17:17:21.986881+0000","last_peered":"2024-09-13T17:17:21.986881+0000","last_clean":"2024-09-13T17:17:21.986881+0000","last_became_active":"2024-09-13T17:17:21.546290+0000","last_became_peered":"2024-09-13T17:17:21.546290+0000","last_unstale":"2024-09-13T17:17:21.986881+0000","last_undegraded":"2024-09-13T17:17:21.986881+0000","last_fullsized":"2024-09-13T17:17:21.986881+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:46:17.950169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.984821+0000","last_change":"2024-09-13T17:17:21.979588+0000","last_active":"2024-09-13T17:17:21.984821+0000","last_peered":"2024-09-13T17:17:21.984821+0000","last_clean":"2024-09-13T17:17:21.984821+0000","last_became_active":"2024-09-13T17:17:21.979317+0000","last_became_peered":"2024-09-13T17:17:21.979317+0000","last_unstale":"2024-09-13T17:17:21.984821+0000","last_undegraded":"2024-09-13T17:17:21.984821+0000","last_fullsized":"2024-09-13T17:17:21.984821+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:29:42.953373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.970867+0000","last_change":"2024-09-13T17:17:21.545670+0000","last_active":"2024-09-13T17:17:21.970867+0000","last_peered":"2024-09-13T17:17:21.970867+0000","last_clean":"2024-09-13T17:17:21.970867+0000","last_became_active":"2024-09-13T17:17:21.543851+0000","last_became_peered":"2024-09-13T17:17:21.543851+0000","last_unstale":"2024-09-13T17:17:21.970867+0000","last_undegraded":"2024-09-13T17:17:21.970867+0000","last_fullsized":"2024-09-13T17:17:21.970867+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:40:25.687649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998312+0000","last_change":"2024-09-13T17:17:21.991030+0000","last_active":"2024-09-13T17:17:21.998312+0000","last_peered":"2024-09-13T17:17:21.998312+0000","last_clean":"2024-09-13T17:17:21.998312+0000","last_became_active":"2024-09-13T17:17:21.990795+0000","last_became_peered":"2024-09-13T17:17:21.990795+0000","last_unstale":"2024-09-13T17:17:21.998312+0000","last_undegraded":"2024-09-13T17:17:21.998312+0000","last_fullsized":"2024-09-13T17:17:21.998312+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:36.628243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.001223+0000","last_change":"2024-09-13T17:17:21.552671+0000","last_active":"2024-09-13T17:17:22.001223+0000","last_peered":"2024-09-13T17:17:22.001223+0000","last_clean":"2024-09-13T17:17:22.001223+0000","last_became_active":"2024-09-13T17:17:21.551524+0000","last_became_peered":"2024-09-13T17:17:21.551524+0000","last_unstale":"2024-09-13T17:17:22.001223+0000","last_undegraded":"2024-09-13T17:17:22.001223+0000","last_fullsized":"2024-09-13T17:17:22.001223+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:50:27.073385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.990082+0000","last_change":"2024-09-13T17:17:21.985804+0000","last_active":"2024-09-13T17:17:21.990082+0000","last_peered":"2024-09-13T17:17:21.990082+0000","last_clean":"2024-09-13T17:17:21.990082+0000","last_became_active":"2024-09-13T17:17:21.985500+0000","last_became_peered":"2024-09-13T17:17:21.985500+0000","last_unstale":"2024-09-13T17:17:21.990082+0000","last_undegraded":"2024-09-13T17:17:21.990082+0000","last_fullsized":"2024-09-13T17:17:21.990082+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:57.067573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.977121+0000","last_change":"2024-09-13T17:17:21.555641+0000","last_active":"2024-09-13T17:17:21.977121+0000","last_peered":"2024-09-13T17:17:21.977121+0000","last_clean":"2024-09-13T17:17:21.977121+0000","last_became_active":"2024-09-13T17:17:21.554741+0000","last_became_peered":"2024-09-13T17:17:21.554741+0000","last_unstale":"2024-09-13T17:17:21.977121+0000","last_undegraded":"2024-09-13T17:17:21.977121+0000","last_fullsized":"2024-09-13T17:17:21.977121+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:01.417167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.986432+0000","last_change":"2024-09-13T17:17:21.981864+0000","last_active":"2024-09-13T17:17:21.986432+0000","last_peered":"2024-09-13T17:17:21.986432+0000","last_clean":"2024-09-13T17:17:21.986432+0000","last_became_active":"2024-09-13T17:17:21.981602+0000","last_became_peered":"2024-09-13T17:17:21.981602+0000","last_unstale":"2024-09-13T17:17:21.986432+0000","last_undegraded":"2024-09-13T17:17:21.986432+0000","last_fullsized":"2024-09-13T17:17:21.986432+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:35:50.733604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.980913+0000","last_change":"2024-09-13T17:17:21.541881+0000","last_active":"2024-09-13T17:17:21.980913+0000","last_peered":"2024-09-13T17:17:21.980913+0000","last_clean":"2024-09-13T17:17:21.980913+0000","last_became_active":"2024-09-13T17:17:21.541253+0000","last_became_peered":"2024-09-13T17:17:21.541253+0000","last_unstale":"2024-09-13T17:17:21.980913+0000","last_undegraded":"2024-09-13T17:17:21.980913+0000","last_fullsized":"2024-09-13T17:17:21.980913+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:07.974799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002671+0000","last_change":"2024-09-13T17:17:22.001879+0000","last_active":"2024-09-13T17:17:22.002671+0000","last_peered":"2024-09-13T17:17:22.002671+0000","last_clean":"2024-09-13T17:17:22.002671+0000","last_became_active":"2024-09-13T17:17:22.001581+0000","last_became_peered":"2024-09-13T17:17:22.001581+0000","last_unstale":"2024-09-13T17:17:22.002671+0000","last_undegraded":"2024-09-13T17:17:22.002671+0000","last_fullsized":"2024-09-13T17:17:22.002671+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:08.982232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.973394+0000","last_change":"2024-09-13T17:17:21.549737+0000","last_active":"2024-09-13T17:17:21.973394+0000","last_peered":"2024-09-13T17:17:21.973394+0000","last_clean":"2024-09-13T17:17:21.973394+0000","last_became_active":"2024-09-13T17:17:21.549415+0000","last_became_peered":"2024-09-13T17:17:21.549415+0000","last_unstale":"2024-09-13T17:17:21.973394+0000","last_undegraded":"2024-09-13T17:17:21.973394+0000","last_fullsized":"2024-09-13T17:17:21.973394+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:05.464294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.976673+0000","last_change":"2024-09-13T17:17:21.974701+0000","last_active":"2024-09-13T17:17:21.976673+0000","last_peered":"2024-09-13T17:17:21.976673+0000","last_clean":"2024-09-13T17:17:21.976673+0000","last_became_active":"2024-09-13T17:17:21.974399+0000","last_became_peered":"2024-09-13T17:17:21.974399+0000","last_unstale":"2024-09-13T17:17:21.976673+0000","last_undegraded":"2024-09-13T17:17:21.976673+0000","last_fullsized":"2024-09-13T17:17:21.976673+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:30.412399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002938+0000","last_change":"2024-09-13T17:17:21.551133+0000","last_active":"2024-09-13T17:17:22.002938+0000","last_peered":"2024-09-13T17:17:22.002938+0000","last_clean":"2024-09-13T17:17:22.002938+0000","last_became_active":"2024-09-13T17:17:21.550782+0000","last_became_peered":"2024-09-13T17:17:21.550782+0000","last_unstale":"2024-09-13T17:17:22.002938+0000","last_undegraded":"2024-09-13T17:17:22.002938+0000","last_fullsized":"2024-09-13T17:17:22.002938+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:25.672114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.000258+0000","last_change":"2024-09-13T17:17:21.998885+0000","last_active":"2024-09-13T17:17:22.000258+0000","last_peered":"2024-09-13T17:17:22.000258+0000","last_clean":"2024-09-13T17:17:22.000258+0000","last_became_active":"2024-09-13T17:17:21.995692+0000","last_became_peered":"2024-09-13T17:17:21.995692+0000","last_unstale":"2024-09-13T17:17:22.000258+0000","last_undegraded":"2024-09-13T17:17:22.000258+0000","last_fullsized":"2024-09-13T17:17:22.000258+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:08.790760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.980646+0000","last_change":"2024-09-13T17:17:21.554984+0000","last_active":"2024-09-13T17:17:21.980646+0000","last_peered":"2024-09-13T17:17:21.980646+0000","last_clean":"2024-09-13T17:17:21.980646+0000","last_became_active":"2024-09-13T17:17:21.554700+0000","last_became_peered":"2024-09-13T17:17:21.554700+0000","last_unstale":"2024-09-13T17:17:21.980646+0000","last_undegraded":"2024-09-13T17:17:21.980646+0000","last_fullsized":"2024-09-13T17:17:21.980646+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:10.989830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.993323+0000","last_change":"2024-09-13T17:17:21.990869+0000","last_active":"2024-09-13T17:17:21.993323+0000","last_peered":"2024-09-13T17:17:21.993323+0000","last_clean":"2024-09-13T17:17:21.993323+0000","last_became_active":"2024-09-13T17:17:21.990638+0000","last_became_peered":"2024-09-13T17:17:21.990638+0000","last_unstale":"2024-09-13T17:17:21.993323+0000","last_undegraded":"2024-09-13T17:17:21.993323+0000","last_fullsized":"2024-09-13T17:17:21.993323+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:11.997263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.969156+0000","last_change":"2024-09-13T17:17:21.549987+0000","last_active":"2024-09-13T17:17:21.969156+0000","last_peered":"2024-09-13T17:17:21.969156+0000","last_clean":"2024-09-13T17:17:21.969156+0000","last_became_active":"2024-09-13T17:17:21.549725+0000","last_became_peered":"2024-09-13T17:17:21.549725+0000","last_unstale":"2024-09-13T17:17:21.969156+0000","last_undegraded":"2024-09-13T17:17:21.969156+0000","last_fullsized":"2024-09-13T17:17:21.969156+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:27.223607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.996064+0000","last_change":"2024-09-13T17:17:21.993795+0000","last_active":"2024-09-13T17:17:21.996064+0000","last_peered":"2024-09-13T17:17:21.996064+0000","last_clean":"2024-09-13T17:17:21.996064+0000","last_became_active":"2024-09-13T17:17:21.993520+0000","last_became_peered":"2024-09-13T17:17:21.993520+0000","last_unstale":"2024-09-13T17:17:21.996064+0000","last_undegraded":"2024-09-13T17:17:21.996064+0000","last_fullsized":"2024-09-13T17:17:21.996064+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:26.727294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.974149+0000","last_change":"2024-09-13T17:17:21.552928+0000","last_active":"2024-09-13T17:17:21.974149+0000","last_peered":"2024-09-13T17:17:21.974149+0000","last_clean":"2024-09-13T17:17:21.974149+0000","last_became_active":"2024-09-13T17:17:21.552628+0000","last_became_peered":"2024-09-13T17:17:21.552628+0000","last_unstale":"2024-09-13T17:17:21.974149+0000","last_undegraded":"2024-09-13T17:17:21.974149+0000","last_fullsized":"2024-09-13T17:17:21.974149+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:53.189574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.999210+0000","last_change":"2024-09-13T17:17:21.996620+0000","last_active":"2024-09-13T17:17:21.999210+0000","last_peered":"2024-09-13T17:17:21.999210+0000","last_clean":"2024-09-13T17:17:21.999210+0000","last_became_active":"2024-09-13T17:17:21.996322+0000","last_became_peered":"2024-09-13T17:17:21.996322+0000","last_unstale":"2024-09-13T17:17:21.999210+0000","last_undegraded":"2024-09-13T17:17:21.999210+0000","last_fullsized":"2024-09-13T17:17:21.999210+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:03.967219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.968628+0000","last_change":"2024-09-13T17:17:21.554349+0000","last_active":"2024-09-13T17:17:21.968628+0000","last_peered":"2024-09-13T17:17:21.968628+0000","last_clean":"2024-09-13T17:17:21.968628+0000","last_became_active":"2024-09-13T17:17:21.553680+0000","last_became_peered":"2024-09-13T17:17:21.553680+0000","last_unstale":"2024-09-13T17:17:21.968628+0000","last_undegraded":"2024-09-13T17:17:21.968628+0000","last_fullsized":"2024-09-13T17:17:21.968628+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:08.719729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.989352+0000","last_change":"2024-09-13T17:17:21.987327+0000","last_active":"2024-09-13T17:17:21.989352+0000","last_peered":"2024-09-13T17:17:21.989352+0000","last_clean":"2024-09-13T17:17:21.989352+0000","last_became_active":"2024-09-13T17:17:21.986322+0000","last_became_peered":"2024-09-13T17:17:21.986322+0000","last_unstale":"2024-09-13T17:17:21.989352+0000","last_undegraded":"2024-09-13T17:17:21.989352+0000","last_fullsized":"2024-09-13T17:17:21.989352+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:04.216711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.979465+0000","last_change":"2024-09-13T17:17:21.556477+0000","last_active":"2024-09-13T17:17:21.979465+0000","last_peered":"2024-09-13T17:17:21.979465+0000","last_clean":"2024-09-13T17:17:21.979465+0000","last_became_active":"2024-09-13T17:17:21.556142+0000","last_became_peered":"2024-09-13T17:17:21.556142+0000","last_unstale":"2024-09-13T17:17:21.979465+0000","last_undegraded":"2024-09-13T17:17:21.979465+0000","last_fullsized":"2024-09-13T17:17:21.979465+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:39.171300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.998819+0000","last_change":"2024-09-13T17:17:21.997075+0000","last_active":"2024-09-13T17:17:21.998819+0000","last_peered":"2024-09-13T17:17:21.998819+0000","last_clean":"2024-09-13T17:17:21.998819+0000","last_became_active":"2024-09-13T17:17:21.996300+0000","last_became_peered":"2024-09-13T17:17:21.996300+0000","last_unstale":"2024-09-13T17:17:21.998819+0000","last_undegraded":"2024-09-13T17:17:21.998819+0000","last_fullsized":"2024-09-13T17:17:21.998819+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:06.471727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.981025+0000","last_change":"2024-09-13T17:17:21.553019+0000","last_active":"2024-09-13T17:17:21.981025+0000","last_peered":"2024-09-13T17:17:21.981025+0000","last_clean":"2024-09-13T17:17:21.981025+0000","last_became_active":"2024-09-13T17:17:21.552745+0000","last_became_peered":"2024-09-13T17:17:21.552745+0000","last_unstale":"2024-09-13T17:17:21.981025+0000","last_undegraded":"2024-09-13T17:17:21.981025+0000","last_fullsized":"2024-09-13T17:17:21.981025+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:56.825607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.997988+0000","last_change":"2024-09-13T17:17:21.997306+0000","last_active":"2024-09-13T17:17:21.997988+0000","last_peered":"2024-09-13T17:17:21.997988+0000","last_clean":"2024-09-13T17:17:21.997988+0000","last_became_active":"2024-09-13T17:17:21.997057+0000","last_became_peered":"2024-09-13T17:17:21.997057+0000","last_unstale":"2024-09-13T17:17:21.997988+0000","last_undegraded":"2024-09-13T17:17:21.997988+0000","last_fullsized":"2024-09-13T17:17:21.997988+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:21.454835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":10,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.973562+0000","last_change":"2024-09-13T17:17:21.543552+0000","last_active":"2024-09-13T17:17:21.973562+0000","last_peered":"2024-09-13T17:17:21.973562+0000","last_clean":"2024-09-13T17:17:21.973562+0000","last_became_active":"2024-09-13T17:17:21.543275+0000","last_became_peered":"2024-09-13T17:17:21.543275+0000","last_unstale":"2024-09-13T17:17:21.973562+0000","last_undegraded":"2024-09-13T17:17:21.973562+0000","last_fullsized":"2024-09-13T17:17:21.973562+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:39.416409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":9,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002714+0000","last_change":"2024-09-13T17:17:22.001979+0000","last_active":"2024-09-13T17:17:22.002714+0000","last_peered":"2024-09-13T17:17:22.002714+0000","last_clean":"2024-09-13T17:17:22.002714+0000","last_became_active":"2024-09-13T17:17:22.001642+0000","last_became_peered":"2024-09-13T17:17:22.001642+0000","last_unstale":"2024-09-13T17:17:22.002714+0000","last_undegraded":"2024-09-13T17:17:22.002714+0000","last_fullsized":"2024-09-13T17:17:22.002714+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:21.523186+0000","last_clean_scrub_stamp":"2024-09-13T17:17:21.523186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:47:57.833040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"138'2","reported_seq":12,"reported_epoch":138,"state":"active+clean","last_fresh":"2024-09-13T17:17:22.002015+0000","last_change":"2024-09-13T17:17:21.559463+0000","last_active":"2024-09-13T17:17:22.002015+0000","last_peered":"2024-09-13T17:17:22.002015+0000","last_clean":"2024-09-13T17:17:22.002015+0000","last_became_active":"2024-09-13T17:17:21.559131+0000","last_became_peered":"2024-09-13T17:17:21.559131+0000","last_unstale":"2024-09-13T17:17:22.002015+0000","last_undegraded":"2024-09-13T17:17:22.002015+0000","last_fullsized":"2024-09-13T17:17:22.002015+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":136,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:17:20.515753+0000","last_clean_scrub_stamp":"2024-09-13T17:17:20.515753+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:33.787674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":263,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-09-13T17:17:21.550175+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:17:21.550175+0000","last_peered":"2024-09-13T17:17:21.550175+0000","last_clean":"2024-09-13T17:17:21.550175+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:17:21.550175+0000","last_undegraded":"2024-09-13T17:17:21.550175+0000","last_fullsized":"2024-09-13T17:17:21.550175+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574964,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73536,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298304,"statfs":{"total":96636764160,"available":96561463296,"internally_reserved":0,"allocated":2338816,"data_stored":1347787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":72939979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.64700000000000002},"min":{"1min":0.36599999999999999,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.96899999999999997,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.44900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.004,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68400000000000005,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0149999999999999,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.748,"15min":0.70799999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.86399999999999999,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.68799999999999994,"15min":0.69699999999999995},"min":{"1min":0.51800000000000002,"5min":0.48299999999999998,"15min":0.42399999999999999},"max":{"1min":0.91500000000000004,"5min":1.544,"15min":1.544},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.69399999999999995,"15min":0.70699999999999996},"min":{"1min":0.45300000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":1.605,"15min":1.605},"last":0.63}]},{"osd":3,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.433,"5min":0.41299999999999998,"15min":0.38800000000000001},"max":{"1min":0.92200000000000004,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.496,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93600000000000005,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.66100000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.60699999999999998,"15min":0.64000000000000001},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.879,"5min":0.90400000000000003,"15min":1.054},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.64700000000000002,"15min":0.67800000000000005},"min":{"1min":0.432,"5min":0.39800000000000002,"15min":0.35399999999999998},"max":{"1min":0.90400000000000003,"5min":0.92900000000000005,"15min":1.1220000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.65500000000000003,"15min":0.68100000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.80400000000000005,"5min":0.94899999999999995,"15min":1.143},"last":0.375},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.74399999999999999,"15min":0.76800000000000002},"min":{"1min":0.48199999999999998,"5min":0.48099999999999998,"15min":0.42599999999999999},"max":{"1min":0.91100000000000003,"5min":1.026,"15min":1.337},"last":0.57999999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:14 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.71699999999999997,"15min":0.751},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"max":{"1min":0.89300000000000002,"5min":0.99199999999999999,"15min":1.1930000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.72499999999999998,"15min":0.75900000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":1.2030000000000001},"last":0.46000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54960,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316880,"statfs":{"total":96636764160,"available":96580485120,"internally_reserved":0,"allocated":1732608,"data_stored":756781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.67400000000000004,"15min":0.66600000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.32900000000000001},"max":{"1min":0.871,"5min":1.298,"15min":3.6629999999999998},"last":0.627},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.51000000000000001,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":0.88900000000000001,"5min":1.5680000000000001,"15min":3.1709999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.67500000000000004,"15min":0.65800000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77300000000000002,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.68899999999999995,"15min":0.65500000000000003},"min":{"1min":0.375,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.84999999999999998,"5min":1.46,"15min":3.3079999999999998},"last":0.54500000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.68500000000000005,"15min":0.66900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.004,"5min":2.3359999999999999,"15min":2.3359999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.62,"5min":0.70099999999999996,"15min":0.68999999999999995},"min":{"1min":0.46300000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":3.2629999999999999},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.73499999999999999,"15min":0.71199999999999997},"min":{"1min":0.46000000000000002,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":1.0660000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.47399999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.91100000000000003,"5min":3.6720000000000002,"15min":3.742},"last":0.68300000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.64400000000000002,"15min":0.629},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":1.1599999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.64300000000000002,"15min":0.65200000000000002},"min":{"1min":0.40600000000000003,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.83399999999999996,"5min":1.075,"15min":4.7750000000000004},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.71199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.304,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65600000000000003,"15min":0.70099999999999996},"min":{"1min":0.39200000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.90000000000000002,"5min":1.127,"15min":3.609},"last":0.73799999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:17:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72099999999999997,"15min":0.746},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.88500000000000001,"5min":1.601,"15min":4.9669999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.747,"5min":0.77900000000000003,"15min":0.79900000000000004},"min":{"1min":0.52300000000000002,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.238,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.78300000000000003}]}]},{"osd":5,"up_from":13,"seq":55834574964,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50856,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94320984,"statfs":{"total":96636764160,"available":96584687616,"internally_reserved":0,"allocated":1728512,"data_stored":755329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.64200000000000002,"15min":0.77200000000000002},"min":{"1min":0.40899999999999997,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.95399999999999996,"5min":1.498,"15min":38.005000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.65500000000000003,"15min":0.751},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.749,"5min":1.577,"15min":21.376999999999999},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.66000000000000003,"15min":0.72799999999999998},"min":{"1min":0.38800000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.877,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.66800000000000004,"15min":0.69699999999999995},"min":{"1min":0.47399999999999998,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":0.98799999999999999,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.66300000000000003,"15min":0.748},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81699999999999995,"5min":1.524,"15min":17.268000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.69699999999999995,"15min":0.746},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.91800000000000004,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.72599999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.73599999999999999,"15min":1.103},"min":{"1min":0.47699999999999998,"5min":0.44,"15min":0.435},"max":{"1min":0.89600000000000002,"5min":2.8690000000000002,"15min":113.363},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71999999999999997,"15min":0.75800000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.97599999999999998,"5min":2.359,"15min":4.1180000000000003},"last":0.86399999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.37,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90500000000000003,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.66200000000000003,"15min":0.70599999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88800000000000001,"5min":2.827,"15min":2.827},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.68300000000000005,"15min":0.70099999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.86499999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.70199999999999996,"15min":0.72099999999999997},"min":{"1min":0.42599999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.018,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.75600000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.73599999999999999,"15min":0.75600000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96199999999999997,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.74299999999999999,"15min":0.76100000000000001},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.94999999999999996,"5min":1.9379999999999999,"15min":2.02},"last":0.89000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574963,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59052,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312788,"statfs":{"total":96636764160,"available":96576294912,"internally_reserved":0,"allocated":1724416,"data_stored":755287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68200000000000005,"15min":0.65200000000000002},"min":{"1min":0.47999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86899999999999999,"5min":1.056,"15min":1.0620000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.71799999999999997,"15min":0.72899999999999998},"min":{"1min":0.55200000000000005,"5min":0.373,"15min":0.373},"max":{"1min":0.88100000000000001,"5min":1.3400000000000001,"15min":1.341},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.66500000000000004,"15min":0.66000000000000003},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":0.86499999999999999,"5min":1.173,"15min":1.173},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65900000000000003,"15min":0.63200000000000001},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88700000000000001,"5min":0.92000000000000004,"15min":0.93999999999999995},"last":0.59499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.68000000000000005,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.96799999999999997,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.71699999999999997,"15min":0.71799999999999997},"min":{"1min":0.495,"5min":0.44900000000000001,"15min":0.42199999999999999},"max":{"1min":0.91100000000000003,"5min":1.22,"15min":2.1259999999999999},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.70199999999999996,"15min":0.67400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.97799999999999998,"5min":1.006,"15min":1.006},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72599999999999998,"15min":0.69899999999999995},"min":{"1min":0.55700000000000005,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.94099999999999995,"5min":1.232,"15min":1.232},"last":0.624}]},{"osd":5,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63100000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.97899999999999998,"5min":1.284,"15min":1.284},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.65700000000000003,"15min":0.65500000000000003},"min":{"1min":0.42699999999999999,"5min":0.39700000000000002,"15min":0.30299999999999999},"max":{"1min":0.83699999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.68899999999999995,"15min":0.66300000000000003},"min":{"1min":0.40899999999999997,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1.03,"5min":1.274,"15min":1.274},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71899999999999997,"15min":0.73399999999999999},"min":{"1min":0.503,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":1.0169999999999999,"5min":1.292,"15min":1.413},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:13 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.71799999999999997,"15min":0.69899999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.018,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.748},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.77200000000000002,"15min":0.73699999999999999},"min":{"1min":0.58199999999999996,"5min":0.44500000000000001,"15min":0.371},"max":{"1min":1.0700000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.875}]}]},{"osd":3,"up_from":13,"seq":55834574964,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77036,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94294804,"statfs":{"total":96636764160,"available":96557879296,"internally_reserved":0,"allocated":1728512,"data_stored":755377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77134279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.42799999999999999,"15min":0.44700000000000001},"min":{"1min":0.28499999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.622,"5min":1.177,"15min":1.9810000000000001},"last":0.436},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.45800000000000002,"15min":0.52900000000000003},"min":{"1min":0.38800000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.621,"5min":0.73499999999999999,"15min":2.96},"last":0.46000000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48999999999999999,"15min":0.51600000000000001},"min":{"1min":0.29799999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68000000000000005,"5min":0.91900000000000004,"15min":3.5600000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.55600000000000005,"15min":0.67100000000000004},"min":{"1min":0.41999999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71399999999999997,"5min":1.0309999999999999,"15min":3.6829999999999998},"last":0.62}]},{"osd":2,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.38600000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.69499999999999995,"5min":0.88,"15min":1.9590000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53800000000000003,"15min":0.58099999999999996},"min":{"1min":0.375,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.73499999999999999,"5min":0.97699999999999998,"15min":1.3819999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.72399999999999998,"15min":0.78300000000000003},"min":{"1min":0.42699999999999999,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.1040000000000001,"5min":1.1339999999999999,"15min":2.8610000000000002},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.74299999999999999,"15min":0.76700000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.48199999999999998},"max":{"1min":0.91000000000000003,"5min":1.012,"15min":1.3640000000000001},"last":0.71399999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.746,"15min":0.77600000000000002},"min":{"1min":0.496,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.032,"5min":1.0580000000000001,"15min":1.391},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.80100000000000005},"min":{"1min":0.47999999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":0.76700000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.746,"15min":0.76500000000000001},"min":{"1min":0.48699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.401},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.76300000000000001,"15min":0.78600000000000003},"min":{"1min":0.53000000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":0.87}]},{"osd":7,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.755,"15min":0.82899999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.45600000000000002},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":2.899},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.81200000000000006},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0489999999999999,"5min":1.075,"15min":1.4359999999999999},"last":0.96599999999999997}]}]},{"osd":2,"up_from":13,"seq":55834574964,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54952,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316888,"statfs":{"total":96636764160,"available":96580493312,"internally_reserved":0,"allocated":1724416,"data_stored":754814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54524359},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.43099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.83899999999999997,"5min":1.097,"15min":1.097},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.81699999999999995,"15min":0.56000000000000005},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.28100000000000003},"max":{"1min":0.56000000000000005,"5min":30.023,"15min":30.023},"last":0.45400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.49399999999999999,"15min":1.3460000000000001},"min":{"1min":0.33100000000000002,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.67600000000000005,"5min":1.046,"15min":260.53399999999999},"last":0.441},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.75900000000000001,"15min":1.448},"min":{"1min":0.40799999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.65800000000000003,"5min":21.279,"15min":262.61399999999998},"last":0.53000000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.81799999999999995,"15min":0.59599999999999997},"min":{"1min":0.36399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.59199999999999997,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.375},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59099999999999997,"15min":0.57799999999999996},"min":{"1min":0.45900000000000002,"5min":0.42799999999999999,"15min":0.42199999999999999},"max":{"1min":0.745,"5min":1.214,"15min":1.214},"last":0.47699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.89400000000000002,"15min":0.80300000000000005},"min":{"1min":0.58599999999999997,"5min":0.495,"15min":0.49099999999999999},"max":{"1min":1.0209999999999999,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":1.018,"15min":0.85899999999999999},"min":{"1min":0.48799999999999999,"5min":0.371,"15min":0.371},"max":{"1min":1.0569999999999999,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.748}]},{"osd":5,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":1.0900000000000001,"15min":0.88700000000000001},"min":{"1min":0.54600000000000004,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":1.099,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.78200000000000003,"15min":0.77600000000000002},"min":{"1min":0.52600000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0469999999999999,"5min":1.2789999999999999,"15min":1.569},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.90200000000000002,"15min":0.81100000000000005},"min":{"1min":0.53700000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0289999999999999,"5min":10.664,"15min":10.664},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":1.091,"15min":0.90000000000000002},"min":{"1min":0.57999999999999996,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0760000000000001,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.91000000000000003,"15min":0.82299999999999995},"min":{"1min":0.59999999999999998,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.99199999999999999,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.81999999999999995,"5min":1.1240000000000001,"15min":0.90700000000000003},"min":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.46100000000000002},"max":{"1min":1.05,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.59499999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574963,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86876,"kb_used_data":1680,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284964,"statfs":{"total":96636764160,"available":96547803136,"internally_reserved":0,"allocated":1720320,"data_stored":755217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45000000000000001,"15min":0.48999999999999999},"min":{"1min":0.29499999999999998,"5min":0.27200000000000002,"15min":0.26100000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.498,"15min":0.51900000000000002},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.67000000000000004,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.436}]},{"osd":2,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44900000000000001,"15min":0.46899999999999997},"min":{"1min":0.29299999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.62,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.504,"15min":0.54800000000000004},"min":{"1min":0.40600000000000003,"5min":0.316,"15min":0.316},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.59099999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59099999999999997,"15min":0.68000000000000005},"min":{"1min":0.50900000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.73499999999999999,"5min":0.84499999999999997,"15min":16.382999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.52700000000000002,"15min":0.60999999999999999},"min":{"1min":0.42299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.76400000000000001,"5min":0.81499999999999995,"15min":18.318000000000001},"last":0.48599999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.76500000000000001,"15min":0.80500000000000005},"min":{"1min":0.496,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.159,"5min":1.165,"15min":6.1630000000000003},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78900000000000003,"15min":0.81299999999999994},"min":{"1min":0.55700000000000005,"5min":0.53000000000000003,"15min":0.49399999999999999},"max":{"1min":1.149,"5min":1.329,"15min":1.78},"last":0.91100000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.75,"15min":0.77100000000000002},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.6870000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.754,"15min":0.79600000000000004},"min":{"1min":0.53800000000000003,"5min":0.502,"15min":0.502},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.95199999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.749,"15min":0.79800000000000004},"min":{"1min":0.58199999999999996,"5min":0.56200000000000006,"15min":0.53200000000000003},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.77500000000000002,"15min":0.81699999999999995},"min":{"1min":0.503,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.204,"5min":1.204,"15min":6.1970000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:17:08 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.76500000000000001,"15min":0.81200000000000006},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.78900000000000003,"15min":0.82099999999999995},"min":{"1min":0.57099999999999995,"5min":0.56399999999999995,"15min":0.53800000000000003},"max":{"1min":1.145,"5min":1.145,"15min":1.6359999999999999},"last":0.93200000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574965,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63716,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94308124,"statfs":{"total":96636764160,"available":96571518976,"internally_reserved":0,"allocated":2310144,"data_stored":1344566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":62912967},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.45500000000000002,"15min":0.61099999999999999},"min":{"1min":0.30499999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.872,"5min":1.018,"15min":5.5259999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52600000000000002,"15min":0.629},"min":{"1min":0.32600000000000001,"5min":0.309,"15min":0.27000000000000002},"max":{"1min":0.84699999999999998,"5min":3.004,"15min":5.4889999999999999},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55600000000000005,"15min":0.66200000000000003},"min":{"1min":0.35999999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.79400000000000004,"5min":3.605,"15min":6.0430000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52400000000000002,"15min":0.66400000000000003},"min":{"1min":0.41999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.90500000000000003,"5min":1.29,"15min":4.6760000000000002},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.51600000000000001,"15min":0.621},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.314},"max":{"1min":0.75700000000000001,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.60199999999999998,"15min":0.68400000000000005},"min":{"1min":0.375,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.88300000000000001,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.78000000000000003,"15min":0.86399999999999999},"min":{"1min":0.56000000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.137,"5min":1.9490000000000001,"15min":4.734},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.77800000000000002,"15min":0.872},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.3109999999999999,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.79500000000000004,"15min":0.88700000000000001},"min":{"1min":0.47799999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.1839999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.79700000000000004,"15min":0.878},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.234,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.80400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.78900000000000003,"15min":0.98299999999999998},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.099,"5min":1.194,"15min":4.8470000000000004},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.80000000000000004,"15min":0.89700000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.48599999999999999},"max":{"1min":1.2889999999999999,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.63700000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:17:16 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.81399999999999995,"15min":0.92500000000000004},"min":{"1min":0.59299999999999997,"5min":0.51100000000000001,"15min":0.47399999999999998},"max":{"1min":1.375,"5min":1.405,"15min":5.0540000000000003},"last":0.748},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.82099999999999995,"15min":0.91500000000000004},"min":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.505},"max":{"1min":1.2649999999999999,"5min":2.706,"15min":4.9950000000000001},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:17:26.591 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:17:26.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:17:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.787+0000 7f8274200640 1 Processor -- start 2024-09-13T17:17:26.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.787+0000 7f8274200640 1 -- start start 2024-09-13T17:17:26.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.788+0000 7f8274200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f826c15a780 0x7f826c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.788+0000 7f8274200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f826c06c250 0x7f826c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.788+0000 7f8274200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c06c660 con 0x7f826c15ab60 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.788+0000 7f8274200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c06c8b0 con 0x7f826c15a780 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.788+0000 7f8274200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c06cb90 con 0x7f826c06c250 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8270e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f826c15ab60 0x7f826c06bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58476/0 (socket says 172.21.15.28:58476) 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8270e00640 1 -- 172.21.15.28:0/2511740548 learned_addr learned my addr 172.21.15.28:0/2511740548 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8271800640 1 --2- 172.21.15.28:0/2511740548 >> v2:172.21.15.28:3300/0 conn(0x7f826c15a780 0x7f826c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8272200640 1 --2- 172.21.15.28:0/2511740548 >> v2:172.21.15.189:3300/0 conn(0x7f826c06c250 0x7f826c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8271800640 1 -- 172.21.15.28:0/2511740548 >> v2:172.21.15.189:3300/0 conn(0x7f826c06c250 msgr2=0x7f826c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8271800640 1 --2- 172.21.15.28:0/2511740548 >> v2:172.21.15.189:3300/0 conn(0x7f826c06c250 0x7f826c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8271800640 1 -- 172.21.15.28:0/2511740548 >> v1:172.21.15.28:6789/0 conn(0x7f826c15ab60 legacy=0x7f826c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f8271800640 1 -- 172.21.15.28:0/2511740548 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0cc980 con 0x7f826c15a780 2024-09-13T17:17:26.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.789+0000 7f826be00640 1 -- 172.21.15.28:0/2511740548 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531258012 0 0) 0x7f826c06c660 con 0x7f826c15ab60 2024-09-13T17:17:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.790+0000 7f8271800640 1 --2- 172.21.15.28:0/2511740548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070d80 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f826c0cc370 tx=0x7f826002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a80e4729e293f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f826be00640 1 -- 172.21.15.28:0/2511740548 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260032440 con 0x7f826c15a780 2024-09-13T17:17:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f826be00640 1 -- 172.21.15.28:0/2511740548 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82600325a0 con 0x7f826c15a780 2024-09-13T17:17:26.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f826be00640 1 -- 172.21.15.28:0/2511740548 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260032930 con 0x7f826c15a780 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 -- 172.21.15.28:0/2511740548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 msgr2=0x7f826c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 --2- 172.21.15.28:0/2511740548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070d80 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f826c0cc370 tx=0x7f826002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 -- 172.21.15.28:0/2511740548 shutdown_connections 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 --2- 172.21.15.28:0/2511740548 >> v2:172.21.15.189:3300/0 conn(0x7f826c06c250 0x7f826c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 --2- 172.21.15.28:0/2511740548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070d80 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 -- 172.21.15.28:0/2511740548 >> 172.21.15.28:0/2511740548 conn(0x7f826c0b0270 msgr2=0x7f826c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 -- 172.21.15.28:0/2511740548 shutdown_connections 2024-09-13T17:17:26.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.791+0000 7f8274200640 1 -- 172.21.15.28:0/2511740548 wait complete. 2024-09-13T17:17:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.793+0000 7f8274200640 1 Processor -- start 2024-09-13T17:17:26.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.793+0000 7f8274200640 1 -- start start 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 0x7f826c151b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 0x7f826c070c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c1526a0 con 0x7f826c15a780 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0a9020 con 0x7f826c15ab60 2024-09-13T17:17:26.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8274200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0a9300 con 0x7f826c06c250 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8270e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8270e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45676/0 (socket says 172.21.15.28:45676) 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8270e00640 1 -- 172.21.15.28:0/3277952645 learned_addr learned my addr 172.21.15.28:0/3277952645 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8272200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 0x7f826c070c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8271800640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 0x7f826c151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.794+0000 7f8270e00640 1 -- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 msgr2=0x7f826c151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8270e00640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 0x7f826c151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8270e00640 1 -- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 msgr2=0x7f826c070c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8270e00640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 0x7f826c070c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8270e00640 1 -- 172.21.15.28:0/3277952645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260009400 con 0x7f826c15a780 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8272200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 0x7f826c070c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8271800640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 0x7f826c151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8270e00640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f825800c430 tx=0x7f825800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82580074e0 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8258007660 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c0a95e0 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8258004a20 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.795+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c0a9ba0 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.796+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c0a9dc0 con 0x7f826c15a780 2024-09-13T17:17:26.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.797+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8258004b80 con 0x7f826c15a780 2024-09-13T17:17:26.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.798+0000 7f8269600640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 0x7f824c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:26.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.799+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f82580891f0 con 0x7f826c15a780 2024-09-13T17:17:26.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.799+0000 7f8271800640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 0x7f824c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:26.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.800+0000 7f8271800640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 0x7f824c06d0c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8260034000 tx=0x7f826002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:26.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:26.803+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f82580543d0 con 0x7f826c15a780 2024-09-13T17:17:27.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.047+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f826c071de0 con 0x7f826c15a780 2024-09-13T17:17:27.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.051+0000 7f8269600640 1 -- 172.21.15.28:0/3277952645 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+0 (secure 0 0 0) 0x7f82580579f0 con 0x7f826c15a780 2024-09-13T17:17:27.052 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:17:27.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 msgr2=0x7f824c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 0x7f824c06d0c0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8260034000 tx=0x7f826002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:27.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 msgr2=0x7f826c070700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f825800c430 tx=0x7f825800c8f0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 shutdown_connections 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f824c06ac00 0x7f824c06d0c0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f826c15ab60 0x7f826c070c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f826c15a780 0x7f826c070700 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 --2- 172.21.15.28:0/3277952645 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f826c06c250 0x7f826c151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 >> 172.21.15.28:0/3277952645 conn(0x7f826c0b0270 msgr2=0x7f826c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 shutdown_connections 2024-09-13T17:17:27.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.053+0000 7f8274200640 1 -- 172.21.15.28:0/3277952645 wait complete. 2024-09-13T17:17:27.071 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.225+0000 7fdaab400640 1 Processor -- start 2024-09-13T17:17:27.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.225+0000 7fdaab400640 1 -- start start 2024-09-13T17:17:27.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaab400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa41592a0 0x7fdaa415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaab400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdaa406f970 0x7fdaa4159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaab400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa40b2040 con 0x7fdaa406c2d0 2024-09-13T17:17:27.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaab400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaa415b0b0 con 0x7fdaa406f970 2024-09-13T17:17:27.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaab400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa4159ff0 con 0x7fdaa41592a0 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fda9be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdaa406f970 0x7fdaa4159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fda9be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdaa406f970 0x7fdaa4159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:51986/0 (socket says 172.21.15.28:51986) 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fdaa8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa41592a0 0x7fdaa415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.227+0000 7fda9be00640 1 -- 172.21.15.28:0/624480642 learned_addr learned my addr 172.21.15.28:0/624480642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.228+0000 7fda9be00640 1 -- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa41592a0 msgr2=0x7fdaa415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.228+0000 7fda9be00640 1 --2- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa41592a0 0x7fdaa415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.228+0000 7fda9be00640 1 -- 172.21.15.28:0/624480642 >> v1:172.21.15.189:6789/0 conn(0x7fdaa406c2d0 legacy=0x7fdaa40b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.228+0000 7fda9be00640 1 -- 172.21.15.28:0/624480642 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa415a2d0 con 0x7fdaa406f970 2024-09-13T17:17:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fda9be00640 1 --2- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa406f970 0x7fdaa4159680 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009870 tx=0x7fda8c02c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37b16478dfb1b267 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fdaa3e00640 1 -- 172.21.15.28:0/624480642 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c031710 con 0x7fdaa406f970 2024-09-13T17:17:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fdaa3e00640 1 -- 172.21.15.28:0/624480642 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c02f050 con 0x7fdaa406f970 2024-09-13T17:17:27.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fdaa3e00640 1 -- 172.21.15.28:0/624480642 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c031a90 con 0x7fdaa406f970 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fdaab400640 1 -- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa406f970 msgr2=0x7fdaa4159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.229+0000 7fdaab400640 1 --2- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa406f970 0x7fdaa4159680 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009870 tx=0x7fda8c02c460 comp rx=0 tx=0).stop 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 -- 172.21.15.28:0/624480642 shutdown_connections 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 --2- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa406f970 0x7fdaa4159680 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 --2- 172.21.15.28:0/624480642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa41592a0 0x7fdaa415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 -- 172.21.15.28:0/624480642 >> 172.21.15.28:0/624480642 conn(0x7fdaa4069e80 msgr2=0x7fdaa406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 -- 172.21.15.28:0/624480642 shutdown_connections 2024-09-13T17:17:27.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.230+0000 7fdaab400640 1 -- 172.21.15.28:0/624480642 wait complete. 2024-09-13T17:17:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.231+0000 7fdaab400640 1 Processor -- start 2024-09-13T17:17:27.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 -- start start 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa406f970 0x7fdaa40af090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 0x7fdaa414abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa40b0b80 con 0x7fdaa406c2d0 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa40afae0 con 0x7fdaa41592a0 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.232+0000 7fdaab400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa40afdc0 con 0x7fdaa406f970 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45682/0 (socket says 172.21.15.28:45682) 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 -- 172.21.15.28:0/315503954 learned_addr learned my addr 172.21.15.28:0/315503954 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa9400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 0x7fdaa414abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fda9be00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa406f970 0x7fdaa40af090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 -- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa406f970 msgr2=0x7fdaa40af090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa406f970 0x7fdaa40af090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 -- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 msgr2=0x7fdaa414abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 0x7fdaa414abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 -- 172.21.15.28:0/315503954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0094d0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa9400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 0x7fdaa414abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:27.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.233+0000 7fdaa8a00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fda9400e870 tx=0x7fda9400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.234+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94009510 con 0x7fdaa406c2d0 2024-09-13T17:17:27.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.234+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9400eec0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.234+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9400cb80 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.234+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa40b00a0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.234+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa40b0660 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.235+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa415ab70 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.236+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fda9400cd20 con 0x7fdaa406c2d0 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.236+0000 7fdaa1600640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 0x7fda8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:27.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.237+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda94089580 con 0x7fdaa406c2d0 2024-09-13T17:17:27.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.237+0000 7fda9be00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 0x7fda8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:27.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.238+0000 7fda9be00640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 0x7fda8006d0c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fda8c035000 tx=0x7fda8c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:27.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.241+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fda940546c0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.453+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdaa41571e0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.453+0000 7fdaa1600640 1 -- 172.21.15.28:0/315503954 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fda94057ce0 con 0x7fdaa406c2d0 2024-09-13T17:17:27.454 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:27.455 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:27.455 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.455+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 msgr2=0x7fda8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.455+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 0x7fda8006d0c0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fda8c035000 tx=0x7fda8c008030 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.455+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 msgr2=0x7fdaa413f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.455+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fda9400e870 tx=0x7fda9400ec70 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 shutdown_connections 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda8006ac00 0x7fda8006d0c0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdaa41592a0 0x7fdaa414abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdaa406f970 0x7fdaa40af090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 --2- 172.21.15.28:0/315503954 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdaa406c2d0 0x7fdaa413f8c0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 >> 172.21.15.28:0/315503954 conn(0x7fdaa4069e80 msgr2=0x7fdaa4157700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:27.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 shutdown_connections 2024-09-13T17:17:27.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:27.456+0000 7fdaab400640 1 -- 172.21.15.28:0/315503954 wait complete. 2024-09-13T17:17:27.508 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:17:27.507+0000 7f8065400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:17:27.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 122, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:17:21.964025+0000', 'modified': '2024-09-13T17:17:24.982429+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8904}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8904': {'gid': 8904, 'name': 'c', 'rank': 0, 'incarnation': 118, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/4121307324', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 4121307324}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 4121307324}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:17:27.877 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:17:27.877 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:17:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.037+0000 7fed03600640 1 Processor -- start 2024-09-13T17:17:28.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.037+0000 7fed03600640 1 -- start start 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed03600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 0x7fecfc06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed03600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc06c8b0 con 0x7fecfc071490 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed03600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecfc06cb90 con 0x7fecfc15a780 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed03600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc0cc980 con 0x7fecfc15ab60 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed01600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fecfc071490 0x7fecfc06c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58484/0 (socket says 172.21.15.28:58484) 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fed01600640 1 -- 172.21.15.28:0/2747156672 learned_addr learned my addr 172.21.15.28:0/2747156672 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:28.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2747156672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 0x7fecfc06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044630532 0 0) 0x7fecfc06c8b0 con 0x7fecfc071490 2024-09-13T17:17:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.039+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece0003520 con 0x7fecfc071490 2024-09-13T17:17:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.040+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687175545 0 0) 0x7fecfc06cb90 con 0x7fecfc15a780 2024-09-13T17:17:28.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.040+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecfc06c8b0 con 0x7fecfc15a780 2024-09-13T17:17:28.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.040+0000 7fecf3e00640 1 -- 172.21.15.28:0/2747156672 >> v1:172.21.15.189:6789/0 conn(0x7fecfc15a780 legacy=0x7fecfc070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.040+0000 7fecf3e00640 1 -- 172.21.15.28:0/2747156672 >> v1:172.21.15.28:6789/0 conn(0x7fecfc071490 legacy=0x7fecfc06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.040+0000 7fecf3e00640 1 -- 172.21.15.28:0/2747156672 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc0ccc60 con 0x7fecfc15ab60 2024-09-13T17:17:28.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2747156672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 0x7fecfc06bb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fece40097d0 tx=0x7fece402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=774938ff7356faf5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4032440 con 0x7fecfc15ab60 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece40325a0 con 0x7fecfc15ab60 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fecfbe00640 1 -- 172.21.15.28:0/2747156672 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4032930 con 0x7fecfc15ab60 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fed03600640 1 -- 172.21.15.28:0/2747156672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 msgr2=0x7fecfc06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.041+0000 7fed03600640 1 --2- 172.21.15.28:0/2747156672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 0x7fecfc06bb40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fece40097d0 tx=0x7fece402a200 comp rx=0 tx=0).stop 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.042+0000 7fed03600640 1 -- 172.21.15.28:0/2747156672 shutdown_connections 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.042+0000 7fed03600640 1 --2- 172.21.15.28:0/2747156672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15ab60 0x7fecfc06bb40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.042+0000 7fed03600640 1 -- 172.21.15.28:0/2747156672 >> 172.21.15.28:0/2747156672 conn(0x7fecfc0b0270 msgr2=0x7fecfc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.042+0000 7fed03600640 1 -- 172.21.15.28:0/2747156672 shutdown_connections 2024-09-13T17:17:28.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.042+0000 7fed03600640 1 -- 172.21.15.28:0/2747156672 wait complete. 2024-09-13T17:17:28.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.043+0000 7fed03600640 1 Processor -- start 2024-09-13T17:17:28.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.043+0000 7fed03600640 1 -- start start 2024-09-13T17:17:28.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 0x7fecfc0a86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 0x7fecfc13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14df80 con 0x7fecfc15ab60 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14e8e0 con 0x7fecfc071490 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.044+0000 7fed03600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc14d300 con 0x7fecfc15a780 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed00c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 0x7fecfc13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 0x7fecfc0a86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed00c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:52002/0 (socket says 172.21.15.28:52002) 2024-09-13T17:17:28.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed00c00640 1 -- 172.21.15.28:0/2002991041 learned_addr learned my addr 172.21.15.28:0/2002991041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 -- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 msgr2=0x7fecfc0a86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 0x7fecfc0a86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 -- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 msgr2=0x7fecfc14e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed01600640 1 -- 172.21.15.28:0/2002991041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece4009400 con 0x7fecfc15ab60 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 0x7fecfc0a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.045+0000 7fed00c00640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fed01600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 0x7fecfc13aa80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fecfc06f3a0 tx=0x7fecf400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fecf9600640 1 -- 172.21.15.28:0/2002991041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf400aba0 con 0x7fecfc15ab60 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fecf9600640 1 -- 172.21.15.28:0/2002991041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf400ad00 con 0x7fecfc15ab60 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fecf9600640 1 -- 172.21.15.28:0/2002991041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4012620 con 0x7fecfc15ab60 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc14d5e0 con 0x7fecfc15ab60 2024-09-13T17:17:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.046+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc14dba0 con 0x7fecfc15ab60 2024-09-13T17:17:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.048+0000 7fecf9600640 1 -- 172.21.15.28:0/2002991041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fecf4007070 con 0x7fecfc15ab60 2024-09-13T17:17:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.049+0000 7fecf9600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 0x7fecd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.049+0000 7fecf9600640 1 -- 172.21.15.28:0/2002991041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecf4089120 con 0x7fecfc15ab60 2024-09-13T17:17:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.049+0000 7fed00c00640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 0x7fecd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.050+0000 7fed063f5740 1 Processor -- start 2024-09-13T17:17:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.050+0000 7fed063f5740 1 -- start start 2024-09-13T17:17:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.050+0000 7fed00c00640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 0x7fecd806d0c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fecec007fd0 tx=0x7fecec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e234170 0x56400e238860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 0x56400e2627f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e1a1ab0 con 0x56400e238da0 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e19f240 con 0x56400e27c690 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e19ab30 con 0x56400e234170 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45688/0 (socket says 172.21.15.28:45688) 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 -- 172.21.15.28:0/2319356605 learned_addr learned my addr 172.21.15.28:0/2319356605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed00c00640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e234170 0x56400e238860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed01600640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 0x56400e2627f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 -- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e234170 msgr2=0x56400e238860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e234170 0x56400e238860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 -- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 msgr2=0x56400e2627f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 0x56400e2627f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fecf3e00640 1 -- 172.21.15.28:0/2319356605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56400e165bd0 con 0x56400e238da0 2024-09-13T17:17:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.051+0000 7fed01600640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 0x56400e2627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:28.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.052+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fece40097d0 tx=0x7fece402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e7716467ed34740 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.052+0000 7fecf1600640 1 -- 172.21.15.28:0/2319356605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4032d40 con 0x56400e238da0 2024-09-13T17:17:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.052+0000 7fecf1600640 1 -- 172.21.15.28:0/2319356605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece4047410 con 0x56400e238da0 2024-09-13T17:17:28.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.052+0000 7fecf1600640 1 -- 172.21.15.28:0/2319356605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4032ea0 con 0x56400e238da0 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- 172.21.15.28:0/2319356605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 msgr2=0x56400e2622b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fece40097d0 tx=0x7fece402c000 comp rx=0 tx=0).stop 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- 172.21.15.28:0/2319356605 shutdown_connections 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e27c690 0x56400e2627f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e238da0 0x56400e2622b0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 --2- 172.21.15.28:0/2319356605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e234170 0x56400e238860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- 172.21.15.28:0/2319356605 >> 172.21.15.28:0/2319356605 conn(0x56400e234e00 msgr2=0x56400e27d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- 172.21.15.28:0/2319356605 shutdown_connections 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- 172.21.15.28:0/2319356605 wait complete. 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.053+0000 7fed063f5740 1 -- start start 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.054+0000 7fed063f5740 1 Processor -- start 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 0x56400e2646b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 0x56400e26dec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e13c260 con 0x56400e27c690 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e13bf00 con 0x56400e26da40 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400df01f20 con 0x56400e238da0 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed063f5740 10 client.0 _ll_register_callbacks cb 0x56400e1941a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45690/0 (socket says 172.21.15.28:45690) 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 -- 172.21.15.28:0/2684677096 learned_addr learned my addr 172.21.15.28:0/2684677096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed00c00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 0x56400e2646b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed01600640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 0x56400e26dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 -- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 msgr2=0x56400e2646b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 0x56400e2646b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 -- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 msgr2=0x56400e26dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 0x56400e26dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fecf3e00640 1 -- 172.21.15.28:0/2684677096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc0ccc60 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed00c00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 0x56400e2646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.055+0000 7fed01600640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 0x56400e26dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecf3e00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fece40098f0 tx=0x7fece4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=874de752f9e6ad53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece4004440 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fece4051af0 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56400e1c4cd0 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fece4006a20 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 10 client.9027 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x56400e1991b0 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56400e1c4cd0 con 0x56400e27c690 2024-09-13T17:17:28.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fece4051cb0 con 0x56400e27c690 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 10 client.9027 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 10 client.9027 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.056+0000 7fed063f5740 20 client.9027 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26b6f0 0x56400e26bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fed063f5740 4 client.9027 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56400de8e3c0 con 0x56400e26b6f0 2024-09-13T17:17:28.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fed00c00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26b6f0 0x56400e26bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fed00c00640 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26b6f0 0x56400e26bad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a593b1c870cb33 server_cookie=31ab1022e4b5495d in_seq=0 out_seq=0 2024-09-13T17:17:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.057+0000 7fecea000640 10 client.9027 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fecea000640 1 -- 172.21.15.28:0/2684677096 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56400de8e3c0 con 0x56400e26b6f0 2024-09-13T17:17:28.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fecea000640 10 client.9027 handle_command_reply: tid=0 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 client.9027 shutdown 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 msgr2=0x56400e27a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7fece40098f0 tx=0x7fece4004730 comp rx=0 tx=0).stop 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 shutdown_connections 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26b6f0 0x56400e26bad0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e26da40 0x56400e26dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fecea000640 0 client.9027 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e27a4c0 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 --2- 172.21.15.28:0/2684677096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e238da0 0x56400e2646b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.058+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 >> 172.21.15.28:0/2684677096 conn(0x56400e234170 msgr2=0x56400e280fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.059+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 shutdown_connections 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.059+0000 7fed063f5740 1 -- 172.21.15.28:0/2684677096 wait complete. 2024-09-13T17:17:28.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.059+0000 7fed063f5740 20 client.9027 trim_cache size 0 max 16384 2024-09-13T17:17:28.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.074+0000 7fed063f5740 1 Processor -- start 2024-09-13T17:17:28.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.074+0000 7fed063f5740 1 -- start start 2024-09-13T17:17:28.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e28feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e233480 con 0x56400e27c690 2024-09-13T17:17:28.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e26e0c0 con 0x56400e238da0 2024-09-13T17:17:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e2669b0 con 0x56400e26da40 2024-09-13T17:17:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45706/0 (socket says 172.21.15.28:45706) 2024-09-13T17:17:28.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.075+0000 7fed01600640 1 -- 172.21.15.28:0/3964018985 learned_addr learned my addr 172.21.15.28:0/3964018985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fecf3e00640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e28feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 -- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 msgr2=0x56400e28feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed00c00640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e28feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 -- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 msgr2=0x56400e293950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 -- 172.21.15.28:0/3964018985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56400e13b470 con 0x56400e27c690 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fecf3e00640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e28feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed00c00640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fed01600640 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fecf4087680 tx=0x7fecf4057a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db396cf4c15e4f2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fecf2000640 1 -- 172.21.15.28:0/3964018985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4060440 con 0x56400e27c690 2024-09-13T17:17:28.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fecf2000640 1 -- 172.21.15.28:0/3964018985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf406be90 con 0x56400e27c690 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.076+0000 7fecf2000640 1 -- 172.21.15.28:0/3964018985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf40564a0 con 0x56400e27c690 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 -- 172.21.15.28:0/3964018985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 msgr2=0x56400e2375e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7fecf4087680 tx=0x7fecf4057a60 comp rx=0 tx=0).stop 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 -- 172.21.15.28:0/3964018985 shutdown_connections 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2375e0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e28feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 --2- 172.21.15.28:0/3964018985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 -- 172.21.15.28:0/3964018985 >> 172.21.15.28:0/3964018985 conn(0x56400e234170 msgr2=0x56400e27f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 -- 172.21.15.28:0/3964018985 shutdown_connections 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.077+0000 7fed063f5740 1 -- 172.21.15.28:0/3964018985 wait complete. 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.078+0000 7fed063f5740 1 -- start start 2024-09-13T17:17:28.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.078+0000 7fed063f5740 1 Processor -- start 2024-09-13T17:17:28.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e2eabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e2ebb60 con 0x56400e27c690 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e279d70 con 0x56400e238da0 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e281ba0 con 0x56400e26da40 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed063f5740 10 client.0 _ll_register_callbacks cb 0x7fece4041070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed01600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45714/0 (socket says 172.21.15.28:45714) 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed01600640 1 -- 172.21.15.28:0/1336393226 learned_addr learned my addr 172.21.15.28:0/1336393226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed01600640 1 -- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 msgr2=0x56400e2eabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.079+0000 7fed01600640 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e2eabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed01600640 1 -- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 msgr2=0x56400e293440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed01600640 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed01600640 1 -- 172.21.15.28:0/1336393226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc14dba0 con 0x56400e27c690 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed01600640 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fecf4087730 tx=0x7fecf40014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e909fb6df88fb8f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4073070 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fecf40564a0 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fece4047410 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecf40871f0 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed063f5740 10 client.9033 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x56400e160980 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf407a070 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.080+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fecf4076030 con 0x56400e27c690 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 10 client.9033 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 10 client.9033 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 20 client.9033 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26bc80 0x56400e26c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 4 client.9033 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:17:28.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56400de9f6c0 con 0x56400e26bc80 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed00c00640 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26bc80 0x56400e26c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fed00c00640 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26bc80 0x56400e26c060 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d95344c06e4b406 server_cookie=381c7040c5aa72a5 in_seq=0 out_seq=0 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.081+0000 7fecea000640 10 client.9033 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fecea000640 1 -- 172.21.15.28:0/1336393226 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x56400de9f6c0 con 0x56400e26bc80 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fecea000640 10 client.9033 handle_command_reply: tid=0 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 client.9033 shutdown 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 msgr2=0x56400e2eb100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7fecf4087730 tx=0x7fecf40014d0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 shutdown_connections 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x56400e26bc80 0x56400e26c060 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400e27c690 0x56400e2eb100 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x56400e26da40 0x56400e2eabc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 --2- 172.21.15.28:0/1336393226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56400e238da0 0x56400e293440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fecea000640 0 client.9033 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 >> 172.21.15.28:0/1336393226 conn(0x56400e234170 msgr2=0x56400e265f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 shutdown_connections 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 1 -- 172.21.15.28:0/1336393226 wait complete. 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.082+0000 7fed063f5740 20 client.9033 trim_cache size 0 max 16384 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:17:28.087 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 8904, 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1157912239999996, 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 122, 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 139, 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 139, 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.246754374 2024-09-13T17:17:28.088 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 msgr2=0x7fecd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 0x7fecd806d0c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fecec007fd0 tx=0x7fecec009040 comp rx=0 tx=0).stop 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 msgr2=0x7fecfc13aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 0x7fecfc13aa80 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fecfc06f3a0 tx=0x7fecf400cf00 comp rx=0 tx=0).stop 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 shutdown_connections 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecd806ac00 0x7fecd806d0c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecfc15ab60 0x7fecfc13aa80 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecfc15a780 0x7fecfc0a86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 --2- 172.21.15.28:0/2002991041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecfc071490 0x7fecfc14e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 >> 172.21.15.28:0/2002991041 conn(0x7fecfc0b0270 msgr2=0x7fecfc1598d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 shutdown_connections 2024-09-13T17:17:28.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.083+0000 7fed03600640 1 -- 172.21.15.28:0/2002991041 wait complete. 2024-09-13T17:17:28.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:17:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.253+0000 7ff5de000640 1 Processor -- start 2024-09-13T17:17:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.254+0000 7ff5de000640 1 -- start start 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5de000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159440 0x7ff5d80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5de000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff5d80b29f0 0x7ff5d80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5de000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5d80c3850 con 0x7ff5d8159820 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5de000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5d815a930 con 0x7ff5d80b29f0 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5de000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8159d20 con 0x7ff5d8159440 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5dd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff5d80b29f0 0x7ff5d80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5cfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff5d8159820 0x7ff5d80b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58494/0 (socket says 172.21.15.28:58494) 2024-09-13T17:17:28.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5cfe00640 1 -- 172.21.15.28:0/2925017023 learned_addr learned my addr 172.21.15.28:0/2925017023 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.255+0000 7ff5dcc00640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159440 0x7ff5d80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.256+0000 7ff5dd600640 1 -- 172.21.15.28:0/2925017023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159440 msgr2=0x7ff5d80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.256+0000 7ff5dd600640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159440 0x7ff5d80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.256+0000 7ff5dd600640 1 -- 172.21.15.28:0/2925017023 >> v1:172.21.15.28:6789/0 conn(0x7ff5d8159820 legacy=0x7ff5d80b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.256+0000 7ff5dd600640 1 -- 172.21.15.28:0/2925017023 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d815a000 con 0x7ff5d80b29f0 2024-09-13T17:17:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5dd600640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d80b29f0 0x7ff5d80b0980 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009d90 tx=0x7ff5d002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d5179ed0af3e202 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5d7e00640 1 -- 172.21.15.28:0/2925017023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0007740 con 0x7ff5d80b29f0 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5d7e00640 1 -- 172.21.15.28:0/2925017023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d002d050 con 0x7ff5d80b29f0 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5d7e00640 1 -- 172.21.15.28:0/2925017023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0007ac0 con 0x7ff5d80b29f0 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5de000640 1 -- 172.21.15.28:0/2925017023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d80b29f0 msgr2=0x7ff5d80b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.257+0000 7ff5de000640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d80b29f0 0x7ff5d80b0980 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009d90 tx=0x7ff5d002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 -- 172.21.15.28:0/2925017023 shutdown_connections 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d80b29f0 0x7ff5d80b0980 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 --2- 172.21.15.28:0/2925017023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159440 0x7ff5d80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 -- 172.21.15.28:0/2925017023 >> 172.21.15.28:0/2925017023 conn(0x7ff5d8072e40 msgr2=0x7ff5d8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 -- 172.21.15.28:0/2925017023 shutdown_connections 2024-09-13T17:17:28.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.258+0000 7ff5de000640 1 -- 172.21.15.28:0/2925017023 wait complete. 2024-09-13T17:17:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.259+0000 7ff5de000640 1 Processor -- start 2024-09-13T17:17:28.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.260+0000 7ff5de000640 1 -- start start 2024-09-13T17:17:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.260+0000 7ff5de000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5d80b29f0 0x7ff5d814b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.260+0000 7ff5de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5de000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 0x7ff5d80ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5de000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8157270 con 0x7ff5d8159440 2024-09-13T17:17:28.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5de000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d8156310 con 0x7ff5d80b29f0 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5de000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d81565f0 con 0x7ff5d8159820 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45724/0 (socket says 172.21.15.28:45724) 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 -- 172.21.15.28:0/3770534797 learned_addr learned my addr 172.21.15.28:0/3770534797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5dcc00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5d80b29f0 0x7ff5d814b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5dd600640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 0x7ff5d80ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 -- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 msgr2=0x7ff5d80ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 0x7ff5d80ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 -- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5d80b29f0 msgr2=0x7ff5d814b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5d80b29f0 0x7ff5d814b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5cfe00640 1 -- 172.21.15.28:0/3770534797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d0009a70 con 0x7ff5d8159440 2024-09-13T17:17:28.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.261+0000 7ff5dd600640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 0x7ff5d80ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:28.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5cfe00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000c480 tx=0x7ff5c000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c0005ce0 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5c0005e40 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c00072f0 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d81568d0 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.262+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d8156e90 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.263+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5d80b22e0 con 0x7ff5d8159440 2024-09-13T17:17:28.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.264+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5c0007450 con 0x7ff5d8159440 2024-09-13T17:17:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.265+0000 7ff5d5600640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 0x7ff5b406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:28.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.265+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5c0088850 con 0x7ff5d8159440 2024-09-13T17:17:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.266+0000 7ff5dcc00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 0x7ff5b406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:28.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.267+0000 7ff5dcc00640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 0x7ff5b406d0f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009b40 tx=0x7ff5c8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:28.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.270+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff5c0053ad0 con 0x7ff5d8159440 2024-09-13T17:17:28.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.483+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff5d80af640 con 0x7ff5d8159440 2024-09-13T17:17:28.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.483+0000 7ff5d5600640 1 -- 172.21.15.28:0/3770534797 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v122) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff5c0017070 con 0x7ff5d8159440 2024-09-13T17:17:28.484 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 msgr2=0x7ff5b406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 0x7ff5b406d0f0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7ff5c8009b40 tx=0x7ff5c8009190 comp rx=0 tx=0).stop 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 msgr2=0x7ff5d80aed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000c480 tx=0x7ff5c000c940 comp rx=0 tx=0).stop 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 shutdown_connections 2024-09-13T17:17:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5b406ac30 0x7ff5b406d0f0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5d8159820 0x7ff5d80ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5d8159440 0x7ff5d80aed10 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 --2- 172.21.15.28:0/3770534797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5d80b29f0 0x7ff5d814b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 >> 172.21.15.28:0/3770534797 conn(0x7ff5d8072e40 msgr2=0x7ff5d815ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 shutdown_connections 2024-09-13T17:17:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:28.486+0000 7ff5de000640 1 -- 172.21.15.28:0/3770534797 wait complete. 2024-09-13T17:17:28.502 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:17:28.503 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:17:28.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.503 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:17:28.539 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:17:28.539 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:17:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:17:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:17:28.540 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42133sec preferred_lft 42133sec 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.541 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:17:28.542 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:17:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:17:28.543 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:17:28.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.543 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:17:28.633 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:17:28.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.636 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:17:28.672 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:17:28.672 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:17:28.672 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:17:28.672 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:17:28.672 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:17:28.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:17:28.673 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:28.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.820 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:17:28.855 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:17:28.855 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:17:28.855 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:17:28.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:28.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:17:28.952 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:17:28.952 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:28.987 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:17:28.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:17:28.987 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:29.063 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:17:29.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:29.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:17:29.192 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:17:29.193 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:17:29.219 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:17:29.219 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:17:29.219 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:17:29.219 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:17:29.219 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:17:29.221 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:17:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:17:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:17:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:17:29.221 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:17:29.224 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:17:29.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:29.224 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:29.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:29.281 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:17:29.326 INFO:teuthology.orchestra.run.smithi028.stdout:client9045 2024-09-13T17:17:29.327 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:29.327 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:17:29.403 INFO:tasks.cephfs.test_scrub_checks:mountpoint: /home/ubuntu/cephtest/mnt.0 2024-09-13T17:17:29.403 INFO:tasks.cephfs.test_scrub_checks:client_path: /home/ubuntu/cephtest/mnt.0/scrub_control_test_path 2024-09-13T17:17:29.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:29.404 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:29.404 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:29.404 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.0 2024-09-13T17:17:29.405 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:29.405 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:29.405 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:29.405 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171026 s, 61.3 MB/s 2024-09-13T17:17:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174807 s, 60.0 MB/s 2024-09-13T17:17:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151601 s, 69.2 MB/s 2024-09-13T17:17:29.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.564 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146089 s, 71.8 MB/s 2024-09-13T17:17:29.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169987 s, 61.7 MB/s 2024-09-13T17:17:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153088 s, 68.5 MB/s 2024-09-13T17:17:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140112 s, 74.8 MB/s 2024-09-13T17:17:29.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.635 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149445 s, 70.2 MB/s 2024-09-13T17:17:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144566 s, 72.5 MB/s 2024-09-13T17:17:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140678 s, 74.5 MB/s 2024-09-13T17:17:29.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.684 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.684 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140341 s, 74.7 MB/s 2024-09-13T17:17:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147638 s, 71.0 MB/s 2024-09-13T17:17:29.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163672 s, 64.1 MB/s 2024-09-13T17:17:29.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141352 s, 74.2 MB/s 2024-09-13T17:17:29.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.752 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138591 s, 75.7 MB/s 2024-09-13T17:17:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136771 s, 76.7 MB/s 2024-09-13T17:17:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131765 s, 79.6 MB/s 2024-09-13T17:17:29.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.799 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136705 s, 76.7 MB/s 2024-09-13T17:17:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155791 s, 67.3 MB/s 2024-09-13T17:17:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.835 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158628 s, 66.1 MB/s 2024-09-13T17:17:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.853 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157424 s, 66.6 MB/s 2024-09-13T17:17:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016353 s, 64.1 MB/s 2024-09-13T17:17:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.888 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142035 s, 73.8 MB/s 2024-09-13T17:17:29.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013769 s, 76.2 MB/s 2024-09-13T17:17:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.921 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147733 s, 71.0 MB/s 2024-09-13T17:17:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.937 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137084 s, 76.5 MB/s 2024-09-13T17:17:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148794 s, 70.5 MB/s 2024-09-13T17:17:29.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189203 s, 55.4 MB/s 2024-09-13T17:17:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:29.992 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014635 s, 71.6 MB/s 2024-09-13T17:17:30.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.011 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.011 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155231 s, 67.5 MB/s 2024-09-13T17:17:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132676 s, 79.0 MB/s 2024-09-13T17:17:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138186 s, 75.9 MB/s 2024-09-13T17:17:30.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.1 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:30.044 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:30.045 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152158 s, 68.9 MB/s 2024-09-13T17:17:30.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172452 s, 60.8 MB/s 2024-09-13T17:17:30.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123536 s, 84.9 MB/s 2024-09-13T17:17:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179589 s, 58.4 MB/s 2024-09-13T17:17:30.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143802 s, 72.9 MB/s 2024-09-13T17:17:30.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166907 s, 62.8 MB/s 2024-09-13T17:17:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160834 s, 65.2 MB/s 2024-09-13T17:17:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141555 s, 74.1 MB/s 2024-09-13T17:17:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.242 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144115 s, 72.8 MB/s 2024-09-13T17:17:30.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138585 s, 75.7 MB/s 2024-09-13T17:17:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128647 s, 81.5 MB/s 2024-09-13T17:17:30.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.290 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143398 s, 73.1 MB/s 2024-09-13T17:17:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.307 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148563 s, 70.6 MB/s 2024-09-13T17:17:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.326 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017189 s, 61.0 MB/s 2024-09-13T17:17:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135668 s, 77.3 MB/s 2024-09-13T17:17:30.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01409 s, 74.4 MB/s 2024-09-13T17:17:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138192 s, 75.9 MB/s 2024-09-13T17:17:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.390 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136554 s, 76.8 MB/s 2024-09-13T17:17:30.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.406 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130535 s, 80.3 MB/s 2024-09-13T17:17:30.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013662 s, 76.8 MB/s 2024-09-13T17:17:30.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017135 s, 61.2 MB/s 2024-09-13T17:17:30.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135138 s, 77.6 MB/s 2024-09-13T17:17:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.475 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166686 s, 62.9 MB/s 2024-09-13T17:17:30.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154092 s, 68.0 MB/s 2024-09-13T17:17:30.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.509 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135886 s, 77.2 MB/s 2024-09-13T17:17:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145561 s, 72.0 MB/s 2024-09-13T17:17:30.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137909 s, 76.0 MB/s 2024-09-13T17:17:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129839 s, 80.8 MB/s 2024-09-13T17:17:30.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.573 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135375 s, 77.5 MB/s 2024-09-13T17:17:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.588 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131112 s, 80.0 MB/s 2024-09-13T17:17:30.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015463 s, 67.8 MB/s 2024-09-13T17:17:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164729 s, 63.7 MB/s 2024-09-13T17:17:30.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.2 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:30.626 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.682 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135369 s, 77.5 MB/s 2024-09-13T17:17:30.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.700 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.700 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164086 s, 63.9 MB/s 2024-09-13T17:17:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140882 s, 74.4 MB/s 2024-09-13T17:17:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128164 s, 81.8 MB/s 2024-09-13T17:17:30.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.747 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136784 s, 76.7 MB/s 2024-09-13T17:17:30.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158225 s, 66.3 MB/s 2024-09-13T17:17:30.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.781 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132802 s, 79.0 MB/s 2024-09-13T17:17:30.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012751 s, 82.2 MB/s 2024-09-13T17:17:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137786 s, 76.1 MB/s 2024-09-13T17:17:30.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.828 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135615 s, 77.3 MB/s 2024-09-13T17:17:30.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138415 s, 75.8 MB/s 2024-09-13T17:17:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138169 s, 75.9 MB/s 2024-09-13T17:17:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165307 s, 63.4 MB/s 2024-09-13T17:17:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131734 s, 79.6 MB/s 2024-09-13T17:17:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168769 s, 62.1 MB/s 2024-09-13T17:17:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165236 s, 63.5 MB/s 2024-09-13T17:17:30.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.947 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128942 s, 81.3 MB/s 2024-09-13T17:17:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.966 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161369 s, 65.0 MB/s 2024-09-13T17:17:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01679 s, 62.5 MB/s 2024-09-13T17:17:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:30.999 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126646 s, 82.8 MB/s 2024-09-13T17:17:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163523 s, 64.1 MB/s 2024-09-13T17:17:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131508 s, 79.7 MB/s 2024-09-13T17:17:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157146 s, 66.7 MB/s 2024-09-13T17:17:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134053 s, 78.2 MB/s 2024-09-13T17:17:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149528 s, 70.1 MB/s 2024-09-13T17:17:31.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140043 s, 74.9 MB/s 2024-09-13T17:17:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129017 s, 81.3 MB/s 2024-09-13T17:17:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.131 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132013 s, 79.4 MB/s 2024-09-13T17:17:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.146 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128223 s, 81.8 MB/s 2024-09-13T17:17:31.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131542 s, 79.7 MB/s 2024-09-13T17:17:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.181 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172351 s, 60.8 MB/s 2024-09-13T17:17:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135144 s, 77.6 MB/s 2024-09-13T17:17:31.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.3 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:31.199 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138158 s, 75.9 MB/s 2024-09-13T17:17:31.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123508 s, 84.9 MB/s 2024-09-13T17:17:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132242 s, 79.3 MB/s 2024-09-13T17:17:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.299 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128915 s, 81.3 MB/s 2024-09-13T17:17:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125983 s, 83.2 MB/s 2024-09-13T17:17:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144835 s, 72.4 MB/s 2024-09-13T17:17:31.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167736 s, 62.5 MB/s 2024-09-13T17:17:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.369 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166235 s, 63.1 MB/s 2024-09-13T17:17:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136002 s, 77.1 MB/s 2024-09-13T17:17:31.402 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164507 s, 63.7 MB/s 2024-09-13T17:17:31.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168735 s, 62.1 MB/s 2024-09-13T17:17:31.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.437 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127978 s, 81.9 MB/s 2024-09-13T17:17:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.453 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140987 s, 74.4 MB/s 2024-09-13T17:17:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.469 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135131 s, 77.6 MB/s 2024-09-13T17:17:31.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128502 s, 81.6 MB/s 2024-09-13T17:17:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158947 s, 66.0 MB/s 2024-09-13T17:17:31.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141194 s, 74.3 MB/s 2024-09-13T17:17:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144102 s, 72.8 MB/s 2024-09-13T17:17:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138212 s, 75.9 MB/s 2024-09-13T17:17:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.568 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144811 s, 72.4 MB/s 2024-09-13T17:17:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172815 s, 60.7 MB/s 2024-09-13T17:17:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129325 s, 81.1 MB/s 2024-09-13T17:17:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136095 s, 77.0 MB/s 2024-09-13T17:17:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151078 s, 69.4 MB/s 2024-09-13T17:17:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142919 s, 73.4 MB/s 2024-09-13T17:17:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177472 s, 59.1 MB/s 2024-09-13T17:17:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153675 s, 68.2 MB/s 2024-09-13T17:17:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161428 s, 65.0 MB/s 2024-09-13T17:17:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138222 s, 75.9 MB/s 2024-09-13T17:17:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178052 s, 58.9 MB/s 2024-09-13T17:17:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165013 s, 63.5 MB/s 2024-09-13T17:17:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.781 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155552 s, 67.4 MB/s 2024-09-13T17:17:31.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.4 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:31.782 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130322 s, 80.5 MB/s 2024-09-13T17:17:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205411 s, 51.0 MB/s 2024-09-13T17:17:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145074 s, 72.3 MB/s 2024-09-13T17:17:31.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.896 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171007 s, 61.3 MB/s 2024-09-13T17:17:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113802 s, 92.1 MB/s 2024-09-13T17:17:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134308 s, 78.1 MB/s 2024-09-13T17:17:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130779 s, 80.2 MB/s 2024-09-13T17:17:31.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158184 s, 66.3 MB/s 2024-09-13T17:17:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133634 s, 78.5 MB/s 2024-09-13T17:17:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131046 s, 80.0 MB/s 2024-09-13T17:17:32.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140367 s, 74.7 MB/s 2024-09-13T17:17:32.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.022 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140052 s, 74.9 MB/s 2024-09-13T17:17:32.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148988 s, 70.4 MB/s 2024-09-13T17:17:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151076 s, 69.4 MB/s 2024-09-13T17:17:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013652 s, 76.8 MB/s 2024-09-13T17:17:32.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.088 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131784 s, 79.6 MB/s 2024-09-13T17:17:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136512 s, 76.8 MB/s 2024-09-13T17:17:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015447 s, 67.9 MB/s 2024-09-13T17:17:32.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162072 s, 64.7 MB/s 2024-09-13T17:17:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138501 s, 75.7 MB/s 2024-09-13T17:17:32.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129381 s, 81.0 MB/s 2024-09-13T17:17:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.186 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129342 s, 81.1 MB/s 2024-09-13T17:17:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130138 s, 80.6 MB/s 2024-09-13T17:17:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137691 s, 76.2 MB/s 2024-09-13T17:17:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137838 s, 76.1 MB/s 2024-09-13T17:17:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129522 s, 81.0 MB/s 2024-09-13T17:17:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132679 s, 79.0 MB/s 2024-09-13T17:17:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132741 s, 79.0 MB/s 2024-09-13T17:17:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132021 s, 79.4 MB/s 2024-09-13T17:17:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158919 s, 66.0 MB/s 2024-09-13T17:17:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132301 s, 79.3 MB/s 2024-09-13T17:17:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012899 s, 81.3 MB/s 2024-09-13T17:17:32.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.5 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:32.346 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154063 s, 68.1 MB/s 2024-09-13T17:17:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167039 s, 62.8 MB/s 2024-09-13T17:17:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.437 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132984 s, 78.8 MB/s 2024-09-13T17:17:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166858 s, 62.8 MB/s 2024-09-13T17:17:32.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134693 s, 77.8 MB/s 2024-09-13T17:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132237 s, 79.3 MB/s 2024-09-13T17:17:32.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.505 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152835 s, 68.6 MB/s 2024-09-13T17:17:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135924 s, 77.1 MB/s 2024-09-13T17:17:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.539 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159053 s, 65.9 MB/s 2024-09-13T17:17:32.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.555 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133243 s, 78.7 MB/s 2024-09-13T17:17:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.572 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156878 s, 66.8 MB/s 2024-09-13T17:17:32.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127772 s, 82.1 MB/s 2024-09-13T17:17:32.603 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141191 s, 74.3 MB/s 2024-09-13T17:17:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160531 s, 65.3 MB/s 2024-09-13T17:17:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.640 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159273 s, 65.8 MB/s 2024-09-13T17:17:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124815 s, 84.0 MB/s 2024-09-13T17:17:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132743 s, 79.0 MB/s 2024-09-13T17:17:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158532 s, 66.1 MB/s 2024-09-13T17:17:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.704 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129892 s, 80.7 MB/s 2024-09-13T17:17:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126924 s, 82.6 MB/s 2024-09-13T17:17:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135129 s, 77.6 MB/s 2024-09-13T17:17:32.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015039 s, 69.7 MB/s 2024-09-13T17:17:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169402 s, 61.9 MB/s 2024-09-13T17:17:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129272 s, 81.1 MB/s 2024-09-13T17:17:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169127 s, 62.0 MB/s 2024-09-13T17:17:32.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137547 s, 76.2 MB/s 2024-09-13T17:17:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152423 s, 68.8 MB/s 2024-09-13T17:17:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136861 s, 76.6 MB/s 2024-09-13T17:17:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.873 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161577 s, 64.9 MB/s 2024-09-13T17:17:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.889 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132057 s, 79.4 MB/s 2024-09-13T17:17:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147253 s, 71.2 MB/s 2024-09-13T17:17:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174458 s, 60.1 MB/s 2024-09-13T17:17:32.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.6 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:32.927 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:32.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:32.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:32.983 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013536 s, 77.5 MB/s 2024-09-13T17:17:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.000 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016208 s, 64.7 MB/s 2024-09-13T17:17:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144486 s, 72.6 MB/s 2024-09-13T17:17:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130497 s, 80.4 MB/s 2024-09-13T17:17:33.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162456 s, 64.5 MB/s 2024-09-13T17:17:33.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138558 s, 75.7 MB/s 2024-09-13T17:17:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.085 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160136 s, 65.5 MB/s 2024-09-13T17:17:33.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.103 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145509 s, 72.1 MB/s 2024-09-13T17:17:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162309 s, 64.6 MB/s 2024-09-13T17:17:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.139 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161527 s, 64.9 MB/s 2024-09-13T17:17:33.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130584 s, 80.3 MB/s 2024-09-13T17:17:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132465 s, 79.2 MB/s 2024-09-13T17:17:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139468 s, 75.2 MB/s 2024-09-13T17:17:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167068 s, 62.8 MB/s 2024-09-13T17:17:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014036 s, 74.7 MB/s 2024-09-13T17:17:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155127 s, 67.6 MB/s 2024-09-13T17:17:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132574 s, 79.1 MB/s 2024-09-13T17:17:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126118 s, 83.1 MB/s 2024-09-13T17:17:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.285 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133406 s, 78.6 MB/s 2024-09-13T17:17:33.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.300 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130462 s, 80.4 MB/s 2024-09-13T17:17:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.316 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135219 s, 77.5 MB/s 2024-09-13T17:17:33.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013547 s, 77.4 MB/s 2024-09-13T17:17:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138227 s, 75.9 MB/s 2024-09-13T17:17:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.363 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137527 s, 76.2 MB/s 2024-09-13T17:17:33.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151188 s, 69.4 MB/s 2024-09-13T17:17:33.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160973 s, 65.1 MB/s 2024-09-13T17:17:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130699 s, 80.2 MB/s 2024-09-13T17:17:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169431 s, 61.9 MB/s 2024-09-13T17:17:33.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.449 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134353 s, 78.0 MB/s 2024-09-13T17:17:33.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.465 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137411 s, 76.3 MB/s 2024-09-13T17:17:33.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169759 s, 61.8 MB/s 2024-09-13T17:17:33.499 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134778 s, 77.8 MB/s 2024-09-13T17:17:33.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:33.501 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.7 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:33.502 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152138 s, 68.9 MB/s 2024-09-13T17:17:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158404 s, 66.2 MB/s 2024-09-13T17:17:33.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129211 s, 81.2 MB/s 2024-09-13T17:17:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170217 s, 61.6 MB/s 2024-09-13T17:17:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.627 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141167 s, 74.3 MB/s 2024-09-13T17:17:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011364 s, 92.3 MB/s 2024-09-13T17:17:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137114 s, 76.5 MB/s 2024-09-13T17:17:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140132 s, 74.8 MB/s 2024-09-13T17:17:33.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016452 s, 63.7 MB/s 2024-09-13T17:17:33.707 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136153 s, 77.0 MB/s 2024-09-13T17:17:33.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169654 s, 61.8 MB/s 2024-09-13T17:17:33.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.742 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135699 s, 77.3 MB/s 2024-09-13T17:17:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135053 s, 77.6 MB/s 2024-09-13T17:17:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166914 s, 62.8 MB/s 2024-09-13T17:17:33.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.793 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135481 s, 77.4 MB/s 2024-09-13T17:17:33.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159624 s, 65.7 MB/s 2024-09-13T17:17:33.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.830 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01667 s, 62.9 MB/s 2024-09-13T17:17:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161123 s, 65.1 MB/s 2024-09-13T17:17:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136897 s, 76.6 MB/s 2024-09-13T17:17:33.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126393 s, 83.0 MB/s 2024-09-13T17:17:33.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.895 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133942 s, 78.3 MB/s 2024-09-13T17:17:33.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.911 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135552 s, 77.4 MB/s 2024-09-13T17:17:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130828 s, 80.1 MB/s 2024-09-13T17:17:33.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.942 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132399 s, 79.2 MB/s 2024-09-13T17:17:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131406 s, 79.8 MB/s 2024-09-13T17:17:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131041 s, 80.0 MB/s 2024-09-13T17:17:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133943 s, 78.3 MB/s 2024-09-13T17:17:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161475 s, 64.9 MB/s 2024-09-13T17:17:34.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.023 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142369 s, 73.7 MB/s 2024-09-13T17:17:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138384 s, 75.8 MB/s 2024-09-13T17:17:34.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136993 s, 76.5 MB/s 2024-09-13T17:17:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135691 s, 77.3 MB/s 2024-09-13T17:17:34.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:34.072 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:34.072 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:34.072 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.8 2024-09-13T17:17:34.072 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:34.073 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:34.073 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:34.073 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140969 s, 74.4 MB/s 2024-09-13T17:17:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133408 s, 78.6 MB/s 2024-09-13T17:17:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132448 s, 79.2 MB/s 2024-09-13T17:17:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128343 s, 81.7 MB/s 2024-09-13T17:17:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134199 s, 78.1 MB/s 2024-09-13T17:17:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127818 s, 82.0 MB/s 2024-09-13T17:17:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127148 s, 82.5 MB/s 2024-09-13T17:17:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162264 s, 64.6 MB/s 2024-09-13T17:17:34.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129305 s, 81.1 MB/s 2024-09-13T17:17:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139035 s, 75.4 MB/s 2024-09-13T17:17:34.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134199 s, 78.1 MB/s 2024-09-13T17:17:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131488 s, 79.7 MB/s 2024-09-13T17:17:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168483 s, 62.2 MB/s 2024-09-13T17:17:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131763 s, 79.6 MB/s 2024-09-13T17:17:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.351 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131984 s, 79.4 MB/s 2024-09-13T17:17:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.369 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159503 s, 65.7 MB/s 2024-09-13T17:17:34.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167371 s, 62.6 MB/s 2024-09-13T17:17:34.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138045 s, 76.0 MB/s 2024-09-13T17:17:34.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131393 s, 79.8 MB/s 2024-09-13T17:17:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.437 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015097 s, 69.5 MB/s 2024-09-13T17:17:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158172 s, 66.3 MB/s 2024-09-13T17:17:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129044 s, 81.3 MB/s 2024-09-13T17:17:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124425 s, 84.3 MB/s 2024-09-13T17:17:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131229 s, 79.9 MB/s 2024-09-13T17:17:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.516 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132955 s, 78.9 MB/s 2024-09-13T17:17:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014155 s, 74.1 MB/s 2024-09-13T17:17:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129676 s, 80.9 MB/s 2024-09-13T17:17:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130325 s, 80.5 MB/s 2024-09-13T17:17:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.581 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153712 s, 68.2 MB/s 2024-09-13T17:17:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012951 s, 81.0 MB/s 2024-09-13T17:17:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134223 s, 78.1 MB/s 2024-09-13T17:17:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136683 s, 76.7 MB/s 2024-09-13T17:17:34.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:34.629 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:34.629 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:34.629 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.9 2024-09-13T17:17:34.629 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:34.630 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:34.630 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:34.630 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:34.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.683 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126838 s, 82.7 MB/s 2024-09-13T17:17:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131034 s, 80.0 MB/s 2024-09-13T17:17:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137043 s, 76.5 MB/s 2024-09-13T17:17:34.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137005 s, 76.5 MB/s 2024-09-13T17:17:34.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170126 s, 61.6 MB/s 2024-09-13T17:17:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.768 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160714 s, 65.2 MB/s 2024-09-13T17:17:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015672 s, 66.9 MB/s 2024-09-13T17:17:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133639 s, 78.5 MB/s 2024-09-13T17:17:34.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133187 s, 78.7 MB/s 2024-09-13T17:17:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133901 s, 78.3 MB/s 2024-09-13T17:17:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135131 s, 77.6 MB/s 2024-09-13T17:17:34.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132321 s, 79.2 MB/s 2024-09-13T17:17:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177786 s, 59.0 MB/s 2024-09-13T17:17:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016343 s, 64.2 MB/s 2024-09-13T17:17:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133372 s, 78.6 MB/s 2024-09-13T17:17:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013661 s, 76.8 MB/s 2024-09-13T17:17:34.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137103 s, 76.5 MB/s 2024-09-13T17:17:34.968 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169007 s, 62.0 MB/s 2024-09-13T17:17:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:34.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143931 s, 72.9 MB/s 2024-09-13T17:17:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157547 s, 66.6 MB/s 2024-09-13T17:17:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135648 s, 77.3 MB/s 2024-09-13T17:17:35.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.038 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162826 s, 64.4 MB/s 2024-09-13T17:17:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.053 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132697 s, 79.0 MB/s 2024-09-13T17:17:35.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133912 s, 78.3 MB/s 2024-09-13T17:17:35.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128988 s, 81.3 MB/s 2024-09-13T17:17:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139259 s, 75.3 MB/s 2024-09-13T17:17:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014055 s, 74.6 MB/s 2024-09-13T17:17:35.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170963 s, 61.3 MB/s 2024-09-13T17:17:35.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133779 s, 78.4 MB/s 2024-09-13T17:17:35.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01368 s, 76.7 MB/s 2024-09-13T17:17:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.181 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115981 s, 90.4 MB/s 2024-09-13T17:17:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117525 s, 89.2 MB/s 2024-09-13T17:17:35.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:35.196 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:35.196 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:35.196 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.10 2024-09-13T17:17:35.196 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:35.196 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:35.197 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:35.197 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138729 s, 75.6 MB/s 2024-09-13T17:17:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.267 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012888 s, 81.4 MB/s 2024-09-13T17:17:35.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147932 s, 70.9 MB/s 2024-09-13T17:17:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131297 s, 79.9 MB/s 2024-09-13T17:17:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129431 s, 81.0 MB/s 2024-09-13T17:17:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128356 s, 81.7 MB/s 2024-09-13T17:17:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.345 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132828 s, 78.9 MB/s 2024-09-13T17:17:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.360 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129902 s, 80.7 MB/s 2024-09-13T17:17:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131935 s, 79.5 MB/s 2024-09-13T17:17:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.390 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012894 s, 81.3 MB/s 2024-09-13T17:17:35.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172801 s, 60.7 MB/s 2024-09-13T17:17:35.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158595 s, 66.1 MB/s 2024-09-13T17:17:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158722 s, 66.1 MB/s 2024-09-13T17:17:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129895 s, 80.7 MB/s 2024-09-13T17:17:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131275 s, 79.9 MB/s 2024-09-13T17:17:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128712 s, 81.5 MB/s 2024-09-13T17:17:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132293 s, 79.3 MB/s 2024-09-13T17:17:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.523 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134232 s, 78.1 MB/s 2024-09-13T17:17:35.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171581 s, 61.1 MB/s 2024-09-13T17:17:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133918 s, 78.3 MB/s 2024-09-13T17:17:35.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013221 s, 79.3 MB/s 2024-09-13T17:17:35.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.589 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.589 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129013 s, 81.3 MB/s 2024-09-13T17:17:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150525 s, 69.7 MB/s 2024-09-13T17:17:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156036 s, 67.2 MB/s 2024-09-13T17:17:35.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160799 s, 65.2 MB/s 2024-09-13T17:17:35.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132149 s, 79.3 MB/s 2024-09-13T17:17:35.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128136 s, 81.8 MB/s 2024-09-13T17:17:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129702 s, 80.8 MB/s 2024-09-13T17:17:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.704 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139277 s, 75.3 MB/s 2024-09-13T17:17:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131535 s, 79.7 MB/s 2024-09-13T17:17:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01264 s, 83.0 MB/s 2024-09-13T17:17:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134342 s, 78.1 MB/s 2024-09-13T17:17:35.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.11 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:35.751 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131553 s, 79.7 MB/s 2024-09-13T17:17:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127321 s, 82.4 MB/s 2024-09-13T17:17:35.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127544 s, 82.2 MB/s 2024-09-13T17:17:35.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.852 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013612 s, 77.0 MB/s 2024-09-13T17:17:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128932 s, 81.3 MB/s 2024-09-13T17:17:35.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174232 s, 60.2 MB/s 2024-09-13T17:17:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164574 s, 63.7 MB/s 2024-09-13T17:17:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127329 s, 82.4 MB/s 2024-09-13T17:17:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016832 s, 62.3 MB/s 2024-09-13T17:17:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143143 s, 73.3 MB/s 2024-09-13T17:17:35.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167396 s, 62.6 MB/s 2024-09-13T17:17:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:35.989 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125445 s, 83.6 MB/s 2024-09-13T17:17:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.009 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174014 s, 60.3 MB/s 2024-09-13T17:17:36.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012326 s, 85.1 MB/s 2024-09-13T17:17:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142792 s, 73.4 MB/s 2024-09-13T17:17:36.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111783 s, 93.8 MB/s 2024-09-13T17:17:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131169 s, 79.9 MB/s 2024-09-13T17:17:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138856 s, 75.5 MB/s 2024-09-13T17:17:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017105 s, 61.3 MB/s 2024-09-13T17:17:36.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137759 s, 76.1 MB/s 2024-09-13T17:17:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136963 s, 76.6 MB/s 2024-09-13T17:17:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115906 s, 90.5 MB/s 2024-09-13T17:17:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171762 s, 61.0 MB/s 2024-09-13T17:17:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.189 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016819 s, 62.3 MB/s 2024-09-13T17:17:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133835 s, 78.3 MB/s 2024-09-13T17:17:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017903 s, 58.6 MB/s 2024-09-13T17:17:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129829 s, 80.8 MB/s 2024-09-13T17:17:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137441 s, 76.3 MB/s 2024-09-13T17:17:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.274 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163601 s, 64.1 MB/s 2024-09-13T17:17:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134063 s, 78.2 MB/s 2024-09-13T17:17:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.308 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160344 s, 65.4 MB/s 2024-09-13T17:17:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139191 s, 75.3 MB/s 2024-09-13T17:17:36.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.12 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:36.326 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01288 s, 81.4 MB/s 2024-09-13T17:17:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016589 s, 63.2 MB/s 2024-09-13T17:17:36.418 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.419 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175727 s, 59.7 MB/s 2024-09-13T17:17:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135351 s, 77.5 MB/s 2024-09-13T17:17:36.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146602 s, 71.5 MB/s 2024-09-13T17:17:36.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162835 s, 64.4 MB/s 2024-09-13T17:17:36.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159338 s, 65.8 MB/s 2024-09-13T17:17:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168793 s, 62.1 MB/s 2024-09-13T17:17:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160236 s, 65.4 MB/s 2024-09-13T17:17:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137978 s, 76.0 MB/s 2024-09-13T17:17:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.556 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131781 s, 79.6 MB/s 2024-09-13T17:17:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.572 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130957 s, 80.1 MB/s 2024-09-13T17:17:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133435 s, 78.6 MB/s 2024-09-13T17:17:36.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016062 s, 65.3 MB/s 2024-09-13T17:17:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.621 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133678 s, 78.4 MB/s 2024-09-13T17:17:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.636 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126803 s, 82.7 MB/s 2024-09-13T17:17:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136564 s, 76.8 MB/s 2024-09-13T17:17:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131686 s, 79.6 MB/s 2024-09-13T17:17:36.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.682 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126967 s, 82.6 MB/s 2024-09-13T17:17:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165595 s, 63.3 MB/s 2024-09-13T17:17:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134665 s, 77.9 MB/s 2024-09-13T17:17:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.733 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140452 s, 74.7 MB/s 2024-09-13T17:17:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133501 s, 78.5 MB/s 2024-09-13T17:17:36.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131032 s, 80.0 MB/s 2024-09-13T17:17:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130229 s, 80.5 MB/s 2024-09-13T17:17:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.795 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140659 s, 74.5 MB/s 2024-09-13T17:17:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138634 s, 75.6 MB/s 2024-09-13T17:17:36.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.827 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.827 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130344 s, 80.4 MB/s 2024-09-13T17:17:36.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145428 s, 72.1 MB/s 2024-09-13T17:17:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018046 s, 58.1 MB/s 2024-09-13T17:17:36.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.880 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139818 s, 75.0 MB/s 2024-09-13T17:17:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162651 s, 64.5 MB/s 2024-09-13T17:17:36.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.13 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:36.901 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131333 s, 79.8 MB/s 2024-09-13T17:17:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137406 s, 76.3 MB/s 2024-09-13T17:17:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:36.992 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135593 s, 77.3 MB/s 2024-09-13T17:17:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012771 s, 82.1 MB/s 2024-09-13T17:17:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.023 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138235 s, 75.9 MB/s 2024-09-13T17:17:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137153 s, 76.5 MB/s 2024-09-13T17:17:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125544 s, 83.5 MB/s 2024-09-13T17:17:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.070 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138692 s, 75.6 MB/s 2024-09-13T17:17:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138767 s, 75.6 MB/s 2024-09-13T17:17:37.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.103 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137788 s, 76.1 MB/s 2024-09-13T17:17:37.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.119 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140892 s, 74.4 MB/s 2024-09-13T17:17:37.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014606 s, 71.8 MB/s 2024-09-13T17:17:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167654 s, 62.5 MB/s 2024-09-13T17:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129832 s, 80.8 MB/s 2024-09-13T17:17:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.188 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160912 s, 65.2 MB/s 2024-09-13T17:17:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.203 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128424 s, 81.6 MB/s 2024-09-13T17:17:37.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.222 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163347 s, 64.2 MB/s 2024-09-13T17:17:37.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141271 s, 74.2 MB/s 2024-09-13T17:17:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.256 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160248 s, 65.4 MB/s 2024-09-13T17:17:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127952 s, 82.0 MB/s 2024-09-13T17:17:37.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134145 s, 78.2 MB/s 2024-09-13T17:17:37.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130647 s, 80.3 MB/s 2024-09-13T17:17:37.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013167 s, 79.6 MB/s 2024-09-13T17:17:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131463 s, 79.8 MB/s 2024-09-13T17:17:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128 s, 81.9 MB/s 2024-09-13T17:17:37.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131384 s, 79.8 MB/s 2024-09-13T17:17:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124994 s, 83.9 MB/s 2024-09-13T17:17:37.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127143 s, 82.5 MB/s 2024-09-13T17:17:37.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013962 s, 75.1 MB/s 2024-09-13T17:17:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132035 s, 79.4 MB/s 2024-09-13T17:17:37.442 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150587 s, 69.6 MB/s 2024-09-13T17:17:37.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156798 s, 66.9 MB/s 2024-09-13T17:17:37.462 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.14 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:37.462 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:37.463 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147679 s, 71.0 MB/s 2024-09-13T17:17:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164406 s, 63.8 MB/s 2024-09-13T17:17:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.553 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140699 s, 74.5 MB/s 2024-09-13T17:17:37.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.571 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156503 s, 67.0 MB/s 2024-09-13T17:17:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127891 s, 82.0 MB/s 2024-09-13T17:17:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.605 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162631 s, 64.5 MB/s 2024-09-13T17:17:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133326 s, 78.6 MB/s 2024-09-13T17:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.636 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135295 s, 77.5 MB/s 2024-09-13T17:17:37.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.651 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122581 s, 85.5 MB/s 2024-09-13T17:17:37.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.666 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131953 s, 79.5 MB/s 2024-09-13T17:17:37.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119611 s, 87.7 MB/s 2024-09-13T17:17:37.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129477 s, 81.0 MB/s 2024-09-13T17:17:37.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125593 s, 83.5 MB/s 2024-09-13T17:17:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131522 s, 79.7 MB/s 2024-09-13T17:17:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.741 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128788 s, 81.4 MB/s 2024-09-13T17:17:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.760 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167701 s, 62.5 MB/s 2024-09-13T17:17:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.776 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138427 s, 75.7 MB/s 2024-09-13T17:17:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160574 s, 65.3 MB/s 2024-09-13T17:17:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017643 s, 59.4 MB/s 2024-09-13T17:17:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153442 s, 68.3 MB/s 2024-09-13T17:17:37.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135103 s, 77.6 MB/s 2024-09-13T17:17:37.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132477 s, 79.2 MB/s 2024-09-13T17:17:37.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129757 s, 80.8 MB/s 2024-09-13T17:17:37.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.898 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177248 s, 59.2 MB/s 2024-09-13T17:17:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012568 s, 83.4 MB/s 2024-09-13T17:17:37.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.929 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141249 s, 74.2 MB/s 2024-09-13T17:17:37.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169081 s, 62.0 MB/s 2024-09-13T17:17:37.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166809 s, 62.9 MB/s 2024-09-13T17:17:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:37.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:37.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139825 s, 75.0 MB/s 2024-09-13T17:17:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148082 s, 70.8 MB/s 2024-09-13T17:17:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.016 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128623 s, 81.5 MB/s 2024-09-13T17:17:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134035 s, 78.2 MB/s 2024-09-13T17:17:38.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:38.033 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:38.033 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:38.034 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.15 2024-09-13T17:17:38.034 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:38.034 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:38.034 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:38.034 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:38.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.088 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128954 s, 81.3 MB/s 2024-09-13T17:17:38.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140812 s, 74.5 MB/s 2024-09-13T17:17:38.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130294 s, 80.5 MB/s 2024-09-13T17:17:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133955 s, 78.3 MB/s 2024-09-13T17:17:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132364 s, 79.2 MB/s 2024-09-13T17:17:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139029 s, 75.4 MB/s 2024-09-13T17:17:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132149 s, 79.3 MB/s 2024-09-13T17:17:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164069 s, 63.9 MB/s 2024-09-13T17:17:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.216 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125211 s, 83.7 MB/s 2024-09-13T17:17:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.235 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171281 s, 61.2 MB/s 2024-09-13T17:17:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.250 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129919 s, 80.7 MB/s 2024-09-13T17:17:38.266 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.266 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.266 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013576 s, 77.2 MB/s 2024-09-13T17:17:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015966 s, 65.7 MB/s 2024-09-13T17:17:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.300 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135932 s, 77.1 MB/s 2024-09-13T17:17:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125691 s, 83.4 MB/s 2024-09-13T17:17:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.330 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136835 s, 76.6 MB/s 2024-09-13T17:17:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148015 s, 70.8 MB/s 2024-09-13T17:17:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128477 s, 81.6 MB/s 2024-09-13T17:17:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162891 s, 64.4 MB/s 2024-09-13T17:17:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141287 s, 74.2 MB/s 2024-09-13T17:17:38.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139689 s, 75.1 MB/s 2024-09-13T17:17:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162171 s, 64.7 MB/s 2024-09-13T17:17:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.447 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131981 s, 79.4 MB/s 2024-09-13T17:17:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129584 s, 80.9 MB/s 2024-09-13T17:17:38.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143442 s, 73.1 MB/s 2024-09-13T17:17:38.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.495 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132463 s, 79.2 MB/s 2024-09-13T17:17:38.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138961 s, 75.5 MB/s 2024-09-13T17:17:38.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137423 s, 76.3 MB/s 2024-09-13T17:17:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.542 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132994 s, 78.8 MB/s 2024-09-13T17:17:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127507 s, 82.2 MB/s 2024-09-13T17:17:38.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.573 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132297 s, 79.3 MB/s 2024-09-13T17:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.588 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134229 s, 78.1 MB/s 2024-09-13T17:17:38.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.16 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:38.590 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:38.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014106 s, 74.3 MB/s 2024-09-13T17:17:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134546 s, 77.9 MB/s 2024-09-13T17:17:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137651 s, 76.2 MB/s 2024-09-13T17:17:38.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131696 s, 79.6 MB/s 2024-09-13T17:17:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.709 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133485 s, 78.6 MB/s 2024-09-13T17:17:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013321 s, 78.7 MB/s 2024-09-13T17:17:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129789 s, 80.8 MB/s 2024-09-13T17:17:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126881 s, 82.6 MB/s 2024-09-13T17:17:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127451 s, 82.3 MB/s 2024-09-13T17:17:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.785 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136603 s, 76.8 MB/s 2024-09-13T17:17:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135184 s, 77.6 MB/s 2024-09-13T17:17:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137635 s, 76.2 MB/s 2024-09-13T17:17:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170662 s, 61.4 MB/s 2024-09-13T17:17:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.851 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126691 s, 82.8 MB/s 2024-09-13T17:17:38.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173371 s, 60.5 MB/s 2024-09-13T17:17:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144657 s, 72.5 MB/s 2024-09-13T17:17:38.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013001 s, 80.7 MB/s 2024-09-13T17:17:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127431 s, 82.3 MB/s 2024-09-13T17:17:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117558 s, 89.2 MB/s 2024-09-13T17:17:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.948 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139142 s, 75.4 MB/s 2024-09-13T17:17:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.963 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133066 s, 78.8 MB/s 2024-09-13T17:17:38.978 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.979 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131175 s, 79.9 MB/s 2024-09-13T17:17:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134161 s, 78.2 MB/s 2024-09-13T17:17:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138345 s, 75.8 MB/s 2024-09-13T17:17:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132839 s, 78.9 MB/s 2024-09-13T17:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.041 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130274 s, 80.5 MB/s 2024-09-13T17:17:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127212 s, 82.4 MB/s 2024-09-13T17:17:39.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.071 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124881 s, 84.0 MB/s 2024-09-13T17:17:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130073 s, 80.6 MB/s 2024-09-13T17:17:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133847 s, 78.3 MB/s 2024-09-13T17:17:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163049 s, 64.3 MB/s 2024-09-13T17:17:39.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137863 s, 76.1 MB/s 2024-09-13T17:17:39.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:39.138 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:39.138 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:39.138 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.17 2024-09-13T17:17:39.138 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:39.138 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:39.139 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:39.139 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129081 s, 81.2 MB/s 2024-09-13T17:17:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142031 s, 73.8 MB/s 2024-09-13T17:17:39.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129934 s, 80.7 MB/s 2024-09-13T17:17:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161525 s, 64.9 MB/s 2024-09-13T17:17:39.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135631 s, 77.3 MB/s 2024-09-13T17:17:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.274 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135518 s, 77.4 MB/s 2024-09-13T17:17:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.291 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136596 s, 76.8 MB/s 2024-09-13T17:17:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170601 s, 61.5 MB/s 2024-09-13T17:17:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161983 s, 64.7 MB/s 2024-09-13T17:17:39.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135271 s, 77.5 MB/s 2024-09-13T17:17:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134134 s, 78.2 MB/s 2024-09-13T17:17:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133525 s, 78.5 MB/s 2024-09-13T17:17:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168426 s, 62.3 MB/s 2024-09-13T17:17:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123336 s, 85.0 MB/s 2024-09-13T17:17:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143349 s, 73.1 MB/s 2024-09-13T17:17:39.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01683 s, 62.3 MB/s 2024-09-13T17:17:39.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143855 s, 72.9 MB/s 2024-09-13T17:17:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128874 s, 81.4 MB/s 2024-09-13T17:17:39.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137269 s, 76.4 MB/s 2024-09-13T17:17:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144998 s, 72.3 MB/s 2024-09-13T17:17:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136405 s, 76.9 MB/s 2024-09-13T17:17:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.540 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01359 s, 77.2 MB/s 2024-09-13T17:17:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.556 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129656 s, 80.9 MB/s 2024-09-13T17:17:39.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.572 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139107 s, 75.4 MB/s 2024-09-13T17:17:39.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127202 s, 82.4 MB/s 2024-09-13T17:17:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133697 s, 78.4 MB/s 2024-09-13T17:17:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013617 s, 77.0 MB/s 2024-09-13T17:17:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126384 s, 83.0 MB/s 2024-09-13T17:17:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.649 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132182 s, 79.3 MB/s 2024-09-13T17:17:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157747 s, 66.5 MB/s 2024-09-13T17:17:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134661 s, 77.9 MB/s 2024-09-13T17:17:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.698 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132395 s, 79.2 MB/s 2024-09-13T17:17:39.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:39.699 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:39.699 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:39.700 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.18 2024-09-13T17:17:39.700 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:39.700 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:39.700 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:39.700 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:39.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.755 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142911 s, 73.4 MB/s 2024-09-13T17:17:39.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127848 s, 82.0 MB/s 2024-09-13T17:17:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015854 s, 66.1 MB/s 2024-09-13T17:17:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128811 s, 81.4 MB/s 2024-09-13T17:17:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126454 s, 82.9 MB/s 2024-09-13T17:17:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.837 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160732 s, 65.2 MB/s 2024-09-13T17:17:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.853 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139867 s, 75.0 MB/s 2024-09-13T17:17:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016193 s, 64.8 MB/s 2024-09-13T17:17:39.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133772 s, 78.4 MB/s 2024-09-13T17:17:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135724 s, 77.3 MB/s 2024-09-13T17:17:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130855 s, 80.1 MB/s 2024-09-13T17:17:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145319 s, 72.2 MB/s 2024-09-13T17:17:39.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132294 s, 79.3 MB/s 2024-09-13T17:17:39.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.966 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144673 s, 72.5 MB/s 2024-09-13T17:17:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130944 s, 80.1 MB/s 2024-09-13T17:17:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:39.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140646 s, 74.6 MB/s 2024-09-13T17:17:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128499 s, 81.6 MB/s 2024-09-13T17:17:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136645 s, 76.7 MB/s 2024-09-13T17:17:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.044 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128636 s, 81.5 MB/s 2024-09-13T17:17:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125209 s, 83.7 MB/s 2024-09-13T17:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.075 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142532 s, 73.6 MB/s 2024-09-13T17:17:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.092 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137103 s, 76.5 MB/s 2024-09-13T17:17:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016706 s, 62.8 MB/s 2024-09-13T17:17:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139016 s, 75.4 MB/s 2024-09-13T17:17:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.142 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128384 s, 81.7 MB/s 2024-09-13T17:17:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016423 s, 63.8 MB/s 2024-09-13T17:17:40.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137754 s, 76.1 MB/s 2024-09-13T17:17:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133301 s, 78.7 MB/s 2024-09-13T17:17:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134197 s, 78.1 MB/s 2024-09-13T17:17:40.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.228 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173775 s, 60.3 MB/s 2024-09-13T17:17:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133142 s, 78.8 MB/s 2024-09-13T17:17:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130494 s, 80.4 MB/s 2024-09-13T17:17:40.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.19 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:40.260 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:40.261 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131719 s, 79.6 MB/s 2024-09-13T17:17:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137291 s, 76.4 MB/s 2024-09-13T17:17:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132663 s, 79.0 MB/s 2024-09-13T17:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141887 s, 73.9 MB/s 2024-09-13T17:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136879 s, 76.6 MB/s 2024-09-13T17:17:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167951 s, 62.4 MB/s 2024-09-13T17:17:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.413 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124543 s, 84.2 MB/s 2024-09-13T17:17:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.428 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128696 s, 81.5 MB/s 2024-09-13T17:17:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129551 s, 80.9 MB/s 2024-09-13T17:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016797 s, 62.4 MB/s 2024-09-13T17:17:40.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164788 s, 63.6 MB/s 2024-09-13T17:17:40.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014446 s, 72.6 MB/s 2024-09-13T17:17:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134979 s, 77.7 MB/s 2024-09-13T17:17:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129048 s, 81.3 MB/s 2024-09-13T17:17:40.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155395 s, 67.5 MB/s 2024-09-13T17:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130277 s, 80.5 MB/s 2024-09-13T17:17:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133672 s, 78.4 MB/s 2024-09-13T17:17:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134548 s, 77.9 MB/s 2024-09-13T17:17:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131324 s, 79.8 MB/s 2024-09-13T17:17:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.623 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131315 s, 79.9 MB/s 2024-09-13T17:17:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136659 s, 76.7 MB/s 2024-09-13T17:17:40.648 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:17:40.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:17:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136177 s, 77.0 MB/s 2024-09-13T17:17:40.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139112 s, 75.4 MB/s 2024-09-13T17:17:40.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155263 s, 67.5 MB/s 2024-09-13T17:17:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.705 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139518 s, 75.2 MB/s 2024-09-13T17:17:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.723 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160062 s, 65.5 MB/s 2024-09-13T17:17:40.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139115 s, 75.4 MB/s 2024-09-13T17:17:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167311 s, 62.7 MB/s 2024-09-13T17:17:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109912 s, 95.4 MB/s 2024-09-13T17:17:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.787 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138456 s, 75.7 MB/s 2024-09-13T17:17:40.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129265 s, 81.1 MB/s 2024-09-13T17:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166464 s, 63.0 MB/s 2024-09-13T17:17:40.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:40.822 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:40.822 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:40.822 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.20 2024-09-13T17:17:40.823 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:40.823 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:40.823 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:40.823 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132224 s, 79.3 MB/s 2024-09-13T17:17:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157333 s, 66.6 MB/s 2024-09-13T17:17:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129635 s, 80.9 MB/s 2024-09-13T17:17:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129205 s, 81.2 MB/s 2024-09-13T17:17:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.979 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120526 s, 87.0 MB/s 2024-09-13T17:17:40.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:40.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:40.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164239 s, 63.8 MB/s 2024-09-13T17:17:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168786 s, 62.1 MB/s 2024-09-13T17:17:41.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.033 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133733 s, 78.4 MB/s 2024-09-13T17:17:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162339 s, 64.6 MB/s 2024-09-13T17:17:41.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132852 s, 78.9 MB/s 2024-09-13T17:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.083 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138039 s, 76.0 MB/s 2024-09-13T17:17:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016979 s, 61.8 MB/s 2024-09-13T17:17:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162355 s, 64.6 MB/s 2024-09-13T17:17:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131412 s, 79.8 MB/s 2024-09-13T17:17:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.155 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169601 s, 61.8 MB/s 2024-09-13T17:17:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129966 s, 80.7 MB/s 2024-09-13T17:17:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169461 s, 61.9 MB/s 2024-09-13T17:17:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134154 s, 78.2 MB/s 2024-09-13T17:17:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132422 s, 79.2 MB/s 2024-09-13T17:17:41.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136955 s, 76.6 MB/s 2024-09-13T17:17:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132981 s, 78.9 MB/s 2024-09-13T17:17:41.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140778 s, 74.5 MB/s 2024-09-13T17:17:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132427 s, 79.2 MB/s 2024-09-13T17:17:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.304 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168676 s, 62.2 MB/s 2024-09-13T17:17:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.322 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.323 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167888 s, 62.5 MB/s 2024-09-13T17:17:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.338 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013015 s, 80.6 MB/s 2024-09-13T17:17:41.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.355 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149249 s, 70.3 MB/s 2024-09-13T17:17:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.372 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150928 s, 69.5 MB/s 2024-09-13T17:17:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.387 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130055 s, 80.6 MB/s 2024-09-13T17:17:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174177 s, 60.2 MB/s 2024-09-13T17:17:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132014 s, 79.4 MB/s 2024-09-13T17:17:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132182 s, 79.3 MB/s 2024-09-13T17:17:41.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.21 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:41.440 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.496 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139034 s, 75.4 MB/s 2024-09-13T17:17:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156982 s, 66.8 MB/s 2024-09-13T17:17:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132294 s, 79.3 MB/s 2024-09-13T17:17:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130348 s, 80.4 MB/s 2024-09-13T17:17:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143831 s, 72.9 MB/s 2024-09-13T17:17:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138419 s, 75.8 MB/s 2024-09-13T17:17:41.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140304 s, 74.7 MB/s 2024-09-13T17:17:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133239 s, 78.7 MB/s 2024-09-13T17:17:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132332 s, 79.2 MB/s 2024-09-13T17:17:41.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117225 s, 89.5 MB/s 2024-09-13T17:17:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160941 s, 65.2 MB/s 2024-09-13T17:17:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129304 s, 81.1 MB/s 2024-09-13T17:17:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139802 s, 75.0 MB/s 2024-09-13T17:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122071 s, 85.9 MB/s 2024-09-13T17:17:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131933 s, 79.5 MB/s 2024-09-13T17:17:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119358 s, 87.9 MB/s 2024-09-13T17:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164833 s, 63.6 MB/s 2024-09-13T17:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125429 s, 83.6 MB/s 2024-09-13T17:17:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137643 s, 76.2 MB/s 2024-09-13T17:17:41.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139782 s, 75.0 MB/s 2024-09-13T17:17:41.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140023 s, 74.9 MB/s 2024-09-13T17:17:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159785 s, 65.6 MB/s 2024-09-13T17:17:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164945 s, 63.6 MB/s 2024-09-13T17:17:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235264 s, 44.6 MB/s 2024-09-13T17:17:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170336 s, 61.6 MB/s 2024-09-13T17:17:41.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158368 s, 66.2 MB/s 2024-09-13T17:17:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136659 s, 76.7 MB/s 2024-09-13T17:17:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.952 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164835 s, 63.6 MB/s 2024-09-13T17:17:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124949 s, 83.9 MB/s 2024-09-13T17:17:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129291 s, 81.1 MB/s 2024-09-13T17:17:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133332 s, 78.6 MB/s 2024-09-13T17:17:42.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128876 s, 81.4 MB/s 2024-09-13T17:17:42.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:42.014 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.22 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:42.015 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168216 s, 62.3 MB/s 2024-09-13T17:17:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136765 s, 76.7 MB/s 2024-09-13T17:17:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.108 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137257 s, 76.4 MB/s 2024-09-13T17:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166927 s, 62.8 MB/s 2024-09-13T17:17:42.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172943 s, 60.6 MB/s 2024-09-13T17:17:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128058 s, 81.9 MB/s 2024-09-13T17:17:42.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157612 s, 66.5 MB/s 2024-09-13T17:17:42.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.192 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128846 s, 81.4 MB/s 2024-09-13T17:17:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.207 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128386 s, 81.7 MB/s 2024-09-13T17:17:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.222 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131937 s, 79.5 MB/s 2024-09-13T17:17:42.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.241 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165792 s, 63.2 MB/s 2024-09-13T17:17:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.257 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134016 s, 78.2 MB/s 2024-09-13T17:17:42.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.276 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170374 s, 61.5 MB/s 2024-09-13T17:17:42.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166672 s, 62.9 MB/s 2024-09-13T17:17:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164471 s, 63.8 MB/s 2024-09-13T17:17:42.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130094 s, 80.6 MB/s 2024-09-13T17:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124824 s, 84.0 MB/s 2024-09-13T17:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156997 s, 66.8 MB/s 2024-09-13T17:17:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.377 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130808 s, 80.2 MB/s 2024-09-13T17:17:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165886 s, 63.2 MB/s 2024-09-13T17:17:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161607 s, 64.9 MB/s 2024-09-13T17:17:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129164 s, 81.2 MB/s 2024-09-13T17:17:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129988 s, 80.7 MB/s 2024-09-13T17:17:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.463 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168595 s, 62.2 MB/s 2024-09-13T17:17:42.482 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.482 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.482 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162238 s, 64.6 MB/s 2024-09-13T17:17:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012888 s, 81.4 MB/s 2024-09-13T17:17:42.512 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134342 s, 78.1 MB/s 2024-09-13T17:17:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.528 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135345 s, 77.5 MB/s 2024-09-13T17:17:42.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158517 s, 66.1 MB/s 2024-09-13T17:17:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.565 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167285 s, 62.7 MB/s 2024-09-13T17:17:42.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.580 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125473 s, 83.6 MB/s 2024-09-13T17:17:42.595 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013535 s, 77.5 MB/s 2024-09-13T17:17:42.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:42.597 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:42.597 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:42.597 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.23 2024-09-13T17:17:42.598 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:42.598 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:42.598 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:42.598 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159613 s, 65.7 MB/s 2024-09-13T17:17:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130457 s, 80.4 MB/s 2024-09-13T17:17:42.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139946 s, 74.9 MB/s 2024-09-13T17:17:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.703 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130979 s, 80.1 MB/s 2024-09-13T17:17:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.722 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163381 s, 64.2 MB/s 2024-09-13T17:17:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.736 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127999 s, 81.9 MB/s 2024-09-13T17:17:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.752 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130271 s, 80.5 MB/s 2024-09-13T17:17:42.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.767 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124719 s, 84.1 MB/s 2024-09-13T17:17:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013663 s, 76.7 MB/s 2024-09-13T17:17:42.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.799 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137154 s, 76.5 MB/s 2024-09-13T17:17:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134249 s, 78.1 MB/s 2024-09-13T17:17:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135118 s, 77.6 MB/s 2024-09-13T17:17:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160601 s, 65.3 MB/s 2024-09-13T17:17:42.868 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.868 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.868 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169616 s, 61.8 MB/s 2024-09-13T17:17:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.884 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141017 s, 74.4 MB/s 2024-09-13T17:17:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146642 s, 71.5 MB/s 2024-09-13T17:17:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.917 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140374 s, 74.7 MB/s 2024-09-13T17:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.933 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013271 s, 79.0 MB/s 2024-09-13T17:17:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014455 s, 72.5 MB/s 2024-09-13T17:17:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013244 s, 79.2 MB/s 2024-09-13T17:17:42.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143533 s, 73.1 MB/s 2024-09-13T17:17:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:42.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136913 s, 76.6 MB/s 2024-09-13T17:17:43.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.014 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138583 s, 75.7 MB/s 2024-09-13T17:17:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013451 s, 78.0 MB/s 2024-09-13T17:17:43.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139238 s, 75.3 MB/s 2024-09-13T17:17:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.062 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139267 s, 75.3 MB/s 2024-09-13T17:17:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.077 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136075 s, 77.1 MB/s 2024-09-13T17:17:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138085 s, 75.9 MB/s 2024-09-13T17:17:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133668 s, 78.4 MB/s 2024-09-13T17:17:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128239 s, 81.8 MB/s 2024-09-13T17:17:43.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130799 s, 80.2 MB/s 2024-09-13T17:17:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137741 s, 76.1 MB/s 2024-09-13T17:17:43.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:43.157 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:43.157 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:43.158 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.24 2024-09-13T17:17:43.158 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:43.158 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:43.158 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:43.158 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.214 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148132 s, 70.8 MB/s 2024-09-13T17:17:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134828 s, 77.8 MB/s 2024-09-13T17:17:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175052 s, 59.9 MB/s 2024-09-13T17:17:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132011 s, 79.4 MB/s 2024-09-13T17:17:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143061 s, 73.3 MB/s 2024-09-13T17:17:43.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.298 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141307 s, 74.2 MB/s 2024-09-13T17:17:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128876 s, 81.4 MB/s 2024-09-13T17:17:43.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130267 s, 80.5 MB/s 2024-09-13T17:17:43.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120118 s, 87.3 MB/s 2024-09-13T17:17:43.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132939 s, 78.9 MB/s 2024-09-13T17:17:43.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130757 s, 80.2 MB/s 2024-09-13T17:17:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.389 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134012 s, 78.2 MB/s 2024-09-13T17:17:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129001 s, 81.3 MB/s 2024-09-13T17:17:43.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165757 s, 63.3 MB/s 2024-09-13T17:17:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160901 s, 65.2 MB/s 2024-09-13T17:17:43.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.458 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152576 s, 68.7 MB/s 2024-09-13T17:17:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168128 s, 62.4 MB/s 2024-09-13T17:17:43.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.494 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148015 s, 70.8 MB/s 2024-09-13T17:17:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.514 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175453 s, 59.8 MB/s 2024-09-13T17:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.529 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012756 s, 82.2 MB/s 2024-09-13T17:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.544 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130636 s, 80.3 MB/s 2024-09-13T17:17:43.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.559 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012722 s, 82.4 MB/s 2024-09-13T17:17:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132982 s, 78.9 MB/s 2024-09-13T17:17:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172707 s, 60.7 MB/s 2024-09-13T17:17:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134072 s, 78.2 MB/s 2024-09-13T17:17:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.629 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167168 s, 62.7 MB/s 2024-09-13T17:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132433 s, 79.2 MB/s 2024-09-13T17:17:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013396 s, 78.3 MB/s 2024-09-13T17:17:43.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143532 s, 73.1 MB/s 2024-09-13T17:17:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138131 s, 75.9 MB/s 2024-09-13T17:17:43.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135086 s, 77.6 MB/s 2024-09-13T17:17:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016559 s, 63.3 MB/s 2024-09-13T17:17:43.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.25 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:43.729 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:43.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.785 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01359 s, 77.2 MB/s 2024-09-13T17:17:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.800 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129306 s, 81.1 MB/s 2024-09-13T17:17:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129021 s, 81.3 MB/s 2024-09-13T17:17:43.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138182 s, 75.9 MB/s 2024-09-13T17:17:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015658 s, 67.0 MB/s 2024-09-13T17:17:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128869 s, 81.4 MB/s 2024-09-13T17:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132627 s, 79.1 MB/s 2024-09-13T17:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129798 s, 80.8 MB/s 2024-09-13T17:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125284 s, 83.7 MB/s 2024-09-13T17:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132963 s, 78.9 MB/s 2024-09-13T17:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131234 s, 79.9 MB/s 2024-09-13T17:17:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161451 s, 64.9 MB/s 2024-09-13T17:17:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133574 s, 78.5 MB/s 2024-09-13T17:17:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:43.994 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017279 s, 60.7 MB/s 2024-09-13T17:17:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.009 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134907 s, 77.7 MB/s 2024-09-13T17:17:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148402 s, 70.7 MB/s 2024-09-13T17:17:44.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.041 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126829 s, 82.7 MB/s 2024-09-13T17:17:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.057 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132362 s, 79.2 MB/s 2024-09-13T17:17:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128522 s, 81.6 MB/s 2024-09-13T17:17:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.088 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137638 s, 76.2 MB/s 2024-09-13T17:17:44.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.105 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151753 s, 69.1 MB/s 2024-09-13T17:17:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131769 s, 79.6 MB/s 2024-09-13T17:17:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140643 s, 74.6 MB/s 2024-09-13T17:17:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123713 s, 84.8 MB/s 2024-09-13T17:17:44.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132136 s, 79.4 MB/s 2024-09-13T17:17:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.183 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131828 s, 79.5 MB/s 2024-09-13T17:17:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.199 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142949 s, 73.4 MB/s 2024-09-13T17:17:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163474 s, 64.1 MB/s 2024-09-13T17:17:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134499 s, 78.0 MB/s 2024-09-13T17:17:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.250 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140835 s, 74.5 MB/s 2024-09-13T17:17:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126718 s, 82.7 MB/s 2024-09-13T17:17:44.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016286 s, 64.4 MB/s 2024-09-13T17:17:44.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.26 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:44.285 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134767 s, 77.8 MB/s 2024-09-13T17:17:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.356 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013322 s, 78.7 MB/s 2024-09-13T17:17:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129964 s, 80.7 MB/s 2024-09-13T17:17:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131032 s, 80.0 MB/s 2024-09-13T17:17:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.402 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138519 s, 75.7 MB/s 2024-09-13T17:17:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160554 s, 65.3 MB/s 2024-09-13T17:17:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.437 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134332 s, 78.1 MB/s 2024-09-13T17:17:44.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131604 s, 79.7 MB/s 2024-09-13T17:17:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.467 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013234 s, 79.2 MB/s 2024-09-13T17:17:44.482 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.483 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130323 s, 80.5 MB/s 2024-09-13T17:17:44.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.498 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129715 s, 80.8 MB/s 2024-09-13T17:17:44.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.514 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140596 s, 74.6 MB/s 2024-09-13T17:17:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158603 s, 66.1 MB/s 2024-09-13T17:17:44.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.548 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133252 s, 78.7 MB/s 2024-09-13T17:17:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134099 s, 78.2 MB/s 2024-09-13T17:17:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.582 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165306 s, 63.4 MB/s 2024-09-13T17:17:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.597 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131003 s, 80.0 MB/s 2024-09-13T17:17:44.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.613 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133179 s, 78.7 MB/s 2024-09-13T17:17:44.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126451 s, 82.9 MB/s 2024-09-13T17:17:44.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.643 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132079 s, 79.4 MB/s 2024-09-13T17:17:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.659 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136391 s, 76.9 MB/s 2024-09-13T17:17:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.675 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135597 s, 77.3 MB/s 2024-09-13T17:17:44.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.689 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125443 s, 83.6 MB/s 2024-09-13T17:17:44.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.705 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133437 s, 78.6 MB/s 2024-09-13T17:17:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.720 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124304 s, 84.4 MB/s 2024-09-13T17:17:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126233 s, 83.1 MB/s 2024-09-13T17:17:44.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.755 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174731 s, 60.0 MB/s 2024-09-13T17:17:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134605 s, 77.9 MB/s 2024-09-13T17:17:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.785 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127807 s, 82.0 MB/s 2024-09-13T17:17:44.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128658 s, 81.5 MB/s 2024-09-13T17:17:44.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136645 s, 76.7 MB/s 2024-09-13T17:17:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012979 s, 80.8 MB/s 2024-09-13T17:17:44.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.27 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:44.833 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:44.834 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129889 s, 80.7 MB/s 2024-09-13T17:17:44.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130186 s, 80.5 MB/s 2024-09-13T17:17:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012241 s, 85.7 MB/s 2024-09-13T17:17:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130895 s, 80.1 MB/s 2024-09-13T17:17:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012575 s, 83.4 MB/s 2024-09-13T17:17:44.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164157 s, 63.9 MB/s 2024-09-13T17:17:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:44.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155365 s, 67.5 MB/s 2024-09-13T17:17:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.002 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145796 s, 71.9 MB/s 2024-09-13T17:17:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130246 s, 80.5 MB/s 2024-09-13T17:17:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140879 s, 74.4 MB/s 2024-09-13T17:17:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171096 s, 61.3 MB/s 2024-09-13T17:17:45.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167935 s, 62.4 MB/s 2024-09-13T17:17:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.087 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132708 s, 79.0 MB/s 2024-09-13T17:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.103 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135581 s, 77.3 MB/s 2024-09-13T17:17:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.122 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165924 s, 63.2 MB/s 2024-09-13T17:17:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141964 s, 73.9 MB/s 2024-09-13T17:17:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132497 s, 79.1 MB/s 2024-09-13T17:17:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129654 s, 80.9 MB/s 2024-09-13T17:17:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.188 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016969 s, 61.8 MB/s 2024-09-13T17:17:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132148 s, 79.3 MB/s 2024-09-13T17:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134736 s, 77.8 MB/s 2024-09-13T17:17:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.236 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141338 s, 74.2 MB/s 2024-09-13T17:17:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156782 s, 66.9 MB/s 2024-09-13T17:17:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138909 s, 75.5 MB/s 2024-09-13T17:17:45.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.286 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135129 s, 77.6 MB/s 2024-09-13T17:17:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133968 s, 78.3 MB/s 2024-09-13T17:17:45.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.316 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120702 s, 86.9 MB/s 2024-09-13T17:17:45.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129363 s, 81.1 MB/s 2024-09-13T17:17:45.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165979 s, 63.2 MB/s 2024-09-13T17:17:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132282 s, 79.3 MB/s 2024-09-13T17:17:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135209 s, 77.6 MB/s 2024-09-13T17:17:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135249 s, 77.5 MB/s 2024-09-13T17:17:45.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:45.398 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:45.398 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:45.398 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.28 2024-09-13T17:17:45.399 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:45.399 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:45.399 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:45.399 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168058 s, 62.4 MB/s 2024-09-13T17:17:45.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124853 s, 84.0 MB/s 2024-09-13T17:17:45.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138668 s, 75.6 MB/s 2024-09-13T17:17:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141677 s, 74.0 MB/s 2024-09-13T17:17:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01288 s, 81.4 MB/s 2024-09-13T17:17:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.538 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158427 s, 66.2 MB/s 2024-09-13T17:17:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172879 s, 60.7 MB/s 2024-09-13T17:17:45.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016366 s, 64.1 MB/s 2024-09-13T17:17:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012641 s, 83.0 MB/s 2024-09-13T17:17:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133917 s, 78.3 MB/s 2024-09-13T17:17:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159978 s, 65.5 MB/s 2024-09-13T17:17:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013128 s, 79.9 MB/s 2024-09-13T17:17:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138711 s, 75.6 MB/s 2024-09-13T17:17:45.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142601 s, 73.5 MB/s 2024-09-13T17:17:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128226 s, 81.8 MB/s 2024-09-13T17:17:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126878 s, 82.6 MB/s 2024-09-13T17:17:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132133 s, 79.4 MB/s 2024-09-13T17:17:45.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.733 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132902 s, 78.9 MB/s 2024-09-13T17:17:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136071 s, 77.1 MB/s 2024-09-13T17:17:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127296 s, 82.4 MB/s 2024-09-13T17:17:45.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.780 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135328 s, 77.5 MB/s 2024-09-13T17:17:45.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016747 s, 62.6 MB/s 2024-09-13T17:17:45.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132163 s, 79.3 MB/s 2024-09-13T17:17:45.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133051 s, 78.8 MB/s 2024-09-13T17:17:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120821 s, 86.8 MB/s 2024-09-13T17:17:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.859 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130058 s, 80.6 MB/s 2024-09-13T17:17:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139909 s, 74.9 MB/s 2024-09-13T17:17:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.893 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159341 s, 65.8 MB/s 2024-09-13T17:17:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:45.908 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127708 s, 82.1 MB/s 2024-09-13T17:17:45.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.016 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141093 s, 74.3 MB/s 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130306 s, 80.5 MB/s 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166808 s, 62.9 MB/s 2024-09-13T17:17:46.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.29 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:46.018 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136852 s, 76.6 MB/s 2024-09-13T17:17:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014083 s, 74.5 MB/s 2024-09-13T17:17:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160365 s, 65.4 MB/s 2024-09-13T17:17:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.124 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141932 s, 73.9 MB/s 2024-09-13T17:17:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133766 s, 78.4 MB/s 2024-09-13T17:17:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.158 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160908 s, 65.2 MB/s 2024-09-13T17:17:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138503 s, 75.7 MB/s 2024-09-13T17:17:46.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167791 s, 62.5 MB/s 2024-09-13T17:17:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012919 s, 81.2 MB/s 2024-09-13T17:17:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133741 s, 78.4 MB/s 2024-09-13T17:17:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123436 s, 84.9 MB/s 2024-09-13T17:17:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128874 s, 81.4 MB/s 2024-09-13T17:17:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.272 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158603 s, 66.1 MB/s 2024-09-13T17:17:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013317 s, 78.7 MB/s 2024-09-13T17:17:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132985 s, 78.8 MB/s 2024-09-13T17:17:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156006 s, 67.2 MB/s 2024-09-13T17:17:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.336 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134761 s, 77.8 MB/s 2024-09-13T17:17:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.352 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132383 s, 79.2 MB/s 2024-09-13T17:17:46.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.370 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158212 s, 66.3 MB/s 2024-09-13T17:17:46.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.385 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127507 s, 82.2 MB/s 2024-09-13T17:17:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.400 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.401 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133439 s, 78.6 MB/s 2024-09-13T17:17:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127649 s, 82.1 MB/s 2024-09-13T17:17:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128787 s, 81.4 MB/s 2024-09-13T17:17:46.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.447 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013421 s, 78.1 MB/s 2024-09-13T17:17:46.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133498 s, 78.5 MB/s 2024-09-13T17:17:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127979 s, 81.9 MB/s 2024-09-13T17:17:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.495 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158484 s, 66.2 MB/s 2024-09-13T17:17:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.510 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130084 s, 80.6 MB/s 2024-09-13T17:17:46.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140938 s, 74.4 MB/s 2024-09-13T17:17:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133395 s, 78.6 MB/s 2024-09-13T17:17:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012305 s, 85.2 MB/s 2024-09-13T17:17:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.572 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132578 s, 79.1 MB/s 2024-09-13T17:17:46.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.30 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:46.574 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160953 s, 65.1 MB/s 2024-09-13T17:17:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135925 s, 77.1 MB/s 2024-09-13T17:17:46.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.666 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.666 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164182 s, 63.9 MB/s 2024-09-13T17:17:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129932 s, 80.7 MB/s 2024-09-13T17:17:46.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.699 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01565 s, 67.0 MB/s 2024-09-13T17:17:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.714 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130289 s, 80.5 MB/s 2024-09-13T17:17:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.730 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135618 s, 77.3 MB/s 2024-09-13T17:17:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119156 s, 88.0 MB/s 2024-09-13T17:17:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.760 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135502 s, 77.4 MB/s 2024-09-13T17:17:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138078 s, 75.9 MB/s 2024-09-13T17:17:46.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.795 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164773 s, 63.6 MB/s 2024-09-13T17:17:46.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.814 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162688 s, 64.5 MB/s 2024-09-13T17:17:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165398 s, 63.4 MB/s 2024-09-13T17:17:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133249 s, 78.7 MB/s 2024-09-13T17:17:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166632 s, 62.9 MB/s 2024-09-13T17:17:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.888 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178576 s, 58.7 MB/s 2024-09-13T17:17:46.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157075 s, 66.8 MB/s 2024-09-13T17:17:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01686 s, 62.2 MB/s 2024-09-13T17:17:46.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127523 s, 82.2 MB/s 2024-09-13T17:17:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128901 s, 81.3 MB/s 2024-09-13T17:17:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120439 s, 87.1 MB/s 2024-09-13T17:17:46.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:46.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:46.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127942 s, 82.0 MB/s 2024-09-13T17:17:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.004 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016351 s, 64.1 MB/s 2024-09-13T17:17:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.022 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016271 s, 64.4 MB/s 2024-09-13T17:17:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.038 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139161 s, 75.3 MB/s 2024-09-13T17:17:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161836 s, 64.8 MB/s 2024-09-13T17:17:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013174 s, 79.6 MB/s 2024-09-13T17:17:47.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.088 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134487 s, 78.0 MB/s 2024-09-13T17:17:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.104 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139255 s, 75.3 MB/s 2024-09-13T17:17:47.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132683 s, 79.0 MB/s 2024-09-13T17:17:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130781 s, 80.2 MB/s 2024-09-13T17:17:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164678 s, 63.7 MB/s 2024-09-13T17:17:47.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.31 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:17:47.155 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:17:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173028 s, 60.6 MB/s 2024-09-13T17:17:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131031 s, 80.0 MB/s 2024-09-13T17:17:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131826 s, 79.5 MB/s 2024-09-13T17:17:47.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.264 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135378 s, 77.5 MB/s 2024-09-13T17:17:47.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014175 s, 74.0 MB/s 2024-09-13T17:17:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.299 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156993 s, 66.8 MB/s 2024-09-13T17:17:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014249 s, 73.6 MB/s 2024-09-13T17:17:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.330 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131008 s, 80.0 MB/s 2024-09-13T17:17:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.346 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138808 s, 75.5 MB/s 2024-09-13T17:17:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162728 s, 64.4 MB/s 2024-09-13T17:17:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133616 s, 78.5 MB/s 2024-09-13T17:17:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013927 s, 75.3 MB/s 2024-09-13T17:17:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133288 s, 78.7 MB/s 2024-09-13T17:17:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016282 s, 64.4 MB/s 2024-09-13T17:17:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126122 s, 83.1 MB/s 2024-09-13T17:17:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01354 s, 77.4 MB/s 2024-09-13T17:17:47.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132712 s, 79.0 MB/s 2024-09-13T17:17:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133144 s, 78.8 MB/s 2024-09-13T17:17:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170604 s, 61.5 MB/s 2024-09-13T17:17:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.530 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166034 s, 63.2 MB/s 2024-09-13T17:17:47.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130961 s, 80.1 MB/s 2024-09-13T17:17:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012923 s, 81.1 MB/s 2024-09-13T17:17:47.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137234 s, 76.4 MB/s 2024-09-13T17:17:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134893 s, 77.7 MB/s 2024-09-13T17:17:47.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134104 s, 78.2 MB/s 2024-09-13T17:17:47.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162311 s, 64.6 MB/s 2024-09-13T17:17:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126951 s, 82.6 MB/s 2024-09-13T17:17:47.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132004 s, 79.4 MB/s 2024-09-13T17:17:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126557 s, 82.9 MB/s 2024-09-13T17:17:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134426 s, 78.0 MB/s 2024-09-13T17:17:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125272 s, 83.7 MB/s 2024-09-13T17:17:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:17:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:17:47.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013632 s, 76.9 MB/s 2024-09-13T17:17:47.719 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.9:0 scrub start /scrub_control_test_path recursive 2024-09-13T17:17:47.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.876+0000 7fbb72800640 1 Processor -- start 2024-09-13T17:17:47.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.877+0000 7fbb72800640 1 -- start start 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb72800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb6c143d90 0x7fbb6c06b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb72800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb6c06b8a0 0x7fbb6c0cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb72800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c06ca10 con 0x7fbb6c0cd840 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb72800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c144170 con 0x7fbb6c143d90 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb72800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c06be00 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb71400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb6c143d90 0x7fbb6c06b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb71e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbb6c0cd840 0x7fbb6c15df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35128/0 (socket says 172.21.15.28:35128) 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb71400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbb6c143d90 0x7fbb6c06b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45492/0 (socket says 172.21.15.28:45492) 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb71e00640 1 -- 172.21.15.28:0/2501661516 learned_addr learned my addr 172.21.15.28:0/2501661516 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.878+0000 7fbb70a00640 1 --2- 172.21.15.28:0/2501661516 >> v2:172.21.15.189:3300/0 conn(0x7fbb6c06b8a0 0x7fbb6c0cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.879+0000 7fbb71400640 1 -- 172.21.15.28:0/2501661516 >> v2:172.21.15.189:3300/0 conn(0x7fbb6c06b8a0 msgr2=0x7fbb6c0cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.879+0000 7fbb71400640 1 --2- 172.21.15.28:0/2501661516 >> v2:172.21.15.189:3300/0 conn(0x7fbb6c06b8a0 0x7fbb6c0cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.879+0000 7fbb71400640 1 -- 172.21.15.28:0/2501661516 >> v1:172.21.15.28:6789/0 conn(0x7fbb6c0cd840 legacy=0x7fbb6c15df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.879+0000 7fbb71400640 1 -- 172.21.15.28:0/2501661516 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c06c0e0 con 0x7fbb6c143d90 2024-09-13T17:17:47.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.879+0000 7fbb6be00640 1 -- 172.21.15.28:0/2501661516 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051914271 0 0) 0x7fbb6c06ca10 con 0x7fbb6c0cd840 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.880+0000 7fbb71400640 1 --2- 172.21.15.28:0/2501661516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c143d90 0x7fbb6c06b360 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0ae020 tx=0x7fbb6002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62491e6d53b63f91 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.880+0000 7fbb6be00640 1 -- 172.21.15.28:0/2501661516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6002aea0 con 0x7fbb6c143d90 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.880+0000 7fbb6be00640 1 -- 172.21.15.28:0/2501661516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6002d050 con 0x7fbb6c143d90 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.880+0000 7fbb6be00640 1 -- 172.21.15.28:0/2501661516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60007960 con 0x7fbb6c143d90 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 -- 172.21.15.28:0/2501661516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c143d90 msgr2=0x7fbb6c06b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 --2- 172.21.15.28:0/2501661516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c143d90 0x7fbb6c06b360 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0ae020 tx=0x7fbb6002a680 comp rx=0 tx=0).stop 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 -- 172.21.15.28:0/2501661516 shutdown_connections 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 --2- 172.21.15.28:0/2501661516 >> v2:172.21.15.189:3300/0 conn(0x7fbb6c06b8a0 0x7fbb6c0cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 --2- 172.21.15.28:0/2501661516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c143d90 0x7fbb6c06b360 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 -- 172.21.15.28:0/2501661516 >> 172.21.15.28:0/2501661516 conn(0x7fbb6c073470 msgr2=0x7fbb6c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 -- 172.21.15.28:0/2501661516 shutdown_connections 2024-09-13T17:17:47.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.881+0000 7fbb72800640 1 -- 172.21.15.28:0/2501661516 wait complete. 2024-09-13T17:17:47.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.882+0000 7fbb72800640 1 Processor -- start 2024-09-13T17:17:47.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.882+0000 7fbb72800640 1 -- start start 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6c0cd840 0x7fbb6c15c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 0x7fbb6c15dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15d4a0 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15a5a0 con 0x7fbb6c0cd840 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.883+0000 7fbb72800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6c15d1c0 con 0x7fbb6c143d90 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45500/0 (socket says 172.21.15.28:45500) 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 -- 172.21.15.28:0/2714509639 learned_addr learned my addr 172.21.15.28:0/2714509639 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb70a00640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6c0cd840 0x7fbb6c15c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71e00640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 0x7fbb6c15dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 -- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 msgr2=0x7fbb6c15dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 0x7fbb6c15dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 -- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6c0cd840 msgr2=0x7fbb6c15c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6c0cd840 0x7fbb6c15c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 -- 172.21.15.28:0/2714509639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb60009a70 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71e00640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 0x7fbb6c15dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.884+0000 7fbb71400640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbb6002ac70 tx=0x7fbb60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.885+0000 7fbb69600640 1 -- 172.21.15.28:0/2714509639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6003c830 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.885+0000 7fbb69600640 1 -- 172.21.15.28:0/2714509639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6003c990 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.885+0000 7fbb69600640 1 -- 172.21.15.28:0/2714509639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6003cd20 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.885+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c1f5980 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.885+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c1f5f40 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.887+0000 7fbb69600640 1 -- 172.21.15.28:0/2714509639 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbb60031400 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.888+0000 7fbb69600640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 0x7fbb4c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.888+0000 7fbb69600640 1 -- 172.21.15.28:0/2714509639 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb600aa680 con 0x7fbb6c06b8a0 2024-09-13T17:17:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.888+0000 7fbb70a00640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 0x7fbb4c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.889+0000 7fbb76ca2740 1 Processor -- start 2024-09-13T17:17:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.889+0000 7fbb76ca2740 1 -- start start 2024-09-13T17:17:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.889+0000 7fbb70a00640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 0x7fbb4c06d0c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009b20 tx=0x7fbb58009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x556540475bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x5565404765b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556540391f00 con 0x5565404283e0 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654038f690 con 0x55654044fc40 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55654038af80 con 0x556540476100 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb71400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45514/0 (socket says 172.21.15.28:45514) 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb71400640 1 -- 172.21.15.28:0/1179079480 learned_addr learned my addr 172.21.15.28:0/1179079480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb70a00640 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x556540475bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.890+0000 7fbb71e00640 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x5565404765b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 -- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 msgr2=0x5565404765b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x5565404765b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 -- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 msgr2=0x556540475bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x556540475bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 -- 172.21.15.28:0/1179079480 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556540355390 con 0x5565404283e0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb71400640 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbb60081340 tx=0x7fbb60009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153cb3bc33874b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb5e000640 1 -- 172.21.15.28:0/1179079480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6007d2b0 con 0x5565404283e0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb5e000640 1 -- 172.21.15.28:0/1179079480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60081870 con 0x5565404283e0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.891+0000 7fbb5e000640 1 -- 172.21.15.28:0/1179079480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600775c0 con 0x5565404283e0 2024-09-13T17:17:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- 172.21.15.28:0/1179079480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 msgr2=0x5565404287c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7fbb60081340 tx=0x7fbb60009450 comp rx=0 tx=0).stop 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- 172.21.15.28:0/1179079480 shutdown_connections 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x5565404765b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x556540475bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/1179079480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5565404283e0 0x5565404287c0 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- 172.21.15.28:0/1179079480 >> 172.21.15.28:0/1179079480 conn(0x556540424c30 msgr2=0x55654046d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- 172.21.15.28:0/1179079480 shutdown_connections 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- 172.21.15.28:0/1179079480 wait complete. 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.892+0000 7fbb76ca2740 1 -- start start 2024-09-13T17:17:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.893+0000 7fbb76ca2740 1 Processor -- start 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044fc40 0x556540472f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x55654044cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565403492c0 con 0x556540476100 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556540348f60 con 0x55654044c8c0 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5565400f23e0 con 0x55654044fc40 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb76ca2740 10 client.0 _ll_register_callbacks cb 0x55654035fdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45524/0 (socket says 172.21.15.28:45524) 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 -- 172.21.15.28:0/3404064046 learned_addr learned my addr 172.21.15.28:0/3404064046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb71e00640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x55654044cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb71400640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044fc40 0x556540472f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 -- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044fc40 msgr2=0x556540472f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044fc40 0x556540472f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 -- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 msgr2=0x55654044cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x55654044cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.894+0000 7fbb70a00640 1 -- 172.21.15.28:0/3404064046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c1f5f40 con 0x556540476100 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb70a00640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbb580045a0 tx=0x7fbb580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84cfa86ca769c53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5801c040 con 0x556540476100 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565403b3ab0 con 0x556540476100 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb58014440 con 0x556540476100 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb58014600 con 0x556540476100 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb76ca2740 10 client.9060 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:47.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x556540389600 con 0x556540476100 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5565403b3ab0 con 0x556540476100 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.895+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbb58014760 con 0x556540476100 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 10 client.9060 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 10 client.9060 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 20 client.9060 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540426510 0x5565404268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 4 client.9060 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55654037c320 con 0x556540426510 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb71400640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540426510 0x5565404268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb71400640 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540426510 0x5565404268f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f83a4580126c67c1 server_cookie=57090fffcbcfffe0 in_seq=0 out_seq=0 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.896+0000 7fbb52a00640 10 client.9060 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb52a00640 1 -- 172.21.15.28:0/3404064046 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55654037c320 con 0x556540426510 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb52a00640 10 client.9060 handle_command_reply: tid=0 2024-09-13T17:17:47.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 client.9060 shutdown 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 msgr2=0x55654044c020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7fbb580045a0 tx=0x7fbb580046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 shutdown_connections 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540426510 0x5565404268f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x55654044cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654044c020 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb52a00640 0 client.9060 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/3404064046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044fc40 0x556540472f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 >> 172.21.15.28:0/3404064046 conn(0x5565404283e0 msgr2=0x55654046e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 shutdown_connections 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.897+0000 7fbb76ca2740 1 -- 172.21.15.28:0/3404064046 wait complete. 2024-09-13T17:17:47.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.898+0000 7fbb76ca2740 20 client.9060 trim_cache size 0 max 16384 2024-09-13T17:17:47.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.912+0000 7fbb76ca2740 1 Processor -- start 2024-09-13T17:17:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 -- start start 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x556540426e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x556540469f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654046cf20 con 0x55654044fc40 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654046de90 con 0x55654044c8c0 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556540422470 con 0x556540476100 2024-09-13T17:17:47.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.913+0000 7fbb70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45526/0 (socket says 172.21.15.28:45526) 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 -- 172.21.15.28:0/4211475622 learned_addr learned my addr 172.21.15.28:0/4211475622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb71400640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x556540426e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb71e00640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x556540469f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 -- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 msgr2=0x556540469f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x556540469f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 -- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 msgr2=0x556540426e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x556540426e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 -- 172.21.15.28:0/4211475622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565403bd5b0 con 0x55654044fc40 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb71400640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x556540426e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb70a00640 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011f60 tx=0x7fbb58024000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69f18bbf3a8c6fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb5ea00640 1 -- 172.21.15.28:0/4211475622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb580045d0 con 0x55654044fc40 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb5ea00640 1 -- 172.21.15.28:0/4211475622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb58003aa0 con 0x55654044fc40 2024-09-13T17:17:47.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.914+0000 7fbb5ea00640 1 -- 172.21.15.28:0/4211475622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58014640 con 0x55654044fc40 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 -- 172.21.15.28:0/4211475622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 msgr2=0x5565404273c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fbb58011f60 tx=0x7fbb58024000 comp rx=0 tx=0).stop 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 -- 172.21.15.28:0/4211475622 shutdown_connections 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556540476100 0x556540469f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55654044fc40 0x5565404273c0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/4211475622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044c8c0 0x556540426e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 -- 172.21.15.28:0/4211475622 >> 172.21.15.28:0/4211475622 conn(0x5565404283e0 msgr2=0x55654041e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 -- 172.21.15.28:0/4211475622 shutdown_connections 2024-09-13T17:17:47.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.915+0000 7fbb76ca2740 1 -- 172.21.15.28:0/4211475622 wait complete. 2024-09-13T17:17:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.916+0000 7fbb76ca2740 1 -- start start 2024-09-13T17:17:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.916+0000 7fbb76ca2740 1 Processor -- start 2024-09-13T17:17:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044c8c0 0x5565404d7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x55654046b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556540427420 con 0x556540476100 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55654046c460 con 0x55654044fc40 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556540446f30 con 0x55654044c8c0 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb76ca2740 10 client.0 _ll_register_callbacks cb 0x7fbb600757c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45538/0 (socket says 172.21.15.28:45538) 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 -- 172.21.15.28:0/80232661 learned_addr learned my addr 172.21.15.28:0/80232661 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 -- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044c8c0 msgr2=0x5565404d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044c8c0 0x5565404d7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 -- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 msgr2=0x55654046b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x55654046b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.917+0000 7fbb71e00640 1 -- 172.21.15.28:0/80232661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556540389600 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb71e00640 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fbb64016440 tx=0x7fbb64016900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62de682663cd568 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb52a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb64017700 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb64017860 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb60081870 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb64017a20 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb76ca2740 10 client.9066 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55654032b400 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.918+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60081870 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbb64017b80 con 0x556540476100 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 10 client.9066 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 10 client.9066 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 20 client.9066 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540428b80 0x556540469f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 4 client.9066 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x55654007e880 con 0x556540428b80 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb71400640 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540428b80 0x556540469f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb71400640 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540428b80 0x556540469f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e169b65840fa56b server_cookie=10a19a30009d7496 in_seq=0 out_seq=0 2024-09-13T17:17:47.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.919+0000 7fbb52a00640 10 client.9066 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.923+0000 7fbb52a00640 1 -- 172.21.15.28:0/80232661 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55654007e880 con 0x556540428b80 2024-09-13T17:17:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.923+0000 7fbb52a00640 10 client.9066 handle_command_reply: tid=0 2024-09-13T17:17:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.923+0000 7fbb76ca2740 1 client.9066 shutdown 2024-09-13T17:17:47.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.923+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 msgr2=0x55654047ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7fbb64016440 tx=0x7fbb64016900 comp rx=0 tx=0).stop 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 shutdown_connections 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x556540428b80 0x556540469f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556540476100 0x55654047ec50 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55654044fc40 0x55654046b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 --2- 172.21.15.28:0/80232661 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55654044c8c0 0x5565404d7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 >> 172.21.15.28:0/80232661 conn(0x5565404283e0 msgr2=0x5565404219b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb52a00640 0 client.9066 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 shutdown_connections 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 1 -- 172.21.15.28:0/80232661 wait complete. 2024-09-13T17:17:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.924+0000 7fbb76ca2740 20 client.9066 trim_cache size 0 max 16384 2024-09-13T17:17:47.928 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:17:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "751a0805-c59c-4315-962f-202cb16c4407", 2024-09-13T17:17:47.928 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:17:47.928 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 msgr2=0x7fbb4c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 0x7fbb4c06d0c0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009b20 tx=0x7fbb58009190 comp rx=0 tx=0).stop 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 msgr2=0x7fbb6c13e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7fbb6002ac70 tx=0x7fbb60004a40 comp rx=0 tx=0).stop 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 shutdown_connections 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb4c06ac00 0x7fbb4c06d0c0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb6c143d90 0x7fbb6c15dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb6c0cd840 0x7fbb6c15c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 --2- 172.21.15.28:0/2714509639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb6c06b8a0 0x7fbb6c13e630 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:47.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 >> 172.21.15.28:0/2714509639 conn(0x7fbb6c073470 msgr2=0x7fbb6c0ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:47.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 shutdown_connections 2024-09-13T17:17:47.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:47.925+0000 7fbb72800640 1 -- 172.21.15.28:0/2714509639 wait complete. 2024-09-13T17:17:47.948 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.9:0 scrub pause 2024-09-13T17:17:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.146+0000 7f5e4c200640 1 Processor -- start 2024-09-13T17:17:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.146+0000 7f5e4c200640 1 -- start start 2024-09-13T17:17:48.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4c200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 0x7f5e440712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4c200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4406c8b0 con 0x7f5e4415ab60 2024-09-13T17:17:48.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4c200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e4406c6a0 con 0x7f5e4406c250 2024-09-13T17:17:48.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4406cb90 con 0x7f5e4415a780 2024-09-13T17:17:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4a200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 0x7f5e440712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e48e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5e4415ab60 0x7f5e4406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:49916/0 (socket says 172.21.15.28:49916) 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e48e00640 1 -- 172.21.15.28:0/2645696674 learned_addr learned my addr 172.21.15.28:0/2645696674 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:33980/0 (socket says 172.21.15.28:33980) 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e43e00640 1 -- 172.21.15.28:0/2645696674 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869055160 0 0) 0x7f5e4406c8b0 con 0x7f5e4415ab60 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e43e00640 1 -- 172.21.15.28:0/2645696674 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003520 con 0x7f5e4415ab60 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 -- 172.21.15.28:0/2645696674 >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 msgr2=0x7f5e440712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 --2- 172.21.15.28:0/2645696674 >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 0x7f5e440712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 -- 172.21.15.28:0/2645696674 >> v1:172.21.15.189:6789/0 conn(0x7f5e4415ab60 legacy=0x7f5e4406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e49800640 1 -- 172.21.15.28:0/2645696674 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e440cc980 con 0x7f5e4415a780 2024-09-13T17:17:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.147+0000 7f5e4a200640 1 --2- 172.21.15.28:0/2645696674 >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 0x7f5e440712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.148+0000 7f5e49800640 1 --2- 172.21.15.28:0/2645696674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5e380097a0 tx=0x7f5e3802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8add6e17735cc3d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.149+0000 7f5e43e00640 1 -- 172.21.15.28:0/2645696674 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38032440 con 0x7f5e4415a780 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.149+0000 7f5e43e00640 1 -- 172.21.15.28:0/2645696674 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e380325a0 con 0x7f5e4415a780 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.149+0000 7f5e43e00640 1 -- 172.21.15.28:0/2645696674 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38032930 con 0x7f5e4415a780 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.149+0000 7f5e4c200640 1 -- 172.21.15.28:0/2645696674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 msgr2=0x7f5e44070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.149+0000 7f5e4c200640 1 --2- 172.21.15.28:0/2645696674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5e380097a0 tx=0x7f5e3802a180 comp rx=0 tx=0).stop 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 -- 172.21.15.28:0/2645696674 shutdown_connections 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 --2- 172.21.15.28:0/2645696674 >> v2:172.21.15.189:3300/0 conn(0x7f5e4406c250 0x7f5e440712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 --2- 172.21.15.28:0/2645696674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e44070d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 -- 172.21.15.28:0/2645696674 >> 172.21.15.28:0/2645696674 conn(0x7f5e440b0270 msgr2=0x7f5e440b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 -- 172.21.15.28:0/2645696674 shutdown_connections 2024-09-13T17:17:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.150+0000 7f5e4c200640 1 -- 172.21.15.28:0/2645696674 wait complete. 2024-09-13T17:17:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 Processor -- start 2024-09-13T17:17:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 -- start start 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 0x7f5e4414be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4414ba20 con 0x7f5e4406c250 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4414ada0 con 0x7f5e4415ab60 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.152+0000 7f5e4c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4414b080 con 0x7f5e4415a780 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e4a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 0x7f5e4414be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45550/0 (socket says 172.21.15.28:45550) 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 -- 172.21.15.28:0/1499829714 learned_addr learned my addr 172.21.15.28:0/1499829714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e4a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 0x7f5e4414be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:44510/0 (socket says 172.21.15.28:44510) 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:33984/0 (socket says 172.21.15.28:33984) 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 -- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 msgr2=0x7f5e440af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 -- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 msgr2=0x7f5e4414be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 0x7f5e4414be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e49800640 1 -- 172.21.15.28:0/1499829714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c009a70 con 0x7f5e4406c250 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.153+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e49800640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5e38035040 tx=0x7f5e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e41600640 1 -- 172.21.15.28:0/1499829714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3803d830 con 0x7f5e4406c250 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e41600640 1 -- 172.21.15.28:0/1499829714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3803d9b0 con 0x7f5e4406c250 2024-09-13T17:17:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e38009400 con 0x7f5e4406c250 2024-09-13T17:17:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e41600640 1 -- 172.21.15.28:0/1499829714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3803dd60 con 0x7f5e4406c250 2024-09-13T17:17:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.154+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4414b640 con 0x7f5e4406c250 2024-09-13T17:17:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.156+0000 7f5e41600640 1 -- 172.21.15.28:0/1499829714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e38039040 con 0x7f5e4406c250 2024-09-13T17:17:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.157+0000 7f5e41600640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 0x7f5e2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.157+0000 7f5e41600640 1 -- 172.21.15.28:0/1499829714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e380a9a30 con 0x7f5e4406c250 2024-09-13T17:17:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.157+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 0x7f5e2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 Processor -- start 2024-09-13T17:17:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 -- start start 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55817560d380 0x55817560d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55817560ab40 0x55817565a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 0x7f5e2406d0c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5e44067240 tx=0x7f5e30007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558175518000 con 0x558175634830 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558175575d40 con 0x55817560d380 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.158+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581755734d0 con 0x55817560ab40 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55817560ab40 0x55817565a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45556/0 (socket says 172.21.15.28:45556) 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 -- 172.21.15.28:0/546696678 learned_addr learned my addr 172.21.15.28:0/546696678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e49800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55817560d380 0x55817560d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 -- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55817560ab40 msgr2=0x55817565a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55817560ab40 0x55817565a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 -- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55817560d380 msgr2=0x55817560d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55817560d380 0x55817560d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 -- 172.21.15.28:0/546696678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55817552a3a0 con 0x558175634830 2024-09-13T17:17:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.159+0000 7f5e4a200640 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0115b0 tx=0x7f5e3c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f277f9337ccde5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e36000640 1 -- 172.21.15.28:0/546696678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c00e040 con 0x558175634830 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e36000640 1 -- 172.21.15.28:0/546696678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c00f040 con 0x558175634830 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e36000640 1 -- 172.21.15.28:0/546696678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3c003dd0 con 0x558175634830 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 -- 172.21.15.28:0/546696678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 msgr2=0x55817565ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c0115b0 tx=0x7f5e3c009750 comp rx=0 tx=0).stop 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 -- 172.21.15.28:0/546696678 shutdown_connections 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558175634830 0x55817565ad90 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55817560ab40 0x55817565a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 --2- 172.21.15.28:0/546696678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55817560d380 0x55817560d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 -- 172.21.15.28:0/546696678 >> 172.21.15.28:0/546696678 conn(0x558175609ab0 msgr2=0x558175651f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 -- 172.21.15.28:0/546696678 shutdown_connections 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.160+0000 7f5e4f032740 1 -- 172.21.15.28:0/546696678 wait complete. 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.161+0000 7f5e4f032740 1 -- start start 2024-09-13T17:17:48.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.161+0000 7f5e4f032740 1 Processor -- start 2024-09-13T17:17:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558175634830 0x558175607460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581756079a0 0x5581756598b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817556edc0 con 0x55817560d380 2024-09-13T17:17:48.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55817552d3e0 con 0x558175634830 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55817552d080 con 0x5581756079a0 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e4f032740 10 client.0 _ll_register_callbacks cb 0x558175568430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.162+0000 7f5e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558175634830 0x558175607460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 -- 172.21.15.28:0/2028663002 learned_addr learned my addr 172.21.15.28:0/2028663002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e4a200640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581756079a0 0x5581756598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 -- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581756079a0 msgr2=0x5581756598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581756079a0 0x5581756598b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 -- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558175634830 msgr2=0x558175607460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558175634830 0x558175607460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 -- 172.21.15.28:0/2028663002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c011290 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e49800640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5e380a7ef0 tx=0x7f5e38078bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0f60504ebdd8ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e38076ef0 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e38077f80 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.163+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558175598600 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e38078140 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 10 client.9075 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55817556d440 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e380a5030 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e380901b0 con 0x55817560d380 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 10 client.9075 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:48.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 10 client.9075 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 20 client.9075 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175620ca0 0x5581756210b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 4 client.9075 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558175560c90 con 0x558175620ca0 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.164+0000 7f5e48e00640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175620ca0 0x5581756210b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.165+0000 7f5e48e00640 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175620ca0 0x5581756210b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5d220ac95787f0e server_cookie=656dc7bc7ef2e246 in_seq=0 out_seq=0 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.165+0000 7f5e2aa00640 10 client.9075 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.165+0000 7f5e2aa00640 1 -- 172.21.15.28:0/2028663002 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x558175560c90 con 0x558175620ca0 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.165+0000 7f5e2aa00640 10 client.9075 handle_command_reply: tid=0 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 client.9075 shutdown 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 msgr2=0x55817564ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f5e380a7ef0 tx=0x7f5e38078bd0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 shutdown_connections 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175620ca0 0x5581756210b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581756079a0 0x5581756598b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558175634830 0x558175607460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 --2- 172.21.15.28:0/2028663002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817564ef50 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e2aa00640 0 client.9075 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 >> 172.21.15.28:0/2028663002 conn(0x55817560ab40 msgr2=0x558175652a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 shutdown_connections 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 1 -- 172.21.15.28:0/2028663002 wait complete. 2024-09-13T17:17:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.166+0000 7f5e4f032740 20 client.9075 trim_cache size 0 max 16384 2024-09-13T17:17:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.181+0000 7f5e4f032740 1 Processor -- start 2024-09-13T17:17:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.181+0000 7f5e4f032740 1 -- start start 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x558175608700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x558175659720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581752d6040 con 0x55817560d380 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558175608c40 con 0x5581756079a0 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558175656840 con 0x558175634830 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45578/0 (socket says 172.21.15.28:45578) 2024-09-13T17:17:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 -- 172.21.15.28:0/161812041 learned_addr learned my addr 172.21.15.28:0/161812041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e4a200640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x558175659720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e49800640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x558175608700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 -- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 msgr2=0x558175659720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x558175659720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 -- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 msgr2=0x558175608700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.182+0000 7f5e48e00640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x558175608700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e48e00640 1 -- 172.21.15.28:0/161812041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55817550f5d0 con 0x55817560d380 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e4a200640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x558175659720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e49800640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x558175608700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e48e00640 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001eb0 tx=0x7f5e3000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3a9b638478092 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e36a00640 1 -- 172.21.15.28:0/161812041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30028070 con 0x55817560d380 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e36a00640 1 -- 172.21.15.28:0/161812041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e300238a0 con 0x55817560d380 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e36a00640 1 -- 172.21.15.28:0/161812041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30023a30 con 0x55817560d380 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e4f032740 1 -- 172.21.15.28:0/161812041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 msgr2=0x5581756591e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.183+0000 7f5e4f032740 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f5e30001eb0 tx=0x7f5e3000a000 comp rx=0 tx=0).stop 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 -- 172.21.15.28:0/161812041 shutdown_connections 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x558175659720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x5581756591e0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 --2- 172.21.15.28:0/161812041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x558175608700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 -- 172.21.15.28:0/161812041 >> 172.21.15.28:0/161812041 conn(0x55817560ab40 msgr2=0x55817564f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 -- 172.21.15.28:0/161812041 shutdown_connections 2024-09-13T17:17:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 -- 172.21.15.28:0/161812041 wait complete. 2024-09-13T17:17:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 -- start start 2024-09-13T17:17:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.184+0000 7f5e4f032740 1 Processor -- start 2024-09-13T17:17:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x5581756073b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x5581756a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558175671430 con 0x55817560d380 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581756511d0 con 0x5581756079a0 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558175651700 con 0x558175634830 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.185+0000 7f5e4f032740 10 client.0 _ll_register_callbacks cb 0x7f5e380783f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45584/0 (socket says 172.21.15.28:45584) 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 -- 172.21.15.28:0/1329798347 learned_addr learned my addr 172.21.15.28:0/1329798347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e4a200640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x5581756a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e49800640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x5581756073b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 -- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 msgr2=0x5581756a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x5581756a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 -- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 msgr2=0x5581756073b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x5581756073b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 -- 172.21.15.28:0/1329798347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55817556d440 con 0x55817560d380 2024-09-13T17:17:48.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e4a200640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x5581756a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e49800640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x5581756073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e48e00640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5e300011e0 tx=0x7f5e30000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18370bf7ab09b82e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.186+0000 7f5e2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e30024ef0 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e3001c040 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e3c00f040 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e30005850 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 10 client.9081 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55817552a3a0 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3c00f040 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e30012070 con 0x55817560d380 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 10 client.9081 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 10 client.9081 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:48.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 20 client.9081 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175670630 0x558175670a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 4 client.9081 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.187+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x5581752624e0 con 0x558175670630 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e49800640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175670630 0x558175670a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e49800640 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175670630 0x558175670a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da3c1a89d703861 server_cookie=726f869ea621c2f1 in_seq=0 out_seq=0 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e2aa00640 10 client.9081 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e2aa00640 1 -- 172.21.15.28:0/1329798347 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x5581752624e0 con 0x558175670630 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e2aa00640 10 client.9081 handle_command_reply: tid=0 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e4f032740 1 client.9081 shutdown 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 msgr2=0x55817560c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.188+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f5e300011e0 tx=0x7f5e30000ed0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 shutdown_connections 2024-09-13T17:17:48.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x558175670630 0x558175670a40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558175634830 0x5581756a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55817560d380 0x55817560c1f0 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 --2- 172.21.15.28:0/1329798347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581756079a0 0x5581756073b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e2aa00640 0 client.9081 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 >> 172.21.15.28:0/1329798347 conn(0x55817560ab40 msgr2=0x558175651c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 shutdown_connections 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 1 -- 172.21.15.28:0/1329798347 wait complete. 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4f032740 20 client.9081 trim_cache size 0 max 16384 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:48.195 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:17:48.196 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 msgr2=0x7f5e2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 0x7f5e2406d0c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f5e44067240 tx=0x7f5e30007d30 comp rx=0 tx=0).stop 2024-09-13T17:17:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 msgr2=0x7f5e4414a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f5e38035040 tx=0x7f5e380046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 shutdown_connections 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e2406ac00 0x7f5e2406d0c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e4415ab60 0x7f5e4414be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e4415a780 0x7f5e440af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 --2- 172.21.15.28:0/1499829714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e4406c250 0x7f5e4414a7c0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.189+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 >> 172.21.15.28:0/1499829714 conn(0x7f5e440b0270 msgr2=0x7f5e440b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.190+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 shutdown_connections 2024-09-13T17:17:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.190+0000 7f5e4c200640 1 -- 172.21.15.28:0/1499829714 wait complete. 2024-09-13T17:17:48.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.9:0 scrub status 2024-09-13T17:17:48.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.362+0000 7f81b1e00640 1 Processor -- start 2024-09-13T17:17:48.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.362+0000 7f81b1e00640 1 -- start start 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.363+0000 7f81b1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81b1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac15ab60 con 0x7f81ac159770 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81b1e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac06f920 con 0x7f81ac06f3f0 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81b1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac15b290 con 0x7f81ac070430 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81aaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81ac159770 0x7f81ac06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35138/0 (socket says 172.21.15.28:35138) 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81aaa00640 1 -- 172.21.15.28:0/1711559943 learned_addr learned my addr 172.21.15.28:0/1711559943 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81a2000640 1 --2- 172.21.15.28:0/1711559943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81a2000640 1 -- 172.21.15.28:0/1711559943 >> v1:172.21.15.189:6789/0 conn(0x7f81ac06f3f0 legacy=0x7f81ac15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81aa000640 1 -- 172.21.15.28:0/1711559943 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108455192 0 0) 0x7f819c001110 con 0x7f81ac159770 2024-09-13T17:17:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81a2000640 1 -- 172.21.15.28:0/1711559943 >> v1:172.21.15.28:6789/0 conn(0x7f81ac159770 legacy=0x7f81ac06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.364+0000 7f81a2000640 1 -- 172.21.15.28:0/1711559943 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac15b570 con 0x7f81ac070430 2024-09-13T17:17:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81a2000640 1 --2- 172.21.15.28:0/1711559943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac15bd70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8194009750 tx=0x7f819402a010 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6b2775082a766f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81aa000640 1 -- 172.21.15.28:0/1711559943 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819402ad20 con 0x7f81ac070430 2024-09-13T17:17:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81aa000640 1 -- 172.21.15.28:0/1711559943 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819402ae80 con 0x7f81ac070430 2024-09-13T17:17:48.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81aa000640 1 -- 172.21.15.28:0/1711559943 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194032660 con 0x7f81ac070430 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81b1e00640 1 -- 172.21.15.28:0/1711559943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 msgr2=0x7f81ac15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.366+0000 7f81b1e00640 1 --2- 172.21.15.28:0/1711559943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac15bd70 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8194009750 tx=0x7f819402a010 comp rx=0 tx=0).stop 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.367+0000 7f81b1e00640 1 -- 172.21.15.28:0/1711559943 shutdown_connections 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.367+0000 7f81b1e00640 1 --2- 172.21.15.28:0/1711559943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac15bd70 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.367+0000 7f81b1e00640 1 -- 172.21.15.28:0/1711559943 >> 172.21.15.28:0/1711559943 conn(0x7f81ac0b0270 msgr2=0x7f81ac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.367+0000 7f81b1e00640 1 -- 172.21.15.28:0/1711559943 shutdown_connections 2024-09-13T17:17:48.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.367+0000 7f81b1e00640 1 -- 172.21.15.28:0/1711559943 wait complete. 2024-09-13T17:17:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 Processor -- start 2024-09-13T17:17:48.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 -- start start 2024-09-13T17:17:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac1421b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81ac159770 0x7f81ac14f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac1426f0 con 0x7f81ac06f3f0 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac153fb0 con 0x7f81ac159770 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.369+0000 7f81b1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac14f700 con 0x7f81ac070430 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45592/0 (socket says 172.21.15.28:45592) 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 -- 172.21.15.28:0/3549781405 learned_addr learned my addr 172.21.15.28:0/3549781405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81a2000640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac1421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81ab400640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81ac159770 0x7f81ac14f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 -- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 msgr2=0x7f81ac1421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac1421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 -- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81ac159770 msgr2=0x7f81ac14f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81ac159770 0x7f81ac14f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 -- 172.21.15.28:0/3549781405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8194009400 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.370+0000 7f81aaa00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f819c00c3f0 tx=0x7f819c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.371+0000 7f81a3e00640 1 -- 172.21.15.28:0/3549781405 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c0074e0 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.371+0000 7f81a3e00640 1 -- 172.21.15.28:0/3549781405 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c007640 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.371+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac14d850 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.371+0000 7f81a3e00640 1 -- 172.21.15.28:0/3549781405 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c004a00 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.371+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac154200 con 0x7f81ac06f3f0 2024-09-13T17:17:48.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81a3e00640 1 -- 172.21.15.28:0/3549781405 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f819c004b60 con 0x7f81ac06f3f0 2024-09-13T17:17:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81a3e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 0x7f818806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81a3e00640 1 -- 172.21.15.28:0/3549781405 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f819c088fa0 con 0x7f81ac06f3f0 2024-09-13T17:17:48.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81a2000640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 0x7f818806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81b4dec740 1 Processor -- start 2024-09-13T17:17:48.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.374+0000 7f81b4dec740 1 -- start start 2024-09-13T17:17:48.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81a2000640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 0x7f818806d0c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8194034000 tx=0x7f819402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539df7c60 0x55e539df8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1f110 0x55e539e45670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539cfe6a0 con 0x55e539df5420 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539d61cc0 con 0x55e539df7c60 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.375+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e539d5f450 con 0x55e539e1f110 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45598/0 (socket says 172.21.15.28:45598) 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 -- 172.21.15.28:0/3159858489 learned_addr learned my addr 172.21.15.28:0/3159858489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81aaa00640 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539df7c60 0x55e539df8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81ab400640 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1f110 0x55e539e45670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 -- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1f110 msgr2=0x55e539e45670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1f110 0x55e539e45670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 -- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539df7c60 msgr2=0x55e539df8040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539df7c60 0x55e539df8040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 -- 172.21.15.28:0/3159858489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e539d2b2e0 con 0x55e539df5420 2024-09-13T17:17:48.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.376+0000 7f81a2000640 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f819402a600 tx=0x7f819402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99bd3e07c7746f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f819be00640 1 -- 172.21.15.28:0/3159858489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194003a10 con 0x55e539df5420 2024-09-13T17:17:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f819be00640 1 -- 172.21.15.28:0/3159858489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194003b70 con 0x55e539df5420 2024-09-13T17:17:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f819be00640 1 -- 172.21.15.28:0/3159858489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194003ee0 con 0x55e539df5420 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 -- 172.21.15.28:0/3159858489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 msgr2=0x55e539e45130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f819402a600 tx=0x7f819402c000 comp rx=0 tx=0).stop 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 -- 172.21.15.28:0/3159858489 shutdown_connections 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1f110 0x55e539e45670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df5420 0x55e539e45130 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 --2- 172.21.15.28:0/3159858489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539df7c60 0x55e539df8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 -- 172.21.15.28:0/3159858489 >> 172.21.15.28:0/3159858489 conn(0x55e539df4390 msgr2=0x55e539e3c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 -- 172.21.15.28:0/3159858489 shutdown_connections 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.377+0000 7f81b4dec740 1 -- 172.21.15.28:0/3159858489 wait complete. 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.378+0000 7f81b4dec740 1 -- start start 2024-09-13T17:17:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.378+0000 7f81b4dec740 1 Processor -- start 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539e1cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1e7d0 0x55e539df2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539d5ad40 con 0x55e539e1f110 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539cfc280 con 0x55e539e1e7d0 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e539cfbf20 con 0x55e539df7c60 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81b4dec740 10 client.0 _ll_register_callbacks cb 0x55e539d543b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.379+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81aaa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539e1cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45600/0 (socket says 172.21.15.28:45600) 2024-09-13T17:17:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 -- 172.21.15.28:0/772873594 learned_addr learned my addr 172.21.15.28:0/772873594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81ab400640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1e7d0 0x55e539df2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 -- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 msgr2=0x55e539e1cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539e1cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 -- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1e7d0 msgr2=0x55e539df2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1e7d0 0x55e539df2680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 -- 172.21.15.28:0/772873594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac15b570 con 0x55e539e1f110 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81aaa00640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539e1cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f81a2000640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f819404d040 tx=0x7f8194004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9ce5317aa3f85a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f8198c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.380+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819403d890 con 0x55e539e1f110 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8194043040 con 0x55e539e1f110 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e539a4e4e0 con 0x55e539e1f110 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f819403d9f0 con 0x55e539e1f110 2024-09-13T17:17:48.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 10 client.9093 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55e539d593c0 con 0x55e539e1f110 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e539a4e4e0 con 0x55e539e1f110 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8194059020 con 0x55e539e1f110 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 10 client.9093 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 10 client.9093 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 20 client.9093 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e1d740 0x55e539e1db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 4 client.9093 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e539a5f7e0 con 0x55e539e1d740 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.381+0000 7f81aaa00640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e1d740 0x55e539e1db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.382+0000 7f81aaa00640 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e1d740 0x55e539e1db50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67a421b872b4db63 server_cookie=e28c2565b3a97a5b in_seq=0 out_seq=0 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.382+0000 7f8198c00640 10 client.9093 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.382+0000 7f8198c00640 1 -- 172.21.15.28:0/772873594 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55e539a5f7e0 con 0x55e539e1d740 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.382+0000 7f8198c00640 10 client.9093 handle_command_reply: tid=0 2024-09-13T17:17:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 client.9093 shutdown 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 msgr2=0x55e539df2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f819404d040 tx=0x7f8194004730 comp rx=0 tx=0).stop 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 shutdown_connections 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e1d740 0x55e539e1db50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1e7d0 0x55e539df2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1f110 0x55e539df2140 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 --2- 172.21.15.28:0/772873594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539e1cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f8198c00640 0 client.9093 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 >> 172.21.15.28:0/772873594 conn(0x55e539df5420 msgr2=0x55e539e40550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 shutdown_connections 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 1 -- 172.21.15.28:0/772873594 wait complete. 2024-09-13T17:17:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.383+0000 7f81b4dec740 20 client.9093 trim_cache size 0 max 16384 2024-09-13T17:17:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.398+0000 7f81b4dec740 1 Processor -- start 2024-09-13T17:17:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.398+0000 7f81b4dec740 1 -- start start 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 0x55e539e42650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539df74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539ac2040 con 0x55e539df7c60 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539e3f8d0 con 0x55e539e1f110 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e539e22ba0 con 0x55e539e1e7d0 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45618/0 (socket says 172.21.15.28:45618) 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 -- 172.21.15.28:0/2694658151 learned_addr learned my addr 172.21.15.28:0/2694658151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81a2000640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 0x55e539e42650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81ab400640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539df74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 -- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 msgr2=0x55e539e42650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 0x55e539e42650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 -- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 msgr2=0x55e539df74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539df74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81aaa00640 1 -- 172.21.15.28:0/2694658151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e539cfb610 con 0x55e539df7c60 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.399+0000 7f81a2000640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 0x55e539e42650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81aaa00640 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f819c057b30 tx=0x7f819c057d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52070c4d515691a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81a0c00640 1 -- 172.21.15.28:0/2694658151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c060220 con 0x55e539df7c60 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81a0c00640 1 -- 172.21.15.28:0/2694658151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819c004e10 con 0x55e539df7c60 2024-09-13T17:17:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81a0c00640 1 -- 172.21.15.28:0/2694658151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819c087540 con 0x55e539df7c60 2024-09-13T17:17:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81b4dec740 1 -- 172.21.15.28:0/2694658151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 msgr2=0x55e539e21500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.400+0000 7f81b4dec740 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f819c057b30 tx=0x7f819c057d30 comp rx=0 tx=0).stop 2024-09-13T17:17:48.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 -- 172.21.15.28:0/2694658151 shutdown_connections 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539df74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539e1e7d0 0x55e539e42650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 --2- 172.21.15.28:0/2694658151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539df7c60 0x55e539e21500 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 -- 172.21.15.28:0/2694658151 >> 172.21.15.28:0/2694658151 conn(0x55e539df5420 msgr2=0x55e539e21a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 -- 172.21.15.28:0/2694658151 shutdown_connections 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 -- 172.21.15.28:0/2694658151 wait complete. 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 -- start start 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.401+0000 7f81b4dec740 1 Processor -- start 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539ea0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539eaa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539e44770 con 0x55e539e1e7d0 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e539e4bb00 con 0x55e539e1f110 2024-09-13T17:17:48.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e539e43f60 con 0x55e539df7c60 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.402+0000 7f81b4dec740 10 client.0 _ll_register_callbacks cb 0x7f819404d070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45632/0 (socket says 172.21.15.28:45632) 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 -- 172.21.15.28:0/124012176 learned_addr learned my addr 172.21.15.28:0/124012176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81aaa00640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539ea0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81ab400640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539eaa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 -- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 msgr2=0x55e539ea0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539ea0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 -- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 msgr2=0x55e539eaa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539eaa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 -- 172.21.15.28:0/124012176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac154200 con 0x55e539e1e7d0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f81a2000640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f8194004690 tx=0x7f8194008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4f465549ebeb2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.403+0000 7f8198c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819403dba0 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8194043070 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8194003b70 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8194006e10 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 10 client.9099 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55e539d20b20 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194003b70 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8194059040 con 0x55e539e1e7d0 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 10 client.9099 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 10 client.9099 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 20 client.9099 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e4aef0 0x55e539e3a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 4 client.9099 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:17:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.404+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e539ab2e30 con 0x55e539e4aef0 2024-09-13T17:17:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f81aaa00640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e4aef0 0x55e539e3a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f81aaa00640 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e4aef0 0x55e539e3a7f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63d040536b1ebf9e server_cookie=bc9ace08bc30f841 in_seq=0 out_seq=0 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f8198c00640 10 client.9099 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f8198c00640 1 -- 172.21.15.28:0/124012176 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x55e539ab2e30 con 0x55e539e4aef0 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f8198c00640 10 client.9099 handle_command_reply: tid=0 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f81b4dec740 1 client.9099 shutdown 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 msgr2=0x55e539e3b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.405+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f8194004690 tx=0x7f8194008030 comp rx=0 tx=0).stop 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 shutdown_connections 2024-09-13T17:17:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55e539e4aef0 0x55e539e3a7f0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e539e1f110 0x55e539eaa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e539e1e7d0 0x55e539e3b9b0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 --2- 172.21.15.28:0/124012176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e539df7c60 0x55e539ea0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f8198c00640 0 client.9099 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 >> 172.21.15.28:0/124012176 conn(0x55e539df5420 msgr2=0x55e539e21a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 shutdown_connections 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 1 -- 172.21.15.28:0/124012176 wait complete. 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b4dec740 20 client.9099 trim_cache size 0 max 16384 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:48.412 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: "751a0805-c59c-4315-962f-202cb16c4407": { 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_control_test_path", 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "751a0805-c59c-4315-962f-202cb16c4407", 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:17:48.413 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 msgr2=0x7f818806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 0x7f818806d0c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8194034000 tx=0x7f819402e000 comp rx=0 tx=0).stop 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 msgr2=0x7f81ac145d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f819c00c3f0 tx=0x7f819c00c8b0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 shutdown_connections 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f818806ac00 0x7f818806d0c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.406+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81ac159770 0x7f81ac14f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.407+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81ac070430 0x7f81ac1421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.407+0000 7f81b1e00640 1 --2- 172.21.15.28:0/3549781405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81ac06f3f0 0x7f81ac145d70 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.407+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 >> 172.21.15.28:0/3549781405 conn(0x7f81ac0b0270 msgr2=0x7f81ac06c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.407+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 shutdown_connections 2024-09-13T17:17:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.407+0000 7f81b1e00640 1 -- 172.21.15.28:0/3549781405 wait complete. 2024-09-13T17:17:48.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.580+0000 7f9950600640 1 Processor -- start 2024-09-13T17:17:48.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.580+0000 7f9950600640 1 -- start start 2024-09-13T17:17:48.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 0x7f9948161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 0x7f9948070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99480711d0 0x7f994806b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99481618f0 con 0x7f99480711d0 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99480b35c0 con 0x7f994806cde0 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.581+0000 7f9950600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480b38a0 con 0x7f994815e860 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99480711d0 0x7f994806b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f99480711d0 0x7f994806b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45636/0 (socket says 172.21.15.28:45636) 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 -- 172.21.15.28:0/86881532 learned_addr learned my addr 172.21.15.28:0/86881532 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994dc00640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 0x7f9948161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994d200640 1 --2- 172.21.15.28:0/86881532 >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 0x7f9948070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 -- 172.21.15.28:0/86881532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 msgr2=0x7f9948161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 0x7f9948161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 -- 172.21.15.28:0/86881532 >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 msgr2=0x7f9948070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 --2- 172.21.15.28:0/86881532 >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 0x7f9948070c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 -- 172.21.15.28:0/86881532 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9948161230 con 0x7f99480711d0 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994dc00640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 0x7f9948161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994d200640 1 --2- 172.21.15.28:0/86881532 >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 0x7f9948070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f994e600640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f994806b1c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f994806d6c0 tx=0x7f994002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f0b8dbda2ec046 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.582+0000 7f9947e00640 1 -- 172.21.15.28:0/86881532 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994002ae90 con 0x7f99480711d0 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9947e00640 1 -- 172.21.15.28:0/86881532 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9940032440 con 0x7f99480711d0 2024-09-13T17:17:48.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9947e00640 1 -- 172.21.15.28:0/86881532 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99400327d0 con 0x7f99480711d0 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 -- 172.21.15.28:0/86881532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 msgr2=0x7f994806b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f994806b1c0 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f994806d6c0 tx=0x7f994002a140 comp rx=0 tx=0).stop 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 -- 172.21.15.28:0/86881532 shutdown_connections 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f994806b1c0 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 --2- 172.21.15.28:0/86881532 >> v2:172.21.15.189:3300/0 conn(0x7f994806cde0 0x7f9948070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 --2- 172.21.15.28:0/86881532 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994815e860 0x7f9948161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 -- 172.21.15.28:0/86881532 >> 172.21.15.28:0/86881532 conn(0x7f9948069e80 msgr2=0x7f994806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 -- 172.21.15.28:0/86881532 shutdown_connections 2024-09-13T17:17:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.583+0000 7f9950600640 1 -- 172.21.15.28:0/86881532 wait complete. 2024-09-13T17:17:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.585+0000 7f9950600640 1 Processor -- start 2024-09-13T17:17:48.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.585+0000 7f9950600640 1 -- start start 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994806cde0 0x7f9948154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f994815e860 0x7f994815a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99481eb760 con 0x7f99480711d0 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948154fa0 con 0x7f994815e860 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f9950600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99481ea750 con 0x7f994806cde0 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f994d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f994d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45638/0 (socket says 172.21.15.28:45638) 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f994d200640 1 -- 172.21.15.28:0/1005983230 learned_addr learned my addr 172.21.15.28:0/1005983230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f994dc00640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994806cde0 0x7f9948154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.586+0000 7f994e600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f994815e860 0x7f994815a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 -- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994806cde0 msgr2=0x7f9948154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994806cde0 0x7f9948154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 -- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f994815e860 msgr2=0x7f994815a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f994815e860 0x7f994815a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 -- 172.21.15.28:0/1005983230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9934009400 con 0x7f99480711d0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f994d200640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f993400e890 tx=0x7f993400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993400c940 con 0x7f99480711d0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993400caa0 con 0x7f99480711d0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993400ce30 con 0x7f99480711d0 2024-09-13T17:17:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c009a70 con 0x7f99480711d0 2024-09-13T17:17:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.587+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99400094d0 con 0x7f99480711d0 2024-09-13T17:17:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.588+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99480b31c0 con 0x7f99480711d0 2024-09-13T17:17:48.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.590+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9934013470 con 0x7f99480711d0 2024-09-13T17:17:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.590+0000 7f9945600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 0x7f992806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.590+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9934089500 con 0x7f99480711d0 2024-09-13T17:17:48.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.590+0000 7f994dc00640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 0x7f992806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.591+0000 7f994dc00640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 0x7f992806d0c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f993c009fd0 tx=0x7f993c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.596+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f99340546f0 con 0x7f99480711d0 2024-09-13T17:17:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.814+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f994806f5c0 con 0x7f99480711d0 2024-09-13T17:17:48.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.814+0000 7f9945600640 1 -- 172.21.15.28:0/1005983230 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f9934057d10 con 0x7f99480711d0 2024-09-13T17:17:48.815 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:48.816 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:48.816 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 msgr2=0x7f992806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 0x7f992806d0c0 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f993c009fd0 tx=0x7f993c00b040 comp rx=0 tx=0).stop 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 msgr2=0x7f9948159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f993400e890 tx=0x7f993400ed50 comp rx=0 tx=0).stop 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 shutdown_connections 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f992806ac00 0x7f992806d0c0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f994815e860 0x7f994815a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f99480711d0 0x7f9948159c70 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 --2- 172.21.15.28:0/1005983230 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f994806cde0 0x7f9948154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 >> 172.21.15.28:0/1005983230 conn(0x7f9948069e80 msgr2=0x7f994815ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 shutdown_connections 2024-09-13T17:17:48.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.817+0000 7f9950600640 1 -- 172.21.15.28:0/1005983230 wait complete. 2024-09-13T17:17:48.833 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:48.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.987+0000 7f5d91a00640 1 Processor -- start 2024-09-13T17:17:48.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.987+0000 7f5d91a00640 1 -- start start 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d91a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5d8c0cdfa0 0x7f5d8c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d91a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 0x7f5d8c0ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d91a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8c0b3970 con 0x7f5d8c06f440 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d91a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8c159230 con 0x7f5d8c0cdfa0 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d91a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c159430 con 0x7f5d8c0b3160 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d8b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5d8c06f440 0x7f5d8c0b1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35144/0 (socket says 172.21.15.28:35144) 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d8b400640 1 -- 172.21.15.28:0/3794614253 learned_addr learned my addr 172.21.15.28:0/3794614253 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d82000640 1 --2- 172.21.15.28:0/3794614253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 0x7f5d8c0ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.988+0000 7f5d8aa00640 1 --2- 172.21.15.28:0/3794614253 >> v2:172.21.15.189:3300/0 conn(0x7f5d8c0cdfa0 0x7f5d8c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.989+0000 7f5d82000640 1 -- 172.21.15.28:0/3794614253 >> v2:172.21.15.189:3300/0 conn(0x7f5d8c0cdfa0 msgr2=0x7f5d8c15d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.989+0000 7f5d82000640 1 --2- 172.21.15.28:0/3794614253 >> v2:172.21.15.189:3300/0 conn(0x7f5d8c0cdfa0 0x7f5d8c15d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.989+0000 7f5d82000640 1 -- 172.21.15.28:0/3794614253 >> v1:172.21.15.28:6789/0 conn(0x7f5d8c06f440 legacy=0x7f5d8c0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.989+0000 7f5d8a000640 1 -- 172.21.15.28:0/3794614253 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 915752146 0 0) 0x7f5d84001110 con 0x7f5d8c06f440 2024-09-13T17:17:48.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.989+0000 7f5d82000640 1 -- 172.21.15.28:0/3794614253 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8c159710 con 0x7f5d8c0b3160 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.990+0000 7f5d82000640 1 --2- 172.21.15.28:0/3794614253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 0x7f5d8c0ce560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5d740097a0 tx=0x7f5d7402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44a4cb90abcf1647 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.990+0000 7f5d8a000640 1 -- 172.21.15.28:0/3794614253 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74032440 con 0x7f5d8c0b3160 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.990+0000 7f5d8a000640 1 -- 172.21.15.28:0/3794614253 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d740325a0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.990+0000 7f5d8a000640 1 -- 172.21.15.28:0/3794614253 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74032930 con 0x7f5d8c0b3160 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 -- 172.21.15.28:0/3794614253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 msgr2=0x7f5d8c0ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 --2- 172.21.15.28:0/3794614253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 0x7f5d8c0ce560 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5d740097a0 tx=0x7f5d7402a180 comp rx=0 tx=0).stop 2024-09-13T17:17:48.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 -- 172.21.15.28:0/3794614253 shutdown_connections 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 --2- 172.21.15.28:0/3794614253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c0b3160 0x7f5d8c0ce560 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 --2- 172.21.15.28:0/3794614253 >> v2:172.21.15.189:3300/0 conn(0x7f5d8c0cdfa0 0x7f5d8c15d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 -- 172.21.15.28:0/3794614253 >> 172.21.15.28:0/3794614253 conn(0x7f5d8c069e80 msgr2=0x7f5d8c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 -- 172.21.15.28:0/3794614253 shutdown_connections 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.991+0000 7f5d91a00640 1 -- 172.21.15.28:0/3794614253 wait complete. 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.993+0000 7f5d91a00640 1 Processor -- start 2024-09-13T17:17:48.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.993+0000 7f5d91a00640 1 -- start start 2024-09-13T17:17:48.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 0x7f5d8c1f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 0x7f5d8c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f28b0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f2b90 con 0x7f5d8c0cdfa0 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d91a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8c1f2e70 con 0x7f5d8c06f440 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d82000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d82000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45650/0 (socket says 172.21.15.28:45650) 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d82000640 1 -- 172.21.15.28:0/2002038416 learned_addr learned my addr 172.21.15.28:0/2002038416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d8b400640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 0x7f5d8c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.994+0000 7f5d8aa00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 0x7f5d8c1f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 -- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 msgr2=0x7f5d8c1f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 0x7f5d8c1f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 -- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 msgr2=0x7f5d8c157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 0x7f5d8c157aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 -- 172.21.15.28:0/2002038416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d74009400 con 0x7f5d8c0b3160 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d8aa00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 0x7f5d8c1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:48.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d8b400640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 0x7f5d8c157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d82000640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5d74035040 tx=0x7f5d740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74006e10 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d74032e90 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d74030be0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8c1f34a0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.995+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8c158ca0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.996+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c15d340 con 0x7f5d8c0b3160 2024-09-13T17:17:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.997+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5d74039040 con 0x7f5d8c0b3160 2024-09-13T17:17:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.998+0000 7f5d83400640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 0x7f5d6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.998+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d740a9cd0 con 0x7f5d8c0b3160 2024-09-13T17:17:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.999+0000 7f5d8aa00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 0x7f5d6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:49.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:48.999+0000 7f5d8aa00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 0x7f5d6806d0c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c007520 tx=0x7f5d7c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:49.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.003+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5d74074f30 con 0x7f5d8c0b3160 2024-09-13T17:17:49.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.248+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5d8c0cf280 con 0x7f5d8c0b3160 2024-09-13T17:17:49.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.248+0000 7f5d83400640 1 -- 172.21.15.28:0/2002038416 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1027 (secure 0 0 0) 0x7f5d74030d40 con 0x7f5d8c0b3160 2024-09-13T17:17:49.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:49.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":139,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076474378,"bytes_used":2640166912,"bytes_avail":770453946368,"bytes_total":773094113280,"write_bytes_sec":72847653,"read_op_per_sec":0,"write_op_per_sec":70},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-09-13T17:16:17.009496+0000","services":{}},"progress_events":{}} 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 msgr2=0x7f5d6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 0x7f5d6806d0c0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5d7c007520 tx=0x7f5d7c00b040 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 msgr2=0x7f5d8c150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f5d74035040 tx=0x7f5d740046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 shutdown_connections 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5d6806ac00 0x7f5d6806d0c0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d8c0cdfa0 0x7f5d8c157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d8c0b3160 0x7f5d8c150210 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 --2- 172.21.15.28:0/2002038416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5d8c06f440 0x7f5d8c1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:49.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.250+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 >> 172.21.15.28:0/2002038416 conn(0x7f5d8c069e80 msgr2=0x7f5d8c15af60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.251+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 shutdown_connections 2024-09-13T17:17:49.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:49.251+0000 7f5d91a00640 1 -- 172.21.15.28:0/2002038416 wait complete. 2024-09-13T17:17:50.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.434+0000 7fd6eb400640 1 Processor -- start 2024-09-13T17:17:50.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.434+0000 7fd6eb400640 1 -- start start 2024-09-13T17:17:50.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6eb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6e40cabc0 0x7fd6e406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6eb400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e415e100 con 0x7fd6e4070cc0 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6eb400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6e415e3e0 con 0x7fd6e415a540 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6eb400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e40adb00 con 0x7fd6e40cabc0 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6e3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6e40cabc0 0x7fd6e406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6e3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd6e40cabc0 0x7fd6e406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45664/0 (socket says 172.21.15.28:45664) 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6e9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6e4070cc0 0x7fd6e406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35160/0 (socket says 172.21.15.28:35160) 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.435+0000 7fd6e3e00640 1 -- 172.21.15.28:0/3330951879 learned_addr learned my addr 172.21.15.28:0/3330951879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.436+0000 7fd6e3e00640 1 -- 172.21.15.28:0/3330951879 >> v1:172.21.15.189:6789/0 conn(0x7fd6e415a540 legacy=0x7fd6e415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:50.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.436+0000 7fd6e3e00640 1 -- 172.21.15.28:0/3330951879 >> v1:172.21.15.28:6789/0 conn(0x7fd6e4070cc0 legacy=0x7fd6e406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.436+0000 7fd6e3e00640 1 -- 172.21.15.28:0/3330951879 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e415ef80 con 0x7fd6e40cabc0 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.437+0000 7fd6e3e00640 1 --2- 172.21.15.28:0/3330951879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e40cabc0 0x7fd6e406c430 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009870 tx=0x7fd6d002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b6fc3b63f756476 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.437+0000 7fd6e3400640 1 -- 172.21.15.28:0/3330951879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0032440 con 0x7fd6e40cabc0 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.437+0000 7fd6e3400640 1 -- 172.21.15.28:0/3330951879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d00325a0 con 0x7fd6e40cabc0 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.437+0000 7fd6e3400640 1 -- 172.21.15.28:0/3330951879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d0032930 con 0x7fd6e40cabc0 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 -- 172.21.15.28:0/3330951879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e40cabc0 msgr2=0x7fd6e406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3330951879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e40cabc0 0x7fd6e406c430 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0009870 tx=0x7fd6d002a210 comp rx=0 tx=0).stop 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 -- 172.21.15.28:0/3330951879 shutdown_connections 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3330951879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e40cabc0 0x7fd6e406c430 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 -- 172.21.15.28:0/3330951879 >> 172.21.15.28:0/3330951879 conn(0x7fd6e4069e80 msgr2=0x7fd6e406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 -- 172.21.15.28:0/3330951879 shutdown_connections 2024-09-13T17:17:50.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.438+0000 7fd6eb400640 1 -- 172.21.15.28:0/3330951879 wait complete. 2024-09-13T17:17:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.439+0000 7fd6eb400640 1 Processor -- start 2024-09-13T17:17:50.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.440+0000 7fd6eb400640 1 -- start start 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.440+0000 7fd6eb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6eb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e40cabc0 0x7fd6e4145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6eb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 0x7fd6e415b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6eb400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4150140 con 0x7fd6e4070cc0 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6eb400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e4145580 con 0x7fd6e415a540 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6eb400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b1710 con 0x7fd6e40cabc0 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45670/0 (socket says 172.21.15.28:45670) 2024-09-13T17:17:50.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 -- 172.21.15.28:0/3561088764 learned_addr learned my addr 172.21.15.28:0/3561088764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e9400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 0x7fd6e415b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e3e00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e40cabc0 0x7fd6e4145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 -- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e40cabc0 msgr2=0x7fd6e4145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e40cabc0 0x7fd6e4145040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 -- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 msgr2=0x7fd6e415b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 0x7fd6e415b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e8a00640 1 -- 172.21.15.28:0/3561088764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d00094d0 con 0x7fd6e4070cc0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.441+0000 7fd6e9400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 0x7fd6e415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6e8a00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b3f0 tx=0x7fd6d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8011830 con 0x7fd6e4070cc0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d8011990 con 0x7fd6e4070cc0 2024-09-13T17:17:50.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8011d40 con 0x7fd6e4070cc0 2024-09-13T17:17:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e40b2570 con 0x7fd6e4070cc0 2024-09-13T17:17:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.442+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e40af110 con 0x7fd6e4070cc0 2024-09-13T17:17:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.443+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e406c430 con 0x7fd6e4070cc0 2024-09-13T17:17:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.444+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6d801a400 con 0x7fd6e4070cc0 2024-09-13T17:17:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.445+0000 7fd6e0c00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 0x7fd6c406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.445+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6d80892c0 con 0x7fd6e4070cc0 2024-09-13T17:17:50.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.445+0000 7fd6e3e00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 0x7fd6c406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.446+0000 7fd6e3e00640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 0x7fd6c406d0f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0034000 tx=0x7fd6d002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.449+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6d8054420 con 0x7fd6e4070cc0 2024-09-13T17:17:50.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.661+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6e415e910 con 0x7fd6e4070cc0 2024-09-13T17:17:50.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.662+0000 7fd6e0c00640 1 -- 172.21.15.28:0/3561088764 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fd6d8057a40 con 0x7fd6e4070cc0 2024-09-13T17:17:50.663 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:50.663 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:50.663 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:50.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 msgr2=0x7fd6c406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 0x7fd6c406d0f0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd6d0034000 tx=0x7fd6d002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 msgr2=0x7fd6e414fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fd6d800b3f0 tx=0x7fd6d800b8b0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 shutdown_connections 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6c406ac30 0x7fd6c406d0f0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6e415a540 0x7fd6e415b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6e40cabc0 0x7fd6e4145040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 --2- 172.21.15.28:0/3561088764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6e4070cc0 0x7fd6e414fc00 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 >> 172.21.15.28:0/3561088764 conn(0x7fd6e4069e80 msgr2=0x7fd6e40cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 shutdown_connections 2024-09-13T17:17:50.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.664+0000 7fd6eb400640 1 -- 172.21.15.28:0/3561088764 wait complete. 2024-09-13T17:17:50.680 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:50.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.833+0000 7f7f28a00640 1 Processor -- start 2024-09-13T17:17:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.833+0000 7f7f28a00640 1 -- start start 2024-09-13T17:17:50.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f28a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f2415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f28a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7f2406c2d0 0x7f7f2406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f28a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f24159d90 con 0x7f7f241592a0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f28a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2415b0b0 con 0x7f7f2406c2d0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f28a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2406cca0 con 0x7f7f2406f970 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f21e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f241592a0 0x7f7f24159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35162/0 (socket says 172.21.15.28:35162) 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f21e00640 1 -- 172.21.15.28:0/2866276522 learned_addr learned my addr 172.21.15.28:0/2866276522 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f22800640 1 --2- 172.21.15.28:0/2866276522 >> v2:172.21.15.189:3300/0 conn(0x7f7f2406c2d0 0x7f7f2406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f19400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f2415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696964276 0 0) 0x7f7f24159d90 con 0x7f7f241592a0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.835+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f08005730 con 0x7f7f241592a0 2024-09-13T17:17:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2161251060 0 0) 0x7f7f08005730 con 0x7f7f241592a0 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 msgr2=0x7f7f2415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f2415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 >> v2:172.21.15.189:3300/0 conn(0x7f7f2406c2d0 msgr2=0x7f7f2406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 --2- 172.21.15.28:0/2866276522 >> v2:172.21.15.189:3300/0 conn(0x7f7f2406c2d0 0x7f7f2406c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240b2040 con 0x7f7f241592a0 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7f14001690 con 0x7f7f241592a0 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 >> v1:172.21.15.28:6789/0 conn(0x7f7f241592a0 legacy=0x7f7f24159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f08005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f080069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f08006f10 0x7f7f08007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2415b0b0 con 0x7f7f08006f10 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2406cca0 con 0x7f7f08005b80 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0800db90 con 0x7f7f2406f970 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f22800640 1 --2- 172.21.15.28:0/2866276522 >> v2:172.21.15.189:3300/0 conn(0x7f7f2406c2d0 0x7f7f2406c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f21e00640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f08005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f19400640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f080069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.836+0000 7f7f22800640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f08006f10 0x7f7f08007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.837+0000 7f7f21e00640 1 -- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 msgr2=0x7f7f080069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.837+0000 7f7f21e00640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f2406f970 0x7f7f080069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.837+0000 7f7f21e00640 1 -- 172.21.15.28:0/2866276522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f08006f10 msgr2=0x7f7f08007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.837+0000 7f7f21e00640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f08006f10 0x7f7f08007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.837+0000 7f7f21e00640 1 -- 172.21.15.28:0/2866276522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08005730 con 0x7f7f08005b80 2024-09-13T17:17:50.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.838+0000 7f7f21e00640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f08005fa0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c240 tx=0x7f7f08007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fecbc178eaeaaf78 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.838+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14005ad0 con 0x7f7f08005b80 2024-09-13T17:17:50.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.838+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14005c30 con 0x7f7f08005b80 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.838+0000 7f7f21400640 1 -- 172.21.15.28:0/2866276522 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f14007070 con 0x7f7f08005b80 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 -- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 msgr2=0x7f7f08005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 --2- 172.21.15.28:0/2866276522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f08005fa0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f7f1400c240 tx=0x7f7f08007a00 comp rx=0 tx=0).stop 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f22800640 1 -- 172.21.15.28:0/2866276522 reap_dead start 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 -- 172.21.15.28:0/2866276522 shutdown_connections 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 -- 172.21.15.28:0/2866276522 >> 172.21.15.28:0/2866276522 conn(0x7f7f24069e80 msgr2=0x7f7f2406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 -- 172.21.15.28:0/2866276522 shutdown_connections 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.839+0000 7f7f28a00640 1 -- 172.21.15.28:0/2866276522 wait complete. 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.841+0000 7f7f28a00640 1 Processor -- start 2024-09-13T17:17:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.841+0000 7f7f28a00640 1 -- start start 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f240c7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 0x7f7f24155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f240c82d0 con 0x7f7f240cd4f0 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f240cdec0 con 0x7f7f08005b80 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f28a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f240c4cf0 con 0x7f7f240c9690 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45700/0 (socket says 172.21.15.28:45700) 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 -- 172.21.15.28:0/3519840117 learned_addr learned my addr 172.21.15.28:0/3519840117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f19400640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 0x7f7f24155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f21e00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f240c7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 -- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 msgr2=0x7f7f24155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 0x7f7f24155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 -- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 msgr2=0x7f7f240c7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f240c7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f22800640 1 -- 172.21.15.28:0/3519840117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240c9c50 con 0x7f7f240cd4f0 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f19400640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 0x7f7f24155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.842+0000 7f7f21e00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f240c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f22800640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7f2406f6d0 tx=0x7f7f1c00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c00e040 con 0x7f7f240cd4f0 2024-09-13T17:17:50.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c00f040 con 0x7f7f240cd4f0 2024-09-13T17:17:50.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1c015670 con 0x7f7f240cd4f0 2024-09-13T17:17:50.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f240c9f30 con 0x7f7f240cd4f0 2024-09-13T17:17:50.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.843+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f240ca4f0 con 0x7f7f240cd4f0 2024-09-13T17:17:50.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.844+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f240708e0 con 0x7f7f240cd4f0 2024-09-13T17:17:50.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.845+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7f1c0157d0 con 0x7f7f240cd4f0 2024-09-13T17:17:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.846+0000 7f7f1aa00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 0x7f7f0006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.846+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f1c089b70 con 0x7f7f240cd4f0 2024-09-13T17:17:50.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.847+0000 7f7f21e00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 0x7f7f0006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.847+0000 7f7f21e00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 0x7f7f0006d0f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7f14033000 tx=0x7f7f14035040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:50.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:50.851+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7f1c054d40 con 0x7f7f240cd4f0 2024-09-13T17:17:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.095+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7f240c4e60 con 0x7f7f240cd4f0 2024-09-13T17:17:51.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.096+0000 7f7f1aa00640 1 -- 172.21.15.28:0/3519840117 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1027 (secure 0 0 0) 0x7f7f1c058360 con 0x7f7f240cd4f0 2024-09-13T17:17:51.097 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:51.097 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":604,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":139,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076474378,"bytes_used":2640166912,"bytes_avail":770453946368,"bytes_total":773094113280,"write_bytes_sec":72847653,"read_op_per_sec":0,"write_op_per_sec":70},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-09-13T17:16:17.009496+0000","services":{}},"progress_events":{}} 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.097+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 msgr2=0x7f7f0006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.097+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 0x7f7f0006d0f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7f14033000 tx=0x7f7f14035040 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 msgr2=0x7f7f240cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f7f2406f6d0 tx=0x7f7f1c00f580 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 shutdown_connections 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f0006ac30 0x7f7f0006d0f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f240cd4f0 0x7f7f240cd980 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f240c9690 0x7f7f24155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 --2- 172.21.15.28:0/3519840117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08005b80 0x7f7f240c7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 >> 172.21.15.28:0/3519840117 conn(0x7f7f24069e80 msgr2=0x7f7f240ad850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:51.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 shutdown_connections 2024-09-13T17:17:51.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:51.098+0000 7f7f28a00640 1 -- 172.21.15.28:0/3519840117 wait complete. 2024-09-13T17:17:52.115 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:52.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.280+0000 7fb32d800640 1 Processor -- start 2024-09-13T17:17:52.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.281+0000 7fb32d800640 1 -- start start 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb32d800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb328159820 0x7fb328164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb32d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3280c3850 con 0x7fb328159440 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb32d800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32815a930 con 0x7fb3280b0b50 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb32d800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb328159d20 con 0x7fb328159820 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb327e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb328159440 0x7fb3280b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35178/0 (socket says 172.21.15.28:35178) 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb327e00640 1 -- 172.21.15.28:0/1352778865 learned_addr learned my addr 172.21.15.28:0/1352778865 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:52.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.282+0000 7fb32ce00640 1 --2- 172.21.15.28:0/1352778865 >> v2:172.21.15.189:3300/0 conn(0x7fb328159820 0x7fb328164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.283+0000 7fb327400640 1 -- 172.21.15.28:0/1352778865 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750248370 0 0) 0x7fb3280c3850 con 0x7fb328159440 2024-09-13T17:17:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.283+0000 7fb327400640 1 -- 172.21.15.28:0/1352778865 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c003520 con 0x7fb328159440 2024-09-13T17:17:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.283+0000 7fb32ce00640 1 -- 172.21.15.28:0/1352778865 >> v1:172.21.15.189:6789/0 conn(0x7fb3280b0b50 legacy=0x7fb3280b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.283+0000 7fb32ce00640 1 -- 172.21.15.28:0/1352778865 >> v1:172.21.15.28:6789/0 conn(0x7fb328159440 legacy=0x7fb3280b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.283+0000 7fb32ce00640 1 -- 172.21.15.28:0/1352778865 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32815a000 con 0x7fb328159820 2024-09-13T17:17:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.284+0000 7fb32ce00640 1 --2- 172.21.15.28:0/1352778865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb328159820 0x7fb328164260 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb320009d90 tx=0x7fb32002a780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=455efcb9a8c947e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.284+0000 7fb327400640 1 -- 172.21.15.28:0/1352778865 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320007740 con 0x7fb328159820 2024-09-13T17:17:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.284+0000 7fb327400640 1 -- 172.21.15.28:0/1352778865 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3200078a0 con 0x7fb328159820 2024-09-13T17:17:52.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.284+0000 7fb327400640 1 -- 172.21.15.28:0/1352778865 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320007c10 con 0x7fb328159820 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 -- 172.21.15.28:0/1352778865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb328159820 msgr2=0x7fb328164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 --2- 172.21.15.28:0/1352778865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb328159820 0x7fb328164260 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb320009d90 tx=0x7fb32002a780 comp rx=0 tx=0).stop 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 -- 172.21.15.28:0/1352778865 shutdown_connections 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 --2- 172.21.15.28:0/1352778865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb328159820 0x7fb328164260 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 -- 172.21.15.28:0/1352778865 >> 172.21.15.28:0/1352778865 conn(0x7fb328072e40 msgr2=0x7fb328073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 -- 172.21.15.28:0/1352778865 shutdown_connections 2024-09-13T17:17:52.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.285+0000 7fb32d800640 1 -- 172.21.15.28:0/1352778865 wait complete. 2024-09-13T17:17:52.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.287+0000 7fb32d800640 1 Processor -- start 2024-09-13T17:17:52.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.287+0000 7fb32d800640 1 -- start start 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3280b0b50 0x7fb3281494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 0x7fb3280c6940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3281f1bf0 con 0x7fb328159820 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3281572c0 con 0x7fb3280b0b50 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3281575a0 con 0x7fb328159440 2024-09-13T17:17:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45716/0 (socket says 172.21.15.28:45716) 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb32ce00640 1 -- 172.21.15.28:0/3989421400 learned_addr learned my addr 172.21.15.28:0/3989421400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb327e00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3280b0b50 0x7fb3281494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.288+0000 7fb31f400640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 0x7fb3280c6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 -- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 msgr2=0x7fb3280c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 0x7fb3280c6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 -- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3280b0b50 msgr2=0x7fb3281494a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3280b0b50 0x7fb3281494a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 -- 172.21.15.28:0/3989421400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320009a70 con 0x7fb328159820 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb31f400640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 0x7fb3280c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32ce00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb320009d90 tx=0x7fb320004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320032c80 con 0x7fb328159820 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb320032de0 con 0x7fb328159820 2024-09-13T17:17:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320003b90 con 0x7fb328159820 2024-09-13T17:17:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb328157880 con 0x7fb328159820 2024-09-13T17:17:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.289+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb328157e40 con 0x7fb328159820 2024-09-13T17:17:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.290+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb328073710 con 0x7fb328159820 2024-09-13T17:17:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.291+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb320003cf0 con 0x7fb328159820 2024-09-13T17:17:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.292+0000 7fb324c00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 0x7fb30406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.292+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb3200aa3e0 con 0x7fb328159820 2024-09-13T17:17:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.293+0000 7fb327e00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 0x7fb30406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.293+0000 7fb327e00640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 0x7fb30406d0c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb3180011e0 tx=0x7fb318007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.297+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb320075620 con 0x7fb328159820 2024-09-13T17:17:52.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.514+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3280b22e0 con 0x7fb328159820 2024-09-13T17:17:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.515+0000 7fb324c00640 1 -- 172.21.15.28:0/3989421400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fb320009450 con 0x7fb328159820 2024-09-13T17:17:52.516 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:52.516 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:52.516 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 msgr2=0x7fb30406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 0x7fb30406d0c0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fb3180011e0 tx=0x7fb318007d30 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 msgr2=0x7fb3281568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb320009d90 tx=0x7fb320004a40 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 shutdown_connections 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb30406ac00 0x7fb30406d0c0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb328159820 0x7fb3281568e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb328159440 0x7fb3280c6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 --2- 172.21.15.28:0/3989421400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3280b0b50 0x7fb3281494a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.517+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 >> 172.21.15.28:0/3989421400 conn(0x7fb328072e40 msgr2=0x7fb32815d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.518+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 shutdown_connections 2024-09-13T17:17:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.518+0000 7fb32d800640 1 -- 172.21.15.28:0/3989421400 wait complete. 2024-09-13T17:17:52.535 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:52.535 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:52.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.688+0000 7fbf4d000640 1 Processor -- start 2024-09-13T17:17:52.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.689+0000 7fbf4d000640 1 -- start start 2024-09-13T17:17:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf4d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf48154a90 0x7fbf4815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf4d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf480b0b50 0x7fbf480af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf4d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf4815bb10 con 0x7fbf4806b360 2024-09-13T17:17:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf4d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4806b900 con 0x7fbf480b0b50 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf4d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf4806bbe0 con 0x7fbf48154a90 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf47e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf480b0b50 0x7fbf480af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf47e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbf480b0b50 0x7fbf480af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45732/0 (socket says 172.21.15.28:45732) 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf47e00640 1 -- 172.21.15.28:0/3360734225 learned_addr learned my addr 172.21.15.28:0/3360734225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.690+0000 7fbf47400640 1 --2- 172.21.15.28:0/3360734225 >> v2:172.21.15.189:3300/0 conn(0x7fbf48154a90 0x7fbf4815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.691+0000 7fbf47e00640 1 -- 172.21.15.28:0/3360734225 >> v2:172.21.15.189:3300/0 conn(0x7fbf48154a90 msgr2=0x7fbf4815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.691+0000 7fbf47e00640 1 --2- 172.21.15.28:0/3360734225 >> v2:172.21.15.189:3300/0 conn(0x7fbf48154a90 0x7fbf4815b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.691+0000 7fbf47e00640 1 -- 172.21.15.28:0/3360734225 >> v1:172.21.15.189:6789/0 conn(0x7fbf4806b360 legacy=0x7fbf480b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.691+0000 7fbf47e00640 1 -- 172.21.15.28:0/3360734225 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4806bec0 con 0x7fbf480b0b50 2024-09-13T17:17:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf47e00640 1 --2- 172.21.15.28:0/3360734225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf480b0b50 0x7fbf480af4b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fbf480b3d70 tx=0x7fbf4002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e1a83c6453540c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf46a00640 1 -- 172.21.15.28:0/3360734225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40031600 con 0x7fbf480b0b50 2024-09-13T17:17:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf46a00640 1 -- 172.21.15.28:0/3360734225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4002d050 con 0x7fbf480b0b50 2024-09-13T17:17:52.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf46a00640 1 -- 172.21.15.28:0/3360734225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf40031980 con 0x7fbf480b0b50 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 -- 172.21.15.28:0/3360734225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf480b0b50 msgr2=0x7fbf480af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 --2- 172.21.15.28:0/3360734225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf480b0b50 0x7fbf480af4b0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fbf480b3d70 tx=0x7fbf4002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 -- 172.21.15.28:0/3360734225 shutdown_connections 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 --2- 172.21.15.28:0/3360734225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf480b0b50 0x7fbf480af4b0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 --2- 172.21.15.28:0/3360734225 >> v2:172.21.15.189:3300/0 conn(0x7fbf48154a90 0x7fbf4815b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 -- 172.21.15.28:0/3360734225 >> 172.21.15.28:0/3360734225 conn(0x7fbf48072e40 msgr2=0x7fbf48073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.692+0000 7fbf4d000640 1 -- 172.21.15.28:0/3360734225 shutdown_connections 2024-09-13T17:17:52.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.693+0000 7fbf4d000640 1 -- 172.21.15.28:0/3360734225 wait complete. 2024-09-13T17:17:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.694+0000 7fbf4d000640 1 Processor -- start 2024-09-13T17:17:52.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.694+0000 7fbf4d000640 1 -- start start 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf480b0b50 0x7fbf481e2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf48154a90 0x7fbf480add90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf481e40d0 con 0x7fbf4806b360 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf481e34c0 con 0x7fbf480b0b50 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf4d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf481e37a0 con 0x7fbf48154a90 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45740/0 (socket says 172.21.15.28:45740) 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.695+0000 7fbf47400640 1 -- 172.21.15.28:0/1866888977 learned_addr learned my addr 172.21.15.28:0/1866888977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47e00640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf48154a90 0x7fbf480add90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf3ea00640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf480b0b50 0x7fbf481e2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 -- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf48154a90 msgr2=0x7fbf480add90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf48154a90 0x7fbf480add90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 -- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf480b0b50 msgr2=0x7fbf481e2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf480b0b50 0x7fbf481e2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 -- 172.21.15.28:0/1866888977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf40009b30 con 0x7fbf4806b360 2024-09-13T17:17:52.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf47400640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800d5f0 tx=0x7fbf3800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.696+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380125f0 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.697+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf38012750 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.697+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38012b00 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.697+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf481e3a80 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.697+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf480ae2d0 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.698+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf480af4b0 con 0x7fbf4806b360 2024-09-13T17:17:52.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.699+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf38012c60 con 0x7fbf4806b360 2024-09-13T17:17:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.699+0000 7fbf3fe00640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 0x7fbf2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:52.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.700+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf38089780 con 0x7fbf4806b360 2024-09-13T17:17:52.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.700+0000 7fbf3ea00640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 0x7fbf2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:52.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.701+0000 7fbf3ea00640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 0x7fbf2406d0c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbf481547f0 tx=0x7fbf30007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:52.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.704+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf38054930 con 0x7fbf4806b360 2024-09-13T17:17:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.947+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fbf4815b080 con 0x7fbf4806b360 2024-09-13T17:17:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.947+0000 7fbf3fe00640 1 -- 172.21.15.28:0/1866888977 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1027 (secure 0 0 0) 0x7fbf38057f50 con 0x7fbf4806b360 2024-09-13T17:17:52.948 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:52.949 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":606,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":140,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076474378,"bytes_used":2640166912,"bytes_avail":770453946368,"bytes_total":773094113280,"write_bytes_sec":47894101,"read_op_per_sec":0,"write_op_per_sec":46},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-09-13T17:16:17.009496+0000","services":{}},"progress_events":{}} 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.949+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 msgr2=0x7fbf2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.949+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 0x7fbf2406d0c0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fbf481547f0 tx=0x7fbf30007e30 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.949+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 msgr2=0x7fbf48157c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.949+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800d5f0 tx=0x7fbf3800dab0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 shutdown_connections 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf2406ac00 0x7fbf2406d0c0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf48154a90 0x7fbf480add90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf480b0b50 0x7fbf481e2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 --2- 172.21.15.28:0/1866888977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf4806b360 0x7fbf48157c50 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 >> 172.21.15.28:0/1866888977 conn(0x7fbf48072e40 msgr2=0x7fbf48070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 shutdown_connections 2024-09-13T17:17:52.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:52.950+0000 7fbf4d000640 1 -- 172.21.15.28:0/1866888977 wait complete. 2024-09-13T17:17:53.967 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.130+0000 7fa72e600640 1 Processor -- start 2024-09-13T17:17:54.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.130+0000 7fa72e600640 1 -- start start 2024-09-13T17:17:54.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.131+0000 7fa72e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 0x7fa72815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.131+0000 7fa72e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.131+0000 7fa72e600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7280b2040 con 0x7fa72806c2d0 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.131+0000 7fa72e600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72815b0b0 con 0x7fa7281592a0 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.131+0000 7fa72e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa728159ff0 con 0x7fa72806f970 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa727400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 0x7fa72815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:51098/0 (socket says 172.21.15.28:51098) 2024-09-13T17:17:54.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 -- 172.21.15.28:0/1097830534 learned_addr learned my addr 172.21.15.28:0/1097830534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 -- 172.21.15.28:0/1097830534 >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 msgr2=0x7fa72815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 --2- 172.21.15.28:0/1097830534 >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 0x7fa72815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 -- 172.21.15.28:0/1097830534 >> v1:172.21.15.189:6789/0 conn(0x7fa72806c2d0 legacy=0x7fa7280b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa71ea00640 1 -- 172.21.15.28:0/1097830534 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa72815a2d0 con 0x7fa72806f970 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.132+0000 7fa727400640 1 --2- 172.21.15.28:0/1097830534 >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 0x7fa72815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:54.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.133+0000 7fa71ea00640 1 --2- 172.21.15.28:0/1097830534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54f011aae670fa27 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.134+0000 7fa726a00640 1 -- 172.21.15.28:0/1097830534 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710032440 con 0x7fa72806f970 2024-09-13T17:17:54.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.134+0000 7fa726a00640 1 -- 172.21.15.28:0/1097830534 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7100325a0 con 0x7fa72806f970 2024-09-13T17:17:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.134+0000 7fa726a00640 1 -- 172.21.15.28:0/1097830534 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710032930 con 0x7fa72806f970 2024-09-13T17:17:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 -- 172.21.15.28:0/1097830534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 msgr2=0x7fa728159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 --2- 172.21.15.28:0/1097830534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa710009870 tx=0x7fa71002a250 comp rx=0 tx=0).stop 2024-09-13T17:17:54.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 -- 172.21.15.28:0/1097830534 shutdown_connections 2024-09-13T17:17:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 --2- 172.21.15.28:0/1097830534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa728159680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 --2- 172.21.15.28:0/1097830534 >> v2:172.21.15.189:3300/0 conn(0x7fa7281592a0 0x7fa72815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 -- 172.21.15.28:0/1097830534 >> 172.21.15.28:0/1097830534 conn(0x7fa728069e80 msgr2=0x7fa72806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 -- 172.21.15.28:0/1097830534 shutdown_connections 2024-09-13T17:17:54.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.135+0000 7fa72e600640 1 -- 172.21.15.28:0/1097830534 wait complete. 2024-09-13T17:17:54.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.137+0000 7fa72e600640 1 Processor -- start 2024-09-13T17:17:54.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.137+0000 7fa72e600640 1 -- start start 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa7280b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 0x7fa7280b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa728164080 con 0x7fa72806c2d0 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72814e020 con 0x7fa7281592a0 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa72e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7280b0f50 con 0x7fa72806f970 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa727400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa727400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33250/0 (socket says 172.21.15.28:33250) 2024-09-13T17:17:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa727400640 1 -- 172.21.15.28:0/3081435285 learned_addr learned my addr 172.21.15.28:0/3081435285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa71ea00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa7280b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa727400640 1 -- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 msgr2=0x7fa7280b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.138+0000 7fa727e00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 0x7fa7280b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727400640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa7280b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727400640 1 -- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 msgr2=0x7fa7280b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727400640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 0x7fa7280b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727400640 1 -- 172.21.15.28:0/3081435285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa718009400 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa71ea00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa7280b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727e00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 0x7fa7280b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa727400640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e960 tx=0x7fa71800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71800ca00 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa71800cb60 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa718013470 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7100094d0 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.139+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72814d220 con 0x7fa72806c2d0 2024-09-13T17:17:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.140+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72815ab70 con 0x7fa72806c2d0 2024-09-13T17:17:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.141+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa7180135d0 con 0x7fa72806c2d0 2024-09-13T17:17:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.142+0000 7fa71fe00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 0x7fa70406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.142+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa7180896b0 con 0x7fa72806c2d0 2024-09-13T17:17:54.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.143+0000 7fa71ea00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 0x7fa70406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.143+0000 7fa71ea00640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 0x7fa70406d0c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa710034000 tx=0x7fa71002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.146+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa7180547b0 con 0x7fa72806c2d0 2024-09-13T17:17:54.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.365+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa728067040 con 0x7fa72806c2d0 2024-09-13T17:17:54.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.366+0000 7fa71fe00640 1 -- 172.21.15.28:0/3081435285 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fa718057dd0 con 0x7fa72806c2d0 2024-09-13T17:17:54.367 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:54.368 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:54.368 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:54.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 msgr2=0x7fa70406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 0x7fa70406d0c0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fa710034000 tx=0x7fa71002e000 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 msgr2=0x7fa72814dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fa71800e960 tx=0x7fa71800ee20 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 shutdown_connections 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa70406ac00 0x7fa70406d0c0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7281592a0 0x7fa7280b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa72806f970 0x7fa7280b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 --2- 172.21.15.28:0/3081435285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa72806c2d0 0x7fa72814dae0 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.368+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 >> 172.21.15.28:0/3081435285 conn(0x7fa728069e80 msgr2=0x7fa7280ad620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.369+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 shutdown_connections 2024-09-13T17:17:54.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.369+0000 7fa72e600640 1 -- 172.21.15.28:0/3081435285 wait complete. 2024-09-13T17:17:54.385 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:54.385 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.539+0000 7f4fe7200640 1 Processor -- start 2024-09-13T17:17:54.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.539+0000 7f4fe7200640 1 -- start start 2024-09-13T17:17:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe7200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fe0156a20 0x7f4fe00b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe015d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe006bde0 con 0x7f4fe0156e00 2024-09-13T17:17:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe7200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe006b820 con 0x7f4fe0156a20 2024-09-13T17:17:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe006bb00 con 0x7f4fe015d370 2024-09-13T17:17:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe5400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4fe0156e00 0x7f4fe0071dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36224/0 (socket says 172.21.15.28:36224) 2024-09-13T17:17:54.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.540+0000 7f4fe5400640 1 -- 172.21.15.28:0/1836319514 learned_addr learned my addr 172.21.15.28:0/1836319514 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe5e00640 1 --2- 172.21.15.28:0/1836319514 >> v2:172.21.15.189:3300/0 conn(0x7f4fe0156a20 0x7f4fe00b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe6800640 1 --2- 172.21.15.28:0/1836319514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe015d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624686210 0 0) 0x7f4fe006bde0 con 0x7f4fe0156e00 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc8003520 con 0x7f4fe0156e00 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1654423274 0 0) 0x7f4fc8003520 con 0x7f4fe0156e00 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe6800640 1 -- 172.21.15.28:0/1836319514 >> v2:172.21.15.189:3300/0 conn(0x7f4fe0156a20 msgr2=0x7f4fe00b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe6800640 1 --2- 172.21.15.28:0/1836319514 >> v2:172.21.15.189:3300/0 conn(0x7f4fe0156a20 0x7f4fe00b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe6800640 1 -- 172.21.15.28:0/1836319514 >> v1:172.21.15.28:6789/0 conn(0x7f4fe0156e00 legacy=0x7f4fe0071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.541+0000 7f4fe6800640 1 -- 172.21.15.28:0/1836319514 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe015ce50 con 0x7f4fe015d370 2024-09-13T17:17:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.542+0000 7f4fe6800640 1 --2- 172.21.15.28:0/1836319514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe015d7d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009d90 tx=0x7f4fdc02a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a366139277a4cba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0001670 con 0x7f4fe015d370 2024-09-13T17:17:54.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc02d050 con 0x7f4fe015d370 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe4a00640 1 -- 172.21.15.28:0/1836319514 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc007760 con 0x7f4fe015d370 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 -- 172.21.15.28:0/1836319514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 msgr2=0x7f4fe015d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 --2- 172.21.15.28:0/1836319514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe015d7d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009d90 tx=0x7f4fdc02a5f0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 -- 172.21.15.28:0/1836319514 shutdown_connections 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 --2- 172.21.15.28:0/1836319514 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe015d7d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 --2- 172.21.15.28:0/1836319514 >> v2:172.21.15.189:3300/0 conn(0x7f4fe0156a20 0x7f4fe00b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 -- 172.21.15.28:0/1836319514 >> 172.21.15.28:0/1836319514 conn(0x7f4fe006d4e0 msgr2=0x7f4fe006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:54.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 -- 172.21.15.28:0/1836319514 shutdown_connections 2024-09-13T17:17:54.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.543+0000 7f4fe7200640 1 -- 172.21.15.28:0/1836319514 wait complete. 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.545+0000 7f4fe7200640 1 Processor -- start 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.545+0000 7f4fe7200640 1 -- start start 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe0156a20 0x7f4fe01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe00b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe015ca50 con 0x7f4fe0156e00 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe015a950 con 0x7f4fe0156a20 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe015ac30 con 0x7f4fe015d370 2024-09-13T17:17:54.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33266/0 (socket says 172.21.15.28:33266) 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.546+0000 7f4fe5400640 1 -- 172.21.15.28:0/3736105240 learned_addr learned my addr 172.21.15.28:0/3736105240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe6800640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe00b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5e00640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe0156a20 0x7f4fe01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 -- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 msgr2=0x7f4fe00b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe00b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 -- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe0156a20 msgr2=0x7f4fe01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe0156a20 0x7f4fe01593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 -- 172.21.15.28:0/3736105240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc009a70 con 0x7f4fe0156e00 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe5400640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00011e0 tx=0x7f4fd000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0005ad0 con 0x7f4fe0156e00 2024-09-13T17:17:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd0005c50 con 0x7f4fe0156e00 2024-09-13T17:17:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0007070 con 0x7f4fe0156e00 2024-09-13T17:17:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.547+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe015af10 con 0x7f4fe0156e00 2024-09-13T17:17:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.548+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe015b710 con 0x7f4fe0156e00 2024-09-13T17:17:54.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.549+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe015d7d0 con 0x7f4fe0156e00 2024-09-13T17:17:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.549+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4fd00071d0 con 0x7f4fe0156e00 2024-09-13T17:17:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.550+0000 7f4fda000640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 0x7f4fc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.550+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4fd0088b50 con 0x7f4fe0156e00 2024-09-13T17:17:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.551+0000 7f4fe5e00640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 0x7f4fc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.551+0000 7f4fe5e00640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 0x7f4fc406d0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00c9ca0 tx=0x7f4fd4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.555+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4fd0053c80 con 0x7f4fe0156e00 2024-09-13T17:17:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.805+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f4fe00720b0 con 0x7f4fe0156e00 2024-09-13T17:17:54.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.805+0000 7f4fda000640 1 -- 172.21.15.28:0/3736105240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1675 (secure 0 0 0) 0x7f4fd00572a0 con 0x7f4fe0156e00 2024-09-13T17:17:54.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:54.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":608,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":141,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":95},{"state_name":"peering","count":2}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076576690,"bytes_used":2685558784,"bytes_avail":770408554496,"bytes_total":773094113280,"inactive_pgs_ratio":0.020618556067347527,"write_bytes_sec":28278064,"read_op_per_sec":0,"write_op_per_sec":27},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-09-13T17:17:53.032878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 msgr2=0x7f4fc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 0x7f4fc406d0c0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00c9ca0 tx=0x7f4fd4009190 comp rx=0 tx=0).stop 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 msgr2=0x7f4fe00b0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f4fd00011e0 tx=0x7f4fd000cc00 comp rx=0 tx=0).stop 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 shutdown_connections 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fc406ac00 0x7f4fc406d0c0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fe015d370 0x7f4fe00b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fe0156e00 0x7f4fe00b0e10 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 --2- 172.21.15.28:0/3736105240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe0156a20 0x7f4fe01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 >> 172.21.15.28:0/3736105240 conn(0x7f4fe006d4e0 msgr2=0x7f4fe00b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 shutdown_connections 2024-09-13T17:17:54.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.808+0000 7f4fe7200640 1 -- 172.21.15.28:0/3736105240 wait complete. 2024-09-13T17:17:54.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:17:54.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.9:0 scrub resume 2024-09-13T17:17:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.980+0000 7f290c000640 1 Processor -- start 2024-09-13T17:17:54.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.980+0000 7f290c000640 1 -- start start 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.981+0000 7f290c000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 0x7f290406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.981+0000 7f290c000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f290406f3f0 0x7f2904070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f290c000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290406f920 con 0x7f29040add00 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f290c000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f290415b290 con 0x7f290406f3f0 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f290c000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290415b570 con 0x7f2904159770 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f290a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29040add00 0x7f2904159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36236/0 (socket says 172.21.15.28:36236) 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f290a000640 1 -- 172.21.15.28:0/945883555 learned_addr learned my addr 172.21.15.28:0/945883555 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2908c00640 1 --2- 172.21.15.28:0/945883555 >> v2:172.21.15.189:3300/0 conn(0x7f290406f3f0 0x7f2904070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2909600640 1 --2- 172.21.15.28:0/945883555 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 0x7f290406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2903e00640 1 -- 172.21.15.28:0/945883555 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007852280 0 0) 0x7f290406f920 con 0x7f29040add00 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2903e00640 1 -- 172.21.15.28:0/945883555 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ec003520 con 0x7f29040add00 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2909600640 1 -- 172.21.15.28:0/945883555 >> v2:172.21.15.189:3300/0 conn(0x7f290406f3f0 msgr2=0x7f2904070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2909600640 1 --2- 172.21.15.28:0/945883555 >> v2:172.21.15.189:3300/0 conn(0x7f290406f3f0 0x7f2904070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.982+0000 7f2909600640 1 -- 172.21.15.28:0/945883555 >> v1:172.21.15.28:6789/0 conn(0x7f29040add00 legacy=0x7f2904159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.983+0000 7f2909600640 1 -- 172.21.15.28:0/945883555 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290415b850 con 0x7f2904159770 2024-09-13T17:17:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f2909600640 1 --2- 172.21.15.28:0/945883555 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 0x7f290406fd20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f28f80097a0 tx=0x7f28f802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9febddd16cb2c2a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f2903e00640 1 -- 172.21.15.28:0/945883555 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f802ae50 con 0x7f2904159770 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f2903e00640 1 -- 172.21.15.28:0/945883555 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f8032440 con 0x7f2904159770 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f2903e00640 1 -- 172.21.15.28:0/945883555 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f80327d0 con 0x7f2904159770 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f290c000640 1 -- 172.21.15.28:0/945883555 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 msgr2=0x7f290406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.984+0000 7f290c000640 1 --2- 172.21.15.28:0/945883555 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 0x7f290406fd20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f28f80097a0 tx=0x7f28f802a100 comp rx=0 tx=0).stop 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 -- 172.21.15.28:0/945883555 shutdown_connections 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 --2- 172.21.15.28:0/945883555 >> v2:172.21.15.189:3300/0 conn(0x7f290406f3f0 0x7f2904070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 --2- 172.21.15.28:0/945883555 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2904159770 0x7f290406fd20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 -- 172.21.15.28:0/945883555 >> 172.21.15.28:0/945883555 conn(0x7f29040b0270 msgr2=0x7f29040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 -- 172.21.15.28:0/945883555 shutdown_connections 2024-09-13T17:17:54.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.985+0000 7f290c000640 1 -- 172.21.15.28:0/945883555 wait complete. 2024-09-13T17:17:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.986+0000 7f290c000640 1 Processor -- start 2024-09-13T17:17:54.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.986+0000 7f290c000640 1 -- start start 2024-09-13T17:17:54.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29040add00 0x7f29040ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2904159770 0x7f29040c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040a8ef0 con 0x7f290406f3f0 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29040a9190 con 0x7f2904159770 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f290c000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29040a9470 con 0x7f29040add00 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33282/0 (socket says 172.21.15.28:33282) 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.987+0000 7f2909600640 1 -- 172.21.15.28:0/3104009696 learned_addr learned my addr 172.21.15.28:0/3104009696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2908c00640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29040add00 0x7f29040ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f290a000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2904159770 0x7f29040c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 -- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29040add00 msgr2=0x7f29040ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29040add00 0x7f29040ca260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 -- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2904159770 msgr2=0x7f29040c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2904159770 0x7f29040c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 -- 172.21.15.28:0/3104009696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8009400 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2909600640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f28f8035040 tx=0x7f28f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2901600640 1 -- 172.21.15.28:0/3104009696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f803d850 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2901600640 1 -- 172.21.15.28:0/3104009696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f803d9d0 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f2901600640 1 -- 172.21.15.28:0/3104009696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f803dd80 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.988+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29040a9750 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.989+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290414f610 con 0x7f290406f3f0 2024-09-13T17:17:54.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.990+0000 7f2901600640 1 -- 172.21.15.28:0/3104009696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28f8039040 con 0x7f290406f3f0 2024-09-13T17:17:54.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.991+0000 7f2901600640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 0x7f28e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.991+0000 7f2901600640 1 -- 172.21.15.28:0/3104009696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28f80a9dc0 con 0x7f290406f3f0 2024-09-13T17:17:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.992+0000 7f2908c00640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 0x7f28e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.992+0000 7f290ee27740 1 Processor -- start 2024-09-13T17:17:54.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.992+0000 7f290ee27740 1 -- start start 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.992+0000 7f2908c00640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 0x7f28e406d0c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f29040a8e20 tx=0x7f28f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76d8450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e7725a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e75da6a0 con 0x55b9e76d5830 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e763dcc0 con 0x55b9e76d8070 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9e763b450 con 0x55b9e76ff520 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33294/0 (socket says 172.21.15.28:33294) 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 -- 172.21.15.28:0/107393908 learned_addr learned my addr 172.21.15.28:0/107393908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:54.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f290a000640 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e7725a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2909600640 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76d8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 -- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 msgr2=0x55b9e7725a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e7725a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 -- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 msgr2=0x55b9e76d8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.993+0000 7f2908c00640 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76d8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.994+0000 7f2908c00640 1 -- 172.21.15.28:0/107393908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9e76072e0 con 0x55b9e76d5830 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.994+0000 7f2908c00640 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f28f0011cb0 tx=0x7f28f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bbffb56a31e4565 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.994+0000 7f28f6000640 1 -- 172.21.15.28:0/107393908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011ce0 con 0x55b9e76d5830 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.994+0000 7f28f6000640 1 -- 172.21.15.28:0/107393908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0011e40 con 0x55b9e76d5830 2024-09-13T17:17:54.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.994+0000 7f28f6000640 1 -- 172.21.15.28:0/107393908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0021640 con 0x55b9e76d5830 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 -- 172.21.15.28:0/107393908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 msgr2=0x55b9e7725540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f28f0011cb0 tx=0x7f28f0014000 comp rx=0 tx=0).stop 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 -- 172.21.15.28:0/107393908 shutdown_connections 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e7725a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d5830 0x55b9e7725540 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 --2- 172.21.15.28:0/107393908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76d8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 -- 172.21.15.28:0/107393908 >> 172.21.15.28:0/107393908 conn(0x55b9e76d47a0 msgr2=0x55b9e771cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 -- 172.21.15.28:0/107393908 shutdown_connections 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.995+0000 7f290ee27740 1 -- 172.21.15.28:0/107393908 wait complete. 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.996+0000 7f290ee27740 1 -- start start 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.996+0000 7f290ee27740 1 Processor -- start 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e7700820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e7636d40 con 0x55b9e76ff520 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e75d8280 con 0x55b9e76d8070 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9e75d7f20 con 0x55b9e76fc730 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290ee27740 10 client.0 _ll_register_callbacks cb 0x55b9e76303b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f2908c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f2908c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33310/0 (socket says 172.21.15.28:33310) 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f2908c00640 1 -- 172.21.15.28:0/4128519763 learned_addr learned my addr 172.21.15.28:0/4128519763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f2909600640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.997+0000 7f290a000640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e7700820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 -- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 msgr2=0x55b9e7700820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e7700820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 -- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 msgr2=0x55b9e76fc1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76fc1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 -- 172.21.15.28:0/4128519763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f0011990 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f2908c00640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f28f0022000 tx=0x7f28f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bc7d28d2882db7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f28eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f0011ce0 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9e732a4e0 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28f0031070 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28f0021d40 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.998+0000 7f290ee27740 10 client.9159 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x55b9e76353c0 con 0x55b9e76ff520 2024-09-13T17:17:55.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b9e732a4e0 con 0x55b9e76ff520 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f28f0036480 con 0x55b9e76ff520 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 10 client.9159 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 10 client.9159 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 20 client.9159 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e7700d60 0x55b9e76f7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 4 client.9159 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9e733b7e0 con 0x55b9e7700d60 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:54.999+0000 7f2909600640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e7700d60 0x55b9e76f7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f2909600640 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e7700d60 0x55b9e76f7d00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b44677992b94a355 server_cookie=1b77239167a41a62 in_seq=0 out_seq=0 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f28eaa00640 10 client.9159 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f28eaa00640 1 -- 172.21.15.28:0/4128519763 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55b9e733b7e0 con 0x55b9e7700d60 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f28eaa00640 10 client.9159 handle_command_reply: tid=0 2024-09-13T17:17:55.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f290ee27740 1 client.9159 shutdown 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 msgr2=0x55b9e77002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f28f0022000 tx=0x7f28f00046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 shutdown_connections 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.000+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e7700d60 0x55b9e76f7d00 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e7700820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76ff520 0x55b9e77002e0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f28eaa00640 0 client.9159 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 --2- 172.21.15.28:0/4128519763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76d8070 0x55b9e76fc1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 >> 172.21.15.28:0/4128519763 conn(0x55b9e76d5830 msgr2=0x55b9e771ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 shutdown_connections 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 1 -- 172.21.15.28:0/4128519763 wait complete. 2024-09-13T17:17:55.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.001+0000 7f290ee27740 20 client.9159 trim_cache size 0 max 16384 2024-09-13T17:17:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.015+0000 7f290ee27740 1 Processor -- start 2024-09-13T17:17:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 -- start start 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76fc730 0x55b9e7701560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e76cd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e739e040 con 0x55b9e76d8070 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e76cdb50 con 0x55b9e76fc730 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9e771bb20 con 0x55b9e76ff520 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.016+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33322/0 (socket says 172.21.15.28:33322) 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 -- 172.21.15.28:0/855812339 learned_addr learned my addr 172.21.15.28:0/855812339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2908c00640 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76fc730 0x55b9e7701560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 -- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 msgr2=0x55b9e76cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e76cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 -- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76fc730 msgr2=0x55b9e7701560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76fc730 0x55b9e7701560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 -- 172.21.15.28:0/855812339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9e771eb50 con 0x55b9e76d8070 2024-09-13T17:17:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f2909600640 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006f80 tx=0x7f28f80a80e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba10363c94736f08 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f28f6a00640 1 -- 172.21.15.28:0/855812339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8080f70 con 0x55b9e76d8070 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f28f6a00640 1 -- 172.21.15.28:0/855812339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f80a8320 con 0x55b9e76d8070 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.017+0000 7f28f6a00640 1 -- 172.21.15.28:0/855812339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f804dde0 con 0x55b9e76d8070 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 -- 172.21.15.28:0/855812339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 msgr2=0x55b9e7701d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f28f8006f80 tx=0x7f28f80a80e0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 -- 172.21.15.28:0/855812339 shutdown_connections 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76ff520 0x55b9e76cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76fc730 0x55b9e7701560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 --2- 172.21.15.28:0/855812339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e7701d70 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 -- 172.21.15.28:0/855812339 >> 172.21.15.28:0/855812339 conn(0x55b9e76d5830 msgr2=0x55b9e76f7e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 -- 172.21.15.28:0/855812339 shutdown_connections 2024-09-13T17:17:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.018+0000 7f290ee27740 1 -- 172.21.15.28:0/855812339 wait complete. 2024-09-13T17:17:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.019+0000 7f290ee27740 1 -- start start 2024-09-13T17:17:55.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.019+0000 7f290ee27740 1 Processor -- start 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e76fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 0x55b9e771aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e76fed00 con 0x55b9e76d8070 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9e771ccd0 con 0x55b9e76ff520 2024-09-13T17:17:55.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b9e776fd80 con 0x55b9e76fc730 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290ee27740 10 client.0 _ll_register_callbacks cb 0x7f28f0011bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33324/0 (socket says 172.21.15.28:33324) 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 -- 172.21.15.28:0/2031063128 learned_addr learned my addr 172.21.15.28:0/2031063128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f290a000640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 0x55b9e771aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2908c00640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e76fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 -- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 msgr2=0x55b9e76fd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e76fd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 -- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 msgr2=0x55b9e771aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.020+0000 7f2909600640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 0x55b9e771aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f2909600640 1 -- 172.21.15.28:0/2031063128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290414f610 con 0x55b9e76d8070 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f290a000640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 0x55b9e771aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f2908c00640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e76fd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:55.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f2909600640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f28f8078b10 tx=0x7f28f803f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f006c77ada22abc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f28eaa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f804dde0 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f28f8075290 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28f0011e40 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28f8075450 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f290ee27740 10 client.9165 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x7f2904150700 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.021+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f0011e40 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f28f80a1070 con 0x55b9e76d8070 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 10 client.9165 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 10 client.9165 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 20 client.9165 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e776cf50 0x55b9e771b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 4 client.9165 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-09-13T17:17:55.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "scrub resume"}) v1 -- 0x55b9e738ee30 con 0x55b9e776cf50 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f2908c00640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e776cf50 0x55b9e771b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f2908c00640 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e776cf50 0x55b9e771b1e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83c0cd7bbee3eae3 server_cookie=828164016f95bcb2 in_seq=0 out_seq=0 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.022+0000 7f28eaa00640 10 client.9165 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f28eaa00640 1 -- 172.21.15.28:0/2031063128 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x55b9e738ee30 con 0x55b9e776cf50 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f28eaa00640 10 client.9165 handle_command_reply: tid=0 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 client.9165 shutdown 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 msgr2=0x55b9e771c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f28f8078b10 tx=0x7f28f803f040 comp rx=0 tx=0).stop 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 shutdown_connections 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x55b9e776cf50 0x55b9e771b1e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9e76ff520 0x55b9e771aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b9e76fc730 0x55b9e76fd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 --2- 172.21.15.28:0/2031063128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b9e76d8070 0x55b9e771c220 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f28eaa00640 0 client.9165 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 >> 172.21.15.28:0/2031063128 conn(0x55b9e76d5830 msgr2=0x55b9e771a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 shutdown_connections 2024-09-13T17:17:55.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 1 -- 172.21.15.28:0/2031063128 wait complete. 2024-09-13T17:17:55.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.023+0000 7f290ee27740 20 client.9165 trim_cache size 0 max 16384 2024-09-13T17:17:55.028 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:55.028 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:17:55.028 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 msgr2=0x7f28e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 0x7f28e406d0c0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f29040a8e20 tx=0x7f28f0008040 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 msgr2=0x7f290414f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7f28f8035040 tx=0x7f28f80046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 shutdown_connections 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28e406ac00 0x7f28e406d0c0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2904159770 0x7f29040c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29040add00 0x7f29040ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 --2- 172.21.15.28:0/3104009696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f290406f3f0 0x7f290414f0d0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 >> 172.21.15.28:0/3104009696 conn(0x7f29040b0270 msgr2=0x7f29040af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 shutdown_connections 2024-09-13T17:17:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.024+0000 7f290c000640 1 -- 172.21.15.28:0/3104009696 wait complete. 2024-09-13T17:17:55.040 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.9:0 scrub status 2024-09-13T17:17:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.197+0000 7fb8d2e00640 1 Processor -- start 2024-09-13T17:17:55.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.197+0000 7fb8d2e00640 1 -- start start 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.198+0000 7fb8d2e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb8cc143320 0x7fb8cc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.198+0000 7fb8d2e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8cc143910 0x7fb8cc071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.198+0000 7fb8d2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8cc071630 con 0x7fb8cc072000 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.198+0000 7fb8d2e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8cc143700 con 0x7fb8cc143910 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.198+0000 7fb8d2e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8cc070a20 con 0x7fb8cc143320 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8cc143910 0x7fb8cc071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb8cc143910 0x7fb8cc071ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33330/0 (socket says 172.21.15.28:33330) 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 -- 172.21.15.28:0/643131283 learned_addr learned my addr 172.21.15.28:0/643131283 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d2400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb8cc072000 0x7fb8cc0cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36250/0 (socket says 172.21.15.28:36250) 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/643131283 >> v2:172.21.15.189:3300/0 conn(0x7fb8cc143320 0x7fb8cc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 -- 172.21.15.28:0/643131283 >> v2:172.21.15.189:3300/0 conn(0x7fb8cc143320 msgr2=0x7fb8cc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 --2- 172.21.15.28:0/643131283 >> v2:172.21.15.189:3300/0 conn(0x7fb8cc143320 0x7fb8cc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 -- 172.21.15.28:0/643131283 >> v1:172.21.15.28:6789/0 conn(0x7fb8cc072000 legacy=0x7fb8cc0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.199+0000 7fb8d1000640 1 -- 172.21.15.28:0/643131283 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc070d00 con 0x7fb8cc143910 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.200+0000 7fb8d1000640 1 --2- 172.21.15.28:0/643131283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc143910 0x7fb8cc071ac0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc06c690 tx=0x7fb8b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e31c2a370fcb0db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.200+0000 7fb8cbe00640 1 -- 172.21.15.28:0/643131283 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8032440 con 0x7fb8cc143910 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.200+0000 7fb8cbe00640 1 -- 172.21.15.28:0/643131283 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b80325a0 con 0x7fb8cc143910 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.200+0000 7fb8cbe00640 1 -- 172.21.15.28:0/643131283 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8032930 con 0x7fb8cc143910 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 -- 172.21.15.28:0/643131283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc143910 msgr2=0x7fb8cc071ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/643131283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc143910 0x7fb8cc071ac0 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc06c690 tx=0x7fb8b802a270 comp rx=0 tx=0).stop 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 -- 172.21.15.28:0/643131283 shutdown_connections 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/643131283 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc143910 0x7fb8cc071ac0 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/643131283 >> v2:172.21.15.189:3300/0 conn(0x7fb8cc143320 0x7fb8cc070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 -- 172.21.15.28:0/643131283 >> 172.21.15.28:0/643131283 conn(0x7fb8cc069e80 msgr2=0x7fb8cc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 -- 172.21.15.28:0/643131283 shutdown_connections 2024-09-13T17:17:55.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.201+0000 7fb8d2e00640 1 -- 172.21.15.28:0/643131283 wait complete. 2024-09-13T17:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.203+0000 7fb8d2e00640 1 Processor -- start 2024-09-13T17:17:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.203+0000 7fb8d2e00640 1 -- start start 2024-09-13T17:17:55.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc143320 0x7fb8cc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 0x7fb8cc13e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0cc0f0 con 0x7fb8cc072000 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0cc3d0 con 0x7fb8cc143910 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8cc0cc6b0 con 0x7fb8cc143320 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33334/0 (socket says 172.21.15.28:33334) 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.204+0000 7fb8d1a00640 1 -- 172.21.15.28:0/3707513677 learned_addr learned my addr 172.21.15.28:0/3707513677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1000640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc143320 0x7fb8cc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d2400640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 0x7fb8cc13e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 -- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc143320 msgr2=0x7fb8cc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc143320 0x7fb8cc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 -- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 msgr2=0x7fb8cc13e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 0x7fb8cc13e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 -- 172.21.15.28:0/3707513677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80094d0 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d2400640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 0x7fb8cc13e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d800 tx=0x7fb8c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8c9600640 1 -- 172.21.15.28:0/3707513677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000b6c0 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8c9600640 1 -- 172.21.15.28:0/3707513677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c000b820 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.205+0000 7fb8c9600640 1 -- 172.21.15.28:0/3707513677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000bbb0 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.206+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8cc0cc990 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.206+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc1f3500 con 0x7fb8cc072000 2024-09-13T17:17:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.207+0000 7fb8c9600640 1 -- 172.21.15.28:0/3707513677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb8c000bd10 con 0x7fb8cc072000 2024-09-13T17:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.208+0000 7fb8c9600640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 0x7fb8ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.208+0000 7fb8c9600640 1 -- 172.21.15.28:0/3707513677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8c0089890 con 0x7fb8cc072000 2024-09-13T17:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.209+0000 7fb8d1000640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 0x7fb8ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.209+0000 7fb8d73ec740 1 Processor -- start 2024-09-13T17:17:55.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.209+0000 7fb8d73ec740 1 -- start start 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d1000640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 0x7fb8ac06d0c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8034000 tx=0x7fb8b802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 0x563447e391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 0x563447e862a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e60280 0x563447e867e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447d3f6a0 con 0x563447e36590 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447da2cc0 con 0x563447e60280 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563447da0450 con 0x563447e38dd0 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.210+0000 7fb8d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 0x563447e862a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 0x563447e862a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33342/0 (socket says 172.21.15.28:33342) 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 0x563447e391b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1000640 1 -- 172.21.15.28:0/1415143963 learned_addr learned my addr 172.21.15.28:0/1415143963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d2400640 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e60280 0x563447e867e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 -- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e60280 msgr2=0x563447e867e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e60280 0x563447e867e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 -- 172.21.15.28:0/1415143963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 msgr2=0x563447e862a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 0x563447e862a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 -- 172.21.15.28:0/1415143963 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563447d6c2e0 con 0x563447e38dd0 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.211+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 0x563447e391b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d7d0 tx=0x7fb8c0058170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=672732225ffd95e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8be000640 1 -- 172.21.15.28:0/1415143963 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0060ad0 con 0x563447e38dd0 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8be000640 1 -- 172.21.15.28:0/1415143963 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c0089050 con 0x563447e38dd0 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8be000640 1 -- 172.21.15.28:0/1415143963 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0054df0 con 0x563447e38dd0 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 -- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 msgr2=0x563447e391b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 0x563447e391b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d7d0 tx=0x7fb8c0058170 comp rx=0 tx=0).stop 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 -- 172.21.15.28:0/1415143963 shutdown_connections 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e60280 0x563447e867e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e36590 0x563447e862a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/1415143963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e38dd0 0x563447e391b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 -- 172.21.15.28:0/1415143963 >> 172.21.15.28:0/1415143963 conn(0x563447e35500 msgr2=0x563447e7d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.212+0000 7fb8d73ec740 1 -- 172.21.15.28:0/1415143963 shutdown_connections 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.213+0000 7fb8d73ec740 1 -- 172.21.15.28:0/1415143963 wait complete. 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.213+0000 7fb8d73ec740 1 -- start start 2024-09-13T17:17:55.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.213+0000 7fb8d73ec740 1 Processor -- start 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e82e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e84750 0x563447e7bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447d9bd40 con 0x563447e60280 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447d3d280 con 0x563447e38dd0 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563447d3cf20 con 0x563447e84750 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d73ec740 10 client.0 _ll_register_callbacks cb 0x563447d953b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.214+0000 7fb8d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33350/0 (socket says 172.21.15.28:33350) 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 -- 172.21.15.28:0/627284251 learned_addr learned my addr 172.21.15.28:0/627284251 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e82e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d2400640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e84750 0x563447e7bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 -- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e84750 msgr2=0x563447e7bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e84750 0x563447e7bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 -- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 msgr2=0x563447e82e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e82e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 -- 172.21.15.28:0/627284251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8cc1f3500 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8d1000640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb8b802a860 tx=0x7fb8b8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cca958dbd17a6b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.215+0000 7fb8b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b8054680 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563447a8f4e0 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8b804f070 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8b80547e0 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 10 client.9180 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x563447d9a3c0 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563447a8f4e0 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb8b8044040 con 0x563447e60280 2024-09-13T17:17:55.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 10 client.9180 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 10 client.9180 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 20 client.9180 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e86230 0x563447e86610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 4 client.9180 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.216+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563447aa07e0 con 0x563447e86230 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.217+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e86230 0x563447e86610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.217+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e86230 0x563447e86610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d547290310b7c32 server_cookie=a0b49e07e11636b3 in_seq=0 out_seq=0 2024-09-13T17:17:55.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.217+0000 7fb8b2a00640 10 client.9180 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8b2a00640 1 -- 172.21.15.28:0/627284251 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x563447aa07e0 con 0x563447e86230 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8b2a00640 10 client.9180 handle_command_reply: tid=0 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 client.9180 shutdown 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 msgr2=0x563447e5f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb8b802a860 tx=0x7fb8b8004730 comp rx=0 tx=0).stop 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 shutdown_connections 2024-09-13T17:17:55.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e86230 0x563447e86610 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e84750 0x563447e7bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e60280 0x563447e5f090 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/627284251 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e82e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 >> 172.21.15.28:0/627284251 conn(0x563447e36590 msgr2=0x563447e816c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8b2a00640 0 client.9180 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 shutdown_connections 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 1 -- 172.21.15.28:0/627284251 wait complete. 2024-09-13T17:17:55.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.218+0000 7fb8d73ec740 20 client.9180 trim_cache size 0 max 16384 2024-09-13T17:17:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.233+0000 7fb8d73ec740 1 Processor -- start 2024-09-13T17:17:55.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.233+0000 7fb8d73ec740 1 -- start start 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e8fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e7fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447b03040 con 0x563447e84750 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447e34f30 con 0x563447e38dd0 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563447e38860 con 0x563447e60280 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33362/0 (socket says 172.21.15.28:33362) 2024-09-13T17:17:55.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.234+0000 7fb8d2400640 1 -- 172.21.15.28:0/3928004802 learned_addr learned my addr 172.21.15.28:0/3928004802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e8fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d1000640 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e7fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 -- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 msgr2=0x563447e7fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e7fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 -- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 msgr2=0x563447e8fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e8fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 -- 172.21.15.28:0/3928004802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563447d3c610 con 0x563447e84750 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8d2400640 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40076c0 tx=0x7fb8c4007b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca5269c4a80a45e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8bea00640 1 -- 172.21.15.28:0/3928004802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4014c90 con 0x563447e84750 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8bea00640 1 -- 172.21.15.28:0/3928004802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c4014df0 con 0x563447e84750 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.235+0000 7fb8bea00640 1 -- 172.21.15.28:0/3928004802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4016620 con 0x563447e84750 2024-09-13T17:17:55.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 -- 172.21.15.28:0/3928004802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 msgr2=0x563447e7a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40076c0 tx=0x7fb8c4007b80 comp rx=0 tx=0).stop 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 -- 172.21.15.28:0/3928004802 shutdown_connections 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e7a070 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e7fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/3928004802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447e8fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 -- 172.21.15.28:0/3928004802 >> 172.21.15.28:0/3928004802 conn(0x563447e36590 msgr2=0x563447e7a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 -- 172.21.15.28:0/3928004802 shutdown_connections 2024-09-13T17:17:55.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.236+0000 7fb8d73ec740 1 -- 172.21.15.28:0/3928004802 wait complete. 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.237+0000 7fb8d73ec740 1 -- start start 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.237+0000 7fb8d73ec740 1 Processor -- start 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447ee8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e81760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447e5da60 con 0x563447e84750 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563447ee8c00 con 0x563447e38dd0 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563447ee8ee0 con 0x563447e60280 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d73ec740 10 client.0 _ll_register_callbacks cb 0x7fb8b80096f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33370/0 (socket says 172.21.15.28:33370) 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 -- 172.21.15.28:0/2599679507 learned_addr learned my addr 172.21.15.28:0/2599679507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d1000640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e81760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447ee8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 -- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 msgr2=0x563447e81760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e81760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 -- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 msgr2=0x563447ee8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447ee8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.238+0000 7fb8d2400640 1 -- 172.21.15.28:0/2599679507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c400f040 con 0x563447e84750 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8d2400640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40012e0 tx=0x7fb8c4001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e93b5daecc9b4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8b2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4014c90 con 0x563447e84750 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb8c4016e10 con 0x563447e84750 2024-09-13T17:17:55.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb8c0089050 con 0x563447e84750 2024-09-13T17:17:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=122) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb8c4028440 con 0x563447e84750 2024-09-13T17:17:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8d73ec740 10 client.9186 fetch_fsmap learned FSMap version 122 2024-09-13T17:17:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.239+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=122,monmap=2+}) v3 -- 0x563447d61b20 con 0x563447e84750 2024-09-13T17:17:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8c0089050 con 0x563447e84750 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 122) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb8c401c020 con 0x563447e84750 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 10 client.9186 fetch_fsmap finished waiting for FSMap version 122 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 10 client.9186 resolve_mds: resolved 9:0 to role '9:0' aka daemon mds.c 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 20 client.9186 populate_metadata read hostname 'smithi028' 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e62800 0x563447e5c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 4 client.9186 mds_command: new command op to 8904 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 --> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563447af3e30 con 0x563447e62800 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e62800 0x563447e5c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8d1a00640 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e62800 0x563447e5c1a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5576718cd0595805 server_cookie=96f0438194012e4d in_seq=0 out_seq=0 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.240+0000 7fb8b2a00640 10 client.9186 ms_handle_connect on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8b2a00640 1 -- 172.21.15.28:0/2599679507 <== mds.0 v2:172.21.15.28:6834/4121307324 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563447af3e30 con 0x563447e62800 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8b2a00640 10 client.9186 handle_command_reply: tid=0 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 client.9186 shutdown 2024-09-13T17:17:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 msgr2=0x563447e81ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7fb8c40012e0 tx=0x7fb8c4001250 comp rx=0 tx=0).stop 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 shutdown_connections 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:6834/4121307324,v1:172.21.15.28:6835/4121307324] conn(0x563447e62800 0x563447e5c1a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563447e84750 0x563447e81ca0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563447e60280 0x563447e81760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 --2- 172.21.15.28:0/2599679507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563447e38dd0 0x563447ee8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 >> 172.21.15.28:0/2599679507 conn(0x563447e36590 msgr2=0x563447e829f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8b2a00640 0 client.9186 ms_handle_reset on v2:172.21.15.28:6834/4121307324 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 shutdown_connections 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 1 -- 172.21.15.28:0/2599679507 wait complete. 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.241+0000 7fb8d73ec740 20 client.9186 trim_cache size 0 max 16384 2024-09-13T17:17:55.245 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:17:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:17:55.246 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:17:55.246 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:17:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 msgr2=0x7fb8ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 0x7fb8ac06d0c0 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8034000 tx=0x7fb8b802e000 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 msgr2=0x7fb8cc1f2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d800 tx=0x7fb8c000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 shutdown_connections 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8ac06ac00 0x7fb8ac06d0c0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8cc143910 0x7fb8cc13e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8cc143320 0x7fb8cc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 --2- 172.21.15.28:0/3707513677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8cc072000 0x7fb8cc1f2fc0 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 >> 172.21.15.28:0/3707513677 conn(0x7fb8cc069e80 msgr2=0x7fb8cc0b3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 shutdown_connections 2024-09-13T17:17:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.242+0000 7fb8d2e00640 1 -- 172.21.15.28:0/3707513677 wait complete. 2024-09-13T17:17:55.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.411+0000 7f1cd8c00640 1 Processor -- start 2024-09-13T17:17:55.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.411+0000 7f1cd8c00640 1 -- start start 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd41592a0 0x7f1cd415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd8c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd4159d90 con 0x7f1cd406f970 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd8c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40b2040 con 0x7f1cd406c2d0 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd415b0b0 con 0x7f1cd41592a0 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd41592a0 0x7f1cd415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1cd41592a0 0x7f1cd415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33380/0 (socket says 172.21.15.28:33380) 2024-09-13T17:17:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 -- 172.21.15.28:0/3238791700 learned_addr learned my addr 172.21.15.28:0/3238791700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 -- 172.21.15.28:0/3238791700 >> v1:172.21.15.189:6789/0 conn(0x7f1cd406c2d0 legacy=0x7f1cd40b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 -- 172.21.15.28:0/3238791700 >> v1:172.21.15.28:6789/0 conn(0x7f1cd406f970 legacy=0x7f1cd4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1e00640 1 -- 172.21.15.28:0/3238791700 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4159ff0 con 0x7f1cd41592a0 2024-09-13T17:17:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.413+0000 7f1cd1400640 1 -- 172.21.15.28:0/3238791700 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920084464 0 0) 0x7f1cd4159d90 con 0x7f1cd406f970 2024-09-13T17:17:55.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.414+0000 7f1cd1e00640 1 --2- 172.21.15.28:0/3238791700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd41592a0 0x7f1cd415ab70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009870 tx=0x7f1cc402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713a893aaf2656cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd1400640 1 -- 172.21.15.28:0/3238791700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4032640 con 0x7f1cd41592a0 2024-09-13T17:17:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd1400640 1 -- 172.21.15.28:0/3238791700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc40327a0 con 0x7f1cd41592a0 2024-09-13T17:17:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd1400640 1 -- 172.21.15.28:0/3238791700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc4032b30 con 0x7f1cd41592a0 2024-09-13T17:17:55.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 -- 172.21.15.28:0/3238791700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd41592a0 msgr2=0x7f1cd415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/3238791700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd41592a0 0x7f1cd415ab70 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4009870 tx=0x7f1cc402a300 comp rx=0 tx=0).stop 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 -- 172.21.15.28:0/3238791700 shutdown_connections 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/3238791700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd41592a0 0x7f1cd415ab70 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 -- 172.21.15.28:0/3238791700 >> 172.21.15.28:0/3238791700 conn(0x7f1cd4069e80 msgr2=0x7f1cd406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 -- 172.21.15.28:0/3238791700 shutdown_connections 2024-09-13T17:17:55.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.415+0000 7f1cd8c00640 1 -- 172.21.15.28:0/3238791700 wait complete. 2024-09-13T17:17:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.417+0000 7f1cd8c00640 1 Processor -- start 2024-09-13T17:17:55.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.417+0000 7f1cd8c00640 1 -- start start 2024-09-13T17:17:55.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cd406c2d0 0x7f1cd4150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd41592a0 0x7f1cd40acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41f2f80 con 0x7f1cd406f970 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41f23b0 con 0x7f1cd41592a0 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41f2620 con 0x7f1cd406c2d0 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cc9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cc9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33386/0 (socket says 172.21.15.28:33386) 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cc9400640 1 -- 172.21.15.28:0/2053545893 learned_addr learned my addr 172.21.15.28:0/2053545893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd1e00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cd406c2d0 0x7f1cd4150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.418+0000 7f1cd2800640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd41592a0 0x7f1cd40acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 -- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cd406c2d0 msgr2=0x7f1cd4150c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cd406c2d0 0x7f1cd4150c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 -- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd41592a0 msgr2=0x7f1cd40acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd41592a0 0x7f1cd40acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 -- 172.21.15.28:0/2053545893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40094d0 con 0x7f1cd406f970 2024-09-13T17:17:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cc9400640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc00c430 tx=0x7f1cbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0052d0 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc005430 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0037f0 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd4153f00 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.419+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd41f2ba0 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.421+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd41f30c0 con 0x7f1cd406f970 2024-09-13T17:17:55.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.421+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1cbc003950 con 0x7f1cd406f970 2024-09-13T17:17:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.422+0000 7f1ccaa00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 0x7f1cb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.422+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1cbc088930 con 0x7f1cd406f970 2024-09-13T17:17:55.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.423+0000 7f1cd1e00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 0x7f1cb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.423+0000 7f1cd1e00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 0x7f1cb006d0c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4034000 tx=0x7f1cc402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.426+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1cd41f30c0 con 0x7f1cd406f970 2024-09-13T17:17:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.640+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd41f30c0 con 0x7f1cd406f970 2024-09-13T17:17:55.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.640+0000 7f1ccaa00640 1 -- 172.21.15.28:0/2053545893 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f1cd41f30c0 con 0x7f1cd406f970 2024-09-13T17:17:55.642 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:55.642 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:55.642 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.642+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 msgr2=0x7f1cb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.642+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 0x7f1cb006d0c0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1cc4034000 tx=0x7f1cc402e000 comp rx=0 tx=0).stop 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.642+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 msgr2=0x7f1cd40ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.642+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc00c430 tx=0x7f1cbc00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 shutdown_connections 2024-09-13T17:17:55.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cb006ac00 0x7f1cb006d0c0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cd41592a0 0x7f1cd40acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1cd406f970 0x7f1cd40ac850 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 --2- 172.21.15.28:0/2053545893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1cd406c2d0 0x7f1cd4150c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 >> 172.21.15.28:0/2053545893 conn(0x7f1cd4069e80 msgr2=0x7f1cd40b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 shutdown_connections 2024-09-13T17:17:55.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.643+0000 7f1cd8c00640 1 -- 172.21.15.28:0/2053545893 wait complete. 2024-09-13T17:17:55.659 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:55.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.813+0000 7f9272200640 1 Processor -- start 2024-09-13T17:17:55.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.813+0000 7f9272200640 1 -- start start 2024-09-13T17:17:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9272200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f926c1592a0 0x7f926c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9272200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f926c06c2d0 0x7f926c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9272200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c06cc70 con 0x7f926c06f970 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9272200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c15b0b0 con 0x7f926c06c2d0 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9272200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c159d90 con 0x7f926c1592a0 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9263e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f926c06f970 0x7f926c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36272/0 (socket says 172.21.15.28:36272) 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.814+0000 7f9263e00640 1 -- 172.21.15.28:0/4185245489 learned_addr learned my addr 172.21.15.28:0/4185245489 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926be00640 1 --2- 172.21.15.28:0/4185245489 >> v2:172.21.15.28:3300/0 conn(0x7f926c06c2d0 0x7f926c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926b400640 1 --2- 172.21.15.28:0/4185245489 >> v2:172.21.15.189:3300/0 conn(0x7f926c1592a0 0x7f926c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926aa00640 1 -- 172.21.15.28:0/4185245489 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620871224 0 0) 0x7f926c06cc70 con 0x7f926c06f970 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926aa00640 1 -- 172.21.15.28:0/4185245489 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9250003520 con 0x7f926c06f970 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926be00640 1 -- 172.21.15.28:0/4185245489 >> v2:172.21.15.189:3300/0 conn(0x7f926c1592a0 msgr2=0x7f926c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926be00640 1 --2- 172.21.15.28:0/4185245489 >> v2:172.21.15.189:3300/0 conn(0x7f926c1592a0 0x7f926c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926be00640 1 -- 172.21.15.28:0/4185245489 >> v1:172.21.15.28:6789/0 conn(0x7f926c06f970 legacy=0x7f926c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.815+0000 7f926be00640 1 -- 172.21.15.28:0/4185245489 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0b2040 con 0x7f926c06c2d0 2024-09-13T17:17:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.816+0000 7f926be00640 1 --2- 172.21.15.28:0/4185245489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c06c730 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f926c06d370 tx=0x7f926402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d814bc69128d792 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.816+0000 7f926aa00640 1 -- 172.21.15.28:0/4185245489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926402aea0 con 0x7f926c06c2d0 2024-09-13T17:17:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.816+0000 7f926aa00640 1 -- 172.21.15.28:0/4185245489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926402d050 con 0x7f926c06c2d0 2024-09-13T17:17:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.816+0000 7f926aa00640 1 -- 172.21.15.28:0/4185245489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264007940 con 0x7f926c06c2d0 2024-09-13T17:17:55.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 -- 172.21.15.28:0/4185245489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 msgr2=0x7f926c06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 --2- 172.21.15.28:0/4185245489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c06c730 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f926c06d370 tx=0x7f926402a680 comp rx=0 tx=0).stop 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 -- 172.21.15.28:0/4185245489 shutdown_connections 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 --2- 172.21.15.28:0/4185245489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c06c730 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 --2- 172.21.15.28:0/4185245489 >> v2:172.21.15.189:3300/0 conn(0x7f926c1592a0 0x7f926c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 -- 172.21.15.28:0/4185245489 >> 172.21.15.28:0/4185245489 conn(0x7f926c069e80 msgr2=0x7f926c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 -- 172.21.15.28:0/4185245489 shutdown_connections 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.817+0000 7f9272200640 1 -- 172.21.15.28:0/4185245489 wait complete. 2024-09-13T17:17:55.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.819+0000 7f9272200640 1 Processor -- start 2024-09-13T17:17:55.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.819+0000 7f9272200640 1 -- start start 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f926c06f970 0x7f926c0af310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 0x7f926c0af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cdae0 con 0x7f926c06c2d0 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cced0 con 0x7f926c1592a0 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9272200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0cd1b0 con 0x7f926c06f970 2024-09-13T17:17:55.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33392/0 (socket says 172.21.15.28:33392) 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 -- 172.21.15.28:0/244578048 learned_addr learned my addr 172.21.15.28:0/244578048 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926be00640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 0x7f926c0af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f9263e00640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f926c06f970 0x7f926c0af310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 -- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f926c06f970 msgr2=0x7f926c0af310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f926c06f970 0x7f926c0af310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 -- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 msgr2=0x7f926c0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 0x7f926c0af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926b400640 1 -- 172.21.15.28:0/244578048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9264009a70 con 0x7f926c06c2d0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.820+0000 7f926be00640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 0x7f926c0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f926b400640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e870 tx=0x7f925c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c0094f0 con 0x7f926c06c2d0 2024-09-13T17:17:55.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925c00c940 con 0x7f926c06c2d0 2024-09-13T17:17:55.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925c00ccf0 con 0x7f926c06c2d0 2024-09-13T17:17:55.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c0cd490 con 0x7f926c06c2d0 2024-09-13T17:17:55.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.821+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c1484b0 con 0x7f926c06c2d0 2024-09-13T17:17:55.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.822+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c0b3740 con 0x7f926c06c2d0 2024-09-13T17:17:55.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.823+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f925c013470 con 0x7f926c06c2d0 2024-09-13T17:17:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.824+0000 7f9263400640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 0x7f924806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.824+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f925c089430 con 0x7f926c06c2d0 2024-09-13T17:17:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.824+0000 7f9263e00640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 0x7f924806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.825+0000 7f9263e00640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 0x7f924806d0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f926c06bac0 tx=0x7f9254009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:55.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:55.828+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f925c0545c0 con 0x7f926c06c2d0 2024-09-13T17:17:56.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.075+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f926c06bfb0 con 0x7f926c06c2d0 2024-09-13T17:17:56.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.075+0000 7f9263400640 1 -- 172.21.15.28:0/244578048 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1675 (secure 0 0 0) 0x7f925c057be0 con 0x7f926c06c2d0 2024-09-13T17:17:56.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:56.077 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":609,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":141,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":95},{"state_name":"peering","count":2}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076576690,"bytes_used":2685558784,"bytes_avail":770408554496,"bytes_total":773094113280,"inactive_pgs_ratio":0.020618556067347527,"write_bytes_sec":35346341,"read_op_per_sec":0,"write_op_per_sec":34},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-09-13T17:17:53.032878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 msgr2=0x7f924806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 0x7f924806d0f0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f926c06bac0 tx=0x7f9254009040 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 msgr2=0x7f926c147f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f925c00e870 tx=0x7f925c00ed30 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 shutdown_connections 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f924806ac30 0x7f924806d0f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f926c1592a0 0x7f926c0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f926c06f970 0x7f926c0af310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 --2- 172.21.15.28:0/244578048 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f926c06c2d0 0x7f926c147f70 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:56.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 >> 172.21.15.28:0/244578048 conn(0x7f926c069e80 msgr2=0x7f926c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 shutdown_connections 2024-09-13T17:17:56.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:56.078+0000 7f9272200640 1 -- 172.21.15.28:0/244578048 wait complete. 2024-09-13T17:17:56.094 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:17:57.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:57.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.250+0000 7f2b8e800640 1 Processor -- start 2024-09-13T17:17:57.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.250+0000 7f2b8e800640 1 -- start start 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.251+0000 7f2b8e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b8815a540 0x7f2b8806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b8e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8815e100 con 0x7f2b880cabc0 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b8e800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8815e3e0 con 0x7f2b88070cc0 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b8e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b880adb00 con 0x7f2b8815a540 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b8815a540 0x7f2b8806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b8815a540 0x7f2b8806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33404/0 (socket says 172.21.15.28:33404) 2024-09-13T17:17:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 -- 172.21.15.28:0/1226434429 learned_addr learned my addr 172.21.15.28:0/1226434429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 -- 172.21.15.28:0/1226434429 >> v1:172.21.15.189:6789/0 conn(0x7f2b88070cc0 legacy=0x7f2b8806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 -- 172.21.15.28:0/1226434429 >> v1:172.21.15.28:6789/0 conn(0x7f2b880cabc0 legacy=0x7f2b8815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.252+0000 7f2b87400640 1 -- 172.21.15.28:0/1226434429 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8815ef80 con 0x7f2b8815a540 2024-09-13T17:17:57.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.253+0000 7f2b87400640 1 --2- 172.21.15.28:0/1226434429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8815a540 0x7f2b8806c430 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009870 tx=0x7f2b7802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f26096357d568e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.253+0000 7f2b86000640 1 -- 172.21.15.28:0/1226434429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78032440 con 0x7f2b8815a540 2024-09-13T17:17:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b86000640 1 -- 172.21.15.28:0/1226434429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b780325a0 con 0x7f2b8815a540 2024-09-13T17:17:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b86000640 1 -- 172.21.15.28:0/1226434429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78032930 con 0x7f2b8815a540 2024-09-13T17:17:57.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 -- 172.21.15.28:0/1226434429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8815a540 msgr2=0x7f2b8806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 --2- 172.21.15.28:0/1226434429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8815a540 0x7f2b8806c430 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009870 tx=0x7f2b7802a270 comp rx=0 tx=0).stop 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 -- 172.21.15.28:0/1226434429 shutdown_connections 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 --2- 172.21.15.28:0/1226434429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b8815a540 0x7f2b8806c430 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 -- 172.21.15.28:0/1226434429 >> 172.21.15.28:0/1226434429 conn(0x7f2b88069e80 msgr2=0x7f2b8806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 -- 172.21.15.28:0/1226434429 shutdown_connections 2024-09-13T17:17:57.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.254+0000 7f2b8e800640 1 -- 172.21.15.28:0/1226434429 wait complete. 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.256+0000 7f2b8e800640 1 Processor -- start 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.256+0000 7f2b8e800640 1 -- start start 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b880cabc0 0x7f2b88139010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b8815a540 0x7f2b8815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880b0b40 con 0x7f2b880cabc0 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880b0170 con 0x7f2b88070cc0 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b8e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b880af5a0 con 0x7f2b8815a540 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b86a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b880cabc0 0x7f2b88139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b87400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b87400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:33374/0 (socket says 172.21.15.28:33374) 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b87400640 1 -- 172.21.15.28:0/2930060969 learned_addr learned my addr 172.21.15.28:0/2930060969 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.257+0000 7f2b87e00640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b8815a540 0x7f2b8815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 -- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b8815a540 msgr2=0x7f2b8815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b8815a540 0x7f2b8815d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 -- 172.21.15.28:0/2930060969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b880cabc0 msgr2=0x7f2b88139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b880cabc0 0x7f2b88139010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 -- 172.21.15.28:0/2930060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780094d0 con 0x7f2b88070cc0 2024-09-13T17:17:57.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b87400640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2b78000c00 tx=0x7f2b780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b78032440 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b78030ba0 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b7803b400 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.258+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b880af840 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.259+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b880afe00 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.260+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b880affc0 con 0x7f2b88070cc0 2024-09-13T17:17:57.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.260+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2b78039040 con 0x7f2b88070cc0 2024-09-13T17:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.261+0000 7f2b7f400640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 0x7f2b6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.261+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2b780a9c50 con 0x7f2b88070cc0 2024-09-13T17:17:57.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.262+0000 7f2b86a00640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 0x7f2b6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.262+0000 7f2b86a00640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 0x7f2b6806d0c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2b88142700 tx=0x7f2b74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.267+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2b78074e00 con 0x7f2b88070cc0 2024-09-13T17:17:57.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.480+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b8806c430 con 0x7f2b88070cc0 2024-09-13T17:17:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.481+0000 7f2b7f400640 1 -- 172.21.15.28:0/2930060969 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f2b78037090 con 0x7f2b88070cc0 2024-09-13T17:17:57.482 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:57.482 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:57.482 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 msgr2=0x7f2b6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 0x7f2b6806d0c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f2b88142700 tx=0x7f2b74009040 comp rx=0 tx=0).stop 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 msgr2=0x7f2b880b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f2b78000c00 tx=0x7f2b780046c0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 shutdown_connections 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b6806ac00 0x7f2b6806d0c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b8815a540 0x7f2b8815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b880cabc0 0x7f2b88139010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 --2- 172.21.15.28:0/2930060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b88070cc0 0x7f2b880b0600 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 >> 172.21.15.28:0/2930060969 conn(0x7f2b88069e80 msgr2=0x7f2b880711c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.483+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 shutdown_connections 2024-09-13T17:17:57.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.484+0000 7f2b8e800640 1 -- 172.21.15.28:0/2930060969 wait complete. 2024-09-13T17:17:57.500 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:57.500 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.654+0000 7f6409200640 1 Processor -- start 2024-09-13T17:17:57.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.654+0000 7f6409200640 1 -- start start 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6409200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64041592a0 0x7f640415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6409200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f640406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6409200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640406cc70 con 0x7f640406f970 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6409200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640415b0b0 con 0x7f64041592a0 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6409200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6404159d90 con 0x7f640406c2d0 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f63f9400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f640406f970 0x7f6404159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36296/0 (socket says 172.21.15.28:36296) 2024-09-13T17:17:57.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f63f9400640 1 -- 172.21.15.28:0/1960578087 learned_addr learned my addr 172.21.15.28:0/1960578087 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6401e00640 1 --2- 172.21.15.28:0/1960578087 >> v2:172.21.15.189:3300/0 conn(0x7f64041592a0 0x7f640415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6402800640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f640406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6401400640 1 -- 172.21.15.28:0/1960578087 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2501283992 0 0) 0x7f640406cc70 con 0x7f640406f970 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.656+0000 7f6401400640 1 -- 172.21.15.28:0/1960578087 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63e8005730 con 0x7f640406f970 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.657+0000 7f6401e00640 1 -- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 msgr2=0x7f640406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.657+0000 7f6401e00640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f640406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.657+0000 7f6401e00640 1 -- 172.21.15.28:0/1960578087 >> v1:172.21.15.28:6789/0 conn(0x7f640406f970 legacy=0x7f6404159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.657+0000 7f6401e00640 1 -- 172.21.15.28:0/1960578087 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64040b2040 con 0x7f64041592a0 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.657+0000 7f6402800640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f640406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.658+0000 7f6401e00640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f640415ab70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f63f40097a0 tx=0x7f63f402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386dd787050b1f34 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.658+0000 7f6401400640 1 -- 172.21.15.28:0/1960578087 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4032440 con 0x7f64041592a0 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.658+0000 7f6401400640 1 -- 172.21.15.28:0/1960578087 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f40325a0 con 0x7f64041592a0 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.658+0000 7f6401400640 1 -- 172.21.15.28:0/1960578087 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f4032930 con 0x7f64041592a0 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 -- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 msgr2=0x7f640415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f640415ab70 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f63f40097a0 tx=0x7f63f402a180 comp rx=0 tx=0).stop 2024-09-13T17:17:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 -- 172.21.15.28:0/1960578087 shutdown_connections 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f640406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 --2- 172.21.15.28:0/1960578087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f640415ab70 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 -- 172.21.15.28:0/1960578087 >> 172.21.15.28:0/1960578087 conn(0x7f6404069e80 msgr2=0x7f640406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 -- 172.21.15.28:0/1960578087 shutdown_connections 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.659+0000 7f6409200640 1 -- 172.21.15.28:0/1960578087 wait complete. 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.660+0000 7f6409200640 1 Processor -- start 2024-09-13T17:17:57.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 -- start start 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f6404151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f64040cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404151960 con 0x7f640406f970 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.661+0000 7f6409200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6404152370 con 0x7f64041592a0 2024-09-13T17:17:57.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f6409200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6404155730 con 0x7f640406c2d0 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33428/0 (socket says 172.21.15.28:33428) 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 -- 172.21.15.28:0/3446595809 learned_addr learned my addr 172.21.15.28:0/3446595809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f6401e00640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f6404151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f6402800640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f64040cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 -- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 msgr2=0x7f6404151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f6404151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 -- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 msgr2=0x7f64040cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f64040cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 -- 172.21.15.28:0/3446595809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc009a70 con 0x7f640406f970 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f6402800640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f64040cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:17:57.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.662+0000 7f63f9400640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f63ec001eb0 tx=0x7f63ec00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.663+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec005ce0 con 0x7f640406f970 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.663+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ec005e40 con 0x7f640406f970 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.663+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ec007270 con 0x7f640406f970 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.663+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f4009400 con 0x7f640406f970 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.663+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6404155cb0 con 0x7f640406f970 2024-09-13T17:17:57.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.664+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64040b3740 con 0x7f640406f970 2024-09-13T17:17:57.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.665+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f63ec0073d0 con 0x7f640406f970 2024-09-13T17:17:57.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.666+0000 7f63faa00640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 0x7f63e006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.666+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f63ec088810 con 0x7f640406f970 2024-09-13T17:17:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.666+0000 7f6401e00640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 0x7f63e006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:57.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.667+0000 7f6401e00640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 0x7f63e006d0f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f63f4034000 tx=0x7f63f402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:57.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.670+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f63ec0574e0 con 0x7f640406f970 2024-09-13T17:17:57.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.915+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f640406c9d0 con 0x7f640406f970 2024-09-13T17:17:57.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.916+0000 7f63faa00640 1 -- 172.21.15.28:0/3446595809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1671 (secure 0 0 0) 0x7f63ec057070 con 0x7f640406f970 2024-09-13T17:17:57.917 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:57.917 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":611,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":141,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":95},{"state_name":"peering","count":2}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076576690,"bytes_used":2685558784,"bytes_avail":770408554496,"bytes_total":773094113280,"inactive_pgs_ratio":0.020618556067347527,"write_bytes_sec":15483,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":26,"modified":"2024-09-13T17:17:53.032878+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 msgr2=0x7f63e006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 0x7f63e006d0f0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f63f4034000 tx=0x7f63f402e000 comp rx=0 tx=0).stop 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 msgr2=0x7f6404154c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f63ec001eb0 tx=0x7f63ec00c960 comp rx=0 tx=0).stop 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 shutdown_connections 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f63e006ac30 0x7f63e006d0f0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64041592a0 0x7f64040cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f640406f970 0x7f6404154c90 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 --2- 172.21.15.28:0/3446595809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f640406c2d0 0x7f6404151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 >> 172.21.15.28:0/3446595809 conn(0x7f6404069e80 msgr2=0x7f640415c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 shutdown_connections 2024-09-13T17:17:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:57.918+0000 7f6409200640 1 -- 172.21.15.28:0/3446595809 wait complete. 2024-09-13T17:17:57.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:17:58.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:17:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.101+0000 7f929ce00640 1 Processor -- start 2024-09-13T17:17:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.102+0000 7f929ce00640 1 -- start start 2024-09-13T17:17:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f929ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9298159440 0x7f92980b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f929ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f92980b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f929ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92980c3850 con 0x7f92980b22e0 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f929ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929815a930 con 0x7f9298159440 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f929ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9298159d20 con 0x7f9298159820 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f9297e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f92980b22e0 0x7f92980b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36312/0 (socket says 172.21.15.28:36312) 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f9297e00640 1 -- 172.21.15.28:0/4240025176 learned_addr learned my addr 172.21.15.28:0/4240025176 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f928ea00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f92980b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.103+0000 7f9297400640 1 --2- 172.21.15.28:0/4240025176 >> v2:172.21.15.189:3300/0 conn(0x7f9298159440 0x7f92980b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621315110 0 0) 0x7f92980c3850 con 0x7f92980b22e0 2024-09-13T17:17:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f927c005730 con 0x7f92980b22e0 2024-09-13T17:17:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820871849 0 0) 0x7f927c005730 con 0x7f92980b22e0 2024-09-13T17:17:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 msgr2=0x7f92980b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f92980b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 >> v2:172.21.15.189:3300/0 conn(0x7f9298159440 msgr2=0x7f92980b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:59.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 --2- 172.21.15.28:0/4240025176 >> v2:172.21.15.189:3300/0 conn(0x7f9298159440 0x7f92980b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929815a000 con 0x7f92980b22e0 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f92900019f0 con 0x7f92980b22e0 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 >> v1:172.21.15.28:6789/0 conn(0x7f92980b22e0 legacy=0x7f92980b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f927c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f927c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f927c006f10 0x7f927c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929815a930 con 0x7f927c005b80 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298159d20 con 0x7f927c006f10 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927c00db90 con 0x7f9298159820 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.104+0000 7f9297400640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f927c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297e00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f927c006f10 0x7f927c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f928ea00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f927c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297400640 1 -- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 msgr2=0x7f927c0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297400640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9298159820 0x7f927c0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297400640 1 -- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f927c006f10 msgr2=0x7f927c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297400640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f927c006f10 0x7f927c007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297400640 1 -- 172.21.15.28:0/4240025176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c005730 con 0x7f927c005b80 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.105+0000 7f9297e00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f927c006f10 0x7f927c007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.106+0000 7f9297400640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f927c005fa0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f9288009ac0 tx=0x7f927c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c6ae9f06c9f166a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.106+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288031400 con 0x7f927c005b80 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.106+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928802d050 con 0x7f927c005b80 2024-09-13T17:17:59.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.106+0000 7f9296a00640 1 -- 172.21.15.28:0/4240025176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288031780 con 0x7f927c005b80 2024-09-13T17:17:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 -- 172.21.15.28:0/4240025176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 msgr2=0x7f927c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 --2- 172.21.15.28:0/4240025176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f927c005fa0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f9288009ac0 tx=0x7f927c007a00 comp rx=0 tx=0).stop 2024-09-13T17:17:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f9297e00640 1 -- 172.21.15.28:0/4240025176 reap_dead start 2024-09-13T17:17:59.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 -- 172.21.15.28:0/4240025176 shutdown_connections 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 -- 172.21.15.28:0/4240025176 >> 172.21.15.28:0/4240025176 conn(0x7f9298072e40 msgr2=0x7f9298073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 -- 172.21.15.28:0/4240025176 shutdown_connections 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.107+0000 7f929ce00640 1 -- 172.21.15.28:0/4240025176 wait complete. 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.109+0000 7f929ce00640 1 Processor -- start 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.109+0000 7f929ce00640 1 -- start start 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92981f1dd0 0x7f929815ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92980aea20 0x7f92980aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92980c3740 con 0x7f927c005b80 2024-09-13T17:17:59.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92981f2300 con 0x7f92980aea20 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f929ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92981f25e0 con 0x7f92981f1dd0 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33438/0 (socket says 172.21.15.28:33438) 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 -- 172.21.15.28:0/2441361011 learned_addr learned my addr 172.21.15.28:0/2441361011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f928ea00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92981f1dd0 0x7f929815ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297e00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92980aea20 0x7f92980aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 -- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92981f1dd0 msgr2=0x7f929815ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92981f1dd0 0x7f929815ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 -- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92980aea20 msgr2=0x7f92980aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92980aea20 0x7f92980aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.110+0000 7f9297400640 1 -- 172.21.15.28:0/2441361011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92981f28c0 con 0x7f927c005b80 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f9297400640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f9288034040 tx=0x7f9288004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288003b90 con 0x7f927c005b80 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9288003d10 con 0x7f927c005b80 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288044650 con 0x7f927c005b80 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92981f2ba0 con 0x7f927c005b80 2024-09-13T17:17:59.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.111+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929815bfc0 con 0x7f927c005b80 2024-09-13T17:17:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.112+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929815c1e0 con 0x7f927c005b80 2024-09-13T17:17:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.113+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9288038070 con 0x7f927c005b80 2024-09-13T17:17:59.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.114+0000 7f928fe00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.114+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f92880a9480 con 0x7f927c005b80 2024-09-13T17:17:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.114+0000 7f928ea00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.115+0000 7f928ea00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 0x7f927406d0c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f92980b1c30 tx=0x7f9280008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.118+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f92880746c0 con 0x7f927c005b80 2024-09-13T17:17:59.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.334+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92980b0440 con 0x7f927c005b80 2024-09-13T17:17:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.335+0000 7f928fe00640 1 -- 172.21.15.28:0/2441361011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f9288077ce0 con 0x7f927c005b80 2024-09-13T17:17:59.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:59.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:17:59.336 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:17:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 msgr2=0x7f927406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 0x7f927406d0c0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f92980b1c30 tx=0x7f9280008040 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 msgr2=0x7f929806c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7f9288034040 tx=0x7f9288004730 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 shutdown_connections 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92980aea20 0x7f92980aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92981f1dd0 0x7f929815ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 --2- 172.21.15.28:0/2441361011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f927c005b80 0x7f929806c070 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 >> 172.21.15.28:0/2441361011 conn(0x7f9298072e40 msgr2=0x7f9298158310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 shutdown_connections 2024-09-13T17:17:59.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.337+0000 7f929ce00640 1 -- 172.21.15.28:0/2441361011 wait complete. 2024-09-13T17:17:59.354 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:17:59.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:17:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.508+0000 7fdebe800640 1 Processor -- start 2024-09-13T17:17:59.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.508+0000 7fdebe800640 1 -- start start 2024-09-13T17:17:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.509+0000 7fdebe800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdeb81592a0 0x7fdeb815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.509+0000 7fdebe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.509+0000 7fdebe800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb806cc70 con 0x7fdeb806f970 2024-09-13T17:17:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.509+0000 7fdebe800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb815b0b0 con 0x7fdeb81592a0 2024-09-13T17:17:59.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.509+0000 7fdebe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb8159d90 con 0x7fdeb806c2d0 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdeb81592a0 0x7fdeb815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:51314/0 (socket says 172.21.15.28:51314) 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 -- 172.21.15.28:0/3830250261 learned_addr learned my addr 172.21.15.28:0/3830250261 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 -- 172.21.15.28:0/3830250261 >> v2:172.21.15.189:3300/0 conn(0x7fdeb81592a0 msgr2=0x7fdeb815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:17:59.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 --2- 172.21.15.28:0/3830250261 >> v2:172.21.15.189:3300/0 conn(0x7fdeb81592a0 0x7fdeb815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 -- 172.21.15.28:0/3830250261 >> v1:172.21.15.189:6789/0 conn(0x7fdeb806f970 legacy=0x7fdeb8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.510+0000 7fdeb7e00640 1 -- 172.21.15.28:0/3830250261 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb80b2040 con 0x7fdeb806c2d0 2024-09-13T17:17:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.511+0000 7fdeb7e00640 1 --2- 172.21.15.28:0/3830250261 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009e50 tx=0x7fdeb002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09023526bc081b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3830250261 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0031600 con 0x7fdeb806c2d0 2024-09-13T17:17:59.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3830250261 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb002d050 con 0x7fdeb806c2d0 2024-09-13T17:17:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3830250261 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb0031980 con 0x7fdeb806c2d0 2024-09-13T17:17:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 -- 172.21.15.28:0/3830250261 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 msgr2=0x7fdeb806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 --2- 172.21.15.28:0/3830250261 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009e50 tx=0x7fdeb002a710 comp rx=0 tx=0).stop 2024-09-13T17:17:59.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 -- 172.21.15.28:0/3830250261 shutdown_connections 2024-09-13T17:17:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 --2- 172.21.15.28:0/3830250261 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb806c2d0 0x7fdeb806c730 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 --2- 172.21.15.28:0/3830250261 >> v2:172.21.15.189:3300/0 conn(0x7fdeb81592a0 0x7fdeb815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 -- 172.21.15.28:0/3830250261 >> 172.21.15.28:0/3830250261 conn(0x7fdeb8069e80 msgr2=0x7fdeb806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 -- 172.21.15.28:0/3830250261 shutdown_connections 2024-09-13T17:17:59.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.512+0000 7fdebe800640 1 -- 172.21.15.28:0/3830250261 wait complete. 2024-09-13T17:17:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.514+0000 7fdebe800640 1 Processor -- start 2024-09-13T17:17:59.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.514+0000 7fdebe800640 1 -- start start 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 0x7fdeb80ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb81592a0 0x7fdeb81f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f4190 con 0x7fdeb806f970 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb80cb5b0 con 0x7fdeb806c2d0 2024-09-13T17:17:59.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdebe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb81f43a0 con 0x7fdeb81592a0 2024-09-13T17:17:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdeaea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdeaea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33444/0 (socket says 172.21.15.28:33444) 2024-09-13T17:17:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdeaea00640 1 -- 172.21.15.28:0/2188589004 learned_addr learned my addr 172.21.15.28:0/2188589004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:17:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdeb7400640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 0x7fdeb80ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.515+0000 7fdeb7e00640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb81592a0 0x7fdeb81f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 -- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb81592a0 msgr2=0x7fdeb81f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb81592a0 0x7fdeb81f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 -- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 msgr2=0x7fdeb80ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 0x7fdeb80ce0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 -- 172.21.15.28:0/2188589004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb0009b30 con 0x7fdeb806f970 2024-09-13T17:17:59.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeb7400640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 0x7fdeb80ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeaea00640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c500 tx=0x7fdea000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00052d0 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea0005430 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb81f4680 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.516+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0012850 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.517+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb80ab760 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.517+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb80ab980 con 0x7fdeb806f970 2024-09-13T17:17:59.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.519+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdea00129b0 con 0x7fdeb806f970 2024-09-13T17:17:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.519+0000 7fdeafe00640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 0x7fde9406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:17:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.519+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fdea0088e10 con 0x7fdeb806f970 2024-09-13T17:17:59.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.520+0000 7fdeb7400640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 0x7fde9406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:17:59.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.521+0000 7fdeb7400640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 0x7fde9406d0c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009e90 tx=0x7fdea8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:17:59.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.523+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdea008c030 con 0x7fdeb806f970 2024-09-13T17:17:59.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.767+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fdeb80b3740 con 0x7fdeb806f970 2024-09-13T17:17:59.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.768+0000 7fdeafe00640 1 -- 172.21.15.28:0/2188589004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1748 (secure 0 0 0) 0x7fdea0053fd0 con 0x7fdeb806f970 2024-09-13T17:17:59.769 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:17:59.769 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":613,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":141,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1080,"data_bytes":1076581037,"bytes_used":2685665280,"bytes_avail":770408448000,"bytes_total":773094113280,"recovering_objects_per_sec":4,"recovering_bytes_per_sec":4586356,"recovering_keys_per_sec":0,"num_objects_recovered":35,"num_bytes_recovered":36700160,"num_keys_recovered":0,"write_bytes_sec":99046,"read_op_per_sec":0,"write_op_per_sec":262},"fsmap":{"epoch":122,"id":9,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":9,"rank":0,"name":"c","status":"up:active","gid":8904}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":27,"modified":"2024-09-13T17:17:59.034344+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 msgr2=0x7fde9406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 0x7fde9406d0c0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdea8009e90 tx=0x7fdea8009190 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 msgr2=0x7fdeb8150dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7fdea000c500 tx=0x7fdea000c9c0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 shutdown_connections 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9406ac00 0x7fde9406d0c0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdeb81592a0 0x7fdeb81f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdeb806f970 0x7fdeb8150dd0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 --2- 172.21.15.28:0/2188589004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdeb806c2d0 0x7fdeb80ce0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:17:59.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 >> 172.21.15.28:0/2188589004 conn(0x7fdeb8069e80 msgr2=0x7fdeb8155110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:17:59.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 shutdown_connections 2024-09-13T17:17:59.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:17:59.770+0000 7fdebe800640 1 -- 172.21.15.28:0/2188589004 wait complete. 2024-09-13T17:17:59.786 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:17:59.787 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:17:59.787 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:59.880 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:17:59.970 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:17:59.970 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:17:59.970 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:00.006 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:18:00.007 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:00.082 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.281+0000 7fbb9a400640 1 Processor -- start 2024-09-13T17:18:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.281+0000 7fbb9a400640 1 -- start start 2024-09-13T17:18:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.282+0000 7fbb9a400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb940cb760 0x7fbb94160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.282+0000 7fbb9a400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.282+0000 7fbb9a400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb940c4af0 con 0x7fbb940c4380 2024-09-13T17:18:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.282+0000 7fbb9a400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb940cd460 con 0x7fbb940cb760 2024-09-13T17:18:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.282+0000 7fbb9a400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940cd740 con 0x7fbb9406b5a0 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:51320/0 (socket says 172.21.15.28:51320) 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 -- 172.21.15.28:0/2151461318 learned_addr learned my addr 172.21.15.28:0/2151461318 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbb940cb760 0x7fbb94160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 -- 172.21.15.28:0/2151461318 >> v2:172.21.15.189:3300/0 conn(0x7fbb940cb760 msgr2=0x7fbb94160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 --2- 172.21.15.28:0/2151461318 >> v2:172.21.15.189:3300/0 conn(0x7fbb940cb760 0x7fbb94160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 -- 172.21.15.28:0/2151461318 >> v1:172.21.15.189:6789/0 conn(0x7fbb940c4380 legacy=0x7fbb9406ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.283+0000 7fbb99a00640 1 -- 172.21.15.28:0/2151461318 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb940cda20 con 0x7fbb9406b5a0 2024-09-13T17:18:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.284+0000 7fbb99a00640 1 --2- 172.21.15.28:0/2151461318 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009e50 tx=0x7fbb8c02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22461ccdc93944d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.284+0000 7fbb93e00640 1 -- 172.21.15.28:0/2151461318 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c02aea0 con 0x7fbb9406b5a0 2024-09-13T17:18:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb93e00640 1 -- 172.21.15.28:0/2151461318 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c02d050 con 0x7fbb9406b5a0 2024-09-13T17:18:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb93e00640 1 -- 172.21.15.28:0/2151461318 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c031820 con 0x7fbb9406b5a0 2024-09-13T17:18:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 -- 172.21.15.28:0/2151461318 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 msgr2=0x7fbb940ce220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 --2- 172.21.15.28:0/2151461318 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009e50 tx=0x7fbb8c02a680 comp rx=0 tx=0).stop 2024-09-13T17:18:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 -- 172.21.15.28:0/2151461318 shutdown_connections 2024-09-13T17:18:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 --2- 172.21.15.28:0/2151461318 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940ce220 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 --2- 172.21.15.28:0/2151461318 >> v2:172.21.15.189:3300/0 conn(0x7fbb940cb760 0x7fbb94160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 -- 172.21.15.28:0/2151461318 >> 172.21.15.28:0/2151461318 conn(0x7fbb940703a0 msgr2=0x7fbb940707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 -- 172.21.15.28:0/2151461318 shutdown_connections 2024-09-13T17:18:00.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.285+0000 7fbb9a400640 1 -- 172.21.15.28:0/2151461318 wait complete. 2024-09-13T17:18:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.287+0000 7fbb9a400640 1 Processor -- start 2024-09-13T17:18:00.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.287+0000 7fbb9a400640 1 -- start start 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940b4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb940cb760 0x7fbb9415f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940c8b00 con 0x7fbb940c4380 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb941f26a0 con 0x7fbb940cb760 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb9a400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb940b4e50 con 0x7fbb9406b5a0 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33446/0 (socket says 172.21.15.28:33446) 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.288+0000 7fbb8be00640 1 -- 172.21.15.28:0/93331808 learned_addr learned my addr 172.21.15.28:0/93331808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb99a00640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb940cb760 0x7fbb9415f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb99000640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940b4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 -- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 msgr2=0x7fbb940b4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940b4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 -- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb940cb760 msgr2=0x7fbb9415f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb940cb760 0x7fbb9415f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 -- 172.21.15.28:0/93331808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c009b30 con 0x7fbb940c4380 2024-09-13T17:18:00.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb8be00640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00c450 tx=0x7fbb7c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c0052d0 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb7c005430 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.289+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7c012870 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.290+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb94138b30 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.290+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb940c7e80 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.291+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb940c63b0 con 0x7fbb940c4380 2024-09-13T17:18:00.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.291+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbb7c0129d0 con 0x7fbb940c4380 2024-09-13T17:18:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.292+0000 7fbb91600640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 0x7fbb7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.292+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbb7c088e70 con 0x7fbb940c4380 2024-09-13T17:18:00.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.293+0000 7fbb99000640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 0x7fbb7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.293+0000 7fbb99000640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 0x7fbb7006d0c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009aa0 tx=0x7fbb84009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.297+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbb7c053f90 con 0x7fbb940c4380 2024-09-13T17:18:00.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.477+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb940ce220 con 0x7fbb940c4380 2024-09-13T17:18:00.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.478+0000 7fbb91600640 1 -- 172.21.15.28:0/93331808 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fbb7c0575b0 con 0x7fbb940c4380 2024-09-13T17:18:00.479 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:00.480 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":141,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:53.123298+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.16","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 msgr2=0x7fbb7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 0x7fbb7006d0c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009aa0 tx=0x7fbb84009190 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 msgr2=0x7fbb94135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c00c450 tx=0x7fbb7c00c910 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 shutdown_connections 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbb7006ac00 0x7fbb7006d0c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbb940cb760 0x7fbb9415f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb940c4380 0x7fbb94135c90 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 --2- 172.21.15.28:0/93331808 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbb9406b5a0 0x7fbb940b4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 >> 172.21.15.28:0/93331808 conn(0x7fbb940703a0 msgr2=0x7fbb940ad990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 shutdown_connections 2024-09-13T17:18:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.480+0000 7fbb9a400640 1 -- 172.21.15.28:0/93331808 wait complete. 2024-09-13T17:18:00.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:17:19.853908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:17:20.955756+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:18:00.497 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:18:00.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.655+0000 7f5052200640 1 Processor -- start 2024-09-13T17:18:00.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.655+0000 7f5052200640 1 -- start start 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f5052200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 0x7f504c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f5052200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f504c06f3f0 0x7f504c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f5052200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f504c06f920 con 0x7f504c0add00 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f5052200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f504c15b290 con 0x7f504c06f3f0 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f5052200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c15b570 con 0x7f504c159770 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f504be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f504c0add00 0x7f504c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36324/0 (socket says 172.21.15.28:36324) 2024-09-13T17:18:00.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.656+0000 7f504be00640 1 -- 172.21.15.28:0/1328629719 learned_addr learned my addr 172.21.15.28:0/1328629719 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504b400640 1 --2- 172.21.15.28:0/1328629719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 0x7f504c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f5042a00640 1 --2- 172.21.15.28:0/1328629719 >> v2:172.21.15.189:3300/0 conn(0x7f504c06f3f0 0x7f504c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504aa00640 1 -- 172.21.15.28:0/1328629719 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761420962 0 0) 0x7f504c06f920 con 0x7f504c0add00 2024-09-13T17:18:00.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504aa00640 1 -- 172.21.15.28:0/1328629719 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030003520 con 0x7f504c0add00 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504b400640 1 -- 172.21.15.28:0/1328629719 >> v2:172.21.15.189:3300/0 conn(0x7f504c06f3f0 msgr2=0x7f504c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504b400640 1 --2- 172.21.15.28:0/1328629719 >> v2:172.21.15.189:3300/0 conn(0x7f504c06f3f0 0x7f504c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504b400640 1 -- 172.21.15.28:0/1328629719 >> v1:172.21.15.28:6789/0 conn(0x7f504c0add00 legacy=0x7f504c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.657+0000 7f504b400640 1 -- 172.21.15.28:0/1328629719 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504c15b850 con 0x7f504c159770 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.658+0000 7f504b400640 1 --2- 172.21.15.28:0/1328629719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 0x7f504c06fd20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f503c0097a0 tx=0x7f503c02a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2942995f01e8bba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f504aa00640 1 -- 172.21.15.28:0/1328629719 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c02ae40 con 0x7f504c159770 2024-09-13T17:18:00.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f504aa00640 1 -- 172.21.15.28:0/1328629719 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c032440 con 0x7f504c159770 2024-09-13T17:18:00.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f504aa00640 1 -- 172.21.15.28:0/1328629719 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c0327b0 con 0x7f504c159770 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 -- 172.21.15.28:0/1328629719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 msgr2=0x7f504c06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 --2- 172.21.15.28:0/1328629719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 0x7f504c06fd20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f503c0097a0 tx=0x7f503c02a0f0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 -- 172.21.15.28:0/1328629719 shutdown_connections 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 --2- 172.21.15.28:0/1328629719 >> v2:172.21.15.189:3300/0 conn(0x7f504c06f3f0 0x7f504c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 --2- 172.21.15.28:0/1328629719 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c159770 0x7f504c06fd20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.659+0000 7f5052200640 1 -- 172.21.15.28:0/1328629719 >> 172.21.15.28:0/1328629719 conn(0x7f504c0b0270 msgr2=0x7f504c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.660+0000 7f5052200640 1 -- 172.21.15.28:0/1328629719 shutdown_connections 2024-09-13T17:18:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.660+0000 7f5052200640 1 -- 172.21.15.28:0/1328629719 wait complete. 2024-09-13T17:18:00.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.661+0000 7f5052200640 1 Processor -- start 2024-09-13T17:18:00.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.661+0000 7f5052200640 1 -- start start 2024-09-13T17:18:00.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 0x7f504c147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504c159770 0x7f504c157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c1476d0 con 0x7f504c0add00 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f504c0aaff0 con 0x7f504c159770 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5052200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f504c0ab2d0 con 0x7f504c06f3f0 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.662+0000 7f5042a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33460/0 (socket says 172.21.15.28:33460) 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 -- 172.21.15.28:0/2844163756 learned_addr learned my addr 172.21.15.28:0/2844163756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f504b400640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 0x7f504c147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f504be00640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504c159770 0x7f504c157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 -- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 msgr2=0x7f504c147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 0x7f504c147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 -- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504c159770 msgr2=0x7f504c157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504c159770 0x7f504c157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 -- 172.21.15.28:0/2844163756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c009400 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f504b400640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 0x7f504c147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5042a00640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f503400e840 tx=0x7f503400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50340094f0 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.663+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f504c0ab5b0 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.664+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503400c940 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.664+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503400ccd0 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.664+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f504c0abb70 con 0x7f504c0add00 2024-09-13T17:18:00.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.665+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f504c06fd20 con 0x7f504c0add00 2024-09-13T17:18:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.666+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5034013470 con 0x7f504c0add00 2024-09-13T17:18:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.666+0000 7f5043e00640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 0x7f502806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.667+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f50340894b0 con 0x7f504c0add00 2024-09-13T17:18:00.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.667+0000 7f504b400640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 0x7f502806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:00.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.668+0000 7f504b400640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 0x7f502806d0c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f503c034000 tx=0x7f503c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:00.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.671+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f50340545d0 con 0x7f504c0add00 2024-09-13T17:18:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.846+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f504c0ceb50 con 0x7f504c0add00 2024-09-13T17:18:00.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.846+0000 7f5043e00640 1 -- 172.21.15.28:0/2844163756 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5034057bf0 con 0x7f504c0add00 2024-09-13T17:18:00.848 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.848+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 msgr2=0x7f502806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.848+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 0x7f502806d0c0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f503c034000 tx=0x7f503c02e000 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.848+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 msgr2=0x7f504c0a9ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f503400e840 tx=0x7f503400ed00 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 shutdown_connections 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f502806ac00 0x7f502806d0c0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f504c159770 0x7f504c157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f504c0add00 0x7f504c0a9ea0 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 --2- 172.21.15.28:0/2844163756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f504c06f3f0 0x7f504c147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 >> 172.21.15.28:0/2844163756 conn(0x7f504c0b0270 msgr2=0x7f504c0cd600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:00.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 shutdown_connections 2024-09-13T17:18:00.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:00.849+0000 7f5052200640 1 -- 172.21.15.28:0/2844163756 wait complete. 2024-09-13T17:18:00.865 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:18:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.023+0000 7ff3a3200640 1 Processor -- start 2024-09-13T17:18:01.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.023+0000 7ff3a3200640 1 -- start start 2024-09-13T17:18:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3a4072140 0x7ff3a40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a4159e10 con 0x7ff3a4071210 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a415a0f0 con 0x7ff3a4072140 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a415a3d0 con 0x7ff3a4070e30 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff399400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff3a4072140 0x7ff3a40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3a4071210 0x7ff3a415d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36328/0 (socket says 172.21.15.28:36328) 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a2800640 1 -- 172.21.15.28:0/1599963159 learned_addr learned my addr 172.21.15.28:0/1599963159 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.024+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.025+0000 7ff399400640 1 -- 172.21.15.28:0/1599963159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 msgr2=0x7ff3a4071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.025+0000 7ff399400640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.025+0000 7ff3a1400640 1 -- 172.21.15.28:0/1599963159 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158738576 0 0) 0x7ff3a4159e10 con 0x7ff3a4071210 2024-09-13T17:18:01.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.025+0000 7ff399400640 1 -- 172.21.15.28:0/1599963159 >> v1:172.21.15.28:6789/0 conn(0x7ff3a4071210 legacy=0x7ff3a415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.025+0000 7ff399400640 1 -- 172.21.15.28:0/1599963159 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a415a6b0 con 0x7ff3a4072140 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.026+0000 7ff399400640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4072140 0x7ff3a40cd870 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0097a0 tx=0x7ff38c02c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76b45b729b27b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.026+0000 7ff3a1400640 1 -- 172.21.15.28:0/1599963159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c034640 con 0x7ff3a4072140 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.026+0000 7ff3a1400640 1 -- 172.21.15.28:0/1599963159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c0347a0 con 0x7ff3a4072140 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.026+0000 7ff3a1400640 1 -- 172.21.15.28:0/1599963159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c034b30 con 0x7ff3a4072140 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 -- 172.21.15.28:0/1599963159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4072140 msgr2=0x7ff3a40cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4072140 0x7ff3a40cd870 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7ff38c0097a0 tx=0x7ff38c02c390 comp rx=0 tx=0).stop 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 -- 172.21.15.28:0/1599963159 shutdown_connections 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4072140 0x7ff3a40cd870 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 --2- 172.21.15.28:0/1599963159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 -- 172.21.15.28:0/1599963159 >> 172.21.15.28:0/1599963159 conn(0x7ff3a4069e80 msgr2=0x7ff3a406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 -- 172.21.15.28:0/1599963159 shutdown_connections 2024-09-13T17:18:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.027+0000 7ff3a3200640 1 -- 172.21.15.28:0/1599963159 wait complete. 2024-09-13T17:18:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.028+0000 7ff3a3200640 1 Processor -- start 2024-09-13T17:18:01.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.028+0000 7ff3a3200640 1 -- start start 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 0x7ff3a40c5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a4159320 con 0x7ff3a4071210 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a4158710 con 0x7ff3a4072140 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.029+0000 7ff3a3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a41589f0 con 0x7ff3a4070e30 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33478/0 (socket says 172.21.15.28:33478) 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 -- 172.21.15.28:0/3699569395 learned_addr learned my addr 172.21.15.28:0/3699569395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff3a2800640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 0x7ff3a40c5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 -- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 msgr2=0x7ff3a4150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 -- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 msgr2=0x7ff3a40c5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 0x7ff3a40c5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 -- 172.21.15.28:0/3699569395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38c009400 con 0x7ff3a4071210 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff3a2800640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 0x7ff3a40c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.030+0000 7ff399400640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff38c037040 tx=0x7ff38c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.031+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c032910 con 0x7ff3a4071210 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.031+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff38c032a90 con 0x7ff3a4071210 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.031+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a4158cd0 con 0x7ff3a4071210 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.031+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff38c032e40 con 0x7ff3a4071210 2024-09-13T17:18:01.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.031+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a4151030 con 0x7ff3a4071210 2024-09-13T17:18:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.032+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a4159e10 con 0x7ff3a4071210 2024-09-13T17:18:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.033+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff38c03b040 con 0x7ff3a4071210 2024-09-13T17:18:01.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.034+0000 7ff39aa00640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 0x7ff38006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.034+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff38c0abda0 con 0x7ff3a4071210 2024-09-13T17:18:01.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.034+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 0x7ff38006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.035+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 0x7ff38006d0c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff394009b40 tx=0x7ff394009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.039+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff38c076f50 con 0x7ff3a4071210 2024-09-13T17:18:01.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.220+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff3a40719e0 con 0x7ff3a4071210 2024-09-13T17:18:01.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.220+0000 7ff39aa00640 1 -- 172.21.15.28:0/3699569395 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff38c07a570 con 0x7ff3a4071210 2024-09-13T17:18:01.221 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.222+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 msgr2=0x7ff38006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.222+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 0x7ff38006d0c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff394009b40 tx=0x7ff394009190 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.222+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 msgr2=0x7ff3a4157cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.222+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7ff38c037040 tx=0x7ff38c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 shutdown_connections 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff38006ac00 0x7ff38006d0c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3a4072140 0x7ff3a40c5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4071210 0x7ff3a4157cc0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 --2- 172.21.15.28:0/3699569395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3a4070e30 0x7ff3a4150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 >> 172.21.15.28:0/3699569395 conn(0x7ff3a4069e80 msgr2=0x7ff3a40cd640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 shutdown_connections 2024-09-13T17:18:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.223+0000 7ff3a3200640 1 -- 172.21.15.28:0/3699569395 wait complete. 2024-09-13T17:18:01.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:18:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.397+0000 7f750ca00640 1 Processor -- start 2024-09-13T17:18:01.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.397+0000 7f750ca00640 1 -- start start 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 0x7f750806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f750806f3f0 0x7f7508070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f75080add00 0x7f7508159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750806f920 con 0x7f75080add00 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750815b290 con 0x7f7508159770 2024-09-13T17:18:01.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.398+0000 7f750ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750815b570 con 0x7f750806f3f0 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f75080add00 0x7f7508159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f75080add00 0x7f7508159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33488/0 (socket says 172.21.15.28:33488) 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 -- 172.21.15.28:0/3373113058 learned_addr learned my addr 172.21.15.28:0/3373113058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7505400640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f750806f3f0 0x7f7508070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7505e00640 1 --2- 172.21.15.28:0/3373113058 >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 0x7f750806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 -- 172.21.15.28:0/3373113058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f750806f3f0 msgr2=0x7f7508070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f750806f3f0 0x7f7508070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 -- 172.21.15.28:0/3373113058 >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 msgr2=0x7f750806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 --2- 172.21.15.28:0/3373113058 >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 0x7f750806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7506800640 1 -- 172.21.15.28:0/3373113058 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750815b850 con 0x7f75080add00 2024-09-13T17:18:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.399+0000 7f7505e00640 1 --2- 172.21.15.28:0/3373113058 >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 0x7f750806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.400+0000 7f7506800640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75080add00 0x7f7508159b50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f7508067f80 tx=0x7f750002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e620d07b5039f9a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.400+0000 7f7504a00640 1 -- 172.21.15.28:0/3373113058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500031600 con 0x7f75080add00 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.400+0000 7f7504a00640 1 -- 172.21.15.28:0/3373113058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750002d050 con 0x7f75080add00 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.400+0000 7f7504a00640 1 -- 172.21.15.28:0/3373113058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7500031980 con 0x7f75080add00 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 -- 172.21.15.28:0/3373113058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75080add00 msgr2=0x7f7508159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75080add00 0x7f7508159b50 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f7508067f80 tx=0x7f750002a610 comp rx=0 tx=0).stop 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 -- 172.21.15.28:0/3373113058 shutdown_connections 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75080add00 0x7f7508159b50 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 --2- 172.21.15.28:0/3373113058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f750806f3f0 0x7f7508070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 --2- 172.21.15.28:0/3373113058 >> v2:172.21.15.189:3300/0 conn(0x7f7508159770 0x7f750806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 -- 172.21.15.28:0/3373113058 >> 172.21.15.28:0/3373113058 conn(0x7f75080b0270 msgr2=0x7f75080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 -- 172.21.15.28:0/3373113058 shutdown_connections 2024-09-13T17:18:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.401+0000 7f750ca00640 1 -- 172.21.15.28:0/3373113058 wait complete. 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.403+0000 7f750ca00640 1 Processor -- start 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.403+0000 7f750ca00640 1 -- start start 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75080add00 0x7f750813bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7508159770 0x7f7508150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75080cacc0 con 0x7f750806f3f0 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75081514a0 con 0x7f7508159770 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f750ca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75080b26a0 con 0x7f75080add00 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33504/0 (socket says 172.21.15.28:33504) 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 -- 172.21.15.28:0/1982342022 learned_addr learned my addr 172.21.15.28:0/1982342022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 -- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75080add00 msgr2=0x7f750813bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75080add00 0x7f750813bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 -- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7508159770 msgr2=0x7f7508150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7508159770 0x7f7508150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.404+0000 7f7505e00640 1 -- 172.21.15.28:0/1982342022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009b30 con 0x7f750806f3f0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f7505e00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f74f800d7d0 tx=0x7f74f800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8004030 con 0x7f750806f3f0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8004190 con 0x7f750806f3f0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f8004520 con 0x7f750806f3f0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75080b2980 con 0x7f750806f3f0 2024-09-13T17:18:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.405+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750814ff80 con 0x7f750806f3f0 2024-09-13T17:18:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.406+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7508159b50 con 0x7f750806f3f0 2024-09-13T17:18:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.407+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f74f800b6c0 con 0x7f750806f3f0 2024-09-13T17:18:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.408+0000 7f74fe000640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 0x7f74e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.408+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f74f8089550 con 0x7f750806f3f0 2024-09-13T17:18:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.408+0000 7f7505400640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 0x7f74e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.409+0000 7f7505400640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 0x7f74e806d0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006a90 tx=0x7f74f4006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.412+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f74f8054650 con 0x7f750806f3f0 2024-09-13T17:18:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.592+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75080af050 con 0x7f750806f3f0 2024-09-13T17:18:01.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.592+0000 7f74fe000640 1 -- 172.21.15.28:0/1982342022 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f74f8057c70 con 0x7f750806f3f0 2024-09-13T17:18:01.593 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:18:01.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 msgr2=0x7f74e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 0x7f74e806d0c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f74f4006a90 tx=0x7f74f4006a20 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 msgr2=0x7f75080ca780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f74f800d7d0 tx=0x7f74f800dc90 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 shutdown_connections 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74e806ac00 0x7f74e806d0c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7508159770 0x7f7508150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75080add00 0x7f750813bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 --2- 172.21.15.28:0/1982342022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f750806f3f0 0x7f75080ca780 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.594+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 >> 172.21.15.28:0/1982342022 conn(0x7f75080b0270 msgr2=0x7f75081542b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.595+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 shutdown_connections 2024-09-13T17:18:01.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.595+0000 7f750ca00640 1 -- 172.21.15.28:0/1982342022 wait complete. 2024-09-13T17:18:01.611 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:18:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.767+0000 7f0abd200640 1 Processor -- start 2024-09-13T17:18:01.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.767+0000 7f0abd200640 1 -- start start 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0abd200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ab8070430 0x7f0ab815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0abd200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab815ab60 con 0x7f0ab8159770 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0abd200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab806f920 con 0x7f0ab806f3f0 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0abd200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab815b290 con 0x7f0ab8070430 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0ab5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ab8159770 0x7f0ab806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36336/0 (socket says 172.21.15.28:36336) 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.768+0000 7f0ab5e00640 1 -- 172.21.15.28:0/851457037 learned_addr learned my addr 172.21.15.28:0/851457037 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0aafe00640 1 --2- 172.21.15.28:0/851457037 >> v2:172.21.15.189:3300/0 conn(0x7f0ab8070430 0x7f0ab815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596229689 0 0) 0x7f0aa8001110 con 0x7f0ab8159770 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c003520 con 0x7f0ab8159770 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2430053725 0 0) 0x7f0a9c003520 con 0x7f0ab8159770 2024-09-13T17:18:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0aafe00640 1 -- 172.21.15.28:0/851457037 >> v1:172.21.15.189:6789/0 conn(0x7f0ab806f3f0 legacy=0x7f0ab815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0aafe00640 1 -- 172.21.15.28:0/851457037 >> v1:172.21.15.28:6789/0 conn(0x7f0ab8159770 legacy=0x7f0ab806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.769+0000 7f0aafe00640 1 -- 172.21.15.28:0/851457037 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab815b570 con 0x7f0ab8070430 2024-09-13T17:18:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.770+0000 7f0aafe00640 1 --2- 172.21.15.28:0/851457037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8070430 0x7f0ab815bd70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00097a0 tx=0x7f0aa002a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9e9d216c41c0013 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8001800 con 0x7f0ab8070430 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa002abc0 con 0x7f0ab8070430 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0ab5400640 1 -- 172.21.15.28:0/851457037 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0032440 con 0x7f0ab8070430 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 -- 172.21.15.28:0/851457037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8070430 msgr2=0x7f0ab815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 --2- 172.21.15.28:0/851457037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8070430 0x7f0ab815bd70 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00097a0 tx=0x7f0aa002a130 comp rx=0 tx=0).stop 2024-09-13T17:18:01.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 -- 172.21.15.28:0/851457037 shutdown_connections 2024-09-13T17:18:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 --2- 172.21.15.28:0/851457037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8070430 0x7f0ab815bd70 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 -- 172.21.15.28:0/851457037 >> 172.21.15.28:0/851457037 conn(0x7f0ab80b0270 msgr2=0x7f0ab80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 -- 172.21.15.28:0/851457037 shutdown_connections 2024-09-13T17:18:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.771+0000 7f0abd200640 1 -- 172.21.15.28:0/851457037 wait complete. 2024-09-13T17:18:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.773+0000 7f0abd200640 1 Processor -- start 2024-09-13T17:18:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.773+0000 7f0abd200640 1 -- start start 2024-09-13T17:18:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab806f3f0 0x7f0ab814f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159770 0x7f0ab8157de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80cc700 con 0x7f0ab8070430 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab814e6d0 con 0x7f0ab806f3f0 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0abd200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80cc950 con 0x7f0ab8159770 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0aafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0aafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33510/0 (socket says 172.21.15.28:33510) 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0aafe00640 1 -- 172.21.15.28:0/2623280358 learned_addr learned my addr 172.21.15.28:0/2623280358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0ab6800640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159770 0x7f0ab8157de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.774+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab806f3f0 0x7f0ab814f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 -- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159770 msgr2=0x7f0ab8157de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159770 0x7f0ab8157de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 -- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab806f3f0 msgr2=0x7f0ab814f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab806f3f0 0x7f0ab814f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 -- 172.21.15.28:0/2623280358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa0009400 con 0x7f0ab8070430 2024-09-13T17:18:01.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aafe00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f0aa002a720 tx=0x7f0aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0003920 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa0003a80 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0003e10 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab80ccc30 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.775+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab80cd1f0 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.776+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab806fd20 con 0x7f0ab8070430 2024-09-13T17:18:01.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.777+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0aa0032440 con 0x7f0ab8070430 2024-09-13T17:18:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.779+0000 7f0aae000640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 0x7f0a9406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:01.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.779+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0aa00a9d20 con 0x7f0ab8070430 2024-09-13T17:18:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.779+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 0x7f0a9406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:01.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.780+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 0x7f0a9406d0c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8007fd0 tx=0x7f0aa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:01.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.783+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0aa0074f40 con 0x7f0ab8070430 2024-09-13T17:18:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.977+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0ab814ea40 con 0x7f0ab8070430 2024-09-13T17:18:01.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.977+0000 7f0aae000640 1 -- 172.21.15.28:0/2623280358 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v141) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0aa0041080 con 0x7f0ab8070430 2024-09-13T17:18:01.979 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 msgr2=0x7f0a9406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 0x7f0a9406d0c0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8007fd0 tx=0x7f0aa8009040 comp rx=0 tx=0).stop 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 msgr2=0x7f0ab814e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f0aa002a720 tx=0x7f0aa00046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 shutdown_connections 2024-09-13T17:18:01.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac00 0x7f0a9406d0c0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159770 0x7f0ab8157de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8070430 0x7f0ab814e190 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 --2- 172.21.15.28:0/2623280358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab806f3f0 0x7f0ab814f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 >> 172.21.15.28:0/2623280358 conn(0x7f0ab80b0270 msgr2=0x7f0ab80ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 shutdown_connections 2024-09-13T17:18:01.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:01.980+0000 7f0abd200640 1 -- 172.21.15.28:0/2623280358 wait complete. 2024-09-13T17:18:01.998 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:18:01.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:01.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:18:02.826 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:18:02.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:02.826 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:18:02.862 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:18:02.862 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:02.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:18:02.901 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:18:02.901 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:02.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:18:02.938 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:18:02.938 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:02.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:18:02.980 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.135+0000 7fde9f400640 1 Processor -- start 2024-09-13T17:18:03.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.135+0000 7fde9f400640 1 -- start start 2024-09-13T17:18:03.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fde9815a540 0x7fde9806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde98070cc0 0x7fde98071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9f400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9815e100 con 0x7fde980cabc0 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9f400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde980adb00 con 0x7fde98070cc0 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9f400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde98071670 con 0x7fde9815a540 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde98070cc0 0x7fde98071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fde98070cc0 0x7fde98071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33526/0 (socket says 172.21.15.28:33526) 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.136+0000 7fde9d400640 1 -- 172.21.15.28:0/1886652840 learned_addr learned my addr 172.21.15.28:0/1886652840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.137+0000 7fde9ca00640 1 --2- 172.21.15.28:0/1886652840 >> v2:172.21.15.189:3300/0 conn(0x7fde9815a540 0x7fde9806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.137+0000 7fde9d400640 1 -- 172.21.15.28:0/1886652840 >> v2:172.21.15.189:3300/0 conn(0x7fde9815a540 msgr2=0x7fde9806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.137+0000 7fde9d400640 1 --2- 172.21.15.28:0/1886652840 >> v2:172.21.15.189:3300/0 conn(0x7fde9815a540 0x7fde9806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.137+0000 7fde9d400640 1 -- 172.21.15.28:0/1886652840 >> v1:172.21.15.189:6789/0 conn(0x7fde980cabc0 legacy=0x7fde9815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:03.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.137+0000 7fde9d400640 1 -- 172.21.15.28:0/1886652840 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9815e3e0 con 0x7fde98070cc0 2024-09-13T17:18:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.138+0000 7fde9d400640 1 --2- 172.21.15.28:0/1886652840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98071130 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fde980b3a00 tx=0x7fde9002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9df6dcd4fc955e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.138+0000 7fde97400640 1 -- 172.21.15.28:0/1886652840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde90031600 con 0x7fde98070cc0 2024-09-13T17:18:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.138+0000 7fde97400640 1 -- 172.21.15.28:0/1886652840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde9002d050 con 0x7fde98070cc0 2024-09-13T17:18:03.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.138+0000 7fde97400640 1 -- 172.21.15.28:0/1886652840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde90031980 con 0x7fde98070cc0 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 -- 172.21.15.28:0/1886652840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 msgr2=0x7fde98071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 --2- 172.21.15.28:0/1886652840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98071130 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fde980b3a00 tx=0x7fde9002a680 comp rx=0 tx=0).stop 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 -- 172.21.15.28:0/1886652840 shutdown_connections 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 --2- 172.21.15.28:0/1886652840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98071130 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 --2- 172.21.15.28:0/1886652840 >> v2:172.21.15.189:3300/0 conn(0x7fde9815a540 0x7fde9806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 -- 172.21.15.28:0/1886652840 >> 172.21.15.28:0/1886652840 conn(0x7fde98069e80 msgr2=0x7fde9806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 -- 172.21.15.28:0/1886652840 shutdown_connections 2024-09-13T17:18:03.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.139+0000 7fde9f400640 1 -- 172.21.15.28:0/1886652840 wait complete. 2024-09-13T17:18:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.140+0000 7fde9f400640 1 Processor -- start 2024-09-13T17:18:03.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.141+0000 7fde9f400640 1 -- start start 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.141+0000 7fde9f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde980cabc0 0x7fde98146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde9815a540 0x7fde981566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981582e0 con 0x7fde98070cc0 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981584f0 con 0x7fde9815a540 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde981587d0 con 0x7fde980cabc0 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33536/0 (socket says 172.21.15.28:33536) 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 -- 172.21.15.28:0/700122388 learned_addr learned my addr 172.21.15.28:0/700122388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9d400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde9815a540 0x7fde981566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde97e00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde980cabc0 0x7fde98146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 -- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde980cabc0 msgr2=0x7fde98146f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde980cabc0 0x7fde98146f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 -- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde9815a540 msgr2=0x7fde981566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde9815a540 0x7fde981566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.142+0000 7fde9ca00640 1 -- 172.21.15.28:0/700122388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde90009b30 con 0x7fde98070cc0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde9ca00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00d5f0 tx=0x7fde8c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c0125f0 con 0x7fde98070cc0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c012750 con 0x7fde98070cc0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c012b00 con 0x7fde98070cc0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde98158ab0 con 0x7fde98070cc0 2024-09-13T17:18:03.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.143+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde98159070 con 0x7fde98070cc0 2024-09-13T17:18:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.144+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde98159200 con 0x7fde98070cc0 2024-09-13T17:18:03.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.145+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fde8c012c60 con 0x7fde98070cc0 2024-09-13T17:18:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.146+0000 7fde94c00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 0x7fde7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.146+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fde8c089870 con 0x7fde98070cc0 2024-09-13T17:18:03.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.146+0000 7fde97e00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 0x7fde7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.147+0000 7fde97e00640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 0x7fde7806d0c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fde980cd870 tx=0x7fde84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.150+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fde8c054970 con 0x7fde98070cc0 2024-09-13T17:18:03.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.370+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde98071130 con 0x7fde98070cc0 2024-09-13T17:18:03.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.370+0000 7fde94c00640 1 -- 172.21.15.28:0/700122388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fde8c057f90 con 0x7fde98070cc0 2024-09-13T17:18:03.371 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:03.371 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:18:03.372 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.372+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 msgr2=0x7fde7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.372+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 0x7fde7806d0c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fde980cd870 tx=0x7fde84009040 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.372+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 msgr2=0x7fde98159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.372+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fde8c00d5f0 tx=0x7fde8c00dab0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 shutdown_connections 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde7806ac00 0x7fde7806d0c0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde9815a540 0x7fde981566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde980cabc0 0x7fde98146f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 --2- 172.21.15.28:0/700122388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde98070cc0 0x7fde98159590 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 >> 172.21.15.28:0/700122388 conn(0x7fde98069e80 msgr2=0x7fde9806f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 shutdown_connections 2024-09-13T17:18:03.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.373+0000 7fde9f400640 1 -- 172.21.15.28:0/700122388 wait complete. 2024-09-13T17:18:03.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.547+0000 7f2ebe800640 1 Processor -- start 2024-09-13T17:18:03.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.547+0000 7f2ebe800640 1 -- start start 2024-09-13T17:18:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2ebe800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2eb806c9e0 0x7f2eb815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2ebe800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb8159d90 con 0x7f2eb806f970 2024-09-13T17:18:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2ebe800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb80b2040 con 0x7f2eb81592a0 2024-09-13T17:18:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2ebe800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb815b0b0 con 0x7f2eb806c9e0 2024-09-13T17:18:03.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2eaea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2eb806f970 0x7f2eb806c2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:36350/0 (socket says 172.21.15.28:36350) 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.548+0000 7f2eaea00640 1 -- 172.21.15.28:0/247024527 learned_addr learned my addr 172.21.15.28:0/247024527 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb7e00640 1 --2- 172.21.15.28:0/247024527 >> v2:172.21.15.189:3300/0 conn(0x7f2eb806c9e0 0x7f2eb815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924922902 0 0) 0x7f2eb8159d90 con 0x7f2eb806f970 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c003520 con 0x7f2eb806f970 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199822715 0 0) 0x7f2e9c003520 con 0x7f2eb806f970 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 >> v2:172.21.15.189:3300/0 conn(0x7f2eb806c9e0 msgr2=0x7f2eb815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 --2- 172.21.15.28:0/247024527 >> v2:172.21.15.189:3300/0 conn(0x7f2eb806c9e0 0x7f2eb815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 >> v1:172.21.15.189:6789/0 conn(0x7f2eb81592a0 legacy=0x7f2eb8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8159ff0 con 0x7f2eb806f970 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f2ea0001690 con 0x7f2eb806f970 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 >> v1:172.21.15.28:6789/0 conn(0x7f2eb806f970 legacy=0x7f2eb806c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c003970 0x7f2e9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 0x7f2e9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e9c005180 0x7f2e9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb815b0b0 con 0x7f2e9c0047c0 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c00c020 con 0x7f2e9c003970 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.549+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9c00c1e0 con 0x7f2e9c005180 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 0x7f2e9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eb7400640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c003970 0x7f2e9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eb7e00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e9c005180 0x7f2e9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 -- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e9c005180 msgr2=0x7f2e9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e9c005180 0x7f2e9c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 -- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c003970 msgr2=0x7f2e9c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c003970 0x7f2e9c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eaea00640 1 -- 172.21.15.28:0/247024527 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c003520 con 0x7f2e9c0047c0 2024-09-13T17:18:03.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.550+0000 7f2eb7e00640 1 -- 172.21.15.28:0/247024527 reap_dead start 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.551+0000 7f2eaea00640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 0x7f2e9c004c40 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8159f20 tx=0x7f2e9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4613524424d7e35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.551+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0005ad0 con 0x7f2e9c0047c0 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.551+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea002d070 con 0x7f2e9c0047c0 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.551+0000 7f2eb6a00640 1 -- 172.21.15.28:0/247024527 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0005e90 con 0x7f2e9c0047c0 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 -- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 msgr2=0x7f2e9c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 0x7f2e9c004c40 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8159f20 tx=0x7f2e9c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 -- 172.21.15.28:0/247024527 shutdown_connections 2024-09-13T17:18:03.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 --2- 172.21.15.28:0/247024527 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e9c0047c0 0x7f2e9c004c40 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 -- 172.21.15.28:0/247024527 >> 172.21.15.28:0/247024527 conn(0x7f2eb8069e80 msgr2=0x7f2eb806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 -- 172.21.15.28:0/247024527 shutdown_connections 2024-09-13T17:18:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.552+0000 7f2ebe800640 1 -- 172.21.15.28:0/247024527 wait complete. 2024-09-13T17:18:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.554+0000 7f2ebe800640 1 Processor -- start 2024-09-13T17:18:03.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.554+0000 7f2ebe800640 1 -- start start 2024-09-13T17:18:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c0047c0 0x7f2eb8143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2eb8140f40 0x7f2eb8142580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81420f0 con 0x7f2eb8142ac0 2024-09-13T17:18:03.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81414e0 con 0x7f2e9c0047c0 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2ebe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb81417c0 con 0x7f2eb8140f40 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33558/0 (socket says 172.21.15.28:33558) 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 -- 172.21.15.28:0/3431321974 learned_addr learned my addr 172.21.15.28:0/3431321974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7400640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c0047c0 0x7f2eb8143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eaea00640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2eb8140f40 0x7f2eb8142580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 -- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2eb8140f40 msgr2=0x7f2eb8142580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2eb8140f40 0x7f2eb8142580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 -- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c0047c0 msgr2=0x7f2eb8143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c0047c0 0x7f2eb8143900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.555+0000 7f2eb7e00640 1 -- 172.21.15.28:0/3431321974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb8141aa0 con 0x7f2eb8142ac0 2024-09-13T17:18:03.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2eb7e00640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806f6d0 tx=0x7f2eb00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb000e070 con 0x7f2eb8142ac0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0011e10 con 0x7f2eb8142ac0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0003990 con 0x7f2eb8142ac0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb8141d80 con 0x7f2eb8142ac0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.556+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb8140910 con 0x7f2eb8142ac0 2024-09-13T17:18:03.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.557+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb8141900 con 0x7f2eb8142ac0 2024-09-13T17:18:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.558+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2eb0003af0 con 0x7f2eb8142ac0 2024-09-13T17:18:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.559+0000 7f2eafe00640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 0x7f2e9406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.559+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2eb0089a00 con 0x7f2eb8142ac0 2024-09-13T17:18:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.559+0000 7f2eb7400640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 0x7f2e9406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.560+0000 7f2eb7400640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 0x7f2e9406d0c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8001760 tx=0x7f2ea8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.563+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2eb8141900 con 0x7f2eb8142ac0 2024-09-13T17:18:03.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.742+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2eb8141900 con 0x7f2eb8142ac0 2024-09-13T17:18:03.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.743+0000 7f2eafe00640 1 -- 172.21.15.28:0/3431321974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16620 (secure 0 0 0) 0x7f2eb8141900 con 0x7f2eb8142ac0 2024-09-13T17:18:03.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":141,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:53.123298+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.16","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 msgr2=0x7f2e9406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 0x7f2e9406d0c0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8001760 tx=0x7f2ea8009040 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 msgr2=0x7f2eb8144430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806f6d0 tx=0x7f2eb00115a0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 shutdown_connections 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e9406ac00 0x7f2e9406d0c0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2eb8142ac0 0x7f2eb8144430 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2eb8140f40 0x7f2eb8142580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 --2- 172.21.15.28:0/3431321974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e9c0047c0 0x7f2eb8143900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 >> 172.21.15.28:0/3431321974 conn(0x7f2eb8069e80 msgr2=0x7f2eb8157540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 shutdown_connections 2024-09-13T17:18:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.745+0000 7f2ebe800640 1 -- 172.21.15.28:0/3431321974 wait complete. 2024-09-13T17:18:03.762 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:17:19.853908+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:17:20.955756+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:18:03.762 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:18:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.920+0000 7f76a7200640 1 Processor -- start 2024-09-13T17:18:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.920+0000 7f76a7200640 1 -- start start 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f76a7200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76a015a780 0x7f76a0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f76a7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a006c660 con 0x7f76a015ab60 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f76a7200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a006c8b0 con 0x7f76a006c250 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f76a7200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a006cb90 con 0x7f76a015a780 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f7697400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76a015ab60 0x7f76a006bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33186/0 (socket says 172.21.15.28:33186) 2024-09-13T17:18:03.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.921+0000 7f7697400640 1 -- 172.21.15.28:0/2285595725 learned_addr learned my addr 172.21.15.28:0/2285595725 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769fe00640 1 --2- 172.21.15.28:0/2285595725 >> v2:172.21.15.189:3300/0 conn(0x7f76a015a780 0x7f76a0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769f400640 1 -- 172.21.15.28:0/2285595725 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197835909 0 0) 0x7f76a006c660 con 0x7f76a015ab60 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769f400640 1 -- 172.21.15.28:0/2285595725 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7684003520 con 0x7f76a015ab60 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769fe00640 1 -- 172.21.15.28:0/2285595725 >> v1:172.21.15.189:6789/0 conn(0x7f76a006c250 legacy=0x7f76a00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769fe00640 1 -- 172.21.15.28:0/2285595725 >> v1:172.21.15.28:6789/0 conn(0x7f76a015ab60 legacy=0x7f76a006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.922+0000 7f769fe00640 1 -- 172.21.15.28:0/2285595725 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a00cc980 con 0x7f76a015a780 2024-09-13T17:18:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.923+0000 7f769fe00640 1 --2- 172.21.15.28:0/2285595725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015a780 0x7f76a0070d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f76900097a0 tx=0x7f769002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33ae4ec822b3d61f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.923+0000 7f769f400640 1 -- 172.21.15.28:0/2285595725 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690032440 con 0x7f76a015a780 2024-09-13T17:18:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f769f400640 1 -- 172.21.15.28:0/2285595725 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76900325a0 con 0x7f76a015a780 2024-09-13T17:18:03.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f769f400640 1 -- 172.21.15.28:0/2285595725 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7690032930 con 0x7f76a015a780 2024-09-13T17:18:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 -- 172.21.15.28:0/2285595725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015a780 msgr2=0x7f76a0070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 --2- 172.21.15.28:0/2285595725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015a780 0x7f76a0070d80 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f76900097a0 tx=0x7f769002a230 comp rx=0 tx=0).stop 2024-09-13T17:18:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 -- 172.21.15.28:0/2285595725 shutdown_connections 2024-09-13T17:18:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 --2- 172.21.15.28:0/2285595725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015a780 0x7f76a0070d80 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 -- 172.21.15.28:0/2285595725 >> 172.21.15.28:0/2285595725 conn(0x7f76a00b0270 msgr2=0x7f76a00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 -- 172.21.15.28:0/2285595725 shutdown_connections 2024-09-13T17:18:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.924+0000 7f76a7200640 1 -- 172.21.15.28:0/2285595725 wait complete. 2024-09-13T17:18:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.926+0000 7f76a7200640 1 Processor -- start 2024-09-13T17:18:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.926+0000 7f76a7200640 1 -- start start 2024-09-13T17:18:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76a006c250 0x7f76a015c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 0x7f76a00c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a015c340 con 0x7f76a015a780 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0149d70 con 0x7f76a015ab60 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a015b730 con 0x7f76a006c250 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f7697400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f7697400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33960/0 (socket says 172.21.15.28:33960) 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f7697400640 1 -- 172.21.15.28:0/3053351262 learned_addr learned my addr 172.21.15.28:0/3053351262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f769fe00640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76a006c250 0x7f76a015c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.927+0000 7f76a5200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 0x7f76a00c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 -- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76a006c250 msgr2=0x7f76a015c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76a006c250 0x7f76a015c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 -- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 msgr2=0x7f76a00c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 0x7f76a00c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 -- 172.21.15.28:0/3053351262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7690009400 con 0x7f76a015a780 2024-09-13T17:18:03.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f76a5200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 0x7f76a00c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f7697400640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f768800c440 tx=0x7f768800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7688005ce0 con 0x7f76a015a780 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7688005e40 con 0x7f76a015a780 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76880072b0 con 0x7f76a015a780 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.928+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a015ba10 con 0x7f76a015a780 2024-09-13T17:18:03.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.929+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a015bfd0 con 0x7f76a015a780 2024-09-13T17:18:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.930+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a015c4f0 con 0x7f76a015a780 2024-09-13T17:18:03.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.931+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7688007410 con 0x7f76a015a780 2024-09-13T17:18:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.932+0000 7f769cc00640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 0x7f767c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.932+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76880888e0 con 0x7f76a015a780 2024-09-13T17:18:03.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.932+0000 7f769fe00640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 0x7f767c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:03.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.933+0000 7f769fe00640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 0x7f767c06d0c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7690034000 tx=0x7f769002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:03.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:03.936+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f76a015c4f0 con 0x7f76a015a780 2024-09-13T17:18:04.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.115+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76a015c4f0 con 0x7f76a015a780 2024-09-13T17:18:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.116+0000 7f769cc00640 1 -- 172.21.15.28:0/3053351262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f76a015c4f0 con 0x7f76a015a780 2024-09-13T17:18:04.117 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.118+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 msgr2=0x7f767c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.118+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 0x7f767c06d0c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f7690034000 tx=0x7f769002e000 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.118+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 msgr2=0x7f76a00ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.118+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f768800c440 tx=0x7f768800c900 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 shutdown_connections 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f767c06ac00 0x7f767c06d0c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76a015ab60 0x7f76a00c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76a015a780 0x7f76a00ca000 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 --2- 172.21.15.28:0/3053351262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76a006c250 0x7f76a015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 >> 172.21.15.28:0/3053351262 conn(0x7f76a00b0270 msgr2=0x7f76a00af600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:04.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 shutdown_connections 2024-09-13T17:18:04.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.119+0000 7f76a7200640 1 -- 172.21.15.28:0/3053351262 wait complete. 2024-09-13T17:18:04.135 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:18:04.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.293+0000 7f086d400640 1 Processor -- start 2024-09-13T17:18:04.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.293+0000 7f086d400640 1 -- start start 2024-09-13T17:18:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f086d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0868158cd0 0x7f08681590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f086d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f086806c110 0x7f08680b1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f086d400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08681595f0 con 0x7f086806ccd0 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f086d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08681598c0 con 0x7f086806c110 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f086d400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0868159ba0 con 0x7f0868158cd0 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f086806c110 0x7f08680b1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f086806c110 0x7f08680b1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33976/0 (socket says 172.21.15.28:33976) 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 -- 172.21.15.28:0/4118790165 learned_addr learned my addr 172.21.15.28:0/4118790165 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0866800640 1 --2- 172.21.15.28:0/4118790165 >> v2:172.21.15.189:3300/0 conn(0x7f0868158cd0 0x7f08681590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 -- 172.21.15.28:0/4118790165 >> v2:172.21.15.189:3300/0 conn(0x7f0868158cd0 msgr2=0x7f08681590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 --2- 172.21.15.28:0/4118790165 >> v2:172.21.15.189:3300/0 conn(0x7f0868158cd0 0x7f08681590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 -- 172.21.15.28:0/4118790165 >> v1:172.21.15.189:6789/0 conn(0x7f086806ccd0 legacy=0x7f086806ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:04.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.295+0000 7f0867200640 1 -- 172.21.15.28:0/4118790165 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0868159e80 con 0x7f086806c110 2024-09-13T17:18:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.296+0000 7f0867200640 1 --2- 172.21.15.28:0/4118790165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806c110 0x7f08680b1a90 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0868067f80 tx=0x7f086002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aba4f85908caa28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f0865e00640 1 -- 172.21.15.28:0/4118790165 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860031600 con 0x7f086806c110 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f0865e00640 1 -- 172.21.15.28:0/4118790165 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f086002d050 con 0x7f086806c110 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f0865e00640 1 -- 172.21.15.28:0/4118790165 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0860031980 con 0x7f086806c110 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 -- 172.21.15.28:0/4118790165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806c110 msgr2=0x7f08680b1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 --2- 172.21.15.28:0/4118790165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806c110 0x7f08680b1a90 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f0868067f80 tx=0x7f086002a660 comp rx=0 tx=0).stop 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 -- 172.21.15.28:0/4118790165 shutdown_connections 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 --2- 172.21.15.28:0/4118790165 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806c110 0x7f08680b1a90 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 --2- 172.21.15.28:0/4118790165 >> v2:172.21.15.189:3300/0 conn(0x7f0868158cd0 0x7f08681590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 -- 172.21.15.28:0/4118790165 >> 172.21.15.28:0/4118790165 conn(0x7f08680b0270 msgr2=0x7f08680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 -- 172.21.15.28:0/4118790165 shutdown_connections 2024-09-13T17:18:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.297+0000 7f086d400640 1 -- 172.21.15.28:0/4118790165 wait complete. 2024-09-13T17:18:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.299+0000 7f086d400640 1 Processor -- start 2024-09-13T17:18:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.299+0000 7f086d400640 1 -- start start 2024-09-13T17:18:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 0x7f086806b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0868158cd0 0x7f08680a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08680cbff0 con 0x7f086806ccd0 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08680cc2d0 con 0x7f0868158cd0 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f086d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08680cc5b0 con 0x7f086806c110 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f085fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f085fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33992/0 (socket says 172.21.15.28:33992) 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f085fe00640 1 -- 172.21.15.28:0/2983545876 learned_addr learned my addr 172.21.15.28:0/2983545876 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f0867200640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0868158cd0 0x7f08680a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.300+0000 7f0866800640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 0x7f086806b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 -- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 msgr2=0x7f086806b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 0x7f086806b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 -- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0868158cd0 msgr2=0x7f08680a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0868158cd0 0x7f08680a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 -- 172.21.15.28:0/2983545876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860009b30 con 0x7f086806ccd0 2024-09-13T17:18:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f0866800640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 0x7f086806b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085fe00640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f085000c430 tx=0x7f085000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850011790 con 0x7f086806ccd0 2024-09-13T17:18:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08500118f0 con 0x7f086806ccd0 2024-09-13T17:18:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0850011ce0 con 0x7f086806ccd0 2024-09-13T17:18:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08680cc890 con 0x7f086806ccd0 2024-09-13T17:18:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.301+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08680cbc10 con 0x7f086806ccd0 2024-09-13T17:18:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.302+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08680ccdb0 con 0x7f086806ccd0 2024-09-13T17:18:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.303+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f085001c460 con 0x7f086806ccd0 2024-09-13T17:18:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.304+0000 7f085ea00640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 0x7f084406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.304+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0850089310 con 0x7f086806ccd0 2024-09-13T17:18:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.305+0000 7f0866800640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 0x7f084406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.305+0000 7f0866800640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 0x7f084406d0c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0858006fd0 tx=0x7f0858008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.308+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f08680ccdb0 con 0x7f086806ccd0 2024-09-13T17:18:04.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.488+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08680ccdb0 con 0x7f086806ccd0 2024-09-13T17:18:04.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.489+0000 7f085ea00640 1 -- 172.21.15.28:0/2983545876 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08680ccdb0 con 0x7f086806ccd0 2024-09-13T17:18:04.490 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:18:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 msgr2=0x7f084406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 0x7f084406d0c0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f0858006fd0 tx=0x7f0858008040 comp rx=0 tx=0).stop 2024-09-13T17:18:04.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 msgr2=0x7f086814db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f085000c430 tx=0x7f085000c8f0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 shutdown_connections 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f084406ac00 0x7f084406d0c0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0868158cd0 0x7f08680a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f086806ccd0 0x7f086814db90 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 --2- 172.21.15.28:0/2983545876 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f086806c110 0x7f086806b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 >> 172.21.15.28:0/2983545876 conn(0x7f08680b0270 msgr2=0x7f08680aefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 shutdown_connections 2024-09-13T17:18:04.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.491+0000 7f086d400640 1 -- 172.21.15.28:0/2983545876 wait complete. 2024-09-13T17:18:04.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:18:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.666+0000 7f8e93600640 1 Processor -- start 2024-09-13T17:18:04.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.666+0000 7f8e93600640 1 -- start start 2024-09-13T17:18:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.667+0000 7f8e93600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e8c070430 0x7f8e8c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.667+0000 7f8e93600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c06f3f0 0x7f8e8c15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.667+0000 7f8e93600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8c15ab60 con 0x7f8e8c159770 2024-09-13T17:18:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.667+0000 7f8e93600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8c06f920 con 0x7f8e8c070430 2024-09-13T17:18:04.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.667+0000 7f8e93600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c15b290 con 0x7f8e8c06f3f0 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e8c070430 0x7f8e8c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e91600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c06f3f0 0x7f8e8c15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e8c070430 0x7f8e8c15bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:49844/0 (socket says 172.21.15.28:49844) 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 -- 172.21.15.28:0/3651103660 learned_addr learned my addr 172.21.15.28:0/3651103660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e90c00640 1 --1- 172.21.15.28:0/3651103660 >> v1:172.21.15.189:6789/0 conn(0x7f8e8c159770 0x7f8e8c06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:58256/0 (socket says 172.21.15.28:58256) 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 -- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c06f3f0 msgr2=0x7f8e8c15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:04.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 --2- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c06f3f0 0x7f8e8c15c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 -- 172.21.15.28:0/3651103660 >> v1:172.21.15.189:6789/0 conn(0x7f8e8c159770 legacy=0x7f8e8c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.668+0000 7f8e83e00640 1 -- 172.21.15.28:0/3651103660 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8c15b570 con 0x7f8e8c070430 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.669+0000 7f8e83e00640 1 --2- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c070430 0x7f8e8c15bd70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009870 tx=0x7f8e7402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=979c961674134d10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.669+0000 7f8e8be00640 1 -- 172.21.15.28:0/3651103660 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74032440 con 0x7f8e8c070430 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.669+0000 7f8e8be00640 1 -- 172.21.15.28:0/3651103660 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e740325a0 con 0x7f8e8c070430 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e8be00640 1 -- 172.21.15.28:0/3651103660 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74032930 con 0x7f8e8c070430 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 -- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c070430 msgr2=0x7f8e8c15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 --2- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c070430 0x7f8e8c15bd70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8e74009870 tx=0x7f8e7402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 -- 172.21.15.28:0/3651103660 shutdown_connections 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 --2- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c06f3f0 0x7f8e8c15c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 --2- 172.21.15.28:0/3651103660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c070430 0x7f8e8c15bd70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 -- 172.21.15.28:0/3651103660 >> 172.21.15.28:0/3651103660 conn(0x7f8e8c0b0270 msgr2=0x7f8e8c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:04.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 -- 172.21.15.28:0/3651103660 shutdown_connections 2024-09-13T17:18:04.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.670+0000 7f8e93600640 1 -- 172.21.15.28:0/3651103660 wait complete. 2024-09-13T17:18:04.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.672+0000 7f8e93600640 1 Processor -- start 2024-09-13T17:18:04.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.672+0000 7f8e93600640 1 -- start start 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c06f3f0 0x7f8e8c152c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c159770 0x7f8e8c0acc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1e1be0 con 0x7f8e8c070430 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1e0f60 con 0x7f8e8c06f3f0 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e93600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8c1e1240 con 0x7f8e8c159770 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34002/0 (socket says 172.21.15.28:34002) 2024-09-13T17:18:04.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e83e00640 1 -- 172.21.15.28:0/771684851 learned_addr learned my addr 172.21.15.28:0/771684851 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e90c00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c06f3f0 0x7f8e8c152c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e91600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c159770 0x7f8e8c0acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e83e00640 1 -- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c159770 msgr2=0x7f8e8c0acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.673+0000 7f8e83e00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c159770 0x7f8e8c0acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e83e00640 1 -- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c06f3f0 msgr2=0x7f8e8c152c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e83e00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c06f3f0 0x7f8e8c152c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e83e00640 1 -- 172.21.15.28:0/771684851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e740094d0 con 0x7f8e8c070430 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e83e00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f8e74000c00 tx=0x7f8e740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74032440 con 0x7f8e8c070430 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e74030980 con 0x7f8e8c070430 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e74030d30 con 0x7f8e8c070430 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8c1e1520 con 0x7f8e8c070430 2024-09-13T17:18:04.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.674+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e8c157600 con 0x7f8e8c070430 2024-09-13T17:18:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.675+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8c157790 con 0x7f8e8c070430 2024-09-13T17:18:04.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.676+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e74039040 con 0x7f8e8c070430 2024-09-13T17:18:04.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.677+0000 7f8e89600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 0x7f8e6806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:04.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.677+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8e740a9b40 con 0x7f8e8c070430 2024-09-13T17:18:04.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.677+0000 7f8e90c00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 0x7f8e6806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:04.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.678+0000 7f8e90c00640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 0x7f8e6806d0f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c000f60 tx=0x7f8e7c000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:04.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.681+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e74074d80 con 0x7f8e8c070430 2024-09-13T17:18:04.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.859+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8e8c1e1d20 con 0x7f8e8c070430 2024-09-13T17:18:04.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.859+0000 7f8e89600640 1 -- 172.21.15.28:0/771684851 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e8c1e1d20 con 0x7f8e8c070430 2024-09-13T17:18:04.860 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:18:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 msgr2=0x7f8e6806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 0x7f8e6806d0f0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c000f60 tx=0x7f8e7c000ed0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 msgr2=0x7f8e8c154190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f8e74000c00 tx=0x7f8e740046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 shutdown_connections 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e6806ac30 0x7f8e6806d0f0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e8c159770 0x7f8e8c0acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e8c070430 0x7f8e8c154190 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 --2- 172.21.15.28:0/771684851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e8c06f3f0 0x7f8e8c152c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 >> 172.21.15.28:0/771684851 conn(0x7f8e8c0b0270 msgr2=0x7f8e8c0ae210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 shutdown_connections 2024-09-13T17:18:04.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:04.861+0000 7f8e93600640 1 -- 172.21.15.28:0/771684851 wait complete. 2024-09-13T17:18:04.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.032+0000 7f7947200640 1 Processor -- start 2024-09-13T17:18:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.032+0000 7f7947200640 1 -- start start 2024-09-13T17:18:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.033+0000 7f7947200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f794006f970 0x7f794015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.033+0000 7f7947200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7940159d90 con 0x7f79401592a0 2024-09-13T17:18:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f7947200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79400b2040 con 0x7f794006c2d0 2024-09-13T17:18:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f7947200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794015b0b0 con 0x7f794006f970 2024-09-13T17:18:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f793fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f79401592a0 0x7f7940159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33190/0 (socket says 172.21.15.28:33190) 2024-09-13T17:18:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f793fe00640 1 -- 172.21.15.28:0/4077556521 learned_addr learned my addr 172.21.15.28:0/4077556521 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f7937e00640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f794006f970 0x7f794015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2091836970 0 0) 0x7f7940159d90 con 0x7f79401592a0 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.034+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7924003520 con 0x7f79401592a0 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 920123769 0 0) 0x7f7924003520 con 0x7f79401592a0 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f794006f970 msgr2=0x7f794015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f794006f970 0x7f794015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 >> v1:172.21.15.189:6789/0 conn(0x7f794006c2d0 legacy=0x7f79400b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7940159ff0 con 0x7f79401592a0 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7930001690 con 0x7f79401592a0 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 >> v1:172.21.15.28:6789/0 conn(0x7f79401592a0 legacy=0x7f7940159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79240039a0 0x7f7924003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7924004350 0x7f7924004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f792400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794015b0b0 con 0x7f7924004fd0 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792400bef0 con 0x7f79240039a0 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f792400e2c0 con 0x7f7924004350 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7937e00640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f794006f970 0x7f794015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f792400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f793fe00640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79240039a0 0x7f7924003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7937e00640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7924004350 0x7f7924004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 -- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7924004350 msgr2=0x7f7924004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7924004350 0x7f7924004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 -- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79240039a0 msgr2=0x7f7924003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79240039a0 0x7f7924003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.035+0000 7f7945200640 1 -- 172.21.15.28:0/4077556521 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924003520 con 0x7f7924004fd0 2024-09-13T17:18:05.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.036+0000 7f7945200640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f792400b980 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f7938001f60 tx=0x7f79240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de4367c2bda3846 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7938031600 con 0x7f7924004fd0 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793802d050 con 0x7f7924004fd0 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f793f400640 1 -- 172.21.15.28:0/4077556521 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79380319c0 con 0x7f7924004fd0 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 -- 172.21.15.28:0/4077556521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 msgr2=0x7f792400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 --2- 172.21.15.28:0/4077556521 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f792400b980 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f7938001f60 tx=0x7f79240057f0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7945200640 1 -- 172.21.15.28:0/4077556521 reap_dead start 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 -- 172.21.15.28:0/4077556521 shutdown_connections 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 -- 172.21.15.28:0/4077556521 >> 172.21.15.28:0/4077556521 conn(0x7f7940069e80 msgr2=0x7f794006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 -- 172.21.15.28:0/4077556521 shutdown_connections 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.037+0000 7f7947200640 1 -- 172.21.15.28:0/4077556521 wait complete. 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.039+0000 7f7947200640 1 Processor -- start 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.039+0000 7f7947200640 1 -- start start 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 0x7f79401ed600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 0x7f794013f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79400cb900 con 0x7f7924004fd0 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794013e0a0 con 0x7f794013f390 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f7947200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794013ef00 con 0x7f7940143bf0 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f793fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f793fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34022/0 (socket says 172.21.15.28:34022) 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.040+0000 7f793fe00640 1 -- 172.21.15.28:0/126239499 learned_addr learned my addr 172.21.15.28:0/126239499 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7937e00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 0x7f79401ed600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7945200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 0x7f794013f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 -- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 msgr2=0x7f79401ed600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 0x7f79401ed600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 -- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 msgr2=0x7f794013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 0x7f794013f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 -- 172.21.15.28:0/126239499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794013e2f0 con 0x7f7924004fd0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7937e00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 0x7f79401ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7945200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 0x7f794013f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793fe00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f79300072f0 tx=0x7f79300077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793000f960 con 0x7f7924004fd0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793000fae0 con 0x7f7924004fd0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793000fe90 con 0x7f7924004fd0 2024-09-13T17:18:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794013e5d0 con 0x7f7924004fd0 2024-09-13T17:18:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.041+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f794013eb90 con 0x7f7924004fd0 2024-09-13T17:18:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.042+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f794015ab70 con 0x7f7924004fd0 2024-09-13T17:18:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.043+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f793001d360 con 0x7f7924004fd0 2024-09-13T17:18:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.045+0000 7f793cc00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 0x7f791c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.045+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7930088e80 con 0x7f7924004fd0 2024-09-13T17:18:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.045+0000 7f7937e00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 0x7f791c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.046+0000 7f7937e00640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 0x7f791c06d0c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f794014d0b0 tx=0x7f7928008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.049+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7930053ef0 con 0x7f7924004fd0 2024-09-13T17:18:05.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.267+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7940156ae0 con 0x7f7924004fd0 2024-09-13T17:18:05.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.268+0000 7f793cc00640 1 -- 172.21.15.28:0/126239499 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7f793001a070 con 0x7f7924004fd0 2024-09-13T17:18:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:05.270 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:18:05.270 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 msgr2=0x7f791c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 0x7f791c06d0c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f794014d0b0 tx=0x7f7928008040 comp rx=0 tx=0).stop 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 msgr2=0x7f7940141590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f79300072f0 tx=0x7f79300077b0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 shutdown_connections 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f791c06ac00 0x7f791c06d0c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f794013f390 0x7f794013f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7940143bf0 0x7f79401ed600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 --2- 172.21.15.28:0/126239499 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7924004fd0 0x7f7940141590 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 >> 172.21.15.28:0/126239499 conn(0x7f7940069e80 msgr2=0x7f79400b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 shutdown_connections 2024-09-13T17:18:05.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.270+0000 7f7947200640 1 -- 172.21.15.28:0/126239499 wait complete. 2024-09-13T17:18:05.288 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.442+0000 7fe360800640 1 Processor -- start 2024-09-13T17:18:05.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.442+0000 7fe360800640 1 -- start start 2024-09-13T17:18:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe360800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3581592a0 0x7fe35815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe360800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe35806f970 0x7fe358159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe360800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3580b2040 con 0x7fe35806c2d0 2024-09-13T17:18:05.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe360800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35815b0b0 con 0x7fe3581592a0 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe360800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe358159ff0 con 0x7fe35806f970 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3581592a0 0x7fe35815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe35806c2d0 0x7fe3580b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33198/0 (socket says 172.21.15.28:33198) 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35e800640 1 -- 172.21.15.28:0/746780 learned_addr learned my addr 172.21.15.28:0/746780 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35d400640 1 --2- 172.21.15.28:0/746780 >> v2:172.21.15.189:3300/0 conn(0x7fe35806f970 0x7fe358159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35de00640 1 -- 172.21.15.28:0/746780 >> v2:172.21.15.189:3300/0 conn(0x7fe35806f970 msgr2=0x7fe358159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35de00640 1 --2- 172.21.15.28:0/746780 >> v2:172.21.15.189:3300/0 conn(0x7fe35806f970 0x7fe358159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35de00640 1 -- 172.21.15.28:0/746780 >> v1:172.21.15.28:6789/0 conn(0x7fe35806c2d0 legacy=0x7fe3580b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35de00640 1 -- 172.21.15.28:0/746780 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35815a2d0 con 0x7fe3581592a0 2024-09-13T17:18:05.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.444+0000 7fe35ca00640 1 -- 172.21.15.28:0/746780 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003873669 0 0) 0x7fe3580b2040 con 0x7fe35806c2d0 2024-09-13T17:18:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.445+0000 7fe35de00640 1 --2- 172.21.15.28:0/746780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe35815ab70 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fe35806ac10 tx=0x7fe34c02c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f2678dcf0fef29 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe35ca00640 1 -- 172.21.15.28:0/746780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c032770 con 0x7fe3581592a0 2024-09-13T17:18:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe35ca00640 1 -- 172.21.15.28:0/746780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34c0328d0 con 0x7fe3581592a0 2024-09-13T17:18:05.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe35ca00640 1 -- 172.21.15.28:0/746780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c032c60 con 0x7fe3581592a0 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 -- 172.21.15.28:0/746780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 msgr2=0x7fe35815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 --2- 172.21.15.28:0/746780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe35815ab70 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fe35806ac10 tx=0x7fe34c02c870 comp rx=0 tx=0).stop 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 -- 172.21.15.28:0/746780 shutdown_connections 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 --2- 172.21.15.28:0/746780 >> v2:172.21.15.189:3300/0 conn(0x7fe35806f970 0x7fe358159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 --2- 172.21.15.28:0/746780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe35815ab70 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 -- 172.21.15.28:0/746780 >> 172.21.15.28:0/746780 conn(0x7fe358069e80 msgr2=0x7fe35806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 -- 172.21.15.28:0/746780 shutdown_connections 2024-09-13T17:18:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.446+0000 7fe360800640 1 -- 172.21.15.28:0/746780 wait complete. 2024-09-13T17:18:05.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.448+0000 7fe360800640 1 Processor -- start 2024-09-13T17:18:05.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.448+0000 7fe360800640 1 -- start start 2024-09-13T17:18:05.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe35806c2d0 0x7fe3581f1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35806f970 0x7fe35814b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35814ef90 con 0x7fe3581592a0 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35814b700 con 0x7fe35806f970 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe360800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe358151210 con 0x7fe35806c2d0 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe35e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe35e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34042/0 (socket says 172.21.15.28:34042) 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.449+0000 7fe35e800640 1 -- 172.21.15.28:0/781757704 learned_addr learned my addr 172.21.15.28:0/781757704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35de00640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe35806c2d0 0x7fe3581f1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35d400640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35806f970 0x7fe35814b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 -- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe35806c2d0 msgr2=0x7fe3581f1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe35806c2d0 0x7fe3581f1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 -- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35806f970 msgr2=0x7fe35814b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35806f970 0x7fe35814b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 -- 172.21.15.28:0/781757704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34c00b6a0 con 0x7fe3581592a0 2024-09-13T17:18:05.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe35e800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c7a0 tx=0x7fe35400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400a640 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35400a7a0 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe35400abd0 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3581f2400 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.450+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3581f29c0 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.451+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35814f160 con 0x7fe3581592a0 2024-09-13T17:18:05.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.452+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe35400ad30 con 0x7fe3581592a0 2024-09-13T17:18:05.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.453+0000 7fe352000640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 0x7fe33c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.453+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe354088930 con 0x7fe3581592a0 2024-09-13T17:18:05.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.454+0000 7fe35de00640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 0x7fe33c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.454+0000 7fe35de00640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 0x7fe33c06d0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe34c037000 tx=0x7fe34c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.457+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe35814f160 con 0x7fe3581592a0 2024-09-13T17:18:05.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.637+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe35815ab70 con 0x7fe3581592a0 2024-09-13T17:18:05.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.638+0000 7fe352000640 1 -- 172.21.15.28:0/781757704 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fe354018070 con 0x7fe3581592a0 2024-09-13T17:18:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:05.641 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":141,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:53.123298+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.16","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 msgr2=0x7fe33c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 0x7fe33c06d0c0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe34c037000 tx=0x7fe34c032000 comp rx=0 tx=0).stop 2024-09-13T17:18:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 msgr2=0x7fe358150cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fe35400c7a0 tx=0x7fe35400cc60 comp rx=0 tx=0).stop 2024-09-13T17:18:05.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 shutdown_connections 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33c06ac00 0x7fe33c06d0c0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3581592a0 0x7fe358150cd0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35806f970 0x7fe35814b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 --2- 172.21.15.28:0/781757704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe35806c2d0 0x7fe3581f1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 >> 172.21.15.28:0/781757704 conn(0x7fe358069e80 msgr2=0x7fe35815c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 shutdown_connections 2024-09-13T17:18:05.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.641+0000 7fe360800640 1 -- 172.21.15.28:0/781757704 wait complete. 2024-09-13T17:18:05.659 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:18:05.660 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.813+0000 7fbfaf000640 1 Processor -- start 2024-09-13T17:18:05.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.813+0000 7fbfaf000640 1 -- start start 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfaf000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbfa8154a90 0x7fbfa815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfaf000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 0x7fbfa80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfaf000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfa815bb10 con 0x7fbfa80b0980 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfaf000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa80b0d80 con 0x7fbfa8154a90 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfaf000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa806b900 con 0x7fbfa806b360 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfadc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbfa8154a90 0x7fbfa815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfadc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbfa8154a90 0x7fbfa815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34056/0 (socket says 172.21.15.28:34056) 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.814+0000 7fbfadc00640 1 -- 172.21.15.28:0/2047280429 learned_addr learned my addr 172.21.15.28:0/2047280429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfad200640 1 --2- 172.21.15.28:0/2047280429 >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 0x7fbfa80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfadc00640 1 -- 172.21.15.28:0/2047280429 >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 msgr2=0x7fbfa80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:05.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfadc00640 1 --2- 172.21.15.28:0/2047280429 >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 0x7fbfa80b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfadc00640 1 -- 172.21.15.28:0/2047280429 >> v1:172.21.15.189:6789/0 conn(0x7fbfa80b0980 legacy=0x7fbfa80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfadc00640 1 -- 172.21.15.28:0/2047280429 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa806bbe0 con 0x7fbfa8154a90 2024-09-13T17:18:05.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.815+0000 7fbfad200640 1 --2- 172.21.15.28:0/2047280429 >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 0x7fbfa80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.816+0000 7fbfadc00640 1 --2- 172.21.15.28:0/2047280429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8154a90 0x7fbfa815b5d0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fbfa813dab0 tx=0x7fbf9c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2123dc9cee428007 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.816+0000 7fbfa7e00640 1 -- 172.21.15.28:0/2047280429 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c032440 con 0x7fbfa8154a90 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.816+0000 7fbfa7e00640 1 -- 172.21.15.28:0/2047280429 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9c0325a0 con 0x7fbfa8154a90 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.816+0000 7fbfa7e00640 1 -- 172.21.15.28:0/2047280429 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9c032930 con 0x7fbfa8154a90 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 -- 172.21.15.28:0/2047280429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8154a90 msgr2=0x7fbfa815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 --2- 172.21.15.28:0/2047280429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8154a90 0x7fbfa815b5d0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fbfa813dab0 tx=0x7fbf9c02a270 comp rx=0 tx=0).stop 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 -- 172.21.15.28:0/2047280429 shutdown_connections 2024-09-13T17:18:05.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 --2- 172.21.15.28:0/2047280429 >> v2:172.21.15.189:3300/0 conn(0x7fbfa806b360 0x7fbfa80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 --2- 172.21.15.28:0/2047280429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa8154a90 0x7fbfa815b5d0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 -- 172.21.15.28:0/2047280429 >> 172.21.15.28:0/2047280429 conn(0x7fbfa8072e40 msgr2=0x7fbfa8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 -- 172.21.15.28:0/2047280429 shutdown_connections 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.817+0000 7fbfaf000640 1 -- 172.21.15.28:0/2047280429 wait complete. 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.819+0000 7fbfaf000640 1 Processor -- start 2024-09-13T17:18:05.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.819+0000 7fbfaf000640 1 -- start start 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa806b360 0x7fbfa8158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfa8154a90 0x7fbfa8149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa8147130 con 0x7fbfa80b0980 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa81557f0 con 0x7fbfa806b360 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfaf000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa813c780 con 0x7fbfa8154a90 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34070/0 (socket says 172.21.15.28:34070) 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 -- 172.21.15.28:0/1812585760 learned_addr learned my addr 172.21.15.28:0/1812585760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfadc00640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa806b360 0x7fbfa8158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfae600640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfa8154a90 0x7fbfa8149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 -- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfa8154a90 msgr2=0x7fbfa8149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfa8154a90 0x7fbfa8149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 -- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa806b360 msgr2=0x7fbfa8158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa806b360 0x7fbfa8158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.820+0000 7fbfad200640 1 -- 172.21.15.28:0/1812585760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0094d0 con 0x7fbfa80b0980 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfad200640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e7e0 tx=0x7fbf9400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94004030 con 0x7fbfa80b0980 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf94004190 con 0x7fbfa80b0980 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94004520 con 0x7fbfa80b0980 2024-09-13T17:18:05.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa813ca00 con 0x7fbfa80b0980 2024-09-13T17:18:05.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.821+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81551a0 con 0x7fbfa80b0980 2024-09-13T17:18:05.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.822+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa81559a0 con 0x7fbfa80b0980 2024-09-13T17:18:05.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.823+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf9400c6d0 con 0x7fbfa80b0980 2024-09-13T17:18:05.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.824+0000 7fbfa5600640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 0x7fbf8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:05.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.824+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbf94088fe0 con 0x7fbfa80b0980 2024-09-13T17:18:05.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.824+0000 7fbfadc00640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 0x7fbf8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:05.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.825+0000 7fbfadc00640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 0x7fbf8806d0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c034000 tx=0x7fbf9c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:05.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:05.828+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbfa81559a0 con 0x7fbfa80b0980 2024-09-13T17:18:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.046+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfa81559a0 con 0x7fbfa80b0980 2024-09-13T17:18:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.047+0000 7fbfa5600640 1 -- 172.21.15.28:0/1812585760 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fbfa81559a0 con 0x7fbfa80b0980 2024-09-13T17:18:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:06.049 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:18:06.049 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 msgr2=0x7fbf8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 0x7fbf8806d0c0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbf9c034000 tx=0x7fbf9c02e000 comp rx=0 tx=0).stop 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 msgr2=0x7fbfa813c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400e7e0 tx=0x7fbf9400eca0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 shutdown_connections 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf8806ac00 0x7fbf8806d0c0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfa8154a90 0x7fbfa8149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfa80b0980 0x7fbfa813c240 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 --2- 172.21.15.28:0/1812585760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfa806b360 0x7fbfa8158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 >> 172.21.15.28:0/1812585760 conn(0x7fbfa8072e40 msgr2=0x7fbfa806f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 shutdown_connections 2024-09-13T17:18:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.049+0000 7fbfaf000640 1 -- 172.21.15.28:0/1812585760 wait complete. 2024-09-13T17:18:06.067 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:06.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.220+0000 7fbac5c00640 1 Processor -- start 2024-09-13T17:18:06.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.220+0000 7fbac5c00640 1 -- start start 2024-09-13T17:18:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbac006b360 0x7fbac00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbac00b0980 0x7fbac00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00b0d80 con 0x7fbac0154a90 2024-09-13T17:18:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac006b900 con 0x7fbac00b0980 2024-09-13T17:18:06.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac006bbe0 con 0x7fbac006b360 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbac00b0980 0x7fbac00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbac00b0980 0x7fbac00af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34078/0 (socket says 172.21.15.28:34078) 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 -- 172.21.15.28:0/2262119524 learned_addr learned my addr 172.21.15.28:0/2262119524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbab7400640 1 --2- 172.21.15.28:0/2262119524 >> v2:172.21.15.189:3300/0 conn(0x7fbac006b360 0x7fbac00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 -- 172.21.15.28:0/2262119524 >> v2:172.21.15.189:3300/0 conn(0x7fbac006b360 msgr2=0x7fbac00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 --2- 172.21.15.28:0/2262119524 >> v2:172.21.15.189:3300/0 conn(0x7fbac006b360 0x7fbac00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 -- 172.21.15.28:0/2262119524 >> v1:172.21.15.189:6789/0 conn(0x7fbac0154a90 legacy=0x7fbac015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:06.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.222+0000 7fbac5200640 1 -- 172.21.15.28:0/2262119524 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac006bec0 con 0x7fbac00b0980 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.223+0000 7fbac5200640 1 --2- 172.21.15.28:0/2262119524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00af4b0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbac013dab0 tx=0x7fbab802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c39995b625f7a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbabf400640 1 -- 172.21.15.28:0/2262119524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8031600 con 0x7fbac00b0980 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbabf400640 1 -- 172.21.15.28:0/2262119524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab802d050 con 0x7fbac00b0980 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbabf400640 1 -- 172.21.15.28:0/2262119524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab8031980 con 0x7fbac00b0980 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 -- 172.21.15.28:0/2262119524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 msgr2=0x7fbac00af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 --2- 172.21.15.28:0/2262119524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00af4b0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7fbac013dab0 tx=0x7fbab802a6f0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 -- 172.21.15.28:0/2262119524 shutdown_connections 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 --2- 172.21.15.28:0/2262119524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00af4b0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 --2- 172.21.15.28:0/2262119524 >> v2:172.21.15.189:3300/0 conn(0x7fbac006b360 0x7fbac00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 -- 172.21.15.28:0/2262119524 >> 172.21.15.28:0/2262119524 conn(0x7fbac0072e40 msgr2=0x7fbac0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 -- 172.21.15.28:0/2262119524 shutdown_connections 2024-09-13T17:18:06.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.224+0000 7fbac5c00640 1 -- 172.21.15.28:0/2262119524 wait complete. 2024-09-13T17:18:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.226+0000 7fbac5c00640 1 Processor -- start 2024-09-13T17:18:06.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.226+0000 7fbac5c00640 1 -- start start 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac006b360 0x7fbac0139700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbac0154a90 0x7fbac0156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac0139d50 con 0x7fbac00b0980 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac013a030 con 0x7fbac006b360 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac013a310 con 0x7fbac0154a90 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbab7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbab7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34094/0 (socket says 172.21.15.28:34094) 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbab7400640 1 -- 172.21.15.28:0/471677568 learned_addr learned my addr 172.21.15.28:0/471677568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbabfe00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac006b360 0x7fbac0139700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.227+0000 7fbac5200640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbac0154a90 0x7fbac0156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 -- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbac0154a90 msgr2=0x7fbac0156670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbac0154a90 0x7fbac0156670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 -- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac006b360 msgr2=0x7fbac0139700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac006b360 0x7fbac0139700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 -- 172.21.15.28:0/471677568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8009b30 con 0x7fbac00b0980 2024-09-13T17:18:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbab7400640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800d800 tx=0x7fbaa800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa800b6c0 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa800b820 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa800bbb0 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.228+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac013a5f0 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.229+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac0154370 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.230+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac00a8c30 con 0x7fbac00b0980 2024-09-13T17:18:06.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.231+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbaa800bd10 con 0x7fbac00b0980 2024-09-13T17:18:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.231+0000 7fbabcc00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 0x7fba9c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.231+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fbaa8089820 con 0x7fbac00b0980 2024-09-13T17:18:06.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.232+0000 7fbabfe00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 0x7fba9c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.233+0000 7fbabfe00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 0x7fba9c06d0f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006fd0 tx=0x7fbab0009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.236+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbaa80549d0 con 0x7fbac00b0980 2024-09-13T17:18:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.450+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac01545f0 con 0x7fbac00b0980 2024-09-13T17:18:06.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.450+0000 7fbabcc00640 1 -- 172.21.15.28:0/471677568 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+4748 (secure 0 0 0) 0x7fbaa8057ff0 con 0x7fbac00b0980 2024-09-13T17:18:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:06.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":122,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:17:21.964025+0000","modified":"2024-09-13T17:17:24.982429+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8904},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8904":{"gid":8904,"name":"c","rank":0,"incarnation":118,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/4121307324","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":4121307324},{"type":"v1","addr":"172.21.15.28:6835","nonce":4121307324}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2024-09-13T17:18:06.453 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 122 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.452+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 msgr2=0x7fba9c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.452+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 0x7fba9c06d0f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fbab0006fd0 tx=0x7fbab0009060 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.452+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 msgr2=0x7fbac00a9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.452+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7fbaa800d800 tx=0x7fbaa800dcc0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 shutdown_connections 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fba9c06ac30 0x7fba9c06d0f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbac0154a90 0x7fbac0156670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbac00b0980 0x7fbac00a9050 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 --2- 172.21.15.28:0/471677568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbac006b360 0x7fbac0139700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 >> 172.21.15.28:0/471677568 conn(0x7fbac0072e40 msgr2=0x7fbac015b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 shutdown_connections 2024-09-13T17:18:06.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.453+0000 7fbac5c00640 1 -- 172.21.15.28:0/471677568 wait complete. 2024-09-13T17:18:06.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:06.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.624+0000 7fa1dc000640 1 Processor -- start 2024-09-13T17:18:06.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.624+0000 7fa1dc000640 1 -- start start 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1dc000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 0x7fa1d415d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1dc000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40b3970 con 0x7fa1d40b3160 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1dc000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4159430 con 0x7fa1d406fb50 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1dc000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d4159230 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1d8c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa1d40b3160 0x7fa1d406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33206/0 (socket says 172.21.15.28:33206) 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1d8c00640 1 -- 172.21.15.28:0/4106969336 learned_addr learned my addr 172.21.15.28:0/4106969336 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:06.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1d9600640 1 --2- 172.21.15.28:0/4106969336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 0x7fa1d415d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.626+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558837424 0 0) 0x7fa1c0001110 con 0x7fa1d40b3160 2024-09-13T17:18:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.627+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc005730 con 0x7fa1d40b3160 2024-09-13T17:18:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.627+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1027261963 0 0) 0x7fa1bc005730 con 0x7fa1d40b3160 2024-09-13T17:18:06.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.627+0000 7fa1d9600640 1 -- 172.21.15.28:0/4106969336 >> v1:172.21.15.189:6789/0 conn(0x7fa1d406fb50 legacy=0x7fa1d406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.627+0000 7fa1d9600640 1 -- 172.21.15.28:0/4106969336 >> v1:172.21.15.28:6789/0 conn(0x7fa1d40b3160 legacy=0x7fa1d406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.627+0000 7fa1d9600640 1 -- 172.21.15.28:0/4106969336 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4159710 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.628+0000 7fa1d9600640 1 --2- 172.21.15.28:0/4106969336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 0x7fa1d415d340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80097d0 tx=0x7fa1c802a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a2c4f551d68c040 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.628+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0001800 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.628+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c802ab40 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.628+0000 7fa1d3e00640 1 -- 172.21.15.28:0/4106969336 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8032440 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 -- 172.21.15.28:0/4106969336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 msgr2=0x7fa1d415d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 --2- 172.21.15.28:0/4106969336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 0x7fa1d415d340 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fa1c80097d0 tx=0x7fa1c802a0b0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 -- 172.21.15.28:0/4106969336 shutdown_connections 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 --2- 172.21.15.28:0/4106969336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d40cdfa0 0x7fa1d415d340 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 -- 172.21.15.28:0/4106969336 >> 172.21.15.28:0/4106969336 conn(0x7fa1d4069e80 msgr2=0x7fa1d406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:06.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 -- 172.21.15.28:0/4106969336 shutdown_connections 2024-09-13T17:18:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.629+0000 7fa1dc000640 1 -- 172.21.15.28:0/4106969336 wait complete. 2024-09-13T17:18:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.630+0000 7fa1dc000640 1 Processor -- start 2024-09-13T17:18:06.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.631+0000 7fa1dc000640 1 -- start start 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.631+0000 7fa1dc000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d406fb50 0x7fa1d40b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.631+0000 7fa1dc000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.631+0000 7fa1dc000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d40cdfa0 0x7fa1d40b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1dc000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d41edf70 con 0x7fa1d40b3160 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1dc000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40a9fe0 con 0x7fa1d40cdfa0 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1dc000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d40a9460 con 0x7fa1d406fb50 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34096/0 (socket says 172.21.15.28:34096) 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 -- 172.21.15.28:0/609276597 learned_addr learned my addr 172.21.15.28:0/609276597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d9600640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d406fb50 0x7fa1d40b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1da000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d40cdfa0 0x7fa1d40b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 -- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d406fb50 msgr2=0x7fa1d40b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d406fb50 0x7fa1d40b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 -- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d40cdfa0 msgr2=0x7fa1d40b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d40cdfa0 0x7fa1d40b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.632+0000 7fa1d8c00640 1 -- 172.21.15.28:0/609276597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8009400 con 0x7fa1d40b3160 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1d8c00640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0001eb0 tx=0x7fa1c000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0005ad0 con 0x7fa1d40b3160 2024-09-13T17:18:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c0005c30 con 0x7fa1d40b3160 2024-09-13T17:18:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0007070 con 0x7fa1d40b3160 2024-09-13T17:18:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d40a9740 con 0x7fa1d40b3160 2024-09-13T17:18:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.633+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d40a9d00 con 0x7fa1d40b3160 2024-09-13T17:18:06.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.634+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d40cf260 con 0x7fa1d40b3160 2024-09-13T17:18:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.635+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa1c00071d0 con 0x7fa1d40b3160 2024-09-13T17:18:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.636+0000 7fa1d1600640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 0x7fa1b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.636+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa1c00887c0 con 0x7fa1d40b3160 2024-09-13T17:18:06.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.636+0000 7fa1d9600640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 0x7fa1b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:06.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.637+0000 7fa1d9600640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 0x7fa1b406d0c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8034000 tx=0x7fa1c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:06.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.640+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa1c0053970 con 0x7fa1d40b3160 2024-09-13T17:18:06.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.821+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1d40b3970 con 0x7fa1d40b3160 2024-09-13T17:18:06.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.822+0000 7fa1d1600640 1 -- 172.21.15.28:0/609276597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+16620 (secure 0 0 0) 0x7fa1d40b3970 con 0x7fa1d40b3160 2024-09-13T17:18:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:06.825 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":141,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:17:53.123298+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:17:19.853908+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"138","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2024-09-13T17:17:20.955756+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.c","mappings":[{"from":2,"to":0}]},{"pgid":"19.16","mappings":[{"from":2,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 msgr2=0x7fa1b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 0x7fa1b406d0c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8034000 tx=0x7fa1c802e000 comp rx=0 tx=0).stop 2024-09-13T17:18:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 msgr2=0x7fa1d41ebbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:06.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fa1c0001eb0 tx=0x7fa1c000cbe0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 shutdown_connections 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1b406ac00 0x7fa1b406d0c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d40cdfa0 0x7fa1d40b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d40b3160 0x7fa1d41ebbb0 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 --2- 172.21.15.28:0/609276597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1d406fb50 0x7fa1d40b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 >> 172.21.15.28:0/609276597 conn(0x7fa1d4069e80 msgr2=0x7fa1d415c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 shutdown_connections 2024-09-13T17:18:06.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:06.824+0000 7fa1dc000640 1 -- 172.21.15.28:0/609276597 wait complete. 2024-09-13T17:18:06.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:18:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.008+0000 7f6585e00640 1 Processor -- start 2024-09-13T17:18:07.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.008+0000 7f6585e00640 1 -- start start 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65800704a0 0x7f65800b5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65800b6090 con 0x7f6580070880 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65800ce170 con 0x7f65800cd8d0 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65800cde90 con 0x7f65800704a0 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6580070880 0x7f6580070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33220/0 (socket says 172.21.15.28:33220) 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6585400640 1 -- 172.21.15.28:0/228623691 learned_addr learned my addr 172.21.15.28:0/228623691 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:07.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.009+0000 7f6584a00640 1 --2- 172.21.15.28:0/228623691 >> v2:172.21.15.189:3300/0 conn(0x7f65800704a0 0x7f65800b5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.010+0000 7f657fe00640 1 -- 172.21.15.28:0/228623691 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223202586 0 0) 0x7f65800b6090 con 0x7f6580070880 2024-09-13T17:18:07.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.010+0000 7f657fe00640 1 -- 172.21.15.28:0/228623691 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6564003520 con 0x7f6580070880 2024-09-13T17:18:07.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.010+0000 7f6584a00640 1 -- 172.21.15.28:0/228623691 >> v1:172.21.15.189:6789/0 conn(0x7f65800cd8d0 legacy=0x7f65800cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.010+0000 7f6584a00640 1 -- 172.21.15.28:0/228623691 >> v1:172.21.15.28:6789/0 conn(0x7f6580070880 legacy=0x7f6580070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.010+0000 7f6584a00640 1 -- 172.21.15.28:0/228623691 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800d0e50 con 0x7f65800704a0 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.011+0000 7f6584a00640 1 --2- 172.21.15.28:0/228623691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65800704a0 0x7f65800b5b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f65700097a0 tx=0x7f657002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0fcff4f1feda1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.011+0000 7f657fe00640 1 -- 172.21.15.28:0/228623691 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570032440 con 0x7f65800704a0 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f657fe00640 1 -- 172.21.15.28:0/228623691 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65700325a0 con 0x7f65800704a0 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f657fe00640 1 -- 172.21.15.28:0/228623691 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570032930 con 0x7f65800704a0 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 -- 172.21.15.28:0/228623691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65800704a0 msgr2=0x7f65800b5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 --2- 172.21.15.28:0/228623691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65800704a0 0x7f65800b5b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f65700097a0 tx=0x7f657002a230 comp rx=0 tx=0).stop 2024-09-13T17:18:07.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 -- 172.21.15.28:0/228623691 shutdown_connections 2024-09-13T17:18:07.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 --2- 172.21.15.28:0/228623691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65800704a0 0x7f65800b5b50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 -- 172.21.15.28:0/228623691 >> 172.21.15.28:0/228623691 conn(0x7f65800d2870 msgr2=0x7f65800d2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:07.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 -- 172.21.15.28:0/228623691 shutdown_connections 2024-09-13T17:18:07.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.012+0000 7f6585e00640 1 -- 172.21.15.28:0/228623691 wait complete. 2024-09-13T17:18:07.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.014+0000 7f6585e00640 1 Processor -- start 2024-09-13T17:18:07.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.014+0000 7f6585e00640 1 -- start start 2024-09-13T17:18:07.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6580070880 0x7f658014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65800cd8d0 0x7f658014aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658014a7f0 con 0x7f65800704a0 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658014c8b0 con 0x7f6580070880 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6585e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658014b2d0 con 0x7f65800cd8d0 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34100/0 (socket says 172.21.15.28:34100) 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 -- 172.21.15.28:0/2503624035 learned_addr learned my addr 172.21.15.28:0/2503624035 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 -- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65800cd8d0 msgr2=0x7f658014aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65800cd8d0 0x7f658014aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 -- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6580070880 msgr2=0x7f658014c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6580070880 0x7f658014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.015+0000 7f6584a00640 1 -- 172.21.15.28:0/2503624035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6570009400 con 0x7f65800704a0 2024-09-13T17:18:07.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f6584a00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f6570035040 tx=0x7f65700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570006e10 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570032e90 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6570030be0 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658014b5b0 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.016+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658014bb70 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.017+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65800b5b50 con 0x7f65800704a0 2024-09-13T17:18:07.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.018+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6570039040 con 0x7f65800704a0 2024-09-13T17:18:07.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.019+0000 7f657d600640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.019+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f65700a9e10 con 0x7f65800704a0 2024-09-13T17:18:07.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.019+0000 7f6577e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.020+0000 7f6577e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6568000f60 tx=0x7f65680079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.023+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f65700ad030 con 0x7f65800704a0 2024-09-13T17:18:07.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.241+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f65800c96b0 con 0x7f65800704a0 2024-09-13T17:18:07.243 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:18:07.242+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:18:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.250+0000 7f657d600640 1 -- 172.21.15.28:0/2503624035 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v123) v1 ==== 161+0+0 (secure 0 0 0) 0x7f65700750c0 con 0x7f65800704a0 2024-09-13T17:18:07.252 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:18:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 msgr2=0x7f655c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6568000f60 tx=0x7f65680079b0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 msgr2=0x7f6580150a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f6570035040 tx=0x7f65700046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 shutdown_connections 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65800cd8d0 0x7f658014aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6580070880 0x7f658014c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 --2- 172.21.15.28:0/2503624035 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65800704a0 0x7f6580150a10 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.252+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 >> 172.21.15.28:0/2503624035 conn(0x7f65800d2870 msgr2=0x7f65800b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.253+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 shutdown_connections 2024-09-13T17:18:07.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.253+0000 7f6585e00640 1 -- 172.21.15.28:0/2503624035 wait complete. 2024-09-13T17:18:07.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:18:07.323 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:18:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.433+0000 7ff006000640 1 Processor -- start 2024-09-13T17:18:07.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.433+0000 7ff006000640 1 -- start start 2024-09-13T17:18:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7ff006000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7ff00015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7ff006000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00015ab60 con 0x7ff000159770 2024-09-13T17:18:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7ff006000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00006f920 con 0x7ff00006f3f0 2024-09-13T17:18:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7ff006000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00015b290 con 0x7ff000070430 2024-09-13T17:18:07.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7fefff400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff000159770 0x7ff00006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33240/0 (socket says 172.21.15.28:33240) 2024-09-13T17:18:07.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.434+0000 7fefff400640 1 -- 172.21.15.28:0/845397019 learned_addr learned my addr 172.21.15.28:0/845397019 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffea00640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7ff00015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376595517 0 0) 0x7feff0001110 con 0x7ff000159770 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe4003520 con 0x7ff000159770 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223949206 0 0) 0x7fefe4003520 con 0x7ff000159770 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 msgr2=0x7ff00015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7ff00015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 >> v1:172.21.15.189:6789/0 conn(0x7ff00006f3f0 legacy=0x7ff00015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00015b570 con 0x7ff000159770 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7feff0001800 con 0x7ff000159770 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 >> v1:172.21.15.28:6789/0 conn(0x7ff000159770 legacy=0x7ff00006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.435+0000 7feffe000640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7fefe40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffe000640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe4003ee0 0x7fefe40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffe000640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe4004d00 0x7fefe4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00015b290 con 0x7fefe4003ee0 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400bac0 con 0x7fefe4004d00 2024-09-13T17:18:07.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefe400bc80 con 0x7ff000070430 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7fefe40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffea00640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe4003ee0 0x7fefe40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefffe00640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe4004d00 0x7fefe4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 -- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe4004d00 msgr2=0x7fefe4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe4004d00 0x7fefe4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 -- 172.21.15.28:0/845397019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe4003ee0 msgr2=0x7fefe40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe4003ee0 0x7fefe40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefff400640 1 -- 172.21.15.28:0/845397019 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4003520 con 0x7ff000070430 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7fefffe00640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefe4004d00 0x7fefe4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:07.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.436+0000 7feffea00640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fefe4003ee0 0x7fefe40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.437+0000 7fefff400640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7fefe40039a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feff000c110 tx=0x7fefe40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4203a36ef1b013c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0005ad0 con 0x7ff000070430 2024-09-13T17:18:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff002d070 con 0x7ff000070430 2024-09-13T17:18:07.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7feffe000640 1 -- 172.21.15.28:0/845397019 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0007070 con 0x7ff000070430 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 -- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 msgr2=0x7fefe40039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 --2- 172.21.15.28:0/845397019 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000070430 0x7fefe40039a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7feff000c110 tx=0x7fefe40057f0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7fefffe00640 1 -- 172.21.15.28:0/845397019 reap_dead start 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 -- 172.21.15.28:0/845397019 shutdown_connections 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 -- 172.21.15.28:0/845397019 >> 172.21.15.28:0/845397019 conn(0x7ff0000b0270 msgr2=0x7ff0000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 -- 172.21.15.28:0/845397019 shutdown_connections 2024-09-13T17:18:07.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.438+0000 7ff006000640 1 -- 172.21.15.28:0/845397019 wait complete. 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.440+0000 7ff006000640 1 Processor -- start 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.440+0000 7ff006000640 1 -- start start 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 0x7ff0000a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 0x7ff0001e09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0001e13b0 con 0x7ff0001e0f30 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000aae40 con 0x7ff000070430 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7ff006000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0000a96a0 con 0x7ff000141600 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7fefffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7fefffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34126/0 (socket says 172.21.15.28:34126) 2024-09-13T17:18:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.441+0000 7fefffe00640 1 -- 172.21.15.28:0/2872483955 learned_addr learned my addr 172.21.15.28:0/2872483955 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefff400640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 0x7ff0000a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7feffea00640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 0x7ff0001e09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 -- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 msgr2=0x7ff0001e09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 0x7ff0001e09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 -- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 msgr2=0x7ff0000a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 0x7ff0000a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 -- 172.21.15.28:0/2872483955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000a9980 con 0x7ff0001e0f30 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7feffea00640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 0x7ff0001e09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefff400640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 0x7ff0000a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:07.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7fefffe00640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff000061b50 tx=0x7feff800ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8006b50 con 0x7ff0001e0f30 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff8006cb0 con 0x7ff0001e0f30 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff8003bb0 con 0x7ff0001e0f30 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.442+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0000a9c60 con 0x7ff0001e0f30 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.443+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0000a9390 con 0x7ff0001e0f30 2024-09-13T17:18:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.443+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff00006f920 con 0x7ff0001e0f30 2024-09-13T17:18:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.444+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feff8003d10 con 0x7ff0001e0f30 2024-09-13T17:18:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.446+0000 7feff7400640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 0x7fefe006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.446+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5853+0+0 (secure 0 0 0) 0x7feff8088e20 con 0x7ff0001e0f30 2024-09-13T17:18:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.446+0000 7fefff400640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 0x7fefe006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:07.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.447+0000 7fefff400640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 0x7fefe006d0c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7feff0032000 tx=0x7feff002d000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.450+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feff8053f60 con 0x7ff0001e0f30 2024-09-13T17:18:07.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:07.665+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff0001569b0 con 0x7ff0001e0f30 2024-09-13T17:18:08.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.252+0000 7feff7400640 1 -- 172.21.15.28:0/2872483955 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v124) v1 ==== 104+0+0 (secure 0 0 0) 0x7feff8057580 con 0x7ff0001e0f30 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 msgr2=0x7fefe006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 0x7fefe006d0c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7feff0032000 tx=0x7feff002d000 comp rx=0 tx=0).stop 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 msgr2=0x7ff000140a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7ff000061b50 tx=0x7feff800ebe0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 shutdown_connections 2024-09-13T17:18:08.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fefe006ac00 0x7fefe006d0c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff0001e0f30 0x7ff000140a70 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff000141600 0x7ff0001e09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 --2- 172.21.15.28:0/2872483955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff000070430 0x7ff0000a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 >> 172.21.15.28:0/2872483955 conn(0x7ff0000b0270 msgr2=0x7ff00014ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 shutdown_connections 2024-09-13T17:18:08.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.255+0000 7ff006000640 1 -- 172.21.15.28:0/2872483955 wait complete. 2024-09-13T17:18:08.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:18:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.435+0000 7f1c4fe00640 1 Processor -- start 2024-09-13T17:18:08.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.435+0000 7f1c4fe00640 1 -- start start 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 0x7f1c500cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c5006a590 0x7f1c500b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5015cc70 con 0x7f1c500d0370 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500d08e0 con 0x7f1c5006a590 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c500d0bc0 con 0x7f1c5015c890 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c5006a590 0x7f1c500b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1c5006a590 0x7f1c500b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34134/0 (socket says 172.21.15.28:34134) 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.436+0000 7f1c4f400640 1 -- 172.21.15.28:0/463968435 learned_addr learned my addr 172.21.15.28:0/463968435 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4ea00640 1 --2- 172.21.15.28:0/463968435 >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 0x7f1c500cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4f400640 1 -- 172.21.15.28:0/463968435 >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 msgr2=0x7f1c500cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:08.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4f400640 1 --2- 172.21.15.28:0/463968435 >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 0x7f1c500cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4f400640 1 -- 172.21.15.28:0/463968435 >> v1:172.21.15.189:6789/0 conn(0x7f1c500d0370 legacy=0x7f1c50069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4f400640 1 -- 172.21.15.28:0/463968435 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c500d0ea0 con 0x7f1c5006a590 2024-09-13T17:18:08.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.437+0000 7f1c4ea00640 1 --2- 172.21.15.28:0/463968435 >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 0x7f1c500cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.438+0000 7f1c4f400640 1 --2- 172.21.15.28:0/463968435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500b5c80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1c500cfab0 tx=0x7f1c4802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1fee17c2117a77 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.438+0000 7f1c4e000640 1 -- 172.21.15.28:0/463968435 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48031600 con 0x7f1c5006a590 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.438+0000 7f1c4e000640 1 -- 172.21.15.28:0/463968435 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4802d050 con 0x7f1c5006a590 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.438+0000 7f1c4e000640 1 -- 172.21.15.28:0/463968435 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c480319a0 con 0x7f1c5006a590 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 -- 172.21.15.28:0/463968435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 msgr2=0x7f1c500b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/463968435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500b5c80 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1c500cfab0 tx=0x7f1c4802a7c0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 -- 172.21.15.28:0/463968435 shutdown_connections 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/463968435 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500b5c80 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/463968435 >> v2:172.21.15.189:3300/0 conn(0x7f1c5015c890 0x7f1c500cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 -- 172.21.15.28:0/463968435 >> 172.21.15.28:0/463968435 conn(0x7f1c500708c0 msgr2=0x7f1c50070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 -- 172.21.15.28:0/463968435 shutdown_connections 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.439+0000 7f1c4fe00640 1 -- 172.21.15.28:0/463968435 wait complete. 2024-09-13T17:18:08.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.440+0000 7f1c4fe00640 1 Processor -- start 2024-09-13T17:18:08.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 -- start start 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 0x7f1c501f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c5015c890 0x7f1c500c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500cc830 con 0x7f1c5006a590 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c501f6650 con 0x7f1c500d0370 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.441+0000 7f1c4fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500c8600 con 0x7f1c5015c890 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34148/0 (socket says 172.21.15.28:34148) 2024-09-13T17:18:08.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 -- 172.21.15.28:0/2658640218 learned_addr learned my addr 172.21.15.28:0/2658640218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c47e00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 0x7f1c501f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4f400640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c5015c890 0x7f1c500c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 -- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c5015c890 msgr2=0x7f1c500c62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c5015c890 0x7f1c500c62b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 -- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 msgr2=0x7f1c501f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 0x7f1c501f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 -- 172.21.15.28:0/2658640218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009b30 con 0x7f1c5006a590 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c47e00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 0x7f1c501f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.442+0000 7f1c4ea00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000d7d0 tx=0x7f1c4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.443+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40004030 con 0x7f1c5006a590 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.443+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40004190 con 0x7f1c5006a590 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.443+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40004520 con 0x7f1c5006a590 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.443+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c500c88e0 con 0x7f1c5006a590 2024-09-13T17:18:08.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.443+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c500c8ea0 con 0x7f1c5006a590 2024-09-13T17:18:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.444+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c500cd870 con 0x7f1c5006a590 2024-09-13T17:18:08.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.445+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1c400096a0 con 0x7f1c5006a590 2024-09-13T17:18:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.446+0000 7f1c46a00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 0x7f1c2c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.446+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1c40089680 con 0x7f1c5006a590 2024-09-13T17:18:08.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.447+0000 7f1c47e00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 0x7f1c2c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:08.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.447+0000 7f1c47e00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 0x7f1c2c06d0c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1c38007fd0 tx=0x7f1c38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:08.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.450+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1c40054730 con 0x7f1c5006a590 2024-09-13T17:18:08.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:08.627+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1c500d2060 con 0x7f1c5006a590 2024-09-13T17:18:09.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.258+0000 7f1c46a00640 1 -- 172.21.15.28:0/2658640218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v143) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1c40057d50 con 0x7f1c5006a590 2024-09-13T17:18:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:18:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 msgr2=0x7f1c2c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 0x7f1c2c06d0c0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f1c38007fd0 tx=0x7f1c38009040 comp rx=0 tx=0).stop 2024-09-13T17:18:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 msgr2=0x7f1c500cc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1c4000d7d0 tx=0x7f1c4000dc90 comp rx=0 tx=0).stop 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 shutdown_connections 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1c2c06ac00 0x7f1c2c06d0c0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1c5015c890 0x7f1c500c62b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c500d0370 0x7f1c501f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 --2- 172.21.15.28:0/2658640218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c5006a590 0x7f1c500cc2f0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.262+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 >> 172.21.15.28:0/2658640218 conn(0x7f1c500708c0 msgr2=0x7f1c50069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.263+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 shutdown_connections 2024-09-13T17:18:09.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.263+0000 7f1c4fe00640 1 -- 172.21.15.28:0/2658640218 wait complete. 2024-09-13T17:18:09.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:18:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.443+0000 7fc04b200640 1 Processor -- start 2024-09-13T17:18:09.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.443+0000 7fc04b200640 1 -- start start 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc04c06ae90 0x7fc04c06b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc04c070e30 0x7fc04c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c06b7b0 con 0x7fc04c0c8790 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c072f50 con 0x7fc04c06ae90 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c073230 con 0x7fc04c070e30 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc049e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc04c06ae90 0x7fc04c06b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc04c0c8790 0x7fc04c0c8c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33250/0 (socket says 172.21.15.28:33250) 2024-09-13T17:18:09.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.444+0000 7fc04a800640 1 -- 172.21.15.28:0/2001424052 learned_addr learned my addr 172.21.15.28:0/2001424052 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.445+0000 7fc041400640 1 --2- 172.21.15.28:0/2001424052 >> v2:172.21.15.189:3300/0 conn(0x7fc04c070e30 0x7fc04c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.445+0000 7fc049e00640 1 -- 172.21.15.28:0/2001424052 >> v2:172.21.15.189:3300/0 conn(0x7fc04c070e30 msgr2=0x7fc04c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.445+0000 7fc049e00640 1 --2- 172.21.15.28:0/2001424052 >> v2:172.21.15.189:3300/0 conn(0x7fc04c070e30 0x7fc04c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.445+0000 7fc049e00640 1 -- 172.21.15.28:0/2001424052 >> v1:172.21.15.28:6789/0 conn(0x7fc04c0c8790 legacy=0x7fc04c0c8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.445+0000 7fc049e00640 1 -- 172.21.15.28:0/2001424052 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c073510 con 0x7fc04c06ae90 2024-09-13T17:18:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.446+0000 7fc049e00640 1 --2- 172.21.15.28:0/2001424052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c06ae90 0x7fc04c06b270 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0d0500 tx=0x7fc03c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303ca6f6faf95d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.446+0000 7fc049400640 1 -- 172.21.15.28:0/2001424052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c02ae60 con 0x7fc04c06ae90 2024-09-13T17:18:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.446+0000 7fc049400640 1 -- 172.21.15.28:0/2001424052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03c032440 con 0x7fc04c06ae90 2024-09-13T17:18:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.446+0000 7fc049400640 1 -- 172.21.15.28:0/2001424052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03c0327d0 con 0x7fc04c06ae90 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 -- 172.21.15.28:0/2001424052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c06ae90 msgr2=0x7fc04c06b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 --2- 172.21.15.28:0/2001424052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c06ae90 0x7fc04c06b270 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0d0500 tx=0x7fc03c02a110 comp rx=0 tx=0).stop 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 -- 172.21.15.28:0/2001424052 shutdown_connections 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 --2- 172.21.15.28:0/2001424052 >> v2:172.21.15.189:3300/0 conn(0x7fc04c070e30 0x7fc04c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 --2- 172.21.15.28:0/2001424052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c06ae90 0x7fc04c06b270 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 -- 172.21.15.28:0/2001424052 >> 172.21.15.28:0/2001424052 conn(0x7fc04c070310 msgr2=0x7fc04c070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 -- 172.21.15.28:0/2001424052 shutdown_connections 2024-09-13T17:18:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.447+0000 7fc04b200640 1 -- 172.21.15.28:0/2001424052 wait complete. 2024-09-13T17:18:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.448+0000 7fc04b200640 1 Processor -- start 2024-09-13T17:18:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.449+0000 7fc04b200640 1 -- start start 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.449+0000 7fc04b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 0x7fc04c0ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc04c070e30 0x7fc04c15f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0b45d0 con 0x7fc04c0c8790 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0b39c0 con 0x7fc04c070e30 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0b3ca0 con 0x7fc04c06ae90 2024-09-13T17:18:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34168/0 (socket says 172.21.15.28:34168) 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 -- 172.21.15.28:0/320272047 learned_addr learned my addr 172.21.15.28:0/320272047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc049e00640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 0x7fc04c0ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc041400640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc04c070e30 0x7fc04c15f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 -- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 msgr2=0x7fc04c0ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 0x7fc04c0ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 -- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc04c070e30 msgr2=0x7fc04c15f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc04c070e30 0x7fc04c15f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc04a800640 1 -- 172.21.15.28:0/320272047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c009400 con 0x7fc04c0c8790 2024-09-13T17:18:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.450+0000 7fc049e00640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 0x7fc04c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc04a800640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc04400cbc0 tx=0x7fc04400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400ad40 con 0x7fc04c0c8790 2024-09-13T17:18:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc044012400 con 0x7fc04c0c8790 2024-09-13T17:18:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04400aea0 con 0x7fc04c0c8790 2024-09-13T17:18:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c0b3f80 con 0x7fc04c0c8790 2024-09-13T17:18:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.451+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c0caa20 con 0x7fc04c0c8790 2024-09-13T17:18:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.452+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c0b4780 con 0x7fc04c0c8790 2024-09-13T17:18:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.453+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc044007070 con 0x7fc04c0c8790 2024-09-13T17:18:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.454+0000 7fc042a00640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 0x7fc02806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.454+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fc044089150 con 0x7fc04c0c8790 2024-09-13T17:18:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.454+0000 7fc049e00640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 0x7fc02806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.455+0000 7fc049e00640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 0x7fc02806d0c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc03c034000 tx=0x7fc03c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.458+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc04c0b4780 con 0x7fc04c0c8790 2024-09-13T17:18:09.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:09.642+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc04c06b270 con 0x7fc04c0c8790 2024-09-13T17:18:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.261+0000 7fc042a00640 1 -- 172.21.15.28:0/320272047 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v144) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc044054460 con 0x7fc04c0c8790 2024-09-13T17:18:10.263 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:18:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.265+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 msgr2=0x7fc02806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.265+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 0x7fc02806d0c0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc03c034000 tx=0x7fc03c02e000 comp rx=0 tx=0).stop 2024-09-13T17:18:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.265+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 msgr2=0x7fc04c0b4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.265+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fc04400cbc0 tx=0x7fc04400a430 comp rx=0 tx=0).stop 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 shutdown_connections 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc02806ac00 0x7fc02806d0c0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc04c0c8790 0x7fc04c0b4a60 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc04c070e30 0x7fc04c15f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 --2- 172.21.15.28:0/320272047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc04c06ae90 0x7fc04c0ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.266+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 >> 172.21.15.28:0/320272047 conn(0x7fc04c070310 msgr2=0x7fc04c06c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.267+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 shutdown_connections 2024-09-13T17:18:10.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.267+0000 7fc04b200640 1 -- 172.21.15.28:0/320272047 wait complete. 2024-09-13T17:18:10.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:18:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.485+0000 7f46d3e00640 1 Processor -- start 2024-09-13T17:18:10.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.486+0000 7f46d3e00640 1 -- start start 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46d40cd830 0x7f46d4070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46d3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4071630 con 0x7f46d4143910 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46d3e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d4143700 con 0x7f46d4143320 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46d3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d4070a20 con 0x7f46d40cd830 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46ca000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f46d4143910 0x7f46d40cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33264/0 (socket says 172.21.15.28:33264) 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46ca000640 1 -- 172.21.15.28:0/2490300637 learned_addr learned my addr 172.21.15.28:0/2490300637 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:10.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.487+0000 7f46d3400640 1 --2- 172.21.15.28:0/2490300637 >> v2:172.21.15.28:3300/0 conn(0x7f46d40cd830 0x7f46d4070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.488+0000 7f46d3400640 1 -- 172.21.15.28:0/2490300637 >> v1:172.21.15.189:6789/0 conn(0x7f46d4143320 legacy=0x7f46d4071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.488+0000 7f46d3400640 1 -- 172.21.15.28:0/2490300637 >> v1:172.21.15.28:6789/0 conn(0x7f46d4143910 legacy=0x7f46d40cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.488+0000 7f46d3400640 1 -- 172.21.15.28:0/2490300637 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d4070d00 con 0x7f46d40cd830 2024-09-13T17:18:10.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.489+0000 7f46d3400640 1 --2- 172.21.15.28:0/2490300637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d4070440 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009d90 tx=0x7f46cc02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff52e9753b727480 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.489+0000 7f46d2000640 1 -- 172.21.15.28:0/2490300637 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc007740 con 0x7f46d40cd830 2024-09-13T17:18:10.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.489+0000 7f46d2000640 1 -- 172.21.15.28:0/2490300637 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46cc02d050 con 0x7f46d40cd830 2024-09-13T17:18:10.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.489+0000 7f46d2000640 1 -- 172.21.15.28:0/2490300637 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc007ac0 con 0x7f46d40cd830 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 -- 172.21.15.28:0/2490300637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 msgr2=0x7f46d4070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 --2- 172.21.15.28:0/2490300637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d4070440 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f46cc009d90 tx=0x7f46cc02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 -- 172.21.15.28:0/2490300637 shutdown_connections 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 --2- 172.21.15.28:0/2490300637 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d4070440 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 -- 172.21.15.28:0/2490300637 >> 172.21.15.28:0/2490300637 conn(0x7f46d4069e80 msgr2=0x7f46d406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 -- 172.21.15.28:0/2490300637 shutdown_connections 2024-09-13T17:18:10.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.490+0000 7f46d3e00640 1 -- 172.21.15.28:0/2490300637 wait complete. 2024-09-13T17:18:10.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.491+0000 7f46d3e00640 1 Processor -- start 2024-09-13T17:18:10.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.492+0000 7f46d3e00640 1 -- start start 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 0x7f46d4158100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46d4143910 0x7f46d40cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d4158640 con 0x7f46d40cd830 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40cbae0 con 0x7f46d4143910 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d40cbdc0 con 0x7f46d4143320 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34190/0 (socket says 172.21.15.28:34190) 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 -- 172.21.15.28:0/3404237208 learned_addr learned my addr 172.21.15.28:0/3404237208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46ca000640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 0x7f46d4158100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d3400640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46d4143910 0x7f46d40cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 -- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 msgr2=0x7f46d4158100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 0x7f46d4158100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 -- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46d4143910 msgr2=0x7f46d40cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46d4143910 0x7f46d40cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46d2a00640 1 -- 172.21.15.28:0/3404237208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc009a70 con 0x7f46d40cd830 2024-09-13T17:18:10.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.493+0000 7f46ca000640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 0x7f46d4158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46d2a00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f46c400c450 tx=0x7f46c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40052d0 con 0x7f46d40cd830 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c4005430 con 0x7f46d40cd830 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c4012870 con 0x7f46d40cd830 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d40cc0a0 con 0x7f46d40cd830 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.494+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d40cc660 con 0x7f46d40cd830 2024-09-13T17:18:10.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.495+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d40b3800 con 0x7f46d40cd830 2024-09-13T17:18:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.496+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f46c40129d0 con 0x7f46d40cd830 2024-09-13T17:18:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.497+0000 7f46cb400640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 0x7f46b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.497+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f46c4088bd0 con 0x7f46d40cd830 2024-09-13T17:18:10.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.497+0000 7f46ca000640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 0x7f46b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.498+0000 7f46ca000640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 0x7f46b006d0c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f46d40ca850 tx=0x7f46bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.501+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46c40540a0 con 0x7f46d40cd830 2024-09-13T17:18:10.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.681+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f46d4070440 con 0x7f46d40cd830 2024-09-13T17:18:10.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.685+0000 7f46cb400640 1 -- 172.21.15.28:0/3404237208 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f46c40576c0 con 0x7f46d40cd830 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 msgr2=0x7f46b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 0x7f46b006d0c0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f46d40ca850 tx=0x7f46bc009040 comp rx=0 tx=0).stop 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 msgr2=0x7f46d413ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f46c400c450 tx=0x7f46c400c910 comp rx=0 tx=0).stop 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 shutdown_connections 2024-09-13T17:18:10.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46b006ac00 0x7f46b006d0c0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46d4143910 0x7f46d40cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46d4143320 0x7f46d4158100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 --2- 172.21.15.28:0/3404237208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46d40cd830 0x7f46d413ba70 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.687+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 >> 172.21.15.28:0/3404237208 conn(0x7f46d4069e80 msgr2=0x7f46d40b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.688+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 shutdown_connections 2024-09-13T17:18:10.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.688+0000 7f46d3e00640 1 -- 172.21.15.28:0/3404237208 wait complete. 2024-09-13T17:18:10.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume' 2024-09-13T17:18:10.748 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:18:10.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:18:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.860+0000 7f25cfe00640 1 Processor -- start 2024-09-13T17:18:10.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.860+0000 7f25cfe00640 1 -- start start 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f25d015c750 0x7f25d00d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 0x7f25d00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25d00b4900 0x7f25d0069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d00b4db0 con 0x7f25d00b4900 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d00d0d80 con 0x7f25d015c750 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d00d0960 con 0x7f25d00d1d10 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25d00b4900 0x7f25d0069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f25d00b4900 0x7f25d0069e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34194/0 (socket says 172.21.15.28:34194) 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.861+0000 7f25cf400640 1 -- 172.21.15.28:0/1809565705 learned_addr learned my addr 172.21.15.28:0/1809565705 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:10.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cea00640 1 --2- 172.21.15.28:0/1809565705 >> v2:172.21.15.189:3300/0 conn(0x7f25d015c750 0x7f25d00d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25ce000640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 0x7f25d00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cf400640 1 -- 172.21.15.28:0/1809565705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 msgr2=0x7f25d00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cf400640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 0x7f25d00b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cf400640 1 -- 172.21.15.28:0/1809565705 >> v2:172.21.15.189:3300/0 conn(0x7f25d015c750 msgr2=0x7f25d00d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cf400640 1 --2- 172.21.15.28:0/1809565705 >> v2:172.21.15.189:3300/0 conn(0x7f25d015c750 0x7f25d00d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25cf400640 1 -- 172.21.15.28:0/1809565705 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d006aa60 con 0x7f25d00b4900 2024-09-13T17:18:10.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.862+0000 7f25ce000640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 0x7f25d00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cf400640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d00b4900 0x7f25d0069e80 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f25d008b150 tx=0x7f25c802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50fd9e3db14a8ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cd600640 1 -- 172.21.15.28:0/1809565705 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8031600 con 0x7f25d00b4900 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cd600640 1 -- 172.21.15.28:0/1809565705 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c802d050 con 0x7f25d00b4900 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cd600640 1 -- 172.21.15.28:0/1809565705 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c8031980 con 0x7f25d00b4900 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cfe00640 1 -- 172.21.15.28:0/1809565705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d00b4900 msgr2=0x7f25d0069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.863+0000 7f25cfe00640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d00b4900 0x7f25d0069e80 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f25d008b150 tx=0x7f25c802a610 comp rx=0 tx=0).stop 2024-09-13T17:18:10.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 -- 172.21.15.28:0/1809565705 shutdown_connections 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d00b4900 0x7f25d0069e80 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 --2- 172.21.15.28:0/1809565705 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00d1d10 0x7f25d00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 --2- 172.21.15.28:0/1809565705 >> v2:172.21.15.189:3300/0 conn(0x7f25d015c750 0x7f25d00d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 -- 172.21.15.28:0/1809565705 >> 172.21.15.28:0/1809565705 conn(0x7f25d00cf5a0 msgr2=0x7f25d00cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 -- 172.21.15.28:0/1809565705 shutdown_connections 2024-09-13T17:18:10.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.864+0000 7f25cfe00640 1 -- 172.21.15.28:0/1809565705 wait complete. 2024-09-13T17:18:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.865+0000 7f25cfe00640 1 Processor -- start 2024-09-13T17:18:10.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.865+0000 7f25cfe00640 1 -- start start 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 0x7f25d01595b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d015c750 0x7f25d014ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d0159af0 con 0x7f25d0069e80 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d006a870 con 0x7f25d015c750 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.866+0000 7f25cfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d015a1d0 con 0x7f25d00b4900 2024-09-13T17:18:10.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 0x7f25d01595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d015c750 0x7f25d014ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34208/0 (socket says 172.21.15.28:34208) 2024-09-13T17:18:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 0x7f25d01595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:47486/0 (socket says 172.21.15.28:47486) 2024-09-13T17:18:10.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 -- 172.21.15.28:0/7632654 learned_addr learned my addr 172.21.15.28:0/7632654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 -- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 msgr2=0x7f25d01595b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 0x7f25d01595b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 -- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d015c750 msgr2=0x7f25d014ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d015c750 0x7f25d014ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 -- 172.21.15.28:0/7632654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c8009b30 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.867+0000 7f25cf400640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009f40 tx=0x7f25c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.868+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c803b810 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.868+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c803b990 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.868+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c803bd40 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.868+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d015a4b0 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.868+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d015aa70 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.869+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d00b43c0 con 0x7f25d0069e80 2024-09-13T17:18:10.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.870+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f25c803a3a0 con 0x7f25d0069e80 2024-09-13T17:18:10.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.871+0000 7f25c6a00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 0x7f25b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:10.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.871+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f25c80a9910 con 0x7f25d0069e80 2024-09-13T17:18:10.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.871+0000 7f25cea00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 0x7f25b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:10.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.872+0000 7f25cea00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 0x7f25b006d0c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f25c00079a0 tx=0x7f25c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:10.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:10.875+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f25c8074e90 con 0x7f25d0069e80 2024-09-13T17:18:11.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.118+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]} v 0) v1 -- 0x7f25d00d0b60 con 0x7f25d0069e80 2024-09-13T17:18:11.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.262+0000 7f25c6a00640 1 -- 172.21.15.28:0/7632654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume"]}]=0 v508) v1 ==== 155+0+0 (secure 0 0 0) 0x7f25c8009510 con 0x7f25d0069e80 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 msgr2=0x7f25b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 0x7f25b006d0c0 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f25c00079a0 tx=0x7f25c0008040 comp rx=0 tx=0).stop 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 msgr2=0x7f25d006a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f25c8009f40 tx=0x7f25c8004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 shutdown_connections 2024-09-13T17:18:11.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25b006ac00 0x7f25b006d0c0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25d0069e80 0x7f25d006a330 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25d015c750 0x7f25d014ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 --2- 172.21.15.28:0/7632654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25d00b4900 0x7f25d01595b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 >> 172.21.15.28:0/7632654 conn(0x7f25d00cf5a0 msgr2=0x7f25d01567c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 shutdown_connections 2024-09-13T17:18:11.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.265+0000 7f25cfe00640 1 -- 172.21.15.28:0/7632654 wait complete. 2024-09-13T17:18:11.284 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-09-13T17:18:11.285 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-09-13T17:18:11.285 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:11.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.477+0000 7f2087e00640 1 Processor -- start 2024-09-13T17:18:11.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.478+0000 7f2087e00640 1 -- start start 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20880b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2088159820 0x7f20880b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 0x7f20880b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20880c3850 con 0x7f2088159820 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f208815a930 con 0x7f20880b22e0 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20880b2cd0 con 0x7f2088159440 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f207fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2088159820 0x7f20880b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f207fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2088159820 0x7f20880b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34220/0 (socket says 172.21.15.28:34220) 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f207fe00640 1 -- 172.21.15.28:0/2339685139 learned_addr learned my addr 172.21.15.28:0/2339685139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:11.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2087400640 1 --2- 172.21.15.28:0/2339685139 >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 0x7f20880b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.479+0000 7f2086a00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20880b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f207fe00640 1 -- 172.21.15.28:0/2339685139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 msgr2=0x7f20880b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f207fe00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20880b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f207fe00640 1 -- 172.21.15.28:0/2339685139 >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 msgr2=0x7f20880b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f207fe00640 1 --2- 172.21.15.28:0/2339685139 >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 0x7f20880b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f207fe00640 1 -- 172.21.15.28:0/2339685139 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2088159d20 con 0x7f2088159820 2024-09-13T17:18:11.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.480+0000 7f2087400640 1 --2- 172.21.15.28:0/2339685139 >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 0x7f20880b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.481+0000 7f207fe00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f20880b0980 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f208806d2d0 tx=0x7f207002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73895e8031128fc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.481+0000 7f2086000640 1 -- 172.21.15.28:0/2339685139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070032440 con 0x7f2088159820 2024-09-13T17:18:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.481+0000 7f2086000640 1 -- 172.21.15.28:0/2339685139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20700325a0 con 0x7f2088159820 2024-09-13T17:18:11.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.481+0000 7f2086000640 1 -- 172.21.15.28:0/2339685139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2070032930 con 0x7f2088159820 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 -- 172.21.15.28:0/2339685139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 msgr2=0x7f20880b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f20880b0980 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f208806d2d0 tx=0x7f207002a270 comp rx=0 tx=0).stop 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 -- 172.21.15.28:0/2339685139 shutdown_connections 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f20880b0980 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 --2- 172.21.15.28:0/2339685139 >> v2:172.21.15.189:3300/0 conn(0x7f20880b22e0 0x7f20880b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 --2- 172.21.15.28:0/2339685139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20880b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 -- 172.21.15.28:0/2339685139 >> 172.21.15.28:0/2339685139 conn(0x7f2088072e40 msgr2=0x7f2088073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 -- 172.21.15.28:0/2339685139 shutdown_connections 2024-09-13T17:18:11.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.482+0000 7f2087e00640 1 -- 172.21.15.28:0/2339685139 wait complete. 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.483+0000 7f2087e00640 1 Processor -- start 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 -- start start 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20880b22e0 0x7f20881ed6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20881f0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2088142b20 con 0x7f2088159820 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20881f10c0 con 0x7f20880b22e0 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.484+0000 7f2087e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20881450e0 con 0x7f2088159440 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34234/0 (socket says 172.21.15.28:34234) 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 -- 172.21.15.28:0/2836902000 learned_addr learned my addr 172.21.15.28:0/2836902000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2086a00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20880b22e0 0x7f20881ed6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f207fe00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20881f0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 -- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 msgr2=0x7f20881f0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20881f0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 -- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20880b22e0 msgr2=0x7f20881ed6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20880b22e0 0x7f20881ed6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.485+0000 7f2087400640 1 -- 172.21.15.28:0/2836902000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20700094d0 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f2087400640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f2080007d60 tx=0x7f208000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080003990 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f208000f040 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2080003d30 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20881453c0 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.486+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2088145980 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.487+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f208815aaa0 con 0x7f2088159820 2024-09-13T17:18:11.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.488+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2080004030 con 0x7f2088159820 2024-09-13T17:18:11.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.489+0000 7f207ea00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 0x7f206406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.489+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2080089220 con 0x7f2088159820 2024-09-13T17:18:11.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.489+0000 7f2086a00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 0x7f206406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.490+0000 7f2086a00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 0x7f206406d0f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2078006fd0 tx=0x7f2078008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.493+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f20800547a0 con 0x7f2088159820 2024-09-13T17:18:11.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.673+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20880b2790 con 0x7f2088159820 2024-09-13T17:18:11.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.674+0000 7f207ea00640 1 -- 172.21.15.28:0/2836902000 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v144) v1 ==== 74+0+12996 (secure 0 0 0) 0x7f2080003e90 con 0x7f2088159820 2024-09-13T17:18:11.676 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:11.677 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":144,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:10.258181+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/4121307324":"2024-09-14T17:18:07.243206+0000","172.21.15.28:6834/4121307324":"2024-09-14T17:18:07.243206+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 msgr2=0x7f206406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 0x7f206406d0f0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2078006fd0 tx=0x7f2078008040 comp rx=0 tx=0).stop 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 msgr2=0x7f2088158060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f2080007d60 tx=0x7f208000d4a0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 shutdown_connections 2024-09-13T17:18:11.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f206406ac30 0x7f206406d0f0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2088159820 0x7f2088158060 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2088159440 0x7f20881f0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 --2- 172.21.15.28:0/2836902000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f20880b22e0 0x7f20881ed6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 >> 172.21.15.28:0/2836902000 conn(0x7f2088072e40 msgr2=0x7f2088155b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 shutdown_connections 2024-09-13T17:18:11.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.677+0000 7f2087e00640 1 -- 172.21.15.28:0/2836902000 wait complete. 2024-09-13T17:18:11.696 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:18:11.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:18:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.852+0000 7fc874800640 1 Processor -- start 2024-09-13T17:18:11.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.852+0000 7fc874800640 1 -- start start 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc874800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc86c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc874800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c06f920 con 0x7fc86c06f3f0 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc874800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c15b290 con 0x7fc86c15cd80 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc874800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c15b570 con 0x7fc86c159770 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc871400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc86c06f3f0 0x7fc86c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33280/0 (socket says 172.21.15.28:33280) 2024-09-13T17:18:11.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.853+0000 7fc871400640 1 -- 172.21.15.28:0/1675198059 learned_addr learned my addr 172.21.15.28:0/1675198059 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc871e00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc86c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3282838813 0 0) 0x7fc86c06f920 con 0x7fc86c06f3f0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc854003520 con 0x7fc86c06f3f0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578888964 0 0) 0x7fc86c15b290 con 0x7fc86c15cd80 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c06f920 con 0x7fc86c15cd80 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2649980178 0 0) 0x7fc854003520 con 0x7fc86c06f3f0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 msgr2=0x7fc86c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc86c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 >> v1:172.21.15.189:6789/0 conn(0x7fc86c15cd80 legacy=0x7fc86c15a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c15b850 con 0x7fc86c06f3f0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc85c0015f0 con 0x7fc86c06f3f0 2024-09-13T17:18:11.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 >> v1:172.21.15.28:6789/0 conn(0x7fc86c06f3f0 legacy=0x7fc86c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.854+0000 7fc870a00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc854003b30 0x7fc854003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc870a00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc8540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc870a00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854004d00 0x7fc854005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c15b570 con 0x7fc854004d00 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85400bb10 con 0x7fc854003b30 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85400bca0 con 0x7fc86c159770 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc854003b30 0x7fc854003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc872800640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854004d00 0x7fc854005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871400640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc8540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 -- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 msgr2=0x7fc8540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c159770 0x7fc8540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 -- 172.21.15.28:0/1675198059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854004d00 msgr2=0x7fc854005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:11.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854004d00 0x7fc854005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.855+0000 7fc871e00640 1 -- 172.21.15.28:0/1675198059 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc854003520 con 0x7fc854003b30 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc871e00640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc854003b30 0x7fc854003f30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc860009ac0 tx=0x7fc8540057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72967a8f2038581e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860031400 con 0x7fc854003b30 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86002d050 con 0x7fc854003b30 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc870a00640 1 -- 172.21.15.28:0/1675198059 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860031780 con 0x7fc854003b30 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc874800640 1 -- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc854003b30 msgr2=0x7fc854003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc874800640 1 --2- 172.21.15.28:0/1675198059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc854003b30 0x7fc854003f30 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc860009ac0 tx=0x7fc8540057f0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.857+0000 7fc872800640 1 -- 172.21.15.28:0/1675198059 reap_dead start 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.858+0000 7fc874800640 1 -- 172.21.15.28:0/1675198059 shutdown_connections 2024-09-13T17:18:11.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.858+0000 7fc874800640 1 -- 172.21.15.28:0/1675198059 >> 172.21.15.28:0/1675198059 conn(0x7fc86c0b0270 msgr2=0x7fc86c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.858+0000 7fc874800640 1 -- 172.21.15.28:0/1675198059 shutdown_connections 2024-09-13T17:18:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.858+0000 7fc874800640 1 -- 172.21.15.28:0/1675198059 wait complete. 2024-09-13T17:18:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.859+0000 7fc874800640 1 Processor -- start 2024-09-13T17:18:11.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.859+0000 7fc874800640 1 -- start start 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c14a5e0 0x7fc86c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc86c0cc990 0x7fc86c14ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0cb690 con 0x7fc854003b30 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0cb970 con 0x7fc86c0cc990 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc874800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86c0cbc50 con 0x7fc86c14a5e0 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc871e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc871e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34254/0 (socket says 172.21.15.28:34254) 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.860+0000 7fc871e00640 1 -- 172.21.15.28:0/3864462843 learned_addr learned my addr 172.21.15.28:0/3864462843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871400640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c14a5e0 0x7fc86c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc872800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc86c0cc990 0x7fc86c14ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 -- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c14a5e0 msgr2=0x7fc86c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c14a5e0 0x7fc86c0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 -- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc86c0cc990 msgr2=0x7fc86c14ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc86c0cc990 0x7fc86c14ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 -- 172.21.15.28:0/3864462843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0cbf30 con 0x7fc854003b30 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc871e00640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fc860009be0 tx=0x7fc860004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.861+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860003b90 con 0x7fc854003b30 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.862+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc860003d10 con 0x7fc854003b30 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.862+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c157a00 con 0x7fc854003b30 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.862+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860044650 con 0x7fc854003b30 2024-09-13T17:18:11.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.862+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c1577c0 con 0x7fc854003b30 2024-09-13T17:18:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.863+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c15a430 con 0x7fc854003b30 2024-09-13T17:18:11.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.864+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc860038070 con 0x7fc854003b30 2024-09-13T17:18:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.865+0000 7fc866000640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 0x7fc85006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.865+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8600a9100 con 0x7fc854003b30 2024-09-13T17:18:11.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.866+0000 7fc871400640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 0x7fc85006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:11.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.866+0000 7fc871400640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 0x7fc85006d0c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc86c14a9e0 tx=0x7fc85c001a60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:11.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:11.869+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc8600745f0 con 0x7fc854003b30 2024-09-13T17:18:12.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.047+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc86c1577c0 con 0x7fc854003b30 2024-09-13T17:18:12.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.048+0000 7fc866000640 1 -- 172.21.15.28:0/3864462843 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v144) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc86c1577c0 con 0x7fc854003b30 2024-09-13T17:18:12.050 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 msgr2=0x7fc85006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 0x7fc85006d0c0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc86c14a9e0 tx=0x7fc85c001a60 comp rx=0 tx=0).stop 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 msgr2=0x7fc86c0acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fc860009be0 tx=0x7fc860004730 comp rx=0 tx=0).stop 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 shutdown_connections 2024-09-13T17:18:12.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc85006ac00 0x7fc85006d0c0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc86c0cc990 0x7fc86c14ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc86c14a5e0 0x7fc86c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 --2- 172.21.15.28:0/3864462843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc854003b30 0x7fc86c0acac0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 >> 172.21.15.28:0/3864462843 conn(0x7fc86c0b0270 msgr2=0x7fc86c0cebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 shutdown_connections 2024-09-13T17:18:12.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.050+0000 7fc874800640 1 -- 172.21.15.28:0/3864462843 wait complete. 2024-09-13T17:18:12.068 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort' 2024-09-13T17:18:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.222+0000 7f965c400640 1 Processor -- start 2024-09-13T17:18:12.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.222+0000 7f965c400640 1 -- start start 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f965c400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f96540713d0 0x7f965406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f965c400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965415e100 con 0x7f96540cabc0 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f965c400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965415e3e0 con 0x7f965415a540 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f965c400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96540adb00 con 0x7f96540713d0 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f9659000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f96540cabc0 0x7f9654070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33292/0 (socket says 172.21.15.28:33292) 2024-09-13T17:18:12.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f9659000640 1 -- 172.21.15.28:0/3033852484 learned_addr learned my addr 172.21.15.28:0/3033852484 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.223+0000 7f965a400640 1 --2- 172.21.15.28:0/3033852484 >> v2:172.21.15.189:3300/0 conn(0x7f96540713d0 0x7f965406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.224+0000 7f9653e00640 1 -- 172.21.15.28:0/3033852484 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 8512305 0 0) 0x7f965415e100 con 0x7f96540cabc0 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.224+0000 7f9653e00640 1 -- 172.21.15.28:0/3033852484 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c003520 con 0x7f96540cabc0 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.224+0000 7f965a400640 1 -- 172.21.15.28:0/3033852484 >> v1:172.21.15.189:6789/0 conn(0x7f965415a540 legacy=0x7f965415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.224+0000 7f965a400640 1 -- 172.21.15.28:0/3033852484 >> v1:172.21.15.28:6789/0 conn(0x7f96540cabc0 legacy=0x7f9654070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.224+0000 7f965a400640 1 -- 172.21.15.28:0/3033852484 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f965415ef80 con 0x7f96540713d0 2024-09-13T17:18:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.225+0000 7f965a400640 1 --2- 172.21.15.28:0/3033852484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f965406c430 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f964c009d90 tx=0x7f964c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7b717ebcc54727 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.225+0000 7f9653e00640 1 -- 172.21.15.28:0/3033852484 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c007740 con 0x7f96540713d0 2024-09-13T17:18:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.225+0000 7f9653e00640 1 -- 172.21.15.28:0/3033852484 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c02d050 con 0x7f96540713d0 2024-09-13T17:18:12.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.225+0000 7f9653e00640 1 -- 172.21.15.28:0/3033852484 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c007ac0 con 0x7f96540713d0 2024-09-13T17:18:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 -- 172.21.15.28:0/3033852484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 msgr2=0x7f965406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 --2- 172.21.15.28:0/3033852484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f965406c430 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f964c009d90 tx=0x7f964c02a660 comp rx=0 tx=0).stop 2024-09-13T17:18:12.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 -- 172.21.15.28:0/3033852484 shutdown_connections 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 --2- 172.21.15.28:0/3033852484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f965406c430 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 -- 172.21.15.28:0/3033852484 >> 172.21.15.28:0/3033852484 conn(0x7f9654069e80 msgr2=0x7f965406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 -- 172.21.15.28:0/3033852484 shutdown_connections 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.226+0000 7f965c400640 1 -- 172.21.15.28:0/3033852484 wait complete. 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.228+0000 7f965c400640 1 Processor -- start 2024-09-13T17:18:12.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.228+0000 7f965c400640 1 -- start start 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f9654148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f965415a540 0x7f9654149a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654157e70 con 0x7f96540cabc0 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654158150 con 0x7f96540713d0 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965c400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9654158430 con 0x7f965415a540 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34262/0 (socket says 172.21.15.28:34262) 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 -- 172.21.15.28:0/855140709 learned_addr learned my addr 172.21.15.28:0/855140709 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:12.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659a00640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f9654148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f965a400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f965415a540 0x7f9654149a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:12.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 -- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f965415a540 msgr2=0x7f9654149a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f965415a540 0x7f9654149a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 -- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 msgr2=0x7f9654148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f9654148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.229+0000 7f9659000640 1 -- 172.21.15.28:0/855140709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c009a70 con 0x7f96540cabc0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f9659000640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f964000c500 tx=0x7f964000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640005ce0 con 0x7f96540cabc0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640005e40 con 0x7f96540cabc0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640007330 con 0x7f96540cabc0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9654158710 con 0x7f96540cabc0 2024-09-13T17:18:12.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.230+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96541494c0 con 0x7f96540cabc0 2024-09-13T17:18:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.231+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96540ae670 con 0x7f96540cabc0 2024-09-13T17:18:12.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.232+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9640007550 con 0x7f96540cabc0 2024-09-13T17:18:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.233+0000 7f9651600640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.233+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9640088720 con 0x7f96540cabc0 2024-09-13T17:18:12.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.234+0000 7f9659a00640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:12.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.234+0000 7f9659a00640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 0x7f963406d0c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9648000f60 tx=0x7f9648009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:12.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.237+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9640053cd0 con 0x7f96540cabc0 2024-09-13T17:18:12.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:12.483+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]} v 0) v1 -- 0x7f965406c430 con 0x7f96540cabc0 2024-09-13T17:18:13.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.271+0000 7f9651600640 1 -- 172.21.15.28:0/855140709 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]}]=0 v510) v1 ==== 169+0+0 (secure 0 0 0) 0x7f96400170f0 con 0x7f96540cabc0 2024-09-13T17:18:13.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 msgr2=0x7f963406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 0x7f963406d0c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f9648000f60 tx=0x7f9648009040 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 msgr2=0x7f96540b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f964000c500 tx=0x7f964000c9c0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 shutdown_connections 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f965415a540 0x7f9654149a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96540cabc0 0x7f96540b1710 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 --2- 172.21.15.28:0/855140709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96540713d0 0x7f9654148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 >> 172.21.15.28:0/855140709 conn(0x7f9654069e80 msgr2=0x7f96540ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:13.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.274+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 shutdown_connections 2024-09-13T17:18:13.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.275+0000 7f965c400640 1 -- 172.21.15.28:0/855140709 wait complete. 2024-09-13T17:18:13.295 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:18:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.456+0000 7fa5d3400640 1 Processor -- start 2024-09-13T17:18:13.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.456+0000 7fa5d3400640 1 -- start start 2024-09-13T17:18:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d3400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5cc159770 0x7fa5cc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5cc06f3f0 0x7fa5cc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d3400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5cc06f920 con 0x7fa5cc0add00 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d3400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc15b290 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d3400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5cc15b570 con 0x7fa5cc159770 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5c3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa5cc06f3f0 0x7fa5cc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.457+0000 7fa5d1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5cc0add00 0x7fa5cc159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:33306/0 (socket says 172.21.15.28:33306) 2024-09-13T17:18:13.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5d1400640 1 -- 172.21.15.28:0/3776670785 learned_addr learned my addr 172.21.15.28:0/3776670785 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/3776670785 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc159770 0x7fa5cc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5c3e00640 1 -- 172.21.15.28:0/3776670785 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc159770 msgr2=0x7fa5cc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/3776670785 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc159770 0x7fa5cc06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5c3e00640 1 -- 172.21.15.28:0/3776670785 >> v1:172.21.15.28:6789/0 conn(0x7fa5cc0add00 legacy=0x7fa5cc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5c3e00640 1 -- 172.21.15.28:0/3776670785 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc15b850 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.458+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3776670785 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2289224822 0 0) 0x7fa5cc06f920 con 0x7fa5cc0add00 2024-09-13T17:18:13.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.459+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/3776670785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc070260 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc067f80 tx=0x7fa5b402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1ebd3f5bbda522 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.459+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3776670785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b402add0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.459+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3776670785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4032440 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.459+0000 7fa5cbe00640 1 -- 172.21.15.28:0/3776670785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40327d0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 -- 172.21.15.28:0/3776670785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 msgr2=0x7fa5cc070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3776670785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc070260 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc067f80 tx=0x7fa5b402a080 comp rx=0 tx=0).stop 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 -- 172.21.15.28:0/3776670785 shutdown_connections 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3776670785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc070260 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 --2- 172.21.15.28:0/3776670785 >> v2:172.21.15.189:3300/0 conn(0x7fa5cc159770 0x7fa5cc06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 -- 172.21.15.28:0/3776670785 >> 172.21.15.28:0/3776670785 conn(0x7fa5cc0b0270 msgr2=0x7fa5cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 -- 172.21.15.28:0/3776670785 shutdown_connections 2024-09-13T17:18:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.460+0000 7fa5d3400640 1 -- 172.21.15.28:0/3776670785 wait complete. 2024-09-13T17:18:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.461+0000 7fa5d3400640 1 Processor -- start 2024-09-13T17:18:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 -- start start 2024-09-13T17:18:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 0x7fa5cc1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 0x7fa5cc1505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc157af0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc156f90 con 0x7fa5cc159770 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.462+0000 7fa5d3400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5cc0ca870 con 0x7fa5cc0add00 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34268/0 (socket says 172.21.15.28:34268) 2024-09-13T17:18:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 -- 172.21.15.28:0/4274382018 learned_addr learned my addr 172.21.15.28:0/4274382018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 0x7fa5cc1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d1400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 0x7fa5cc1505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 -- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 msgr2=0x7fa5cc1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 0x7fa5cc1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 -- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 msgr2=0x7fa5cc1505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 0x7fa5cc1505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d0a00640 1 -- 172.21.15.28:0/4274382018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009400 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 0x7fa5cc1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.463+0000 7fa5d1400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 0x7fa5cc1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5d0a00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc007e40 tx=0x7fa5bc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc014620 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5bc00ee90 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc0149c0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5cc1571c0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.464+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5cc157780 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.465+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5cc15a1e0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.466+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa5bc014b20 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.467+0000 7fa5c9600640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 0x7fa5a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.467+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa5bc0899e0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.468+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 0x7fa5a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.468+0000 7fa5c3e00640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 0x7fa5a806d0f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4034000 tx=0x7fa5b402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.471+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa5bc054eb0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.648+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa5cc06f920 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.652+0000 7fa5c9600640 1 -- 172.21.15.28:0/4274382018 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa5bc0584d0 con 0x7fa5cc06f3f0 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 msgr2=0x7fa5a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 0x7fa5a806d0f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4034000 tx=0x7fa5b402e000 comp rx=0 tx=0).stop 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 msgr2=0x7fa5cc157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc007e40 tx=0x7fa5bc00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 shutdown_connections 2024-09-13T17:18:13.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5a806ac30 0x7fa5a806d0f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5cc159770 0x7fa5cc1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5cc0add00 0x7fa5cc1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 --2- 172.21.15.28:0/4274382018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5cc06f3f0 0x7fa5cc157d70 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.654+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 >> 172.21.15.28:0/4274382018 conn(0x7fa5cc0b0270 msgr2=0x7fa5cc06c330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.655+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 shutdown_connections 2024-09-13T17:18:13.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:13.655+0000 7fa5d3400640 1 -- 172.21.15.28:0/4274382018 wait complete. 2024-09-13T17:18:13.672 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:18:13.673 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:18:13.776 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:18:13.874 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:18:13.875 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:18:13.875 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:13.911 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:18:13.911 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:18:13.911 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:13.948 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:18:13.949 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:18:13.949 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:13.988 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:18:13.988 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:18:13.988 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:14.025 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:18:14.025 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:18:14.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.180+0000 7f80b7800640 1 Processor -- start 2024-09-13T17:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.180+0000 7f80b7800640 1 -- start start 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b7800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b006f970 0x7f80b015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b7800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80b006c2d0 0x7f80b006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b0159d90 con 0x7f80b01592a0 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b7800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b015b0b0 con 0x7f80b006c2d0 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b7800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b006cca0 con 0x7f80b006f970 2024-09-13T17:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b5800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80b006c2d0 0x7f80b006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f80b01592a0 0x7f80b0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58298/0 (socket says 172.21.15.28:58298) 2024-09-13T17:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.181+0000 7f80b4e00640 1 -- 172.21.15.28:0/1373150597 learned_addr learned my addr 172.21.15.28:0/1373150597 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80afe00640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b006f970 0x7f80b015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80b5800640 1 -- 172.21.15.28:0/1373150597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b006f970 msgr2=0x7f80b015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80b5800640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b006f970 0x7f80b015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80b5800640 1 -- 172.21.15.28:0/1373150597 >> v1:172.21.15.28:6789/0 conn(0x7f80b01592a0 legacy=0x7f80b0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80b5800640 1 -- 172.21.15.28:0/1373150597 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b00b2040 con 0x7f80b006c2d0 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.182+0000 7f80af400640 1 -- 172.21.15.28:0/1373150597 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593514194 0 0) 0x7f80b0159d90 con 0x7f80b01592a0 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.183+0000 7f80b5800640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006c2d0 0x7f80b006c760 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009d90 tx=0x7f80a802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be6ce2eec3b6dec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.183+0000 7f80af400640 1 -- 172.21.15.28:0/1373150597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8007740 con 0x7f80b006c2d0 2024-09-13T17:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.183+0000 7f80af400640 1 -- 172.21.15.28:0/1373150597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80a802d050 con 0x7f80b006c2d0 2024-09-13T17:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.183+0000 7f80af400640 1 -- 172.21.15.28:0/1373150597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a8007ac0 con 0x7f80b006c2d0 2024-09-13T17:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 -- 172.21.15.28:0/1373150597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006c2d0 msgr2=0x7f80b006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006c2d0 0x7f80b006c760 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f80a8009d90 tx=0x7f80a802a660 comp rx=0 tx=0).stop 2024-09-13T17:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 -- 172.21.15.28:0/1373150597 shutdown_connections 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006c2d0 0x7f80b006c760 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 --2- 172.21.15.28:0/1373150597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b006f970 0x7f80b015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 -- 172.21.15.28:0/1373150597 >> 172.21.15.28:0/1373150597 conn(0x7f80b0069e80 msgr2=0x7f80b006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 -- 172.21.15.28:0/1373150597 shutdown_connections 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.184+0000 7f80b7800640 1 -- 172.21.15.28:0/1373150597 wait complete. 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.186+0000 7f80b7800640 1 Processor -- start 2024-09-13T17:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.186+0000 7f80b7800640 1 -- start start 2024-09-13T17:18:14.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b006c2d0 0x7f80b014b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b01592a0 0x7f80b00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b01e3070 con 0x7f80b006f970 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b01e3350 con 0x7f80b006c2d0 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b7800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b01e3630 con 0x7f80b01592a0 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40268/0 (socket says 172.21.15.28:40268) 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 -- 172.21.15.28:0/693780511 learned_addr learned my addr 172.21.15.28:0/693780511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b4e00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b006c2d0 0x7f80b014b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80b5800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b01592a0 0x7f80b00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 -- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b01592a0 msgr2=0x7f80b00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b01592a0 0x7f80b00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 -- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b006c2d0 msgr2=0x7f80b014b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b006c2d0 0x7f80b014b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.187+0000 7f80afe00640 1 -- 172.21.15.28:0/693780511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80a8009a70 con 0x7f80b006f970 2024-09-13T17:18:14.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80afe00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f809c00d800 tx=0x7f809c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00b6c0 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f809c00b820 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f809c00bbb0 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b01e3910 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.188+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b014b890 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.189+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b01503a0 con 0x7f80b006f970 2024-09-13T17:18:14.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.190+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f809c00bd10 con 0x7f80b006f970 2024-09-13T17:18:14.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.191+0000 7f80acc00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 0x7f809006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:14.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.191+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f809c0894a0 con 0x7f80b006f970 2024-09-13T17:18:14.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.191+0000 7f80b4e00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 0x7f809006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:14.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.192+0000 7f80b4e00640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 0x7f809006d0c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.195+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f80b01503a0 con 0x7f80b006f970 2024-09-13T17:18:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.414+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80b01503a0 con 0x7f80b006f970 2024-09-13T17:18:14.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.414+0000 7f80acc00640 1 -- 172.21.15.28:0/693780511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+3540 (secure 0 0 0) 0x7f80b01503a0 con 0x7f80b006f970 2024-09-13T17:18:14.416 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:14.417 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":124,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8308,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2743122768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2743122768},{"type":"v1","addr":"172.21.15.189:6836","nonce":2743122768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3648027520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3648027520},{"type":"v1","addr":"172.21.15.189:6837","nonce":3648027520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":8901,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2427937779","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2427937779},{"type":"v1","addr":"172.21.15.28:6833","nonce":2427937779}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":117},{"gid":9333,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2960511435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":2960511435},{"type":"v1","addr":"172.21.15.28:6835","nonce":2960511435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":124}],"filesystems":[]} 2024-09-13T17:18:14.417 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 124 2024-09-13T17:18:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 msgr2=0x7f809006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 0x7f809006d0c0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).stop 2024-09-13T17:18:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 msgr2=0x7f80b0151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:14.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f809c00d800 tx=0x7f809c00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 shutdown_connections 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f809006ac00 0x7f809006d0c0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80b01592a0 0x7f80b00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80b006f970 0x7f80b0151cc0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 --2- 172.21.15.28:0/693780511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80b006c2d0 0x7f80b014b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 >> 172.21.15.28:0/693780511 conn(0x7f80b0069e80 msgr2=0x7f80b0156e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 shutdown_connections 2024-09-13T17:18:14.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:14.417+0000 7f80b7800640 1 -- 172.21.15.28:0/693780511 wait complete. 2024-09-13T17:18:14.435 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:18:14.436 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:18:14.436 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:18:14.436 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:18:14.436 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:18:14.436 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:18:14.437 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:18:14.437 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:18:14.437 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:18:14.437 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:18:14.437 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:18:14.437 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:18:14.437 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:18:14.437 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:18:14.438 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:18:14.438 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:18:14.451 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:18:14.452 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:18:14.453 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:18:14.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:18:14.456 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:18:14.456 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:18:14.496 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:18:14.496 DEBUG:teuthology.parallel:result is None 2024-09-13T17:18:14.498 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:18:14.499 DEBUG:teuthology.parallel:result is None 2024-09-13T17:18:14.500 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:18:14.501 DEBUG:teuthology.parallel:result is None 2024-09-13T17:18:14.623 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:18:14.624 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:18:14.627 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:18:14.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-09-13T17:18:20.456 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:18:20.456 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:18:20.461 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:18:20.461 DEBUG:teuthology.parallel:result is None 2024-09-13T17:18:20.461 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:18:20.594 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:18:20.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.629+0000 7f4682600640 1 Processor -- start 2024-09-13T17:18:20.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.630+0000 7f4682600640 1 -- start start 2024-09-13T17:18:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4682600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f467c0b0e10 0x7f467c159a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4682600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f467c06ca80 0x7f467c0b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4682600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f467c0b32f0 con 0x7f467c0b2870 2024-09-13T17:18:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4682600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c0cbac0 con 0x7f467c06ca80 2024-09-13T17:18:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4682600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f467c0b35d0 con 0x7f467c0b0e10 2024-09-13T17:18:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4681c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f467c06ca80 0x7f467c0b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4681c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f467c06ca80 0x7f467c0b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40336/0 (socket says 172.21.15.28:40336) 2024-09-13T17:18:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4681c00640 1 -- 172.21.15.28:0/4278997641 learned_addr learned my addr 172.21.15.28:0/4278997641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.631+0000 7f4681200640 1 --2- 172.21.15.28:0/4278997641 >> v2:172.21.15.189:3300/0 conn(0x7f467c0b0e10 0x7f467c159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.632+0000 7f4681c00640 1 -- 172.21.15.28:0/4278997641 >> v2:172.21.15.189:3300/0 conn(0x7f467c0b0e10 msgr2=0x7f467c159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.632+0000 7f4681c00640 1 --2- 172.21.15.28:0/4278997641 >> v2:172.21.15.189:3300/0 conn(0x7f467c0b0e10 0x7f467c159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.632+0000 7f4681c00640 1 -- 172.21.15.28:0/4278997641 >> v1:172.21.15.189:6789/0 conn(0x7f467c0b2870 legacy=0x7f467c06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.632+0000 7f4681c00640 1 -- 172.21.15.28:0/4278997641 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c0b38b0 con 0x7f467c06ca80 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.633+0000 7f4681c00640 1 --2- 172.21.15.28:0/4278997641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c06ca80 0x7f467c0b2ce0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f467c06cf00 tx=0x7f467402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5269561b39845195 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.633+0000 7f467be00640 1 -- 172.21.15.28:0/4278997641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674031600 con 0x7f467c06ca80 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.633+0000 7f467be00640 1 -- 172.21.15.28:0/4278997641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467402d050 con 0x7f467c06ca80 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.633+0000 7f467be00640 1 -- 172.21.15.28:0/4278997641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674031980 con 0x7f467c06ca80 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 -- 172.21.15.28:0/4278997641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c06ca80 msgr2=0x7f467c0b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 --2- 172.21.15.28:0/4278997641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c06ca80 0x7f467c0b2ce0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f467c06cf00 tx=0x7f467402a6a0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 -- 172.21.15.28:0/4278997641 shutdown_connections 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 --2- 172.21.15.28:0/4278997641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c06ca80 0x7f467c0b2ce0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 --2- 172.21.15.28:0/4278997641 >> v2:172.21.15.189:3300/0 conn(0x7f467c0b0e10 0x7f467c159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 -- 172.21.15.28:0/4278997641 >> 172.21.15.28:0/4278997641 conn(0x7f467c073470 msgr2=0x7f467c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 -- 172.21.15.28:0/4278997641 shutdown_connections 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.634+0000 7f4682600640 1 -- 172.21.15.28:0/4278997641 wait complete. 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.635+0000 7f4682600640 1 Processor -- start 2024-09-13T17:18:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.636+0000 7f4682600640 1 -- start start 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.636+0000 7f4682600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.636+0000 7f4682600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f467c0b0e10 0x7f467c06f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.636+0000 7f4682600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 0x7f467c13dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.636+0000 7f4682600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c070340 con 0x7f467c0b2870 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4682600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467c13b900 con 0x7f467c06ca80 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4682600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467c1389d0 con 0x7f467c0b0e10 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 0x7f467c13dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:38284/0 (socket says 172.21.15.28:38284) 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681200640 1 -- 172.21.15.28:0/4006157148 learned_addr learned my addr 172.21.15.28:0/4006157148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4673e00640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f467c0b0e10 0x7f467c06f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 -- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f467c0b0e10 msgr2=0x7f467c06f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f467c0b0e10 0x7f467c06f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 -- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 msgr2=0x7f467c1ef740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681c00640 1 -- 172.21.15.28:0/4006157148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674009b30 con 0x7f467c0b2870 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.637+0000 7f4681200640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4681c00640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 0x7f467c13dfd0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f4674030000 tx=0x7f4674004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467403d860 con 0x7f467c0b2870 2024-09-13T17:18:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467403d9c0 con 0x7f467c0b2870 2024-09-13T17:18:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467c1efc80 con 0x7f467c0b2870 2024-09-13T17:18:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467403ddb0 con 0x7f467c0b2870 2024-09-13T17:18:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.638+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467c06faa0 con 0x7f467c0b2870 2024-09-13T17:18:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.639+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467c138730 con 0x7f467c0b2870 2024-09-13T17:18:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.640+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f467403b400 con 0x7f467c0b2870 2024-09-13T17:18:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.641+0000 7f4679600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 0x7f465806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.641+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f46740a9d90 con 0x7f467c0b2870 2024-09-13T17:18:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.641+0000 7f4681200640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 0x7f465806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.642+0000 7f4681200640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 0x7f465806d0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f466c0112c0 tx=0x7f466c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.646+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f467c138730 con 0x7f467c0b2870 2024-09-13T17:18:20.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:20.838+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f467c138b10 con 0x7f467c0b2870 2024-09-13T17:18:21.371 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-09-13T17:18:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.613+0000 7f4679600640 1 -- 172.21.15.28:0/4006157148 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v149) v1 ==== 97+0+0 (secure 0 0 0) 0x7f467c138b10 con 0x7f467c0b2870 2024-09-13T17:18:21.616 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:18:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 msgr2=0x7f465806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 0x7f465806d0c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f466c0112c0 tx=0x7f466c009190 comp rx=0 tx=0).stop 2024-09-13T17:18:21.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 msgr2=0x7f467c13dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 0x7f467c13dfd0 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f4674030000 tx=0x7f4674004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 shutdown_connections 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465806ac00 0x7f465806d0c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f467c0b2870 0x7f467c13dfd0 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f467c0b0e10 0x7f467c06f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 --2- 172.21.15.28:0/4006157148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467c06ca80 0x7f467c1ef740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 >> 172.21.15.28:0/4006157148 conn(0x7f467c073470 msgr2=0x7f467c0cd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 shutdown_connections 2024-09-13T17:18:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.616+0000 7f4682600640 1 -- 172.21.15.28:0/4006157148 wait complete. 2024-09-13T17:18:21.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:18:21.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.794+0000 7f16bda00640 1 Processor -- start 2024-09-13T17:18:21.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.794+0000 7f16bda00640 1 -- start start 2024-09-13T17:18:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.795+0000 7f16bda00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f16b81592a0 0x7f16b815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.795+0000 7f16bda00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f16b806c2d0 0x7f16b806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.795+0000 7f16bda00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b806cc70 con 0x7f16b806f970 2024-09-13T17:18:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.795+0000 7f16bda00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b815b0b0 con 0x7f16b806c2d0 2024-09-13T17:18:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.795+0000 7f16bda00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b8159d90 con 0x7f16b81592a0 2024-09-13T17:18:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16ae000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f16b806f970 0x7f16b8159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58306/0 (socket says 172.21.15.28:58306) 2024-09-13T17:18:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16ae000640 1 -- 172.21.15.28:0/2417722422 learned_addr learned my addr 172.21.15.28:0/2417722422 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f16b806c2d0 0x7f16b806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b6a00640 1 --2- 172.21.15.28:0/2417722422 >> v2:172.21.15.189:3300/0 conn(0x7f16b81592a0 0x7f16b815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b7400640 1 -- 172.21.15.28:0/2417722422 >> v2:172.21.15.189:3300/0 conn(0x7f16b81592a0 msgr2=0x7f16b815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b7400640 1 --2- 172.21.15.28:0/2417722422 >> v2:172.21.15.189:3300/0 conn(0x7f16b81592a0 0x7f16b815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b6000640 1 -- 172.21.15.28:0/2417722422 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179038333 0 0) 0x7f16b806cc70 con 0x7f16b806f970 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b7400640 1 -- 172.21.15.28:0/2417722422 >> v1:172.21.15.28:6789/0 conn(0x7f16b806f970 legacy=0x7f16b8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.796+0000 7f16b7400640 1 -- 172.21.15.28:0/2417722422 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80b2040 con 0x7f16b806c2d0 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.797+0000 7f16b7400640 1 --2- 172.21.15.28:0/2417722422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806c2d0 0x7f16b806c730 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f16b806d370 tx=0x7f16b002c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ec10da96df05a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16b6000640 1 -- 172.21.15.28:0/2417722422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0032790 con 0x7f16b806c2d0 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16b6000640 1 -- 172.21.15.28:0/2417722422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b00328f0 con 0x7f16b806c2d0 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16b6000640 1 -- 172.21.15.28:0/2417722422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b0032c80 con 0x7f16b806c2d0 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 -- 172.21.15.28:0/2417722422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806c2d0 msgr2=0x7f16b806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 --2- 172.21.15.28:0/2417722422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806c2d0 0x7f16b806c730 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f16b806d370 tx=0x7f16b002c890 comp rx=0 tx=0).stop 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 -- 172.21.15.28:0/2417722422 shutdown_connections 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 --2- 172.21.15.28:0/2417722422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806c2d0 0x7f16b806c730 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 --2- 172.21.15.28:0/2417722422 >> v2:172.21.15.189:3300/0 conn(0x7f16b81592a0 0x7f16b815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 -- 172.21.15.28:0/2417722422 >> 172.21.15.28:0/2417722422 conn(0x7f16b8069e80 msgr2=0x7f16b806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 -- 172.21.15.28:0/2417722422 shutdown_connections 2024-09-13T17:18:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.798+0000 7f16bda00640 1 -- 172.21.15.28:0/2417722422 wait complete. 2024-09-13T17:18:21.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.800+0000 7f16bda00640 1 Processor -- start 2024-09-13T17:18:21.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.800+0000 7f16bda00640 1 -- start start 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 0x7f16b80afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 0x7f16b80ce7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80c35f0 con 0x7f16b806f970 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b0100 con 0x7f16b81592a0 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16bda00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80b0330 con 0x7f16b806c2d0 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16ae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16ae000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40364/0 (socket says 172.21.15.28:40364) 2024-09-13T17:18:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.801+0000 7f16ae000640 1 -- 172.21.15.28:0/2419860937 learned_addr learned my addr 172.21.15.28:0/2419860937 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16b7400640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 0x7f16b80ce7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 -- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 msgr2=0x7f16b80afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16b6a00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 0x7f16b80afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 0x7f16b80afbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 -- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 msgr2=0x7f16b80ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 0x7f16b80ce7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 -- 172.21.15.28:0/2419860937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0009a70 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16b7400640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 0x7f16b80ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16b6a00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 0x7f16b80afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16ae000640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c4d0 tx=0x7f16a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a00074e0 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0007660 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0004a20 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b80b0610 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.802+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b80b0bd0 con 0x7f16b806f970 2024-09-13T17:18:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.803+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80b3740 con 0x7f16b806f970 2024-09-13T17:18:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.804+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f16a0004b80 con 0x7f16b806f970 2024-09-13T17:18:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.805+0000 7f16af400640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 0x7f169406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.805+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f16a0088290 con 0x7f16b806f970 2024-09-13T17:18:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.805+0000 7f16b6a00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 0x7f169406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.806+0000 7f16b6a00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 0x7f169406d0c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006fd0 tx=0x7f16a8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:21.810+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f16a0053720 con 0x7f16b806f970 2024-09-13T17:18:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.051+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f16b80ad6e0 con 0x7f16b806f970 2024-09-13T17:18:22.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.051+0000 7f16af400640 1 -- 172.21.15.28:0/2419860937 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f16a0056d40 con 0x7f16b806f970 2024-09-13T17:18:22.053 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.054 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:18:22.055 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:18:22.056 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:18:22.057 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.058 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.059 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.060 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.061 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.062 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.063 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.064 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:18:22.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:18:22.067 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.053+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 msgr2=0x7f169406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.053+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 0x7f169406d0c0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f16a8006fd0 tx=0x7f16a8008040 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.053+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 msgr2=0x7f16b80ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.053+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c4d0 tx=0x7f16a000c990 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 shutdown_connections 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f169406ac00 0x7f169406d0c0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16b81592a0 0x7f16b80ce7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16b806f970 0x7f16b80ce260 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 --2- 172.21.15.28:0/2419860937 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16b806c2d0 0x7f16b80afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 >> 172.21.15.28:0/2419860937 conn(0x7f16b8069e80 msgr2=0x7f16b8159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 shutdown_connections 2024-09-13T17:18:22.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.054+0000 7f16bda00640 1 -- 172.21.15.28:0/2419860937 wait complete. 2024-09-13T17:18:22.072 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.226+0000 7f4675e00640 1 Processor -- start 2024-09-13T17:18:22.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.226+0000 7f4675e00640 1 -- start start 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4675e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f46700b0b50 0x7f46700b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4675e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46700b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4675e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46700c3850 con 0x7f4670159440 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4675e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46700b2d60 con 0x7f46700b0b50 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4675e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467015a930 con 0x7f4670159820 2024-09-13T17:18:22.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4674a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4670159440 0x7f46700b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58312/0 (socket says 172.21.15.28:58312) 2024-09-13T17:18:22.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.227+0000 7f4674a00640 1 -- 172.21.15.28:0/1433132137 learned_addr learned my addr 172.21.15.28:0/1433132137 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054194184 0 0) 0x7f46700c3850 con 0x7f4670159440 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f4675400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46700b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4658003520 con 0x7f4670159440 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466fe00640 1 --2- 172.21.15.28:0/1433132137 >> v2:172.21.15.189:3300/0 conn(0x7f46700b0b50 0x7f46700b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506977247 0 0) 0x7f4658003520 con 0x7f4670159440 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 msgr2=0x7f46700b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46700b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 >> v2:172.21.15.189:3300/0 conn(0x7f46700b0b50 msgr2=0x7f46700b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 --2- 172.21.15.28:0/1433132137 >> v2:172.21.15.189:3300/0 conn(0x7f46700b0b50 0x7f46700b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4670159d20 con 0x7f4670159440 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4664001670 con 0x7f4670159440 2024-09-13T17:18:22.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 >> v1:172.21.15.28:6789/0 conn(0x7f4670159440 legacy=0x7f46700b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f4658003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4658004d00 0x7f4658005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46700b2d60 con 0x7f4658003970 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467015a930 con 0x7f4658004d00 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.228+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465800b980 con 0x7f4670159820 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f4658003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f466fe00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4675400640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4658004d00 0x7f4658005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 -- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 msgr2=0x7f46580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4670159820 0x7f46580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 -- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4658004d00 msgr2=0x7f4658005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4658004d00 0x7f4658005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.229+0000 7f4674a00640 1 -- 172.21.15.28:0/1433132137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4658003520 con 0x7f4658003970 2024-09-13T17:18:22.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.230+0000 7f4674a00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f4658003d90 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f46580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ffa8a1433512f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.230+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664005ad0 con 0x7f4658003970 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.230+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4664005c30 con 0x7f4658003970 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.230+0000 7f466f400640 1 -- 172.21.15.28:0/1433132137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664007070 con 0x7f4658003970 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 -- 172.21.15.28:0/1433132137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 msgr2=0x7f4658003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 --2- 172.21.15.28:0/1433132137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f4658003d90 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f46580057f0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675400640 1 -- 172.21.15.28:0/1433132137 reap_dead start 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 -- 172.21.15.28:0/1433132137 shutdown_connections 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 -- 172.21.15.28:0/1433132137 >> 172.21.15.28:0/1433132137 conn(0x7f4670072e40 msgr2=0x7f4670073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 -- 172.21.15.28:0/1433132137 shutdown_connections 2024-09-13T17:18:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.231+0000 7f4675e00640 1 -- 172.21.15.28:0/1433132137 wait complete. 2024-09-13T17:18:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.233+0000 7f4675e00640 1 Processor -- start 2024-09-13T17:18:22.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.233+0000 7f4675e00640 1 -- start start 2024-09-13T17:18:22.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 0x7f467015adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 0x7f4670152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467014f7c0 con 0x7f4658003970 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f467015b810 con 0x7f467015b300 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f467015baf0 con 0x7f46701441d0 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40382/0 (socket says 172.21.15.28:40382) 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 -- 172.21.15.28:0/1700509990 learned_addr learned my addr 172.21.15.28:0/1700509990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 -- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 msgr2=0x7f467015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4675400640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 0x7f4670152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f466fe00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 0x7f467015adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 0x7f467015adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 -- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 msgr2=0x7f4670152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.234+0000 7f4674a00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 0x7f4670152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f4674a00640 1 -- 172.21.15.28:0/1700509990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467015bdd0 con 0x7f4658003970 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f4675400640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 0x7f4670152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f466fe00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 0x7f467015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f4674a00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f4664001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4664007480 con 0x7f4658003970 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46640075e0 con 0x7f4658003970 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466403e670 con 0x7f4658003970 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f467014f3e0 con 0x7f4658003970 2024-09-13T17:18:22.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.235+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f467015c480 con 0x7f4658003970 2024-09-13T17:18:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.236+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f467014fa30 con 0x7f4658003970 2024-09-13T17:18:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.237+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f466403e7d0 con 0x7f4658003970 2024-09-13T17:18:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.238+0000 7f466cc00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 0x7f465006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.238+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f46640a9fc0 con 0x7f4658003970 2024-09-13T17:18:22.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.239+0000 7f466fe00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 0x7f465006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.239+0000 7f466fe00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 0x7f465006d0c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4670150db0 tx=0x7f465c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.242+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46640dd9c0 con 0x7f4658003970 2024-09-13T17:18:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.421+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467015bc30 con 0x7f4658003970 2024-09-13T17:18:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.422+0000 7f466cc00640 1 -- 172.21.15.28:0/1700509990 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f467015bc30 con 0x7f4658003970 2024-09-13T17:18:22.424 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:22.424 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":149,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:21.607121+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":140,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:22.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 msgr2=0x7f465006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 0x7f465006d0c0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4670150db0 tx=0x7f465c008040 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 msgr2=0x7f46701f20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f466400c240 tx=0x7f4664001a00 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 shutdown_connections 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f465006ac00 0x7f465006d0c0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f467015b300 0x7f4670152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46701441d0 0x7f467015adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 --2- 172.21.15.28:0/1700509990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4658003970 0x7f46701f20c0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 >> 172.21.15.28:0/1700509990 conn(0x7f4670072e40 msgr2=0x7f46701571c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 shutdown_connections 2024-09-13T17:18:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.424+0000 7f4675e00640 1 -- 172.21.15.28:0/1700509990 wait complete. 2024-09-13T17:18:22.443 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:18:22.443 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:18:22.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.599+0000 7f9406000640 1 Processor -- start 2024-09-13T17:18:22.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.599+0000 7f9406000640 1 -- start start 2024-09-13T17:18:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f9406000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9400159770 0x7f940006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f9406000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f940006f3f0 0x7f9400070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f9406000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f940006f920 con 0x7f94000add00 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f9406000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940015b290 con 0x7f9400159770 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f9406000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940015b570 con 0x7f940006f3f0 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f93ff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9400159770 0x7f940006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f93ff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9400159770 0x7f940006fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40392/0 (socket says 172.21.15.28:40392) 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f93ff400640 1 -- 172.21.15.28:0/2675758037 learned_addr learned my addr 172.21.15.28:0/2675758037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f93fea00640 1 --2- 172.21.15.28:0/2675758037 >> v2:172.21.15.189:3300/0 conn(0x7f940006f3f0 0x7f9400070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.601+0000 7f93ff400640 1 -- 172.21.15.28:0/2675758037 >> v2:172.21.15.189:3300/0 conn(0x7f940006f3f0 msgr2=0x7f9400070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.602+0000 7f93ff400640 1 --2- 172.21.15.28:0/2675758037 >> v2:172.21.15.189:3300/0 conn(0x7f940006f3f0 0x7f9400070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.602+0000 7f93ff400640 1 -- 172.21.15.28:0/2675758037 >> v1:172.21.15.189:6789/0 conn(0x7f94000add00 legacy=0x7f9400159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:22.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.602+0000 7f93ff400640 1 -- 172.21.15.28:0/2675758037 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f940015b850 con 0x7f9400159770 2024-09-13T17:18:22.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f93ff400640 1 --2- 172.21.15.28:0/2675758037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400159770 0x7f940006fd20 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f940006b950 tx=0x7f93f002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87a26b8dd1fed612 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f93fe000640 1 -- 172.21.15.28:0/2675758037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f002ae20 con 0x7f9400159770 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f93fe000640 1 -- 172.21.15.28:0/2675758037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f0032440 con 0x7f9400159770 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f93fe000640 1 -- 172.21.15.28:0/2675758037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f00327d0 con 0x7f9400159770 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 -- 172.21.15.28:0/2675758037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400159770 msgr2=0x7f940006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 --2- 172.21.15.28:0/2675758037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400159770 0x7f940006fd20 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f940006b950 tx=0x7f93f002a110 comp rx=0 tx=0).stop 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 -- 172.21.15.28:0/2675758037 shutdown_connections 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 --2- 172.21.15.28:0/2675758037 >> v2:172.21.15.189:3300/0 conn(0x7f940006f3f0 0x7f9400070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 --2- 172.21.15.28:0/2675758037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9400159770 0x7f940006fd20 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.603+0000 7f9406000640 1 -- 172.21.15.28:0/2675758037 >> 172.21.15.28:0/2675758037 conn(0x7f94000b0270 msgr2=0x7f94000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.604+0000 7f9406000640 1 -- 172.21.15.28:0/2675758037 shutdown_connections 2024-09-13T17:18:22.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.604+0000 7f9406000640 1 -- 172.21.15.28:0/2675758037 wait complete. 2024-09-13T17:18:22.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.605+0000 7f9406000640 1 Processor -- start 2024-09-13T17:18:22.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.605+0000 7f9406000640 1 -- start start 2024-09-13T17:18:22.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940006f3f0 0x7f9400158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 0x7f940014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9400147ae0 con 0x7f94000add00 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9400153120 con 0x7f940006f3f0 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f9406000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94001527f0 con 0x7f9400159770 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f93fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f93fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40394/0 (socket says 172.21.15.28:40394) 2024-09-13T17:18:22.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.606+0000 7f93fea00640 1 -- 172.21.15.28:0/707069322 learned_addr learned my addr 172.21.15.28:0/707069322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93ff400640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940006f3f0 0x7f9400158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93ffe00640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 0x7f940014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 -- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 msgr2=0x7f940014f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 0x7f940014f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 -- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940006f3f0 msgr2=0x7f9400158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940006f3f0 0x7f9400158c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 -- 172.21.15.28:0/707069322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f00094d0 con 0x7f94000add00 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93ffe00640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 0x7f940014f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93fea00640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00d5f0 tx=0x7f93ec00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec0125f0 con 0x7f94000add00 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ec012750 con 0x7f94000add00 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ec012b00 con 0x7f94000add00 2024-09-13T17:18:22.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.607+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9400152ad0 con 0x7f94000add00 2024-09-13T17:18:22.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.608+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9400153af0 con 0x7f94000add00 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.609+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9400070850 con 0x7f94000add00 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.609+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f93ec012c60 con 0x7f94000add00 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.610+0000 7f93f7400640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 0x7f93e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.610+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f93ec089540 con 0x7f94000add00 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.611+0000 7f93ff400640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 0x7f93e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.611+0000 7f93ff400640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 0x7f93e006d0c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f93f0034000 tx=0x7f93f002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.614+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f93ec054940 con 0x7f94000add00 2024-09-13T17:18:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.791+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f940006fd20 con 0x7f94000add00 2024-09-13T17:18:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.792+0000 7f93f7400640 1 -- 172.21.15.28:0/707069322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v149) v1 ==== 93+0+10 (secure 0 0 0) 0x7f93ec057f60 con 0x7f94000add00 2024-09-13T17:18:22.794 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 msgr2=0x7f93e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 0x7f93e006d0c0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f93f0034000 tx=0x7f93f002e000 comp rx=0 tx=0).stop 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 msgr2=0x7f94001535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00d5f0 tx=0x7f93ec00dab0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 shutdown_connections 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93e006ac00 0x7f93e006d0c0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9400159770 0x7f940014f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94000add00 0x7f94001535b0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 --2- 172.21.15.28:0/707069322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f940006f3f0 0x7f9400158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 >> 172.21.15.28:0/707069322 conn(0x7f94000b0270 msgr2=0x7f9400159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 shutdown_connections 2024-09-13T17:18:22.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.794+0000 7f9406000640 1 -- 172.21.15.28:0/707069322 wait complete. 2024-09-13T17:18:22.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:18:22.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.966+0000 7f29a4c00640 1 Processor -- start 2024-09-13T17:18:22.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.966+0000 7f29a4c00640 1 -- start start 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f29a4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f29a4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a015ab60 con 0x7f29a006f3f0 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f29a4c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a006f920 con 0x7f29a0159770 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f29a4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a015b290 con 0x7f29a0070430 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f299e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29a006f3f0 0x7f29a015c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58328/0 (socket says 172.21.15.28:58328) 2024-09-13T17:18:22.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f299e800640 1 -- 172.21.15.28:0/1652075841 learned_addr learned my addr 172.21.15.28:0/1652075841 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f2997e00640 1 --2- 172.21.15.28:0/1652075841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.968+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350057451 0 0) 0x7f2998001490 con 0x7f29a006f3f0 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2984003520 con 0x7f29a006f3f0 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1585962092 0 0) 0x7f29a006f920 con 0x7f29a0159770 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2998001490 con 0x7f29a0159770 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f2997e00640 1 -- 172.21.15.28:0/1652075841 >> v1:172.21.15.189:6789/0 conn(0x7f29a0159770 legacy=0x7f29a006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f2997e00640 1 -- 172.21.15.28:0/1652075841 >> v1:172.21.15.28:6789/0 conn(0x7f29a006f3f0 legacy=0x7f29a015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.969+0000 7f2997e00640 1 -- 172.21.15.28:0/1652075841 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a015b570 con 0x7f29a0070430 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.970+0000 7f2997e00640 1 --2- 172.21.15.28:0/1652075841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a015bd70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f29880097d0 tx=0x7f298802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221788c565cafe36 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.970+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2988032440 con 0x7f29a0070430 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.970+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29880325a0 con 0x7f29a0070430 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.970+0000 7f299d400640 1 -- 172.21.15.28:0/1652075841 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2988032930 con 0x7f29a0070430 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 -- 172.21.15.28:0/1652075841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 msgr2=0x7f29a015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1652075841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a015bd70 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f29880097d0 tx=0x7f298802a180 comp rx=0 tx=0).stop 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 -- 172.21.15.28:0/1652075841 shutdown_connections 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1652075841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a015bd70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 -- 172.21.15.28:0/1652075841 >> 172.21.15.28:0/1652075841 conn(0x7f29a00b0270 msgr2=0x7f29a00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 -- 172.21.15.28:0/1652075841 shutdown_connections 2024-09-13T17:18:22.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.971+0000 7f29a4c00640 1 -- 172.21.15.28:0/1652075841 wait complete. 2024-09-13T17:18:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.972+0000 7f29a4c00640 1 Processor -- start 2024-09-13T17:18:22.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.973+0000 7f29a4c00640 1 -- start start 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.973+0000 7f29a4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.973+0000 7f29a4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a0150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f29a4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 0x7f29a01512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f29a4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a014d830 con 0x7f29a006f3f0 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f29a4c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0156c10 con 0x7f29a0159770 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f29a4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a0156ef0 con 0x7f29a0070430 2024-09-13T17:18:22.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40410/0 (socket says 172.21.15.28:40410) 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 -- 172.21.15.28:0/1756162609 learned_addr learned my addr 172.21.15.28:0/1756162609 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 -- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 msgr2=0x7f29a0150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299e800640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 0x7f29a01512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f2997e00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a0150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a0150d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 -- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 msgr2=0x7f29a01512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 0x7f29a01512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299de00640 1 -- 172.21.15.28:0/1756162609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2988009400 con 0x7f29a006f3f0 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f2997e00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a0150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.974+0000 7f299e800640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 0x7f29a01512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:22.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f299de00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f299000c430 tx=0x7f299000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29900052d0 con 0x7f29a006f3f0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2990005450 con 0x7f29a006f3f0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a01571d0 con 0x7f29a006f3f0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2990003830 con 0x7f29a006f3f0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.975+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a0157790 con 0x7f29a006f3f0 2024-09-13T17:18:22.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.976+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a006f920 con 0x7f29a006f3f0 2024-09-13T17:18:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.977+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2990003990 con 0x7f29a006f3f0 2024-09-13T17:18:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.978+0000 7f2996000640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.978+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2990088680 con 0x7f29a006f3f0 2024-09-13T17:18:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.978+0000 7f2997e00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.979+0000 7f2997e00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 0x7f297c06d0c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2988034000 tx=0x7f298802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:22.982+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2990053ba0 con 0x7f29a006f3f0 2024-09-13T17:18:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.161+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f29a00af5e0 con 0x7f29a006f3f0 2024-09-13T17:18:23.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.162+0000 7f2996000640 1 -- 172.21.15.28:0/1756162609 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v149) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2990017060 con 0x7f29a006f3f0 2024-09-13T17:18:23.164 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:23.165 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:18:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 msgr2=0x7f297c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 0x7f297c06d0c0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2988034000 tx=0x7f298802e000 comp rx=0 tx=0).stop 2024-09-13T17:18:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 msgr2=0x7f29a014d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7f299000c430 tx=0x7f299000c8f0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 shutdown_connections 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29a0159770 0x7f29a01512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29a0070430 0x7f29a0150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 --2- 172.21.15.28:0/1756162609 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29a006f3f0 0x7f29a014d2f0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 >> 172.21.15.28:0/1756162609 conn(0x7f29a00b0270 msgr2=0x7f29a006fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 shutdown_connections 2024-09-13T17:18:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.164+0000 7f29a4c00640 1 -- 172.21.15.28:0/1756162609 wait complete. 2024-09-13T17:18:23.182 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:18:23.182 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:18:23.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.340+0000 7f0cdfe00640 1 Processor -- start 2024-09-13T17:18:23.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.340+0000 7f0cdfe00640 1 -- start start 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cdfe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0ce0159440 0x7f0ce006b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cdfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ce006b7f0 0x7f0ce00b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cdfe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce006bc40 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cdfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00b46a0 con 0x7f0ce006b7f0 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cdfe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ce00b3600 con 0x7f0ce0159440 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ce006b7f0 0x7f0ce00b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ce006b7f0 0x7f0ce00b2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40414/0 (socket says 172.21.15.28:40414) 2024-09-13T17:18:23.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.341+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2955025944 learned_addr learned my addr 172.21.15.28:0/2955025944 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.342+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2955025944 >> v2:172.21.15.189:3300/0 conn(0x7f0ce0159440 msgr2=0x7f0ce006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.342+0000 7f0cd7e00640 1 --2- 172.21.15.28:0/2955025944 >> v2:172.21.15.189:3300/0 conn(0x7f0ce0159440 0x7f0ce006b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.342+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2955025944 >> v1:172.21.15.189:6789/0 conn(0x7f0ce00b2dd0 legacy=0x7f0ce00caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:23.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.342+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2955025944 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce00b38e0 con 0x7f0ce006b7f0 2024-09-13T17:18:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cd7e00640 1 --2- 172.21.15.28:0/2955025944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce006b7f0 0x7f0ce00b2890 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f0ce006d3d0 tx=0x7f0cc802c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a319abbf48799c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:23.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cde000640 1 -- 172.21.15.28:0/2955025944 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8031710 con 0x7f0ce006b7f0 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cde000640 1 -- 172.21.15.28:0/2955025944 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc802f050 con 0x7f0ce006b7f0 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cde000640 1 -- 172.21.15.28:0/2955025944 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc8031a90 con 0x7f0ce006b7f0 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2955025944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce006b7f0 msgr2=0x7f0ce00b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2955025944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce006b7f0 0x7f0ce00b2890 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f0ce006d3d0 tx=0x7f0cc802c440 comp rx=0 tx=0).stop 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2955025944 shutdown_connections 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.343+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2955025944 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce006b7f0 0x7f0ce00b2890 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.344+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2955025944 >> v2:172.21.15.189:3300/0 conn(0x7f0ce0159440 0x7f0ce006b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.344+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2955025944 >> 172.21.15.28:0/2955025944 conn(0x7f0ce0069e80 msgr2=0x7f0ce006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.344+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2955025944 shutdown_connections 2024-09-13T17:18:23.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.344+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2955025944 wait complete. 2024-09-13T17:18:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.345+0000 7f0cdfe00640 1 Processor -- start 2024-09-13T17:18:23.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.345+0000 7f0cdfe00640 1 -- start start 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ce006b7f0 0x7f0ce01497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 0x7f0ce0144950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00c8260 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00c7970 con 0x7f0ce0159440 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.346+0000 7f0cdfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ce00c7c10 con 0x7f0ce006b7f0 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:23.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40422/0 (socket says 172.21.15.28:40422) 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2592927907 learned_addr learned my addr 172.21.15.28:0/2592927907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cdea00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ce006b7f0 0x7f0ce01497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cdf400640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 0x7f0ce0144950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ce006b7f0 msgr2=0x7f0ce01497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ce006b7f0 0x7f0ce01497a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 msgr2=0x7f0ce0144950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 0x7f0ce0144950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 -- 172.21.15.28:0/2592927907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc80094d0 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cdf400640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 0x7f0ce0144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:23.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.347+0000 7f0cd7e00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8000c00 tx=0x7f0cc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.348+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc803e830 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.348+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cc803e990 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.348+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce00c7ef0 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.348+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc803ed20 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.348+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce0144e90 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.349+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ce0067a10 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.350+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0cc803a040 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.351+0000 7f0cd6a00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 0x7f0cbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:23.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.351+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0cc80a9e80 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.351+0000 7f0cdea00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 0x7f0cbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:23.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.352+0000 7f0cdea00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 0x7f0cbc06d0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000b810 tx=0x7f0cd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:23.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.355+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0cc8075290 con 0x7f0ce00b2dd0 2024-09-13T17:18:23.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:23.533+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0ce006bc40 con 0x7f0ce00b2dd0 2024-09-13T17:18:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.119+0000 7f0cd6a00640 1 -- 172.21.15.28:0/2592927907 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v150) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0ce006bc40 con 0x7f0ce00b2dd0 2024-09-13T17:18:24.121 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 msgr2=0x7f0cbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 0x7f0cbc06d0c0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000b810 tx=0x7f0cd0008040 comp rx=0 tx=0).stop 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 msgr2=0x7f0ce00c93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8000c00 tx=0x7f0cc80046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 shutdown_connections 2024-09-13T17:18:24.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0cbc06ac00 0x7f0cbc06d0c0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ce0159440 0x7f0ce0144950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ce00b2dd0 0x7f0ce00c93e0 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 --2- 172.21.15.28:0/2592927907 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ce006b7f0 0x7f0ce01497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.124+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 >> 172.21.15.28:0/2592927907 conn(0x7f0ce0069e80 msgr2=0x7f0ce00cb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:24.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.125+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 shutdown_connections 2024-09-13T17:18:24.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.125+0000 7f0cdfe00640 1 -- 172.21.15.28:0/2592927907 wait complete. 2024-09-13T17:18:24.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:18:24.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.309+0000 7fc20b200640 1 Processor -- start 2024-09-13T17:18:24.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.309+0000 7fc20b200640 1 -- start start 2024-09-13T17:18:24.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc20b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204070430 0x7fc20415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc20b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc20406f3f0 0x7fc20415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc20b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20415ab60 con 0x7fc204159770 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc20b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20406f920 con 0x7fc20406f3f0 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc20b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20415b290 con 0x7fc204070430 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc20406f3f0 0x7fc20415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc20406f3f0 0x7fc20415c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 -- 172.21.15.28:0/1187107328 learned_addr learned my addr 172.21.15.28:0/1187107328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc203400640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204070430 0x7fc20415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 -- 172.21.15.28:0/1187107328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204070430 msgr2=0x7fc20415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:24.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204070430 0x7fc20415bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 -- 172.21.15.28:0/1187107328 >> v1:172.21.15.189:6789/0 conn(0x7fc204159770 legacy=0x7fc20406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc202a00640 1 -- 172.21.15.28:0/1187107328 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753624569 0 0) 0x7fc1f4001110 con 0x7fc204159770 2024-09-13T17:18:24.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.310+0000 7fc209200640 1 -- 172.21.15.28:0/1187107328 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20415b570 con 0x7fc20406f3f0 2024-09-13T17:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.311+0000 7fc209200640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc20415c2b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009e50 tx=0x7fc1fc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24abd36386d7ae2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.311+0000 7fc202a00640 1 -- 172.21.15.28:0/1187107328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc02aea0 con 0x7fc20406f3f0 2024-09-13T17:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.311+0000 7fc202a00640 1 -- 172.21.15.28:0/1187107328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1fc02d050 con 0x7fc20406f3f0 2024-09-13T17:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.311+0000 7fc202a00640 1 -- 172.21.15.28:0/1187107328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1fc031820 con 0x7fc20406f3f0 2024-09-13T17:18:24.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 -- 172.21.15.28:0/1187107328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 msgr2=0x7fc20415c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc20415c2b0 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fc1fc009e50 tx=0x7fc1fc02a680 comp rx=0 tx=0).stop 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 -- 172.21.15.28:0/1187107328 shutdown_connections 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc20415c2b0 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 --2- 172.21.15.28:0/1187107328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204070430 0x7fc20415bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 -- 172.21.15.28:0/1187107328 >> 172.21.15.28:0/1187107328 conn(0x7fc2040b0270 msgr2=0x7fc2040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 -- 172.21.15.28:0/1187107328 shutdown_connections 2024-09-13T17:18:24.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.312+0000 7fc20b200640 1 -- 172.21.15.28:0/1187107328 wait complete. 2024-09-13T17:18:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.314+0000 7fc20b200640 1 Processor -- start 2024-09-13T17:18:24.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.314+0000 7fc20b200640 1 -- start start 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc204070430 0x7fc204153fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204159770 0x7fc20414d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20414bae0 con 0x7fc20406f3f0 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040cc540 con 0x7fc204070430 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc20b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2040cc7e0 con 0x7fc204159770 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53526/0 (socket says 172.21.15.28:53526) 2024-09-13T17:18:24.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 -- 172.21.15.28:0/1164611193 learned_addr learned my addr 172.21.15.28:0/1164611193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203400640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc204070430 0x7fc204153fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc209200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204159770 0x7fc20414d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 -- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204159770 msgr2=0x7fc20414d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204159770 0x7fc20414d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 -- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc204070430 msgr2=0x7fc204153fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc204070430 0x7fc204153fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.315+0000 7fc203e00640 1 -- 172.21.15.28:0/1164611193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1fc009b30 con 0x7fc20406f3f0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc203e00640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c410 tx=0x7fc1f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f40035f0 con 0x7fc20406f3f0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f4003770 con 0x7fc20406f3f0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4003b20 con 0x7fc20406f3f0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2040ccac0 con 0x7fc20406f3f0 2024-09-13T17:18:24.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.316+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc204154510 con 0x7fc20406f3f0 2024-09-13T17:18:24.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.318+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2040ccfe0 con 0x7fc20406f3f0 2024-09-13T17:18:24.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.319+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc1f4003c80 con 0x7fc20406f3f0 2024-09-13T17:18:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.320+0000 7fc1fbe00640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 0x7fc1e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.320+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc1f4088710 con 0x7fc20406f3f0 2024-09-13T17:18:24.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.320+0000 7fc203400640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 0x7fc1e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:24.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.322+0000 7fc203400640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 0x7fc1e406d0c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc204157b90 tx=0x7fc1f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:24.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.325+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc2040ccfe0 con 0x7fc20406f3f0 2024-09-13T17:18:24.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:24.503+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc20415ab60 con 0x7fc20406f3f0 2024-09-13T17:18:25.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.132+0000 7fc1fbe00640 1 -- 172.21.15.28:0/1164611193 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v151) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc20415ab60 con 0x7fc20406f3f0 2024-09-13T17:18:25.133 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:18:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.142+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 msgr2=0x7fc1e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.142+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 0x7fc1e406d0c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fc204157b90 tx=0x7fc1f0008040 comp rx=0 tx=0).stop 2024-09-13T17:18:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.142+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 msgr2=0x7fc2040cc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.142+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fc1f400c410 tx=0x7fc1f400c8d0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.144+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 shutdown_connections 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.144+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1e406ac00 0x7fc1e406d0c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.144+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc204159770 0x7fc20414d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.145+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc204070430 0x7fc204153fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.145+0000 7fc20b200640 1 --2- 172.21.15.28:0/1164611193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc20406f3f0 0x7fc2040cc000 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.145+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 >> 172.21.15.28:0/1164611193 conn(0x7fc2040b0270 msgr2=0x7fc2041587c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:25.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.146+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 shutdown_connections 2024-09-13T17:18:25.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.146+0000 7fc20b200640 1 -- 172.21.15.28:0/1164611193 wait complete. 2024-09-13T17:18:25.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:18:25.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.329+0000 7fb7e7200640 1 Processor -- start 2024-09-13T17:18:25.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.330+0000 7fb7e7200640 1 -- start start 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e7200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7e80cd7c0 0x7fb7e80c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e80c3ba0 0x7fb7e8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e7200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e815ef70 con 0x7fb7e8143f70 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e80c4320 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e7200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e815e1b0 con 0x7fb7e80cd7c0 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e80c3ba0 0x7fb7e8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7e80c3ba0 0x7fb7e8070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53528/0 (socket says 172.21.15.28:53528) 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 -- 172.21.15.28:0/1627782795 learned_addr learned my addr 172.21.15.28:0/1627782795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e5400640 1 --2- 172.21.15.28:0/1627782795 >> v2:172.21.15.189:3300/0 conn(0x7fb7e80cd7c0 0x7fb7e80c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 -- 172.21.15.28:0/1627782795 >> v2:172.21.15.189:3300/0 conn(0x7fb7e80cd7c0 msgr2=0x7fb7e80c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:25.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 --2- 172.21.15.28:0/1627782795 >> v2:172.21.15.189:3300/0 conn(0x7fb7e80cd7c0 0x7fb7e80c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 -- 172.21.15.28:0/1627782795 >> v1:172.21.15.189:6789/0 conn(0x7fb7e8143f70 legacy=0x7fb7e80cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:25.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.331+0000 7fb7e6800640 1 -- 172.21.15.28:0/1627782795 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e815e490 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.332+0000 7fb7e6800640 1 --2- 172.21.15.28:0/1627782795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e8070440 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80ce2b0 tx=0x7fb7e002a6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c17ad083c65c99d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e4a00640 1 -- 172.21.15.28:0/1627782795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0031600 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e4a00640 1 -- 172.21.15.28:0/1627782795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e002d050 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e4a00640 1 -- 172.21.15.28:0/1627782795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e0031980 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 -- 172.21.15.28:0/1627782795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 msgr2=0x7fb7e8070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1627782795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e8070440 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80ce2b0 tx=0x7fb7e002a6b0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 -- 172.21.15.28:0/1627782795 shutdown_connections 2024-09-13T17:18:25.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1627782795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e8070440 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1627782795 >> v2:172.21.15.189:3300/0 conn(0x7fb7e80cd7c0 0x7fb7e80c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 -- 172.21.15.28:0/1627782795 >> 172.21.15.28:0/1627782795 conn(0x7fb7e8069e80 msgr2=0x7fb7e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 -- 172.21.15.28:0/1627782795 shutdown_connections 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.333+0000 7fb7e7200640 1 -- 172.21.15.28:0/1627782795 wait complete. 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.335+0000 7fb7e7200640 1 Processor -- start 2024-09-13T17:18:25.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.335+0000 7fb7e7200640 1 -- start start 2024-09-13T17:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7e80cd7c0 0x7fb7e8140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7e8143f70 0x7fb7e8159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e813e050 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e815a8b0 con 0x7fb7e80cd7c0 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e815ab90 con 0x7fb7e8143f70 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53536/0 (socket says 172.21.15.28:53536) 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.336+0000 7fb7e5e00640 1 -- 172.21.15.28:0/1553604566 learned_addr learned my addr 172.21.15.28:0/1553604566 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5400640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7e80cd7c0 0x7fb7e8140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e6800640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7e8143f70 0x7fb7e8159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 -- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7e8143f70 msgr2=0x7fb7e8159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7e8143f70 0x7fb7e8159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 -- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7e80cd7c0 msgr2=0x7fb7e8140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7e80cd7c0 0x7fb7e8140010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 -- 172.21.15.28:0/1553604566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e0009b30 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7e5e00640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800b3f0 tx=0x7fb7d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.337+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8011830 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.338+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8011990 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.338+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e815ae70 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.338+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8011d40 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.338+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e813a080 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.339+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e80cad10 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.340+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb7d801a400 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.341+0000 7fb7de000640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 0x7fb7c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.341+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb7d8089080 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.342+0000 7fb7e5400640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 0x7fb7c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.342+0000 7fb7e5400640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 0x7fb7c806d0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb7e81f5750 tx=0x7fb7d4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.346+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb7d80bc9c0 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.558+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb7e813e190 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.565+0000 7fb7de000640 1 -- 172.21.15.28:0/1553604566 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v127) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb7e813e190 con 0x7fb7e80c3ba0 2024-09-13T17:18:25.568 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 20 and data pool 21 2024-09-13T17:18:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 msgr2=0x7fb7c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 0x7fb7c806d0c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb7e81f5750 tx=0x7fb7d4008040 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 msgr2=0x7fb7e813fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fb7d800b3f0 tx=0x7fb7d800b8b0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 shutdown_connections 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7c806ac00 0x7fb7c806d0c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7e8143f70 0x7fb7e8159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7e80cd7c0 0x7fb7e8140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 --2- 172.21.15.28:0/1553604566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7e80c3ba0 0x7fb7e813fad0 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.571+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 >> 172.21.15.28:0/1553604566 conn(0x7fb7e8069e80 msgr2=0x7fb7e8143240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:25.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.573+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 shutdown_connections 2024-09-13T17:18:25.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.576+0000 7fb7e7200640 1 -- 172.21.15.28:0/1553604566 wait complete. 2024-09-13T17:18:25.603 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.758+0000 7fde82a00640 1 Processor -- start 2024-09-13T17:18:25.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.758+0000 7fde82a00640 1 -- start start 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde82a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 0x7fde7c0b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde82a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0d0c90 con 0x7fde7c06fb90 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde82a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde7c0d0f70 con 0x7fde7c15a2f0 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde82a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0b3640 con 0x7fde7c0ccb70 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde80c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fde7c06fb90 0x7fde7c0ce590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55648/0 (socket says 172.21.15.28:55648) 2024-09-13T17:18:25.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde80c00640 1 -- 172.21.15.28:0/2863823865 learned_addr learned my addr 172.21.15.28:0/2863823865 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.759+0000 7fde82000640 1 --2- 172.21.15.28:0/2863823865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 0x7fde7c0b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603029694 0 0) 0x7fde7c0d0c90 con 0x7fde7c06fb90 2024-09-13T17:18:25.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde64003520 con 0x7fde7c06fb90 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050369664 0 0) 0x7fde7c0d0f70 con 0x7fde7c15a2f0 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7c0d0c90 con 0x7fde7c15a2f0 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2085208559 0 0) 0x7fde64003520 con 0x7fde7c06fb90 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde82000640 1 -- 172.21.15.28:0/2863823865 >> v1:172.21.15.189:6789/0 conn(0x7fde7c15a2f0 legacy=0x7fde7c06f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde82000640 1 -- 172.21.15.28:0/2863823865 >> v1:172.21.15.28:6789/0 conn(0x7fde7c06fb90 legacy=0x7fde7c0ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.760+0000 7fde82000640 1 -- 172.21.15.28:0/2863823865 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c0b3920 con 0x7fde7c0ccb70 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.761+0000 7fde82000640 1 --2- 172.21.15.28:0/2863823865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 0x7fde7c0b4120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde74009dc0 tx=0x7fde7402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64cf48cdb62de38d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.761+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde68001670 con 0x7fde7c0ccb70 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.761+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7402d050 con 0x7fde7c0ccb70 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.761+0000 7fde7be00640 1 -- 172.21.15.28:0/2863823865 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde74007760 con 0x7fde7c0ccb70 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 -- 172.21.15.28:0/2863823865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 msgr2=0x7fde7c0b4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 --2- 172.21.15.28:0/2863823865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 0x7fde7c0b4120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fde74009dc0 tx=0x7fde7402a660 comp rx=0 tx=0).stop 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 -- 172.21.15.28:0/2863823865 shutdown_connections 2024-09-13T17:18:25.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 --2- 172.21.15.28:0/2863823865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c0ccb70 0x7fde7c0b4120 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 -- 172.21.15.28:0/2863823865 >> 172.21.15.28:0/2863823865 conn(0x7fde7c06d4e0 msgr2=0x7fde7c06d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 -- 172.21.15.28:0/2863823865 shutdown_connections 2024-09-13T17:18:25.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.762+0000 7fde82a00640 1 -- 172.21.15.28:0/2863823865 wait complete. 2024-09-13T17:18:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.764+0000 7fde82a00640 1 Processor -- start 2024-09-13T17:18:25.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.764+0000 7fde82a00640 1 -- start start 2024-09-13T17:18:25.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c0ccb70 0x7fde7c1e6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 0x7fde7c1f0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1e6cc0 con 0x7fde7c06fb90 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c156810 con 0x7fde7c0ccb70 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c1e3a30 con 0x7fde7c15a2f0 2024-09-13T17:18:25.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde81600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde81600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53540/0 (socket says 172.21.15.28:53540) 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde81600640 1 -- 172.21.15.28:0/1528878861 learned_addr learned my addr 172.21.15.28:0/1528878861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde82000640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 0x7fde7c1f0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.765+0000 7fde80c00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c0ccb70 0x7fde7c1e6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 -- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 msgr2=0x7fde7c1f0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 0x7fde7c1f0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 -- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c0ccb70 msgr2=0x7fde7c1e6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c0ccb70 0x7fde7c1e6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 -- 172.21.15.28:0/1528878861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde74009a70 con 0x7fde7c06fb90 2024-09-13T17:18:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde82000640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 0x7fde7c1f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:25.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde81600640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c430 tx=0x7fde7000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde700052d0 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde70005450 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde70003830 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c1e3d10 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.766+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c1e3700 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.767+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c1e3890 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.768+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fde70003990 con 0x7fde7c06fb90 2024-09-13T17:18:25.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.769+0000 7fde79600640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 0x7fde5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.769+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde70088940 con 0x7fde7c06fb90 2024-09-13T17:18:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.770+0000 7fde80c00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 0x7fde5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:25.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.770+0000 7fde80c00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 0x7fde5c06d0c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fde7c1531d0 tx=0x7fde68001670 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:25.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.773+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fde7c1e3890 con 0x7fde7c06fb90 2024-09-13T17:18:25.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.951+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde7c1e3890 con 0x7fde7c06fb90 2024-09-13T17:18:25.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.953+0000 7fde79600640 1 -- 172.21.15.28:0/1528878861 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16592 (secure 0 0 0) 0x7fde7c1e3890 con 0x7fde7c06fb90 2024-09-13T17:18:25.954 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:25.954 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":152,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:25.559631+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 msgr2=0x7fde5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 0x7fde5c06d0c0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fde7c1531d0 tx=0x7fde68001670 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 msgr2=0x7fde7c156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fde7000c430 tx=0x7fde7000c8f0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 shutdown_connections 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde5c06ac00 0x7fde5c06d0c0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fde7c15a2f0 0x7fde7c1f0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde7c0ccb70 0x7fde7c1e6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 --2- 172.21.15.28:0/1528878861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde7c06fb90 0x7fde7c156110 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 >> 172.21.15.28:0/1528878861 conn(0x7fde7c06d4e0 msgr2=0x7fde7c0cf560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 shutdown_connections 2024-09-13T17:18:25.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:25.955+0000 7fde82a00640 1 -- 172.21.15.28:0/1528878861 wait complete. 2024-09-13T17:18:25.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.127+0000 7f5b13200640 1 Processor -- start 2024-09-13T17:18:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.127+0000 7f5b13200640 1 -- start start 2024-09-13T17:18:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b13200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b0c1592a0 0x7f5b0c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b13200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c06f970 0x7f5b0c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b13200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0c0b2040 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b13200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0c15b0b0 con 0x7f5b0c1592a0 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b13200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c159ff0 con 0x7f5b0c06f970 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b0c1592a0 0x7f5b0c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b0be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5b0c1592a0 0x7f5b0c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53548/0 (socket says 172.21.15.28:53548) 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b0be00640 1 -- 172.21.15.28:0/1116418811 learned_addr learned my addr 172.21.15.28:0/1116418811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.129+0000 7f5b03400640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c06f970 0x7f5b0c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.130+0000 7f5b0be00640 1 -- 172.21.15.28:0/1116418811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c06f970 msgr2=0x7f5b0c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.130+0000 7f5b0be00640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c06f970 0x7f5b0c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.130+0000 7f5b0be00640 1 -- 172.21.15.28:0/1116418811 >> v1:172.21.15.189:6789/0 conn(0x7f5b0c06c2d0 legacy=0x7f5b0c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.130+0000 7f5b0be00640 1 -- 172.21.15.28:0/1116418811 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c15a2d0 con 0x7f5b0c1592a0 2024-09-13T17:18:26.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.131+0000 7f5b0be00640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c1592a0 0x7f5b0c15ab70 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009870 tx=0x7f5afc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931dfd1f5f34ac0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.133+0000 7f5b0b400640 1 -- 172.21.15.28:0/1116418811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc032440 con 0x7f5b0c1592a0 2024-09-13T17:18:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.133+0000 7f5b0b400640 1 -- 172.21.15.28:0/1116418811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc0325a0 con 0x7f5b0c1592a0 2024-09-13T17:18:26.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.133+0000 7f5b0b400640 1 -- 172.21.15.28:0/1116418811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc032930 con 0x7f5b0c1592a0 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 -- 172.21.15.28:0/1116418811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c1592a0 msgr2=0x7f5b0c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c1592a0 0x7f5b0c15ab70 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f5afc009870 tx=0x7f5afc02a270 comp rx=0 tx=0).stop 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 -- 172.21.15.28:0/1116418811 shutdown_connections 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c06f970 0x7f5b0c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 --2- 172.21.15.28:0/1116418811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c1592a0 0x7f5b0c15ab70 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 -- 172.21.15.28:0/1116418811 >> 172.21.15.28:0/1116418811 conn(0x7f5b0c069e80 msgr2=0x7f5b0c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 -- 172.21.15.28:0/1116418811 shutdown_connections 2024-09-13T17:18:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.134+0000 7f5b13200640 1 -- 172.21.15.28:0/1116418811 wait complete. 2024-09-13T17:18:26.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.135+0000 7f5b13200640 1 Processor -- start 2024-09-13T17:18:26.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.136+0000 7f5b13200640 1 -- start start 2024-09-13T17:18:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.136+0000 7f5b13200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.136+0000 7f5b13200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0c06f970 0x7f5b0c14b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b13200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c1592a0 0x7f5b0c14a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b13200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c14bb90 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b13200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c149170 con 0x7f5b0c06f970 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b13200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0c149450 con 0x7f5b0c1592a0 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53558/0 (socket says 172.21.15.28:53558) 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 -- 172.21.15.28:0/3890760188 learned_addr learned my addr 172.21.15.28:0/3890760188 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b03400640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0c06f970 0x7f5b0c14b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b11200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c1592a0 0x7f5b0c14a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 -- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c1592a0 msgr2=0x7f5b0c14a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c1592a0 0x7f5b0c14a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 -- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0c06f970 msgr2=0x7f5b0c14b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0c06f970 0x7f5b0c14b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.137+0000 7f5b0be00640 1 -- 172.21.15.28:0/3890760188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5afc0094d0 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b0be00640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5afc000c00 tx=0x7f5afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc032440 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5afc030b80 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5afc03d440 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b0c149730 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.138+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b0c149cf0 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.139+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b0c15ab70 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.140+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5afc039040 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.141+0000 7f5b08c00640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 0x7f5ae806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.141+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5afc0a9df0 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.141+0000 7f5b03400640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 0x7f5ae806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.142+0000 7f5b03400640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 0x7f5ae806d0c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c1f5620 tx=0x7f5af4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.145+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5afc075080 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.330+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b0c0cba30 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.331+0000 7f5b08c00640 1 -- 172.21.15.28:0/3890760188 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f5afc0786a0 con 0x7f5b0c06c2d0 2024-09-13T17:18:26.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:26.333 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":152,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:25.559631+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 msgr2=0x7f5ae806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 0x7f5ae806d0c0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c1f5620 tx=0x7f5af4008040 comp rx=0 tx=0).stop 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 msgr2=0x7f5b0c0ca480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f5afc000c00 tx=0x7f5afc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 shutdown_connections 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ae806ac00 0x7f5ae806d0c0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b0c1592a0 0x7f5b0c14a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b0c06f970 0x7f5b0c14b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 --2- 172.21.15.28:0/3890760188 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b0c06c2d0 0x7f5b0c0ca480 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 >> 172.21.15.28:0/3890760188 conn(0x7f5b0c069e80 msgr2=0x7f5b0c0b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 shutdown_connections 2024-09-13T17:18:26.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.334+0000 7f5b13200640 1 -- 172.21.15.28:0/3890760188 wait complete. 2024-09-13T17:18:26.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:18:26.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.507+0000 7f2cd9400640 1 Processor -- start 2024-09-13T17:18:26.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.507+0000 7f2cd9400640 1 -- start start 2024-09-13T17:18:26.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.508+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.508+0000 7f2cd9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd406f920 con 0x7f2cd406f3f0 2024-09-13T17:18:26.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd9400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd415b290 con 0x7f2cd415cd80 2024-09-13T17:18:26.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd415b570 con 0x7f2cd4159770 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2ccbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2cd406f3f0 0x7f2cd4159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55654/0 (socket says 172.21.15.28:55654) 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2ccbe00640 1 -- 172.21.15.28:0/1985253082 learned_addr learned my addr 172.21.15.28:0/1985253082 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd2800640 1 --2- 172.21.15.28:0/1985253082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3457512680 0 0) 0x7f2cd406f920 con 0x7f2cd406f3f0 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb8003520 con 0x7f2cd406f3f0 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710711106 0 0) 0x7f2cd415b290 con 0x7f2cd415cd80 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd2800640 1 -- 172.21.15.28:0/1985253082 >> v1:172.21.15.189:6789/0 conn(0x7f2cd415cd80 legacy=0x7f2cd415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.509+0000 7f2cd2800640 1 -- 172.21.15.28:0/1985253082 >> v1:172.21.15.28:6789/0 conn(0x7f2cd406f3f0 legacy=0x7f2cd4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.510+0000 7f2cd2800640 1 -- 172.21.15.28:0/1985253082 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd415b850 con 0x7f2cd4159770 2024-09-13T17:18:26.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.511+0000 7f2cd2800640 1 --2- 172.21.15.28:0/1985253082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd406fd20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40097d0 tx=0x7f2cc402c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4703ce0379b92bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.511+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4034640 con 0x7f2cd4159770 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.511+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc40347a0 con 0x7f2cd4159770 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.511+0000 7f2cd1e00640 1 -- 172.21.15.28:0/1985253082 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4034b30 con 0x7f2cd4159770 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 -- 172.21.15.28:0/1985253082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 msgr2=0x7f2cd406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 --2- 172.21.15.28:0/1985253082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd406fd20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2cc40097d0 tx=0x7f2cc402c390 comp rx=0 tx=0).stop 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 -- 172.21.15.28:0/1985253082 shutdown_connections 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 --2- 172.21.15.28:0/1985253082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd406fd20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 -- 172.21.15.28:0/1985253082 >> 172.21.15.28:0/1985253082 conn(0x7f2cd40b0270 msgr2=0x7f2cd40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 -- 172.21.15.28:0/1985253082 shutdown_connections 2024-09-13T17:18:26.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.512+0000 7f2cd9400640 1 -- 172.21.15.28:0/1985253082 wait complete. 2024-09-13T17:18:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.513+0000 7f2cd9400640 1 Processor -- start 2024-09-13T17:18:26.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.513+0000 7f2cd9400640 1 -- start start 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 0x7f2cd4149600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd4148030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd4149f00 con 0x7f2cd415cd80 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd414a1e0 con 0x7f2cd406f3f0 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.514+0000 7f2cd9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd414a4c0 con 0x7f2cd4159770 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 0x7f2cd4149600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53560/0 (socket says 172.21.15.28:53560) 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 0x7f2cd4149600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46484/0 (socket says 172.21.15.28:46484) 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 -- 172.21.15.28:0/326593783 learned_addr learned my addr 172.21.15.28:0/326593783 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2ccbe00640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd4148030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 -- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 msgr2=0x7f2cd4148030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd4148030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 -- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 msgr2=0x7f2cd4149600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 0x7f2cd4149600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:26.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.515+0000 7f2cd3200640 1 -- 172.21.15.28:0/326593783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc4009400 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2cd3200640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00c780 tx=0x7f2ccc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc003da0 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc00ce30 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc005270 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd414a7a0 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.516+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd414af80 con 0x7f2cd415cd80 2024-09-13T17:18:26.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.517+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd40add00 con 0x7f2cd415cd80 2024-09-13T17:18:26.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.518+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ccc0053d0 con 0x7f2cd415cd80 2024-09-13T17:18:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.519+0000 7f2ccaa00640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.519+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ccc088b60 con 0x7f2cd415cd80 2024-09-13T17:18:26.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.520+0000 7f2cd2800640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:26.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.520+0000 7f2cd2800640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4036000 tx=0x7f2cc403f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:26.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.523+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2ccc053d50 con 0x7f2cd415cd80 2024-09-13T17:18:26.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:26.742+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2cd40ad840 con 0x7f2cd415cd80 2024-09-13T17:18:27.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.572+0000 7f2ccaa00640 1 -- 172.21.15.28:0/326593783 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v129) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2ccc017070 con 0x7f2cd415cd80 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 msgr2=0x7f2cb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4036000 tx=0x7f2cc403f040 comp rx=0 tx=0).stop 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 msgr2=0x7f2cd40adf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc00c780 tx=0x7f2ccc00cc40 comp rx=0 tx=0).stop 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 shutdown_connections 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd415cd80 0x7f2cd40adf10 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd4159770 0x7f2cd4148030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 --2- 172.21.15.28:0/326593783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd406f3f0 0x7f2cd4149600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 >> 172.21.15.28:0/326593783 conn(0x7f2cd40b0270 msgr2=0x7f2cd40cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 shutdown_connections 2024-09-13T17:18:27.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.576+0000 7f2cd9400640 1 -- 172.21.15.28:0/326593783 wait complete. 2024-09-13T17:18:27.595 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:18:27.596 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:18:27.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.758+0000 7f2d89a00640 1 Processor -- start 2024-09-13T17:18:27.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.758+0000 7f2d89a00640 1 -- start start 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.759+0000 7f2d89a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d84071490 0x7f2d8406c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d89a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8406c8b0 con 0x7f2d8415ab60 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d89a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8406cb90 con 0x7f2d8415a780 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d89a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d840cc980 con 0x7f2d84071490 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d83400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d84071490 0x7f2d8406c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d7a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2d8415ab60 0x7f2d8406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55666/0 (socket says 172.21.15.28:55666) 2024-09-13T17:18:27.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d7a000640 1 -- 172.21.15.28:0/1663126832 learned_addr learned my addr 172.21.15.28:0/1663126832 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:27.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d83400640 1 -- 172.21.15.28:0/1663126832 >> v1:172.21.15.189:6789/0 conn(0x7f2d8415a780 legacy=0x7f2d84070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d83400640 1 -- 172.21.15.28:0/1663126832 >> v1:172.21.15.28:6789/0 conn(0x7f2d8415ab60 legacy=0x7f2d8406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d83400640 1 -- 172.21.15.28:0/1663126832 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d840ccc60 con 0x7f2d84071490 2024-09-13T17:18:27.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.760+0000 7f2d82000640 1 -- 172.21.15.28:0/1663126832 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696829497 0 0) 0x7f2d8406c8b0 con 0x7f2d8415ab60 2024-09-13T17:18:27.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.761+0000 7f2d83400640 1 --2- 172.21.15.28:0/1663126832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84071490 0x7f2d8406c250 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2de18105a873187 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:27.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d82000640 1 -- 172.21.15.28:0/1663126832 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c007740 con 0x7f2d84071490 2024-09-13T17:18:27.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d82000640 1 -- 172.21.15.28:0/1663126832 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c02d050 con 0x7f2d84071490 2024-09-13T17:18:27.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d82000640 1 -- 172.21.15.28:0/1663126832 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c007ac0 con 0x7f2d84071490 2024-09-13T17:18:27.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 -- 172.21.15.28:0/1663126832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84071490 msgr2=0x7f2d8406c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1663126832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84071490 0x7f2d8406c250 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 -- 172.21.15.28:0/1663126832 shutdown_connections 2024-09-13T17:18:27.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1663126832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d84071490 0x7f2d8406c250 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 -- 172.21.15.28:0/1663126832 >> 172.21.15.28:0/1663126832 conn(0x7f2d840b0270 msgr2=0x7f2d840b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:27.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 -- 172.21.15.28:0/1663126832 shutdown_connections 2024-09-13T17:18:27.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.762+0000 7f2d89a00640 1 -- 172.21.15.28:0/1663126832 wait complete. 2024-09-13T17:18:27.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.764+0000 7f2d89a00640 1 Processor -- start 2024-09-13T17:18:27.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.764+0000 7f2d89a00640 1 -- start start 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d84071490 0x7f2d84152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8415a780 0x7f2d8415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841525d0 con 0x7f2d8415ab60 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840b1710 con 0x7f2d8415a780 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d89a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8415b8e0 con 0x7f2d84071490 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:27.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d83400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53580/0 (socket says 172.21.15.28:53580) 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.765+0000 7f2d83400640 1 -- 172.21.15.28:0/1630335899 learned_addr learned my addr 172.21.15.28:0/1630335899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d7a000640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8415a780 0x7f2d8415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d82a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d84071490 0x7f2d84152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 -- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d84071490 msgr2=0x7f2d84152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d84071490 0x7f2d84152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 -- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8415a780 msgr2=0x7f2d8415af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8415a780 0x7f2d8415af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 -- 172.21.15.28:0/1630335899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c009a70 con 0x7f2d8415ab60 2024-09-13T17:18:27.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d83400640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c031910 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c031a70 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.766+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c031de0 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.767+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d840b1980 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.767+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8415bbc0 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.768+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d840b1ea0 con 0x7f2d8415ab60 2024-09-13T17:18:27.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.769+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2d7c03c420 con 0x7f2d8415ab60 2024-09-13T17:18:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.769+0000 7f2d7b400640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 0x7f2d6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:27.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.770+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2d7c0aade0 con 0x7f2d8415ab60 2024-09-13T17:18:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.770+0000 7f2d82a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 0x7f2d6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:27.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.771+0000 7f2d82a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 0x7f2d6006d0c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000f60 tx=0x7f2d74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:27.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.774+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2d840b1ea0 con 0x7f2d8415ab60 2024-09-13T17:18:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:27.991+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2d840b1ea0 con 0x7f2d8415ab60 2024-09-13T17:18:28.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.574+0000 7f2d7b400640 1 -- 172.21.15.28:0/1630335899 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v130) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2d840b1ea0 con 0x7f2d8415ab60 2024-09-13T17:18:28.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 msgr2=0x7f2d6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 0x7f2d6006d0c0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000f60 tx=0x7f2d74009040 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 msgr2=0x7f2d841f24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009d90 tx=0x7f2d7c004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 shutdown_connections 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d6006ac00 0x7f2d6006d0c0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d8415ab60 0x7f2d841f24f0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d8415a780 0x7f2d8415af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.577+0000 7f2d89a00640 1 --2- 172.21.15.28:0/1630335899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d84071490 0x7f2d84152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.578+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 >> 172.21.15.28:0/1630335899 conn(0x7f2d840b0270 msgr2=0x7f2d840cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.578+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 shutdown_connections 2024-09-13T17:18:28.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.578+0000 7f2d89a00640 1 -- 172.21.15.28:0/1630335899 wait complete. 2024-09-13T17:18:28.597 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:18:28.597 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:28.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.755+0000 7f8c7ee00640 1 Processor -- start 2024-09-13T17:18:28.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.756+0000 7f8c7ee00640 1 -- start start 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c78159820 0x7f8c78164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7ee00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c780c3850 con 0x7f8c78159440 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7ee00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7815a930 con 0x7f8c780b0b50 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c78159d20 con 0x7f8c78159820 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c78159820 0x7f8c78164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8c78159440 0x7f8c780b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55674/0 (socket says 172.21.15.28:55674) 2024-09-13T17:18:28.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.757+0000 7f8c7da00640 1 -- 172.21.15.28:0/1108453989 learned_addr learned my addr 172.21.15.28:0/1108453989 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.758+0000 7f8c7e400640 1 -- 172.21.15.28:0/1108453989 >> v1:172.21.15.189:6789/0 conn(0x7f8c780b0b50 legacy=0x7f8c780b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.758+0000 7f8c7e400640 1 -- 172.21.15.28:0/1108453989 >> v1:172.21.15.28:6789/0 conn(0x7f8c78159440 legacy=0x7f8c780b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.758+0000 7f8c7e400640 1 -- 172.21.15.28:0/1108453989 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7815a000 con 0x7f8c78159820 2024-09-13T17:18:28.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.758+0000 7f8c77e00640 1 -- 172.21.15.28:0/1108453989 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2488238922 0 0) 0x7f8c780c3850 con 0x7f8c78159440 2024-09-13T17:18:28.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c7e400640 1 --2- 172.21.15.28:0/1108453989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c78159820 0x7f8c78164260 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009d90 tx=0x7f8c7002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc6d9c028ea8c6b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:28.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c77e00640 1 -- 172.21.15.28:0/1108453989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70007740 con 0x7f8c78159820 2024-09-13T17:18:28.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c77e00640 1 -- 172.21.15.28:0/1108453989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c700078a0 con 0x7f8c78159820 2024-09-13T17:18:28.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c77e00640 1 -- 172.21.15.28:0/1108453989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70007c30 con 0x7f8c78159820 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c7ee00640 1 -- 172.21.15.28:0/1108453989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c78159820 msgr2=0x7f8c78164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.759+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/1108453989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c78159820 0x7f8c78164260 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f8c70009d90 tx=0x7f8c7002a780 comp rx=0 tx=0).stop 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.760+0000 7f8c7ee00640 1 -- 172.21.15.28:0/1108453989 shutdown_connections 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.760+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/1108453989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c78159820 0x7f8c78164260 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.760+0000 7f8c7ee00640 1 -- 172.21.15.28:0/1108453989 >> 172.21.15.28:0/1108453989 conn(0x7f8c78072e40 msgr2=0x7f8c78073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.760+0000 7f8c7ee00640 1 -- 172.21.15.28:0/1108453989 shutdown_connections 2024-09-13T17:18:28.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.760+0000 7f8c7ee00640 1 -- 172.21.15.28:0/1108453989 wait complete. 2024-09-13T17:18:28.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.761+0000 7f8c7ee00640 1 Processor -- start 2024-09-13T17:18:28.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.761+0000 7f8c7ee00640 1 -- start start 2024-09-13T17:18:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 0x7f8c7814afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 0x7f8c78157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:28.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7814b4f0 con 0x7f8c780b0b50 2024-09-13T17:18:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c781577c0 con 0x7f8c78159820 2024-09-13T17:18:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c78156bb0 con 0x7f8c78159440 2024-09-13T17:18:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.762+0000 7f8c7da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53602/0 (socket says 172.21.15.28:53602) 2024-09-13T17:18:28.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 -- 172.21.15.28:0/2496739621 learned_addr learned my addr 172.21.15.28:0/2496739621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7e400640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 0x7f8c78157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7d000640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 0x7f8c7814afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 -- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 msgr2=0x7f8c7814afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 0x7f8c7814afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 -- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 msgr2=0x7f8c78157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 0x7f8c78157c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 -- 172.21.15.28:0/2496739621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c70009a70 con 0x7f8c780b0b50 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7e400640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 0x7f8c78157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7d000640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 0x7f8c7814afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c7da00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c4a0 tx=0x7f8c6c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.763+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c0074e0 con 0x7f8c780b0b50 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.764+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c007640 con 0x7f8c780b0b50 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.764+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6c004a00 con 0x7f8c780b0b50 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.764+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c78156e90 con 0x7f8c780b0b50 2024-09-13T17:18:28.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.764+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c78157450 con 0x7f8c780b0b50 2024-09-13T17:18:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.765+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c78067580 con 0x7f8c780b0b50 2024-09-13T17:18:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.766+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c6c004b60 con 0x7f8c780b0b50 2024-09-13T17:18:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.767+0000 7f8c75600640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 0x7f8c5806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:28.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.767+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c6c088510 con 0x7f8c780b0b50 2024-09-13T17:18:28.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.767+0000 7f8c7d000640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 0x7f8c5806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:28.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.768+0000 7f8c7d000640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 0x7f8c5806d0c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8c7815db90 tx=0x7f8c64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:28.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.771+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8c6c053780 con 0x7f8c780b0b50 2024-09-13T17:18:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.985+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c780b22e0 con 0x7f8c780b0b50 2024-09-13T17:18:28.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.986+0000 7f8c75600640 1 -- 172.21.15.28:0/2496739621 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f8c6c086e60 con 0x7f8c780b0b50 2024-09-13T17:18:28.987 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:28.987 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:28.987 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 msgr2=0x7f8c5806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 0x7f8c5806d0c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f8c7815db90 tx=0x7f8c64009040 comp rx=0 tx=0).stop 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 msgr2=0x7f8c78156520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c00c4a0 tx=0x7f8c6c00c960 comp rx=0 tx=0).stop 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 shutdown_connections 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5806ac00 0x7f8c5806d0c0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c78159820 0x7f8c78157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c78159440 0x7f8c7814afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 --2- 172.21.15.28:0/2496739621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c780b0b50 0x7f8c78156520 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 >> 172.21.15.28:0/2496739621 conn(0x7f8c78072e40 msgr2=0x7f8c78070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 shutdown_connections 2024-09-13T17:18:28.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:28.988+0000 7f8c7ee00640 1 -- 172.21.15.28:0/2496739621 wait complete. 2024-09-13T17:18:29.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:18:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.160+0000 7f56c4c00640 1 Processor -- start 2024-09-13T17:18:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.160+0000 7f56c4c00640 1 -- start start 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.161+0000 7f56c4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.161+0000 7f56c4c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56c00b0b50 0x7f56c00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.161+0000 7f56c4c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c015bb10 con 0x7f56c006b360 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.161+0000 7f56c4c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c006b900 con 0x7f56c00b0b50 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.161+0000 7f56c4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c006bbe0 con 0x7f56c0154a90 2024-09-13T17:18:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56c00b0b50 0x7f56c00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f56c00b0b50 0x7f56c00af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46510/0 (socket says 172.21.15.28:46510) 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 -- 172.21.15.28:0/303161694 learned_addr learned my addr 172.21.15.28:0/303161694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56b7e00640 1 --1- 172.21.15.28:0/303161694 >> v1:172.21.15.189:6789/0 conn(0x7f56c006b360 0x7f56c00b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:33330/0 (socket says 172.21.15.28:33330) 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 -- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 msgr2=0x7f56c015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 -- 172.21.15.28:0/303161694 >> v1:172.21.15.189:6789/0 conn(0x7f56c006b360 legacy=0x7f56c00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bfe00640 1 -- 172.21.15.28:0/303161694 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c006bec0 con 0x7f56c00b0b50 2024-09-13T17:18:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bf400640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.162+0000 7f56bea00640 1 -- 172.21.15.28:0/303161694 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616791173 0 0) 0x7f56c015bb10 con 0x7f56c006b360 2024-09-13T17:18:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.163+0000 7f56bfe00640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c00af4b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009e50 tx=0x7f56b802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=138b98a89578afe8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.163+0000 7f56bea00640 1 -- 172.21.15.28:0/303161694 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8031600 con 0x7f56c00b0b50 2024-09-13T17:18:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56bea00640 1 -- 172.21.15.28:0/303161694 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b802d050 con 0x7f56c00b0b50 2024-09-13T17:18:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56bea00640 1 -- 172.21.15.28:0/303161694 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b8031980 con 0x7f56c00b0b50 2024-09-13T17:18:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 -- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 msgr2=0x7f56c00af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c00af4b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f56b8009e50 tx=0x7f56b802a710 comp rx=0 tx=0).stop 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 -- 172.21.15.28:0/303161694 shutdown_connections 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c00af4b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 --2- 172.21.15.28:0/303161694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c015b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 -- 172.21.15.28:0/303161694 >> 172.21.15.28:0/303161694 conn(0x7f56c0072e40 msgr2=0x7f56c0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 -- 172.21.15.28:0/303161694 shutdown_connections 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.164+0000 7f56c4c00640 1 -- 172.21.15.28:0/303161694 wait complete. 2024-09-13T17:18:29.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.166+0000 7f56c4c00640 1 Processor -- start 2024-09-13T17:18:29.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.166+0000 7f56c4c00640 1 -- start start 2024-09-13T17:18:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c0157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c01e1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c013d2e0 con 0x7f56c006b360 2024-09-13T17:18:29.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c01565e0 con 0x7f56c00b0b50 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56c4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c01568c0 con 0x7f56c0154a90 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56bf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53606/0 (socket says 172.21.15.28:53606) 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56bf400640 1 -- 172.21.15.28:0/3589764105 learned_addr learned my addr 172.21.15.28:0/3589764105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56b7e00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c0157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.167+0000 7f56bfe00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c01e1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 -- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 msgr2=0x7f56c01e1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c01e1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 -- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 msgr2=0x7f56c0157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c0157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 -- 172.21.15.28:0/3589764105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b0009400 con 0x7f56c006b360 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bfe00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c01e1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56b7e00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c0157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:29.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56bf400640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f56b000e960 tx=0x7f56b000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b000ca00 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56b000cb60 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0013470 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56b8009b30 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.168+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c0156e80 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.170+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c00c4050 con 0x7f56c006b360 2024-09-13T17:18:29.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.170+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f56b00135d0 con 0x7f56c006b360 2024-09-13T17:18:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.171+0000 7f56b7400640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 0x7f569c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.171+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56b00895d0 con 0x7f56c006b360 2024-09-13T17:18:29.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.172+0000 7f56b7e00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 0x7f569c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.172+0000 7f56b7e00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 0x7f569c06d0c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f56c01547b0 tx=0x7f56a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.175+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f56b00547b0 con 0x7f56c006b360 2024-09-13T17:18:29.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.358+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56c00af4b0 con 0x7f56c006b360 2024-09-13T17:18:29.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.360+0000 7f56b7400640 1 -- 172.21.15.28:0/3589764105 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f56b0057dd0 con 0x7f56c006b360 2024-09-13T17:18:29.361 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:29.361 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":153,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:26.562329+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 msgr2=0x7f569c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 0x7f569c06d0c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f56c01547b0 tx=0x7f56a8009040 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 msgr2=0x7f56c013cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f56b000e960 tx=0x7f56b000ee20 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 shutdown_connections 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f569c06ac00 0x7f569c06d0c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56c0154a90 0x7f56c01e1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56c00b0b50 0x7f56c0157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 --2- 172.21.15.28:0/3589764105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56c006b360 0x7f56c013cda0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 >> 172.21.15.28:0/3589764105 conn(0x7f56c0072e40 msgr2=0x7f56c0159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 shutdown_connections 2024-09-13T17:18:29.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.362+0000 7f56c4c00640 1 -- 172.21.15.28:0/3589764105 wait complete. 2024-09-13T17:18:29.378 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:18:29.378 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:18:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.533+0000 7efd17600640 1 Processor -- start 2024-09-13T17:18:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.533+0000 7efd17600640 1 -- start start 2024-09-13T17:18:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.534+0000 7efd17600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd100cabc0 0x7efd1006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.534+0000 7efd17600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 0x7efd10071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.534+0000 7efd17600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1015e100 con 0x7efd1015a540 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.534+0000 7efd17600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd100adb00 con 0x7efd100cabc0 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.534+0000 7efd17600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10071670 con 0x7efd10070cc0 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd14c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efd1015a540 0x7efd1015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55678/0 (socket says 172.21.15.28:55678) 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd14c00640 1 -- 172.21.15.28:0/3418233425 learned_addr learned my addr 172.21.15.28:0/3418233425 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0fe00640 1 --2- 172.21.15.28:0/3418233425 >> v2:172.21.15.28:3300/0 conn(0x7efd100cabc0 0x7efd1006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd15600640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 0x7efd10071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0f400640 1 -- 172.21.15.28:0/3418233425 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129813137 0 0) 0x7efd1015e100 con 0x7efd1015a540 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0f400640 1 -- 172.21.15.28:0/3418233425 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf8003520 con 0x7efd1015a540 2024-09-13T17:18:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0fe00640 1 -- 172.21.15.28:0/3418233425 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 msgr2=0x7efd10071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0fe00640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 0x7efd10071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0fe00640 1 -- 172.21.15.28:0/3418233425 >> v1:172.21.15.28:6789/0 conn(0x7efd1015a540 legacy=0x7efd1015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd0fe00640 1 -- 172.21.15.28:0/3418233425 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1015e3e0 con 0x7efd100cabc0 2024-09-13T17:18:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.535+0000 7efd15600640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 0x7efd10071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.536+0000 7efd0fe00640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd100cabc0 0x7efd1006c430 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0097a0 tx=0x7efcfc02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf6ab6fef09955d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd0f400640 1 -- 172.21.15.28:0/3418233425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc02ae50 con 0x7efd100cabc0 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd0f400640 1 -- 172.21.15.28:0/3418233425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc032440 con 0x7efd100cabc0 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd0f400640 1 -- 172.21.15.28:0/3418233425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0327d0 con 0x7efd100cabc0 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 -- 172.21.15.28:0/3418233425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd100cabc0 msgr2=0x7efd1006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd100cabc0 0x7efd1006c430 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7efcfc0097a0 tx=0x7efcfc02a100 comp rx=0 tx=0).stop 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 -- 172.21.15.28:0/3418233425 shutdown_connections 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd10070cc0 0x7efd10071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 --2- 172.21.15.28:0/3418233425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd100cabc0 0x7efd1006c430 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 -- 172.21.15.28:0/3418233425 >> 172.21.15.28:0/3418233425 conn(0x7efd10069e80 msgr2=0x7efd1006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 -- 172.21.15.28:0/3418233425 shutdown_connections 2024-09-13T17:18:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.537+0000 7efd17600640 1 -- 172.21.15.28:0/3418233425 wait complete. 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.539+0000 7efd17600640 1 Processor -- start 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.539+0000 7efd17600640 1 -- start start 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd100cabc0 0x7efd10153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 0x7efd10156860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd101592c0 con 0x7efd10070cc0 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd101595a0 con 0x7efd1015a540 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd17600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd10159880 con 0x7efd100cabc0 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd14c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd14c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53630/0 (socket says 172.21.15.28:53630) 2024-09-13T17:18:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd14c00640 1 -- 172.21.15.28:0/2508563667 learned_addr learned my addr 172.21.15.28:0/2508563667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd14c00640 1 -- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd100cabc0 msgr2=0x7efd10153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.540+0000 7efd14c00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd100cabc0 0x7efd10153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd14c00640 1 -- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 msgr2=0x7efd10156860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd15600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 0x7efd10156860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd14c00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 0x7efd10156860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd14c00640 1 -- 172.21.15.28:0/2508563667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc009400 con 0x7efd10070cc0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd15600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 0x7efd10156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd14c00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c410 tx=0x7efd0400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04005ce0 con 0x7efd10070cc0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04005e40 con 0x7efd10070cc0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd040072f0 con 0x7efd10070cc0 2024-09-13T17:18:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd10153a80 con 0x7efd10070cc0 2024-09-13T17:18:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.541+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd101f6940 con 0x7efd10070cc0 2024-09-13T17:18:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.542+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd10153c70 con 0x7efd10070cc0 2024-09-13T17:18:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.543+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efd04007510 con 0x7efd10070cc0 2024-09-13T17:18:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.544+0000 7efd0cc00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 0x7efcf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.544+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efd040889c0 con 0x7efd10070cc0 2024-09-13T17:18:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.545+0000 7efd0fe00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 0x7efcf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.545+0000 7efd0fe00640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 0x7efcf006d0c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7efcfc034000 tx=0x7efcfc02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.548+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efd04053c60 con 0x7efd10070cc0 2024-09-13T17:18:29.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.725+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd10159710 con 0x7efcf006ac00 2024-09-13T17:18:29.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.734+0000 7efd0cc00640 1 -- 172.21.15.28:0/2508563667 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289353 (secure 0 0 0) 0x7efd10159710 con 0x7efcf006ac00 2024-09-13T17:18:29.739 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:29.741 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:18:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 msgr2=0x7efcf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 0x7efcf006d0c0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7efcfc034000 tx=0x7efcfc02c000 comp rx=0 tx=0).stop 2024-09-13T17:18:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 msgr2=0x7efd10158c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7efd0400c410 tx=0x7efd0400c8d0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 shutdown_connections 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efcf006ac00 0x7efcf006d0c0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd1015a540 0x7efd10156860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd100cabc0 0x7efd10153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 --2- 172.21.15.28:0/2508563667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd10070cc0 0x7efd10158c60 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 >> 172.21.15.28:0/2508563667 conn(0x7efd10069e80 msgr2=0x7efd1015ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.738+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 shutdown_connections 2024-09-13T17:18:29.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.739+0000 7efd17600640 1 -- 172.21.15.28:0/2508563667 wait complete. 2024-09-13T17:18:29.755 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-13T17:18:29.041754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":506068,"kb_used_data":16160,"kb_used_omap":12,"kb_used_meta":489779,"kb_avail":754468652,"statfs":{"total":773094113280,"available":772575899648,"internally_reserved":0,"allocated":16547840,"data_stored":7920942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":501534265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.917587"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730403+0000","last_change":"2024-09-13T17:18:25.583283+0000","last_active":"2024-09-13T17:18:26.730403+0000","last_peered":"2024-09-13T17:18:26.730403+0000","last_clean":"2024-09-13T17:18:26.730403+0000","last_became_active":"2024-09-13T17:18:25.583010+0000","last_became_peered":"2024-09-13T17:18:25.583010+0000","last_unstale":"2024-09-13T17:18:26.730403+0000","last_undegraded":"2024-09-13T17:18:26.730403+0000","last_fullsized":"2024-09-13T17:18:26.730403+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566235+0000","last_change":"2024-09-13T17:18:25.582821+0000","last_active":"2024-09-13T17:18:26.566235+0000","last_peered":"2024-09-13T17:18:26.566235+0000","last_clean":"2024-09-13T17:18:26.566235+0000","last_became_active":"2024-09-13T17:18:25.582554+0000","last_became_peered":"2024-09-13T17:18:25.582554+0000","last_unstale":"2024-09-13T17:18:26.566235+0000","last_undegraded":"2024-09-13T17:18:26.566235+0000","last_fullsized":"2024-09-13T17:18:26.566235+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:48.905857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721003+0000","last_change":"2024-09-13T17:18:25.590551+0000","last_active":"2024-09-13T17:18:26.721003+0000","last_peered":"2024-09-13T17:18:26.721003+0000","last_clean":"2024-09-13T17:18:26.721003+0000","last_became_active":"2024-09-13T17:18:25.590188+0000","last_became_peered":"2024-09-13T17:18:25.590188+0000","last_unstale":"2024-09-13T17:18:26.721003+0000","last_undegraded":"2024-09-13T17:18:26.721003+0000","last_fullsized":"2024-09-13T17:18:26.721003+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730644+0000","last_change":"2024-09-13T17:18:25.592532+0000","last_active":"2024-09-13T17:18:26.730644+0000","last_peered":"2024-09-13T17:18:26.730644+0000","last_clean":"2024-09-13T17:18:26.730644+0000","last_became_active":"2024-09-13T17:18:25.591789+0000","last_became_peered":"2024-09-13T17:18:25.591789+0000","last_unstale":"2024-09-13T17:18:26.730644+0000","last_undegraded":"2024-09-13T17:18:26.730644+0000","last_fullsized":"2024-09-13T17:18:26.730644+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:44.382887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568585+0000","last_change":"2024-09-13T17:18:25.582580+0000","last_active":"2024-09-13T17:18:26.568585+0000","last_peered":"2024-09-13T17:18:26.568585+0000","last_clean":"2024-09-13T17:18:26.568585+0000","last_became_active":"2024-09-13T17:18:25.582312+0000","last_became_peered":"2024-09-13T17:18:25.582312+0000","last_unstale":"2024-09-13T17:18:26.568585+0000","last_undegraded":"2024-09-13T17:18:26.568585+0000","last_fullsized":"2024-09-13T17:18:26.568585+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:55.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730390+0000","last_change":"2024-09-13T17:18:25.594734+0000","last_active":"2024-09-13T17:18:26.730390+0000","last_peered":"2024-09-13T17:18:26.730390+0000","last_clean":"2024-09-13T17:18:26.730390+0000","last_became_active":"2024-09-13T17:18:25.594478+0000","last_became_peered":"2024-09-13T17:18:25.594478+0000","last_unstale":"2024-09-13T17:18:26.730390+0000","last_undegraded":"2024-09-13T17:18:26.730390+0000","last_fullsized":"2024-09-13T17:18:26.730390+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:28.803886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730309+0000","last_change":"2024-09-13T17:18:25.575232+0000","last_active":"2024-09-13T17:18:26.730309+0000","last_peered":"2024-09-13T17:18:26.730309+0000","last_clean":"2024-09-13T17:18:26.730309+0000","last_became_active":"2024-09-13T17:18:25.572953+0000","last_became_peered":"2024-09-13T17:18:25.572953+0000","last_unstale":"2024-09-13T17:18:26.730309+0000","last_undegraded":"2024-09-13T17:18:26.730309+0000","last_fullsized":"2024-09-13T17:18:26.730309+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:48.905857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.729943+0000","last_change":"2024-09-13T17:18:25.577944+0000","last_active":"2024-09-13T17:18:26.729943+0000","last_peered":"2024-09-13T17:18:26.729943+0000","last_clean":"2024-09-13T17:18:26.729943+0000","last_became_active":"2024-09-13T17:18:25.577433+0000","last_became_peered":"2024-09-13T17:18:25.577433+0000","last_unstale":"2024-09-13T17:18:26.729943+0000","last_undegraded":"2024-09-13T17:18:26.729943+0000","last_fullsized":"2024-09-13T17:18:26.729943+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566233+0000","last_change":"2024-09-13T17:18:25.587625+0000","last_active":"2024-09-13T17:18:26.566233+0000","last_peered":"2024-09-13T17:18:26.566233+0000","last_clean":"2024-09-13T17:18:26.566233+0000","last_became_active":"2024-09-13T17:18:25.585645+0000","last_became_peered":"2024-09-13T17:18:25.585645+0000","last_unstale":"2024-09-13T17:18:26.566233+0000","last_undegraded":"2024-09-13T17:18:26.566233+0000","last_fullsized":"2024-09-13T17:18:26.566233+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:04.478740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569069+0000","last_change":"2024-09-13T17:18:25.592679+0000","last_active":"2024-09-13T17:18:26.569069+0000","last_peered":"2024-09-13T17:18:26.569069+0000","last_clean":"2024-09-13T17:18:26.569069+0000","last_became_active":"2024-09-13T17:18:25.592198+0000","last_became_peered":"2024-09-13T17:18:25.592198+0000","last_unstale":"2024-09-13T17:18:26.569069+0000","last_undegraded":"2024-09-13T17:18:26.569069+0000","last_fullsized":"2024-09-13T17:18:26.569069+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:17.576724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730525+0000","last_change":"2024-09-13T17:18:25.582501+0000","last_active":"2024-09-13T17:18:26.730525+0000","last_peered":"2024-09-13T17:18:26.730525+0000","last_clean":"2024-09-13T17:18:26.730525+0000","last_became_active":"2024-09-13T17:18:25.582239+0000","last_became_peered":"2024-09-13T17:18:25.582239+0000","last_unstale":"2024-09-13T17:18:26.730525+0000","last_undegraded":"2024-09-13T17:18:26.730525+0000","last_fullsized":"2024-09-13T17:18:26.730525+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:49.058248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569131+0000","last_change":"2024-09-13T17:18:25.587801+0000","last_active":"2024-09-13T17:18:26.569131+0000","last_peered":"2024-09-13T17:18:26.569131+0000","last_clean":"2024-09-13T17:18:26.569131+0000","last_became_active":"2024-09-13T17:18:25.583003+0000","last_became_peered":"2024-09-13T17:18:25.583003+0000","last_unstale":"2024-09-13T17:18:26.569131+0000","last_undegraded":"2024-09-13T17:18:26.569131+0000","last_fullsized":"2024-09-13T17:18:26.569131+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:47.264988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568678+0000","last_change":"2024-09-13T17:18:25.577340+0000","last_active":"2024-09-13T17:18:26.568678+0000","last_peered":"2024-09-13T17:18:26.568678+0000","last_clean":"2024-09-13T17:18:26.568678+0000","last_became_active":"2024-09-13T17:18:25.577072+0000","last_became_peered":"2024-09-13T17:18:25.577072+0000","last_unstale":"2024-09-13T17:18:26.568678+0000","last_undegraded":"2024-09-13T17:18:26.568678+0000","last_fullsized":"2024-09-13T17:18:26.568678+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:26.222696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730463+0000","last_change":"2024-09-13T17:18:25.587672+0000","last_active":"2024-09-13T17:18:26.730463+0000","last_peered":"2024-09-13T17:18:26.730463+0000","last_clean":"2024-09-13T17:18:26.730463+0000","last_became_active":"2024-09-13T17:18:25.587274+0000","last_became_peered":"2024-09-13T17:18:25.587274+0000","last_unstale":"2024-09-13T17:18:26.730463+0000","last_undegraded":"2024-09-13T17:18:26.730463+0000","last_fullsized":"2024-09-13T17:18:26.730463+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:23.154924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567973+0000","last_change":"2024-09-13T17:18:25.583450+0000","last_active":"2024-09-13T17:18:26.567973+0000","last_peered":"2024-09-13T17:18:26.567973+0000","last_clean":"2024-09-13T17:18:26.567973+0000","last_became_active":"2024-09-13T17:18:25.581810+0000","last_became_peered":"2024-09-13T17:18:25.581810+0000","last_unstale":"2024-09-13T17:18:26.567973+0000","last_undegraded":"2024-09-13T17:18:26.567973+0000","last_fullsized":"2024-09-13T17:18:26.567973+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:00.965493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599325+0000","last_change":"2024-09-13T17:18:25.579947+0000","last_active":"2024-09-13T17:18:26.599325+0000","last_peered":"2024-09-13T17:18:26.599325+0000","last_clean":"2024-09-13T17:18:26.599325+0000","last_became_active":"2024-09-13T17:18:25.578900+0000","last_became_peered":"2024-09-13T17:18:25.578900+0000","last_unstale":"2024-09-13T17:18:26.599325+0000","last_undegraded":"2024-09-13T17:18:26.599325+0000","last_fullsized":"2024-09-13T17:18:26.599325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:54.986444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569280+0000","last_change":"2024-09-13T17:18:25.585214+0000","last_active":"2024-09-13T17:18:26.569280+0000","last_peered":"2024-09-13T17:18:26.569280+0000","last_clean":"2024-09-13T17:18:26.569280+0000","last_became_active":"2024-09-13T17:18:25.584938+0000","last_became_peered":"2024-09-13T17:18:25.584938+0000","last_unstale":"2024-09-13T17:18:26.569280+0000","last_undegraded":"2024-09-13T17:18:26.569280+0000","last_fullsized":"2024-09-13T17:18:26.569280+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:53.638212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721058+0000","last_change":"2024-09-13T17:18:25.594139+0000","last_active":"2024-09-13T17:18:26.721058+0000","last_peered":"2024-09-13T17:18:26.721058+0000","last_clean":"2024-09-13T17:18:26.721058+0000","last_became_active":"2024-09-13T17:18:25.593882+0000","last_became_peered":"2024-09-13T17:18:25.593882+0000","last_unstale":"2024-09-13T17:18:26.721058+0000","last_undegraded":"2024-09-13T17:18:26.721058+0000","last_fullsized":"2024-09-13T17:18:26.721058+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:26.222696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730624+0000","last_change":"2024-09-13T17:18:25.586675+0000","last_active":"2024-09-13T17:18:26.730624+0000","last_peered":"2024-09-13T17:18:26.730624+0000","last_clean":"2024-09-13T17:18:26.730624+0000","last_became_active":"2024-09-13T17:18:25.586392+0000","last_became_peered":"2024-09-13T17:18:25.586392+0000","last_unstale":"2024-09-13T17:18:26.730624+0000","last_undegraded":"2024-09-13T17:18:26.730624+0000","last_fullsized":"2024-09-13T17:18:26.730624+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:57.490862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599647+0000","last_change":"2024-09-13T17:18:25.574420+0000","last_active":"2024-09-13T17:18:26.599647+0000","last_peered":"2024-09-13T17:18:26.599647+0000","last_clean":"2024-09-13T17:18:26.599647+0000","last_became_active":"2024-09-13T17:18:25.574176+0000","last_became_peered":"2024-09-13T17:18:25.574176+0000","last_unstale":"2024-09-13T17:18:26.599647+0000","last_undegraded":"2024-09-13T17:18:26.599647+0000","last_fullsized":"2024-09-13T17:18:26.599647+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:53:36.210934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568648+0000","last_change":"2024-09-13T17:18:25.592426+0000","last_active":"2024-09-13T17:18:26.568648+0000","last_peered":"2024-09-13T17:18:26.568648+0000","last_clean":"2024-09-13T17:18:26.568648+0000","last_became_active":"2024-09-13T17:18:25.591556+0000","last_became_peered":"2024-09-13T17:18:25.591556+0000","last_unstale":"2024-09-13T17:18:26.568648+0000","last_undegraded":"2024-09-13T17:18:26.568648+0000","last_fullsized":"2024-09-13T17:18:26.568648+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:23.154924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730483+0000","last_change":"2024-09-13T17:18:25.573292+0000","last_active":"2024-09-13T17:18:26.730483+0000","last_peered":"2024-09-13T17:18:26.730483+0000","last_clean":"2024-09-13T17:18:26.730483+0000","last_became_active":"2024-09-13T17:18:25.572701+0000","last_became_peered":"2024-09-13T17:18:25.572701+0000","last_unstale":"2024-09-13T17:18:26.730483+0000","last_undegraded":"2024-09-13T17:18:26.730483+0000","last_fullsized":"2024-09-13T17:18:26.730483+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:56.512584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730353+0000","last_change":"2024-09-13T17:18:25.583023+0000","last_active":"2024-09-13T17:18:26.730353+0000","last_peered":"2024-09-13T17:18:26.730353+0000","last_clean":"2024-09-13T17:18:26.730353+0000","last_became_active":"2024-09-13T17:18:25.582499+0000","last_became_peered":"2024-09-13T17:18:25.582499+0000","last_unstale":"2024-09-13T17:18:26.730353+0000","last_undegraded":"2024-09-13T17:18:26.730353+0000","last_fullsized":"2024-09-13T17:18:26.730353+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:45.622459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567976+0000","last_change":"2024-09-13T17:18:25.587408+0000","last_active":"2024-09-13T17:18:26.567976+0000","last_peered":"2024-09-13T17:18:26.567976+0000","last_clean":"2024-09-13T17:18:26.567976+0000","last_became_active":"2024-09-13T17:18:25.582533+0000","last_became_peered":"2024-09-13T17:18:25.582533+0000","last_unstale":"2024-09-13T17:18:26.567976+0000","last_undegraded":"2024-09-13T17:18:26.567976+0000","last_fullsized":"2024-09-13T17:18:26.567976+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:08.327631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599767+0000","last_change":"2024-09-13T17:18:25.586618+0000","last_active":"2024-09-13T17:18:26.599767+0000","last_peered":"2024-09-13T17:18:26.599767+0000","last_clean":"2024-09-13T17:18:26.599767+0000","last_became_active":"2024-09-13T17:18:25.586363+0000","last_became_peered":"2024-09-13T17:18:25.586363+0000","last_unstale":"2024-09-13T17:18:26.599767+0000","last_undegraded":"2024-09-13T17:18:26.599767+0000","last_fullsized":"2024-09-13T17:18:26.599767+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:51.366450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730124+0000","last_change":"2024-09-13T17:18:25.588869+0000","last_active":"2024-09-13T17:18:26.730124+0000","last_peered":"2024-09-13T17:18:26.730124+0000","last_clean":"2024-09-13T17:18:26.730124+0000","last_became_active":"2024-09-13T17:18:25.588590+0000","last_became_peered":"2024-09-13T17:18:25.588590+0000","last_unstale":"2024-09-13T17:18:26.730124+0000","last_undegraded":"2024-09-13T17:18:26.730124+0000","last_fullsized":"2024-09-13T17:18:26.730124+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:08.431154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599647+0000","last_change":"2024-09-13T17:18:25.593673+0000","last_active":"2024-09-13T17:18:26.599647+0000","last_peered":"2024-09-13T17:18:26.599647+0000","last_clean":"2024-09-13T17:18:26.599647+0000","last_became_active":"2024-09-13T17:18:25.592650+0000","last_became_peered":"2024-09-13T17:18:25.592650+0000","last_unstale":"2024-09-13T17:18:26.599647+0000","last_undegraded":"2024-09-13T17:18:26.599647+0000","last_fullsized":"2024-09-13T17:18:26.599647+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:19.811125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730866+0000","last_change":"2024-09-13T17:18:25.590226+0000","last_active":"2024-09-13T17:18:26.730866+0000","last_peered":"2024-09-13T17:18:26.730866+0000","last_clean":"2024-09-13T17:18:26.730866+0000","last_became_active":"2024-09-13T17:18:25.589973+0000","last_became_peered":"2024-09-13T17:18:25.589973+0000","last_unstale":"2024-09-13T17:18:26.730866+0000","last_undegraded":"2024-09-13T17:18:26.730866+0000","last_fullsized":"2024-09-13T17:18:26.730866+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.729999+0000","last_change":"2024-09-13T17:18:25.594725+0000","last_active":"2024-09-13T17:18:26.729999+0000","last_peered":"2024-09-13T17:18:26.729999+0000","last_clean":"2024-09-13T17:18:26.729999+0000","last_became_active":"2024-09-13T17:18:25.594437+0000","last_became_peered":"2024-09-13T17:18:25.594437+0000","last_unstale":"2024-09-13T17:18:26.729999+0000","last_undegraded":"2024-09-13T17:18:26.729999+0000","last_fullsized":"2024-09-13T17:18:26.729999+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730785+0000","last_change":"2024-09-13T17:18:25.587517+0000","last_active":"2024-09-13T17:18:26.730785+0000","last_peered":"2024-09-13T17:18:26.730785+0000","last_clean":"2024-09-13T17:18:26.730785+0000","last_became_active":"2024-09-13T17:18:25.586936+0000","last_became_peered":"2024-09-13T17:18:25.586936+0000","last_unstale":"2024-09-13T17:18:26.730785+0000","last_undegraded":"2024-09-13T17:18:26.730785+0000","last_fullsized":"2024-09-13T17:18:26.730785+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:53.638212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565980+0000","last_change":"2024-09-13T17:18:25.590428+0000","last_active":"2024-09-13T17:18:26.565980+0000","last_peered":"2024-09-13T17:18:26.565980+0000","last_clean":"2024-09-13T17:18:26.565980+0000","last_became_active":"2024-09-13T17:18:25.589291+0000","last_became_peered":"2024-09-13T17:18:25.589291+0000","last_unstale":"2024-09-13T17:18:26.565980+0000","last_undegraded":"2024-09-13T17:18:26.565980+0000","last_fullsized":"2024-09-13T17:18:26.565980+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:58.998510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599353+0000","last_change":"2024-09-13T17:18:25.573098+0000","last_active":"2024-09-13T17:18:26.599353+0000","last_peered":"2024-09-13T17:18:26.599353+0000","last_clean":"2024-09-13T17:18:26.599353+0000","last_became_active":"2024-09-13T17:18:25.572732+0000","last_became_peered":"2024-09-13T17:18:25.572732+0000","last_unstale":"2024-09-13T17:18:26.599353+0000","last_undegraded":"2024-09-13T17:18:26.599353+0000","last_fullsized":"2024-09-13T17:18:26.599353+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:03.912654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569183+0000","last_change":"2024-09-13T17:18:25.586629+0000","last_active":"2024-09-13T17:18:26.569183+0000","last_peered":"2024-09-13T17:18:26.569183+0000","last_clean":"2024-09-13T17:18:26.569183+0000","last_became_active":"2024-09-13T17:18:25.586244+0000","last_became_peered":"2024-09-13T17:18:25.586244+0000","last_unstale":"2024-09-13T17:18:26.569183+0000","last_undegraded":"2024-09-13T17:18:26.569183+0000","last_fullsized":"2024-09-13T17:18:26.569183+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:56.189045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"152'4","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569183+0000","last_change":"2024-09-13T17:18:25.160980+0000","last_active":"2024-09-13T17:18:26.569183+0000","last_peered":"2024-09-13T17:18:26.569183+0000","last_clean":"2024-09-13T17:18:26.569183+0000","last_became_active":"2024-09-13T17:18:25.160678+0000","last_became_peered":"2024-09-13T17:18:25.160678+0000","last_unstale":"2024-09-13T17:18:26.569183+0000","last_undegraded":"2024-09-13T17:18:26.569183+0000","last_fullsized":"2024-09-13T17:18:26.569183+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:05.830174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566341+0000","last_change":"2024-09-13T17:18:25.584385+0000","last_active":"2024-09-13T17:18:26.566341+0000","last_peered":"2024-09-13T17:18:26.566341+0000","last_clean":"2024-09-13T17:18:26.566341+0000","last_became_active":"2024-09-13T17:18:25.583039+0000","last_became_peered":"2024-09-13T17:18:25.583039+0000","last_unstale":"2024-09-13T17:18:26.566341+0000","last_undegraded":"2024-09-13T17:18:26.566341+0000","last_fullsized":"2024-09-13T17:18:26.566341+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566283+0000","last_change":"2024-09-13T17:18:25.153960+0000","last_active":"2024-09-13T17:18:26.566283+0000","last_peered":"2024-09-13T17:18:26.566283+0000","last_clean":"2024-09-13T17:18:26.566283+0000","last_became_active":"2024-09-13T17:18:25.153547+0000","last_became_peered":"2024-09-13T17:18:25.153547+0000","last_unstale":"2024-09-13T17:18:26.566283+0000","last_undegraded":"2024-09-13T17:18:26.566283+0000","last_fullsized":"2024-09-13T17:18:26.566283+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.145798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730066+0000","last_change":"2024-09-13T17:18:25.593654+0000","last_active":"2024-09-13T17:18:26.730066+0000","last_peered":"2024-09-13T17:18:26.730066+0000","last_clean":"2024-09-13T17:18:26.730066+0000","last_became_active":"2024-09-13T17:18:25.593332+0000","last_became_peered":"2024-09-13T17:18:25.593332+0000","last_unstale":"2024-09-13T17:18:26.730066+0000","last_undegraded":"2024-09-13T17:18:26.730066+0000","last_fullsized":"2024-09-13T17:18:26.730066+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:20.666847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730655+0000","last_change":"2024-09-13T17:18:25.145149+0000","last_active":"2024-09-13T17:18:26.730655+0000","last_peered":"2024-09-13T17:18:26.730655+0000","last_clean":"2024-09-13T17:18:26.730655+0000","last_became_active":"2024-09-13T17:18:25.144882+0000","last_became_peered":"2024-09-13T17:18:25.144882+0000","last_unstale":"2024-09-13T17:18:26.730655+0000","last_undegraded":"2024-09-13T17:18:26.730655+0000","last_fullsized":"2024-09-13T17:18:26.730655+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:47.827518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568646+0000","last_change":"2024-09-13T17:18:25.584379+0000","last_active":"2024-09-13T17:18:26.568646+0000","last_peered":"2024-09-13T17:18:26.568646+0000","last_clean":"2024-09-13T17:18:26.568646+0000","last_became_active":"2024-09-13T17:18:25.583871+0000","last_became_peered":"2024-09-13T17:18:25.583871+0000","last_unstale":"2024-09-13T17:18:26.568646+0000","last_undegraded":"2024-09-13T17:18:26.568646+0000","last_fullsized":"2024-09-13T17:18:26.568646+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:52.658355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730681+0000","last_change":"2024-09-13T17:18:25.147372+0000","last_active":"2024-09-13T17:18:26.730681+0000","last_peered":"2024-09-13T17:18:26.730681+0000","last_clean":"2024-09-13T17:18:26.730681+0000","last_became_active":"2024-09-13T17:18:25.146276+0000","last_became_peered":"2024-09-13T17:18:25.146276+0000","last_unstale":"2024-09-13T17:18:26.730681+0000","last_undegraded":"2024-09-13T17:18:26.730681+0000","last_fullsized":"2024-09-13T17:18:26.730681+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:24.774066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566215+0000","last_change":"2024-09-13T17:18:25.587085+0000","last_active":"2024-09-13T17:18:26.566215+0000","last_peered":"2024-09-13T17:18:26.566215+0000","last_clean":"2024-09-13T17:18:26.566215+0000","last_became_active":"2024-09-13T17:18:25.586816+0000","last_became_peered":"2024-09-13T17:18:25.586816+0000","last_unstale":"2024-09-13T17:18:26.566215+0000","last_undegraded":"2024-09-13T17:18:26.566215+0000","last_fullsized":"2024-09-13T17:18:26.566215+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:15.552013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599499+0000","last_change":"2024-09-13T17:18:25.158637+0000","last_active":"2024-09-13T17:18:26.599499+0000","last_peered":"2024-09-13T17:18:26.599499+0000","last_clean":"2024-09-13T17:18:26.599499+0000","last_became_active":"2024-09-13T17:18:25.158286+0000","last_became_peered":"2024-09-13T17:18:25.158286+0000","last_unstale":"2024-09-13T17:18:26.599499+0000","last_undegraded":"2024-09-13T17:18:26.599499+0000","last_fullsized":"2024-09-13T17:18:26.599499+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:01.047481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730533+0000","last_change":"2024-09-13T17:18:25.589290+0000","last_active":"2024-09-13T17:18:26.730533+0000","last_peered":"2024-09-13T17:18:26.730533+0000","last_clean":"2024-09-13T17:18:26.730533+0000","last_became_active":"2024-09-13T17:18:25.589023+0000","last_became_peered":"2024-09-13T17:18:25.589023+0000","last_unstale":"2024-09-13T17:18:26.730533+0000","last_undegraded":"2024-09-13T17:18:26.730533+0000","last_fullsized":"2024-09-13T17:18:26.730533+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:17.576724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568298+0000","last_change":"2024-09-13T17:18:25.157554+0000","last_active":"2024-09-13T17:18:26.568298+0000","last_peered":"2024-09-13T17:18:26.568298+0000","last_clean":"2024-09-13T17:18:26.568298+0000","last_became_active":"2024-09-13T17:18:25.157194+0000","last_became_peered":"2024-09-13T17:18:25.157194+0000","last_unstale":"2024-09-13T17:18:26.568298+0000","last_undegraded":"2024-09-13T17:18:26.568298+0000","last_fullsized":"2024-09-13T17:18:26.568298+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:23.668919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565933+0000","last_change":"2024-09-13T17:18:25.575346+0000","last_active":"2024-09-13T17:18:26.565933+0000","last_peered":"2024-09-13T17:18:26.565933+0000","last_clean":"2024-09-13T17:18:26.565933+0000","last_became_active":"2024-09-13T17:18:25.575108+0000","last_became_peered":"2024-09-13T17:18:25.575108+0000","last_unstale":"2024-09-13T17:18:26.565933+0000","last_undegraded":"2024-09-13T17:18:26.565933+0000","last_fullsized":"2024-09-13T17:18:26.565933+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:12.622765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599404+0000","last_change":"2024-09-13T17:18:25.148369+0000","last_active":"2024-09-13T17:18:26.599404+0000","last_peered":"2024-09-13T17:18:26.599404+0000","last_clean":"2024-09-13T17:18:26.599404+0000","last_became_active":"2024-09-13T17:18:25.146221+0000","last_became_peered":"2024-09-13T17:18:25.146221+0000","last_unstale":"2024-09-13T17:18:26.599404+0000","last_undegraded":"2024-09-13T17:18:26.599404+0000","last_fullsized":"2024-09-13T17:18:26.599404+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:59.409452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567710+0000","last_change":"2024-09-13T17:18:25.579818+0000","last_active":"2024-09-13T17:18:26.567710+0000","last_peered":"2024-09-13T17:18:26.567710+0000","last_clean":"2024-09-13T17:18:26.567710+0000","last_became_active":"2024-09-13T17:18:25.579502+0000","last_became_peered":"2024-09-13T17:18:25.579502+0000","last_unstale":"2024-09-13T17:18:26.567710+0000","last_undegraded":"2024-09-13T17:18:26.567710+0000","last_fullsized":"2024-09-13T17:18:26.567710+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:28.001858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730328+0000","last_change":"2024-09-13T17:18:25.154251+0000","last_active":"2024-09-13T17:18:26.730328+0000","last_peered":"2024-09-13T17:18:26.730328+0000","last_clean":"2024-09-13T17:18:26.730328+0000","last_became_active":"2024-09-13T17:18:25.153833+0000","last_became_peered":"2024-09-13T17:18:25.153833+0000","last_unstale":"2024-09-13T17:18:26.730328+0000","last_undegraded":"2024-09-13T17:18:26.730328+0000","last_fullsized":"2024-09-13T17:18:26.730328+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:23.325245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569035+0000","last_change":"2024-09-13T17:18:25.573589+0000","last_active":"2024-09-13T17:18:26.569035+0000","last_peered":"2024-09-13T17:18:26.569035+0000","last_clean":"2024-09-13T17:18:26.569035+0000","last_became_active":"2024-09-13T17:18:25.573093+0000","last_became_peered":"2024-09-13T17:18:25.573093+0000","last_unstale":"2024-09-13T17:18:26.569035+0000","last_undegraded":"2024-09-13T17:18:26.569035+0000","last_fullsized":"2024-09-13T17:18:26.569035+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:49.058248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566408+0000","last_change":"2024-09-13T17:18:25.143056+0000","last_active":"2024-09-13T17:18:26.566408+0000","last_peered":"2024-09-13T17:18:26.566408+0000","last_clean":"2024-09-13T17:18:26.566408+0000","last_became_active":"2024-09-13T17:18:25.142629+0000","last_became_peered":"2024-09-13T17:18:25.142629+0000","last_unstale":"2024-09-13T17:18:26.566408+0000","last_undegraded":"2024-09-13T17:18:26.566408+0000","last_fullsized":"2024-09-13T17:18:26.566408+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:43.373760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569142+0000","last_change":"2024-09-13T17:18:25.586255+0000","last_active":"2024-09-13T17:18:26.569142+0000","last_peered":"2024-09-13T17:18:26.569142+0000","last_clean":"2024-09-13T17:18:26.569142+0000","last_became_active":"2024-09-13T17:18:25.582344+0000","last_became_peered":"2024-09-13T17:18:25.582344+0000","last_unstale":"2024-09-13T17:18:26.569142+0000","last_undegraded":"2024-09-13T17:18:26.569142+0000","last_fullsized":"2024-09-13T17:18:26.569142+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:56.512584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569267+0000","last_change":"2024-09-13T17:18:25.161288+0000","last_active":"2024-09-13T17:18:26.569267+0000","last_peered":"2024-09-13T17:18:26.569267+0000","last_clean":"2024-09-13T17:18:26.569267+0000","last_became_active":"2024-09-13T17:18:25.160983+0000","last_became_peered":"2024-09-13T17:18:25.160983+0000","last_unstale":"2024-09-13T17:18:26.569267+0000","last_undegraded":"2024-09-13T17:18:26.569267+0000","last_fullsized":"2024-09-13T17:18:26.569267+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:58.767168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599717+0000","last_change":"2024-09-13T17:18:25.582346+0000","last_active":"2024-09-13T17:18:26.599717+0000","last_peered":"2024-09-13T17:18:26.599717+0000","last_clean":"2024-09-13T17:18:26.599717+0000","last_became_active":"2024-09-13T17:18:25.581084+0000","last_became_peered":"2024-09-13T17:18:25.581084+0000","last_unstale":"2024-09-13T17:18:26.599717+0000","last_undegraded":"2024-09-13T17:18:26.599717+0000","last_fullsized":"2024-09-13T17:18:26.599717+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:58.673315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"152'6","reported_seq":17,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730986+0000","last_change":"2024-09-13T17:18:25.142894+0000","last_active":"2024-09-13T17:18:26.730986+0000","last_peered":"2024-09-13T17:18:26.730986+0000","last_clean":"2024-09-13T17:18:26.730986+0000","last_became_active":"2024-09-13T17:18:25.142506+0000","last_became_peered":"2024-09-13T17:18:25.142506+0000","last_unstale":"2024-09-13T17:18:26.730986+0000","last_undegraded":"2024-09-13T17:18:26.730986+0000","last_fullsized":"2024-09-13T17:18:26.730986+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.145798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720820+0000","last_change":"2024-09-13T17:18:25.585949+0000","last_active":"2024-09-13T17:18:26.720820+0000","last_peered":"2024-09-13T17:18:26.720820+0000","last_clean":"2024-09-13T17:18:26.720820+0000","last_became_active":"2024-09-13T17:18:25.585706+0000","last_became_peered":"2024-09-13T17:18:25.585706+0000","last_unstale":"2024-09-13T17:18:26.720820+0000","last_undegraded":"2024-09-13T17:18:26.720820+0000","last_fullsized":"2024-09-13T17:18:26.720820+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720902+0000","last_change":"2024-09-13T17:18:25.154349+0000","last_active":"2024-09-13T17:18:26.720902+0000","last_peered":"2024-09-13T17:18:26.720902+0000","last_clean":"2024-09-13T17:18:26.720902+0000","last_became_active":"2024-09-13T17:18:25.152804+0000","last_became_peered":"2024-09-13T17:18:25.152804+0000","last_unstale":"2024-09-13T17:18:26.720902+0000","last_undegraded":"2024-09-13T17:18:26.720902+0000","last_fullsized":"2024-09-13T17:18:26.720902+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:18.506679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565999+0000","last_change":"2024-09-13T17:18:25.580515+0000","last_active":"2024-09-13T17:18:26.565999+0000","last_peered":"2024-09-13T17:18:26.565999+0000","last_clean":"2024-09-13T17:18:26.565999+0000","last_became_active":"2024-09-13T17:18:25.580217+0000","last_became_peered":"2024-09-13T17:18:25.580217+0000","last_unstale":"2024-09-13T17:18:26.565999+0000","last_undegraded":"2024-09-13T17:18:26.565999+0000","last_fullsized":"2024-09-13T17:18:26.565999+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:14.346865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720824+0000","last_change":"2024-09-13T17:18:25.150316+0000","last_active":"2024-09-13T17:18:26.720824+0000","last_peered":"2024-09-13T17:18:26.720824+0000","last_clean":"2024-09-13T17:18:26.720824+0000","last_became_active":"2024-09-13T17:18:25.150010+0000","last_became_peered":"2024-09-13T17:18:25.150010+0000","last_unstale":"2024-09-13T17:18:26.720824+0000","last_undegraded":"2024-09-13T17:18:26.720824+0000","last_fullsized":"2024-09-13T17:18:26.720824+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:27.544810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567911+0000","last_change":"2024-09-13T17:18:25.580865+0000","last_active":"2024-09-13T17:18:26.567911+0000","last_peered":"2024-09-13T17:18:26.567911+0000","last_clean":"2024-09-13T17:18:26.567911+0000","last_became_active":"2024-09-13T17:18:25.580609+0000","last_became_peered":"2024-09-13T17:18:25.580609+0000","last_unstale":"2024-09-13T17:18:26.567911+0000","last_undegraded":"2024-09-13T17:18:26.567911+0000","last_fullsized":"2024-09-13T17:18:26.567911+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730175+0000","last_change":"2024-09-13T17:18:25.143107+0000","last_active":"2024-09-13T17:18:26.730175+0000","last_peered":"2024-09-13T17:18:26.730175+0000","last_clean":"2024-09-13T17:18:26.730175+0000","last_became_active":"2024-09-13T17:18:25.142839+0000","last_became_peered":"2024-09-13T17:18:25.142839+0000","last_unstale":"2024-09-13T17:18:26.730175+0000","last_undegraded":"2024-09-13T17:18:26.730175+0000","last_fullsized":"2024-09-13T17:18:26.730175+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:02.903527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720604+0000","last_change":"2024-09-13T17:18:25.586714+0000","last_active":"2024-09-13T17:18:26.720604+0000","last_peered":"2024-09-13T17:18:26.720604+0000","last_clean":"2024-09-13T17:18:26.720604+0000","last_became_active":"2024-09-13T17:18:25.584465+0000","last_became_peered":"2024-09-13T17:18:25.584465+0000","last_unstale":"2024-09-13T17:18:26.720604+0000","last_undegraded":"2024-09-13T17:18:26.720604+0000","last_fullsized":"2024-09-13T17:18:26.720604+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"152'8","reported_seq":19,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599444+0000","last_change":"2024-09-13T17:18:25.154342+0000","last_active":"2024-09-13T17:18:26.599444+0000","last_peered":"2024-09-13T17:18:26.599444+0000","last_clean":"2024-09-13T17:18:26.599444+0000","last_became_active":"2024-09-13T17:18:25.154088+0000","last_became_peered":"2024-09-13T17:18:25.154088+0000","last_unstale":"2024-09-13T17:18:26.599444+0000","last_undegraded":"2024-09-13T17:18:26.599444+0000","last_fullsized":"2024-09-13T17:18:26.599444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:53.047516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569297+0000","last_change":"2024-09-13T17:18:25.591945+0000","last_active":"2024-09-13T17:18:26.569297+0000","last_peered":"2024-09-13T17:18:26.569297+0000","last_clean":"2024-09-13T17:18:26.569297+0000","last_became_active":"2024-09-13T17:18:25.590685+0000","last_became_peered":"2024-09-13T17:18:25.590685+0000","last_unstale":"2024-09-13T17:18:26.569297+0000","last_undegraded":"2024-09-13T17:18:26.569297+0000","last_fullsized":"2024-09-13T17:18:26.569297+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:40.698757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568774+0000","last_change":"2024-09-13T17:18:25.157365+0000","last_active":"2024-09-13T17:18:26.568774+0000","last_peered":"2024-09-13T17:18:26.568774+0000","last_clean":"2024-09-13T17:18:26.568774+0000","last_became_active":"2024-09-13T17:18:25.156888+0000","last_became_peered":"2024-09-13T17:18:25.156888+0000","last_unstale":"2024-09-13T17:18:26.568774+0000","last_undegraded":"2024-09-13T17:18:26.568774+0000","last_fullsized":"2024-09-13T17:18:26.568774+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:18.680287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599797+0000","last_change":"2024-09-13T17:18:25.585900+0000","last_active":"2024-09-13T17:18:26.599797+0000","last_peered":"2024-09-13T17:18:26.599797+0000","last_clean":"2024-09-13T17:18:26.599797+0000","last_became_active":"2024-09-13T17:18:25.583822+0000","last_became_peered":"2024-09-13T17:18:25.583822+0000","last_unstale":"2024-09-13T17:18:26.599797+0000","last_undegraded":"2024-09-13T17:18:26.599797+0000","last_fullsized":"2024-09-13T17:18:26.599797+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:02:04.937114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566327+0000","last_change":"2024-09-13T17:18:25.154568+0000","last_active":"2024-09-13T17:18:26.566327+0000","last_peered":"2024-09-13T17:18:26.566327+0000","last_clean":"2024-09-13T17:18:26.566327+0000","last_became_active":"2024-09-13T17:18:25.154240+0000","last_became_peered":"2024-09-13T17:18:25.154240+0000","last_unstale":"2024-09-13T17:18:26.566327+0000","last_undegraded":"2024-09-13T17:18:26.566327+0000","last_fullsized":"2024-09-13T17:18:26.566327+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:46.255862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721003+0000","last_change":"2024-09-13T17:18:25.594539+0000","last_active":"2024-09-13T17:18:26.721003+0000","last_peered":"2024-09-13T17:18:26.721003+0000","last_clean":"2024-09-13T17:18:26.721003+0000","last_became_active":"2024-09-13T17:18:25.594221+0000","last_became_peered":"2024-09-13T17:18:25.594221+0000","last_unstale":"2024-09-13T17:18:26.721003+0000","last_undegraded":"2024-09-13T17:18:26.721003+0000","last_fullsized":"2024-09-13T17:18:26.721003+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:55.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599728+0000","last_change":"2024-09-13T17:18:25.152130+0000","last_active":"2024-09-13T17:18:26.599728+0000","last_peered":"2024-09-13T17:18:26.599728+0000","last_clean":"2024-09-13T17:18:26.599728+0000","last_became_active":"2024-09-13T17:18:25.151882+0000","last_became_peered":"2024-09-13T17:18:25.151882+0000","last_unstale":"2024-09-13T17:18:26.599728+0000","last_undegraded":"2024-09-13T17:18:26.599728+0000","last_fullsized":"2024-09-13T17:18:26.599728+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:37.730295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730507+0000","last_change":"2024-09-13T17:18:25.596129+0000","last_active":"2024-09-13T17:18:26.730507+0000","last_peered":"2024-09-13T17:18:26.730507+0000","last_clean":"2024-09-13T17:18:26.730507+0000","last_became_active":"2024-09-13T17:18:25.595037+0000","last_became_peered":"2024-09-13T17:18:25.595037+0000","last_unstale":"2024-09-13T17:18:26.730507+0000","last_undegraded":"2024-09-13T17:18:26.730507+0000","last_fullsized":"2024-09-13T17:18:26.730507+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:28.001858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566157+0000","last_change":"2024-09-13T17:18:25.145279+0000","last_active":"2024-09-13T17:18:26.566157+0000","last_peered":"2024-09-13T17:18:26.566157+0000","last_clean":"2024-09-13T17:18:26.566157+0000","last_became_active":"2024-09-13T17:18:25.144493+0000","last_became_peered":"2024-09-13T17:18:25.144493+0000","last_unstale":"2024-09-13T17:18:26.566157+0000","last_undegraded":"2024-09-13T17:18:26.566157+0000","last_fullsized":"2024-09-13T17:18:26.566157+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:55.503457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730780+0000","last_change":"2024-09-13T17:18:25.588921+0000","last_active":"2024-09-13T17:18:26.730780+0000","last_peered":"2024-09-13T17:18:26.730780+0000","last_clean":"2024-09-13T17:18:26.730780+0000","last_became_active":"2024-09-13T17:18:25.588590+0000","last_became_peered":"2024-09-13T17:18:25.588590+0000","last_unstale":"2024-09-13T17:18:26.730780+0000","last_undegraded":"2024-09-13T17:18:26.730780+0000","last_fullsized":"2024-09-13T17:18:26.730780+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:40.698757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730589+0000","last_change":"2024-09-13T17:18:25.156786+0000","last_active":"2024-09-13T17:18:26.730589+0000","last_peered":"2024-09-13T17:18:26.730589+0000","last_clean":"2024-09-13T17:18:26.730589+0000","last_became_active":"2024-09-13T17:18:25.156504+0000","last_became_peered":"2024-09-13T17:18:25.156504+0000","last_unstale":"2024-09-13T17:18:26.730589+0000","last_undegraded":"2024-09-13T17:18:26.730589+0000","last_fullsized":"2024-09-13T17:18:26.730589+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:51.649229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730506+0000","last_change":"2024-09-13T17:18:25.575414+0000","last_active":"2024-09-13T17:18:26.730506+0000","last_peered":"2024-09-13T17:18:26.730506+0000","last_clean":"2024-09-13T17:18:26.730506+0000","last_became_active":"2024-09-13T17:18:25.573914+0000","last_became_peered":"2024-09-13T17:18:25.573914+0000","last_unstale":"2024-09-13T17:18:26.730506+0000","last_undegraded":"2024-09-13T17:18:26.730506+0000","last_fullsized":"2024-09-13T17:18:26.730506+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:14.346865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568220+0000","last_change":"2024-09-13T17:18:25.157732+0000","last_active":"2024-09-13T17:18:26.568220+0000","last_peered":"2024-09-13T17:18:26.568220+0000","last_clean":"2024-09-13T17:18:26.568220+0000","last_became_active":"2024-09-13T17:18:25.157461+0000","last_became_peered":"2024-09-13T17:18:25.157461+0000","last_unstale":"2024-09-13T17:18:26.568220+0000","last_undegraded":"2024-09-13T17:18:26.568220+0000","last_fullsized":"2024-09-13T17:18:26.568220+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:08.010160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730246+0000","last_change":"2024-09-13T17:18:25.589558+0000","last_active":"2024-09-13T17:18:26.730246+0000","last_peered":"2024-09-13T17:18:26.730246+0000","last_clean":"2024-09-13T17:18:26.730246+0000","last_became_active":"2024-09-13T17:18:25.589187+0000","last_became_peered":"2024-09-13T17:18:25.589187+0000","last_unstale":"2024-09-13T17:18:26.730246+0000","last_undegraded":"2024-09-13T17:18:26.730246+0000","last_fullsized":"2024-09-13T17:18:26.730246+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:00.965493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568060+0000","last_change":"2024-09-13T17:18:25.159414+0000","last_active":"2024-09-13T17:18:26.568060+0000","last_peered":"2024-09-13T17:18:26.568060+0000","last_clean":"2024-09-13T17:18:26.568060+0000","last_became_active":"2024-09-13T17:18:25.158410+0000","last_became_peered":"2024-09-13T17:18:25.158410+0000","last_unstale":"2024-09-13T17:18:26.568060+0000","last_undegraded":"2024-09-13T17:18:26.568060+0000","last_fullsized":"2024-09-13T17:18:26.568060+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:54.738997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730325+0000","last_change":"2024-09-13T17:18:25.573015+0000","last_active":"2024-09-13T17:18:26.730325+0000","last_peered":"2024-09-13T17:18:26.730325+0000","last_clean":"2024-09-13T17:18:26.730325+0000","last_became_active":"2024-09-13T17:18:25.572758+0000","last_became_peered":"2024-09-13T17:18:25.572758+0000","last_unstale":"2024-09-13T17:18:26.730325+0000","last_undegraded":"2024-09-13T17:18:26.730325+0000","last_fullsized":"2024-09-13T17:18:26.730325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:13.683409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720566+0000","last_change":"2024-09-13T17:18:25.153891+0000","last_active":"2024-09-13T17:18:26.720566+0000","last_peered":"2024-09-13T17:18:26.720566+0000","last_clean":"2024-09-13T17:18:26.720566+0000","last_became_active":"2024-09-13T17:18:25.153464+0000","last_became_peered":"2024-09-13T17:18:25.153464+0000","last_unstale":"2024-09-13T17:18:26.720566+0000","last_undegraded":"2024-09-13T17:18:26.720566+0000","last_fullsized":"2024-09-13T17:18:26.720566+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:49.835535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569128+0000","last_change":"2024-09-13T17:18:25.584752+0000","last_active":"2024-09-13T17:18:26.569128+0000","last_peered":"2024-09-13T17:18:26.569128+0000","last_clean":"2024-09-13T17:18:26.569128+0000","last_became_active":"2024-09-13T17:18:25.583587+0000","last_became_peered":"2024-09-13T17:18:25.583587+0000","last_unstale":"2024-09-13T17:18:26.569128+0000","last_undegraded":"2024-09-13T17:18:26.569128+0000","last_fullsized":"2024-09-13T17:18:26.569128+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569296+0000","last_change":"2024-09-13T17:18:25.161166+0000","last_active":"2024-09-13T17:18:26.569296+0000","last_peered":"2024-09-13T17:18:26.569296+0000","last_clean":"2024-09-13T17:18:26.569296+0000","last_became_active":"2024-09-13T17:18:25.160718+0000","last_became_peered":"2024-09-13T17:18:25.160718+0000","last_unstale":"2024-09-13T17:18:26.569296+0000","last_undegraded":"2024-09-13T17:18:26.569296+0000","last_fullsized":"2024-09-13T17:18:26.569296+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:19.734455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721051+0000","last_change":"2024-09-13T17:18:25.585227+0000","last_active":"2024-09-13T17:18:26.721051+0000","last_peered":"2024-09-13T17:18:26.721051+0000","last_clean":"2024-09-13T17:18:26.721051+0000","last_became_active":"2024-09-13T17:18:25.584956+0000","last_became_peered":"2024-09-13T17:18:25.584956+0000","last_unstale":"2024-09-13T17:18:26.721051+0000","last_undegraded":"2024-09-13T17:18:26.721051+0000","last_fullsized":"2024-09-13T17:18:26.721051+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:20.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730217+0000","last_change":"2024-09-13T17:18:25.155260+0000","last_active":"2024-09-13T17:18:26.730217+0000","last_peered":"2024-09-13T17:18:26.730217+0000","last_clean":"2024-09-13T17:18:26.730217+0000","last_became_active":"2024-09-13T17:18:25.154901+0000","last_became_peered":"2024-09-13T17:18:25.154901+0000","last_unstale":"2024-09-13T17:18:26.730217+0000","last_undegraded":"2024-09-13T17:18:26.730217+0000","last_fullsized":"2024-09-13T17:18:26.730217+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:18.801999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566111+0000","last_change":"2024-09-13T17:18:25.575793+0000","last_active":"2024-09-13T17:18:26.566111+0000","last_peered":"2024-09-13T17:18:26.566111+0000","last_clean":"2024-09-13T17:18:26.566111+0000","last_became_active":"2024-09-13T17:18:25.575544+0000","last_became_peered":"2024-09-13T17:18:25.575544+0000","last_unstale":"2024-09-13T17:18:26.566111+0000","last_undegraded":"2024-09-13T17:18:26.566111+0000","last_fullsized":"2024-09-13T17:18:26.566111+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:15.170284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730600+0000","last_change":"2024-09-13T17:18:25.145846+0000","last_active":"2024-09-13T17:18:26.730600+0000","last_peered":"2024-09-13T17:18:26.730600+0000","last_clean":"2024-09-13T17:18:26.730600+0000","last_became_active":"2024-09-13T17:18:25.144449+0000","last_became_peered":"2024-09-13T17:18:25.144449+0000","last_unstale":"2024-09-13T17:18:26.730600+0000","last_undegraded":"2024-09-13T17:18:26.730600+0000","last_fullsized":"2024-09-13T17:18:26.730600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:53.977318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730556+0000","last_change":"2024-09-13T17:18:25.590857+0000","last_active":"2024-09-13T17:18:26.730556+0000","last_peered":"2024-09-13T17:18:26.730556+0000","last_clean":"2024-09-13T17:18:26.730556+0000","last_became_active":"2024-09-13T17:18:25.590568+0000","last_became_peered":"2024-09-13T17:18:25.590568+0000","last_unstale":"2024-09-13T17:18:26.730556+0000","last_undegraded":"2024-09-13T17:18:26.730556+0000","last_fullsized":"2024-09-13T17:18:26.730556+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:33.135490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730065+0000","last_change":"2024-09-13T17:18:25.146418+0000","last_active":"2024-09-13T17:18:26.730065+0000","last_peered":"2024-09-13T17:18:26.730065+0000","last_clean":"2024-09-13T17:18:26.730065+0000","last_became_active":"2024-09-13T17:18:25.144484+0000","last_became_peered":"2024-09-13T17:18:25.144484+0000","last_unstale":"2024-09-13T17:18:26.730065+0000","last_undegraded":"2024-09-13T17:18:26.730065+0000","last_fullsized":"2024-09-13T17:18:26.730065+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:26.290597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599375+0000","last_change":"2024-09-13T17:18:25.573883+0000","last_active":"2024-09-13T17:18:26.599375+0000","last_peered":"2024-09-13T17:18:26.599375+0000","last_clean":"2024-09-13T17:18:26.599375+0000","last_became_active":"2024-09-13T17:18:25.573337+0000","last_became_peered":"2024-09-13T17:18:25.573337+0000","last_unstale":"2024-09-13T17:18:26.599375+0000","last_undegraded":"2024-09-13T17:18:26.599375+0000","last_fullsized":"2024-09-13T17:18:26.599375+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:27.299723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730665+0000","last_change":"2024-09-13T17:18:25.149634+0000","last_active":"2024-09-13T17:18:26.730665+0000","last_peered":"2024-09-13T17:18:26.730665+0000","last_clean":"2024-09-13T17:18:26.730665+0000","last_became_active":"2024-09-13T17:18:25.149209+0000","last_became_peered":"2024-09-13T17:18:25.149209+0000","last_unstale":"2024-09-13T17:18:26.730665+0000","last_undegraded":"2024-09-13T17:18:26.730665+0000","last_fullsized":"2024-09-13T17:18:26.730665+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:46.255862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568698+0000","last_change":"2024-09-13T17:18:25.587553+0000","last_active":"2024-09-13T17:18:26.568698+0000","last_peered":"2024-09-13T17:18:26.568698+0000","last_clean":"2024-09-13T17:18:26.568698+0000","last_became_active":"2024-09-13T17:18:25.587264+0000","last_became_peered":"2024-09-13T17:18:25.587264+0000","last_unstale":"2024-09-13T17:18:26.568698+0000","last_undegraded":"2024-09-13T17:18:26.568698+0000","last_fullsized":"2024-09-13T17:18:26.568698+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:28.803886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568372+0000","last_change":"2024-09-13T17:18:25.159730+0000","last_active":"2024-09-13T17:18:26.568372+0000","last_peered":"2024-09-13T17:18:26.568372+0000","last_clean":"2024-09-13T17:18:26.568372+0000","last_became_active":"2024-09-13T17:18:25.159138+0000","last_became_peered":"2024-09-13T17:18:25.159138+0000","last_unstale":"2024-09-13T17:18:26.568372+0000","last_undegraded":"2024-09-13T17:18:26.568372+0000","last_fullsized":"2024-09-13T17:18:26.568372+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:28.633856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568081+0000","last_change":"2024-09-13T17:18:25.585984+0000","last_active":"2024-09-13T17:18:26.568081+0000","last_peered":"2024-09-13T17:18:26.568081+0000","last_clean":"2024-09-13T17:18:26.568081+0000","last_became_active":"2024-09-13T17:18:25.583547+0000","last_became_peered":"2024-09-13T17:18:25.583547+0000","last_unstale":"2024-09-13T17:18:26.568081+0000","last_undegraded":"2024-09-13T17:18:26.568081+0000","last_fullsized":"2024-09-13T17:18:26.568081+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:44.382887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720714+0000","last_change":"2024-09-13T17:18:25.155074+0000","last_active":"2024-09-13T17:18:26.720714+0000","last_peered":"2024-09-13T17:18:26.720714+0000","last_clean":"2024-09-13T17:18:26.720714+0000","last_became_active":"2024-09-13T17:18:25.152672+0000","last_became_peered":"2024-09-13T17:18:25.152672+0000","last_unstale":"2024-09-13T17:18:26.720714+0000","last_undegraded":"2024-09-13T17:18:26.720714+0000","last_fullsized":"2024-09-13T17:18:26.720714+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:17.508560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567884+0000","last_change":"2024-09-13T17:18:25.573804+0000","last_active":"2024-09-13T17:18:26.567884+0000","last_peered":"2024-09-13T17:18:26.567884+0000","last_clean":"2024-09-13T17:18:26.567884+0000","last_became_active":"2024-09-13T17:18:25.573534+0000","last_became_peered":"2024-09-13T17:18:25.573534+0000","last_unstale":"2024-09-13T17:18:26.567884+0000","last_undegraded":"2024-09-13T17:18:26.567884+0000","last_fullsized":"2024-09-13T17:18:26.567884+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565979+0000","last_change":"2024-09-13T17:18:25.152835+0000","last_active":"2024-09-13T17:18:26.565979+0000","last_peered":"2024-09-13T17:18:26.565979+0000","last_clean":"2024-09-13T17:18:26.565979+0000","last_became_active":"2024-09-13T17:18:25.152548+0000","last_became_peered":"2024-09-13T17:18:25.152548+0000","last_unstale":"2024-09-13T17:18:26.565979+0000","last_undegraded":"2024-09-13T17:18:26.565979+0000","last_fullsized":"2024-09-13T17:18:26.565979+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:16.567597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730974+0000","last_change":"2024-09-13T17:18:25.573606+0000","last_active":"2024-09-13T17:18:26.730974+0000","last_peered":"2024-09-13T17:18:26.730974+0000","last_clean":"2024-09-13T17:18:26.730974+0000","last_became_active":"2024-09-13T17:18:25.573359+0000","last_became_peered":"2024-09-13T17:18:25.573359+0000","last_unstale":"2024-09-13T17:18:26.730974+0000","last_undegraded":"2024-09-13T17:18:26.730974+0000","last_fullsized":"2024-09-13T17:18:26.730974+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:27.547752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"152'3","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721035+0000","last_change":"2024-09-13T17:18:25.154092+0000","last_active":"2024-09-13T17:18:26.721035+0000","last_peered":"2024-09-13T17:18:26.721035+0000","last_clean":"2024-09-13T17:18:26.721035+0000","last_became_active":"2024-09-13T17:18:25.153805+0000","last_became_peered":"2024-09-13T17:18:25.153805+0000","last_unstale":"2024-09-13T17:18:26.721035+0000","last_undegraded":"2024-09-13T17:18:26.721035+0000","last_fullsized":"2024-09-13T17:18:26.721035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:07.422028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T17:18:25.573515+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:18:25.573515+0000","last_peered":"2024-09-13T17:18:25.573515+0000","last_clean":"2024-09-13T17:18:25.573515+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:18:25.573515+0000","last_undegraded":"2024-09-13T17:18:25.573515+0000","last_fullsized":"2024-09-13T17:18:25.573515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574977,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73700,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298140,"statfs":{"total":96636764160,"available":96561295360,"internally_reserved":0,"allocated":2514944,"data_stored":1433210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":72939979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.996,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68200000000000005,"15min":0.65700000000000003},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.127,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.74199999999999999,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92000000000000004,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.51800000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":1.169,"5min":1.544,"15min":1.544},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.605,"15min":1.605},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0660000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.274,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.76200000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60599999999999998,"15min":0.63400000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.90400000000000003,"15min":1.054},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.35399999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.54500000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.375,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.143},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.76500000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.337},"last":0.82899999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71399999999999997,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.1930000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71699999999999997,"15min":0.754},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.2030000000000001},"last":0.77700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574978,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1925120,"data_stored":842336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.32900000000000001},"max":{"1min":1.0209999999999999,"5min":1.298,"15min":3.6629999999999998},"last":0.501},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.76700000000000002,"15min":0.74099999999999999},"min":{"1min":0.51400000000000001,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89000000000000001,"5min":1.46,"15min":3.3079999999999998},"last":0.81200000000000006}]},{"osd":2,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.72499999999999998,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93300000000000005,"5min":1.004,"15min":3.2629999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.495,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":1.167,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85499999999999998,"5min":3.6720000000000002,"15min":3.742},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.623,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.91000000000000003,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85199999999999998,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66600000000000004,"15min":0.70399999999999996},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.83999999999999997,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64000000000000001,"15min":0.69199999999999995},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":0.92100000000000004,"15min":3.609},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70799999999999996,"15min":0.74099999999999999},"min":{"1min":0.45700000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.93700000000000006,"5min":1.601,"15min":4.9669999999999996},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0780000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.98999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574977,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55140,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316700,"statfs":{"total":96636764160,"available":96580300800,"internally_reserved":0,"allocated":1916928,"data_stored":842212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.76800000000000002},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94199999999999995,"5min":1.498,"15min":38.005000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64400000000000002,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89700000000000002,"5min":1.577,"15min":21.376999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65300000000000002,"15min":0.72499999999999998},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95299999999999996,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":0.99199999999999999,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.745},"min":{"1min":0.54100000000000004,"5min":0.379,"15min":0.379},"max":{"1min":1.0489999999999999,"5min":1.524,"15min":17.268000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98199999999999998,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.77800000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72799999999999998,"15min":1.0980000000000001},"min":{"1min":0.51800000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.86299999999999999,"5min":2.8690000000000002,"15min":113.363},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71799999999999997,"15min":0.754},"min":{"1min":0.54900000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.01,"5min":2.359,"15min":4.1180000000000003},"last":0.72999999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81299999999999994,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65700000000000003,"15min":0.69899999999999995},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85599999999999998,"5min":2.827,"15min":2.827},"last":0.878}]},{"osd":6,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.69699999999999995},"min":{"1min":0.41499999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88200000000000001,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69099999999999995,"15min":0.71799999999999997},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79600000000000004,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.81699999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72699999999999998,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72899999999999998,"15min":0.75800000000000001},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.9379999999999999,"15min":2.02},"last":0.82899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574976,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59228,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312612,"statfs":{"total":96636764160,"available":96576114688,"internally_reserved":0,"allocated":1912832,"data_stored":842190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66700000000000004,"15min":0.65000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79800000000000004,"5min":1.056,"15min":1.0620000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71799999999999997,"15min":0.72699999999999998},"min":{"1min":0.5,"5min":0.373,"15min":0.373},"max":{"1min":0.93700000000000006,"5min":1.3400000000000001,"15min":1.341},"last":0.61099999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.374,"15min":0.374},"max":{"1min":1.0509999999999999,"5min":1.173,"15min":1.173},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":1.1100000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70399999999999996,"15min":0.71299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0249999999999999,"5min":1.22,"15min":2.1259999999999999},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.67700000000000005},"min":{"1min":0.53200000000000003,"5min":0.378,"15min":0.378},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.48399999999999999,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46200000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.95699999999999996,"5min":1.284,"15min":1.284},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.30299999999999999},"max":{"1min":0.90100000000000002,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.99099999999999999,"5min":1.274,"15min":1.274},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.57499999999999996,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":1.0720000000000001,"5min":1.292,"15min":1.413},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.752},{"interface":"front","average":{"1min":0.752,"5min":0.76100000000000001,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.44500000000000001,"15min":0.371},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.76700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64924,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306916,"statfs":{"total":96636764160,"available":96570281984,"internally_reserved":0,"allocated":1916928,"data_stored":842280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40500000000000003,"15min":0.443},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.55200000000000005,"5min":0.67500000000000004,"15min":1.9810000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45100000000000001,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57599999999999996,"5min":0.68300000000000005,"15min":2.96},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49199999999999999,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.64100000000000001,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.65700000000000003},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":3.6829999999999998},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.57599999999999996},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70899999999999996,"5min":0.85999999999999999,"15min":1.3819999999999999},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":0.77200000000000002},"min":{"1min":0.45800000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.075,"5min":1.1040000000000001,"15min":2.8610000000000002},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.94099999999999995,"5min":1.012,"15min":1.3640000000000001},"last":0.95199999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.77000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":1.391},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0269999999999999,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":0.97499999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.745,"15min":0.76400000000000001},"min":{"1min":0.51400000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.05,"5min":1.0840000000000001,"15min":1.401},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.746,"15min":0.78100000000000003},"min":{"1min":0.59999999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.01,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":1.004}]},{"osd":7,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.746,"15min":0.81799999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1020000000000001,"5min":1.1339999999999999,"15min":2.899},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.80900000000000005},"min":{"1min":0.60399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.4359999999999999},"last":0.88900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1925120,"data_stored":842805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.504,"5min":1.097,"15min":1.097},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.80300000000000005,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28100000000000003},"max":{"1min":0.65300000000000002,"5min":30.023,"15min":30.023},"last":0.51400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49099999999999999,"15min":1.3460000000000001},"min":{"1min":0.34499999999999997,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.629,"5min":1.046,"15min":260.53399999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.752,"15min":1.4490000000000001},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.69599999999999995,"5min":21.279,"15min":262.61399999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.80600000000000005,"15min":0.59499999999999997},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.29099999999999998},"max":{"1min":0.58499999999999996,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.42199999999999999},"max":{"1min":0.66400000000000003,"5min":1.1559999999999999,"15min":1.214},"last":0.626}]},{"osd":4,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.871,"15min":0.79700000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.98799999999999999,"15min":0.85199999999999998},"min":{"1min":0.48999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.91000000000000003,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.0549999999999999,"15min":0.879},"min":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":0.876,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.51900000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.86199999999999999,"5min":1.1279999999999999,"15min":1.569},"last":0.75700000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.88500000000000001,"15min":0.80700000000000005},"min":{"1min":0.53300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93000000000000005,"5min":10.664,"15min":10.664},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":1.0629999999999999,"15min":0.89200000000000002},"min":{"1min":0.503,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90700000000000003,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.88800000000000001,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.88900000000000001,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.0920000000000001,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.46100000000000002},"max":{"1min":0.93100000000000005,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.66900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87076,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284764,"statfs":{"total":96636764160,"available":96547598336,"internally_reserved":0,"allocated":1916928,"data_stored":842772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.48399999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51800000000000002,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.51800000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.44400000000000001,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58099999999999996,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.54200000000000004},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.378}]},{"osd":3,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59099999999999997,"15min":0.67400000000000004},"min":{"1min":0.47399999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.84499999999999997,"15min":16.382999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.81499999999999995,"15min":18.318000000000001},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.80000000000000004},"min":{"1min":0.45200000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.01,"5min":1.165,"15min":6.1630000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78200000000000003,"15min":0.81000000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.49399999999999999},"max":{"1min":1.0249999999999999,"5min":1.329,"15min":1.78},"last":0.751}]},{"osd":5,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96099999999999997,"5min":1.2549999999999999,"15min":1.6870000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.746,"15min":0.79300000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.085,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.745,"15min":0.79300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0960000000000001,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.81299999999999994},"min":{"1min":0.52000000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.119,"5min":1.204,"15min":6.1970000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76900000000000002,"15min":0.80900000000000005},"min":{"1min":0.63800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.075,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.81699999999999995},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.53800000000000003},"max":{"1min":1.137,"5min":1.145,"15min":1.6359999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574978,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47528,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324312,"statfs":{"total":96636764160,"available":96588095488,"internally_reserved":0,"allocated":2519040,"data_stored":1433137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.58499999999999996},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65300000000000002,"5min":1.018,"15min":5.5259999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.51800000000000002,"15min":0.60699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":0.57599999999999996,"5min":3.004,"15min":5.4889999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.55100000000000005,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.70899999999999996,"5min":3.605,"15min":6.0430000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52900000000000003,"15min":0.64300000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72599999999999998,"5min":1.29,"15min":4.6760000000000002},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51400000000000001,"15min":0.60299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.72299999999999998,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68899999999999995,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.76500000000000001,"15min":0.84499999999999997},"min":{"1min":0.502,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.9490000000000001,"15min":4.734},"last":0.628},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77600000000000002,"15min":0.85499999999999998},"min":{"1min":0.55700000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.137,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.78900000000000003,"15min":0.86799999999999999},"min":{"1min":0.52100000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98399999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.86299999999999999},"min":{"1min":0.54000000000000004,"5min":0.432,"15min":0.432},"max":{"1min":1.1100000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.95199999999999996},"min":{"1min":0.59499999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.161,"5min":1.1859999999999999,"15min":4.8470000000000004},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.878},"min":{"1min":0.495,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.04,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80600000000000005,"15min":0.90300000000000002},"min":{"1min":0.57299999999999995,"5min":0.51100000000000001,"15min":0.47399999999999998},"max":{"1min":1.1259999999999999,"5min":1.405,"15min":5.0540000000000003},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.81999999999999995,"15min":0.89500000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.0980000000000001,"5min":2.706,"15min":4.9950000000000001},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:18:29.760 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:18:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.949+0000 7fd2a9a00640 1 Processor -- start 2024-09-13T17:18:29.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.949+0000 7fd2a9a00640 1 -- start start 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2a40cabc0 0x7fd2a415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd2a415edb0 0x7fd2a4070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40adb00 con 0x7fd2a40cabc0 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a415e100 con 0x7fd2a415edb0 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a415e3e0 con 0x7fd2a415a540 2024-09-13T17:18:29.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2a40cabc0 0x7fd2a415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd2a40cabc0 0x7fd2a415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53634/0 (socket says 172.21.15.28:53634) 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 -- 172.21.15.28:0/1640349790 learned_addr learned my addr 172.21.15.28:0/1640349790 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a2a00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd2a3400640 1 --2- 172.21.15.28:0/1640349790 >> v2:172.21.15.189:3300/0 conn(0x7fd2a415edb0 0x7fd2a4070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 -- 172.21.15.28:0/1640349790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 msgr2=0x7fd2a406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 -- 172.21.15.28:0/1640349790 >> v2:172.21.15.189:3300/0 conn(0x7fd2a415edb0 msgr2=0x7fd2a4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 --2- 172.21.15.28:0/1640349790 >> v2:172.21.15.189:3300/0 conn(0x7fd2a415edb0 0x7fd2a4070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.951+0000 7fd29be00640 1 -- 172.21.15.28:0/1640349790 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a406ce80 con 0x7fd2a40cabc0 2024-09-13T17:18:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.952+0000 7fd29be00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a415e870 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40ca980 tx=0x7fd28c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabc8eb9178d4cec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a2000640 1 -- 172.21.15.28:0/1640349790 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c02ae90 con 0x7fd2a40cabc0 2024-09-13T17:18:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a2000640 1 -- 172.21.15.28:0/1640349790 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c032440 con 0x7fd2a40cabc0 2024-09-13T17:18:29.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a2000640 1 -- 172.21.15.28:0/1640349790 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c0327d0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 -- 172.21.15.28:0/1640349790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 msgr2=0x7fd2a415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a415e870 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40ca980 tx=0x7fd28c02a140 comp rx=0 tx=0).stop 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 -- 172.21.15.28:0/1640349790 shutdown_connections 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a415e870 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/1640349790 >> v2:172.21.15.189:3300/0 conn(0x7fd2a415edb0 0x7fd2a4070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/1640349790 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 -- 172.21.15.28:0/1640349790 >> 172.21.15.28:0/1640349790 conn(0x7fd2a4069e80 msgr2=0x7fd2a406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:29.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 -- 172.21.15.28:0/1640349790 shutdown_connections 2024-09-13T17:18:29.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.953+0000 7fd2a9a00640 1 -- 172.21.15.28:0/1640349790 wait complete. 2024-09-13T17:18:29.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.955+0000 7fd2a9a00640 1 Processor -- start 2024-09-13T17:18:29.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.955+0000 7fd2a9a00640 1 -- start start 2024-09-13T17:18:29.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a4141260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2a41417a0 0x7fd2a40adee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a4153020 con 0x7fd2a40cabc0 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41532e0 con 0x7fd2a41417a0 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a41535c0 con 0x7fd2a415a540 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53648/0 (socket says 172.21.15.28:53648) 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a2a00640 1 -- 172.21.15.28:0/2783713990 learned_addr learned my addr 172.21.15.28:0/2783713990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd29be00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a4141260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.956+0000 7fd2a3400640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2a41417a0 0x7fd2a40adee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 -- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 msgr2=0x7fd2a4141260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a4141260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 -- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2a41417a0 msgr2=0x7fd2a40adee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:29.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2a41417a0 0x7fd2a40adee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 -- 172.21.15.28:0/2783713990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0094d0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a2a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd29400d800 tx=0x7fd29400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400b6c0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400b820 con 0x7fd2a40cabc0 2024-09-13T17:18:29.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a41538a0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400bbb0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.957+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a40b0f90 con 0x7fd2a40cabc0 2024-09-13T17:18:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.958+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a415e870 con 0x7fd2a40cabc0 2024-09-13T17:18:29.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.959+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd29400bd10 con 0x7fd2a40cabc0 2024-09-13T17:18:29.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.960+0000 7fd29aa00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 0x7fd28006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:29.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.960+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2940899a0 con 0x7fd2a40cabc0 2024-09-13T17:18:29.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.961+0000 7fd29be00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 0x7fd28006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:29.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.961+0000 7fd29be00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 0x7fd28006d0c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd28c034000 tx=0x7fd28c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:29.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:29.964+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd294054ae0 con 0x7fd2a40cabc0 2024-09-13T17:18:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.137+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2a4153450 con 0x7fd28006ac00 2024-09-13T17:18:30.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.145+0000 7fd29aa00640 1 -- 172.21.15.28:0/2783713990 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289353 (secure 0 0 0) 0x7fd2a4153450 con 0x7fd28006ac00 2024-09-13T17:18:30.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:30.152 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:18:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 msgr2=0x7fd28006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 0x7fd28006d0c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fd28c034000 tx=0x7fd28c02e000 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 msgr2=0x7fd2a40b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7fd29400d800 tx=0x7fd29400dcc0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 shutdown_connections 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd28006ac00 0x7fd28006d0c0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd2a41417a0 0x7fd2a40adee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd2a415a540 0x7fd2a4141260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 --2- 172.21.15.28:0/2783713990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a40cabc0 0x7fd2a40b0a50 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 >> 172.21.15.28:0/2783713990 conn(0x7fd2a4069e80 msgr2=0x7fd2a406fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 shutdown_connections 2024-09-13T17:18:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.149+0000 7fd2a9a00640 1 -- 172.21.15.28:0/2783713990 wait complete. 2024-09-13T17:18:30.166 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":462,"stamp":"2024-09-13T17:18:29.041754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":506068,"kb_used_data":16160,"kb_used_omap":12,"kb_used_meta":489779,"kb_avail":754468652,"statfs":{"total":773094113280,"available":772575899648,"internally_reserved":0,"allocated":16547840,"data_stored":7920942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":501534265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.917587"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730403+0000","last_change":"2024-09-13T17:18:25.583283+0000","last_active":"2024-09-13T17:18:26.730403+0000","last_peered":"2024-09-13T17:18:26.730403+0000","last_clean":"2024-09-13T17:18:26.730403+0000","last_became_active":"2024-09-13T17:18:25.583010+0000","last_became_peered":"2024-09-13T17:18:25.583010+0000","last_unstale":"2024-09-13T17:18:26.730403+0000","last_undegraded":"2024-09-13T17:18:26.730403+0000","last_fullsized":"2024-09-13T17:18:26.730403+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566235+0000","last_change":"2024-09-13T17:18:25.582821+0000","last_active":"2024-09-13T17:18:26.566235+0000","last_peered":"2024-09-13T17:18:26.566235+0000","last_clean":"2024-09-13T17:18:26.566235+0000","last_became_active":"2024-09-13T17:18:25.582554+0000","last_became_peered":"2024-09-13T17:18:25.582554+0000","last_unstale":"2024-09-13T17:18:26.566235+0000","last_undegraded":"2024-09-13T17:18:26.566235+0000","last_fullsized":"2024-09-13T17:18:26.566235+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:48.905857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721003+0000","last_change":"2024-09-13T17:18:25.590551+0000","last_active":"2024-09-13T17:18:26.721003+0000","last_peered":"2024-09-13T17:18:26.721003+0000","last_clean":"2024-09-13T17:18:26.721003+0000","last_became_active":"2024-09-13T17:18:25.590188+0000","last_became_peered":"2024-09-13T17:18:25.590188+0000","last_unstale":"2024-09-13T17:18:26.721003+0000","last_undegraded":"2024-09-13T17:18:26.721003+0000","last_fullsized":"2024-09-13T17:18:26.721003+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730644+0000","last_change":"2024-09-13T17:18:25.592532+0000","last_active":"2024-09-13T17:18:26.730644+0000","last_peered":"2024-09-13T17:18:26.730644+0000","last_clean":"2024-09-13T17:18:26.730644+0000","last_became_active":"2024-09-13T17:18:25.591789+0000","last_became_peered":"2024-09-13T17:18:25.591789+0000","last_unstale":"2024-09-13T17:18:26.730644+0000","last_undegraded":"2024-09-13T17:18:26.730644+0000","last_fullsized":"2024-09-13T17:18:26.730644+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:44.382887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568585+0000","last_change":"2024-09-13T17:18:25.582580+0000","last_active":"2024-09-13T17:18:26.568585+0000","last_peered":"2024-09-13T17:18:26.568585+0000","last_clean":"2024-09-13T17:18:26.568585+0000","last_became_active":"2024-09-13T17:18:25.582312+0000","last_became_peered":"2024-09-13T17:18:25.582312+0000","last_unstale":"2024-09-13T17:18:26.568585+0000","last_undegraded":"2024-09-13T17:18:26.568585+0000","last_fullsized":"2024-09-13T17:18:26.568585+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:55.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730390+0000","last_change":"2024-09-13T17:18:25.594734+0000","last_active":"2024-09-13T17:18:26.730390+0000","last_peered":"2024-09-13T17:18:26.730390+0000","last_clean":"2024-09-13T17:18:26.730390+0000","last_became_active":"2024-09-13T17:18:25.594478+0000","last_became_peered":"2024-09-13T17:18:25.594478+0000","last_unstale":"2024-09-13T17:18:26.730390+0000","last_undegraded":"2024-09-13T17:18:26.730390+0000","last_fullsized":"2024-09-13T17:18:26.730390+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:28.803886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730309+0000","last_change":"2024-09-13T17:18:25.575232+0000","last_active":"2024-09-13T17:18:26.730309+0000","last_peered":"2024-09-13T17:18:26.730309+0000","last_clean":"2024-09-13T17:18:26.730309+0000","last_became_active":"2024-09-13T17:18:25.572953+0000","last_became_peered":"2024-09-13T17:18:25.572953+0000","last_unstale":"2024-09-13T17:18:26.730309+0000","last_undegraded":"2024-09-13T17:18:26.730309+0000","last_fullsized":"2024-09-13T17:18:26.730309+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:48.905857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.729943+0000","last_change":"2024-09-13T17:18:25.577944+0000","last_active":"2024-09-13T17:18:26.729943+0000","last_peered":"2024-09-13T17:18:26.729943+0000","last_clean":"2024-09-13T17:18:26.729943+0000","last_became_active":"2024-09-13T17:18:25.577433+0000","last_became_peered":"2024-09-13T17:18:25.577433+0000","last_unstale":"2024-09-13T17:18:26.729943+0000","last_undegraded":"2024-09-13T17:18:26.729943+0000","last_fullsized":"2024-09-13T17:18:26.729943+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566233+0000","last_change":"2024-09-13T17:18:25.587625+0000","last_active":"2024-09-13T17:18:26.566233+0000","last_peered":"2024-09-13T17:18:26.566233+0000","last_clean":"2024-09-13T17:18:26.566233+0000","last_became_active":"2024-09-13T17:18:25.585645+0000","last_became_peered":"2024-09-13T17:18:25.585645+0000","last_unstale":"2024-09-13T17:18:26.566233+0000","last_undegraded":"2024-09-13T17:18:26.566233+0000","last_fullsized":"2024-09-13T17:18:26.566233+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:04.478740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569069+0000","last_change":"2024-09-13T17:18:25.592679+0000","last_active":"2024-09-13T17:18:26.569069+0000","last_peered":"2024-09-13T17:18:26.569069+0000","last_clean":"2024-09-13T17:18:26.569069+0000","last_became_active":"2024-09-13T17:18:25.592198+0000","last_became_peered":"2024-09-13T17:18:25.592198+0000","last_unstale":"2024-09-13T17:18:26.569069+0000","last_undegraded":"2024-09-13T17:18:26.569069+0000","last_fullsized":"2024-09-13T17:18:26.569069+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:17.576724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730525+0000","last_change":"2024-09-13T17:18:25.582501+0000","last_active":"2024-09-13T17:18:26.730525+0000","last_peered":"2024-09-13T17:18:26.730525+0000","last_clean":"2024-09-13T17:18:26.730525+0000","last_became_active":"2024-09-13T17:18:25.582239+0000","last_became_peered":"2024-09-13T17:18:25.582239+0000","last_unstale":"2024-09-13T17:18:26.730525+0000","last_undegraded":"2024-09-13T17:18:26.730525+0000","last_fullsized":"2024-09-13T17:18:26.730525+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:49.058248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569131+0000","last_change":"2024-09-13T17:18:25.587801+0000","last_active":"2024-09-13T17:18:26.569131+0000","last_peered":"2024-09-13T17:18:26.569131+0000","last_clean":"2024-09-13T17:18:26.569131+0000","last_became_active":"2024-09-13T17:18:25.583003+0000","last_became_peered":"2024-09-13T17:18:25.583003+0000","last_unstale":"2024-09-13T17:18:26.569131+0000","last_undegraded":"2024-09-13T17:18:26.569131+0000","last_fullsized":"2024-09-13T17:18:26.569131+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:47.264988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568678+0000","last_change":"2024-09-13T17:18:25.577340+0000","last_active":"2024-09-13T17:18:26.568678+0000","last_peered":"2024-09-13T17:18:26.568678+0000","last_clean":"2024-09-13T17:18:26.568678+0000","last_became_active":"2024-09-13T17:18:25.577072+0000","last_became_peered":"2024-09-13T17:18:25.577072+0000","last_unstale":"2024-09-13T17:18:26.568678+0000","last_undegraded":"2024-09-13T17:18:26.568678+0000","last_fullsized":"2024-09-13T17:18:26.568678+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:26.222696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730463+0000","last_change":"2024-09-13T17:18:25.587672+0000","last_active":"2024-09-13T17:18:26.730463+0000","last_peered":"2024-09-13T17:18:26.730463+0000","last_clean":"2024-09-13T17:18:26.730463+0000","last_became_active":"2024-09-13T17:18:25.587274+0000","last_became_peered":"2024-09-13T17:18:25.587274+0000","last_unstale":"2024-09-13T17:18:26.730463+0000","last_undegraded":"2024-09-13T17:18:26.730463+0000","last_fullsized":"2024-09-13T17:18:26.730463+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:23.154924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567973+0000","last_change":"2024-09-13T17:18:25.583450+0000","last_active":"2024-09-13T17:18:26.567973+0000","last_peered":"2024-09-13T17:18:26.567973+0000","last_clean":"2024-09-13T17:18:26.567973+0000","last_became_active":"2024-09-13T17:18:25.581810+0000","last_became_peered":"2024-09-13T17:18:25.581810+0000","last_unstale":"2024-09-13T17:18:26.567973+0000","last_undegraded":"2024-09-13T17:18:26.567973+0000","last_fullsized":"2024-09-13T17:18:26.567973+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:00.965493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599325+0000","last_change":"2024-09-13T17:18:25.579947+0000","last_active":"2024-09-13T17:18:26.599325+0000","last_peered":"2024-09-13T17:18:26.599325+0000","last_clean":"2024-09-13T17:18:26.599325+0000","last_became_active":"2024-09-13T17:18:25.578900+0000","last_became_peered":"2024-09-13T17:18:25.578900+0000","last_unstale":"2024-09-13T17:18:26.599325+0000","last_undegraded":"2024-09-13T17:18:26.599325+0000","last_fullsized":"2024-09-13T17:18:26.599325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:54.986444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569280+0000","last_change":"2024-09-13T17:18:25.585214+0000","last_active":"2024-09-13T17:18:26.569280+0000","last_peered":"2024-09-13T17:18:26.569280+0000","last_clean":"2024-09-13T17:18:26.569280+0000","last_became_active":"2024-09-13T17:18:25.584938+0000","last_became_peered":"2024-09-13T17:18:25.584938+0000","last_unstale":"2024-09-13T17:18:26.569280+0000","last_undegraded":"2024-09-13T17:18:26.569280+0000","last_fullsized":"2024-09-13T17:18:26.569280+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:53.638212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721058+0000","last_change":"2024-09-13T17:18:25.594139+0000","last_active":"2024-09-13T17:18:26.721058+0000","last_peered":"2024-09-13T17:18:26.721058+0000","last_clean":"2024-09-13T17:18:26.721058+0000","last_became_active":"2024-09-13T17:18:25.593882+0000","last_became_peered":"2024-09-13T17:18:25.593882+0000","last_unstale":"2024-09-13T17:18:26.721058+0000","last_undegraded":"2024-09-13T17:18:26.721058+0000","last_fullsized":"2024-09-13T17:18:26.721058+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:26.222696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730624+0000","last_change":"2024-09-13T17:18:25.586675+0000","last_active":"2024-09-13T17:18:26.730624+0000","last_peered":"2024-09-13T17:18:26.730624+0000","last_clean":"2024-09-13T17:18:26.730624+0000","last_became_active":"2024-09-13T17:18:25.586392+0000","last_became_peered":"2024-09-13T17:18:25.586392+0000","last_unstale":"2024-09-13T17:18:26.730624+0000","last_undegraded":"2024-09-13T17:18:26.730624+0000","last_fullsized":"2024-09-13T17:18:26.730624+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:57.490862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599647+0000","last_change":"2024-09-13T17:18:25.574420+0000","last_active":"2024-09-13T17:18:26.599647+0000","last_peered":"2024-09-13T17:18:26.599647+0000","last_clean":"2024-09-13T17:18:26.599647+0000","last_became_active":"2024-09-13T17:18:25.574176+0000","last_became_peered":"2024-09-13T17:18:25.574176+0000","last_unstale":"2024-09-13T17:18:26.599647+0000","last_undegraded":"2024-09-13T17:18:26.599647+0000","last_fullsized":"2024-09-13T17:18:26.599647+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:53:36.210934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568648+0000","last_change":"2024-09-13T17:18:25.592426+0000","last_active":"2024-09-13T17:18:26.568648+0000","last_peered":"2024-09-13T17:18:26.568648+0000","last_clean":"2024-09-13T17:18:26.568648+0000","last_became_active":"2024-09-13T17:18:25.591556+0000","last_became_peered":"2024-09-13T17:18:25.591556+0000","last_unstale":"2024-09-13T17:18:26.568648+0000","last_undegraded":"2024-09-13T17:18:26.568648+0000","last_fullsized":"2024-09-13T17:18:26.568648+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:23.154924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730483+0000","last_change":"2024-09-13T17:18:25.573292+0000","last_active":"2024-09-13T17:18:26.730483+0000","last_peered":"2024-09-13T17:18:26.730483+0000","last_clean":"2024-09-13T17:18:26.730483+0000","last_became_active":"2024-09-13T17:18:25.572701+0000","last_became_peered":"2024-09-13T17:18:25.572701+0000","last_unstale":"2024-09-13T17:18:26.730483+0000","last_undegraded":"2024-09-13T17:18:26.730483+0000","last_fullsized":"2024-09-13T17:18:26.730483+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:56.512584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730353+0000","last_change":"2024-09-13T17:18:25.583023+0000","last_active":"2024-09-13T17:18:26.730353+0000","last_peered":"2024-09-13T17:18:26.730353+0000","last_clean":"2024-09-13T17:18:26.730353+0000","last_became_active":"2024-09-13T17:18:25.582499+0000","last_became_peered":"2024-09-13T17:18:25.582499+0000","last_unstale":"2024-09-13T17:18:26.730353+0000","last_undegraded":"2024-09-13T17:18:26.730353+0000","last_fullsized":"2024-09-13T17:18:26.730353+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:45.622459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567976+0000","last_change":"2024-09-13T17:18:25.587408+0000","last_active":"2024-09-13T17:18:26.567976+0000","last_peered":"2024-09-13T17:18:26.567976+0000","last_clean":"2024-09-13T17:18:26.567976+0000","last_became_active":"2024-09-13T17:18:25.582533+0000","last_became_peered":"2024-09-13T17:18:25.582533+0000","last_unstale":"2024-09-13T17:18:26.567976+0000","last_undegraded":"2024-09-13T17:18:26.567976+0000","last_fullsized":"2024-09-13T17:18:26.567976+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:08.327631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599767+0000","last_change":"2024-09-13T17:18:25.586618+0000","last_active":"2024-09-13T17:18:26.599767+0000","last_peered":"2024-09-13T17:18:26.599767+0000","last_clean":"2024-09-13T17:18:26.599767+0000","last_became_active":"2024-09-13T17:18:25.586363+0000","last_became_peered":"2024-09-13T17:18:25.586363+0000","last_unstale":"2024-09-13T17:18:26.599767+0000","last_undegraded":"2024-09-13T17:18:26.599767+0000","last_fullsized":"2024-09-13T17:18:26.599767+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:51.366450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730124+0000","last_change":"2024-09-13T17:18:25.588869+0000","last_active":"2024-09-13T17:18:26.730124+0000","last_peered":"2024-09-13T17:18:26.730124+0000","last_clean":"2024-09-13T17:18:26.730124+0000","last_became_active":"2024-09-13T17:18:25.588590+0000","last_became_peered":"2024-09-13T17:18:25.588590+0000","last_unstale":"2024-09-13T17:18:26.730124+0000","last_undegraded":"2024-09-13T17:18:26.730124+0000","last_fullsized":"2024-09-13T17:18:26.730124+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:08.431154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599647+0000","last_change":"2024-09-13T17:18:25.593673+0000","last_active":"2024-09-13T17:18:26.599647+0000","last_peered":"2024-09-13T17:18:26.599647+0000","last_clean":"2024-09-13T17:18:26.599647+0000","last_became_active":"2024-09-13T17:18:25.592650+0000","last_became_peered":"2024-09-13T17:18:25.592650+0000","last_unstale":"2024-09-13T17:18:26.599647+0000","last_undegraded":"2024-09-13T17:18:26.599647+0000","last_fullsized":"2024-09-13T17:18:26.599647+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:19.811125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730866+0000","last_change":"2024-09-13T17:18:25.590226+0000","last_active":"2024-09-13T17:18:26.730866+0000","last_peered":"2024-09-13T17:18:26.730866+0000","last_clean":"2024-09-13T17:18:26.730866+0000","last_became_active":"2024-09-13T17:18:25.589973+0000","last_became_peered":"2024-09-13T17:18:25.589973+0000","last_unstale":"2024-09-13T17:18:26.730866+0000","last_undegraded":"2024-09-13T17:18:26.730866+0000","last_fullsized":"2024-09-13T17:18:26.730866+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.729999+0000","last_change":"2024-09-13T17:18:25.594725+0000","last_active":"2024-09-13T17:18:26.729999+0000","last_peered":"2024-09-13T17:18:26.729999+0000","last_clean":"2024-09-13T17:18:26.729999+0000","last_became_active":"2024-09-13T17:18:25.594437+0000","last_became_peered":"2024-09-13T17:18:25.594437+0000","last_unstale":"2024-09-13T17:18:26.729999+0000","last_undegraded":"2024-09-13T17:18:26.729999+0000","last_fullsized":"2024-09-13T17:18:26.729999+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730785+0000","last_change":"2024-09-13T17:18:25.587517+0000","last_active":"2024-09-13T17:18:26.730785+0000","last_peered":"2024-09-13T17:18:26.730785+0000","last_clean":"2024-09-13T17:18:26.730785+0000","last_became_active":"2024-09-13T17:18:25.586936+0000","last_became_peered":"2024-09-13T17:18:25.586936+0000","last_unstale":"2024-09-13T17:18:26.730785+0000","last_undegraded":"2024-09-13T17:18:26.730785+0000","last_fullsized":"2024-09-13T17:18:26.730785+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:53.638212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565980+0000","last_change":"2024-09-13T17:18:25.590428+0000","last_active":"2024-09-13T17:18:26.565980+0000","last_peered":"2024-09-13T17:18:26.565980+0000","last_clean":"2024-09-13T17:18:26.565980+0000","last_became_active":"2024-09-13T17:18:25.589291+0000","last_became_peered":"2024-09-13T17:18:25.589291+0000","last_unstale":"2024-09-13T17:18:26.565980+0000","last_undegraded":"2024-09-13T17:18:26.565980+0000","last_fullsized":"2024-09-13T17:18:26.565980+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:58.998510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599353+0000","last_change":"2024-09-13T17:18:25.573098+0000","last_active":"2024-09-13T17:18:26.599353+0000","last_peered":"2024-09-13T17:18:26.599353+0000","last_clean":"2024-09-13T17:18:26.599353+0000","last_became_active":"2024-09-13T17:18:25.572732+0000","last_became_peered":"2024-09-13T17:18:25.572732+0000","last_unstale":"2024-09-13T17:18:26.599353+0000","last_undegraded":"2024-09-13T17:18:26.599353+0000","last_fullsized":"2024-09-13T17:18:26.599353+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:03.912654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569183+0000","last_change":"2024-09-13T17:18:25.586629+0000","last_active":"2024-09-13T17:18:26.569183+0000","last_peered":"2024-09-13T17:18:26.569183+0000","last_clean":"2024-09-13T17:18:26.569183+0000","last_became_active":"2024-09-13T17:18:25.586244+0000","last_became_peered":"2024-09-13T17:18:25.586244+0000","last_unstale":"2024-09-13T17:18:26.569183+0000","last_undegraded":"2024-09-13T17:18:26.569183+0000","last_fullsized":"2024-09-13T17:18:26.569183+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:12:56.189045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"152'4","reported_seq":15,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569183+0000","last_change":"2024-09-13T17:18:25.160980+0000","last_active":"2024-09-13T17:18:26.569183+0000","last_peered":"2024-09-13T17:18:26.569183+0000","last_clean":"2024-09-13T17:18:26.569183+0000","last_became_active":"2024-09-13T17:18:25.160678+0000","last_became_peered":"2024-09-13T17:18:25.160678+0000","last_unstale":"2024-09-13T17:18:26.569183+0000","last_undegraded":"2024-09-13T17:18:26.569183+0000","last_fullsized":"2024-09-13T17:18:26.569183+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:05.830174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566341+0000","last_change":"2024-09-13T17:18:25.584385+0000","last_active":"2024-09-13T17:18:26.566341+0000","last_peered":"2024-09-13T17:18:26.566341+0000","last_clean":"2024-09-13T17:18:26.566341+0000","last_became_active":"2024-09-13T17:18:25.583039+0000","last_became_peered":"2024-09-13T17:18:25.583039+0000","last_unstale":"2024-09-13T17:18:26.566341+0000","last_undegraded":"2024-09-13T17:18:26.566341+0000","last_fullsized":"2024-09-13T17:18:26.566341+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:26.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566283+0000","last_change":"2024-09-13T17:18:25.153960+0000","last_active":"2024-09-13T17:18:26.566283+0000","last_peered":"2024-09-13T17:18:26.566283+0000","last_clean":"2024-09-13T17:18:26.566283+0000","last_became_active":"2024-09-13T17:18:25.153547+0000","last_became_peered":"2024-09-13T17:18:25.153547+0000","last_unstale":"2024-09-13T17:18:26.566283+0000","last_undegraded":"2024-09-13T17:18:26.566283+0000","last_fullsized":"2024-09-13T17:18:26.566283+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.145798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730066+0000","last_change":"2024-09-13T17:18:25.593654+0000","last_active":"2024-09-13T17:18:26.730066+0000","last_peered":"2024-09-13T17:18:26.730066+0000","last_clean":"2024-09-13T17:18:26.730066+0000","last_became_active":"2024-09-13T17:18:25.593332+0000","last_became_peered":"2024-09-13T17:18:25.593332+0000","last_unstale":"2024-09-13T17:18:26.730066+0000","last_undegraded":"2024-09-13T17:18:26.730066+0000","last_fullsized":"2024-09-13T17:18:26.730066+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:20.666847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730655+0000","last_change":"2024-09-13T17:18:25.145149+0000","last_active":"2024-09-13T17:18:26.730655+0000","last_peered":"2024-09-13T17:18:26.730655+0000","last_clean":"2024-09-13T17:18:26.730655+0000","last_became_active":"2024-09-13T17:18:25.144882+0000","last_became_peered":"2024-09-13T17:18:25.144882+0000","last_unstale":"2024-09-13T17:18:26.730655+0000","last_undegraded":"2024-09-13T17:18:26.730655+0000","last_fullsized":"2024-09-13T17:18:26.730655+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:54:47.827518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568646+0000","last_change":"2024-09-13T17:18:25.584379+0000","last_active":"2024-09-13T17:18:26.568646+0000","last_peered":"2024-09-13T17:18:26.568646+0000","last_clean":"2024-09-13T17:18:26.568646+0000","last_became_active":"2024-09-13T17:18:25.583871+0000","last_became_peered":"2024-09-13T17:18:25.583871+0000","last_unstale":"2024-09-13T17:18:26.568646+0000","last_undegraded":"2024-09-13T17:18:26.568646+0000","last_fullsized":"2024-09-13T17:18:26.568646+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:52.658355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730681+0000","last_change":"2024-09-13T17:18:25.147372+0000","last_active":"2024-09-13T17:18:26.730681+0000","last_peered":"2024-09-13T17:18:26.730681+0000","last_clean":"2024-09-13T17:18:26.730681+0000","last_became_active":"2024-09-13T17:18:25.146276+0000","last_became_peered":"2024-09-13T17:18:25.146276+0000","last_unstale":"2024-09-13T17:18:26.730681+0000","last_undegraded":"2024-09-13T17:18:26.730681+0000","last_fullsized":"2024-09-13T17:18:26.730681+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:24.774066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566215+0000","last_change":"2024-09-13T17:18:25.587085+0000","last_active":"2024-09-13T17:18:26.566215+0000","last_peered":"2024-09-13T17:18:26.566215+0000","last_clean":"2024-09-13T17:18:26.566215+0000","last_became_active":"2024-09-13T17:18:25.586816+0000","last_became_peered":"2024-09-13T17:18:25.586816+0000","last_unstale":"2024-09-13T17:18:26.566215+0000","last_undegraded":"2024-09-13T17:18:26.566215+0000","last_fullsized":"2024-09-13T17:18:26.566215+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:48:15.552013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599499+0000","last_change":"2024-09-13T17:18:25.158637+0000","last_active":"2024-09-13T17:18:26.599499+0000","last_peered":"2024-09-13T17:18:26.599499+0000","last_clean":"2024-09-13T17:18:26.599499+0000","last_became_active":"2024-09-13T17:18:25.158286+0000","last_became_peered":"2024-09-13T17:18:25.158286+0000","last_unstale":"2024-09-13T17:18:26.599499+0000","last_undegraded":"2024-09-13T17:18:26.599499+0000","last_fullsized":"2024-09-13T17:18:26.599499+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:14:01.047481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730533+0000","last_change":"2024-09-13T17:18:25.589290+0000","last_active":"2024-09-13T17:18:26.730533+0000","last_peered":"2024-09-13T17:18:26.730533+0000","last_clean":"2024-09-13T17:18:26.730533+0000","last_became_active":"2024-09-13T17:18:25.589023+0000","last_became_peered":"2024-09-13T17:18:25.589023+0000","last_unstale":"2024-09-13T17:18:26.730533+0000","last_undegraded":"2024-09-13T17:18:26.730533+0000","last_fullsized":"2024-09-13T17:18:26.730533+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:17.576724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568298+0000","last_change":"2024-09-13T17:18:25.157554+0000","last_active":"2024-09-13T17:18:26.568298+0000","last_peered":"2024-09-13T17:18:26.568298+0000","last_clean":"2024-09-13T17:18:26.568298+0000","last_became_active":"2024-09-13T17:18:25.157194+0000","last_became_peered":"2024-09-13T17:18:25.157194+0000","last_unstale":"2024-09-13T17:18:26.568298+0000","last_undegraded":"2024-09-13T17:18:26.568298+0000","last_fullsized":"2024-09-13T17:18:26.568298+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:23.668919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565933+0000","last_change":"2024-09-13T17:18:25.575346+0000","last_active":"2024-09-13T17:18:26.565933+0000","last_peered":"2024-09-13T17:18:26.565933+0000","last_clean":"2024-09-13T17:18:26.565933+0000","last_became_active":"2024-09-13T17:18:25.575108+0000","last_became_peered":"2024-09-13T17:18:25.575108+0000","last_unstale":"2024-09-13T17:18:26.565933+0000","last_undegraded":"2024-09-13T17:18:26.565933+0000","last_fullsized":"2024-09-13T17:18:26.565933+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:12.622765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599404+0000","last_change":"2024-09-13T17:18:25.148369+0000","last_active":"2024-09-13T17:18:26.599404+0000","last_peered":"2024-09-13T17:18:26.599404+0000","last_clean":"2024-09-13T17:18:26.599404+0000","last_became_active":"2024-09-13T17:18:25.146221+0000","last_became_peered":"2024-09-13T17:18:25.146221+0000","last_unstale":"2024-09-13T17:18:26.599404+0000","last_undegraded":"2024-09-13T17:18:26.599404+0000","last_fullsized":"2024-09-13T17:18:26.599404+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:59.409452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567710+0000","last_change":"2024-09-13T17:18:25.579818+0000","last_active":"2024-09-13T17:18:26.567710+0000","last_peered":"2024-09-13T17:18:26.567710+0000","last_clean":"2024-09-13T17:18:26.567710+0000","last_became_active":"2024-09-13T17:18:25.579502+0000","last_became_peered":"2024-09-13T17:18:25.579502+0000","last_unstale":"2024-09-13T17:18:26.567710+0000","last_undegraded":"2024-09-13T17:18:26.567710+0000","last_fullsized":"2024-09-13T17:18:26.567710+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:28.001858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730328+0000","last_change":"2024-09-13T17:18:25.154251+0000","last_active":"2024-09-13T17:18:26.730328+0000","last_peered":"2024-09-13T17:18:26.730328+0000","last_clean":"2024-09-13T17:18:26.730328+0000","last_became_active":"2024-09-13T17:18:25.153833+0000","last_became_peered":"2024-09-13T17:18:25.153833+0000","last_unstale":"2024-09-13T17:18:26.730328+0000","last_undegraded":"2024-09-13T17:18:26.730328+0000","last_fullsized":"2024-09-13T17:18:26.730328+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:23.325245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569035+0000","last_change":"2024-09-13T17:18:25.573589+0000","last_active":"2024-09-13T17:18:26.569035+0000","last_peered":"2024-09-13T17:18:26.569035+0000","last_clean":"2024-09-13T17:18:26.569035+0000","last_became_active":"2024-09-13T17:18:25.573093+0000","last_became_peered":"2024-09-13T17:18:25.573093+0000","last_unstale":"2024-09-13T17:18:26.569035+0000","last_undegraded":"2024-09-13T17:18:26.569035+0000","last_fullsized":"2024-09-13T17:18:26.569035+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:49.058248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566408+0000","last_change":"2024-09-13T17:18:25.143056+0000","last_active":"2024-09-13T17:18:26.566408+0000","last_peered":"2024-09-13T17:18:26.566408+0000","last_clean":"2024-09-13T17:18:26.566408+0000","last_became_active":"2024-09-13T17:18:25.142629+0000","last_became_peered":"2024-09-13T17:18:25.142629+0000","last_unstale":"2024-09-13T17:18:26.566408+0000","last_undegraded":"2024-09-13T17:18:26.566408+0000","last_fullsized":"2024-09-13T17:18:26.566408+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:43.373760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569142+0000","last_change":"2024-09-13T17:18:25.586255+0000","last_active":"2024-09-13T17:18:26.569142+0000","last_peered":"2024-09-13T17:18:26.569142+0000","last_clean":"2024-09-13T17:18:26.569142+0000","last_became_active":"2024-09-13T17:18:25.582344+0000","last_became_peered":"2024-09-13T17:18:25.582344+0000","last_unstale":"2024-09-13T17:18:26.569142+0000","last_undegraded":"2024-09-13T17:18:26.569142+0000","last_fullsized":"2024-09-13T17:18:26.569142+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:56.512584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569267+0000","last_change":"2024-09-13T17:18:25.161288+0000","last_active":"2024-09-13T17:18:26.569267+0000","last_peered":"2024-09-13T17:18:26.569267+0000","last_clean":"2024-09-13T17:18:26.569267+0000","last_became_active":"2024-09-13T17:18:25.160983+0000","last_became_peered":"2024-09-13T17:18:25.160983+0000","last_unstale":"2024-09-13T17:18:26.569267+0000","last_undegraded":"2024-09-13T17:18:26.569267+0000","last_fullsized":"2024-09-13T17:18:26.569267+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:07:58.767168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599717+0000","last_change":"2024-09-13T17:18:25.582346+0000","last_active":"2024-09-13T17:18:26.599717+0000","last_peered":"2024-09-13T17:18:26.599717+0000","last_clean":"2024-09-13T17:18:26.599717+0000","last_became_active":"2024-09-13T17:18:25.581084+0000","last_became_peered":"2024-09-13T17:18:25.581084+0000","last_unstale":"2024-09-13T17:18:26.599717+0000","last_undegraded":"2024-09-13T17:18:26.599717+0000","last_fullsized":"2024-09-13T17:18:26.599717+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:58.673315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"152'6","reported_seq":17,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730986+0000","last_change":"2024-09-13T17:18:25.142894+0000","last_active":"2024-09-13T17:18:26.730986+0000","last_peered":"2024-09-13T17:18:26.730986+0000","last_clean":"2024-09-13T17:18:26.730986+0000","last_became_active":"2024-09-13T17:18:25.142506+0000","last_became_peered":"2024-09-13T17:18:25.142506+0000","last_unstale":"2024-09-13T17:18:26.730986+0000","last_undegraded":"2024-09-13T17:18:26.730986+0000","last_fullsized":"2024-09-13T17:18:26.730986+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:22.145798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720820+0000","last_change":"2024-09-13T17:18:25.585949+0000","last_active":"2024-09-13T17:18:26.720820+0000","last_peered":"2024-09-13T17:18:26.720820+0000","last_clean":"2024-09-13T17:18:26.720820+0000","last_became_active":"2024-09-13T17:18:25.585706+0000","last_became_peered":"2024-09-13T17:18:25.585706+0000","last_unstale":"2024-09-13T17:18:26.720820+0000","last_undegraded":"2024-09-13T17:18:26.720820+0000","last_fullsized":"2024-09-13T17:18:26.720820+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720902+0000","last_change":"2024-09-13T17:18:25.154349+0000","last_active":"2024-09-13T17:18:26.720902+0000","last_peered":"2024-09-13T17:18:26.720902+0000","last_clean":"2024-09-13T17:18:26.720902+0000","last_became_active":"2024-09-13T17:18:25.152804+0000","last_became_peered":"2024-09-13T17:18:25.152804+0000","last_unstale":"2024-09-13T17:18:26.720902+0000","last_undegraded":"2024-09-13T17:18:26.720902+0000","last_fullsized":"2024-09-13T17:18:26.720902+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:18.506679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565999+0000","last_change":"2024-09-13T17:18:25.580515+0000","last_active":"2024-09-13T17:18:26.565999+0000","last_peered":"2024-09-13T17:18:26.565999+0000","last_clean":"2024-09-13T17:18:26.565999+0000","last_became_active":"2024-09-13T17:18:25.580217+0000","last_became_peered":"2024-09-13T17:18:25.580217+0000","last_unstale":"2024-09-13T17:18:26.565999+0000","last_undegraded":"2024-09-13T17:18:26.565999+0000","last_fullsized":"2024-09-13T17:18:26.565999+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:14.346865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720824+0000","last_change":"2024-09-13T17:18:25.150316+0000","last_active":"2024-09-13T17:18:26.720824+0000","last_peered":"2024-09-13T17:18:26.720824+0000","last_clean":"2024-09-13T17:18:26.720824+0000","last_became_active":"2024-09-13T17:18:25.150010+0000","last_became_peered":"2024-09-13T17:18:25.150010+0000","last_unstale":"2024-09-13T17:18:26.720824+0000","last_undegraded":"2024-09-13T17:18:26.720824+0000","last_fullsized":"2024-09-13T17:18:26.720824+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:27.544810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567911+0000","last_change":"2024-09-13T17:18:25.580865+0000","last_active":"2024-09-13T17:18:26.567911+0000","last_peered":"2024-09-13T17:18:26.567911+0000","last_clean":"2024-09-13T17:18:26.567911+0000","last_became_active":"2024-09-13T17:18:25.580609+0000","last_became_peered":"2024-09-13T17:18:25.580609+0000","last_unstale":"2024-09-13T17:18:26.567911+0000","last_undegraded":"2024-09-13T17:18:26.567911+0000","last_fullsized":"2024-09-13T17:18:26.567911+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:58.153857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730175+0000","last_change":"2024-09-13T17:18:25.143107+0000","last_active":"2024-09-13T17:18:26.730175+0000","last_peered":"2024-09-13T17:18:26.730175+0000","last_clean":"2024-09-13T17:18:26.730175+0000","last_became_active":"2024-09-13T17:18:25.142839+0000","last_became_peered":"2024-09-13T17:18:25.142839+0000","last_unstale":"2024-09-13T17:18:26.730175+0000","last_undegraded":"2024-09-13T17:18:26.730175+0000","last_fullsized":"2024-09-13T17:18:26.730175+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:49:02.903527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720604+0000","last_change":"2024-09-13T17:18:25.586714+0000","last_active":"2024-09-13T17:18:26.720604+0000","last_peered":"2024-09-13T17:18:26.720604+0000","last_clean":"2024-09-13T17:18:26.720604+0000","last_became_active":"2024-09-13T17:18:25.584465+0000","last_became_peered":"2024-09-13T17:18:25.584465+0000","last_unstale":"2024-09-13T17:18:26.720604+0000","last_undegraded":"2024-09-13T17:18:26.720604+0000","last_fullsized":"2024-09-13T17:18:26.720604+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"152'8","reported_seq":19,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599444+0000","last_change":"2024-09-13T17:18:25.154342+0000","last_active":"2024-09-13T17:18:26.599444+0000","last_peered":"2024-09-13T17:18:26.599444+0000","last_clean":"2024-09-13T17:18:26.599444+0000","last_became_active":"2024-09-13T17:18:25.154088+0000","last_became_peered":"2024-09-13T17:18:25.154088+0000","last_unstale":"2024-09-13T17:18:26.599444+0000","last_undegraded":"2024-09-13T17:18:26.599444+0000","last_fullsized":"2024-09-13T17:18:26.599444+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:21:53.047516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569297+0000","last_change":"2024-09-13T17:18:25.591945+0000","last_active":"2024-09-13T17:18:26.569297+0000","last_peered":"2024-09-13T17:18:26.569297+0000","last_clean":"2024-09-13T17:18:26.569297+0000","last_became_active":"2024-09-13T17:18:25.590685+0000","last_became_peered":"2024-09-13T17:18:25.590685+0000","last_unstale":"2024-09-13T17:18:26.569297+0000","last_undegraded":"2024-09-13T17:18:26.569297+0000","last_fullsized":"2024-09-13T17:18:26.569297+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:40.698757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568774+0000","last_change":"2024-09-13T17:18:25.157365+0000","last_active":"2024-09-13T17:18:26.568774+0000","last_peered":"2024-09-13T17:18:26.568774+0000","last_clean":"2024-09-13T17:18:26.568774+0000","last_became_active":"2024-09-13T17:18:25.156888+0000","last_became_peered":"2024-09-13T17:18:25.156888+0000","last_unstale":"2024-09-13T17:18:26.568774+0000","last_undegraded":"2024-09-13T17:18:26.568774+0000","last_fullsized":"2024-09-13T17:18:26.568774+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:18.680287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599797+0000","last_change":"2024-09-13T17:18:25.585900+0000","last_active":"2024-09-13T17:18:26.599797+0000","last_peered":"2024-09-13T17:18:26.599797+0000","last_clean":"2024-09-13T17:18:26.599797+0000","last_became_active":"2024-09-13T17:18:25.583822+0000","last_became_peered":"2024-09-13T17:18:25.583822+0000","last_unstale":"2024-09-13T17:18:26.599797+0000","last_undegraded":"2024-09-13T17:18:26.599797+0000","last_fullsized":"2024-09-13T17:18:26.599797+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:02:04.937114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566327+0000","last_change":"2024-09-13T17:18:25.154568+0000","last_active":"2024-09-13T17:18:26.566327+0000","last_peered":"2024-09-13T17:18:26.566327+0000","last_clean":"2024-09-13T17:18:26.566327+0000","last_became_active":"2024-09-13T17:18:25.154240+0000","last_became_peered":"2024-09-13T17:18:25.154240+0000","last_unstale":"2024-09-13T17:18:26.566327+0000","last_undegraded":"2024-09-13T17:18:26.566327+0000","last_fullsized":"2024-09-13T17:18:26.566327+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:46.255862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721003+0000","last_change":"2024-09-13T17:18:25.594539+0000","last_active":"2024-09-13T17:18:26.721003+0000","last_peered":"2024-09-13T17:18:26.721003+0000","last_clean":"2024-09-13T17:18:26.721003+0000","last_became_active":"2024-09-13T17:18:25.594221+0000","last_became_peered":"2024-09-13T17:18:25.594221+0000","last_unstale":"2024-09-13T17:18:26.721003+0000","last_undegraded":"2024-09-13T17:18:26.721003+0000","last_fullsized":"2024-09-13T17:18:26.721003+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:55.748123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"152'1","reported_seq":12,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599728+0000","last_change":"2024-09-13T17:18:25.152130+0000","last_active":"2024-09-13T17:18:26.599728+0000","last_peered":"2024-09-13T17:18:26.599728+0000","last_clean":"2024-09-13T17:18:26.599728+0000","last_became_active":"2024-09-13T17:18:25.151882+0000","last_became_peered":"2024-09-13T17:18:25.151882+0000","last_unstale":"2024-09-13T17:18:26.599728+0000","last_undegraded":"2024-09-13T17:18:26.599728+0000","last_fullsized":"2024-09-13T17:18:26.599728+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:37.730295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730507+0000","last_change":"2024-09-13T17:18:25.596129+0000","last_active":"2024-09-13T17:18:26.730507+0000","last_peered":"2024-09-13T17:18:26.730507+0000","last_clean":"2024-09-13T17:18:26.730507+0000","last_became_active":"2024-09-13T17:18:25.595037+0000","last_became_peered":"2024-09-13T17:18:25.595037+0000","last_unstale":"2024-09-13T17:18:26.730507+0000","last_undegraded":"2024-09-13T17:18:26.730507+0000","last_fullsized":"2024-09-13T17:18:26.730507+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:28.001858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566157+0000","last_change":"2024-09-13T17:18:25.145279+0000","last_active":"2024-09-13T17:18:26.566157+0000","last_peered":"2024-09-13T17:18:26.566157+0000","last_clean":"2024-09-13T17:18:26.566157+0000","last_became_active":"2024-09-13T17:18:25.144493+0000","last_became_peered":"2024-09-13T17:18:25.144493+0000","last_unstale":"2024-09-13T17:18:26.566157+0000","last_undegraded":"2024-09-13T17:18:26.566157+0000","last_fullsized":"2024-09-13T17:18:26.566157+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:55.503457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730780+0000","last_change":"2024-09-13T17:18:25.588921+0000","last_active":"2024-09-13T17:18:26.730780+0000","last_peered":"2024-09-13T17:18:26.730780+0000","last_clean":"2024-09-13T17:18:26.730780+0000","last_became_active":"2024-09-13T17:18:25.588590+0000","last_became_peered":"2024-09-13T17:18:25.588590+0000","last_unstale":"2024-09-13T17:18:26.730780+0000","last_undegraded":"2024-09-13T17:18:26.730780+0000","last_fullsized":"2024-09-13T17:18:26.730780+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:40.698757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730589+0000","last_change":"2024-09-13T17:18:25.156786+0000","last_active":"2024-09-13T17:18:26.730589+0000","last_peered":"2024-09-13T17:18:26.730589+0000","last_clean":"2024-09-13T17:18:26.730589+0000","last_became_active":"2024-09-13T17:18:25.156504+0000","last_became_peered":"2024-09-13T17:18:25.156504+0000","last_unstale":"2024-09-13T17:18:26.730589+0000","last_undegraded":"2024-09-13T17:18:26.730589+0000","last_fullsized":"2024-09-13T17:18:26.730589+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:51.649229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730506+0000","last_change":"2024-09-13T17:18:25.575414+0000","last_active":"2024-09-13T17:18:26.730506+0000","last_peered":"2024-09-13T17:18:26.730506+0000","last_clean":"2024-09-13T17:18:26.730506+0000","last_became_active":"2024-09-13T17:18:25.573914+0000","last_became_peered":"2024-09-13T17:18:25.573914+0000","last_unstale":"2024-09-13T17:18:26.730506+0000","last_undegraded":"2024-09-13T17:18:26.730506+0000","last_fullsized":"2024-09-13T17:18:26.730506+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:14.346865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568220+0000","last_change":"2024-09-13T17:18:25.157732+0000","last_active":"2024-09-13T17:18:26.568220+0000","last_peered":"2024-09-13T17:18:26.568220+0000","last_clean":"2024-09-13T17:18:26.568220+0000","last_became_active":"2024-09-13T17:18:25.157461+0000","last_became_peered":"2024-09-13T17:18:25.157461+0000","last_unstale":"2024-09-13T17:18:26.568220+0000","last_undegraded":"2024-09-13T17:18:26.568220+0000","last_fullsized":"2024-09-13T17:18:26.568220+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:08.010160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730246+0000","last_change":"2024-09-13T17:18:25.589558+0000","last_active":"2024-09-13T17:18:26.730246+0000","last_peered":"2024-09-13T17:18:26.730246+0000","last_clean":"2024-09-13T17:18:26.730246+0000","last_became_active":"2024-09-13T17:18:25.589187+0000","last_became_peered":"2024-09-13T17:18:25.589187+0000","last_unstale":"2024-09-13T17:18:26.730246+0000","last_undegraded":"2024-09-13T17:18:26.730246+0000","last_fullsized":"2024-09-13T17:18:26.730246+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:00.965493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568060+0000","last_change":"2024-09-13T17:18:25.159414+0000","last_active":"2024-09-13T17:18:26.568060+0000","last_peered":"2024-09-13T17:18:26.568060+0000","last_clean":"2024-09-13T17:18:26.568060+0000","last_became_active":"2024-09-13T17:18:25.158410+0000","last_became_peered":"2024-09-13T17:18:25.158410+0000","last_unstale":"2024-09-13T17:18:26.568060+0000","last_undegraded":"2024-09-13T17:18:26.568060+0000","last_fullsized":"2024-09-13T17:18:26.568060+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:54.738997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730325+0000","last_change":"2024-09-13T17:18:25.573015+0000","last_active":"2024-09-13T17:18:26.730325+0000","last_peered":"2024-09-13T17:18:26.730325+0000","last_clean":"2024-09-13T17:18:26.730325+0000","last_became_active":"2024-09-13T17:18:25.572758+0000","last_became_peered":"2024-09-13T17:18:25.572758+0000","last_unstale":"2024-09-13T17:18:26.730325+0000","last_undegraded":"2024-09-13T17:18:26.730325+0000","last_fullsized":"2024-09-13T17:18:26.730325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:13.683409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720566+0000","last_change":"2024-09-13T17:18:25.153891+0000","last_active":"2024-09-13T17:18:26.720566+0000","last_peered":"2024-09-13T17:18:26.720566+0000","last_clean":"2024-09-13T17:18:26.720566+0000","last_became_active":"2024-09-13T17:18:25.153464+0000","last_became_peered":"2024-09-13T17:18:25.153464+0000","last_unstale":"2024-09-13T17:18:26.720566+0000","last_undegraded":"2024-09-13T17:18:26.720566+0000","last_fullsized":"2024-09-13T17:18:26.720566+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:49.835535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569128+0000","last_change":"2024-09-13T17:18:25.584752+0000","last_active":"2024-09-13T17:18:26.569128+0000","last_peered":"2024-09-13T17:18:26.569128+0000","last_clean":"2024-09-13T17:18:26.569128+0000","last_became_active":"2024-09-13T17:18:25.583587+0000","last_became_peered":"2024-09-13T17:18:25.583587+0000","last_unstale":"2024-09-13T17:18:26.569128+0000","last_undegraded":"2024-09-13T17:18:26.569128+0000","last_fullsized":"2024-09-13T17:18:26.569128+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:00.201368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"152'2","reported_seq":13,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.569296+0000","last_change":"2024-09-13T17:18:25.161166+0000","last_active":"2024-09-13T17:18:26.569296+0000","last_peered":"2024-09-13T17:18:26.569296+0000","last_clean":"2024-09-13T17:18:26.569296+0000","last_became_active":"2024-09-13T17:18:25.160718+0000","last_became_peered":"2024-09-13T17:18:25.160718+0000","last_unstale":"2024-09-13T17:18:26.569296+0000","last_undegraded":"2024-09-13T17:18:26.569296+0000","last_fullsized":"2024-09-13T17:18:26.569296+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:19.734455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721051+0000","last_change":"2024-09-13T17:18:25.585227+0000","last_active":"2024-09-13T17:18:26.721051+0000","last_peered":"2024-09-13T17:18:26.721051+0000","last_clean":"2024-09-13T17:18:26.721051+0000","last_became_active":"2024-09-13T17:18:25.584956+0000","last_became_peered":"2024-09-13T17:18:25.584956+0000","last_unstale":"2024-09-13T17:18:26.721051+0000","last_undegraded":"2024-09-13T17:18:26.721051+0000","last_fullsized":"2024-09-13T17:18:26.721051+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:20.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730217+0000","last_change":"2024-09-13T17:18:25.155260+0000","last_active":"2024-09-13T17:18:26.730217+0000","last_peered":"2024-09-13T17:18:26.730217+0000","last_clean":"2024-09-13T17:18:26.730217+0000","last_became_active":"2024-09-13T17:18:25.154901+0000","last_became_peered":"2024-09-13T17:18:25.154901+0000","last_unstale":"2024-09-13T17:18:26.730217+0000","last_undegraded":"2024-09-13T17:18:26.730217+0000","last_fullsized":"2024-09-13T17:18:26.730217+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:18.801999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.566111+0000","last_change":"2024-09-13T17:18:25.575793+0000","last_active":"2024-09-13T17:18:26.566111+0000","last_peered":"2024-09-13T17:18:26.566111+0000","last_clean":"2024-09-13T17:18:26.566111+0000","last_became_active":"2024-09-13T17:18:25.575544+0000","last_became_peered":"2024-09-13T17:18:25.575544+0000","last_unstale":"2024-09-13T17:18:26.566111+0000","last_undegraded":"2024-09-13T17:18:26.566111+0000","last_fullsized":"2024-09-13T17:18:26.566111+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:15.170284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730600+0000","last_change":"2024-09-13T17:18:25.145846+0000","last_active":"2024-09-13T17:18:26.730600+0000","last_peered":"2024-09-13T17:18:26.730600+0000","last_clean":"2024-09-13T17:18:26.730600+0000","last_became_active":"2024-09-13T17:18:25.144449+0000","last_became_peered":"2024-09-13T17:18:25.144449+0000","last_unstale":"2024-09-13T17:18:26.730600+0000","last_undegraded":"2024-09-13T17:18:26.730600+0000","last_fullsized":"2024-09-13T17:18:26.730600+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:50:53.977318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730556+0000","last_change":"2024-09-13T17:18:25.590857+0000","last_active":"2024-09-13T17:18:26.730556+0000","last_peered":"2024-09-13T17:18:26.730556+0000","last_clean":"2024-09-13T17:18:26.730556+0000","last_became_active":"2024-09-13T17:18:25.590568+0000","last_became_peered":"2024-09-13T17:18:25.590568+0000","last_unstale":"2024-09-13T17:18:26.730556+0000","last_undegraded":"2024-09-13T17:18:26.730556+0000","last_fullsized":"2024-09-13T17:18:26.730556+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:33.135490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730065+0000","last_change":"2024-09-13T17:18:25.146418+0000","last_active":"2024-09-13T17:18:26.730065+0000","last_peered":"2024-09-13T17:18:26.730065+0000","last_clean":"2024-09-13T17:18:26.730065+0000","last_became_active":"2024-09-13T17:18:25.144484+0000","last_became_peered":"2024-09-13T17:18:25.144484+0000","last_unstale":"2024-09-13T17:18:26.730065+0000","last_undegraded":"2024-09-13T17:18:26.730065+0000","last_fullsized":"2024-09-13T17:18:26.730065+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:26.290597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.599375+0000","last_change":"2024-09-13T17:18:25.573883+0000","last_active":"2024-09-13T17:18:26.599375+0000","last_peered":"2024-09-13T17:18:26.599375+0000","last_clean":"2024-09-13T17:18:26.599375+0000","last_became_active":"2024-09-13T17:18:25.573337+0000","last_became_peered":"2024-09-13T17:18:25.573337+0000","last_unstale":"2024-09-13T17:18:26.599375+0000","last_undegraded":"2024-09-13T17:18:26.599375+0000","last_fullsized":"2024-09-13T17:18:26.599375+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:27.299723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730665+0000","last_change":"2024-09-13T17:18:25.149634+0000","last_active":"2024-09-13T17:18:26.730665+0000","last_peered":"2024-09-13T17:18:26.730665+0000","last_clean":"2024-09-13T17:18:26.730665+0000","last_became_active":"2024-09-13T17:18:25.149209+0000","last_became_peered":"2024-09-13T17:18:25.149209+0000","last_unstale":"2024-09-13T17:18:26.730665+0000","last_undegraded":"2024-09-13T17:18:26.730665+0000","last_fullsized":"2024-09-13T17:18:26.730665+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:46.255862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568698+0000","last_change":"2024-09-13T17:18:25.587553+0000","last_active":"2024-09-13T17:18:26.568698+0000","last_peered":"2024-09-13T17:18:26.568698+0000","last_clean":"2024-09-13T17:18:26.568698+0000","last_became_active":"2024-09-13T17:18:25.587264+0000","last_became_peered":"2024-09-13T17:18:25.587264+0000","last_unstale":"2024-09-13T17:18:26.568698+0000","last_undegraded":"2024-09-13T17:18:26.568698+0000","last_fullsized":"2024-09-13T17:18:26.568698+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:28.803886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568372+0000","last_change":"2024-09-13T17:18:25.159730+0000","last_active":"2024-09-13T17:18:26.568372+0000","last_peered":"2024-09-13T17:18:26.568372+0000","last_clean":"2024-09-13T17:18:26.568372+0000","last_became_active":"2024-09-13T17:18:25.159138+0000","last_became_peered":"2024-09-13T17:18:25.159138+0000","last_unstale":"2024-09-13T17:18:26.568372+0000","last_undegraded":"2024-09-13T17:18:26.568372+0000","last_fullsized":"2024-09-13T17:18:26.568372+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:28.633856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.568081+0000","last_change":"2024-09-13T17:18:25.585984+0000","last_active":"2024-09-13T17:18:26.568081+0000","last_peered":"2024-09-13T17:18:26.568081+0000","last_clean":"2024-09-13T17:18:26.568081+0000","last_became_active":"2024-09-13T17:18:25.583547+0000","last_became_peered":"2024-09-13T17:18:25.583547+0000","last_unstale":"2024-09-13T17:18:26.568081+0000","last_undegraded":"2024-09-13T17:18:26.568081+0000","last_fullsized":"2024-09-13T17:18:26.568081+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:44.382887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.720714+0000","last_change":"2024-09-13T17:18:25.155074+0000","last_active":"2024-09-13T17:18:26.720714+0000","last_peered":"2024-09-13T17:18:26.720714+0000","last_clean":"2024-09-13T17:18:26.720714+0000","last_became_active":"2024-09-13T17:18:25.152672+0000","last_became_peered":"2024-09-13T17:18:25.152672+0000","last_unstale":"2024-09-13T17:18:26.720714+0000","last_undegraded":"2024-09-13T17:18:26.720714+0000","last_fullsized":"2024-09-13T17:18:26.720714+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:17.508560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.567884+0000","last_change":"2024-09-13T17:18:25.573804+0000","last_active":"2024-09-13T17:18:26.567884+0000","last_peered":"2024-09-13T17:18:26.567884+0000","last_clean":"2024-09-13T17:18:26.567884+0000","last_became_active":"2024-09-13T17:18:25.573534+0000","last_became_peered":"2024-09-13T17:18:25.573534+0000","last_unstale":"2024-09-13T17:18:26.567884+0000","last_undegraded":"2024-09-13T17:18:26.567884+0000","last_fullsized":"2024-09-13T17:18:26.567884+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:07:51.762097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":11,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.565979+0000","last_change":"2024-09-13T17:18:25.152835+0000","last_active":"2024-09-13T17:18:26.565979+0000","last_peered":"2024-09-13T17:18:26.565979+0000","last_clean":"2024-09-13T17:18:26.565979+0000","last_became_active":"2024-09-13T17:18:25.152548+0000","last_became_peered":"2024-09-13T17:18:25.152548+0000","last_unstale":"2024-09-13T17:18:26.565979+0000","last_undegraded":"2024-09-13T17:18:26.565979+0000","last_fullsized":"2024-09-13T17:18:26.565979+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:16.567597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":10,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.730974+0000","last_change":"2024-09-13T17:18:25.573606+0000","last_active":"2024-09-13T17:18:26.730974+0000","last_peered":"2024-09-13T17:18:26.730974+0000","last_clean":"2024-09-13T17:18:26.730974+0000","last_became_active":"2024-09-13T17:18:25.573359+0000","last_became_peered":"2024-09-13T17:18:25.573359+0000","last_unstale":"2024-09-13T17:18:26.730974+0000","last_undegraded":"2024-09-13T17:18:26.730974+0000","last_fullsized":"2024-09-13T17:18:26.730974+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:25.119736+0000","last_clean_scrub_stamp":"2024-09-13T17:18:25.119736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:27.547752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"152'3","reported_seq":14,"reported_epoch":153,"state":"active+clean","last_fresh":"2024-09-13T17:18:26.721035+0000","last_change":"2024-09-13T17:18:25.154092+0000","last_active":"2024-09-13T17:18:26.721035+0000","last_peered":"2024-09-13T17:18:26.721035+0000","last_clean":"2024-09-13T17:18:26.721035+0000","last_became_active":"2024-09-13T17:18:25.153805+0000","last_became_peered":"2024-09-13T17:18:25.153805+0000","last_unstale":"2024-09-13T17:18:26.721035+0000","last_undegraded":"2024-09-13T17:18:26.721035+0000","last_fullsized":"2024-09-13T17:18:26.721035+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":150,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:18:24.110610+0000","last_clean_scrub_stamp":"2024-09-13T17:18:24.110610+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:07.422028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-09-13T17:18:25.573515+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:18:25.573515+0000","last_peered":"2024-09-13T17:18:25.573515+0000","last_clean":"2024-09-13T17:18:25.573515+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:18:25.573515+0000","last_undegraded":"2024-09-13T17:18:25.573515+0000","last_fullsized":"2024-09-13T17:18:25.573515+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574977,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73700,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298140,"statfs":{"total":96636764160,"available":96561295360,"internally_reserved":0,"allocated":2514944,"data_stored":1433210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":72939979},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40899999999999997,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.996,"5min":2.0880000000000001,"15min":2.0880000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.67900000000000005},"min":{"1min":0.42699999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":2.6739999999999999,"15min":2.6739999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68200000000000005,"15min":0.65700000000000003},"min":{"1min":0.47199999999999998,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.127,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.74199999999999999,"15min":0.70799999999999996},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.92000000000000004,"5min":6.4909999999999997,"15min":6.4909999999999997},"last":0.51800000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.69699999999999995,"15min":0.69799999999999995},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":1.169,"5min":1.544,"15min":1.544},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.70499999999999996},"min":{"1min":0.46000000000000002,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.88500000000000001,"5min":1.605,"15min":1.605},"last":0.45300000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.67700000000000005,"15min":0.67900000000000005},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.0660000000000001,"5min":2.4249999999999998,"15min":2.4249999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73799999999999999,"15min":0.74099999999999999},"min":{"1min":0.45400000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.274,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":0.76200000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.60599999999999998,"15min":0.63400000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76500000000000001,"5min":0.90400000000000003,"15min":1.054},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.64800000000000002,"15min":0.67600000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.35399999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.54500000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65100000000000002,"15min":0.67600000000000005},"min":{"1min":0.375,"5min":0.35699999999999998,"15min":0.33800000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":1.143},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.74199999999999999,"15min":0.76500000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.337},"last":0.82899999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.71399999999999997,"15min":0.746},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":1.048,"5min":1.048,"15min":1.1930000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71699999999999997,"15min":0.754},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.101,"5min":1.101,"15min":1.2030000000000001},"last":0.77700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574978,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1925120,"data_stored":842336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.35299999999999998,"5min":0.34399999999999997,"15min":0.32900000000000001},"max":{"1min":1.0209999999999999,"5min":1.298,"15min":3.6629999999999998},"last":0.501},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.76700000000000002,"15min":0.74099999999999999},"min":{"1min":0.51400000000000001,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":5.9800000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.64500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65000000000000002,"15min":0.65800000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90300000000000002,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.47099999999999997,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89000000000000001,"5min":1.46,"15min":3.3079999999999998},"last":0.81200000000000006}]},{"osd":2,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.72499999999999998,"15min":0.68400000000000005},"min":{"1min":0.44,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":5.9240000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.93300000000000005,"5min":1.004,"15min":3.2629999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.71799999999999997,"15min":0.70999999999999996},"min":{"1min":0.495,"5min":0.45400000000000001,"15min":0.42699999999999999},"max":{"1min":1.167,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70799999999999996,"15min":0.70699999999999996},"min":{"1min":0.46200000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85499999999999998,"5min":3.6720000000000002,"15min":3.742},"last":0.61099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.623,"15min":0.624},"min":{"1min":0.42499999999999999,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.91000000000000003,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.625,"15min":0.64900000000000002},"min":{"1min":0.41299999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85199999999999998,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.85699999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.66600000000000004,"15min":0.70399999999999996},"min":{"1min":0.51000000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.83999999999999997,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.64000000000000001,"15min":0.69199999999999995},"min":{"1min":0.376,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.80800000000000005,"5min":0.92100000000000004,"15min":3.609},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:18:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.70799999999999996,"15min":0.74099999999999999},"min":{"1min":0.45700000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.93700000000000006,"5min":1.601,"15min":4.9669999999999996},"last":0.92300000000000004},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.76100000000000001,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0780000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.98999999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574977,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55140,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316700,"statfs":{"total":96636764160,"available":96580300800,"internally_reserved":0,"allocated":1916928,"data_stored":842212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63300000000000001,"15min":0.76800000000000002},"min":{"1min":0.44800000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94199999999999995,"5min":1.498,"15min":38.005000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.64400000000000002,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89700000000000002,"5min":1.577,"15min":21.376999999999999},"last":0.78600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65300000000000002,"15min":0.72499999999999998},"min":{"1min":0.47299999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95299999999999996,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.66700000000000004,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.41899999999999998,"15min":0.41099999999999998},"max":{"1min":0.99199999999999999,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66300000000000003,"15min":0.745},"min":{"1min":0.54100000000000004,"5min":0.379,"15min":0.379},"max":{"1min":1.0489999999999999,"5min":1.524,"15min":17.268000000000001},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69099999999999995,"15min":0.74199999999999999},"min":{"1min":0.47699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.98199999999999998,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.77800000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.72799999999999998,"15min":1.0980000000000001},"min":{"1min":0.51800000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.86299999999999999,"5min":2.8690000000000002,"15min":113.363},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71799999999999997,"15min":0.754},"min":{"1min":0.54900000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.01,"5min":2.359,"15min":4.1180000000000003},"last":0.72999999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.61099999999999999,"15min":0.621},"min":{"1min":0.39500000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.81299999999999994,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65700000000000003,"15min":0.69899999999999995},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.85599999999999998,"5min":2.827,"15min":2.827},"last":0.878}]},{"osd":6,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.66900000000000004,"15min":0.69699999999999995},"min":{"1min":0.41499999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88200000000000001,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.69099999999999995,"15min":0.71799999999999997},"min":{"1min":0.49299999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.79600000000000004,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.81699999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.72699999999999998,"15min":0.755},"min":{"1min":0.44400000000000001,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89800000000000002,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72899999999999998,"15min":0.75800000000000001},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":0.91400000000000003,"5min":1.9379999999999999,"15min":2.02},"last":0.82899999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574976,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59228,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312612,"statfs":{"total":96636764160,"available":96576114688,"internally_reserved":0,"allocated":1912832,"data_stored":842190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.66700000000000004,"15min":0.65000000000000002},"min":{"1min":0.35499999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.79800000000000004,"5min":1.056,"15min":1.0620000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.71799999999999997,"15min":0.72699999999999998},"min":{"1min":0.5,"5min":0.373,"15min":0.373},"max":{"1min":0.93700000000000006,"5min":1.3400000000000001,"15min":1.341},"last":0.61099999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67000000000000004,"15min":0.66100000000000003},"min":{"1min":0.46899999999999997,"5min":0.374,"15min":0.374},"max":{"1min":1.0509999999999999,"5min":1.173,"15min":1.173},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65000000000000002,"15min":0.63400000000000001},"min":{"1min":0.42599999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.66300000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66900000000000004,"15min":0.65500000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":1.1100000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.70399999999999996,"15min":0.71299999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0249999999999999,"5min":1.22,"15min":2.1259999999999999},"last":0.54600000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.70299999999999996,"15min":0.67700000000000005},"min":{"1min":0.53200000000000003,"5min":0.378,"15min":0.378},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.48399999999999999,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":1.304,"5min":1.304,"15min":1.304},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.46200000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.95699999999999996,"5min":1.284,"15min":1.284},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.30299999999999999},"max":{"1min":0.90100000000000002,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.67300000000000004,"15min":0.66200000000000003},"min":{"1min":0.40699999999999997,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.99099999999999999,"5min":1.274,"15min":1.274},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.73399999999999999},"min":{"1min":0.57499999999999996,"5min":0.46600000000000003,"15min":0.41799999999999998},"max":{"1min":1.0720000000000001,"5min":1.292,"15min":1.413},"last":0.72899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:18:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.71799999999999997,"15min":0.70199999999999996},"min":{"1min":0.47499999999999998,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.252,"5min":1.252,"15min":1.252},"last":0.752},{"interface":"front","average":{"1min":0.752,"5min":0.76100000000000001,"15min":0.73799999999999999},"min":{"1min":0.46500000000000002,"5min":0.44500000000000001,"15min":0.371},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.76700000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574977,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64924,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306916,"statfs":{"total":96636764160,"available":96570281984,"internally_reserved":0,"allocated":1916928,"data_stored":842280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40500000000000003,"15min":0.443},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.55200000000000005,"5min":0.67500000000000004,"15min":1.9810000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.435,"5min":0.45100000000000001,"15min":0.51800000000000002},"min":{"1min":0.32000000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57599999999999996,"5min":0.68300000000000005,"15min":2.96},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49199999999999999,"15min":0.51400000000000001},"min":{"1min":0.34599999999999997,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.64100000000000001,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54800000000000004,"15min":0.65700000000000003},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":3.6829999999999998},"last":0.59699999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.52000000000000002,"15min":0.52600000000000002},"min":{"1min":0.38400000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72599999999999998,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.52800000000000002,"15min":0.57599999999999996},"min":{"1min":0.378,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70899999999999996,"5min":0.85999999999999999,"15min":1.3819999999999999},"last":0.53900000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.70799999999999996,"15min":0.77200000000000002},"min":{"1min":0.45800000000000002,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.075,"5min":1.1040000000000001,"15min":2.8610000000000002},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.72999999999999998,"15min":0.76300000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.94099999999999995,"5min":1.012,"15min":1.3640000000000001},"last":0.95199999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.77000000000000002},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.91700000000000004,"5min":1.032,"15min":1.391},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.71199999999999997,"15min":0.79000000000000004},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0269999999999999,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":0.97499999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.745,"15min":0.76400000000000001},"min":{"1min":0.51400000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.05,"5min":1.0840000000000001,"15min":1.401},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.746,"15min":0.78100000000000003},"min":{"1min":0.59999999999999998,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.01,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":1.004}]},{"osd":7,"last update":"Fri Sep 13 17:18:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.746,"15min":0.81799999999999995},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1020000000000001,"5min":1.1339999999999999,"15min":2.899},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76900000000000002,"15min":0.80900000000000005},"min":{"1min":0.60399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.4359999999999999},"last":0.88900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574977,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59236,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312604,"statfs":{"total":96636764160,"available":96576106496,"internally_reserved":0,"allocated":1925120,"data_stored":842805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.42899999999999999,"15min":0.41999999999999998},"min":{"1min":0.28199999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.504,"5min":1.097,"15min":1.097},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.80300000000000005,"15min":0.56000000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28100000000000003},"max":{"1min":0.65300000000000002,"5min":30.023,"15min":30.023},"last":0.51400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.49099999999999999,"15min":1.3460000000000001},"min":{"1min":0.34499999999999997,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.629,"5min":1.046,"15min":260.53399999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.752,"15min":1.4490000000000001},"min":{"1min":0.32500000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.69599999999999995,"5min":21.279,"15min":262.61399999999998},"last":0.41499999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.80600000000000005,"15min":0.59499999999999997},"min":{"1min":0.34300000000000003,"5min":0.32500000000000001,"15min":0.29099999999999998},"max":{"1min":0.58499999999999996,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57799999999999996},"min":{"1min":0.44400000000000001,"5min":0.439,"15min":0.42199999999999999},"max":{"1min":0.66400000000000003,"5min":1.1559999999999999,"15min":1.214},"last":0.626}]},{"osd":4,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.871,"15min":0.79700000000000004},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":0.95399999999999996,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.98799999999999999,"15min":0.85199999999999998},"min":{"1min":0.48999999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.91000000000000003,"5min":21.213000000000001,"15min":21.213000000000001},"last":0.55300000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":1.0549999999999999,"15min":0.879},"min":{"1min":0.44500000000000001,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":0.876,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.75700000000000001,"15min":0.77000000000000002},"min":{"1min":0.51900000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.86199999999999999,"5min":1.1279999999999999,"15min":1.569},"last":0.75700000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.88500000000000001,"15min":0.80700000000000005},"min":{"1min":0.53300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.93000000000000005,"5min":10.664,"15min":10.664},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":1.0629999999999999,"15min":0.89200000000000002},"min":{"1min":0.503,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.90700000000000003,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.88800000000000001,"15min":0.81799999999999995},"min":{"1min":0.55600000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.88900000000000001,"5min":10.712999999999999,"15min":10.712999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":1.0920000000000001,"15min":0.90000000000000002},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.46100000000000002},"max":{"1min":0.93100000000000005,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.66900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87076,"kb_used_data":1872,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284764,"statfs":{"total":96636764160,"available":96547598336,"internally_reserved":0,"allocated":1916928,"data_stored":842772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.442,"15min":0.48399999999999999},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51800000000000002,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.48899999999999999,"15min":0.51800000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66900000000000004,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.52000000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.44400000000000001,"15min":0.46600000000000003},"min":{"1min":0.30099999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.58099999999999996,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.503,"15min":0.54200000000000004},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.70799999999999996,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.378}]},{"osd":3,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.59099999999999997,"15min":0.67400000000000004},"min":{"1min":0.47399999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.75600000000000001,"5min":0.84499999999999997,"15min":16.382999999999999},"last":0.498},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79500000000000004,"5min":0.81499999999999995,"15min":18.318000000000001},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.80000000000000004},"min":{"1min":0.45200000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.01,"5min":1.165,"15min":6.1630000000000003},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78200000000000003,"15min":0.81000000000000005},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.49399999999999999},"max":{"1min":1.0249999999999999,"5min":1.329,"15min":1.78},"last":0.751}]},{"osd":5,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.74299999999999999,"15min":0.76800000000000002},"min":{"1min":0.502,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96099999999999997,"5min":1.2549999999999999,"15min":1.6870000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.746,"15min":0.79300000000000004},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.085,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.745,"15min":0.79300000000000004},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":1.0960000000000001,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.76400000000000001,"15min":0.81299999999999994},"min":{"1min":0.52000000000000002,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.119,"5min":1.204,"15min":6.1970000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:18:09 2024","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.76900000000000002,"15min":0.80900000000000005},"min":{"1min":0.63800000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.075,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.77700000000000002,"15min":0.81699999999999995},"min":{"1min":0.57199999999999995,"5min":0.56399999999999995,"15min":0.53800000000000003},"max":{"1min":1.137,"5min":1.145,"15min":1.6359999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":13,"seq":55834574978,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47528,"kb_used_data":2460,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324312,"statfs":{"total":96636764160,"available":96588095488,"internally_reserved":0,"allocated":2519040,"data_stored":1433137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.44400000000000001,"15min":0.58499999999999996},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65300000000000002,"5min":1.018,"15min":5.5259999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.439,"5min":0.51800000000000002,"15min":0.60699999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27000000000000002},"max":{"1min":0.57599999999999996,"5min":3.004,"15min":5.4889999999999999},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.55100000000000005,"15min":0.64000000000000001},"min":{"1min":0.35599999999999998,"5min":0.313,"15min":0.313},"max":{"1min":0.70899999999999996,"5min":3.605,"15min":6.0430000000000001},"last":0.496},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52900000000000003,"15min":0.64300000000000002},"min":{"1min":0.34300000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72599999999999998,"5min":1.29,"15min":4.6760000000000002},"last":0.48599999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.51400000000000001,"15min":0.60299999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.72299999999999998,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.59199999999999997,"15min":0.66600000000000004},"min":{"1min":0.437,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.68899999999999995,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.76500000000000001,"15min":0.84499999999999997},"min":{"1min":0.502,"5min":0.443,"15min":0.443},"max":{"1min":0.95399999999999996,"5min":1.9490000000000001,"15min":4.734},"last":0.628},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.77600000000000002,"15min":0.85499999999999998},"min":{"1min":0.55700000000000005,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.137,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.78900000000000003,"15min":0.86799999999999999},"min":{"1min":0.52100000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98399999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.79500000000000004,"15min":0.86299999999999999},"min":{"1min":0.54000000000000004,"5min":0.432,"15min":0.432},"max":{"1min":1.1100000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.65100000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.78100000000000003,"15min":0.95199999999999996},"min":{"1min":0.59499999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.161,"5min":1.1859999999999999,"15min":4.8470000000000004},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.79100000000000004,"15min":0.878},"min":{"1min":0.495,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.04,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.78100000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:18:19 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.80600000000000005,"15min":0.90300000000000002},"min":{"1min":0.57299999999999995,"5min":0.51100000000000001,"15min":0.47399999999999998},"max":{"1min":1.1259999999999999,"5min":1.405,"15min":5.0540000000000003},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.749,"5min":0.81999999999999995,"15min":0.89500000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.0980000000000001,"5min":2.706,"15min":4.9950000000000001},"last":0.64200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:18:30.171 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:18:30.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:18:30.283 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:18:30.282+0000 7f4255000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:18:30.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.367+0000 7f3934200640 1 Processor -- start 2024-09-13T17:18:30.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.367+0000 7f3934200640 1 -- start start 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3934200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f392c15c9a0 0x7f392c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3934200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0b2400 0x7f392c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3934200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f392c070470 con 0x7f392c0ad940 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3934200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f392c0ace30 con 0x7f392c15c9a0 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3934200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0ad110 con 0x7f392c0b2400 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3930e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f392c0ad940 0x7f392c0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55692/0 (socket says 172.21.15.28:55692) 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3930e00640 1 -- 172.21.15.28:0/1036564306 learned_addr learned my addr 172.21.15.28:0/1036564306 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.368+0000 7f3931800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f392c15c9a0 0x7f392c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3932200640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0b2400 0x7f392c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3931800640 1 -- 172.21.15.28:0/1036564306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0b2400 msgr2=0x7f392c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3931800640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0b2400 0x7f392c06bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3931800640 1 -- 172.21.15.28:0/1036564306 >> v1:172.21.15.28:6789/0 conn(0x7f392c0ad940 legacy=0x7f392c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f3931800640 1 -- 172.21.15.28:0/1036564306 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f392c06b170 con 0x7f392c15c9a0 2024-09-13T17:18:30.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.369+0000 7f392be00640 1 -- 172.21.15.28:0/1036564306 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636754315 0 0) 0x7f392c070470 con 0x7f392c0ad940 2024-09-13T17:18:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.370+0000 7f3931800640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0ac850 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f39200097d0 tx=0x7f392002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65abacb63d38db0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.370+0000 7f392be00640 1 -- 172.21.15.28:0/1036564306 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920032440 con 0x7f392c15c9a0 2024-09-13T17:18:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.370+0000 7f392be00640 1 -- 172.21.15.28:0/1036564306 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39200325a0 con 0x7f392c15c9a0 2024-09-13T17:18:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.370+0000 7f392be00640 1 -- 172.21.15.28:0/1036564306 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3920032930 con 0x7f392c15c9a0 2024-09-13T17:18:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 -- 172.21.15.28:0/1036564306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 msgr2=0x7f392c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0ac850 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f39200097d0 tx=0x7f392002a250 comp rx=0 tx=0).stop 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 -- 172.21.15.28:0/1036564306 shutdown_connections 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0b2400 0x7f392c06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 --2- 172.21.15.28:0/1036564306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0ac850 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 -- 172.21.15.28:0/1036564306 >> 172.21.15.28:0/1036564306 conn(0x7f392c0b0270 msgr2=0x7f392c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 -- 172.21.15.28:0/1036564306 shutdown_connections 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.371+0000 7f3934200640 1 -- 172.21.15.28:0/1036564306 wait complete. 2024-09-13T17:18:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.373+0000 7f3934200640 1 Processor -- start 2024-09-13T17:18:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.373+0000 7f3934200640 1 -- start start 2024-09-13T17:18:30.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0ad940 0x7f392c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c0b2400 0x7f392c14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0c4270 con 0x7f392c15c9a0 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0c4550 con 0x7f392c0b2400 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3934200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392c0c4830 con 0x7f392c0ad940 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3932200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3932200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53672/0 (socket says 172.21.15.28:53672) 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3932200640 1 -- 172.21.15.28:0/3458636490 learned_addr learned my addr 172.21.15.28:0/3458636490 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3931800640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0ad940 0x7f392c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.374+0000 7f3930e00640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c0b2400 0x7f392c14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 -- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0ad940 msgr2=0x7f392c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0ad940 0x7f392c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 -- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c0b2400 msgr2=0x7f392c14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c0b2400 0x7f392c14cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 -- 172.21.15.28:0/3458636490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3920009400 con 0x7f392c15c9a0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3932200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f392400ef90 tx=0x7f392400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924009ca0 con 0x7f392c15c9a0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f392400f040 con 0x7f392c15c9a0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392c070930 con 0x7f392c15c9a0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39240146c0 con 0x7f392c15c9a0 2024-09-13T17:18:30.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.375+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f392c0c4df0 con 0x7f392c15c9a0 2024-09-13T17:18:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.376+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392c0c5010 con 0x7f392c15c9a0 2024-09-13T17:18:30.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.377+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3924014820 con 0x7f392c15c9a0 2024-09-13T17:18:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.379+0000 7f3929600640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 0x7f390c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.379+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f392408a0e0 con 0x7f392c15c9a0 2024-09-13T17:18:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.379+0000 7f3931800640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 0x7f390c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.380+0000 7f3931800640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 0x7f390c06d0c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3920034000 tx=0x7f392002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.383+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3924055360 con 0x7f392c15c9a0 2024-09-13T17:18:30.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.625+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f392c15ce10 con 0x7f392c15c9a0 2024-09-13T17:18:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.629+0000 7f3929600640 1 -- 172.21.15.28:0/3458636490 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v22) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3924058980 con 0x7f392c15c9a0 2024-09-13T17:18:30.630 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 msgr2=0x7f390c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 0x7f390c06d0c0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3920034000 tx=0x7f392002e000 comp rx=0 tx=0).stop 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 msgr2=0x7f392c0c37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f392400ef90 tx=0x7f392400c490 comp rx=0 tx=0).stop 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 shutdown_connections 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f390c06ac00 0x7f390c06d0c0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f392c15c9a0 0x7f392c0c37f0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f392c0b2400 0x7f392c14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 --2- 172.21.15.28:0/3458636490 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f392c0ad940 0x7f392c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 >> 172.21.15.28:0/3458636490 conn(0x7f392c0b0270 msgr2=0x7f392c15bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 shutdown_connections 2024-09-13T17:18:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.632+0000 7f3934200640 1 -- 172.21.15.28:0/3458636490 wait complete. 2024-09-13T17:18:30.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.803+0000 7fc200000640 1 Processor -- start 2024-09-13T17:18:30.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.803+0000 7fc200000640 1 -- start start 2024-09-13T17:18:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc200000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1f806f970 0x7fc1f815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc200000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f8159d90 con 0x7fc1f806c2d0 2024-09-13T17:18:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc200000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1f80b2040 con 0x7fc1f81592a0 2024-09-13T17:18:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc200000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1f815b0b0 con 0x7fc1f806f970 2024-09-13T17:18:30.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fcc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc1f806f970 0x7fc1f815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fe000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc1f806c2d0 0x7fc1f80b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55704/0 (socket says 172.21.15.28:55704) 2024-09-13T17:18:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fe000640 1 -- 172.21.15.28:0/4109846800 learned_addr learned my addr 172.21.15.28:0/4109846800 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fcc00640 1 -- 172.21.15.28:0/4109846800 >> v1:172.21.15.189:6789/0 conn(0x7fc1f81592a0 legacy=0x7fc1f8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fcc00640 1 -- 172.21.15.28:0/4109846800 >> v1:172.21.15.28:6789/0 conn(0x7fc1f806c2d0 legacy=0x7fc1f80b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.805+0000 7fc1fcc00640 1 -- 172.21.15.28:0/4109846800 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f8159ff0 con 0x7fc1f806f970 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.806+0000 7fc1fcc00640 1 --2- 172.21.15.28:0/4109846800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f815ab70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40097a0 tx=0x7fc1e402c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709ce779d7d01306 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc1f7e00640 1 -- 172.21.15.28:0/4109846800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4034640 con 0x7fc1f806f970 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc1f7e00640 1 -- 172.21.15.28:0/4109846800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e40347a0 con 0x7fc1f806f970 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc1f7e00640 1 -- 172.21.15.28:0/4109846800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4034b30 con 0x7fc1f806f970 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 -- 172.21.15.28:0/4109846800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 msgr2=0x7fc1f815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 --2- 172.21.15.28:0/4109846800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f815ab70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc1e40097a0 tx=0x7fc1e402c3f0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 -- 172.21.15.28:0/4109846800 shutdown_connections 2024-09-13T17:18:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 --2- 172.21.15.28:0/4109846800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f815ab70 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 -- 172.21.15.28:0/4109846800 >> 172.21.15.28:0/4109846800 conn(0x7fc1f8069e80 msgr2=0x7fc1f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 -- 172.21.15.28:0/4109846800 shutdown_connections 2024-09-13T17:18:30.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.807+0000 7fc200000640 1 -- 172.21.15.28:0/4109846800 wait complete. 2024-09-13T17:18:30.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.809+0000 7fc200000640 1 Processor -- start 2024-09-13T17:18:30.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.809+0000 7fc200000640 1 -- start start 2024-09-13T17:18:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc1f806c2d0 0x7fc1f8152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 0x7fc1f80acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80b0900 con 0x7fc1f806f970 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80b0b30 con 0x7fc1f81592a0 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc200000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f80b0e10 con 0x7fc1f806c2d0 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc1fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc1fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53692/0 (socket says 172.21.15.28:53692) 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.810+0000 7fc1fcc00640 1 -- 172.21.15.28:0/3047198604 learned_addr learned my addr 172.21.15.28:0/3047198604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 -- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc1f806c2d0 msgr2=0x7fc1f8152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc1f806c2d0 0x7fc1f8152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fe000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 0x7fc1f80acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 -- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 msgr2=0x7fc1f80acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:30.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 0x7fc1f80acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 -- 172.21.15.28:0/3047198604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e4009400 con 0x7fc1f806f970 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fe000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 0x7fc1f80acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1fcc00640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4000c00 tx=0x7fc1e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4032b80 con 0x7fc1f806f970 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e4032ce0 con 0x7fc1f806f970 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e403e640 con 0x7fc1f806f970 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.811+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f80b10f0 con 0x7fc1f806f970 2024-09-13T17:18:30.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.812+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f814fc00 con 0x7fc1f806f970 2024-09-13T17:18:30.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.813+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f80ada00 con 0x7fc1f806f970 2024-09-13T17:18:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.813+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc1e403e7a0 con 0x7fc1f806f970 2024-09-13T17:18:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.814+0000 7fc1f5600640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 0x7fc1d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.814+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1e40aa460 con 0x7fc1f806f970 2024-09-13T17:18:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.815+0000 7fc1fd600640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 0x7fc1d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:30.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.815+0000 7fc1fd600640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 0x7fc1d806d0c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec006bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:30.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:30.818+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc1e40755a0 con 0x7fc1f806f970 2024-09-13T17:18:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.035+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1f80ada00 con 0x7fc1f806f970 2024-09-13T17:18:31.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.036+0000 7fc1f5600640 1 -- 172.21.15.28:0/3047198604 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fc1e4078bc0 con 0x7fc1f806f970 2024-09-13T17:18:31.037 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:31.037 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:31.038 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:31.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 msgr2=0x7fc1d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 0x7fc1d806d0c0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec006fd0 tx=0x7fc1ec006bc0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 msgr2=0x7fc1f80ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4000c00 tx=0x7fc1e40046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 shutdown_connections 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc1d806ac00 0x7fc1d806d0c0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc1f81592a0 0x7fc1f80acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.038+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc1f806f970 0x7fc1f80ac850 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.039+0000 7fc200000640 1 --2- 172.21.15.28:0/3047198604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc1f806c2d0 0x7fc1f8152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.039+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 >> 172.21.15.28:0/3047198604 conn(0x7fc1f8069e80 msgr2=0x7fc1f8159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.039+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 shutdown_connections 2024-09-13T17:18:31.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.039+0000 7fc200000640 1 -- 172.21.15.28:0/3047198604 wait complete. 2024-09-13T17:18:31.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 130, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:18:25.559972+0000', 'modified': '2024-09-13T17:18:28.571542+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9420}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9420': {'gid': 9420, 'name': 'c', 'rank': 0, 'incarnation': 127, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/3564905441', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3564905441}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3564905441}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:18:31.459 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:18:31.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:18:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.618+0000 7f6d4e000640 1 Processor -- start 2024-09-13T17:18:31.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.618+0000 7f6d4e000640 1 -- start start 2024-09-13T17:18:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.619+0000 7f6d4e000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 0x7f6d4806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.619+0000 7f6d4e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d4806f3f0 0x7f6d48070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d4e000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4806f920 con 0x7f6d480add00 2024-09-13T17:18:31.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d4e000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4815b290 con 0x7f6d4806f3f0 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d4e000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d4815b570 con 0x7f6d48159770 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d3ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6d4806f3f0 0x7f6d48070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d47e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6d480add00 0x7f6d48159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55708/0 (socket says 172.21.15.28:55708) 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d47e00640 1 -- 172.21.15.28:0/991796673 learned_addr learned my addr 172.21.15.28:0/991796673 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d47400640 1 --2- 172.21.15.28:0/991796673 >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 0x7f6d4806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d3ea00640 1 -- 172.21.15.28:0/991796673 >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 msgr2=0x7f6d4806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/991796673 >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 0x7f6d4806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d3ea00640 1 -- 172.21.15.28:0/991796673 >> v1:172.21.15.28:6789/0 conn(0x7f6d480add00 legacy=0x7f6d48159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d3ea00640 1 -- 172.21.15.28:0/991796673 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4815b850 con 0x7f6d4806f3f0 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d46a00640 1 -- 172.21.15.28:0/991796673 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85785433 0 0) 0x7f6d4806f920 con 0x7f6d480add00 2024-09-13T17:18:31.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.620+0000 7f6d47400640 1 --2- 172.21.15.28:0/991796673 >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 0x7f6d4806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.621+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/991796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d48070260 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f6d48068040 tx=0x7f6d3002a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8767c3353dc859 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d46a00640 1 -- 172.21.15.28:0/991796673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3002add0 con 0x7f6d4806f3f0 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d46a00640 1 -- 172.21.15.28:0/991796673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30032440 con 0x7f6d4806f3f0 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d46a00640 1 -- 172.21.15.28:0/991796673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d300327d0 con 0x7f6d4806f3f0 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 -- 172.21.15.28:0/991796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 msgr2=0x7f6d48070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 --2- 172.21.15.28:0/991796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d48070260 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f6d48068040 tx=0x7f6d3002a080 comp rx=0 tx=0).stop 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 -- 172.21.15.28:0/991796673 shutdown_connections 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 --2- 172.21.15.28:0/991796673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d48070260 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 --2- 172.21.15.28:0/991796673 >> v2:172.21.15.189:3300/0 conn(0x7f6d48159770 0x7f6d4806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 -- 172.21.15.28:0/991796673 >> 172.21.15.28:0/991796673 conn(0x7f6d480b0270 msgr2=0x7f6d480b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 -- 172.21.15.28:0/991796673 shutdown_connections 2024-09-13T17:18:31.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.622+0000 7f6d4e000640 1 -- 172.21.15.28:0/991796673 wait complete. 2024-09-13T17:18:31.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.624+0000 7f6d4e000640 1 Processor -- start 2024-09-13T17:18:31.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.624+0000 7f6d4e000640 1 -- start start 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d480add00 0x7f6d48150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d48159770 0x7f6d481535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4814f8c0 con 0x7f6d4806f3f0 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d48150a90 con 0x7f6d48159770 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d4e000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d48150d70 con 0x7f6d480add00 2024-09-13T17:18:31.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53710/0 (socket says 172.21.15.28:53710) 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.625+0000 7f6d47400640 1 -- 172.21.15.28:0/1929560650 learned_addr learned my addr 172.21.15.28:0/1929560650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 -- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d480add00 msgr2=0x7f6d48150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d480add00 0x7f6d48150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 -- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d48159770 msgr2=0x7f6d481535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d48159770 0x7f6d481535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 -- 172.21.15.28:0/1929560650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d30009400 con 0x7f6d4806f3f0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d47400640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800d7d0 tx=0x7f6d3800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d3fe00640 1 -- 172.21.15.28:0/1929560650 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38004030 con 0x7f6d4806f3f0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d3fe00640 1 -- 172.21.15.28:0/1929560650 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d38004190 con 0x7f6d4806f3f0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d3fe00640 1 -- 172.21.15.28:0/1929560650 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38004520 con 0x7f6d4806f3f0 2024-09-13T17:18:31.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d48151050 con 0x7f6d4806f3f0 2024-09-13T17:18:31.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.626+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d48151610 con 0x7f6d4806f3f0 2024-09-13T17:18:31.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.628+0000 7f6d3fe00640 1 -- 172.21.15.28:0/1929560650 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6d3800b6c0 con 0x7f6d4806f3f0 2024-09-13T17:18:31.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.630+0000 7f6d3fe00640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.630+0000 7f6d3fe00640 1 -- 172.21.15.28:0/1929560650 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d38089570 con 0x7f6d4806f3f0 2024-09-13T17:18:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.630+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.630+0000 7f6d50e88740 1 Processor -- start 2024-09-13T17:18:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 -- start start 2024-09-13T17:18:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 0x7f6d2406d0c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6d30034000 tx=0x7f6d3002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927ae90 0x55d2892caba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892cb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2891e6ab0 con 0x55d28927d6d0 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2891e4240 con 0x55d28927ae90 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.631+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d2891dfb30 con 0x55d2892a4b80 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53722/0 (socket says 172.21.15.28:53722) 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 -- 172.21.15.28:0/329580984 learned_addr learned my addr 172.21.15.28:0/329580984 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927ae90 0x55d2892caba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47e00640 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892cb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 -- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 msgr2=0x55d2892cb0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892cb0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 -- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927ae90 msgr2=0x55d2892caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927ae90 0x55d2892caba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.632+0000 7f6d47400640 1 -- 172.21.15.28:0/329580984 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2891aabd0 con 0x55d28927d6d0 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d47400640 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f6d38054900 tx=0x7f6d38058400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98c7f7767dac0be server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d37e00640 1 -- 172.21.15.28:0/329580984 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38056540 con 0x55d28927d6d0 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d37e00640 1 -- 172.21.15.28:0/329580984 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d380567a0 con 0x55d28927d6d0 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d37e00640 1 -- 172.21.15.28:0/329580984 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38060360 con 0x55d28927d6d0 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d50e88740 1 -- 172.21.15.28:0/329580984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 msgr2=0x55d28927dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.633+0000 7f6d50e88740 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f6d38054900 tx=0x7f6d38058400 comp rx=0 tx=0).stop 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 -- 172.21.15.28:0/329580984 shutdown_connections 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892cb0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927ae90 0x55d2892caba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 --2- 172.21.15.28:0/329580984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28927dab0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 -- 172.21.15.28:0/329580984 >> 172.21.15.28:0/329580984 conn(0x55d289279e00 msgr2=0x55d2892c22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 -- 172.21.15.28:0/329580984 shutdown_connections 2024-09-13T17:18:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 -- 172.21.15.28:0/329580984 wait complete. 2024-09-13T17:18:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 -- start start 2024-09-13T17:18:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.634+0000 7f6d50e88740 1 Processor -- start 2024-09-13T17:18:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d2892a6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d289181260 con 0x55d28927d6d0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d289180f00 con 0x55d2892a4b80 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d288f46f20 con 0x55d2892a2960 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.635+0000 7f6d50e88740 10 client.0 _ll_register_callbacks cb 0x55d2891d91a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53730/0 (socket says 172.21.15.28:53730) 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 -- 172.21.15.28:0/2203751114 learned_addr learned my addr 172.21.15.28:0/2203751114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47e00640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d2892a6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 -- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 msgr2=0x55d2892a6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d2892a6610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 -- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 msgr2=0x55d2892a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 -- 172.21.15.28:0/2203751114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d48151610 con 0x55d28927d6d0 2024-09-13T17:18:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d47400640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f6d3805c3f0 tx=0x7f6d3805b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a02efc5a591f3f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.636+0000 7f6d34c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d38060360 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d380604c0 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d289209cd0 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d380890d0 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 10 client.9570 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55d2891de1b0 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d289209cd0 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6d3801b020 con 0x55d28927d6d0 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 10 client.9570 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 10 client.9570 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 20 client.9570 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892a3740 0x55d2892c6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 4 client.9570 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.637+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d288ed33c0 con 0x55d2892a3740 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.638+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892a3740 0x55d2892c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.638+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892a3740 0x55d2892c6980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57c4e84e21ccb2d2 server_cookie=77cac1c1e1c40b33 in_seq=0 out_seq=0 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.638+0000 7f6d34c00640 10 client.9570 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d34c00640 1 -- 172.21.15.28:0/2203751114 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55d288ed33c0 con 0x55d2892a3740 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d34c00640 10 client.9570 handle_command_reply: tid=0 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 client.9570 shutdown 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 msgr2=0x55d2892a7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f6d3805c3f0 tx=0x7f6d3805b070 comp rx=0 tx=0).stop 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 shutdown_connections 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892a3740 0x55d2892c6980 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d2892a6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 --2- 172.21.15.28:0/2203751114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d2892a7310 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d34c00640 0 client.9570 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 >> 172.21.15.28:0/2203751114 conn(0x55d28927ae90 msgr2=0x55d2892c5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 shutdown_connections 2024-09-13T17:18:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.639+0000 7f6d50e88740 1 -- 172.21.15.28:0/2203751114 wait complete. 2024-09-13T17:18:31.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.640+0000 7f6d50e88740 20 client.9570 trim_cache size 0 max 16384 2024-09-13T17:18:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.655+0000 7f6d50e88740 1 Processor -- start 2024-09-13T17:18:31.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.655+0000 7f6d50e88740 1 -- start start 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 0x55d2892c0740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892beaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2892c71e0 con 0x55d2892a2960 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2892c4840 con 0x55d28927d6d0 2024-09-13T17:18:31.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d2892c1cd0 con 0x55d2892a4b80 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53742/0 (socket says 172.21.15.28:53742) 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.656+0000 7f6d3ea00640 1 -- 172.21.15.28:0/43857534 learned_addr learned my addr 172.21.15.28:0/43857534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 -- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 msgr2=0x55d2892beaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892beaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d47400640 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 0x55d2892c0740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 -- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 msgr2=0x55d2892c0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 0x55d2892c0740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 -- 172.21.15.28:0/43857534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d289180470 con 0x55d2892a2960 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d47400640 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 0x55d2892c0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f6d30035040 tx=0x7f6d3002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c4d1f0a455231c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3cc00640 1 -- 172.21.15.28:0/43857534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30041030 con 0x55d2892a2960 2024-09-13T17:18:31.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3cc00640 1 -- 172.21.15.28:0/43857534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30032930 con 0x55d2892a2960 2024-09-13T17:18:31.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.657+0000 7f6d3cc00640 1 -- 172.21.15.28:0/43857534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30044040 con 0x55d2892a2960 2024-09-13T17:18:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 -- 172.21.15.28:0/43857534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 msgr2=0x55d289278de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f6d30035040 tx=0x7f6d3002e000 comp rx=0 tx=0).stop 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 -- 172.21.15.28:0/43857534 shutdown_connections 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a4b80 0x55d2892beaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d2892a2960 0x55d289278de0 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 --2- 172.21.15.28:0/43857534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d28927d6d0 0x55d2892c0740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 -- 172.21.15.28:0/43857534 >> 172.21.15.28:0/43857534 conn(0x55d28927ae90 msgr2=0x55d2892a6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 -- 172.21.15.28:0/43857534 shutdown_connections 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.658+0000 7f6d50e88740 1 -- 172.21.15.28:0/43857534 wait complete. 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.659+0000 7f6d50e88740 1 -- start start 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.659+0000 7f6d50e88740 1 Processor -- start 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d28932c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d28932cd20 con 0x55d28927d6d0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2892c3dc0 con 0x55d2892a4b80 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d2892a23d0 con 0x55d2892a2960 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d50e88740 10 client.0 _ll_register_callbacks cb 0x7f6d30035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53744/0 (socket says 172.21.15.28:53744) 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 -- 172.21.15.28:0/758371627 learned_addr learned my addr 172.21.15.28:0/758371627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d28932c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47e00640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 -- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 msgr2=0x55d28932c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d28932c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 -- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 msgr2=0x55d2892c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.660+0000 7f6d47400640 1 -- 172.21.15.28:0/758371627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4815b850 con 0x55d28927d6d0 2024-09-13T17:18:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d47400640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f6d38054900 tx=0x7f6d38058280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8b77991c87a513 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d380890d0 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d38089230 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d380567a0 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d380893f0 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d50e88740 10 client.9576 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55d2891a5980 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d380567a0 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.661+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6d3801b040 con 0x55d28927d6d0 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 10 client.9576 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 10 client.9576 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 20 client.9576 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892c9f00 0x55d2892a1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 4 client.9576 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d288ee46c0 con 0x55d2892c9f00 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892c9f00 0x55d2892a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d3ea00640 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892c9f00 0x55d2892a1530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b8a3b4c14bd5 server_cookie=1309bc887c4aae23 in_seq=0 out_seq=0 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.662+0000 7f6d34c00640 10 client.9576 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d34c00640 1 -- 172.21.15.28:0/758371627 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55d288ee46c0 con 0x55d2892c9f00 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d34c00640 10 client.9576 handle_command_reply: tid=0 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 client.9576 shutdown 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 msgr2=0x55d28932bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f6d38054900 tx=0x7f6d38058280 comp rx=0 tx=0).stop 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 shutdown_connections 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55d2892c9f00 0x55d2892a1530 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d2892a4b80 0x55d2892c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d2892a2960 0x55d28932c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 --2- 172.21.15.28:0/758371627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d28927d6d0 0x55d28932bf00 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d34c00640 0 client.9576 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 >> 172.21.15.28:0/758371627 conn(0x55d28927ae90 msgr2=0x55d2892c06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 shutdown_connections 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 1 -- 172.21.15.28:0/758371627 wait complete. 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.663+0000 7f6d50e88740 20 client.9576 trim_cache size 0 max 16384 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 9420, 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:18:31.669 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.100423116, 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 130, 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 153, 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 153, 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.034960138999999 2024-09-13T17:18:31.670 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 msgr2=0x7f6d2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 0x7f6d2406d0c0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f6d30034000 tx=0x7f6d3002c000 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 msgr2=0x7f6d4814f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f6d3800d7d0 tx=0x7f6d3800dc90 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 shutdown_connections 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d48159770 0x7f6d481535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6d480add00 0x7f6d48150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 --2- 172.21.15.28:0/1929560650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6d4806f3f0 0x7f6d4814f380 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 >> 172.21.15.28:0/1929560650 conn(0x7f6d480b0270 msgr2=0x7f6d480aca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 shutdown_connections 2024-09-13T17:18:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.664+0000 7f6d4e000640 1 -- 172.21.15.28:0/1929560650 wait complete. 2024-09-13T17:18:31.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:18:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.834+0000 7fdec6400640 1 Processor -- start 2024-09-13T17:18:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.834+0000 7fdec6400640 1 -- start start 2024-09-13T17:18:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdec6400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdec006f970 0x7fdec015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdec6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec0159d90 con 0x7fdec01592a0 2024-09-13T17:18:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdec6400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec00b2040 con 0x7fdec006c2d0 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdec6400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec015b0b0 con 0x7fdec006f970 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdebf400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdec01592a0 0x7fdec0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55718/0 (socket says 172.21.15.28:55718) 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdebf400640 1 -- 172.21.15.28:0/3864030162 learned_addr learned my addr 172.21.15.28:0/3864030162 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdeb6a00640 1 --2- 172.21.15.28:0/3864030162 >> v2:172.21.15.189:3300/0 conn(0x7fdec006f970 0x7fdec015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdebea00640 1 -- 172.21.15.28:0/3864030162 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637215938 0 0) 0x7fdec0159d90 con 0x7fdec01592a0 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.836+0000 7fdebea00640 1 -- 172.21.15.28:0/3864030162 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdea4005730 con 0x7fdec01592a0 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.837+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3864030162 >> v1:172.21.15.189:6789/0 conn(0x7fdec006c2d0 legacy=0x7fdec00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.837+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3864030162 >> v1:172.21.15.28:6789/0 conn(0x7fdec01592a0 legacy=0x7fdec0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.837+0000 7fdeb6a00640 1 -- 172.21.15.28:0/3864030162 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0159ff0 con 0x7fdec006f970 2024-09-13T17:18:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.838+0000 7fdeb6a00640 1 --2- 172.21.15.28:0/3864030162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006f970 0x7fdec015ab70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d7f88420998f20d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.838+0000 7fdebea00640 1 -- 172.21.15.28:0/3864030162 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8032440 con 0x7fdec006f970 2024-09-13T17:18:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.838+0000 7fdebea00640 1 -- 172.21.15.28:0/3864030162 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea80325a0 con 0x7fdec006f970 2024-09-13T17:18:31.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.838+0000 7fdebea00640 1 -- 172.21.15.28:0/3864030162 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8032930 con 0x7fdec006f970 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 -- 172.21.15.28:0/3864030162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006f970 msgr2=0x7fdec015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 --2- 172.21.15.28:0/3864030162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006f970 0x7fdec015ab70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fdea80097a0 tx=0x7fdea802a230 comp rx=0 tx=0).stop 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 -- 172.21.15.28:0/3864030162 shutdown_connections 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 --2- 172.21.15.28:0/3864030162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006f970 0x7fdec015ab70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 -- 172.21.15.28:0/3864030162 >> 172.21.15.28:0/3864030162 conn(0x7fdec0069e80 msgr2=0x7fdec006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 -- 172.21.15.28:0/3864030162 shutdown_connections 2024-09-13T17:18:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.839+0000 7fdec6400640 1 -- 172.21.15.28:0/3864030162 wait complete. 2024-09-13T17:18:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.840+0000 7fdec6400640 1 Processor -- start 2024-09-13T17:18:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.841+0000 7fdec6400640 1 -- start start 2024-09-13T17:18:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.841+0000 7fdec6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006c2d0 0x7fdec00cef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.841+0000 7fdec6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdec6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 0x7fdec00af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdec6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00ceb30 con 0x7fdec006f970 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdec6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00cf490 con 0x7fdec006c2d0 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdec6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec00cdc80 con 0x7fdec01592a0 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53750/0 (socket says 172.21.15.28:53750) 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 -- 172.21.15.28:0/38924847 learned_addr learned my addr 172.21.15.28:0/38924847 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdebf400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006c2d0 0x7fdec00cef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdebfe00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 0x7fdec00af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 -- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 msgr2=0x7fdec00af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 0x7fdec00af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 -- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006c2d0 msgr2=0x7fdec00cef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006c2d0 0x7fdec00cef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdeb6a00640 1 -- 172.21.15.28:0/38924847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea8009400 con 0x7fdec006f970 2024-09-13T17:18:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.842+0000 7fdebfe00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 0x7fdec00af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdeb6a00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fdea80098c0 tx=0x7fdea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8006e10 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8032e90 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8030ba0 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec00cdeb0 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.843+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec00ce470 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.844+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec00b1270 con 0x7fdec006f970 2024-09-13T17:18:31.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.845+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdea8039040 con 0x7fdec006f970 2024-09-13T17:18:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.846+0000 7fdeb7e00640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 0x7fde9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:31.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.847+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdea80a9ce0 con 0x7fdec006f970 2024-09-13T17:18:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.847+0000 7fdebf400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 0x7fde9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.848+0000 7fdebf400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 0x7fde9c06d0c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0005880 tx=0x7fdeb0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:31.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:31.851+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdea8074da0 con 0x7fdec006f970 2024-09-13T17:18:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.066+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdec00cec70 con 0x7fdec006f970 2024-09-13T17:18:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.067+0000 7fdeb7e00640 1 -- 172.21.15.28:0/38924847 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v130) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdec00cec70 con 0x7fdec006f970 2024-09-13T17:18:32.069 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:18:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 msgr2=0x7fde9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 0x7fde9c06d0c0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0005880 tx=0x7fdeb0009040 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 msgr2=0x7fdec014efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7fdea80098c0 tx=0x7fdea80046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 shutdown_connections 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fde9c06ac00 0x7fde9c06d0c0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdec01592a0 0x7fdec00af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdec006f970 0x7fdec014efc0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 --2- 172.21.15.28:0/38924847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdec006c2d0 0x7fdec00cef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 >> 172.21.15.28:0/38924847 conn(0x7fdec0069e80 msgr2=0x7fdec01597a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 shutdown_connections 2024-09-13T17:18:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:32.069+0000 7fdec6400640 1 -- 172.21.15.28:0/38924847 wait complete. 2024-09-13T17:18:32.086 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:18:32.087 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:18:32.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.087 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:18:32.128 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:18:32.129 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:18:32.130 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:18:32.130 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:18:32.130 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:18:32.130 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:18:32.130 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42069sec preferred_lft 42069sec 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:18:32.131 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:18:32.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.132 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:18:32.227 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:18:32.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.231 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:18:32.270 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:18:32.271 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:18:32.271 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:18:32.271 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:18:32.271 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:18:32.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.271 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:18:32.271 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:32.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:18:32.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:18:32.272 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:18:32.272 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:32.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.417 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:18:32.452 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:18:32.452 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:18:32.453 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:18:32.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:18:32.551 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:18:32.551 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:18:32.586 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:18:32.587 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:18:32.587 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:18:32.662 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:18:32.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:18:32.792 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:18:32.793 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:18:32.818 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:18:32.818 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:18:32.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:18:32.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:18:32.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:18:32.820 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:18:32.820 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:18:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:18:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:18:32.821 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:18:32.823 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:18:32.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.824 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:18:32.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:32.880 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:18:32.925 INFO:teuthology.orchestra.run.smithi028.stdout:client9585 2024-09-13T17:18:32.925 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:18:32.925 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:18:33.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.0 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:33.002 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:33.003 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.112 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173782 s, 60.3 MB/s 2024-09-13T17:18:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149616 s, 70.1 MB/s 2024-09-13T17:18:33.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161902 s, 64.8 MB/s 2024-09-13T17:18:33.166 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167204 s, 62.7 MB/s 2024-09-13T17:18:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.186 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171901 s, 61.0 MB/s 2024-09-13T17:18:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135963 s, 77.1 MB/s 2024-09-13T17:18:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.221 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171317 s, 61.2 MB/s 2024-09-13T17:18:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014446 s, 72.6 MB/s 2024-09-13T17:18:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149894 s, 70.0 MB/s 2024-09-13T17:18:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.272 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147941 s, 70.9 MB/s 2024-09-13T17:18:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.291 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170695 s, 61.4 MB/s 2024-09-13T17:18:33.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149331 s, 70.2 MB/s 2024-09-13T17:18:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017151 s, 61.1 MB/s 2024-09-13T17:18:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171608 s, 61.1 MB/s 2024-09-13T17:18:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145178 s, 72.2 MB/s 2024-09-13T17:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130123 s, 80.6 MB/s 2024-09-13T17:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143681 s, 73.0 MB/s 2024-09-13T17:18:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140577 s, 74.6 MB/s 2024-09-13T17:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140974 s, 74.4 MB/s 2024-09-13T17:18:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132685 s, 79.0 MB/s 2024-09-13T17:18:33.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.463 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168799 s, 62.1 MB/s 2024-09-13T17:18:33.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132729 s, 79.0 MB/s 2024-09-13T17:18:33.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.501 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197103 s, 53.2 MB/s 2024-09-13T17:18:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153846 s, 68.2 MB/s 2024-09-13T17:18:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193927 s, 54.1 MB/s 2024-09-13T17:18:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163101 s, 64.3 MB/s 2024-09-13T17:18:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145963 s, 71.8 MB/s 2024-09-13T17:18:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149643 s, 70.1 MB/s 2024-09-13T17:18:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161583 s, 64.9 MB/s 2024-09-13T17:18:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.629 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156143 s, 67.2 MB/s 2024-09-13T17:18:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015898 s, 66.0 MB/s 2024-09-13T17:18:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.663 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013393 s, 78.3 MB/s 2024-09-13T17:18:33.664 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:33.664 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:33.664 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:33.665 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.1 2024-09-13T17:18:33.665 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:33.665 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:33.665 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:33.665 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.720 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133189 s, 78.7 MB/s 2024-09-13T17:18:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.737 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153965 s, 68.1 MB/s 2024-09-13T17:18:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138198 s, 75.9 MB/s 2024-09-13T17:18:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135538 s, 77.4 MB/s 2024-09-13T17:18:33.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158304 s, 66.2 MB/s 2024-09-13T17:18:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129596 s, 80.9 MB/s 2024-09-13T17:18:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.818 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133288 s, 78.7 MB/s 2024-09-13T17:18:33.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.834 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135251 s, 77.5 MB/s 2024-09-13T17:18:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138197 s, 75.9 MB/s 2024-09-13T17:18:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134396 s, 78.0 MB/s 2024-09-13T17:18:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134078 s, 78.2 MB/s 2024-09-13T17:18:33.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152961 s, 68.6 MB/s 2024-09-13T17:18:33.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.915 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131348 s, 79.8 MB/s 2024-09-13T17:18:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132643 s, 79.1 MB/s 2024-09-13T17:18:33.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.947 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138837 s, 75.5 MB/s 2024-09-13T17:18:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162071 s, 64.7 MB/s 2024-09-13T17:18:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015478 s, 67.7 MB/s 2024-09-13T17:18:33.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:33.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:33.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141501 s, 74.1 MB/s 2024-09-13T17:18:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.016 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150045 s, 69.9 MB/s 2024-09-13T17:18:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169413 s, 61.9 MB/s 2024-09-13T17:18:34.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126819 s, 82.7 MB/s 2024-09-13T17:18:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160548 s, 65.3 MB/s 2024-09-13T17:18:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134155 s, 78.2 MB/s 2024-09-13T17:18:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158716 s, 66.1 MB/s 2024-09-13T17:18:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170615 s, 61.5 MB/s 2024-09-13T17:18:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168309 s, 62.3 MB/s 2024-09-13T17:18:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165126 s, 63.5 MB/s 2024-09-13T17:18:34.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139194 s, 75.3 MB/s 2024-09-13T17:18:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127134 s, 82.5 MB/s 2024-09-13T17:18:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130318 s, 80.5 MB/s 2024-09-13T17:18:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.221 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128208 s, 81.8 MB/s 2024-09-13T17:18:34.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.236 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013596 s, 77.1 MB/s 2024-09-13T17:18:34.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.2 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:34.238 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136936 s, 76.6 MB/s 2024-09-13T17:18:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134003 s, 78.3 MB/s 2024-09-13T17:18:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141278 s, 74.2 MB/s 2024-09-13T17:18:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169876 s, 61.7 MB/s 2024-09-13T17:18:34.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.361 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140388 s, 74.7 MB/s 2024-09-13T17:18:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.376 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128949 s, 81.3 MB/s 2024-09-13T17:18:34.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135012 s, 77.7 MB/s 2024-09-13T17:18:34.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.408 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140704 s, 74.5 MB/s 2024-09-13T17:18:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.426 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163547 s, 64.1 MB/s 2024-09-13T17:18:34.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110331 s, 95.0 MB/s 2024-09-13T17:18:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139902 s, 75.0 MB/s 2024-09-13T17:18:34.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135519 s, 77.4 MB/s 2024-09-13T17:18:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156564 s, 67.0 MB/s 2024-09-13T17:18:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.505 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013465 s, 77.9 MB/s 2024-09-13T17:18:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.521 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142112 s, 73.8 MB/s 2024-09-13T17:18:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133675 s, 78.4 MB/s 2024-09-13T17:18:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.556 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168206 s, 62.3 MB/s 2024-09-13T17:18:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171787 s, 61.0 MB/s 2024-09-13T17:18:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013287 s, 78.9 MB/s 2024-09-13T17:18:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141359 s, 74.2 MB/s 2024-09-13T17:18:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.627 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167732 s, 62.5 MB/s 2024-09-13T17:18:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145096 s, 72.3 MB/s 2024-09-13T17:18:34.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155655 s, 67.4 MB/s 2024-09-13T17:18:34.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168715 s, 62.2 MB/s 2024-09-13T17:18:34.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132254 s, 79.3 MB/s 2024-09-13T17:18:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.715 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166136 s, 63.1 MB/s 2024-09-13T17:18:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137343 s, 76.3 MB/s 2024-09-13T17:18:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.747 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135003 s, 77.7 MB/s 2024-09-13T17:18:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151694 s, 69.1 MB/s 2024-09-13T17:18:34.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.780 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128764 s, 81.4 MB/s 2024-09-13T17:18:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136828 s, 76.6 MB/s 2024-09-13T17:18:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125766 s, 83.4 MB/s 2024-09-13T17:18:34.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.3 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:34.812 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016287 s, 64.4 MB/s 2024-09-13T17:18:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115085 s, 91.1 MB/s 2024-09-13T17:18:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133986 s, 78.3 MB/s 2024-09-13T17:18:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.914 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129394 s, 81.0 MB/s 2024-09-13T17:18:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.929 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130902 s, 80.1 MB/s 2024-09-13T17:18:34.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.948 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160155 s, 65.5 MB/s 2024-09-13T17:18:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.964 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136435 s, 76.9 MB/s 2024-09-13T17:18:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013542 s, 77.4 MB/s 2024-09-13T17:18:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:34.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015643 s, 67.0 MB/s 2024-09-13T17:18:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135363 s, 77.5 MB/s 2024-09-13T17:18:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163259 s, 64.2 MB/s 2024-09-13T17:18:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.047 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128302 s, 81.7 MB/s 2024-09-13T17:18:35.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.063 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132469 s, 79.2 MB/s 2024-09-13T17:18:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.077 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130586 s, 80.3 MB/s 2024-09-13T17:18:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129757 s, 80.8 MB/s 2024-09-13T17:18:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185449 s, 56.5 MB/s 2024-09-13T17:18:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.129 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130518 s, 80.3 MB/s 2024-09-13T17:18:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012878 s, 81.4 MB/s 2024-09-13T17:18:35.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129731 s, 80.8 MB/s 2024-09-13T17:18:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137695 s, 76.2 MB/s 2024-09-13T17:18:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.192 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143842 s, 72.9 MB/s 2024-09-13T17:18:35.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.211 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165211 s, 63.5 MB/s 2024-09-13T17:18:35.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.227 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140804 s, 74.5 MB/s 2024-09-13T17:18:35.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.242 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133341 s, 78.6 MB/s 2024-09-13T17:18:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166546 s, 63.0 MB/s 2024-09-13T17:18:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.277 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137978 s, 76.0 MB/s 2024-09-13T17:18:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145353 s, 72.1 MB/s 2024-09-13T17:18:35.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163517 s, 64.1 MB/s 2024-09-13T17:18:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.327 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123501 s, 84.9 MB/s 2024-09-13T17:18:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139232 s, 75.3 MB/s 2024-09-13T17:18:35.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130987 s, 80.1 MB/s 2024-09-13T17:18:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140991 s, 74.4 MB/s 2024-09-13T17:18:35.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:35.376 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.4 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:35.377 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:35.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.432 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130233 s, 80.5 MB/s 2024-09-13T17:18:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129154 s, 81.2 MB/s 2024-09-13T17:18:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127794 s, 82.1 MB/s 2024-09-13T17:18:35.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128304 s, 81.7 MB/s 2024-09-13T17:18:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132798 s, 79.0 MB/s 2024-09-13T17:18:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126691 s, 82.8 MB/s 2024-09-13T17:18:35.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157213 s, 66.7 MB/s 2024-09-13T17:18:35.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161065 s, 65.1 MB/s 2024-09-13T17:18:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.562 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162252 s, 64.6 MB/s 2024-09-13T17:18:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135656 s, 77.3 MB/s 2024-09-13T17:18:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138797 s, 75.5 MB/s 2024-09-13T17:18:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165107 s, 63.5 MB/s 2024-09-13T17:18:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135414 s, 77.4 MB/s 2024-09-13T17:18:35.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141032 s, 74.4 MB/s 2024-09-13T17:18:35.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130433 s, 80.4 MB/s 2024-09-13T17:18:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.674 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127863 s, 82.0 MB/s 2024-09-13T17:18:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130316 s, 80.5 MB/s 2024-09-13T17:18:35.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.705 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128058 s, 81.9 MB/s 2024-09-13T17:18:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.721 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143375 s, 73.1 MB/s 2024-09-13T17:18:35.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.737 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138026 s, 76.0 MB/s 2024-09-13T17:18:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131308 s, 79.9 MB/s 2024-09-13T17:18:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160697 s, 65.3 MB/s 2024-09-13T17:18:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01325 s, 79.1 MB/s 2024-09-13T17:18:35.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160249 s, 65.4 MB/s 2024-09-13T17:18:35.820 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013893 s, 75.5 MB/s 2024-09-13T17:18:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159622 s, 65.7 MB/s 2024-09-13T17:18:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.854 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134095 s, 78.2 MB/s 2024-09-13T17:18:35.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.873 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.873 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167347 s, 62.7 MB/s 2024-09-13T17:18:35.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.889 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136051 s, 77.1 MB/s 2024-09-13T17:18:35.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128512 s, 81.6 MB/s 2024-09-13T17:18:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133103 s, 78.8 MB/s 2024-09-13T17:18:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158 s, 66.4 MB/s 2024-09-13T17:18:35.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:35.939 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:35.939 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:35.939 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.5 2024-09-13T17:18:35.939 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:35.939 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:35.940 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:35.940 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:35.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:35.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165146 s, 63.5 MB/s 2024-09-13T17:18:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.015 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149101 s, 70.3 MB/s 2024-09-13T17:18:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.030 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129948 s, 80.7 MB/s 2024-09-13T17:18:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.045 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130474 s, 80.4 MB/s 2024-09-13T17:18:36.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.062 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151505 s, 69.2 MB/s 2024-09-13T17:18:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.078 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136644 s, 76.7 MB/s 2024-09-13T17:18:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133878 s, 78.3 MB/s 2024-09-13T17:18:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163699 s, 64.1 MB/s 2024-09-13T17:18:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013024 s, 80.5 MB/s 2024-09-13T17:18:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141702 s, 74.0 MB/s 2024-09-13T17:18:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126549 s, 82.9 MB/s 2024-09-13T17:18:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132015 s, 79.4 MB/s 2024-09-13T17:18:36.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167516 s, 62.6 MB/s 2024-09-13T17:18:36.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132817 s, 78.9 MB/s 2024-09-13T17:18:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013173 s, 79.6 MB/s 2024-09-13T17:18:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123509 s, 84.9 MB/s 2024-09-13T17:18:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128663 s, 81.5 MB/s 2024-09-13T17:18:36.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.270 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013463 s, 77.9 MB/s 2024-09-13T17:18:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140176 s, 74.8 MB/s 2024-09-13T17:18:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139268 s, 75.3 MB/s 2024-09-13T17:18:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.319 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149618 s, 70.1 MB/s 2024-09-13T17:18:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140493 s, 74.6 MB/s 2024-09-13T17:18:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166319 s, 63.0 MB/s 2024-09-13T17:18:36.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.372 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155571 s, 67.4 MB/s 2024-09-13T17:18:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165745 s, 63.3 MB/s 2024-09-13T17:18:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163019 s, 64.3 MB/s 2024-09-13T17:18:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.424 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125308 s, 83.7 MB/s 2024-09-13T17:18:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137794 s, 76.1 MB/s 2024-09-13T17:18:36.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.460 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170139 s, 61.6 MB/s 2024-09-13T17:18:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.475 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131944 s, 79.5 MB/s 2024-09-13T17:18:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135214 s, 77.5 MB/s 2024-09-13T17:18:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.506 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131038 s, 80.0 MB/s 2024-09-13T17:18:36.507 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.6 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:36.508 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:36.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130359 s, 80.4 MB/s 2024-09-13T17:18:36.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132794 s, 79.0 MB/s 2024-09-13T17:18:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133647 s, 78.5 MB/s 2024-09-13T17:18:36.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.610 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139058 s, 75.4 MB/s 2024-09-13T17:18:36.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.625 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131626 s, 79.7 MB/s 2024-09-13T17:18:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133718 s, 78.4 MB/s 2024-09-13T17:18:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133054 s, 78.8 MB/s 2024-09-13T17:18:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133267 s, 78.7 MB/s 2024-09-13T17:18:36.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.687 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132581 s, 79.1 MB/s 2024-09-13T17:18:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.704 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139687 s, 75.1 MB/s 2024-09-13T17:18:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134318 s, 78.1 MB/s 2024-09-13T17:18:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.738 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016 s, 65.5 MB/s 2024-09-13T17:18:36.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166109 s, 63.1 MB/s 2024-09-13T17:18:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.774 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151541 s, 69.2 MB/s 2024-09-13T17:18:36.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.790 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134192 s, 78.1 MB/s 2024-09-13T17:18:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135627 s, 77.3 MB/s 2024-09-13T17:18:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131099 s, 80.0 MB/s 2024-09-13T17:18:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.842 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193258 s, 54.3 MB/s 2024-09-13T17:18:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.858 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134642 s, 77.9 MB/s 2024-09-13T17:18:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171049 s, 61.3 MB/s 2024-09-13T17:18:36.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139155 s, 75.4 MB/s 2024-09-13T17:18:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134324 s, 78.1 MB/s 2024-09-13T17:18:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132456 s, 79.2 MB/s 2024-09-13T17:18:36.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166367 s, 63.0 MB/s 2024-09-13T17:18:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131745 s, 79.6 MB/s 2024-09-13T17:18:36.977 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.978 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016577 s, 63.3 MB/s 2024-09-13T17:18:36.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:36.994 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136169 s, 77.0 MB/s 2024-09-13T17:18:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.009 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131061 s, 80.0 MB/s 2024-09-13T17:18:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130828 s, 80.1 MB/s 2024-09-13T17:18:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162332 s, 64.6 MB/s 2024-09-13T17:18:37.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.058 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128537 s, 81.6 MB/s 2024-09-13T17:18:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128623 s, 81.5 MB/s 2024-09-13T17:18:37.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.7 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:37.075 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:37.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.130 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129452 s, 81.0 MB/s 2024-09-13T17:18:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.145 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012749 s, 82.2 MB/s 2024-09-13T17:18:37.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.160 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130309 s, 80.5 MB/s 2024-09-13T17:18:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158098 s, 66.3 MB/s 2024-09-13T17:18:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137005 s, 76.5 MB/s 2024-09-13T17:18:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135566 s, 77.3 MB/s 2024-09-13T17:18:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168386 s, 62.3 MB/s 2024-09-13T17:18:37.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.244 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130221 s, 80.5 MB/s 2024-09-13T17:18:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.260 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133401 s, 78.6 MB/s 2024-09-13T17:18:37.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133009 s, 78.8 MB/s 2024-09-13T17:18:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170067 s, 61.7 MB/s 2024-09-13T17:18:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.310 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132357 s, 79.2 MB/s 2024-09-13T17:18:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126818 s, 82.7 MB/s 2024-09-13T17:18:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016825 s, 62.3 MB/s 2024-09-13T17:18:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.362 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157032 s, 66.8 MB/s 2024-09-13T17:18:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167019 s, 62.8 MB/s 2024-09-13T17:18:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136471 s, 76.8 MB/s 2024-09-13T17:18:37.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.413 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.413 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141148 s, 74.3 MB/s 2024-09-13T17:18:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163975 s, 63.9 MB/s 2024-09-13T17:18:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129679 s, 80.9 MB/s 2024-09-13T17:18:37.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130411 s, 80.4 MB/s 2024-09-13T17:18:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125155 s, 83.8 MB/s 2024-09-13T17:18:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137061 s, 76.5 MB/s 2024-09-13T17:18:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135314 s, 77.5 MB/s 2024-09-13T17:18:37.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132971 s, 78.9 MB/s 2024-09-13T17:18:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153678 s, 68.2 MB/s 2024-09-13T17:18:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.560 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164092 s, 63.9 MB/s 2024-09-13T17:18:37.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130961 s, 80.1 MB/s 2024-09-13T17:18:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155743 s, 67.3 MB/s 2024-09-13T17:18:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130191 s, 80.5 MB/s 2024-09-13T17:18:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136405 s, 76.9 MB/s 2024-09-13T17:18:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.640 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141326 s, 74.2 MB/s 2024-09-13T17:18:37.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.8 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:37.642 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.698 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144376 s, 72.6 MB/s 2024-09-13T17:18:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166103 s, 63.1 MB/s 2024-09-13T17:18:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01332 s, 78.7 MB/s 2024-09-13T17:18:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129646 s, 80.9 MB/s 2024-09-13T17:18:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.767 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016579 s, 63.2 MB/s 2024-09-13T17:18:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.782 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130013 s, 80.7 MB/s 2024-09-13T17:18:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170855 s, 61.4 MB/s 2024-09-13T17:18:37.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012965 s, 80.9 MB/s 2024-09-13T17:18:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124672 s, 84.1 MB/s 2024-09-13T17:18:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167972 s, 62.4 MB/s 2024-09-13T17:18:37.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.866 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132775 s, 79.0 MB/s 2024-09-13T17:18:37.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133952 s, 78.3 MB/s 2024-09-13T17:18:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132427 s, 79.2 MB/s 2024-09-13T17:18:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134144 s, 78.2 MB/s 2024-09-13T17:18:37.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013133 s, 79.8 MB/s 2024-09-13T17:18:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.943 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012532 s, 83.7 MB/s 2024-09-13T17:18:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132958 s, 78.9 MB/s 2024-09-13T17:18:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137009 s, 76.5 MB/s 2024-09-13T17:18:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:37.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130783 s, 80.2 MB/s 2024-09-13T17:18:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127388 s, 82.3 MB/s 2024-09-13T17:18:38.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122336 s, 85.7 MB/s 2024-09-13T17:18:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139148 s, 75.4 MB/s 2024-09-13T17:18:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124999 s, 83.9 MB/s 2024-09-13T17:18:38.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141545 s, 74.1 MB/s 2024-09-13T17:18:38.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.087 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170406 s, 61.5 MB/s 2024-09-13T17:18:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123175 s, 85.1 MB/s 2024-09-13T17:18:38.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172938 s, 60.6 MB/s 2024-09-13T17:18:38.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135186 s, 77.6 MB/s 2024-09-13T17:18:38.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117579 s, 89.2 MB/s 2024-09-13T17:18:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135089 s, 77.6 MB/s 2024-09-13T17:18:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132229 s, 79.3 MB/s 2024-09-13T17:18:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.196 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116153 s, 90.3 MB/s 2024-09-13T17:18:38.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:38.197 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:38.197 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:38.198 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.9 2024-09-13T17:18:38.198 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:38.198 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:38.198 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:38.198 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129559 s, 80.9 MB/s 2024-09-13T17:18:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.268 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138312 s, 75.8 MB/s 2024-09-13T17:18:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.283 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131939 s, 79.5 MB/s 2024-09-13T17:18:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.299 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132956 s, 78.9 MB/s 2024-09-13T17:18:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135875 s, 77.2 MB/s 2024-09-13T17:18:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.332 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144183 s, 72.7 MB/s 2024-09-13T17:18:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140125 s, 74.8 MB/s 2024-09-13T17:18:38.362 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.363 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128142 s, 81.8 MB/s 2024-09-13T17:18:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133938 s, 78.3 MB/s 2024-09-13T17:18:38.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129042 s, 81.3 MB/s 2024-09-13T17:18:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.408 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118556 s, 88.4 MB/s 2024-09-13T17:18:38.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131801 s, 79.6 MB/s 2024-09-13T17:18:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140211 s, 74.8 MB/s 2024-09-13T17:18:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133545 s, 78.5 MB/s 2024-09-13T17:18:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135267 s, 77.5 MB/s 2024-09-13T17:18:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.487 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137535 s, 76.2 MB/s 2024-09-13T17:18:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.503 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141933 s, 73.9 MB/s 2024-09-13T17:18:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133375 s, 78.6 MB/s 2024-09-13T17:18:38.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131468 s, 79.8 MB/s 2024-09-13T17:18:38.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139883 s, 75.0 MB/s 2024-09-13T17:18:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.566 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130105 s, 80.6 MB/s 2024-09-13T17:18:38.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157455 s, 66.6 MB/s 2024-09-13T17:18:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142355 s, 73.7 MB/s 2024-09-13T17:18:38.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.615 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125994 s, 83.2 MB/s 2024-09-13T17:18:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.631 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131418 s, 79.8 MB/s 2024-09-13T17:18:38.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134875 s, 77.7 MB/s 2024-09-13T17:18:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128548 s, 81.6 MB/s 2024-09-13T17:18:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131152 s, 80.0 MB/s 2024-09-13T17:18:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136048 s, 77.1 MB/s 2024-09-13T17:18:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131375 s, 79.8 MB/s 2024-09-13T17:18:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140656 s, 74.5 MB/s 2024-09-13T17:18:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.740 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132848 s, 78.9 MB/s 2024-09-13T17:18:38.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.10 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:38.742 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134946 s, 77.7 MB/s 2024-09-13T17:18:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143232 s, 73.2 MB/s 2024-09-13T17:18:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143072 s, 73.3 MB/s 2024-09-13T17:18:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138344 s, 75.8 MB/s 2024-09-13T17:18:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132543 s, 79.1 MB/s 2024-09-13T17:18:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135374 s, 77.5 MB/s 2024-09-13T17:18:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158552 s, 66.1 MB/s 2024-09-13T17:18:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129818 s, 80.8 MB/s 2024-09-13T17:18:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137987 s, 76.0 MB/s 2024-09-13T17:18:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133751 s, 78.4 MB/s 2024-09-13T17:18:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.961 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154153 s, 68.0 MB/s 2024-09-13T17:18:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.977 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140081 s, 74.9 MB/s 2024-09-13T17:18:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176655 s, 59.4 MB/s 2024-09-13T17:18:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.015 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155479 s, 67.4 MB/s 2024-09-13T17:18:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.030 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128644 s, 81.5 MB/s 2024-09-13T17:18:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.047 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140666 s, 74.5 MB/s 2024-09-13T17:18:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160253 s, 65.4 MB/s 2024-09-13T17:18:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173624 s, 60.4 MB/s 2024-09-13T17:18:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134942 s, 77.7 MB/s 2024-09-13T17:18:39.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.119 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015917 s, 65.9 MB/s 2024-09-13T17:18:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.134 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135405 s, 77.4 MB/s 2024-09-13T17:18:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146213 s, 71.7 MB/s 2024-09-13T17:18:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.166 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130968 s, 80.1 MB/s 2024-09-13T17:18:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135001 s, 77.7 MB/s 2024-09-13T17:18:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.198 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134811 s, 77.8 MB/s 2024-09-13T17:18:39.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168306 s, 62.3 MB/s 2024-09-13T17:18:39.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.234 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146216 s, 71.7 MB/s 2024-09-13T17:18:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.251 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142685 s, 73.5 MB/s 2024-09-13T17:18:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142194 s, 73.7 MB/s 2024-09-13T17:18:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149512 s, 70.1 MB/s 2024-09-13T17:18:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152949 s, 68.6 MB/s 2024-09-13T17:18:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141017 s, 74.4 MB/s 2024-09-13T17:18:39.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.11 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:39.320 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:39.321 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151263 s, 69.3 MB/s 2024-09-13T17:18:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148385 s, 70.7 MB/s 2024-09-13T17:18:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134341 s, 78.1 MB/s 2024-09-13T17:18:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135993 s, 77.1 MB/s 2024-09-13T17:18:39.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129048 s, 81.3 MB/s 2024-09-13T17:18:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.458 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145642 s, 72.0 MB/s 2024-09-13T17:18:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.474 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138456 s, 75.7 MB/s 2024-09-13T17:18:39.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165445 s, 63.4 MB/s 2024-09-13T17:18:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133662 s, 78.4 MB/s 2024-09-13T17:18:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173179 s, 60.5 MB/s 2024-09-13T17:18:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134116 s, 78.2 MB/s 2024-09-13T17:18:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159315 s, 65.8 MB/s 2024-09-13T17:18:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138681 s, 75.6 MB/s 2024-09-13T17:18:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129441 s, 81.0 MB/s 2024-09-13T17:18:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167047 s, 62.8 MB/s 2024-09-13T17:18:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138483 s, 75.7 MB/s 2024-09-13T17:18:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173727 s, 60.4 MB/s 2024-09-13T17:18:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02262 s, 46.4 MB/s 2024-09-13T17:18:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018586 s, 56.4 MB/s 2024-09-13T17:18:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197659 s, 53.0 MB/s 2024-09-13T17:18:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141481 s, 74.1 MB/s 2024-09-13T17:18:39.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160718 s, 65.2 MB/s 2024-09-13T17:18:39.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136861 s, 76.6 MB/s 2024-09-13T17:18:39.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.784 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012892 s, 81.3 MB/s 2024-09-13T17:18:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166311 s, 63.0 MB/s 2024-09-13T17:18:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133894 s, 78.3 MB/s 2024-09-13T17:18:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128828 s, 81.4 MB/s 2024-09-13T17:18:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.851 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155789 s, 67.3 MB/s 2024-09-13T17:18:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134883 s, 77.7 MB/s 2024-09-13T17:18:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132216 s, 79.3 MB/s 2024-09-13T17:18:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.898 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133349 s, 78.6 MB/s 2024-09-13T17:18:39.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132839 s, 78.9 MB/s 2024-09-13T17:18:39.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.12 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:39.915 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:39.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163428 s, 64.2 MB/s 2024-09-13T17:18:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130795 s, 80.2 MB/s 2024-09-13T17:18:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160981 s, 65.1 MB/s 2024-09-13T17:18:40.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.025 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015742 s, 66.6 MB/s 2024-09-13T17:18:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134899 s, 77.7 MB/s 2024-09-13T17:18:40.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128829 s, 81.4 MB/s 2024-09-13T17:18:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.071 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137196 s, 76.4 MB/s 2024-09-13T17:18:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.087 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132599 s, 79.1 MB/s 2024-09-13T17:18:40.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.102 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.102 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126919 s, 82.6 MB/s 2024-09-13T17:18:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162836 s, 64.4 MB/s 2024-09-13T17:18:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133895 s, 78.3 MB/s 2024-09-13T17:18:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.153 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135455 s, 77.4 MB/s 2024-09-13T17:18:40.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.169 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013708 s, 76.5 MB/s 2024-09-13T17:18:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.188 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164117 s, 63.9 MB/s 2024-09-13T17:18:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125061 s, 83.8 MB/s 2024-09-13T17:18:40.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132192 s, 79.3 MB/s 2024-09-13T17:18:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.234 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134881 s, 77.7 MB/s 2024-09-13T17:18:40.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.249 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129112 s, 81.2 MB/s 2024-09-13T17:18:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.264 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132414 s, 79.2 MB/s 2024-09-13T17:18:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137785 s, 76.1 MB/s 2024-09-13T17:18:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136905 s, 76.6 MB/s 2024-09-13T17:18:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131459 s, 79.8 MB/s 2024-09-13T17:18:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.327 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132997 s, 78.8 MB/s 2024-09-13T17:18:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135573 s, 77.3 MB/s 2024-09-13T17:18:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013869 s, 75.6 MB/s 2024-09-13T17:18:40.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137945 s, 76.0 MB/s 2024-09-13T17:18:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135418 s, 77.4 MB/s 2024-09-13T17:18:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139058 s, 75.4 MB/s 2024-09-13T17:18:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.427 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176626 s, 59.4 MB/s 2024-09-13T17:18:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.442 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128259 s, 81.8 MB/s 2024-09-13T17:18:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012792 s, 82.0 MB/s 2024-09-13T17:18:40.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.473 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131257 s, 79.9 MB/s 2024-09-13T17:18:40.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:40.474 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.13 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:40.475 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159536 s, 65.7 MB/s 2024-09-13T17:18:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132072 s, 79.4 MB/s 2024-09-13T17:18:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143965 s, 72.8 MB/s 2024-09-13T17:18:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.580 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135708 s, 77.3 MB/s 2024-09-13T17:18:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.595 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133586 s, 78.5 MB/s 2024-09-13T17:18:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.613 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161658 s, 64.9 MB/s 2024-09-13T17:18:40.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.630 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013862 s, 75.6 MB/s 2024-09-13T17:18:40.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136695 s, 76.7 MB/s 2024-09-13T17:18:40.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012845 s, 81.6 MB/s 2024-09-13T17:18:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013401 s, 78.2 MB/s 2024-09-13T17:18:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137739 s, 76.1 MB/s 2024-09-13T17:18:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165477 s, 63.4 MB/s 2024-09-13T17:18:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.729 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155286 s, 67.5 MB/s 2024-09-13T17:18:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183753 s, 57.1 MB/s 2024-09-13T17:18:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130872 s, 80.1 MB/s 2024-09-13T17:18:40.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.780 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128272 s, 81.7 MB/s 2024-09-13T17:18:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138855 s, 75.5 MB/s 2024-09-13T17:18:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144605 s, 72.5 MB/s 2024-09-13T17:18:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141474 s, 74.1 MB/s 2024-09-13T17:18:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.845 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139356 s, 75.2 MB/s 2024-09-13T17:18:40.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:18:40.856 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:18:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131114 s, 80.0 MB/s 2024-09-13T17:18:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130331 s, 80.5 MB/s 2024-09-13T17:18:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.892 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142834 s, 73.4 MB/s 2024-09-13T17:18:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168052 s, 62.4 MB/s 2024-09-13T17:18:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128907 s, 81.3 MB/s 2024-09-13T17:18:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.942 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138199 s, 75.9 MB/s 2024-09-13T17:18:40.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014603 s, 71.8 MB/s 2024-09-13T17:18:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136127 s, 77.0 MB/s 2024-09-13T17:18:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:40.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128069 s, 81.9 MB/s 2024-09-13T17:18:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.009 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164627 s, 63.7 MB/s 2024-09-13T17:18:41.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178652 s, 58.7 MB/s 2024-09-13T17:18:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.044 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130924 s, 80.1 MB/s 2024-09-13T17:18:41.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.14 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:41.046 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.145 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174647 s, 60.0 MB/s 2024-09-13T17:18:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160135 s, 65.5 MB/s 2024-09-13T17:18:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.182 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161809 s, 64.8 MB/s 2024-09-13T17:18:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01731 s, 60.6 MB/s 2024-09-13T17:18:41.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.217 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134976 s, 77.7 MB/s 2024-09-13T17:18:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127781 s, 82.1 MB/s 2024-09-13T17:18:41.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136458 s, 76.8 MB/s 2024-09-13T17:18:41.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.264 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133276 s, 78.7 MB/s 2024-09-13T17:18:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140752 s, 74.5 MB/s 2024-09-13T17:18:41.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144617 s, 72.5 MB/s 2024-09-13T17:18:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133258 s, 78.7 MB/s 2024-09-13T17:18:41.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132424 s, 79.2 MB/s 2024-09-13T17:18:41.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135768 s, 77.2 MB/s 2024-09-13T17:18:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.360 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138323 s, 75.8 MB/s 2024-09-13T17:18:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.376 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141655 s, 74.0 MB/s 2024-09-13T17:18:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157535 s, 66.6 MB/s 2024-09-13T17:18:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132093 s, 79.4 MB/s 2024-09-13T17:18:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.426 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145351 s, 72.1 MB/s 2024-09-13T17:18:41.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125566 s, 83.5 MB/s 2024-09-13T17:18:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013704 s, 76.5 MB/s 2024-09-13T17:18:41.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130286 s, 80.5 MB/s 2024-09-13T17:18:41.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168261 s, 62.3 MB/s 2024-09-13T17:18:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148002 s, 70.8 MB/s 2024-09-13T17:18:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137936 s, 76.0 MB/s 2024-09-13T17:18:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127855 s, 82.0 MB/s 2024-09-13T17:18:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131962 s, 79.5 MB/s 2024-09-13T17:18:41.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165468 s, 63.4 MB/s 2024-09-13T17:18:41.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159296 s, 65.8 MB/s 2024-09-13T17:18:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137518 s, 76.3 MB/s 2024-09-13T17:18:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.625 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135985 s, 77.1 MB/s 2024-09-13T17:18:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168224 s, 62.3 MB/s 2024-09-13T17:18:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.658 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126524 s, 82.9 MB/s 2024-09-13T17:18:41.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:41.660 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:41.660 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:41.660 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.15 2024-09-13T17:18:41.660 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:41.661 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:41.661 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:41.661 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133519 s, 78.5 MB/s 2024-09-13T17:18:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127699 s, 82.1 MB/s 2024-09-13T17:18:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158739 s, 66.1 MB/s 2024-09-13T17:18:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114508 s, 91.6 MB/s 2024-09-13T17:18:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132928 s, 78.9 MB/s 2024-09-13T17:18:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130114 s, 80.6 MB/s 2024-09-13T17:18:41.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.808 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131409 s, 79.8 MB/s 2024-09-13T17:18:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125429 s, 83.6 MB/s 2024-09-13T17:18:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162781 s, 64.4 MB/s 2024-09-13T17:18:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.856 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128326 s, 81.7 MB/s 2024-09-13T17:18:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016431 s, 63.8 MB/s 2024-09-13T17:18:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139607 s, 75.1 MB/s 2024-09-13T17:18:41.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.907 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132865 s, 78.9 MB/s 2024-09-13T17:18:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172155 s, 60.9 MB/s 2024-09-13T17:18:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131965 s, 79.5 MB/s 2024-09-13T17:18:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131159 s, 79.9 MB/s 2024-09-13T17:18:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015309 s, 68.5 MB/s 2024-09-13T17:18:41.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:41.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:41.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129572 s, 80.9 MB/s 2024-09-13T17:18:42.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132007 s, 79.4 MB/s 2024-09-13T17:18:42.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.021 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134679 s, 77.9 MB/s 2024-09-13T17:18:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131913 s, 79.5 MB/s 2024-09-13T17:18:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128185 s, 81.8 MB/s 2024-09-13T17:18:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154086 s, 68.1 MB/s 2024-09-13T17:18:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129564 s, 80.9 MB/s 2024-09-13T17:18:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013271 s, 79.0 MB/s 2024-09-13T17:18:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014405 s, 72.8 MB/s 2024-09-13T17:18:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129916 s, 80.7 MB/s 2024-09-13T17:18:42.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.148 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137354 s, 76.3 MB/s 2024-09-13T17:18:42.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.164 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136528 s, 76.8 MB/s 2024-09-13T17:18:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124321 s, 84.3 MB/s 2024-09-13T17:18:42.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160617 s, 65.3 MB/s 2024-09-13T17:18:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128741 s, 81.4 MB/s 2024-09-13T17:18:42.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.16 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:42.213 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:42.214 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.268 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013022 s, 80.5 MB/s 2024-09-13T17:18:42.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136788 s, 76.7 MB/s 2024-09-13T17:18:42.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152337 s, 68.8 MB/s 2024-09-13T17:18:42.316 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.317 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131391 s, 79.8 MB/s 2024-09-13T17:18:42.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143115 s, 73.3 MB/s 2024-09-13T17:18:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135769 s, 77.2 MB/s 2024-09-13T17:18:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127533 s, 82.2 MB/s 2024-09-13T17:18:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129527 s, 81.0 MB/s 2024-09-13T17:18:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.395 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137237 s, 76.4 MB/s 2024-09-13T17:18:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139121 s, 75.4 MB/s 2024-09-13T17:18:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165899 s, 63.2 MB/s 2024-09-13T17:18:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132012 s, 79.4 MB/s 2024-09-13T17:18:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.462 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141154 s, 74.3 MB/s 2024-09-13T17:18:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132749 s, 79.0 MB/s 2024-09-13T17:18:42.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127886 s, 82.0 MB/s 2024-09-13T17:18:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132408 s, 79.2 MB/s 2024-09-13T17:18:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.523 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130114 s, 80.6 MB/s 2024-09-13T17:18:42.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.539 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013502 s, 77.7 MB/s 2024-09-13T17:18:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.554 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126896 s, 82.6 MB/s 2024-09-13T17:18:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.570 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141102 s, 74.3 MB/s 2024-09-13T17:18:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142393 s, 73.6 MB/s 2024-09-13T17:18:42.603 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.603 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.603 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137156 s, 76.5 MB/s 2024-09-13T17:18:42.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131971 s, 79.5 MB/s 2024-09-13T17:18:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.638 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172353 s, 60.8 MB/s 2024-09-13T17:18:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156542 s, 67.0 MB/s 2024-09-13T17:18:42.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129567 s, 80.9 MB/s 2024-09-13T17:18:42.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170744 s, 61.4 MB/s 2024-09-13T17:18:42.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130348 s, 80.4 MB/s 2024-09-13T17:18:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.721 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136816 s, 76.6 MB/s 2024-09-13T17:18:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.737 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135592 s, 77.3 MB/s 2024-09-13T17:18:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134448 s, 78.0 MB/s 2024-09-13T17:18:42.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.768 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131545 s, 79.7 MB/s 2024-09-13T17:18:42.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.17 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:42.770 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125796 s, 83.4 MB/s 2024-09-13T17:18:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131507 s, 79.7 MB/s 2024-09-13T17:18:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126227 s, 83.1 MB/s 2024-09-13T17:18:42.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120938 s, 86.7 MB/s 2024-09-13T17:18:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133909 s, 78.3 MB/s 2024-09-13T17:18:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013572 s, 77.3 MB/s 2024-09-13T17:18:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154689 s, 67.8 MB/s 2024-09-13T17:18:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.937 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163918 s, 64.0 MB/s 2024-09-13T17:18:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.953 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130647 s, 80.3 MB/s 2024-09-13T17:18:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.968 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136544 s, 76.8 MB/s 2024-09-13T17:18:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:42.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131654 s, 79.6 MB/s 2024-09-13T17:18:42.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:42.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.000 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135975 s, 77.1 MB/s 2024-09-13T17:18:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161299 s, 65.0 MB/s 2024-09-13T17:18:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135605 s, 77.3 MB/s 2024-09-13T17:18:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146743 s, 71.5 MB/s 2024-09-13T17:18:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.066 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135331 s, 77.5 MB/s 2024-09-13T17:18:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.082 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134645 s, 77.9 MB/s 2024-09-13T17:18:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163391 s, 64.2 MB/s 2024-09-13T17:18:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013217 s, 79.3 MB/s 2024-09-13T17:18:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164353 s, 63.8 MB/s 2024-09-13T17:18:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133909 s, 78.3 MB/s 2024-09-13T17:18:43.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118177 s, 88.7 MB/s 2024-09-13T17:18:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133894 s, 78.3 MB/s 2024-09-13T17:18:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.198 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015858 s, 66.1 MB/s 2024-09-13T17:18:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.213 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012385 s, 84.7 MB/s 2024-09-13T17:18:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134742 s, 77.8 MB/s 2024-09-13T17:18:43.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145765 s, 71.9 MB/s 2024-09-13T17:18:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135318 s, 77.5 MB/s 2024-09-13T17:18:43.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168702 s, 62.2 MB/s 2024-09-13T17:18:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.299 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161712 s, 64.8 MB/s 2024-09-13T17:18:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013773 s, 76.1 MB/s 2024-09-13T17:18:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.330 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132513 s, 79.1 MB/s 2024-09-13T17:18:43.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.18 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:43.332 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143648 s, 73.0 MB/s 2024-09-13T17:18:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128976 s, 81.3 MB/s 2024-09-13T17:18:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.417 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123629 s, 84.8 MB/s 2024-09-13T17:18:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130081 s, 80.6 MB/s 2024-09-13T17:18:43.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.448 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130198 s, 80.5 MB/s 2024-09-13T17:18:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.465 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144368 s, 72.6 MB/s 2024-09-13T17:18:43.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138637 s, 75.6 MB/s 2024-09-13T17:18:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170848 s, 61.4 MB/s 2024-09-13T17:18:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162695 s, 64.5 MB/s 2024-09-13T17:18:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126037 s, 83.2 MB/s 2024-09-13T17:18:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.553 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171225 s, 61.2 MB/s 2024-09-13T17:18:43.567 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.568 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130331 s, 80.5 MB/s 2024-09-13T17:18:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143959 s, 72.8 MB/s 2024-09-13T17:18:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132372 s, 79.2 MB/s 2024-09-13T17:18:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015913 s, 65.9 MB/s 2024-09-13T17:18:43.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126853 s, 82.7 MB/s 2024-09-13T17:18:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126062 s, 83.2 MB/s 2024-09-13T17:18:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.664 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138249 s, 75.8 MB/s 2024-09-13T17:18:43.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013187 s, 79.5 MB/s 2024-09-13T17:18:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123945 s, 84.6 MB/s 2024-09-13T17:18:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.710 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138525 s, 75.7 MB/s 2024-09-13T17:18:43.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136755 s, 76.7 MB/s 2024-09-13T17:18:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.742 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140684 s, 74.5 MB/s 2024-09-13T17:18:43.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136646 s, 76.7 MB/s 2024-09-13T17:18:43.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127273 s, 82.4 MB/s 2024-09-13T17:18:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132528 s, 79.1 MB/s 2024-09-13T17:18:43.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012929 s, 81.1 MB/s 2024-09-13T17:18:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.819 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138635 s, 75.6 MB/s 2024-09-13T17:18:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167616 s, 62.6 MB/s 2024-09-13T17:18:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142895 s, 73.4 MB/s 2024-09-13T17:18:43.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.870 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132218 s, 79.3 MB/s 2024-09-13T17:18:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135048 s, 77.6 MB/s 2024-09-13T17:18:43.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:43.887 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.19 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:43.888 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.942 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132779 s, 79.0 MB/s 2024-09-13T17:18:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135875 s, 77.2 MB/s 2024-09-13T17:18:43.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136073 s, 77.1 MB/s 2024-09-13T17:18:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:43.993 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168944 s, 62.1 MB/s 2024-09-13T17:18:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168214 s, 62.3 MB/s 2024-09-13T17:18:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138774 s, 75.6 MB/s 2024-09-13T17:18:44.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.044 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136538 s, 76.8 MB/s 2024-09-13T17:18:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147447 s, 71.1 MB/s 2024-09-13T17:18:44.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.078 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148872 s, 70.4 MB/s 2024-09-13T17:18:44.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138439 s, 75.7 MB/s 2024-09-13T17:18:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.110 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137977 s, 76.0 MB/s 2024-09-13T17:18:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.126 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132512 s, 79.1 MB/s 2024-09-13T17:18:44.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.142 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138243 s, 75.8 MB/s 2024-09-13T17:18:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.161 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167409 s, 62.6 MB/s 2024-09-13T17:18:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160544 s, 65.3 MB/s 2024-09-13T17:18:44.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120671 s, 86.9 MB/s 2024-09-13T17:18:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013239 s, 79.2 MB/s 2024-09-13T17:18:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127131 s, 82.5 MB/s 2024-09-13T17:18:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127914 s, 82.0 MB/s 2024-09-13T17:18:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140113 s, 74.8 MB/s 2024-09-13T17:18:44.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.273 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015282 s, 68.6 MB/s 2024-09-13T17:18:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136212 s, 77.0 MB/s 2024-09-13T17:18:44.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.304 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133635 s, 78.5 MB/s 2024-09-13T17:18:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137966 s, 76.0 MB/s 2024-09-13T17:18:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.336 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141939 s, 73.9 MB/s 2024-09-13T17:18:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.353 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139951 s, 74.9 MB/s 2024-09-13T17:18:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.370 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150386 s, 69.7 MB/s 2024-09-13T17:18:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136129 s, 77.0 MB/s 2024-09-13T17:18:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159504 s, 65.7 MB/s 2024-09-13T17:18:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013055 s, 80.3 MB/s 2024-09-13T17:18:44.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.435 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134204 s, 78.1 MB/s 2024-09-13T17:18:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142111 s, 73.8 MB/s 2024-09-13T17:18:44.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.20 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:44.453 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:44.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.509 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130399 s, 80.4 MB/s 2024-09-13T17:18:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.523 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129087 s, 81.2 MB/s 2024-09-13T17:18:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159627 s, 65.7 MB/s 2024-09-13T17:18:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.557 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138728 s, 75.6 MB/s 2024-09-13T17:18:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.573 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138724 s, 75.6 MB/s 2024-09-13T17:18:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161007 s, 65.1 MB/s 2024-09-13T17:18:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013416 s, 78.2 MB/s 2024-09-13T17:18:44.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.623 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129307 s, 81.1 MB/s 2024-09-13T17:18:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187762 s, 55.8 MB/s 2024-09-13T17:18:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147298 s, 71.2 MB/s 2024-09-13T17:18:44.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126358 s, 83.0 MB/s 2024-09-13T17:18:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163807 s, 64.0 MB/s 2024-09-13T17:18:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139307 s, 75.3 MB/s 2024-09-13T17:18:44.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127159 s, 82.5 MB/s 2024-09-13T17:18:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133702 s, 78.4 MB/s 2024-09-13T17:18:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117958 s, 88.9 MB/s 2024-09-13T17:18:44.775 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.775 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.775 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167911 s, 62.4 MB/s 2024-09-13T17:18:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165219 s, 63.5 MB/s 2024-09-13T17:18:44.812 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162904 s, 64.4 MB/s 2024-09-13T17:18:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137623 s, 76.2 MB/s 2024-09-13T17:18:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134852 s, 77.8 MB/s 2024-09-13T17:18:44.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013395 s, 78.3 MB/s 2024-09-13T17:18:44.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136773 s, 76.7 MB/s 2024-09-13T17:18:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133064 s, 78.8 MB/s 2024-09-13T17:18:44.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.908 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138678 s, 75.6 MB/s 2024-09-13T17:18:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128717 s, 81.5 MB/s 2024-09-13T17:18:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137284 s, 76.4 MB/s 2024-09-13T17:18:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131242 s, 79.9 MB/s 2024-09-13T17:18:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131039 s, 80.0 MB/s 2024-09-13T17:18:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:44.987 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159742 s, 65.6 MB/s 2024-09-13T17:18:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137193 s, 76.4 MB/s 2024-09-13T17:18:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01272 s, 82.4 MB/s 2024-09-13T17:18:45.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.21 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:45.020 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.076 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132043 s, 79.4 MB/s 2024-09-13T17:18:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145436 s, 72.1 MB/s 2024-09-13T17:18:45.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.108 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131218 s, 79.9 MB/s 2024-09-13T17:18:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166444 s, 63.0 MB/s 2024-09-13T17:18:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140711 s, 74.5 MB/s 2024-09-13T17:18:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.160 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140358 s, 74.7 MB/s 2024-09-13T17:18:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130107 s, 80.6 MB/s 2024-09-13T17:18:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161826 s, 64.8 MB/s 2024-09-13T17:18:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172849 s, 60.7 MB/s 2024-09-13T17:18:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137251 s, 76.4 MB/s 2024-09-13T17:18:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134689 s, 77.9 MB/s 2024-09-13T17:18:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139554 s, 75.1 MB/s 2024-09-13T17:18:45.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.276 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130153 s, 80.6 MB/s 2024-09-13T17:18:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.291 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129572 s, 80.9 MB/s 2024-09-13T17:18:45.307 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.307 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.307 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135472 s, 77.4 MB/s 2024-09-13T17:18:45.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.323 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139382 s, 75.2 MB/s 2024-09-13T17:18:45.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.339 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132857 s, 78.9 MB/s 2024-09-13T17:18:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169467 s, 61.9 MB/s 2024-09-13T17:18:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133861 s, 78.3 MB/s 2024-09-13T17:18:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167117 s, 62.7 MB/s 2024-09-13T17:18:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128029 s, 81.9 MB/s 2024-09-13T17:18:45.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.427 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170345 s, 61.6 MB/s 2024-09-13T17:18:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170962 s, 61.3 MB/s 2024-09-13T17:18:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.465 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016771 s, 62.5 MB/s 2024-09-13T17:18:45.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131417 s, 79.8 MB/s 2024-09-13T17:18:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144404 s, 72.6 MB/s 2024-09-13T17:18:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134666 s, 77.9 MB/s 2024-09-13T17:18:45.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.529 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134188 s, 78.1 MB/s 2024-09-13T17:18:45.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.548 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169211 s, 62.0 MB/s 2024-09-13T17:18:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130617 s, 80.3 MB/s 2024-09-13T17:18:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.583 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176462 s, 59.4 MB/s 2024-09-13T17:18:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130503 s, 80.3 MB/s 2024-09-13T17:18:45.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.22 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:45.600 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:45.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130148 s, 80.6 MB/s 2024-09-13T17:18:45.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016001 s, 65.5 MB/s 2024-09-13T17:18:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161619 s, 64.9 MB/s 2024-09-13T17:18:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162722 s, 64.4 MB/s 2024-09-13T17:18:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.725 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136834 s, 76.6 MB/s 2024-09-13T17:18:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139549 s, 75.1 MB/s 2024-09-13T17:18:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.757 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136306 s, 76.9 MB/s 2024-09-13T17:18:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.772 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129236 s, 81.1 MB/s 2024-09-13T17:18:45.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.791 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167552 s, 62.6 MB/s 2024-09-13T17:18:45.807 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.807 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.807 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138405 s, 75.8 MB/s 2024-09-13T17:18:45.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.822 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126293 s, 83.0 MB/s 2024-09-13T17:18:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170317 s, 61.6 MB/s 2024-09-13T17:18:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.856 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012766 s, 82.1 MB/s 2024-09-13T17:18:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.872 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013566 s, 77.3 MB/s 2024-09-13T17:18:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.888 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136496 s, 76.8 MB/s 2024-09-13T17:18:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.907 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163594 s, 64.1 MB/s 2024-09-13T17:18:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128809 s, 81.4 MB/s 2024-09-13T17:18:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167224 s, 62.7 MB/s 2024-09-13T17:18:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130902 s, 80.1 MB/s 2024-09-13T17:18:45.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138804 s, 75.5 MB/s 2024-09-13T17:18:45.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:45.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:45.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161044 s, 65.1 MB/s 2024-09-13T17:18:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139287 s, 75.3 MB/s 2024-09-13T17:18:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110171 s, 95.2 MB/s 2024-09-13T17:18:46.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.035 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130334 s, 80.5 MB/s 2024-09-13T17:18:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128473 s, 81.6 MB/s 2024-09-13T17:18:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160399 s, 65.4 MB/s 2024-09-13T17:18:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135642 s, 77.3 MB/s 2024-09-13T17:18:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133996 s, 78.3 MB/s 2024-09-13T17:18:46.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.119 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172679 s, 60.7 MB/s 2024-09-13T17:18:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013212 s, 79.4 MB/s 2024-09-13T17:18:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141569 s, 74.1 MB/s 2024-09-13T17:18:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138698 s, 75.6 MB/s 2024-09-13T17:18:46.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.23 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:46.169 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125091 s, 83.8 MB/s 2024-09-13T17:18:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133952 s, 78.3 MB/s 2024-09-13T17:18:46.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133888 s, 78.3 MB/s 2024-09-13T17:18:46.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.273 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162835 s, 64.4 MB/s 2024-09-13T17:18:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138783 s, 75.6 MB/s 2024-09-13T17:18:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.308 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164315 s, 63.8 MB/s 2024-09-13T17:18:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142329 s, 73.7 MB/s 2024-09-13T17:18:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133819 s, 78.4 MB/s 2024-09-13T17:18:46.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.356 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013353 s, 78.5 MB/s 2024-09-13T17:18:46.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131715 s, 79.6 MB/s 2024-09-13T17:18:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145322 s, 72.2 MB/s 2024-09-13T17:18:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129681 s, 80.9 MB/s 2024-09-13T17:18:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.418 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133756 s, 78.4 MB/s 2024-09-13T17:18:46.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.436 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154075 s, 68.1 MB/s 2024-09-13T17:18:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137934 s, 76.0 MB/s 2024-09-13T17:18:46.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.468 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135469 s, 77.4 MB/s 2024-09-13T17:18:46.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134723 s, 77.8 MB/s 2024-09-13T17:18:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.499 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129979 s, 80.7 MB/s 2024-09-13T17:18:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154579 s, 67.8 MB/s 2024-09-13T17:18:46.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013636 s, 76.9 MB/s 2024-09-13T17:18:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142253 s, 73.7 MB/s 2024-09-13T17:18:46.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.564 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.564 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132144 s, 79.4 MB/s 2024-09-13T17:18:46.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.583 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016804 s, 62.4 MB/s 2024-09-13T17:18:46.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159384 s, 65.8 MB/s 2024-09-13T17:18:46.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126763 s, 82.7 MB/s 2024-09-13T17:18:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135273 s, 77.5 MB/s 2024-09-13T17:18:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134361 s, 78.0 MB/s 2024-09-13T17:18:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.663 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130289 s, 80.5 MB/s 2024-09-13T17:18:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134582 s, 77.9 MB/s 2024-09-13T17:18:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.694 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130701 s, 80.2 MB/s 2024-09-13T17:18:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139514 s, 75.2 MB/s 2024-09-13T17:18:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139336 s, 75.3 MB/s 2024-09-13T17:18:46.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:46.728 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:46.728 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:46.728 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.24 2024-09-13T17:18:46.728 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:46.728 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:46.729 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:46.729 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160146 s, 65.5 MB/s 2024-09-13T17:18:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.802 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135697 s, 77.3 MB/s 2024-09-13T17:18:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125294 s, 83.7 MB/s 2024-09-13T17:18:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131877 s, 79.5 MB/s 2024-09-13T17:18:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131287 s, 79.9 MB/s 2024-09-13T17:18:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143431 s, 73.1 MB/s 2024-09-13T17:18:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146862 s, 71.4 MB/s 2024-09-13T17:18:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140771 s, 74.5 MB/s 2024-09-13T17:18:46.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167004 s, 62.8 MB/s 2024-09-13T17:18:46.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131476 s, 79.8 MB/s 2024-09-13T17:18:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165138 s, 63.5 MB/s 2024-09-13T17:18:46.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128225 s, 81.8 MB/s 2024-09-13T17:18:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165369 s, 63.4 MB/s 2024-09-13T17:18:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:46.999 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130882 s, 80.1 MB/s 2024-09-13T17:18:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165975 s, 63.2 MB/s 2024-09-13T17:18:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.033 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127171 s, 82.5 MB/s 2024-09-13T17:18:47.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166616 s, 62.9 MB/s 2024-09-13T17:18:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133935 s, 78.3 MB/s 2024-09-13T17:18:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162344 s, 64.6 MB/s 2024-09-13T17:18:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013353 s, 78.5 MB/s 2024-09-13T17:18:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.117 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130991 s, 80.0 MB/s 2024-09-13T17:18:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163983 s, 63.9 MB/s 2024-09-13T17:18:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165986 s, 63.2 MB/s 2024-09-13T17:18:47.169 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132695 s, 79.0 MB/s 2024-09-13T17:18:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.185 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013301 s, 78.8 MB/s 2024-09-13T17:18:47.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135113 s, 77.6 MB/s 2024-09-13T17:18:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174872 s, 60.0 MB/s 2024-09-13T17:18:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013838 s, 75.8 MB/s 2024-09-13T17:18:47.251 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012921 s, 81.2 MB/s 2024-09-13T17:18:47.270 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163234 s, 64.2 MB/s 2024-09-13T17:18:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.288 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152961 s, 68.6 MB/s 2024-09-13T17:18:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135622 s, 77.3 MB/s 2024-09-13T17:18:47.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:47.305 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:47.305 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:47.305 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.25 2024-09-13T17:18:47.305 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:47.306 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:47.306 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:47.306 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.361 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138165 s, 75.9 MB/s 2024-09-13T17:18:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.377 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141181 s, 74.3 MB/s 2024-09-13T17:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140609 s, 74.6 MB/s 2024-09-13T17:18:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137529 s, 76.2 MB/s 2024-09-13T17:18:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136295 s, 76.9 MB/s 2024-09-13T17:18:47.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127931 s, 82.0 MB/s 2024-09-13T17:18:47.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133397 s, 78.6 MB/s 2024-09-13T17:18:47.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132293 s, 79.3 MB/s 2024-09-13T17:18:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014074 s, 74.5 MB/s 2024-09-13T17:18:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134364 s, 78.0 MB/s 2024-09-13T17:18:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125026 s, 83.9 MB/s 2024-09-13T17:18:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136164 s, 77.0 MB/s 2024-09-13T17:18:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133488 s, 78.6 MB/s 2024-09-13T17:18:47.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.565 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130743 s, 80.2 MB/s 2024-09-13T17:18:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.581 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131618 s, 79.7 MB/s 2024-09-13T17:18:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.599 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164473 s, 63.8 MB/s 2024-09-13T17:18:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.613 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.614 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120973 s, 86.7 MB/s 2024-09-13T17:18:47.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.629 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130192 s, 80.5 MB/s 2024-09-13T17:18:47.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013081 s, 80.2 MB/s 2024-09-13T17:18:47.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.663 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164807 s, 63.6 MB/s 2024-09-13T17:18:47.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138057 s, 76.0 MB/s 2024-09-13T17:18:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161638 s, 64.9 MB/s 2024-09-13T17:18:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136564 s, 76.8 MB/s 2024-09-13T17:18:47.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.729 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133653 s, 78.5 MB/s 2024-09-13T17:18:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136364 s, 76.9 MB/s 2024-09-13T17:18:47.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.761 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138563 s, 75.7 MB/s 2024-09-13T17:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142649 s, 73.5 MB/s 2024-09-13T17:18:47.790 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.791 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114199 s, 91.8 MB/s 2024-09-13T17:18:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.807 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013812 s, 75.9 MB/s 2024-09-13T17:18:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.823 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138661 s, 75.6 MB/s 2024-09-13T17:18:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.838 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013637 s, 76.9 MB/s 2024-09-13T17:18:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142531 s, 73.6 MB/s 2024-09-13T17:18:47.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.26 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:47.857 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.912 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133646 s, 78.5 MB/s 2024-09-13T17:18:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.927 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130582 s, 80.3 MB/s 2024-09-13T17:18:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.942 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129516 s, 81.0 MB/s 2024-09-13T17:18:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.962 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170762 s, 61.4 MB/s 2024-09-13T17:18:47.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.979 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.979 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151143 s, 69.4 MB/s 2024-09-13T17:18:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:47.994 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:47.995 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012958 s, 80.9 MB/s 2024-09-13T17:18:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.010 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133193 s, 78.7 MB/s 2024-09-13T17:18:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135867 s, 77.2 MB/s 2024-09-13T17:18:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155799 s, 67.3 MB/s 2024-09-13T17:18:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016552 s, 63.4 MB/s 2024-09-13T17:18:48.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.078 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131332 s, 79.8 MB/s 2024-09-13T17:18:48.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132752 s, 79.0 MB/s 2024-09-13T17:18:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136763 s, 76.7 MB/s 2024-09-13T17:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187301 s, 56.0 MB/s 2024-09-13T17:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127553 s, 82.2 MB/s 2024-09-13T17:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134762 s, 77.8 MB/s 2024-09-13T17:18:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134868 s, 77.7 MB/s 2024-09-13T17:18:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167761 s, 62.5 MB/s 2024-09-13T17:18:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01693 s, 61.9 MB/s 2024-09-13T17:18:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.231 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139583 s, 75.1 MB/s 2024-09-13T17:18:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127775 s, 82.1 MB/s 2024-09-13T17:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135529 s, 77.4 MB/s 2024-09-13T17:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183856 s, 57.0 MB/s 2024-09-13T17:18:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160301 s, 65.4 MB/s 2024-09-13T17:18:48.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.317 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140124 s, 74.8 MB/s 2024-09-13T17:18:48.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138414 s, 75.8 MB/s 2024-09-13T17:18:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134332 s, 78.1 MB/s 2024-09-13T17:18:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.367 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159957 s, 65.6 MB/s 2024-09-13T17:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132865 s, 78.9 MB/s 2024-09-13T17:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148379 s, 70.7 MB/s 2024-09-13T17:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141129 s, 74.3 MB/s 2024-09-13T17:18:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.434 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171317 s, 61.2 MB/s 2024-09-13T17:18:48.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.27 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:48.436 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129027 s, 81.3 MB/s 2024-09-13T17:18:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133986 s, 78.3 MB/s 2024-09-13T17:18:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166051 s, 63.1 MB/s 2024-09-13T17:18:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.545 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017735 s, 59.1 MB/s 2024-09-13T17:18:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.560 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126028 s, 83.2 MB/s 2024-09-13T17:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127124 s, 82.5 MB/s 2024-09-13T17:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151004 s, 69.4 MB/s 2024-09-13T17:18:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131416 s, 79.8 MB/s 2024-09-13T17:18:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169094 s, 62.0 MB/s 2024-09-13T17:18:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016857 s, 62.2 MB/s 2024-09-13T17:18:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.664 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151765 s, 69.1 MB/s 2024-09-13T17:18:48.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.683 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168818 s, 62.1 MB/s 2024-09-13T17:18:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167089 s, 62.8 MB/s 2024-09-13T17:18:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127918 s, 82.0 MB/s 2024-09-13T17:18:48.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.736 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165464 s, 63.4 MB/s 2024-09-13T17:18:48.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.752 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137039 s, 76.5 MB/s 2024-09-13T17:18:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.768 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138984 s, 75.4 MB/s 2024-09-13T17:18:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134225 s, 78.1 MB/s 2024-09-13T17:18:48.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.802 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161254 s, 65.0 MB/s 2024-09-13T17:18:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133648 s, 78.5 MB/s 2024-09-13T17:18:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148846 s, 70.4 MB/s 2024-09-13T17:18:48.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.851 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013979 s, 75.0 MB/s 2024-09-13T17:18:48.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134838 s, 77.8 MB/s 2024-09-13T17:18:48.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131088 s, 80.0 MB/s 2024-09-13T17:18:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013555 s, 77.4 MB/s 2024-09-13T17:18:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.915 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160401 s, 65.4 MB/s 2024-09-13T17:18:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017109 s, 61.3 MB/s 2024-09-13T17:18:48.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167504 s, 62.6 MB/s 2024-09-13T17:18:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129307 s, 81.1 MB/s 2024-09-13T17:18:48.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128185 s, 81.8 MB/s 2024-09-13T17:18:48.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:48.999 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129452 s, 81.0 MB/s 2024-09-13T17:18:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.014 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132459 s, 79.2 MB/s 2024-09-13T17:18:49.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.28 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:49.016 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.076 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179149 s, 58.5 MB/s 2024-09-13T17:18:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.091 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135893 s, 77.2 MB/s 2024-09-13T17:18:49.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133841 s, 78.3 MB/s 2024-09-13T17:18:49.124 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154697 s, 67.8 MB/s 2024-09-13T17:18:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131925 s, 79.5 MB/s 2024-09-13T17:18:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.157 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143843 s, 72.9 MB/s 2024-09-13T17:18:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159478 s, 65.8 MB/s 2024-09-13T17:18:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131607 s, 79.7 MB/s 2024-09-13T17:18:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.206 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137482 s, 76.3 MB/s 2024-09-13T17:18:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145788 s, 71.9 MB/s 2024-09-13T17:18:49.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170457 s, 61.5 MB/s 2024-09-13T17:18:49.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.259 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138243 s, 75.9 MB/s 2024-09-13T17:18:49.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135441 s, 77.4 MB/s 2024-09-13T17:18:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.289 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123579 s, 84.9 MB/s 2024-09-13T17:18:49.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174507 s, 60.1 MB/s 2024-09-13T17:18:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137658 s, 76.2 MB/s 2024-09-13T17:18:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016556 s, 63.3 MB/s 2024-09-13T17:18:49.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.360 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137962 s, 76.0 MB/s 2024-09-13T17:18:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133582 s, 78.5 MB/s 2024-09-13T17:18:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158636 s, 66.1 MB/s 2024-09-13T17:18:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013733 s, 76.4 MB/s 2024-09-13T17:18:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.424 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132165 s, 79.3 MB/s 2024-09-13T17:18:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016531 s, 63.4 MB/s 2024-09-13T17:18:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159708 s, 65.7 MB/s 2024-09-13T17:18:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.478 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145479 s, 72.1 MB/s 2024-09-13T17:18:49.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163595 s, 64.1 MB/s 2024-09-13T17:18:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.516 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170541 s, 61.5 MB/s 2024-09-13T17:18:49.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134967 s, 77.7 MB/s 2024-09-13T17:18:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130806 s, 80.2 MB/s 2024-09-13T17:18:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.562 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131463 s, 79.8 MB/s 2024-09-13T17:18:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.579 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013875 s, 75.6 MB/s 2024-09-13T17:18:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148305 s, 70.7 MB/s 2024-09-13T17:18:49.597 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.29 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:49.598 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.732 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141361 s, 74.2 MB/s 2024-09-13T17:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.745 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115717 s, 90.6 MB/s 2024-09-13T17:18:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150342 s, 69.7 MB/s 2024-09-13T17:18:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136586 s, 76.8 MB/s 2024-09-13T17:18:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136723 s, 76.7 MB/s 2024-09-13T17:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134834 s, 77.8 MB/s 2024-09-13T17:18:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.827 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143459 s, 73.1 MB/s 2024-09-13T17:18:49.846 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.846 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.846 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170199 s, 61.6 MB/s 2024-09-13T17:18:49.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145922 s, 71.9 MB/s 2024-09-13T17:18:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136137 s, 77.0 MB/s 2024-09-13T17:18:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132218 s, 79.3 MB/s 2024-09-13T17:18:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131881 s, 79.5 MB/s 2024-09-13T17:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135546 s, 77.4 MB/s 2024-09-13T17:18:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01352 s, 77.6 MB/s 2024-09-13T17:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130134 s, 80.6 MB/s 2024-09-13T17:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170716 s, 61.4 MB/s 2024-09-13T17:18:49.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:49.993 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:49.993 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155548 s, 67.4 MB/s 2024-09-13T17:18:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126655 s, 82.8 MB/s 2024-09-13T17:18:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.022 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011579 s, 90.6 MB/s 2024-09-13T17:18:50.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128696 s, 81.5 MB/s 2024-09-13T17:18:50.057 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.058 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179369 s, 58.5 MB/s 2024-09-13T17:18:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135466 s, 77.4 MB/s 2024-09-13T17:18:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132329 s, 79.2 MB/s 2024-09-13T17:18:50.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.105 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137764 s, 76.1 MB/s 2024-09-13T17:18:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.120 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134516 s, 78.0 MB/s 2024-09-13T17:18:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.135 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127231 s, 82.4 MB/s 2024-09-13T17:18:50.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.152 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142039 s, 73.8 MB/s 2024-09-13T17:18:50.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.168 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134144 s, 78.2 MB/s 2024-09-13T17:18:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137926 s, 76.0 MB/s 2024-09-13T17:18:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139519 s, 75.2 MB/s 2024-09-13T17:18:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128773 s, 81.4 MB/s 2024-09-13T17:18:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.234 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167534 s, 62.6 MB/s 2024-09-13T17:18:50.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.30 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:50.236 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128874 s, 81.4 MB/s 2024-09-13T17:18:50.306 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.306 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.307 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142684 s, 73.5 MB/s 2024-09-13T17:18:50.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016141 s, 65.0 MB/s 2024-09-13T17:18:50.341 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.341 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.341 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141162 s, 74.3 MB/s 2024-09-13T17:18:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142245 s, 73.7 MB/s 2024-09-13T17:18:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158866 s, 66.0 MB/s 2024-09-13T17:18:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162133 s, 64.7 MB/s 2024-09-13T17:18:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150309 s, 69.8 MB/s 2024-09-13T17:18:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164883 s, 63.6 MB/s 2024-09-13T17:18:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132339 s, 79.2 MB/s 2024-09-13T17:18:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013856 s, 75.7 MB/s 2024-09-13T17:18:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177347 s, 59.1 MB/s 2024-09-13T17:18:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.496 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125044 s, 83.9 MB/s 2024-09-13T17:18:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.512 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135774 s, 77.2 MB/s 2024-09-13T17:18:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141618 s, 74.0 MB/s 2024-09-13T17:18:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169177 s, 62.0 MB/s 2024-09-13T17:18:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138629 s, 75.6 MB/s 2024-09-13T17:18:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124339 s, 84.3 MB/s 2024-09-13T17:18:50.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115765 s, 90.6 MB/s 2024-09-13T17:18:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016439 s, 63.8 MB/s 2024-09-13T17:18:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.625 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128914 s, 81.3 MB/s 2024-09-13T17:18:50.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139497 s, 75.2 MB/s 2024-09-13T17:18:50.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130191 s, 80.5 MB/s 2024-09-13T17:18:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170399 s, 61.5 MB/s 2024-09-13T17:18:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.691 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127579 s, 82.2 MB/s 2024-09-13T17:18:50.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.707 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.707 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136287 s, 76.9 MB/s 2024-09-13T17:18:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134422 s, 78.0 MB/s 2024-09-13T17:18:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.738 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130853 s, 80.1 MB/s 2024-09-13T17:18:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013342 s, 78.6 MB/s 2024-09-13T17:18:50.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134177 s, 78.1 MB/s 2024-09-13T17:18:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013082 s, 80.2 MB/s 2024-09-13T17:18:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165234 s, 63.5 MB/s 2024-09-13T17:18:50.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:18:50.804 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:18:50.804 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:18:50.805 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.31 2024-09-13T17:18:50.805 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:18:50.805 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:18:50.805 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:18:50.805 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:18:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134949 s, 77.7 MB/s 2024-09-13T17:18:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121814 s, 86.1 MB/s 2024-09-13T17:18:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136002 s, 77.1 MB/s 2024-09-13T17:18:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165975 s, 63.2 MB/s 2024-09-13T17:18:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139361 s, 75.2 MB/s 2024-09-13T17:18:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123391 s, 85.0 MB/s 2024-09-13T17:18:50.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137746 s, 76.1 MB/s 2024-09-13T17:18:50.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138413 s, 75.8 MB/s 2024-09-13T17:18:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:50.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133485 s, 78.6 MB/s 2024-09-13T17:18:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.004 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134685 s, 77.9 MB/s 2024-09-13T17:18:51.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140219 s, 74.8 MB/s 2024-09-13T17:18:51.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136191 s, 77.0 MB/s 2024-09-13T17:18:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016001 s, 65.5 MB/s 2024-09-13T17:18:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167957 s, 62.4 MB/s 2024-09-13T17:18:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.088 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.089 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134682 s, 77.9 MB/s 2024-09-13T17:18:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160374 s, 65.4 MB/s 2024-09-13T17:18:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.122 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135594 s, 77.3 MB/s 2024-09-13T17:18:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.137 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126959 s, 82.6 MB/s 2024-09-13T17:18:51.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.153 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131041 s, 80.0 MB/s 2024-09-13T17:18:51.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.168 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128388 s, 81.7 MB/s 2024-09-13T17:18:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164375 s, 63.8 MB/s 2024-09-13T17:18:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015846 s, 66.2 MB/s 2024-09-13T17:18:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172228 s, 60.9 MB/s 2024-09-13T17:18:51.240 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.240 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.240 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133587 s, 78.5 MB/s 2024-09-13T17:18:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.254 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114132 s, 91.9 MB/s 2024-09-13T17:18:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132122 s, 79.4 MB/s 2024-09-13T17:18:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124286 s, 84.4 MB/s 2024-09-13T17:18:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146147 s, 71.7 MB/s 2024-09-13T17:18:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157016 s, 66.8 MB/s 2024-09-13T17:18:51.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.334 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126618 s, 82.8 MB/s 2024-09-13T17:18:51.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138591 s, 75.7 MB/s 2024-09-13T17:18:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:18:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:18:51.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131455 s, 79.8 MB/s 2024-09-13T17:18:51.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub start /scrub_control_test_path recursive 2024-09-13T17:18:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.525+0000 7f5e5be00640 1 Processor -- start 2024-09-13T17:18:51.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.525+0000 7f5e5be00640 1 -- start start 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e5c159190 0x7f5e5c06fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c0cdf10 0x7f5e5c06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e5c0cd1d0 con 0x7f5e5c06d240 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e5c06d6d0 con 0x7f5e5c159190 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c0cd660 con 0x7f5e5c0cdf10 2024-09-13T17:18:51.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e5c159190 0x7f5e5c06fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e5c159190 0x7f5e5c06fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38492/0 (socket says 172.21.15.28:38492) 2024-09-13T17:18:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e5aa00640 1 -- 172.21.15.28:0/2354902640 learned_addr learned my addr 172.21.15.28:0/2354902640 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.526+0000 7f5e53e00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c0cdf10 0x7f5e5c06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.527+0000 7f5e5aa00640 1 -- 172.21.15.28:0/2354902640 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c0cdf10 msgr2=0x7f5e5c06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.527+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c0cdf10 0x7f5e5c06cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.527+0000 7f5e5aa00640 1 -- 172.21.15.28:0/2354902640 >> v1:172.21.15.28:6789/0 conn(0x7f5e5c06d240 legacy=0x7f5e5c15ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.527+0000 7f5e5aa00640 1 -- 172.21.15.28:0/2354902640 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c0cc5c0 con 0x7f5e5c159190 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.528+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c159190 0x7f5e5c06fd10 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009870 tx=0x7f5e4c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af388dd196edf3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.528+0000 7f5e5a000640 1 -- 172.21.15.28:0/2354902640 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c032440 con 0x7f5e5c159190 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.528+0000 7f5e5a000640 1 -- 172.21.15.28:0/2354902640 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c0325a0 con 0x7f5e5c159190 2024-09-13T17:18:51.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.528+0000 7f5e5a000640 1 -- 172.21.15.28:0/2354902640 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c032930 con 0x7f5e5c159190 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 -- 172.21.15.28:0/2354902640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c159190 msgr2=0x7f5e5c06fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c159190 0x7f5e5c06fd10 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c009870 tx=0x7f5e4c02a250 comp rx=0 tx=0).stop 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 -- 172.21.15.28:0/2354902640 shutdown_connections 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c0cdf10 0x7f5e5c06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 --2- 172.21.15.28:0/2354902640 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c159190 0x7f5e5c06fd10 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 -- 172.21.15.28:0/2354902640 >> 172.21.15.28:0/2354902640 conn(0x7f5e5c073470 msgr2=0x7f5e5c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 -- 172.21.15.28:0/2354902640 shutdown_connections 2024-09-13T17:18:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.529+0000 7f5e5be00640 1 -- 172.21.15.28:0/2354902640 wait complete. 2024-09-13T17:18:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.530+0000 7f5e5be00640 1 Processor -- start 2024-09-13T17:18:51.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.530+0000 7f5e5be00640 1 -- start start 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5c0cdf10 0x7f5e5c151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c159190 0x7f5e5c15cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15bda0 con 0x7f5e5c06d240 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15b190 con 0x7f5e5c0cdf10 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c15b470 con 0x7f5e5c159190 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.531+0000 7f5e5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38502/0 (socket says 172.21.15.28:38502) 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 -- 172.21.15.28:0/1601801809 learned_addr learned my addr 172.21.15.28:0/1601801809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e53e00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5c0cdf10 0x7f5e5c151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5b400640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c159190 0x7f5e5c15cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 -- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c159190 msgr2=0x7f5e5c15cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c159190 0x7f5e5c15cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 -- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5c0cdf10 msgr2=0x7f5e5c151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5c0cdf10 0x7f5e5c151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 -- 172.21.15.28:0/1601801809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0094d0 con 0x7f5e5c06d240 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c000c00 tx=0x7f5e4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.532+0000 7f5e52a00640 1 -- 172.21.15.28:0/1601801809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c032440 con 0x7f5e5c06d240 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.533+0000 7f5e52a00640 1 -- 172.21.15.28:0/1601801809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4c030980 con 0x7f5e5c06d240 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.533+0000 7f5e52a00640 1 -- 172.21.15.28:0/1601801809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4c030d10 con 0x7f5e5c06d240 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.533+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c15b750 con 0x7f5e5c06d240 2024-09-13T17:18:51.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.533+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c146050 con 0x7f5e5c06d240 2024-09-13T17:18:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.535+0000 7f5e52a00640 1 -- 172.21.15.28:0/1601801809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e4c039040 con 0x7f5e5c06d240 2024-09-13T17:18:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.536+0000 7f5e52a00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 0x7f5e3806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.536+0000 7f5e52a00640 1 -- 172.21.15.28:0/1601801809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e4c0a9d30 con 0x7f5e5c06d240 2024-09-13T17:18:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.536+0000 7f5e53e00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 0x7f5e3806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.537+0000 7f5e64a0f740 1 Processor -- start 2024-09-13T17:18:51.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.537+0000 7f5e64a0f740 1 -- start start 2024-09-13T17:18:51.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.537+0000 7f5e53e00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 0x7f5e3806d0c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c153810 tx=0x7f5e44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 0x55c72c4d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4f7380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c40ff00 con 0x55c72c4ea560 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c40d690 con 0x55c72c4d5ab0 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c72c408f80 con 0x55c72c4f7b90 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38514/0 (socket says 172.21.15.28:38514) 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 -- 172.21.15.28:0/629578830 learned_addr learned my addr 172.21.15.28:0/629578830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 0x55c72c4d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e5b400640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 -- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 msgr2=0x55c72c4f7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4f7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 -- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 msgr2=0x55c72c4d5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 0x55c72c4d5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e53e00640 1 -- 172.21.15.28:0/629578830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c72c3d3390 con 0x55c72c4ea560 2024-09-13T17:18:51.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.538+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 0x55c72c4d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:51.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e53e00640 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f5e44011db0 tx=0x7f5e44014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=313810ceeacf7e9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e4b400640 1 -- 172.21.15.28:0/629578830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e4401c040 con 0x55c72c4ea560 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e4b400640 1 -- 172.21.15.28:0/629578830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44011ea0 con 0x55c72c4ea560 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e4b400640 1 -- 172.21.15.28:0/629578830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44003a20 con 0x55c72c4ea560 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e64a0f740 1 -- 172.21.15.28:0/629578830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 msgr2=0x55c72c4f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.539+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f5e44011db0 tx=0x7f5e44014000 comp rx=0 tx=0).stop 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 -- 172.21.15.28:0/629578830 shutdown_connections 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4f7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4ea560 0x55c72c4f6e40 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/629578830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4d5ab0 0x55c72c4d5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 -- 172.21.15.28:0/629578830 >> 172.21.15.28:0/629578830 conn(0x55c72c4a2bd0 msgr2=0x55c72c4a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 -- 172.21.15.28:0/629578830 shutdown_connections 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 -- 172.21.15.28:0/629578830 wait complete. 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 -- start start 2024-09-13T17:18:51.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.540+0000 7f5e64a0f740 1 Processor -- start 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c4d84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 0x55c72c5033e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c3c72c0 con 0x55c72c4d22f0 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c3c6f60 con 0x55c72c4ea560 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c72c1703e0 con 0x55c72c4f7b90 2024-09-13T17:18:51.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.541+0000 7f5e64a0f740 10 client.0 _ll_register_callbacks cb 0x55c72c3dddb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e5aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c4d84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 0x55c72c5033e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55928/0 (socket says 172.21.15.28:55928) 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e5aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c4d84a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:43506/0 (socket says 172.21.15.28:43506) 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 -- 172.21.15.28:0/2852823670 learned_addr learned my addr 172.21.15.28:0/2852823670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 -- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 msgr2=0x55c72c4d84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c4d84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 -- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 msgr2=0x55c72c5033e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 0x55c72c5033e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 -- 172.21.15.28:0/2852823670 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e44011a90 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e5b400640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 0x55c72c5033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e53e00640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e44023000 tx=0x7f5e44004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1c528e9a6f0031f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.542+0000 7f5e3fe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3301/0 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e44022ac0 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c72c431ab0 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mon.2 v2:172.21.15.189:3301/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e44022c20 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e44022de0 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e64a0f740 10 client.7790 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.543+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55c72c407600 con 0x55c72c4f7b90 2024-09-13T17:18:51.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mon.2 v2:172.21.15.189:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c72c431ab0 con 0x55c72c4f7b90 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mon.2 v2:172.21.15.189:3301/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e44014440 con 0x55c72c4f7b90 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 10 client.7790 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 10 client.7790 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 20 client.7790 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d28e0 0x55c72c503920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 4 client.7790 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c72c3fa320 con 0x55c72c4d28e0 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.544+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d28e0 0x55c72c503920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.545+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d28e0 0x55c72c503920 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf86b6b5ec7357f8 server_cookie=719fa2b58869a5c2 in_seq=0 out_seq=0 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.545+0000 7f5e3fe00640 10 client.7790 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.545+0000 7f5e3fe00640 1 -- 172.21.15.28:0/2852823670 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55c72c3fa320 con 0x55c72c4d28e0 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.545+0000 7f5e3fe00640 10 client.7790 handle_command_reply: tid=0 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.545+0000 7f5e64a0f740 1 client.7790 shutdown 2024-09-13T17:18:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 msgr2=0x55c72c4d18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5e44023000 tx=0x7f5e44004730 comp rx=0 tx=0).stop 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 shutdown_connections 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d28e0 0x55c72c503920 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4d22f0 0x55c72c5033e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4f7b90 0x55c72c4d18c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/2852823670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c4d84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 >> 172.21.15.28:0/2852823670 conn(0x55c72c4d5ab0 msgr2=0x55c72c4f69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e3fe00640 0 client.7790 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 shutdown_connections 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 1 -- 172.21.15.28:0/2852823670 wait complete. 2024-09-13T17:18:51.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.546+0000 7f5e64a0f740 20 client.7790 trim_cache size 0 max 16384 2024-09-13T17:18:51.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.561+0000 7f5e64a0f740 1 Processor -- start 2024-09-13T17:18:51.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.561+0000 7f5e64a0f740 1 -- start start 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c502240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c502780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c4f6ff0 con 0x55c72c4f7b90 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c4ae210 con 0x55c72c4ea560 2024-09-13T17:18:51.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c72c4d5470 con 0x55c72c4d22f0 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38534/0 (socket says 172.21.15.28:38534) 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e5b400640 1 -- 172.21.15.28:0/1674852415 learned_addr learned my addr 172.21.15.28:0/1674852415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e53e00640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c502780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.562+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c502240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 -- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 msgr2=0x55c72c502240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c502240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 -- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 msgr2=0x55c72c502780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c502780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 -- 172.21.15.28:0/1674852415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c72c43b5b0 con 0x55c72c4f7b90 2024-09-13T17:18:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c502240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e5b400640 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f5e54001b00 tx=0x7f5e54001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e88ead9e3cb4f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e4be00640 1 -- 172.21.15.28:0/1674852415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e540057c0 con 0x55c72c4f7b90 2024-09-13T17:18:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e4be00640 1 -- 172.21.15.28:0/1674852415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54006b50 con 0x55c72c4f7b90 2024-09-13T17:18:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.563+0000 7f5e4be00640 1 -- 172.21.15.28:0/1674852415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54006d00 con 0x55c72c4f7b90 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- 172.21.15.28:0/1674852415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 msgr2=0x55c72c555990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f5e54001b00 tx=0x7f5e54001670 comp rx=0 tx=0).stop 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- 172.21.15.28:0/1674852415 shutdown_connections 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555990 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c502780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/1674852415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c502240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- 172.21.15.28:0/1674852415 >> 172.21.15.28:0/1674852415 conn(0x55c72c4d5ab0 msgr2=0x55c72c4a19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- 172.21.15.28:0/1674852415 shutdown_connections 2024-09-13T17:18:51.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- 172.21.15.28:0/1674852415 wait complete. 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 -- start start 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.564+0000 7f5e64a0f740 1 Processor -- start 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c559180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c554190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c558af0 con 0x55c72c4f7b90 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c72c4d3040 con 0x55c72c4ea560 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c72c4aeb30 con 0x55c72c4d22f0 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.565+0000 7f5e64a0f740 10 client.0 _ll_register_callbacks cb 0x7f5e440095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38542/0 (socket says 172.21.15.28:38542) 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c559180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 -- 172.21.15.28:0/711084820 learned_addr learned my addr 172.21.15.28:0/711084820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e53e00640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c554190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 -- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 msgr2=0x55c72c559180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c559180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 -- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 msgr2=0x55c72c554190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c554190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 -- 172.21.15.28:0/711084820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c0cd1d0 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e5b400640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f5e54003bd0 tx=0x7f5e54001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e742f1101ac1146 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.566+0000 7f5e3fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e540077f0 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e54007950 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e44011ea0 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e54007b10 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 10 client.9606 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55c72c3a9400 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e44011ea0 con 0x55c72c4f7b90 2024-09-13T17:18:51.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e54020460 con 0x55c72c4f7b90 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 10 client.9606 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 10 client.9606 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 20 client.9606 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d6140 0x55c72c555ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 4 client.9606 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.567+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x55c72c0fc880 con 0x55c72c4d6140 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.568+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d6140 0x55c72c555ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.568+0000 7f5e5aa00640 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d6140 0x55c72c555ca0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac9517162e220b server_cookie=522319e1b4653a88 in_seq=0 out_seq=0 2024-09-13T17:18:51.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.568+0000 7f5e3fe00640 10 client.9606 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e3fe00640 1 -- 172.21.15.28:0/711084820 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c72c0fc880 con 0x55c72c4d6140 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e3fe00640 10 client.9606 handle_command_reply: tid=0 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 client.9606 shutdown 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 msgr2=0x55c72c555760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f5e54003bd0 tx=0x7f5e54001250 comp rx=0 tx=0).stop 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 shutdown_connections 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55c72c4d6140 0x55c72c555ca0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c72c4f7b90 0x55c72c555760 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c72c4ea560 0x55c72c554190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 --2- 172.21.15.28:0/711084820 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c72c4d22f0 0x55c72c559180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 >> 172.21.15.28:0/711084820 conn(0x55c72c4d5ab0 msgr2=0x55c72c4f7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.572+0000 7f5e3fe00640 0 client.9606 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 shutdown_connections 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e64a0f740 1 -- 172.21.15.28:0/711084820 wait complete. 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e64a0f740 20 client.9606 trim_cache size 0 max 16384 2024-09-13T17:18:51.576 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:51.577 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:18:51.577 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "f3d2c57d-c36b-4c28-bf19-d2110b4fd507", 2024-09-13T17:18:51.577 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:18:51.577 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 msgr2=0x7f5e3806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 0x7f5e3806d0c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c153810 tx=0x7f5e44009190 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 msgr2=0x7f5e5c145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c000c00 tx=0x7f5e4c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 shutdown_connections 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e3806ac00 0x7f5e3806d0c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e5c159190 0x7f5e5c15cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e5c0cdf10 0x7f5e5c151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 --2- 172.21.15.28:0/1601801809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e5c06d240 0x7f5e5c145b10 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.573+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 >> 172.21.15.28:0/1601801809 conn(0x7f5e5c073470 msgr2=0x7f5e5c152b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.574+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 shutdown_connections 2024-09-13T17:18:51.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.574+0000 7f5e5be00640 1 -- 172.21.15.28:0/1601801809 wait complete. 2024-09-13T17:18:51.592 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub pause 2024-09-13T17:18:51.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.781+0000 7f890d200640 1 Processor -- start 2024-09-13T17:18:51.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.782+0000 7f890d200640 1 -- start start 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f890d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f8908165190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f890d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890815ef70 con 0x7f89080cd7c0 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f890d200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89080c4320 con 0x7f8908143f70 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f890d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890815e1b0 con 0x7f89080c3d70 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f88fea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89080cd7c0 0x7f89080c3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44266/0 (socket says 172.21.15.28:44266) 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f88fea00640 1 -- 172.21.15.28:0/2859028691 learned_addr learned my addr 172.21.15.28:0/2859028691 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:51.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.783+0000 7f8907e00640 1 --2- 172.21.15.28:0/2859028691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f8908165190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900334561 0 0) 0x7f890815ef70 con 0x7f89080cd7c0 2024-09-13T17:18:51.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88ec003520 con 0x7f89080cd7c0 2024-09-13T17:18:51.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753776272 0 0) 0x7f89080c4320 con 0x7f8908143f70 2024-09-13T17:18:51.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8907e00640 1 -- 172.21.15.28:0/2859028691 >> v1:172.21.15.189:6789/0 conn(0x7f8908143f70 legacy=0x7f89080cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8907e00640 1 -- 172.21.15.28:0/2859028691 >> v1:172.21.15.28:6789/0 conn(0x7f89080cd7c0 legacy=0x7f89080c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.784+0000 7f8907e00640 1 -- 172.21.15.28:0/2859028691 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890815e490 con 0x7f89080c3d70 2024-09-13T17:18:51.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.785+0000 7f8907e00640 1 --2- 172.21.15.28:0/2859028691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f8908165190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8900009dc0 tx=0x7f890002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=968236112350b9d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.785+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900007740 con 0x7f89080c3d70 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.785+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890002d050 con 0x7f89080c3d70 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.785+0000 7f8906a00640 1 -- 172.21.15.28:0/2859028691 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900007ac0 con 0x7f89080c3d70 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 -- 172.21.15.28:0/2859028691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 msgr2=0x7f8908165190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 --2- 172.21.15.28:0/2859028691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f8908165190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8900009dc0 tx=0x7f890002a6d0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 -- 172.21.15.28:0/2859028691 shutdown_connections 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 --2- 172.21.15.28:0/2859028691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f8908165190 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 -- 172.21.15.28:0/2859028691 >> 172.21.15.28:0/2859028691 conn(0x7f8908069e80 msgr2=0x7f890806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 -- 172.21.15.28:0/2859028691 shutdown_connections 2024-09-13T17:18:51.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.786+0000 7f890d200640 1 -- 172.21.15.28:0/2859028691 wait complete. 2024-09-13T17:18:51.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.788+0000 7f890d200640 1 Processor -- start 2024-09-13T17:18:51.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.788+0000 7f890d200640 1 -- start start 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f890815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89080cd7c0 0x7f89080ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890813ac90 con 0x7f8908143f70 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890813a800 con 0x7f89080cd7c0 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f890d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89080caae0 con 0x7f89080c3d70 2024-09-13T17:18:51.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f8907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f8907400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f890815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f88fea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89080cd7c0 0x7f89080ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f8907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38558/0 (socket says 172.21.15.28:38558) 2024-09-13T17:18:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.789+0000 7f8907e00640 1 -- 172.21.15.28:0/4211393962 learned_addr learned my addr 172.21.15.28:0/4211393962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 -- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 msgr2=0x7f890815aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f890815aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 -- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89080cd7c0 msgr2=0x7f89080ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89080cd7c0 0x7f89080ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 -- 172.21.15.28:0/4211393962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8900009a70 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f8907e00640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f8900009eb0 tx=0x7f8900004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f88ffe00640 1 -- 172.21.15.28:0/4211393962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890003b820 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f88ffe00640 1 -- 172.21.15.28:0/4211393962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f890003b980 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8908139f60 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f88ffe00640 1 -- 172.21.15.28:0/4211393962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890003bcf0 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.790+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890813a520 con 0x7f8908143f70 2024-09-13T17:18:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.792+0000 7f88ffe00640 1 -- 172.21.15.28:0/4211393962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8900031400 con 0x7f8908143f70 2024-09-13T17:18:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.793+0000 7f88ffe00640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 0x7f88e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.793+0000 7f88ffe00640 1 -- 172.21.15.28:0/4211393962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89000aaaf0 con 0x7f8908143f70 2024-09-13T17:18:51.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.794+0000 7f8907400640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 0x7f88e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.794+0000 7f8911687740 1 Processor -- start 2024-09-13T17:18:51.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.794+0000 7f8911687740 1 -- start start 2024-09-13T17:18:51.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.794+0000 7f8907400640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 0x7f88e406d0c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f88f8000f60 tx=0x7f88f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89b8dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bdb380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89a94000 con 0x563f89b8b130 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89af1d40 con 0x563f89b8d970 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89aef4d0 con 0x563f89bb4e20 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f88fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8907400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89b8dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f88fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38568/0 (socket says 172.21.15.28:38568) 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f88fea00640 1 -- 172.21.15.28:0/1280476458 learned_addr learned my addr 172.21.15.28:0/1280476458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.795+0000 7f8907e00640 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bdb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 -- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 msgr2=0x563f89bdb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bdb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 -- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 msgr2=0x563f89b8dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89b8dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 -- 172.21.15.28:0/1280476458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f89aa63a0 con 0x563f89b8b130 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f8907e00640 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bdb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88fea00640 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f88f0001670 tx=0x7f88f0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72195ce675e08b69 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88f7e00640 1 -- 172.21.15.28:0/1280476458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0001d10 con 0x563f89b8b130 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88f7e00640 1 -- 172.21.15.28:0/1280476458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0005ac0 con 0x563f89b8b130 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.796+0000 7f88f7e00640 1 -- 172.21.15.28:0/1280476458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0005e70 con 0x563f89b8b130 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 -- 172.21.15.28:0/1280476458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 msgr2=0x563f89bdae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f88f0001670 tx=0x7f88f0000bc0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 -- 172.21.15.28:0/1280476458 shutdown_connections 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bdb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8b130 0x563f89bdae40 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 --2- 172.21.15.28:0/1280476458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89b8dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 -- 172.21.15.28:0/1280476458 >> 172.21.15.28:0/1280476458 conn(0x563f89b8a0a0 msgr2=0x563f89bd2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 -- 172.21.15.28:0/1280476458 shutdown_connections 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.797+0000 7f8911687740 1 -- 172.21.15.28:0/1280476458 wait complete. 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.798+0000 7f8911687740 1 -- start start 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.798+0000 7f8911687740 1 Processor -- start 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89bd3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bad580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89aeadc0 con 0x563f89bcfc60 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89aa93e0 con 0x563f89b8d970 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89aa9080 con 0x563f89bb4e20 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8911687740 10 client.0 _ll_register_callbacks cb 0x563f89ae4430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8907e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38582/0 (socket says 172.21.15.28:38582) 2024-09-13T17:18:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8907e00640 1 -- 172.21.15.28:0/2740543683 learned_addr learned my addr 172.21.15.28:0/2740543683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f88fea00640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bad580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.799+0000 7f8907400640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89bd3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 -- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 msgr2=0x563f89bad580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bad580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 -- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 msgr2=0x563f89bd3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89bd3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 -- 172.21.15.28:0/2740543683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec003520 con 0x563f89bcfc60 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8907e00640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f890003e040 tx=0x7f890007d870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b1159b053eaf497 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f88f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8900079e90 con 0x563f89bcfc60 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8900078f20 con 0x563f89bcfc60 2024-09-13T17:18:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f89b14600 con 0x563f89bcfc60 2024-09-13T17:18:51.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f89000790e0 con 0x563f89bcfc60 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8911687740 10 client.9618 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.800+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x563f89ae9440 con 0x563f89bcfc60 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563f89b14600 con 0x563f89bcfc60 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8900038040 con 0x563f89bcfc60 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 10 client.9618 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 10 client.9618 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 20 client.9618 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bab950 0x563f89babd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 4 client.9618 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f89adcc90 con 0x563f89bab950 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8907400640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bab950 0x563f89babd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f88f4c00640 10 client.9618 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.801+0000 7f8907400640 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bab950 0x563f89babd30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b61cbd99595722dd server_cookie=c4d67c55a7a30b9c in_seq=0 out_seq=0 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f88f4c00640 1 -- 172.21.15.28:0/2740543683 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x563f89adcc90 con 0x563f89bab950 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f88f4c00640 10 client.9618 handle_command_reply: tid=0 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 client.9618 shutdown 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 msgr2=0x563f89baefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f890003e040 tx=0x7f890007d870 comp rx=0 tx=0).stop 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 shutdown_connections 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bab950 0x563f89babd30 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bcfc60 0x563f89baefa0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bb4e20 0x563f89bad580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.802+0000 7f8911687740 1 --2- 172.21.15.28:0/2740543683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89b8d970 0x563f89bd3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.803+0000 7f88f4c00640 0 client.9618 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.803+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 >> 172.21.15.28:0/2740543683 conn(0x563f89b8b130 msgr2=0x563f89b88560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.803+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 shutdown_connections 2024-09-13T17:18:51.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.803+0000 7f8911687740 1 -- 172.21.15.28:0/2740543683 wait complete. 2024-09-13T17:18:51.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.803+0000 7f8911687740 20 client.9618 trim_cache size 0 max 16384 2024-09-13T17:18:51.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 Processor -- start 2024-09-13T17:18:51.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 -- start start 2024-09-13T17:18:51.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89b8d970 0x563f89b82a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 0x563f89b88390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.818+0000 7f8911687740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89852040 con 0x563f89bb4e20 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89bf3de0 con 0x563f89bcfc60 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89baf010 con 0x563f89b8d970 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38586/0 (socket says 172.21.15.28:38586) 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 -- 172.21.15.28:0/3310429838 learned_addr learned my addr 172.21.15.28:0/3310429838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f8907400640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89b8d970 0x563f89b82a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f8907e00640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 0x563f89b88390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 -- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89b8d970 msgr2=0x563f89b82a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89b8d970 0x563f89b82a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 -- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 msgr2=0x563f89b88390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 0x563f89b88390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f88fea00640 1 -- 172.21.15.28:0/3310429838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f89a8b5d0 con 0x563f89bb4e20 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.819+0000 7f8907e00640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 0x563f89b88390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f88fea00640 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f88f0014000 tx=0x7f88f0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87dc294eeec9b4bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f88fcc00640 1 -- 172.21.15.28:0/3310429838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f00072d0 con 0x563f89bb4e20 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f88fcc00640 1 -- 172.21.15.28:0/3310429838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0007430 con 0x563f89bb4e20 2024-09-13T17:18:51.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f88fcc00640 1 -- 172.21.15.28:0/3310429838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f0001d10 con 0x563f89bb4e20 2024-09-13T17:18:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 -- 172.21.15.28:0/3310429838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 msgr2=0x563f89bd05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f88f0014000 tx=0x7f88f0012000 comp rx=0 tx=0).stop 2024-09-13T17:18:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 -- 172.21.15.28:0/3310429838 shutdown_connections 2024-09-13T17:18:51.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bcfc60 0x563f89b88390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89bb4e20 0x563f89bd05a0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 --2- 172.21.15.28:0/3310429838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89b8d970 0x563f89b82a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 -- 172.21.15.28:0/3310429838 >> 172.21.15.28:0/3310429838 conn(0x563f89b8b130 msgr2=0x563f89bd1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 -- 172.21.15.28:0/3310429838 shutdown_connections 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.820+0000 7f8911687740 1 -- 172.21.15.28:0/3310429838 wait complete. 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.821+0000 7f8911687740 1 -- start start 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.821+0000 7f8911687740 1 Processor -- start 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bb4e20 0x563f89c3c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bcfc60 0x563f89b7f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89bebc90 con 0x563f89b8d970 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f89bb44b0 con 0x563f89bb4e20 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f89bac830 con 0x563f89bcfc60 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8911687740 10 client.0 _ll_register_callbacks cb 0x7f8900079390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:51.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.822+0000 7f8907400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38590/0 (socket says 172.21.15.28:38590) 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 -- 172.21.15.28:0/2746154394 learned_addr learned my addr 172.21.15.28:0/2746154394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907e00640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bcfc60 0x563f89b7f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f88fea00640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bb4e20 0x563f89c3c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 -- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bcfc60 msgr2=0x563f89b7f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bcfc60 0x563f89b7f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 -- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bb4e20 msgr2=0x563f89c3c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bb4e20 0x563f89c3c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 -- 172.21.15.28:0/2746154394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f89aa63a0 con 0x563f89b8d970 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f8907400640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x563f89bab0e0 tx=0x7f88f8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6886402269a1e4ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f88f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.823+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80291b0 con 0x563f89b8d970 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88f801c040 con 0x563f89b8d970 2024-09-13T17:18:51.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f88f0005ac0 con 0x563f89b8d970 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f88f8029310 con 0x563f89b8d970 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 10 client.9624 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x563f89ab09b0 con 0x563f89b8d970 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f0005ac0 con 0x563f89b8d970 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f88f8012070 con 0x563f89b8d970 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 10 client.9624 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 10 client.9624 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 20 client.9624 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bad560 0x563f89b7f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 4 client.9624 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x563f897de4e0 con 0x563f89bad560 2024-09-13T17:18:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.824+0000 7f88fea00640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bad560 0x563f89b7f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f88fea00640 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bad560 0x563f89b7f6b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13ec6c9b3deb4d8a server_cookie=9ec9086a456d5cb4 in_seq=0 out_seq=0 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f88f4c00640 10 client.9624 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f88f4c00640 1 -- 172.21.15.28:0/2746154394 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x563f897de4e0 con 0x563f89bad560 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f88f4c00640 10 client.9624 handle_command_reply: tid=0 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 client.9624 shutdown 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 msgr2=0x563f89bcf1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x563f89bab0e0 tx=0x7f88f8001080 comp rx=0 tx=0).stop 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 shutdown_connections 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x563f89bad560 0x563f89b7f6b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f89bcfc60 0x563f89b7f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f89bb4e20 0x563f89c3c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 --2- 172.21.15.28:0/2746154394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f89b8d970 0x563f89bcf1b0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f88f4c00640 0 client.9624 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.825+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 >> 172.21.15.28:0/2746154394 conn(0x563f89b8b130 msgr2=0x563f89b881b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 shutdown_connections 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f8911687740 1 -- 172.21.15.28:0/2746154394 wait complete. 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f8911687740 20 client.9624 trim_cache size 0 max 16384 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:18:51.832 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 msgr2=0x7f88e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 0x7f88e406d0c0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f88f8000f60 tx=0x7f88f8009040 comp rx=0 tx=0).stop 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 msgr2=0x7f890815cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f8900009eb0 tx=0x7f8900004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 shutdown_connections 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f88e406ac00 0x7f88e406d0c0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8908143f70 0x7f890815cc30 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89080cd7c0 0x7f89080ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 --2- 172.21.15.28:0/4211393962 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89080c3d70 0x7f890815aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 >> 172.21.15.28:0/4211393962 conn(0x7f8908069e80 msgr2=0x7f890806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.826+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 shutdown_connections 2024-09-13T17:18:51.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.827+0000 7f890d200640 1 -- 172.21.15.28:0/4211393962 wait complete. 2024-09-13T17:18:51.843 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub status 2024-09-13T17:18:52.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.999+0000 7f6591000640 1 Processor -- start 2024-09-13T17:18:52.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:51.999+0000 7f6591000640 1 -- start start 2024-09-13T17:18:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6591000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f658c071490 0x7f658c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6591000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c06c8b0 con 0x7f658c15a780 2024-09-13T17:18:52.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6591000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658c06cb90 con 0x7f658c15ab60 2024-09-13T17:18:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6591000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658c0cc980 con 0x7f658c071490 2024-09-13T17:18:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6589e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f658c15a780 0x7f658c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44282/0 (socket says 172.21.15.28:44282) 2024-09-13T17:18:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.000+0000 7f6589e00640 1 -- 172.21.15.28:0/2364433625 learned_addr learned my addr 172.21.15.28:0/2364433625 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f658a800640 1 --2- 172.21.15.28:0/2364433625 >> v2:172.21.15.189:3300/0 conn(0x7f658c071490 0x7f658c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483549705 0 0) 0x7f658c06c8b0 con 0x7f658c15a780 2024-09-13T17:18:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6570003520 con 0x7f658c15a780 2024-09-13T17:18:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f658a800640 1 -- 172.21.15.28:0/2364433625 >> v1:172.21.15.189:6789/0 conn(0x7f658c15ab60 legacy=0x7f658c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200376994 0 0) 0x7f658c06cb90 con 0x7f658c15ab60 2024-09-13T17:18:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f658a800640 1 -- 172.21.15.28:0/2364433625 >> v1:172.21.15.28:6789/0 conn(0x7f658c15a780 legacy=0x7f658c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.001+0000 7f658a800640 1 -- 172.21.15.28:0/2364433625 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0ccc60 con 0x7f658c071490 2024-09-13T17:18:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.002+0000 7f658a800640 1 --2- 172.21.15.28:0/2364433625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c071490 0x7f658c06c250 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d75a011bce99137b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.002+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584007740 con 0x7f658c071490 2024-09-13T17:18:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658402d050 con 0x7f658c071490 2024-09-13T17:18:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6588a00640 1 -- 172.21.15.28:0/2364433625 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584007ac0 con 0x7f658c071490 2024-09-13T17:18:52.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 -- 172.21.15.28:0/2364433625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c071490 msgr2=0x7f658c06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 --2- 172.21.15.28:0/2364433625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c071490 0x7f658c06c250 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f658402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 -- 172.21.15.28:0/2364433625 shutdown_connections 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 --2- 172.21.15.28:0/2364433625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c071490 0x7f658c06c250 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 -- 172.21.15.28:0/2364433625 >> 172.21.15.28:0/2364433625 conn(0x7f658c0b0270 msgr2=0x7f658c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 -- 172.21.15.28:0/2364433625 shutdown_connections 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.003+0000 7f6591000640 1 -- 172.21.15.28:0/2364433625 wait complete. 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.005+0000 7f6591000640 1 Processor -- start 2024-09-13T17:18:52.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.005+0000 7f6591000640 1 -- start start 2024-09-13T17:18:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f658c071490 0x7f658c139350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c15a780 0x7f658c135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c152760 con 0x7f658c15ab60 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c14b2f0 con 0x7f658c15a780 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6591000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658c153b20 con 0x7f658c071490 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38598/0 (socket says 172.21.15.28:38598) 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 -- 172.21.15.28:0/3254908175 learned_addr learned my addr 172.21.15.28:0/3254908175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6589400640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c15a780 0x7f658c135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f6589e00640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f658c071490 0x7f658c139350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 -- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f658c071490 msgr2=0x7f658c139350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f658c071490 0x7f658c139350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 -- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c15a780 msgr2=0x7f658c135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c15a780 0x7f658c135ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.006+0000 7f658a800640 1 -- 172.21.15.28:0/3254908175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584009a70 con 0x7f658c15ab60 2024-09-13T17:18:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f658a800640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f6584004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f6582000640 1 -- 172.21.15.28:0/3254908175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658403c8f0 con 0x7f658c15ab60 2024-09-13T17:18:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f6582000640 1 -- 172.21.15.28:0/3254908175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658403ca50 con 0x7f658c15ab60 2024-09-13T17:18:52.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f6582000640 1 -- 172.21.15.28:0/3254908175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658403cde0 con 0x7f658c15ab60 2024-09-13T17:18:52.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c139890 con 0x7f658c15ab60 2024-09-13T17:18:52.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.007+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c1582b0 con 0x7f658c15ab60 2024-09-13T17:18:52.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.010+0000 7f6582000640 1 -- 172.21.15.28:0/3254908175 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6584031400 con 0x7f658c15ab60 2024-09-13T17:18:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.011+0000 7f6582000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 0x7f656c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.011+0000 7f6582000640 1 -- 172.21.15.28:0/3254908175 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65840aa800 con 0x7f658c15ab60 2024-09-13T17:18:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.011+0000 7f6589e00640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 0x7f656c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.011+0000 7f6593fe0740 1 Processor -- start 2024-09-13T17:18:52.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 -- start start 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6589e00640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 0x7f656c06d0f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f657c005880 tx=0x7f657c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 0x55f5d45f9130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d44b26a0 con 0x55f5d45a9420 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d4515cc0 con 0x55f5d45d3110 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.012+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5d4513450 con 0x55f5d45abc60 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 0x55f5d45f9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f658a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45ac040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:56020/0 (socket says 172.21.15.28:56020) 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589e00640 1 -- 172.21.15.28:0/2690144368 learned_addr learned my addr 172.21.15.28:0/2690144368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 -- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 msgr2=0x55f5d45ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 -- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 msgr2=0x55f5d45f9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 -- 172.21.15.28:0/2690144368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5d44df2e0 con 0x55f5d45a9420 2024-09-13T17:18:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.013+0000 7f6589400640 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 0x55f5d45f9130 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f658c071130 tx=0x7f65780019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5dd4adf6f752b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6576a00640 1 -- 172.21.15.28:0/2690144368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65780052e0 con 0x55f5d45a9420 2024-09-13T17:18:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6576a00640 1 -- 172.21.15.28:0/2690144368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578005440 con 0x55f5d45a9420 2024-09-13T17:18:52.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6576a00640 1 -- 172.21.15.28:0/2690144368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65780037f0 con 0x55f5d45a9420 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 -- 172.21.15.28:0/2690144368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 msgr2=0x55f5d45f9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 0x55f5d45f9130 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f658c071130 tx=0x7f65780019c0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 -- 172.21.15.28:0/2690144368 shutdown_connections 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45a9420 0x55f5d45f9130 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2690144368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 -- 172.21.15.28:0/2690144368 >> 172.21.15.28:0/2690144368 conn(0x55f5d45a8390 msgr2=0x55f5d45f0840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 -- 172.21.15.28:0/2690144368 shutdown_connections 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.014+0000 7f6593fe0740 1 -- 172.21.15.28:0/2690144368 wait complete. 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.015+0000 7f6593fe0740 1 -- start start 2024-09-13T17:18:52.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.015+0000 7f6593fe0740 1 Processor -- start 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45d3110 0x55f5d45f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d4d60 0x55f5d45f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d450ed40 con 0x55f5d45abc60 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d44b0280 con 0x55f5d45d4d60 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5d44aff20 con 0x55f5d45d3110 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6593fe0740 10 client.0 _ll_register_callbacks cb 0x55f5d45083b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6589e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.016+0000 7f6589e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38622/0 (socket says 172.21.15.28:38622) 2024-09-13T17:18:52.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 -- 172.21.15.28:0/763042871 learned_addr learned my addr 172.21.15.28:0/763042871 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f658a800640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d4d60 0x55f5d45f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589400640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45d3110 0x55f5d45f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 -- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45d3110 msgr2=0x55f5d45f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45d3110 0x55f5d45f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 -- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d4d60 msgr2=0x55f5d45f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d4d60 0x55f5d45f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 -- 172.21.15.28:0/763042871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657800f040 con 0x55f5d45abc60 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f6589e00640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f657c007fd0 tx=0x7f657c0010a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da16ea62c498fad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f656b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.017+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657c007560 con 0x55f5d45abc60 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f657c0076c0 con 0x55f5d45abc60 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5d42024e0 con 0x55f5d45abc60 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f657c013420 con 0x55f5d45abc60 2024-09-13T17:18:52.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 10 client.9636 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55f5d450d3c0 con 0x55f5d45abc60 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f5d42024e0 con 0x55f5d45abc60 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f657c020070 con 0x55f5d45abc60 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 10 client.9636 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 10 client.9636 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 20 client.9636 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45d1800 0x55f5d45d1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 4 client.9636 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.018+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5d42137e0 con 0x55f5d45d1800 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.019+0000 7f6589400640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45d1800 0x55f5d45d1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.019+0000 7f6589400640 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45d1800 0x55f5d45d1be0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=add85bedf7981c4f server_cookie=28aead65a539bcde in_seq=0 out_seq=0 2024-09-13T17:18:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.019+0000 7f656b400640 10 client.9636 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f656b400640 1 -- 172.21.15.28:0/763042871 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55f5d42137e0 con 0x55f5d45d1800 2024-09-13T17:18:52.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f656b400640 10 client.9636 handle_command_reply: tid=0 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 client.9636 shutdown 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 msgr2=0x55f5d45d4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f657c007fd0 tx=0x7f657c0010a0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 shutdown_connections 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45d1800 0x55f5d45d1be0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d4d60 0x55f5d45f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45d3110 0x55f5d45f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 --2- 172.21.15.28:0/763042871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45abc60 0x55f5d45d4820 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f656b400640 0 client.9636 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 >> 172.21.15.28:0/763042871 conn(0x55f5d45a9420 msgr2=0x55f5d45efda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 shutdown_connections 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 1 -- 172.21.15.28:0/763042871 wait complete. 2024-09-13T17:18:52.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.020+0000 7f6593fe0740 20 client.9636 trim_cache size 0 max 16384 2024-09-13T17:18:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.035+0000 7f6593fe0740 1 Processor -- start 2024-09-13T17:18:52.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.035+0000 7f6593fe0740 1 -- start start 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45f6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d4276040 con 0x55f5d45d4d60 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d45f2610 con 0x55f5d45d3110 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5d45d1e90 con 0x55f5d45abc60 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38638/0 (socket says 172.21.15.28:38638) 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.036+0000 7f658a800640 1 -- 172.21.15.28:0/2605148475 learned_addr learned my addr 172.21.15.28:0/2605148475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6589e00640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45f6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6589400640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 -- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 msgr2=0x55f5d45f6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45f6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 -- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 msgr2=0x55f5d45f3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 -- 172.21.15.28:0/2605148475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5d44af610 con 0x55f5d45d4d60 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6589e00640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6589400640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:52.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f658a800640 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f6584075b90 tx=0x7f6584075a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231223faeba83d2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6577400640 1 -- 172.21.15.28:0/2605148475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840910d0 con 0x55f5d45d4d60 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6577400640 1 -- 172.21.15.28:0/2605148475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6584081a80 con 0x55f5d45d4d60 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.037+0000 7f6577400640 1 -- 172.21.15.28:0/2605148475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6584075da0 con 0x55f5d45d4d60 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 -- 172.21.15.28:0/2605148475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 msgr2=0x55f5d45f4240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f6584075b90 tx=0x7f6584075a50 comp rx=0 tx=0).stop 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 -- 172.21.15.28:0/2605148475 shutdown_connections 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d45f4240 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d45f3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 --2- 172.21.15.28:0/2605148475 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45f6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 -- 172.21.15.28:0/2605148475 >> 172.21.15.28:0/2605148475 conn(0x55f5d45a9420 msgr2=0x55f5d45f28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 -- 172.21.15.28:0/2605148475 shutdown_connections 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.038+0000 7f6593fe0740 1 -- 172.21.15.28:0/2605148475 wait complete. 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.039+0000 7f6593fe0740 1 -- start start 2024-09-13T17:18:52.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.039+0000 7f6593fe0740 1 Processor -- start 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45d34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d465e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d465ee80 con 0x55f5d45d4d60 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5d4640d80 con 0x55f5d45d3110 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f5d45d3d80 con 0x55f5d45abc60 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f6593fe0740 10 client.0 _ll_register_callbacks cb 0x7f657c001de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38652/0 (socket says 172.21.15.28:38652) 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 -- 172.21.15.28:0/3235433065 learned_addr learned my addr 172.21.15.28:0/3235433065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 -- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 msgr2=0x55f5d45d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45d34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 -- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 msgr2=0x55f5d465e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d465e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.040+0000 7f658a800640 1 -- 172.21.15.28:0/3235433065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c1582b0 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f658a800640 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f65840779e0 tx=0x7f6584009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6320249a1af8e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658407d2b0 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65840798f0 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6578005440 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6584079ad0 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 10 client.9642 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55f5d44d4b20 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578005440 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6584038070 con 0x55f5d45d4d60 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 10 client.9642 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:52.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 10 client.9642 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 20 client.9642 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45ef370 0x55f5d45ef780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 4 client.9642 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:18:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.041+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f5d4266e30 con 0x55f5d45ef370 2024-09-13T17:18:52.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f6589e00640 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45ef370 0x55f5d45ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f6589e00640 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45ef370 0x55f5d45ef780 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3f397548addb7e2 server_cookie=1b94bc9a1581f21b in_seq=0 out_seq=0 2024-09-13T17:18:52.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f656b400640 10 client.9642 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:52.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f656b400640 1 -- 172.21.15.28:0/3235433065 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x55f5d4266e30 con 0x55f5d45ef370 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f656b400640 10 client.9642 handle_command_reply: tid=0 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.042+0000 7f6593fe0740 1 client.9642 shutdown 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 msgr2=0x55f5d4640840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f65840779e0 tx=0x7f6584009750 comp rx=0 tx=0).stop 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 shutdown_connections 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f5d45ef370 0x55f5d45ef780 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f5d45d4d60 0x55f5d4640840 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f5d45d3110 0x55f5d465e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 --2- 172.21.15.28:0/3235433065 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f5d45abc60 0x55f5d45d34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 >> 172.21.15.28:0/3235433065 conn(0x55f5d45a9420 msgr2=0x55f5d45f4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f656b400640 0 client.9642 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 shutdown_connections 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 1 -- 172.21.15.28:0/3235433065 wait complete. 2024-09-13T17:18:52.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6593fe0740 20 client.9642 trim_cache size 0 max 16384 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "f3d2c57d-c36b-4c28-bf19-d2110b4fd507": { 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_control_test_path", 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "f3d2c57d-c36b-4c28-bf19-d2110b4fd507", 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:18:52.050 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 msgr2=0x7f656c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 0x7f656c06d0f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f657c005880 tx=0x7f657c009040 comp rx=0 tx=0).stop 2024-09-13T17:18:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 msgr2=0x7f658c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.043+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f6584009d90 tx=0x7f6584004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 shutdown_connections 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f656c06ac30 0x7f656c06d0f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f658c15ab60 0x7f658c0ac850 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f658c15a780 0x7f658c135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 --2- 172.21.15.28:0/3254908175 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f658c071490 0x7f658c139350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 >> 172.21.15.28:0/3254908175 conn(0x7f658c0b0270 msgr2=0x7f658c0cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 shutdown_connections 2024-09-13T17:18:52.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.044+0000 7f6591000640 1 -- 172.21.15.28:0/3254908175 wait complete. 2024-09-13T17:18:52.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.215+0000 7f1b7a200640 1 Processor -- start 2024-09-13T17:18:52.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.215+0000 7f1b7a200640 1 -- start start 2024-09-13T17:18:52.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b7a200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b7415a540 0x7f1b7406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b7a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b740cabc0 0x7f1b7415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b7a200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7415e100 con 0x7f1b7415edb0 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b7a200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b740adb00 con 0x7f1b7415a540 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b7a200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7415e3e0 con 0x7f1b740cabc0 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b73400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b7415a540 0x7f1b7406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b73e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b7415edb0 0x7f1b74070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44294/0 (socket says 172.21.15.28:44294) 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.216+0000 7f1b73e00640 1 -- 172.21.15.28:0/4237165584 learned_addr learned my addr 172.21.15.28:0/4237165584 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b6aa00640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b740cabc0 0x7f1b7415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b73400640 1 -- 172.21.15.28:0/4237165584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b740cabc0 msgr2=0x7f1b7415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b73400640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b740cabc0 0x7f1b7415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b73400640 1 -- 172.21.15.28:0/4237165584 >> v1:172.21.15.28:6789/0 conn(0x7f1b7415edb0 legacy=0x7f1b74070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b73400640 1 -- 172.21.15.28:0/4237165584 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7406ce80 con 0x7f1b7415a540 2024-09-13T17:18:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.217+0000 7f1b72a00640 1 -- 172.21.15.28:0/4237165584 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44242142 0 0) 0x7f1b7415e100 con 0x7f1b7415edb0 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b73400640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7415a540 0x7f1b7406c430 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f1b640097a0 tx=0x7f1b6402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a53c5634a3907f5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b72a00640 1 -- 172.21.15.28:0/4237165584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6402ae50 con 0x7f1b7415a540 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b72a00640 1 -- 172.21.15.28:0/4237165584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64032440 con 0x7f1b7415a540 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b72a00640 1 -- 172.21.15.28:0/4237165584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640327d0 con 0x7f1b7415a540 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b7a200640 1 -- 172.21.15.28:0/4237165584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7415a540 msgr2=0x7f1b7406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.218+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7415a540 0x7f1b7406c430 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f1b640097a0 tx=0x7f1b6402a100 comp rx=0 tx=0).stop 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 -- 172.21.15.28:0/4237165584 shutdown_connections 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b740cabc0 0x7f1b7415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4237165584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7415a540 0x7f1b7406c430 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 -- 172.21.15.28:0/4237165584 >> 172.21.15.28:0/4237165584 conn(0x7f1b74069e80 msgr2=0x7f1b7406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 -- 172.21.15.28:0/4237165584 shutdown_connections 2024-09-13T17:18:52.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.219+0000 7f1b7a200640 1 -- 172.21.15.28:0/4237165584 wait complete. 2024-09-13T17:18:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.220+0000 7f1b7a200640 1 Processor -- start 2024-09-13T17:18:52.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.220+0000 7f1b7a200640 1 -- start start 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b7415a540 0x7f1b7414d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7414ded0 0x7f1b7415cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741599d0 con 0x7f1b740cabc0 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7415c710 con 0x7f1b7415a540 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b7a200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741414b0 con 0x7f1b7414ded0 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.221+0000 7f1b73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38682/0 (socket says 172.21.15.28:38682) 2024-09-13T17:18:52.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 -- 172.21.15.28:0/4252807004 learned_addr learned my addr 172.21.15.28:0/4252807004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b6aa00640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b7415a540 0x7f1b7414d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73e00640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7414ded0 0x7f1b7415cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 -- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7414ded0 msgr2=0x7f1b7415cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7414ded0 0x7f1b7415cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 -- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b7415a540 msgr2=0x7f1b7414d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b7415a540 0x7f1b7414d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 -- 172.21.15.28:0/4252807004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009400 con 0x7f1b740cabc0 2024-09-13T17:18:52.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b73400640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f1b64035040 tx=0x7f1b640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.222+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6403d710 con 0x7f1b740cabc0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.223+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6403d890 con 0x7f1b740cabc0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.223+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6403dc40 con 0x7f1b740cabc0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.223+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b74159150 con 0x7f1b740cabc0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.223+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b741596f0 con 0x7f1b740cabc0 2024-09-13T17:18:52.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.224+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7406beb0 con 0x7f1b740cabc0 2024-09-13T17:18:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.225+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1b64039040 con 0x7f1b740cabc0 2024-09-13T17:18:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.225+0000 7f1b6be00640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 0x7f1b5006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.225+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b640a99d0 con 0x7f1b740cabc0 2024-09-13T17:18:52.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.226+0000 7f1b6aa00640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 0x7f1b5006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.226+0000 7f1b6aa00640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 0x7f1b5006d0f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1b7415db00 tx=0x7f1b5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.230+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1b64074d70 con 0x7f1b740cabc0 2024-09-13T17:18:52.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.443+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b740b0ba0 con 0x7f1b740cabc0 2024-09-13T17:18:52.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.443+0000 7f1b6be00640 1 -- 172.21.15.28:0/4252807004 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f1b64078390 con 0x7f1b740cabc0 2024-09-13T17:18:52.446 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:52.446 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:52.446 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 msgr2=0x7f1b5006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 0x7f1b5006d0f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1b7415db00 tx=0x7f1b5c008040 comp rx=0 tx=0).stop 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 msgr2=0x7f1b741416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7f1b64035040 tx=0x7f1b640046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 shutdown_connections 2024-09-13T17:18:52.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5006ac30 0x7f1b5006d0f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7414ded0 0x7f1b7415cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b7415a540 0x7f1b7414d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 --2- 172.21.15.28:0/4252807004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b740cabc0 0x7f1b741416f0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 >> 172.21.15.28:0/4252807004 conn(0x7f1b74069e80 msgr2=0x7f1b7406c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 shutdown_connections 2024-09-13T17:18:52.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.446+0000 7f1b7a200640 1 -- 172.21.15.28:0/4252807004 wait complete. 2024-09-13T17:18:52.464 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:18:52.464 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:18:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.617+0000 7fae0b800640 1 Processor -- start 2024-09-13T17:18:52.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.617+0000 7fae0b800640 1 -- start start 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae0b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae04161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae0b800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae040713a0 0x7fae0406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae0b800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae040b35c0 con 0x7fae0406cde0 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae0b800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae041618f0 con 0x7fae040713a0 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae0b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040b38a0 con 0x7fae0415e860 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.618+0000 7fae09800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae040713a0 0x7fae0406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae040713a0 0x7fae0406b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38684/0 (socket says 172.21.15.28:38684) 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 -- 172.21.15.28:0/3545216216 learned_addr learned my addr 172.21.15.28:0/3545216216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae08e00640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae04161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 -- 172.21.15.28:0/3545216216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 msgr2=0x7fae04161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae04161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 -- 172.21.15.28:0/3545216216 >> v1:172.21.15.189:6789/0 conn(0x7fae0406cde0 legacy=0x7fae04070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:52.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.619+0000 7fae09800640 1 -- 172.21.15.28:0/3545216216 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae04161230 con 0x7fae040713a0 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.620+0000 7fae09800640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae040713a0 0x7fae0406b1c0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009e50 tx=0x7fadfc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5986876db858ccd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.620+0000 7fae03e00640 1 -- 172.21.15.28:0/3545216216 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc02aea0 con 0x7fae040713a0 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.620+0000 7fae03e00640 1 -- 172.21.15.28:0/3545216216 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadfc02d050 con 0x7fae040713a0 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.620+0000 7fae03e00640 1 -- 172.21.15.28:0/3545216216 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadfc031820 con 0x7fae040713a0 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 -- 172.21.15.28:0/3545216216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae040713a0 msgr2=0x7fae0406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae040713a0 0x7fae0406b1c0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fadfc009e50 tx=0x7fadfc02a680 comp rx=0 tx=0).stop 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 -- 172.21.15.28:0/3545216216 shutdown_connections 2024-09-13T17:18:52.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae040713a0 0x7fae0406b1c0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 --2- 172.21.15.28:0/3545216216 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae04161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 -- 172.21.15.28:0/3545216216 >> 172.21.15.28:0/3545216216 conn(0x7fae04069e80 msgr2=0x7fae0406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 -- 172.21.15.28:0/3545216216 shutdown_connections 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.621+0000 7fae0b800640 1 -- 172.21.15.28:0/3545216216 wait complete. 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.623+0000 7fae0b800640 1 Processor -- start 2024-09-13T17:18:52.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.623+0000 7fae0b800640 1 -- start start 2024-09-13T17:18:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae040713a0 0x7fae040b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae040aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040ae270 con 0x7fae0406cde0 2024-09-13T17:18:52.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae040aa900 con 0x7fae040713a0 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae0b800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae040af9d0 con 0x7fae0415e860 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38700/0 (socket says 172.21.15.28:38700) 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 -- 172.21.15.28:0/256227491 learned_addr learned my addr 172.21.15.28:0/256227491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fadfbe00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae040713a0 0x7fae040b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae09800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae040aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 -- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 msgr2=0x7fae040aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae040aac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 -- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae040713a0 msgr2=0x7fae040b0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae040713a0 0x7fae040b0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.624+0000 7fae08e00640 1 -- 172.21.15.28:0/256227491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadfc009b30 con 0x7fae0406cde0 2024-09-13T17:18:52.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae08e00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d800 tx=0x7fadf400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400b6c0 con 0x7fae0406cde0 2024-09-13T17:18:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf400b820 con 0x7fae0406cde0 2024-09-13T17:18:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf400bbb0 con 0x7fae0406cde0 2024-09-13T17:18:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae040afcb0 con 0x7fae0406cde0 2024-09-13T17:18:52.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.625+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae040b0040 con 0x7fae0406cde0 2024-09-13T17:18:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.626+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040cdcd0 con 0x7fae0406cde0 2024-09-13T17:18:52.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.627+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fadf400bd10 con 0x7fae0406cde0 2024-09-13T17:18:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.628+0000 7fae01600640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 0x7fade006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.628+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadf4089910 con 0x7fae0406cde0 2024-09-13T17:18:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.628+0000 7fadfbe00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 0x7fade006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.629+0000 7fadfbe00640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 0x7fade006d0c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fae041492b0 tx=0x7fadec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:52.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.632+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fadf4054ae0 con 0x7fae0406cde0 2024-09-13T17:18:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.878+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fae040b31c0 con 0x7fae0406cde0 2024-09-13T17:18:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.879+0000 7fae01600640 1 -- 172.21.15.28:0/256227491 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1446 (secure 0 0 0) 0x7fadf4058100 con 0x7fae0406cde0 2024-09-13T17:18:52.881 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:52.881 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":666,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":153,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":860,"data_bytes":878622232,"bytes_used":2276442112,"bytes_avail":770817671168,"bytes_total":773094113280,"write_bytes_sec":49117848,"read_op_per_sec":0,"write_op_per_sec":47},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-09-13T17:18:19.039587+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:18:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 msgr2=0x7fade006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 0x7fade006d0c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fae041492b0 tx=0x7fadec009040 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 msgr2=0x7fae041f5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fadf400d800 tx=0x7fadf400dcc0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 shutdown_connections 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fade006ac00 0x7fade006d0c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae0415e860 0x7fae040aac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae040713a0 0x7fae040b0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 --2- 172.21.15.28:0/256227491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae0406cde0 0x7fae041f5d60 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.881+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 >> 172.21.15.28:0/256227491 conn(0x7fae04069e80 msgr2=0x7fae040b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.882+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 shutdown_connections 2024-09-13T17:18:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:52.882+0000 7fae0b800640 1 -- 172.21.15.28:0/256227491 wait complete. 2024-09-13T17:18:52.899 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:18:53.900 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:54.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.064+0000 7f8cd5000640 1 Processor -- start 2024-09-13T17:18:54.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.065+0000 7f8cd5000640 1 -- start start 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cd5000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8cd015d340 0x7f8cd006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cd5000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd00b3970 con 0x7f8cd006f980 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cd5000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd006fdb0 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cd5000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd0159230 con 0x7f8cd015d340 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cc5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8cd015d340 0x7f8cd006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cc5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8cd015d340 0x7f8cd006f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49518/0 (socket says 172.21.15.28:49518) 2024-09-13T17:18:54.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cc5400640 1 -- 172.21.15.28:0/4001540191 learned_addr learned my addr 172.21.15.28:0/4001540191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.066+0000 7f8cce800640 1 --1- 172.21.15.28:0/4001540191 >> v1:172.21.15.28:6789/0 conn(0x7f8cd006f980 0x7f8cd006b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58586/0 (socket says 172.21.15.28:58586) 2024-09-13T17:18:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.067+0000 7f8cc5400640 1 -- 172.21.15.28:0/4001540191 >> v1:172.21.15.189:6789/0 conn(0x7f8cd00cdfa0 legacy=0x7f8cd00b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.067+0000 7f8cc5400640 1 -- 172.21.15.28:0/4001540191 >> v1:172.21.15.28:6789/0 conn(0x7f8cd006f980 legacy=0x7f8cd006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.067+0000 7f8cc5400640 1 -- 172.21.15.28:0/4001540191 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0159430 con 0x7f8cd015d340 2024-09-13T17:18:54.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8cc5400640 1 --2- 172.21.15.28:0/4001540191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd015d340 0x7f8cd006f440 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800bb00 tx=0x7f8cb802c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e08abc7be6a7747 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8ccd400640 1 -- 172.21.15.28:0/4001540191 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8032480 con 0x7f8cd015d340 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8ccd400640 1 -- 172.21.15.28:0/4001540191 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb802f030 con 0x7f8cd015d340 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8ccd400640 1 -- 172.21.15.28:0/4001540191 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8032800 con 0x7f8cd015d340 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8cd5000640 1 -- 172.21.15.28:0/4001540191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd015d340 msgr2=0x7f8cd006f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.068+0000 7f8cd5000640 1 --2- 172.21.15.28:0/4001540191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd015d340 0x7f8cd006f440 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f8cb800bb00 tx=0x7f8cb802c790 comp rx=0 tx=0).stop 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.069+0000 7f8cd5000640 1 -- 172.21.15.28:0/4001540191 shutdown_connections 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.069+0000 7f8cd5000640 1 --2- 172.21.15.28:0/4001540191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd015d340 0x7f8cd006f440 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.069+0000 7f8cd5000640 1 -- 172.21.15.28:0/4001540191 >> 172.21.15.28:0/4001540191 conn(0x7f8cd0069e80 msgr2=0x7f8cd006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.069+0000 7f8cd5000640 1 -- 172.21.15.28:0/4001540191 shutdown_connections 2024-09-13T17:18:54.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.069+0000 7f8cd5000640 1 -- 172.21.15.28:0/4001540191 wait complete. 2024-09-13T17:18:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.070+0000 7f8cd5000640 1 Processor -- start 2024-09-13T17:18:54.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.070+0000 7f8cd5000640 1 -- start start 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd006f980 0x7f8cd013e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8cd015d340 0x7f8cd0154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0156ba0 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0155650 con 0x7f8cd006f980 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cd5000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd0154770 con 0x7f8cd015d340 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.071+0000 7f8cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49522/0 (socket says 172.21.15.28:49522) 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 -- 172.21.15.28:0/2212152616 learned_addr learned my addr 172.21.15.28:0/2212152616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cce800640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8cd015d340 0x7f8cd0154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8ccde00640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd006f980 0x7f8cd013e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 -- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8cd015d340 msgr2=0x7f8cd0154230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8cd015d340 0x7f8cd0154230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 -- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd006f980 msgr2=0x7f8cd013e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd006f980 0x7f8cd013e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 -- 172.21.15.28:0/2212152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb800b760 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.072+0000 7f8cc5400640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c00 tx=0x7f8cb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.073+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8035820 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.073+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb8035980 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.073+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb8035d30 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.073+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd0154a40 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.073+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd0155000 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.074+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd0155800 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.075+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8cb803c040 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.076+0000 7f8cc6a00640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 0x7f8cac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.076+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8cb80ab9d0 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.076+0000 7f8ccde00640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 0x7f8cac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.077+0000 7f8ccde00640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 0x7f8cac06d0c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000f60 tx=0x7f8cc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.080+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8cd0155800 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.313+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cd0155800 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.313+0000 7f8cc6a00640 1 -- 172.21.15.28:0/2212152616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f8cd0155800 con 0x7f8cd00cdfa0 2024-09-13T17:18:54.315 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:54.315 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:54.316 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 msgr2=0x7f8cac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 0x7f8cac06d0c0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f8cc0000f60 tx=0x7f8cc0009040 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 msgr2=0x7f8cd0156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f8cb8000c00 tx=0x7f8cb80046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 shutdown_connections 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8cac06ac00 0x7f8cac06d0c0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8cd015d340 0x7f8cd0154230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8cd00cdfa0 0x7f8cd0156660 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 --2- 172.21.15.28:0/2212152616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8cd006f980 0x7f8cd013e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 >> 172.21.15.28:0/2212152616 conn(0x7f8cd0069e80 msgr2=0x7f8cd0152620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 shutdown_connections 2024-09-13T17:18:54.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.315+0000 7f8cd5000640 1 -- 172.21.15.28:0/2212152616 wait complete. 2024-09-13T17:18:54.332 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:18:54.332 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:18:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.485+0000 7f391e000640 1 Processor -- start 2024-09-13T17:18:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.485+0000 7f391e000640 1 -- start start 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39181592a0 0x7f391815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f391806f970 0x7f3918159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f391806c2d0 0x7f391806c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391815b0b0 con 0x7f39181592a0 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391806cca0 con 0x7f391806c2d0 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.486+0000 7f391e000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39180b2040 con 0x7f391806f970 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39181592a0 0x7f391815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f39181592a0 0x7f391815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49536/0 (socket says 172.21.15.28:49536) 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 -- 172.21.15.28:0/3986497141 learned_addr learned my addr 172.21.15.28:0/3986497141 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917e00640 1 --2- 172.21.15.28:0/3986497141 >> v2:172.21.15.189:3300/0 conn(0x7f391806c2d0 0x7f391806c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3916a00640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f391806f970 0x7f3918159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 -- 172.21.15.28:0/3986497141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f391806f970 msgr2=0x7f3918159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f391806f970 0x7f3918159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 -- 172.21.15.28:0/3986497141 >> v2:172.21.15.189:3300/0 conn(0x7f391806c2d0 msgr2=0x7f391806c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 --2- 172.21.15.28:0/3986497141 >> v2:172.21.15.189:3300/0 conn(0x7f391806c2d0 0x7f391806c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.487+0000 7f3917400640 1 -- 172.21.15.28:0/3986497141 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3918159ff0 con 0x7f39181592a0 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.488+0000 7f3917400640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39181592a0 0x7f391815ab70 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f391806f700 tx=0x7f390802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a120e60c6a8ed0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.488+0000 7f3916000640 1 -- 172.21.15.28:0/3986497141 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908032440 con 0x7f39181592a0 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.488+0000 7f3916000640 1 -- 172.21.15.28:0/3986497141 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39080325a0 con 0x7f39181592a0 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.488+0000 7f3916000640 1 -- 172.21.15.28:0/3986497141 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908032930 con 0x7f39181592a0 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 -- 172.21.15.28:0/3986497141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39181592a0 msgr2=0x7f391815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39181592a0 0x7f391815ab70 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f391806f700 tx=0x7f390802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 -- 172.21.15.28:0/3986497141 shutdown_connections 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 --2- 172.21.15.28:0/3986497141 >> v2:172.21.15.189:3300/0 conn(0x7f391806c2d0 0x7f391806c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f391806f970 0x7f3918159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 --2- 172.21.15.28:0/3986497141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39181592a0 0x7f391815ab70 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 -- 172.21.15.28:0/3986497141 >> 172.21.15.28:0/3986497141 conn(0x7f3918069e80 msgr2=0x7f391806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 -- 172.21.15.28:0/3986497141 shutdown_connections 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.489+0000 7f391e000640 1 -- 172.21.15.28:0/3986497141 wait complete. 2024-09-13T17:18:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.491+0000 7f391e000640 1 Processor -- start 2024-09-13T17:18:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.491+0000 7f391e000640 1 -- start start 2024-09-13T17:18:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 0x7f39180cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39181592a0 0x7f39180a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39180ccd20 con 0x7f391806c2d0 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39180cc0a0 con 0x7f391806f970 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f391e000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39180cc380 con 0x7f39181592a0 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49552/0 (socket says 172.21.15.28:49552) 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 -- 172.21.15.28:0/1261904218 learned_addr learned my addr 172.21.15.28:0/1261904218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3916a00640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 0x7f39180cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917e00640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39181592a0 0x7f39180a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 -- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39181592a0 msgr2=0x7f39180a8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39181592a0 0x7f39180a8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 -- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 msgr2=0x7f39180cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 0x7f39180cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3917400640 1 -- 172.21.15.28:0/1261904218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080094d0 con 0x7f391806c2d0 2024-09-13T17:18:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.492+0000 7f3916a00640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 0x7f39180cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f3917400640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f390802a220 tx=0x7f39080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3908032440 con 0x7f391806c2d0 2024-09-13T17:18:54.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3908030ba0 con 0x7f391806c2d0 2024-09-13T17:18:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390803d440 con 0x7f391806c2d0 2024-09-13T17:18:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39180cc660 con 0x7f391806c2d0 2024-09-13T17:18:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.493+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39180a9bf0 con 0x7f391806c2d0 2024-09-13T17:18:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.494+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39180cce60 con 0x7f391806c2d0 2024-09-13T17:18:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.495+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3908039040 con 0x7f391806c2d0 2024-09-13T17:18:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.496+0000 7f390f400640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.496+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f39080a9ec0 con 0x7f391806c2d0 2024-09-13T17:18:54.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.496+0000 7f3916a00640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:54.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.497+0000 7f3916a00640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 0x7f38f806d0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3918152e50 tx=0x7f3904008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.500+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f39180cce60 con 0x7f391806c2d0 2024-09-13T17:18:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.755+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f39180cce60 con 0x7f391806c2d0 2024-09-13T17:18:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.756+0000 7f390f400640 1 -- 172.21.15.28:0/1261904218 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1448 (secure 0 0 0) 0x7f39080375c0 con 0x7f391806c2d0 2024-09-13T17:18:54.758 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:54.758 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":668,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076362607,"bytes_used":2698141696,"bytes_avail":770395971584,"bytes_total":773094113280,"write_bytes_sec":65596216,"read_op_per_sec":0,"write_op_per_sec":63},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-09-13T17:18:19.039587+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:18:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 msgr2=0x7f38f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 0x7f38f806d0c0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3918152e50 tx=0x7f3904008040 comp rx=0 tx=0).stop 2024-09-13T17:18:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 msgr2=0x7f3918152110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f390802a220 tx=0x7f39080046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 shutdown_connections 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f39181592a0 0x7f39180a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f391806f970 0x7f39180cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 --2- 172.21.15.28:0/1261904218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f391806c2d0 0x7f3918152110 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 >> 172.21.15.28:0/1261904218 conn(0x7f3918069e80 msgr2=0x7f39180ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 shutdown_connections 2024-09-13T17:18:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:54.758+0000 7f391e000640 1 -- 172.21.15.28:0/1261904218 wait complete. 2024-09-13T17:18:54.775 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:18:55.777 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:55.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.942+0000 7fe327c00640 1 Processor -- start 2024-09-13T17:18:55.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.942+0000 7fe327c00640 1 -- start start 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe327c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 0x7fe32015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe327c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe320159d90 con 0x7fe3201592a0 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe327c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3200b2040 con 0x7fe32006f970 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe327c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015b0b0 con 0x7fe32006c9e0 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe325200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3201592a0 0x7fe320159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58598/0 (socket says 172.21.15.28:58598) 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe325200640 1 -- 172.21.15.28:0/2239286632 learned_addr learned my addr 172.21.15.28:0/2239286632 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:55.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe325c00640 1 --2- 172.21.15.28:0/2239286632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 0x7fe32015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978363400 0 0) 0x7fe320159d90 con 0x7fe3201592a0 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.944+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe304003520 con 0x7fe3201592a0 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.945+0000 7fe325c00640 1 -- 172.21.15.28:0/2239286632 >> v1:172.21.15.189:6789/0 conn(0x7fe32006f970 legacy=0x7fe32006c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.945+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232459581 0 0) 0x7fe304003520 con 0x7fe3201592a0 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.945+0000 7fe325c00640 1 -- 172.21.15.28:0/2239286632 >> v1:172.21.15.28:6789/0 conn(0x7fe3201592a0 legacy=0x7fe320159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.945+0000 7fe325c00640 1 -- 172.21.15.28:0/2239286632 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320159ff0 con 0x7fe32006c9e0 2024-09-13T17:18:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.946+0000 7fe325c00640 1 --2- 172.21.15.28:0/2239286632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 0x7fe32015ab70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe318009dc0 tx=0x7fe31802a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ced1bc6f9c10eb5d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.946+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310001690 con 0x7fe32006c9e0 2024-09-13T17:18:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.946+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31802d050 con 0x7fe32006c9e0 2024-09-13T17:18:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.946+0000 7fe31fe00640 1 -- 172.21.15.28:0/2239286632 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe318007740 con 0x7fe32006c9e0 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 -- 172.21.15.28:0/2239286632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 msgr2=0x7fe32015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 --2- 172.21.15.28:0/2239286632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 0x7fe32015ab70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe318009dc0 tx=0x7fe31802a580 comp rx=0 tx=0).stop 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 -- 172.21.15.28:0/2239286632 shutdown_connections 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 --2- 172.21.15.28:0/2239286632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006c9e0 0x7fe32015ab70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 -- 172.21.15.28:0/2239286632 >> 172.21.15.28:0/2239286632 conn(0x7fe320069e80 msgr2=0x7fe32006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 -- 172.21.15.28:0/2239286632 shutdown_connections 2024-09-13T17:18:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.947+0000 7fe327c00640 1 -- 172.21.15.28:0/2239286632 wait complete. 2024-09-13T17:18:55.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.948+0000 7fe327c00640 1 Processor -- start 2024-09-13T17:18:55.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.948+0000 7fe327c00640 1 -- start start 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 0x7fe3200b01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3201592a0 0x7fe3200cced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200ccab0 con 0x7fe32006c9e0 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200cbe30 con 0x7fe3201592a0 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.949+0000 7fe327c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200cc110 con 0x7fe32006f970 2024-09-13T17:18:55.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49560/0 (socket says 172.21.15.28:49560) 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 -- 172.21.15.28:0/1177207930 learned_addr learned my addr 172.21.15.28:0/1177207930 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3201592a0 0x7fe3200cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe317e00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 0x7fe3200b01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 -- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 msgr2=0x7fe3200b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 0x7fe3200b01e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 -- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3201592a0 msgr2=0x7fe3200cced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3201592a0 0x7fe3200cced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 -- 172.21.15.28:0/1177207930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe318009a70 con 0x7fe32006c9e0 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe317e00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 0x7fe3200b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:55.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.950+0000 7fe325200640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe31000c780 tx=0x7fe31000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.951+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100074e0 con 0x7fe32006c9e0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.951+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe310007660 con 0x7fe32006c9e0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.951+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3200cc3f0 con 0x7fe32006c9e0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.951+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe310004a20 con 0x7fe32006c9e0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.951+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3200cba20 con 0x7fe32006c9e0 2024-09-13T17:18:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.952+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3200ccbf0 con 0x7fe32006c9e0 2024-09-13T17:18:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.953+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe310004b80 con 0x7fe32006c9e0 2024-09-13T17:18:55.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.954+0000 7fe31d600640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 0x7fe2fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:55.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.954+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fe3100889c0 con 0x7fe32006c9e0 2024-09-13T17:18:55.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.954+0000 7fe317e00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 0x7fe2fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:55.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.955+0000 7fe317e00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 0x7fe2fc06d0c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe3201f5af0 tx=0x7fe308009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:55.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:55.958+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe3200ccbf0 con 0x7fe32006c9e0 2024-09-13T17:18:56.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.177+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3200ccbf0 con 0x7fe32006c9e0 2024-09-13T17:18:56.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.177+0000 7fe31d600640 1 -- 172.21.15.28:0/1177207930 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fe3200ccbf0 con 0x7fe32006c9e0 2024-09-13T17:18:56.179 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:56.179 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:56.180 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 msgr2=0x7fe2fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 0x7fe2fc06d0c0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fe3201f5af0 tx=0x7fe308009040 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 msgr2=0x7fe3200cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fe31000c780 tx=0x7fe31000cc40 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 shutdown_connections 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2fc06ac00 0x7fe2fc06d0c0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3201592a0 0x7fe3200cced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32006f970 0x7fe3200b01e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 --2- 172.21.15.28:0/1177207930 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32006c9e0 0x7fe3200cb4e0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 >> 172.21.15.28:0/1177207930 conn(0x7fe320069e80 msgr2=0x7fe3200acfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:56.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 shutdown_connections 2024-09-13T17:18:56.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.180+0000 7fe327c00640 1 -- 172.21.15.28:0/1177207930 wait complete. 2024-09-13T17:18:56.198 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:18:56.198 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:18:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.352+0000 7f56ffe00640 1 Processor -- start 2024-09-13T17:18:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.352+0000 7f56ffe00640 1 -- start start 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f57000b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5700159820 0x7f57000b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57000b22e0 0x7f57000b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57000c3850 con 0x7f57000b22e0 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570015a930 con 0x7f5700159820 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ffe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57000b2cd0 con 0x7f5700159440 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57000b22e0 0x7f57000b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f57000b22e0 0x7f57000b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49572/0 (socket says 172.21.15.28:49572) 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.353+0000 7f56ff400640 1 -- 172.21.15.28:0/526583451 learned_addr learned my addr 172.21.15.28:0/526583451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56f6000640 1 --2- 172.21.15.28:0/526583451 >> v2:172.21.15.189:3300/0 conn(0x7f5700159820 0x7f57000b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56fea00640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f57000b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56ff400640 1 -- 172.21.15.28:0/526583451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 msgr2=0x7f57000b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56ff400640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f57000b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56ff400640 1 -- 172.21.15.28:0/526583451 >> v2:172.21.15.189:3300/0 conn(0x7f5700159820 msgr2=0x7f57000b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56ff400640 1 --2- 172.21.15.28:0/526583451 >> v2:172.21.15.189:3300/0 conn(0x7f5700159820 0x7f57000b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.354+0000 7f56ff400640 1 -- 172.21.15.28:0/526583451 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5700159d20 con 0x7f57000b22e0 2024-09-13T17:18:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.355+0000 7f56ff400640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57000b22e0 0x7f57000b2790 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f57000b10b0 tx=0x7f56f802c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337d81529b755176 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.355+0000 7f56fe000640 1 -- 172.21.15.28:0/526583451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8032430 con 0x7f57000b22e0 2024-09-13T17:18:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.355+0000 7f56fe000640 1 -- 172.21.15.28:0/526583451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8032590 con 0x7f57000b22e0 2024-09-13T17:18:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.355+0000 7f56fe000640 1 -- 172.21.15.28:0/526583451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8032920 con 0x7f57000b22e0 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 -- 172.21.15.28:0/526583451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57000b22e0 msgr2=0x7f57000b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57000b22e0 0x7f57000b2790 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f57000b10b0 tx=0x7f56f802c940 comp rx=0 tx=0).stop 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 -- 172.21.15.28:0/526583451 shutdown_connections 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57000b22e0 0x7f57000b2790 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 --2- 172.21.15.28:0/526583451 >> v2:172.21.15.189:3300/0 conn(0x7f5700159820 0x7f57000b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 --2- 172.21.15.28:0/526583451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f57000b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 -- 172.21.15.28:0/526583451 >> 172.21.15.28:0/526583451 conn(0x7f5700072e40 msgr2=0x7f5700073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 -- 172.21.15.28:0/526583451 shutdown_connections 2024-09-13T17:18:56.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.356+0000 7f56ffe00640 1 -- 172.21.15.28:0/526583451 wait complete. 2024-09-13T17:18:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.357+0000 7f56ffe00640 1 Processor -- start 2024-09-13T17:18:56.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 -- start start 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57000b22e0 0x7f570014dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f5700155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57000a8440 con 0x7f5700159820 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57001566a0 con 0x7f57000b22e0 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.358+0000 7f56ffe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5700156980 con 0x7f5700159440 2024-09-13T17:18:56.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49582/0 (socket says 172.21.15.28:49582) 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 -- 172.21.15.28:0/276089097 learned_addr learned my addr 172.21.15.28:0/276089097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56fea00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57000b22e0 0x7f570014dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56f6000640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f5700155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 -- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 msgr2=0x7f5700155320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f5700155320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 -- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57000b22e0 msgr2=0x7f570014dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57000b22e0 0x7f570014dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 -- 172.21.15.28:0/276089097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009b30 con 0x7f5700159820 2024-09-13T17:18:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.359+0000 7f56ff400640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f56f8034040 tx=0x7f56f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.360+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8032430 con 0x7f5700159820 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.360+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8040070 con 0x7f5700159820 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.360+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8007e10 con 0x7f5700159820 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.360+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700156c60 con 0x7f5700159820 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.360+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570014e520 con 0x7f5700159820 2024-09-13T17:18:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.361+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570014e6b0 con 0x7f5700159820 2024-09-13T17:18:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.362+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f56f803b400 con 0x7f5700159820 2024-09-13T17:18:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.363+0000 7f56f7400640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 0x7f56dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:56.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.363+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f56f8042070 con 0x7f5700159820 2024-09-13T17:18:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.363+0000 7f56fea00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 0x7f56dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.364+0000 7f56fea00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 0x7f56dc06d0c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006fd0 tx=0x7f56f0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.367+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f56f8077b40 con 0x7f5700159820 2024-09-13T17:18:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.612+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f57000b02b0 con 0x7f5700159820 2024-09-13T17:18:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.612+0000 7f56f7400640 1 -- 172.21.15.28:0/276089097 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1448 (secure 0 0 0) 0x7f56f804c5c0 con 0x7f5700159820 2024-09-13T17:18:56.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:56.614 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":670,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076362607,"bytes_used":2698141696,"bytes_avail":770395971584,"bytes_total":773094113280,"write_bytes_sec":61749045,"read_op_per_sec":0,"write_op_per_sec":59},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":28,"modified":"2024-09-13T17:18:19.039587+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:18:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 msgr2=0x7f56dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 0x7f56dc06d0c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f56f0006fd0 tx=0x7f56f0008040 comp rx=0 tx=0).stop 2024-09-13T17:18:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 msgr2=0x7f5700155860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f56f8034040 tx=0x7f56f8004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:56.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 shutdown_connections 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56dc06ac00 0x7f56dc06d0c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5700159820 0x7f5700155860 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5700159440 0x7f5700155320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 --2- 172.21.15.28:0/276089097 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f57000b22e0 0x7f570014dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 >> 172.21.15.28:0/276089097 conn(0x7f5700072e40 msgr2=0x7f570015ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 shutdown_connections 2024-09-13T17:18:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:56.614+0000 7f56ffe00640 1 -- 172.21.15.28:0/276089097 wait complete. 2024-09-13T17:18:56.632 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:18:57.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:57.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.798+0000 7f80a2400640 1 Processor -- start 2024-09-13T17:18:57.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.799+0000 7f80a2400640 1 -- start start 2024-09-13T17:18:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a2400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a2400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f809c0b21c0 0x7f809c06b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f809c0cad20 con 0x7f809c0b0440 2024-09-13T17:18:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a2400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f809c06b8a0 con 0x7f809c0b21c0 2024-09-13T17:18:57.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a2400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c06bb40 con 0x7f809c15fb90 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a1a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f809c0b0440 0x7f809c160330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58612/0 (socket says 172.21.15.28:58612) 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a1a00640 1 -- 172.21.15.28:0/858831527 learned_addr learned my addr 172.21.15.28:0/858831527 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f80a1000640 1 --2- 172.21.15.28:0/858831527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.800+0000 7f8093e00640 1 --2- 172.21.15.28:0/858831527 >> v2:172.21.15.189:3300/0 conn(0x7f809c0b21c0 0x7f809c06b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f809be00640 1 -- 172.21.15.28:0/858831527 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094589136 0 0) 0x7f809c0cad20 con 0x7f809c0b0440 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f809be00640 1 -- 172.21.15.28:0/858831527 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8080003520 con 0x7f809c0b0440 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f80a1000640 1 -- 172.21.15.28:0/858831527 >> v2:172.21.15.189:3300/0 conn(0x7f809c0b21c0 msgr2=0x7f809c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f80a1000640 1 --2- 172.21.15.28:0/858831527 >> v2:172.21.15.189:3300/0 conn(0x7f809c0b21c0 0x7f809c06b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f80a1000640 1 -- 172.21.15.28:0/858831527 >> v1:172.21.15.28:6789/0 conn(0x7f809c0b0440 legacy=0x7f809c160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.801+0000 7f80a1000640 1 -- 172.21.15.28:0/858831527 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f809c06be20 con 0x7f809c15fb90 2024-09-13T17:18:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.802+0000 7f80a1000640 1 --2- 172.21.15.28:0/858831527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0b19d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f808c0097a0 tx=0x7f808c02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2bb3d881a9fb98c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.802+0000 7f809be00640 1 -- 172.21.15.28:0/858831527 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c02adb0 con 0x7f809c15fb90 2024-09-13T17:18:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.802+0000 7f809be00640 1 -- 172.21.15.28:0/858831527 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c032440 con 0x7f809c15fb90 2024-09-13T17:18:57.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f809be00640 1 -- 172.21.15.28:0/858831527 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c0327b0 con 0x7f809c15fb90 2024-09-13T17:18:57.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 -- 172.21.15.28:0/858831527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 msgr2=0x7f809c0b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:57.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 --2- 172.21.15.28:0/858831527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0b19d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f808c0097a0 tx=0x7f808c02a060 comp rx=0 tx=0).stop 2024-09-13T17:18:57.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 -- 172.21.15.28:0/858831527 shutdown_connections 2024-09-13T17:18:57.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 --2- 172.21.15.28:0/858831527 >> v2:172.21.15.189:3300/0 conn(0x7f809c0b21c0 0x7f809c06b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:57.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 --2- 172.21.15.28:0/858831527 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0b19d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:57.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 -- 172.21.15.28:0/858831527 >> 172.21.15.28:0/858831527 conn(0x7f809c072e40 msgr2=0x7f809c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:57.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 -- 172.21.15.28:0/858831527 shutdown_connections 2024-09-13T17:18:57.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.803+0000 7f80a2400640 1 -- 172.21.15.28:0/858831527 wait complete. 2024-09-13T17:18:57.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.805+0000 7f80a2400640 1 Processor -- start 2024-09-13T17:18:57.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.805+0000 7f80a2400640 1 -- start start 2024-09-13T17:18:57.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f809c0b21c0 0x7f809c0c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c15d110 con 0x7f809c0b0440 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809c154320 con 0x7f809c0b21c0 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a2400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f809c154600 con 0x7f809c15fb90 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49594/0 (socket says 172.21.15.28:49594) 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a1000640 1 -- 172.21.15.28:0/3293319653 learned_addr learned my addr 172.21.15.28:0/3293319653 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f8093e00640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f809c0b21c0 0x7f809c0c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.806+0000 7f80a1a00640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 -- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 msgr2=0x7f809c0c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 -- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f809c0b21c0 msgr2=0x7f809c0c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f809c0b21c0 0x7f809c0c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 -- 172.21.15.28:0/3293319653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c009400 con 0x7f809c0b0440 2024-09-13T17:18:57.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a1000640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f808c02a0a0 tx=0x7f808c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:57.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c03c810 con 0x7f809c0b0440 2024-09-13T17:18:57.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808c03c970 con 0x7f809c0b0440 2024-09-13T17:18:57.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809c15d380 con 0x7f809c0b0440 2024-09-13T17:18:57.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808c03cd00 con 0x7f809c0b0440 2024-09-13T17:18:57.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.807+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f809c150eb0 con 0x7f809c0b0440 2024-09-13T17:18:57.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.808+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c0b11f0 con 0x7f809c0b0440 2024-09-13T17:18:57.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.809+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f808c03b3a0 con 0x7f809c0b0440 2024-09-13T17:18:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.810+0000 7f8099600640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 0x7f807806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.810+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f808c0aa050 con 0x7f809c0b0440 2024-09-13T17:18:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.811+0000 7f8093e00640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 0x7f807806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.811+0000 7f8093e00640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 0x7f807806d0c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f809c15ae90 tx=0x7f8084009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:57.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:57.814+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f808c0dd9c0 con 0x7f809c0b0440 2024-09-13T17:18:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.028+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f809c15abe0 con 0x7f809c0b0440 2024-09-13T17:18:58.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.029+0000 7f8099600640 1 -- 172.21.15.28:0/3293319653 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f808c03ce60 con 0x7f809c0b0440 2024-09-13T17:18:58.031 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:58.031 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:58.031 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 msgr2=0x7f807806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 0x7f807806d0c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f809c15ae90 tx=0x7f8084009190 comp rx=0 tx=0).stop 2024-09-13T17:18:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 msgr2=0x7f809c14fed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f808c02a0a0 tx=0x7f808c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 shutdown_connections 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f807806ac00 0x7f807806d0c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f809c15fb90 0x7f809c0c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f809c0b21c0 0x7f809c0c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 --2- 172.21.15.28:0/3293319653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f809c0b0440 0x7f809c14fed0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 >> 172.21.15.28:0/3293319653 conn(0x7f809c072e40 msgr2=0x7f809c1602a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 shutdown_connections 2024-09-13T17:18:58.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.031+0000 7f80a2400640 1 -- 172.21.15.28:0/3293319653 wait complete. 2024-09-13T17:18:58.049 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:18:58.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.203+0000 7fb7f4600640 1 Processor -- start 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.203+0000 7fb7f4600640 1 -- start start 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7ec0add10 0x7fb7ec06c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb7ec0b1610 0x7fb7ec06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0b3750 con 0x7fb7ec0add10 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7ec0cef90 con 0x7fb7ec0b1610 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f4600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0b2e20 con 0x7fb7ec0ad930 2024-09-13T17:18:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.204+0000 7fb7f1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7ec0add10 0x7fb7ec06c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.205+0000 7fb7f1200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb7ec0add10 0x7fb7ec06c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49598/0 (socket says 172.21.15.28:49598) 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.205+0000 7fb7f2600640 1 --2- 172.21.15.28:0/3966339020 >> v2:172.21.15.189:3300/0 conn(0x7fb7ec0b1610 0x7fb7ec06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.205+0000 7fb7f1200640 1 -- 172.21.15.28:0/3966339020 learned_addr learned my addr 172.21.15.28:0/3966339020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.205+0000 7fb7f1c00640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.223+0000 7fb7f1200640 1 -- 172.21.15.28:0/3966339020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 msgr2=0x7fb7ec0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.223+0000 7fb7f1200640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.223+0000 7fb7f1200640 1 -- 172.21.15.28:0/3966339020 >> v2:172.21.15.189:3300/0 conn(0x7fb7ec0b1610 msgr2=0x7fb7ec06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.223+0000 7fb7f1200640 1 --2- 172.21.15.28:0/3966339020 >> v2:172.21.15.189:3300/0 conn(0x7fb7ec0b1610 0x7fb7ec06cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.223+0000 7fb7f1200640 1 -- 172.21.15.28:0/3966339020 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0b3100 con 0x7fb7ec0add10 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7f1200640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0add10 0x7fb7ec06c680 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b36c0 tx=0x7fb7d8032a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=805765d5d261ff67 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7ebe00640 1 -- 172.21.15.28:0/3966339020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8037890 con 0x7fb7ec0add10 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7ebe00640 1 -- 172.21.15.28:0/3966339020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d80379f0 con 0x7fb7ec0add10 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7ebe00640 1 -- 172.21.15.28:0/3966339020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8037d80 con 0x7fb7ec0add10 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7f4600640 1 -- 172.21.15.28:0/3966339020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0add10 msgr2=0x7fb7ec06c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.224+0000 7fb7f4600640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0add10 0x7fb7ec06c680 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec0b36c0 tx=0x7fb7d8032a30 comp rx=0 tx=0).stop 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 -- 172.21.15.28:0/3966339020 shutdown_connections 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0add10 0x7fb7ec06c680 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 --2- 172.21.15.28:0/3966339020 >> v2:172.21.15.189:3300/0 conn(0x7fb7ec0b1610 0x7fb7ec06cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 --2- 172.21.15.28:0/3966339020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 -- 172.21.15.28:0/3966339020 >> 172.21.15.28:0/3966339020 conn(0x7fb7ec069e80 msgr2=0x7fb7ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 -- 172.21.15.28:0/3966339020 shutdown_connections 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.225+0000 7fb7f4600640 1 -- 172.21.15.28:0/3966339020 wait complete. 2024-09-13T17:18:58.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.226+0000 7fb7f4600640 1 Processor -- start 2024-09-13T17:18:58.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.226+0000 7fb7f4600640 1 -- start start 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec1638d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7ec0add10 0x7fb7ec1fe820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0ab780 con 0x7fb7ec0b1610 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec163ed0 con 0x7fb7ec0add10 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.227+0000 7fb7f4600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec1fed60 con 0x7fb7ec0ad930 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec1638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f1200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7ec0add10 0x7fb7ec1fe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49606/0 (socket says 172.21.15.28:49606) 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f1c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec1638d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55626/0 (socket says 172.21.15.28:55626) 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 -- 172.21.15.28:0/1830934006 learned_addr learned my addr 172.21.15.28:0/1830934006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 -- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 msgr2=0x7fb7ec1638d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec1638d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 -- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7ec0add10 msgr2=0x7fb7ec1fe820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7ec0add10 0x7fb7ec1fe820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 -- 172.21.15.28:0/1830934006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8014c10 con 0x7fb7ec0b1610 2024-09-13T17:18:58.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.228+0000 7fb7f2600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400e630 tx=0x7fb7e400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.229+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4016620 con 0x7fb7ec0b1610 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.229+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e4016780 con 0x7fb7ec0b1610 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.229+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4016b10 con 0x7fb7ec0b1610 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.229+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec1ffe60 con 0x7fb7ec0b1610 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.229+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec0abc90 con 0x7fb7ec0b1610 2024-09-13T17:18:58.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.230+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec08a7d0 con 0x7fb7ec0b1610 2024-09-13T17:18:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.231+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb7e4016c70 con 0x7fb7ec0b1610 2024-09-13T17:18:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.231+0000 7fb7e9600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 0x7fb7cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.232+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb7e4089f20 con 0x7fb7ec0b1610 2024-09-13T17:18:58.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.232+0000 7fb7f1c00640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 0x7fb7cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.233+0000 7fb7f1c00640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 0x7fb7cc06d0c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0004150 tx=0x7fb7e0009100 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.236+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb7e4055180 con 0x7fb7ec0b1610 2024-09-13T17:18:58.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.479+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb7ec0ce490 con 0x7fb7ec0b1610 2024-09-13T17:18:58.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.479+0000 7fb7e9600640 1 -- 172.21.15.28:0/1830934006 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1633 (secure 0 0 0) 0x7fb7e400ed30 con 0x7fb7ec0b1610 2024-09-13T17:18:58.481 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:58.481 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":672,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076362607,"bytes_used":2698141696,"bytes_avail":770395971584,"bytes_total":773094113280,"write_bytes_sec":24715940,"read_op_per_sec":0,"write_op_per_sec":24},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-09-13T17:18:57.048172+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:18:58.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 msgr2=0x7fb7cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 0x7fb7cc06d0c0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0004150 tx=0x7fb7e0009100 comp rx=0 tx=0).stop 2024-09-13T17:18:58.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 msgr2=0x7fb7ec1ff920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400e630 tx=0x7fb7e400cf90 comp rx=0 tx=0).stop 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 shutdown_connections 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb7cc06ac00 0x7fb7cc06d0c0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb7ec0b1610 0x7fb7ec1ff920 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7ec0add10 0x7fb7ec1fe820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 --2- 172.21.15.28:0/1830934006 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7ec0ad930 0x7fb7ec1638d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 >> 172.21.15.28:0/1830934006 conn(0x7fb7ec069e80 msgr2=0x7fb7ec1698b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 shutdown_connections 2024-09-13T17:18:58.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.481+0000 7fb7f4600640 1 -- 172.21.15.28:0/1830934006 wait complete. 2024-09-13T17:18:58.499 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-09-13T17:18:58.499 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub abort 2024-09-13T17:18:58.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.654+0000 7fb159600640 1 Processor -- start 2024-09-13T17:18:58.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.654+0000 7fb159600640 1 -- start start 2024-09-13T17:18:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.655+0000 7fb159600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb15415ab60 0x7fb15406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.655+0000 7fb159600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 0x7fb15406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb159600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15406c8b0 con 0x7fb15415a780 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb159600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15406cb90 con 0x7fb15415ab60 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb159600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1540cc980 con 0x7fb154071490 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb152a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb15415a780 0x7fb154070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58614/0 (socket says 172.21.15.28:58614) 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb152a00640 1 -- 172.21.15.28:0/615891762 learned_addr learned my addr 172.21.15.28:0/615891762 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:58.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb14a000640 1 --2- 172.21.15.28:0/615891762 >> v2:172.21.15.28:3300/0 conn(0x7fb15415ab60 0x7fb15406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb153400640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 0x7fb15406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb14a000640 1 -- 172.21.15.28:0/615891762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 msgr2=0x7fb15406c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb14a000640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 0x7fb15406c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb14a000640 1 -- 172.21.15.28:0/615891762 >> v1:172.21.15.28:6789/0 conn(0x7fb15415a780 legacy=0x7fb154070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb14a000640 1 -- 172.21.15.28:0/615891762 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1540ccc60 con 0x7fb15415ab60 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb152000640 1 -- 172.21.15.28:0/615891762 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603563841 0 0) 0x7fb15406c8b0 con 0x7fb15415a780 2024-09-13T17:18:58.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.656+0000 7fb153400640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 0x7fb15406c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.657+0000 7fb14a000640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb15406bb40 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2ec8746a1d660a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.657+0000 7fb152000640 1 -- 172.21.15.28:0/615891762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c032440 con 0x7fb15415ab60 2024-09-13T17:18:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb152000640 1 -- 172.21.15.28:0/615891762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c0325a0 con 0x7fb15415ab60 2024-09-13T17:18:58.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb152000640 1 -- 172.21.15.28:0/615891762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c032930 con 0x7fb15415ab60 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 -- 172.21.15.28:0/615891762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 msgr2=0x7fb15406bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb15406bb40 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fb13c0097a0 tx=0x7fb13c02a180 comp rx=0 tx=0).stop 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 -- 172.21.15.28:0/615891762 shutdown_connections 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb154071490 0x7fb15406c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 --2- 172.21.15.28:0/615891762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb15406bb40 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 -- 172.21.15.28:0/615891762 >> 172.21.15.28:0/615891762 conn(0x7fb1540b0270 msgr2=0x7fb1540b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 -- 172.21.15.28:0/615891762 shutdown_connections 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.658+0000 7fb159600640 1 -- 172.21.15.28:0/615891762 wait complete. 2024-09-13T17:18:58.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.660+0000 7fb159600640 1 Processor -- start 2024-09-13T17:18:58.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.660+0000 7fb159600640 1 -- start start 2024-09-13T17:18:58.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb154071490 0x7fb154153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb15415a780 0x7fb1541531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541536e0 con 0x7fb15415ab60 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541542c0 con 0x7fb154071490 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb159600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1541545a0 con 0x7fb15415a780 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49618/0 (socket says 172.21.15.28:49618) 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.661+0000 7fb153400640 1 -- 172.21.15.28:0/1090216030 learned_addr learned my addr 172.21.15.28:0/1090216030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb14a000640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb15415a780 0x7fb1541531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb152a00640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb154071490 0x7fb154153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 -- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb15415a780 msgr2=0x7fb1541531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb15415a780 0x7fb1541531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 -- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb154071490 msgr2=0x7fb154153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb154071490 0x7fb154153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 -- 172.21.15.28:0/1090216030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c009400 con 0x7fb15415ab60 2024-09-13T17:18:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb153400640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00cea0 tx=0x7fb14c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb14b400640 1 -- 172.21.15.28:0/1090216030 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c003b90 con 0x7fb15415ab60 2024-09-13T17:18:58.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb14b400640 1 -- 172.21.15.28:0/1090216030 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c003d10 con 0x7fb15415ab60 2024-09-13T17:18:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb14b400640 1 -- 172.21.15.28:0/1090216030 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c004260 con 0x7fb15415ab60 2024-09-13T17:18:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb154154880 con 0x7fb15415ab60 2024-09-13T17:18:58.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.662+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb154156f30 con 0x7fb15415ab60 2024-09-13T17:18:58.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.665+0000 7fb14b400640 1 -- 172.21.15.28:0/1090216030 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb14c0043c0 con 0x7fb15415ab60 2024-09-13T17:18:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.666+0000 7fb14b400640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 0x7fb13006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.666+0000 7fb14b400640 1 -- 172.21.15.28:0/1090216030 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb14c089c70 con 0x7fb15415ab60 2024-09-13T17:18:58.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.666+0000 7fb152a00640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 0x7fb13006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.667+0000 7fb15c4d9740 1 Processor -- start 2024-09-13T17:18:58.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.667+0000 7fb15c4d9740 1 -- start start 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.667+0000 7fb152a00640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 0x7fb13006d110 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb144000f60 tx=0x7fb144007b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x5615827fc760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615827f9b40 0x561582849850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561582707000 con 0x561582823830 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561582764d40 con 0x5615827f9b40 2024-09-13T17:18:58.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615827624d0 con 0x5615827fc380 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49624/0 (socket says 172.21.15.28:49624) 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 -- 172.21.15.28:0/1328327046 learned_addr learned my addr 172.21.15.28:0/1328327046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb152a00640 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x5615827fc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb14a000640 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615827f9b40 0x561582849850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 -- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 msgr2=0x5615827fc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x5615827fc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 -- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615827f9b40 msgr2=0x561582849850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615827f9b40 0x561582849850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.668+0000 7fb153400640 1 -- 172.21.15.28:0/1328327046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615827193a0 con 0x561582823830 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb153400640 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fb14c060a20 tx=0x7fb14c0584c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c428efdfa91d01 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb143400640 1 -- 172.21.15.28:0/1328327046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c05c990 con 0x561582823830 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb143400640 1 -- 172.21.15.28:0/1328327046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c060f50 con 0x561582823830 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb143400640 1 -- 172.21.15.28:0/1328327046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14c055270 con 0x561582823830 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb15c4d9740 1 -- 172.21.15.28:0/1328327046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 msgr2=0x561582849d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.669+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7fb14c060a20 tx=0x7fb14c0584c0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 -- 172.21.15.28:0/1328327046 shutdown_connections 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x561582849d90 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615827f9b40 0x561582849850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/1328327046 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x5615827fc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 -- 172.21.15.28:0/1328327046 >> 172.21.15.28:0/1328327046 conn(0x5615827f8ab0 msgr2=0x561582840f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 -- 172.21.15.28:0/1328327046 shutdown_connections 2024-09-13T17:18:58.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 -- 172.21.15.28:0/1328327046 wait complete. 2024-09-13T17:18:58.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.670+0000 7fb15c4d9740 1 -- start start 2024-09-13T17:18:58.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.671+0000 7fb15c4d9740 1 Processor -- start 2024-09-13T17:18:58.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 0x5615827f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615827f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56158275ddc0 con 0x5615827fc380 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56158271c3e0 con 0x561582827310 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x56158271c080 con 0x561582823830 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb15c4d9740 10 client.0 _ll_register_callbacks cb 0x561582757430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49628/0 (socket says 172.21.15.28:49628) 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 -- 172.21.15.28:0/2280619221 learned_addr learned my addr 172.21.15.28:0/2280619221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb14a000640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 0x5615827f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 -- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 msgr2=0x5615827f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb153400640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615827f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 0x5615827f67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 -- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 msgr2=0x5615827f6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615827f6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb152a00640 1 -- 172.21.15.28:0/2280619221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb154156f30 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb14a000640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 0x5615827f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.672+0000 7fb153400640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615827f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb152a00640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb144001e50 tx=0x7fb144000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da1eb15cc1b8c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb137e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144014e40 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb144012590 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561582787600 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb144012750 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb15c4d9740 10 client.9711 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.673+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x56158275c440 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561582787600 con 0x5615827fc380 2024-09-13T17:18:58.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb144020070 con 0x5615827fc380 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 10 client.9711 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 10 client.9711 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 20 client.9711 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x561582821d80 0x561582822160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 4 client.9711 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56158274fc90 con 0x561582821d80 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb14a000640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x561582821d80 0x561582822160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb137e00640 10 client.9711 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.674+0000 7fb14a000640 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x561582821d80 0x561582822160 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40f1678e4dc6e242 server_cookie=177024baa936a8c0 in_seq=0 out_seq=0 2024-09-13T17:18:58.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb137e00640 1 -- 172.21.15.28:0/2280619221 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x56158274fc90 con 0x561582821d80 2024-09-13T17:18:58.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb137e00640 10 client.9711 handle_command_reply: tid=0 2024-09-13T17:18:58.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 client.9711 shutdown 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 msgr2=0x561582821150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7fb144001e50 tx=0x7fb144000ed0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 shutdown_connections 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x561582821d80 0x561582822160 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615827f6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582823830 0x5615827f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/2280619221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x561582821150 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb137e00640 0 client.9711 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.675+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 >> 172.21.15.28:0/2280619221 conn(0x5615827f9b40 msgr2=0x561582844420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.676+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 shutdown_connections 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.676+0000 7fb15c4d9740 1 -- 172.21.15.28:0/2280619221 wait complete. 2024-09-13T17:18:58.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.676+0000 7fb15c4d9740 20 client.9711 trim_cache size 0 max 16384 2024-09-13T17:18:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.690+0000 7fb15c4d9740 1 Processor -- start 2024-09-13T17:18:58.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.690+0000 7fb15c4d9740 1 -- start start 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x561582841130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615828202f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615827f7690 con 0x561582823830 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615824c5040 con 0x561582827310 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561582826bb0 con 0x5615827fc380 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.691+0000 7fb14a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49634/0 (socket says 172.21.15.28:49634) 2024-09-13T17:18:58.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 -- 172.21.15.28:0/3719092651 learned_addr learned my addr 172.21.15.28:0/3719092651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb153400640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615828202f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb152a00640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x561582841130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 -- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 msgr2=0x561582841130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x561582841130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 -- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 msgr2=0x5615828202f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615828202f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 -- 172.21.15.28:0/3719092651 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615826fe5d0 con 0x561582823830 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb152a00640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x561582841130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:18:58.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb14a000640 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02a770 tx=0x7fb13c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61d1788e7782a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb143e00640 1 -- 172.21.15.28:0/3719092651 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c044d70 con 0x561582823830 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.692+0000 7fb143e00640 1 -- 172.21.15.28:0/3719092651 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c032b10 con 0x561582823830 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb143e00640 1 -- 172.21.15.28:0/3719092651 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c032e80 con 0x561582823830 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3719092651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 msgr2=0x5615827fab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fb13c02a770 tx=0x7fb13c02e000 comp rx=0 tx=0).stop 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3719092651 shutdown_connections 2024-09-13T17:18:58.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582827310 0x5615828202f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561582823830 0x5615827fab60 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3719092651 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615827fc380 0x561582841130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3719092651 >> 172.21.15.28:0/3719092651 conn(0x5615827f9b40 msgr2=0x561582820b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3719092651 shutdown_connections 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.693+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3719092651 wait complete. 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.694+0000 7fb15c4d9740 1 -- start start 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.694+0000 7fb15c4d9740 1 Processor -- start 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582823830 0x5615828489c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 0x5615828ab010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x56158285a640 con 0x5615827fc380 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615827f64c0 con 0x561582823830 2024-09-13T17:18:58.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615827fa170 con 0x561582827310 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb15c4d9740 10 client.0 _ll_register_callbacks cb 0x7fb144007360 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb152a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb152a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49638/0 (socket says 172.21.15.28:49638) 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb152a00640 1 -- 172.21.15.28:0/3789817724 learned_addr learned my addr 172.21.15.28:0/3789817724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb14a000640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582823830 0x5615828489c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb153400640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 0x5615828ab010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb152a00640 1 -- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 msgr2=0x5615828ab010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.695+0000 7fb152a00640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 0x5615828ab010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb152a00640 1 -- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582823830 msgr2=0x5615828489c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb152a00640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582823830 0x5615828489c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb152a00640 1 -- 172.21.15.28:0/3789817724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1540ccc60 con 0x5615827fc380 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb153400640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 0x5615828ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb152a00640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fb144012e70 tx=0x7fb14400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1cf859e975f7ee5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb137e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144032600 con 0x5615827fc380 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb144032760 con 0x5615827fc380 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb14c060f50 con 0x5615827fc380 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb144032920 con 0x5615827fc380 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb15c4d9740 10 client.9717 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.696+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x5615827239b0 con 0x5615827fc380 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14c060f50 con 0x5615827fc380 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb144032a80 con 0x5615827fc380 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 10 client.9717 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 10 client.9717 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 20 client.9717 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x5615827f57a0 0x5615827f5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 4 client.9717 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub abort"}) v1 -- 0x5615824514e0 con 0x5615827f57a0 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.697+0000 7fb14a000640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x5615827f57a0 0x5615827f5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb137e00640 10 client.9717 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb14a000640 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x5615827f57a0 0x5615827f5b80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbb46395311d6982 server_cookie=da7b5a7046f3269e in_seq=0 out_seq=0 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb137e00640 1 -- 172.21.15.28:0/3789817724 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x5615824514e0 con 0x5615827f57a0 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb137e00640 10 client.9717 handle_command_reply: tid=0 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 client.9717 shutdown 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 msgr2=0x5615828402e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fb144012e70 tx=0x7fb14400a000 comp rx=0 tx=0).stop 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 shutdown_connections 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x5615827f57a0 0x5615827f5b80 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561582827310 0x5615828ab010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561582823830 0x5615828489c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb137e00640 0 client.9717 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 --2- 172.21.15.28:0/3789817724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615827fc380 0x5615828402e0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 >> 172.21.15.28:0/3789817724 conn(0x5615827f9b40 msgr2=0x561582844420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 shutdown_connections 2024-09-13T17:18:58.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 1 -- 172.21.15.28:0/3789817724 wait complete. 2024-09-13T17:18:58.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.698+0000 7fb15c4d9740 20 client.9717 trim_cache size 0 max 16384 2024-09-13T17:18:58.705 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:58.705 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:18:58.705 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 msgr2=0x7fb13006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 0x7fb13006d110 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb144000f60 tx=0x7fb144007b70 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 msgr2=0x7fb1540ce610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fb14c00cea0 tx=0x7fb14c00cf90 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 shutdown_connections 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb13006ac50 0x7fb13006d110 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb15415ab60 0x7fb1540ce610 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb15415a780 0x7fb1541531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 --2- 172.21.15.28:0/1090216030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb154071490 0x7fb154153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 >> 172.21.15.28:0/1090216030 conn(0x7fb1540b0270 msgr2=0x7fb154070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 shutdown_connections 2024-09-13T17:18:58.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.699+0000 7fb159600640 1 -- 172.21.15.28:0/1090216030 wait complete. 2024-09-13T17:18:58.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub status 2024-09-13T17:18:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.871+0000 7fd355800640 1 Processor -- start 2024-09-13T17:18:58.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.871+0000 7fd355800640 1 -- start start 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.872+0000 7fd355800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 0x7fd35015c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd355800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35015ab60 con 0x7fd350070430 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd355800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35006f920 con 0x7fd350159770 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd355800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35015b290 con 0x7fd35006f3f0 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd346000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd350070430 0x7fd35015bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58628/0 (socket says 172.21.15.28:58628) 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd346000640 1 -- 172.21.15.28:0/2341616390 learned_addr learned my addr 172.21.15.28:0/2341616390 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:18:58.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd34f400640 1 --2- 172.21.15.28:0/2341616390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 0x7fd35015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908918677 0 0) 0x7fd338001110 con 0x7fd350070430 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd334003520 con 0x7fd350070430 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186820702 0 0) 0x7fd35006f920 con 0x7fd350159770 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.873+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd338001110 con 0x7fd350159770 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.874+0000 7fd34f400640 1 -- 172.21.15.28:0/2341616390 >> v1:172.21.15.189:6789/0 conn(0x7fd350159770 legacy=0x7fd35006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.874+0000 7fd34f400640 1 -- 172.21.15.28:0/2341616390 >> v1:172.21.15.28:6789/0 conn(0x7fd350070430 legacy=0x7fd35015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.874+0000 7fd34f400640 1 -- 172.21.15.28:0/2341616390 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35015b570 con 0x7fd35006f3f0 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd34f400640 1 --2- 172.21.15.28:0/2341616390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 0x7fd35015c480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd348009dc0 tx=0x7fd34802a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52ce69978fe8f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348007740 con 0x7fd35006f3f0 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34802d050 con 0x7fd35006f3f0 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd34e000640 1 -- 172.21.15.28:0/2341616390 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348007aa0 con 0x7fd35006f3f0 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd355800640 1 -- 172.21.15.28:0/2341616390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 msgr2=0x7fd35015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.875+0000 7fd355800640 1 --2- 172.21.15.28:0/2341616390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 0x7fd35015c480 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd348009dc0 tx=0x7fd34802a640 comp rx=0 tx=0).stop 2024-09-13T17:18:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.876+0000 7fd355800640 1 -- 172.21.15.28:0/2341616390 shutdown_connections 2024-09-13T17:18:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.876+0000 7fd355800640 1 --2- 172.21.15.28:0/2341616390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd35006f3f0 0x7fd35015c480 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.876+0000 7fd355800640 1 -- 172.21.15.28:0/2341616390 >> 172.21.15.28:0/2341616390 conn(0x7fd3500b0270 msgr2=0x7fd3500b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.876+0000 7fd355800640 1 -- 172.21.15.28:0/2341616390 shutdown_connections 2024-09-13T17:18:58.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.876+0000 7fd355800640 1 -- 172.21.15.28:0/2341616390 wait complete. 2024-09-13T17:18:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.877+0000 7fd355800640 1 Processor -- start 2024-09-13T17:18:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.877+0000 7fd355800640 1 -- start start 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006f3f0 0x7fd350157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd350070430 0x7fd350157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350148650 con 0x7fd350159770 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350141f20 con 0x7fd35006f3f0 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.878+0000 7fd355800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500ccf60 con 0x7fd350070430 2024-09-13T17:18:58.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49642/0 (socket says 172.21.15.28:49642) 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 -- 172.21.15.28:0/3293989363 learned_addr learned my addr 172.21.15.28:0/3293989363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006f3f0 0x7fd350157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd346000640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd350070430 0x7fd350157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 -- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd350070430 msgr2=0x7fd350157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd350070430 0x7fd350157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 -- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006f3f0 msgr2=0x7fd350157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006f3f0 0x7fd350157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 -- 172.21.15.28:0/3293989363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348009a70 con 0x7fd350159770 2024-09-13T17:18:58.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.879+0000 7fd34f400640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd348009eb0 tx=0x7fd348004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.880+0000 7fd347400640 1 -- 172.21.15.28:0/3293989363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34803c830 con 0x7fd350159770 2024-09-13T17:18:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.880+0000 7fd347400640 1 -- 172.21.15.28:0/3293989363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34803c990 con 0x7fd350159770 2024-09-13T17:18:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.880+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3500ccc80 con 0x7fd350159770 2024-09-13T17:18:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.880+0000 7fd347400640 1 -- 172.21.15.28:0/3293989363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34803cd20 con 0x7fd350159770 2024-09-13T17:18:58.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.880+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd350142570 con 0x7fd350159770 2024-09-13T17:18:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.882+0000 7fd347400640 1 -- 172.21.15.28:0/3293989363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd348031400 con 0x7fd350159770 2024-09-13T17:18:58.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.883+0000 7fd347400640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 0x7fd32c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.883+0000 7fd347400640 1 -- 172.21.15.28:0/3293989363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd3480aadd0 con 0x7fd350159770 2024-09-13T17:18:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.883+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 0x7fd32c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.883+0000 7fd3585d8740 1 Processor -- start 2024-09-13T17:18:58.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 -- start start 2024-09-13T17:18:58.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 0x7fd32c06d0c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd340007fd0 tx=0x7fd340009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b581cf330 0x560b581f8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b580d56a0 con 0x560b581ca700 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b58138cc0 con 0x560b581cf330 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.884+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b58136450 con 0x560b58212c20 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49644/0 (socket says 172.21.15.28:49644) 2024-09-13T17:18:58.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 -- 172.21.15.28:0/864797172 learned_addr learned my addr 172.21.15.28:0/864797172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd346000640 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b581cf330 0x560b581f8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34f400640 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 -- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 msgr2=0x560b581f8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 -- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b581cf330 msgr2=0x560b581f8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b581cf330 0x560b581f8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 -- 172.21.15.28:0/864797172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b581022e0 con 0x560b581ca700 2024-09-13T17:18:58.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.885+0000 7fd34ea00640 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fd340001e70 tx=0x7fd340012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45bad781c44ade0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd33f400640 1 -- 172.21.15.28:0/864797172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400149d0 con 0x560b581ca700 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd33f400640 1 -- 172.21.15.28:0/864797172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340014b30 con 0x560b581ca700 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd33f400640 1 -- 172.21.15.28:0/864797172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd340020410 con 0x560b581ca700 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 -- 172.21.15.28:0/864797172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 msgr2=0x560b581cedf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fd340001e70 tx=0x7fd340012000 comp rx=0 tx=0).stop 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 -- 172.21.15.28:0/864797172 shutdown_connections 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b581cf330 0x560b581f8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 --2- 172.21.15.28:0/864797172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581ca700 0x560b581cedf0 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.886+0000 7fd3585d8740 1 -- 172.21.15.28:0/864797172 >> 172.21.15.28:0/864797172 conn(0x560b581cb390 msgr2=0x560b58213840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.887+0000 7fd3585d8740 1 -- 172.21.15.28:0/864797172 shutdown_connections 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.887+0000 7fd3585d8740 1 -- 172.21.15.28:0/864797172 wait complete. 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.887+0000 7fd3585d8740 1 -- start start 2024-09-13T17:18:58.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.887+0000 7fd3585d8740 1 Processor -- start 2024-09-13T17:18:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.888+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.888+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.888+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b581f89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd3585d8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b58131d40 con 0x560b581cf330 2024-09-13T17:18:58.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b580d3280 con 0x560b58201cb0 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b580d2f20 con 0x560b58212c20 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd3585d8740 10 client.0 _ll_register_callbacks cb 0x560b5812b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49648/0 (socket says 172.21.15.28:49648) 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 -- 172.21.15.28:0/3206571718 learned_addr learned my addr 172.21.15.28:0/3206571718 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34f400640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b581f89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd346000640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 -- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 msgr2=0x560b581f8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 -- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 msgr2=0x560b581f89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b581f89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34ea00640 1 -- 172.21.15.28:0/3206571718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd338001110 con 0x560b581cf330 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.889+0000 7fd34f400640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b581f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:18:58.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fd340014820 tx=0x7fd340000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d7f4d13eb44f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd333e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400149d0 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560b57e254e0 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd34001c040 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd340020c80 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd3585d8740 10 client.9729 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x560b581303c0 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.890+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x560b57e254e0 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd340012070 con 0x560b581cf330 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 10 client.9729 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 10 client.9729 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 20 client.9729 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581e9d70 0x560b581ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 4 client.9729 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:18:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b57e367e0 con 0x560b581e9d70 2024-09-13T17:18:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd346000640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581e9d70 0x560b581ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd346000640 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581e9d70 0x560b581ea5f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fca37e08b0d60d6 server_cookie=5b31d2aa5b3588ed in_seq=0 out_seq=0 2024-09-13T17:18:58.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.891+0000 7fd333e00640 10 client.9729 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd333e00640 1 -- 172.21.15.28:0/3206571718 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x560b57e367e0 con 0x560b581e9d70 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd333e00640 10 client.9729 handle_command_reply: tid=0 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 client.9729 shutdown 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 msgr2=0x560b58201770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fd340014820 tx=0x7fd340000ed0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 shutdown_connections 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581e9d70 0x560b581ea5f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b581f89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58212c20 0x560b581f8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd333e00640 0 client.9729 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3206571718 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b58201770 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.892+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 >> 172.21.15.28:0/3206571718 conn(0x560b581ca700 msgr2=0x560b58211e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.893+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 shutdown_connections 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.893+0000 7fd3585d8740 1 -- 172.21.15.28:0/3206571718 wait complete. 2024-09-13T17:18:58.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.893+0000 7fd3585d8740 20 client.9729 trim_cache size 0 max 16384 2024-09-13T17:18:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 Processor -- start 2024-09-13T17:18:58.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 -- start start 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b581cf330 0x560b58210850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b58210d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57e99040 con 0x560b58212c20 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b582310c0 con 0x560b58201cb0 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.908+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b581cdaf0 con 0x560b581cf330 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49652/0 (socket says 172.21.15.28:49652) 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 -- 172.21.15.28:0/3363496070 learned_addr learned my addr 172.21.15.28:0/3363496070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd346000640 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b58210d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34ea00640 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b581cf330 0x560b58210850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 -- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b581cf330 msgr2=0x560b58210850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b581cf330 0x560b58210850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 -- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 msgr2=0x560b58210d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b58210d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 -- 172.21.15.28:0/3363496070 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b580d2610 con 0x560b58212c20 2024-09-13T17:18:58.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.909+0000 7fd34f400640 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd348079d20 tx=0x7fd3480760d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0896b2d399ebac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd33fe00640 1 -- 172.21.15.28:0/3363496070 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348079d90 con 0x560b58212c20 2024-09-13T17:18:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd33fe00640 1 -- 172.21.15.28:0/3363496070 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34807db10 con 0x560b58212c20 2024-09-13T17:18:58.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd33fe00640 1 -- 172.21.15.28:0/3363496070 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3480820d0 con 0x560b58212c20 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 -- 172.21.15.28:0/3363496070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 msgr2=0x560b58225e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fd348079d20 tx=0x7fd3480760d0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 -- 172.21.15.28:0/3363496070 shutdown_connections 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b58212c20 0x560b58225e40 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58201cb0 0x560b58210d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 --2- 172.21.15.28:0/3363496070 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b581cf330 0x560b58210850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 -- 172.21.15.28:0/3363496070 >> 172.21.15.28:0/3363496070 conn(0x560b581ca700 msgr2=0x560b581cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 -- 172.21.15.28:0/3363496070 shutdown_connections 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.910+0000 7fd3585d8740 1 -- 172.21.15.28:0/3363496070 wait complete. 2024-09-13T17:18:58.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.911+0000 7fd3585d8740 1 -- start start 2024-09-13T17:18:58.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.911+0000 7fd3585d8740 1 Processor -- start 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58201cb0 0x560b58239700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58212c20 0x560b581f9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b5820fe50 con 0x560b581cf330 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b5823a2e0 con 0x560b58212c20 2024-09-13T17:18:58.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x560b58239c40 con 0x560b58201cb0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd3585d8740 10 client.0 _ll_register_callbacks cb 0x7fd340023070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd34ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49662/0 (socket says 172.21.15.28:49662) 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd34ea00640 1 -- 172.21.15.28:0/2973922927 learned_addr learned my addr 172.21.15.28:0/2973922927 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd346000640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58201cb0 0x560b58239700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.912+0000 7fd34f400640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58212c20 0x560b581f9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 -- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58201cb0 msgr2=0x560b58239700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58201cb0 0x560b58239700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 -- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58212c20 msgr2=0x560b581f9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58212c20 0x560b581f9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 -- 172.21.15.28:0/2973922927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd350142570 con 0x560b581cf330 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd34ea00640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd340000f10 tx=0x7fd34000a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07665a3783e7be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd333e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:18:58.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3400206c0 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd340014b30 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.913+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd340034a40 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd340034c00 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 10 client.9735 fetch_fsmap learned FSMap version 130 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x560b580f7b20 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd340014b30 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd340012050 con 0x560b581cf330 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 10 client.9735 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 10 client.9735 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 20 client.9735 populate_metadata read hostname 'smithi028' 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581f9aa0 0x560b58225e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 4 client.9735 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560b57e89e30 con 0x560b581f9aa0 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.914+0000 7fd346000640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581f9aa0 0x560b58225e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:58.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd346000640 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581f9aa0 0x560b58225e40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=705a0c060e304739 server_cookie=54ff3498a9b09b5d in_seq=0 out_seq=0 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd333e00640 10 client.9735 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd333e00640 1 -- 172.21.15.28:0/2973922927 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+69 (crc 0 0 0) 0x560b57e89e30 con 0x560b581f9aa0 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd333e00640 10 client.9735 handle_command_reply: tid=0 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 client.9735 shutdown 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 msgr2=0x560b5820f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7fd340000f10 tx=0x7fd34000a000 comp rx=0 tx=0).stop 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 shutdown_connections 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x560b581f9aa0 0x560b58225e40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b58212c20 0x560b581f9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x560b58201cb0 0x560b58239700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 --2- 172.21.15.28:0/2973922927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560b581cf330 0x560b5820f550 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd333e00640 0 client.9735 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:18:58.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.915+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 >> 172.21.15.28:0/2973922927 conn(0x560b581ca700 msgr2=0x560b58211d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 shutdown_connections 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd3585d8740 1 -- 172.21.15.28:0/2973922927 wait complete. 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd3585d8740 20 client.9735 trim_cache size 0 max 16384 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:18:58.922 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 msgr2=0x7fd32c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 0x7fd32c06d0c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd340007fd0 tx=0x7fd340009040 comp rx=0 tx=0).stop 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 msgr2=0x7fd350147700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fd348009eb0 tx=0x7fd348004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 shutdown_connections 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd32c06ac00 0x7fd32c06d0c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd350159770 0x7fd350147700 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd350070430 0x7fd350157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 --2- 172.21.15.28:0/3293989363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd35006f3f0 0x7fd350157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.916+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 >> 172.21.15.28:0/3293989363 conn(0x7fd3500b0270 msgr2=0x7fd3500b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.917+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 shutdown_connections 2024-09-13T17:18:58.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:58.917+0000 7fd355800640 1 -- 172.21.15.28:0/3293989363 wait complete. 2024-09-13T17:18:58.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:18:59.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.088+0000 7f8d61000640 1 Processor -- start 2024-09-13T17:18:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.089+0000 7f8d61000640 1 -- start start 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d5c159440 0x7f8d5c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159820 0x7f8d5c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c3850 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c15a930 con 0x7f8d5c159440 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d61000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0b2cd0 con 0x7f8d5c159820 2024-09-13T17:18:59.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d5be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d5b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d5c159440 0x7f8d5c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d52a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159820 0x7f8d5c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d5be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49678/0 (socket says 172.21.15.28:49678) 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.090+0000 7f8d5be00640 1 -- 172.21.15.28:0/1714361503 learned_addr learned my addr 172.21.15.28:0/1714361503 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.091+0000 7f8d5be00640 1 -- 172.21.15.28:0/1714361503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159820 msgr2=0x7f8d5c0b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.091+0000 7f8d5be00640 1 --2- 172.21.15.28:0/1714361503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159820 0x7f8d5c0b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.091+0000 7f8d5be00640 1 -- 172.21.15.28:0/1714361503 >> v2:172.21.15.189:3300/0 conn(0x7f8d5c159440 msgr2=0x7f8d5c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.091+0000 7f8d5be00640 1 --2- 172.21.15.28:0/1714361503 >> v2:172.21.15.189:3300/0 conn(0x7f8d5c159440 0x7f8d5c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.091+0000 7f8d5be00640 1 -- 172.21.15.28:0/1714361503 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c159d20 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d5be00640 1 --2- 172.21.15.28:0/1714361503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c06d4d0 tx=0x7f8d5402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f8b32a0b959e92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d5aa00640 1 -- 172.21.15.28:0/1714361503 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54031600 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d5aa00640 1 -- 172.21.15.28:0/1714361503 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5402d050 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d5aa00640 1 -- 172.21.15.28:0/1714361503 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d54031980 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d61000640 1 -- 172.21.15.28:0/1714361503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c0b22e0 msgr2=0x7f8d5c0b2790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.092+0000 7f8d61000640 1 --2- 172.21.15.28:0/1714361503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c06d4d0 tx=0x7f8d5402a730 comp rx=0 tx=0).stop 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 -- 172.21.15.28:0/1714361503 shutdown_connections 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 --2- 172.21.15.28:0/1714361503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c0b2790 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 --2- 172.21.15.28:0/1714361503 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159820 0x7f8d5c0b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 --2- 172.21.15.28:0/1714361503 >> v2:172.21.15.189:3300/0 conn(0x7f8d5c159440 0x7f8d5c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 -- 172.21.15.28:0/1714361503 >> 172.21.15.28:0/1714361503 conn(0x7f8d5c072e40 msgr2=0x7f8d5c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 -- 172.21.15.28:0/1714361503 shutdown_connections 2024-09-13T17:18:59.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.093+0000 7f8d61000640 1 -- 172.21.15.28:0/1714361503 wait complete. 2024-09-13T17:18:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.094+0000 7f8d61000640 1 Processor -- start 2024-09-13T17:18:59.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.094+0000 7f8d61000640 1 -- start start 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 0x7f8d5c070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c157ee0 con 0x7f8d5c159820 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1f2110 con 0x7f8d5c0b22e0 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d61000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c6f40 con 0x7f8d5c159440 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.095+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49680/0 (socket says 172.21.15.28:49680) 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 -- 172.21.15.28:0/3267220028 learned_addr learned my addr 172.21.15.28:0/3267220028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5b400640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d52a00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 0x7f8d5c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 -- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 msgr2=0x7f8d5c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 0x7f8d5c070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 -- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 msgr2=0x7f8d5c1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 -- 172.21.15.28:0/3267220028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54009b30 con 0x7f8d5c159820 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5b400640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d52a00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 0x7f8d5c070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d5be00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f8d54034040 tx=0x7f8d54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.096+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5403b820 con 0x7f8d5c159820 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.097+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5403b9a0 con 0x7f8d5c159820 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.097+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5403bd50 con 0x7f8d5c159820 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.097+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c1f3110 con 0x7f8d5c159820 2024-09-13T17:18:59.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.097+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c1f2630 con 0x7f8d5c159820 2024-09-13T17:18:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.098+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c0b2790 con 0x7f8d5c159820 2024-09-13T17:18:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.099+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d54038050 con 0x7f8d5c159820 2024-09-13T17:18:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.099+0000 7f8d53e00640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 0x7f8d3806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.099+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8d540a9c80 con 0x7f8d5c159820 2024-09-13T17:18:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.100+0000 7f8d5b400640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 0x7f8d3806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.100+0000 7f8d5b400640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 0x7f8d3806d0c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b240 tx=0x7f8d4c00b3a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.103+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8d54074e50 con 0x7f8d5c159820 2024-09-13T17:18:59.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.318+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d5c15c440 con 0x7f8d5c159820 2024-09-13T17:18:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.319+0000 7f8d53e00640 1 -- 172.21.15.28:0/3267220028 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f8d5404b090 con 0x7f8d5c159820 2024-09-13T17:18:59.321 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:59.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:18:59.321 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 msgr2=0x7f8d3806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 0x7f8d3806d0c0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c00b240 tx=0x7f8d4c00b3a0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 msgr2=0x7f8d5c070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f8d54034040 tx=0x7f8d54004a40 comp rx=0 tx=0).stop 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 shutdown_connections 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d3806ac00 0x7f8d3806d0c0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d5c159820 0x7f8d5c070680 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d5c159440 0x7f8d5c070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 --2- 172.21.15.28:0/3267220028 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d5c0b22e0 0x7f8d5c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 >> 172.21.15.28:0/3267220028 conn(0x7f8d5c072e40 msgr2=0x7f8d5c15da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:59.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 shutdown_connections 2024-09-13T17:18:59.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.321+0000 7f8d61000640 1 -- 172.21.15.28:0/3267220028 wait complete. 2024-09-13T17:18:59.338 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:18:59.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:18:59.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.491+0000 7fc7b7a00640 1 Processor -- start 2024-09-13T17:18:59.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.491+0000 7fc7b7a00640 1 -- start start 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.492+0000 7fc7b7a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7b00cabc0 0x7fc7b006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.492+0000 7fc7b7a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc7b0070cc0 0x7fc7b0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7b7a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7b015e100 con 0x7fc7b015a540 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7b7a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b00adb00 con 0x7fc7b00cabc0 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7b7a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7b0071670 con 0x7fc7b0070cc0 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7b00cabc0 0x7fc7b006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc7b00cabc0 0x7fc7b006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49690/0 (socket says 172.21.15.28:49690) 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 -- 172.21.15.28:0/2447162378 learned_addr learned my addr 172.21.15.28:0/2447162378 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:59.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7b5a00640 1 --2- 172.21.15.28:0/2447162378 >> v2:172.21.15.189:3300/0 conn(0x7fc7b0070cc0 0x7fc7b0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 -- 172.21.15.28:0/2447162378 >> v2:172.21.15.189:3300/0 conn(0x7fc7b0070cc0 msgr2=0x7fc7b0071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:18:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 --2- 172.21.15.28:0/2447162378 >> v2:172.21.15.189:3300/0 conn(0x7fc7b0070cc0 0x7fc7b0071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 -- 172.21.15.28:0/2447162378 >> v1:172.21.15.189:6789/0 conn(0x7fc7b015a540 legacy=0x7fc7b015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.493+0000 7fc7afe00640 1 -- 172.21.15.28:0/2447162378 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7b015e3e0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.494+0000 7fc7afe00640 1 --2- 172.21.15.28:0/2447162378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b006c430 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0070900 tx=0x7fc79c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4a3f5235df4e1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.494+0000 7fc7af400640 1 -- 172.21.15.28:0/2447162378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c032440 con 0x7fc7b00cabc0 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7af400640 1 -- 172.21.15.28:0/2447162378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c0325a0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7af400640 1 -- 172.21.15.28:0/2447162378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c032930 con 0x7fc7b00cabc0 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 -- 172.21.15.28:0/2447162378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 msgr2=0x7fc7b006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/2447162378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b006c430 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0070900 tx=0x7fc79c02a230 comp rx=0 tx=0).stop 2024-09-13T17:18:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 -- 172.21.15.28:0/2447162378 shutdown_connections 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/2447162378 >> v2:172.21.15.189:3300/0 conn(0x7fc7b0070cc0 0x7fc7b0071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/2447162378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b006c430 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 -- 172.21.15.28:0/2447162378 >> 172.21.15.28:0/2447162378 conn(0x7fc7b0069e80 msgr2=0x7fc7b006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 -- 172.21.15.28:0/2447162378 shutdown_connections 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.495+0000 7fc7b7a00640 1 -- 172.21.15.28:0/2447162378 wait complete. 2024-09-13T17:18:59.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.497+0000 7fc7b7a00640 1 Processor -- start 2024-09-13T17:18:59.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.497+0000 7fc7b7a00640 1 -- start start 2024-09-13T17:18:59.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 0x7fc7b014c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 0x7fc7b00acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b014c9c0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b014e180 con 0x7fc7b015a540 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b7a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7b014e460 con 0x7fc7b0070cc0 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49704/0 (socket says 172.21.15.28:49704) 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7afe00640 1 -- 172.21.15.28:0/3395264098 learned_addr learned my addr 172.21.15.28:0/3395264098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b5000640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 0x7fc7b014c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.498+0000 7fc7b5a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 0x7fc7b00acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 -- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 msgr2=0x7fc7b014c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 0x7fc7b014c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 -- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 msgr2=0x7fc7b00acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 0x7fc7b00acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 -- 172.21.15.28:0/3395264098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79c0094d0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7b5a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 0x7fc7b00acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7b5000640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 0x7fc7b014c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:18:59.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7afe00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fc79c000c00 tx=0x7fc79c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c032440 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c030b80 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7b014e740 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c03d440 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.499+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7b014ed00 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.500+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7b0071130 con 0x7fc7b00cabc0 2024-09-13T17:18:59.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.501+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc79c039040 con 0x7fc7b00cabc0 2024-09-13T17:18:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.502+0000 7fc7acc00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 0x7fc79006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:18:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.502+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc79c0aa060 con 0x7fc7b00cabc0 2024-09-13T17:18:59.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.504+0000 7fc7b5000640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 0x7fc79006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:18:59.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.504+0000 7fc7b5000640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 0x7fc79006d0c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4007fd0 tx=0x7fc7a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:18:59.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.507+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc79c0dd9c0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.864+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc7b014e5a0 con 0x7fc7b00cabc0 2024-09-13T17:18:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.864+0000 7fc7acc00640 1 -- 172.21.15.28:0/3395264098 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1808 (secure 0 0 0) 0x7fc79c041720 con 0x7fc7b00cabc0 2024-09-13T17:18:59.866 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:18:59.866 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":673,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076572536,"bytes_used":2698801152,"bytes_avail":770395312128,"bytes_total":773094113280,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":2227669,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":24745461,"read_op_per_sec":0,"write_op_per_sec":24},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-09-13T17:18:57.048172+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.866+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 msgr2=0x7fc79006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.866+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 0x7fc79006d0c0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fc7a4007fd0 tx=0x7fc7a4009040 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.866+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 msgr2=0x7fc7b00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.866+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fc79c000c00 tx=0x7fc79c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 shutdown_connections 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc79006ac00 0x7fc79006d0c0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc7b015a540 0x7fc7b00acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc7b00cabc0 0x7fc7b00ac850 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 --2- 172.21.15.28:0/3395264098 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7b0070cc0 0x7fc7b014c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:18:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 >> 172.21.15.28:0/3395264098 conn(0x7fc7b0069e80 msgr2=0x7fc7b00cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:18:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 shutdown_connections 2024-09-13T17:18:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:18:59.867+0000 7fc7b7a00640 1 -- 172.21.15.28:0/3395264098 wait complete. 2024-09-13T17:18:59.885 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-09-13T17:18:59.885 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub resume 2024-09-13T17:19:00.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.041+0000 7f2b73200640 1 Processor -- start 2024-09-13T17:19:00.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.042+0000 7f2b73200640 1 -- start start 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b73200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2b74143910 0x7f2b74070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b73200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b740cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b73200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b74071630 con 0x7f2b74143320 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b73200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b740cdaf0 con 0x7f2b74143910 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b73200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b74143700 con 0x7f2b740cd120 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b71e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2b74143320 0x7f2b74071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58630/0 (socket says 172.21.15.28:58630) 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b71e00640 1 -- 172.21.15.28:0/1312612976 learned_addr learned my addr 172.21.15.28:0/1312612976 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.043+0000 7f2b69400640 1 --2- 172.21.15.28:0/1312612976 >> v2:172.21.15.28:3300/0 conn(0x7f2b74143910 0x7f2b74070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b72800640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b740cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b71400640 1 -- 172.21.15.28:0/1312612976 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278488586 0 0) 0x7f2b74071630 con 0x7f2b74143320 2024-09-13T17:19:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b69400640 1 -- 172.21.15.28:0/1312612976 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 msgr2=0x7f2b740cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b69400640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b740cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b69400640 1 -- 172.21.15.28:0/1312612976 >> v1:172.21.15.28:6789/0 conn(0x7f2b74143320 legacy=0x7f2b74071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b69400640 1 -- 172.21.15.28:0/1312612976 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b74070a20 con 0x7f2b74143910 2024-09-13T17:19:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.044+0000 7f2b72800640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b740cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b69400640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143910 0x7f2b74070440 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f4a5d515959d58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b71400640 1 -- 172.21.15.28:0/1312612976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c032440 con 0x7f2b74143910 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b71400640 1 -- 172.21.15.28:0/1312612976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0325a0 con 0x7f2b74143910 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b71400640 1 -- 172.21.15.28:0/1312612976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c032930 con 0x7f2b74143910 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 -- 172.21.15.28:0/1312612976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143910 msgr2=0x7f2b74070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143910 0x7f2b74070440 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0097a0 tx=0x7f2b5c02a180 comp rx=0 tx=0).stop 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 -- 172.21.15.28:0/1312612976 shutdown_connections 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b740cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 --2- 172.21.15.28:0/1312612976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143910 0x7f2b74070440 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.045+0000 7f2b73200640 1 -- 172.21.15.28:0/1312612976 >> 172.21.15.28:0/1312612976 conn(0x7f2b74069e80 msgr2=0x7f2b7406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.046+0000 7f2b73200640 1 -- 172.21.15.28:0/1312612976 shutdown_connections 2024-09-13T17:19:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.046+0000 7f2b73200640 1 -- 172.21.15.28:0/1312612976 wait complete. 2024-09-13T17:19:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.047+0000 7f2b73200640 1 Processor -- start 2024-09-13T17:19:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.047+0000 7f2b73200640 1 -- start start 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b7414dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b74143910 0x7f2b74138ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7413a8b0 con 0x7f2b74143320 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7413ab90 con 0x7f2b74143910 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b73200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7413ae70 con 0x7f2b740cd120 2024-09-13T17:19:00.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.048+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49718/0 (socket says 172.21.15.28:49718) 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 -- 172.21.15.28:0/2392789602 learned_addr learned my addr 172.21.15.28:0/2392789602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b71e00640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b7414dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b72800640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b74143910 0x7f2b74138ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 -- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 msgr2=0x7f2b7414dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b7414dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 -- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b74143910 msgr2=0x7f2b74138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b74143910 0x7f2b74138ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 -- 172.21.15.28:0/2392789602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c009400 con 0x7f2b74143320 2024-09-13T17:19:00.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.049+0000 7f2b69400640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000c00 tx=0x7f2b5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.050+0000 7f2b6aa00640 1 -- 172.21.15.28:0/2392789602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c030b80 con 0x7f2b74143320 2024-09-13T17:19:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.050+0000 7f2b6aa00640 1 -- 172.21.15.28:0/2392789602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c030d00 con 0x7f2b74143320 2024-09-13T17:19:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.050+0000 7f2b6aa00640 1 -- 172.21.15.28:0/2392789602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c03d680 con 0x7f2b74143320 2024-09-13T17:19:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.050+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7413b150 con 0x7f2b74143320 2024-09-13T17:19:00.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.050+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7414e110 con 0x7f2b74143320 2024-09-13T17:19:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.052+0000 7f2b6aa00640 1 -- 172.21.15.28:0/2392789602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2b5c039040 con 0x7f2b74143320 2024-09-13T17:19:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.053+0000 7f2b6aa00640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 0x7f2b5006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.053+0000 7f2b6aa00640 1 -- 172.21.15.28:0/2392789602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b5c0a9ea0 con 0x7f2b74143320 2024-09-13T17:19:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.053+0000 7f2b71e00640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 0x7f2b5006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.053+0000 7f2b7be6d740 1 Processor -- start 2024-09-13T17:19:00.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.053+0000 7f2b7be6d740 1 -- start start 2024-09-13T17:19:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b71e00640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 0x7f2b5006d0c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 0x559c3368ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336b99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c335966a0 con 0x559c336d3d90 2024-09-13T17:19:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c335f9cc0 con 0x559c336904a0 2024-09-13T17:19:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.054+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c335f7450 con 0x559c3368b870 2024-09-13T17:19:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49720/0 (socket says 172.21.15.28:49720) 2024-09-13T17:19:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 -- 172.21.15.28:0/2034934217 learned_addr learned my addr 172.21.15.28:0/2034934217 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b71e00640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 0x559c3368ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b69400640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336b99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 -- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 msgr2=0x559c3368ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 0x559c3368ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 -- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 msgr2=0x559c336b99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336b99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 -- 172.21.15.28:0/2034934217 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c335c32e0 con 0x559c336d3d90 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b71e00640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 0x559c3368ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b69400640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.055+0000 7f2b72800640 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00e3b0 tx=0x7f2b6c00e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c3fbb222732a4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b62a00640 1 -- 172.21.15.28:0/2034934217 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00f6b0 con 0x559c336d3d90 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b62a00640 1 -- 172.21.15.28:0/2034934217 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6c013070 con 0x559c336d3d90 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b62a00640 1 -- 172.21.15.28:0/2034934217 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c00fa10 con 0x559c336d3d90 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 -- 172.21.15.28:0/2034934217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 msgr2=0x559c336b9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c00e3b0 tx=0x7f2b6c00e870 comp rx=0 tx=0).stop 2024-09-13T17:19:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 -- 172.21.15.28:0/2034934217 shutdown_connections 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336d3d90 0x559c336b9ef0 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336b99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/2034934217 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c3368b870 0x559c3368ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 -- 172.21.15.28:0/2034934217 >> 172.21.15.28:0/2034934217 conn(0x559c3368c500 msgr2=0x559c336d49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 -- 172.21.15.28:0/2034934217 shutdown_connections 2024-09-13T17:19:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.056+0000 7f2b7be6d740 1 -- 172.21.15.28:0/2034934217 wait complete. 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.057+0000 7f2b7be6d740 1 -- start start 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.057+0000 7f2b7be6d740 1 Processor -- start 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c336bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336c5f30 0x559c336af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c335f2d40 con 0x559c336904a0 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c33594280 con 0x559c336c5f30 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c33593f20 con 0x559c336d3d90 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.058+0000 7f2b7be6d740 10 client.0 _ll_register_callbacks cb 0x559c335ec3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49724/0 (socket says 172.21.15.28:49724) 2024-09-13T17:19:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 -- 172.21.15.28:0/318580769 learned_addr learned my addr 172.21.15.28:0/318580769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b69400640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c336bc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b72800640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336c5f30 0x559c336af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 -- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 msgr2=0x559c336bc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c336bc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 -- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336c5f30 msgr2=0x559c336af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336c5f30 0x559c336af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 -- 172.21.15.28:0/318580769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6c007c40 con 0x559c336904a0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b71e00640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2b64013670 tx=0x7f2b64000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9140833affcf8097 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.059+0000 7f2b57400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64012560 con 0x559c336904a0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b6401b070 con 0x559c336904a0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c332e64e0 con 0x559c336904a0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b640126c0 con 0x559c336904a0 2024-09-13T17:19:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 10 client.9762 fetch_fsmap learned FSMap version 130 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x559c335f13c0 con 0x559c336904a0 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c332e64e0 con 0x559c336904a0 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2b64012070 con 0x559c336904a0 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 10 client.9762 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 10 client.9762 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 20 client.9762 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336ebd70 0x559c336ec150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 4 client.9762 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.060+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559c332f77e0 con 0x559c336ebd70 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.061+0000 7f2b69400640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336ebd70 0x559c336ec150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.061+0000 7f2b69400640 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336ebd70 0x559c336ec150 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27eebeadb69692e9 server_cookie=e56440e49c3680d in_seq=0 out_seq=0 2024-09-13T17:19:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.061+0000 7f2b57400640 10 client.9762 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b57400640 1 -- 172.21.15.28:0/318580769 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x559c332f77e0 con 0x559c336ebd70 2024-09-13T17:19:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b57400640 10 client.9762 handle_command_reply: tid=0 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 client.9762 shutdown 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 msgr2=0x559c336c59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2b64013670 tx=0x7f2b64000ed0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 shutdown_connections 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336ebd70 0x559c336ec150 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336c5f30 0x559c336af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c336bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/318580769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336904a0 0x559c336c59f0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 >> 172.21.15.28:0/318580769 conn(0x559c3368b870 msgr2=0x559c336d8700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b57400640 0 client.9762 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 shutdown_connections 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 1 -- 172.21.15.28:0/318580769 wait complete. 2024-09-13T17:19:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.062+0000 7f2b7be6d740 20 client.9762 trim_cache size 0 max 16384 2024-09-13T17:19:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.077+0000 7f2b7be6d740 1 Processor -- start 2024-09-13T17:19:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.077+0000 7f2b7be6d740 1 -- start start 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336bda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c33739dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c336d7990 con 0x559c336c5f30 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c3335a040 con 0x559c336904a0 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c336c7750 con 0x559c336d3d90 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49732/0 (socket says 172.21.15.28:49732) 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b69400640 1 -- 172.21.15.28:0/154091672 learned_addr learned my addr 172.21.15.28:0/154091672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b71e00640 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336bda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.078+0000 7f2b72800640 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c33739dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 -- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 msgr2=0x559c33739dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c33739dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 -- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 msgr2=0x559c336bda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336bda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 -- 172.21.15.28:0/154091672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559c33593610 con 0x559c336c5f30 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b69400640 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c02a1c0 tx=0x7f2b5c03f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97baee7784127c39 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b63400640 1 -- 172.21.15.28:0/154091672 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c081100 con 0x559c336c5f30 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b63400640 1 -- 172.21.15.28:0/154091672 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b5c0a8420 con 0x559c336c5f30 2024-09-13T17:19:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.079+0000 7f2b63400640 1 -- 172.21.15.28:0/154091672 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c075420 con 0x559c336c5f30 2024-09-13T17:19:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- 172.21.15.28:0/154091672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 msgr2=0x559c336e7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c02a1c0 tx=0x7f2b5c03f040 comp rx=0 tx=0).stop 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- 172.21.15.28:0/154091672 shutdown_connections 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336d3d90 0x559c33739dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c336e7ef0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/154091672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336904a0 0x559c336bda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- 172.21.15.28:0/154091672 >> 172.21.15.28:0/154091672 conn(0x559c3368b870 msgr2=0x559c336d3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- 172.21.15.28:0/154091672 shutdown_connections 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- 172.21.15.28:0/154091672 wait complete. 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.080+0000 7f2b7be6d740 1 -- start start 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.081+0000 7f2b7be6d740 1 Processor -- start 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336904a0 0x559c337392c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 0x559c336890b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c336ef5b0 con 0x559c336c5f30 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559c336bddc0 con 0x559c336d3d90 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559c3372b220 con 0x559c336904a0 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b7be6d740 10 client.0 _ll_register_callbacks cb 0x559c33691680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49736/0 (socket says 172.21.15.28:49736) 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 -- 172.21.15.28:0/3164720940 learned_addr learned my addr 172.21.15.28:0/3164720940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 -- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336904a0 msgr2=0x559c337392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336904a0 0x559c337392c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 -- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 msgr2=0x559c336890b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b72800640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 0x559c336890b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 0x559c336890b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b69400640 1 -- 172.21.15.28:0/3164720940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7414e110 con 0x559c336c5f30 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.082+0000 7f2b72800640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 0x559c336890b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b69400640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c075210 tx=0x7f2b64013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=653a125a7a2f4dbd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c078ae0 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f2b5c081100 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559c336ecc50 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b5c078f70 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b7be6d740 10 client.9768 fetch_fsmap learned FSMap version 130 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x559c335b8b20 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559c336ecc50 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2b5c079d20 con 0x559c336c5f30 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b7be6d740 10 client.9768 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.083+0000 7f2b7be6d740 10 client.9768 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b7be6d740 20 client.9768 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336eedd0 0x559c336895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b7be6d740 4 client.9768 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-09-13T17:19:00.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub resume"}) v1 -- 0x7f2b6c013070 con 0x559c336eedd0 2024-09-13T17:19:00.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b71e00640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336eedd0 0x559c336895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b71e00640 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336eedd0 0x559c336895f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f36ea5cd7cd7e1bd server_cookie=ae680a6ea76c4f87 in_seq=0 out_seq=0 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.084+0000 7f2b57400640 10 client.9768 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b57400640 1 -- 172.21.15.28:0/3164720940 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x7f2b6c013070 con 0x559c336eedd0 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b57400640 10 client.9768 handle_command_reply: tid=0 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 client.9768 shutdown 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 msgr2=0x559c33725a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c075210 tx=0x7f2b64013000 comp rx=0 tx=0).stop 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 shutdown_connections 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x559c336eedd0 0x559c336895f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c336d3d90 0x559c336890b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559c336c5f30 0x559c33725a60 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b57400640 0 client.9768 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 --2- 172.21.15.28:0/3164720940 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559c336904a0 0x559c337392c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 >> 172.21.15.28:0/3164720940 conn(0x559c3368b870 msgr2=0x559c336d62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 shutdown_connections 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 1 -- 172.21.15.28:0/3164720940 wait complete. 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.085+0000 7f2b7be6d740 20 client.9768 trim_cache size 0 max 16384 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:19:00.090 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:00.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 msgr2=0x7f2b5006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 0x7f2b5006d0c0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000f60 tx=0x7f2b64007d50 comp rx=0 tx=0).stop 2024-09-13T17:19:00.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 msgr2=0x7f2b741389a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c000c00 tx=0x7f2b5c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 shutdown_connections 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2b5006ac00 0x7f2b5006d0c0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2b74143910 0x7f2b74138ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b74143320 0x7f2b741389a0 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 --2- 172.21.15.28:0/2392789602 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2b740cd120 0x7f2b7414dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 >> 172.21.15.28:0/2392789602 conn(0x7f2b74069e80 msgr2=0x7f2b74071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 shutdown_connections 2024-09-13T17:19:00.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.086+0000 7f2b73200640 1 -- 172.21.15.28:0/2392789602 wait complete. 2024-09-13T17:19:00.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.10:0 scrub status 2024-09-13T17:19:00.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.259+0000 7f21d7e00640 1 Processor -- start 2024-09-13T17:19:00.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.260+0000 7f21d7e00640 1 -- start start 2024-09-13T17:19:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21d8143320 0x7f21d8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21d80cd120 0x7f21d80cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21d8071630 con 0x7f21d8143910 2024-09-13T17:19:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d80cdad0 con 0x7f21d8143320 2024-09-13T17:19:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21d8143700 con 0x7f21d80cd120 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21d8143320 0x7f21d8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21d8143320 0x7f21d8070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49738/0 (socket says 172.21.15.28:49738) 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d6a00640 1 -- 172.21.15.28:0/2384075909 learned_addr learned my addr 172.21.15.28:0/2384075909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.261+0000 7f21d7400640 1 --2- 172.21.15.28:0/2384075909 >> v2:172.21.15.189:3300/0 conn(0x7f21d80cd120 0x7f21d80cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.262+0000 7f21d6a00640 1 -- 172.21.15.28:0/2384075909 >> v2:172.21.15.189:3300/0 conn(0x7f21d80cd120 msgr2=0x7f21d80cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.262+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2384075909 >> v2:172.21.15.189:3300/0 conn(0x7f21d80cd120 0x7f21d80cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.262+0000 7f21d6a00640 1 -- 172.21.15.28:0/2384075909 >> v1:172.21.15.189:6789/0 conn(0x7f21d8143910 legacy=0x7f21d8071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.262+0000 7f21d6a00640 1 -- 172.21.15.28:0/2384075909 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21d8070a20 con 0x7f21d8143320 2024-09-13T17:19:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.263+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2384075909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143320 0x7f21d8070440 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f21d80ce320 tx=0x7f21c802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef87d6ffc06f092 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.263+0000 7f21d6000640 1 -- 172.21.15.28:0/2384075909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8032440 con 0x7f21d8143320 2024-09-13T17:19:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.263+0000 7f21d6000640 1 -- 172.21.15.28:0/2384075909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c80325a0 con 0x7f21d8143320 2024-09-13T17:19:00.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.263+0000 7f21d6000640 1 -- 172.21.15.28:0/2384075909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c8032930 con 0x7f21d8143320 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 -- 172.21.15.28:0/2384075909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143320 msgr2=0x7f21d8070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 --2- 172.21.15.28:0/2384075909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143320 0x7f21d8070440 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f21d80ce320 tx=0x7f21c802a270 comp rx=0 tx=0).stop 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 -- 172.21.15.28:0/2384075909 shutdown_connections 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 --2- 172.21.15.28:0/2384075909 >> v2:172.21.15.189:3300/0 conn(0x7f21d80cd120 0x7f21d80cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 --2- 172.21.15.28:0/2384075909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143320 0x7f21d8070440 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 -- 172.21.15.28:0/2384075909 >> 172.21.15.28:0/2384075909 conn(0x7f21d8069e80 msgr2=0x7f21d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 -- 172.21.15.28:0/2384075909 shutdown_connections 2024-09-13T17:19:00.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.264+0000 7f21d7e00640 1 -- 172.21.15.28:0/2384075909 wait complete. 2024-09-13T17:19:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.265+0000 7f21d7e00640 1 Processor -- start 2024-09-13T17:19:00.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 -- start start 2024-09-13T17:19:00.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21d80cd120 0x7f21d80cb040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8143320 0x7f21d81e5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80ccc00 con 0x7f21d8143910 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80cc230 con 0x7f21d8143320 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.266+0000 7f21d7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21d80cb620 con 0x7f21d80cd120 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49740/0 (socket says 172.21.15.28:49740) 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 -- 172.21.15.28:0/194445531 learned_addr learned my addr 172.21.15.28:0/194445531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21cfe00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8143320 0x7f21d81e5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d6a00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21d80cd120 0x7f21d80cb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 -- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21d80cd120 msgr2=0x7f21d80cb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21d80cd120 0x7f21d80cb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 -- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8143320 msgr2=0x7f21d81e5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8143320 0x7f21d81e5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 -- 172.21.15.28:0/194445531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c80094d0 con 0x7f21d8143910 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.267+0000 7f21d7400640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f21d000efc0 tx=0x7f21d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.268+0000 7f21cea00640 1 -- 172.21.15.28:0/194445531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d0009ba0 con 0x7f21d8143910 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.268+0000 7f21cea00640 1 -- 172.21.15.28:0/194445531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d000f040 con 0x7f21d8143910 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.268+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21d80cb900 con 0x7f21d8143910 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.268+0000 7f21cea00640 1 -- 172.21.15.28:0/194445531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00146c0 con 0x7f21d8143910 2024-09-13T17:19:00.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.268+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80cbec0 con 0x7f21d8143910 2024-09-13T17:19:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.270+0000 7f21cea00640 1 -- 172.21.15.28:0/194445531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f21d0014820 con 0x7f21d8143910 2024-09-13T17:19:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.271+0000 7f21cea00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 0x7f21b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.271+0000 7f21cea00640 1 -- 172.21.15.28:0/194445531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21d008a120 con 0x7f21d8143910 2024-09-13T17:19:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.271+0000 7f21d6a00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 0x7f21b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.271+0000 7f21e067d740 1 Processor -- start 2024-09-13T17:19:00.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.271+0000 7f21e067d740 1 -- start start 2024-09-13T17:19:00.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21d6a00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 0x7f21b406d0c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f21c8034000 tx=0x7f21c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e665a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e66a77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e65606a0 con 0x55f3e6657590 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e65c3cc0 con 0x55f3e6681280 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.272+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3e65c1450 con 0x55f3e6659dd0 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49754/0 (socket says 172.21.15.28:49754) 2024-09-13T17:19:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 -- 172.21.15.28:0/3527878287 learned_addr learned my addr 172.21.15.28:0/3527878287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21d6a00640 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e665a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21d7400640 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e66a77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 -- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 msgr2=0x55f3e665a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e665a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 -- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 msgr2=0x55f3e66a77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e66a77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 -- 172.21.15.28:0/3527878287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3e658d2e0 con 0x55f3e6657590 2024-09-13T17:19:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21cfe00640 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f21c0001680 tx=0x7f21c0007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b82894afffdd105 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21c7400640 1 -- 172.21.15.28:0/3527878287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0001d20 con 0x55f3e6657590 2024-09-13T17:19:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21c7400640 1 -- 172.21.15.28:0/3527878287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c00052e0 con 0x55f3e6657590 2024-09-13T17:19:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.273+0000 7f21c7400640 1 -- 172.21.15.28:0/3527878287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c000f9d0 con 0x55f3e6657590 2024-09-13T17:19:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 -- 172.21.15.28:0/3527878287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 msgr2=0x55f3e66a72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f21c0001680 tx=0x7f21c0007e30 comp rx=0 tx=0).stop 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 -- 172.21.15.28:0/3527878287 shutdown_connections 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e66a77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6657590 0x55f3e66a72a0 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 --2- 172.21.15.28:0/3527878287 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e665a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 -- 172.21.15.28:0/3527878287 >> 172.21.15.28:0/3527878287 conn(0x55f3e6656500 msgr2=0x55f3e669e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 -- 172.21.15.28:0/3527878287 shutdown_connections 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.274+0000 7f21e067d740 1 -- 172.21.15.28:0/3527878287 wait complete. 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.275+0000 7f21e067d740 1 -- start start 2024-09-13T17:19:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.275+0000 7f21e067d740 1 Processor -- start 2024-09-13T17:19:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e6682940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6682e80 0x55f3e66a4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e65bcd40 con 0x55f3e6681280 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e655e280 con 0x55f3e6659dd0 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3e655df20 con 0x55f3e6682e80 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21e067d740 10 client.0 _ll_register_callbacks cb 0x55f3e65b63b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21cfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49760/0 (socket says 172.21.15.28:49760) 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.276+0000 7f21cfe00640 1 -- 172.21.15.28:0/2581890079 learned_addr learned my addr 172.21.15.28:0/2581890079 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21d7400640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6682e80 0x55f3e66a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e6682940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 -- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6682e80 msgr2=0x55f3e66a4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6682e80 0x55f3e66a4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 -- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 msgr2=0x55f3e6682940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e6682940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 -- 172.21.15.28:0/2581890079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d8071630 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21cfe00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f21c000f5a0 tx=0x7f21c0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb5b938c0528336 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c0001d20 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21c00069e0 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.277+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f3e62b04e0 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21c0006ba0 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 10 client.9780 fetch_fsmap learned FSMap version 130 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55f3e65bb3c0 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f3e62b04e0 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f21c0012070 con 0x55f3e6681280 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 10 client.9780 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:19:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 10 client.9780 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:19:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 20 client.9780 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e667f9e0 0x55f3e667fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 4 client.9780 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3e62c17e0 con 0x55f3e667f9e0 2024-09-13T17:19:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.278+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e667f9e0 0x55f3e667fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.279+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e667f9e0 0x55f3e667fdc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=579f7507ecff60a server_cookie=b722ab0aad8edb39 in_seq=0 out_seq=0 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.279+0000 7f21bbe00640 10 client.9780 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21bbe00640 1 -- 172.21.15.28:0/2581890079 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55f3e62c17e0 con 0x55f3e667f9e0 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21bbe00640 10 client.9780 handle_command_reply: tid=0 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 client.9780 shutdown 2024-09-13T17:19:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 msgr2=0x55f3e66a4120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f21c000f5a0 tx=0x7f21c0000ed0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 shutdown_connections 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e667f9e0 0x55f3e667fdc0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6682e80 0x55f3e66a4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6681280 0x55f3e66a4120 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 --2- 172.21.15.28:0/2581890079 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e6682940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21bbe00640 0 client.9780 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 >> 172.21.15.28:0/2581890079 conn(0x55f3e6657590 msgr2=0x55f3e66a1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 shutdown_connections 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 1 -- 172.21.15.28:0/2581890079 wait complete. 2024-09-13T17:19:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.280+0000 7f21e067d740 20 client.9780 trim_cache size 0 max 16384 2024-09-13T17:19:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.295+0000 7f21e067d740 1 Processor -- start 2024-09-13T17:19:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.295+0000 7f21e067d740 1 -- start start 2024-09-13T17:19:00.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e665a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6681280 0x55f3e66b7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e6324040 con 0x55f3e6682e80 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e6680df0 con 0x55f3e6659dd0 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3e62f6a60 con 0x55f3e6681280 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49772/0 (socket says 172.21.15.28:49772) 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21d7400640 1 -- 172.21.15.28:0/1606615900 learned_addr learned my addr 172.21.15.28:0/1606615900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21d6a00640 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e665a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.296+0000 7f21cfe00640 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6681280 0x55f3e66b7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 -- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6681280 msgr2=0x55f3e66b7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6681280 0x55f3e66b7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 -- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 msgr2=0x55f3e665a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e665a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 -- 172.21.15.28:0/1606615900 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3e655d610 con 0x55f3e6682e80 2024-09-13T17:19:00.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21d7400640 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f21d000ef90 tx=0x7f21d005f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ce84b8a99e7260 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21c7e00640 1 -- 172.21.15.28:0/1606615900 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00612f0 con 0x55f3e6682e80 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21c7e00640 1 -- 172.21.15.28:0/1606615900 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d0057350 con 0x55f3e6682e80 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.297+0000 7f21c7e00640 1 -- 172.21.15.28:0/1606615900 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00584a0 con 0x55f3e6682e80 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 -- 172.21.15.28:0/1606615900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 msgr2=0x55f3e66a18e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f21d000ef90 tx=0x7f21d005f040 comp rx=0 tx=0).stop 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 -- 172.21.15.28:0/1606615900 shutdown_connections 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e66a18e0 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6681280 0x55f3e66b7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 --2- 172.21.15.28:0/1606615900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6659dd0 0x55f3e665a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 -- 172.21.15.28:0/1606615900 >> 172.21.15.28:0/1606615900 conn(0x55f3e6657590 msgr2=0x55f3e66835d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 -- 172.21.15.28:0/1606615900 shutdown_connections 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.298+0000 7f21e067d740 1 -- 172.21.15.28:0/1606615900 wait complete. 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.299+0000 7f21e067d740 1 -- start start 2024-09-13T17:19:00.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.299+0000 7f21e067d740 1 Processor -- start 2024-09-13T17:19:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e66a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e6700f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e67014a0 con 0x55f3e6682e80 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3e66823f0 con 0x55f3e6681280 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f3e66eab40 con 0x55f3e6659dd0 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21e067d740 10 client.0 _ll_register_callbacks cb 0x7f21d0055370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49784/0 (socket says 172.21.15.28:49784) 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 -- 172.21.15.28:0/2640398476 learned_addr learned my addr 172.21.15.28:0/2640398476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e66a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21cfe00640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e6700f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 -- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 msgr2=0x55f3e66a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e66a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 -- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 msgr2=0x55f3e6700f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e6700f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d7400640 1 -- 172.21.15.28:0/2640398476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21d80cbec0 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.300+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e66a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21d7400640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c990 tx=0x7f21d005b070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f34c43645c0322 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d00612f0 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f21d0060e60 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21c00052e0 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=130) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d0061020 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 10 client.9786 fetch_fsmap learned FSMap version 130 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=130,monmap=2+}) v3 -- 0x55f3e6582b20 con 0x55f3e6682e80 2024-09-13T17:19:00.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21c00052e0 con 0x55f3e6682e80 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 130) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f21d0024020 con 0x55f3e6682e80 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 10 client.9786 fetch_fsmap finished waiting for FSMap version 130 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 10 client.9786 resolve_mds: resolved 10:0 to role '10:0' aka daemon mds.c 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 20 client.9786 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e66fe770 0x55f3e6681660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 4 client.9786 mds_command: new command op to 9420 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.301+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 --> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f3e6314e30 con 0x55f3e66fe770 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e66fe770 0x55f3e6681660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21d6a00640 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e66fe770 0x55f3e6681660 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cfd2a72cde873d0 server_cookie=396adedf48f1894d in_seq=0 out_seq=0 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21bbe00640 10 client.9786 ms_handle_connect on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21bbe00640 1 -- 172.21.15.28:0/2640398476 <== mds.0 v2:172.21.15.28:6833/3564905441 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f3e6314e30 con 0x55f3e66fe770 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21bbe00640 10 client.9786 handle_command_reply: tid=0 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.302+0000 7f21e067d740 1 client.9786 shutdown 2024-09-13T17:19:00.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 msgr2=0x55f3e669f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f21d000c990 tx=0x7f21d005b070 comp rx=0 tx=0).stop 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 shutdown_connections 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:6833/3564905441,v1:172.21.15.28:6835/3564905441] conn(0x55f3e66fe770 0x55f3e6681660 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f3e6682e80 0x55f3e669f660 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3e6681280 0x55f3e6700f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21bbe00640 0 client.9786 ms_handle_reset on v2:172.21.15.28:6833/3564905441 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 --2- 172.21.15.28:0/2640398476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55f3e6659dd0 0x55f3e66a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 >> 172.21.15.28:0/2640398476 conn(0x55f3e6657590 msgr2=0x55f3e66b7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 shutdown_connections 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 1 -- 172.21.15.28:0/2640398476 wait complete. 2024-09-13T17:19:00.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21e067d740 20 client.9786 trim_cache size 0 max 16384 2024-09-13T17:19:00.309 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:00.309 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:19:00.309 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:19:00.309 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 msgr2=0x7f21b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.303+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 0x7f21b406d0c0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f21c8034000 tx=0x7f21c802e000 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 msgr2=0x7f21d80cc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f21d000efc0 tx=0x7f21d000c490 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 shutdown_connections 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f21b406ac00 0x7f21b406d0c0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21d8143910 0x7f21d80cc6c0 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21d8143320 0x7f21d81e5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 --2- 172.21.15.28:0/194445531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21d80cd120 0x7f21d80cb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 >> 172.21.15.28:0/194445531 conn(0x7f21d8069e80 msgr2=0x7f21d80b1930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 shutdown_connections 2024-09-13T17:19:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.304+0000 7f21d7e00640 1 -- 172.21.15.28:0/194445531 wait complete. 2024-09-13T17:19:00.320 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:19:00.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.476+0000 7f10c1600640 1 Processor -- start 2024-09-13T17:19:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.476+0000 7f10c1600640 1 -- start start 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c1600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f10bc154a90 0x7f10bc15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c1600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10bc0b0b50 0x7f10bc0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c1600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10bc15bb10 con 0x7f10bc06b360 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c1600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc06b900 con 0x7f10bc154a90 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c1600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10bc06bbe0 con 0x7f10bc0b0b50 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f10bc154a90 0x7f10bc15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f10bc154a90 0x7f10bc15b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49796/0 (socket says 172.21.15.28:49796) 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 -- 172.21.15.28:0/13028255 learned_addr learned my addr 172.21.15.28:0/13028255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10c0c00640 1 --2- 172.21.15.28:0/13028255 >> v2:172.21.15.189:3300/0 conn(0x7f10bc0b0b50 0x7f10bc0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 -- 172.21.15.28:0/13028255 >> v2:172.21.15.189:3300/0 conn(0x7f10bc0b0b50 msgr2=0x7f10bc0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 --2- 172.21.15.28:0/13028255 >> v2:172.21.15.189:3300/0 conn(0x7f10bc0b0b50 0x7f10bc0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.478+0000 7f10bbe00640 1 -- 172.21.15.28:0/13028255 >> v1:172.21.15.189:6789/0 conn(0x7f10bc06b360 legacy=0x7f10bc0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.479+0000 7f10bbe00640 1 -- 172.21.15.28:0/13028255 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc06bec0 con 0x7f10bc154a90 2024-09-13T17:19:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10bbe00640 1 --2- 172.21.15.28:0/13028255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc15b5d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0b14d0 tx=0x7f10ac02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1304f36f0b69ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10baa00640 1 -- 172.21.15.28:0/13028255 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac032440 con 0x7f10bc154a90 2024-09-13T17:19:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10baa00640 1 -- 172.21.15.28:0/13028255 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac0325a0 con 0x7f10bc154a90 2024-09-13T17:19:00.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10baa00640 1 -- 172.21.15.28:0/13028255 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac032930 con 0x7f10bc154a90 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10c1600640 1 -- 172.21.15.28:0/13028255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 msgr2=0x7f10bc15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.480+0000 7f10c1600640 1 --2- 172.21.15.28:0/13028255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc15b5d0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0b14d0 tx=0x7f10ac02a270 comp rx=0 tx=0).stop 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 -- 172.21.15.28:0/13028255 shutdown_connections 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 --2- 172.21.15.28:0/13028255 >> v2:172.21.15.189:3300/0 conn(0x7f10bc0b0b50 0x7f10bc0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 --2- 172.21.15.28:0/13028255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc15b5d0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 -- 172.21.15.28:0/13028255 >> 172.21.15.28:0/13028255 conn(0x7f10bc072e40 msgr2=0x7f10bc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 -- 172.21.15.28:0/13028255 shutdown_connections 2024-09-13T17:19:00.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.481+0000 7f10c1600640 1 -- 172.21.15.28:0/13028255 wait complete. 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.482+0000 7f10c1600640 1 Processor -- start 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.482+0000 7f10c1600640 1 -- start start 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc06b360 0x7f10bc153e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10bc0b0b50 0x7f10bc0a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0a9e00 con 0x7f10bc154a90 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0a8820 con 0x7f10bc06b360 2024-09-13T17:19:00.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.483+0000 7f10c1600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10bc0a8b00 con 0x7f10bc0b0b50 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49800/0 (socket says 172.21.15.28:49800) 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 -- 172.21.15.28:0/1633837986 learned_addr learned my addr 172.21.15.28:0/1633837986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10bbe00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc06b360 0x7f10bc153e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10bb400640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10bc0b0b50 0x7f10bc0a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 -- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10bc0b0b50 msgr2=0x7f10bc0a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10bc0b0b50 0x7f10bc0a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 -- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc06b360 msgr2=0x7f10bc153e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc06b360 0x7f10bc153e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 -- 172.21.15.28:0/1633837986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0094d0 con 0x7f10bc154a90 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.484+0000 7f10c0c00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f10b400efc0 tx=0x7f10b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.485+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b4009b10 con 0x7f10bc154a90 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.485+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b400f040 con 0x7f10bc154a90 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.485+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b40146c0 con 0x7f10bc154a90 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.485+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10bc0a8de0 con 0x7f10bc154a90 2024-09-13T17:19:00.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.485+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10bc0a93a0 con 0x7f10bc154a90 2024-09-13T17:19:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.486+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10bc0a95c0 con 0x7f10bc154a90 2024-09-13T17:19:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.487+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f10b4014820 con 0x7f10bc154a90 2024-09-13T17:19:00.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.488+0000 7f10b3e00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 0x7f109c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.488+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f10b4089d60 con 0x7f10bc154a90 2024-09-13T17:19:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.488+0000 7f10bbe00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 0x7f109c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.489+0000 7f10bbe00640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 0x7f109c06d0c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f10ac034000 tx=0x7f10ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.492+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f10b4054fc0 con 0x7f10bc154a90 2024-09-13T17:19:00.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.709+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10bc0c45b0 con 0x7f10bc154a90 2024-09-13T17:19:00.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.710+0000 7f10b3e00640 1 -- 172.21.15.28:0/1633837986 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f10b4018090 con 0x7f10bc154a90 2024-09-13T17:19:00.711 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:00.711 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 msgr2=0x7f109c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 0x7f109c06d0c0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f10ac034000 tx=0x7f10ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 msgr2=0x7f10bc13b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f10b400efc0 tx=0x7f10b400c490 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 shutdown_connections 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f109c06ac00 0x7f109c06d0c0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10bc154a90 0x7f10bc13b3d0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10bc0b0b50 0x7f10bc0a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 --2- 172.21.15.28:0/1633837986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10bc06b360 0x7f10bc153e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 >> 172.21.15.28:0/1633837986 conn(0x7f10bc072e40 msgr2=0x7f10bc0cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 shutdown_connections 2024-09-13T17:19:00.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.712+0000 7f10c1600640 1 -- 172.21.15.28:0/1633837986 wait complete. 2024-09-13T17:19:00.730 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:00.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.884+0000 7fc6e2a00640 1 Processor -- start 2024-09-13T17:19:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.884+0000 7fc6e2a00640 1 -- start start 2024-09-13T17:19:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.885+0000 7fc6e2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6dc15a540 0x7fc6dc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.885+0000 7fc6e2a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc15e100 con 0x7fc6dc070cc0 2024-09-13T17:19:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.885+0000 7fc6e2a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6dc15e3e0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.885+0000 7fc6e2a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6dc0adb00 con 0x7fc6dc15a540 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.885+0000 7fc6dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6dc15a540 0x7fc6dc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.886+0000 7fc6dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc6dc15a540 0x7fc6dc06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49812/0 (socket says 172.21.15.28:49812) 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.886+0000 7fc6dbe00640 1 -- 172.21.15.28:0/3268659913 learned_addr learned my addr 172.21.15.28:0/3268659913 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.886+0000 7fc6dbe00640 1 -- 172.21.15.28:0/3268659913 >> v1:172.21.15.189:6789/0 conn(0x7fc6dc0cabc0 legacy=0x7fc6dc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.886+0000 7fc6dbe00640 1 -- 172.21.15.28:0/3268659913 >> v1:172.21.15.28:6789/0 conn(0x7fc6dc070cc0 legacy=0x7fc6dc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.886+0000 7fc6dbe00640 1 -- 172.21.15.28:0/3268659913 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6dc15ef80 con 0x7fc6dc15a540 2024-09-13T17:19:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.887+0000 7fc6dbe00640 1 --2- 172.21.15.28:0/3268659913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc15a540 0x7fc6dc06c430 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0165e14f89f04f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.887+0000 7fc6daa00640 1 -- 172.21.15.28:0/3268659913 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc032440 con 0x7fc6dc15a540 2024-09-13T17:19:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.887+0000 7fc6daa00640 1 -- 172.21.15.28:0/3268659913 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc0325a0 con 0x7fc6dc15a540 2024-09-13T17:19:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.887+0000 7fc6daa00640 1 -- 172.21.15.28:0/3268659913 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc032930 con 0x7fc6dc15a540 2024-09-13T17:19:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 -- 172.21.15.28:0/3268659913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc15a540 msgr2=0x7fc6dc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/3268659913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc15a540 0x7fc6dc06c430 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009870 tx=0x7fc6cc02a280 comp rx=0 tx=0).stop 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 -- 172.21.15.28:0/3268659913 shutdown_connections 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/3268659913 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc15a540 0x7fc6dc06c430 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 -- 172.21.15.28:0/3268659913 >> 172.21.15.28:0/3268659913 conn(0x7fc6dc069e80 msgr2=0x7fc6dc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 -- 172.21.15.28:0/3268659913 shutdown_connections 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.888+0000 7fc6e2a00640 1 -- 172.21.15.28:0/3268659913 wait complete. 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.890+0000 7fc6e2a00640 1 Processor -- start 2024-09-13T17:19:00.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.890+0000 7fc6e2a00640 1 -- start start 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 0x7fc6dc0ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 0x7fc6dc0ac880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0b1130 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc0b0e50 con 0x7fc6dc070cc0 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e2a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6dc1f6410 con 0x7fc6dc15a540 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49824/0 (socket says 172.21.15.28:49824) 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 -- 172.21.15.28:0/900046957 learned_addr learned my addr 172.21.15.28:0/900046957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6dbe00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 0x7fc6dc0ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 -- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 msgr2=0x7fc6dc0ac880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e0a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 0x7fc6dc0ac880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 0x7fc6dc0ac880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 -- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 msgr2=0x7fc6dc0ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 0x7fc6dc0ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6db400640 1 -- 172.21.15.28:0/900046957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc0094d0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6dbe00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 0x7fc6dc0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.891+0000 7fc6e0a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 0x7fc6dc0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6db400640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c450 tx=0x7fc6c800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80052d0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c8005430 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8012870 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6dc1f66f0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.892+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6dc14f1b0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.893+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6dc1f6270 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.894+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6c80129d0 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.895+0000 7fc6d3e00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 0x7fc6bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:00.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.895+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc6c8088f30 con 0x7fc6dc0cabc0 2024-09-13T17:19:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.895+0000 7fc6dbe00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 0x7fc6bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.896+0000 7fc6dbe00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 0x7fc6bc06d0c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc034000 tx=0x7fc6cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:00.899+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6dc1f6270 con 0x7fc6dc0cabc0 2024-09-13T17:19:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.145+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fc6dc1f6270 con 0x7fc6dc0cabc0 2024-09-13T17:19:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.146+0000 7fc6d3e00640 1 -- 172.21.15.28:0/900046957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1808 (secure 0 0 0) 0x7fc6dc1f6270 con 0x7fc6dc0cabc0 2024-09-13T17:19:01.147 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:01.147 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":675,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076572536,"bytes_used":2698801152,"bytes_avail":770395312128,"bytes_total":773094113280,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":2227669,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":24745461,"read_op_per_sec":0,"write_op_per_sec":24},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":29,"modified":"2024-09-13T17:18:57.048172+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused paths [/scrub_control_test_path]"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 msgr2=0x7fc6bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 0x7fc6bc06d0c0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc034000 tx=0x7fc6cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 msgr2=0x7fc6dc14e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fc6c800c450 tx=0x7fc6c800c910 comp rx=0 tx=0).stop 2024-09-13T17:19:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 shutdown_connections 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc6bc06ac00 0x7fc6bc06d0c0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6dc15a540 0x7fc6dc0ac880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6dc0cabc0 0x7fc6dc14e9f0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 --2- 172.21.15.28:0/900046957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6dc070cc0 0x7fc6dc0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 >> 172.21.15.28:0/900046957 conn(0x7fc6dc069e80 msgr2=0x7fc6dc15c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 shutdown_connections 2024-09-13T17:19:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:01.148+0000 7fc6e2a00640 1 -- 172.21.15.28:0/900046957 wait complete. 2024-09-13T17:19:01.165 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused paths [/scrub_control_test_path]'} 2024-09-13T17:19:02.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.333+0000 7ffa4d600640 1 Processor -- start 2024-09-13T17:19:02.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.333+0000 7ffa4d600640 1 -- start start 2024-09-13T17:19:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa4d600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffa48158b20 0x7ffa480afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa4d600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa480b39d0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa4d600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa480b2dc0 con 0x7ffa4806ba70 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa4d600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa480b30a0 con 0x7ffa48158b20 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa3f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffa4806c3f0 0x7ffa4806b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58654/0 (socket says 172.21.15.28:58654) 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.334+0000 7ffa3f400640 1 -- 172.21.15.28:0/3771351053 learned_addr learned my addr 172.21.15.28:0/3771351053 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3771351053 >> v2:172.21.15.189:3300/0 conn(0x7ffa48158b20 0x7ffa480afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286173722 0 0) 0x7ffa480b39d0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa2c003440 con 0x7ffa4806c3f0 2024-09-13T17:19:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47e00640 1 -- 172.21.15.28:0/3771351053 >> v1:172.21.15.189:6789/0 conn(0x7ffa4806ba70 legacy=0x7ffa48158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47e00640 1 -- 172.21.15.28:0/3771351053 >> v1:172.21.15.28:6789/0 conn(0x7ffa4806c3f0 legacy=0x7ffa4806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039724488 0 0) 0x7ffa2c003440 con 0x7ffa4806c3f0 2024-09-13T17:19:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.335+0000 7ffa47e00640 1 -- 172.21.15.28:0/3771351053 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa480b3380 con 0x7ffa48158b20 2024-09-13T17:19:02.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.336+0000 7ffa47e00640 1 --2- 172.21.15.28:0/3771351053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa48158b20 0x7ffa480afbd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa380097d0 tx=0x7ffa3802a160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cef01e5130ef9b2b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.336+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30001670 con 0x7ffa48158b20 2024-09-13T17:19:02.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3802abf0 con 0x7ffa48158b20 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa47400640 1 -- 172.21.15.28:0/3771351053 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38032440 con 0x7ffa48158b20 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 -- 172.21.15.28:0/3771351053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa48158b20 msgr2=0x7ffa480afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 --2- 172.21.15.28:0/3771351053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa48158b20 0x7ffa480afbd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffa380097d0 tx=0x7ffa3802a160 comp rx=0 tx=0).stop 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 -- 172.21.15.28:0/3771351053 shutdown_connections 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 --2- 172.21.15.28:0/3771351053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa48158b20 0x7ffa480afbd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 -- 172.21.15.28:0/3771351053 >> 172.21.15.28:0/3771351053 conn(0x7ffa48072e40 msgr2=0x7ffa48073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 -- 172.21.15.28:0/3771351053 shutdown_connections 2024-09-13T17:19:02.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.337+0000 7ffa4d600640 1 -- 172.21.15.28:0/3771351053 wait complete. 2024-09-13T17:19:02.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.339+0000 7ffa4d600640 1 Processor -- start 2024-09-13T17:19:02.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.339+0000 7ffa4d600640 1 -- start start 2024-09-13T17:19:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 0x7ffa481509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 0x7ffa481e14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa48150ee0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480a8440 con 0x7ffa4806ba70 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa4d600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa480a86e0 con 0x7ffa48158b20 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa3f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa3f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49826/0 (socket says 172.21.15.28:49826) 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.340+0000 7ffa3f400640 1 -- 172.21.15.28:0/2530944775 learned_addr learned my addr 172.21.15.28:0/2530944775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 0x7ffa481509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 -- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 msgr2=0x7ffa481e14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa4cc00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 0x7ffa481e14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 0x7ffa481e14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 -- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 msgr2=0x7ffa481509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 0x7ffa481509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 -- 172.21.15.28:0/2530944775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa38009400 con 0x7ffa4806c3f0 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 0x7ffa481509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa4cc00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 0x7ffa481e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa3f400640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c6b0 tx=0x7ffa3000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30005ad0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa30005c30 con 0x7ffa4806c3f0 2024-09-13T17:19:02.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa30007070 con 0x7ffa4806c3f0 2024-09-13T17:19:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.341+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa480a89c0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.342+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa480a8f80 con 0x7ffa4806c3f0 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.343+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa480b0950 con 0x7ffa4806c3f0 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.344+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffa300071d0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.344+0000 7ffa44c00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 0x7ffa2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.344+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ffa30088b80 con 0x7ffa4806c3f0 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.345+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 0x7ffa2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.346+0000 7ffa47e00640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 0x7ffa2406d0c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ffa38034000 tx=0x7ffa3802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.349+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffa30053ca0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.561+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa481571a0 con 0x7ffa4806c3f0 2024-09-13T17:19:02.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.562+0000 7ffa44c00640 1 -- 172.21.15.28:0/2530944775 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7ffa30088e60 con 0x7ffa4806c3f0 2024-09-13T17:19:02.563 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:02.563 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:02.564 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 msgr2=0x7ffa2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 0x7ffa2406d0c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7ffa38034000 tx=0x7ffa3802e000 comp rx=0 tx=0).stop 2024-09-13T17:19:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 msgr2=0x7ffa481e3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000c6b0 tx=0x7ffa3000cb70 comp rx=0 tx=0).stop 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 shutdown_connections 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffa2406ac00 0x7ffa2406d0c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffa48158b20 0x7ffa481e14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa4806c3f0 0x7ffa481e3460 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 --2- 172.21.15.28:0/2530944775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa4806ba70 0x7ffa481509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 >> 172.21.15.28:0/2530944775 conn(0x7ffa48072e40 msgr2=0x7ffa48070720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 shutdown_connections 2024-09-13T17:19:02.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.564+0000 7ffa4d600640 1 -- 172.21.15.28:0/2530944775 wait complete. 2024-09-13T17:19:02.581 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:02.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:02.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.735+0000 7faf8fe00640 1 Processor -- start 2024-09-13T17:19:02.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.735+0000 7faf8fe00640 1 -- start start 2024-09-13T17:19:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.736+0000 7faf8fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf9006ce80 0x7faf90160e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.736+0000 7faf8fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf90061110 0x7faf9006d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.736+0000 7faf8fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf9006bf50 con 0x7faf9006c3e0 2024-09-13T17:19:02.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf900ccd70 con 0x7faf9006ce80 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8fe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf901613b0 con 0x7faf90061110 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf9006ce80 0x7faf90160e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf9006ce80 0x7faf90160e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49834/0 (socket says 172.21.15.28:49834) 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 -- 172.21.15.28:0/4206266047 learned_addr learned my addr 172.21.15.28:0/4206266047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf87e00640 1 --2- 172.21.15.28:0/4206266047 >> v2:172.21.15.189:3300/0 conn(0x7faf90061110 0x7faf9006d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 -- 172.21.15.28:0/4206266047 >> v2:172.21.15.189:3300/0 conn(0x7faf90061110 msgr2=0x7faf9006d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 --2- 172.21.15.28:0/4206266047 >> v2:172.21.15.189:3300/0 conn(0x7faf90061110 0x7faf9006d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 -- 172.21.15.28:0/4206266047 >> v1:172.21.15.189:6789/0 conn(0x7faf9006c3e0 legacy=0x7faf9015cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.737+0000 7faf8ea00640 1 -- 172.21.15.28:0/4206266047 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9006d7a0 con 0x7faf9006ce80 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8ea00640 1 --2- 172.21.15.28:0/4206266047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf90160e70 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7faf9004e860 tx=0x7faf8002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b2a28138cbf957 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8e000640 1 -- 172.21.15.28:0/4206266047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80032440 con 0x7faf9006ce80 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8e000640 1 -- 172.21.15.28:0/4206266047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf800325a0 con 0x7faf9006ce80 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8e000640 1 -- 172.21.15.28:0/4206266047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf80032930 con 0x7faf9006ce80 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 -- 172.21.15.28:0/4206266047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 msgr2=0x7faf90160e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 --2- 172.21.15.28:0/4206266047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf90160e70 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7faf9004e860 tx=0x7faf8002a270 comp rx=0 tx=0).stop 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 -- 172.21.15.28:0/4206266047 shutdown_connections 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 --2- 172.21.15.28:0/4206266047 >> v2:172.21.15.189:3300/0 conn(0x7faf90061110 0x7faf9006d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 --2- 172.21.15.28:0/4206266047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf90160e70 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.739+0000 7faf8fe00640 1 -- 172.21.15.28:0/4206266047 >> 172.21.15.28:0/4206266047 conn(0x7faf900cb6d0 msgr2=0x7faf900cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:02.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.740+0000 7faf8fe00640 1 -- 172.21.15.28:0/4206266047 shutdown_connections 2024-09-13T17:19:02.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.740+0000 7faf8fe00640 1 -- 172.21.15.28:0/4206266047 wait complete. 2024-09-13T17:19:02.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.741+0000 7faf8fe00640 1 Processor -- start 2024-09-13T17:19:02.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.741+0000 7faf8fe00640 1 -- start start 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf90061110 0x7faf901520b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9006c3e0 0x7faf900b3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf901525f0 con 0x7faf9006ce80 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf900b1b20 con 0x7faf9006c3e0 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf900b47f0 con 0x7faf90061110 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.742+0000 7faf8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49850/0 (socket says 172.21.15.28:49850) 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 -- 172.21.15.28:0/2483607238 learned_addr learned my addr 172.21.15.28:0/2483607238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf87e00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9006c3e0 0x7faf900b3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8ea00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf90061110 0x7faf901520b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 -- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf90061110 msgr2=0x7faf901520b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf90061110 0x7faf901520b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 -- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9006c3e0 msgr2=0x7faf900b3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9006c3e0 0x7faf900b3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:02.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 -- 172.21.15.28:0/2483607238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf800094d0 con 0x7faf9006ce80 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.743+0000 7faf8f400640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7a0 tx=0x7faf8800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.744+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003b70 con 0x7faf9006ce80 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.744+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88003cf0 con 0x7faf9006ce80 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.744+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88003ea0 con 0x7faf9006ce80 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.744+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf900b1d70 con 0x7faf9006ce80 2024-09-13T17:19:02.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.744+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf900b2330 con 0x7faf9006ce80 2024-09-13T17:19:02.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.745+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9006c850 con 0x7faf9006ce80 2024-09-13T17:19:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.746+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faf8800aa90 con 0x7faf9006ce80 2024-09-13T17:19:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.747+0000 7faf86a00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 0x7faf6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.747+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faf88089130 con 0x7faf9006ce80 2024-09-13T17:19:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.747+0000 7faf8ea00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 0x7faf6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:02.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.748+0000 7faf8ea00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 0x7faf6c06d0c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faf80034000 tx=0x7faf8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.751+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faf88054300 con 0x7faf9006ce80 2024-09-13T17:19:02.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.997+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7faf900ae570 con 0x7faf9006ce80 2024-09-13T17:19:03.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:02.998+0000 7faf86a00640 1 -- 172.21.15.28:0/2483607238 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1771 (secure 0 0 0) 0x7faf88057920 con 0x7faf9006ce80 2024-09-13T17:19:03.000 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:03.000 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076572536,"bytes_used":2698801152,"bytes_avail":770395312128,"bytes_total":773094113280,"recovering_objects_per_sec":2,"recovering_bytes_per_sec":2227592,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":29431,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":30,"modified":"2024-09-13T17:19:01.049303+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:03.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 msgr2=0x7faf6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:03.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 0x7faf6c06d0c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7faf80034000 tx=0x7faf8002e000 comp rx=0 tx=0).stop 2024-09-13T17:19:03.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 msgr2=0x7faf900c4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:03.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c7a0 tx=0x7faf8800cc60 comp rx=0 tx=0).stop 2024-09-13T17:19:03.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 shutdown_connections 2024-09-13T17:19:03.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6c06ac00 0x7faf6c06d0c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf9006ce80 0x7faf900c4440 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:03.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf9006c3e0 0x7faf900b3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 --2- 172.21.15.28:0/2483607238 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf90061110 0x7faf901520b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 >> 172.21.15.28:0/2483607238 conn(0x7faf900cb6d0 msgr2=0x7faf900ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:03.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 shutdown_connections 2024-09-13T17:19:03.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:03.000+0000 7faf8fe00640 1 -- 172.21.15.28:0/2483607238 wait complete. 2024-09-13T17:19:03.018 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:19:04.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:04.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.184+0000 7f3546800640 1 Processor -- start 2024-09-13T17:19:04.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.185+0000 7f3546800640 1 -- start start 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3546800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3540159820 0x7f3540164260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3546800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35400c3850 con 0x7f3540159440 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3546800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f354015a930 con 0x7f35400b0b50 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3546800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3540159d20 con 0x7f3540159820 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3545400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3540159440 0x7f35400b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35042/0 (socket says 172.21.15.28:35042) 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3545400640 1 -- 172.21.15.28:0/1382861620 learned_addr learned my addr 172.21.15.28:0/1382861620 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:04.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.186+0000 7f3545e00640 1 --2- 172.21.15.28:0/1382861620 >> v2:172.21.15.28:3300/0 conn(0x7f3540159820 0x7f3540164260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.187+0000 7f353fe00640 1 -- 172.21.15.28:0/1382861620 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2707724210 0 0) 0x7f35400c3850 con 0x7f3540159440 2024-09-13T17:19:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.187+0000 7f353fe00640 1 -- 172.21.15.28:0/1382861620 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3528003520 con 0x7f3540159440 2024-09-13T17:19:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.187+0000 7f3545e00640 1 -- 172.21.15.28:0/1382861620 >> v1:172.21.15.189:6789/0 conn(0x7f35400b0b50 legacy=0x7f35400b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.187+0000 7f3545e00640 1 -- 172.21.15.28:0/1382861620 >> v1:172.21.15.28:6789/0 conn(0x7f3540159440 legacy=0x7f35400b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.187+0000 7f3545e00640 1 -- 172.21.15.28:0/1382861620 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354015a000 con 0x7f3540159820 2024-09-13T17:19:04.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.188+0000 7f3545e00640 1 --2- 172.21.15.28:0/1382861620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540159820 0x7f3540164260 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f3538009d90 tx=0x7f353802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d812943ef73e8556 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.188+0000 7f353fe00640 1 -- 172.21.15.28:0/1382861620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538007740 con 0x7f3540159820 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.188+0000 7f353fe00640 1 -- 172.21.15.28:0/1382861620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35380078a0 con 0x7f3540159820 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.188+0000 7f353fe00640 1 -- 172.21.15.28:0/1382861620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3538007c30 con 0x7f3540159820 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 -- 172.21.15.28:0/1382861620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540159820 msgr2=0x7f3540164260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 --2- 172.21.15.28:0/1382861620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540159820 0x7f3540164260 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f3538009d90 tx=0x7f353802a780 comp rx=0 tx=0).stop 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 -- 172.21.15.28:0/1382861620 shutdown_connections 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 --2- 172.21.15.28:0/1382861620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3540159820 0x7f3540164260 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 -- 172.21.15.28:0/1382861620 >> 172.21.15.28:0/1382861620 conn(0x7f3540072e40 msgr2=0x7f3540073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 -- 172.21.15.28:0/1382861620 shutdown_connections 2024-09-13T17:19:04.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.189+0000 7f3546800640 1 -- 172.21.15.28:0/1382861620 wait complete. 2024-09-13T17:19:04.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.190+0000 7f3546800640 1 Processor -- start 2024-09-13T17:19:04.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.191+0000 7f3546800640 1 -- start start 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 0x7f354013fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 0x7f35401402b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35401407f0 con 0x7f35400b0b50 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f354014c920 con 0x7f3540159820 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3546800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3540140a10 con 0x7f3540159440 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36484/0 (socket says 172.21.15.28:36484) 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 -- 172.21.15.28:0/2905053809 learned_addr learned my addr 172.21.15.28:0/2905053809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545e00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 0x7f35401402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3544a00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 0x7f354013fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 -- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 msgr2=0x7f354013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 0x7f354013fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 -- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 msgr2=0x7f35401402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 0x7f35401402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 -- 172.21.15.28:0/2905053809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3538009a70 con 0x7f35400b0b50 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545e00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 0x7f35401402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3544a00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 0x7f354013fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.192+0000 7f3545400640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f353400c500 tx=0x7f353400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.193+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534005ce0 con 0x7f35400b0b50 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.193+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3534005e40 con 0x7f35400b0b50 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.193+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3534007330 con 0x7f35400b0b50 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.193+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3540140cf0 con 0x7f35400b0b50 2024-09-13T17:19:04.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.193+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35401412b0 con 0x7f35400b0b50 2024-09-13T17:19:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.194+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35400b22e0 con 0x7f35400b0b50 2024-09-13T17:19:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.195+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3534007490 con 0x7f35400b0b50 2024-09-13T17:19:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.196+0000 7f353d600640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 0x7f352006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.196+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3534088a00 con 0x7f35400b0b50 2024-09-13T17:19:04.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.196+0000 7f3544a00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 0x7f352006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.197+0000 7f3544a00640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 0x7f352006d0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f354006b760 tx=0x7f352c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.200+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3534053c70 con 0x7f35400b0b50 2024-09-13T17:19:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.412+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35401587e0 con 0x7f35400b0b50 2024-09-13T17:19:04.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.412+0000 7f353d600640 1 -- 172.21.15.28:0/2905053809 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f3534017020 con 0x7f35400b0b50 2024-09-13T17:19:04.413 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:04.413 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:04.414 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.414+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 msgr2=0x7f352006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.414+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 0x7f352006d0c0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f354006b760 tx=0x7f352c009040 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.414+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 msgr2=0x7f354006bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.414+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f353400c500 tx=0x7f353400c9c0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 shutdown_connections 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f352006ac00 0x7f352006d0c0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3540159820 0x7f35401402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3540159440 0x7f354013fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 --2- 172.21.15.28:0/2905053809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f35400b0b50 0x7f354006bc40 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 >> 172.21.15.28:0/2905053809 conn(0x7f3540072e40 msgr2=0x7f3540153ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 shutdown_connections 2024-09-13T17:19:04.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.415+0000 7f3546800640 1 -- 172.21.15.28:0/2905053809 wait complete. 2024-09-13T17:19:04.431 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:04.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:04.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.584+0000 7f0e25800640 1 Processor -- start 2024-09-13T17:19:04.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.584+0000 7f0e25800640 1 -- start start 2024-09-13T17:19:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.585+0000 7f0e25800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006f970 0x7f0e2015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.585+0000 7f0e25800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e2006c2d0 0x7f0e2006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.585+0000 7f0e25800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e20159d90 con 0x7f0e201592a0 2024-09-13T17:19:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.585+0000 7f0e25800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2015b0b0 con 0x7f0e2006c2d0 2024-09-13T17:19:04.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.585+0000 7f0e25800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2006cca0 con 0x7f0e2006f970 2024-09-13T17:19:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e2006c2d0 0x7f0e2006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e2006c2d0 0x7f0e2006c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36494/0 (socket says 172.21.15.28:36494) 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 -- 172.21.15.28:0/3688347762 learned_addr learned my addr 172.21.15.28:0/3688347762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1e000640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006f970 0x7f0e2015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 -- 172.21.15.28:0/3688347762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006f970 msgr2=0x7f0e2015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006f970 0x7f0e2015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 -- 172.21.15.28:0/3688347762 >> v1:172.21.15.189:6789/0 conn(0x7f0e201592a0 legacy=0x7f0e20159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.586+0000 7f0e1f400640 1 -- 172.21.15.28:0/3688347762 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e200b2040 con 0x7f0e2006c2d0 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.587+0000 7f0e1f400640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e2006c2d0 0x7f0e2006c760 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009e50 tx=0x7f0e1802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27bcdc812171803 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.587+0000 7f0e1d600640 1 -- 172.21.15.28:0/3688347762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18031600 con 0x7f0e2006c2d0 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.587+0000 7f0e1d600640 1 -- 172.21.15.28:0/3688347762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1802d050 con 0x7f0e2006c2d0 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.587+0000 7f0e1d600640 1 -- 172.21.15.28:0/3688347762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18031980 con 0x7f0e2006c2d0 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 -- 172.21.15.28:0/3688347762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e2006c2d0 msgr2=0x7f0e2006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e2006c2d0 0x7f0e2006c760 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009e50 tx=0x7f0e1802a710 comp rx=0 tx=0).stop 2024-09-13T17:19:04.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 -- 172.21.15.28:0/3688347762 shutdown_connections 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e2006c2d0 0x7f0e2006c760 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 --2- 172.21.15.28:0/3688347762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006f970 0x7f0e2015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 -- 172.21.15.28:0/3688347762 >> 172.21.15.28:0/3688347762 conn(0x7f0e20069e80 msgr2=0x7f0e2006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 -- 172.21.15.28:0/3688347762 shutdown_connections 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.588+0000 7f0e25800640 1 -- 172.21.15.28:0/3688347762 wait complete. 2024-09-13T17:19:04.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.590+0000 7f0e25800640 1 Processor -- start 2024-09-13T17:19:04.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.590+0000 7f0e25800640 1 -- start start 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006c2d0 0x7f0e200caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 0x7f0e200af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e200af0f0 con 0x7f0e201592a0 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2014b940 con 0x7f0e2006f970 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e25800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2014bbe0 con 0x7f0e2006c2d0 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36506/0 (socket says 172.21.15.28:36506) 2024-09-13T17:19:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 -- 172.21.15.28:0/13047821 learned_addr learned my addr 172.21.15.28:0/13047821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1e000640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 0x7f0e200af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 -- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006c2d0 msgr2=0x7f0e200caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006c2d0 0x7f0e200caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 -- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 msgr2=0x7f0e200af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 0x7f0e200af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1f400640 1 -- 172.21.15.28:0/13047821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18009b30 con 0x7f0e201592a0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.591+0000 7f0e1e000640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 0x7f0e200af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e1f400640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f0e18034040 tx=0x7f0e18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1803d860 con 0x7f0e201592a0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1803d9e0 con 0x7f0e201592a0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1803dd90 con 0x7f0e201592a0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2014bec0 con 0x7f0e201592a0 2024-09-13T17:19:04.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.592+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e201425f0 con 0x7f0e201592a0 2024-09-13T17:19:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.593+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2015a2b0 con 0x7f0e201592a0 2024-09-13T17:19:04.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.594+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e1803b400 con 0x7f0e201592a0 2024-09-13T17:19:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.595+0000 7f0e16a00640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 0x7f0e0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.595+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f0e180aa390 con 0x7f0e201592a0 2024-09-13T17:19:04.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.595+0000 7f0e1ea00640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 0x7f0e0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:04.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.596+0000 7f0e1ea00640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 0x7f0e0006d0c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0e20159f20 tx=0x7f0e10007950 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:04.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.599+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0e2015a2b0 con 0x7f0e201592a0 2024-09-13T17:19:04.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.842+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0e200aef30 con 0x7f0e201592a0 2024-09-13T17:19:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.842+0000 7f0e16a00640 1 -- 172.21.15.28:0/13047821 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1773 (secure 0 0 0) 0x7f0e200aef30 con 0x7f0e201592a0 2024-09-13T17:19:04.843 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:04.843 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":678,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1080,"data_bytes":1076576883,"bytes_used":2698764288,"bytes_avail":770395348992,"bytes_total":773094113280,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":1796884,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":91557,"read_op_per_sec":0,"write_op_per_sec":212},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":30,"modified":"2024-09-13T17:19:01.049303+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 msgr2=0x7f0e0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 0x7f0e0006d0c0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f0e20159f20 tx=0x7f0e10007950 comp rx=0 tx=0).stop 2024-09-13T17:19:04.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 msgr2=0x7f0e2014c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f0e18034040 tx=0x7f0e18004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 shutdown_connections 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e0006ac00 0x7f0e0006d0c0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e201592a0 0x7f0e2014c270 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e2006f970 0x7f0e200af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 --2- 172.21.15.28:0/13047821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e2006c2d0 0x7f0e200caa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 >> 172.21.15.28:0/13047821 conn(0x7f0e20069e80 msgr2=0x7f0e2015b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 shutdown_connections 2024-09-13T17:19:04.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:04.844+0000 7f0e25800640 1 -- 172.21.15.28:0/13047821 wait complete. 2024-09-13T17:19:04.861 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:19:05.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:06.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.026+0000 7fb0a9200640 1 Processor -- start 2024-09-13T17:19:06.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.026+0000 7fb0a9200640 1 -- start start 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a9200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb0a40b0b50 0x7fb0a40b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a40cad20 con 0x7fb0a415fb90 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a9200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a406bb40 con 0x7fb0a40b21c0 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a9200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a406be20 con 0x7fb0a40b0b50 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb0a415fb90 0x7fb0a406b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35050/0 (socket says 172.21.15.28:35050) 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a3400640 1 -- 172.21.15.28:0/311972922 learned_addr learned my addr 172.21.15.28:0/311972922 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:06.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.028+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/311972922 >> v2:172.21.15.189:3300/0 conn(0x7fb0a40b0b50 0x7fb0a40b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466860216 0 0) 0x7fb0a40cad20 con 0x7fb0a415fb90 2024-09-13T17:19:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb088003520 con 0x7fb0a415fb90 2024-09-13T17:19:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709922349 0 0) 0x7fb088003520 con 0x7fb0a415fb90 2024-09-13T17:19:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 >> v2:172.21.15.189:3300/0 conn(0x7fb0a40b0b50 msgr2=0x7fb0a40b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:06.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 --2- 172.21.15.28:0/311972922 >> v2:172.21.15.189:3300/0 conn(0x7fb0a40b0b50 0x7fb0a40b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 >> v1:172.21.15.189:6789/0 conn(0x7fb0a40b21c0 legacy=0x7fb0a40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a406c100 con 0x7fb0a415fb90 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fb094001670 con 0x7fb0a415fb90 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 >> v1:172.21.15.28:6789/0 conn(0x7fb0a415fb90 legacy=0x7fb0a406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 0x7fb088003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0880052a0 0x7fb08800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a406be20 con 0x7fb0880039c0 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb08800c1a0 con 0x7fb0880052a0 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb08800c360 con 0x7fb0880047c0 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/311972922 >> v2:172.21.15.189:3300/0 conn(0x7fb0a40b0b50 0x7fb0a40b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb0a3400640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 0x7fb088003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.029+0000 7fb09aa00640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0880052a0 0x7fb08800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3400640 1 -- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880047c0 msgr2=0x7fb088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3400640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880047c0 0x7fb088004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3400640 1 -- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0880052a0 msgr2=0x7fb08800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3400640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0880052a0 0x7fb08800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3400640 1 -- 172.21.15.28:0/311972922 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb088003520 con 0x7fb0880039c0 2024-09-13T17:19:06.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.030+0000 7fb0a3e00640 1 -- 172.21.15.28:0/311972922 reap_dead start 2024-09-13T17:19:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a3400640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 0x7fb088003de0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fb09400c0e0 tx=0x7fb0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a98fe275a79370 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094005ad0 con 0x7fb0880039c0 2024-09-13T17:19:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09402d070 con 0x7fb0880039c0 2024-09-13T17:19:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a2a00640 1 -- 172.21.15.28:0/311972922 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094007070 con 0x7fb0880039c0 2024-09-13T17:19:06.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a9200640 1 -- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 msgr2=0x7fb088003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.031+0000 7fb0a9200640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 0x7fb088003de0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7fb09400c0e0 tx=0x7fb0880057f0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.032+0000 7fb0a9200640 1 -- 172.21.15.28:0/311972922 shutdown_connections 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.032+0000 7fb0a9200640 1 --2- 172.21.15.28:0/311972922 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0880039c0 0x7fb088003de0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.032+0000 7fb0a9200640 1 -- 172.21.15.28:0/311972922 >> 172.21.15.28:0/311972922 conn(0x7fb0a4072e40 msgr2=0x7fb0a4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.032+0000 7fb0a9200640 1 -- 172.21.15.28:0/311972922 shutdown_connections 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.032+0000 7fb0a9200640 1 -- 172.21.15.28:0/311972922 wait complete. 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.033+0000 7fb0a9200640 1 Processor -- start 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 -- start start 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880039c0 0x7fb0a4153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a41520b0 0x7fb0a41f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a41f30c0 con 0x7fb0a41f2700 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4153dd0 con 0x7fb0a41520b0 2024-09-13T17:19:06.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.034+0000 7fb0a9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a41540b0 con 0x7fb0880039c0 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36534/0 (socket says 172.21.15.28:36534) 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 -- 172.21.15.28:0/1048682081 learned_addr learned my addr 172.21.15.28:0/1048682081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3400640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880039c0 0x7fb0a4153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb09aa00640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a41520b0 0x7fb0a41f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 -- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880039c0 msgr2=0x7fb0a4153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880039c0 0x7fb0a4153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 -- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a41520b0 msgr2=0x7fb0a41f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a41520b0 0x7fb0a41f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.035+0000 7fb0a3e00640 1 -- 172.21.15.28:0/1048682081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a4154390 con 0x7fb0a41f2700 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb0a3e00640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb0a406d620 tx=0x7fb09c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00e070 con 0x7fb0a41f2700 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c011e10 con 0x7fb0a41f2700 2024-09-13T17:19:06.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c003990 con 0x7fb0a41f2700 2024-09-13T17:19:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a4154670 con 0x7fb0a41f2700 2024-09-13T17:19:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.036+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a4153ac0 con 0x7fb0a41f2700 2024-09-13T17:19:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.037+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a4154b90 con 0x7fb0a41f2700 2024-09-13T17:19:06.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.038+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb09c003af0 con 0x7fb0a41f2700 2024-09-13T17:19:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.039+0000 7fb09be00640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 0x7fb08006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.039+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb09c089920 con 0x7fb0a41f2700 2024-09-13T17:19:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.040+0000 7fb0a3400640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 0x7fb08006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.041+0000 7fb0a3400640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 0x7fb08006d0c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb094032000 tx=0x7fb09402d000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.044+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb0a4154b90 con 0x7fb0a41f2700 2024-09-13T17:19:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.258+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a4067580 con 0x7fb0a41f2700 2024-09-13T17:19:06.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.258+0000 7fb09be00640 1 -- 172.21.15.28:0/1048682081 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fb09c054b80 con 0x7fb0a41f2700 2024-09-13T17:19:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:06.260 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:06.260 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.260+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 msgr2=0x7fb08006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 0x7fb08006d0c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fb094032000 tx=0x7fb09402d000 comp rx=0 tx=0).stop 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 msgr2=0x7fb0a41f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7fb0a406d620 tx=0x7fb09c0115a0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 shutdown_connections 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb08006ac00 0x7fb08006d0c0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0a41f2700 0x7fb0a41f2b80 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0a41520b0 0x7fb0a41f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 --2- 172.21.15.28:0/1048682081 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb0880039c0 0x7fb0a4153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 >> 172.21.15.28:0/1048682081 conn(0x7fb0a4072e40 msgr2=0x7fb0a406b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 shutdown_connections 2024-09-13T17:19:06.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.261+0000 7fb0a9200640 1 -- 172.21.15.28:0/1048682081 wait complete. 2024-09-13T17:19:06.278 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:06.278 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.431+0000 7f3f0e400640 1 Processor -- start 2024-09-13T17:19:06.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.431+0000 7f3f0e400640 1 -- start start 2024-09-13T17:19:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.432+0000 7f3f0e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f0806f970 0x7f3f0815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.432+0000 7f3f0e400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f0806c2d0 0x7f3f0806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.432+0000 7f3f0e400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f0806cc70 con 0x7f3f081592a0 2024-09-13T17:19:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.432+0000 7f3f0e400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f0815b0b0 con 0x7f3f0806f970 2024-09-13T17:19:06.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.432+0000 7f3f0e400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f08159d90 con 0x7f3f0806c2d0 2024-09-13T17:19:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f0806f970 0x7f3f0815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3f0806f970 0x7f3f0815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36538/0 (socket says 172.21.15.28:36538) 2024-09-13T17:19:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 -- 172.21.15.28:0/4205117584 learned_addr learned my addr 172.21.15.28:0/4205117584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f07e00640 1 --2- 172.21.15.28:0/4205117584 >> v2:172.21.15.189:3300/0 conn(0x7f3f0806c2d0 0x7f3f0806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 -- 172.21.15.28:0/4205117584 >> v2:172.21.15.189:3300/0 conn(0x7f3f0806c2d0 msgr2=0x7f3f0806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4205117584 >> v2:172.21.15.189:3300/0 conn(0x7f3f0806c2d0 0x7f3f0806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 -- 172.21.15.28:0/4205117584 >> v1:172.21.15.189:6789/0 conn(0x7f3f081592a0 legacy=0x7f3f08159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06000640 1 -- 172.21.15.28:0/4205117584 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633211589 0 0) 0x7f3f0806cc70 con 0x7f3f081592a0 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.433+0000 7f3f06a00640 1 -- 172.21.15.28:0/4205117584 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f080b2040 con 0x7f3f0806f970 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.434+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4205117584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806f970 0x7f3f0815ab70 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3f080b1dd0 tx=0x7f3ef402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5bc39f79a15d3c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f06000640 1 -- 172.21.15.28:0/4205117584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4032440 con 0x7f3f0806f970 2024-09-13T17:19:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f06000640 1 -- 172.21.15.28:0/4205117584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef40325a0 con 0x7f3f0806f970 2024-09-13T17:19:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f06000640 1 -- 172.21.15.28:0/4205117584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef4032930 con 0x7f3f0806f970 2024-09-13T17:19:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 -- 172.21.15.28:0/4205117584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806f970 msgr2=0x7f3f0815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4205117584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806f970 0x7f3f0815ab70 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f3f080b1dd0 tx=0x7f3ef402a270 comp rx=0 tx=0).stop 2024-09-13T17:19:06.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 -- 172.21.15.28:0/4205117584 shutdown_connections 2024-09-13T17:19:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4205117584 >> v2:172.21.15.189:3300/0 conn(0x7f3f0806c2d0 0x7f3f0806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4205117584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806f970 0x7f3f0815ab70 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 -- 172.21.15.28:0/4205117584 >> 172.21.15.28:0/4205117584 conn(0x7f3f08069e80 msgr2=0x7f3f0806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 -- 172.21.15.28:0/4205117584 shutdown_connections 2024-09-13T17:19:06.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.435+0000 7f3f0e400640 1 -- 172.21.15.28:0/4205117584 wait complete. 2024-09-13T17:19:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.437+0000 7f3f0e400640 1 Processor -- start 2024-09-13T17:19:06.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.437+0000 7f3f0e400640 1 -- start start 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 0x7f3f080acaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 0x7f3f080acfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080cdd20 con 0x7f3f0806c2d0 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080ccc80 con 0x7f3f0806f970 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f0e400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f080ccf60 con 0x7f3f081592a0 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f07400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f07400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36550/0 (socket says 172.21.15.28:36550) 2024-09-13T17:19:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f07400640 1 -- 172.21.15.28:0/4040441362 learned_addr learned my addr 172.21.15.28:0/4040441362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07e00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 0x7f3f080acfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.438+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 0x7f3f080acaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 -- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 msgr2=0x7f3f080acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 0x7f3f080acfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 -- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 msgr2=0x7f3f080acaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 0x7f3f080acaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 -- 172.21.15.28:0/4040441362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ef40094d0 con 0x7f3f0806c2d0 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 0x7f3f080acaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07e00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 0x7f3f080acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f07400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c410 tx=0x7f3ef800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80035f0 con 0x7f3f0806c2d0 2024-09-13T17:19:06.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ef8003770 con 0x7f3f0806c2d0 2024-09-13T17:19:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8003b20 con 0x7f3f0806c2d0 2024-09-13T17:19:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f080cd240 con 0x7f3f0806c2d0 2024-09-13T17:19:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.439+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f080cd800 con 0x7f3f0806c2d0 2024-09-13T17:19:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.440+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f080b3740 con 0x7f3f0806c2d0 2024-09-13T17:19:06.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.441+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3ef8003c80 con 0x7f3f0806c2d0 2024-09-13T17:19:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.442+0000 7f3eff400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 0x7f3ee806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.442+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f3ef8088a30 con 0x7f3f0806c2d0 2024-09-13T17:19:06.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.443+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 0x7f3ee806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:06.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.443+0000 7f3f06a00640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 0x7f3ee806d0c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4034000 tx=0x7f3ef402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:06.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.446+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3ef8053be0 con 0x7f3f0806c2d0 2024-09-13T17:19:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.689+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3f0806a400 con 0x7f3f0806c2d0 2024-09-13T17:19:06.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.689+0000 7f3eff400640 1 -- 172.21.15.28:0/4040441362 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1773 (secure 0 0 0) 0x7f3ef8017080 con 0x7f3f0806c2d0 2024-09-13T17:19:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:06.691 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1080,"data_bytes":1076576883,"bytes_used":2698764288,"bytes_avail":770395348992,"bytes_total":773094113280,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":1633307,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":83222,"read_op_per_sec":0,"write_op_per_sec":192},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":30,"modified":"2024-09-13T17:19:01.049303+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 msgr2=0x7f3ee806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 0x7f3ee806d0c0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f3ef4034000 tx=0x7f3ef402e000 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 msgr2=0x7f3f08152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800c410 tx=0x7f3ef800c8d0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 shutdown_connections 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3ee806ac00 0x7f3ee806d0c0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f081592a0 0x7f3f080acfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f0806f970 0x7f3f080acaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 --2- 172.21.15.28:0/4040441362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0806c2d0 0x7f3f08152df0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 >> 172.21.15.28:0/4040441362 conn(0x7f3f08069e80 msgr2=0x7f3f080ae890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 shutdown_connections 2024-09-13T17:19:06.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:06.691+0000 7f3f0e400640 1 -- 172.21.15.28:0/4040441362 wait complete. 2024-09-13T17:19:06.708 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:19:07.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:07.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.876+0000 7f7cc2e00640 1 Processor -- start 2024-09-13T17:19:07.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.876+0000 7f7cc2e00640 1 -- start start 2024-09-13T17:19:07.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cc2e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cbc15a540 0x7f7cbc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cc2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc070cc0 0x7f7cbc071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cc2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc15e100 con 0x7f7cbc0cabc0 2024-09-13T17:19:07.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cc2e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc0adb00 con 0x7f7cbc15a540 2024-09-13T17:19:07.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cc2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc071670 con 0x7f7cbc070cc0 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cbbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cbc15a540 0x7f7cbc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cbbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7cbc15a540 0x7f7cbc06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36566/0 (socket says 172.21.15.28:36566) 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.877+0000 7f7cbbe00640 1 -- 172.21.15.28:0/1085409220 learned_addr learned my addr 172.21.15.28:0/1085409220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cc0e00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc070cc0 0x7f7cbc071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cbbe00640 1 -- 172.21.15.28:0/1085409220 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc070cc0 msgr2=0x7f7cbc071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cbbe00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc070cc0 0x7f7cbc071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cbbe00640 1 -- 172.21.15.28:0/1085409220 >> v1:172.21.15.28:6789/0 conn(0x7f7cbc0cabc0 legacy=0x7f7cbc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cbbe00640 1 -- 172.21.15.28:0/1085409220 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc15e3e0 con 0x7f7cbc15a540 2024-09-13T17:19:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.878+0000 7f7cbb400640 1 -- 172.21.15.28:0/1085409220 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250793259 0 0) 0x7f7cbc15e100 con 0x7f7cbc0cabc0 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.879+0000 7f7cbbe00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc15a540 0x7f7cbc06c430 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d442b8e97fec5a36 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.879+0000 7f7cbb400640 1 -- 172.21.15.28:0/1085409220 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032440 con 0x7f7cbc15a540 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.879+0000 7f7cbb400640 1 -- 172.21.15.28:0/1085409220 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac0325a0 con 0x7f7cbc15a540 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.879+0000 7f7cbb400640 1 -- 172.21.15.28:0/1085409220 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032930 con 0x7f7cbc15a540 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 -- 172.21.15.28:0/1085409220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc15a540 msgr2=0x7f7cbc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc15a540 0x7f7cbc06c430 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f7cac009870 tx=0x7f7cac02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 -- 172.21.15.28:0/1085409220 shutdown_connections 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc070cc0 0x7f7cbc071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/1085409220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc15a540 0x7f7cbc06c430 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 -- 172.21.15.28:0/1085409220 >> 172.21.15.28:0/1085409220 conn(0x7f7cbc069e80 msgr2=0x7f7cbc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 -- 172.21.15.28:0/1085409220 shutdown_connections 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.880+0000 7f7cc2e00640 1 -- 172.21.15.28:0/1085409220 wait complete. 2024-09-13T17:19:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.881+0000 7f7cc2e00640 1 Processor -- start 2024-09-13T17:19:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.882+0000 7f7cc2e00640 1 -- start start 2024-09-13T17:19:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.882+0000 7f7cc2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.882+0000 7f7cc2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 0x7f7cbc14f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 0x7f7cbc1ec1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b3720 con 0x7f7cbc070cc0 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc14caf0 con 0x7f7cbc15a540 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc0b3160 con 0x7f7cbc0cabc0 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36570/0 (socket says 172.21.15.28:36570) 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 -- 172.21.15.28:0/123437095 learned_addr learned my addr 172.21.15.28:0/123437095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc0e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 0x7f7cbc1ec1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 0x7f7cbc14f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 -- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 msgr2=0x7f7cbc14f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 0x7f7cbc14f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 -- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 msgr2=0x7f7cbc1ec1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 0x7f7cbc1ec1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cbbe00640 1 -- 172.21.15.28:0/123437095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cac0094d0 con 0x7f7cbc070cc0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cc0e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 0x7f7cbc1ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.883+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 0x7f7cbc14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cbbe00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac032440 con 0x7f7cbc070cc0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac030980 con 0x7f7cbc070cc0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc0b3440 con 0x7f7cbc070cc0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cac030d30 con 0x7f7cbc070cc0 2024-09-13T17:19:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.884+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc14d150 con 0x7f7cbc070cc0 2024-09-13T17:19:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.885+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc1f23a0 con 0x7f7cbc070cc0 2024-09-13T17:19:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.886+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7cac039040 con 0x7f7cbc070cc0 2024-09-13T17:19:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.887+0000 7f7cb8c00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 0x7f7c9806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:07.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.887+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7cac0a9cf0 con 0x7f7cbc070cc0 2024-09-13T17:19:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.887+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 0x7f7c9806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.888+0000 7f7cb3e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 0x7f7c9806d110 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4000f60 tx=0x7f7ca4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:07.891+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7cbc1f23a0 con 0x7f7cbc070cc0 2024-09-13T17:19:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.108+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cbc1f23a0 con 0x7f7cbc070cc0 2024-09-13T17:19:08.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.109+0000 7f7cb8c00640 1 -- 172.21.15.28:0/123437095 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f7cbc1f23a0 con 0x7f7cbc070cc0 2024-09-13T17:19:08.110 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:08.110 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:08.110 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 msgr2=0x7f7c9806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 0x7f7c9806d110 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f7ca4000f60 tx=0x7f7ca4009040 comp rx=0 tx=0).stop 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 msgr2=0x7f7cbc14c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f7cac000c00 tx=0x7f7cac0046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 shutdown_connections 2024-09-13T17:19:08.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c9806ac50 0x7f7c9806d110 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7cbc15a540 0x7f7cbc1ec1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7cbc0cabc0 0x7f7cbc14f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 --2- 172.21.15.28:0/123437095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cbc070cc0 0x7f7cbc14c510 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 >> 172.21.15.28:0/123437095 conn(0x7f7cbc069e80 msgr2=0x7f7cbc06f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 shutdown_connections 2024-09-13T17:19:08.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.111+0000 7f7cc2e00640 1 -- 172.21.15.28:0/123437095 wait complete. 2024-09-13T17:19:08.128 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:08.128 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:08.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.283+0000 7fd88ac00640 1 Processor -- start 2024-09-13T17:19:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.283+0000 7fd88ac00640 1 -- start start 2024-09-13T17:19:08.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88ac00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd884154a90 0x7fd88415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 0x7fd8840af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88ac00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88415bb10 con 0x7fd88406b360 2024-09-13T17:19:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88ac00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88406b900 con 0x7fd884154a90 2024-09-13T17:19:08.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88406bbe0 con 0x7fd8840b0b50 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd889800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd884154a90 0x7fd88415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 0x7fd8840af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd889800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd884154a90 0x7fd88415b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:34058/0 (socket says 172.21.15.28:34058) 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd88a200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 0x7fd8840af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:57340/0 (socket says 172.21.15.28:57340) 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.285+0000 7fd889800640 1 -- 172.21.15.28:0/3547109682 learned_addr learned my addr 172.21.15.28:0/3547109682 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.286+0000 7fd889800640 1 -- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 msgr2=0x7fd8840af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.286+0000 7fd889800640 1 --2- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 0x7fd8840af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.286+0000 7fd889800640 1 -- 172.21.15.28:0/3547109682 >> v1:172.21.15.189:6789/0 conn(0x7fd88406b360 legacy=0x7fd8840b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.286+0000 7fd883e00640 1 -- 172.21.15.28:0/3547109682 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559249024 0 0) 0x7fd88415bb10 con 0x7fd88406b360 2024-09-13T17:19:08.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.286+0000 7fd889800640 1 -- 172.21.15.28:0/3547109682 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88406bec0 con 0x7fd884154a90 2024-09-13T17:19:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.287+0000 7fd889800640 1 --2- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd884154a90 0x7fd88415b5d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd878009870 tx=0x7fd87802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434a515a7b1e1f48 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.287+0000 7fd883e00640 1 -- 172.21.15.28:0/3547109682 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878032440 con 0x7fd884154a90 2024-09-13T17:19:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.287+0000 7fd883e00640 1 -- 172.21.15.28:0/3547109682 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8780325a0 con 0x7fd884154a90 2024-09-13T17:19:08.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.287+0000 7fd883e00640 1 -- 172.21.15.28:0/3547109682 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878032930 con 0x7fd884154a90 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 -- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd884154a90 msgr2=0x7fd88415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd884154a90 0x7fd88415b5d0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fd878009870 tx=0x7fd87802a270 comp rx=0 tx=0).stop 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 -- 172.21.15.28:0/3547109682 shutdown_connections 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8840b0b50 0x7fd8840af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3547109682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd884154a90 0x7fd88415b5d0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 -- 172.21.15.28:0/3547109682 >> 172.21.15.28:0/3547109682 conn(0x7fd884072e40 msgr2=0x7fd884073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 -- 172.21.15.28:0/3547109682 shutdown_connections 2024-09-13T17:19:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.288+0000 7fd88ac00640 1 -- 172.21.15.28:0/3547109682 wait complete. 2024-09-13T17:19:08.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.290+0000 7fd88ac00640 1 Processor -- start 2024-09-13T17:19:08.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.290+0000 7fd88ac00640 1 -- start start 2024-09-13T17:19:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 0x7fd8841e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd884154a90 0x7fd88413c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841f3d20 con 0x7fd88406b360 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841e3b10 con 0x7fd8840b0b50 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd88ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841f33f0 con 0x7fd884154a90 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd889800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd889800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36584/0 (socket says 172.21.15.28:36584) 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.291+0000 7fd889800640 1 -- 172.21.15.28:0/3950284574 learned_addr learned my addr 172.21.15.28:0/3950284574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 -- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd884154a90 msgr2=0x7fd88413c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd884154a90 0x7fd88413c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 -- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 msgr2=0x7fd8841e35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd888e00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 0x7fd8841e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 0x7fd8841e35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 -- 172.21.15.28:0/3950284574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780094d0 con 0x7fd88406b360 2024-09-13T17:19:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd888e00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 0x7fd8841e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd889800640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fd878009870 tx=0x7fd8780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878032440 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd878030900 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878030cb0 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8841f36d0 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.292+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88413cd70 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.294+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8840af4b0 con 0x7fd88406b360 2024-09-13T17:19:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.294+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd878039040 con 0x7fd88406b360 2024-09-13T17:19:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.295+0000 7fd881600640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 0x7fd86406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.295+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd8780a9ce0 con 0x7fd88406b360 2024-09-13T17:19:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.296+0000 7fd888e00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 0x7fd86406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.296+0000 7fd888e00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 0x7fd86406d0f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd870007fd0 tx=0x7fd870009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:08.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.299+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd878075050 con 0x7fd88406b360 2024-09-13T17:19:08.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.543+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fd8840b0570 con 0x7fd88406b360 2024-09-13T17:19:08.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.543+0000 7fd881600640 1 -- 172.21.15.28:0/3950284574 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1750 (secure 0 0 0) 0x7fd878078670 con 0x7fd88406b360 2024-09-13T17:19:08.544 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:08.545 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":682,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":155,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1080,"data_bytes":1076576883,"bytes_used":2698764288,"bytes_avail":770395348992,"bytes_total":773094113280,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":1485037,"recovering_keys_per_sec":0,"num_objects_recovered":17,"num_bytes_recovered":17825792,"num_keys_recovered":0,"write_bytes_sec":75667,"read_op_per_sec":0,"write_op_per_sec":175},"fsmap":{"epoch":130,"id":10,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":10,"rank":0,"name":"c","status":"up:active","gid":9420}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":31,"modified":"2024-09-13T17:19:07.051288+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:08.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 msgr2=0x7fd86406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 0x7fd86406d0f0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fd870007fd0 tx=0x7fd870009040 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 msgr2=0x7fd8841f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fd878009870 tx=0x7fd8780046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 shutdown_connections 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86406ac30 0x7fd86406d0f0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd884154a90 0x7fd88413c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8840b0b50 0x7fd8841e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 --2- 172.21.15.28:0/3950284574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88406b360 0x7fd8841f2cf0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 >> 172.21.15.28:0/3950284574 conn(0x7fd884072e40 msgr2=0x7fd8840707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.545+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 shutdown_connections 2024-09-13T17:19:08.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:08.546+0000 7fd88ac00640 1 -- 172.21.15.28:0/3950284574 wait complete. 2024-09-13T17:19:08.562 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:19:08.563 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:19:08.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:19:08.663 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:19:08.764 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:19:08.764 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:08.764 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:08.800 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:08.800 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:08.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.071+0000 7efd29600640 1 Processor -- start 2024-09-13T17:19:09.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.071+0000 7efd29600640 1 -- start start 2024-09-13T17:19:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efd241592a0 0x7efd2415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 0x7efd24159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd2406c2d0 0x7efd2406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd2415b0b0 con 0x7efd2406c2d0 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd2406cca0 con 0x7efd241592a0 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd29600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd240b2040 con 0x7efd2406f970 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.072+0000 7efd23400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd2406c2d0 0x7efd2406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efd2406c2d0 0x7efd2406c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36600/0 (socket says 172.21.15.28:36600) 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 -- 172.21.15.28:0/350618322 learned_addr learned my addr 172.21.15.28:0/350618322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd22000640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 0x7efd24159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd22a00640 1 --2- 172.21.15.28:0/350618322 >> v2:172.21.15.189:3300/0 conn(0x7efd241592a0 0x7efd2415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 -- 172.21.15.28:0/350618322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 msgr2=0x7efd24159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 0x7efd24159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 -- 172.21.15.28:0/350618322 >> v2:172.21.15.189:3300/0 conn(0x7efd241592a0 msgr2=0x7efd2415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 --2- 172.21.15.28:0/350618322 >> v2:172.21.15.189:3300/0 conn(0x7efd241592a0 0x7efd2415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd23400640 1 -- 172.21.15.28:0/350618322 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd24159ff0 con 0x7efd2406c2d0 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.073+0000 7efd22000640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 0x7efd24159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:09.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.074+0000 7efd23400640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2406c2d0 0x7efd2406c760 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7efd2406d370 tx=0x7efd1c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b1a95aad05f74b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.074+0000 7efd21600640 1 -- 172.21.15.28:0/350618322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c031600 con 0x7efd2406c2d0 2024-09-13T17:19:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.074+0000 7efd21600640 1 -- 172.21.15.28:0/350618322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1c02d050 con 0x7efd2406c2d0 2024-09-13T17:19:09.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.074+0000 7efd21600640 1 -- 172.21.15.28:0/350618322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c031980 con 0x7efd2406c2d0 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 -- 172.21.15.28:0/350618322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2406c2d0 msgr2=0x7efd2406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2406c2d0 0x7efd2406c760 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7efd2406d370 tx=0x7efd1c02a6a0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 -- 172.21.15.28:0/350618322 shutdown_connections 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd2406c2d0 0x7efd2406c760 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 --2- 172.21.15.28:0/350618322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406f970 0x7efd24159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 --2- 172.21.15.28:0/350618322 >> v2:172.21.15.189:3300/0 conn(0x7efd241592a0 0x7efd2415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 -- 172.21.15.28:0/350618322 >> 172.21.15.28:0/350618322 conn(0x7efd24069e80 msgr2=0x7efd2406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 -- 172.21.15.28:0/350618322 shutdown_connections 2024-09-13T17:19:09.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.075+0000 7efd29600640 1 -- 172.21.15.28:0/350618322 wait complete. 2024-09-13T17:19:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.077+0000 7efd29600640 1 Processor -- start 2024-09-13T17:19:09.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.077+0000 7efd29600640 1 -- start start 2024-09-13T17:19:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406c2d0 0x7efd2414dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 0x7efd240aa650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240ab820 con 0x7efd241592a0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd240aaf80 con 0x7efd2406f970 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd29600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd240ab260 con 0x7efd2406c2d0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36616/0 (socket says 172.21.15.28:36616) 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 -- 172.21.15.28:0/3020836440 learned_addr learned my addr 172.21.15.28:0/3020836440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd22a00640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406c2d0 0x7efd2414dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd22000640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 0x7efd240aa650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 -- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406c2d0 msgr2=0x7efd2414dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406c2d0 0x7efd2414dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 -- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 msgr2=0x7efd240aa650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 0x7efd240aa650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd23400640 1 -- 172.21.15.28:0/3020836440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c009b30 con 0x7efd241592a0 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.078+0000 7efd22000640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 0x7efd240aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:09.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd23400640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7efd1c034040 tx=0x7efd1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c03d860 con 0x7efd241592a0 2024-09-13T17:19:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd1c03d9e0 con 0x7efd241592a0 2024-09-13T17:19:09.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd240ab540 con 0x7efd241592a0 2024-09-13T17:19:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c03dd90 con 0x7efd241592a0 2024-09-13T17:19:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.079+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2414e150 con 0x7efd241592a0 2024-09-13T17:19:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.080+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd240abd60 con 0x7efd241592a0 2024-09-13T17:19:09.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.081+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efd1c03b400 con 0x7efd241592a0 2024-09-13T17:19:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.082+0000 7efd1aa00640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 0x7efd0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.082+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7efd1c0aa310 con 0x7efd241592a0 2024-09-13T17:19:09.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.082+0000 7efd22a00640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 0x7efd0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.083+0000 7efd22a00640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 0x7efd0406d0c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efd14007920 tx=0x7efd14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.086+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efd240abd60 con 0x7efd241592a0 2024-09-13T17:19:09.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.269+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd240abd60 con 0x7efd241592a0 2024-09-13T17:19:09.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.270+0000 7efd1aa00640 1 -- 172.21.15.28:0/3020836440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16629 (secure 0 0 0) 0x7efd240abd60 con 0x7efd241592a0 2024-09-13T17:19:09.271 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:09.271 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:54.127866+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.29","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 msgr2=0x7efd0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 0x7efd0406d0c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efd14007920 tx=0x7efd14008040 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 msgr2=0x7efd2406b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7efd1c034040 tx=0x7efd1c004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 shutdown_connections 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efd0406ac00 0x7efd0406d0c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd241592a0 0x7efd2406b1c0 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efd2406f970 0x7efd240aa650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 --2- 172.21.15.28:0/3020836440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efd2406c2d0 0x7efd2414dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 >> 172.21.15.28:0/3020836440 conn(0x7efd24069e80 msgr2=0x7efd24156890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 shutdown_connections 2024-09-13T17:19:09.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.272+0000 7efd29600640 1 -- 172.21.15.28:0/3020836440 wait complete. 2024-09-13T17:19:09.289 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:18:23.576110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:18:24.545207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:19:09.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:19:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.446+0000 7f4deca00640 1 Processor -- start 2024-09-13T17:19:09.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.446+0000 7f4deca00640 1 -- start start 2024-09-13T17:19:09.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4deca00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4de815c9a0 0x7f4de80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4deca00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de8070470 con 0x7f4de80ad940 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4deca00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de80ace30 con 0x7f4de80b2400 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4deca00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4de80ad110 con 0x7f4de815c9a0 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4ddfe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4de80ad940 0x7f4de80b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35060/0 (socket says 172.21.15.28:35060) 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.447+0000 7f4ddfe00640 1 -- 172.21.15.28:0/4190787969 learned_addr learned my addr 172.21.15.28:0/4190787969 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5e00640 1 --2- 172.21.15.28:0/4190787969 >> v2:172.21.15.189:3300/0 conn(0x7f4de815c9a0 0x7f4de80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348349973 0 0) 0x7f4de8070470 con 0x7f4de80ad940 2024-09-13T17:19:09.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dcc003520 con 0x7f4de80ad940 2024-09-13T17:19:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355493094 0 0) 0x7f4dcc003520 con 0x7f4de80ad940 2024-09-13T17:19:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5e00640 1 -- 172.21.15.28:0/4190787969 >> v1:172.21.15.189:6789/0 conn(0x7f4de80b2400 legacy=0x7f4de806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5e00640 1 -- 172.21.15.28:0/4190787969 >> v1:172.21.15.28:6789/0 conn(0x7f4de80ad940 legacy=0x7f4de80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.448+0000 7f4de5e00640 1 -- 172.21.15.28:0/4190787969 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de806b170 con 0x7f4de815c9a0 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.449+0000 7f4de5e00640 1 --2- 172.21.15.28:0/4190787969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80ac850 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097d0 tx=0x7f4dd802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb22d2e60a000957 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd0001670 con 0x7f4de815c9a0 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd802ad00 con 0x7f4de815c9a0 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4de5400640 1 -- 172.21.15.28:0/4190787969 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd8032660 con 0x7f4de815c9a0 2024-09-13T17:19:09.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 -- 172.21.15.28:0/4190787969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 msgr2=0x7f4de80ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 --2- 172.21.15.28:0/4190787969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80ac850 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f4dd80097d0 tx=0x7f4dd802a270 comp rx=0 tx=0).stop 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 -- 172.21.15.28:0/4190787969 shutdown_connections 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 --2- 172.21.15.28:0/4190787969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80ac850 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 -- 172.21.15.28:0/4190787969 >> 172.21.15.28:0/4190787969 conn(0x7f4de80b0270 msgr2=0x7f4de80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 -- 172.21.15.28:0/4190787969 shutdown_connections 2024-09-13T17:19:09.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.450+0000 7f4deca00640 1 -- 172.21.15.28:0/4190787969 wait complete. 2024-09-13T17:19:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.452+0000 7f4deca00640 1 Processor -- start 2024-09-13T17:19:09.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.452+0000 7f4deca00640 1 -- start start 2024-09-13T17:19:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 0x7f4de8155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de81530b0 con 0x7f4de80ad940 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de8156a40 con 0x7f4de815c9a0 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4deca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4de80cd5e0 con 0x7f4de80b2400 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4de5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4de5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36620/0 (socket says 172.21.15.28:36620) 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4de5e00640 1 -- 172.21.15.28:0/2406040692 learned_addr learned my addr 172.21.15.28:0/2406040692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4ddfe00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 0x7f4de8155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.453+0000 7f4de6800640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 -- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 msgr2=0x7f4de8155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 0x7f4de8155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 -- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 msgr2=0x7f4de80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 -- 172.21.15.28:0/2406040692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd8009400 con 0x7f4de80ad940 2024-09-13T17:19:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4ddfe00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 0x7f4de8155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4de5e00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8035040 tx=0x7f4dd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd803b810 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dd803b970 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd803bd20 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de8070930 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.454+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de8156480 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.455+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de815cd80 con 0x7f4de80ad940 2024-09-13T17:19:09.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.456+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4dd8039040 con 0x7f4de80ad940 2024-09-13T17:19:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.457+0000 7f4dde000640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 0x7f4dc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.457+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f4dd80a98a0 con 0x7f4de80ad940 2024-09-13T17:19:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.458+0000 7f4ddfe00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 0x7f4dc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.458+0000 7f4ddfe00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 0x7f4dc406d0c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4de814a3d0 tx=0x7f4dd0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.461+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4dd8074b00 con 0x7f4de80ad940 2024-09-13T17:19:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.636+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4de8061300 con 0x7f4de80ad940 2024-09-13T17:19:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.637+0000 7f4dde000640 1 -- 172.21.15.28:0/2406040692 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4dd80370a0 con 0x7f4de80ad940 2024-09-13T17:19:09.638 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:19:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 msgr2=0x7f4dc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 0x7f4dc406d0c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f4de814a3d0 tx=0x7f4dd0009040 comp rx=0 tx=0).stop 2024-09-13T17:19:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 msgr2=0x7f4de81532d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8035040 tx=0x7f4dd80046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 shutdown_connections 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4dc406ac00 0x7f4dc406d0c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4de815c9a0 0x7f4de80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4de80b2400 0x7f4de8155b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 --2- 172.21.15.28:0/2406040692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4de80ad940 0x7f4de81532d0 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 >> 172.21.15.28:0/2406040692 conn(0x7f4de80b0270 msgr2=0x7f4de80b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 shutdown_connections 2024-09-13T17:19:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.639+0000 7f4deca00640 1 -- 172.21.15.28:0/2406040692 wait complete. 2024-09-13T17:19:09.656 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:19:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.814+0000 7fc09fe00640 1 Processor -- start 2024-09-13T17:19:09.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.814+0000 7fc09fe00640 1 -- start start 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.815+0000 7fc09fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc098159770 0x7fc09806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09806f920 con 0x7fc09815cd80 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09815b290 con 0x7fc09806f3f0 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09815b570 con 0x7fc098159770 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc098159770 0x7fc09806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc09815cd80 0x7fc09815a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35068/0 (socket says 172.21.15.28:35068) 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc098159770 0x7fc09806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36636/0 (socket says 172.21.15.28:36636) 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09de00640 1 -- 172.21.15.28:0/3818824684 learned_addr learned my addr 172.21.15.28:0/3818824684 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09d400640 1 -- 172.21.15.28:0/3818824684 >> v1:172.21.15.189:6789/0 conn(0x7fc09806f3f0 legacy=0x7fc098159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09d400640 1 -- 172.21.15.28:0/3818824684 >> v1:172.21.15.28:6789/0 conn(0x7fc09815cd80 legacy=0x7fc09815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc097e00640 1 -- 172.21.15.28:0/3818824684 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 50580935 0 0) 0x7fc09806f920 con 0x7fc09815cd80 2024-09-13T17:19:09.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.816+0000 7fc09d400640 1 -- 172.21.15.28:0/3818824684 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc09815b850 con 0x7fc098159770 2024-09-13T17:19:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.817+0000 7fc09d400640 1 --2- 172.21.15.28:0/3818824684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098159770 0x7fc09806fd20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0097a0 tx=0x7fc08c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad079aaee85d0243 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc097e00640 1 -- 172.21.15.28:0/3818824684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c02ae50 con 0x7fc098159770 2024-09-13T17:19:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc097e00640 1 -- 172.21.15.28:0/3818824684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08c032440 con 0x7fc098159770 2024-09-13T17:19:09.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc097e00640 1 -- 172.21.15.28:0/3818824684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08c0327d0 con 0x7fc098159770 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 -- 172.21.15.28:0/3818824684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098159770 msgr2=0x7fc09806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 --2- 172.21.15.28:0/3818824684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098159770 0x7fc09806fd20 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fc08c0097a0 tx=0x7fc08c02a140 comp rx=0 tx=0).stop 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 -- 172.21.15.28:0/3818824684 shutdown_connections 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 --2- 172.21.15.28:0/3818824684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc098159770 0x7fc09806fd20 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 -- 172.21.15.28:0/3818824684 >> 172.21.15.28:0/3818824684 conn(0x7fc0980b0270 msgr2=0x7fc0980b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 -- 172.21.15.28:0/3818824684 shutdown_connections 2024-09-13T17:19:09.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.818+0000 7fc09fe00640 1 -- 172.21.15.28:0/3818824684 wait complete. 2024-09-13T17:19:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.820+0000 7fc09fe00640 1 Processor -- start 2024-09-13T17:19:09.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.820+0000 7fc09fe00640 1 -- start start 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09806f3f0 0x7fc09814fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 0x7fc098158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc098152400 con 0x7fc09815cd80 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc098153cc0 con 0x7fc09806f3f0 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0981585b0 con 0x7fc098159770 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.821+0000 7fc09de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36644/0 (socket says 172.21.15.28:36644) 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 -- 172.21.15.28:0/963162326 learned_addr learned my addr 172.21.15.28:0/963162326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09d400640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09806f3f0 0x7fc09814fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09ca00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 0x7fc098158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 -- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 msgr2=0x7fc098158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 0x7fc098158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 -- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09806f3f0 msgr2=0x7fc09814fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09806f3f0 0x7fc09814fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 -- 172.21.15.28:0/963162326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c009400 con 0x7fc09815cd80 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09ca00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 0x7fc098158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.822+0000 7fc09de00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fc09000c700 tx=0x7fc09000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.823+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000ce10 con 0x7fc09815cd80 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.823+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc09000a640 con 0x7fc09815cd80 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.823+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc09000aa70 con 0x7fc09815cd80 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.823+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0981587e0 con 0x7fc09815cd80 2024-09-13T17:19:09.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.823+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc098152020 con 0x7fc09815cd80 2024-09-13T17:19:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.824+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc098152540 con 0x7fc09815cd80 2024-09-13T17:19:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.825+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc09000abd0 con 0x7fc09815cd80 2024-09-13T17:19:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.826+0000 7fc095600640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 0x7fc07806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.826+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc090088860 con 0x7fc09815cd80 2024-09-13T17:19:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.826+0000 7fc09d400640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 0x7fc07806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.827+0000 7fc09d400640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 0x7fc07806d0c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc08c034000 tx=0x7fc08c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:09.830+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc098152540 con 0x7fc09815cd80 2024-09-13T17:19:10.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.008+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc098152540 con 0x7fc09815cd80 2024-09-13T17:19:10.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.008+0000 7fc095600640 1 -- 172.21.15.28:0/963162326 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v155) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc098152540 con 0x7fc09815cd80 2024-09-13T17:19:10.009 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:19:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 msgr2=0x7fc07806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 0x7fc07806d0c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc08c034000 tx=0x7fc08c02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 msgr2=0x7fc098153e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fc09000c700 tx=0x7fc09000cbc0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 shutdown_connections 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc07806ac00 0x7fc07806d0c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc09815cd80 0x7fc098153e30 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc098159770 0x7fc098158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 --2- 172.21.15.28:0/963162326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc09806f3f0 0x7fc09814fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 >> 172.21.15.28:0/963162326 conn(0x7fc0980b0270 msgr2=0x7fc0980af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 shutdown_connections 2024-09-13T17:19:10.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.010+0000 7fc09fe00640 1 -- 172.21.15.28:0/963162326 wait complete. 2024-09-13T17:19:10.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:19:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.184+0000 7fa3f4c00640 1 Processor -- start 2024-09-13T17:19:10.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.184+0000 7fa3f4c00640 1 -- start start 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.185+0000 7fa3f4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 0x7fa3f006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.185+0000 7fa3f4c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa3f006f3f0 0x7fa3f0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3f4c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f006f920 con 0x7fa3f00add00 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3f4c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f015b290 con 0x7fa3f006f3f0 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3f4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f015b570 con 0x7fa3f0159770 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3ee800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa3f00add00 0x7fa3f0159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35072/0 (socket says 172.21.15.28:35072) 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3ee800640 1 -- 172.21.15.28:0/3355705021 learned_addr learned my addr 172.21.15.28:0/3355705021 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3ede00640 1 --2- 172.21.15.28:0/3355705021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 0x7fa3f006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3e7e00640 1 --2- 172.21.15.28:0/3355705021 >> v2:172.21.15.189:3300/0 conn(0x7fa3f006f3f0 0x7fa3f0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3ed400640 1 -- 172.21.15.28:0/3355705021 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882121793 0 0) 0x7fa3f006f920 con 0x7fa3f00add00 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.186+0000 7fa3ed400640 1 -- 172.21.15.28:0/3355705021 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d4003520 con 0x7fa3f00add00 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.187+0000 7fa3ede00640 1 -- 172.21.15.28:0/3355705021 >> v2:172.21.15.189:3300/0 conn(0x7fa3f006f3f0 msgr2=0x7fa3f0070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.187+0000 7fa3ede00640 1 --2- 172.21.15.28:0/3355705021 >> v2:172.21.15.189:3300/0 conn(0x7fa3f006f3f0 0x7fa3f0070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.187+0000 7fa3ede00640 1 -- 172.21.15.28:0/3355705021 >> v1:172.21.15.28:6789/0 conn(0x7fa3f00add00 legacy=0x7fa3f0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.187+0000 7fa3ede00640 1 -- 172.21.15.28:0/3355705021 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f015b850 con 0x7fa3f0159770 2024-09-13T17:19:10.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.188+0000 7fa3ede00640 1 --2- 172.21.15.28:0/3355705021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 0x7fa3f006fd20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00097a0 tx=0x7fa3e002c310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c110cfc8e932fffb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.188+0000 7fa3ed400640 1 -- 172.21.15.28:0/3355705021 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0034640 con 0x7fa3f0159770 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.188+0000 7fa3ed400640 1 -- 172.21.15.28:0/3355705021 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e00347a0 con 0x7fa3f0159770 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.188+0000 7fa3ed400640 1 -- 172.21.15.28:0/3355705021 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e0034b30 con 0x7fa3f0159770 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 -- 172.21.15.28:0/3355705021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 msgr2=0x7fa3f006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/3355705021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 0x7fa3f006fd20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa3e00097a0 tx=0x7fa3e002c310 comp rx=0 tx=0).stop 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 -- 172.21.15.28:0/3355705021 shutdown_connections 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/3355705021 >> v2:172.21.15.189:3300/0 conn(0x7fa3f006f3f0 0x7fa3f0070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/3355705021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f0159770 0x7fa3f006fd20 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 -- 172.21.15.28:0/3355705021 >> 172.21.15.28:0/3355705021 conn(0x7fa3f00b0270 msgr2=0x7fa3f00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 -- 172.21.15.28:0/3355705021 shutdown_connections 2024-09-13T17:19:10.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.189+0000 7fa3f4c00640 1 -- 172.21.15.28:0/3355705021 wait complete. 2024-09-13T17:19:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.190+0000 7fa3f4c00640 1 Processor -- start 2024-09-13T17:19:10.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.190+0000 7fa3f4c00640 1 -- start start 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f00add00 0x7fa3f01583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3f0159770 0x7fa3f014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c55e0 con 0x7fa3f006f3f0 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f00c5150 con 0x7fa3f0159770 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.191+0000 7fa3f4c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f01588f0 con 0x7fa3f00add00 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36650/0 (socket says 172.21.15.28:36650) 2024-09-13T17:19:10.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 -- 172.21.15.28:0/585269177 learned_addr learned my addr 172.21.15.28:0/585269177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ee800640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3f0159770 0x7fa3f014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3e7e00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f00add00 0x7fa3f01583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 -- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f00add00 msgr2=0x7fa3f01583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f00add00 0x7fa3f01583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 -- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3f0159770 msgr2=0x7fa3f014ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3f0159770 0x7fa3f014ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.192+0000 7fa3ede00640 1 -- 172.21.15.28:0/585269177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e0009400 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3ede00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0000c00 tx=0x7fa3e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e003e7f0 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e003e970 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e003ed20 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f014f200 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.193+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f00c4820 con 0x7fa3f006f3f0 2024-09-13T17:19:10.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.194+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f006fd20 con 0x7fa3f006f3f0 2024-09-13T17:19:10.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.195+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa3e003f480 con 0x7fa3f006f3f0 2024-09-13T17:19:10.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.196+0000 7fa3e6000640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 0x7fa3cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.196+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa3e00aa100 con 0x7fa3f006f3f0 2024-09-13T17:19:10.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.196+0000 7fa3e7e00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 0x7fa3cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.197+0000 7fa3e7e00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 0x7fa3cc06d0f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00ad500 tx=0x7fa3d8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.201+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa3e00752d0 con 0x7fa3f006f3f0 2024-09-13T17:19:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.382+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3f0073bb0 con 0x7fa3f006f3f0 2024-09-13T17:19:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.382+0000 7fa3e6000640 1 -- 172.21.15.28:0/585269177 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v155) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa3e0039020 con 0x7fa3f006f3f0 2024-09-13T17:19:10.383 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 msgr2=0x7fa3cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 0x7fa3cc06d0f0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa3f00ad500 tx=0x7fa3d8009190 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 msgr2=0x7fa3f00c3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0000c00 tx=0x7fa3e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 shutdown_connections 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3cc06ac30 0x7fa3cc06d0f0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3f0159770 0x7fa3f014ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3f00add00 0x7fa3f01583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 --2- 172.21.15.28:0/585269177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3f006f3f0 0x7fa3f00c3ef0 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 >> 172.21.15.28:0/585269177 conn(0x7fa3f00b0270 msgr2=0x7fa3f00aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:10.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 shutdown_connections 2024-09-13T17:19:10.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.385+0000 7fa3f4c00640 1 -- 172.21.15.28:0/585269177 wait complete. 2024-09-13T17:19:10.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:19:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.557+0000 7f811ac00640 1 Processor -- start 2024-09-13T17:19:10.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.557+0000 7f811ac00640 1 -- start start 2024-09-13T17:19:10.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.558+0000 7f811ac00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f811415a780 0x7f8114070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.558+0000 7f811ac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811406c660 con 0x7f811406c250 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.558+0000 7f811ac00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811406c8b0 con 0x7f811415ab60 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.558+0000 7f811ac00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f811406cb90 con 0x7f811415a780 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8118c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f811406c250 0x7f81140cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35082/0 (socket says 172.21.15.28:35082) 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8118c00640 1 -- 172.21.15.28:0/1040912375 learned_addr learned my addr 172.21.15.28:0/1040912375 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:10.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8113e00640 1 --2- 172.21.15.28:0/1040912375 >> v2:172.21.15.189:3300/0 conn(0x7f811415a780 0x7f8114070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503579865 0 0) 0x7f811406c660 con 0x7f811406c250 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f8003520 con 0x7f811406c250 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8113e00640 1 -- 172.21.15.28:0/1040912375 >> v1:172.21.15.189:6789/0 conn(0x7f811415ab60 legacy=0x7f811406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607370342 0 0) 0x7f811406c8b0 con 0x7f811415ab60 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8113e00640 1 -- 172.21.15.28:0/1040912375 >> v1:172.21.15.28:6789/0 conn(0x7f811406c250 legacy=0x7f81140cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.559+0000 7f8113e00640 1 -- 172.21.15.28:0/1040912375 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81140cc980 con 0x7f811415a780 2024-09-13T17:19:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.560+0000 7f8113e00640 1 --2- 172.21.15.28:0/1040912375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415a780 0x7f8114070d80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f81040097a0 tx=0x7f810402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bea94aabbe3efb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104032440 con 0x7f811415a780 2024-09-13T17:19:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81040325a0 con 0x7f811415a780 2024-09-13T17:19:10.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f8112a00640 1 -- 172.21.15.28:0/1040912375 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8104032910 con 0x7f811415a780 2024-09-13T17:19:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f811ac00640 1 -- 172.21.15.28:0/1040912375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415a780 msgr2=0x7f8114070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f811ac00640 1 --2- 172.21.15.28:0/1040912375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415a780 0x7f8114070d80 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f81040097a0 tx=0x7f810402a230 comp rx=0 tx=0).stop 2024-09-13T17:19:10.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f811ac00640 1 -- 172.21.15.28:0/1040912375 shutdown_connections 2024-09-13T17:19:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f811ac00640 1 --2- 172.21.15.28:0/1040912375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415a780 0x7f8114070d80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.561+0000 7f811ac00640 1 -- 172.21.15.28:0/1040912375 >> 172.21.15.28:0/1040912375 conn(0x7f81140b0270 msgr2=0x7f81140b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.562+0000 7f811ac00640 1 -- 172.21.15.28:0/1040912375 shutdown_connections 2024-09-13T17:19:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.562+0000 7f811ac00640 1 -- 172.21.15.28:0/1040912375 wait complete. 2024-09-13T17:19:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.563+0000 7f811ac00640 1 Processor -- start 2024-09-13T17:19:10.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.563+0000 7f811ac00640 1 -- start start 2024-09-13T17:19:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f811406c250 0x7f81140a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415ab60 0x7f81140af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140abe50 con 0x7f811415a780 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140c3800 con 0x7f811415ab60 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f811ac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81140ab240 con 0x7f811406c250 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f8113400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f8113400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36658/0 (socket says 172.21.15.28:36658) 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.564+0000 7f8113400640 1 -- 172.21.15.28:0/2002784118 learned_addr learned my addr 172.21.15.28:0/2002784118 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8118c00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415ab60 0x7f81140af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113e00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f811406c250 0x7f81140a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 -- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f811406c250 msgr2=0x7f81140a9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f811406c250 0x7f81140a9fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 -- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415ab60 msgr2=0x7f81140af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415ab60 0x7f81140af490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 -- 172.21.15.28:0/2002784118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8104009400 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f8113400640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f810000c3c0 tx=0x7f810000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81000035f0 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8100003770 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81140ab520 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100003b20 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.565+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81140abae0 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.567+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81140ac000 con 0x7f811415a780 2024-09-13T17:19:10.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.567+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8100003c80 con 0x7f811415a780 2024-09-13T17:19:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.568+0000 7f810be00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 0x7f80f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.568+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8100088a40 con 0x7f811415a780 2024-09-13T17:19:10.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.569+0000 7f8113e00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 0x7f80f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:10.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.569+0000 7f8113e00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 0x7f80f406d0c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8104034000 tx=0x7f810402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:10.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.572+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f81140ac000 con 0x7f811415a780 2024-09-13T17:19:10.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.762+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f81140cb320 con 0x7f811415a780 2024-09-13T17:19:10.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.762+0000 7f810be00640 1 -- 172.21.15.28:0/2002784118 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v155) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8100053ae0 con 0x7f811415a780 2024-09-13T17:19:10.764 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:19:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 msgr2=0x7f80f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 0x7f80f406d0c0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f8104034000 tx=0x7f810402e000 comp rx=0 tx=0).stop 2024-09-13T17:19:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 msgr2=0x7f8114147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:10.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f810000c3c0 tx=0x7f810000c880 comp rx=0 tx=0).stop 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 shutdown_connections 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80f406ac00 0x7f80f406d0c0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811415ab60 0x7f81140af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f811415a780 0x7f8114147090 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 --2- 172.21.15.28:0/2002784118 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f811406c250 0x7f81140a9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 >> 172.21.15.28:0/2002784118 conn(0x7f81140b0270 msgr2=0x7f811415cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 shutdown_connections 2024-09-13T17:19:10.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:10.764+0000 7f811ac00640 1 -- 172.21.15.28:0/2002784118 wait complete. 2024-09-13T17:19:10.780 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:19:10.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:10.780 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:19:10.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:19:10.965 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:19:11.623 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:19:11.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:11.624 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:19:11.659 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:19:11.660 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:11.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:19:11.698 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:11.698 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:11.735 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:19:11.735 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:19:11.735 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:11.774 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:19:11.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:11.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.930+0000 7fcf09600640 1 Processor -- start 2024-09-13T17:19:11.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.930+0000 7fcf09600640 1 -- start start 2024-09-13T17:19:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf0415a540 0x7fcf0406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf040cabc0 0x7fcf0415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415edb0 0x7fcf04070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf040adb00 con 0x7fcf040cabc0 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0415e100 con 0x7fcf0415a540 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf09600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0415e3e0 con 0x7fcf0415edb0 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf02000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf040cabc0 0x7fcf0415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf02000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf040cabc0 0x7fcf0415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36672/0 (socket says 172.21.15.28:36672) 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf02000640 1 -- 172.21.15.28:0/4160711753 learned_addr learned my addr 172.21.15.28:0/4160711753 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf02a00640 1 --2- 172.21.15.28:0/4160711753 >> v2:172.21.15.189:3300/0 conn(0x7fcf0415a540 0x7fcf0406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.931+0000 7fcf03400640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415edb0 0x7fcf04070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.932+0000 7fcf02000640 1 -- 172.21.15.28:0/4160711753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415edb0 msgr2=0x7fcf04070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.932+0000 7fcf02000640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415edb0 0x7fcf04070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.932+0000 7fcf02000640 1 -- 172.21.15.28:0/4160711753 >> v2:172.21.15.189:3300/0 conn(0x7fcf0415a540 msgr2=0x7fcf0406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.932+0000 7fcf02000640 1 --2- 172.21.15.28:0/4160711753 >> v2:172.21.15.189:3300/0 conn(0x7fcf0415a540 0x7fcf0406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.932+0000 7fcf02000640 1 -- 172.21.15.28:0/4160711753 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0406ce80 con 0x7fcf040cabc0 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.933+0000 7fcf02000640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0415e870 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fcf040b3a00 tx=0x7fcef002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f2abdb501a6cec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.933+0000 7fcf01600640 1 -- 172.21.15.28:0/4160711753 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef002ae90 con 0x7fcf040cabc0 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.933+0000 7fcf01600640 1 -- 172.21.15.28:0/4160711753 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef0032440 con 0x7fcf040cabc0 2024-09-13T17:19:11.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.933+0000 7fcf01600640 1 -- 172.21.15.28:0/4160711753 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef00327d0 con 0x7fcf040cabc0 2024-09-13T17:19:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 -- 172.21.15.28:0/4160711753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 msgr2=0x7fcf0415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0415e870 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7fcf040b3a00 tx=0x7fcef002a140 comp rx=0 tx=0).stop 2024-09-13T17:19:11.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 -- 172.21.15.28:0/4160711753 shutdown_connections 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0415e870 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 --2- 172.21.15.28:0/4160711753 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415edb0 0x7fcf04070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 --2- 172.21.15.28:0/4160711753 >> v2:172.21.15.189:3300/0 conn(0x7fcf0415a540 0x7fcf0406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 -- 172.21.15.28:0/4160711753 >> 172.21.15.28:0/4160711753 conn(0x7fcf04069e80 msgr2=0x7fcf0406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 -- 172.21.15.28:0/4160711753 shutdown_connections 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.934+0000 7fcf09600640 1 -- 172.21.15.28:0/4160711753 wait complete. 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.935+0000 7fcf09600640 1 Processor -- start 2024-09-13T17:19:11.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.936+0000 7fcf09600640 1 -- start start 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.936+0000 7fcf09600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.936+0000 7fcf09600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 0x7fcf0414cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.936+0000 7fcf09600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 0x7fcf040cc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.936+0000 7fcf09600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf040b37d0 con 0x7fcf040cabc0 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf09600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0414d7e0 con 0x7fcf0414d2d0 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf09600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf041f3020 con 0x7fcf0415a540 2024-09-13T17:19:11.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36674/0 (socket says 172.21.15.28:36674) 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 -- 172.21.15.28:0/3684764262 learned_addr learned my addr 172.21.15.28:0/3684764262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02000640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 0x7fcf0414cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf03400640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 0x7fcf040cc5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 -- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 msgr2=0x7fcf0414cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 0x7fcf0414cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 -- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 msgr2=0x7fcf040cc5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 0x7fcf040cc5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 -- 172.21.15.28:0/3684764262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef00094d0 con 0x7fcf040cabc0 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02000640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 0x7fcf0414cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf03400640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 0x7fcf040cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:11.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.937+0000 7fcf02a00640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9b0 tx=0x7fcef400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.938+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4009670 con 0x7fcf040cabc0 2024-09-13T17:19:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.938+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef400c9a0 con 0x7fcf040cabc0 2024-09-13T17:19:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.938+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef400cd50 con 0x7fcf040cabc0 2024-09-13T17:19:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.938+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf041f3300 con 0x7fcf040cabc0 2024-09-13T17:19:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.938+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf041f2ca0 con 0x7fcf040cabc0 2024-09-13T17:19:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.939+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf041f2e90 con 0x7fcf040cabc0 2024-09-13T17:19:11.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.940+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcef4013470 con 0x7fcf040cabc0 2024-09-13T17:19:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.941+0000 7fcefaa00640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 0x7fcee406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.941+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fcef40896b0 con 0x7fcf040cabc0 2024-09-13T17:19:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.942+0000 7fcf02000640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 0x7fcee406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.942+0000 7fcf02000640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 0x7fcee406d0c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcef0034000 tx=0x7fcef002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:11.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:11.946+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcef40547d0 con 0x7fcf040cabc0 2024-09-13T17:19:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.164+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf040aed30 con 0x7fcf040cabc0 2024-09-13T17:19:12.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.164+0000 7fcefaa00640 1 -- 172.21.15.28:0/3684764262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fcef4057df0 con 0x7fcf040cabc0 2024-09-13T17:19:12.166 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:12.166 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:12.166 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 msgr2=0x7fcee406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 0x7fcee406d0c0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fcef0034000 tx=0x7fcef002e000 comp rx=0 tx=0).stop 2024-09-13T17:19:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 msgr2=0x7fcf0414bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7fcef400e9b0 tx=0x7fcef400ee70 comp rx=0 tx=0).stop 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 shutdown_connections 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcee406ac00 0x7fcee406d0c0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf0414d2d0 0x7fcf040cc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf0415a540 0x7fcf0414cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 --2- 172.21.15.28:0/3684764262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf040cabc0 0x7fcf0414bc80 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 >> 172.21.15.28:0/3684764262 conn(0x7fcf04069e80 msgr2=0x7fcf040ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:12.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 shutdown_connections 2024-09-13T17:19:12.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.167+0000 7fcf09600640 1 -- 172.21.15.28:0/3684764262 wait complete. 2024-09-13T17:19:12.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:12.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.338+0000 7f7d31400640 1 Processor -- start 2024-09-13T17:19:12.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.339+0000 7f7d31400640 1 -- start start 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d31400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 0x7f7d2c06d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d31400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d31400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c069f10 con 0x7f7d2c06e500 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d31400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c06e9b0 con 0x7f7d2c0afe40 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d31400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0cf5b0 con 0x7f7d2c15cb60 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d21e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d2a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 0x7f7d2c06d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d21e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:34176/0 (socket says 172.21.15.28:34176) 2024-09-13T17:19:12.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.340+0000 7f7d21e00640 1 -- 172.21.15.28:0/1319176838 learned_addr learned my addr 172.21.15.28:0/1319176838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.341+0000 7f7d21e00640 1 -- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 msgr2=0x7f7d2c06d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.341+0000 7f7d21e00640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 0x7f7d2c06d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.341+0000 7f7d21e00640 1 -- 172.21.15.28:0/1319176838 >> v1:172.21.15.189:6789/0 conn(0x7f7d2c06e500 legacy=0x7f7d2c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.341+0000 7f7d21e00640 1 -- 172.21.15.28:0/1319176838 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c0cf890 con 0x7f7d2c0afe40 2024-09-13T17:19:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.341+0000 7f7d2a800640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 0x7f7d2c06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d21e00640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009870 tx=0x7f7d1402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9f1d69ea0be84f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d29e00640 1 -- 172.21.15.28:0/1319176838 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14032440 con 0x7f7d2c0afe40 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d29e00640 1 -- 172.21.15.28:0/1319176838 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d140325a0 con 0x7f7d2c0afe40 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d29e00640 1 -- 172.21.15.28:0/1319176838 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d14032930 con 0x7f7d2c0afe40 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d31400640 1 -- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0afe40 msgr2=0x7f7d2c06dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.342+0000 7f7d31400640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7d14009870 tx=0x7f7d1402a250 comp rx=0 tx=0).stop 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 -- 172.21.15.28:0/1319176838 shutdown_connections 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0afe40 0x7f7d2c06dfc0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 --2- 172.21.15.28:0/1319176838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c15cb60 0x7f7d2c06d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 -- 172.21.15.28:0/1319176838 >> 172.21.15.28:0/1319176838 conn(0x7f7d2c0b22c0 msgr2=0x7f7d2c0b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 -- 172.21.15.28:0/1319176838 shutdown_connections 2024-09-13T17:19:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.343+0000 7f7d31400640 1 -- 172.21.15.28:0/1319176838 wait complete. 2024-09-13T17:19:12.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.344+0000 7f7d31400640 1 Processor -- start 2024-09-13T17:19:12.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.344+0000 7f7d31400640 1 -- start start 2024-09-13T17:19:12.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c06e500 0x7f7d2c158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0ae080 0x7f7d2c0ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1f2e90 con 0x7f7d2c1f3140 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0aea40 con 0x7f7d2c0ae080 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.345+0000 7f7d31400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c1f73d0 con 0x7f7d2c06e500 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36684/0 (socket says 172.21.15.28:36684) 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 -- 172.21.15.28:0/1271961394 learned_addr learned my addr 172.21.15.28:0/1271961394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d21e00640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0ae080 0x7f7d2c0ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2a800640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c06e500 0x7f7d2c158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 -- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c06e500 msgr2=0x7f7d2c158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c06e500 0x7f7d2c158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 -- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0ae080 msgr2=0x7f7d2c0ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0ae080 0x7f7d2c0ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 -- 172.21.15.28:0/1271961394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c009400 con 0x7f7d2c1f3140 2024-09-13T17:19:12.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.346+0000 7f7d2b200640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c7a0 tx=0x7f7d2400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.347+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24003b70 con 0x7f7d2c1f3140 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.347+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24003cf0 con 0x7f7d2c1f3140 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.347+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d140094d0 con 0x7f7d2c1f3140 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.347+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2400aa90 con 0x7f7d2c1f3140 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.347+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c1f3e70 con 0x7f7d2c1f3140 2024-09-13T17:19:12.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.348+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c06cd80 con 0x7f7d2c1f3140 2024-09-13T17:19:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.349+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d2400abf0 con 0x7f7d2c1f3140 2024-09-13T17:19:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.350+0000 7f7d23400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.350+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f7d24089100 con 0x7f7d2c1f3140 2024-09-13T17:19:12.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.350+0000 7f7d2a800640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.351+0000 7f7d2a800640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 0x7f7d0806d0f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0b6750 tx=0x7f7d1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.354+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7d240542d0 con 0x7f7d2c1f3140 2024-09-13T17:19:12.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.533+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d2c0b0450 con 0x7f7d2c1f3140 2024-09-13T17:19:12.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.535+0000 7f7d23400640 1 -- 172.21.15.28:0/1271961394 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f7d24003e60 con 0x7f7d2c1f3140 2024-09-13T17:19:12.536 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:12.536 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:54.127866+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.29","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:12.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 msgr2=0x7f7d0806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 0x7f7d0806d0f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c0b6750 tx=0x7f7d1c009190 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 msgr2=0x7f7d2c1f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400c7a0 tx=0x7f7d2400cc60 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 shutdown_connections 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d2c1f3140 0x7f7d2c1f35c0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d2c0ae080 0x7f7d2c0ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 --2- 172.21.15.28:0/1271961394 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d2c06e500 0x7f7d2c158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 >> 172.21.15.28:0/1271961394 conn(0x7f7d2c0b22c0 msgr2=0x7f7d2c15a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 shutdown_connections 2024-09-13T17:19:12.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.537+0000 7f7d31400640 1 -- 172.21.15.28:0/1271961394 wait complete. 2024-09-13T17:19:12.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:18:23.576110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '152', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:18:24.545207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:19:12.556 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:19:12.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.714+0000 7fc6c2400640 1 Processor -- start 2024-09-13T17:19:12.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.714+0000 7fc6c2400640 1 -- start start 2024-09-13T17:19:12.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c2400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc6bc143320 0x7fc6bc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c2400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 0x7fc6bc0cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c2400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc071630 con 0x7fc6bc143910 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c2400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc0cdaf0 con 0x7fc6bc143320 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c2400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc143700 con 0x7fc6bc0cd120 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6bbe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc6bc143910 0x7fc6bc071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35092/0 (socket says 172.21.15.28:35092) 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6bbe00640 1 -- 172.21.15.28:0/753359077 learned_addr learned my addr 172.21.15.28:0/753359077 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c1a00640 1 --2- 172.21.15.28:0/753359077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 0x7fc6bc0cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6c1000640 1 --2- 172.21.15.28:0/753359077 >> v2:172.21.15.189:3300/0 conn(0x7fc6bc143320 0x7fc6bc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.716+0000 7fc6bb400640 1 -- 172.21.15.28:0/753359077 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635683607 0 0) 0x7fc6bc071630 con 0x7fc6bc143910 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.717+0000 7fc6bb400640 1 -- 172.21.15.28:0/753359077 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a4003520 con 0x7fc6bc143910 2024-09-13T17:19:12.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.717+0000 7fc6c1a00640 1 -- 172.21.15.28:0/753359077 >> v2:172.21.15.189:3300/0 conn(0x7fc6bc143320 msgr2=0x7fc6bc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.717+0000 7fc6c1a00640 1 --2- 172.21.15.28:0/753359077 >> v2:172.21.15.189:3300/0 conn(0x7fc6bc143320 0x7fc6bc070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.717+0000 7fc6c1a00640 1 -- 172.21.15.28:0/753359077 >> v1:172.21.15.28:6789/0 conn(0x7fc6bc143910 legacy=0x7fc6bc071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.717+0000 7fc6c1a00640 1 -- 172.21.15.28:0/753359077 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc070a20 con 0x7fc6bc0cd120 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.718+0000 7fc6c1a00640 1 --2- 172.21.15.28:0/753359077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 0x7fc6bc0cd5b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009d90 tx=0x7fc6b402a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d647c614ab43543 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.718+0000 7fc6bb400640 1 -- 172.21.15.28:0/753359077 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4007740 con 0x7fc6bc0cd120 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.718+0000 7fc6bb400640 1 -- 172.21.15.28:0/753359077 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b402d050 con 0x7fc6bc0cd120 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.718+0000 7fc6bb400640 1 -- 172.21.15.28:0/753359077 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b4007ac0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 -- 172.21.15.28:0/753359077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 msgr2=0x7fc6bc0cd5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 --2- 172.21.15.28:0/753359077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 0x7fc6bc0cd5b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fc6b4009d90 tx=0x7fc6b402a660 comp rx=0 tx=0).stop 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 -- 172.21.15.28:0/753359077 shutdown_connections 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 --2- 172.21.15.28:0/753359077 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc0cd120 0x7fc6bc0cd5b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 --2- 172.21.15.28:0/753359077 >> v2:172.21.15.189:3300/0 conn(0x7fc6bc143320 0x7fc6bc070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 -- 172.21.15.28:0/753359077 >> 172.21.15.28:0/753359077 conn(0x7fc6bc069e80 msgr2=0x7fc6bc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 -- 172.21.15.28:0/753359077 shutdown_connections 2024-09-13T17:19:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.719+0000 7fc6c2400640 1 -- 172.21.15.28:0/753359077 wait complete. 2024-09-13T17:19:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.721+0000 7fc6c2400640 1 Processor -- start 2024-09-13T17:19:12.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.721+0000 7fc6c2400640 1 -- start start 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc143320 0x7fc6bc13d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc143910 0x7fc6bc1e6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc13ee50 con 0x7fc6bc0cd120 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc130030 con 0x7fc6bc143910 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c2400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc13e240 con 0x7fc6bc143320 2024-09-13T17:19:12.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36698/0 (socket says 172.21.15.28:36698) 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1000640 1 -- 172.21.15.28:0/3809486427 learned_addr learned my addr 172.21.15.28:0/3809486427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc143320 0x7fc6bc13d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1a00640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc143910 0x7fc6bc1e6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1000640 1 -- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc143320 msgr2=0x7fc6bc13d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.722+0000 7fc6c1000640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc143320 0x7fc6bc13d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c1000640 1 -- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc143910 msgr2=0x7fc6bc1e6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c1000640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc143910 0x7fc6bc1e6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c1000640 1 -- 172.21.15.28:0/3809486427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4009a70 con 0x7fc6bc0cd120 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c1000640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000e9b0 tx=0x7fc6b000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b0009670 con 0x7fc6bc0cd120 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b000c9a0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc13e520 con 0x7fc6bc0cd120 2024-09-13T17:19:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6b000cd50 con 0x7fc6bc0cd120 2024-09-13T17:19:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.723+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc13eae0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.724+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc0b37e0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.725+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc6b0013470 con 0x7fc6bc0cd120 2024-09-13T17:19:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.726+0000 7fc6b8c00640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 0x7fc69c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.726+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fc6b0089980 con 0x7fc6bc0cd120 2024-09-13T17:19:12.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.727+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 0x7fc69c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:12.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.727+0000 7fc6bbe00640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 0x7fc69c06d0f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc1e49b0 tx=0x7fc6a8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:12.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.730+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc6b0054ac0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.906+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc6bc144010 con 0x7fc6bc0cd120 2024-09-13T17:19:12.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.907+0000 7fc6b8c00640 1 -- 172.21.15.28:0/3809486427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v155) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc6b00210a0 con 0x7fc6bc0cd120 2024-09-13T17:19:12.908 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:19:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 msgr2=0x7fc69c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 0x7fc69c06d0f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc1e49b0 tx=0x7fc6a8007d30 comp rx=0 tx=0).stop 2024-09-13T17:19:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 msgr2=0x7fc6bc12faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fc6b000e9b0 tx=0x7fc6b000ee70 comp rx=0 tx=0).stop 2024-09-13T17:19:12.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 shutdown_connections 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc69c06ac30 0x7fc69c06d0f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6bc143910 0x7fc6bc1e6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc6bc143320 0x7fc6bc13d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 --2- 172.21.15.28:0/3809486427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc6bc0cd120 0x7fc6bc12faf0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 >> 172.21.15.28:0/3809486427 conn(0x7fc6bc069e80 msgr2=0x7fc6bc140f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 shutdown_connections 2024-09-13T17:19:12.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:12.909+0000 7fc6c2400640 1 -- 172.21.15.28:0/3809486427 wait complete. 2024-09-13T17:19:12.926 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:19:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.084+0000 7f4968a00640 1 Processor -- start 2024-09-13T17:19:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.084+0000 7f4968a00640 1 -- start start 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 0x7f4964070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f496415ab60 0x7f49640712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496406bb40 0x7f496406bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496406c8b0 con 0x7f496415ab60 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f496406cb90 con 0x7f496415a780 2024-09-13T17:19:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4968a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49640cc980 con 0x7f496406bb40 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4959400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f496415ab60 0x7f49640712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4959400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f496415ab60 0x7f49640712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36706/0 (socket says 172.21.15.28:36706) 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.085+0000 7f4959400640 1 -- 172.21.15.28:0/122474124 learned_addr learned my addr 172.21.15.28:0/122474124 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4962800640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496406bb40 0x7f496406bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4961e00640 1 --2- 172.21.15.28:0/122474124 >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 0x7f4964070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4959400640 1 -- 172.21.15.28:0/122474124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496406bb40 msgr2=0x7f496406bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4959400640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496406bb40 0x7f496406bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4959400640 1 -- 172.21.15.28:0/122474124 >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 msgr2=0x7f4964070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4959400640 1 --2- 172.21.15.28:0/122474124 >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 0x7f4964070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4959400640 1 -- 172.21.15.28:0/122474124 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49640ccc60 con 0x7f496415ab60 2024-09-13T17:19:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.086+0000 7f4961e00640 1 --2- 172.21.15.28:0/122474124 >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 0x7f4964070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.087+0000 7f4959400640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f49640712c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f49640b1330 tx=0x7f494c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=859b603ea12c2228 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.087+0000 7f4961400640 1 -- 172.21.15.28:0/122474124 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c032440 con 0x7f496415ab60 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.087+0000 7f4961400640 1 -- 172.21.15.28:0/122474124 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f494c0325a0 con 0x7f496415ab60 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.087+0000 7f4961400640 1 -- 172.21.15.28:0/122474124 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c032930 con 0x7f496415ab60 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 -- 172.21.15.28:0/122474124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 msgr2=0x7f49640712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f49640712c0 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f49640b1330 tx=0x7f494c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 -- 172.21.15.28:0/122474124 shutdown_connections 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f49640712c0 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 --2- 172.21.15.28:0/122474124 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496406bb40 0x7f496406bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 --2- 172.21.15.28:0/122474124 >> v2:172.21.15.189:3300/0 conn(0x7f496415a780 0x7f4964070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 -- 172.21.15.28:0/122474124 >> 172.21.15.28:0/122474124 conn(0x7f49640b0270 msgr2=0x7f49640b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 -- 172.21.15.28:0/122474124 shutdown_connections 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.088+0000 7f4968a00640 1 -- 172.21.15.28:0/122474124 wait complete. 2024-09-13T17:19:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.089+0000 7f4968a00640 1 Processor -- start 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.090+0000 7f4968a00640 1 -- start start 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 0x7f496414d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496415a780 0x7f496414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496414e750 con 0x7f496415ab60 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496414dad0 con 0x7f496406bb40 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4968a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496414ddb0 con 0x7f496415a780 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36712/0 (socket says 172.21.15.28:36712) 2024-09-13T17:19:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 -- 172.21.15.28:0/3324745737 learned_addr learned my addr 172.21.15.28:0/3324745737 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4959400640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496415a780 0x7f496414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4961e00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 0x7f496414d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 -- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496415a780 msgr2=0x7f496414bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496415a780 0x7f496414bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 -- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 msgr2=0x7f496414d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 0x7f496414d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4962800640 1 -- 172.21.15.28:0/3324745737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c0094d0 con 0x7f496415ab60 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.091+0000 7f4961e00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 0x7f496414d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f4962800640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f495c00efc0 tx=0x7f495c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c014480 con 0x7f496415ab60 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f495c00f040 con 0x7f496415ab60 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c014820 con 0x7f496415ab60 2024-09-13T17:19:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496414e090 con 0x7f496415ab60 2024-09-13T17:19:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.092+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496414d7c0 con 0x7f496415ab60 2024-09-13T17:19:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.093+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f496406bf70 con 0x7f496415ab60 2024-09-13T17:19:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.094+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f495c014980 con 0x7f496415ab60 2024-09-13T17:19:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.095+0000 7f495aa00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 0x7f494006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.095+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f495c089ec0 con 0x7f496415ab60 2024-09-13T17:19:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.095+0000 7f4961e00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 0x7f494006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.096+0000 7f4961e00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 0x7f494006d0f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4954006fd0 tx=0x7f4954008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.099+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f495c054fe0 con 0x7f496415ab60 2024-09-13T17:19:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.283+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f496406b920 con 0x7f496415ab60 2024-09-13T17:19:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.283+0000 7f495aa00640 1 -- 172.21.15.28:0/3324745737 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v155) v1 ==== 104+0+11 (secure 0 0 0) 0x7f495c018020 con 0x7f496415ab60 2024-09-13T17:19:13.288 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:19:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.285+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 msgr2=0x7f494006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.285+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 0x7f494006d0f0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4954006fd0 tx=0x7f4954008040 comp rx=0 tx=0).stop 2024-09-13T17:19:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.285+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 msgr2=0x7f4964151080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.285+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f495c00efc0 tx=0x7f495c00c490 comp rx=0 tx=0).stop 2024-09-13T17:19:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 shutdown_connections 2024-09-13T17:19:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f494006ac30 0x7f494006d0f0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f496415ab60 0x7f4964151080 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f496415a780 0x7f496414bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 --2- 172.21.15.28:0/3324745737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f496406bb40 0x7f496414d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 >> 172.21.15.28:0/3324745737 conn(0x7f49640b0270 msgr2=0x7f4964070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 shutdown_connections 2024-09-13T17:19:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.286+0000 7f4968a00640 1 -- 172.21.15.28:0/3324745737 wait complete. 2024-09-13T17:19:13.306 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:19:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.467+0000 7ff5e5e00640 1 Processor -- start 2024-09-13T17:19:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.467+0000 7ff5e5e00640 1 -- start start 2024-09-13T17:19:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 0x7ff5e006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5e5e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e006c8b0 con 0x7ff5e015a780 2024-09-13T17:19:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5e5e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e006cb90 con 0x7ff5e0071490 2024-09-13T17:19:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5e5e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e00cc980 con 0x7ff5e015ab60 2024-09-13T17:19:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5dea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff5e015a780 0x7ff5e0070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:35094/0 (socket says 172.21.15.28:35094) 2024-09-13T17:19:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.468+0000 7ff5dea00640 1 -- 172.21.15.28:0/833991307 learned_addr learned my addr 172.21.15.28:0/833991307 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5de000640 1 --2- 172.21.15.28:0/833991307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 0x7ff5e006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789745971 0 0) 0x7ff5e006c8b0 con 0x7ff5e015a780 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5c4003520 con 0x7ff5e015a780 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3073696502 0 0) 0x7ff5e006cb90 con 0x7ff5e0071490 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e006c8b0 con 0x7ff5e0071490 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5de000640 1 -- 172.21.15.28:0/833991307 >> v1:172.21.15.189:6789/0 conn(0x7ff5e0071490 legacy=0x7ff5e006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4090794530 0 0) 0x7ff5c4003520 con 0x7ff5e015a780 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5de000640 1 -- 172.21.15.28:0/833991307 >> v1:172.21.15.28:6789/0 conn(0x7ff5e015a780 legacy=0x7ff5e0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.469+0000 7ff5de000640 1 -- 172.21.15.28:0/833991307 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e00ccc60 con 0x7ff5e015ab60 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.470+0000 7ff5de000640 1 --2- 172.21.15.28:0/833991307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 0x7ff5e006bb40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0097d0 tx=0x7ff5cc02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6091bee27ef944c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d0001670 con 0x7ff5e015ab60 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5cc02ac30 con 0x7ff5e015ab60 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5dd600640 1 -- 172.21.15.28:0/833991307 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc032440 con 0x7ff5e015ab60 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 -- 172.21.15.28:0/833991307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 msgr2=0x7ff5e006bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/833991307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 0x7ff5e006bb40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0097d0 tx=0x7ff5cc02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 -- 172.21.15.28:0/833991307 shutdown_connections 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/833991307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e015ab60 0x7ff5e006bb40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 -- 172.21.15.28:0/833991307 >> 172.21.15.28:0/833991307 conn(0x7ff5e00b0270 msgr2=0x7ff5e00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 -- 172.21.15.28:0/833991307 shutdown_connections 2024-09-13T17:19:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.471+0000 7ff5e5e00640 1 -- 172.21.15.28:0/833991307 wait complete. 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.473+0000 7ff5e5e00640 1 Processor -- start 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.473+0000 7ff5e5e00640 1 -- start start 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e0071490 0x7ff5e014ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e015a780 0x7ff5e00b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0156800 con 0x7ff5e015ab60 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0156ae0 con 0x7ff5e015a780 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5e5e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e0156dc0 con 0x7ff5e0071490 2024-09-13T17:19:13.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36724/0 (socket says 172.21.15.28:36724) 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5df400640 1 -- 172.21.15.28:0/1406853830 learned_addr learned my addr 172.21.15.28:0/1406853830 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5dea00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e0071490 0x7ff5e014ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.474+0000 7ff5de000640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e015a780 0x7ff5e00b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 -- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e0071490 msgr2=0x7ff5e014ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e0071490 0x7ff5e014ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 -- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e015a780 msgr2=0x7ff5e00b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e015a780 0x7ff5e00b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 -- 172.21.15.28:0/1406853830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc009400 con 0x7ff5e015ab60 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5df400640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c730 tx=0x7ff5d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8003b70 con 0x7ff5e015ab60 2024-09-13T17:19:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d8003cd0 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800aa90 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.475+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e01570a0 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.476+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e01504d0 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.477+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e00cb8a0 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.478+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5d800abf0 con 0x7ff5e015ab60 2024-09-13T17:19:13.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.478+0000 7ff5d7400640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 0x7ff5c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.478+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff5d8089280 con 0x7ff5e015ab60 2024-09-13T17:19:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.479+0000 7ff5dea00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 0x7ff5c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.480+0000 7ff5dea00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 0x7ff5c006d0c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000f60 tx=0x7ff5d0001670 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.483+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff5d80543c0 con 0x7ff5e015ab60 2024-09-13T17:19:13.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.664+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff5e00cab40 con 0x7ff5e015ab60 2024-09-13T17:19:13.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.665+0000 7ff5d7400640 1 -- 172.21.15.28:0/1406853830 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v155) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff5d80579e0 con 0x7ff5e015ab60 2024-09-13T17:19:13.666 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 msgr2=0x7ff5c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 0x7ff5c006d0c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0000f60 tx=0x7ff5d0001670 comp rx=0 tx=0).stop 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 msgr2=0x7ff5e00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800c730 tx=0x7ff5d800cbf0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 shutdown_connections 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5c006ac00 0x7ff5c006d0c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5e015ab60 0x7ff5e00ac850 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff5e015a780 0x7ff5e00b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 --2- 172.21.15.28:0/1406853830 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff5e0071490 0x7ff5e014ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 >> 172.21.15.28:0/1406853830 conn(0x7ff5e00b0270 msgr2=0x7ff5e00ce470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 shutdown_connections 2024-09-13T17:19:13.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.667+0000 7ff5e5e00640 1 -- 172.21.15.28:0/1406853830 wait complete. 2024-09-13T17:19:13.684 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:13.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.839+0000 7fbc9dc00640 1 Processor -- start 2024-09-13T17:19:13.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.839+0000 7fbc9dc00640 1 -- start start 2024-09-13T17:19:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.840+0000 7fbc9dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbc981592a0 0x7fbc9815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.840+0000 7fbc9dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.840+0000 7fbc9dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9806cc70 con 0x7fbc9806f970 2024-09-13T17:19:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.840+0000 7fbc9dc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9815b0b0 con 0x7fbc981592a0 2024-09-13T17:19:13.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.840+0000 7fbc9dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc98159d90 con 0x7fbc9806c2d0 2024-09-13T17:19:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbc981592a0 0x7fbc9815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc97400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbc981592a0 0x7fbc9815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:37452/0 (socket says 172.21.15.28:37452) 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 -- 172.21.15.28:0/3724661746 learned_addr learned my addr 172.21.15.28:0/3724661746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 -- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 msgr2=0x7fbc9806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 -- 172.21.15.28:0/3724661746 >> v1:172.21.15.189:6789/0 conn(0x7fbc9806f970 legacy=0x7fbc98159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc96a00640 1 -- 172.21.15.28:0/3724661746 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc980b2040 con 0x7fbc981592a0 2024-09-13T17:19:13.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.841+0000 7fbc97400640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.842+0000 7fbc96a00640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc9815ab70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8802c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e3cb47eede9dc70 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc95600640 1 -- 172.21.15.28:0/3724661746 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88031710 con 0x7fbc981592a0 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc95600640 1 -- 172.21.15.28:0/3724661746 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc8802f050 con 0x7fbc981592a0 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc95600640 1 -- 172.21.15.28:0/3724661746 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc88031a90 con 0x7fbc981592a0 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 -- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 msgr2=0x7fbc9815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc9815ab70 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fbc88009870 tx=0x7fbc8802c460 comp rx=0 tx=0).stop 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 -- 172.21.15.28:0/3724661746 shutdown_connections 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/3724661746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc9815ab70 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 -- 172.21.15.28:0/3724661746 >> 172.21.15.28:0/3724661746 conn(0x7fbc98069e80 msgr2=0x7fbc9806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 -- 172.21.15.28:0/3724661746 shutdown_connections 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.843+0000 7fbc9dc00640 1 -- 172.21.15.28:0/3724661746 wait complete. 2024-09-13T17:19:13.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.845+0000 7fbc9dc00640 1 Processor -- start 2024-09-13T17:19:13.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.845+0000 7fbc9dc00640 1 -- start start 2024-09-13T17:19:13.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9813fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc980ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9814f750 con 0x7fbc9806f970 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9814eb40 con 0x7fbc981592a0 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc9dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9814ee20 con 0x7fbc9806c2d0 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc96000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc96000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39008/0 (socket says 172.21.15.28:39008) 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc96000640 1 -- 172.21.15.28:0/306304983 learned_addr learned my addr 172.21.15.28:0/306304983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc97400640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc980ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.846+0000 7fbc96a00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9813fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 -- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 msgr2=0x7fbc9813fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9813fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 -- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 msgr2=0x7fbc980ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc980ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:13.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 -- 172.21.15.28:0/306304983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90009a70 con 0x7fbc9806f970 2024-09-13T17:19:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc96000640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c4d0 tx=0x7fbc8400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc840052d0 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc84005430 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc84012850 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc880094d0 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.847+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9814f3e0 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.848+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc980b3740 con 0x7fbc9806f970 2024-09-13T17:19:13.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.849+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbc840129b0 con 0x7fbc9806f970 2024-09-13T17:19:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.850+0000 7fbc8ea00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 0x7fbc7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.850+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fbc84088f40 con 0x7fbc9806f970 2024-09-13T17:19:13.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.851+0000 7fbc96a00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 0x7fbc7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:13.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.852+0000 7fbc96a00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 0x7fbc7806d0c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbc88035000 tx=0x7fbc88008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:13.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:13.855+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbc84054060 con 0x7fbc9806f970 2024-09-13T17:19:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.069+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc981579e0 con 0x7fbc9806f970 2024-09-13T17:19:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.070+0000 7fbc8ea00640 1 -- 172.21.15.28:0/306304983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fbc84057680 con 0x7fbc9806f970 2024-09-13T17:19:14.071 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:14.071 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:14.071 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 msgr2=0x7fbc7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 0x7fbc7806d0c0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fbc88035000 tx=0x7fbc88008030 comp rx=0 tx=0).stop 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 msgr2=0x7fbc9814e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fbc8400c4d0 tx=0x7fbc8400c990 comp rx=0 tx=0).stop 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 shutdown_connections 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbc7806ac00 0x7fbc7806d0c0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc981592a0 0x7fbc980ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc9806f970 0x7fbc9814e080 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 --2- 172.21.15.28:0/306304983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbc9806c2d0 0x7fbc9813fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 >> 172.21.15.28:0/306304983 conn(0x7fbc98069e80 msgr2=0x7fbc981581e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 shutdown_connections 2024-09-13T17:19:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.072+0000 7fbc9dc00640 1 -- 172.21.15.28:0/306304983 wait complete. 2024-09-13T17:19:14.090 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.244+0000 7f6a8c800640 1 Processor -- start 2024-09-13T17:19:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.244+0000 7f6a8c800640 1 -- start start 2024-09-13T17:19:14.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a8c800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a841592a0 0x7f6a8415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a8c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a84159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a8c800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a840b2040 con 0x7f6a8406c2d0 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a8c800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a8415b0b0 con 0x7f6a841592a0 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a8c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84159ff0 con 0x7f6a8406f970 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a89e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a841592a0 0x7f6a8415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a89e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6a841592a0 0x7f6a8415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39020/0 (socket says 172.21.15.28:39020) 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.245+0000 7f6a89e00640 1 -- 172.21.15.28:0/377027414 learned_addr learned my addr 172.21.15.28:0/377027414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89400640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a84159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89e00640 1 -- 172.21.15.28:0/377027414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 msgr2=0x7f6a84159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89e00640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a84159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89e00640 1 -- 172.21.15.28:0/377027414 >> v1:172.21.15.189:6789/0 conn(0x7f6a8406c2d0 legacy=0x7f6a840b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89e00640 1 -- 172.21.15.28:0/377027414 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a8415a2d0 con 0x7f6a841592a0 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.246+0000 7f6a89400640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a84159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:14.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.247+0000 7f6a89e00640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a841592a0 0x7f6a8415ab70 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a7802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af058b486de31d2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.247+0000 7f6a88a00640 1 -- 172.21.15.28:0/377027414 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78032440 con 0x7f6a841592a0 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.247+0000 7f6a88a00640 1 -- 172.21.15.28:0/377027414 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a780325a0 con 0x7f6a841592a0 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.247+0000 7f6a88a00640 1 -- 172.21.15.28:0/377027414 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78032930 con 0x7f6a841592a0 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 -- 172.21.15.28:0/377027414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a841592a0 msgr2=0x7f6a8415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a841592a0 0x7f6a8415ab70 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a7802a200 comp rx=0 tx=0).stop 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 -- 172.21.15.28:0/377027414 shutdown_connections 2024-09-13T17:19:14.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a84159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 --2- 172.21.15.28:0/377027414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a841592a0 0x7f6a8415ab70 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 -- 172.21.15.28:0/377027414 >> 172.21.15.28:0/377027414 conn(0x7f6a84069e80 msgr2=0x7f6a8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 -- 172.21.15.28:0/377027414 shutdown_connections 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.248+0000 7f6a8c800640 1 -- 172.21.15.28:0/377027414 wait complete. 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.249+0000 7f6a8c800640 1 Processor -- start 2024-09-13T17:19:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.250+0000 7f6a8c800640 1 -- start start 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.250+0000 7f6a8c800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.250+0000 7f6a8c800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a840cec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a8c800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a841592a0 0x7f6a8414cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a8c800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840ca860 con 0x7f6a8406c2d0 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a8c800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840c8960 con 0x7f6a841592a0 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a8c800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a840c8c40 con 0x7f6a8406f970 2024-09-13T17:19:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39030/0 (socket says 172.21.15.28:39030) 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 -- 172.21.15.28:0/4029162206 learned_addr learned my addr 172.21.15.28:0/4029162206 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 -- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 msgr2=0x7f6a840cec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a840cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 -- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a841592a0 msgr2=0x7f6a8414cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a841592a0 0x7f6a8414cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 -- 172.21.15.28:0/4029162206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a780094d0 con 0x7f6a8406c2d0 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.251+0000 7f6a89e00640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.252+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78030b00 con 0x7f6a8406c2d0 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.252+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a78030c60 con 0x7f6a8406c2d0 2024-09-13T17:19:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.252+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7803d420 con 0x7f6a8406c2d0 2024-09-13T17:19:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.252+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a840c8f20 con 0x7f6a8406c2d0 2024-09-13T17:19:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.252+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a840c94e0 con 0x7f6a8406c2d0 2024-09-13T17:19:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.253+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a8415ab70 con 0x7f6a8406c2d0 2024-09-13T17:19:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.254+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a78039040 con 0x7f6a8406c2d0 2024-09-13T17:19:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.255+0000 7f6a7e000640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 0x7f6a6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.255+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f6a780aa0c0 con 0x7f6a8406c2d0 2024-09-13T17:19:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.255+0000 7f6a89400640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 0x7f6a6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.256+0000 7f6a89400640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 0x7f6a6806d0c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6a74006fd0 tx=0x7f6a74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.259+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6a78075270 con 0x7f6a8406c2d0 2024-09-13T17:19:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.438+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a840710d0 con 0x7f6a8406c2d0 2024-09-13T17:19:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.439+0000 7f6a7e000640 1 -- 172.21.15.28:0/4029162206 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f6a78078890 con 0x7f6a8406c2d0 2024-09-13T17:19:14.441 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:14.441 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:54.127866+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.29","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 msgr2=0x7f6a6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 0x7f6a6806d0c0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f6a74006fd0 tx=0x7f6a74008040 comp rx=0 tx=0).stop 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 msgr2=0x7f6a840ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f6a78009870 tx=0x7f6a780046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 shutdown_connections 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a6806ac00 0x7f6a6806d0c0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a841592a0 0x7f6a8414cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a8406f970 0x7f6a840cec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 --2- 172.21.15.28:0/4029162206 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a8406c2d0 0x7f6a840ca320 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 >> 172.21.15.28:0/4029162206 conn(0x7f6a84069e80 msgr2=0x7f6a840cce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 shutdown_connections 2024-09-13T17:19:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.442+0000 7f6a8c800640 1 -- 172.21.15.28:0/4029162206 wait complete. 2024-09-13T17:19:14.458 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:19:14.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:14.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.613+0000 7fd7bea00640 1 Processor -- start 2024-09-13T17:19:14.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.613+0000 7fd7bea00640 1 -- start start 2024-09-13T17:19:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.614+0000 7fd7bea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.614+0000 7fd7bea00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7b8070cc0 0x7fd7b8071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.614+0000 7fd7bea00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b815e100 con 0x7fd7b80cabc0 2024-09-13T17:19:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.614+0000 7fd7bea00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7b80716b0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.614+0000 7fd7bea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b80adb00 con 0x7fd7b815a540 2024-09-13T17:19:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7bca00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7b8070cc0 0x7fd7b8071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:58244/0 (socket says 172.21.15.28:58244) 2024-09-13T17:19:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7b7e00640 1 -- 172.21.15.28:0/1643545225 learned_addr learned my addr 172.21.15.28:0/1643545225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:14.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7bca00640 1 -- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 msgr2=0x7fd7b806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:14.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7bca00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7bca00640 1 -- 172.21.15.28:0/1643545225 >> v1:172.21.15.189:6789/0 conn(0x7fd7b80cabc0 legacy=0x7fd7b815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7bca00640 1 -- 172.21.15.28:0/1643545225 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b815e3e0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.615+0000 7fd7b7e00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:14.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.616+0000 7fd7bca00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b8070cc0 0x7fd7b8071170 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009dc0 tx=0x7fd7b002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6d8d15c0dd20576 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7b7400640 1 -- 172.21.15.28:0/1643545225 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0007740 con 0x7fd7b8070cc0 2024-09-13T17:19:14.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7b7400640 1 -- 172.21.15.28:0/1643545225 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b002d050 con 0x7fd7b8070cc0 2024-09-13T17:19:14.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7b7400640 1 -- 172.21.15.28:0/1643545225 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b0007ac0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 -- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b8070cc0 msgr2=0x7fd7b8071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b8070cc0 0x7fd7b8071170 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009dc0 tx=0x7fd7b002a5f0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 -- 172.21.15.28:0/1643545225 shutdown_connections 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b8070cc0 0x7fd7b8071170 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 --2- 172.21.15.28:0/1643545225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 -- 172.21.15.28:0/1643545225 >> 172.21.15.28:0/1643545225 conn(0x7fd7b8069e80 msgr2=0x7fd7b806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 -- 172.21.15.28:0/1643545225 shutdown_connections 2024-09-13T17:19:14.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.617+0000 7fd7bea00640 1 -- 172.21.15.28:0/1643545225 wait complete. 2024-09-13T17:19:14.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.619+0000 7fd7bea00640 1 Processor -- start 2024-09-13T17:19:14.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.619+0000 7fd7bea00640 1 -- start start 2024-09-13T17:19:14.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b80cabc0 0x7fd7b81539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b8142e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8154d60 con 0x7fd7b8070cc0 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8153f20 con 0x7fd7b80cabc0 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b8154150 con 0x7fd7b815a540 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39038/0 (socket says 172.21.15.28:39038) 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7b7e00640 1 -- 172.21.15.28:0/3794652438 learned_addr learned my addr 172.21.15.28:0/3794652438 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7af400640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b80cabc0 0x7fd7b81539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.620+0000 7fd7bca00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b8142e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 -- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 msgr2=0x7fd7b8142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b8142e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 -- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b80cabc0 msgr2=0x7fd7b81539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b80cabc0 0x7fd7b81539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 -- 172.21.15.28:0/3794652438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a80094d0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b7e00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800e9b0 tx=0x7fd7a800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a800c9a0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7a800cb00 con 0x7fd7b8070cc0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7a800ce70 con 0x7fd7b8070cc0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b0009a70 con 0x7fd7b8070cc0 2024-09-13T17:19:14.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.621+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b8154710 con 0x7fd7b8070cc0 2024-09-13T17:19:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.622+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b80cb620 con 0x7fd7b8070cc0 2024-09-13T17:19:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.623+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd7a8013470 con 0x7fd7b8070cc0 2024-09-13T17:19:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.624+0000 7fd7b4c00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 0x7fd79406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:14.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.624+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd7a80899a0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.625+0000 7fd7af400640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 0x7fd79406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:14.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.625+0000 7fd7af400640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 0x7fd79406d0c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd7b81f5cb0 tx=0x7fd7a0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.628+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd7a8054ac0 con 0x7fd7b8070cc0 2024-09-13T17:19:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.854+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7b8071170 con 0x7fd7b8070cc0 2024-09-13T17:19:14.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.855+0000 7fd7b4c00640 1 -- 172.21.15.28:0/3794652438 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fd7a8017090 con 0x7fd7b8070cc0 2024-09-13T17:19:14.856 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:14.856 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:14.857 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:14.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 msgr2=0x7fd79406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 0x7fd79406d0c0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fd7b81f5cb0 tx=0x7fd7a0009040 comp rx=0 tx=0).stop 2024-09-13T17:19:14.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 msgr2=0x7fd7b814e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800e9b0 tx=0x7fd7a800ee70 comp rx=0 tx=0).stop 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 shutdown_connections 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd79406ac00 0x7fd79406d0c0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7b815a540 0x7fd7b8142e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7b80cabc0 0x7fd7b81539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 --2- 172.21.15.28:0/3794652438 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7b8070cc0 0x7fd7b814e390 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 >> 172.21.15.28:0/3794652438 conn(0x7fd7b8069e80 msgr2=0x7fd7b8155ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 shutdown_connections 2024-09-13T17:19:14.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:14.857+0000 7fd7bea00640 1 -- 172.21.15.28:0/3794652438 wait complete. 2024-09-13T17:19:14.875 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.030+0000 7fd91f600640 1 Processor -- start 2024-09-13T17:19:15.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.030+0000 7fd91f600640 1 -- start start 2024-09-13T17:19:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91f600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd91806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91f600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd918070cc0 0x7fd918071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91f600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91815e100 con 0x7fd91815a540 2024-09-13T17:19:15.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91f600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180716b0 con 0x7fd918070cc0 2024-09-13T17:19:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91f600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180adb00 con 0x7fd9180cabc0 2024-09-13T17:19:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91cc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd91815a540 0x7fd91815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59212/0 (socket says 172.21.15.28:59212) 2024-09-13T17:19:15.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.031+0000 7fd91cc00640 1 -- 172.21.15.28:0/2887598154 learned_addr learned my addr 172.21.15.28:0/2887598154 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd917e00640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd91806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd91d600640 1 --2- 172.21.15.28:0/2887598154 >> v2:172.21.15.189:3300/0 conn(0x7fd918070cc0 0x7fd918071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590286451 0 0) 0x7fd91815e100 con 0x7fd91815a540 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd900003520 con 0x7fd91815a540 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd91d600640 1 -- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 msgr2=0x7fd91806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd91d600640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd91806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1279885471 0 0) 0x7fd900003520 con 0x7fd91815a540 2024-09-13T17:19:15.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd91d600640 1 -- 172.21.15.28:0/2887598154 >> v1:172.21.15.28:6789/0 conn(0x7fd91815a540 legacy=0x7fd91815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd91d600640 1 -- 172.21.15.28:0/2887598154 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91815e3e0 con 0x7fd918070cc0 2024-09-13T17:19:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.032+0000 7fd917e00640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd91806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:15.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.033+0000 7fd91d600640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd918070cc0 0x7fd918071170 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd910009dc0 tx=0x7fd91002a590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c167deee0f27505 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.034+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c001670 con 0x7fd918070cc0 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.034+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91002d050 con 0x7fd918070cc0 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.034+0000 7fd917400640 1 -- 172.21.15.28:0/2887598154 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910007760 con 0x7fd918070cc0 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 -- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd918070cc0 msgr2=0x7fd918071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd918070cc0 0x7fd918071170 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fd910009dc0 tx=0x7fd91002a590 comp rx=0 tx=0).stop 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 -- 172.21.15.28:0/2887598154 shutdown_connections 2024-09-13T17:19:15.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd918070cc0 0x7fd918071170 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 --2- 172.21.15.28:0/2887598154 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd91806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 -- 172.21.15.28:0/2887598154 >> 172.21.15.28:0/2887598154 conn(0x7fd918069e80 msgr2=0x7fd91806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:15.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 -- 172.21.15.28:0/2887598154 shutdown_connections 2024-09-13T17:19:15.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.035+0000 7fd91f600640 1 -- 172.21.15.28:0/2887598154 wait complete. 2024-09-13T17:19:15.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.036+0000 7fd91f600640 1 Processor -- start 2024-09-13T17:19:15.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.036+0000 7fd91f600640 1 -- start start 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd9181556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd91815a540 0x7fd9180b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91815ca90 con 0x7fd918070cc0 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91815cd30 con 0x7fd91815a540 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.037+0000 7fd91f600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91815d010 con 0x7fd9180cabc0 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39048/0 (socket says 172.21.15.28:39048) 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 -- 172.21.15.28:0/1979293 learned_addr learned my addr 172.21.15.28:0/1979293 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:15.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd917e00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd9181556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91d600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd91815a540 0x7fd9180b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 -- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 msgr2=0x7fd9181556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd9181556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 -- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd91815a540 msgr2=0x7fd9180b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd91815a540 0x7fd9180b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 -- 172.21.15.28:0/1979293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd904009400 con 0x7fd918070cc0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.038+0000 7fd91cc00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0011e0 tx=0x7fd90c00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.039+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c005ad0 con 0x7fd918070cc0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.039+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90c005c50 con 0x7fd918070cc0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.039+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd910009a70 con 0x7fd918070cc0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.039+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c007070 con 0x7fd918070cc0 2024-09-13T17:19:15.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.039+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91815d5d0 con 0x7fd918070cc0 2024-09-13T17:19:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.040+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd918156640 con 0x7fd918070cc0 2024-09-13T17:19:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.041+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd90c0071d0 con 0x7fd918070cc0 2024-09-13T17:19:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.042+0000 7fd914c00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 0x7fd8f806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.042+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fd90c088a30 con 0x7fd918070cc0 2024-09-13T17:19:15.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.042+0000 7fd917e00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 0x7fd8f806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.043+0000 7fd917e00640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 0x7fd8f806d0f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd9180ad6c0 tx=0x7fd904009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.046+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd918156640 con 0x7fd918070cc0 2024-09-13T17:19:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.262+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd918156640 con 0x7fd918070cc0 2024-09-13T17:19:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.262+0000 7fd914c00640 1 -- 172.21.15.28:0/1979293 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fd918156640 con 0x7fd918070cc0 2024-09-13T17:19:15.264 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:15.264 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":130,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:18:25.559972+0000","modified":"2024-09-13T17:18:28.571542+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9420},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9420":{"gid":9420,"name":"c","rank":0,"incarnation":127,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/3564905441","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3564905441},{"type":"v1","addr":"172.21.15.28:6835","nonce":3564905441}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2024-09-13T17:19:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 130 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 msgr2=0x7fd8f806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 0x7fd8f806d0f0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fd9180ad6c0 tx=0x7fd904009190 comp rx=0 tx=0).stop 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 msgr2=0x7fd9181f0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0011e0 tx=0x7fd90c00cbe0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 shutdown_connections 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd8f806ac30 0x7fd8f806d0f0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd91815a540 0x7fd9180b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9180cabc0 0x7fd9181556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 --2- 172.21.15.28:0/1979293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd918070cc0 0x7fd9181f0ef0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 >> 172.21.15.28:0/1979293 conn(0x7fd918069e80 msgr2=0x7fd91815ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 shutdown_connections 2024-09-13T17:19:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.265+0000 7fd91f600640 1 -- 172.21.15.28:0/1979293 wait complete. 2024-09-13T17:19:15.283 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.439+0000 7eff99800640 1 Processor -- start 2024-09-13T17:19:15.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.439+0000 7eff99800640 1 -- start start 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff99800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff941592a0 0x7eff9415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff99800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7eff9406c2d0 0x7eff9406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff99800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff9406cc70 con 0x7eff9406f970 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff99800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9415b0b0 con 0x7eff941592a0 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff99800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff94159d90 con 0x7eff9406c2d0 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff92a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff941592a0 0x7eff9415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.440+0000 7eff92a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7eff941592a0 0x7eff9415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39062/0 (socket says 172.21.15.28:39062) 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff92a00640 1 -- 172.21.15.28:0/417264998 learned_addr learned my addr 172.21.15.28:0/417264998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff93400640 1 --2- 172.21.15.28:0/417264998 >> v2:172.21.15.189:3300/0 conn(0x7eff9406c2d0 0x7eff9406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff92a00640 1 -- 172.21.15.28:0/417264998 >> v2:172.21.15.189:3300/0 conn(0x7eff9406c2d0 msgr2=0x7eff9406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff92a00640 1 --2- 172.21.15.28:0/417264998 >> v2:172.21.15.189:3300/0 conn(0x7eff9406c2d0 0x7eff9406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff92a00640 1 -- 172.21.15.28:0/417264998 >> v1:172.21.15.189:6789/0 conn(0x7eff9406f970 legacy=0x7eff94159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:15.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.441+0000 7eff92a00640 1 -- 172.21.15.28:0/417264998 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff940b2040 con 0x7eff941592a0 2024-09-13T17:19:15.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.442+0000 7eff92a00640 1 --2- 172.21.15.28:0/417264998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff9415ab70 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7eff940b1dd0 tx=0x7eff8402c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724df79cf040a7e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.442+0000 7eff92000640 1 -- 172.21.15.28:0/417264998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8402f070 con 0x7eff941592a0 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.442+0000 7eff92000640 1 -- 172.21.15.28:0/417264998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff840325a0 con 0x7eff941592a0 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.442+0000 7eff92000640 1 -- 172.21.15.28:0/417264998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff84032930 con 0x7eff941592a0 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 -- 172.21.15.28:0/417264998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 msgr2=0x7eff9415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 --2- 172.21.15.28:0/417264998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff9415ab70 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7eff940b1dd0 tx=0x7eff8402c8e0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 -- 172.21.15.28:0/417264998 shutdown_connections 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 --2- 172.21.15.28:0/417264998 >> v2:172.21.15.189:3300/0 conn(0x7eff9406c2d0 0x7eff9406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 --2- 172.21.15.28:0/417264998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff9415ab70 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 -- 172.21.15.28:0/417264998 >> 172.21.15.28:0/417264998 conn(0x7eff94069e80 msgr2=0x7eff9406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 -- 172.21.15.28:0/417264998 shutdown_connections 2024-09-13T17:19:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.443+0000 7eff99800640 1 -- 172.21.15.28:0/417264998 wait complete. 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.445+0000 7eff99800640 1 Processor -- start 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.445+0000 7eff99800640 1 -- start start 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9406c2d0 0x7eff94154a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9406f970 0x7eff94151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff940ce620 con 0x7eff941592a0 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff940cf2a0 con 0x7eff9406c2d0 2024-09-13T17:19:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff99800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff94152ef0 con 0x7eff9406f970 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39078/0 (socket says 172.21.15.28:39078) 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 -- 172.21.15.28:0/403125300 learned_addr learned my addr 172.21.15.28:0/403125300 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff92a00640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9406c2d0 0x7eff94154a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff8be00640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9406f970 0x7eff94151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 -- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9406f970 msgr2=0x7eff94151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9406f970 0x7eff94151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 -- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9406c2d0 msgr2=0x7eff94154a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9406c2d0 0x7eff94154a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.446+0000 7eff93400640 1 -- 172.21.15.28:0/403125300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8400b770 con 0x7eff941592a0 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff93400640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7eff8c007e40 tx=0x7eff8c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c014620 con 0x7eff941592a0 2024-09-13T17:19:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c00ee90 con 0x7eff941592a0 2024-09-13T17:19:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0149e0 con 0x7eff941592a0 2024-09-13T17:19:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff941522e0 con 0x7eff941592a0 2024-09-13T17:19:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.447+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff941528a0 con 0x7eff941592a0 2024-09-13T17:19:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.448+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff94153080 con 0x7eff941592a0 2024-09-13T17:19:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.449+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7eff8c014b40 con 0x7eff941592a0 2024-09-13T17:19:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.450+0000 7eff8aa00640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 0x7eff7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.450+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7eff8c089ce0 con 0x7eff941592a0 2024-09-13T17:19:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.450+0000 7eff92a00640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 0x7eff7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.451+0000 7eff92a00640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 0x7eff7006d0c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7eff84037000 tx=0x7eff84032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.454+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7eff8c054e90 con 0x7eff941592a0 2024-09-13T17:19:15.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.633+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff940b1b80 con 0x7eff941592a0 2024-09-13T17:19:15.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.634+0000 7eff8aa00640 1 -- 172.21.15.28:0/403125300 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v155) v1 ==== 74+0+16629 (secure 0 0 0) 0x7eff8c0584b0 con 0x7eff941592a0 2024-09-13T17:19:15.635 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:15.635 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:18:54.127866+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:18:23.576110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"152","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2024-09-13T17:18:24.545207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"21.29","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 msgr2=0x7eff7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 0x7eff7006d0c0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7eff84037000 tx=0x7eff84032000 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 msgr2=0x7eff94153380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7eff8c007e40 tx=0x7eff8c00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 shutdown_connections 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7eff7006ac00 0x7eff7006d0c0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff941592a0 0x7eff94153380 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7eff9406f970 0x7eff94151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 --2- 172.21.15.28:0/403125300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff9406c2d0 0x7eff94154a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 >> 172.21.15.28:0/403125300 conn(0x7eff94069e80 msgr2=0x7eff940aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 shutdown_connections 2024-09-13T17:19:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.636+0000 7eff99800640 1 -- 172.21.15.28:0/403125300 wait complete. 2024-09-13T17:19:15.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:19:15.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.810+0000 7faac8c00640 1 Processor -- start 2024-09-13T17:19:15.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.810+0000 7faac8c00640 1 -- start start 2024-09-13T17:19:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.811+0000 7faac8c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 0x7faac406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.811+0000 7faac8c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faac4071490 0x7faac406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.811+0000 7faac8c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac406c8b0 con 0x7faac415a780 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.811+0000 7faac8c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac406cb90 con 0x7faac4071490 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.811+0000 7faac8c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac40cc980 con 0x7faac415ab60 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faac4071490 0x7faac406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faac415a780 0x7faac4070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59228/0 (socket says 172.21.15.28:59228) 2024-09-13T17:19:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac1e00640 1 -- 172.21.15.28:0/4201160239 learned_addr learned my addr 172.21.15.28:0/4201160239 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac1400640 1 --2- 172.21.15.28:0/4201160239 >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 0x7faac406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac2800640 1 -- 172.21.15.28:0/4201160239 >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 msgr2=0x7faac406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac0a00640 1 -- 172.21.15.28:0/4201160239 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656076190 0 0) 0x7faac406c8b0 con 0x7faac415a780 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac2800640 1 --2- 172.21.15.28:0/4201160239 >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 0x7faac406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac2800640 1 -- 172.21.15.28:0/4201160239 >> v1:172.21.15.28:6789/0 conn(0x7faac415a780 legacy=0x7faac4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac2800640 1 -- 172.21.15.28:0/4201160239 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac40ccc60 con 0x7faac4071490 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.812+0000 7faac1400640 1 --2- 172.21.15.28:0/4201160239 >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 0x7faac406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.813+0000 7faac2800640 1 --2- 172.21.15.28:0/4201160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4071490 0x7faac406c080 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7faac406f380 tx=0x7faabc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91b656b4ed161054 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.813+0000 7faac0a00640 1 -- 172.21.15.28:0/4201160239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc02aea0 con 0x7faac4071490 2024-09-13T17:19:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.813+0000 7faac0a00640 1 -- 172.21.15.28:0/4201160239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc02d050 con 0x7faac4071490 2024-09-13T17:19:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.813+0000 7faac0a00640 1 -- 172.21.15.28:0/4201160239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc007960 con 0x7faac4071490 2024-09-13T17:19:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 -- 172.21.15.28:0/4201160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4071490 msgr2=0x7faac406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 --2- 172.21.15.28:0/4201160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4071490 0x7faac406c080 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7faac406f380 tx=0x7faabc02a680 comp rx=0 tx=0).stop 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 -- 172.21.15.28:0/4201160239 shutdown_connections 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 --2- 172.21.15.28:0/4201160239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac4071490 0x7faac406c080 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 --2- 172.21.15.28:0/4201160239 >> v2:172.21.15.189:3300/0 conn(0x7faac415ab60 0x7faac406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 -- 172.21.15.28:0/4201160239 >> 172.21.15.28:0/4201160239 conn(0x7faac40b0270 msgr2=0x7faac40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 -- 172.21.15.28:0/4201160239 shutdown_connections 2024-09-13T17:19:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.814+0000 7faac8c00640 1 -- 172.21.15.28:0/4201160239 wait complete. 2024-09-13T17:19:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.816+0000 7faac8c00640 1 Processor -- start 2024-09-13T17:19:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.816+0000 7faac8c00640 1 -- start start 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4071490 0x7faac415ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 0x7faac4155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac415bf90 con 0x7faac415a780 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac414b260 con 0x7faac4071490 2024-09-13T17:19:15.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac8c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac414b540 con 0x7faac415ab60 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39096/0 (socket says 172.21.15.28:39096) 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 -- 172.21.15.28:0/1488894531 learned_addr learned my addr 172.21.15.28:0/1488894531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1e00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4071490 0x7faac415ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac2800640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 0x7faac4155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 -- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 msgr2=0x7faac4155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 0x7faac4155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 -- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4071490 msgr2=0x7faac415ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4071490 0x7faac415ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac1400640 1 -- 172.21.15.28:0/1488894531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc009a70 con 0x7faac415a780 2024-09-13T17:19:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.817+0000 7faac2800640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 0x7faac4155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faac1400640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7faab000c7f0 tx=0x7faab000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0007820 con 0x7faac415a780 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab0007980 con 0x7faac415a780 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab0007cf0 con 0x7faac415a780 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac415af70 con 0x7faac415a780 2024-09-13T17:19:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.818+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac415b530 con 0x7faac415a780 2024-09-13T17:19:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.819+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac4070d80 con 0x7faac415a780 2024-09-13T17:19:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.820+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faab00113f0 con 0x7faac415a780 2024-09-13T17:19:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.821+0000 7faaba000640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 0x7faaa406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.821+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 5738+0+0 (secure 0 0 0) 0x7faab00895d0 con 0x7faac415a780 2024-09-13T17:19:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.822+0000 7faac1e00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 0x7faaa406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.822+0000 7faac1e00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 0x7faaa406d0f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faab4000f60 tx=0x7faab4007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:15.825+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faab0054830 con 0x7faac415a780 2024-09-13T17:19:16.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.041+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7faac40cbb90 con 0x7faac415a780 2024-09-13T17:19:16.043 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:19:16.042+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:19:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.051+0000 7faaba000640 1 -- 172.21.15.28:0/1488894531 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v131) v1 ==== 161+0+0 (secure 0 0 0) 0x7faab0087e60 con 0x7faac415a780 2024-09-13T17:19:16.052 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:19:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 msgr2=0x7faaa406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 0x7faaa406d0f0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7faab4000f60 tx=0x7faab4007d50 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 msgr2=0x7faac4147e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7faab000c7f0 tx=0x7faab000ccb0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 shutdown_connections 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faaa406ac30 0x7faaa406d0f0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faac415ab60 0x7faac4155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faac415a780 0x7faac4147e50 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 --2- 172.21.15.28:0/1488894531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faac4071490 0x7faac415ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.053+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 >> 172.21.15.28:0/1488894531 conn(0x7faac40b0270 msgr2=0x7faac40ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.054+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 shutdown_connections 2024-09-13T17:19:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.054+0000 7faac8c00640 1 -- 172.21.15.28:0/1488894531 wait complete. 2024-09-13T17:19:16.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:19:16.121 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:19:16.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.229+0000 7f06c1200640 1 Processor -- start 2024-09-13T17:19:16.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.229+0000 7f06c1200640 1 -- start start 2024-09-13T17:19:16.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06c1200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06c1200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 0x7f06bc06a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06c1200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06bc0ae200 con 0x7f06bc0b2dc0 2024-09-13T17:19:16.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06c1200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06bc06ae10 con 0x7f06bc06a440 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06c1200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0ae4e0 con 0x7f06bc15c900 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06bb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06bbe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 0x7f06bc06a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06bb400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:58320/0 (socket says 172.21.15.28:58320) 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06bb400640 1 -- 172.21.15.28:0/2923893267 learned_addr learned my addr 172.21.15.28:0/2923893267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:16.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06baa00640 1 -- 172.21.15.28:0/2923893267 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746560298 0 0) 0x7f06bc0ae200 con 0x7f06bc0b2dc0 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.231+0000 7f06baa00640 1 -- 172.21.15.28:0/2923893267 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0003520 con 0x7f06bc0b2dc0 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.232+0000 7f06bb400640 1 -- 172.21.15.28:0/2923893267 >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 msgr2=0x7f06bc06a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.232+0000 7f06bb400640 1 --2- 172.21.15.28:0/2923893267 >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 0x7f06bc06a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.232+0000 7f06bb400640 1 -- 172.21.15.28:0/2923893267 >> v1:172.21.15.189:6789/0 conn(0x7f06bc0b2dc0 legacy=0x7f06bc0af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.232+0000 7f06bb400640 1 -- 172.21.15.28:0/2923893267 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06bc0ae7c0 con 0x7f06bc15c900 2024-09-13T17:19:16.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.232+0000 7f06bbe00640 1 --2- 172.21.15.28:0/2923893267 >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 0x7f06bc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:16.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.233+0000 7f06bb400640 1 --2- 172.21.15.28:0/2923893267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009870 tx=0x7f06ac02a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c471b1fb76f47f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.233+0000 7f06baa00640 1 -- 172.21.15.28:0/2923893267 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac032440 con 0x7f06bc15c900 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.233+0000 7f06baa00640 1 -- 172.21.15.28:0/2923893267 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac0325a0 con 0x7f06bc15c900 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.233+0000 7f06baa00640 1 -- 172.21.15.28:0/2923893267 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac032930 con 0x7f06bc15c900 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 -- 172.21.15.28:0/2923893267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 msgr2=0x7f06bc069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 --2- 172.21.15.28:0/2923893267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f06ac009870 tx=0x7f06ac02a250 comp rx=0 tx=0).stop 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 -- 172.21.15.28:0/2923893267 shutdown_connections 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 --2- 172.21.15.28:0/2923893267 >> v2:172.21.15.189:3300/0 conn(0x7f06bc06a440 0x7f06bc06a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 --2- 172.21.15.28:0/2923893267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc15c900 0x7f06bc069e80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 -- 172.21.15.28:0/2923893267 >> 172.21.15.28:0/2923893267 conn(0x7f06bc0b2150 msgr2=0x7f06bc0b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 -- 172.21.15.28:0/2923893267 shutdown_connections 2024-09-13T17:19:16.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.234+0000 7f06c1200640 1 -- 172.21.15.28:0/2923893267 wait complete. 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.236+0000 7f06c1200640 1 Processor -- start 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.236+0000 7f06c1200640 1 -- start start 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 0x7f06bc15b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc0b2dc0 0x7f06bc0ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc15b710 con 0x7f06bc15c900 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0c4f00 con 0x7f06bc06a440 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06c1200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06bc0c51a0 con 0x7f06bc0b2dc0 2024-09-13T17:19:16.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39118/0 (socket says 172.21.15.28:39118) 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 -- 172.21.15.28:0/3883954706 learned_addr learned my addr 172.21.15.28:0/3883954706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bb400640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 0x7f06bc15b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06b2a00640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc0b2dc0 0x7f06bc0ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 -- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc0b2dc0 msgr2=0x7f06bc0ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc0b2dc0 0x7f06bc0ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 -- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 msgr2=0x7f06bc15b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 0x7f06bc15b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bbe00640 1 -- 172.21.15.28:0/3883954706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b4009a70 con 0x7f06bc15c900 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.237+0000 7f06bb400640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 0x7f06bc15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06bbe00640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efc0 tx=0x7f06b4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4009d70 con 0x7f06bc15c900 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b400f040 con 0x7f06bc15c900 2024-09-13T17:19:16.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ac0094d0 con 0x7f06bc15c900 2024-09-13T17:19:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b4003990 con 0x7f06bc15c900 2024-09-13T17:19:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.238+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06bc0c5760 con 0x7f06bc15c900 2024-09-13T17:19:16.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.239+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06bc15cce0 con 0x7f06bc15c900 2024-09-13T17:19:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.240+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f06b40144a0 con 0x7f06bc15c900 2024-09-13T17:19:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.241+0000 7f06b3e00640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 0x7f069806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.241+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f06b408a100 con 0x7f06bc15c900 2024-09-13T17:19:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.242+0000 7f06bb400640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 0x7f069806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:16.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.242+0000 7f06bb400640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 0x7f069806d0f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f06ac034000 tx=0x7f06ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:16.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.246+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f06b40551f0 con 0x7f06bc15c900 2024-09-13T17:19:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:16.459+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f06bc0ad710 con 0x7f06bc15c900 2024-09-13T17:19:17.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.053+0000 7f06b3e00640 1 -- 172.21.15.28:0/3883954706 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v132) v1 ==== 104+0+0 (secure 0 0 0) 0x7f06b4058810 con 0x7f06bc15c900 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 msgr2=0x7f069806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 0x7f069806d0f0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f06ac034000 tx=0x7f06ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 msgr2=0x7f06bc0ced80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f06b400efc0 tx=0x7f06b4009f90 comp rx=0 tx=0).stop 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 shutdown_connections 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f069806ac30 0x7f069806d0f0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06bc15c900 0x7f06bc0ced80 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06bc0b2dc0 0x7f06bc0ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 --2- 172.21.15.28:0/3883954706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06bc06a440 0x7f06bc15b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 >> 172.21.15.28:0/3883954706 conn(0x7f06bc0b2150 msgr2=0x7f06bc0b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 shutdown_connections 2024-09-13T17:19:17.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.056+0000 7f06c1200640 1 -- 172.21.15.28:0/3883954706 wait complete. 2024-09-13T17:19:17.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:19:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.238+0000 7f10b1e00640 1 Processor -- start 2024-09-13T17:19:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.238+0000 7f10b1e00640 1 -- start start 2024-09-13T17:19:17.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.239+0000 7f10b1e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f10ac157000 0x7f10ac06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.239+0000 7f10b1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 0x7f10ac15d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.239+0000 7f10b1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10ac15d950 con 0x7f10ac06b6b0 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10b1e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10ac15dc30 con 0x7f10ac157000 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10b1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac0b1b80 con 0x7f10ac15cdd0 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10aa000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f10ac06b6b0 0x7f10ac0b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59252/0 (socket says 172.21.15.28:59252) 2024-09-13T17:19:17.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10aa000640 1 -- 172.21.15.28:0/460534677 learned_addr learned my addr 172.21.15.28:0/460534677 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10aaa00640 1 --2- 172.21.15.28:0/460534677 >> v2:172.21.15.189:3300/0 conn(0x7f10ac157000 0x7f10ac06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10ab400640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 0x7f10ac15d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131716992 0 0) 0x7f10ac15d950 con 0x7f10ac06b6b0 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.240+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1090003520 con 0x7f10ac06b6b0 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10aaa00640 1 -- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 msgr2=0x7f10ac15d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10aaa00640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 0x7f10ac15d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10aaa00640 1 -- 172.21.15.28:0/460534677 >> v1:172.21.15.28:6789/0 conn(0x7f10ac06b6b0 legacy=0x7f10ac0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3235180775 0 0) 0x7f1090003520 con 0x7f10ac06b6b0 2024-09-13T17:19:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10aaa00640 1 -- 172.21.15.28:0/460534677 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10ac0b1e60 con 0x7f10ac157000 2024-09-13T17:19:17.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.241+0000 7f10ab400640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 0x7f10ac15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10aaa00640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac06b170 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f109c0097a0 tx=0x7f109c02a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b097e5d2c69c382 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1098001670 con 0x7f10ac157000 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c02aba0 con 0x7f10ac157000 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10a9600640 1 -- 172.21.15.28:0/460534677 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c032440 con 0x7f10ac157000 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10b1e00640 1 -- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 msgr2=0x7f10ac06b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.242+0000 7f10b1e00640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac06b170 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f109c0097a0 tx=0x7f109c02a110 comp rx=0 tx=0).stop 2024-09-13T17:19:17.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 -- 172.21.15.28:0/460534677 shutdown_connections 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac15cdd0 0x7f10ac15d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 --2- 172.21.15.28:0/460534677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac06b170 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 -- 172.21.15.28:0/460534677 >> 172.21.15.28:0/460534677 conn(0x7f10ac069e80 msgr2=0x7f10ac06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 -- 172.21.15.28:0/460534677 shutdown_connections 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.243+0000 7f10b1e00640 1 -- 172.21.15.28:0/460534677 wait complete. 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.244+0000 7f10b1e00640 1 Processor -- start 2024-09-13T17:19:17.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.244+0000 7f10b1e00640 1 -- start start 2024-09-13T17:19:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 0x7f10ac14f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac151c00 con 0x7f10ac15cdd0 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac153bb0 con 0x7f10ac157000 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10b1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10ac152fe0 con 0x7f10ac06b6b0 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.245+0000 7f10ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39124/0 (socket says 172.21.15.28:39124) 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 -- 172.21.15.28:0/3141636781 learned_addr learned my addr 172.21.15.28:0/3141636781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10aaa00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 0x7f10ac14f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10aa000640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 -- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 msgr2=0x7f10ac14f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 0x7f10ac14f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 -- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 msgr2=0x7f10ac151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 -- 172.21.15.28:0/3141636781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a4009a70 con 0x7f10ac15cdd0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10aaa00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 0x7f10ac14f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10ab400640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f10a400efc0 tx=0x7f10a4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.246+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4009d70 con 0x7f10ac15cdd0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.247+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a400f040 con 0x7f10ac15cdd0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.247+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109c009400 con 0x7f10ac15cdd0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.247+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4003990 con 0x7f10ac15cdd0 2024-09-13T17:19:17.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.247+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10ac153560 con 0x7f10ac15cdd0 2024-09-13T17:19:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.248+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10ac153d60 con 0x7f10ac15cdd0 2024-09-13T17:19:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.249+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f10a40144a0 con 0x7f10ac15cdd0 2024-09-13T17:19:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.249+0000 7f10a3400640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 0x7f108c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:17.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.250+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(156..156 src has 1..156) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f10a408a150 con 0x7f10ac15cdd0 2024-09-13T17:19:17.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.250+0000 7f10aaa00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 0x7f108c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:17.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.251+0000 7f10aaa00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 0x7f108c06d0c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f109c034000 tx=0x7f109c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:17.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.254+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f10ac153d60 con 0x7f10ac15cdd0 2024-09-13T17:19:17.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:17.434+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10ac153d60 con 0x7f10ac15cdd0 2024-09-13T17:19:18.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.235+0000 7f10a3400640 1 -- 172.21.15.28:0/3141636781 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v157) v1 ==== 181+0+0 (secure 0 0 0) 0x7f10ac153d60 con 0x7f10ac15cdd0 2024-09-13T17:19:18.237 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:19:18.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 msgr2=0x7f108c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:18.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 0x7f108c06d0c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f109c034000 tx=0x7f109c02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:18.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 msgr2=0x7f10ac1516c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f10a400efc0 tx=0x7f10a4009f90 comp rx=0 tx=0).stop 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 shutdown_connections 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f108c06ac00 0x7f108c06d0c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10ac15cdd0 0x7f10ac1516c0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10ac157000 0x7f10ac151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 --2- 172.21.15.28:0/3141636781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f10ac06b6b0 0x7f10ac14f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.239+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 >> 172.21.15.28:0/3141636781 conn(0x7f10ac069e80 msgr2=0x7f10ac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.240+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 shutdown_connections 2024-09-13T17:19:18.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.240+0000 7f10b1e00640 1 -- 172.21.15.28:0/3141636781 wait complete. 2024-09-13T17:19:18.262 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:19:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.425+0000 7f9f2bc00640 1 Processor -- start 2024-09-13T17:19:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.425+0000 7f9f2bc00640 1 -- start start 2024-09-13T17:19:18.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f2406b2f0 0x7f9f240ae500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f240aea40 0x7f9f24070980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 0x7f9f24072e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f24071350 con 0x7f9f240aea40 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f241603a0 con 0x7f9f24070ec0 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f2bc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240aefa0 con 0x7f9f2406b2f0 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f23e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f240aea40 0x7f9f24070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f23e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9f240aea40 0x7f9f24070980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39136/0 (socket says 172.21.15.28:39136) 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.426+0000 7f9f23e00640 1 -- 172.21.15.28:0/3073032322 learned_addr learned my addr 172.21.15.28:0/3073032322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f29200640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f2406b2f0 0x7f9f240ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f29c00640 1 --2- 172.21.15.28:0/3073032322 >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 0x7f9f24072e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f23e00640 1 -- 172.21.15.28:0/3073032322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f2406b2f0 msgr2=0x7f9f240ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f23e00640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f2406b2f0 0x7f9f240ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f23e00640 1 -- 172.21.15.28:0/3073032322 >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 msgr2=0x7f9f24072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f23e00640 1 --2- 172.21.15.28:0/3073032322 >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 0x7f9f24072e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f23e00640 1 -- 172.21.15.28:0/3073032322 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f241605f0 con 0x7f9f240aea40 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.427+0000 7f9f29c00640 1 --2- 172.21.15.28:0/3073032322 >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 0x7f9f24072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.428+0000 7f9f23e00640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f240aea40 0x7f9f24070980 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9f240b4ce0 tx=0x7f9f1002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ef1abb31cc04da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.428+0000 7f9f23400640 1 -- 172.21.15.28:0/3073032322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10032440 con 0x7f9f240aea40 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.428+0000 7f9f23400640 1 -- 172.21.15.28:0/3073032322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f100325a0 con 0x7f9f240aea40 2024-09-13T17:19:18.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.428+0000 7f9f23400640 1 -- 172.21.15.28:0/3073032322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10032930 con 0x7f9f240aea40 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 -- 172.21.15.28:0/3073032322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f240aea40 msgr2=0x7f9f24070980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f240aea40 0x7f9f24070980 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f9f240b4ce0 tx=0x7f9f1002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 -- 172.21.15.28:0/3073032322 shutdown_connections 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f240aea40 0x7f9f24070980 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/3073032322 >> v2:172.21.15.189:3300/0 conn(0x7f9f24070ec0 0x7f9f24072e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/3073032322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f2406b2f0 0x7f9f240ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 -- 172.21.15.28:0/3073032322 >> 172.21.15.28:0/3073032322 conn(0x7f9f240b17b0 msgr2=0x7f9f240b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 -- 172.21.15.28:0/3073032322 shutdown_connections 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.429+0000 7f9f2bc00640 1 -- 172.21.15.28:0/3073032322 wait complete. 2024-09-13T17:19:18.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.431+0000 7f9f2bc00640 1 Processor -- start 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.431+0000 7f9f2bc00640 1 -- start start 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.431+0000 7f9f2bc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f2406b2f0 0x7f9f24152370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f2bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f2bc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f240aea40 0x7f9f241588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f2bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f240cdf30 con 0x7f9f24070ec0 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f2bc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f241528b0 con 0x7f9f2406b2f0 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f2bc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f241f2d10 con 0x7f9f240aea40 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39152/0 (socket says 172.21.15.28:39152) 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 -- 172.21.15.28:0/1006960562 learned_addr learned my addr 172.21.15.28:0/1006960562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f29200640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f2406b2f0 0x7f9f24152370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f29c00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f240aea40 0x7f9f241588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 -- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f240aea40 msgr2=0x7f9f241588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f240aea40 0x7f9f241588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 -- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f2406b2f0 msgr2=0x7f9f24152370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f2406b2f0 0x7f9f24152370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.432+0000 7f9f23e00640 1 -- 172.21.15.28:0/1006960562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f100094d0 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f23e00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f9f10000c00 tx=0x7f9f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f10032440 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f10030b00 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1003d440 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f240cd590 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.433+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f240cdb50 con 0x7f9f24070ec0 2024-09-13T17:19:18.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.434+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f240ce070 con 0x7f9f24070ec0 2024-09-13T17:19:18.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.435+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9f10039040 con 0x7f9f24070ec0 2024-09-13T17:19:18.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.436+0000 7f9f20c00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 0x7f9f0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:18.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.436+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(157..157 src has 1..157) v4 ==== 5386+0+0 (secure 0 0 0) 0x7f9f100a9d10 con 0x7f9f24070ec0 2024-09-13T17:19:18.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.437+0000 7f9f29200640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 0x7f9f0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:18.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.437+0000 7f9f29200640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 0x7f9f0406d0c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9f18006fd0 tx=0x7f9f18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.440+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9f240ce070 con 0x7f9f24070ec0 2024-09-13T17:19:18.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:18.624+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9f240ce070 con 0x7f9f24070ec0 2024-09-13T17:19:19.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.241+0000 7f9f20c00640 1 -- 172.21.15.28:0/1006960562 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v158) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9f240ce070 con 0x7f9f24070ec0 2024-09-13T17:19:19.244 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:19:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 msgr2=0x7f9f0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 0x7f9f0406d0c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9f18006fd0 tx=0x7f9f18008040 comp rx=0 tx=0).stop 2024-09-13T17:19:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 msgr2=0x7f9f241f27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f9f10000c00 tx=0x7f9f100046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 shutdown_connections 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9f0406ac00 0x7f9f0406d0c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9f240aea40 0x7f9f241588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9f24070ec0 0x7f9f241f27d0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 --2- 172.21.15.28:0/1006960562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9f2406b2f0 0x7f9f24152370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.246+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 >> 172.21.15.28:0/1006960562 conn(0x7f9f240b17b0 msgr2=0x7f9f240b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.247+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 shutdown_connections 2024-09-13T17:19:19.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.247+0000 7f9f2bc00640 1 -- 172.21.15.28:0/1006960562 wait complete. 2024-09-13T17:19:19.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:19:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.448+0000 7fa4d7e00640 1 Processor -- start 2024-09-13T17:19:19.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.448+0000 7fa4d7e00640 1 -- start start 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.449+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015a780 0x7fa4d0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.449+0000 7fa4d7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4d006c250 0x7fa4d00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.449+0000 7fa4d7e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d006c8b0 con 0x7fa4d015ab60 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.449+0000 7fa4d7e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d006c6a0 con 0x7fa4d006c250 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.449+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d006cb90 con 0x7fa4d015a780 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4d006c250 0x7fa4d00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4d006c250 0x7fa4d00712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39168/0 (socket says 172.21.15.28:39168) 2024-09-13T17:19:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 -- 172.21.15.28:0/1887384638 learned_addr learned my addr 172.21.15.28:0/1887384638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5400640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015a780 0x7fa4d0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d4a00640 1 --1- 172.21.15.28:0/1887384638 >> v1:172.21.15.189:6789/0 conn(0x7fa4d015ab60 0x7fa4d006bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:60290/0 (socket says 172.21.15.28:60290) 2024-09-13T17:19:19.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 -- 172.21.15.28:0/1887384638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015a780 msgr2=0x7fa4d0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015a780 0x7fa4d0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 -- 172.21.15.28:0/1887384638 >> v1:172.21.15.189:6789/0 conn(0x7fa4d015ab60 legacy=0x7fa4d006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4d5e00640 1 -- 172.21.15.28:0/1887384638 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d00cc980 con 0x7fa4d006c250 2024-09-13T17:19:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.450+0000 7fa4cfe00640 1 -- 172.21.15.28:0/1887384638 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192907522 0 0) 0x7fa4d006c8b0 con 0x7fa4d015ab60 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.451+0000 7fa4d5e00640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d00712c0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009e50 tx=0x7fa4c802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752d63fa194994b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.451+0000 7fa4cfe00640 1 -- 172.21.15.28:0/1887384638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8031600 con 0x7fa4d006c250 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.451+0000 7fa4cfe00640 1 -- 172.21.15.28:0/1887384638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c802d050 con 0x7fa4d006c250 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.451+0000 7fa4cfe00640 1 -- 172.21.15.28:0/1887384638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8031980 con 0x7fa4d006c250 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 -- 172.21.15.28:0/1887384638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 msgr2=0x7fa4d00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d00712c0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009e50 tx=0x7fa4c802a710 comp rx=0 tx=0).stop 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 -- 172.21.15.28:0/1887384638 shutdown_connections 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d00712c0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/1887384638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015a780 0x7fa4d0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 -- 172.21.15.28:0/1887384638 >> 172.21.15.28:0/1887384638 conn(0x7fa4d00b0270 msgr2=0x7fa4d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 -- 172.21.15.28:0/1887384638 shutdown_connections 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.452+0000 7fa4d7e00640 1 -- 172.21.15.28:0/1887384638 wait complete. 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.454+0000 7fa4d7e00640 1 Processor -- start 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.454+0000 7fa4d7e00640 1 -- start start 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d015a780 0x7fa4d0157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015ab60 0x7fa4d0157d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0147a20 con 0x7fa4d006c250 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0145e20 con 0x7fa4d015a780 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d0146e10 con 0x7fa4d015ab60 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39170/0 (socket says 172.21.15.28:39170) 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 -- 172.21.15.28:0/3702089032 learned_addr learned my addr 172.21.15.28:0/3702089032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 -- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015ab60 msgr2=0x7fa4d0157d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015ab60 0x7fa4d0157d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 -- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d015a780 msgr2=0x7fa4d0157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d015a780 0x7fa4d0157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.455+0000 7fa4d5400640 1 -- 172.21.15.28:0/3702089032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8009b30 con 0x7fa4d006c250 2024-09-13T17:19:19.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4d5400640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400e9b0 tx=0x7fa4c400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4009670 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c400c9a0 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c400cd50 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d01470f0 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.456+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d01476b0 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.457+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d006c8b0 con 0x7fa4d006c250 2024-09-13T17:19:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.458+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa4c4013470 con 0x7fa4d006c250 2024-09-13T17:19:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.459+0000 7fa4cd600640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 0x7fa4b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.459+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa4c4089660 con 0x7fa4d006c250 2024-09-13T17:19:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.459+0000 7fa4d4a00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 0x7fa4b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.460+0000 7fa4d4a00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 0x7fa4b006d0c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc007fd0 tx=0x7fa4bc007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.463+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa4c4054b30 con 0x7fa4d006c250 2024-09-13T17:19:19.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.642+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa4d00af070 con 0x7fa4d006c250 2024-09-13T17:19:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.647+0000 7fa4cd600640 1 -- 172.21.15.28:0/3702089032 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa4c4058150 con 0x7fa4d006c250 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 msgr2=0x7fa4b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 0x7fa4b006d0c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc007fd0 tx=0x7fa4bc007960 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 msgr2=0x7fa4d01458e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fa4c400e9b0 tx=0x7fa4c400ee70 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 shutdown_connections 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b006ac00 0x7fa4b006d0c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d015ab60 0x7fa4d0157d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d015a780 0x7fa4d0157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 --2- 172.21.15.28:0/3702089032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d006c250 0x7fa4d01458e0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.649+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 >> 172.21.15.28:0/3702089032 conn(0x7fa4d00b0270 msgr2=0x7fa4d00ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.650+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 shutdown_connections 2024-09-13T17:19:19.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.650+0000 7fa4d7e00640 1 -- 172.21.15.28:0/3702089032 wait complete. 2024-09-13T17:19:19.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort' 2024-09-13T17:19:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.820+0000 7fddf7a00640 1 Processor -- start 2024-09-13T17:19:19.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.820+0000 7fddf7a00640 1 -- start start 2024-09-13T17:19:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.821+0000 7fddf7a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fddf006f970 0x7fddf015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddf7a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf0159d90 con 0x7fddf01592a0 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddf7a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf00b2040 con 0x7fddf006c2d0 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddf7a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf015b0b0 con 0x7fddf006f970 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddf5000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fddf01592a0 0x7fddf0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59254/0 (socket says 172.21.15.28:59254) 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddf5000640 1 -- 172.21.15.28:0/2921792495 learned_addr learned my addr 172.21.15.28:0/2921792495 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddefe00640 1 --2- 172.21.15.28:0/2921792495 >> v2:172.21.15.189:3300/0 conn(0x7fddf006f970 0x7fddf015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772082624 0 0) 0x7fddf0159d90 con 0x7fddf01592a0 2024-09-13T17:19:19.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.822+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd8003520 con 0x7fddf01592a0 2024-09-13T17:19:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.823+0000 7fddefe00640 1 -- 172.21.15.28:0/2921792495 >> v1:172.21.15.189:6789/0 conn(0x7fddf006c2d0 legacy=0x7fddf00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.823+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720360431 0 0) 0x7fddf00b2040 con 0x7fddf006c2d0 2024-09-13T17:19:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.823+0000 7fddefe00640 1 -- 172.21.15.28:0/2921792495 >> v1:172.21.15.28:6789/0 conn(0x7fddf01592a0 legacy=0x7fddf0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.823+0000 7fddefe00640 1 -- 172.21.15.28:0/2921792495 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf0159ff0 con 0x7fddf006f970 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.824+0000 7fddefe00640 1 --2- 172.21.15.28:0/2921792495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006f970 0x7fddf015ab70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdddc0097a0 tx=0x7fdddc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c50c885fdf6ba4b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.824+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc032440 con 0x7fddf006f970 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.824+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc0325a0 con 0x7fddf006f970 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.824+0000 7fddef400640 1 -- 172.21.15.28:0/2921792495 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc032930 con 0x7fddf006f970 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 -- 172.21.15.28:0/2921792495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006f970 msgr2=0x7fddf015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2921792495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006f970 0x7fddf015ab70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fdddc0097a0 tx=0x7fdddc02a230 comp rx=0 tx=0).stop 2024-09-13T17:19:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 -- 172.21.15.28:0/2921792495 shutdown_connections 2024-09-13T17:19:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2921792495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf006f970 0x7fddf015ab70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 -- 172.21.15.28:0/2921792495 >> 172.21.15.28:0/2921792495 conn(0x7fddf0069e80 msgr2=0x7fddf006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 -- 172.21.15.28:0/2921792495 shutdown_connections 2024-09-13T17:19:19.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.825+0000 7fddf7a00640 1 -- 172.21.15.28:0/2921792495 wait complete. 2024-09-13T17:19:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.827+0000 7fddf7a00640 1 Processor -- start 2024-09-13T17:19:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.827+0000 7fddf7a00640 1 -- start start 2024-09-13T17:19:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fddf006c2d0 0x7fddf0143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf01592a0 0x7fddf0153700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf00cf4d0 con 0x7fddf006f970 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0151a70 con 0x7fddf01592a0 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf7a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf0151d50 con 0x7fddf006c2d0 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39174/0 (socket says 172.21.15.28:39174) 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 -- 172.21.15.28:0/2753649049 learned_addr learned my addr 172.21.15.28:0/2753649049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddf5000640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fddf006c2d0 0x7fddf0143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 -- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fddf006c2d0 msgr2=0x7fddf0143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fddf006c2d0 0x7fddf0143d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 -- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf01592a0 msgr2=0x7fddf0153700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf01592a0 0x7fddf0153700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:19.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.828+0000 7fddefe00640 1 -- 172.21.15.28:0/2753649049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc009400 con 0x7fddf006f970 2024-09-13T17:19:19.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddefe00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fdddc035040 tx=0x7fdddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc032440 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdddc030b00 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf0152030 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdddc030e90 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.829+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf01442d0 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.830+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf0152550 con 0x7fddf006f970 2024-09-13T17:19:19.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.831+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdddc039040 con 0x7fddf006f970 2024-09-13T17:19:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.832+0000 7fddecc00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 0x7fddd006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.832+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdddc0a9bb0 con 0x7fddf006f970 2024-09-13T17:19:19.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.833+0000 7fddf5000640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 0x7fddd006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:19.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.834+0000 7fddf5000640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 0x7fddd006d0f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdde4001200 tx=0x7fdde4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:19.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:19.836+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fddf0152550 con 0x7fddf006f970 2024-09-13T17:19:20.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.095+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]} v 0) v1 -- 0x7fddf0152550 con 0x7fddf006f970 2024-09-13T17:19:20.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.242+0000 7fddecc00640 1 -- 172.21.15.28:0/2753649049 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_pause_and_resume_with_abort"]}]=0 v565) v1 ==== 166+0+0 (secure 0 0 0) 0x7fddf0152550 con 0x7fddf006f970 2024-09-13T17:19:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 msgr2=0x7fddd006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 0x7fddd006d0f0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fdde4001200 tx=0x7fdde4009040 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 msgr2=0x7fddf01f2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fdddc035040 tx=0x7fdddc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 shutdown_connections 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fddd006ac30 0x7fddd006d0f0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fddf01592a0 0x7fddf0153700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fddf006f970 0x7fddf01f2240 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 --2- 172.21.15.28:0/2753649049 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fddf006c2d0 0x7fddf0143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 >> 172.21.15.28:0/2753649049 conn(0x7fddf0069e80 msgr2=0x7fddf0154e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 shutdown_connections 2024-09-13T17:19:20.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.244+0000 7fddf7a00640 1 -- 172.21.15.28:0/2753649049 wait complete. 2024-09-13T17:19:20.262 INFO:tasks.cephfs_test_runner:test_scrub_pause_and_resume_with_abort (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-09-13T17:19:20.263 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-09-13T17:19:20.263 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:20.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.417+0000 7fc85fe00640 1 Processor -- start 2024-09-13T17:19:20.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.417+0000 7fc85fe00640 1 -- start start 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.418+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858146170 0x7fc85815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.418+0000 7fc85fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc858160310 0x7fc858160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.418+0000 7fc85fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc858146550 con 0x7fc8580ac850 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.418+0000 7fc85fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc85815cd20 con 0x7fc858160310 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.418+0000 7fc85fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85815d280 con 0x7fc858146170 2024-09-13T17:19:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc858160310 0x7fc858160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc8580ac850 0x7fc85806b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59264/0 (socket says 172.21.15.28:59264) 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85de00640 1 -- 172.21.15.28:0/3263745017 learned_addr learned my addr 172.21.15.28:0/3263745017 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85d400640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858146170 0x7fc85815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85ca00640 1 -- 172.21.15.28:0/3263745017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858146170 msgr2=0x7fc85815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85ca00640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858146170 0x7fc85815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85ca00640 1 -- 172.21.15.28:0/3263745017 >> v1:172.21.15.28:6789/0 conn(0x7fc8580ac850 legacy=0x7fc85806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc85ca00640 1 -- 172.21.15.28:0/3263745017 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85815dc50 con 0x7fc858160310 2024-09-13T17:19:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.419+0000 7fc857e00640 1 -- 172.21.15.28:0/3263745017 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3011468189 0 0) 0x7fc858146550 con 0x7fc8580ac850 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.420+0000 7fc85ca00640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858160310 0x7fc858160750 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc84402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0ad8edf424cac0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc857e00640 1 -- 172.21.15.28:0/3263745017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844032440 con 0x7fc858160310 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc857e00640 1 -- 172.21.15.28:0/3263745017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8440325a0 con 0x7fc858160310 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc857e00640 1 -- 172.21.15.28:0/3263745017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844032930 con 0x7fc858160310 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 -- 172.21.15.28:0/3263745017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858160310 msgr2=0x7fc858160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858160310 0x7fc858160750 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc84402a180 comp rx=0 tx=0).stop 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 -- 172.21.15.28:0/3263745017 shutdown_connections 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858160310 0x7fc858160750 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 --2- 172.21.15.28:0/3263745017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858146170 0x7fc85815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 -- 172.21.15.28:0/3263745017 >> 172.21.15.28:0/3263745017 conn(0x7fc858069e80 msgr2=0x7fc85806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 -- 172.21.15.28:0/3263745017 shutdown_connections 2024-09-13T17:19:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.421+0000 7fc85fe00640 1 -- 172.21.15.28:0/3263745017 wait complete. 2024-09-13T17:19:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.423+0000 7fc85fe00640 1 Processor -- start 2024-09-13T17:19:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.423+0000 7fc85fe00640 1 -- start start 2024-09-13T17:19:20.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 0x7fc8581307b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858160310 0x7fc85813e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85813da80 con 0x7fc858146170 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85813dd60 con 0x7fc8580ac850 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc85813e040 con 0x7fc858160310 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85ca00640 1 -- 172.21.15.28:0/92414916 learned_addr learned my addr 172.21.15.28:0/92414916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85de00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858160310 0x7fc85813e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.424+0000 7fc85d400640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 0x7fc8581307b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 -- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858160310 msgr2=0x7fc85813e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858160310 0x7fc85813e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 -- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 msgr2=0x7fc8581307b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 0x7fc8581307b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 -- 172.21.15.28:0/92414916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc844009400 con 0x7fc858146170 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85d400640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 0x7fc8581307b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85ca00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc8440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844030b80 con 0x7fc858146170 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc844032e90 con 0x7fc858146170 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844006e10 con 0x7fc858146170 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.425+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85813e320 con 0x7fc858146170 2024-09-13T17:19:20.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.426+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85813d420 con 0x7fc858146170 2024-09-13T17:19:20.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.427+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc858066e70 con 0x7fc858146170 2024-09-13T17:19:20.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.427+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc844039040 con 0x7fc858146170 2024-09-13T17:19:20.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.428+0000 7fc855600640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 0x7fc83806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.428+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc8440a9870 con 0x7fc858146170 2024-09-13T17:19:20.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.429+0000 7fc85d400640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 0x7fc83806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.429+0000 7fc85d400640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 0x7fc83806d0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009b90 tx=0x7fc84c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.432+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc844074d60 con 0x7fc858146170 2024-09-13T17:19:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.631+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc85813dbc0 con 0x7fc858146170 2024-09-13T17:19:20.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.632+0000 7fc855600640 1 -- 172.21.15.28:0/92414916 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v158) v1 ==== 74+0+12996 (secure 0 0 0) 0x7fc85813dbc0 con 0x7fc858146170 2024-09-13T17:19:20.634 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:20.634 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":158,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:19:19.234745+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3564905441":"2024-09-14T17:19:16.042603+0000","172.21.15.28:6833/3564905441":"2024-09-14T17:19:16.042603+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 msgr2=0x7fc83806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 0x7fc83806d0c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009b90 tx=0x7fc84c008040 comp rx=0 tx=0).stop 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 msgr2=0x7fc85806c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fc8440097a0 tx=0x7fc8440046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 shutdown_connections 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc83806ac00 0x7fc83806d0c0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc858160310 0x7fc85813e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc858146170 0x7fc85806c5b0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 --2- 172.21.15.28:0/92414916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8580ac850 0x7fc8581307b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 >> 172.21.15.28:0/92414916 conn(0x7fc858069e80 msgr2=0x7fc8580ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 shutdown_connections 2024-09-13T17:19:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.635+0000 7fc85fe00640 1 -- 172.21.15.28:0/92414916 wait complete. 2024-09-13T17:19:20.652 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:19:20.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.809+0000 7f91c0600640 1 Processor -- start 2024-09-13T17:19:20.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.809+0000 7f91c0600640 1 -- start start 2024-09-13T17:19:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91c0600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91b806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91c0600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b806f920 con 0x7f91b815cd80 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91c0600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91b815b290 con 0x7f91b806f3f0 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91c0600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815b570 con 0x7f91b8159770 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91be600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91b815cd80 0x7f91b815a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59276/0 (socket says 172.21.15.28:59276) 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.810+0000 7f91be600640 1 -- 172.21.15.28:0/3737877422 learned_addr learned my addr 172.21.15.28:0/3737877422 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91bdc00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91b806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483113791 0 0) 0x7f91b806f920 con 0x7f91b815cd80 2024-09-13T17:19:20.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91a0003520 con 0x7f91b815cd80 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465063473 0 0) 0x7f91b815b290 con 0x7f91b806f3f0 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91b806f920 con 0x7f91b806f3f0 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 535972640 0 0) 0x7f91a0003520 con 0x7f91b815cd80 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 msgr2=0x7f91b806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91b806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 >> v1:172.21.15.189:6789/0 conn(0x7f91b806f3f0 legacy=0x7f91b8159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b815b850 con 0x7f91b815cd80 2024-09-13T17:19:20.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f91b00019f0 con 0x7f91b815cd80 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 >> v1:172.21.15.28:6789/0 conn(0x7f91b815cd80 legacy=0x7f91b815a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0003b30 0x7f91a0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a0004d00 0x7f91a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b815b570 con 0x7f91a0004d00 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000dd20 con 0x7f91a0003b30 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.811+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91a000deb0 con 0x7f91b8159770 2024-09-13T17:19:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a0004d00 0x7f91a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91bd200640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91bdc00640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0003b30 0x7f91a0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 -- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 msgr2=0x7f91a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 -- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0003b30 msgr2=0x7f91a0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0003b30 0x7f91a0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91be600640 1 -- 172.21.15.28:0/3737877422 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0003520 con 0x7f91a0004d00 2024-09-13T17:19:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.812+0000 7f91bd200640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91b8159770 0x7f91a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.813+0000 7f91be600640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a0004d00 0x7f91a0005180 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f91b80b0e40 tx=0x7f91a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bb00f96d8f44cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.813+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b002b870 con 0x7f91a0004d00 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.813+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b002b9d0 con 0x7f91a0004d00 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.813+0000 7f91b7e00640 1 -- 172.21.15.28:0/3737877422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b002bda0 con 0x7f91a0004d00 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 -- 172.21.15.28:0/3737877422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a0004d00 msgr2=0x7f91a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 --2- 172.21.15.28:0/3737877422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91a0004d00 0x7f91a0005180 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f91b80b0e40 tx=0x7f91a00057f0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91be600640 1 -- 172.21.15.28:0/3737877422 reap_dead start 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 -- 172.21.15.28:0/3737877422 shutdown_connections 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 -- 172.21.15.28:0/3737877422 >> 172.21.15.28:0/3737877422 conn(0x7f91b80b0270 msgr2=0x7f91b80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 -- 172.21.15.28:0/3737877422 shutdown_connections 2024-09-13T17:19:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.814+0000 7f91c0600640 1 -- 172.21.15.28:0/3737877422 wait complete. 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.816+0000 7f91c0600640 1 Processor -- start 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.816+0000 7f91c0600640 1 -- start start 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a0004d00 0x7f91b8158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 0x7f91b813faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b8159060 con 0x7f91b80aede0 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80cba10 con 0x7f91b80ae960 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91c0600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91b80cdde0 con 0x7f91a0004d00 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39224/0 (socket says 172.21.15.28:39224) 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 -- 172.21.15.28:0/3096178260 learned_addr learned my addr 172.21.15.28:0/3096178260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91bdc00640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a0004d00 0x7f91b8158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91bd200640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 0x7f91b813faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 -- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a0004d00 msgr2=0x7f91b8158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a0004d00 0x7f91b8158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.817+0000 7f91be600640 1 -- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 msgr2=0x7f91b813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91be600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 0x7f91b813faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91be600640 1 -- 172.21.15.28:0/3096178260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91b80ce0c0 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91bd200640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 0x7f91b813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91be600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f91b0000f80 tx=0x7f91b0001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0038070 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91b00011c0 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b80ce3a0 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91b0034670 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.818+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b814d200 con 0x7f91b80aede0 2024-09-13T17:19:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.819+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b806d190 con 0x7f91b80aede0 2024-09-13T17:19:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.820+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f91b0031070 con 0x7f91b80aede0 2024-09-13T17:19:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.821+0000 7f91b5600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 0x7f919806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.821+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f91b003b040 con 0x7f91b80aede0 2024-09-13T17:19:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.822+0000 7f91bdc00640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 0x7f919806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.822+0000 7f91bdc00640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 0x7f919806d0c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009b40 tx=0x7f91ac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:20.825+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f91b0074290 con 0x7f91b80aede0 2024-09-13T17:19:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.003+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91b80b38e0 con 0x7f91b80aede0 2024-09-13T17:19:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.003+0000 7f91b5600640 1 -- 172.21.15.28:0/3096178260 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v158) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91b00778b0 con 0x7f91b80aede0 2024-09-13T17:19:21.006 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 msgr2=0x7f919806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 0x7f919806d0c0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f91ac009b40 tx=0x7f91ac009190 comp rx=0 tx=0).stop 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 msgr2=0x7f91b8140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f91b0000f80 tx=0x7f91b0001cb0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 shutdown_connections 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f919806ac00 0x7f919806d0c0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91b80aede0 0x7f91b8140030 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91b80ae960 0x7f91b813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.005+0000 7f91c0600640 1 --2- 172.21.15.28:0/3096178260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91a0004d00 0x7f91b8158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.006+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 >> 172.21.15.28:0/3096178260 conn(0x7f91b80b0270 msgr2=0x7f91b806fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.006+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 shutdown_connections 2024-09-13T17:19:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.006+0000 7f91c0600640 1 -- 172.21.15.28:0/3096178260 wait complete. 2024-09-13T17:19:21.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover' 2024-09-13T17:19:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.178+0000 7f4567a00640 1 Processor -- start 2024-09-13T17:19:21.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.178+0000 7f4567a00640 1 -- start start 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4567a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45600cabc0 0x7f456006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4567a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4560070cc0 0x7f4560071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4567a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456015e100 con 0x7f456015a540 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4567a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45600adb00 con 0x7f45600cabc0 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4567a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4560071670 con 0x7f4560070cc0 2024-09-13T17:19:21.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45600cabc0 0x7f456006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f45600cabc0 0x7f456006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39232/0 (socket says 172.21.15.28:39232) 2024-09-13T17:19:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 -- 172.21.15.28:0/1734069934 learned_addr learned my addr 172.21.15.28:0/1734069934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4565a00640 1 --2- 172.21.15.28:0/1734069934 >> v2:172.21.15.189:3300/0 conn(0x7f4560070cc0 0x7f4560071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 -- 172.21.15.28:0/1734069934 >> v2:172.21.15.189:3300/0 conn(0x7f4560070cc0 msgr2=0x7f4560071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:21.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 --2- 172.21.15.28:0/1734069934 >> v2:172.21.15.189:3300/0 conn(0x7f4560070cc0 0x7f4560071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 -- 172.21.15.28:0/1734069934 >> v1:172.21.15.189:6789/0 conn(0x7f456015a540 legacy=0x7f456015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:21.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.180+0000 7f4557e00640 1 -- 172.21.15.28:0/1734069934 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456015e3e0 con 0x7f45600cabc0 2024-09-13T17:19:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.181+0000 7f4557e00640 1 --2- 172.21.15.28:0/1734069934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45600cabc0 0x7f456006c430 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f4560070900 tx=0x7f454802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cd12c95be7557f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f455fe00640 1 -- 172.21.15.28:0/1734069934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548032440 con 0x7f45600cabc0 2024-09-13T17:19:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f455fe00640 1 -- 172.21.15.28:0/1734069934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45480325a0 con 0x7f45600cabc0 2024-09-13T17:19:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f455fe00640 1 -- 172.21.15.28:0/1734069934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548032930 con 0x7f45600cabc0 2024-09-13T17:19:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 -- 172.21.15.28:0/1734069934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45600cabc0 msgr2=0x7f456006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 --2- 172.21.15.28:0/1734069934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45600cabc0 0x7f456006c430 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f4560070900 tx=0x7f454802a1c0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 -- 172.21.15.28:0/1734069934 shutdown_connections 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 --2- 172.21.15.28:0/1734069934 >> v2:172.21.15.189:3300/0 conn(0x7f4560070cc0 0x7f4560071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 --2- 172.21.15.28:0/1734069934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45600cabc0 0x7f456006c430 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 -- 172.21.15.28:0/1734069934 >> 172.21.15.28:0/1734069934 conn(0x7f4560069e80 msgr2=0x7f456006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 -- 172.21.15.28:0/1734069934 shutdown_connections 2024-09-13T17:19:21.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.182+0000 7f4567a00640 1 -- 172.21.15.28:0/1734069934 wait complete. 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.184+0000 7f4567a00640 1 Processor -- start 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.184+0000 7f4567a00640 1 -- start start 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4560070cc0 0x7f4560156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45600cabc0 0x7f456014bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560157990 con 0x7f456015a540 2024-09-13T17:19:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560156ad0 con 0x7f45600cabc0 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4567a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4560156d10 con 0x7f4560070cc0 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4565a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4565a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39242/0 (socket says 172.21.15.28:39242) 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.185+0000 7f4565a00640 1 -- 172.21.15.28:0/483557591 learned_addr learned my addr 172.21.15.28:0/483557591 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4557e00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45600cabc0 0x7f456014bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565000640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4560070cc0 0x7f4560156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 -- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4560070cc0 msgr2=0x7f4560156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4560070cc0 0x7f4560156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 -- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45600cabc0 msgr2=0x7f456014bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45600cabc0 0x7f456014bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 -- 172.21.15.28:0/483557591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480094d0 con 0x7f456015a540 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4565a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f4558007e40 tx=0x7f455800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558014620 con 0x7f456015a540 2024-09-13T17:19:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45580147a0 con 0x7f456015a540 2024-09-13T17:19:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558014b50 con 0x7f456015a540 2024-09-13T17:19:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.186+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560156ff0 con 0x7f456015a540 2024-09-13T17:19:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.187+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45601575b0 con 0x7f456015a540 2024-09-13T17:19:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.188+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560157ad0 con 0x7f456015a540 2024-09-13T17:19:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.189+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4558014cb0 con 0x7f456015a540 2024-09-13T17:19:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.189+0000 7f455d600640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 0x7f453c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.189+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4558089990 con 0x7f456015a540 2024-09-13T17:19:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.190+0000 7f4565000640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 0x7f453c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.191+0000 7f4565000640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 0x7f453c06d0f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.194+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4560157ad0 con 0x7f456015a540 2024-09-13T17:19:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:21.439+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]} v 0) v1 -- 0x7f4560157ad0 con 0x7f456015a540 2024-09-13T17:19:22.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.254+0000 7f455d600640 1 -- 172.21.15.28:0/483557591 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]}]=0 v567) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4560157ad0 con 0x7f456015a540 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.257+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 msgr2=0x7f453c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.257+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 0x7f453c06d0f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f4550006fd0 tx=0x7f4550008040 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 msgr2=0x7f45600af510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f4558007e40 tx=0x7f455800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 shutdown_connections 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f453c06ac30 0x7f453c06d0f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f456015a540 0x7f45600af510 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45600cabc0 0x7f456014bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 --2- 172.21.15.28:0/483557591 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4560070cc0 0x7f4560156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 >> 172.21.15.28:0/483557591 conn(0x7f4560069e80 msgr2=0x7f4560155c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:22.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 shutdown_connections 2024-09-13T17:19:22.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.258+0000 7f4567a00640 1 -- 172.21.15.28:0/483557591 wait complete. 2024-09-13T17:19:22.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:19:22.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.440+0000 7fcf24400640 1 Processor -- start 2024-09-13T17:19:22.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.440+0000 7fcf24400640 1 -- start start 2024-09-13T17:19:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.441+0000 7fcf24400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf24400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf1c06c250 0x7fcf1c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf24400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1c06c8b0 con 0x7fcf1c15ab60 2024-09-13T17:19:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf24400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf1c06c6a0 con 0x7fcf1c06c250 2024-09-13T17:19:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf24400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c06cb90 con 0x7fcf1c15a780 2024-09-13T17:19:22.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf21a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf21a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:58450/0 (socket says 172.21.15.28:58450) 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf22400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcf1c06c250 0x7fcf1c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.442+0000 7fcf21a00640 1 -- 172.21.15.28:0/3864338299 learned_addr learned my addr 172.21.15.28:0/3864338299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.443+0000 7fcf21a00640 1 -- 172.21.15.28:0/3864338299 >> v2:172.21.15.189:3300/0 conn(0x7fcf1c06c250 msgr2=0x7fcf1c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.443+0000 7fcf21a00640 1 --2- 172.21.15.28:0/3864338299 >> v2:172.21.15.189:3300/0 conn(0x7fcf1c06c250 0x7fcf1c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.443+0000 7fcf21a00640 1 -- 172.21.15.28:0/3864338299 >> v1:172.21.15.189:6789/0 conn(0x7fcf1c15ab60 legacy=0x7fcf1c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.443+0000 7fcf21a00640 1 -- 172.21.15.28:0/3864338299 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1c0cc980 con 0x7fcf1c15a780 2024-09-13T17:19:22.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.444+0000 7fcf21a00640 1 --2- 172.21.15.28:0/3864338299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009870 tx=0x7fcf1002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8896493762eae831 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:22.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.444+0000 7fcf1be00640 1 -- 172.21.15.28:0/3864338299 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10032440 con 0x7fcf1c15a780 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.444+0000 7fcf1be00640 1 -- 172.21.15.28:0/3864338299 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf100325a0 con 0x7fcf1c15a780 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.444+0000 7fcf1be00640 1 -- 172.21.15.28:0/3864338299 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10032930 con 0x7fcf1c15a780 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 -- 172.21.15.28:0/3864338299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 msgr2=0x7fcf1c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 --2- 172.21.15.28:0/3864338299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009870 tx=0x7fcf1002a250 comp rx=0 tx=0).stop 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 -- 172.21.15.28:0/3864338299 shutdown_connections 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 --2- 172.21.15.28:0/3864338299 >> v2:172.21.15.189:3300/0 conn(0x7fcf1c06c250 0x7fcf1c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 --2- 172.21.15.28:0/3864338299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c070d80 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 -- 172.21.15.28:0/3864338299 >> 172.21.15.28:0/3864338299 conn(0x7fcf1c0b0270 msgr2=0x7fcf1c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 -- 172.21.15.28:0/3864338299 shutdown_connections 2024-09-13T17:19:22.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.445+0000 7fcf24400640 1 -- 172.21.15.28:0/3864338299 wait complete. 2024-09-13T17:19:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.446+0000 7fcf24400640 1 Processor -- start 2024-09-13T17:19:22.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 -- start start 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c1e1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c15ab60 0x7fcf1c1e3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1e1d50 con 0x7fcf1c06c250 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1e2030 con 0x7fcf1c15ab60 2024-09-13T17:19:22.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.447+0000 7fcf24400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1c1e2310 con 0x7fcf1c15a780 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39258/0 (socket says 172.21.15.28:39258) 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 -- 172.21.15.28:0/3144088635 learned_addr learned my addr 172.21.15.28:0/3144088635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21000640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c1e1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf22400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c15ab60 0x7fcf1c1e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 -- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 msgr2=0x7fcf1c1e1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c1e1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 -- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c15ab60 msgr2=0x7fcf1c1e3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c15ab60 0x7fcf1c1e3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 -- 172.21.15.28:0/3144088635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100094d0 con 0x7fcf1c06c250 2024-09-13T17:19:22.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.448+0000 7fcf21a00640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.449+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10032440 con 0x7fcf1c06c250 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.449+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf10030980 con 0x7fcf1c06c250 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.449+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf10030d30 con 0x7fcf1c06c250 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.449+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1c1e25f0 con 0x7fcf1c06c250 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.449+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf1c1f5a60 con 0x7fcf1c06c250 2024-09-13T17:19:22.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.450+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1c06bb40 con 0x7fcf1c06c250 2024-09-13T17:19:22.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.451+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf10039040 con 0x7fcf1c06c250 2024-09-13T17:19:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.452+0000 7fcf19600640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.452+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcf100a9900 con 0x7fcf1c06c250 2024-09-13T17:19:22.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.453+0000 7fcf21000640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:22.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.453+0000 7fcf21000640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1e2c40 tx=0x7fcf08009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:22.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.456+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcf10074df0 con 0x7fcf1c06c250 2024-09-13T17:19:22.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.634+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcf1c0ae4c0 con 0x7fcf1c06c250 2024-09-13T17:19:22.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.638+0000 7fcf19600640 1 -- 172.21.15.28:0/3144088635 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcf10078410 con 0x7fcf1c06c250 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.640+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 msgr2=0x7fcefc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.640+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c1e2c40 tx=0x7fcf08009040 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.640+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 msgr2=0x7fcf1c1f5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.640+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcf10000c00 tx=0x7fcf100046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 shutdown_connections 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcefc06ac00 0x7fcefc06d0c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf1c15ab60 0x7fcf1c1e3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf1c15a780 0x7fcf1c1e1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 --2- 172.21.15.28:0/3144088635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf1c06c250 0x7fcf1c1f5520 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:22.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 >> 172.21.15.28:0/3144088635 conn(0x7fcf1c0b0270 msgr2=0x7fcf1c0ce7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 shutdown_connections 2024-09-13T17:19:22.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:22.641+0000 7fcf24400640 1 -- 172.21.15.28:0/3144088635 wait complete. 2024-09-13T17:19:22.659 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:19:22.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:19:22.744 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:19:22.840 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:19:22.840 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:22.840 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:22.876 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:19:22.876 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:19:22.876 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:22.913 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:19:22.914 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:22.914 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:22.951 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:19:22.951 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:19:22.951 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:22.989 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:19:22.990 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:19:22.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.144+0000 7f5e71000640 1 Processor -- start 2024-09-13T17:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.144+0000 7f5e71000640 1 -- start start 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e71000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5e6c159440 0x7f5e6c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e71000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c159820 0x7f5e6c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e71000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c0c3850 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e71000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c15a930 con 0x7f5e6c159820 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e71000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c159d20 con 0x7f5e6c159440 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e62a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c159820 0x7f5e6c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e62a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e6c159820 0x7f5e6c0b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39270/0 (socket says 172.21.15.28:39270) 2024-09-13T17:19:23.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e62a00640 1 -- 172.21.15.28:0/2821892505 learned_addr learned my addr 172.21.15.28:0/2821892505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.145+0000 7f5e6b400640 1 --2- 172.21.15.28:0/2821892505 >> v2:172.21.15.189:3300/0 conn(0x7f5e6c159440 0x7f5e6c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.146+0000 7f5e62a00640 1 -- 172.21.15.28:0/2821892505 >> v2:172.21.15.189:3300/0 conn(0x7f5e6c159440 msgr2=0x7f5e6c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.146+0000 7f5e62a00640 1 --2- 172.21.15.28:0/2821892505 >> v2:172.21.15.189:3300/0 conn(0x7f5e6c159440 0x7f5e6c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.146+0000 7f5e62a00640 1 -- 172.21.15.28:0/2821892505 >> v1:172.21.15.189:6789/0 conn(0x7f5e6c0b22e0 legacy=0x7f5e6c0b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.146+0000 7f5e62a00640 1 -- 172.21.15.28:0/2821892505 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c15a000 con 0x7f5e6c159820 2024-09-13T17:19:23.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e62a00640 1 --2- 172.21.15.28:0/2821892505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c159820 0x7f5e6c0b0980 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009870 tx=0x7f5e5402a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96faa8306887d5ee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e6aa00640 1 -- 172.21.15.28:0/2821892505 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54032640 con 0x7f5e6c159820 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e6aa00640 1 -- 172.21.15.28:0/2821892505 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e540327a0 con 0x7f5e6c159820 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e6aa00640 1 -- 172.21.15.28:0/2821892505 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54032b30 con 0x7f5e6c159820 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 -- 172.21.15.28:0/2821892505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c159820 msgr2=0x7f5e6c0b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 --2- 172.21.15.28:0/2821892505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c159820 0x7f5e6c0b0980 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f5e54009870 tx=0x7f5e5402a330 comp rx=0 tx=0).stop 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 -- 172.21.15.28:0/2821892505 shutdown_connections 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 --2- 172.21.15.28:0/2821892505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c159820 0x7f5e6c0b0980 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 --2- 172.21.15.28:0/2821892505 >> v2:172.21.15.189:3300/0 conn(0x7f5e6c159440 0x7f5e6c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.147+0000 7f5e71000640 1 -- 172.21.15.28:0/2821892505 >> 172.21.15.28:0/2821892505 conn(0x7f5e6c072e40 msgr2=0x7f5e6c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.148+0000 7f5e71000640 1 -- 172.21.15.28:0/2821892505 shutdown_connections 2024-09-13T17:19:23.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.148+0000 7f5e71000640 1 -- 172.21.15.28:0/2821892505 wait complete. 2024-09-13T17:19:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.149+0000 7f5e71000640 1 Processor -- start 2024-09-13T17:19:23.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.149+0000 7f5e71000640 1 -- start start 2024-09-13T17:19:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 0x7f5e6c0c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 0x7f5e6c0c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0ad730 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0abcb0 con 0x7f5e6c159820 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e71000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0abf90 con 0x7f5e6c159440 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39274/0 (socket says 172.21.15.28:39274) 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.150+0000 7f5e6b400640 1 -- 172.21.15.28:0/4046334208 learned_addr learned my addr 172.21.15.28:0/4046334208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e62a00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 0x7f5e6c0c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6be00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 0x7f5e6c0c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 -- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 msgr2=0x7f5e6c0c3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 0x7f5e6c0c3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 -- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 msgr2=0x7f5e6c0c4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 0x7f5e6c0c4510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 -- 172.21.15.28:0/4046334208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e540094d0 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e62a00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 0x7f5e6c0c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6be00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 0x7f5e6c0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e6b400640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00e930 tx=0x7f5e5c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c009690 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e5c00ca00 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.151+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c0ada60 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.152+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5c00cdb0 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.152+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c0ae020 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.153+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c0ae210 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.154+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e5c013470 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.154+0000 7f5e63e00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 0x7f5e4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.155+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5e5c089310 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.155+0000 7f5e62a00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 0x7f5e4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.155+0000 7f5e62a00640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 0x7f5e4806d0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5e54034000 tx=0x7f5e5402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.158+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5e5c0548b0 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.376+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e6c0b2750 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.377+0000 7f5e63e00640 1 -- 172.21.15.28:0/4046334208 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+3534 (secure 0 0 0) 0x7f5e5c017de0 con 0x7f5e6c0b22e0 2024-09-13T17:19:23.379 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:23.379 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":132,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/1586790115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":1586790115},{"type":"v1","addr":"172.21.15.189:6835","nonce":1586790115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":8788,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/888024955","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":888024955},{"type":"v1","addr":"172.21.15.189:6837","nonce":888024955}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":126},{"gid":9417,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/253017710","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":253017710},{"type":"v1","addr":"172.21.15.28:6834","nonce":253017710}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":125},{"gid":9954,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/2399709922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":2399709922},{"type":"v1","addr":"172.21.15.28:6835","nonce":2399709922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":132}],"filesystems":[]} 2024-09-13T17:19:23.380 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 132 2024-09-13T17:19:23.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 msgr2=0x7f5e4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 0x7f5e4806d0c0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5e54034000 tx=0x7f5e5402e000 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 msgr2=0x7f5e6c14af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00e930 tx=0x7f5e5c00edf0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 shutdown_connections 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e4806ac00 0x7f5e4806d0c0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e6c159820 0x7f5e6c0c4510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e6c159440 0x7f5e6c0c3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 --2- 172.21.15.28:0/4046334208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e6c0b22e0 0x7f5e6c14af10 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 >> 172.21.15.28:0/4046334208 conn(0x7f5e6c072e40 msgr2=0x7f5e6c15b540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.379+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 shutdown_connections 2024-09-13T17:19:23.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.380+0000 7f5e71000640 1 -- 172.21.15.28:0/4046334208 wait complete. 2024-09-13T17:19:23.399 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:19:23.399 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:19:23.399 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:19:23.399 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:19:23.400 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:19:23.400 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:19:23.400 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:19:23.400 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:19:23.400 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:19:23.400 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:19:23.400 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:19:23.401 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:19:23.401 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:19:23.401 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:19:23.401 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:19:23.401 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:19:23.414 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:19:23.414 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:19:23.415 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:19:23.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:19:23.416 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:19:23.416 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:19:23.417 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:19:23.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:19:23.460 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:19:23.460 DEBUG:teuthology.parallel:result is None 2024-09-13T17:19:23.461 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:19:23.461 DEBUG:teuthology.parallel:result is None 2024-09-13T17:19:23.462 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:19:23.462 DEBUG:teuthology.parallel:result is None 2024-09-13T17:19:23.464 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:19:23.464 DEBUG:teuthology.parallel:result is None 2024-09-13T17:19:23.465 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:19:23.588 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:19:23.588 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:19:23.590 INFO:tasks.ceph.mds.d.smithi189.stderr:2024-09-13T17:19:23.589+0000 7f26af600640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-09-13T17:19:23.591 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:19:23.592 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:19:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.631+0000 7fb406400640 1 Processor -- start 2024-09-13T17:19:23.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.631+0000 7fb406400640 1 -- start start 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb400159770 0x7fb40006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb40006f3f0 0x7fb400070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4000add00 0x7fb400159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40006f920 con 0x7fb4000add00 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40015b290 con 0x7fb40006f3f0 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.632+0000 7fb406400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40015b570 con 0x7fb400159770 2024-09-13T17:19:23.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4000add00 0x7fb400159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb400159770 0x7fb40006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4000add00 0x7fb400159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39308/0 (socket says 172.21.15.28:39308) 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 -- 172.21.15.28:0/565530213 learned_addr learned my addr 172.21.15.28:0/565530213 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3fea00640 1 --2- 172.21.15.28:0/565530213 >> v2:172.21.15.189:3300/0 conn(0x7fb40006f3f0 0x7fb400070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 -- 172.21.15.28:0/565530213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb400159770 msgr2=0x7fb40006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 --2- 172.21.15.28:0/565530213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb400159770 0x7fb40006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 -- 172.21.15.28:0/565530213 >> v2:172.21.15.189:3300/0 conn(0x7fb40006f3f0 msgr2=0x7fb400070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 --2- 172.21.15.28:0/565530213 >> v2:172.21.15.189:3300/0 conn(0x7fb40006f3f0 0x7fb400070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.633+0000 7fb3ffe00640 1 -- 172.21.15.28:0/565530213 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40015b850 con 0x7fb4000add00 2024-09-13T17:19:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.634+0000 7fb3ffe00640 1 --2- 172.21.15.28:0/565530213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb400159b50 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fb4000cf5c0 tx=0x7fb3f802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a97ec6c0dfe653 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.634+0000 7fb3fe000640 1 -- 172.21.15.28:0/565530213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8031600 con 0x7fb4000add00 2024-09-13T17:19:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.634+0000 7fb3fe000640 1 -- 172.21.15.28:0/565530213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f802d050 con 0x7fb4000add00 2024-09-13T17:19:23.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.634+0000 7fb3fe000640 1 -- 172.21.15.28:0/565530213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8031980 con 0x7fb4000add00 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 -- 172.21.15.28:0/565530213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 msgr2=0x7fb400159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 --2- 172.21.15.28:0/565530213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb400159b50 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7fb4000cf5c0 tx=0x7fb3f802a610 comp rx=0 tx=0).stop 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 -- 172.21.15.28:0/565530213 shutdown_connections 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 --2- 172.21.15.28:0/565530213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb400159b50 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 --2- 172.21.15.28:0/565530213 >> v2:172.21.15.189:3300/0 conn(0x7fb40006f3f0 0x7fb400070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 --2- 172.21.15.28:0/565530213 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb400159770 0x7fb40006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 -- 172.21.15.28:0/565530213 >> 172.21.15.28:0/565530213 conn(0x7fb4000b0270 msgr2=0x7fb4000b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 -- 172.21.15.28:0/565530213 shutdown_connections 2024-09-13T17:19:23.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.635+0000 7fb406400640 1 -- 172.21.15.28:0/565530213 wait complete. 2024-09-13T17:19:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.637+0000 7fb406400640 1 Processor -- start 2024-09-13T17:19:23.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.637+0000 7fb406400640 1 -- start start 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb40006f3f0 0x7fb4001560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb400159770 0x7fb4001e1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000ac240 con 0x7fb4000add00 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000ac520 con 0x7fb400159770 2024-09-13T17:19:23.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb406400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4000ac800 con 0x7fb40006f3f0 2024-09-13T17:19:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb40006f3f0 0x7fb4001560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb3fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39314/0 (socket says 172.21.15.28:39314) 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb3fea00640 1 -- 172.21.15.28:0/2253323344 learned_addr learned my addr 172.21.15.28:0/2253323344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.638+0000 7fb3ffe00640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb400159770 0x7fb4001e1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 -- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb40006f3f0 msgr2=0x7fb4001560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb40006f3f0 0x7fb4001560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 -- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb400159770 msgr2=0x7fb4001e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb400159770 0x7fb4001e1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 -- 172.21.15.28:0/2253323344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f8009b30 con 0x7fb4000add00 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3fea00640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e980 tx=0x7fb3ec00ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec009670 con 0x7fb4000add00 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3ec00ca00 con 0x7fb4000add00 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.639+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3ec00cd90 con 0x7fb4000add00 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.640+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4000acae0 con 0x7fb4000add00 2024-09-13T17:19:23.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.640+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb400156610 con 0x7fb4000add00 2024-09-13T17:19:23.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.641+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000b38e0 con 0x7fb4000add00 2024-09-13T17:19:23.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.642+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb3ec013470 con 0x7fb4000add00 2024-09-13T17:19:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.643+0000 7fb3f7400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 0x7fb3e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.643+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb3ec0899f0 con 0x7fb4000add00 2024-09-13T17:19:23.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.643+0000 7fb3ff400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 0x7fb3e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:23.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.644+0000 7fb3ff400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 0x7fb3e006d0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000df40 tx=0x7fb3f000f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:23.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.646+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb3ec054f60 con 0x7fb4000add00 2024-09-13T17:19:23.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:23.837+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb400159b50 con 0x7fb4000add00 2024-09-13T17:19:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.642+0000 7fb3f7400640 1 -- 172.21.15.28:0/2253323344 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v163) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb3ec017080 con 0x7fb4000add00 2024-09-13T17:19:24.644 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.644+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 msgr2=0x7fb3e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.644+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 0x7fb3e006d0c0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fb3f000df40 tx=0x7fb3f000f040 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.644+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 msgr2=0x7fb4000ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.644+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec00e980 tx=0x7fb3ec00ee40 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 shutdown_connections 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb3e006ac00 0x7fb3e006d0c0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb400159770 0x7fb4001e1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4000add00 0x7fb4000ad2e0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 --2- 172.21.15.28:0/2253323344 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb40006f3f0 0x7fb4001560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 >> 172.21.15.28:0/2253323344 conn(0x7fb4000b0270 msgr2=0x7fb4000b1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:24.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 shutdown_connections 2024-09-13T17:19:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.645+0000 7fb406400640 1 -- 172.21.15.28:0/2253323344 wait complete. 2024-09-13T17:19:24.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:19:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.825+0000 7fd803200640 1 Processor -- start 2024-09-13T17:19:24.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.825+0000 7fd803200640 1 -- start start 2024-09-13T17:19:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd803200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8040b0b50 0x7fd8040b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd803200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd804159820 0x7fd8040b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd803200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8040c3850 con 0x7fd804159440 2024-09-13T17:19:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd803200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8040b2d60 con 0x7fd804159820 2024-09-13T17:19:24.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd803200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80415a930 con 0x7fd8040b0b50 2024-09-13T17:19:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.826+0000 7fd7fbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8040b0b50 0x7fd8040b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd802800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd804159820 0x7fd8040b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd801e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fd804159440 0x7fd8040b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:46126/0 (socket says 172.21.15.28:46126) 2024-09-13T17:19:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd801e00640 1 -- 172.21.15.28:0/2559538161 learned_addr learned my addr 172.21.15.28:0/2559538161 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd802800640 1 -- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8040b0b50 msgr2=0x7fd8040b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd802800640 1 --2- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8040b0b50 0x7fd8040b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd802800640 1 -- 172.21.15.28:0/2559538161 >> v1:172.21.15.189:6789/0 conn(0x7fd804159440 legacy=0x7fd8040b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.827+0000 7fd802800640 1 -- 172.21.15.28:0/2559538161 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804159d20 con 0x7fd804159820 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.828+0000 7fd802800640 1 --2- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd804159820 0x7fd8040b2820 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc02c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94cafaade0f5e477 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd801400640 1 -- 172.21.15.28:0/2559538161 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc032800 con 0x7fd804159820 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd801400640 1 -- 172.21.15.28:0/2559538161 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc032960 con 0x7fd804159820 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd801400640 1 -- 172.21.15.28:0/2559538161 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc032cf0 con 0x7fd804159820 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 -- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd804159820 msgr2=0x7fd8040b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 --2- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd804159820 0x7fd8040b2820 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc02c900 comp rx=0 tx=0).stop 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 -- 172.21.15.28:0/2559538161 shutdown_connections 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 --2- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd804159820 0x7fd8040b2820 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 --2- 172.21.15.28:0/2559538161 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd8040b0b50 0x7fd8040b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 -- 172.21.15.28:0/2559538161 >> 172.21.15.28:0/2559538161 conn(0x7fd804072e40 msgr2=0x7fd804073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 -- 172.21.15.28:0/2559538161 shutdown_connections 2024-09-13T17:19:24.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.829+0000 7fd803200640 1 -- 172.21.15.28:0/2559538161 wait complete. 2024-09-13T17:19:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.831+0000 7fd803200640 1 Processor -- start 2024-09-13T17:19:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.831+0000 7fd803200640 1 -- start start 2024-09-13T17:19:24.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 0x7fd804143f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd804159440 0x7fd804155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8041504f0 con 0x7fd804159820 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80414fbe0 con 0x7fd8040b0b50 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd803200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd804158b00 con 0x7fd804159440 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd802800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd802800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44578/0 (socket says 172.21.15.28:44578) 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd802800640 1 -- 172.21.15.28:0/503398779 learned_addr learned my addr 172.21.15.28:0/503398779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.832+0000 7fd7fbe00640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd804159440 0x7fd804155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd801e00640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 0x7fd804143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 -- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd804159440 msgr2=0x7fd804155df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd804159440 0x7fd804155df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 -- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 msgr2=0x7fd804143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:24.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 0x7fd804143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 -- 172.21.15.28:0/503398779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc009a70 con 0x7fd804159820 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd801e00640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 0x7fd804143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd802800640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc03a840 con 0x7fd804159820 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.833+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80414fe30 con 0x7fd804159820 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.834+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc03f070 con 0x7fd804159820 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.834+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc03abe0 con 0x7fd804159820 2024-09-13T17:19:24.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.834+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8041444a0 con 0x7fd804159820 2024-09-13T17:19:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.835+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd804150630 con 0x7fd804159820 2024-09-13T17:19:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.836+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd7fc03ad40 con 0x7fd804159820 2024-09-13T17:19:24.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.837+0000 7fd7fa000640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 0x7fd7e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.837+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7fc02f070 con 0x7fd804159820 2024-09-13T17:19:24.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.837+0000 7fd801e00640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 0x7fd7e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:24.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.838+0000 7fd801e00640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 0x7fd7e006d0c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4000f60 tx=0x7fd7f4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:24.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:24.841+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd804150630 con 0x7fd804159820 2024-09-13T17:19:25.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.089+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd804150630 con 0x7fd804159820 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.090+0000 7fd7fa000640 1 -- 172.21.15.28:0/503398779 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v22) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd804150630 con 0x7fd804159820 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:19:25.092 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.094 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:19:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:19:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:19:25.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:19:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.096 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:19:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:19:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:19:25.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:19:25.098 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:19:25.101 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.103 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:19:25.104 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:19:25.107 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:19:25.109 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:19:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.110 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:19:25.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:19:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:19:25.112 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:19:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:19:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:19:25.113 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:19:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:19:25.114 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:19:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:19:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:19:25.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:19:25.116 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:19:25.117 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:19:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:19:25.118 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:19:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:19:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:19:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.119 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:19:25.120 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:19:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:19:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:25.121 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:19:25.122 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:25.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 msgr2=0x7fd7e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 0x7fd7e006d0c0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4000f60 tx=0x7fd7f4009040 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 msgr2=0x7fd80415c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009d90 tx=0x7fd7fc004ab0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 shutdown_connections 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd7e006ac00 0x7fd7e006d0c0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd804159820 0x7fd80415c270 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd804159440 0x7fd804155df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 --2- 172.21.15.28:0/503398779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8040b0b50 0x7fd804143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 >> 172.21.15.28:0/503398779 conn(0x7fd804072e40 msgr2=0x7fd80406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 shutdown_connections 2024-09-13T17:19:25.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.092+0000 7fd803200640 1 -- 172.21.15.28:0/503398779 wait complete. 2024-09-13T17:19:25.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.280+0000 7fed87c00640 1 Processor -- start 2024-09-13T17:19:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.280+0000 7fed87c00640 1 -- start start 2024-09-13T17:19:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.281+0000 7fed87c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fed8006f970 0x7fed8015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.281+0000 7fed87c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed80159d90 con 0x7fed8006c2d0 2024-09-13T17:19:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.281+0000 7fed87c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800b2040 con 0x7fed801592a0 2024-09-13T17:19:25.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.281+0000 7fed87c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8015b0b0 con 0x7fed8006f970 2024-09-13T17:19:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed85c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fed8006c2d0 0x7fed800b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43000/0 (socket says 172.21.15.28:43000) 2024-09-13T17:19:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed85c00640 1 -- 172.21.15.28:0/1474436633 learned_addr learned my addr 172.21.15.28:0/1474436633 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7fe00640 1 --2- 172.21.15.28:0/1474436633 >> v2:172.21.15.189:3300/0 conn(0x7fed8006f970 0x7fed8015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7f400640 1 -- 172.21.15.28:0/1474436633 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077153950 0 0) 0x7fed80159d90 con 0x7fed8006c2d0 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7f400640 1 -- 172.21.15.28:0/1474436633 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed68003520 con 0x7fed8006c2d0 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7fe00640 1 -- 172.21.15.28:0/1474436633 >> v1:172.21.15.189:6789/0 conn(0x7fed801592a0 legacy=0x7fed80159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7fe00640 1 -- 172.21.15.28:0/1474436633 >> v1:172.21.15.28:6789/0 conn(0x7fed8006c2d0 legacy=0x7fed800b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.282+0000 7fed7fe00640 1 -- 172.21.15.28:0/1474436633 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed80159ff0 con 0x7fed8006f970 2024-09-13T17:19:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.283+0000 7fed7fe00640 1 --2- 172.21.15.28:0/1474436633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed8006f970 0x7fed8015ab70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0097a0 tx=0x7fed6c02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=226dcdb82406dcf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed7f400640 1 -- 172.21.15.28:0/1474436633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c032440 con 0x7fed8006f970 2024-09-13T17:19:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed7f400640 1 -- 172.21.15.28:0/1474436633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c0325a0 con 0x7fed8006f970 2024-09-13T17:19:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed7f400640 1 -- 172.21.15.28:0/1474436633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c032930 con 0x7fed8006f970 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 -- 172.21.15.28:0/1474436633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed8006f970 msgr2=0x7fed8015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 --2- 172.21.15.28:0/1474436633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed8006f970 0x7fed8015ab70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fed6c0097a0 tx=0x7fed6c02a230 comp rx=0 tx=0).stop 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 -- 172.21.15.28:0/1474436633 shutdown_connections 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 --2- 172.21.15.28:0/1474436633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed8006f970 0x7fed8015ab70 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 -- 172.21.15.28:0/1474436633 >> 172.21.15.28:0/1474436633 conn(0x7fed80069e80 msgr2=0x7fed8006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 -- 172.21.15.28:0/1474436633 shutdown_connections 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.284+0000 7fed87c00640 1 -- 172.21.15.28:0/1474436633 wait complete. 2024-09-13T17:19:25.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.286+0000 7fed87c00640 1 Processor -- start 2024-09-13T17:19:25.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.286+0000 7fed87c00640 1 -- start start 2024-09-13T17:19:25.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 0x7fed8014c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 0x7fed800cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed800cd0f0 con 0x7fed8006f970 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed800cc050 con 0x7fed801592a0 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed87c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed800cc330 con 0x7fed8006c2d0 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44594/0 (socket says 172.21.15.28:44594) 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.287+0000 7fed7fe00640 1 -- 172.21.15.28:0/1086856507 learned_addr learned my addr 172.21.15.28:0/1086856507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed85c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 0x7fed800cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 -- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 msgr2=0x7fed8014c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed85200640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 0x7fed8014c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 0x7fed8014c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 -- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 msgr2=0x7fed800cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 0x7fed800cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 -- 172.21.15.28:0/1086856507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c009400 con 0x7fed8006f970 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed85c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 0x7fed800cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed85200640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 0x7fed8014c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7fe00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fed6c035040 tx=0x7fed6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c006e10 con 0x7fed8006f970 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c032e90 con 0x7fed8006f970 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c030be0 con 0x7fed8006f970 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.288+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed800cc610 con 0x7fed8006f970 2024-09-13T17:19:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.289+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed800ccbd0 con 0x7fed8006f970 2024-09-13T17:19:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.290+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8015ab70 con 0x7fed8006f970 2024-09-13T17:19:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.290+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fed6c039040 con 0x7fed8006f970 2024-09-13T17:19:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.291+0000 7fed7cc00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 0x7fed6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.291+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fed6c0a99d0 con 0x7fed8006f970 2024-09-13T17:19:25.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.292+0000 7fed85200640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 0x7fed6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.292+0000 7fed85200640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 0x7fed6006d0c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fed800ad3e0 tx=0x7fed74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.295+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fed6c074f80 con 0x7fed8006f970 2024-09-13T17:19:25.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.477+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed800b1a20 con 0x7fed8006f970 2024-09-13T17:19:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.477+0000 7fed7cc00640 1 -- 172.21.15.28:0/1086856507 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fed6c030d40 con 0x7fed8006f970 2024-09-13T17:19:25.480 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:25.480 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":163,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:19:24.639052+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 msgr2=0x7fed6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 0x7fed6006d0c0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fed800ad3e0 tx=0x7fed74009040 comp rx=0 tx=0).stop 2024-09-13T17:19:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 msgr2=0x7fed800cb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fed6c035040 tx=0x7fed6c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 shutdown_connections 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fed6006ac00 0x7fed6006d0c0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed801592a0 0x7fed800cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed8006f970 0x7fed800cb4e0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 --2- 172.21.15.28:0/1086856507 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed8006c2d0 0x7fed8014c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 >> 172.21.15.28:0/1086856507 conn(0x7fed80069e80 msgr2=0x7fed8006fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 shutdown_connections 2024-09-13T17:19:25.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.480+0000 7fed87c00640 1 -- 172.21.15.28:0/1086856507 wait complete. 2024-09-13T17:19:25.498 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:19:25.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:19:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.655+0000 7f52aee00640 1 Processor -- start 2024-09-13T17:19:25.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.655+0000 7f52aee00640 1 -- start start 2024-09-13T17:19:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 0x7f52a8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 0x7f52a80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52a806bb40 0x7f52a806bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a806c8b0 con 0x7f52a806bb40 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a806cb90 con 0x7f52a815ab60 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.656+0000 7f52aee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80cc980 con 0x7f52a815a780 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52a806bb40 0x7f52a806bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f52a806bb40 0x7f52a806bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44608/0 (socket says 172.21.15.28:44608) 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 -- 172.21.15.28:0/3391008841 learned_addr learned my addr 172.21.15.28:0/3391008841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52a7e00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 0x7f52a8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f529f400640 1 --2- 172.21.15.28:0/3391008841 >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 0x7f52a80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 -- 172.21.15.28:0/3391008841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 msgr2=0x7f52a8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 0x7f52a8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 -- 172.21.15.28:0/3391008841 >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 msgr2=0x7f52a80712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 --2- 172.21.15.28:0/3391008841 >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 0x7f52a80712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52ace00640 1 -- 172.21.15.28:0/3391008841 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80ccc60 con 0x7f52a806bb40 2024-09-13T17:19:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f52a7e00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 0x7f52a8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:25.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.657+0000 7f529f400640 1 --2- 172.21.15.28:0/3391008841 >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 0x7f52a80712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.658+0000 7f52ace00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a806bb40 0x7f52a806bf90 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f52a80cf5c0 tx=0x7f52a002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb389a641591d6b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.658+0000 7f52a7400640 1 -- 172.21.15.28:0/3391008841 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0031600 con 0x7f52a806bb40 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.658+0000 7f52a7400640 1 -- 172.21.15.28:0/3391008841 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a002d050 con 0x7f52a806bb40 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.658+0000 7f52a7400640 1 -- 172.21.15.28:0/3391008841 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a0031980 con 0x7f52a806bb40 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 -- 172.21.15.28:0/3391008841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a806bb40 msgr2=0x7f52a806bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a806bb40 0x7f52a806bf90 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f52a80cf5c0 tx=0x7f52a002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 -- 172.21.15.28:0/3391008841 shutdown_connections 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a806bb40 0x7f52a806bf90 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 --2- 172.21.15.28:0/3391008841 >> v2:172.21.15.189:3300/0 conn(0x7f52a815ab60 0x7f52a80712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 --2- 172.21.15.28:0/3391008841 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815a780 0x7f52a8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 -- 172.21.15.28:0/3391008841 >> 172.21.15.28:0/3391008841 conn(0x7f52a80b0270 msgr2=0x7f52a80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:25.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 -- 172.21.15.28:0/3391008841 shutdown_connections 2024-09-13T17:19:25.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.659+0000 7f52aee00640 1 -- 172.21.15.28:0/3391008841 wait complete. 2024-09-13T17:19:25.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.661+0000 7f52aee00640 1 Processor -- start 2024-09-13T17:19:25.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.661+0000 7f52aee00640 1 -- start start 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 0x7f52a8150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815ab60 0x7f52a80af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8152990 con 0x7f52a815a780 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8151d10 con 0x7f52a806bb40 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52aee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8151ff0 con 0x7f52a815ab60 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44624/0 (socket says 172.21.15.28:44624) 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 -- 172.21.15.28:0/2869852635 learned_addr learned my addr 172.21.15.28:0/2869852635 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:25.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52ace00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815ab60 0x7f52a80af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f52a7e00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 0x7f52a8150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 -- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815ab60 msgr2=0x7f52a80af2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815ab60 0x7f52a80af2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.662+0000 7f529f400640 1 -- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 msgr2=0x7f52a8150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f529f400640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 0x7f52a8150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f529f400640 1 -- 172.21.15.28:0/2869852635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0009b30 con 0x7f52a815a780 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52a7e00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 0x7f52a8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f529f400640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f529000d800 tx=0x7f529000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5290004030 con 0x7f52a815a780 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5290004190 con 0x7f52a815a780 2024-09-13T17:19:25.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529000b6c0 con 0x7f52a815a780 2024-09-13T17:19:25.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a81522d0 con 0x7f52a815a780 2024-09-13T17:19:25.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.663+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a8151430 con 0x7f52a815a780 2024-09-13T17:19:25.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.664+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a806bf90 con 0x7f52a815a780 2024-09-13T17:19:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.665+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f529000b820 con 0x7f52a815a780 2024-09-13T17:19:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.667+0000 7f52a4c00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 0x7f528406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.667+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f52900890e0 con 0x7f52a815a780 2024-09-13T17:19:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.667+0000 7f52a7e00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 0x7f528406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.668+0000 7f52a7e00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 0x7f528406d0c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f52a806c4a0 tx=0x7f5298008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:25.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.671+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5290054600 con 0x7f52a815a780 2024-09-13T17:19:25.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.848+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52a80afc60 con 0x7f52a815a780 2024-09-13T17:19:25.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.849+0000 7f52a4c00640 1 -- 172.21.15.28:0/2869852635 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v163) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5290057c20 con 0x7f52a815a780 2024-09-13T17:19:25.851 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 msgr2=0x7f528406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 0x7f528406d0c0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f52a806c4a0 tx=0x7f5298008040 comp rx=0 tx=0).stop 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 msgr2=0x7f52a8152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f529000d800 tx=0x7f529000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 shutdown_connections 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f528406ac00 0x7f528406d0c0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f52a815ab60 0x7f52a80af2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f52a815a780 0x7f52a8152db0 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 --2- 172.21.15.28:0/2869852635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f52a806bb40 0x7f52a8150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 >> 172.21.15.28:0/2869852635 conn(0x7f52a80b0270 msgr2=0x7f52a80ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 shutdown_connections 2024-09-13T17:19:25.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:25.851+0000 7f52aee00640 1 -- 172.21.15.28:0/2869852635 wait complete. 2024-09-13T17:19:25.868 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:19:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.022+0000 7f4e9a600640 1 Processor -- start 2024-09-13T17:19:26.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.022+0000 7f4e9a600640 1 -- start start 2024-09-13T17:19:26.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.023+0000 7f4e9a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 0x7f4e9406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.023+0000 7f4e9a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9406f920 con 0x7f4e9406f3f0 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e9a600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9415b290 con 0x7f4e9415cd80 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e9a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9415b570 con 0x7f4e94159770 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e8be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4e9406f3f0 0x7f4e94159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43012/0 (socket says 172.21.15.28:43012) 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e8be00640 1 -- 172.21.15.28:0/2477783063 learned_addr learned my addr 172.21.15.28:0/2477783063 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:26.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e93400640 1 --2- 172.21.15.28:0/2477783063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 0x7f4e9406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709060014 0 0) 0x7f4e9406f920 con 0x7f4e9406f3f0 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e78003520 con 0x7f4e9406f3f0 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781105187 0 0) 0x7f4e9415b290 con 0x7f4e9415cd80 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9406f920 con 0x7f4e9415cd80 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.024+0000 7f4e93400640 1 -- 172.21.15.28:0/2477783063 >> v1:172.21.15.189:6789/0 conn(0x7f4e9415cd80 legacy=0x7f4e9415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.025+0000 7f4e93400640 1 -- 172.21.15.28:0/2477783063 >> v1:172.21.15.28:6789/0 conn(0x7f4e9406f3f0 legacy=0x7f4e94159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.025+0000 7f4e93400640 1 -- 172.21.15.28:0/2477783063 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9415b850 con 0x7f4e94159770 2024-09-13T17:19:26.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.026+0000 7f4e93400640 1 --2- 172.21.15.28:0/2477783063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 0x7f4e9406fd20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097d0 tx=0x7f4e8402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2483b6a1748817f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.026+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e8402ae40 con 0x7f4e94159770 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.026+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e84032440 con 0x7f4e94159770 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.026+0000 7f4e92a00640 1 -- 172.21.15.28:0/2477783063 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e840327d0 con 0x7f4e94159770 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 -- 172.21.15.28:0/2477783063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 msgr2=0x7f4e9406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2477783063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 0x7f4e9406fd20 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097d0 tx=0x7f4e8402a130 comp rx=0 tx=0).stop 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 -- 172.21.15.28:0/2477783063 shutdown_connections 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2477783063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e94159770 0x7f4e9406fd20 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 -- 172.21.15.28:0/2477783063 >> 172.21.15.28:0/2477783063 conn(0x7f4e940b0270 msgr2=0x7f4e940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 -- 172.21.15.28:0/2477783063 shutdown_connections 2024-09-13T17:19:26.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.027+0000 7f4e9a600640 1 -- 172.21.15.28:0/2477783063 wait complete. 2024-09-13T17:19:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.028+0000 7f4e9a600640 1 Processor -- start 2024-09-13T17:19:26.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.028+0000 7f4e9a600640 1 -- start start 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f3f0 0x7f4e9414ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9415cd80 0x7f4e94145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e941456b0 con 0x7f4e94159770 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940cc130 con 0x7f4e9415cd80 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.029+0000 7f4e9a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e940cc3d0 con 0x7f4e9406f3f0 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44636/0 (socket says 172.21.15.28:44636) 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 -- 172.21.15.28:0/2965020688 learned_addr learned my addr 172.21.15.28:0/2965020688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:26.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e93e00640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9415cd80 0x7f4e94145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e93400640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f3f0 0x7f4e9414ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 -- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f3f0 msgr2=0x7f4e9414ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f3f0 0x7f4e9414ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 -- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9415cd80 msgr2=0x7f4e94145170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9415cd80 0x7f4e94145170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.030+0000 7f4e8be00640 1 -- 172.21.15.28:0/2965020688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84009400 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e8be00640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00c4d0 tx=0x7f4e7c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c005ce0 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7c005e40 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e940cc6b0 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7c007270 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.031+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e94158650 con 0x7f4e94159770 2024-09-13T17:19:26.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.032+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e940ccbd0 con 0x7f4e94159770 2024-09-13T17:19:26.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.033+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e7c007490 con 0x7f4e94159770 2024-09-13T17:19:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.034+0000 7f4e8b400640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 0x7f4e7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.034+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e7c0884b0 con 0x7f4e94159770 2024-09-13T17:19:26.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.035+0000 7f4e93400640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 0x7f4e7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.035+0000 7f4e93400640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 0x7f4e7006d0c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f4e84034000 tx=0x7f4e8402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.038+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4e940ccbd0 con 0x7f4e94159770 2024-09-13T17:19:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.217+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4e940ccbd0 con 0x7f4e94159770 2024-09-13T17:19:26.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.218+0000 7f4e8b400640 1 -- 172.21.15.28:0/2965020688 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v163) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4e940ccbd0 con 0x7f4e94159770 2024-09-13T17:19:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:26.220 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:19:26.220 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:19:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:19:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:19:26.221 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:26.221 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 msgr2=0x7f4e7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 0x7f4e7006d0c0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f4e84034000 tx=0x7f4e8402e000 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 msgr2=0x7f4e94158910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c00c4d0 tx=0x7f4e7c00c990 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 shutdown_connections 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e7006ac00 0x7f4e7006d0c0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e9415cd80 0x7f4e94145170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e94159770 0x7f4e94158910 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 --2- 172.21.15.28:0/2965020688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e9406f3f0 0x7f4e9414ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.220+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 >> 172.21.15.28:0/2965020688 conn(0x7f4e940b0270 msgr2=0x7f4e940adf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:26.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.221+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 shutdown_connections 2024-09-13T17:19:26.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.221+0000 7f4e9a600640 1 -- 172.21.15.28:0/2965020688 wait complete. 2024-09-13T17:19:26.239 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:19:26.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:19:26.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.397+0000 7fed05c00640 1 Processor -- start 2024-09-13T17:19:26.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.398+0000 7fed05c00640 1 -- start start 2024-09-13T17:19:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed0006b9c0 0x7fed000b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed000b3600 con 0x7fed000b2dd0 2024-09-13T17:19:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed000b38e0 con 0x7fed00159440 2024-09-13T17:19:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed000b46a0 con 0x7fed0006b9c0 2024-09-13T17:19:26.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fecf7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fed0006b9c0 0x7fed000b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fed000b2dd0 0x7fed00159820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43026/0 (socket says 172.21.15.28:43026) 2024-09-13T17:19:26.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.399+0000 7fed05200640 1 -- 172.21.15.28:0/773599959 learned_addr learned my addr 172.21.15.28:0/773599959 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.400+0000 7fecf7e00640 1 -- 172.21.15.28:0/773599959 >> v1:172.21.15.189:6789/0 conn(0x7fed00159440 legacy=0x7fed0006b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.400+0000 7fecf7e00640 1 -- 172.21.15.28:0/773599959 >> v1:172.21.15.28:6789/0 conn(0x7fed000b2dd0 legacy=0x7fed00159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.400+0000 7fecf7e00640 1 -- 172.21.15.28:0/773599959 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000b3bc0 con 0x7fed0006b9c0 2024-09-13T17:19:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.400+0000 7fecff400640 1 -- 172.21.15.28:0/773599959 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805917207 0 0) 0x7fed000b3600 con 0x7fed000b2dd0 2024-09-13T17:19:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.401+0000 7fecf7e00640 1 --2- 172.21.15.28:0/773599959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed0006b9c0 0x7fed000b2890 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fece80097a0 tx=0x7fece802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c216c53e4187157f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.401+0000 7fecff400640 1 -- 172.21.15.28:0/773599959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8032440 con 0x7fed0006b9c0 2024-09-13T17:19:26.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.401+0000 7fecff400640 1 -- 172.21.15.28:0/773599959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fece80325a0 con 0x7fed0006b9c0 2024-09-13T17:19:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.401+0000 7fecff400640 1 -- 172.21.15.28:0/773599959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fece8032930 con 0x7fed0006b9c0 2024-09-13T17:19:26.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.401+0000 7fed05c00640 1 -- 172.21.15.28:0/773599959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed0006b9c0 msgr2=0x7fed000b2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 --2- 172.21.15.28:0/773599959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed0006b9c0 0x7fed000b2890 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7fece80097a0 tx=0x7fece802a230 comp rx=0 tx=0).stop 2024-09-13T17:19:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 -- 172.21.15.28:0/773599959 shutdown_connections 2024-09-13T17:19:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 --2- 172.21.15.28:0/773599959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed0006b9c0 0x7fed000b2890 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 -- 172.21.15.28:0/773599959 >> 172.21.15.28:0/773599959 conn(0x7fed00069e80 msgr2=0x7fed0006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:26.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 -- 172.21.15.28:0/773599959 shutdown_connections 2024-09-13T17:19:26.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.402+0000 7fed05c00640 1 -- 172.21.15.28:0/773599959 wait complete. 2024-09-13T17:19:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.403+0000 7fed05c00640 1 Processor -- start 2024-09-13T17:19:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.403+0000 7fed05c00640 1 -- start start 2024-09-13T17:19:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed0006b9c0 0x7fed0014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed000b2dd0 0x7fed0014fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0006f370 con 0x7fed00159440 2024-09-13T17:19:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0006f610 con 0x7fed0006b9c0 2024-09-13T17:19:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.404+0000 7fed05c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0006f8f0 con 0x7fed000b2dd0 2024-09-13T17:19:26.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44648/0 (socket says 172.21.15.28:44648) 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 -- 172.21.15.28:0/3633080601 learned_addr learned my addr 172.21.15.28:0/3633080601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fecf7e00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed000b2dd0 0x7fed0014fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fecffe00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed0006b9c0 0x7fed0014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 -- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed000b2dd0 msgr2=0x7fed0014fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed000b2dd0 0x7fed0014fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 -- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed0006b9c0 msgr2=0x7fed0014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed0006b9c0 0x7fed0014ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.405+0000 7fed05200640 1 -- 172.21.15.28:0/3633080601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8009400 con 0x7fed00159440 2024-09-13T17:19:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fed05200640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c7a0 tx=0x7fecf800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800a640 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf800a7a0 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800abd0 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0006fbd0 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.406+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed00070190 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.407+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed00067a10 con 0x7fed00159440 2024-09-13T17:19:26.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.408+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fecf800ad30 con 0x7fed00159440 2024-09-13T17:19:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.409+0000 7fecfcc00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 0x7fecdc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.409+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fecf80884f0 con 0x7fed00159440 2024-09-13T17:19:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.410+0000 7fecffe00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 0x7fecdc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:26.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.410+0000 7fecffe00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 0x7fecdc06d0c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000f60 tx=0x7fecf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:26.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.414+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fecf8053ac0 con 0x7fed00159440 2024-09-13T17:19:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:26.594+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fed000b18b0 con 0x7fed00159440 2024-09-13T17:19:27.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.270+0000 7fecfcc00640 1 -- 172.21.15.28:0/3633080601 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v164) v1 ==== 138+0+0 (secure 0 0 0) 0x7fecf8018020 con 0x7fed00159440 2024-09-13T17:19:27.273 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:19:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.275+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 msgr2=0x7fecdc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.275+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 0x7fecdc06d0c0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fecf0000f60 tx=0x7fecf0009040 comp rx=0 tx=0).stop 2024-09-13T17:19:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.275+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 msgr2=0x7fed00144dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.275+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c7a0 tx=0x7fecf800cc60 comp rx=0 tx=0).stop 2024-09-13T17:19:27.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 shutdown_connections 2024-09-13T17:19:27.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecdc06ac00 0x7fecdc06d0c0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed00159440 0x7fed00144dd0 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fed000b2dd0 0x7fed0014fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 --2- 172.21.15.28:0/3633080601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fed0006b9c0 0x7fed0014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.276+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 >> 172.21.15.28:0/3633080601 conn(0x7fed00069e80 msgr2=0x7fed0006b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.277+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 shutdown_connections 2024-09-13T17:19:27.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.277+0000 7fed05c00640 1 -- 172.21.15.28:0/3633080601 wait complete. 2024-09-13T17:19:27.298 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:19:27.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.461+0000 7f59db600640 1 Processor -- start 2024-09-13T17:19:27.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.462+0000 7f59db600640 1 -- start start 2024-09-13T17:19:27.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59db600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59db600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59d40b2400 0x7f59d406bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59db600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d4070470 con 0x7f59d40ad940 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59db600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d40ace30 con 0x7f59d40b2400 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59db600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d40ad110 con 0x7f59d415c9a0 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59d9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59d40b2400 0x7f59d406bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59d9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f59d40b2400 0x7f59d406bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44660/0 (socket says 172.21.15.28:44660) 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59d9600640 1 -- 172.21.15.28:0/1595888022 learned_addr learned my addr 172.21.15.28:0/1595888022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.463+0000 7f59d8c00640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.464+0000 7f59d9600640 1 -- 172.21.15.28:0/1595888022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 msgr2=0x7f59d40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.464+0000 7f59d9600640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.464+0000 7f59d9600640 1 -- 172.21.15.28:0/1595888022 >> v1:172.21.15.189:6789/0 conn(0x7f59d40ad940 legacy=0x7f59d40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.464+0000 7f59d9600640 1 -- 172.21.15.28:0/1595888022 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d406b170 con 0x7f59d40b2400 2024-09-13T17:19:27.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.465+0000 7f59d9600640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40b2400 0x7f59d406bc50 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009e50 tx=0x7f59cc02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3572654afd1bb32 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.465+0000 7f59d3400640 1 -- 172.21.15.28:0/1595888022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031600 con 0x7f59d40b2400 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.465+0000 7f59d3400640 1 -- 172.21.15.28:0/1595888022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59cc02d050 con 0x7f59d40b2400 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.465+0000 7f59d3400640 1 -- 172.21.15.28:0/1595888022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59cc031980 con 0x7f59d40b2400 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 -- 172.21.15.28:0/1595888022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40b2400 msgr2=0x7f59d406bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40b2400 0x7f59d406bc50 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f59cc009e50 tx=0x7f59cc02a7a0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 -- 172.21.15.28:0/1595888022 shutdown_connections 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40b2400 0x7f59d406bc50 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 --2- 172.21.15.28:0/1595888022 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 -- 172.21.15.28:0/1595888022 >> 172.21.15.28:0/1595888022 conn(0x7f59d40b0270 msgr2=0x7f59d40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 -- 172.21.15.28:0/1595888022 shutdown_connections 2024-09-13T17:19:27.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.466+0000 7f59db600640 1 -- 172.21.15.28:0/1595888022 wait complete. 2024-09-13T17:19:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.467+0000 7f59db600640 1 Processor -- start 2024-09-13T17:19:27.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.467+0000 7f59db600640 1 -- start start 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 0x7f59d41463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d4159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d41f5670 con 0x7f59d40ad940 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d41468f0 con 0x7f59d40b2400 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.468+0000 7f59db600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d4160230 con 0x7f59d415c9a0 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44670/0 (socket says 172.21.15.28:44670) 2024-09-13T17:19:27.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 -- 172.21.15.28:0/2665821388 learned_addr learned my addr 172.21.15.28:0/2665821388 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d9600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d4159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d3e00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 0x7f59d41463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 -- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 msgr2=0x7f59d4159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d4159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 -- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 msgr2=0x7f59d41463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 0x7f59d41463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d8c00640 1 -- 172.21.15.28:0/2665821388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc009b30 con 0x7f59d40ad940 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.469+0000 7f59d3e00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 0x7f59d41463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59d8c00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e870 tx=0x7f59c800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c80094f0 con 0x7f59d40ad940 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800c940 con 0x7f59d40ad940 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c800ccf0 con 0x7f59d40ad940 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d4070930 con 0x7f59d40ad940 2024-09-13T17:19:27.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.470+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d41f5ba0 con 0x7f59d40ad940 2024-09-13T17:19:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.471+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d4061090 con 0x7f59d40ad940 2024-09-13T17:19:27.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.472+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f59c8013470 con 0x7f59d40ad940 2024-09-13T17:19:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.473+0000 7f59d0c00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 0x7f59b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.473+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f59c8089540 con 0x7f59d40ad940 2024-09-13T17:19:27.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.473+0000 7f59d3e00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 0x7f59b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.474+0000 7f59d3e00640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 0x7f59b406d0c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007fd0 tx=0x7f59c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.478+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f59c80549a0 con 0x7f59d40ad940 2024-09-13T17:19:27.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:27.657+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f59d415cd80 con 0x7f59d40ad940 2024-09-13T17:19:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.276+0000 7f59d0c00640 1 -- 172.21.15.28:0/2665821388 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v165) v1 ==== 170+0+0 (secure 0 0 0) 0x7f59c8057fc0 con 0x7f59d40ad940 2024-09-13T17:19:28.278 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:19:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.284+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 msgr2=0x7f59b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.284+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 0x7f59b406d0c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007fd0 tx=0x7f59c0009040 comp rx=0 tx=0).stop 2024-09-13T17:19:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.284+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 msgr2=0x7f59d415fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.284+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f59c800e870 tx=0x7f59c800ed30 comp rx=0 tx=0).stop 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 shutdown_connections 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f59b406ac00 0x7f59b406d0c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59d415c9a0 0x7f59d4159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59d40b2400 0x7f59d41463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 --2- 172.21.15.28:0/2665821388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59d40ad940 0x7f59d415fbc0 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.286+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 >> 172.21.15.28:0/2665821388 conn(0x7f59d40b0270 msgr2=0x7f59d406baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.287+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 shutdown_connections 2024-09-13T17:19:28.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.287+0000 7f59db600640 1 -- 172.21.15.28:0/2665821388 wait complete. 2024-09-13T17:19:28.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:19:28.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.475+0000 7fa5d7600640 1 Processor -- start 2024-09-13T17:19:28.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.475+0000 7fa5d7600640 1 -- start start 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d7600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 0x7fa5d00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d7600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 0x7fa5d006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d7600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d0070470 con 0x7fa5d015c9a0 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d7600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d00ace30 con 0x7fa5d00ad940 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d7600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d00ad110 con 0x7fa5d00b2400 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d4c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa5d015c9a0 0x7fa5d00b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43034/0 (socket says 172.21.15.28:43034) 2024-09-13T17:19:28.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d4c00640 1 -- 172.21.15.28:0/3526259955 learned_addr learned my addr 172.21.15.28:0/3526259955 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:28.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/3526259955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 0x7fa5d00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5d5600640 1 --2- 172.21.15.28:0/3526259955 >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 0x7fa5d006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3971880454 0 0) 0x7fa5d0070470 con 0x7fa5d015c9a0 2024-09-13T17:19:28.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.477+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5b4003520 con 0x7fa5d015c9a0 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2554170728 0 0) 0x7fa5b4003520 con 0x7fa5d015c9a0 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5c7e00640 1 -- 172.21.15.28:0/3526259955 >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 msgr2=0x7fa5d006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/3526259955 >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 0x7fa5d006bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5c7e00640 1 -- 172.21.15.28:0/3526259955 >> v1:172.21.15.28:6789/0 conn(0x7fa5d015c9a0 legacy=0x7fa5d00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5c7e00640 1 -- 172.21.15.28:0/3526259955 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d006b170 con 0x7fa5d00b2400 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.478+0000 7fa5d5600640 1 --2- 172.21.15.28:0/3526259955 >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 0x7fa5d006bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.479+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/3526259955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 0x7fa5d00ac850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097a0 tx=0x7fa5b802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4aa78299cc41a00b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.479+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0001670 con 0x7fa5d00b2400 2024-09-13T17:19:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.479+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b802ac30 con 0x7fa5d00b2400 2024-09-13T17:19:28.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.479+0000 7fa5cfe00640 1 -- 172.21.15.28:0/3526259955 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8032440 con 0x7fa5d00b2400 2024-09-13T17:19:28.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 -- 172.21.15.28:0/3526259955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 msgr2=0x7fa5d00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 --2- 172.21.15.28:0/3526259955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 0x7fa5d00ac850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa5b80097a0 tx=0x7fa5b802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 -- 172.21.15.28:0/3526259955 shutdown_connections 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 --2- 172.21.15.28:0/3526259955 >> v2:172.21.15.189:3300/0 conn(0x7fa5d00ad940 0x7fa5d006bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 --2- 172.21.15.28:0/3526259955 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00b2400 0x7fa5d00ac850 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 -- 172.21.15.28:0/3526259955 >> 172.21.15.28:0/3526259955 conn(0x7fa5d00b0270 msgr2=0x7fa5d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 -- 172.21.15.28:0/3526259955 shutdown_connections 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.480+0000 7fa5d7600640 1 -- 172.21.15.28:0/3526259955 wait complete. 2024-09-13T17:19:28.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.482+0000 7fa5d7600640 1 Processor -- start 2024-09-13T17:19:28.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.482+0000 7fa5d7600640 1 -- start start 2024-09-13T17:19:28.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00ad940 0x7fa5d01f1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d00b2400 0x7fa5d01f4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01f2510 con 0x7fa5d015c9a0 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d01f52f0 con 0x7fa5d00b2400 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d7600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d015d500 con 0x7fa5d00ad940 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d4c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00ad940 0x7fa5d01f1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d00b2400 0x7fa5d01f4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44684/0 (socket says 172.21.15.28:44684) 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 -- 172.21.15.28:0/4035399214 learned_addr learned my addr 172.21.15.28:0/4035399214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 -- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00ad940 msgr2=0x7fa5d01f1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00ad940 0x7fa5d01f1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 -- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d00b2400 msgr2=0x7fa5d01f4db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d00b2400 0x7fa5d01f4db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.483+0000 7fa5d5600640 1 -- 172.21.15.28:0/4035399214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c8009a70 con 0x7fa5d015c9a0 2024-09-13T17:19:28.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5d5600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800ef90 tx=0x7fa5c8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8009d70 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c800f040 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b8009400 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c80146c0 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.484+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d0070930 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.485+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d0070730 con 0x7fa5d015c9a0 2024-09-13T17:19:28.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.487+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa5c8014820 con 0x7fa5d015c9a0 2024-09-13T17:19:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.487+0000 7fa5cd600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 0x7fa5ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.488+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa5c808a2f0 con 0x7fa5d015c9a0 2024-09-13T17:19:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.488+0000 7fa5d4c00640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 0x7fa5ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.489+0000 7fa5d4c00640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 0x7fa5ac06d0c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0005880 tx=0x7fa5c0001ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.492+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa5c80554c0 con 0x7fa5d015c9a0 2024-09-13T17:19:28.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.711+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa5d0070d10 con 0x7fa5d015c9a0 2024-09-13T17:19:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.718+0000 7fa5cd600640 1 -- 172.21.15.28:0/4035399214 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v134) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa5c8058ae0 con 0x7fa5d015c9a0 2024-09-13T17:19:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 22 and data pool 23 2024-09-13T17:19:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.724+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 msgr2=0x7fa5ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.724+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 0x7fa5ac06d0c0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa5c0005880 tx=0x7fa5c0001ce0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.724+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 msgr2=0x7fa5d0155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.724+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800ef90 tx=0x7fa5c8009f90 comp rx=0 tx=0).stop 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 shutdown_connections 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa5ac06ac00 0x7fa5ac06d0c0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5d015c9a0 0x7fa5d0155cc0 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5d00b2400 0x7fa5d01f4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 --2- 172.21.15.28:0/4035399214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa5d00ad940 0x7fa5d01f1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 >> 172.21.15.28:0/4035399214 conn(0x7fa5d00b0270 msgr2=0x7fa5d00af230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 shutdown_connections 2024-09-13T17:19:28.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.726+0000 7fa5d7600640 1 -- 172.21.15.28:0/4035399214 wait complete. 2024-09-13T17:19:28.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:28.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.910+0000 7f0d6e200640 1 Processor -- start 2024-09-13T17:19:28.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.910+0000 7f0d6e200640 1 -- start start 2024-09-13T17:19:28.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0d681592a0 0x7f0d6815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 0x7f0d68159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d6806c2d0 0x7f0d6806c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6815b0b0 con 0x7f0d6806c2d0 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6806cca0 con 0x7f0d681592a0 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.911+0000 7f0d6e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d680b2040 con 0x7f0d6806f970 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d6806c2d0 0x7f0d6806c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0d6806c2d0 0x7f0d6806c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44700/0 (socket says 172.21.15.28:44700) 2024-09-13T17:19:28.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 -- 172.21.15.28:0/2313845688 learned_addr learned my addr 172.21.15.28:0/2313845688 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 -- 172.21.15.28:0/2313845688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 msgr2=0x7f0d68159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d66a00640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 0x7f0d68159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 0x7f0d68159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 -- 172.21.15.28:0/2313845688 >> v2:172.21.15.189:3300/0 conn(0x7f0d681592a0 msgr2=0x7f0d6815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 --2- 172.21.15.28:0/2313845688 >> v2:172.21.15.189:3300/0 conn(0x7f0d681592a0 0x7f0d6815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d67e00640 1 -- 172.21.15.28:0/2313845688 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d68159ff0 con 0x7f0d6806c2d0 2024-09-13T17:19:28.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.912+0000 7f0d66a00640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 0x7f0d68159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:28.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.913+0000 7f0d67e00640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d6806c2d0 0x7f0d6806c760 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0d6806d370 tx=0x7f0d6002c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabe978ac2a7600c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.913+0000 7f0d66000640 1 -- 172.21.15.28:0/2313845688 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6002f070 con 0x7f0d6806c2d0 2024-09-13T17:19:28.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.913+0000 7f0d66000640 1 -- 172.21.15.28:0/2313845688 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6000b2e0 con 0x7f0d6806c2d0 2024-09-13T17:19:28.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.913+0000 7f0d66000640 1 -- 172.21.15.28:0/2313845688 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60009820 con 0x7f0d6806c2d0 2024-09-13T17:19:28.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 -- 172.21.15.28:0/2313845688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d6806c2d0 msgr2=0x7f0d6806c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d6806c2d0 0x7f0d6806c760 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0d6806d370 tx=0x7f0d6002c8a0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 -- 172.21.15.28:0/2313845688 shutdown_connections 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d6806c2d0 0x7f0d6806c760 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 --2- 172.21.15.28:0/2313845688 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806f970 0x7f0d68159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 --2- 172.21.15.28:0/2313845688 >> v2:172.21.15.189:3300/0 conn(0x7f0d681592a0 0x7f0d6815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 -- 172.21.15.28:0/2313845688 >> 172.21.15.28:0/2313845688 conn(0x7f0d68069e80 msgr2=0x7f0d6806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 -- 172.21.15.28:0/2313845688 shutdown_connections 2024-09-13T17:19:28.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.914+0000 7f0d6e200640 1 -- 172.21.15.28:0/2313845688 wait complete. 2024-09-13T17:19:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.916+0000 7f0d6e200640 1 Processor -- start 2024-09-13T17:19:28.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.916+0000 7f0d6e200640 1 -- start start 2024-09-13T17:19:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 0x7f0d6814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 0x7f0d680b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d681edc80 con 0x7f0d681592a0 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814fff0 con 0x7f0d6806f970 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d6e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d68150da0 con 0x7f0d6806c2d0 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44710/0 (socket says 172.21.15.28:44710) 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 -- 172.21.15.28:0/122828336 learned_addr learned my addr 172.21.15.28:0/122828336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d66a00640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 0x7f0d680b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 -- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 msgr2=0x7f0d6814f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67400640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 0x7f0d6814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 0x7f0d6814f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 -- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 msgr2=0x7f0d680b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 0x7f0d680b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67e00640 1 -- 172.21.15.28:0/122828336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6000bd40 con 0x7f0d681592a0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d66a00640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 0x7f0d680b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.917+0000 7f0d67400640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 0x7f0d6814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d67e00640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0d6002ce90 tx=0x7f0d60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60034bc0 con 0x7f0d681592a0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d60034d20 con 0x7f0d681592a0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d60047630 con 0x7f0d681592a0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d68150220 con 0x7f0d681592a0 2024-09-13T17:19:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.918+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d681507e0 con 0x7f0d681592a0 2024-09-13T17:19:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.919+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d681eddd0 con 0x7f0d681592a0 2024-09-13T17:19:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.920+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0d6003d410 con 0x7f0d681592a0 2024-09-13T17:19:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.921+0000 7f0d5f400640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 0x7f0d4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.921+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d600ac290 con 0x7f0d681592a0 2024-09-13T17:19:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.922+0000 7f0d67400640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 0x7f0d4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.922+0000 7f0d67400640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 0x7f0d4806d0c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800a870 tx=0x7f0d58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:28.925+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0d60077480 con 0x7f0d681592a0 2024-09-13T17:19:29.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.103+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d6806c760 con 0x7f0d681592a0 2024-09-13T17:19:29.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.105+0000 7f0d5f400640 1 -- 172.21.15.28:0/122828336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f0d6000b720 con 0x7f0d681592a0 2024-09-13T17:19:29.107 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:29.107 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":166,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:19:28.713665+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 msgr2=0x7f0d4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 0x7f0d4806d0c0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f0d5800a870 tx=0x7f0d58008040 comp rx=0 tx=0).stop 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 msgr2=0x7f0d6814fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0d6002ce90 tx=0x7f0d60004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 shutdown_connections 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0d4806ac00 0x7f0d4806d0c0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d681592a0 0x7f0d6814fab0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d6806f970 0x7f0d680b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 --2- 172.21.15.28:0/122828336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0d6806c2d0 0x7f0d6814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 >> 172.21.15.28:0/122828336 conn(0x7f0d68069e80 msgr2=0x7f0d68157110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 shutdown_connections 2024-09-13T17:19:29.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.107+0000 7f0d6e200640 1 -- 172.21.15.28:0/122828336 wait complete. 2024-09-13T17:19:29.126 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.280+0000 7f2477600640 1 Processor -- start 2024-09-13T17:19:29.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.280+0000 7f2477600640 1 -- start start 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.281+0000 7f2477600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 0x7f247006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.281+0000 7f2477600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 0x7f2470071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.281+0000 7f2477600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247015e100 con 0x7f247015a540 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.281+0000 7f2477600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24700adb00 con 0x7f24700cabc0 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.281+0000 7f2477600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2470071670 con 0x7f2470070cc0 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2475600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 0x7f2470071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2474c00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f247015a540 0x7f247015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:46172/0 (socket says 172.21.15.28:46172) 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2474c00640 1 -- 172.21.15.28:0/1405912927 learned_addr learned my addr 172.21.15.28:0/1405912927 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:29.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f246fe00640 1 --2- 172.21.15.28:0/1405912927 >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 0x7f247006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2475600640 1 -- 172.21.15.28:0/1405912927 >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 msgr2=0x7f247006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2475600640 1 --2- 172.21.15.28:0/1405912927 >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 0x7f247006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2475600640 1 -- 172.21.15.28:0/1405912927 >> v1:172.21.15.189:6789/0 conn(0x7f247015a540 legacy=0x7f247015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.282+0000 7f2475600640 1 -- 172.21.15.28:0/1405912927 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247015e3e0 con 0x7f2470070cc0 2024-09-13T17:19:29.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.283+0000 7f246fe00640 1 --2- 172.21.15.28:0/1405912927 >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 0x7f247006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:29.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.284+0000 7f2475600640 1 --2- 172.21.15.28:0/1405912927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 0x7f2470071130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2468009d90 tx=0x7f246802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81020c8c57059457 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.289+0000 7f246f400640 1 -- 172.21.15.28:0/1405912927 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468031600 con 0x7f2470070cc0 2024-09-13T17:19:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.289+0000 7f246f400640 1 -- 172.21.15.28:0/1405912927 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246802d050 con 0x7f2470070cc0 2024-09-13T17:19:29.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.289+0000 7f246f400640 1 -- 172.21.15.28:0/1405912927 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2468031980 con 0x7f2470070cc0 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.289+0000 7f2477600640 1 -- 172.21.15.28:0/1405912927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 msgr2=0x7f2470071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.289+0000 7f2477600640 1 --2- 172.21.15.28:0/1405912927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 0x7f2470071130 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2468009d90 tx=0x7f246802a5d0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 -- 172.21.15.28:0/1405912927 shutdown_connections 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 --2- 172.21.15.28:0/1405912927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2470070cc0 0x7f2470071130 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 --2- 172.21.15.28:0/1405912927 >> v2:172.21.15.189:3300/0 conn(0x7f24700cabc0 0x7f247006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 -- 172.21.15.28:0/1405912927 >> 172.21.15.28:0/1405912927 conn(0x7f2470069e80 msgr2=0x7f247006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 -- 172.21.15.28:0/1405912927 shutdown_connections 2024-09-13T17:19:29.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.290+0000 7f2477600640 1 -- 172.21.15.28:0/1405912927 wait complete. 2024-09-13T17:19:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.291+0000 7f2477600640 1 Processor -- start 2024-09-13T17:19:29.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.291+0000 7f2477600640 1 -- start start 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24700cabc0 0x7f247015c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247015a540 0x7f247015ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2470157e40 con 0x7f2470070cc0 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24700adf20 con 0x7f247015a540 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.292+0000 7f2477600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f247014ee80 con 0x7f24700cabc0 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44720/0 (socket says 172.21.15.28:44720) 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 -- 172.21.15.28:0/3592281839 learned_addr learned my addr 172.21.15.28:0/3592281839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2475600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247015a540 0x7f247015ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f246fe00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24700cabc0 0x7f247015c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 -- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24700cabc0 msgr2=0x7f247015c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24700cabc0 0x7f247015c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 -- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247015a540 msgr2=0x7f247015ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247015a540 0x7f247015ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 -- 172.21.15.28:0/3592281839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c009400 con 0x7f2470070cc0 2024-09-13T17:19:29.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.293+0000 7f2474c00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f246400c480 tx=0x7f246400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.294+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464005ce0 con 0x7f2470070cc0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.294+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464005e40 con 0x7f2470070cc0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.294+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640072d0 con 0x7f2470070cc0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.294+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2468009a70 con 0x7f2470070cc0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.294+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24700b0ca0 con 0x7f2470070cc0 2024-09-13T17:19:29.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.295+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2470071130 con 0x7f2470070cc0 2024-09-13T17:19:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.296+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f24640074f0 con 0x7f2470070cc0 2024-09-13T17:19:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.297+0000 7f246cc00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 0x7f245006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.297+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2464088440 con 0x7f2470070cc0 2024-09-13T17:19:29.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.297+0000 7f246fe00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 0x7f245006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.298+0000 7f246fe00640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 0x7f245006d0c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f245c009cd0 tx=0x7f245c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.301+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2464053640 con 0x7f2470070cc0 2024-09-13T17:19:29.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.480+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2470070560 con 0x7f2470070cc0 2024-09-13T17:19:29.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.481+0000 7f246cc00640 1 -- 172.21.15.28:0/3592281839 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+16584 (secure 0 0 0) 0x7f246406ee60 con 0x7f2470070cc0 2024-09-13T17:19:29.484 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:29.484 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":166,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:19:28.713665+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 msgr2=0x7f245006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 0x7f245006d0c0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f245c009cd0 tx=0x7f245c009190 comp rx=0 tx=0).stop 2024-09-13T17:19:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 msgr2=0x7f247014e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f246400c480 tx=0x7f246400c940 comp rx=0 tx=0).stop 2024-09-13T17:19:29.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 shutdown_connections 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f245006ac00 0x7f245006d0c0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f247015a540 0x7f247015ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24700cabc0 0x7f247015c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 --2- 172.21.15.28:0/3592281839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2470070cc0 0x7f247014e940 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 >> 172.21.15.28:0/3592281839 conn(0x7f2470069e80 msgr2=0x7f247006f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 shutdown_connections 2024-09-13T17:19:29.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.483+0000 7f2477600640 1 -- 172.21.15.28:0/3592281839 wait complete. 2024-09-13T17:19:29.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:19:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.657+0000 7f74db200640 1 Processor -- start 2024-09-13T17:19:29.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.657+0000 7f74db200640 1 -- start start 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74db200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 0x7f74d40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74db200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74d40ad940 0x7f74d406bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74db200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d4070470 con 0x7f74d415c9a0 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74db200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d40ace30 con 0x7f74d40ad940 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74db200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d40ad110 con 0x7f74d40b2400 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74d9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74d40ad940 0x7f74d406bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74d9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f74d40ad940 0x7f74d406bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44736/0 (socket says 172.21.15.28:44736) 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.658+0000 7f74d9200640 1 -- 172.21.15.28:0/11296037 learned_addr learned my addr 172.21.15.28:0/11296037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74cbe00640 1 --2- 172.21.15.28:0/11296037 >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 0x7f74d40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74d9200640 1 -- 172.21.15.28:0/11296037 >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 msgr2=0x7f74d40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74d9200640 1 --2- 172.21.15.28:0/11296037 >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 0x7f74d40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74d9200640 1 -- 172.21.15.28:0/11296037 >> v1:172.21.15.189:6789/0 conn(0x7f74d415c9a0 legacy=0x7f74d40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74d9200640 1 -- 172.21.15.28:0/11296037 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d406b170 con 0x7f74d40ad940 2024-09-13T17:19:29.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.659+0000 7f74cbe00640 1 --2- 172.21.15.28:0/11296037 >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 0x7f74d40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.660+0000 7f74d9200640 1 --2- 172.21.15.28:0/11296037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d406bc50 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f74d40ce450 tx=0x7f74cc02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd734916c1c673d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.660+0000 7f74d3400640 1 -- 172.21.15.28:0/11296037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc031600 con 0x7f74d40ad940 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.660+0000 7f74d3400640 1 -- 172.21.15.28:0/11296037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74cc02d050 con 0x7f74d40ad940 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.660+0000 7f74d3400640 1 -- 172.21.15.28:0/11296037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74cc031980 con 0x7f74d40ad940 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 -- 172.21.15.28:0/11296037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 msgr2=0x7f74d406bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 --2- 172.21.15.28:0/11296037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d406bc50 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f74d40ce450 tx=0x7f74cc02a730 comp rx=0 tx=0).stop 2024-09-13T17:19:29.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 -- 172.21.15.28:0/11296037 shutdown_connections 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 --2- 172.21.15.28:0/11296037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d406bc50 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 --2- 172.21.15.28:0/11296037 >> v2:172.21.15.189:3300/0 conn(0x7f74d40b2400 0x7f74d40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 -- 172.21.15.28:0/11296037 >> 172.21.15.28:0/11296037 conn(0x7f74d40b0270 msgr2=0x7f74d40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 -- 172.21.15.28:0/11296037 shutdown_connections 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.661+0000 7f74db200640 1 -- 172.21.15.28:0/11296037 wait complete. 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.662+0000 7f74db200640 1 Processor -- start 2024-09-13T17:19:29.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.663+0000 7f74db200640 1 -- start start 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.663+0000 7f74db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.663+0000 7f74db200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d40b2400 0x7f74d414c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74db200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f74d415c9a0 0x7f74d414b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40cc9c0 con 0x7f74d40ad940 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74db200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40ccc60 con 0x7f74d40b2400 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74db200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40ccf40 con 0x7f74d415c9a0 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44752/0 (socket says 172.21.15.28:44752) 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 -- 172.21.15.28:0/1882192369 learned_addr learned my addr 172.21.15.28:0/1882192369 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74cbe00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d40b2400 0x7f74d414c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d9200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f74d415c9a0 0x7f74d414b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 -- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f74d415c9a0 msgr2=0x7f74d414b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f74d415c9a0 0x7f74d414b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 -- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d40b2400 msgr2=0x7f74d414c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d40b2400 0x7f74d414c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.664+0000 7f74d3e00640 1 -- 172.21.15.28:0/1882192369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc009b30 con 0x7f74d40ad940 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74d3e00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c400 tx=0x7f74c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c40052d0 con 0x7f74d40ad940 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c4005430 con 0x7f74d40ad940 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c4003830 con 0x7f74d40ad940 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d4070930 con 0x7f74d40ad940 2024-09-13T17:19:29.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.665+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d40cd500 con 0x7f74d40ad940 2024-09-13T17:19:29.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.666+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d40cd710 con 0x7f74d40ad940 2024-09-13T17:19:29.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.667+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f74c4003990 con 0x7f74d40ad940 2024-09-13T17:19:29.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.668+0000 7f74d0c00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 0x7f74b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:29.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.668+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74c40888c0 con 0x7f74d40ad940 2024-09-13T17:19:29.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.668+0000 7f74cbe00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 0x7f74b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:29.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.669+0000 7f74cbe00640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 0x7f74b006d0c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f74d40afbc0 tx=0x7f74bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:29.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.672+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f74c4053ab0 con 0x7f74d40ad940 2024-09-13T17:19:29.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:29.895+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f74d415cd80 con 0x7f74d40ad940 2024-09-13T17:19:30.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.733+0000 7f74d0c00640 1 -- 172.21.15.28:0/1882192369 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v137) v1 ==== 118+0+0 (secure 0 0 0) 0x7f74c40170b0 con 0x7f74d40ad940 2024-09-13T17:19:30.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 msgr2=0x7f74b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 0x7f74b006d0c0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f74d40afbc0 tx=0x7f74bc008040 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 msgr2=0x7f74d4141d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f74c400c400 tx=0x7f74c400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 shutdown_connections 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f74b006ac00 0x7f74b006d0c0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f74d415c9a0 0x7f74d414b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74d40b2400 0x7f74d414c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 --2- 172.21.15.28:0/1882192369 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f74d40ad940 0x7f74d4141d20 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 >> 172.21.15.28:0/1882192369 conn(0x7f74d40b0270 msgr2=0x7f74d4156740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 shutdown_connections 2024-09-13T17:19:30.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.736+0000 7f74db200640 1 -- 172.21.15.28:0/1882192369 wait complete. 2024-09-13T17:19:30.757 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:19:30.757 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:19:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.920+0000 7f1d22600640 1 Processor -- start 2024-09-13T17:19:30.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.920+0000 7f1d22600640 1 -- start start 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d22600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d1c0cdc70 0x7f1d1c0b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d22600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c06f5c0 con 0x7f1d1c071240 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d22600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1c06f8a0 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d22600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d1c06fb80 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d12a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d1c0cdc70 0x7f1d1c0b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:30.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d1be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1d1c071240 0x7f1d1c06bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43046/0 (socket says 172.21.15.28:43046) 2024-09-13T17:19:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.921+0000 7f1d1be00640 1 -- 172.21.15.28:0/4266481126 learned_addr learned my addr 172.21.15.28:0/4266481126 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.922+0000 7f1d12a00640 1 -- 172.21.15.28:0/4266481126 >> v1:172.21.15.189:6789/0 conn(0x7f1d1c15ccf0 legacy=0x7f1d1c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.922+0000 7f1d12a00640 1 -- 172.21.15.28:0/4266481126 >> v1:172.21.15.28:6789/0 conn(0x7f1d1c071240 legacy=0x7f1d1c06bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.922+0000 7f1d12a00640 1 -- 172.21.15.28:0/4266481126 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c06b610 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d12a00640 1 --2- 172.21.15.28:0/4266481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c0b1f10 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f1d040097a0 tx=0x7f1d0402a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a1de755da063d4d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d1aa00640 1 -- 172.21.15.28:0/4266481126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04032640 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d1aa00640 1 -- 172.21.15.28:0/4266481126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d040327a0 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d1aa00640 1 -- 172.21.15.28:0/4266481126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04032b30 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d22600640 1 -- 172.21.15.28:0/4266481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c0cdc70 msgr2=0x7f1d1c0b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.923+0000 7f1d22600640 1 --2- 172.21.15.28:0/4266481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c0b1f10 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f1d040097a0 tx=0x7f1d0402a2c0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.924+0000 7f1d22600640 1 -- 172.21.15.28:0/4266481126 shutdown_connections 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.924+0000 7f1d22600640 1 --2- 172.21.15.28:0/4266481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c0b1f10 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.924+0000 7f1d22600640 1 -- 172.21.15.28:0/4266481126 >> 172.21.15.28:0/4266481126 conn(0x7f1d1c0b0200 msgr2=0x7f1d1c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.924+0000 7f1d22600640 1 -- 172.21.15.28:0/4266481126 shutdown_connections 2024-09-13T17:19:30.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.924+0000 7f1d22600640 1 -- 172.21.15.28:0/4266481126 wait complete. 2024-09-13T17:19:30.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.925+0000 7f1d22600640 1 Processor -- start 2024-09-13T17:19:30.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.925+0000 7f1d22600640 1 -- start start 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d1c071240 0x7f1d1c14f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c1f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c14f750 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c153e20 con 0x7f1d1c0cdc70 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.926+0000 7f1d22600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c153210 con 0x7f1d1c071240 2024-09-13T17:19:30.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44766/0 (socket says 172.21.15.28:44766) 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 -- 172.21.15.28:0/3352266865 learned_addr learned my addr 172.21.15.28:0/3352266865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d12a00640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c1f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1b400640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d1c071240 0x7f1d1c14f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 -- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d1c071240 msgr2=0x7f1d1c14f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d1c071240 0x7f1d1c14f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 -- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d1c0cdc70 msgr2=0x7f1d1c1f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c1f2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 -- 172.21.15.28:0/3352266865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04009400 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.927+0000 7f1d1be00640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400cbc0 tx=0x7f1d1400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.928+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400ad40 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.928+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d14012400 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.928+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d1400aea0 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.928+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d1c1534f0 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.928+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d1c153ab0 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.929+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d1c070710 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.930+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1d14007070 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.930+0000 7f1d13e00640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 0x7f1cf806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.931+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1d14089260 con 0x7f1d1c15ccf0 2024-09-13T17:19:30.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.931+0000 7f1d1b400640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 0x7f1cf806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:30.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.932+0000 7f1d1b400640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 0x7f1cf806d0c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000f60 tx=0x7f1d0c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:30.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:30.935+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1d14054450 con 0x7f1d1c15ccf0 2024-09-13T17:19:31.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.148+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f1d1c153f60 con 0x7f1d1c15ccf0 2024-09-13T17:19:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.739+0000 7f1d13e00640 1 -- 172.21.15.28:0/3352266865 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v138) v1 ==== 122+0+0 (secure 0 0 0) 0x7f1d14057a70 con 0x7f1d1c15ccf0 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.741+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 msgr2=0x7f1cf806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.741+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 0x7f1cf806d0c0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c000f60 tx=0x7f1d0c009040 comp rx=0 tx=0).stop 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 msgr2=0x7f1d1c1f3130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f1d1400cbc0 tx=0x7f1d1400a430 comp rx=0 tx=0).stop 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 shutdown_connections 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cf806ac00 0x7f1cf806d0c0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d1c15ccf0 0x7f1d1c1f3130 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d1c0cdc70 0x7f1d1c1f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 --2- 172.21.15.28:0/3352266865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d1c071240 0x7f1d1c14f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 >> 172.21.15.28:0/3352266865 conn(0x7f1d1c0b0200 msgr2=0x7f1d1c156610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 shutdown_connections 2024-09-13T17:19:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.742+0000 7f1d22600640 1 -- 172.21.15.28:0/3352266865 wait complete. 2024-09-13T17:19:31.763 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:19:31.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.918+0000 7f256c600640 1 Processor -- start 2024-09-13T17:19:31.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.918+0000 7f256c600640 1 -- start start 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.919+0000 7f256c600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f256406c9e0 0x7f256415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.919+0000 7f256c600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2564159d90 con 0x7f25641592a0 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.919+0000 7f256c600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25640b2040 con 0x7f256406f970 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.919+0000 7f256c600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f256415b0b0 con 0x7f256406c9e0 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f256a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f256406c9e0 0x7f256415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:31.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f2569c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f25641592a0 0x7f2564159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43062/0 (socket says 172.21.15.28:43062) 2024-09-13T17:19:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f2569c00640 1 -- 172.21.15.28:0/2803013144 learned_addr learned my addr 172.21.15.28:0/2803013144 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f256a600640 1 -- 172.21.15.28:0/2803013144 >> v1:172.21.15.189:6789/0 conn(0x7f256406f970 legacy=0x7f256406c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f256a600640 1 -- 172.21.15.28:0/2803013144 >> v1:172.21.15.28:6789/0 conn(0x7f25641592a0 legacy=0x7f2564159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f256a600640 1 -- 172.21.15.28:0/2803013144 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2564159ff0 con 0x7f256406c9e0 2024-09-13T17:19:31.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.920+0000 7f2563e00640 1 -- 172.21.15.28:0/2803013144 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673885024 0 0) 0x7f2564159d90 con 0x7f25641592a0 2024-09-13T17:19:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.921+0000 7f256a600640 1 --2- 172.21.15.28:0/2803013144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f256415ab70 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f255c009d90 tx=0x7f255c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1130d514f7e89e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.921+0000 7f2563e00640 1 -- 172.21.15.28:0/2803013144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c007740 con 0x7f256406c9e0 2024-09-13T17:19:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.921+0000 7f2563e00640 1 -- 172.21.15.28:0/2803013144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255c02d050 con 0x7f256406c9e0 2024-09-13T17:19:31.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.921+0000 7f2563e00640 1 -- 172.21.15.28:0/2803013144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255c007ac0 con 0x7f256406c9e0 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 -- 172.21.15.28:0/2803013144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 msgr2=0x7f256415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 --2- 172.21.15.28:0/2803013144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f256415ab70 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f255c009d90 tx=0x7f255c02a6a0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 -- 172.21.15.28:0/2803013144 shutdown_connections 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 --2- 172.21.15.28:0/2803013144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f256415ab70 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 -- 172.21.15.28:0/2803013144 >> 172.21.15.28:0/2803013144 conn(0x7f2564069e80 msgr2=0x7f256406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 -- 172.21.15.28:0/2803013144 shutdown_connections 2024-09-13T17:19:31.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.922+0000 7f256c600640 1 -- 172.21.15.28:0/2803013144 wait complete. 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.924+0000 7f256c600640 1 Processor -- start 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.924+0000 7f256c600640 1 -- start start 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f256406f970 0x7f25641f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25641592a0 0x7f25641542f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25640ce050 con 0x7f256406c9e0 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25641e2990 con 0x7f256406f970 2024-09-13T17:19:31.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256c600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25641e2c70 con 0x7f25641592a0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f2569c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f2569c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44794/0 (socket says 172.21.15.28:44794) 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f2569c00640 1 -- 172.21.15.28:0/2599654714 learned_addr learned my addr 172.21.15.28:0/2599654714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f2569200640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f256406f970 0x7f25641f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.925+0000 7f256a600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25641592a0 0x7f25641542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 -- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25641592a0 msgr2=0x7f25641542f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25641592a0 0x7f25641542f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 -- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f256406f970 msgr2=0x7f25641f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f256406f970 0x7f25641f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 -- 172.21.15.28:0/2599654714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c009a70 con 0x7f256406c9e0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2569c00640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f255800c4a0 tx=0x7f255800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2558005ce0 con 0x7f256406c9e0 2024-09-13T17:19:31.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2558005e40 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25640ce2f0 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25580072b0 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.926+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25641e2620 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.927+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256415ab70 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.928+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2558007410 con 0x7f256406c9e0 2024-09-13T17:19:31.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.930+0000 7f2561600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 0x7f254406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:31.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.930+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2558088920 con 0x7f256406c9e0 2024-09-13T17:19:31.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.930+0000 7f2569200640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 0x7f254406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:31.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.931+0000 7f2569200640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 0x7f254406d0c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2564154d40 tx=0x7f2550007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:31.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:31.934+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2558053b10 con 0x7f256406c9e0 2024-09-13T17:19:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.149+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25641e2e30 con 0x7f256406c9e0 2024-09-13T17:19:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.150+0000 7f2561600640 1 -- 172.21.15.28:0/2599654714 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f2558017020 con 0x7f256406c9e0 2024-09-13T17:19:32.152 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:32.152 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:32.152 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 msgr2=0x7f254406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 0x7f254406d0c0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2564154d40 tx=0x7f2550007e30 comp rx=0 tx=0).stop 2024-09-13T17:19:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 msgr2=0x7f2564158020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f255800c4a0 tx=0x7f255800c960 comp rx=0 tx=0).stop 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 shutdown_connections 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f254406ac00 0x7f254406d0c0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f25641592a0 0x7f25641542f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f256406f970 0x7f25641f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 --2- 172.21.15.28:0/2599654714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f256406c9e0 0x7f2564158020 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 >> 172.21.15.28:0/2599654714 conn(0x7f2564069e80 msgr2=0x7f256406fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 shutdown_connections 2024-09-13T17:19:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.152+0000 7f256c600640 1 -- 172.21.15.28:0/2599654714 wait complete. 2024-09-13T17:19:32.170 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:19:32.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.323+0000 7fa484000640 1 Processor -- start 2024-09-13T17:19:32.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.324+0000 7fa484000640 1 -- start start 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa484000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa47c1592a0 0x7fa47c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa484000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa47c06f970 0x7fa47c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa484000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa47c0b2040 con 0x7fa47c06c2d0 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa484000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c15b0b0 con 0x7fa47c06f970 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa484000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47c159ff0 con 0x7fa47c1592a0 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa480c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa47c06f970 0x7fa47c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa482000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa47c06c2d0 0x7fa47c0b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43066/0 (socket says 172.21.15.28:43066) 2024-09-13T17:19:32.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa482000640 1 -- 172.21.15.28:0/1001543744 learned_addr learned my addr 172.21.15.28:0/1001543744 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:32.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.325+0000 7fa481600640 1 --2- 172.21.15.28:0/1001543744 >> v2:172.21.15.189:3300/0 conn(0x7fa47c1592a0 0x7fa47c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.326+0000 7fa480c00640 1 -- 172.21.15.28:0/1001543744 >> v2:172.21.15.189:3300/0 conn(0x7fa47c1592a0 msgr2=0x7fa47c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:32.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.326+0000 7fa480c00640 1 --2- 172.21.15.28:0/1001543744 >> v2:172.21.15.189:3300/0 conn(0x7fa47c1592a0 0x7fa47c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.326+0000 7fa480c00640 1 -- 172.21.15.28:0/1001543744 >> v1:172.21.15.28:6789/0 conn(0x7fa47c06c2d0 legacy=0x7fa47c0b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.326+0000 7fa480c00640 1 -- 172.21.15.28:0/1001543744 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47c15a2d0 con 0x7fa47c06f970 2024-09-13T17:19:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.326+0000 7fa47be00640 1 -- 172.21.15.28:0/1001543744 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862560492 0 0) 0x7fa47c0b2040 con 0x7fa47c06c2d0 2024-09-13T17:19:32.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.327+0000 7fa480c00640 1 --2- 172.21.15.28:0/1001543744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06f970 0x7fa47c159680 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa47c06d370 tx=0x7fa46802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115e9a64ee5aa3aa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.327+0000 7fa47be00640 1 -- 172.21.15.28:0/1001543744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468032440 con 0x7fa47c06f970 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.327+0000 7fa47be00640 1 -- 172.21.15.28:0/1001543744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4680325a0 con 0x7fa47c06f970 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.327+0000 7fa47be00640 1 -- 172.21.15.28:0/1001543744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468032930 con 0x7fa47c06f970 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 -- 172.21.15.28:0/1001543744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06f970 msgr2=0x7fa47c159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 --2- 172.21.15.28:0/1001543744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06f970 0x7fa47c159680 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fa47c06d370 tx=0x7fa46802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 -- 172.21.15.28:0/1001543744 shutdown_connections 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 --2- 172.21.15.28:0/1001543744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06f970 0x7fa47c159680 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 --2- 172.21.15.28:0/1001543744 >> v2:172.21.15.189:3300/0 conn(0x7fa47c1592a0 0x7fa47c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 -- 172.21.15.28:0/1001543744 >> 172.21.15.28:0/1001543744 conn(0x7fa47c069e80 msgr2=0x7fa47c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:32.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 -- 172.21.15.28:0/1001543744 shutdown_connections 2024-09-13T17:19:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.328+0000 7fa484000640 1 -- 172.21.15.28:0/1001543744 wait complete. 2024-09-13T17:19:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.329+0000 7fa484000640 1 Processor -- start 2024-09-13T17:19:32.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.329+0000 7fa484000640 1 -- start start 2024-09-13T17:19:32.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 0x7fa47c144260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa47c1592a0 0x7fa47c1447a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c12cde0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c144dd0 con 0x7fa47c06f970 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.330+0000 7fa484000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa47c1450b0 con 0x7fa47c1592a0 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44822/0 (socket says 172.21.15.28:44822) 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 -- 172.21.15.28:0/3483077948 learned_addr learned my addr 172.21.15.28:0/3483077948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa482000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa47c1592a0 0x7fa47c1447a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa480c00640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 0x7fa47c144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 -- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa47c1592a0 msgr2=0x7fa47c1447a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa47c1592a0 0x7fa47c1447a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 -- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 msgr2=0x7fa47c144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 0x7fa47c144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 -- 172.21.15.28:0/3483077948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa468009400 con 0x7fa47c06c2d0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa480c00640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 0x7fa47c144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa481600640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa47000e870 tx=0x7fa47000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.331+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4700094f0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.332+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47000c940 con 0x7fa47c06c2d0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.332+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47c145390 con 0x7fa47c06c2d0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.332+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47000ccf0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.332+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47c0cb870 con 0x7fa47c06c2d0 2024-09-13T17:19:32.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.333+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47c1455d0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.334+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa470013470 con 0x7fa47c06c2d0 2024-09-13T17:19:32.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.334+0000 7fa479600640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 0x7fa45c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.335+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4700894e0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.335+0000 7fa480c00640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 0x7fa45c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.336+0000 7fa480c00640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 0x7fa45c06d0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa468034000 tx=0x7fa46802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.338+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa47c1455d0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.518+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa47c1455d0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.520+0000 7fa479600640 1 -- 172.21.15.28:0/3483077948 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v167) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fa47c1455d0 con 0x7fa47c06c2d0 2024-09-13T17:19:32.522 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:32.522 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":167,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:19:29.716517+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:19:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 msgr2=0x7fa45c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 0x7fa45c06d0c0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fa468034000 tx=0x7fa46802e000 comp rx=0 tx=0).stop 2024-09-13T17:19:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 msgr2=0x7fa47c140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7fa47000e870 tx=0x7fa47000ed30 comp rx=0 tx=0).stop 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 shutdown_connections 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa45c06ac00 0x7fa45c06d0c0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa47c1592a0 0x7fa47c1447a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47c06f970 0x7fa47c144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 --2- 172.21.15.28:0/3483077948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa47c06c2d0 0x7fa47c140520 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 >> 172.21.15.28:0/3483077948 conn(0x7fa47c069e80 msgr2=0x7fa47c06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 shutdown_connections 2024-09-13T17:19:32.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.522+0000 7fa484000640 1 -- 172.21.15.28:0/3483077948 wait complete. 2024-09-13T17:19:32.540 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:19:32.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:19:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.694+0000 7fe298c00640 1 Processor -- start 2024-09-13T17:19:32.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.694+0000 7fe298c00640 1 -- start start 2024-09-13T17:19:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.695+0000 7fe298c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2941592a0 0x7fe29415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.695+0000 7fe298c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe294159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.695+0000 7fe298c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2940b2040 con 0x7fe29406c2d0 2024-09-13T17:19:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.695+0000 7fe298c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29415b0b0 con 0x7fe2941592a0 2024-09-13T17:19:32.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.695+0000 7fe298c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe294159ff0 con 0x7fe29406f970 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2941592a0 0x7fe29415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe2941592a0 0x7fe29415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44826/0 (socket says 172.21.15.28:44826) 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 -- 172.21.15.28:0/4010896325 learned_addr learned my addr 172.21.15.28:0/4010896325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291400640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe294159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 -- 172.21.15.28:0/4010896325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 msgr2=0x7fe294159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe294159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 -- 172.21.15.28:0/4010896325 >> v1:172.21.15.189:6789/0 conn(0x7fe29406c2d0 legacy=0x7fe2940b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.696+0000 7fe291e00640 1 -- 172.21.15.28:0/4010896325 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29415a2d0 con 0x7fe2941592a0 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.697+0000 7fe291e00640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe29415ab70 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe284009870 tx=0x7fe28402c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ffd607667b23e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.697+0000 7fe290a00640 1 -- 172.21.15.28:0/4010896325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284031710 con 0x7fe2941592a0 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.697+0000 7fe290a00640 1 -- 172.21.15.28:0/4010896325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28402f050 con 0x7fe2941592a0 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.697+0000 7fe290a00640 1 -- 172.21.15.28:0/4010896325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe284031a90 con 0x7fe2941592a0 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 -- 172.21.15.28:0/4010896325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 msgr2=0x7fe29415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe29415ab70 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7fe284009870 tx=0x7fe28402c460 comp rx=0 tx=0).stop 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 -- 172.21.15.28:0/4010896325 shutdown_connections 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe294159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 --2- 172.21.15.28:0/4010896325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe29415ab70 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 -- 172.21.15.28:0/4010896325 >> 172.21.15.28:0/4010896325 conn(0x7fe294069e80 msgr2=0x7fe29406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 -- 172.21.15.28:0/4010896325 shutdown_connections 2024-09-13T17:19:32.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.698+0000 7fe298c00640 1 -- 172.21.15.28:0/4010896325 wait complete. 2024-09-13T17:19:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.700+0000 7fe298c00640 1 Processor -- start 2024-09-13T17:19:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.700+0000 7fe298c00640 1 -- start start 2024-09-13T17:19:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29406c2d0 0x7fe2940cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe2940cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe294151a10 con 0x7fe2941592a0 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940c35f0 con 0x7fe29406c2d0 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe298c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2940cbec0 con 0x7fe29406f970 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44840/0 (socket says 172.21.15.28:44840) 2024-09-13T17:19:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 -- 172.21.15.28:0/4053378353 learned_addr learned my addr 172.21.15.28:0/4053378353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe291400640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe2940cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe291e00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29406c2d0 0x7fe2940cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 -- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 msgr2=0x7fe2940cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe2940cb980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 -- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29406c2d0 msgr2=0x7fe2940cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29406c2d0 0x7fe2940cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.701+0000 7fe292800640 1 -- 172.21.15.28:0/4053378353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2840094d0 con 0x7fe2941592a0 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe292800640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c7d0 tx=0x7fe28c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003b70 con 0x7fe2941592a0 2024-09-13T17:19:32.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe28c003cf0 con 0x7fe2941592a0 2024-09-13T17:19:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe28c00aa90 con 0x7fe2941592a0 2024-09-13T17:19:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2941f2480 con 0x7fe2941592a0 2024-09-13T17:19:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.702+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2941f2d20 con 0x7fe2941592a0 2024-09-13T17:19:32.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.703+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe294151b80 con 0x7fe2941592a0 2024-09-13T17:19:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.704+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe28c00abf0 con 0x7fe2941592a0 2024-09-13T17:19:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.705+0000 7fe28a000640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 0x7fe27406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.705+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe28c089210 con 0x7fe2941592a0 2024-09-13T17:19:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.705+0000 7fe291e00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 0x7fe27406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.706+0000 7fe291e00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 0x7fe27406d0c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe284035000 tx=0x7fe284008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:32.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.709+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe294151b80 con 0x7fe2941592a0 2024-09-13T17:19:32.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.889+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe29415a890 con 0x7fe27406ac00 2024-09-13T17:19:32.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.897+0000 7fe28a000640 1 -- 172.21.15.28:0/4053378353 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278273 (secure 0 0 0) 0x7fe2840317e0 con 0x7fe27406ac00 2024-09-13T17:19:32.902 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:32.903 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 msgr2=0x7fe27406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 0x7fe27406d0c0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe284035000 tx=0x7fe284008030 comp rx=0 tx=0).stop 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 msgr2=0x7fe2940c9f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7fe28c00c7d0 tx=0x7fe28c00cc90 comp rx=0 tx=0).stop 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 shutdown_connections 2024-09-13T17:19:32.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe27406ac00 0x7fe27406d0c0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2941592a0 0x7fe2940c9f40 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe29406f970 0x7fe2940cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 --2- 172.21.15.28:0/4053378353 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29406c2d0 0x7fe2940cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 >> 172.21.15.28:0/4053378353 conn(0x7fe294069e80 msgr2=0x7fe2940b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 shutdown_connections 2024-09-13T17:19:32.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:32.901+0000 7fe298c00640 1 -- 172.21.15.28:0/4053378353 wait complete. 2024-09-13T17:19:32.919 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":507,"stamp":"2024-09-13T17:19:31.057236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":534872,"kb_used_data":17192,"kb_used_omap":12,"kb_used_meta":517555,"kb_avail":754439848,"statfs":{"total":773094113280,"available":772546404352,"internally_reserved":0,"allocated":17604608,"data_stored":8393808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":529976889},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000946"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.279713+0000","last_change":"2024-09-13T17:19:27.279713+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.279713+0000","last_undegraded":"2024-09-13T17:19:27.279713+0000","last_fullsized":"2024-09-13T17:19:27.279713+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.743250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.273852+0000","last_change":"2024-09-13T17:19:27.273852+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.273852+0000","last_undegraded":"2024-09-13T17:19:27.273852+0000","last_fullsized":"2024-09-13T17:19:27.273852+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:54.257710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.276511+0000","last_change":"2024-09-13T17:19:27.276511+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.276511+0000","last_undegraded":"2024-09-13T17:19:27.276511+0000","last_fullsized":"2024-09-13T17:19:27.276511+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:48.531947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.276885+0000","last_change":"2024-09-13T17:19:27.276885+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.276885+0000","last_undegraded":"2024-09-13T17:19:27.276885+0000","last_fullsized":"2024-09-13T17:19:27.276885+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:02.252414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.274982+0000","last_change":"2024-09-13T17:19:27.274982+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.274982+0000","last_undegraded":"2024-09-13T17:19:27.274982+0000","last_fullsized":"2024-09-13T17:19:27.274982+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277958+0000","last_change":"2024-09-13T17:19:27.277958+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277958+0000","last_undegraded":"2024-09-13T17:19:27.277958+0000","last_fullsized":"2024-09-13T17:19:27.277958+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:21.639686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277889+0000","last_change":"2024-09-13T17:19:27.277889+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277889+0000","last_undegraded":"2024-09-13T17:19:27.277889+0000","last_fullsized":"2024-09-13T17:19:27.277889+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.273133+0000","last_change":"2024-09-13T17:19:27.273133+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.273133+0000","last_undegraded":"2024-09-13T17:19:27.273133+0000","last_fullsized":"2024-09-13T17:19:27.273133+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:47.594797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.275548+0000","last_change":"2024-09-13T17:19:27.275548+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.275548+0000","last_undegraded":"2024-09-13T17:19:27.275548+0000","last_fullsized":"2024-09-13T17:19:27.275548+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:44.360094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.275438+0000","last_change":"2024-09-13T17:19:27.275438+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.275438+0000","last_undegraded":"2024-09-13T17:19:27.275438+0000","last_fullsized":"2024-09-13T17:19:27.275438+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:50.986775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.274632+0000","last_change":"2024-09-13T17:19:27.274632+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.274632+0000","last_undegraded":"2024-09-13T17:19:27.274632+0000","last_fullsized":"2024-09-13T17:19:27.274632+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:41.661263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277151+0000","last_change":"2024-09-13T17:19:27.277151+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277151+0000","last_undegraded":"2024-09-13T17:19:27.277151+0000","last_fullsized":"2024-09-13T17:19:27.277151+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277472+0000","last_change":"2024-09-13T17:19:27.277472+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277472+0000","last_undegraded":"2024-09-13T17:19:27.277472+0000","last_fullsized":"2024-09-13T17:19:27.277472+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:47.633546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277097+0000","last_change":"2024-09-13T17:19:27.277097+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277097+0000","last_undegraded":"2024-09-13T17:19:27.277097+0000","last_fullsized":"2024-09-13T17:19:27.277097+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:18.253231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.275299+0000","last_change":"2024-09-13T17:19:27.275299+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.275299+0000","last_undegraded":"2024-09-13T17:19:27.275299+0000","last_fullsized":"2024-09-13T17:19:27.275299+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:11.092897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.275551+0000","last_change":"2024-09-13T17:19:27.275551+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.275551+0000","last_undegraded":"2024-09-13T17:19:27.275551+0000","last_fullsized":"2024-09-13T17:19:27.275551+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:18.253231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.276210+0000","last_change":"2024-09-13T17:19:27.276210+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.276210+0000","last_undegraded":"2024-09-13T17:19:27.276210+0000","last_fullsized":"2024-09-13T17:19:27.276210+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.529755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.279579+0000","last_change":"2024-09-13T17:19:27.279579+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.279579+0000","last_undegraded":"2024-09-13T17:19:27.279579+0000","last_fullsized":"2024-09-13T17:19:27.279579+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:34.008890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.279771+0000","last_change":"2024-09-13T17:19:27.279771+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.279771+0000","last_undegraded":"2024-09-13T17:19:27.279771+0000","last_fullsized":"2024-09-13T17:19:27.279771+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:33.813305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.276431+0000","last_change":"2024-09-13T17:19:27.276431+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.276431+0000","last_undegraded":"2024-09-13T17:19:27.276431+0000","last_fullsized":"2024-09-13T17:19:27.276431+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:10.695940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.279589+0000","last_change":"2024-09-13T17:19:27.279589+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.279589+0000","last_undegraded":"2024-09-13T17:19:27.279589+0000","last_fullsized":"2024-09-13T17:19:27.279589+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:11.092897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.276498+0000","last_change":"2024-09-13T17:19:27.276498+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.276498+0000","last_undegraded":"2024-09-13T17:19:27.276498+0000","last_fullsized":"2024-09-13T17:19:27.276498+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.279567+0000","last_change":"2024-09-13T17:19:27.279567+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.279567+0000","last_undegraded":"2024-09-13T17:19:27.279567+0000","last_fullsized":"2024-09-13T17:19:27.279567+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:41.661263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.278046+0000","last_change":"2024-09-13T17:19:27.278046+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.278046+0000","last_undegraded":"2024-09-13T17:19:27.278046+0000","last_fullsized":"2024-09-13T17:19:27.278046+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:45.379894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.274999+0000","last_change":"2024-09-13T17:19:27.274999+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.274999+0000","last_undegraded":"2024-09-13T17:19:27.274999+0000","last_fullsized":"2024-09-13T17:19:27.274999+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.278603+0000","last_change":"2024-09-13T17:19:27.278603+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.278603+0000","last_undegraded":"2024-09-13T17:19:27.278603+0000","last_fullsized":"2024-09-13T17:19:27.278603+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277431+0000","last_change":"2024-09-13T17:19:27.277431+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277431+0000","last_undegraded":"2024-09-13T17:19:27.277431+0000","last_fullsized":"2024-09-13T17:19:27.277431+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:31:37.915597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.274016+0000","last_change":"2024-09-13T17:19:27.274016+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.274016+0000","last_undegraded":"2024-09-13T17:19:27.274016+0000","last_fullsized":"2024-09-13T17:19:27.274016+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:33.813305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277489+0000","last_change":"2024-09-13T17:19:27.277489+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277489+0000","last_undegraded":"2024-09-13T17:19:27.277489+0000","last_fullsized":"2024-09-13T17:19:27.277489+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.529755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.277184+0000","last_change":"2024-09-13T17:19:27.277184+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.277184+0000","last_undegraded":"2024-09-13T17:19:27.277184+0000","last_fullsized":"2024-09-13T17:19:27.277184+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:12.974081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.273984+0000","last_change":"2024-09-13T17:19:27.273984+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.273984+0000","last_undegraded":"2024-09-13T17:19:27.273984+0000","last_fullsized":"2024-09-13T17:19:27.273984+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:37.033265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T17:19:28.270223+0000","last_change":"2024-09-13T17:19:28.270223+0000","last_active":"2024-09-13T17:19:28.270223+0000","last_peered":"2024-09-13T17:19:28.270223+0000","last_clean":"2024-09-13T17:19:28.270223+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:28.270223+0000","last_undegraded":"2024-09-13T17:19:28.270223+0000","last_fullsized":"2024-09-13T17:19:28.270223+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1,"reported_epoch":164,"state":"creating+peering","last_fresh":"2024-09-13T17:19:27.273837+0000","last_change":"2024-09-13T17:19:27.273837+0000","last_active":"2024-09-13T17:19:27.264095+0000","last_peered":"2024-09-13T17:19:27.264095+0000","last_clean":"2024-09-13T17:19:27.264095+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T17:19:27.273837+0000","last_undegraded":"2024-09-13T17:19:27.273837+0000","last_fullsized":"2024-09-13T17:19:27.273837+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.743250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":284,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-09-13T17:19:24.643896+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:19:24.643896+0000","last_peered":"2024-09-13T17:19:24.643896+0000","last_clean":"2024-09-13T17:19:24.643896+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:19:24.643896+0000","last_undegraded":"2024-09-13T17:19:24.643896+0000","last_fullsized":"2024-09-13T17:19:24.643896+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574989,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77928,"kb_used_data":2588,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94293912,"statfs":{"total":96636764160,"available":96556965888,"internally_reserved":0,"allocated":2650112,"data_stored":1493711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77134283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":0.996,"15min":2.0880000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":1.004,"15min":2.6739999999999999},"last":0.88400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64700000000000002,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84699999999999998,"5min":1.127,"15min":2.5030000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64300000000000002,"15min":0.70299999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84299999999999997,"5min":0.92000000000000004,"15min":6.4909999999999997},"last":0.80800000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":1.169,"15min":1.544},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.69999999999999996},"min":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.95499999999999996,"15min":1.605},"last":0.85199999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.091,"5min":1.091,"15min":2.4249999999999998},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69499999999999995,"15min":0.73799999999999999},"min":{"1min":0.53500000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.875,"5min":1.274,"15min":2.8239999999999998},"last":0.79000000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.627},"min":{"1min":0.374,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78600000000000003,"5min":0.90400000000000003,"15min":1.054},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.67500000000000004},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.35399999999999998},"max":{"1min":1.04,"5min":1.234,"15min":1.234},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.67100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.88100000000000001,"5min":0.99299999999999999,"15min":1.143},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72199999999999998,"15min":0.75800000000000001},"min":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.82899999999999996,"5min":1.1659999999999999,"15min":1.337},"last":0.69399999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70799999999999996,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.93100000000000005,"5min":1.048,"15min":1.1930000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.52000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85699999999999998,"5min":1.101,"15min":1.2030000000000001},"last":0.75900000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574990,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59364,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312476,"statfs":{"total":96636764160,"available":96575975424,"internally_reserved":0,"allocated":2056192,"data_stored":903343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":1.298,"15min":3.6629999999999998},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.749,"15min":0.73799999999999999},"min":{"1min":0.52000000000000002,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":0.79400000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.71799999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63100000000000001,"15min":0.65300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.745,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81200000000000006,"5min":1.46,"15min":3.3079999999999998},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.71099999999999997,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83599999999999997,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.63},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66100000000000003,"15min":0.68799999999999994},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81999999999999995,"5min":0.96899999999999997,"15min":3.2629999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95199999999999996,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.49099999999999999,"5min":0.46200000000000002,"15min":0.46000000000000002},"max":{"1min":0.73299999999999998,"5min":3.6720000000000002,"15min":3.742},"last":0.60899999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.47899999999999998,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.628,"15min":0.65000000000000002},"min":{"1min":0.51100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.88300000000000001,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.70199999999999996},"min":{"1min":0.44900000000000001,"5min":0.374,"15min":0.374},"max":{"1min":1.0509999999999999,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86299999999999999,"5min":0.92100000000000004,"15min":3.609},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71699999999999997,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.378,"15min":0.378},"max":{"1min":0.93200000000000005,"5min":1.601,"15min":4.9669999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76500000000000001,"15min":0.79300000000000004},"min":{"1min":0.61399999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0940000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.86899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55256,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316584,"statfs":{"total":96636764160,"available":96580182016,"internally_reserved":0,"allocated":2043904,"data_stored":897526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.65000000000000002,"15min":0.77100000000000002},"min":{"1min":0.48999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4390000000000001,"5min":1.498,"15min":38.005000000000003},"last":0.75},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.747},"min":{"1min":0.54700000000000004,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":21.376999999999999},"last":0.47899999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.65600000000000003,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0149999999999999,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68100000000000005,"15min":0.69799999999999995},"min":{"1min":0.49399999999999999,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":1.482,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.70799999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.65900000000000003,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":1.524,"15min":17.268000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69499999999999995,"15min":0.73899999999999999},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0649999999999999,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.55500000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72999999999999998,"15min":1.095},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.91100000000000003,"5min":2.8690000000000002,"15min":113.363},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.753,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.52900000000000003,"5min":0.433,"15min":0.433},"max":{"1min":1.113,"5min":2.359,"15min":4.1180000000000003},"last":0.66500000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.628,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5669999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.69799999999999995},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.03,"5min":2.827,"15min":2.827},"last":0.54300000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.68200000000000005,"15min":0.69999999999999996},"min":{"1min":0.46999999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.73,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71299999999999997,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.532,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.73899999999999999,"15min":0.75700000000000001},"min":{"1min":0.53000000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0700000000000001,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.74199999999999999,"15min":0.76200000000000001},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.9379999999999999,"15min":2.02},"last":0.60199999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574988,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65064,"kb_used_data":1996,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306776,"statfs":{"total":96636764160,"available":96570138624,"internally_reserved":0,"allocated":2043904,"data_stored":897526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.0620000000000001},"last":0.745},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70099999999999996,"15min":0.72099999999999997},"min":{"1min":0.46000000000000002,"5min":0.40699999999999997,"15min":0.373},"max":{"1min":0.99099999999999999,"5min":1.3400000000000001,"15min":1.341},"last":0.78800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.874,"5min":1.173,"15min":1.173},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88200000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.46200000000000002,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88800000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69599999999999995,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0409999999999999,"5min":1.22,"15min":2.1259999999999999},"last":0.72799999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.44500000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.69999999999999996,"15min":0.69899999999999995},"min":{"1min":0.47199999999999998,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":1.304,"15min":1.304},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.073,"5min":1.284,"15min":1.284},"last":0.503},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64300000000000002,"15min":0.65300000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.96499999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.46100000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.67000000000000004,"15min":0.66600000000000004},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.96599999999999997,"5min":1.274,"15min":1.274},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.292,"15min":1.413},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":0.92900000000000005,"5min":1.252,"15min":1.252},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.754,"15min":0.73999999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.371},"max":{"1min":1.0109999999999999,"5min":1.351,"15min":1.351},"last":0.61799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574989,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65060,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306780,"statfs":{"total":96636764160,"available":96570142720,"internally_reserved":0,"allocated":2056192,"data_stored":903343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39900000000000002,"15min":0.439},"min":{"1min":0.27700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":1.9810000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.68300000000000005,"15min":2.96},"last":0.41099999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49399999999999999,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54200000000000004,"15min":0.64200000000000002},"min":{"1min":0.45300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66700000000000004,"5min":0.77600000000000002,"15min":3.6829999999999998},"last":0.51900000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84999999999999998,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52300000000000002,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":1.3819999999999999},"last":0.65500000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.76300000000000001},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.90800000000000003,"5min":1.1040000000000001,"15min":2.8610000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.51500000000000001,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.99299999999999999,"5min":1.012,"15min":1.3640000000000001},"last":0.72199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71999999999999997,"15min":0.76600000000000001},"min":{"1min":0.58299999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89000000000000001,"5min":1.032,"15min":1.391},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71999999999999997,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.0069999999999999,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":0.77500000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.746,"15min":0.76300000000000001},"min":{"1min":0.51100000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.997,"5min":1.0840000000000001,"15min":1.401},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74099999999999999,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0209999999999999,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":0.76000000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.747,"15min":0.80900000000000005},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":2.899},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.80500000000000005},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0920000000000001,"15min":1.4359999999999999},"last":0.79200000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574989,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":2004,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":2052096,"data_stored":902324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.41699999999999998},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51400000000000001,"5min":1.097,"15min":1.097},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.78600000000000003,"15min":0.56200000000000006},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":30.023,"15min":30.023},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48499999999999999,"15min":1.347},"min":{"1min":0.378,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.73699999999999999,"5min":1.046,"15min":260.53399999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.73799999999999999,"15min":1.45},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":21.279,"15min":262.61399999999998},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.81499999999999995,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.29099999999999998},"max":{"1min":1.016,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.443},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.42199999999999999},"max":{"1min":1.1539999999999999,"5min":1.1559999999999999,"15min":1.214},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.84999999999999998,"15min":0.79400000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95299999999999996,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.96599999999999997,"15min":0.84599999999999997},"min":{"1min":0.53300000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.085,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.012}]},{"osd":5,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.048,"15min":0.876},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":1.121,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.76700000000000002},"min":{"1min":0.59099999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.042,"5min":1.1279999999999999,"15min":1.569},"last":0.91300000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.85999999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.018,"5min":10.664,"15min":10.664},"last":1.077},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.056,"15min":0.88800000000000001},"min":{"1min":0.52100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2210000000000001,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.85199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.87,"15min":0.81599999999999995},"min":{"1min":0.54500000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.194,"5min":10.712999999999999,"15min":10.712999999999999},"last":1.048},{"interface":"front","average":{"1min":0.747,"5min":1.079,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.46100000000000002},"max":{"1min":1.0469999999999999,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.90100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574988,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87204,"kb_used_data":2008,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284636,"statfs":{"total":96636764160,"available":96547467264,"internally_reserved":0,"allocated":2056192,"data_stored":903343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.47799999999999998},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55900000000000005,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48799999999999999,"15min":0.51600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59699999999999998,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.38900000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42999999999999999,"15min":0.46100000000000002},"min":{"1min":0.30399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54600000000000004,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.372},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.53600000000000003},"min":{"1min":0.378,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.66700000000000004},"min":{"1min":0.44600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.79200000000000004,"15min":16.382999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52200000000000002,"15min":0.60299999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69699999999999995,"5min":0.79500000000000004,"15min":18.318000000000001},"last":0.55100000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.753,"15min":0.79400000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92600000000000005,"5min":1.165,"15min":6.1630000000000003},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49399999999999999},"max":{"1min":1.087,"5min":1.149,"15min":1.78},"last":0.74399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.747,"15min":0.78800000000000003},"min":{"1min":0.48499999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0309999999999999,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.67300000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.753,"15min":0.78900000000000003},"min":{"1min":0.55600000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.624},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.80700000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1140000000000001,"5min":1.204,"15min":6.1970000000000001},"last":0.77200000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76600000000000001,"15min":0.80200000000000005},"min":{"1min":0.54700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98199999999999998,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.6359999999999999},"last":0.65500000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574990,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65636,"kb_used_data":2584,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306204,"statfs":{"total":96636764160,"available":96569552896,"internally_reserved":0,"allocated":2646016,"data_stored":1492692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45800000000000002,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.986,"5min":1.986,"15min":5.5259999999999998},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51600000000000001,"15min":0.58599999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":3.004,"15min":5.4889999999999999},"last":0.87}]},{"osd":2,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.621},"min":{"1min":0.34599999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.69899999999999995,"5min":3.605,"15min":6.0430000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.62},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":1.29,"15min":4.6760000000000002},"last":0.92500000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51100000000000001,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.73299999999999998,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58999999999999997,"15min":0.65000000000000002},"min":{"1min":0.42599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69399999999999995,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.94899999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76200000000000001,"15min":0.82899999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.443},"max":{"1min":1.022,"5min":1.9490000000000001,"15min":4.734},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76800000000000002,"15min":0.83699999999999997},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0369999999999999,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.93799999999999994}]},{"osd":5,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.84999999999999998},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.99099999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.84899999999999998},"min":{"1min":0.56999999999999995,"5min":0.432,"15min":0.432},"max":{"1min":0.99399999999999999,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":1.006}]},{"osd":6,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.92200000000000004},"min":{"1min":0.57599999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94199999999999995,"5min":1.161,"15min":4.8470000000000004},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.86199999999999999},"min":{"1min":0.64500000000000002,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":0.995,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":1.014}]},{"osd":7,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81899999999999995,"15min":0.88800000000000001},"min":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.47399999999999998},"max":{"1min":2.044,"5min":2.044,"15min":5.0540000000000003},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82499999999999996,"15min":0.88100000000000001},"min":{"1min":0.59499999999999997,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.1220000000000001,"5min":2.706,"15min":4.9950000000000001},"last":1.081}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:19:32.925 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:19:33.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.112+0000 7f9b8ae00640 1 Processor -- start 2024-09-13T17:19:33.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.113+0000 7f9b8ae00640 1 -- start start 2024-09-13T17:19:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b8ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b84154a90 0x7f9b8415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b8ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8415bb10 con 0x7f9b8406b360 2024-09-13T17:19:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b8ae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8406b900 con 0x7f9b840b0b50 2024-09-13T17:19:33.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b8ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b8406bbe0 con 0x7f9b84154a90 2024-09-13T17:19:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b89a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9b84154a90 0x7f9b8415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b89000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9b8406b360 0x7f9b840b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43076/0 (socket says 172.21.15.28:43076) 2024-09-13T17:19:33.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.114+0000 7f9b89000640 1 -- 172.21.15.28:0/687291432 learned_addr learned my addr 172.21.15.28:0/687291432 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.115+0000 7f9b89a00640 1 -- 172.21.15.28:0/687291432 >> v1:172.21.15.189:6789/0 conn(0x7f9b840b0b50 legacy=0x7f9b840af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.115+0000 7f9b89a00640 1 -- 172.21.15.28:0/687291432 >> v1:172.21.15.28:6789/0 conn(0x7f9b8406b360 legacy=0x7f9b840b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.115+0000 7f9b89a00640 1 -- 172.21.15.28:0/687291432 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8406bec0 con 0x7f9b84154a90 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.115+0000 7f9b83e00640 1 -- 172.21.15.28:0/687291432 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790400674 0 0) 0x7f9b8415bb10 con 0x7f9b8406b360 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b89a00640 1 --2- 172.21.15.28:0/687291432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b8415b5d0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f9b780097a0 tx=0x7f9b7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e0be507c5b978f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b83e00640 1 -- 172.21.15.28:0/687291432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b78032440 con 0x7f9b84154a90 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b83e00640 1 -- 172.21.15.28:0/687291432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b780325a0 con 0x7f9b84154a90 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b83e00640 1 -- 172.21.15.28:0/687291432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b78032930 con 0x7f9b84154a90 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b8ae00640 1 -- 172.21.15.28:0/687291432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 msgr2=0x7f9b8415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.116+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/687291432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b8415b5d0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f9b780097a0 tx=0x7f9b7802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.117+0000 7f9b8ae00640 1 -- 172.21.15.28:0/687291432 shutdown_connections 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.117+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/687291432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b8415b5d0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.117+0000 7f9b8ae00640 1 -- 172.21.15.28:0/687291432 >> 172.21.15.28:0/687291432 conn(0x7f9b84072e40 msgr2=0x7f9b84073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.117+0000 7f9b8ae00640 1 -- 172.21.15.28:0/687291432 shutdown_connections 2024-09-13T17:19:33.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.117+0000 7f9b8ae00640 1 -- 172.21.15.28:0/687291432 wait complete. 2024-09-13T17:19:33.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.118+0000 7f9b8ae00640 1 Processor -- start 2024-09-13T17:19:33.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 -- start start 2024-09-13T17:19:33.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8406b360 0x7f9b840c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b840b0b50 0x7f9b840ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b840c50d0 con 0x7f9b84154a90 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b841307f0 con 0x7f9b8406b360 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.119+0000 7f9b8ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8415a6e0 con 0x7f9b840b0b50 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44856/0 (socket says 172.21.15.28:44856) 2024-09-13T17:19:33.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 -- 172.21.15.28:0/825951931 learned_addr learned my addr 172.21.15.28:0/825951931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b89a00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8406b360 0x7f9b840c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b89000640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b840b0b50 0x7f9b840ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 -- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b840b0b50 msgr2=0x7f9b840ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b840b0b50 0x7f9b840ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 -- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8406b360 msgr2=0x7f9b840c49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8406b360 0x7f9b840c49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.120+0000 7f9b8a400640 1 -- 172.21.15.28:0/825951931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78009400 con 0x7f9b84154a90 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b8a400640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00c780 tx=0x7f9b7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c004b10 con 0x7f9b84154a90 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7c004c70 con 0x7f9b84154a90 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b7c00aa90 con 0x7f9b84154a90 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8412fec0 con 0x7f9b84154a90 2024-09-13T17:19:33.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.121+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b84130480 con 0x7f9b84154a90 2024-09-13T17:19:33.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.122+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b8415b5d0 con 0x7f9b84154a90 2024-09-13T17:19:33.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.123+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9b7c00abf0 con 0x7f9b84154a90 2024-09-13T17:19:33.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.124+0000 7f9b81600640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 0x7f9b6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.124+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b7c089430 con 0x7f9b84154a90 2024-09-13T17:19:33.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.124+0000 7f9b89a00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 0x7f9b6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.125+0000 7f9b89a00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 0x7f9b6406d0c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9b78034000 tx=0x7f9b7802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.128+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9b7c054620 con 0x7f9b84154a90 2024-09-13T17:19:33.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.303+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9b84154510 con 0x7f9b6406ac00 2024-09-13T17:19:33.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.311+0000 7f9b81600640 1 -- 172.21.15.28:0/825951931 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289201 (secure 0 0 0) 0x7f9b84154510 con 0x7f9b6406ac00 2024-09-13T17:19:33.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:33.318 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:19:33.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 msgr2=0x7f9b6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 0x7f9b6406d0c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9b78034000 tx=0x7f9b7802e000 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 msgr2=0x7f9b84131dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c00c780 tx=0x7f9b7c00cc40 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 shutdown_connections 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6406ac00 0x7f9b6406d0c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b84154a90 0x7f9b84131dd0 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b840b0b50 0x7f9b840ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 --2- 172.21.15.28:0/825951931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8406b360 0x7f9b840c49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 >> 172.21.15.28:0/825951931 conn(0x7f9b84072e40 msgr2=0x7f9b840cd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 shutdown_connections 2024-09-13T17:19:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.315+0000 7f9b8ae00640 1 -- 172.21.15.28:0/825951931 wait complete. 2024-09-13T17:19:33.333 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-13T17:19:33.057706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":535252,"kb_used_data":17568,"kb_used_omap":12,"kb_used_meta":517555,"kb_avail":754439468,"statfs":{"total":773094113280,"available":772546015232,"internally_reserved":0,"allocated":17989632,"data_stored":8564966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":529976889},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.785982"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.729500+0000","last_change":"2024-09-13T17:19:28.748975+0000","last_active":"2024-09-13T17:19:29.729500+0000","last_peered":"2024-09-13T17:19:29.729500+0000","last_clean":"2024-09-13T17:19:29.729500+0000","last_became_active":"2024-09-13T17:19:28.748486+0000","last_became_peered":"2024-09-13T17:19:28.748486+0000","last_unstale":"2024-09-13T17:19:29.729500+0000","last_undegraded":"2024-09-13T17:19:29.729500+0000","last_fullsized":"2024-09-13T17:19:29.729500+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:23:15.850934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140940+0000","last_change":"2024-09-13T17:19:28.739480+0000","last_active":"2024-09-13T17:19:30.140940+0000","last_peered":"2024-09-13T17:19:30.140940+0000","last_clean":"2024-09-13T17:19:30.140940+0000","last_became_active":"2024-09-13T17:19:28.739229+0000","last_became_peered":"2024-09-13T17:19:28.739229+0000","last_unstale":"2024-09-13T17:19:30.140940+0000","last_undegraded":"2024-09-13T17:19:30.140940+0000","last_fullsized":"2024-09-13T17:19:30.140940+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:36:49.161520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728546+0000","last_change":"2024-09-13T17:19:28.746043+0000","last_active":"2024-09-13T17:19:29.728546+0000","last_peered":"2024-09-13T17:19:29.728546+0000","last_clean":"2024-09-13T17:19:29.728546+0000","last_became_active":"2024-09-13T17:19:28.740420+0000","last_became_peered":"2024-09-13T17:19:28.740420+0000","last_unstale":"2024-09-13T17:19:29.728546+0000","last_undegraded":"2024-09-13T17:19:29.728546+0000","last_fullsized":"2024-09-13T17:19:29.728546+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:12.577060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725802+0000","last_change":"2024-09-13T17:19:28.735154+0000","last_active":"2024-09-13T17:19:29.725802+0000","last_peered":"2024-09-13T17:19:29.725802+0000","last_clean":"2024-09-13T17:19:29.725802+0000","last_became_active":"2024-09-13T17:19:28.734640+0000","last_became_peered":"2024-09-13T17:19:28.734640+0000","last_unstale":"2024-09-13T17:19:29.725802+0000","last_undegraded":"2024-09-13T17:19:29.725802+0000","last_fullsized":"2024-09-13T17:19:29.725802+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:14.570067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140960+0000","last_change":"2024-09-13T17:19:28.736040+0000","last_active":"2024-09-13T17:19:30.140960+0000","last_peered":"2024-09-13T17:19:30.140960+0000","last_clean":"2024-09-13T17:19:30.140960+0000","last_became_active":"2024-09-13T17:19:28.735734+0000","last_became_peered":"2024-09-13T17:19:28.735734+0000","last_unstale":"2024-09-13T17:19:30.140960+0000","last_undegraded":"2024-09-13T17:19:30.140960+0000","last_fullsized":"2024-09-13T17:19:30.140960+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:06.867792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140347+0000","last_change":"2024-09-13T17:19:28.732324+0000","last_active":"2024-09-13T17:19:30.140347+0000","last_peered":"2024-09-13T17:19:30.140347+0000","last_clean":"2024-09-13T17:19:30.140347+0000","last_became_active":"2024-09-13T17:19:28.732050+0000","last_became_peered":"2024-09-13T17:19:28.732050+0000","last_unstale":"2024-09-13T17:19:30.140347+0000","last_undegraded":"2024-09-13T17:19:30.140347+0000","last_fullsized":"2024-09-13T17:19:30.140347+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:38.345656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855500+0000","last_change":"2024-09-13T17:19:28.742302+0000","last_active":"2024-09-13T17:19:29.855500+0000","last_peered":"2024-09-13T17:19:29.855500+0000","last_clean":"2024-09-13T17:19:29.855500+0000","last_became_active":"2024-09-13T17:19:28.742016+0000","last_became_peered":"2024-09-13T17:19:28.742016+0000","last_unstale":"2024-09-13T17:19:29.855500+0000","last_undegraded":"2024-09-13T17:19:29.855500+0000","last_fullsized":"2024-09-13T17:19:29.855500+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:18.306229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141126+0000","last_change":"2024-09-13T17:19:28.750673+0000","last_active":"2024-09-13T17:19:30.141126+0000","last_peered":"2024-09-13T17:19:30.141126+0000","last_clean":"2024-09-13T17:19:30.141126+0000","last_became_active":"2024-09-13T17:19:28.750395+0000","last_became_peered":"2024-09-13T17:19:28.750395+0000","last_unstale":"2024-09-13T17:19:30.141126+0000","last_undegraded":"2024-09-13T17:19:30.141126+0000","last_fullsized":"2024-09-13T17:19:30.141126+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:22.652493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141164+0000","last_change":"2024-09-13T17:19:28.750888+0000","last_active":"2024-09-13T17:19:30.141164+0000","last_peered":"2024-09-13T17:19:30.141164+0000","last_clean":"2024-09-13T17:19:30.141164+0000","last_became_active":"2024-09-13T17:19:28.749226+0000","last_became_peered":"2024-09-13T17:19:28.749226+0000","last_unstale":"2024-09-13T17:19:30.141164+0000","last_undegraded":"2024-09-13T17:19:30.141164+0000","last_fullsized":"2024-09-13T17:19:30.141164+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:37.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141107+0000","last_change":"2024-09-13T17:19:28.740261+0000","last_active":"2024-09-13T17:19:30.141107+0000","last_peered":"2024-09-13T17:19:30.141107+0000","last_clean":"2024-09-13T17:19:30.141107+0000","last_became_active":"2024-09-13T17:19:28.739996+0000","last_became_peered":"2024-09-13T17:19:28.739996+0000","last_unstale":"2024-09-13T17:19:30.141107+0000","last_undegraded":"2024-09-13T17:19:30.141107+0000","last_fullsized":"2024-09-13T17:19:30.141107+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:12.577060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140944+0000","last_change":"2024-09-13T17:19:28.744034+0000","last_active":"2024-09-13T17:19:30.140944+0000","last_peered":"2024-09-13T17:19:30.140944+0000","last_clean":"2024-09-13T17:19:30.140944+0000","last_became_active":"2024-09-13T17:19:28.743517+0000","last_became_peered":"2024-09-13T17:19:28.743517+0000","last_unstale":"2024-09-13T17:19:30.140944+0000","last_undegraded":"2024-09-13T17:19:30.140944+0000","last_fullsized":"2024-09-13T17:19:30.140944+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:53:37.646562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140783+0000","last_change":"2024-09-13T17:19:28.738545+0000","last_active":"2024-09-13T17:19:30.140783+0000","last_peered":"2024-09-13T17:19:30.140783+0000","last_clean":"2024-09-13T17:19:30.140783+0000","last_became_active":"2024-09-13T17:19:28.738279+0000","last_became_peered":"2024-09-13T17:19:28.738279+0000","last_unstale":"2024-09-13T17:19:30.140783+0000","last_undegraded":"2024-09-13T17:19:30.140783+0000","last_fullsized":"2024-09-13T17:19:30.140783+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:40.368293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728383+0000","last_change":"2024-09-13T17:19:28.739125+0000","last_active":"2024-09-13T17:19:29.728383+0000","last_peered":"2024-09-13T17:19:29.728383+0000","last_clean":"2024-09-13T17:19:29.728383+0000","last_became_active":"2024-09-13T17:19:28.738431+0000","last_became_peered":"2024-09-13T17:19:28.738431+0000","last_unstale":"2024-09-13T17:19:29.728383+0000","last_undegraded":"2024-09-13T17:19:29.728383+0000","last_fullsized":"2024-09-13T17:19:29.728383+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:39.658258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728161+0000","last_change":"2024-09-13T17:19:28.736709+0000","last_active":"2024-09-13T17:19:29.728161+0000","last_peered":"2024-09-13T17:19:29.728161+0000","last_clean":"2024-09-13T17:19:29.728161+0000","last_became_active":"2024-09-13T17:19:28.736390+0000","last_became_peered":"2024-09-13T17:19:28.736390+0000","last_unstale":"2024-09-13T17:19:29.728161+0000","last_undegraded":"2024-09-13T17:19:29.728161+0000","last_fullsized":"2024-09-13T17:19:29.728161+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:40.368293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728973+0000","last_change":"2024-09-13T17:19:28.748211+0000","last_active":"2024-09-13T17:19:29.728973+0000","last_peered":"2024-09-13T17:19:29.728973+0000","last_clean":"2024-09-13T17:19:29.728973+0000","last_became_active":"2024-09-13T17:19:28.747350+0000","last_became_peered":"2024-09-13T17:19:28.747350+0000","last_unstale":"2024-09-13T17:19:29.728973+0000","last_undegraded":"2024-09-13T17:19:29.728973+0000","last_fullsized":"2024-09-13T17:19:29.728973+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:48.600925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855200+0000","last_change":"2024-09-13T17:19:28.743643+0000","last_active":"2024-09-13T17:19:29.855200+0000","last_peered":"2024-09-13T17:19:29.855200+0000","last_clean":"2024-09-13T17:19:29.855200+0000","last_became_active":"2024-09-13T17:19:28.742747+0000","last_became_peered":"2024-09-13T17:19:28.742747+0000","last_unstale":"2024-09-13T17:19:29.855200+0000","last_undegraded":"2024-09-13T17:19:29.855200+0000","last_fullsized":"2024-09-13T17:19:29.855200+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:51.028062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140417+0000","last_change":"2024-09-13T17:19:28.739879+0000","last_active":"2024-09-13T17:19:30.140417+0000","last_peered":"2024-09-13T17:19:30.140417+0000","last_clean":"2024-09-13T17:19:30.140417+0000","last_became_active":"2024-09-13T17:19:28.736307+0000","last_became_peered":"2024-09-13T17:19:28.736307+0000","last_unstale":"2024-09-13T17:19:30.140417+0000","last_undegraded":"2024-09-13T17:19:30.140417+0000","last_fullsized":"2024-09-13T17:19:30.140417+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:21:10.403087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791802+0000","last_change":"2024-09-13T17:19:28.741367+0000","last_active":"2024-09-13T17:19:29.791802+0000","last_peered":"2024-09-13T17:19:29.791802+0000","last_clean":"2024-09-13T17:19:29.791802+0000","last_became_active":"2024-09-13T17:19:28.741125+0000","last_became_peered":"2024-09-13T17:19:28.741125+0000","last_unstale":"2024-09-13T17:19:29.791802+0000","last_undegraded":"2024-09-13T17:19:29.791802+0000","last_fullsized":"2024-09-13T17:19:29.791802+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:54:13.981455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140425+0000","last_change":"2024-09-13T17:19:28.737239+0000","last_active":"2024-09-13T17:19:30.140425+0000","last_peered":"2024-09-13T17:19:30.140425+0000","last_clean":"2024-09-13T17:19:30.140425+0000","last_became_active":"2024-09-13T17:19:28.736092+0000","last_became_peered":"2024-09-13T17:19:28.736092+0000","last_unstale":"2024-09-13T17:19:30.140425+0000","last_undegraded":"2024-09-13T17:19:30.140425+0000","last_fullsized":"2024-09-13T17:19:30.140425+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:17.047375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728498+0000","last_change":"2024-09-13T17:19:28.739850+0000","last_active":"2024-09-13T17:19:29.728498+0000","last_peered":"2024-09-13T17:19:29.728498+0000","last_clean":"2024-09-13T17:19:29.728498+0000","last_became_active":"2024-09-13T17:19:28.739543+0000","last_became_peered":"2024-09-13T17:19:28.739543+0000","last_unstale":"2024-09-13T17:19:29.728498+0000","last_undegraded":"2024-09-13T17:19:29.728498+0000","last_fullsized":"2024-09-13T17:19:29.728498+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:08.213640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140322+0000","last_change":"2024-09-13T17:19:28.737093+0000","last_active":"2024-09-13T17:19:30.140322+0000","last_peered":"2024-09-13T17:19:30.140322+0000","last_clean":"2024-09-13T17:19:30.140322+0000","last_became_active":"2024-09-13T17:19:28.736817+0000","last_became_peered":"2024-09-13T17:19:28.736817+0000","last_unstale":"2024-09-13T17:19:30.140322+0000","last_undegraded":"2024-09-13T17:19:30.140322+0000","last_fullsized":"2024-09-13T17:19:30.140322+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:39:16.822565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140730+0000","last_change":"2024-09-13T17:19:28.734205+0000","last_active":"2024-09-13T17:19:30.140730+0000","last_peered":"2024-09-13T17:19:30.140730+0000","last_clean":"2024-09-13T17:19:30.140730+0000","last_became_active":"2024-09-13T17:19:28.733887+0000","last_became_peered":"2024-09-13T17:19:28.733887+0000","last_unstale":"2024-09-13T17:19:30.140730+0000","last_undegraded":"2024-09-13T17:19:30.140730+0000","last_fullsized":"2024-09-13T17:19:30.140730+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:15:14.570067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140252+0000","last_change":"2024-09-13T17:19:28.733228+0000","last_active":"2024-09-13T17:19:30.140252+0000","last_peered":"2024-09-13T17:19:30.140252+0000","last_clean":"2024-09-13T17:19:30.140252+0000","last_became_active":"2024-09-13T17:19:28.732802+0000","last_became_peered":"2024-09-13T17:19:28.732802+0000","last_unstale":"2024-09-13T17:19:30.140252+0000","last_undegraded":"2024-09-13T17:19:30.140252+0000","last_fullsized":"2024-09-13T17:19:30.140252+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:46:08.676396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140606+0000","last_change":"2024-09-13T17:19:28.735933+0000","last_active":"2024-09-13T17:19:30.140606+0000","last_peered":"2024-09-13T17:19:30.140606+0000","last_clean":"2024-09-13T17:19:30.140606+0000","last_became_active":"2024-09-13T17:19:28.735655+0000","last_became_peered":"2024-09-13T17:19:28.735655+0000","last_unstale":"2024-09-13T17:19:30.140606+0000","last_undegraded":"2024-09-13T17:19:30.140606+0000","last_fullsized":"2024-09-13T17:19:30.140606+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:03.122408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728724+0000","last_change":"2024-09-13T17:19:28.740906+0000","last_active":"2024-09-13T17:19:29.728724+0000","last_peered":"2024-09-13T17:19:29.728724+0000","last_clean":"2024-09-13T17:19:29.728724+0000","last_became_active":"2024-09-13T17:19:28.732458+0000","last_became_peered":"2024-09-13T17:19:28.732458+0000","last_unstale":"2024-09-13T17:19:29.728724+0000","last_undegraded":"2024-09-13T17:19:29.728724+0000","last_fullsized":"2024-09-13T17:19:29.728724+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:37.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855339+0000","last_change":"2024-09-13T17:19:28.747742+0000","last_active":"2024-09-13T17:19:29.855339+0000","last_peered":"2024-09-13T17:19:29.855339+0000","last_clean":"2024-09-13T17:19:29.855339+0000","last_became_active":"2024-09-13T17:19:28.746609+0000","last_became_peered":"2024-09-13T17:19:28.746609+0000","last_unstale":"2024-09-13T17:19:29.855339+0000","last_undegraded":"2024-09-13T17:19:29.855339+0000","last_fullsized":"2024-09-13T17:19:29.855339+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:56.155192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728664+0000","last_change":"2024-09-13T17:19:28.736656+0000","last_active":"2024-09-13T17:19:29.728664+0000","last_peered":"2024-09-13T17:19:29.728664+0000","last_clean":"2024-09-13T17:19:29.728664+0000","last_became_active":"2024-09-13T17:19:28.736379+0000","last_became_peered":"2024-09-13T17:19:28.736379+0000","last_unstale":"2024-09-13T17:19:29.728664+0000","last_undegraded":"2024-09-13T17:19:29.728664+0000","last_fullsized":"2024-09-13T17:19:29.728664+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:51.387588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725891+0000","last_change":"2024-09-13T17:19:28.731489+0000","last_active":"2024-09-13T17:19:29.725891+0000","last_peered":"2024-09-13T17:19:29.725891+0000","last_clean":"2024-09-13T17:19:29.725891+0000","last_became_active":"2024-09-13T17:19:28.731278+0000","last_became_peered":"2024-09-13T17:19:28.731278+0000","last_unstale":"2024-09-13T17:19:29.725891+0000","last_undegraded":"2024-09-13T17:19:29.725891+0000","last_fullsized":"2024-09-13T17:19:29.725891+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:22.652493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728459+0000","last_change":"2024-09-13T17:19:28.742556+0000","last_active":"2024-09-13T17:19:29.728459+0000","last_peered":"2024-09-13T17:19:29.728459+0000","last_clean":"2024-09-13T17:19:29.728459+0000","last_became_active":"2024-09-13T17:19:28.742294+0000","last_became_peered":"2024-09-13T17:19:28.742294+0000","last_unstale":"2024-09-13T17:19:29.728459+0000","last_undegraded":"2024-09-13T17:19:29.728459+0000","last_fullsized":"2024-09-13T17:19:29.728459+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:40.376580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141255+0000","last_change":"2024-09-13T17:19:28.741953+0000","last_active":"2024-09-13T17:19:30.141255+0000","last_peered":"2024-09-13T17:19:30.141255+0000","last_clean":"2024-09-13T17:19:30.141255+0000","last_became_active":"2024-09-13T17:19:28.741680+0000","last_became_peered":"2024-09-13T17:19:28.741680+0000","last_unstale":"2024-09-13T17:19:30.141255+0000","last_undegraded":"2024-09-13T17:19:30.141255+0000","last_fullsized":"2024-09-13T17:19:30.141255+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:42.655997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141027+0000","last_change":"2024-09-13T17:19:28.737180+0000","last_active":"2024-09-13T17:19:30.141027+0000","last_peered":"2024-09-13T17:19:30.141027+0000","last_clean":"2024-09-13T17:19:30.141027+0000","last_became_active":"2024-09-13T17:19:28.735637+0000","last_became_peered":"2024-09-13T17:19:28.735637+0000","last_unstale":"2024-09-13T17:19:30.141027+0000","last_undegraded":"2024-09-13T17:19:30.141027+0000","last_fullsized":"2024-09-13T17:19:30.141027+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:22:28.532456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791455+0000","last_change":"2024-09-13T17:19:28.736966+0000","last_active":"2024-09-13T17:19:29.791455+0000","last_peered":"2024-09-13T17:19:29.791455+0000","last_clean":"2024-09-13T17:19:29.791455+0000","last_became_active":"2024-09-13T17:19:28.736729+0000","last_became_peered":"2024-09-13T17:19:28.736729+0000","last_unstale":"2024-09-13T17:19:29.791455+0000","last_undegraded":"2024-09-13T17:19:29.791455+0000","last_fullsized":"2024-09-13T17:19:29.791455+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:04:42.305887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855385+0000","last_change":"2024-09-13T17:19:28.745525+0000","last_active":"2024-09-13T17:19:29.855385+0000","last_peered":"2024-09-13T17:19:29.855385+0000","last_clean":"2024-09-13T17:19:29.855385+0000","last_became_active":"2024-09-13T17:19:28.745280+0000","last_became_peered":"2024-09-13T17:19:28.745280+0000","last_unstale":"2024-09-13T17:19:29.855385+0000","last_undegraded":"2024-09-13T17:19:29.855385+0000","last_fullsized":"2024-09-13T17:19:29.855385+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:11.757039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"166'4","reported_seq":15,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.751253+0000","last_change":"2024-09-13T17:19:28.299428+0000","last_active":"2024-09-13T17:19:28.751253+0000","last_peered":"2024-09-13T17:19:28.751253+0000","last_clean":"2024-09-13T17:19:28.751253+0000","last_became_active":"2024-09-13T17:19:28.299144+0000","last_became_peered":"2024-09-13T17:19:28.299144+0000","last_unstale":"2024-09-13T17:19:28.751253+0000","last_undegraded":"2024-09-13T17:19:28.751253+0000","last_fullsized":"2024-09-13T17:19:28.751253+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.743250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141728+0000","last_change":"2024-09-13T17:19:28.748193+0000","last_active":"2024-09-13T17:19:30.141728+0000","last_peered":"2024-09-13T17:19:30.141728+0000","last_clean":"2024-09-13T17:19:30.141728+0000","last_became_active":"2024-09-13T17:19:28.744081+0000","last_became_peered":"2024-09-13T17:19:28.744081+0000","last_unstale":"2024-09-13T17:19:30.141728+0000","last_undegraded":"2024-09-13T17:19:30.141728+0000","last_fullsized":"2024-09-13T17:19:30.141728+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:23.214378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"166'1","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.750642+0000","last_change":"2024-09-13T17:19:28.296175+0000","last_active":"2024-09-13T17:19:28.750642+0000","last_peered":"2024-09-13T17:19:28.750642+0000","last_clean":"2024-09-13T17:19:28.750642+0000","last_became_active":"2024-09-13T17:19:28.295897+0000","last_became_peered":"2024-09-13T17:19:28.295897+0000","last_unstale":"2024-09-13T17:19:28.750642+0000","last_undegraded":"2024-09-13T17:19:28.750642+0000","last_fullsized":"2024-09-13T17:19:28.750642+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:54.257710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140493+0000","last_change":"2024-09-13T17:19:28.742470+0000","last_active":"2024-09-13T17:19:30.140493+0000","last_peered":"2024-09-13T17:19:30.140493+0000","last_clean":"2024-09-13T17:19:30.140493+0000","last_became_active":"2024-09-13T17:19:28.742220+0000","last_became_peered":"2024-09-13T17:19:28.742220+0000","last_unstale":"2024-09-13T17:19:30.140493+0000","last_undegraded":"2024-09-13T17:19:30.140493+0000","last_fullsized":"2024-09-13T17:19:30.140493+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:27.560188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.748509+0000","last_change":"2024-09-13T17:19:28.298957+0000","last_active":"2024-09-13T17:19:28.748509+0000","last_peered":"2024-09-13T17:19:28.748509+0000","last_clean":"2024-09-13T17:19:28.748509+0000","last_became_active":"2024-09-13T17:19:28.298510+0000","last_became_peered":"2024-09-13T17:19:28.298510+0000","last_unstale":"2024-09-13T17:19:28.748509+0000","last_undegraded":"2024-09-13T17:19:28.748509+0000","last_fullsized":"2024-09-13T17:19:28.748509+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:48.531947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728792+0000","last_change":"2024-09-13T17:19:28.731784+0000","last_active":"2024-09-13T17:19:29.728792+0000","last_peered":"2024-09-13T17:19:29.728792+0000","last_clean":"2024-09-13T17:19:29.728792+0000","last_became_active":"2024-09-13T17:19:28.731497+0000","last_became_peered":"2024-09-13T17:19:28.731497+0000","last_unstale":"2024-09-13T17:19:29.728792+0000","last_undegraded":"2024-09-13T17:19:29.728792+0000","last_fullsized":"2024-09-13T17:19:29.728792+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:59.970407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.751073+0000","last_change":"2024-09-13T17:19:28.298011+0000","last_active":"2024-09-13T17:19:28.751073+0000","last_peered":"2024-09-13T17:19:28.751073+0000","last_clean":"2024-09-13T17:19:28.751073+0000","last_became_active":"2024-09-13T17:19:28.297528+0000","last_became_peered":"2024-09-13T17:19:28.297528+0000","last_unstale":"2024-09-13T17:19:28.751073+0000","last_undegraded":"2024-09-13T17:19:28.751073+0000","last_fullsized":"2024-09-13T17:19:28.751073+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:35:02.252414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855399+0000","last_change":"2024-09-13T17:19:28.735781+0000","last_active":"2024-09-13T17:19:29.855399+0000","last_peered":"2024-09-13T17:19:29.855399+0000","last_clean":"2024-09-13T17:19:29.855399+0000","last_became_active":"2024-09-13T17:19:28.735559+0000","last_became_peered":"2024-09-13T17:19:28.735559+0000","last_unstale":"2024-09-13T17:19:29.855399+0000","last_undegraded":"2024-09-13T17:19:29.855399+0000","last_fullsized":"2024-09-13T17:19:29.855399+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:00:18.021677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"166'1","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.751765+0000","last_change":"2024-09-13T17:19:28.286041+0000","last_active":"2024-09-13T17:19:28.751765+0000","last_peered":"2024-09-13T17:19:28.751765+0000","last_clean":"2024-09-13T17:19:28.751765+0000","last_became_active":"2024-09-13T17:19:28.285788+0000","last_became_peered":"2024-09-13T17:19:28.285788+0000","last_unstale":"2024-09-13T17:19:28.751765+0000","last_undegraded":"2024-09-13T17:19:28.751765+0000","last_fullsized":"2024-09-13T17:19:28.751765+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725511+0000","last_change":"2024-09-13T17:19:28.739482+0000","last_active":"2024-09-13T17:19:29.725511+0000","last_peered":"2024-09-13T17:19:29.725511+0000","last_clean":"2024-09-13T17:19:29.725511+0000","last_became_active":"2024-09-13T17:19:28.738992+0000","last_became_peered":"2024-09-13T17:19:28.738992+0000","last_unstale":"2024-09-13T17:19:29.725511+0000","last_undegraded":"2024-09-13T17:19:29.725511+0000","last_fullsized":"2024-09-13T17:19:29.725511+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:55:08.213640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"166'1","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.749532+0000","last_change":"2024-09-13T17:19:28.300550+0000","last_active":"2024-09-13T17:19:28.749532+0000","last_peered":"2024-09-13T17:19:28.749532+0000","last_clean":"2024-09-13T17:19:28.749532+0000","last_became_active":"2024-09-13T17:19:28.300036+0000","last_became_peered":"2024-09-13T17:19:28.300036+0000","last_unstale":"2024-09-13T17:19:28.749532+0000","last_undegraded":"2024-09-13T17:19:28.749532+0000","last_fullsized":"2024-09-13T17:19:28.749532+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:21.639686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725306+0000","last_change":"2024-09-13T17:19:28.745191+0000","last_active":"2024-09-13T17:19:29.725306+0000","last_peered":"2024-09-13T17:19:29.725306+0000","last_clean":"2024-09-13T17:19:29.725306+0000","last_became_active":"2024-09-13T17:19:28.743546+0000","last_became_peered":"2024-09-13T17:19:28.743546+0000","last_unstale":"2024-09-13T17:19:29.725306+0000","last_undegraded":"2024-09-13T17:19:29.725306+0000","last_fullsized":"2024-09-13T17:19:29.725306+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:37.938672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.728714+0000","last_change":"2024-09-13T17:19:28.297262+0000","last_active":"2024-09-13T17:19:28.728714+0000","last_peered":"2024-09-13T17:19:28.728714+0000","last_clean":"2024-09-13T17:19:28.728714+0000","last_became_active":"2024-09-13T17:19:28.296994+0000","last_became_peered":"2024-09-13T17:19:28.296994+0000","last_unstale":"2024-09-13T17:19:28.728714+0000","last_undegraded":"2024-09-13T17:19:28.728714+0000","last_fullsized":"2024-09-13T17:19:28.728714+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140292+0000","last_change":"2024-09-13T17:19:28.733069+0000","last_active":"2024-09-13T17:19:30.140292+0000","last_peered":"2024-09-13T17:19:30.140292+0000","last_clean":"2024-09-13T17:19:30.140292+0000","last_became_active":"2024-09-13T17:19:28.732400+0000","last_became_peered":"2024-09-13T17:19:28.732400+0000","last_unstale":"2024-09-13T17:19:30.140292+0000","last_undegraded":"2024-09-13T17:19:30.140292+0000","last_fullsized":"2024-09-13T17:19:30.140292+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:30.699844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.730851+0000","last_change":"2024-09-13T17:19:28.297745+0000","last_active":"2024-09-13T17:19:28.730851+0000","last_peered":"2024-09-13T17:19:28.730851+0000","last_clean":"2024-09-13T17:19:28.730851+0000","last_became_active":"2024-09-13T17:19:28.297327+0000","last_became_peered":"2024-09-13T17:19:28.297327+0000","last_unstale":"2024-09-13T17:19:28.730851+0000","last_undegraded":"2024-09-13T17:19:28.730851+0000","last_fullsized":"2024-09-13T17:19:28.730851+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:47.594797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791778+0000","last_change":"2024-09-13T17:19:28.735519+0000","last_active":"2024-09-13T17:19:29.791778+0000","last_peered":"2024-09-13T17:19:29.791778+0000","last_clean":"2024-09-13T17:19:29.791778+0000","last_became_active":"2024-09-13T17:19:28.735242+0000","last_became_peered":"2024-09-13T17:19:28.735242+0000","last_unstale":"2024-09-13T17:19:29.791778+0000","last_undegraded":"2024-09-13T17:19:29.791778+0000","last_fullsized":"2024-09-13T17:19:29.791778+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:11.223640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.718625+0000","last_change":"2024-09-13T17:19:28.292833+0000","last_active":"2024-09-13T17:19:28.718625+0000","last_peered":"2024-09-13T17:19:28.718625+0000","last_clean":"2024-09-13T17:19:28.718625+0000","last_became_active":"2024-09-13T17:19:28.292524+0000","last_became_peered":"2024-09-13T17:19:28.292524+0000","last_unstale":"2024-09-13T17:19:28.718625+0000","last_undegraded":"2024-09-13T17:19:28.718625+0000","last_fullsized":"2024-09-13T17:19:28.718625+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:44.360094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791690+0000","last_change":"2024-09-13T17:19:28.736280+0000","last_active":"2024-09-13T17:19:29.791690+0000","last_peered":"2024-09-13T17:19:29.791690+0000","last_clean":"2024-09-13T17:19:29.791690+0000","last_became_active":"2024-09-13T17:19:28.736040+0000","last_became_peered":"2024-09-13T17:19:28.736040+0000","last_unstale":"2024-09-13T17:19:29.791690+0000","last_undegraded":"2024-09-13T17:19:29.791690+0000","last_fullsized":"2024-09-13T17:19:29.791690+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:04.459243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"166'3","reported_seq":14,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.745536+0000","last_change":"2024-09-13T17:19:28.296261+0000","last_active":"2024-09-13T17:19:28.745536+0000","last_peered":"2024-09-13T17:19:28.745536+0000","last_clean":"2024-09-13T17:19:28.745536+0000","last_became_active":"2024-09-13T17:19:28.295238+0000","last_became_peered":"2024-09-13T17:19:28.295238+0000","last_unstale":"2024-09-13T17:19:28.745536+0000","last_undegraded":"2024-09-13T17:19:28.745536+0000","last_fullsized":"2024-09-13T17:19:28.745536+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:50.986775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141659+0000","last_change":"2024-09-13T17:19:28.738985+0000","last_active":"2024-09-13T17:19:30.141659+0000","last_peered":"2024-09-13T17:19:30.141659+0000","last_clean":"2024-09-13T17:19:30.141659+0000","last_became_active":"2024-09-13T17:19:28.738693+0000","last_became_peered":"2024-09-13T17:19:28.738693+0000","last_unstale":"2024-09-13T17:19:30.141659+0000","last_undegraded":"2024-09-13T17:19:30.141659+0000","last_fullsized":"2024-09-13T17:19:30.141659+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:35.034599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"166'6","reported_seq":17,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.754498+0000","last_change":"2024-09-13T17:19:28.289921+0000","last_active":"2024-09-13T17:19:28.754498+0000","last_peered":"2024-09-13T17:19:28.754498+0000","last_clean":"2024-09-13T17:19:28.754498+0000","last_became_active":"2024-09-13T17:19:28.289629+0000","last_became_peered":"2024-09-13T17:19:28.289629+0000","last_unstale":"2024-09-13T17:19:28.754498+0000","last_undegraded":"2024-09-13T17:19:28.754498+0000","last_fullsized":"2024-09-13T17:19:28.754498+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:41.661263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791576+0000","last_change":"2024-09-13T17:19:28.734756+0000","last_active":"2024-09-13T17:19:29.791576+0000","last_peered":"2024-09-13T17:19:29.791576+0000","last_clean":"2024-09-13T17:19:29.791576+0000","last_became_active":"2024-09-13T17:19:28.734494+0000","last_became_peered":"2024-09-13T17:19:28.734494+0000","last_unstale":"2024-09-13T17:19:29.791576+0000","last_undegraded":"2024-09-13T17:19:29.791576+0000","last_fullsized":"2024-09-13T17:19:29.791576+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:42.568100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.750930+0000","last_change":"2024-09-13T17:19:28.295029+0000","last_active":"2024-09-13T17:19:28.750930+0000","last_peered":"2024-09-13T17:19:28.750930+0000","last_clean":"2024-09-13T17:19:28.750930+0000","last_became_active":"2024-09-13T17:19:28.294755+0000","last_became_peered":"2024-09-13T17:19:28.294755+0000","last_unstale":"2024-09-13T17:19:28.750930+0000","last_undegraded":"2024-09-13T17:19:28.750930+0000","last_fullsized":"2024-09-13T17:19:28.750930+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:58:14.667648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855482+0000","last_change":"2024-09-13T17:19:28.738160+0000","last_active":"2024-09-13T17:19:29.855482+0000","last_peered":"2024-09-13T17:19:29.855482+0000","last_clean":"2024-09-13T17:19:29.855482+0000","last_became_active":"2024-09-13T17:19:28.737909+0000","last_became_peered":"2024-09-13T17:19:28.737909+0000","last_unstale":"2024-09-13T17:19:29.855482+0000","last_undegraded":"2024-09-13T17:19:29.855482+0000","last_fullsized":"2024-09-13T17:19:29.855482+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:41:07.221542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.724536+0000","last_change":"2024-09-13T17:19:28.290154+0000","last_active":"2024-09-13T17:19:28.724536+0000","last_peered":"2024-09-13T17:19:28.724536+0000","last_clean":"2024-09-13T17:19:28.724536+0000","last_became_active":"2024-09-13T17:19:28.289669+0000","last_became_peered":"2024-09-13T17:19:28.289669+0000","last_unstale":"2024-09-13T17:19:28.724536+0000","last_undegraded":"2024-09-13T17:19:28.724536+0000","last_fullsized":"2024-09-13T17:19:28.724536+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:47.633546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725347+0000","last_change":"2024-09-13T17:19:28.737669+0000","last_active":"2024-09-13T17:19:29.725347+0000","last_peered":"2024-09-13T17:19:29.725347+0000","last_clean":"2024-09-13T17:19:29.725347+0000","last_became_active":"2024-09-13T17:19:28.734152+0000","last_became_peered":"2024-09-13T17:19:28.734152+0000","last_unstale":"2024-09-13T17:19:29.725347+0000","last_undegraded":"2024-09-13T17:19:29.725347+0000","last_fullsized":"2024-09-13T17:19:29.725347+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:14:12.577060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.724590+0000","last_change":"2024-09-13T17:19:28.298211+0000","last_active":"2024-09-13T17:19:28.724590+0000","last_peered":"2024-09-13T17:19:28.724590+0000","last_clean":"2024-09-13T17:19:28.724590+0000","last_became_active":"2024-09-13T17:19:28.297960+0000","last_became_peered":"2024-09-13T17:19:28.297960+0000","last_unstale":"2024-09-13T17:19:28.724590+0000","last_undegraded":"2024-09-13T17:19:28.724590+0000","last_fullsized":"2024-09-13T17:19:28.724590+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:18.253231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728142+0000","last_change":"2024-09-13T17:19:28.734420+0000","last_active":"2024-09-13T17:19:29.728142+0000","last_peered":"2024-09-13T17:19:29.728142+0000","last_clean":"2024-09-13T17:19:29.728142+0000","last_became_active":"2024-09-13T17:19:28.734142+0000","last_became_peered":"2024-09-13T17:19:28.734142+0000","last_unstale":"2024-09-13T17:19:29.728142+0000","last_undegraded":"2024-09-13T17:19:29.728142+0000","last_fullsized":"2024-09-13T17:19:29.728142+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:03.122408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"166'8","reported_seq":19,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.753308+0000","last_change":"2024-09-13T17:19:28.289214+0000","last_active":"2024-09-13T17:19:28.753308+0000","last_peered":"2024-09-13T17:19:28.753308+0000","last_clean":"2024-09-13T17:19:28.753308+0000","last_became_active":"2024-09-13T17:19:28.288961+0000","last_became_peered":"2024-09-13T17:19:28.288961+0000","last_unstale":"2024-09-13T17:19:28.753308+0000","last_undegraded":"2024-09-13T17:19:28.753308+0000","last_fullsized":"2024-09-13T17:19:28.753308+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:11.092897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725933+0000","last_change":"2024-09-13T17:19:28.729830+0000","last_active":"2024-09-13T17:19:29.725933+0000","last_peered":"2024-09-13T17:19:29.725933+0000","last_clean":"2024-09-13T17:19:29.725933+0000","last_became_active":"2024-09-13T17:19:28.729490+0000","last_became_peered":"2024-09-13T17:19:28.729490+0000","last_unstale":"2024-09-13T17:19:29.725933+0000","last_undegraded":"2024-09-13T17:19:29.725933+0000","last_fullsized":"2024-09-13T17:19:29.725933+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:03.122408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"166'1","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.746805+0000","last_change":"2024-09-13T17:19:28.292526+0000","last_active":"2024-09-13T17:19:28.746805+0000","last_peered":"2024-09-13T17:19:28.746805+0000","last_clean":"2024-09-13T17:19:28.746805+0000","last_became_active":"2024-09-13T17:19:28.292210+0000","last_became_peered":"2024-09-13T17:19:28.292210+0000","last_unstale":"2024-09-13T17:19:28.746805+0000","last_undegraded":"2024-09-13T17:19:28.746805+0000","last_fullsized":"2024-09-13T17:19:28.746805+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:18.253231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791817+0000","last_change":"2024-09-13T17:19:28.744033+0000","last_active":"2024-09-13T17:19:29.791817+0000","last_peered":"2024-09-13T17:19:29.791817+0000","last_clean":"2024-09-13T17:19:29.791817+0000","last_became_active":"2024-09-13T17:19:28.743761+0000","last_became_peered":"2024-09-13T17:19:28.743761+0000","last_unstale":"2024-09-13T17:19:29.791817+0000","last_undegraded":"2024-09-13T17:19:29.791817+0000","last_fullsized":"2024-09-13T17:19:29.791817+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:40.339402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.745376+0000","last_change":"2024-09-13T17:19:28.294780+0000","last_active":"2024-09-13T17:19:28.745376+0000","last_peered":"2024-09-13T17:19:28.745376+0000","last_clean":"2024-09-13T17:19:28.745376+0000","last_became_active":"2024-09-13T17:19:28.294442+0000","last_became_peered":"2024-09-13T17:19:28.294442+0000","last_unstale":"2024-09-13T17:19:28.745376+0000","last_undegraded":"2024-09-13T17:19:28.745376+0000","last_fullsized":"2024-09-13T17:19:28.745376+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.529755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.725867+0000","last_change":"2024-09-13T17:19:28.734026+0000","last_active":"2024-09-13T17:19:29.725867+0000","last_peered":"2024-09-13T17:19:29.725867+0000","last_clean":"2024-09-13T17:19:29.725867+0000","last_became_active":"2024-09-13T17:19:28.733631+0000","last_became_peered":"2024-09-13T17:19:28.733631+0000","last_unstale":"2024-09-13T17:19:29.725867+0000","last_undegraded":"2024-09-13T17:19:29.725867+0000","last_fullsized":"2024-09-13T17:19:29.725867+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:40.368293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"166'1","reported_seq":12,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.752117+0000","last_change":"2024-09-13T17:19:28.301227+0000","last_active":"2024-09-13T17:19:28.752117+0000","last_peered":"2024-09-13T17:19:28.752117+0000","last_clean":"2024-09-13T17:19:28.752117+0000","last_became_active":"2024-09-13T17:19:28.300956+0000","last_became_peered":"2024-09-13T17:19:28.300956+0000","last_unstale":"2024-09-13T17:19:28.752117+0000","last_undegraded":"2024-09-13T17:19:28.752117+0000","last_fullsized":"2024-09-13T17:19:28.752117+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:31:34.008890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855282+0000","last_change":"2024-09-13T17:19:28.742820+0000","last_active":"2024-09-13T17:19:29.855282+0000","last_peered":"2024-09-13T17:19:29.855282+0000","last_clean":"2024-09-13T17:19:29.855282+0000","last_became_active":"2024-09-13T17:19:28.740714+0000","last_became_peered":"2024-09-13T17:19:28.740714+0000","last_unstale":"2024-09-13T17:19:29.855282+0000","last_undegraded":"2024-09-13T17:19:29.855282+0000","last_fullsized":"2024-09-13T17:19:29.855282+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:24.309783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.735224+0000","last_change":"2024-09-13T17:19:28.301547+0000","last_active":"2024-09-13T17:19:28.735224+0000","last_peered":"2024-09-13T17:19:28.735224+0000","last_clean":"2024-09-13T17:19:28.735224+0000","last_became_active":"2024-09-13T17:19:28.301271+0000","last_became_peered":"2024-09-13T17:19:28.301271+0000","last_unstale":"2024-09-13T17:19:28.735224+0000","last_undegraded":"2024-09-13T17:19:28.735224+0000","last_fullsized":"2024-09-13T17:19:28.735224+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:33.813305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728184+0000","last_change":"2024-09-13T17:19:28.737038+0000","last_active":"2024-09-13T17:19:29.728184+0000","last_peered":"2024-09-13T17:19:29.728184+0000","last_clean":"2024-09-13T17:19:29.728184+0000","last_became_active":"2024-09-13T17:19:28.733233+0000","last_became_peered":"2024-09-13T17:19:28.733233+0000","last_unstale":"2024-09-13T17:19:29.728184+0000","last_undegraded":"2024-09-13T17:19:29.728184+0000","last_fullsized":"2024-09-13T17:19:29.728184+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:56:50.346483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.752789+0000","last_change":"2024-09-13T17:19:28.299500+0000","last_active":"2024-09-13T17:19:28.752789+0000","last_peered":"2024-09-13T17:19:28.752789+0000","last_clean":"2024-09-13T17:19:28.752789+0000","last_became_active":"2024-09-13T17:19:28.298962+0000","last_became_peered":"2024-09-13T17:19:28.298962+0000","last_unstale":"2024-09-13T17:19:28.752789+0000","last_undegraded":"2024-09-13T17:19:28.752789+0000","last_fullsized":"2024-09-13T17:19:28.752789+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:10.695940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855502+0000","last_change":"2024-09-13T17:19:28.739158+0000","last_active":"2024-09-13T17:19:29.855502+0000","last_peered":"2024-09-13T17:19:29.855502+0000","last_clean":"2024-09-13T17:19:29.855502+0000","last_became_active":"2024-09-13T17:19:28.738777+0000","last_became_peered":"2024-09-13T17:19:28.738777+0000","last_unstale":"2024-09-13T17:19:29.855502+0000","last_undegraded":"2024-09-13T17:19:29.855502+0000","last_fullsized":"2024-09-13T17:19:29.855502+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:46.386022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.734769+0000","last_change":"2024-09-13T17:19:28.299309+0000","last_active":"2024-09-13T17:19:28.734769+0000","last_peered":"2024-09-13T17:19:28.734769+0000","last_clean":"2024-09-13T17:19:28.734769+0000","last_became_active":"2024-09-13T17:19:28.298408+0000","last_became_peered":"2024-09-13T17:19:28.298408+0000","last_unstale":"2024-09-13T17:19:28.734769+0000","last_undegraded":"2024-09-13T17:19:28.734769+0000","last_fullsized":"2024-09-13T17:19:28.734769+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:11.092897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728974+0000","last_change":"2024-09-13T17:19:28.742578+0000","last_active":"2024-09-13T17:19:29.728974+0000","last_peered":"2024-09-13T17:19:29.728974+0000","last_clean":"2024-09-13T17:19:29.728974+0000","last_became_active":"2024-09-13T17:19:28.742298+0000","last_became_peered":"2024-09-13T17:19:28.742298+0000","last_unstale":"2024-09-13T17:19:29.728974+0000","last_undegraded":"2024-09-13T17:19:29.728974+0000","last_fullsized":"2024-09-13T17:19:29.728974+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:35.034599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.724684+0000","last_change":"2024-09-13T17:19:28.294205+0000","last_active":"2024-09-13T17:19:28.724684+0000","last_peered":"2024-09-13T17:19:28.724684+0000","last_clean":"2024-09-13T17:19:28.724684+0000","last_became_active":"2024-09-13T17:19:28.293765+0000","last_became_peered":"2024-09-13T17:19:28.293765+0000","last_unstale":"2024-09-13T17:19:28.724684+0000","last_undegraded":"2024-09-13T17:19:28.724684+0000","last_fullsized":"2024-09-13T17:19:28.724684+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140513+0000","last_change":"2024-09-13T17:19:28.733276+0000","last_active":"2024-09-13T17:19:30.140513+0000","last_peered":"2024-09-13T17:19:30.140513+0000","last_clean":"2024-09-13T17:19:30.140513+0000","last_became_active":"2024-09-13T17:19:28.733001+0000","last_became_peered":"2024-09-13T17:19:28.733001+0000","last_unstale":"2024-09-13T17:19:30.140513+0000","last_undegraded":"2024-09-13T17:19:30.140513+0000","last_fullsized":"2024-09-13T17:19:30.140513+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:33.361454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.734387+0000","last_change":"2024-09-13T17:19:28.296655+0000","last_active":"2024-09-13T17:19:28.734387+0000","last_peered":"2024-09-13T17:19:28.734387+0000","last_clean":"2024-09-13T17:19:28.734387+0000","last_became_active":"2024-09-13T17:19:28.296393+0000","last_became_peered":"2024-09-13T17:19:28.296393+0000","last_unstale":"2024-09-13T17:19:28.734387+0000","last_undegraded":"2024-09-13T17:19:28.734387+0000","last_fullsized":"2024-09-13T17:19:28.734387+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:41.661263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791942+0000","last_change":"2024-09-13T17:19:28.740825+0000","last_active":"2024-09-13T17:19:29.791942+0000","last_peered":"2024-09-13T17:19:29.791942+0000","last_clean":"2024-09-13T17:19:29.791942+0000","last_became_active":"2024-09-13T17:19:28.740548+0000","last_became_peered":"2024-09-13T17:19:28.740548+0000","last_unstale":"2024-09-13T17:19:29.791942+0000","last_undegraded":"2024-09-13T17:19:29.791942+0000","last_fullsized":"2024-09-13T17:19:29.791942+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:26.766558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"166'2","reported_seq":13,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.742117+0000","last_change":"2024-09-13T17:19:28.296945+0000","last_active":"2024-09-13T17:19:28.742117+0000","last_peered":"2024-09-13T17:19:28.742117+0000","last_clean":"2024-09-13T17:19:28.742117+0000","last_became_active":"2024-09-13T17:19:28.296010+0000","last_became_peered":"2024-09-13T17:19:28.296010+0000","last_unstale":"2024-09-13T17:19:28.742117+0000","last_undegraded":"2024-09-13T17:19:28.742117+0000","last_fullsized":"2024-09-13T17:19:28.742117+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:45.379894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855305+0000","last_change":"2024-09-13T17:19:28.733639+0000","last_active":"2024-09-13T17:19:29.855305+0000","last_peered":"2024-09-13T17:19:29.855305+0000","last_clean":"2024-09-13T17:19:29.855305+0000","last_became_active":"2024-09-13T17:19:28.733375+0000","last_became_peered":"2024-09-13T17:19:28.733375+0000","last_unstale":"2024-09-13T17:19:29.855305+0000","last_undegraded":"2024-09-13T17:19:29.855305+0000","last_fullsized":"2024-09-13T17:19:29.855305+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:31:38.921726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.729412+0000","last_change":"2024-09-13T17:19:28.287391+0000","last_active":"2024-09-13T17:19:28.729412+0000","last_peered":"2024-09-13T17:19:28.729412+0000","last_clean":"2024-09-13T17:19:28.729412+0000","last_became_active":"2024-09-13T17:19:28.287096+0000","last_became_peered":"2024-09-13T17:19:28.287096+0000","last_unstale":"2024-09-13T17:19:28.729412+0000","last_undegraded":"2024-09-13T17:19:28.729412+0000","last_fullsized":"2024-09-13T17:19:28.729412+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.140481+0000","last_change":"2024-09-13T17:19:28.740530+0000","last_active":"2024-09-13T17:19:30.140481+0000","last_peered":"2024-09-13T17:19:30.140481+0000","last_clean":"2024-09-13T17:19:30.140481+0000","last_became_active":"2024-09-13T17:19:28.740259+0000","last_became_peered":"2024-09-13T17:19:28.740259+0000","last_unstale":"2024-09-13T17:19:30.140481+0000","last_undegraded":"2024-09-13T17:19:30.140481+0000","last_fullsized":"2024-09-13T17:19:30.140481+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:05.588158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.729064+0000","last_change":"2024-09-13T17:19:28.297103+0000","last_active":"2024-09-13T17:19:28.729064+0000","last_peered":"2024-09-13T17:19:28.729064+0000","last_clean":"2024-09-13T17:19:28.729064+0000","last_became_active":"2024-09-13T17:19:28.296208+0000","last_became_peered":"2024-09-13T17:19:28.296208+0000","last_unstale":"2024-09-13T17:19:28.729064+0000","last_undegraded":"2024-09-13T17:19:28.729064+0000","last_fullsized":"2024-09-13T17:19:28.729064+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:39:04.543707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791724+0000","last_change":"2024-09-13T17:19:28.739723+0000","last_active":"2024-09-13T17:19:29.791724+0000","last_peered":"2024-09-13T17:19:29.791724+0000","last_clean":"2024-09-13T17:19:29.791724+0000","last_became_active":"2024-09-13T17:19:28.738616+0000","last_became_peered":"2024-09-13T17:19:28.738616+0000","last_unstale":"2024-09-13T17:19:29.791724+0000","last_undegraded":"2024-09-13T17:19:29.791724+0000","last_fullsized":"2024-09-13T17:19:29.791724+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:40.112255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.725420+0000","last_change":"2024-09-13T17:19:28.285891+0000","last_active":"2024-09-13T17:19:28.725420+0000","last_peered":"2024-09-13T17:19:28.725420+0000","last_clean":"2024-09-13T17:19:28.725420+0000","last_became_active":"2024-09-13T17:19:28.285609+0000","last_became_peered":"2024-09-13T17:19:28.285609+0000","last_unstale":"2024-09-13T17:19:28.725420+0000","last_undegraded":"2024-09-13T17:19:28.725420+0000","last_fullsized":"2024-09-13T17:19:28.725420+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:31:37.915597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791578+0000","last_change":"2024-09-13T17:19:28.739774+0000","last_active":"2024-09-13T17:19:29.791578+0000","last_peered":"2024-09-13T17:19:29.791578+0000","last_clean":"2024-09-13T17:19:29.791578+0000","last_became_active":"2024-09-13T17:19:28.739198+0000","last_became_peered":"2024-09-13T17:19:28.739198+0000","last_unstale":"2024-09-13T17:19:29.791578+0000","last_undegraded":"2024-09-13T17:19:29.791578+0000","last_fullsized":"2024-09-13T17:19:29.791578+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:27.920113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.731581+0000","last_change":"2024-09-13T17:19:28.295374+0000","last_active":"2024-09-13T17:19:28.731581+0000","last_peered":"2024-09-13T17:19:28.731581+0000","last_clean":"2024-09-13T17:19:28.731581+0000","last_became_active":"2024-09-13T17:19:28.294745+0000","last_became_peered":"2024-09-13T17:19:28.294745+0000","last_unstale":"2024-09-13T17:19:28.731581+0000","last_undegraded":"2024-09-13T17:19:28.731581+0000","last_fullsized":"2024-09-13T17:19:28.731581+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:04:33.813305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:30.141438+0000","last_change":"2024-09-13T17:19:28.733582+0000","last_active":"2024-09-13T17:19:30.141438+0000","last_peered":"2024-09-13T17:19:30.141438+0000","last_clean":"2024-09-13T17:19:30.141438+0000","last_became_active":"2024-09-13T17:19:28.733205+0000","last_became_peered":"2024-09-13T17:19:28.733205+0000","last_unstale":"2024-09-13T17:19:30.141438+0000","last_undegraded":"2024-09-13T17:19:30.141438+0000","last_fullsized":"2024-09-13T17:19:30.141438+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:16:54.593844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.726959+0000","last_change":"2024-09-13T17:19:28.296181+0000","last_active":"2024-09-13T17:19:28.726959+0000","last_peered":"2024-09-13T17:19:28.726959+0000","last_clean":"2024-09-13T17:19:28.726959+0000","last_became_active":"2024-09-13T17:19:28.295926+0000","last_became_peered":"2024-09-13T17:19:28.295926+0000","last_unstale":"2024-09-13T17:19:28.726959+0000","last_undegraded":"2024-09-13T17:19:28.726959+0000","last_fullsized":"2024-09-13T17:19:28.726959+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:38.529755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.855574+0000","last_change":"2024-09-13T17:19:28.742093+0000","last_active":"2024-09-13T17:19:29.855574+0000","last_peered":"2024-09-13T17:19:29.855574+0000","last_clean":"2024-09-13T17:19:29.855574+0000","last_became_active":"2024-09-13T17:19:28.739130+0000","last_became_peered":"2024-09-13T17:19:28.739130+0000","last_unstale":"2024-09-13T17:19:29.855574+0000","last_undegraded":"2024-09-13T17:19:29.855574+0000","last_fullsized":"2024-09-13T17:19:29.855574+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:48.639674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.729043+0000","last_change":"2024-09-13T17:19:28.298693+0000","last_active":"2024-09-13T17:19:28.729043+0000","last_peered":"2024-09-13T17:19:28.729043+0000","last_clean":"2024-09-13T17:19:28.729043+0000","last_became_active":"2024-09-13T17:19:28.298286+0000","last_became_peered":"2024-09-13T17:19:28.298286+0000","last_unstale":"2024-09-13T17:19:28.729043+0000","last_undegraded":"2024-09-13T17:19:28.729043+0000","last_fullsized":"2024-09-13T17:19:28.729043+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:12.974081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.728142+0000","last_change":"2024-09-13T17:19:28.735073+0000","last_active":"2024-09-13T17:19:29.728142+0000","last_peered":"2024-09-13T17:19:29.728142+0000","last_clean":"2024-09-13T17:19:29.728142+0000","last_became_active":"2024-09-13T17:19:28.732525+0000","last_became_peered":"2024-09-13T17:19:28.732525+0000","last_unstale":"2024-09-13T17:19:29.728142+0000","last_undegraded":"2024-09-13T17:19:29.728142+0000","last_fullsized":"2024-09-13T17:19:29.728142+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:22.652493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.729848+0000","last_change":"2024-09-13T17:19:28.298044+0000","last_active":"2024-09-13T17:19:28.729848+0000","last_peered":"2024-09-13T17:19:28.729848+0000","last_clean":"2024-09-13T17:19:28.729848+0000","last_became_active":"2024-09-13T17:19:28.297686+0000","last_became_peered":"2024-09-13T17:19:28.297686+0000","last_unstale":"2024-09-13T17:19:28.729848+0000","last_undegraded":"2024-09-13T17:19:28.729848+0000","last_fullsized":"2024-09-13T17:19:28.729848+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:37.033265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":10,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-13T17:19:29.791866+0000","last_change":"2024-09-13T17:19:28.746700+0000","last_active":"2024-09-13T17:19:29.791866+0000","last_peered":"2024-09-13T17:19:29.791866+0000","last_clean":"2024-09-13T17:19:29.791866+0000","last_became_active":"2024-09-13T17:19:28.746068+0000","last_became_peered":"2024-09-13T17:19:28.746068+0000","last_unstale":"2024-09-13T17:19:29.791866+0000","last_undegraded":"2024-09-13T17:19:29.791866+0000","last_fullsized":"2024-09-13T17:19:29.791866+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":165,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:28.270223+0000","last_clean_scrub_stamp":"2024-09-13T17:19:28.270223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:14:06.167774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":11,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.732190+0000","last_change":"2024-09-13T17:19:28.300949+0000","last_active":"2024-09-13T17:19:28.732190+0000","last_peered":"2024-09-13T17:19:28.732190+0000","last_clean":"2024-09-13T17:19:28.732190+0000","last_became_active":"2024-09-13T17:19:28.300197+0000","last_became_peered":"2024-09-13T17:19:28.300197+0000","last_unstale":"2024-09-13T17:19:28.732190+0000","last_undegraded":"2024-09-13T17:19:28.732190+0000","last_fullsized":"2024-09-13T17:19:28.732190+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":164,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:19:27.264095+0000","last_clean_scrub_stamp":"2024-09-13T17:19:27.264095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:10:22.743250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":287,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-13T17:19:28.734481+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:19:28.734481+0000","last_peered":"2024-09-13T17:19:28.734481+0000","last_clean":"2024-09-13T17:19:28.734481+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:19:28.734481+0000","last_undegraded":"2024-09-13T17:19:28.734481+0000","last_fullsized":"2024-09-13T17:19:28.734481+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574990,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77988,"kb_used_data":2648,"kb_used_omap":1,"kb_used_meta":75326,"kb_avail":94293852,"statfs":{"total":96636764160,"available":96556904448,"internally_reserved":0,"allocated":2711552,"data_stored":1515367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77134283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.64500000000000002},"min":{"1min":0.40100000000000002,"5min":0.33800000000000002,"15min":0.33700000000000002},"max":{"1min":0.77600000000000002,"5min":0.996,"15min":2.0880000000000001},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.64500000000000002,"15min":0.67600000000000005},"min":{"1min":0.41899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.84199999999999997,"5min":1.004,"15min":2.6739999999999999},"last":0.79800000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.64700000000000002,"15min":0.65800000000000003},"min":{"1min":0.40100000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.84699999999999998,"5min":1.127,"15min":2.5030000000000001},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.64300000000000002,"15min":0.70299999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.84299999999999997,"5min":0.92000000000000004,"15min":6.4909999999999997},"last":0.93500000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67400000000000004,"15min":0.69499999999999995},"min":{"1min":0.46400000000000002,"5min":0.42999999999999999,"15min":0.42399999999999999},"max":{"1min":0.91200000000000003,"5min":1.169,"15min":1.544},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66100000000000003,"15min":0.69999999999999996},"min":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.95499999999999996,"15min":1.605},"last":0.879}]},{"osd":3,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.443,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.091,"5min":1.091,"15min":2.4249999999999998},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.69499999999999995,"15min":0.73799999999999999},"min":{"1min":0.53500000000000003,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.875,"5min":1.274,"15min":2.8239999999999998},"last":0.81999999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.58899999999999997,"15min":0.627},"min":{"1min":0.374,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78600000000000003,"5min":0.90400000000000003,"15min":1.054},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.65400000000000003,"15min":0.67500000000000004},"min":{"1min":0.47299999999999998,"5min":0.376,"15min":0.35399999999999998},"max":{"1min":1.04,"5min":1.234,"15min":1.234},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.67100000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.88100000000000001,"5min":0.99299999999999999,"15min":1.143},"last":0.499},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.72199999999999998,"15min":0.75800000000000001},"min":{"1min":0.55200000000000005,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.82899999999999996,"5min":1.1659999999999999,"15min":1.337},"last":0.78700000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70799999999999996,"15min":0.74199999999999999},"min":{"1min":0.47499999999999998,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.93100000000000005,"5min":1.048,"15min":1.1930000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70499999999999996,"15min":0.748},"min":{"1min":0.52000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.85699999999999998,"5min":1.101,"15min":1.2030000000000001},"last":0.73199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574991,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59404,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312436,"statfs":{"total":96636764160,"available":96575934464,"internally_reserved":0,"allocated":2097152,"data_stored":922715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63300000000000001,"15min":0.66300000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.77600000000000002,"5min":1.298,"15min":3.6629999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.749,"15min":0.73799999999999999},"min":{"1min":0.52000000000000002,"5min":0.47199999999999998,"15min":0.36099999999999999},"max":{"1min":0.79400000000000004,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.89100000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63100000000000001,"15min":0.65300000000000002},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.745,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.65500000000000003,"15min":0.65800000000000003},"min":{"1min":0.39600000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.81200000000000006,"5min":1.46,"15min":3.3079999999999998},"last":0.86699999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.71099999999999997,"15min":0.68100000000000005},"min":{"1min":0.46800000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.83599999999999997,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66100000000000003,"15min":0.68799999999999994},"min":{"1min":0.47399999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.81999999999999995,"5min":0.96899999999999997,"15min":3.2629999999999999},"last":0.746}]},{"osd":3,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.95199999999999996,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.70199999999999996},"min":{"1min":0.49099999999999999,"5min":0.46200000000000002,"15min":0.46000000000000002},"max":{"1min":0.73299999999999998,"5min":3.6720000000000002,"15min":3.742},"last":0.84499999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.62,"15min":0.625},"min":{"1min":0.47899999999999998,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.84399999999999997,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.628,"15min":0.65000000000000002},"min":{"1min":0.51100000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.88300000000000001,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.70199999999999996},"min":{"1min":0.44900000000000001,"5min":0.374,"15min":0.374},"max":{"1min":1.0509999999999999,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63500000000000001,"15min":0.68600000000000005},"min":{"1min":0.45500000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86299999999999999,"5min":0.92100000000000004,"15min":3.609},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.71699999999999997,"15min":0.73999999999999999},"min":{"1min":0.56899999999999995,"5min":0.378,"15min":0.378},"max":{"1min":0.93200000000000005,"5min":1.601,"15min":4.9669999999999996},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.76500000000000001,"15min":0.79300000000000004},"min":{"1min":0.61399999999999999,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0940000000000001,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.89800000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574990,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55320,"kb_used_data":2052,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316520,"statfs":{"total":96636764160,"available":96580116480,"internally_reserved":0,"allocated":2101248,"data_stored":923297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.65000000000000002,"15min":0.77100000000000002},"min":{"1min":0.48999999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4390000000000001,"5min":1.498,"15min":38.005000000000003},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.65800000000000003,"15min":0.747},"min":{"1min":0.54700000000000004,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.91800000000000004,"5min":1.577,"15min":21.376999999999999},"last":0.501}]},{"osd":1,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.65600000000000003,"15min":0.72699999999999998},"min":{"1min":0.46600000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.0149999999999999,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68100000000000005,"15min":0.69799999999999995},"min":{"1min":0.49399999999999999,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":1.482,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.67200000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.65900000000000003,"15min":0.74199999999999999},"min":{"1min":0.45400000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.96299999999999997,"5min":1.524,"15min":17.268000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69499999999999995,"15min":0.73899999999999999},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.0649999999999999,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.57199999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.72999999999999998,"15min":1.095},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.91100000000000003,"5min":2.8690000000000002,"15min":113.363},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.72399999999999998,"15min":0.752},"min":{"1min":0.52900000000000003,"5min":0.433,"15min":0.433},"max":{"1min":1.113,"5min":2.359,"15min":4.1180000000000003},"last":0.624}]},{"osd":4,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.628,"15min":0.627},"min":{"1min":0.42099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.5669999999999999,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.66900000000000004,"15min":0.69799999999999995},"min":{"1min":0.39900000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.03,"5min":2.827,"15min":2.827},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.68200000000000005,"15min":0.69999999999999996},"min":{"1min":0.46999999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.73,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.71299999999999997,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.532,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.83099999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:19:22 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":0.73899999999999999,"15min":0.75700000000000001},"min":{"1min":0.53000000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.0700000000000001,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.74199999999999999,"15min":0.76200000000000001},"min":{"1min":0.56799999999999995,"5min":0.44,"15min":0.44},"max":{"1min":1.363,"5min":1.9379999999999999,"15min":2.02},"last":0.77200000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65116,"kb_used_data":2060,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306724,"statfs":{"total":96636764160,"available":96570085376,"internally_reserved":0,"allocated":2109440,"data_stored":924327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.64200000000000002,"15min":0.64700000000000002},"min":{"1min":0.34999999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":1.0620000000000001},"last":0.752},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.70099999999999996,"15min":0.72099999999999997},"min":{"1min":0.46000000000000002,"5min":0.40699999999999997,"15min":0.373},"max":{"1min":0.99099999999999999,"5min":1.3400000000000001,"15min":1.341},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65500000000000003,"15min":0.66000000000000003},"min":{"1min":0.44600000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.874,"5min":1.173,"15min":1.173},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.39900000000000002,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.88200000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.46200000000000002,"5min":0.38900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88800000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.69599999999999995,"15min":0.70899999999999996},"min":{"1min":0.47299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0409999999999999,"5min":1.22,"15min":2.1259999999999999},"last":0.60599999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.68100000000000005,"15min":0.67600000000000005},"min":{"1min":0.44500000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.86499999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.69999999999999996,"15min":0.69899999999999995},"min":{"1min":0.47199999999999998,"5min":0.46600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":1.304,"15min":1.304},"last":0.51900000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.64400000000000002,"15min":0.64100000000000001},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.073,"5min":1.284,"15min":1.284},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64300000000000002,"15min":0.65300000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":0.96499999999999997,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.55300000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.67000000000000004,"15min":0.66600000000000004},"min":{"1min":0.36699999999999999,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.96599999999999997,"5min":1.274,"15min":1.274},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73099999999999998,"15min":0.73299999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.41799999999999998},"max":{"1min":0.94599999999999995,"5min":1.292,"15min":1.413},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:19:16 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70799999999999996,"15min":0.70499999999999996},"min":{"1min":0.48299999999999998,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":0.92900000000000005,"5min":1.252,"15min":1.252},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.754,"15min":0.73999999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.371},"max":{"1min":1.0109999999999999,"5min":1.351,"15min":1.351},"last":0.78700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834574990,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65100,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":2097152,"data_stored":922727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39900000000000002,"15min":0.439},"min":{"1min":0.27700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.59499999999999997,"5min":0.67500000000000004,"15min":1.9810000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.45500000000000002,"15min":0.50900000000000001},"min":{"1min":0.28199999999999997,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.66400000000000003,"5min":0.68300000000000005,"15min":2.96},"last":0.54200000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49399999999999999,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.29499999999999998,"15min":0.28299999999999997},"max":{"1min":0.69099999999999995,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54200000000000004,"15min":0.64200000000000002},"min":{"1min":0.45300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66700000000000004,"5min":0.77600000000000002,"15min":3.6829999999999998},"last":0.63}]},{"osd":2,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.36399999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84999999999999998,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.52300000000000002,"15min":0.56899999999999995},"min":{"1min":0.38,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71099999999999997,"5min":0.73499999999999999,"15min":1.3819999999999999},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.76300000000000001},"min":{"1min":0.40899999999999997,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":0.90800000000000003,"5min":1.1040000000000001,"15min":2.8610000000000002},"last":1.171},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72699999999999998,"15min":0.76000000000000001},"min":{"1min":0.51500000000000001,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.99299999999999999,"5min":1.012,"15min":1.3640000000000001},"last":1.1120000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.71999999999999997,"15min":0.76600000000000001},"min":{"1min":0.58299999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.89000000000000001,"5min":1.032,"15min":1.391},"last":1.25},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.71999999999999997,"15min":0.78200000000000003},"min":{"1min":0.54300000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.0069999999999999,"5min":1.0660000000000001,"15min":2.9300000000000002},"last":1.224}]},{"osd":6,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.746,"15min":0.76300000000000001},"min":{"1min":0.51100000000000001,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.997,"5min":1.0840000000000001,"15min":1.401},"last":1.091},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.74099999999999999,"15min":0.77900000000000003},"min":{"1min":0.53500000000000003,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.0209999999999999,"5min":1.1850000000000001,"15min":1.3300000000000001},"last":1.1879999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.747,"15min":0.80900000000000005},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.97399999999999998,"5min":1.1339999999999999,"15min":2.899},"last":1.27},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77100000000000002,"15min":0.80500000000000005},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.02,"5min":1.0920000000000001,"15min":1.4359999999999999},"last":1.2070000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574990,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59400,"kb_used_data":2044,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312440,"statfs":{"total":96636764160,"available":96575938560,"internally_reserved":0,"allocated":2093056,"data_stored":921764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58718663},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.41799999999999998,"15min":0.41699999999999998},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51400000000000001,"5min":1.097,"15min":1.097},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.78600000000000003,"15min":0.56200000000000006},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":30.023,"15min":30.023},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.48499999999999999,"15min":1.347},"min":{"1min":0.378,"5min":0.33000000000000002,"15min":0.307},"max":{"1min":0.73699999999999999,"5min":1.046,"15min":260.53399999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.73799999999999999,"15min":1.45},"min":{"1min":0.35999999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.79800000000000004,"5min":21.279,"15min":262.61399999999998},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.81499999999999995,"15min":0.59899999999999998},"min":{"1min":0.36399999999999999,"5min":0.32500000000000001,"15min":0.29099999999999998},"max":{"1min":1.016,"5min":27.271999999999998,"15min":27.271999999999998},"last":0.443},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.47499999999999998,"5min":0.439,"15min":0.42199999999999999},"max":{"1min":1.1539999999999999,"5min":1.1559999999999999,"15min":1.214},"last":0.56299999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.84999999999999998,"15min":0.79400000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95299999999999996,"5min":10.598000000000001,"15min":10.598000000000001},"last":0.998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.96599999999999997,"15min":0.84599999999999997},"min":{"1min":0.53300000000000003,"5min":0.371,"15min":0.371},"max":{"1min":1.085,"5min":21.213000000000001,"15min":21.213000000000001},"last":1.012}]},{"osd":5,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":1.048,"15min":0.876},"min":{"1min":0.48299999999999998,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":1.121,"5min":27.135000000000002,"15min":27.135000000000002},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.74399999999999999,"15min":0.76700000000000002},"min":{"1min":0.59099999999999997,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.042,"5min":1.1279999999999999,"15min":1.569},"last":0.91300000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.85999999999999999,"15min":0.80400000000000005},"min":{"1min":0.55700000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.018,"5min":10.664,"15min":10.664},"last":1.077},{"interface":"front","average":{"1min":0.76500000000000001,"5min":1.056,"15min":0.88800000000000001},"min":{"1min":0.52100000000000002,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.2210000000000001,"5min":27.236999999999998,"15min":27.236999999999998},"last":0.85199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.87,"15min":0.81599999999999995},"min":{"1min":0.54500000000000004,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.194,"5min":10.712999999999999,"15min":10.712999999999999},"last":1.048},{"interface":"front","average":{"1min":0.747,"5min":1.079,"15min":0.89600000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.46100000000000002},"max":{"1min":1.0469999999999999,"5min":27.199999999999999,"15min":27.199999999999999},"last":0.90100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574989,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87252,"kb_used_data":2048,"kb_used_omap":1,"kb_used_meta":85182,"kb_avail":94284588,"statfs":{"total":96636764160,"available":96547418112,"internally_reserved":0,"allocated":2097152,"data_stored":922727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87226822},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.42899999999999999,"15min":0.47799999999999998},"min":{"1min":0.249,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55900000000000005,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.48799999999999999,"15min":0.51600000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.59699999999999998,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.45800000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.42999999999999999,"15min":0.46100000000000002},"min":{"1min":0.30399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.54600000000000004,"5min":0.68300000000000005,"15min":2.3119999999999998},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.497,"15min":0.53600000000000003},"min":{"1min":0.378,"5min":0.316,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.438}]},{"osd":3,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58099999999999996,"15min":0.66700000000000004},"min":{"1min":0.44600000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.70099999999999996,"5min":0.79200000000000004,"15min":16.382999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52200000000000002,"15min":0.60299999999999998},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.69699999999999995,"5min":0.79500000000000004,"15min":18.318000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.753,"15min":0.79400000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92600000000000005,"5min":1.165,"15min":6.1630000000000003},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78300000000000003,"15min":0.80700000000000005},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.49399999999999999},"max":{"1min":1.087,"5min":1.149,"15min":1.78},"last":0.85599999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.75600000000000001,"15min":0.76800000000000002},"min":{"1min":0.56000000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.747,"15min":0.78800000000000003},"min":{"1min":0.48499999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0309999999999999,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.753,"15min":0.78900000000000003},"min":{"1min":0.55600000000000005,"5min":0.499,"15min":0.499},"max":{"1min":1.0489999999999999,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.76700000000000002,"15min":0.80700000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1140000000000001,"5min":1.204,"15min":6.1970000000000001},"last":0.84199999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.76600000000000001,"15min":0.80200000000000005},"min":{"1min":0.54700000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.98199999999999998,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.78100000000000003,"15min":0.81299999999999994},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.6359999999999999},"last":0.78600000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574991,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65672,"kb_used_data":2620,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306168,"statfs":{"total":96636764160,"available":96569516032,"internally_reserved":0,"allocated":2682880,"data_stored":1512042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45800000000000002,"15min":0.56599999999999995},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.986,"5min":1.986,"15min":5.5259999999999998},"last":0.371},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.51600000000000001,"15min":0.58599999999999997},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.64700000000000002,"5min":3.004,"15min":5.4889999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.55600000000000005,"15min":0.621},"min":{"1min":0.34599999999999997,"5min":0.313,"15min":0.313},"max":{"1min":0.69899999999999995,"5min":3.605,"15min":6.0430000000000001},"last":0.441},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.52500000000000002,"15min":0.62},"min":{"1min":0.35799999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.73399999999999999,"5min":1.29,"15min":4.6760000000000002},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.51100000000000001,"15min":0.58399999999999996},"min":{"1min":0.34399999999999997,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.73299999999999998,"5min":2.6400000000000001,"15min":4.7549999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.58999999999999997,"15min":0.65000000000000002},"min":{"1min":0.42599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.69399999999999995,"5min":3.7709999999999999,"15min":4.5940000000000003},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.76200000000000001,"15min":0.82899999999999996},"min":{"1min":0.61299999999999999,"5min":0.502,"15min":0.443},"max":{"1min":1.022,"5min":1.9490000000000001,"15min":4.734},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.76800000000000002,"15min":0.83699999999999997},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0369999999999999,"5min":2.6760000000000002,"15min":4.8079999999999998},"last":0.624}]},{"osd":5,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.78800000000000003,"15min":0.84999999999999998},"min":{"1min":0.55600000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.99099999999999999,"5min":2.7320000000000002,"15min":4.944},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.78600000000000003,"15min":0.84899999999999998},"min":{"1min":0.56999999999999995,"5min":0.432,"15min":0.432},"max":{"1min":0.99399999999999999,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.77800000000000002,"15min":0.92200000000000004},"min":{"1min":0.57599999999999996,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94199999999999995,"5min":1.161,"15min":4.8470000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.79300000000000004,"15min":0.86199999999999999},"min":{"1min":0.64500000000000002,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":0.995,"5min":1.3720000000000001,"15min":5.0170000000000003},"last":0.745}]},{"osd":7,"last update":"Fri Sep 13 17:19:20 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.81899999999999995,"15min":0.88800000000000001},"min":{"1min":0.59799999999999998,"5min":0.52200000000000002,"15min":0.47399999999999998},"max":{"1min":2.044,"5min":2.044,"15min":5.0540000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.82499999999999996,"15min":0.88100000000000001},"min":{"1min":0.59499999999999997,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.1220000000000001,"5min":2.706,"15min":4.9950000000000001},"last":0.72599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:19:33.337 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:19:33.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:19:33.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.533+0000 7fe3c1e00640 1 Processor -- start 2024-09-13T17:19:33.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.533+0000 7fe3c1e00640 1 -- start start 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c1e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 0x7fe3bc0cb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c1e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc0722e0 con 0x7fe3bc0b0c30 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c1e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cb3e0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c1e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3bc0cbdb0 con 0x7fe3bc0cc4a0 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44868/0 (socket says 172.21.15.28:44868) 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3bbe00640 1 -- 172.21.15.28:0/2209367981 learned_addr learned my addr 172.21.15.28:0/2209367981 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3bbe00640 1 -- 172.21.15.28:0/2209367981 >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 msgr2=0x7fe3bc0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:33.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/2209367981 >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 0x7fe3bc0cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.535+0000 7fe3bbe00640 1 --2- 172.21.15.28:0/2209367981 >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 0x7fe3bc0cb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.536+0000 7fe3bbe00640 1 -- 172.21.15.28:0/2209367981 >> v1:172.21.15.189:6789/0 conn(0x7fe3bc0b0c30 legacy=0x7fe3bc0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:33.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.536+0000 7fe3bbe00640 1 -- 172.21.15.28:0/2209367981 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc072000 con 0x7fe3bc0cc880 2024-09-13T17:19:33.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.536+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/2209367981 >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 0x7fe3bc0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.536+0000 7fe3bbe00640 1 --2- 172.21.15.28:0/2209367981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009870 tx=0x7fe3a802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d12bf4d9f723e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3bb400640 1 -- 172.21.15.28:0/2209367981 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8032440 con 0x7fe3bc0cc880 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3bb400640 1 -- 172.21.15.28:0/2209367981 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a80325a0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3bb400640 1 -- 172.21.15.28:0/2209367981 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8032930 con 0x7fe3bc0cc880 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 -- 172.21.15.28:0/2209367981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 msgr2=0x7fe3bc0b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/2209367981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8009870 tx=0x7fe3a802a1c0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 -- 172.21.15.28:0/2209367981 shutdown_connections 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/2209367981 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc0b06f0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/2209367981 >> v2:172.21.15.189:3300/0 conn(0x7fe3bc0cc4a0 0x7fe3bc0cb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.537+0000 7fe3c1e00640 1 -- 172.21.15.28:0/2209367981 >> 172.21.15.28:0/2209367981 conn(0x7fe3bc069e80 msgr2=0x7fe3bc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.538+0000 7fe3c1e00640 1 -- 172.21.15.28:0/2209367981 shutdown_connections 2024-09-13T17:19:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.538+0000 7fe3c1e00640 1 -- 172.21.15.28:0/2209367981 wait complete. 2024-09-13T17:19:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.539+0000 7fe3c1e00640 1 Processor -- start 2024-09-13T17:19:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.539+0000 7fe3c1e00640 1 -- start start 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3bc0b0c30 0x7fe3bc1f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3bc0cc4a0 0x7fe3bc0b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc1f60f0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0b14c0 con 0x7fe3bc0cc4a0 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0b1710 con 0x7fe3bc0b0c30 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44878/0 (socket says 172.21.15.28:44878) 2024-09-13T17:19:33.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.540+0000 7fe3c1400640 1 -- 172.21.15.28:0/1282612788 learned_addr learned my addr 172.21.15.28:0/1282612788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3bc0b0c30 0x7fe3bc1f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3bbe00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3bc0cc4a0 0x7fe3bc0b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 -- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3bc0b0c30 msgr2=0x7fe3bc1f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3bc0b0c30 0x7fe3bc1f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 -- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3bc0cc4a0 msgr2=0x7fe3bc0b27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3bc0cc4a0 0x7fe3bc0b27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 -- 172.21.15.28:0/1282612788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a80094d0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1400640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400c780 tx=0x7fe3b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4004b60 con 0x7fe3bc0cc880 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b400ce90 con 0x7fe3bc0cc880 2024-09-13T17:19:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc0b19f0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.541+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b400a850 con 0x7fe3bc0cc880 2024-09-13T17:19:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.542+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0b2290 con 0x7fe3bc0cc880 2024-09-13T17:19:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.542+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0b1be0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.543+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe3b400a9b0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.544+0000 7fe3b8c00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.544+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3b4089460 con 0x7fe3bc0cc880 2024-09-13T17:19:33.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.545+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.546+0000 7fe3c0a00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 0x7fe39c06d0c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000a990 tx=0x7fe3b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.548+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe3b4054630 con 0x7fe3bc0cc880 2024-09-13T17:19:33.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.794+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fe3bc0b1de0 con 0x7fe3bc0cc880 2024-09-13T17:19:33.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.798+0000 7fe3b8c00640 1 -- 172.21.15.28:0/1282612788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v23) v1 ==== 190+0+0 (secure 0 0 0) 0x7fe3b4057c50 con 0x7fe3bc0cc880 2024-09-13T17:19:33.800 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 msgr2=0x7fe39c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 0x7fe39c06d0c0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000a990 tx=0x7fe3b0008040 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 msgr2=0x7fe3bc157c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fe3b400c780 tx=0x7fe3b400cc40 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 shutdown_connections 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe39c06ac00 0x7fe39c06d0c0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3bc0cc880 0x7fe3bc157c40 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3bc0cc4a0 0x7fe3bc0b27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.800+0000 7fe3c1e00640 1 --2- 172.21.15.28:0/1282612788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3bc0b0c30 0x7fe3bc1f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.801+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 >> 172.21.15.28:0/1282612788 conn(0x7fe3bc069e80 msgr2=0x7fe3bc15ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:33.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.801+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 shutdown_connections 2024-09-13T17:19:33.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.801+0000 7fe3c1e00640 1 -- 172.21.15.28:0/1282612788 wait complete. 2024-09-13T17:19:33.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.972+0000 7f5752800640 1 Processor -- start 2024-09-13T17:19:33.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.972+0000 7f5752800640 1 -- start start 2024-09-13T17:19:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.973+0000 7f5752800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f574c15a540 0x7f574c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.973+0000 7f5752800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f574c070cc0 0x7f574c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.973+0000 7f5752800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c15e100 con 0x7f574c0cabc0 2024-09-13T17:19:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.973+0000 7f5752800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0adb00 con 0x7f574c15a540 2024-09-13T17:19:33.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.973+0000 7f5752800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c071670 con 0x7f574c070cc0 2024-09-13T17:19:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f574c0cabc0 0x7f574c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45668/0 (socket says 172.21.15.28:45668) 2024-09-13T17:19:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574aa00640 1 -- 172.21.15.28:0/602712859 learned_addr learned my addr 172.21.15.28:0/602712859 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574b400640 1 --2- 172.21.15.28:0/602712859 >> v2:172.21.15.28:3300/0 conn(0x7f574c15a540 0x7f574c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574be00640 1 --2- 172.21.15.28:0/602712859 >> v2:172.21.15.189:3300/0 conn(0x7f574c070cc0 0x7f574c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574b400640 1 -- 172.21.15.28:0/602712859 >> v2:172.21.15.189:3300/0 conn(0x7f574c070cc0 msgr2=0x7f574c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574b400640 1 --2- 172.21.15.28:0/602712859 >> v2:172.21.15.189:3300/0 conn(0x7f574c070cc0 0x7f574c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574b400640 1 -- 172.21.15.28:0/602712859 >> v1:172.21.15.28:6789/0 conn(0x7f574c0cabc0 legacy=0x7f574c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574b400640 1 -- 172.21.15.28:0/602712859 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c15e3e0 con 0x7f574c15a540 2024-09-13T17:19:33.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.974+0000 7f574a000640 1 -- 172.21.15.28:0/602712859 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456286397 0 0) 0x7f574c15e100 con 0x7f574c0cabc0 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.975+0000 7f574b400640 1 --2- 172.21.15.28:0/602712859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c06c430 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f574c070900 tx=0x7f573c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48fdc3d0f48fdee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.975+0000 7f574a000640 1 -- 172.21.15.28:0/602712859 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c02ae50 con 0x7f574c15a540 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.975+0000 7f574a000640 1 -- 172.21.15.28:0/602712859 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c032440 con 0x7f574c15a540 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.975+0000 7f574a000640 1 -- 172.21.15.28:0/602712859 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0327d0 con 0x7f574c15a540 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 -- 172.21.15.28:0/602712859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 msgr2=0x7f574c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 --2- 172.21.15.28:0/602712859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c06c430 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f574c070900 tx=0x7f573c02a140 comp rx=0 tx=0).stop 2024-09-13T17:19:33.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 -- 172.21.15.28:0/602712859 shutdown_connections 2024-09-13T17:19:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 --2- 172.21.15.28:0/602712859 >> v2:172.21.15.189:3300/0 conn(0x7f574c070cc0 0x7f574c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 --2- 172.21.15.28:0/602712859 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c06c430 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 -- 172.21.15.28:0/602712859 >> 172.21.15.28:0/602712859 conn(0x7f574c069e80 msgr2=0x7f574c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 -- 172.21.15.28:0/602712859 shutdown_connections 2024-09-13T17:19:33.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.976+0000 7f5752800640 1 -- 172.21.15.28:0/602712859 wait complete. 2024-09-13T17:19:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.978+0000 7f5752800640 1 Processor -- start 2024-09-13T17:19:33.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.978+0000 7f5752800640 1 -- start start 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f574c070cc0 0x7f574c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0cabc0 0x7f574c159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c157e10 con 0x7f574c15a540 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c155a30 con 0x7f574c0cabc0 2024-09-13T17:19:33.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f5752800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c1577c0 con 0x7f574c070cc0 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36414/0 (socket says 172.21.15.28:36414) 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 -- 172.21.15.28:0/3231048041 learned_addr learned my addr 172.21.15.28:0/3231048041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574b400640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f574c070cc0 0x7f574c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574aa00640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0cabc0 0x7f574c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 -- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f574c070cc0 msgr2=0x7f574c1582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f574c070cc0 0x7f574c1582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 -- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0cabc0 msgr2=0x7f574c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0cabc0 0x7f574c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.979+0000 7f574be00640 1 -- 172.21.15.28:0/3231048041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c009400 con 0x7f574c15a540 2024-09-13T17:19:33.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f574be00640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f5744007e40 tx=0x7f574400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5744014620 con 0x7f574c15a540 2024-09-13T17:19:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574400ee90 con 0x7f574c15a540 2024-09-13T17:19:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57440149e0 con 0x7f574c15a540 2024-09-13T17:19:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c157aa0 con 0x7f574c15a540 2024-09-13T17:19:33.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.980+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c1557a0 con 0x7f574c15a540 2024-09-13T17:19:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.981+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c157f80 con 0x7f574c15a540 2024-09-13T17:19:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.982+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5744014b40 con 0x7f574c15a540 2024-09-13T17:19:33.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.983+0000 7f5743400640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 0x7f572c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.983+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5744089e10 con 0x7f574c15a540 2024-09-13T17:19:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.984+0000 7f574b400640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 0x7f572c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:33.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.984+0000 7f574b400640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 0x7f572c06d0c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f573c034000 tx=0x7f573c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:33.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:33.987+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f574408d030 con 0x7f574c15a540 2024-09-13T17:19:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.203+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f574c071130 con 0x7f574c15a540 2024-09-13T17:19:34.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.204+0000 7f5743400640 1 -- 172.21.15.28:0/3231048041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f5744054ec0 con 0x7f574c15a540 2024-09-13T17:19:34.206 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:34.206 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:34.206 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:34.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 msgr2=0x7f572c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 0x7f572c06d0c0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f573c034000 tx=0x7f573c02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:34.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 msgr2=0x7f574c159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f5744007e40 tx=0x7f574400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 shutdown_connections 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f572c06ac00 0x7f572c06d0c0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f574c15a540 0x7f574c159840 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f574c0cabc0 0x7f574c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 --2- 172.21.15.28:0/3231048041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f574c070cc0 0x7f574c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 >> 172.21.15.28:0/3231048041 conn(0x7f574c069e80 msgr2=0x7f574c15e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 shutdown_connections 2024-09-13T17:19:34.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.206+0000 7f5752800640 1 -- 172.21.15.28:0/3231048041 wait complete. 2024-09-13T17:19:34.267 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:19:34.265+0000 7f3ace200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:19:34.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 138, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:19:28.714008+0000', 'modified': '2024-09-13T17:19:31.732739+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10041}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10041': {'gid': 10041, 'name': 'c', 'rank': 0, 'incarnation': 134, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3709710897', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 3709710897}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3709710897}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:19:34.627 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:19:34.628 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:19:34.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.785+0000 7f1b86400640 1 Processor -- start 2024-09-13T17:19:34.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.785+0000 7f1b86400640 1 -- start start 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b86400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1b80159770 0x7f1b8006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b86400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1b8015cd80 0x7f1b80070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b86400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8006f920 con 0x7f1b8006f3f0 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b86400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8015b290 con 0x7f1b8015cd80 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b86400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8015b570 con 0x7f1b80159770 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b77e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b8006f3f0 0x7f1b80159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45682/0 (socket says 172.21.15.28:45682) 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b77e00640 1 -- 172.21.15.28:0/1742040491 learned_addr learned my addr 172.21.15.28:0/1742040491 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.786+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1742040491 >> v2:172.21.15.28:3300/0 conn(0x7f1b8015cd80 0x7f1b80070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1742040491 >> v2:172.21.15.189:3300/0 conn(0x7f1b80159770 0x7f1b8006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7ea00640 1 -- 172.21.15.28:0/1742040491 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20256874 0 0) 0x7f1b8006f920 con 0x7f1b8006f3f0 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7ea00640 1 -- 172.21.15.28:0/1742040491 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b64003520 con 0x7f1b8006f3f0 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1742040491 >> v2:172.21.15.189:3300/0 conn(0x7f1b80159770 msgr2=0x7f1b8006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1742040491 >> v2:172.21.15.189:3300/0 conn(0x7f1b80159770 0x7f1b8006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1742040491 >> v1:172.21.15.28:6789/0 conn(0x7f1b8006f3f0 legacy=0x7f1b80159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.787+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1742040491 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8015b850 con 0x7f1b8015cd80 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.788+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1742040491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80070260 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f1b80068040 tx=0x7f1b7802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9688eed815cff864 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.788+0000 7f1b7ea00640 1 -- 172.21.15.28:0/1742040491 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78007740 con 0x7f1b8015cd80 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.788+0000 7f1b7ea00640 1 -- 172.21.15.28:0/1742040491 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7802d050 con 0x7f1b8015cd80 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.788+0000 7f1b7ea00640 1 -- 172.21.15.28:0/1742040491 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78007ac0 con 0x7f1b8015cd80 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 -- 172.21.15.28:0/1742040491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 msgr2=0x7f1b80070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 --2- 172.21.15.28:0/1742040491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80070260 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7f1b80068040 tx=0x7f1b7802a5f0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 -- 172.21.15.28:0/1742040491 shutdown_connections 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 --2- 172.21.15.28:0/1742040491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80070260 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 --2- 172.21.15.28:0/1742040491 >> v2:172.21.15.189:3300/0 conn(0x7f1b80159770 0x7f1b8006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 -- 172.21.15.28:0/1742040491 >> 172.21.15.28:0/1742040491 conn(0x7f1b800b0270 msgr2=0x7f1b800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 -- 172.21.15.28:0/1742040491 shutdown_connections 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.789+0000 7f1b86400640 1 -- 172.21.15.28:0/1742040491 wait complete. 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.790+0000 7f1b86400640 1 Processor -- start 2024-09-13T17:19:34.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.791+0000 7f1b86400640 1 -- start start 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.791+0000 7f1b86400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b8006f3f0 0x7f1b8014cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.791+0000 7f1b86400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 0x7f1b80155380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b86400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b86400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800cd3e0 con 0x7f1b8015cd80 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b86400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b80155cf0 con 0x7f1b80159770 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b86400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b800ccd90 con 0x7f1b8006f3f0 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36434/0 (socket says 172.21.15.28:36434) 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1981688601 learned_addr learned my addr 172.21.15.28:0/1981688601 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 0x7f1b80155380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b8006f3f0 0x7f1b8014cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b8006f3f0 msgr2=0x7f1b8014cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b8006f3f0 0x7f1b8014cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 msgr2=0x7f1b80155380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 0x7f1b80155380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1981688601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78009a70 con 0x7f1b8015cd80 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.792+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 0x7f1b80155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f1b7802abe0 tx=0x7f1b78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b77400640 1 -- 172.21.15.28:0/1981688601 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7803c830 con 0x7f1b8015cd80 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b77400640 1 -- 172.21.15.28:0/1981688601 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7803c9b0 con 0x7f1b8015cd80 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b77400640 1 -- 172.21.15.28:0/1981688601 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7803cd60 con 0x7f1b8015cd80 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b800cd070 con 0x7f1b8015cd80 2024-09-13T17:19:34.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.793+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8014d480 con 0x7f1b8015cd80 2024-09-13T17:19:34.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.795+0000 7f1b77400640 1 -- 172.21.15.28:0/1981688601 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1b78031400 con 0x7f1b8015cd80 2024-09-13T17:19:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.796+0000 7f1b77400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 0x7f1b5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.796+0000 7f1b77400640 1 -- 172.21.15.28:0/1981688601 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b780aa820 con 0x7f1b8015cd80 2024-09-13T17:19:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.797+0000 7f1b89289740 1 Processor -- start 2024-09-13T17:19:34.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.797+0000 7f1b89289740 1 -- start start 2024-09-13T17:19:34.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.797+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 0x7f1b5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 0x7f1b5c06d0c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009ad0 tx=0x7f1b70008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68cae90 0x55b5e691aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e6836ab0 con 0x55b5e68f4b80 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e6834240 con 0x55b5e68cae90 2024-09-13T17:19:34.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5e682fb30 con 0x55b5e68cd6d0 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36438/0 (socket says 172.21.15.28:36438) 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1914225986 learned_addr learned my addr 172.21.15.28:0/1914225986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:48770/0 (socket says 172.21.15.28:48770) 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68cae90 0x55b5e691aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 msgr2=0x55b5e68cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68cae90 msgr2=0x55b5e691aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68cae90 0x55b5e691aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.798+0000 7f1b7fe00640 1 -- 172.21.15.28:0/1914225986 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5e67fabd0 con 0x55b5e68f4b80 2024-09-13T17:19:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f1b7802a630 tx=0x7f1b78079040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9826a1f1c043b35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b6fe00640 1 -- 172.21.15.28:0/1914225986 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78091070 con 0x55b5e68f4b80 2024-09-13T17:19:34.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b6fe00640 1 -- 172.21.15.28:0/1914225986 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7807d450 con 0x55b5e68f4b80 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b6fe00640 1 -- 172.21.15.28:0/1914225986 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b78081a10 con 0x55b5e68f4b80 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b89289740 1 -- 172.21.15.28:0/1914225986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 msgr2=0x55b5e691b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.799+0000 7f1b89289740 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f1b7802a630 tx=0x7f1b78079040 comp rx=0 tx=0).stop 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 -- 172.21.15.28:0/1914225986 shutdown_connections 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4b80 0x55b5e691b0e0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68cae90 0x55b5e691aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 --2- 172.21.15.28:0/1914225986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 -- 172.21.15.28:0/1914225986 >> 172.21.15.28:0/1914225986 conn(0x55b5e68c9e00 msgr2=0x55b5e69122b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 -- 172.21.15.28:0/1914225986 shutdown_connections 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 -- 172.21.15.28:0/1914225986 wait complete. 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 -- start start 2024-09-13T17:19:34.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.800+0000 7f1b89289740 1 Processor -- start 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e68eb080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e68eb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e67d1260 con 0x55b5e68cd6d0 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e67d0f00 con 0x55b5e68f4b80 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5e6596f20 con 0x55b5e68f4020 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.801+0000 7f1b89289740 10 client.0 _ll_register_callbacks cb 0x55b5e68291a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36440/0 (socket says 172.21.15.28:36440) 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 -- 172.21.15.28:0/1275555087 learned_addr learned my addr 172.21.15.28:0/1275555087 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e68eb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e68eb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 -- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 msgr2=0x55b5e68eb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e68eb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 -- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 msgr2=0x55b5e68eb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e68eb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 -- 172.21.15.28:0/1275555087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8014d480 con 0x55b5e68cd6d0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b7f400640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f1b700157e0 tx=0x7f1b700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b25219380ea5b64a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.802+0000 7f1b6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70014890 con 0x55b5e68cd6d0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b7001b040 con 0x55b5e68cd6d0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b5e6859cd0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b700149f0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 10 client.10176 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55b5e682e1b0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b5e6859cd0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1b70014b50 con 0x55b5e68cd6d0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 10 client.10176 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 10 client.10176 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 20 client.10176 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e68f10c0 0x55b5e68cb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 4 client.10176 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.803+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5e65233c0 con 0x55b5e68f10c0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.804+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e68f10c0 0x55b5e68cb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.804+0000 7f1b77e00640 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e68f10c0 0x55b5e68cb3a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcee2eddbed89948 server_cookie=6902245273366b4e in_seq=0 out_seq=0 2024-09-13T17:19:34.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.804+0000 7f1b6cc00640 10 client.10176 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b6cc00640 1 -- 172.21.15.28:0/1275555087 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55b5e65233c0 con 0x55b5e68f10c0 2024-09-13T17:19:34.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b6cc00640 10 client.10176 handle_command_reply: tid=0 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 client.10176 shutdown 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 msgr2=0x55b5e68f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7f1b700157e0 tx=0x7f1b700046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 shutdown_connections 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e68f10c0 0x55b5e68cb3a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e68eb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e68eb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 --2- 172.21.15.28:0/1275555087 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e68f3ae0 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b6cc00640 0 client.10176 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 >> 172.21.15.28:0/1275555087 conn(0x55b5e68cae90 msgr2=0x55b5e6912a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 shutdown_connections 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.805+0000 7f1b89289740 1 -- 172.21.15.28:0/1275555087 wait complete. 2024-09-13T17:19:34.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.806+0000 7f1b89289740 20 client.10176 trim_cache size 0 max 16384 2024-09-13T17:19:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.821+0000 7f1b89289740 1 Processor -- start 2024-09-13T17:19:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.821+0000 7f1b89289740 1 -- start start 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e6974570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e690e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e68ef2c0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e690f8c0 con 0x55b5e68f4b80 2024-09-13T17:19:34.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5e68cab10 con 0x55b5e68f4020 2024-09-13T17:19:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36442/0 (socket says 172.21.15.28:36442) 2024-09-13T17:19:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.822+0000 7f1b7f400640 1 -- 172.21.15.28:0/3307891034 learned_addr learned my addr 172.21.15.28:0/3307891034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 -- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 msgr2=0x55b5e6974570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e6974570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 -- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 msgr2=0x55b5e690e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e690e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 -- 172.21.15.28:0/3307891034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5e68eebd0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b7f400640 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f1b700116b0 tx=0x7f1b70022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88145104bb570fed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b74c00640 1 -- 172.21.15.28:0/3307891034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b700143e0 con 0x55b5e68cd6d0 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b74c00640 1 -- 172.21.15.28:0/3307891034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b70014890 con 0x55b5e68cd6d0 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.823+0000 7f1b74c00640 1 -- 172.21.15.28:0/3307891034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70014c00 con 0x55b5e68cd6d0 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 -- 172.21.15.28:0/3307891034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 msgr2=0x55b5e6974030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f1b700116b0 tx=0x7f1b70022000 comp rx=0 tx=0).stop 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 -- 172.21.15.28:0/3307891034 shutdown_connections 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e690e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68f4020 0x55b5e6974570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 --2- 172.21.15.28:0/3307891034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68cd6d0 0x55b5e6974030 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 -- 172.21.15.28:0/3307891034 >> 172.21.15.28:0/3307891034 conn(0x55b5e68cae90 msgr2=0x55b5e6911d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 -- 172.21.15.28:0/3307891034 shutdown_connections 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.824+0000 7f1b89289740 1 -- 172.21.15.28:0/3307891034 wait complete. 2024-09-13T17:19:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.825+0000 7f1b89289740 1 -- start start 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.825+0000 7f1b89289740 1 Processor -- start 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68c9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e6972a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e690f760 con 0x55b5e68f4020 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5e68ed3d0 con 0x55b5e68f4b80 2024-09-13T17:19:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b5e68c9080 con 0x55b5e68cd6d0 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b89289740 10 client.0 _ll_register_callbacks cb 0x7f1b700119e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36454/0 (socket says 172.21.15.28:36454) 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 -- 172.21.15.28:0/424239225 learned_addr learned my addr 172.21.15.28:0/424239225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b7f400640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b7fe00640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e6972a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 -- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 msgr2=0x55b5e68c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68c9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 -- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 msgr2=0x55b5e6972a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e6972a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b77e00640 1 -- 172.21.15.28:0/424239225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5e682e1b0 con 0x55b5e68f4020 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.826+0000 7f1b7f400640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b77e00640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f1b801f55b0 tx=0x7f1b6801ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb3de5c6ccafdad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b6cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b6801c070 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b680070c0 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b7807d450 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b68007280 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b89289740 10 client.10182 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55b5e6863170 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.827+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7807d450 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1b680073e0 con 0x55b5e68f4020 2024-09-13T17:19:34.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 10 client.10182 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 10 client.10182 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 20 client.10182 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e6973d30 0x55b5e6974110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 4 client.10182 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b5e65346c0 con 0x55b5e6973d30 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b7f400640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e6973d30 0x55b5e6974110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b7f400640 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e6973d30 0x55b5e6974110 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1334915bf3946df2 server_cookie=5b5532277f4c1a33 in_seq=0 out_seq=0 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.828+0000 7f1b6cc00640 10 client.10182 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b6cc00640 1 -- 172.21.15.28:0/424239225 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55b5e65346c0 con 0x55b5e6973d30 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b6cc00640 10 client.10182 handle_command_reply: tid=0 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 client.10182 shutdown 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 msgr2=0x55b5e68ec990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f1b801f55b0 tx=0x7f1b6801ced0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 shutdown_connections 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b5e6973d30 0x55b5e6974110 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b5e68f4b80 0x55b5e6972a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b5e68f4020 0x55b5e68ec990 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b6cc00640 0 client.10182 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 --2- 172.21.15.28:0/424239225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b5e68cd6d0 0x55b5e68c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 >> 172.21.15.28:0/424239225 conn(0x55b5e68cae90 msgr2=0x55b5e68cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 shutdown_connections 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 1 -- 172.21.15.28:0/424239225 wait complete. 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.829+0000 7f1b89289740 20 client.10182 trim_cache size 0 max 16384 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:34.835 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 10041, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.113783068, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 138, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 167, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 167, 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.233841748 2024-09-13T17:19:34.836 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 msgr2=0x7f1b5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 0x7f1b5c06d0c0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1b70009ad0 tx=0x7f1b70008040 comp rx=0 tx=0).stop 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 msgr2=0x7f1b80150b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f1b7802abe0 tx=0x7f1b78004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 shutdown_connections 2024-09-13T17:19:34.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5c06ac00 0x7f1b5c06d0c0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b8015cd80 0x7f1b80150b60 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b80159770 0x7f1b80155380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 --2- 172.21.15.28:0/1981688601 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b8006f3f0 0x7f1b8014cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 >> 172.21.15.28:0/1981688601 conn(0x7f1b800b0270 msgr2=0x7f1b8015a370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 shutdown_connections 2024-09-13T17:19:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:34.830+0000 7f1b86400640 1 -- 172.21.15.28:0/1981688601 wait complete. 2024-09-13T17:19:34.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:19:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.001+0000 7ffb07200640 1 Processor -- start 2024-09-13T17:19:35.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.001+0000 7ffb07200640 1 -- start start 2024-09-13T17:19:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.002+0000 7ffb07200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 0x7ffb0015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb07200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb00159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb07200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb0006c2d0 0x7ffb0006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb07200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0015b0b0 con 0x7ffb0006c2d0 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb07200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb0006cca0 con 0x7ffb001592a0 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb07200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000b2040 con 0x7ffb0006f970 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb05200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb0006c2d0 0x7ffb0006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb05200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ffb0006c2d0 0x7ffb0006c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36462/0 (socket says 172.21.15.28:36462) 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb05200640 1 -- 172.21.15.28:0/1228777500 learned_addr learned my addr 172.21.15.28:0/1228777500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffaffe00640 1 --2- 172.21.15.28:0/1228777500 >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 0x7ffb0015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb00159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.003+0000 7ffb05200640 1 -- 172.21.15.28:0/1228777500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 msgr2=0x7ffb00159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffb05200640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb00159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffb05200640 1 -- 172.21.15.28:0/1228777500 >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 msgr2=0x7ffb0015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffb05200640 1 --2- 172.21.15.28:0/1228777500 >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 0x7ffb0015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffb05200640 1 -- 172.21.15.28:0/1228777500 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb00159ff0 con 0x7ffb0006c2d0 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffaffe00640 1 --2- 172.21.15.28:0/1228777500 >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 0x7ffb0015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.004+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb00159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb05200640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006c2d0 0x7ffb0006c760 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7ffb0006d370 tx=0x7ffaf802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e80f7c1503119 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:35.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffaff400640 1 -- 172.21.15.28:0/1228777500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8031600 con 0x7ffb0006c2d0 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffaff400640 1 -- 172.21.15.28:0/1228777500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf802d050 con 0x7ffb0006c2d0 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffaff400640 1 -- 172.21.15.28:0/1228777500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8031980 con 0x7ffb0006c2d0 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 -- 172.21.15.28:0/1228777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006c2d0 msgr2=0x7ffb0006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006c2d0 0x7ffb0006c760 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7ffb0006d370 tx=0x7ffaf802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 -- 172.21.15.28:0/1228777500 shutdown_connections 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb0006c2d0 0x7ffb0006c760 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 --2- 172.21.15.28:0/1228777500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb00159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 --2- 172.21.15.28:0/1228777500 >> v2:172.21.15.189:3300/0 conn(0x7ffb001592a0 0x7ffb0015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.005+0000 7ffb07200640 1 -- 172.21.15.28:0/1228777500 >> 172.21.15.28:0/1228777500 conn(0x7ffb00069e80 msgr2=0x7ffb0006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:35.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.006+0000 7ffb07200640 1 -- 172.21.15.28:0/1228777500 shutdown_connections 2024-09-13T17:19:35.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.006+0000 7ffb07200640 1 -- 172.21.15.28:0/1228777500 wait complete. 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.007+0000 7ffb07200640 1 Processor -- start 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.007+0000 7ffb07200640 1 -- start start 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0006c2d0 0x7ffb001467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb000cb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000cc2d0 con 0x7ffb001592a0 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000cbd10 con 0x7ffb0006c2d0 2024-09-13T17:19:35.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb07200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb000cbff0 con 0x7ffb0006f970 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36464/0 (socket says 172.21.15.28:36464) 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.008+0000 7ffb05200640 1 -- 172.21.15.28:0/509929067 learned_addr learned my addr 172.21.15.28:0/509929067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffaffe00640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0006c2d0 0x7ffb001467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffaf7e00640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb000cb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 -- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 msgr2=0x7ffb000cb4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb000cb4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 -- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0006c2d0 msgr2=0x7ffb001467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0006c2d0 0x7ffb001467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 -- 172.21.15.28:0/509929067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae8009400 con 0x7ffb001592a0 2024-09-13T17:19:35.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.009+0000 7ffb05200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8034040 tx=0x7ffaf8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.010+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf803d860 con 0x7ffb001592a0 2024-09-13T17:19:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.010+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf803d9c0 con 0x7ffb001592a0 2024-09-13T17:19:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.010+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf803dd70 con 0x7ffb001592a0 2024-09-13T17:19:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.010+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf0009400 con 0x7ffb001592a0 2024-09-13T17:19:35.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.010+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf8009b30 con 0x7ffb001592a0 2024-09-13T17:19:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.011+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb0014a9d0 con 0x7ffb001592a0 2024-09-13T17:19:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.012+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffaf803b400 con 0x7ffb001592a0 2024-09-13T17:19:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.012+0000 7ffafcc00640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 0x7ffadc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.013+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffaf80a9fe0 con 0x7ffb001592a0 2024-09-13T17:19:35.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.013+0000 7ffaffe00640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 0x7ffadc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:35.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.014+0000 7ffaffe00640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 0x7ffadc06d0c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009bb0 tx=0x7ffaf0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.017+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffb0014a9d0 con 0x7ffb001592a0 2024-09-13T17:19:35.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.230+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffb0014a9d0 con 0x7ffb001592a0 2024-09-13T17:19:35.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.230+0000 7ffafcc00640 1 -- 172.21.15.28:0/509929067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v138) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffb0014a9d0 con 0x7ffb001592a0 2024-09-13T17:19:35.232 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 msgr2=0x7ffadc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 0x7ffadc06d0c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0009bb0 tx=0x7ffaf0009190 comp rx=0 tx=0).stop 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 msgr2=0x7ffb00152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7ffaf8034040 tx=0x7ffaf8004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 shutdown_connections 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffadc06ac00 0x7ffadc06d0c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb001592a0 0x7ffb00152520 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffb0006f970 0x7ffb000cb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 --2- 172.21.15.28:0/509929067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb0006c2d0 0x7ffb001467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:35.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 >> 172.21.15.28:0/509929067 conn(0x7ffb00069e80 msgr2=0x7ffb0006fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:35.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 shutdown_connections 2024-09-13T17:19:35.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:35.233+0000 7ffb07200640 1 -- 172.21.15.28:0/509929067 wait complete. 2024-09-13T17:19:35.250 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:19:35.250 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:19:35.251 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:19:35.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.251 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:19:35.287 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:19:35.288 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 42006sec preferred_lft 42006sec 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.289 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:19:35.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.291 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:19:35.382 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:19:35.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.385 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:19:35.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.422 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:19:35.422 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:35.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:19:35.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:19:35.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:19:35.423 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:35.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.567 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:19:35.604 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:19:35.604 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:19:35.604 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:19:35.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.604 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:19:35.701 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:19:35.701 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:19:35.737 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:19:35.737 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:19:35.737 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:19:35.811 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:19:35.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.812 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:19:35.940 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:19:35.940 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:19:35.967 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:19:35.967 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:19:35.967 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:19:35.967 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:19:35.967 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:19:35.968 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:19:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:19:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:19:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:19:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:19:35.972 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:19:35.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:35.972 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:19:36.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:36.028 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:19:36.072 INFO:teuthology.orchestra.run.smithi028.stdout:client9430 2024-09-13T17:19:36.073 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:19:36.073 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:19:36.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:36.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.345+0000 7fd148a00640 1 Processor -- start 2024-09-13T17:19:36.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.345+0000 7fd148a00640 1 -- start start 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd148a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1440b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd148a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd144159820 0x7fd1440b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd148a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1440c3850 con 0x7fd144159440 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd148a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1440b2d60 con 0x7fd144159820 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd148a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14415a930 con 0x7fd1440b0b50 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd143400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd144159440 0x7fd1440b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45696/0 (socket says 172.21.15.28:45696) 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd143400640 1 -- 172.21.15.28:0/1979692469 learned_addr learned my addr 172.21.15.28:0/1979692469 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd13aa00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1440b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.346+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> v2:172.21.15.189:3300/0 conn(0x7fd144159820 0x7fd1440b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049666549 0 0) 0x7fd1440c3850 con 0x7fd144159440 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd128003520 con 0x7fd144159440 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3884295835 0 0) 0x7fd128003520 con 0x7fd144159440 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 msgr2=0x7fd1440b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1440b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 >> v2:172.21.15.189:3300/0 conn(0x7fd144159820 msgr2=0x7fd1440b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 --2- 172.21.15.28:0/1979692469 >> v2:172.21.15.189:3300/0 conn(0x7fd144159820 0x7fd1440b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144159d20 con 0x7fd144159440 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fd134001670 con 0x7fd144159440 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 >> v1:172.21.15.28:6789/0 conn(0x7fd144159440 legacy=0x7fd1440b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128003970 0x7fd128003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128004d00 0x7fd128005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1440b2d60 con 0x7fd128004d00 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14415a930 con 0x7fd128003970 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12800db90 con 0x7fd1440b0b50 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> v2:172.21.15.189:3300/0 conn(0x7fd144159820 0x7fd1440b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128004d00 0x7fd128005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd143400640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128003970 0x7fd128003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.347+0000 7fd13aa00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.348+0000 7fd143e00640 1 -- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 msgr2=0x7fd1280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.348+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1440b0b50 0x7fd1280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.348+0000 7fd143e00640 1 -- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128003970 msgr2=0x7fd128003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.348+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128003970 0x7fd128003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.348+0000 7fd143e00640 1 -- 172.21.15.28:0/1979692469 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd128003520 con 0x7fd128004d00 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd143e00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128004d00 0x7fd128005180 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd14406d640 tx=0x7fd1280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcad2d1228224d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c02e610 con 0x7fd128004d00 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c00f040 con 0x7fd128004d00 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd142a00640 1 -- 172.21.15.28:0/1979692469 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c02e9b0 con 0x7fd128004d00 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd148a00640 1 -- 172.21.15.28:0/1979692469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128004d00 msgr2=0x7fd128005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd148a00640 1 --2- 172.21.15.28:0/1979692469 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd128004d00 0x7fd128005180 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fd14406d640 tx=0x7fd1280057f0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.349+0000 7fd143e00640 1 -- 172.21.15.28:0/1979692469 reap_dead start 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.350+0000 7fd148a00640 1 -- 172.21.15.28:0/1979692469 shutdown_connections 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.350+0000 7fd148a00640 1 -- 172.21.15.28:0/1979692469 >> 172.21.15.28:0/1979692469 conn(0x7fd144072e40 msgr2=0x7fd144073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.350+0000 7fd148a00640 1 -- 172.21.15.28:0/1979692469 shutdown_connections 2024-09-13T17:19:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.350+0000 7fd148a00640 1 -- 172.21.15.28:0/1979692469 wait complete. 2024-09-13T17:19:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.351+0000 7fd148a00640 1 Processor -- start 2024-09-13T17:19:36.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.351+0000 7fd148a00640 1 -- start start 2024-09-13T17:19:36.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128004d00 0x7fd1441e7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1441f5440 0x7fd1441f58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd144157dc0 con 0x7fd1441f5e00 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1441581e0 con 0x7fd128004d00 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.352+0000 7fd148a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd144157420 con 0x7fd1441f5440 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36476/0 (socket says 172.21.15.28:36476) 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 -- 172.21.15.28:0/3380498509 learned_addr learned my addr 172.21.15.28:0/3380498509 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143400640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128004d00 0x7fd1441e7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd13aa00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1441f5440 0x7fd1441f58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 -- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1441f5440 msgr2=0x7fd1441f58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1441f5440 0x7fd1441f58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 -- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128004d00 msgr2=0x7fd1441e7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128004d00 0x7fd1441e7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 -- 172.21.15.28:0/3380498509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144157700 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.353+0000 7fd143e00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02afc0 tx=0x7fd13c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.354+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c03d070 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.354+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c003990 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.354+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c003d00 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.354+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1441579e0 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.354+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1441420e0 con 0x7fd1441f5e00 2024-09-13T17:19:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.355+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1440b0440 con 0x7fd1441f5e00 2024-09-13T17:19:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.356+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd13c02c040 con 0x7fd1441f5e00 2024-09-13T17:19:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.356+0000 7fd13be00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 0x7fd12006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.357+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd13c0a9400 con 0x7fd1441f5e00 2024-09-13T17:19:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.357+0000 7fd143400640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 0x7fd12006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.358+0000 7fd143400640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 0x7fd12006d0f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd1441e3060 tx=0x7fd1340078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.361+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd13c0746a0 con 0x7fd1441f5e00 2024-09-13T17:19:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.575+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd144070340 con 0x7fd1441f5e00 2024-09-13T17:19:36.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.575+0000 7fd13be00640 1 -- 172.21.15.28:0/3380498509 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fd13c009de0 con 0x7fd1441f5e00 2024-09-13T17:19:36.577 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:36.577 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:36.577 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 msgr2=0x7fd12006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 0x7fd12006d0f0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fd1441e3060 tx=0x7fd1340078f0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 msgr2=0x7fd144141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fd13c02afc0 tx=0x7fd13c009750 comp rx=0 tx=0).stop 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 shutdown_connections 2024-09-13T17:19:36.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd12006ac30 0x7fd12006d0f0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1441f5e00 0x7fd144141ba0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd1441f5440 0x7fd1441f58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 --2- 172.21.15.28:0/3380498509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd128004d00 0x7fd1441e7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 >> 172.21.15.28:0/3380498509 conn(0x7fd144072e40 msgr2=0x7fd1440b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 shutdown_connections 2024-09-13T17:19:36.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.578+0000 7fd148a00640 1 -- 172.21.15.28:0/3380498509 wait complete. 2024-09-13T17:19:36.594 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:36.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.750+0000 7f9be7e00640 1 Processor -- start 2024-09-13T17:19:36.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.751+0000 7f9be7e00640 1 -- start start 2024-09-13T17:19:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 0x7f9be80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 0x7f9be80b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be80c3850 con 0x7f9be80b22e0 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be815a930 con 0x7f9be8159820 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be8159d20 con 0x7f9be8159440 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9be80b22e0 0x7f9be80b2790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:45708/0 (socket says 172.21.15.28:45708) 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be7400640 1 -- 172.21.15.28:0/3482592461 learned_addr learned my addr 172.21.15.28:0/3482592461 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be6a00640 1 --2- 172.21.15.28:0/3482592461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 0x7f9be80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9bdfe00640 1 --2- 172.21.15.28:0/3482592461 >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 0x7f9be80b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.752+0000 7f9be6000640 1 -- 172.21.15.28:0/3482592461 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2157578929 0 0) 0x7f9be80c3850 con 0x7f9be80b22e0 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9be6000640 1 -- 172.21.15.28:0/3482592461 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bcc003520 con 0x7f9be80b22e0 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9be6a00640 1 -- 172.21.15.28:0/3482592461 >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 msgr2=0x7f9be80b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9be6a00640 1 --2- 172.21.15.28:0/3482592461 >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 0x7f9be80b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9be6a00640 1 -- 172.21.15.28:0/3482592461 >> v1:172.21.15.28:6789/0 conn(0x7f9be80b22e0 legacy=0x7f9be80b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9be6a00640 1 -- 172.21.15.28:0/3482592461 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be815a000 con 0x7f9be8159440 2024-09-13T17:19:36.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.753+0000 7f9bdfe00640 1 --2- 172.21.15.28:0/3482592461 >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 0x7f9be80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:36.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.754+0000 7f9be6a00640 1 --2- 172.21.15.28:0/3482592461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 0x7f9be80b0440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800ba40 tx=0x7f9bd802c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a947004ac3aeefdc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.754+0000 7f9be6000640 1 -- 172.21.15.28:0/3482592461 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd80327e0 con 0x7f9be8159440 2024-09-13T17:19:36.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be6000640 1 -- 172.21.15.28:0/3482592461 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd8032940 con 0x7f9be8159440 2024-09-13T17:19:36.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be6000640 1 -- 172.21.15.28:0/3482592461 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd8032cb0 con 0x7f9be8159440 2024-09-13T17:19:36.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 -- 172.21.15.28:0/3482592461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 msgr2=0x7f9be80b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 --2- 172.21.15.28:0/3482592461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 0x7f9be80b0440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9bd800ba40 tx=0x7f9bd802c8e0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 -- 172.21.15.28:0/3482592461 shutdown_connections 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 --2- 172.21.15.28:0/3482592461 >> v2:172.21.15.189:3300/0 conn(0x7f9be8159820 0x7f9be80b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 --2- 172.21.15.28:0/3482592461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be8159440 0x7f9be80b0440 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 -- 172.21.15.28:0/3482592461 >> 172.21.15.28:0/3482592461 conn(0x7f9be8072e40 msgr2=0x7f9be8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.755+0000 7f9be7e00640 1 -- 172.21.15.28:0/3482592461 shutdown_connections 2024-09-13T17:19:36.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.756+0000 7f9be7e00640 1 -- 172.21.15.28:0/3482592461 wait complete. 2024-09-13T17:19:36.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.758+0000 7f9be7e00640 1 Processor -- start 2024-09-13T17:19:36.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.758+0000 7f9be7e00640 1 -- start start 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be80b22e0 0x7f9be81544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be8159440 0x7f9be814bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be814c8b0 con 0x7f9be8159820 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81f1270 con 0x7f9be8159440 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be81f1550 con 0x7f9be80b22e0 2024-09-13T17:19:36.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36478/0 (socket says 172.21.15.28:36478) 2024-09-13T17:19:36.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be7400640 1 -- 172.21.15.28:0/646570223 learned_addr learned my addr 172.21.15.28:0/646570223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:36.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9be6a00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be80b22e0 0x7f9be81544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.759+0000 7f9bdfe00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be8159440 0x7f9be814bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 -- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be80b22e0 msgr2=0x7f9be81544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be80b22e0 0x7f9be81544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 -- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be8159440 msgr2=0x7f9be814bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be8159440 0x7f9be814bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 -- 172.21.15.28:0/646570223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd0009400 con 0x7f9be8159820 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7400640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9be000caa0 tx=0x7f9be000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be000aba0 con 0x7f9be8159820 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be000ad00 con 0x7f9be8159820 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be0012620 con 0x7f9be8159820 2024-09-13T17:19:36.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd800b6a0 con 0x7f9be8159820 2024-09-13T17:19:36.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.760+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be81f1b10 con 0x7f9be8159820 2024-09-13T17:19:36.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.761+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be81f1d30 con 0x7f9be8159820 2024-09-13T17:19:36.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.762+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9be0007070 con 0x7f9be8159820 2024-09-13T17:19:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.763+0000 7f9bdf400640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 0x7f9bc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.763+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9be0089280 con 0x7f9be8159820 2024-09-13T17:19:36.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.763+0000 7f9be6a00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 0x7f9bc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:36.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.764+0000 7f9be6a00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 0x7f9bc406d0c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8037000 tx=0x7f9bd803e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:36.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.767+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9be0054470 con 0x7f9be8159820 2024-09-13T17:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.982+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9be80b2790 con 0x7f9be8159820 2024-09-13T17:19:36.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.983+0000 7f9bdf400640 1 -- 172.21.15.28:0/646570223 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f9be0057a90 con 0x7f9be8159820 2024-09-13T17:19:36.984 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:36.984 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:36.984 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 msgr2=0x7f9bc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 0x7f9bc406d0c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8037000 tx=0x7f9bd803e040 comp rx=0 tx=0).stop 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 msgr2=0x7f9be814c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9be000caa0 tx=0x7f9be000cf60 comp rx=0 tx=0).stop 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 shutdown_connections 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bc406ac00 0x7f9bc406d0c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be8159820 0x7f9be814c280 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be8159440 0x7f9be814bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 --2- 172.21.15.28:0/646570223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be80b22e0 0x7f9be81544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 >> 172.21.15.28:0/646570223 conn(0x7f9be8072e40 msgr2=0x7f9be8153050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 shutdown_connections 2024-09-13T17:19:36.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:36.985+0000 7f9be7e00640 1 -- 172.21.15.28:0/646570223 wait complete. 2024-09-13T17:19:37.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:37.002 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:37.002 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:37.002 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.0 2024-09-13T17:19:37.002 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:37.002 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:37.003 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:37.003 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:37.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205089 s, 51.1 MB/s 2024-09-13T17:19:37.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018257 s, 57.4 MB/s 2024-09-13T17:19:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167948 s, 62.4 MB/s 2024-09-13T17:19:37.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.133 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166686 s, 62.9 MB/s 2024-09-13T17:19:37.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.150 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158663 s, 66.1 MB/s 2024-09-13T17:19:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174103 s, 60.2 MB/s 2024-09-13T17:19:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.192 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201011 s, 52.2 MB/s 2024-09-13T17:19:37.207 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013273 s, 79.0 MB/s 2024-09-13T17:19:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136438 s, 76.9 MB/s 2024-09-13T17:19:37.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172729 s, 60.7 MB/s 2024-09-13T17:19:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132434 s, 79.2 MB/s 2024-09-13T17:19:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.276 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159333 s, 65.8 MB/s 2024-09-13T17:19:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.294 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151892 s, 69.0 MB/s 2024-09-13T17:19:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134136 s, 78.2 MB/s 2024-09-13T17:19:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136347 s, 76.9 MB/s 2024-09-13T17:19:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159921 s, 65.6 MB/s 2024-09-13T17:19:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134816 s, 77.8 MB/s 2024-09-13T17:19:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.378 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168283 s, 62.3 MB/s 2024-09-13T17:19:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.398 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177941 s, 58.9 MB/s 2024-09-13T17:19:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141492 s, 74.1 MB/s 2024-09-13T17:19:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133554 s, 78.5 MB/s 2024-09-13T17:19:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.447 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152057 s, 69.0 MB/s 2024-09-13T17:19:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.464 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148015 s, 70.8 MB/s 2024-09-13T17:19:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129194 s, 81.2 MB/s 2024-09-13T17:19:37.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155058 s, 67.6 MB/s 2024-09-13T17:19:37.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.515 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154409 s, 67.9 MB/s 2024-09-13T17:19:37.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.530 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012976 s, 80.8 MB/s 2024-09-13T17:19:37.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.547 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143469 s, 73.1 MB/s 2024-09-13T17:19:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.565 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164947 s, 63.6 MB/s 2024-09-13T17:19:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.583 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153732 s, 68.2 MB/s 2024-09-13T17:19:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.601 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162927 s, 64.4 MB/s 2024-09-13T17:19:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013478 s, 77.8 MB/s 2024-09-13T17:19:37.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.1 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:37.619 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171229 s, 61.2 MB/s 2024-09-13T17:19:37.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.694 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134774 s, 77.8 MB/s 2024-09-13T17:19:37.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126829 s, 82.7 MB/s 2024-09-13T17:19:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135414 s, 77.4 MB/s 2024-09-13T17:19:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.740 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013174 s, 79.6 MB/s 2024-09-13T17:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163091 s, 64.3 MB/s 2024-09-13T17:19:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.774 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134561 s, 77.9 MB/s 2024-09-13T17:19:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.789 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124941 s, 83.9 MB/s 2024-09-13T17:19:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142318 s, 73.7 MB/s 2024-09-13T17:19:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137255 s, 76.4 MB/s 2024-09-13T17:19:37.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128268 s, 81.7 MB/s 2024-09-13T17:19:37.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.856 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167851 s, 62.5 MB/s 2024-09-13T17:19:37.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.870 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126623 s, 82.8 MB/s 2024-09-13T17:19:37.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133172 s, 78.7 MB/s 2024-09-13T17:19:37.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.902 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133973 s, 78.3 MB/s 2024-09-13T17:19:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170151 s, 61.6 MB/s 2024-09-13T17:19:37.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145932 s, 71.9 MB/s 2024-09-13T17:19:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140952 s, 74.4 MB/s 2024-09-13T17:19:37.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122376 s, 85.7 MB/s 2024-09-13T17:19:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:37.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145627 s, 72.0 MB/s 2024-09-13T17:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135946 s, 77.1 MB/s 2024-09-13T17:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132691 s, 79.0 MB/s 2024-09-13T17:19:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.035 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156468 s, 67.0 MB/s 2024-09-13T17:19:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129834 s, 80.8 MB/s 2024-09-13T17:19:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132202 s, 79.3 MB/s 2024-09-13T17:19:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.081 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135971 s, 77.1 MB/s 2024-09-13T17:19:38.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.097 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.097 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133661 s, 78.5 MB/s 2024-09-13T17:19:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013815 s, 75.9 MB/s 2024-09-13T17:19:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128244 s, 81.8 MB/s 2024-09-13T17:19:38.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.143 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133389 s, 78.6 MB/s 2024-09-13T17:19:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133048 s, 78.8 MB/s 2024-09-13T17:19:38.172 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.172 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.172 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117844 s, 89.0 MB/s 2024-09-13T17:19:38.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.2 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:38.174 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165933 s, 63.2 MB/s 2024-09-13T17:19:38.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134388 s, 78.0 MB/s 2024-09-13T17:19:38.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151344 s, 69.3 MB/s 2024-09-13T17:19:38.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122533 s, 85.6 MB/s 2024-09-13T17:19:38.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131862 s, 79.5 MB/s 2024-09-13T17:19:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174664 s, 60.0 MB/s 2024-09-13T17:19:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158191 s, 66.3 MB/s 2024-09-13T17:19:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.348 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133548 s, 78.5 MB/s 2024-09-13T17:19:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133404 s, 78.6 MB/s 2024-09-13T17:19:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139679 s, 75.1 MB/s 2024-09-13T17:19:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133707 s, 78.4 MB/s 2024-09-13T17:19:38.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.416 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.416 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171437 s, 61.2 MB/s 2024-09-13T17:19:38.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.432 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143307 s, 73.2 MB/s 2024-09-13T17:19:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.451 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170954 s, 61.3 MB/s 2024-09-13T17:19:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.466 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130846 s, 80.1 MB/s 2024-09-13T17:19:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.483 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142366 s, 73.7 MB/s 2024-09-13T17:19:38.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.498 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.498 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126675 s, 82.8 MB/s 2024-09-13T17:19:38.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131392 s, 79.8 MB/s 2024-09-13T17:19:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.529 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136114 s, 77.0 MB/s 2024-09-13T17:19:38.544 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.544 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.544 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127586 s, 82.2 MB/s 2024-09-13T17:19:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132832 s, 78.9 MB/s 2024-09-13T17:19:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013592 s, 77.1 MB/s 2024-09-13T17:19:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147626 s, 71.0 MB/s 2024-09-13T17:19:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135938 s, 77.1 MB/s 2024-09-13T17:19:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.623 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126793 s, 82.7 MB/s 2024-09-13T17:19:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139128 s, 75.4 MB/s 2024-09-13T17:19:38.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136974 s, 76.6 MB/s 2024-09-13T17:19:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.674 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.675 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167597 s, 62.6 MB/s 2024-09-13T17:19:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133166 s, 78.7 MB/s 2024-09-13T17:19:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135994 s, 77.1 MB/s 2024-09-13T17:19:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.722 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142057 s, 73.8 MB/s 2024-09-13T17:19:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136549 s, 76.8 MB/s 2024-09-13T17:19:38.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.3 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:38.740 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:38.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156144 s, 67.2 MB/s 2024-09-13T17:19:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.816 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170079 s, 61.7 MB/s 2024-09-13T17:19:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134822 s, 77.8 MB/s 2024-09-13T17:19:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130546 s, 80.3 MB/s 2024-09-13T17:19:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128405 s, 81.7 MB/s 2024-09-13T17:19:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130941 s, 80.1 MB/s 2024-09-13T17:19:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.893 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137123 s, 76.5 MB/s 2024-09-13T17:19:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164126 s, 63.9 MB/s 2024-09-13T17:19:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130231 s, 80.5 MB/s 2024-09-13T17:19:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.943 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138893 s, 75.5 MB/s 2024-09-13T17:19:38.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.960 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141463 s, 74.1 MB/s 2024-09-13T17:19:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131621 s, 79.7 MB/s 2024-09-13T17:19:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:38.989 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011718 s, 89.5 MB/s 2024-09-13T17:19:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131508 s, 79.7 MB/s 2024-09-13T17:19:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132131 s, 79.4 MB/s 2024-09-13T17:19:39.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142253 s, 73.7 MB/s 2024-09-13T17:19:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166148 s, 63.1 MB/s 2024-09-13T17:19:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115784 s, 90.6 MB/s 2024-09-13T17:19:39.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140055 s, 74.9 MB/s 2024-09-13T17:19:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128513 s, 81.6 MB/s 2024-09-13T17:19:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131806 s, 79.6 MB/s 2024-09-13T17:19:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.132 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136159 s, 77.0 MB/s 2024-09-13T17:19:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130223 s, 80.5 MB/s 2024-09-13T17:19:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139752 s, 75.0 MB/s 2024-09-13T17:19:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138825 s, 75.5 MB/s 2024-09-13T17:19:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.199 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162259 s, 64.6 MB/s 2024-09-13T17:19:39.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.214 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130748 s, 80.2 MB/s 2024-09-13T17:19:39.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127301 s, 82.4 MB/s 2024-09-13T17:19:39.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144253 s, 72.7 MB/s 2024-09-13T17:19:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016921 s, 62.0 MB/s 2024-09-13T17:19:39.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136451 s, 76.8 MB/s 2024-09-13T17:19:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136874 s, 76.6 MB/s 2024-09-13T17:19:39.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.4 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:39.298 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:39.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.356 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160673 s, 65.3 MB/s 2024-09-13T17:19:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133369 s, 78.6 MB/s 2024-09-13T17:19:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012974 s, 80.8 MB/s 2024-09-13T17:19:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.406 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177423 s, 59.1 MB/s 2024-09-13T17:19:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.424 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155534 s, 67.4 MB/s 2024-09-13T17:19:39.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128018 s, 81.9 MB/s 2024-09-13T17:19:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136759 s, 76.7 MB/s 2024-09-13T17:19:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142244 s, 73.7 MB/s 2024-09-13T17:19:39.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.487 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130327 s, 80.5 MB/s 2024-09-13T17:19:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113583 s, 92.3 MB/s 2024-09-13T17:19:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.515 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129677 s, 80.9 MB/s 2024-09-13T17:19:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141343 s, 74.2 MB/s 2024-09-13T17:19:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016072 s, 65.2 MB/s 2024-09-13T17:19:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.565 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128668 s, 81.5 MB/s 2024-09-13T17:19:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.579 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125042 s, 83.9 MB/s 2024-09-13T17:19:39.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.595 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127152 s, 82.5 MB/s 2024-09-13T17:19:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147084 s, 71.3 MB/s 2024-09-13T17:19:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.627 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132145 s, 79.4 MB/s 2024-09-13T17:19:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130422 s, 80.4 MB/s 2024-09-13T17:19:39.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.658 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131626 s, 79.7 MB/s 2024-09-13T17:19:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123574 s, 84.9 MB/s 2024-09-13T17:19:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154088 s, 68.1 MB/s 2024-09-13T17:19:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132016 s, 79.4 MB/s 2024-09-13T17:19:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.721 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133123 s, 78.8 MB/s 2024-09-13T17:19:39.737 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.738 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014327 s, 73.2 MB/s 2024-09-13T17:19:39.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162864 s, 64.4 MB/s 2024-09-13T17:19:39.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129071 s, 81.2 MB/s 2024-09-13T17:19:39.787 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138649 s, 75.6 MB/s 2024-09-13T17:19:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134083 s, 78.2 MB/s 2024-09-13T17:19:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.818 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133185 s, 78.7 MB/s 2024-09-13T17:19:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.834 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013466 s, 77.9 MB/s 2024-09-13T17:19:39.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134924 s, 77.7 MB/s 2024-09-13T17:19:39.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:39.851 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:39.851 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:39.851 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.5 2024-09-13T17:19:39.851 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:39.851 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:39.852 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:39.852 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:39.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129856 s, 80.7 MB/s 2024-09-13T17:19:39.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.921 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134263 s, 78.1 MB/s 2024-09-13T17:19:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171504 s, 61.1 MB/s 2024-09-13T17:19:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137473 s, 76.3 MB/s 2024-09-13T17:19:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124197 s, 84.4 MB/s 2024-09-13T17:19:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:39.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145353 s, 72.1 MB/s 2024-09-13T17:19:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158868 s, 66.0 MB/s 2024-09-13T17:19:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.022 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134841 s, 77.8 MB/s 2024-09-13T17:19:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149267 s, 70.2 MB/s 2024-09-13T17:19:40.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171735 s, 61.1 MB/s 2024-09-13T17:19:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013259 s, 79.1 MB/s 2024-09-13T17:19:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168043 s, 62.4 MB/s 2024-09-13T17:19:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134574 s, 77.9 MB/s 2024-09-13T17:19:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136487 s, 76.8 MB/s 2024-09-13T17:19:40.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.141 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134802 s, 77.8 MB/s 2024-09-13T17:19:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130801 s, 80.2 MB/s 2024-09-13T17:19:40.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125631 s, 83.5 MB/s 2024-09-13T17:19:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013844 s, 75.7 MB/s 2024-09-13T17:19:40.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160132 s, 65.5 MB/s 2024-09-13T17:19:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128545 s, 81.6 MB/s 2024-09-13T17:19:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147385 s, 71.1 MB/s 2024-09-13T17:19:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135592 s, 77.3 MB/s 2024-09-13T17:19:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.272 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167015 s, 62.8 MB/s 2024-09-13T17:19:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.288 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138046 s, 76.0 MB/s 2024-09-13T17:19:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.304 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135562 s, 77.4 MB/s 2024-09-13T17:19:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.323 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172196 s, 60.9 MB/s 2024-09-13T17:19:40.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013215 s, 79.3 MB/s 2024-09-13T17:19:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.355 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136889 s, 76.6 MB/s 2024-09-13T17:19:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135391 s, 77.4 MB/s 2024-09-13T17:19:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151105 s, 69.4 MB/s 2024-09-13T17:19:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012987 s, 80.7 MB/s 2024-09-13T17:19:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134622 s, 77.9 MB/s 2024-09-13T17:19:40.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.6 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:40.421 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137135 s, 76.5 MB/s 2024-09-13T17:19:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.492 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136795 s, 76.7 MB/s 2024-09-13T17:19:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145256 s, 72.2 MB/s 2024-09-13T17:19:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141212 s, 74.3 MB/s 2024-09-13T17:19:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135937 s, 77.1 MB/s 2024-09-13T17:19:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.556 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132264 s, 79.3 MB/s 2024-09-13T17:19:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169659 s, 61.8 MB/s 2024-09-13T17:19:40.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137429 s, 76.3 MB/s 2024-09-13T17:19:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.607 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132454 s, 79.2 MB/s 2024-09-13T17:19:40.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.622 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.622 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012717 s, 82.5 MB/s 2024-09-13T17:19:40.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.638 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132671 s, 79.0 MB/s 2024-09-13T17:19:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168438 s, 62.3 MB/s 2024-09-13T17:19:40.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144614 s, 72.5 MB/s 2024-09-13T17:19:40.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.689 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129283 s, 81.1 MB/s 2024-09-13T17:19:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.704 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130609 s, 80.3 MB/s 2024-09-13T17:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125317 s, 83.7 MB/s 2024-09-13T17:19:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013908 s, 75.4 MB/s 2024-09-13T17:19:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170117 s, 61.6 MB/s 2024-09-13T17:19:40.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137669 s, 76.2 MB/s 2024-09-13T17:19:40.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129466 s, 81.0 MB/s 2024-09-13T17:19:40.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129803 s, 80.8 MB/s 2024-09-13T17:19:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.816 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135061 s, 77.6 MB/s 2024-09-13T17:19:40.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137479 s, 76.3 MB/s 2024-09-13T17:19:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124707 s, 84.1 MB/s 2024-09-13T17:19:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136255 s, 77.0 MB/s 2024-09-13T17:19:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135563 s, 77.3 MB/s 2024-09-13T17:19:40.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.895 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134836 s, 77.8 MB/s 2024-09-13T17:19:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013591 s, 77.2 MB/s 2024-09-13T17:19:40.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129819 s, 80.8 MB/s 2024-09-13T17:19:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133363 s, 78.6 MB/s 2024-09-13T17:19:40.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142298 s, 73.7 MB/s 2024-09-13T17:19:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:40.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015937 s, 65.8 MB/s 2024-09-13T17:19:40.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:40.977 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.7 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:40.978 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:41.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013281 s, 79.0 MB/s 2024-09-13T17:19:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.051 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164032 s, 63.9 MB/s 2024-09-13T17:19:41.066 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:19:41.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130388 s, 80.4 MB/s 2024-09-13T17:19:41.070 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:19:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.082 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013462 s, 77.9 MB/s 2024-09-13T17:19:41.097 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.097 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133836 s, 78.3 MB/s 2024-09-13T17:19:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.117 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171016 s, 61.3 MB/s 2024-09-13T17:19:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128927 s, 81.3 MB/s 2024-09-13T17:19:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.148 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135624 s, 77.3 MB/s 2024-09-13T17:19:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.164 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135556 s, 77.4 MB/s 2024-09-13T17:19:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134495 s, 78.0 MB/s 2024-09-13T17:19:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119502 s, 87.7 MB/s 2024-09-13T17:19:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130564 s, 80.3 MB/s 2024-09-13T17:19:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.226 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145255 s, 72.2 MB/s 2024-09-13T17:19:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163796 s, 64.0 MB/s 2024-09-13T17:19:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153501 s, 68.3 MB/s 2024-09-13T17:19:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.279 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143461 s, 73.1 MB/s 2024-09-13T17:19:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.297 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158933 s, 66.0 MB/s 2024-09-13T17:19:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124755 s, 84.1 MB/s 2024-09-13T17:19:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013988 s, 75.0 MB/s 2024-09-13T17:19:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133104 s, 78.8 MB/s 2024-09-13T17:19:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129233 s, 81.1 MB/s 2024-09-13T17:19:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135823 s, 77.2 MB/s 2024-09-13T17:19:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.389 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118058 s, 88.8 MB/s 2024-09-13T17:19:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140199 s, 74.8 MB/s 2024-09-13T17:19:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012563 s, 83.5 MB/s 2024-09-13T17:19:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.436 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138557 s, 75.7 MB/s 2024-09-13T17:19:41.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.454 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156014 s, 67.2 MB/s 2024-09-13T17:19:41.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138241 s, 75.9 MB/s 2024-09-13T17:19:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.485 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130218 s, 80.5 MB/s 2024-09-13T17:19:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167732 s, 62.5 MB/s 2024-09-13T17:19:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130401 s, 80.4 MB/s 2024-09-13T17:19:41.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132356 s, 79.2 MB/s 2024-09-13T17:19:41.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:41.536 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:41.536 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:41.537 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.8 2024-09-13T17:19:41.537 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:41.537 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:41.537 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:41.537 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148027 s, 70.8 MB/s 2024-09-13T17:19:41.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128969 s, 81.3 MB/s 2024-09-13T17:19:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172254 s, 60.9 MB/s 2024-09-13T17:19:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168948 s, 62.1 MB/s 2024-09-13T17:19:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161421 s, 65.0 MB/s 2024-09-13T17:19:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.723 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156594 s, 67.0 MB/s 2024-09-13T17:19:41.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135004 s, 77.7 MB/s 2024-09-13T17:19:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134218 s, 78.1 MB/s 2024-09-13T17:19:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131829 s, 79.5 MB/s 2024-09-13T17:19:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138307 s, 75.8 MB/s 2024-09-13T17:19:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167133 s, 62.7 MB/s 2024-09-13T17:19:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.821 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013922 s, 75.3 MB/s 2024-09-13T17:19:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128485 s, 81.6 MB/s 2024-09-13T17:19:41.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163621 s, 64.1 MB/s 2024-09-13T17:19:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133316 s, 78.7 MB/s 2024-09-13T17:19:41.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137508 s, 76.3 MB/s 2024-09-13T17:19:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129096 s, 81.2 MB/s 2024-09-13T17:19:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016492 s, 63.6 MB/s 2024-09-13T17:19:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131352 s, 79.8 MB/s 2024-09-13T17:19:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127307 s, 82.4 MB/s 2024-09-13T17:19:41.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140999 s, 74.4 MB/s 2024-09-13T17:19:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013547 s, 77.4 MB/s 2024-09-13T17:19:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126114 s, 83.1 MB/s 2024-09-13T17:19:42.011 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.011 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119171 s, 88.0 MB/s 2024-09-13T17:19:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134565 s, 77.9 MB/s 2024-09-13T17:19:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164371 s, 63.8 MB/s 2024-09-13T17:19:42.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.061 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130695 s, 80.2 MB/s 2024-09-13T17:19:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013984 s, 75.0 MB/s 2024-09-13T17:19:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157721 s, 66.5 MB/s 2024-09-13T17:19:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.115 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169391 s, 61.9 MB/s 2024-09-13T17:19:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169305 s, 61.9 MB/s 2024-09-13T17:19:42.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.150 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.150 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140139 s, 74.8 MB/s 2024-09-13T17:19:42.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.9 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:42.152 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131221 s, 79.9 MB/s 2024-09-13T17:19:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165053 s, 63.5 MB/s 2024-09-13T17:19:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.244 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169221 s, 62.0 MB/s 2024-09-13T17:19:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.260 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130734 s, 80.2 MB/s 2024-09-13T17:19:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129335 s, 81.1 MB/s 2024-09-13T17:19:42.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.291 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.291 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136053 s, 77.1 MB/s 2024-09-13T17:19:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157804 s, 66.4 MB/s 2024-09-13T17:19:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128349 s, 81.7 MB/s 2024-09-13T17:19:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131024 s, 80.0 MB/s 2024-09-13T17:19:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129728 s, 80.8 MB/s 2024-09-13T17:19:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.374 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170782 s, 61.4 MB/s 2024-09-13T17:19:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.392 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163537 s, 64.1 MB/s 2024-09-13T17:19:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128214 s, 81.8 MB/s 2024-09-13T17:19:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130049 s, 80.6 MB/s 2024-09-13T17:19:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153933 s, 68.1 MB/s 2024-09-13T17:19:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128741 s, 81.4 MB/s 2024-09-13T17:19:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.470 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122215 s, 85.8 MB/s 2024-09-13T17:19:42.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.489 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165065 s, 63.5 MB/s 2024-09-13T17:19:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132741 s, 79.0 MB/s 2024-09-13T17:19:42.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132935 s, 78.9 MB/s 2024-09-13T17:19:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135695 s, 77.3 MB/s 2024-09-13T17:19:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136229 s, 77.0 MB/s 2024-09-13T17:19:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.566 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127068 s, 82.5 MB/s 2024-09-13T17:19:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166172 s, 63.1 MB/s 2024-09-13T17:19:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128803 s, 81.4 MB/s 2024-09-13T17:19:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.615 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131149 s, 80.0 MB/s 2024-09-13T17:19:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.631 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134336 s, 78.1 MB/s 2024-09-13T17:19:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128746 s, 81.4 MB/s 2024-09-13T17:19:42.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116222 s, 90.2 MB/s 2024-09-13T17:19:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.678 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165451 s, 63.4 MB/s 2024-09-13T17:19:42.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125283 s, 83.7 MB/s 2024-09-13T17:19:42.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.709 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.709 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126624 s, 82.8 MB/s 2024-09-13T17:19:42.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:42.710 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.10 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:42.711 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:42.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124362 s, 84.3 MB/s 2024-09-13T17:19:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.782 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135261 s, 77.5 MB/s 2024-09-13T17:19:42.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128841 s, 81.4 MB/s 2024-09-13T17:19:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141101 s, 74.3 MB/s 2024-09-13T17:19:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168294 s, 62.3 MB/s 2024-09-13T17:19:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172582 s, 60.8 MB/s 2024-09-13T17:19:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132978 s, 78.9 MB/s 2024-09-13T17:19:42.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.883 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135073 s, 77.6 MB/s 2024-09-13T17:19:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.898 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126595 s, 82.8 MB/s 2024-09-13T17:19:42.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132019 s, 79.4 MB/s 2024-09-13T17:19:42.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.929 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.929 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132642 s, 79.1 MB/s 2024-09-13T17:19:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.944 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130153 s, 80.6 MB/s 2024-09-13T17:19:42.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130673 s, 80.2 MB/s 2024-09-13T17:19:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.974 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127982 s, 81.9 MB/s 2024-09-13T17:19:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:42.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131657 s, 79.6 MB/s 2024-09-13T17:19:43.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135973 s, 77.1 MB/s 2024-09-13T17:19:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163792 s, 64.0 MB/s 2024-09-13T17:19:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.040 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140837 s, 74.5 MB/s 2024-09-13T17:19:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133556 s, 78.5 MB/s 2024-09-13T17:19:43.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.075 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016512 s, 63.5 MB/s 2024-09-13T17:19:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.091 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134213 s, 78.1 MB/s 2024-09-13T17:19:43.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.105 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.105 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128238 s, 81.8 MB/s 2024-09-13T17:19:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171351 s, 61.2 MB/s 2024-09-13T17:19:43.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012708 s, 82.5 MB/s 2024-09-13T17:19:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135637 s, 77.3 MB/s 2024-09-13T17:19:43.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.172 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.172 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140239 s, 74.8 MB/s 2024-09-13T17:19:43.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.191 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.191 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170444 s, 61.5 MB/s 2024-09-13T17:19:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145781 s, 71.9 MB/s 2024-09-13T17:19:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128927 s, 81.3 MB/s 2024-09-13T17:19:43.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181762 s, 57.7 MB/s 2024-09-13T17:19:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135065 s, 77.6 MB/s 2024-09-13T17:19:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.274 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127775 s, 82.1 MB/s 2024-09-13T17:19:43.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.11 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:43.276 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.332 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133119 s, 78.8 MB/s 2024-09-13T17:19:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168145 s, 62.4 MB/s 2024-09-13T17:19:43.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141794 s, 74.0 MB/s 2024-09-13T17:19:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127718 s, 82.1 MB/s 2024-09-13T17:19:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158795 s, 66.0 MB/s 2024-09-13T17:19:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134343 s, 78.1 MB/s 2024-09-13T17:19:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013445 s, 78.0 MB/s 2024-09-13T17:19:43.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.446 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.446 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134899 s, 77.7 MB/s 2024-09-13T17:19:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012775 s, 82.1 MB/s 2024-09-13T17:19:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130377 s, 80.4 MB/s 2024-09-13T17:19:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.496 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172925 s, 60.6 MB/s 2024-09-13T17:19:43.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127038 s, 82.5 MB/s 2024-09-13T17:19:43.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.526 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127488 s, 82.2 MB/s 2024-09-13T17:19:43.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127546 s, 82.2 MB/s 2024-09-13T17:19:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153548 s, 68.3 MB/s 2024-09-13T17:19:43.573 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130203 s, 80.5 MB/s 2024-09-13T17:19:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142792 s, 73.4 MB/s 2024-09-13T17:19:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186288 s, 56.3 MB/s 2024-09-13T17:19:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127807 s, 82.0 MB/s 2024-09-13T17:19:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146126 s, 71.8 MB/s 2024-09-13T17:19:43.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150781 s, 69.5 MB/s 2024-09-13T17:19:43.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.678 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.678 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128294 s, 81.7 MB/s 2024-09-13T17:19:43.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.697 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157082 s, 66.8 MB/s 2024-09-13T17:19:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.714 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156577 s, 67.0 MB/s 2024-09-13T17:19:43.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.730 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130588 s, 80.3 MB/s 2024-09-13T17:19:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.746 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140253 s, 74.8 MB/s 2024-09-13T17:19:43.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154528 s, 67.9 MB/s 2024-09-13T17:19:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128755 s, 81.4 MB/s 2024-09-13T17:19:43.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129649 s, 80.9 MB/s 2024-09-13T17:19:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125042 s, 83.9 MB/s 2024-09-13T17:19:43.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.823 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115356 s, 90.9 MB/s 2024-09-13T17:19:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136892 s, 76.6 MB/s 2024-09-13T17:19:43.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.12 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:43.840 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127743 s, 82.1 MB/s 2024-09-13T17:19:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016612 s, 63.1 MB/s 2024-09-13T17:19:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126153 s, 83.1 MB/s 2024-09-13T17:19:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.947 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169195 s, 62.0 MB/s 2024-09-13T17:19:43.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.963 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.964 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142498 s, 73.6 MB/s 2024-09-13T17:19:43.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.980 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140793 s, 74.5 MB/s 2024-09-13T17:19:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:43.996 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140002 s, 74.9 MB/s 2024-09-13T17:19:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166284 s, 63.1 MB/s 2024-09-13T17:19:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127885 s, 82.0 MB/s 2024-09-13T17:19:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.046 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01412 s, 74.3 MB/s 2024-09-13T17:19:44.063 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.064 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155781 s, 67.3 MB/s 2024-09-13T17:19:44.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.083 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165569 s, 63.3 MB/s 2024-09-13T17:19:44.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149902 s, 70.0 MB/s 2024-09-13T17:19:44.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.115 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127188 s, 82.4 MB/s 2024-09-13T17:19:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.134 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017171 s, 61.1 MB/s 2024-09-13T17:19:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013185 s, 79.5 MB/s 2024-09-13T17:19:44.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130969 s, 80.1 MB/s 2024-09-13T17:19:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.181 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131163 s, 79.9 MB/s 2024-09-13T17:19:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.198 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158335 s, 66.2 MB/s 2024-09-13T17:19:44.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.214 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126551 s, 82.9 MB/s 2024-09-13T17:19:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153574 s, 68.3 MB/s 2024-09-13T17:19:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152904 s, 68.6 MB/s 2024-09-13T17:19:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123812 s, 84.7 MB/s 2024-09-13T17:19:44.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.280 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.280 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014437 s, 72.6 MB/s 2024-09-13T17:19:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141246 s, 74.2 MB/s 2024-09-13T17:19:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139207 s, 75.3 MB/s 2024-09-13T17:19:44.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134172 s, 78.2 MB/s 2024-09-13T17:19:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012849 s, 81.6 MB/s 2024-09-13T17:19:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137044 s, 76.5 MB/s 2024-09-13T17:19:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171818 s, 61.0 MB/s 2024-09-13T17:19:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154517 s, 67.9 MB/s 2024-09-13T17:19:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131941 s, 79.5 MB/s 2024-09-13T17:19:44.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.13 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:44.414 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138832 s, 75.5 MB/s 2024-09-13T17:19:44.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123139 s, 85.2 MB/s 2024-09-13T17:19:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134523 s, 77.9 MB/s 2024-09-13T17:19:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141775 s, 74.0 MB/s 2024-09-13T17:19:44.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013968 s, 75.1 MB/s 2024-09-13T17:19:44.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135998 s, 77.1 MB/s 2024-09-13T17:19:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.569 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160331 s, 65.4 MB/s 2024-09-13T17:19:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144587 s, 72.5 MB/s 2024-09-13T17:19:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121023 s, 86.6 MB/s 2024-09-13T17:19:44.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.615 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.615 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129924 s, 80.7 MB/s 2024-09-13T17:19:44.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.629 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110989 s, 94.5 MB/s 2024-09-13T17:19:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170799 s, 61.4 MB/s 2024-09-13T17:19:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171011 s, 61.3 MB/s 2024-09-13T17:19:44.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.683 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135541 s, 77.4 MB/s 2024-09-13T17:19:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.698 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129979 s, 80.7 MB/s 2024-09-13T17:19:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.714 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137608 s, 76.2 MB/s 2024-09-13T17:19:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.729 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133403 s, 78.6 MB/s 2024-09-13T17:19:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150779 s, 69.5 MB/s 2024-09-13T17:19:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132509 s, 79.1 MB/s 2024-09-13T17:19:44.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133089 s, 78.8 MB/s 2024-09-13T17:19:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.793 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131179 s, 79.9 MB/s 2024-09-13T17:19:44.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.809 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134125 s, 78.2 MB/s 2024-09-13T17:19:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130083 s, 80.6 MB/s 2024-09-13T17:19:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013748 s, 76.3 MB/s 2024-09-13T17:19:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130597 s, 80.3 MB/s 2024-09-13T17:19:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.871 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136449 s, 76.8 MB/s 2024-09-13T17:19:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148607 s, 70.6 MB/s 2024-09-13T17:19:44.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132407 s, 79.2 MB/s 2024-09-13T17:19:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167955 s, 62.4 MB/s 2024-09-13T17:19:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129542 s, 80.9 MB/s 2024-09-13T17:19:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162759 s, 64.4 MB/s 2024-09-13T17:19:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:44.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134522 s, 77.9 MB/s 2024-09-13T17:19:44.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.14 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:44.974 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:45.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.028 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128335 s, 81.7 MB/s 2024-09-13T17:19:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136806 s, 76.6 MB/s 2024-09-13T17:19:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133029 s, 78.8 MB/s 2024-09-13T17:19:45.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.075 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130424 s, 80.4 MB/s 2024-09-13T17:19:45.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.090 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.090 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134663 s, 77.9 MB/s 2024-09-13T17:19:45.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159006 s, 65.9 MB/s 2024-09-13T17:19:45.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169164 s, 62.0 MB/s 2024-09-13T17:19:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132606 s, 79.1 MB/s 2024-09-13T17:19:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128261 s, 81.8 MB/s 2024-09-13T17:19:45.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132697 s, 79.0 MB/s 2024-09-13T17:19:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.189 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128861 s, 81.4 MB/s 2024-09-13T17:19:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126119 s, 83.1 MB/s 2024-09-13T17:19:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.219 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128939 s, 81.3 MB/s 2024-09-13T17:19:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162075 s, 64.7 MB/s 2024-09-13T17:19:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.256 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156986 s, 66.8 MB/s 2024-09-13T17:19:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129992 s, 80.7 MB/s 2024-09-13T17:19:45.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140415 s, 74.7 MB/s 2024-09-13T17:19:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134356 s, 78.0 MB/s 2024-09-13T17:19:45.322 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.322 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.322 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016915 s, 62.0 MB/s 2024-09-13T17:19:45.337 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.338 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135452 s, 77.4 MB/s 2024-09-13T17:19:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.354 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134671 s, 77.9 MB/s 2024-09-13T17:19:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.367 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116486 s, 90.0 MB/s 2024-09-13T17:19:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.383 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131241 s, 79.9 MB/s 2024-09-13T17:19:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141127 s, 74.3 MB/s 2024-09-13T17:19:45.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.417 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155227 s, 67.6 MB/s 2024-09-13T17:19:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142146 s, 73.8 MB/s 2024-09-13T17:19:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.451 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155463 s, 67.4 MB/s 2024-09-13T17:19:45.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.467 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130566 s, 80.3 MB/s 2024-09-13T17:19:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167772 s, 62.5 MB/s 2024-09-13T17:19:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.505 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169031 s, 62.0 MB/s 2024-09-13T17:19:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127799 s, 82.0 MB/s 2024-09-13T17:19:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.537 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144636 s, 72.5 MB/s 2024-09-13T17:19:45.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.15 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:45.538 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:45.539 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133526 s, 78.5 MB/s 2024-09-13T17:19:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013096 s, 80.1 MB/s 2024-09-13T17:19:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.627 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166532 s, 63.0 MB/s 2024-09-13T17:19:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126944 s, 82.6 MB/s 2024-09-13T17:19:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157501 s, 66.6 MB/s 2024-09-13T17:19:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172079 s, 60.9 MB/s 2024-09-13T17:19:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126433 s, 82.9 MB/s 2024-09-13T17:19:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137039 s, 76.5 MB/s 2024-09-13T17:19:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147322 s, 71.2 MB/s 2024-09-13T17:19:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164565 s, 63.7 MB/s 2024-09-13T17:19:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155784 s, 67.3 MB/s 2024-09-13T17:19:45.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127605 s, 82.2 MB/s 2024-09-13T17:19:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130056 s, 80.6 MB/s 2024-09-13T17:19:45.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140878 s, 74.4 MB/s 2024-09-13T17:19:45.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.828 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161575 s, 64.9 MB/s 2024-09-13T17:19:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.843 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126479 s, 82.9 MB/s 2024-09-13T17:19:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.858 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128494 s, 81.6 MB/s 2024-09-13T17:19:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.874 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137426 s, 76.3 MB/s 2024-09-13T17:19:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.889 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129705 s, 80.8 MB/s 2024-09-13T17:19:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125275 s, 83.7 MB/s 2024-09-13T17:19:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132787 s, 79.0 MB/s 2024-09-13T17:19:45.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163217 s, 64.2 MB/s 2024-09-13T17:19:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139187 s, 75.3 MB/s 2024-09-13T17:19:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127033 s, 82.5 MB/s 2024-09-13T17:19:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:45.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159598 s, 65.7 MB/s 2024-09-13T17:19:46.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165395 s, 63.4 MB/s 2024-09-13T17:19:46.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.021 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125552 s, 83.5 MB/s 2024-09-13T17:19:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123912 s, 84.6 MB/s 2024-09-13T17:19:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137086 s, 76.5 MB/s 2024-09-13T17:19:46.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.068 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134138 s, 78.2 MB/s 2024-09-13T17:19:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138897 s, 75.5 MB/s 2024-09-13T17:19:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.099 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134335 s, 78.1 MB/s 2024-09-13T17:19:46.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.16 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:46.101 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.156 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134146 s, 78.2 MB/s 2024-09-13T17:19:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129344 s, 81.1 MB/s 2024-09-13T17:19:46.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013548 s, 77.4 MB/s 2024-09-13T17:19:46.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.202 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134365 s, 78.0 MB/s 2024-09-13T17:19:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.219 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137479 s, 76.3 MB/s 2024-09-13T17:19:46.234 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.235 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133368 s, 78.6 MB/s 2024-09-13T17:19:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.250 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136567 s, 76.8 MB/s 2024-09-13T17:19:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.265 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129349 s, 81.1 MB/s 2024-09-13T17:19:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139952 s, 74.9 MB/s 2024-09-13T17:19:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012378 s, 84.7 MB/s 2024-09-13T17:19:46.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136019 s, 77.1 MB/s 2024-09-13T17:19:46.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.326 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116935 s, 89.7 MB/s 2024-09-13T17:19:46.340 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.341 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127694 s, 82.1 MB/s 2024-09-13T17:19:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.357 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138837 s, 75.5 MB/s 2024-09-13T17:19:46.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.372 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134546 s, 77.9 MB/s 2024-09-13T17:19:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131388 s, 79.8 MB/s 2024-09-13T17:19:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.403 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130718 s, 80.2 MB/s 2024-09-13T17:19:46.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016009 s, 65.5 MB/s 2024-09-13T17:19:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143396 s, 73.1 MB/s 2024-09-13T17:19:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.453 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122644 s, 85.5 MB/s 2024-09-13T17:19:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.466 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116878 s, 89.7 MB/s 2024-09-13T17:19:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124687 s, 84.1 MB/s 2024-09-13T17:19:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136364 s, 76.9 MB/s 2024-09-13T17:19:46.512 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.513 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013166 s, 79.6 MB/s 2024-09-13T17:19:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.531 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161685 s, 64.9 MB/s 2024-09-13T17:19:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128332 s, 81.7 MB/s 2024-09-13T17:19:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131307 s, 79.9 MB/s 2024-09-13T17:19:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146379 s, 71.6 MB/s 2024-09-13T17:19:46.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128749 s, 81.4 MB/s 2024-09-13T17:19:46.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.610 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014346 s, 73.1 MB/s 2024-09-13T17:19:46.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157899 s, 66.4 MB/s 2024-09-13T17:19:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.643 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129252 s, 81.1 MB/s 2024-09-13T17:19:46.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.17 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:46.645 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.703 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162601 s, 64.5 MB/s 2024-09-13T17:19:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132185 s, 79.3 MB/s 2024-09-13T17:19:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136661 s, 76.7 MB/s 2024-09-13T17:19:46.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174239 s, 60.2 MB/s 2024-09-13T17:19:46.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013329 s, 78.7 MB/s 2024-09-13T17:19:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.784 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127517 s, 82.2 MB/s 2024-09-13T17:19:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.800 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138695 s, 75.6 MB/s 2024-09-13T17:19:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.816 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128834 s, 81.4 MB/s 2024-09-13T17:19:46.834 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.835 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167246 s, 62.7 MB/s 2024-09-13T17:19:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131975 s, 79.5 MB/s 2024-09-13T17:19:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.865 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126618 s, 82.8 MB/s 2024-09-13T17:19:46.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144304 s, 72.7 MB/s 2024-09-13T17:19:46.896 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129327 s, 81.1 MB/s 2024-09-13T17:19:46.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.913 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137187 s, 76.4 MB/s 2024-09-13T17:19:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130271 s, 80.5 MB/s 2024-09-13T17:19:46.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.944 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136703 s, 76.7 MB/s 2024-09-13T17:19:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133682 s, 78.4 MB/s 2024-09-13T17:19:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132479 s, 79.2 MB/s 2024-09-13T17:19:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:46.990 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135619 s, 77.3 MB/s 2024-09-13T17:19:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132509 s, 79.1 MB/s 2024-09-13T17:19:47.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.025 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166146 s, 63.1 MB/s 2024-09-13T17:19:47.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.042 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148966 s, 70.4 MB/s 2024-09-13T17:19:47.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.056 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119556 s, 87.7 MB/s 2024-09-13T17:19:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.072 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012256 s, 85.6 MB/s 2024-09-13T17:19:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.086 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135729 s, 77.3 MB/s 2024-09-13T17:19:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128449 s, 81.6 MB/s 2024-09-13T17:19:47.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.118 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142796 s, 73.4 MB/s 2024-09-13T17:19:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157677 s, 66.5 MB/s 2024-09-13T17:19:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.154 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.155 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166647 s, 62.9 MB/s 2024-09-13T17:19:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.170 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131035 s, 80.0 MB/s 2024-09-13T17:19:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.189 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167142 s, 62.7 MB/s 2024-09-13T17:19:47.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134143 s, 78.2 MB/s 2024-09-13T17:19:47.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:47.206 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:47.206 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:47.206 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.18 2024-09-13T17:19:47.207 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:47.207 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:47.207 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:47.207 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:47.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.265 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.266 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170663 s, 61.4 MB/s 2024-09-13T17:19:47.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131608 s, 79.7 MB/s 2024-09-13T17:19:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128179 s, 81.8 MB/s 2024-09-13T17:19:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166067 s, 63.1 MB/s 2024-09-13T17:19:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136984 s, 76.5 MB/s 2024-09-13T17:19:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141007 s, 74.4 MB/s 2024-09-13T17:19:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143317 s, 73.2 MB/s 2024-09-13T17:19:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.379 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013101 s, 80.0 MB/s 2024-09-13T17:19:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139645 s, 75.1 MB/s 2024-09-13T17:19:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167277 s, 62.7 MB/s 2024-09-13T17:19:47.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134063 s, 78.2 MB/s 2024-09-13T17:19:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130235 s, 80.5 MB/s 2024-09-13T17:19:47.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.460 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128326 s, 81.7 MB/s 2024-09-13T17:19:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.478 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159543 s, 65.7 MB/s 2024-09-13T17:19:47.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123439 s, 84.9 MB/s 2024-09-13T17:19:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161029 s, 65.1 MB/s 2024-09-13T17:19:47.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140609 s, 74.6 MB/s 2024-09-13T17:19:47.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168124 s, 62.4 MB/s 2024-09-13T17:19:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128972 s, 81.3 MB/s 2024-09-13T17:19:47.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130728 s, 80.2 MB/s 2024-09-13T17:19:47.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138253 s, 75.8 MB/s 2024-09-13T17:19:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140148 s, 74.8 MB/s 2024-09-13T17:19:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.624 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130575 s, 80.3 MB/s 2024-09-13T17:19:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156342 s, 67.1 MB/s 2024-09-13T17:19:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153107 s, 68.5 MB/s 2024-09-13T17:19:47.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.675 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130295 s, 80.5 MB/s 2024-09-13T17:19:47.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128646 s, 81.5 MB/s 2024-09-13T17:19:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.705 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130418 s, 80.4 MB/s 2024-09-13T17:19:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167149 s, 62.7 MB/s 2024-09-13T17:19:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.740 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139513 s, 75.2 MB/s 2024-09-13T17:19:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126648 s, 82.8 MB/s 2024-09-13T17:19:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015174 s, 69.1 MB/s 2024-09-13T17:19:47.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:47.775 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:47.775 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:47.775 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.19 2024-09-13T17:19:47.775 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:47.776 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:47.776 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:47.776 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:47.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127818 s, 82.0 MB/s 2024-09-13T17:19:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.846 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133193 s, 78.7 MB/s 2024-09-13T17:19:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140479 s, 74.6 MB/s 2024-09-13T17:19:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124483 s, 84.2 MB/s 2024-09-13T17:19:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.892 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124995 s, 83.9 MB/s 2024-09-13T17:19:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.908 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136525 s, 76.8 MB/s 2024-09-13T17:19:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.924 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138933 s, 75.5 MB/s 2024-09-13T17:19:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139485 s, 75.2 MB/s 2024-09-13T17:19:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.955 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129605 s, 80.9 MB/s 2024-09-13T17:19:47.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136233 s, 77.0 MB/s 2024-09-13T17:19:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:47.987 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140472 s, 74.6 MB/s 2024-09-13T17:19:48.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.005 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.005 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155572 s, 67.4 MB/s 2024-09-13T17:19:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.021 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135518 s, 77.4 MB/s 2024-09-13T17:19:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013571 s, 77.3 MB/s 2024-09-13T17:19:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130629 s, 80.3 MB/s 2024-09-13T17:19:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.068 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130146 s, 80.6 MB/s 2024-09-13T17:19:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.083 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131925 s, 79.5 MB/s 2024-09-13T17:19:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123358 s, 85.0 MB/s 2024-09-13T17:19:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131296 s, 79.9 MB/s 2024-09-13T17:19:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131015 s, 80.0 MB/s 2024-09-13T17:19:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142173 s, 73.8 MB/s 2024-09-13T17:19:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133541 s, 78.5 MB/s 2024-09-13T17:19:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164465 s, 63.8 MB/s 2024-09-13T17:19:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153883 s, 68.1 MB/s 2024-09-13T17:19:48.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.211 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126834 s, 82.7 MB/s 2024-09-13T17:19:48.226 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.227 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127919 s, 82.0 MB/s 2024-09-13T17:19:48.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.241 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125455 s, 83.6 MB/s 2024-09-13T17:19:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165988 s, 63.2 MB/s 2024-09-13T17:19:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.279 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166702 s, 62.9 MB/s 2024-09-13T17:19:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135226 s, 77.5 MB/s 2024-09-13T17:19:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.310 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135912 s, 77.2 MB/s 2024-09-13T17:19:48.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.325 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.325 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125423 s, 83.6 MB/s 2024-09-13T17:19:48.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.20 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:48.327 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128508 s, 81.6 MB/s 2024-09-13T17:19:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132232 s, 79.3 MB/s 2024-09-13T17:19:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130774 s, 80.2 MB/s 2024-09-13T17:19:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140318 s, 74.7 MB/s 2024-09-13T17:19:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129298 s, 81.1 MB/s 2024-09-13T17:19:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013075 s, 80.2 MB/s 2024-09-13T17:19:48.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.475 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.475 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134776 s, 77.8 MB/s 2024-09-13T17:19:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136684 s, 76.7 MB/s 2024-09-13T17:19:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.510 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167297 s, 62.7 MB/s 2024-09-13T17:19:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117083 s, 89.6 MB/s 2024-09-13T17:19:48.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167029 s, 62.8 MB/s 2024-09-13T17:19:48.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.559 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138368 s, 75.8 MB/s 2024-09-13T17:19:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01333 s, 78.7 MB/s 2024-09-13T17:19:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01359 s, 77.2 MB/s 2024-09-13T17:19:48.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167698 s, 62.5 MB/s 2024-09-13T17:19:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197054 s, 53.2 MB/s 2024-09-13T17:19:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267409 s, 39.2 MB/s 2024-09-13T17:19:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175088 s, 59.9 MB/s 2024-09-13T17:19:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133198 s, 78.7 MB/s 2024-09-13T17:19:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.712 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138022 s, 76.0 MB/s 2024-09-13T17:19:48.729 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.730 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150573 s, 69.6 MB/s 2024-09-13T17:19:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.745 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126402 s, 83.0 MB/s 2024-09-13T17:19:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.759 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125904 s, 83.3 MB/s 2024-09-13T17:19:48.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.775 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130289 s, 80.5 MB/s 2024-09-13T17:19:48.790 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.791 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013655 s, 76.8 MB/s 2024-09-13T17:19:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.806 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01323 s, 79.3 MB/s 2024-09-13T17:19:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.822 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133711 s, 78.4 MB/s 2024-09-13T17:19:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134631 s, 77.9 MB/s 2024-09-13T17:19:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.853 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134916 s, 77.7 MB/s 2024-09-13T17:19:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136267 s, 77.0 MB/s 2024-09-13T17:19:48.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137294 s, 76.4 MB/s 2024-09-13T17:19:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135545 s, 77.4 MB/s 2024-09-13T17:19:48.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.21 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:48.902 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:48.903 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:48.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.956 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129514 s, 81.0 MB/s 2024-09-13T17:19:48.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138161 s, 75.9 MB/s 2024-09-13T17:19:48.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:48.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:48.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159246 s, 65.8 MB/s 2024-09-13T17:19:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134249 s, 78.1 MB/s 2024-09-13T17:19:49.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.022 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132389 s, 79.2 MB/s 2024-09-13T17:19:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126632 s, 82.8 MB/s 2024-09-13T17:19:49.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131905 s, 79.5 MB/s 2024-09-13T17:19:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.068 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134005 s, 78.2 MB/s 2024-09-13T17:19:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.083 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125041 s, 83.9 MB/s 2024-09-13T17:19:49.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.099 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136294 s, 76.9 MB/s 2024-09-13T17:19:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160916 s, 65.2 MB/s 2024-09-13T17:19:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.132 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134098 s, 78.2 MB/s 2024-09-13T17:19:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.148 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133953 s, 78.3 MB/s 2024-09-13T17:19:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.167 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167181 s, 62.7 MB/s 2024-09-13T17:19:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142577 s, 73.5 MB/s 2024-09-13T17:19:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157251 s, 66.7 MB/s 2024-09-13T17:19:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.216 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123432 s, 85.0 MB/s 2024-09-13T17:19:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136177 s, 77.0 MB/s 2024-09-13T17:19:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.247 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132411 s, 79.2 MB/s 2024-09-13T17:19:49.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130766 s, 80.2 MB/s 2024-09-13T17:19:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.282 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173598 s, 60.4 MB/s 2024-09-13T17:19:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163087 s, 64.3 MB/s 2024-09-13T17:19:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.316 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131344 s, 79.8 MB/s 2024-09-13T17:19:49.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.332 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.332 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126945 s, 82.6 MB/s 2024-09-13T17:19:49.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016565 s, 63.3 MB/s 2024-09-13T17:19:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131451 s, 79.8 MB/s 2024-09-13T17:19:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128362 s, 81.7 MB/s 2024-09-13T17:19:49.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129015 s, 81.3 MB/s 2024-09-13T17:19:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166889 s, 62.8 MB/s 2024-09-13T17:19:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160554 s, 65.3 MB/s 2024-09-13T17:19:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.447 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.448 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125862 s, 83.3 MB/s 2024-09-13T17:19:49.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.464 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139039 s, 75.4 MB/s 2024-09-13T17:19:49.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:49.465 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.22 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:49.466 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.524 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138251 s, 75.8 MB/s 2024-09-13T17:19:49.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.540 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.540 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143448 s, 73.1 MB/s 2024-09-13T17:19:49.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.559 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167537 s, 62.6 MB/s 2024-09-13T17:19:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134098 s, 78.2 MB/s 2024-09-13T17:19:49.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135249 s, 77.5 MB/s 2024-09-13T17:19:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131324 s, 79.8 MB/s 2024-09-13T17:19:49.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.621 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134739 s, 77.8 MB/s 2024-09-13T17:19:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136346 s, 76.9 MB/s 2024-09-13T17:19:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159879 s, 65.6 MB/s 2024-09-13T17:19:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.671 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137281 s, 76.4 MB/s 2024-09-13T17:19:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130579 s, 80.3 MB/s 2024-09-13T17:19:49.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135981 s, 77.1 MB/s 2024-09-13T17:19:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.718 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135091 s, 77.6 MB/s 2024-09-13T17:19:49.733 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136126 s, 77.0 MB/s 2024-09-13T17:19:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124408 s, 84.3 MB/s 2024-09-13T17:19:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117824 s, 89.0 MB/s 2024-09-13T17:19:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.780 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158036 s, 66.4 MB/s 2024-09-13T17:19:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.796 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132006 s, 79.4 MB/s 2024-09-13T17:19:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126672 s, 82.8 MB/s 2024-09-13T17:19:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.826 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133993 s, 78.3 MB/s 2024-09-13T17:19:49.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.844 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157185 s, 66.7 MB/s 2024-09-13T17:19:49.859 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133603 s, 78.5 MB/s 2024-09-13T17:19:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132311 s, 79.3 MB/s 2024-09-13T17:19:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126454 s, 82.9 MB/s 2024-09-13T17:19:49.905 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132589 s, 79.1 MB/s 2024-09-13T17:19:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.922 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138471 s, 75.7 MB/s 2024-09-13T17:19:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131684 s, 79.6 MB/s 2024-09-13T17:19:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128114 s, 81.8 MB/s 2024-09-13T17:19:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170221 s, 61.6 MB/s 2024-09-13T17:19:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:49.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:49.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016719 s, 62.7 MB/s 2024-09-13T17:19:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.008 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156043 s, 67.2 MB/s 2024-09-13T17:19:50.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.024 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.024 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133788 s, 78.4 MB/s 2024-09-13T17:19:50.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.23 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:50.026 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:50.080 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.081 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131371 s, 79.8 MB/s 2024-09-13T17:19:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159467 s, 65.8 MB/s 2024-09-13T17:19:50.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.114 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130784 s, 80.2 MB/s 2024-09-13T17:19:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.129 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013249 s, 79.1 MB/s 2024-09-13T17:19:50.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.144 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126456 s, 82.9 MB/s 2024-09-13T17:19:50.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.159 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.159 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122944 s, 85.3 MB/s 2024-09-13T17:19:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130317 s, 80.5 MB/s 2024-09-13T17:19:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126809 s, 82.7 MB/s 2024-09-13T17:19:50.207 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163115 s, 64.3 MB/s 2024-09-13T17:19:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139665 s, 75.1 MB/s 2024-09-13T17:19:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130828 s, 80.1 MB/s 2024-09-13T17:19:50.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.258 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.258 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016265 s, 64.5 MB/s 2024-09-13T17:19:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.273 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013639 s, 76.9 MB/s 2024-09-13T17:19:50.289 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.290 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139509 s, 75.2 MB/s 2024-09-13T17:19:50.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012714 s, 82.5 MB/s 2024-09-13T17:19:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.319 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119896 s, 87.5 MB/s 2024-09-13T17:19:50.337 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.337 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.337 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158112 s, 66.3 MB/s 2024-09-13T17:19:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.353 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138667 s, 75.6 MB/s 2024-09-13T17:19:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129465 s, 81.0 MB/s 2024-09-13T17:19:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130465 s, 80.4 MB/s 2024-09-13T17:19:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137045 s, 76.5 MB/s 2024-09-13T17:19:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134165 s, 78.2 MB/s 2024-09-13T17:19:50.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132877 s, 78.9 MB/s 2024-09-13T17:19:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125324 s, 83.7 MB/s 2024-09-13T17:19:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132143 s, 79.4 MB/s 2024-09-13T17:19:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.480 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016778 s, 62.5 MB/s 2024-09-13T17:19:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.495 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127752 s, 82.1 MB/s 2024-09-13T17:19:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.510 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128475 s, 81.6 MB/s 2024-09-13T17:19:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.526 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136237 s, 77.0 MB/s 2024-09-13T17:19:50.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148433 s, 70.6 MB/s 2024-09-13T17:19:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161644 s, 64.9 MB/s 2024-09-13T17:19:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.577 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129129 s, 81.2 MB/s 2024-09-13T17:19:50.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.24 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:50.578 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:50.579 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129102 s, 81.2 MB/s 2024-09-13T17:19:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012881 s, 81.4 MB/s 2024-09-13T17:19:50.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168056 s, 62.4 MB/s 2024-09-13T17:19:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150056 s, 69.9 MB/s 2024-09-13T17:19:50.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.698 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.698 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124189 s, 84.4 MB/s 2024-09-13T17:19:50.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.714 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.714 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013383 s, 78.4 MB/s 2024-09-13T17:19:50.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175731 s, 59.7 MB/s 2024-09-13T17:19:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.753 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173077 s, 60.6 MB/s 2024-09-13T17:19:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.769 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133948 s, 78.3 MB/s 2024-09-13T17:19:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.784 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.785 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133558 s, 78.5 MB/s 2024-09-13T17:19:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.800 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133451 s, 78.6 MB/s 2024-09-13T17:19:50.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01318 s, 79.6 MB/s 2024-09-13T17:19:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136487 s, 76.8 MB/s 2024-09-13T17:19:50.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170582 s, 61.5 MB/s 2024-09-13T17:19:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134362 s, 78.0 MB/s 2024-09-13T17:19:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108441 s, 96.7 MB/s 2024-09-13T17:19:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127339 s, 82.3 MB/s 2024-09-13T17:19:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131883 s, 79.5 MB/s 2024-09-13T17:19:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.925 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137167 s, 76.4 MB/s 2024-09-13T17:19:50.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130897 s, 80.1 MB/s 2024-09-13T17:19:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.959 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164956 s, 63.6 MB/s 2024-09-13T17:19:50.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.974 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128266 s, 81.8 MB/s 2024-09-13T17:19:50.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:50.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:50.993 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159842 s, 65.6 MB/s 2024-09-13T17:19:51.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.008 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133638 s, 78.5 MB/s 2024-09-13T17:19:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.027 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166322 s, 63.0 MB/s 2024-09-13T17:19:51.042 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.042 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132133 s, 79.4 MB/s 2024-09-13T17:19:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.058 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133828 s, 78.4 MB/s 2024-09-13T17:19:51.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.078 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171368 s, 61.2 MB/s 2024-09-13T17:19:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.096 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166377 s, 63.0 MB/s 2024-09-13T17:19:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.112 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131261 s, 79.9 MB/s 2024-09-13T17:19:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131521 s, 79.7 MB/s 2024-09-13T17:19:51.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.146 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169552 s, 61.8 MB/s 2024-09-13T17:19:51.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:51.148 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:51.148 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:51.148 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.25 2024-09-13T17:19:51.149 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:51.149 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:51.149 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:51.149 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:51.203 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.204 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134178 s, 78.1 MB/s 2024-09-13T17:19:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.219 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138635 s, 75.6 MB/s 2024-09-13T17:19:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.235 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134531 s, 77.9 MB/s 2024-09-13T17:19:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160795 s, 65.2 MB/s 2024-09-13T17:19:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.269 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131546 s, 79.7 MB/s 2024-09-13T17:19:51.284 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.285 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134498 s, 78.0 MB/s 2024-09-13T17:19:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169999 s, 61.7 MB/s 2024-09-13T17:19:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.319 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134855 s, 77.8 MB/s 2024-09-13T17:19:51.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.335 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129172 s, 81.2 MB/s 2024-09-13T17:19:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.353 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161428 s, 65.0 MB/s 2024-09-13T17:19:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131602 s, 79.7 MB/s 2024-09-13T17:19:51.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130845 s, 80.1 MB/s 2024-09-13T17:19:51.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133635 s, 78.5 MB/s 2024-09-13T17:19:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128859 s, 81.4 MB/s 2024-09-13T17:19:51.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.430 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.430 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137646 s, 76.2 MB/s 2024-09-13T17:19:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128351 s, 81.7 MB/s 2024-09-13T17:19:51.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.460 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127593 s, 82.2 MB/s 2024-09-13T17:19:51.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013249 s, 79.1 MB/s 2024-09-13T17:19:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130032 s, 80.6 MB/s 2024-09-13T17:19:51.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135019 s, 77.7 MB/s 2024-09-13T17:19:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.526 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169945 s, 61.7 MB/s 2024-09-13T17:19:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.545 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168833 s, 62.1 MB/s 2024-09-13T17:19:51.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134366 s, 78.0 MB/s 2024-09-13T17:19:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134746 s, 77.8 MB/s 2024-09-13T17:19:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.591 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012817 s, 81.8 MB/s 2024-09-13T17:19:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.607 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013619 s, 77.0 MB/s 2024-09-13T17:19:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.623 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013424 s, 78.1 MB/s 2024-09-13T17:19:51.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.639 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139099 s, 75.4 MB/s 2024-09-13T17:19:51.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136837 s, 76.6 MB/s 2024-09-13T17:19:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.670 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127797 s, 82.0 MB/s 2024-09-13T17:19:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171612 s, 61.1 MB/s 2024-09-13T17:19:51.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139878 s, 75.0 MB/s 2024-09-13T17:19:51.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:51.707 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:51.707 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.26 2024-09-13T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:51.708 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:51.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.764 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.764 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137252 s, 76.4 MB/s 2024-09-13T17:19:51.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129531 s, 81.0 MB/s 2024-09-13T17:19:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127016 s, 82.6 MB/s 2024-09-13T17:19:51.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.810 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142635 s, 73.5 MB/s 2024-09-13T17:19:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166545 s, 63.0 MB/s 2024-09-13T17:19:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.845 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136575 s, 76.8 MB/s 2024-09-13T17:19:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.861 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135662 s, 77.3 MB/s 2024-09-13T17:19:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.880 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165984 s, 63.2 MB/s 2024-09-13T17:19:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.895 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138101 s, 75.9 MB/s 2024-09-13T17:19:51.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120428 s, 87.1 MB/s 2024-09-13T17:19:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157885 s, 66.4 MB/s 2024-09-13T17:19:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.944 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137503 s, 76.3 MB/s 2024-09-13T17:19:51.959 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.960 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136153 s, 77.0 MB/s 2024-09-13T17:19:51.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136028 s, 77.1 MB/s 2024-09-13T17:19:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:51.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134996 s, 77.7 MB/s 2024-09-13T17:19:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.008 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013781 s, 76.1 MB/s 2024-09-13T17:19:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.023 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134328 s, 78.1 MB/s 2024-09-13T17:19:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.038 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129925 s, 80.7 MB/s 2024-09-13T17:19:52.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139709 s, 75.1 MB/s 2024-09-13T17:19:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.074 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169606 s, 61.8 MB/s 2024-09-13T17:19:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.092 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158973 s, 66.0 MB/s 2024-09-13T17:19:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132122 s, 79.4 MB/s 2024-09-13T17:19:52.122 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.123 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013591 s, 77.2 MB/s 2024-09-13T17:19:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.137 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.138 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128741 s, 81.4 MB/s 2024-09-13T17:19:52.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.153 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.153 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013429 s, 78.1 MB/s 2024-09-13T17:19:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159857 s, 65.6 MB/s 2024-09-13T17:19:52.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.188 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141827 s, 73.9 MB/s 2024-09-13T17:19:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148897 s, 70.4 MB/s 2024-09-13T17:19:52.219 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.220 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127967 s, 81.9 MB/s 2024-09-13T17:19:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.238 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162937 s, 64.4 MB/s 2024-09-13T17:19:52.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131605 s, 79.7 MB/s 2024-09-13T17:19:52.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.273 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167399 s, 62.6 MB/s 2024-09-13T17:19:52.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.27 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:52.274 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:52.275 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:52.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129678 s, 80.9 MB/s 2024-09-13T17:19:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.345 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013664 s, 76.7 MB/s 2024-09-13T17:19:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.361 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135722 s, 77.3 MB/s 2024-09-13T17:19:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122589 s, 85.5 MB/s 2024-09-13T17:19:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132064 s, 79.4 MB/s 2024-09-13T17:19:52.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.407 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01331 s, 78.8 MB/s 2024-09-13T17:19:52.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.423 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.423 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139991 s, 74.9 MB/s 2024-09-13T17:19:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133047 s, 78.8 MB/s 2024-09-13T17:19:52.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153637 s, 68.3 MB/s 2024-09-13T17:19:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.472 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013754 s, 76.2 MB/s 2024-09-13T17:19:52.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01418 s, 73.9 MB/s 2024-09-13T17:19:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.505 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01364 s, 76.9 MB/s 2024-09-13T17:19:52.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.519 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119013 s, 88.1 MB/s 2024-09-13T17:19:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133744 s, 78.4 MB/s 2024-09-13T17:19:52.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.549 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129387 s, 81.0 MB/s 2024-09-13T17:19:52.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.565 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.565 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137164 s, 76.4 MB/s 2024-09-13T17:19:52.583 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167464 s, 62.6 MB/s 2024-09-13T17:19:52.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133924 s, 78.3 MB/s 2024-09-13T17:19:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142269 s, 73.7 MB/s 2024-09-13T17:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.635 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016908 s, 62.0 MB/s 2024-09-13T17:19:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.651 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136333 s, 76.9 MB/s 2024-09-13T17:19:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.669 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157701 s, 66.5 MB/s 2024-09-13T17:19:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.685 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013842 s, 75.8 MB/s 2024-09-13T17:19:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.703 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160596 s, 65.3 MB/s 2024-09-13T17:19:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.719 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013504 s, 77.6 MB/s 2024-09-13T17:19:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135617 s, 77.3 MB/s 2024-09-13T17:19:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138036 s, 76.0 MB/s 2024-09-13T17:19:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135548 s, 77.4 MB/s 2024-09-13T17:19:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133288 s, 78.7 MB/s 2024-09-13T17:19:52.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131995 s, 79.4 MB/s 2024-09-13T17:19:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136162 s, 77.0 MB/s 2024-09-13T17:19:52.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.830 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128727 s, 81.5 MB/s 2024-09-13T17:19:52.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:52.831 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:52.831 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:52.831 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.28 2024-09-13T17:19:52.832 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:52.832 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:52.832 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:52.832 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:52.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119266 s, 87.9 MB/s 2024-09-13T17:19:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137139 s, 76.5 MB/s 2024-09-13T17:19:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138434 s, 75.7 MB/s 2024-09-13T17:19:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136609 s, 76.8 MB/s 2024-09-13T17:19:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.948 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137918 s, 76.0 MB/s 2024-09-13T17:19:52.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.968 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168695 s, 62.2 MB/s 2024-09-13T17:19:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158365 s, 66.2 MB/s 2024-09-13T17:19:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.000 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126423 s, 82.9 MB/s 2024-09-13T17:19:53.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012718 s, 82.4 MB/s 2024-09-13T17:19:53.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.031 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.031 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013103 s, 80.0 MB/s 2024-09-13T17:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.045 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123624 s, 84.8 MB/s 2024-09-13T17:19:53.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01707 s, 61.4 MB/s 2024-09-13T17:19:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.080 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128325 s, 81.7 MB/s 2024-09-13T17:19:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163888 s, 64.0 MB/s 2024-09-13T17:19:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133845 s, 78.3 MB/s 2024-09-13T17:19:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.133 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168247 s, 62.3 MB/s 2024-09-13T17:19:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163601 s, 64.1 MB/s 2024-09-13T17:19:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.171 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172935 s, 60.6 MB/s 2024-09-13T17:19:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.186 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131477 s, 79.8 MB/s 2024-09-13T17:19:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.202 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132264 s, 79.3 MB/s 2024-09-13T17:19:53.217 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.218 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137561 s, 76.2 MB/s 2024-09-13T17:19:53.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132694 s, 79.0 MB/s 2024-09-13T17:19:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.252 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165054 s, 63.5 MB/s 2024-09-13T17:19:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.268 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134409 s, 78.0 MB/s 2024-09-13T17:19:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016454 s, 63.7 MB/s 2024-09-13T17:19:53.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137726 s, 76.1 MB/s 2024-09-13T17:19:53.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.317 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125264 s, 83.7 MB/s 2024-09-13T17:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137802 s, 76.1 MB/s 2024-09-13T17:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167331 s, 62.7 MB/s 2024-09-13T17:19:53.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134234 s, 78.1 MB/s 2024-09-13T17:19:53.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013635 s, 76.9 MB/s 2024-09-13T17:19:53.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.400 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134277 s, 78.1 MB/s 2024-09-13T17:19:53.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:53.401 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:53.401 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:53.401 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.29 2024-09-13T17:19:53.401 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:53.402 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:53.402 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:53.402 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.459 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161588 s, 64.9 MB/s 2024-09-13T17:19:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.474 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124289 s, 84.4 MB/s 2024-09-13T17:19:53.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118095 s, 88.8 MB/s 2024-09-13T17:19:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.503 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129815 s, 80.8 MB/s 2024-09-13T17:19:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151465 s, 69.2 MB/s 2024-09-13T17:19:53.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.535 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.535 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125849 s, 83.3 MB/s 2024-09-13T17:19:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141103 s, 74.3 MB/s 2024-09-13T17:19:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.571 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176888 s, 59.3 MB/s 2024-09-13T17:19:53.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120311 s, 87.2 MB/s 2024-09-13T17:19:53.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182917 s, 57.3 MB/s 2024-09-13T17:19:53.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.628 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184231 s, 56.9 MB/s 2024-09-13T17:19:53.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188245 s, 55.7 MB/s 2024-09-13T17:19:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.664 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135609 s, 77.3 MB/s 2024-09-13T17:19:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.680 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134118 s, 78.2 MB/s 2024-09-13T17:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139935 s, 74.9 MB/s 2024-09-13T17:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144916 s, 72.4 MB/s 2024-09-13T17:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.728 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127907 s, 82.0 MB/s 2024-09-13T17:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01299 s, 80.7 MB/s 2024-09-13T17:19:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131609 s, 79.7 MB/s 2024-09-13T17:19:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.774 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133255 s, 78.7 MB/s 2024-09-13T17:19:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.788 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123137 s, 85.2 MB/s 2024-09-13T17:19:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141504 s, 74.1 MB/s 2024-09-13T17:19:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.820 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130488 s, 80.4 MB/s 2024-09-13T17:19:53.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.835 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132655 s, 79.0 MB/s 2024-09-13T17:19:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.850 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127261 s, 82.4 MB/s 2024-09-13T17:19:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161732 s, 64.8 MB/s 2024-09-13T17:19:53.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.884 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.885 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136643 s, 76.7 MB/s 2024-09-13T17:19:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016064 s, 65.3 MB/s 2024-09-13T17:19:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135076 s, 77.6 MB/s 2024-09-13T17:19:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173444 s, 60.5 MB/s 2024-09-13T17:19:53.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134917 s, 77.7 MB/s 2024-09-13T17:19:53.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:53.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:53.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138509 s, 75.7 MB/s 2024-09-13T17:19:53.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:53.971 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:53.971 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:53.971 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.30 2024-09-13T17:19:53.972 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:53.972 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:53.972 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:53.972 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:54.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.027 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.027 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133369 s, 78.6 MB/s 2024-09-13T17:19:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138649 s, 75.6 MB/s 2024-09-13T17:19:54.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.058 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.058 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129338 s, 81.1 MB/s 2024-09-13T17:19:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.076 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159264 s, 65.8 MB/s 2024-09-13T17:19:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.092 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136912 s, 76.6 MB/s 2024-09-13T17:19:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147725 s, 71.0 MB/s 2024-09-13T17:19:54.126 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.127 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157347 s, 66.6 MB/s 2024-09-13T17:19:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.142 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133508 s, 78.5 MB/s 2024-09-13T17:19:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177998 s, 58.9 MB/s 2024-09-13T17:19:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138065 s, 75.9 MB/s 2024-09-13T17:19:54.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143282 s, 73.2 MB/s 2024-09-13T17:19:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012608 s, 83.2 MB/s 2024-09-13T17:19:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.227 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149278 s, 70.2 MB/s 2024-09-13T17:19:54.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136628 s, 76.7 MB/s 2024-09-13T17:19:54.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.262 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164231 s, 63.8 MB/s 2024-09-13T17:19:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136371 s, 76.9 MB/s 2024-09-13T17:19:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134807 s, 77.8 MB/s 2024-09-13T17:19:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.309 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132622 s, 79.1 MB/s 2024-09-13T17:19:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129438 s, 81.0 MB/s 2024-09-13T17:19:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.339 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128337 s, 81.7 MB/s 2024-09-13T17:19:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.354 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133695 s, 78.4 MB/s 2024-09-13T17:19:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016017 s, 65.5 MB/s 2024-09-13T17:19:54.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128735 s, 81.5 MB/s 2024-09-13T17:19:54.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.405 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131196 s, 79.9 MB/s 2024-09-13T17:19:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.419 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01366 s, 76.8 MB/s 2024-09-13T17:19:54.435 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144092 s, 72.8 MB/s 2024-09-13T17:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016693 s, 62.8 MB/s 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160201 s, 65.5 MB/s 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137362 s, 76.3 MB/s 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.508 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166518 s, 63.0 MB/s 2024-09-13T17:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155424 s, 67.5 MB/s 2024-09-13T17:19:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.541 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132747 s, 79.0 MB/s 2024-09-13T17:19:54.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.31 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:19:54.543 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:19:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157847 s, 66.4 MB/s 2024-09-13T17:19:54.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153522 s, 68.3 MB/s 2024-09-13T17:19:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129842 s, 80.8 MB/s 2024-09-13T17:19:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128295 s, 81.7 MB/s 2024-09-13T17:19:54.663 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.664 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131615 s, 79.7 MB/s 2024-09-13T17:19:54.678 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127781 s, 82.1 MB/s 2024-09-13T17:19:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140705 s, 74.5 MB/s 2024-09-13T17:19:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136334 s, 76.9 MB/s 2024-09-13T17:19:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128391 s, 81.7 MB/s 2024-09-13T17:19:54.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.741 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124994 s, 83.9 MB/s 2024-09-13T17:19:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132873 s, 78.9 MB/s 2024-09-13T17:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130838 s, 80.1 MB/s 2024-09-13T17:19:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.786 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.787 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127294 s, 82.4 MB/s 2024-09-13T17:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138163 s, 75.9 MB/s 2024-09-13T17:19:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.818 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130566 s, 80.3 MB/s 2024-09-13T17:19:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012574 s, 83.4 MB/s 2024-09-13T17:19:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133394 s, 78.6 MB/s 2024-09-13T17:19:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01225 s, 85.6 MB/s 2024-09-13T17:19:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132582 s, 79.1 MB/s 2024-09-13T17:19:54.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.893 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012911 s, 81.2 MB/s 2024-09-13T17:19:54.908 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.909 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130731 s, 80.2 MB/s 2024-09-13T17:19:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.922 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115224 s, 91.0 MB/s 2024-09-13T17:19:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.938 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134839 s, 77.8 MB/s 2024-09-13T17:19:54.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.953 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131756 s, 79.6 MB/s 2024-09-13T17:19:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013597 s, 77.1 MB/s 2024-09-13T17:19:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:54.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138621 s, 75.6 MB/s 2024-09-13T17:19:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133402 s, 78.6 MB/s 2024-09-13T17:19:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140182 s, 74.8 MB/s 2024-09-13T17:19:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130913 s, 80.1 MB/s 2024-09-13T17:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132157 s, 79.3 MB/s 2024-09-13T17:19:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.063 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128785 s, 81.4 MB/s 2024-09-13T17:19:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:19:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:19:55.079 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141883 s, 73.9 MB/s 2024-09-13T17:19:55.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.11:0 scrub start /scrub_control_test_path recursive 2024-09-13T17:19:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.240+0000 7f1497200640 1 Processor -- start 2024-09-13T17:19:55.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.240+0000 7f1497200640 1 -- start start 2024-09-13T17:19:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1497200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1498143d90 0x7f149806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1497200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f14980cd300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1497200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f149806ca10 con 0x7f14980cd840 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1497200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1498144170 con 0x7f1498143d90 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1497200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f149806be00 con 0x7f149806b8a0 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1495e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1498143d90 0x7f149806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1496800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f14980cd840 0x7f149815df30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55620/0 (socket says 172.21.15.28:55620) 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.241+0000 7f1496800640 1 -- 172.21.15.28:0/1813106998 learned_addr learned my addr 172.21.15.28:0/1813106998 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:55.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f148fe00640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f14980cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f1495e00640 1 -- 172.21.15.28:0/1813106998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 msgr2=0x7f14980cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f1495e00640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f14980cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f1495e00640 1 -- 172.21.15.28:0/1813106998 >> v1:172.21.15.28:6789/0 conn(0x7f14980cd840 legacy=0x7f149815df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f1495e00640 1 -- 172.21.15.28:0/1813106998 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149806c0e0 con 0x7f1498143d90 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.242+0000 7f1495400640 1 -- 172.21.15.28:0/1813106998 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635453503 0 0) 0x7f149806ca10 con 0x7f14980cd840 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.243+0000 7f1495e00640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f149806b360 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f14880097a0 tx=0x7f148802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed5a8e77f320d2b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.243+0000 7f1495400640 1 -- 172.21.15.28:0/1813106998 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488032440 con 0x7f1498143d90 2024-09-13T17:19:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.243+0000 7f1495400640 1 -- 172.21.15.28:0/1813106998 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14880325a0 con 0x7f1498143d90 2024-09-13T17:19:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.243+0000 7f1495400640 1 -- 172.21.15.28:0/1813106998 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488032930 con 0x7f1498143d90 2024-09-13T17:19:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 -- 172.21.15.28:0/1813106998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 msgr2=0x7f149806b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f149806b360 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f14880097a0 tx=0x7f148802a180 comp rx=0 tx=0).stop 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 -- 172.21.15.28:0/1813106998 shutdown_connections 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f14980cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 --2- 172.21.15.28:0/1813106998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f149806b360 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 -- 172.21.15.28:0/1813106998 >> 172.21.15.28:0/1813106998 conn(0x7f1498073470 msgr2=0x7f1498069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 -- 172.21.15.28:0/1813106998 shutdown_connections 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.244+0000 7f1497200640 1 -- 172.21.15.28:0/1813106998 wait complete. 2024-09-13T17:19:55.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.246+0000 7f1497200640 1 Processor -- start 2024-09-13T17:19:55.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.246+0000 7f1497200640 1 -- start start 2024-09-13T17:19:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f149813a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 0x7f14980b1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14981f0620 con 0x7f1498143d90 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980ac870 con 0x7f14980cd840 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1497200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980acb50 con 0x7f149806b8a0 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36828/0 (socket says 172.21.15.28:36828) 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 -- 172.21.15.28:0/233264428 learned_addr learned my addr 172.21.15.28:0/233264428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f148fe00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 0x7f14980b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1495e00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f149813a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 -- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 msgr2=0x7f149813a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f149813a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 -- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 msgr2=0x7f14980b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 0x7f14980b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1496800640 1 -- 172.21.15.28:0/233264428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1488009400 con 0x7f1498143d90 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f148fe00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 0x7f14980b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.247+0000 7f1495e00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f149813a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:19:55.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f1496800640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f149000c780 tx=0x7f149000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f148e000640 1 -- 172.21.15.28:0/233264428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149000ce90 con 0x7f1498143d90 2024-09-13T17:19:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f148e000640 1 -- 172.21.15.28:0/233264428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149000a640 con 0x7f1498143d90 2024-09-13T17:19:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f148e000640 1 -- 172.21.15.28:0/233264428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149000a9d0 con 0x7f1498143d90 2024-09-13T17:19:55.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14980ace30 con 0x7f1498143d90 2024-09-13T17:19:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.248+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14980abb20 con 0x7f1498143d90 2024-09-13T17:19:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.250+0000 7f148e000640 1 -- 172.21.15.28:0/233264428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f149000ab30 con 0x7f1498143d90 2024-09-13T17:19:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.252+0000 7f148e000640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 0x7f147406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.252+0000 7f148e000640 1 -- 172.21.15.28:0/233264428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1490088960 con 0x7f1498143d90 2024-09-13T17:19:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.252+0000 7f1495e00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 0x7f147406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.253+0000 7f149f7f8740 1 Processor -- start 2024-09-13T17:19:55.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.253+0000 7f149f7f8740 1 -- start start 2024-09-13T17:19:55.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.253+0000 7f1495e00640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 0x7f147406d0c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1488034000 tx=0x7f148802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b153e0 0x5581b5b157c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5a7ef00 con 0x5581b5b3cc40 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5a7c690 con 0x5581b5b63100 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581b5a77f80 con 0x5581b5b153e0 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36844/0 (socket says 172.21.15.28:36844) 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 -- 172.21.15.28:0/4068702221 learned_addr learned my addr 172.21.15.28:0/4068702221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f1495e00640 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b153e0 0x5581b5b157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f1496800640 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 -- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b153e0 msgr2=0x5581b5b157c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b153e0 0x5581b5b157c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 -- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 msgr2=0x5581b5b635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.254+0000 7f148fe00640 1 -- 172.21.15.28:0/4068702221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b5a42390 con 0x5581b5b3cc40 2024-09-13T17:19:55.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f148fe00640 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f1480009a90 tx=0x7f1480009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e9c2ff245af54a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f1486a00640 1 -- 172.21.15.28:0/4068702221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480010d60 con 0x5581b5b3cc40 2024-09-13T17:19:55.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f1486a00640 1 -- 172.21.15.28:0/4068702221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480010ec0 con 0x5581b5b3cc40 2024-09-13T17:19:55.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f1486a00640 1 -- 172.21.15.28:0/4068702221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1480014610 con 0x5581b5b3cc40 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f149f7f8740 1 -- 172.21.15.28:0/4068702221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 msgr2=0x5581b5b62bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.255+0000 7f149f7f8740 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f1480009a90 tx=0x7f1480009190 comp rx=0 tx=0).stop 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 -- 172.21.15.28:0/4068702221 shutdown_connections 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b62bc0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 --2- 172.21.15.28:0/4068702221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b153e0 0x5581b5b157c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 -- 172.21.15.28:0/4068702221 >> 172.21.15.28:0/4068702221 conn(0x5581b5b11c30 msgr2=0x5581b5b5a0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 -- 172.21.15.28:0/4068702221 shutdown_connections 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 -- 172.21.15.28:0/4068702221 wait complete. 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 -- start start 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.256+0000 7f149f7f8740 1 Processor -- start 2024-09-13T17:19:55.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b5d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b3b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5a362c0 con 0x5581b5b63100 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5a35f60 con 0x5581b5b5dc70 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581b57df3e0 con 0x5581b5b3cc40 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.257+0000 7f149f7f8740 10 client.0 _ll_register_callbacks cb 0x5581b5a4cdb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f148fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b5d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:60760/0 (socket says 172.21.15.28:60760) 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 -- 172.21.15.28:0/2749364172 learned_addr learned my addr 172.21.15.28:0/2749364172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1496800640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b3b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 -- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 msgr2=0x5581b5b3b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b3b580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 -- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 msgr2=0x5581b5b5d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b5d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 -- 172.21.15.28:0/2749364172 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14800096c0 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f1495e00640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f148802a770 tx=0x7f1488004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba385606a89470 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.258+0000 7f147b400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3301/0 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148804c070 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mon.2 v2:172.21.15.189:3301/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1488032e40 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581b5a69320 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1488003c70 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f149f7f8740 10 client.8201 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.259+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x5581b5a76600 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mon.2 v2:172.21.15.189:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5581b5a69320 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mon.2 v2:172.21.15.189:3301/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f148804f020 con 0x5581b5b3cc40 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 10 client.8201 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 10 client.8201 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 20 client.8201 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b593e0 0x5581b5b597c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 4 client.8201 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581b576b880 con 0x5581b5b593e0 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.260+0000 7f148fe00640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b593e0 0x5581b5b597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.261+0000 7f148fe00640 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b593e0 0x5581b5b597c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77cdd41f750d32bd server_cookie=e651ba5765fe2088 in_seq=0 out_seq=0 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.261+0000 7f147b400640 10 client.8201 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.261+0000 7f147b400640 1 -- 172.21.15.28:0/2749364172 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5581b576b880 con 0x5581b5b593e0 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.261+0000 7f147b400640 10 client.8201 handle_command_reply: tid=0 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.261+0000 7f149f7f8740 1 client.8201 shutdown 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 msgr2=0x5581b5b5e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f148802a770 tx=0x7f1488004730 comp rx=0 tx=0).stop 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 shutdown_connections 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b593e0 0x5581b5b597c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b3b580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b5d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f147b400640 0 client.8201 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2749364172 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5b5e920 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 >> 172.21.15.28:0/2749364172 conn(0x5581b5b153e0 msgr2=0x5581b5b10950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 shutdown_connections 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 1 -- 172.21.15.28:0/2749364172 wait complete. 2024-09-13T17:19:55.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.262+0000 7f149f7f8740 20 client.8201 trim_cache size 0 max 16384 2024-09-13T17:19:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.277+0000 7f149f7f8740 1 Processor -- start 2024-09-13T17:19:55.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.277+0000 7f149f7f8740 1 -- start start 2024-09-13T17:19:55.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b5dc70 0x5581b5b577d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b10b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b58000 con 0x5581b5b3cc40 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b80a30 con 0x5581b5b63100 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b5ff60 con 0x5581b5b5dc70 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f1495e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f1495e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36860/0 (socket says 172.21.15.28:36860) 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f1495e00640 1 -- 172.21.15.28:0/938592606 learned_addr learned my addr 172.21.15.28:0/938592606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f1496800640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b10b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.278+0000 7f148fe00640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b5dc70 0x5581b5b577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 -- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b5dc70 msgr2=0x5581b5b577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b5dc70 0x5581b5b577d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 -- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 msgr2=0x5581b5b10b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b10b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 -- 172.21.15.28:0/938592606 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b5a18400 con 0x5581b5b3cc40 2024-09-13T17:19:55.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1496800640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b10b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1495e00640 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f1488006f80 tx=0x7f1488046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05e61e8c766510c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1487400640 1 -- 172.21.15.28:0/938592606 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148804c070 con 0x5581b5b3cc40 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1487400640 1 -- 172.21.15.28:0/938592606 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488042040 con 0x5581b5b3cc40 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.279+0000 7f1487400640 1 -- 172.21.15.28:0/938592606 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488003ec0 con 0x5581b5b3cc40 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 -- 172.21.15.28:0/938592606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 msgr2=0x5581b5b57290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f1488006f80 tx=0x7f1488046000 comp rx=0 tx=0).stop 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 -- 172.21.15.28:0/938592606 shutdown_connections 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b63100 0x5581b5b10b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b5dc70 0x5581b5b577d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 --2- 172.21.15.28:0/938592606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b3cc40 0x5581b5b57290 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 -- 172.21.15.28:0/938592606 >> 172.21.15.28:0/938592606 conn(0x5581b5b153e0 msgr2=0x5581b5b157c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 -- 172.21.15.28:0/938592606 shutdown_connections 2024-09-13T17:19:55.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.280+0000 7f149f7f8740 1 -- 172.21.15.28:0/938592606 wait complete. 2024-09-13T17:19:55.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.281+0000 7f149f7f8740 1 -- start start 2024-09-13T17:19:55.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.281+0000 7f149f7f8740 1 Processor -- start 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5bc88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b7d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b7dc30 con 0x5581b5b63100 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b5e050 con 0x5581b5b5dc70 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5581b5b60650 con 0x5581b5b3cc40 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f149f7f8740 10 client.0 _ll_register_callbacks cb 0x7f1480009420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36864/0 (socket says 172.21.15.28:36864) 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 -- 172.21.15.28:0/2051480897 learned_addr learned my addr 172.21.15.28:0/2051480897 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f148fe00640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b7d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1495e00640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5bc88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 -- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 msgr2=0x5581b5bc88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5bc88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 -- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 msgr2=0x5581b5b7d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b7d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1496800640 1 -- 172.21.15.28:0/2051480897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581b5a76600 con 0x5581b5b63100 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.282+0000 7f1495e00640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5bc88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f1496800640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f149005f620 tx=0x7f149005fb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3828e577293f37f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149005b370 con 0x5581b5b63100 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f14900868b0 con 0x5581b5b63100 2024-09-13T17:19:55.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581b5aa0ab0 con 0x5581b5b63100 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1490086a70 con 0x5581b5b63100 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 10 client.10221 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x5581b5a3d9d0 con 0x5581b5b63100 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1490075070 con 0x5581b5b63100 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f149005f6c0 con 0x5581b5b63100 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 10 client.10221 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 10 client.10221 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 20 client.10221 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b3b360 0x5581b5b101e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 4 client.10221 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.283+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x5581b577cb80 con 0x5581b5b3b360 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.284+0000 7f1495e00640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b3b360 0x5581b5b101e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.284+0000 7f1495e00640 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b3b360 0x5581b5b101e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50ea76225e3b2712 server_cookie=3aa46a0a15cac5fd in_seq=0 out_seq=0 2024-09-13T17:19:55.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.284+0000 7f147b400640 10 client.10221 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f147b400640 1 -- 172.21.15.28:0/2051480897 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5581b577cb80 con 0x5581b5b3b360 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f147b400640 10 client.10221 handle_command_reply: tid=0 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 client.10221 shutdown 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 msgr2=0x5581b5b0fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f149005f620 tx=0x7f149005fb50 comp rx=0 tx=0).stop 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 shutdown_connections 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x5581b5b3b360 0x5581b5b101e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5581b5b63100 0x5581b5b0fca0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5581b5b5dc70 0x5581b5b7d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 --2- 172.21.15.28:0/2051480897 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5581b5b3cc40 0x5581b5bc88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 >> 172.21.15.28:0/2051480897 conn(0x5581b5b153e0 msgr2=0x5581b5b57290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f147b400640 0 client.10221 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 shutdown_connections 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.287+0000 7f149f7f8740 1 -- 172.21.15.28:0/2051480897 wait complete. 2024-09-13T17:19:55.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f149f7f8740 20 client.10221 trim_cache size 0 max 16384 2024-09-13T17:19:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:19:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "bdef086f-0d4a-4728-a2b8-a3c324f1bfc0", 2024-09-13T17:19:55.291 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:19:55.291 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 msgr2=0x7f147406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 0x7f147406d0c0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f1488034000 tx=0x7f148802e000 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 msgr2=0x7f14980b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f149000c780 tx=0x7f149000cc40 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 shutdown_connections 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f147406ac00 0x7f147406d0c0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1498143d90 0x7f14980b1940 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14980cd840 0x7f14980b1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 --2- 172.21.15.28:0/233264428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f149806b8a0 0x7f149813a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.288+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 >> 172.21.15.28:0/233264428 conn(0x7f1498073470 msgr2=0x7f14980af370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.289+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 shutdown_connections 2024-09-13T17:19:55.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.289+0000 7f1497200640 1 -- 172.21.15.28:0/233264428 wait complete. 2024-09-13T17:19:55.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.11:0 scrub pause 2024-09-13T17:19:55.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.503+0000 7f05d8800640 1 Processor -- start 2024-09-13T17:19:55.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.503+0000 7f05d8800640 1 -- start start 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.504+0000 7f05d8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d015ab60 0x7f05d006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d8800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05d0071490 0x7f05d006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d8800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d006c8b0 con 0x7f05d015a780 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d8800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d006cb90 con 0x7f05d0071490 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d00cc980 con 0x7f05d015ab60 2024-09-13T17:19:55.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05d0071490 0x7f05d006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05d0071490 0x7f05d006c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36868/0 (socket says 172.21.15.28:36868) 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 -- 172.21.15.28:0/1140740542 learned_addr learned my addr 172.21.15.28:0/1140740542 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d5400640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d015ab60 0x7f05d006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 -- 172.21.15.28:0/1140740542 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d015ab60 msgr2=0x7f05d006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d015ab60 0x7f05d006bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 -- 172.21.15.28:0/1140740542 >> v1:172.21.15.189:6789/0 conn(0x7f05d015a780 legacy=0x7f05d0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.505+0000 7f05d6800640 1 -- 172.21.15.28:0/1140740542 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d00ccc60 con 0x7f05d0071490 2024-09-13T17:19:55.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.506+0000 7f05d6800640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0071490 0x7f05d006c080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009e50 tx=0x7f05cc02eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d6e6c6283632268 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d4a00640 1 -- 172.21.15.28:0/1140740542 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc030690 con 0x7f05d0071490 2024-09-13T17:19:55.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d4a00640 1 -- 172.21.15.28:0/1140740542 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc0307f0 con 0x7f05d0071490 2024-09-13T17:19:55.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d4a00640 1 -- 172.21.15.28:0/1140740542 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc030b80 con 0x7f05d0071490 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 -- 172.21.15.28:0/1140740542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0071490 msgr2=0x7f05d006c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0071490 0x7f05d006c080 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009e50 tx=0x7f05cc02eb40 comp rx=0 tx=0).stop 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 -- 172.21.15.28:0/1140740542 shutdown_connections 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d0071490 0x7f05d006c080 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 --2- 172.21.15.28:0/1140740542 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d015ab60 0x7f05d006bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 -- 172.21.15.28:0/1140740542 >> 172.21.15.28:0/1140740542 conn(0x7f05d00b0270 msgr2=0x7f05d00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 -- 172.21.15.28:0/1140740542 shutdown_connections 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.507+0000 7f05d8800640 1 -- 172.21.15.28:0/1140740542 wait complete. 2024-09-13T17:19:55.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.509+0000 7f05d8800640 1 Processor -- start 2024-09-13T17:19:55.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.509+0000 7f05d8800640 1 -- start start 2024-09-13T17:19:55.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d0071490 0x7f05d0152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d015a780 0x7f05d014edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0150a90 con 0x7f05d015ab60 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0153320 con 0x7f05d015a780 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d0153600 con 0x7f05d0071490 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36884/0 (socket says 172.21.15.28:36884) 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d6800640 1 -- 172.21.15.28:0/2674078649 learned_addr learned my addr 172.21.15.28:0/2674078649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d5e00640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d0071490 0x7f05d0152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.510+0000 7f05d5400640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d015a780 0x7f05d014edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 -- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d0071490 msgr2=0x7f05d0152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d0071490 0x7f05d0152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 -- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d015a780 msgr2=0x7f05d014edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d015a780 0x7f05d014edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 -- 172.21.15.28:0/2674078649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc009b30 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d6800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009e50 tx=0x7f05cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05ca000640 1 -- 172.21.15.28:0/2674078649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc041070 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05ca000640 1 -- 172.21.15.28:0/2674078649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc009510 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05ca000640 1 -- 172.21.15.28:0/2674078649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc031bf0 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d01538e0 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.511+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d0153010 con 0x7f05d015ab60 2024-09-13T17:19:55.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.513+0000 7f05ca000640 1 -- 172.21.15.28:0/2674078649 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f05cc031d50 con 0x7f05d015ab60 2024-09-13T17:19:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.515+0000 7f05ca000640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 0x7f05b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.515+0000 7f05ca000640 1 -- 172.21.15.28:0/2674078649 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05cc0adf80 con 0x7f05d015ab60 2024-09-13T17:19:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.515+0000 7f05d5e00640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 0x7f05b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.516+0000 7f05db6a2740 1 Processor -- start 2024-09-13T17:19:55.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.516+0000 7f05db6a2740 1 -- start start 2024-09-13T17:19:55.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.516+0000 7f05d5e00640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 0x7f05b406d0c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f60 tx=0x7f05c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.516+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d95633380 0x562d95633760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d9565a830 0x562d95680d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05db6a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9553e000 con 0x562d95630b40 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9559bd40 con 0x562d95633380 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562d955994d0 con 0x562d9565a830 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36890/0 (socket says 172.21.15.28:36890) 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 -- 172.21.15.28:0/928138571 learned_addr learned my addr 172.21.15.28:0/928138571 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5e00640 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d95633380 0x562d95633760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d6800640 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d9565a830 0x562d95680d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 -- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d9565a830 msgr2=0x562d95680d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d9565a830 0x562d95680d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 -- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d95633380 msgr2=0x562d95633760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d95633380 0x562d95633760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.517+0000 7f05d5400640 1 -- 172.21.15.28:0/928138571 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d955503a0 con 0x562d95630b40 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05d5400640 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f05c0007890 tx=0x7f05c0007d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c67c05a6dce91 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05bea00640 1 -- 172.21.15.28:0/928138571 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0010b00 con 0x562d95630b40 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05bea00640 1 -- 172.21.15.28:0/928138571 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c0010c60 con 0x562d95630b40 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05bea00640 1 -- 172.21.15.28:0/928138571 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0004030 con 0x562d95630b40 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05db6a2740 1 -- 172.21.15.28:0/928138571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 msgr2=0x562d95680850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05db6a2740 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f05c0007890 tx=0x7f05c0007d50 comp rx=0 tx=0).stop 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.518+0000 7f05db6a2740 1 -- 172.21.15.28:0/928138571 shutdown_connections 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d9565a830 0x562d95680d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95630b40 0x562d95680850 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 --2- 172.21.15.28:0/928138571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d95633380 0x562d95633760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 -- 172.21.15.28:0/928138571 >> 172.21.15.28:0/928138571 conn(0x562d9562fab0 msgr2=0x562d95677f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 -- 172.21.15.28:0/928138571 shutdown_connections 2024-09-13T17:19:55.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 -- 172.21.15.28:0/928138571 wait complete. 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 -- start start 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.519+0000 7f05db6a2740 1 Processor -- start 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d95633380 0x562d95644760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d956467f0 0x562d95659280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d95594dc0 con 0x562d9565a830 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d955533e0 con 0x562d956467f0 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562d95553080 con 0x562d95633380 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.520+0000 7f05db6a2740 10 client.0 _ll_register_callbacks cb 0x562d9558e430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36904/0 (socket says 172.21.15.28:36904) 2024-09-13T17:19:55.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 -- 172.21.15.28:0/866763516 learned_addr learned my addr 172.21.15.28:0/866763516 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5e00640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d95633380 0x562d95644760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d6800640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d956467f0 0x562d95659280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 -- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d95633380 msgr2=0x562d95644760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d95633380 0x562d95644760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 -- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d956467f0 msgr2=0x562d95659280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d956467f0 0x562d95659280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 -- 172.21.15.28:0/866763516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c000e040 con 0x562d9565a830 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05d5400640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f05c0004000 tx=0x7f05c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff69530cb9f8531 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.521+0000 7f05b3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0023070 con 0x562d9565a830 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562d955be600 con 0x562d9565a830 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05c0029ad0 con 0x562d9565a830 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05c0010b00 con 0x562d9565a830 2024-09-13T17:19:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 10 client.10233 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x562d95593440 con 0x562d9565a830 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562d955be600 con 0x562d9565a830 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f05c0029c90 con 0x562d9565a830 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 10 client.10233 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 10 client.10233 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.522+0000 7f05db6a2740 20 client.10233 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9567b450 0x562d9567b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05db6a2740 4 client.10233 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562d95586c90 con 0x562d9567b450 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05d5e00640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9567b450 0x562d9567b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05d5e00640 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9567b450 0x562d9567b830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be9afb7fe78203b4 server_cookie=723f0e7bcaa56147 in_seq=0 out_seq=0 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.523+0000 7f05b3400640 10 client.10233 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05b3400640 1 -- 172.21.15.28:0/866763516 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562d95586c90 con 0x562d9567b450 2024-09-13T17:19:55.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05b3400640 10 client.10233 handle_command_reply: tid=0 2024-09-13T17:19:55.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 client.10233 shutdown 2024-09-13T17:19:55.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 msgr2=0x562d95675010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f05c0004000 tx=0x7f05c0004730 comp rx=0 tx=0).stop 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 shutdown_connections 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9567b450 0x562d9567b830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d956467f0 0x562d95659280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d9565a830 0x562d95675010 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 --2- 172.21.15.28:0/866763516 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d95633380 0x562d95644760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05b3400640 0 client.10233 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.524+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 >> 172.21.15.28:0/866763516 conn(0x562d95630b40 msgr2=0x562d95678a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.525+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 shutdown_connections 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.525+0000 7f05db6a2740 1 -- 172.21.15.28:0/866763516 wait complete. 2024-09-13T17:19:55.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.525+0000 7f05db6a2740 20 client.10233 trim_cache size 0 max 16384 2024-09-13T17:19:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.539+0000 7f05db6a2740 1 Processor -- start 2024-09-13T17:19:55.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.539+0000 7f05db6a2740 1 -- start start 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d9565cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d95678a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d952fc040 con 0x562d95633380 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d9568a660 con 0x562d9565a830 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562d95675820 con 0x562d956467f0 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36906/0 (socket says 172.21.15.28:36906) 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.540+0000 7f05d5e00640 1 -- 172.21.15.28:0/516357957 learned_addr learned my addr 172.21.15.28:0/516357957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5400640 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d9565cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d6800640 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d95678a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 -- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 msgr2=0x562d9565cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d9565cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 -- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 msgr2=0x562d95678a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d95678a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 -- 172.21.15.28:0/516357957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562d955355d0 con 0x562d95633380 2024-09-13T17:19:55.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05d5e00640 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x562d9567bf10 tx=0x7f05c4012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e217032ba149e71f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05bf400640 1 -- 172.21.15.28:0/516357957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4028070 con 0x562d95633380 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05bf400640 1 -- 172.21.15.28:0/516357957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c4029310 con 0x562d95633380 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.541+0000 7f05bf400640 1 -- 172.21.15.28:0/516357957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4029480 con 0x562d95633380 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 -- 172.21.15.28:0/516357957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 msgr2=0x562d9562f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x562d9567bf10 tx=0x7f05c4012000 comp rx=0 tx=0).stop 2024-09-13T17:19:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 -- 172.21.15.28:0/516357957 shutdown_connections 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d95678a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d9565cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 --2- 172.21.15.28:0/516357957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d9562f0f0 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 -- 172.21.15.28:0/516357957 >> 172.21.15.28:0/516357957 conn(0x562d95630b40 msgr2=0x562d956579a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 -- 172.21.15.28:0/516357957 shutdown_connections 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.542+0000 7f05db6a2740 1 -- 172.21.15.28:0/516357957 wait complete. 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.543+0000 7f05db6a2740 1 -- start start 2024-09-13T17:19:55.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.543+0000 7f05db6a2740 1 Processor -- start 2024-09-13T17:19:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d956e65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d956e1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d956e2990 con 0x562d95633380 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562d956592a0 con 0x562d9565a830 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562d956780d0 con 0x562d956467f0 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05db6a2740 10 client.0 _ll_register_callbacks cb 0x7f05c4017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36916/0 (socket says 172.21.15.28:36916) 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05d5e00640 1 -- 172.21.15.28:0/1907995430 learned_addr learned my addr 172.21.15.28:0/1907995430 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05d6800640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d956e1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.544+0000 7f05d5400640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d956e65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 -- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 msgr2=0x562d956e65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d956e65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 -- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 msgr2=0x562d956e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d956e1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 -- 172.21.15.28:0/1907995430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d006c8b0 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05d5e00640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f05c402d040 tx=0x7f05c4029a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b595c7aa22a1773 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05b3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c4023860 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05c401c040 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05c0010c60 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05c4029090 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05db6a2740 10 client.10239 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.545+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x562d9555a9b0 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c0010c60 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f05c4031070 con 0x562d95633380 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 10 client.10239 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 10 client.10239 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 20 client.10239 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9562f0f0 0x562d95691430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 4 client.10239 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x562d952884e0 con 0x562d9562f0f0 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.546+0000 7f05d5400640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9562f0f0 0x562d95691430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05d5400640 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9562f0f0 0x562d95691430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34de2fe8fabecde1 server_cookie=d7f102803ff5e8a3 in_seq=0 out_seq=0 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05b3400640 10 client.10239 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05b3400640 1 -- 172.21.15.28:0/1907995430 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+25 (crc 0 0 0) 0x562d952884e0 con 0x562d9562f0f0 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05b3400640 10 client.10239 handle_command_reply: tid=0 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 client.10239 shutdown 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 msgr2=0x562d95645c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f05c402d040 tx=0x7f05c4029a90 comp rx=0 tx=0).stop 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 shutdown_connections 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x562d9562f0f0 0x562d95691430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562d9565a830 0x562d956e1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562d956467f0 0x562d956e65a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 --2- 172.21.15.28:0/1907995430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562d95633380 0x562d95645c20 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05b3400640 0 client.10239 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 >> 172.21.15.28:0/1907995430 conn(0x562d95630b40 msgr2=0x562d95679cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 shutdown_connections 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 1 -- 172.21.15.28:0/1907995430 wait complete. 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.547+0000 7f05db6a2740 20 client.10239 trim_cache size 0 max 16384 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0 2024-09-13T17:19:55.553 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 msgr2=0x7f05b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 0x7f05b406d0c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f60 tx=0x7f05c4009040 comp rx=0 tx=0).stop 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 msgr2=0x7f05d006b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009e50 tx=0x7f05cc004a40 comp rx=0 tx=0).stop 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 shutdown_connections 2024-09-13T17:19:55.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b406ac00 0x7f05b406d0c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d015ab60 0x7f05d006b580 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d015a780 0x7f05d014edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 --2- 172.21.15.28:0/2674078649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d0071490 0x7f05d0152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 >> 172.21.15.28:0/2674078649 conn(0x7f05d00b0270 msgr2=0x7f05d0154480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 shutdown_connections 2024-09-13T17:19:55.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.548+0000 7f05d8800640 1 -- 172.21.15.28:0/2674078649 wait complete. 2024-09-13T17:19:55.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.11:0 scrub status 2024-09-13T17:19:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.721+0000 7f288f800640 1 Processor -- start 2024-09-13T17:19:55.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.721+0000 7f288f800640 1 -- start start 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288f800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2888159770 0x7f288806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f2888070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288f800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f288806f920 con 0x7f28880add00 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288f800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f288815b290 con 0x7f2888159770 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288815b570 con 0x7f288806f3f0 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.722+0000 7f288ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2888159770 0x7f288806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2888159770 0x7f288806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36922/0 (socket says 172.21.15.28:36922) 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 -- 172.21.15.28:0/4118122169 learned_addr learned my addr 172.21.15.28:0/4118122169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f2887e00640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f2888070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 -- 172.21.15.28:0/4118122169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 msgr2=0x7f2888070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f2888070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 -- 172.21.15.28:0/4118122169 >> v1:172.21.15.189:6789/0 conn(0x7f28880add00 legacy=0x7f2888159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.723+0000 7f288ce00640 1 -- 172.21.15.28:0/4118122169 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288815b850 con 0x7f2888159770 2024-09-13T17:19:55.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.724+0000 7f288ce00640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2888159770 0x7f288806fd20 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f287c009870 tx=0x7f287c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f8e1717d4eeda9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.724+0000 7f2887400640 1 -- 172.21.15.28:0/4118122169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c032440 con 0x7f2888159770 2024-09-13T17:19:55.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.724+0000 7f2887400640 1 -- 172.21.15.28:0/4118122169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c0325a0 con 0x7f2888159770 2024-09-13T17:19:55.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.724+0000 7f2887400640 1 -- 172.21.15.28:0/4118122169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c032930 con 0x7f2888159770 2024-09-13T17:19:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 -- 172.21.15.28:0/4118122169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2888159770 msgr2=0x7f288806fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2888159770 0x7f288806fd20 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f287c009870 tx=0x7f287c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 -- 172.21.15.28:0/4118122169 shutdown_connections 2024-09-13T17:19:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f2888070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 --2- 172.21.15.28:0/4118122169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2888159770 0x7f288806fd20 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 -- 172.21.15.28:0/4118122169 >> 172.21.15.28:0/4118122169 conn(0x7f28880b0270 msgr2=0x7f28880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 -- 172.21.15.28:0/4118122169 shutdown_connections 2024-09-13T17:19:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.725+0000 7f288f800640 1 -- 172.21.15.28:0/4118122169 wait complete. 2024-09-13T17:19:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.727+0000 7f288f800640 1 Processor -- start 2024-09-13T17:19:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.727+0000 7f288f800640 1 -- start start 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f28881e4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888159770 0x7f28880af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888145d60 con 0x7f28880add00 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2888142a90 con 0x7f2888159770 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28881437d0 con 0x7f288806f3f0 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36926/0 (socket says 172.21.15.28:36926) 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 -- 172.21.15.28:0/1955785113 learned_addr learned my addr 172.21.15.28:0/1955785113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288ce00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f28881e4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f288d800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888159770 0x7f28880af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 -- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 msgr2=0x7f28881e4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f28881e4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 -- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888159770 msgr2=0x7f28880af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888159770 0x7f28880af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.728+0000 7f2887e00640 1 -- 172.21.15.28:0/1955785113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c0094d0 con 0x7f28880add00 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f2887e00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f287400e870 tx=0x7f287400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f2884c00640 1 -- 172.21.15.28:0/1955785113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740094f0 con 0x7f28880add00 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f2884c00640 1 -- 172.21.15.28:0/1955785113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287400c940 con 0x7f28880add00 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f2884c00640 1 -- 172.21.15.28:0/1955785113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287400ccf0 con 0x7f28880add00 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28881e5490 con 0x7f28880add00 2024-09-13T17:19:55.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.729+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2888143180 con 0x7f28880add00 2024-09-13T17:19:55.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.731+0000 7f2884c00640 1 -- 172.21.15.28:0/1955785113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2874013470 con 0x7f28880add00 2024-09-13T17:19:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.732+0000 7f2884c00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 0x7f286806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.732+0000 7f2884c00640 1 -- 172.21.15.28:0/1955785113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28740893d0 con 0x7f28880add00 2024-09-13T17:19:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.733+0000 7f28927ee740 1 Processor -- start 2024-09-13T17:19:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.733+0000 7f288ce00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 0x7f286806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.733+0000 7f28927ee740 1 -- start start 2024-09-13T17:19:55.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.733+0000 7f288ce00640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 0x7f286806d0f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f287c034000 tx=0x7f287c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 0x55b0f8b65040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8a6b6a0 con 0x55b0f8b62420 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8acecc0 con 0x55b0f8b64c60 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8acc450 con 0x55b0f8b8c110 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36932/0 (socket says 172.21.15.28:36932) 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 -- 172.21.15.28:0/2161890456 learned_addr learned my addr 172.21.15.28:0/2161890456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f288d800640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f288ce00640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 0x55b0f8b65040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 -- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 msgr2=0x55b0f8bb2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 -- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 msgr2=0x55b0f8b65040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 0x55b0f8b65040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f2887e00640 1 -- 172.21.15.28:0/2161890456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0f8a982e0 con 0x55b0f8b62420 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.734+0000 7f288ce00640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 0x55b0f8b65040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f2887e00640 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f2874054780 tx=0x7f2874058280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c69d56d7ed0719a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f2879600640 1 -- 172.21.15.28:0/2161890456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740605c0 con 0x55b0f8b62420 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f2879600640 1 -- 172.21.15.28:0/2161890456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874056620 con 0x55b0f8b62420 2024-09-13T17:19:55.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f2879600640 1 -- 172.21.15.28:0/2161890456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874054990 con 0x55b0f8b62420 2024-09-13T17:19:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f28927ee740 1 -- 172.21.15.28:0/2161890456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 msgr2=0x55b0f8bb2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.735+0000 7f28927ee740 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f2874054780 tx=0x7f2874058280 comp rx=0 tx=0).stop 2024-09-13T17:19:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 -- 172.21.15.28:0/2161890456 shutdown_connections 2024-09-13T17:19:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b62420 0x55b0f8bb2130 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 --2- 172.21.15.28:0/2161890456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b64c60 0x55b0f8b65040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 -- 172.21.15.28:0/2161890456 >> 172.21.15.28:0/2161890456 conn(0x55b0f8b61390 msgr2=0x55b0f8ba9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 -- 172.21.15.28:0/2161890456 shutdown_connections 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 -- 172.21.15.28:0/2161890456 wait complete. 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.736+0000 7f28927ee740 1 -- start start 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.737+0000 7f28927ee740 1 Processor -- start 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.737+0000 7f28927ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.737+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f28927ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8ac7d40 con 0x55b0f8b64c60 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8a69280 con 0x55b0f8bb1a20 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8a68f20 con 0x55b0f8b8c110 2024-09-13T17:19:55.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f28927ee740 10 client.0 _ll_register_callbacks cb 0x55b0f8ac13b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36946/0 (socket says 172.21.15.28:36946) 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 -- 172.21.15.28:0/2769032728 learned_addr learned my addr 172.21.15.28:0/2769032728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288d800640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f2887e00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 -- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 msgr2=0x55b0f8bb14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 -- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 msgr2=0x55b0f8baf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.738+0000 7f288ce00640 1 -- 172.21.15.28:0/2769032728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2888143180 con 0x55b0f8b64c60 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f288ce00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f287c009990 tx=0x7f287c02c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b97be7a0079700 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:55.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c03ce60 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287c050410 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0f87bb4e0 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f287c0505d0 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f28927ee740 10 client.10251 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55b0f8ac63c0 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b0f87bb4e0 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.739+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f287c044040 con 0x55b0f8b64c60 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 10 client.10251 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 10 client.10251 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 20 client.10251 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8b8a6f0 0x55b0f8b8aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 4 client.10251 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:19:55.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0f87cc7e0 con 0x55b0f8b8a6f0 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f2887e00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8b8a6f0 0x55b0f8b8aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f2887e00640 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8b8a6f0 0x55b0f8b8aad0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2acd147350bf7171 server_cookie=95ca1f76ba896297 in_seq=0 out_seq=0 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.740+0000 7f286e000640 10 client.10251 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f286e000640 1 -- 172.21.15.28:0/2769032728 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55b0f87cc7e0 con 0x55b0f8b8a6f0 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f286e000640 10 client.10251 handle_command_reply: tid=0 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 client.10251 shutdown 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 msgr2=0x55b0f8b89a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f287c009990 tx=0x7f287c02c070 comp rx=0 tx=0).stop 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 shutdown_connections 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8b8a6f0 0x55b0f8b8aad0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b8c110 0x55b0f8bb14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 --2- 172.21.15.28:0/2769032728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8b89a30 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f286e000640 0 client.10251 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 >> 172.21.15.28:0/2769032728 conn(0x55b0f8b62420 msgr2=0x55b0f8baa330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.741+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 shutdown_connections 2024-09-13T17:19:55.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.742+0000 7f28927ee740 1 -- 172.21.15.28:0/2769032728 wait complete. 2024-09-13T17:19:55.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.742+0000 7f28927ee740 20 client.10251 trim_cache size 0 max 16384 2024-09-13T17:19:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 Processor -- start 2024-09-13T17:19:55.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 -- start start 2024-09-13T17:19:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b64c60 0x55b0f8bb0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baa300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8baddd0 con 0x55b0f8b8c110 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f882f040 con 0x55b0f8bb1a20 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.757+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8b8bf50 con 0x55b0f8b64c60 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36954/0 (socket says 172.21.15.28:36954) 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 -- 172.21.15.28:0/2740547342 learned_addr learned my addr 172.21.15.28:0/2740547342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f288ce00640 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b64c60 0x55b0f8bb0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f288d800640 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 -- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b64c60 msgr2=0x55b0f8bb0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b64c60 0x55b0f8bb0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 -- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 msgr2=0x55b0f8baa300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baa300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 -- 172.21.15.28:0/2740547342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0f8a68610 con 0x55b0f8b8c110 2024-09-13T17:19:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f2887e00640 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2874054780 tx=0x7f287405d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b9e9d85e6a7dcb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.758+0000 7f287a000640 1 -- 172.21.15.28:0/2740547342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28740890c0 con 0x55b0f8b8c110 2024-09-13T17:19:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f287a000640 1 -- 172.21.15.28:0/2740547342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874085070 con 0x55b0f8b8c110 2024-09-13T17:19:55.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f287a000640 1 -- 172.21.15.28:0/2740547342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874089220 con 0x55b0f8b8c110 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 -- 172.21.15.28:0/2740547342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 msgr2=0x55b0f8bb11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f2874054780 tx=0x7f287405d000 comp rx=0 tx=0).stop 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 -- 172.21.15.28:0/2740547342 shutdown_connections 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8bb1a20 0x55b0f8baa300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b8c110 0x55b0f8bb11a0 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 --2- 172.21.15.28:0/2740547342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8b64c60 0x55b0f8bb0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 -- 172.21.15.28:0/2740547342 >> 172.21.15.28:0/2740547342 conn(0x55b0f8b62420 msgr2=0x55b0f8ba98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 -- 172.21.15.28:0/2740547342 shutdown_connections 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.759+0000 7f28927ee740 1 -- 172.21.15.28:0/2740547342 wait complete. 2024-09-13T17:19:55.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.760+0000 7f28927ee740 1 -- start start 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.760+0000 7f28927ee740 1 Processor -- start 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b8c110 0x55b0f8c16e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8bb1a20 0x55b0f8bc7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8afc0f0 con 0x55b0f8b64c60 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8c0d3a0 con 0x55b0f8b8c110 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0f8baec30 con 0x55b0f8bb1a20 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f28927ee740 10 client.0 _ll_register_callbacks cb 0x7f2874009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f288ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f288ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36964/0 (socket says 172.21.15.28:36964) 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.761+0000 7f288ce00640 1 -- 172.21.15.28:0/1147107831 learned_addr learned my addr 172.21.15.28:0/1147107831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 -- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8bb1a20 msgr2=0x55b0f8bc7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8bb1a20 0x55b0f8bc7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 -- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b8c110 msgr2=0x55b0f8c16e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b8c110 0x55b0f8c16e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 -- 172.21.15.28:0/1147107831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0f8a982e0 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f288ce00640 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f287c009870 tx=0x7f287c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7cdc2674bf7a016 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f286e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c050dc0 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f287c060aa0 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2874056620 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7f287c03c950 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f28927ee740 10 client.10257 fetch_fsmap learned FSMap version 138 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.762+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=2+}) v3 -- 0x55b0f8a8db20 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2874056620 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 138) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f287c066020 con 0x55b0f8b64c60 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 10 client.10257 fetch_fsmap finished waiting for FSMap version 138 2024-09-13T17:19:55.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 10 client.10257 resolve_mds: resolved 11:0 to role '11:0' aka daemon mds.c 2024-09-13T17:19:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 20 client.10257 populate_metadata read hostname 'smithi028' 2024-09-13T17:19:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8c12f40 0x55b0f8b8aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 4 client.10257 mds_command: new command op to 10041 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:19:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 --> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b0f881fe30 con 0x55b0f8c12f40 2024-09-13T17:19:55.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f2887e00640 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8c12f40 0x55b0f8b8aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f2887e00640 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8c12f40 0x55b0f8b8aad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599cc1399e31eb49 server_cookie=a73b05ac8af9a217 in_seq=0 out_seq=0 2024-09-13T17:19:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.763+0000 7f286e000640 10 client.10257 ms_handle_connect on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f286e000640 1 -- 172.21.15.28:0/1147107831 <== mds.0 v2:172.21.15.28:6833/3709710897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+276 (crc 0 0 0) 0x55b0f881fe30 con 0x55b0f8c12f40 2024-09-13T17:19:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f286e000640 10 client.10257 handle_command_reply: tid=0 2024-09-13T17:19:55.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 client.10257 shutdown 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 msgr2=0x55b0f8c0ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f287c009870 tx=0x7f287c009240 comp rx=0 tx=0).stop 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 shutdown_connections 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:6833/3709710897,v1:172.21.15.28:6835/3709710897] conn(0x55b0f8c12f40 0x55b0f8b8aad0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55b0f8bb1a20 0x55b0f8bc7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0f8b8c110 0x55b0f8c16e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 --2- 172.21.15.28:0/1147107831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b0f8b64c60 0x55b0f8c0ce60 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 >> 172.21.15.28:0/1147107831 conn(0x55b0f8b62420 msgr2=0x55b0f8bada60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f286e000640 0 client.10257 ms_handle_reset on v2:172.21.15.28:6833/3709710897 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 shutdown_connections 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 1 -- 172.21.15.28:0/1147107831 wait complete. 2024-09-13T17:19:55.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.764+0000 7f28927ee740 20 client.10257 trim_cache size 0 max 16384 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "PAUSED (0 inodes in the stack)", 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "bdef086f-0d4a-4728-a2b8-a3c324f1bfc0": { 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/scrub_control_test_path", 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "bdef086f-0d4a-4728-a2b8-a3c324f1bfc0", 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:19:55.771 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:19:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 msgr2=0x7f286806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 0x7f286806d0f0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f287c034000 tx=0x7f287c02e000 comp rx=0 tx=0).stop 2024-09-13T17:19:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 msgr2=0x7f28880af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f287400e870 tx=0x7f287400ed30 comp rx=0 tx=0).stop 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 shutdown_connections 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f286806ac30 0x7f286806d0f0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2888159770 0x7f28880af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28880add00 0x7f28880af120 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 --2- 172.21.15.28:0/1955785113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f288806f3f0 0x7f28881e4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 >> 172.21.15.28:0/1955785113 conn(0x7f28880b0270 msgr2=0x7f28880ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 shutdown_connections 2024-09-13T17:19:55.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.765+0000 7f288f800640 1 -- 172.21.15.28:0/1955785113 wait complete. 2024-09-13T17:19:55.782 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.936+0000 7fd760400640 1 Processor -- start 2024-09-13T17:19:55.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.936+0000 7fd760400640 1 -- start start 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd760400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd758163ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd760400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7580b3970 con 0x7fd7580cdfa0 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd760400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd758159430 con 0x7fd75815d340 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd760400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd758159230 con 0x7fd75806fb50 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd75da00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd7580cdfa0 0x7fd7580b3160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55630/0 (socket says 172.21.15.28:55630) 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.937+0000 7fd75da00640 1 -- 172.21.15.28:0/2295046535 learned_addr learned my addr 172.21.15.28:0/2295046535 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd75e400640 1 --2- 172.21.15.28:0/2295046535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd758163ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502691744 0 0) 0x7fd74c001110 con 0x7fd7580cdfa0 2024-09-13T17:19:55.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003440 con 0x7fd7580cdfa0 2024-09-13T17:19:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179752132 0 0) 0x7fd758159430 con 0x7fd75815d340 2024-09-13T17:19:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c001110 con 0x7fd75815d340 2024-09-13T17:19:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd75e400640 1 -- 172.21.15.28:0/2295046535 >> v1:172.21.15.189:6789/0 conn(0x7fd75815d340 legacy=0x7fd75806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd75e400640 1 -- 172.21.15.28:0/2295046535 >> v1:172.21.15.28:6789/0 conn(0x7fd7580cdfa0 legacy=0x7fd7580b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.938+0000 7fd75e400640 1 -- 172.21.15.28:0/2295046535 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758159710 con 0x7fd75806fb50 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.939+0000 7fd75e400640 1 --2- 172.21.15.28:0/2295046535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd758163ba0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd750009dc0 tx=0x7fd75002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6c2ec1bf348fe6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.939+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750007740 con 0x7fd75806fb50 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.939+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75002d050 con 0x7fd75806fb50 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.939+0000 7fd757e00640 1 -- 172.21.15.28:0/2295046535 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750007aa0 con 0x7fd75806fb50 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 -- 172.21.15.28:0/2295046535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 msgr2=0x7fd758163ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 --2- 172.21.15.28:0/2295046535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd758163ba0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd750009dc0 tx=0x7fd75002a6d0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 -- 172.21.15.28:0/2295046535 shutdown_connections 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 --2- 172.21.15.28:0/2295046535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd758163ba0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 -- 172.21.15.28:0/2295046535 >> 172.21.15.28:0/2295046535 conn(0x7fd758069e80 msgr2=0x7fd75806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:55.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 -- 172.21.15.28:0/2295046535 shutdown_connections 2024-09-13T17:19:55.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.940+0000 7fd760400640 1 -- 172.21.15.28:0/2295046535 wait complete. 2024-09-13T17:19:55.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.942+0000 7fd760400640 1 Processor -- start 2024-09-13T17:19:55.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.942+0000 7fd760400640 1 -- start start 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7580cdfa0 0x7fd758147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd75815d340 0x7fd7580aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580af580 con 0x7fd7580cdfa0 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580af860 con 0x7fd75815d340 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd760400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580afb40 con 0x7fd75806fb50 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7580cdfa0 0x7fd758147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:60880/0 (socket says 172.21.15.28:60880) 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 -- 172.21.15.28:0/996444732 learned_addr learned my addr 172.21.15.28:0/996444732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:55.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75e400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd75815d340 0x7fd7580aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 -- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd75815d340 msgr2=0x7fd7580aeba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd75815d340 0x7fd7580aeba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 -- 172.21.15.28:0/996444732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7580cdfa0 msgr2=0x7fd758147ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.943+0000 7fd75da00640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7580cdfa0 0x7fd758147ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd75da00640 1 -- 172.21.15.28:0/996444732 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750009a70 con 0x7fd75806fb50 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd75da00640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00c730 tx=0x7fd74c00cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c005ab0 con 0x7fd75806fb50 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c005c10 con 0x7fd75806fb50 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7580afe20 con 0x7fd75806fb50 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c007070 con 0x7fd75806fb50 2024-09-13T17:19:55.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.944+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd758148410 con 0x7fd75806fb50 2024-09-13T17:19:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.945+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7580b0340 con 0x7fd75806fb50 2024-09-13T17:19:55.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.946+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd74c0073e0 con 0x7fd75806fb50 2024-09-13T17:19:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.947+0000 7fd755600640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 0x7fd73806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.947+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd74c088ac0 con 0x7fd75806fb50 2024-09-13T17:19:55.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.947+0000 7fd75d000640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 0x7fd73806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:55.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.948+0000 7fd75d000640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 0x7fd73806d0f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd744007fd0 tx=0x7fd744009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:55.952+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd7580b0340 con 0x7fd75806fb50 2024-09-13T17:19:56.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.164+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7580b3970 con 0x7fd75806fb50 2024-09-13T17:19:56.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.165+0000 7fd755600640 1 -- 172.21.15.28:0/996444732 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fd7580b3970 con 0x7fd75806fb50 2024-09-13T17:19:56.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:56.167 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:56.168 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.167+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 msgr2=0x7fd73806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.167+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 0x7fd73806d0f0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd744007fd0 tx=0x7fd744009040 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.167+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 msgr2=0x7fd7580b0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.167+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd74c00c730 tx=0x7fd74c00cbf0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 shutdown_connections 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd73806ac30 0x7fd73806d0f0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd75815d340 0x7fd7580aeba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7580cdfa0 0x7fd758147ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 --2- 172.21.15.28:0/996444732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd75806fb50 0x7fd7580b0620 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 >> 172.21.15.28:0/996444732 conn(0x7fd758069e80 msgr2=0x7fd7580b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 shutdown_connections 2024-09-13T17:19:56.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.168+0000 7fd760400640 1 -- 172.21.15.28:0/996444732 wait complete. 2024-09-13T17:19:56.185 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:56.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.338+0000 7fb4ee400640 1 Processor -- start 2024-09-13T17:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.338+0000 7fb4ee400640 1 -- start start 2024-09-13T17:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4ee400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e8146170 0x7fb4e815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4ee400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4e8160310 0x7fb4e8160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4ee400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4e8146550 con 0x7fb4e80ac850 2024-09-13T17:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4ee400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4e815cd20 con 0x7fb4e8160310 2024-09-13T17:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4ee400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e815d280 con 0x7fb4e8146170 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4dea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4e8160310 0x7fb4e8160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4dea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4e8160310 0x7fb4e8160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36982/0 (socket says 172.21.15.28:36982) 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.339+0000 7fb4dea00640 1 -- 172.21.15.28:0/1002744207 learned_addr learned my addr 172.21.15.28:0/1002744207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.340+0000 7fb4e7400640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e8146170 0x7fb4e815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.340+0000 7fb4dea00640 1 -- 172.21.15.28:0/1002744207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e8146170 msgr2=0x7fb4e815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.340+0000 7fb4dea00640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e8146170 0x7fb4e815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.340+0000 7fb4dea00640 1 -- 172.21.15.28:0/1002744207 >> v1:172.21.15.189:6789/0 conn(0x7fb4e80ac850 legacy=0x7fb4e806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.340+0000 7fb4dea00640 1 -- 172.21.15.28:0/1002744207 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e815dc50 con 0x7fb4e8160310 2024-09-13T17:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.341+0000 7fb4dea00640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8160750 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009870 tx=0x7fb4d002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8f938005fc7614 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.341+0000 7fb4e6a00640 1 -- 172.21.15.28:0/1002744207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0032440 con 0x7fb4e8160310 2024-09-13T17:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.341+0000 7fb4e6a00640 1 -- 172.21.15.28:0/1002744207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d00325a0 con 0x7fb4e8160310 2024-09-13T17:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.341+0000 7fb4e6a00640 1 -- 172.21.15.28:0/1002744207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0032930 con 0x7fb4e8160310 2024-09-13T17:19:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 -- 172.21.15.28:0/1002744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 msgr2=0x7fb4e8160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8160750 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009870 tx=0x7fb4d002a250 comp rx=0 tx=0).stop 2024-09-13T17:19:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 -- 172.21.15.28:0/1002744207 shutdown_connections 2024-09-13T17:19:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8160750 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1002744207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e8146170 0x7fb4e815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 -- 172.21.15.28:0/1002744207 >> 172.21.15.28:0/1002744207 conn(0x7fb4e8069e80 msgr2=0x7fb4e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 -- 172.21.15.28:0/1002744207 shutdown_connections 2024-09-13T17:19:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.342+0000 7fb4ee400640 1 -- 172.21.15.28:0/1002744207 wait complete. 2024-09-13T17:19:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.344+0000 7fb4ee400640 1 Processor -- start 2024-09-13T17:19:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.344+0000 7fb4ee400640 1 -- start start 2024-09-13T17:19:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8156c70 con 0x7fb4e8160310 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8155150 con 0x7fb4e8146170 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4ee400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4e8155430 con 0x7fb4e80ac850 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4e7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4dea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4dea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:33598/0 (socket says 172.21.15.28:33598) 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4dea00640 1 -- 172.21.15.28:0/1737293212 learned_addr learned my addr 172.21.15.28:0/1737293212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.345+0000 7fb4e7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:60894/0 (socket says 172.21.15.28:60894) 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 -- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 msgr2=0x7fb4e8143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 -- 172.21.15.28:0/1737293212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 msgr2=0x7fb4e8159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 -- 172.21.15.28:0/1737293212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d00094d0 con 0x7fb4e8146170 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4e7400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dea00640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009870 tx=0x7fb4d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d0032440 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d0030ba0 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4d003b400 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4e8155710 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.346+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4e8143ab0 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.347+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4e8157150 con 0x7fb4e8146170 2024-09-13T17:19:56.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.349+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4d0039040 con 0x7fb4e8146170 2024-09-13T17:19:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.349+0000 7fb4dfe00640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 0x7fb4c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.350+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4d00a9cd0 con 0x7fb4e8146170 2024-09-13T17:19:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.350+0000 7fb4e7400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 0x7fb4c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:56.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.351+0000 7fb4e7400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 0x7fb4c406d0c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8006fd0 tx=0x7fb4d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:56.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.353+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb4d0075010 con 0x7fb4e8146170 2024-09-13T17:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.598+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fb4e815d710 con 0x7fb4e8146170 2024-09-13T17:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.599+0000 7fb4dfe00640 1 -- 172.21.15.28:0/1737293212 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1189 (secure 0 0 0) 0x7fb4d004c4c0 con 0x7fb4e8146170 2024-09-13T17:19:56.601 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:56.601 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":730,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":167,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":926,"data_bytes":947840769,"bytes_used":2417369088,"bytes_avail":770676744192,"bytes_total":773094113280,"write_bytes_sec":49729246,"read_op_per_sec":0,"write_op_per_sec":48},"fsmap":{"epoch":138,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":10041}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":33,"modified":"2024-09-13T17:19:23.055454+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 msgr2=0x7fb4c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 0x7fb4c406d0c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fb4d8006fd0 tx=0x7fb4d8008040 comp rx=0 tx=0).stop 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 msgr2=0x7fb4e8154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009870 tx=0x7fb4d00046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 shutdown_connections 2024-09-13T17:19:56.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4c406ac00 0x7fb4c406d0c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4e8160310 0x7fb4e8159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4e8146170 0x7fb4e8154800 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 --2- 172.21.15.28:0/1737293212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4e80ac850 0x7fb4e8143570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 >> 172.21.15.28:0/1737293212 conn(0x7fb4e8069e80 msgr2=0x7fb4e80af3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 shutdown_connections 2024-09-13T17:19:56.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:56.601+0000 7fb4ee400640 1 -- 172.21.15.28:0/1737293212 wait complete. 2024-09-13T17:19:56.618 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:19:57.619 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.784+0000 7f8c85c00640 1 Processor -- start 2024-09-13T17:19:57.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.785+0000 7f8c85c00640 1 -- start start 2024-09-13T17:19:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c85c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 0x7f8c800b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c85c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c800c3850 con 0x7f8c80159440 2024-09-13T17:19:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c85c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c8015a930 con 0x7f8c80159820 2024-09-13T17:19:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c85c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80159d20 con 0x7f8c800b0b50 2024-09-13T17:19:57.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c7fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8c80159440 0x7f8c800b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55636/0 (socket says 172.21.15.28:55636) 2024-09-13T17:19:57.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c7fe00640 1 -- 172.21.15.28:0/3662414462 learned_addr learned my addr 172.21.15.28:0/3662414462 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:57.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.786+0000 7f8c77e00640 1 --2- 172.21.15.28:0/3662414462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 0x7f8c800b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323117544 0 0) 0x7f8c800c3850 con 0x7f8c80159440 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c64003520 con 0x7f8c80159440 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190846393 0 0) 0x7f8c8015a930 con 0x7f8c80159820 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c800c3850 con 0x7f8c80159820 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c77e00640 1 -- 172.21.15.28:0/3662414462 >> v1:172.21.15.189:6789/0 conn(0x7f8c80159820 legacy=0x7f8c8015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c77e00640 1 -- 172.21.15.28:0/3662414462 >> v1:172.21.15.28:6789/0 conn(0x7f8c80159440 legacy=0x7f8c800b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:57.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1622893911 0 0) 0x7f8c64003520 con 0x7f8c80159440 2024-09-13T17:19:57.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.787+0000 7f8c77e00640 1 -- 172.21.15.28:0/3662414462 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c8015a000 con 0x7f8c800b0b50 2024-09-13T17:19:57.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.788+0000 7f8c77e00640 1 --2- 172.21.15.28:0/3662414462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 0x7f8c800b22e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c680097d0 tx=0x7f8c6802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32d4fb2c2e18d628 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c70001670 con 0x7f8c800b0b50 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6802acc0 con 0x7f8c800b0b50 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c7f400640 1 -- 172.21.15.28:0/3662414462 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c68032660 con 0x7f8c800b0b50 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 -- 172.21.15.28:0/3662414462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 msgr2=0x7f8c800b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 --2- 172.21.15.28:0/3662414462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 0x7f8c800b22e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8c680097d0 tx=0x7f8c6802a230 comp rx=0 tx=0).stop 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 -- 172.21.15.28:0/3662414462 shutdown_connections 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 --2- 172.21.15.28:0/3662414462 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c800b0b50 0x7f8c800b22e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:57.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 -- 172.21.15.28:0/3662414462 >> 172.21.15.28:0/3662414462 conn(0x7f8c80072e40 msgr2=0x7f8c80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:57.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 -- 172.21.15.28:0/3662414462 shutdown_connections 2024-09-13T17:19:57.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.789+0000 7f8c85c00640 1 -- 172.21.15.28:0/3662414462 wait complete. 2024-09-13T17:19:57.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.791+0000 7f8c85c00640 1 Processor -- start 2024-09-13T17:19:57.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.791+0000 7f8c85c00640 1 -- start start 2024-09-13T17:19:57.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c800b0b50 0x7f8c800c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:57.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:57.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c80159820 0x7f8c8006b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80146f20 con 0x7f8c80159440 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80149ab0 con 0x7f8c800b0b50 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c85c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c80149d90 con 0x7f8c80159820 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36992/0 (socket says 172.21.15.28:36992) 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.792+0000 7f8c77e00640 1 -- 172.21.15.28:0/2656822636 learned_addr learned my addr 172.21.15.28:0/2656822636 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c7fe00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c800b0b50 0x7f8c800c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:57.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c85200640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c80159820 0x7f8c8006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 -- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c80159820 msgr2=0x7f8c8006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c80159820 0x7f8c8006b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 -- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c800b0b50 msgr2=0x7f8c800c68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c800b0b50 0x7f8c800c68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:57.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 -- 172.21.15.28:0/2656822636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c68009400 con 0x7f8c80159440 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.793+0000 7f8c77e00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f8c68035040 tx=0x7f8c680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.794+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6803b810 con 0x7f8c80159440 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.794+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6803b990 con 0x7f8c80159440 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.794+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c6803bd40 con 0x7f8c80159440 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.794+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c80147170 con 0x7f8c80159440 2024-09-13T17:19:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.794+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c800c6de0 con 0x7f8c80159440 2024-09-13T17:19:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.795+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c80147690 con 0x7f8c80159440 2024-09-13T17:19:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.796+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c68039040 con 0x7f8c80159440 2024-09-13T17:19:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.797+0000 7f8c7cc00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 0x7f8c5c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.797+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c680a9880 con 0x7f8c80159440 2024-09-13T17:19:57.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.797+0000 7f8c7fe00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 0x7f8c5c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:57.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.798+0000 7f8c7fe00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 0x7f8c5c06d0f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8c700011e0 tx=0x7f8c700016d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:57.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:57.801+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8c80147690 con 0x7f8c80159440 2024-09-13T17:19:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.015+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c80147690 con 0x7f8c80159440 2024-09-13T17:19:58.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.016+0000 7f8c7cc00640 1 -- 172.21.15.28:0/2656822636 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f8c80147690 con 0x7f8c80159440 2024-09-13T17:19:58.018 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:58.018 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:58.018 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 msgr2=0x7f8c5c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 0x7f8c5c06d0f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f8c700011e0 tx=0x7f8c700016d0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 msgr2=0x7f8c80148fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f8c68035040 tx=0x7f8c680046c0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 shutdown_connections 2024-09-13T17:19:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8c5c06ac30 0x7f8c5c06d0f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c80159820 0x7f8c8006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c80159440 0x7f8c80148fb0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 --2- 172.21.15.28:0/2656822636 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c800b0b50 0x7f8c800c68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 >> 172.21.15.28:0/2656822636 conn(0x7f8c80072e40 msgr2=0x7f8c800ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 shutdown_connections 2024-09-13T17:19:58.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.018+0000 7f8c85c00640 1 -- 172.21.15.28:0/2656822636 wait complete. 2024-09-13T17:19:58.036 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:58.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:19:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.190+0000 7f3f61a00640 1 Processor -- start 2024-09-13T17:19:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.190+0000 7f3f61a00640 1 -- start start 2024-09-13T17:19:58.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.191+0000 7f3f61a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f5c146170 0x7f3f5c15d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.191+0000 7f3f61a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c146550 con 0x7f3f5c0acf60 2024-09-13T17:19:58.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.191+0000 7f3f61a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5c15d280 con 0x7f3f5c160310 2024-09-13T17:19:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.191+0000 7f3f61a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5c15cd20 con 0x7f3f5c146170 2024-09-13T17:19:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f5b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f5c0acf60 0x7f3f5c06b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55642/0 (socket says 172.21.15.28:55642) 2024-09-13T17:19:58.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f5b400640 1 -- 172.21.15.28:0/1655477100 learned_addr learned my addr 172.21.15.28:0/1655477100 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f5aa00640 1 --2- 172.21.15.28:0/1655477100 >> v2:172.21.15.189:3300/0 conn(0x7f3f5c146170 0x7f3f5c15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2443180862 0 0) 0x7f3f5c146550 con 0x7f3f5c0acf60 2024-09-13T17:19:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f40003520 con 0x7f3f5c0acf60 2024-09-13T17:19:58.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939735037 0 0) 0x7f3f40003520 con 0x7f3f5c0acf60 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 >> v2:172.21.15.189:3300/0 conn(0x7f3f5c146170 msgr2=0x7f3f5c15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 --2- 172.21.15.28:0/1655477100 >> v2:172.21.15.189:3300/0 conn(0x7f3f5c146170 0x7f3f5c15d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 >> v1:172.21.15.189:6789/0 conn(0x7f3f5c160310 legacy=0x7f3f5c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c15dc50 con 0x7f3f5c0acf60 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3f540019f0 con 0x7f3f5c0acf60 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 >> v1:172.21.15.28:6789/0 conn(0x7f3f5c0acf60 legacy=0x7f3f5c06b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f40003970 0x7f3f40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 0x7f3f40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.192+0000 7f3f59600640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f40005180 0x7f3f4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4000bff0 con 0x7f3f400047c0 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4000c160 con 0x7f3f40003970 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4000c320 con 0x7f3f40005180 2024-09-13T17:19:58.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 0x7f3f40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5aa00640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f40003970 0x7f3f40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5b400640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f40005180 0x7f3f4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 -- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f40005180 msgr2=0x7f3f4000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f40005180 0x7f3f4000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 -- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f40003970 msgr2=0x7f3f40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f40003970 0x7f3f40003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5a000640 1 -- 172.21.15.28:0/1655477100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f40003520 con 0x7f3f400047c0 2024-09-13T17:19:58.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.193+0000 7f3f5b400640 1 -- 172.21.15.28:0/1655477100 reap_dead start 2024-09-13T17:19:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.194+0000 7f3f5a000640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 0x7f3f40004c40 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f3f48001750 tx=0x7f3f400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b14e3a00c84e0a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.194+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f480052d0 con 0x7f3f400047c0 2024-09-13T17:19:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.194+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f48005430 con 0x7f3f400047c0 2024-09-13T17:19:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.194+0000 7f3f59600640 1 -- 172.21.15.28:0/1655477100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802abe0 con 0x7f3f400047c0 2024-09-13T17:19:58.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 -- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 msgr2=0x7f3f40004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 0x7f3f40004c40 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f3f48001750 tx=0x7f3f400057f0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 -- 172.21.15.28:0/1655477100 shutdown_connections 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 --2- 172.21.15.28:0/1655477100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f400047c0 0x7f3f40004c40 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 -- 172.21.15.28:0/1655477100 >> 172.21.15.28:0/1655477100 conn(0x7f3f5c069e80 msgr2=0x7f3f5c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 -- 172.21.15.28:0/1655477100 shutdown_connections 2024-09-13T17:19:58.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.195+0000 7f3f61a00640 1 -- 172.21.15.28:0/1655477100 wait complete. 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.197+0000 7f3f61a00640 1 Processor -- start 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.197+0000 7f3f61a00640 1 -- start start 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f400047c0 0x7f3f5c144b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 0x7f3f5c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c158b70 con 0x7f3f5c1563e0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c15b480 con 0x7f3f5c15aae0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f61a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f5c156db0 con 0x7f3f400047c0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f5b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37018/0 (socket says 172.21.15.28:37018) 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f5b400640 1 -- 172.21.15.28:0/4243311771 learned_addr learned my addr 172.21.15.28:0/4243311771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f5aa00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f400047c0 0x7f3f5c144b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.198+0000 7f3f5a000640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 0x7f3f5c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 -- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f400047c0 msgr2=0x7f3f5c144b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f400047c0 0x7f3f5c144b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 -- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 msgr2=0x7f3f5c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 0x7f3f5c15af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 -- 172.21.15.28:0/4243311771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c1450a0 con 0x7f3f5c1563e0 2024-09-13T17:19:58.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5a000640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 0x7f3f5c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f5b400640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f3f54002230 tx=0x7f3f54007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ea40 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5400ebe0 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54013420 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f5c159e20 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.199+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f5c159060 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.200+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5c15d710 con 0x7f3f5c1563e0 2024-09-13T17:19:58.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.201+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3f54013580 con 0x7f3f5c1563e0 2024-09-13T17:19:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.202+0000 7f3f52a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 0x7f3f3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.202+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3f54089600 con 0x7f3f5c1563e0 2024-09-13T17:19:58.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.203+0000 7f3f5aa00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 0x7f3f3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:58.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.203+0000 7f3f5aa00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 0x7f3f3c06d0c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c068da0 tx=0x7f3f4c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:58.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.206+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3f540547d0 con 0x7f3f5c1563e0 2024-09-13T17:19:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.450+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3f5c0cdd00 con 0x7f3f5c1563e0 2024-09-13T17:19:58.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.451+0000 7f3f52a00640 1 -- 172.21.15.28:0/4243311771 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1189 (secure 0 0 0) 0x7f3f54057df0 con 0x7f3f5c1563e0 2024-09-13T17:19:58.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:58.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":167,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":926,"data_bytes":947840769,"bytes_used":2417369088,"bytes_avail":770676744192,"bytes_total":773094113280,"write_bytes_sec":49730315,"read_op_per_sec":0,"write_op_per_sec":48},"fsmap":{"epoch":138,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":10041}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":33,"modified":"2024-09-13T17:19:23.055454+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.452+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 msgr2=0x7f3f3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.452+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 0x7f3f3c06d0c0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c068da0 tx=0x7f3f4c009190 comp rx=0 tx=0).stop 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.452+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 msgr2=0x7f3f5c156870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.452+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f3f54002230 tx=0x7f3f54007b00 comp rx=0 tx=0).stop 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 shutdown_connections 2024-09-13T17:19:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f3c06ac00 0x7f3f3c06d0c0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f5c1563e0 0x7f3f5c156870 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5c15aae0 0x7f3f5c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 --2- 172.21.15.28:0/4243311771 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f400047c0 0x7f3f5c144b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 >> 172.21.15.28:0/4243311771 conn(0x7f3f5c069e80 msgr2=0x7f3f5c0ae110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 shutdown_connections 2024-09-13T17:19:58.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:58.453+0000 7f3f61a00640 1 -- 172.21.15.28:0/4243311771 wait complete. 2024-09-13T17:19:58.471 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:19:59.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:19:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.631+0000 7f62ff200640 1 Processor -- start 2024-09-13T17:19:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.631+0000 7f62ff200640 1 -- start start 2024-09-13T17:19:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.632+0000 7f62ff200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62f8160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.632+0000 7f62ff200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8146550 con 0x7f62f8146170 2024-09-13T17:19:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.632+0000 7f62ff200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f815d280 con 0x7f62f8160310 2024-09-13T17:19:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.632+0000 7f62ff200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f815cd20 con 0x7f62f80acf60 2024-09-13T17:19:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f62f8146170 0x7f62f815d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55652/0 (socket says 172.21.15.28:55652) 2024-09-13T17:19:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f7e00640 1 -- 172.21.15.28:0/1791270331 learned_addr learned my addr 172.21.15.28:0/1791270331 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:19:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62fd200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62f8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897157154 0 0) 0x7f62f8146550 con 0x7f62f8146170 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc003520 con 0x7f62f8146170 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605617083 0 0) 0x7f62dc003520 con 0x7f62f8146170 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 msgr2=0x7f62f8160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62f8160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 >> v1:172.21.15.189:6789/0 conn(0x7f62f8160310 legacy=0x7f62f80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8160d60 con 0x7f62f8146170 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f62e8001670 con 0x7f62f8146170 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 >> v1:172.21.15.28:6789/0 conn(0x7f62f8146170 legacy=0x7f62f815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62dc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc003f20 0x7f62dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62dc004d00 0x7f62dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00ba30 con 0x7f62dc004d00 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00bba0 con 0x7f62dc003f20 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.633+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62dc00bd60 con 0x7f62f80acf60 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62dc004d00 0x7f62dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62f7400640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc003f20 0x7f62dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62f7e00640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62dc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 -- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 msgr2=0x7f62dc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f80acf60 0x7f62dc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 -- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc003f20 msgr2=0x7f62dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62dc003f20 0x7f62dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.634+0000 7f62fd200640 1 -- 172.21.15.28:0/1791270331 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc003520 con 0x7f62dc004d00 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.635+0000 7f62fd200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62dc004d00 0x7f62dc005180 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f62f806d4d0 tx=0x7f62dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29b1a9b7f770e70 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.635+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f0032640 con 0x7f62dc004d00 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.635+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f000f040 con 0x7f62dc004d00 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.635+0000 7f62f6a00640 1 -- 172.21.15.28:0/1791270331 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f00329a0 con 0x7f62dc004d00 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 -- 172.21.15.28:0/1791270331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62dc004d00 msgr2=0x7f62dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 --2- 172.21.15.28:0/1791270331 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62dc004d00 0x7f62dc005180 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f62f806d4d0 tx=0x7f62dc0057f0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62fd200640 1 -- 172.21.15.28:0/1791270331 reap_dead start 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 -- 172.21.15.28:0/1791270331 shutdown_connections 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 -- 172.21.15.28:0/1791270331 >> 172.21.15.28:0/1791270331 conn(0x7f62f8069e80 msgr2=0x7f62f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 -- 172.21.15.28:0/1791270331 shutdown_connections 2024-09-13T17:19:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.636+0000 7f62ff200640 1 -- 172.21.15.28:0/1791270331 wait complete. 2024-09-13T17:19:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.638+0000 7f62ff200640 1 Processor -- start 2024-09-13T17:19:59.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.638+0000 7f62ff200640 1 -- start start 2024-09-13T17:19:59.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62dc004d00 0x7f62f8141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 0x7f62f81f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f80b0e50 con 0x7f62f81f2a20 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f813ed40 con 0x7f62f81f2060 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62ff200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f813f020 con 0x7f62dc004d00 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37034/0 (socket says 172.21.15.28:37034) 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 -- 172.21.15.28:0/2228241058 learned_addr learned my addr 172.21.15.28:0/2228241058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62f7e00640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62dc004d00 0x7f62f8141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62f7400640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 0x7f62f81f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 -- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62dc004d00 msgr2=0x7f62f8141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62dc004d00 0x7f62f8141030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 -- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 msgr2=0x7f62f81f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 0x7f62f81f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62fd200640 1 -- 172.21.15.28:0/2228241058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f813f300 con 0x7f62f81f2a20 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.639+0000 7f62f7400640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 0x7f62f81f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62fd200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f62f002adb0 tx=0x7f62f0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f003b600 con 0x7f62f81f2a20 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f003b780 con 0x7f62f81f2a20 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f813f5e0 con 0x7f62f81f2a20 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f003bb30 con 0x7f62f81f2a20 2024-09-13T17:19:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.640+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f81f2ea0 con 0x7f62f81f2a20 2024-09-13T17:19:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.641+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f8160730 con 0x7f62f81f2a20 2024-09-13T17:19:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.642+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f62f0039040 con 0x7f62f81f2a20 2024-09-13T17:19:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.643+0000 7f62efe00640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 0x7f62d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:19:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.643+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62f00a9600 con 0x7f62f81f2a20 2024-09-13T17:19:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.643+0000 7f62f7e00640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 0x7f62d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:19:59.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.644+0000 7f62f7e00640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 0x7f62d806d0f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f62e80017a0 tx=0x7f62e80015b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:19:59.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.647+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f62f0078430 con 0x7f62f81f2a20 2024-09-13T17:19:59.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.863+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62f80cdd10 con 0x7f62f81f2a20 2024-09-13T17:19:59.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.864+0000 7f62efe00640 1 -- 172.21.15.28:0/2228241058 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f62f0077fc0 con 0x7f62f81f2a20 2024-09-13T17:19:59.867 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:19:59.867 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:19:59.867 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 msgr2=0x7f62d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 0x7f62d806d0f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f62e80017a0 tx=0x7f62e80015b0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 msgr2=0x7f62f80ae110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f62f002adb0 tx=0x7f62f0009750 comp rx=0 tx=0).stop 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 shutdown_connections 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62d806ac30 0x7f62d806d0f0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f81f2a20 0x7f62f80ae110 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f81f2060 0x7f62f81f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 --2- 172.21.15.28:0/2228241058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62dc004d00 0x7f62f8141030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:19:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 >> 172.21.15.28:0/2228241058 conn(0x7f62f8069e80 msgr2=0x7f62f80ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:19:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 shutdown_connections 2024-09-13T17:19:59.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:19:59.866+0000 7f62ff200640 1 -- 172.21.15.28:0/2228241058 wait complete. 2024-09-13T17:19:59.883 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:19:59.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:20:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.037+0000 7f5b9e800640 1 Processor -- start 2024-09-13T17:20:00.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.037+0000 7f5b9e800640 1 -- start start 2024-09-13T17:20:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.038+0000 7f5b9e800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.038+0000 7f5b9e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.038+0000 7f5b9e800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b9815e100 con 0x7f5b9815edb0 2024-09-13T17:20:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.038+0000 7f5b9e800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b980adb00 con 0x7f5b9815a540 2024-09-13T17:20:00.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.038+0000 7f5b9e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9815e3e0 con 0x7f5b980cabc0 2024-09-13T17:20:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b97400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:60972/0 (socket says 172.21.15.28:60972) 2024-09-13T17:20:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 -- 172.21.15.28:0/2537290693 learned_addr learned my addr 172.21.15.28:0/2537290693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:00.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b97400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:33658/0 (socket says 172.21.15.28:33658) 2024-09-13T17:20:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96000640 1 -- 172.21.15.28:0/2537290693 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335540264 0 0) 0x7f5b9815e100 con 0x7f5b9815edb0 2024-09-13T17:20:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 -- 172.21.15.28:0/2537290693 >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 msgr2=0x7f5b9806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 --2- 172.21.15.28:0/2537290693 >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 -- 172.21.15.28:0/2537290693 >> v1:172.21.15.189:6789/0 conn(0x7f5b9815edb0 legacy=0x7f5b98070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b96a00640 1 -- 172.21.15.28:0/2537290693 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9806ce80 con 0x7f5b980cabc0 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.039+0000 7f5b97400640 1 --2- 172.21.15.28:0/2537290693 >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.040+0000 7f5b96a00640 1 --2- 172.21.15.28:0/2537290693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009870 tx=0x7f5b8402a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=228ebcc007d9763f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b96000640 1 -- 172.21.15.28:0/2537290693 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84032440 con 0x7f5b980cabc0 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b96000640 1 -- 172.21.15.28:0/2537290693 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b840325a0 con 0x7f5b980cabc0 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b96000640 1 -- 172.21.15.28:0/2537290693 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84032930 con 0x7f5b980cabc0 2024-09-13T17:20:00.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 -- 172.21.15.28:0/2537290693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 msgr2=0x7f5b9815e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2537290693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009870 tx=0x7f5b8402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 -- 172.21.15.28:0/2537290693 shutdown_connections 2024-09-13T17:20:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2537290693 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b980cabc0 0x7f5b9815e870 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2537290693 >> v2:172.21.15.189:3300/0 conn(0x7f5b9815a540 0x7f5b9806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 -- 172.21.15.28:0/2537290693 >> 172.21.15.28:0/2537290693 conn(0x7f5b98069e80 msgr2=0x7f5b9806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 -- 172.21.15.28:0/2537290693 shutdown_connections 2024-09-13T17:20:00.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.041+0000 7f5b9e800640 1 -- 172.21.15.28:0/2537290693 wait complete. 2024-09-13T17:20:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.043+0000 7f5b9e800640 1 Processor -- start 2024-09-13T17:20:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.043+0000 7f5b9e800640 1 -- start start 2024-09-13T17:20:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b980cabc0 0x7f5b980b0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b9815a540 0x7f5b981408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b980b08f0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b980affc0 con 0x7f5b980cabc0 2024-09-13T17:20:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b9e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b980b02a0 con 0x7f5b9815a540 2024-09-13T17:20:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37038/0 (socket says 172.21.15.28:37038) 2024-09-13T17:20:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.044+0000 7f5b97e00640 1 -- 172.21.15.28:0/2052315535 learned_addr learned my addr 172.21.15.28:0/2052315535 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b96a00640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b9815a540 0x7f5b981408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97400640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b980cabc0 0x7f5b980b0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 -- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b9815a540 msgr2=0x7f5b981408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b9815a540 0x7f5b981408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 -- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b980cabc0 msgr2=0x7f5b980b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b980cabc0 0x7f5b980b0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 -- 172.21.15.28:0/2052315535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b88009400 con 0x7f5b9814f1f0 2024-09-13T17:20:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b97e00640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000c710 tx=0x7f5b9000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:00.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b90003da0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.045+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b9000ce20 con 0x7f5b9814f1f0 2024-09-13T17:20:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.046+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b840094d0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.046+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b90005290 con 0x7f5b9814f1f0 2024-09-13T17:20:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.046+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b980afbe0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.047+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b980b0a80 con 0x7f5b9814f1f0 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.048+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b900053f0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.048+0000 7f5b8f400640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 0x7f5b7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.049+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b90088ba0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.049+0000 7f5b97400640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 0x7f5b7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.050+0000 7f5b97400640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 0x7f5b7806d0c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009cd0 tx=0x7f5b88009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:00.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.053+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5b90053da0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.298+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5b98067720 con 0x7f5b9814f1f0 2024-09-13T17:20:00.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.298+0000 7f5b8f400640 1 -- 172.21.15.28:0/2052315535 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1191 (secure 0 0 0) 0x7f5b900573c0 con 0x7f5b9814f1f0 2024-09-13T17:20:00.300 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:00.300 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":734,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":167,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076396270,"bytes_used":2724814848,"bytes_avail":770369298432,"bytes_total":773094113280,"write_bytes_sec":60444300,"read_op_per_sec":0,"write_op_per_sec":58},"fsmap":{"epoch":138,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":10041}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":33,"modified":"2024-09-13T17:19:23.055454+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 msgr2=0x7f5b7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 0x7f5b7806d0c0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009cd0 tx=0x7f5b88009190 comp rx=0 tx=0).stop 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 msgr2=0x7f5b980adf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f5b9000c710 tx=0x7f5b9000cbd0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 shutdown_connections 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b7806ac00 0x7f5b7806d0c0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b9814f1f0 0x7f5b980adf20 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b9815a540 0x7f5b981408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 --2- 172.21.15.28:0/2052315535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b980cabc0 0x7f5b980b0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 >> 172.21.15.28:0/2052315535 conn(0x7f5b98069e80 msgr2=0x7f5b980acf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 shutdown_connections 2024-09-13T17:20:00.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:00.301+0000 7f5b9e800640 1 -- 172.21.15.28:0/2052315535 wait complete. 2024-09-13T17:20:00.318 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:20:01.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.486+0000 7fee67e00640 1 Processor -- start 2024-09-13T17:20:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.486+0000 7fee67e00640 1 -- start start 2024-09-13T17:20:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 0x7fee68160f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee680c4380 0x7fee6806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6806b820 con 0x7fee6806b3d0 2024-09-13T17:20:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee680cf230 con 0x7fee680c4380 2024-09-13T17:20:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee680c4af0 con 0x7fee680cb760 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee6806b3d0 0x7fee68072e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55662/0 (socket says 172.21.15.28:55662) 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee67400640 1 -- 172.21.15.28:0/2361433480 learned_addr learned my addr 172.21.15.28:0/2361433480 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.487+0000 7fee5e000640 1 --2- 172.21.15.28:0/2361433480 >> v2:172.21.15.28:3300/0 conn(0x7fee680c4380 0x7fee6806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee66a00640 1 --2- 172.21.15.28:0/2361433480 >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 0x7fee68160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee5e000640 1 -- 172.21.15.28:0/2361433480 >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 msgr2=0x7fee68160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee5e000640 1 --2- 172.21.15.28:0/2361433480 >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 0x7fee68160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee5e000640 1 -- 172.21.15.28:0/2361433480 >> v1:172.21.15.28:6789/0 conn(0x7fee6806b3d0 legacy=0x7fee68072e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee5e000640 1 -- 172.21.15.28:0/2361433480 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee680cd460 con 0x7fee680c4380 2024-09-13T17:20:01.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee66a00640 1 --2- 172.21.15.28:0/2361433480 >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 0x7fee68160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.488+0000 7fee66000640 1 -- 172.21.15.28:0/2361433480 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614812147 0 0) 0x7fee60001490 con 0x7fee6806b3d0 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.489+0000 7fee5e000640 1 --2- 172.21.15.28:0/2361433480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6806ae90 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fee680c9ac0 tx=0x7fee5002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2910dd8d804020d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.489+0000 7fee66000640 1 -- 172.21.15.28:0/2361433480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5002ae60 con 0x7fee680c4380 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.489+0000 7fee66000640 1 -- 172.21.15.28:0/2361433480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50032440 con 0x7fee680c4380 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.489+0000 7fee66000640 1 -- 172.21.15.28:0/2361433480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee500327d0 con 0x7fee680c4380 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 -- 172.21.15.28:0/2361433480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 msgr2=0x7fee6806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 --2- 172.21.15.28:0/2361433480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6806ae90 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7fee680c9ac0 tx=0x7fee5002a110 comp rx=0 tx=0).stop 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 -- 172.21.15.28:0/2361433480 shutdown_connections 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 --2- 172.21.15.28:0/2361433480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6806ae90 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 --2- 172.21.15.28:0/2361433480 >> v2:172.21.15.189:3300/0 conn(0x7fee680cb760 0x7fee68160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 -- 172.21.15.28:0/2361433480 >> 172.21.15.28:0/2361433480 conn(0x7fee680703a0 msgr2=0x7fee680707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 -- 172.21.15.28:0/2361433480 shutdown_connections 2024-09-13T17:20:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.490+0000 7fee67e00640 1 -- 172.21.15.28:0/2361433480 wait complete. 2024-09-13T17:20:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.492+0000 7fee67e00640 1 Processor -- start 2024-09-13T17:20:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.492+0000 7fee67e00640 1 -- start start 2024-09-13T17:20:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6806b3d0 0x7fee6806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee680cb760 0x7fee6806c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680b2a00 con 0x7fee680c4380 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680b2410 con 0x7fee6806b3d0 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6813f0d0 con 0x7fee680cb760 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37050/0 (socket says 172.21.15.28:37050) 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 -- 172.21.15.28:0/1973675014 learned_addr learned my addr 172.21.15.28:0/1973675014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee66a00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6806b3d0 0x7fee6806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee67400640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee680cb760 0x7fee6806c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 -- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee680cb760 msgr2=0x7fee6806c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee680cb760 0x7fee6806c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 -- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6806b3d0 msgr2=0x7fee6806e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6806b3d0 0x7fee6806e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.493+0000 7fee5e000640 1 -- 172.21.15.28:0/1973675014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee50009400 con 0x7fee680c4380 2024-09-13T17:20:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee5e000640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fee50000c00 tx=0x7fee500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee50030990 con 0x7fee680c4380 2024-09-13T17:20:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee50030b10 con 0x7fee680c4380 2024-09-13T17:20:01.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee5003d420 con 0x7fee680c4380 2024-09-13T17:20:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6815f170 con 0x7fee680c4380 2024-09-13T17:20:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.494+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680b2620 con 0x7fee680c4380 2024-09-13T17:20:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.495+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee680b2b40 con 0x7fee680c4380 2024-09-13T17:20:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.496+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee50039040 con 0x7fee680c4380 2024-09-13T17:20:01.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.497+0000 7fee5fe00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.497+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee500a9ea0 con 0x7fee680c4380 2024-09-13T17:20:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.497+0000 7fee66a00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.498+0000 7fee66a00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 0x7fee4406d0f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fee58006fd0 tx=0x7fee58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.501+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee680b2b40 con 0x7fee680c4380 2024-09-13T17:20:01.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.712+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee680b2b40 con 0x7fee680c4380 2024-09-13T17:20:01.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.712+0000 7fee5fe00640 1 -- 172.21.15.28:0/1973675014 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fee680b2b40 con 0x7fee680c4380 2024-09-13T17:20:01.714 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:01.715 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:01.715 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 msgr2=0x7fee4406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 0x7fee4406d0f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fee58006fd0 tx=0x7fee58008040 comp rx=0 tx=0).stop 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 msgr2=0x7fee6813eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7fee50000c00 tx=0x7fee500046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 shutdown_connections 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee4406ac30 0x7fee4406d0f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee680cb760 0x7fee6806c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee680c4380 0x7fee6813eb90 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 --2- 172.21.15.28:0/1973675014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee6806b3d0 0x7fee6806e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 >> 172.21.15.28:0/1973675014 conn(0x7fee680703a0 msgr2=0x7fee680ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 shutdown_connections 2024-09-13T17:20:01.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.715+0000 7fee67e00640 1 -- 172.21.15.28:0/1973675014 wait complete. 2024-09-13T17:20:01.732 DEBUG:tasks.cephfs.test_scrub_checks:current active=['c'] 2024-09-13T17:20:01.733 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:20:01.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.886+0000 7f9902000640 1 Processor -- start 2024-09-13T17:20:01.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.886+0000 7f9902000640 1 -- start start 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9902000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 0x7f98fc0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9902000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc0c3850 con 0x7f98fc159820 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9902000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98fc15a930 con 0x7f98fc159440 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9902000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc159d20 con 0x7f98fc0b0b50 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9901600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f98fc159820 0x7f98fc15adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55670/0 (socket says 172.21.15.28:55670) 2024-09-13T17:20:01.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f9901600640 1 -- 172.21.15.28:0/547898985 learned_addr learned my addr 172.21.15.28:0/547898985 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.887+0000 7f98fbe00640 1 --2- 172.21.15.28:0/547898985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 0x7f98fc0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.888+0000 7f98fb400640 1 -- 172.21.15.28:0/547898985 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324165474 0 0) 0x7f98fc0c3850 con 0x7f98fc159820 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.888+0000 7f98fb400640 1 -- 172.21.15.28:0/547898985 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e4005730 con 0x7f98fc159820 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.888+0000 7f98fbe00640 1 -- 172.21.15.28:0/547898985 >> v1:172.21.15.189:6789/0 conn(0x7f98fc159440 legacy=0x7f98fc0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.888+0000 7f98fbe00640 1 -- 172.21.15.28:0/547898985 >> v1:172.21.15.28:6789/0 conn(0x7f98fc159820 legacy=0x7f98fc15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.888+0000 7f98fbe00640 1 -- 172.21.15.28:0/547898985 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc15a000 con 0x7f98fc0b0b50 2024-09-13T17:20:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.889+0000 7f98fbe00640 1 --2- 172.21.15.28:0/547898985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 0x7f98fc0b22e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f98e80097d0 tx=0x7f98e802a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f620d6c4f368b28c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.889+0000 7f98fb400640 1 -- 172.21.15.28:0/547898985 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8032440 con 0x7f98fc0b0b50 2024-09-13T17:20:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f98fb400640 1 -- 172.21.15.28:0/547898985 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e80325a0 con 0x7f98fc0b0b50 2024-09-13T17:20:01.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f98fb400640 1 -- 172.21.15.28:0/547898985 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98e8032910 con 0x7f98fc0b0b50 2024-09-13T17:20:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 -- 172.21.15.28:0/547898985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 msgr2=0x7f98fc0b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 --2- 172.21.15.28:0/547898985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 0x7f98fc0b22e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f98e80097d0 tx=0x7f98e802a290 comp rx=0 tx=0).stop 2024-09-13T17:20:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 -- 172.21.15.28:0/547898985 shutdown_connections 2024-09-13T17:20:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 --2- 172.21.15.28:0/547898985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc0b0b50 0x7f98fc0b22e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 -- 172.21.15.28:0/547898985 >> 172.21.15.28:0/547898985 conn(0x7f98fc072e40 msgr2=0x7f98fc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 -- 172.21.15.28:0/547898985 shutdown_connections 2024-09-13T17:20:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.890+0000 7f9902000640 1 -- 172.21.15.28:0/547898985 wait complete. 2024-09-13T17:20:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.892+0000 7f9902000640 1 Processor -- start 2024-09-13T17:20:01.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.892+0000 7f9902000640 1 -- start start 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc159440 0x7f98fc0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98fc159820 0x7f98fc0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc14ab10 con 0x7f98fc0b0b50 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc14a830 con 0x7f98fc159820 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9902000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0ada40 con 0x7f98fc159440 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37054/0 (socket says 172.21.15.28:37054) 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 -- 172.21.15.28:0/3393123023 learned_addr learned my addr 172.21.15.28:0/3393123023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f98fbe00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc159440 0x7f98fc0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9901600640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98fc159820 0x7f98fc0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 -- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc159440 msgr2=0x7f98fc0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc159440 0x7f98fc0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 -- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98fc159820 msgr2=0x7f98fc0c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98fc159820 0x7f98fc0c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.893+0000 7f9900c00640 1 -- 172.21.15.28:0/3393123023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8009400 con 0x7f98fc0b0b50 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f9900c00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c450 tx=0x7f98f000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f00052d0 con 0x7f98fc0b0b50 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f0005430 con 0x7f98fc0b0b50 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0012870 con 0x7f98fc0b0b50 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0add20 con 0x7f98fc0b0b50 2024-09-13T17:20:01.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.894+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc149b30 con 0x7f98fc0b0b50 2024-09-13T17:20:01.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.895+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc0ad8a0 con 0x7f98fc0b0b50 2024-09-13T17:20:01.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.896+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f98f00129d0 con 0x7f98fc0b0b50 2024-09-13T17:20:01.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.897+0000 7f98f8c00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 0x7f98dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:01.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.897+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98f0088f60 con 0x7f98fc0b0b50 2024-09-13T17:20:01.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.897+0000 7f98fbe00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 0x7f98dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:01.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.898+0000 7f98fbe00640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 0x7f98dc06d0c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f98e8034000 tx=0x7f98e802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:01.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:01.901+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f98fc0ad8a0 con 0x7f98fc0b0b50 2024-09-13T17:20:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.147+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f98fc0ae240 con 0x7f98fc0b0b50 2024-09-13T17:20:02.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.148+0000 7f98f8c00640 1 -- 172.21.15.28:0/3393123023 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1601 (secure 0 0 0) 0x7f98fc0ae240 con 0x7f98fc0b0b50 2024-09-13T17:20:02.150 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:02.150 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":167,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076396270,"bytes_used":2724814848,"bytes_avail":770369298432,"bytes_total":773094113280,"write_bytes_sec":35577984,"read_op_per_sec":0,"write_op_per_sec":34},"fsmap":{"epoch":138,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"c","status":"up:active","gid":10041}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":34,"modified":"2024-09-13T17:20:01.064600+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{"scrub status":"paused"}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 msgr2=0x7f98dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 0x7f98dc06d0c0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f98e8034000 tx=0x7f98e802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 msgr2=0x7f98fc147ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f98f000c450 tx=0x7f98f000c910 comp rx=0 tx=0).stop 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 shutdown_connections 2024-09-13T17:20:02.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f98dc06ac00 0x7f98dc06d0c0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98fc159820 0x7f98fc0c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f98fc159440 0x7f98fc0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 --2- 172.21.15.28:0/3393123023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98fc0b0b50 0x7f98fc147ae0 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 >> 172.21.15.28:0/3393123023 conn(0x7f98fc072e40 msgr2=0x7f98fc1578a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 shutdown_connections 2024-09-13T17:20:02.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.150+0000 7f9902000640 1 -- 172.21.15.28:0/3393123023 wait complete. 2024-09-13T17:20:02.167 INFO:tasks.cephfs.filesystem.ceph_manager:{'c': 'paused'} 2024-09-13T17:20:02.168 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:20:02.168 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:20:02.186 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:20:02.186 INFO:tasks.cephfs.test_scrub_checks:Waiting for promotion of one of the original standbys {'a', 'd', 'b'} 2024-09-13T17:20:02.187 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_beacon_grace 2024-09-13T17:20:02.391 INFO:teuthology.orchestra.run.smithi028.stdout:{"mds_beacon_grace":"15.000000"} 2024-09-13T17:20:02.392 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_beacon_grace": "15.000000" } 2024-09-13T17:20:02.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:02.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.545+0000 7ff89ce00640 1 Processor -- start 2024-09-13T17:20:02.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.545+0000 7ff89ce00640 1 -- start start 2024-09-13T17:20:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.546+0000 7ff89ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff8980acf60 0x7ff898160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:02.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.546+0000 7ff89ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff898146550 con 0x7ff898146170 2024-09-13T17:20:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.546+0000 7ff89ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff89815d280 con 0x7ff898160310 2024-09-13T17:20:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.546+0000 7ff89ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89815cd20 con 0x7ff8980acf60 2024-09-13T17:20:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff895e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff898146170 0x7ff89815d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55682/0 (socket says 172.21.15.28:55682) 2024-09-13T17:20:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff895e00640 1 -- 172.21.15.28:0/460157427 learned_addr learned my addr 172.21.15.28:0/460157427 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:02.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff896800640 1 --2- 172.21.15.28:0/460157427 >> v2:172.21.15.28:3300/0 conn(0x7ff8980acf60 0x7ff898160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff896800640 1 -- 172.21.15.28:0/460157427 >> v1:172.21.15.189:6789/0 conn(0x7ff898160310 legacy=0x7ff8980ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff896800640 1 -- 172.21.15.28:0/460157427 >> v1:172.21.15.28:6789/0 conn(0x7ff898146170 legacy=0x7ff89815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff896800640 1 -- 172.21.15.28:0/460157427 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff898160d60 con 0x7ff8980acf60 2024-09-13T17:20:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.547+0000 7ff894a00640 1 -- 172.21.15.28:0/460157427 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940164003 0 0) 0x7ff898146550 con 0x7ff898146170 2024-09-13T17:20:02.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.548+0000 7ff896800640 1 --2- 172.21.15.28:0/460157427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff898160730 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7ff890009d90 tx=0x7ff89002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0c6433dcdd4b22 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.548+0000 7ff894a00640 1 -- 172.21.15.28:0/460157427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890007740 con 0x7ff8980acf60 2024-09-13T17:20:02.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.548+0000 7ff894a00640 1 -- 172.21.15.28:0/460157427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89002d050 con 0x7ff8980acf60 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.548+0000 7ff894a00640 1 -- 172.21.15.28:0/460157427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890007ac0 con 0x7ff8980acf60 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 -- 172.21.15.28:0/460157427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 msgr2=0x7ff898160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 --2- 172.21.15.28:0/460157427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff898160730 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7ff890009d90 tx=0x7ff89002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 -- 172.21.15.28:0/460157427 shutdown_connections 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 --2- 172.21.15.28:0/460157427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff898160730 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 -- 172.21.15.28:0/460157427 >> 172.21.15.28:0/460157427 conn(0x7ff898069e80 msgr2=0x7ff89806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 -- 172.21.15.28:0/460157427 shutdown_connections 2024-09-13T17:20:02.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.549+0000 7ff89ce00640 1 -- 172.21.15.28:0/460157427 wait complete. 2024-09-13T17:20:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.551+0000 7ff89ce00640 1 Processor -- start 2024-09-13T17:20:02.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.551+0000 7ff89ce00640 1 -- start start 2024-09-13T17:20:02.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:02.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 0x7ff8981e7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:02.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898160310 0x7ff8981e7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:02.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff898135040 con 0x7ff8980acf60 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff898135320 con 0x7ff898160310 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff89ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff898135600 con 0x7ff898146170 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff895e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff895e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37066/0 (socket says 172.21.15.28:37066) 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff895e00640 1 -- 172.21.15.28:0/714509183 learned_addr learned my addr 172.21.15.28:0/714509183 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff896800640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898160310 0x7ff8981e7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.552+0000 7ff895400640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 0x7ff8981e7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 -- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 msgr2=0x7ff8981e7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 0x7ff8981e7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 -- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898160310 msgr2=0x7ff8981e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898160310 0x7ff8981e7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 -- 172.21.15.28:0/714509183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890009a70 con 0x7ff8980acf60 2024-09-13T17:20:02.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895400640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 0x7ff8981e7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff895e00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c4a0 tx=0x7ff88800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8880074e0 con 0x7ff8980acf60 2024-09-13T17:20:02.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff888007640 con 0x7ff8980acf60 2024-09-13T17:20:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8981358e0 con 0x7ff8980acf60 2024-09-13T17:20:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888004a00 con 0x7ff8980acf60 2024-09-13T17:20:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.553+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff898134cd0 con 0x7ff8980acf60 2024-09-13T17:20:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.554+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898160730 con 0x7ff8980acf60 2024-09-13T17:20:02.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.555+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff888004b60 con 0x7ff8980acf60 2024-09-13T17:20:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.556+0000 7ff88e000640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 0x7ff87806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.556+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8880883f0 con 0x7ff8980acf60 2024-09-13T17:20:02.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.557+0000 7ff895400640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 0x7ff87806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:02.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.557+0000 7ff895400640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 0x7ff87806d0c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff884007fd0 tx=0x7ff884009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:02.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.561+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff888053670 con 0x7ff8980acf60 2024-09-13T17:20:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.774+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff898135e00 con 0x7ff8980acf60 2024-09-13T17:20:02.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.775+0000 7ff88e000640 1 -- 172.21.15.28:0/714509183 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7ff888056c90 con 0x7ff8980acf60 2024-09-13T17:20:02.777 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:02.777 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:02.778 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 msgr2=0x7ff87806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 0x7ff87806d0c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7ff884007fd0 tx=0x7ff884009040 comp rx=0 tx=0).stop 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 msgr2=0x7ff8981e8350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c4a0 tx=0x7ff88800c960 comp rx=0 tx=0).stop 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 shutdown_connections 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac00 0x7ff87806d0c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff898160310 0x7ff8981e7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff898146170 0x7ff8981e7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 --2- 172.21.15.28:0/714509183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8980acf60 0x7ff8981e8350 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 >> 172.21.15.28:0/714509183 conn(0x7ff898069e80 msgr2=0x7ff8980af770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 shutdown_connections 2024-09-13T17:20:02.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:02.777+0000 7ff89ce00640 1 -- 172.21.15.28:0/714509183 wait complete. 2024-09-13T17:20:02.795 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-13T17:20:07.796 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.961+0000 7ff8a0400640 1 Processor -- start 2024-09-13T17:20:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.961+0000 7ff8a0400640 1 -- start start 2024-09-13T17:20:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff89815a540 0x7ff89806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff8980cabc0 0x7ff89815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff89815edb0 0x7ff898070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8980adb00 con 0x7ff89815edb0 2024-09-13T17:20:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff89815e100 con 0x7ff89815a540 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.962+0000 7ff8a0400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff89815e3e0 con 0x7ff8980cabc0 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff89815edb0 0x7ff898070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff89815edb0 0x7ff898070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45530/0 (socket says 172.21.15.28:45530) 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 -- 172.21.15.28:0/396467061 learned_addr learned my addr 172.21.15.28:0/396467061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89da00640 1 --2- 172.21.15.28:0/396467061 >> v2:172.21.15.189:3300/0 conn(0x7ff89815a540 0x7ff89806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89d000640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff8980cabc0 0x7ff89815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 -- 172.21.15.28:0/396467061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff8980cabc0 msgr2=0x7ff89815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff8980cabc0 0x7ff89815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 -- 172.21.15.28:0/396467061 >> v2:172.21.15.189:3300/0 conn(0x7ff89815a540 msgr2=0x7ff89806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 --2- 172.21.15.28:0/396467061 >> v2:172.21.15.189:3300/0 conn(0x7ff89815a540 0x7ff89806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.963+0000 7ff89e400640 1 -- 172.21.15.28:0/396467061 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89806ce80 con 0x7ff89815edb0 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.964+0000 7ff89e400640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89815edb0 0x7ff898070cc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7ff8980b3a00 tx=0x7ff89002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c23979f18e13398 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.964+0000 7ff897e00640 1 -- 172.21.15.28:0/396467061 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890031600 con 0x7ff89815edb0 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.964+0000 7ff897e00640 1 -- 172.21.15.28:0/396467061 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89002d050 con 0x7ff89815edb0 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.964+0000 7ff897e00640 1 -- 172.21.15.28:0/396467061 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890031980 con 0x7ff89815edb0 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 -- 172.21.15.28:0/396467061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89815edb0 msgr2=0x7ff898070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89815edb0 0x7ff898070cc0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7ff8980b3a00 tx=0x7ff89002a610 comp rx=0 tx=0).stop 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 -- 172.21.15.28:0/396467061 shutdown_connections 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff89815edb0 0x7ff898070cc0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 --2- 172.21.15.28:0/396467061 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff8980cabc0 0x7ff89815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 --2- 172.21.15.28:0/396467061 >> v2:172.21.15.189:3300/0 conn(0x7ff89815a540 0x7ff89806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 -- 172.21.15.28:0/396467061 >> 172.21.15.28:0/396467061 conn(0x7ff898069e80 msgr2=0x7ff89806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 -- 172.21.15.28:0/396467061 shutdown_connections 2024-09-13T17:20:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.965+0000 7ff8a0400640 1 -- 172.21.15.28:0/396467061 wait complete. 2024-09-13T17:20:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.967+0000 7ff8a0400640 1 Processor -- start 2024-09-13T17:20:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.967+0000 7ff8a0400640 1 -- start start 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8980cabc0 0x7ff8981f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 0x7ff89815cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89814c2a0 con 0x7ff8981f75f0 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8981f0d90 con 0x7ff8980cabc0 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff8a0400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8981509c0 con 0x7ff89815a540 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff89e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff89e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45542/0 (socket says 172.21.15.28:45542) 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff89e400640 1 -- 172.21.15.28:0/3984467471 learned_addr learned my addr 172.21.15.28:0/3984467471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff89d000640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 0x7ff89815cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.968+0000 7ff89da00640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8980cabc0 0x7ff8981f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 -- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 msgr2=0x7ff89815cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 0x7ff89815cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 -- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8980cabc0 msgr2=0x7ff8981f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8980cabc0 0x7ff8981f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 -- 172.21.15.28:0/3984467471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890009b30 con 0x7ff8981f75f0 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89d000640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 0x7ff89815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff89e400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff890009f40 tx=0x7ff890004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89003b810 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89003b990 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89003bd40 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89815d900 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.969+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff898150370 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.970+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff898150b50 con 0x7ff8981f75f0 2024-09-13T17:20:07.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.971+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff89003a3a0 con 0x7ff8981f75f0 2024-09-13T17:20:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.972+0000 7ff895600640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 0x7ff87806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.972+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff8900a9c30 con 0x7ff8981f75f0 2024-09-13T17:20:07.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.973+0000 7ff89da00640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 0x7ff87806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:07.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.974+0000 7ff89da00640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 0x7ff87806d0f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff88c006fd0 tx=0x7ff88c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:07.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:07.977+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff890074e20 con 0x7ff8981f75f0 2024-09-13T17:20:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.193+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff89815e870 con 0x7ff8981f75f0 2024-09-13T17:20:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.194+0000 7ff895600640 1 -- 172.21.15.28:0/3984467471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7ff890009510 con 0x7ff8981f75f0 2024-09-13T17:20:08.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:08.196 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:08.197 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.196+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 msgr2=0x7ff87806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.196+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 0x7ff87806d0f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff88c006fd0 tx=0x7ff88c008040 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.196+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 msgr2=0x7ff898155360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.196+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7ff890009f40 tx=0x7ff890004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 shutdown_connections 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff87806ac30 0x7ff87806d0f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8981f75f0 0x7ff898155360 unknown :-1 s=CLOSED pgs=2031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff89815a540 0x7ff89815cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 --2- 172.21.15.28:0/3984467471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8980cabc0 0x7ff8981f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 >> 172.21.15.28:0/3984467471 conn(0x7ff898069e80 msgr2=0x7ff8980ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:08.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 shutdown_connections 2024-09-13T17:20:08.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:08.197+0000 7ff8a0400640 1 -- 172.21.15.28:0/3984467471 wait complete. 2024-09-13T17:20:08.215 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-13T17:20:11.160 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:20:11.165 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:20:13.216 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.381+0000 7fb4f4a00640 1 Processor -- start 2024-09-13T17:20:13.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.381+0000 7fb4f4a00640 1 -- start start 2024-09-13T17:20:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.382+0000 7fb4f4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f015d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.382+0000 7fb4f4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb4f006fb50 0x7fb4f006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.382+0000 7fb4f4a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4f00b3970 con 0x7fb4f00b3160 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.382+0000 7fb4f4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f0159230 con 0x7fb4f006fb50 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.382+0000 7fb4f4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0159430 con 0x7fb4f00cdfa0 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ed400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb4f00b3160 0x7fb4f006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:57710/0 (socket says 172.21.15.28:57710) 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ed400640 1 -- 172.21.15.28:0/1976794500 learned_addr learned my addr 172.21.15.28:0/1976794500 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ee800640 1 --2- 172.21.15.28:0/1976794500 >> v2:172.21.15.28:3300/0 conn(0x7fb4f006fb50 0x7fb4f006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ede00640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4eca00640 1 -- 172.21.15.28:0/1976794500 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805315853 0 0) 0x7fb4dc001110 con 0x7fb4f00b3160 2024-09-13T17:20:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4eca00640 1 -- 172.21.15.28:0/1976794500 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4d4003520 con 0x7fb4f00b3160 2024-09-13T17:20:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ee800640 1 -- 172.21.15.28:0/1976794500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 msgr2=0x7fb4f015d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ee800640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f015d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ee800640 1 -- 172.21.15.28:0/1976794500 >> v1:172.21.15.28:6789/0 conn(0x7fb4f00b3160 legacy=0x7fb4f006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.383+0000 7fb4ee800640 1 -- 172.21.15.28:0/1976794500 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f0159710 con 0x7fb4f006fb50 2024-09-13T17:20:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.384+0000 7fb4ee800640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f006fb50 0x7fb4f006b1c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c8e3d8e6aa01269 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.384+0000 7fb4eca00640 1 -- 172.21.15.28:0/1976794500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8007740 con 0x7fb4f006fb50 2024-09-13T17:20:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.384+0000 7fb4eca00640 1 -- 172.21.15.28:0/1976794500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e802d050 con 0x7fb4f006fb50 2024-09-13T17:20:13.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.384+0000 7fb4eca00640 1 -- 172.21.15.28:0/1976794500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8007aa0 con 0x7fb4f006fb50 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1976794500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f006fb50 msgr2=0x7fb4f006b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f006fb50 0x7fb4f006b1c0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009d90 tx=0x7fb4e802a660 comp rx=0 tx=0).stop 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1976794500 shutdown_connections 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f006fb50 0x7fb4f006b1c0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1976794500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f015d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1976794500 >> 172.21.15.28:0/1976794500 conn(0x7fb4f0069e80 msgr2=0x7fb4f006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1976794500 shutdown_connections 2024-09-13T17:20:13.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.385+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1976794500 wait complete. 2024-09-13T17:20:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.387+0000 7fb4f4a00640 1 Processor -- start 2024-09-13T17:20:13.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.387+0000 7fb4f4a00640 1 -- start start 2024-09-13T17:20:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f006fb50 0x7fb4f00a8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f013ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0152940 con 0x7fb4f00b3160 2024-09-13T17:20:13.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0152c20 con 0x7fb4f006fb50 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4f4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f0152f00 con 0x7fb4f00cdfa0 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45566/0 (socket says 172.21.15.28:45566) 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 -- 172.21.15.28:0/1234070746 learned_addr learned my addr 172.21.15.28:0/1234070746 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ee800640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f013ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ede00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f006fb50 0x7fb4f00a8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 -- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 msgr2=0x7fb4f013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f013ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 -- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f006fb50 msgr2=0x7fb4f00a8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f006fb50 0x7fb4f00a8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.388+0000 7fb4ed400640 1 -- 172.21.15.28:0/1234070746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e8009a70 con 0x7fb4f00b3160 2024-09-13T17:20:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4ed400640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c730 tx=0x7fb4dc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc005ab0 con 0x7fb4f00b3160 2024-09-13T17:20:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4dc005c10 con 0x7fb4f00b3160 2024-09-13T17:20:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4dc001d30 con 0x7fb4f00b3160 2024-09-13T17:20:13.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f01531e0 con 0x7fb4f00b3160 2024-09-13T17:20:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.389+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f00a9370 con 0x7fb4f00b3160 2024-09-13T17:20:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.390+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f00a9560 con 0x7fb4f00b3160 2024-09-13T17:20:13.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.391+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb4dc007070 con 0x7fb4f00b3160 2024-09-13T17:20:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.392+0000 7fb4e6000640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 0x7fb4d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.392+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb4dc088740 con 0x7fb4f00b3160 2024-09-13T17:20:13.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.392+0000 7fb4ede00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 0x7fb4d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:13.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.393+0000 7fb4ede00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 0x7fb4d006d0f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0006fd0 tx=0x7fb4e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:13.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.397+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb4dc053910 con 0x7fb4f00b3160 2024-09-13T17:20:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.612+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4f00b3970 con 0x7fb4f00b3160 2024-09-13T17:20:13.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.612+0000 7fb4e6000640 1 -- 172.21.15.28:0/1234070746 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fb4f00b3970 con 0x7fb4f00b3160 2024-09-13T17:20:13.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:13.614 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":138,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10038,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:19:31.732739+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10041},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10041":{"gid":10041,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3709710897","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3709710897},{"type":"v1","addr":"172.21.15.28:6835","nonce":3709710897}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:13.614 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 138 2024-09-13T17:20:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 msgr2=0x7fb4d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 0x7fb4d006d0f0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fb4e0006fd0 tx=0x7fb4e0008040 comp rx=0 tx=0).stop 2024-09-13T17:20:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 msgr2=0x7fb4f01500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:13.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fb4dc00c730 tx=0x7fb4dc00cbf0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 shutdown_connections 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb4d006ac30 0x7fb4d006d0f0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb4f00cdfa0 0x7fb4f013ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb4f00b3160 0x7fb4f01500d0 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 --2- 172.21.15.28:0/1234070746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4f006fb50 0x7fb4f00a8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 >> 172.21.15.28:0/1234070746 conn(0x7fb4f0069e80 msgr2=0x7fb4f0158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 shutdown_connections 2024-09-13T17:20:13.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:13.615+0000 7fb4f4a00640 1 -- 172.21.15.28:0/1234070746 wait complete. 2024-09-13T17:20:13.631 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=45.0 retry_count=0)... 2024-09-13T17:20:18.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:18.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.798+0000 7ff4de200640 1 Processor -- start 2024-09-13T17:20:18.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.798+0000 7ff4de200640 1 -- start start 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.799+0000 7ff4de200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff4d81592a0 0x7ff4d815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4de200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d8159d90 con 0x7ff4d806c2d0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4de200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d80b2040 con 0x7ff4d806f970 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4de200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d815b0b0 con 0x7ff4d81592a0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4d7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4d806c2d0 0x7ff4d80b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50754/0 (socket says 172.21.15.28:50754) 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4d7e00640 1 -- 172.21.15.28:0/3686310391 learned_addr learned my addr 172.21.15.28:0/3686310391 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4d7400640 1 --2- 172.21.15.28:0/3686310391 >> v2:172.21.15.189:3300/0 conn(0x7ff4d81592a0 0x7ff4d815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670370274 0 0) 0x7ff4d8159d90 con 0x7ff4d806c2d0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.800+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4bc003520 con 0x7ff4d806c2d0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.801+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2686337059 0 0) 0x7ff4bc003520 con 0x7ff4d806c2d0 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.801+0000 7ff4d7400640 1 -- 172.21.15.28:0/3686310391 >> v1:172.21.15.189:6789/0 conn(0x7ff4d806f970 legacy=0x7ff4d8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:18.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.801+0000 7ff4d7400640 1 -- 172.21.15.28:0/3686310391 >> v1:172.21.15.28:6789/0 conn(0x7ff4d806c2d0 legacy=0x7ff4d80b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:18.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.801+0000 7ff4d7400640 1 -- 172.21.15.28:0/3686310391 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d8159ff0 con 0x7ff4d81592a0 2024-09-13T17:20:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.802+0000 7ff4d7400640 1 --2- 172.21.15.28:0/3686310391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81592a0 0x7ff4d815ab70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009d90 tx=0x7ff4c802a5b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ecc19a1181dd0e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.802+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0001690 con 0x7ff4d81592a0 2024-09-13T17:20:18.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.802+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c802d050 con 0x7ff4d81592a0 2024-09-13T17:20:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.802+0000 7ff4d6a00640 1 -- 172.21.15.28:0/3686310391 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8007760 con 0x7ff4d81592a0 2024-09-13T17:20:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 -- 172.21.15.28:0/3686310391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81592a0 msgr2=0x7ff4d815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 --2- 172.21.15.28:0/3686310391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81592a0 0x7ff4d815ab70 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009d90 tx=0x7ff4c802a5b0 comp rx=0 tx=0).stop 2024-09-13T17:20:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 -- 172.21.15.28:0/3686310391 shutdown_connections 2024-09-13T17:20:18.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 --2- 172.21.15.28:0/3686310391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d81592a0 0x7ff4d815ab70 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 -- 172.21.15.28:0/3686310391 >> 172.21.15.28:0/3686310391 conn(0x7ff4d8069e80 msgr2=0x7ff4d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 -- 172.21.15.28:0/3686310391 shutdown_connections 2024-09-13T17:20:18.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.803+0000 7ff4de200640 1 -- 172.21.15.28:0/3686310391 wait complete. 2024-09-13T17:20:18.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.805+0000 7ff4de200640 1 Processor -- start 2024-09-13T17:20:18.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.805+0000 7ff4de200640 1 -- start start 2024-09-13T17:20:18.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 0x7ff4d80cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4d81592a0 0x7ff4d81575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8146af0 con 0x7ff4d806c2d0 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8144540 con 0x7ff4d806f970 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4de200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8143c10 con 0x7ff4d81592a0 2024-09-13T17:20:18.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41994/0 (socket says 172.21.15.28:41994) 2024-09-13T17:20:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 -- 172.21.15.28:0/791077322 learned_addr learned my addr 172.21.15.28:0/791077322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4cea00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 0x7ff4d80cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7e00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4d81592a0 0x7ff4d81575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:18.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 -- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4d81592a0 msgr2=0x7ff4d81575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4d81592a0 0x7ff4d81575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 -- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 msgr2=0x7ff4d80cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 0x7ff4d80cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4d7400640 1 -- 172.21.15.28:0/791077322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c8009a70 con 0x7ff4d806c2d0 2024-09-13T17:20:18.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.806+0000 7ff4cea00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 0x7ff4d80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4d7400640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8034040 tx=0x7ff4c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c803b930 con 0x7ff4d806c2d0 2024-09-13T17:20:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c803ba90 con 0x7ff4d806c2d0 2024-09-13T17:20:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d8143ef0 con 0x7ff4d806c2d0 2024-09-13T17:20:18.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c803be40 con 0x7ff4d806c2d0 2024-09-13T17:20:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.807+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d8157b30 con 0x7ff4d806c2d0 2024-09-13T17:20:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.808+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d815ab70 con 0x7ff4d806c2d0 2024-09-13T17:20:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.809+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff4c8031400 con 0x7ff4d806c2d0 2024-09-13T17:20:18.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.810+0000 7ff4cfe00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 0x7ff4b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.810+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff4c80aaa20 con 0x7ff4d806c2d0 2024-09-13T17:20:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.810+0000 7ff4cea00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 0x7ff4b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:18.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.811+0000 7ff4cea00640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 0x7ff4b406d0c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff4d814a1a0 tx=0x7ff4c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:18.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:18.814+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff4c80759f0 con 0x7ff4d806c2d0 2024-09-13T17:20:19.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.027+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4d8143d50 con 0x7ff4d806c2d0 2024-09-13T17:20:19.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.028+0000 7ff4cfe00640 1 -- 172.21.15.28:0/791077322 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7ff4d8143d50 con 0x7ff4d806c2d0 2024-09-13T17:20:19.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:19.030 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:19.031 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:19.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 msgr2=0x7ff4b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 0x7ff4b406d0c0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7ff4d814a1a0 tx=0x7ff4c0009040 comp rx=0 tx=0).stop 2024-09-13T17:20:19.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 msgr2=0x7ff4d80cd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8034040 tx=0x7ff4c8004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 shutdown_connections 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4b406ac00 0x7ff4b406d0c0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4d81592a0 0x7ff4d81575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4d806f970 0x7ff4d80cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 --2- 172.21.15.28:0/791077322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4d806c2d0 0x7ff4d80cd420 secure :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8034040 tx=0x7ff4c8004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:19.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.031+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 >> 172.21.15.28:0/791077322 conn(0x7ff4d8069e80 msgr2=0x7ff4d80b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:19.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.032+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 shutdown_connections 2024-09-13T17:20:19.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.032+0000 7ff4de200640 1 -- 172.21.15.28:0/791077322 wait complete. 2024-09-13T17:20:19.050 DEBUG:tasks.ceph_test_case:wait_until_true: success in 15s and 0 retries 2024-09-13T17:20:19.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.215+0000 7fa0b9800640 1 Processor -- start 2024-09-13T17:20:19.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.215+0000 7fa0b9800640 1 -- start start 2024-09-13T17:20:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.216+0000 7fa0b9800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0b40cfae0 0x7fa0b40b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.216+0000 7fa0b9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0b4160580 0x7fa0b41609b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.216+0000 7fa0b9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b40b6200 con 0x7fa0b415c970 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.216+0000 7fa0b9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b40d1690 con 0x7fa0b4160580 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.216+0000 7fa0b9800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b40d09b0 con 0x7fa0b40cfae0 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b2a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0b415c970 0x7fa0b40cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50770/0 (socket says 172.21.15.28:50770) 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b3400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa0b4160580 0x7fa0b41609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b2a00640 1 -- 172.21.15.28:0/940611146 learned_addr learned my addr 172.21.15.28:0/940611146 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b2000640 1 --2- 172.21.15.28:0/940611146 >> v2:172.21.15.189:3300/0 conn(0x7fa0b40cfae0 0x7fa0b40b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b1600640 1 -- 172.21.15.28:0/940611146 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 944785957 0 0) 0x7fa0b40b6200 con 0x7fa0b415c970 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.217+0000 7fa0b1600640 1 -- 172.21.15.28:0/940611146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa098003520 con 0x7fa0b415c970 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.218+0000 7fa0b3400640 1 -- 172.21.15.28:0/940611146 >> v2:172.21.15.189:3300/0 conn(0x7fa0b40cfae0 msgr2=0x7fa0b40b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.218+0000 7fa0b3400640 1 --2- 172.21.15.28:0/940611146 >> v2:172.21.15.189:3300/0 conn(0x7fa0b40cfae0 0x7fa0b40b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.218+0000 7fa0b3400640 1 -- 172.21.15.28:0/940611146 >> v1:172.21.15.28:6789/0 conn(0x7fa0b415c970 legacy=0x7fa0b40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.218+0000 7fa0b3400640 1 -- 172.21.15.28:0/940611146 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b4160ef0 con 0x7fa0b4160580 2024-09-13T17:20:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.219+0000 7fa0b3400640 1 --2- 172.21.15.28:0/940611146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b4160580 0x7fa0b41609b0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40b4e60 tx=0x7fa0ac02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=666028144c4f8309 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.219+0000 7fa0b1600640 1 -- 172.21.15.28:0/940611146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac007740 con 0x7fa0b4160580 2024-09-13T17:20:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.219+0000 7fa0b1600640 1 -- 172.21.15.28:0/940611146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac02d050 con 0x7fa0b4160580 2024-09-13T17:20:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.219+0000 7fa0b1600640 1 -- 172.21.15.28:0/940611146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac007ac0 con 0x7fa0b4160580 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 -- 172.21.15.28:0/940611146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b4160580 msgr2=0x7fa0b41609b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 --2- 172.21.15.28:0/940611146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b4160580 0x7fa0b41609b0 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fa0b40b4e60 tx=0x7fa0ac02a5f0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 -- 172.21.15.28:0/940611146 shutdown_connections 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 --2- 172.21.15.28:0/940611146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b4160580 0x7fa0b41609b0 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 --2- 172.21.15.28:0/940611146 >> v2:172.21.15.189:3300/0 conn(0x7fa0b40cfae0 0x7fa0b40b5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 -- 172.21.15.28:0/940611146 >> 172.21.15.28:0/940611146 conn(0x7fa0b4070ad0 msgr2=0x7fa0b40b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 -- 172.21.15.28:0/940611146 shutdown_connections 2024-09-13T17:20:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.220+0000 7fa0b9800640 1 -- 172.21.15.28:0/940611146 wait complete. 2024-09-13T17:20:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.222+0000 7fa0b9800640 1 Processor -- start 2024-09-13T17:20:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.222+0000 7fa0b9800640 1 -- start start 2024-09-13T17:20:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b415c970 0x7fa0b41f59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b4160580 0x7fa0b4069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b414d290 con 0x7fa0b40cfae0 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41f68d0 con 0x7fa0b415c970 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b9800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b41f6310 con 0x7fa0b4160580 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.223+0000 7fa0b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42006/0 (socket says 172.21.15.28:42006) 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 -- 172.21.15.28:0/596923401 learned_addr learned my addr 172.21.15.28:0/596923401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2000640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b415c970 0x7fa0b41f59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b3400640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b4160580 0x7fa0b4069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 -- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b4160580 msgr2=0x7fa0b4069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b4160580 0x7fa0b4069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 -- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b415c970 msgr2=0x7fa0b41f59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b415c970 0x7fa0b41f59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 -- 172.21.15.28:0/596923401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009a70 con 0x7fa0b40cfae0 2024-09-13T17:20:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.224+0000 7fa0b2a00640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c500 tx=0x7fa0a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.225+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4005ce0 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.225+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a4005e40 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.225+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4007330 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.225+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b41f65f0 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.225+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b41f21e0 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.226+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b415cd50 con 0x7fa0b40cfae0 2024-09-13T17:20:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.228+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0a4007550 con 0x7fa0b40cfae0 2024-09-13T17:20:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.229+0000 7fa0aaa00640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.229+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa0a4088970 con 0x7fa0b40cfae0 2024-09-13T17:20:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.229+0000 7fa0b2000640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.230+0000 7fa0b2000640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4066fb0 tx=0x7fa0a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.233+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0a4053c50 con 0x7fa0b40cfae0 2024-09-13T17:20:19.268 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-13T17:20:19.267+0000 7fc14b200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:20:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.470+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0b40b4af0 con 0x7fa0b40cfae0 2024-09-13T17:20:19.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.471+0000 7fa0aaa00640 1 -- 172.21.15.28:0/596923401 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7fa0a4017020 con 0x7fa0b40cfae0 2024-09-13T17:20:19.472 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:19.472 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:19.472 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 msgr2=0x7fa09406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa0b4066fb0 tx=0x7fa0a0008040 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 msgr2=0x7fa0b41f1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c500 tx=0x7fa0a400c9c0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 shutdown_connections 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa09406ac00 0x7fa09406d0c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0b4160580 0x7fa0b4069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0b415c970 0x7fa0b41f59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 --2- 172.21.15.28:0/596923401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0b40cfae0 0x7fa0b41f1ca0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 >> 172.21.15.28:0/596923401 conn(0x7fa0b4070ad0 msgr2=0x7fa0b415ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 shutdown_connections 2024-09-13T17:20:19.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.473+0000 7fa0b9800640 1 -- 172.21.15.28:0/596923401 wait complete. 2024-09-13T17:20:19.491 DEBUG:tasks.cephfs.test_scrub_checks:current active=['a'] 2024-09-13T17:20:19.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T17:20:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.666+0000 7f26a6400640 1 Processor -- start 2024-09-13T17:20:19.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.666+0000 7f26a6400640 1 -- start start 2024-09-13T17:20:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a6400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26a0159440 0x7f26a00b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a6400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26a0159820 0x7f26a00b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26a00c3850 con 0x7f26a00b22e0 2024-09-13T17:20:19.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a6400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a015a930 con 0x7f26a0159820 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a6400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26a0159d20 con 0x7f26a0159440 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f269fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f26a0159820 0x7f26a00b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a5a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26a00b22e0 0x7f26a00b2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50774/0 (socket says 172.21.15.28:50774) 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a5a00640 1 -- 172.21.15.28:0/837174876 learned_addr learned my addr 172.21.15.28:0/837174876 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.668+0000 7f26a5000640 1 --2- 172.21.15.28:0/837174876 >> v2:172.21.15.189:3300/0 conn(0x7f26a0159440 0x7f26a00b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.669+0000 7f269fe00640 1 -- 172.21.15.28:0/837174876 >> v2:172.21.15.189:3300/0 conn(0x7f26a0159440 msgr2=0x7f26a00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.669+0000 7f269fe00640 1 --2- 172.21.15.28:0/837174876 >> v2:172.21.15.189:3300/0 conn(0x7f26a0159440 0x7f26a00b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.669+0000 7f269fe00640 1 -- 172.21.15.28:0/837174876 >> v1:172.21.15.28:6789/0 conn(0x7f26a00b22e0 legacy=0x7f26a00b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.669+0000 7f269fe00640 1 -- 172.21.15.28:0/837174876 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a015a000 con 0x7f26a0159820 2024-09-13T17:20:19.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.671+0000 7f269fe00640 1 --2- 172.21.15.28:0/837174876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0159820 0x7f26a00b0980 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c00c2cedbea97fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.671+0000 7f269f400640 1 -- 172.21.15.28:0/837174876 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c032440 con 0x7f26a0159820 2024-09-13T17:20:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.671+0000 7f269f400640 1 -- 172.21.15.28:0/837174876 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f268c0325a0 con 0x7f26a0159820 2024-09-13T17:20:19.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.671+0000 7f269f400640 1 -- 172.21.15.28:0/837174876 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f268c032930 con 0x7f26a0159820 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 -- 172.21.15.28:0/837174876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0159820 msgr2=0x7f26a00b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 --2- 172.21.15.28:0/837174876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0159820 0x7f26a00b0980 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f268c0097a0 tx=0x7f268c02a270 comp rx=0 tx=0).stop 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 -- 172.21.15.28:0/837174876 shutdown_connections 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 --2- 172.21.15.28:0/837174876 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a0159820 0x7f26a00b0980 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 --2- 172.21.15.28:0/837174876 >> v2:172.21.15.189:3300/0 conn(0x7f26a0159440 0x7f26a00b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 -- 172.21.15.28:0/837174876 >> 172.21.15.28:0/837174876 conn(0x7f26a0072e40 msgr2=0x7f26a0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:19.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 -- 172.21.15.28:0/837174876 shutdown_connections 2024-09-13T17:20:19.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.672+0000 7f26a6400640 1 -- 172.21.15.28:0/837174876 wait complete. 2024-09-13T17:20:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.674+0000 7f26a6400640 1 Processor -- start 2024-09-13T17:20:19.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.674+0000 7f26a6400640 1 -- start start 2024-09-13T17:20:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0159440 0x7f26a014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 0x7f26a013ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0150100 con 0x7f26a00b22e0 2024-09-13T17:20:19.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0150ad0 con 0x7f26a0159820 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26a014f4f0 con 0x7f26a0159440 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42012/0 (socket says 172.21.15.28:42012) 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5000640 1 -- 172.21.15.28:0/2390405050 learned_addr learned my addr 172.21.15.28:0/2390405050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f269fe00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0159440 0x7f26a014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5a00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 0x7f26a013ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5000640 1 -- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0159440 msgr2=0x7f26a014ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.676+0000 7f26a5000640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0159440 0x7f26a014ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a5000640 1 -- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 msgr2=0x7f26a013ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a5000640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 0x7f26a013ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a5000640 1 -- 172.21.15.28:0/2390405050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f268c009400 con 0x7f26a00b22e0 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a5a00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 0x7f26a013ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:19.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a5000640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f269400e9e0 tx=0x7f269400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2694009670 con 0x7f26a00b22e0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f269400c9d0 con 0x7f26a00b22e0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269400cd80 con 0x7f26a00b22e0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a014f7d0 con 0x7f26a00b22e0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.677+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a014fd90 con 0x7f26a00b22e0 2024-09-13T17:20:19.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.679+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0164040 con 0x7f26a00b22e0 2024-09-13T17:20:19.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.680+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2694013470 con 0x7f26a00b22e0 2024-09-13T17:20:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.681+0000 7f269cc00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 0x7f268006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.681+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2694089590 con 0x7f26a00b22e0 2024-09-13T17:20:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.682+0000 7f269fe00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 0x7f268006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:19.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.683+0000 7f269fe00640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 0x7f268006d0f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f268c034000 tx=0x7f268c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:19.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.690+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f26940547a0 con 0x7f26a00b22e0 2024-09-13T17:20:19.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.944+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f26a00afd70 con 0x7f26a00b22e0 2024-09-13T17:20:19.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.945+0000 7f269cc00640 1 -- 172.21.15.28:0/2390405050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1380 (secure 0 0 0) 0x7f2694087e60 con 0x7f26a00b22e0 2024-09-13T17:20:19.946 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:19.946 INFO:teuthology.orchestra.run.smithi028.stdout:{"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":168,"num_osds":8,"num_up_osds":8,"osd_up_since":1726247277,"num_in_osds":8,"osd_in_since":1726247268,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":1048,"data_bytes":1076574590,"bytes_used":2725175296,"bytes_avail":770368937984,"bytes_total":773094113280},"fsmap":{"epoch":142,"id":11,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":11,"rank":0,"name":"a","status":"up:active","gid":10038}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":35,"modified":"2024-09-13T17:20:17.068804+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 msgr2=0x7f268006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 0x7f268006d0f0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f268c034000 tx=0x7f268c02e000 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 msgr2=0x7f26a0150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f269400e9e0 tx=0x7f269400eea0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 shutdown_connections 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f268006ac30 0x7f268006d0f0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f26a0159820 0x7f26a013ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26a0159440 0x7f26a014ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.947+0000 7f26a6400640 1 --2- 172.21.15.28:0/2390405050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f26a00b22e0 0x7f26a0150590 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.948+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 >> 172.21.15.28:0/2390405050 conn(0x7f26a0072e40 msgr2=0x7f26a0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.948+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 shutdown_connections 2024-09-13T17:20:19.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:19.948+0000 7f26a6400640 1 -- 172.21.15.28:0/2390405050 wait complete. 2024-09-13T17:20:19.965 INFO:tasks.cephfs.filesystem.ceph_manager:{} 2024-09-13T17:20:19.965 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:20:19.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:20:20.052 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:20:20.135 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:20:20.136 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:20.136 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:20.172 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:20.172 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:20.247 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.441+0000 7fd166600640 1 Processor -- start 2024-09-13T17:20:20.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.441+0000 7fd166600640 1 -- start start 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd166600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1600cabc0 0x7fd16006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd166600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16015e100 con 0x7fd160070cc0 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd166600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16015e3e0 con 0x7fd16015a540 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd166600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1600adb00 con 0x7fd1600cabc0 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd156a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd1600cabc0 0x7fd16006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd15fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd160070cc0 0x7fd16006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50776/0 (socket says 172.21.15.28:50776) 2024-09-13T17:20:20.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd15fe00640 1 -- 172.21.15.28:0/4015602602 learned_addr learned my addr 172.21.15.28:0/4015602602 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd156a00640 1 -- 172.21.15.28:0/4015602602 >> v1:172.21.15.189:6789/0 conn(0x7fd16015a540 legacy=0x7fd16015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.443+0000 7fd156a00640 1 -- 172.21.15.28:0/4015602602 >> v1:172.21.15.28:6789/0 conn(0x7fd160070cc0 legacy=0x7fd16006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.444+0000 7fd156a00640 1 -- 172.21.15.28:0/4015602602 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16015ef80 con 0x7fd1600cabc0 2024-09-13T17:20:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.444+0000 7fd15ea00640 1 -- 172.21.15.28:0/4015602602 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831346308 0 0) 0x7fd16015e100 con 0x7fd160070cc0 2024-09-13T17:20:20.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd156a00640 1 --2- 172.21.15.28:0/4015602602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd16006c430 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097a0 tx=0x7fd14802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144f28f960d9db1b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd15ea00640 1 -- 172.21.15.28:0/4015602602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148032440 con 0x7fd1600cabc0 2024-09-13T17:20:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd15ea00640 1 -- 172.21.15.28:0/4015602602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1480325a0 con 0x7fd1600cabc0 2024-09-13T17:20:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd15ea00640 1 -- 172.21.15.28:0/4015602602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148032930 con 0x7fd1600cabc0 2024-09-13T17:20:20.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 -- 172.21.15.28:0/4015602602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 msgr2=0x7fd16006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 --2- 172.21.15.28:0/4015602602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd16006c430 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fd1480097a0 tx=0x7fd14802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 -- 172.21.15.28:0/4015602602 shutdown_connections 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 --2- 172.21.15.28:0/4015602602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd16006c430 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 -- 172.21.15.28:0/4015602602 >> 172.21.15.28:0/4015602602 conn(0x7fd160069e80 msgr2=0x7fd16006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.445+0000 7fd166600640 1 -- 172.21.15.28:0/4015602602 shutdown_connections 2024-09-13T17:20:20.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.446+0000 7fd166600640 1 -- 172.21.15.28:0/4015602602 wait complete. 2024-09-13T17:20:20.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.447+0000 7fd166600640 1 Processor -- start 2024-09-13T17:20:20.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.447+0000 7fd166600640 1 -- start start 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd160070cc0 0x7fd160148070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd16015a540 0x7fd1601550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16014fc90 con 0x7fd1600cabc0 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601485b0 con 0x7fd160070cc0 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd166600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600aeef0 con 0x7fd16015a540 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd156a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd156a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42020/0 (socket says 172.21.15.28:42020) 2024-09-13T17:20:20.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd156a00640 1 -- 172.21.15.28:0/3440669373 learned_addr learned my addr 172.21.15.28:0/3440669373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd15f400640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd160070cc0 0x7fd160148070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.448+0000 7fd15fe00640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd16015a540 0x7fd1601550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 -- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd16015a540 msgr2=0x7fd1601550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd16015a540 0x7fd1601550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 -- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd160070cc0 msgr2=0x7fd160148070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd160070cc0 0x7fd160148070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 -- 172.21.15.28:0/3440669373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd148009400 con 0x7fd1600cabc0 2024-09-13T17:20:20.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.449+0000 7fd156a00640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.450+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd148030b80 con 0x7fd1600cabc0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.450+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd148030ce0 con 0x7fd1600cabc0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.450+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14803d680 con 0x7fd1600cabc0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.450+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1600aec10 con 0x7fd1600cabc0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.450+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16014f8b0 con 0x7fd1600cabc0 2024-09-13T17:20:20.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.451+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16014fdd0 con 0x7fd1600cabc0 2024-09-13T17:20:20.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.452+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd148039040 con 0x7fd1600cabc0 2024-09-13T17:20:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.453+0000 7fd157e00640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 0x7fd13c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.453+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd1480a9e30 con 0x7fd1600cabc0 2024-09-13T17:20:20.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.453+0000 7fd15f400640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 0x7fd13c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.454+0000 7fd15f400640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 0x7fd13c06d0c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd150000f60 tx=0x7fd150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.457+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd16014fdd0 con 0x7fd1600cabc0 2024-09-13T17:20:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.637+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16014fdd0 con 0x7fd1600cabc0 2024-09-13T17:20:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.639+0000 7fd157e00640 1 -- 172.21.15.28:0/3440669373 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16700 (secure 0 0 0) 0x7fd16014fdd0 con 0x7fd1600cabc0 2024-09-13T17:20:20.640 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:20.640 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":168,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:16.004870+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3709710897":"2024-09-14T17:20:16.004790+0000","172.21.15.28:6833/3709710897":"2024-09-14T17:20:16.004790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 msgr2=0x7fd13c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 0x7fd13c06d0c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fd150000f60 tx=0x7fd150009040 comp rx=0 tx=0).stop 2024-09-13T17:20:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 msgr2=0x7fd1601500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fd148000c00 tx=0x7fd1480046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 shutdown_connections 2024-09-13T17:20:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd13c06ac00 0x7fd13c06d0c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd16015a540 0x7fd1601550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd1600cabc0 0x7fd1601500b0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 --2- 172.21.15.28:0/3440669373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd160070cc0 0x7fd160148070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 >> 172.21.15.28:0/3440669373 conn(0x7fd160069e80 msgr2=0x7fd1600cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:20.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 shutdown_connections 2024-09-13T17:20:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.641+0000 7fd166600640 1 -- 172.21.15.28:0/3440669373 wait complete. 2024-09-13T17:20:20.658 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:19:26.637775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:19:27.700541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:20:20.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:20:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.815+0000 7fa9b6a00640 1 Processor -- start 2024-09-13T17:20:20.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.815+0000 7fa9b6a00640 1 -- start start 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9b6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9b015ab60 0x7fa9b006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9b6a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b006c8b0 con 0x7fa9b015a780 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9b6a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b006cb90 con 0x7fa9b0071490 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9b6a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00cc980 con 0x7fa9b015ab60 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9a7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9b015ab60 0x7fa9b006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9afe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa9b015a780 0x7fa9b0070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50790/0 (socket says 172.21.15.28:50790) 2024-09-13T17:20:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.816+0000 7fa9afe00640 1 -- 172.21.15.28:0/3134833157 learned_addr learned my addr 172.21.15.28:0/3134833157 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.817+0000 7fa9a7400640 1 -- 172.21.15.28:0/3134833157 >> v1:172.21.15.189:6789/0 conn(0x7fa9b0071490 legacy=0x7fa9b006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.817+0000 7fa9a7400640 1 -- 172.21.15.28:0/3134833157 >> v1:172.21.15.28:6789/0 conn(0x7fa9b015a780 legacy=0x7fa9b0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.817+0000 7fa9a7400640 1 -- 172.21.15.28:0/3134833157 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00ccc60 con 0x7fa9b015ab60 2024-09-13T17:20:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.817+0000 7fa9af400640 1 -- 172.21.15.28:0/3134833157 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360048732 0 0) 0x7fa9b006c8b0 con 0x7fa9b015a780 2024-09-13T17:20:20.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9a7400640 1 --2- 172.21.15.28:0/3134833157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b015ab60 0x7fa9b006bb40 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097a0 tx=0x7fa99802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81562497f7a26a80 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9af400640 1 -- 172.21.15.28:0/3134833157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998032440 con 0x7fa9b015ab60 2024-09-13T17:20:20.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9af400640 1 -- 172.21.15.28:0/3134833157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9980325a0 con 0x7fa9b015ab60 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9af400640 1 -- 172.21.15.28:0/3134833157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998032930 con 0x7fa9b015ab60 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9b6a00640 1 -- 172.21.15.28:0/3134833157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b015ab60 msgr2=0x7fa9b006bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.818+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/3134833157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b015ab60 0x7fa9b006bb40 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097a0 tx=0x7fa99802a230 comp rx=0 tx=0).stop 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.819+0000 7fa9b6a00640 1 -- 172.21.15.28:0/3134833157 shutdown_connections 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.819+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/3134833157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b015ab60 0x7fa9b006bb40 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.819+0000 7fa9b6a00640 1 -- 172.21.15.28:0/3134833157 >> 172.21.15.28:0/3134833157 conn(0x7fa9b00b0270 msgr2=0x7fa9b00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.819+0000 7fa9b6a00640 1 -- 172.21.15.28:0/3134833157 shutdown_connections 2024-09-13T17:20:20.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.819+0000 7fa9b6a00640 1 -- 172.21.15.28:0/3134833157 wait complete. 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.820+0000 7fa9b6a00640 1 Processor -- start 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.820+0000 7fa9b6a00640 1 -- start start 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b015a780 0x7fa9b014f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 0x7fa9b01460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b014a5e0 con 0x7fa9b0071490 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0151d10 con 0x7fa9b015a780 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.821+0000 7fa9b6a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b014f700 con 0x7fa9b015ab60 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42030/0 (socket says 172.21.15.28:42030) 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 -- 172.21.15.28:0/158029654 learned_addr learned my addr 172.21.15.28:0/158029654 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9a7400640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b015a780 0x7fa9b014f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9b4a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 0x7fa9b01460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 -- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 msgr2=0x7fa9b01460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 0x7fa9b01460f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 -- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b015a780 msgr2=0x7fa9b014f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b015a780 0x7fa9b014f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 -- 172.21.15.28:0/158029654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998009400 con 0x7fa9b0071490 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9b4a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 0x7fa9b01460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.822+0000 7fa9afe00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4a0 tx=0x7fa9a000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.823+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a00074e0 con 0x7fa9b0071490 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.823+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a0007640 con 0x7fa9b0071490 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.823+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0004a00 con 0x7fa9b0071490 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.823+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b01513e0 con 0x7fa9b0071490 2024-09-13T17:20:20.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.823+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b01519a0 con 0x7fa9b0071490 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.824+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0070d80 con 0x7fa9b0071490 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.825+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa9a0004b60 con 0x7fa9b0071490 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.826+0000 7fa9acc00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 0x7fa98c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.826+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa9a0088500 con 0x7fa9b0071490 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.826+0000 7fa9a7400640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 0x7fa98c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.827+0000 7fa9a7400640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 0x7fa98c06d0c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa998034000 tx=0x7fa99802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:20.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:20.830+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa9a0053690 con 0x7fa9b0071490 2024-09-13T17:20:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.007+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9b0153cb0 con 0x7fa9b0071490 2024-09-13T17:20:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.007+0000 7fa9acc00640 1 -- 172.21.15.28:0/158029654 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9a0018020 con 0x7fa9b0071490 2024-09-13T17:20:21.009 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 msgr2=0x7fa98c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 0x7fa98c06d0c0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa998034000 tx=0x7fa99802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 msgr2=0x7fa9b01521a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fa9a000c4a0 tx=0x7fa9a000c960 comp rx=0 tx=0).stop 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 shutdown_connections 2024-09-13T17:20:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98c06ac00 0x7fa98c06d0c0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9b015ab60 0x7fa9b01460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9b015a780 0x7fa9b014f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 --2- 172.21.15.28:0/158029654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9b0071490 0x7fa9b01521a0 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 >> 172.21.15.28:0/158029654 conn(0x7fa9b00b0270 msgr2=0x7fa9b0154960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 shutdown_connections 2024-09-13T17:20:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.010+0000 7fa9b6a00640 1 -- 172.21.15.28:0/158029654 wait complete. 2024-09-13T17:20:21.027 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:20:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.185+0000 7fe48d000640 1 Processor -- start 2024-09-13T17:20:21.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.185+0000 7fe48d000640 1 -- start start 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.186+0000 7fe48d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe488159770 0x7fe48806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.186+0000 7fe48d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe48815cd80 0x7fe488070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.186+0000 7fe48d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48806f920 con 0x7fe48806f3f0 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.186+0000 7fe48d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48815b290 con 0x7fe48815cd80 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.186+0000 7fe48d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48815b570 con 0x7fe488159770 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe48815cd80 0x7fe488070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe485e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe488159770 0x7fe48806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe48815cd80 0x7fe488070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:34590/0 (socket says 172.21.15.28:34590) 2024-09-13T17:20:21.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 -- 172.21.15.28:0/3403426515 learned_addr learned my addr 172.21.15.28:0/3403426515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 -- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe488159770 msgr2=0x7fe48806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 --2- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe488159770 0x7fe48806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 -- 172.21.15.28:0/3403426515 >> v1:172.21.15.189:6789/0 conn(0x7fe48806f3f0 legacy=0x7fe488159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe486800640 1 -- 172.21.15.28:0/3403426515 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48815b850 con 0x7fe48815cd80 2024-09-13T17:20:21.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.187+0000 7fe484a00640 1 -- 172.21.15.28:0/3403426515 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129838860 0 0) 0x7fe48806f920 con 0x7fe48806f3f0 2024-09-13T17:20:21.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.188+0000 7fe486800640 1 --2- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe48815cd80 0x7fe488070260 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe48000d150 tx=0x7fe48002c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cad1587540cad159 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe484a00640 1 -- 172.21.15.28:0/3403426515 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480009600 con 0x7fe48815cd80 2024-09-13T17:20:21.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe484a00640 1 -- 172.21.15.28:0/3403426515 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe480009760 con 0x7fe48815cd80 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe484a00640 1 -- 172.21.15.28:0/3403426515 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe480009af0 con 0x7fe48815cd80 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 -- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe48815cd80 msgr2=0x7fe488070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 --2- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe48815cd80 0x7fe488070260 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe48000d150 tx=0x7fe48002c890 comp rx=0 tx=0).stop 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 -- 172.21.15.28:0/3403426515 shutdown_connections 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 --2- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe48815cd80 0x7fe488070260 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 --2- 172.21.15.28:0/3403426515 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe488159770 0x7fe48806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.189+0000 7fe48d000640 1 -- 172.21.15.28:0/3403426515 >> 172.21.15.28:0/3403426515 conn(0x7fe4880b0270 msgr2=0x7fe4880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.190+0000 7fe48d000640 1 -- 172.21.15.28:0/3403426515 shutdown_connections 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.190+0000 7fe48d000640 1 -- 172.21.15.28:0/3403426515 wait complete. 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.191+0000 7fe48d000640 1 Processor -- start 2024-09-13T17:20:21.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.191+0000 7fe48d000640 1 -- start start 2024-09-13T17:20:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe488159770 0x7fe4880ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe48815cd80 0x7fe4880acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe488156340 con 0x7fe48806f3f0 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4881474b0 con 0x7fe488159770 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe48d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe488155730 con 0x7fe48815cd80 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe485e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe485e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42036/0 (socket says 172.21.15.28:42036) 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.192+0000 7fe485e00640 1 -- 172.21.15.28:0/1932868762 learned_addr learned my addr 172.21.15.28:0/1932868762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe486800640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe48815cd80 0x7fe4880acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485400640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe488159770 0x7fe4880ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 -- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe48815cd80 msgr2=0x7fe4880acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe48815cd80 0x7fe4880acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 -- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe488159770 msgr2=0x7fe4880ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe488159770 0x7fe4880ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 -- 172.21.15.28:0/1932868762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48000bd40 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe485e00640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe47800e9b0 tx=0x7fe47800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe478009670 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47800c9a0 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47800cd50 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.193+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe488155a10 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.194+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe488155fd0 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.195+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48806fd20 con 0x7fe48806f3f0 2024-09-13T17:20:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.196+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe478013470 con 0x7fe48806f3f0 2024-09-13T17:20:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.197+0000 7fe47e000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 0x7fe46806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.197+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe4780899a0 con 0x7fe48806f3f0 2024-09-13T17:20:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.198+0000 7fe485400640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 0x7fe46806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.198+0000 7fe485400640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 0x7fe46806d0c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe474007fd0 tx=0x7fe474009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.202+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe478054b30 con 0x7fe48806f3f0 2024-09-13T17:20:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.379+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4880cd830 con 0x7fe48806f3f0 2024-09-13T17:20:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.379+0000 7fe47e000640 1 -- 172.21.15.28:0/1932868762 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe478017020 con 0x7fe48806f3f0 2024-09-13T17:20:21.380 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 msgr2=0x7fe46806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 0x7fe46806d0c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe474007fd0 tx=0x7fe474009040 comp rx=0 tx=0).stop 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 msgr2=0x7fe488146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fe47800e9b0 tx=0x7fe47800ee70 comp rx=0 tx=0).stop 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 shutdown_connections 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe46806ac00 0x7fe46806d0c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe48815cd80 0x7fe4880acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe488159770 0x7fe4880ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 --2- 172.21.15.28:0/1932868762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe48806f3f0 0x7fe488146f70 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 >> 172.21.15.28:0/1932868762 conn(0x7fe4880b0270 msgr2=0x7fe488156bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 shutdown_connections 2024-09-13T17:20:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.381+0000 7fe48d000640 1 -- 172.21.15.28:0/1932868762 wait complete. 2024-09-13T17:20:21.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:20:21.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.555+0000 7feb83200640 1 Processor -- start 2024-09-13T17:20:21.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.556+0000 7feb83200640 1 -- start start 2024-09-13T17:20:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb83200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feb84071210 0x7feb840cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb83200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb84159e10 con 0x7feb84070e30 2024-09-13T17:20:21.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb83200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb8415a0f0 con 0x7feb840cddb0 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb83200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb840ce240 con 0x7feb84071210 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb81e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feb84070e30 0x7feb84071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50802/0 (socket says 172.21.15.28:50802) 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb81e00640 1 -- 172.21.15.28:0/2154855346 learned_addr learned my addr 172.21.15.28:0/2154855346 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.557+0000 7feb79400640 1 --2- 172.21.15.28:0/2154855346 >> v2:172.21.15.189:3300/0 conn(0x7feb84071210 0x7feb840cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554232819 0 0) 0x7feb84159e10 con 0x7feb84070e30 2024-09-13T17:20:21.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb68003520 con 0x7feb84070e30 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268398100 0 0) 0x7feb8415a0f0 con 0x7feb840cddb0 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb84159e10 con 0x7feb840cddb0 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2012332820 0 0) 0x7feb68003520 con 0x7feb84070e30 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb79400640 1 -- 172.21.15.28:0/2154855346 >> v1:172.21.15.189:6789/0 conn(0x7feb840cddb0 legacy=0x7feb8415aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb79400640 1 -- 172.21.15.28:0/2154855346 >> v1:172.21.15.28:6789/0 conn(0x7feb84070e30 legacy=0x7feb84071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.558+0000 7feb79400640 1 -- 172.21.15.28:0/2154855346 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb8415a3d0 con 0x7feb84071210 2024-09-13T17:20:21.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.559+0000 7feb79400640 1 --2- 172.21.15.28:0/2154855346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb84071210 0x7feb840cd870 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097a0 tx=0x7feb6c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265ab9400305f724 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.559+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74001670 con 0x7feb84071210 2024-09-13T17:20:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.559+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb6c02ac50 con 0x7feb84071210 2024-09-13T17:20:21.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.559+0000 7feb81400640 1 -- 172.21.15.28:0/2154855346 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c032440 con 0x7feb84071210 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 -- 172.21.15.28:0/2154855346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb84071210 msgr2=0x7feb840cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 --2- 172.21.15.28:0/2154855346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb84071210 0x7feb840cd870 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb6c0097a0 tx=0x7feb6c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 -- 172.21.15.28:0/2154855346 shutdown_connections 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 --2- 172.21.15.28:0/2154855346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb84071210 0x7feb840cd870 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 -- 172.21.15.28:0/2154855346 >> 172.21.15.28:0/2154855346 conn(0x7feb84069e80 msgr2=0x7feb8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 -- 172.21.15.28:0/2154855346 shutdown_connections 2024-09-13T17:20:21.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.560+0000 7feb83200640 1 -- 172.21.15.28:0/2154855346 wait complete. 2024-09-13T17:20:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.562+0000 7feb83200640 1 Processor -- start 2024-09-13T17:20:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.562+0000 7feb83200640 1 -- start start 2024-09-13T17:20:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb84071210 0x7feb841f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 0x7feb840cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840c5110 con 0x7feb84070e30 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb840c5360 con 0x7feb840cddb0 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb83200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb840c5640 con 0x7feb84071210 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb81e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42050/0 (socket says 172.21.15.28:42050) 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb81e00640 1 -- 172.21.15.28:0/4256083657 learned_addr learned my addr 172.21.15.28:0/4256083657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb79400640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb84071210 0x7feb841f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.563+0000 7feb82800640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 0x7feb840cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 -- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb84071210 msgr2=0x7feb841f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb84071210 0x7feb841f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 -- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 msgr2=0x7feb840cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 0x7feb840cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 -- 172.21.15.28:0/4256083657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c009400 con 0x7feb84070e30 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb82800640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 0x7feb840cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb81e00640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7feb740010e0 tx=0x7feb7400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74005ad0 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb74005c30 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb74007070 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb840c5920 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.564+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb840c5ee0 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.565+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb840c60d0 con 0x7feb84070e30 2024-09-13T17:20:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.566+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feb740071d0 con 0x7feb84070e30 2024-09-13T17:20:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.567+0000 7feb7aa00640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 0x7feb6006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.567+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feb74088ad0 con 0x7feb84070e30 2024-09-13T17:20:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.568+0000 7feb79400640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 0x7feb6006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.568+0000 7feb79400640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 0x7feb6006d0f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb6c034000 tx=0x7feb6c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.571+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feb740576f0 con 0x7feb84070e30 2024-09-13T17:20:21.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.751+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feb840cd870 con 0x7feb84070e30 2024-09-13T17:20:21.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.751+0000 7feb7aa00640 1 -- 172.21.15.28:0/4256083657 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7feb74017070 con 0x7feb84070e30 2024-09-13T17:20:21.752 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.753+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 msgr2=0x7feb6006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.753+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 0x7feb6006d0f0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7feb6c034000 tx=0x7feb6c02e000 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 msgr2=0x7feb840afd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7feb740010e0 tx=0x7feb7400cb70 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 shutdown_connections 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb6006ac30 0x7feb6006d0f0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb840cddb0 0x7feb840cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb84071210 0x7feb841f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 --2- 172.21.15.28:0/4256083657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb84070e30 0x7feb840afd10 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 >> 172.21.15.28:0/4256083657 conn(0x7feb84069e80 msgr2=0x7feb8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 shutdown_connections 2024-09-13T17:20:21.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.754+0000 7feb83200640 1 -- 172.21.15.28:0/4256083657 wait complete. 2024-09-13T17:20:21.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:20:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.929+0000 7ff951600640 1 Processor -- start 2024-09-13T17:20:21.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.929+0000 7ff951600640 1 -- start start 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff951600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff94c159770 0x7ff94c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff951600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff94c15cd80 0x7ff94c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff951600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff94c06f920 con 0x7ff94c06f3f0 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff951600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c15b290 con 0x7ff94c15cd80 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff951600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c15b570 con 0x7ff94c159770 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff943e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff94c06f3f0 0x7ff94c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50804/0 (socket says 172.21.15.28:50804) 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff943e00640 1 -- 172.21.15.28:0/2030804245 learned_addr learned my addr 172.21.15.28:0/2030804245 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.930+0000 7ff94b400640 1 --2- 172.21.15.28:0/2030804245 >> v2:172.21.15.28:3300/0 conn(0x7ff94c15cd80 0x7ff94c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.931+0000 7ff94aa00640 1 --2- 172.21.15.28:0/2030804245 >> v2:172.21.15.189:3300/0 conn(0x7ff94c159770 0x7ff94c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.931+0000 7ff94b400640 1 -- 172.21.15.28:0/2030804245 >> v2:172.21.15.189:3300/0 conn(0x7ff94c159770 msgr2=0x7ff94c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.931+0000 7ff94b400640 1 --2- 172.21.15.28:0/2030804245 >> v2:172.21.15.189:3300/0 conn(0x7ff94c159770 0x7ff94c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.931+0000 7ff94b400640 1 -- 172.21.15.28:0/2030804245 >> v1:172.21.15.28:6789/0 conn(0x7ff94c06f3f0 legacy=0x7ff94c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.931+0000 7ff94b400640 1 -- 172.21.15.28:0/2030804245 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c15b850 con 0x7ff94c15cd80 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.932+0000 7ff94b400640 1 --2- 172.21.15.28:0/2030804245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c070260 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff94c068040 tx=0x7ff94402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d4916a65db925ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.932+0000 7ff94a000640 1 -- 172.21.15.28:0/2030804245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944007740 con 0x7ff94c15cd80 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.932+0000 7ff94a000640 1 -- 172.21.15.28:0/2030804245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94402d050 con 0x7ff94c15cd80 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.932+0000 7ff94a000640 1 -- 172.21.15.28:0/2030804245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944007ac0 con 0x7ff94c15cd80 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 -- 172.21.15.28:0/2030804245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 msgr2=0x7ff94c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 --2- 172.21.15.28:0/2030804245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c070260 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff94c068040 tx=0x7ff94402a5f0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 -- 172.21.15.28:0/2030804245 shutdown_connections 2024-09-13T17:20:21.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 --2- 172.21.15.28:0/2030804245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c070260 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 --2- 172.21.15.28:0/2030804245 >> v2:172.21.15.189:3300/0 conn(0x7ff94c159770 0x7ff94c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 -- 172.21.15.28:0/2030804245 >> 172.21.15.28:0/2030804245 conn(0x7ff94c0b0270 msgr2=0x7ff94c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 -- 172.21.15.28:0/2030804245 shutdown_connections 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.933+0000 7ff951600640 1 -- 172.21.15.28:0/2030804245 wait complete. 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.934+0000 7ff951600640 1 Processor -- start 2024-09-13T17:20:21.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.935+0000 7ff951600640 1 -- start start 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.935+0000 7ff951600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff94c06f3f0 0x7ff94c1f6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.935+0000 7ff951600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c159770 0x7ff94c153810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.935+0000 7ff951600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff951600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c1566b0 con 0x7ff94c15cd80 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff951600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c152840 con 0x7ff94c159770 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff951600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c153d50 con 0x7ff94c06f3f0 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42068/0 (socket says 172.21.15.28:42068) 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 -- 172.21.15.28:0/2748331731 learned_addr learned my addr 172.21.15.28:0/2748331731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:21.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94aa00640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff94c06f3f0 0x7ff94c1f6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff943e00640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c159770 0x7ff94c153810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 -- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff94c06f3f0 msgr2=0x7ff94c1f6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff94c06f3f0 0x7ff94c1f6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 -- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c159770 msgr2=0x7ff94c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c159770 0x7ff94c153810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.936+0000 7ff94b400640 1 -- 172.21.15.28:0/2748331731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009a70 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff94b400640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ff94402abe0 tx=0x7ff944004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944031890 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440319f0 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff944031d60 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c153380 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.937+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c152d30 con 0x7ff94c15cd80 2024-09-13T17:20:21.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.938+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c153510 con 0x7ff94c15cd80 2024-09-13T17:20:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.939+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff94403c420 con 0x7ff94c15cd80 2024-09-13T17:20:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.940+0000 7ff942a00640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 0x7ff92806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.940+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff9440aafb0 con 0x7ff94c15cd80 2024-09-13T17:20:21.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.940+0000 7ff94aa00640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 0x7ff92806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:21.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.941+0000 7ff94aa00640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 0x7ff92806d0c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff93c006fd0 tx=0x7ff93c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:21.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:21.944+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff944076260 con 0x7ff94c15cd80 2024-09-13T17:20:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.138+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff94c061700 con 0x7ff94c15cd80 2024-09-13T17:20:22.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.138+0000 7ff942a00640 1 -- 172.21.15.28:0/2748331731 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v168) v1 ==== 80+0+122 (secure 0 0 0) 0x7ff944040090 con 0x7ff94c15cd80 2024-09-13T17:20:22.140 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6835/3709710897 2024-09-14T17:20:16.004790+0000 2024-09-13T17:20:22.140 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6833/3709710897 2024-09-14T17:20:16.004790+0000 2024-09-13T17:20:22.140 INFO:teuthology.orchestra.run.smithi028.stderr:listed 2 entries 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.140+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 msgr2=0x7ff92806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.140+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 0x7ff92806d0c0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff93c006fd0 tx=0x7ff93c008040 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.140+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 msgr2=0x7ff94c0acbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.140+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7ff94402abe0 tx=0x7ff944004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 shutdown_connections 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff92806ac00 0x7ff92806d0c0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff94c15cd80 0x7ff94c0acbc0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff94c159770 0x7ff94c153810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 --2- 172.21.15.28:0/2748331731 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff94c06f3f0 0x7ff94c1f6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 >> 172.21.15.28:0/2748331731 conn(0x7ff94c0b0270 msgr2=0x7ff94c0ce480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 shutdown_connections 2024-09-13T17:20:22.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.141+0000 7ff951600640 1 -- 172.21.15.28:0/2748331731 wait complete. 2024-09-13T17:20:22.158 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:20:22.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:22.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:20:22.228 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:20:22.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:22.228 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:20:22.264 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:20:22.265 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:22.305 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:20:22.305 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:22.305 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:22.340 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:20:22.340 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:20:22.340 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:22.383 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:20:22.383 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.537+0000 7f3196200640 1 Processor -- start 2024-09-13T17:20:22.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.538+0000 7f3196200640 1 -- start start 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f3196200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3190154a90 0x7f319015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f3196200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319015bb10 con 0x7f319006b360 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f3196200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f319006b900 con 0x7f31900b0b50 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f3196200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f319006bbe0 con 0x7f3190154a90 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f318fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f319006b360 0x7f31900b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50810/0 (socket says 172.21.15.28:50810) 2024-09-13T17:20:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.539+0000 7f318fe00640 1 -- 172.21.15.28:0/394293538 learned_addr learned my addr 172.21.15.28:0/394293538 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f3194e00640 1 --2- 172.21.15.28:0/394293538 >> v2:172.21.15.189:3300/0 conn(0x7f3190154a90 0x7f319015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905905512 0 0) 0x7f319015bb10 con 0x7f319006b360 2024-09-13T17:20:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3178003520 con 0x7f319006b360 2024-09-13T17:20:22.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479259616 0 0) 0x7f319006b900 con 0x7f31900b0b50 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319015bb10 con 0x7f31900b0b50 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2679276329 0 0) 0x7f3178003520 con 0x7f319006b360 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 >> v2:172.21.15.189:3300/0 conn(0x7f3190154a90 msgr2=0x7f319015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 --2- 172.21.15.28:0/394293538 >> v2:172.21.15.189:3300/0 conn(0x7f3190154a90 0x7f319015b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 >> v1:172.21.15.189:6789/0 conn(0x7f31900b0b50 legacy=0x7f31900af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319006bec0 con 0x7f319006b360 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f317c001670 con 0x7f319006b360 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 >> v1:172.21.15.28:6789/0 conn(0x7f319006b360 legacy=0x7f31900b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 0x7f3178003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31780047c0 0x7f3178004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178005180 0x7f317800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319006bbe0 con 0x7f3178003b50 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f317800c050 con 0x7f31780047c0 2024-09-13T17:20:22.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.540+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317800c210 con 0x7f3178005180 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 0x7f3178003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f318fe00640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31780047c0 0x7f3178004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3195800640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178005180 0x7f317800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 -- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178005180 msgr2=0x7f317800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178005180 0x7f317800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 -- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31780047c0 msgr2=0x7f3178004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31780047c0 0x7f3178004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3194e00640 1 -- 172.21.15.28:0/394293538 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3178003520 con 0x7f3178003b50 2024-09-13T17:20:22.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.541+0000 7f3195800640 1 -- 172.21.15.28:0/394293538 reap_dead start 2024-09-13T17:20:22.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.542+0000 7f3194e00640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 0x7f3178003f50 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f3184009ac0 tx=0x7f31780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5795eee98a99ec6f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.542+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184031400 con 0x7f3178003b50 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.542+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318402d050 con 0x7f3178003b50 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.542+0000 7f318f400640 1 -- 172.21.15.28:0/394293538 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3184031780 con 0x7f3178003b50 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 -- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 msgr2=0x7f3178003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 0x7f3178003f50 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f3184009ac0 tx=0x7f31780057f0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 -- 172.21.15.28:0/394293538 shutdown_connections 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 --2- 172.21.15.28:0/394293538 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3178003b50 0x7f3178003f50 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 -- 172.21.15.28:0/394293538 >> 172.21.15.28:0/394293538 conn(0x7f3190072e40 msgr2=0x7f3190073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 -- 172.21.15.28:0/394293538 shutdown_connections 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.543+0000 7f3196200640 1 -- 172.21.15.28:0/394293538 wait complete. 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.544+0000 7f3196200640 1 Processor -- start 2024-09-13T17:20:22.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.545+0000 7f3196200640 1 -- start start 2024-09-13T17:20:22.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.545+0000 7f3196200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178003b50 0x7f31901f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3196200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319013cfd0 0x7f31900ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3196200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3196200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31900c6300 con 0x7f31900aec10 2024-09-13T17:20:22.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3196200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31901f5ab0 con 0x7f319013cfd0 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3196200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31901f5d90 con 0x7f3178003b50 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42092/0 (socket says 172.21.15.28:42092) 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 -- 172.21.15.28:0/1941745463 learned_addr learned my addr 172.21.15.28:0/1941745463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3194e00640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178003b50 0x7f31901f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f318fe00640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319013cfd0 0x7f31900ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 -- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178003b50 msgr2=0x7f31901f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178003b50 0x7f31901f2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 -- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319013cfd0 msgr2=0x7f31900ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319013cfd0 0x7f31900ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.546+0000 7f3195800640 1 -- 172.21.15.28:0/1941745463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31901f6070 con 0x7f31900aec10 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f3195800640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f319015ab50 tx=0x7f318800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188003970 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3188003af0 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3188003ea0 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31901f6350 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.547+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31901f5740 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.548+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31900af060 con 0x7f31900aec10 2024-09-13T17:20:22.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.549+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f31880050d0 con 0x7f31900aec10 2024-09-13T17:20:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.550+0000 7f318cc00640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 0x7f317006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.550+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3188088ff0 con 0x7f31900aec10 2024-09-13T17:20:22.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.551+0000 7f3194e00640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 0x7f317006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.551+0000 7f3194e00640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 0x7f317006d0c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3184033000 tx=0x7f3184008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.555+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f31880bc9c0 con 0x7f31900aec10 2024-09-13T17:20:22.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.770+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f319006b900 con 0x7f31900aec10 2024-09-13T17:20:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.771+0000 7f318cc00640 1 -- 172.21.15.28:0/1941745463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7f31900af060 con 0x7f31900aec10 2024-09-13T17:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:22.772 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:22.772 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 msgr2=0x7f317006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 0x7f317006d0c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f3184033000 tx=0x7f3184008030 comp rx=0 tx=0).stop 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 msgr2=0x7f31901589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f319015ab50 tx=0x7f318800eb90 comp rx=0 tx=0).stop 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 shutdown_connections 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f317006ac00 0x7f317006d0c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f31900aec10 0x7f31901589f0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319013cfd0 0x7f31900ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 --2- 172.21.15.28:0/1941745463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3178003b50 0x7f31901f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 >> 172.21.15.28:0/1941745463 conn(0x7f3190072e40 msgr2=0x7f31900b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 shutdown_connections 2024-09-13T17:20:22.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.773+0000 7f3196200640 1 -- 172.21.15.28:0/1941745463 wait complete. 2024-09-13T17:20:22.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:22.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.944+0000 7fea8ce00640 1 Processor -- start 2024-09-13T17:20:22.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.944+0000 7fea8ce00640 1 -- start start 2024-09-13T17:20:22.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.945+0000 7fea8ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.945+0000 7fea8ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea88069e80 0x7fea8806a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.945+0000 7fea8ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea880d0d80 con 0x7fea8815c750 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.945+0000 7fea8ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea8806a850 con 0x7fea88069e80 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.945+0000 7fea8ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880b4d40 con 0x7fea880b43c0 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fea8815c750 0x7fea880d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50824/0 (socket says 172.21.15.28:50824) 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87400640 1 -- 172.21.15.28:0/2163116774 learned_addr learned my addr 172.21.15.28:0/2163116774 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87e00640 1 --2- 172.21.15.28:0/2163116774 >> v2:172.21.15.28:3300/0 conn(0x7fea88069e80 0x7fea8806a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea86a00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea86000640 1 -- 172.21.15.28:0/2163116774 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469548937 0 0) 0x7fea880d0d80 con 0x7fea8815c750 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea86000640 1 -- 172.21.15.28:0/2163116774 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea6c003520 con 0x7fea8815c750 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87e00640 1 -- 172.21.15.28:0/2163116774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 msgr2=0x7fea880b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87e00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87e00640 1 -- 172.21.15.28:0/2163116774 >> v1:172.21.15.28:6789/0 conn(0x7fea8815c750 legacy=0x7fea880d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea87e00640 1 -- 172.21.15.28:0/2163116774 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea880d0960 con 0x7fea88069e80 2024-09-13T17:20:22.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.946+0000 7fea86a00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.947+0000 7fea87e00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea8806a310 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fea80009d90 tx=0x7fea8002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=115f8a40e960e2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.947+0000 7fea86000640 1 -- 172.21.15.28:0/2163116774 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80031600 con 0x7fea88069e80 2024-09-13T17:20:22.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea86000640 1 -- 172.21.15.28:0/2163116774 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8002d050 con 0x7fea88069e80 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea86000640 1 -- 172.21.15.28:0/2163116774 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80031980 con 0x7fea88069e80 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 -- 172.21.15.28:0/2163116774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 msgr2=0x7fea8806a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea8806a310 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fea80009d90 tx=0x7fea8002a5d0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 -- 172.21.15.28:0/2163116774 shutdown_connections 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea8806a310 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 --2- 172.21.15.28:0/2163116774 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 -- 172.21.15.28:0/2163116774 >> 172.21.15.28:0/2163116774 conn(0x7fea880cf5a0 msgr2=0x7fea880cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 -- 172.21.15.28:0/2163116774 shutdown_connections 2024-09-13T17:20:22.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.948+0000 7fea8ce00640 1 -- 172.21.15.28:0/2163116774 wait complete. 2024-09-13T17:20:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.950+0000 7fea8ce00640 1 Processor -- start 2024-09-13T17:20:22.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.950+0000 7fea8ce00640 1 -- start start 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880c6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea8815c750 0x7fea880c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880cb3a0 con 0x7fea88069e80 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea880c6dd0 con 0x7fea8815c750 2024-09-13T17:20:22.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea8ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea880c70b0 con 0x7fea880b43c0 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42100/0 (socket says 172.21.15.28:42100) 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.951+0000 7fea87400640 1 -- 172.21.15.28:0/3119306404 learned_addr learned my addr 172.21.15.28:0/3119306404 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87e00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea8815c750 0x7fea880c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea86a00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 -- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 msgr2=0x7fea880c6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880c6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 -- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea8815c750 msgr2=0x7fea880c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea8815c750 0x7fea880c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 -- 172.21.15.28:0/3119306404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74009400 con 0x7fea88069e80 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea86a00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea87400640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c410 tx=0x7fea7800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78005ce0 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea78005e40 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea780072f0 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.952+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea80009a70 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.953+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea880c7670 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.953+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea880d12b0 con 0x7fea88069e80 2024-09-13T17:20:22.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.955+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fea78007510 con 0x7fea88069e80 2024-09-13T17:20:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.955+0000 7fea7f400640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 0x7fea6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:22.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.956+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fea78088ac0 con 0x7fea88069e80 2024-09-13T17:20:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.956+0000 7fea86a00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 0x7fea6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:22.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.957+0000 7fea86a00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 0x7fea6806d0c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fea880d08e0 tx=0x7fea74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:22.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:22.959+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fea78053c60 con 0x7fea88069e80 2024-09-13T17:20:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.141+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea88154c10 con 0x7fea88069e80 2024-09-13T17:20:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.142+0000 7fea7f400640 1 -- 172.21.15.28:0/3119306404 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16700 (secure 0 0 0) 0x7fea78017090 con 0x7fea88069e80 2024-09-13T17:20:23.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:23.143 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":168,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:16.004870+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3709710897":"2024-09-14T17:20:16.004790+0000","172.21.15.28:6833/3709710897":"2024-09-14T17:20:16.004790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 msgr2=0x7fea6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 0x7fea6806d0c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fea880d08e0 tx=0x7fea74008040 comp rx=0 tx=0).stop 2024-09-13T17:20:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 msgr2=0x7fea881f5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fea7800c410 tx=0x7fea7800c8d0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 shutdown_connections 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea6806ac00 0x7fea6806d0c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea8815c750 0x7fea880c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea880b43c0 0x7fea880c6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 --2- 172.21.15.28:0/3119306404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea88069e80 0x7fea881f5180 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 >> 172.21.15.28:0/3119306404 conn(0x7fea880cf5a0 msgr2=0x7fea880d2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 shutdown_connections 2024-09-13T17:20:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.144+0000 7fea8ce00640 1 -- 172.21.15.28:0/3119306404 wait complete. 2024-09-13T17:20:23.161 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:19:26.637775+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '166', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:19:27.700541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '167', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:20:23.161 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:20:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.320+0000 7f6694200640 1 Processor -- start 2024-09-13T17:20:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.320+0000 7f6694200640 1 -- start start 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6694200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f668c071490 0x7f668c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6694200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c06c8b0 con 0x7f668c15a780 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6694200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f668c06cb90 con 0x7f668c15ab60 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6694200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668c0cc980 con 0x7f668c071490 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6692200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f668c071490 0x7f668c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6691800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f668c15a780 0x7f668c070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50832/0 (socket says 172.21.15.28:50832) 2024-09-13T17:20:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.321+0000 7f6691800640 1 -- 172.21.15.28:0/124170949 learned_addr learned my addr 172.21.15.28:0/124170949 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.322+0000 7f6692200640 1 -- 172.21.15.28:0/124170949 >> v1:172.21.15.189:6789/0 conn(0x7f668c15ab60 legacy=0x7f668c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.322+0000 7f6692200640 1 -- 172.21.15.28:0/124170949 >> v1:172.21.15.28:6789/0 conn(0x7f668c15a780 legacy=0x7f668c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.322+0000 7f6692200640 1 -- 172.21.15.28:0/124170949 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f668c0ccc60 con 0x7f668c071490 2024-09-13T17:20:23.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.322+0000 7f668be00640 1 -- 172.21.15.28:0/124170949 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254833235 0 0) 0x7f668c06c8b0 con 0x7f668c15a780 2024-09-13T17:20:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.323+0000 7f6692200640 1 --2- 172.21.15.28:0/124170949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c06c250 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6684009d90 tx=0x7f668402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957588b258ef3ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.323+0000 7f668be00640 1 -- 172.21.15.28:0/124170949 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684007740 con 0x7f668c071490 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.323+0000 7f668be00640 1 -- 172.21.15.28:0/124170949 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668402d050 con 0x7f668c071490 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.323+0000 7f668be00640 1 -- 172.21.15.28:0/124170949 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6684007ac0 con 0x7f668c071490 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 -- 172.21.15.28:0/124170949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 msgr2=0x7f668c06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 --2- 172.21.15.28:0/124170949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c06c250 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f6684009d90 tx=0x7f668402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 -- 172.21.15.28:0/124170949 shutdown_connections 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 --2- 172.21.15.28:0/124170949 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c06c250 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 -- 172.21.15.28:0/124170949 >> 172.21.15.28:0/124170949 conn(0x7f668c0b0270 msgr2=0x7f668c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 -- 172.21.15.28:0/124170949 shutdown_connections 2024-09-13T17:20:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.324+0000 7f6694200640 1 -- 172.21.15.28:0/124170949 wait complete. 2024-09-13T17:20:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.325+0000 7f6694200640 1 Processor -- start 2024-09-13T17:20:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.325+0000 7f6694200640 1 -- start start 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668c15a780 0x7f668c15d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f668c15ab60 0x7f668c15bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15b880 con 0x7f668c071490 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f668c15af70 con 0x7f668c15a780 2024-09-13T17:20:23.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.326+0000 7f6694200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668c154260 con 0x7f668c15ab60 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42114/0 (socket says 172.21.15.28:42114) 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 -- 172.21.15.28:0/2725100770 learned_addr learned my addr 172.21.15.28:0/2725100770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6690e00640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668c15a780 0x7f668c15d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6692200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f668c15ab60 0x7f668c15bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 -- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f668c15ab60 msgr2=0x7f668c15bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f668c15ab60 0x7f668c15bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 -- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668c15a780 msgr2=0x7f668c15d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668c15a780 0x7f668c15d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.327+0000 7f6691800640 1 -- 172.21.15.28:0/2725100770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6684009a70 con 0x7f668c071490 2024-09-13T17:20:23.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6691800640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f668000c4d0 tx=0x7f668000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800074e0 con 0x7f668c071490 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6680007640 con 0x7f668c071490 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668c0afb70 con 0x7f668c071490 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6680004a20 con 0x7f668c071490 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.328+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f668c15b4a0 con 0x7f668c071490 2024-09-13T17:20:23.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.329+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668c070d80 con 0x7f668c071490 2024-09-13T17:20:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.330+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6680004b80 con 0x7f668c071490 2024-09-13T17:20:23.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.331+0000 7f6689600640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 0x7f666c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.331+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6680088830 con 0x7f668c071490 2024-09-13T17:20:23.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.331+0000 7f6690e00640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 0x7f666c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.332+0000 7f6690e00640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 0x7f666c06d0c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f668c1f5700 tx=0x7f6678009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.335+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6680053a50 con 0x7f668c071490 2024-09-13T17:20:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.511+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f668c0cee50 con 0x7f668c071490 2024-09-13T17:20:23.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.511+0000 7f6689600640 1 -- 172.21.15.28:0/2725100770 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v168) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66800180a0 con 0x7f668c071490 2024-09-13T17:20:23.512 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 msgr2=0x7f666c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 0x7f666c06d0c0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f668c1f5700 tx=0x7f6678009040 comp rx=0 tx=0).stop 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 msgr2=0x7f668c153710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f668000c4d0 tx=0x7f668000c990 comp rx=0 tx=0).stop 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 shutdown_connections 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f666c06ac00 0x7f666c06d0c0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f668c15ab60 0x7f668c15bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f668c15a780 0x7f668c15d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 --2- 172.21.15.28:0/2725100770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f668c071490 0x7f668c153710 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 >> 172.21.15.28:0/2725100770 conn(0x7f668c0b0270 msgr2=0x7f668c0ce3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 shutdown_connections 2024-09-13T17:20:23.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.514+0000 7f6694200640 1 -- 172.21.15.28:0/2725100770 wait complete. 2024-09-13T17:20:23.531 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:20:23.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.751+0000 7f8d91a00640 1 Processor -- start 2024-09-13T17:20:23.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.752+0000 7f8d91a00640 1 -- start start 2024-09-13T17:20:23.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.756+0000 7f8d91a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.757+0000 7f8d91a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 0x7f8d8c0b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.757+0000 7f8d91a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8d8c0b2100 0x7f8d8c06b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.757+0000 7f8d91a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c1610a0 con 0x7f8d8c06d070 2024-09-13T17:20:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.757+0000 7f8d91a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d8c0b2620 con 0x7f8d8c0b2100 2024-09-13T17:20:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.757+0000 7f8d91a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c0b2900 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.760+0000 7f8d8aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.760+0000 7f8d8aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:50722/0 (socket says 172.21.15.28:50722) 2024-09-13T17:20:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.760+0000 7f8d8aa00640 1 -- 172.21.15.28:0/1571248700 learned_addr learned my addr 172.21.15.28:0/1571248700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:23.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.761+0000 7f8d8a000640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 0x7f8d8c0b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.761+0000 7f8d8aa00640 1 -- 172.21.15.28:0/1571248700 >> v2:172.21.15.189:3300/0 conn(0x7f8d8c0b2100 msgr2=0x7f8d8c06b530 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T17:20:23.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.761+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.189:3300/0 conn(0x7f8d8c0b2100 0x7f8d8c06b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.761+0000 7f8d8aa00640 1 -- 172.21.15.28:0/1571248700 >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 msgr2=0x7f8d8c0b1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:23.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.763+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 0x7f8d8c0b1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.763+0000 7f8d8aa00640 1 -- 172.21.15.28:0/1571248700 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c0b2be0 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.765+0000 7f8d8a000640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 0x7f8d8c0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:23.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.766+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/1571248700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c071880 tx=0x7f8d7c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaea1663b123265c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.768+0000 7f8d89600640 1 -- 172.21.15.28:0/1571248700 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c032440 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.768+0000 7f8d89600640 1 -- 172.21.15.28:0/1571248700 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c0325a0 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.768+0000 7f8d89600640 1 -- 172.21.15.28:0/1571248700 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c032970 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.769+0000 7f8d91a00640 1 -- 172.21.15.28:0/1571248700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 msgr2=0x7f8d8c0b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.769+0000 7f8d91a00640 1 --2- 172.21.15.28:0/1571248700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c071880 tx=0x7f8d7c02a210 comp rx=0 tx=0).stop 2024-09-13T17:20:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.771+0000 7f8d91a00640 1 -- 172.21.15.28:0/1571248700 shutdown_connections 2024-09-13T17:20:23.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.771+0000 7f8d91a00640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.28:3300/0 conn(0x7f8d8c06d070 0x7f8d8c0b1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.771+0000 7f8d91a00640 1 --2- 172.21.15.28:0/1571248700 >> v2:172.21.15.189:3300/0 conn(0x7f8d8c0b2100 0x7f8d8c06b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.771+0000 7f8d91a00640 1 --2- 172.21.15.28:0/1571248700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c0b3480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.771+0000 7f8d91a00640 1 -- 172.21.15.28:0/1571248700 >> 172.21.15.28:0/1571248700 conn(0x7f8d8c0b0270 msgr2=0x7f8d8c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.775+0000 7f8d91a00640 1 -- 172.21.15.28:0/1571248700 shutdown_connections 2024-09-13T17:20:23.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.775+0000 7f8d91a00640 1 -- 172.21.15.28:0/1571248700 wait complete. 2024-09-13T17:20:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.778+0000 7f8d91a00640 1 Processor -- start 2024-09-13T17:20:23.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.779+0000 7f8d91a00640 1 -- start start 2024-09-13T17:20:23.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c1eb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0b2100 0x7f8d8c1e9490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c1e6e10 con 0x7f8d8c06d070 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c1e70b0 con 0x7f8d8c0b2100 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.780+0000 7f8d91a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c1e7390 con 0x7f8d8c0b17e0 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.782+0000 7f8d8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.782+0000 7f8d8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:42136/0 (socket says 172.21.15.28:42136) 2024-09-13T17:20:23.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.782+0000 7f8d8aa00640 1 -- 172.21.15.28:0/4081010418 learned_addr learned my addr 172.21.15.28:0/4081010418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:23.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.784+0000 7f8d8aa00640 1 -- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 msgr2=0x7f8d8c1eb680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:20:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.785+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c1eb680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.785+0000 7f8d8aa00640 1 -- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0b2100 msgr2=0x7f8d8c1e9490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T17:20:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.785+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0b2100 0x7f8d8c1e9490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.785+0000 7f8d8aa00640 1 -- 172.21.15.28:0/4081010418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7c0094d0 con 0x7f8d8c06d070 2024-09-13T17:20:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.789+0000 7f8d8aa00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009960 tx=0x7f8d7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.789+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c032440 con 0x7f8d8c06d070 2024-09-13T17:20:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.789+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7c030b20 con 0x7f8d8c06d070 2024-09-13T17:20:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.789+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7c03b400 con 0x7f8d8c06d070 2024-09-13T17:20:23.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.789+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c1e7670 con 0x7f8d8c06d070 2024-09-13T17:20:23.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.790+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c150b80 con 0x7f8d8c06d070 2024-09-13T17:20:23.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.791+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c1e7b90 con 0x7f8d8c06d070 2024-09-13T17:20:23.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.792+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8d7c039040 con 0x7f8d8c06d070 2024-09-13T17:20:23.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.794+0000 7f8d82a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 0x7f8d6c06d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.794+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8d7c0a9bb0 con 0x7f8d8c06d070 2024-09-13T17:20:23.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.794+0000 7f8d8a000640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 0x7f8d6c06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:23.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.795+0000 7f8d8a000640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 0x7f8d6c06d190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c15fe20 tx=0x7f8d840073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:23.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.800+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8d8c1e7b90 con 0x7f8d8c06d070 2024-09-13T17:20:23.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.980+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8d8c1e7b90 con 0x7f8d8c06d070 2024-09-13T17:20:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.981+0000 7f8d82a00640 1 -- 172.21.15.28:0/4081010418 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v168) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8d8c1e7b90 con 0x7f8d8c06d070 2024-09-13T17:20:23.982 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:20:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 msgr2=0x7f8d6c06d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 0x7f8d6c06d190 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c15fe20 tx=0x7f8d840073c0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 msgr2=0x7f8d8c150640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009960 tx=0x7f8d7c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 shutdown_connections 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8d6c06acd0 0x7f8d6c06d190 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d8c0b2100 0x7f8d8c1e9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8d8c0b17e0 0x7f8d8c1eb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 --2- 172.21.15.28:0/4081010418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d8c06d070 0x7f8d8c150640 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 >> 172.21.15.28:0/4081010418 conn(0x7f8d8c0b0270 msgr2=0x7f8d8c0cd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 shutdown_connections 2024-09-13T17:20:23.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:23.983+0000 7f8d91a00640 1 -- 172.21.15.28:0/4081010418 wait complete. 2024-09-13T17:20:23.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:20:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.158+0000 7f3466400640 1 Processor -- start 2024-09-13T17:20:24.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.158+0000 7f3466400640 1 -- start start 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3466400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3460070430 0x7f346015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3466400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f346015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3466400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f346015ab60 con 0x7f3460159770 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3466400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006f920 con 0x7f3460070430 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3466400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346015b290 con 0x7f346006f3f0 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3457e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3460070430 0x7f346015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3457e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3460070430 0x7f346015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33022/0 (socket says 172.21.15.28:33022) 2024-09-13T17:20:24.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.159+0000 7f3457e00640 1 -- 172.21.15.28:0/1641114528 learned_addr learned my addr 172.21.15.28:0/1641114528 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.160+0000 7f345fe00640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f346015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.160+0000 7f3457e00640 1 -- 172.21.15.28:0/1641114528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 msgr2=0x7f346015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.160+0000 7f3457e00640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f346015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.160+0000 7f3457e00640 1 -- 172.21.15.28:0/1641114528 >> v1:172.21.15.189:6789/0 conn(0x7f3460159770 legacy=0x7f346006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.160+0000 7f3457e00640 1 -- 172.21.15.28:0/1641114528 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f346015b570 con 0x7f3460070430 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.161+0000 7f3457e00640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f346015bd70 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3448009870 tx=0x7f344802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df0b07831e1c13b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.161+0000 7f345ea00640 1 -- 172.21.15.28:0/1641114528 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448032440 con 0x7f3460070430 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.161+0000 7f345ea00640 1 -- 172.21.15.28:0/1641114528 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34480325a0 con 0x7f3460070430 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.161+0000 7f345ea00640 1 -- 172.21.15.28:0/1641114528 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448032930 con 0x7f3460070430 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 -- 172.21.15.28:0/1641114528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 msgr2=0x7f346015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f346015bd70 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f3448009870 tx=0x7f344802a1c0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 -- 172.21.15.28:0/1641114528 shutdown_connections 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f346015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 --2- 172.21.15.28:0/1641114528 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f346015bd70 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 -- 172.21.15.28:0/1641114528 >> 172.21.15.28:0/1641114528 conn(0x7f34600b0270 msgr2=0x7f34600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 -- 172.21.15.28:0/1641114528 shutdown_connections 2024-09-13T17:20:24.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.162+0000 7f3466400640 1 -- 172.21.15.28:0/1641114528 wait complete. 2024-09-13T17:20:24.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.164+0000 7f3466400640 1 Processor -- start 2024-09-13T17:20:24.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.164+0000 7f3466400640 1 -- start start 2024-09-13T17:20:24.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f3460144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3460159770 0x7f3460159b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460152110 con 0x7f3460070430 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3460151e30 con 0x7f3460159770 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3466400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346014ff30 con 0x7f346006f3f0 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f3460144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f345f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f345f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:53004/0 (socket says 172.21.15.28:53004) 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f345f400640 1 -- 172.21.15.28:0/2956508034 learned_addr learned my addr 172.21.15.28:0/2956508034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f345fe00640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3460159770 0x7f3460159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 -- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 msgr2=0x7f34601535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 -- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3460159770 msgr2=0x7f3460159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3460159770 0x7f3460159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f3457e00640 1 -- 172.21.15.28:0/2956508034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34480094d0 con 0x7f3460070430 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.165+0000 7f345f400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3457e00640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f3460144b00 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f3448031000 tx=0x7f34480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3448032440 con 0x7f3460070430 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3448030ba0 con 0x7f3460070430 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344803d440 con 0x7f3460070430 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3460150210 con 0x7f3460070430 2024-09-13T17:20:24.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.166+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460153ae0 con 0x7f3460070430 2024-09-13T17:20:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.167+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346014fd90 con 0x7f3460070430 2024-09-13T17:20:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.168+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3448039040 con 0x7f3460070430 2024-09-13T17:20:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.169+0000 7f3457400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.169+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f34480a9cf0 con 0x7f3460070430 2024-09-13T17:20:24.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.170+0000 7f345f400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.170+0000 7f345f400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 0x7f343c06d0c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3450007fd0 tx=0x7f3450009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.173+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f346014fd90 con 0x7f3460070430 2024-09-13T17:20:24.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.357+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f346014fd90 con 0x7f3460070430 2024-09-13T17:20:24.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.358+0000 7f3457400640 1 -- 172.21.15.28:0/2956508034 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v168) v1 ==== 100+0+11 (secure 0 0 0) 0x7f346014fd90 con 0x7f3460070430 2024-09-13T17:20:24.359 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:20:24.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 msgr2=0x7f343c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 0x7f343c06d0c0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f3450007fd0 tx=0x7f3450009040 comp rx=0 tx=0).stop 2024-09-13T17:20:24.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 msgr2=0x7f3460144b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f3460144b00 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f3448031000 tx=0x7f34480046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 shutdown_connections 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3460159770 0x7f3460159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3460070430 0x7f3460144b00 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 --2- 172.21.15.28:0/2956508034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346006f3f0 0x7f34601535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 >> 172.21.15.28:0/2956508034 conn(0x7f34600b0270 msgr2=0x7f346014daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 shutdown_connections 2024-09-13T17:20:24.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.360+0000 7f3466400640 1 -- 172.21.15.28:0/2956508034 wait complete. 2024-09-13T17:20:24.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:24.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.531+0000 7f6b1e200640 1 Processor -- start 2024-09-13T17:20:24.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.532+0000 7f6b1e200640 1 -- start start 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b180b0b50 0x7f6b180af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1806b900 con 0x7f6b18154a90 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1e200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1806bbe0 con 0x7f6b1806b360 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1806bec0 con 0x7f6b180b0b50 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b180b0b50 0x7f6b180af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1ce00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b18154a90 0x7f6b1815b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54860/0 (socket says 172.21.15.28:54860) 2024-09-13T17:20:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.533+0000 7f6b1ce00640 1 -- 172.21.15.28:0/1644698426 learned_addr learned my addr 172.21.15.28:0/1644698426 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.534+0000 7f6b1d800640 1 -- 172.21.15.28:0/1644698426 >> v1:172.21.15.189:6789/0 conn(0x7f6b1806b360 legacy=0x7f6b180b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.534+0000 7f6b1d800640 1 -- 172.21.15.28:0/1644698426 >> v1:172.21.15.28:6789/0 conn(0x7f6b18154a90 legacy=0x7f6b1815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.534+0000 7f6b1d800640 1 -- 172.21.15.28:0/1644698426 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1806c1a0 con 0x7f6b180b0b50 2024-09-13T17:20:24.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b1d800640 1 --2- 172.21.15.28:0/1644698426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180b0b50 0x7f6b180af4b0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009d90 tx=0x7f6b1002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73545205fc5a8647 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b17400640 1 -- 172.21.15.28:0/1644698426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10007740 con 0x7f6b180b0b50 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b17400640 1 -- 172.21.15.28:0/1644698426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1002d050 con 0x7f6b180b0b50 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b17400640 1 -- 172.21.15.28:0/1644698426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10007ac0 con 0x7f6b180b0b50 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b1e200640 1 -- 172.21.15.28:0/1644698426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180b0b50 msgr2=0x7f6b180af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.535+0000 7f6b1e200640 1 --2- 172.21.15.28:0/1644698426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180b0b50 0x7f6b180af4b0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009d90 tx=0x7f6b1002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.536+0000 7f6b1e200640 1 -- 172.21.15.28:0/1644698426 shutdown_connections 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.536+0000 7f6b1e200640 1 --2- 172.21.15.28:0/1644698426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b180b0b50 0x7f6b180af4b0 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.536+0000 7f6b1e200640 1 -- 172.21.15.28:0/1644698426 >> 172.21.15.28:0/1644698426 conn(0x7f6b18072e40 msgr2=0x7f6b18073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.536+0000 7f6b1e200640 1 -- 172.21.15.28:0/1644698426 shutdown_connections 2024-09-13T17:20:24.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.536+0000 7f6b1e200640 1 -- 172.21.15.28:0/1644698426 wait complete. 2024-09-13T17:20:24.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.537+0000 7f6b1e200640 1 Processor -- start 2024-09-13T17:20:24.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.538+0000 7f6b1e200640 1 -- start start 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.538+0000 7f6b1e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.538+0000 7f6b1e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b180b0b50 0x7f6b18153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.538+0000 7f6b1e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 0x7f6b18157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1813c3a0 con 0x7f6b1806b360 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180b3060 con 0x7f6b180b0b50 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181f68e0 con 0x7f6b18154a90 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33058/0 (socket says 172.21.15.28:33058) 2024-09-13T17:20:24.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 -- 172.21.15.28:0/284169745 learned_addr learned my addr 172.21.15.28:0/284169745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b17e00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b180b0b50 0x7f6b18153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1d800640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 0x7f6b18157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 -- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 msgr2=0x7f6b18157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 0x7f6b18157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 -- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b180b0b50 msgr2=0x7f6b18153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b180b0b50 0x7f6b18153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 -- 172.21.15.28:0/284169745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b10009a70 con 0x7f6b1806b360 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1d800640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 0x7f6b18157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.539+0000 7f6b1ce00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00c4b0 tx=0x7f6b0c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.540+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c005ce0 con 0x7f6b1806b360 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.540+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0c005e40 con 0x7f6b1806b360 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.540+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c007270 con 0x7f6b1806b360 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.540+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b181f6600 con 0x7f6b1806b360 2024-09-13T17:20:24.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.540+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b180b2cf0 con 0x7f6b1806b360 2024-09-13T17:20:24.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.541+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b180b0440 con 0x7f6b1806b360 2024-09-13T17:20:24.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.542+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6b0c0073d0 con 0x7f6b1806b360 2024-09-13T17:20:24.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.543+0000 7f6b14c00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 0x7f6af806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.543+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6b0c0887e0 con 0x7f6b1806b360 2024-09-13T17:20:24.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.544+0000 7f6b17e00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 0x7f6af806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.544+0000 7f6b17e00640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 0x7f6af806d0c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6b1813eb10 tx=0x7f6b04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.548+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6b0c053a00 con 0x7f6b1806b360 2024-09-13T17:20:24.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.764+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b18159400 con 0x7f6b1806b360 2024-09-13T17:20:24.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.764+0000 7f6b14c00640 1 -- 172.21.15.28:0/284169745 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7f6b0c017090 con 0x7f6b1806b360 2024-09-13T17:20:24.765 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:24.766 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:24.766 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:24.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 msgr2=0x7f6af806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 0x7f6af806d0c0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6b1813eb10 tx=0x7f6b04009040 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 msgr2=0x7f6b180b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00c4b0 tx=0x7f6b0c00c970 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 shutdown_connections 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6af806ac00 0x7f6af806d0c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b18154a90 0x7f6b18157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b180b0b50 0x7f6b18153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 --2- 172.21.15.28:0/284169745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b1806b360 0x7f6b180b34f0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 >> 172.21.15.28:0/284169745 conn(0x7f6b18072e40 msgr2=0x7f6b181583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.766+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 shutdown_connections 2024-09-13T17:20:24.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.767+0000 7f6b1e200640 1 -- 172.21.15.28:0/284169745 wait complete. 2024-09-13T17:20:24.783 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:24.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.937+0000 7fc78aa00640 1 Processor -- start 2024-09-13T17:20:24.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.937+0000 7fc78aa00640 1 -- start start 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc78aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 0x7fc78406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc78aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc784070cc0 0x7fc784071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc78aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78415e100 con 0x7fc7840cabc0 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc78aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7840716b0 con 0x7fc784070cc0 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc78aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840adb00 con 0x7fc78415a540 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc77b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc7840cabc0 0x7fc78415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54864/0 (socket says 172.21.15.28:54864) 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc77b400640 1 -- 172.21.15.28:0/4034414665 learned_addr learned my addr 172.21.15.28:0/4034414665 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc783e00640 1 --2- 172.21.15.28:0/4034414665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 0x7fc78406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc788a00640 1 --2- 172.21.15.28:0/4034414665 >> v2:172.21.15.189:3300/0 conn(0x7fc784070cc0 0x7fc784071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.939+0000 7fc783400640 1 -- 172.21.15.28:0/4034414665 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474325986 0 0) 0x7fc78415e100 con 0x7fc7840cabc0 2024-09-13T17:20:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.940+0000 7fc783400640 1 -- 172.21.15.28:0/4034414665 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc768003520 con 0x7fc7840cabc0 2024-09-13T17:20:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.940+0000 7fc783e00640 1 -- 172.21.15.28:0/4034414665 >> v2:172.21.15.189:3300/0 conn(0x7fc784070cc0 msgr2=0x7fc784071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.940+0000 7fc783e00640 1 --2- 172.21.15.28:0/4034414665 >> v2:172.21.15.189:3300/0 conn(0x7fc784070cc0 0x7fc784071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.940+0000 7fc783e00640 1 -- 172.21.15.28:0/4034414665 >> v1:172.21.15.28:6789/0 conn(0x7fc7840cabc0 legacy=0x7fc78415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.940+0000 7fc783e00640 1 -- 172.21.15.28:0/4034414665 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78415e3e0 con 0x7fc78415a540 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.941+0000 7fc783e00640 1 --2- 172.21.15.28:0/4034414665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 0x7fc78406c430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097a0 tx=0x7fc77402a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7507161b745e36bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.941+0000 7fc783400640 1 -- 172.21.15.28:0/4034414665 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77402ae40 con 0x7fc78415a540 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.941+0000 7fc783400640 1 -- 172.21.15.28:0/4034414665 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774032440 con 0x7fc78415a540 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.941+0000 7fc783400640 1 -- 172.21.15.28:0/4034414665 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7740327b0 con 0x7fc78415a540 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 -- 172.21.15.28:0/4034414665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 msgr2=0x7fc78406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 --2- 172.21.15.28:0/4034414665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 0x7fc78406c430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc7740097a0 tx=0x7fc77402a0f0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 -- 172.21.15.28:0/4034414665 shutdown_connections 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 --2- 172.21.15.28:0/4034414665 >> v2:172.21.15.189:3300/0 conn(0x7fc784070cc0 0x7fc784071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 --2- 172.21.15.28:0/4034414665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc78415a540 0x7fc78406c430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 -- 172.21.15.28:0/4034414665 >> 172.21.15.28:0/4034414665 conn(0x7fc784069e80 msgr2=0x7fc78406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 -- 172.21.15.28:0/4034414665 shutdown_connections 2024-09-13T17:20:24.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.942+0000 7fc78aa00640 1 -- 172.21.15.28:0/4034414665 wait complete. 2024-09-13T17:20:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.943+0000 7fc78aa00640 1 Processor -- start 2024-09-13T17:20:24.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.944+0000 7fc78aa00640 1 -- start start 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 0x7fc78414b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 0x7fc78414fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78414bf00 con 0x7fc784070cc0 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc784152020 con 0x7fc78415a540 2024-09-13T17:20:24.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc78aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7841efcc0 con 0x7fc7840cabc0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33074/0 (socket says 172.21.15.28:33074) 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 -- 172.21.15.28:0/1082403700 learned_addr learned my addr 172.21.15.28:0/1082403700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc77b400640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 0x7fc78414b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 -- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 msgr2=0x7fc78414b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 0x7fc78414b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 -- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 msgr2=0x7fc78414fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc788a00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 0x7fc78414fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 0x7fc78414fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc783e00640 1 -- 172.21.15.28:0/1082403700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774009400 con 0x7fc784070cc0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc77b400640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 0x7fc78414b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.945+0000 7fc788a00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 0x7fc78414fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc783e00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc774000c00 tx=0x7fc7740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774004230 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc774004390 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78414f170 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc774003810 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.946+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7840ccf60 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.947+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7840cbeb0 con 0x7fc784070cc0 2024-09-13T17:20:24.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.948+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc774039050 con 0x7fc784070cc0 2024-09-13T17:20:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.949+0000 7fc780c00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 0x7fc76006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.949+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc7740a9a80 con 0x7fc784070cc0 2024-09-13T17:20:24.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.950+0000 7fc77b400640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 0x7fc76006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:24.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.951+0000 7fc77b400640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 0x7fc76006d0c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc784159ba0 tx=0x7fc76c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:24.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:24.954+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc774074b70 con 0x7fc784070cc0 2024-09-13T17:20:25.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.137+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc784071170 con 0x7fc784070cc0 2024-09-13T17:20:25.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.138+0000 7fc780c00640 1 -- 172.21.15.28:0/1082403700 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16700 (secure 0 0 0) 0x7fc774037090 con 0x7fc784070cc0 2024-09-13T17:20:25.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:25.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":168,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:16.004870+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3709710897":"2024-09-14T17:20:16.004790+0000","172.21.15.28:6833/3709710897":"2024-09-14T17:20:16.004790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 msgr2=0x7fc76006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 0x7fc76006d0c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fc784159ba0 tx=0x7fc76c009040 comp rx=0 tx=0).stop 2024-09-13T17:20:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 msgr2=0x7fc78414eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7fc774000c00 tx=0x7fc7740046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 shutdown_connections 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc76006ac00 0x7fc76006d0c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc78415a540 0x7fc78414fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7840cabc0 0x7fc78414b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 --2- 172.21.15.28:0/1082403700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc784070cc0 0x7fc78414eb90 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.140+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 >> 172.21.15.28:0/1082403700 conn(0x7fc784069e80 msgr2=0x7fc784155360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.141+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 shutdown_connections 2024-09-13T17:20:25.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.141+0000 7fc78aa00640 1 -- 172.21.15.28:0/1082403700 wait complete. 2024-09-13T17:20:25.158 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:20:25.158 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:25.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.313+0000 7f6371c00640 1 Processor -- start 2024-09-13T17:20:25.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.313+0000 7f6371c00640 1 -- start start 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6371c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c15a540 0x7f636c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6371c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f636c0cabc0 0x7f636c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6371c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636c15e100 con 0x7f636c15edb0 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6371c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f636c0adb00 con 0x7f636c0cabc0 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6371c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c15e3e0 con 0x7f636c15a540 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6362000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f636c0cabc0 0x7f636c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6362000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f636c0cabc0 0x7f636c15e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33080/0 (socket says 172.21.15.28:33080) 2024-09-13T17:20:25.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.314+0000 7f6362000640 1 -- 172.21.15.28:0/2324667563 learned_addr learned my addr 172.21.15.28:0/2324667563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.315+0000 7f636aa00640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c15a540 0x7f636c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.315+0000 7f6362000640 1 -- 172.21.15.28:0/2324667563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c15a540 msgr2=0x7f636c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.315+0000 7f6362000640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c15a540 0x7f636c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.315+0000 7f6362000640 1 -- 172.21.15.28:0/2324667563 >> v1:172.21.15.189:6789/0 conn(0x7f636c15edb0 legacy=0x7f636c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.315+0000 7f6362000640 1 -- 172.21.15.28:0/2324667563 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c06ce80 con 0x7f636c0cabc0 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.316+0000 7f6362000640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c0cabc0 0x7f636c15e870 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f6354009870 tx=0x7f635402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c848960889c515e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.316+0000 7f636a000640 1 -- 172.21.15.28:0/2324667563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354032440 con 0x7f636c0cabc0 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.316+0000 7f636a000640 1 -- 172.21.15.28:0/2324667563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63540325a0 con 0x7f636c0cabc0 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.316+0000 7f636a000640 1 -- 172.21.15.28:0/2324667563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6354032930 con 0x7f636c0cabc0 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 -- 172.21.15.28:0/2324667563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c0cabc0 msgr2=0x7f636c15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c0cabc0 0x7f636c15e870 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f6354009870 tx=0x7f635402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 -- 172.21.15.28:0/2324667563 shutdown_connections 2024-09-13T17:20:25.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c0cabc0 0x7f636c15e870 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 --2- 172.21.15.28:0/2324667563 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c15a540 0x7f636c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 -- 172.21.15.28:0/2324667563 >> 172.21.15.28:0/2324667563 conn(0x7f636c069e80 msgr2=0x7f636c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 -- 172.21.15.28:0/2324667563 shutdown_connections 2024-09-13T17:20:25.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.317+0000 7f6371c00640 1 -- 172.21.15.28:0/2324667563 wait complete. 2024-09-13T17:20:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.319+0000 7f6371c00640 1 Processor -- start 2024-09-13T17:20:25.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.319+0000 7f6371c00640 1 -- start start 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c0cabc0 0x7f636c13baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 0x7f636c153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c153090 con 0x7f636c14eca0 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f636c152db0 con 0x7f636c15a540 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f6371c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636c14fe80 con 0x7f636c0cabc0 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.320+0000 7f636b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33094/0 (socket says 172.21.15.28:33094) 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 -- 172.21.15.28:0/2965469805 learned_addr learned my addr 172.21.15.28:0/2965469805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636aa00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c0cabc0 0x7f636c13baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f6362000640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 0x7f636c153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 -- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c0cabc0 msgr2=0x7f636c13baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c0cabc0 0x7f636c13baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 -- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 msgr2=0x7f636c153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 0x7f636c153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 -- 172.21.15.28:0/2965469805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63540094d0 con 0x7f636c14eca0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f6362000640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 0x7f636c153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.321+0000 7f636b400640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f636400c6b0 tx=0x7f636400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.322+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364003b70 con 0x7f636c14eca0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.322+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364003cf0 con 0x7f636c14eca0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.322+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364003ea0 con 0x7f636c14eca0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.322+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636c150160 con 0x7f636c14eca0 2024-09-13T17:20:25.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.322+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f636c154a20 con 0x7f636c14eca0 2024-09-13T17:20:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.323+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636c14fce0 con 0x7f636c14eca0 2024-09-13T17:20:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.324+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6364005310 con 0x7f636c14eca0 2024-09-13T17:20:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.325+0000 7f6363400640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.325+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6364089110 con 0x7f636c14eca0 2024-09-13T17:20:25.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.325+0000 7f636aa00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.326+0000 7f636aa00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 0x7f634806d0c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f636c1555d0 tx=0x7f635c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.329+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f636c14fce0 con 0x7f636c14eca0 2024-09-13T17:20:25.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.540+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f636c14fce0 con 0x7f636c14eca0 2024-09-13T17:20:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.541+0000 7f6363400640 1 -- 172.21.15.28:0/2965469805 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7f636c14fce0 con 0x7f636c14eca0 2024-09-13T17:20:25.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:25.542 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:25.542 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 msgr2=0x7f634806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 0x7f634806d0c0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f636c1555d0 tx=0x7f635c009190 comp rx=0 tx=0).stop 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 msgr2=0x7f636c1544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f636400c6b0 tx=0x7f636400cb70 comp rx=0 tx=0).stop 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 shutdown_connections 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f636c14eca0 0x7f636c1544e0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f636c15a540 0x7f636c153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 --2- 172.21.15.28:0/2965469805 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f636c0cabc0 0x7f636c13baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 >> 172.21.15.28:0/2965469805 conn(0x7f636c069e80 msgr2=0x7f636c15b7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 shutdown_connections 2024-09-13T17:20:25.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.543+0000 7f6371c00640 1 -- 172.21.15.28:0/2965469805 wait complete. 2024-09-13T17:20:25.560 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:25.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.716+0000 7fa4dbe00640 1 Processor -- start 2024-09-13T17:20:25.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.716+0000 7fa4dbe00640 1 -- start start 2024-09-13T17:20:25.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4dbe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 0x7fa4d406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4d4070cc0 0x7fa4d4071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4dbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d415e100 con 0x7fa4d40cabc0 2024-09-13T17:20:25.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4dbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d40adb00 con 0x7fa4d4070cc0 2024-09-13T17:20:25.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4dbe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d4071670 con 0x7fa4d415a540 2024-09-13T17:20:25.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4d9e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa4d4070cc0 0x7fa4d4071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4d8a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa4d40cabc0 0x7fa4d415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54868/0 (socket says 172.21.15.28:54868) 2024-09-13T17:20:25.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.717+0000 7fa4d8a00640 1 -- 172.21.15.28:0/1724433483 learned_addr learned my addr 172.21.15.28:0/1724433483 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9400640 1 --2- 172.21.15.28:0/1724433483 >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 0x7fa4d406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9e00640 1 -- 172.21.15.28:0/1724433483 >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 msgr2=0x7fa4d406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9e00640 1 --2- 172.21.15.28:0/1724433483 >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 0x7fa4d406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9e00640 1 -- 172.21.15.28:0/1724433483 >> v1:172.21.15.28:6789/0 conn(0x7fa4d40cabc0 legacy=0x7fa4d415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d3e00640 1 -- 172.21.15.28:0/1724433483 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773279175 0 0) 0x7fa4d415e100 con 0x7fa4d40cabc0 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9e00640 1 -- 172.21.15.28:0/1724433483 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d415e3e0 con 0x7fa4d4070cc0 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.718+0000 7fa4d9400640 1 --2- 172.21.15.28:0/1724433483 >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 0x7fa4d406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:25.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.719+0000 7fa4d9e00640 1 --2- 172.21.15.28:0/1724433483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d4070cc0 0x7fa4d4071130 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40b3a00 tx=0x7fa4cc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a78dea7a0e1c860 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.719+0000 7fa4d3e00640 1 -- 172.21.15.28:0/1724433483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc031600 con 0x7fa4d4070cc0 2024-09-13T17:20:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.719+0000 7fa4d3e00640 1 -- 172.21.15.28:0/1724433483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc02d050 con 0x7fa4d4070cc0 2024-09-13T17:20:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.719+0000 7fa4d3e00640 1 -- 172.21.15.28:0/1724433483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc031980 con 0x7fa4d4070cc0 2024-09-13T17:20:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 -- 172.21.15.28:0/1724433483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d4070cc0 msgr2=0x7fa4d4071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/1724433483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d4070cc0 0x7fa4d4071130 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7fa4d40b3a00 tx=0x7fa4cc02a610 comp rx=0 tx=0).stop 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 -- 172.21.15.28:0/1724433483 shutdown_connections 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/1724433483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d4070cc0 0x7fa4d4071130 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/1724433483 >> v2:172.21.15.189:3300/0 conn(0x7fa4d415a540 0x7fa4d406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 -- 172.21.15.28:0/1724433483 >> 172.21.15.28:0/1724433483 conn(0x7fa4d4069e80 msgr2=0x7fa4d406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 -- 172.21.15.28:0/1724433483 shutdown_connections 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.720+0000 7fa4dbe00640 1 -- 172.21.15.28:0/1724433483 wait complete. 2024-09-13T17:20:25.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.722+0000 7fa4dbe00640 1 Processor -- start 2024-09-13T17:20:25.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.722+0000 7fa4dbe00640 1 -- start start 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 0x7fa4d4147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 0x7fa4d4152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40aee30 con 0x7fa4d40cabc0 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d41475c0 con 0x7fa4d415a540 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4dbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d40af660 con 0x7fa4d4070cc0 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33118/0 (socket says 172.21.15.28:33118) 2024-09-13T17:20:25.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 -- 172.21.15.28:0/2035049665 learned_addr learned my addr 172.21.15.28:0/2035049665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d9400640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 0x7fa4d4147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d9e00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 0x7fa4d4152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 -- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 msgr2=0x7fa4d4147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 0x7fa4d4147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 -- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 msgr2=0x7fa4d4152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 0x7fa4d4152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d8a00640 1 -- 172.21.15.28:0/2035049665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc009a70 con 0x7fa4d40cabc0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d9400640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 0x7fa4d4147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.723+0000 7fa4d9e00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 0x7fa4d4152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4d8a00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c4d0 tx=0x7fa4c000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c00074e0 con 0x7fa4d40cabc0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c0007660 con 0x7fa4d40cabc0 2024-09-13T17:20:25.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c0004a20 con 0x7fa4d40cabc0 2024-09-13T17:20:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d41529a0 con 0x7fa4d40cabc0 2024-09-13T17:20:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.724+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d40af010 con 0x7fa4d40cabc0 2024-09-13T17:20:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.725+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d40adf20 con 0x7fa4d40cabc0 2024-09-13T17:20:25.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.726+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa4c0004b80 con 0x7fa4d40cabc0 2024-09-13T17:20:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.727+0000 7fa4d1600640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 0x7fa4b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.727+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa4c0088620 con 0x7fa4d40cabc0 2024-09-13T17:20:25.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.727+0000 7fa4d9400640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 0x7fa4b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:25.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.728+0000 7fa4d9400640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 0x7fa4b406d0c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8006fd0 tx=0x7fa4c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:25.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.731+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa4c0053700 con 0x7fa4d40cabc0 2024-09-13T17:20:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.944+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4d40cb340 con 0x7fa4d40cabc0 2024-09-13T17:20:25.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.944+0000 7fa4d1600640 1 -- 172.21.15.28:0/2035049665 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+3998 (secure 0 0 0) 0x7fa4c0056d20 con 0x7fa4d40cabc0 2024-09-13T17:20:25.945 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:25.946 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":142,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:19:28.714008+0000","modified":"2024-09-13T17:20:19.021956+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":168,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10038},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10038":{"gid":10038,"name":"a","rank":0,"incarnation":139,"state":"up:active","state_seq":17,"addr":"172.21.15.28:6834/1290280670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1290280670},{"type":"v1","addr":"172.21.15.28:6834","nonce":1290280670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2024-09-13T17:20:25.946 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 142 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 msgr2=0x7fa4b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 0x7fa4b406d0c0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8006fd0 tx=0x7fa4c8008040 comp rx=0 tx=0).stop 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 msgr2=0x7fa4d4151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7fa4c000c4d0 tx=0x7fa4c000c990 comp rx=0 tx=0).stop 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 shutdown_connections 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa4b406ac00 0x7fa4b406d0c0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4d415a540 0x7fa4d4152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4d40cabc0 0x7fa4d4151f20 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 --2- 172.21.15.28:0/2035049665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa4d4070cc0 0x7fa4d4147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 >> 172.21.15.28:0/2035049665 conn(0x7fa4d4069e80 msgr2=0x7fa4d40ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 shutdown_connections 2024-09-13T17:20:25.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:25.947+0000 7fa4dbe00640 1 -- 172.21.15.28:0/2035049665 wait complete. 2024-09-13T17:20:25.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.119+0000 7f28e8e00640 1 Processor -- start 2024-09-13T17:20:26.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.119+0000 7f28e8e00640 1 -- start start 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28e40cabc0 0x7f28e415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 0x7f28e4070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e40adb00 con 0x7f28e40cabc0 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e415e100 con 0x7f28e415edb0 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e8e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e415e3e0 con 0x7f28e415a540 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28e40cabc0 0x7f28e415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28d9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f28e40cabc0 0x7f28e415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33122/0 (socket says 172.21.15.28:33122) 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28d9400640 1 -- 172.21.15.28:0/1893537385 learned_addr learned my addr 172.21.15.28:0/1893537385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.120+0000 7f28e1e00640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28e2800640 1 --2- 172.21.15.28:0/1893537385 >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 0x7f28e4070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28d9400640 1 -- 172.21.15.28:0/1893537385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 msgr2=0x7f28e406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28d9400640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28d9400640 1 -- 172.21.15.28:0/1893537385 >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 msgr2=0x7f28e4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28d9400640 1 --2- 172.21.15.28:0/1893537385 >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 0x7f28e4070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28d9400640 1 -- 172.21.15.28:0/1893537385 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e406ce80 con 0x7f28e40cabc0 2024-09-13T17:20:26.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.121+0000 7f28e2800640 1 --2- 172.21.15.28:0/1893537385 >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 0x7f28e4070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28d9400640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e415e870 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f28e40ca980 tx=0x7f28cc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d0c2ea761713d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28e1400640 1 -- 172.21.15.28:0/1893537385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc02ae90 con 0x7f28e40cabc0 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28e1400640 1 -- 172.21.15.28:0/1893537385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28cc032440 con 0x7f28e40cabc0 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28e1400640 1 -- 172.21.15.28:0/1893537385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28cc0327d0 con 0x7f28e40cabc0 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28e8e00640 1 -- 172.21.15.28:0/1893537385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 msgr2=0x7f28e415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.122+0000 7f28e8e00640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e415e870 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f28e40ca980 tx=0x7f28cc02a140 comp rx=0 tx=0).stop 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 -- 172.21.15.28:0/1893537385 shutdown_connections 2024-09-13T17:20:26.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e415e870 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 --2- 172.21.15.28:0/1893537385 >> v2:172.21.15.189:3300/0 conn(0x7f28e415edb0 0x7f28e4070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 --2- 172.21.15.28:0/1893537385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 -- 172.21.15.28:0/1893537385 >> 172.21.15.28:0/1893537385 conn(0x7f28e4069e80 msgr2=0x7f28e406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 -- 172.21.15.28:0/1893537385 shutdown_connections 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.123+0000 7f28e8e00640 1 -- 172.21.15.28:0/1893537385 wait complete. 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.124+0000 7f28e8e00640 1 Processor -- start 2024-09-13T17:20:26.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.124+0000 7f28e8e00640 1 -- start start 2024-09-13T17:20:26.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e4150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e41508d0 0x7f28e40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4152e10 con 0x7f28e40cabc0 2024-09-13T17:20:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e41f6080 con 0x7f28e41508d0 2024-09-13T17:20:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.125+0000 7f28e8e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e41f6360 con 0x7f28e415a540 2024-09-13T17:20:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33138/0 (socket says 172.21.15.28:33138) 2024-09-13T17:20:26.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 -- 172.21.15.28:0/794687137 learned_addr learned my addr 172.21.15.28:0/794687137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e2800640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e41508d0 0x7f28e40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28d9400640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e4150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 -- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 msgr2=0x7f28e4150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e4150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 -- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e41508d0 msgr2=0x7f28e40ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e41508d0 0x7f28e40ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 -- 172.21.15.28:0/794687137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28cc0094d0 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28e1e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e9b0 tx=0x7f28d400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.126+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4009670 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.127+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d400c9a0 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.127+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d400cd50 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.127+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e41f6640 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.127+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e41f6c00 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.128+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e406d6d0 con 0x7f28e40cabc0 2024-09-13T17:20:26.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.129+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f28d4013470 con 0x7f28e40cabc0 2024-09-13T17:20:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.129+0000 7f28daa00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 0x7f28c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.130+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f28d4089660 con 0x7f28e40cabc0 2024-09-13T17:20:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.130+0000 7f28d9400640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 0x7f28c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.131+0000 7f28d9400640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 0x7f28c006d0c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f28cc034000 tx=0x7f28cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.134+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f28d40547d0 con 0x7f28e40cabc0 2024-09-13T17:20:26.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.313+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28e41559b0 con 0x7f28e40cabc0 2024-09-13T17:20:26.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.314+0000 7f28daa00640 1 -- 172.21.15.28:0/794687137 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v168) v1 ==== 74+0+16700 (secure 0 0 0) 0x7f28d4057df0 con 0x7f28e40cabc0 2024-09-13T17:20:26.316 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:26.316 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":168,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:16.004870+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:19:26.637775+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2024-09-13T17:19:27.700541+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"167","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3709710897":"2024-09-14T17:20:16.004790+0000","172.21.15.28:6833/3709710897":"2024-09-14T17:20:16.004790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.316+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 msgr2=0x7f28c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.316+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 0x7f28c006d0c0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f28cc034000 tx=0x7f28cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 msgr2=0x7f28e41528d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f28d400e9b0 tx=0x7f28d400ee70 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 shutdown_connections 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f28c006ac00 0x7f28c006d0c0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f28e41508d0 0x7f28e40ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f28e415a540 0x7f28e4150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 --2- 172.21.15.28:0/794687137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28e40cabc0 0x7f28e41528d0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 >> 172.21.15.28:0/794687137 conn(0x7f28e4069e80 msgr2=0x7f28e415b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:26.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 shutdown_connections 2024-09-13T17:20:26.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.317+0000 7f28e8e00640 1 -- 172.21.15.28:0/794687137 wait complete. 2024-09-13T17:20:26.334 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:20:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.489+0000 7fb06e200640 1 Processor -- start 2024-09-13T17:20:26.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.489+0000 7fb06e200640 1 -- start start 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.490+0000 7fb06e200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb06815a780 0x7fb068070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.490+0000 7fb06e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815ab60 0x7fb0680712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.490+0000 7fb06e200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06806c8b0 con 0x7fb06806bb40 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.490+0000 7fb06e200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06806cb90 con 0x7fb06815a780 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.490+0000 7fb06e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680cc980 con 0x7fb06815ab60 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb06815a780 0x7fb068070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb06806bb40 0x7fb06806bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54872/0 (socket says 172.21.15.28:54872) 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb06815a780 0x7fb068070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33152/0 (socket says 172.21.15.28:33152) 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067e00640 1 -- 172.21.15.28:0/158463078 learned_addr learned my addr 172.21.15.28:0/158463078 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb05ea00640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815ab60 0x7fb0680712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 -- 172.21.15.28:0/158463078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815ab60 msgr2=0x7fb0680712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815ab60 0x7fb0680712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 -- 172.21.15.28:0/158463078 >> v1:172.21.15.28:6789/0 conn(0x7fb06806bb40 legacy=0x7fb06806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.491+0000 7fb067400640 1 -- 172.21.15.28:0/158463078 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0680ccc60 con 0x7fb06815a780 2024-09-13T17:20:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.492+0000 7fb067400640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06815a780 0x7fb068070d80 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fb0580097a0 tx=0x7fb05802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79efde657e25ff29 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb066a00640 1 -- 172.21.15.28:0/158463078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058032440 con 0x7fb06815a780 2024-09-13T17:20:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb066a00640 1 -- 172.21.15.28:0/158463078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0580325a0 con 0x7fb06815a780 2024-09-13T17:20:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb066a00640 1 -- 172.21.15.28:0/158463078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058032930 con 0x7fb06815a780 2024-09-13T17:20:26.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 -- 172.21.15.28:0/158463078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06815a780 msgr2=0x7fb068070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06815a780 0x7fb068070d80 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fb0580097a0 tx=0x7fb05802a180 comp rx=0 tx=0).stop 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 -- 172.21.15.28:0/158463078 shutdown_connections 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815ab60 0x7fb0680712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 --2- 172.21.15.28:0/158463078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06815a780 0x7fb068070d80 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 -- 172.21.15.28:0/158463078 >> 172.21.15.28:0/158463078 conn(0x7fb0680b0270 msgr2=0x7fb0680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.493+0000 7fb06e200640 1 -- 172.21.15.28:0/158463078 shutdown_connections 2024-09-13T17:20:26.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.494+0000 7fb06e200640 1 -- 172.21.15.28:0/158463078 wait complete. 2024-09-13T17:20:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.495+0000 7fb06e200640 1 Processor -- start 2024-09-13T17:20:26.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.495+0000 7fb06e200640 1 -- start start 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06806bb40 0x7fb06814e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815a780 0x7fb068155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06815ab60 0x7fb0680ab2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680b1670 con 0x7fb06806bb40 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680b1950 con 0x7fb06815ab60 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb06e200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0680b1c30 con 0x7fb06815a780 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb067400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06806bb40 0x7fb06814e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb067400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06806bb40 0x7fb06814e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33168/0 (socket says 172.21.15.28:33168) 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.496+0000 7fb067400640 1 -- 172.21.15.28:0/3906207104 learned_addr learned my addr 172.21.15.28:0/3906207104 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb05ea00640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815a780 0x7fb068155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 -- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815a780 msgr2=0x7fb068155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb06815a780 0x7fb068155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 -- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06815ab60 msgr2=0x7fb0680ab2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb06815ab60 0x7fb0680ab2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 -- 172.21.15.28:0/3906207104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009400 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb067400640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb06806bb40 0x7fb06814e0e0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fb0580098c0 tx=0x7fb0580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058030b80 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb058032e90 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.497+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058006e10 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.498+0000 7fb06e200640 1 -- 172.21.15.28:0/3906207104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680b1f10 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.498+0000 7fb06e200640 1 -- 172.21.15.28:0/3906207104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb06814e620 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.499+0000 7fb06e200640 1 -- 172.21.15.28:0/3906207104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06806bfd0 con 0x7fb06806bb40 2024-09-13T17:20:26.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.500+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb05803c400 con 0x7fb06806bb40 2024-09-13T17:20:26.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.500+0000 7fb05fe00640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04406ac00 0x7fb04406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.501+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb0580aa190 con 0x7fb06806bb40 2024-09-13T17:20:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.501+0000 7fb05ea00640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04406ac00 0x7fb04406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.502+0000 7fb05ea00640 1 --2- 172.21.15.28:0/3906207104 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb04406ac00 0x7fb04406d0c0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb0681f67b0 tx=0x7fb050008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.505+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb058075300 con 0x7fb06806bb40 2024-09-13T17:20:26.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.716+0000 7fb06e200640 1 -- 172.21.15.28:0/3906207104 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb068159c80 con 0x7fb06806bb40 2024-09-13T17:20:26.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.717+0000 7fb05fe00640 1 -- 172.21.15.28:0/3906207104 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=-1 MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it v143) v1 ==== 316+0+0 (secure 0 0 0) 0x7fb058078b00 con 0x7fb06806bb40 2024-09-13T17:20:26.718 INFO:teuthology.orchestra.run.smithi028.stderr:Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it 2024-09-13T17:20:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:20:26.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs --yes-i-really-mean-it 2024-09-13T17:20:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.885+0000 7ff590600640 1 Processor -- start 2024-09-13T17:20:26.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.885+0000 7ff590600640 1 -- start start 2024-09-13T17:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff590600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff58806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff590600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff58806f3f0 0x7ff588070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff590600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58806f920 con 0x7ff5880add00 2024-09-13T17:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff590600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58815b290 con 0x7ff58806f3f0 2024-09-13T17:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff590600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58815b570 con 0x7ff588159770 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.886+0000 7ff58d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff58806f3f0 0x7ff588070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff58806f3f0 0x7ff588070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33180/0 (socket says 172.21.15.28:33180) 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 -- 172.21.15.28:0/1981189218 learned_addr learned my addr 172.21.15.28:0/1981189218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58dc00640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff58806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 -- 172.21.15.28:0/1981189218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 msgr2=0x7ff58806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff58806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 -- 172.21.15.28:0/1981189218 >> v1:172.21.15.189:6789/0 conn(0x7ff5880add00 legacy=0x7ff588159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58d200640 1 -- 172.21.15.28:0/1981189218 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58815b850 con 0x7ff58806f3f0 2024-09-13T17:20:26.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.887+0000 7ff58dc00640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff58806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.888+0000 7ff58d200640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58806f3f0 0x7ff588070260 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7ff574009870 tx=0x7ff57402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7018ac7b8766c96d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.888+0000 7ff587e00640 1 -- 172.21.15.28:0/1981189218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574032440 con 0x7ff58806f3f0 2024-09-13T17:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.888+0000 7ff587e00640 1 -- 172.21.15.28:0/1981189218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5740325a0 con 0x7ff58806f3f0 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.888+0000 7ff587e00640 1 -- 172.21.15.28:0/1981189218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574032930 con 0x7ff58806f3f0 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 -- 172.21.15.28:0/1981189218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58806f3f0 msgr2=0x7ff588070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58806f3f0 0x7ff588070260 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7ff574009870 tx=0x7ff57402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 -- 172.21.15.28:0/1981189218 shutdown_connections 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff58806f3f0 0x7ff588070260 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 --2- 172.21.15.28:0/1981189218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff58806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 -- 172.21.15.28:0/1981189218 >> 172.21.15.28:0/1981189218 conn(0x7ff5880b0270 msgr2=0x7ff5880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 -- 172.21.15.28:0/1981189218 shutdown_connections 2024-09-13T17:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.889+0000 7ff590600640 1 -- 172.21.15.28:0/1981189218 wait complete. 2024-09-13T17:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.891+0000 7ff590600640 1 Processor -- start 2024-09-13T17:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.891+0000 7ff590600640 1 -- start start 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 0x7ff588155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff5880cc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880cbd50 con 0x7ff5880add00 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588151480 con 0x7ff58806f3f0 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff590600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5881511a0 con 0x7ff588159770 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33190/0 (socket says 172.21.15.28:33190) 2024-09-13T17:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 -- 172.21.15.28:0/133436242 learned_addr learned my addr 172.21.15.28:0/133436242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:26.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58dc00640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 0x7ff588155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 -- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 msgr2=0x7ff5880cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58e600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff5880cc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff5880cc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 -- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 msgr2=0x7ff588155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 0x7ff588155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58d200640 1 -- 172.21.15.28:0/133436242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740094d0 con 0x7ff5880add00 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58dc00640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 0x7ff588155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.892+0000 7ff58e600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff5880cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff58d200640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7ff57402a200 tx=0x7ff5740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574032440 con 0x7ff5880add00 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574030b80 con 0x7ff5880add00 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57403d440 con 0x7ff5880add00 2024-09-13T17:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5880cb700 con 0x7ff5880add00 2024-09-13T17:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.893+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5881548d0 con 0x7ff5880add00 2024-09-13T17:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.894+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5880cbec0 con 0x7ff5880add00 2024-09-13T17:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.895+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff574039040 con 0x7ff5880add00 2024-09-13T17:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.896+0000 7ff585600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 0x7ff56806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.896+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff5740a9f40 con 0x7ff5880add00 2024-09-13T17:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.897+0000 7ff58dc00640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 0x7ff56806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.897+0000 7ff58dc00640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 0x7ff56806d0c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff57c006fd0 tx=0x7ff57c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:26.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:26.900+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff5740750c0 con 0x7ff5880add00 2024-09-13T17:20:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.120+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff58806fd20 con 0x7ff5880add00 2024-09-13T17:20:27.122 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:20:27.121+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.130+0000 7ff585600640 1 -- 172.21.15.28:0/133436242 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v144) v1 ==== 191+0+0 (secure 0 0 0) 0x7ff5740786e0 con 0x7ff5880add00 2024-09-13T17:20:27.131 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 msgr2=0x7ff56806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 0x7ff56806d0c0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7ff57c006fd0 tx=0x7ff57c008040 comp rx=0 tx=0).stop 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 msgr2=0x7ff588151a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7ff57402a200 tx=0x7ff5740046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 shutdown_connections 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff56806ac00 0x7ff56806d0c0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff588159770 0x7ff5880cc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff5880add00 0x7ff588151a30 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 --2- 172.21.15.28:0/133436242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff58806f3f0 0x7ff588155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 >> 172.21.15.28:0/133436242 conn(0x7ff5880b0270 msgr2=0x7ff58815a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 shutdown_connections 2024-09-13T17:20:27.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.133+0000 7ff590600640 1 -- 172.21.15.28:0/133436242 wait complete. 2024-09-13T17:20:27.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:20:27.198 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:20:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.310+0000 7f89c2000640 1 Processor -- start 2024-09-13T17:20:27.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.310+0000 7f89c2000640 1 -- start start 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.311+0000 7f89c2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89bc159770 0x7f89bc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.311+0000 7f89c2000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 0x7f89bc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.311+0000 7f89c2000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89bc06f920 con 0x7f89bc06f3f0 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.311+0000 7f89c2000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc15b290 con 0x7f89bc159770 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.311+0000 7f89c2000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89bc15b570 con 0x7f89bc15cd80 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89bc159770 0x7f89bc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89b3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89bc06f3f0 0x7f89bc159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54878/0 (socket says 172.21.15.28:54878) 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89b3e00640 1 -- 172.21.15.28:0/3058778194 learned_addr learned my addr 172.21.15.28:0/3058778194 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bbe00640 1 --2- 172.21.15.28:0/3058778194 >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 0x7f89bc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bb400640 1 -- 172.21.15.28:0/3058778194 >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 msgr2=0x7f89bc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bb400640 1 --2- 172.21.15.28:0/3058778194 >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 0x7f89bc070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bb400640 1 -- 172.21.15.28:0/3058778194 >> v1:172.21.15.28:6789/0 conn(0x7f89bc06f3f0 legacy=0x7f89bc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bb400640 1 -- 172.21.15.28:0/3058778194 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89bc15b850 con 0x7f89bc159770 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89baa00640 1 -- 172.21.15.28:0/3058778194 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269355227 0 0) 0x7f89bc06f920 con 0x7f89bc06f3f0 2024-09-13T17:20:27.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.312+0000 7f89bbe00640 1 --2- 172.21.15.28:0/3058778194 >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 0x7f89bc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.313+0000 7f89bb400640 1 --2- 172.21.15.28:0/3058778194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc06fd20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06f380 tx=0x7f89ac02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=904ed0018b3b4480 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89baa00640 1 -- 172.21.15.28:0/3058778194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac02ae60 con 0x7f89bc159770 2024-09-13T17:20:27.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89baa00640 1 -- 172.21.15.28:0/3058778194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89ac032440 con 0x7f89bc159770 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89baa00640 1 -- 172.21.15.28:0/3058778194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89ac0327d0 con 0x7f89bc159770 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 -- 172.21.15.28:0/3058778194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 msgr2=0x7f89bc06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 --2- 172.21.15.28:0/3058778194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc06fd20 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f89bc06f380 tx=0x7f89ac02a110 comp rx=0 tx=0).stop 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 -- 172.21.15.28:0/3058778194 shutdown_connections 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 --2- 172.21.15.28:0/3058778194 >> v2:172.21.15.189:3300/0 conn(0x7f89bc15cd80 0x7f89bc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 --2- 172.21.15.28:0/3058778194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc06fd20 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 -- 172.21.15.28:0/3058778194 >> 172.21.15.28:0/3058778194 conn(0x7f89bc0b0270 msgr2=0x7f89bc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 -- 172.21.15.28:0/3058778194 shutdown_connections 2024-09-13T17:20:27.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.314+0000 7f89c2000640 1 -- 172.21.15.28:0/3058778194 wait complete. 2024-09-13T17:20:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.316+0000 7f89c2000640 1 Processor -- start 2024-09-13T17:20:27.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.316+0000 7f89c2000640 1 -- start start 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89bc06f3f0 0x7f89bc154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc15cd80 0x7f89bc1508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0cde60 con 0x7f89bc159770 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc150470 con 0x7f89bc15cd80 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.317+0000 7f89c2000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89bc0cb680 con 0x7f89bc06f3f0 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33234/0 (socket says 172.21.15.28:33234) 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 -- 172.21.15.28:0/2768753262 learned_addr learned my addr 172.21.15.28:0/2768753262 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:27.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89bbe00640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc15cd80 0x7f89bc1508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89bb400640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89bc06f3f0 0x7f89bc154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 -- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89bc06f3f0 msgr2=0x7f89bc154840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89bc06f3f0 0x7f89bc154840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 -- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc15cd80 msgr2=0x7f89bc1508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc15cd80 0x7f89bc1508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:27.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 -- 172.21.15.28:0/2768753262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac009400 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.318+0000 7f89b3e00640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c460 tx=0x7f89a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.319+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40074e0 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.319+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a4007660 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.319+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4004a20 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.319+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89bc154f40 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.319+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89bc155500 con 0x7f89bc159770 2024-09-13T17:20:27.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.320+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89bc06fd20 con 0x7f89bc159770 2024-09-13T17:20:27.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.321+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89a4004b80 con 0x7f89bc159770 2024-09-13T17:20:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.322+0000 7f89b3400640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 0x7f899806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.322+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f89a4088ff0 con 0x7f89bc159770 2024-09-13T17:20:27.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.322+0000 7f89bb400640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 0x7f899806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:27.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.323+0000 7f89bb400640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 0x7f899806d110 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f89ac034000 tx=0x7f89ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:27.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.326+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f89a40bc9c0 con 0x7f89bc159770 2024-09-13T17:20:27.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:27.543+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f89bc0cb7c0 con 0x7f89bc159770 2024-09-13T17:20:28.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.131+0000 7f89b3400640 1 -- 172.21.15.28:0/2768753262 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v145) v1 ==== 104+0+0 (secure 0 0 0) 0x7f89a4054210 con 0x7f89bc159770 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.134+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 msgr2=0x7f899806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.134+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 0x7f899806d110 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f89ac034000 tx=0x7f89ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.134+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 msgr2=0x7f89bc14f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.134+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f89a400c460 tx=0x7f89a400c920 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 shutdown_connections 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f899806ac50 0x7f899806d110 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89bc15cd80 0x7f89bc1508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89bc159770 0x7f89bc14f980 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 --2- 172.21.15.28:0/2768753262 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89bc06f3f0 0x7f89bc154840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 >> 172.21.15.28:0/2768753262 conn(0x7f89bc0b0270 msgr2=0x7f89bc0ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:28.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 shutdown_connections 2024-09-13T17:20:28.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.135+0000 7f89c2000640 1 -- 172.21.15.28:0/2768753262 wait complete. 2024-09-13T17:20:28.155 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:20:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.315+0000 7f5ef5800640 1 Processor -- start 2024-09-13T17:20:28.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.315+0000 7f5ef5800640 1 -- start start 2024-09-13T17:20:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef5800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ef00b1920 0x7f5ef00b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00b87e0 con 0x7f5ef0069e80 2024-09-13T17:20:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef5800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ef00d0ca0 con 0x7f5ef015c920 2024-09-13T17:20:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef5800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ef00b7ca0 con 0x7f5ef00b1920 2024-09-13T17:20:28.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ee7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ef00b1920 0x7f5ef00b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:28.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef4e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ef0069e80 0x7f5ef006a2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54890/0 (socket says 172.21.15.28:54890) 2024-09-13T17:20:28.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.316+0000 7f5ef4e00640 1 -- 172.21.15.28:0/952740556 learned_addr learned my addr 172.21.15.28:0/952740556 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:28.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.317+0000 7f5ee7e00640 1 -- 172.21.15.28:0/952740556 >> v1:172.21.15.189:6789/0 conn(0x7f5ef015c920 legacy=0x7f5ef00b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.317+0000 7f5ee7e00640 1 -- 172.21.15.28:0/952740556 >> v1:172.21.15.28:6789/0 conn(0x7f5ef0069e80 legacy=0x7f5ef006a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.317+0000 7f5ee7e00640 1 -- 172.21.15.28:0/952740556 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef00d0f40 con 0x7f5ef00b1920 2024-09-13T17:20:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.318+0000 7f5ee7e00640 1 --2- 172.21.15.28:0/952740556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef00b1920 0x7f5ef00b5830 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80097a0 tx=0x7f5ed802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4052affbe7f5d9c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.318+0000 7f5eef400640 1 -- 172.21.15.28:0/952740556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8032440 con 0x7f5ef00b1920 2024-09-13T17:20:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.318+0000 7f5eef400640 1 -- 172.21.15.28:0/952740556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ed80325a0 con 0x7f5ef00b1920 2024-09-13T17:20:28.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.318+0000 7f5eef400640 1 -- 172.21.15.28:0/952740556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ed8032930 con 0x7f5ef00b1920 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 -- 172.21.15.28:0/952740556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef00b1920 msgr2=0x7f5ef00b5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 --2- 172.21.15.28:0/952740556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef00b1920 0x7f5ef00b5830 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f5ed80097a0 tx=0x7f5ed802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 -- 172.21.15.28:0/952740556 shutdown_connections 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 --2- 172.21.15.28:0/952740556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef00b1920 0x7f5ef00b5830 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 -- 172.21.15.28:0/952740556 >> 172.21.15.28:0/952740556 conn(0x7f5ef00708c0 msgr2=0x7f5ef0070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 -- 172.21.15.28:0/952740556 shutdown_connections 2024-09-13T17:20:28.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.319+0000 7f5ef5800640 1 -- 172.21.15.28:0/952740556 wait complete. 2024-09-13T17:20:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.320+0000 7f5ef5800640 1 Processor -- start 2024-09-13T17:20:28.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.320+0000 7f5ef5800640 1 -- start start 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ef0069e80 0x7f5ef014ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef00b1920 0x7f5ef00cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00b1e50 con 0x7f5ef015c920 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00cbdd0 con 0x7f5ef00b1920 2024-09-13T17:20:28.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.321+0000 7f5ef5800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef00cc0b0 con 0x7f5ef0069e80 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33262/0 (socket says 172.21.15.28:33262) 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 -- 172.21.15.28:0/3163236013 learned_addr learned my addr 172.21.15.28:0/3163236013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ee7e00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef00b1920 0x7f5ef00cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5eefe00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ef0069e80 0x7f5ef014ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 -- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ef0069e80 msgr2=0x7f5ef014ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ef0069e80 0x7f5ef014ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 -- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef00b1920 msgr2=0x7f5ef00cb380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef00b1920 0x7f5ef00cb380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 -- 172.21.15.28:0/3163236013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed8009400 con 0x7f5ef015c920 2024-09-13T17:20:28.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.322+0000 7f5ef4e00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800ca80 tx=0x7f5ee800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:28.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.323+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee800ac30 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.323+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ee800ad90 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.323+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef00b20a0 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.323+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ee8012600 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.323+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef01f3630 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.324+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef00b25c0 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.325+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5ee8007070 con 0x7f5ef015c920 2024-09-13T17:20:28.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.326+0000 7f5eecc00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 0x7f5ecc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:28.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.326+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f5ee80892a0 con 0x7f5ef015c920 2024-09-13T17:20:28.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.326+0000 7f5eefe00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 0x7f5ecc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:28.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.327+0000 7f5eefe00640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 0x7f5ecc06d0c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00b3f60 tx=0x7f5ee0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:28.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.330+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5ef00b25c0 con 0x7f5ef015c920 2024-09-13T17:20:28.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:28.505+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5ef00b25c0 con 0x7f5ef015c920 2024-09-13T17:20:29.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.134+0000 7f5eecc00640 1 -- 172.21.15.28:0/3163236013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v170) v1 ==== 181+0+0 (secure 0 0 0) 0x7f5ef00b25c0 con 0x7f5ef015c920 2024-09-13T17:20:29.137 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:20:29.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.138+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 msgr2=0x7f5ecc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:29.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 0x7f5ecc06d0c0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f5ef00b3f60 tx=0x7f5ee0009040 comp rx=0 tx=0).stop 2024-09-13T17:20:29.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 msgr2=0x7f5ef01f5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:29.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f5ee800ca80 tx=0x7f5ee800cf40 comp rx=0 tx=0).stop 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 shutdown_connections 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ecc06ac00 0x7f5ecc06d0c0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ef015c920 0x7f5ef01f5a60 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ef00b1920 0x7f5ef00cb380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 --2- 172.21.15.28:0/3163236013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ef0069e80 0x7f5ef014ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 >> 172.21.15.28:0/3163236013 conn(0x7f5ef00708c0 msgr2=0x7f5ef015d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 shutdown_connections 2024-09-13T17:20:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.139+0000 7f5ef5800640 1 -- 172.21.15.28:0/3163236013 wait complete. 2024-09-13T17:20:29.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:20:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.320+0000 7f9713000640 1 Processor -- start 2024-09-13T17:20:29.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.320+0000 7f9713000640 1 -- start start 2024-09-13T17:20:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.321+0000 7f9713000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970c159590 0x7f970c070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f9713000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f970c159970 0x7f970c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f9713000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f970c15a4c0 con 0x7f970c0b2630 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f9713000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c0b2ae0 con 0x7f970c159590 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f9713000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f970c071580 con 0x7f970c159970 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970c159590 0x7f970c070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f970c159590 0x7f970c070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33264/0 (socket says 172.21.15.28:33264) 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 -- 172.21.15.28:0/2741485241 learned_addr learned my addr 172.21.15.28:0/2741485241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f9703400640 1 --2- 172.21.15.28:0/2741485241 >> v2:172.21.15.189:3300/0 conn(0x7f970c159970 0x7f970c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 -- 172.21.15.28:0/2741485241 >> v2:172.21.15.189:3300/0 conn(0x7f970c159970 msgr2=0x7f970c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 --2- 172.21.15.28:0/2741485241 >> v2:172.21.15.189:3300/0 conn(0x7f970c159970 0x7f970c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 -- 172.21.15.28:0/2741485241 >> v1:172.21.15.189:6789/0 conn(0x7f970c0b2630 legacy=0x7f970c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:29.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.322+0000 7f970be00640 1 -- 172.21.15.28:0/2741485241 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c159e70 con 0x7f970c159590 2024-09-13T17:20:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.323+0000 7f970be00640 1 --2- 172.21.15.28:0/2741485241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159590 0x7f970c070c80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cf5e0 tx=0x7f96fc02c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3dd25ca287e7ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:29.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f970b400640 1 -- 172.21.15.28:0/2741485241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc032640 con 0x7f970c159590 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f970b400640 1 -- 172.21.15.28:0/2741485241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96fc0327a0 con 0x7f970c159590 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f970b400640 1 -- 172.21.15.28:0/2741485241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96fc032b30 con 0x7f970c159590 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 -- 172.21.15.28:0/2741485241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159590 msgr2=0x7f970c070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 --2- 172.21.15.28:0/2741485241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159590 0x7f970c070c80 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7f970c0cf5e0 tx=0x7f96fc02c940 comp rx=0 tx=0).stop 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 -- 172.21.15.28:0/2741485241 shutdown_connections 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 --2- 172.21.15.28:0/2741485241 >> v2:172.21.15.189:3300/0 conn(0x7f970c159970 0x7f970c0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 --2- 172.21.15.28:0/2741485241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159590 0x7f970c070c80 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 -- 172.21.15.28:0/2741485241 >> 172.21.15.28:0/2741485241 conn(0x7f970c069e80 msgr2=0x7f970c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 -- 172.21.15.28:0/2741485241 shutdown_connections 2024-09-13T17:20:29.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.324+0000 7f9713000640 1 -- 172.21.15.28:0/2741485241 wait complete. 2024-09-13T17:20:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.326+0000 7f9713000640 1 Processor -- start 2024-09-13T17:20:29.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.326+0000 7f9713000640 1 -- start start 2024-09-13T17:20:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c0b2630 0x7f970c1ee8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970c159590 0x7f970c155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c1edf60 con 0x7f970c159970 2024-09-13T17:20:29.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0cc630 con 0x7f970c0b2630 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9713000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970c0cc910 con 0x7f970c159590 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9711000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9711000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33270/0 (socket says 172.21.15.28:33270) 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9711000640 1 -- 172.21.15.28:0/404307786 learned_addr learned my addr 172.21.15.28:0/404307786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f970be00640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c0b2630 0x7f970c1ee8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.327+0000 7f9703400640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970c159590 0x7f970c155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 -- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970c159590 msgr2=0x7f970c155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970c159590 0x7f970c155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 -- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c0b2630 msgr2=0x7f970c1ee8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c0b2630 0x7f970c1ee8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:29.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 -- 172.21.15.28:0/404307786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96fc00b770 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9711000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7a0 tx=0x7f970400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704003b70 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9704003cf0 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704003ea0 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.328+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f970c0ccbf0 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.329+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f970c0cd3f0 con 0x7f970c159970 2024-09-13T17:20:29.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.329+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f970c0cd110 con 0x7f970c159970 2024-09-13T17:20:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.330+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f970400aa90 con 0x7f970c159970 2024-09-13T17:20:29.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.331+0000 7f9708c00640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 0x7f96e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.331+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(170..170 src has 1..170) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f9704088fe0 con 0x7f970c159970 2024-09-13T17:20:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.332+0000 7f970be00640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 0x7f96e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:29.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.332+0000 7f970be00640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 0x7f96e806d0c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f96fc037000 tx=0x7f96fc03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:29.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.335+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f970c0cd110 con 0x7f970c159970 2024-09-13T17:20:29.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:29.518+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f970c0cd110 con 0x7f970c159970 2024-09-13T17:20:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.135+0000 7f9708c00640 1 -- 172.21.15.28:0/404307786 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v171) v1 ==== 169+0+0 (secure 0 0 0) 0x7f970c0cd110 con 0x7f970c159970 2024-09-13T17:20:30.138 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:20:30.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 msgr2=0x7f96e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 0x7f96e806d0c0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f96fc037000 tx=0x7f96fc03d040 comp rx=0 tx=0).stop 2024-09-13T17:20:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 msgr2=0x7f970c1557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f970400c7a0 tx=0x7f970400cc60 comp rx=0 tx=0).stop 2024-09-13T17:20:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 shutdown_connections 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f96e806ac00 0x7f96e806d0c0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f970c159970 0x7f970c1557a0 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f970c159590 0x7f970c155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 --2- 172.21.15.28:0/404307786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f970c0b2630 0x7f970c1ee8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.140+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 >> 172.21.15.28:0/404307786 conn(0x7f970c069e80 msgr2=0x7f970c15c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.141+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 shutdown_connections 2024-09-13T17:20:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.141+0000 7f9713000640 1 -- 172.21.15.28:0/404307786 wait complete. 2024-09-13T17:20:30.172 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:20:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.331+0000 7f27f1400640 1 Processor -- start 2024-09-13T17:20:30.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.332+0000 7f27f1400640 1 -- start start 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27f1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27ec070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27f1400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec06c8b0 con 0x7f27ec06c250 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27f1400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec06cb90 con 0x7f27ec15ab60 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27f1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec06c6a0 con 0x7f27ec15a780 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27eb200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f27ec06c250 0x7f27ec0cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54894/0 (socket says 172.21.15.28:54894) 2024-09-13T17:20:30.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27eb200640 1 -- 172.21.15.28:0/427461232 learned_addr learned my addr 172.21.15.28:0/427461232 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.333+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27ec070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306693005 0 0) 0x7f27ec06c8b0 con 0x7f27ec06c250 2024-09-13T17:20:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d0003520 con 0x7f27ec06c250 2024-09-13T17:20:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853764048 0 0) 0x7f27ec06cb90 con 0x7f27ec15ab60 2024-09-13T17:20:30.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27ec06c8b0 con 0x7f27ec15ab60 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104319582 0 0) 0x7f27d0003520 con 0x7f27ec06c250 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 msgr2=0x7f27ec070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27ec070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 >> v1:172.21.15.189:6789/0 conn(0x7f27ec15ab60 legacy=0x7f27ec06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec0cc980 con 0x7f27ec06c250 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f27e40019f0 con 0x7f27ec06c250 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27ec070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 >> v1:172.21.15.28:6789/0 conn(0x7f27ec06c250 legacy=0x7f27ec0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d0003b30 0x7f27d0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27d00047c0 0x7f27d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec06c6a0 con 0x7f27d0003b30 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d000ba30 con 0x7f27d00047c0 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d000bbc0 con 0x7f27ec15a780 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d0003b30 0x7f27d0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.334+0000 7f27e3e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27d00047c0 0x7f27d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27eb200640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27ea800640 1 -- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 msgr2=0x7f27d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27ec15a780 0x7f27d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27ea800640 1 -- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27d00047c0 msgr2=0x7f27d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27d00047c0 0x7f27d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27ea800640 1 -- 172.21.15.28:0/427461232 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc009400 con 0x7f27d0003b30 2024-09-13T17:20:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.335+0000 7f27e3e00640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27d00047c0 0x7f27d0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.336+0000 7f27ea800640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d0003b30 0x7f27d0003f30 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009c60 tx=0x7f27d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8c9fed967e1845 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.336+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc031400 con 0x7f27d0003b30 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.336+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc02d050 con 0x7f27d0003b30 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.336+0000 7f27e9e00640 1 -- 172.21.15.28:0/427461232 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc031760 con 0x7f27d0003b30 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 -- 172.21.15.28:0/427461232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d0003b30 msgr2=0x7f27d0003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 --2- 172.21.15.28:0/427461232 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27d0003b30 0x7f27d0003f30 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009c60 tx=0x7f27d00057f0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27eb200640 1 -- 172.21.15.28:0/427461232 reap_dead start 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 -- 172.21.15.28:0/427461232 shutdown_connections 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 -- 172.21.15.28:0/427461232 >> 172.21.15.28:0/427461232 conn(0x7f27ec0b0270 msgr2=0x7f27ec0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 -- 172.21.15.28:0/427461232 shutdown_connections 2024-09-13T17:20:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.337+0000 7f27f1400640 1 -- 172.21.15.28:0/427461232 wait complete. 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.338+0000 7f27f1400640 1 Processor -- start 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.339+0000 7f27f1400640 1 -- start start 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.339+0000 7f27f1400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27d0003b30 0x7f27ec14f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.339+0000 7f27f1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27ec0ad860 0x7f27ec150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27f1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec0a87e0 con 0x7f27ec150ad0 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27f1400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec0a84e0 con 0x7f27ec0ad860 2024-09-13T17:20:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27f1400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec150f50 con 0x7f27d0003b30 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33284/0 (socket says 172.21.15.28:33284) 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 -- 172.21.15.28:0/3001353037 learned_addr learned my addr 172.21.15.28:0/3001353037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27ea800640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27d0003b30 0x7f27ec14f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27e3e00640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27ec0ad860 0x7f27ec150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 -- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27d0003b30 msgr2=0x7f27ec14f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27d0003b30 0x7f27ec14f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 -- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27ec0ad860 msgr2=0x7f27ec150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27ec0ad860 0x7f27ec150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.340+0000 7f27eb200640 1 -- 172.21.15.28:0/3001353037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec151230 con 0x7f27ec150ad0 2024-09-13T17:20:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27eb200640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0b12e0 tx=0x7f27e4007b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e400e9c0 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e400eb40 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4013420 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec151510 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.341+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec152630 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.342+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec0a8950 con 0x7f27ec150ad0 2024-09-13T17:20:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.343+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f27e4013580 con 0x7f27ec150ad0 2024-09-13T17:20:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.344+0000 7f27e2a00640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.344+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f27e4089350 con 0x7f27ec150ad0 2024-09-13T17:20:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.344+0000 7f27ea800640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.345+0000 7f27ea800640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 0x7f27c806d0c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f27dc033000 tx=0x7f27dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.348+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f27e4054730 con 0x7f27ec150ad0 2024-09-13T17:20:30.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.527+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f27ec06cb90 con 0x7f27ec150ad0 2024-09-13T17:20:30.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.531+0000 7f27e2a00640 1 -- 172.21.15.28:0/3001353037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f27e4057d50 con 0x7f27ec150ad0 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.533+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 msgr2=0x7f27c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.533+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 0x7f27c806d0c0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f27dc033000 tx=0x7f27dc008030 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.533+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 msgr2=0x7f27ec1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.533+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0b12e0 tx=0x7f27e4007b00 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 shutdown_connections 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f27ec150ad0 0x7f27ec1520f0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f27ec0ad860 0x7f27ec150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 --2- 172.21.15.28:0/3001353037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f27d0003b30 0x7f27ec14f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 >> 172.21.15.28:0/3001353037 conn(0x7f27ec0b0270 msgr2=0x7f27ec0afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 shutdown_connections 2024-09-13T17:20:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.534+0000 7f27f1400640 1 -- 172.21.15.28:0/3001353037 wait complete. 2024-09-13T17:20:30.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover' 2024-09-13T17:20:30.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.706+0000 7f0792400640 1 Processor -- start 2024-09-13T17:20:30.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.706+0000 7f0792400640 1 -- start start 2024-09-13T17:20:30.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0792400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f078c159440 0x7f078c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0792400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f078c0b29f0 0x7f078c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0792400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f078c0c3850 con 0x7f078c159820 2024-09-13T17:20:30.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0792400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c15a930 con 0x7f078c0b29f0 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0792400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f078c159d20 con 0x7f078c159440 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.707+0000 7f0791a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f078c0b29f0 0x7f078c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f078c0b29f0 0x7f078c0b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33294/0 (socket says 172.21.15.28:33294) 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 -- 172.21.15.28:0/4069651504 learned_addr learned my addr 172.21.15.28:0/4069651504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791000640 1 --2- 172.21.15.28:0/4069651504 >> v2:172.21.15.189:3300/0 conn(0x7f078c159440 0x7f078c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 -- 172.21.15.28:0/4069651504 >> v2:172.21.15.189:3300/0 conn(0x7f078c159440 msgr2=0x7f078c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 --2- 172.21.15.28:0/4069651504 >> v2:172.21.15.189:3300/0 conn(0x7f078c159440 0x7f078c0b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 -- 172.21.15.28:0/4069651504 >> v1:172.21.15.189:6789/0 conn(0x7f078c159820 legacy=0x7f078c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.708+0000 7f0791a00640 1 -- 172.21.15.28:0/4069651504 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c15a000 con 0x7f078c0b29f0 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.709+0000 7f0791a00640 1 --2- 172.21.15.28:0/4069651504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c0b0980 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f0784009e50 tx=0x7f078402a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631c60d7abc3f2b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.709+0000 7f078b400640 1 -- 172.21.15.28:0/4069651504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784031600 con 0x7f078c0b29f0 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.709+0000 7f078b400640 1 -- 172.21.15.28:0/4069651504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078402d050 con 0x7f078c0b29f0 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.709+0000 7f078b400640 1 -- 172.21.15.28:0/4069651504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0784031980 con 0x7f078c0b29f0 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 -- 172.21.15.28:0/4069651504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 msgr2=0x7f078c0b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 --2- 172.21.15.28:0/4069651504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c0b0980 secure :-1 s=READY pgs=2097 cs=0 l=1 rev1=1 crypto rx=0x7f0784009e50 tx=0x7f078402a7f0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 -- 172.21.15.28:0/4069651504 shutdown_connections 2024-09-13T17:20:30.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 --2- 172.21.15.28:0/4069651504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c0b0980 unknown :-1 s=CLOSED pgs=2097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 --2- 172.21.15.28:0/4069651504 >> v2:172.21.15.189:3300/0 conn(0x7f078c159440 0x7f078c0b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 -- 172.21.15.28:0/4069651504 >> 172.21.15.28:0/4069651504 conn(0x7f078c072e40 msgr2=0x7f078c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 -- 172.21.15.28:0/4069651504 shutdown_connections 2024-09-13T17:20:30.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.710+0000 7f0792400640 1 -- 172.21.15.28:0/4069651504 wait complete. 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.712+0000 7f0792400640 1 Processor -- start 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.712+0000 7f0792400640 1 -- start start 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f078c159440 0x7f078c156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 0x7f078c157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c1504a0 con 0x7f078c0b29f0 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078c150780 con 0x7f078c159820 2024-09-13T17:20:30.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0792400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078c150a60 con 0x7f078c159440 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33306/0 (socket says 172.21.15.28:33306) 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 -- 172.21.15.28:0/1266506431 learned_addr learned my addr 172.21.15.28:0/1266506431 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f078be00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f078c159440 0x7f078c156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 -- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f078c159440 msgr2=0x7f078c156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791a00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 0x7f078c157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f078c159440 0x7f078c156f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 -- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 msgr2=0x7f078c157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 0x7f078c157450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791000640 1 -- 172.21.15.28:0/1266506431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784009b30 con 0x7f078c0b29f0 2024-09-13T17:20:30.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.713+0000 7f0791a00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 0x7f078c157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0791000640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f078000e9b0 tx=0x7f078000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0780009670 con 0x7f078c0b29f0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078000c9a0 con 0x7f078c0b29f0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078000cd50 con 0x7f078c0b29f0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078c150d40 con 0x7f078c0b29f0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.714+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078c14f920 con 0x7f078c0b29f0 2024-09-13T17:20:30.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.715+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078c0b22e0 con 0x7f078c0b29f0 2024-09-13T17:20:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.716+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0780013470 con 0x7f078c0b29f0 2024-09-13T17:20:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.717+0000 7f0788c00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 0x7f076c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.717+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0780089310 con 0x7f078c0b29f0 2024-09-13T17:20:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.717+0000 7f078be00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 0x7f076c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:30.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.718+0000 7f078be00640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 0x7f076c06d0c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f078c14da70 tx=0x7f0778009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:30.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:30.721+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0780054780 con 0x7f078c0b29f0 2024-09-13T17:20:31.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.004+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]} v 0) v1 -- 0x7f078c0a8440 con 0x7f078c0b29f0 2024-09-13T17:20:31.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.146+0000 7f0788c00640 1 -- 172.21.15.28:0/1266506431 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_task_status_on_mds_failover"]}]=0 v620) v1 ==== 166+0+0 (secure 0 0 0) 0x7f0780087e60 con 0x7f078c0b29f0 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 msgr2=0x7f076c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 0x7f076c06d0c0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f078c14da70 tx=0x7f0778009040 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 msgr2=0x7f078c140120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7f078000e9b0 tx=0x7f078000ee70 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 shutdown_connections 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f076c06ac00 0x7f076c06d0c0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078c159820 0x7f078c157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f078c159440 0x7f078c156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.148+0000 7f0792400640 1 --2- 172.21.15.28:0/1266506431 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f078c0b29f0 0x7f078c140120 unknown :-1 s=CLOSED pgs=2098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.149+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 >> 172.21.15.28:0/1266506431 conn(0x7f078c072e40 msgr2=0x7f078c06f910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:31.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.149+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 shutdown_connections 2024-09-13T17:20:31.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.149+0000 7f0792400640 1 -- 172.21.15.28:0/1266506431 wait complete. 2024-09-13T17:20:31.167 INFO:tasks.cephfs_test_runner:test_scrub_task_status_on_mds_failover (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-09-13T17:20:31.168 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls) 2024-09-13T17:20:31.168 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.322+0000 7f57a1e00640 1 Processor -- start 2024-09-13T17:20:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.322+0000 7f57a1e00640 1 -- start start 2024-09-13T17:20:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f57a1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f579c0cabc0 0x7f579c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f57a1e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 0x7f579c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f57a1e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f579c15e100 con 0x7f579c15a540 2024-09-13T17:20:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f57a1e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579c0adb00 con 0x7f579c0cabc0 2024-09-13T17:20:31.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f57a1e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f579c071670 con 0x7f579c070cc0 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.323+0000 7f579a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f579c0cabc0 0x7f579c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f579c0cabc0 0x7f579c06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33310/0 (socket says 172.21.15.28:33310) 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 -- 172.21.15.28:0/2414084805 learned_addr learned my addr 172.21.15.28:0/2414084805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579aa00640 1 --1- 172.21.15.28:0/2414084805 >> v1:172.21.15.28:6789/0 conn(0x7f579c15a540 0x7f579c15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54896/0 (socket says 172.21.15.28:54896) 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579b400640 1 --2- 172.21.15.28:0/2414084805 >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 0x7f579c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 -- 172.21.15.28:0/2414084805 >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 msgr2=0x7f579c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 --2- 172.21.15.28:0/2414084805 >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 0x7f579c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 -- 172.21.15.28:0/2414084805 >> v1:172.21.15.28:6789/0 conn(0x7f579c15a540 legacy=0x7f579c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579a000640 1 -- 172.21.15.28:0/2414084805 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579c15e3e0 con 0x7f579c0cabc0 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f5799600640 1 -- 172.21.15.28:0/2414084805 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935545074 0 0) 0x7f579c15e100 con 0x7f579c15a540 2024-09-13T17:20:31.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.324+0000 7f579b400640 1 --2- 172.21.15.28:0/2414084805 >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 0x7f579c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.325+0000 7f579a000640 1 --2- 172.21.15.28:0/2414084805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c0cabc0 0x7f579c06c430 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f579c070900 tx=0x7f578802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80afca65279059a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.325+0000 7f5799600640 1 -- 172.21.15.28:0/2414084805 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788032440 con 0x7f579c0cabc0 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.325+0000 7f5799600640 1 -- 172.21.15.28:0/2414084805 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57880325a0 con 0x7f579c0cabc0 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.325+0000 7f5799600640 1 -- 172.21.15.28:0/2414084805 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788032930 con 0x7f579c0cabc0 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 -- 172.21.15.28:0/2414084805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c0cabc0 msgr2=0x7f579c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 --2- 172.21.15.28:0/2414084805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c0cabc0 0x7f579c06c430 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7f579c070900 tx=0x7f578802a230 comp rx=0 tx=0).stop 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 -- 172.21.15.28:0/2414084805 shutdown_connections 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 --2- 172.21.15.28:0/2414084805 >> v2:172.21.15.189:3300/0 conn(0x7f579c070cc0 0x7f579c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 --2- 172.21.15.28:0/2414084805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c0cabc0 0x7f579c06c430 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 -- 172.21.15.28:0/2414084805 >> 172.21.15.28:0/2414084805 conn(0x7f579c069e80 msgr2=0x7f579c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:31.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 -- 172.21.15.28:0/2414084805 shutdown_connections 2024-09-13T17:20:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.326+0000 7f57a1e00640 1 -- 172.21.15.28:0/2414084805 wait complete. 2024-09-13T17:20:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.328+0000 7f57a1e00640 1 Processor -- start 2024-09-13T17:20:31.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.328+0000 7f57a1e00640 1 -- start start 2024-09-13T17:20:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f579c070cc0 0x7f579c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579c0cabc0 0x7f579c156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c0af350 con 0x7f579c15a540 2024-09-13T17:20:31.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c1edd10 con 0x7f579c0cabc0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f57a1e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f579c150160 con 0x7f579c070cc0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f579b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f579b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33312/0 (socket says 172.21.15.28:33312) 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f579b400640 1 -- 172.21.15.28:0/761340996 learned_addr learned my addr 172.21.15.28:0/761340996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f579a000640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579c0cabc0 0x7f579c156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.329+0000 7f579aa00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f579c070cc0 0x7f579c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 -- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f579c070cc0 msgr2=0x7f579c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f579c070cc0 0x7f579c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 -- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579c0cabc0 msgr2=0x7f579c156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579c0cabc0 0x7f579c156ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 -- 172.21.15.28:0/761340996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880094d0 con 0x7f579c15a540 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f579b400640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f579400cef0 tx=0x7f579400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400d840 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579400d9c0 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400dd70 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.330+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c159670 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.331+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c159c30 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.332+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579c0af490 con 0x7f579c15a540 2024-09-13T17:20:31.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.332+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f579401e3b0 con 0x7f579c15a540 2024-09-13T17:20:31.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.333+0000 7f5793400640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 0x7f577c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.333+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5794089af0 con 0x7f579c15a540 2024-09-13T17:20:31.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.334+0000 7f579aa00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 0x7f577c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.334+0000 7f579aa00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 0x7f577c06d0c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f578c007fd0 tx=0x7f578c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.338+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f579c0af490 con 0x7f579c15a540 2024-09-13T17:20:31.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.515+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f579c0af490 con 0x7f579c15a540 2024-09-13T17:20:31.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.516+0000 7f5793400640 1 -- 172.21.15.28:0/761340996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v171) v1 ==== 74+0+13126 (secure 0 0 0) 0x7f579c0af490 con 0x7f579c15a540 2024-09-13T17:20:31.518 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:31.518 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":171,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:30.133880+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/1290280670":"2024-09-14T17:20:27.121642+0000","172.21.15.28:6832/1290280670":"2024-09-14T17:20:27.121642+0000","172.21.15.28:6835/3709710897":"2024-09-14T17:20:16.004790+0000","172.21.15.28:6833/3709710897":"2024-09-14T17:20:16.004790+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:31.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 msgr2=0x7f577c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 0x7f577c06d0c0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f578c007fd0 tx=0x7f578c007de0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 msgr2=0x7f579c1f01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7f579400cef0 tx=0x7f579400a710 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 shutdown_connections 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f577c06ac00 0x7f577c06d0c0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f579c15a540 0x7f579c1f01e0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f579c0cabc0 0x7f579c156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 --2- 172.21.15.28:0/761340996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f579c070cc0 0x7f579c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 >> 172.21.15.28:0/761340996 conn(0x7f579c069e80 msgr2=0x7f579c15ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 shutdown_connections 2024-09-13T17:20:31.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.518+0000 7f57a1e00640 1 -- 172.21.15.28:0/761340996 wait complete. 2024-09-13T17:20:31.537 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:20:31.537 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:20:31.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.694+0000 7f46af200640 1 Processor -- start 2024-09-13T17:20:31.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.694+0000 7f46af200640 1 -- start start 2024-09-13T17:20:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.695+0000 7f46af200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 0x7f46a806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.695+0000 7f46af200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f46a806f3f0 0x7f46a8070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.695+0000 7f46af200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46a80add00 0x7f46a8159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.695+0000 7f46af200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a806f920 con 0x7f46a80add00 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46af200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a815b290 con 0x7f46a806f3f0 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46af200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a815b570 con 0x7f46a8159770 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46a80add00 0x7f46a8159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46a80add00 0x7f46a8159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33316/0 (socket says 172.21.15.28:33316) 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 -- 172.21.15.28:0/1496687762 learned_addr learned my addr 172.21.15.28:0/1496687762 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46a7e00640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 0x7f46a806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f469f400640 1 --2- 172.21.15.28:0/1496687762 >> v2:172.21.15.189:3300/0 conn(0x7f46a806f3f0 0x7f46a8070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 -- 172.21.15.28:0/1496687762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 msgr2=0x7f46a806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 0x7f46a806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 -- 172.21.15.28:0/1496687762 >> v2:172.21.15.189:3300/0 conn(0x7f46a806f3f0 msgr2=0x7f46a8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:31.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 --2- 172.21.15.28:0/1496687762 >> v2:172.21.15.189:3300/0 conn(0x7f46a806f3f0 0x7f46a8070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46ad200640 1 -- 172.21.15.28:0/1496687762 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a815b850 con 0x7f46a80add00 2024-09-13T17:20:31.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.696+0000 7f46a7e00640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 0x7f46a806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.697+0000 7f46ad200640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a80add00 0x7f46a8159b50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f46a80cf5c0 tx=0x7f46a002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4fe923df6a306b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46a7400640 1 -- 172.21.15.28:0/1496687762 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0031600 con 0x7f46a80add00 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46a7400640 1 -- 172.21.15.28:0/1496687762 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a002d050 con 0x7f46a80add00 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46a7400640 1 -- 172.21.15.28:0/1496687762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0031980 con 0x7f46a80add00 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 -- 172.21.15.28:0/1496687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a80add00 msgr2=0x7f46a8159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a80add00 0x7f46a8159b50 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7f46a80cf5c0 tx=0x7f46a002a610 comp rx=0 tx=0).stop 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 -- 172.21.15.28:0/1496687762 shutdown_connections 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a80add00 0x7f46a8159b50 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 --2- 172.21.15.28:0/1496687762 >> v2:172.21.15.189:3300/0 conn(0x7f46a806f3f0 0x7f46a8070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 --2- 172.21.15.28:0/1496687762 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a8159770 0x7f46a806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 -- 172.21.15.28:0/1496687762 >> 172.21.15.28:0/1496687762 conn(0x7f46a80b0270 msgr2=0x7f46a80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:31.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 -- 172.21.15.28:0/1496687762 shutdown_connections 2024-09-13T17:20:31.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.698+0000 7f46af200640 1 -- 172.21.15.28:0/1496687762 wait complete. 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.700+0000 7f46af200640 1 Processor -- start 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.700+0000 7f46af200640 1 -- start start 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 0x7f46a8150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 0x7f46a8151b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a81514d0 con 0x7f46a8159770 2024-09-13T17:20:31.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8164c00 con 0x7f46a80add00 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46af200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a8166220 con 0x7f46a806f3f0 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46ad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46ad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33324/0 (socket says 172.21.15.28:33324) 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.701+0000 7f46ad200640 1 -- 172.21.15.28:0/2357381444 learned_addr learned my addr 172.21.15.28:0/2357381444 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46a7e00640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 0x7f46a8150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f469f400640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 0x7f46a8151b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 -- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 msgr2=0x7f46a8150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 0x7f46a8150900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 -- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 msgr2=0x7f46a8151b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 0x7f46a8151b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 -- 172.21.15.28:0/2357381444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0009b30 con 0x7f46a8159770 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46a7e00640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 0x7f46a8150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f469f400640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 0x7f46a8151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46ad200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f46a0034040 tx=0x7f46a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a003b820 con 0x7f46a8159770 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46a003b980 con 0x7f46a8159770 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a81642d0 con 0x7f46a8159770 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.702+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a003bd30 con 0x7f46a8159770 2024-09-13T17:20:31.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.703+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a8164890 con 0x7f46a8159770 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.704+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a8164d70 con 0x7f46a8159770 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.704+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f46a0038050 con 0x7f46a8159770 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.705+0000 7f46a4c00640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 0x7f468406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.705+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f46a00a9c60 con 0x7f46a8159770 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.706+0000 7f46a7e00640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 0x7f468406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:31.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.706+0000 7f46a7e00640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 0x7f468406d0c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f46a80ca990 tx=0x7f4698008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:31.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.709+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46a0075040 con 0x7f46a8159770 2024-09-13T17:20:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.885+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46a8159b50 con 0x7f46a8159770 2024-09-13T17:20:31.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.886+0000 7f46a4c00640 1 -- 172.21.15.28:0/2357381444 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v171) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46a004b0a0 con 0x7f46a8159770 2024-09-13T17:20:31.888 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 msgr2=0x7f468406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 0x7f468406d0c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f46a80ca990 tx=0x7f4698008040 comp rx=0 tx=0).stop 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 msgr2=0x7f46a8152050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f46a0034040 tx=0x7f46a0004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 shutdown_connections 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f468406ac00 0x7f468406d0c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46a8159770 0x7f46a8152050 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46a80add00 0x7f46a8151b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 --2- 172.21.15.28:0/2357381444 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46a806f3f0 0x7f46a8150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 >> 172.21.15.28:0/2357381444 conn(0x7f46a80b0270 msgr2=0x7f46a80ae130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 shutdown_connections 2024-09-13T17:20:31.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:31.888+0000 7f46af200640 1 -- 172.21.15.28:0/2357381444 wait complete. 2024-09-13T17:20:31.906 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive' 2024-09-13T17:20:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.059+0000 7f1f86000640 1 Processor -- start 2024-09-13T17:20:32.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.059+0000 7f1f86000640 1 -- start start 2024-09-13T17:20:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.060+0000 7f1f86000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f8015b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.060+0000 7f1f86000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 0x7f1f800af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.060+0000 7f1f86000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8015bb10 con 0x7f1f8006b360 2024-09-13T17:20:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.060+0000 7f1f86000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f8006b900 con 0x7f1f800b0b50 2024-09-13T17:20:32.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.060+0000 7f1f86000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8006bbe0 con 0x7f1f80154a90 2024-09-13T17:20:32.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f84c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f8015b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f85600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 0x7f1f800af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f7fe00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f1f8006b360 0x7f1f800b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:56124/0 (socket says 172.21.15.28:56124) 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f7fe00640 1 -- 172.21.15.28:0/1871091835 learned_addr learned my addr 172.21.15.28:0/1871091835 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f7f400640 1 -- 172.21.15.28:0/1871091835 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1749656226 0 0) 0x7f1f8015bb10 con 0x7f1f8006b360 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f84c00640 1 -- 172.21.15.28:0/1871091835 >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 msgr2=0x7f1f800af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f84c00640 1 --2- 172.21.15.28:0/1871091835 >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 0x7f1f800af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f84c00640 1 -- 172.21.15.28:0/1871091835 >> v1:172.21.15.189:6789/0 conn(0x7f1f8006b360 legacy=0x7f1f800b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f84c00640 1 -- 172.21.15.28:0/1871091835 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8006bec0 con 0x7f1f80154a90 2024-09-13T17:20:32.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.061+0000 7f1f85600640 1 --2- 172.21.15.28:0/1871091835 >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 0x7f1f800af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.062+0000 7f1f84c00640 1 --2- 172.21.15.28:0/1871091835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f8015b5d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097a0 tx=0x7f1f7402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6765d9d401ea2596 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f7f400640 1 -- 172.21.15.28:0/1871091835 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74032440 con 0x7f1f80154a90 2024-09-13T17:20:32.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f7f400640 1 -- 172.21.15.28:0/1871091835 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f740325a0 con 0x7f1f80154a90 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f7f400640 1 -- 172.21.15.28:0/1871091835 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74032930 con 0x7f1f80154a90 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 -- 172.21.15.28:0/1871091835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 msgr2=0x7f1f8015b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 --2- 172.21.15.28:0/1871091835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f8015b5d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1f740097a0 tx=0x7f1f7402a180 comp rx=0 tx=0).stop 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 -- 172.21.15.28:0/1871091835 shutdown_connections 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 --2- 172.21.15.28:0/1871091835 >> v2:172.21.15.189:3300/0 conn(0x7f1f800b0b50 0x7f1f800af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 --2- 172.21.15.28:0/1871091835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f8015b5d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 -- 172.21.15.28:0/1871091835 >> 172.21.15.28:0/1871091835 conn(0x7f1f80072e40 msgr2=0x7f1f80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 -- 172.21.15.28:0/1871091835 shutdown_connections 2024-09-13T17:20:32.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.063+0000 7f1f86000640 1 -- 172.21.15.28:0/1871091835 wait complete. 2024-09-13T17:20:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.065+0000 7f1f86000640 1 Processor -- start 2024-09-13T17:20:32.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.065+0000 7f1f86000640 1 -- start start 2024-09-13T17:20:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8006b360 0x7f1f800c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f80153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800b1840 con 0x7f1f800b0b50 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80158250 con 0x7f1f8006b360 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f86000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800b1b00 con 0x7f1f80154a90 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33328/0 (socket says 172.21.15.28:33328) 2024-09-13T17:20:32.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.066+0000 7f1f7fe00640 1 -- 172.21.15.28:0/3285676631 learned_addr learned my addr 172.21.15.28:0/3285676631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:32.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f85600640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f80153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f84c00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8006b360 0x7f1f800c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 -- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 msgr2=0x7f1f80153930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f80153930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 -- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8006b360 msgr2=0x7f1f800c4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8006b360 0x7f1f800c4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 -- 172.21.15.28:0/3285676631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009a70 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7fe00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c4b0 tx=0x7f1f6c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.067+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c0074e0 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.068+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f6c007660 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.068+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6c004a20 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.068+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f74009400 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.068+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800b20c0 con 0x7f1f800b0b50 2024-09-13T17:20:32.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.069+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f800af4b0 con 0x7f1f800b0b50 2024-09-13T17:20:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.070+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1f6c004b80 con 0x7f1f800b0b50 2024-09-13T17:20:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.070+0000 7f1f7cc00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 0x7f1f6006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:32.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.071+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f1f6c0881c0 con 0x7f1f800b0b50 2024-09-13T17:20:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.071+0000 7f1f84c00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 0x7f1f6006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:32.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.072+0000 7f1f84c00640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 0x7f1f6006d0f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1f74034000 tx=0x7f1f7402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:32.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.075+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1f6c0536f0 con 0x7f1f800b0b50 2024-09-13T17:20:32.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:32.322+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]} v 0) v1 -- 0x7f1f80158fa0 con 0x7f1f800b0b50 2024-09-13T17:20:33.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.153+0000 7f1f7cc00640 1 -- 172.21.15.28:0/3285676631 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]}]=0 v622) v1 ==== 162+0+0 (secure 0 0 0) 0x7f1f6c056d10 con 0x7f1f800b0b50 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 msgr2=0x7f1f6006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 0x7f1f6006d0f0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f1f74034000 tx=0x7f1f7402e000 comp rx=0 tx=0).stop 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 msgr2=0x7f1f801579e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7f1f6c00c4b0 tx=0x7f1f6c00c970 comp rx=0 tx=0).stop 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 shutdown_connections 2024-09-13T17:20:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1f6006ac30 0x7f1f6006d0f0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1f80154a90 0x7f1f80153930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f800b0b50 0x7f1f801579e0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 --2- 172.21.15.28:0/3285676631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1f8006b360 0x7f1f800c4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 >> 172.21.15.28:0/3285676631 conn(0x7f1f80072e40 msgr2=0x7f1f80159280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 shutdown_connections 2024-09-13T17:20:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.156+0000 7f1f86000640 1 -- 172.21.15.28:0/3285676631 wait complete. 2024-09-13T17:20:33.176 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:20:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.340+0000 7f9be5400640 1 Processor -- start 2024-09-13T17:20:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.340+0000 7f9be5400640 1 -- start start 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9be5400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9be0159770 0x7f9be006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9be5400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be006f3f0 0x7f9be0070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9be5400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9be006f920 con 0x7f9be00add00 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9be5400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be015b290 con 0x7f9be006f3f0 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9be5400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9be015b570 con 0x7f9be0159770 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9bd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be006f3f0 0x7f9be0070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9bd7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9be006f3f0 0x7f9be0070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33334/0 (socket says 172.21.15.28:33334) 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.341+0000 7f9bd7e00640 1 -- 172.21.15.28:0/1237242022 learned_addr learned my addr 172.21.15.28:0/1237242022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.342+0000 7f9bde800640 1 --2- 172.21.15.28:0/1237242022 >> v2:172.21.15.189:3300/0 conn(0x7f9be0159770 0x7f9be006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.342+0000 7f9bd7e00640 1 -- 172.21.15.28:0/1237242022 >> v2:172.21.15.189:3300/0 conn(0x7f9be0159770 msgr2=0x7f9be006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.342+0000 7f9bd7e00640 1 --2- 172.21.15.28:0/1237242022 >> v2:172.21.15.189:3300/0 conn(0x7f9be0159770 0x7f9be006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.342+0000 7f9bd7e00640 1 -- 172.21.15.28:0/1237242022 >> v1:172.21.15.189:6789/0 conn(0x7f9be00add00 legacy=0x7f9be0159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.342+0000 7f9bd7e00640 1 -- 172.21.15.28:0/1237242022 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be015b850 con 0x7f9be006f3f0 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.343+0000 7f9bd7e00640 1 --2- 172.21.15.28:0/1237242022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be0070260 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f9be0067f80 tx=0x7f9bc802a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c66a853dde8b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.343+0000 7f9bdde00640 1 -- 172.21.15.28:0/1237242022 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc802aea0 con 0x7f9be006f3f0 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.343+0000 7f9bdde00640 1 -- 172.21.15.28:0/1237242022 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc8032440 con 0x7f9be006f3f0 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.343+0000 7f9bdde00640 1 -- 172.21.15.28:0/1237242022 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc80327d0 con 0x7f9be006f3f0 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 -- 172.21.15.28:0/1237242022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 msgr2=0x7f9be0070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 --2- 172.21.15.28:0/1237242022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be0070260 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f9be0067f80 tx=0x7f9bc802a150 comp rx=0 tx=0).stop 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 -- 172.21.15.28:0/1237242022 shutdown_connections 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 --2- 172.21.15.28:0/1237242022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be0070260 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 --2- 172.21.15.28:0/1237242022 >> v2:172.21.15.189:3300/0 conn(0x7f9be0159770 0x7f9be006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 -- 172.21.15.28:0/1237242022 >> 172.21.15.28:0/1237242022 conn(0x7f9be00b0270 msgr2=0x7f9be00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 -- 172.21.15.28:0/1237242022 shutdown_connections 2024-09-13T17:20:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.344+0000 7f9be5400640 1 -- 172.21.15.28:0/1237242022 wait complete. 2024-09-13T17:20:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.345+0000 7f9be5400640 1 Processor -- start 2024-09-13T17:20:33.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.346+0000 7f9be5400640 1 -- start start 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.346+0000 7f9be5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9be5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be00add00 0x7f9be00cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9be5400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be0159770 0x7f9be00cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9be5400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0141c10 con 0x7f9be006f3f0 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9be5400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0142070 con 0x7f9be00add00 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9be5400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9be0142350 con 0x7f9be0159770 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33346/0 (socket says 172.21.15.28:33346) 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 -- 172.21.15.28:0/237277983 learned_addr learned my addr 172.21.15.28:0/237277983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bd7e00640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be00add00 0x7f9be00cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bdf200640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be0159770 0x7f9be00cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 -- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be0159770 msgr2=0x7f9be00cccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be0159770 0x7f9be00cccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 -- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be00add00 msgr2=0x7f9be00cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be00add00 0x7f9be00cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.347+0000 7f9bde800640 1 -- 172.21.15.28:0/237277983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc80094d0 con 0x7f9be006f3f0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9bde800640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000e870 tx=0x7f9bd000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd00094f0 con 0x7f9be006f3f0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bd000c940 con 0x7f9be006f3f0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd000ccf0 con 0x7f9be006f3f0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9be0142630 con 0x7f9be006f3f0 2024-09-13T17:20:33.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.348+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9be00a8e20 con 0x7f9be006f3f0 2024-09-13T17:20:33.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.349+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9be015a650 con 0x7f9be006f3f0 2024-09-13T17:20:33.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.350+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9bd0013470 con 0x7f9be006f3f0 2024-09-13T17:20:33.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.351+0000 7f9bd6a00640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 0x7f9bbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:33.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.351+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f9bd0089180 con 0x7f9be006f3f0 2024-09-13T17:20:33.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.351+0000 7f9bd7e00640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 0x7f9bbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:33.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.352+0000 7f9bd7e00640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 0x7f9bbc06d0c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8034000 tx=0x7f9bc802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:33.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.355+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9bd0054610 con 0x7f9be006f3f0 2024-09-13T17:20:33.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.535+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9be0070bf0 con 0x7f9be006f3f0 2024-09-13T17:20:33.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.539+0000 7f9bd6a00640 1 -- 172.21.15.28:0/237277983 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v44)=0 v44) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9bd0057c30 con 0x7f9be006f3f0 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.541+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 msgr2=0x7f9bbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.541+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 0x7f9bbc06d0c0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8034000 tx=0x7f9bc802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.541+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 msgr2=0x7f9be00a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.541+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f9bd000e870 tx=0x7f9bd000ed30 comp rx=0 tx=0).stop 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 shutdown_connections 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bbc06ac00 0x7f9bbc06d0c0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9be0159770 0x7f9be00cccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9be00add00 0x7f9be00cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 --2- 172.21.15.28:0/237277983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9be006f3f0 0x7f9be00a88e0 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 >> 172.21.15.28:0/237277983 conn(0x7f9be00b0270 msgr2=0x7f9be0159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 shutdown_connections 2024-09-13T17:20:33.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:33.542+0000 7f9be5400640 1 -- 172.21.15.28:0/237277983 wait complete. 2024-09-13T17:20:33.559 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:20:33.559 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:20:33.660 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:20:33.751 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:20:33.751 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:33.751 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:33.787 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:20:33.787 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:20:33.787 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:33.824 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:20:33.824 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:33.825 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:33.862 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:20:33.862 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:20:33.862 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:33.900 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:20:33.901 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:20:33.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:34.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.055+0000 7f289b200640 1 Processor -- start 2024-09-13T17:20:34.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.055+0000 7f289b200640 1 -- start start 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.056+0000 7f289b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f289415a540 0x7f289406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f289b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2894070cc0 0x7f2894071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f289b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f289415e100 con 0x7f28940cabc0 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f289b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28940adb00 con 0x7f2894070cc0 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f289b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2894071670 con 0x7f289415a540 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2894070cc0 0x7f2894071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2894070cc0 0x7f2894071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49816/0 (socket says 172.21.15.28:49816) 2024-09-13T17:20:34.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 -- 172.21.15.28:0/800369017 learned_addr learned my addr 172.21.15.28:0/800369017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:34.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2893e00640 1 --2- 172.21.15.28:0/800369017 >> v2:172.21.15.189:3300/0 conn(0x7f289415a540 0x7f289406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 -- 172.21.15.28:0/800369017 >> v2:172.21.15.189:3300/0 conn(0x7f289415a540 msgr2=0x7f289406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:34.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 --2- 172.21.15.28:0/800369017 >> v2:172.21.15.189:3300/0 conn(0x7f289415a540 0x7f289406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 -- 172.21.15.28:0/800369017 >> v1:172.21.15.189:6789/0 conn(0x7f28940cabc0 legacy=0x7f289415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:34.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.057+0000 7f2899200640 1 -- 172.21.15.28:0/800369017 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289415e3e0 con 0x7f2894070cc0 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.058+0000 7f2899200640 1 --2- 172.21.15.28:0/800369017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2894070cc0 0x7f2894071130 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f28940b3a00 tx=0x7f288c02a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f56f90cf095d96ba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f2893400640 1 -- 172.21.15.28:0/800369017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c031600 con 0x7f2894070cc0 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f2893400640 1 -- 172.21.15.28:0/800369017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f288c02d050 con 0x7f2894070cc0 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f2893400640 1 -- 172.21.15.28:0/800369017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f288c031980 con 0x7f2894070cc0 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 -- 172.21.15.28:0/800369017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2894070cc0 msgr2=0x7f2894071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 --2- 172.21.15.28:0/800369017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2894070cc0 0x7f2894071130 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f28940b3a00 tx=0x7f288c02a6d0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 -- 172.21.15.28:0/800369017 shutdown_connections 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 --2- 172.21.15.28:0/800369017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2894070cc0 0x7f2894071130 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 --2- 172.21.15.28:0/800369017 >> v2:172.21.15.189:3300/0 conn(0x7f289415a540 0x7f289406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 -- 172.21.15.28:0/800369017 >> 172.21.15.28:0/800369017 conn(0x7f2894069e80 msgr2=0x7f289406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 -- 172.21.15.28:0/800369017 shutdown_connections 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.059+0000 7f289b200640 1 -- 172.21.15.28:0/800369017 wait complete. 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.061+0000 7f289b200640 1 Processor -- start 2024-09-13T17:20:34.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.061+0000 7f289b200640 1 -- start start 2024-09-13T17:20:34.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2894070cc0 0x7f28940b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f289415a540 0x7f28941406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2894156dd0 con 0x7f28940cabc0 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28940b00e0 con 0x7f289415a540 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f289b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2894156710 con 0x7f2894070cc0 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f288b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f288b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49826/0 (socket says 172.21.15.28:49826) 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f288b400640 1 -- 172.21.15.28:0/830501763 learned_addr learned my addr 172.21.15.28:0/830501763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.062+0000 7f2893e00640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2894070cc0 0x7f28940b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f2899200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f289415a540 0x7f28941406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 -- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2894070cc0 msgr2=0x7f28940b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2894070cc0 0x7f28940b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 -- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f289415a540 msgr2=0x7f28941406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f289415a540 0x7f28941406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 -- 172.21.15.28:0/830501763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f288c009b30 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f288b400640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f287c00c450 tx=0x7f287c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c0052d0 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f287c005430 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.063+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c012870 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.064+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28941569f0 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.064+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28940b03c0 con 0x7f28940cabc0 2024-09-13T17:20:34.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.065+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2894156f10 con 0x7f28940cabc0 2024-09-13T17:20:34.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.065+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f287c0129d0 con 0x7f28940cabc0 2024-09-13T17:20:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.066+0000 7f2890c00640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 0x7f287006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.066+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(171..171 src has 1..171) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f287c088b10 con 0x7f28940cabc0 2024-09-13T17:20:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.067+0000 7f2893e00640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 0x7f287006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.067+0000 7f2893e00640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 0x7f287006d0c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f28840079a0 tx=0x7f2884008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.070+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f287c053fa0 con 0x7f28940cabc0 2024-09-13T17:20:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.287+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2894071130 con 0x7f28940cabc0 2024-09-13T17:20:34.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.287+0000 7f2890c00640 1 -- 172.21.15.28:0/830501763 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+2783 (secure 0 0 0) 0x7f287c0575c0 con 0x7f28940cabc0 2024-09-13T17:20:34.290 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:34.290 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":145,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8084,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/1522756541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":1522756541},{"type":"v1","addr":"172.21.15.189:6837","nonce":1522756541}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":9298,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2868294891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2868294891},{"type":"v1","addr":"172.21.15.189:6836","nonce":2868294891}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":133},{"gid":10479,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/1310585676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1310585676},{"type":"v1","addr":"172.21.15.28:6833","nonce":1310585676}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":145}],"filesystems":[]} 2024-09-13T17:20:34.290 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 145 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.289+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 msgr2=0x7f287006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.289+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 0x7f287006d0c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f28840079a0 tx=0x7f2884008040 comp rx=0 tx=0).stop 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.289+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 msgr2=0x7f28940acac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.289+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f287c00c450 tx=0x7f287c00c910 comp rx=0 tx=0).stop 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 shutdown_connections 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f287006ac00 0x7f287006d0c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f289415a540 0x7f28941406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f28940cabc0 0x7f28940acac0 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 --2- 172.21.15.28:0/830501763 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2894070cc0 0x7f28940b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 >> 172.21.15.28:0/830501763 conn(0x7f2894069e80 msgr2=0x7f289406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 shutdown_connections 2024-09-13T17:20:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.290+0000 7f289b200640 1 -- 172.21.15.28:0/830501763 wait complete. 2024-09-13T17:20:34.308 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:20:34.308 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:20:34.308 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:20:34.308 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:20:34.309 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:20:34.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:20:34.309 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:20:34.309 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:20:34.310 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:20:34.310 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:20:34.310 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:20:34.310 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:20:34.310 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:20:34.310 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:20:34.313 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:20:34.314 DEBUG:teuthology.parallel:result is None 2024-09-13T17:20:34.322 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:20:34.322 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:20:34.326 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:20:34.326 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:20:34.327 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:20:34.327 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:20:34.367 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:20:34.367 DEBUG:teuthology.parallel:result is None 2024-09-13T17:20:34.371 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:20:34.372 DEBUG:teuthology.parallel:result is None 2024-09-13T17:20:34.374 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:20:34.374 DEBUG:teuthology.parallel:result is None 2024-09-13T17:20:34.374 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:20:34.443 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:20:34.500 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:20:34.506 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:20:34.506 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:20:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.537+0000 7fe35aa00640 1 Processor -- start 2024-09-13T17:20:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.537+0000 7fe35aa00640 1 -- start start 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3540cf100 0x7fe35406a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe35415d890 0x7fe3540cf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe354069e80 con 0x7fe3540d0d70 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe35406ae10 con 0x7fe3540cf100 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3540d2590 con 0x7fe35415d890 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe359600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe3540cf100 0x7fe35406a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe3540d0d70 0x7fe3540d1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38458/0 (socket says 172.21.15.28:38458) 2024-09-13T17:20:34.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe35a000640 1 -- 172.21.15.28:0/1366028493 learned_addr learned my addr 172.21.15.28:0/1366028493 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.538+0000 7fe358c00640 1 --2- 172.21.15.28:0/1366028493 >> v2:172.21.15.189:3300/0 conn(0x7fe35415d890 0x7fe3540cf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe353e00640 1 -- 172.21.15.28:0/1366028493 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458196179 0 0) 0x7fe354069e80 con 0x7fe3540d0d70 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe353e00640 1 -- 172.21.15.28:0/1366028493 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe33c003520 con 0x7fe3540d0d70 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe359600640 1 -- 172.21.15.28:0/1366028493 >> v2:172.21.15.189:3300/0 conn(0x7fe35415d890 msgr2=0x7fe3540cf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe359600640 1 --2- 172.21.15.28:0/1366028493 >> v2:172.21.15.189:3300/0 conn(0x7fe35415d890 0x7fe3540cf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe359600640 1 -- 172.21.15.28:0/1366028493 >> v1:172.21.15.28:6789/0 conn(0x7fe3540d0d70 legacy=0x7fe3540d1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.539+0000 7fe359600640 1 -- 172.21.15.28:0/1366028493 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3540cdc40 con 0x7fe3540cf100 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.540+0000 7fe359600640 1 --2- 172.21.15.28:0/1366028493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540cf100 0x7fe35406a8d0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe3540b5060 tx=0x7fe34802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3737218fac1d1f90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.540+0000 7fe353e00640 1 -- 172.21.15.28:0/1366028493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34802ae60 con 0x7fe3540cf100 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.540+0000 7fe353e00640 1 -- 172.21.15.28:0/1366028493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe348032440 con 0x7fe3540cf100 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.540+0000 7fe353e00640 1 -- 172.21.15.28:0/1366028493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3480327b0 con 0x7fe3540cf100 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 -- 172.21.15.28:0/1366028493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540cf100 msgr2=0x7fe35406a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 --2- 172.21.15.28:0/1366028493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540cf100 0x7fe35406a8d0 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fe3540b5060 tx=0x7fe34802a110 comp rx=0 tx=0).stop 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 -- 172.21.15.28:0/1366028493 shutdown_connections 2024-09-13T17:20:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 --2- 172.21.15.28:0/1366028493 >> v2:172.21.15.189:3300/0 conn(0x7fe35415d890 0x7fe3540cf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 --2- 172.21.15.28:0/1366028493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540cf100 0x7fe35406a8d0 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 -- 172.21.15.28:0/1366028493 >> 172.21.15.28:0/1366028493 conn(0x7fe3540b4390 msgr2=0x7fe3540b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 -- 172.21.15.28:0/1366028493 shutdown_connections 2024-09-13T17:20:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.541+0000 7fe35aa00640 1 -- 172.21.15.28:0/1366028493 wait complete. 2024-09-13T17:20:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.542+0000 7fe35aa00640 1 Processor -- start 2024-09-13T17:20:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 -- start start 2024-09-13T17:20:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3540cf100 0x7fe35414ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35415d890 0x7fe35414c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35414ba40 con 0x7fe3540d0d70 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35414ae30 con 0x7fe35415d890 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.543+0000 7fe35aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe35414b110 con 0x7fe3540cf100 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49872/0 (socket says 172.21.15.28:49872) 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 -- 172.21.15.28:0/2248906644 learned_addr learned my addr 172.21.15.28:0/2248906644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe359600640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3540cf100 0x7fe35414ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe35a000640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35415d890 0x7fe35414c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 -- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3540cf100 msgr2=0x7fe35414ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3540cf100 0x7fe35414ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 -- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35415d890 msgr2=0x7fe35414c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35415d890 0x7fe35414c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 -- 172.21.15.28:0/2248906644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe348009400 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.544+0000 7fe358c00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7fe34000e9b0 tx=0x7fe34000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.545+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340009670 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.545+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe34000c9a0 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.545+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000cd50 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.545+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35414b3f0 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.545+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe354159c60 con 0x7fe3540d0d70 2024-09-13T17:20:34.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.546+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe354061910 con 0x7fe3540d0d70 2024-09-13T17:20:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.547+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe340013470 con 0x7fe3540d0d70 2024-09-13T17:20:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.547+0000 7fe351600640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 0x7fe33406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.548+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe340089380 con 0x7fe3540d0d70 2024-09-13T17:20:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.548+0000 7fe359600640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 0x7fe33406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:34.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.549+0000 7fe359600640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 0x7fe33406d0f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe348034000 tx=0x7fe34802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:34.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.552+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe340054780 con 0x7fe3540d0d70 2024-09-13T17:20:34.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:34.744+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe3540cd250 con 0x7fe3540d0d70 2024-09-13T17:20:35.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.531+0000 7fe351600640 1 -- 172.21.15.28:0/2248906644 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v175) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe340057da0 con 0x7fe3540d0d70 2024-09-13T17:20:35.533 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:20:35.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 msgr2=0x7fe33406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 0x7fe33406d0f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fe348034000 tx=0x7fe34802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:35.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 msgr2=0x7fe35414bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7fe34000e9b0 tx=0x7fe34000ee70 comp rx=0 tx=0).stop 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 shutdown_connections 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe33406ac30 0x7fe33406d0f0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe35415d890 0x7fe35414c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe3540d0d70 0x7fe35414bed0 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 --2- 172.21.15.28:0/2248906644 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe3540cf100 0x7fe35414ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 >> 172.21.15.28:0/2248906644 conn(0x7fe3540b4390 msgr2=0x7fe3540b3c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 shutdown_connections 2024-09-13T17:20:35.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.533+0000 7fe35aa00640 1 -- 172.21.15.28:0/2248906644 wait complete. 2024-09-13T17:20:35.552 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:20:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.707+0000 7fa9a7a00640 1 Processor -- start 2024-09-13T17:20:35.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.707+0000 7fa9a7a00640 1 -- start start 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a7a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa9a00cabc0 0x7fa9a006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a7a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9a0070cc0 0x7fa9a0071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a7a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a015e100 con 0x7fa9a015a540 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a7a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a00adb00 con 0x7fa9a0070cc0 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a7a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a0071670 con 0x7fa9a00cabc0 2024-09-13T17:20:35.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9a0070cc0 0x7fa9a0071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa9a0070cc0 0x7fa9a0071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49876/0 (socket says 172.21.15.28:49876) 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.708+0000 7fa9a5a00640 1 -- 172.21.15.28:0/504359379 learned_addr learned my addr 172.21.15.28:0/504359379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.709+0000 7fa99fe00640 1 --2- 172.21.15.28:0/504359379 >> v2:172.21.15.189:3300/0 conn(0x7fa9a00cabc0 0x7fa9a006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.709+0000 7fa9a5a00640 1 -- 172.21.15.28:0/504359379 >> v2:172.21.15.189:3300/0 conn(0x7fa9a00cabc0 msgr2=0x7fa9a006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.709+0000 7fa9a5a00640 1 --2- 172.21.15.28:0/504359379 >> v2:172.21.15.189:3300/0 conn(0x7fa9a00cabc0 0x7fa9a006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.709+0000 7fa9a5a00640 1 -- 172.21.15.28:0/504359379 >> v1:172.21.15.189:6789/0 conn(0x7fa9a015a540 legacy=0x7fa9a015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.709+0000 7fa9a5a00640 1 -- 172.21.15.28:0/504359379 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a015e3e0 con 0x7fa9a0070cc0 2024-09-13T17:20:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.710+0000 7fa9a5a00640 1 --2- 172.21.15.28:0/504359379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a0070cc0 0x7fa9a0071130 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00ae250 tx=0x7fa99802a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f21d86cb1c04d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.710+0000 7fa99f400640 1 -- 172.21.15.28:0/504359379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998031600 con 0x7fa9a0070cc0 2024-09-13T17:20:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.710+0000 7fa99f400640 1 -- 172.21.15.28:0/504359379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa99802d050 con 0x7fa9a0070cc0 2024-09-13T17:20:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.710+0000 7fa99f400640 1 -- 172.21.15.28:0/504359379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998031980 con 0x7fa9a0070cc0 2024-09-13T17:20:35.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 -- 172.21.15.28:0/504359379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a0070cc0 msgr2=0x7fa9a0071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/504359379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a0070cc0 0x7fa9a0071130 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00ae250 tx=0x7fa99802a6d0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 -- 172.21.15.28:0/504359379 shutdown_connections 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/504359379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a0070cc0 0x7fa9a0071130 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/504359379 >> v2:172.21.15.189:3300/0 conn(0x7fa9a00cabc0 0x7fa9a006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 -- 172.21.15.28:0/504359379 >> 172.21.15.28:0/504359379 conn(0x7fa9a0069e80 msgr2=0x7fa9a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 -- 172.21.15.28:0/504359379 shutdown_connections 2024-09-13T17:20:35.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.711+0000 7fa9a7a00640 1 -- 172.21.15.28:0/504359379 wait complete. 2024-09-13T17:20:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.712+0000 7fa9a7a00640 1 Processor -- start 2024-09-13T17:20:35.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 -- start start 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9a0070cc0 0x7fa9a01f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a015a540 0x7fa9a00b00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01482c0 con 0x7fa9a00cabc0 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0147020 con 0x7fa9a015a540 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.713+0000 7fa9a7a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a01476b0 con 0x7fa9a0070cc0 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49878/0 (socket says 172.21.15.28:49878) 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 -- 172.21.15.28:0/1631493440 learned_addr learned my addr 172.21.15.28:0/1631493440 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa9a5a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a015a540 0x7fa9a00b00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa9a5000640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9a0070cc0 0x7fa9a01f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 -- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9a0070cc0 msgr2=0x7fa9a01f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9a0070cc0 0x7fa9a01f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 -- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a015a540 msgr2=0x7fa9a00b00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a015a540 0x7fa9a00b00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 -- 172.21.15.28:0/1631493440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998009b30 con 0x7fa9a00cabc0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.714+0000 7fa99fe00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00d800 tx=0x7fa98c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.715+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00b6c0 con 0x7fa9a00cabc0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.715+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa98c00b820 con 0x7fa9a00cabc0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.715+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a0147990 con 0x7fa9a00cabc0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.715+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa98c00bbb0 con 0x7fa9a00cabc0 2024-09-13T17:20:35.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.715+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a0147f50 con 0x7fa9a00cabc0 2024-09-13T17:20:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.716+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a0071130 con 0x7fa9a00cabc0 2024-09-13T17:20:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.717+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa98c00bd10 con 0x7fa9a00cabc0 2024-09-13T17:20:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.718+0000 7fa99cc00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 0x7fa98006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.718+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa98c089560 con 0x7fa9a00cabc0 2024-09-13T17:20:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.718+0000 7fa9a5000640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 0x7fa98006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.719+0000 7fa9a5000640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 0x7fa98006d0c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa994000f60 tx=0x7fa994009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.722+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa98c054a10 con 0x7fa9a00cabc0 2024-09-13T17:20:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.962+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa9a0067040 con 0x7fa9a00cabc0 2024-09-13T17:20:35.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.962+0000 7fa99cc00640 1 -- 172.21.15.28:0/1631493440 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa98c058030 con 0x7fa9a00cabc0 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.965 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.966 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.971 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.973 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:20:35.974 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.975 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.976 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:20:35.978 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:20:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 msgr2=0x7fa98006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 0x7fa98006d0c0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fa994000f60 tx=0x7fa994009040 comp rx=0 tx=0).stop 2024-09-13T17:20:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 msgr2=0x7fa9a00b3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:35.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7fa98c00d800 tx=0x7fa98c00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 shutdown_connections 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa98006ac00 0x7fa98006d0c0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9a015a540 0x7fa9a00b00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9a00cabc0 0x7fa9a00b3020 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 --2- 172.21.15.28:0/1631493440 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9a0070cc0 0x7fa9a01f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.964+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 >> 172.21.15.28:0/1631493440 conn(0x7fa9a0069e80 msgr2=0x7fa9a015e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.965+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 shutdown_connections 2024-09-13T17:20:35.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:35.965+0000 7fa9a7a00640 1 -- 172.21.15.28:0/1631493440 wait complete. 2024-09-13T17:20:35.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.135+0000 7f4e02c00640 1 Processor -- start 2024-09-13T17:20:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.136+0000 7f4e02c00640 1 -- start start 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.137+0000 7f4e02c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.137+0000 7f4e02c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc159820 0x7f4dfc0b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.137+0000 7f4e02c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dfc0c3850 con 0x7f4dfc159440 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.137+0000 7f4e02c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dfc0b2d60 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.137+0000 7f4e02c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc15a930 con 0x7f4dfc159820 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e02200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc159820 0x7f4dfc0b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:41474/0 (socket says 172.21.15.28:41474) 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 -- 172.21.15.28:0/247433066 learned_addr learned my addr 172.21.15.28:0/247433066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e01800640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f4dfc159440 0x7f4dfc0b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:33794/0 (socket says 172.21.15.28:33794) 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 -- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc159820 msgr2=0x7f4dfc0b2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 --2- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc159820 0x7f4dfc0b2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 -- 172.21.15.28:0/247433066 >> v1:172.21.15.189:6789/0 conn(0x7f4dfc159440 legacy=0x7f4dfc0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.138+0000 7f4e00e00640 1 -- 172.21.15.28:0/247433066 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dfc159d20 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.139+0000 7f4e00e00640 1 --2- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009870 tx=0x7f4de802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d91ef28f9914fb2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.139+0000 7f4dfbe00640 1 -- 172.21.15.28:0/247433066 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de8032640 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.139+0000 7f4dfbe00640 1 -- 172.21.15.28:0/247433066 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4de80327a0 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.139+0000 7f4dfbe00640 1 -- 172.21.15.28:0/247433066 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4de8032b30 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 -- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc0b0b50 msgr2=0x7f4dfc0b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 --2- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4de8009870 tx=0x7f4de802a300 comp rx=0 tx=0).stop 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 -- 172.21.15.28:0/247433066 shutdown_connections 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 --2- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc159820 0x7f4dfc0b2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 --2- 172.21.15.28:0/247433066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc0b0b50 0x7f4dfc0b22e0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 -- 172.21.15.28:0/247433066 >> 172.21.15.28:0/247433066 conn(0x7f4dfc072e40 msgr2=0x7f4dfc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 -- 172.21.15.28:0/247433066 shutdown_connections 2024-09-13T17:20:36.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.140+0000 7f4e02c00640 1 -- 172.21.15.28:0/247433066 wait complete. 2024-09-13T17:20:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.142+0000 7f4e02c00640 1 Processor -- start 2024-09-13T17:20:36.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.142+0000 7f4e02c00640 1 -- start start 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc0b0b50 0x7f4dfc155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc159440 0x7f4dfc14da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc150790 con 0x7f4dfc159820 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc14fe80 con 0x7f4dfc159440 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dfc158890 con 0x7f4dfc0b0b50 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49884/0 (socket says 172.21.15.28:49884) 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.143+0000 7f4e02200640 1 -- 172.21.15.28:0/1480841511 learned_addr learned my addr 172.21.15.28:0/1480841511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:36.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e00e00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc159440 0x7f4dfc14da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e01800640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc0b0b50 0x7f4dfc155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 -- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc0b0b50 msgr2=0x7f4dfc155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc0b0b50 0x7f4dfc155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 -- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc159440 msgr2=0x7f4dfc14da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc159440 0x7f4dfc14da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 -- 172.21.15.28:0/1480841511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de80094d0 con 0x7f4dfc159820 2024-09-13T17:20:36.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4e02200640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4df400ebe0 tx=0x7f4df400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4009a70 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df400f040 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.144+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4df4014480 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.145+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dfc1500d0 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.145+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dfc155bd0 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.146+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dfc1508d0 con 0x7f4dfc159820 2024-09-13T17:20:36.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.146+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4df40145e0 con 0x7f4dfc159820 2024-09-13T17:20:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.147+0000 7f4df9600640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 0x7f4ddc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.147+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4df4089c20 con 0x7f4dfc159820 2024-09-13T17:20:36.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.148+0000 7f4e01800640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 0x7f4ddc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.148+0000 7f4e01800640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 0x7f4ddc06d0c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4df0000f60 tx=0x7f4df0000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.152+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4dfc1508d0 con 0x7f4dfc159820 2024-09-13T17:20:36.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.332+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dfc1508d0 con 0x7f4dfc159820 2024-09-13T17:20:36.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.333+0000 7f4df9600640 1 -- 172.21.15.28:0/1480841511 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v175) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f4dfc1508d0 con 0x7f4dfc159820 2024-09-13T17:20:36.335 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:36.335 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":175,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:35.526626+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":165,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 msgr2=0x7f4ddc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 0x7f4ddc06d0c0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f4df0000f60 tx=0x7f4df0000ed0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 msgr2=0x7f4dfc142b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f4df400ebe0 tx=0x7f4df400c490 comp rx=0 tx=0).stop 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 shutdown_connections 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4ddc06ac00 0x7f4ddc06d0c0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4dfc159820 0x7f4dfc142b10 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dfc159440 0x7f4dfc14da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 --2- 172.21.15.28:0/1480841511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4dfc0b0b50 0x7f4dfc155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 >> 172.21.15.28:0/1480841511 conn(0x7f4dfc072e40 msgr2=0x7f4dfc15ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 shutdown_connections 2024-09-13T17:20:36.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.335+0000 7f4e02c00640 1 -- 172.21.15.28:0/1480841511 wait complete. 2024-09-13T17:20:36.353 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:20:36.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:20:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.511+0000 7f3290800640 1 Processor -- start 2024-09-13T17:20:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.511+0000 7f3290800640 1 -- start start 2024-09-13T17:20:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32880cec20 0x7f3288071010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32880b2ce0 0x7f32880b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f328806c610 0x7f328806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328806cfe0 con 0x7f328806c610 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32880cf000 con 0x7f32880b2ce0 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f3290800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880b1980 con 0x7f32880cec20 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.512+0000 7f328e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f328806c610 0x7f328806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.513+0000 7f328e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f328806c610 0x7f328806caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49888/0 (socket says 172.21.15.28:49888) 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.513+0000 7f328e800640 1 -- 172.21.15.28:0/706547787 learned_addr learned my addr 172.21.15.28:0/706547787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.513+0000 7f328de00640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32880cec20 0x7f3288071010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.513+0000 7f328d400640 1 --2- 172.21.15.28:0/706547787 >> v2:172.21.15.189:3300/0 conn(0x7f32880b2ce0 0x7f32880b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.586+0000 7f328e800640 1 -- 172.21.15.28:0/706547787 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32880cec20 msgr2=0x7f3288071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.586+0000 7f328e800640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32880cec20 0x7f3288071010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.586+0000 7f328e800640 1 -- 172.21.15.28:0/706547787 >> v2:172.21.15.189:3300/0 conn(0x7f32880b2ce0 msgr2=0x7f32880b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:36.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.586+0000 7f328e800640 1 --2- 172.21.15.28:0/706547787 >> v2:172.21.15.189:3300/0 conn(0x7f32880b2ce0 0x7f32880b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.586+0000 7f328e800640 1 -- 172.21.15.28:0/706547787 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880b31e0 con 0x7f328806c610 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.587+0000 7f328e800640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f328806c610 0x7f328806caa0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f328806ad70 tx=0x7f328402ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ec4e5eca6b5bd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.587+0000 7f328ca00640 1 -- 172.21.15.28:0/706547787 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32840094e0 con 0x7f328806c610 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.587+0000 7f328ca00640 1 -- 172.21.15.28:0/706547787 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3284007720 con 0x7f328806c610 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.587+0000 7f328ca00640 1 -- 172.21.15.28:0/706547787 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3284007ab0 con 0x7f328806c610 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 -- 172.21.15.28:0/706547787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f328806c610 msgr2=0x7f328806caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f328806c610 0x7f328806caa0 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f328806ad70 tx=0x7f328402ad40 comp rx=0 tx=0).stop 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 -- 172.21.15.28:0/706547787 shutdown_connections 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f328806c610 0x7f328806caa0 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 --2- 172.21.15.28:0/706547787 >> v2:172.21.15.189:3300/0 conn(0x7f32880b2ce0 0x7f32880b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 --2- 172.21.15.28:0/706547787 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32880cec20 0x7f3288071010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 -- 172.21.15.28:0/706547787 >> 172.21.15.28:0/706547787 conn(0x7f3288069e80 msgr2=0x7f328806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 -- 172.21.15.28:0/706547787 shutdown_connections 2024-09-13T17:20:36.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.588+0000 7f3290800640 1 -- 172.21.15.28:0/706547787 wait complete. 2024-09-13T17:20:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.590+0000 7f3290800640 1 Processor -- start 2024-09-13T17:20:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.590+0000 7f3290800640 1 -- start start 2024-09-13T17:20:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f328806c610 0x7f32881fe9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32880cec20 0x7f32881f4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3288168820 con 0x7f32880b2ce0 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32881f5970 con 0x7f32880cec20 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f3290800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32880a94e0 con 0x7f328806c610 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49900/0 (socket says 172.21.15.28:49900) 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 -- 172.21.15.28:0/3717534865 learned_addr learned my addr 172.21.15.28:0/3717534865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328e800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32880cec20 0x7f32881f4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328de00640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f328806c610 0x7f32881fe9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 -- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f328806c610 msgr2=0x7f32881fe9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f328806c610 0x7f32881fe9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 -- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32880cec20 msgr2=0x7f32881f4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32880cec20 0x7f32881f4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.591+0000 7f328d400640 1 -- 172.21.15.28:0/3717534865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328400aa90 con 0x7f32880b2ce0 2024-09-13T17:20:36.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f328d400640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f3278011a60 tx=0x7f3278011f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327800fbc0 con 0x7f32880b2ce0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3278013040 con 0x7f32880b2ce0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3278019400 con 0x7f32880b2ce0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32880a97c0 con 0x7f32880b2ce0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.592+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32880a9d80 con 0x7f32880b2ce0 2024-09-13T17:20:36.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.593+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32880a9f40 con 0x7f32880b2ce0 2024-09-13T17:20:36.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.594+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3278019560 con 0x7f32880b2ce0 2024-09-13T17:20:36.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.595+0000 7f3282000640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 0x7f326c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.595+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f327808ce80 con 0x7f32880b2ce0 2024-09-13T17:20:36.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.595+0000 7f328de00640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 0x7f326c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.596+0000 7f328de00640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 0x7f326c06d0c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f327c006fd0 tx=0x7f327c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.599+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32780583c0 con 0x7f32880b2ce0 2024-09-13T17:20:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.779+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f328806caa0 con 0x7f32880b2ce0 2024-09-13T17:20:36.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.780+0000 7f3282000640 1 -- 172.21.15.28:0/3717534865 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v175) v1 ==== 93+0+10 (secure 0 0 0) 0x7f327801e0a0 con 0x7f32880b2ce0 2024-09-13T17:20:36.782 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:20:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 msgr2=0x7f326c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 0x7f326c06d0c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f327c006fd0 tx=0x7f327c008040 comp rx=0 tx=0).stop 2024-09-13T17:20:36.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 msgr2=0x7f3288201740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f3278011a60 tx=0x7f3278011f20 comp rx=0 tx=0).stop 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 shutdown_connections 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f326c06ac00 0x7f326c06d0c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32880cec20 0x7f32881f4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32880b2ce0 0x7f3288201740 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 --2- 172.21.15.28:0/3717534865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f328806c610 0x7f32881fe9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 >> 172.21.15.28:0/3717534865 conn(0x7f3288069e80 msgr2=0x7f328816b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:36.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 shutdown_connections 2024-09-13T17:20:36.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.782+0000 7f3290800640 1 -- 172.21.15.28:0/3717534865 wait complete. 2024-09-13T17:20:36.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:20:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.954+0000 7f06aa000640 1 Processor -- start 2024-09-13T17:20:36.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.954+0000 7f06aa000640 1 -- start start 2024-09-13T17:20:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.955+0000 7f06aa000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 0x7f06a406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.955+0000 7f06aa000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 0x7f06a4070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.955+0000 7f06aa000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a406f920 con 0x7f06a406f3f0 2024-09-13T17:20:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.955+0000 7f06aa000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a415b290 con 0x7f06a4159770 2024-09-13T17:20:36.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.955+0000 7f06aa000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a415b570 con 0x7f06a415cd80 2024-09-13T17:20:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f069be00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f06a406f3f0 0x7f06a4159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:33806/0 (socket says 172.21.15.28:33806) 2024-09-13T17:20:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 0x7f06a406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f069be00640 1 -- 172.21.15.28:0/207948478 learned_addr learned my addr 172.21.15.28:0/207948478 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3e00640 1 --2- 172.21.15.28:0/207948478 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 0x7f06a4070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a2a00640 1 -- 172.21.15.28:0/207948478 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766973375 0 0) 0x7f06a406f920 con 0x7f06a406f3f0 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a2a00640 1 -- 172.21.15.28:0/207948478 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003520 con 0x7f06a406f3f0 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3e00640 1 -- 172.21.15.28:0/207948478 >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 msgr2=0x7f06a406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3e00640 1 --2- 172.21.15.28:0/207948478 >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 0x7f06a406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3e00640 1 -- 172.21.15.28:0/207948478 >> v1:172.21.15.189:6789/0 conn(0x7f06a406f3f0 legacy=0x7f06a4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3e00640 1 -- 172.21.15.28:0/207948478 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a415b850 con 0x7f06a415cd80 2024-09-13T17:20:36.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.956+0000 7f06a3400640 1 --2- 172.21.15.28:0/207948478 >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 0x7f06a406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.957+0000 7f06a3e00640 1 --2- 172.21.15.28:0/207948478 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 0x7f06a4070260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f069c009d90 tx=0x7f069c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0153f77c5b4ffd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.957+0000 7f06a2a00640 1 -- 172.21.15.28:0/207948478 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c031600 con 0x7f06a415cd80 2024-09-13T17:20:36.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06a2a00640 1 -- 172.21.15.28:0/207948478 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c02d050 con 0x7f06a415cd80 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06a2a00640 1 -- 172.21.15.28:0/207948478 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069c031980 con 0x7f06a415cd80 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 -- 172.21.15.28:0/207948478 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 msgr2=0x7f06a4070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 --2- 172.21.15.28:0/207948478 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 0x7f06a4070260 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f069c009d90 tx=0x7f069c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 -- 172.21.15.28:0/207948478 shutdown_connections 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 --2- 172.21.15.28:0/207948478 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a415cd80 0x7f06a4070260 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 --2- 172.21.15.28:0/207948478 >> v2:172.21.15.189:3300/0 conn(0x7f06a4159770 0x7f06a406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 -- 172.21.15.28:0/207948478 >> 172.21.15.28:0/207948478 conn(0x7f06a40b0270 msgr2=0x7f06a40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 -- 172.21.15.28:0/207948478 shutdown_connections 2024-09-13T17:20:36.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.958+0000 7f06aa000640 1 -- 172.21.15.28:0/207948478 wait complete. 2024-09-13T17:20:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.960+0000 7f06aa000640 1 Processor -- start 2024-09-13T17:20:36.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.960+0000 7f06aa000640 1 -- start start 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a4159770 0x7f06a414e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06a415cd80 0x7f06a4144d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a414c220 con 0x7f06a406f3f0 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a414da60 con 0x7f06a415cd80 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06aa000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a414dd40 con 0x7f06a4159770 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06a3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49908/0 (socket says 172.21.15.28:49908) 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.961+0000 7f06a3400640 1 -- 172.21.15.28:0/4216694655 learned_addr learned my addr 172.21.15.28:0/4216694655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3e00640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06a415cd80 0x7f06a4144d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f069be00640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a4159770 0x7f06a414e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 -- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a4159770 msgr2=0x7f06a414e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a4159770 0x7f06a414e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 -- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06a415cd80 msgr2=0x7f06a4144d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06a415cd80 0x7f06a4144d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 -- 172.21.15.28:0/4216694655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0694009400 con 0x7f06a406f3f0 2024-09-13T17:20:36.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06a3400640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f069400ebe0 tx=0x7f069400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694009760 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069400cc30 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069400eee0 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c009a70 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.962+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a414e300 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.963+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a406fd20 con 0x7f06a406f3f0 2024-09-13T17:20:36.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.964+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0694013470 con 0x7f06a406f3f0 2024-09-13T17:20:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.965+0000 7f069b400640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 0x7f068006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.965+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0694089370 con 0x7f06a406f3f0 2024-09-13T17:20:36.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.966+0000 7f069be00640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 0x7f068006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:36.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.966+0000 7f069be00640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 0x7f068006d0c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f06a40b3420 tx=0x7f068c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:36.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:36.969+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0694054890 con 0x7f06a406f3f0 2024-09-13T17:20:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.150+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f06a40cf170 con 0x7f06a406f3f0 2024-09-13T17:20:37.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.150+0000 7f069b400640 1 -- 172.21.15.28:0/4216694655 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v175) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0694057eb0 con 0x7f06a406f3f0 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:20:37.153 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 msgr2=0x7f068006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 0x7f068006d0c0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f06a40b3420 tx=0x7f068c009040 comp rx=0 tx=0).stop 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 msgr2=0x7f06a40b1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f069400ebe0 tx=0x7f069400c520 comp rx=0 tx=0).stop 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 shutdown_connections 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f068006ac00 0x7f068006d0c0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06a415cd80 0x7f06a4144d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f06a4159770 0x7f06a414e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 --2- 172.21.15.28:0/4216694655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06a406f3f0 0x7f06a40b1f50 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 >> 172.21.15.28:0/4216694655 conn(0x7f06a40b0270 msgr2=0x7f06a406c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:37.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 shutdown_connections 2024-09-13T17:20:37.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.153+0000 7f06aa000640 1 -- 172.21.15.28:0/4216694655 wait complete. 2024-09-13T17:20:37.172 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:20:37.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:20:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.330+0000 7fb30f000640 1 Processor -- start 2024-09-13T17:20:37.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.330+0000 7fb30f000640 1 -- start start 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.331+0000 7fb30f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb308159970 0x7fb3080b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.331+0000 7fb30f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb3080b2630 0x7fb30815a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb30f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30815a4c0 con 0x7fb308159590 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb30f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3080b2ae0 con 0x7fb308159970 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb30f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb308071390 con 0x7fb3080b2630 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb307e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb308159590 0x7fb308070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38466/0 (socket says 172.21.15.28:38466) 2024-09-13T17:20:37.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb307e00640 1 -- 172.21.15.28:0/571824276 learned_addr learned my addr 172.21.15.28:0/571824276 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb2ff400640 1 --2- 172.21.15.28:0/571824276 >> v2:172.21.15.28:3300/0 conn(0x7fb308159970 0x7fb3080b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb30d000640 1 --2- 172.21.15.28:0/571824276 >> v2:172.21.15.189:3300/0 conn(0x7fb3080b2630 0x7fb30815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb307400640 1 -- 172.21.15.28:0/571824276 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96764419 0 0) 0x7fb30815a4c0 con 0x7fb308159590 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb307400640 1 -- 172.21.15.28:0/571824276 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2ec003520 con 0x7fb308159590 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb2ff400640 1 -- 172.21.15.28:0/571824276 >> v2:172.21.15.189:3300/0 conn(0x7fb3080b2630 msgr2=0x7fb30815a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb2ff400640 1 --2- 172.21.15.28:0/571824276 >> v2:172.21.15.189:3300/0 conn(0x7fb3080b2630 0x7fb30815a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb2ff400640 1 -- 172.21.15.28:0/571824276 >> v1:172.21.15.28:6789/0 conn(0x7fb308159590 legacy=0x7fb308070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.332+0000 7fb2ff400640 1 -- 172.21.15.28:0/571824276 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308071580 con 0x7fb308159970 2024-09-13T17:20:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.333+0000 7fb2ff400640 1 --2- 172.21.15.28:0/571824276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159970 0x7fb3080b20f0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fb308071860 tx=0x7fb2f002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b08c25b2bff1976 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb307400640 1 -- 172.21.15.28:0/571824276 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0032440 con 0x7fb308159970 2024-09-13T17:20:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb307400640 1 -- 172.21.15.28:0/571824276 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f00325a0 con 0x7fb308159970 2024-09-13T17:20:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb307400640 1 -- 172.21.15.28:0/571824276 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0032930 con 0x7fb308159970 2024-09-13T17:20:37.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 -- 172.21.15.28:0/571824276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159970 msgr2=0x7fb3080b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 --2- 172.21.15.28:0/571824276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159970 0x7fb3080b20f0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7fb308071860 tx=0x7fb2f002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 -- 172.21.15.28:0/571824276 shutdown_connections 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 --2- 172.21.15.28:0/571824276 >> v2:172.21.15.189:3300/0 conn(0x7fb3080b2630 0x7fb30815a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 --2- 172.21.15.28:0/571824276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159970 0x7fb3080b20f0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 -- 172.21.15.28:0/571824276 >> 172.21.15.28:0/571824276 conn(0x7fb308069e80 msgr2=0x7fb30806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 -- 172.21.15.28:0/571824276 shutdown_connections 2024-09-13T17:20:37.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.334+0000 7fb30f000640 1 -- 172.21.15.28:0/571824276 wait complete. 2024-09-13T17:20:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.336+0000 7fb30f000640 1 Processor -- start 2024-09-13T17:20:37.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.336+0000 7fb30f000640 1 -- start start 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 0x7fb3080ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb308159970 0x7fb30815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080ac790 con 0x7fb308159590 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080ab8e0 con 0x7fb3080b2630 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb30f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3080abb80 con 0x7fb308159970 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb2ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb2ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49912/0 (socket says 172.21.15.28:49912) 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb2ff400640 1 -- 172.21.15.28:0/3184830826 learned_addr learned my addr 172.21.15.28:0/3184830826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.337+0000 7fb307e00640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 0x7fb3080ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb30d000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb308159970 0x7fb30815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 -- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb308159970 msgr2=0x7fb30815bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb308159970 0x7fb30815bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 -- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 msgr2=0x7fb3080ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 0x7fb3080ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 -- 172.21.15.28:0/3184830826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0009400 con 0x7fb308159590 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb307e00640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 0x7fb3080ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:37.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb2ff400640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0000c00 tx=0x7fb2f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0030b00 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f0032e90 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f0030ea0 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3080abe60 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.338+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3080ac420 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.339+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3080ac900 con 0x7fb308159590 2024-09-13T17:20:37.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.340+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb2f0039040 con 0x7fb308159590 2024-09-13T17:20:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.341+0000 7fb304c00640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 0x7fb2e406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.341+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb2f00a9a10 con 0x7fb308159590 2024-09-13T17:20:37.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.342+0000 7fb307e00640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 0x7fb2e406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:37.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.342+0000 7fb307e00640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 0x7fb2e406d0f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80011e0 tx=0x7fb2f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:37.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.345+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb2f0074f30 con 0x7fb308159590 2024-09-13T17:20:37.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:37.525+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb30815cdc0 con 0x7fb308159590 2024-09-13T17:20:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.175+0000 7fb304c00640 1 -- 172.21.15.28:0/3184830826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v176) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb2f0006e10 con 0x7fb308159590 2024-09-13T17:20:38.177 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 msgr2=0x7fb2e406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 0x7fb2e406d0f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80011e0 tx=0x7fb2f8009040 comp rx=0 tx=0).stop 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 msgr2=0x7fb3080ab3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7fb2f0000c00 tx=0x7fb2f00046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 shutdown_connections 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb2e406ac30 0x7fb2e406d0f0 secure :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fb2f80011e0 tx=0x7fb2f8009040 comp rx=0 tx=0).stop 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb308159970 0x7fb30815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb308159590 0x7fb3080ab3a0 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 --2- 172.21.15.28:0/3184830826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3080b2630 0x7fb3080ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.179+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 >> 172.21.15.28:0/3184830826 conn(0x7fb308069e80 msgr2=0x7fb3081579a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.180+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 shutdown_connections 2024-09-13T17:20:38.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.180+0000 7fb30f000640 1 -- 172.21.15.28:0/3184830826 wait complete. 2024-09-13T17:20:38.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:20:38.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.362+0000 7f517be00640 1 Processor -- start 2024-09-13T17:20:38.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.362+0000 7f517be00640 1 -- start start 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f517c154a90 0x7f517c15b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 0x7f517c0af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517be00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f517c15bb10 con 0x7f517c06b360 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517be00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517c06b900 con 0x7f517c154a90 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c06bbe0 con 0x7f517c0b0b50 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f5173e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f517c06b360 0x7f517c0b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38482/0 (socket says 172.21.15.28:38482) 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f5173e00640 1 -- 172.21.15.28:0/1805584648 learned_addr learned my addr 172.21.15.28:0/1805584648 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517aa00640 1 --2- 172.21.15.28:0/1805584648 >> v2:172.21.15.189:3300/0 conn(0x7f517c154a90 0x7f517c15b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517b400640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 0x7f517c0af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517a000640 1 -- 172.21.15.28:0/1805584648 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3992146870 0 0) 0x7f517c15bb10 con 0x7f517c06b360 2024-09-13T17:20:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.364+0000 7f517a000640 1 -- 172.21.15.28:0/1805584648 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5160003520 con 0x7f517c06b360 2024-09-13T17:20:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.365+0000 7f517aa00640 1 -- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 msgr2=0x7f517c0af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.365+0000 7f517aa00640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 0x7f517c0af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.365+0000 7f517aa00640 1 -- 172.21.15.28:0/1805584648 >> v1:172.21.15.28:6789/0 conn(0x7f517c06b360 legacy=0x7f517c0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.365+0000 7f517aa00640 1 -- 172.21.15.28:0/1805584648 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f517c06bec0 con 0x7f517c154a90 2024-09-13T17:20:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.365+0000 7f517b400640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 0x7f517c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:38.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.366+0000 7f517aa00640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c154a90 0x7f517c15b5d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f516c0097a0 tx=0x7f516c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af32347b556681c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.367+0000 7f517a000640 1 -- 172.21.15.28:0/1805584648 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c032440 con 0x7f517c154a90 2024-09-13T17:20:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.367+0000 7f517a000640 1 -- 172.21.15.28:0/1805584648 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c0325a0 con 0x7f517c154a90 2024-09-13T17:20:38.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.367+0000 7f517a000640 1 -- 172.21.15.28:0/1805584648 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c032930 con 0x7f517c154a90 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 -- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c154a90 msgr2=0x7f517c15b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c154a90 0x7f517c15b5d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f516c0097a0 tx=0x7f516c02a180 comp rx=0 tx=0).stop 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 -- 172.21.15.28:0/1805584648 shutdown_connections 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c0b0b50 0x7f517c0af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 --2- 172.21.15.28:0/1805584648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c154a90 0x7f517c15b5d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 -- 172.21.15.28:0/1805584648 >> 172.21.15.28:0/1805584648 conn(0x7f517c072e40 msgr2=0x7f517c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 -- 172.21.15.28:0/1805584648 shutdown_connections 2024-09-13T17:20:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.368+0000 7f517be00640 1 -- 172.21.15.28:0/1805584648 wait complete. 2024-09-13T17:20:38.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.370+0000 7f517be00640 1 Processor -- start 2024-09-13T17:20:38.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.370+0000 7f517be00640 1 -- start start 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c0b0b50 0x7f517c1e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c154a90 0x7f517c15c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c155640 con 0x7f517c06b360 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517c13a820 con 0x7f517c0b0b50 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517c13ab00 con 0x7f517c154a90 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49920/0 (socket says 172.21.15.28:49920) 2024-09-13T17:20:38.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517aa00640 1 -- 172.21.15.28:0/1888876084 learned_addr learned my addr 172.21.15.28:0/1888876084 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f5173e00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c0b0b50 0x7f517c1e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.371+0000 7f517b400640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c154a90 0x7f517c15c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 -- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c154a90 msgr2=0x7f517c15c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c154a90 0x7f517c15c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 -- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c0b0b50 msgr2=0x7f517c1e6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c0b0b50 0x7f517c1e6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 -- 172.21.15.28:0/1888876084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174009a70 con 0x7f517c06b360 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517aa00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f516c02a770 tx=0x7f516c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c03e920 con 0x7f517c06b360 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f516c03eaa0 con 0x7f517c06b360 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f516c009400 con 0x7f517c06b360 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f516c03ee50 con 0x7f517c06b360 2024-09-13T17:20:38.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.372+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517c13b0c0 con 0x7f517c06b360 2024-09-13T17:20:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.373+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f517c158e80 con 0x7f517c06b360 2024-09-13T17:20:38.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.375+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f516c050430 con 0x7f517c06b360 2024-09-13T17:20:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.376+0000 7f5172a00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:38.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.376+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f516c0aa330 con 0x7f517c06b360 2024-09-13T17:20:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.376+0000 7f5173e00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:38.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.377+0000 7f5173e00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f517c15d550 tx=0x7f5164000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:38.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.379+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f516c075780 con 0x7f517c06b360 2024-09-13T17:20:38.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:38.558+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f517c0af4b0 con 0x7f517c06b360 2024-09-13T17:20:39.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.183+0000 7f5172a00640 1 -- 172.21.15.28:0/1888876084 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v177) v1 ==== 170+0+0 (secure 0 0 0) 0x7f516c078da0 con 0x7f517c06b360 2024-09-13T17:20:39.186 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 msgr2=0x7f515806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f517c15d550 tx=0x7f5164000bc0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 msgr2=0x7f517c153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f516c02a770 tx=0x7f516c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 shutdown_connections 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f515806ac00 0x7f515806d0c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517c154a90 0x7f517c15c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517c0b0b50 0x7f517c1e6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 --2- 172.21.15.28:0/1888876084 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517c06b360 0x7f517c153400 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.193+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 >> 172.21.15.28:0/1888876084 conn(0x7f517c072e40 msgr2=0x7f517c158770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.194+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 shutdown_connections 2024-09-13T17:20:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.194+0000 7f517be00640 1 -- 172.21.15.28:0/1888876084 wait complete. 2024-09-13T17:20:39.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:20:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.379+0000 7f6395000640 1 Processor -- start 2024-09-13T17:20:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.379+0000 7f6395000640 1 -- start start 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f6395000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f639015c9a0 0x7f63900ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f6395000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63900ad940 0x7f63900b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f6395000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63900b2680 con 0x7f63900b2230 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f6395000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900ace30 con 0x7f63900ad940 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f6395000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63900ad110 con 0x7f639015c9a0 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f638e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63900b2230 0x7f639006f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38484/0 (socket says 172.21.15.28:38484) 2024-09-13T17:20:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f638d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f63900ad940 0x7f63900b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.380+0000 7f638e800640 1 -- 172.21.15.28:0/2480128099 learned_addr learned my addr 172.21.15.28:0/2480128099 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638de00640 1 --2- 172.21.15.28:0/2480128099 >> v2:172.21.15.189:3300/0 conn(0x7f639015c9a0 0x7f63900ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638ca00640 1 -- 172.21.15.28:0/2480128099 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784042718 0 0) 0x7f6388001490 con 0x7f63900b2230 2024-09-13T17:20:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638d400640 1 -- 172.21.15.28:0/2480128099 >> v2:172.21.15.189:3300/0 conn(0x7f639015c9a0 msgr2=0x7f63900ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638d400640 1 --2- 172.21.15.28:0/2480128099 >> v2:172.21.15.189:3300/0 conn(0x7f639015c9a0 0x7f63900ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638d400640 1 -- 172.21.15.28:0/2480128099 >> v1:172.21.15.28:6789/0 conn(0x7f63900b2230 legacy=0x7f639006f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.381+0000 7f638d400640 1 -- 172.21.15.28:0/2480128099 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639006b170 con 0x7f63900ad940 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.382+0000 7f638d400640 1 --2- 172.21.15.28:0/2480128099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63900ad940 0x7f63900b1cf0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f637c0097a0 tx=0x7f637c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9296cbc38cbc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.382+0000 7f638ca00640 1 -- 172.21.15.28:0/2480128099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c032440 con 0x7f63900ad940 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.382+0000 7f638ca00640 1 -- 172.21.15.28:0/2480128099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f637c0325a0 con 0x7f63900ad940 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.382+0000 7f638ca00640 1 -- 172.21.15.28:0/2480128099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f637c032930 con 0x7f63900ad940 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 -- 172.21.15.28:0/2480128099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63900ad940 msgr2=0x7f63900b1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 --2- 172.21.15.28:0/2480128099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63900ad940 0x7f63900b1cf0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f637c0097a0 tx=0x7f637c02a270 comp rx=0 tx=0).stop 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 -- 172.21.15.28:0/2480128099 shutdown_connections 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 --2- 172.21.15.28:0/2480128099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63900ad940 0x7f63900b1cf0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 --2- 172.21.15.28:0/2480128099 >> v2:172.21.15.189:3300/0 conn(0x7f639015c9a0 0x7f63900ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 -- 172.21.15.28:0/2480128099 >> 172.21.15.28:0/2480128099 conn(0x7f63900b0270 msgr2=0x7f63900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 -- 172.21.15.28:0/2480128099 shutdown_connections 2024-09-13T17:20:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.383+0000 7f6395000640 1 -- 172.21.15.28:0/2480128099 wait complete. 2024-09-13T17:20:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.384+0000 7f6395000640 1 Processor -- start 2024-09-13T17:20:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 -- start start 2024-09-13T17:20:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 0x7f63901416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63900b2230 0x7f6390158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63901525c0 con 0x7f639015c9a0 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6390141be0 con 0x7f63900b2230 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.385+0000 7f6395000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f639013f0c0 con 0x7f63900ad940 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49936/0 (socket says 172.21.15.28:49936) 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 0x7f63901416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 -- 172.21.15.28:0/1661716498 learned_addr learned my addr 172.21.15.28:0/1661716498 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638d400640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63900b2230 0x7f6390158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 -- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 msgr2=0x7f63901416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 0x7f63901416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 -- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63900b2230 msgr2=0x7f6390158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63900b2230 0x7f6390158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638e800640 1 -- 172.21.15.28:0/1661716498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c009400 con 0x7f639015c9a0 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.386+0000 7f638de00640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 0x7f63901416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:39.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f638e800640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f638800c7d0 tx=0x7f638800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800a640 con 0x7f639015c9a0 2024-09-13T17:20:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638800a7c0 con 0x7f639015c9a0 2024-09-13T17:20:39.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6390070930 con 0x7f639015c9a0 2024-09-13T17:20:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638800abb0 con 0x7f639015c9a0 2024-09-13T17:20:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.387+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f639013f960 con 0x7f639015c9a0 2024-09-13T17:20:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.388+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639013f4c0 con 0x7f639015c9a0 2024-09-13T17:20:39.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.389+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f638800ad10 con 0x7f639015c9a0 2024-09-13T17:20:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.390+0000 7f6386000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 0x7f637006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.390+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f63880889a0 con 0x7f639015c9a0 2024-09-13T17:20:39.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.391+0000 7f638de00640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 0x7f637006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.391+0000 7f638de00640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 0x7f637006d0c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f63900b38f0 tx=0x7f6380008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.395+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6388053bf0 con 0x7f639015c9a0 2024-09-13T17:20:39.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.610+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f63900b2680 con 0x7f639015c9a0 2024-09-13T17:20:39.621 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 24 and data pool 25 2024-09-13T17:20:39.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.619+0000 7f6386000640 1 -- 172.21.15.28:0/1661716498 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v147) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6388018020 con 0x7f639015c9a0 2024-09-13T17:20:39.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.627+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 msgr2=0x7f637006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.627+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 0x7f637006d0c0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f63900b38f0 tx=0x7f6380008040 comp rx=0 tx=0).stop 2024-09-13T17:20:39.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.627+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 msgr2=0x7f63901420f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.627+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f638800c7d0 tx=0x7f638800cc90 comp rx=0 tx=0).stop 2024-09-13T17:20:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 shutdown_connections 2024-09-13T17:20:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f637006ac00 0x7f637006d0c0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f639015c9a0 0x7f63901420f0 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63900b2230 0x7f6390158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 --2- 172.21.15.28:0/1661716498 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63900ad940 0x7f63901416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.629+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 >> 172.21.15.28:0/1661716498 conn(0x7f63900b0270 msgr2=0x7f639006f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:39.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.630+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 shutdown_connections 2024-09-13T17:20:39.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.632+0000 7f6395000640 1 -- 172.21.15.28:0/1661716498 wait complete. 2024-09-13T17:20:39.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:39.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.812+0000 7f922dc00640 1 Processor -- start 2024-09-13T17:20:39.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.812+0000 7f922dc00640 1 -- start start 2024-09-13T17:20:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f922dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 0x7f922806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f922dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92280cabc0 0x7f922815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f922dc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f922815e100 con 0x7f922815edb0 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f922dc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92280adb00 con 0x7f92280cabc0 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f922dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f922815e3e0 con 0x7f922815a540 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92280cabc0 0x7f922815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f92280cabc0 0x7f922815e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49946/0 (socket says 172.21.15.28:49946) 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 -- 172.21.15.28:0/4087527086 learned_addr learned my addr 172.21.15.28:0/4087527086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f9226a00640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 0x7f922806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 -- 172.21.15.28:0/4087527086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 msgr2=0x7f922806c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 0x7f922806c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 -- 172.21.15.28:0/4087527086 >> v1:172.21.15.189:6789/0 conn(0x7f922815edb0 legacy=0x7f9228070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f921e000640 1 -- 172.21.15.28:0/4087527086 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f922806ce80 con 0x7f92280cabc0 2024-09-13T17:20:39.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.814+0000 7f9226a00640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 0x7f922806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.815+0000 7f921e000640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92280cabc0 0x7f922815e870 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1f2a9abfd16751 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f9226000640 1 -- 172.21.15.28:0/4087527086 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921002ae90 con 0x7f92280cabc0 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f9226000640 1 -- 172.21.15.28:0/4087527086 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9210032440 con 0x7f92280cabc0 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f9226000640 1 -- 172.21.15.28:0/4087527086 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92100327d0 con 0x7f92280cabc0 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 -- 172.21.15.28:0/4087527086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92280cabc0 msgr2=0x7f922815e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92280cabc0 0x7f922815e870 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f921002a180 comp rx=0 tx=0).stop 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 -- 172.21.15.28:0/4087527086 shutdown_connections 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92280cabc0 0x7f922815e870 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 --2- 172.21.15.28:0/4087527086 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f922815a540 0x7f922806c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 -- 172.21.15.28:0/4087527086 >> 172.21.15.28:0/4087527086 conn(0x7f9228069e80 msgr2=0x7f922806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.816+0000 7f922dc00640 1 -- 172.21.15.28:0/4087527086 shutdown_connections 2024-09-13T17:20:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.817+0000 7f922dc00640 1 -- 172.21.15.28:0/4087527086 wait complete. 2024-09-13T17:20:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.818+0000 7f922dc00640 1 Processor -- start 2024-09-13T17:20:39.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.818+0000 7f922dc00640 1 -- start start 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92280cabc0 0x7f92280b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 0x7f9228153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9228143330 con 0x7f922815a540 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281521e0 con 0x7f9228151c70 2024-09-13T17:20:39.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f922dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92281524c0 con 0x7f92280cabc0 2024-09-13T17:20:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f921e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f921e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49948/0 (socket says 172.21.15.28:49948) 2024-09-13T17:20:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.819+0000 7f921e000640 1 -- 172.21.15.28:0/1569147887 learned_addr learned my addr 172.21.15.28:0/1569147887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f9226a00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92280cabc0 0x7f92280b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f9227400640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 0x7f9228153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 -- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92280cabc0 msgr2=0x7f92280b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92280cabc0 0x7f92280b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 -- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 msgr2=0x7f9228153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 0x7f9228153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 -- 172.21.15.28:0/1569147887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92100094d0 con 0x7f922815a540 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f9227400640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 0x7f9228153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921e000640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f92100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.820+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9210030b80 con 0x7f922815a540 2024-09-13T17:20:39.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.821+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9210030ce0 con 0x7f922815a540 2024-09-13T17:20:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.821+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92281527a0 con 0x7f922815a540 2024-09-13T17:20:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.821+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921003d660 con 0x7f922815a540 2024-09-13T17:20:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.821+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9228152d60 con 0x7f922815a540 2024-09-13T17:20:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.822+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9228152600 con 0x7f922815a540 2024-09-13T17:20:39.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.823+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9210039040 con 0x7f922815a540 2024-09-13T17:20:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.823+0000 7f921f400640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 0x7f920406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:39.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.824+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92100aa0e0 con 0x7f922815a540 2024-09-13T17:20:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.824+0000 7f9226a00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 0x7f920406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:39.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.825+0000 7f9226a00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 0x7f920406d0f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f921800a970 tx=0x7f9218008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:39.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:39.828+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9228152600 con 0x7f922815a540 2024-09-13T17:20:40.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.006+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9228152600 con 0x7f922815a540 2024-09-13T17:20:40.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.007+0000 7f921f400640 1 -- 172.21.15.28:0/1569147887 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f9228152600 con 0x7f922815a540 2024-09-13T17:20:40.009 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:40.009 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":178,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:39.611980+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 msgr2=0x7f920406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 0x7f920406d0f0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f921800a970 tx=0x7f9218008040 comp rx=0 tx=0).stop 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 msgr2=0x7f9228142df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7f9210009870 tx=0x7f92100046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 shutdown_connections 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f920406ac30 0x7f920406d0f0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228151c70 0x7f9228153280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f922815a540 0x7f9228142df0 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 --2- 172.21.15.28:0/1569147887 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f92280cabc0 0x7f92280b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 >> 172.21.15.28:0/1569147887 conn(0x7f9228069e80 msgr2=0x7f92280cd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 shutdown_connections 2024-09-13T17:20:40.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.010+0000 7f922dc00640 1 -- 172.21.15.28:0/1569147887 wait complete. 2024-09-13T17:20:40.028 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:40.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.182+0000 7f7613800640 1 Processor -- start 2024-09-13T17:20:40.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.182+0000 7f7613800640 1 -- start start 2024-09-13T17:20:40.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c15a540 0x7f760c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f760c0cabc0 0x7f760c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f760c15edb0 0x7f760c070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c0adb00 con 0x7f760c15edb0 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f760c15e100 con 0x7f760c0cabc0 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.183+0000 7f7613800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c15e3e0 con 0x7f760c15a540 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f760c15edb0 0x7f760c070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7610e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c15a540 0x7f760c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f760be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f760c0cabc0 0x7f760c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f760c15edb0 0x7f760c070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49956/0 (socket says 172.21.15.28:49956) 2024-09-13T17:20:40.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 -- 172.21.15.28:0/880411340 learned_addr learned my addr 172.21.15.28:0/880411340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 -- 172.21.15.28:0/880411340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c15a540 msgr2=0x7f760c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 --2- 172.21.15.28:0/880411340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c15a540 0x7f760c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 -- 172.21.15.28:0/880411340 >> v2:172.21.15.189:3300/0 conn(0x7f760c0cabc0 msgr2=0x7f760c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 --2- 172.21.15.28:0/880411340 >> v2:172.21.15.189:3300/0 conn(0x7f760c0cabc0 0x7f760c15e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.184+0000 7f7611800640 1 -- 172.21.15.28:0/880411340 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c06ce80 con 0x7f760c15edb0 2024-09-13T17:20:40.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.185+0000 7f7611800640 1 --2- 172.21.15.28:0/880411340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c15edb0 0x7f760c070cc0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f760c06ae00 tx=0x7f760402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d84e32428ce4ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f760b400640 1 -- 172.21.15.28:0/880411340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604031600 con 0x7f760c15edb0 2024-09-13T17:20:40.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f760b400640 1 -- 172.21.15.28:0/880411340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760402d050 con 0x7f760c15edb0 2024-09-13T17:20:40.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f760b400640 1 -- 172.21.15.28:0/880411340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604031980 con 0x7f760c15edb0 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 -- 172.21.15.28:0/880411340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c15edb0 msgr2=0x7f760c070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 --2- 172.21.15.28:0/880411340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c15edb0 0x7f760c070cc0 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7f760c06ae00 tx=0x7f760402a610 comp rx=0 tx=0).stop 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 -- 172.21.15.28:0/880411340 shutdown_connections 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 --2- 172.21.15.28:0/880411340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c15edb0 0x7f760c070cc0 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 --2- 172.21.15.28:0/880411340 >> v2:172.21.15.189:3300/0 conn(0x7f760c0cabc0 0x7f760c15e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 --2- 172.21.15.28:0/880411340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c15a540 0x7f760c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 -- 172.21.15.28:0/880411340 >> 172.21.15.28:0/880411340 conn(0x7f760c069e80 msgr2=0x7f760c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 -- 172.21.15.28:0/880411340 shutdown_connections 2024-09-13T17:20:40.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.186+0000 7f7613800640 1 -- 172.21.15.28:0/880411340 wait complete. 2024-09-13T17:20:40.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.188+0000 7f7613800640 1 Processor -- start 2024-09-13T17:20:40.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.188+0000 7f7613800640 1 -- start start 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 0x7f760c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 0x7f760c143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c15b670 con 0x7f760c0cabc0 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f760c15b8c0 con 0x7f760c15a540 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7613800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f760c15bba0 con 0x7f760c0b0150 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7610e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7610e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49962/0 (socket says 172.21.15.28:49962) 2024-09-13T17:20:40.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.189+0000 7f7610e00640 1 -- 172.21.15.28:0/1532552356 learned_addr learned my addr 172.21.15.28:0/1532552356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f760be00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 0x7f760c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 -- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 msgr2=0x7f760c143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7611800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 0x7f760c143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 0x7f760c143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 -- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 msgr2=0x7f760c14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 0x7f760c14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 -- 172.21.15.28:0/1532552356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604009b30 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f760be00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 0x7f760c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7611800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 0x7f760c143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7610e00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f760000e9b0 tx=0x7f760000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600009670 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760000c9a0 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f760c15be80 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000cd50 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.190+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f760c0b05e0 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.191+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760c0b0800 con 0x7f760c0cabc0 2024-09-13T17:20:40.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.192+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7600013470 con 0x7f760c0cabc0 2024-09-13T17:20:40.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.193+0000 7f7608c00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 0x7f75ec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.193+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76000894d0 con 0x7f760c0cabc0 2024-09-13T17:20:40.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.194+0000 7f760be00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 0x7f75ec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.194+0000 7f760be00640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 0x7f75ec06d0f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f760c1f5a30 tx=0x7f75f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.197+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f76000546a0 con 0x7f760c0cabc0 2024-09-13T17:20:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.377+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f760c15e870 con 0x7f760c0cabc0 2024-09-13T17:20:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.379+0000 7f7608c00640 1 -- 172.21.15.28:0/1532552356 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f7600057cc0 con 0x7f760c0cabc0 2024-09-13T17:20:40.381 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:40.381 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":178,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:39.611980+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 msgr2=0x7f75ec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 0x7f75ec06d0f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f760c1f5a30 tx=0x7f75f8008040 comp rx=0 tx=0).stop 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 msgr2=0x7f760c14cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7f760000e9b0 tx=0x7f760000ee70 comp rx=0 tx=0).stop 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 shutdown_connections 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f75ec06ac30 0x7f75ec06d0f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f760c0b0150 0x7f760c143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f760c15a540 0x7f760c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 --2- 172.21.15.28:0/1532552356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f760c0cabc0 0x7f760c14cfc0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 >> 172.21.15.28:0/1532552356 conn(0x7f760c069e80 msgr2=0x7f760c0ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 shutdown_connections 2024-09-13T17:20:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.381+0000 7f7613800640 1 -- 172.21.15.28:0/1532552356 wait complete. 2024-09-13T17:20:40.399 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:20:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.556+0000 7f29cea00640 1 Processor -- start 2024-09-13T17:20:40.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.556+0000 7f29cea00640 1 -- start start 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29c80acf70 0x7f29c806fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c81604b0 con 0x7f29c81600d0 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cea00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c80cd740 con 0x7f29c80c98f0 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c80ccee0 con 0x7f29c80acf70 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29c80acf70 0x7f29c806fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f29c81600d0 0x7f29c80cab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38498/0 (socket says 172.21.15.28:38498) 2024-09-13T17:20:40.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29cca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f29c80acf70 0x7f29c806fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49972/0 (socket says 172.21.15.28:49972) 2024-09-13T17:20:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.557+0000 7f29c7e00640 1 -- 172.21.15.28:0/1653456935 learned_addr learned my addr 172.21.15.28:0/1653456935 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.558+0000 7f29cca00640 1 -- 172.21.15.28:0/1653456935 >> v1:172.21.15.189:6789/0 conn(0x7f29c80c98f0 legacy=0x7f29c80ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.558+0000 7f29cca00640 1 -- 172.21.15.28:0/1653456935 >> v1:172.21.15.28:6789/0 conn(0x7f29c81600d0 legacy=0x7f29c80cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.558+0000 7f29cca00640 1 -- 172.21.15.28:0/1653456935 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c80cd180 con 0x7f29c80acf70 2024-09-13T17:20:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.559+0000 7f29cca00640 1 --2- 172.21.15.28:0/1653456935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c806fb00 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009d90 tx=0x7f29c002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccc942402f19d99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.559+0000 7f29c6a00640 1 -- 172.21.15.28:0/1653456935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0007740 con 0x7f29c80acf70 2024-09-13T17:20:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.559+0000 7f29c6a00640 1 -- 172.21.15.28:0/1653456935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c002d050 con 0x7f29c80acf70 2024-09-13T17:20:40.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.559+0000 7f29c6a00640 1 -- 172.21.15.28:0/1653456935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c0007ac0 con 0x7f29c80acf70 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 -- 172.21.15.28:0/1653456935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 msgr2=0x7f29c806fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 --2- 172.21.15.28:0/1653456935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c806fb00 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009d90 tx=0x7f29c002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 -- 172.21.15.28:0/1653456935 shutdown_connections 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 --2- 172.21.15.28:0/1653456935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c806fb00 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 -- 172.21.15.28:0/1653456935 >> 172.21.15.28:0/1653456935 conn(0x7f29c80ce900 msgr2=0x7f29c80cece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 -- 172.21.15.28:0/1653456935 shutdown_connections 2024-09-13T17:20:40.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.560+0000 7f29cea00640 1 -- 172.21.15.28:0/1653456935 wait complete. 2024-09-13T17:20:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.561+0000 7f29cea00640 1 Processor -- start 2024-09-13T17:20:40.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.562+0000 7f29cea00640 1 -- start start 2024-09-13T17:20:40.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.562+0000 7f29cea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29c80c98f0 0x7f29c8159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 0x7f29c8070e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c814c340 con 0x7f29c80acf70 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49974/0 (socket says 172.21.15.28:49974) 2024-09-13T17:20:40.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c815b860 con 0x7f29c81600d0 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c815a7c0 con 0x7f29c80c98f0 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 -- 172.21.15.28:0/3391956959 learned_addr learned my addr 172.21.15.28:0/3391956959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 -- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29c80c98f0 msgr2=0x7f29c8159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29c80c98f0 0x7f29c8159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 -- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 msgr2=0x7f29c8070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cca00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 0x7f29c8070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 0x7f29c8070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29c7e00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c0009a70 con 0x7f29c80acf70 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.563+0000 7f29cca00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 0x7f29c8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29c7e00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f29b800c480 tx=0x7f29b800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b8005ce0 con 0x7f29c80acf70 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b8005e40 con 0x7f29c80acf70 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b80072f0 con 0x7f29c80acf70 2024-09-13T17:20:40.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c815aaa0 con 0x7f29c80acf70 2024-09-13T17:20:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.564+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c815b060 con 0x7f29c80acf70 2024-09-13T17:20:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.565+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c80c9d60 con 0x7f29c80acf70 2024-09-13T17:20:40.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.566+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f29b8007450 con 0x7f29c80acf70 2024-09-13T17:20:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.567+0000 7f29bfe00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 0x7f29a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.567+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f29b8088880 con 0x7f29c80acf70 2024-09-13T17:20:40.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.568+0000 7f29c7400640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 0x7f29a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:40.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.568+0000 7f29c7400640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 0x7f29a806d0c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b40079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:40.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.571+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f29b8053b00 con 0x7f29c80acf70 2024-09-13T17:20:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:40.789+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f29c80ad540 con 0x7f29c80acf70 2024-09-13T17:20:41.267 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:20:41.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:20:41.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.633+0000 7f29bfe00640 1 -- 172.21.15.28:0/3391956959 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v150) v1 ==== 118+0+0 (secure 0 0 0) 0x7f29b80170b0 con 0x7f29c80acf70 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 msgr2=0x7f29a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 0x7f29a806d0c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f29b4000f60 tx=0x7f29b40079b0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 msgr2=0x7f29c814be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f29b800c480 tx=0x7f29b800c940 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 shutdown_connections 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f29a806ac00 0x7f29a806d0c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29c81600d0 0x7f29c8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f29c80c98f0 0x7f29c8159ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 --2- 172.21.15.28:0/3391956959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29c80acf70 0x7f29c814be00 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 >> 172.21.15.28:0/3391956959 conn(0x7f29c80ce900 msgr2=0x7f29c80adbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.635+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 shutdown_connections 2024-09-13T17:20:41.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.636+0000 7f29cea00640 1 -- 172.21.15.28:0/3391956959 wait complete. 2024-09-13T17:20:41.654 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:20:41.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:20:41.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.811+0000 7f8116600640 1 Processor -- start 2024-09-13T17:20:41.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.811+0000 7f8116600640 1 -- start start 2024-09-13T17:20:41.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.812+0000 7f8116600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f81100cc530 0x7f8110072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8116600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100c78e0 con 0x7f81100b06f0 2024-09-13T17:20:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8116600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81100cbe40 con 0x7f81100cc910 2024-09-13T17:20:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8116600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8110070c90 con 0x7f81100cc530 2024-09-13T17:20:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8115c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81100b06f0 0x7f81100b0b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38506/0 (socket says 172.21.15.28:38506) 2024-09-13T17:20:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8115c00640 1 -- 172.21.15.28:0/145567131 learned_addr learned my addr 172.21.15.28:0/145567131 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f8115200640 1 --2- 172.21.15.28:0/145567131 >> v2:172.21.15.189:3300/0 conn(0x7f81100cc530 0x7f8110072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3254251747 0 0) 0x7f81100c78e0 con 0x7f81100b06f0 2024-09-13T17:20:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.813+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80f4003520 con 0x7f81100b06f0 2024-09-13T17:20:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829117133 0 0) 0x7f81100cbe40 con 0x7f81100cc910 2024-09-13T17:20:41.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81100c78e0 con 0x7f81100cc910 2024-09-13T17:20:41.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f8115200640 1 -- 172.21.15.28:0/145567131 >> v1:172.21.15.189:6789/0 conn(0x7f81100cc910 legacy=0x7f811015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f8115200640 1 -- 172.21.15.28:0/145567131 >> v1:172.21.15.28:6789/0 conn(0x7f81100b06f0 legacy=0x7f81100b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2016218314 0 0) 0x7f80f4003520 con 0x7f81100b06f0 2024-09-13T17:20:41.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.814+0000 7f8115200640 1 -- 172.21.15.28:0/145567131 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8110071000 con 0x7f81100cc530 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.815+0000 7f8115200640 1 --2- 172.21.15.28:0/145567131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc530 0x7f8110072060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f81000097a0 tx=0x7f810002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f56de48dadbc88ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.815+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81080019f0 con 0x7f81100cc530 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810002ace0 con 0x7f81100cc530 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f810fe00640 1 -- 172.21.15.28:0/145567131 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8100032660 con 0x7f81100cc530 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 -- 172.21.15.28:0/145567131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc530 msgr2=0x7f8110072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 --2- 172.21.15.28:0/145567131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc530 0x7f8110072060 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f81000097a0 tx=0x7f810002a250 comp rx=0 tx=0).stop 2024-09-13T17:20:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 -- 172.21.15.28:0/145567131 shutdown_connections 2024-09-13T17:20:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 --2- 172.21.15.28:0/145567131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc530 0x7f8110072060 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 -- 172.21.15.28:0/145567131 >> 172.21.15.28:0/145567131 conn(0x7f8110069e80 msgr2=0x7f811006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 -- 172.21.15.28:0/145567131 shutdown_connections 2024-09-13T17:20:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.816+0000 7f8116600640 1 -- 172.21.15.28:0/145567131 wait complete. 2024-09-13T17:20:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.818+0000 7f8116600640 1 Processor -- start 2024-09-13T17:20:41.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.818+0000 7f8116600640 1 -- start start 2024-09-13T17:20:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81100b06f0 0x7f81101476a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:41.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc910 0x7f81101f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81100c81d0 con 0x7f81100cc530 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8110147be0 con 0x7f81100cc910 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8116600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8110158fb0 con 0x7f81100b06f0 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8107e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8107e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49984/0 (socket says 172.21.15.28:49984) 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8107e00640 1 -- 172.21.15.28:0/3502598134 learned_addr learned my addr 172.21.15.28:0/3502598134 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8115c00640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc910 0x7f81101f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.819+0000 7f8115200640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81100b06f0 0x7f81101476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 -- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81100b06f0 msgr2=0x7f81101476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81100b06f0 0x7f81101476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 -- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc910 msgr2=0x7f81101f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc910 0x7f81101f56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 -- 172.21.15.28:0/3502598134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100009400 con 0x7f81100cc530 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8107e00640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c490 tx=0x7f80f800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f80052d0 con 0x7f81100cc530 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f8005450 con 0x7f81100cc530 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8110157300 con 0x7f81100cc530 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f8003830 con 0x7f81100cc530 2024-09-13T17:20:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.820+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8110159f30 con 0x7f81100cc530 2024-09-13T17:20:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.821+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81100c8310 con 0x7f81100cc530 2024-09-13T17:20:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.822+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f80f8003990 con 0x7f81100cc530 2024-09-13T17:20:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.823+0000 7f810d600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 0x7f80ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.823+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80f8088870 con 0x7f81100cc530 2024-09-13T17:20:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.824+0000 7f8115200640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 0x7f80ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.825+0000 7f8115200640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 0x7f80ec06d0c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8100034000 tx=0x7f810002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:41.828+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f81100c8310 con 0x7f81100cc530 2024-09-13T17:20:42.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.042+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f81100c8310 con 0x7f81100cc530 2024-09-13T17:20:42.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.635+0000 7f810d600640 1 -- 172.21.15.28:0/3502598134 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v151) v1 ==== 122+0+0 (secure 0 0 0) 0x7f81100c8310 con 0x7f81100cc530 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 msgr2=0x7f80ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 0x7f80ec06d0c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f8100034000 tx=0x7f810002e000 comp rx=0 tx=0).stop 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 msgr2=0x7f81101f51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7f80f800c490 tx=0x7f80f800c950 comp rx=0 tx=0).stop 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 shutdown_connections 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80ec06ac00 0x7f80ec06d0c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81100cc910 0x7f81101f56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f81100cc530 0x7f81101f51b0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 --2- 172.21.15.28:0/3502598134 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81100b06f0 0x7f81101476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 >> 172.21.15.28:0/3502598134 conn(0x7f8110069e80 msgr2=0x7f8110155b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 shutdown_connections 2024-09-13T17:20:42.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.638+0000 7f8116600640 1 -- 172.21.15.28:0/3502598134 wait complete. 2024-09-13T17:20:42.657 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:20:42.658 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.814+0000 7f80e2200640 1 Processor -- start 2024-09-13T17:20:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.814+0000 7f80e2200640 1 -- start start 2024-09-13T17:20:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80e2200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80dc161020 0x7f80dc161460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80e2200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f80dc0b2a80 0x7f80dc0b2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80e2200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80dc06c300 con 0x7f80dc160c40 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80e2200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc06ba70 con 0x7f80dc161020 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80e2200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80dc06bd40 con 0x7f80dc0b2a80 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.815+0000 7f80dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80dc161020 0x7f80dc161460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f80dc161020 0x7f80dc161460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49994/0 (socket says 172.21.15.28:49994) 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 -- 172.21.15.28:0/1876680393 learned_addr learned my addr 172.21.15.28:0/1876680393 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:42.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80e1800640 1 --2- 172.21.15.28:0/1876680393 >> v2:172.21.15.189:3300/0 conn(0x7f80dc0b2a80 0x7f80dc0b2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 -- 172.21.15.28:0/1876680393 >> v2:172.21.15.189:3300/0 conn(0x7f80dc0b2a80 msgr2=0x7f80dc0b2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 --2- 172.21.15.28:0/1876680393 >> v2:172.21.15.189:3300/0 conn(0x7f80dc0b2a80 0x7f80dc0b2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 -- 172.21.15.28:0/1876680393 >> v1:172.21.15.189:6789/0 conn(0x7f80dc160c40 legacy=0x7f80dc06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.816+0000 7f80dbe00640 1 -- 172.21.15.28:0/1876680393 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80dc06c020 con 0x7f80dc161020 2024-09-13T17:20:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.817+0000 7f80dbe00640 1 --2- 172.21.15.28:0/1876680393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc161020 0x7f80dc161460 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f80dc0af990 tx=0x7f80c802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1463b94ceca64dd3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.817+0000 7f80db400640 1 -- 172.21.15.28:0/1876680393 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c8032440 con 0x7f80dc161020 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.817+0000 7f80db400640 1 -- 172.21.15.28:0/1876680393 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80c80325a0 con 0x7f80dc161020 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.817+0000 7f80db400640 1 -- 172.21.15.28:0/1876680393 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80c8032930 con 0x7f80dc161020 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 -- 172.21.15.28:0/1876680393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc161020 msgr2=0x7f80dc161460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 --2- 172.21.15.28:0/1876680393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc161020 0x7f80dc161460 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7f80dc0af990 tx=0x7f80c802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 -- 172.21.15.28:0/1876680393 shutdown_connections 2024-09-13T17:20:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 --2- 172.21.15.28:0/1876680393 >> v2:172.21.15.189:3300/0 conn(0x7f80dc0b2a80 0x7f80dc0b2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 --2- 172.21.15.28:0/1876680393 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc161020 0x7f80dc161460 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 -- 172.21.15.28:0/1876680393 >> 172.21.15.28:0/1876680393 conn(0x7f80dc072e40 msgr2=0x7f80dc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 -- 172.21.15.28:0/1876680393 shutdown_connections 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.818+0000 7f80e2200640 1 -- 172.21.15.28:0/1876680393 wait complete. 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.820+0000 7f80e2200640 1 Processor -- start 2024-09-13T17:20:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.820+0000 7f80e2200640 1 -- start start 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80dc160c40 0x7f80dc155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 0x7f80dc0aaf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc0a84e0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc0a87c0 con 0x7f80dc160c40 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80dc0a8aa0 con 0x7f80dc161020 2024-09-13T17:20:42.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49996/0 (socket says 172.21.15.28:49996) 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 -- 172.21.15.28:0/1547539703 learned_addr learned my addr 172.21.15.28:0/1547539703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80dbe00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80dc160c40 0x7f80dc155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e1800640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 0x7f80dc0aaf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 -- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 msgr2=0x7f80dc0aaf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 0x7f80dc0aaf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 -- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80dc160c40 msgr2=0x7f80dc155dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80dc160c40 0x7f80dc155dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e0e00640 1 -- 172.21.15.28:0/1547539703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c80094d0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.821+0000 7f80e1800640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 0x7f80dc0aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80e0e00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c450 tx=0x7f80d000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d00052d0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d0005430 con 0x7f80dc0b2a80 2024-09-13T17:20:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80dc0a8d80 con 0x7f80dc0b2a80 2024-09-13T17:20:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d0012870 con 0x7f80dc0b2a80 2024-09-13T17:20:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.822+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80dc15b160 con 0x7f80dc0b2a80 2024-09-13T17:20:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.823+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80dc0a92a0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.824+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f80d00129d0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.825+0000 7f80d8c00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 0x7f80bc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.825+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f80d0088ea0 con 0x7f80dc0b2a80 2024-09-13T17:20:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.826+0000 7f80dbe00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 0x7f80bc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.827+0000 7f80dbe00640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 0x7f80bc06d0c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f80c8034000 tx=0x7f80c802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:42.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:42.830+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f80dc0a92a0 con 0x7f80dc0b2a80 2024-09-13T17:20:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.044+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80dc0a92a0 con 0x7f80dc0b2a80 2024-09-13T17:20:43.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.045+0000 7f80d8c00640 1 -- 172.21.15.28:0/1547539703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4754 (secure 0 0 0) 0x7f80dc0a92a0 con 0x7f80dc0b2a80 2024-09-13T17:20:43.047 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:43.047 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":10551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:20:42.633390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:20:43.047 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 151 2024-09-13T17:20:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 msgr2=0x7f80bc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 0x7f80bc06d0c0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f80c8034000 tx=0x7f80c802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 msgr2=0x7f80dc15a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f80d000c450 tx=0x7f80d000c910 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 shutdown_connections 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80bc06ac00 0x7f80bc06d0c0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f80dc161020 0x7f80dc0aaf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80dc160c40 0x7f80dc155dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 --2- 172.21.15.28:0/1547539703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f80dc0b2a80 0x7f80dc15a8b0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 >> 172.21.15.28:0/1547539703 conn(0x7f80dc072e40 msgr2=0x7f80dc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 shutdown_connections 2024-09-13T17:20:43.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.047+0000 7f80e2200640 1 -- 172.21.15.28:0/1547539703 wait complete. 2024-09-13T17:20:43.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:20:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.218+0000 7fbfb9400640 1 Processor -- start 2024-09-13T17:20:43.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.218+0000 7fbfb9400640 1 -- start start 2024-09-13T17:20:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.219+0000 7fbfb9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 0x7fbfb406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.219+0000 7fbfb9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb415e100 con 0x7fbfb415a540 2024-09-13T17:20:43.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.219+0000 7fbfb9400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfb415e3e0 con 0x7fbfb40cabc0 2024-09-13T17:20:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.219+0000 7fbfb9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40adb00 con 0x7fbfb40713d0 2024-09-13T17:20:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbfb415a540 0x7fbfb415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38518/0 (socket says 172.21.15.28:38518) 2024-09-13T17:20:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb2800640 1 -- 172.21.15.28:0/2939704675 learned_addr learned my addr 172.21.15.28:0/2939704675 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:43.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb3200640 1 --2- 172.21.15.28:0/2939704675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 0x7fbfb406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141014249 0 0) 0x7fbfb415e100 con 0x7fbfb415a540 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf98003520 con 0x7fbfb415a540 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2145281250 0 0) 0x7fbf98003520 con 0x7fbfb415a540 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb3200640 1 -- 172.21.15.28:0/2939704675 >> v1:172.21.15.189:6789/0 conn(0x7fbfb40cabc0 legacy=0x7fbfb4070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb3200640 1 -- 172.21.15.28:0/2939704675 >> v1:172.21.15.28:6789/0 conn(0x7fbfb415a540 legacy=0x7fbfb415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.220+0000 7fbfb3200640 1 -- 172.21.15.28:0/2939704675 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfb415ef80 con 0x7fbfb40713d0 2024-09-13T17:20:43.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb3200640 1 --2- 172.21.15.28:0/2939704675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 0x7fbfb406c430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009dc0 tx=0x7fbfac02a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd88b1fe3c9e928 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfa4001670 con 0x7fbfb40713d0 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac02d050 con 0x7fbfb40713d0 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb1400640 1 -- 172.21.15.28:0/2939704675 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac007740 con 0x7fbfb40713d0 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb9400640 1 -- 172.21.15.28:0/2939704675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 msgr2=0x7fbfb406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb9400640 1 --2- 172.21.15.28:0/2939704675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 0x7fbfb406c430 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009dc0 tx=0x7fbfac02a500 comp rx=0 tx=0).stop 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb9400640 1 -- 172.21.15.28:0/2939704675 shutdown_connections 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb9400640 1 --2- 172.21.15.28:0/2939704675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40713d0 0x7fbfb406c430 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.222+0000 7fbfb9400640 1 -- 172.21.15.28:0/2939704675 >> 172.21.15.28:0/2939704675 conn(0x7fbfb4069e80 msgr2=0x7fbfb406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.223+0000 7fbfb9400640 1 -- 172.21.15.28:0/2939704675 shutdown_connections 2024-09-13T17:20:43.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.223+0000 7fbfb9400640 1 -- 172.21.15.28:0/2939704675 wait complete. 2024-09-13T17:20:43.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.224+0000 7fbfb9400640 1 Processor -- start 2024-09-13T17:20:43.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.224+0000 7fbfb9400640 1 -- start start 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 0x7fbfb414f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40cabc0 0x7fbfb4150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40af940 con 0x7fbfb415a540 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb40aefe0 con 0x7fbfb40713d0 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.225+0000 7fbfb9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb414fd50 con 0x7fbfb40cabc0 2024-09-13T17:20:43.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50002/0 (socket says 172.21.15.28:50002) 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 -- 172.21.15.28:0/1717905099 learned_addr learned my addr 172.21.15.28:0/1717905099 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb2800640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 0x7fbfb414f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb1e00640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40cabc0 0x7fbfb4150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 -- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40cabc0 msgr2=0x7fbfb4150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40cabc0 0x7fbfb4150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 -- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 msgr2=0x7fbfb414f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 0x7fbfb414f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 -- 172.21.15.28:0/1717905099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfac009a70 con 0x7fbfb415a540 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb2800640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 0x7fbfb414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:43.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfb3200640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fbfac02aaf0 tx=0x7fbfac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.226+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac003b90 con 0x7fbfb415a540 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.227+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfac003cf0 con 0x7fbfb415a540 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.227+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfac03c960 con 0x7fbfb415a540 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.227+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb40af280 con 0x7fbfb415a540 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.227+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb4147660 con 0x7fbfb415a540 2024-09-13T17:20:43.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.228+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb406c430 con 0x7fbfb415a540 2024-09-13T17:20:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.229+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbfac03cac0 con 0x7fbfb415a540 2024-09-13T17:20:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.229+0000 7fbfaaa00640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 0x7fbf9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.230+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbfac0aa9d0 con 0x7fbfb415a540 2024-09-13T17:20:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.230+0000 7fbfb2800640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 0x7fbf9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.231+0000 7fbfb2800640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 0x7fbf9406d0f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4001eb0 tx=0x7fbfa4001ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.233+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbfac075ba0 con 0x7fbfb415a540 2024-09-13T17:20:43.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.423+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfb4155ca0 con 0x7fbfb415a540 2024-09-13T17:20:43.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.424+0000 7fbfaaa00640 1 -- 172.21.15.28:0/1717905099 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+16579 (secure 0 0 0) 0x7fbfac035090 con 0x7fbfb415a540 2024-09-13T17:20:43.426 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:43.426 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":179,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:40.614423+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 msgr2=0x7fbf9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 0x7fbf9406d0f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbfa4001eb0 tx=0x7fbfa4001ca0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 msgr2=0x7fbfb4147120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fbfac02aaf0 tx=0x7fbfac004a40 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 shutdown_connections 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf9406ac30 0x7fbf9406d0f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbfb415a540 0x7fbfb4147120 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbfb40cabc0 0x7fbfb4150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 --2- 172.21.15.28:0/1717905099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbfb40713d0 0x7fbfb414f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.426+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 >> 172.21.15.28:0/1717905099 conn(0x7fbfb4069e80 msgr2=0x7fbfb40ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.427+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 shutdown_connections 2024-09-13T17:20:43.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.427+0000 7fbfb9400640 1 -- 172.21.15.28:0/1717905099 wait complete. 2024-09-13T17:20:43.445 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:20:43.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:20:43.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.599+0000 7f3d4d800640 1 Processor -- start 2024-09-13T17:20:43.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.599+0000 7f3d4d800640 1 -- start start 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d4d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 0x7f3d4806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d4d800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3d48070cc0 0x7f3d48071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d4d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4815e100 con 0x7f3d480cabc0 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d4d800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d480716b0 con 0x7f3d48070cc0 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d4d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d480adb00 con 0x7f3d4815a540 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d46000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3d480cabc0 0x7f3d4815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38534/0 (socket says 172.21.15.28:38534) 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.600+0000 7f3d46000640 1 -- 172.21.15.28:0/637330173 learned_addr learned my addr 172.21.15.28:0/637330173 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d46a00640 1 --2- 172.21.15.28:0/637330173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 0x7f3d4806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d47400640 1 --2- 172.21.15.28:0/637330173 >> v2:172.21.15.189:3300/0 conn(0x7f3d48070cc0 0x7f3d48071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d45600640 1 -- 172.21.15.28:0/637330173 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901197477 0 0) 0x7f3d4815e100 con 0x7f3d480cabc0 2024-09-13T17:20:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d45600640 1 -- 172.21.15.28:0/637330173 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d2c003520 con 0x7f3d480cabc0 2024-09-13T17:20:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d46a00640 1 -- 172.21.15.28:0/637330173 >> v2:172.21.15.189:3300/0 conn(0x7f3d48070cc0 msgr2=0x7f3d48071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d46a00640 1 --2- 172.21.15.28:0/637330173 >> v2:172.21.15.189:3300/0 conn(0x7f3d48070cc0 0x7f3d48071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d46a00640 1 -- 172.21.15.28:0/637330173 >> v1:172.21.15.28:6789/0 conn(0x7f3d480cabc0 legacy=0x7f3d4815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.601+0000 7f3d46a00640 1 -- 172.21.15.28:0/637330173 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4815e3e0 con 0x7f3d4815a540 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.602+0000 7f3d46a00640 1 --2- 172.21.15.28:0/637330173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 0x7f3d4806c430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e6582b47b51fd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.602+0000 7f3d45600640 1 -- 172.21.15.28:0/637330173 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3802ae50 con 0x7f3d4815a540 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d45600640 1 -- 172.21.15.28:0/637330173 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d38032440 con 0x7f3d4815a540 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d45600640 1 -- 172.21.15.28:0/637330173 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d380327b0 con 0x7f3d4815a540 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 -- 172.21.15.28:0/637330173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 msgr2=0x7f3d4806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 --2- 172.21.15.28:0/637330173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 0x7f3d4806c430 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802a100 comp rx=0 tx=0).stop 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 -- 172.21.15.28:0/637330173 shutdown_connections 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 --2- 172.21.15.28:0/637330173 >> v2:172.21.15.189:3300/0 conn(0x7f3d48070cc0 0x7f3d48071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 --2- 172.21.15.28:0/637330173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d4815a540 0x7f3d4806c430 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 -- 172.21.15.28:0/637330173 >> 172.21.15.28:0/637330173 conn(0x7f3d48069e80 msgr2=0x7f3d4806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:43.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 -- 172.21.15.28:0/637330173 shutdown_connections 2024-09-13T17:20:43.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.603+0000 7f3d4d800640 1 -- 172.21.15.28:0/637330173 wait complete. 2024-09-13T17:20:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.605+0000 7f3d4d800640 1 Processor -- start 2024-09-13T17:20:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.605+0000 7f3d4d800640 1 -- start start 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d48070cc0 0x7f3d4814eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4815a540 0x7f3d4814fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4814bf00 con 0x7f3d480cabc0 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d48152020 con 0x7f3d4815a540 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d4d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d481efcc0 con 0x7f3d48070cc0 2024-09-13T17:20:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d46000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50010/0 (socket says 172.21.15.28:50010) 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d46000640 1 -- 172.21.15.28:0/816476371 learned_addr learned my addr 172.21.15.28:0/816476371 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d47400640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4815a540 0x7f3d4814fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.606+0000 7f3d46a00640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d48070cc0 0x7f3d4814eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 -- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d48070cc0 msgr2=0x7f3d4814eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d48070cc0 0x7f3d4814eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 -- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4815a540 msgr2=0x7f3d4814fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4815a540 0x7f3d4814fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 -- 172.21.15.28:0/816476371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d38009400 con 0x7f3d480cabc0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d46000640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c400 tx=0x7f3d3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34005ce0 con 0x7f3d480cabc0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34005e60 con 0x7f3d480cabc0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d340072f0 con 0x7f3d480cabc0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4814f170 con 0x7f3d480cabc0 2024-09-13T17:20:43.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.607+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d480ccf60 con 0x7f3d480cabc0 2024-09-13T17:20:43.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.608+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d480cbeb0 con 0x7f3d480cabc0 2024-09-13T17:20:43.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.609+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3d34007450 con 0x7f3d480cabc0 2024-09-13T17:20:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.611+0000 7f3d3ea00640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 0x7f3d2806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.611+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d34088990 con 0x7f3d480cabc0 2024-09-13T17:20:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.611+0000 7f3d46a00640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 0x7f3d2806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.612+0000 7f3d46a00640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 0x7f3d2806d0c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.615+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3d34053c10 con 0x7f3d480cabc0 2024-09-13T17:20:43.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.792+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3d4806d160 con 0x7f3d2806ac00 2024-09-13T17:20:43.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.801+0000 7f3d3ea00640 1 -- 172.21.15.28:0/816476371 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289146 (secure 0 0 0) 0x7f3d38041070 con 0x7f3d2806ac00 2024-09-13T17:20:43.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:43.808 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:20:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 msgr2=0x7f3d2806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 0x7f3d2806d0c0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f3d380097a0 tx=0x7f3d3802e000 comp rx=0 tx=0).stop 2024-09-13T17:20:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 msgr2=0x7f3d4814b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:43.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c400 tx=0x7f3d3400c8c0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 shutdown_connections 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d2806ac00 0x7f3d2806d0c0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d4815a540 0x7f3d4814fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d480cabc0 0x7f3d4814b9c0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 --2- 172.21.15.28:0/816476371 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d48070cc0 0x7f3d4814eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 >> 172.21.15.28:0/816476371 conn(0x7f3d48069e80 msgr2=0x7f3d48155360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.804+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 shutdown_connections 2024-09-13T17:20:43.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:43.805+0000 7f3d4d800640 1 -- 172.21.15.28:0/816476371 wait complete. 2024-09-13T17:20:43.823 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2024-09-13T17:20:43.075077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":520116,"kb_used_data":18784,"kb_used_omap":12,"kb_used_meta":501171,"kb_avail":754454604,"statfs":{"total":773094113280,"available":772561514496,"internally_reserved":0,"allocated":19234816,"data_stored":9125278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":513199673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.896552"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144039+0000","last_change":"2024-09-13T17:20:39.639513+0000","last_active":"2024-09-13T17:20:41.144039+0000","last_peered":"2024-09-13T17:20:41.144039+0000","last_clean":"2024-09-13T17:20:41.144039+0000","last_became_active":"2024-09-13T17:20:39.639243+0000","last_became_peered":"2024-09-13T17:20:39.639243+0000","last_unstale":"2024-09-13T17:20:41.144039+0000","last_undegraded":"2024-09-13T17:20:41.144039+0000","last_fullsized":"2024-09-13T17:20:41.144039+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:02.805053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620671+0000","last_change":"2024-09-13T17:20:39.644427+0000","last_active":"2024-09-13T17:20:40.620671+0000","last_peered":"2024-09-13T17:20:40.620671+0000","last_clean":"2024-09-13T17:20:40.620671+0000","last_became_active":"2024-09-13T17:20:39.644157+0000","last_became_peered":"2024-09-13T17:20:39.644157+0000","last_unstale":"2024-09-13T17:20:40.620671+0000","last_undegraded":"2024-09-13T17:20:40.620671+0000","last_fullsized":"2024-09-13T17:20:40.620671+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984986+0000","last_change":"2024-09-13T17:20:39.627269+0000","last_active":"2024-09-13T17:20:40.984986+0000","last_peered":"2024-09-13T17:20:40.984986+0000","last_clean":"2024-09-13T17:20:40.984986+0000","last_became_active":"2024-09-13T17:20:39.627012+0000","last_became_peered":"2024-09-13T17:20:39.627012+0000","last_unstale":"2024-09-13T17:20:40.984986+0000","last_undegraded":"2024-09-13T17:20:40.984986+0000","last_fullsized":"2024-09-13T17:20:40.984986+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620600+0000","last_change":"2024-09-13T17:20:39.637469+0000","last_active":"2024-09-13T17:20:40.620600+0000","last_peered":"2024-09-13T17:20:40.620600+0000","last_clean":"2024-09-13T17:20:40.620600+0000","last_became_active":"2024-09-13T17:20:39.637144+0000","last_became_peered":"2024-09-13T17:20:39.637144+0000","last_unstale":"2024-09-13T17:20:40.620600+0000","last_undegraded":"2024-09-13T17:20:40.620600+0000","last_fullsized":"2024-09-13T17:20:40.620600+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:00.369481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144258+0000","last_change":"2024-09-13T17:20:39.637845+0000","last_active":"2024-09-13T17:20:41.144258+0000","last_peered":"2024-09-13T17:20:41.144258+0000","last_clean":"2024-09-13T17:20:41.144258+0000","last_became_active":"2024-09-13T17:20:39.637143+0000","last_became_peered":"2024-09-13T17:20:39.637143+0000","last_unstale":"2024-09-13T17:20:41.144258+0000","last_undegraded":"2024-09-13T17:20:41.144258+0000","last_fullsized":"2024-09-13T17:20:41.144258+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:05.927764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984963+0000","last_change":"2024-09-13T17:20:39.627407+0000","last_active":"2024-09-13T17:20:40.984963+0000","last_peered":"2024-09-13T17:20:40.984963+0000","last_clean":"2024-09-13T17:20:40.984963+0000","last_became_active":"2024-09-13T17:20:39.627160+0000","last_became_peered":"2024-09-13T17:20:39.627160+0000","last_unstale":"2024-09-13T17:20:40.984963+0000","last_undegraded":"2024-09-13T17:20:40.984963+0000","last_fullsized":"2024-09-13T17:20:40.984963+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:00.369481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620271+0000","last_change":"2024-09-13T17:20:39.649468+0000","last_active":"2024-09-13T17:20:40.620271+0000","last_peered":"2024-09-13T17:20:40.620271+0000","last_clean":"2024-09-13T17:20:40.620271+0000","last_became_active":"2024-09-13T17:20:39.649168+0000","last_became_peered":"2024-09-13T17:20:39.649168+0000","last_unstale":"2024-09-13T17:20:40.620271+0000","last_undegraded":"2024-09-13T17:20:40.620271+0000","last_fullsized":"2024-09-13T17:20:40.620271+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:28:37.235135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143444+0000","last_change":"2024-09-13T17:20:39.629646+0000","last_active":"2024-09-13T17:20:41.143444+0000","last_peered":"2024-09-13T17:20:41.143444+0000","last_clean":"2024-09-13T17:20:41.143444+0000","last_became_active":"2024-09-13T17:20:39.628797+0000","last_became_peered":"2024-09-13T17:20:39.628797+0000","last_unstale":"2024-09-13T17:20:41.143444+0000","last_undegraded":"2024-09-13T17:20:41.143444+0000","last_fullsized":"2024-09-13T17:20:41.143444+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:26.903757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678457+0000","last_change":"2024-09-13T17:20:39.638900+0000","last_active":"2024-09-13T17:20:40.678457+0000","last_peered":"2024-09-13T17:20:40.678457+0000","last_clean":"2024-09-13T17:20:40.678457+0000","last_became_active":"2024-09-13T17:20:39.638635+0000","last_became_peered":"2024-09-13T17:20:39.638635+0000","last_unstale":"2024-09-13T17:20:40.678457+0000","last_undegraded":"2024-09-13T17:20:40.678457+0000","last_fullsized":"2024-09-13T17:20:40.678457+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:48.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618431+0000","last_change":"2024-09-13T17:20:39.643856+0000","last_active":"2024-09-13T17:20:40.618431+0000","last_peered":"2024-09-13T17:20:40.618431+0000","last_clean":"2024-09-13T17:20:40.618431+0000","last_became_active":"2024-09-13T17:20:39.641868+0000","last_became_peered":"2024-09-13T17:20:39.641868+0000","last_unstale":"2024-09-13T17:20:40.618431+0000","last_undegraded":"2024-09-13T17:20:40.618431+0000","last_fullsized":"2024-09-13T17:20:40.618431+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:25.682499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620646+0000","last_change":"2024-09-13T17:20:39.629329+0000","last_active":"2024-09-13T17:20:40.620646+0000","last_peered":"2024-09-13T17:20:40.620646+0000","last_clean":"2024-09-13T17:20:40.620646+0000","last_became_active":"2024-09-13T17:20:39.628159+0000","last_became_peered":"2024-09-13T17:20:39.628159+0000","last_unstale":"2024-09-13T17:20:40.620646+0000","last_undegraded":"2024-09-13T17:20:40.620646+0000","last_fullsized":"2024-09-13T17:20:40.620646+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:35.251677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620568+0000","last_change":"2024-09-13T17:20:39.638963+0000","last_active":"2024-09-13T17:20:40.620568+0000","last_peered":"2024-09-13T17:20:40.620568+0000","last_clean":"2024-09-13T17:20:40.620568+0000","last_became_active":"2024-09-13T17:20:39.637723+0000","last_became_peered":"2024-09-13T17:20:39.637723+0000","last_unstale":"2024-09-13T17:20:40.620568+0000","last_undegraded":"2024-09-13T17:20:40.620568+0000","last_fullsized":"2024-09-13T17:20:40.620568+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:37.192146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619696+0000","last_change":"2024-09-13T17:20:39.638804+0000","last_active":"2024-09-13T17:20:40.619696+0000","last_peered":"2024-09-13T17:20:40.619696+0000","last_clean":"2024-09-13T17:20:40.619696+0000","last_became_active":"2024-09-13T17:20:39.636208+0000","last_became_peered":"2024-09-13T17:20:39.636208+0000","last_unstale":"2024-09-13T17:20:40.619696+0000","last_undegraded":"2024-09-13T17:20:40.619696+0000","last_fullsized":"2024-09-13T17:20:40.619696+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678161+0000","last_change":"2024-09-13T17:20:39.641632+0000","last_active":"2024-09-13T17:20:40.678161+0000","last_peered":"2024-09-13T17:20:40.678161+0000","last_clean":"2024-09-13T17:20:40.678161+0000","last_became_active":"2024-09-13T17:20:39.641389+0000","last_became_peered":"2024-09-13T17:20:39.641389+0000","last_unstale":"2024-09-13T17:20:40.678161+0000","last_undegraded":"2024-09-13T17:20:40.678161+0000","last_fullsized":"2024-09-13T17:20:40.678161+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620446+0000","last_change":"2024-09-13T17:20:39.641174+0000","last_active":"2024-09-13T17:20:40.620446+0000","last_peered":"2024-09-13T17:20:40.620446+0000","last_clean":"2024-09-13T17:20:40.620446+0000","last_became_active":"2024-09-13T17:20:39.640504+0000","last_became_peered":"2024-09-13T17:20:39.640504+0000","last_unstale":"2024-09-13T17:20:40.620446+0000","last_undegraded":"2024-09-13T17:20:40.620446+0000","last_fullsized":"2024-09-13T17:20:40.620446+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143680+0000","last_change":"2024-09-13T17:20:39.632785+0000","last_active":"2024-09-13T17:20:41.143680+0000","last_peered":"2024-09-13T17:20:41.143680+0000","last_clean":"2024-09-13T17:20:41.143680+0000","last_became_active":"2024-09-13T17:20:39.632510+0000","last_became_peered":"2024-09-13T17:20:39.632510+0000","last_unstale":"2024-09-13T17:20:41.143680+0000","last_undegraded":"2024-09-13T17:20:41.143680+0000","last_fullsized":"2024-09-13T17:20:41.143680+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:57.973890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144375+0000","last_change":"2024-09-13T17:20:39.645897+0000","last_active":"2024-09-13T17:20:41.144375+0000","last_peered":"2024-09-13T17:20:41.144375+0000","last_clean":"2024-09-13T17:20:41.144375+0000","last_became_active":"2024-09-13T17:20:39.643353+0000","last_became_peered":"2024-09-13T17:20:39.643353+0000","last_unstale":"2024-09-13T17:20:41.144375+0000","last_undegraded":"2024-09-13T17:20:41.144375+0000","last_fullsized":"2024-09-13T17:20:41.144375+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:38:04.265039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143956+0000","last_change":"2024-09-13T17:20:39.634096+0000","last_active":"2024-09-13T17:20:41.143956+0000","last_peered":"2024-09-13T17:20:41.143956+0000","last_clean":"2024-09-13T17:20:41.143956+0000","last_became_active":"2024-09-13T17:20:39.633806+0000","last_became_peered":"2024-09-13T17:20:39.633806+0000","last_unstale":"2024-09-13T17:20:41.143956+0000","last_undegraded":"2024-09-13T17:20:41.143956+0000","last_fullsized":"2024-09-13T17:20:41.143956+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:24.516630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618278+0000","last_change":"2024-09-13T17:20:39.634066+0000","last_active":"2024-09-13T17:20:40.618278+0000","last_peered":"2024-09-13T17:20:40.618278+0000","last_clean":"2024-09-13T17:20:40.618278+0000","last_became_active":"2024-09-13T17:20:39.633182+0000","last_became_peered":"2024-09-13T17:20:39.633182+0000","last_unstale":"2024-09-13T17:20:40.618278+0000","last_undegraded":"2024-09-13T17:20:40.618278+0000","last_fullsized":"2024-09-13T17:20:40.618278+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:26.903757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144120+0000","last_change":"2024-09-13T17:20:39.646961+0000","last_active":"2024-09-13T17:20:41.144120+0000","last_peered":"2024-09-13T17:20:41.144120+0000","last_clean":"2024-09-13T17:20:41.144120+0000","last_became_active":"2024-09-13T17:20:39.645949+0000","last_became_peered":"2024-09-13T17:20:39.645949+0000","last_unstale":"2024-09-13T17:20:41.144120+0000","last_undegraded":"2024-09-13T17:20:41.144120+0000","last_fullsized":"2024-09-13T17:20:41.144120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:37.428112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144178+0000","last_change":"2024-09-13T17:20:39.637383+0000","last_active":"2024-09-13T17:20:41.144178+0000","last_peered":"2024-09-13T17:20:41.144178+0000","last_clean":"2024-09-13T17:20:41.144178+0000","last_became_active":"2024-09-13T17:20:39.636741+0000","last_became_peered":"2024-09-13T17:20:39.636741+0000","last_unstale":"2024-09-13T17:20:41.144178+0000","last_undegraded":"2024-09-13T17:20:41.144178+0000","last_fullsized":"2024-09-13T17:20:41.144178+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:16.534672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618298+0000","last_change":"2024-09-13T17:20:39.643510+0000","last_active":"2024-09-13T17:20:40.618298+0000","last_peered":"2024-09-13T17:20:40.618298+0000","last_clean":"2024-09-13T17:20:40.618298+0000","last_became_active":"2024-09-13T17:20:39.643224+0000","last_became_peered":"2024-09-13T17:20:39.643224+0000","last_unstale":"2024-09-13T17:20:40.618298+0000","last_undegraded":"2024-09-13T17:20:40.618298+0000","last_fullsized":"2024-09-13T17:20:40.618298+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:42.523979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984785+0000","last_change":"2024-09-13T17:20:39.649180+0000","last_active":"2024-09-13T17:20:40.984785+0000","last_peered":"2024-09-13T17:20:40.984785+0000","last_clean":"2024-09-13T17:20:40.984785+0000","last_became_active":"2024-09-13T17:20:39.648919+0000","last_became_peered":"2024-09-13T17:20:39.648919+0000","last_unstale":"2024-09-13T17:20:40.984785+0000","last_undegraded":"2024-09-13T17:20:40.984785+0000","last_fullsized":"2024-09-13T17:20:40.984785+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143783+0000","last_change":"2024-09-13T17:20:39.632244+0000","last_active":"2024-09-13T17:20:41.143783+0000","last_peered":"2024-09-13T17:20:41.143783+0000","last_clean":"2024-09-13T17:20:41.143783+0000","last_became_active":"2024-09-13T17:20:39.631383+0000","last_became_peered":"2024-09-13T17:20:39.631383+0000","last_unstale":"2024-09-13T17:20:41.143783+0000","last_undegraded":"2024-09-13T17:20:41.143783+0000","last_fullsized":"2024-09-13T17:20:41.143783+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:56.936624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620674+0000","last_change":"2024-09-13T17:20:39.641267+0000","last_active":"2024-09-13T17:20:40.620674+0000","last_peered":"2024-09-13T17:20:40.620674+0000","last_clean":"2024-09-13T17:20:40.620674+0000","last_became_active":"2024-09-13T17:20:39.639766+0000","last_became_peered":"2024-09-13T17:20:39.639766+0000","last_unstale":"2024-09-13T17:20:40.620674+0000","last_undegraded":"2024-09-13T17:20:40.620674+0000","last_fullsized":"2024-09-13T17:20:40.620674+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:25.220159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620102+0000","last_change":"2024-09-13T17:20:39.639141+0000","last_active":"2024-09-13T17:20:40.620102+0000","last_peered":"2024-09-13T17:20:40.620102+0000","last_clean":"2024-09-13T17:20:40.620102+0000","last_became_active":"2024-09-13T17:20:39.638838+0000","last_became_peered":"2024-09-13T17:20:39.638838+0000","last_unstale":"2024-09-13T17:20:40.620102+0000","last_undegraded":"2024-09-13T17:20:40.620102+0000","last_fullsized":"2024-09-13T17:20:40.620102+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.554603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985000+0000","last_change":"2024-09-13T17:20:39.626846+0000","last_active":"2024-09-13T17:20:40.985000+0000","last_peered":"2024-09-13T17:20:40.985000+0000","last_clean":"2024-09-13T17:20:40.985000+0000","last_became_active":"2024-09-13T17:20:39.626502+0000","last_became_peered":"2024-09-13T17:20:39.626502+0000","last_unstale":"2024-09-13T17:20:40.985000+0000","last_undegraded":"2024-09-13T17:20:40.985000+0000","last_fullsized":"2024-09-13T17:20:40.985000+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144177+0000","last_change":"2024-09-13T17:20:39.641571+0000","last_active":"2024-09-13T17:20:41.144177+0000","last_peered":"2024-09-13T17:20:41.144177+0000","last_clean":"2024-09-13T17:20:41.144177+0000","last_became_active":"2024-09-13T17:20:39.641313+0000","last_became_peered":"2024-09-13T17:20:39.641313+0000","last_unstale":"2024-09-13T17:20:41.144177+0000","last_undegraded":"2024-09-13T17:20:41.144177+0000","last_fullsized":"2024-09-13T17:20:41.144177+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.246356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620568+0000","last_change":"2024-09-13T17:20:39.639587+0000","last_active":"2024-09-13T17:20:40.620568+0000","last_peered":"2024-09-13T17:20:40.620568+0000","last_clean":"2024-09-13T17:20:40.620568+0000","last_became_active":"2024-09-13T17:20:39.639337+0000","last_became_peered":"2024-09-13T17:20:39.639337+0000","last_unstale":"2024-09-13T17:20:40.620568+0000","last_undegraded":"2024-09-13T17:20:40.620568+0000","last_fullsized":"2024-09-13T17:20:40.620568+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:31.957995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678098+0000","last_change":"2024-09-13T17:20:39.624500+0000","last_active":"2024-09-13T17:20:40.678098+0000","last_peered":"2024-09-13T17:20:40.678098+0000","last_clean":"2024-09-13T17:20:40.678098+0000","last_became_active":"2024-09-13T17:20:39.624188+0000","last_became_peered":"2024-09-13T17:20:39.624188+0000","last_unstale":"2024-09-13T17:20:40.678098+0000","last_undegraded":"2024-09-13T17:20:40.678098+0000","last_fullsized":"2024-09-13T17:20:40.678098+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984817+0000","last_change":"2024-09-13T17:20:39.635575+0000","last_active":"2024-09-13T17:20:40.984817+0000","last_peered":"2024-09-13T17:20:40.984817+0000","last_clean":"2024-09-13T17:20:40.984817+0000","last_became_active":"2024-09-13T17:20:39.635246+0000","last_became_peered":"2024-09-13T17:20:39.635246+0000","last_unstale":"2024-09-13T17:20:40.984817+0000","last_undegraded":"2024-09-13T17:20:40.984817+0000","last_fullsized":"2024-09-13T17:20:40.984817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143914+0000","last_change":"2024-09-13T17:20:39.630955+0000","last_active":"2024-09-13T17:20:41.143914+0000","last_peered":"2024-09-13T17:20:41.143914+0000","last_clean":"2024-09-13T17:20:41.143914+0000","last_became_active":"2024-09-13T17:20:39.630697+0000","last_became_peered":"2024-09-13T17:20:39.630697+0000","last_unstale":"2024-09-13T17:20:41.143914+0000","last_undegraded":"2024-09-13T17:20:41.143914+0000","last_fullsized":"2024-09-13T17:20:41.143914+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:00.743598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143803+0000","last_change":"2024-09-13T17:20:39.631816+0000","last_active":"2024-09-13T17:20:41.143803+0000","last_peered":"2024-09-13T17:20:41.143803+0000","last_clean":"2024-09-13T17:20:41.143803+0000","last_became_active":"2024-09-13T17:20:39.631588+0000","last_became_peered":"2024-09-13T17:20:39.631588+0000","last_unstale":"2024-09-13T17:20:41.143803+0000","last_undegraded":"2024-09-13T17:20:41.143803+0000","last_fullsized":"2024-09-13T17:20:41.143803+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:52.109474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"178'4","reported_seq":15,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144475+0000","last_change":"2024-09-13T17:20:39.198543+0000","last_active":"2024-09-13T17:20:41.144475+0000","last_peered":"2024-09-13T17:20:41.144475+0000","last_clean":"2024-09-13T17:20:41.144475+0000","last_became_active":"2024-09-13T17:20:39.198183+0000","last_became_peered":"2024-09-13T17:20:39.198183+0000","last_unstale":"2024-09-13T17:20:41.144475+0000","last_undegraded":"2024-09-13T17:20:41.144475+0000","last_fullsized":"2024-09-13T17:20:41.144475+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:09.570377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618473+0000","last_change":"2024-09-13T17:20:39.643611+0000","last_active":"2024-09-13T17:20:40.618473+0000","last_peered":"2024-09-13T17:20:40.618473+0000","last_clean":"2024-09-13T17:20:40.618473+0000","last_became_active":"2024-09-13T17:20:39.641581+0000","last_became_peered":"2024-09-13T17:20:39.641581+0000","last_unstale":"2024-09-13T17:20:40.618473+0000","last_undegraded":"2024-09-13T17:20:40.618473+0000","last_fullsized":"2024-09-13T17:20:40.618473+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:35.572828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678375+0000","last_change":"2024-09-13T17:20:39.190999+0000","last_active":"2024-09-13T17:20:40.678375+0000","last_peered":"2024-09-13T17:20:40.678375+0000","last_clean":"2024-09-13T17:20:40.678375+0000","last_became_active":"2024-09-13T17:20:39.190656+0000","last_became_peered":"2024-09-13T17:20:39.190656+0000","last_unstale":"2024-09-13T17:20:40.678375+0000","last_undegraded":"2024-09-13T17:20:40.678375+0000","last_fullsized":"2024-09-13T17:20:40.678375+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:13.591611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619833+0000","last_change":"2024-09-13T17:20:39.636918+0000","last_active":"2024-09-13T17:20:40.619833+0000","last_peered":"2024-09-13T17:20:40.619833+0000","last_clean":"2024-09-13T17:20:40.619833+0000","last_became_active":"2024-09-13T17:20:39.636647+0000","last_became_peered":"2024-09-13T17:20:39.636647+0000","last_unstale":"2024-09-13T17:20:40.619833+0000","last_undegraded":"2024-09-13T17:20:40.619833+0000","last_fullsized":"2024-09-13T17:20:40.619833+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:44.679816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985078+0000","last_change":"2024-09-13T17:20:39.195861+0000","last_active":"2024-09-13T17:20:40.985078+0000","last_peered":"2024-09-13T17:20:40.985078+0000","last_clean":"2024-09-13T17:20:40.985078+0000","last_became_active":"2024-09-13T17:20:39.195603+0000","last_became_peered":"2024-09-13T17:20:39.195603+0000","last_unstale":"2024-09-13T17:20:40.985078+0000","last_undegraded":"2024-09-13T17:20:40.985078+0000","last_fullsized":"2024-09-13T17:20:40.985078+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:20.042947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678349+0000","last_change":"2024-09-13T17:20:39.626164+0000","last_active":"2024-09-13T17:20:40.678349+0000","last_peered":"2024-09-13T17:20:40.678349+0000","last_clean":"2024-09-13T17:20:40.678349+0000","last_became_active":"2024-09-13T17:20:39.625823+0000","last_became_peered":"2024-09-13T17:20:39.625823+0000","last_unstale":"2024-09-13T17:20:40.678349+0000","last_undegraded":"2024-09-13T17:20:40.678349+0000","last_fullsized":"2024-09-13T17:20:40.678349+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620686+0000","last_change":"2024-09-13T17:20:39.199632+0000","last_active":"2024-09-13T17:20:40.620686+0000","last_peered":"2024-09-13T17:20:40.620686+0000","last_clean":"2024-09-13T17:20:40.620686+0000","last_became_active":"2024-09-13T17:20:39.199352+0000","last_became_peered":"2024-09-13T17:20:39.199352+0000","last_unstale":"2024-09-13T17:20:40.620686+0000","last_undegraded":"2024-09-13T17:20:40.620686+0000","last_fullsized":"2024-09-13T17:20:40.620686+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:58.689861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618407+0000","last_change":"2024-09-13T17:20:39.648663+0000","last_active":"2024-09-13T17:20:40.618407+0000","last_peered":"2024-09-13T17:20:40.618407+0000","last_clean":"2024-09-13T17:20:40.618407+0000","last_became_active":"2024-09-13T17:20:39.646946+0000","last_became_peered":"2024-09-13T17:20:39.646946+0000","last_unstale":"2024-09-13T17:20:40.618407+0000","last_undegraded":"2024-09-13T17:20:40.618407+0000","last_fullsized":"2024-09-13T17:20:40.618407+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:25.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144268+0000","last_change":"2024-09-13T17:20:39.197366+0000","last_active":"2024-09-13T17:20:41.144268+0000","last_peered":"2024-09-13T17:20:41.144268+0000","last_clean":"2024-09-13T17:20:41.144268+0000","last_became_active":"2024-09-13T17:20:39.197112+0000","last_became_peered":"2024-09-13T17:20:39.197112+0000","last_unstale":"2024-09-13T17:20:41.144268+0000","last_undegraded":"2024-09-13T17:20:41.144268+0000","last_fullsized":"2024-09-13T17:20:41.144268+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:43.295845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143602+0000","last_change":"2024-09-13T17:20:39.632993+0000","last_active":"2024-09-13T17:20:41.143602+0000","last_peered":"2024-09-13T17:20:41.143602+0000","last_clean":"2024-09-13T17:20:41.143602+0000","last_became_active":"2024-09-13T17:20:39.632593+0000","last_became_peered":"2024-09-13T17:20:39.632593+0000","last_unstale":"2024-09-13T17:20:41.143602+0000","last_undegraded":"2024-09-13T17:20:41.143602+0000","last_fullsized":"2024-09-13T17:20:41.143602+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:43.557625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143612+0000","last_change":"2024-09-13T17:20:39.206148+0000","last_active":"2024-09-13T17:20:41.143612+0000","last_peered":"2024-09-13T17:20:41.143612+0000","last_clean":"2024-09-13T17:20:41.143612+0000","last_became_active":"2024-09-13T17:20:39.205888+0000","last_became_peered":"2024-09-13T17:20:39.205888+0000","last_unstale":"2024-09-13T17:20:41.143612+0000","last_undegraded":"2024-09-13T17:20:41.143612+0000","last_fullsized":"2024-09-13T17:20:41.143612+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:24.675079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620379+0000","last_change":"2024-09-13T17:20:39.629179+0000","last_active":"2024-09-13T17:20:40.620379+0000","last_peered":"2024-09-13T17:20:40.620379+0000","last_clean":"2024-09-13T17:20:40.620379+0000","last_became_active":"2024-09-13T17:20:39.626826+0000","last_became_peered":"2024-09-13T17:20:39.626826+0000","last_unstale":"2024-09-13T17:20:40.620379+0000","last_undegraded":"2024-09-13T17:20:40.620379+0000","last_fullsized":"2024-09-13T17:20:40.620379+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:41.217125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143590+0000","last_change":"2024-09-13T17:20:39.201413+0000","last_active":"2024-09-13T17:20:41.143590+0000","last_peered":"2024-09-13T17:20:41.143590+0000","last_clean":"2024-09-13T17:20:41.143590+0000","last_became_active":"2024-09-13T17:20:39.200200+0000","last_became_peered":"2024-09-13T17:20:39.200200+0000","last_unstale":"2024-09-13T17:20:41.143590+0000","last_undegraded":"2024-09-13T17:20:41.143590+0000","last_fullsized":"2024-09-13T17:20:41.143590+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:47.990493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985066+0000","last_change":"2024-09-13T17:20:39.625360+0000","last_active":"2024-09-13T17:20:40.985066+0000","last_peered":"2024-09-13T17:20:40.985066+0000","last_clean":"2024-09-13T17:20:40.985066+0000","last_became_active":"2024-09-13T17:20:39.625153+0000","last_became_peered":"2024-09-13T17:20:39.625153+0000","last_unstale":"2024-09-13T17:20:40.985066+0000","last_undegraded":"2024-09-13T17:20:40.985066+0000","last_fullsized":"2024-09-13T17:20:40.985066+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.761569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620298+0000","last_change":"2024-09-13T17:20:39.199376+0000","last_active":"2024-09-13T17:20:40.620298+0000","last_peered":"2024-09-13T17:20:40.620298+0000","last_clean":"2024-09-13T17:20:40.620298+0000","last_became_active":"2024-09-13T17:20:39.198269+0000","last_became_peered":"2024-09-13T17:20:39.198269+0000","last_unstale":"2024-09-13T17:20:40.620298+0000","last_undegraded":"2024-09-13T17:20:40.620298+0000","last_fullsized":"2024-09-13T17:20:40.620298+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:37.898138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143763+0000","last_change":"2024-09-13T17:20:39.640835+0000","last_active":"2024-09-13T17:20:41.143763+0000","last_peered":"2024-09-13T17:20:41.143763+0000","last_clean":"2024-09-13T17:20:41.143763+0000","last_became_active":"2024-09-13T17:20:39.640548+0000","last_became_peered":"2024-09-13T17:20:39.640548+0000","last_unstale":"2024-09-13T17:20:41.143763+0000","last_undegraded":"2024-09-13T17:20:41.143763+0000","last_fullsized":"2024-09-13T17:20:41.143763+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:01.126089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144172+0000","last_change":"2024-09-13T17:20:39.205506+0000","last_active":"2024-09-13T17:20:41.144172+0000","last_peered":"2024-09-13T17:20:41.144172+0000","last_clean":"2024-09-13T17:20:41.144172+0000","last_became_active":"2024-09-13T17:20:39.204481+0000","last_became_peered":"2024-09-13T17:20:39.204481+0000","last_unstale":"2024-09-13T17:20:41.144172+0000","last_undegraded":"2024-09-13T17:20:41.144172+0000","last_fullsized":"2024-09-13T17:20:41.144172+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:57.501218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619833+0000","last_change":"2024-09-13T17:20:39.625709+0000","last_active":"2024-09-13T17:20:40.619833+0000","last_peered":"2024-09-13T17:20:40.619833+0000","last_clean":"2024-09-13T17:20:40.619833+0000","last_became_active":"2024-09-13T17:20:39.625450+0000","last_became_peered":"2024-09-13T17:20:39.625450+0000","last_unstale":"2024-09-13T17:20:40.619833+0000","last_undegraded":"2024-09-13T17:20:40.619833+0000","last_fullsized":"2024-09-13T17:20:40.619833+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:01.811707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"178'3","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144192+0000","last_change":"2024-09-13T17:20:39.197453+0000","last_active":"2024-09-13T17:20:41.144192+0000","last_peered":"2024-09-13T17:20:41.144192+0000","last_clean":"2024-09-13T17:20:41.144192+0000","last_became_active":"2024-09-13T17:20:39.197185+0000","last_became_peered":"2024-09-13T17:20:39.197185+0000","last_unstale":"2024-09-13T17:20:41.144192+0000","last_undegraded":"2024-09-13T17:20:41.144192+0000","last_fullsized":"2024-09-13T17:20:41.144192+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:00.761299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618270+0000","last_change":"2024-09-13T17:20:39.642856+0000","last_active":"2024-09-13T17:20:40.618270+0000","last_peered":"2024-09-13T17:20:40.618270+0000","last_clean":"2024-09-13T17:20:40.618270+0000","last_became_active":"2024-09-13T17:20:39.641881+0000","last_became_peered":"2024-09-13T17:20:39.641881+0000","last_unstale":"2024-09-13T17:20:40.618270+0000","last_undegraded":"2024-09-13T17:20:40.618270+0000","last_fullsized":"2024-09-13T17:20:40.618270+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:35.337787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"178'6","reported_seq":17,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984922+0000","last_change":"2024-09-13T17:20:39.199323+0000","last_active":"2024-09-13T17:20:40.984922+0000","last_peered":"2024-09-13T17:20:40.984922+0000","last_clean":"2024-09-13T17:20:40.984922+0000","last_became_active":"2024-09-13T17:20:39.199063+0000","last_became_peered":"2024-09-13T17:20:39.199063+0000","last_unstale":"2024-09-13T17:20:40.984922+0000","last_undegraded":"2024-09-13T17:20:40.984922+0000","last_fullsized":"2024-09-13T17:20:40.984922+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:13.591611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984876+0000","last_change":"2024-09-13T17:20:39.641027+0000","last_active":"2024-09-13T17:20:40.984876+0000","last_peered":"2024-09-13T17:20:40.984876+0000","last_clean":"2024-09-13T17:20:40.984876+0000","last_became_active":"2024-09-13T17:20:39.640780+0000","last_became_peered":"2024-09-13T17:20:39.640780+0000","last_unstale":"2024-09-13T17:20:40.984876+0000","last_undegraded":"2024-09-13T17:20:40.984876+0000","last_fullsized":"2024-09-13T17:20:40.984876+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144032+0000","last_change":"2024-09-13T17:20:39.201708+0000","last_active":"2024-09-13T17:20:41.144032+0000","last_peered":"2024-09-13T17:20:41.144032+0000","last_clean":"2024-09-13T17:20:41.144032+0000","last_became_active":"2024-09-13T17:20:39.201308+0000","last_became_peered":"2024-09-13T17:20:39.201308+0000","last_unstale":"2024-09-13T17:20:41.144032+0000","last_undegraded":"2024-09-13T17:20:41.144032+0000","last_fullsized":"2024-09-13T17:20:41.144032+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:37.898138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620014+0000","last_change":"2024-09-13T17:20:39.642348+0000","last_active":"2024-09-13T17:20:40.620014+0000","last_peered":"2024-09-13T17:20:40.620014+0000","last_clean":"2024-09-13T17:20:40.620014+0000","last_became_active":"2024-09-13T17:20:39.640237+0000","last_became_peered":"2024-09-13T17:20:39.640237+0000","last_unstale":"2024-09-13T17:20:40.620014+0000","last_undegraded":"2024-09-13T17:20:40.620014+0000","last_fullsized":"2024-09-13T17:20:40.620014+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:55.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618501+0000","last_change":"2024-09-13T17:20:39.207181+0000","last_active":"2024-09-13T17:20:40.618501+0000","last_peered":"2024-09-13T17:20:40.618501+0000","last_clean":"2024-09-13T17:20:40.618501+0000","last_became_active":"2024-09-13T17:20:39.206785+0000","last_became_peered":"2024-09-13T17:20:39.206785+0000","last_unstale":"2024-09-13T17:20:40.618501+0000","last_undegraded":"2024-09-13T17:20:40.618501+0000","last_fullsized":"2024-09-13T17:20:40.618501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:27.598580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.677983+0000","last_change":"2024-09-13T17:20:39.640381+0000","last_active":"2024-09-13T17:20:40.677983+0000","last_peered":"2024-09-13T17:20:40.677983+0000","last_clean":"2024-09-13T17:20:40.677983+0000","last_became_active":"2024-09-13T17:20:39.640051+0000","last_became_peered":"2024-09-13T17:20:39.640051+0000","last_unstale":"2024-09-13T17:20:40.677983+0000","last_undegraded":"2024-09-13T17:20:40.677983+0000","last_fullsized":"2024-09-13T17:20:40.677983+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143404+0000","last_change":"2024-09-13T17:20:39.204522+0000","last_active":"2024-09-13T17:20:41.143404+0000","last_peered":"2024-09-13T17:20:41.143404+0000","last_clean":"2024-09-13T17:20:41.143404+0000","last_became_active":"2024-09-13T17:20:39.204217+0000","last_became_peered":"2024-09-13T17:20:39.204217+0000","last_unstale":"2024-09-13T17:20:41.143404+0000","last_undegraded":"2024-09-13T17:20:41.143404+0000","last_fullsized":"2024-09-13T17:20:41.143404+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:24.405013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618079+0000","last_change":"2024-09-13T17:20:39.643620+0000","last_active":"2024-09-13T17:20:40.618079+0000","last_peered":"2024-09-13T17:20:40.618079+0000","last_clean":"2024-09-13T17:20:40.618079+0000","last_became_active":"2024-09-13T17:20:39.637801+0000","last_became_peered":"2024-09-13T17:20:39.637801+0000","last_unstale":"2024-09-13T17:20:40.618079+0000","last_undegraded":"2024-09-13T17:20:40.618079+0000","last_fullsized":"2024-09-13T17:20:40.618079+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:23.440511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"178'8","reported_seq":19,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143759+0000","last_change":"2024-09-13T17:20:39.202745+0000","last_active":"2024-09-13T17:20:41.143759+0000","last_peered":"2024-09-13T17:20:41.143759+0000","last_clean":"2024-09-13T17:20:41.143759+0000","last_became_active":"2024-09-13T17:20:39.199696+0000","last_became_peered":"2024-09-13T17:20:39.199696+0000","last_unstale":"2024-09-13T17:20:41.143759+0000","last_undegraded":"2024-09-13T17:20:41.143759+0000","last_fullsized":"2024-09-13T17:20:41.143759+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:24.754149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618741+0000","last_change":"2024-09-13T17:20:39.637107+0000","last_active":"2024-09-13T17:20:40.618741+0000","last_peered":"2024-09-13T17:20:40.618741+0000","last_clean":"2024-09-13T17:20:40.618741+0000","last_became_active":"2024-09-13T17:20:39.636743+0000","last_became_peered":"2024-09-13T17:20:39.636743+0000","last_unstale":"2024-09-13T17:20:40.618741+0000","last_undegraded":"2024-09-13T17:20:40.618741+0000","last_fullsized":"2024-09-13T17:20:40.618741+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:50.231822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144476+0000","last_change":"2024-09-13T17:20:39.204155+0000","last_active":"2024-09-13T17:20:41.144476+0000","last_peered":"2024-09-13T17:20:41.144476+0000","last_clean":"2024-09-13T17:20:41.144476+0000","last_became_active":"2024-09-13T17:20:39.203844+0000","last_became_peered":"2024-09-13T17:20:39.203844+0000","last_unstale":"2024-09-13T17:20:41.144476+0000","last_undegraded":"2024-09-13T17:20:41.144476+0000","last_fullsized":"2024-09-13T17:20:41.144476+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:00:52.830750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678466+0000","last_change":"2024-09-13T17:20:39.641302+0000","last_active":"2024-09-13T17:20:40.678466+0000","last_peered":"2024-09-13T17:20:40.678466+0000","last_clean":"2024-09-13T17:20:40.678466+0000","last_became_active":"2024-09-13T17:20:39.641028+0000","last_became_peered":"2024-09-13T17:20:39.641028+0000","last_unstale":"2024-09-13T17:20:40.678466+0000","last_undegraded":"2024-09-13T17:20:40.678466+0000","last_fullsized":"2024-09-13T17:20:40.678466+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.761569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143793+0000","last_change":"2024-09-13T17:20:39.206500+0000","last_active":"2024-09-13T17:20:41.143793+0000","last_peered":"2024-09-13T17:20:41.143793+0000","last_clean":"2024-09-13T17:20:41.143793+0000","last_became_active":"2024-09-13T17:20:39.204754+0000","last_became_peered":"2024-09-13T17:20:39.204754+0000","last_unstale":"2024-09-13T17:20:41.143793+0000","last_undegraded":"2024-09-13T17:20:41.143793+0000","last_fullsized":"2024-09-13T17:20:41.143793+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:41.516559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678209+0000","last_change":"2024-09-13T17:20:39.627103+0000","last_active":"2024-09-13T17:20:40.678209+0000","last_peered":"2024-09-13T17:20:40.678209+0000","last_clean":"2024-09-13T17:20:40.678209+0000","last_became_active":"2024-09-13T17:20:39.626856+0000","last_became_peered":"2024-09-13T17:20:39.626856+0000","last_unstale":"2024-09-13T17:20:40.678209+0000","last_undegraded":"2024-09-13T17:20:40.678209+0000","last_fullsized":"2024-09-13T17:20:40.678209+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144004+0000","last_change":"2024-09-13T17:20:39.204614+0000","last_active":"2024-09-13T17:20:41.144004+0000","last_peered":"2024-09-13T17:20:41.144004+0000","last_clean":"2024-09-13T17:20:41.144004+0000","last_became_active":"2024-09-13T17:20:39.204225+0000","last_became_peered":"2024-09-13T17:20:39.204225+0000","last_unstale":"2024-09-13T17:20:41.144004+0000","last_undegraded":"2024-09-13T17:20:41.144004+0000","last_fullsized":"2024-09-13T17:20:41.144004+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:24.741949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620517+0000","last_change":"2024-09-13T17:20:39.642381+0000","last_active":"2024-09-13T17:20:40.620517+0000","last_peered":"2024-09-13T17:20:40.620517+0000","last_clean":"2024-09-13T17:20:40.620517+0000","last_became_active":"2024-09-13T17:20:39.642111+0000","last_became_peered":"2024-09-13T17:20:39.642111+0000","last_unstale":"2024-09-13T17:20:40.620517+0000","last_undegraded":"2024-09-13T17:20:40.620517+0000","last_fullsized":"2024-09-13T17:20:40.620517+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619923+0000","last_change":"2024-09-13T17:20:39.200576+0000","last_active":"2024-09-13T17:20:40.619923+0000","last_peered":"2024-09-13T17:20:40.619923+0000","last_clean":"2024-09-13T17:20:40.619923+0000","last_became_active":"2024-09-13T17:20:39.200219+0000","last_became_peered":"2024-09-13T17:20:39.200219+0000","last_unstale":"2024-09-13T17:20:40.619923+0000","last_undegraded":"2024-09-13T17:20:40.619923+0000","last_fullsized":"2024-09-13T17:20:40.619923+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:24.741949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618445+0000","last_change":"2024-09-13T17:20:39.641324+0000","last_active":"2024-09-13T17:20:40.618445+0000","last_peered":"2024-09-13T17:20:40.618445+0000","last_clean":"2024-09-13T17:20:40.618445+0000","last_became_active":"2024-09-13T17:20:39.640177+0000","last_became_peered":"2024-09-13T17:20:39.640177+0000","last_unstale":"2024-09-13T17:20:40.618445+0000","last_undegraded":"2024-09-13T17:20:40.618445+0000","last_fullsized":"2024-09-13T17:20:40.618445+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:40.520569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984788+0000","last_change":"2024-09-13T17:20:39.206177+0000","last_active":"2024-09-13T17:20:40.984788+0000","last_peered":"2024-09-13T17:20:40.984788+0000","last_clean":"2024-09-13T17:20:40.984788+0000","last_became_active":"2024-09-13T17:20:39.205861+0000","last_became_peered":"2024-09-13T17:20:39.205861+0000","last_unstale":"2024-09-13T17:20:40.984788+0000","last_undegraded":"2024-09-13T17:20:40.984788+0000","last_fullsized":"2024-09-13T17:20:40.984788+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:02.840549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620121+0000","last_change":"2024-09-13T17:20:39.625469+0000","last_active":"2024-09-13T17:20:40.620121+0000","last_peered":"2024-09-13T17:20:40.620121+0000","last_clean":"2024-09-13T17:20:40.620121+0000","last_became_active":"2024-09-13T17:20:39.625159+0000","last_became_peered":"2024-09-13T17:20:39.625159+0000","last_unstale":"2024-09-13T17:20:40.620121+0000","last_undegraded":"2024-09-13T17:20:40.620121+0000","last_fullsized":"2024-09-13T17:20:40.620121+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:28.606000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144067+0000","last_change":"2024-09-13T17:20:39.201866+0000","last_active":"2024-09-13T17:20:41.144067+0000","last_peered":"2024-09-13T17:20:41.144067+0000","last_clean":"2024-09-13T17:20:41.144067+0000","last_became_active":"2024-09-13T17:20:39.201615+0000","last_became_peered":"2024-09-13T17:20:39.201615+0000","last_unstale":"2024-09-13T17:20:41.144067+0000","last_undegraded":"2024-09-13T17:20:41.144067+0000","last_fullsized":"2024-09-13T17:20:41.144067+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:39:48.919267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619546+0000","last_change":"2024-09-13T17:20:39.641296+0000","last_active":"2024-09-13T17:20:40.619546+0000","last_peered":"2024-09-13T17:20:40.619546+0000","last_clean":"2024-09-13T17:20:40.619546+0000","last_became_active":"2024-09-13T17:20:39.640974+0000","last_became_peered":"2024-09-13T17:20:39.640974+0000","last_unstale":"2024-09-13T17:20:40.619546+0000","last_undegraded":"2024-09-13T17:20:40.619546+0000","last_fullsized":"2024-09-13T17:20:40.619546+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:06.952818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984889+0000","last_change":"2024-09-13T17:20:39.207372+0000","last_active":"2024-09-13T17:20:40.984889+0000","last_peered":"2024-09-13T17:20:40.984889+0000","last_clean":"2024-09-13T17:20:40.984889+0000","last_became_active":"2024-09-13T17:20:39.207086+0000","last_became_peered":"2024-09-13T17:20:39.207086+0000","last_unstale":"2024-09-13T17:20:40.984889+0000","last_undegraded":"2024-09-13T17:20:40.984889+0000","last_fullsized":"2024-09-13T17:20:40.984889+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:03.491122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143958+0000","last_change":"2024-09-13T17:20:39.632790+0000","last_active":"2024-09-13T17:20:41.143958+0000","last_peered":"2024-09-13T17:20:41.143958+0000","last_clean":"2024-09-13T17:20:41.143958+0000","last_became_active":"2024-09-13T17:20:39.632550+0000","last_became_peered":"2024-09-13T17:20:39.632550+0000","last_unstale":"2024-09-13T17:20:41.143958+0000","last_undegraded":"2024-09-13T17:20:41.143958+0000","last_fullsized":"2024-09-13T17:20:41.143958+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:10.019707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678075+0000","last_change":"2024-09-13T17:20:39.201922+0000","last_active":"2024-09-13T17:20:40.678075+0000","last_peered":"2024-09-13T17:20:40.678075+0000","last_clean":"2024-09-13T17:20:40.678075+0000","last_became_active":"2024-09-13T17:20:39.201679+0000","last_became_peered":"2024-09-13T17:20:39.201679+0000","last_unstale":"2024-09-13T17:20:40.678075+0000","last_undegraded":"2024-09-13T17:20:40.678075+0000","last_fullsized":"2024-09-13T17:20:40.678075+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:34.705513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144179+0000","last_change":"2024-09-13T17:20:39.632118+0000","last_active":"2024-09-13T17:20:41.144179+0000","last_peered":"2024-09-13T17:20:41.144179+0000","last_clean":"2024-09-13T17:20:41.144179+0000","last_became_active":"2024-09-13T17:20:39.631858+0000","last_became_peered":"2024-09-13T17:20:39.631858+0000","last_unstale":"2024-09-13T17:20:41.144179+0000","last_undegraded":"2024-09-13T17:20:41.144179+0000","last_fullsized":"2024-09-13T17:20:41.144179+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:38.561207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144047+0000","last_change":"2024-09-13T17:20:39.205299+0000","last_active":"2024-09-13T17:20:41.144047+0000","last_peered":"2024-09-13T17:20:41.144047+0000","last_clean":"2024-09-13T17:20:41.144047+0000","last_became_active":"2024-09-13T17:20:39.204944+0000","last_became_peered":"2024-09-13T17:20:39.204944+0000","last_unstale":"2024-09-13T17:20:41.144047+0000","last_undegraded":"2024-09-13T17:20:41.144047+0000","last_fullsized":"2024-09-13T17:20:41.144047+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:48.671196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618557+0000","last_change":"2024-09-13T17:20:39.639222+0000","last_active":"2024-09-13T17:20:40.618557+0000","last_peered":"2024-09-13T17:20:40.618557+0000","last_clean":"2024-09-13T17:20:40.618557+0000","last_became_active":"2024-09-13T17:20:39.638967+0000","last_became_peered":"2024-09-13T17:20:39.638967+0000","last_unstale":"2024-09-13T17:20:40.618557+0000","last_undegraded":"2024-09-13T17:20:40.618557+0000","last_fullsized":"2024-09-13T17:20:40.618557+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:37.374048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618695+0000","last_change":"2024-09-13T17:20:39.195809+0000","last_active":"2024-09-13T17:20:40.618695+0000","last_peered":"2024-09-13T17:20:40.618695+0000","last_clean":"2024-09-13T17:20:40.618695+0000","last_became_active":"2024-09-13T17:20:39.195286+0000","last_became_peered":"2024-09-13T17:20:39.195286+0000","last_unstale":"2024-09-13T17:20:40.618695+0000","last_undegraded":"2024-09-13T17:20:40.618695+0000","last_fullsized":"2024-09-13T17:20:40.618695+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:34.244257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143641+0000","last_change":"2024-09-13T17:20:39.626675+0000","last_active":"2024-09-13T17:20:41.143641+0000","last_peered":"2024-09-13T17:20:41.143641+0000","last_clean":"2024-09-13T17:20:41.143641+0000","last_became_active":"2024-09-13T17:20:39.625923+0000","last_became_peered":"2024-09-13T17:20:39.625923+0000","last_unstale":"2024-09-13T17:20:41.143641+0000","last_undegraded":"2024-09-13T17:20:41.143641+0000","last_fullsized":"2024-09-13T17:20:41.143641+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:11.695712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618362+0000","last_change":"2024-09-13T17:20:39.200677+0000","last_active":"2024-09-13T17:20:40.618362+0000","last_peered":"2024-09-13T17:20:40.618362+0000","last_clean":"2024-09-13T17:20:40.618362+0000","last_became_active":"2024-09-13T17:20:39.200389+0000","last_became_peered":"2024-09-13T17:20:39.200389+0000","last_unstale":"2024-09-13T17:20:40.618362+0000","last_undegraded":"2024-09-13T17:20:40.618362+0000","last_fullsized":"2024-09-13T17:20:40.618362+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:51.547183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143735+0000","last_change":"2024-09-13T17:20:39.632884+0000","last_active":"2024-09-13T17:20:41.143735+0000","last_peered":"2024-09-13T17:20:41.143735+0000","last_clean":"2024-09-13T17:20:41.143735+0000","last_became_active":"2024-09-13T17:20:39.632114+0000","last_became_peered":"2024-09-13T17:20:39.632114+0000","last_unstale":"2024-09-13T17:20:41.143735+0000","last_undegraded":"2024-09-13T17:20:41.143735+0000","last_fullsized":"2024-09-13T17:20:41.143735+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:16:56.033401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143663+0000","last_change":"2024-09-13T17:20:39.200932+0000","last_active":"2024-09-13T17:20:41.143663+0000","last_peered":"2024-09-13T17:20:41.143663+0000","last_clean":"2024-09-13T17:20:41.143663+0000","last_became_active":"2024-09-13T17:20:39.200611+0000","last_became_peered":"2024-09-13T17:20:39.200611+0000","last_unstale":"2024-09-13T17:20:41.143663+0000","last_undegraded":"2024-09-13T17:20:41.143663+0000","last_fullsized":"2024-09-13T17:20:41.143663+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:35.251693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143907+0000","last_change":"2024-09-13T17:20:39.643281+0000","last_active":"2024-09-13T17:20:41.143907+0000","last_peered":"2024-09-13T17:20:41.143907+0000","last_clean":"2024-09-13T17:20:41.143907+0000","last_became_active":"2024-09-13T17:20:39.642969+0000","last_became_peered":"2024-09-13T17:20:39.642969+0000","last_unstale":"2024-09-13T17:20:41.143907+0000","last_undegraded":"2024-09-13T17:20:41.143907+0000","last_fullsized":"2024-09-13T17:20:41.143907+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:30.506534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678225+0000","last_change":"2024-09-13T17:20:39.199454+0000","last_active":"2024-09-13T17:20:40.678225+0000","last_peered":"2024-09-13T17:20:40.678225+0000","last_clean":"2024-09-13T17:20:40.678225+0000","last_became_active":"2024-09-13T17:20:39.198310+0000","last_became_peered":"2024-09-13T17:20:39.198310+0000","last_unstale":"2024-09-13T17:20:40.678225+0000","last_undegraded":"2024-09-13T17:20:40.678225+0000","last_fullsized":"2024-09-13T17:20:40.678225+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:20.042947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143907+0000","last_change":"2024-09-13T17:20:39.631945+0000","last_active":"2024-09-13T17:20:41.143907+0000","last_peered":"2024-09-13T17:20:41.143907+0000","last_clean":"2024-09-13T17:20:41.143907+0000","last_became_active":"2024-09-13T17:20:39.631096+0000","last_became_peered":"2024-09-13T17:20:39.631096+0000","last_unstale":"2024-09-13T17:20:41.143907+0000","last_undegraded":"2024-09-13T17:20:41.143907+0000","last_fullsized":"2024-09-13T17:20:41.143907+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:37.139362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678199+0000","last_change":"2024-09-13T17:20:39.196860+0000","last_active":"2024-09-13T17:20:40.678199+0000","last_peered":"2024-09-13T17:20:40.678199+0000","last_clean":"2024-09-13T17:20:40.678199+0000","last_became_active":"2024-09-13T17:20:39.196597+0000","last_became_peered":"2024-09-13T17:20:39.196597+0000","last_unstale":"2024-09-13T17:20:40.678199+0000","last_undegraded":"2024-09-13T17:20:40.678199+0000","last_fullsized":"2024-09-13T17:20:40.678199+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:03.491122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619655+0000","last_change":"2024-09-13T17:20:39.635620+0000","last_active":"2024-09-13T17:20:40.619655+0000","last_peered":"2024-09-13T17:20:40.619655+0000","last_clean":"2024-09-13T17:20:40.619655+0000","last_became_active":"2024-09-13T17:20:39.635324+0000","last_became_peered":"2024-09-13T17:20:39.635324+0000","last_unstale":"2024-09-13T17:20:40.619655+0000","last_undegraded":"2024-09-13T17:20:40.619655+0000","last_fullsized":"2024-09-13T17:20:40.619655+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620144+0000","last_change":"2024-09-13T17:20:39.201780+0000","last_active":"2024-09-13T17:20:40.620144+0000","last_peered":"2024-09-13T17:20:40.620144+0000","last_clean":"2024-09-13T17:20:40.620144+0000","last_became_active":"2024-09-13T17:20:39.201334+0000","last_became_peered":"2024-09-13T17:20:39.201334+0000","last_unstale":"2024-09-13T17:20:40.620144+0000","last_undegraded":"2024-09-13T17:20:40.620144+0000","last_fullsized":"2024-09-13T17:20:40.620144+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:13.936247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144048+0000","last_change":"2024-09-13T17:20:39.647486+0000","last_active":"2024-09-13T17:20:41.144048+0000","last_peered":"2024-09-13T17:20:41.144048+0000","last_clean":"2024-09-13T17:20:41.144048+0000","last_became_active":"2024-09-13T17:20:39.647208+0000","last_became_peered":"2024-09-13T17:20:39.647208+0000","last_unstale":"2024-09-13T17:20:41.144048+0000","last_undegraded":"2024-09-13T17:20:41.144048+0000","last_fullsized":"2024-09-13T17:20:41.144048+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:03.716102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984803+0000","last_change":"2024-09-13T17:20:39.205816+0000","last_active":"2024-09-13T17:20:40.984803+0000","last_peered":"2024-09-13T17:20:40.984803+0000","last_clean":"2024-09-13T17:20:40.984803+0000","last_became_active":"2024-09-13T17:20:39.205546+0000","last_became_peered":"2024-09-13T17:20:39.205546+0000","last_unstale":"2024-09-13T17:20:40.984803+0000","last_undegraded":"2024-09-13T17:20:40.984803+0000","last_fullsized":"2024-09-13T17:20:40.984803+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:44.158088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618014+0000","last_change":"2024-09-13T17:20:39.639611+0000","last_active":"2024-09-13T17:20:40.618014+0000","last_peered":"2024-09-13T17:20:40.618014+0000","last_clean":"2024-09-13T17:20:40.618014+0000","last_became_active":"2024-09-13T17:20:39.638883+0000","last_became_peered":"2024-09-13T17:20:39.638883+0000","last_unstale":"2024-09-13T17:20:40.618014+0000","last_undegraded":"2024-09-13T17:20:40.618014+0000","last_fullsized":"2024-09-13T17:20:40.618014+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:04.149998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143761+0000","last_change":"2024-09-13T17:20:39.205036+0000","last_active":"2024-09-13T17:20:41.143761+0000","last_peered":"2024-09-13T17:20:41.143761+0000","last_clean":"2024-09-13T17:20:41.143761+0000","last_became_active":"2024-09-13T17:20:39.204468+0000","last_became_peered":"2024-09-13T17:20:39.204468+0000","last_unstale":"2024-09-13T17:20:41.143761+0000","last_undegraded":"2024-09-13T17:20:41.143761+0000","last_fullsized":"2024-09-13T17:20:41.143761+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:41.908698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":298,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T17:20:39.625031+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:20:39.625031+0000","last_peered":"2024-09-13T17:20:39.625031+0000","last_clean":"2024-09-13T17:20:39.625031+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:20:39.625031+0000","last_undegraded":"2024-09-13T17:20:39.625031+0000","last_fullsized":"2024-09-13T17:20:39.625031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65844,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305996,"statfs":{"total":96636764160,"available":96569339904,"internally_reserved":0,"allocated":2850816,"data_stored":1583204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64551371},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.625,"15min":0.64500000000000002},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.996,"15min":2.0880000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.65200000000000002,"15min":0.67300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":1.004,"15min":2.6739999999999999},"last":0.71199999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89500000000000002,"5min":1.127,"15min":2.5030000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64600000000000002,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":6.4909999999999997},"last":0.63200000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.67500000000000004,"15min":0.69099999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91700000000000004,"5min":1.169,"15min":1.544},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":1.605},"last":0.77200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83999999999999997,"5min":1.091,"15min":2.4249999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84599999999999997,"5min":1.274,"15min":2.8239999999999998},"last":0.84599999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56899999999999995,"15min":0.61699999999999999},"min":{"1min":0.35799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73899999999999999,"5min":0.879,"15min":1.054},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.747,"5min":1.234,"15min":1.234},"last":0.81100000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61899999999999999,"15min":0.66200000000000003},"min":{"1min":0.39100000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.83699999999999997,"5min":0.99299999999999999,"15min":1.143},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.751},"min":{"1min":0.47999999999999998,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":1.1659999999999999,"15min":1.337},"last":0.73799999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.87,"5min":1.048,"15min":1.1930000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.68600000000000005,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85099999999999998,"5min":1.101,"15min":1.2030000000000001},"last":0.65600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47272,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324568,"statfs":{"total":96636764160,"available":96588357632,"internally_reserved":0,"allocated":2248704,"data_stored":992732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63400000000000001,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89200000000000002,"5min":1.298,"15min":3.6629999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.752,"15min":0.73599999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.81399999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.629,"15min":0.65300000000000002},"min":{"1min":0.32200000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81699999999999995,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40600000000000003,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96899999999999997,"5min":1.46,"15min":3.3079999999999998},"last":0.745}]},{"osd":2,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72299999999999998,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.68700000000000006},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89300000000000002,"5min":0.96899999999999997,"15min":3.2629999999999999},"last":0.871}]},{"osd":3,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.51100000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70099999999999996},"min":{"1min":0.46600000000000003,"5min":0.46200000000000002,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":3.6720000000000002,"15min":3.742},"last":0.79600000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.75800000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.64700000000000002},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.29199999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.39800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.871,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.67600000000000005},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":0.84099999999999997,"5min":0.92100000000000004,"15min":3.609},"last":0.84799999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71299999999999997,"15min":0.73599999999999999},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.91200000000000003,"5min":1.601,"15min":4.9669999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78900000000000003},"min":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.76400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59552,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312288,"statfs":{"total":96636764160,"available":96575782912,"internally_reserved":0,"allocated":2248704,"data_stored":992732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.76600000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85699999999999998,"5min":1.498,"15min":38.005000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.66400000000000003,"15min":0.73999999999999999},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.88800000000000001,"5min":1.577,"15min":21.376999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.72399999999999998},"min":{"1min":0.42599999999999999,"5min":0.38800000000000001,"15min":0.30499999999999999},"max":{"1min":0.85899999999999999,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68200000000000005,"15min":0.69499999999999995},"min":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.90800000000000003,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66600000000000004,"15min":0.73899999999999999},"min":{"1min":0.40600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.99299999999999999,"5min":1.524,"15min":17.268000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68600000000000005,"15min":0.73199999999999998},"min":{"1min":0.439,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90600000000000003,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.82699999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.71499999999999997,"15min":1.0880000000000001},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.95099999999999996,"5min":2.8690000000000002,"15min":113.363},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72499999999999998,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.86599999999999999,"5min":2.359,"15min":4.1180000000000003},"last":0.752}]},{"osd":4,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79500000000000004,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90900000000000003,"5min":2.827,"15min":2.827},"last":0.55400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83399999999999996,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.69699999999999995,"15min":0.71699999999999997},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.83099999999999996,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":1.9379999999999999,"15min":2.02},"last":0.64800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69372,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302468,"statfs":{"total":96636764160,"available":96565727232,"internally_reserved":0,"allocated":2265088,"data_stored":994878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68745670},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0620000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71599999999999997},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.373},"max":{"1min":0.997,"5min":1.3400000000000001,"15min":1.341},"last":0.83899999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95399999999999996,"5min":1.173,"15min":1.173},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96599999999999997,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.65600000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.1100000000000001,"15min":1.1910000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.22,"15min":2.1259999999999999},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.496,"5min":0.378,"15min":0.378},"max":{"1min":0.96699999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":1.089,"5min":1.304,"15min":1.304},"last":0.875}]},{"osd":5,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":1.284,"15min":1.284},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":1.01,"5min":1.3069999999999999,"15min":1.3240000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1,"5min":1.03,"15min":1.274},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.72399999999999998,"15min":0.72899999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":1.069,"5min":1.254,"15min":1.413},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70499999999999996},"min":{"1min":0.46600000000000003,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.127,"5min":1.252,"15min":1.252},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.351,"15min":1.351},"last":0.69299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69364,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302476,"statfs":{"total":96636764160,"available":96565735424,"internally_reserved":0,"allocated":2260992,"data_stored":992926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.28299999999999997,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.316},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.502},"min":{"1min":0.372,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.65300000000000002,"5min":0.68300000000000005,"15min":2.96},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.48199999999999998,"15min":0.50800000000000001},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.28299999999999997},"max":{"1min":0.60199999999999998,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55300000000000005,"15min":0.63},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":3.6829999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.3819999999999999},"last":0.42499999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71499999999999997,"15min":0.75800000000000001},"min":{"1min":0.40600000000000003,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.21,"5min":1.21,"15min":2.8610000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.73699999999999999,"15min":0.76000000000000001},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.3640000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.73299999999999998,"15min":0.76700000000000002},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.391},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.73399999999999999,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.442,"15min":0.442},"max":{"1min":1.224,"5min":1.224,"15min":2.9300000000000002},"last":0.54000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.76400000000000001},"min":{"1min":0.54800000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.401},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.76300000000000001,"15min":0.78200000000000003},"min":{"1min":0.55400000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.3300000000000001},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.77000000000000002,"15min":0.80600000000000005},"min":{"1min":0.53400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.27,"5min":1.27,"15min":2.899},"last":0.626},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.80100000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.67700000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47280,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324560,"statfs":{"total":96636764160,"available":96588349440,"internally_reserved":0,"allocated":2256896,"data_stored":992407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.46899999999999997,"15min":0.436},"min":{"1min":0.27800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.45900000000000002,"15min":0.56799999999999995},"min":{"1min":0.316,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.593,"5min":1.593,"15min":30.023},"last":0.497}]},{"osd":1,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.53600000000000003,"15min":1.365},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":5.7409999999999997,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.52900000000000003,"15min":1.462},"min":{"1min":0.39800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.443}]},{"osd":3,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.54500000000000004,"15min":0.61599999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":5.0229999999999997,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.376},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.65800000000000003,"15min":0.60599999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":8.4149999999999991,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.0039999999999996,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.78800000000000003,"15min":0.86399999999999999},"min":{"1min":0.56499999999999995,"5min":0.371,"15min":0.371},"max":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.78500000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.76700000000000002,"15min":0.88400000000000001},"min":{"1min":0.55100000000000005,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":5.8890000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.80400000000000005,"15min":0.82299999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.44700000000000001},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":10.664},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.79500000000000004,"15min":0.89900000000000002},"min":{"1min":0.57199999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.81499999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.80800000000000005,"15min":0.90900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.86899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91500,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280340,"statfs":{"total":96636764160,"available":96543068160,"internally_reserved":0,"allocated":2260992,"data_stored":994296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42499999999999999,"15min":0.47199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56000000000000005,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.48599999999999999,"15min":0.51600000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70199999999999996,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.69499999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.45900000000000002},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":2.3119999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.66100000000000003},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.63700000000000001,"5min":0.76100000000000001,"15min":16.382999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.81299999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.748,"15min":0.78800000000000003},"min":{"1min":0.52600000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93400000000000005,"5min":1.159,"15min":6.1630000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91800000000000004,"5min":1.149,"15min":1.78},"last":1.23}]},{"osd":5,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.752,"15min":0.76500000000000001},"min":{"1min":0.57799999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88700000000000001,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":1.159},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.74399999999999999,"15min":0.78300000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.96899999999999997,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":1.175}]},{"osd":6,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.78300000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.96699999999999997,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":1.093},{"interface":"front","average":{"1min":0.752,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.47399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94799999999999995,"5min":1.204,"15min":6.1970000000000001},"last":1.2150000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76700000000000002,"15min":0.79500000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95699999999999996,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":0.94099999999999995,"5min":1.1679999999999999,"15min":1.6359999999999999},"last":1.2609999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575005,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69932,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301908,"statfs":{"total":96636764160,"available":96565153792,"internally_reserved":0,"allocated":2842624,"data_stored":1582103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.45100000000000001,"15min":0.54000000000000004},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.84399999999999997,"5min":1.986,"15min":5.5259999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.52500000000000002,"15min":0.57399999999999995},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":2.371,"5min":2.738,"15min":5.4889999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":3.605,"15min":6.0430000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.499}]},{"osd":3,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.878,"5min":0.98999999999999999,"15min":4.7549999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56699999999999995,"15min":0.63400000000000001},"min":{"1min":0.38,"5min":0.375,"15min":0.34999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":4.734},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.745,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93799999999999994,"5min":1.3109999999999999,"15min":4.8079999999999998},"last":1.0009999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76000000000000001,"15min":0.82999999999999996},"min":{"1min":0.54200000000000004,"5min":0.47799999999999998,"15min":0.44500000000000001},"max":{"1min":0.96099999999999997,"5min":1.2190000000000001,"15min":4.944},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.78400000000000003,"15min":0.83299999999999996},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.0820000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.89100000000000001},"min":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.161,"15min":4.8470000000000004},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.82799999999999996,"15min":0.85499999999999998},"min":{"1min":0.497,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.97299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.88200000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.47399999999999998},"max":{"1min":3.9209999999999998,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":1.02},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80400000000000005,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.081,"5min":1.3149999999999999,"15min":4.9950000000000001},"last":0.91200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:20:43.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:20:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.017+0000 7f6e42c00640 1 Processor -- start 2024-09-13T17:20:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.017+0000 7f6e42c00640 1 -- start start 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e42c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6e3c06f970 0x7f6e3c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e42c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06c2d0 0x7f6e3c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e42c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e3c06cc70 con 0x7f6e3c1592a0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e42c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e3c15b0b0 con 0x7f6e3c06f970 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e42c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c159d90 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e3be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6e3c1592a0 0x7f6e3c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44528/0 (socket says 172.21.15.28:44528) 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e3be00640 1 -- 172.21.15.28:0/44362605 learned_addr learned my addr 172.21.15.28:0/44362605 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.018+0000 7f6e33400640 1 --2- 172.21.15.28:0/44362605 >> v2:172.21.15.28:3300/0 conn(0x7f6e3c06f970 0x7f6e3c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e40c00640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06c2d0 0x7f6e3c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e3b400640 1 -- 172.21.15.28:0/44362605 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2030170596 0 0) 0x7f6e3c06cc70 con 0x7f6e3c1592a0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e3b400640 1 -- 172.21.15.28:0/44362605 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e20005730 con 0x7f6e3c1592a0 2024-09-13T17:20:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e33400640 1 -- 172.21.15.28:0/44362605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06c2d0 msgr2=0x7f6e3c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e33400640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06c2d0 0x7f6e3c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e33400640 1 -- 172.21.15.28:0/44362605 >> v1:172.21.15.28:6789/0 conn(0x7f6e3c1592a0 legacy=0x7f6e3c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.019+0000 7f6e33400640 1 -- 172.21.15.28:0/44362605 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c0b2040 con 0x7f6e3c06f970 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.020+0000 7f6e33400640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06f970 0x7f6e3c15ab70 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f6e240097a0 tx=0x7f6e2402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b0b8f85fbe603f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.020+0000 7f6e3b400640 1 -- 172.21.15.28:0/44362605 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24032440 con 0x7f6e3c06f970 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.020+0000 7f6e3b400640 1 -- 172.21.15.28:0/44362605 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e240325a0 con 0x7f6e3c06f970 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.020+0000 7f6e3b400640 1 -- 172.21.15.28:0/44362605 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e24032910 con 0x7f6e3c06f970 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 -- 172.21.15.28:0/44362605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06f970 msgr2=0x7f6e3c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06f970 0x7f6e3c15ab70 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f6e240097a0 tx=0x7f6e2402a180 comp rx=0 tx=0).stop 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 -- 172.21.15.28:0/44362605 shutdown_connections 2024-09-13T17:20:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06c2d0 0x7f6e3c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 --2- 172.21.15.28:0/44362605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06f970 0x7f6e3c15ab70 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 -- 172.21.15.28:0/44362605 >> 172.21.15.28:0/44362605 conn(0x7f6e3c069e80 msgr2=0x7f6e3c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 -- 172.21.15.28:0/44362605 shutdown_connections 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.021+0000 7f6e42c00640 1 -- 172.21.15.28:0/44362605 wait complete. 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.023+0000 7f6e42c00640 1 Processor -- start 2024-09-13T17:20:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.023+0000 7f6e42c00640 1 -- start start 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06f970 0x7f6e3c153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 0x7f6e3c1ecd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c145b20 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c144f10 con 0x7f6e3c1592a0 2024-09-13T17:20:44.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e42c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c1451f0 con 0x7f6e3c06f970 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57134/0 (socket says 172.21.15.28:57134) 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 -- 172.21.15.28:0/1334804553 learned_addr learned my addr 172.21.15.28:0/1334804553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e40c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 0x7f6e3c1ecd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e33400640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06f970 0x7f6e3c153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 -- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06f970 msgr2=0x7f6e3c153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06f970 0x7f6e3c153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 -- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 msgr2=0x7f6e3c1ecd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 0x7f6e3c1ecd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e3be00640 1 -- 172.21.15.28:0/1334804553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24009400 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.024+0000 7f6e40c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 0x7f6e3c1ecd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e3be00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0010e0 tx=0x7f6e2c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c005ce0 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e2c005e60 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e2c0072f0 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3c1454d0 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.025+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3c144ba0 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.026+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3c0b3740 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.027+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6e2c007450 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.028+0000 7f6e38c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 0x7f6e1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.028+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e2c088950 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.028+0000 7f6e33400640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 0x7f6e1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.029+0000 7f6e33400640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 0x7f6e1806d0c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f6e24034000 tx=0x7f6e2402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.032+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6e2c053b20 con 0x7f6e3c06c2d0 2024-09-13T17:20:44.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.207+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6e3c15a5b0 con 0x7f6e1806ac00 2024-09-13T17:20:44.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.215+0000 7f6e38c00640 1 -- 172.21.15.28:0/1334804553 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289146 (secure 0 0 0) 0x7f6e24043090 con 0x7f6e1806ac00 2024-09-13T17:20:44.220 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:44.222 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:20:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 msgr2=0x7f6e1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 0x7f6e1806d0c0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f6e24034000 tx=0x7f6e2402e000 comp rx=0 tx=0).stop 2024-09-13T17:20:44.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 msgr2=0x7f6e3c152b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c0010e0 tx=0x7f6e2c00c890 comp rx=0 tx=0).stop 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 shutdown_connections 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6e1806ac00 0x7f6e1806d0c0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e3c1592a0 0x7f6e3c1ecd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6e3c06f970 0x7f6e3c153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 --2- 172.21.15.28:0/1334804553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e3c06c2d0 0x7f6e3c152b10 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 >> 172.21.15.28:0/1334804553 conn(0x7f6e3c069e80 msgr2=0x7f6e3c15bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 shutdown_connections 2024-09-13T17:20:44.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.219+0000 7f6e42c00640 1 -- 172.21.15.28:0/1334804553 wait complete. 2024-09-13T17:20:44.238 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":555,"stamp":"2024-09-13T17:20:43.075077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":520116,"kb_used_data":18784,"kb_used_omap":12,"kb_used_meta":501171,"kb_avail":754454604,"statfs":{"total":773094113280,"available":772561514496,"internally_reserved":0,"allocated":19234816,"data_stored":9125278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":513199673},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.896552"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144039+0000","last_change":"2024-09-13T17:20:39.639513+0000","last_active":"2024-09-13T17:20:41.144039+0000","last_peered":"2024-09-13T17:20:41.144039+0000","last_clean":"2024-09-13T17:20:41.144039+0000","last_became_active":"2024-09-13T17:20:39.639243+0000","last_became_peered":"2024-09-13T17:20:39.639243+0000","last_unstale":"2024-09-13T17:20:41.144039+0000","last_undegraded":"2024-09-13T17:20:41.144039+0000","last_fullsized":"2024-09-13T17:20:41.144039+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:02.805053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620671+0000","last_change":"2024-09-13T17:20:39.644427+0000","last_active":"2024-09-13T17:20:40.620671+0000","last_peered":"2024-09-13T17:20:40.620671+0000","last_clean":"2024-09-13T17:20:40.620671+0000","last_became_active":"2024-09-13T17:20:39.644157+0000","last_became_peered":"2024-09-13T17:20:39.644157+0000","last_unstale":"2024-09-13T17:20:40.620671+0000","last_undegraded":"2024-09-13T17:20:40.620671+0000","last_fullsized":"2024-09-13T17:20:40.620671+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984986+0000","last_change":"2024-09-13T17:20:39.627269+0000","last_active":"2024-09-13T17:20:40.984986+0000","last_peered":"2024-09-13T17:20:40.984986+0000","last_clean":"2024-09-13T17:20:40.984986+0000","last_became_active":"2024-09-13T17:20:39.627012+0000","last_became_peered":"2024-09-13T17:20:39.627012+0000","last_unstale":"2024-09-13T17:20:40.984986+0000","last_undegraded":"2024-09-13T17:20:40.984986+0000","last_fullsized":"2024-09-13T17:20:40.984986+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620600+0000","last_change":"2024-09-13T17:20:39.637469+0000","last_active":"2024-09-13T17:20:40.620600+0000","last_peered":"2024-09-13T17:20:40.620600+0000","last_clean":"2024-09-13T17:20:40.620600+0000","last_became_active":"2024-09-13T17:20:39.637144+0000","last_became_peered":"2024-09-13T17:20:39.637144+0000","last_unstale":"2024-09-13T17:20:40.620600+0000","last_undegraded":"2024-09-13T17:20:40.620600+0000","last_fullsized":"2024-09-13T17:20:40.620600+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:00.369481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144258+0000","last_change":"2024-09-13T17:20:39.637845+0000","last_active":"2024-09-13T17:20:41.144258+0000","last_peered":"2024-09-13T17:20:41.144258+0000","last_clean":"2024-09-13T17:20:41.144258+0000","last_became_active":"2024-09-13T17:20:39.637143+0000","last_became_peered":"2024-09-13T17:20:39.637143+0000","last_unstale":"2024-09-13T17:20:41.144258+0000","last_undegraded":"2024-09-13T17:20:41.144258+0000","last_fullsized":"2024-09-13T17:20:41.144258+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:55:05.927764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984963+0000","last_change":"2024-09-13T17:20:39.627407+0000","last_active":"2024-09-13T17:20:40.984963+0000","last_peered":"2024-09-13T17:20:40.984963+0000","last_clean":"2024-09-13T17:20:40.984963+0000","last_became_active":"2024-09-13T17:20:39.627160+0000","last_became_peered":"2024-09-13T17:20:39.627160+0000","last_unstale":"2024-09-13T17:20:40.984963+0000","last_undegraded":"2024-09-13T17:20:40.984963+0000","last_fullsized":"2024-09-13T17:20:40.984963+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:34:00.369481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620271+0000","last_change":"2024-09-13T17:20:39.649468+0000","last_active":"2024-09-13T17:20:40.620271+0000","last_peered":"2024-09-13T17:20:40.620271+0000","last_clean":"2024-09-13T17:20:40.620271+0000","last_became_active":"2024-09-13T17:20:39.649168+0000","last_became_peered":"2024-09-13T17:20:39.649168+0000","last_unstale":"2024-09-13T17:20:40.620271+0000","last_undegraded":"2024-09-13T17:20:40.620271+0000","last_fullsized":"2024-09-13T17:20:40.620271+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:28:37.235135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143444+0000","last_change":"2024-09-13T17:20:39.629646+0000","last_active":"2024-09-13T17:20:41.143444+0000","last_peered":"2024-09-13T17:20:41.143444+0000","last_clean":"2024-09-13T17:20:41.143444+0000","last_became_active":"2024-09-13T17:20:39.628797+0000","last_became_peered":"2024-09-13T17:20:39.628797+0000","last_unstale":"2024-09-13T17:20:41.143444+0000","last_undegraded":"2024-09-13T17:20:41.143444+0000","last_fullsized":"2024-09-13T17:20:41.143444+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:26.903757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678457+0000","last_change":"2024-09-13T17:20:39.638900+0000","last_active":"2024-09-13T17:20:40.678457+0000","last_peered":"2024-09-13T17:20:40.678457+0000","last_clean":"2024-09-13T17:20:40.678457+0000","last_became_active":"2024-09-13T17:20:39.638635+0000","last_became_peered":"2024-09-13T17:20:39.638635+0000","last_unstale":"2024-09-13T17:20:40.678457+0000","last_undegraded":"2024-09-13T17:20:40.678457+0000","last_fullsized":"2024-09-13T17:20:40.678457+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:48.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618431+0000","last_change":"2024-09-13T17:20:39.643856+0000","last_active":"2024-09-13T17:20:40.618431+0000","last_peered":"2024-09-13T17:20:40.618431+0000","last_clean":"2024-09-13T17:20:40.618431+0000","last_became_active":"2024-09-13T17:20:39.641868+0000","last_became_peered":"2024-09-13T17:20:39.641868+0000","last_unstale":"2024-09-13T17:20:40.618431+0000","last_undegraded":"2024-09-13T17:20:40.618431+0000","last_fullsized":"2024-09-13T17:20:40.618431+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:25.682499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620646+0000","last_change":"2024-09-13T17:20:39.629329+0000","last_active":"2024-09-13T17:20:40.620646+0000","last_peered":"2024-09-13T17:20:40.620646+0000","last_clean":"2024-09-13T17:20:40.620646+0000","last_became_active":"2024-09-13T17:20:39.628159+0000","last_became_peered":"2024-09-13T17:20:39.628159+0000","last_unstale":"2024-09-13T17:20:40.620646+0000","last_undegraded":"2024-09-13T17:20:40.620646+0000","last_fullsized":"2024-09-13T17:20:40.620646+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:35.251677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620568+0000","last_change":"2024-09-13T17:20:39.638963+0000","last_active":"2024-09-13T17:20:40.620568+0000","last_peered":"2024-09-13T17:20:40.620568+0000","last_clean":"2024-09-13T17:20:40.620568+0000","last_became_active":"2024-09-13T17:20:39.637723+0000","last_became_peered":"2024-09-13T17:20:39.637723+0000","last_unstale":"2024-09-13T17:20:40.620568+0000","last_undegraded":"2024-09-13T17:20:40.620568+0000","last_fullsized":"2024-09-13T17:20:40.620568+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:37.192146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619696+0000","last_change":"2024-09-13T17:20:39.638804+0000","last_active":"2024-09-13T17:20:40.619696+0000","last_peered":"2024-09-13T17:20:40.619696+0000","last_clean":"2024-09-13T17:20:40.619696+0000","last_became_active":"2024-09-13T17:20:39.636208+0000","last_became_peered":"2024-09-13T17:20:39.636208+0000","last_unstale":"2024-09-13T17:20:40.619696+0000","last_undegraded":"2024-09-13T17:20:40.619696+0000","last_fullsized":"2024-09-13T17:20:40.619696+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678161+0000","last_change":"2024-09-13T17:20:39.641632+0000","last_active":"2024-09-13T17:20:40.678161+0000","last_peered":"2024-09-13T17:20:40.678161+0000","last_clean":"2024-09-13T17:20:40.678161+0000","last_became_active":"2024-09-13T17:20:39.641389+0000","last_became_peered":"2024-09-13T17:20:39.641389+0000","last_unstale":"2024-09-13T17:20:40.678161+0000","last_undegraded":"2024-09-13T17:20:40.678161+0000","last_fullsized":"2024-09-13T17:20:40.678161+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620446+0000","last_change":"2024-09-13T17:20:39.641174+0000","last_active":"2024-09-13T17:20:40.620446+0000","last_peered":"2024-09-13T17:20:40.620446+0000","last_clean":"2024-09-13T17:20:40.620446+0000","last_became_active":"2024-09-13T17:20:39.640504+0000","last_became_peered":"2024-09-13T17:20:39.640504+0000","last_unstale":"2024-09-13T17:20:40.620446+0000","last_undegraded":"2024-09-13T17:20:40.620446+0000","last_fullsized":"2024-09-13T17:20:40.620446+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143680+0000","last_change":"2024-09-13T17:20:39.632785+0000","last_active":"2024-09-13T17:20:41.143680+0000","last_peered":"2024-09-13T17:20:41.143680+0000","last_clean":"2024-09-13T17:20:41.143680+0000","last_became_active":"2024-09-13T17:20:39.632510+0000","last_became_peered":"2024-09-13T17:20:39.632510+0000","last_unstale":"2024-09-13T17:20:41.143680+0000","last_undegraded":"2024-09-13T17:20:41.143680+0000","last_fullsized":"2024-09-13T17:20:41.143680+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:57.973890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144375+0000","last_change":"2024-09-13T17:20:39.645897+0000","last_active":"2024-09-13T17:20:41.144375+0000","last_peered":"2024-09-13T17:20:41.144375+0000","last_clean":"2024-09-13T17:20:41.144375+0000","last_became_active":"2024-09-13T17:20:39.643353+0000","last_became_peered":"2024-09-13T17:20:39.643353+0000","last_unstale":"2024-09-13T17:20:41.144375+0000","last_undegraded":"2024-09-13T17:20:41.144375+0000","last_fullsized":"2024-09-13T17:20:41.144375+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:38:04.265039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143956+0000","last_change":"2024-09-13T17:20:39.634096+0000","last_active":"2024-09-13T17:20:41.143956+0000","last_peered":"2024-09-13T17:20:41.143956+0000","last_clean":"2024-09-13T17:20:41.143956+0000","last_became_active":"2024-09-13T17:20:39.633806+0000","last_became_peered":"2024-09-13T17:20:39.633806+0000","last_unstale":"2024-09-13T17:20:41.143956+0000","last_undegraded":"2024-09-13T17:20:41.143956+0000","last_fullsized":"2024-09-13T17:20:41.143956+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:24.516630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618278+0000","last_change":"2024-09-13T17:20:39.634066+0000","last_active":"2024-09-13T17:20:40.618278+0000","last_peered":"2024-09-13T17:20:40.618278+0000","last_clean":"2024-09-13T17:20:40.618278+0000","last_became_active":"2024-09-13T17:20:39.633182+0000","last_became_peered":"2024-09-13T17:20:39.633182+0000","last_unstale":"2024-09-13T17:20:40.618278+0000","last_undegraded":"2024-09-13T17:20:40.618278+0000","last_fullsized":"2024-09-13T17:20:40.618278+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:42:26.903757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144120+0000","last_change":"2024-09-13T17:20:39.646961+0000","last_active":"2024-09-13T17:20:41.144120+0000","last_peered":"2024-09-13T17:20:41.144120+0000","last_clean":"2024-09-13T17:20:41.144120+0000","last_became_active":"2024-09-13T17:20:39.645949+0000","last_became_peered":"2024-09-13T17:20:39.645949+0000","last_unstale":"2024-09-13T17:20:41.144120+0000","last_undegraded":"2024-09-13T17:20:41.144120+0000","last_fullsized":"2024-09-13T17:20:41.144120+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:37.428112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144178+0000","last_change":"2024-09-13T17:20:39.637383+0000","last_active":"2024-09-13T17:20:41.144178+0000","last_peered":"2024-09-13T17:20:41.144178+0000","last_clean":"2024-09-13T17:20:41.144178+0000","last_became_active":"2024-09-13T17:20:39.636741+0000","last_became_peered":"2024-09-13T17:20:39.636741+0000","last_unstale":"2024-09-13T17:20:41.144178+0000","last_undegraded":"2024-09-13T17:20:41.144178+0000","last_fullsized":"2024-09-13T17:20:41.144178+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:16.534672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618298+0000","last_change":"2024-09-13T17:20:39.643510+0000","last_active":"2024-09-13T17:20:40.618298+0000","last_peered":"2024-09-13T17:20:40.618298+0000","last_clean":"2024-09-13T17:20:40.618298+0000","last_became_active":"2024-09-13T17:20:39.643224+0000","last_became_peered":"2024-09-13T17:20:39.643224+0000","last_unstale":"2024-09-13T17:20:40.618298+0000","last_undegraded":"2024-09-13T17:20:40.618298+0000","last_fullsized":"2024-09-13T17:20:40.618298+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:42.523979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984785+0000","last_change":"2024-09-13T17:20:39.649180+0000","last_active":"2024-09-13T17:20:40.984785+0000","last_peered":"2024-09-13T17:20:40.984785+0000","last_clean":"2024-09-13T17:20:40.984785+0000","last_became_active":"2024-09-13T17:20:39.648919+0000","last_became_peered":"2024-09-13T17:20:39.648919+0000","last_unstale":"2024-09-13T17:20:40.984785+0000","last_undegraded":"2024-09-13T17:20:40.984785+0000","last_fullsized":"2024-09-13T17:20:40.984785+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143783+0000","last_change":"2024-09-13T17:20:39.632244+0000","last_active":"2024-09-13T17:20:41.143783+0000","last_peered":"2024-09-13T17:20:41.143783+0000","last_clean":"2024-09-13T17:20:41.143783+0000","last_became_active":"2024-09-13T17:20:39.631383+0000","last_became_peered":"2024-09-13T17:20:39.631383+0000","last_unstale":"2024-09-13T17:20:41.143783+0000","last_undegraded":"2024-09-13T17:20:41.143783+0000","last_fullsized":"2024-09-13T17:20:41.143783+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:52:56.936624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620674+0000","last_change":"2024-09-13T17:20:39.641267+0000","last_active":"2024-09-13T17:20:40.620674+0000","last_peered":"2024-09-13T17:20:40.620674+0000","last_clean":"2024-09-13T17:20:40.620674+0000","last_became_active":"2024-09-13T17:20:39.639766+0000","last_became_peered":"2024-09-13T17:20:39.639766+0000","last_unstale":"2024-09-13T17:20:40.620674+0000","last_undegraded":"2024-09-13T17:20:40.620674+0000","last_fullsized":"2024-09-13T17:20:40.620674+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:25.220159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620102+0000","last_change":"2024-09-13T17:20:39.639141+0000","last_active":"2024-09-13T17:20:40.620102+0000","last_peered":"2024-09-13T17:20:40.620102+0000","last_clean":"2024-09-13T17:20:40.620102+0000","last_became_active":"2024-09-13T17:20:39.638838+0000","last_became_peered":"2024-09-13T17:20:39.638838+0000","last_unstale":"2024-09-13T17:20:40.620102+0000","last_undegraded":"2024-09-13T17:20:40.620102+0000","last_fullsized":"2024-09-13T17:20:40.620102+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:52.554603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985000+0000","last_change":"2024-09-13T17:20:39.626846+0000","last_active":"2024-09-13T17:20:40.985000+0000","last_peered":"2024-09-13T17:20:40.985000+0000","last_clean":"2024-09-13T17:20:40.985000+0000","last_became_active":"2024-09-13T17:20:39.626502+0000","last_became_peered":"2024-09-13T17:20:39.626502+0000","last_unstale":"2024-09-13T17:20:40.985000+0000","last_undegraded":"2024-09-13T17:20:40.985000+0000","last_fullsized":"2024-09-13T17:20:40.985000+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144177+0000","last_change":"2024-09-13T17:20:39.641571+0000","last_active":"2024-09-13T17:20:41.144177+0000","last_peered":"2024-09-13T17:20:41.144177+0000","last_clean":"2024-09-13T17:20:41.144177+0000","last_became_active":"2024-09-13T17:20:39.641313+0000","last_became_peered":"2024-09-13T17:20:39.641313+0000","last_unstale":"2024-09-13T17:20:41.144177+0000","last_undegraded":"2024-09-13T17:20:41.144177+0000","last_fullsized":"2024-09-13T17:20:41.144177+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:15.246356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620568+0000","last_change":"2024-09-13T17:20:39.639587+0000","last_active":"2024-09-13T17:20:40.620568+0000","last_peered":"2024-09-13T17:20:40.620568+0000","last_clean":"2024-09-13T17:20:40.620568+0000","last_became_active":"2024-09-13T17:20:39.639337+0000","last_became_peered":"2024-09-13T17:20:39.639337+0000","last_unstale":"2024-09-13T17:20:40.620568+0000","last_undegraded":"2024-09-13T17:20:40.620568+0000","last_fullsized":"2024-09-13T17:20:40.620568+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:31.957995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678098+0000","last_change":"2024-09-13T17:20:39.624500+0000","last_active":"2024-09-13T17:20:40.678098+0000","last_peered":"2024-09-13T17:20:40.678098+0000","last_clean":"2024-09-13T17:20:40.678098+0000","last_became_active":"2024-09-13T17:20:39.624188+0000","last_became_peered":"2024-09-13T17:20:39.624188+0000","last_unstale":"2024-09-13T17:20:40.678098+0000","last_undegraded":"2024-09-13T17:20:40.678098+0000","last_fullsized":"2024-09-13T17:20:40.678098+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984817+0000","last_change":"2024-09-13T17:20:39.635575+0000","last_active":"2024-09-13T17:20:40.984817+0000","last_peered":"2024-09-13T17:20:40.984817+0000","last_clean":"2024-09-13T17:20:40.984817+0000","last_became_active":"2024-09-13T17:20:39.635246+0000","last_became_peered":"2024-09-13T17:20:39.635246+0000","last_unstale":"2024-09-13T17:20:40.984817+0000","last_undegraded":"2024-09-13T17:20:40.984817+0000","last_fullsized":"2024-09-13T17:20:40.984817+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:49.678616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143914+0000","last_change":"2024-09-13T17:20:39.630955+0000","last_active":"2024-09-13T17:20:41.143914+0000","last_peered":"2024-09-13T17:20:41.143914+0000","last_clean":"2024-09-13T17:20:41.143914+0000","last_became_active":"2024-09-13T17:20:39.630697+0000","last_became_peered":"2024-09-13T17:20:39.630697+0000","last_unstale":"2024-09-13T17:20:41.143914+0000","last_undegraded":"2024-09-13T17:20:41.143914+0000","last_fullsized":"2024-09-13T17:20:41.143914+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:00.743598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143803+0000","last_change":"2024-09-13T17:20:39.631816+0000","last_active":"2024-09-13T17:20:41.143803+0000","last_peered":"2024-09-13T17:20:41.143803+0000","last_clean":"2024-09-13T17:20:41.143803+0000","last_became_active":"2024-09-13T17:20:39.631588+0000","last_became_peered":"2024-09-13T17:20:39.631588+0000","last_unstale":"2024-09-13T17:20:41.143803+0000","last_undegraded":"2024-09-13T17:20:41.143803+0000","last_fullsized":"2024-09-13T17:20:41.143803+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:52.109474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"178'4","reported_seq":15,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144475+0000","last_change":"2024-09-13T17:20:39.198543+0000","last_active":"2024-09-13T17:20:41.144475+0000","last_peered":"2024-09-13T17:20:41.144475+0000","last_clean":"2024-09-13T17:20:41.144475+0000","last_became_active":"2024-09-13T17:20:39.198183+0000","last_became_peered":"2024-09-13T17:20:39.198183+0000","last_unstale":"2024-09-13T17:20:41.144475+0000","last_undegraded":"2024-09-13T17:20:41.144475+0000","last_fullsized":"2024-09-13T17:20:41.144475+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:23:09.570377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618473+0000","last_change":"2024-09-13T17:20:39.643611+0000","last_active":"2024-09-13T17:20:40.618473+0000","last_peered":"2024-09-13T17:20:40.618473+0000","last_clean":"2024-09-13T17:20:40.618473+0000","last_became_active":"2024-09-13T17:20:39.641581+0000","last_became_peered":"2024-09-13T17:20:39.641581+0000","last_unstale":"2024-09-13T17:20:40.618473+0000","last_undegraded":"2024-09-13T17:20:40.618473+0000","last_fullsized":"2024-09-13T17:20:40.618473+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:35.572828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678375+0000","last_change":"2024-09-13T17:20:39.190999+0000","last_active":"2024-09-13T17:20:40.678375+0000","last_peered":"2024-09-13T17:20:40.678375+0000","last_clean":"2024-09-13T17:20:40.678375+0000","last_became_active":"2024-09-13T17:20:39.190656+0000","last_became_peered":"2024-09-13T17:20:39.190656+0000","last_unstale":"2024-09-13T17:20:40.678375+0000","last_undegraded":"2024-09-13T17:20:40.678375+0000","last_fullsized":"2024-09-13T17:20:40.678375+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:13.591611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619833+0000","last_change":"2024-09-13T17:20:39.636918+0000","last_active":"2024-09-13T17:20:40.619833+0000","last_peered":"2024-09-13T17:20:40.619833+0000","last_clean":"2024-09-13T17:20:40.619833+0000","last_became_active":"2024-09-13T17:20:39.636647+0000","last_became_peered":"2024-09-13T17:20:39.636647+0000","last_unstale":"2024-09-13T17:20:40.619833+0000","last_undegraded":"2024-09-13T17:20:40.619833+0000","last_fullsized":"2024-09-13T17:20:40.619833+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:44.679816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985078+0000","last_change":"2024-09-13T17:20:39.195861+0000","last_active":"2024-09-13T17:20:40.985078+0000","last_peered":"2024-09-13T17:20:40.985078+0000","last_clean":"2024-09-13T17:20:40.985078+0000","last_became_active":"2024-09-13T17:20:39.195603+0000","last_became_peered":"2024-09-13T17:20:39.195603+0000","last_unstale":"2024-09-13T17:20:40.985078+0000","last_undegraded":"2024-09-13T17:20:40.985078+0000","last_fullsized":"2024-09-13T17:20:40.985078+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:20.042947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678349+0000","last_change":"2024-09-13T17:20:39.626164+0000","last_active":"2024-09-13T17:20:40.678349+0000","last_peered":"2024-09-13T17:20:40.678349+0000","last_clean":"2024-09-13T17:20:40.678349+0000","last_became_active":"2024-09-13T17:20:39.625823+0000","last_became_peered":"2024-09-13T17:20:39.625823+0000","last_unstale":"2024-09-13T17:20:40.678349+0000","last_undegraded":"2024-09-13T17:20:40.678349+0000","last_fullsized":"2024-09-13T17:20:40.678349+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:04.193554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620686+0000","last_change":"2024-09-13T17:20:39.199632+0000","last_active":"2024-09-13T17:20:40.620686+0000","last_peered":"2024-09-13T17:20:40.620686+0000","last_clean":"2024-09-13T17:20:40.620686+0000","last_became_active":"2024-09-13T17:20:39.199352+0000","last_became_peered":"2024-09-13T17:20:39.199352+0000","last_unstale":"2024-09-13T17:20:40.620686+0000","last_undegraded":"2024-09-13T17:20:40.620686+0000","last_fullsized":"2024-09-13T17:20:40.620686+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:58.689861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618407+0000","last_change":"2024-09-13T17:20:39.648663+0000","last_active":"2024-09-13T17:20:40.618407+0000","last_peered":"2024-09-13T17:20:40.618407+0000","last_clean":"2024-09-13T17:20:40.618407+0000","last_became_active":"2024-09-13T17:20:39.646946+0000","last_became_peered":"2024-09-13T17:20:39.646946+0000","last_unstale":"2024-09-13T17:20:40.618407+0000","last_undegraded":"2024-09-13T17:20:40.618407+0000","last_fullsized":"2024-09-13T17:20:40.618407+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:25.412433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144268+0000","last_change":"2024-09-13T17:20:39.197366+0000","last_active":"2024-09-13T17:20:41.144268+0000","last_peered":"2024-09-13T17:20:41.144268+0000","last_clean":"2024-09-13T17:20:41.144268+0000","last_became_active":"2024-09-13T17:20:39.197112+0000","last_became_peered":"2024-09-13T17:20:39.197112+0000","last_unstale":"2024-09-13T17:20:41.144268+0000","last_undegraded":"2024-09-13T17:20:41.144268+0000","last_fullsized":"2024-09-13T17:20:41.144268+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:43.295845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143602+0000","last_change":"2024-09-13T17:20:39.632993+0000","last_active":"2024-09-13T17:20:41.143602+0000","last_peered":"2024-09-13T17:20:41.143602+0000","last_clean":"2024-09-13T17:20:41.143602+0000","last_became_active":"2024-09-13T17:20:39.632593+0000","last_became_peered":"2024-09-13T17:20:39.632593+0000","last_unstale":"2024-09-13T17:20:41.143602+0000","last_undegraded":"2024-09-13T17:20:41.143602+0000","last_fullsized":"2024-09-13T17:20:41.143602+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:43.557625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143612+0000","last_change":"2024-09-13T17:20:39.206148+0000","last_active":"2024-09-13T17:20:41.143612+0000","last_peered":"2024-09-13T17:20:41.143612+0000","last_clean":"2024-09-13T17:20:41.143612+0000","last_became_active":"2024-09-13T17:20:39.205888+0000","last_became_peered":"2024-09-13T17:20:39.205888+0000","last_unstale":"2024-09-13T17:20:41.143612+0000","last_undegraded":"2024-09-13T17:20:41.143612+0000","last_fullsized":"2024-09-13T17:20:41.143612+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:24:24.675079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620379+0000","last_change":"2024-09-13T17:20:39.629179+0000","last_active":"2024-09-13T17:20:40.620379+0000","last_peered":"2024-09-13T17:20:40.620379+0000","last_clean":"2024-09-13T17:20:40.620379+0000","last_became_active":"2024-09-13T17:20:39.626826+0000","last_became_peered":"2024-09-13T17:20:39.626826+0000","last_unstale":"2024-09-13T17:20:40.620379+0000","last_undegraded":"2024-09-13T17:20:40.620379+0000","last_fullsized":"2024-09-13T17:20:40.620379+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:41.217125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143590+0000","last_change":"2024-09-13T17:20:39.201413+0000","last_active":"2024-09-13T17:20:41.143590+0000","last_peered":"2024-09-13T17:20:41.143590+0000","last_clean":"2024-09-13T17:20:41.143590+0000","last_became_active":"2024-09-13T17:20:39.200200+0000","last_became_peered":"2024-09-13T17:20:39.200200+0000","last_unstale":"2024-09-13T17:20:41.143590+0000","last_undegraded":"2024-09-13T17:20:41.143590+0000","last_fullsized":"2024-09-13T17:20:41.143590+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:21:47.990493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.985066+0000","last_change":"2024-09-13T17:20:39.625360+0000","last_active":"2024-09-13T17:20:40.985066+0000","last_peered":"2024-09-13T17:20:40.985066+0000","last_clean":"2024-09-13T17:20:40.985066+0000","last_became_active":"2024-09-13T17:20:39.625153+0000","last_became_peered":"2024-09-13T17:20:39.625153+0000","last_unstale":"2024-09-13T17:20:40.985066+0000","last_undegraded":"2024-09-13T17:20:40.985066+0000","last_fullsized":"2024-09-13T17:20:40.985066+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.761569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620298+0000","last_change":"2024-09-13T17:20:39.199376+0000","last_active":"2024-09-13T17:20:40.620298+0000","last_peered":"2024-09-13T17:20:40.620298+0000","last_clean":"2024-09-13T17:20:40.620298+0000","last_became_active":"2024-09-13T17:20:39.198269+0000","last_became_peered":"2024-09-13T17:20:39.198269+0000","last_unstale":"2024-09-13T17:20:40.620298+0000","last_undegraded":"2024-09-13T17:20:40.620298+0000","last_fullsized":"2024-09-13T17:20:40.620298+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:37.898138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143763+0000","last_change":"2024-09-13T17:20:39.640835+0000","last_active":"2024-09-13T17:20:41.143763+0000","last_peered":"2024-09-13T17:20:41.143763+0000","last_clean":"2024-09-13T17:20:41.143763+0000","last_became_active":"2024-09-13T17:20:39.640548+0000","last_became_peered":"2024-09-13T17:20:39.640548+0000","last_unstale":"2024-09-13T17:20:41.143763+0000","last_undegraded":"2024-09-13T17:20:41.143763+0000","last_fullsized":"2024-09-13T17:20:41.143763+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:01.126089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144172+0000","last_change":"2024-09-13T17:20:39.205506+0000","last_active":"2024-09-13T17:20:41.144172+0000","last_peered":"2024-09-13T17:20:41.144172+0000","last_clean":"2024-09-13T17:20:41.144172+0000","last_became_active":"2024-09-13T17:20:39.204481+0000","last_became_peered":"2024-09-13T17:20:39.204481+0000","last_unstale":"2024-09-13T17:20:41.144172+0000","last_undegraded":"2024-09-13T17:20:41.144172+0000","last_fullsized":"2024-09-13T17:20:41.144172+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:57.501218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619833+0000","last_change":"2024-09-13T17:20:39.625709+0000","last_active":"2024-09-13T17:20:40.619833+0000","last_peered":"2024-09-13T17:20:40.619833+0000","last_clean":"2024-09-13T17:20:40.619833+0000","last_became_active":"2024-09-13T17:20:39.625450+0000","last_became_peered":"2024-09-13T17:20:39.625450+0000","last_unstale":"2024-09-13T17:20:40.619833+0000","last_undegraded":"2024-09-13T17:20:40.619833+0000","last_fullsized":"2024-09-13T17:20:40.619833+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:01.811707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"178'3","reported_seq":14,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144192+0000","last_change":"2024-09-13T17:20:39.197453+0000","last_active":"2024-09-13T17:20:41.144192+0000","last_peered":"2024-09-13T17:20:41.144192+0000","last_clean":"2024-09-13T17:20:41.144192+0000","last_became_active":"2024-09-13T17:20:39.197185+0000","last_became_peered":"2024-09-13T17:20:39.197185+0000","last_unstale":"2024-09-13T17:20:41.144192+0000","last_undegraded":"2024-09-13T17:20:41.144192+0000","last_fullsized":"2024-09-13T17:20:41.144192+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:00.761299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618270+0000","last_change":"2024-09-13T17:20:39.642856+0000","last_active":"2024-09-13T17:20:40.618270+0000","last_peered":"2024-09-13T17:20:40.618270+0000","last_clean":"2024-09-13T17:20:40.618270+0000","last_became_active":"2024-09-13T17:20:39.641881+0000","last_became_peered":"2024-09-13T17:20:39.641881+0000","last_unstale":"2024-09-13T17:20:40.618270+0000","last_undegraded":"2024-09-13T17:20:40.618270+0000","last_fullsized":"2024-09-13T17:20:40.618270+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:35.337787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"178'6","reported_seq":17,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984922+0000","last_change":"2024-09-13T17:20:39.199323+0000","last_active":"2024-09-13T17:20:40.984922+0000","last_peered":"2024-09-13T17:20:40.984922+0000","last_clean":"2024-09-13T17:20:40.984922+0000","last_became_active":"2024-09-13T17:20:39.199063+0000","last_became_peered":"2024-09-13T17:20:39.199063+0000","last_unstale":"2024-09-13T17:20:40.984922+0000","last_undegraded":"2024-09-13T17:20:40.984922+0000","last_fullsized":"2024-09-13T17:20:40.984922+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:13.591611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984876+0000","last_change":"2024-09-13T17:20:39.641027+0000","last_active":"2024-09-13T17:20:40.984876+0000","last_peered":"2024-09-13T17:20:40.984876+0000","last_clean":"2024-09-13T17:20:40.984876+0000","last_became_active":"2024-09-13T17:20:39.640780+0000","last_became_peered":"2024-09-13T17:20:39.640780+0000","last_unstale":"2024-09-13T17:20:40.984876+0000","last_undegraded":"2024-09-13T17:20:40.984876+0000","last_fullsized":"2024-09-13T17:20:40.984876+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144032+0000","last_change":"2024-09-13T17:20:39.201708+0000","last_active":"2024-09-13T17:20:41.144032+0000","last_peered":"2024-09-13T17:20:41.144032+0000","last_clean":"2024-09-13T17:20:41.144032+0000","last_became_active":"2024-09-13T17:20:39.201308+0000","last_became_peered":"2024-09-13T17:20:39.201308+0000","last_unstale":"2024-09-13T17:20:41.144032+0000","last_undegraded":"2024-09-13T17:20:41.144032+0000","last_fullsized":"2024-09-13T17:20:41.144032+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:37.898138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620014+0000","last_change":"2024-09-13T17:20:39.642348+0000","last_active":"2024-09-13T17:20:40.620014+0000","last_peered":"2024-09-13T17:20:40.620014+0000","last_clean":"2024-09-13T17:20:40.620014+0000","last_became_active":"2024-09-13T17:20:39.640237+0000","last_became_peered":"2024-09-13T17:20:39.640237+0000","last_unstale":"2024-09-13T17:20:40.620014+0000","last_undegraded":"2024-09-13T17:20:40.620014+0000","last_fullsized":"2024-09-13T17:20:40.620014+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:55.829472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618501+0000","last_change":"2024-09-13T17:20:39.207181+0000","last_active":"2024-09-13T17:20:40.618501+0000","last_peered":"2024-09-13T17:20:40.618501+0000","last_clean":"2024-09-13T17:20:40.618501+0000","last_became_active":"2024-09-13T17:20:39.206785+0000","last_became_peered":"2024-09-13T17:20:39.206785+0000","last_unstale":"2024-09-13T17:20:40.618501+0000","last_undegraded":"2024-09-13T17:20:40.618501+0000","last_fullsized":"2024-09-13T17:20:40.618501+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:27.598580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.677983+0000","last_change":"2024-09-13T17:20:39.640381+0000","last_active":"2024-09-13T17:20:40.677983+0000","last_peered":"2024-09-13T17:20:40.677983+0000","last_clean":"2024-09-13T17:20:40.677983+0000","last_became_active":"2024-09-13T17:20:39.640051+0000","last_became_peered":"2024-09-13T17:20:39.640051+0000","last_unstale":"2024-09-13T17:20:40.677983+0000","last_undegraded":"2024-09-13T17:20:40.677983+0000","last_fullsized":"2024-09-13T17:20:40.677983+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:42.916118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143404+0000","last_change":"2024-09-13T17:20:39.204522+0000","last_active":"2024-09-13T17:20:41.143404+0000","last_peered":"2024-09-13T17:20:41.143404+0000","last_clean":"2024-09-13T17:20:41.143404+0000","last_became_active":"2024-09-13T17:20:39.204217+0000","last_became_peered":"2024-09-13T17:20:39.204217+0000","last_unstale":"2024-09-13T17:20:41.143404+0000","last_undegraded":"2024-09-13T17:20:41.143404+0000","last_fullsized":"2024-09-13T17:20:41.143404+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:24.405013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618079+0000","last_change":"2024-09-13T17:20:39.643620+0000","last_active":"2024-09-13T17:20:40.618079+0000","last_peered":"2024-09-13T17:20:40.618079+0000","last_clean":"2024-09-13T17:20:40.618079+0000","last_became_active":"2024-09-13T17:20:39.637801+0000","last_became_peered":"2024-09-13T17:20:39.637801+0000","last_unstale":"2024-09-13T17:20:40.618079+0000","last_undegraded":"2024-09-13T17:20:40.618079+0000","last_fullsized":"2024-09-13T17:20:40.618079+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:23.440511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"178'8","reported_seq":19,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143759+0000","last_change":"2024-09-13T17:20:39.202745+0000","last_active":"2024-09-13T17:20:41.143759+0000","last_peered":"2024-09-13T17:20:41.143759+0000","last_clean":"2024-09-13T17:20:41.143759+0000","last_became_active":"2024-09-13T17:20:39.199696+0000","last_became_peered":"2024-09-13T17:20:39.199696+0000","last_unstale":"2024-09-13T17:20:41.143759+0000","last_undegraded":"2024-09-13T17:20:41.143759+0000","last_fullsized":"2024-09-13T17:20:41.143759+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:24.754149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618741+0000","last_change":"2024-09-13T17:20:39.637107+0000","last_active":"2024-09-13T17:20:40.618741+0000","last_peered":"2024-09-13T17:20:40.618741+0000","last_clean":"2024-09-13T17:20:40.618741+0000","last_became_active":"2024-09-13T17:20:39.636743+0000","last_became_peered":"2024-09-13T17:20:39.636743+0000","last_unstale":"2024-09-13T17:20:40.618741+0000","last_undegraded":"2024-09-13T17:20:40.618741+0000","last_fullsized":"2024-09-13T17:20:40.618741+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:37:50.231822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144476+0000","last_change":"2024-09-13T17:20:39.204155+0000","last_active":"2024-09-13T17:20:41.144476+0000","last_peered":"2024-09-13T17:20:41.144476+0000","last_clean":"2024-09-13T17:20:41.144476+0000","last_became_active":"2024-09-13T17:20:39.203844+0000","last_became_peered":"2024-09-13T17:20:39.203844+0000","last_unstale":"2024-09-13T17:20:41.144476+0000","last_undegraded":"2024-09-13T17:20:41.144476+0000","last_fullsized":"2024-09-13T17:20:41.144476+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:00:52.830750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678466+0000","last_change":"2024-09-13T17:20:39.641302+0000","last_active":"2024-09-13T17:20:40.678466+0000","last_peered":"2024-09-13T17:20:40.678466+0000","last_clean":"2024-09-13T17:20:40.678466+0000","last_became_active":"2024-09-13T17:20:39.641028+0000","last_became_peered":"2024-09-13T17:20:39.641028+0000","last_unstale":"2024-09-13T17:20:40.678466+0000","last_undegraded":"2024-09-13T17:20:40.678466+0000","last_fullsized":"2024-09-13T17:20:40.678466+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:34:25.761569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143793+0000","last_change":"2024-09-13T17:20:39.206500+0000","last_active":"2024-09-13T17:20:41.143793+0000","last_peered":"2024-09-13T17:20:41.143793+0000","last_clean":"2024-09-13T17:20:41.143793+0000","last_became_active":"2024-09-13T17:20:39.204754+0000","last_became_peered":"2024-09-13T17:20:39.204754+0000","last_unstale":"2024-09-13T17:20:41.143793+0000","last_undegraded":"2024-09-13T17:20:41.143793+0000","last_fullsized":"2024-09-13T17:20:41.143793+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:56:41.516559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678209+0000","last_change":"2024-09-13T17:20:39.627103+0000","last_active":"2024-09-13T17:20:40.678209+0000","last_peered":"2024-09-13T17:20:40.678209+0000","last_clean":"2024-09-13T17:20:40.678209+0000","last_became_active":"2024-09-13T17:20:39.626856+0000","last_became_peered":"2024-09-13T17:20:39.626856+0000","last_unstale":"2024-09-13T17:20:40.678209+0000","last_undegraded":"2024-09-13T17:20:40.678209+0000","last_fullsized":"2024-09-13T17:20:40.678209+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"178'1","reported_seq":12,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144004+0000","last_change":"2024-09-13T17:20:39.204614+0000","last_active":"2024-09-13T17:20:41.144004+0000","last_peered":"2024-09-13T17:20:41.144004+0000","last_clean":"2024-09-13T17:20:41.144004+0000","last_became_active":"2024-09-13T17:20:39.204225+0000","last_became_peered":"2024-09-13T17:20:39.204225+0000","last_unstale":"2024-09-13T17:20:41.144004+0000","last_undegraded":"2024-09-13T17:20:41.144004+0000","last_fullsized":"2024-09-13T17:20:41.144004+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:24.741949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620517+0000","last_change":"2024-09-13T17:20:39.642381+0000","last_active":"2024-09-13T17:20:40.620517+0000","last_peered":"2024-09-13T17:20:40.620517+0000","last_clean":"2024-09-13T17:20:40.620517+0000","last_became_active":"2024-09-13T17:20:39.642111+0000","last_became_peered":"2024-09-13T17:20:39.642111+0000","last_unstale":"2024-09-13T17:20:40.620517+0000","last_undegraded":"2024-09-13T17:20:40.620517+0000","last_fullsized":"2024-09-13T17:20:40.620517+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:51.426108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619923+0000","last_change":"2024-09-13T17:20:39.200576+0000","last_active":"2024-09-13T17:20:40.619923+0000","last_peered":"2024-09-13T17:20:40.619923+0000","last_clean":"2024-09-13T17:20:40.619923+0000","last_became_active":"2024-09-13T17:20:39.200219+0000","last_became_peered":"2024-09-13T17:20:39.200219+0000","last_unstale":"2024-09-13T17:20:40.619923+0000","last_undegraded":"2024-09-13T17:20:40.619923+0000","last_fullsized":"2024-09-13T17:20:40.619923+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:24.741949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618445+0000","last_change":"2024-09-13T17:20:39.641324+0000","last_active":"2024-09-13T17:20:40.618445+0000","last_peered":"2024-09-13T17:20:40.618445+0000","last_clean":"2024-09-13T17:20:40.618445+0000","last_became_active":"2024-09-13T17:20:39.640177+0000","last_became_peered":"2024-09-13T17:20:39.640177+0000","last_unstale":"2024-09-13T17:20:40.618445+0000","last_undegraded":"2024-09-13T17:20:40.618445+0000","last_fullsized":"2024-09-13T17:20:40.618445+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:40.520569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984788+0000","last_change":"2024-09-13T17:20:39.206177+0000","last_active":"2024-09-13T17:20:40.984788+0000","last_peered":"2024-09-13T17:20:40.984788+0000","last_clean":"2024-09-13T17:20:40.984788+0000","last_became_active":"2024-09-13T17:20:39.205861+0000","last_became_peered":"2024-09-13T17:20:39.205861+0000","last_unstale":"2024-09-13T17:20:40.984788+0000","last_undegraded":"2024-09-13T17:20:40.984788+0000","last_fullsized":"2024-09-13T17:20:40.984788+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:02.840549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620121+0000","last_change":"2024-09-13T17:20:39.625469+0000","last_active":"2024-09-13T17:20:40.620121+0000","last_peered":"2024-09-13T17:20:40.620121+0000","last_clean":"2024-09-13T17:20:40.620121+0000","last_became_active":"2024-09-13T17:20:39.625159+0000","last_became_peered":"2024-09-13T17:20:39.625159+0000","last_unstale":"2024-09-13T17:20:40.620121+0000","last_undegraded":"2024-09-13T17:20:40.620121+0000","last_fullsized":"2024-09-13T17:20:40.620121+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:28.606000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144067+0000","last_change":"2024-09-13T17:20:39.201866+0000","last_active":"2024-09-13T17:20:41.144067+0000","last_peered":"2024-09-13T17:20:41.144067+0000","last_clean":"2024-09-13T17:20:41.144067+0000","last_became_active":"2024-09-13T17:20:39.201615+0000","last_became_peered":"2024-09-13T17:20:39.201615+0000","last_unstale":"2024-09-13T17:20:41.144067+0000","last_undegraded":"2024-09-13T17:20:41.144067+0000","last_fullsized":"2024-09-13T17:20:41.144067+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:39:48.919267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619546+0000","last_change":"2024-09-13T17:20:39.641296+0000","last_active":"2024-09-13T17:20:40.619546+0000","last_peered":"2024-09-13T17:20:40.619546+0000","last_clean":"2024-09-13T17:20:40.619546+0000","last_became_active":"2024-09-13T17:20:39.640974+0000","last_became_peered":"2024-09-13T17:20:39.640974+0000","last_unstale":"2024-09-13T17:20:40.619546+0000","last_undegraded":"2024-09-13T17:20:40.619546+0000","last_fullsized":"2024-09-13T17:20:40.619546+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:06.952818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984889+0000","last_change":"2024-09-13T17:20:39.207372+0000","last_active":"2024-09-13T17:20:40.984889+0000","last_peered":"2024-09-13T17:20:40.984889+0000","last_clean":"2024-09-13T17:20:40.984889+0000","last_became_active":"2024-09-13T17:20:39.207086+0000","last_became_peered":"2024-09-13T17:20:39.207086+0000","last_unstale":"2024-09-13T17:20:40.984889+0000","last_undegraded":"2024-09-13T17:20:40.984889+0000","last_fullsized":"2024-09-13T17:20:40.984889+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:03.491122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143958+0000","last_change":"2024-09-13T17:20:39.632790+0000","last_active":"2024-09-13T17:20:41.143958+0000","last_peered":"2024-09-13T17:20:41.143958+0000","last_clean":"2024-09-13T17:20:41.143958+0000","last_became_active":"2024-09-13T17:20:39.632550+0000","last_became_peered":"2024-09-13T17:20:39.632550+0000","last_unstale":"2024-09-13T17:20:41.143958+0000","last_undegraded":"2024-09-13T17:20:41.143958+0000","last_fullsized":"2024-09-13T17:20:41.143958+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:10.019707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678075+0000","last_change":"2024-09-13T17:20:39.201922+0000","last_active":"2024-09-13T17:20:40.678075+0000","last_peered":"2024-09-13T17:20:40.678075+0000","last_clean":"2024-09-13T17:20:40.678075+0000","last_became_active":"2024-09-13T17:20:39.201679+0000","last_became_peered":"2024-09-13T17:20:39.201679+0000","last_unstale":"2024-09-13T17:20:40.678075+0000","last_undegraded":"2024-09-13T17:20:40.678075+0000","last_fullsized":"2024-09-13T17:20:40.678075+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:34.705513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144179+0000","last_change":"2024-09-13T17:20:39.632118+0000","last_active":"2024-09-13T17:20:41.144179+0000","last_peered":"2024-09-13T17:20:41.144179+0000","last_clean":"2024-09-13T17:20:41.144179+0000","last_became_active":"2024-09-13T17:20:39.631858+0000","last_became_peered":"2024-09-13T17:20:39.631858+0000","last_unstale":"2024-09-13T17:20:41.144179+0000","last_undegraded":"2024-09-13T17:20:41.144179+0000","last_fullsized":"2024-09-13T17:20:41.144179+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:28:38.561207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"178'2","reported_seq":13,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144047+0000","last_change":"2024-09-13T17:20:39.205299+0000","last_active":"2024-09-13T17:20:41.144047+0000","last_peered":"2024-09-13T17:20:41.144047+0000","last_clean":"2024-09-13T17:20:41.144047+0000","last_became_active":"2024-09-13T17:20:39.204944+0000","last_became_peered":"2024-09-13T17:20:39.204944+0000","last_unstale":"2024-09-13T17:20:41.144047+0000","last_undegraded":"2024-09-13T17:20:41.144047+0000","last_fullsized":"2024-09-13T17:20:41.144047+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:48.671196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618557+0000","last_change":"2024-09-13T17:20:39.639222+0000","last_active":"2024-09-13T17:20:40.618557+0000","last_peered":"2024-09-13T17:20:40.618557+0000","last_clean":"2024-09-13T17:20:40.618557+0000","last_became_active":"2024-09-13T17:20:39.638967+0000","last_became_peered":"2024-09-13T17:20:39.638967+0000","last_unstale":"2024-09-13T17:20:40.618557+0000","last_undegraded":"2024-09-13T17:20:40.618557+0000","last_fullsized":"2024-09-13T17:20:40.618557+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:37.374048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618695+0000","last_change":"2024-09-13T17:20:39.195809+0000","last_active":"2024-09-13T17:20:40.618695+0000","last_peered":"2024-09-13T17:20:40.618695+0000","last_clean":"2024-09-13T17:20:40.618695+0000","last_became_active":"2024-09-13T17:20:39.195286+0000","last_became_peered":"2024-09-13T17:20:39.195286+0000","last_unstale":"2024-09-13T17:20:40.618695+0000","last_undegraded":"2024-09-13T17:20:40.618695+0000","last_fullsized":"2024-09-13T17:20:40.618695+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:34.244257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143641+0000","last_change":"2024-09-13T17:20:39.626675+0000","last_active":"2024-09-13T17:20:41.143641+0000","last_peered":"2024-09-13T17:20:41.143641+0000","last_clean":"2024-09-13T17:20:41.143641+0000","last_became_active":"2024-09-13T17:20:39.625923+0000","last_became_peered":"2024-09-13T17:20:39.625923+0000","last_unstale":"2024-09-13T17:20:41.143641+0000","last_undegraded":"2024-09-13T17:20:41.143641+0000","last_fullsized":"2024-09-13T17:20:41.143641+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:11.695712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618362+0000","last_change":"2024-09-13T17:20:39.200677+0000","last_active":"2024-09-13T17:20:40.618362+0000","last_peered":"2024-09-13T17:20:40.618362+0000","last_clean":"2024-09-13T17:20:40.618362+0000","last_became_active":"2024-09-13T17:20:39.200389+0000","last_became_peered":"2024-09-13T17:20:39.200389+0000","last_unstale":"2024-09-13T17:20:40.618362+0000","last_undegraded":"2024-09-13T17:20:40.618362+0000","last_fullsized":"2024-09-13T17:20:40.618362+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:31:51.547183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143735+0000","last_change":"2024-09-13T17:20:39.632884+0000","last_active":"2024-09-13T17:20:41.143735+0000","last_peered":"2024-09-13T17:20:41.143735+0000","last_clean":"2024-09-13T17:20:41.143735+0000","last_became_active":"2024-09-13T17:20:39.632114+0000","last_became_peered":"2024-09-13T17:20:39.632114+0000","last_unstale":"2024-09-13T17:20:41.143735+0000","last_undegraded":"2024-09-13T17:20:41.143735+0000","last_fullsized":"2024-09-13T17:20:41.143735+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:16:56.033401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143663+0000","last_change":"2024-09-13T17:20:39.200932+0000","last_active":"2024-09-13T17:20:41.143663+0000","last_peered":"2024-09-13T17:20:41.143663+0000","last_clean":"2024-09-13T17:20:41.143663+0000","last_became_active":"2024-09-13T17:20:39.200611+0000","last_became_peered":"2024-09-13T17:20:39.200611+0000","last_unstale":"2024-09-13T17:20:41.143663+0000","last_undegraded":"2024-09-13T17:20:41.143663+0000","last_fullsized":"2024-09-13T17:20:41.143663+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:35.251693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143907+0000","last_change":"2024-09-13T17:20:39.643281+0000","last_active":"2024-09-13T17:20:41.143907+0000","last_peered":"2024-09-13T17:20:41.143907+0000","last_clean":"2024-09-13T17:20:41.143907+0000","last_became_active":"2024-09-13T17:20:39.642969+0000","last_became_peered":"2024-09-13T17:20:39.642969+0000","last_unstale":"2024-09-13T17:20:41.143907+0000","last_undegraded":"2024-09-13T17:20:41.143907+0000","last_fullsized":"2024-09-13T17:20:41.143907+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:30.506534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678225+0000","last_change":"2024-09-13T17:20:39.199454+0000","last_active":"2024-09-13T17:20:40.678225+0000","last_peered":"2024-09-13T17:20:40.678225+0000","last_clean":"2024-09-13T17:20:40.678225+0000","last_became_active":"2024-09-13T17:20:39.198310+0000","last_became_peered":"2024-09-13T17:20:39.198310+0000","last_unstale":"2024-09-13T17:20:40.678225+0000","last_undegraded":"2024-09-13T17:20:40.678225+0000","last_fullsized":"2024-09-13T17:20:40.678225+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:20.042947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143907+0000","last_change":"2024-09-13T17:20:39.631945+0000","last_active":"2024-09-13T17:20:41.143907+0000","last_peered":"2024-09-13T17:20:41.143907+0000","last_clean":"2024-09-13T17:20:41.143907+0000","last_became_active":"2024-09-13T17:20:39.631096+0000","last_became_peered":"2024-09-13T17:20:39.631096+0000","last_unstale":"2024-09-13T17:20:41.143907+0000","last_undegraded":"2024-09-13T17:20:41.143907+0000","last_fullsized":"2024-09-13T17:20:41.143907+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:58:37.139362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.678199+0000","last_change":"2024-09-13T17:20:39.196860+0000","last_active":"2024-09-13T17:20:40.678199+0000","last_peered":"2024-09-13T17:20:40.678199+0000","last_clean":"2024-09-13T17:20:40.678199+0000","last_became_active":"2024-09-13T17:20:39.196597+0000","last_became_peered":"2024-09-13T17:20:39.196597+0000","last_unstale":"2024-09-13T17:20:40.678199+0000","last_undegraded":"2024-09-13T17:20:40.678199+0000","last_fullsized":"2024-09-13T17:20:40.678199+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:03.491122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.619655+0000","last_change":"2024-09-13T17:20:39.635620+0000","last_active":"2024-09-13T17:20:40.619655+0000","last_peered":"2024-09-13T17:20:40.619655+0000","last_clean":"2024-09-13T17:20:40.619655+0000","last_became_active":"2024-09-13T17:20:39.635324+0000","last_became_peered":"2024-09-13T17:20:39.635324+0000","last_unstale":"2024-09-13T17:20:40.619655+0000","last_undegraded":"2024-09-13T17:20:40.619655+0000","last_fullsized":"2024-09-13T17:20:40.619655+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:25:12.072035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.620144+0000","last_change":"2024-09-13T17:20:39.201780+0000","last_active":"2024-09-13T17:20:40.620144+0000","last_peered":"2024-09-13T17:20:40.620144+0000","last_clean":"2024-09-13T17:20:40.620144+0000","last_became_active":"2024-09-13T17:20:39.201334+0000","last_became_peered":"2024-09-13T17:20:39.201334+0000","last_unstale":"2024-09-13T17:20:40.620144+0000","last_undegraded":"2024-09-13T17:20:40.620144+0000","last_fullsized":"2024-09-13T17:20:40.620144+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:13.936247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.144048+0000","last_change":"2024-09-13T17:20:39.647486+0000","last_active":"2024-09-13T17:20:41.144048+0000","last_peered":"2024-09-13T17:20:41.144048+0000","last_clean":"2024-09-13T17:20:41.144048+0000","last_became_active":"2024-09-13T17:20:39.647208+0000","last_became_peered":"2024-09-13T17:20:39.647208+0000","last_unstale":"2024-09-13T17:20:41.144048+0000","last_undegraded":"2024-09-13T17:20:41.144048+0000","last_fullsized":"2024-09-13T17:20:41.144048+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:03.716102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.984803+0000","last_change":"2024-09-13T17:20:39.205816+0000","last_active":"2024-09-13T17:20:40.984803+0000","last_peered":"2024-09-13T17:20:40.984803+0000","last_clean":"2024-09-13T17:20:40.984803+0000","last_became_active":"2024-09-13T17:20:39.205546+0000","last_became_peered":"2024-09-13T17:20:39.205546+0000","last_unstale":"2024-09-13T17:20:40.984803+0000","last_undegraded":"2024-09-13T17:20:40.984803+0000","last_fullsized":"2024-09-13T17:20:40.984803+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:44.158088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":10,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:40.618014+0000","last_change":"2024-09-13T17:20:39.639611+0000","last_active":"2024-09-13T17:20:40.618014+0000","last_peered":"2024-09-13T17:20:40.618014+0000","last_clean":"2024-09-13T17:20:40.618014+0000","last_became_active":"2024-09-13T17:20:39.638883+0000","last_became_peered":"2024-09-13T17:20:39.638883+0000","last_unstale":"2024-09-13T17:20:40.618014+0000","last_undegraded":"2024-09-13T17:20:40.618014+0000","last_fullsized":"2024-09-13T17:20:40.618014+0000","mapping_epoch":177,"log_start":"0'0","ondisk_log_start":"0'0","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:39.175604+0000","last_clean_scrub_stamp":"2024-09-13T17:20:39.175604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:04.149998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-09-13T17:20:41.143761+0000","last_change":"2024-09-13T17:20:39.205036+0000","last_active":"2024-09-13T17:20:41.143761+0000","last_peered":"2024-09-13T17:20:41.143761+0000","last_clean":"2024-09-13T17:20:41.143761+0000","last_became_active":"2024-09-13T17:20:39.204468+0000","last_became_peered":"2024-09-13T17:20:39.204468+0000","last_unstale":"2024-09-13T17:20:41.143761+0000","last_undegraded":"2024-09-13T17:20:41.143761+0000","last_fullsized":"2024-09-13T17:20:41.143761+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:20:38.168184+0000","last_clean_scrub_stamp":"2024-09-13T17:20:38.168184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:59:41.908698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":298,"reported_epoch":178,"state":"active+clean","last_fresh":"2024-09-13T17:20:39.625031+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:20:39.625031+0000","last_peered":"2024-09-13T17:20:39.625031+0000","last_clean":"2024-09-13T17:20:39.625031+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:20:39.625031+0000","last_undegraded":"2024-09-13T17:20:39.625031+0000","last_fullsized":"2024-09-13T17:20:39.625031+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575004,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65844,"kb_used_data":2784,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305996,"statfs":{"total":96636764160,"available":96569339904,"internally_reserved":0,"allocated":2850816,"data_stored":1583204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":64551371},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.625,"15min":0.64500000000000002},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.33700000000000002},"max":{"1min":0.86499999999999999,"5min":0.996,"15min":2.0880000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.65200000000000002,"15min":0.67300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.316},"max":{"1min":0.88400000000000001,"5min":1.004,"15min":2.6739999999999999},"last":0.71199999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89500000000000002,"5min":1.127,"15min":2.5030000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64600000000000002,"15min":0.69999999999999996},"min":{"1min":0.42899999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":6.4909999999999997},"last":0.63200000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.67500000000000004,"15min":0.69099999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.91700000000000004,"5min":1.169,"15min":1.544},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.66100000000000003,"15min":0.69599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.879,"5min":0.95499999999999996,"15min":1.605},"last":0.77200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.47699999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.83999999999999997,"5min":1.091,"15min":2.4249999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.69199999999999995,"15min":0.73299999999999998},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.84599999999999997,"5min":1.274,"15min":2.8239999999999998},"last":0.84599999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.56899999999999995,"15min":0.61699999999999999},"min":{"1min":0.35799999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73899999999999999,"5min":0.879,"15min":1.054},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.63500000000000001,"15min":0.66600000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.747,"5min":1.234,"15min":1.234},"last":0.81100000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61899999999999999,"15min":0.66200000000000003},"min":{"1min":0.39100000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.83699999999999997,"5min":0.99299999999999999,"15min":1.143},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.70599999999999996,"15min":0.751},"min":{"1min":0.47999999999999998,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.85799999999999998,"5min":1.1659999999999999,"15min":1.337},"last":0.73799999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.68300000000000005,"15min":0.73199999999999998},"min":{"1min":0.39300000000000002,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.87,"5min":1.048,"15min":1.1930000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.68600000000000005,"15min":0.73799999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85099999999999998,"5min":1.101,"15min":1.2030000000000001},"last":0.65600000000000003}]}]},{"osd":6,"up_from":13,"seq":55834575005,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47272,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324568,"statfs":{"total":96636764160,"available":96588357632,"internally_reserved":0,"allocated":2248704,"data_stored":992732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.63400000000000001,"15min":0.66400000000000003},"min":{"1min":0.38300000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89200000000000002,"5min":1.298,"15min":3.6629999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.752,"15min":0.73599999999999999},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.36099999999999999},"max":{"1min":0.95799999999999996,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.81399999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.629,"15min":0.65300000000000002},"min":{"1min":0.32200000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81699999999999995,"5min":2.3100000000000001,"15min":3.7040000000000002},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.40600000000000003,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.96899999999999997,"5min":1.46,"15min":3.3079999999999998},"last":0.745}]},{"osd":2,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.72299999999999998,"15min":0.68000000000000005},"min":{"1min":0.47499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.68700000000000006},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89300000000000002,"5min":0.96899999999999997,"15min":3.2629999999999999},"last":0.871}]},{"osd":3,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.51100000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88800000000000001,"5min":3.6520000000000001,"15min":3.6520000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69499999999999995,"15min":0.70099999999999996},"min":{"1min":0.46600000000000003,"5min":0.46200000000000002,"15min":0.46000000000000002},"max":{"1min":0.871,"5min":3.6720000000000002,"15min":3.742},"last":0.79600000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.61299999999999999,"15min":0.62},"min":{"1min":0.375,"5min":0.33800000000000002,"15min":0.32600000000000001},"max":{"1min":0.75800000000000001,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.501},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.623,"15min":0.64700000000000002},"min":{"1min":0.41999999999999998,"5min":0.36699999999999999,"15min":0.29199999999999998},"max":{"1min":0.84999999999999998,"5min":0.98999999999999999,"15min":4.7750000000000004},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.39800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.871,"5min":1.8260000000000001,"15min":2.3700000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.629,"15min":0.67600000000000005},"min":{"1min":0.39500000000000002,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":0.84099999999999997,"5min":0.92100000000000004,"15min":3.609},"last":0.84799999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.71299999999999997,"15min":0.73599999999999999},"min":{"1min":0.44700000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.91200000000000003,"5min":1.601,"15min":4.9669999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.76800000000000002,"15min":0.78900000000000003},"min":{"1min":0.55900000000000005,"5min":0.52300000000000002,"15min":0.46800000000000003},"max":{"1min":0.93799999999999994,"5min":1.5720000000000001,"15min":3.8100000000000001},"last":0.76400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59552,"kb_used_data":2196,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312288,"statfs":{"total":96636764160,"available":96575782912,"internally_reserved":0,"allocated":2248704,"data_stored":992732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.65100000000000002,"15min":0.76600000000000001},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.85699999999999998,"5min":1.498,"15min":38.005000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.66400000000000003,"15min":0.73999999999999999},"min":{"1min":0.377,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.88800000000000001,"5min":1.577,"15min":21.376999999999999},"last":0.66100000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.66500000000000004,"15min":0.72399999999999998},"min":{"1min":0.42599999999999999,"5min":0.38800000000000001,"15min":0.30499999999999999},"max":{"1min":0.85899999999999999,"5min":1.4770000000000001,"15min":17.231000000000002},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.68200000000000005,"15min":0.69499999999999995},"min":{"1min":0.45800000000000002,"5min":0.41999999999999998,"15min":0.41099999999999998},"max":{"1min":0.90800000000000003,"5min":1.5569999999999999,"15min":7.7370000000000001},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.66600000000000004,"15min":0.73899999999999999},"min":{"1min":0.40600000000000003,"5min":0.379,"15min":0.379},"max":{"1min":0.99299999999999999,"5min":1.524,"15min":17.268000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.68600000000000005,"15min":0.73199999999999998},"min":{"1min":0.439,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.90600000000000003,"5min":1.8680000000000001,"15min":9.6549999999999994},"last":0.82699999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.71499999999999997,"15min":1.0880000000000001},"min":{"1min":0.45700000000000002,"5min":0.44,"15min":0.435},"max":{"1min":0.95099999999999996,"5min":2.8690000000000002,"15min":113.363},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72499999999999998,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.433,"15min":0.433},"max":{"1min":0.86599999999999999,"5min":2.359,"15min":4.1180000000000003},"last":0.752}]},{"osd":4,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.61199999999999999,"15min":0.623},"min":{"1min":0.36099999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79500000000000004,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.64500000000000002,"15min":0.68700000000000006},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.90900000000000003,"5min":2.827,"15min":2.827},"last":0.55400000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.66900000000000004,"15min":0.69599999999999995},"min":{"1min":0.40400000000000003,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.83399999999999996,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.69699999999999995,"15min":0.71699999999999997},"min":{"1min":0.44800000000000001,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":0.83099999999999996,"5min":2.9929999999999999,"15min":4.1500000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:20:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.72599999999999998,"15min":0.752},"min":{"1min":0.442,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.034,"5min":2.9319999999999999,"15min":4.1029999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.73199999999999998,"15min":0.75700000000000001},"min":{"1min":0.47099999999999997,"5min":0.44,"15min":0.44},"max":{"1min":0.89700000000000002,"5min":1.9379999999999999,"15min":2.02},"last":0.64800000000000002}]}]},{"osd":4,"up_from":13,"seq":55834575003,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69372,"kb_used_data":2212,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302468,"statfs":{"total":96636764160,"available":96565727232,"internally_reserved":0,"allocated":2265088,"data_stored":994878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68745670},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.63900000000000001,"15min":0.64800000000000002},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":1.0620000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.69899999999999995,"15min":0.71599999999999997},"min":{"1min":0.45600000000000002,"5min":0.40699999999999997,"15min":0.373},"max":{"1min":0.997,"5min":1.3400000000000001,"15min":1.341},"last":0.83899999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.64900000000000002,"15min":0.65500000000000003},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95399999999999996,"5min":1.173,"15min":1.173},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.40400000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.96599999999999997,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.71299999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.64900000000000002,"15min":0.65600000000000003},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":1.042,"5min":1.1100000000000001,"15min":1.1910000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.68700000000000006,"15min":0.70599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.026,"5min":1.22,"15min":2.1259999999999999},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.496,"5min":0.378,"15min":0.378},"max":{"1min":0.96699999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68899999999999995,"15min":0.69799999999999995},"min":{"1min":0.443,"5min":0.443,"15min":0.40600000000000003},"max":{"1min":1.089,"5min":1.304,"15min":1.304},"last":0.875}]},{"osd":5,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88300000000000001,"5min":1.284,"15min":1.284},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40500000000000003,"5min":0.34799999999999998,"15min":0.30299999999999999},"max":{"1min":1.01,"5min":1.3069999999999999,"15min":1.3240000000000001},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.65900000000000003,"15min":0.66600000000000004},"min":{"1min":0.371,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1,"5min":1.03,"15min":1.274},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.72399999999999998,"15min":0.72899999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.41799999999999998},"max":{"1min":1.069,"5min":1.254,"15min":1.413},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:18 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70499999999999996},"min":{"1min":0.46600000000000003,"5min":0.46100000000000002,"15min":0.39200000000000002},"max":{"1min":1.127,"5min":1.252,"15min":1.252},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.746,"5min":0.745,"15min":0.74099999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.371},"max":{"1min":1.1419999999999999,"5min":1.351,"15min":1.351},"last":0.69299999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575004,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69364,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302476,"statfs":{"total":96636764160,"available":96565735424,"internally_reserved":0,"allocated":2260992,"data_stored":992926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.41899999999999998,"15min":0.44},"min":{"1min":0.28299999999999997,"5min":0.25700000000000001,"15min":0.25},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.316},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46999999999999997,"15min":0.502},"min":{"1min":0.372,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.65300000000000002,"5min":0.68300000000000005,"15min":2.96},"last":0.33600000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.48199999999999998,"15min":0.50800000000000001},"min":{"1min":0.32100000000000001,"5min":0.29799999999999999,"15min":0.28299999999999997},"max":{"1min":0.60199999999999998,"5min":0.81499999999999995,"15min":3.5600000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.55300000000000005,"15min":0.63},"min":{"1min":0.379,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":3.6829999999999998},"last":0.46899999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.84399999999999997,"5min":0.85999999999999999,"15min":1.9590000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53800000000000003,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.34799999999999998,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":1.3819999999999999},"last":0.42499999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.71499999999999997,"15min":0.75800000000000001},"min":{"1min":0.40600000000000003,"5min":0.39700000000000002,"15min":0.39100000000000001},"max":{"1min":1.21,"5min":1.21,"15min":2.8610000000000002},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.73699999999999999,"15min":0.76000000000000001},"min":{"1min":0.52200000000000002,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.23,"5min":1.23,"15min":1.3640000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.73299999999999998,"15min":0.76700000000000002},"min":{"1min":0.49199999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.323,"5min":1.323,"15min":1.391},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.73399999999999999,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.442,"15min":0.442},"max":{"1min":1.224,"5min":1.224,"15min":2.9300000000000002},"last":0.54000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.755,"15min":0.76400000000000001},"min":{"1min":0.54800000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.21,"5min":1.21,"15min":1.401},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.76300000000000001,"15min":0.78200000000000003},"min":{"1min":0.55400000000000005,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.25,"5min":1.25,"15min":1.3300000000000001},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.77000000000000002,"15min":0.80600000000000005},"min":{"1min":0.53400000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.27,"5min":1.27,"15min":2.899},"last":0.626},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.77800000000000002,"15min":0.80100000000000005},"min":{"1min":0.56399999999999995,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.2749999999999999,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.67700000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47280,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324560,"statfs":{"total":96636764160,"available":96588349440,"internally_reserved":0,"allocated":2256896,"data_stored":992407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.46899999999999997,"15min":0.436},"min":{"1min":0.27800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":5.6769999999999996,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.45900000000000002,"15min":0.56799999999999995},"min":{"1min":0.316,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.593,"5min":1.593,"15min":30.023},"last":0.497}]},{"osd":1,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.53600000000000003,"15min":1.365},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":5.7409999999999997,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.52900000000000003,"15min":1.462},"min":{"1min":0.39800000000000002,"5min":0.316,"15min":0.316},"max":{"1min":3.3620000000000001,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.443}]},{"osd":3,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.54500000000000004,"15min":0.61599999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":5.0229999999999997,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.376},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.65800000000000003,"15min":0.60599999999999998},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":8.4149999999999991,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.57899999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.78600000000000003,"15min":0.81100000000000005},"min":{"1min":0.47899999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":6.0039999999999996,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.78800000000000003,"15min":0.86399999999999999},"min":{"1min":0.56499999999999995,"5min":0.371,"15min":0.371},"max":{"1min":6.7080000000000002,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.78500000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.76700000000000002,"15min":0.88400000000000001},"min":{"1min":0.55100000000000005,"5min":0.40300000000000002,"15min":0.376},"max":{"1min":3.8279999999999998,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.995,"5min":0.78400000000000003,"15min":0.78100000000000003},"min":{"1min":0.5,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":5.8890000000000002,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.89600000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":0.80400000000000005,"15min":0.82299999999999995},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.44700000000000001},"max":{"1min":6.4800000000000004,"5min":6.4800000000000004,"15min":10.664},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.98399999999999999,"5min":0.79500000000000004,"15min":0.89900000000000002},"min":{"1min":0.57199999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":4.9480000000000004,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.81499999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:20:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.78500000000000003,"15min":0.82499999999999996},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":3.3929999999999998,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.80800000000000005,"15min":0.90900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":4.9809999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.86899999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575003,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91500,"kb_used_data":2208,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280340,"statfs":{"total":96636764160,"available":96543068160,"internally_reserved":0,"allocated":2260992,"data_stored":994296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.42499999999999999,"15min":0.47199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56000000000000005,"5min":0.64700000000000002,"15min":5.4939999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.48599999999999999,"15min":0.51600000000000001},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.70199999999999996,"5min":0.73399999999999999,"15min":6.7240000000000002},"last":0.69499999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.45900000000000002},"min":{"1min":0.32200000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":2.3119999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.496,"15min":0.52900000000000003},"min":{"1min":0.34699999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.67600000000000005,"5min":0.75900000000000001,"15min":1.5089999999999999},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57099999999999995,"15min":0.66100000000000003},"min":{"1min":0.48599999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.63700000000000001,"5min":0.76100000000000001,"15min":16.382999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52500000000000002,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.81299999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.748,"15min":0.78800000000000003},"min":{"1min":0.52600000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.93400000000000005,"5min":1.159,"15min":6.1630000000000003},"last":1.1140000000000001},{"interface":"front","average":{"1min":0.748,"5min":0.78000000000000003,"15min":0.80200000000000005},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.91800000000000004,"5min":1.149,"15min":1.78},"last":1.23}]},{"osd":5,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.752,"15min":0.76500000000000001},"min":{"1min":0.57799999999999996,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88700000000000001,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":1.159},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.74399999999999999,"15min":0.78300000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.96899999999999997,"5min":1.1319999999999999,"15min":6.1189999999999998},"last":1.175}]},{"osd":6,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.78300000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.96699999999999997,"5min":1.1850000000000001,"15min":1.7310000000000001},"last":1.093},{"interface":"front","average":{"1min":0.752,"5min":0.76500000000000001,"15min":0.80300000000000005},"min":{"1min":0.47399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.94799999999999995,"5min":1.204,"15min":6.1970000000000001},"last":1.2150000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:20:14 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76700000000000002,"15min":0.79500000000000004},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.95699999999999996,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78400000000000003,"15min":0.81000000000000005},"min":{"1min":0.57299999999999995,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":0.94099999999999995,"5min":1.1679999999999999,"15min":1.6359999999999999},"last":1.2609999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575005,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69932,"kb_used_data":2776,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301908,"statfs":{"total":96636764160,"available":96565153792,"internally_reserved":0,"allocated":2842624,"data_stored":1582103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.45100000000000001,"15min":0.54000000000000004},"min":{"1min":0.26500000000000001,"5min":0.247,"15min":0.247},"max":{"1min":0.84399999999999997,"5min":1.986,"15min":5.5259999999999998},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.52500000000000002,"15min":0.57399999999999995},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":2.371,"5min":2.738,"15min":5.4889999999999999},"last":0.56100000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.54400000000000004,"15min":0.59699999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.89900000000000002,"5min":3.605,"15min":6.0430000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.60399999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":2.4039999999999999,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.499}]},{"osd":3,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.498,"15min":0.56799999999999995},"min":{"1min":0.36099999999999999,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.878,"5min":0.98999999999999999,"15min":4.7549999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.56699999999999995,"15min":0.63400000000000001},"min":{"1min":0.38,"5min":0.375,"15min":0.34999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.52200000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.80000000000000004,"15min":0.82599999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":3.8849999999999998,"5min":3.8849999999999998,"15min":4.734},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.745,"15min":0.81799999999999995},"min":{"1min":0.51600000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.93799999999999994,"5min":1.3109999999999999,"15min":4.8079999999999998},"last":1.0009999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.76000000000000001,"15min":0.82999999999999996},"min":{"1min":0.54200000000000004,"5min":0.47799999999999998,"15min":0.44500000000000001},"max":{"1min":0.96099999999999997,"5min":1.2190000000000001,"15min":4.944},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.78400000000000003,"15min":0.83299999999999996},"min":{"1min":0.50700000000000001,"5min":0.432,"15min":0.432},"max":{"1min":1.0820000000000001,"5min":1.8740000000000001,"15min":4.9710000000000001},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.77700000000000002,"15min":0.89100000000000001},"min":{"1min":0.47099999999999997,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.97999999999999998,"5min":1.161,"15min":4.8470000000000004},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.82799999999999996,"15min":0.85499999999999998},"min":{"1min":0.497,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.97299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:20:20 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.86499999999999999,"15min":0.88200000000000001},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.47399999999999998},"max":{"1min":3.9209999999999998,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":1.02},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.80400000000000005,"15min":0.86199999999999999},"min":{"1min":0.58299999999999996,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.081,"5min":1.3149999999999999,"15min":4.9950000000000001},"last":0.91200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:20:44.242 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:20:44.243 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:20:44.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.437+0000 7f18a6e00640 1 Processor -- start 2024-09-13T17:20:44.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.438+0000 7f18a6e00640 1 -- start start 2024-09-13T17:20:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f18a00cd260 0x7f18a0070910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a0070480 con 0x7f18a0159b30 2024-09-13T17:20:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a015a010 con 0x7f18a00cd260 2024-09-13T17:20:44.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a006fb50 con 0x7f18a0156650 2024-09-13T17:20:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a5a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:35172/0 (socket says 172.21.15.28:35172) 2024-09-13T17:20:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a5a00640 1 -- 172.21.15.28:0/3737167518 learned_addr learned my addr 172.21.15.28:0/3737167518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.439+0000 7f18a6400640 1 --2- 172.21.15.28:0/3737167518 >> v2:172.21.15.189:3300/0 conn(0x7f18a00cd260 0x7f18a0070910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.440+0000 7f18a5a00640 1 -- 172.21.15.28:0/3737167518 >> v2:172.21.15.189:3300/0 conn(0x7f18a00cd260 msgr2=0x7f18a0070910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.440+0000 7f18a5a00640 1 --2- 172.21.15.28:0/3737167518 >> v2:172.21.15.189:3300/0 conn(0x7f18a00cd260 0x7f18a0070910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.440+0000 7f18a5a00640 1 -- 172.21.15.28:0/3737167518 >> v1:172.21.15.189:6789/0 conn(0x7f18a0159b30 legacy=0x7f18a00ccb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.440+0000 7f18a5a00640 1 -- 172.21.15.28:0/3737167518 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a006fe30 con 0x7f18a0156650 2024-09-13T17:20:44.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.441+0000 7f18a5a00640 1 --2- 172.21.15.28:0/3737167518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1894009870 tx=0x7f189402a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=865f8443384cd9be server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.441+0000 7f189fe00640 1 -- 172.21.15.28:0/3737167518 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894032440 con 0x7f18a0156650 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.441+0000 7f189fe00640 1 -- 172.21.15.28:0/3737167518 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18940325a0 con 0x7f18a0156650 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.441+0000 7f189fe00640 1 -- 172.21.15.28:0/3737167518 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1894032930 con 0x7f18a0156650 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 -- 172.21.15.28:0/3737167518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 msgr2=0x7f18a015a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 --2- 172.21.15.28:0/3737167518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1894009870 tx=0x7f189402a250 comp rx=0 tx=0).stop 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 -- 172.21.15.28:0/3737167518 shutdown_connections 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 --2- 172.21.15.28:0/3737167518 >> v2:172.21.15.189:3300/0 conn(0x7f18a00cd260 0x7f18a0070910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 --2- 172.21.15.28:0/3737167518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0156650 0x7f18a015a270 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 -- 172.21.15.28:0/3737167518 >> 172.21.15.28:0/3737167518 conn(0x7f18a0069e80 msgr2=0x7f18a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 -- 172.21.15.28:0/3737167518 shutdown_connections 2024-09-13T17:20:44.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.442+0000 7f18a6e00640 1 -- 172.21.15.28:0/3737167518 wait complete. 2024-09-13T17:20:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.444+0000 7f18a6e00640 1 Processor -- start 2024-09-13T17:20:44.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.444+0000 7f18a6e00640 1 -- start start 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a00cd260 0x7f18a015bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0159b30 0x7f18a0167000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a015c480 con 0x7f18a0156650 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0166240 con 0x7f18a00cd260 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a0166520 con 0x7f18a0159b30 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57136/0 (socket says 172.21.15.28:57136) 2024-09-13T17:20:44.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 -- 172.21.15.28:0/576356000 learned_addr learned my addr 172.21.15.28:0/576356000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5a00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a00cd260 0x7f18a015bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a6400640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0159b30 0x7f18a0167000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 -- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0159b30 msgr2=0x7f18a0167000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0159b30 0x7f18a0167000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 -- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a00cd260 msgr2=0x7f18a015bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a00cd260 0x7f18a015bc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.445+0000 7f18a5000640 1 -- 172.21.15.28:0/576356000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18940094d0 con 0x7f18a0156650 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f18a5000640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f188c00c3c0 tx=0x7f188c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c0052d0 con 0x7f18a0156650 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c005450 con 0x7f18a0156650 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c003830 con 0x7f18a0156650 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a0166800 con 0x7f18a0156650 2024-09-13T17:20:44.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.446+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a015c170 con 0x7f18a0156650 2024-09-13T17:20:44.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.447+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a0166380 con 0x7f18a0156650 2024-09-13T17:20:44.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.448+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f188c003990 con 0x7f18a0156650 2024-09-13T17:20:44.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.449+0000 7f189d600640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 0x7f188006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.449+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f188c088790 con 0x7f18a0156650 2024-09-13T17:20:44.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.449+0000 7f18a5a00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 0x7f188006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.450+0000 7f18a5a00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 0x7f188006d0c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f189402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.453+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f18a0166380 con 0x7f18a0156650 2024-09-13T17:20:44.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.703+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f18a0166d20 con 0x7f18a0156650 2024-09-13T17:20:44.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.708+0000 7f189d600640 1 -- 172.21.15.28:0/576356000 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v25) v1 ==== 190+0+0 (secure 0 0 0) 0x7f18a0166d20 con 0x7f18a0156650 2024-09-13T17:20:44.710 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 msgr2=0x7f188006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 0x7f188006d0c0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f1894034000 tx=0x7f189402e000 comp rx=0 tx=0).stop 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 msgr2=0x7f18a014de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f188c00c3c0 tx=0x7f188c00c880 comp rx=0 tx=0).stop 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 shutdown_connections 2024-09-13T17:20:44.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f188006ac00 0x7f188006d0c0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f18a0159b30 0x7f18a0167000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f18a0156650 0x7f18a014de30 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 --2- 172.21.15.28:0/576356000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f18a00cd260 0x7f18a015bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 >> 172.21.15.28:0/576356000 conn(0x7f18a0069e80 msgr2=0x7f18a01583f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 shutdown_connections 2024-09-13T17:20:44.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.710+0000 7f18a6e00640 1 -- 172.21.15.28:0/576356000 wait complete. 2024-09-13T17:20:44.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:20:44.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.881+0000 7fb86a600640 1 Processor -- start 2024-09-13T17:20:44.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.881+0000 7fb86a600640 1 -- start start 2024-09-13T17:20:44.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.882+0000 7fb86a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb86415a540 0x7fb86406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.882+0000 7fb86a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8640cabc0 0x7fb86415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.882+0000 7fb86a600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86415e100 con 0x7fb86415edb0 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.882+0000 7fb86a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8640adb00 con 0x7fb86415a540 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.882+0000 7fb86a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86415e3e0 con 0x7fb8640cabc0 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb86415a540 0x7fb86406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb86415edb0 0x7fb864070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44544/0 (socket says 172.21.15.28:44544) 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863e00640 1 -- 172.21.15.28:0/770034506 learned_addr learned my addr 172.21.15.28:0/770034506 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:44.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb862a00640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8640cabc0 0x7fb86415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863400640 1 -- 172.21.15.28:0/770034506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8640cabc0 msgr2=0x7fb86415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863400640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8640cabc0 0x7fb86415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863400640 1 -- 172.21.15.28:0/770034506 >> v1:172.21.15.28:6789/0 conn(0x7fb86415edb0 legacy=0x7fb864070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb863400640 1 -- 172.21.15.28:0/770034506 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86406ce80 con 0x7fb86415a540 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.883+0000 7fb862000640 1 -- 172.21.15.28:0/770034506 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988676684 0 0) 0x7fb86415e100 con 0x7fb86415edb0 2024-09-13T17:20:44.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.884+0000 7fb863400640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb86415a540 0x7fb86406c430 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4884aa06d95bca5c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.884+0000 7fb862000640 1 -- 172.21.15.28:0/770034506 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85402ae50 con 0x7fb86415a540 2024-09-13T17:20:44.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb862000640 1 -- 172.21.15.28:0/770034506 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb854032440 con 0x7fb86415a540 2024-09-13T17:20:44.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb862000640 1 -- 172.21.15.28:0/770034506 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8540327d0 con 0x7fb86415a540 2024-09-13T17:20:44.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 -- 172.21.15.28:0/770034506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb86415a540 msgr2=0x7fb86406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb86415a540 0x7fb86406c430 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7fb8540097a0 tx=0x7fb85402a100 comp rx=0 tx=0).stop 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 -- 172.21.15.28:0/770034506 shutdown_connections 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8640cabc0 0x7fb86415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 --2- 172.21.15.28:0/770034506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb86415a540 0x7fb86406c430 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 -- 172.21.15.28:0/770034506 >> 172.21.15.28:0/770034506 conn(0x7fb864069e80 msgr2=0x7fb86406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 -- 172.21.15.28:0/770034506 shutdown_connections 2024-09-13T17:20:44.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.885+0000 7fb86a600640 1 -- 172.21.15.28:0/770034506 wait complete. 2024-09-13T17:20:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.887+0000 7fb86a600640 1 Processor -- start 2024-09-13T17:20:44.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.887+0000 7fb86a600640 1 -- start start 2024-09-13T17:20:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 0x7fb86415d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 0x7fb864155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864157700 con 0x7fb8640cabc0 2024-09-13T17:20:44.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb864156b10 con 0x7fb86415a540 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb86a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb864156dd0 con 0x7fb864148fa0 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb863400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb863400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57154/0 (socket says 172.21.15.28:57154) 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.888+0000 7fb863400640 1 -- 172.21.15.28:0/1065608463 learned_addr learned my addr 172.21.15.28:0/1065608463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 -- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 msgr2=0x7fb864155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863e00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 0x7fb864155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb862a00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 0x7fb86415d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 0x7fb864155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 -- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 msgr2=0x7fb86415d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 0x7fb86415d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 -- 172.21.15.28:0/1065608463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854009400 con 0x7fb8640cabc0 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863e00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 0x7fb864155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:44.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb862a00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 0x7fb86415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb863400640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb854035040 tx=0x7fb8540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85403d710 con 0x7fb8640cabc0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85403d890 con 0x7fb8640cabc0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85403dc40 con 0x7fb8640cabc0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.889+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8641570b0 con 0x7fb8640cabc0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.890+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8641580d0 con 0x7fb8640cabc0 2024-09-13T17:20:44.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.890+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8640704b0 con 0x7fb8640cabc0 2024-09-13T17:20:44.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.891+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb854039040 con 0x7fb8640cabc0 2024-09-13T17:20:44.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.892+0000 7fb85b400640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 0x7fb84406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:44.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.892+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8540a9c10 con 0x7fb8640cabc0 2024-09-13T17:20:44.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.893+0000 7fb862a00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 0x7fb84406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:44.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.893+0000 7fb862a00640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 0x7fb84406d0c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb864152620 tx=0x7fb850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:44.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:44.896+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb854074d50 con 0x7fb8640cabc0 2024-09-13T17:20:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.109+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8640701d0 con 0x7fb8640cabc0 2024-09-13T17:20:45.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.109+0000 7fb85b400640 1 -- 172.21.15.28:0/1065608463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+4754 (secure 0 0 0) 0x7fb854078370 con 0x7fb8640cabc0 2024-09-13T17:20:45.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:20:45.112 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":151,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":10551,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:20:42.633390+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:20:45.112 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 151 2024-09-13T17:20:45.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 msgr2=0x7fb84406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 0x7fb84406d0c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fb864152620 tx=0x7fb850008040 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 msgr2=0x7fb864157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7fb854035040 tx=0x7fb8540046c0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 shutdown_connections 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb84406ac00 0x7fb84406d0c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb864148fa0 0x7fb864155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb86415a540 0x7fb86415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 --2- 172.21.15.28:0/1065608463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8640cabc0 0x7fb864157b90 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.111+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 >> 172.21.15.28:0/1065608463 conn(0x7fb864069e80 msgr2=0x7fb8640cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.112+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 shutdown_connections 2024-09-13T17:20:45.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.112+0000 7fb86a600640 1 -- 172.21.15.28:0/1065608463 wait complete. 2024-09-13T17:20:45.164 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-13T17:20:45.163+0000 7fd086c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:20:45.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 151, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:20:39.612305+0000', 'modified': '2024-09-13T17:20:42.633390+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10566}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10566': {'gid': 10566, 'name': 'a', 'rank': 0, 'incarnation': 147, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3229070352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3229070352}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3229070352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:20:45.533 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:20:45.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:20:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.690+0000 7f05bdc00640 1 Processor -- start 2024-09-13T17:20:45.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.691+0000 7f05bdc00640 1 -- start start 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bdc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05b80c40a0 0x7f05b80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bdc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b806c370 0x7f05b806c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bdc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b80b2090 con 0x7f05b80b1ba0 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bdc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b806cd20 con 0x7f05b806c370 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bdc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b80b2370 con 0x7f05b80c40a0 2024-09-13T17:20:45.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05bd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f05b806c370 0x7f05b806c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05afe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05b80b1ba0 0x7f05b815ab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44554/0 (socket says 172.21.15.28:44554) 2024-09-13T17:20:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05afe00640 1 -- 172.21.15.28:0/3204945706 learned_addr learned my addr 172.21.15.28:0/3204945706 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:45.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.692+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3204945706 >> v2:172.21.15.189:3300/0 conn(0x7f05b80c40a0 0x7f05b80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.693+0000 7f05bd200640 1 -- 172.21.15.28:0/3204945706 >> v2:172.21.15.189:3300/0 conn(0x7f05b80c40a0 msgr2=0x7f05b80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.693+0000 7f05bd200640 1 --2- 172.21.15.28:0/3204945706 >> v2:172.21.15.189:3300/0 conn(0x7f05b80c40a0 0x7f05b80b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.693+0000 7f05bd200640 1 -- 172.21.15.28:0/3204945706 >> v1:172.21.15.28:6789/0 conn(0x7f05b80b1ba0 legacy=0x7f05b815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.693+0000 7f05bd200640 1 -- 172.21.15.28:0/3204945706 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b80b2650 con 0x7f05b806c370 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.693+0000 7f05b7400640 1 -- 172.21.15.28:0/3204945706 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637879273 0 0) 0x7f05b80b2090 con 0x7f05b80b1ba0 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.694+0000 7f05bd200640 1 --2- 172.21.15.28:0/3204945706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b806c370 0x7f05b806c7e0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f05b80ce360 tx=0x7f05b002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b37cfd68d7d8722 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.694+0000 7f05b7400640 1 -- 172.21.15.28:0/3204945706 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b002aea0 con 0x7f05b806c370 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.694+0000 7f05b7400640 1 -- 172.21.15.28:0/3204945706 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b002d050 con 0x7f05b806c370 2024-09-13T17:20:45.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.694+0000 7f05b7400640 1 -- 172.21.15.28:0/3204945706 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0007960 con 0x7f05b806c370 2024-09-13T17:20:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 -- 172.21.15.28:0/3204945706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b806c370 msgr2=0x7f05b806c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3204945706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b806c370 0x7f05b806c7e0 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f05b80ce360 tx=0x7f05b002a680 comp rx=0 tx=0).stop 2024-09-13T17:20:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 -- 172.21.15.28:0/3204945706 shutdown_connections 2024-09-13T17:20:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3204945706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b806c370 0x7f05b806c7e0 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3204945706 >> v2:172.21.15.189:3300/0 conn(0x7f05b80c40a0 0x7f05b80b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 -- 172.21.15.28:0/3204945706 >> 172.21.15.28:0/3204945706 conn(0x7f05b8073470 msgr2=0x7f05b8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 -- 172.21.15.28:0/3204945706 shutdown_connections 2024-09-13T17:20:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.695+0000 7f05bdc00640 1 -- 172.21.15.28:0/3204945706 wait complete. 2024-09-13T17:20:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.696+0000 7f05bdc00640 1 Processor -- start 2024-09-13T17:20:45.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.697+0000 7f05bdc00640 1 -- start start 2024-09-13T17:20:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.697+0000 7f05bdc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05b806c370 0x7f05b81efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bdc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bdc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80c40a0 0x7f05b81f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bdc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b81f4010 con 0x7f05b80b1ba0 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bdc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05b81f1c90 con 0x7f05b80c40a0 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bdc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b81f3390 con 0x7f05b806c370 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57184/0 (socket says 172.21.15.28:57184) 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 -- 172.21.15.28:0/3291882558 learned_addr learned my addr 172.21.15.28:0/3291882558 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05b806c370 0x7f05b81efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05bd200640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80c40a0 0x7f05b81f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 -- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05b806c370 msgr2=0x7f05b81efaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05b806c370 0x7f05b81efaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 -- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80c40a0 msgr2=0x7f05b81f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80c40a0 0x7f05b81f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.698+0000 7f05afe00640 1 -- 172.21.15.28:0/3291882558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b0009a70 con 0x7f05b80b1ba0 2024-09-13T17:20:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05afe00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c4b0 tx=0x7f05a000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05b4c00640 1 -- 172.21.15.28:0/3291882558 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a00074e0 con 0x7f05b80b1ba0 2024-09-13T17:20:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05b4c00640 1 -- 172.21.15.28:0/3291882558 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a0007660 con 0x7f05b80b1ba0 2024-09-13T17:20:45.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05b4c00640 1 -- 172.21.15.28:0/3291882558 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0004a20 con 0x7f05b80b1ba0 2024-09-13T17:20:45.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b815b400 con 0x7f05b80b1ba0 2024-09-13T17:20:45.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.699+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b81f3950 con 0x7f05b80b1ba0 2024-09-13T17:20:45.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.701+0000 7f05b4c00640 1 -- 172.21.15.28:0/3291882558 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f05a0004b80 con 0x7f05b80b1ba0 2024-09-13T17:20:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.702+0000 7f05b4c00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 0x7f059406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.702+0000 7f05b4c00640 1 -- 172.21.15.28:0/3291882558 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a00884a0 con 0x7f05b80b1ba0 2024-09-13T17:20:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.702+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 0x7f059406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.703+0000 7f05c207b740 1 Processor -- start 2024-09-13T17:20:45.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.703+0000 7f05c207b740 1 -- start start 2024-09-13T17:20:45.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.703+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 0x7f059406d0f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009aa0 tx=0x7f05a8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb31000 0x5630dbb80d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb81250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dba9cab0 con 0x5630dbb33840 2024-09-13T17:20:45.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dba9a240 con 0x5630dbb5acf0 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630dba95b30 con 0x5630dbb31000 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57192/0 (socket says 172.21.15.28:57192) 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 -- 172.21.15.28:0/2633571071 learned_addr learned my addr 172.21.15.28:0/2633571071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05bd200640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb81250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05afe00640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb31000 0x5630dbb80d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 -- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb31000 msgr2=0x5630dbb80d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb31000 0x5630dbb80d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 -- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 msgr2=0x5630dbb81250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.704+0000 7f05b7e00640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb81250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05b7e00640 1 -- 172.21.15.28:0/2633571071 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630dba60bd0 con 0x5630dbb33840 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05bd200640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb81250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05b7e00640 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f05a8004440 tx=0x7f05a8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5c55c45e9703e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05acc00640 1 -- 172.21.15.28:0/2633571071 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a801c040 con 0x5630dbb33840 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05acc00640 1 -- 172.21.15.28:0/2633571071 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8003820 con 0x5630dbb33840 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.705+0000 7f05acc00640 1 -- 172.21.15.28:0/2633571071 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8003b90 con 0x5630dbb33840 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 -- 172.21.15.28:0/2633571071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 msgr2=0x5630dbb33c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f05a8004440 tx=0x7f05a8014000 comp rx=0 tx=0).stop 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 -- 172.21.15.28:0/2633571071 shutdown_connections 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb81250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb31000 0x5630dbb80d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 --2- 172.21.15.28:0/2633571071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb33c20 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 -- 172.21.15.28:0/2633571071 >> 172.21.15.28:0/2633571071 conn(0x5630dbb2ff70 msgr2=0x5630dbb78420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 -- 172.21.15.28:0/2633571071 shutdown_connections 2024-09-13T17:20:45.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.706+0000 7f05c207b740 1 -- 172.21.15.28:0/2633571071 wait complete. 2024-09-13T17:20:45.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.707+0000 7f05c207b740 1 -- start start 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.707+0000 7f05c207b740 1 Processor -- start 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 0x5630dbb5c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dba37260 con 0x5630dbb33840 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dba36f00 con 0x5630dbb5c8f0 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630db7fcf20 con 0x5630dbb5acf0 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05c207b740 10 client.0 _ll_register_callbacks cb 0x5630dba8f1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57206/0 (socket says 172.21.15.28:57206) 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05b7e00640 1 -- 172.21.15.28:0/772359169 learned_addr learned my addr 172.21.15.28:0/772359169 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.708+0000 7f05afe00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 0x5630dbb5c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05bd200640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 -- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 msgr2=0x5630dbb5c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 0x5630dbb5c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 -- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 msgr2=0x5630dbb7dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 -- 172.21.15.28:0/772359169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a8004120 con 0x5630dbb33840 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05afe00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 0x5630dbb5c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05bd200640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05b7e00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f05a8011dd0 tx=0x7f05a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54baf386a8a995e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05a5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8022670 con 0x5630dbb33840 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.709+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a8037ae0 con 0x5630dbb33840 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630dbabfcd0 con 0x5630dbb33840 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05a8037ca0 con 0x5630dbb33840 2024-09-13T17:20:45.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 10 client.10704 fetch_fsmap learned FSMap version 151 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=2+}) v3 -- 0x5630dba941b0 con 0x5630dbb33840 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5630dbabfcd0 con 0x5630dbb33840 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 151) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f05a80227d0 con 0x5630dbb33840 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 10 client.10704 fetch_fsmap finished waiting for FSMap version 151 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 10 client.10704 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 20 client.10704 populate_metadata read hostname 'smithi028' 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb7e680 0x5630dbb953e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 4 client.10704 mds_command: new command op to 10566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.710+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 --> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630db7893c0 con 0x5630dbb7e680 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.711+0000 7f05afe00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb7e680 0x5630dbb953e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.711+0000 7f05afe00640 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb7e680 0x5630dbb953e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=251ab17e628b71c7 server_cookie=573459c4ff1321fe in_seq=0 out_seq=0 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.711+0000 7f05a5600640 10 client.10704 ms_handle_connect on v2:172.21.15.28:6834/3229070352 2024-09-13T17:20:45.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05a5600640 1 -- 172.21.15.28:0/772359169 <== mds.0 v2:172.21.15.28:6834/3229070352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5630db7893c0 con 0x5630dbb7e680 2024-09-13T17:20:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05a5600640 10 client.10704 handle_command_reply: tid=0 2024-09-13T17:20:45.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 client.10704 shutdown 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 msgr2=0x5630dbb5a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f05a8011dd0 tx=0x7f05a8004730 comp rx=0 tx=0).stop 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 shutdown_connections 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb7e680 0x5630dbb953e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb5acf0 0x5630dbb5c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 --2- 172.21.15.28:0/772359169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb33840 0x5630dbb5a360 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05a5600640 0 client.10704 ms_handle_reset on v2:172.21.15.28:6834/3229070352 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 >> 172.21.15.28:0/772359169 conn(0x5630dbb31000 msgr2=0x5630dbb7b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 shutdown_connections 2024-09-13T17:20:45.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.712+0000 7f05c207b740 1 -- 172.21.15.28:0/772359169 wait complete. 2024-09-13T17:20:45.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.713+0000 7f05c207b740 20 client.10704 trim_cache size 0 max 16384 2024-09-13T17:20:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.728+0000 7f05c207b740 1 Processor -- start 2024-09-13T17:20:45.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.728+0000 7f05c207b740 1 -- start start 2024-09-13T17:20:45.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbb97820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb7c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dbb7c2a0 con 0x5630dbb5c8f0 2024-09-13T17:20:45.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dbb7fe10 con 0x5630dbb5acf0 2024-09-13T17:20:45.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630dbb5a240 con 0x5630dbb33840 2024-09-13T17:20:45.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05bd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05bd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57212/0 (socket says 172.21.15.28:57212) 2024-09-13T17:20:45.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.729+0000 7f05bd200640 1 -- 172.21.15.28:0/3071795743 learned_addr learned my addr 172.21.15.28:0/3071795743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05afe00640 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb7c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbb97820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 -- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 msgr2=0x5630dbb97820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbb97820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 -- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 msgr2=0x5630dbb7c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb7c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 -- 172.21.15.28:0/3071795743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630dbac9170 con 0x5630dbb5c8f0 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05bd200640 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e80 tx=0x7f05b002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9235a9c1b9911d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.730+0000 7f05ad600640 1 -- 172.21.15.28:0/3071795743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b004b070 con 0x5630dbb5c8f0 2024-09-13T17:20:45.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05ad600640 1 -- 172.21.15.28:0/3071795743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b0043040 con 0x5630dbb5c8f0 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05ad600640 1 -- 172.21.15.28:0/3071795743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0007760 con 0x5630dbb5c8f0 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 -- 172.21.15.28:0/3071795743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 msgr2=0x5630dbb7cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f05b0009e80 tx=0x7f05b002e000 comp rx=0 tx=0).stop 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 -- 172.21.15.28:0/3071795743 shutdown_connections 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7cdf0 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5acf0 0x5630dbb7c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 --2- 172.21.15.28:0/3071795743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbb97820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 -- 172.21.15.28:0/3071795743 >> 172.21.15.28:0/3071795743 conn(0x5630dbb31000 msgr2=0x5630dbb79890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 -- 172.21.15.28:0/3071795743 shutdown_connections 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.731+0000 7f05c207b740 1 -- 172.21.15.28:0/3071795743 wait complete. 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.732+0000 7f05c207b740 1 -- start start 2024-09-13T17:20:45.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.732+0000 7f05c207b740 1 Processor -- start 2024-09-13T17:20:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbbc8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dbb958f0 con 0x5630dbb5acf0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630dbbe3ce0 con 0x5630dbb5c8f0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5630dbb95580 con 0x5630dbb33840 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05c207b740 10 client.0 _ll_register_callbacks cb 0x7f05a8009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05afe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57216/0 (socket says 172.21.15.28:57216) 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05afe00640 1 -- 172.21.15.28:0/3855136750 learned_addr learned my addr 172.21.15.28:0/3855136750 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbbc8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.733+0000 7f05bd200640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 -- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 msgr2=0x5630dbbc8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbbc8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 -- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 msgr2=0x5630dbb7c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 -- 172.21.15.28:0/3855136750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b80b2650 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05afe00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f05a0086930 tx=0x7f05a0009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a84e6e80c94ca19 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05a5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a0086070 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05a0053b20 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.734+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f05a8003820 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=151) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05a0053ce0 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 10 client.10710 fetch_fsmap learned FSMap version 151 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=151,monmap=2+}) v3 -- 0x5630dba36470 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8003820 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 151) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f05a0056900 con 0x5630dbb5acf0 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 10 client.10710 fetch_fsmap finished waiting for FSMap version 151 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 10 client.10710 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:20:45.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 20 client.10710 populate_metadata read hostname 'smithi028' 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb57ab0 0x5630dbbee510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 4 client.10710 mds_command: new command op to 10566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 --> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5630db79a6c0 con 0x5630dbb57ab0 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.735+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb57ab0 0x5630dbbee510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05b7e00640 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb57ab0 0x5630dbbee510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1ad269db6d32997 server_cookie=ccf55f734ec9149d in_seq=0 out_seq=0 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05a5600640 10 client.10710 ms_handle_connect on v2:172.21.15.28:6834/3229070352 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05a5600640 1 -- 172.21.15.28:0/3855136750 <== mds.0 v2:172.21.15.28:6834/3229070352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x5630db79a6c0 con 0x5630dbb57ab0 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05a5600640 10 client.10710 handle_command_reply: tid=0 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 client.10710 shutdown 2024-09-13T17:20:45.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 msgr2=0x5630dbbc8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f05a0086930 tx=0x7f05a0009040 comp rx=0 tx=0).stop 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 shutdown_connections 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x5630dbb57ab0 0x5630dbbee510 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5630dbb5c8f0 0x5630dbb7c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5630dbb5acf0 0x5630dbbc8b70 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05a5600640 0 client.10710 ms_handle_reset on v2:172.21.15.28:6834/3229070352 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 --2- 172.21.15.28:0/3855136750 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5630dbb33840 0x5630dbbc8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 >> 172.21.15.28:0/3855136750 conn(0x5630dbb31000 msgr2=0x5630dbb7a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 shutdown_connections 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.736+0000 7f05c207b740 1 -- 172.21.15.28:0/3855136750 wait complete. 2024-09-13T17:20:45.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05c207b740 20 client.10710 trim_cache size 0 max 16384 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 10566, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1225265069999999, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 151, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 179, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 179, 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.234056313 2024-09-13T17:20:45.744 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:20:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 msgr2=0x7f059406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 0x7f059406d0f0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009aa0 tx=0x7f05a8009190 comp rx=0 tx=0).stop 2024-09-13T17:20:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 msgr2=0x7f05b81419f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f05a000c4b0 tx=0x7f05a000c970 comp rx=0 tx=0).stop 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 shutdown_connections 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f059406ac30 0x7f059406d0f0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05b80c40a0 0x7f05b81f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05b80b1ba0 0x7f05b81419f0 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 --2- 172.21.15.28:0/3291882558 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05b806c370 0x7f05b81efaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 >> 172.21.15.28:0/3291882558 conn(0x7f05b8073470 msgr2=0x7f05b815d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 shutdown_connections 2024-09-13T17:20:45.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.737+0000 7f05bdc00640 1 -- 172.21.15.28:0/3291882558 wait complete. 2024-09-13T17:20:45.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:20:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.908+0000 7ff2f9800640 1 Processor -- start 2024-09-13T17:20:45.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.909+0000 7ff2f9800640 1 -- start start 2024-09-13T17:20:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f9800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2f41602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f406bde0 con 0x7ff2f4156e00 2024-09-13T17:20:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f9800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f40b3a80 con 0x7ff2f4156a20 2024-09-13T17:20:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f9800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f406b820 con 0x7ff2f40b3620 2024-09-13T17:20:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2f4156e00 0x7ff2f415d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44566/0 (socket says 172.21.15.28:44566) 2024-09-13T17:20:45.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.910+0000 7ff2f3400640 1 -- 172.21.15.28:0/154929653 learned_addr learned my addr 172.21.15.28:0/154929653 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:20:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2f41602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948767540 0 0) 0x7ff2f406bde0 con 0x7ff2f4156e00 2024-09-13T17:20:45.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d8003520 con 0x7ff2f4156e00 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1359384637 0 0) 0x7ff2d8003520 con 0x7ff2f4156e00 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 msgr2=0x7ff2f41602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2f41602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 >> v1:172.21.15.189:6789/0 conn(0x7ff2f4156a20 legacy=0x7ff2f4071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f406bb00 con 0x7ff2f4156e00 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7ff2e0001670 con 0x7ff2f4156e00 2024-09-13T17:20:45.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 >> v1:172.21.15.28:6789/0 conn(0x7ff2f4156e00 legacy=0x7ff2f415d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2d80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d8003ee0 0x7ff2d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f406b820 con 0x7ff2d8004d00 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d800b9e0 con 0x7ff2d8003ee0 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.911+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d800bba0 con 0x7ff2f40b3620 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f3e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2d80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f3400640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d8003ee0 0x7ff2d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 -- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 msgr2=0x7ff2d80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2d80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 -- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d8003ee0 msgr2=0x7ff2d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2d8003ee0 0x7ff2d80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f8e00640 1 -- 172.21.15.28:0/154929653 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8003520 con 0x7ff2d8004d00 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.912+0000 7ff2f3e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f40b3620 0x7ff2d80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.913+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2d8005180 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009fc0 tx=0x7ff2d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef46346e1723395e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.913+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec031640 con 0x7ff2d8004d00 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.913+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2ec02d070 con 0x7ff2d8004d00 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.913+0000 7ff2f2a00640 1 -- 172.21.15.28:0/154929653 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2ec0319e0 con 0x7ff2d8004d00 2024-09-13T17:20:45.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 -- 172.21.15.28:0/154929653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 msgr2=0x7ff2d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 --2- 172.21.15.28:0/154929653 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2d8005180 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7ff2ec009fc0 tx=0x7ff2d80057f0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f8e00640 1 -- 172.21.15.28:0/154929653 reap_dead start 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 -- 172.21.15.28:0/154929653 shutdown_connections 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 -- 172.21.15.28:0/154929653 >> 172.21.15.28:0/154929653 conn(0x7ff2f406d4e0 msgr2=0x7ff2f406d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 -- 172.21.15.28:0/154929653 shutdown_connections 2024-09-13T17:20:45.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.914+0000 7ff2f9800640 1 -- 172.21.15.28:0/154929653 wait complete. 2024-09-13T17:20:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.916+0000 7ff2f9800640 1 Processor -- start 2024-09-13T17:20:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.916+0000 7ff2f9800640 1 -- start start 2024-09-13T17:20:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 0x7ff2f40caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f41556e0 0x7ff2f40cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f4159c60 con 0x7ff2d8004d00 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f40cac70 con 0x7ff2f4155210 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f9800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f415b980 con 0x7ff2f41556e0 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57222/0 (socket says 172.21.15.28:57222) 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 -- 172.21.15.28:0/3472533402 learned_addr learned my addr 172.21.15.28:0/3472533402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:20:45.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f8e00640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f41556e0 0x7ff2f40cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3400640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 0x7ff2f40caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 -- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f41556e0 msgr2=0x7ff2f40cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f41556e0 0x7ff2f40cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 -- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 msgr2=0x7ff2f40caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 0x7ff2f40caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:45.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3e00640 1 -- 172.21.15.28:0/3472533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f4159e50 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.917+0000 7ff2f3400640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 0x7ff2f40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2f3e00640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400fb30 tx=0x7ff2e4001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e40052e0 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e4005440 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e400d9b0 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f415a130 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.918+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f415a6f0 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.919+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f41607b0 con 0x7ff2d8004d00 2024-09-13T17:20:45.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.920+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff2e400db10 con 0x7ff2d8004d00 2024-09-13T17:20:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.921+0000 7ff2ebe00640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 0x7ff2d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:20:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.921+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2e4089500 con 0x7ff2d8004d00 2024-09-13T17:20:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.921+0000 7ff2f3400640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 0x7ff2d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:20:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.922+0000 7ff2f3400640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 0x7ff2d406d0c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406b720 tx=0x7ff2e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:20:45.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:45.925+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff2e40546d0 con 0x7ff2d8004d00 2024-09-13T17:20:46.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.136+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff2f415bbb0 con 0x7ff2d8004d00 2024-09-13T17:20:46.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.137+0000 7ff2ebe00640 1 -- 172.21.15.28:0/3472533402 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v151) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff2e4057cf0 con 0x7ff2d8004d00 2024-09-13T17:20:46.139 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:20:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 msgr2=0x7ff2d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 0x7ff2d406d0c0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7ff2f406b720 tx=0x7ff2e0009040 comp rx=0 tx=0).stop 2024-09-13T17:20:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 msgr2=0x7ff2f415b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:20:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400fb30 tx=0x7ff2e4001d00 comp rx=0 tx=0).stop 2024-09-13T17:20:46.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 shutdown_connections 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff2d406ac00 0x7ff2d406d0c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2f41556e0 0x7ff2f40cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2f4155210 0x7ff2f40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 --2- 172.21.15.28:0/3472533402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2d8004d00 0x7ff2f415b440 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 >> 172.21.15.28:0/3472533402 conn(0x7ff2f406d4e0 msgr2=0x7ff2f40b1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 shutdown_connections 2024-09-13T17:20:46.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:20:46.139+0000 7ff2f9800640 1 -- 172.21.15.28:0/3472533402 wait complete. 2024-09-13T17:20:46.156 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:20:46.156 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:20:46.157 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:20:46.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.157 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:20:46.193 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:20:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:20:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:20:46.193 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:20:46.194 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41935sec preferred_lft 41935sec 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:20:46.195 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:20:46.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.196 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:20:46.287 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:20:46.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.291 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:20:46.327 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:20:46.327 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:20:46.327 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:20:46.327 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:20:46.327 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:20:46.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:20:46.328 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:46.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.472 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:20:46.507 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:20:46.508 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:20:46.508 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:20:46.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.508 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:20:46.604 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:20:46.604 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:20:46.640 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:20:46.640 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:20:46.640 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:20:46.715 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:20:46.715 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.715 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:20:46.844 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:20:46.845 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:20:46.874 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:20:46.874 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:20:46.874 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:20:46.874 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:20:46.874 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:20:46.875 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:20:46.876 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:20:46.876 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:20:46.876 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:20:46.876 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:20:46.879 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:20:46.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.879 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:20:46.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:46.935 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:20:46.978 INFO:teuthology.orchestra.run.smithi028.stdout:client8525 2024-09-13T17:20:46.979 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:20:46.979 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:20:47.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:47.054 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:47.054 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:47.054 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.0 2024-09-13T17:20:47.054 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:47.055 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.0/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:47.055 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:47.055 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.161 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174773 s, 60.0 MB/s 2024-09-13T17:20:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155608 s, 67.4 MB/s 2024-09-13T17:20:47.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157699 s, 66.5 MB/s 2024-09-13T17:20:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163798 s, 64.0 MB/s 2024-09-13T17:20:47.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169474 s, 61.9 MB/s 2024-09-13T17:20:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.257 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185135 s, 56.6 MB/s 2024-09-13T17:20:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.277 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185121 s, 56.6 MB/s 2024-09-13T17:20:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.298 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184107 s, 57.0 MB/s 2024-09-13T17:20:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142783 s, 73.4 MB/s 2024-09-13T17:20:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151592 s, 69.2 MB/s 2024-09-13T17:20:47.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.350 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161061 s, 65.1 MB/s 2024-09-13T17:20:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128935 s, 81.3 MB/s 2024-09-13T17:20:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167059 s, 62.8 MB/s 2024-09-13T17:20:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127217 s, 82.4 MB/s 2024-09-13T17:20:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140906 s, 74.4 MB/s 2024-09-13T17:20:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153725 s, 68.2 MB/s 2024-09-13T17:20:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.449 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140347 s, 74.7 MB/s 2024-09-13T17:20:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.465 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135795 s, 77.2 MB/s 2024-09-13T17:20:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137453 s, 76.3 MB/s 2024-09-13T17:20:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137909 s, 76.0 MB/s 2024-09-13T17:20:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.515 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163748 s, 64.0 MB/s 2024-09-13T17:20:47.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158029 s, 66.4 MB/s 2024-09-13T17:20:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149528 s, 70.1 MB/s 2024-09-13T17:20:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.567 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138402 s, 75.8 MB/s 2024-09-13T17:20:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169433 s, 61.9 MB/s 2024-09-13T17:20:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133859 s, 78.3 MB/s 2024-09-13T17:20:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012898 s, 81.3 MB/s 2024-09-13T17:20:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.637 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171885 s, 61.0 MB/s 2024-09-13T17:20:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155828 s, 67.3 MB/s 2024-09-13T17:20:47.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.670 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130317 s, 80.5 MB/s 2024-09-13T17:20:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.689 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168829 s, 62.1 MB/s 2024-09-13T17:20:47.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.703 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.703 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122635 s, 85.5 MB/s 2024-09-13T17:20:47.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.1 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.1/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:47.705 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:47.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150202 s, 69.8 MB/s 2024-09-13T17:20:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142733 s, 73.5 MB/s 2024-09-13T17:20:47.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158436 s, 66.2 MB/s 2024-09-13T17:20:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.817 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176496 s, 59.4 MB/s 2024-09-13T17:20:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013944 s, 75.2 MB/s 2024-09-13T17:20:47.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013796 s, 76.0 MB/s 2024-09-13T17:20:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.865 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140172 s, 74.8 MB/s 2024-09-13T17:20:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134465 s, 78.0 MB/s 2024-09-13T17:20:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161356 s, 65.0 MB/s 2024-09-13T17:20:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.915 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140953 s, 74.4 MB/s 2024-09-13T17:20:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165025 s, 63.5 MB/s 2024-09-13T17:20:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.950 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139186 s, 75.3 MB/s 2024-09-13T17:20:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.966 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136648 s, 76.7 MB/s 2024-09-13T17:20:47.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.981 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127029 s, 82.5 MB/s 2024-09-13T17:20:47.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:47.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:47.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134241 s, 78.1 MB/s 2024-09-13T17:20:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.016 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.017 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017166 s, 61.1 MB/s 2024-09-13T17:20:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135964 s, 77.1 MB/s 2024-09-13T17:20:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133945 s, 78.3 MB/s 2024-09-13T17:20:48.064 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014997 s, 69.9 MB/s 2024-09-13T17:20:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.080 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.081 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134411 s, 78.0 MB/s 2024-09-13T17:20:48.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.096 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.096 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135472 s, 77.4 MB/s 2024-09-13T17:20:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133285 s, 78.7 MB/s 2024-09-13T17:20:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140889 s, 74.4 MB/s 2024-09-13T17:20:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165049 s, 63.5 MB/s 2024-09-13T17:20:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139076 s, 75.4 MB/s 2024-09-13T17:20:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121068 s, 86.6 MB/s 2024-09-13T17:20:48.196 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016611 s, 63.1 MB/s 2024-09-13T17:20:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160078 s, 65.5 MB/s 2024-09-13T17:20:48.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129999 s, 80.7 MB/s 2024-09-13T17:20:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140356 s, 74.7 MB/s 2024-09-13T17:20:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149822 s, 70.0 MB/s 2024-09-13T17:20:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.282 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164126 s, 63.9 MB/s 2024-09-13T17:20:48.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.2 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.2/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:48.284 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173578 s, 60.4 MB/s 2024-09-13T17:20:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132335 s, 79.2 MB/s 2024-09-13T17:20:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126935 s, 82.6 MB/s 2024-09-13T17:20:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130434 s, 80.4 MB/s 2024-09-13T17:20:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133224 s, 78.7 MB/s 2024-09-13T17:20:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154844 s, 67.7 MB/s 2024-09-13T17:20:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170858 s, 61.4 MB/s 2024-09-13T17:20:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121747 s, 86.1 MB/s 2024-09-13T17:20:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128644 s, 81.5 MB/s 2024-09-13T17:20:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.486 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132013 s, 79.4 MB/s 2024-09-13T17:20:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132894 s, 78.9 MB/s 2024-09-13T17:20:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131694 s, 79.6 MB/s 2024-09-13T17:20:48.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01343 s, 78.1 MB/s 2024-09-13T17:20:48.547 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.548 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126319 s, 83.0 MB/s 2024-09-13T17:20:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.562 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.563 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126585 s, 82.8 MB/s 2024-09-13T17:20:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128317 s, 81.7 MB/s 2024-09-13T17:20:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127757 s, 82.1 MB/s 2024-09-13T17:20:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160517 s, 65.3 MB/s 2024-09-13T17:20:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.627 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.628 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143795 s, 72.9 MB/s 2024-09-13T17:20:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127569 s, 82.2 MB/s 2024-09-13T17:20:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138507 s, 75.7 MB/s 2024-09-13T17:20:48.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158386 s, 66.2 MB/s 2024-09-13T17:20:48.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135144 s, 77.6 MB/s 2024-09-13T17:20:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160758 s, 65.2 MB/s 2024-09-13T17:20:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133748 s, 78.4 MB/s 2024-09-13T17:20:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.742 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013991 s, 74.9 MB/s 2024-09-13T17:20:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.761 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165619 s, 63.3 MB/s 2024-09-13T17:20:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136002 s, 77.1 MB/s 2024-09-13T17:20:48.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.793 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136085 s, 77.1 MB/s 2024-09-13T17:20:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.811 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156968 s, 66.8 MB/s 2024-09-13T17:20:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.826 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131718 s, 79.6 MB/s 2024-09-13T17:20:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128081 s, 81.9 MB/s 2024-09-13T17:20:48.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.3 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.3/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:48.843 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.898 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132662 s, 79.0 MB/s 2024-09-13T17:20:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.913 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.914 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139101 s, 75.4 MB/s 2024-09-13T17:20:48.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.931 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150492 s, 69.7 MB/s 2024-09-13T17:20:48.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.946 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01313 s, 79.9 MB/s 2024-09-13T17:20:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.965 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169942 s, 61.7 MB/s 2024-09-13T17:20:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.981 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136668 s, 76.7 MB/s 2024-09-13T17:20:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:48.997 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139256 s, 75.3 MB/s 2024-09-13T17:20:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136092 s, 77.0 MB/s 2024-09-13T17:20:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130666 s, 80.2 MB/s 2024-09-13T17:20:49.048 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.048 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.048 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171733 s, 61.1 MB/s 2024-09-13T17:20:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.064 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137539 s, 76.2 MB/s 2024-09-13T17:20:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.079 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131024 s, 80.0 MB/s 2024-09-13T17:20:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128547 s, 81.6 MB/s 2024-09-13T17:20:49.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.110 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135963 s, 77.1 MB/s 2024-09-13T17:20:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134194 s, 78.1 MB/s 2024-09-13T17:20:49.144 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.145 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168353 s, 62.3 MB/s 2024-09-13T17:20:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163342 s, 64.2 MB/s 2024-09-13T17:20:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132158 s, 79.3 MB/s 2024-09-13T17:20:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133467 s, 78.6 MB/s 2024-09-13T17:20:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128326 s, 81.7 MB/s 2024-09-13T17:20:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.224 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130026 s, 80.6 MB/s 2024-09-13T17:20:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.240 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131438 s, 79.8 MB/s 2024-09-13T17:20:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.254 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130943 s, 80.1 MB/s 2024-09-13T17:20:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137566 s, 76.2 MB/s 2024-09-13T17:20:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.286 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137479 s, 76.3 MB/s 2024-09-13T17:20:49.301 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129416 s, 81.0 MB/s 2024-09-13T17:20:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135662 s, 77.3 MB/s 2024-09-13T17:20:49.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.334 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139383 s, 75.2 MB/s 2024-09-13T17:20:49.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.349 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127561 s, 82.2 MB/s 2024-09-13T17:20:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132952 s, 78.9 MB/s 2024-09-13T17:20:49.379 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130018 s, 80.6 MB/s 2024-09-13T17:20:49.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.395 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132064 s, 79.4 MB/s 2024-09-13T17:20:49.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.4 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.4/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:49.397 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132604 s, 79.1 MB/s 2024-09-13T17:20:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.467 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126337 s, 83.0 MB/s 2024-09-13T17:20:49.482 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012894 s, 81.3 MB/s 2024-09-13T17:20:49.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.501 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.501 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016933 s, 61.9 MB/s 2024-09-13T17:20:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.523 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199746 s, 52.5 MB/s 2024-09-13T17:20:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171496 s, 61.1 MB/s 2024-09-13T17:20:49.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128772 s, 81.4 MB/s 2024-09-13T17:20:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136483 s, 76.8 MB/s 2024-09-13T17:20:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158538 s, 66.1 MB/s 2024-09-13T17:20:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116078 s, 90.3 MB/s 2024-09-13T17:20:49.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135555 s, 77.4 MB/s 2024-09-13T17:20:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129731 s, 80.8 MB/s 2024-09-13T17:20:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159845 s, 65.6 MB/s 2024-09-13T17:20:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160128 s, 65.5 MB/s 2024-09-13T17:20:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136197 s, 77.0 MB/s 2024-09-13T17:20:49.707 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169859 s, 61.7 MB/s 2024-09-13T17:20:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.723 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130723 s, 80.2 MB/s 2024-09-13T17:20:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.739 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135651 s, 77.3 MB/s 2024-09-13T17:20:49.755 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143201 s, 73.2 MB/s 2024-09-13T17:20:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135915 s, 77.1 MB/s 2024-09-13T17:20:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.790 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162274 s, 64.6 MB/s 2024-09-13T17:20:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.805 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130575 s, 80.3 MB/s 2024-09-13T17:20:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171975 s, 61.0 MB/s 2024-09-13T17:20:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163093 s, 64.3 MB/s 2024-09-13T17:20:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.858 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137022 s, 76.5 MB/s 2024-09-13T17:20:49.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143006 s, 73.3 MB/s 2024-09-13T17:20:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134652 s, 77.9 MB/s 2024-09-13T17:20:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.904 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120339 s, 87.1 MB/s 2024-09-13T17:20:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.921 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145266 s, 72.2 MB/s 2024-09-13T17:20:49.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.940 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161362 s, 65.0 MB/s 2024-09-13T17:20:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162031 s, 64.7 MB/s 2024-09-13T17:20:49.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:49.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:49.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131333 s, 79.8 MB/s 2024-09-13T17:20:49.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.5 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.5/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:49.975 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.030 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135409 s, 77.4 MB/s 2024-09-13T17:20:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.045 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130546 s, 80.3 MB/s 2024-09-13T17:20:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.060 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013153 s, 79.7 MB/s 2024-09-13T17:20:50.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.079 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016403 s, 63.9 MB/s 2024-09-13T17:20:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128359 s, 81.7 MB/s 2024-09-13T17:20:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129513 s, 81.0 MB/s 2024-09-13T17:20:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.128 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165145 s, 63.5 MB/s 2024-09-13T17:20:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.146 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160266 s, 65.4 MB/s 2024-09-13T17:20:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143799 s, 72.9 MB/s 2024-09-13T17:20:50.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013469 s, 77.9 MB/s 2024-09-13T17:20:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159254 s, 65.8 MB/s 2024-09-13T17:20:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.212 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131933 s, 79.5 MB/s 2024-09-13T17:20:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154748 s, 67.8 MB/s 2024-09-13T17:20:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134084 s, 78.2 MB/s 2024-09-13T17:20:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128902 s, 81.3 MB/s 2024-09-13T17:20:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.277 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137071 s, 76.5 MB/s 2024-09-13T17:20:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.293 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013858 s, 75.7 MB/s 2024-09-13T17:20:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.308 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131978 s, 79.5 MB/s 2024-09-13T17:20:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.324 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133332 s, 78.6 MB/s 2024-09-13T17:20:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.342 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159686 s, 65.7 MB/s 2024-09-13T17:20:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.357 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131544 s, 79.7 MB/s 2024-09-13T17:20:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.373 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132824 s, 78.9 MB/s 2024-09-13T17:20:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.388 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126816 s, 82.7 MB/s 2024-09-13T17:20:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.402 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118657 s, 88.4 MB/s 2024-09-13T17:20:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.417 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129632 s, 80.9 MB/s 2024-09-13T17:20:50.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.434 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140687 s, 74.5 MB/s 2024-09-13T17:20:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.448 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120611 s, 86.9 MB/s 2024-09-13T17:20:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.463 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01292 s, 81.2 MB/s 2024-09-13T17:20:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134564 s, 77.9 MB/s 2024-09-13T17:20:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.494 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134243 s, 78.1 MB/s 2024-09-13T17:20:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.509 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126542 s, 82.9 MB/s 2024-09-13T17:20:50.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.529 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165799 s, 63.2 MB/s 2024-09-13T17:20:50.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.6 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.6/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:50.531 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:50.532 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013254 s, 79.1 MB/s 2024-09-13T17:20:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120865 s, 86.8 MB/s 2024-09-13T17:20:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132603 s, 79.1 MB/s 2024-09-13T17:20:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.632 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134749 s, 77.8 MB/s 2024-09-13T17:20:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168994 s, 62.0 MB/s 2024-09-13T17:20:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.669 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157646 s, 66.5 MB/s 2024-09-13T17:20:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.684 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125212 s, 83.7 MB/s 2024-09-13T17:20:50.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.699 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130132 s, 80.6 MB/s 2024-09-13T17:20:50.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.715 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139293 s, 75.3 MB/s 2024-09-13T17:20:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127953 s, 82.0 MB/s 2024-09-13T17:20:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.748 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153834 s, 68.2 MB/s 2024-09-13T17:20:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125737 s, 83.4 MB/s 2024-09-13T17:20:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126061 s, 83.2 MB/s 2024-09-13T17:20:50.796 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166291 s, 63.1 MB/s 2024-09-13T17:20:50.812 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.812 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.812 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132831 s, 78.9 MB/s 2024-09-13T17:20:50.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.828 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134712 s, 77.8 MB/s 2024-09-13T17:20:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167966 s, 62.4 MB/s 2024-09-13T17:20:50.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.863 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.863 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140906 s, 74.4 MB/s 2024-09-13T17:20:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.878 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132682 s, 79.0 MB/s 2024-09-13T17:20:50.896 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.897 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015906 s, 65.9 MB/s 2024-09-13T17:20:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134758 s, 77.8 MB/s 2024-09-13T17:20:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.927 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127356 s, 82.3 MB/s 2024-09-13T17:20:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.943 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136029 s, 77.1 MB/s 2024-09-13T17:20:50.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.958 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128148 s, 81.8 MB/s 2024-09-13T17:20:50.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152786 s, 68.6 MB/s 2024-09-13T17:20:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127857 s, 82.0 MB/s 2024-09-13T17:20:51.011 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.011 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.011 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177068 s, 59.2 MB/s 2024-09-13T17:20:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.026 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132947 s, 78.9 MB/s 2024-09-13T17:20:51.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.045 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161472 s, 64.9 MB/s 2024-09-13T17:20:51.060 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.061 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.061 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133002 s, 78.8 MB/s 2024-09-13T17:20:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.076 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133651 s, 78.5 MB/s 2024-09-13T17:20:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154181 s, 68.0 MB/s 2024-09-13T17:20:51.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.7 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.7/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:51.102 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:51.182 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.183 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131322 s, 79.8 MB/s 2024-09-13T17:20:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137996 s, 76.0 MB/s 2024-09-13T17:20:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137577 s, 76.2 MB/s 2024-09-13T17:20:51.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133522 s, 78.5 MB/s 2024-09-13T17:20:51.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130158 s, 80.6 MB/s 2024-09-13T17:20:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164995 s, 63.6 MB/s 2024-09-13T17:20:51.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.279 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131372 s, 79.8 MB/s 2024-09-13T17:20:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.294 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134364 s, 78.0 MB/s 2024-09-13T17:20:51.309 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.310 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013231 s, 79.3 MB/s 2024-09-13T17:20:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165506 s, 63.4 MB/s 2024-09-13T17:20:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.347 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168798 s, 62.1 MB/s 2024-09-13T17:20:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.363 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136751 s, 76.7 MB/s 2024-09-13T17:20:51.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.378 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128575 s, 81.6 MB/s 2024-09-13T17:20:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.393 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134097 s, 78.2 MB/s 2024-09-13T17:20:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132231 s, 79.3 MB/s 2024-09-13T17:20:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132542 s, 79.1 MB/s 2024-09-13T17:20:51.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.440 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.440 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131314 s, 79.9 MB/s 2024-09-13T17:20:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.455 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126568 s, 82.8 MB/s 2024-09-13T17:20:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.473 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158364 s, 66.2 MB/s 2024-09-13T17:20:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130238 s, 80.5 MB/s 2024-09-13T17:20:51.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136109 s, 77.0 MB/s 2024-09-13T17:20:51.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.519 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133198 s, 78.7 MB/s 2024-09-13T17:20:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.537 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.538 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161468 s, 64.9 MB/s 2024-09-13T17:20:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110013 s, 95.3 MB/s 2024-09-13T17:20:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.567 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137747 s, 76.1 MB/s 2024-09-13T17:20:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166359 s, 63.0 MB/s 2024-09-13T17:20:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.601 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137279 s, 76.4 MB/s 2024-09-13T17:20:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137859 s, 76.1 MB/s 2024-09-13T17:20:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.634 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140255 s, 74.8 MB/s 2024-09-13T17:20:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.649 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126631 s, 82.8 MB/s 2024-09-13T17:20:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.665 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135331 s, 77.5 MB/s 2024-09-13T17:20:51.682 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.683 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157342 s, 66.6 MB/s 2024-09-13T17:20:51.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:51.684 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:51.684 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:51.684 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.8 2024-09-13T17:20:51.684 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:51.685 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.8/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:51.685 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:51.685 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:51.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169405 s, 61.9 MB/s 2024-09-13T17:20:51.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165279 s, 63.4 MB/s 2024-09-13T17:20:51.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125825 s, 83.3 MB/s 2024-09-13T17:20:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136149 s, 77.0 MB/s 2024-09-13T17:20:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.808 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131179 s, 79.9 MB/s 2024-09-13T17:20:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.825 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140173 s, 74.8 MB/s 2024-09-13T17:20:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.839 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.840 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126254 s, 83.1 MB/s 2024-09-13T17:20:51.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.858 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.858 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166665 s, 62.9 MB/s 2024-09-13T17:20:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.874 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137288 s, 76.4 MB/s 2024-09-13T17:20:51.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.889 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.889 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121699 s, 86.2 MB/s 2024-09-13T17:20:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123869 s, 84.7 MB/s 2024-09-13T17:20:51.922 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.923 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171387 s, 61.2 MB/s 2024-09-13T17:20:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160494 s, 65.3 MB/s 2024-09-13T17:20:51.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.955 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.956 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124701 s, 84.1 MB/s 2024-09-13T17:20:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.971 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137515 s, 76.3 MB/s 2024-09-13T17:20:51.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120734 s, 86.9 MB/s 2024-09-13T17:20:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.004 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162454 s, 64.5 MB/s 2024-09-13T17:20:52.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135533 s, 77.4 MB/s 2024-09-13T17:20:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170793 s, 61.4 MB/s 2024-09-13T17:20:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136024 s, 77.1 MB/s 2024-09-13T17:20:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121888 s, 86.0 MB/s 2024-09-13T17:20:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129832 s, 80.8 MB/s 2024-09-13T17:20:52.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.101 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.101 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013702 s, 76.5 MB/s 2024-09-13T17:20:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.119 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01646 s, 63.7 MB/s 2024-09-13T17:20:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.133 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114337 s, 91.7 MB/s 2024-09-13T17:20:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.149 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138524 s, 75.7 MB/s 2024-09-13T17:20:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135416 s, 77.4 MB/s 2024-09-13T17:20:52.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126938 s, 82.6 MB/s 2024-09-13T17:20:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130908 s, 80.1 MB/s 2024-09-13T17:20:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.211 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135433 s, 77.4 MB/s 2024-09-13T17:20:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.228 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137543 s, 76.2 MB/s 2024-09-13T17:20:52.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.245 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153589 s, 68.3 MB/s 2024-09-13T17:20:52.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:52.246 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:52.246 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:52.246 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.9 2024-09-13T17:20:52.246 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:52.247 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.9/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:52.247 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:52.247 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:52.304 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164009 s, 63.9 MB/s 2024-09-13T17:20:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129423 s, 81.0 MB/s 2024-09-13T17:20:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.335 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126265 s, 83.0 MB/s 2024-09-13T17:20:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.352 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138155 s, 75.9 MB/s 2024-09-13T17:20:52.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013325 s, 78.7 MB/s 2024-09-13T17:20:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170735 s, 61.4 MB/s 2024-09-13T17:20:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.404 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162016 s, 64.7 MB/s 2024-09-13T17:20:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139351 s, 75.2 MB/s 2024-09-13T17:20:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.437 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.438 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148919 s, 70.4 MB/s 2024-09-13T17:20:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.454 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136906 s, 76.6 MB/s 2024-09-13T17:20:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.469 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135405 s, 77.4 MB/s 2024-09-13T17:20:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.489 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170314 s, 61.6 MB/s 2024-09-13T17:20:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.504 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133351 s, 78.6 MB/s 2024-09-13T17:20:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108729 s, 96.4 MB/s 2024-09-13T17:20:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01347 s, 77.8 MB/s 2024-09-13T17:20:52.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.552 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.552 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169825 s, 61.7 MB/s 2024-09-13T17:20:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.568 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137606 s, 76.2 MB/s 2024-09-13T17:20:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.584 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135338 s, 77.5 MB/s 2024-09-13T17:20:52.601 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154968 s, 67.7 MB/s 2024-09-13T17:20:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.616 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124251 s, 84.4 MB/s 2024-09-13T17:20:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.630 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.631 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116884 s, 89.7 MB/s 2024-09-13T17:20:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137552 s, 76.2 MB/s 2024-09-13T17:20:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.661 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012279 s, 85.4 MB/s 2024-09-13T17:20:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129427 s, 81.0 MB/s 2024-09-13T17:20:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133931 s, 78.3 MB/s 2024-09-13T17:20:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129267 s, 81.1 MB/s 2024-09-13T17:20:52.722 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.723 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.723 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137632 s, 76.2 MB/s 2024-09-13T17:20:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.738 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135291 s, 77.5 MB/s 2024-09-13T17:20:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136359 s, 76.9 MB/s 2024-09-13T17:20:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143659 s, 73.0 MB/s 2024-09-13T17:20:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124157 s, 84.5 MB/s 2024-09-13T17:20:52.800 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.801 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127676 s, 82.1 MB/s 2024-09-13T17:20:52.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:52.802 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:52.802 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:52.803 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.10 2024-09-13T17:20:52.803 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:52.803 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.10/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:52.803 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:52.803 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:52.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159742 s, 65.6 MB/s 2024-09-13T17:20:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136373 s, 76.9 MB/s 2024-09-13T17:20:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132789 s, 79.0 MB/s 2024-09-13T17:20:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141625 s, 74.0 MB/s 2024-09-13T17:20:52.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.923 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.923 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128576 s, 81.6 MB/s 2024-09-13T17:20:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.938 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134081 s, 78.2 MB/s 2024-09-13T17:20:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161164 s, 65.1 MB/s 2024-09-13T17:20:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133983 s, 78.3 MB/s 2024-09-13T17:20:52.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:52.988 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130832 s, 80.1 MB/s 2024-09-13T17:20:53.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.004 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.004 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137448 s, 76.3 MB/s 2024-09-13T17:20:53.022 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.023 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016671 s, 62.9 MB/s 2024-09-13T17:20:53.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117904 s, 88.9 MB/s 2024-09-13T17:20:53.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131388 s, 79.8 MB/s 2024-09-13T17:20:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130901 s, 80.1 MB/s 2024-09-13T17:20:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.081 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118641 s, 88.4 MB/s 2024-09-13T17:20:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.097 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137197 s, 76.4 MB/s 2024-09-13T17:20:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.111 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116978 s, 89.6 MB/s 2024-09-13T17:20:53.129 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.130 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163882 s, 64.0 MB/s 2024-09-13T17:20:53.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153646 s, 68.2 MB/s 2024-09-13T17:20:53.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.164 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141215 s, 74.3 MB/s 2024-09-13T17:20:53.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139861 s, 75.0 MB/s 2024-09-13T17:20:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135443 s, 77.4 MB/s 2024-09-13T17:20:53.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129689 s, 80.9 MB/s 2024-09-13T17:20:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.230 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168322 s, 62.3 MB/s 2024-09-13T17:20:53.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.244 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116425 s, 90.1 MB/s 2024-09-13T17:20:53.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.259 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.259 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136422 s, 76.9 MB/s 2024-09-13T17:20:53.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.274 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.275 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130126 s, 80.6 MB/s 2024-09-13T17:20:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.290 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126777 s, 82.7 MB/s 2024-09-13T17:20:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132632 s, 79.1 MB/s 2024-09-13T17:20:53.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.320 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.320 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012983 s, 80.8 MB/s 2024-09-13T17:20:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.336 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134106 s, 78.2 MB/s 2024-09-13T17:20:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.352 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135609 s, 77.3 MB/s 2024-09-13T17:20:53.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:53.353 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:53.353 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:53.354 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.11 2024-09-13T17:20:53.354 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:53.354 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.11/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:53.354 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:53.354 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156186 s, 67.1 MB/s 2024-09-13T17:20:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.426 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133638 s, 78.5 MB/s 2024-09-13T17:20:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.441 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129914 s, 80.7 MB/s 2024-09-13T17:20:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135332 s, 77.5 MB/s 2024-09-13T17:20:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171592 s, 61.1 MB/s 2024-09-13T17:20:53.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.495 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163049 s, 64.3 MB/s 2024-09-13T17:20:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.512 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01464 s, 71.6 MB/s 2024-09-13T17:20:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132746 s, 79.0 MB/s 2024-09-13T17:20:53.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134932 s, 77.7 MB/s 2024-09-13T17:20:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.558 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128836 s, 81.4 MB/s 2024-09-13T17:20:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160806 s, 65.2 MB/s 2024-09-13T17:20:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153435 s, 68.3 MB/s 2024-09-13T17:20:53.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132188 s, 79.3 MB/s 2024-09-13T17:20:53.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149933 s, 69.9 MB/s 2024-09-13T17:20:53.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134084 s, 78.2 MB/s 2024-09-13T17:20:53.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127784 s, 82.1 MB/s 2024-09-13T17:20:53.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.672 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.672 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133331 s, 78.6 MB/s 2024-09-13T17:20:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161226 s, 65.0 MB/s 2024-09-13T17:20:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130292 s, 80.5 MB/s 2024-09-13T17:20:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.725 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165754 s, 63.3 MB/s 2024-09-13T17:20:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128138 s, 81.8 MB/s 2024-09-13T17:20:53.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.756 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.756 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014244 s, 73.6 MB/s 2024-09-13T17:20:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.773 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014551 s, 72.1 MB/s 2024-09-13T17:20:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155663 s, 67.4 MB/s 2024-09-13T17:20:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.809 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154162 s, 68.0 MB/s 2024-09-13T17:20:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.825 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140658 s, 74.5 MB/s 2024-09-13T17:20:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.841 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143192 s, 73.2 MB/s 2024-09-13T17:20:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.857 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132641 s, 79.1 MB/s 2024-09-13T17:20:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133408 s, 78.6 MB/s 2024-09-13T17:20:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125333 s, 83.7 MB/s 2024-09-13T17:20:53.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.903 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131026 s, 80.0 MB/s 2024-09-13T17:20:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.918 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132126 s, 79.4 MB/s 2024-09-13T17:20:53.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.12 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.12/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:53.920 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117508 s, 89.2 MB/s 2024-09-13T17:20:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:53.989 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135156 s, 77.6 MB/s 2024-09-13T17:20:54.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.007 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.007 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160972 s, 65.1 MB/s 2024-09-13T17:20:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.025 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158694 s, 66.1 MB/s 2024-09-13T17:20:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.041 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139038 s, 75.4 MB/s 2024-09-13T17:20:54.059 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.060 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162912 s, 64.4 MB/s 2024-09-13T17:20:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.075 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136271 s, 76.9 MB/s 2024-09-13T17:20:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161736 s, 64.8 MB/s 2024-09-13T17:20:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132511 s, 79.1 MB/s 2024-09-13T17:20:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.125 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138853 s, 75.5 MB/s 2024-09-13T17:20:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.141 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128987 s, 81.3 MB/s 2024-09-13T17:20:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182546 s, 57.4 MB/s 2024-09-13T17:20:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.183 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018688 s, 56.1 MB/s 2024-09-13T17:20:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.201 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156289 s, 67.1 MB/s 2024-09-13T17:20:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.216 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130079 s, 80.6 MB/s 2024-09-13T17:20:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136501 s, 76.8 MB/s 2024-09-13T17:20:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.247 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131097 s, 80.0 MB/s 2024-09-13T17:20:54.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133087 s, 78.8 MB/s 2024-09-13T17:20:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131412 s, 79.8 MB/s 2024-09-13T17:20:54.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.297 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.297 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161485 s, 64.9 MB/s 2024-09-13T17:20:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134001 s, 78.3 MB/s 2024-09-13T17:20:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135441 s, 77.4 MB/s 2024-09-13T17:20:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135126 s, 77.6 MB/s 2024-09-13T17:20:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130246 s, 80.5 MB/s 2024-09-13T17:20:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.378 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167635 s, 62.6 MB/s 2024-09-13T17:20:54.393 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.394 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.394 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131333 s, 79.8 MB/s 2024-09-13T17:20:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134128 s, 78.2 MB/s 2024-09-13T17:20:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130966 s, 80.1 MB/s 2024-09-13T17:20:54.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119115 s, 88.0 MB/s 2024-09-13T17:20:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:54.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161588 s, 64.9 MB/s 2024-09-13T17:20:55.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.773202 s, 1.4 MB/s 2024-09-13T17:20:55.252 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176216 s, 59.5 MB/s 2024-09-13T17:20:55.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.13 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.13/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:55.255 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014897 s, 70.4 MB/s 2024-09-13T17:20:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.330 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.331 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132348 s, 79.2 MB/s 2024-09-13T17:20:55.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.350 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168138 s, 62.4 MB/s 2024-09-13T17:20:55.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159888 s, 65.6 MB/s 2024-09-13T17:20:55.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137042 s, 76.5 MB/s 2024-09-13T17:20:55.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.399 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.399 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131217 s, 79.9 MB/s 2024-09-13T17:20:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131806 s, 79.6 MB/s 2024-09-13T17:20:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.431 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141589 s, 74.1 MB/s 2024-09-13T17:20:55.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.445 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.445 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118219 s, 88.7 MB/s 2024-09-13T17:20:55.460 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.461 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135622 s, 77.3 MB/s 2024-09-13T17:20:55.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.476 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.476 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137413 s, 76.3 MB/s 2024-09-13T17:20:55.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.495 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.495 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158733 s, 66.1 MB/s 2024-09-13T17:20:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.514 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168084 s, 62.4 MB/s 2024-09-13T17:20:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.527 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112173 s, 93.5 MB/s 2024-09-13T17:20:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.545 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016079 s, 65.2 MB/s 2024-09-13T17:20:55.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013253 s, 79.1 MB/s 2024-09-13T17:20:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134753 s, 77.8 MB/s 2024-09-13T17:20:55.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.591 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120065 s, 87.3 MB/s 2024-09-13T17:20:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.605 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130566 s, 80.3 MB/s 2024-09-13T17:20:55.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.621 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133469 s, 78.6 MB/s 2024-09-13T17:20:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.638 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.639 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151106 s, 69.4 MB/s 2024-09-13T17:20:55.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.657 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154073 s, 68.1 MB/s 2024-09-13T17:20:55.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169405 s, 61.9 MB/s 2024-09-13T17:20:55.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.692 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.692 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139134 s, 75.4 MB/s 2024-09-13T17:20:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135407 s, 77.4 MB/s 2024-09-13T17:20:55.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.728 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174546 s, 60.1 MB/s 2024-09-13T17:20:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.743 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127295 s, 82.4 MB/s 2024-09-13T17:20:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.761 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166055 s, 63.1 MB/s 2024-09-13T17:20:55.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.777 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130641 s, 80.3 MB/s 2024-09-13T17:20:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.792 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131255 s, 79.9 MB/s 2024-09-13T17:20:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.810 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155803 s, 67.3 MB/s 2024-09-13T17:20:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.825 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131485 s, 79.7 MB/s 2024-09-13T17:20:55.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.14 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.14/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:55.827 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.883 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144214 s, 72.7 MB/s 2024-09-13T17:20:55.898 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.899 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136103 s, 77.0 MB/s 2024-09-13T17:20:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.917 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162452 s, 64.5 MB/s 2024-09-13T17:20:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.936 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016206 s, 64.7 MB/s 2024-09-13T17:20:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166472 s, 63.0 MB/s 2024-09-13T17:20:55.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131008 s, 80.0 MB/s 2024-09-13T17:20:55.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:55.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:55.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133363 s, 78.6 MB/s 2024-09-13T17:20:56.000 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130989 s, 80.1 MB/s 2024-09-13T17:20:56.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.019 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.019 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161236 s, 65.0 MB/s 2024-09-13T17:20:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131228 s, 79.9 MB/s 2024-09-13T17:20:56.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.052 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.052 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157779 s, 66.5 MB/s 2024-09-13T17:20:56.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128584 s, 81.5 MB/s 2024-09-13T17:20:56.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.082 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129517 s, 81.0 MB/s 2024-09-13T17:20:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.098 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135481 s, 77.4 MB/s 2024-09-13T17:20:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.117 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164994 s, 63.6 MB/s 2024-09-13T17:20:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.132 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131711 s, 79.6 MB/s 2024-09-13T17:20:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127492 s, 82.2 MB/s 2024-09-13T17:20:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.163 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130626 s, 80.3 MB/s 2024-09-13T17:20:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125865 s, 83.3 MB/s 2024-09-13T17:20:56.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136921 s, 76.6 MB/s 2024-09-13T17:20:56.209 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138882 s, 75.5 MB/s 2024-09-13T17:20:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128489 s, 81.6 MB/s 2024-09-13T17:20:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.239 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.240 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127791 s, 82.1 MB/s 2024-09-13T17:20:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.255 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131312 s, 79.9 MB/s 2024-09-13T17:20:56.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140236 s, 74.8 MB/s 2024-09-13T17:20:56.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134832 s, 77.8 MB/s 2024-09-13T17:20:56.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.305 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.305 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158711 s, 66.1 MB/s 2024-09-13T17:20:56.321 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.321 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.321 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012898 s, 81.3 MB/s 2024-09-13T17:20:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160993 s, 65.1 MB/s 2024-09-13T17:20:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128752 s, 81.4 MB/s 2024-09-13T17:20:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126439 s, 82.9 MB/s 2024-09-13T17:20:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.385 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140884 s, 74.4 MB/s 2024-09-13T17:20:56.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:56.415 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:56.415 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:56.416 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.15 2024-09-13T17:20:56.416 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:56.416 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.15/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:56.416 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:56.416 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:56.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.471 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.471 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014262 s, 73.5 MB/s 2024-09-13T17:20:56.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.487 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133652 s, 78.5 MB/s 2024-09-13T17:20:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129133 s, 81.2 MB/s 2024-09-13T17:20:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127793 s, 82.1 MB/s 2024-09-13T17:20:56.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.532 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.532 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128296 s, 81.7 MB/s 2024-09-13T17:20:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163118 s, 64.3 MB/s 2024-09-13T17:20:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.568 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153059 s, 68.5 MB/s 2024-09-13T17:20:56.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.587 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.587 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016523 s, 63.5 MB/s 2024-09-13T17:20:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128743 s, 81.4 MB/s 2024-09-13T17:20:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.617 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132459 s, 79.2 MB/s 2024-09-13T17:20:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138301 s, 75.8 MB/s 2024-09-13T17:20:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011349 s, 92.4 MB/s 2024-09-13T17:20:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013034 s, 80.4 MB/s 2024-09-13T17:20:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.677 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130848 s, 80.1 MB/s 2024-09-13T17:20:56.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.693 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.693 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133935 s, 78.3 MB/s 2024-09-13T17:20:56.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.712 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167358 s, 62.7 MB/s 2024-09-13T17:20:56.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.728 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137204 s, 76.4 MB/s 2024-09-13T17:20:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141126 s, 74.3 MB/s 2024-09-13T17:20:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.762 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152691 s, 68.7 MB/s 2024-09-13T17:20:56.777 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.778 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136244 s, 77.0 MB/s 2024-09-13T17:20:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122255 s, 85.8 MB/s 2024-09-13T17:20:56.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.809 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140947 s, 74.4 MB/s 2024-09-13T17:20:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130046 s, 80.6 MB/s 2024-09-13T17:20:56.842 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.843 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165254 s, 63.5 MB/s 2024-09-13T17:20:56.861 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167072 s, 62.8 MB/s 2024-09-13T17:20:56.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.877 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122898 s, 85.3 MB/s 2024-09-13T17:20:56.893 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.894 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151705 s, 69.1 MB/s 2024-09-13T17:20:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.912 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165632 s, 63.3 MB/s 2024-09-13T17:20:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.927 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125912 s, 83.3 MB/s 2024-09-13T17:20:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.943 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.944 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139764 s, 75.0 MB/s 2024-09-13T17:20:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.960 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145302 s, 72.2 MB/s 2024-09-13T17:20:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:56.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140074 s, 74.9 MB/s 2024-09-13T17:20:56.977 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:56.977 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:56.977 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:56.978 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.16 2024-09-13T17:20:56.978 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:56.978 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.16/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:56.978 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:56.978 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:57.031 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.032 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127439 s, 82.3 MB/s 2024-09-13T17:20:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.047 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133622 s, 78.5 MB/s 2024-09-13T17:20:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.065 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133017 s, 78.8 MB/s 2024-09-13T17:20:57.078 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.079 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134299 s, 78.1 MB/s 2024-09-13T17:20:57.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.093 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.093 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126479 s, 82.9 MB/s 2024-09-13T17:20:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.109 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137021 s, 76.5 MB/s 2024-09-13T17:20:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.125 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.126 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139539 s, 75.1 MB/s 2024-09-13T17:20:57.141 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.142 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138456 s, 75.7 MB/s 2024-09-13T17:20:57.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.158 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140048 s, 74.9 MB/s 2024-09-13T17:20:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.174 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.175 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014337 s, 73.1 MB/s 2024-09-13T17:20:57.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.190 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.190 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132568 s, 79.1 MB/s 2024-09-13T17:20:57.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129042 s, 81.3 MB/s 2024-09-13T17:20:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.221 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135063 s, 77.6 MB/s 2024-09-13T17:20:57.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.237 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.237 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136362 s, 76.9 MB/s 2024-09-13T17:20:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.253 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138687 s, 75.6 MB/s 2024-09-13T17:20:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.268 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132613 s, 79.1 MB/s 2024-09-13T17:20:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.288 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171103 s, 61.3 MB/s 2024-09-13T17:20:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129369 s, 81.1 MB/s 2024-09-13T17:20:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.317 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124793 s, 84.0 MB/s 2024-09-13T17:20:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.335 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157213 s, 66.7 MB/s 2024-09-13T17:20:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.351 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013003 s, 80.6 MB/s 2024-09-13T17:20:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.370 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.371 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169529 s, 61.9 MB/s 2024-09-13T17:20:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.386 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135556 s, 77.4 MB/s 2024-09-13T17:20:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.401 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130641 s, 80.3 MB/s 2024-09-13T17:20:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168512 s, 62.2 MB/s 2024-09-13T17:20:57.438 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.439 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164328 s, 63.8 MB/s 2024-09-13T17:20:57.453 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.454 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012653 s, 82.9 MB/s 2024-09-13T17:20:57.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.468 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123291 s, 85.0 MB/s 2024-09-13T17:20:57.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.484 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013464 s, 77.9 MB/s 2024-09-13T17:20:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.499 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134884 s, 77.7 MB/s 2024-09-13T17:20:57.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.518 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161129 s, 65.1 MB/s 2024-09-13T17:20:57.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.536 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159279 s, 65.8 MB/s 2024-09-13T17:20:57.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:57.538 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:57.538 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:57.538 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.17 2024-09-13T17:20:57.538 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:57.539 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.17/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:57.539 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:57.539 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:57.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.596 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.596 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015844 s, 66.2 MB/s 2024-09-13T17:20:57.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140854 s, 74.4 MB/s 2024-09-13T17:20:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147251 s, 71.2 MB/s 2024-09-13T17:20:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.644 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129047 s, 81.3 MB/s 2024-09-13T17:20:57.659 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135077 s, 77.6 MB/s 2024-09-13T17:20:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.676 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124203 s, 84.4 MB/s 2024-09-13T17:20:57.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.690 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.690 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139403 s, 75.2 MB/s 2024-09-13T17:20:57.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.705 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.706 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012726 s, 82.4 MB/s 2024-09-13T17:20:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.721 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137397 s, 76.3 MB/s 2024-09-13T17:20:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115972 s, 90.4 MB/s 2024-09-13T17:20:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134665 s, 77.9 MB/s 2024-09-13T17:20:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.766 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130154 s, 80.6 MB/s 2024-09-13T17:20:57.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.781 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.781 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127819 s, 82.0 MB/s 2024-09-13T17:20:57.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140384 s, 74.7 MB/s 2024-09-13T17:20:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131976 s, 79.5 MB/s 2024-09-13T17:20:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.828 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012659 s, 82.8 MB/s 2024-09-13T17:20:57.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.843 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131284 s, 79.9 MB/s 2024-09-13T17:20:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.860 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139256 s, 75.3 MB/s 2024-09-13T17:20:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.875 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132312 s, 79.3 MB/s 2024-09-13T17:20:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.890 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.891 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128117 s, 81.8 MB/s 2024-09-13T17:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.906 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130093 s, 80.6 MB/s 2024-09-13T17:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.920 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124277 s, 84.4 MB/s 2024-09-13T17:20:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161713 s, 64.8 MB/s 2024-09-13T17:20:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127594 s, 82.2 MB/s 2024-09-13T17:20:57.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.970 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.970 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135862 s, 77.2 MB/s 2024-09-13T17:20:57.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:57.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:57.985 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132779 s, 79.0 MB/s 2024-09-13T17:20:58.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.001 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132659 s, 79.0 MB/s 2024-09-13T17:20:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.020 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167909 s, 62.4 MB/s 2024-09-13T17:20:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.035 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129988 s, 80.7 MB/s 2024-09-13T17:20:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168732 s, 62.1 MB/s 2024-09-13T17:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129633 s, 80.9 MB/s 2024-09-13T17:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.085 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137982 s, 76.0 MB/s 2024-09-13T17:20:58.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.18 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.18/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:58.087 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:58.139 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.140 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110362 s, 95.0 MB/s 2024-09-13T17:20:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.158 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016555 s, 63.3 MB/s 2024-09-13T17:20:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016212 s, 64.7 MB/s 2024-09-13T17:20:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.192 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134101 s, 78.2 MB/s 2024-09-13T17:20:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159379 s, 65.8 MB/s 2024-09-13T17:20:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.225 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124476 s, 84.2 MB/s 2024-09-13T17:20:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.241 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137506 s, 76.3 MB/s 2024-09-13T17:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.256 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013183 s, 79.5 MB/s 2024-09-13T17:20:58.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.271 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.271 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124994 s, 83.9 MB/s 2024-09-13T17:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.287 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140337 s, 74.7 MB/s 2024-09-13T17:20:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012984 s, 80.8 MB/s 2024-09-13T17:20:58.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.318 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.318 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133421 s, 78.6 MB/s 2024-09-13T17:20:58.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.334 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132838 s, 78.9 MB/s 2024-09-13T17:20:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128263 s, 81.8 MB/s 2024-09-13T17:20:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132702 s, 79.0 MB/s 2024-09-13T17:20:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140141 s, 74.8 MB/s 2024-09-13T17:20:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130458 s, 80.4 MB/s 2024-09-13T17:20:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.411 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129961 s, 80.7 MB/s 2024-09-13T17:20:58.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.427 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.427 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013426 s, 78.1 MB/s 2024-09-13T17:20:58.442 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.442 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.442 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131265 s, 79.9 MB/s 2024-09-13T17:20:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130249 s, 80.5 MB/s 2024-09-13T17:20:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132396 s, 79.2 MB/s 2024-09-13T17:20:58.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.487 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.487 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120573 s, 87.0 MB/s 2024-09-13T17:20:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.506 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165557 s, 63.3 MB/s 2024-09-13T17:20:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140096 s, 74.8 MB/s 2024-09-13T17:20:58.541 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.542 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.542 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172038 s, 61.0 MB/s 2024-09-13T17:20:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.560 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163605 s, 64.1 MB/s 2024-09-13T17:20:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138597 s, 75.7 MB/s 2024-09-13T17:20:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.594 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158151 s, 66.3 MB/s 2024-09-13T17:20:58.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.611 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138513 s, 75.7 MB/s 2024-09-13T17:20:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135563 s, 77.3 MB/s 2024-09-13T17:20:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.642 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133903 s, 78.3 MB/s 2024-09-13T17:20:58.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:58.643 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:58.643 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:58.643 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.19 2024-09-13T17:20:58.643 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:58.644 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.19/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:58.644 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:58.644 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.701 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01606 s, 65.3 MB/s 2024-09-13T17:20:58.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.716 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.716 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130591 s, 80.3 MB/s 2024-09-13T17:20:58.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.735 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162347 s, 64.6 MB/s 2024-09-13T17:20:58.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.750 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129269 s, 81.1 MB/s 2024-09-13T17:20:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.765 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126701 s, 82.8 MB/s 2024-09-13T17:20:58.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162903 s, 64.4 MB/s 2024-09-13T17:20:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.798 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013147 s, 79.8 MB/s 2024-09-13T17:20:58.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126443 s, 82.9 MB/s 2024-09-13T17:20:58.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.828 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.829 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130507 s, 80.3 MB/s 2024-09-13T17:20:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.847 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168314 s, 62.3 MB/s 2024-09-13T17:20:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.864 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143779 s, 72.9 MB/s 2024-09-13T17:20:58.879 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.880 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134406 s, 78.0 MB/s 2024-09-13T17:20:58.899 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177321 s, 59.1 MB/s 2024-09-13T17:20:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138739 s, 75.6 MB/s 2024-09-13T17:20:58.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.931 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.932 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135715 s, 77.3 MB/s 2024-09-13T17:20:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.950 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.951 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166853 s, 62.8 MB/s 2024-09-13T17:20:58.966 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.967 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.967 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138063 s, 75.9 MB/s 2024-09-13T17:20:58.981 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.982 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.982 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127601 s, 82.2 MB/s 2024-09-13T17:20:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:58.998 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138384 s, 75.8 MB/s 2024-09-13T17:20:59.012 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.013 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.013 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127221 s, 82.4 MB/s 2024-09-13T17:20:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.028 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132129 s, 79.4 MB/s 2024-09-13T17:20:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.043 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127405 s, 82.3 MB/s 2024-09-13T17:20:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.062 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166578 s, 62.9 MB/s 2024-09-13T17:20:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.077 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.078 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134348 s, 78.0 MB/s 2024-09-13T17:20:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.091 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113741 s, 92.2 MB/s 2024-09-13T17:20:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.108 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129822 s, 80.8 MB/s 2024-09-13T17:20:59.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.121 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.121 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127201 s, 82.4 MB/s 2024-09-13T17:20:59.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129918 s, 80.7 MB/s 2024-09-13T17:20:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.152 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133059 s, 78.8 MB/s 2024-09-13T17:20:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.168 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134808 s, 77.8 MB/s 2024-09-13T17:20:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165103 s, 63.5 MB/s 2024-09-13T17:20:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.205 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165133 s, 63.5 MB/s 2024-09-13T17:20:59.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.20 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.20/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:59.207 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:59.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138989 s, 75.4 MB/s 2024-09-13T17:20:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.278 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133892 s, 78.3 MB/s 2024-09-13T17:20:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.294 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133916 s, 78.3 MB/s 2024-09-13T17:20:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.312 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016099 s, 65.1 MB/s 2024-09-13T17:20:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.328 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136893 s, 76.6 MB/s 2024-09-13T17:20:59.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.343 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.343 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131216 s, 79.9 MB/s 2024-09-13T17:20:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.359 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013467 s, 77.9 MB/s 2024-09-13T17:20:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131908 s, 79.5 MB/s 2024-09-13T17:20:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135635 s, 77.3 MB/s 2024-09-13T17:20:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.406 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133687 s, 78.4 MB/s 2024-09-13T17:20:59.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.421 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.421 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133135 s, 78.8 MB/s 2024-09-13T17:20:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.436 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013074 s, 80.2 MB/s 2024-09-13T17:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138193 s, 75.9 MB/s 2024-09-13T17:20:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.468 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135026 s, 77.7 MB/s 2024-09-13T17:20:59.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.481 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.481 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109761 s, 95.5 MB/s 2024-09-13T17:20:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.496 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.497 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129116 s, 81.2 MB/s 2024-09-13T17:20:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018166 s, 57.7 MB/s 2024-09-13T17:20:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.539 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197604 s, 53.1 MB/s 2024-09-13T17:20:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.559 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173205 s, 60.5 MB/s 2024-09-13T17:20:59.577 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.578 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015312 s, 68.5 MB/s 2024-09-13T17:20:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135295 s, 77.5 MB/s 2024-09-13T17:20:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164288 s, 63.8 MB/s 2024-09-13T17:20:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157242 s, 66.7 MB/s 2024-09-13T17:20:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.645 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130357 s, 80.4 MB/s 2024-09-13T17:20:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.660 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131474 s, 79.8 MB/s 2024-09-13T17:20:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.680 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169412 s, 61.9 MB/s 2024-09-13T17:20:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139258 s, 75.3 MB/s 2024-09-13T17:20:59.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133628 s, 78.5 MB/s 2024-09-13T17:20:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.727 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138546 s, 75.7 MB/s 2024-09-13T17:20:59.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.745 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.746 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157492 s, 66.6 MB/s 2024-09-13T17:20:59.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.761 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133999 s, 78.3 MB/s 2024-09-13T17:20:59.775 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.776 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127491 s, 82.2 MB/s 2024-09-13T17:20:59.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.21 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.21/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:20:59.778 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:20:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128715 s, 81.5 MB/s 2024-09-13T17:20:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.849 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014003 s, 74.9 MB/s 2024-09-13T17:20:59.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.865 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.865 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139974 s, 74.9 MB/s 2024-09-13T17:20:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.880 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130623 s, 80.3 MB/s 2024-09-13T17:20:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133291 s, 78.7 MB/s 2024-09-13T17:20:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.910 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122661 s, 85.5 MB/s 2024-09-13T17:20:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.928 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131613 s, 79.7 MB/s 2024-09-13T17:20:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.942 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137917 s, 76.0 MB/s 2024-09-13T17:20:59.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127324 s, 82.4 MB/s 2024-09-13T17:20:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.975 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159818 s, 65.6 MB/s 2024-09-13T17:20:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:20:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:20:59.991 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133142 s, 78.8 MB/s 2024-09-13T17:21:00.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.006 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.006 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129771 s, 80.8 MB/s 2024-09-13T17:21:00.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.021 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129496 s, 81.0 MB/s 2024-09-13T17:21:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013089 s, 80.1 MB/s 2024-09-13T17:21:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.053 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142793 s, 73.4 MB/s 2024-09-13T17:21:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116636 s, 89.9 MB/s 2024-09-13T17:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127864 s, 82.0 MB/s 2024-09-13T17:21:00.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.099 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152217 s, 68.9 MB/s 2024-09-13T17:21:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.118 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165636 s, 63.3 MB/s 2024-09-13T17:21:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.136 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161328 s, 65.0 MB/s 2024-09-13T17:21:00.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.151 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.151 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126459 s, 82.9 MB/s 2024-09-13T17:21:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.168 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.169 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013818 s, 75.9 MB/s 2024-09-13T17:21:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.184 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013337 s, 78.6 MB/s 2024-09-13T17:21:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.199 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013082 s, 80.2 MB/s 2024-09-13T17:21:00.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.215 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.215 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136277 s, 76.9 MB/s 2024-09-13T17:21:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.232 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140322 s, 74.7 MB/s 2024-09-13T17:21:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132567 s, 79.1 MB/s 2024-09-13T17:21:00.262 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133236 s, 78.7 MB/s 2024-09-13T17:21:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162969 s, 64.3 MB/s 2024-09-13T17:21:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.296 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125748 s, 83.4 MB/s 2024-09-13T17:21:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.311 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130071 s, 80.6 MB/s 2024-09-13T17:21:00.326 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.327 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.327 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013223 s, 79.3 MB/s 2024-09-13T17:21:00.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:00.328 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:00.328 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:00.328 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.22 2024-09-13T17:21:00.328 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:00.328 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.22/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:00.329 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:00.329 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:00.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.384 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133258 s, 78.7 MB/s 2024-09-13T17:21:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127632 s, 82.2 MB/s 2024-09-13T17:21:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013116 s, 79.9 MB/s 2024-09-13T17:21:00.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.429 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.429 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125686 s, 83.4 MB/s 2024-09-13T17:21:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128232 s, 81.8 MB/s 2024-09-13T17:21:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.459 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128375 s, 81.7 MB/s 2024-09-13T17:21:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.477 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016153 s, 64.9 MB/s 2024-09-13T17:21:00.492 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.493 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131726 s, 79.6 MB/s 2024-09-13T17:21:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165155 s, 63.5 MB/s 2024-09-13T17:21:00.527 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.528 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142633 s, 73.5 MB/s 2024-09-13T17:21:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.546 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160822 s, 65.2 MB/s 2024-09-13T17:21:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.561 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128351 s, 81.7 MB/s 2024-09-13T17:21:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.576 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133297 s, 78.7 MB/s 2024-09-13T17:21:00.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140725 s, 74.5 MB/s 2024-09-13T17:21:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.610 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155187 s, 67.6 MB/s 2024-09-13T17:21:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129903 s, 80.7 MB/s 2024-09-13T17:21:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.645 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.646 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176827 s, 59.3 MB/s 2024-09-13T17:21:00.661 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.662 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.662 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013709 s, 76.5 MB/s 2024-09-13T17:21:00.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.681 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171628 s, 61.1 MB/s 2024-09-13T17:21:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.696 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125567 s, 83.5 MB/s 2024-09-13T17:21:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.712 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138681 s, 75.6 MB/s 2024-09-13T17:21:00.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.728 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140686 s, 74.5 MB/s 2024-09-13T17:21:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135798 s, 77.2 MB/s 2024-09-13T17:21:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.760 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136292 s, 76.9 MB/s 2024-09-13T17:21:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.776 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141965 s, 73.9 MB/s 2024-09-13T17:21:00.791 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.792 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.792 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129603 s, 80.9 MB/s 2024-09-13T17:21:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.808 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147481 s, 71.1 MB/s 2024-09-13T17:21:00.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.824 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.824 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013449 s, 78.0 MB/s 2024-09-13T17:21:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.840 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134022 s, 78.2 MB/s 2024-09-13T17:21:00.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.855 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.855 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131234 s, 79.9 MB/s 2024-09-13T17:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128446 s, 81.6 MB/s 2024-09-13T17:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140007 s, 74.9 MB/s 2024-09-13T17:21:00.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.23 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.23/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:00.888 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.944 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.945 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130399 s, 80.4 MB/s 2024-09-13T17:21:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.958 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118262 s, 88.7 MB/s 2024-09-13T17:21:00.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.976 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.976 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166241 s, 63.1 MB/s 2024-09-13T17:21:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:00.992 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013905 s, 75.4 MB/s 2024-09-13T17:21:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.008 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131757 s, 79.6 MB/s 2024-09-13T17:21:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.025 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150549 s, 69.7 MB/s 2024-09-13T17:21:01.038 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.039 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112394 s, 93.3 MB/s 2024-09-13T17:21:01.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129807 s, 80.8 MB/s 2024-09-13T17:21:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129823 s, 80.8 MB/s 2024-09-13T17:21:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133178 s, 78.7 MB/s 2024-09-13T17:21:01.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132139 s, 79.4 MB/s 2024-09-13T17:21:01.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137198 s, 76.4 MB/s 2024-09-13T17:21:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.131 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132937 s, 78.9 MB/s 2024-09-13T17:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132435 s, 79.2 MB/s 2024-09-13T17:21:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132353 s, 79.2 MB/s 2024-09-13T17:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125189 s, 83.8 MB/s 2024-09-13T17:21:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132954 s, 78.9 MB/s 2024-09-13T17:21:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136081 s, 77.1 MB/s 2024-09-13T17:21:01.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.227 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.227 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163043 s, 64.3 MB/s 2024-09-13T17:21:01.241 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.242 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.242 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125963 s, 83.2 MB/s 2024-09-13T17:21:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.257 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129772 s, 80.8 MB/s 2024-09-13T17:21:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.272 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126475 s, 82.9 MB/s 2024-09-13T17:21:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.287 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.288 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133016 s, 78.8 MB/s 2024-09-13T17:21:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.302 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118808 s, 88.3 MB/s 2024-09-13T17:21:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.317 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013462 s, 77.9 MB/s 2024-09-13T17:21:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.333 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138059 s, 76.0 MB/s 2024-09-13T17:21:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.349 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131727 s, 79.6 MB/s 2024-09-13T17:21:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.368 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171529 s, 61.1 MB/s 2024-09-13T17:21:01.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.383 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.383 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126589 s, 82.8 MB/s 2024-09-13T17:21:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.401 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155901 s, 67.3 MB/s 2024-09-13T17:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163614 s, 64.1 MB/s 2024-09-13T17:21:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125741 s, 83.4 MB/s 2024-09-13T17:21:01.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.24 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.24/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:01.436 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125517 s, 83.5 MB/s 2024-09-13T17:21:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.505 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123646 s, 84.8 MB/s 2024-09-13T17:21:01.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.520 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.520 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127547 s, 82.2 MB/s 2024-09-13T17:21:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.536 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124919 s, 83.9 MB/s 2024-09-13T17:21:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.551 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131215 s, 79.9 MB/s 2024-09-13T17:21:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.569 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162342 s, 64.6 MB/s 2024-09-13T17:21:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.588 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151319 s, 69.3 MB/s 2024-09-13T17:21:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130876 s, 80.1 MB/s 2024-09-13T17:21:01.617 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133491 s, 78.6 MB/s 2024-09-13T17:21:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.634 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139897 s, 75.0 MB/s 2024-09-13T17:21:01.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.649 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.650 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136036 s, 77.1 MB/s 2024-09-13T17:21:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120867 s, 86.8 MB/s 2024-09-13T17:21:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.679 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130229 s, 80.5 MB/s 2024-09-13T17:21:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137346 s, 76.3 MB/s 2024-09-13T17:21:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.711 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139871 s, 75.0 MB/s 2024-09-13T17:21:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.728 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143007 s, 73.3 MB/s 2024-09-13T17:21:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.743 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132943 s, 78.9 MB/s 2024-09-13T17:21:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.758 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128426 s, 81.6 MB/s 2024-09-13T17:21:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131925 s, 79.5 MB/s 2024-09-13T17:21:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.789 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127463 s, 82.3 MB/s 2024-09-13T17:21:01.803 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121771 s, 86.1 MB/s 2024-09-13T17:21:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.819 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132572 s, 79.1 MB/s 2024-09-13T17:21:01.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.838 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.838 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166342 s, 63.0 MB/s 2024-09-13T17:21:01.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.854 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133984 s, 78.3 MB/s 2024-09-13T17:21:01.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130499 s, 80.4 MB/s 2024-09-13T17:21:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.883 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123004 s, 85.2 MB/s 2024-09-13T17:21:01.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.900 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.900 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014308 s, 73.3 MB/s 2024-09-13T17:21:01.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013262 s, 79.1 MB/s 2024-09-13T17:21:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.930 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127804 s, 82.0 MB/s 2024-09-13T17:21:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.946 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133849 s, 78.3 MB/s 2024-09-13T17:21:01.964 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.964 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.964 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159651 s, 65.7 MB/s 2024-09-13T17:21:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:01.980 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013372 s, 78.4 MB/s 2024-09-13T17:21:01.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.25 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.25/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:01.982 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:02.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.037 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.037 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132066 s, 79.4 MB/s 2024-09-13T17:21:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.053 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131854 s, 79.5 MB/s 2024-09-13T17:21:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.068 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012923 s, 81.1 MB/s 2024-09-13T17:21:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.083 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130127 s, 80.6 MB/s 2024-09-13T17:21:02.098 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.099 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130498 s, 80.4 MB/s 2024-09-13T17:21:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.113 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123487 s, 84.9 MB/s 2024-09-13T17:21:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.130 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145983 s, 71.8 MB/s 2024-09-13T17:21:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.149 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171788 s, 61.0 MB/s 2024-09-13T17:21:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133716 s, 78.4 MB/s 2024-09-13T17:21:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128077 s, 81.9 MB/s 2024-09-13T17:21:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.194 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011963 s, 87.7 MB/s 2024-09-13T17:21:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.213 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164589 s, 63.7 MB/s 2024-09-13T17:21:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.232 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.233 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173833 s, 60.3 MB/s 2024-09-13T17:21:02.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.247 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126851 s, 82.7 MB/s 2024-09-13T17:21:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135953 s, 77.1 MB/s 2024-09-13T17:21:02.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.278 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.278 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127152 s, 82.5 MB/s 2024-09-13T17:21:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.294 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139334 s, 75.3 MB/s 2024-09-13T17:21:02.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163595 s, 64.1 MB/s 2024-09-13T17:21:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140655 s, 74.5 MB/s 2024-09-13T17:21:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.345 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132639 s, 79.1 MB/s 2024-09-13T17:21:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.363 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158929 s, 66.0 MB/s 2024-09-13T17:21:02.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.377 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118048 s, 88.8 MB/s 2024-09-13T17:21:02.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.391 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.391 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124963 s, 83.9 MB/s 2024-09-13T17:21:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153313 s, 68.4 MB/s 2024-09-13T17:21:02.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.425 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.425 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135772 s, 77.2 MB/s 2024-09-13T17:21:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.443 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157624 s, 66.5 MB/s 2024-09-13T17:21:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.458 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133574 s, 78.5 MB/s 2024-09-13T17:21:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.474 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128244 s, 81.8 MB/s 2024-09-13T17:21:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.488 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125578 s, 83.5 MB/s 2024-09-13T17:21:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.503 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012323 s, 85.1 MB/s 2024-09-13T17:21:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123967 s, 84.6 MB/s 2024-09-13T17:21:02.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.533 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130916 s, 80.1 MB/s 2024-09-13T17:21:02.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:02.534 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:02.534 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:02.534 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.26 2024-09-13T17:21:02.535 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:02.535 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.26/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:02.535 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:02.535 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.589 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131441 s, 79.8 MB/s 2024-09-13T17:21:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.605 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134737 s, 77.8 MB/s 2024-09-13T17:21:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.620 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012741 s, 82.3 MB/s 2024-09-13T17:21:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.636 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140814 s, 74.5 MB/s 2024-09-13T17:21:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.650 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113269 s, 92.6 MB/s 2024-09-13T17:21:02.665 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.665 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.665 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131663 s, 79.6 MB/s 2024-09-13T17:21:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.680 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.681 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131714 s, 79.6 MB/s 2024-09-13T17:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160386 s, 65.4 MB/s 2024-09-13T17:21:02.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.713 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.714 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127442 s, 82.3 MB/s 2024-09-13T17:21:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.732 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159449 s, 65.8 MB/s 2024-09-13T17:21:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.751 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169014 s, 62.0 MB/s 2024-09-13T17:21:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.770 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169274 s, 61.9 MB/s 2024-09-13T17:21:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.789 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168609 s, 62.2 MB/s 2024-09-13T17:21:02.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.804 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.804 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129902 s, 80.7 MB/s 2024-09-13T17:21:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.820 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127868 s, 82.0 MB/s 2024-09-13T17:21:02.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.835 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.835 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133059 s, 78.8 MB/s 2024-09-13T17:21:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.851 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136801 s, 76.6 MB/s 2024-09-13T17:21:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.869 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159061 s, 65.9 MB/s 2024-09-13T17:21:02.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.887 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.887 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160438 s, 65.4 MB/s 2024-09-13T17:21:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.902 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128303 s, 81.7 MB/s 2024-09-13T17:21:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.921 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016662 s, 62.9 MB/s 2024-09-13T17:21:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.939 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160273 s, 65.4 MB/s 2024-09-13T17:21:02.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.954 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.954 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130024 s, 80.6 MB/s 2024-09-13T17:21:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127283 s, 82.4 MB/s 2024-09-13T17:21:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:02.985 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:02.986 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014601 s, 71.8 MB/s 2024-09-13T17:21:03.002 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.002 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144505 s, 72.6 MB/s 2024-09-13T17:21:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167128 s, 62.7 MB/s 2024-09-13T17:21:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128308 s, 81.7 MB/s 2024-09-13T17:21:03.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.055 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163425 s, 64.2 MB/s 2024-09-13T17:21:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.073 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016818 s, 62.3 MB/s 2024-09-13T17:21:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.092 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163896 s, 64.0 MB/s 2024-09-13T17:21:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.107 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133774 s, 78.4 MB/s 2024-09-13T17:21:03.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.27 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.27/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:03.109 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.165 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014743 s, 71.1 MB/s 2024-09-13T17:21:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.181 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136846 s, 76.6 MB/s 2024-09-13T17:21:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.197 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131696 s, 79.6 MB/s 2024-09-13T17:21:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.214 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146865 s, 71.4 MB/s 2024-09-13T17:21:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132456 s, 79.2 MB/s 2024-09-13T17:21:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.247 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155707 s, 67.3 MB/s 2024-09-13T17:21:03.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.264 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142637 s, 73.5 MB/s 2024-09-13T17:21:03.282 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.283 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168054 s, 62.4 MB/s 2024-09-13T17:21:03.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.298 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.298 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135343 s, 77.5 MB/s 2024-09-13T17:21:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.314 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129869 s, 80.7 MB/s 2024-09-13T17:21:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012819 s, 81.8 MB/s 2024-09-13T17:21:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.347 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161525 s, 64.9 MB/s 2024-09-13T17:21:03.365 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.366 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.366 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161834 s, 64.8 MB/s 2024-09-13T17:21:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.381 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136239 s, 77.0 MB/s 2024-09-13T17:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.397 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127337 s, 82.3 MB/s 2024-09-13T17:21:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.415 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016125 s, 65.0 MB/s 2024-09-13T17:21:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.433 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161287 s, 65.0 MB/s 2024-09-13T17:21:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.451 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.452 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163226 s, 64.2 MB/s 2024-09-13T17:21:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.467 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134987 s, 77.7 MB/s 2024-09-13T17:21:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.483 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137014 s, 76.5 MB/s 2024-09-13T17:21:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.502 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167105 s, 62.7 MB/s 2024-09-13T17:21:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.517 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01258 s, 83.4 MB/s 2024-09-13T17:21:03.533 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.534 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144016 s, 72.8 MB/s 2024-09-13T17:21:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.549 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.550 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137457 s, 76.3 MB/s 2024-09-13T17:21:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.566 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139816 s, 75.0 MB/s 2024-09-13T17:21:03.584 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.585 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.586 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165656 s, 63.3 MB/s 2024-09-13T17:21:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.599 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127889 s, 82.0 MB/s 2024-09-13T17:21:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.618 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160383 s, 65.4 MB/s 2024-09-13T17:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137294 s, 76.4 MB/s 2024-09-13T17:21:03.654 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.655 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.655 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183454 s, 57.2 MB/s 2024-09-13T17:21:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.673 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160816 s, 65.2 MB/s 2024-09-13T17:21:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.688 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130353 s, 80.4 MB/s 2024-09-13T17:21:03.690 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:03.690 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.28 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.28/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:03.691 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:03.746 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.747 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139716 s, 75.1 MB/s 2024-09-13T17:21:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.765 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155852 s, 67.3 MB/s 2024-09-13T17:21:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.783 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154352 s, 67.9 MB/s 2024-09-13T17:21:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.799 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135399 s, 77.4 MB/s 2024-09-13T17:21:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.815 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142837 s, 73.4 MB/s 2024-09-13T17:21:03.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.832 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.832 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146878 s, 71.4 MB/s 2024-09-13T17:21:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.848 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135729 s, 77.3 MB/s 2024-09-13T17:21:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.862 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118249 s, 88.7 MB/s 2024-09-13T17:21:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.876 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121237 s, 86.5 MB/s 2024-09-13T17:21:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.892 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137141 s, 76.5 MB/s 2024-09-13T17:21:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.910 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.911 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161261 s, 65.0 MB/s 2024-09-13T17:21:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.925 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.926 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128632 s, 81.5 MB/s 2024-09-13T17:21:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.941 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013099 s, 80.0 MB/s 2024-09-13T17:21:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.957 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137983 s, 76.0 MB/s 2024-09-13T17:21:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.972 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132739 s, 79.0 MB/s 2024-09-13T17:21:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:03.987 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124929 s, 83.9 MB/s 2024-09-13T17:21:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.003 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136647 s, 76.7 MB/s 2024-09-13T17:21:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131396 s, 79.8 MB/s 2024-09-13T17:21:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.034 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138819 s, 75.5 MB/s 2024-09-13T17:21:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.050 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013693 s, 76.6 MB/s 2024-09-13T17:21:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.066 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132465 s, 79.2 MB/s 2024-09-13T17:21:04.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157553 s, 66.6 MB/s 2024-09-13T17:21:04.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.099 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.099 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126146 s, 83.1 MB/s 2024-09-13T17:21:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.115 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141319 s, 74.2 MB/s 2024-09-13T17:21:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.130 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131856 s, 79.5 MB/s 2024-09-13T17:21:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.146 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.147 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138155 s, 75.9 MB/s 2024-09-13T17:21:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.162 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136772 s, 76.7 MB/s 2024-09-13T17:21:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131452 s, 79.8 MB/s 2024-09-13T17:21:04.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.193 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.193 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128668 s, 81.5 MB/s 2024-09-13T17:21:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.212 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163457 s, 64.1 MB/s 2024-09-13T17:21:04.228 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134334 s, 78.1 MB/s 2024-09-13T17:21:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.243 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136224 s, 77.0 MB/s 2024-09-13T17:21:04.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.29 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.29/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:04.245 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.300 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013578 s, 77.2 MB/s 2024-09-13T17:21:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127934 s, 82.0 MB/s 2024-09-13T17:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125599 s, 83.5 MB/s 2024-09-13T17:21:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.345 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131933 s, 79.5 MB/s 2024-09-13T17:21:04.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.361 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.361 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137276 s, 76.4 MB/s 2024-09-13T17:21:04.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.377 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.378 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143208 s, 73.2 MB/s 2024-09-13T17:21:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.395 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149412 s, 70.2 MB/s 2024-09-13T17:21:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.410 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133166 s, 78.7 MB/s 2024-09-13T17:21:04.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.426 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.426 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131975 s, 79.5 MB/s 2024-09-13T17:21:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.444 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157596 s, 66.5 MB/s 2024-09-13T17:21:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.459 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128964 s, 81.3 MB/s 2024-09-13T17:21:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.473 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113275 s, 92.6 MB/s 2024-09-13T17:21:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.491 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160018 s, 65.5 MB/s 2024-09-13T17:21:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.511 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177785 s, 59.0 MB/s 2024-09-13T17:21:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.530 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164637 s, 63.7 MB/s 2024-09-13T17:21:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.553 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206748 s, 50.7 MB/s 2024-09-13T17:21:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.574 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018211 s, 57.6 MB/s 2024-09-13T17:21:04.592 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.593 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160697 s, 65.3 MB/s 2024-09-13T17:21:04.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.609 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.609 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136493 s, 76.8 MB/s 2024-09-13T17:21:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147984 s, 70.9 MB/s 2024-09-13T17:21:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.641 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013224 s, 79.3 MB/s 2024-09-13T17:21:04.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.656 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.656 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129319 s, 81.1 MB/s 2024-09-13T17:21:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.675 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169488 s, 61.9 MB/s 2024-09-13T17:21:04.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.694 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.694 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164675 s, 63.7 MB/s 2024-09-13T17:21:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.708 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123458 s, 84.9 MB/s 2024-09-13T17:21:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.726 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156446 s, 67.0 MB/s 2024-09-13T17:21:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.744 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156329 s, 67.1 MB/s 2024-09-13T17:21:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.763 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169179 s, 62.0 MB/s 2024-09-13T17:21:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.779 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138964 s, 75.5 MB/s 2024-09-13T17:21:04.793 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.794 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120583 s, 87.0 MB/s 2024-09-13T17:21:04.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169718 s, 61.8 MB/s 2024-09-13T17:21:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.829 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.830 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144894 s, 72.4 MB/s 2024-09-13T17:21:04.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.30 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.30/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:04.831 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:04.885 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.886 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127656 s, 82.1 MB/s 2024-09-13T17:21:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013191 s, 79.5 MB/s 2024-09-13T17:21:04.915 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.916 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012632 s, 83.0 MB/s 2024-09-13T17:21:04.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016638 s, 63.0 MB/s 2024-09-13T17:21:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.953 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158907 s, 66.0 MB/s 2024-09-13T17:21:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135994 s, 77.1 MB/s 2024-09-13T17:21:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128119 s, 81.8 MB/s 2024-09-13T17:21:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155708 s, 67.3 MB/s 2024-09-13T17:21:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140648 s, 74.6 MB/s 2024-09-13T17:21:05.035 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156604 s, 67.0 MB/s 2024-09-13T17:21:05.053 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.054 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157321 s, 66.7 MB/s 2024-09-13T17:21:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.069 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131783 s, 79.6 MB/s 2024-09-13T17:21:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.084 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130584 s, 80.3 MB/s 2024-09-13T17:21:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132483 s, 79.1 MB/s 2024-09-13T17:21:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135031 s, 77.7 MB/s 2024-09-13T17:21:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133987 s, 78.3 MB/s 2024-09-13T17:21:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.147 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.148 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137424 s, 76.3 MB/s 2024-09-13T17:21:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.164 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014079 s, 74.5 MB/s 2024-09-13T17:21:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.179 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.180 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132322 s, 79.2 MB/s 2024-09-13T17:21:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.195 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130173 s, 80.6 MB/s 2024-09-13T17:21:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.210 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130684 s, 80.2 MB/s 2024-09-13T17:21:05.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.229 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.229 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170737 s, 61.4 MB/s 2024-09-13T17:21:05.245 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.248 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136887 s, 76.6 MB/s 2024-09-13T17:21:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.263 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157631 s, 66.5 MB/s 2024-09-13T17:21:05.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.279 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.279 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133968 s, 78.3 MB/s 2024-09-13T17:21:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.295 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138353 s, 75.8 MB/s 2024-09-13T17:21:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.313 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160335 s, 65.4 MB/s 2024-09-13T17:21:05.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.328 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.329 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133367 s, 78.6 MB/s 2024-09-13T17:21:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.344 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129194 s, 81.2 MB/s 2024-09-13T17:21:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.358 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124718 s, 84.1 MB/s 2024-09-13T17:21:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.374 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135277 s, 77.5 MB/s 2024-09-13T17:21:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.389 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130109 s, 80.6 MB/s 2024-09-13T17:21:05.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c ' 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p scrub_control_test_path/dir.31 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> for ((i = 0; i < 32; i++)); do 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/dev/urandom of=scrub_control_test_path/dir.31/filename.$i bs=1M conv=fdatasync count=1 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> done 2024-09-13T17:21:05.391 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.449 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159475 s, 65.8 MB/s 2024-09-13T17:21:05.462 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.463 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118495 s, 88.5 MB/s 2024-09-13T17:21:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135229 s, 77.5 MB/s 2024-09-13T17:21:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.493 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.494 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013078 s, 80.2 MB/s 2024-09-13T17:21:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132661 s, 79.0 MB/s 2024-09-13T17:21:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.525 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013567 s, 77.3 MB/s 2024-09-13T17:21:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158448 s, 66.2 MB/s 2024-09-13T17:21:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.559 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140723 s, 74.5 MB/s 2024-09-13T17:21:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.575 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134827 s, 77.8 MB/s 2024-09-13T17:21:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.590 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129099 s, 81.2 MB/s 2024-09-13T17:21:05.606 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.607 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143287 s, 73.2 MB/s 2024-09-13T17:21:05.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.621 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.621 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125401 s, 83.6 MB/s 2024-09-13T17:21:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.636 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.637 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128423 s, 81.6 MB/s 2024-09-13T17:21:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.652 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129415 s, 81.0 MB/s 2024-09-13T17:21:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.667 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128182 s, 81.8 MB/s 2024-09-13T17:21:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.683 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139951 s, 74.9 MB/s 2024-09-13T17:21:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.702 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167553 s, 62.6 MB/s 2024-09-13T17:21:05.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.717 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.717 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132392 s, 79.2 MB/s 2024-09-13T17:21:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.734 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140924 s, 74.4 MB/s 2024-09-13T17:21:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.749 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128907 s, 81.3 MB/s 2024-09-13T17:21:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.768 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165178 s, 63.5 MB/s 2024-09-13T17:21:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.782 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012239 s, 85.7 MB/s 2024-09-13T17:21:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.797 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126528 s, 82.9 MB/s 2024-09-13T17:21:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134871 s, 77.7 MB/s 2024-09-13T17:21:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.836 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213691 s, 49.1 MB/s 2024-09-13T17:21:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.853 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143301 s, 73.2 MB/s 2024-09-13T17:21:05.866 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.867 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.867 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117044 s, 89.6 MB/s 2024-09-13T17:21:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.882 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127469 s, 82.3 MB/s 2024-09-13T17:21:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.901 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168716 s, 62.2 MB/s 2024-09-13T17:21:05.918 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.919 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.919 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155367 s, 67.5 MB/s 2024-09-13T17:21:05.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.934 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.934 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133063 s, 78.8 MB/s 2024-09-13T17:21:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-09-13T17:21:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-09-13T17:21:05.949 INFO:teuthology.orchestra.run.smithi028.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128469 s, 81.6 MB/s 2024-09-13T17:21:05.950 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2024-09-13T17:21:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.107+0000 7f9e0f200640 1 Processor -- start 2024-09-13T17:21:06.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.108+0000 7f9e0f200640 1 -- start start 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 0x7f9e0806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9e0815cd80 0x7f9e08070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0806f920 con 0x7f9e0806f3f0 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0815b290 con 0x7f9e0815cd80 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0815b570 con 0x7f9e08159770 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9dff400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9e0806f3f0 0x7f9e08159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41334/0 (socket says 172.21.15.28:41334) 2024-09-13T17:21:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9dff400640 1 -- 172.21.15.28:0/406280143 learned_addr learned my addr 172.21.15.28:0/406280143 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.109+0000 7f9e0d200640 1 --2- 172.21.15.28:0/406280143 >> v2:172.21.15.28:3300/0 conn(0x7f9e0815cd80 0x7f9e08070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e07e00640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 0x7f9e0806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e0d200640 1 -- 172.21.15.28:0/406280143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 msgr2=0x7f9e0806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e0d200640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 0x7f9e0806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e0d200640 1 -- 172.21.15.28:0/406280143 >> v1:172.21.15.28:6789/0 conn(0x7f9e0806f3f0 legacy=0x7f9e08159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e0d200640 1 -- 172.21.15.28:0/406280143 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0815b850 con 0x7f9e0815cd80 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e07400640 1 -- 172.21.15.28:0/406280143 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994249460 0 0) 0x7f9e0806f920 con 0x7f9e0806f3f0 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.110+0000 7f9e07e00640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 0x7f9e0806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e0d200640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e08070260 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009d90 tx=0x7f9e0002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bb1204024c9fb1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e07400640 1 -- 172.21.15.28:0/406280143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00031600 con 0x7f9e0815cd80 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e07400640 1 -- 172.21.15.28:0/406280143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e0002d050 con 0x7f9e0815cd80 2024-09-13T17:21:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e07400640 1 -- 172.21.15.28:0/406280143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00031980 con 0x7f9e0815cd80 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e0f200640 1 -- 172.21.15.28:0/406280143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 msgr2=0x7f9e08070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.111+0000 7f9e0f200640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e08070260 secure :-1 s=READY pgs=2166 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009d90 tx=0x7f9e0002a5d0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 -- 172.21.15.28:0/406280143 shutdown_connections 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e08070260 unknown :-1 s=CLOSED pgs=2166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 --2- 172.21.15.28:0/406280143 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e08159770 0x7f9e0806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 -- 172.21.15.28:0/406280143 >> 172.21.15.28:0/406280143 conn(0x7f9e080b0270 msgr2=0x7f9e080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 -- 172.21.15.28:0/406280143 shutdown_connections 2024-09-13T17:21:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.112+0000 7f9e0f200640 1 -- 172.21.15.28:0/406280143 wait complete. 2024-09-13T17:21:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.113+0000 7f9e0f200640 1 Processor -- start 2024-09-13T17:21:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.113+0000 7f9e0f200640 1 -- start start 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e0806f3f0 0x7f9e081421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e08159770 0x7f9e080cbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08142710 con 0x7f9e0815cd80 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08146b70 con 0x7f9e08159770 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.114+0000 7f9e0f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08145150 con 0x7f9e0806f3f0 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60438/0 (socket says 172.21.15.28:60438) 2024-09-13T17:21:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 -- 172.21.15.28:0/2709840240 learned_addr learned my addr 172.21.15.28:0/2709840240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9dff400640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e08159770 0x7f9e080cbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e07e00640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e0806f3f0 0x7f9e081421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 -- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e0806f3f0 msgr2=0x7f9e081421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e0806f3f0 0x7f9e081421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 -- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e08159770 msgr2=0x7f9e080cbf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e08159770 0x7f9e080cbf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.115+0000 7f9e0d200640 1 -- 172.21.15.28:0/2709840240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00009a70 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e0d200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009eb0 tx=0x7f9e00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00003b90 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e00003cf0 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08146240 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00004230 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.116+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08146800 con 0x7f9e0815cd80 2024-09-13T17:21:06.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.117+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08146d20 con 0x7f9e0815cd80 2024-09-13T17:21:06.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.118+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9e00004390 con 0x7f9e0815cd80 2024-09-13T17:21:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.119+0000 7f9e04c00640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 0x7f9de406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:06.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.120+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9e000aa440 con 0x7f9e0815cd80 2024-09-13T17:21:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.120+0000 7f9e07e00640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 0x7f9de406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:06.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.120+0000 7f9e07e00640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 0x7f9de406d0c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9e081476d0 tx=0x7f9df8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:06.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.124+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9e08146d20 con 0x7f9e0815cd80 2024-09-13T17:21:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:06.339+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7f9e08146d20 con 0x7f9e0815cd80 2024-09-13T17:21:07.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.169+0000 7f9e04c00640 1 -- 172.21.15.28:0/2709840240 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v152) v1 ==== 105+0+0 (secure 0 0 0) 0x7f9e08146d20 con 0x7f9e0815cd80 2024-09-13T17:21:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 msgr2=0x7f9de406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 0x7f9de406d0c0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f9e081476d0 tx=0x7f9df8008040 comp rx=0 tx=0).stop 2024-09-13T17:21:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 msgr2=0x7f9e080cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009eb0 tx=0x7f9e00004a40 comp rx=0 tx=0).stop 2024-09-13T17:21:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 shutdown_connections 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9de406ac00 0x7f9de406d0c0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9e0815cd80 0x7f9e080cc4b0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e08159770 0x7f9e080cbf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 --2- 172.21.15.28:0/2709840240 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9e0806f3f0 0x7f9e081421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 >> 172.21.15.28:0/2709840240 conn(0x7f9e080b0270 msgr2=0x7f9e0806ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 shutdown_connections 2024-09-13T17:21:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.172+0000 7f9e0f200640 1 -- 172.21.15.28:0/2709840240 wait complete. 2024-09-13T17:21:07.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-09-13T17:21:07.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.355+0000 7fcafb200640 1 Processor -- start 2024-09-13T17:21:07.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.356+0000 7fcafb200640 1 -- start start 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafb200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcafc0cc530 0x7fcafc072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcafc15d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcafc0c78e0 con 0x7fcafc0b06f0 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafb200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcafc070c90 con 0x7fcafc0cc530 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0cbe40 con 0x7fcafc0cc910 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafa800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcafc0b06f0 0x7fcafc06f220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41340/0 (socket says 172.21.15.28:41340) 2024-09-13T17:21:07.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.357+0000 7fcafa800640 1 -- 172.21.15.28:0/3173515412 learned_addr learned my addr 172.21.15.28:0/3173515412 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3655919818 0 0) 0x7fcafc0c78e0 con 0x7fcafc0b06f0 2024-09-13T17:21:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/3173515412 >> v2:172.21.15.189:3300/0 conn(0x7fcafc0cc530 0x7fcafc072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf1400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcafc15d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae0003520 con 0x7fcafc0b06f0 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4071350929 0 0) 0x7fcae0003520 con 0x7fcafc0b06f0 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 msgr2=0x7fcafc15d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcafc15d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 >> v2:172.21.15.189:3300/0 conn(0x7fcafc0cc530 msgr2=0x7fcafc072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 --2- 172.21.15.28:0/3173515412 >> v2:172.21.15.189:3300/0 conn(0x7fcafc0cc530 0x7fcafc072060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc071000 con 0x7fcafc0b06f0 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fcaf40019f0 con 0x7fcafc0b06f0 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 >> v1:172.21.15.28:6789/0 conn(0x7fcafc0b06f0 legacy=0x7fcafc06f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae0003970 0x7fcae0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcae00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcae0004d00 0x7fcae0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc070c90 con 0x7fcae0003970 2024-09-13T17:21:07.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc0cbe40 con 0x7fcae0004d00 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae000b980 con 0x7fcafc0cc910 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae0003970 0x7fcae0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.358+0000 7fcaf1400640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcae00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcafa800640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcae0004d00 0x7fcae0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcaf9e00640 1 -- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 msgr2=0x7fcae00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcafc0cc910 0x7fcae00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcaf9e00640 1 -- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcae0004d00 msgr2=0x7fcae0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcae0004d00 0x7fcae0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcaf9e00640 1 -- 172.21.15.28:0/3173515412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae0003520 con 0x7fcae0003970 2024-09-13T17:21:07.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.359+0000 7fcafa800640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcae0004d00 0x7fcae0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:07.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.360+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae0003970 0x7fcae0003d90 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009ac0 tx=0x7fcae00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29de5d6d6416b866 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.360+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec031400 con 0x7fcae0003970 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.360+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec02d050 con 0x7fcae0003970 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.360+0000 7fcaf9400640 1 -- 172.21.15.28:0/3173515412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec031780 con 0x7fcae0003970 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 -- 172.21.15.28:0/3173515412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae0003970 msgr2=0x7fcae0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 --2- 172.21.15.28:0/3173515412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcae0003970 0x7fcae0003d90 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009ac0 tx=0x7fcae00057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafa800640 1 -- 172.21.15.28:0/3173515412 reap_dead start 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 -- 172.21.15.28:0/3173515412 shutdown_connections 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 -- 172.21.15.28:0/3173515412 >> 172.21.15.28:0/3173515412 conn(0x7fcafc069e80 msgr2=0x7fcafc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 -- 172.21.15.28:0/3173515412 shutdown_connections 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.361+0000 7fcafb200640 1 -- 172.21.15.28:0/3173515412 wait complete. 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.362+0000 7fcafb200640 1 Processor -- start 2024-09-13T17:21:07.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 -- start start 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcae0003970 0x7fcafc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcafc149a00 0x7fcafc160090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1498a0 con 0x7fcafc15f4d0 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15def0 con 0x7fcafc149a00 2024-09-13T17:21:07.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.363+0000 7fcafb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15e1d0 con 0x7fcae0003970 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60464/0 (socket says 172.21.15.28:60464) 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 -- 172.21.15.28:0/136376114 learned_addr learned my addr 172.21.15.28:0/136376114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcae0003970 0x7fcafc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcafa800640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcafc149a00 0x7fcafc160090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 -- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcae0003970 msgr2=0x7fcafc15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcae0003970 0x7fcafc15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 -- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcafc149a00 msgr2=0x7fcafc160090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcafc149a00 0x7fcafc160090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.364+0000 7fcaf1400640 1 -- 172.21.15.28:0/136376114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc15e4b0 con 0x7fcafc15f4d0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcaf1400640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fcae40119a0 tx=0x7fcae4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae400f9d0 con 0x7fcafc15f4d0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae400fb30 con 0x7fcafc15f4d0 2024-09-13T17:21:07.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae400fea0 con 0x7fcafc15f4d0 2024-09-13T17:21:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc15e790 con 0x7fcafc15f4d0 2024-09-13T17:21:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.365+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc1f1b10 con 0x7fcafc15f4d0 2024-09-13T17:21:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.366+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc072060 con 0x7fcafc15f4d0 2024-09-13T17:21:07.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.367+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcae4014440 con 0x7fcafc15f4d0 2024-09-13T17:21:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.368+0000 7fcaf2a00640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 0x7fcad806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.368+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcae40897e0 con 0x7fcafc15f4d0 2024-09-13T17:21:07.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.368+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 0x7fcad806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:07.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.369+0000 7fcaf9e00640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 0x7fcad806d0c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcaec033000 tx=0x7fcaec008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:07.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.372+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcae4054980 con 0x7fcafc15f4d0 2024-09-13T17:21:07.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:07.586+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fcafc156640 con 0x7fcafc15f4d0 2024-09-13T17:21:08.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.171+0000 7fcaf2a00640 1 -- 172.21.15.28:0/136376114 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v153) v1 ==== 121+0+0 (secure 0 0 0) 0x7fcae40220c0 con 0x7fcafc15f4d0 2024-09-13T17:21:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 msgr2=0x7fcad806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 0x7fcad806d0c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcaec033000 tx=0x7fcaec008030 comp rx=0 tx=0).stop 2024-09-13T17:21:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 msgr2=0x7fcafc15fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7fcae40119a0 tx=0x7fcae4011da0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 shutdown_connections 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcad806ac00 0x7fcad806d0c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcafc149a00 0x7fcafc160090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcafc15f4d0 0x7fcafc15fb50 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 --2- 172.21.15.28:0/136376114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcae0003970 0x7fcafc15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 >> 172.21.15.28:0/136376114 conn(0x7fcafc069e80 msgr2=0x7fcafc1558b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 shutdown_connections 2024-09-13T17:21:08.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.174+0000 7fcafb200640 1 -- 172.21.15.28:0/136376114 wait complete. 2024-09-13T17:21:08.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.354+0000 7f9177600640 1 Processor -- start 2024-09-13T17:21:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.354+0000 7f9177600640 1 -- start start 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.355+0000 7f9177600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f917015a540 0x7f917006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.355+0000 7f9177600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9170070cc0 0x7f9170071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.355+0000 7f9177600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917015e100 con 0x7f91700cabc0 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.355+0000 7f9177600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91700adb00 con 0x7f917015a540 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.355+0000 7f9177600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9170071670 con 0x7f9170070cc0 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f917015a540 0x7f917006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f917015a540 0x7f917006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60474/0 (socket says 172.21.15.28:60474) 2024-09-13T17:21:08.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 -- 172.21.15.28:0/2462173948 learned_addr learned my addr 172.21.15.28:0/2462173948 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9175600640 1 --2- 172.21.15.28:0/2462173948 >> v2:172.21.15.189:3300/0 conn(0x7f9170070cc0 0x7f9170071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 -- 172.21.15.28:0/2462173948 >> v2:172.21.15.189:3300/0 conn(0x7f9170070cc0 msgr2=0x7f9170071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 --2- 172.21.15.28:0/2462173948 >> v2:172.21.15.189:3300/0 conn(0x7f9170070cc0 0x7f9170071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 -- 172.21.15.28:0/2462173948 >> v1:172.21.15.189:6789/0 conn(0x7f91700cabc0 legacy=0x7f917015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:08.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.356+0000 7f9174c00640 1 -- 172.21.15.28:0/2462173948 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917015e3e0 con 0x7f917015a540 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.357+0000 7f9174c00640 1 --2- 172.21.15.28:0/2462173948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f917015a540 0x7f917006c430 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f9170070900 tx=0x7f916002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753fad60fa22818d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.357+0000 7f916fe00640 1 -- 172.21.15.28:0/2462173948 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160032440 con 0x7f917015a540 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f916fe00640 1 -- 172.21.15.28:0/2462173948 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91600325a0 con 0x7f917015a540 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f916fe00640 1 -- 172.21.15.28:0/2462173948 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9160032930 con 0x7f917015a540 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 -- 172.21.15.28:0/2462173948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f917015a540 msgr2=0x7f917006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 --2- 172.21.15.28:0/2462173948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f917015a540 0x7f917006c430 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f9170070900 tx=0x7f916002a210 comp rx=0 tx=0).stop 2024-09-13T17:21:08.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 -- 172.21.15.28:0/2462173948 shutdown_connections 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 --2- 172.21.15.28:0/2462173948 >> v2:172.21.15.189:3300/0 conn(0x7f9170070cc0 0x7f9170071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 --2- 172.21.15.28:0/2462173948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f917015a540 0x7f917006c430 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 -- 172.21.15.28:0/2462173948 >> 172.21.15.28:0/2462173948 conn(0x7f9170069e80 msgr2=0x7f917006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 -- 172.21.15.28:0/2462173948 shutdown_connections 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.358+0000 7f9177600640 1 -- 172.21.15.28:0/2462173948 wait complete. 2024-09-13T17:21:08.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.360+0000 7f9177600640 1 Processor -- start 2024-09-13T17:21:08.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.360+0000 7f9177600640 1 -- start start 2024-09-13T17:21:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9170070cc0 0x7f91700b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 0x7f91700ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91700afde0 con 0x7f91700cabc0 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91700b2870 con 0x7f917015a540 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9177600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91700b1cf0 con 0x7f9170070cc0 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60484/0 (socket says 172.21.15.28:60484) 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9167e00640 1 -- 172.21.15.28:0/3747627523 learned_addr learned my addr 172.21.15.28:0/3747627523 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9174c00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9170070cc0 0x7f91700b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.361+0000 7f9175600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 0x7f91700ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 -- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9170070cc0 msgr2=0x7f91700b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9170070cc0 0x7f91700b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 -- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 msgr2=0x7f91700ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 0x7f91700ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 -- 172.21.15.28:0/3747627523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91600094d0 con 0x7f91700cabc0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9175600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 0x7f91700ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9167e00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f915800c450 tx=0x7f915800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91580052d0 con 0x7f91700cabc0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158005430 con 0x7f91700cabc0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158012870 con 0x7f91700cabc0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91700b1fd0 con 0x7f91700cabc0 2024-09-13T17:21:08.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.362+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91700b2590 con 0x7f91700cabc0 2024-09-13T17:21:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.363+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91700aff20 con 0x7f91700cabc0 2024-09-13T17:21:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.364+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f91580129d0 con 0x7f91700cabc0 2024-09-13T17:21:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.365+0000 7f916d600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 0x7f914c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.365+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9158088f70 con 0x7f91700cabc0 2024-09-13T17:21:08.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.366+0000 7f9174c00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 0x7f914c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:08.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.366+0000 7f9174c00640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 0x7f914c06d0c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9160034000 tx=0x7f916002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:08.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.369+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9158053fe0 con 0x7f91700cabc0 2024-09-13T17:21:08.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.582+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9170071130 con 0x7f91700cabc0 2024-09-13T17:21:08.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.583+0000 7f916d600640 1 -- 172.21.15.28:0/3747627523 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4761 (secure 0 0 0) 0x7f9158057600 con 0x7f91700cabc0 2024-09-13T17:21:08.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:08.585 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":154,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":154,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:21:08.173139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10551":{"gid":10551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:21:08.585 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 154 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 msgr2=0x7f914c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 0x7f914c06d0c0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f9160034000 tx=0x7f916002e000 comp rx=0 tx=0).stop 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 msgr2=0x7f91701440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f915800c450 tx=0x7f915800c910 comp rx=0 tx=0).stop 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 shutdown_connections 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f914c06ac00 0x7f914c06d0c0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f917015a540 0x7f91700ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f91700cabc0 0x7f91701440c0 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 --2- 172.21.15.28:0/3747627523 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9170070cc0 0x7f91700b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:08.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 >> 172.21.15.28:0/3747627523 conn(0x7f9170069e80 msgr2=0x7f91700cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:08.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 shutdown_connections 2024-09-13T17:21:08.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:08.585+0000 7f9177600640 1 -- 172.21.15.28:0/3747627523 wait complete. 2024-09-13T17:21:09.005 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 154, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-09-13T17:20:39.612305+0000', 'modified': '2024-09-13T17:21:08.173139+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10566}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10551': {'gid': 10551, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.28:6833/3568343573', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3568343573}, {'type': 'v1', 'addr': '172.21.15.28:6833', 'nonce': 3568343573}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_10566': {'gid': 10566, 'name': 'a', 'rank': 0, 'incarnation': 147, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/3229070352', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 3229070352}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 3229070352}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:21:09.006 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:21:09.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:21:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.164+0000 7f2a2cc00640 1 Processor -- start 2024-09-13T17:21:09.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.164+0000 7f2a2cc00640 1 -- start start 2024-09-13T17:21:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 0x7f2a2806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 0x7f2a28070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2a280add00 0x7f2a28159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2806f920 con 0x7f2a280add00 2024-09-13T17:21:09.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a2815b290 con 0x7f2a28159770 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.165+0000 7f2a2cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2815b570 con 0x7f2a2806f3f0 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2a280add00 0x7f2a28159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2a280add00 0x7f2a28159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60490/0 (socket says 172.21.15.28:60490) 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 -- 172.21.15.28:0/2112687759 learned_addr learned my addr 172.21.15.28:0/2112687759 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a25400640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 0x7f2a28070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2112687759 >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 0x7f2a2806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 -- 172.21.15.28:0/2112687759 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 msgr2=0x7f2a28070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 0x7f2a28070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 -- 172.21.15.28:0/2112687759 >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 msgr2=0x7f2a2806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 --2- 172.21.15.28:0/2112687759 >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 0x7f2a2806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a26800640 1 -- 172.21.15.28:0/2112687759 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2815b850 con 0x7f2a280add00 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a25400640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 0x7f2a28070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.166+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2112687759 >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 0x7f2a2806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.167+0000 7f2a26800640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a280add00 0x7f2a28159b50 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f2a28068040 tx=0x7f2a2002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3620d6e6739e093 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.167+0000 7f2a24a00640 1 -- 172.21.15.28:0/2112687759 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20031600 con 0x7f2a280add00 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.167+0000 7f2a24a00640 1 -- 172.21.15.28:0/2112687759 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2002d050 con 0x7f2a280add00 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.167+0000 7f2a24a00640 1 -- 172.21.15.28:0/2112687759 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a20031980 con 0x7f2a280add00 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2112687759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a280add00 msgr2=0x7f2a28159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a280add00 0x7f2a28159b50 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7f2a28068040 tx=0x7f2a2002a610 comp rx=0 tx=0).stop 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2112687759 shutdown_connections 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a280add00 0x7f2a28159b50 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2112687759 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a2806f3f0 0x7f2a28070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2112687759 >> v2:172.21.15.189:3300/0 conn(0x7f2a28159770 0x7f2a2806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2112687759 >> 172.21.15.28:0/2112687759 conn(0x7f2a280b0270 msgr2=0x7f2a280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2112687759 shutdown_connections 2024-09-13T17:21:09.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.168+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2112687759 wait complete. 2024-09-13T17:21:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.170+0000 7f2a2cc00640 1 Processor -- start 2024-09-13T17:21:09.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.170+0000 7f2a2cc00640 1 -- start start 2024-09-13T17:21:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 0x7f2a281f6500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 0x7f2a281f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a281565a0 con 0x7f2a2806f3f0 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280aa500 con 0x7f2a280add00 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a2cc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a280aa7e0 con 0x7f2a28159770 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60492/0 (socket says 172.21.15.28:60492) 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 -- 172.21.15.28:0/2798312453 learned_addr learned my addr 172.21.15.28:0/2798312453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a26800640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 0x7f2a281f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25400640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 0x7f2a281f6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 -- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 msgr2=0x7f2a281f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 0x7f2a281f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 -- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 msgr2=0x7f2a281f6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 0x7f2a281f6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25e00640 1 -- 172.21.15.28:0/2798312453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a20009b30 con 0x7f2a2806f3f0 2024-09-13T17:21:09.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a25400640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 0x7f2a281f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.171+0000 7f2a26800640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 0x7f2a281f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d960 tx=0x7f2a1800dd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a1e000640 1 -- 172.21.15.28:0/2798312453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18004230 con 0x7f2a2806f3f0 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a1e000640 1 -- 172.21.15.28:0/2798312453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a18004390 con 0x7f2a2806f3f0 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a1e000640 1 -- 172.21.15.28:0/2798312453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18007b30 con 0x7f2a2806f3f0 2024-09-13T17:21:09.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a280aaac0 con 0x7f2a2806f3f0 2024-09-13T17:21:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.172+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a280ac550 con 0x7f2a2806f3f0 2024-09-13T17:21:09.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.174+0000 7f2a1e000640 1 -- 172.21.15.28:0/2798312453 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2a18007c90 con 0x7f2a2806f3f0 2024-09-13T17:21:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.175+0000 7f2a1e000640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 0x7f2a0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.175+0000 7f2a1e000640 1 -- 172.21.15.28:0/2798312453 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a18089500 con 0x7f2a2806f3f0 2024-09-13T17:21:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.176+0000 7f2a25400640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 0x7f2a0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.176+0000 7f2a2faaa740 1 Processor -- start 2024-09-13T17:21:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.176+0000 7f2a2faaa740 1 -- start start 2024-09-13T17:21:09.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.176+0000 7f2a25400640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 0x7f2a0806d0c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2a14006fd0 tx=0x7f2a14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b418bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 0x55d4b41d8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b40f4ab0 con 0x55d4b41b2b80 2024-09-13T17:21:09.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b40f2240 con 0x55d4b418b6d0 2024-09-13T17:21:09.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.177+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b40edb30 con 0x55d4b4188e90 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a25400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 0x55d4b41d8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b418bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60498/0 (socket says 172.21.15.28:60498) 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 -- 172.21.15.28:0/1823776357 learned_addr learned my addr 172.21.15.28:0/1823776357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a25400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 0x55d4b41d8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:54040/0 (socket says 172.21.15.28:54040) 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 -- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 msgr2=0x55d4b41d8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 0x55d4b41d8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 -- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 msgr2=0x55d4b418bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b418bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a26800640 1 -- 172.21.15.28:0/1823776357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b40b8bd0 con 0x55d4b41b2b80 2024-09-13T17:21:09.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.178+0000 7f2a25e00640 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b418bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.179+0000 7f2a26800640 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f2a20034040 tx=0x7f2a2002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207256e8e386bd49 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.179+0000 7f2a12a00640 1 -- 172.21.15.28:0/1823776357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2003b610 con 0x55d4b41b2b80 2024-09-13T17:21:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.179+0000 7f2a12a00640 1 -- 172.21.15.28:0/1823776357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a2003b770 con 0x55d4b41b2b80 2024-09-13T17:21:09.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.179+0000 7f2a12a00640 1 -- 172.21.15.28:0/1823776357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2003bae0 con 0x55d4b41b2b80 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1823776357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 msgr2=0x55d4b41d90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7f2a20034040 tx=0x7f2a2002e000 comp rx=0 tx=0).stop 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1823776357 shutdown_connections 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41b2b80 0x55d4b41d90e0 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b4188e90 0x55d4b41d8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1823776357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b418bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1823776357 >> 172.21.15.28:0/1823776357 conn(0x55d4b4187e00 msgr2=0x55d4b41d02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1823776357 shutdown_connections 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.180+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1823776357 wait complete. 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.181+0000 7f2a2faaa740 1 -- start start 2024-09-13T17:21:09.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.181+0000 7f2a2faaa740 1 Processor -- start 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d58e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b41d5e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b408f260 con 0x55d4b418b6d0 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b408ef00 con 0x55d4b41b2b80 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b3e54f20 con 0x55d4b41d85b0 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a2faaa740 10 client.0 _ll_register_callbacks cb 0x55d4b40e71a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.182+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60508/0 (socket says 172.21.15.28:60508) 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 -- 172.21.15.28:0/2714836879 learned_addr learned my addr 172.21.15.28:0/2714836879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a26800640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b41d5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25400640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 -- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 msgr2=0x55d4b41d5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b41d5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 -- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 msgr2=0x55d4b41d58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 -- 172.21.15.28:0/2714836879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2815b850 con 0x55d4b418b6d0 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a25e00640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f2a180074f0 tx=0x7f2a18058b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cf294c6204104a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.183+0000 7f2a07400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18089270 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b4117cd0 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a1806d070 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a180602b0 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a2faaa740 10 client.10749 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.184+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55d4b40ec1b0 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d4b4117cd0 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f2a1805e110 con 0x55d4b418b6d0 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 10 client.10749 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 10 client.10749 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 20 client.10749 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41b1be0 0x55d4b41ed200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 4 client.10749 mds_command: new command op to 10566 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 --> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4b3de13c0 con 0x55d4b41b1be0 2024-09-13T17:21:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.185+0000 7f2a25400640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41b1be0 0x55d4b41ed200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.186+0000 7f2a07400640 10 client.10749 ms_handle_connect on v2:172.21.15.28:6834/3229070352 2024-09-13T17:21:09.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.186+0000 7f2a25400640 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41b1be0 0x55d4b41ed200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2606b9d86f0c2c63 server_cookie=ca08f71afd0d9e88 in_seq=0 out_seq=0 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.186+0000 7f2a07400640 1 -- 172.21.15.28:0/2714836879 <== mds.0 v2:172.21.15.28:6834/3229070352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55d4b3de13c0 con 0x55d4b41b1be0 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.186+0000 7f2a07400640 10 client.10749 handle_command_reply: tid=0 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.186+0000 7f2a2faaa740 1 client.10749 shutdown 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 msgr2=0x55d4b41d8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7f2a180074f0 tx=0x7f2a18058b90 comp rx=0 tx=0).stop 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 shutdown_connections 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41b1be0 0x55d4b41ed200 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b41d5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a07400640 0 client.10749 ms_handle_reset on v2:172.21.15.28:6834/3229070352 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/2714836879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b41d8070 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 >> 172.21.15.28:0/2714836879 conn(0x55d4b4188e90 msgr2=0x55d4b41ce5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 shutdown_connections 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 1 -- 172.21.15.28:0/2714836879 wait complete. 2024-09-13T17:21:09.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.187+0000 7f2a2faaa740 20 client.10749 trim_cache size 0 max 16384 2024-09-13T17:21:09.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.203+0000 7f2a2faaa740 1 Processor -- start 2024-09-13T17:21:09.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.203+0000 7f2a2faaa740 1 -- start start 2024-09-13T17:21:09.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.203+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b4231cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b41d0190 con 0x55d4b418b6d0 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b41cf650 con 0x55d4b41b2b80 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b4181d90 con 0x55d4b41d85b0 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60510/0 (socket says 172.21.15.28:60510) 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 -- 172.21.15.28:0/231284458 learned_addr learned my addr 172.21.15.28:0/231284458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 -- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 msgr2=0x55d4b4231cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b4231cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 -- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 msgr2=0x55d4b41d5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 -- 172.21.15.28:0/231284458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4b4121170 con 0x55d4b418b6d0 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.204+0000 7f2a25e00640 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f2a18000c00 tx=0x7f2a18089000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab72a2e338bd040 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a13400640 1 -- 172.21.15.28:0/231284458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a18006e50 con 0x55d4b418b6d0 2024-09-13T17:21:09.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a13400640 1 -- 172.21.15.28:0/231284458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1805c180 con 0x55d4b418b6d0 2024-09-13T17:21:09.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a13400640 1 -- 172.21.15.28:0/231284458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a180604b0 con 0x55d4b418b6d0 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 -- 172.21.15.28:0/231284458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 msgr2=0x55d4b4187280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f2a18000c00 tx=0x7f2a18089000 comp rx=0 tx=0).stop 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 -- 172.21.15.28:0/231284458 shutdown_connections 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41d85b0 0x55d4b4231cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b41b2b80 0x55d4b41d5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/231284458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b418b6d0 0x55d4b4187280 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 -- 172.21.15.28:0/231284458 >> 172.21.15.28:0/231284458 conn(0x55d4b4188e90 msgr2=0x55d4b41af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 -- 172.21.15.28:0/231284458 shutdown_connections 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.205+0000 7f2a2faaa740 1 -- 172.21.15.28:0/231284458 wait complete. 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.206+0000 7f2a2faaa740 1 -- start start 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.206+0000 7f2a2faaa740 1 Processor -- start 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b421b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41b2b80 0x55d4b41b6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b4232690 con 0x55d4b41d85b0 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4b42319a0 con 0x55d4b418b6d0 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4b4230410 con 0x55d4b41b2b80 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a2faaa740 10 client.0 _ll_register_callbacks cb 0x7f2a180581b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a26800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60524/0 (socket says 172.21.15.28:60524) 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a26800640 1 -- 172.21.15.28:0/1824803642 learned_addr learned my addr 172.21.15.28:0/1824803642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a25e00640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b421b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.207+0000 7f2a25400640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41b2b80 0x55d4b41b6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 -- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41b2b80 msgr2=0x55d4b41b6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41b2b80 0x55d4b41b6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 -- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 msgr2=0x55d4b421b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b421b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 -- 172.21.15.28:0/1824803642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4b40ec1b0 con 0x55d4b41d85b0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a25e00640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b421b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a26800640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009f40 tx=0x7f2a20004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e56dba6af8206167 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a07400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a2003bae0 con 0x55d4b41d85b0 2024-09-13T17:21:09.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2a2003bc40 con 0x55d4b41d85b0 2024-09-13T17:21:09.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4b3df26c0 con 0x55d4b41d85b0 2024-09-13T17:21:09.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2a2003be00 con 0x55d4b41d85b0 2024-09-13T17:21:09.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a2faaa740 10 client.10755 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.208+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55d4b408e470 con 0x55d4b41d85b0 2024-09-13T17:21:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d4b3df26c0 con 0x55d4b41d85b0 2024-09-13T17:21:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f2a2004c030 con 0x55d4b41d85b0 2024-09-13T17:21:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 10 client.10755 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 10 client.10755 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 20 client.10755 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41d1430 0x55d4b41d1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 4 client.10755 mds_command: new command op to 10566 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 --> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d4b3e45d10 con 0x55d4b41d1430 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.209+0000 7f2a25e00640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41d1430 0x55d4b41d1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a25e00640 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41d1430 0x55d4b41d1810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe3ada221aa68c63 server_cookie=c82b9d16ae0149d1 in_seq=0 out_seq=0 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a07400640 10 client.10755 ms_handle_connect on v2:172.21.15.28:6834/3229070352 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a07400640 1 -- 172.21.15.28:0/1824803642 <== mds.0 v2:172.21.15.28:6834/3229070352 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55d4b3e45d10 con 0x55d4b41d1430 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a07400640 10 client.10755 handle_command_reply: tid=0 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 client.10755 shutdown 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 msgr2=0x55d4b41d3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f2a20009f40 tx=0x7f2a20004ab0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 shutdown_connections 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:6834/3229070352,v1:172.21.15.28:6835/3229070352] conn(0x55d4b41d1430 0x55d4b41d1810 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d4b41d85b0 0x55d4b41d3770 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d4b41b2b80 0x55d4b41b6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 --2- 172.21.15.28:0/1824803642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4b418b6d0 0x55d4b421b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a07400640 0 client.10755 ms_handle_reset on v2:172.21.15.28:6834/3229070352 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 >> 172.21.15.28:0/1824803642 conn(0x55d4b4188e90 msgr2=0x55d4b418aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 shutdown_connections 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 1 -- 172.21.15.28:0/1824803642 wait complete. 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.210+0000 7f2a2faaa740 20 client.10755 trim_cache size 0 max 16384 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:21:09.216 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 10566, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 29.596340911999999, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 154, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 181, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 181, 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 34.707871369999999 2024-09-13T17:21:09.217 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 msgr2=0x7f2a0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 0x7f2a0806d0c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f2a14006fd0 tx=0x7f2a14008040 comp rx=0 tx=0).stop 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 msgr2=0x7f2a280ac010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7f2a1800d960 tx=0x7f2a1800dd60 comp rx=0 tx=0).stop 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 shutdown_connections 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a0806ac00 0x7f2a0806d0c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a28159770 0x7f2a281f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a280add00 0x7f2a281f6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 --2- 172.21.15.28:0/2798312453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a2806f3f0 0x7f2a280ac010 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 >> 172.21.15.28:0/2798312453 conn(0x7f2a280b0270 msgr2=0x7f2a280ad180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 shutdown_connections 2024-09-13T17:21:09.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.211+0000 7f2a2cc00640 1 -- 172.21.15.28:0/2798312453 wait complete. 2024-09-13T17:21:09.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c scrub start /scrub_control_test_path recursive 2024-09-13T17:21:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.386+0000 7f6329200640 1 Processor -- start 2024-09-13T17:21:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.386+0000 7f6329200640 1 -- start start 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.387+0000 7f6329200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 0x7f63240b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6329200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632415d950 con 0x7f6324157000 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6329200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632415dc30 con 0x7f632415cdd0 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6329200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63240b1b80 con 0x7f63240b2380 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6321e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6324157000 0x7f632406b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41354/0 (socket says 172.21.15.28:41354) 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6321e00640 1 -- 172.21.15.28:0/191836322 learned_addr learned my addr 172.21.15.28:0/191836322 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6319400640 1 --2- 172.21.15.28:0/191836322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 0x7f63240b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 52226635 0 0) 0x7f632415d950 con 0x7f6324157000 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308003520 con 0x7f6324157000 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.388+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551038354 0 0) 0x7f632415dc30 con 0x7f632415cdd0 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.389+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f632415d950 con 0x7f632415cdd0 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.389+0000 7f6319400640 1 -- 172.21.15.28:0/191836322 >> v1:172.21.15.189:6789/0 conn(0x7f632415cdd0 legacy=0x7f63240ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.389+0000 7f6319400640 1 -- 172.21.15.28:0/191836322 >> v1:172.21.15.28:6789/0 conn(0x7f6324157000 legacy=0x7f632406b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.389+0000 7f6319400640 1 -- 172.21.15.28:0/191836322 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63240b1e60 con 0x7f63240b2380 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6319400640 1 --2- 172.21.15.28:0/191836322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 0x7f63240b27e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f630c0097d0 tx=0x7f630c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78d688b6b5faee18 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c032440 con 0x7f63240b2380 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c0325a0 con 0x7f63240b2380 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6321400640 1 -- 172.21.15.28:0/191836322 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c032930 con 0x7f63240b2380 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6329200640 1 -- 172.21.15.28:0/191836322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 msgr2=0x7f63240b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.390+0000 7f6329200640 1 --2- 172.21.15.28:0/191836322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 0x7f63240b27e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f630c0097d0 tx=0x7f630c02a210 comp rx=0 tx=0).stop 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.391+0000 7f6329200640 1 -- 172.21.15.28:0/191836322 shutdown_connections 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.391+0000 7f6329200640 1 --2- 172.21.15.28:0/191836322 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63240b2380 0x7f63240b27e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.391+0000 7f6329200640 1 -- 172.21.15.28:0/191836322 >> 172.21.15.28:0/191836322 conn(0x7f6324069e80 msgr2=0x7f632406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.391+0000 7f6329200640 1 -- 172.21.15.28:0/191836322 shutdown_connections 2024-09-13T17:21:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.391+0000 7f6329200640 1 -- 172.21.15.28:0/191836322 wait complete. 2024-09-13T17:21:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.392+0000 7f6329200640 1 Processor -- start 2024-09-13T17:21:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 -- start start 2024-09-13T17:21:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63240b2380 0x7f6324153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6324157000 0x7f63241573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632415cdd0 0x7f6324157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324154630 con 0x7f63240b2380 2024-09-13T17:21:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.393+0000 7f6329200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324154910 con 0x7f632415cdd0 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6329200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6324154bf0 con 0x7f6324157000 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63240b2380 0x7f6324153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63240b2380 0x7f6324153de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60530/0 (socket says 172.21.15.28:60530) 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 -- 172.21.15.28:0/2920337695 learned_addr learned my addr 172.21.15.28:0/2920337695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6319400640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6324157000 0x7f63241573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6322800640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632415cdd0 0x7f6324157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 -- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6324157000 msgr2=0x7f63241573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6324157000 0x7f63241573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 -- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632415cdd0 msgr2=0x7f6324157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632415cdd0 0x7f6324157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.394+0000 7f6321e00640 1 -- 172.21.15.28:0/2920337695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c009400 con 0x7f63240b2380 2024-09-13T17:21:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f6321e00640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f63240b2380 0x7f6324153de0 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f631400c500 tx=0x7f631400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f631aa00640 1 -- 172.21.15.28:0/2920337695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6314005ce0 con 0x7f63240b2380 2024-09-13T17:21:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f631aa00640 1 -- 172.21.15.28:0/2920337695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6314005e40 con 0x7f63240b2380 2024-09-13T17:21:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f631aa00640 1 -- 172.21.15.28:0/2920337695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63140072f0 con 0x7f63240b2380 2024-09-13T17:21:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f6329200640 1 -- 172.21.15.28:0/2920337695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6324154ed0 con 0x7f63240b2380 2024-09-13T17:21:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.395+0000 7f6329200640 1 -- 172.21.15.28:0/2920337695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63241556d0 con 0x7f63240b2380 2024-09-13T17:21:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.397+0000 7f631aa00640 1 -- 172.21.15.28:0/2920337695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6314007450 con 0x7f63240b2380 2024-09-13T17:21:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.398+0000 7f631aa00640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630006ac00 0x7f630006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.399+0000 7f631aa00640 1 -- 172.21.15.28:0/2920337695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6314088960 con 0x7f63240b2380 2024-09-13T17:21:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.399+0000 7f6319400640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630006ac00 0x7f630006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.399+0000 7f632c07c740 1 Processor -- start 2024-09-13T17:21:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.399+0000 7f632c07c740 1 -- start start 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f6319400640 1 --2- 172.21.15.28:0/2920337695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630006ac00 0x7f630006d0c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f630c034000 tx=0x7f630c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605cb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 0x55e605d01590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605bbec30 con 0x55e605cb1340 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605c1cce0 con 0x55e605cb3b80 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e605c1a470 con 0x55e605cdb030 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60546/0 (socket says 172.21.15.28:60546) 2024-09-13T17:21:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.400+0000 7f6319400640 1 -- 172.21.15.28:0/3338909932 learned_addr learned my addr 172.21.15.28:0/3338909932 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6321e00640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605cb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6322800640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 0x55e605d01590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 -- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 msgr2=0x55e605d01590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 0x55e605d01590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 -- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 msgr2=0x55e605cb3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605cb3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 -- 172.21.15.28:0/3338909932 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e605bb79c0 con 0x55e605cb1340 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6322800640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 0x55e605d01590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.401+0000 7f6319400640 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f630c02a800 tx=0x7f630c02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6a0eca303c647c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f6312a00640 1 -- 172.21.15.28:0/3338909932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c0496e0 con 0x55e605cb1340 2024-09-13T17:21:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f6312a00640 1 -- 172.21.15.28:0/3338909932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c049840 con 0x55e605cb1340 2024-09-13T17:21:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f6312a00640 1 -- 172.21.15.28:0/3338909932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c049bb0 con 0x55e605cb1340 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 -- 172.21.15.28:0/3338909932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 msgr2=0x55e605d01050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f630c02a800 tx=0x7f630c02c000 comp rx=0 tx=0).stop 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 -- 172.21.15.28:0/3338909932 shutdown_connections 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdb030 0x55e605d01590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb1340 0x55e605d01050 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 --2- 172.21.15.28:0/3338909932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605cb3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 -- 172.21.15.28:0/3338909932 >> 172.21.15.28:0/3338909932 conn(0x55e605cb02b0 msgr2=0x55e605cf8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 -- 172.21.15.28:0/3338909932 shutdown_connections 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.402+0000 7f632c07c740 1 -- 172.21.15.28:0/3338909932 wait complete. 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.403+0000 7f632c07c740 1 -- start start 2024-09-13T17:21:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.403+0000 7f632c07c740 1 Processor -- start 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605d007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cdc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605c15d60 con 0x55e605cdb030 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605bd4190 con 0x55e605cb3b80 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e605bd3e30 con 0x55e605cdecd0 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f632c07c740 10 client.0 _ll_register_callbacks cb 0x55e605c05ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60554/0 (socket says 172.21.15.28:60554) 2024-09-13T17:21:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.404+0000 7f6319400640 1 -- 172.21.15.28:0/3612584012 learned_addr learned my addr 172.21.15.28:0/3612584012 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6321e00640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605d007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6322800640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cdc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 -- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 msgr2=0x55e605cdc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cdc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 -- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 msgr2=0x55e605d007d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605d007d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 -- 172.21.15.28:0/3612584012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63240b1e60 con 0x55e605cdb030 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6319400640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f630c04e000 tx=0x7f630c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a7b87425dec7d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6307400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c049ea0 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f630c052670 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.405+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e605c3ff00 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f630c052830 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 10 client.10767 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55e605c143e0 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e605c3ff00 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f630c045070 con 0x55e605cdb030 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 10 client.10767 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 10 client.10767 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 20 client.10767 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605cfdc90 0x55e605cc7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 4 client.10767 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e605c08a40 con 0x55e605cfdc90 2024-09-13T17:21:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.406+0000 7f6321e00640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605cfdc90 0x55e605cc7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.407+0000 7f6321e00640 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605cfdc90 0x55e605cc7170 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=a3a0f543aa0b250b server_cookie=ef18cbc64d754245 in_seq=0 out_seq=0 2024-09-13T17:21:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.407+0000 7f6307400640 10 client.10767 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.407+0000 7f6307400640 1 -- 172.21.15.28:0/3612584012 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55e605c08a40 con 0x55e605cfdc90 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.407+0000 7f6307400640 10 client.10767 handle_command_reply: tid=0 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 client.10767 shutdown 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 msgr2=0x55e605cf5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f630c04e000 tx=0x7f630c004730 comp rx=0 tx=0).stop 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 shutdown_connections 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605cfdc90 0x55e605cc7170 crc :-1 s=READY pgs=9 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cdc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf5cc0 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 --2- 172.21.15.28:0/3612584012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cb3b80 0x55e605d007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f6307400640 0 client.10767 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 >> 172.21.15.28:0/3612584012 conn(0x55e605cb1340 msgr2=0x55e605cf9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 shutdown_connections 2024-09-13T17:21:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 1 -- 172.21.15.28:0/3612584012 wait complete. 2024-09-13T17:21:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.408+0000 7f632c07c740 20 client.10767 trim_cache size 0 max 16384 2024-09-13T17:21:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.423+0000 7f632c07c740 1 Processor -- start 2024-09-13T17:21:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.423+0000 7f632c07c740 1 -- start start 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 0x55e605cdfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 0x55e605cf9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e60597d180 con 0x55e605cdb030 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605cfd290 con 0x55e605cdecd0 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e605cc69d0 con 0x55e605cb3b80 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f6319400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60562/0 (socket says 172.21.15.28:60562) 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.424+0000 7f6319400640 1 -- 172.21.15.28:0/2741524699 learned_addr learned my addr 172.21.15.28:0/2741524699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6321e00640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 0x55e605cdfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6322800640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 0x55e605cf9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 -- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 msgr2=0x55e605cdfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 0x55e605cdfb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 -- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 msgr2=0x55e605cf9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 0x55e605cf9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 -- 172.21.15.28:0/2741524699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e605bb6200 con 0x55e605cdb030 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6322800640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 0x55e605cf9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6321e00640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 0x55e605cdfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6319400640 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f630c0328e0 tx=0x7f630c055040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713af168e4943fbe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6313400640 1 -- 172.21.15.28:0/2741524699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c052670 con 0x55e605cdb030 2024-09-13T17:21:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6313400640 1 -- 172.21.15.28:0/2741524699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630c0527d0 con 0x55e605cdb030 2024-09-13T17:21:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.425+0000 7f6313400640 1 -- 172.21.15.28:0/2741524699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630c05bd10 con 0x55e605cdb030 2024-09-13T17:21:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 -- 172.21.15.28:0/2741524699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 msgr2=0x55e605cf7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f630c0328e0 tx=0x7f630c055040 comp rx=0 tx=0).stop 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 -- 172.21.15.28:0/2741524699 shutdown_connections 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdecd0 0x55e605cf9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cdb030 0x55e605cf7d50 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 --2- 172.21.15.28:0/2741524699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cb3b80 0x55e605cdfb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 -- 172.21.15.28:0/2741524699 >> 172.21.15.28:0/2741524699 conn(0x55e605cb1340 msgr2=0x55e605cf9ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 -- 172.21.15.28:0/2741524699 shutdown_connections 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.426+0000 7f632c07c740 1 -- 172.21.15.28:0/2741524699 wait complete. 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.427+0000 7f632c07c740 1 -- start start 2024-09-13T17:21:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.427+0000 7f632c07c740 1 Processor -- start 2024-09-13T17:21:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdb030 0x55e605cc39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cc3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605cd98a0 con 0x55e605cb3b80 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e605cb2a20 con 0x55e605cdb030 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e605cd9ab0 con 0x55e605cdecd0 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f632c07c740 10 client.0 _ll_register_callbacks cb 0x7f630c041050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60566/0 (socket says 172.21.15.28:60566) 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 -- 172.21.15.28:0/2538063559 learned_addr learned my addr 172.21.15.28:0/2538063559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 -- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 msgr2=0x55e605cc3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cc3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 -- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdb030 msgr2=0x55e605cc39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdb030 0x55e605cc39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.428+0000 7f6321e00640 1 -- 172.21.15.28:0/2538063559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e605c143e0 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6321e00640 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f631400cec0 tx=0x7f6314057930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1f7bb63d6b9889 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631406f1d0 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6314056d50 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e605cfe3e0 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6314056f10 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 10 client.10773 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55e605bdb7d0 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e605cfe3e0 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f631405f710 con 0x55e605cb3b80 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 10 client.10773 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 10 client.10773 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 20 client.10773 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605d66b10 0x55e605d66f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 4 client.10773 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}] 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.429+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "scrub start", "path": "/scrub_control_test_path", "scrubops": ["recursive"]}) v1 -- 0x7f630c049840 con 0x55e605d66b10 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f6319400640 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605d66b10 0x55e605d66f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f6319400640 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605d66b10 0x55e605d66f10 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=e920e5c7391ccd6 server_cookie=f86f8110df4ad2e2 in_seq=0 out_seq=0 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f6307400640 10 client.10773 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f6307400640 1 -- 172.21.15.28:0/2538063559 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: -22 MDS is not active) v1 ==== 25+0+0 (crc 0 0 0) 0x7f630c049840 con 0x55e605d66b10 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f6307400640 10 client.10773 handle_command_reply: tid=0 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.430+0000 7f632c07c740 1 client.10773 shutdown 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 msgr2=0x55e605d62830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f631400cec0 tx=0x7f6314057930 comp rx=0 tx=0).stop 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 shutdown_connections 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e605d66b10 0x55e605d66f10 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e605cdecd0 0x55e605cc3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e605cdb030 0x55e605cc39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 --2- 172.21.15.28:0/2538063559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e605cb3b80 0x55e605d62830 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f6307400640 0 client.10773 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 >> 172.21.15.28:0/2538063559 conn(0x55e605cb1340 msgr2=0x55e605d11940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 shutdown_connections 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 1 -- 172.21.15.28:0/2538063559 wait complete. 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.431+0000 7f632c07c740 20 client.10773 trim_cache size 0 max 16384 2024-09-13T17:21:09.437 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: MDS is not active 2024-09-13T17:21:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T17:21:09.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c scrub pause 2024-09-13T17:21:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.610+0000 7f76dae00640 1 Processor -- start 2024-09-13T17:21:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.610+0000 7f76dae00640 1 -- start start 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.611+0000 7f76dae00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f76d4159770 0x7f76d406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76dae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d406f920 con 0x7f76d415cd80 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76dae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d415b290 con 0x7f76d406f3f0 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76dae00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d415b570 con 0x7f76d4159770 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76d8e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f76d415cd80 0x7f76d415a260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41368/0 (socket says 172.21.15.28:41368) 2024-09-13T17:21:09.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76d8e00640 1 -- 172.21.15.28:0/151496445 learned_addr learned my addr 172.21.15.28:0/151496445 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76d3e00640 1 --2- 172.21.15.28:0/151496445 >> v2:172.21.15.189:3300/0 conn(0x7f76d4159770 0x7f76d406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486645067 0 0) 0x7f76d406f920 con 0x7f76d415cd80 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.612+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76b8003520 con 0x7f76d415cd80 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1384339397 0 0) 0x7f76b8003520 con 0x7f76d415cd80 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 >> v2:172.21.15.189:3300/0 conn(0x7f76d4159770 msgr2=0x7f76d406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 --2- 172.21.15.28:0/151496445 >> v2:172.21.15.189:3300/0 conn(0x7f76d4159770 0x7f76d406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 >> v1:172.21.15.189:6789/0 conn(0x7f76d406f3f0 legacy=0x7f76d4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d415b850 con 0x7f76d415cd80 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f76cc0019f0 con 0x7f76d415cd80 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 >> v1:172.21.15.28:6789/0 conn(0x7f76d415cd80 legacy=0x7f76d415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76b80039c0 0x7f76b8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76b80051d0 0x7f76b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d415b570 con 0x7f76b80047c0 2024-09-13T17:21:09.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c120 con 0x7f76b80051d0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b800c2e0 con 0x7f76b80039c0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d3e00640 1 --2- 172.21.15.28:0/151496445 >> v2:172.21.15.189:3300/0 conn(0x7f76d4159770 0x7f76d406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76cb400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.613+0000 7f76d8e00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76b80051d0 0x7f76b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76d3e00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76b80039c0 0x7f76b8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76cb400640 1 -- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76b80039c0 msgr2=0x7f76b8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76cb400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76b80039c0 0x7f76b8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76cb400640 1 -- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76b80051d0 msgr2=0x7f76b800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76cb400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76b80051d0 0x7f76b800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76cb400640 1 -- 172.21.15.28:0/151496445 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b8003520 con 0x7f76b80047c0 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76d8e00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76b80051d0 0x7f76b800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76d8e00640 1 -- 172.21.15.28:0/151496445 reap_dead start 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.614+0000 7f76d3e00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76b80039c0 0x7f76b8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76cb400640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76b8004c40 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f76bc001720 tx=0x7f76b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931ae20f4775a96b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc0052d0 con 0x7f76b80047c0 2024-09-13T17:21:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc005430 con 0x7f76b80047c0 2024-09-13T17:21:09.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76d3400640 1 -- 172.21.15.28:0/151496445 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc02ab90 con 0x7f76b80047c0 2024-09-13T17:21:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76dae00640 1 -- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 msgr2=0x7f76b8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76dae00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76b8004c40 secure :-1 s=READY pgs=2186 cs=0 l=1 rev1=1 crypto rx=0x7f76bc001720 tx=0x7f76b80057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76dae00640 1 -- 172.21.15.28:0/151496445 shutdown_connections 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76dae00640 1 --2- 172.21.15.28:0/151496445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76b8004c40 unknown :-1 s=CLOSED pgs=2186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.615+0000 7f76dae00640 1 -- 172.21.15.28:0/151496445 >> 172.21.15.28:0/151496445 conn(0x7f76d40b0270 msgr2=0x7f76d40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.616+0000 7f76dae00640 1 -- 172.21.15.28:0/151496445 shutdown_connections 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.616+0000 7f76dae00640 1 -- 172.21.15.28:0/151496445 wait complete. 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.617+0000 7f76dae00640 1 Processor -- start 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.617+0000 7f76dae00640 1 -- start start 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76d4132d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76d40acb50 0x7f76d40acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d40c34e0 0x7f76d40c3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40c3ea0 con 0x7f76b80047c0 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40a84e0 con 0x7f76d40c34e0 2024-09-13T17:21:09.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76dae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d40a87c0 con 0x7f76d40acb50 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.618+0000 7f76d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76d4132d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76d4132d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60578/0 (socket says 172.21.15.28:60578) 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 -- 172.21.15.28:0/4182540952 learned_addr learned my addr 172.21.15.28:0/4182540952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76cb400640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76d40acb50 0x7f76d40acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d8e00640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d40c34e0 0x7f76d40c3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 -- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76d40acb50 msgr2=0x7f76d40acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f76d40acb50 0x7f76d40acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 -- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d40c34e0 msgr2=0x7f76d40c3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d40c34e0 0x7f76d40c3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 -- 172.21.15.28:0/4182540952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40a8aa0 con 0x7f76b80047c0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76b80047c0 0x7f76d4132d70 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f76c40108d0 tx=0x7f76c4010d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.619+0000 7f76d0c00640 1 -- 172.21.15.28:0/4182540952 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c400eb60 con 0x7f76b80047c0 2024-09-13T17:21:09.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.620+0000 7f76d0c00640 1 -- 172.21.15.28:0/4182540952 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c400ecc0 con 0x7f76b80047c0 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.620+0000 7f76d0c00640 1 -- 172.21.15.28:0/4182540952 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c401c400 con 0x7f76b80047c0 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.620+0000 7f76dae00640 1 -- 172.21.15.28:0/4182540952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d40a8d80 con 0x7f76b80047c0 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.620+0000 7f76dae00640 1 -- 172.21.15.28:0/4182540952 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d414f5d0 con 0x7f76b80047c0 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.622+0000 7f76d0c00640 1 -- 172.21.15.28:0/4182540952 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f76c401c560 con 0x7f76b80047c0 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.622+0000 7f76d0c00640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76b006ac00 0x7f76b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.622+0000 7f76d0c00640 1 -- 172.21.15.28:0/4182540952 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76c4089a10 con 0x7f76b80047c0 2024-09-13T17:21:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.623+0000 7f76cb400640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76b006ac00 0x7f76b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.623+0000 7f76ddcbe740 1 Processor -- start 2024-09-13T17:21:09.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.623+0000 7f76ddcbe740 1 -- start start 2024-09-13T17:21:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76cb400640 1 --2- 172.21.15.28:0/4182540952 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f76b006ac00 0x7f76b006d0c0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f76bc032000 tx=0x7f76bc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e4418d00 con 0x5577e44ad0a0 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e4416490 con 0x5577e44af8e0 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577e4411d80 con 0x5577e44f2e10 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.624+0000 7f76cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60582/0 (socket says 172.21.15.28:60582) 2024-09-13T17:21:09.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 -- 172.21.15.28:0/1285722569 learned_addr learned my addr 172.21.15.28:0/1285722569 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76d3e00640 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76d8e00640 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 -- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 msgr2=0x5577e44f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44f32c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 -- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 msgr2=0x5577e44afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 -- 172.21.15.28:0/1285722569 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e43c3b80 con 0x5577e44ad0a0 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.625+0000 7f76cb400640 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02a540 tx=0x7f76bc02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122c8efe0e0da3b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76c8c00640 1 -- 172.21.15.28:0/1285722569 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc02aea0 con 0x5577e44ad0a0 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76c8c00640 1 -- 172.21.15.28:0/1285722569 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc003b40 con 0x5577e44ad0a0 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76c8c00640 1 -- 172.21.15.28:0/1285722569 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc03d440 con 0x5577e44ad0a0 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1285722569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 msgr2=0x5577e44aa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f76bc02a540 tx=0x7f76bc02c000 comp rx=0 tx=0).stop 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1285722569 shutdown_connections 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44ad0a0 0x5577e44aa160 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1285722569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1285722569 >> 172.21.15.28:0/1285722569 conn(0x5577e44ac330 msgr2=0x5577e44f4760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1285722569 shutdown_connections 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.626+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1285722569 wait complete. 2024-09-13T17:21:09.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.627+0000 7f76ddcbe740 1 -- start start 2024-09-13T17:21:09.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.627+0000 7f76ddcbe740 1 Processor -- start 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44fb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44fb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e43b3270 con 0x5577e44c4dc0 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e43b2f10 con 0x5577e44af8e0 2024-09-13T17:21:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577e4178d70 con 0x5577e44f2e10 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76ddcbe740 10 client.0 _ll_register_callbacks cb 0x5577e440b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.628+0000 7f76d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60596/0 (socket says 172.21.15.28:60596) 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 -- 172.21.15.28:0/4226255768 learned_addr learned my addr 172.21.15.28:0/4226255768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44fb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76cb400640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44fb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 -- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 msgr2=0x5577e44fb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44fb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 -- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 msgr2=0x5577e44fb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44fb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 -- 172.21.15.28:0/4226255768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc036040 con 0x5577e44c4dc0 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76cb400640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76d8e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f76d406ce40 tx=0x7f76cc007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ceab8d77872214b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76c1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc01a070 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76cc011990 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.629+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577e4405080 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76cc011b50 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 10 client.10785 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x5577e4410400 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc023020 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f76cc01d020 con 0x5577e44c4dc0 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 10 client.10785 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 10 client.10785 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 20 client.10785 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e450f420 0x5577e4511810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 4 client.10785 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577e4105240 con 0x5577e450f420 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.630+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e450f420 0x5577e4511810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.631+0000 7f76d3e00640 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e450f420 0x5577e4511810 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=b486784efe71fa33 server_cookie=740d29c6c830a7cf in_seq=0 out_seq=0 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.631+0000 7f76c1600640 10 client.10785 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.631+0000 7f76c1600640 1 -- 172.21.15.28:0/4226255768 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5577e4105240 con 0x5577e450f420 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.631+0000 7f76c1600640 10 client.10785 handle_command_reply: tid=0 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.631+0000 7f76ddcbe740 1 client.10785 shutdown 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 msgr2=0x5577e44d7800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 secure :-1 s=READY pgs=2189 cs=0 l=1 rev1=1 crypto rx=0x7f76d406ce40 tx=0x7f76cc007a90 comp rx=0 tx=0).stop 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 shutdown_connections 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e450f420 0x5577e4511810 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44d7800 unknown :-1 s=CLOSED pgs=2189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44f2e10 0x5577e44fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76c1600640 0 client.10785 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/4226255768 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44af8e0 0x5577e44fb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 >> 172.21.15.28:0/4226255768 conn(0x5577e44ad0a0 msgr2=0x5577e44ab320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 shutdown_connections 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 1 -- 172.21.15.28:0/4226255768 wait complete. 2024-09-13T17:21:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.632+0000 7f76ddcbe740 20 client.10785 trim_cache size 0 max 16384 2024-09-13T17:21:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.647+0000 7f76ddcbe740 1 Processor -- start 2024-09-13T17:21:09.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.647+0000 7f76ddcbe740 1 -- start start 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e44f9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44c4dc0 0x5577e44f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e4410f10 con 0x5577e44f2e10 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e44f5220 con 0x5577e44c4dc0 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577e44f4640 con 0x5577e44af8e0 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76d8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60598/0 (socket says 172.21.15.28:60598) 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.648+0000 7f76d8e00640 1 -- 172.21.15.28:0/3655456064 learned_addr learned my addr 172.21.15.28:0/3655456064 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76cb400640 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44c4dc0 0x5577e44f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d3e00640 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e44f9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 -- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 msgr2=0x5577e44f9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e44f9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 -- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44c4dc0 msgr2=0x5577e44f59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44c4dc0 0x5577e44f59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 -- 172.21.15.28:0/3655456064 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577e43b2480 con 0x5577e44f2e10 2024-09-13T17:21:09.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76d8e00640 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f76cc007ad0 tx=0x7f76cc00a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a448a8b8ab59e489 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76c9600640 1 -- 172.21.15.28:0/3655456064 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc01a040 con 0x5577e44f2e10 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76c9600640 1 -- 172.21.15.28:0/3655456064 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5577e4405080 con 0x5577e44f2e10 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.649+0000 7f76c9600640 1 -- 172.21.15.28:0/3655456064 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc001ff0 con 0x5577e44f2e10 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 -- 172.21.15.28:0/3655456064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 msgr2=0x5577e44f5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f76cc007ad0 tx=0x7f76cc00a000 comp rx=0 tx=0).stop 2024-09-13T17:21:09.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 -- 172.21.15.28:0/3655456064 shutdown_connections 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44f2e10 0x5577e44f5ef0 unknown :-1 s=CLOSED pgs=2190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44c4dc0 0x5577e44f59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/3655456064 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e44f9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 -- 172.21.15.28:0/3655456064 >> 172.21.15.28:0/3655456064 conn(0x5577e44ad0a0 msgr2=0x5577e44d8c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 -- 172.21.15.28:0/3655456064 shutdown_connections 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.650+0000 7f76ddcbe740 1 -- 172.21.15.28:0/3655456064 wait complete. 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.651+0000 7f76ddcbe740 1 -- start start 2024-09-13T17:21:09.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.651+0000 7f76ddcbe740 1 Processor -- start 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e4515cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 0x5577e44f9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e455f1b0 con 0x5577e44c4dc0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577e44d8f40 con 0x5577e44f2e10 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5577e4515830 con 0x5577e44af8e0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76ddcbe740 10 client.0 _ll_register_callbacks cb 0x7f76cc0017f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60614/0 (socket says 172.21.15.28:60614) 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76cb400640 1 -- 172.21.15.28:0/1219282584 learned_addr learned my addr 172.21.15.28:0/1219282584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76d3e00640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e4515cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76d8e00640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 0x5577e44f9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76cb400640 1 -- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 msgr2=0x5577e4515cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.652+0000 7f76cb400640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e4515cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76cb400640 1 -- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 msgr2=0x5577e44f9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76cb400640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 0x5577e44f9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76cb400640 1 -- 172.21.15.28:0/1219282584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577e4410400 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76d8e00640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 0x5577e44f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76cb400640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f76bc001840 tx=0x7f76bc001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5886283fbd61c3e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76c1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76bc02aea0 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f76bc042050 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76bc003b40 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76bc03dac0 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76ddcbe740 10 client.10791 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.653+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x5577e43d7990 con 0x5577e44c4dc0 2024-09-13T17:21:09.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76bc003b40 con 0x5577e44c4dc0 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7f76bc03dc20 con 0x5577e44c4dc0 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 10 client.10791 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 10 client.10791 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 20 client.10791 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e455e020 0x5577e455e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 4 client.10791 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "scrub pause"}] 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "scrub pause"}) v1 -- 0x5577e4116510 con 0x5577e455e020 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76d3e00640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e455e020 0x5577e455e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76d3e00640 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e455e020 0x5577e455e400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=18fdc40694aa58aa server_cookie=7df6b4a9a7c50098 in_seq=0 out_seq=0 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.654+0000 7f76c1600640 10 client.10791 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76c1600640 1 -- 172.21.15.28:0/1219282584 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: -22 MDS is not active) v1 ==== 25+0+0 (crc 0 0 0) 0x5577e4116510 con 0x5577e455e020 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76c1600640 10 client.10791 handle_command_reply: tid=0 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 client.10791 shutdown 2024-09-13T17:21:09.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 msgr2=0x5577e44f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f76bc001840 tx=0x7f76bc001190 comp rx=0 tx=0).stop 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 shutdown_connections 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x5577e455e020 0x5577e455e400 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577e44f2e10 0x5577e44f9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577e44c4dc0 0x5577e44f8e30 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 --2- 172.21.15.28:0/1219282584 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5577e44af8e0 0x5577e4515cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 >> 172.21.15.28:0/1219282584 conn(0x5577e44ad0a0 msgr2=0x5577e44f5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76c1600640 0 client.10791 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 shutdown_connections 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 1 -- 172.21.15.28:0/1219282584 wait complete. 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.655+0000 7f76ddcbe740 20 client.10791 trim_cache size 0 max 16384 2024-09-13T17:21:09.661 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: MDS is not active 2024-09-13T17:21:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T17:21:09.664 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c scrub abort 2024-09-13T17:21:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.820+0000 7fea97e00640 1 Processor -- start 2024-09-13T17:21:09.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.820+0000 7fea97e00640 1 -- start start 2024-09-13T17:21:09.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea97e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fea90159770 0x7fea9006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea97e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea9015cd80 0x7fea90070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea97e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea9006f920 con 0x7fea9006f3f0 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea97e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9015b290 con 0x7fea9015cd80 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea97e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea9015b570 con 0x7fea90159770 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea95e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fea9015cd80 0x7fea90070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea94a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fea9006f3f0 0x7fea90159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41370/0 (socket says 172.21.15.28:41370) 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea94a00640 1 -- 172.21.15.28:0/534296815 learned_addr learned my addr 172.21.15.28:0/534296815 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.821+0000 7fea95400640 1 --2- 172.21.15.28:0/534296815 >> v2:172.21.15.189:3300/0 conn(0x7fea90159770 0x7fea9006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.822+0000 7fea95e00640 1 -- 172.21.15.28:0/534296815 >> v2:172.21.15.189:3300/0 conn(0x7fea90159770 msgr2=0x7fea9006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.822+0000 7fea95e00640 1 --2- 172.21.15.28:0/534296815 >> v2:172.21.15.189:3300/0 conn(0x7fea90159770 0x7fea9006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.822+0000 7fea95e00640 1 -- 172.21.15.28:0/534296815 >> v1:172.21.15.28:6789/0 conn(0x7fea9006f3f0 legacy=0x7fea90159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.822+0000 7fea95e00640 1 -- 172.21.15.28:0/534296815 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea9015b850 con 0x7fea9015cd80 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.823+0000 7fea95e00640 1 --2- 172.21.15.28:0/534296815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea90070260 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fea90067f80 tx=0x7fea8802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92c287a3f7a6c53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.823+0000 7fea8fe00640 1 -- 172.21.15.28:0/534296815 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88007740 con 0x7fea9015cd80 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.823+0000 7fea8fe00640 1 -- 172.21.15.28:0/534296815 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8802d050 con 0x7fea9015cd80 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.823+0000 7fea8fe00640 1 -- 172.21.15.28:0/534296815 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88007ac0 con 0x7fea9015cd80 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 -- 172.21.15.28:0/534296815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 msgr2=0x7fea90070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 --2- 172.21.15.28:0/534296815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea90070260 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7fea90067f80 tx=0x7fea8802a5f0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 -- 172.21.15.28:0/534296815 shutdown_connections 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 --2- 172.21.15.28:0/534296815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea90070260 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 --2- 172.21.15.28:0/534296815 >> v2:172.21.15.189:3300/0 conn(0x7fea90159770 0x7fea9006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 -- 172.21.15.28:0/534296815 >> 172.21.15.28:0/534296815 conn(0x7fea900b0270 msgr2=0x7fea900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 -- 172.21.15.28:0/534296815 shutdown_connections 2024-09-13T17:21:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.824+0000 7fea97e00640 1 -- 172.21.15.28:0/534296815 wait complete. 2024-09-13T17:21:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.825+0000 7fea97e00640 1 Processor -- start 2024-09-13T17:21:09.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 -- start start 2024-09-13T17:21:09.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea9006f3f0 0x7fea90143fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea90159770 0x7fea900b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea900cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea90144510 con 0x7fea9015cd80 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea900b33f0 con 0x7fea9006f3f0 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.826+0000 7fea97e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea9014e220 con 0x7fea90159770 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea900cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea900cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60628/0 (socket says 172.21.15.28:60628) 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 -- 172.21.15.28:0/1099990800 learned_addr learned my addr 172.21.15.28:0/1099990800 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea94a00640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea90159770 0x7fea900b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95400640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea9006f3f0 0x7fea90143fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 -- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea90159770 msgr2=0x7fea900b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fea90159770 0x7fea900b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 -- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea9006f3f0 msgr2=0x7fea90143fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea9006f3f0 0x7fea90143fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.827+0000 7fea95e00640 1 -- 172.21.15.28:0/1099990800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88009a70 con 0x7fea9015cd80 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea95e00640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea9015cd80 0x7fea900cc4c0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7fea8802abe0 tx=0x7fea88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea8d600640 1 -- 172.21.15.28:0/1099990800 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8803c830 con 0x7fea9015cd80 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea8d600640 1 -- 172.21.15.28:0/1099990800 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8803c990 con 0x7fea9015cd80 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea8d600640 1 -- 172.21.15.28:0/1099990800 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8803cd00 con 0x7fea9015cd80 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea97e00640 1 -- 172.21.15.28:0/1099990800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea900b2e30 con 0x7fea9015cd80 2024-09-13T17:21:09.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.828+0000 7fea97e00640 1 -- 172.21.15.28:0/1099990800 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea90145930 con 0x7fea9015cd80 2024-09-13T17:21:09.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.831+0000 7fea8d600640 1 -- 172.21.15.28:0/1099990800 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fea88031400 con 0x7fea9015cd80 2024-09-13T17:21:09.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.832+0000 7fea8d600640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea7006ac00 0x7fea7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.832+0000 7fea8d600640 1 -- 172.21.15.28:0/1099990800 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea880aaeb0 con 0x7fea9015cd80 2024-09-13T17:21:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.832+0000 7fea95400640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea7006ac00 0x7fea7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.832+0000 7fea9ac3f740 1 Processor -- start 2024-09-13T17:21:09.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.832+0000 7fea9ac3f740 1 -- start start 2024-09-13T17:21:09.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea95400640 1 --2- 172.21.15.28:0/1099990800 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea7006ac00 0x7fea7006d0c0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fea84009b40 tx=0x7fea84009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 0x562385b1fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b632c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385a88d00 con 0x562385b1d0a0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385a86490 con 0x562385b1f8e0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562385a81d80 con 0x562385b62e10 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60630/0 (socket says 172.21.15.28:60630) 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.833+0000 7fea94a00640 1 -- 172.21.15.28:0/657635496 learned_addr learned my addr 172.21.15.28:0/657635496 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea95400640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 0x562385b1fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea95e00640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b632c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 -- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 msgr2=0x562385b632c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b632c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 -- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 msgr2=0x562385b1fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 0x562385b1fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 -- 172.21.15.28:0/657635496 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562385a33b80 con 0x562385b1d0a0 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea95400640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 0x562385b1fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:09.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea94a00640 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001630 tx=0x7fea7c007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4df4e6fcdbb18b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea82000640 1 -- 172.21.15.28:0/657635496 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0058b0 con 0x562385b1d0a0 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea82000640 1 -- 172.21.15.28:0/657635496 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c005a10 con 0x562385b1d0a0 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.834+0000 7fea82000640 1 -- 172.21.15.28:0/657635496 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c005d80 con 0x562385b1d0a0 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 -- 172.21.15.28:0/657635496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 msgr2=0x562385b1a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7fea7c001630 tx=0x7fea7c007ec0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 -- 172.21.15.28:0/657635496 shutdown_connections 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b632c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1d0a0 0x562385b1a160 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/657635496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b1f8e0 0x562385b1fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 -- 172.21.15.28:0/657635496 >> 172.21.15.28:0/657635496 conn(0x562385b1c330 msgr2=0x562385b64760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 -- 172.21.15.28:0/657635496 shutdown_connections 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.835+0000 7fea9ac3f740 1 -- 172.21.15.28:0/657635496 wait complete. 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.836+0000 7fea9ac3f740 1 -- start start 2024-09-13T17:21:09.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.836+0000 7fea9ac3f740 1 Processor -- start 2024-09-13T17:21:09.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b49c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b65250 0x562385b656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385a23270 con 0x562385b1f8e0 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385a22f10 con 0x562385b65250 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5623857e8d70 con 0x562385b62e10 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea9ac3f740 10 client.0 _ll_register_callbacks cb 0x562385a7b3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea95400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60638/0 (socket says 172.21.15.28:60638) 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea95400640 1 -- 172.21.15.28:0/710301358 learned_addr learned my addr 172.21.15.28:0/710301358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea94a00640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b49c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.837+0000 7fea95e00640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b65250 0x562385b656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 -- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 msgr2=0x562385b49c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b49c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 -- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b65250 msgr2=0x562385b656d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b65250 0x562385b656d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 -- 172.21.15.28:0/710301358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea9006f920 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea94a00640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b49c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea95400640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fea840045c0 tx=0x7fea840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c25b34dcda8d833 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea76a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea84023440 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea840235a0 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562385a75080 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea84023760 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea9ac3f740 10 client.10806 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.838+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x562385a80400 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x562385a75080 con 0x562385b1f8e0 2024-09-13T17:21:09.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7fea840238c0 con 0x562385b1f8e0 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 10 client.10806 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 10 client.10806 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 20 client.10806 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b22370 0x562385b81190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 4 client.10806 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562385775240 con 0x562385b22370 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.839+0000 7fea94a00640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b22370 0x562385b81190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea94a00640 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b22370 0x562385b81190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=7bb3f2f457591f4f server_cookie=be005484175b0be3 in_seq=0 out_seq=0 2024-09-13T17:21:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea76a00640 10 client.10806 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea76a00640 1 -- 172.21.15.28:0/710301358 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x562385775240 con 0x562385b22370 2024-09-13T17:21:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea76a00640 10 client.10806 handle_command_reply: tid=0 2024-09-13T17:21:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea9ac3f740 1 client.10806 shutdown 2024-09-13T17:21:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 msgr2=0x562385b46a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.840+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fea840045c0 tx=0x7fea840046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 shutdown_connections 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b22370 0x562385b81190 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b65250 0x562385b656d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b62e10 0x562385b49c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/710301358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b1f8e0 0x562385b46a90 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 >> 172.21.15.28:0/710301358 conn(0x562385b1d0a0 msgr2=0x562385b63ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea76a00640 0 client.10806 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 shutdown_connections 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 1 -- 172.21.15.28:0/710301358 wait complete. 2024-09-13T17:21:09.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.841+0000 7fea9ac3f740 20 client.10806 trim_cache size 0 max 16384 2024-09-13T17:21:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.856+0000 7fea9ac3f740 1 Processor -- start 2024-09-13T17:21:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.856+0000 7fea9ac3f740 1 -- start start 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385b7d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b5a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385b14dc0 con 0x562385b65250 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385b766a0 con 0x562385b62e10 2024-09-13T17:21:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562385b48060 con 0x562385b1f8e0 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60646/0 (socket says 172.21.15.28:60646) 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 -- 172.21.15.28:0/3579236214 learned_addr learned my addr 172.21.15.28:0/3579236214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea94a00640 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b5a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95400640 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385b7d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 -- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 msgr2=0x562385b7d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385b7d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 -- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 msgr2=0x562385b5a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b5a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.857+0000 7fea95e00640 1 -- 172.21.15.28:0/3579236214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562385a22480 con 0x562385b65250 2024-09-13T17:21:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea95e00640 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fea880763f0 tx=0x7fea88079910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3fd2a0bf3daf4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea82a00640 1 -- 172.21.15.28:0/3579236214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea88082230 con 0x562385b65250 2024-09-13T17:21:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea82a00640 1 -- 172.21.15.28:0/3579236214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea88078290 con 0x562385b65250 2024-09-13T17:21:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea82a00640 1 -- 172.21.15.28:0/3579236214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8807a150 con 0x562385b65250 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea9ac3f740 1 -- 172.21.15.28:0/3579236214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 msgr2=0x562385b1da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.858+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fea880763f0 tx=0x7fea88079910 comp rx=0 tx=0).stop 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 -- 172.21.15.28:0/3579236214 shutdown_connections 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b1da50 unknown :-1 s=CLOSED pgs=2197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b5a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/3579236214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385b7d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 -- 172.21.15.28:0/3579236214 >> 172.21.15.28:0/3579236214 conn(0x562385b1d0a0 msgr2=0x562385b7d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 -- 172.21.15.28:0/3579236214 shutdown_connections 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 -- 172.21.15.28:0/3579236214 wait complete. 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 -- start start 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.859+0000 7fea9ac3f740 1 Processor -- start 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385bb1360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b67240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385b68a30 con 0x562385b65250 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562385b5b160 con 0x562385b62e10 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x562385bd24b0 con 0x562385b1f8e0 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.860+0000 7fea9ac3f740 10 client.0 _ll_register_callbacks cb 0x7fea840095b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:09.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385bb1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60654/0 (socket says 172.21.15.28:60654) 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea94a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b67240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 -- 172.21.15.28:0/912218696 learned_addr learned my addr 172.21.15.28:0/912218696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 -- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 msgr2=0x562385bb1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385bb1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 -- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 msgr2=0x562385b67240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b67240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 -- 172.21.15.28:0/912218696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea90145930 con 0x562385b65250 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea95e00640 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fea88079bf0 tx=0x7fea88009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2addde3a6cdadbae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.861+0000 7fea76a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea880795e0 con 0x562385b65250 2024-09-13T17:21:09.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea7c005a10 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fea88082230 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea88097280 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 10 client.10812 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x562385a47990 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea7c005a10 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7fea8804d070 con 0x562385b65250 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 10 client.10812 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 10 client.10812 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 20 client.10812 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b47b60 0x562385b1dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 4 client.10812 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "scrub abort"}] 2024-09-13T17:21:09.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.862+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "scrub abort"}) v1 -- 0x562385786510 con 0x562385b47b60 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea95400640 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b47b60 0x562385b1dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea95400640 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b47b60 0x562385b1dda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=5d4ede39d4855113 server_cookie=21c5ce2d7baf8798 in_seq=0 out_seq=0 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea76a00640 10 client.10812 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea76a00640 1 -- 172.21.15.28:0/912218696 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: -22 MDS is not active) v1 ==== 25+0+0 (crc 0 0 0) 0x562385786510 con 0x562385b47b60 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea76a00640 10 client.10812 handle_command_reply: tid=0 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.863+0000 7fea9ac3f740 1 client.10812 shutdown 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 msgr2=0x562385b67780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7fea88079bf0 tx=0x7fea88009750 comp rx=0 tx=0).stop 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 shutdown_connections 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x562385b47b60 0x562385b1dda0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562385b65250 0x562385b67780 unknown :-1 s=CLOSED pgs=2198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562385b62e10 0x562385b67240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 --2- 172.21.15.28:0/912218696 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x562385b1f8e0 0x562385bb1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:09.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 >> 172.21.15.28:0/912218696 conn(0x562385b1d0a0 msgr2=0x562385b64670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:09.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea76a00640 0 client.10812 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:09.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 shutdown_connections 2024-09-13T17:21:09.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 1 -- 172.21.15.28:0/912218696 wait complete. 2024-09-13T17:21:09.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:09.864+0000 7fea9ac3f740 20 client.10812 trim_cache size 0 max 16384 2024-09-13T17:21:09.872 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: MDS is not active 2024-09-13T17:21:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T17:21:09.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c scrub resume 2024-09-13T17:21:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.028+0000 7fecdd600640 1 Processor -- start 2024-09-13T17:21:10.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.028+0000 7fecdd600640 1 -- start start 2024-09-13T17:21:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecdd600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fecd815ca60 0x7fecd80cde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecdd600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd80cf9b0 0x7fecd80b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecdd600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd80d15d0 con 0x7fecd80d0e60 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecdd600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd80ce370 con 0x7fecd815ca60 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecdd600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd80d18b0 con 0x7fecd80cf9b0 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecd6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fecd815ca60 0x7fecd80cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.029+0000 7fecd6000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fecd80d0e60 0x7fecd80cf2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41374/0 (socket says 172.21.15.28:41374) 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd6000640 1 -- 172.21.15.28:0/200920328 learned_addr learned my addr 172.21.15.28:0/200920328 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:10.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd7400640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd80cf9b0 0x7fecd80b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd6a00640 1 -- 172.21.15.28:0/200920328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd80cf9b0 msgr2=0x7fecd80b5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd6a00640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd80cf9b0 0x7fecd80b5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd6a00640 1 -- 172.21.15.28:0/200920328 >> v1:172.21.15.28:6789/0 conn(0x7fecd80d0e60 legacy=0x7fecd80cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.030+0000 7fecd6a00640 1 -- 172.21.15.28:0/200920328 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd80d1b90 con 0x7fecd815ca60 2024-09-13T17:21:10.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.031+0000 7fecd6a00640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd815ca60 0x7fecd80cde30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097a0 tx=0x7fecc802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1da5914c4cb5a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.031+0000 7fecd5600640 1 -- 172.21.15.28:0/200920328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc802ae50 con 0x7fecd815ca60 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.031+0000 7fecd5600640 1 -- 172.21.15.28:0/200920328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc8032440 con 0x7fecd815ca60 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.031+0000 7fecd5600640 1 -- 172.21.15.28:0/200920328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc80327d0 con 0x7fecd815ca60 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 -- 172.21.15.28:0/200920328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd815ca60 msgr2=0x7fecd80cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd815ca60 0x7fecd80cde30 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fecc80097a0 tx=0x7fecc802a100 comp rx=0 tx=0).stop 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 -- 172.21.15.28:0/200920328 shutdown_connections 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd80cf9b0 0x7fecd80b5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 --2- 172.21.15.28:0/200920328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd815ca60 0x7fecd80cde30 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 -- 172.21.15.28:0/200920328 >> 172.21.15.28:0/200920328 conn(0x7fecd8070ad0 msgr2=0x7fecd80b4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 -- 172.21.15.28:0/200920328 shutdown_connections 2024-09-13T17:21:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.032+0000 7fecdd600640 1 -- 172.21.15.28:0/200920328 wait complete. 2024-09-13T17:21:10.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.034+0000 7fecdd600640 1 Processor -- start 2024-09-13T17:21:10.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.034+0000 7fecdd600640 1 -- start start 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd80cf9b0 0x7fecd814ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd80d0e60 0x7fecd80aeb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd815ca60 0x7fecd80aa110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd815b8c0 con 0x7fecd80d0e60 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd80aad20 con 0x7fecd80cf9b0 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecdd600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd80ab000 con 0x7fecd815ca60 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecd6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd80d0e60 0x7fecd80aeb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecd6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd80d0e60 0x7fecd80aeb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60670/0 (socket says 172.21.15.28:60670) 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecd6000640 1 -- 172.21.15.28:0/2901470673 learned_addr learned my addr 172.21.15.28:0/2901470673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd80cf9b0 0x7fecd814ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.035+0000 7fecd7400640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd815ca60 0x7fecd80aa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 -- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd815ca60 msgr2=0x7fecd80aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fecd815ca60 0x7fecd80aa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 -- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd80cf9b0 msgr2=0x7fecd814ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecd80cf9b0 0x7fecd814ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 -- 172.21.15.28:0/2901470673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc8009400 con 0x7fecd80d0e60 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecd6000640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecd80d0e60 0x7fecd80aeb20 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fecc400c450 tx=0x7fecc400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7feccea00640 1 -- 172.21.15.28:0/2901470673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4005ce0 con 0x7fecd80d0e60 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7feccea00640 1 -- 172.21.15.28:0/2901470673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc4005e40 con 0x7fecd80d0e60 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7feccea00640 1 -- 172.21.15.28:0/2901470673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc40072b0 con 0x7fecd80d0e60 2024-09-13T17:21:10.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecdd600640 1 -- 172.21.15.28:0/2901470673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd80ab2e0 con 0x7fecd80d0e60 2024-09-13T17:21:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.036+0000 7fecdd600640 1 -- 172.21.15.28:0/2901470673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd80ab8a0 con 0x7fecd80d0e60 2024-09-13T17:21:10.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.038+0000 7feccea00640 1 -- 172.21.15.28:0/2901470673 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fecc4007410 con 0x7fecd80d0e60 2024-09-13T17:21:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.039+0000 7feccea00640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecb806ac30 0x7fecb806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.039+0000 7feccea00640 1 -- 172.21.15.28:0/2901470673 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fecc4088860 con 0x7fecd80d0e60 2024-09-13T17:21:10.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.040+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecb806ac30 0x7fecb806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.040+0000 7fece040a740 1 Processor -- start 2024-09-13T17:21:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.040+0000 7fece040a740 1 -- start start 2024-09-13T17:21:10.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.040+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2901470673 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fecb806ac30 0x7fecb806d0f0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fecc8034000 tx=0x7fecc802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2d0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2cdb40 0x55e24e31d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e1db000 con 0x55e24e2f7830 2024-09-13T17:21:10.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e238d40 con 0x55e24e2cdb40 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e24e2364d0 con 0x55e24e2d0380 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60676/0 (socket says 172.21.15.28:60676) 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fecd7400640 1 -- 172.21.15.28:0/1033393119 learned_addr learned my addr 172.21.15.28:0/1033393119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fecd6000640 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2cdb40 0x55e24e31d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.041+0000 7fecd6a00640 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 -- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 msgr2=0x55e24e2d0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2d0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 -- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2cdb40 msgr2=0x55e24e31d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2cdb40 0x55e24e31d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 -- 172.21.15.28:0/1033393119 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24e1ed3a0 con 0x55e24e2f7830 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecd7400640 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fecd00114a0 tx=0x7fecd0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf91406ab40d460 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecc3400640 1 -- 172.21.15.28:0/1033393119 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd000e040 con 0x55e24e2f7830 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecc3400640 1 -- 172.21.15.28:0/1033393119 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd000f040 con 0x55e24e2f7830 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.042+0000 7fecc3400640 1 -- 172.21.15.28:0/1033393119 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0015a40 con 0x55e24e2f7830 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 -- 172.21.15.28:0/1033393119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 msgr2=0x55e24e31dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fecd00114a0 tx=0x7fecd0009750 comp rx=0 tx=0).stop 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 -- 172.21.15.28:0/1033393119 shutdown_connections 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e31dd90 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2cdb40 0x55e24e31d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 --2- 172.21.15.28:0/1033393119 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2d0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 -- 172.21.15.28:0/1033393119 >> 172.21.15.28:0/1033393119 conn(0x55e24e2ccab0 msgr2=0x55e24e314f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 -- 172.21.15.28:0/1033393119 shutdown_connections 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.043+0000 7fece040a740 1 -- 172.21.15.28:0/1033393119 wait complete. 2024-09-13T17:21:10.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.044+0000 7fece040a740 1 -- start start 2024-09-13T17:21:10.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.044+0000 7fece040a740 1 Processor -- start 2024-09-13T17:21:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e31a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e231dc0 con 0x55e24e2caf20 2024-09-13T17:21:10.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e1f03e0 con 0x55e24e2f7830 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e24e1f0080 con 0x55e24e2d0380 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fece040a740 10 client.0 _ll_register_callbacks cb 0x55e24e22b430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60690/0 (socket says 172.21.15.28:60690) 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 -- 172.21.15.28:0/2796121386 learned_addr learned my addr 172.21.15.28:0/2796121386 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd6000640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e31a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 -- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 msgr2=0x55e24e2ca9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2ca9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 -- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 msgr2=0x55e24e31a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e31a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd7400640 1 -- 172.21.15.28:0/2796121386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd0011230 con 0x55e24e2caf20 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:10.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.045+0000 7fecd6000640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e31a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecd7400640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fecd0011df0 tx=0x7fecd0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418195a1bf082ef9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecb7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd000e040 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fecd0015c00 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e24e25b600 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecd0015dc0 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fece040a740 10 client.10827 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55e24e230440 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.046+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55e24e25b600 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7fecd001b020 con 0x55e24e2caf20 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 10 client.10827 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 10 client.10827 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 20 client.10827 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:10.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e32ff10 0x55e24e332300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 4 client.10827 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e24e223c90 con 0x55e24e32ff10 2024-09-13T17:21:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e32ff10 0x55e24e332300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e32ff10 0x55e24e332300 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=dc9614a08ff4d2f server_cookie=b3f554700f227e28 in_seq=0 out_seq=0 2024-09-13T17:21:10.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.047+0000 7fecb7e00640 10 client.10827 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fecb7e00640 1 -- 172.21.15.28:0/2796121386 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55e24e223c90 con 0x55e24e32ff10 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fecb7e00640 10 client.10827 handle_command_reply: tid=0 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 client.10827 shutdown 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 msgr2=0x55e24e31a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fecd0011df0 tx=0x7fecd0004ab0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 shutdown_connections 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e32ff10 0x55e24e332300 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2caf20 0x55e24e31a6f0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e31a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 --2- 172.21.15.28:0/2796121386 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2d0380 0x55e24e2ca9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 >> 172.21.15.28:0/2796121386 conn(0x55e24e2cdb40 msgr2=0x55e24e3132a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.048+0000 7fecb7e00640 0 client.10827 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.049+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 shutdown_connections 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.049+0000 7fece040a740 1 -- 172.21.15.28:0/2796121386 wait complete. 2024-09-13T17:21:10.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.049+0000 7fece040a740 20 client.10827 trim_cache size 0 max 16384 2024-09-13T17:21:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.063+0000 7fece040a740 1 Processor -- start 2024-09-13T17:21:10.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.063+0000 7fece040a740 1 -- start start 2024-09-13T17:21:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e2cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 0x55e24e3131a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24df99040 con 0x55e24e2f7830 2024-09-13T17:21:10.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e3171e0 con 0x55e24e2d0380 2024-09-13T17:21:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e24e2fbd00 con 0x55e24e2caf20 2024-09-13T17:21:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.064+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60702/0 (socket says 172.21.15.28:60702) 2024-09-13T17:21:10.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 -- 172.21.15.28:0/448608578 learned_addr learned my addr 172.21.15.28:0/448608578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 -- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 msgr2=0x55e24e2cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd6a00640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e2cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd6000640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 0x55e24e3131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e2cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 -- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 msgr2=0x55e24e3131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 0x55e24e3131a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 -- 172.21.15.28:0/448608578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e24e1d25d0 con 0x55e24e2f7830 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd6a00640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e2cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd6000640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 0x55e24e3131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.065+0000 7fecd7400640 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fecd0028750 tx=0x7fecd0021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d6fd7c71dec647 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fecc3e00640 1 -- 172.21.15.28:0/448608578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd000e040 con 0x55e24e2f7830 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fecc3e00640 1 -- 172.21.15.28:0/448608578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0009750 con 0x55e24e2f7830 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fecc3e00640 1 -- 172.21.15.28:0/448608578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0015c40 con 0x55e24e2f7830 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 -- 172.21.15.28:0/448608578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 msgr2=0x55e24e319f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7fecd0028750 tx=0x7fecd0021000 comp rx=0 tx=0).stop 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 -- 172.21.15.28:0/448608578 shutdown_connections 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2f7830 0x55e24e319f70 unknown :-1 s=CLOSED pgs=2204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2d0380 0x55e24e3131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 --2- 172.21.15.28:0/448608578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e2cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 -- 172.21.15.28:0/448608578 >> 172.21.15.28:0/448608578 conn(0x55e24e2cdb40 msgr2=0x55e24e312690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 -- 172.21.15.28:0/448608578 shutdown_connections 2024-09-13T17:21:10.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.066+0000 7fece040a740 1 -- 172.21.15.28:0/448608578 wait complete. 2024-09-13T17:21:10.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.067+0000 7fece040a740 1 -- start start 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.067+0000 7fece040a740 1 Processor -- start 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e316400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e3131a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e37f440 con 0x55e24e2d0380 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e24e31a090 con 0x55e24e2f7830 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e24e2f39d0 con 0x55e24e2caf20 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fece040a740 10 client.0 _ll_register_callbacks cb 0x7fecd0009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fecd6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fecd6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60710/0 (socket says 172.21.15.28:60710) 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fecd6000640 1 -- 172.21.15.28:0/2928180663 learned_addr learned my addr 172.21.15.28:0/2928180663 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e316400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.068+0000 7fecd7400640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e3131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 -- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 msgr2=0x55e24e316400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e316400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 -- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 msgr2=0x55e24e3131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e3131a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 -- 172.21.15.28:0/2928180663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e24e230440 con 0x55e24e2d0380 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e316400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecd6000640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fecc4086c20 tx=0x7fecc405b510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7813c2a4d252761c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecb7e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4056bc0 con 0x55e24e2d0380 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fecc40868f0 con 0x55e24e2d0380 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecd000f040 con 0x55e24e2d0380 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecc4056d20 con 0x55e24e2d0380 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fece040a740 10 client.10833 fetch_fsmap learned FSMap version 154 2024-09-13T17:21:10.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.069+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=2+}) v3 -- 0x55e24e1f79b0 con 0x55e24e2d0380 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd000f040 con 0x55e24e2d0380 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 154) v1 ==== 3102+0+0 (secure 0 0 0) 0x7fecc405f580 con 0x55e24e2d0380 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 10 client.10833 fetch_fsmap finished waiting for FSMap version 154 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 10 client.10833 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 20 client.10833 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e336320 0x55e24e336700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 4 client.10833 mds_command: new command op to 10551 tid=0 multi_id=0 [{"prefix": "scrub resume"}] 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 --> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] -- command(tid 0: {"prefix": "scrub resume"}) v1 -- 0x55e24df254e0 con 0x55e24e336320 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.070+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e336320 0x55e24e336700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fecd6a00640 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e336320 0x55e24e336700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.10551 client_cookie=43bdd7a33a3a0a33 server_cookie=c698c1671322f8f9 in_seq=0 out_seq=0 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fecb7e00640 10 client.10833 ms_handle_connect on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fecb7e00640 1 -- 172.21.15.28:0/2928180663 <== mds.10551 v2:172.21.15.28:6832/3568343573 1 ==== command_reply(tid 0: -22 MDS is not active) v1 ==== 25+0+0 (crc 0 0 0) 0x55e24df254e0 con 0x55e24e336320 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fecb7e00640 10 client.10833 handle_command_reply: tid=0 2024-09-13T17:21:10.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 client.10833 shutdown 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 msgr2=0x55e24e367810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fecc4086c20 tx=0x7fecc405b510 comp rx=0 tx=0).stop 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 shutdown_connections 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:6832/3568343573,v1:172.21.15.28:6833/3568343573] conn(0x55e24e336320 0x55e24e336700 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e24e2f7830 0x55e24e3131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e24e2d0380 0x55e24e367810 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 --2- 172.21.15.28:0/2928180663 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55e24e2caf20 0x55e24e316400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 >> 172.21.15.28:0/2928180663 conn(0x55e24e2cdb40 msgr2=0x55e24e2cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fecb7e00640 0 client.10833 ms_handle_reset on v2:172.21.15.28:6832/3568343573 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 shutdown_connections 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 1 -- 172.21.15.28:0/2928180663 wait complete. 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.071+0000 7fece040a740 20 client.10833 trim_cache size 0 max 16384 2024-09-13T17:21:10.077 INFO:teuthology.orchestra.run.smithi028.stderr:Error EINVAL: MDS is not active 2024-09-13T17:21:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-13T17:21:10.081 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:21:10.081 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:21:10.174 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:21:10.273 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:21:10.273 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:10.273 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:10.309 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:10.309 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:10.384 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.577+0000 7f592a000640 1 Processor -- start 2024-09-13T17:21:10.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.577+0000 7f592a000640 1 -- start start 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.578+0000 7f592a000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f592415e860 0x7f5924161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f592a000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240b35c0 con 0x7f592406cde0 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f592a000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240b38a0 con 0x7f59240713a0 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f592a000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59241618f0 con 0x7f592415e860 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f591be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f592406cde0 0x7f5924070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41386/0 (socket says 172.21.15.28:41386) 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f591be00640 1 -- 172.21.15.28:0/1782234456 learned_addr learned my addr 172.21.15.28:0/1782234456 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f5923400640 1 --2- 172.21.15.28:0/1782234456 >> v2:172.21.15.189:3300/0 conn(0x7f592415e860 0x7f5924161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281579505 0 0) 0x7f59240b35c0 con 0x7f592406cde0 2024-09-13T17:21:10.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.579+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5908003520 con 0x7f592406cde0 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2833819585 0 0) 0x7f5908003520 con 0x7f592406cde0 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 >> v2:172.21.15.189:3300/0 conn(0x7f592415e860 msgr2=0x7f5924161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 --2- 172.21.15.28:0/1782234456 >> v2:172.21.15.189:3300/0 conn(0x7f592415e860 0x7f5924161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 >> v1:172.21.15.189:6789/0 conn(0x7f59240713a0 legacy=0x7f592406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924161230 con 0x7f592406cde0 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f590c001670 con 0x7f592406cde0 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 >> v1:172.21.15.28:6789/0 conn(0x7f592406cde0 legacy=0x7f5924070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080039c0 0x7f5908003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59080047c0 0x7f5908004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 0x7f590800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59241618f0 con 0x7f59080051d0 2024-09-13T17:21:10.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590800c120 con 0x7f59080047c0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590800c2e0 con 0x7f59080039c0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5923e00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 0x7f590800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f591be00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59080047c0 0x7f5908004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.580+0000 7f5923400640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080039c0 0x7f5908003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 -- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080039c0 msgr2=0x7f5908003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080039c0 0x7f5908003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 -- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59080047c0 msgr2=0x7f5908004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59080047c0 0x7f5908004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 -- 172.21.15.28:0/1782234456 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908003520 con 0x7f59080051d0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.581+0000 7f5923e00640 1 -- 172.21.15.28:0/1782234456 reap_dead start 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f5923e00640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 0x7f590800b980 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f591c001e20 tx=0x7f59080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8fe053af61e8598 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c02ae40 con 0x7f59080051d0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c003970 con 0x7f59080051d0 2024-09-13T17:21:10.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f5922a00640 1 -- 172.21.15.28:0/1782234456 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c003d20 con 0x7f59080051d0 2024-09-13T17:21:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 -- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 msgr2=0x7f590800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 0x7f590800b980 secure :-1 s=READY pgs=2207 cs=0 l=1 rev1=1 crypto rx=0x7f591c001e20 tx=0x7f59080057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 -- 172.21.15.28:0/1782234456 shutdown_connections 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 --2- 172.21.15.28:0/1782234456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59080051d0 0x7f590800b980 unknown :-1 s=CLOSED pgs=2207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 -- 172.21.15.28:0/1782234456 >> 172.21.15.28:0/1782234456 conn(0x7f5924069e80 msgr2=0x7f592406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 -- 172.21.15.28:0/1782234456 shutdown_connections 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.582+0000 7f592a000640 1 -- 172.21.15.28:0/1782234456 wait complete. 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.584+0000 7f592a000640 1 Processor -- start 2024-09-13T17:21:10.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.584+0000 7f592a000640 1 -- start start 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080051d0 0x7f5924156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924158ee0 0x7f59240a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924156eb0 con 0x7f59240a9c50 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240aa610 con 0x7f5924158ee0 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f592a000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59240a8d20 con 0x7f59080051d0 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.585+0000 7f5923e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60736/0 (socket says 172.21.15.28:60736) 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 -- 172.21.15.28:0/3955147618 learned_addr learned my addr 172.21.15.28:0/3955147618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923400640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080051d0 0x7f5924156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f591be00640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924158ee0 0x7f59240a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 -- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080051d0 msgr2=0x7f5924156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080051d0 0x7f5924156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 -- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924158ee0 msgr2=0x7f59240a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924158ee0 0x7f59240a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 -- 172.21.15.28:0/3955147618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59240a9000 con 0x7f59240a9c50 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f5923e00640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f591c001f40 tx=0x7f591c001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.586+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c006d50 con 0x7f59240a9c50 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.587+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c007030 con 0x7f59240a9c50 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.587+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c0073c0 con 0x7f59240a9c50 2024-09-13T17:21:10.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.587+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59240a92e0 con 0x7f59240a9c50 2024-09-13T17:21:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.587+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59240a8980 con 0x7f59240a9c50 2024-09-13T17:21:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.588+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59240b31a0 con 0x7f59240a9c50 2024-09-13T17:21:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.589+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f591c007520 con 0x7f59240a9c50 2024-09-13T17:21:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.589+0000 7f591b400640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 0x7f590006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.589+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f591c0a8a50 con 0x7f59240a9c50 2024-09-13T17:21:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.590+0000 7f5923400640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 0x7f590006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.591+0000 7f5923400640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 0x7f590006d0f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5914009a80 tx=0x7f5914008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.593+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f591c073c60 con 0x7f59240a9c50 2024-09-13T17:21:10.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.772+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f592406a670 con 0x7f59240a9c50 2024-09-13T17:21:10.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.774+0000 7f591b400640 1 -- 172.21.15.28:0/3955147618 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f591c077280 con 0x7f59240a9c50 2024-09-13T17:21:10.776 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:10.776 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:55.141358+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.2d","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 msgr2=0x7f590006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 0x7f590006d0f0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f5914009a80 tx=0x7f5914008040 comp rx=0 tx=0).stop 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 msgr2=0x7f59240aa0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f591c001f40 tx=0x7f591c001510 comp rx=0 tx=0).stop 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 shutdown_connections 2024-09-13T17:21:10.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590006ac30 0x7f590006d0f0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f59240a9c50 0x7f59240aa0d0 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5924158ee0 0x7f59240a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 --2- 172.21.15.28:0/3955147618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f59080051d0 0x7f5924156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 >> 172.21.15.28:0/3955147618 conn(0x7f5924069e80 msgr2=0x7f59241608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 shutdown_connections 2024-09-13T17:21:10.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.776+0000 7f592a000640 1 -- 172.21.15.28:0/3955147618 wait complete. 2024-09-13T17:21:10.794 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:20:37.568535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:20:38.601719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:21:10.794 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:21:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.951+0000 7fd7a6800640 1 Processor -- start 2024-09-13T17:21:10.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.951+0000 7fd7a6800640 1 -- start start 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a6800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7a00cd830 0x7fd7a0070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a6800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0071630 con 0x7fd7a0143910 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a6800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a0143700 con 0x7fd7a0143320 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a6800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7a0070a20 con 0x7fd7a00cd830 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a4a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd7a0143910 0x7fd7a00cd120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41400/0 (socket says 172.21.15.28:41400) 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a4a00640 1 -- 172.21.15.28:0/2171067021 learned_addr learned my addr 172.21.15.28:0/2171067021 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:10.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd7a5e00640 1 --2- 172.21.15.28:0/2171067021 >> v2:172.21.15.189:3300/0 conn(0x7fd7a00cd830 0x7fd7a0070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967794011 0 0) 0x7fd7a0071630 con 0x7fd7a0143910 2024-09-13T17:21:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.953+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788003520 con 0x7fd7a0143910 2024-09-13T17:21:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3307593820 0 0) 0x7fd788003520 con 0x7fd7a0143910 2024-09-13T17:21:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 >> v2:172.21.15.189:3300/0 conn(0x7fd7a00cd830 msgr2=0x7fd7a0070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 --2- 172.21.15.28:0/2171067021 >> v2:172.21.15.189:3300/0 conn(0x7fd7a00cd830 0x7fd7a0070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 >> v1:172.21.15.189:6789/0 conn(0x7fd7a0143320 legacy=0x7fd7a0071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0070d00 con 0x7fd7a0143910 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fd78c001670 con 0x7fd7a0143910 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 >> v1:172.21.15.28:6789/0 conn(0x7fd7a0143910 legacy=0x7fd7a00cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd788003970 0x7fd788003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 0x7fd78800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a0070a20 con 0x7fd788005180 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800c020 con 0x7fd7880047c0 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd78800c1e0 con 0x7fd788003970 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd7a5e00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 0x7fd78800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd7a4a00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.954+0000 7fd7a5400640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd788003970 0x7fd788003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 -- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd788003970 msgr2=0x7fd788003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd788003970 0x7fd788003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 -- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7880047c0 msgr2=0x7fd788004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7880047c0 0x7fd788004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 -- 172.21.15.28:0/2171067021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788003520 con 0x7fd788005180 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5400640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd788003970 0x7fd788003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.955+0000 7fd7a5e00640 1 -- 172.21.15.28:0/2171067021 reap_dead start 2024-09-13T17:21:10.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a5e00640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 0x7fd78800b980 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fd798009fc0 tx=0x7fd7880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959800df0664c273 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798003b90 con 0x7fd788005180 2024-09-13T17:21:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd79800f040 con 0x7fd788005180 2024-09-13T17:21:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd79fe00640 1 -- 172.21.15.28:0/2171067021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd798004020 con 0x7fd788005180 2024-09-13T17:21:10.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a6800640 1 -- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 msgr2=0x7fd78800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a6800640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 0x7fd78800b980 secure :-1 s=READY pgs=2210 cs=0 l=1 rev1=1 crypto rx=0x7fd798009fc0 tx=0x7fd7880057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a6800640 1 -- 172.21.15.28:0/2171067021 shutdown_connections 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a6800640 1 --2- 172.21.15.28:0/2171067021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd788005180 0x7fd78800b980 unknown :-1 s=CLOSED pgs=2210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.956+0000 7fd7a6800640 1 -- 172.21.15.28:0/2171067021 >> 172.21.15.28:0/2171067021 conn(0x7fd7a0069e80 msgr2=0x7fd7a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.957+0000 7fd7a6800640 1 -- 172.21.15.28:0/2171067021 shutdown_connections 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.957+0000 7fd7a6800640 1 -- 172.21.15.28:0/2171067021 wait complete. 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.958+0000 7fd7a6800640 1 Processor -- start 2024-09-13T17:21:10.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.958+0000 7fd7a6800640 1 -- start start 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd788005180 0x7fd7a01ed690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7a015ad00 0x7fd7a015b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01360b0 con 0x7fd7a015d880 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a01edbd0 con 0x7fd788005180 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.959+0000 7fd7a6800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a0138df0 con 0x7fd7a015ad00 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60754/0 (socket says 172.21.15.28:60754) 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 -- 172.21.15.28:0/1284318543 learned_addr learned my addr 172.21.15.28:0/1284318543 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a5400640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd788005180 0x7fd7a01ed690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 -- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7a015ad00 msgr2=0x7fd7a015b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7a015ad00 0x7fd7a015b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 -- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd788005180 msgr2=0x7fd7a01ed690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd788005180 0x7fd7a01ed690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 -- 172.21.15.28:0/1284318543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a01384c0 con 0x7fd7a015d880 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a4a00640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0072c0 tx=0x7fd78c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c0015d0 con 0x7fd7a015d880 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78c00f960 con 0x7fd7a015d880 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78c00fcd0 con 0x7fd7a015d880 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.960+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a01387a0 con 0x7fd7a015d880 2024-09-13T17:21:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.961+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a0135cf0 con 0x7fd7a015d880 2024-09-13T17:21:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.962+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a00b3800 con 0x7fd7a015d880 2024-09-13T17:21:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.963+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd78c01d360 con 0x7fd7a015d880 2024-09-13T17:21:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.963+0000 7fd79d600640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 0x7fd78006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:10.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.964+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd78c088bb0 con 0x7fd7a015d880 2024-09-13T17:21:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.964+0000 7fd7a5400640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 0x7fd78006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:10.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.965+0000 7fd7a5400640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 0x7fd78006d0c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd79400f740 tx=0x7fd794009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:10.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:10.968+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd78c053de0 con 0x7fd7a015d880 2024-09-13T17:21:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.147+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd7a0135e80 con 0x7fd7a015d880 2024-09-13T17:21:11.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.148+0000 7fd79d600640 1 -- 172.21.15.28:0/1284318543 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd78c01a070 con 0x7fd7a015d880 2024-09-13T17:21:11.150 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:21:11.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 msgr2=0x7fd78006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 0x7fd78006d0c0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd79400f740 tx=0x7fd794009040 comp rx=0 tx=0).stop 2024-09-13T17:21:11.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 msgr2=0x7fd7a01357b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fd78c0072c0 tx=0x7fd78c007780 comp rx=0 tx=0).stop 2024-09-13T17:21:11.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 shutdown_connections 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd78006ac00 0x7fd78006d0c0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7a015ad00 0x7fd7a015b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a015d880 0x7fd7a01357b0 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 --2- 172.21.15.28:0/1284318543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd788005180 0x7fd7a01ed690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 >> 172.21.15.28:0/1284318543 conn(0x7fd7a0069e80 msgr2=0x7fd7a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 shutdown_connections 2024-09-13T17:21:11.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.150+0000 7fd7a6800640 1 -- 172.21.15.28:0/1284318543 wait complete. 2024-09-13T17:21:11.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:21:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.326+0000 7fe333800640 1 Processor -- start 2024-09-13T17:21:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.326+0000 7fe333800640 1 -- start start 2024-09-13T17:21:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.327+0000 7fe333800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe32c0b2400 0x7fe32c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.327+0000 7fe333800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe32c0ad940 0x7fe32c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.327+0000 7fe333800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32c070470 con 0x7fe32c15c9a0 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.327+0000 7fe333800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c0ace30 con 0x7fe32c0b2400 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.327+0000 7fe333800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c0ad110 con 0x7fe32c0ad940 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe323e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe32c0b2400 0x7fe32c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe330e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe32c15c9a0 0x7fe32c0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41402/0 (socket says 172.21.15.28:41402) 2024-09-13T17:21:11.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe330e00640 1 -- 172.21.15.28:0/75200645 learned_addr learned my addr 172.21.15.28:0/75200645 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe331800640 1 --2- 172.21.15.28:0/75200645 >> v2:172.21.15.189:3300/0 conn(0x7fe32c0ad940 0x7fe32c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe323e00640 1 -- 172.21.15.28:0/75200645 >> v2:172.21.15.189:3300/0 conn(0x7fe32c0ad940 msgr2=0x7fe32c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe323e00640 1 --2- 172.21.15.28:0/75200645 >> v2:172.21.15.189:3300/0 conn(0x7fe32c0ad940 0x7fe32c06bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe323e00640 1 -- 172.21.15.28:0/75200645 >> v1:172.21.15.28:6789/0 conn(0x7fe32c15c9a0 legacy=0x7fe32c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe32be00640 1 -- 172.21.15.28:0/75200645 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1143335813 0 0) 0x7fe32c070470 con 0x7fe32c15c9a0 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.328+0000 7fe323e00640 1 -- 172.21.15.28:0/75200645 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c06b170 con 0x7fe32c0b2400 2024-09-13T17:21:11.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.329+0000 7fe323e00640 1 --2- 172.21.15.28:0/75200645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c0ac850 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fe32c160550 tx=0x7fe31402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407fef8d1c91ae84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe32be00640 1 -- 172.21.15.28:0/75200645 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314032440 con 0x7fe32c0b2400 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe32be00640 1 -- 172.21.15.28:0/75200645 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3140325a0 con 0x7fe32c0b2400 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe32be00640 1 -- 172.21.15.28:0/75200645 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314032930 con 0x7fe32c0b2400 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 -- 172.21.15.28:0/75200645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 msgr2=0x7fe32c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 --2- 172.21.15.28:0/75200645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c0ac850 secure :-1 s=READY pgs=2213 cs=0 l=1 rev1=1 crypto rx=0x7fe32c160550 tx=0x7fe31402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 -- 172.21.15.28:0/75200645 shutdown_connections 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 --2- 172.21.15.28:0/75200645 >> v2:172.21.15.189:3300/0 conn(0x7fe32c0ad940 0x7fe32c06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 --2- 172.21.15.28:0/75200645 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c0ac850 unknown :-1 s=CLOSED pgs=2213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 -- 172.21.15.28:0/75200645 >> 172.21.15.28:0/75200645 conn(0x7fe32c0b0270 msgr2=0x7fe32c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 -- 172.21.15.28:0/75200645 shutdown_connections 2024-09-13T17:21:11.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.330+0000 7fe333800640 1 -- 172.21.15.28:0/75200645 wait complete. 2024-09-13T17:21:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.332+0000 7fe333800640 1 Processor -- start 2024-09-13T17:21:11.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.332+0000 7fe333800640 1 -- start start 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 0x7fe32c1f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 0x7fe32c15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0cefa0 con 0x7fe32c0b2400 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c1f3810 con 0x7fe32c0ad940 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe333800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0ce420 con 0x7fe32c15c9a0 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe323e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe323e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60776/0 (socket says 172.21.15.28:60776) 2024-09-13T17:21:11.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe323e00640 1 -- 172.21.15.28:0/1657063088 learned_addr learned my addr 172.21.15.28:0/1657063088 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe330e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 0x7fe32c1f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.333+0000 7fe331800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 0x7fe32c15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 -- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 msgr2=0x7fe32c15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 0x7fe32c15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 -- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 msgr2=0x7fe32c1f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 0x7fe32c1f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 -- 172.21.15.28:0/1657063088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe314009400 con 0x7fe32c0b2400 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe331800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 0x7fe32c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe330e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 0x7fe32c1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe323e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fe314009890 tx=0x7fe3140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314004230 con 0x7fe32c0b2400 2024-09-13T17:21:11.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3140043b0 con 0x7fe32c0b2400 2024-09-13T17:21:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe314004520 con 0x7fe32c0b2400 2024-09-13T17:21:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c070930 con 0x7fe32c0b2400 2024-09-13T17:21:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.334+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c0ce9e0 con 0x7fe32c0b2400 2024-09-13T17:21:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.335+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c0cadf0 con 0x7fe32c0b2400 2024-09-13T17:21:11.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.336+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe314039050 con 0x7fe32c0b2400 2024-09-13T17:21:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.337+0000 7fe329600640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 0x7fe30806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.337+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe3140a9d70 con 0x7fe32c0b2400 2024-09-13T17:21:11.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.338+0000 7fe330e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 0x7fe30806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.338+0000 7fe330e00640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 0x7fe30806d0c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe31c000f60 tx=0x7fe31c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.342+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe314074f00 con 0x7fe32c0b2400 2024-09-13T17:21:11.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:21:11.402 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:21:11.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.520+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe32c0aca60 con 0x7fe32c0b2400 2024-09-13T17:21:11.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.521+0000 7fe329600640 1 -- 172.21.15.28:0/1657063088 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v181) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe31404b0a0 con 0x7fe32c0b2400 2024-09-13T17:21:11.523 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:21:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 msgr2=0x7fe30806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 0x7fe30806d0c0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fe31c000f60 tx=0x7fe31c009040 comp rx=0 tx=0).stop 2024-09-13T17:21:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 msgr2=0x7fe32c153fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7fe314009890 tx=0x7fe3140046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 shutdown_connections 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe30806ac00 0x7fe30806d0c0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe32c15c9a0 0x7fe32c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe32c0b2400 0x7fe32c153fb0 unknown :-1 s=CLOSED pgs=2214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 --2- 172.21.15.28:0/1657063088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe32c0ad940 0x7fe32c1f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 >> 172.21.15.28:0/1657063088 conn(0x7fe32c0b0270 msgr2=0x7fe32c15afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 shutdown_connections 2024-09-13T17:21:11.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.523+0000 7fe333800640 1 -- 172.21.15.28:0/1657063088 wait complete. 2024-09-13T17:21:11.540 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:21:11.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.740+0000 7fad0d200640 1 Processor -- start 2024-09-13T17:21:11.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.740+0000 7fad0d200640 1 -- start start 2024-09-13T17:21:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad0d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7fad0815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad0d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0815ab60 con 0x7fad08159770 2024-09-13T17:21:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad0d200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0806f920 con 0x7fad0806f3f0 2024-09-13T17:21:11.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad0d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0815b290 con 0x7fad08070430 2024-09-13T17:21:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad05e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fad08159770 0x7fad0806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41412/0 (socket says 172.21.15.28:41412) 2024-09-13T17:21:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.741+0000 7fad05e00640 1 -- 172.21.15.28:0/4207692943 learned_addr learned my addr 172.21.15.28:0/4207692943 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:11.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7facfd400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7fad0815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376672598 0 0) 0x7facf8001110 con 0x7fad08159770 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facec005730 con 0x7fad08159770 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442631572 0 0) 0x7fad0806f920 con 0x7fad0806f3f0 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facf8001110 con 0x7fad0806f3f0 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4133578494 0 0) 0x7facec005730 con 0x7fad08159770 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 msgr2=0x7fad0815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7fad0815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 >> v1:172.21.15.189:6789/0 conn(0x7fad0806f3f0 legacy=0x7fad0815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0815b570 con 0x7fad08159770 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7facf8001800 con 0x7fad08159770 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 >> v1:172.21.15.28:6789/0 conn(0x7fad08159770 legacy=0x7fad0806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facec005d40 0x7facec006140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7facec0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec006f10 0x7facec007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad0815b290 con 0x7facec006f10 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facec00dd20 con 0x7facec005d40 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.742+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facec00deb0 con 0x7fad08070430 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad06800640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec006f10 0x7facec007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facec005d40 0x7facec006140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7facfd400640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7facec0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 -- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 msgr2=0x7facec0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad08070430 0x7facec0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 -- 172.21.15.28:0/4207692943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec006f10 msgr2=0x7facec007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec006f10 0x7facec007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad05e00640 1 -- 172.21.15.28:0/4207692943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facec005730 con 0x7facec005d40 2024-09-13T17:21:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.743+0000 7fad06800640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec006f10 0x7facec007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:11.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.744+0000 7fad05e00640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facec005d40 0x7facec006140 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7facf8007f80 tx=0x7facec007a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5281c858430d608 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.744+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8005ad0 con 0x7facec005d40 2024-09-13T17:21:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.744+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf802d070 con 0x7facec005d40 2024-09-13T17:21:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.744+0000 7fad05400640 1 -- 172.21.15.28:0/4207692943 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8005e70 con 0x7facec005d40 2024-09-13T17:21:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 -- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facec005d40 msgr2=0x7facec006140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 --2- 172.21.15.28:0/4207692943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facec005d40 0x7facec006140 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7facf8007f80 tx=0x7facec007a00 comp rx=0 tx=0).stop 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad06800640 1 -- 172.21.15.28:0/4207692943 reap_dead start 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 -- 172.21.15.28:0/4207692943 shutdown_connections 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 -- 172.21.15.28:0/4207692943 >> 172.21.15.28:0/4207692943 conn(0x7fad080b0270 msgr2=0x7fad080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 -- 172.21.15.28:0/4207692943 shutdown_connections 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.745+0000 7fad0d200640 1 -- 172.21.15.28:0/4207692943 wait complete. 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.747+0000 7fad0d200640 1 Processor -- start 2024-09-13T17:21:11.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.747+0000 7fad0d200640 1 -- start start 2024-09-13T17:21:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad0814d160 0x7fad080ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad080acd90 0x7fad080cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080ad210 con 0x7facec005d40 2024-09-13T17:21:11.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad080cc440 con 0x7fad080acd90 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad0d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad08150f30 con 0x7fad0814d160 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad05e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad05e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60806/0 (socket says 172.21.15.28:60806) 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad05e00640 1 -- 172.21.15.28:0/3492810465 learned_addr learned my addr 172.21.15.28:0/3492810465 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7facfd400640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad0814d160 0x7fad080ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.748+0000 7fad06800640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad080acd90 0x7fad080cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 -- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad0814d160 msgr2=0x7fad080ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad0814d160 0x7fad080ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 -- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad080acd90 msgr2=0x7fad080cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad080acd90 0x7fad080cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 -- 172.21.15.28:0/3492810465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad08151210 con 0x7facec005d40 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad05e00640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7facf802abd0 tx=0x7facf8001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8004bf0 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8004d50 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad081514f0 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf8007290 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.749+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0814d5e0 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.750+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0806f920 con 0x7facec005d40 2024-09-13T17:21:11.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.751+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7facf802f030 con 0x7facec005d40 2024-09-13T17:21:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.752+0000 7facfea00640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 0x7face406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.752+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7facf80a8c70 con 0x7facec005d40 2024-09-13T17:21:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.753+0000 7facfd400640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 0x7face406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:11.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.753+0000 7facfd400640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 0x7face406d0c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fad08156700 tx=0x7facf0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:11.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.756+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7facf8073eb0 con 0x7facec005d40 2024-09-13T17:21:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.938+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fad0815aae0 con 0x7facec005d40 2024-09-13T17:21:11.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.938+0000 7facfea00640 1 -- 172.21.15.28:0/3492810465 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v181) v1 ==== 100+0+11 (secure 0 0 0) 0x7facf8044020 con 0x7facec005d40 2024-09-13T17:21:11.941 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:21:11.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 msgr2=0x7face406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 0x7face406d0c0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fad08156700 tx=0x7facf0009190 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 msgr2=0x7fad08151cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 secure :-1 s=READY pgs=2216 cs=0 l=1 rev1=1 crypto rx=0x7facf802abd0 tx=0x7facf8001ac0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 shutdown_connections 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7face406ac00 0x7face406d0c0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad080acd90 0x7fad080cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fad0814d160 0x7fad080ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 --2- 172.21.15.28:0/3492810465 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facec005d40 0x7fad08151cf0 unknown :-1 s=CLOSED pgs=2216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.940+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 >> 172.21.15.28:0/3492810465 conn(0x7fad080b0270 msgr2=0x7fad080b16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.941+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 shutdown_connections 2024-09-13T17:21:11.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:11.941+0000 7fad0d200640 1 -- 172.21.15.28:0/3492810465 wait complete. 2024-09-13T17:21:11.959 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:21:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.112+0000 7f21bae00640 1 Processor -- start 2024-09-13T17:21:12.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.112+0000 7f21bae00640 1 -- start start 2024-09-13T17:21:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.113+0000 7f21bae00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f21b4070430 0x7f21b415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.113+0000 7f21bae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b415ab60 con 0x7f21b4159770 2024-09-13T17:21:12.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.113+0000 7f21bae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b406f920 con 0x7f21b406f3f0 2024-09-13T17:21:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.113+0000 7f21bae00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b415b290 con 0x7f21b4070430 2024-09-13T17:21:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f21b4159770 0x7f21b406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41414/0 (socket says 172.21.15.28:41414) 2024-09-13T17:21:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3e00640 1 -- 172.21.15.28:0/48565317 learned_addr learned my addr 172.21.15.28:0/48565317 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3400640 1 --2- 172.21.15.28:0/48565317 >> v2:172.21.15.189:3300/0 conn(0x7f21b4070430 0x7f21b415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:12.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b2a00640 1 -- 172.21.15.28:0/48565317 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1620835932 0 0) 0x7f21a4001110 con 0x7f21b4159770 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b2a00640 1 -- 172.21.15.28:0/48565317 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2198003520 con 0x7f21b4159770 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3400640 1 -- 172.21.15.28:0/48565317 >> v1:172.21.15.189:6789/0 conn(0x7f21b406f3f0 legacy=0x7f21b415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3400640 1 -- 172.21.15.28:0/48565317 >> v1:172.21.15.28:6789/0 conn(0x7f21b4159770 legacy=0x7f21b406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.114+0000 7f21b3400640 1 -- 172.21.15.28:0/48565317 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b415b570 con 0x7f21b4070430 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.115+0000 7f21b3400640 1 --2- 172.21.15.28:0/48565317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b4070430 0x7f21b415bd70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3b1e386f8dc9f89 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21b2a00640 1 -- 172.21.15.28:0/48565317 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032440 con 0x7f21b4070430 2024-09-13T17:21:12.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21b2a00640 1 -- 172.21.15.28:0/48565317 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a00325a0 con 0x7f21b4070430 2024-09-13T17:21:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21b2a00640 1 -- 172.21.15.28:0/48565317 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0032930 con 0x7f21b4070430 2024-09-13T17:21:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 -- 172.21.15.28:0/48565317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b4070430 msgr2=0x7f21b415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 --2- 172.21.15.28:0/48565317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b4070430 0x7f21b415bd70 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f21a00097a0 tx=0x7f21a002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 -- 172.21.15.28:0/48565317 shutdown_connections 2024-09-13T17:21:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 --2- 172.21.15.28:0/48565317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b4070430 0x7f21b415bd70 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 -- 172.21.15.28:0/48565317 >> 172.21.15.28:0/48565317 conn(0x7f21b40b0270 msgr2=0x7f21b40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 -- 172.21.15.28:0/48565317 shutdown_connections 2024-09-13T17:21:12.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.116+0000 7f21bae00640 1 -- 172.21.15.28:0/48565317 wait complete. 2024-09-13T17:21:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.118+0000 7f21bae00640 1 Processor -- start 2024-09-13T17:21:12.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.118+0000 7f21bae00640 1 -- start start 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 0x7f21b41f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 0x7f21b41e8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b4155a30 con 0x7f21b4070430 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b41f2b80 con 0x7f21b406f3f0 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21bae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b4155ca0 con 0x7f21b4159770 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21b3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60820/0 (socket says 172.21.15.28:60820) 2024-09-13T17:21:12.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.119+0000 7f21b3400640 1 -- 172.21.15.28:0/903692082 learned_addr learned my addr 172.21.15.28:0/903692082 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 0x7f21b41f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 -- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 msgr2=0x7f21b41e8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b8e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 0x7f21b41e8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 0x7f21b41e8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 -- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 msgr2=0x7f21b41f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 0x7f21b41f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 -- 172.21.15.28:0/903692082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a0009400 con 0x7f21b4070430 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 0x7f21b41f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b8e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 0x7f21b41e8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:12.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.120+0000 7f21b3400640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f21a0035040 tx=0x7f21a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.121+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0006e10 con 0x7f21b4070430 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.121+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0032e90 con 0x7f21b4070430 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.121+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b4155f80 con 0x7f21b4070430 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.121+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21a0030be0 con 0x7f21b4070430 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.121+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b4156540 con 0x7f21b4070430 2024-09-13T17:21:12.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.122+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b406fd20 con 0x7f21b4070430 2024-09-13T17:21:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.123+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f21a0039040 con 0x7f21b4070430 2024-09-13T17:21:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.123+0000 7f21abe00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 0x7f219406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:12.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.124+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f21a00a9d60 con 0x7f21b4070430 2024-09-13T17:21:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.124+0000 7f21b3e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 0x7f219406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:12.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.125+0000 7f21b3e00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 0x7f219406d0c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f21a4001370 tx=0x7f21a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:12.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.127+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f21a0074f80 con 0x7f21b4070430 2024-09-13T17:21:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.322+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f21b406c660 con 0x7f21b4070430 2024-09-13T17:21:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.322+0000 7f21abe00640 1 -- 172.21.15.28:0/903692082 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v181) v1 ==== 80+0+0 (secure 0 0 0) 0x7f21a0030d40 con 0x7f21b4070430 2024-09-13T17:21:12.324 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:21:12.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 msgr2=0x7f219406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 0x7f219406d0c0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f21a4001370 tx=0x7f21a4009040 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 msgr2=0x7f21b4145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7f21a0035040 tx=0x7f21a00046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 shutdown_connections 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f219406ac00 0x7f219406d0c0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f21b4159770 0x7f21b41e8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21b4070430 0x7f21b4145e30 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 --2- 172.21.15.28:0/903692082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21b406f3f0 0x7f21b41f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 >> 172.21.15.28:0/903692082 conn(0x7f21b40b0270 msgr2=0x7f21b4153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 shutdown_connections 2024-09-13T17:21:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:12.324+0000 7f21bae00640 1 -- 172.21.15.28:0/903692082 wait complete. 2024-09-13T17:21:12.342 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:21:12.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:12.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:21:13.156 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:21:13.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:13.156 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:21:13.193 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:21:13.193 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:13.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:21:13.232 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:13.232 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:13.270 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:21:13.270 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:21:13.270 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:13.309 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:21:13.309 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:13.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.463+0000 7f46f3200640 1 Processor -- start 2024-09-13T17:21:13.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.464+0000 7f46f3200640 1 -- start start 2024-09-13T17:21:13.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46f3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46f4154a90 0x7f46f415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46f3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f415bb10 con 0x7f46f406b360 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46f3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f406b900 con 0x7f46f40b0b50 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46f3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f406bbe0 con 0x7f46f4154a90 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46f1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46f4154a90 0x7f46f415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46ebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f46f406b360 0x7f46f40b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:41430/0 (socket says 172.21.15.28:41430) 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.465+0000 7f46ebe00640 1 -- 172.21.15.28:0/2821339090 learned_addr learned my addr 172.21.15.28:0/2821339090 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.466+0000 7f46f1e00640 1 -- 172.21.15.28:0/2821339090 >> v1:172.21.15.189:6789/0 conn(0x7f46f40b0b50 legacy=0x7f46f40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.466+0000 7f46f1e00640 1 -- 172.21.15.28:0/2821339090 >> v1:172.21.15.28:6789/0 conn(0x7f46f406b360 legacy=0x7f46f40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.466+0000 7f46f1e00640 1 -- 172.21.15.28:0/2821339090 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f406bec0 con 0x7f46f4154a90 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f1e00640 1 --2- 172.21.15.28:0/2821339090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f415b5d0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f46e40097a0 tx=0x7f46e402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7712bf2b83404775 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f1400640 1 -- 172.21.15.28:0/2821339090 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4032440 con 0x7f46f4154a90 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f1400640 1 -- 172.21.15.28:0/2821339090 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e40325a0 con 0x7f46f4154a90 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f1400640 1 -- 172.21.15.28:0/2821339090 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4032930 con 0x7f46f4154a90 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f3200640 1 -- 172.21.15.28:0/2821339090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 msgr2=0x7f46f415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.467+0000 7f46f3200640 1 --2- 172.21.15.28:0/2821339090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f415b5d0 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f46e40097a0 tx=0x7f46e402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.468+0000 7f46f3200640 1 -- 172.21.15.28:0/2821339090 shutdown_connections 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.468+0000 7f46f3200640 1 --2- 172.21.15.28:0/2821339090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f415b5d0 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.468+0000 7f46f3200640 1 -- 172.21.15.28:0/2821339090 >> 172.21.15.28:0/2821339090 conn(0x7f46f4072e40 msgr2=0x7f46f4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:13.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.468+0000 7f46f3200640 1 -- 172.21.15.28:0/2821339090 shutdown_connections 2024-09-13T17:21:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.468+0000 7f46f3200640 1 -- 172.21.15.28:0/2821339090 wait complete. 2024-09-13T17:21:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.469+0000 7f46f3200640 1 Processor -- start 2024-09-13T17:21:13.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.469+0000 7f46f3200640 1 -- start start 2024-09-13T17:21:13.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f406b360 0x7f46f40ad710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f40b0b50 0x7f46f40a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40a9b10 con 0x7f46f4154a90 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40a8f00 con 0x7f46f40b0b50 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.470+0000 7f46f3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40a91e0 con 0x7f46f406b360 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60842/0 (socket says 172.21.15.28:60842) 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 -- 172.21.15.28:0/3885298943 learned_addr learned my addr 172.21.15.28:0/3885298943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46ebe00640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f40b0b50 0x7f46f40a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f1e00640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f406b360 0x7f46f40ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 -- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f406b360 msgr2=0x7f46f40ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f406b360 0x7f46f40ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 -- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f40b0b50 msgr2=0x7f46f40a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f40b0b50 0x7f46f40a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 -- 172.21.15.28:0/3885298943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e4009400 con 0x7f46f4154a90 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.471+0000 7f46f2800640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00c780 tx=0x7f46ec00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.472+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec004b10 con 0x7f46f4154a90 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.472+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec004c70 con 0x7f46f4154a90 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.472+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec00aa90 con 0x7f46f4154a90 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.472+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f40a94c0 con 0x7f46f4154a90 2024-09-13T17:21:13.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.472+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f40adc50 con 0x7f46f4154a90 2024-09-13T17:21:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.473+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f40a9cc0 con 0x7f46f4154a90 2024-09-13T17:21:13.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.474+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f46ec00abf0 con 0x7f46f4154a90 2024-09-13T17:21:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.474+0000 7f46ea000640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.475+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f46ec089370 con 0x7f46f4154a90 2024-09-13T17:21:13.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.475+0000 7f46f1e00640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.476+0000 7f46f1e00640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f46e4034000 tx=0x7f46e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.479+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46ec0545b0 con 0x7f46f4154a90 2024-09-13T17:21:13.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.692+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46f415b5d0 con 0x7f46f4154a90 2024-09-13T17:21:13.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.692+0000 7f46ea000640 1 -- 172.21.15.28:0/3885298943 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+4761 (secure 0 0 0) 0x7f46ec057bd0 con 0x7f46f4154a90 2024-09-13T17:21:13.694 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:13.695 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":154,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":154,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:21:08.173139+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10551":{"gid":10551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:21:13.695 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 154 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 msgr2=0x7f46d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f46e4034000 tx=0x7f46e402e000 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 msgr2=0x7f46f40af160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00c780 tx=0x7f46ec00cc40 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 shutdown_connections 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4154a90 0x7f46f40af160 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f40b0b50 0x7f46f40a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.694+0000 7f46f3200640 1 --2- 172.21.15.28:0/3885298943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f406b360 0x7f46f40ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.695+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 >> 172.21.15.28:0/3885298943 conn(0x7f46f4072e40 msgr2=0x7f46f4158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:13.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.695+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 shutdown_connections 2024-09-13T17:21:13.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.695+0000 7f46f3200640 1 -- 172.21.15.28:0/3885298943 wait complete. 2024-09-13T17:21:13.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.866+0000 7f6b0f000640 1 Processor -- start 2024-09-13T17:21:13.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.866+0000 7f6b0f000640 1 -- start start 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b081592a0 0x7f6b0815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0f000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b0806f970 0x7f6b08159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b080b2040 con 0x7f6b0806c2d0 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0f000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b0815b0b0 con 0x7f6b0806f970 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08159ff0 con 0x7f6b081592a0 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6b0806c2d0 0x7f6b080b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34642/0 (socket says 172.21.15.28:34642) 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6aff400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6b0806f970 0x7f6b08159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b0d000640 1 -- 172.21.15.28:0/3321450659 learned_addr learned my addr 172.21.15.28:0/3321450659 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:13.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6b07e00640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b081592a0 0x7f6b0815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6aff400640 1 -- 172.21.15.28:0/3321450659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b081592a0 msgr2=0x7f6b0815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:13.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.868+0000 7f6aff400640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b081592a0 0x7f6b0815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.869+0000 7f6aff400640 1 -- 172.21.15.28:0/3321450659 >> v1:172.21.15.28:6789/0 conn(0x7f6b0806c2d0 legacy=0x7f6b080b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.869+0000 7f6aff400640 1 -- 172.21.15.28:0/3321450659 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0815a2d0 con 0x7f6b0806f970 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.869+0000 7f6b07400640 1 -- 172.21.15.28:0/3321450659 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143907301 0 0) 0x7f6b080b2040 con 0x7f6b0806c2d0 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.869+0000 7f6aff400640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0806f970 0x7f6b08159680 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f3440b3c23731b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b07400640 1 -- 172.21.15.28:0/3321450659 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0032440 con 0x7f6b0806f970 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b07400640 1 -- 172.21.15.28:0/3321450659 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af00325a0 con 0x7f6b0806f970 2024-09-13T17:21:13.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b07400640 1 -- 172.21.15.28:0/3321450659 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0032930 con 0x7f6b0806f970 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 -- 172.21.15.28:0/3321450659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0806f970 msgr2=0x7f6b08159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0806f970 0x7f6b08159680 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7f6af00097a0 tx=0x7f6af002a180 comp rx=0 tx=0).stop 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 -- 172.21.15.28:0/3321450659 shutdown_connections 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b0806f970 0x7f6b08159680 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 --2- 172.21.15.28:0/3321450659 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b081592a0 0x7f6b0815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 -- 172.21.15.28:0/3321450659 >> 172.21.15.28:0/3321450659 conn(0x7f6b08069e80 msgr2=0x7f6b0806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.870+0000 7f6b0f000640 1 -- 172.21.15.28:0/3321450659 shutdown_connections 2024-09-13T17:21:13.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.871+0000 7f6b0f000640 1 -- 172.21.15.28:0/3321450659 wait complete. 2024-09-13T17:21:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.872+0000 7f6b0f000640 1 Processor -- start 2024-09-13T17:21:13.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.872+0000 7f6b0f000640 1 -- start start 2024-09-13T17:21:13.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b0806c2d0 0x7f6b08148bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0806f970 0x7f6b0814c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08152fd0 con 0x7f6b081592a0 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0814cff0 con 0x7f6b0806f970 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.873+0000 7f6b0f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b08153200 con 0x7f6b0806c2d0 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60914/0 (socket says 172.21.15.28:60914) 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 -- 172.21.15.28:0/604165691 learned_addr learned my addr 172.21.15.28:0/604165691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b07e00640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b0806c2d0 0x7f6b08148bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6aff400640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0806f970 0x7f6b0814c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 -- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b0806c2d0 msgr2=0x7f6b08148bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b0806c2d0 0x7f6b08148bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 -- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0806f970 msgr2=0x7f6b0814c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0806f970 0x7f6b0814c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 -- 172.21.15.28:0/604165691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af0009400 con 0x7f6b081592a0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.874+0000 7f6b0d000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c7e0 tx=0x7f6b0000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.875+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0000a640 con 0x7f6b081592a0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.875+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0000a7c0 con 0x7f6b081592a0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.875+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b081534a0 con 0x7f6b081592a0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.875+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0000ab70 con 0x7f6b081592a0 2024-09-13T17:21:13.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.875+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b08153a60 con 0x7f6b081592a0 2024-09-13T17:21:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.876+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b08153c50 con 0x7f6b081592a0 2024-09-13T17:21:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.877+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6b0000acd0 con 0x7f6b081592a0 2024-09-13T17:21:13.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.877+0000 7f6b04c00640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 0x7f6ae406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.878+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b00088c30 con 0x7f6b081592a0 2024-09-13T17:21:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.878+0000 7f6b07e00640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 0x7f6ae406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:13.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.879+0000 7f6b07e00640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 0x7f6ae406d0c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009a30 tx=0x7f6af8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:13.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:13.881+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6b00053dd0 con 0x7f6b081592a0 2024-09-13T17:21:14.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.061+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b0815ab70 con 0x7f6b081592a0 2024-09-13T17:21:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.063+0000 7f6b04c00640 1 -- 172.21.15.28:0/604165691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f6b00018020 con 0x7f6b081592a0 2024-09-13T17:21:14.065 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:14.065 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:55.141358+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.2d","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 msgr2=0x7f6ae406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 0x7f6ae406d0c0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009a30 tx=0x7f6af8008040 comp rx=0 tx=0).stop 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 msgr2=0x7f6b081e5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c7e0 tx=0x7f6b0000cca0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 shutdown_connections 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6ae406ac00 0x7f6ae406d0c0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b081592a0 0x7f6b081e5ec0 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0806f970 0x7f6b0814c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 --2- 172.21.15.28:0/604165691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6b0806c2d0 0x7f6b08148bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 >> 172.21.15.28:0/604165691 conn(0x7f6b08069e80 msgr2=0x7f6b081572c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 shutdown_connections 2024-09-13T17:21:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.065+0000 7f6b0f000640 1 -- 172.21.15.28:0/604165691 wait complete. 2024-09-13T17:21:14.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:20:37.568535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:20:38.601719+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:21:14.083 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:21:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.240+0000 7f64fb200640 1 Processor -- start 2024-09-13T17:21:14.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.240+0000 7f64fb200640 1 -- start start 2024-09-13T17:21:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.241+0000 7f64fb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.241+0000 7f64fb200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 0x7f64f415c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.241+0000 7f64fb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f415ab60 con 0x7f64f4159770 2024-09-13T17:21:14.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.241+0000 7f64fb200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f406f920 con 0x7f64f406f3f0 2024-09-13T17:21:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.241+0000 7f64fb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f415b290 con 0x7f64f4070430 2024-09-13T17:21:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64f4159770 0x7f64f406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34652/0 (socket says 172.21.15.28:34652) 2024-09-13T17:21:14.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3e00640 1 -- 172.21.15.28:0/1968044291 learned_addr learned my addr 172.21.15.28:0/1968044291 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f9200640 1 --2- 172.21.15.28:0/1968044291 >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 0x7f64f415c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3400640 1 --2- 172.21.15.28:0/1968044291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f2a00640 1 -- 172.21.15.28:0/1968044291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629756085 0 0) 0x7f64e4001110 con 0x7f64f4159770 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f2a00640 1 -- 172.21.15.28:0/1968044291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64d8003520 con 0x7f64f4159770 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3400640 1 -- 172.21.15.28:0/1968044291 >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 msgr2=0x7f64f415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3400640 1 --2- 172.21.15.28:0/1968044291 >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 0x7f64f415c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3400640 1 -- 172.21.15.28:0/1968044291 >> v1:172.21.15.28:6789/0 conn(0x7f64f4159770 legacy=0x7f64f406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f3400640 1 -- 172.21.15.28:0/1968044291 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f415b570 con 0x7f64f4070430 2024-09-13T17:21:14.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.242+0000 7f64f9200640 1 --2- 172.21.15.28:0/1968044291 >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 0x7f64f415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64f3400640 1 --2- 172.21.15.28:0/1968044291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f415bd70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8298088d8c8c13b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64f2a00640 1 -- 172.21.15.28:0/1968044291 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e002ae50 con 0x7f64f4070430 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64f2a00640 1 -- 172.21.15.28:0/1968044291 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0032440 con 0x7f64f4070430 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64f2a00640 1 -- 172.21.15.28:0/1968044291 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e00327b0 con 0x7f64f4070430 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64fb200640 1 -- 172.21.15.28:0/1968044291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 msgr2=0x7f64f415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.244+0000 7f64fb200640 1 --2- 172.21.15.28:0/1968044291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f415bd70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f64e00097a0 tx=0x7f64e002a100 comp rx=0 tx=0).stop 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 -- 172.21.15.28:0/1968044291 shutdown_connections 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 --2- 172.21.15.28:0/1968044291 >> v2:172.21.15.189:3300/0 conn(0x7f64f406f3f0 0x7f64f415c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 --2- 172.21.15.28:0/1968044291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f415bd70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 -- 172.21.15.28:0/1968044291 >> 172.21.15.28:0/1968044291 conn(0x7f64f40b0270 msgr2=0x7f64f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 -- 172.21.15.28:0/1968044291 shutdown_connections 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.245+0000 7f64fb200640 1 -- 172.21.15.28:0/1968044291 wait complete. 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.246+0000 7f64fb200640 1 Processor -- start 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.246+0000 7f64fb200640 1 -- start start 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f40a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 0x7f64f40acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40b1f80 con 0x7f64f406f3f0 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40b2820 con 0x7f64f4159770 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.247+0000 7f64fb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f41e6a90 con 0x7f64f4070430 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60930/0 (socket says 172.21.15.28:60930) 2024-09-13T17:21:14.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 -- 172.21.15.28:0/705093540 learned_addr learned my addr 172.21.15.28:0/705093540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3400640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f40a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f9200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 0x7f64f40acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 -- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 msgr2=0x7f64f40a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f40a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 -- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 msgr2=0x7f64f40acfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 0x7f64f40acfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 -- 172.21.15.28:0/705093540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec009a70 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f9200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 0x7f64f40acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.248+0000 7f64f3e00640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c3c0 tx=0x7f64e400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.249+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4005ce0 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.249+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4005e60 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.249+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4007310 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.249+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64e0009400 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.249+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f40b24b0 con 0x7f64f406f3f0 2024-09-13T17:21:14.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.250+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f406fd20 con 0x7f64f406f3f0 2024-09-13T17:21:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.251+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64e4007470 con 0x7f64f406f3f0 2024-09-13T17:21:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.252+0000 7f64ebe00640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 0x7f64d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.252+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f64e40887d0 con 0x7f64f406f3f0 2024-09-13T17:21:14.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.253+0000 7f64f3400640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 0x7f64d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.253+0000 7f64f3400640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 0x7f64d406d0c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f64e0034000 tx=0x7f64e002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.256+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f64e4053960 con 0x7f64f406f3f0 2024-09-13T17:21:14.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.432+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64f415ab60 con 0x7f64f406f3f0 2024-09-13T17:21:14.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.433+0000 7f64ebe00640 1 -- 172.21.15.28:0/705093540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v181) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64f415ab60 con 0x7f64f406f3f0 2024-09-13T17:21:14.435 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:21:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 msgr2=0x7f64d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 0x7f64d406d0c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f64e0034000 tx=0x7f64e002e000 comp rx=0 tx=0).stop 2024-09-13T17:21:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 msgr2=0x7f64f41e6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c3c0 tx=0x7f64e400c880 comp rx=0 tx=0).stop 2024-09-13T17:21:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 shutdown_connections 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f64d406ac00 0x7f64d406d0c0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64f4159770 0x7f64f40acfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f64f4070430 0x7f64f40a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 --2- 172.21.15.28:0/705093540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64f406f3f0 0x7f64f41e6460 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 >> 172.21.15.28:0/705093540 conn(0x7f64f40b0270 msgr2=0x7f64f40ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 shutdown_connections 2024-09-13T17:21:14.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.435+0000 7f64fb200640 1 -- 172.21.15.28:0/705093540 wait complete. 2024-09-13T17:21:14.453 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:21:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.681+0000 7f2cd5600640 1 Processor -- start 2024-09-13T17:21:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.681+0000 7f2cd5600640 1 -- start start 2024-09-13T17:21:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 0x7f2cd0159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2cd00abd10 0x7f2cd00ad320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cd00ad860 0x7f2cd00b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd0158f40 con 0x7f2cd00ad860 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd015a020 con 0x7f2cd00abd10 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.683+0000 7f2cd5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd00ac280 con 0x7f2cd0158b60 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cd00ad860 0x7f2cd00b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2cd00ad860 0x7f2cd00b1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60934/0 (socket says 172.21.15.28:60934) 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3157316155 learned_addr learned my addr 172.21.15.28:0/3157316155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:14.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2ccf400640 1 --2- 172.21.15.28:0/3157316155 >> v2:172.21.15.189:3300/0 conn(0x7f2cd00abd10 0x7f2cd00ad320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2ccfe00640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 0x7f2cd0159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3157316155 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 msgr2=0x7f2cd0159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 0x7f2cd0159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3157316155 >> v2:172.21.15.189:3300/0 conn(0x7f2cd00abd10 msgr2=0x7f2cd00ad320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3157316155 >> v2:172.21.15.189:3300/0 conn(0x7f2cd00abd10 0x7f2cd00ad320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3157316155 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd00ac560 con 0x7f2cd00ad860 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.684+0000 7f2ccfe00640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 0x7f2cd0159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.685+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00ad860 0x7f2cd00b1b50 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00cfbe0 tx=0x7f2cc802c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2a88cb1e4f1ea0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.686+0000 7f2ccea00640 1 -- 172.21.15.28:0/3157316155 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc802f070 con 0x7f2cd00ad860 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.686+0000 7f2ccea00640 1 -- 172.21.15.28:0/3157316155 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc800b2e0 con 0x7f2cd00ad860 2024-09-13T17:21:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.686+0000 7f2ccea00640 1 -- 172.21.15.28:0/3157316155 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8009820 con 0x7f2cd00ad860 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.686+0000 7f2cd5600640 1 -- 172.21.15.28:0/3157316155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00ad860 msgr2=0x7f2cd00b1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.686+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00ad860 0x7f2cd00b1b50 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00cfbe0 tx=0x7f2cc802c8a0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 -- 172.21.15.28:0/3157316155 shutdown_connections 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd00ad860 0x7f2cd00b1b50 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3157316155 >> v2:172.21.15.189:3300/0 conn(0x7f2cd00abd10 0x7f2cd00ad320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3157316155 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd0158b60 0x7f2cd0159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 -- 172.21.15.28:0/3157316155 >> 172.21.15.28:0/3157316155 conn(0x7f2cd0069e80 msgr2=0x7f2cd006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 -- 172.21.15.28:0/3157316155 shutdown_connections 2024-09-13T17:21:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.687+0000 7f2cd5600640 1 -- 172.21.15.28:0/3157316155 wait complete. 2024-09-13T17:21:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.688+0000 7f2cd5600640 1 Processor -- start 2024-09-13T17:21:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.688+0000 7f2cd5600640 1 -- start start 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd00abd10 0x7f2cd00ce6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 0x7f2cd00c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0099b80 con 0x7f2cd01540d0 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd0099e60 con 0x7f2cd0158b60 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.689+0000 7f2cd5600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd009a140 con 0x7f2cd00abd10 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60936/0 (socket says 172.21.15.28:60936) 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3179721743 learned_addr learned my addr 172.21.15.28:0/3179721743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2ccf400640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 0x7f2cd00c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2ccfe00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd00abd10 0x7f2cd00ce6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd00abd10 msgr2=0x7f2cd00ce6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd00abd10 0x7f2cd00ce6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 msgr2=0x7f2cd00c7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 0x7f2cd00c7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2cd4c00640 1 -- 172.21.15.28:0/3179721743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc800bd40 con 0x7f2cd01540d0 2024-09-13T17:21:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.690+0000 7f2ccf400640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 0x7f2cd00c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cd4c00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f2cc802ce90 tx=0x7f2cc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8034b40 con 0x7f2cd01540d0 2024-09-13T17:21:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc8034ca0 con 0x7f2cd01540d0 2024-09-13T17:21:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd009a420 con 0x7f2cd01540d0 2024-09-13T17:21:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc800b720 con 0x7f2cd01540d0 2024-09-13T17:21:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.691+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd0154760 con 0x7f2cd01540d0 2024-09-13T17:21:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.692+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd009a660 con 0x7f2cd01540d0 2024-09-13T17:21:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.693+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2cc803c430 con 0x7f2cd01540d0 2024-09-13T17:21:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.695+0000 7f2cc7e00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.695+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f2cc80aa400 con 0x7f2cd01540d0 2024-09-13T17:21:14.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.695+0000 7f2ccfe00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.696+0000 7f2ccfe00640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0006fd0 tx=0x7f2cc0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.699+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2cd009a660 con 0x7f2cd01540d0 2024-09-13T17:21:14.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.877+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cd0154760 con 0x7f2cd01540d0 2024-09-13T17:21:14.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.878+0000 7f2cc7e00640 1 -- 172.21.15.28:0/3179721743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v181) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cd0154760 con 0x7f2cd01540d0 2024-09-13T17:21:14.880 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:21:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 msgr2=0x7f2cb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0006fd0 tx=0x7f2cc0008040 comp rx=0 tx=0).stop 2024-09-13T17:21:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 msgr2=0x7f2cd00c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f2cc802ce90 tx=0x7f2cc8004a40 comp rx=0 tx=0).stop 2024-09-13T17:21:14.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 shutdown_connections 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2cb006ac00 0x7f2cb006d0c0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2cd01540d0 0x7f2cd00c8850 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2cd0158b60 0x7f2cd00c7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 --2- 172.21.15.28:0/3179721743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2cd00abd10 0x7f2cd00ce6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 >> 172.21.15.28:0/3179721743 conn(0x7f2cd0069e80 msgr2=0x7f2cd0155670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 shutdown_connections 2024-09-13T17:21:14.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:14.880+0000 7f2cd5600640 1 -- 172.21.15.28:0/3179721743 wait complete. 2024-09-13T17:21:14.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:21:15.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.054+0000 7f09d2c00640 1 Processor -- start 2024-09-13T17:21:15.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.054+0000 7f09d2c00640 1 -- start start 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 0x7f09cc071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09cc0cdf80 0x7f09cc15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09cc159e10 con 0x7f09cc072140 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc15a0f0 con 0x7f09cc0cdf80 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09cc15a3d0 con 0x7f09cc070e30 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d0e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f09cc072140 0x7f09cc0cd870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34654/0 (socket says 172.21.15.28:34654) 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d0e00640 1 -- 172.21.15.28:0/690122480 learned_addr learned my addr 172.21.15.28:0/690122480 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d2200640 1 --2- 172.21.15.28:0/690122480 >> v2:172.21.15.28:3300/0 conn(0x7f09cc0cdf80 0x7f09cc15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09d1800640 1 --2- 172.21.15.28:0/690122480 >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 0x7f09cc071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.056+0000 7f09cbe00640 1 -- 172.21.15.28:0/690122480 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1052637674 0 0) 0x7f09cc159e10 con 0x7f09cc072140 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09cbe00640 1 -- 172.21.15.28:0/690122480 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09b4003520 con 0x7f09cc072140 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09d2200640 1 -- 172.21.15.28:0/690122480 >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 msgr2=0x7f09cc071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09d2200640 1 --2- 172.21.15.28:0/690122480 >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 0x7f09cc071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09d2200640 1 -- 172.21.15.28:0/690122480 >> v1:172.21.15.28:6789/0 conn(0x7f09cc072140 legacy=0x7f09cc0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09d2200640 1 -- 172.21.15.28:0/690122480 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09cc15a6b0 con 0x7f09cc0cdf80 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.057+0000 7f09d1800640 1 --2- 172.21.15.28:0/690122480 >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 0x7f09cc071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2200640 1 --2- 172.21.15.28:0/690122480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0cdf80 0x7f09cc15aed0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f09cc06d430 tx=0x7f09c402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d452df23c8373cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09cbe00640 1 -- 172.21.15.28:0/690122480 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c402aea0 con 0x7f09cc0cdf80 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09cbe00640 1 -- 172.21.15.28:0/690122480 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c402d050 con 0x7f09cc0cdf80 2024-09-13T17:21:15.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09cbe00640 1 -- 172.21.15.28:0/690122480 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4007960 con 0x7f09cc0cdf80 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 -- 172.21.15.28:0/690122480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0cdf80 msgr2=0x7f09cc15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 --2- 172.21.15.28:0/690122480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0cdf80 0x7f09cc15aed0 secure :-1 s=READY pgs=2230 cs=0 l=1 rev1=1 crypto rx=0x7f09cc06d430 tx=0x7f09c402a680 comp rx=0 tx=0).stop 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 -- 172.21.15.28:0/690122480 shutdown_connections 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 --2- 172.21.15.28:0/690122480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc0cdf80 0x7f09cc15aed0 unknown :-1 s=CLOSED pgs=2230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 --2- 172.21.15.28:0/690122480 >> v2:172.21.15.189:3300/0 conn(0x7f09cc070e30 0x7f09cc071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.058+0000 7f09d2c00640 1 -- 172.21.15.28:0/690122480 >> 172.21.15.28:0/690122480 conn(0x7f09cc069e80 msgr2=0x7f09cc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.059+0000 7f09d2c00640 1 -- 172.21.15.28:0/690122480 shutdown_connections 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.059+0000 7f09d2c00640 1 -- 172.21.15.28:0/690122480 wait complete. 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.060+0000 7f09d2c00640 1 Processor -- start 2024-09-13T17:21:15.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.060+0000 7f09d2c00640 1 -- start start 2024-09-13T17:21:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09cc072140 0x7f09cc134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09cc0cdf80 0x7f09cc1e9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc152450 con 0x7f09cc070e30 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc150520 con 0x7f09cc0cdf80 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d2c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc152660 con 0x7f09cc072140 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60960/0 (socket says 172.21.15.28:60960) 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.061+0000 7f09d1800640 1 -- 172.21.15.28:0/115937234 learned_addr learned my addr 172.21.15.28:0/115937234 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d0e00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09cc072140 0x7f09cc134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d2200640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09cc0cdf80 0x7f09cc1e9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 -- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09cc072140 msgr2=0x7f09cc134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09cc072140 0x7f09cc134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 -- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09cc0cdf80 msgr2=0x7f09cc1e9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09cc0cdf80 0x7f09cc1e9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 -- 172.21.15.28:0/115937234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c0009400 con 0x7f09cc070e30 2024-09-13T17:21:15.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09d1800640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f09c000ebe0 tx=0x7f09c000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c0009760 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09c000cc30 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.062+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c000eee0 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.063+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c4009a70 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.063+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc152c20 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.064+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09cc0cd280 con 0x7f09cc070e30 2024-09-13T17:21:15.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.064+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09c0013470 con 0x7f09cc070e30 2024-09-13T17:21:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.065+0000 7f09c9600640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 0x7f09ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.065+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09c0089780 con 0x7f09cc070e30 2024-09-13T17:21:15.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.066+0000 7f09d0e00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 0x7f09ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.066+0000 7f09d0e00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 0x7f09ac06d0c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f09cc14ce30 tx=0x7f09b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.070+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f09c0054880 con 0x7f09cc070e30 2024-09-13T17:21:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.249+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09cc06f720 con 0x7f09cc070e30 2024-09-13T17:21:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.250+0000 7f09c9600640 1 -- 172.21.15.28:0/115937234 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v181) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09c0057ea0 con 0x7f09cc070e30 2024-09-13T17:21:15.252 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:21:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 msgr2=0x7f09ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 0x7f09ac06d0c0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f09cc14ce30 tx=0x7f09b8009040 comp rx=0 tx=0).stop 2024-09-13T17:21:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 msgr2=0x7f09cc14ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 secure :-1 s=READY pgs=2231 cs=0 l=1 rev1=1 crypto rx=0x7f09c000ebe0 tx=0x7f09c000c520 comp rx=0 tx=0).stop 2024-09-13T17:21:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 shutdown_connections 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09ac06ac00 0x7f09ac06d0c0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09cc0cdf80 0x7f09cc1e9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09cc072140 0x7f09cc134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 --2- 172.21.15.28:0/115937234 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09cc070e30 0x7f09cc14ffe0 unknown :-1 s=CLOSED pgs=2231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 >> 172.21.15.28:0/115937234 conn(0x7f09cc069e80 msgr2=0x7f09cc15bab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 shutdown_connections 2024-09-13T17:21:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.252+0000 7f09d2c00640 1 -- 172.21.15.28:0/115937234 wait complete. 2024-09-13T17:21:15.270 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:15.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.423+0000 7fa951000640 1 Processor -- start 2024-09-13T17:21:15.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.424+0000 7fa951000640 1 -- start start 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa951000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa94c06ba70 0x7fa94c0afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa951000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94c0b39d0 con 0x7fa94c158b20 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa951000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94c06c880 con 0x7fa94c06c3f0 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa951000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94c0b2dc0 con 0x7fa94c06ba70 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa94b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa94c158b20 0x7fa94c06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34662/0 (socket says 172.21.15.28:34662) 2024-09-13T17:21:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa94b400640 1 -- 172.21.15.28:0/2330892129 learned_addr learned my addr 172.21.15.28:0/2330892129 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa942a00640 1 --2- 172.21.15.28:0/2330892129 >> v2:172.21.15.189:3300/0 conn(0x7fa94c06ba70 0x7fa94c0afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.425+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 583304580 0 0) 0x7fa94c0b39d0 con 0x7fa94c158b20 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa930003520 con 0x7fa94c158b20 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3352941499 0 0) 0x7fa930003520 con 0x7fa94c158b20 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 >> v2:172.21.15.189:3300/0 conn(0x7fa94c06ba70 msgr2=0x7fa94c0afbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 --2- 172.21.15.28:0/2330892129 >> v2:172.21.15.189:3300/0 conn(0x7fa94c06ba70 0x7fa94c0afbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 >> v1:172.21.15.189:6789/0 conn(0x7fa94c06c3f0 legacy=0x7fa94c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0b30a0 con 0x7fa94c158b20 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa93c001670 con 0x7fa94c158b20 2024-09-13T17:21:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 >> v1:172.21.15.28:6789/0 conn(0x7fa94c158b20 legacy=0x7fa94c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9300039a0 0x7fa930003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 0x7fa93000ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c0b2dc0 con 0x7fa9300051d0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93000d210 con 0x7fa9300047c0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93000d3d0 con 0x7fa9300039a0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94be00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 0x7fa93000ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa94b400640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9300039a0 0x7fa930003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.426+0000 7fa942a00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa94be00640 1 -- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9300039a0 msgr2=0x7fa930003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa94be00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa9300039a0 0x7fa930003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa94be00640 1 -- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300047c0 msgr2=0x7fa930004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa94be00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa94be00640 1 -- 172.21.15.28:0/2330892129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930003520 con 0x7fa9300051d0 2024-09-13T17:21:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.427+0000 7fa942a00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300047c0 0x7fa930004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa94be00640 1 -- 172.21.15.28:0/2330892129 reap_dead start 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa94be00640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 0x7fa93000ca90 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fa944001a30 tx=0x7fa9300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc873eba54656c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944003b70 con 0x7fa9300051d0 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94402f050 con 0x7fa9300051d0 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa94aa00640 1 -- 172.21.15.28:0/2330892129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9440050d0 con 0x7fa9300051d0 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa951000640 1 -- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 msgr2=0x7fa93000ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.428+0000 7fa951000640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 0x7fa93000ca90 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7fa944001a30 tx=0x7fa9300057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.429+0000 7fa951000640 1 -- 172.21.15.28:0/2330892129 shutdown_connections 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.429+0000 7fa951000640 1 --2- 172.21.15.28:0/2330892129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9300051d0 0x7fa93000ca90 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.429+0000 7fa951000640 1 -- 172.21.15.28:0/2330892129 >> 172.21.15.28:0/2330892129 conn(0x7fa94c072e40 msgr2=0x7fa94c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.429+0000 7fa951000640 1 -- 172.21.15.28:0/2330892129 shutdown_connections 2024-09-13T17:21:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.429+0000 7fa951000640 1 -- 172.21.15.28:0/2330892129 wait complete. 2024-09-13T17:21:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.430+0000 7fa951000640 1 Processor -- start 2024-09-13T17:21:15.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 -- start start 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 0x7fa94c0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 0x7fa94c0af0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c14b590 con 0x7fa94c154bf0 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c1555b0 con 0x7fa9300051d0 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.431+0000 7fa951000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c13b0f0 con 0x7fa94c0aec20 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60974/0 (socket says 172.21.15.28:60974) 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 -- 172.21.15.28:0/3438722658 learned_addr learned my addr 172.21.15.28:0/3438722658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa942a00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 0x7fa94c0af0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94b400640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 0x7fa94c0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 -- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 msgr2=0x7fa94c0af0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 0x7fa94c0af0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 -- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 msgr2=0x7fa94c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 0x7fa94c0ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 -- 172.21.15.28:0/3438722658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0ad7a0 con 0x7fa94c154bf0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa942a00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 0x7fa94c0af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94b400640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 0x7fa94c0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.432+0000 7fa94be00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fa944001f10 tx=0x7fa9440014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.433+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944038070 con 0x7fa94c154bf0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.433+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa944003b80 con 0x7fa94c154bf0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.433+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa944006d50 con 0x7fa94c154bf0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.433+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c13a4e0 con 0x7fa94c154bf0 2024-09-13T17:21:15.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.433+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c13aaa0 con 0x7fa94c154bf0 2024-09-13T17:21:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.434+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c1380e0 con 0x7fa94c154bf0 2024-09-13T17:21:15.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.435+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa944007030 con 0x7fa94c154bf0 2024-09-13T17:21:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.436+0000 7fa943e00640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 0x7fa92806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.436+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9440a8bc0 con 0x7fa94c154bf0 2024-09-13T17:21:15.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.437+0000 7fa94b400640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 0x7fa92806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.437+0000 7fa94b400640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 0x7fa92806d0f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa93c007c60 tx=0x7fa93c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.441+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa94c1380e0 con 0x7fa94c154bf0 2024-09-13T17:21:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.654+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa94c1380e0 con 0x7fa94c154bf0 2024-09-13T17:21:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.654+0000 7fa943e00640 1 -- 172.21.15.28:0/3438722658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+4762 (secure 0 0 0) 0x7fa94c1380e0 con 0x7fa94c154bf0 2024-09-13T17:21:15.657 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:15.657 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":155,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":155,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:21:15.201092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10551":{"gid":10551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":11,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:21:15.657 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 155 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 msgr2=0x7fa92806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 0x7fa92806d0f0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa93c007c60 tx=0x7fa93c009040 comp rx=0 tx=0).stop 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 msgr2=0x7fa94c155070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7fa944001f10 tx=0x7fa9440014a0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 shutdown_connections 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa92806ac30 0x7fa92806d0f0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa94c154bf0 0x7fa94c155070 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa94c0aec20 0x7fa94c0af0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 --2- 172.21.15.28:0/3438722658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9300051d0 0x7fa94c0ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 >> 172.21.15.28:0/3438722658 conn(0x7fa94c072e40 msgr2=0x7fa94c155f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 shutdown_connections 2024-09-13T17:21:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.657+0000 7fa951000640 1 -- 172.21.15.28:0/3438722658 wait complete. 2024-09-13T17:21:15.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.829+0000 7f0f4b600640 1 Processor -- start 2024-09-13T17:21:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.829+0000 7f0f4b600640 1 -- start start 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f4b600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0f4406f970 0x7f0f4415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f4b600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0f4406c2d0 0x7f0f4406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f4b600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4406cc70 con 0x7f0f441592a0 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f4b600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f4415b0b0 con 0x7f0f4406f970 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f4b600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f44159d90 con 0x7f0f4406c2d0 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f48c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0f441592a0 0x7f0f44159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34668/0 (socket says 172.21.15.28:34668) 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f48c00640 1 -- 172.21.15.28:0/564015543 learned_addr learned my addr 172.21.15.28:0/564015543 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f3be00640 1 --2- 172.21.15.28:0/564015543 >> v2:172.21.15.28:3300/0 conn(0x7f0f4406f970 0x7f0f4415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f49600640 1 --2- 172.21.15.28:0/564015543 >> v2:172.21.15.189:3300/0 conn(0x7f0f4406c2d0 0x7f0f4406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f43e00640 1 -- 172.21.15.28:0/564015543 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194729819 0 0) 0x7f0f4406cc70 con 0x7f0f441592a0 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f43e00640 1 -- 172.21.15.28:0/564015543 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f28003520 con 0x7f0f441592a0 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f3be00640 1 -- 172.21.15.28:0/564015543 >> v2:172.21.15.189:3300/0 conn(0x7f0f4406c2d0 msgr2=0x7f0f4406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f3be00640 1 --2- 172.21.15.28:0/564015543 >> v2:172.21.15.189:3300/0 conn(0x7f0f4406c2d0 0x7f0f4406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f3be00640 1 -- 172.21.15.28:0/564015543 >> v1:172.21.15.28:6789/0 conn(0x7f0f441592a0 legacy=0x7f0f44159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.831+0000 7f0f3be00640 1 -- 172.21.15.28:0/564015543 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f440b2040 con 0x7f0f4406f970 2024-09-13T17:21:15.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f3be00640 1 --2- 172.21.15.28:0/564015543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4406f970 0x7f0f4415ab70 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f0f440b1dd0 tx=0x7f0f2c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f3d11708798bc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f43e00640 1 -- 172.21.15.28:0/564015543 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c032440 con 0x7f0f4406f970 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f43e00640 1 -- 172.21.15.28:0/564015543 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c0325a0 con 0x7f0f4406f970 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f43e00640 1 -- 172.21.15.28:0/564015543 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c032910 con 0x7f0f4406f970 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 -- 172.21.15.28:0/564015543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4406f970 msgr2=0x7f0f4415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 --2- 172.21.15.28:0/564015543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4406f970 0x7f0f4415ab70 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f0f440b1dd0 tx=0x7f0f2c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 -- 172.21.15.28:0/564015543 shutdown_connections 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 --2- 172.21.15.28:0/564015543 >> v2:172.21.15.189:3300/0 conn(0x7f0f4406c2d0 0x7f0f4406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 --2- 172.21.15.28:0/564015543 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f4406f970 0x7f0f4415ab70 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.833+0000 7f0f4b600640 1 -- 172.21.15.28:0/564015543 >> 172.21.15.28:0/564015543 conn(0x7f0f44069e80 msgr2=0x7f0f4406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.834+0000 7f0f4b600640 1 -- 172.21.15.28:0/564015543 shutdown_connections 2024-09-13T17:21:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.834+0000 7f0f4b600640 1 -- 172.21.15.28:0/564015543 wait complete. 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.835+0000 7f0f4b600640 1 Processor -- start 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.835+0000 7f0f4b600640 1 -- start start 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 0x7f0f441498c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0f4406f970 0x7f0f44150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4414a1d0 con 0x7f0f441592a0 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f44150d50 con 0x7f0f4406c2d0 2024-09-13T17:21:15.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f4b600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f440cea00 con 0x7f0f4406f970 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f49600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f49600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60988/0 (socket says 172.21.15.28:60988) 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.836+0000 7f0f49600640 1 -- 172.21.15.28:0/2772559698 learned_addr learned my addr 172.21.15.28:0/2772559698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f3be00640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0f4406f970 0x7f0f44150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f48c00640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 0x7f0f441498c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 -- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0f4406f970 msgr2=0x7f0f44150650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0f4406f970 0x7f0f44150650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 -- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 msgr2=0x7f0f441498c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 0x7f0f441498c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 -- 172.21.15.28:0/2772559698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c009400 con 0x7f0f441592a0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f48c00640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 0x7f0f441498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f49600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c007e70 tx=0x7f0f3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c014620 con 0x7f0f441592a0 2024-09-13T17:21:15.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c0147a0 con 0x7f0f441592a0 2024-09-13T17:21:15.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c014b50 con 0x7f0f441592a0 2024-09-13T17:21:15.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.837+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f440cece0 con 0x7f0f441592a0 2024-09-13T17:21:15.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.838+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f440cf2a0 con 0x7f0f441592a0 2024-09-13T17:21:15.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.839+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f440cf430 con 0x7f0f441592a0 2024-09-13T17:21:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.839+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0f3c014cb0 con 0x7f0f441592a0 2024-09-13T17:21:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.840+0000 7f0f41600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 0x7f0f2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.840+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0f3c089cd0 con 0x7f0f441592a0 2024-09-13T17:21:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.841+0000 7f0f48c00640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 0x7f0f2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:15.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.842+0000 7f0f48c00640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 0x7f0f2006d0c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0f34007fd0 tx=0x7f0f34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:15.845+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0f3c054e60 con 0x7f0f441592a0 2024-09-13T17:21:16.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.026+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f440b3740 con 0x7f0f441592a0 2024-09-13T17:21:16.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.027+0000 7f0f41600640 1 -- 172.21.15.28:0/2772559698 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f0f3c058480 con 0x7f0f441592a0 2024-09-13T17:21:16.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:16.029 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:55.141358+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.2d","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.029+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 msgr2=0x7f0f2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.029+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 0x7f0f2006d0c0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f0f34007fd0 tx=0x7f0f34009040 comp rx=0 tx=0).stop 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.029+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 msgr2=0x7f0f4414bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.029+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 secure :-1 s=READY pgs=2237 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c007e70 tx=0x7f0f3c00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 shutdown_connections 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0f2006ac00 0x7f0f2006d0c0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f441592a0 0x7f0f4414bbb0 unknown :-1 s=CLOSED pgs=2237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0f4406f970 0x7f0f44150650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 --2- 172.21.15.28:0/2772559698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0f4406c2d0 0x7f0f441498c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 >> 172.21.15.28:0/2772559698 conn(0x7f0f44069e80 msgr2=0x7f0f4406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 shutdown_connections 2024-09-13T17:21:16.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.030+0000 7f0f4b600640 1 -- 172.21.15.28:0/2772559698 wait complete. 2024-09-13T17:21:16.049 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:21:16.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:16.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.203+0000 7f19dbe00640 1 Processor -- start 2024-09-13T17:21:16.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.204+0000 7f19dbe00640 1 -- start start 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f19dc160e70 0x7f19dc0ad870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19dbe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc06bf50 con 0x7f19dc061110 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19dbe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19dc161300 con 0x7f19dc06ce80 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19dbe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19dc0ccd70 con 0x7f19dc160e70 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19d3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f19dc061110 0x7f19dc06c3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34670/0 (socket says 172.21.15.28:34670) 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19d3e00640 1 -- 172.21.15.28:0/4098374577 learned_addr learned my addr 172.21.15.28:0/4098374577 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:16.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.205+0000 7f19db400640 1 --2- 172.21.15.28:0/4098374577 >> v2:172.21.15.28:3300/0 conn(0x7f19dc160e70 0x7f19dc0ad870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.206+0000 7f19da000640 1 -- 172.21.15.28:0/4098374577 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204816456 0 0) 0x7f19dc06bf50 con 0x7f19dc061110 2024-09-13T17:21:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.206+0000 7f19da000640 1 -- 172.21.15.28:0/4098374577 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0003520 con 0x7f19dc061110 2024-09-13T17:21:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.206+0000 7f19db400640 1 -- 172.21.15.28:0/4098374577 >> v1:172.21.15.189:6789/0 conn(0x7f19dc06ce80 legacy=0x7f19dc06d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.206+0000 7f19db400640 1 -- 172.21.15.28:0/4098374577 >> v1:172.21.15.28:6789/0 conn(0x7f19dc061110 legacy=0x7f19dc06c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.206+0000 7f19db400640 1 -- 172.21.15.28:0/4098374577 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc0ce3d0 con 0x7f19dc160e70 2024-09-13T17:21:16.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.207+0000 7f19db400640 1 --2- 172.21.15.28:0/4098374577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0ad870 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009d90 tx=0x7f19d402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3adc94a6e3d52cf3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19da000640 1 -- 172.21.15.28:0/4098374577 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4007740 con 0x7f19dc160e70 2024-09-13T17:21:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19da000640 1 -- 172.21.15.28:0/4098374577 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d402d050 con 0x7f19dc160e70 2024-09-13T17:21:16.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19da000640 1 -- 172.21.15.28:0/4098374577 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4007aa0 con 0x7f19dc160e70 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19dbe00640 1 -- 172.21.15.28:0/4098374577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 msgr2=0x7f19dc0ad870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19dbe00640 1 --2- 172.21.15.28:0/4098374577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0ad870 secure :-1 s=READY pgs=2239 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009d90 tx=0x7f19d402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19dbe00640 1 -- 172.21.15.28:0/4098374577 shutdown_connections 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19dbe00640 1 --2- 172.21.15.28:0/4098374577 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0ad870 unknown :-1 s=CLOSED pgs=2239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.209+0000 7f19dbe00640 1 -- 172.21.15.28:0/4098374577 >> 172.21.15.28:0/4098374577 conn(0x7f19dc0cb6d0 msgr2=0x7f19dc0cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.210+0000 7f19dbe00640 1 -- 172.21.15.28:0/4098374577 shutdown_connections 2024-09-13T17:21:16.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.210+0000 7f19dbe00640 1 -- 172.21.15.28:0/4098374577 wait complete. 2024-09-13T17:21:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.211+0000 7f19dbe00640 1 Processor -- start 2024-09-13T17:21:16.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.211+0000 7f19dbe00640 1 -- start start 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f19dc061110 0x7f19dc1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc06ce80 0x7f19dc153cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc158b10 con 0x7f19dc160e70 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc0b1d30 con 0x7f19dc06ce80 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.212+0000 7f19dbe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc159b60 con 0x7f19dc061110 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32776/0 (socket says 172.21.15.28:32776) 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 -- 172.21.15.28:0/876426729 learned_addr learned my addr 172.21.15.28:0/876426729 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19daa00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f19dc061110 0x7f19dc1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19d3e00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc06ce80 0x7f19dc153cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 -- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f19dc061110 msgr2=0x7f19dc1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f19dc061110 0x7f19dc1585d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 -- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc06ce80 msgr2=0x7f19dc153cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc06ce80 0x7f19dc153cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.213+0000 7f19db400640 1 -- 172.21.15.28:0/876426729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d4009a70 con 0x7f19dc160e70 2024-09-13T17:21:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.214+0000 7f19db400640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009eb0 tx=0x7f19d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.215+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4003b90 con 0x7f19dc160e70 2024-09-13T17:21:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.215+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc0b1f80 con 0x7f19dc160e70 2024-09-13T17:21:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.215+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d4003cf0 con 0x7f19dc160e70 2024-09-13T17:21:16.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.215+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d403b900 con 0x7f19dc160e70 2024-09-13T17:21:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.215+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc154230 con 0x7f19dc160e70 2024-09-13T17:21:16.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.216+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc0b24a0 con 0x7f19dc160e70 2024-09-13T17:21:16.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.217+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f19d403ba60 con 0x7f19dc160e70 2024-09-13T17:21:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.218+0000 7f19d2a00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 0x7f19b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.218+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f19d40aa260 con 0x7f19dc160e70 2024-09-13T17:21:16.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.218+0000 7f19daa00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 0x7f19b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.219+0000 7f19daa00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 0x7f19b806d0c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000f60 tx=0x7f19cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.222+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f19dc0b24a0 con 0x7f19dc160e70 2024-09-13T17:21:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.438+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19dc159930 con 0x7f19dc160e70 2024-09-13T17:21:16.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.438+0000 7f19d2a00640 1 -- 172.21.15.28:0/876426729 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+4763 (secure 0 0 0) 0x7f19dc159930 con 0x7f19dc160e70 2024-09-13T17:21:16.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:16.441 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":156,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":156,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:21:16.206193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10551":{"gid":10551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":11,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":12,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:21:16.441 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 156 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 msgr2=0x7f19b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 0x7f19b806d0c0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f19cc000f60 tx=0x7f19cc009040 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 msgr2=0x7f19dc0b2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 secure :-1 s=READY pgs=2240 cs=0 l=1 rev1=1 crypto rx=0x7f19d4009eb0 tx=0x7f19d4004a40 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 shutdown_connections 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f19b806ac00 0x7f19b806d0c0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19dc160e70 0x7f19dc0b2660 unknown :-1 s=CLOSED pgs=2240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc06ce80 0x7f19dc153cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.440+0000 7f19dbe00640 1 --2- 172.21.15.28:0/876426729 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f19dc061110 0x7f19dc1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.441+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 >> 172.21.15.28:0/876426729 conn(0x7f19dc0cb6d0 msgr2=0x7f19dc0af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.441+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 shutdown_connections 2024-09-13T17:21:16.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.441+0000 7f19dbe00640 1 -- 172.21.15.28:0/876426729 wait complete. 2024-09-13T17:21:16.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:16.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.612+0000 7f0a04200640 1 Processor -- start 2024-09-13T17:21:16.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.612+0000 7f0a04200640 1 -- start start 2024-09-13T17:21:16.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a04200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a04200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 0x7f09fc06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a04200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09fc159d90 con 0x7f09fc1592a0 2024-09-13T17:21:16.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a04200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc15b0b0 con 0x7f09fc06c2d0 2024-09-13T17:21:16.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a04200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc06cca0 con 0x7f09fc06f970 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a00e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a02200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 0x7f09fc06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a00e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:57246/0 (socket says 172.21.15.28:57246) 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.614+0000 7f0a00e00640 1 -- 172.21.15.28:0/1871153608 learned_addr learned my addr 172.21.15.28:0/1871153608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.615+0000 7f0a00e00640 1 -- 172.21.15.28:0/1871153608 >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 msgr2=0x7f09fc06c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.615+0000 7f0a00e00640 1 --2- 172.21.15.28:0/1871153608 >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 0x7f09fc06c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.615+0000 7f0a00e00640 1 -- 172.21.15.28:0/1871153608 >> v1:172.21.15.189:6789/0 conn(0x7f09fc1592a0 legacy=0x7f09fc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.615+0000 7f0a00e00640 1 -- 172.21.15.28:0/1871153608 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc0b2040 con 0x7f09fc06f970 2024-09-13T17:21:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.615+0000 7f0a02200640 1 --2- 172.21.15.28:0/1871153608 >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 0x7f09fc06c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.616+0000 7f0a00e00640 1 --2- 172.21.15.28:0/1871153608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009870 tx=0x7f09e802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3232e5fc8ea5349 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.616+0000 7f09fbe00640 1 -- 172.21.15.28:0/1871153608 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8032440 con 0x7f09fc06f970 2024-09-13T17:21:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.616+0000 7f09fbe00640 1 -- 172.21.15.28:0/1871153608 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e80325a0 con 0x7f09fc06f970 2024-09-13T17:21:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.616+0000 7f09fbe00640 1 -- 172.21.15.28:0/1871153608 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8032930 con 0x7f09fc06f970 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 -- 172.21.15.28:0/1871153608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 msgr2=0x7f09fc15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 --2- 172.21.15.28:0/1871153608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009870 tx=0x7f09e802a270 comp rx=0 tx=0).stop 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 -- 172.21.15.28:0/1871153608 shutdown_connections 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 --2- 172.21.15.28:0/1871153608 >> v2:172.21.15.189:3300/0 conn(0x7f09fc06c2d0 0x7f09fc06c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 --2- 172.21.15.28:0/1871153608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc15ab70 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 -- 172.21.15.28:0/1871153608 >> 172.21.15.28:0/1871153608 conn(0x7f09fc069e80 msgr2=0x7f09fc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 -- 172.21.15.28:0/1871153608 shutdown_connections 2024-09-13T17:21:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.617+0000 7f0a04200640 1 -- 172.21.15.28:0/1871153608 wait complete. 2024-09-13T17:21:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.619+0000 7f0a04200640 1 Processor -- start 2024-09-13T17:21:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.619+0000 7f0a04200640 1 -- start start 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 0x7f09fc1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc0cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0cde90 con 0x7f09fc1592a0 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0ce130 con 0x7f09fc06c2d0 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a04200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc0ce410 con 0x7f09fc06f970 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a02200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a02200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32778/0 (socket says 172.21.15.28:32778) 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a02200640 1 -- 172.21.15.28:0/105724268 learned_addr learned my addr 172.21.15.28:0/105724268 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:16.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a00e00640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc0cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a01800640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 0x7f09fc1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a02200640 1 -- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 msgr2=0x7f09fc0cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.620+0000 7f0a02200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc0cd950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a02200640 1 -- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 msgr2=0x7f09fc1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a02200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 0x7f09fc1524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a02200640 1 -- 172.21.15.28:0/105724268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009a70 con 0x7f09fc1592a0 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a01800640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 0x7f09fc1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a02200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f09f400efc0 tx=0x7f09f4009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4009d70 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f400f040 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4003990 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e80094d0 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.621+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc0ce9d0 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.622+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc0b3740 con 0x7f09fc1592a0 2024-09-13T17:21:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.623+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f09f40144a0 con 0x7f09fc1592a0 2024-09-13T17:21:16.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.624+0000 7f09f9600640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 0x7f09dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:16.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.624+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f09f408a000 con 0x7f09fc1592a0 2024-09-13T17:21:16.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.625+0000 7f0a01800640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 0x7f09dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:16.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.625+0000 7f0a01800640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 0x7f09dc06d0c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000f60 tx=0x7f09f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.628+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f09f40551b0 con 0x7f09fc1592a0 2024-09-13T17:21:16.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.842+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09fc06a4b0 con 0x7f09fc1592a0 2024-09-13T17:21:16.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.842+0000 7f09f9600640 1 -- 172.21.15.28:0/105724268 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+4763 (secure 0 0 0) 0x7f09f40587d0 con 0x7f09fc1592a0 2024-09-13T17:21:16.844 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:16.844 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":156,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146}],"filesystems":[{"mdsmap":{"epoch":156,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-09-13T17:20:39.612305+0000","modified":"2024-09-13T17:21:16.206193+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":10566},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10551":{"gid":10551,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":11,"addr":"172.21.15.28:6833/3568343573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3568343573},{"type":"v1","addr":"172.21.15.28:6833","nonce":3568343573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_10566":{"gid":10566,"name":"a","rank":0,"incarnation":147,"state":"up:active","state_seq":12,"addr":"172.21.15.28:6835/3229070352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":3229070352},{"type":"v1","addr":"172.21.15.28:6835","nonce":3229070352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2024-09-13T17:21:16.845 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 156 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.844+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 msgr2=0x7f09dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.844+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 0x7f09dc06d0c0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f09f0000f60 tx=0x7f09f0009040 comp rx=0 tx=0).stop 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.844+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 msgr2=0x7f09fc1e70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.844+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 secure :-1 s=READY pgs=2241 cs=0 l=1 rev1=1 crypto rx=0x7f09f400efc0 tx=0x7f09f4009f90 comp rx=0 tx=0).stop 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 shutdown_connections 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f09dc06ac00 0x7f09dc06d0c0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09fc1592a0 0x7f09fc1e70d0 unknown :-1 s=CLOSED pgs=2241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09fc06f970 0x7f09fc0cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 --2- 172.21.15.28:0/105724268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09fc06c2d0 0x7f09fc1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:16.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 >> 172.21.15.28:0/105724268 conn(0x7f09fc069e80 msgr2=0x7f09fc0ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:16.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 shutdown_connections 2024-09-13T17:21:16.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:16.845+0000 7f0a04200640 1 -- 172.21.15.28:0/105724268 wait complete. 2024-09-13T17:21:16.863 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:17.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.016+0000 7f6318600640 1 Processor -- start 2024-09-13T17:21:17.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.016+0000 7f6318600640 1 -- start start 2024-09-13T17:21:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.017+0000 7f6318600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f631006c9e0 0x7f631015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.017+0000 7f6318600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6310159d90 con 0x7f63101592a0 2024-09-13T17:21:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.017+0000 7f6318600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100b2040 con 0x7f631006f970 2024-09-13T17:21:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.017+0000 7f6318600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f631015b0b0 con 0x7f631006c9e0 2024-09-13T17:21:17.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f6315c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f63101592a0 0x7f6310159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34678/0 (socket says 172.21.15.28:34678) 2024-09-13T17:21:17.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f6315c00640 1 -- 172.21.15.28:0/772354517 learned_addr learned my addr 172.21.15.28:0/772354517 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:17.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f6316600640 1 --2- 172.21.15.28:0/772354517 >> v2:172.21.15.189:3300/0 conn(0x7f631006c9e0 0x7f631015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274572183 0 0) 0x7f6310159d90 con 0x7f63101592a0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8003520 con 0x7f63101592a0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10639960 0 0) 0x7f63100b2040 con 0x7f631006f970 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6310159d90 con 0x7f631006f970 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2386377741 0 0) 0x7f62f8003520 con 0x7f63101592a0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 >> v2:172.21.15.189:3300/0 conn(0x7f631006c9e0 msgr2=0x7f631015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 --2- 172.21.15.28:0/772354517 >> v2:172.21.15.189:3300/0 conn(0x7f631006c9e0 0x7f631015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 >> v1:172.21.15.189:6789/0 conn(0x7f631006f970 legacy=0x7f631006c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6310159ff0 con 0x7f63101592a0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6304001690 con 0x7f63101592a0 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 >> v1:172.21.15.28:6789/0 conn(0x7f63101592a0 legacy=0x7f6310159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f62f8003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f80047c0 0x7f62f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f8005180 0x7f62f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f631015b0b0 con 0x7f62f8003b00 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f800e210 con 0x7f62f80047c0 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.018+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f800e3d0 con 0x7f62f8005180 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6316600640 1 --2- 172.21.15.28:0/772354517 >> v2:172.21.15.189:3300/0 conn(0x7f631006c9e0 0x7f631015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f62f8003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315200640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f80047c0 0x7f62f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6316600640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f8005180 0x7f62f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 -- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f8005180 msgr2=0x7f62f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62f8005180 0x7f62f800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 -- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f80047c0 msgr2=0x7f62f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62f80047c0 0x7f62f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6315c00640 1 -- 172.21.15.28:0/772354517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f8003520 con 0x7f62f8003b00 2024-09-13T17:21:17.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.019+0000 7f6316600640 1 -- 172.21.15.28:0/772354517 reap_dead start 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.020+0000 7f6315c00640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f62f8003f00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f6310159f20 tx=0x7f62f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c662a42dd28cebd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304005ad0 con 0x7f62f8003b00 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630402d070 con 0x7f62f8003b00 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f630fe00640 1 -- 172.21.15.28:0/772354517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304005e70 con 0x7f62f8003b00 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 -- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 msgr2=0x7f62f8003f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f62f8003f00 secure :-1 s=READY pgs=2243 cs=0 l=1 rev1=1 crypto rx=0x7f6310159f20 tx=0x7f62f80057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 -- 172.21.15.28:0/772354517 shutdown_connections 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 --2- 172.21.15.28:0/772354517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f62f8003f00 unknown :-1 s=CLOSED pgs=2243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 -- 172.21.15.28:0/772354517 >> 172.21.15.28:0/772354517 conn(0x7f6310069e80 msgr2=0x7f631006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 -- 172.21.15.28:0/772354517 shutdown_connections 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.021+0000 7f6318600640 1 -- 172.21.15.28:0/772354517 wait complete. 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.023+0000 7f6318600640 1 Processor -- start 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.023+0000 7f6318600640 1 -- start start 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 0x7f63100b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63101f1b90 0x7f63101f2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63101ee440 con 0x7f62f8003b00 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63100aa620 con 0x7f63100ce450 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6318600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63101ee670 con 0x7f63101f1b90 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6315c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6315c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32796/0 (socket says 172.21.15.28:32796) 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.024+0000 7f6315c00640 1 -- 172.21.15.28:0/189192798 learned_addr learned my addr 172.21.15.28:0/189192798 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 -- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63101f1b90 msgr2=0x7f63101f2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63101f1b90 0x7f63101f2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 -- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 msgr2=0x7f63100b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315200640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 0x7f63100b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 0x7f63100b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 -- 172.21.15.28:0/189192798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63101ee950 con 0x7f62f8003b00 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315200640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 0x7f63100b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:17.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6315c00640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f630402abd0 tx=0x7f63040019a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63040049f0 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6304004b50 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304007070 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63101eec30 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.025+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63100b05d0 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.026+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63101eee70 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.027+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f630402f030 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.028+0000 7f630d600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 0x7f62f006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.028+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f63040a8ea0 con 0x7f62f8003b00 2024-09-13T17:21:17.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.029+0000 7f6315200640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 0x7f62f006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.029+0000 7f6315200640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 0x7f62f006d0f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f62fc000c00 tx=0x7f62fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.032+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6304074020 con 0x7f62f8003b00 2024-09-13T17:21:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.214+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63100b0760 con 0x7f62f8003b00 2024-09-13T17:21:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.215+0000 7f630d600640 1 -- 172.21.15.28:0/189192798 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v181) v1 ==== 74+0+16621 (secure 0 0 0) 0x7f6304077640 con 0x7f62f8003b00 2024-09-13T17:21:17.218 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:17.218 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:20:55.141358+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:20:37.568535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2024-09-13T17:20:38.601719+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"179","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.26","mappings":[{"from":0,"to":1}]},{"pgid":"25.2d","mappings":[{"from":0,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.217+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 msgr2=0x7f62f006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 0x7f62f006d0f0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f62fc000c00 tx=0x7f62fc009040 comp rx=0 tx=0).stop 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 msgr2=0x7f63101edf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 secure :-1 s=READY pgs=2244 cs=0 l=1 rev1=1 crypto rx=0x7f630402abd0 tx=0x7f63040019a0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 shutdown_connections 2024-09-13T17:21:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f62f006ac30 0x7f62f006d0f0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f63101f1b90 0x7f63101f2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63100ce450 0x7f63100b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 --2- 172.21.15.28:0/189192798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62f8003b00 0x7f63101edf00 unknown :-1 s=CLOSED pgs=2244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 >> 172.21.15.28:0/189192798 conn(0x7f6310069e80 msgr2=0x7f631006c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 shutdown_connections 2024-09-13T17:21:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.218+0000 7f6318600640 1 -- 172.21.15.28:0/189192798 wait complete. 2024-09-13T17:21:17.238 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:21:17.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.393+0000 7f9bc8200640 1 Processor -- start 2024-09-13T17:21:17.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.393+0000 7f9bc8200640 1 -- start start 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9bc015a780 0x7f9bc0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc015ab60 0x7f9bc00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bc006bb40 0x7f9bc006bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc006c8b0 con 0x7f9bc006bb40 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc006cb90 con 0x7f9bc015a780 2024-09-13T17:21:17.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.394+0000 7f9bc8200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00cc980 con 0x7f9bc015ab60 2024-09-13T17:21:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bc006bb40 0x7f9bc006bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9bc006bb40 0x7f9bc006bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32800/0 (socket says 172.21.15.28:32800) 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 -- 172.21.15.28:0/1481729419 learned_addr learned my addr 172.21.15.28:0/1481729419 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc4e00640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc015ab60 0x7f9bc00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc5800640 1 --2- 172.21.15.28:0/1481729419 >> v2:172.21.15.189:3300/0 conn(0x7f9bc015a780 0x7f9bc0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 -- 172.21.15.28:0/1481729419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc015ab60 msgr2=0x7f9bc00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc015ab60 0x7f9bc00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 -- 172.21.15.28:0/1481729419 >> v2:172.21.15.189:3300/0 conn(0x7f9bc015a780 msgr2=0x7f9bc0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 --2- 172.21.15.28:0/1481729419 >> v2:172.21.15.189:3300/0 conn(0x7f9bc015a780 0x7f9bc0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.395+0000 7f9bc6200640 1 -- 172.21.15.28:0/1481729419 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00ccc60 con 0x7f9bc006bb40 2024-09-13T17:21:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.396+0000 7f9bc6200640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc006bb40 0x7f9bc006bfd0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00b1330 tx=0x7f9bb802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2695f1ffe22e5d3b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.396+0000 7f9bbfe00640 1 -- 172.21.15.28:0/1481729419 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8031600 con 0x7f9bc006bb40 2024-09-13T17:21:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.396+0000 7f9bbfe00640 1 -- 172.21.15.28:0/1481729419 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb802d050 con 0x7f9bc006bb40 2024-09-13T17:21:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.396+0000 7f9bbfe00640 1 -- 172.21.15.28:0/1481729419 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb8031980 con 0x7f9bc006bb40 2024-09-13T17:21:17.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 -- 172.21.15.28:0/1481729419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc006bb40 msgr2=0x7f9bc006bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc006bb40 0x7f9bc006bfd0 secure :-1 s=READY pgs=2245 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00b1330 tx=0x7f9bb802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 -- 172.21.15.28:0/1481729419 shutdown_connections 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc006bb40 0x7f9bc006bfd0 unknown :-1 s=CLOSED pgs=2245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 --2- 172.21.15.28:0/1481729419 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc015ab60 0x7f9bc00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 --2- 172.21.15.28:0/1481729419 >> v2:172.21.15.189:3300/0 conn(0x7f9bc015a780 0x7f9bc0070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 -- 172.21.15.28:0/1481729419 >> 172.21.15.28:0/1481729419 conn(0x7f9bc00b0270 msgr2=0x7f9bc00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 -- 172.21.15.28:0/1481729419 shutdown_connections 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.397+0000 7f9bc8200640 1 -- 172.21.15.28:0/1481729419 wait complete. 2024-09-13T17:21:17.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.399+0000 7f9bc8200640 1 Processor -- start 2024-09-13T17:21:17.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.399+0000 7f9bc8200640 1 -- start start 2024-09-13T17:21:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc006bb40 0x7f9bc00ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc015a780 0x7f9bc0154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc015ab60 0x7f9bc0154b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00c73f0 con 0x7f9bc015a780 2024-09-13T17:21:17.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00c6b50 con 0x7f9bc015ab60 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc8200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc00c6e30 con 0x7f9bc006bb40 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc015a780 0x7f9bc0154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc015a780 0x7f9bc0154640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32808/0 (socket says 172.21.15.28:32808) 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 -- 172.21.15.28:0/1584189773 learned_addr learned my addr 172.21.15.28:0/1584189773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc6200640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc015ab60 0x7f9bc0154b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc5800640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc006bb40 0x7f9bc00ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 -- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc006bb40 msgr2=0x7f9bc00ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc006bb40 0x7f9bc00ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 -- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc015ab60 msgr2=0x7f9bc0154b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bc015ab60 0x7f9bc0154b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc4e00640 1 -- 172.21.15.28:0/1584189773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb8009b30 con 0x7f9bc015a780 2024-09-13T17:21:17.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.400+0000 7f9bc5800640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bc006bb40 0x7f9bc00ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bc4e00640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bc015a780 0x7f9bc0154640 secure :-1 s=READY pgs=2246 cs=0 l=1 rev1=1 crypto rx=0x7f9bac00d5f0 tx=0x7f9bac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac0125f0 con 0x7f9bc015a780 2024-09-13T17:21:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac012750 con 0x7f9bc015a780 2024-09-13T17:21:17.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bac012b00 con 0x7f9bc015a780 2024-09-13T17:21:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bc8200640 1 -- 172.21.15.28:0/1584189773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc00c7110 con 0x7f9bc015a780 2024-09-13T17:21:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.401+0000 7f9bc8200640 1 -- 172.21.15.28:0/1584189773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc00ce7d0 con 0x7f9bc015a780 2024-09-13T17:21:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.402+0000 7f9bc8200640 1 -- 172.21.15.28:0/1584189773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc00c34e0 con 0x7f9bc015a780 2024-09-13T17:21:17.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.403+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9bac012c60 con 0x7f9bc015a780 2024-09-13T17:21:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.404+0000 7f9bbd600640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9ba006ac00 0x7f9ba006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.404+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f9bac0898d0 con 0x7f9bc015a780 2024-09-13T17:21:17.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.405+0000 7f9bc5800640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9ba006ac00 0x7f9ba006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.405+0000 7f9bc5800640 1 --2- 172.21.15.28:0/1584189773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9ba006ac00 0x7f9ba006d0c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4006fd0 tx=0x7f9bb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.408+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9bc00c34e0 con 0x7f9bc015a780 2024-09-13T17:21:17.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.619+0000 7f9bc8200640 1 -- 172.21.15.28:0/1584189773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9bc00c34e0 con 0x7f9bc015a780 2024-09-13T17:21:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.620+0000 7f9bbd600640 1 -- 172.21.15.28:0/1584189773 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=-1 MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it v156) v1 ==== 316+0+0 (secure 0 0 0) 0x7f9bac0549d0 con 0x7f9bc015a780 2024-09-13T17:21:17.622 INFO:teuthology.orchestra.run.smithi028.stderr:Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it 2024-09-13T17:21:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:21:17.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs --yes-i-really-mean-it 2024-09-13T17:21:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.788+0000 7fb63f200640 1 Processor -- start 2024-09-13T17:21:17.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.788+0000 7fb63f200640 1 -- start start 2024-09-13T17:21:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6400cde30 0x7fb640071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6400712d0 0x7fb64015af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb640159e50 con 0x7fb640071f60 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6400ce210 con 0x7fb6400cde30 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64015a130 con 0x7fb6400712d0 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6400cde30 0x7fb640071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb6400cde30 0x7fb640071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32814/0 (socket says 172.21.15.28:32814) 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63de00640 1 -- 172.21.15.28:0/994221802 learned_addr learned my addr 172.21.15.28:0/994221802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.789+0000 7fb63e800640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6400712d0 0x7fb64015af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.790+0000 7fb63de00640 1 -- 172.21.15.28:0/994221802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6400712d0 msgr2=0x7fb64015af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.790+0000 7fb63de00640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6400712d0 0x7fb64015af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.790+0000 7fb63de00640 1 -- 172.21.15.28:0/994221802 >> v1:172.21.15.189:6789/0 conn(0x7fb640071f60 legacy=0x7fb640070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:17.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.790+0000 7fb63de00640 1 -- 172.21.15.28:0/994221802 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb64015a410 con 0x7fb6400cde30 2024-09-13T17:21:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63de00640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400cde30 0x7fb640071a20 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb630009870 tx=0x7fb63002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8964360e01475e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63d400640 1 -- 172.21.15.28:0/994221802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630032440 con 0x7fb6400cde30 2024-09-13T17:21:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63d400640 1 -- 172.21.15.28:0/994221802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6300325a0 con 0x7fb6400cde30 2024-09-13T17:21:17.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63d400640 1 -- 172.21.15.28:0/994221802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630032930 con 0x7fb6400cde30 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63f200640 1 -- 172.21.15.28:0/994221802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400cde30 msgr2=0x7fb640071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.791+0000 7fb63f200640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400cde30 0x7fb640071a20 secure :-1 s=READY pgs=2247 cs=0 l=1 rev1=1 crypto rx=0x7fb630009870 tx=0x7fb63002a250 comp rx=0 tx=0).stop 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 -- 172.21.15.28:0/994221802 shutdown_connections 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb6400712d0 0x7fb64015af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 --2- 172.21.15.28:0/994221802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400cde30 0x7fb640071a20 unknown :-1 s=CLOSED pgs=2247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 -- 172.21.15.28:0/994221802 >> 172.21.15.28:0/994221802 conn(0x7fb640069e80 msgr2=0x7fb64006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 -- 172.21.15.28:0/994221802 shutdown_connections 2024-09-13T17:21:17.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.792+0000 7fb63f200640 1 -- 172.21.15.28:0/994221802 wait complete. 2024-09-13T17:21:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.793+0000 7fb63f200640 1 Processor -- start 2024-09-13T17:21:17.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.793+0000 7fb63f200640 1 -- start start 2024-09-13T17:21:17.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb640071f60 0x7fb6401dfb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6400cde30 0x7fb6401e00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6401e1330 con 0x7fb6400712d0 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb640153910 con 0x7fb6400cde30 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.794+0000 7fb63f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6401e0750 con 0x7fb640071f60 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32820/0 (socket says 172.21.15.28:32820) 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 -- 172.21.15.28:0/129107148 learned_addr learned my addr 172.21.15.28:0/129107148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63e800640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6400cde30 0x7fb6401e00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb635400640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb640071f60 0x7fb6401dfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 -- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb640071f60 msgr2=0x7fb6401dfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb640071f60 0x7fb6401dfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 -- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6400cde30 msgr2=0x7fb6401e00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6400cde30 0x7fb6401e00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 -- 172.21.15.28:0/129107148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6300094d0 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.795+0000 7fb63de00640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fb63002a290 tx=0x7fb6300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.796+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb630032440 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.796+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb630030b80 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.796+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63003d440 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.796+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6401e0a30 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.796+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6401e1510 con 0x7fb6400712d0 2024-09-13T17:21:17.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.797+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6401e0f50 con 0x7fb6400712d0 2024-09-13T17:21:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.798+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb630039040 con 0x7fb6400712d0 2024-09-13T17:21:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.798+0000 7fb636a00640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 0x7fb61c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:17.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.799+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb6300a9e00 con 0x7fb6400712d0 2024-09-13T17:21:17.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.799+0000 7fb635400640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 0x7fb61c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:17.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.800+0000 7fb635400640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 0x7fb61c06d0c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb64014c6f0 tx=0x7fb628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:17.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:17.802+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb6401e0f50 con 0x7fb6400712d0 2024-09-13T17:21:18.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.017+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb6401e0f50 con 0x7fb6400712d0 2024-09-13T17:21:18.020 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:21:18.018+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:21:18.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.028+0000 7fb636a00640 1 -- 172.21.15.28:0/129107148 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v157) v1 ==== 191+0+0 (secure 0 0 0) 0x7fb6401e0f50 con 0x7fb6400712d0 2024-09-13T17:21:18.030 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:21:18.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 msgr2=0x7fb61c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:18.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 0x7fb61c06d0c0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fb64014c6f0 tx=0x7fb628009040 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 msgr2=0x7fb6401483e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7fb63002a290 tx=0x7fb6300046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 shutdown_connections 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb61c06ac00 0x7fb61c06d0c0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6400cde30 0x7fb6401e00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb640071f60 0x7fb6401dfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 --2- 172.21.15.28:0/129107148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb6400712d0 0x7fb6401483e0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 >> 172.21.15.28:0/129107148 conn(0x7fb640069e80 msgr2=0x7fb6400b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 shutdown_connections 2024-09-13T17:21:18.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.030+0000 7fb63f200640 1 -- 172.21.15.28:0/129107148 wait complete. 2024-09-13T17:21:18.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:21:18.098 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:21:18.098 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:21:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.212+0000 7f1337e00640 1 Processor -- start 2024-09-13T17:21:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.212+0000 7f1337e00640 1 -- start start 2024-09-13T17:21:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 0x7f133815a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1338159b30 0x7f13380ccb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 0x7f13380cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1338070480 con 0x7f1338159b30 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133815a010 con 0x7f1338156650 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f1337e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133806fb50 con 0x7f13380b0a90 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f132e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1338159b30 0x7f13380ccb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f132e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1338159b30 0x7f13380ccb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32856/0 (socket says 172.21.15.28:32856) 2024-09-13T17:21:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.213+0000 7f132e000640 1 -- 172.21.15.28:0/2207701603 learned_addr learned my addr 172.21.15.28:0/2207701603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f1336a00640 1 --2- 172.21.15.28:0/2207701603 >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 0x7f133815a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f132e000640 1 -- 172.21.15.28:0/2207701603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 msgr2=0x7f13380cd090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f1337400640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 0x7f13380cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f132e000640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 0x7f13380cd090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f132e000640 1 -- 172.21.15.28:0/2207701603 >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 msgr2=0x7f133815a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f132e000640 1 --2- 172.21.15.28:0/2207701603 >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 0x7f133815a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f132e000640 1 -- 172.21.15.28:0/2207701603 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133806fe30 con 0x7f1338159b30 2024-09-13T17:21:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f1336a00640 1 --2- 172.21.15.28:0/2207701603 >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 0x7f133815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.214+0000 7f1337400640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 0x7f13380cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.215+0000 7f132e000640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1338159b30 0x7f13380ccb50 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f133806d3b0 tx=0x7f132002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721487a47b5a2050 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1336000640 1 -- 172.21.15.28:0/2207701603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132002ae90 con 0x7f1338159b30 2024-09-13T17:21:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1336000640 1 -- 172.21.15.28:0/2207701603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1320032440 con 0x7f1338159b30 2024-09-13T17:21:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1336000640 1 -- 172.21.15.28:0/2207701603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13200327d0 con 0x7f1338159b30 2024-09-13T17:21:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 -- 172.21.15.28:0/2207701603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1338159b30 msgr2=0x7f13380ccb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1338159b30 0x7f13380ccb50 secure :-1 s=READY pgs=2251 cs=0 l=1 rev1=1 crypto rx=0x7f133806d3b0 tx=0x7f132002a140 comp rx=0 tx=0).stop 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 -- 172.21.15.28:0/2207701603 shutdown_connections 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1338159b30 0x7f13380ccb50 unknown :-1 s=CLOSED pgs=2251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 --2- 172.21.15.28:0/2207701603 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f13380b0a90 0x7f13380cd090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 --2- 172.21.15.28:0/2207701603 >> v2:172.21.15.189:3300/0 conn(0x7f1338156650 0x7f133815a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 -- 172.21.15.28:0/2207701603 >> 172.21.15.28:0/2207701603 conn(0x7f1338069e80 msgr2=0x7f133806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 -- 172.21.15.28:0/2207701603 shutdown_connections 2024-09-13T17:21:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.216+0000 7f1337e00640 1 -- 172.21.15.28:0/2207701603 wait complete. 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.218+0000 7f1337e00640 1 Processor -- start 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.218+0000 7f1337e00640 1 -- start start 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1338156650 0x7f13380aa590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1338159b30 0x7f1338163d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13381532e0 con 0x7f13380b0a90 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13380a94f0 con 0x7f1338159b30 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1337e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13380a97d0 con 0x7f1338156650 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1336a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1336a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32858/0 (socket says 172.21.15.28:32858) 2024-09-13T17:21:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.219+0000 7f1336a00640 1 -- 172.21.15.28:0/3246296068 learned_addr learned my addr 172.21.15.28:0/3246296068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f132e000640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1338156650 0x7f13380aa590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1337400640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1338159b30 0x7f1338163d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 -- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1338156650 msgr2=0x7f13380aa590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1338156650 0x7f13380aa590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 -- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1338159b30 msgr2=0x7f1338163d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1338159b30 0x7f1338163d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 -- 172.21.15.28:0/3246296068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13200094d0 con 0x7f13380b0a90 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f1336a00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f132800cf60 tx=0x7f132800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.220+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800d840 con 0x7f13380b0a90 2024-09-13T17:21:18.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.221+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132800d9a0 con 0x7f13380b0a90 2024-09-13T17:21:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.221+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132800dd30 con 0x7f13380b0a90 2024-09-13T17:21:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.221+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13380a9ab0 con 0x7f13380b0a90 2024-09-13T17:21:18.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.221+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13380aa070 con 0x7f13380b0a90 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.222+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13380cd360 con 0x7f13380b0a90 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.223+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1328015440 con 0x7f13380b0a90 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.223+0000 7f132f400640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 0x7f131406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.224+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f1328089e50 con 0x7f13380b0a90 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.224+0000 7f132e000640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 0x7f131406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:18.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.225+0000 7f132e000640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 0x7f131406d0f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:18.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.227+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1328054fb0 con 0x7f13380b0a90 2024-09-13T17:21:18.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:18.440+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f133815a270 con 0x7f13380b0a90 2024-09-13T17:21:19.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.030+0000 7f132f400640 1 -- 172.21.15.28:0/3246296068 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v158) v1 ==== 104+0+0 (secure 0 0 0) 0x7f132801a0a0 con 0x7f13380b0a90 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.032+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 msgr2=0x7f131406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.032+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 0x7f131406d0f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1320034000 tx=0x7f132002e000 comp rx=0 tx=0).stop 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.032+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 msgr2=0x7f13380abfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.032+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f132800cf60 tx=0x7f132800cf90 comp rx=0 tx=0).stop 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 shutdown_connections 2024-09-13T17:21:19.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f131406ac30 0x7f131406d0f0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1338159b30 0x7f1338163d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1338156650 0x7f13380aa590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 --2- 172.21.15.28:0/3246296068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13380b0a90 0x7f13380abfa0 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 >> 172.21.15.28:0/3246296068 conn(0x7f1338069e80 msgr2=0x7f1338156bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 shutdown_connections 2024-09-13T17:21:19.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.033+0000 7f1337e00640 1 -- 172.21.15.28:0/3246296068 wait complete. 2024-09-13T17:21:19.054 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:21:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.217+0000 7ff23b200640 1 Processor -- start 2024-09-13T17:21:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.217+0000 7ff23b200640 1 -- start start 2024-09-13T17:21:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.218+0000 7ff23b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff23c069e80 0x7ff23c06a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.218+0000 7ff23b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c06a7e0 con 0x7ff23c0b1920 2024-09-13T17:21:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.218+0000 7ff23b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23c0b87e0 con 0x7ff23c15c920 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.218+0000 7ff23b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23c0b7ca0 con 0x7ff23c069e80 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff239e00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7ff23c15c920 0x7ff23c0b8020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:37320/0 (socket says 172.21.15.28:37320) 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff239e00640 1 -- 172.21.15.28:0/3813866611 learned_addr learned my addr 172.21.15.28:0/3813866611 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff23a800640 1 --2- 172.21.15.28:0/3813866611 >> v2:172.21.15.28:3300/0 conn(0x7ff23c069e80 0x7ff23c06a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff23a800640 1 -- 172.21.15.28:0/3813866611 >> v1:172.21.15.189:6789/0 conn(0x7ff23c15c920 legacy=0x7ff23c0b8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff23a800640 1 -- 172.21.15.28:0/3813866611 >> v1:172.21.15.28:6789/0 conn(0x7ff23c0b1920 legacy=0x7ff23c0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.219+0000 7ff23a800640 1 -- 172.21.15.28:0/3813866611 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23c0d0ca0 con 0x7ff23c069e80 2024-09-13T17:21:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.220+0000 7ff23a800640 1 --2- 172.21.15.28:0/3813866611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c069e80 0x7ff23c06a2a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7ff234009d90 tx=0x7ff23402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297b60281fdef169 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.221+0000 7ff239400640 1 -- 172.21.15.28:0/3813866611 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234007740 con 0x7ff23c069e80 2024-09-13T17:21:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff239400640 1 -- 172.21.15.28:0/3813866611 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23402d050 con 0x7ff23c069e80 2024-09-13T17:21:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff239400640 1 -- 172.21.15.28:0/3813866611 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff234007ac0 con 0x7ff23c069e80 2024-09-13T17:21:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 -- 172.21.15.28:0/3813866611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c069e80 msgr2=0x7ff23c06a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 --2- 172.21.15.28:0/3813866611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c069e80 0x7ff23c06a2a0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7ff234009d90 tx=0x7ff23402a660 comp rx=0 tx=0).stop 2024-09-13T17:21:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 -- 172.21.15.28:0/3813866611 shutdown_connections 2024-09-13T17:21:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 --2- 172.21.15.28:0/3813866611 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c069e80 0x7ff23c06a2a0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 -- 172.21.15.28:0/3813866611 >> 172.21.15.28:0/3813866611 conn(0x7ff23c0708c0 msgr2=0x7ff23c070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 -- 172.21.15.28:0/3813866611 shutdown_connections 2024-09-13T17:21:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.222+0000 7ff23b200640 1 -- 172.21.15.28:0/3813866611 wait complete. 2024-09-13T17:21:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.224+0000 7ff23b200640 1 Processor -- start 2024-09-13T17:21:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.224+0000 7ff23b200640 1 -- start start 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff23c069e80 0x7ff23c0c8150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff23c15c920 0x7ff23c0c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0cb9b0 con 0x7ff23c0b1920 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c1540c0 con 0x7ff23c069e80 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23c0b47b0 con 0x7ff23c15c920 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32874/0 (socket says 172.21.15.28:32874) 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff233e00640 1 -- 172.21.15.28:0/315075568 learned_addr learned my addr 172.21.15.28:0/315075568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff23a800640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff23c15c920 0x7ff23c0c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:19.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.225+0000 7ff239e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff23c069e80 0x7ff23c0c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 -- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff23c15c920 msgr2=0x7ff23c0c61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff23c15c920 0x7ff23c0c61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 -- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff23c069e80 msgr2=0x7ff23c0c8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff23c069e80 0x7ff23c0c8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 -- 172.21.15.28:0/315075568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234009a70 con 0x7ff23c0b1920 2024-09-13T17:21:19.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff233e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c450 tx=0x7ff22400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2240052d0 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff224005430 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff224012870 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.226+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23c0ca440 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23c0caa00 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.228+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff23c0cabc0 con 0x7ff23c0b1920 2024-09-13T17:21:19.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.228+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff2240129d0 con 0x7ff23c0b1920 2024-09-13T17:21:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.229+0000 7ff232000640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 0x7ff21806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.229+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 5853+0+0 (secure 0 0 0) 0x7ff224088e30 con 0x7ff23c0b1920 2024-09-13T17:21:19.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.230+0000 7ff239e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 0x7ff21806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:19.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.230+0000 7ff239e00640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 0x7ff21806d0c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff22c005880 tx=0x7ff22c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:19.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.233+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff224053f90 con 0x7ff23c0b1920 2024-09-13T17:21:19.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:19.412+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff23c15c4d0 con 0x7ff23c0b1920 2024-09-13T17:21:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.223+0000 7ff232000640 1 -- 172.21.15.28:0/315075568 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v183) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff2240575b0 con 0x7ff23c0b1920 2024-09-13T17:21:20.226 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:21:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 msgr2=0x7ff21806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 0x7ff21806d0c0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7ff22c005880 tx=0x7ff22c009040 comp rx=0 tx=0).stop 2024-09-13T17:21:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 msgr2=0x7ff23c0c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7ff22400c450 tx=0x7ff22400c910 comp rx=0 tx=0).stop 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 shutdown_connections 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff21806ac00 0x7ff21806d0c0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff23c15c920 0x7ff23c0c61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff23c0b1920 0x7ff23c0c9ae0 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 --2- 172.21.15.28:0/315075568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff23c069e80 0x7ff23c0c8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 >> 172.21.15.28:0/315075568 conn(0x7ff23c0708c0 msgr2=0x7ff23c15dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.227+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 shutdown_connections 2024-09-13T17:21:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.228+0000 7ff23b200640 1 -- 172.21.15.28:0/315075568 wait complete. 2024-09-13T17:21:20.249 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:21:20.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.410+0000 7fd330600640 1 Processor -- start 2024-09-13T17:21:20.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.410+0000 7fd330600640 1 -- start start 2024-09-13T17:21:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.411+0000 7fd330600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 0x7fd3280b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.411+0000 7fd330600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3280b2630 0x7fd32815a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.411+0000 7fd330600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32815a4c0 con 0x7fd328159590 2024-09-13T17:21:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.411+0000 7fd330600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3280b2ae0 con 0x7fd3280b2630 2024-09-13T17:21:20.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.411+0000 7fd330600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd328071390 con 0x7fd328159970 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3280b2630 0x7fd32815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 0x7fd3280b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3280b2630 0x7fd32815a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:57026/0 (socket says 172.21.15.28:57026) 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 -- 172.21.15.28:0/2072440400 learned_addr learned my addr 172.21.15.28:0/2072440400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 -- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 msgr2=0x7fd3280b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:20.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 0x7fd3280b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 -- 172.21.15.28:0/2072440400 >> v1:172.21.15.189:6789/0 conn(0x7fd328159590 legacy=0x7fd328070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32e600640 1 -- 172.21.15.28:0/2072440400 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328071580 con 0x7fd3280b2630 2024-09-13T17:21:20.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.412+0000 7fd32d200640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 0x7fd3280b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.413+0000 7fd32e600640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3280b2630 0x7fd32815a950 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd320009e50 tx=0x7fd32002a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dfc9fd950d6688 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.413+0000 7fd327e00640 1 -- 172.21.15.28:0/2072440400 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320031600 con 0x7fd3280b2630 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.413+0000 7fd327e00640 1 -- 172.21.15.28:0/2072440400 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32002d050 con 0x7fd3280b2630 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.413+0000 7fd327e00640 1 -- 172.21.15.28:0/2072440400 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320031980 con 0x7fd3280b2630 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 -- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3280b2630 msgr2=0x7fd32815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3280b2630 0x7fd32815a950 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd320009e50 tx=0x7fd32002a710 comp rx=0 tx=0).stop 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 -- 172.21.15.28:0/2072440400 shutdown_connections 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3280b2630 0x7fd32815a950 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 --2- 172.21.15.28:0/2072440400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd328159970 0x7fd3280b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 -- 172.21.15.28:0/2072440400 >> 172.21.15.28:0/2072440400 conn(0x7fd328069e80 msgr2=0x7fd32806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 -- 172.21.15.28:0/2072440400 shutdown_connections 2024-09-13T17:21:20.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.414+0000 7fd330600640 1 -- 172.21.15.28:0/2072440400 wait complete. 2024-09-13T17:21:20.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.416+0000 7fd330600640 1 Processor -- start 2024-09-13T17:21:20.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.416+0000 7fd330600640 1 -- start start 2024-09-13T17:21:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3280b2630 0x7fd3280a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd328159970 0x7fd3281eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280a97e0 con 0x7fd328159590 2024-09-13T17:21:20.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280cc740 con 0x7fd328159970 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd330600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280cca20 con 0x7fd3280b2630 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32882/0 (socket says 172.21.15.28:32882) 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 -- 172.21.15.28:0/1847576814 learned_addr learned my addr 172.21.15.28:0/1847576814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32dc00640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3280b2630 0x7fd3280a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32e600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd328159970 0x7fd3281eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 -- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3280b2630 msgr2=0x7fd3280a92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3280b2630 0x7fd3280a92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 -- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd328159970 msgr2=0x7fd3281eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.417+0000 7fd32d200640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd328159970 0x7fd3281eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd32d200640 1 -- 172.21.15.28:0/1847576814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314009400 con 0x7fd328159590 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd32d200640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd31400e980 tx=0x7fd31400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:20.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31400ca00 con 0x7fd328159590 2024-09-13T17:21:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31400cb60 con 0x7fd328159590 2024-09-13T17:21:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd314013470 con 0x7fd328159590 2024-09-13T17:21:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd320009b30 con 0x7fd328159590 2024-09-13T17:21:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.418+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3280ccfe0 con 0x7fd328159590 2024-09-13T17:21:20.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.419+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32815a950 con 0x7fd328159590 2024-09-13T17:21:20.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.420+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3140135d0 con 0x7fd328159590 2024-09-13T17:21:20.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.421+0000 7fd325600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 0x7fd30806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:20.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.421+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5415+0+0 (secure 0 0 0) 0x7fd3140894d0 con 0x7fd328159590 2024-09-13T17:21:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.422+0000 7fd32dc00640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 0x7fd30806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.422+0000 7fd32dc00640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 0x7fd30806d0c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd31c000f60 tx=0x7fd31c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:20.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.425+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd3140547e0 con 0x7fd328159590 2024-09-13T17:21:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:20.606+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd32806b5f0 con 0x7fd328159590 2024-09-13T17:21:21.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.238+0000 7fd325600640 1 -- 172.21.15.28:0/1847576814 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v184) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd314057e00 con 0x7fd328159590 2024-09-13T17:21:21.240 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:21:21.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.242+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 msgr2=0x7fd30806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.242+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 0x7fd30806d0c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fd31c000f60 tx=0x7fd31c009040 comp rx=0 tx=0).stop 2024-09-13T17:21:21.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.242+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 msgr2=0x7fd328153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.242+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 secure :-1 s=READY pgs=2256 cs=0 l=1 rev1=1 crypto rx=0x7fd31400e980 tx=0x7fd31400ee40 comp rx=0 tx=0).stop 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 shutdown_connections 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd30806ac00 0x7fd30806d0c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd328159970 0x7fd3281eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd328159590 0x7fd328153400 unknown :-1 s=CLOSED pgs=2256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 --2- 172.21.15.28:0/1847576814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3280b2630 0x7fd3280a92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 >> 172.21.15.28:0/1847576814 conn(0x7fd328069e80 msgr2=0x7fd32806b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:21.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 shutdown_connections 2024-09-13T17:21:21.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.243+0000 7fd330600640 1 -- 172.21.15.28:0/1847576814 wait complete. 2024-09-13T17:21:21.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:21:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.428+0000 7f795b200640 1 Processor -- start 2024-09-13T17:21:21.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.428+0000 7f795b200640 1 -- start start 2024-09-13T17:21:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 0x7f795c071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 0x7f795c070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c159e50 con 0x7f795c071100 2024-09-13T17:21:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f795c0715b0 con 0x7f795c071f60 2024-09-13T17:21:21.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c0ce210 con 0x7f795c0cde30 2024-09-13T17:21:21.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795a800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f795c071100 0x7f795c15cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34712/0 (socket says 172.21.15.28:34712) 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.429+0000 7f795a800640 1 -- 172.21.15.28:0/1651864069 learned_addr learned my addr 172.21.15.28:0/1651864069 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959e00640 1 --2- 172.21.15.28:0/1651864069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 0x7f795c071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7951400640 1 --2- 172.21.15.28:0/1651864069 >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 0x7f795c070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959400640 1 -- 172.21.15.28:0/1651864069 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690867176 0 0) 0x7f795c159e50 con 0x7f795c071100 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959400640 1 -- 172.21.15.28:0/1651864069 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940003520 con 0x7f795c071100 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959e00640 1 -- 172.21.15.28:0/1651864069 >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 msgr2=0x7f795c070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959e00640 1 --2- 172.21.15.28:0/1651864069 >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 0x7f795c070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959e00640 1 -- 172.21.15.28:0/1651864069 >> v1:172.21.15.28:6789/0 conn(0x7f795c071100 legacy=0x7f795c15cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7959e00640 1 -- 172.21.15.28:0/1651864069 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c15a130 con 0x7f795c0cde30 2024-09-13T17:21:21.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.430+0000 7f7951400640 1 --2- 172.21.15.28:0/1651864069 >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 0x7f795c070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.431+0000 7f7959e00640 1 --2- 172.21.15.28:0/1651864069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 0x7f795c071a20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f794c0097a0 tx=0x7f794c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6569a2025eb0e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f7959400640 1 -- 172.21.15.28:0/1651864069 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c032440 con 0x7f795c0cde30 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f7959400640 1 -- 172.21.15.28:0/1651864069 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f794c0325a0 con 0x7f795c0cde30 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f7959400640 1 -- 172.21.15.28:0/1651864069 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f794c032910 con 0x7f795c0cde30 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 -- 172.21.15.28:0/1651864069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 msgr2=0x7f795c071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 --2- 172.21.15.28:0/1651864069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 0x7f795c071a20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f794c0097a0 tx=0x7f794c02a180 comp rx=0 tx=0).stop 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 -- 172.21.15.28:0/1651864069 shutdown_connections 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 --2- 172.21.15.28:0/1651864069 >> v2:172.21.15.189:3300/0 conn(0x7f795c071f60 0x7f795c070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 --2- 172.21.15.28:0/1651864069 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c0cde30 0x7f795c071a20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 -- 172.21.15.28:0/1651864069 >> 172.21.15.28:0/1651864069 conn(0x7f795c069e80 msgr2=0x7f795c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 -- 172.21.15.28:0/1651864069 shutdown_connections 2024-09-13T17:21:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.432+0000 7f795b200640 1 -- 172.21.15.28:0/1651864069 wait complete. 2024-09-13T17:21:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.434+0000 7f795b200640 1 Processor -- start 2024-09-13T17:21:21.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.434+0000 7f795b200640 1 -- start start 2024-09-13T17:21:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 0x7f795c1e4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c071f60 0x7f795c14dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 0x7f795c150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c14f050 con 0x7f795c0cde30 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f795c1510c0 con 0x7f795c071100 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.435+0000 7f795b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f795c150810 con 0x7f795c071f60 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 0x7f795c150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f7959e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 0x7f795c1e4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f7951400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c071f60 0x7f795c14dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f7959e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 0x7f795c1e4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:57042/0 (socket says 172.21.15.28:57042) 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f7959e00640 1 -- 172.21.15.28:0/2338153669 learned_addr learned my addr 172.21.15.28:0/2338153669 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 -- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c071f60 msgr2=0x7f795c14dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c071f60 0x7f795c14dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 -- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 msgr2=0x7f795c1e4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 0x7f795c1e4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 -- 172.21.15.28:0/2338153669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944009400 con 0x7f795c0cde30 2024-09-13T17:21:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.436+0000 7f795a800640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 0x7f795c150b80 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f795400ca80 tx=0x7f795400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.437+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795400ac30 con 0x7f795c0cde30 2024-09-13T17:21:21.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.437+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795400ad90 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.437+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7954012600 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.437+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f794c009400 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.437+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c150530 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.438+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795c14f190 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.439+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7954007070 con 0x7f795c0cde30 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.440+0000 7f7952a00640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 0x7f793806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.440+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7954088f90 con 0x7f795c0cde30 2024-09-13T17:21:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.440+0000 7f7959e00640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 0x7f793806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.441+0000 7f7959e00640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 0x7f793806d0c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f794c0097a0 tx=0x7f794c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.444+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f795c14f190 con 0x7f795c0cde30 2024-09-13T17:21:21.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.623+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f795c14f190 con 0x7f795c0cde30 2024-09-13T17:21:21.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.627+0000 7f7952a00640 1 -- 172.21.15.28:0/2338153669 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f795c14f190 con 0x7f795c0cde30 2024-09-13T17:21:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 msgr2=0x7f793806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 0x7f793806d0c0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f794c0097a0 tx=0x7f794c02e000 comp rx=0 tx=0).stop 2024-09-13T17:21:21.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 msgr2=0x7f795c150b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 0x7f795c150b80 secure :-1 s=READY pgs=2258 cs=0 l=1 rev1=1 crypto rx=0x7f795400ca80 tx=0x7f795400cf40 comp rx=0 tx=0).stop 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 shutdown_connections 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f793806ac00 0x7f793806d0c0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795c0cde30 0x7f795c150b80 unknown :-1 s=CLOSED pgs=2258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f795c071f60 0x7f795c14dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 --2- 172.21.15.28:0/2338153669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f795c071100 0x7f795c1e4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 >> 172.21.15.28:0/2338153669 conn(0x7f795c069e80 msgr2=0x7f795c0b06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 shutdown_connections 2024-09-13T17:21:21.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.629+0000 7f795b200640 1 -- 172.21.15.28:0/2338153669 wait complete. 2024-09-13T17:21:21.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive' 2024-09-13T17:21:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.801+0000 7f472f200640 1 Processor -- start 2024-09-13T17:21:21.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.801+0000 7f472f200640 1 -- start start 2024-09-13T17:21:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.802+0000 7f472f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f473006b360 0x7f47300b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.802+0000 7f472f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47300b0d80 con 0x7f47300b0980 2024-09-13T17:21:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.802+0000 7f472f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473006b900 con 0x7f4730154a90 2024-09-13T17:21:21.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.802+0000 7f472f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473006bbe0 con 0x7f473006b360 2024-09-13T17:21:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.802+0000 7f472e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f47300b0980 0x7f47300af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34724/0 (socket says 172.21.15.28:34724) 2024-09-13T17:21:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472e800640 1 -- 172.21.15.28:0/3647885522 learned_addr learned my addr 172.21.15.28:0/3647885522 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630282536 0 0) 0x7f47300b0d80 con 0x7f47300b0980 2024-09-13T17:21:21.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4714003520 con 0x7f47300b0980 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 667403261 0 0) 0x7f4714003520 con 0x7f47300b0980 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 >> v2:172.21.15.28:3300/0 conn(0x7f473006b360 msgr2=0x7f47300b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 --2- 172.21.15.28:0/3647885522 >> v2:172.21.15.28:3300/0 conn(0x7f473006b360 0x7f47300b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 >> v1:172.21.15.189:6789/0 conn(0x7f4730154a90 legacy=0x7f473015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473006bec0 con 0x7f47300b0980 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f47280019f0 con 0x7f47300b0980 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 >> v1:172.21.15.28:6789/0 conn(0x7f47300b0980 legacy=0x7f47300af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 0x7f4714003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47140047c0 0x7f4714004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47140052a0 0x7f471400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473006bbe0 con 0x7f4714003970 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471400c1a0 con 0x7f47140047c0 2024-09-13T17:21:21.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.803+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471400c360 con 0x7f47140052a0 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 0x7f4714003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472e800640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47140052a0 0x7f471400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472d400640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47140047c0 0x7f4714004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 -- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47140052a0 msgr2=0x7f471400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47140052a0 0x7f471400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 -- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47140047c0 msgr2=0x7f4714004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f47140047c0 0x7f4714004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472de00640 1 -- 172.21.15.28:0/3647885522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714003520 con 0x7f4714003970 2024-09-13T17:21:21.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.804+0000 7f472e800640 1 -- 172.21.15.28:0/3647885522 reap_dead start 2024-09-13T17:21:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.805+0000 7f472de00640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 0x7f4714003d70 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f4720001620 tx=0x7f47140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64e39dc1a3b055f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.805+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720005270 con 0x7f4714003970 2024-09-13T17:21:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.805+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47200053f0 con 0x7f4714003970 2024-09-13T17:21:21.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.805+0000 7f472ca00640 1 -- 172.21.15.28:0/3647885522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472002a9b0 con 0x7f4714003970 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 -- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 msgr2=0x7f4714003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 0x7f4714003d70 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f4720001620 tx=0x7f47140057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 -- 172.21.15.28:0/3647885522 shutdown_connections 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 --2- 172.21.15.28:0/3647885522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4714003970 0x7f4714003d70 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 -- 172.21.15.28:0/3647885522 >> 172.21.15.28:0/3647885522 conn(0x7f4730072e40 msgr2=0x7f4730073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 -- 172.21.15.28:0/3647885522 shutdown_connections 2024-09-13T17:21:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.806+0000 7f472f200640 1 -- 172.21.15.28:0/3647885522 wait complete. 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.808+0000 7f472f200640 1 Processor -- start 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.808+0000 7f472f200640 1 -- start start 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 0x7f47301f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 0x7f473013a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4730139ff0 con 0x7f473013a9c0 2024-09-13T17:21:21.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47301f3170 con 0x7f4714003970 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47301393e0 con 0x7f47301580e0 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32912/0 (socket says 172.21.15.28:32912) 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472e800640 1 -- 172.21.15.28:0/2343654646 learned_addr learned my addr 172.21.15.28:0/2343654646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472d400640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 0x7f473013a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.809+0000 7f472de00640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 0x7f47301f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 -- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 msgr2=0x7f473013a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 0x7f473013a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 -- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 msgr2=0x7f47301f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 0x7f47301f2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 -- 172.21.15.28:0/2343654646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47301396c0 con 0x7f473013a9c0 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472d400640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 0x7f473013a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472de00640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 0x7f47301f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472e800640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f473015e080 tx=0x7f4728007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472800ab80 con 0x7f473013a9c0 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472800ace0 con 0x7f473013a9c0 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728013630 con 0x7f473013a9c0 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47301399a0 con 0x7f473013a9c0 2024-09-13T17:21:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.810+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4730153c90 con 0x7f473013a9c0 2024-09-13T17:21:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.811+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f473013a1a0 con 0x7f473013a9c0 2024-09-13T17:21:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.812+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4728013790 con 0x7f473013a9c0 2024-09-13T17:21:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.813+0000 7f4726000640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 0x7f471006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.813+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4728089770 con 0x7f473013a9c0 2024-09-13T17:21:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.813+0000 7f472de00640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 0x7f471006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.814+0000 7f472de00640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 0x7f471006d0c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4720032000 tx=0x7f4720005000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:21.817+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f473013a1a0 con 0x7f473013a9c0 2024-09-13T17:21:22.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.062+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]} v 0) v1 -- 0x7f473013a1a0 con 0x7f473013a9c0 2024-09-13T17:21:22.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.239+0000 7f4726000640 1 -- 172.21.15.28:0/2343654646 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub_checks.TestScrubControls.test_scrub_when_mds_is_inactive"]}]=0 v662) v1 ==== 159+0+0 (secure 0 0 0) 0x7f473013a1a0 con 0x7f473013a9c0 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 msgr2=0x7f471006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 0x7f471006d0c0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f4720032000 tx=0x7f4720005000 comp rx=0 tx=0).stop 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 msgr2=0x7f4730155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f473015e080 tx=0x7f4728007bd0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 shutdown_connections 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471006ac00 0x7f471006d0c0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473013a9c0 0x7f4730155e20 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f47301580e0 0x7f473013a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 --2- 172.21.15.28:0/2343654646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4714003970 0x7f47301f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 >> 172.21.15.28:0/2343654646 conn(0x7f4730072e40 msgr2=0x7f473015c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 shutdown_connections 2024-09-13T17:21:22.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.242+0000 7f472f200640 1 -- 172.21.15.28:0/2343654646 wait complete. 2024-09-13T17:21:22.260 INFO:tasks.cephfs_test_runner:test_scrub_when_mds_is_inactive (tasks.cephfs.test_scrub_checks.TestScrubControls) ... ok 2024-09-13T17:21:22.261 INFO:tasks.cephfs_test_runner:Starting test: test_mdsdir_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) 2024-09-13T17:21:22.261 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:22.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 Processor -- start 2024-09-13T17:21:22.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 -- start start 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7febb8160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb8146550 con 0x7febb8146170 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febb815d280 con 0x7febb80ac850 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febbe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb815cd20 con 0x7febb8160310 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febb7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7febb8146170 0x7febb815d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34728/0 (socket says 172.21.15.28:34728) 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.416+0000 7febb7400640 1 -- 172.21.15.28:0/1750776549 learned_addr learned my addr 172.21.15.28:0/1750776549 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7febb8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518399268 0 0) 0x7febb8146550 con 0x7febb8146170 2024-09-13T17:21:22.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c003520 con 0x7febb8146170 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335580617 0 0) 0x7febb815d280 con 0x7febb80ac850 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb8146550 con 0x7febb80ac850 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397720482 0 0) 0x7feb9c003520 con 0x7febb8146170 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 msgr2=0x7febb8160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7febb8160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 >> v1:172.21.15.189:6789/0 conn(0x7febb80ac850 legacy=0x7febb806b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb8160d60 con 0x7febb8146170 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7feba8001670 con 0x7febb8146170 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7febb8160730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 >> v1:172.21.15.28:6789/0 conn(0x7febb8146170 legacy=0x7febb815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7feb9c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.417+0000 7febb6000640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7feb9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6000640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9c004d00 0x7feb9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00ba60 con 0x7feb9c0040c0 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00bbd0 con 0x7feb9c004d00 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00bd90 con 0x7febb8160310 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7feb9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb7400640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7feb9c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb7e00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9c004d00 0x7feb9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 -- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 msgr2=0x7feb9c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb8160310 0x7feb9c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 -- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9c004d00 msgr2=0x7feb9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9c004d00 0x7feb9c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.418+0000 7febb6a00640 1 -- 172.21.15.28:0/1750776549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4009400 con 0x7feb9c0040c0 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.419+0000 7febb6a00640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7feb9c0047c0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7febb806d550 tx=0x7feb9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd444f22aa56a64 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.419+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba402aea0 con 0x7feb9c0040c0 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.419+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba402d050 con 0x7feb9c0040c0 2024-09-13T17:21:22.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.419+0000 7febb6000640 1 -- 172.21.15.28:0/1750776549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba4031620 con 0x7feb9c0040c0 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 -- 172.21.15.28:0/1750776549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 msgr2=0x7feb9c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 --2- 172.21.15.28:0/1750776549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7feb9c0047c0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7febb806d550 tx=0x7feb9c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febb7e00640 1 -- 172.21.15.28:0/1750776549 reap_dead start 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 -- 172.21.15.28:0/1750776549 shutdown_connections 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 -- 172.21.15.28:0/1750776549 >> 172.21.15.28:0/1750776549 conn(0x7febb8069e80 msgr2=0x7febb806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 -- 172.21.15.28:0/1750776549 shutdown_connections 2024-09-13T17:21:22.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.420+0000 7febbe800640 1 -- 172.21.15.28:0/1750776549 wait complete. 2024-09-13T17:21:22.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.422+0000 7febbe800640 1 Processor -- start 2024-09-13T17:21:22.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.422+0000 7febbe800640 1 -- start start 2024-09-13T17:21:22.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb80af160 0x7febb80af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb815b660 0x7febb815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb815c020 con 0x7feb9c0040c0 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb80b08e0 con 0x7febb80af160 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febbe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febb81f65e0 con 0x7febb815b660 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32928/0 (socket says 172.21.15.28:32928) 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 -- 172.21.15.28:0/3170333789 learned_addr learned my addr 172.21.15.28:0/3170333789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb6a00640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb80af160 0x7febb80af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7e00640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb815b660 0x7febb815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 -- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb815b660 msgr2=0x7febb815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb815b660 0x7febb815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 -- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb80af160 msgr2=0x7febb80af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb80af160 0x7febb80af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.423+0000 7febb7400640 1 -- 172.21.15.28:0/3170333789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febb80afcd0 con 0x7feb9c0040c0 2024-09-13T17:21:22.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febb7400640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7feba800e760 tx=0x7feba800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800ee70 con 0x7feb9c0040c0 2024-09-13T17:21:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feba800c7f0 con 0x7feb9c0040c0 2024-09-13T17:21:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feba800cba0 con 0x7feb9c0040c0 2024-09-13T17:21:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb80affb0 con 0x7feb9c0040c0 2024-09-13T17:21:22.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.424+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb80b0570 con 0x7feb9c0040c0 2024-09-13T17:21:22.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.425+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb8160730 con 0x7feb9c0040c0 2024-09-13T17:21:22.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.426+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feba800cd00 con 0x7feb9c0040c0 2024-09-13T17:21:22.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.428+0000 7febaf400640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 0x7feb9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.428+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feba8088ea0 con 0x7feb9c0040c0 2024-09-13T17:21:22.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.428+0000 7febb6a00640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 0x7feb9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.428+0000 7febb6a00640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 0x7feb9806d0c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7feba4033000 tx=0x7feba4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.431+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feba8054390 con 0x7feb9c0040c0 2024-09-13T17:21:22.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.610+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febb80710e0 con 0x7feb9c0040c0 2024-09-13T17:21:22.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.610+0000 7febaf400640 1 -- 172.21.15.28:0/3170333789 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v184) v1 ==== 74+0+12996 (secure 0 0 0) 0x7feba80579b0 con 0x7feb9c0040c0 2024-09-13T17:21:22.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:22.613 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":184,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:21.230826+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/3229070352":"2024-09-14T17:21:18.019641+0000","172.21.15.28:6834/3229070352":"2024-09-14T17:21:18.019641+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.612+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 msgr2=0x7feb9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.612+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 0x7feb9806d0c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7feba4033000 tx=0x7feba4008030 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.612+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 msgr2=0x7febb81f5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.612+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7feba800e760 tx=0x7feba800ec20 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 shutdown_connections 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb9806ac00 0x7feb9806d0c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7febb815b660 0x7febb815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7febb80af160 0x7febb80af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 --2- 172.21.15.28:0/3170333789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9c0040c0 0x7febb81f5a40 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 >> 172.21.15.28:0/3170333789 conn(0x7febb8069e80 msgr2=0x7febb815f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 shutdown_connections 2024-09-13T17:21:22.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.613+0000 7febbe800640 1 -- 172.21.15.28:0/3170333789 wait complete. 2024-09-13T17:21:22.630 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:21:22.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:21:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.787+0000 7fb828200640 1 Processor -- start 2024-09-13T17:21:22.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.787+0000 7fb828200640 1 -- start start 2024-09-13T17:21:22.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.788+0000 7fb828200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb82015a780 0x7fb820070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.788+0000 7fb828200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8200712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.788+0000 7fb828200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82006c680 con 0x7fb82015ab60 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb828200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82006c8b0 con 0x7fb82015a780 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb828200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb82006cb90 con 0x7fb82006c250 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb824e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb82015ab60 0x7fb82006bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34742/0 (socket says 172.21.15.28:34742) 2024-09-13T17:21:22.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb824e00640 1 -- 172.21.15.28:0/505731524 learned_addr learned my addr 172.21.15.28:0/505731524 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3397462127 0 0) 0x7fb82006c680 con 0x7fb82015ab60 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb808003520 con 0x7fb82015ab60 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb825800640 1 --2- 172.21.15.28:0/505731524 >> v2:172.21.15.189:3300/0 conn(0x7fb82015a780 0x7fb820070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.789+0000 7fb826200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8200712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745250954 0 0) 0x7fb808003520 con 0x7fb82015ab60 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 msgr2=0x7fb8200712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8200712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 >> v2:172.21.15.189:3300/0 conn(0x7fb82015a780 msgr2=0x7fb820070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 --2- 172.21.15.28:0/505731524 >> v2:172.21.15.189:3300/0 conn(0x7fb82015a780 0x7fb820070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8200cc980 con 0x7fb82015ab60 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fb80c001670 con 0x7fb82015ab60 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 >> v1:172.21.15.28:6789/0 conn(0x7fb82015ab60 legacy=0x7fb82006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb808004d00 0x7fb808005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82006c8b0 con 0x7fb808004d00 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb82006cb90 con 0x7fb808003970 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80800b980 con 0x7fb82006c250 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb826200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb808004d00 0x7fb808005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb825800640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.790+0000 7fb824e00640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.791+0000 7fb826200640 1 -- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 msgr2=0x7fb8080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.791+0000 7fb826200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb82006c250 0x7fb8080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.791+0000 7fb826200640 1 -- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808003970 msgr2=0x7fb808003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.791+0000 7fb826200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808003970 0x7fb808003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.791+0000 7fb826200640 1 -- 172.21.15.28:0/505731524 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808003520 con 0x7fb808004d00 2024-09-13T17:21:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb826200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb808004d00 0x7fb808005180 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb818009fc0 tx=0x7fb8080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29f40f060473f8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818031640 con 0x7fb808004d00 2024-09-13T17:21:22.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81802d070 con 0x7fb808004d00 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb81fe00640 1 -- 172.21.15.28:0/505731524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8180319e0 con 0x7fb808004d00 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb828200640 1 -- 172.21.15.28:0/505731524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb808004d00 msgr2=0x7fb808005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb828200640 1 --2- 172.21.15.28:0/505731524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb808004d00 0x7fb808005180 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7fb818009fc0 tx=0x7fb8080057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb826200640 1 -- 172.21.15.28:0/505731524 reap_dead start 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb828200640 1 -- 172.21.15.28:0/505731524 shutdown_connections 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.792+0000 7fb828200640 1 -- 172.21.15.28:0/505731524 >> 172.21.15.28:0/505731524 conn(0x7fb8200b0270 msgr2=0x7fb8200b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.793+0000 7fb828200640 1 -- 172.21.15.28:0/505731524 shutdown_connections 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.793+0000 7fb828200640 1 -- 172.21.15.28:0/505731524 wait complete. 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.794+0000 7fb828200640 1 Processor -- start 2024-09-13T17:21:22.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.794+0000 7fb828200640 1 -- start start 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808004d00 0x7fb82014c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8200acac0 0x7fb8200c52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c5cb0 con 0x7fb8200c5810 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c34e0 con 0x7fb808004d00 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb828200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200c37c0 con 0x7fb8200acac0 2024-09-13T17:21:22.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb826200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.795+0000 7fb826200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32946/0 (socket says 172.21.15.28:32946) 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 -- 172.21.15.28:0/1245866842 learned_addr learned my addr 172.21.15.28:0/1245866842 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb825800640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808004d00 0x7fb82014c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb824e00640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8200acac0 0x7fb8200c52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 -- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8200acac0 msgr2=0x7fb8200c52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8200acac0 0x7fb8200c52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 -- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808004d00 msgr2=0x7fb82014c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808004d00 0x7fb82014c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 -- 172.21.15.28:0/1245866842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8200c3aa0 con 0x7fb8200c5810 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb826200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e230 tx=0x7fb818009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.796+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818031d30 con 0x7fb8200c5810 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.797+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818040070 con 0x7fb8200c5810 2024-09-13T17:21:22.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.797+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb818003bd0 con 0x7fb8200c5810 2024-09-13T17:21:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.797+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8200c3d80 con 0x7fb8200c5810 2024-09-13T17:21:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.797+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82014d010 con 0x7fb8200c5810 2024-09-13T17:21:22.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.798+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8200676b0 con 0x7fb8200c5810 2024-09-13T17:21:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.799+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb818042070 con 0x7fb8200c5810 2024-09-13T17:21:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.799+0000 7fb81d600640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 0x7fb80006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.799+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fb8180a9e60 con 0x7fb8200c5810 2024-09-13T17:21:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.800+0000 7fb825800640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 0x7fb80006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:22.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.801+0000 7fb825800640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 0x7fb80006d0c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb814009c40 tx=0x7fb814009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:22.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.804+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb8180753c0 con 0x7fb8200c5810 2024-09-13T17:21:22.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.979+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb82014d010 con 0x7fb8200c5810 2024-09-13T17:21:22.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.980+0000 7fb81d600640 1 -- 172.21.15.28:0/1245866842 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v184) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb82014d010 con 0x7fb8200c5810 2024-09-13T17:21:22.981 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:21:22.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 msgr2=0x7fb80006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 0x7fb80006d0c0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fb814009c40 tx=0x7fb814009190 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 msgr2=0x7fb8200c4280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7fb81800e230 tx=0x7fb818009750 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 shutdown_connections 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb80006ac00 0x7fb80006d0c0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8200c5810 0x7fb8200c4280 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8200acac0 0x7fb8200c52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 --2- 172.21.15.28:0/1245866842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb808004d00 0x7fb82014c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 >> 172.21.15.28:0/1245866842 conn(0x7fb8200b0270 msgr2=0x7fb8200cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 shutdown_connections 2024-09-13T17:21:22.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:22.982+0000 7fb828200640 1 -- 172.21.15.28:0/1245866842 wait complete. 2024-09-13T17:21:22.999 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace' 2024-09-13T17:21:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.184+0000 7f0683c00640 1 Processor -- start 2024-09-13T17:21:23.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.184+0000 7f0683c00640 1 -- start start 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0683c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0683c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f067c15cd80 0x7f067c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0683c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f067c06f920 con 0x7f067c06f3f0 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0683c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c15b290 con 0x7f067c15cd80 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0683c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c15b570 con 0x7f067c159770 2024-09-13T17:21:23.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f0681c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f067c15cd80 0x7f067c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f067be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f067c06f3f0 0x7f067c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:34756/0 (socket says 172.21.15.28:34756) 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.185+0000 7f067be00640 1 -- 172.21.15.28:0/845125378 learned_addr learned my addr 172.21.15.28:0/845125378 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f0681200640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f0681c00640 1 -- 172.21.15.28:0/845125378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 msgr2=0x7f067c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f0681c00640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f0681c00640 1 -- 172.21.15.28:0/845125378 >> v1:172.21.15.28:6789/0 conn(0x7f067c06f3f0 legacy=0x7f067c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f0681c00640 1 -- 172.21.15.28:0/845125378 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c15b850 con 0x7f067c15cd80 2024-09-13T17:21:23.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.186+0000 7f067b400640 1 -- 172.21.15.28:0/845125378 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243593102 0 0) 0x7f067c06f920 con 0x7f067c06f3f0 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.187+0000 7f0681c00640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c15cd80 0x7f067c070260 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0674009d90 tx=0x7f067402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df7a079b0b2e3b2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.187+0000 7f067b400640 1 -- 172.21.15.28:0/845125378 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674031600 con 0x7f067c15cd80 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.187+0000 7f067b400640 1 -- 172.21.15.28:0/845125378 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067402d050 con 0x7f067c15cd80 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.187+0000 7f067b400640 1 -- 172.21.15.28:0/845125378 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0674031980 con 0x7f067c15cd80 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 -- 172.21.15.28:0/845125378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c15cd80 msgr2=0x7f067c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c15cd80 0x7f067c070260 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f0674009d90 tx=0x7f067402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 -- 172.21.15.28:0/845125378 shutdown_connections 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c15cd80 0x7f067c070260 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 --2- 172.21.15.28:0/845125378 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 -- 172.21.15.28:0/845125378 >> 172.21.15.28:0/845125378 conn(0x7f067c0b0270 msgr2=0x7f067c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 -- 172.21.15.28:0/845125378 shutdown_connections 2024-09-13T17:21:23.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.188+0000 7f0683c00640 1 -- 172.21.15.28:0/845125378 wait complete. 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.189+0000 7f0683c00640 1 Processor -- start 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 -- start start 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067c15cd80 0x7f067c14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c148ff0 con 0x7f067c06f3f0 2024-09-13T17:21:23.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c14b800 con 0x7f067c15cd80 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.190+0000 7f0683c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c14cec0 con 0x7f067c159770 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:32968/0 (socket says 172.21.15.28:32968) 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 -- 172.21.15.28:0/2206923005 learned_addr learned my addr 172.21.15.28:0/2206923005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f067be00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067c15cd80 0x7f067c14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 -- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 msgr2=0x7f067c0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 -- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067c15cd80 msgr2=0x7f067c14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067c15cd80 0x7f067c14ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.191+0000 7f0681200640 1 -- 172.21.15.28:0/2206923005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0674009a70 con 0x7f067c06f3f0 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0681200640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f067000d800 tx=0x7f067000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:23.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067000b6c0 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f067000b820 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f067000bbb0 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c14d1a0 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.192+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c14d760 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.193+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c0700e0 con 0x7f067c06f3f0 2024-09-13T17:21:23.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.194+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f067000bd10 con 0x7f067c06f3f0 2024-09-13T17:21:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.195+0000 7f0678c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 0x7f065c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.195+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f06700895a0 con 0x7f067c06f3f0 2024-09-13T17:21:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.195+0000 7f067be00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 0x7f065c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:23.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.196+0000 7f067be00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 0x7f065c06d0c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f067c1547a0 tx=0x7f0668007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:23.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.199+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0670054a90 con 0x7f067c06f3f0 2024-09-13T17:21:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:23.445+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace"]} v 0) v1 -- 0x7f067c06ca10 con 0x7f067c06f3f0 2024-09-13T17:21:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.248+0000 7f0678c00640 1 -- 172.21.15.28:0/2206923005 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace"]}]=0 v664) v1 ==== 143+0+0 (secure 0 0 0) 0x7f06700580b0 con 0x7f067c06f3f0 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 msgr2=0x7f065c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 0x7f065c06d0c0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f067c1547a0 tx=0x7f0668007d50 comp rx=0 tx=0).stop 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 msgr2=0x7f067c148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f067000d800 tx=0x7f067000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 shutdown_connections 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f065c06ac00 0x7f065c06d0c0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f067c15cd80 0x7f067c14ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f067c159770 0x7f067c0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 --2- 172.21.15.28:0/2206923005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067c06f3f0 0x7f067c148ab0 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 >> 172.21.15.28:0/2206923005 conn(0x7f067c0b0270 msgr2=0x7f067c159b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 shutdown_connections 2024-09-13T17:21:24.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.251+0000 7f0683c00640 1 -- 172.21.15.28:0/2206923005 wait complete. 2024-09-13T17:21:24.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:21:24.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.437+0000 7f0dd3e00640 1 Processor -- start 2024-09-13T17:21:24.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.437+0000 7f0dd3e00640 1 -- start start 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.438+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd4071c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.438+0000 7f0dd3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dd4072140 0x7f0dd40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.438+0000 7f0dd3e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4159e10 con 0x7f0dd4071210 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.438+0000 7f0dd3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd415a0f0 con 0x7f0dd4072140 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.438+0000 7f0dd3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd415a3d0 con 0x7f0dd4070e30 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dd4072140 0x7f0dd40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0dd4072140 0x7f0dd40cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50296/0 (socket says 172.21.15.28:50296) 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 -- 172.21.15.28:0/871731029 learned_addr learned my addr 172.21.15.28:0/871731029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:24.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dd2a00640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd4071c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 -- 172.21.15.28:0/871731029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 msgr2=0x7f0dd4071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd4071c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 -- 172.21.15.28:0/871731029 >> v1:172.21.15.189:6789/0 conn(0x7f0dd4071210 legacy=0x7f0dd415d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.439+0000 7f0dca000640 1 -- 172.21.15.28:0/871731029 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd415a6b0 con 0x7f0dd4072140 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.440+0000 7f0dca000640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4072140 0x7f0dd40cd870 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009870 tx=0x7f0dbc02c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852f629dbf2aeaf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.440+0000 7f0dd2000640 1 -- 172.21.15.28:0/871731029 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc031710 con 0x7f0dd4072140 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.440+0000 7f0dd2000640 1 -- 172.21.15.28:0/871731029 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc02f050 con 0x7f0dd4072140 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.440+0000 7f0dd2000640 1 -- 172.21.15.28:0/871731029 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc031a90 con 0x7f0dd4072140 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 -- 172.21.15.28:0/871731029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4072140 msgr2=0x7f0dd40cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4072140 0x7f0dd40cd870 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc009870 tx=0x7f0dbc02c480 comp rx=0 tx=0).stop 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 -- 172.21.15.28:0/871731029 shutdown_connections 2024-09-13T17:21:24.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4072140 0x7f0dd40cd870 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/871731029 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd4071c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 -- 172.21.15.28:0/871731029 >> 172.21.15.28:0/871731029 conn(0x7f0dd4069e80 msgr2=0x7f0dd406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 -- 172.21.15.28:0/871731029 shutdown_connections 2024-09-13T17:21:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.441+0000 7f0dd3e00640 1 -- 172.21.15.28:0/871731029 wait complete. 2024-09-13T17:21:24.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.443+0000 7f0dd3e00640 1 Processor -- start 2024-09-13T17:21:24.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.443+0000 7f0dd3e00640 1 -- start start 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd414ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd4072140 0x7f0dd414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd414ba90 con 0x7f0dd4071210 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd414b7b0 con 0x7f0dd4072140 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd414e090 con 0x7f0dd4070e30 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50302/0 (socket says 172.21.15.28:50302) 2024-09-13T17:21:24.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 -- 172.21.15.28:0/3175912785 learned_addr learned my addr 172.21.15.28:0/3175912785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd2a00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd414ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dd3400640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd4072140 0x7f0dd414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 -- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 msgr2=0x7f0dd414ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd414ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 -- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd4072140 msgr2=0x7f0dd414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd4072140 0x7f0dd414d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.444+0000 7f0dca000640 1 -- 172.21.15.28:0/3175912785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dbc0094d0 con 0x7f0dd4071210 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dca000640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000c00 tx=0x7f0dbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc03c830 con 0x7f0dd4071210 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc03c990 con 0x7f0dd4071210 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dbc03cd40 con 0x7f0dd4071210 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd414e370 con 0x7f0dd4071210 2024-09-13T17:21:24.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.445+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd414eb70 con 0x7f0dd4071210 2024-09-13T17:21:24.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.446+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd406f720 con 0x7f0dd4071210 2024-09-13T17:21:24.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.447+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0dbc03a070 con 0x7f0dd4071210 2024-09-13T17:21:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.448+0000 7f0dcbe00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 0x7f0db006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.448+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0dbc0a9ff0 con 0x7f0dd4071210 2024-09-13T17:21:24.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.448+0000 7f0dd2a00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 0x7f0db006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:24.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.449+0000 7f0dd2a00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 0x7f0db006d0c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4006fd0 tx=0x7f0dc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:24.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.452+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0dbc0dd9c0 con 0x7f0dd4071210 2024-09-13T17:21:24.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.629+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0dd4158bf0 con 0x7f0dd4071210 2024-09-13T17:21:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.634+0000 7f0dcbe00640 1 -- 172.21.15.28:0/3175912785 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0dbc0753d0 con 0x7f0dd4071210 2024-09-13T17:21:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 msgr2=0x7f0db006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 0x7f0db006d0c0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4006fd0 tx=0x7f0dc4008040 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 msgr2=0x7f0dd414b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 secure :-1 s=READY pgs=2272 cs=0 l=1 rev1=1 crypto rx=0x7f0dbc000c00 tx=0x7f0dbc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 shutdown_connections 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0db006ac00 0x7f0db006d0c0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd4072140 0x7f0dd414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0dd4071210 0x7f0dd414b1d0 unknown :-1 s=CLOSED pgs=2272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 --2- 172.21.15.28:0/3175912785 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0dd4070e30 0x7f0dd414ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 >> 172.21.15.28:0/3175912785 conn(0x7f0dd4069e80 msgr2=0x7f0dd40b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 shutdown_connections 2024-09-13T17:21:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:24.636+0000 7f0dd3e00640 1 -- 172.21.15.28:0/3175912785 wait complete. 2024-09-13T17:21:24.653 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:21:24.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:21:24.743 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:21:24.847 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:21:24.847 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:24.847 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:24.886 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:21:24.886 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:21:24.886 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:24.925 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:21:24.925 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:24.925 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:24.962 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:21:24.962 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:21:24.962 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:25.002 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:21:25.002 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:21:25.002 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.157+0000 7f8bb2e00640 1 Processor -- start 2024-09-13T17:21:25.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.157+0000 7f8bb2e00640 1 -- start start 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.158+0000 7f8bb2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 0x7f8bac06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.158+0000 7f8bb2e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8bac070cc0 0x7f8bac071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.158+0000 7f8bb2e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac15e100 con 0x7f8bac15a540 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.158+0000 7f8bb2e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0716b0 con 0x7f8bac070cc0 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.158+0000 7f8bb2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0adb00 con 0x7f8bac0cabc0 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8babe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8bac15a540 0x7f8bac15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52496/0 (socket says 172.21.15.28:52496) 2024-09-13T17:21:25.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8babe00640 1 -- 172.21.15.28:0/2354860624 learned_addr learned my addr 172.21.15.28:0/2354860624 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bab400640 1 --2- 172.21.15.28:0/2354860624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 0x7f8bac06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bb0e00640 1 --2- 172.21.15.28:0/2354860624 >> v2:172.21.15.189:3300/0 conn(0x7f8bac070cc0 0x7f8bac071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8baaa00640 1 -- 172.21.15.28:0/2354860624 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534546430 0 0) 0x7f8bac15e100 con 0x7f8bac15a540 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8baaa00640 1 -- 172.21.15.28:0/2354860624 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b90003520 con 0x7f8bac15a540 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bab400640 1 -- 172.21.15.28:0/2354860624 >> v2:172.21.15.189:3300/0 conn(0x7f8bac070cc0 msgr2=0x7f8bac071170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bab400640 1 --2- 172.21.15.28:0/2354860624 >> v2:172.21.15.189:3300/0 conn(0x7f8bac070cc0 0x7f8bac071170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bab400640 1 -- 172.21.15.28:0/2354860624 >> v1:172.21.15.28:6789/0 conn(0x7f8bac15a540 legacy=0x7f8bac15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.159+0000 7f8bab400640 1 -- 172.21.15.28:0/2354860624 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac15e3e0 con 0x7f8bac0cabc0 2024-09-13T17:21:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.160+0000 7f8bab400640 1 --2- 172.21.15.28:0/2354860624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 0x7f8bac06c430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097a0 tx=0x7f8b9802a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3465ecf54de5b90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8baaa00640 1 -- 172.21.15.28:0/2354860624 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9802ae40 con 0x7f8bac0cabc0 2024-09-13T17:21:25.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8baaa00640 1 -- 172.21.15.28:0/2354860624 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b98032440 con 0x7f8bac0cabc0 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8baaa00640 1 -- 172.21.15.28:0/2354860624 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b980327d0 con 0x7f8bac0cabc0 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 -- 172.21.15.28:0/2354860624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 msgr2=0x7f8bac06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/2354860624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 0x7f8bac06c430 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8b980097a0 tx=0x7f8b9802a0f0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 -- 172.21.15.28:0/2354860624 shutdown_connections 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/2354860624 >> v2:172.21.15.189:3300/0 conn(0x7f8bac070cc0 0x7f8bac071170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/2354860624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac0cabc0 0x7f8bac06c430 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 -- 172.21.15.28:0/2354860624 >> 172.21.15.28:0/2354860624 conn(0x7f8bac069e80 msgr2=0x7f8bac06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 -- 172.21.15.28:0/2354860624 shutdown_connections 2024-09-13T17:21:25.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.161+0000 7f8bb2e00640 1 -- 172.21.15.28:0/2354860624 wait complete. 2024-09-13T17:21:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.163+0000 7f8bb2e00640 1 Processor -- start 2024-09-13T17:21:25.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.163+0000 7f8bb2e00640 1 -- start start 2024-09-13T17:21:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac070cc0 0x7f8bac1450d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bac0cabc0 0x7f8bac0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0b2700 con 0x7f8bac15a540 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0b1a80 con 0x7f8bac0cabc0 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb2e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac0b1d60 con 0x7f8bac070cc0 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50312/0 (socket says 172.21.15.28:50312) 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.164+0000 7f8bb0e00640 1 -- 172.21.15.28:0/1461224407 learned_addr learned my addr 172.21.15.28:0/1461224407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8babe00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac070cc0 0x7f8bac1450d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bab400640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bac0cabc0 0x7f8bac0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 -- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac070cc0 msgr2=0x7f8bac1450d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac070cc0 0x7f8bac1450d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 -- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bac0cabc0 msgr2=0x7f8bac0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bac0cabc0 0x7f8bac0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 -- 172.21.15.28:0/1461224407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b98009400 con 0x7f8bac15a540 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb0e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400efc0 tx=0x7f8ba400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4014480 con 0x7f8bac15a540 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba400f040 con 0x7f8bac15a540 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba4014820 con 0x7f8bac15a540 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac0b2040 con 0x7f8bac15a540 2024-09-13T17:21:25.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.165+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac0b1710 con 0x7f8bac15a540 2024-09-13T17:21:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.166+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac0b2840 con 0x7f8bac15a540 2024-09-13T17:21:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.167+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8ba4014980 con 0x7f8bac15a540 2024-09-13T17:21:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.168+0000 7f8ba3e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 0x7f8b8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:25.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.168+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(184..184 src has 1..184) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8ba4089bd0 con 0x7f8bac15a540 2024-09-13T17:21:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.169+0000 7f8babe00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 0x7f8b8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:25.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.169+0000 7f8babe00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 0x7f8b8c06d0c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0011e0 tx=0x7f8b9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:25.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.172+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8bac0b2840 con 0x7f8bac15a540 2024-09-13T17:21:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.387+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bac0b2840 con 0x7f8bac15a540 2024-09-13T17:21:25.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.387+0000 7f8ba3e00640 1 -- 172.21.15.28:0/1461224407 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+3537 (secure 0 0 0) 0x7f8bac0b2840 con 0x7f8bac15a540 2024-09-13T17:21:25.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:25.388 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":158,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8423,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/2115409263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2115409263},{"type":"v1","addr":"172.21.15.189:6836","nonce":2115409263}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8426,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/4225418124","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":4225418124},{"type":"v1","addr":"172.21.15.189:6837","nonce":4225418124}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":146},{"gid":8663,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/2731071087","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2731071087},{"type":"v1","addr":"172.21.15.28:6834","nonce":2731071087}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":158},{"gid":8666,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/741882939","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":741882939},{"type":"v1","addr":"172.21.15.28:6835","nonce":741882939}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":158}],"filesystems":[]} 2024-09-13T17:21:25.389 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 158 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 msgr2=0x7f8b8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 0x7f8b8c06d0c0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c0011e0 tx=0x7f8b9c009040 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 msgr2=0x7f8bac0acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7f8ba400efc0 tx=0x7f8ba400c490 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 shutdown_connections 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8c06ac00 0x7f8b8c06d0c0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8bac15a540 0x7f8bac0acd90 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8bac0cabc0 0x7f8bac0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 --2- 172.21.15.28:0/1461224407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8bac070cc0 0x7f8bac1450d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.389+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 >> 172.21.15.28:0/1461224407 conn(0x7f8bac069e80 msgr2=0x7f8bac06b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.390+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 shutdown_connections 2024-09-13T17:21:25.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:25.390+0000 7f8bb2e00640 1 -- 172.21.15.28:0/1461224407 wait complete. 2024-09-13T17:21:25.406 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:21:25.406 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:21:25.407 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:21:25.407 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:21:25.407 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:21:25.407 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:21:25.407 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:21:25.407 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:21:25.408 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:21:25.408 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:21:25.408 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:21:25.408 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:21:25.408 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:21:25.408 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:21:25.408 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:21:25.409 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:21:25.423 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:21:25.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:21:25.426 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:21:25.426 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:21:25.427 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:21:25.427 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:21:25.468 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:21:25.468 DEBUG:teuthology.parallel:result is None 2024-09-13T17:21:25.470 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:21:25.470 DEBUG:teuthology.parallel:result is None 2024-09-13T17:21:25.472 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:21:25.473 DEBUG:teuthology.parallel:result is None 2024-09-13T17:21:25.594 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:21:25.609 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:21:25.612 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:21:28.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c is failed for ~0s 2024-09-13T17:21:31.423 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:21:31.424 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:21:31.429 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:21:31.429 DEBUG:teuthology.parallel:result is None 2024-09-13T17:21:31.429 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:21:31.567 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:21:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.596+0000 7f0b96c00640 1 Processor -- start 2024-09-13T17:21:31.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.596+0000 7f0b96c00640 1 -- start start 2024-09-13T17:21:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b96c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b9015a780 0x7f0b90070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b96c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b9015ab60 0x7f0b900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b96c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b9006c660 con 0x7f0b9006bb40 2024-09-13T17:21:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b96c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9006c8b0 con 0x7f0b9015ab60 2024-09-13T17:21:31.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b96c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b9006cb90 con 0x7f0b9015a780 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b87400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b9015ab60 0x7f0b900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b94c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0b9006bb40 0x7f0b9006bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52500/0 (socket says 172.21.15.28:52500) 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b94c00640 1 -- 172.21.15.28:0/2179278415 learned_addr learned my addr 172.21.15.28:0/2179278415 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b8fe00640 1 --2- 172.21.15.28:0/2179278415 >> v2:172.21.15.189:3300/0 conn(0x7f0b9015a780 0x7f0b90070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b87400640 1 -- 172.21.15.28:0/2179278415 >> v2:172.21.15.189:3300/0 conn(0x7f0b9015a780 msgr2=0x7f0b90070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b87400640 1 --2- 172.21.15.28:0/2179278415 >> v2:172.21.15.189:3300/0 conn(0x7f0b9015a780 0x7f0b90070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b87400640 1 -- 172.21.15.28:0/2179278415 >> v1:172.21.15.28:6789/0 conn(0x7f0b9006bb40 legacy=0x7f0b9006bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:31.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.598+0000 7f0b87400640 1 -- 172.21.15.28:0/2179278415 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b900cc980 con 0x7f0b9015ab60 2024-09-13T17:21:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.599+0000 7f0b87400640 1 --2- 172.21.15.28:0/2179278415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9015ab60 0x7f0b900712c0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0b900b1330 tx=0x7f0b7802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f388145507e4d0b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b8f400640 1 -- 172.21.15.28:0/2179278415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78032440 con 0x7f0b9015ab60 2024-09-13T17:21:31.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b8f400640 1 -- 172.21.15.28:0/2179278415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b780325a0 con 0x7f0b9015ab60 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b8f400640 1 -- 172.21.15.28:0/2179278415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b78032930 con 0x7f0b9015ab60 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 -- 172.21.15.28:0/2179278415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9015ab60 msgr2=0x7f0b900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 --2- 172.21.15.28:0/2179278415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9015ab60 0x7f0b900712c0 secure :-1 s=READY pgs=2284 cs=0 l=1 rev1=1 crypto rx=0x7f0b900b1330 tx=0x7f0b7802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 -- 172.21.15.28:0/2179278415 shutdown_connections 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 --2- 172.21.15.28:0/2179278415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9015ab60 0x7f0b900712c0 unknown :-1 s=CLOSED pgs=2284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 --2- 172.21.15.28:0/2179278415 >> v2:172.21.15.189:3300/0 conn(0x7f0b9015a780 0x7f0b90070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 -- 172.21.15.28:0/2179278415 >> 172.21.15.28:0/2179278415 conn(0x7f0b900b0270 msgr2=0x7f0b900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.600+0000 7f0b96c00640 1 -- 172.21.15.28:0/2179278415 shutdown_connections 2024-09-13T17:21:31.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.601+0000 7f0b96c00640 1 -- 172.21.15.28:0/2179278415 wait complete. 2024-09-13T17:21:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.602+0000 7f0b96c00640 1 Processor -- start 2024-09-13T17:21:31.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.602+0000 7f0b96c00640 1 -- start start 2024-09-13T17:21:31.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b9015a780 0x7f0b90155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 0x7f0b901f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b900c8d20 con 0x7f0b9006bb40 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b901f2730 con 0x7f0b9015a780 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b96c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b901f2a10 con 0x7f0b9015ab60 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.603+0000 7f0b8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50364/0 (socket says 172.21.15.28:50364) 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 -- 172.21.15.28:0/1255583223 learned_addr learned my addr 172.21.15.28:0/1255583223 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b87400640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b9015a780 0x7f0b90155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b94c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 0x7f0b901f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 -- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 msgr2=0x7f0b901f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 0x7f0b901f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 -- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b9015a780 msgr2=0x7f0b90155050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b9015a780 0x7f0b90155050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 -- 172.21.15.28:0/1255583223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b78009400 con 0x7f0b9006bb40 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b94c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 0x7f0b901f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.604+0000 7f0b8fe00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000e9b0 tx=0x7f0b8000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.605+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b80009670 con 0x7f0b9006bb40 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.605+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8000c9a0 con 0x7f0b9006bb40 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.605+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b8000cd50 con 0x7f0b9006bb40 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.605+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b901f2cf0 con 0x7f0b9006bb40 2024-09-13T17:21:31.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.605+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b901f23b0 con 0x7f0b9006bb40 2024-09-13T17:21:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.606+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b901f2e80 con 0x7f0b9006bb40 2024-09-13T17:21:31.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.607+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0b80013470 con 0x7f0b9006bb40 2024-09-13T17:21:31.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.608+0000 7f0b8cc00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 0x7f0b6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:31.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.608+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0b800892b0 con 0x7f0b9006bb40 2024-09-13T17:21:31.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.608+0000 7f0b87400640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 0x7f0b6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:31.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.609+0000 7f0b87400640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 0x7f0b6c06d0c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0b78034000 tx=0x7f0b7802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:31.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.613+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0b80054830 con 0x7f0b9006bb40 2024-09-13T17:21:31.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:31.803+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0b9006bf50 con 0x7f0b9006bb40 2024-09-13T17:21:32.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.586+0000 7f0b8cc00640 1 -- 172.21.15.28:0/1255583223 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v189) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0b80087e60 con 0x7f0b9006bb40 2024-09-13T17:21:32.587 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:21:32.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 msgr2=0x7f0b6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:32.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 0x7f0b6c06d0c0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0b78034000 tx=0x7f0b7802e000 comp rx=0 tx=0).stop 2024-09-13T17:21:32.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 msgr2=0x7f0b900c7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:32.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 secure :-1 s=READY pgs=2285 cs=0 l=1 rev1=1 crypto rx=0x7f0b8000e9b0 tx=0x7f0b8000ee70 comp rx=0 tx=0).stop 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 shutdown_connections 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b6c06ac00 0x7f0b6c06d0c0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b9015ab60 0x7f0b901f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b9015a780 0x7f0b90155050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 --2- 172.21.15.28:0/1255583223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b9006bb40 0x7f0b900c7060 unknown :-1 s=CLOSED pgs=2285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 >> 172.21.15.28:0/1255583223 conn(0x7f0b900b0270 msgr2=0x7f0b90070620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 shutdown_connections 2024-09-13T17:21:32.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.589+0000 7f0b96c00640 1 -- 172.21.15.28:0/1255583223 wait complete. 2024-09-13T17:21:32.609 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:21:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.768+0000 7f4d3ee00640 1 Processor -- start 2024-09-13T17:21:32.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.768+0000 7f4d3ee00640 1 -- start start 2024-09-13T17:21:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d3815a540 0x7f4d3806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d3815e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4d3815edb0 0x7f4d38070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d380adb00 con 0x7f4d3815a540 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d3815e100 con 0x7f4d3815edb0 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.769+0000 7f4d3ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3815e3e0 con 0x7f4d380cabc0 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d3815a540 0x7f4d3806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4d3815a540 0x7f4d3806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50370/0 (socket says 172.21.15.28:50370) 2024-09-13T17:21:32.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 -- 172.21.15.28:0/3207944416 learned_addr learned my addr 172.21.15.28:0/3207944416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37400640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d3815e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d3ce00640 1 --2- 172.21.15.28:0/3207944416 >> v2:172.21.15.189:3300/0 conn(0x7f4d3815edb0 0x7f4d38070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 -- 172.21.15.28:0/3207944416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 msgr2=0x7f4d3815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d3815e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 -- 172.21.15.28:0/3207944416 >> v2:172.21.15.189:3300/0 conn(0x7f4d3815edb0 msgr2=0x7f4d38070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:32.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 --2- 172.21.15.28:0/3207944416 >> v2:172.21.15.189:3300/0 conn(0x7f4d3815edb0 0x7f4d38070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.770+0000 7f4d37e00640 1 -- 172.21.15.28:0/3207944416 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d3806ce80 con 0x7f4d3815a540 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.771+0000 7f4d37e00640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815a540 0x7f4d3806c430 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f4d380ca980 tx=0x7f4d2802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89018e56e64c023a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.771+0000 7f4d36a00640 1 -- 172.21.15.28:0/3207944416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d2802ae90 con 0x7f4d3815a540 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.771+0000 7f4d36a00640 1 -- 172.21.15.28:0/3207944416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d28032440 con 0x7f4d3815a540 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.771+0000 7f4d36a00640 1 -- 172.21.15.28:0/3207944416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d280327d0 con 0x7f4d3815a540 2024-09-13T17:21:32.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 -- 172.21.15.28:0/3207944416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815a540 msgr2=0x7f4d3806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815a540 0x7f4d3806c430 secure :-1 s=READY pgs=2286 cs=0 l=1 rev1=1 crypto rx=0x7f4d380ca980 tx=0x7f4d2802a140 comp rx=0 tx=0).stop 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 -- 172.21.15.28:0/3207944416 shutdown_connections 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/3207944416 >> v2:172.21.15.189:3300/0 conn(0x7f4d3815edb0 0x7f4d38070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d3815e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/3207944416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815a540 0x7f4d3806c430 unknown :-1 s=CLOSED pgs=2286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 -- 172.21.15.28:0/3207944416 >> 172.21.15.28:0/3207944416 conn(0x7f4d38069e80 msgr2=0x7f4d3806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 -- 172.21.15.28:0/3207944416 shutdown_connections 2024-09-13T17:21:32.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.772+0000 7f4d3ee00640 1 -- 172.21.15.28:0/3207944416 wait complete. 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.774+0000 7f4d3ee00640 1 Processor -- start 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.774+0000 7f4d3ee00640 1 -- start start 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d380acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3815a540 0x7f4d3815baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d381520b0 con 0x7f4d3815c030 2024-09-13T17:21:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3815cb70 con 0x7f4d3815a540 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d3815c5b0 con 0x7f4d380cabc0 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50386/0 (socket says 172.21.15.28:50386) 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ce00640 1 -- 172.21.15.28:0/1913209703 learned_addr learned my addr 172.21.15.28:0/1913209703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d37400640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3815a540 0x7f4d3815baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d37e00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d380acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.775+0000 7f4d3ce00640 1 -- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 msgr2=0x7f4d380acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ce00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d380acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ce00640 1 -- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3815a540 msgr2=0x7f4d3815baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ce00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3815a540 0x7f4d3815baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ce00640 1 -- 172.21.15.28:0/1913209703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d280094d0 con 0x7f4d3815c030 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ce00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f4d30007e40 tx=0x7f4d3000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d30014620 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3000ee90 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d300149e0 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d3815c890 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.776+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d3815d720 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.777+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d38151b30 con 0x7f4d3815c030 2024-09-13T17:21:32.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.778+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4d30014b40 con 0x7f4d3815c030 2024-09-13T17:21:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.779+0000 7f4d2fe00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 0x7f4d1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.779+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4d30089830 con 0x7f4d3815c030 2024-09-13T17:21:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.780+0000 7f4d37e00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 0x7f4d1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:32.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.780+0000 7f4d37e00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 0x7f4d1806d0c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4d28034000 tx=0x7f4d2802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:32.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:32.783+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4d38151b30 con 0x7f4d3815c030 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.024+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4d381521f0 con 0x7f4d3815c030 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.025+0000 7f4d2fe00640 1 -- 172.21.15.28:0/1913209703 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v25) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4d381521f0 con 0x7f4d3815c030 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:21:33.026 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:21:33.027 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.028 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:21:33.029 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.030 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:21:33.031 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:21:33.033 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.035 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:21:33.037 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:21:33.038 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:21:33.039 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.040 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:33.040 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:21:33.040 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:21:33.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 msgr2=0x7f4d1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 0x7f4d1806d0c0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4d28034000 tx=0x7f4d2802e000 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 msgr2=0x7f4d381524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 secure :-1 s=READY pgs=2287 cs=0 l=1 rev1=1 crypto rx=0x7f4d30007e40 tx=0x7f4d3000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 shutdown_connections 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d1806ac00 0x7f4d1806d0c0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d3815c030 0x7f4d381524d0 unknown :-1 s=CLOSED pgs=2287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3815a540 0x7f4d3815baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 --2- 172.21.15.28:0/1913209703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d380cabc0 0x7f4d380acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 >> 172.21.15.28:0/1913209703 conn(0x7f4d38069e80 msgr2=0x7f4d380ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 shutdown_connections 2024-09-13T17:21:33.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.027+0000 7f4d3ee00640 1 -- 172.21.15.28:0/1913209703 wait complete. 2024-09-13T17:21:33.044 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.200+0000 7f784bc00640 1 Processor -- start 2024-09-13T17:21:33.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.200+0000 7f784bc00640 1 -- start start 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f784bc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f78440713d0 0x7f784406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f784bc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784415e100 con 0x7f784415a540 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f784bc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784415e3e0 con 0x7f78440cabc0 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f784bc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78440adb00 con 0x7f78440713d0 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f7849200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f784415a540 0x7f784415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:52504/0 (socket says 172.21.15.28:52504) 2024-09-13T17:21:33.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.201+0000 7f7849200640 1 -- 172.21.15.28:0/320578723 learned_addr learned my addr 172.21.15.28:0/320578723 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7849c00640 1 --2- 172.21.15.28:0/320578723 >> v2:172.21.15.189:3300/0 conn(0x7f78440713d0 0x7f784406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7843400640 1 -- 172.21.15.28:0/320578723 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1390690932 0 0) 0x7f784415e100 con 0x7f784415a540 2024-09-13T17:21:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7843400640 1 -- 172.21.15.28:0/320578723 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f782c003520 con 0x7f784415a540 2024-09-13T17:21:33.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7849c00640 1 -- 172.21.15.28:0/320578723 >> v1:172.21.15.189:6789/0 conn(0x7f78440cabc0 legacy=0x7f7844070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7849c00640 1 -- 172.21.15.28:0/320578723 >> v1:172.21.15.28:6789/0 conn(0x7f784415a540 legacy=0x7f784415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.202+0000 7f7849c00640 1 -- 172.21.15.28:0/320578723 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784415ef80 con 0x7f78440713d0 2024-09-13T17:21:33.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.203+0000 7f7849c00640 1 --2- 172.21.15.28:0/320578723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440713d0 0x7f784406c430 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f783c009d90 tx=0x7f783c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9b73f80d47ade83 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f7843400640 1 -- 172.21.15.28:0/320578723 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c007740 con 0x7f78440713d0 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f7843400640 1 -- 172.21.15.28:0/320578723 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f783c02d050 con 0x7f78440713d0 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f7843400640 1 -- 172.21.15.28:0/320578723 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f783c007aa0 con 0x7f78440713d0 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 -- 172.21.15.28:0/320578723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440713d0 msgr2=0x7f784406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 --2- 172.21.15.28:0/320578723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440713d0 0x7f784406c430 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f783c009d90 tx=0x7f783c02a660 comp rx=0 tx=0).stop 2024-09-13T17:21:33.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 -- 172.21.15.28:0/320578723 shutdown_connections 2024-09-13T17:21:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 --2- 172.21.15.28:0/320578723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440713d0 0x7f784406c430 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 -- 172.21.15.28:0/320578723 >> 172.21.15.28:0/320578723 conn(0x7f7844069e80 msgr2=0x7f784406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 -- 172.21.15.28:0/320578723 shutdown_connections 2024-09-13T17:21:33.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.204+0000 7f784bc00640 1 -- 172.21.15.28:0/320578723 wait complete. 2024-09-13T17:21:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.206+0000 7f784bc00640 1 Processor -- start 2024-09-13T17:21:33.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.206+0000 7f784bc00640 1 -- start start 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440cabc0 0x7f78440b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f784415a540 0x7f7844144030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784415d160 con 0x7f78440713d0 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784414f6c0 con 0x7f78440cabc0 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f784bc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78440b1080 con 0x7f784415a540 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f7849200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f7849200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50392/0 (socket says 172.21.15.28:50392) 2024-09-13T17:21:33.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f7849200640 1 -- 172.21.15.28:0/3656762648 learned_addr learned my addr 172.21.15.28:0/3656762648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f7849c00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f784415a540 0x7f7844144030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.207+0000 7f7843e00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440cabc0 0x7f78440b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 -- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f784415a540 msgr2=0x7f7844144030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f784415a540 0x7f7844144030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 -- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440cabc0 msgr2=0x7f78440b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440cabc0 0x7f78440b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 -- 172.21.15.28:0/3656762648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f783c009a70 con 0x7f78440713d0 2024-09-13T17:21:33.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7849200640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f783800c410 tx=0x7f783800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7838005ce0 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7838005e40 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784415b200 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78380072f0 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.208+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784415af20 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.209+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78440b2430 con 0x7f78440713d0 2024-09-13T17:21:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.210+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7838007450 con 0x7f78440713d0 2024-09-13T17:21:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.211+0000 7f7840c00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 0x7f782406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.211+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7838088690 con 0x7f78440713d0 2024-09-13T17:21:33.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.211+0000 7f7843e00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 0x7f782406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.212+0000 7f7843e00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 0x7f782406d0f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7844159a60 tx=0x7f7830009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.215+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7838057700 con 0x7f78440713d0 2024-09-13T17:21:33.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.395+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f784406c430 con 0x7f78440713d0 2024-09-13T17:21:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.396+0000 7f7840c00640 1 -- 172.21.15.28:0/3656762648 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v189) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f7838057290 con 0x7f78440713d0 2024-09-13T17:21:33.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:33.398 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":189,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:32.582624+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":180,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":177,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.398+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 msgr2=0x7f782406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.398+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 0x7f782406d0f0 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f7844159a60 tx=0x7f7830009040 comp rx=0 tx=0).stop 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 msgr2=0x7f78440b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 secure :-1 s=READY pgs=2289 cs=0 l=1 rev1=1 crypto rx=0x7f783800c410 tx=0x7f783800c8d0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 shutdown_connections 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f782406ac30 0x7f782406d0f0 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f784415a540 0x7f7844144030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78440cabc0 0x7f78440b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 --2- 172.21.15.28:0/3656762648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78440713d0 0x7f78440b0aa0 unknown :-1 s=CLOSED pgs=2289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 >> 172.21.15.28:0/3656762648 conn(0x7f7844069e80 msgr2=0x7f78440cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 shutdown_connections 2024-09-13T17:21:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.399+0000 7f784bc00640 1 -- 172.21.15.28:0/3656762648 wait complete. 2024-09-13T17:21:33.417 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:21:33.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:21:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.575+0000 7f6332a00640 1 Processor -- start 2024-09-13T17:21:33.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.575+0000 7f6332a00640 1 -- start start 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f632c15a780 0x7f632c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f632c06bb40 0x7f632c06bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c06c8b0 con 0x7f632c06bb40 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f632c06cb90 con 0x7f632c15a780 2024-09-13T17:21:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6332a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0cc980 con 0x7f632c15ab60 2024-09-13T17:21:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.576+0000 7f6330a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f632c06bb40 0x7f632c06bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f632be00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f632c15a780 0x7f632c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f632b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f632c06bb40 0x7f632c06bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50404/0 (socket says 172.21.15.28:50404) 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 -- 172.21.15.28:0/4089770299 learned_addr learned my addr 172.21.15.28:0/4089770299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 -- 172.21.15.28:0/4089770299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 msgr2=0x7f632c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 --2- 172.21.15.28:0/4089770299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 -- 172.21.15.28:0/4089770299 >> v2:172.21.15.189:3300/0 conn(0x7f632c15a780 msgr2=0x7f632c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 --2- 172.21.15.28:0/4089770299 >> v2:172.21.15.189:3300/0 conn(0x7f632c15a780 0x7f632c070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.577+0000 7f6330a00640 1 -- 172.21.15.28:0/4089770299 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632c0ccc60 con 0x7f632c06bb40 2024-09-13T17:21:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.578+0000 7f6330a00640 1 --2- 172.21.15.28:0/4089770299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c06bb40 0x7f632c06bfd0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f632c0b1330 tx=0x7f632402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=152f7fea6e3050ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.578+0000 7f632aa00640 1 -- 172.21.15.28:0/4089770299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324031600 con 0x7f632c06bb40 2024-09-13T17:21:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.578+0000 7f632aa00640 1 -- 172.21.15.28:0/4089770299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632402d050 con 0x7f632c06bb40 2024-09-13T17:21:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.578+0000 7f632aa00640 1 -- 172.21.15.28:0/4089770299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6324031980 con 0x7f632c06bb40 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 -- 172.21.15.28:0/4089770299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c06bb40 msgr2=0x7f632c06bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 --2- 172.21.15.28:0/4089770299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c06bb40 0x7f632c06bfd0 secure :-1 s=READY pgs=2290 cs=0 l=1 rev1=1 crypto rx=0x7f632c0b1330 tx=0x7f632402a6a0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 -- 172.21.15.28:0/4089770299 shutdown_connections 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 --2- 172.21.15.28:0/4089770299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c06bb40 0x7f632c06bfd0 unknown :-1 s=CLOSED pgs=2290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 --2- 172.21.15.28:0/4089770299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 --2- 172.21.15.28:0/4089770299 >> v2:172.21.15.189:3300/0 conn(0x7f632c15a780 0x7f632c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 -- 172.21.15.28:0/4089770299 >> 172.21.15.28:0/4089770299 conn(0x7f632c0b0270 msgr2=0x7f632c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 -- 172.21.15.28:0/4089770299 shutdown_connections 2024-09-13T17:21:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.579+0000 7f6332a00640 1 -- 172.21.15.28:0/4089770299 wait complete. 2024-09-13T17:21:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.581+0000 7f6332a00640 1 Processor -- start 2024-09-13T17:21:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.581+0000 7f6332a00640 1 -- start start 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632c06bb40 0x7f632c153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c1efe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c1547a0 con 0x7f632c15a780 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632c153890 con 0x7f632c06bb40 2024-09-13T17:21:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6332a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632c0ab4f0 con 0x7f632c15ab60 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50414/0 (socket says 172.21.15.28:50414) 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 -- 172.21.15.28:0/2604374575 learned_addr learned my addr 172.21.15.28:0/2604374575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f6330a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c1efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632be00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632c06bb40 0x7f632c153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 -- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 msgr2=0x7f632c1efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c1efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 -- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632c06bb40 msgr2=0x7f632c153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632c06bb40 0x7f632c153350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.582+0000 7f632b400640 1 -- 172.21.15.28:0/2604374575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6324009b30 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f632b400640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f631800e870 tx=0x7f631800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63180094f0 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631800c940 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631800ccf0 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632c0ab7d0 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.583+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632c0abd90 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.584+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f632c06bfd0 con 0x7f632c15a780 2024-09-13T17:21:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.585+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6318013470 con 0x7f632c15a780 2024-09-13T17:21:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.586+0000 7f6323e00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 0x7f630c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.586+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6318089150 con 0x7f632c15a780 2024-09-13T17:21:33.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.587+0000 7f632be00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 0x7f630c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.587+0000 7f632be00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 0x7f630c06d0c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f631c009b40 tx=0x7f631c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.592+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6318054600 con 0x7f632c15a780 2024-09-13T17:21:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.771+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f632c073bb0 con 0x7f632c15a780 2024-09-13T17:21:33.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.772+0000 7f6323e00640 1 -- 172.21.15.28:0/2604374575 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v189) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6318057c20 con 0x7f632c15a780 2024-09-13T17:21:33.773 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 msgr2=0x7f630c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 0x7f630c06d0c0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f631c009b40 tx=0x7f631c009190 comp rx=0 tx=0).stop 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 msgr2=0x7f632c0ac590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 secure :-1 s=READY pgs=2291 cs=0 l=1 rev1=1 crypto rx=0x7f631800e870 tx=0x7f631800ed30 comp rx=0 tx=0).stop 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 shutdown_connections 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f630c06ac00 0x7f630c06d0c0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f632c15ab60 0x7f632c1efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f632c15a780 0x7f632c0ac590 unknown :-1 s=CLOSED pgs=2291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 --2- 172.21.15.28:0/2604374575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f632c06bb40 0x7f632c153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 >> 172.21.15.28:0/2604374575 conn(0x7f632c0b0270 msgr2=0x7f632c0ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 shutdown_connections 2024-09-13T17:21:33.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.774+0000 7f6332a00640 1 -- 172.21.15.28:0/2604374575 wait complete. 2024-09-13T17:21:33.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:21:33.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.945+0000 7fca81c00640 1 Processor -- start 2024-09-13T17:21:33.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.945+0000 7fca81c00640 1 -- start start 2024-09-13T17:21:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.946+0000 7fca81c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c0b2400 0x7fca7c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.946+0000 7fca81c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 0x7fca7c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.946+0000 7fca81c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7c070470 con 0x7fca7c15c9a0 2024-09-13T17:21:33.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.946+0000 7fca81c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0ace30 con 0x7fca7c0b2400 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.946+0000 7fca81c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7c0ad110 con 0x7fca7c0ad940 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c0b2400 0x7fca7c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca7c0b2400 0x7fca7c0ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39774/0 (socket says 172.21.15.28:39774) 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 -- 172.21.15.28:0/1488364073 learned_addr learned my addr 172.21.15.28:0/1488364073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7b400640 1 --2- 172.21.15.28:0/1488364073 >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 0x7fca7c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 -- 172.21.15.28:0/1488364073 >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 msgr2=0x7fca7c06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 --2- 172.21.15.28:0/1488364073 >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 0x7fca7c06bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 -- 172.21.15.28:0/1488364073 >> v1:172.21.15.189:6789/0 conn(0x7fca7c15c9a0 legacy=0x7fca7c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7a000640 1 -- 172.21.15.28:0/1488364073 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7c06b170 con 0x7fca7c0b2400 2024-09-13T17:21:33.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.947+0000 7fca7b400640 1 --2- 172.21.15.28:0/1488364073 >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 0x7fca7c06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:33.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.948+0000 7fca7a000640 1 --2- 172.21.15.28:0/1488364073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0b2400 0x7fca7c0ac850 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fca7c160550 tx=0x7fca6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3dad26b7707afa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.948+0000 7fca79600640 1 -- 172.21.15.28:0/1488364073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68032440 con 0x7fca7c0b2400 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca79600640 1 -- 172.21.15.28:0/1488364073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680325a0 con 0x7fca7c0b2400 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca79600640 1 -- 172.21.15.28:0/1488364073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68032930 con 0x7fca7c0b2400 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 -- 172.21.15.28:0/1488364073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0b2400 msgr2=0x7fca7c0ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 --2- 172.21.15.28:0/1488364073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0b2400 0x7fca7c0ac850 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fca7c160550 tx=0x7fca6802a270 comp rx=0 tx=0).stop 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 -- 172.21.15.28:0/1488364073 shutdown_connections 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 --2- 172.21.15.28:0/1488364073 >> v2:172.21.15.189:3300/0 conn(0x7fca7c0ad940 0x7fca7c06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 --2- 172.21.15.28:0/1488364073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0b2400 0x7fca7c0ac850 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 -- 172.21.15.28:0/1488364073 >> 172.21.15.28:0/1488364073 conn(0x7fca7c0b0270 msgr2=0x7fca7c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 -- 172.21.15.28:0/1488364073 shutdown_connections 2024-09-13T17:21:33.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.949+0000 7fca81c00640 1 -- 172.21.15.28:0/1488364073 wait complete. 2024-09-13T17:21:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.951+0000 7fca81c00640 1 Processor -- start 2024-09-13T17:21:33.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.951+0000 7fca81c00640 1 -- start start 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c0b2400 0x7fca7c14f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca7c15c9a0 0x7fca7c15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cced0 con 0x7fca7c0ad940 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cba30 con 0x7fca7c0b2400 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca81c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7c0cd150 con 0x7fca7c15c9a0 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39790/0 (socket says 172.21.15.28:39790) 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 -- 172.21.15.28:0/3459635702 learned_addr learned my addr 172.21.15.28:0/3459635702 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:33.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7a000640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c0b2400 0x7fca7c14f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7b400640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca7c15c9a0 0x7fca7c15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 -- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca7c15c9a0 msgr2=0x7fca7c15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca7c15c9a0 0x7fca7c15b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 -- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c0b2400 msgr2=0x7fca7c14f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c0b2400 0x7fca7c14f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.952+0000 7fca7aa00640 1 -- 172.21.15.28:0/3459635702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca680094d0 con 0x7fca7c0ad940 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca7aa00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c400 tx=0x7fca6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c0052d0 con 0x7fca7c0ad940 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6c005430 con 0x7fca7c0ad940 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6c003830 con 0x7fca7c0ad940 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca7c070930 con 0x7fca7c0ad940 2024-09-13T17:21:33.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.953+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7c0cd710 con 0x7fca7c0ad940 2024-09-13T17:21:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.954+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca7c0acb70 con 0x7fca7c0ad940 2024-09-13T17:21:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.955+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca6c003990 con 0x7fca7c0ad940 2024-09-13T17:21:33.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.956+0000 7fca72a00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 0x7fca5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.956+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fca6c088620 con 0x7fca7c0ad940 2024-09-13T17:21:33.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.956+0000 7fca7a000640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 0x7fca5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:33.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.957+0000 7fca7a000640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 0x7fca5c06d0c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fca68034000 tx=0x7fca6802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:33.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:33.960+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fca6c053ab0 con 0x7fca7c0ad940 2024-09-13T17:21:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.140+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fca7c06c7b0 con 0x7fca7c0ad940 2024-09-13T17:21:34.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.140+0000 7fca72a00640 1 -- 172.21.15.28:0/3459635702 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v189) v1 ==== 84+0+65 (secure 0 0 0) 0x7fca6c017060 con 0x7fca7c0ad940 2024-09-13T17:21:34.141 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:34.142 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 msgr2=0x7fca5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 0x7fca5c06d0c0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fca68034000 tx=0x7fca6802e000 comp rx=0 tx=0).stop 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 msgr2=0x7fca7c0cdf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 secure :-1 s=READY pgs=2293 cs=0 l=1 rev1=1 crypto rx=0x7fca6c00c400 tx=0x7fca6c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 shutdown_connections 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5c06ac00 0x7fca5c06d0c0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca7c15c9a0 0x7fca7c15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca7c0b2400 0x7fca7c14f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 --2- 172.21.15.28:0/3459635702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca7c0ad940 0x7fca7c0cdf10 unknown :-1 s=CLOSED pgs=2293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.142+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 >> 172.21.15.28:0/3459635702 conn(0x7fca7c0b0270 msgr2=0x7fca7c06bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.143+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 shutdown_connections 2024-09-13T17:21:34.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.143+0000 7fca81c00640 1 -- 172.21.15.28:0/3459635702 wait complete. 2024-09-13T17:21:34.159 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:21:34.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:21:34.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.317+0000 7f095da00640 1 Processor -- start 2024-09-13T17:21:34.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.318+0000 7f095da00640 1 -- start start 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f095da00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f095815ebd0 0x7f09580cd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f095da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09580cd6d0 0x7f09580b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f095da00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0958144280 con 0x7f0958143ea0 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f095da00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09580b3060 con 0x7f095815ebd0 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f095da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09580cdb20 con 0x7f09580cd6d0 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f0957e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0958143ea0 0x7f09580b1b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58402/0 (socket says 172.21.15.28:58402) 2024-09-13T17:21:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.319+0000 7f0957e00640 1 -- 172.21.15.28:0/2379177024 learned_addr learned my addr 172.21.15.28:0/2379177024 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f095d000640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09580cd6d0 0x7f09580b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f094f400640 1 --2- 172.21.15.28:0/2379177024 >> v2:172.21.15.189:3300/0 conn(0x7f095815ebd0 0x7f09580cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503088295 0 0) 0x7f0948001110 con 0x7f0958143ea0 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003520 con 0x7f0958143ea0 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2724473446 0 0) 0x7f093c003520 con 0x7f0958143ea0 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09580cd6d0 msgr2=0x7f09580b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09580cd6d0 0x7f09580b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 >> v2:172.21.15.189:3300/0 conn(0x7f095815ebd0 msgr2=0x7f09580cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 --2- 172.21.15.28:0/2379177024 >> v2:172.21.15.189:3300/0 conn(0x7f095815ebd0 0x7f09580cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09580b23e0 con 0x7f0958143ea0 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f0948001820 con 0x7f0958143ea0 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 >> v1:172.21.15.28:6789/0 conn(0x7f0958143ea0 legacy=0x7f09580b1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f093c003970 0x7f093c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f093c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093c005180 0x7f093c00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580b3060 con 0x7f093c0047c0 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09580cdb20 con 0x7f093c003970 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093c010300 con 0x7f093c005180 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f095d000640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09580cd6d0 0x7f09580b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.320+0000 7f094f400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f093c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f0957e00640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f093c003970 0x7f093c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f095d000640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093c005180 0x7f093c00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f094f400640 1 -- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093c005180 msgr2=0x7f093c00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f094f400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093c005180 0x7f093c00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f094f400640 1 -- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f093c003970 msgr2=0x7f093c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f094f400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f093c003970 0x7f093c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.321+0000 7f094f400640 1 -- 172.21.15.28:0/2379177024 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093c003520 con 0x7f093c0047c0 2024-09-13T17:21:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.322+0000 7f094f400640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f093c004c40 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0940009a90 tx=0x7f093c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3510abecdeccb33d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.322+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940031400 con 0x7f093c0047c0 2024-09-13T17:21:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.322+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094002d050 con 0x7f093c0047c0 2024-09-13T17:21:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.322+0000 7f0957400640 1 -- 172.21.15.28:0/2379177024 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0940031780 con 0x7f093c0047c0 2024-09-13T17:21:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 -- 172.21.15.28:0/2379177024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 msgr2=0x7f093c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 --2- 172.21.15.28:0/2379177024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f093c004c40 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7f0940009a90 tx=0x7f093c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095d000640 1 -- 172.21.15.28:0/2379177024 reap_dead start 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 -- 172.21.15.28:0/2379177024 shutdown_connections 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 -- 172.21.15.28:0/2379177024 >> 172.21.15.28:0/2379177024 conn(0x7f0958069e80 msgr2=0x7f095806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 -- 172.21.15.28:0/2379177024 shutdown_connections 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.323+0000 7f095da00640 1 -- 172.21.15.28:0/2379177024 wait complete. 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.324+0000 7f095da00640 1 Processor -- start 2024-09-13T17:21:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.325+0000 7f095da00640 1 -- start start 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.325+0000 7f095da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.325+0000 7f095da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0958159e30 0x7f09581edec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.325+0000 7f095da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0958139d90 0x7f095813bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f095da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09581370c0 con 0x7f093c0047c0 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f095da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095815ae60 con 0x7f0958159e30 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f095da00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09581ee400 con 0x7f0958139d90 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39816/0 (socket says 172.21.15.28:39816) 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 -- 172.21.15.28:0/574086067 learned_addr learned my addr 172.21.15.28:0/574086067 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f094f400640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0958159e30 0x7f09581edec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f095d000640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0958139d90 0x7f095813bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 -- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0958139d90 msgr2=0x7f095813bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0958139d90 0x7f095813bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 -- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0958159e30 msgr2=0x7f09581edec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0958159e30 0x7f09581edec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.326+0000 7f0957e00640 1 -- 172.21.15.28:0/574086067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095813a290 con 0x7f093c0047c0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f0957e00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f0948007310 tx=0x7f09480077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0948001110 con 0x7f093c0047c0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094800f9c0 con 0x7f093c0047c0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09581372d0 con 0x7f093c0047c0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094800fd90 con 0x7f093c0047c0 2024-09-13T17:21:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.327+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0958137890 con 0x7f093c0047c0 2024-09-13T17:21:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.328+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0958067a10 con 0x7f093c0047c0 2024-09-13T17:21:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.329+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f094801d460 con 0x7f093c0047c0 2024-09-13T17:21:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.330+0000 7f0954c00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 0x7f093406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.330+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0948088f70 con 0x7f093c0047c0 2024-09-13T17:21:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.331+0000 7f094f400640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 0x7f093406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:34.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.331+0000 7f094f400640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 0x7f093406d0c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0940033000 tx=0x7f0940008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.334+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f09480544b0 con 0x7f093c0047c0 2024-09-13T17:21:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:34.517+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0958144280 con 0x7f093c0047c0 2024-09-13T17:21:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.287+0000 7f0954c00640 1 -- 172.21.15.28:0/574086067 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v190) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0958144280 con 0x7f093c0047c0 2024-09-13T17:21:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:21:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.292+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 msgr2=0x7f093406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.292+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 0x7f093406d0c0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0940033000 tx=0x7f0940008030 comp rx=0 tx=0).stop 2024-09-13T17:21:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.292+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 msgr2=0x7f09581f0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:35.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.292+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 secure :-1 s=READY pgs=2296 cs=0 l=1 rev1=1 crypto rx=0x7f0948007310 tx=0x7f09480077d0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 shutdown_connections 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f093406ac00 0x7f093406d0c0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0958139d90 0x7f095813bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0958159e30 0x7f09581edec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 --2- 172.21.15.28:0/574086067 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093c0047c0 0x7f09581f0d80 unknown :-1 s=CLOSED pgs=2296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.294+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 >> 172.21.15.28:0/574086067 conn(0x7f0958069e80 msgr2=0x7f09580cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.295+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 shutdown_connections 2024-09-13T17:21:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.295+0000 7f095da00640 1 -- 172.21.15.28:0/574086067 wait complete. 2024-09-13T17:21:35.314 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:21:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.479+0000 7fc2e8a00640 1 Processor -- start 2024-09-13T17:21:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.479+0000 7fc2e8a00640 1 -- start start 2024-09-13T17:21:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.480+0000 7fc2e8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e4160c40 0x7fc2e406b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.480+0000 7fc2e8a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2e40b2a80 0x7fc2e40b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.480+0000 7fc2e8a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2e406c300 con 0x7fc2e406b8a0 2024-09-13T17:21:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.480+0000 7fc2e8a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e406bd40 con 0x7fc2e40b2a80 2024-09-13T17:21:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.480+0000 7fc2e8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e406c020 con 0x7fc2e4160c40 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2e40b2a80 0x7fc2e40b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2e40b2a80 0x7fc2e40b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39830/0 (socket says 172.21.15.28:39830) 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 -- 172.21.15.28:0/3596644401 learned_addr learned my addr 172.21.15.28:0/3596644401 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 -- 172.21.15.28:0/3596644401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e4160c40 msgr2=0x7fc2e406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 --2- 172.21.15.28:0/3596644401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e4160c40 0x7fc2e406b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 -- 172.21.15.28:0/3596644401 >> v1:172.21.15.189:6789/0 conn(0x7fc2e406b8a0 legacy=0x7fc2e4161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:35.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.481+0000 7fc2e3e00640 1 -- 172.21.15.28:0/3596644401 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e40b2280 con 0x7fc2e40b2a80 2024-09-13T17:21:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.482+0000 7fc2e3e00640 1 --2- 172.21.15.28:0/3596644401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e40b2a80 0x7fc2e40b2f10 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009e50 tx=0x7fc2dc02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=128e323da0d052f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.482+0000 7fc2e2000640 1 -- 172.21.15.28:0/3596644401 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc031600 con 0x7fc2e40b2a80 2024-09-13T17:21:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.482+0000 7fc2e2000640 1 -- 172.21.15.28:0/3596644401 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2dc02d050 con 0x7fc2e40b2a80 2024-09-13T17:21:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.482+0000 7fc2e2000640 1 -- 172.21.15.28:0/3596644401 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2dc031980 con 0x7fc2e40b2a80 2024-09-13T17:21:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 -- 172.21.15.28:0/3596644401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e40b2a80 msgr2=0x7fc2e40b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/3596644401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e40b2a80 0x7fc2e40b2f10 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc009e50 tx=0x7fc2dc02a640 comp rx=0 tx=0).stop 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 -- 172.21.15.28:0/3596644401 shutdown_connections 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/3596644401 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e40b2a80 0x7fc2e40b2f10 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/3596644401 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e4160c40 0x7fc2e406b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 -- 172.21.15.28:0/3596644401 >> 172.21.15.28:0/3596644401 conn(0x7fc2e4072e40 msgr2=0x7fc2e4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 -- 172.21.15.28:0/3596644401 shutdown_connections 2024-09-13T17:21:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.483+0000 7fc2e8a00640 1 -- 172.21.15.28:0/3596644401 wait complete. 2024-09-13T17:21:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.485+0000 7fc2e8a00640 1 Processor -- start 2024-09-13T17:21:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.485+0000 7fc2e8a00640 1 -- start start 2024-09-13T17:21:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e40b2a80 0x7fc2e40a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e4160c40 0x7fc2e406fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e414e870 con 0x7fc2e406b8a0 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e40aa3b0 con 0x7fc2e4160c40 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e8a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e414dbf0 con 0x7fc2e40b2a80 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39836/0 (socket says 172.21.15.28:39836) 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 -- 172.21.15.28:0/456508345 learned_addr learned my addr 172.21.15.28:0/456508345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 -- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e40b2a80 msgr2=0x7fc2e40a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e40b2a80 0x7fc2e40a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 -- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e4160c40 msgr2=0x7fc2e406fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e4160c40 0x7fc2e406fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.486+0000 7fc2e3400640 1 -- 172.21.15.28:0/456508345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc009b30 con 0x7fc2e406b8a0 2024-09-13T17:21:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2e3400640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400b410 tx=0x7fc2d400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011830 con 0x7fc2e406b8a0 2024-09-13T17:21:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d40119b0 con 0x7fc2e406b8a0 2024-09-13T17:21:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4011d60 con 0x7fc2e406b8a0 2024-09-13T17:21:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e414ded0 con 0x7fc2e406b8a0 2024-09-13T17:21:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.487+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e414e490 con 0x7fc2e406b8a0 2024-09-13T17:21:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.488+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e414b7f0 con 0x7fc2e406b8a0 2024-09-13T17:21:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.489+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc2d401a400 con 0x7fc2e406b8a0 2024-09-13T17:21:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.490+0000 7fc2db400640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 0x7fc2c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.491+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc2d4089220 con 0x7fc2e406b8a0 2024-09-13T17:21:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.491+0000 7fc2e2a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 0x7fc2c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.492+0000 7fc2e2a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 0x7fc2c406d0c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aabe0 tx=0x7fc2d0007990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.495+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc2e414b7f0 con 0x7fc2e406b8a0 2024-09-13T17:21:35.519 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.c has been restored 2024-09-13T17:21:35.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:35.677+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc2e414b7f0 con 0x7fc2e406b8a0 2024-09-13T17:21:36.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.290+0000 7fc2db400640 1 -- 172.21.15.28:0/456508345 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v191) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc2d4054670 con 0x7fc2e406b8a0 2024-09-13T17:21:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:21:36.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.296+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 msgr2=0x7fc2c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.296+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 0x7fc2c406d0c0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fc2e40aabe0 tx=0x7fc2d0007990 comp rx=0 tx=0).stop 2024-09-13T17:21:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.296+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 msgr2=0x7fc2e414b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.296+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400b410 tx=0x7fc2d400b8d0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 shutdown_connections 2024-09-13T17:21:36.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc2c406ac00 0x7fc2c406d0c0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2e4160c40 0x7fc2e406fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2e40b2a80 0x7fc2e40a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 --2- 172.21.15.28:0/456508345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2e406b8a0 0x7fc2e414b150 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.298+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 >> 172.21.15.28:0/456508345 conn(0x7fc2e4072e40 msgr2=0x7fc2e406f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.299+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 shutdown_connections 2024-09-13T17:21:36.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.299+0000 7fc2e8a00640 1 -- 172.21.15.28:0/456508345 wait complete. 2024-09-13T17:21:36.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:21:36.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.489+0000 7fd55ce00640 1 Processor -- start 2024-09-13T17:21:36.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.489+0000 7fd55ce00640 1 -- start start 2024-09-13T17:21:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd55ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd55815a780 0x7fd558070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd55ce00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55806c660 con 0x7fd55806c250 2024-09-13T17:21:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd55ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd55806c8b0 con 0x7fd55815ab60 2024-09-13T17:21:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd55ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd55806cb90 con 0x7fd55815a780 2024-09-13T17:21:36.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd556800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd55806c250 0x7fd5580cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58404/0 (socket says 172.21.15.28:58404) 2024-09-13T17:21:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.490+0000 7fd556800640 1 -- 172.21.15.28:0/3225708016 learned_addr learned my addr 172.21.15.28:0/3225708016 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555e00640 1 --2- 172.21.15.28:0/3225708016 >> v2:172.21.15.189:3300/0 conn(0x7fd55815a780 0x7fd558070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555400640 1 -- 172.21.15.28:0/3225708016 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077042657 0 0) 0x7fd55806c660 con 0x7fd55806c250 2024-09-13T17:21:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555400640 1 -- 172.21.15.28:0/3225708016 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd53c003520 con 0x7fd55806c250 2024-09-13T17:21:36.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555e00640 1 -- 172.21.15.28:0/3225708016 >> v1:172.21.15.189:6789/0 conn(0x7fd55815ab60 legacy=0x7fd55806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555e00640 1 -- 172.21.15.28:0/3225708016 >> v1:172.21.15.28:6789/0 conn(0x7fd55806c250 legacy=0x7fd5580cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.491+0000 7fd555e00640 1 -- 172.21.15.28:0/3225708016 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5580cc980 con 0x7fd55815a780 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.492+0000 7fd555e00640 1 --2- 172.21.15.28:0/3225708016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558070d80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5480097a0 tx=0x7fd54802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3000b672bd920922 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd555400640 1 -- 172.21.15.28:0/3225708016 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548032440 con 0x7fd55815a780 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd555400640 1 -- 172.21.15.28:0/3225708016 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5480325a0 con 0x7fd55815a780 2024-09-13T17:21:36.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd555400640 1 -- 172.21.15.28:0/3225708016 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548032910 con 0x7fd55815a780 2024-09-13T17:21:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 -- 172.21.15.28:0/3225708016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 msgr2=0x7fd558070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 --2- 172.21.15.28:0/3225708016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558070d80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd5480097a0 tx=0x7fd54802a230 comp rx=0 tx=0).stop 2024-09-13T17:21:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 -- 172.21.15.28:0/3225708016 shutdown_connections 2024-09-13T17:21:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 --2- 172.21.15.28:0/3225708016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558070d80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 -- 172.21.15.28:0/3225708016 >> 172.21.15.28:0/3225708016 conn(0x7fd5580b0270 msgr2=0x7fd5580b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 -- 172.21.15.28:0/3225708016 shutdown_connections 2024-09-13T17:21:36.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.493+0000 7fd55ce00640 1 -- 172.21.15.28:0/3225708016 wait complete. 2024-09-13T17:21:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.495+0000 7fd55ce00640 1 Processor -- start 2024-09-13T17:21:36.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.495+0000 7fd55ce00640 1 -- start start 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd55815ab60 0x7fd5581f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55815dfe0 con 0x7fd55806c250 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd558154640 con 0x7fd55815a780 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd55ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd558154920 con 0x7fd55815ab60 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd555e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd555e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39848/0 (socket says 172.21.15.28:39848) 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.496+0000 7fd555e00640 1 -- 172.21.15.28:0/961093734 learned_addr learned my addr 172.21.15.28:0/961093734 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd556800640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd55815ab60 0x7fd5581f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd54fe00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 -- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd55815ab60 msgr2=0x7fd5581f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd55815ab60 0x7fd5581f5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 -- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 msgr2=0x7fd558145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 -- 172.21.15.28:0/961093734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd548009400 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.497+0000 7fd555e00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fd548000c00 tx=0x7fd5480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.498+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548032440 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.498+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd548004230 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.498+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd548006e10 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.498+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd558154c00 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.498+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd558146430 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.499+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd558155120 con 0x7fd55806c250 2024-09-13T17:21:36.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.500+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd548039050 con 0x7fd55806c250 2024-09-13T17:21:36.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.501+0000 7fd54e000640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 0x7fd53406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.501+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fd5480a9c60 con 0x7fd55806c250 2024-09-13T17:21:36.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.501+0000 7fd54fe00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 0x7fd53406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.502+0000 7fd54fe00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 0x7fd53406d0c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd5581e2fa0 tx=0x7fd540009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.506+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd548074eb0 con 0x7fd55806c250 2024-09-13T17:21:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.719+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd558071680 con 0x7fd55806c250 2024-09-13T17:21:36.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.728+0000 7fd54e000640 1 -- 172.21.15.28:0/961093734 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v161) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd548037020 con 0x7fd55806c250 2024-09-13T17:21:36.732 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 26 and data pool 27 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.733+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 msgr2=0x7fd53406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.733+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 0x7fd53406d0c0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd5581e2fa0 tx=0x7fd540009040 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.733+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 msgr2=0x7fd55815daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.733+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7fd548000c00 tx=0x7fd5480046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 shutdown_connections 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd53406ac00 0x7fd53406d0c0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd55815ab60 0x7fd5581f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd55815a780 0x7fd558145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 --2- 172.21.15.28:0/961093734 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd55806c250 0x7fd55815daa0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.734+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 >> 172.21.15.28:0/961093734 conn(0x7fd5580b0270 msgr2=0x7fd5580ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.735+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 shutdown_connections 2024-09-13T17:21:36.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.735+0000 7fd55ce00640 1 -- 172.21.15.28:0/961093734 wait complete. 2024-09-13T17:21:36.767 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.922+0000 7f9a04a00640 1 Processor -- start 2024-09-13T17:21:36.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.922+0000 7f9a04a00640 1 -- start start 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9a0015a540 0x7f9a0006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a000cabc0 0x7f9a0015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a0015edb0 0x7f9a00070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a000adb00 con 0x7f9a0015edb0 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a0015e100 con 0x7f9a0015a540 2024-09-13T17:21:36.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f9a04a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0015e3e0 con 0x7f9a000cabc0 2024-09-13T17:21:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f99fe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a0015edb0 0x7f9a00070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f99fe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9a0015edb0 0x7f9a00070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39864/0 (socket says 172.21.15.28:39864) 2024-09-13T17:21:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.923+0000 7f99fe800640 1 -- 172.21.15.28:0/41652335 learned_addr learned my addr 172.21.15.28:0/41652335 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fde00640 1 --2- 172.21.15.28:0/41652335 >> v2:172.21.15.189:3300/0 conn(0x7f9a0015a540 0x7f9a0006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fd400640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a000cabc0 0x7f9a0015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fe800640 1 -- 172.21.15.28:0/41652335 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a000cabc0 msgr2=0x7f9a0015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fe800640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a000cabc0 0x7f9a0015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fe800640 1 -- 172.21.15.28:0/41652335 >> v2:172.21.15.189:3300/0 conn(0x7f9a0015a540 msgr2=0x7f9a0006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fe800640 1 --2- 172.21.15.28:0/41652335 >> v2:172.21.15.189:3300/0 conn(0x7f9a0015a540 0x7f9a0006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.924+0000 7f99fe800640 1 -- 172.21.15.28:0/41652335 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0006ce80 con 0x7f9a0015edb0 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.925+0000 7f99fe800640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0015edb0 0x7f9a00070cc0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f9a000b3a00 tx=0x7f99f802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40acde01a79439df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.925+0000 7f99fca00640 1 -- 172.21.15.28:0/41652335 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8031600 con 0x7f9a0015edb0 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.925+0000 7f99fca00640 1 -- 172.21.15.28:0/41652335 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f802d050 con 0x7f9a0015edb0 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.925+0000 7f99fca00640 1 -- 172.21.15.28:0/41652335 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8031980 con 0x7f9a0015edb0 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 -- 172.21.15.28:0/41652335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0015edb0 msgr2=0x7f9a00070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0015edb0 0x7f9a00070cc0 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f9a000b3a00 tx=0x7f99f802a610 comp rx=0 tx=0).stop 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 -- 172.21.15.28:0/41652335 shutdown_connections 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0015edb0 0x7f9a00070cc0 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 --2- 172.21.15.28:0/41652335 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a000cabc0 0x7f9a0015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 --2- 172.21.15.28:0/41652335 >> v2:172.21.15.189:3300/0 conn(0x7f9a0015a540 0x7f9a0006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 -- 172.21.15.28:0/41652335 >> 172.21.15.28:0/41652335 conn(0x7f9a00069e80 msgr2=0x7f9a0006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 -- 172.21.15.28:0/41652335 shutdown_connections 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.926+0000 7f9a04a00640 1 -- 172.21.15.28:0/41652335 wait complete. 2024-09-13T17:21:36.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.928+0000 7f9a04a00640 1 Processor -- start 2024-09-13T17:21:36.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.928+0000 7f9a04a00640 1 -- start start 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a000cabc0 0x7f9a000b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 0x7f9a0014d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00153d80 con 0x7f9a0014dec0 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00154060 con 0x7f9a000cabc0 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f9a04a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00154340 con 0x7f9a0015a540 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f99fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f99fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39874/0 (socket says 172.21.15.28:39874) 2024-09-13T17:21:36.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f99fe800640 1 -- 172.21.15.28:0/397244263 learned_addr learned my addr 172.21.15.28:0/397244263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f99fde00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a000cabc0 0x7f9a000b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.929+0000 7f99fd400640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 0x7f9a0014d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 -- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 msgr2=0x7f9a0014d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 0x7f9a0014d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 -- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a000cabc0 msgr2=0x7f9a000b0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a000cabc0 0x7f9a000b0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 -- 172.21.15.28:0/397244263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8009b30 con 0x7f9a0014dec0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fd400640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 0x7f9a0014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99fe800640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f99f8034040 tx=0x7f99f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f803d860 con 0x7f9a0014dec0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f803d9c0 con 0x7f9a0014dec0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f803dd70 con 0x7f9a0014dec0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a00154620 con 0x7f9a0014dec0 2024-09-13T17:21:36.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.930+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a00154be0 con 0x7f9a0014dec0 2024-09-13T17:21:36.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.931+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a00154d70 con 0x7f9a0014dec0 2024-09-13T17:21:36.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.932+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f99f803b400 con 0x7f9a0014dec0 2024-09-13T17:21:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.934+0000 7f99f6000640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 0x7f99e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:36.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.934+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99f80aa180 con 0x7f9a0014dec0 2024-09-13T17:21:36.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.934+0000 7f99fde00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 0x7f99e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:36.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.935+0000 7f99fde00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 0x7f99e006d0c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009ac0 tx=0x7f99f0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:36.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:36.938+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f99f80752c0 con 0x7f9a0014dec0 2024-09-13T17:21:37.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.119+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a0015e870 con 0x7f9a0014dec0 2024-09-13T17:21:37.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.120+0000 7f99f6000640 1 -- 172.21.15.28:0/397244263 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f99f8009510 con 0x7f9a0014dec0 2024-09-13T17:21:37.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:37.122 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:36.720812+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:37.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 msgr2=0x7f99e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 0x7f99e006d0c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009ac0 tx=0x7f99f0009190 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 msgr2=0x7f9a000b1ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f99f8034040 tx=0x7f99f8004a40 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 shutdown_connections 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f99e006ac00 0x7f99e006d0c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9a0014dec0 0x7f9a000b1ed0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9a0015a540 0x7f9a0014d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 --2- 172.21.15.28:0/397244263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a000cabc0 0x7f9a000b0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.122+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 >> 172.21.15.28:0/397244263 conn(0x7f9a00069e80 msgr2=0x7f9a000ce9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.123+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 shutdown_connections 2024-09-13T17:21:37.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.123+0000 7f9a04a00640 1 -- 172.21.15.28:0/397244263 wait complete. 2024-09-13T17:21:37.139 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.294+0000 7f89aac00640 1 Processor -- start 2024-09-13T17:21:37.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.294+0000 7f89aac00640 1 -- start start 2024-09-13T17:21:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.295+0000 7f89aac00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89a41592a0 0x7f89a415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.295+0000 7f89aac00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a4159d90 con 0x7f89a406c2d0 2024-09-13T17:21:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.295+0000 7f89aac00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a40b2040 con 0x7f89a406f970 2024-09-13T17:21:37.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.295+0000 7f89aac00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a415b0b0 con 0x7f89a41592a0 2024-09-13T17:21:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a8c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f89a406c2d0 0x7f89a40b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58420/0 (socket says 172.21.15.28:58420) 2024-09-13T17:21:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a8c00640 1 -- 172.21.15.28:0/1560732634 learned_addr learned my addr 172.21.15.28:0/1560732634 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a3e00640 1 --2- 172.21.15.28:0/1560732634 >> v2:172.21.15.189:3300/0 conn(0x7f89a41592a0 0x7f89a415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411975777 0 0) 0x7f89a4159d90 con 0x7f89a406c2d0 2024-09-13T17:21:37.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988003520 con 0x7f89a406c2d0 2024-09-13T17:21:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a3e00640 1 -- 172.21.15.28:0/1560732634 >> v1:172.21.15.189:6789/0 conn(0x7f89a406f970 legacy=0x7f89a4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3495836807 0 0) 0x7f8988003520 con 0x7f89a406c2d0 2024-09-13T17:21:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a3e00640 1 -- 172.21.15.28:0/1560732634 >> v1:172.21.15.28:6789/0 conn(0x7f89a406c2d0 legacy=0x7f89a40b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.296+0000 7f89a3e00640 1 -- 172.21.15.28:0/1560732634 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4159ff0 con 0x7f89a41592a0 2024-09-13T17:21:37.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.297+0000 7f89a3e00640 1 --2- 172.21.15.28:0/1560732634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a415ab70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f89940097a0 tx=0x7f899402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40f78fb7761b7922 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.298+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899c001a10 con 0x7f89a41592a0 2024-09-13T17:21:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.298+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899402ac50 con 0x7f89a41592a0 2024-09-13T17:21:37.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.298+0000 7f89a2a00640 1 -- 172.21.15.28:0/1560732634 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994032440 con 0x7f89a41592a0 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 -- 172.21.15.28:0/1560732634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 msgr2=0x7f89a415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 --2- 172.21.15.28:0/1560732634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a415ab70 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f89940097a0 tx=0x7f899402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 -- 172.21.15.28:0/1560732634 shutdown_connections 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 --2- 172.21.15.28:0/1560732634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a415ab70 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 -- 172.21.15.28:0/1560732634 >> 172.21.15.28:0/1560732634 conn(0x7f89a4069e80 msgr2=0x7f89a406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 -- 172.21.15.28:0/1560732634 shutdown_connections 2024-09-13T17:21:37.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.299+0000 7f89aac00640 1 -- 172.21.15.28:0/1560732634 wait complete. 2024-09-13T17:21:37.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.300+0000 7f89aac00640 1 Processor -- start 2024-09-13T17:21:37.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.300+0000 7f89aac00640 1 -- start start 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a406f970 0x7f89a4152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a40a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40a98d0 con 0x7f89a406c2d0 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40abd80 con 0x7f89a41592a0 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.301+0000 7f89aac00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40ac060 con 0x7f89a406f970 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39878/0 (socket says 172.21.15.28:39878) 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 -- 172.21.15.28:0/1088896416 learned_addr learned my addr 172.21.15.28:0/1088896416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:37.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a8c00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a40a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3400640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a406f970 0x7f89a4152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 -- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a406f970 msgr2=0x7f89a4152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a406f970 0x7f89a4152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 -- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 msgr2=0x7f89a40a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a40a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.302+0000 7f89a3e00640 1 -- 172.21.15.28:0/1088896416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8994009400 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f89a3e00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8994035040 tx=0x7f89940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899403c680 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899403c7e0 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899403cb70 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a40ac340 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.303+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a414a4f0 con 0x7f89a406c2d0 2024-09-13T17:21:37.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.304+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a40ac860 con 0x7f89a406c2d0 2024-09-13T17:21:37.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.305+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8994039040 con 0x7f89a406c2d0 2024-09-13T17:21:37.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.306+0000 7f899be00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.306+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89940a9aa0 con 0x7f89a406c2d0 2024-09-13T17:21:37.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.307+0000 7f89a3400640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.307+0000 7f89a3400640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 0x7f898406d0c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f89a40ad8c0 tx=0x7f8990009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.310+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f89a40ac860 con 0x7f89a406c2d0 2024-09-13T17:21:37.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.490+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89a40ac860 con 0x7f89a406c2d0 2024-09-13T17:21:37.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.491+0000 7f899be00640 1 -- 172.21.15.28:0/1088896416 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16596 (secure 0 0 0) 0x7f89a40ac860 con 0x7f89a406c2d0 2024-09-13T17:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:36.720812+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.493+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 msgr2=0x7f898406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.493+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 0x7f898406d0c0 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f89a40ad8c0 tx=0x7f8990009040 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.493+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 msgr2=0x7f89a4149fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.493+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 secure :-1 s=READY pgs=2304 cs=0 l=1 rev1=1 crypto rx=0x7f8994035040 tx=0x7f89940046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 shutdown_connections 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a41592a0 0x7f89a40a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a406f970 0x7f89a4152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 --2- 172.21.15.28:0/1088896416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a406c2d0 0x7f89a4149fb0 unknown :-1 s=CLOSED pgs=2304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 >> 172.21.15.28:0/1088896416 conn(0x7f89a4069e80 msgr2=0x7f89a406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 shutdown_connections 2024-09-13T17:21:37.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.494+0000 7f89aac00640 1 -- 172.21.15.28:0/1088896416 wait complete. 2024-09-13T17:21:37.511 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:21:37.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.668+0000 7fa6d1000640 1 Processor -- start 2024-09-13T17:21:37.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.668+0000 7fa6d1000640 1 -- start start 2024-09-13T17:21:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6d1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 0x7fa6cc1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6d1000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa6cc06ccd0 0x7fa6cc06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6d1000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6cc1595f0 con 0x7fa6cc06bf40 2024-09-13T17:21:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6d1000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6cc06c3f0 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6d1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc1598c0 con 0x7fa6cc158cd0 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6c9e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 0x7fa6cc1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6c1400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa6cc06ccd0 0x7fa6cc06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6c9e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 0x7fa6cc1590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55192/0 (socket says 172.21.15.28:55192) 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.669+0000 7fa6c9e00640 1 -- 172.21.15.28:0/3068657895 learned_addr learned my addr 172.21.15.28:0/3068657895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.670+0000 7fa6c1400640 1 -- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 msgr2=0x7fa6cc1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:37.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.670+0000 7fa6c1400640 1 --2- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 0x7fa6cc1590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.670+0000 7fa6c1400640 1 -- 172.21.15.28:0/3068657895 >> v1:172.21.15.189:6789/0 conn(0x7fa6cc06bf40 legacy=0x7fa6cc15c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.670+0000 7fa6c1400640 1 -- 172.21.15.28:0/3068657895 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc159ba0 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.671+0000 7fa6c1400640 1 --2- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc06ccd0 0x7fa6cc06ba00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40097a0 tx=0x7fa6b402c390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eec521aae53a037c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.671+0000 7fa6c9400640 1 -- 172.21.15.28:0/3068657895 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4034640 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.671+0000 7fa6c9400640 1 -- 172.21.15.28:0/3068657895 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b40347a0 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.671+0000 7fa6c9400640 1 -- 172.21.15.28:0/3068657895 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4034b30 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 -- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc06ccd0 msgr2=0x7fa6cc06ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 --2- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc06ccd0 0x7fa6cc06ba00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa6b40097a0 tx=0x7fa6b402c390 comp rx=0 tx=0).stop 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 -- 172.21.15.28:0/3068657895 shutdown_connections 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 --2- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc06ccd0 0x7fa6cc06ba00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 --2- 172.21.15.28:0/3068657895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc158cd0 0x7fa6cc1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 -- 172.21.15.28:0/3068657895 >> 172.21.15.28:0/3068657895 conn(0x7fa6cc0b0270 msgr2=0x7fa6cc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:37.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 -- 172.21.15.28:0/3068657895 shutdown_connections 2024-09-13T17:21:37.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.672+0000 7fa6d1000640 1 -- 172.21.15.28:0/3068657895 wait complete. 2024-09-13T17:21:37.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.674+0000 7fa6d1000640 1 Processor -- start 2024-09-13T17:21:37.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.674+0000 7fa6d1000640 1 -- start start 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc06ccd0 0x7fa6cc06b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 0x7fa6cc06b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc13f2c0 con 0x7fa6cc06bf40 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc13f780 con 0x7fa6cc158cd0 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6d1000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc13efe0 con 0x7fa6cc06ccd0 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39890/0 (socket says 172.21.15.28:39890) 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 -- 172.21.15.28:0/2049816252 learned_addr learned my addr 172.21.15.28:0/2049816252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c1400640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc06ccd0 0x7fa6cc06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6ca800640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 0x7fa6cc06b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 -- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc06ccd0 msgr2=0x7fa6cc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc06ccd0 0x7fa6cc06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 -- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 msgr2=0x7fa6cc06b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 0x7fa6cc06b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6c9e00640 1 -- 172.21.15.28:0/2049816252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b4009400 con 0x7fa6cc06bf40 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.675+0000 7fa6ca800640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 0x7fa6cc06b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6c9e00640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009fd0 tx=0x7fa6bc00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00c940 con 0x7fa6cc06bf40 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00caa0 con 0x7fa6cc06bf40 2024-09-13T17:21:37.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc13c430 con 0x7fa6cc06bf40 2024-09-13T17:21:37.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00ce30 con 0x7fa6cc06bf40 2024-09-13T17:21:37.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.676+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc13c9f0 con 0x7fa6cc06bf40 2024-09-13T17:21:37.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.677+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc13c290 con 0x7fa6cc06bf40 2024-09-13T17:21:37.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.678+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa6bc013470 con 0x7fa6cc06bf40 2024-09-13T17:21:37.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.679+0000 7fa6c2a00640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 0x7fa6a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:37.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.679+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6bc089540 con 0x7fa6cc06bf40 2024-09-13T17:21:37.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.679+0000 7fa6c1400640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 0x7fa6a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:37.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.680+0000 7fa6c1400640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 0x7fa6a806d0c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4036000 tx=0x7fa6b4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:37.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.683+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa6cc13c290 con 0x7fa6cc06bf40 2024-09-13T17:21:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:37.907+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa6cc13c290 con 0x7fa6cc06bf40 2024-09-13T17:21:38.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.738+0000 7fa6c2a00640 1 -- 172.21.15.28:0/2049816252 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v164) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa6cc13c290 con 0x7fa6cc06bf40 2024-09-13T17:21:38.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 msgr2=0x7fa6a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:38.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 0x7fa6a806d0c0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fa6b4036000 tx=0x7fa6b4030000 comp rx=0 tx=0).stop 2024-09-13T17:21:38.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 msgr2=0x7fa6cc13bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:38.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 secure :-1 s=READY pgs=2305 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009fd0 tx=0x7fa6bc00ed40 comp rx=0 tx=0).stop 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 shutdown_connections 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa6a806ac00 0x7fa6a806d0c0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6cc158cd0 0x7fa6cc06b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa6cc06ccd0 0x7fa6cc06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 --2- 172.21.15.28:0/2049816252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6cc06bf40 0x7fa6cc13bb90 unknown :-1 s=CLOSED pgs=2305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 >> 172.21.15.28:0/2049816252 conn(0x7fa6cc0b0270 msgr2=0x7fa6cc0ad610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.741+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 shutdown_connections 2024-09-13T17:21:38.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.742+0000 7fa6d1000640 1 -- 172.21.15.28:0/2049816252 wait complete. 2024-09-13T17:21:38.760 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:21:38.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:21:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.923+0000 7f210aa00640 1 Processor -- start 2024-09-13T17:21:38.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.923+0000 7f210aa00640 1 -- start start 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2104159770 0x7f210406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f2104070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21040add00 0x7f2104159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210406f920 con 0x7f21040add00 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f210415b290 con 0x7f2104159770 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f210aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f210415b570 con 0x7f210406f3f0 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2108a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21040add00 0x7f2104159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2108a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f21040add00 0x7f2104159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39892/0 (socket says 172.21.15.28:39892) 2024-09-13T17:21:38.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2108a00640 1 -- 172.21.15.28:0/3324550408 learned_addr learned my addr 172.21.15.28:0/3324550408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2103e00640 1 --2- 172.21.15.28:0/3324550408 >> v2:172.21.15.189:3300/0 conn(0x7f2104159770 0x7f210406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f20fb400640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f2104070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2108a00640 1 -- 172.21.15.28:0/3324550408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 msgr2=0x7f2104070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.925+0000 7f2108a00640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f2104070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.926+0000 7f2108a00640 1 -- 172.21.15.28:0/3324550408 >> v2:172.21.15.189:3300/0 conn(0x7f2104159770 msgr2=0x7f210406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.926+0000 7f2108a00640 1 --2- 172.21.15.28:0/3324550408 >> v2:172.21.15.189:3300/0 conn(0x7f2104159770 0x7f210406fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.926+0000 7f2108a00640 1 -- 172.21.15.28:0/3324550408 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f210415b850 con 0x7f21040add00 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.926+0000 7f20fb400640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f2104070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f2108a00640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21040add00 0x7f2104159b50 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f2104067f80 tx=0x7f20fc02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1720241421f40b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f2103400640 1 -- 172.21.15.28:0/3324550408 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc031600 con 0x7f21040add00 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f2103400640 1 -- 172.21.15.28:0/3324550408 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc02d050 con 0x7f21040add00 2024-09-13T17:21:38.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f2103400640 1 -- 172.21.15.28:0/3324550408 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc031980 con 0x7f21040add00 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 -- 172.21.15.28:0/3324550408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21040add00 msgr2=0x7f2104159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21040add00 0x7f2104159b50 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7f2104067f80 tx=0x7f20fc02a610 comp rx=0 tx=0).stop 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 -- 172.21.15.28:0/3324550408 shutdown_connections 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f21040add00 0x7f2104159b50 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 --2- 172.21.15.28:0/3324550408 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f2104070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 --2- 172.21.15.28:0/3324550408 >> v2:172.21.15.189:3300/0 conn(0x7f2104159770 0x7f210406fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 -- 172.21.15.28:0/3324550408 >> 172.21.15.28:0/3324550408 conn(0x7f21040b0270 msgr2=0x7f21040b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:38.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.927+0000 7f210aa00640 1 -- 172.21.15.28:0/3324550408 shutdown_connections 2024-09-13T17:21:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.928+0000 7f210aa00640 1 -- 172.21.15.28:0/3324550408 wait complete. 2024-09-13T17:21:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.929+0000 7f210aa00640 1 Processor -- start 2024-09-13T17:21:38.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.929+0000 7f210aa00640 1 -- start start 2024-09-13T17:21:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f21041e47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21040add00 0x7f21040af7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041e7940 con 0x7f2104159770 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21041e7380 con 0x7f21040add00 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f210aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21041e7660 con 0x7f210406f3f0 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f2108a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.930+0000 7f2108a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39904/0 (socket says 172.21.15.28:39904) 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 -- 172.21.15.28:0/2508363740 learned_addr learned my addr 172.21.15.28:0/2508363740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2103e00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f21041e47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f20fb400640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21040add00 0x7f21040af7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 -- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 msgr2=0x7f21041e47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f21041e47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 -- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21040add00 msgr2=0x7f21040af7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21040add00 0x7f21040af7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:38.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 -- 172.21.15.28:0/2508363740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc009b30 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2108a00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f20fc034040 tx=0x7f20fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc03d860 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.931+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20fc03d9e0 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.932+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21041e19d0 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.932+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20fc03dd90 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.932+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21041e4d20 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.933+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21041e4eb0 con 0x7f2104159770 2024-09-13T17:21:38.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.933+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f20fc03b400 con 0x7f2104159770 2024-09-13T17:21:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.934+0000 7f2100c00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 0x7f20e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.934+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20fc0aa1c0 con 0x7f2104159770 2024-09-13T17:21:38.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.935+0000 7f2103e00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 0x7f20e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:38.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.936+0000 7f2103e00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 0x7f20e006d0c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009a20 tx=0x7f20f4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:38.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:38.938+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f21041e4eb0 con 0x7f2104159770 2024-09-13T17:21:39.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.151+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f21041e4eb0 con 0x7f2104159770 2024-09-13T17:21:39.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.745+0000 7f2100c00640 1 -- 172.21.15.28:0/2508363740 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v165) v1 ==== 122+0+0 (secure 0 0 0) 0x7f21041e4eb0 con 0x7f2104159770 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.747+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 msgr2=0x7f20e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.747+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 0x7f20e006d0c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f20f4009a20 tx=0x7f20f4009190 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 msgr2=0x7f2104149a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7f20fc034040 tx=0x7f20fc004a40 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 shutdown_connections 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f20e006ac00 0x7f20e006d0c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2104159770 0x7f2104149a70 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f21040add00 0x7f21040af7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 --2- 172.21.15.28:0/2508363740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f210406f3f0 0x7f21041e47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 >> 172.21.15.28:0/2508363740 conn(0x7f21040b0270 msgr2=0x7f210406fa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:39.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 shutdown_connections 2024-09-13T17:21:39.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.748+0000 7f210aa00640 1 -- 172.21.15.28:0/2508363740 wait complete. 2024-09-13T17:21:39.771 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:21:39.771 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.929+0000 7f4fb3200640 1 Processor -- start 2024-09-13T17:21:39.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.929+0000 7f4fb3200640 1 -- start start 2024-09-13T17:21:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fb3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fb40b43c0 0x7f4fb40b4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fb3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb40d0d80 con 0x7f4fb415c750 2024-09-13T17:21:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fb3200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb40b4d40 con 0x7f4fb4069e80 2024-09-13T17:21:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fb3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb40d0960 con 0x7f4fb40b43c0 2024-09-13T17:21:39.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fb40b43c0 0x7f4fb40b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fabe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fb40b43c0 0x7f4fb40b4800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39908/0 (socket says 172.21.15.28:39908) 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fabe00640 1 -- 172.21.15.28:0/4220905612 learned_addr learned my addr 172.21.15.28:0/4220905612 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.930+0000 7f4fb1e00640 1 --1- 172.21.15.28:0/4220905612 >> v1:172.21.15.28:6789/0 conn(0x7f4fb415c750 0x7f4fb40d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58434/0 (socket says 172.21.15.28:58434) 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.931+0000 7f4fabe00640 1 -- 172.21.15.28:0/4220905612 >> v1:172.21.15.189:6789/0 conn(0x7f4fb4069e80 legacy=0x7f4fb40b5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.931+0000 7f4fabe00640 1 -- 172.21.15.28:0/4220905612 >> v1:172.21.15.28:6789/0 conn(0x7f4fb415c750 legacy=0x7f4fb40d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.931+0000 7f4fb1400640 1 -- 172.21.15.28:0/4220905612 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907066244 0 0) 0x7f4fb40d0d80 con 0x7f4fb415c750 2024-09-13T17:21:39.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.931+0000 7f4fabe00640 1 -- 172.21.15.28:0/4220905612 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb40d1ee0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.932+0000 7f4fabe00640 1 --2- 172.21.15.28:0/4220905612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb40b4800 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009870 tx=0x7f4f9c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6e1879ec59e328 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.932+0000 7f4fb1400640 1 -- 172.21.15.28:0/4220905612 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c032440 con 0x7f4fb40b43c0 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.932+0000 7f4fb1400640 1 -- 172.21.15.28:0/4220905612 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c0325a0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.932+0000 7f4fb1400640 1 -- 172.21.15.28:0/4220905612 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c032930 con 0x7f4fb40b43c0 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 -- 172.21.15.28:0/4220905612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 msgr2=0x7f4fb40b4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 --2- 172.21.15.28:0/4220905612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb40b4800 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009870 tx=0x7f4f9c02a210 comp rx=0 tx=0).stop 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 -- 172.21.15.28:0/4220905612 shutdown_connections 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 --2- 172.21.15.28:0/4220905612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb40b4800 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 -- 172.21.15.28:0/4220905612 >> 172.21.15.28:0/4220905612 conn(0x7f4fb40cf5a0 msgr2=0x7f4fb40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:39.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 -- 172.21.15.28:0/4220905612 shutdown_connections 2024-09-13T17:21:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.933+0000 7f4fb3200640 1 -- 172.21.15.28:0/4220905612 wait complete. 2024-09-13T17:21:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.934+0000 7f4fb3200640 1 Processor -- start 2024-09-13T17:21:39.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 -- start start 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 0x7f4fb4149c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb415c750 0x7f4fb40cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40ca920 con 0x7f4fb40b43c0 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40cac00 con 0x7f4fb415c750 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.935+0000 7f4fb3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb40caee0 con 0x7f4fb4069e80 2024-09-13T17:21:39.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39918/0 (socket says 172.21.15.28:39918) 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 -- 172.21.15.28:0/3528891037 learned_addr learned my addr 172.21.15.28:0/3528891037 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fb2800640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb415c750 0x7f4fb40cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fb1e00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 0x7f4fb4149c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 -- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 msgr2=0x7f4fb4149c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 0x7f4fb4149c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 -- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb415c750 msgr2=0x7f4fb40cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb415c750 0x7f4fb40cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 -- 172.21.15.28:0/3528891037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f9c0094d0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fb1e00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 0x7f4fb4149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:39.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.936+0000 7f4fabe00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009870 tx=0x7f4f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.937+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c032440 con 0x7f4fb40b43c0 2024-09-13T17:21:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.937+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f9c004250 con 0x7f4fb40b43c0 2024-09-13T17:21:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.937+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f9c030980 con 0x7f4fb40b43c0 2024-09-13T17:21:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.937+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb40cb1c0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.937+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb41543c0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.938+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb40cb6e0 con 0x7f4fb40b43c0 2024-09-13T17:21:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.939+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f9c039050 con 0x7f4fb40b43c0 2024-09-13T17:21:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.940+0000 7f4faa000640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 0x7f4f9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:39.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.940+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4f9c0a9d30 con 0x7f4fb40b43c0 2024-09-13T17:21:39.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.940+0000 7f4fb1e00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 0x7f4f9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:39.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.941+0000 7f4fb1e00640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 0x7f4f9006d0c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4000f60 tx=0x7f4fa4007cc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:39.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:39.944+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4fb40cb6e0 con 0x7f4fb40b43c0 2024-09-13T17:21:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.158+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fb40cb020 con 0x7f4fb40b43c0 2024-09-13T17:21:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.159+0000 7f4faa000640 1 -- 172.21.15.28:0/3528891037 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f4fb40cb020 con 0x7f4fb40b43c0 2024-09-13T17:21:40.161 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:40.161 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:21:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:21:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 msgr2=0x7f4f9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 0x7f4f9006d0c0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f4fa4000f60 tx=0x7f4fa4007cc0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 msgr2=0x7f4fb41f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c009870 tx=0x7f4f9c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 shutdown_connections 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f9006ac00 0x7f4f9006d0c0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fb415c750 0x7f4fb40cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fb40b43c0 0x7f4fb41f2800 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 --2- 172.21.15.28:0/3528891037 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fb4069e80 0x7f4fb4149c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 >> 172.21.15.28:0/3528891037 conn(0x7f4fb40cf5a0 msgr2=0x7f4fb415b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 shutdown_connections 2024-09-13T17:21:40.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.161+0000 7f4fb3200640 1 -- 172.21.15.28:0/3528891037 wait complete. 2024-09-13T17:21:40.179 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:21:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.333+0000 7f260c400640 1 Processor -- start 2024-09-13T17:21:40.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.333+0000 7f260c400640 1 -- start start 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260c400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f2604146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260c400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f26040ad260 0x7f26040b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260c400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26040b2a40 con 0x7f26040b25f0 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260c400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2604143070 con 0x7f26040ad260 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260c400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2604071fe0 con 0x7f2604071c00 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.334+0000 7f260a400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f26040b25f0 0x7f26041604a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58450/0 (socket says 172.21.15.28:58450) 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f260a400640 1 -- 172.21.15.28:0/3056106387 learned_addr learned my addr 172.21.15.28:0/3056106387 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2609000640 1 --2- 172.21.15.28:0/3056106387 >> v2:172.21.15.189:3300/0 conn(0x7f26040ad260 0x7f26040b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2609a00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f2604146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261133324 0 0) 0x7f25fc001490 con 0x7f26040b25f0 2024-09-13T17:21:40.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25ec003520 con 0x7f26040b25f0 2024-09-13T17:21:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133777596 0 0) 0x7f25ec003520 con 0x7f26040b25f0 2024-09-13T17:21:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 msgr2=0x7f2604146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f2604146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 >> v2:172.21.15.189:3300/0 conn(0x7f26040ad260 msgr2=0x7f26040b20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:40.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 --2- 172.21.15.28:0/3056106387 >> v2:172.21.15.189:3300/0 conn(0x7f26040ad260 0x7f26040b20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26040ada70 con 0x7f26040b25f0 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f25fc001b80 con 0x7f26040b25f0 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 >> v1:172.21.15.28:6789/0 conn(0x7f26040b25f0 legacy=0x7f26041604a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f25ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.335+0000 7f2603e00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f25ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2603e00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25ec004d00 0x7f25ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2604143070 con 0x7f25ec004d00 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2604071fe0 con 0x7f25ec003970 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ec00b980 con 0x7f2604071c00 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f25ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609000640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f25ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f260a400640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25ec004d00 0x7f25ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 -- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 msgr2=0x7f25ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f25ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 -- 172.21.15.28:0/3056106387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25ec004d00 msgr2=0x7f25ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25ec004d00 0x7f25ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609a00640 1 -- 172.21.15.28:0/3056106387 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ec003520 con 0x7f25ec003970 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f2609000640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2604071c00 0x7f25ec0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.336+0000 7f260a400640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f25ec004d00 0x7f25ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:40.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.337+0000 7f2609a00640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f25ec003d90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009ac0 tx=0x7f25ec0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ba19193aa34fa6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8031400 con 0x7f25ec003970 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25f802d050 con 0x7f25ec003970 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f2603e00640 1 -- 172.21.15.28:0/3056106387 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25f8031780 con 0x7f25ec003970 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f260c400640 1 -- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 msgr2=0x7f25ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f260c400640 1 --2- 172.21.15.28:0/3056106387 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f25ec003d90 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f25f8009ac0 tx=0x7f25ec0057f0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f260a400640 1 -- 172.21.15.28:0/3056106387 reap_dead start 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f260c400640 1 -- 172.21.15.28:0/3056106387 shutdown_connections 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.338+0000 7f260c400640 1 -- 172.21.15.28:0/3056106387 >> 172.21.15.28:0/3056106387 conn(0x7f26040b0b60 msgr2=0x7f26040b0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.339+0000 7f260c400640 1 -- 172.21.15.28:0/3056106387 shutdown_connections 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.339+0000 7f260c400640 1 -- 172.21.15.28:0/3056106387 wait complete. 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.340+0000 7f260c400640 1 Processor -- start 2024-09-13T17:21:40.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.340+0000 7f260c400640 1 -- start start 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f26040cc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26040ccd60 0x7f260413b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26040cde10 con 0x7f260413b720 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26040cd200 con 0x7f25ec003970 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.341+0000 7f260c400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26040cd4e0 con 0x7f26040ccd60 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39936/0 (socket says 172.21.15.28:39936) 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 -- 172.21.15.28:0/2813858671 learned_addr learned my addr 172.21.15.28:0/2813858671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f2609000640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26040ccd60 0x7f260413b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f2609a00640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f26040cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 -- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26040ccd60 msgr2=0x7f260413b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26040ccd60 0x7f260413b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 -- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 msgr2=0x7f26040cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f26040cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 -- 172.21.15.28:0/2813858671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26040cd7c0 con 0x7f260413b720 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f2609a00640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f26040cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.342+0000 7f260a400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f26040cedc0 tx=0x7f25fc00ae50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.343+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc007840 con 0x7f260413b720 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.343+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc0079a0 con 0x7f260413b720 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.343+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc007b10 con 0x7f260413b720 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.343+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26040cdaa0 con 0x7f260413b720 2024-09-13T17:21:40.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.343+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2604138d90 con 0x7f260413b720 2024-09-13T17:21:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.344+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26040cdf80 con 0x7f260413b720 2024-09-13T17:21:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.345+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f25fc013630 con 0x7f260413b720 2024-09-13T17:21:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.346+0000 7f2601600640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 0x7f25e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.346+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25fc089990 con 0x7f260413b720 2024-09-13T17:21:40.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.346+0000 7f2609a00640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 0x7f25e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.347+0000 7f2609a00640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 0x7f25e406d0c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f25f8033000 tx=0x7f25f8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.349+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f25fc054b60 con 0x7f260413b720 2024-09-13T17:21:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.530+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26040b2a40 con 0x7f260413b720 2024-09-13T17:21:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.532+0000 7f2601600640 1 -- 172.21.15.28:0/2813858671 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16583 (secure 0 0 0) 0x7f26040b2a40 con 0x7f260413b720 2024-09-13T17:21:40.534 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:40.534 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:37.723750+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:21:40.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 msgr2=0x7f25e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 0x7f25e406d0c0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f25f8033000 tx=0x7f25f8008030 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 msgr2=0x7f26041445d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 secure :-1 s=READY pgs=2312 cs=0 l=1 rev1=1 crypto rx=0x7f26040cedc0 tx=0x7f25fc00ae50 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 shutdown_connections 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f25e406ac00 0x7f25e406d0c0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f260413b720 0x7f26041445d0 unknown :-1 s=CLOSED pgs=2312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f26040ccd60 0x7f260413b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 --2- 172.21.15.28:0/2813858671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25ec003970 0x7f26040cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 >> 172.21.15.28:0/2813858671 conn(0x7f26040b0b60 msgr2=0x7f2604070190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 shutdown_connections 2024-09-13T17:21:40.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.534+0000 7f260c400640 1 -- 172.21.15.28:0/2813858671 wait complete. 2024-09-13T17:21:40.553 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:21:40.553 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:21:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.707+0000 7f16ace00640 1 Processor -- start 2024-09-13T17:21:40.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.707+0000 7f16ace00640 1 -- start start 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16ace00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f16a81592a0 0x7f16a815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16ace00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a8159d90 con 0x7f16a806f970 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16ace00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a80b2040 con 0x7f16a806c2d0 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16ace00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a815b0b0 con 0x7f16a81592a0 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16a5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f16a81592a0 0x7f16a815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16a5e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f16a81592a0 0x7f16a815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39948/0 (socket says 172.21.15.28:39948) 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f169fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f16a806f970 0x7f16a8159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58454/0 (socket says 172.21.15.28:58454) 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.708+0000 7f16a5e00640 1 -- 172.21.15.28:0/582771266 learned_addr learned my addr 172.21.15.28:0/582771266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.709+0000 7f16a5e00640 1 -- 172.21.15.28:0/582771266 >> v1:172.21.15.189:6789/0 conn(0x7f16a806c2d0 legacy=0x7f16a80b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.709+0000 7f16a5e00640 1 -- 172.21.15.28:0/582771266 >> v1:172.21.15.28:6789/0 conn(0x7f16a806f970 legacy=0x7f16a8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.709+0000 7f16a5e00640 1 -- 172.21.15.28:0/582771266 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a8159ff0 con 0x7f16a81592a0 2024-09-13T17:21:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.710+0000 7f16a5e00640 1 --2- 172.21.15.28:0/582771266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a815ab70 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f1698009870 tx=0x7f169802c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d04b6f5ad09634 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.710+0000 7f16a5400640 1 -- 172.21.15.28:0/582771266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698031710 con 0x7f16a81592a0 2024-09-13T17:21:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.710+0000 7f16a5400640 1 -- 172.21.15.28:0/582771266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169802f050 con 0x7f16a81592a0 2024-09-13T17:21:40.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.710+0000 7f16a5400640 1 -- 172.21.15.28:0/582771266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698031a90 con 0x7f16a81592a0 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 -- 172.21.15.28:0/582771266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 msgr2=0x7f16a815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 --2- 172.21.15.28:0/582771266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a815ab70 secure :-1 s=READY pgs=2314 cs=0 l=1 rev1=1 crypto rx=0x7f1698009870 tx=0x7f169802c510 comp rx=0 tx=0).stop 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 -- 172.21.15.28:0/582771266 shutdown_connections 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 --2- 172.21.15.28:0/582771266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a815ab70 unknown :-1 s=CLOSED pgs=2314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 -- 172.21.15.28:0/582771266 >> 172.21.15.28:0/582771266 conn(0x7f16a8069e80 msgr2=0x7f16a806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 -- 172.21.15.28:0/582771266 shutdown_connections 2024-09-13T17:21:40.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.711+0000 7f16ace00640 1 -- 172.21.15.28:0/582771266 wait complete. 2024-09-13T17:21:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.712+0000 7f16ace00640 1 Processor -- start 2024-09-13T17:21:40.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.713+0000 7f16ace00640 1 -- start start 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.713+0000 7f16ace00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16a806c2d0 0x7f16a8143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.713+0000 7f16ace00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16a806f970 0x7f16a80c4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16ace00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16ace00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c4750 con 0x7f16a81592a0 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16ace00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c95f0 con 0x7f16a806c2d0 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16ace00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c7610 con 0x7f16a806f970 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39958/0 (socket says 172.21.15.28:39958) 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 -- 172.21.15.28:0/3127449357 learned_addr learned my addr 172.21.15.28:0/3127449357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a5e00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16a806c2d0 0x7f16a8143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f169fe00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16a806f970 0x7f16a80c4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 -- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16a806f970 msgr2=0x7f16a80c4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16a806f970 0x7f16a80c4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 -- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16a806c2d0 msgr2=0x7f16a8143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16a806c2d0 0x7f16a8143a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.714+0000 7f16a6800640 1 -- 172.21.15.28:0/3127449357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980094d0 con 0x7f16a81592a0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f16a6800640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7d0 tx=0x7f16a000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0003b70 con 0x7f16a81592a0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0003cf0 con 0x7f16a81592a0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a000aa90 con 0x7f16a81592a0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a80c8a60 con 0x7f16a81592a0 2024-09-13T17:21:40.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.715+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a80c8fa0 con 0x7f16a81592a0 2024-09-13T17:21:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.716+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a80c9780 con 0x7f16a81592a0 2024-09-13T17:21:40.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.717+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f16a000abf0 con 0x7f16a81592a0 2024-09-13T17:21:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.718+0000 7f169e000640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 0x7f168406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.718+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f16a0088ff0 con 0x7f16a81592a0 2024-09-13T17:21:40.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.718+0000 7f16a5e00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 0x7f168406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:40.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.719+0000 7f16a5e00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 0x7f168406d0c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1698035000 tx=0x7f1698008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:40.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.722+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f16a00bc9c0 con 0x7f16a81592a0 2024-09-13T17:21:40.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.900+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16a815a890 con 0x7f168406ac00 2024-09-13T17:21:40.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.909+0000 7f169e000640 1 -- 172.21.15.28:0/3127449357 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289504 (secure 0 0 0) 0x7f1698043090 con 0x7f168406ac00 2024-09-13T17:21:40.914 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:21:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 msgr2=0x7f168406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 0x7f168406d0c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f1698035000 tx=0x7f1698008030 comp rx=0 tx=0).stop 2024-09-13T17:21:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 msgr2=0x7f16a80c70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 secure :-1 s=READY pgs=2315 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7d0 tx=0x7f16a000cc90 comp rx=0 tx=0).stop 2024-09-13T17:21:40.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 shutdown_connections 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f168406ac00 0x7f168406d0c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a81592a0 0x7f16a80c70d0 unknown :-1 s=CLOSED pgs=2315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f16a806f970 0x7f16a80c4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 --2- 172.21.15.28:0/3127449357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16a806c2d0 0x7f16a8143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 >> 172.21.15.28:0/3127449357 conn(0x7f16a8069e80 msgr2=0x7f16a806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 shutdown_connections 2024-09-13T17:21:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:40.913+0000 7f16ace00640 1 -- 172.21.15.28:0/3127449357 wait complete. 2024-09-13T17:21:40.932 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":597,"stamp":"2024-09-13T17:21:39.088638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":547628,"kb_used_data":20120,"kb_used_omap":12,"kb_used_meta":527347,"kb_avail":754427092,"statfs":{"total":773094113280,"available":772533342208,"internally_reserved":0,"allocated":20602880,"data_stored":9732862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":540003897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001005"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986010+0000","last_change":"2024-09-13T17:21:36.756178+0000","last_active":"2024-09-13T17:21:37.986010+0000","last_peered":"2024-09-13T17:21:37.986010+0000","last_clean":"2024-09-13T17:21:37.986010+0000","last_became_active":"2024-09-13T17:21:36.755914+0000","last_became_peered":"2024-09-13T17:21:36.755914+0000","last_unstale":"2024-09-13T17:21:37.986010+0000","last_undegraded":"2024-09-13T17:21:37.986010+0000","last_fullsized":"2024-09-13T17:21:37.986010+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:38.898990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733821+0000","last_change":"2024-09-13T17:21:36.736750+0000","last_active":"2024-09-13T17:21:37.733821+0000","last_peered":"2024-09-13T17:21:37.733821+0000","last_clean":"2024-09-13T17:21:37.733821+0000","last_became_active":"2024-09-13T17:21:36.736484+0000","last_became_peered":"2024-09-13T17:21:36.736484+0000","last_unstale":"2024-09-13T17:21:37.733821+0000","last_undegraded":"2024-09-13T17:21:37.733821+0000","last_fullsized":"2024-09-13T17:21:37.733821+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:29.353327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758339+0000","last_change":"2024-09-13T17:21:36.754492+0000","last_active":"2024-09-13T17:21:36.758339+0000","last_peered":"2024-09-13T17:21:36.758339+0000","last_clean":"2024-09-13T17:21:36.758339+0000","last_became_active":"2024-09-13T17:21:36.754237+0000","last_became_peered":"2024-09-13T17:21:36.754237+0000","last_unstale":"2024-09-13T17:21:36.758339+0000","last_undegraded":"2024-09-13T17:21:36.758339+0000","last_fullsized":"2024-09-13T17:21:36.758339+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.400778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733591+0000","last_change":"2024-09-13T17:21:36.752368+0000","last_active":"2024-09-13T17:21:37.733591+0000","last_peered":"2024-09-13T17:21:37.733591+0000","last_clean":"2024-09-13T17:21:37.733591+0000","last_became_active":"2024-09-13T17:21:36.750473+0000","last_became_peered":"2024-09-13T17:21:36.750473+0000","last_unstale":"2024-09-13T17:21:37.733591+0000","last_undegraded":"2024-09-13T17:21:37.733591+0000","last_fullsized":"2024-09-13T17:21:37.733591+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:08.997780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987041+0000","last_change":"2024-09-13T17:21:36.753080+0000","last_active":"2024-09-13T17:21:37.987041+0000","last_peered":"2024-09-13T17:21:37.987041+0000","last_clean":"2024-09-13T17:21:37.987041+0000","last_became_active":"2024-09-13T17:21:36.751006+0000","last_became_peered":"2024-09-13T17:21:36.751006+0000","last_unstale":"2024-09-13T17:21:37.987041+0000","last_undegraded":"2024-09-13T17:21:37.987041+0000","last_fullsized":"2024-09-13T17:21:37.987041+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:11.944621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986585+0000","last_change":"2024-09-13T17:21:36.737539+0000","last_active":"2024-09-13T17:21:37.986585+0000","last_peered":"2024-09-13T17:21:37.986585+0000","last_clean":"2024-09-13T17:21:37.986585+0000","last_became_active":"2024-09-13T17:21:36.736183+0000","last_became_peered":"2024-09-13T17:21:36.736183+0000","last_unstale":"2024-09-13T17:21:37.986585+0000","last_undegraded":"2024-09-13T17:21:37.986585+0000","last_fullsized":"2024-09-13T17:21:37.986585+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:53.816777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985683+0000","last_change":"2024-09-13T17:21:36.740091+0000","last_active":"2024-09-13T17:21:37.985683+0000","last_peered":"2024-09-13T17:21:37.985683+0000","last_clean":"2024-09-13T17:21:37.985683+0000","last_became_active":"2024-09-13T17:21:36.739810+0000","last_became_peered":"2024-09-13T17:21:36.739810+0000","last_unstale":"2024-09-13T17:21:37.985683+0000","last_undegraded":"2024-09-13T17:21:37.985683+0000","last_fullsized":"2024-09-13T17:21:37.985683+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:11.949573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986295+0000","last_change":"2024-09-13T17:21:36.736763+0000","last_active":"2024-09-13T17:21:37.986295+0000","last_peered":"2024-09-13T17:21:37.986295+0000","last_clean":"2024-09-13T17:21:37.986295+0000","last_became_active":"2024-09-13T17:21:36.736536+0000","last_became_peered":"2024-09-13T17:21:36.736536+0000","last_unstale":"2024-09-13T17:21:37.986295+0000","last_undegraded":"2024-09-13T17:21:37.986295+0000","last_fullsized":"2024-09-13T17:21:37.986295+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:29.353327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734378+0000","last_change":"2024-09-13T17:21:36.754642+0000","last_active":"2024-09-13T17:21:37.734378+0000","last_peered":"2024-09-13T17:21:37.734378+0000","last_clean":"2024-09-13T17:21:37.734378+0000","last_became_active":"2024-09-13T17:21:36.752872+0000","last_became_peered":"2024-09-13T17:21:36.752872+0000","last_unstale":"2024-09-13T17:21:37.734378+0000","last_undegraded":"2024-09-13T17:21:37.734378+0000","last_fullsized":"2024-09-13T17:21:37.734378+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:42.168423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732088+0000","last_change":"2024-09-13T17:21:36.745486+0000","last_active":"2024-09-13T17:21:37.732088+0000","last_peered":"2024-09-13T17:21:37.732088+0000","last_clean":"2024-09-13T17:21:37.732088+0000","last_became_active":"2024-09-13T17:21:36.745240+0000","last_became_peered":"2024-09-13T17:21:36.745240+0000","last_unstale":"2024-09-13T17:21:37.732088+0000","last_undegraded":"2024-09-13T17:21:37.732088+0000","last_fullsized":"2024-09-13T17:21:37.732088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:16.541840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757806+0000","last_change":"2024-09-13T17:21:36.751897+0000","last_active":"2024-09-13T17:21:36.757806+0000","last_peered":"2024-09-13T17:21:36.757806+0000","last_clean":"2024-09-13T17:21:36.757806+0000","last_became_active":"2024-09-13T17:21:36.751635+0000","last_became_peered":"2024-09-13T17:21:36.751635+0000","last_unstale":"2024-09-13T17:21:36.757806+0000","last_undegraded":"2024-09-13T17:21:36.757806+0000","last_fullsized":"2024-09-13T17:21:36.757806+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:06.879345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986547+0000","last_change":"2024-09-13T17:21:36.748467+0000","last_active":"2024-09-13T17:21:37.986547+0000","last_peered":"2024-09-13T17:21:37.986547+0000","last_clean":"2024-09-13T17:21:37.986547+0000","last_became_active":"2024-09-13T17:21:36.748162+0000","last_became_peered":"2024-09-13T17:21:36.748162+0000","last_unstale":"2024-09-13T17:21:37.986547+0000","last_undegraded":"2024-09-13T17:21:37.986547+0000","last_fullsized":"2024-09-13T17:21:37.986547+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:42.429017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986709+0000","last_change":"2024-09-13T17:21:36.743612+0000","last_active":"2024-09-13T17:21:37.986709+0000","last_peered":"2024-09-13T17:21:37.986709+0000","last_clean":"2024-09-13T17:21:37.986709+0000","last_became_active":"2024-09-13T17:21:36.743060+0000","last_became_peered":"2024-09-13T17:21:36.743060+0000","last_unstale":"2024-09-13T17:21:37.986709+0000","last_undegraded":"2024-09-13T17:21:37.986709+0000","last_fullsized":"2024-09-13T17:21:37.986709+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:05.733702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985726+0000","last_change":"2024-09-13T17:21:36.743186+0000","last_active":"2024-09-13T17:21:37.985726+0000","last_peered":"2024-09-13T17:21:37.985726+0000","last_clean":"2024-09-13T17:21:37.985726+0000","last_became_active":"2024-09-13T17:21:36.742943+0000","last_became_peered":"2024-09-13T17:21:36.742943+0000","last_unstale":"2024-09-13T17:21:37.985726+0000","last_undegraded":"2024-09-13T17:21:37.985726+0000","last_fullsized":"2024-09-13T17:21:37.985726+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:51.986633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986042+0000","last_change":"2024-09-13T17:21:36.746435+0000","last_active":"2024-09-13T17:21:37.986042+0000","last_peered":"2024-09-13T17:21:37.986042+0000","last_clean":"2024-09-13T17:21:37.986042+0000","last_became_active":"2024-09-13T17:21:36.744928+0000","last_became_peered":"2024-09-13T17:21:36.744928+0000","last_unstale":"2024-09-13T17:21:37.986042+0000","last_undegraded":"2024-09-13T17:21:37.986042+0000","last_fullsized":"2024-09-13T17:21:37.986042+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:13.412833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731689+0000","last_change":"2024-09-13T17:21:36.750007+0000","last_active":"2024-09-13T17:21:37.731689+0000","last_peered":"2024-09-13T17:21:37.731689+0000","last_clean":"2024-09-13T17:21:37.731689+0000","last_became_active":"2024-09-13T17:21:36.744658+0000","last_became_peered":"2024-09-13T17:21:36.744658+0000","last_unstale":"2024-09-13T17:21:37.731689+0000","last_undegraded":"2024-09-13T17:21:37.731689+0000","last_fullsized":"2024-09-13T17:21:37.731689+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:38.898990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986133+0000","last_change":"2024-09-13T17:21:36.756978+0000","last_active":"2024-09-13T17:21:37.986133+0000","last_peered":"2024-09-13T17:21:37.986133+0000","last_clean":"2024-09-13T17:21:37.986133+0000","last_became_active":"2024-09-13T17:21:36.756574+0000","last_became_peered":"2024-09-13T17:21:36.756574+0000","last_unstale":"2024-09-13T17:21:37.986133+0000","last_undegraded":"2024-09-13T17:21:37.986133+0000","last_fullsized":"2024-09-13T17:21:37.986133+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:22.135159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732088+0000","last_change":"2024-09-13T17:21:36.747673+0000","last_active":"2024-09-13T17:21:37.732088+0000","last_peered":"2024-09-13T17:21:37.732088+0000","last_clean":"2024-09-13T17:21:37.732088+0000","last_became_active":"2024-09-13T17:21:36.747233+0000","last_became_peered":"2024-09-13T17:21:36.747233+0000","last_unstale":"2024-09-13T17:21:37.732088+0000","last_undegraded":"2024-09-13T17:21:37.732088+0000","last_fullsized":"2024-09-13T17:21:37.732088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:28.804370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732167+0000","last_change":"2024-09-13T17:21:36.750165+0000","last_active":"2024-09-13T17:21:37.732167+0000","last_peered":"2024-09-13T17:21:37.732167+0000","last_clean":"2024-09-13T17:21:37.732167+0000","last_became_active":"2024-09-13T17:21:36.744559+0000","last_became_peered":"2024-09-13T17:21:36.744559+0000","last_unstale":"2024-09-13T17:21:37.732167+0000","last_undegraded":"2024-09-13T17:21:37.732167+0000","last_fullsized":"2024-09-13T17:21:37.732167+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:11.949573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986844+0000","last_change":"2024-09-13T17:21:36.748081+0000","last_active":"2024-09-13T17:21:37.986844+0000","last_peered":"2024-09-13T17:21:37.986844+0000","last_clean":"2024-09-13T17:21:37.986844+0000","last_became_active":"2024-09-13T17:21:36.747392+0000","last_became_peered":"2024-09-13T17:21:36.747392+0000","last_unstale":"2024-09-13T17:21:37.986844+0000","last_undegraded":"2024-09-13T17:21:37.986844+0000","last_fullsized":"2024-09-13T17:21:37.986844+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:38.521775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733940+0000","last_change":"2024-09-13T17:21:36.756830+0000","last_active":"2024-09-13T17:21:37.733940+0000","last_peered":"2024-09-13T17:21:37.733940+0000","last_clean":"2024-09-13T17:21:37.733940+0000","last_became_active":"2024-09-13T17:21:36.756397+0000","last_became_peered":"2024-09-13T17:21:36.756397+0000","last_unstale":"2024-09-13T17:21:37.733940+0000","last_undegraded":"2024-09-13T17:21:37.733940+0000","last_fullsized":"2024-09-13T17:21:37.733940+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:38.289392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987068+0000","last_change":"2024-09-13T17:21:36.749217+0000","last_active":"2024-09-13T17:21:37.987068+0000","last_peered":"2024-09-13T17:21:37.987068+0000","last_clean":"2024-09-13T17:21:37.987068+0000","last_became_active":"2024-09-13T17:21:36.748949+0000","last_became_peered":"2024-09-13T17:21:36.748949+0000","last_unstale":"2024-09-13T17:21:37.987068+0000","last_undegraded":"2024-09-13T17:21:37.987068+0000","last_fullsized":"2024-09-13T17:21:37.987068+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:10.179759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986797+0000","last_change":"2024-09-13T17:21:36.750807+0000","last_active":"2024-09-13T17:21:37.986797+0000","last_peered":"2024-09-13T17:21:37.986797+0000","last_clean":"2024-09-13T17:21:37.986797+0000","last_became_active":"2024-09-13T17:21:36.750529+0000","last_became_peered":"2024-09-13T17:21:36.750529+0000","last_unstale":"2024-09-13T17:21:37.986797+0000","last_undegraded":"2024-09-13T17:21:37.986797+0000","last_fullsized":"2024-09-13T17:21:37.986797+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:34.509612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733603+0000","last_change":"2024-09-13T17:21:36.759033+0000","last_active":"2024-09-13T17:21:37.733603+0000","last_peered":"2024-09-13T17:21:37.733603+0000","last_clean":"2024-09-13T17:21:37.733603+0000","last_became_active":"2024-09-13T17:21:36.758792+0000","last_became_peered":"2024-09-13T17:21:36.758792+0000","last_unstale":"2024-09-13T17:21:37.733603+0000","last_undegraded":"2024-09-13T17:21:37.733603+0000","last_fullsized":"2024-09-13T17:21:37.733603+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:28.758053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734249+0000","last_change":"2024-09-13T17:21:36.747690+0000","last_active":"2024-09-13T17:21:37.734249+0000","last_peered":"2024-09-13T17:21:37.734249+0000","last_clean":"2024-09-13T17:21:37.734249+0000","last_became_active":"2024-09-13T17:21:36.747422+0000","last_became_peered":"2024-09-13T17:21:36.747422+0000","last_unstale":"2024-09-13T17:21:37.734249+0000","last_undegraded":"2024-09-13T17:21:37.734249+0000","last_fullsized":"2024-09-13T17:21:37.734249+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:10.020415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986258+0000","last_change":"2024-09-13T17:21:36.743297+0000","last_active":"2024-09-13T17:21:37.986258+0000","last_peered":"2024-09-13T17:21:37.986258+0000","last_clean":"2024-09-13T17:21:37.986258+0000","last_became_active":"2024-09-13T17:21:36.743062+0000","last_became_peered":"2024-09-13T17:21:36.743062+0000","last_unstale":"2024-09-13T17:21:37.986258+0000","last_undegraded":"2024-09-13T17:21:37.986258+0000","last_fullsized":"2024-09-13T17:21:37.986258+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:38.055703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986506+0000","last_change":"2024-09-13T17:21:36.749920+0000","last_active":"2024-09-13T17:21:37.986506+0000","last_peered":"2024-09-13T17:21:37.986506+0000","last_clean":"2024-09-13T17:21:37.986506+0000","last_became_active":"2024-09-13T17:21:36.747974+0000","last_became_peered":"2024-09-13T17:21:36.747974+0000","last_unstale":"2024-09-13T17:21:37.986506+0000","last_undegraded":"2024-09-13T17:21:37.986506+0000","last_fullsized":"2024-09-13T17:21:37.986506+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:54.825955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.755370+0000","last_change":"2024-09-13T17:21:36.745077+0000","last_active":"2024-09-13T17:21:36.755370+0000","last_peered":"2024-09-13T17:21:36.755370+0000","last_clean":"2024-09-13T17:21:36.755370+0000","last_became_active":"2024-09-13T17:21:36.744089+0000","last_became_peered":"2024-09-13T17:21:36.744089+0000","last_unstale":"2024-09-13T17:21:36.755370+0000","last_undegraded":"2024-09-13T17:21:36.755370+0000","last_fullsized":"2024-09-13T17:21:36.755370+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:15.351494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986404+0000","last_change":"2024-09-13T17:21:36.748284+0000","last_active":"2024-09-13T17:21:37.986404+0000","last_peered":"2024-09-13T17:21:37.986404+0000","last_clean":"2024-09-13T17:21:37.986404+0000","last_became_active":"2024-09-13T17:21:36.747990+0000","last_became_peered":"2024-09-13T17:21:36.747990+0000","last_unstale":"2024-09-13T17:21:37.986404+0000","last_undegraded":"2024-09-13T17:21:37.986404+0000","last_fullsized":"2024-09-13T17:21:37.986404+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:33.924677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733504+0000","last_change":"2024-09-13T17:21:36.738847+0000","last_active":"2024-09-13T17:21:37.733504+0000","last_peered":"2024-09-13T17:21:37.733504+0000","last_clean":"2024-09-13T17:21:37.733504+0000","last_became_active":"2024-09-13T17:21:36.738528+0000","last_became_peered":"2024-09-13T17:21:36.738528+0000","last_unstale":"2024-09-13T17:21:37.733504+0000","last_undegraded":"2024-09-13T17:21:37.733504+0000","last_fullsized":"2024-09-13T17:21:37.733504+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:19.045283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731689+0000","last_change":"2024-09-13T17:21:36.754967+0000","last_active":"2024-09-13T17:21:37.731689+0000","last_peered":"2024-09-13T17:21:37.731689+0000","last_clean":"2024-09-13T17:21:37.731689+0000","last_became_active":"2024-09-13T17:21:36.754744+0000","last_became_peered":"2024-09-13T17:21:36.754744+0000","last_unstale":"2024-09-13T17:21:37.731689+0000","last_undegraded":"2024-09-13T17:21:37.731689+0000","last_fullsized":"2024-09-13T17:21:37.731689+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:22.135159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986088+0000","last_change":"2024-09-13T17:21:36.736750+0000","last_active":"2024-09-13T17:21:37.986088+0000","last_peered":"2024-09-13T17:21:37.986088+0000","last_clean":"2024-09-13T17:21:37.986088+0000","last_became_active":"2024-09-13T17:21:36.736062+0000","last_became_peered":"2024-09-13T17:21:36.736062+0000","last_unstale":"2024-09-13T17:21:37.986088+0000","last_undegraded":"2024-09-13T17:21:37.986088+0000","last_fullsized":"2024-09-13T17:21:37.986088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:17.123975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733918+0000","last_change":"2024-09-13T17:21:36.746377+0000","last_active":"2024-09-13T17:21:37.733918+0000","last_peered":"2024-09-13T17:21:37.733918+0000","last_clean":"2024-09-13T17:21:37.733918+0000","last_became_active":"2024-09-13T17:21:36.745648+0000","last_became_peered":"2024-09-13T17:21:36.745648+0000","last_unstale":"2024-09-13T17:21:37.733918+0000","last_undegraded":"2024-09-13T17:21:37.733918+0000","last_fullsized":"2024-09-13T17:21:37.733918+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:53.376685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"192'4","reported_seq":15,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986762+0000","last_change":"2024-09-13T17:21:36.309004+0000","last_active":"2024-09-13T17:21:37.986762+0000","last_peered":"2024-09-13T17:21:37.986762+0000","last_clean":"2024-09-13T17:21:37.986762+0000","last_became_active":"2024-09-13T17:21:36.308738+0000","last_became_peered":"2024-09-13T17:21:36.308738+0000","last_unstale":"2024-09-13T17:21:37.986762+0000","last_undegraded":"2024-09-13T17:21:37.986762+0000","last_fullsized":"2024-09-13T17:21:37.986762+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:33.142468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987210+0000","last_change":"2024-09-13T17:21:36.747527+0000","last_active":"2024-09-13T17:21:37.987210+0000","last_peered":"2024-09-13T17:21:37.987210+0000","last_clean":"2024-09-13T17:21:37.987210+0000","last_became_active":"2024-09-13T17:21:36.747131+0000","last_became_peered":"2024-09-13T17:21:36.747131+0000","last_unstale":"2024-09-13T17:21:37.987210+0000","last_undegraded":"2024-09-13T17:21:37.987210+0000","last_fullsized":"2024-09-13T17:21:37.987210+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:53.049699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734098+0000","last_change":"2024-09-13T17:21:36.307661+0000","last_active":"2024-09-13T17:21:37.734098+0000","last_peered":"2024-09-13T17:21:37.734098+0000","last_clean":"2024-09-13T17:21:37.734098+0000","last_became_active":"2024-09-13T17:21:36.307004+0000","last_became_peered":"2024-09-13T17:21:36.307004+0000","last_unstale":"2024-09-13T17:21:37.734098+0000","last_undegraded":"2024-09-13T17:21:37.734098+0000","last_fullsized":"2024-09-13T17:21:37.734098+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:44.202163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986047+0000","last_change":"2024-09-13T17:21:36.739149+0000","last_active":"2024-09-13T17:21:37.986047+0000","last_peered":"2024-09-13T17:21:37.986047+0000","last_clean":"2024-09-13T17:21:37.986047+0000","last_became_active":"2024-09-13T17:21:36.738897+0000","last_became_peered":"2024-09-13T17:21:36.738897+0000","last_unstale":"2024-09-13T17:21:37.986047+0000","last_undegraded":"2024-09-13T17:21:37.986047+0000","last_fullsized":"2024-09-13T17:21:37.986047+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:51.289471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"192'2","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758430+0000","last_change":"2024-09-13T17:21:36.308002+0000","last_active":"2024-09-13T17:21:36.758430+0000","last_peered":"2024-09-13T17:21:36.758430+0000","last_clean":"2024-09-13T17:21:36.758430+0000","last_became_active":"2024-09-13T17:21:36.307473+0000","last_became_peered":"2024-09-13T17:21:36.307473+0000","last_unstale":"2024-09-13T17:21:36.758430+0000","last_undegraded":"2024-09-13T17:21:36.758430+0000","last_fullsized":"2024-09-13T17:21:36.758430+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:51.374000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732038+0000","last_change":"2024-09-13T17:21:36.745188+0000","last_active":"2024-09-13T17:21:37.732038+0000","last_peered":"2024-09-13T17:21:37.732038+0000","last_clean":"2024-09-13T17:21:37.732038+0000","last_became_active":"2024-09-13T17:21:36.744544+0000","last_became_peered":"2024-09-13T17:21:36.744544+0000","last_unstale":"2024-09-13T17:21:37.732038+0000","last_undegraded":"2024-09-13T17:21:37.732038+0000","last_fullsized":"2024-09-13T17:21:37.732038+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:51.986633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734536+0000","last_change":"2024-09-13T17:21:36.312756+0000","last_active":"2024-09-13T17:21:37.734536+0000","last_peered":"2024-09-13T17:21:37.734536+0000","last_clean":"2024-09-13T17:21:37.734536+0000","last_became_active":"2024-09-13T17:21:36.312487+0000","last_became_peered":"2024-09-13T17:21:36.312487+0000","last_unstale":"2024-09-13T17:21:37.734536+0000","last_undegraded":"2024-09-13T17:21:37.734536+0000","last_fullsized":"2024-09-13T17:21:37.734536+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:09.013124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734063+0000","last_change":"2024-09-13T17:21:36.746808+0000","last_active":"2024-09-13T17:21:37.734063+0000","last_peered":"2024-09-13T17:21:37.734063+0000","last_clean":"2024-09-13T17:21:37.734063+0000","last_became_active":"2024-09-13T17:21:36.746526+0000","last_became_peered":"2024-09-13T17:21:36.746526+0000","last_unstale":"2024-09-13T17:21:37.734063+0000","last_undegraded":"2024-09-13T17:21:37.734063+0000","last_fullsized":"2024-09-13T17:21:37.734063+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:33.299639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986685+0000","last_change":"2024-09-13T17:21:36.299723+0000","last_active":"2024-09-13T17:21:37.986685+0000","last_peered":"2024-09-13T17:21:37.986685+0000","last_clean":"2024-09-13T17:21:37.986685+0000","last_became_active":"2024-09-13T17:21:36.299432+0000","last_became_peered":"2024-09-13T17:21:36.299432+0000","last_unstale":"2024-09-13T17:21:37.986685+0000","last_undegraded":"2024-09-13T17:21:37.986685+0000","last_fullsized":"2024-09-13T17:21:37.986685+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:02.934115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986340+0000","last_change":"2024-09-13T17:21:36.751769+0000","last_active":"2024-09-13T17:21:37.986340+0000","last_peered":"2024-09-13T17:21:37.986340+0000","last_clean":"2024-09-13T17:21:37.986340+0000","last_became_active":"2024-09-13T17:21:36.751503+0000","last_became_peered":"2024-09-13T17:21:36.751503+0000","last_unstale":"2024-09-13T17:21:37.986340+0000","last_undegraded":"2024-09-13T17:21:37.986340+0000","last_fullsized":"2024-09-13T17:21:37.986340+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:51.528433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986487+0000","last_change":"2024-09-13T17:21:36.314738+0000","last_active":"2024-09-13T17:21:37.986487+0000","last_peered":"2024-09-13T17:21:37.986487+0000","last_clean":"2024-09-13T17:21:37.986487+0000","last_became_active":"2024-09-13T17:21:36.313576+0000","last_became_peered":"2024-09-13T17:21:36.313576+0000","last_unstale":"2024-09-13T17:21:37.986487+0000","last_undegraded":"2024-09-13T17:21:37.986487+0000","last_fullsized":"2024-09-13T17:21:37.986487+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:32.917385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734458+0000","last_change":"2024-09-13T17:21:36.753712+0000","last_active":"2024-09-13T17:21:37.734458+0000","last_peered":"2024-09-13T17:21:37.734458+0000","last_clean":"2024-09-13T17:21:37.734458+0000","last_became_active":"2024-09-13T17:21:36.752544+0000","last_became_peered":"2024-09-13T17:21:36.752544+0000","last_unstale":"2024-09-13T17:21:37.734458+0000","last_undegraded":"2024-09-13T17:21:37.734458+0000","last_fullsized":"2024-09-13T17:21:37.734458+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:00.490825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986530+0000","last_change":"2024-09-13T17:21:36.299554+0000","last_active":"2024-09-13T17:21:37.986530+0000","last_peered":"2024-09-13T17:21:37.986530+0000","last_clean":"2024-09-13T17:21:37.986530+0000","last_became_active":"2024-09-13T17:21:36.299250+0000","last_became_peered":"2024-09-13T17:21:36.299250+0000","last_unstale":"2024-09-13T17:21:37.986530+0000","last_undegraded":"2024-09-13T17:21:37.986530+0000","last_fullsized":"2024-09-13T17:21:37.986530+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:59.483533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734015+0000","last_change":"2024-09-13T17:21:36.751006+0000","last_active":"2024-09-13T17:21:37.734015+0000","last_peered":"2024-09-13T17:21:37.734015+0000","last_clean":"2024-09-13T17:21:37.734015+0000","last_became_active":"2024-09-13T17:21:36.750753+0000","last_became_peered":"2024-09-13T17:21:36.750753+0000","last_unstale":"2024-09-13T17:21:37.734015+0000","last_undegraded":"2024-09-13T17:21:37.734015+0000","last_fullsized":"2024-09-13T17:21:37.734015+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:38.404283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987154+0000","last_change":"2024-09-13T17:21:36.301369+0000","last_active":"2024-09-13T17:21:37.987154+0000","last_peered":"2024-09-13T17:21:37.987154+0000","last_clean":"2024-09-13T17:21:37.987154+0000","last_became_active":"2024-09-13T17:21:36.301048+0000","last_became_peered":"2024-09-13T17:21:36.301048+0000","last_unstale":"2024-09-13T17:21:37.987154+0000","last_undegraded":"2024-09-13T17:21:37.987154+0000","last_fullsized":"2024-09-13T17:21:37.987154+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:20:38.392162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758424+0000","last_change":"2024-09-13T17:21:36.753522+0000","last_active":"2024-09-13T17:21:36.758424+0000","last_peered":"2024-09-13T17:21:36.758424+0000","last_clean":"2024-09-13T17:21:36.758424+0000","last_became_active":"2024-09-13T17:21:36.751908+0000","last_became_peered":"2024-09-13T17:21:36.751908+0000","last_unstale":"2024-09-13T17:21:36.758424+0000","last_undegraded":"2024-09-13T17:21:36.758424+0000","last_fullsized":"2024-09-13T17:21:36.758424+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:34.149760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987346+0000","last_change":"2024-09-13T17:21:36.309447+0000","last_active":"2024-09-13T17:21:37.987346+0000","last_peered":"2024-09-13T17:21:37.987346+0000","last_clean":"2024-09-13T17:21:37.987346+0000","last_became_active":"2024-09-13T17:21:36.308414+0000","last_became_peered":"2024-09-13T17:21:36.308414+0000","last_unstale":"2024-09-13T17:21:37.987346+0000","last_undegraded":"2024-09-13T17:21:37.987346+0000","last_fullsized":"2024-09-13T17:21:37.987346+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:37.396991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733804+0000","last_change":"2024-09-13T17:21:36.753468+0000","last_active":"2024-09-13T17:21:37.733804+0000","last_peered":"2024-09-13T17:21:37.733804+0000","last_clean":"2024-09-13T17:21:37.733804+0000","last_became_active":"2024-09-13T17:21:36.753099+0000","last_became_peered":"2024-09-13T17:21:36.753099+0000","last_unstale":"2024-09-13T17:21:37.733804+0000","last_undegraded":"2024-09-13T17:21:37.733804+0000","last_fullsized":"2024-09-13T17:21:37.733804+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:58.354697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"192'3","reported_seq":14,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732264+0000","last_change":"2024-09-13T17:21:36.309438+0000","last_active":"2024-09-13T17:21:37.732264+0000","last_peered":"2024-09-13T17:21:37.732264+0000","last_clean":"2024-09-13T17:21:37.732264+0000","last_became_active":"2024-09-13T17:21:36.309174+0000","last_became_peered":"2024-09-13T17:21:36.309174+0000","last_unstale":"2024-09-13T17:21:37.732264+0000","last_undegraded":"2024-09-13T17:21:37.732264+0000","last_fullsized":"2024-09-13T17:21:37.732264+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:09.538090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986224+0000","last_change":"2024-09-13T17:21:36.742783+0000","last_active":"2024-09-13T17:21:37.986224+0000","last_peered":"2024-09-13T17:21:37.986224+0000","last_clean":"2024-09-13T17:21:37.986224+0000","last_became_active":"2024-09-13T17:21:36.742452+0000","last_became_peered":"2024-09-13T17:21:36.742452+0000","last_unstale":"2024-09-13T17:21:37.986224+0000","last_undegraded":"2024-09-13T17:21:37.986224+0000","last_fullsized":"2024-09-13T17:21:37.986224+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:16.541840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"192'6","reported_seq":17,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734530+0000","last_change":"2024-09-13T17:21:36.308875+0000","last_active":"2024-09-13T17:21:37.734530+0000","last_peered":"2024-09-13T17:21:37.734530+0000","last_clean":"2024-09-13T17:21:37.734530+0000","last_became_active":"2024-09-13T17:21:36.308623+0000","last_became_peered":"2024-09-13T17:21:36.308623+0000","last_unstale":"2024-09-13T17:21:37.734530+0000","last_undegraded":"2024-09-13T17:21:37.734530+0000","last_fullsized":"2024-09-13T17:21:37.734530+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:53.028588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986667+0000","last_change":"2024-09-13T17:21:36.746168+0000","last_active":"2024-09-13T17:21:37.986667+0000","last_peered":"2024-09-13T17:21:37.986667+0000","last_clean":"2024-09-13T17:21:37.986667+0000","last_became_active":"2024-09-13T17:21:36.745926+0000","last_became_peered":"2024-09-13T17:21:36.745926+0000","last_unstale":"2024-09-13T17:21:37.986667+0000","last_undegraded":"2024-09-13T17:21:37.986667+0000","last_fullsized":"2024-09-13T17:21:37.986667+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:03.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987067+0000","last_change":"2024-09-13T17:21:36.302066+0000","last_active":"2024-09-13T17:21:37.987067+0000","last_peered":"2024-09-13T17:21:37.987067+0000","last_clean":"2024-09-13T17:21:37.987067+0000","last_became_active":"2024-09-13T17:21:36.301803+0000","last_became_peered":"2024-09-13T17:21:36.301803+0000","last_unstale":"2024-09-13T17:21:37.987067+0000","last_undegraded":"2024-09-13T17:21:37.987067+0000","last_fullsized":"2024-09-13T17:21:37.987067+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:32.292347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732144+0000","last_change":"2024-09-13T17:21:36.749244+0000","last_active":"2024-09-13T17:21:37.732144+0000","last_peered":"2024-09-13T17:21:37.732144+0000","last_clean":"2024-09-13T17:21:37.732144+0000","last_became_active":"2024-09-13T17:21:36.748557+0000","last_became_peered":"2024-09-13T17:21:36.748557+0000","last_unstale":"2024-09-13T17:21:37.732144+0000","last_undegraded":"2024-09-13T17:21:37.732144+0000","last_fullsized":"2024-09-13T17:21:37.732144+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.298517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987083+0000","last_change":"2024-09-13T17:21:36.304740+0000","last_active":"2024-09-13T17:21:37.987083+0000","last_peered":"2024-09-13T17:21:37.987083+0000","last_clean":"2024-09-13T17:21:37.987083+0000","last_became_active":"2024-09-13T17:21:36.304485+0000","last_became_peered":"2024-09-13T17:21:36.304485+0000","last_unstale":"2024-09-13T17:21:37.987083+0000","last_undegraded":"2024-09-13T17:21:37.987083+0000","last_fullsized":"2024-09-13T17:21:37.987083+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:07.990488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757904+0000","last_change":"2024-09-13T17:21:36.750869+0000","last_active":"2024-09-13T17:21:36.757904+0000","last_peered":"2024-09-13T17:21:36.757904+0000","last_clean":"2024-09-13T17:21:36.757904+0000","last_became_active":"2024-09-13T17:21:36.749555+0000","last_became_peered":"2024-09-13T17:21:36.749555+0000","last_unstale":"2024-09-13T17:21:36.757904+0000","last_undegraded":"2024-09-13T17:21:36.757904+0000","last_fullsized":"2024-09-13T17:21:36.757904+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:51.007537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731816+0000","last_change":"2024-09-13T17:21:36.313796+0000","last_active":"2024-09-13T17:21:37.731816+0000","last_peered":"2024-09-13T17:21:37.731816+0000","last_clean":"2024-09-13T17:21:37.731816+0000","last_became_active":"2024-09-13T17:21:36.313384+0000","last_became_peered":"2024-09-13T17:21:36.313384+0000","last_unstale":"2024-09-13T17:21:37.731816+0000","last_undegraded":"2024-09-13T17:21:37.731816+0000","last_fullsized":"2024-09-13T17:21:37.731816+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:03.457394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985822+0000","last_change":"2024-09-13T17:21:36.735315+0000","last_active":"2024-09-13T17:21:37.985822+0000","last_peered":"2024-09-13T17:21:37.985822+0000","last_clean":"2024-09-13T17:21:37.985822+0000","last_became_active":"2024-09-13T17:21:36.735019+0000","last_became_peered":"2024-09-13T17:21:36.735019+0000","last_unstale":"2024-09-13T17:21:37.985822+0000","last_undegraded":"2024-09-13T17:21:37.985822+0000","last_fullsized":"2024-09-13T17:21:37.985822+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:23.911415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"192'8","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.760616+0000","last_change":"2024-09-13T17:21:36.306747+0000","last_active":"2024-09-13T17:21:36.760616+0000","last_peered":"2024-09-13T17:21:36.760616+0000","last_clean":"2024-09-13T17:21:36.760616+0000","last_became_active":"2024-09-13T17:21:36.306416+0000","last_became_peered":"2024-09-13T17:21:36.306416+0000","last_unstale":"2024-09-13T17:21:36.760616+0000","last_undegraded":"2024-09-13T17:21:36.760616+0000","last_fullsized":"2024-09-13T17:21:36.760616+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:11.973871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734007+0000","last_change":"2024-09-13T17:21:36.755992+0000","last_active":"2024-09-13T17:21:37.734007+0000","last_peered":"2024-09-13T17:21:37.734007+0000","last_clean":"2024-09-13T17:21:37.734007+0000","last_became_active":"2024-09-13T17:21:36.753414+0000","last_became_peered":"2024-09-13T17:21:36.753414+0000","last_unstale":"2024-09-13T17:21:37.734007+0000","last_undegraded":"2024-09-13T17:21:37.734007+0000","last_fullsized":"2024-09-13T17:21:37.734007+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:38.521775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987165+0000","last_change":"2024-09-13T17:21:36.308772+0000","last_active":"2024-09-13T17:21:37.987165+0000","last_peered":"2024-09-13T17:21:37.987165+0000","last_clean":"2024-09-13T17:21:37.987165+0000","last_became_active":"2024-09-13T17:21:36.308498+0000","last_became_peered":"2024-09-13T17:21:36.308498+0000","last_unstale":"2024-09-13T17:21:37.987165+0000","last_undegraded":"2024-09-13T17:21:37.987165+0000","last_fullsized":"2024-09-13T17:21:37.987165+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:00.880376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734277+0000","last_change":"2024-09-13T17:21:36.747817+0000","last_active":"2024-09-13T17:21:37.734277+0000","last_peered":"2024-09-13T17:21:37.734277+0000","last_clean":"2024-09-13T17:21:37.734277+0000","last_became_active":"2024-09-13T17:21:36.747554+0000","last_became_peered":"2024-09-13T17:21:36.747554+0000","last_unstale":"2024-09-13T17:21:37.734277+0000","last_undegraded":"2024-09-13T17:21:37.734277+0000","last_fullsized":"2024-09-13T17:21:37.734277+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:54.035880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734259+0000","last_change":"2024-09-13T17:21:36.300595+0000","last_active":"2024-09-13T17:21:37.734259+0000","last_peered":"2024-09-13T17:21:37.734259+0000","last_clean":"2024-09-13T17:21:37.734259+0000","last_became_active":"2024-09-13T17:21:36.300031+0000","last_became_peered":"2024-09-13T17:21:36.300031+0000","last_unstale":"2024-09-13T17:21:37.734259+0000","last_undegraded":"2024-09-13T17:21:37.734259+0000","last_fullsized":"2024-09-13T17:21:37.734259+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:05.872053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.752755+0000","last_change":"2024-09-13T17:21:36.742622+0000","last_active":"2024-09-13T17:21:36.752755+0000","last_peered":"2024-09-13T17:21:36.752755+0000","last_clean":"2024-09-13T17:21:36.752755+0000","last_became_active":"2024-09-13T17:21:36.742357+0000","last_became_peered":"2024-09-13T17:21:36.742357+0000","last_unstale":"2024-09-13T17:21:36.752755+0000","last_undegraded":"2024-09-13T17:21:36.752755+0000","last_fullsized":"2024-09-13T17:21:36.752755+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:53.099300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732354+0000","last_change":"2024-09-13T17:21:36.300292+0000","last_active":"2024-09-13T17:21:37.732354+0000","last_peered":"2024-09-13T17:21:37.732354+0000","last_clean":"2024-09-13T17:21:37.732354+0000","last_became_active":"2024-09-13T17:21:36.300030+0000","last_became_peered":"2024-09-13T17:21:36.300030+0000","last_unstale":"2024-09-13T17:21:37.732354+0000","last_undegraded":"2024-09-13T17:21:37.732354+0000","last_fullsized":"2024-09-13T17:21:37.732354+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:24.398739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732444+0000","last_change":"2024-09-13T17:21:36.756635+0000","last_active":"2024-09-13T17:21:37.732444+0000","last_peered":"2024-09-13T17:21:37.732444+0000","last_clean":"2024-09-13T17:21:37.732444+0000","last_became_active":"2024-09-13T17:21:36.755698+0000","last_became_peered":"2024-09-13T17:21:36.755698+0000","last_unstale":"2024-09-13T17:21:37.732444+0000","last_undegraded":"2024-09-13T17:21:37.732444+0000","last_fullsized":"2024-09-13T17:21:37.732444+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:18.013523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.738731+0000","last_change":"2024-09-13T17:21:36.310694+0000","last_active":"2024-09-13T17:21:36.738731+0000","last_peered":"2024-09-13T17:21:36.738731+0000","last_clean":"2024-09-13T17:21:36.738731+0000","last_became_active":"2024-09-13T17:21:36.309451+0000","last_became_peered":"2024-09-13T17:21:36.309451+0000","last_unstale":"2024-09-13T17:21:36.738731+0000","last_undegraded":"2024-09-13T17:21:36.738731+0000","last_fullsized":"2024-09-13T17:21:36.738731+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:51.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731948+0000","last_change":"2024-09-13T17:21:36.745379+0000","last_active":"2024-09-13T17:21:37.731948+0000","last_peered":"2024-09-13T17:21:37.731948+0000","last_clean":"2024-09-13T17:21:37.731948+0000","last_became_active":"2024-09-13T17:21:36.744448+0000","last_became_peered":"2024-09-13T17:21:36.744448+0000","last_unstale":"2024-09-13T17:21:37.731948+0000","last_undegraded":"2024-09-13T17:21:37.731948+0000","last_fullsized":"2024-09-13T17:21:37.731948+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:38.055703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732238+0000","last_change":"2024-09-13T17:21:36.313327+0000","last_active":"2024-09-13T17:21:37.732238+0000","last_peered":"2024-09-13T17:21:37.732238+0000","last_clean":"2024-09-13T17:21:37.732238+0000","last_became_active":"2024-09-13T17:21:36.313037+0000","last_became_peered":"2024-09-13T17:21:36.313037+0000","last_unstale":"2024-09-13T17:21:37.732238+0000","last_undegraded":"2024-09-13T17:21:37.732238+0000","last_fullsized":"2024-09-13T17:21:37.732238+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:54.858905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.741284+0000","last_change":"2024-09-13T17:21:36.737995+0000","last_active":"2024-09-13T17:21:36.741284+0000","last_peered":"2024-09-13T17:21:36.741284+0000","last_clean":"2024-09-13T17:21:36.741284+0000","last_became_active":"2024-09-13T17:21:36.737725+0000","last_became_peered":"2024-09-13T17:21:36.737725+0000","last_unstale":"2024-09-13T17:21:36.741284+0000","last_undegraded":"2024-09-13T17:21:36.741284+0000","last_fullsized":"2024-09-13T17:21:36.741284+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:40.827453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986524+0000","last_change":"2024-09-13T17:21:36.302171+0000","last_active":"2024-09-13T17:21:37.986524+0000","last_peered":"2024-09-13T17:21:37.986524+0000","last_clean":"2024-09-13T17:21:37.986524+0000","last_became_active":"2024-09-13T17:21:36.301525+0000","last_became_peered":"2024-09-13T17:21:36.301525+0000","last_unstale":"2024-09-13T17:21:37.986524+0000","last_undegraded":"2024-09-13T17:21:37.986524+0000","last_fullsized":"2024-09-13T17:21:37.986524+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:43.526946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733040+0000","last_change":"2024-09-13T17:21:36.738161+0000","last_active":"2024-09-13T17:21:37.733040+0000","last_peered":"2024-09-13T17:21:37.733040+0000","last_clean":"2024-09-13T17:21:37.733040+0000","last_became_active":"2024-09-13T17:21:36.737885+0000","last_became_peered":"2024-09-13T17:21:36.737885+0000","last_unstale":"2024-09-13T17:21:37.733040+0000","last_undegraded":"2024-09-13T17:21:37.733040+0000","last_fullsized":"2024-09-13T17:21:37.733040+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:53.816777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985914+0000","last_change":"2024-09-13T17:21:36.303299+0000","last_active":"2024-09-13T17:21:37.985914+0000","last_peered":"2024-09-13T17:21:37.985914+0000","last_clean":"2024-09-13T17:21:37.985914+0000","last_became_active":"2024-09-13T17:21:36.302858+0000","last_became_peered":"2024-09-13T17:21:36.302858+0000","last_unstale":"2024-09-13T17:21:37.985914+0000","last_undegraded":"2024-09-13T17:21:37.985914+0000","last_fullsized":"2024-09-13T17:21:37.985914+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:04.631923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986827+0000","last_change":"2024-09-13T17:21:36.747167+0000","last_active":"2024-09-13T17:21:37.986827+0000","last_peered":"2024-09-13T17:21:37.986827+0000","last_clean":"2024-09-13T17:21:37.986827+0000","last_became_active":"2024-09-13T17:21:36.746664+0000","last_became_peered":"2024-09-13T17:21:36.746664+0000","last_unstale":"2024-09-13T17:21:37.986827+0000","last_undegraded":"2024-09-13T17:21:37.986827+0000","last_fullsized":"2024-09-13T17:21:37.986827+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:17:16.322908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986376+0000","last_change":"2024-09-13T17:21:36.310792+0000","last_active":"2024-09-13T17:21:37.986376+0000","last_peered":"2024-09-13T17:21:37.986376+0000","last_clean":"2024-09-13T17:21:37.986376+0000","last_became_active":"2024-09-13T17:21:36.310527+0000","last_became_peered":"2024-09-13T17:21:36.310527+0000","last_unstale":"2024-09-13T17:21:37.986376+0000","last_undegraded":"2024-09-13T17:21:37.986376+0000","last_fullsized":"2024-09-13T17:21:37.986376+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:04.726410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986638+0000","last_change":"2024-09-13T17:21:36.747238+0000","last_active":"2024-09-13T17:21:37.986638+0000","last_peered":"2024-09-13T17:21:37.986638+0000","last_clean":"2024-09-13T17:21:37.986638+0000","last_became_active":"2024-09-13T17:21:36.746981+0000","last_became_peered":"2024-09-13T17:21:36.746981+0000","last_unstale":"2024-09-13T17:21:37.986638+0000","last_undegraded":"2024-09-13T17:21:37.986638+0000","last_fullsized":"2024-09-13T17:21:37.986638+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:50.020081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"192'2","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758371+0000","last_change":"2024-09-13T17:21:36.309805+0000","last_active":"2024-09-13T17:21:36.758371+0000","last_peered":"2024-09-13T17:21:36.758371+0000","last_clean":"2024-09-13T17:21:36.758371+0000","last_became_active":"2024-09-13T17:21:36.309140+0000","last_became_peered":"2024-09-13T17:21:36.309140+0000","last_unstale":"2024-09-13T17:21:36.758371+0000","last_undegraded":"2024-09-13T17:21:36.758371+0000","last_fullsized":"2024-09-13T17:21:36.758371+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:11.894842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986219+0000","last_change":"2024-09-13T17:21:36.749941+0000","last_active":"2024-09-13T17:21:37.986219+0000","last_peered":"2024-09-13T17:21:37.986219+0000","last_clean":"2024-09-13T17:21:37.986219+0000","last_became_active":"2024-09-13T17:21:36.749595+0000","last_became_peered":"2024-09-13T17:21:36.749595+0000","last_unstale":"2024-09-13T17:21:37.986219+0000","last_undegraded":"2024-09-13T17:21:37.986219+0000","last_fullsized":"2024-09-13T17:21:37.986219+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:28.804370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734108+0000","last_change":"2024-09-13T17:21:36.309579+0000","last_active":"2024-09-13T17:21:37.734108+0000","last_peered":"2024-09-13T17:21:37.734108+0000","last_clean":"2024-09-13T17:21:37.734108+0000","last_became_active":"2024-09-13T17:21:36.308175+0000","last_became_peered":"2024-09-13T17:21:36.308175+0000","last_unstale":"2024-09-13T17:21:37.734108+0000","last_undegraded":"2024-09-13T17:21:37.734108+0000","last_fullsized":"2024-09-13T17:21:37.734108+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:52.369393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733756+0000","last_change":"2024-09-13T17:21:36.736311+0000","last_active":"2024-09-13T17:21:37.733756+0000","last_peered":"2024-09-13T17:21:37.733756+0000","last_clean":"2024-09-13T17:21:37.733756+0000","last_became_active":"2024-09-13T17:21:36.735759+0000","last_became_peered":"2024-09-13T17:21:36.735759+0000","last_unstale":"2024-09-13T17:21:37.733756+0000","last_undegraded":"2024-09-13T17:21:37.733756+0000","last_fullsized":"2024-09-13T17:21:37.733756+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:03.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986889+0000","last_change":"2024-09-13T17:21:36.313125+0000","last_active":"2024-09-13T17:21:37.986889+0000","last_peered":"2024-09-13T17:21:37.986889+0000","last_clean":"2024-09-13T17:21:37.986889+0000","last_became_active":"2024-09-13T17:21:36.312821+0000","last_became_peered":"2024-09-13T17:21:36.312821+0000","last_unstale":"2024-09-13T17:21:37.986889+0000","last_undegraded":"2024-09-13T17:21:37.986889+0000","last_fullsized":"2024-09-13T17:21:37.986889+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:01.963163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987209+0000","last_change":"2024-09-13T17:21:36.750620+0000","last_active":"2024-09-13T17:21:37.987209+0000","last_peered":"2024-09-13T17:21:37.987209+0000","last_clean":"2024-09-13T17:21:37.987209+0000","last_became_active":"2024-09-13T17:21:36.750347+0000","last_became_peered":"2024-09-13T17:21:36.750347+0000","last_unstale":"2024-09-13T17:21:37.987209+0000","last_undegraded":"2024-09-13T17:21:37.987209+0000","last_fullsized":"2024-09-13T17:21:37.987209+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:34.814185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.738823+0000","last_change":"2024-09-13T17:21:36.305629+0000","last_active":"2024-09-13T17:21:36.738823+0000","last_peered":"2024-09-13T17:21:36.738823+0000","last_clean":"2024-09-13T17:21:36.738823+0000","last_became_active":"2024-09-13T17:21:36.305358+0000","last_became_peered":"2024-09-13T17:21:36.305358+0000","last_unstale":"2024-09-13T17:21:36.738823+0000","last_undegraded":"2024-09-13T17:21:36.738823+0000","last_fullsized":"2024-09-13T17:21:36.738823+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:38.635179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986430+0000","last_change":"2024-09-13T17:21:36.736928+0000","last_active":"2024-09-13T17:21:37.986430+0000","last_peered":"2024-09-13T17:21:37.986430+0000","last_clean":"2024-09-13T17:21:37.986430+0000","last_became_active":"2024-09-13T17:21:36.736670+0000","last_became_peered":"2024-09-13T17:21:36.736670+0000","last_unstale":"2024-09-13T17:21:37.986430+0000","last_undegraded":"2024-09-13T17:21:37.986430+0000","last_fullsized":"2024-09-13T17:21:37.986430+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:56.819148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734158+0000","last_change":"2024-09-13T17:21:36.313511+0000","last_active":"2024-09-13T17:21:37.734158+0000","last_peered":"2024-09-13T17:21:37.734158+0000","last_clean":"2024-09-13T17:21:37.734158+0000","last_became_active":"2024-09-13T17:21:36.313126+0000","last_became_peered":"2024-09-13T17:21:36.313126+0000","last_unstale":"2024-09-13T17:21:37.734158+0000","last_undegraded":"2024-09-13T17:21:37.734158+0000","last_fullsized":"2024-09-13T17:21:37.734158+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:18.037991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986673+0000","last_change":"2024-09-13T17:21:36.751622+0000","last_active":"2024-09-13T17:21:37.986673+0000","last_peered":"2024-09-13T17:21:37.986673+0000","last_clean":"2024-09-13T17:21:37.986673+0000","last_became_active":"2024-09-13T17:21:36.751280+0000","last_became_peered":"2024-09-13T17:21:36.751280+0000","last_unstale":"2024-09-13T17:21:37.986673+0000","last_undegraded":"2024-09-13T17:21:37.986673+0000","last_fullsized":"2024-09-13T17:21:37.986673+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:54.035880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986758+0000","last_change":"2024-09-13T17:21:36.311612+0000","last_active":"2024-09-13T17:21:37.986758+0000","last_peered":"2024-09-13T17:21:37.986758+0000","last_clean":"2024-09-13T17:21:37.986758+0000","last_became_active":"2024-09-13T17:21:36.311312+0000","last_became_peered":"2024-09-13T17:21:36.311312+0000","last_unstale":"2024-09-13T17:21:37.986758+0000","last_undegraded":"2024-09-13T17:21:37.986758+0000","last_fullsized":"2024-09-13T17:21:37.986758+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:00.342646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757882+0000","last_change":"2024-09-13T17:21:36.749577+0000","last_active":"2024-09-13T17:21:36.757882+0000","last_peered":"2024-09-13T17:21:36.757882+0000","last_clean":"2024-09-13T17:21:36.757882+0000","last_became_active":"2024-09-13T17:21:36.748510+0000","last_became_peered":"2024-09-13T17:21:36.748510+0000","last_unstale":"2024-09-13T17:21:36.757882+0000","last_undegraded":"2024-09-13T17:21:36.757882+0000","last_fullsized":"2024-09-13T17:21:36.757882+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:45.209455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986943+0000","last_change":"2024-09-13T17:21:36.306469+0000","last_active":"2024-09-13T17:21:37.986943+0000","last_peered":"2024-09-13T17:21:37.986943+0000","last_clean":"2024-09-13T17:21:37.986943+0000","last_became_active":"2024-09-13T17:21:36.306116+0000","last_became_peered":"2024-09-13T17:21:36.306116+0000","last_unstale":"2024-09-13T17:21:37.986943+0000","last_undegraded":"2024-09-13T17:21:37.986943+0000","last_fullsized":"2024-09-13T17:21:37.986943+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:57.347405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733243+0000","last_change":"2024-09-13T17:21:36.744596+0000","last_active":"2024-09-13T17:21:37.733243+0000","last_peered":"2024-09-13T17:21:37.733243+0000","last_clean":"2024-09-13T17:21:37.733243+0000","last_became_active":"2024-09-13T17:21:36.744274+0000","last_became_peered":"2024-09-13T17:21:36.744274+0000","last_unstale":"2024-09-13T17:21:37.733243+0000","last_undegraded":"2024-09-13T17:21:37.733243+0000","last_fullsized":"2024-09-13T17:21:37.733243+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:51.528433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986244+0000","last_change":"2024-09-13T17:21:36.306340+0000","last_active":"2024-09-13T17:21:37.986244+0000","last_peered":"2024-09-13T17:21:37.986244+0000","last_clean":"2024-09-13T17:21:37.986244+0000","last_became_active":"2024-09-13T17:21:36.305674+0000","last_became_peered":"2024-09-13T17:21:36.305674+0000","last_unstale":"2024-09-13T17:21:37.986244+0000","last_undegraded":"2024-09-13T17:21:37.986244+0000","last_fullsized":"2024-09-13T17:21:37.986244+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:05.872053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986327+0000","last_change":"2024-09-13T17:21:36.745187+0000","last_active":"2024-09-13T17:21:37.986327+0000","last_peered":"2024-09-13T17:21:37.986327+0000","last_clean":"2024-09-13T17:21:37.986327+0000","last_became_active":"2024-09-13T17:21:36.744577+0000","last_became_peered":"2024-09-13T17:21:36.744577+0000","last_unstale":"2024-09-13T17:21:37.986327+0000","last_undegraded":"2024-09-13T17:21:37.986327+0000","last_fullsized":"2024-09-13T17:21:37.986327+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:18.013523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986763+0000","last_change":"2024-09-13T17:21:36.313777+0000","last_active":"2024-09-13T17:21:37.986763+0000","last_peered":"2024-09-13T17:21:37.986763+0000","last_clean":"2024-09-13T17:21:37.986763+0000","last_became_active":"2024-09-13T17:21:36.313340+0000","last_became_peered":"2024-09-13T17:21:36.313340+0000","last_unstale":"2024-09-13T17:21:37.986763+0000","last_undegraded":"2024-09-13T17:21:37.986763+0000","last_fullsized":"2024-09-13T17:21:37.986763+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:39.820161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.741338+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:21:36.741338+0000","last_peered":"2024-09-13T17:21:36.741338+0000","last_clean":"2024-09-13T17:21:36.741338+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:21:36.741338+0000","last_undegraded":"2024-09-13T17:21:36.741338+0000","last_fullsized":"2024-09-13T17:21:36.741338+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70120,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301720,"statfs":{"total":96636764160,"available":96564961280,"internally_reserved":0,"allocated":3035136,"data_stored":1665337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":1.008,"5min":1.008,"15min":2.0880000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65600000000000003,"15min":0.67300000000000004},"min":{"1min":0.47599999999999998,"5min":0.373,"15min":0.316},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.79200000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.66100000000000003},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.161,"5min":1.161,"15min":2.5030000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64200000000000002,"15min":0.69799999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.399,"5min":1.399,"15min":6.4909999999999997},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.105,"5min":1.169,"15min":1.544},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.69399999999999995},"min":{"1min":0.504,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.605},"last":0.74399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.65700000000000003,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69499999999999995,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.2649999999999999,"5min":1.274,"15min":2.8239999999999998},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.44600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2170000000000001,"5min":1.234,"15min":1.234},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.623,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.143},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.90800000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.73599999999999999},"min":{"1min":0.52200000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.94499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65420,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306420,"statfs":{"total":96636764160,"available":96569774080,"internally_reserved":0,"allocated":2420736,"data_stored":1067687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.499,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.95299999999999996,"5min":1.0209999999999999,"15min":3.6629999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.73599999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.97399999999999998,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.78600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.65300000000000002},"min":{"1min":0.443,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":3.7040000000000002},"last":0.626},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.45500000000000002,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.90700000000000003,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.72099999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.53700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64000000000000001,"15min":0.68200000000000005},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.93400000000000005,"15min":3.2629999999999999},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.53200000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94899999999999995,"5min":1.167,"15min":3.6520000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.66400000000000003,"15min":0.69999999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86399999999999999,"5min":0.91100000000000003,"15min":3.742},"last":0.746}]},{"osd":4,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.5,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.6439999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.29199999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":4.7750000000000004},"last":0.75600000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.91000000000000003,"5min":1.304,"15min":2.3700000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63100000000000001,"15min":0.67100000000000004},"min":{"1min":0.39200000000000002,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":3.609},"last":0.66900000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68700000000000006,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.93400000000000005,"5min":0.93700000000000006,"15min":4.9669999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.754,"15min":0.78500000000000003},"min":{"1min":0.60599999999999998,"5min":0.52300000000000002,"15min":0.46800000000000003},"max":{"1min":1.016,"5min":1.238,"15min":3.8100000000000001},"last":0.84599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59728,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312112,"statfs":{"total":96636764160,"available":96575602688,"internally_reserved":0,"allocated":2420736,"data_stored":1067687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64300000000000002,"15min":0.76600000000000001},"min":{"1min":0.53800000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81100000000000005,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.65600000000000003,"15min":0.73699999999999999},"min":{"1min":0.44600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90700000000000003,"5min":0.91800000000000004,"15min":21.376999999999999},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.72099999999999997},"min":{"1min":0.41999999999999998,"5min":0.38800000000000001,"15min":0.30499999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.874},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67400000000000004,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":0.91500000000000004,"5min":1.482,"15min":7.7370000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.73599999999999999},"min":{"1min":0.505,"5min":0.379,"15min":0.379},"max":{"1min":0.88800000000000001,"5min":1.0489999999999999,"15min":17.268000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66100000000000003,"15min":0.72599999999999998},"min":{"1min":0.47699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.76000000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68600000000000005,"15min":1.083},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":0.91200000000000003,"5min":0.95099999999999996,"15min":113.363},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.45800000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.88100000000000001,"5min":1.113,"15min":4.1180000000000003},"last":0.95199999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":1.0760000000000001,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64100000000000001,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":1.03,"15min":2.827},"last":0.89800000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65800000000000003,"15min":0.69099999999999995},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.871,"5min":1.73,"15min":2.2320000000000002},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.71499999999999997},"min":{"1min":0.47099999999999997,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.0249999999999999,"5min":1.532,"15min":4.1500000000000004},"last":0.92600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.751},"min":{"1min":0.52900000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":1.0700000000000001,"15min":4.1029999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73499999999999999,"15min":0.75600000000000001},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.97899999999999998,"5min":1.363,"15min":2.02},"last":0.91400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575014,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73632,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298208,"statfs":{"total":96636764160,"available":96561364992,"internally_reserved":0,"allocated":2437120,"data_stored":1068773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":1.165,"15min":0.82399999999999995},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":46.860999999999997,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.498},{"interface":"front","average":{"1min":3.1190000000000002,"5min":1.1739999999999999,"15min":0.874},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":45.140000000000001,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":0.83299999999999996,"15min":0.71699999999999997},"min":{"1min":0.442,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":18.071000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.70399999999999996,"15min":0.66200000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":8.3320000000000007,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":1.071,"15min":0.80900000000000005},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":36.725999999999999,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.749,"15min":0.71799999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.70299999999999996,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.63600000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.41699999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":0.625},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.77700000000000002,"15min":0.74299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.78200000000000003,"15min":0.752},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575015,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69540,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302300,"statfs":{"total":96636764160,"available":96565555200,"internally_reserved":0,"allocated":2441216,"data_stored":1073587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41999999999999998,"15min":0.434},"min":{"1min":0.28999999999999998,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":0.496,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45700000000000002,"15min":0.48899999999999999},"min":{"1min":0.30599999999999999,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":2.96},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47599999999999998,"15min":0.505},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59999999999999998,"5min":0.69099999999999995,"15min":3.5600000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.61399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.61899999999999999,"5min":0.79200000000000004,"15min":3.6829999999999998},"last":0.495}]},{"osd":2,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.84999999999999998,"15min":1.9590000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52800000000000002,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.34499999999999997},"max":{"1min":0.57699999999999996,"5min":0.77500000000000002,"15min":1.3819999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":0.90400000000000003,"5min":1.21,"15min":2.8610000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.755},"min":{"1min":0.51200000000000001,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.90800000000000003,"5min":1.23,"15min":1.3640000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91200000000000003,"5min":1.323,"15min":1.391},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72499999999999998,"15min":0.76500000000000001},"min":{"1min":0.503,"5min":0.442,"15min":0.442},"max":{"1min":0.83799999999999997,"5min":1.224,"15min":2.9300000000000002},"last":0.67000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.52900000000000003,"5min":0.48699999999999999,"15min":0.47199999999999998},"max":{"1min":0.90900000000000003,"5min":1.21,"15min":1.401},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.77800000000000002},"min":{"1min":0.57699999999999996,"5min":0.53000000000000003,"15min":0.46300000000000002},"max":{"1min":0.88100000000000001,"5min":1.25,"15min":1.3300000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":0.79600000000000004},"min":{"1min":0.53700000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93400000000000005,"5min":1.27,"15min":2.899},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77300000000000002,"15min":0.79500000000000004},"min":{"1min":0.55400000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.90100000000000002,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.878}]}]},{"osd":2,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47428,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324412,"statfs":{"total":96636764160,"available":96588197888,"internally_reserved":0,"allocated":2412544,"data_stored":1066086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.46999999999999997,"15min":0.435},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.621,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.31},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45100000000000001,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.623,"5min":1.593,"15min":30.023},"last":0.432}]},{"osd":1,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53400000000000003,"15min":1.365},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.52100000000000002,"15min":1.4610000000000001},"min":{"1min":0.34999999999999998,"5min":0.32500000000000001,"15min":0.316},"max":{"1min":0.73499999999999999,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.38}]},{"osd":3,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53600000000000003,"15min":0.61599999999999999},"min":{"1min":0.33400000000000002,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.76500000000000001,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65400000000000003,"15min":0.60599999999999998},"min":{"1min":0.45000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88400000000000001,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.80800000000000005},"min":{"1min":0.55100000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94899999999999995,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.78900000000000003,"15min":0.85899999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.371},"max":{"1min":0.91200000000000003,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.878},"min":{"1min":0.54700000000000004,"5min":0.44500000000000001,"15min":0.376},"max":{"1min":0.89900000000000002,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.78900000000000003,"15min":0.77900000000000003},"min":{"1min":0.52400000000000002,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":0.93700000000000006,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.79600000000000004,"15min":0.81899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":6.4800000000000004,"15min":10.664},"last":0.626},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78300000000000003,"15min":0.89100000000000001},"min":{"1min":0.53500000000000003,"5min":0.503,"15min":0.47999999999999998},"max":{"1min":0.97999999999999998,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.83299999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.77400000000000002,"15min":0.82099999999999995},"min":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.752},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.79900000000000004,"15min":0.90400000000000003},"min":{"1min":0.52300000000000002,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":0.95699999999999996,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91668,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280172,"statfs":{"total":96636764160,"available":96542896128,"internally_reserved":0,"allocated":2424832,"data_stored":1067161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.434,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":5.4939999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48699999999999999,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":2.3119999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57699999999999996,"15min":0.65800000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":16.382999999999999},"last":0.5},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53400000000000003,"15min":0.60299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81299999999999994,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.442}]},{"osd":4,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.76000000000000001,"15min":0.78900000000000003},"min":{"1min":0.47199999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.28,"5min":1.28,"15min":6.1630000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.56699999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.78},"last":0.86699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.76800000000000002},"min":{"1min":0.52200000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.159,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.75800000000000001,"15min":0.78400000000000003},"min":{"1min":0.50600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.175,"5min":1.175,"15min":6.1189999999999998},"last":0.747}]},{"osd":6,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.77500000000000002,"15min":0.80400000000000005},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.246,"5min":1.246,"15min":6.1970000000000001},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.58999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1359999999999999,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.80200000000000005,"15min":0.81200000000000006},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.76400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575016,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70092,"kb_used_data":2940,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301748,"statfs":{"total":96636764160,"available":96564989952,"internally_reserved":0,"allocated":3010560,"data_stored":1656544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.432,"15min":0.51400000000000001},"min":{"1min":0.27600000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":1.986,"15min":5.5259999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.48599999999999999,"15min":0.55200000000000005},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.56100000000000005,"5min":2.371,"15min":5.4889999999999999},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.57599999999999996},"min":{"1min":0.33100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.61199999999999999,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52100000000000002,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.60999999999999999,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.434}]},{"osd":3,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47799999999999998,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.63600000000000001,"5min":0.878,"15min":4.7549999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.61699999999999999},"min":{"1min":0.39000000000000001,"5min":0.375,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.76400000000000001,"15min":0.81100000000000005},"min":{"1min":0.53100000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.079,"5min":3.8849999999999998,"15min":4.734},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72599999999999998,"15min":0.80000000000000004},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0009999999999999,"5min":1.3109999999999999,"15min":4.8079999999999998},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73599999999999999,"15min":0.81100000000000005},"min":{"1min":0.51600000000000001,"5min":0.47799999999999998,"15min":0.44500000000000001},"max":{"1min":1.0980000000000001,"5min":1.1839999999999999,"15min":4.944},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76300000000000001,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.139,"5min":1.234,"15min":4.9710000000000001},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.76400000000000001,"15min":0.86099999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0489999999999999,"5min":1.161,"15min":4.8470000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.79200000000000004,"15min":0.83699999999999997},"min":{"1min":0.50600000000000001,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.133,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83099999999999996,"15min":0.86299999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.1080000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.84299999999999997},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.0649999999999999,"5min":1.2649999999999999,"15min":4.9950000000000001},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:21:40.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:21:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.124+0000 7f62a6800640 1 Processor -- start 2024-09-13T17:21:41.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.124+0000 7f62a6800640 1 -- start start 2024-09-13T17:21:41.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.125+0000 7f62a6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f62a6800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62a006c2d0 0x7f62a006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f62a6800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a0159d90 con 0x7f62a01592a0 2024-09-13T17:21:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f62a6800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a015b0b0 con 0x7f62a006c2d0 2024-09-13T17:21:41.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f62a6800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a006cca0 con 0x7f62a006f970 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f629fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62a006c2d0 0x7f62a006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f6296a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f629fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f62a006c2d0 0x7f62a006c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:53468/0 (socket says 172.21.15.28:53468) 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f629f400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f62a01592a0 0x7f62a0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:48532/0 (socket says 172.21.15.28:48532) 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f6296a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55254/0 (socket says 172.21.15.28:55254) 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.126+0000 7f629fe00640 1 -- 172.21.15.28:0/2925922194 learned_addr learned my addr 172.21.15.28:0/2925922194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f629fe00640 1 -- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 msgr2=0x7f62a015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:41.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f629fe00640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f629fe00640 1 -- 172.21.15.28:0/2925922194 >> v1:172.21.15.189:6789/0 conn(0x7f62a01592a0 legacy=0x7f62a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f629ea00640 1 -- 172.21.15.28:0/2925922194 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234743006 0 0) 0x7f62a0159d90 con 0x7f62a01592a0 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f629fe00640 1 -- 172.21.15.28:0/2925922194 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a00b2040 con 0x7f62a006c2d0 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.127+0000 7f6296a00640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.128+0000 7f629fe00640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a006c2d0 0x7f62a006c760 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6298009e50 tx=0x7f629802a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92d49268b8606f82 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.128+0000 7f629ea00640 1 -- 172.21.15.28:0/2925922194 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298031600 con 0x7f62a006c2d0 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.128+0000 7f629ea00640 1 -- 172.21.15.28:0/2925922194 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629802d050 con 0x7f62a006c2d0 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.128+0000 7f629ea00640 1 -- 172.21.15.28:0/2925922194 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6298031980 con 0x7f62a006c2d0 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 -- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a006c2d0 msgr2=0x7f62a006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a006c2d0 0x7f62a006c760 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f6298009e50 tx=0x7f629802a710 comp rx=0 tx=0).stop 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 -- 172.21.15.28:0/2925922194 shutdown_connections 2024-09-13T17:21:41.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a006c2d0 0x7f62a006c760 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 --2- 172.21.15.28:0/2925922194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006f970 0x7f62a015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 -- 172.21.15.28:0/2925922194 >> 172.21.15.28:0/2925922194 conn(0x7f62a0069e80 msgr2=0x7f62a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 -- 172.21.15.28:0/2925922194 shutdown_connections 2024-09-13T17:21:41.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.129+0000 7f62a6800640 1 -- 172.21.15.28:0/2925922194 wait complete. 2024-09-13T17:21:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.131+0000 7f62a6800640 1 Processor -- start 2024-09-13T17:21:41.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.131+0000 7f62a6800640 1 -- start start 2024-09-13T17:21:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006c2d0 0x7f62a01ef9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a01592a0 0x7f62a00b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01eff00 con 0x7f62a006f970 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00cc510 con 0x7f62a01592a0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f62a6800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00cb900 con 0x7f62a006c2d0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f6296a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f6296a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39960/0 (socket says 172.21.15.28:39960) 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.132+0000 7f6296a00640 1 -- 172.21.15.28:0/3824429328 learned_addr learned my addr 172.21.15.28:0/3824429328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f629fe00640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a01592a0 0x7f62a00b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f629f400640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006c2d0 0x7f62a01ef9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 -- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006c2d0 msgr2=0x7f62a01ef9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006c2d0 0x7f62a01ef9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 -- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a01592a0 msgr2=0x7f62a00b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a01592a0 0x7f62a00b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 -- 172.21.15.28:0/3824429328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6288009400 con 0x7f62a006f970 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6296a00640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f628800ec10 tx=0x7f628800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288009760 con 0x7f62a006f970 2024-09-13T17:21:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628800cc30 con 0x7f62a006f970 2024-09-13T17:21:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6298009b30 con 0x7f62a006f970 2024-09-13T17:21:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6288013470 con 0x7f62a006f970 2024-09-13T17:21:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.133+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a00cc1a0 con 0x7f62a006f970 2024-09-13T17:21:41.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.135+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a00af550 con 0x7f62a006f970 2024-09-13T17:21:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.135+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f62880135d0 con 0x7f62a006f970 2024-09-13T17:21:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.136+0000 7f6297e00640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 0x7f627c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.136+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6288089680 con 0x7f62a006f970 2024-09-13T17:21:41.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.137+0000 7f629f400640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 0x7f627c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.138+0000 7f629f400640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 0x7f627c06d0c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6290007fd0 tx=0x7f6290007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.141+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f62a00af550 con 0x7f62a006f970 2024-09-13T17:21:41.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.319+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62a015a5b0 con 0x7f627c06ac00 2024-09-13T17:21:41.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.327+0000 7f6297e00640 1 -- 172.21.15.28:0/3824429328 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289504 (secure 0 0 0) 0x7f6290007270 con 0x7f627c06ac00 2024-09-13T17:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:41.334 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:21:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 msgr2=0x7f627c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 0x7f627c06d0c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6290007fd0 tx=0x7f6290007c20 comp rx=0 tx=0).stop 2024-09-13T17:21:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 msgr2=0x7f62a0150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f628800ec10 tx=0x7f628800c520 comp rx=0 tx=0).stop 2024-09-13T17:21:41.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 shutdown_connections 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f627c06ac00 0x7f627c06d0c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f62a01592a0 0x7f62a00b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f62a006f970 0x7f62a0150f40 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 --2- 172.21.15.28:0/3824429328 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f62a006c2d0 0x7f62a01ef9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 >> 172.21.15.28:0/3824429328 conn(0x7f62a0069e80 msgr2=0x7f62a015c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 shutdown_connections 2024-09-13T17:21:41.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.331+0000 7f62a6800640 1 -- 172.21.15.28:0/3824429328 wait complete. 2024-09-13T17:21:41.349 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-13T17:21:41.089261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":113,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":547628,"kb_used_data":20120,"kb_used_omap":12,"kb_used_meta":527347,"kb_avail":754427092,"statfs":{"total":773094113280,"available":772533342208,"internally_reserved":0,"allocated":20602880,"data_stored":9732862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":540003897},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.802595"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986010+0000","last_change":"2024-09-13T17:21:36.756178+0000","last_active":"2024-09-13T17:21:37.986010+0000","last_peered":"2024-09-13T17:21:37.986010+0000","last_clean":"2024-09-13T17:21:37.986010+0000","last_became_active":"2024-09-13T17:21:36.755914+0000","last_became_peered":"2024-09-13T17:21:36.755914+0000","last_unstale":"2024-09-13T17:21:37.986010+0000","last_undegraded":"2024-09-13T17:21:37.986010+0000","last_fullsized":"2024-09-13T17:21:37.986010+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:38.898990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733821+0000","last_change":"2024-09-13T17:21:36.736750+0000","last_active":"2024-09-13T17:21:37.733821+0000","last_peered":"2024-09-13T17:21:37.733821+0000","last_clean":"2024-09-13T17:21:37.733821+0000","last_became_active":"2024-09-13T17:21:36.736484+0000","last_became_peered":"2024-09-13T17:21:36.736484+0000","last_unstale":"2024-09-13T17:21:37.733821+0000","last_undegraded":"2024-09-13T17:21:37.733821+0000","last_fullsized":"2024-09-13T17:21:37.733821+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:29.353327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758339+0000","last_change":"2024-09-13T17:21:36.754492+0000","last_active":"2024-09-13T17:21:36.758339+0000","last_peered":"2024-09-13T17:21:36.758339+0000","last_clean":"2024-09-13T17:21:36.758339+0000","last_became_active":"2024-09-13T17:21:36.754237+0000","last_became_peered":"2024-09-13T17:21:36.754237+0000","last_unstale":"2024-09-13T17:21:36.758339+0000","last_undegraded":"2024-09-13T17:21:36.758339+0000","last_fullsized":"2024-09-13T17:21:36.758339+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:36.400778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733591+0000","last_change":"2024-09-13T17:21:36.752368+0000","last_active":"2024-09-13T17:21:37.733591+0000","last_peered":"2024-09-13T17:21:37.733591+0000","last_clean":"2024-09-13T17:21:37.733591+0000","last_became_active":"2024-09-13T17:21:36.750473+0000","last_became_peered":"2024-09-13T17:21:36.750473+0000","last_unstale":"2024-09-13T17:21:37.733591+0000","last_undegraded":"2024-09-13T17:21:37.733591+0000","last_fullsized":"2024-09-13T17:21:37.733591+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:08.997780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987041+0000","last_change":"2024-09-13T17:21:36.753080+0000","last_active":"2024-09-13T17:21:37.987041+0000","last_peered":"2024-09-13T17:21:37.987041+0000","last_clean":"2024-09-13T17:21:37.987041+0000","last_became_active":"2024-09-13T17:21:36.751006+0000","last_became_peered":"2024-09-13T17:21:36.751006+0000","last_unstale":"2024-09-13T17:21:37.987041+0000","last_undegraded":"2024-09-13T17:21:37.987041+0000","last_fullsized":"2024-09-13T17:21:37.987041+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:11.944621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986585+0000","last_change":"2024-09-13T17:21:36.737539+0000","last_active":"2024-09-13T17:21:37.986585+0000","last_peered":"2024-09-13T17:21:37.986585+0000","last_clean":"2024-09-13T17:21:37.986585+0000","last_became_active":"2024-09-13T17:21:36.736183+0000","last_became_peered":"2024-09-13T17:21:36.736183+0000","last_unstale":"2024-09-13T17:21:37.986585+0000","last_undegraded":"2024-09-13T17:21:37.986585+0000","last_fullsized":"2024-09-13T17:21:37.986585+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:53.816777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985683+0000","last_change":"2024-09-13T17:21:36.740091+0000","last_active":"2024-09-13T17:21:37.985683+0000","last_peered":"2024-09-13T17:21:37.985683+0000","last_clean":"2024-09-13T17:21:37.985683+0000","last_became_active":"2024-09-13T17:21:36.739810+0000","last_became_peered":"2024-09-13T17:21:36.739810+0000","last_unstale":"2024-09-13T17:21:37.985683+0000","last_undegraded":"2024-09-13T17:21:37.985683+0000","last_fullsized":"2024-09-13T17:21:37.985683+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:11.949573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986295+0000","last_change":"2024-09-13T17:21:36.736763+0000","last_active":"2024-09-13T17:21:37.986295+0000","last_peered":"2024-09-13T17:21:37.986295+0000","last_clean":"2024-09-13T17:21:37.986295+0000","last_became_active":"2024-09-13T17:21:36.736536+0000","last_became_peered":"2024-09-13T17:21:36.736536+0000","last_unstale":"2024-09-13T17:21:37.986295+0000","last_undegraded":"2024-09-13T17:21:37.986295+0000","last_fullsized":"2024-09-13T17:21:37.986295+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:29.353327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734378+0000","last_change":"2024-09-13T17:21:36.754642+0000","last_active":"2024-09-13T17:21:37.734378+0000","last_peered":"2024-09-13T17:21:37.734378+0000","last_clean":"2024-09-13T17:21:37.734378+0000","last_became_active":"2024-09-13T17:21:36.752872+0000","last_became_peered":"2024-09-13T17:21:36.752872+0000","last_unstale":"2024-09-13T17:21:37.734378+0000","last_undegraded":"2024-09-13T17:21:37.734378+0000","last_fullsized":"2024-09-13T17:21:37.734378+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:42.168423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732088+0000","last_change":"2024-09-13T17:21:36.745486+0000","last_active":"2024-09-13T17:21:37.732088+0000","last_peered":"2024-09-13T17:21:37.732088+0000","last_clean":"2024-09-13T17:21:37.732088+0000","last_became_active":"2024-09-13T17:21:36.745240+0000","last_became_peered":"2024-09-13T17:21:36.745240+0000","last_unstale":"2024-09-13T17:21:37.732088+0000","last_undegraded":"2024-09-13T17:21:37.732088+0000","last_fullsized":"2024-09-13T17:21:37.732088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:16.541840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757806+0000","last_change":"2024-09-13T17:21:36.751897+0000","last_active":"2024-09-13T17:21:36.757806+0000","last_peered":"2024-09-13T17:21:36.757806+0000","last_clean":"2024-09-13T17:21:36.757806+0000","last_became_active":"2024-09-13T17:21:36.751635+0000","last_became_peered":"2024-09-13T17:21:36.751635+0000","last_unstale":"2024-09-13T17:21:36.757806+0000","last_undegraded":"2024-09-13T17:21:36.757806+0000","last_fullsized":"2024-09-13T17:21:36.757806+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:06.879345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986547+0000","last_change":"2024-09-13T17:21:36.748467+0000","last_active":"2024-09-13T17:21:37.986547+0000","last_peered":"2024-09-13T17:21:37.986547+0000","last_clean":"2024-09-13T17:21:37.986547+0000","last_became_active":"2024-09-13T17:21:36.748162+0000","last_became_peered":"2024-09-13T17:21:36.748162+0000","last_unstale":"2024-09-13T17:21:37.986547+0000","last_undegraded":"2024-09-13T17:21:37.986547+0000","last_fullsized":"2024-09-13T17:21:37.986547+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:37:42.429017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986709+0000","last_change":"2024-09-13T17:21:36.743612+0000","last_active":"2024-09-13T17:21:37.986709+0000","last_peered":"2024-09-13T17:21:37.986709+0000","last_clean":"2024-09-13T17:21:37.986709+0000","last_became_active":"2024-09-13T17:21:36.743060+0000","last_became_peered":"2024-09-13T17:21:36.743060+0000","last_unstale":"2024-09-13T17:21:37.986709+0000","last_undegraded":"2024-09-13T17:21:37.986709+0000","last_fullsized":"2024-09-13T17:21:37.986709+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:05.733702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985726+0000","last_change":"2024-09-13T17:21:36.743186+0000","last_active":"2024-09-13T17:21:37.985726+0000","last_peered":"2024-09-13T17:21:37.985726+0000","last_clean":"2024-09-13T17:21:37.985726+0000","last_became_active":"2024-09-13T17:21:36.742943+0000","last_became_peered":"2024-09-13T17:21:36.742943+0000","last_unstale":"2024-09-13T17:21:37.985726+0000","last_undegraded":"2024-09-13T17:21:37.985726+0000","last_fullsized":"2024-09-13T17:21:37.985726+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:51.986633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986042+0000","last_change":"2024-09-13T17:21:36.746435+0000","last_active":"2024-09-13T17:21:37.986042+0000","last_peered":"2024-09-13T17:21:37.986042+0000","last_clean":"2024-09-13T17:21:37.986042+0000","last_became_active":"2024-09-13T17:21:36.744928+0000","last_became_peered":"2024-09-13T17:21:36.744928+0000","last_unstale":"2024-09-13T17:21:37.986042+0000","last_undegraded":"2024-09-13T17:21:37.986042+0000","last_fullsized":"2024-09-13T17:21:37.986042+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:51:13.412833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731689+0000","last_change":"2024-09-13T17:21:36.750007+0000","last_active":"2024-09-13T17:21:37.731689+0000","last_peered":"2024-09-13T17:21:37.731689+0000","last_clean":"2024-09-13T17:21:37.731689+0000","last_became_active":"2024-09-13T17:21:36.744658+0000","last_became_peered":"2024-09-13T17:21:36.744658+0000","last_unstale":"2024-09-13T17:21:37.731689+0000","last_undegraded":"2024-09-13T17:21:37.731689+0000","last_fullsized":"2024-09-13T17:21:37.731689+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:38.898990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986133+0000","last_change":"2024-09-13T17:21:36.756978+0000","last_active":"2024-09-13T17:21:37.986133+0000","last_peered":"2024-09-13T17:21:37.986133+0000","last_clean":"2024-09-13T17:21:37.986133+0000","last_became_active":"2024-09-13T17:21:36.756574+0000","last_became_peered":"2024-09-13T17:21:36.756574+0000","last_unstale":"2024-09-13T17:21:37.986133+0000","last_undegraded":"2024-09-13T17:21:37.986133+0000","last_fullsized":"2024-09-13T17:21:37.986133+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:22.135159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732088+0000","last_change":"2024-09-13T17:21:36.747673+0000","last_active":"2024-09-13T17:21:37.732088+0000","last_peered":"2024-09-13T17:21:37.732088+0000","last_clean":"2024-09-13T17:21:37.732088+0000","last_became_active":"2024-09-13T17:21:36.747233+0000","last_became_peered":"2024-09-13T17:21:36.747233+0000","last_unstale":"2024-09-13T17:21:37.732088+0000","last_undegraded":"2024-09-13T17:21:37.732088+0000","last_fullsized":"2024-09-13T17:21:37.732088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:28.804370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732167+0000","last_change":"2024-09-13T17:21:36.750165+0000","last_active":"2024-09-13T17:21:37.732167+0000","last_peered":"2024-09-13T17:21:37.732167+0000","last_clean":"2024-09-13T17:21:37.732167+0000","last_became_active":"2024-09-13T17:21:36.744559+0000","last_became_peered":"2024-09-13T17:21:36.744559+0000","last_unstale":"2024-09-13T17:21:37.732167+0000","last_undegraded":"2024-09-13T17:21:37.732167+0000","last_fullsized":"2024-09-13T17:21:37.732167+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:11.949573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986844+0000","last_change":"2024-09-13T17:21:36.748081+0000","last_active":"2024-09-13T17:21:37.986844+0000","last_peered":"2024-09-13T17:21:37.986844+0000","last_clean":"2024-09-13T17:21:37.986844+0000","last_became_active":"2024-09-13T17:21:36.747392+0000","last_became_peered":"2024-09-13T17:21:36.747392+0000","last_unstale":"2024-09-13T17:21:37.986844+0000","last_undegraded":"2024-09-13T17:21:37.986844+0000","last_fullsized":"2024-09-13T17:21:37.986844+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:38.521775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733940+0000","last_change":"2024-09-13T17:21:36.756830+0000","last_active":"2024-09-13T17:21:37.733940+0000","last_peered":"2024-09-13T17:21:37.733940+0000","last_clean":"2024-09-13T17:21:37.733940+0000","last_became_active":"2024-09-13T17:21:36.756397+0000","last_became_peered":"2024-09-13T17:21:36.756397+0000","last_unstale":"2024-09-13T17:21:37.733940+0000","last_undegraded":"2024-09-13T17:21:37.733940+0000","last_fullsized":"2024-09-13T17:21:37.733940+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:38.289392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987068+0000","last_change":"2024-09-13T17:21:36.749217+0000","last_active":"2024-09-13T17:21:37.987068+0000","last_peered":"2024-09-13T17:21:37.987068+0000","last_clean":"2024-09-13T17:21:37.987068+0000","last_became_active":"2024-09-13T17:21:36.748949+0000","last_became_peered":"2024-09-13T17:21:36.748949+0000","last_unstale":"2024-09-13T17:21:37.987068+0000","last_undegraded":"2024-09-13T17:21:37.987068+0000","last_fullsized":"2024-09-13T17:21:37.987068+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:01:10.179759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986797+0000","last_change":"2024-09-13T17:21:36.750807+0000","last_active":"2024-09-13T17:21:37.986797+0000","last_peered":"2024-09-13T17:21:37.986797+0000","last_clean":"2024-09-13T17:21:37.986797+0000","last_became_active":"2024-09-13T17:21:36.750529+0000","last_became_peered":"2024-09-13T17:21:36.750529+0000","last_unstale":"2024-09-13T17:21:37.986797+0000","last_undegraded":"2024-09-13T17:21:37.986797+0000","last_fullsized":"2024-09-13T17:21:37.986797+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:34.509612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733603+0000","last_change":"2024-09-13T17:21:36.759033+0000","last_active":"2024-09-13T17:21:37.733603+0000","last_peered":"2024-09-13T17:21:37.733603+0000","last_clean":"2024-09-13T17:21:37.733603+0000","last_became_active":"2024-09-13T17:21:36.758792+0000","last_became_peered":"2024-09-13T17:21:36.758792+0000","last_unstale":"2024-09-13T17:21:37.733603+0000","last_undegraded":"2024-09-13T17:21:37.733603+0000","last_fullsized":"2024-09-13T17:21:37.733603+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:32:28.758053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734249+0000","last_change":"2024-09-13T17:21:36.747690+0000","last_active":"2024-09-13T17:21:37.734249+0000","last_peered":"2024-09-13T17:21:37.734249+0000","last_clean":"2024-09-13T17:21:37.734249+0000","last_became_active":"2024-09-13T17:21:36.747422+0000","last_became_peered":"2024-09-13T17:21:36.747422+0000","last_unstale":"2024-09-13T17:21:37.734249+0000","last_undegraded":"2024-09-13T17:21:37.734249+0000","last_fullsized":"2024-09-13T17:21:37.734249+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:10.020415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986258+0000","last_change":"2024-09-13T17:21:36.743297+0000","last_active":"2024-09-13T17:21:37.986258+0000","last_peered":"2024-09-13T17:21:37.986258+0000","last_clean":"2024-09-13T17:21:37.986258+0000","last_became_active":"2024-09-13T17:21:36.743062+0000","last_became_peered":"2024-09-13T17:21:36.743062+0000","last_unstale":"2024-09-13T17:21:37.986258+0000","last_undegraded":"2024-09-13T17:21:37.986258+0000","last_fullsized":"2024-09-13T17:21:37.986258+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:38.055703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986506+0000","last_change":"2024-09-13T17:21:36.749920+0000","last_active":"2024-09-13T17:21:37.986506+0000","last_peered":"2024-09-13T17:21:37.986506+0000","last_clean":"2024-09-13T17:21:37.986506+0000","last_became_active":"2024-09-13T17:21:36.747974+0000","last_became_peered":"2024-09-13T17:21:36.747974+0000","last_unstale":"2024-09-13T17:21:37.986506+0000","last_undegraded":"2024-09-13T17:21:37.986506+0000","last_fullsized":"2024-09-13T17:21:37.986506+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:16:54.825955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.755370+0000","last_change":"2024-09-13T17:21:36.745077+0000","last_active":"2024-09-13T17:21:36.755370+0000","last_peered":"2024-09-13T17:21:36.755370+0000","last_clean":"2024-09-13T17:21:36.755370+0000","last_became_active":"2024-09-13T17:21:36.744089+0000","last_became_peered":"2024-09-13T17:21:36.744089+0000","last_unstale":"2024-09-13T17:21:36.755370+0000","last_undegraded":"2024-09-13T17:21:36.755370+0000","last_fullsized":"2024-09-13T17:21:36.755370+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:15.351494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986404+0000","last_change":"2024-09-13T17:21:36.748284+0000","last_active":"2024-09-13T17:21:37.986404+0000","last_peered":"2024-09-13T17:21:37.986404+0000","last_clean":"2024-09-13T17:21:37.986404+0000","last_became_active":"2024-09-13T17:21:36.747990+0000","last_became_peered":"2024-09-13T17:21:36.747990+0000","last_unstale":"2024-09-13T17:21:37.986404+0000","last_undegraded":"2024-09-13T17:21:37.986404+0000","last_fullsized":"2024-09-13T17:21:37.986404+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:33.924677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733504+0000","last_change":"2024-09-13T17:21:36.738847+0000","last_active":"2024-09-13T17:21:37.733504+0000","last_peered":"2024-09-13T17:21:37.733504+0000","last_clean":"2024-09-13T17:21:37.733504+0000","last_became_active":"2024-09-13T17:21:36.738528+0000","last_became_peered":"2024-09-13T17:21:36.738528+0000","last_unstale":"2024-09-13T17:21:37.733504+0000","last_undegraded":"2024-09-13T17:21:37.733504+0000","last_fullsized":"2024-09-13T17:21:37.733504+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:19.045283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731689+0000","last_change":"2024-09-13T17:21:36.754967+0000","last_active":"2024-09-13T17:21:37.731689+0000","last_peered":"2024-09-13T17:21:37.731689+0000","last_clean":"2024-09-13T17:21:37.731689+0000","last_became_active":"2024-09-13T17:21:36.754744+0000","last_became_peered":"2024-09-13T17:21:36.754744+0000","last_unstale":"2024-09-13T17:21:37.731689+0000","last_undegraded":"2024-09-13T17:21:37.731689+0000","last_fullsized":"2024-09-13T17:21:37.731689+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:51:22.135159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986088+0000","last_change":"2024-09-13T17:21:36.736750+0000","last_active":"2024-09-13T17:21:37.986088+0000","last_peered":"2024-09-13T17:21:37.986088+0000","last_clean":"2024-09-13T17:21:37.986088+0000","last_became_active":"2024-09-13T17:21:36.736062+0000","last_became_peered":"2024-09-13T17:21:36.736062+0000","last_unstale":"2024-09-13T17:21:37.986088+0000","last_undegraded":"2024-09-13T17:21:37.986088+0000","last_fullsized":"2024-09-13T17:21:37.986088+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:00:17.123975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733918+0000","last_change":"2024-09-13T17:21:36.746377+0000","last_active":"2024-09-13T17:21:37.733918+0000","last_peered":"2024-09-13T17:21:37.733918+0000","last_clean":"2024-09-13T17:21:37.733918+0000","last_became_active":"2024-09-13T17:21:36.745648+0000","last_became_peered":"2024-09-13T17:21:36.745648+0000","last_unstale":"2024-09-13T17:21:37.733918+0000","last_undegraded":"2024-09-13T17:21:37.733918+0000","last_fullsized":"2024-09-13T17:21:37.733918+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:53.376685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"192'4","reported_seq":15,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986762+0000","last_change":"2024-09-13T17:21:36.309004+0000","last_active":"2024-09-13T17:21:37.986762+0000","last_peered":"2024-09-13T17:21:37.986762+0000","last_clean":"2024-09-13T17:21:37.986762+0000","last_became_active":"2024-09-13T17:21:36.308738+0000","last_became_peered":"2024-09-13T17:21:36.308738+0000","last_unstale":"2024-09-13T17:21:37.986762+0000","last_undegraded":"2024-09-13T17:21:37.986762+0000","last_fullsized":"2024-09-13T17:21:37.986762+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:33.142468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987210+0000","last_change":"2024-09-13T17:21:36.747527+0000","last_active":"2024-09-13T17:21:37.987210+0000","last_peered":"2024-09-13T17:21:37.987210+0000","last_clean":"2024-09-13T17:21:37.987210+0000","last_became_active":"2024-09-13T17:21:36.747131+0000","last_became_peered":"2024-09-13T17:21:36.747131+0000","last_unstale":"2024-09-13T17:21:37.987210+0000","last_undegraded":"2024-09-13T17:21:37.987210+0000","last_fullsized":"2024-09-13T17:21:37.987210+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:53.049699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734098+0000","last_change":"2024-09-13T17:21:36.307661+0000","last_active":"2024-09-13T17:21:37.734098+0000","last_peered":"2024-09-13T17:21:37.734098+0000","last_clean":"2024-09-13T17:21:37.734098+0000","last_became_active":"2024-09-13T17:21:36.307004+0000","last_became_peered":"2024-09-13T17:21:36.307004+0000","last_unstale":"2024-09-13T17:21:37.734098+0000","last_undegraded":"2024-09-13T17:21:37.734098+0000","last_fullsized":"2024-09-13T17:21:37.734098+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:44.202163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986047+0000","last_change":"2024-09-13T17:21:36.739149+0000","last_active":"2024-09-13T17:21:37.986047+0000","last_peered":"2024-09-13T17:21:37.986047+0000","last_clean":"2024-09-13T17:21:37.986047+0000","last_became_active":"2024-09-13T17:21:36.738897+0000","last_became_peered":"2024-09-13T17:21:36.738897+0000","last_unstale":"2024-09-13T17:21:37.986047+0000","last_undegraded":"2024-09-13T17:21:37.986047+0000","last_fullsized":"2024-09-13T17:21:37.986047+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:51.289471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"192'2","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758430+0000","last_change":"2024-09-13T17:21:36.308002+0000","last_active":"2024-09-13T17:21:36.758430+0000","last_peered":"2024-09-13T17:21:36.758430+0000","last_clean":"2024-09-13T17:21:36.758430+0000","last_became_active":"2024-09-13T17:21:36.307473+0000","last_became_peered":"2024-09-13T17:21:36.307473+0000","last_unstale":"2024-09-13T17:21:36.758430+0000","last_undegraded":"2024-09-13T17:21:36.758430+0000","last_fullsized":"2024-09-13T17:21:36.758430+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:42:51.374000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732038+0000","last_change":"2024-09-13T17:21:36.745188+0000","last_active":"2024-09-13T17:21:37.732038+0000","last_peered":"2024-09-13T17:21:37.732038+0000","last_clean":"2024-09-13T17:21:37.732038+0000","last_became_active":"2024-09-13T17:21:36.744544+0000","last_became_peered":"2024-09-13T17:21:36.744544+0000","last_unstale":"2024-09-13T17:21:37.732038+0000","last_undegraded":"2024-09-13T17:21:37.732038+0000","last_fullsized":"2024-09-13T17:21:37.732038+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:51.986633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734536+0000","last_change":"2024-09-13T17:21:36.312756+0000","last_active":"2024-09-13T17:21:37.734536+0000","last_peered":"2024-09-13T17:21:37.734536+0000","last_clean":"2024-09-13T17:21:37.734536+0000","last_became_active":"2024-09-13T17:21:36.312487+0000","last_became_peered":"2024-09-13T17:21:36.312487+0000","last_unstale":"2024-09-13T17:21:37.734536+0000","last_undegraded":"2024-09-13T17:21:37.734536+0000","last_fullsized":"2024-09-13T17:21:37.734536+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:09.013124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734063+0000","last_change":"2024-09-13T17:21:36.746808+0000","last_active":"2024-09-13T17:21:37.734063+0000","last_peered":"2024-09-13T17:21:37.734063+0000","last_clean":"2024-09-13T17:21:37.734063+0000","last_became_active":"2024-09-13T17:21:36.746526+0000","last_became_peered":"2024-09-13T17:21:36.746526+0000","last_unstale":"2024-09-13T17:21:37.734063+0000","last_undegraded":"2024-09-13T17:21:37.734063+0000","last_fullsized":"2024-09-13T17:21:37.734063+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:33.299639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986685+0000","last_change":"2024-09-13T17:21:36.299723+0000","last_active":"2024-09-13T17:21:37.986685+0000","last_peered":"2024-09-13T17:21:37.986685+0000","last_clean":"2024-09-13T17:21:37.986685+0000","last_became_active":"2024-09-13T17:21:36.299432+0000","last_became_peered":"2024-09-13T17:21:36.299432+0000","last_unstale":"2024-09-13T17:21:37.986685+0000","last_undegraded":"2024-09-13T17:21:37.986685+0000","last_fullsized":"2024-09-13T17:21:37.986685+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:02.934115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986340+0000","last_change":"2024-09-13T17:21:36.751769+0000","last_active":"2024-09-13T17:21:37.986340+0000","last_peered":"2024-09-13T17:21:37.986340+0000","last_clean":"2024-09-13T17:21:37.986340+0000","last_became_active":"2024-09-13T17:21:36.751503+0000","last_became_peered":"2024-09-13T17:21:36.751503+0000","last_unstale":"2024-09-13T17:21:37.986340+0000","last_undegraded":"2024-09-13T17:21:37.986340+0000","last_fullsized":"2024-09-13T17:21:37.986340+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:51.528433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986487+0000","last_change":"2024-09-13T17:21:36.314738+0000","last_active":"2024-09-13T17:21:37.986487+0000","last_peered":"2024-09-13T17:21:37.986487+0000","last_clean":"2024-09-13T17:21:37.986487+0000","last_became_active":"2024-09-13T17:21:36.313576+0000","last_became_peered":"2024-09-13T17:21:36.313576+0000","last_unstale":"2024-09-13T17:21:37.986487+0000","last_undegraded":"2024-09-13T17:21:37.986487+0000","last_fullsized":"2024-09-13T17:21:37.986487+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:05:32.917385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734458+0000","last_change":"2024-09-13T17:21:36.753712+0000","last_active":"2024-09-13T17:21:37.734458+0000","last_peered":"2024-09-13T17:21:37.734458+0000","last_clean":"2024-09-13T17:21:37.734458+0000","last_became_active":"2024-09-13T17:21:36.752544+0000","last_became_peered":"2024-09-13T17:21:36.752544+0000","last_unstale":"2024-09-13T17:21:37.734458+0000","last_undegraded":"2024-09-13T17:21:37.734458+0000","last_fullsized":"2024-09-13T17:21:37.734458+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:00.490825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986530+0000","last_change":"2024-09-13T17:21:36.299554+0000","last_active":"2024-09-13T17:21:37.986530+0000","last_peered":"2024-09-13T17:21:37.986530+0000","last_clean":"2024-09-13T17:21:37.986530+0000","last_became_active":"2024-09-13T17:21:36.299250+0000","last_became_peered":"2024-09-13T17:21:36.299250+0000","last_unstale":"2024-09-13T17:21:37.986530+0000","last_undegraded":"2024-09-13T17:21:37.986530+0000","last_fullsized":"2024-09-13T17:21:37.986530+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:59.483533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734015+0000","last_change":"2024-09-13T17:21:36.751006+0000","last_active":"2024-09-13T17:21:37.734015+0000","last_peered":"2024-09-13T17:21:37.734015+0000","last_clean":"2024-09-13T17:21:37.734015+0000","last_became_active":"2024-09-13T17:21:36.750753+0000","last_became_peered":"2024-09-13T17:21:36.750753+0000","last_unstale":"2024-09-13T17:21:37.734015+0000","last_undegraded":"2024-09-13T17:21:37.734015+0000","last_fullsized":"2024-09-13T17:21:37.734015+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:38.404283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987154+0000","last_change":"2024-09-13T17:21:36.301369+0000","last_active":"2024-09-13T17:21:37.987154+0000","last_peered":"2024-09-13T17:21:37.987154+0000","last_clean":"2024-09-13T17:21:37.987154+0000","last_became_active":"2024-09-13T17:21:36.301048+0000","last_became_peered":"2024-09-13T17:21:36.301048+0000","last_unstale":"2024-09-13T17:21:37.987154+0000","last_undegraded":"2024-09-13T17:21:37.987154+0000","last_fullsized":"2024-09-13T17:21:37.987154+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:20:38.392162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758424+0000","last_change":"2024-09-13T17:21:36.753522+0000","last_active":"2024-09-13T17:21:36.758424+0000","last_peered":"2024-09-13T17:21:36.758424+0000","last_clean":"2024-09-13T17:21:36.758424+0000","last_became_active":"2024-09-13T17:21:36.751908+0000","last_became_peered":"2024-09-13T17:21:36.751908+0000","last_unstale":"2024-09-13T17:21:36.758424+0000","last_undegraded":"2024-09-13T17:21:36.758424+0000","last_fullsized":"2024-09-13T17:21:36.758424+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:34.149760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987346+0000","last_change":"2024-09-13T17:21:36.309447+0000","last_active":"2024-09-13T17:21:37.987346+0000","last_peered":"2024-09-13T17:21:37.987346+0000","last_clean":"2024-09-13T17:21:37.987346+0000","last_became_active":"2024-09-13T17:21:36.308414+0000","last_became_peered":"2024-09-13T17:21:36.308414+0000","last_unstale":"2024-09-13T17:21:37.987346+0000","last_undegraded":"2024-09-13T17:21:37.987346+0000","last_fullsized":"2024-09-13T17:21:37.987346+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:15:37.396991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733804+0000","last_change":"2024-09-13T17:21:36.753468+0000","last_active":"2024-09-13T17:21:37.733804+0000","last_peered":"2024-09-13T17:21:37.733804+0000","last_clean":"2024-09-13T17:21:37.733804+0000","last_became_active":"2024-09-13T17:21:36.753099+0000","last_became_peered":"2024-09-13T17:21:36.753099+0000","last_unstale":"2024-09-13T17:21:37.733804+0000","last_undegraded":"2024-09-13T17:21:37.733804+0000","last_fullsized":"2024-09-13T17:21:37.733804+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:58.354697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"192'3","reported_seq":14,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732264+0000","last_change":"2024-09-13T17:21:36.309438+0000","last_active":"2024-09-13T17:21:37.732264+0000","last_peered":"2024-09-13T17:21:37.732264+0000","last_clean":"2024-09-13T17:21:37.732264+0000","last_became_active":"2024-09-13T17:21:36.309174+0000","last_became_peered":"2024-09-13T17:21:36.309174+0000","last_unstale":"2024-09-13T17:21:37.732264+0000","last_undegraded":"2024-09-13T17:21:37.732264+0000","last_fullsized":"2024-09-13T17:21:37.732264+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:09.538090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986224+0000","last_change":"2024-09-13T17:21:36.742783+0000","last_active":"2024-09-13T17:21:37.986224+0000","last_peered":"2024-09-13T17:21:37.986224+0000","last_clean":"2024-09-13T17:21:37.986224+0000","last_became_active":"2024-09-13T17:21:36.742452+0000","last_became_peered":"2024-09-13T17:21:36.742452+0000","last_unstale":"2024-09-13T17:21:37.986224+0000","last_undegraded":"2024-09-13T17:21:37.986224+0000","last_fullsized":"2024-09-13T17:21:37.986224+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:52:16.541840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"192'6","reported_seq":17,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734530+0000","last_change":"2024-09-13T17:21:36.308875+0000","last_active":"2024-09-13T17:21:37.734530+0000","last_peered":"2024-09-13T17:21:37.734530+0000","last_clean":"2024-09-13T17:21:37.734530+0000","last_became_active":"2024-09-13T17:21:36.308623+0000","last_became_peered":"2024-09-13T17:21:36.308623+0000","last_unstale":"2024-09-13T17:21:37.734530+0000","last_undegraded":"2024-09-13T17:21:37.734530+0000","last_fullsized":"2024-09-13T17:21:37.734530+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:53.028588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986667+0000","last_change":"2024-09-13T17:21:36.746168+0000","last_active":"2024-09-13T17:21:37.986667+0000","last_peered":"2024-09-13T17:21:37.986667+0000","last_clean":"2024-09-13T17:21:37.986667+0000","last_became_active":"2024-09-13T17:21:36.745926+0000","last_became_peered":"2024-09-13T17:21:36.745926+0000","last_unstale":"2024-09-13T17:21:37.986667+0000","last_undegraded":"2024-09-13T17:21:37.986667+0000","last_fullsized":"2024-09-13T17:21:37.986667+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:03.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987067+0000","last_change":"2024-09-13T17:21:36.302066+0000","last_active":"2024-09-13T17:21:37.987067+0000","last_peered":"2024-09-13T17:21:37.987067+0000","last_clean":"2024-09-13T17:21:37.987067+0000","last_became_active":"2024-09-13T17:21:36.301803+0000","last_became_peered":"2024-09-13T17:21:36.301803+0000","last_unstale":"2024-09-13T17:21:37.987067+0000","last_undegraded":"2024-09-13T17:21:37.987067+0000","last_fullsized":"2024-09-13T17:21:37.987067+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:32.292347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732144+0000","last_change":"2024-09-13T17:21:36.749244+0000","last_active":"2024-09-13T17:21:37.732144+0000","last_peered":"2024-09-13T17:21:37.732144+0000","last_clean":"2024-09-13T17:21:37.732144+0000","last_became_active":"2024-09-13T17:21:36.748557+0000","last_became_peered":"2024-09-13T17:21:36.748557+0000","last_unstale":"2024-09-13T17:21:37.732144+0000","last_undegraded":"2024-09-13T17:21:37.732144+0000","last_fullsized":"2024-09-13T17:21:37.732144+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:49:41.298517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987083+0000","last_change":"2024-09-13T17:21:36.304740+0000","last_active":"2024-09-13T17:21:37.987083+0000","last_peered":"2024-09-13T17:21:37.987083+0000","last_clean":"2024-09-13T17:21:37.987083+0000","last_became_active":"2024-09-13T17:21:36.304485+0000","last_became_peered":"2024-09-13T17:21:36.304485+0000","last_unstale":"2024-09-13T17:21:37.987083+0000","last_undegraded":"2024-09-13T17:21:37.987083+0000","last_fullsized":"2024-09-13T17:21:37.987083+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:07.990488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757904+0000","last_change":"2024-09-13T17:21:36.750869+0000","last_active":"2024-09-13T17:21:36.757904+0000","last_peered":"2024-09-13T17:21:36.757904+0000","last_clean":"2024-09-13T17:21:36.757904+0000","last_became_active":"2024-09-13T17:21:36.749555+0000","last_became_peered":"2024-09-13T17:21:36.749555+0000","last_unstale":"2024-09-13T17:21:36.757904+0000","last_undegraded":"2024-09-13T17:21:36.757904+0000","last_fullsized":"2024-09-13T17:21:36.757904+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:51:51.007537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731816+0000","last_change":"2024-09-13T17:21:36.313796+0000","last_active":"2024-09-13T17:21:37.731816+0000","last_peered":"2024-09-13T17:21:37.731816+0000","last_clean":"2024-09-13T17:21:37.731816+0000","last_became_active":"2024-09-13T17:21:36.313384+0000","last_became_peered":"2024-09-13T17:21:36.313384+0000","last_unstale":"2024-09-13T17:21:37.731816+0000","last_undegraded":"2024-09-13T17:21:37.731816+0000","last_fullsized":"2024-09-13T17:21:37.731816+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:03.457394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985822+0000","last_change":"2024-09-13T17:21:36.735315+0000","last_active":"2024-09-13T17:21:37.985822+0000","last_peered":"2024-09-13T17:21:37.985822+0000","last_clean":"2024-09-13T17:21:37.985822+0000","last_became_active":"2024-09-13T17:21:36.735019+0000","last_became_peered":"2024-09-13T17:21:36.735019+0000","last_unstale":"2024-09-13T17:21:37.985822+0000","last_undegraded":"2024-09-13T17:21:37.985822+0000","last_fullsized":"2024-09-13T17:21:37.985822+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:10:23.911415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"192'8","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.760616+0000","last_change":"2024-09-13T17:21:36.306747+0000","last_active":"2024-09-13T17:21:36.760616+0000","last_peered":"2024-09-13T17:21:36.760616+0000","last_clean":"2024-09-13T17:21:36.760616+0000","last_became_active":"2024-09-13T17:21:36.306416+0000","last_became_peered":"2024-09-13T17:21:36.306416+0000","last_unstale":"2024-09-13T17:21:36.760616+0000","last_undegraded":"2024-09-13T17:21:36.760616+0000","last_fullsized":"2024-09-13T17:21:36.760616+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:11.973871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734007+0000","last_change":"2024-09-13T17:21:36.755992+0000","last_active":"2024-09-13T17:21:37.734007+0000","last_peered":"2024-09-13T17:21:37.734007+0000","last_clean":"2024-09-13T17:21:37.734007+0000","last_became_active":"2024-09-13T17:21:36.753414+0000","last_became_peered":"2024-09-13T17:21:36.753414+0000","last_unstale":"2024-09-13T17:21:37.734007+0000","last_undegraded":"2024-09-13T17:21:37.734007+0000","last_fullsized":"2024-09-13T17:21:37.734007+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:38.521775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987165+0000","last_change":"2024-09-13T17:21:36.308772+0000","last_active":"2024-09-13T17:21:37.987165+0000","last_peered":"2024-09-13T17:21:37.987165+0000","last_clean":"2024-09-13T17:21:37.987165+0000","last_became_active":"2024-09-13T17:21:36.308498+0000","last_became_peered":"2024-09-13T17:21:36.308498+0000","last_unstale":"2024-09-13T17:21:37.987165+0000","last_undegraded":"2024-09-13T17:21:37.987165+0000","last_fullsized":"2024-09-13T17:21:37.987165+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:00.880376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734277+0000","last_change":"2024-09-13T17:21:36.747817+0000","last_active":"2024-09-13T17:21:37.734277+0000","last_peered":"2024-09-13T17:21:37.734277+0000","last_clean":"2024-09-13T17:21:37.734277+0000","last_became_active":"2024-09-13T17:21:36.747554+0000","last_became_peered":"2024-09-13T17:21:36.747554+0000","last_unstale":"2024-09-13T17:21:37.734277+0000","last_undegraded":"2024-09-13T17:21:37.734277+0000","last_fullsized":"2024-09-13T17:21:37.734277+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:54.035880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734259+0000","last_change":"2024-09-13T17:21:36.300595+0000","last_active":"2024-09-13T17:21:37.734259+0000","last_peered":"2024-09-13T17:21:37.734259+0000","last_clean":"2024-09-13T17:21:37.734259+0000","last_became_active":"2024-09-13T17:21:36.300031+0000","last_became_peered":"2024-09-13T17:21:36.300031+0000","last_unstale":"2024-09-13T17:21:37.734259+0000","last_undegraded":"2024-09-13T17:21:37.734259+0000","last_fullsized":"2024-09-13T17:21:37.734259+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:05.872053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.752755+0000","last_change":"2024-09-13T17:21:36.742622+0000","last_active":"2024-09-13T17:21:36.752755+0000","last_peered":"2024-09-13T17:21:36.752755+0000","last_clean":"2024-09-13T17:21:36.752755+0000","last_became_active":"2024-09-13T17:21:36.742357+0000","last_became_peered":"2024-09-13T17:21:36.742357+0000","last_unstale":"2024-09-13T17:21:36.752755+0000","last_undegraded":"2024-09-13T17:21:36.752755+0000","last_fullsized":"2024-09-13T17:21:36.752755+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:53.099300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732354+0000","last_change":"2024-09-13T17:21:36.300292+0000","last_active":"2024-09-13T17:21:37.732354+0000","last_peered":"2024-09-13T17:21:37.732354+0000","last_clean":"2024-09-13T17:21:37.732354+0000","last_became_active":"2024-09-13T17:21:36.300030+0000","last_became_peered":"2024-09-13T17:21:36.300030+0000","last_unstale":"2024-09-13T17:21:37.732354+0000","last_undegraded":"2024-09-13T17:21:37.732354+0000","last_fullsized":"2024-09-13T17:21:37.732354+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:24.398739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732444+0000","last_change":"2024-09-13T17:21:36.756635+0000","last_active":"2024-09-13T17:21:37.732444+0000","last_peered":"2024-09-13T17:21:37.732444+0000","last_clean":"2024-09-13T17:21:37.732444+0000","last_became_active":"2024-09-13T17:21:36.755698+0000","last_became_peered":"2024-09-13T17:21:36.755698+0000","last_unstale":"2024-09-13T17:21:37.732444+0000","last_undegraded":"2024-09-13T17:21:37.732444+0000","last_fullsized":"2024-09-13T17:21:37.732444+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:18.013523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.738731+0000","last_change":"2024-09-13T17:21:36.310694+0000","last_active":"2024-09-13T17:21:36.738731+0000","last_peered":"2024-09-13T17:21:36.738731+0000","last_clean":"2024-09-13T17:21:36.738731+0000","last_became_active":"2024-09-13T17:21:36.309451+0000","last_became_peered":"2024-09-13T17:21:36.309451+0000","last_unstale":"2024-09-13T17:21:36.738731+0000","last_undegraded":"2024-09-13T17:21:36.738731+0000","last_fullsized":"2024-09-13T17:21:36.738731+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:51.263112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.731948+0000","last_change":"2024-09-13T17:21:36.745379+0000","last_active":"2024-09-13T17:21:37.731948+0000","last_peered":"2024-09-13T17:21:37.731948+0000","last_clean":"2024-09-13T17:21:37.731948+0000","last_became_active":"2024-09-13T17:21:36.744448+0000","last_became_peered":"2024-09-13T17:21:36.744448+0000","last_unstale":"2024-09-13T17:21:37.731948+0000","last_undegraded":"2024-09-13T17:21:37.731948+0000","last_fullsized":"2024-09-13T17:21:37.731948+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:38.055703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"192'2","reported_seq":13,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.732238+0000","last_change":"2024-09-13T17:21:36.313327+0000","last_active":"2024-09-13T17:21:37.732238+0000","last_peered":"2024-09-13T17:21:37.732238+0000","last_clean":"2024-09-13T17:21:37.732238+0000","last_became_active":"2024-09-13T17:21:36.313037+0000","last_became_peered":"2024-09-13T17:21:36.313037+0000","last_unstale":"2024-09-13T17:21:37.732238+0000","last_undegraded":"2024-09-13T17:21:37.732238+0000","last_fullsized":"2024-09-13T17:21:37.732238+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:02:54.858905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.741284+0000","last_change":"2024-09-13T17:21:36.737995+0000","last_active":"2024-09-13T17:21:36.741284+0000","last_peered":"2024-09-13T17:21:36.741284+0000","last_clean":"2024-09-13T17:21:36.741284+0000","last_became_active":"2024-09-13T17:21:36.737725+0000","last_became_peered":"2024-09-13T17:21:36.737725+0000","last_unstale":"2024-09-13T17:21:36.741284+0000","last_undegraded":"2024-09-13T17:21:36.741284+0000","last_fullsized":"2024-09-13T17:21:36.741284+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:40.827453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986524+0000","last_change":"2024-09-13T17:21:36.302171+0000","last_active":"2024-09-13T17:21:37.986524+0000","last_peered":"2024-09-13T17:21:37.986524+0000","last_clean":"2024-09-13T17:21:37.986524+0000","last_became_active":"2024-09-13T17:21:36.301525+0000","last_became_peered":"2024-09-13T17:21:36.301525+0000","last_unstale":"2024-09-13T17:21:37.986524+0000","last_undegraded":"2024-09-13T17:21:37.986524+0000","last_fullsized":"2024-09-13T17:21:37.986524+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:43.526946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733040+0000","last_change":"2024-09-13T17:21:36.738161+0000","last_active":"2024-09-13T17:21:37.733040+0000","last_peered":"2024-09-13T17:21:37.733040+0000","last_clean":"2024-09-13T17:21:37.733040+0000","last_became_active":"2024-09-13T17:21:36.737885+0000","last_became_peered":"2024-09-13T17:21:36.737885+0000","last_unstale":"2024-09-13T17:21:37.733040+0000","last_undegraded":"2024-09-13T17:21:37.733040+0000","last_fullsized":"2024-09-13T17:21:37.733040+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:53.816777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.985914+0000","last_change":"2024-09-13T17:21:36.303299+0000","last_active":"2024-09-13T17:21:37.985914+0000","last_peered":"2024-09-13T17:21:37.985914+0000","last_clean":"2024-09-13T17:21:37.985914+0000","last_became_active":"2024-09-13T17:21:36.302858+0000","last_became_peered":"2024-09-13T17:21:36.302858+0000","last_unstale":"2024-09-13T17:21:37.985914+0000","last_undegraded":"2024-09-13T17:21:37.985914+0000","last_fullsized":"2024-09-13T17:21:37.985914+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:04.631923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986827+0000","last_change":"2024-09-13T17:21:36.747167+0000","last_active":"2024-09-13T17:21:37.986827+0000","last_peered":"2024-09-13T17:21:37.986827+0000","last_clean":"2024-09-13T17:21:37.986827+0000","last_became_active":"2024-09-13T17:21:36.746664+0000","last_became_peered":"2024-09-13T17:21:36.746664+0000","last_unstale":"2024-09-13T17:21:37.986827+0000","last_undegraded":"2024-09-13T17:21:37.986827+0000","last_fullsized":"2024-09-13T17:21:37.986827+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:17:16.322908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986376+0000","last_change":"2024-09-13T17:21:36.310792+0000","last_active":"2024-09-13T17:21:37.986376+0000","last_peered":"2024-09-13T17:21:37.986376+0000","last_clean":"2024-09-13T17:21:37.986376+0000","last_became_active":"2024-09-13T17:21:36.310527+0000","last_became_peered":"2024-09-13T17:21:36.310527+0000","last_unstale":"2024-09-13T17:21:37.986376+0000","last_undegraded":"2024-09-13T17:21:37.986376+0000","last_fullsized":"2024-09-13T17:21:37.986376+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:46:04.726410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986638+0000","last_change":"2024-09-13T17:21:36.747238+0000","last_active":"2024-09-13T17:21:37.986638+0000","last_peered":"2024-09-13T17:21:37.986638+0000","last_clean":"2024-09-13T17:21:37.986638+0000","last_became_active":"2024-09-13T17:21:36.746981+0000","last_became_peered":"2024-09-13T17:21:36.746981+0000","last_unstale":"2024-09-13T17:21:37.986638+0000","last_undegraded":"2024-09-13T17:21:37.986638+0000","last_fullsized":"2024-09-13T17:21:37.986638+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:50.020081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"192'2","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.758371+0000","last_change":"2024-09-13T17:21:36.309805+0000","last_active":"2024-09-13T17:21:36.758371+0000","last_peered":"2024-09-13T17:21:36.758371+0000","last_clean":"2024-09-13T17:21:36.758371+0000","last_became_active":"2024-09-13T17:21:36.309140+0000","last_became_peered":"2024-09-13T17:21:36.309140+0000","last_unstale":"2024-09-13T17:21:36.758371+0000","last_undegraded":"2024-09-13T17:21:36.758371+0000","last_fullsized":"2024-09-13T17:21:36.758371+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:03:11.894842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986219+0000","last_change":"2024-09-13T17:21:36.749941+0000","last_active":"2024-09-13T17:21:37.986219+0000","last_peered":"2024-09-13T17:21:37.986219+0000","last_clean":"2024-09-13T17:21:37.986219+0000","last_became_active":"2024-09-13T17:21:36.749595+0000","last_became_peered":"2024-09-13T17:21:36.749595+0000","last_unstale":"2024-09-13T17:21:37.986219+0000","last_undegraded":"2024-09-13T17:21:37.986219+0000","last_fullsized":"2024-09-13T17:21:37.986219+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:15:28.804370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734108+0000","last_change":"2024-09-13T17:21:36.309579+0000","last_active":"2024-09-13T17:21:37.734108+0000","last_peered":"2024-09-13T17:21:37.734108+0000","last_clean":"2024-09-13T17:21:37.734108+0000","last_became_active":"2024-09-13T17:21:36.308175+0000","last_became_peered":"2024-09-13T17:21:36.308175+0000","last_unstale":"2024-09-13T17:21:37.734108+0000","last_undegraded":"2024-09-13T17:21:37.734108+0000","last_fullsized":"2024-09-13T17:21:37.734108+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:52.369393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733756+0000","last_change":"2024-09-13T17:21:36.736311+0000","last_active":"2024-09-13T17:21:37.733756+0000","last_peered":"2024-09-13T17:21:37.733756+0000","last_clean":"2024-09-13T17:21:37.733756+0000","last_became_active":"2024-09-13T17:21:36.735759+0000","last_became_peered":"2024-09-13T17:21:36.735759+0000","last_unstale":"2024-09-13T17:21:37.733756+0000","last_undegraded":"2024-09-13T17:21:37.733756+0000","last_fullsized":"2024-09-13T17:21:37.733756+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:03.148877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986889+0000","last_change":"2024-09-13T17:21:36.313125+0000","last_active":"2024-09-13T17:21:37.986889+0000","last_peered":"2024-09-13T17:21:37.986889+0000","last_clean":"2024-09-13T17:21:37.986889+0000","last_became_active":"2024-09-13T17:21:36.312821+0000","last_became_peered":"2024-09-13T17:21:36.312821+0000","last_unstale":"2024-09-13T17:21:37.986889+0000","last_undegraded":"2024-09-13T17:21:37.986889+0000","last_fullsized":"2024-09-13T17:21:37.986889+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:01.963163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.987209+0000","last_change":"2024-09-13T17:21:36.750620+0000","last_active":"2024-09-13T17:21:37.987209+0000","last_peered":"2024-09-13T17:21:37.987209+0000","last_clean":"2024-09-13T17:21:37.987209+0000","last_became_active":"2024-09-13T17:21:36.750347+0000","last_became_peered":"2024-09-13T17:21:36.750347+0000","last_unstale":"2024-09-13T17:21:37.987209+0000","last_undegraded":"2024-09-13T17:21:37.987209+0000","last_fullsized":"2024-09-13T17:21:37.987209+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:39:34.814185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":10,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.738823+0000","last_change":"2024-09-13T17:21:36.305629+0000","last_active":"2024-09-13T17:21:36.738823+0000","last_peered":"2024-09-13T17:21:36.738823+0000","last_clean":"2024-09-13T17:21:36.738823+0000","last_became_active":"2024-09-13T17:21:36.305358+0000","last_became_peered":"2024-09-13T17:21:36.305358+0000","last_unstale":"2024-09-13T17:21:36.738823+0000","last_undegraded":"2024-09-13T17:21:36.738823+0000","last_fullsized":"2024-09-13T17:21:36.738823+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:38.635179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986430+0000","last_change":"2024-09-13T17:21:36.736928+0000","last_active":"2024-09-13T17:21:37.986430+0000","last_peered":"2024-09-13T17:21:37.986430+0000","last_clean":"2024-09-13T17:21:37.986430+0000","last_became_active":"2024-09-13T17:21:36.736670+0000","last_became_peered":"2024-09-13T17:21:36.736670+0000","last_unstale":"2024-09-13T17:21:37.986430+0000","last_undegraded":"2024-09-13T17:21:37.986430+0000","last_fullsized":"2024-09-13T17:21:37.986430+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:24:56.819148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.734158+0000","last_change":"2024-09-13T17:21:36.313511+0000","last_active":"2024-09-13T17:21:37.734158+0000","last_peered":"2024-09-13T17:21:37.734158+0000","last_clean":"2024-09-13T17:21:37.734158+0000","last_became_active":"2024-09-13T17:21:36.313126+0000","last_became_peered":"2024-09-13T17:21:36.313126+0000","last_unstale":"2024-09-13T17:21:37.734158+0000","last_undegraded":"2024-09-13T17:21:37.734158+0000","last_fullsized":"2024-09-13T17:21:37.734158+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:18.037991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986673+0000","last_change":"2024-09-13T17:21:36.751622+0000","last_active":"2024-09-13T17:21:37.986673+0000","last_peered":"2024-09-13T17:21:37.986673+0000","last_clean":"2024-09-13T17:21:37.986673+0000","last_became_active":"2024-09-13T17:21:36.751280+0000","last_became_peered":"2024-09-13T17:21:36.751280+0000","last_unstale":"2024-09-13T17:21:37.986673+0000","last_undegraded":"2024-09-13T17:21:37.986673+0000","last_fullsized":"2024-09-13T17:21:37.986673+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:08:54.035880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986758+0000","last_change":"2024-09-13T17:21:36.311612+0000","last_active":"2024-09-13T17:21:37.986758+0000","last_peered":"2024-09-13T17:21:37.986758+0000","last_clean":"2024-09-13T17:21:37.986758+0000","last_became_active":"2024-09-13T17:21:36.311312+0000","last_became_peered":"2024-09-13T17:21:36.311312+0000","last_unstale":"2024-09-13T17:21:37.986758+0000","last_undegraded":"2024-09-13T17:21:37.986758+0000","last_fullsized":"2024-09-13T17:21:37.986758+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:45:00.342646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":9,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.757882+0000","last_change":"2024-09-13T17:21:36.749577+0000","last_active":"2024-09-13T17:21:36.757882+0000","last_peered":"2024-09-13T17:21:36.757882+0000","last_clean":"2024-09-13T17:21:36.757882+0000","last_became_active":"2024-09-13T17:21:36.748510+0000","last_became_peered":"2024-09-13T17:21:36.748510+0000","last_unstale":"2024-09-13T17:21:36.757882+0000","last_undegraded":"2024-09-13T17:21:36.757882+0000","last_fullsized":"2024-09-13T17:21:36.757882+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:45.209455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986943+0000","last_change":"2024-09-13T17:21:36.306469+0000","last_active":"2024-09-13T17:21:37.986943+0000","last_peered":"2024-09-13T17:21:37.986943+0000","last_clean":"2024-09-13T17:21:37.986943+0000","last_became_active":"2024-09-13T17:21:36.306116+0000","last_became_peered":"2024-09-13T17:21:36.306116+0000","last_unstale":"2024-09-13T17:21:37.986943+0000","last_undegraded":"2024-09-13T17:21:37.986943+0000","last_fullsized":"2024-09-13T17:21:37.986943+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:57.347405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.733243+0000","last_change":"2024-09-13T17:21:36.744596+0000","last_active":"2024-09-13T17:21:37.733243+0000","last_peered":"2024-09-13T17:21:37.733243+0000","last_clean":"2024-09-13T17:21:37.733243+0000","last_became_active":"2024-09-13T17:21:36.744274+0000","last_became_peered":"2024-09-13T17:21:36.744274+0000","last_unstale":"2024-09-13T17:21:37.733243+0000","last_undegraded":"2024-09-13T17:21:37.733243+0000","last_fullsized":"2024-09-13T17:21:37.733243+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:51.528433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":11,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986244+0000","last_change":"2024-09-13T17:21:36.306340+0000","last_active":"2024-09-13T17:21:37.986244+0000","last_peered":"2024-09-13T17:21:37.986244+0000","last_clean":"2024-09-13T17:21:37.986244+0000","last_became_active":"2024-09-13T17:21:36.305674+0000","last_became_peered":"2024-09-13T17:21:36.305674+0000","last_unstale":"2024-09-13T17:21:37.986244+0000","last_undegraded":"2024-09-13T17:21:37.986244+0000","last_fullsized":"2024-09-13T17:21:37.986244+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:39:05.872053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":10,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986327+0000","last_change":"2024-09-13T17:21:36.745187+0000","last_active":"2024-09-13T17:21:37.986327+0000","last_peered":"2024-09-13T17:21:37.986327+0000","last_clean":"2024-09-13T17:21:37.986327+0000","last_became_active":"2024-09-13T17:21:36.744577+0000","last_became_peered":"2024-09-13T17:21:36.744577+0000","last_unstale":"2024-09-13T17:21:37.986327+0000","last_undegraded":"2024-09-13T17:21:37.986327+0000","last_fullsized":"2024-09-13T17:21:37.986327+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":191,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:36.285868+0000","last_clean_scrub_stamp":"2024-09-13T17:21:36.285868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:18.013523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"192'1","reported_seq":12,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-13T17:21:37.986763+0000","last_change":"2024-09-13T17:21:36.313777+0000","last_active":"2024-09-13T17:21:37.986763+0000","last_peered":"2024-09-13T17:21:37.986763+0000","last_clean":"2024-09-13T17:21:37.986763+0000","last_became_active":"2024-09-13T17:21:36.313340+0000","last_became_peered":"2024-09-13T17:21:36.313340+0000","last_unstale":"2024-09-13T17:21:37.986763+0000","last_undegraded":"2024-09-13T17:21:37.986763+0000","last_fullsized":"2024-09-13T17:21:37.986763+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:21:35.278576+0000","last_clean_scrub_stamp":"2024-09-13T17:21:35.278576+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:39.820161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-09-13T17:21:36.741338+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:21:36.741338+0000","last_peered":"2024-09-13T17:21:36.741338+0000","last_clean":"2024-09-13T17:21:36.741338+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:21:36.741338+0000","last_undegraded":"2024-09-13T17:21:36.741338+0000","last_fullsized":"2024-09-13T17:21:36.741338+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70120,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301720,"statfs":{"total":96636764160,"available":96564961280,"internally_reserved":0,"allocated":3035136,"data_stored":1665337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.628,"15min":0.64300000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":1.008,"5min":1.008,"15min":2.0880000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.65600000000000003,"15min":0.67300000000000004},"min":{"1min":0.47599999999999998,"5min":0.373,"15min":0.316},"max":{"1min":1.5169999999999999,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.79200000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65800000000000003,"15min":0.66100000000000003},"min":{"1min":0.39800000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.161,"5min":1.161,"15min":2.5030000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64200000000000002,"15min":0.69799999999999995},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.399,"5min":1.399,"15min":6.4909999999999997},"last":0.68100000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67300000000000004,"15min":0.68899999999999995},"min":{"1min":0.436,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.105,"5min":1.169,"15min":1.544},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66600000000000004,"15min":0.69399999999999995},"min":{"1min":0.504,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.605},"last":0.74399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.65700000000000003,"15min":0.67500000000000004},"min":{"1min":0.48599999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.6539999999999999,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.69499999999999995,"15min":0.73299999999999998},"min":{"1min":0.47499999999999998,"5min":0.437,"15min":0.437},"max":{"1min":1.2649999999999999,"5min":1.274,"15min":2.8239999999999998},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.44600000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4910000000000001,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.66100000000000003},"min":{"1min":0.35499999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2170000000000001,"5min":1.234,"15min":1.234},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.623,"15min":0.65800000000000003},"min":{"1min":0.443,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":1.135,"5min":1.135,"15min":1.143},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.751},"min":{"1min":0.51900000000000002,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":1.915,"5min":1.915,"15min":1.915},"last":0.90800000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:21:22 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.69699999999999995,"15min":0.73099999999999998},"min":{"1min":0.53700000000000003,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.69599999999999995,"15min":0.73599999999999999},"min":{"1min":0.52200000000000002,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.607,"5min":1.607,"15min":1.607},"last":0.94499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575016,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65420,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306420,"statfs":{"total":96636764160,"available":96569774080,"internally_reserved":0,"allocated":2420736,"data_stored":1067687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.63900000000000001,"15min":0.66800000000000004},"min":{"1min":0.499,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.95299999999999996,"5min":1.0209999999999999,"15min":3.6629999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.74099999999999999,"15min":0.73599999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.97399999999999998,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.78600000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.59999999999999998,"15min":0.65300000000000002},"min":{"1min":0.443,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":3.7040000000000002},"last":0.626},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.64100000000000001,"15min":0.66400000000000003},"min":{"1min":0.45500000000000002,"5min":0.375,"15min":0.36299999999999999},"max":{"1min":0.90700000000000003,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.72099999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70299999999999996,"15min":0.67900000000000005},"min":{"1min":0.53700000000000003,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.84199999999999997,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64000000000000001,"15min":0.68200000000000005},"min":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.93400000000000005,"15min":3.2629999999999999},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67300000000000004,"15min":0.70599999999999996},"min":{"1min":0.53200000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94899999999999995,"5min":1.167,"15min":3.6520000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.66400000000000003,"15min":0.69999999999999996},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.86399999999999999,"5min":0.91100000000000003,"15min":3.742},"last":0.746}]},{"osd":4,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.61199999999999999,"15min":0.621},"min":{"1min":0.5,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.6439999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.62,"15min":0.64500000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.29199999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":4.7750000000000004},"last":0.75600000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66100000000000003,"15min":0.69099999999999995},"min":{"1min":0.48599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.91000000000000003,"5min":1.304,"15min":2.3700000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.63100000000000001,"15min":0.67100000000000004},"min":{"1min":0.39200000000000002,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":3.609},"last":0.66900000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:21:26 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68700000000000006,"15min":0.72899999999999998},"min":{"1min":0.379,"5min":0.378,"15min":0.378},"max":{"1min":0.93400000000000005,"5min":0.93700000000000006,"15min":4.9669999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.749,"5min":0.754,"15min":0.78500000000000003},"min":{"1min":0.60599999999999998,"5min":0.52300000000000002,"15min":0.46800000000000003},"max":{"1min":1.016,"5min":1.238,"15min":3.8100000000000001},"last":0.84599999999999997}]}]},{"osd":5,"up_from":13,"seq":55834575015,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59728,"kb_used_data":2364,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312112,"statfs":{"total":96636764160,"available":96575602688,"internally_reserved":0,"allocated":2420736,"data_stored":1067687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.64300000000000002,"15min":0.76600000000000001},"min":{"1min":0.53800000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81100000000000005,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.65600000000000003,"15min":0.73699999999999999},"min":{"1min":0.44600000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.90700000000000003,"5min":0.91800000000000004,"15min":21.376999999999999},"last":0.56999999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.65600000000000003,"15min":0.72099999999999997},"min":{"1min":0.41999999999999998,"5min":0.38800000000000001,"15min":0.30499999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.874},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67400000000000004,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":0.91500000000000004,"5min":1.482,"15min":7.7370000000000001},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.73599999999999999},"min":{"1min":0.505,"5min":0.379,"15min":0.379},"max":{"1min":0.88800000000000001,"5min":1.0489999999999999,"15min":17.268000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.66100000000000003,"15min":0.72599999999999998},"min":{"1min":0.47699999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.76000000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68600000000000005,"15min":1.083},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":0.91200000000000003,"5min":0.95099999999999996,"15min":113.363},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.45800000000000002,"5min":0.433,"15min":0.433},"max":{"1min":0.88100000000000001,"5min":1.113,"15min":4.1180000000000003},"last":0.95199999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.622},"min":{"1min":0.36699999999999999,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":1.0760000000000001,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64100000000000001,"15min":0.68200000000000005},"min":{"1min":0.38800000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.89900000000000002,"5min":1.03,"15min":2.827},"last":0.89800000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.65800000000000003,"15min":0.69099999999999995},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.871,"5min":1.73,"15min":2.2320000000000002},"last":0.99099999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68700000000000006,"15min":0.71499999999999997},"min":{"1min":0.47099999999999997,"5min":0.42599999999999999,"15min":0.40200000000000002},"max":{"1min":1.0249999999999999,"5min":1.532,"15min":4.1500000000000004},"last":0.92600000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:21:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.72399999999999998,"15min":0.751},"min":{"1min":0.52900000000000003,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.95899999999999996,"5min":1.0700000000000001,"15min":4.1029999999999998},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73499999999999999,"15min":0.75600000000000001},"min":{"1min":0.55300000000000005,"5min":0.44,"15min":0.44},"max":{"1min":0.97899999999999998,"5min":1.363,"15min":2.02},"last":0.91400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575014,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73632,"kb_used_data":2380,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298208,"statfs":{"total":96636764160,"available":96561364992,"internally_reserved":0,"allocated":2437120,"data_stored":1068773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":3.2959999999999998,"5min":1.165,"15min":0.82399999999999995},"min":{"1min":0.40999999999999998,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":46.860999999999997,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.498},{"interface":"front","average":{"1min":3.1190000000000002,"5min":1.1739999999999999,"15min":0.874},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":45.140000000000001,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":1.601,"5min":0.83299999999999996,"15min":0.71699999999999997},"min":{"1min":0.442,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":18.071000000000002,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.70399999999999996,"15min":0.66200000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":8.3320000000000007,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.68300000000000005,"15min":0.66600000000000004},"min":{"1min":0.44900000000000001,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":3.1190000000000002,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.67900000000000005,"15min":0.70399999999999996},"min":{"1min":0.44600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":2.653,"5min":1.071,"15min":0.80900000000000005},"min":{"1min":0.437,"5min":0.378,"15min":0.378},"max":{"1min":36.725999999999999,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.749,"15min":0.71799999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.70299999999999996,"15min":0.65600000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":7.2610000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.63600000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.69599999999999995,"15min":0.67700000000000005},"min":{"1min":0.41699999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":2.976,"5min":2.976,"15min":2.976},"last":0.625},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.77700000000000002,"15min":0.74299999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.60599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:21:18 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.71499999999999997,"15min":0.70799999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":2.3780000000000001,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.78200000000000003,"15min":0.752},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":2.9129999999999998,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.71899999999999997}]}]},{"osd":3,"up_from":13,"seq":55834575015,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69540,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302300,"statfs":{"total":96636764160,"available":96565555200,"internally_reserved":0,"allocated":2441216,"data_stored":1073587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41999999999999998,"15min":0.434},"min":{"1min":0.28999999999999998,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":0.496,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.45700000000000002,"15min":0.48899999999999999},"min":{"1min":0.30599999999999999,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.58699999999999997,"5min":0.66400000000000003,"15min":2.96},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47599999999999998,"15min":0.505},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.59999999999999998,"5min":0.69099999999999995,"15min":3.5600000000000001},"last":0.443},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.55700000000000005,"15min":0.61399999999999999},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.61899999999999999,"5min":0.79200000000000004,"15min":3.6829999999999998},"last":0.495}]},{"osd":2,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.51900000000000002,"15min":0.52500000000000002},"min":{"1min":0.38100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78800000000000003,"5min":0.84999999999999998,"15min":1.9590000000000001},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.52800000000000002,"15min":0.55700000000000005},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.34499999999999997},"max":{"1min":0.57699999999999996,"5min":0.77500000000000002,"15min":1.3819999999999999},"last":0.52700000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.747},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":0.90400000000000003,"5min":1.21,"15min":2.8610000000000002},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.73499999999999999,"15min":0.755},"min":{"1min":0.51200000000000001,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":0.90800000000000003,"5min":1.23,"15min":1.3640000000000001},"last":0.74199999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.73199999999999998,"15min":0.76100000000000001},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91200000000000003,"5min":1.323,"15min":1.391},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72499999999999998,"15min":0.76500000000000001},"min":{"1min":0.503,"5min":0.442,"15min":0.442},"max":{"1min":0.83799999999999997,"5min":1.224,"15min":2.9300000000000002},"last":0.67000000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.76200000000000001},"min":{"1min":0.52900000000000003,"5min":0.48699999999999999,"15min":0.47199999999999998},"max":{"1min":0.90900000000000003,"5min":1.21,"15min":1.401},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76700000000000002,"15min":0.77800000000000002},"min":{"1min":0.57699999999999996,"5min":0.53000000000000003,"15min":0.46300000000000002},"max":{"1min":0.88100000000000001,"5min":1.25,"15min":1.3300000000000001},"last":0.72699999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76200000000000001,"15min":0.79600000000000004},"min":{"1min":0.53700000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93400000000000005,"5min":1.27,"15min":2.899},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77300000000000002,"15min":0.79500000000000004},"min":{"1min":0.55400000000000005,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.90100000000000002,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.878}]}]},{"osd":2,"up_from":13,"seq":55834575015,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47428,"kb_used_data":2356,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324412,"statfs":{"total":96636764160,"available":96588197888,"internally_reserved":0,"allocated":2412544,"data_stored":1066086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.46999999999999997,"15min":0.435},"min":{"1min":0.27300000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.621,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.31},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.45100000000000001,"15min":0.56799999999999995},"min":{"1min":0.29799999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.623,"5min":1.593,"15min":30.023},"last":0.432}]},{"osd":1,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.53400000000000003,"15min":1.365},"min":{"1min":0.34300000000000003,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.80400000000000005,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.52100000000000002,"15min":1.4610000000000001},"min":{"1min":0.34999999999999998,"5min":0.32500000000000001,"15min":0.316},"max":{"1min":0.73499999999999999,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.38}]},{"osd":3,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.53600000000000003,"15min":0.61599999999999999},"min":{"1min":0.33400000000000002,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.76500000000000001,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.65400000000000003,"15min":0.60599999999999998},"min":{"1min":0.45000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.88400000000000001,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.56699999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.78000000000000003,"15min":0.80800000000000005},"min":{"1min":0.55100000000000005,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.94899999999999995,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.78900000000000003,"15min":0.85899999999999999},"min":{"1min":0.54200000000000004,"5min":0.48799999999999999,"15min":0.371},"max":{"1min":0.91200000000000003,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.71699999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.76500000000000001,"15min":0.878},"min":{"1min":0.54700000000000004,"5min":0.44500000000000001,"15min":0.376},"max":{"1min":0.89900000000000002,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.78900000000000003,"15min":0.77900000000000003},"min":{"1min":0.52400000000000002,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":0.93700000000000006,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.73499999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.79600000000000004,"15min":0.81899999999999995},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":0.91300000000000003,"5min":6.4800000000000004,"15min":10.664},"last":0.626},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.78300000000000003,"15min":0.89100000000000001},"min":{"1min":0.53500000000000003,"5min":0.503,"15min":0.47999999999999998},"max":{"1min":0.97999999999999998,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.83299999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:21:13 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.77400000000000002,"15min":0.82099999999999995},"min":{"1min":0.57799999999999996,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":0.999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.752},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.79900000000000004,"15min":0.90400000000000003},"min":{"1min":0.52300000000000002,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":0.95699999999999996,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.76200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91668,"kb_used_data":2368,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280172,"statfs":{"total":96636764160,"available":96542896128,"internally_reserved":0,"allocated":2424832,"data_stored":1067161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.434,"15min":0.46999999999999997},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":5.4939999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48699999999999999,"15min":0.51800000000000002},"min":{"1min":0.39900000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":2.3119999999999998},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.57699999999999996,"15min":0.65800000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.376},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":16.382999999999999},"last":0.5},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.53400000000000003,"15min":0.60299999999999998},"min":{"1min":0.35499999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81299999999999994,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.442}]},{"osd":4,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.76000000000000001,"15min":0.78900000000000003},"min":{"1min":0.47199999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.28,"5min":1.28,"15min":6.1630000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.79700000000000004,"15min":0.80400000000000005},"min":{"1min":0.56699999999999995,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.23,"5min":1.23,"15min":1.78},"last":0.86699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.77100000000000002,"15min":0.76800000000000002},"min":{"1min":0.52200000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.159,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.75800000000000001,"15min":0.78400000000000003},"min":{"1min":0.50600000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.175,"5min":1.175,"15min":6.1189999999999998},"last":0.747}]},{"osd":6,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.77100000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.77500000000000002,"15min":0.80400000000000005},"min":{"1min":0.503,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.246,"5min":1.246,"15min":6.1970000000000001},"last":0.88600000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:21:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.78600000000000003,"15min":0.79400000000000004},"min":{"1min":0.58999999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.1359999999999999,"5min":1.1759999999999999,"15min":1.8320000000000001},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.80200000000000005,"15min":0.81200000000000006},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.76400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575016,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70092,"kb_used_data":2940,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301748,"statfs":{"total":96636764160,"available":96564989952,"internally_reserved":0,"allocated":3010560,"data_stored":1656544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.432,"15min":0.51400000000000001},"min":{"1min":0.27600000000000002,"5min":0.247,"15min":0.247},"max":{"1min":0.56599999999999995,"5min":1.986,"15min":5.5259999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.48599999999999999,"15min":0.55200000000000005},"min":{"1min":0.32400000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.56100000000000005,"5min":2.371,"15min":5.4889999999999999},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.499,"15min":0.57599999999999996},"min":{"1min":0.33100000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.61199999999999999,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52100000000000002,"15min":0.58199999999999996},"min":{"1min":0.377,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.60999999999999999,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.434}]},{"osd":3,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47799999999999998,"15min":0.54900000000000004},"min":{"1min":0.36899999999999999,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.63600000000000001,"5min":0.878,"15min":4.7549999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55100000000000005,"15min":0.61699999999999999},"min":{"1min":0.39000000000000001,"5min":0.375,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.50700000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.76400000000000001,"15min":0.81100000000000005},"min":{"1min":0.53100000000000003,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.079,"5min":3.8849999999999998,"15min":4.734},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.72599999999999998,"15min":0.80000000000000004},"min":{"1min":0.46999999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0009999999999999,"5min":1.3109999999999999,"15min":4.8079999999999998},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.73599999999999999,"15min":0.81100000000000005},"min":{"1min":0.51600000000000001,"5min":0.47799999999999998,"15min":0.44500000000000001},"max":{"1min":1.0980000000000001,"5min":1.1839999999999999,"15min":4.944},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.76300000000000001,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.432,"15min":0.432},"max":{"1min":1.139,"5min":1.234,"15min":4.9710000000000001},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.76400000000000001,"15min":0.86099999999999999},"min":{"1min":0.52600000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0489999999999999,"5min":1.161,"15min":4.8470000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.79200000000000004,"15min":0.83699999999999997},"min":{"1min":0.50600000000000001,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.133,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.61399999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:21:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.83099999999999996,"15min":0.86299999999999999},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.1080000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.77900000000000003,"15min":0.84299999999999997},"min":{"1min":0.58599999999999997,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":1.0649999999999999,"5min":1.2649999999999999,"15min":4.9950000000000001},"last":0.68600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:21:41.354 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:21:41.354 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:21:41.496 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:21:41.500 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:21:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.552+0000 7f3ea3e00640 1 Processor -- start 2024-09-13T17:21:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.552+0000 7f3ea3e00640 1 -- start start 2024-09-13T17:21:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea415c890 0x7f3ea40cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea40d0370 0x7f3ea4069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea415cc70 con 0x7f3ea406a3c0 2024-09-13T17:21:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea3e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea406a830 con 0x7f3ea40d0370 2024-09-13T17:21:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea40d08e0 con 0x7f3ea415c890 2024-09-13T17:21:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea40d0370 0x7f3ea4069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea2000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3ea40d0370 0x7f3ea4069e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39968/0 (socket says 172.21.15.28:39968) 2024-09-13T17:21:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.553+0000 7f3ea2000640 1 -- 172.21.15.28:0/3383722524 learned_addr learned my addr 172.21.15.28:0/3383722524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea2a00640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea415c890 0x7f3ea40cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea2000640 1 -- 172.21.15.28:0/3383722524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea415c890 msgr2=0x7f3ea40cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea2000640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea415c890 0x7f3ea40cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea2000640 1 -- 172.21.15.28:0/3383722524 >> v1:172.21.15.28:6789/0 conn(0x7f3ea406a3c0 legacy=0x7f3ea40b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea2000640 1 -- 172.21.15.28:0/3383722524 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40d0bc0 con 0x7f3ea40d0370 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.554+0000 7f3ea1600640 1 -- 172.21.15.28:0/3383722524 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422875846 0 0) 0x7f3ea415cc70 con 0x7f3ea406a3c0 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.555+0000 7f3ea2000640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40d0370 0x7f3ea4069e80 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009870 tx=0x7f3e9002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61f80ec8417b5fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.555+0000 7f3ea1600640 1 -- 172.21.15.28:0/3383722524 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90032640 con 0x7f3ea40d0370 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.555+0000 7f3ea1600640 1 -- 172.21.15.28:0/3383722524 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e900327a0 con 0x7f3ea40d0370 2024-09-13T17:21:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.555+0000 7f3ea1600640 1 -- 172.21.15.28:0/3383722524 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90032b30 con 0x7f3ea40d0370 2024-09-13T17:21:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 -- 172.21.15.28:0/3383722524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40d0370 msgr2=0x7f3ea4069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40d0370 0x7f3ea4069e80 secure :-1 s=READY pgs=2318 cs=0 l=1 rev1=1 crypto rx=0x7f3e90009870 tx=0x7f3e9002a300 comp rx=0 tx=0).stop 2024-09-13T17:21:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 -- 172.21.15.28:0/3383722524 shutdown_connections 2024-09-13T17:21:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea40d0370 0x7f3ea4069e80 unknown :-1 s=CLOSED pgs=2318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/3383722524 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea415c890 0x7f3ea40cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 -- 172.21.15.28:0/3383722524 >> 172.21.15.28:0/3383722524 conn(0x7f3ea40708c0 msgr2=0x7f3ea4070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 -- 172.21.15.28:0/3383722524 shutdown_connections 2024-09-13T17:21:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.556+0000 7f3ea3e00640 1 -- 172.21.15.28:0/3383722524 wait complete. 2024-09-13T17:21:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.558+0000 7f3ea3e00640 1 Processor -- start 2024-09-13T17:21:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.558+0000 7f3ea3e00640 1 -- start start 2024-09-13T17:21:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.558+0000 7f3ea3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea406a3c0 0x7f3ea4150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 0x7f3ea40c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4151f20 con 0x7f3ea415c890 2024-09-13T17:21:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4148000 con 0x7f3ea406a3c0 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea4151c40 con 0x7f3ea40d0370 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39972/0 (socket says 172.21.15.28:39972) 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea406a3c0 0x7f3ea4150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 0x7f3ea40c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3400640 1 -- 172.21.15.28:0/1356202975 learned_addr learned my addr 172.21.15.28:0/1356202975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 -- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea406a3c0 msgr2=0x7f3ea4150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea406a3c0 0x7f3ea4150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 -- 172.21.15.28:0/1356202975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 msgr2=0x7f3ea41533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea2000640 1 -- 172.21.15.28:0/1356202975 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900094d0 con 0x7f3ea40d0370 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.559+0000 7f3ea3400640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3ea2000640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 0x7f3ea40c8f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40ccda0 tx=0x7f3e900046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90030850 con 0x7f3ea40d0370 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e900309b0 con 0x7f3ea40d0370 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e90030d60 con 0x7f3ea40d0370 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea4154180 con 0x7f3ea40d0370 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.560+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea4154740 con 0x7f3ea40d0370 2024-09-13T17:21:41.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.561+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea4069e80 con 0x7f3ea40d0370 2024-09-13T17:21:41.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.562+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e90039040 con 0x7f3ea40d0370 2024-09-13T17:21:41.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.563+0000 7f3e9aa00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 0x7f3e8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.563+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e900a9af0 con 0x7f3ea40d0370 2024-09-13T17:21:41.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.563+0000 7f3ea2a00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 0x7f3e8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.564+0000 7f3ea2a00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 0x7f3e8406d0c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009aa0 tx=0x7f3e94009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:41.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.568+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3e90074ce0 con 0x7f3ea40d0370 2024-09-13T17:21:41.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.815+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f3ea4154900 con 0x7f3ea40d0370 2024-09-13T17:21:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.822+0000 7f3e9aa00640 1 -- 172.21.15.28:0/1356202975 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v26) v1 ==== 190+0+0 (secure 0 0 0) 0x7f3e900370e0 con 0x7f3ea40d0370 2024-09-13T17:21:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:21:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 msgr2=0x7f3e8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 0x7f3e8406d0c0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f3e94009aa0 tx=0x7f3e94009190 comp rx=0 tx=0).stop 2024-09-13T17:21:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 msgr2=0x7f3ea40c8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 0x7f3ea40c8f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ea40ccda0 tx=0x7f3e900046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 shutdown_connections 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e8406ac00 0x7f3e8406d0c0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3ea415c890 0x7f3ea41533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3ea40d0370 0x7f3ea40c8f50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 --2- 172.21.15.28:0/1356202975 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ea406a3c0 0x7f3ea4150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 >> 172.21.15.28:0/1356202975 conn(0x7f3ea40708c0 msgr2=0x7f3ea4159100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 shutdown_connections 2024-09-13T17:21:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.824+0000 7f3ea3e00640 1 -- 172.21.15.28:0/1356202975 wait complete. 2024-09-13T17:21:41.842 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:21:41.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.995+0000 7fee5b200640 1 Processor -- start 2024-09-13T17:21:41.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.996+0000 7fee5b200640 1 -- start start 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee5b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee5406b360 0x7fee540b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee5b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee540b0d80 con 0x7fee54154a90 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee5b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5406b900 con 0x7fee540b0980 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee5b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee5406bbe0 con 0x7fee5406b360 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee59400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fee5406b360 0x7fee540b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee59e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fee54154a90 0x7fee5415b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58480/0 (socket says 172.21.15.28:58480) 2024-09-13T17:21:41.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee59e00640 1 -- 172.21.15.28:0/59772179 learned_addr learned my addr 172.21.15.28:0/59772179 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.997+0000 7fee59400640 1 -- 172.21.15.28:0/59772179 >> v1:172.21.15.189:6789/0 conn(0x7fee540b0980 legacy=0x7fee540af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.998+0000 7fee59400640 1 -- 172.21.15.28:0/59772179 >> v1:172.21.15.28:6789/0 conn(0x7fee54154a90 legacy=0x7fee5415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.998+0000 7fee58a00640 1 -- 172.21.15.28:0/59772179 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286672933 0 0) 0x7fee540b0d80 con 0x7fee54154a90 2024-09-13T17:21:42.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.998+0000 7fee59400640 1 -- 172.21.15.28:0/59772179 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5406bec0 con 0x7fee5406b360 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee59400640 1 --2- 172.21.15.28:0/59772179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee540b0440 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fee440097a0 tx=0x7fee4402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f14ba351749a2b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee58a00640 1 -- 172.21.15.28:0/59772179 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44032440 con 0x7fee5406b360 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee58a00640 1 -- 172.21.15.28:0/59772179 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee440325a0 con 0x7fee5406b360 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee58a00640 1 -- 172.21.15.28:0/59772179 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee44032930 con 0x7fee5406b360 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee5b200640 1 -- 172.21.15.28:0/59772179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 msgr2=0x7fee540b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee5b200640 1 --2- 172.21.15.28:0/59772179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee540b0440 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7fee440097a0 tx=0x7fee4402a230 comp rx=0 tx=0).stop 2024-09-13T17:21:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee5b200640 1 -- 172.21.15.28:0/59772179 shutdown_connections 2024-09-13T17:21:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee5b200640 1 --2- 172.21.15.28:0/59772179 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee540b0440 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:41.999+0000 7fee5b200640 1 -- 172.21.15.28:0/59772179 >> 172.21.15.28:0/59772179 conn(0x7fee54072e40 msgr2=0x7fee54073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.000+0000 7fee5b200640 1 -- 172.21.15.28:0/59772179 shutdown_connections 2024-09-13T17:21:42.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.000+0000 7fee5b200640 1 -- 172.21.15.28:0/59772179 wait complete. 2024-09-13T17:21:42.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.001+0000 7fee5b200640 1 Processor -- start 2024-09-13T17:21:42.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.001+0000 7fee5b200640 1 -- start start 2024-09-13T17:21:42.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee540b0980 0x7fee540af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 0x7fee540afa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee540ab8a0 con 0x7fee5406b360 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee540abb80 con 0x7fee54154a90 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee5b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee540abe60 con 0x7fee540b0980 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.002+0000 7fee59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39986/0 (socket says 172.21.15.28:39986) 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 -- 172.21.15.28:0/1619120189 learned_addr learned my addr 172.21.15.28:0/1619120189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59400640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee540b0980 0x7fee540af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee5a800640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 0x7fee540afa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 -- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee540b0980 msgr2=0x7fee540af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee540b0980 0x7fee540af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 -- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 msgr2=0x7fee540afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 0x7fee540afa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 -- 172.21.15.28:0/1619120189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44009400 con 0x7fee5406b360 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee5a800640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 0x7fee540afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.003+0000 7fee59e00640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c3f0 tx=0x7fee4800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.004+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee480074e0 con 0x7fee5406b360 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.004+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee48007640 con 0x7fee5406b360 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.004+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee48004a00 con 0x7fee5406b360 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.004+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee540ac140 con 0x7fee5406b360 2024-09-13T17:21:42.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.004+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee540aab00 con 0x7fee5406b360 2024-09-13T17:21:42.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.005+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5413ef30 con 0x7fee5406b360 2024-09-13T17:21:42.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.006+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee48004b60 con 0x7fee5406b360 2024-09-13T17:21:42.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.007+0000 7fee4e000640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 0x7fee3806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.007+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee48089090 con 0x7fee5406b360 2024-09-13T17:21:42.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.007+0000 7fee59400640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 0x7fee3806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.008+0000 7fee59400640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 0x7fee3806d0f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fee44034000 tx=0x7fee4402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.011+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee48054270 con 0x7fee5406b360 2024-09-13T17:21:42.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.223+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee540b0440 con 0x7fee5406b360 2024-09-13T17:21:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.224+0000 7fee4e000640 1 -- 172.21.15.28:0/1619120189 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fee48057890 con 0x7fee5406b360 2024-09-13T17:21:42.226 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:42.226 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:21:42.226 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 msgr2=0x7fee3806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 0x7fee3806d0f0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fee44034000 tx=0x7fee4402e000 comp rx=0 tx=0).stop 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 msgr2=0x7fee5413d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 secure :-1 s=READY pgs=2321 cs=0 l=1 rev1=1 crypto rx=0x7fee4800c3f0 tx=0x7fee4800c8b0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 shutdown_connections 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee3806ac30 0x7fee3806d0f0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee54154a90 0x7fee540afa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fee540b0980 0x7fee540af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 --2- 172.21.15.28:0/1619120189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee5406b360 0x7fee5413d040 unknown :-1 s=CLOSED pgs=2321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 >> 172.21.15.28:0/1619120189 conn(0x7fee54072e40 msgr2=0x7fee54159a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 shutdown_connections 2024-09-13T17:21:42.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.226+0000 7fee5b200640 1 -- 172.21.15.28:0/1619120189 wait complete. 2024-09-13T17:21:42.279 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:21:42.277+0000 7f9981400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:21:42.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 165, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:21:36.721122+0000', 'modified': '2024-09-13T17:21:39.737482+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11049}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11049': {'gid': 11049, 'name': 'c', 'rank': 0, 'incarnation': 161, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/762464937', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6834', 'nonce': 762464937}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 762464937}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:21:42.647 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:21:42.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:21:42.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.803+0000 7f4277800640 1 Processor -- start 2024-09-13T17:21:42.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.803+0000 7f4277800640 1 -- start start 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.804+0000 7f4277800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 0x7f42700ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.804+0000 7f4277800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42700ad940 0x7f427006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.804+0000 7f4277800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4270070470 con 0x7f427015c9a0 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.804+0000 7f4277800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42700ace30 con 0x7f42700ad940 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.804+0000 7f4277800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42700ad110 con 0x7f42700b2400 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42700ad940 0x7f427006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f42700ad940 0x7f427006bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39996/0 (socket says 172.21.15.28:39996) 2024-09-13T17:21:42.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 -- 172.21.15.28:0/2021769764 learned_addr learned my addr 172.21.15.28:0/2021769764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f426fe00640 1 --2- 172.21.15.28:0/2021769764 >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 0x7f42700ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 -- 172.21.15.28:0/2021769764 >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 msgr2=0x7f42700ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 --2- 172.21.15.28:0/2021769764 >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 0x7f42700ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 -- 172.21.15.28:0/2021769764 >> v1:172.21.15.189:6789/0 conn(0x7f427015c9a0 legacy=0x7f42700b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f4275800640 1 -- 172.21.15.28:0/2021769764 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427006b170 con 0x7f42700ad940 2024-09-13T17:21:42.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.805+0000 7f426fe00640 1 --2- 172.21.15.28:0/2021769764 >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 0x7f42700ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.806+0000 7f4275800640 1 --2- 172.21.15.28:0/2021769764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700ad940 0x7f427006bc50 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f42700ce450 tx=0x7f426802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c279b0d1fcabeaad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.806+0000 7f426f400640 1 -- 172.21.15.28:0/2021769764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268031600 con 0x7f42700ad940 2024-09-13T17:21:42.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.806+0000 7f426f400640 1 -- 172.21.15.28:0/2021769764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426802d050 con 0x7f42700ad940 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.806+0000 7f426f400640 1 -- 172.21.15.28:0/2021769764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4268031980 con 0x7f42700ad940 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 -- 172.21.15.28:0/2021769764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700ad940 msgr2=0x7f427006bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 --2- 172.21.15.28:0/2021769764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700ad940 0x7f427006bc50 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f42700ce450 tx=0x7f426802a6f0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 -- 172.21.15.28:0/2021769764 shutdown_connections 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 --2- 172.21.15.28:0/2021769764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700ad940 0x7f427006bc50 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 --2- 172.21.15.28:0/2021769764 >> v2:172.21.15.189:3300/0 conn(0x7f42700b2400 0x7f42700ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 -- 172.21.15.28:0/2021769764 >> 172.21.15.28:0/2021769764 conn(0x7f42700b0270 msgr2=0x7f42700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 -- 172.21.15.28:0/2021769764 shutdown_connections 2024-09-13T17:21:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.807+0000 7f4277800640 1 -- 172.21.15.28:0/2021769764 wait complete. 2024-09-13T17:21:42.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.809+0000 7f4277800640 1 Processor -- start 2024-09-13T17:21:42.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.809+0000 7f4277800640 1 -- start start 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.809+0000 7f4277800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42700ad940 0x7f4270155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4277800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4277800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 0x7f42700cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4277800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42700cca80 con 0x7f42700b2400 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4277800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42701569e0 con 0x7f427015c9a0 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4277800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4270152bd0 con 0x7f42700ad940 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40000/0 (socket says 172.21.15.28:40000) 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 -- 172.21.15.28:0/3526509679 learned_addr learned my addr 172.21.15.28:0/3526509679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4274e00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42700ad940 0x7f4270155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4275800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 0x7f42700cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 -- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42700ad940 msgr2=0x7f4270155d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42700ad940 0x7f4270155d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 -- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 msgr2=0x7f42700cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 0x7f42700cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f426fe00640 1 -- 172.21.15.28:0/3526509679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4268009b30 con 0x7f42700b2400 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.810+0000 7f4275800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 0x7f42700cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f426fe00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f425c00e730 tx=0x7f425c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f426cc00640 1 -- 172.21.15.28:0/3526509679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c004230 con 0x7f42700b2400 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f426cc00640 1 -- 172.21.15.28:0/3526509679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c004390 con 0x7f42700b2400 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f426cc00640 1 -- 172.21.15.28:0/3526509679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c00c8f0 con 0x7f42700b2400 2024-09-13T17:21:42.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4270070930 con 0x7f42700b2400 2024-09-13T17:21:42.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.811+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42701564d0 con 0x7f42700b2400 2024-09-13T17:21:42.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.814+0000 7f426cc00640 1 -- 172.21.15.28:0/3526509679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f425c00ca50 con 0x7f42700b2400 2024-09-13T17:21:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.815+0000 7f426cc00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 0x7f425006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.815+0000 7f426cc00640 1 -- 172.21.15.28:0/3526509679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f425c088eb0 con 0x7f42700b2400 2024-09-13T17:21:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.815+0000 7f427a65c740 1 Processor -- start 2024-09-13T17:21:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.815+0000 7f427a65c740 1 -- start start 2024-09-13T17:21:42.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.815+0000 7f4274e00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 0x7f425006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f4274e00640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 0x7f425006d0c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4264006fd0 tx=0x7f4264009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e8d5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8d2e90 0x563f3e922ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e83eab0 con 0x563f3e8fcb80 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e83c240 con 0x563f3e8d56d0 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.816+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f3e837b30 con 0x563f3e8d2e90 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f426fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8d2e90 0x563f3e922ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40002/0 (socket says 172.21.15.28:40002) 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 -- 172.21.15.28:0/1470530443 learned_addr learned my addr 172.21.15.28:0/1470530443 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4274e00640 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e8d5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 -- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8d2e90 msgr2=0x563f3e922ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8d2e90 0x563f3e922ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 -- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 msgr2=0x563f3e8d5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e8d5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 -- 172.21.15.28:0/1470530443 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f3e802bd0 con 0x563f3e8fcb80 2024-09-13T17:21:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.817+0000 7f4275800640 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f4268009f40 tx=0x7f426802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34846bdaf1a1ff4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f4261600640 1 -- 172.21.15.28:0/1470530443 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426803c680 con 0x563f3e8fcb80 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f4261600640 1 -- 172.21.15.28:0/1470530443 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426803c7e0 con 0x563f3e8fcb80 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f4261600640 1 -- 172.21.15.28:0/1470530443 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426803cb50 con 0x563f3e8fcb80 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 -- 172.21.15.28:0/1470530443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 msgr2=0x563f3e9230e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 secure :-1 s=READY pgs=2324 cs=0 l=1 rev1=1 crypto rx=0x7f4268009f40 tx=0x7f426802e000 comp rx=0 tx=0).stop 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 -- 172.21.15.28:0/1470530443 shutdown_connections 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e9230e0 unknown :-1 s=CLOSED pgs=2324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8d2e90 0x563f3e922ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 --2- 172.21.15.28:0/1470530443 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e8d5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 -- 172.21.15.28:0/1470530443 >> 172.21.15.28:0/1470530443 conn(0x563f3e8d1e00 msgr2=0x563f3e91a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.818+0000 7f427a65c740 1 -- 172.21.15.28:0/1470530443 shutdown_connections 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.819+0000 7f427a65c740 1 -- 172.21.15.28:0/1470530443 wait complete. 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.819+0000 7f427a65c740 1 -- start start 2024-09-13T17:21:42.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.819+0000 7f427a65c740 1 Processor -- start 2024-09-13T17:21:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e9006d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e7d9260 con 0x563f3e8fe820 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e7d8f00 con 0x563f3e8d56d0 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f3e59ef20 con 0x563f3e8fcb80 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f427a65c740 10 client.0 _ll_register_callbacks cb 0x563f3e8311a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.820+0000 7f4275800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40004/0 (socket says 172.21.15.28:40004) 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 -- 172.21.15.28:0/1962379105 learned_addr learned my addr 172.21.15.28:0/1962379105 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f426fe00640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4274e00640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e9006d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 -- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 msgr2=0x563f3e8fe2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8fe2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 -- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 msgr2=0x563f3e9006d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e9006d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 -- 172.21.15.28:0/1962379105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427006b170 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f426fe00640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4275800640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f426802a730 tx=0x7f426802c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93019c1b515b94c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4256000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.821+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426803ce20 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f3e861cd0 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4268031480 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4268031640 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 10 client.11178 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x563f3e8361b0 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563f3e861cd0 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f42680317a0 con 0x563f3e8fe820 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 10 client.11178 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 10 client.11178 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:42.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 20 client.11178 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e8e9480 0x563f3e8e7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 4 client.11178 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.822+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f3e52b3c0 con 0x563f3e8e9480 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.823+0000 7f4274e00640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e8e9480 0x563f3e8e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.823+0000 7f4274e00640 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e8e9480 0x563f3e8e7910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=628005cc7bfaffe5 server_cookie=b99757a6179f276 in_seq=0 out_seq=0 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.823+0000 7f4256000640 10 client.11178 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f4256000640 1 -- 172.21.15.28:0/1962379105 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x563f3e52b3c0 con 0x563f3e8e9480 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f4256000640 10 client.11178 handle_command_reply: tid=0 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 client.11178 shutdown 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 msgr2=0x563f3e9174f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f426802a730 tx=0x7f426802c000 comp rx=0 tx=0).stop 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 shutdown_connections 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e8e9480 0x563f3e8e7910 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fe820 0x563f3e9174f0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 --2- 172.21.15.28:0/1962379105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e9006d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f4256000640 0 client.11178 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 >> 172.21.15.28:0/1962379105 conn(0x563f3e8d2e90 msgr2=0x563f3e91d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 shutdown_connections 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.824+0000 7f427a65c740 1 -- 172.21.15.28:0/1962379105 wait complete. 2024-09-13T17:21:42.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.825+0000 7f427a65c740 20 client.11178 trim_cache size 0 max 16384 2024-09-13T17:21:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.840+0000 7f427a65c740 1 Processor -- start 2024-09-13T17:21:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.840+0000 7f427a65c740 1 -- start start 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8d46f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 0x563f3e8e6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e8ecd20 con 0x563f3e8d56d0 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e93c940 con 0x563f3e8fe820 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f3e8fa9f0 con 0x563f3e8fcb80 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f4274e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f4274e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40016/0 (socket says 172.21.15.28:40016) 2024-09-13T17:21:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.841+0000 7f4274e00640 1 -- 172.21.15.28:0/1316597926 learned_addr learned my addr 172.21.15.28:0/1316597926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4275800640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 0x563f3e8e6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f426fe00640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8d46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 -- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 msgr2=0x563f3e8d46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8d46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 -- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 msgr2=0x563f3e8e6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 0x563f3e8e6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 -- 172.21.15.28:0/1316597926 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f3e86b170 con 0x563f3e8d56d0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4275800640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 0x563f3e8e6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f426fe00640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4274e00640 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f426400e5a0 tx=0x7f4264015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55621756e04da254 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4262000640 1 -- 172.21.15.28:0/1316597926 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426400edb0 con 0x563f3e8d56d0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4262000640 1 -- 172.21.15.28:0/1316597926 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4264016990 con 0x563f3e8d56d0 2024-09-13T17:21:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.842+0000 7f4262000640 1 -- 172.21.15.28:0/1316597926 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264016d20 con 0x563f3e8d56d0 2024-09-13T17:21:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 -- 172.21.15.28:0/1316597926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 msgr2=0x563f3e91ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 secure :-1 s=READY pgs=2326 cs=0 l=1 rev1=1 crypto rx=0x7f426400e5a0 tx=0x7f4264015000 comp rx=0 tx=0).stop 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 -- 172.21.15.28:0/1316597926 shutdown_connections 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8fe820 0x563f3e8e6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fcb80 0x563f3e8d46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 --2- 172.21.15.28:0/1316597926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8d56d0 0x563f3e91ad70 unknown :-1 s=CLOSED pgs=2326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 -- 172.21.15.28:0/1316597926 >> 172.21.15.28:0/1316597926 conn(0x563f3e8d2e90 msgr2=0x563f3e9004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 -- 172.21.15.28:0/1316597926 shutdown_connections 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.843+0000 7f427a65c740 1 -- 172.21.15.28:0/1316597926 wait complete. 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.844+0000 7f427a65c740 1 -- start start 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.844+0000 7f427a65c740 1 Processor -- start 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e981dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fe820 0x563f3e8e55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e984f50 con 0x563f3e8fcb80 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f3e8d48f0 con 0x563f3e8d56d0 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x563f3e8d4bd0 con 0x563f3e8fe820 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f427a65c740 10 client.0 _ll_register_callbacks cb 0x7f42640092d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40018/0 (socket says 172.21.15.28:40018) 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 -- 172.21.15.28:0/1214040957 learned_addr learned my addr 172.21.15.28:0/1214040957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 -- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fe820 msgr2=0x563f3e8e55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fe820 0x563f3e8e55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 -- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 msgr2=0x563f3e981dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e981dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.845+0000 7f426fe00640 1 -- 172.21.15.28:0/1214040957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426401a040 con 0x563f3e8fcb80 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f426fe00640 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f425c05fc00 tx=0x7f425c00ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e02214eb3a9204f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c05fca0 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f425c05fe00 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f426803c7e0 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f425c087110 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 10 client.11184 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x563f3e7d8470 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c074070 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f425c06bb80 con 0x563f3e8fcb80 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 10 client.11184 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 10 client.11184 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 20 client.11184 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.846+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e9332e0 0x563f3e9336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f427a65c740 4 client.11184 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:21:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563f3e53c6c0 con 0x563f3e9332e0 2024-09-13T17:21:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f4274e00640 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e9332e0 0x563f3e9336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f4274e00640 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e9332e0 0x563f3e9336e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c406c0891e7d1c2a server_cookie=53b3f66c49201d52 in_seq=0 out_seq=0 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f4256000640 10 client.11184 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.847+0000 7f4256000640 1 -- 172.21.15.28:0/1214040957 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+329 (crc 0 0 0) 0x563f3e53c6c0 con 0x563f3e9332e0 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f4256000640 10 client.11184 handle_command_reply: tid=0 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 client.11184 shutdown 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 msgr2=0x563f3e8e5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f425c05fc00 tx=0x7f425c00ef80 comp rx=0 tx=0).stop 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 shutdown_connections 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x563f3e9332e0 0x563f3e9336e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x563f3e8fe820 0x563f3e8e55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3e8fcb80 0x563f3e8e5080 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 --2- 172.21.15.28:0/1214040957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563f3e8d56d0 0x563f3e981dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 >> 172.21.15.28:0/1214040957 conn(0x563f3e8d2e90 msgr2=0x563f3e91ad70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f4256000640 0 client.11184 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 shutdown_connections 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 1 -- 172.21.15.28:0/1214040957 wait complete. 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.848+0000 7f427a65c740 20 client.11184 trim_cache size 0 max 16384 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 11049, 2024-09-13T17:21:42.854 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1252611659999996, 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 165, 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 193, 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 193, 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.277708398 2024-09-13T17:21:42.855 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 msgr2=0x7f425006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 0x7f425006d0c0 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4264006fd0 tx=0x7f4264009060 comp rx=0 tx=0).stop 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 msgr2=0x7f4270150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f425c00e730 tx=0x7f425c00ebf0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 shutdown_connections 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f425006ac00 0x7f425006d0c0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f427015c9a0 0x7f42700cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f42700b2400 0x7f4270150200 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 --2- 172.21.15.28:0/3526509679 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42700ad940 0x7f4270155d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 >> 172.21.15.28:0/3526509679 conn(0x7f42700b0270 msgr2=0x7f42700af850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 shutdown_connections 2024-09-13T17:21:42.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:42.849+0000 7f4277800640 1 -- 172.21.15.28:0/3526509679 wait complete. 2024-09-13T17:21:42.867 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:21:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.020+0000 7f323ba00640 1 Processor -- start 2024-09-13T17:21:43.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.020+0000 7f323ba00640 1 -- start start 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f323415a540 0x7f323406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32340cabc0 0x7f323415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323415edb0 0x7f3234070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32340adb00 con 0x7f323415edb0 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f323415e100 con 0x7f323415a540 2024-09-13T17:21:43.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.021+0000 7f323ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f323415e3e0 con 0x7f32340cabc0 2024-09-13T17:21:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323415edb0 0x7f3234070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f323415edb0 0x7f3234070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40020/0 (socket says 172.21.15.28:40020) 2024-09-13T17:21:43.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 -- 172.21.15.28:0/2458621 learned_addr learned my addr 172.21.15.28:0/2458621 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239000640 1 --2- 172.21.15.28:0/2458621 >> v2:172.21.15.189:3300/0 conn(0x7f323415a540 0x7f323406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3233e00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32340cabc0 0x7f323415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 -- 172.21.15.28:0/2458621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32340cabc0 msgr2=0x7f323415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32340cabc0 0x7f323415e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 -- 172.21.15.28:0/2458621 >> v2:172.21.15.189:3300/0 conn(0x7f323415a540 msgr2=0x7f323406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 --2- 172.21.15.28:0/2458621 >> v2:172.21.15.189:3300/0 conn(0x7f323415a540 0x7f323406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.022+0000 7f3239a00640 1 -- 172.21.15.28:0/2458621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f323406ce80 con 0x7f323415edb0 2024-09-13T17:21:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.023+0000 7f3239a00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415edb0 0x7f3234070cc0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f32340b3a00 tx=0x7f322c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99e230eaba6cce52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:43.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.023+0000 7f3233400640 1 -- 172.21.15.28:0/2458621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c031600 con 0x7f323415edb0 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.023+0000 7f3233400640 1 -- 172.21.15.28:0/2458621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f322c02d050 con 0x7f323415edb0 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.023+0000 7f3233400640 1 -- 172.21.15.28:0/2458621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322c031980 con 0x7f323415edb0 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 -- 172.21.15.28:0/2458621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415edb0 msgr2=0x7f3234070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415edb0 0x7f3234070cc0 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f32340b3a00 tx=0x7f322c02a610 comp rx=0 tx=0).stop 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 -- 172.21.15.28:0/2458621 shutdown_connections 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415edb0 0x7f3234070cc0 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 --2- 172.21.15.28:0/2458621 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32340cabc0 0x7f323415e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 --2- 172.21.15.28:0/2458621 >> v2:172.21.15.189:3300/0 conn(0x7f323415a540 0x7f323406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 -- 172.21.15.28:0/2458621 >> 172.21.15.28:0/2458621 conn(0x7f3234069e80 msgr2=0x7f323406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 -- 172.21.15.28:0/2458621 shutdown_connections 2024-09-13T17:21:43.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.024+0000 7f323ba00640 1 -- 172.21.15.28:0/2458621 wait complete. 2024-09-13T17:21:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.026+0000 7f323ba00640 1 Processor -- start 2024-09-13T17:21:43.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.026+0000 7f323ba00640 1 -- start start 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32340cabc0 0x7f323414b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323414b760 0x7f323414ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323414b5b0 con 0x7f323415a540 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32341f4570 con 0x7f32340cabc0 2024-09-13T17:21:43.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f323ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32341f4aa0 con 0x7f323414b760 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f3233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f3233e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40028/0 (socket says 172.21.15.28:40028) 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f3233e00640 1 -- 172.21.15.28:0/767199608 learned_addr learned my addr 172.21.15.28:0/767199608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f3239a00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323414b760 0x7f323414ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.027+0000 7f3239000640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32340cabc0 0x7f323414b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 -- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323414b760 msgr2=0x7f323414ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323414b760 0x7f323414ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 -- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32340cabc0 msgr2=0x7f323414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32340cabc0 0x7f323414b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 -- 172.21.15.28:0/767199608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c009b30 con 0x7f323415a540 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3233e00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f322000d5d0 tx=0x7f322000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220004240 con 0x7f323415a540 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32200043a0 con 0x7f323415a540 2024-09-13T17:21:43.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32341508f0 con 0x7f323415a540 2024-09-13T17:21:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3220003830 con 0x7f323415a540 2024-09-13T17:21:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.028+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323414d3c0 con 0x7f323415a540 2024-09-13T17:21:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.029+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32340ca9b0 con 0x7f323415a540 2024-09-13T17:21:43.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.030+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3220003990 con 0x7f323415a540 2024-09-13T17:21:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.032+0000 7f3230c00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 0x7f321406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:43.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.032+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3220089410 con 0x7f323415a540 2024-09-13T17:21:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.032+0000 7f3239000640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 0x7f321406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:43.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.033+0000 7f3239000640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 0x7f321406d0c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f32340ad820 tx=0x7f3228009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:43.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.036+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3220054690 con 0x7f323415a540 2024-09-13T17:21:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.253+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f323415e870 con 0x7f323415a540 2024-09-13T17:21:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.253+0000 7f3230c00640 1 -- 172.21.15.28:0/767199608 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v165) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3220057cb0 con 0x7f323415a540 2024-09-13T17:21:43.255 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 msgr2=0x7f321406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 0x7f321406d0c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f32340ad820 tx=0x7f3228009190 comp rx=0 tx=0).stop 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 msgr2=0x7f323414fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7f322000d5d0 tx=0x7f322000da90 comp rx=0 tx=0).stop 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 shutdown_connections 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f321406ac00 0x7f321406d0c0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f323414b760 0x7f323414ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f323415a540 0x7f323414fa00 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 --2- 172.21.15.28:0/767199608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32340cabc0 0x7f323414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.255+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 >> 172.21.15.28:0/767199608 conn(0x7f3234069e80 msgr2=0x7f32340cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.256+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 shutdown_connections 2024-09-13T17:21:43.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:43.256+0000 7f323ba00640 1 -- 172.21.15.28:0/767199608 wait complete. 2024-09-13T17:21:43.274 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:21:43.274 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:21:43.274 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:21:43.274 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:21:43.274 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:21:43.275 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:21:43.275 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:21:43.275 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:21:43.275 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:21:43.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.275 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:21:43.311 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:21:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:21:43.311 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:21:43.312 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41878sec preferred_lft 41878sec 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:21:43.313 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:21:43.314 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:21:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:21:43.315 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:21:43.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.315 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:21:43.405 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:21:43.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.409 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:21:43.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:21:43.446 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:21:43.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.591 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:21:43.626 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:21:43.626 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:21:43.627 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:21:43.627 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.627 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:21:43.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:21:43.723 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:21:43.758 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:21:43.758 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:21:43.759 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:21:43.833 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:21:43.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:21:43.962 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:21:43.962 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:21:43.989 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:21:43.990 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:21:43.990 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:21:43.990 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:21:43.990 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:21:43.992 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:21:43.992 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:21:43.992 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:21:43.992 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:21:43.992 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:21:43.995 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:21:43.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:43.995 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:21:44.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:21:44.051 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:21:44.097 INFO:teuthology.orchestra.run.smithi028.stdout:client8828 2024-09-13T17:21:44.097 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:21:44.098 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:21:44.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.13:0 damage ls 2024-09-13T17:21:44.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.370+0000 7f4edaa00640 1 Processor -- start 2024-09-13T17:21:44.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.370+0000 7f4edaa00640 1 -- start start 2024-09-13T17:21:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.371+0000 7f4edaa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.371+0000 7f4edaa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.371+0000 7f4edaa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed406c8b0 con 0x7f4ed406bb40 2024-09-13T17:21:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.371+0000 7f4edaa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed406cb90 con 0x7f4ed415a780 2024-09-13T17:21:44.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.371+0000 7f4edaa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40cc980 con 0x7f4ed415ab60 2024-09-13T17:21:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:45048/0 (socket says 172.21.15.28:45048) 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2725775152 learned_addr learned my addr 172.21.15.28:0/2725775152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3400640 1 -- 172.21.15.28:0/2725775152 >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 msgr2=0x7f4ed4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.372+0000 7f4ed3400640 1 --2- 172.21.15.28:0/2725775152 >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.373+0000 7f4ed3400640 1 -- 172.21.15.28:0/2725775152 >> v1:172.21.15.189:6789/0 conn(0x7f4ed406bb40 legacy=0x7f4ed406bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.373+0000 7f4ed3400640 1 -- 172.21.15.28:0/2725775152 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40ccc60 con 0x7f4ed415ab60 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.373+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2725775152 >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:21:44.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4ed3400640 1 --2- 172.21.15.28:0/2725775152 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed40712c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf5d6a751e7db7f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4ed2a00640 1 -- 172.21.15.28:0/2725775152 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0032440 con 0x7f4ed415ab60 2024-09-13T17:21:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4ed2a00640 1 -- 172.21.15.28:0/2725775152 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec00325a0 con 0x7f4ed415ab60 2024-09-13T17:21:44.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4ed2a00640 1 -- 172.21.15.28:0/2725775152 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0032930 con 0x7f4ed415ab60 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4edaa00640 1 -- 172.21.15.28:0/2725775152 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 msgr2=0x7f4ed40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.374+0000 7f4edaa00640 1 --2- 172.21.15.28:0/2725775152 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed40712c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f4ec00097a0 tx=0x7f4ec002a180 comp rx=0 tx=0).stop 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 -- 172.21.15.28:0/2725775152 shutdown_connections 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 --2- 172.21.15.28:0/2725775152 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed40712c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 --2- 172.21.15.28:0/2725775152 >> v2:172.21.15.189:3300/0 conn(0x7f4ed415a780 0x7f4ed4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 -- 172.21.15.28:0/2725775152 >> 172.21.15.28:0/2725775152 conn(0x7f4ed40b0270 msgr2=0x7f4ed40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 -- 172.21.15.28:0/2725775152 shutdown_connections 2024-09-13T17:21:44.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.375+0000 7f4edaa00640 1 -- 172.21.15.28:0/2725775152 wait complete. 2024-09-13T17:21:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.376+0000 7f4edaa00640 1 Processor -- start 2024-09-13T17:21:44.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.376+0000 7f4edaa00640 1 -- start start 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed415a780 0x7f4ed40a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed41579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40a8440 con 0x7f4ed406bb40 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed40a8c10 con 0x7f4ed415a780 2024-09-13T17:21:44.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.377+0000 7f4edaa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed4151480 con 0x7f4ed415ab60 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43396/0 (socket says 172.21.15.28:43396) 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 -- 172.21.15.28:0/873395989 learned_addr learned my addr 172.21.15.28:0/873395989 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3400640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed415a780 0x7f4ed40a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed41579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 -- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 msgr2=0x7f4ed41579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed41579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 -- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed415a780 msgr2=0x7f4ed40a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed415a780 0x7f4ed40a8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 -- 172.21.15.28:0/873395989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec40094d0 con 0x7f4ed406bb40 2024-09-13T17:21:44.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.378+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e9d0 tx=0x7f4ec400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.379+0000 7f4ecbe00640 1 -- 172.21.15.28:0/873395989 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec400ca60 con 0x7f4ed406bb40 2024-09-13T17:21:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.379+0000 7f4ecbe00640 1 -- 172.21.15.28:0/873395989 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec400cbc0 con 0x7f4ed406bb40 2024-09-13T17:21:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.379+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec0009400 con 0x7f4ed406bb40 2024-09-13T17:21:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.379+0000 7f4ecbe00640 1 -- 172.21.15.28:0/873395989 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4013470 con 0x7f4ed406bb40 2024-09-13T17:21:44.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.379+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40a8930 con 0x7f4ed406bb40 2024-09-13T17:21:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.381+0000 7f4ecbe00640 1 -- 172.21.15.28:0/873395989 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4ec40135d0 con 0x7f4ed406bb40 2024-09-13T17:21:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.382+0000 7f4ecbe00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 0x7f4eb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.382+0000 7f4ecbe00640 1 -- 172.21.15.28:0/873395989 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ec40895f0 con 0x7f4ed406bb40 2024-09-13T17:21:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.382+0000 7f4edd862740 1 Processor -- start 2024-09-13T17:21:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.383+0000 7f4ed3400640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 0x7f4eb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.383+0000 7f4edd862740 1 -- start start 2024-09-13T17:21:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.383+0000 7f4edd862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.383+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646d5d0 0x556dc64bd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc64972c0 0x556dc64bd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4edd862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc637c020 con 0x556dc646fe10 2024-09-13T17:21:44.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc63d9ba0 con 0x556dc646d5d0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556dc63d7330 con 0x556dc64972c0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3400640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 0x7f4eb406d0c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0034000 tx=0x7f4ec002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646d5d0 0x556dc64bd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43408/0 (socket says 172.21.15.28:43408) 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed8a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc64972c0 0x556dc64bd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2260002083 learned_addr learned my addr 172.21.15.28:0/2260002083 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc64972c0 msgr2=0x556dc64bd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc64972c0 0x556dc64bd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646d5d0 msgr2=0x556dc64bd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646d5d0 0x556dc64bd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.384+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2260002083 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dc6384cf0 con 0x556dc646fe10 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4058230 tx=0x7f4ec4057f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c415a3a341010fbd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4ec8c00640 1 -- 172.21.15.28:0/2260002083 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4060870 con 0x556dc646fe10 2024-09-13T17:21:44.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4ec8c00640 1 -- 172.21.15.28:0/2260002083 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4056670 con 0x556dc646fe10 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4ec8c00640 1 -- 172.21.15.28:0/2260002083 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4054c40 con 0x556dc646fe10 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 -- 172.21.15.28:0/2260002083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 msgr2=0x556dc64701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 secure :-1 s=READY pgs=2331 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4058230 tx=0x7f4ec4057f50 comp rx=0 tx=0).stop 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 -- 172.21.15.28:0/2260002083 shutdown_connections 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc64972c0 0x556dc64bd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646d5d0 0x556dc64bd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 --2- 172.21.15.28:0/2260002083 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646fe10 0x556dc64701f0 unknown :-1 s=CLOSED pgs=2331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.385+0000 7f4edd862740 1 -- 172.21.15.28:0/2260002083 >> 172.21.15.28:0/2260002083 conn(0x556dc646c540 msgr2=0x556dc64b49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.386+0000 7f4edd862740 1 -- 172.21.15.28:0/2260002083 shutdown_connections 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.386+0000 7f4edd862740 1 -- 172.21.15.28:0/2260002083 wait complete. 2024-09-13T17:21:44.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.386+0000 7f4edd862740 1 -- start start 2024-09-13T17:21:44.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.386+0000 7f4edd862740 1 Processor -- start 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc646a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64ba9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc63d2c20 con 0x556dc64972c0 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc63741b0 con 0x556dc646abf0 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556dc6373e50 con 0x556dc646fe10 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.387+0000 7f4edd862740 10 client.0 _ll_register_callbacks cb 0x556dc63cc290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:44.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43424/0 (socket says 172.21.15.28:43424) 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 -- 172.21.15.28:0/1187621372 learned_addr learned my addr 172.21.15.28:0/1187621372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc646a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64ba9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 -- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 msgr2=0x556dc646a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc646a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 -- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 msgr2=0x556dc64ba9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64ba9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 -- 172.21.15.28:0/1187621372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed40a8930 con 0x556dc64972c0 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc646a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64ba9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:44.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ed3400640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f4ec002a770 tx=0x7f4ec0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726d9aa129d0d328 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.388+0000 7f4ebd600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0052430 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ec0053c40 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556dc63fbeb0 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec0053e00 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 10 client.11199 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x556dc63d12a0 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556dc63fbeb0 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4ec0044070 con 0x556dc64972c0 2024-09-13T17:21:44.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 10 client.11199 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 10 client.11199 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.389+0000 7f4edd862740 20 client.11199 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64baf90 0x556dc64959d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4edd862740 4 client.11199 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556dc63c7340 con 0x556dc64baf90 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64baf90 0x556dc64959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64baf90 0x556dc64959d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18e33d9d7a6f275 server_cookie=e23581aa44d5334f in_seq=0 out_seq=0 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.390+0000 7f4ebd600640 10 client.11199 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4ebd600640 1 -- 172.21.15.28:0/1187621372 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x556dc63c7340 con 0x556dc64baf90 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4ebd600640 10 client.11199 handle_command_reply: tid=0 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 client.11199 shutdown 2024-09-13T17:21:44.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 msgr2=0x556dc64ba470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f4ec002a770 tx=0x7f4ec0004730 comp rx=0 tx=0).stop 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 shutdown_connections 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64baf90 0x556dc64959d0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64ba9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64ba470 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 --2- 172.21.15.28:0/1187621372 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc646a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4ebd600640 0 client.11199 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 >> 172.21.15.28:0/1187621372 conn(0x556dc646d5d0 msgr2=0x556dc64b7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 shutdown_connections 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.391+0000 7f4edd862740 1 -- 172.21.15.28:0/1187621372 wait complete. 2024-09-13T17:21:44.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.392+0000 7f4edd862740 20 client.11199 trim_cache size 0 max 16384 2024-09-13T17:21:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.403+0000 7f4edd862740 1 Processor -- start 2024-09-13T17:21:44.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.403+0000 7f4edd862740 1 -- start start 2024-09-13T17:21:44.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc613a040 con 0x556dc64972c0 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc64b6410 con 0x556dc646abf0 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556dc610ca60 con 0x556dc646fe10 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.404+0000 7f4ed8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43428/0 (socket says 172.21.15.28:43428) 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 -- 172.21.15.28:0/4036655489 learned_addr learned my addr 172.21.15.28:0/4036655489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed3400640 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 -- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 msgr2=0x556dc64b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 -- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 msgr2=0x556dc64b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 -- 172.21.15.28:0/4036655489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556dc6405200 con 0x556dc64972c0 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc003ce0 tx=0x7f4ecc001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7041e007a8b0540f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.405+0000 7f4ec9600640 1 -- 172.21.15.28:0/4036655489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc0020d0 con 0x556dc64972c0 2024-09-13T17:21:44.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4ec9600640 1 -- 172.21.15.28:0/4036655489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ecc0055b0 con 0x556dc64972c0 2024-09-13T17:21:44.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4ec9600640 1 -- 172.21.15.28:0/4036655489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc016450 con 0x556dc64972c0 2024-09-13T17:21:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 -- 172.21.15.28:0/4036655489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 msgr2=0x556dc64b9030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc003ce0 tx=0x7f4ecc001670 comp rx=0 tx=0).stop 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 -- 172.21.15.28:0/4036655489 shutdown_connections 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc64972c0 0x556dc64b9030 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 --2- 172.21.15.28:0/4036655489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc646abf0 0x556dc64b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 -- 172.21.15.28:0/4036655489 >> 172.21.15.28:0/4036655489 conn(0x556dc646d5d0 msgr2=0x556dc64b54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 -- 172.21.15.28:0/4036655489 shutdown_connections 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.406+0000 7f4edd862740 1 -- 172.21.15.28:0/4036655489 wait complete. 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.407+0000 7f4edd862740 1 -- start start 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.407+0000 7f4edd862740 1 Processor -- start 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 0x556dc64b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc64b1e80 con 0x556dc646abf0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556dc651f220 con 0x556dc64972c0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556dc6469a60 con 0x556dc646fe10 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4edd862740 10 client.0 _ll_register_callbacks cb 0x7f4ec40090e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4ed3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43444/0 (socket says 172.21.15.28:43444) 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2339422384 learned_addr learned my addr 172.21.15.28:0/2339422384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4ed3400640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.408+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 0x556dc64b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 msgr2=0x556dc64b2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 msgr2=0x556dc64b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 0x556dc64b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 -- 172.21.15.28:0/2339422384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed406c8b0 con 0x556dc646abf0 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed8a00640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 0x556dc64b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ed3e00640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40586f0 tx=0x7f4ec4089040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d860c0628e6d7269 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ebd600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4072070 con 0x556dc646abf0 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ec4060870 con 0x556dc646abf0 2024-09-13T17:21:44.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ec4056670 con 0x556dc646abf0 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ec4054c40 con 0x556dc646abf0 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4edd862740 10 client.11205 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x556dc63735f0 con 0x556dc646abf0 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.409+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ec4056670 con 0x556dc646abf0 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4ec4011070 con 0x556dc646abf0 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 10 client.11205 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 10 client.11205 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 20 client.11205 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64d5720 0x556dc651f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 4 client.11205 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:21:44.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556dc60c64e0 con 0x556dc64d5720 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4ed3400640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64d5720 0x556dc651f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4ebd600640 10 client.11205 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.410+0000 7f4ed3400640 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64d5720 0x556dc651f670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d9497359655be33 server_cookie=61de12fc6dea1d79 in_seq=0 out_seq=0 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4ebd600640 1 -- 172.21.15.28:0/2339422384 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556dc60c64e0 con 0x556dc64d5720 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4ebd600640 10 client.11205 handle_command_reply: tid=0 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 client.11205 shutdown 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 msgr2=0x556dc64b1940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f4ec40586f0 tx=0x7f4ec4089040 comp rx=0 tx=0).stop 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 shutdown_connections 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556dc64d5720 0x556dc651f670 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556dc64972c0 0x556dc64b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556dc646fe10 0x556dc64b2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 --2- 172.21.15.28:0/2339422384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556dc646abf0 0x556dc64b1940 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4ebd600640 0 client.11205 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 >> 172.21.15.28:0/2339422384 conn(0x556dc646d5d0 msgr2=0x556dc64b84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 shutdown_connections 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 1 -- 172.21.15.28:0/2339422384 wait complete. 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.411+0000 7f4edd862740 20 client.11205 trim_cache size 0 max 16384 2024-09-13T17:21:44.421 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 msgr2=0x7f4eb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 0x7f4eb406d0c0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0034000 tx=0x7f4ec002e000 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 msgr2=0x7f4ed414ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e9d0 tx=0x7f4ec400ee90 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 shutdown_connections 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4eb406ac00 0x7f4eb406d0c0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4ed415ab60 0x7f4ed41579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ed415a780 0x7f4ed40a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 --2- 172.21.15.28:0/873395989 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ed406bb40 0x7f4ed414ea80 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 >> 172.21.15.28:0/873395989 conn(0x7f4ed40b0270 msgr2=0x7f4ed40cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 shutdown_connections 2024-09-13T17:21:44.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.412+0000 7f4edaa00640 1 -- 172.21.15.28:0/873395989 wait complete. 2024-09-13T17:21:44.430 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T17:21:44.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.581+0000 7fcf3b200640 1 Processor -- start 2024-09-13T17:21:44.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.581+0000 7fcf3b200640 1 -- start start 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf3b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf3406f970 0x7fcf3415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf3b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf3406c2d0 0x7fcf3406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf3b200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf34159d90 con 0x7fcf341592a0 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf3b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf3415b0b0 con 0x7fcf3406c2d0 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf3b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3406cca0 con 0x7fcf3406f970 2024-09-13T17:21:44.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf39200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf3406c2d0 0x7fcf3406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf39200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf3406c2d0 0x7fcf3406c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43460/0 (socket says 172.21.15.28:43460) 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf39200640 1 -- 172.21.15.28:0/2200934136 learned_addr learned my addr 172.21.15.28:0/2200934136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.583+0000 7fcf2b400640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf3406f970 0x7fcf3415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.584+0000 7fcf39200640 1 -- 172.21.15.28:0/2200934136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf3406f970 msgr2=0x7fcf3415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.584+0000 7fcf39200640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf3406f970 0x7fcf3415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.584+0000 7fcf39200640 1 -- 172.21.15.28:0/2200934136 >> v1:172.21.15.189:6789/0 conn(0x7fcf341592a0 legacy=0x7fcf34159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.584+0000 7fcf39200640 1 -- 172.21.15.28:0/2200934136 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340b2040 con 0x7fcf3406c2d0 2024-09-13T17:21:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf39200640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf3406c760 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009e50 tx=0x7fcf2c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58f192d35435dac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf33400640 1 -- 172.21.15.28:0/2200934136 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c031600 con 0x7fcf3406c2d0 2024-09-13T17:21:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf33400640 1 -- 172.21.15.28:0/2200934136 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2c02d050 con 0x7fcf3406c2d0 2024-09-13T17:21:44.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf33400640 1 -- 172.21.15.28:0/2200934136 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2c031980 con 0x7fcf3406c2d0 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 -- 172.21.15.28:0/2200934136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 msgr2=0x7fcf3406c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf3406c760 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c009e50 tx=0x7fcf2c02a710 comp rx=0 tx=0).stop 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 -- 172.21.15.28:0/2200934136 shutdown_connections 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf3406c760 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 --2- 172.21.15.28:0/2200934136 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf3406f970 0x7fcf3415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.585+0000 7fcf3b200640 1 -- 172.21.15.28:0/2200934136 >> 172.21.15.28:0/2200934136 conn(0x7fcf34069e80 msgr2=0x7fcf3406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.586+0000 7fcf3b200640 1 -- 172.21.15.28:0/2200934136 shutdown_connections 2024-09-13T17:21:44.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.586+0000 7fcf3b200640 1 -- 172.21.15.28:0/2200934136 wait complete. 2024-09-13T17:21:44.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.587+0000 7fcf3b200640 1 Processor -- start 2024-09-13T17:21:44.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.587+0000 7fcf3b200640 1 -- start start 2024-09-13T17:21:44.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf3406f970 0x7fcf3414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf341592a0 0x7fcf3414d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34165890 con 0x7fcf3406c2d0 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34156890 con 0x7fcf3406f970 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf3b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf341f6320 con 0x7fcf341592a0 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.588+0000 7fcf33e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43468/0 (socket says 172.21.15.28:43468) 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 -- 172.21.15.28:0/824317116 learned_addr learned my addr 172.21.15.28:0/824317116 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf39200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf341592a0 0x7fcf3414d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf2b400640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf3406f970 0x7fcf3414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 -- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf341592a0 msgr2=0x7fcf3414d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf341592a0 0x7fcf3414d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 -- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf3406f970 msgr2=0x7fcf3414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf3406f970 0x7fcf3414d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 -- 172.21.15.28:0/824317116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c009b30 con 0x7fcf3406c2d0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.589+0000 7fcf33e00640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400c390 tx=0x7fcf2400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.590+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240052d0 con 0x7fcf3406c2d0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.590+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf24005430 con 0x7fcf3406c2d0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.590+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24003830 con 0x7fcf3406c2d0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.590+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf34165aa0 con 0x7fcf3406c2d0 2024-09-13T17:21:44.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.590+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf341497d0 con 0x7fcf3406c2d0 2024-09-13T17:21:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.591+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf340b3740 con 0x7fcf3406c2d0 2024-09-13T17:21:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.592+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf24003990 con 0x7fcf3406c2d0 2024-09-13T17:21:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.593+0000 7fcf30c00640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 0x7fcf1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.593+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf24088810 con 0x7fcf3406c2d0 2024-09-13T17:21:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.593+0000 7fcf2b400640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 0x7fcf1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.594+0000 7fcf2b400640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 0x7fcf1006d0c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcf341672c0 tx=0x7fcf1c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:44.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.597+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcf24053a80 con 0x7fcf3406c2d0 2024-09-13T17:21:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.815+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf340b0f60 con 0x7fcf3406c2d0 2024-09-13T17:21:44.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.816+0000 7fcf30c00640 1 -- 172.21.15.28:0/824317116 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fcf24017040 con 0x7fcf3406c2d0 2024-09-13T17:21:44.818 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:21:44.818 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T17:21:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 msgr2=0x7fcf1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 0x7fcf1006d0c0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fcf341672c0 tx=0x7fcf1c009190 comp rx=0 tx=0).stop 2024-09-13T17:21:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 msgr2=0x7fcf341453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fcf2400c390 tx=0x7fcf2400c850 comp rx=0 tx=0).stop 2024-09-13T17:21:44.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 shutdown_connections 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf1006ac00 0x7fcf1006d0c0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf341592a0 0x7fcf3414d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf3406f970 0x7fcf3414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 --2- 172.21.15.28:0/824317116 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf3406c2d0 0x7fcf341453f0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 >> 172.21.15.28:0/824317116 conn(0x7fcf34069e80 msgr2=0x7fcf3406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 shutdown_connections 2024-09-13T17:21:44.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.818+0000 7fcf3b200640 1 -- 172.21.15.28:0/824317116 wait complete. 2024-09-13T17:21:44.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.13:0 scrub start '~mdsdir' recursive 2024-09-13T17:21:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.994+0000 7f53dfe00640 1 Processor -- start 2024-09-13T17:21:44.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.994+0000 7f53dfe00640 1 -- start start 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53e0143ea0 0x7f53e00b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e00cd8a0 0x7f53e00b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dfe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e0144280 con 0x7f53e015ebd0 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e00b3060 con 0x7f53e0143ea0 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00b23e0 con 0x7f53e00cd8a0 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.995+0000 7f53dea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53e0143ea0 0x7f53e00b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f53e0143ea0 0x7f53e00b1b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43480/0 (socket says 172.21.15.28:43480) 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 -- 172.21.15.28:0/4236615305 learned_addr learned my addr 172.21.15.28:0/4236615305 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53df400640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e00cd8a0 0x7f53e00b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:44.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 -- 172.21.15.28:0/4236615305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e00cd8a0 msgr2=0x7f53e00b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e00cd8a0 0x7f53e00b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 -- 172.21.15.28:0/4236615305 >> v1:172.21.15.189:6789/0 conn(0x7f53e015ebd0 legacy=0x7f53e00cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:44.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.996+0000 7f53dea00640 1 -- 172.21.15.28:0/4236615305 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e00b26c0 con 0x7f53e0143ea0 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.997+0000 7f53dea00640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0143ea0 0x7f53e00b1b00 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009870 tx=0x7f53d002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184395a3191c04a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.997+0000 7f53dd600640 1 -- 172.21.15.28:0/4236615305 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0032440 con 0x7f53e0143ea0 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.997+0000 7f53dd600640 1 -- 172.21.15.28:0/4236615305 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d00325a0 con 0x7f53e0143ea0 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.997+0000 7f53dd600640 1 -- 172.21.15.28:0/4236615305 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0032930 con 0x7f53e0143ea0 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 -- 172.21.15.28:0/4236615305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0143ea0 msgr2=0x7f53e00b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0143ea0 0x7f53e00b1b00 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f53d0009870 tx=0x7f53d002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 -- 172.21.15.28:0/4236615305 shutdown_connections 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e00cd8a0 0x7f53e00b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4236615305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e0143ea0 0x7f53e00b1b00 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 -- 172.21.15.28:0/4236615305 >> 172.21.15.28:0/4236615305 conn(0x7f53e0069e80 msgr2=0x7f53e006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 -- 172.21.15.28:0/4236615305 shutdown_connections 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:44.998+0000 7f53dfe00640 1 -- 172.21.15.28:0/4236615305 wait complete. 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.000+0000 7f53dfe00640 1 Processor -- start 2024-09-13T17:21:45.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.000+0000 7f53dfe00640 1 -- start start 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 0x7f53e0139280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ebd0 0x7f53e00b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e013a230 con 0x7f53e00cd8a0 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0139990 con 0x7f53e0143ea0 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e0139c70 con 0x7f53e015ebd0 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43484/0 (socket says 172.21.15.28:43484) 2024-09-13T17:21:45.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 -- 172.21.15.28:0/4062815218 learned_addr learned my addr 172.21.15.28:0/4062815218 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 -- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ebd0 msgr2=0x7f53e00b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ebd0 0x7f53e00b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 -- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 msgr2=0x7f53e0139280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53de000640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 0x7f53e0139280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 0x7f53e0139280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53dea00640 1 -- 172.21.15.28:0/4062815218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00094d0 con 0x7f53e00cd8a0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.001+0000 7f53de000640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 0x7f53e0139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53dea00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f53d0000c00 tx=0x7f53d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53d7400640 1 -- 172.21.15.28:0/4062815218 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0032440 con 0x7f53e00cd8a0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53d7400640 1 -- 172.21.15.28:0/4062815218 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d0030900 con 0x7f53e00cd8a0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e0139f50 con 0x7f53e00cd8a0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53d7400640 1 -- 172.21.15.28:0/4062815218 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0030cb0 con 0x7f53e00cd8a0 2024-09-13T17:21:45.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.002+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e00b0a50 con 0x7f53e00cd8a0 2024-09-13T17:21:45.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.004+0000 7f53d7400640 1 -- 172.21.15.28:0/4062815218 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f53d0039040 con 0x7f53e00cd8a0 2024-09-13T17:21:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.005+0000 7f53d7400640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 0x7f53c006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.005+0000 7f53d7400640 1 -- 172.21.15.28:0/4062815218 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d00a9bf0 con 0x7f53e00cd8a0 2024-09-13T17:21:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.006+0000 7f53de000640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 0x7f53c006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.006+0000 7f53e8c20740 1 Processor -- start 2024-09-13T17:21:45.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.006+0000 7f53e8c20740 1 -- start start 2024-09-13T17:21:45.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53de000640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 0x7f53c006d110 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f53e013aa60 tx=0x7f53cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d74544760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 0x561d74591850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d744ff5e0 con 0x561d7456b830 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d744addf0 con 0x561d74544380 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d744ab580 con 0x561d74541b40 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43498/0 (socket says 172.21.15.28:43498) 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53df400640 1 -- 172.21.15.28:0/3214502017 learned_addr learned my addr 172.21.15.28:0/3214502017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53de000640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 0x561d74591850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.007+0000 7f53dea00640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d74544760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 -- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 msgr2=0x561d74591850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 0x561d74591850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 -- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 msgr2=0x561d74544760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d74544760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 -- 172.21.15.28:0/3214502017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d7446e0e0 con 0x561d7456b830 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53dea00640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d74544760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53de000640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 0x561d74591850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53df400640 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e300 tx=0x7f53d8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b379119ed537fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53cbe00640 1 -- 172.21.15.28:0/3214502017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8009ca0 con 0x561d7456b830 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53cbe00640 1 -- 172.21.15.28:0/3214502017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d800f040 con 0x561d7456b830 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.008+0000 7f53cbe00640 1 -- 172.21.15.28:0/3214502017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8014af0 con 0x561d7456b830 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 -- 172.21.15.28:0/3214502017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 msgr2=0x561d74591d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 secure :-1 s=READY pgs=2339 cs=0 l=1 rev1=1 crypto rx=0x7f53d800e300 tx=0x7f53d8009f90 comp rx=0 tx=0).stop 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 -- 172.21.15.28:0/3214502017 shutdown_connections 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d74591d90 unknown :-1 s=CLOSED pgs=2339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74541b40 0x561d74591850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3214502017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d74544760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 -- 172.21.15.28:0/3214502017 >> 172.21.15.28:0/3214502017 conn(0x561d74540ab0 msgr2=0x561d74588f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 -- 172.21.15.28:0/3214502017 shutdown_connections 2024-09-13T17:21:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.009+0000 7f53e8c20740 1 -- 172.21.15.28:0/3214502017 wait complete. 2024-09-13T17:21:45.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.010+0000 7f53e8c20740 1 -- start start 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.010+0000 7f53e8c20740 1 Processor -- start 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d7453e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d7453ee80 0x561d74561de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d744a6e70 con 0x561d7456b830 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d744480a0 con 0x561d74544380 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d74447d40 con 0x561d7453ee80 2024-09-13T17:21:45.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53e8c20740 10 client.0 _ll_register_callbacks cb 0x561d7446d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43508/0 (socket says 172.21.15.28:43508) 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 -- 172.21.15.28:0/4134471782 learned_addr learned my addr 172.21.15.28:0/4134471782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53dea00640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d7453e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53df400640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d7453ee80 0x561d74561de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 -- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d7453ee80 msgr2=0x561d74561de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d7453ee80 0x561d74561de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.011+0000 7f53de000640 1 -- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 msgr2=0x561d7453e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53de000640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d7453e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53de000640 1 -- 172.21.15.28:0/4134471782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d8009e00 con 0x561d7456b830 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53de000640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f53e015f020 tx=0x7f53cc000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76cec3e6263a29e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:45.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53cc014dc0 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53cc005110 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d74498d10 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53cc005310 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53e8c20740 10 client.11223 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.012+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x561d744a54f0 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561d74498d10 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f53cc003610 con 0x561d7456b830 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 10 client.11223 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 10 client.11223 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 20 client.11223 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74569090 0x561d74590670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 4 client.11223 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d7419a700 con 0x561d74569090 2024-09-13T17:21:45.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53dea00640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74569090 0x561d74590670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53dea00640 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74569090 0x561d74590670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e967cb3b9503700 server_cookie=37e195a016c6dcde in_seq=0 out_seq=0 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.013+0000 7f53c8c00640 10 client.11223 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53c8c00640 1 -- 172.21.15.28:0/4134471782 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x561d7419a700 con 0x561d74569090 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53c8c00640 10 client.11223 handle_command_reply: tid=0 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 client.11223 shutdown 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 msgr2=0x561d745618a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7f53e015f020 tx=0x7f53cc000ed0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 shutdown_connections 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74569090 0x561d74590670 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d7453ee80 0x561d74561de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745618a0 unknown :-1 s=CLOSED pgs=2340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53c8c00640 0 client.11223 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 --2- 172.21.15.28:0/4134471782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d74544380 0x561d7453e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.014+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 >> 172.21.15.28:0/4134471782 conn(0x561d74541b40 msgr2=0x561d7458c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.015+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 shutdown_connections 2024-09-13T17:21:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.015+0000 7f53e8c20740 1 -- 172.21.15.28:0/4134471782 wait complete. 2024-09-13T17:21:45.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.015+0000 7f53e8c20740 20 client.11223 trim_cache size 0 max 16384 2024-09-13T17:21:45.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.030+0000 7f53e8c20740 1 Processor -- start 2024-09-13T17:21:45.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.030+0000 7f53e8c20740 1 -- start start 2024-09-13T17:21:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7453ee80 0x561d7456dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7456e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d7420e260 con 0x561d7456b830 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d745909d0 con 0x561d7453ee80 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d7456ac80 con 0x561d74544380 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43522/0 (socket says 172.21.15.28:43522) 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 -- 172.21.15.28:0/3982933678 learned_addr learned my addr 172.21.15.28:0/3982933678 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53de000640 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7456e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53dea00640 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7453ee80 0x561d7456dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 -- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 msgr2=0x561d7456e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7456e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 -- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7453ee80 msgr2=0x561d7456dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7453ee80 0x561d7456dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.031+0000 7f53df400640 1 -- 172.21.15.28:0/3982933678 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d74447240 con 0x561d7456b830 2024-09-13T17:21:45.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53df400640 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f53d8016000 tx=0x7f53d8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8053518f3ad40f82 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53d4c00640 1 -- 172.21.15.28:0/3982933678 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8003990 con 0x561d7456b830 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53d4c00640 1 -- 172.21.15.28:0/3982933678 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d8003b10 con 0x561d7456b830 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53d4c00640 1 -- 172.21.15.28:0/3982933678 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d8014690 con 0x561d7456b830 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53e8c20740 1 -- 172.21.15.28:0/3982933678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 msgr2=0x561d745861c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.032+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7f53d8016000 tx=0x7f53d8013000 comp rx=0 tx=0).stop 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 -- 172.21.15.28:0/3982933678 shutdown_connections 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7456b830 0x561d745861c0 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7456e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3982933678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7453ee80 0x561d7456dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 -- 172.21.15.28:0/3982933678 >> 172.21.15.28:0/3982933678 conn(0x561d74541b40 msgr2=0x561d74569470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 -- 172.21.15.28:0/3982933678 shutdown_connections 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 -- 172.21.15.28:0/3982933678 wait complete. 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 -- start start 2024-09-13T17:21:45.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.033+0000 7f53e8c20740 1 Processor -- start 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7458f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7456b830 0x561d745644b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d74540040 con 0x561d7453ee80 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d7453e110 con 0x561d7456b830 2024-09-13T17:21:45.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561d74540290 con 0x561d74544380 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.034+0000 7f53e8c20740 10 client.0 _ll_register_callbacks cb 0x7f53d8009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43530/0 (socket says 172.21.15.28:43530) 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 -- 172.21.15.28:0/3920550613 learned_addr learned my addr 172.21.15.28:0/3920550613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53df400640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7456b830 0x561d745644b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53de000640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7458f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 -- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 msgr2=0x561d7458f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7458f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 -- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7456b830 msgr2=0x561d745644b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7456b830 0x561d745644b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 -- 172.21.15.28:0/3920550613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d7446e0e0 con 0x561d7453ee80 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53dea00640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f53d007caf0 tx=0x7f53d002a740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2c67dd33ffe03f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.035+0000 7f53c8c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d00a4030 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f53d00a5070 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53d800f040 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53d00a82d0 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 10 client.11229 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x561d7446c940 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d800f040 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f53d004d020 con 0x561d7453ee80 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 10 client.11229 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 10 client.11229 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 20 client.11229 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:45.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74542610 0x561d745649f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 4 client.11229 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}] 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "scrub start", "path": "~mdsdir", "scrubops": ["recursive"]}) v1 -- 0x561d741aba00 con 0x561d74542610 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.036+0000 7f53de000640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74542610 0x561d745649f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.037+0000 7f53de000640 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74542610 0x561d745649f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff24519fd6485019 server_cookie=af1061a67cb72e64 in_seq=0 out_seq=0 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.037+0000 7f53c8c00640 10 client.11229 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53c8c00640 1 -- 172.21.15.28:0/3920550613 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561d741aba00 con 0x561d74542610 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53c8c00640 10 client.11229 handle_command_reply: tid=0 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 client.11229 shutdown 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 msgr2=0x561d7458ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 secure :-1 s=READY pgs=2342 cs=0 l=1 rev1=1 crypto rx=0x7f53d007caf0 tx=0x7f53d002a740 comp rx=0 tx=0).stop 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 shutdown_connections 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x561d74542610 0x561d745649f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561d7456b830 0x561d745644b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561d74544380 0x561d7458f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 --2- 172.21.15.28:0/3920550613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561d7453ee80 0x561d7458ed80 unknown :-1 s=CLOSED pgs=2342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 >> 172.21.15.28:0/3920550613 conn(0x561d74541b40 msgr2=0x561d745af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53c8c00640 0 client.11229 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 shutdown_connections 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 1 -- 172.21.15.28:0/3920550613 wait complete. 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.038+0000 7f53e8c20740 20 client.11229 trim_cache size 0 max 16384 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:21:45.045 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "0ec1f2dd-781c-4334-880e-775654e4ccdc", 2024-09-13T17:21:45.046 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:21:45.046 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:21:45.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 msgr2=0x7f53c006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 0x7f53c006d110 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f53e013aa60 tx=0x7f53cc009040 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 msgr2=0x7f53e01f3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f53d0000c00 tx=0x7f53d00046c0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 shutdown_connections 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53c006ac50 0x7f53c006d110 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53e015ebd0 0x7f53e00b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53e0143ea0 0x7f53e0139280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 --2- 172.21.15.28:0/4062815218 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53e00cd8a0 0x7f53e01f3d20 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 >> 172.21.15.28:0/4062815218 conn(0x7f53e0069e80 msgr2=0x7f53e015b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 shutdown_connections 2024-09-13T17:21:45.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.039+0000 7f53dfe00640 1 -- 172.21.15.28:0/4062815218 wait complete. 2024-09-13T17:21:45.056 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.13:0 scrub status 2024-09-13T17:21:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.211+0000 7f75b0e00640 1 Processor -- start 2024-09-13T17:21:45.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.211+0000 7f75b0e00640 1 -- start start 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75b0e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f75ac159770 0x7f75ac06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75b0e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f75ac06f3f0 0x7f75ac070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75b0e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac06f920 con 0x7f75ac0add00 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75b0e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac15b290 con 0x7f75ac06f3f0 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75b0e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac15b570 con 0x7f75ac159770 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75a9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f75ac06f3f0 0x7f75ac070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75aa800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f75ac0add00 0x7f75ac159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:50874/0 (socket says 172.21.15.28:50874) 2024-09-13T17:21:45.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.212+0000 7f75aa800640 1 -- 172.21.15.28:0/2366220596 learned_addr learned my addr 172.21.15.28:0/2366220596 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a9e00640 1 --2- 172.21.15.28:0/2366220596 >> v2:172.21.15.189:3300/0 conn(0x7f75ac159770 0x7f75ac06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a9400640 1 -- 172.21.15.28:0/2366220596 >> v2:172.21.15.189:3300/0 conn(0x7f75ac159770 msgr2=0x7f75ac06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a9400640 1 --2- 172.21.15.28:0/2366220596 >> v2:172.21.15.189:3300/0 conn(0x7f75ac159770 0x7f75ac06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a9400640 1 -- 172.21.15.28:0/2366220596 >> v1:172.21.15.28:6789/0 conn(0x7f75ac0add00 legacy=0x7f75ac159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a8a00640 1 -- 172.21.15.28:0/2366220596 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851539676 0 0) 0x7f75ac06f920 con 0x7f75ac0add00 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.213+0000 7f75a9400640 1 -- 172.21.15.28:0/2366220596 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac15b850 con 0x7f75ac06f3f0 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.214+0000 7f75a9400640 1 --2- 172.21.15.28:0/2366220596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac06f3f0 0x7f75ac070260 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f75ac067f80 tx=0x7f759802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ab08488ab03281 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.214+0000 7f75a8a00640 1 -- 172.21.15.28:0/2366220596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759802add0 con 0x7f75ac06f3f0 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.214+0000 7f75a8a00640 1 -- 172.21.15.28:0/2366220596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7598032440 con 0x7f75ac06f3f0 2024-09-13T17:21:45.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.214+0000 7f75a8a00640 1 -- 172.21.15.28:0/2366220596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75980327d0 con 0x7f75ac06f3f0 2024-09-13T17:21:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 -- 172.21.15.28:0/2366220596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac06f3f0 msgr2=0x7f75ac070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 --2- 172.21.15.28:0/2366220596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac06f3f0 0x7f75ac070260 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f75ac067f80 tx=0x7f759802a080 comp rx=0 tx=0).stop 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 -- 172.21.15.28:0/2366220596 shutdown_connections 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 --2- 172.21.15.28:0/2366220596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac06f3f0 0x7f75ac070260 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 --2- 172.21.15.28:0/2366220596 >> v2:172.21.15.189:3300/0 conn(0x7f75ac159770 0x7f75ac06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 -- 172.21.15.28:0/2366220596 >> 172.21.15.28:0/2366220596 conn(0x7f75ac0b0270 msgr2=0x7f75ac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 -- 172.21.15.28:0/2366220596 shutdown_connections 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.215+0000 7f75b0e00640 1 -- 172.21.15.28:0/2366220596 wait complete. 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.217+0000 7f75b0e00640 1 Processor -- start 2024-09-13T17:21:45.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.217+0000 7f75b0e00640 1 -- start start 2024-09-13T17:21:45.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.217+0000 7f75b0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 0x7f75ac146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.217+0000 7f75b0e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75ac0add00 0x7f75ac1e1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.217+0000 7f75b0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75b0e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1e27e0 con 0x7f75ac159770 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75b0e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1e2ac0 con 0x7f75ac06f3f0 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75b0e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac1e2da0 con 0x7f75ac0add00 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43544/0 (socket says 172.21.15.28:43544) 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 -- 172.21.15.28:0/3346397120 learned_addr learned my addr 172.21.15.28:0/3346397120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 0x7f75ac146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75a9400640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75ac0add00 0x7f75ac1e1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 -- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75ac0add00 msgr2=0x7f75ac1e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75ac0add00 0x7f75ac1e1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 -- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 msgr2=0x7f75ac146f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 0x7f75ac146f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 -- 172.21.15.28:0/3346397120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7598009400 con 0x7f75ac159770 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 0x7f75ac146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.218+0000 7f75aa800640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c7e0 tx=0x7f75a400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.219+0000 7f75a2000640 1 -- 172.21.15.28:0/3346397120 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400a640 con 0x7f75ac159770 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.219+0000 7f75a2000640 1 -- 172.21.15.28:0/3346397120 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75a400a7c0 con 0x7f75ac159770 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.219+0000 7f75a2000640 1 -- 172.21.15.28:0/3346397120 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75a400ab70 con 0x7f75ac159770 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.219+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac1e3080 con 0x7f75ac159770 2024-09-13T17:21:45.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.219+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac153fb0 con 0x7f75ac159770 2024-09-13T17:21:45.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.221+0000 7f75a2000640 1 -- 172.21.15.28:0/3346397120 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f75a400acd0 con 0x7f75ac159770 2024-09-13T17:21:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.223+0000 7f75a2000640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 0x7f758c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.223+0000 7f75a2000640 1 -- 172.21.15.28:0/3346397120 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75a40889e0 con 0x7f75ac159770 2024-09-13T17:21:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.223+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 0x7f758c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.223+0000 7f75b3c22740 1 Processor -- start 2024-09-13T17:21:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.223+0000 7f75b3c22740 1 -- start start 2024-09-13T17:21:45.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 0x7f758c06d0c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f759c006fd0 tx=0x7f759c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 0x55a809e0d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de7110 0x55a809e0d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809cc66a0 con 0x55a809dbfc60 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809d29cc0 con 0x55a809dbd420 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a809d27450 con 0x55a809de7110 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.224+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43550/0 (socket says 172.21.15.28:43550) 2024-09-13T17:21:45.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 -- 172.21.15.28:0/271203689 learned_addr learned my addr 172.21.15.28:0/271203689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9400640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 0x55a809e0d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75aa800640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de7110 0x55a809e0d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 -- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de7110 msgr2=0x55a809e0d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de7110 0x55a809e0d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 -- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 msgr2=0x55a809e0d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 0x55a809e0d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 -- 172.21.15.28:0/271203689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a809cf32e0 con 0x55a809dbfc60 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9400640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 0x55a809e0d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.225+0000 7f75a9e00640 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f759c00fa80 tx=0x7f759c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17dc297b0ede48ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f7596a00640 1 -- 172.21.15.28:0/271203689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c014be0 con 0x55a809dbfc60 2024-09-13T17:21:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f7596a00640 1 -- 172.21.15.28:0/271203689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c019070 con 0x55a809dbfc60 2024-09-13T17:21:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f7596a00640 1 -- 172.21.15.28:0/271203689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c020440 con 0x55a809dbfc60 2024-09-13T17:21:45.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 -- 172.21.15.28:0/271203689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 msgr2=0x55a809dc0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 secure :-1 s=READY pgs=2346 cs=0 l=1 rev1=1 crypto rx=0x7f759c00fa80 tx=0x7f759c012000 comp rx=0 tx=0).stop 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 -- 172.21.15.28:0/271203689 shutdown_connections 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de7110 0x55a809e0d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbd420 0x55a809e0d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 --2- 172.21.15.28:0/271203689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dc0040 unknown :-1 s=CLOSED pgs=2346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 -- 172.21.15.28:0/271203689 >> 172.21.15.28:0/271203689 conn(0x55a809dbc390 msgr2=0x55a809e04840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 -- 172.21.15.28:0/271203689 shutdown_connections 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.226+0000 7f75b3c22740 1 -- 172.21.15.28:0/271203689 wait complete. 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.227+0000 7f75b3c22740 1 -- start start 2024-09-13T17:21:45.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.227+0000 7f75b3c22740 1 Processor -- start 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 0x55a809e183b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e0b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809d22d40 con 0x55a809de7110 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809cc4280 con 0x55a809dbfc60 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a809cc3f20 con 0x55a809de90b0 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75b3c22740 10 client.0 _ll_register_callbacks cb 0x55a809d1c3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75a9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75a9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43564/0 (socket says 172.21.15.28:43564) 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.228+0000 7f75a9400640 1 -- 172.21.15.28:0/62921225 learned_addr learned my addr 172.21.15.28:0/62921225 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9e00640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 0x55a809e183b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75aa800640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e0b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 -- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 msgr2=0x55a809e0b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e0b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 -- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 msgr2=0x55a809e183b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 0x55a809e183b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 -- 172.21.15.28:0/62921225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f759c00e040 con 0x55a809de7110 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9e00640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 0x55a809e183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75a9400640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f7598009860 tx=0x7f7598004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae18c09d7dbb05cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f758b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759803c470 con 0x55a809de7110 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a809a164e0 con 0x55a809de7110 2024-09-13T17:21:45.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.229+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f759803c5d0 con 0x55a809de7110 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f759803c790 con 0x55a809de7110 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 10 client.11244 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55a809d213c0 con 0x55a809de7110 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a809a164e0 con 0x55a809de7110 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f759803c8f0 con 0x55a809de7110 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 10 client.11244 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 10 client.11244 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 20 client.11244 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809e0c300 0x55a809e0c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 4 client.11244 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a809a277e0 con 0x55a809e0c300 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.230+0000 7f75a9e00640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809e0c300 0x55a809e0c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.231+0000 7f75a9e00640 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809e0c300 0x55a809e0c6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ecdffc6716f3630 server_cookie=ebaa5692c98dc1b7 in_seq=0 out_seq=0 2024-09-13T17:21:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.231+0000 7f758b400640 10 client.11244 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f758b400640 1 -- 172.21.15.28:0/62921225 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55a809a277e0 con 0x55a809e0c300 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f758b400640 10 client.11244 handle_command_reply: tid=0 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 client.11244 shutdown 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 msgr2=0x55a809ddb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 secure :-1 s=READY pgs=2347 cs=0 l=1 rev1=1 crypto rx=0x7f7598009860 tx=0x7f7598004730 comp rx=0 tx=0).stop 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 shutdown_connections 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809e0c300 0x55a809e0c6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e0b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809de7110 0x55a809ddb450 unknown :-1 s=CLOSED pgs=2347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 --2- 172.21.15.28:0/62921225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809dbfc60 0x55a809e183b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f758b400640 0 client.11244 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 >> 172.21.15.28:0/62921225 conn(0x55a809dbd420 msgr2=0x55a809e08550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 shutdown_connections 2024-09-13T17:21:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 1 -- 172.21.15.28:0/62921225 wait complete. 2024-09-13T17:21:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.232+0000 7f75b3c22740 20 client.11244 trim_cache size 0 max 16384 2024-09-13T17:21:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.247+0000 7f75b3c22740 1 Processor -- start 2024-09-13T17:21:45.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.247+0000 7f75b3c22740 1 -- start start 2024-09-13T17:21:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809e03990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e03ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809a8a040 con 0x55a809dbfc60 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809e0b5d0 con 0x55a809de7110 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a809de9590 con 0x55a809de90b0 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43572/0 (socket says 172.21.15.28:43572) 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 -- 172.21.15.28:0/1586352041 learned_addr learned my addr 172.21.15.28:0/1586352041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 -- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 msgr2=0x55a809e03ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e03ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9400640 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809e03990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 -- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 msgr2=0x55a809e03990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809e03990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.248+0000 7f75a9e00640 1 -- 172.21.15.28:0/1586352041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a809cc3610 con 0x55a809dbfc60 2024-09-13T17:21:45.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f75a9e00640 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f759c021000 tx=0x7f759c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ac487500e23159 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f7597400640 1 -- 172.21.15.28:0/1586352041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c030040 con 0x55a809dbfc60 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f7597400640 1 -- 172.21.15.28:0/1586352041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c0353b0 con 0x55a809dbfc60 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f7597400640 1 -- 172.21.15.28:0/1586352041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c035740 con 0x55a809dbfc60 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f75b3c22740 1 -- 172.21.15.28:0/1586352041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 msgr2=0x55a809dbb400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.249+0000 7f75b3c22740 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 secure :-1 s=READY pgs=2348 cs=0 l=1 rev1=1 crypto rx=0x7f759c021000 tx=0x7f759c014000 comp rx=0 tx=0).stop 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 -- 172.21.15.28:0/1586352041 shutdown_connections 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809e03ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809e03990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 --2- 172.21.15.28:0/1586352041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809dbb400 unknown :-1 s=CLOSED pgs=2348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 -- 172.21.15.28:0/1586352041 >> 172.21.15.28:0/1586352041 conn(0x55a809dbd420 msgr2=0x55a809e2ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 -- 172.21.15.28:0/1586352041 shutdown_connections 2024-09-13T17:21:45.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 -- 172.21.15.28:0/1586352041 wait complete. 2024-09-13T17:21:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 -- start start 2024-09-13T17:21:45.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.250+0000 7f75b3c22740 1 Processor -- start 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809de75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809de7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809def8c0 con 0x55a809dbfc60 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a809ddab70 con 0x55a809de7110 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a809e06ca0 con 0x55a809de90b0 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.251+0000 7f75b3c22740 10 client.0 _ll_register_callbacks cb 0x7f7598035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43574/0 (socket says 172.21.15.28:43574) 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 -- 172.21.15.28:0/3169855622 learned_addr learned my addr 172.21.15.28:0/3169855622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:21:45.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9400640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809de75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75aa800640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809de7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 -- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 msgr2=0x55a809de7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809de7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 -- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 msgr2=0x55a809de75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809de75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 -- 172.21.15.28:0/3169855622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a809cf32e0 con 0x55a809dbfc60 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75aa800640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809de7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f75a9e00640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f759c00fa20 tx=0x7f759c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4a54bc0adabdc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.252+0000 7f758b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c030040 con 0x55a809dbfc60 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f759c019070 con 0x55a809dbfc60 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f759c03f450 con 0x55a809dbfc60 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f759c035930 con 0x55a809dbfc60 2024-09-13T17:21:45.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 10 client.11250 fetch_fsmap learned FSMap version 165 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55a809ce8b20 con 0x55a809dbfc60 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c019070 con 0x55a809dbfc60 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f759c03f610 con 0x55a809dbfc60 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 10 client.11250 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 10 client.11250 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 20 client.11250 populate_metadata read hostname 'smithi028' 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809ddb010 0x55a809e05d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 4 client.11250 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a809a7ae30 con 0x55a809ddb010 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.253+0000 7f75a9400640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809ddb010 0x55a809e05d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f75a9400640 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809ddb010 0x55a809e05d10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd248f3421dcb601 server_cookie=1fcf97f46cc46c75 in_seq=0 out_seq=0 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f758b400640 10 client.11250 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f758b400640 1 -- 172.21.15.28:0/3169855622 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+257 (crc 0 0 0) 0x55a809a7ae30 con 0x55a809ddb010 2024-09-13T17:21:45.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f758b400640 10 client.11250 handle_command_reply: tid=0 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f75b3c22740 1 client.11250 shutdown 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 msgr2=0x55a809e72370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.254+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7f759c00fa20 tx=0x7f759c004730 comp rx=0 tx=0).stop 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 shutdown_connections 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55a809ddb010 0x55a809e05d10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a809de90b0 0x55a809de7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a809de7110 0x55a809de75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 --2- 172.21.15.28:0/3169855622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a809dbfc60 0x55a809e72370 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f758b400640 0 client.11250 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 >> 172.21.15.28:0/3169855622 conn(0x55a809dbd420 msgr2=0x55a809e038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 shutdown_connections 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 1 -- 172.21.15.28:0/3169855622 wait complete. 2024-09-13T17:21:45.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b3c22740 20 client.11250 trim_cache size 0 max 16384 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "0ec1f2dd-781c-4334-880e-775654e4ccdc": { 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "~mds0", 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "0ec1f2dd-781c-4334-880e-775654e4ccdc", 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive" 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:21:45.261 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:21:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 msgr2=0x7f758c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 0x7f758c06d0c0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f759c006fd0 tx=0x7f759c008040 comp rx=0 tx=0).stop 2024-09-13T17:21:45.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 msgr2=0x7f75ac155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 secure :-1 s=READY pgs=2345 cs=0 l=1 rev1=1 crypto rx=0x7f75a400c7e0 tx=0x7f75a400cca0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.255+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 shutdown_connections 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f758c06ac00 0x7f758c06d0c0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75ac159770 0x7f75ac155840 unknown :-1 s=CLOSED pgs=2345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f75ac0add00 0x7f75ac1e1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 --2- 172.21.15.28:0/3346397120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f75ac06f3f0 0x7f75ac146f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 >> 172.21.15.28:0/3346397120 conn(0x7f75ac0b0270 msgr2=0x7f75ac1589a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 shutdown_connections 2024-09-13T17:21:45.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:21:45.256+0000 7f75b0e00640 1 -- 172.21.15.28:0/3346397120 wait complete. 2024-09-13T17:21:45.273 INFO:tasks.cephfs.filesystem:scrub status for tag:0ec1f2dd-781c-4334-880e-775654e4ccdc - {'path': '~mds0', 'tag': '0ec1f2dd-781c-4334-880e-775654e4ccdc', 'options': 'recursive'} 2024-09-13T17:22:11.599 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:22:11.604 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:22:15.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.13:0 scrub status 2024-09-13T17:22:15.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.442+0000 7f5422200640 1 Processor -- start 2024-09-13T17:22:15.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.442+0000 7f5422200640 1 -- start start 2024-09-13T17:22:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5422200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f541c143910 0x7f541c070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5422200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f541c0cd120 0x7f541c0cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5422200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f541c071630 con 0x7f541c143320 2024-09-13T17:22:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5422200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c0cdad0 con 0x7f541c0cd120 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5422200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f541c143700 con 0x7f541c143910 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5421800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f541c0cd120 0x7f541c0cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5421800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f541c0cd120 0x7f541c0cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52036/0 (socket says 172.21.15.28:52036) 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5421800640 1 -- 172.21.15.28:0/2615284943 learned_addr learned my addr 172.21.15.28:0/2615284943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.443+0000 7f5420e00640 1 --1- 172.21.15.28:0/2615284943 >> v1:172.21.15.28:6789/0 conn(0x7f541c143320 0x7f541c071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59808/0 (socket says 172.21.15.28:59808) 2024-09-13T17:22:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.444+0000 7f5413e00640 1 --2- 172.21.15.28:0/2615284943 >> v2:172.21.15.189:3300/0 conn(0x7f541c143910 0x7f541c070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.444+0000 7f5421800640 1 -- 172.21.15.28:0/2615284943 >> v2:172.21.15.189:3300/0 conn(0x7f541c143910 msgr2=0x7f541c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.444+0000 7f5421800640 1 --2- 172.21.15.28:0/2615284943 >> v2:172.21.15.189:3300/0 conn(0x7f541c143910 0x7f541c070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.444+0000 7f5421800640 1 -- 172.21.15.28:0/2615284943 >> v1:172.21.15.28:6789/0 conn(0x7f541c143320 legacy=0x7f541c071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.444+0000 7f5421800640 1 -- 172.21.15.28:0/2615284943 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f541c070a20 con 0x7f541c0cd120 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.445+0000 7f5421800640 1 --2- 172.21.15.28:0/2615284943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c0cd120 0x7f541c0cd590 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f541c0b0cc0 tx=0x7f541402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac400e3691f0625 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.445+0000 7f541be00640 1 -- 172.21.15.28:0/2615284943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414031600 con 0x7f541c0cd120 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.445+0000 7f541be00640 1 -- 172.21.15.28:0/2615284943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541402d050 con 0x7f541c0cd120 2024-09-13T17:22:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.445+0000 7f541be00640 1 -- 172.21.15.28:0/2615284943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414031980 con 0x7f541c0cd120 2024-09-13T17:22:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 -- 172.21.15.28:0/2615284943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c0cd120 msgr2=0x7f541c0cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 --2- 172.21.15.28:0/2615284943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c0cd120 0x7f541c0cd590 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f541c0b0cc0 tx=0x7f541402a730 comp rx=0 tx=0).stop 2024-09-13T17:22:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 -- 172.21.15.28:0/2615284943 shutdown_connections 2024-09-13T17:22:15.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 --2- 172.21.15.28:0/2615284943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c0cd120 0x7f541c0cd590 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 --2- 172.21.15.28:0/2615284943 >> v2:172.21.15.189:3300/0 conn(0x7f541c143910 0x7f541c070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 -- 172.21.15.28:0/2615284943 >> 172.21.15.28:0/2615284943 conn(0x7f541c069e80 msgr2=0x7f541c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 -- 172.21.15.28:0/2615284943 shutdown_connections 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.446+0000 7f5422200640 1 -- 172.21.15.28:0/2615284943 wait complete. 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.447+0000 7f5422200640 1 Processor -- start 2024-09-13T17:22:15.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.448+0000 7f5422200640 1 -- start start 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.448+0000 7f5422200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 0x7f541c13d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.448+0000 7f5422200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5422200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f541c143910 0x7f541c159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5422200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0c5c10 con 0x7f541c143320 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5422200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f541c13de20 con 0x7f541c0cd120 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5422200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f541c0c5e60 con 0x7f541c143910 2024-09-13T17:22:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52048/0 (socket says 172.21.15.28:52048) 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 -- 172.21.15.28:0/3646635235 learned_addr learned my addr 172.21.15.28:0/3646635235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 -- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f541c143910 msgr2=0x7f541c159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f541c143910 0x7f541c159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 -- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 msgr2=0x7f541c13d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5420e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 0x7f541c13d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 0x7f541c13d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5413e00640 1 -- 172.21.15.28:0/3646635235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414009b30 con 0x7f541c143320 2024-09-13T17:22:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.449+0000 7f5420e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 0x7f541c13d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5413e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f540400d830 tx=0x7f540400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5419600640 1 -- 172.21.15.28:0/3646635235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400b6c0 con 0x7f541c143320 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5419600640 1 -- 172.21.15.28:0/3646635235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f540400b840 con 0x7f541c143320 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5419600640 1 -- 172.21.15.28:0/3646635235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540400bbf0 con 0x7f541c143320 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f541c0c6140 con 0x7f541c143320 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.450+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c0c6700 con 0x7f541c143320 2024-09-13T17:22:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.452+0000 7f5419600640 1 -- 172.21.15.28:0/3646635235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f540401e460 con 0x7f541c143320 2024-09-13T17:22:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.453+0000 7f5419600640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 0x7f53f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.453+0000 7f5419600640 1 -- 172.21.15.28:0/3646635235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5404089b90 con 0x7f541c143320 2024-09-13T17:22:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.453+0000 7f542662e740 1 Processor -- start 2024-09-13T17:22:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f5420e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 0x7f53f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 -- start start 2024-09-13T17:22:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f5420e00640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 0x7f53f806d0c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f60 tx=0x7f540c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d0a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d07590 0x55de83d572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83c106a0 con 0x55de83d31280 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83c73cc0 con 0x55de83d07590 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.454+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de83c71450 con 0x55de83d09dd0 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52060/0 (socket says 172.21.15.28:52060) 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 -- 172.21.15.28:0/850322893 learned_addr learned my addr 172.21.15.28:0/850322893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5420e00640 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d0a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 -- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 msgr2=0x55de83d0a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5413e00640 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d07590 0x55de83d572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d0a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 -- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d07590 msgr2=0x55de83d572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d07590 0x55de83d572a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 -- 172.21.15.28:0/850322893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de83c3d2e0 con 0x55de83d31280 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.455+0000 7f5421800640 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f5414034040 tx=0x7f541402c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b1b9039f0f52d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f5411600640 1 -- 172.21.15.28:0/850322893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403d650 con 0x55de83d31280 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f5411600640 1 -- 172.21.15.28:0/850322893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541403d7d0 con 0x55de83d31280 2024-09-13T17:22:15.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f5411600640 1 -- 172.21.15.28:0/850322893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f541403db80 con 0x55de83d31280 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 -- 172.21.15.28:0/850322893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 msgr2=0x55de83d577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7f5414034040 tx=0x7f541402c000 comp rx=0 tx=0).stop 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 -- 172.21.15.28:0/850322893 shutdown_connections 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d577e0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d07590 0x55de83d572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 --2- 172.21.15.28:0/850322893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d0a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 -- 172.21.15.28:0/850322893 >> 172.21.15.28:0/850322893 conn(0x55de83d06500 msgr2=0x55de83d4e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 -- 172.21.15.28:0/850322893 shutdown_connections 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.456+0000 7f542662e740 1 -- 172.21.15.28:0/850322893 wait complete. 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.457+0000 7f542662e740 1 -- start start 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.457+0000 7f542662e740 1 Processor -- start 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d04f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83d251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83c6cd40 con 0x55de83d09dd0 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83c0e280 con 0x55de83d31280 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de83c0df20 con 0x55de83d05470 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f542662e740 10 client.0 _ll_register_callbacks cb 0x55de83c663b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f5420e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f5420e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52068/0 (socket says 172.21.15.28:52068) 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.458+0000 7f5420e00640 1 -- 172.21.15.28:0/3902038840 learned_addr learned my addr 172.21.15.28:0/3902038840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5421800640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83d251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5413e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d04f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 -- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 msgr2=0x55de83d251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83d251f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 -- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 msgr2=0x55de83d04f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d04f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 -- 172.21.15.28:0/3902038840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f541c070a20 con 0x55de83d09dd0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f5420e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f540c005a70 tx=0x7f540c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a2d127fe1b5723 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f540a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c021630 con 0x55de83d09dd0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f540c020a80 con 0x55de83d09dd0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.459+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de839604e0 con 0x55de83d09dd0 2024-09-13T17:22:15.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f540c020c40 con 0x55de83d09dd0 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 10 client.11265 fetch_fsmap learned FSMap version 165 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55de83c6b3c0 con 0x55de83d09dd0 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55de839604e0 con 0x55de83d09dd0 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f540c020420 con 0x55de83d09dd0 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 10 client.11265 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 10 client.11265 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 20 client.11265 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d53060 0x55de83d53470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 4 client.11265 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de839717e0 con 0x55de83d53060 2024-09-13T17:22:15.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.460+0000 7f5413e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d53060 0x55de83d53470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.461+0000 7f5413e00640 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d53060 0x55de83d53470 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a56f4387c1bce17 server_cookie=493cc262c309e9b1 in_seq=0 out_seq=0 2024-09-13T17:22:15.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.461+0000 7f540a000640 10 client.11265 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.461+0000 7f540a000640 1 -- 172.21.15.28:0/3902038840 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55de839717e0 con 0x55de83d53060 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.461+0000 7f540a000640 10 client.11265 handle_command_reply: tid=0 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 client.11265 shutdown 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 msgr2=0x55de83d56b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7f540c005a70 tx=0x7f540c000ed0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 shutdown_connections 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d53060 0x55de83d53470 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83d251f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d04f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 --2- 172.21.15.28:0/3902038840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d09dd0 0x55de83d56b00 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f540a000640 0 client.11265 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 >> 172.21.15.28:0/3902038840 conn(0x55de83d07590 msgr2=0x55de83d4ccf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 shutdown_connections 2024-09-13T17:22:15.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 1 -- 172.21.15.28:0/3902038840 wait complete. 2024-09-13T17:22:15.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.462+0000 7f542662e740 20 client.11265 trim_cache size 0 max 16384 2024-09-13T17:22:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.477+0000 7f542662e740 1 Processor -- start 2024-09-13T17:22:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.477+0000 7f542662e740 1 -- start start 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83db7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 0x55de83d61760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de839d4040 con 0x55de83d31280 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83d506b0 con 0x55de83d09dd0 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de83d4c020 con 0x55de83d05470 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f5421800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f5421800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52080/0 (socket says 172.21.15.28:52080) 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.478+0000 7f5421800640 1 -- 172.21.15.28:0/1918373817 learned_addr learned my addr 172.21.15.28:0/1918373817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5413e00640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 0x55de83d61760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5420e00640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83db7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 -- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 msgr2=0x55de83db7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83db7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 -- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 msgr2=0x55de83d61760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 0x55de83d61760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 -- 172.21.15.28:0/1918373817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de83c0d610 con 0x55de83d31280 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5420e00640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83db7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5413e00640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 0x55de83d61760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:15.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5421800640 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f541403f040 tx=0x7f5414009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb38195973338150 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5412000640 1 -- 172.21.15.28:0/1918373817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414003990 con 0x55de83d31280 2024-09-13T17:22:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5412000640 1 -- 172.21.15.28:0/1918373817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f541404c070 con 0x55de83d31280 2024-09-13T17:22:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.479+0000 7f5412000640 1 -- 172.21.15.28:0/1918373817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5414003d30 con 0x55de83d31280 2024-09-13T17:22:15.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 -- 172.21.15.28:0/1918373817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 msgr2=0x55de83d61ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 secure :-1 s=READY pgs=2355 cs=0 l=1 rev1=1 crypto rx=0x7f541403f040 tx=0x7f5414009000 comp rx=0 tx=0).stop 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 -- 172.21.15.28:0/1918373817 shutdown_connections 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d31280 0x55de83d61ca0 unknown :-1 s=CLOSED pgs=2355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d09dd0 0x55de83d61760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 --2- 172.21.15.28:0/1918373817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d05470 0x55de83db7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 -- 172.21.15.28:0/1918373817 >> 172.21.15.28:0/1918373817 conn(0x55de83d07590 msgr2=0x55de83d05850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 -- 172.21.15.28:0/1918373817 shutdown_connections 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.480+0000 7f542662e740 1 -- 172.21.15.28:0/1918373817 wait complete. 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.481+0000 7f542662e740 1 -- start start 2024-09-13T17:22:15.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.481+0000 7f542662e740 1 Processor -- start 2024-09-13T17:22:15.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d6c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d6c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83db61a0 con 0x55de83d05470 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de83db6000 con 0x55de83d31280 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55de83dbd500 con 0x55de83d09dd0 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f542662e740 10 client.0 _ll_register_callbacks cb 0x7f540c001e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52092/0 (socket says 172.21.15.28:52092) 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 -- 172.21.15.28:0/3331109597 learned_addr learned my addr 172.21.15.28:0/3331109597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5413e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d6c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5421800640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d6c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 -- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 msgr2=0x55de83d6c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d6c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 -- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 msgr2=0x55de83d6c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d6c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.482+0000 7f5420e00640 1 -- 172.21.15.28:0/3331109597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de83c3d2e0 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f5421800640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d6c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f5420e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f540c020da0 tx=0x7f540c007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eb8b0462397cb84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f540a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f540c037070 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f540c00d5a0 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55de83d33fa0 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7f540c00db40 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f542662e740 10 client.11271 fetch_fsmap learned FSMap version 165 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.483+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x55de83c32b20 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55de83d33fa0 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f540c03a020 con 0x55de83d05470 2024-09-13T17:22:15.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 10 client.11271 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 10 client.11271 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 20 client.11271 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d61210 0x55de83d4ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 4 client.11271 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f541403d7d0 con 0x55de83d61210 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f5413e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d61210 0x55de83d4ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f5413e00640 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d61210 0x55de83d4ace0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5a5dbc101a10eb server_cookie=44a76bfd84a97e8 in_seq=0 out_seq=0 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.484+0000 7f540a000640 10 client.11271 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f540a000640 1 -- 172.21.15.28:0/3331109597 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7f541403d7d0 con 0x55de83d61210 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f540a000640 10 client.11271 handle_command_reply: tid=0 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 client.11271 shutdown 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 msgr2=0x55de83db5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7f540c020da0 tx=0x7f540c007000 comp rx=0 tx=0).stop 2024-09-13T17:22:15.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 shutdown_connections 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x55de83d61210 0x55de83d4ace0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de83d31280 0x55de83d6c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55de83d09dd0 0x55de83d6c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 --2- 172.21.15.28:0/3331109597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55de83d05470 0x55de83db5ac0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f540a000640 0 client.11271 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 >> 172.21.15.28:0/3331109597 conn(0x55de83d07590 msgr2=0x55de83cff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 shutdown_connections 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 1 -- 172.21.15.28:0/3331109597 wait complete. 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.485+0000 7f542662e740 20 client.11271 trim_cache size 0 max 16384 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:15.491 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:22:15.492 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:22:15.492 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:15.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 msgr2=0x7f53f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 0x7f53f806d0c0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f540c000f60 tx=0x7f540c009040 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 msgr2=0x7f541c0c6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 secure :-1 s=READY pgs=2352 cs=0 l=1 rev1=1 crypto rx=0x7f540400d830 tx=0x7f540400dcf0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 shutdown_connections 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f53f806ac00 0x7f53f806d0c0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f541c143910 0x7f541c159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f541c143320 0x7f541c0c6c20 unknown :-1 s=CLOSED pgs=2352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 --2- 172.21.15.28:0/3646635235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f541c0cd120 0x7f541c13d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 >> 172.21.15.28:0/3646635235 conn(0x7f541c069e80 msgr2=0x7f541c0c3d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 shutdown_connections 2024-09-13T17:22:15.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.486+0000 7f5422200640 1 -- 172.21.15.28:0/3646635235 wait complete. 2024-09-13T17:22:15.503 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:22:15.503 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.13:0 damage ls 2024-09-13T17:22:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.659+0000 7faa11600640 1 Processor -- start 2024-09-13T17:22:15.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.659+0000 7faa11600640 1 -- start start 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa11600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa11600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa0c0ad940 0x7faa0c06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa11600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0c070470 con 0x7faa0c15c9a0 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa11600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0c0ace30 con 0x7faa0c0ad940 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa11600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c0ad110 con 0x7faa0c0b2400 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa0b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa0c0ad940 0x7faa0c06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa0b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa0c0ad940 0x7faa0c06bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52106/0 (socket says 172.21.15.28:52106) 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.660+0000 7faa0b400640 1 -- 172.21.15.28:0/2947107810 learned_addr learned my addr 172.21.15.28:0/2947107810 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0a000640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0b400640 1 -- 172.21.15.28:0/2947107810 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 msgr2=0x7faa0c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0b400640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0b400640 1 -- 172.21.15.28:0/2947107810 >> v1:172.21.15.189:6789/0 conn(0x7faa0c15c9a0 legacy=0x7faa0c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0b400640 1 -- 172.21.15.28:0/2947107810 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0c06b170 con 0x7faa0c0ad940 2024-09-13T17:22:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.661+0000 7faa0a000640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.662+0000 7faa0b400640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c06bc50 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7faa04009e50 tx=0x7faa0402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e806c0ae8692cb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.662+0000 7faa09600640 1 -- 172.21.15.28:0/2947107810 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04031600 con 0x7faa0c0ad940 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.662+0000 7faa09600640 1 -- 172.21.15.28:0/2947107810 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0402d050 con 0x7faa0c0ad940 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.662+0000 7faa09600640 1 -- 172.21.15.28:0/2947107810 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04031980 con 0x7faa0c0ad940 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 -- 172.21.15.28:0/2947107810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 msgr2=0x7faa0c06bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c06bc50 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7faa04009e50 tx=0x7faa0402a7a0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 -- 172.21.15.28:0/2947107810 shutdown_connections 2024-09-13T17:22:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c06bc50 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 --2- 172.21.15.28:0/2947107810 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 -- 172.21.15.28:0/2947107810 >> 172.21.15.28:0/2947107810 conn(0x7faa0c0b0270 msgr2=0x7faa0c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 -- 172.21.15.28:0/2947107810 shutdown_connections 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.663+0000 7faa11600640 1 -- 172.21.15.28:0/2947107810 wait complete. 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.664+0000 7faa11600640 1 Processor -- start 2024-09-13T17:22:15.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 -- start start 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c1541d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c15c9a0 0x7faa0c154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c154710 con 0x7faa0c0ad940 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c15e2e0 con 0x7faa0c15c9a0 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.665+0000 7faa11600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c0c3e80 con 0x7faa0c0b2400 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52112/0 (socket says 172.21.15.28:52112) 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 -- 172.21.15.28:0/3736781020 learned_addr learned my addr 172.21.15.28:0/3736781020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0a000640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c1541d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0b400640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c15c9a0 0x7faa0c154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 -- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 msgr2=0x7faa0c1541d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c1541d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 -- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c15c9a0 msgr2=0x7faa0c154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c15c9a0 0x7faa0c154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 -- 172.21.15.28:0/3736781020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04009b30 con 0x7faa0c0ad940 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.666+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c400 tx=0x7fa9fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.667+0000 7faa02a00640 1 -- 172.21.15.28:0/3736781020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0052d0 con 0x7faa0c0ad940 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.667+0000 7faa02a00640 1 -- 172.21.15.28:0/3736781020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc005430 con 0x7faa0c0ad940 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.667+0000 7faa02a00640 1 -- 172.21.15.28:0/3736781020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc003830 con 0x7faa0c0ad940 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.667+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c070930 con 0x7faa0c0ad940 2024-09-13T17:22:15.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.667+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c0cdc10 con 0x7faa0c0ad940 2024-09-13T17:22:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.669+0000 7faa02a00640 1 -- 172.21.15.28:0/3736781020 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa9fc003990 con 0x7faa0c0ad940 2024-09-13T17:22:15.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.670+0000 7faa02a00640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 0x7fa9ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.670+0000 7faa02a00640 1 -- 172.21.15.28:0/3736781020 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9fc088910 con 0x7faa0c0ad940 2024-09-13T17:22:15.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.670+0000 7faa0a000640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 0x7fa9ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.670+0000 7faa14441740 1 Processor -- start 2024-09-13T17:22:15.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.670+0000 7faa14441740 1 -- start start 2024-09-13T17:22:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa0a000640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 0x7fa9ec06d0c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7faa0c1560d0 tx=0x7fa9f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c101f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 0x556747c5d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747b1c020 con 0x556747c372c0 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747b79ba0 con 0x556747c0d5d0 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556747b77330 con 0x556747c0fe10 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.671+0000 7faa0a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 0x556747c5d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c101f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52126/0 (socket says 172.21.15.28:52126) 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c101f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:60160/0 (socket says 172.21.15.28:60160) 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 -- 172.21.15.28:0/2471703476 learned_addr learned my addr 172.21.15.28:0/2471703476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 0x556747c5d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:47500/0 (socket says 172.21.15.28:47500) 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 -- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 msgr2=0x556747c101f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c101f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 -- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 msgr2=0x556747c5d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 0x556747c5d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 -- 172.21.15.28:0/2471703476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556747b24cf0 con 0x556747c372c0 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.672+0000 7faa0b400640 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7faa04034040 tx=0x7faa0402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdc812653d43184b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7fa9f7400640 1 -- 172.21.15.28:0/2471703476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0403c680 con 0x556747c372c0 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7fa9f7400640 1 -- 172.21.15.28:0/2471703476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0403c800 con 0x556747c372c0 2024-09-13T17:22:15.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7fa9f7400640 1 -- 172.21.15.28:0/2471703476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0403cbb0 con 0x556747c372c0 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 -- 172.21.15.28:0/2471703476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 msgr2=0x556747c5d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7faa04034040 tx=0x7faa0402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 -- 172.21.15.28:0/2471703476 shutdown_connections 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5d820 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c0d5d0 0x556747c5d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 --2- 172.21.15.28:0/2471703476 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 -- 172.21.15.28:0/2471703476 >> 172.21.15.28:0/2471703476 conn(0x556747c0c540 msgr2=0x556747c549f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 -- 172.21.15.28:0/2471703476 shutdown_connections 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.673+0000 7faa14441740 1 -- 172.21.15.28:0/2471703476 wait complete. 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.674+0000 7faa14441740 1 -- start start 2024-09-13T17:22:15.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.674+0000 7faa14441740 1 Processor -- start 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c372c0 0x556747c38810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c59ad0 0x556747c38d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747b72c20 con 0x556747c0fe10 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747b141b0 con 0x556747c372c0 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556747b13e50 con 0x556747c59ad0 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa14441740 10 client.0 _ll_register_callbacks cb 0x556747b6c290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa0aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52138/0 (socket says 172.21.15.28:52138) 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.675+0000 7faa0aa00640 1 -- 172.21.15.28:0/1760789804 learned_addr learned my addr 172.21.15.28:0/1760789804 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0a000640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c372c0 0x556747c38810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0b400640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c59ad0 0x556747c38d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 -- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c59ad0 msgr2=0x556747c38d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c59ad0 0x556747c38d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 -- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c372c0 msgr2=0x556747c38810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c372c0 0x556747c38810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 -- 172.21.15.28:0/1760789804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c06b170 con 0x556747c0fe10 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa0aa00640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0574a0 tx=0x7fa9fc0578e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75b2b619f8d9b27d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7fa9ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc05b3b0 con 0x556747c0fe10 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9fc055c00 con 0x556747c0fe10 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556747b9beb0 con 0x556747c0fe10 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9fc06f180 con 0x556747c0fe10 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa14441740 10 client.11283 fetch_fsmap learned FSMap version 165 2024-09-13T17:22:15.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.676+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x556747b712a0 con 0x556747c0fe10 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556747b9beb0 con 0x556747c0fe10 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa9fc05f700 con 0x556747c0fe10 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 10 client.11283 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 10 client.11283 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 20 client.11283 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c23910 0x556747c23d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 4 client.11283 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556747b67340 con 0x556747c23910 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.677+0000 7faa0a000640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c23910 0x556747c23d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7faa0a000640 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c23910 0x556747c23d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc0169175132d974 server_cookie=d165aa0a8d69edd0 in_seq=0 out_seq=0 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7fa9ebe00640 10 client.11283 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7fa9ebe00640 1 -- 172.21.15.28:0/1760789804 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x556747b67340 con 0x556747c23910 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7fa9ebe00640 10 client.11283 handle_command_reply: tid=0 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7faa14441740 1 client.11283 shutdown 2024-09-13T17:22:15.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 msgr2=0x556747c51f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.678+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0574a0 tx=0x7fa9fc0578e0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 shutdown_connections 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c23910 0x556747c23d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c59ad0 0x556747c38d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c372c0 0x556747c38810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7fa9ebe00640 0 client.11283 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 --2- 172.21.15.28:0/1760789804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c0fe10 0x556747c51f10 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 >> 172.21.15.28:0/1760789804 conn(0x556747c0d5d0 msgr2=0x556747c52d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 shutdown_connections 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 1 -- 172.21.15.28:0/1760789804 wait complete. 2024-09-13T17:22:15.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.679+0000 7faa14441740 20 client.11283 trim_cache size 0 max 16384 2024-09-13T17:22:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.691+0000 7faa14441740 1 Processor -- start 2024-09-13T17:22:15.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.691+0000 7faa14441740 1 -- start start 2024-09-13T17:22:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c56a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c5c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567478da040 con 0x556747c372c0 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747c38ff0 con 0x556747c59ad0 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556747c67510 con 0x556747c0fe10 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52142/0 (socket says 172.21.15.28:52142) 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 -- 172.21.15.28:0/738716102 learned_addr learned my addr 172.21.15.28:0/738716102 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0aa00640 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c56a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0b400640 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c5c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 -- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 msgr2=0x556747c56a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c56a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 -- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 msgr2=0x556747c5c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c5c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.692+0000 7faa0a000640 1 -- 172.21.15.28:0/738716102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556747ba5200 con 0x556747c372c0 2024-09-13T17:22:15.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa0a000640 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8006fd0 tx=0x7fa9f8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99737a58f2ff51ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7fa9f7e00640 1 -- 172.21.15.28:0/738716102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8029070 con 0x556747c372c0 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7fa9f7e00640 1 -- 172.21.15.28:0/738716102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f802a310 con 0x556747c372c0 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7fa9f7e00640 1 -- 172.21.15.28:0/738716102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f802a680 con 0x556747c372c0 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 -- 172.21.15.28:0/738716102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 msgr2=0x556747c5c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 secure :-1 s=READY pgs=2361 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8006fd0 tx=0x7fa9f8016000 comp rx=0 tx=0).stop 2024-09-13T17:22:15.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 -- 172.21.15.28:0/738716102 shutdown_connections 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c5c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c5c320 unknown :-1 s=CLOSED pgs=2361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 --2- 172.21.15.28:0/738716102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747c56a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 -- 172.21.15.28:0/738716102 >> 172.21.15.28:0/738716102 conn(0x556747c0d5d0 msgr2=0x556747c05b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 -- 172.21.15.28:0/738716102 shutdown_connections 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.693+0000 7faa14441740 1 -- 172.21.15.28:0/738716102 wait complete. 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.694+0000 7faa14441740 1 -- start start 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.694+0000 7faa14441740 1 Processor -- start 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747ca4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c52f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747c63650 con 0x556747c372c0 2024-09-13T17:22:15.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556747c66da0 con 0x556747c59ad0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556747c54dd0 con 0x556747c0fe10 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.695+0000 7faa14441740 10 client.0 _ll_register_callbacks cb 0x7fa9fc0572e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52154/0 (socket says 172.21.15.28:52154) 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 -- 172.21.15.28:0/3083109831 learned_addr learned my addr 172.21.15.28:0/3083109831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747ca4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0b400640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c52f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 -- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 msgr2=0x556747ca4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747ca4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 -- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 msgr2=0x556747c52f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c52f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 -- 172.21.15.28:0/3083109831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f800e050 con 0x556747c372c0 2024-09-13T17:22:15.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7faa0a000640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8008010 tx=0x7fa9f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162ed7b9e1eee22c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.696+0000 7fa9ebe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f802a8b0 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556747c3bf10 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9f8005ba0 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=165) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9f8005d60 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 10 client.11289 fetch_fsmap learned FSMap version 165 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=165,monmap=2+}) v3 -- 0x556747b135f0 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556747c3bf10 con 0x556747c372c0 2024-09-13T17:22:15.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 165) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa9f8032070 con 0x556747c372c0 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 10 client.11289 fetch_fsmap finished waiting for FSMap version 165 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 10 client.11289 resolve_mds: resolved 13:0 to role '13:0' aka daemon mds.c 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 20 client.11289 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c63230 0x556747cbf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 4 client.11289 mds_command: new command op to 11049 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.697+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 --> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7faa0403c800 con 0x556747c63230 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c63230 0x556747cbf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa0aa00640 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c63230 0x556747cbf090 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df22f0a0613f2544 server_cookie=8cbccac8ca59935c in_seq=0 out_seq=0 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7fa9ebe00640 10 client.11289 ms_handle_connect on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7fa9ebe00640 1 -- 172.21.15.28:0/3083109831 <== mds.0 v2:172.21.15.28:6834/762464937 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7faa0403c800 con 0x556747c63230 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7fa9ebe00640 10 client.11289 handle_command_reply: tid=0 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 client.11289 shutdown 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 msgr2=0x556747c52a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8008010 tx=0x7fa9f8004730 comp rx=0 tx=0).stop 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 shutdown_connections 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:6834/762464937,v1:172.21.15.28:6835/762464937] conn(0x556747c63230 0x556747cbf090 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556747c59ad0 0x556747c52f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.698+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556747c372c0 0x556747c52a40 unknown :-1 s=CLOSED pgs=2362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa14441740 1 --2- 172.21.15.28:0/3083109831 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556747c0fe10 0x556747ca4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 >> 172.21.15.28:0/3083109831 conn(0x556747c0d5d0 msgr2=0x556747c57eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7fa9ebe00640 0 client.11289 ms_handle_reset on v2:172.21.15.28:6834/762464937 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 shutdown_connections 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa14441740 1 -- 172.21.15.28:0/3083109831 wait complete. 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa14441740 20 client.11289 trim_cache size 0 max 16384 2024-09-13T17:22:15.705 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-13T17:22:15.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 msgr2=0x7fa9ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 0x7fa9ec06d0c0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7faa0c1560d0 tx=0x7fa9f8008040 comp rx=0 tx=0).stop 2024-09-13T17:22:15.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 msgr2=0x7faa0c1f2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 secure :-1 s=READY pgs=2358 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00c400 tx=0x7fa9fc00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 shutdown_connections 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa9ec06ac00 0x7fa9ec06d0c0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa0c15c9a0 0x7faa0c154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa0c0b2400 0x7faa0c1541d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 --2- 172.21.15.28:0/3736781020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa0c0ad940 0x7faa0c1f2c60 unknown :-1 s=CLOSED pgs=2358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.699+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 >> 172.21.15.28:0/3736781020 conn(0x7faa0c0b0270 msgr2=0x7faa0c0b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.700+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 shutdown_connections 2024-09-13T17:22:15.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.700+0000 7faa11600640 1 -- 172.21.15.28:0/3736781020 wait complete. 2024-09-13T17:22:15.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T17:22:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.869+0000 7f7d4f400640 1 Processor -- start 2024-09-13T17:22:15.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.869+0000 7f7d4f400640 1 -- start start 2024-09-13T17:22:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.870+0000 7f7d4f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d480cabc0 0x7f7d4806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.870+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d48070cc0 0x7f7d48071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d4f400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4815e100 con 0x7f7d4815a540 2024-09-13T17:22:15.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d4f400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480adb00 con 0x7f7d480cabc0 2024-09-13T17:22:15.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d4f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d48071670 con 0x7f7d48070cc0 2024-09-13T17:22:15.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d480cabc0 0x7f7d4806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7d480cabc0 0x7f7d4806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:47530/0 (socket says 172.21.15.28:47530) 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 -- 172.21.15.28:0/2716961301 learned_addr learned my addr 172.21.15.28:0/2716961301 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d4ca00640 1 --1- 172.21.15.28:0/2716961301 >> v1:172.21.15.189:6789/0 conn(0x7f7d4815a540 0x7f7d4815e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:44196/0 (socket says 172.21.15.28:44196) 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d4d400640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d48070cc0 0x7f7d48071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 -- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d48070cc0 msgr2=0x7f7d48071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d48070cc0 0x7f7d48071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 -- 172.21.15.28:0/2716961301 >> v1:172.21.15.189:6789/0 conn(0x7f7d4815a540 legacy=0x7f7d4815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.871+0000 7f7d3fe00640 1 -- 172.21.15.28:0/2716961301 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4815e3e0 con 0x7f7d480cabc0 2024-09-13T17:22:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d3fe00640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d480cabc0 0x7f7d4806c430 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d08c0d83f369da90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d47e00640 1 -- 172.21.15.28:0/2716961301 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30032440 con 0x7f7d480cabc0 2024-09-13T17:22:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d47e00640 1 -- 172.21.15.28:0/2716961301 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d300325a0 con 0x7f7d480cabc0 2024-09-13T17:22:15.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d47e00640 1 -- 172.21.15.28:0/2716961301 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30032930 con 0x7f7d480cabc0 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 -- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d480cabc0 msgr2=0x7f7d4806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d480cabc0 0x7f7d4806c430 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009870 tx=0x7f7d3002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 -- 172.21.15.28:0/2716961301 shutdown_connections 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d48070cc0 0x7f7d48071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 --2- 172.21.15.28:0/2716961301 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d480cabc0 0x7f7d4806c430 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.873+0000 7f7d4f400640 1 -- 172.21.15.28:0/2716961301 >> 172.21.15.28:0/2716961301 conn(0x7f7d48069e80 msgr2=0x7f7d4806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.874+0000 7f7d4f400640 1 -- 172.21.15.28:0/2716961301 shutdown_connections 2024-09-13T17:22:15.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.874+0000 7f7d4f400640 1 -- 172.21.15.28:0/2716961301 wait complete. 2024-09-13T17:22:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.875+0000 7f7d4f400640 1 Processor -- start 2024-09-13T17:22:15.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.875+0000 7f7d4f400640 1 -- start start 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d48070cc0 0x7f7d480ab530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d4815a540 0x7f7d480a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480aba70 con 0x7f7d480cabc0 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480aa490 con 0x7f7d48070cc0 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d4f400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d480aa770 con 0x7f7d4815a540 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52166/0 (socket says 172.21.15.28:52166) 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.876+0000 7f7d3fe00640 1 -- 172.21.15.28:0/3217004812 learned_addr learned my addr 172.21.15.28:0/3217004812 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d4d400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d4815a540 0x7f7d480a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d4ca00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d48070cc0 0x7f7d480ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 -- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d4815a540 msgr2=0x7f7d480a9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d4815a540 0x7f7d480a9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 -- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d48070cc0 msgr2=0x7f7d480ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d48070cc0 0x7f7d480ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 -- 172.21.15.28:0/3217004812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d300094d0 con 0x7f7d480cabc0 2024-09-13T17:22:15.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d3fe00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f7d30000c00 tx=0x7f7d300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.877+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30032440 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.878+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d30030980 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.878+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d30030d30 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.878+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d480aaa50 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.878+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d480ab010 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.879+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d48071130 con 0x7f7d480cabc0 2024-09-13T17:22:15.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.880+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7d30039040 con 0x7f7d480cabc0 2024-09-13T17:22:15.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.880+0000 7f7d45600640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 0x7f7d2406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:15.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.880+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d300a9ca0 con 0x7f7d480cabc0 2024-09-13T17:22:15.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.881+0000 7f7d4ca00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 0x7f7d2406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:15.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.881+0000 7f7d4ca00640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 0x7f7d2406d0f0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000f60 tx=0x7f7d38007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:15.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:15.885+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7d30074e70 con 0x7f7d480cabc0 2024-09-13T17:22:16.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.099+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7d480aa8b0 con 0x7f7d480cabc0 2024-09-13T17:22:16.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.100+0000 7f7d45600640 1 -- 172.21.15.28:0/3217004812 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7d480aa8b0 con 0x7f7d480cabc0 2024-09-13T17:22:16.102 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:16.102 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T17:22:16.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 msgr2=0x7f7d2406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 0x7f7d2406d0f0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7d38000f60 tx=0x7f7d38007c80 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 msgr2=0x7f7d48147180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f7d30000c00 tx=0x7f7d300046c0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 shutdown_connections 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7d2406ac30 0x7f7d2406d0f0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7d4815a540 0x7f7d480a9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d480cabc0 0x7f7d48147180 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 --2- 172.21.15.28:0/3217004812 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d48070cc0 0x7f7d480ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 >> 172.21.15.28:0/3217004812 conn(0x7f7d48069e80 msgr2=0x7f7d480cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:16.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 shutdown_connections 2024-09-13T17:22:16.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.102+0000 7f7d4f400640 1 -- 172.21.15.28:0/3217004812 wait complete. 2024-09-13T17:22:16.121 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:22:16.121 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:22:16.219 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:22:16.307 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:22:16.307 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:16.308 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:16.344 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:16.344 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:16.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:16.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.613+0000 7fcf80e00640 1 Processor -- start 2024-09-13T17:22:16.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.613+0000 7fcf80e00640 1 -- start start 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf80e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf7c06f970 0x7fcf7c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf80e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 0x7fcf7c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf80e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c06cc70 con 0x7fcf7c1592a0 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf80e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf7c15b0b0 con 0x7fcf7c06f970 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf80e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c159d90 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.614+0000 7fcf71400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf7c06f970 0x7fcf7c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcf7c06f970 0x7fcf7c15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52178/0 (socket says 172.21.15.28:52178) 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 -- 172.21.15.28:0/2176120764 learned_addr learned my addr 172.21.15.28:0/2176120764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf7a800640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 0x7fcf7c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 -- 172.21.15.28:0/2176120764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 msgr2=0x7fcf7c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 0x7fcf7c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 -- 172.21.15.28:0/2176120764 >> v1:172.21.15.189:6789/0 conn(0x7fcf7c1592a0 legacy=0x7fcf7c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf71400640 1 -- 172.21.15.28:0/2176120764 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c0b2040 con 0x7fcf7c06f970 2024-09-13T17:22:16.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.615+0000 7fcf7a800640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 0x7fcf7c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.616+0000 7fcf71400640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06f970 0x7fcf7c15ab70 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c06ac10 tx=0x7fcf6402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a0077d64181d15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.616+0000 7fcf79400640 1 -- 172.21.15.28:0/2176120764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64032440 con 0x7fcf7c06f970 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.616+0000 7fcf79400640 1 -- 172.21.15.28:0/2176120764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf640325a0 con 0x7fcf7c06f970 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.616+0000 7fcf79400640 1 -- 172.21.15.28:0/2176120764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf64032930 con 0x7fcf7c06f970 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 -- 172.21.15.28:0/2176120764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06f970 msgr2=0x7fcf7c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06f970 0x7fcf7c15ab70 secure :-1 s=READY pgs=2364 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c06ac10 tx=0x7fcf6402a1e0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 -- 172.21.15.28:0/2176120764 shutdown_connections 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06c2d0 0x7fcf7c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 --2- 172.21.15.28:0/2176120764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06f970 0x7fcf7c15ab70 unknown :-1 s=CLOSED pgs=2364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 -- 172.21.15.28:0/2176120764 >> 172.21.15.28:0/2176120764 conn(0x7fcf7c069e80 msgr2=0x7fcf7c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 -- 172.21.15.28:0/2176120764 shutdown_connections 2024-09-13T17:22:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.617+0000 7fcf80e00640 1 -- 172.21.15.28:0/2176120764 wait complete. 2024-09-13T17:22:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.619+0000 7fcf80e00640 1 Processor -- start 2024-09-13T17:22:16.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.619+0000 7fcf80e00640 1 -- start start 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.619+0000 7fcf80e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.619+0000 7fcf80e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 0x7fcf7c1f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf80e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 0x7fcf7c1f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf80e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0aee90 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf80e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0af170 con 0x7fcf7c1592a0 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf80e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf7c0af450 con 0x7fcf7c06f970 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52194/0 (socket says 172.21.15.28:52194) 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 -- 172.21.15.28:0/1563979288 learned_addr learned my addr 172.21.15.28:0/1563979288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf7a800640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 0x7fcf7c1f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf71400640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 0x7fcf7c1f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 -- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 msgr2=0x7fcf7c1f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 0x7fcf7c1f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 -- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 msgr2=0x7fcf7c1f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 0x7fcf7c1f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf79e00640 1 -- 172.21.15.28:0/1563979288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf640094d0 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf7a800640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 0x7fcf7c1f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.620+0000 7fcf71400640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 0x7fcf7c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf79e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00b3f0 tx=0x7fcf6c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c011830 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf6c011990 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6c011d40 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c0af730 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.621+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c14c1e0 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.622+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7c0b3740 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.623+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcf6c01a400 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.624+0000 7fcf72a00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 0x7fcf5806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.624+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf6c089200 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.624+0000 7fcf71400640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 0x7fcf5806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.625+0000 7fcf71400640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 0x7fcf5806d0c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcf64034000 tx=0x7fcf6402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.628+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcf6c0543d0 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.812+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf7c159960 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.813+0000 7fcf72a00640 1 -- 172.21.15.28:0/1563979288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16583 (secure 0 0 0) 0x7fcf6c0579f0 con 0x7fcf7c06c2d0 2024-09-13T17:22:16.815 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:16.815 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:37.723750+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:16.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 msgr2=0x7fcf5806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 0x7fcf5806d0c0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcf64034000 tx=0x7fcf6402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 msgr2=0x7fcf7c0b0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00b3f0 tx=0x7fcf6c00b8b0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 shutdown_connections 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcf5806ac00 0x7fcf5806d0c0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcf7c1592a0 0x7fcf7c1f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcf7c06f970 0x7fcf7c1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 --2- 172.21.15.28:0/1563979288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcf7c06c2d0 0x7fcf7c0b0300 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 >> 172.21.15.28:0/1563979288 conn(0x7fcf7c069e80 msgr2=0x7fcf7c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 shutdown_connections 2024-09-13T17:22:16.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.815+0000 7fcf80e00640 1 -- 172.21.15.28:0/1563979288 wait complete. 2024-09-13T17:22:16.833 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:21:34.559482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:21:35.719796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '193', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:22:16.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:22:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.989+0000 7f8793000640 1 Processor -- start 2024-09-13T17:22:16.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.989+0000 7f8793000640 1 -- start start 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.990+0000 7f8793000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f878c15a780 0x7f878c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8793000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 0x7f878c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8793000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 0x7f878c06bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8793000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c06c8b0 con 0x7f878c15a780 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8793000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f878c06cb90 con 0x7f878c15ab60 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8793000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c0cc980 con 0x7f878c06bb40 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f878c15a780 0x7f878c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f878c15a780 0x7f878c070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52206/0 (socket says 172.21.15.28:52206) 2024-09-13T17:22:16.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 -- 172.21.15.28:0/1770790707 learned_addr learned my addr 172.21.15.28:0/1770790707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:16.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8783e00640 1 --2- 172.21.15.28:0/1770790707 >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 0x7f878c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 -- 172.21.15.28:0/1770790707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 msgr2=0x7f878c06bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8791000640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 0x7f878c06bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 0x7f878c06bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 -- 172.21.15.28:0/1770790707 >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 msgr2=0x7f878c0712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 --2- 172.21.15.28:0/1770790707 >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 0x7f878c0712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f878be00640 1 -- 172.21.15.28:0/1770790707 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0ccc60 con 0x7f878c15a780 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8783e00640 1 --2- 172.21.15.28:0/1770790707 >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 0x7f878c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.991+0000 7f8791000640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 0x7f878c06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.992+0000 7f878be00640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15a780 0x7f878c070d80 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f878c06f380 tx=0x7f877c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f18ccef3e09e7870 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f878b400640 1 -- 172.21.15.28:0/1770790707 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c032440 con 0x7f878c15a780 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f878b400640 1 -- 172.21.15.28:0/1770790707 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c0325a0 con 0x7f878c15a780 2024-09-13T17:22:16.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f878b400640 1 -- 172.21.15.28:0/1770790707 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c032930 con 0x7f878c15a780 2024-09-13T17:22:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 -- 172.21.15.28:0/1770790707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15a780 msgr2=0x7f878c070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15a780 0x7f878c070d80 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7f878c06f380 tx=0x7f877c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 -- 172.21.15.28:0/1770790707 shutdown_connections 2024-09-13T17:22:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c06bb40 0x7f878c06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 --2- 172.21.15.28:0/1770790707 >> v2:172.21.15.189:3300/0 conn(0x7f878c15ab60 0x7f878c0712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 --2- 172.21.15.28:0/1770790707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15a780 0x7f878c070d80 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 -- 172.21.15.28:0/1770790707 >> 172.21.15.28:0/1770790707 conn(0x7f878c0b0270 msgr2=0x7f878c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 -- 172.21.15.28:0/1770790707 shutdown_connections 2024-09-13T17:22:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.993+0000 7f8793000640 1 -- 172.21.15.28:0/1770790707 wait complete. 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.995+0000 7f8793000640 1 Processor -- start 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.995+0000 7f8793000640 1 -- start start 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 0x7f878c152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 0x7f878c154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c154f70 con 0x7f878c15ab60 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c155250 con 0x7f878c06bb40 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8793000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f878c155530 con 0x7f878c15a780 2024-09-13T17:22:16.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8791000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8791000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52214/0 (socket says 172.21.15.28:52214) 2024-09-13T17:22:16.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.996+0000 7f8791000640 1 -- 172.21.15.28:0/2987250980 learned_addr learned my addr 172.21.15.28:0/2987250980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f878be00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 0x7f878c152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8783e00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 0x7f878c154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 -- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 msgr2=0x7f878c154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 0x7f878c154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 -- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 msgr2=0x7f878c152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 0x7f878c152000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 -- 172.21.15.28:0/2987250980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c0094d0 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f878be00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 0x7f878c152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8783e00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 0x7f878c154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8791000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f878400cef0 tx=0x7f878400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400d840 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878400d9c0 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878400dd70 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c155810 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.997+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c152540 con 0x7f878c15ab60 2024-09-13T17:22:17.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.998+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c06bf70 con 0x7f878c15ab60 2024-09-13T17:22:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:16.999+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f878401e3b0 con 0x7f878c15ab60 2024-09-13T17:22:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.000+0000 7f8788c00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 0x7f876806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.000+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8784089e40 con 0x7f878c15ab60 2024-09-13T17:22:17.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.001+0000 7f878be00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 0x7f876806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.001+0000 7f878be00640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 0x7f876806d0c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f877c034000 tx=0x7f877c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.005+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f87840550d0 con 0x7f878c15ab60 2024-09-13T17:22:17.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.180+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f878c0cdc30 con 0x7f878c15ab60 2024-09-13T17:22:17.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.180+0000 7f8788c00640 1 -- 172.21.15.28:0/2987250980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f878401a020 con 0x7f878c15ab60 2024-09-13T17:22:17.183 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.182+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 msgr2=0x7f876806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.182+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 0x7f876806d0c0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f877c034000 tx=0x7f877c02e000 comp rx=0 tx=0).stop 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.182+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 msgr2=0x7f878c0aec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.182+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 secure :-1 s=READY pgs=2367 cs=0 l=1 rev1=1 crypto rx=0x7f878400cef0 tx=0x7f878400a710 comp rx=0 tx=0).stop 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 shutdown_connections 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f876806ac00 0x7f876806d0c0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f878c15ab60 0x7f878c0aec40 unknown :-1 s=CLOSED pgs=2367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f878c15a780 0x7f878c154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 --2- 172.21.15.28:0/2987250980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f878c06bb40 0x7f878c152000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 >> 172.21.15.28:0/2987250980 conn(0x7f878c0b0270 msgr2=0x7f878c157a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 shutdown_connections 2024-09-13T17:22:17.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.183+0000 7f8793000640 1 -- 172.21.15.28:0/2987250980 wait complete. 2024-09-13T17:22:17.201 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:22:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.361+0000 7f8166800640 1 Processor -- start 2024-09-13T17:22:17.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.361+0000 7f8166800640 1 -- start start 2024-09-13T17:22:17.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8166800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816015ab60 0x7f816006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8166800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8160071490 0x7f816006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8166800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f816006c8b0 con 0x7f816015a780 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8166800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f816006cb90 con 0x7f816015ab60 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8166800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81600cc980 con 0x7f8160071490 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8156a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816015ab60 0x7f816006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8156a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f816015ab60 0x7f816006bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52222/0 (socket says 172.21.15.28:52222) 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.362+0000 7f8156a00640 1 -- 172.21.15.28:0/237514272 learned_addr learned my addr 172.21.15.28:0/237514272 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:17.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.363+0000 7f815fe00640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8160071490 0x7f816006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.363+0000 7f8156a00640 1 -- 172.21.15.28:0/237514272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8160071490 msgr2=0x7f816006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.363+0000 7f8156a00640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8160071490 0x7f816006c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.363+0000 7f8156a00640 1 -- 172.21.15.28:0/237514272 >> v1:172.21.15.189:6789/0 conn(0x7f816015a780 legacy=0x7f8160070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.363+0000 7f8156a00640 1 -- 172.21.15.28:0/237514272 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81600ccc60 con 0x7f816015ab60 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.364+0000 7f8156a00640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015ab60 0x7f816006bb40 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f814802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faad7c3abcbbed53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.364+0000 7f815ea00640 1 -- 172.21.15.28:0/237514272 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148032440 con 0x7f816015ab60 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.364+0000 7f815ea00640 1 -- 172.21.15.28:0/237514272 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81480325a0 con 0x7f816015ab60 2024-09-13T17:22:17.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.364+0000 7f815ea00640 1 -- 172.21.15.28:0/237514272 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148032930 con 0x7f816015ab60 2024-09-13T17:22:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 -- 172.21.15.28:0/237514272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015ab60 msgr2=0x7f816006bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015ab60 0x7f816006bb40 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f8148009870 tx=0x7f814802a270 comp rx=0 tx=0).stop 2024-09-13T17:22:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 -- 172.21.15.28:0/237514272 shutdown_connections 2024-09-13T17:22:17.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8160071490 0x7f816006c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 --2- 172.21.15.28:0/237514272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015ab60 0x7f816006bb40 unknown :-1 s=CLOSED pgs=2368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 -- 172.21.15.28:0/237514272 >> 172.21.15.28:0/237514272 conn(0x7f81600b0270 msgr2=0x7f81600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 -- 172.21.15.28:0/237514272 shutdown_connections 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.365+0000 7f8166800640 1 -- 172.21.15.28:0/237514272 wait complete. 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.366+0000 7f8166800640 1 Processor -- start 2024-09-13T17:22:17.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 -- start start 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8160071490 0x7f81601549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816015ab60 0x7f81600b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8160151e30 con 0x7f816015a780 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81600b2820 con 0x7f8160071490 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.367+0000 7f8166800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81600b2b00 con 0x7f816015ab60 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52234/0 (socket says 172.21.15.28:52234) 2024-09-13T17:22:17.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 -- 172.21.15.28:0/3241128610 learned_addr learned my addr 172.21.15.28:0/3241128610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f815f400640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8160071490 0x7f81601549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f815fe00640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816015ab60 0x7f81600b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 -- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816015ab60 msgr2=0x7f81600b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816015ab60 0x7f81600b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 -- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8160071490 msgr2=0x7f81601549a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8160071490 0x7f81601549a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.368+0000 7f8156a00640 1 -- 172.21.15.28:0/3241128610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81480094d0 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8156a00640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f8148000c00 tx=0x7f81480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148032440 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8148030980 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8148030d30 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81600b2de0 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.369+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81601515a0 con 0x7f816015a780 2024-09-13T17:22:17.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.370+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8160070d80 con 0x7f816015a780 2024-09-13T17:22:17.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.371+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8148039040 con 0x7f816015a780 2024-09-13T17:22:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.372+0000 7f8157e00640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 0x7f813c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.372+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f81480a9c00 con 0x7f816015a780 2024-09-13T17:22:17.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.372+0000 7f815f400640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 0x7f813c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.373+0000 7f815f400640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 0x7f813c06d0c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f81600b36b0 tx=0x7f8150009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.376+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8148074df0 con 0x7f816015a780 2024-09-13T17:22:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.556+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8160157990 con 0x7f816015a780 2024-09-13T17:22:17.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.556+0000 7f8157e00640 1 -- 172.21.15.28:0/3241128610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v193) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8148078410 con 0x7f816015a780 2024-09-13T17:22:17.558 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 msgr2=0x7f813c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 0x7f813c06d0c0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f81600b36b0 tx=0x7f8150009040 comp rx=0 tx=0).stop 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 msgr2=0x7f81600b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f8148000c00 tx=0x7f81480046c0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 shutdown_connections 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f813c06ac00 0x7f813c06d0c0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f816015ab60 0x7f81600b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f816015a780 0x7f81600b1670 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.558+0000 7f8166800640 1 --2- 172.21.15.28:0/3241128610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8160071490 0x7f81601549a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.559+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 >> 172.21.15.28:0/3241128610 conn(0x7f81600b0270 msgr2=0x7f81600cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.559+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 shutdown_connections 2024-09-13T17:22:17.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.559+0000 7f8166800640 1 -- 172.21.15.28:0/3241128610 wait complete. 2024-09-13T17:22:17.577 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:22:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.733+0000 7fc3fde00640 1 Processor -- start 2024-09-13T17:22:17.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.733+0000 7fc3fde00640 1 -- start start 2024-09-13T17:22:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.734+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f815a780 0x7fc3f8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3fde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3f806c250 0x7fc3f80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3fde00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f806c8b0 con 0x7fc3f815ab60 2024-09-13T17:22:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3fde00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f806c6a0 con 0x7fc3f806c250 2024-09-13T17:22:17.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3fde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f806cb90 con 0x7fc3f815a780 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc3f806c250 0x7fc3f80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3ee000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc3f815ab60 0x7fc3f806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59818/0 (socket says 172.21.15.28:59818) 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3ee000640 1 -- 172.21.15.28:0/1963898664 learned_addr learned my addr 172.21.15.28:0/1963898664 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f815a780 0x7fc3f8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f7400640 1 -- 172.21.15.28:0/1963898664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f815a780 msgr2=0x7fc3f8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f7400640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f815a780 0x7fc3f8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f7400640 1 -- 172.21.15.28:0/1963898664 >> v1:172.21.15.28:6789/0 conn(0x7fc3f815ab60 legacy=0x7fc3f806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f7400640 1 -- 172.21.15.28:0/1963898664 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80cc980 con 0x7fc3f806c250 2024-09-13T17:22:17.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.735+0000 7fc3f6000640 1 -- 172.21.15.28:0/1963898664 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260728091 0 0) 0x7fc3f806c8b0 con 0x7fc3f815ab60 2024-09-13T17:22:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.736+0000 7fc3f7400640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f806c250 0x7fc3f80712c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b1c393a87723a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3f6000640 1 -- 172.21.15.28:0/1963898664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0007740 con 0x7fc3f806c250 2024-09-13T17:22:17.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3f6000640 1 -- 172.21.15.28:0/1963898664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f002d050 con 0x7fc3f806c250 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3f6000640 1 -- 172.21.15.28:0/1963898664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f0007ac0 con 0x7fc3f806c250 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 -- 172.21.15.28:0/1963898664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f806c250 msgr2=0x7fc3f80712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f806c250 0x7fc3f80712c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f002a660 comp rx=0 tx=0).stop 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 -- 172.21.15.28:0/1963898664 shutdown_connections 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f806c250 0x7fc3f80712c0 unknown :-1 s=CLOSED pgs=2371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 --2- 172.21.15.28:0/1963898664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f815a780 0x7fc3f8070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 -- 172.21.15.28:0/1963898664 >> 172.21.15.28:0/1963898664 conn(0x7fc3f80b0270 msgr2=0x7fc3f80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 -- 172.21.15.28:0/1963898664 shutdown_connections 2024-09-13T17:22:17.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.737+0000 7fc3fde00640 1 -- 172.21.15.28:0/1963898664 wait complete. 2024-09-13T17:22:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.739+0000 7fc3fde00640 1 Processor -- start 2024-09-13T17:22:17.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.739+0000 7fc3fde00640 1 -- start start 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f806c250 0x7fc3f81f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f815a780 0x7fc3f80c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80c6de0 con 0x7fc3f815ab60 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f8158170 con 0x7fc3f815a780 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3fde00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80c8580 con 0x7fc3f806c250 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.740+0000 7fc3f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52254/0 (socket says 172.21.15.28:52254) 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 -- 172.21.15.28:0/3180161400 learned_addr learned my addr 172.21.15.28:0/3180161400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:17.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f806c250 0x7fc3f81f25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3ee000640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f815a780 0x7fc3f80c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 -- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f806c250 msgr2=0x7fc3f81f25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f806c250 0x7fc3f81f25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 -- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f815a780 msgr2=0x7fc3f80c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f815a780 0x7fc3f80c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 -- 172.21.15.28:0/3180161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f0009a70 con 0x7fc3f815ab60 2024-09-13T17:22:17.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.741+0000 7fc3f7400640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.742+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003c830 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.742+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f003c990 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.742+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f80c8860 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.742+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f003cd00 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.742+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f80c8e20 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.743+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f8067fd0 con 0x7fc3f815ab60 2024-09-13T17:22:17.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.744+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc3f0031400 con 0x7fc3f815ab60 2024-09-13T17:22:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.744+0000 7fc3efe00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:17.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.745+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3f00aaaf0 con 0x7fc3f815ab60 2024-09-13T17:22:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.745+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:17.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.746+0000 7fc3f6a00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009b40 tx=0x7fc3e8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:17.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.749+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc3f0075cf0 con 0x7fc3f815ab60 2024-09-13T17:22:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.926+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3f8067080 con 0x7fc3f815ab60 2024-09-13T17:22:17.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.926+0000 7fc3efe00640 1 -- 172.21.15.28:0/3180161400 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v193) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3f003ce60 con 0x7fc3f815ab60 2024-09-13T17:22:17.928 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:22:17.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 msgr2=0x7fc3d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009b40 tx=0x7fc3e8009190 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 msgr2=0x7fc3f80c68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7fc3f0009d90 tx=0x7fc3f0004a40 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 shutdown_connections 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc3d406ac00 0x7fc3d406d0c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3f815ab60 0x7fc3f80c68a0 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3f815a780 0x7fc3f80c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 --2- 172.21.15.28:0/3180161400 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc3f806c250 0x7fc3f81f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.928+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 >> 172.21.15.28:0/3180161400 conn(0x7fc3f80b0270 msgr2=0x7fc3f80acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.929+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 shutdown_connections 2024-09-13T17:22:17.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:17.929+0000 7fc3fde00640 1 -- 172.21.15.28:0/3180161400 wait complete. 2024-09-13T17:22:17.946 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:22:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.101+0000 7f3d3d800640 1 Processor -- start 2024-09-13T17:22:18.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.101+0000 7f3d3d800640 1 -- start start 2024-09-13T17:22:18.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.102+0000 7f3d3d800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d3815c9a0 0x7f3d380ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.102+0000 7f3d3d800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d38070470 con 0x7f3d380b2400 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d3d800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d380ace30 con 0x7f3d380ad940 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d3d800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d380ad110 con 0x7f3d3815c9a0 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d3815c9a0 0x7f3d380ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3d3815c9a0 0x7f3d380ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52268/0 (socket says 172.21.15.28:52268) 2024-09-13T17:22:18.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 -- 172.21.15.28:0/1034076934 learned_addr learned my addr 172.21.15.28:0/1034076934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 -- 172.21.15.28:0/1034076934 >> v1:172.21.15.189:6789/0 conn(0x7f3d380ad940 legacy=0x7f3d380b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 -- 172.21.15.28:0/1034076934 >> v1:172.21.15.28:6789/0 conn(0x7f3d380b2400 legacy=0x7f3d3806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.103+0000 7f3d36a00640 1 -- 172.21.15.28:0/1034076934 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3806b170 con 0x7f3d3815c9a0 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.104+0000 7f3d36a00640 1 --2- 172.21.15.28:0/1034076934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d380ac850 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098a0 tx=0x7f3d2802a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3f1ba8ab65d774 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d36000640 1 -- 172.21.15.28:0/1034076934 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28032640 con 0x7f3d3815c9a0 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d36000640 1 -- 172.21.15.28:0/1034076934 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d280327a0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d36000640 1 -- 172.21.15.28:0/1034076934 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d28032b30 con 0x7f3d3815c9a0 2024-09-13T17:22:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 -- 172.21.15.28:0/1034076934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 msgr2=0x7f3d380ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 --2- 172.21.15.28:0/1034076934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d380ac850 secure :-1 s=READY pgs=2374 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098a0 tx=0x7f3d2802a3c0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 -- 172.21.15.28:0/1034076934 shutdown_connections 2024-09-13T17:22:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 --2- 172.21.15.28:0/1034076934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d380ac850 unknown :-1 s=CLOSED pgs=2374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 -- 172.21.15.28:0/1034076934 >> 172.21.15.28:0/1034076934 conn(0x7f3d380b0270 msgr2=0x7f3d380b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 -- 172.21.15.28:0/1034076934 shutdown_connections 2024-09-13T17:22:18.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.105+0000 7f3d3d800640 1 -- 172.21.15.28:0/1034076934 wait complete. 2024-09-13T17:22:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.107+0000 7f3d3d800640 1 Processor -- start 2024-09-13T17:22:18.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.107+0000 7f3d3d800640 1 -- start start 2024-09-13T17:22:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d380ad940 0x7f3d380c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d380b2400 0x7f3d380c5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c6410 con 0x7f3d3815c9a0 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c5580 con 0x7f3d380ad940 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d3d800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380c5800 con 0x7f3d380b2400 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d380ad940 0x7f3d380c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d37400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52270/0 (socket says 172.21.15.28:52270) 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.108+0000 7f3d37400640 1 -- 172.21.15.28:0/2386918421 learned_addr learned my addr 172.21.15.28:0/2386918421 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:18.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d2fe00640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d380b2400 0x7f3d380c5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 -- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d380b2400 msgr2=0x7f3d380c5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d380b2400 0x7f3d380c5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 -- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d380ad940 msgr2=0x7f3d380c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d380ad940 0x7f3d380c7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 -- 172.21.15.28:0/2386918421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280094d0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d37400640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c760 tx=0x7f3d3000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d30003b70 con 0x7f3d3815c9a0 2024-09-13T17:22:18.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d30003cd0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d3000aa90 con 0x7f3d3815c9a0 2024-09-13T17:22:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.109+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38070930 con 0x7f3d3815c9a0 2024-09-13T17:22:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.110+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d380c5dc0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.111+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d380c65c0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.111+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3d3000abf0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.112+0000 7f3d2ea00640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 0x7f3d1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.112+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d300890c0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.113+0000 7f3d36a00640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 0x7f3d1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.113+0000 7f3d36a00640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 0x7f3d1406d0c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3d28034000 tx=0x7f3d2802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.117+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3d380c65c0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.308+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3d380c65c0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.308+0000 7f3d2ea00640 1 -- 172.21.15.28:0/2386918421 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v193) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3d380c65c0 con 0x7f3d3815c9a0 2024-09-13T17:22:18.310 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.310+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 msgr2=0x7f3d1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.310+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 0x7f3d1406d0c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3d28034000 tx=0x7f3d2802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.310+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 msgr2=0x7f3d3806bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.310+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f3d3000c760 tx=0x7f3d3000cc20 comp rx=0 tx=0).stop 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 shutdown_connections 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3d1406ac00 0x7f3d1406d0c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d3815c9a0 0x7f3d3806bdc0 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3d380b2400 0x7f3d380c5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 --2- 172.21.15.28:0/2386918421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d380ad940 0x7f3d380c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 >> 172.21.15.28:0/2386918421 conn(0x7f3d380b0270 msgr2=0x7f3d3815a510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 shutdown_connections 2024-09-13T17:22:18.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.311+0000 7f3d3d800640 1 -- 172.21.15.28:0/2386918421 wait complete. 2024-09-13T17:22:18.328 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:22:18.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:18.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:22:18.391 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:22:18.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:18.392 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:22:18.428 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:22:18.428 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:18.468 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:22:18.468 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:18.468 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:18.504 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:22:18.504 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:22:18.504 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:18.546 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:22:18.546 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.700+0000 7f6a51c00640 1 Processor -- start 2024-09-13T17:22:18.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.700+0000 7f6a51c00640 1 -- start start 2024-09-13T17:22:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.701+0000 7f6a51c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6a4c0ce9f0 0x7f6a4c070820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.701+0000 7f6a51c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c070d60 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a51c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a4c0cd190 con 0x7f6a4c0b16e0 2024-09-13T17:22:18.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a51c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a4c0cd470 con 0x7f6a4c0ce9f0 2024-09-13T17:22:18.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4aa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6a4c15a5f0 0x7f6a4c0ce2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59836/0 (socket says 172.21.15.28:59836) 2024-09-13T17:22:18.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4aa00640 1 -- 172.21.15.28:0/1729520888 learned_addr learned my addr 172.21.15.28:0/1729520888 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:18.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a42000640 1 --2- 172.21.15.28:0/1729520888 >> v2:172.21.15.189:3300/0 conn(0x7f6a4c0ce9f0 0x7f6a4c070820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962948337 0 0) 0x7f6a4c070d60 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30003520 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2907490941 0 0) 0x7f6a30003520 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 >> v2:172.21.15.189:3300/0 conn(0x7f6a4c0ce9f0 msgr2=0x7f6a4c070820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 --2- 172.21.15.28:0/1729520888 >> v2:172.21.15.189:3300/0 conn(0x7f6a4c0ce9f0 0x7f6a4c070820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 >> v1:172.21.15.189:6789/0 conn(0x7f6a4c0b16e0 legacy=0x7f6a4c0b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c0cd750 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.702+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6a3c001670 con 0x7f6a4c15a5f0 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 >> v1:172.21.15.28:6789/0 conn(0x7f6a4c15a5f0 legacy=0x7f6a4c0ce2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a300039c0 0x7f6a30003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a30005180 0x7f6a3000fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c0cd470 con 0x7f6a300047c0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30010490 con 0x7f6a30005180 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a30010650 con 0x7f6a300039c0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4aa00640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a300039c0 0x7f6a30003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4b400640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a30005180 0x7f6a3000fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 -- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a300039c0 msgr2=0x7f6a30003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a300039c0 0x7f6a30003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 -- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a30005180 msgr2=0x7f6a3000fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a30005180 0x7f6a3000fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a42000640 1 -- 172.21.15.28:0/1729520888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30003520 con 0x7f6a300047c0 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4b400640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a30005180 0x7f6a3000fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:18.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.703+0000 7f6a4b400640 1 -- 172.21.15.28:0/1729520888 reap_dead start 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.704+0000 7f6a42000640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a30004c40 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009ac0 tx=0x7f6a300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f02e3bb715f2829 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34031400 con 0x7f6a300047c0 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3402d050 con 0x7f6a300047c0 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a4a000640 1 -- 172.21.15.28:0/1729520888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34031780 con 0x7f6a300047c0 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 -- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 msgr2=0x7f6a30004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a30004c40 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f6a34009ac0 tx=0x7f6a300057f0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 -- 172.21.15.28:0/1729520888 shutdown_connections 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 --2- 172.21.15.28:0/1729520888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a30004c40 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 -- 172.21.15.28:0/1729520888 >> 172.21.15.28:0/1729520888 conn(0x7f6a4c073580 msgr2=0x7f6a4c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 -- 172.21.15.28:0/1729520888 shutdown_connections 2024-09-13T17:22:18.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.705+0000 7f6a51c00640 1 -- 172.21.15.28:0/1729520888 wait complete. 2024-09-13T17:22:18.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.707+0000 7f6a51c00640 1 Processor -- start 2024-09-13T17:22:18.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.707+0000 7f6a51c00640 1 -- start start 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a4c153f00 0x7f6a4c154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 0x7f6a4c15cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c1548c0 con 0x7f6a300047c0 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c15adc0 con 0x7f6a4c15c770 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a51c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c15b0a0 con 0x7f6a4c153f00 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a4aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a4aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52286/0 (socket says 172.21.15.28:52286) 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a4aa00640 1 -- 172.21.15.28:0/663721346 learned_addr learned my addr 172.21.15.28:0/663721346 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a42000640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a4c153f00 0x7f6a4c154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.708+0000 7f6a4b400640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 0x7f6a4c15cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 -- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a4c153f00 msgr2=0x7f6a4c154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a4c153f00 0x7f6a4c154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 -- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 msgr2=0x7f6a4c15cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 0x7f6a4c15cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 -- 172.21.15.28:0/663721346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4c15b380 con 0x7f6a300047c0 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4b400640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 0x7f6a4c15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a4aa00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c0072c0 tx=0x7f6a3c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c00f940 con 0x7f6a300047c0 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3c00fac0 con 0x7f6a300047c0 2024-09-13T17:22:18.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c00fe70 con 0x7f6a300047c0 2024-09-13T17:22:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c15b660 con 0x7f6a300047c0 2024-09-13T17:22:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.709+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c15bb80 con 0x7f6a300047c0 2024-09-13T17:22:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.710+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c15b850 con 0x7f6a300047c0 2024-09-13T17:22:18.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.711+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6a3c01d460 con 0x7f6a300047c0 2024-09-13T17:22:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.712+0000 7f6a43400640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 0x7f6a2806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.712+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6a3c089510 con 0x7f6a300047c0 2024-09-13T17:22:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.713+0000 7f6a42000640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 0x7f6a2806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:18.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.714+0000 7f6a42000640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 0x7f6a2806d0f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6a34033000 tx=0x7f6a34008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:18.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.716+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6a3c054650 con 0x7f6a300047c0 2024-09-13T17:22:18.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.932+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a4c0712d0 con 0x7f6a300047c0 2024-09-13T17:22:18.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.933+0000 7f6a43400640 1 -- 172.21.15.28:0/663721346 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f6a3c057c70 con 0x7f6a300047c0 2024-09-13T17:22:18.935 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:18.935 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:22:18.936 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 msgr2=0x7f6a2806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 0x7f6a2806d0f0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6a34033000 tx=0x7f6a34008030 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 msgr2=0x7f6a4c14e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c0072c0 tx=0x7f6a3c007780 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 shutdown_connections 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6a2806ac30 0x7f6a2806d0f0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6a4c15c770 0x7f6a4c15cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6a4c153f00 0x7f6a4c154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 --2- 172.21.15.28:0/663721346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a300047c0 0x7f6a4c14e980 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:18.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 >> 172.21.15.28:0/663721346 conn(0x7f6a4c073580 msgr2=0x7f6a4c156430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:18.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 shutdown_connections 2024-09-13T17:22:18.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:18.935+0000 7f6a51c00640 1 -- 172.21.15.28:0/663721346 wait complete. 2024-09-13T17:22:18.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.109+0000 7fcfeba00640 1 Processor -- start 2024-09-13T17:22:19.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.109+0000 7fcfeba00640 1 -- start start 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.110+0000 7fcfeba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcfe406f970 0x7fcfe415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.110+0000 7fcfeba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 0x7fcfe406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.110+0000 7fcfeba00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe406cc70 con 0x7fcfe41592a0 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.110+0000 7fcfeba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe415b0b0 con 0x7fcfe406f970 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.110+0000 7fcfeba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe4159d90 con 0x7fcfe406c2d0 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcfe406f970 0x7fcfe415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcfe406f970 0x7fcfe415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52288/0 (socket says 172.21.15.28:52288) 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 -- 172.21.15.28:0/4256233402 learned_addr learned my addr 172.21.15.28:0/4256233402 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe9a00640 1 --2- 172.21.15.28:0/4256233402 >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 0x7fcfe406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 -- 172.21.15.28:0/4256233402 >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 msgr2=0x7fcfe406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 --2- 172.21.15.28:0/4256233402 >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 0x7fcfe406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 -- 172.21.15.28:0/4256233402 >> v1:172.21.15.189:6789/0 conn(0x7fcfe41592a0 legacy=0x7fcfe4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe3e00640 1 -- 172.21.15.28:0/4256233402 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe40b2040 con 0x7fcfe406f970 2024-09-13T17:22:19.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.111+0000 7fcfe9a00640 1 --2- 172.21.15.28:0/4256233402 >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 0x7fcfe406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.112+0000 7fcfe3e00640 1 --2- 172.21.15.28:0/4256233402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406f970 0x7fcfe415ab70 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40b1dd0 tx=0x7fcfd002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b8b3f9737d0ea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.112+0000 7fcfe3400640 1 -- 172.21.15.28:0/4256233402 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0032440 con 0x7fcfe406f970 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.112+0000 7fcfe3400640 1 -- 172.21.15.28:0/4256233402 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd00325a0 con 0x7fcfe406f970 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.112+0000 7fcfe3400640 1 -- 172.21.15.28:0/4256233402 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd0032930 con 0x7fcfe406f970 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 -- 172.21.15.28:0/4256233402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406f970 msgr2=0x7fcfe415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 --2- 172.21.15.28:0/4256233402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406f970 0x7fcfe415ab70 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7fcfe40b1dd0 tx=0x7fcfd002a270 comp rx=0 tx=0).stop 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 -- 172.21.15.28:0/4256233402 shutdown_connections 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 --2- 172.21.15.28:0/4256233402 >> v2:172.21.15.189:3300/0 conn(0x7fcfe406c2d0 0x7fcfe406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 --2- 172.21.15.28:0/4256233402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406f970 0x7fcfe415ab70 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 -- 172.21.15.28:0/4256233402 >> 172.21.15.28:0/4256233402 conn(0x7fcfe4069e80 msgr2=0x7fcfe406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 -- 172.21.15.28:0/4256233402 shutdown_connections 2024-09-13T17:22:19.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.113+0000 7fcfeba00640 1 -- 172.21.15.28:0/4256233402 wait complete. 2024-09-13T17:22:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.115+0000 7fcfeba00640 1 Processor -- start 2024-09-13T17:22:19.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.115+0000 7fcfeba00640 1 -- start start 2024-09-13T17:22:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfe406f970 0x7fcfe413c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe41592a0 0x7fcfe406b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe41502b0 con 0x7fcfe406c2d0 2024-09-13T17:22:19.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe40ce220 con 0x7fcfe41592a0 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfeba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe413c990 con 0x7fcfe406f970 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfe9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfe9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52294/0 (socket says 172.21.15.28:52294) 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfe9000640 1 -- 172.21.15.28:0/22827329 learned_addr learned my addr 172.21.15.28:0/22827329 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfe3e00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfe406f970 0x7fcfe413c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.116+0000 7fcfe9a00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe41592a0 0x7fcfe406b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 -- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfe406f970 msgr2=0x7fcfe413c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfe406f970 0x7fcfe413c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 -- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe41592a0 msgr2=0x7fcfe406b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe41592a0 0x7fcfe406b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 -- 172.21.15.28:0/22827329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd00094d0 con 0x7fcfe406c2d0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe9000640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800c400 tx=0x7fcfd800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80052d0 con 0x7fcfe406c2d0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd8005430 con 0x7fcfe406c2d0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe40ce450 con 0x7fcfe406c2d0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8003830 con 0x7fcfe406c2d0 2024-09-13T17:22:19.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.117+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe40cea10 con 0x7fcfe406c2d0 2024-09-13T17:22:19.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.118+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe413cb00 con 0x7fcfe406c2d0 2024-09-13T17:22:19.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.119+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcfd8003990 con 0x7fcfe406c2d0 2024-09-13T17:22:19.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.120+0000 7fcfe0c00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 0x7fcfc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.120+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcfd80888c0 con 0x7fcfe406c2d0 2024-09-13T17:22:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.121+0000 7fcfe3e00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 0x7fcfc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.122+0000 7fcfe3e00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 0x7fcfc406d0c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0034000 tx=0x7fcfd002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.125+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcfd8053ab0 con 0x7fcfe406c2d0 2024-09-13T17:22:19.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.308+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfe40b3740 con 0x7fcfe406c2d0 2024-09-13T17:22:19.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.309+0000 7fcfe0c00640 1 -- 172.21.15.28:0/22827329 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16583 (secure 0 0 0) 0x7fcfd8017090 con 0x7fcfe406c2d0 2024-09-13T17:22:19.311 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:19.311 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:37.723750+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 msgr2=0x7fcfc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 0x7fcfc406d0c0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fcfd0034000 tx=0x7fcfd002e000 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 msgr2=0x7fcfe40cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800c400 tx=0x7fcfd800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 shutdown_connections 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcfc406ac00 0x7fcfc406d0c0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcfe41592a0 0x7fcfe406b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcfe406f970 0x7fcfe413c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 --2- 172.21.15.28:0/22827329 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcfe406c2d0 0x7fcfe40cdce0 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.311+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 >> 172.21.15.28:0/22827329 conn(0x7fcfe4069e80 msgr2=0x7fcfe415c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.312+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 shutdown_connections 2024-09-13T17:22:19.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.312+0000 7fcfeba00640 1 -- 172.21.15.28:0/22827329 wait complete. 2024-09-13T17:22:19.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:21:34.559482+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:21:35.719796+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '193', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:22:19.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:22:19.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.488+0000 7f0cc5c00640 1 Processor -- start 2024-09-13T17:22:19.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.488+0000 7f0cc5c00640 1 -- start start 2024-09-13T17:22:19.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cc5c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cc0070430 0x7f0cc015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cc5c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 0x7f0cc015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cc5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc015ab60 con 0x7f0cc0159770 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cc5c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc006f920 con 0x7f0cc0070430 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cc5c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc015b290 con 0x7f0cc006f3f0 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cc0070430 0x7f0cc015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cb7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0cc0070430 0x7f0cc015bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52306/0 (socket says 172.21.15.28:52306) 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.489+0000 7f0cb7e00640 1 -- 172.21.15.28:0/2492832732 learned_addr learned my addr 172.21.15.28:0/2492832732 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cbf400640 1 --2- 172.21.15.28:0/2492832732 >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 0x7f0cc015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cb7e00640 1 -- 172.21.15.28:0/2492832732 >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 msgr2=0x7f0cc015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cb7e00640 1 --2- 172.21.15.28:0/2492832732 >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 0x7f0cc015c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cb7e00640 1 -- 172.21.15.28:0/2492832732 >> v1:172.21.15.28:6789/0 conn(0x7f0cc0159770 legacy=0x7f0cc006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cb7e00640 1 -- 172.21.15.28:0/2492832732 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc015b570 con 0x7f0cc0070430 2024-09-13T17:22:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cbe000640 1 -- 172.21.15.28:0/2492832732 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569697544 0 0) 0x7f0cb0001110 con 0x7f0cc0159770 2024-09-13T17:22:19.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.490+0000 7f0cbf400640 1 --2- 172.21.15.28:0/2492832732 >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 0x7f0cc015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.491+0000 7f0cb7e00640 1 --2- 172.21.15.28:0/2492832732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0070430 0x7f0cc015bd70 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f0cc006f380 tx=0x7f0ca802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d54653daa5157f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.491+0000 7f0cbe000640 1 -- 172.21.15.28:0/2492832732 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8032440 con 0x7f0cc0070430 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.491+0000 7f0cbe000640 1 -- 172.21.15.28:0/2492832732 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca80325a0 con 0x7f0cc0070430 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.491+0000 7f0cbe000640 1 -- 172.21.15.28:0/2492832732 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca8032930 con 0x7f0cc0070430 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2492832732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0070430 msgr2=0x7f0cc015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2492832732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0070430 0x7f0cc015bd70 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f0cc006f380 tx=0x7f0ca802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2492832732 shutdown_connections 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2492832732 >> v2:172.21.15.189:3300/0 conn(0x7f0cc006f3f0 0x7f0cc015c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2492832732 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc0070430 0x7f0cc015bd70 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2492832732 >> 172.21.15.28:0/2492832732 conn(0x7f0cc00b0270 msgr2=0x7f0cc00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2492832732 shutdown_connections 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.492+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2492832732 wait complete. 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.493+0000 7f0cc5c00640 1 Processor -- start 2024-09-13T17:22:19.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.494+0000 7f0cc5c00640 1 -- start start 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.494+0000 7f0cc5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cc5c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc0070430 0x7f0cc014eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cc5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 0x7f0cc00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cc5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc014fff0 con 0x7f0cc006f3f0 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cc5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0145490 con 0x7f0cc0159770 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cc5c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc014f3e0 con 0x7f0cc0070430 2024-09-13T17:22:19.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52318/0 (socket says 172.21.15.28:52318) 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 -- 172.21.15.28:0/2330302580 learned_addr learned my addr 172.21.15.28:0/2330302580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cb7e00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc0070430 0x7f0cc014eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbf400640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 0x7f0cc00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 -- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc0070430 msgr2=0x7f0cc014eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc0070430 0x7f0cc014eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 -- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 msgr2=0x7f0cc00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 0x7f0cc00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbea00640 1 -- 172.21.15.28:0/2330302580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca80094d0 con 0x7f0cc006f3f0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.495+0000 7f0cbf400640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 0x7f0cc00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cbea00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000c410 tx=0x7f0cb000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00035f0 con 0x7f0cc006f3f0 2024-09-13T17:22:19.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb0003770 con 0x7f0cc006f3f0 2024-09-13T17:22:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0003b20 con 0x7f0cc006f3f0 2024-09-13T17:22:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc014f6c0 con 0x7f0cc006f3f0 2024-09-13T17:22:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.496+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc014fc80 con 0x7f0cc006f3f0 2024-09-13T17:22:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.497+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc006fd20 con 0x7f0cc006f3f0 2024-09-13T17:22:19.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.498+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0cb0003c80 con 0x7f0cc006f3f0 2024-09-13T17:22:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.499+0000 7f0cb6a00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 0x7f0c9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.499+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0cb0088780 con 0x7f0cc006f3f0 2024-09-13T17:22:19.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.500+0000 7f0cb7e00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 0x7f0c9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.501+0000 7f0cb7e00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 0x7f0c9c06d0c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8034000 tx=0x7f0ca802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.504+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0cb0053a10 con 0x7f0cc006f3f0 2024-09-13T17:22:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.680+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0cc015ab60 con 0x7f0cc006f3f0 2024-09-13T17:22:19.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.680+0000 7f0cb6a00640 1 -- 172.21.15.28:0/2330302580 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v193) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0cb00170a0 con 0x7f0cc006f3f0 2024-09-13T17:22:19.682 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 msgr2=0x7f0c9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 0x7f0c9c06d0c0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8034000 tx=0x7f0ca802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 msgr2=0x7f0cc0144f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000c410 tx=0x7f0cb000c8d0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 shutdown_connections 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0c9c06ac00 0x7f0c9c06d0c0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0cc0159770 0x7f0cc00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0cc0070430 0x7f0cc014eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 --2- 172.21.15.28:0/2330302580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0cc006f3f0 0x7f0cc0144f50 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 >> 172.21.15.28:0/2330302580 conn(0x7f0cc00b0270 msgr2=0x7f0cc00af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 shutdown_connections 2024-09-13T17:22:19.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.682+0000 7f0cc5c00640 1 -- 172.21.15.28:0/2330302580 wait complete. 2024-09-13T17:22:19.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:22:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.856+0000 7f3bf8e00640 1 Processor -- start 2024-09-13T17:22:19.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.856+0000 7f3bf8e00640 1 -- start start 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf8e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3bf415a780 0x7f3bf4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf8e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf406c660 con 0x7f3bf415ab60 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf8e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bf406c8b0 con 0x7f3bf406c250 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf8e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bf406cb90 con 0x7f3bf415a780 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3bf415ab60 0x7f3bf406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59844/0 (socket says 172.21.15.28:59844) 2024-09-13T17:22:19.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bebe00640 1 -- 172.21.15.28:0/2254319297 learned_addr learned my addr 172.21.15.28:0/2254319297 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/2254319297 >> v2:172.21.15.189:3300/0 conn(0x7f3bf415a780 0x7f3bf4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420904245 0 0) 0x7f3bf406c660 con 0x7f3bf415ab60 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.858+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd8003520 con 0x7f3bf415ab60 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71489743 0 0) 0x7f3bd8003520 con 0x7f3bf415ab60 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 >> v2:172.21.15.189:3300/0 conn(0x7f3bf415a780 msgr2=0x7f3bf4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 --2- 172.21.15.28:0/2254319297 >> v2:172.21.15.189:3300/0 conn(0x7f3bf415a780 0x7f3bf4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 >> v1:172.21.15.189:6789/0 conn(0x7f3bf406c250 legacy=0x7f3bf40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf40cc980 con 0x7f3bf415ab60 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3bdc001670 con 0x7f3bf415ab60 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 >> v1:172.21.15.28:6789/0 conn(0x7f3bf415ab60 legacy=0x7f3bf406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 0x7f3bd8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80047c0 0x7f3bd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd8005180 0x7f3bd800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf406cb90 con 0x7f3bd80047c0 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd800c080 con 0x7f3bd80039c0 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd800c240 con 0x7f3bd8005180 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bebe00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80047c0 0x7f3bd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 0x7f3bd8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.859+0000 7f3bf2800640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd8005180 0x7f3bd800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf1e00640 1 -- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd8005180 msgr2=0x7f3bd800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bd8005180 0x7f3bd800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf1e00640 1 -- 172.21.15.28:0/2254319297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80047c0 msgr2=0x7f3bd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80047c0 0x7f3bd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf1e00640 1 -- 172.21.15.28:0/2254319297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd8003520 con 0x7f3bd80039c0 2024-09-13T17:22:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bebe00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80047c0 0x7f3bd8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:19.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.860+0000 7f3bf2800640 1 -- 172.21.15.28:0/2254319297 reap_dead start 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.861+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 0x7f3bd8003de0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009ac0 tx=0x7f3bd80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adeb1bd31535c394 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.861+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4031400 con 0x7f3bd80039c0 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.861+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be402d050 con 0x7f3bd80039c0 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.861+0000 7f3bf1400640 1 -- 172.21.15.28:0/2254319297 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4031780 con 0x7f3bd80039c0 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 -- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 msgr2=0x7f3bd8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 0x7f3bd8003de0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009ac0 tx=0x7f3bd80057f0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 -- 172.21.15.28:0/2254319297 shutdown_connections 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/2254319297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd80039c0 0x7f3bd8003de0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 -- 172.21.15.28:0/2254319297 >> 172.21.15.28:0/2254319297 conn(0x7f3bf40b0270 msgr2=0x7f3bf40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 -- 172.21.15.28:0/2254319297 shutdown_connections 2024-09-13T17:22:19.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.862+0000 7f3bf8e00640 1 -- 172.21.15.28:0/2254319297 wait complete. 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.863+0000 7f3bf8e00640 1 Processor -- start 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.864+0000 7f3bf8e00640 1 -- start start 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.864+0000 7f3bf8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bf41ef910 0x7f3bf415d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf8e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bf415dcd0 0x7f3bf40ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf8e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf41f1700 con 0x7f3bd80039c0 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf8e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf415af70 con 0x7f3bf41ef910 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf8e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bf415b250 con 0x7f3bf415dcd0 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52336/0 (socket says 172.21.15.28:52336) 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 -- 172.21.15.28:0/3163994683 learned_addr learned my addr 172.21.15.28:0/3163994683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf2800640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bf415dcd0 0x7f3bf40ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bebe00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bf41ef910 0x7f3bf415d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 -- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bf415dcd0 msgr2=0x7f3bf40ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bf415dcd0 0x7f3bf40ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 -- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bf41ef910 msgr2=0x7f3bf415d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bf41ef910 0x7f3bf415d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.865+0000 7f3bf1e00640 1 -- 172.21.15.28:0/3163994683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf415b530 con 0x7f3bd80039c0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bf1e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f3be402aae0 tx=0x7f3be4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4003e90 con 0x7f3bd80039c0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be403b400 con 0x7f3bd80039c0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be403b790 con 0x7f3bd80039c0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bf415b810 con 0x7f3bd80039c0 2024-09-13T17:22:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.866+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bf41efd90 con 0x7f3bd80039c0 2024-09-13T17:22:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.867+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf406bed0 con 0x7f3bd80039c0 2024-09-13T17:22:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.868+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3be403b8f0 con 0x7f3bd80039c0 2024-09-13T17:22:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.869+0000 7f3bea000640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 0x7f3bd006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.869+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3be40a99a0 con 0x7f3bd80039c0 2024-09-13T17:22:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.870+0000 7f3bebe00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 0x7f3bd006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.870+0000 7f3bebe00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 0x7f3bd006d0f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4154c80 tx=0x7f3bdc007bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:19.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:19.874+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3be4074b90 con 0x7f3bd80039c0 2024-09-13T17:22:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.050+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3bf41f1700 con 0x7f3bd80039c0 2024-09-13T17:22:20.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.050+0000 7f3bea000640 1 -- 172.21.15.28:0/3163994683 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v193) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3be4048020 con 0x7f3bd80039c0 2024-09-13T17:22:20.052 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:22:20.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 msgr2=0x7f3bd006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 0x7f3bd006d0f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3bf4154c80 tx=0x7f3bdc007bb0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 msgr2=0x7f3bf41f11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7f3be402aae0 tx=0x7f3be4004730 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 shutdown_connections 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3bd006ac30 0x7f3bd006d0f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bf415dcd0 0x7f3bf40ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bf41ef910 0x7f3bf415d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 --2- 172.21.15.28:0/3163994683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bd80039c0 0x7f3bf41f11c0 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 >> 172.21.15.28:0/3163994683 conn(0x7f3bf40b0270 msgr2=0x7f3bf40ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 shutdown_connections 2024-09-13T17:22:20.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.052+0000 7f3bf8e00640 1 -- 172.21.15.28:0/3163994683 wait complete. 2024-09-13T17:22:20.070 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:22:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.226+0000 7fd737200640 1 Processor -- start 2024-09-13T17:22:20.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.226+0000 7fd737200640 1 -- start start 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd73015cb30 0x7fd7300cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd7300d0480 0x7fd7300cf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 0x7fd730069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300cfc10 con 0x7fd73015cb30 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7300d0bf0 con 0x7fd7300d0480 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd737200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300d0ed0 con 0x7fd7300cf7a0 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd735e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd73015cb30 0x7fd7300cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd735e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd73015cb30 0x7fd7300cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52350/0 (socket says 172.21.15.28:52350) 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.227+0000 7fd735e00640 1 -- 172.21.15.28:0/621693618 learned_addr learned my addr 172.21.15.28:0/621693618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd736800640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 0x7fd730069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735400640 1 --2- 172.21.15.28:0/621693618 >> v2:172.21.15.189:3300/0 conn(0x7fd7300d0480 0x7fd7300cf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735e00640 1 -- 172.21.15.28:0/621693618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 msgr2=0x7fd730069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735e00640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 0x7fd730069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735e00640 1 -- 172.21.15.28:0/621693618 >> v2:172.21.15.189:3300/0 conn(0x7fd7300d0480 msgr2=0x7fd7300cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735e00640 1 --2- 172.21.15.28:0/621693618 >> v2:172.21.15.189:3300/0 conn(0x7fd7300d0480 0x7fd7300cf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd735e00640 1 -- 172.21.15.28:0/621693618 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7300d11b0 con 0x7fd73015cb30 2024-09-13T17:22:20.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.228+0000 7fd736800640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 0x7fd730069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.229+0000 7fd735e00640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd73015cb30 0x7fd7300cdc00 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fd7300d2880 tx=0x7fd72402a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241588b7733013db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.229+0000 7fd734a00640 1 -- 172.21.15.28:0/621693618 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724032440 con 0x7fd73015cb30 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.229+0000 7fd734a00640 1 -- 172.21.15.28:0/621693618 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7240325a0 con 0x7fd73015cb30 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.229+0000 7fd734a00640 1 -- 172.21.15.28:0/621693618 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd724032930 con 0x7fd73015cb30 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 -- 172.21.15.28:0/621693618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd73015cb30 msgr2=0x7fd7300cdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd73015cb30 0x7fd7300cdc00 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fd7300d2880 tx=0x7fd72402a1d0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 -- 172.21.15.28:0/621693618 shutdown_connections 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd7300cf7a0 0x7fd730069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 --2- 172.21.15.28:0/621693618 >> v2:172.21.15.189:3300/0 conn(0x7fd7300d0480 0x7fd7300cf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 --2- 172.21.15.28:0/621693618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd73015cb30 0x7fd7300cdc00 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 -- 172.21.15.28:0/621693618 >> 172.21.15.28:0/621693618 conn(0x7fd730070ad0 msgr2=0x7fd7300b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 -- 172.21.15.28:0/621693618 shutdown_connections 2024-09-13T17:22:20.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.230+0000 7fd737200640 1 -- 172.21.15.28:0/621693618 wait complete. 2024-09-13T17:22:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.231+0000 7fd737200640 1 Processor -- start 2024-09-13T17:22:20.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.231+0000 7fd737200640 1 -- start start 2024-09-13T17:22:20.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7300cf7a0 0x7fd7300cb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd73015cb30 0x7fd73015ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300cb2d0 con 0x7fd7300d0480 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300ca6c0 con 0x7fd7300cf7a0 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.232+0000 7fd737200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7300ca9a0 con 0x7fd73015cb30 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52366/0 (socket says 172.21.15.28:52366) 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 -- 172.21.15.28:0/4250799340 learned_addr learned my addr 172.21.15.28:0/4250799340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd736800640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd73015cb30 0x7fd73015ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735e00640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7300cf7a0 0x7fd7300cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 -- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd73015cb30 msgr2=0x7fd73015ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd73015cb30 0x7fd73015ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 -- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7300cf7a0 msgr2=0x7fd7300cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7300cf7a0 0x7fd7300cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 -- 172.21.15.28:0/4250799340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7240094d0 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.233+0000 7fd735400640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d800 tx=0x7fd72000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.234+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000b6c0 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.234+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72000b820 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.234+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72000bbb0 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.234+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7300cac80 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.234+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7300ca340 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.235+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7300cb440 con 0x7fd7300d0480 2024-09-13T17:22:20.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.236+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd72000bd10 con 0x7fd7300d0480 2024-09-13T17:22:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.237+0000 7fd72a000640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 0x7fd71406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.237+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd720089830 con 0x7fd7300d0480 2024-09-13T17:22:20.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.238+0000 7fd735e00640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 0x7fd71406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.238+0000 7fd735e00640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 0x7fd71406d0c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd724034000 tx=0x7fd72402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.241+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd720054a20 con 0x7fd7300d0480 2024-09-13T17:22:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.420+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd730069e80 con 0x7fd7300d0480 2024-09-13T17:22:20.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.420+0000 7fd72a000640 1 -- 172.21.15.28:0/4250799340 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v193) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd720058040 con 0x7fd7300d0480 2024-09-13T17:22:20.422 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:22:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 msgr2=0x7fd71406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 0x7fd71406d0c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fd724034000 tx=0x7fd72402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 msgr2=0x7fd730159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 secure :-1 s=READY pgs=2387 cs=0 l=1 rev1=1 crypto rx=0x7fd72000d800 tx=0x7fd72000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 shutdown_connections 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd71406ac00 0x7fd71406d0c0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd73015cb30 0x7fd73015ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7300d0480 0x7fd730159140 unknown :-1 s=CLOSED pgs=2387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 --2- 172.21.15.28:0/4250799340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd7300cf7a0 0x7fd7300cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 >> 172.21.15.28:0/4250799340 conn(0x7fd730070ad0 msgr2=0x7fd7300b6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 shutdown_connections 2024-09-13T17:22:20.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.422+0000 7fd737200640 1 -- 172.21.15.28:0/4250799340 wait complete. 2024-09-13T17:22:20.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:20.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.592+0000 7f45c3e00640 1 Processor -- start 2024-09-13T17:22:20.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.592+0000 7f45c3e00640 1 -- start start 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c3e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc15e100 con 0x7f45bc15a540 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c3e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45bc15e3e0 con 0x7f45bc070cc0 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc0adb00 con 0x7f45bc0cabc0 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c1400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f45bc15a540 0x7f45bc15e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59846/0 (socket says 172.21.15.28:59846) 2024-09-13T17:22:20.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.593+0000 7f45c1400640 1 -- 172.21.15.28:0/998272276 learned_addr learned my addr 172.21.15.28:0/998272276 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:20.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45c0a00640 1 --2- 172.21.15.28:0/998272276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955967943 0 0) 0x7f45bc15e100 con 0x7f45bc15a540 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45a4003520 con 0x7f45bc15a540 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284989691 0 0) 0x7f45bc15e3e0 con 0x7f45bc070cc0 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45bc15e100 con 0x7f45bc070cc0 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45c0a00640 1 -- 172.21.15.28:0/998272276 >> v1:172.21.15.189:6789/0 conn(0x7f45bc070cc0 legacy=0x7f45bc06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2910453453 0 0) 0x7f45a4003520 con 0x7f45bc15a540 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45c0a00640 1 -- 172.21.15.28:0/998272276 >> v1:172.21.15.28:6789/0 conn(0x7f45bc15a540 legacy=0x7f45bc15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.594+0000 7f45c0a00640 1 -- 172.21.15.28:0/998272276 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc15ef80 con 0x7f45bc0cabc0 2024-09-13T17:22:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.595+0000 7f45c0a00640 1 --2- 172.21.15.28:0/998272276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc06c430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f45a80097d0 tx=0x7f45a802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9353002c0f43509 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8032440 con 0x7f45bc0cabc0 2024-09-13T17:22:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45a80325a0 con 0x7f45bc0cabc0 2024-09-13T17:22:20.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45bbe00640 1 -- 172.21.15.28:0/998272276 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a8032930 con 0x7f45bc0cabc0 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 -- 172.21.15.28:0/998272276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 msgr2=0x7f45bc06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 --2- 172.21.15.28:0/998272276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc06c430 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f45a80097d0 tx=0x7f45a802a180 comp rx=0 tx=0).stop 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 -- 172.21.15.28:0/998272276 shutdown_connections 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 --2- 172.21.15.28:0/998272276 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc06c430 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 -- 172.21.15.28:0/998272276 >> 172.21.15.28:0/998272276 conn(0x7f45bc069e80 msgr2=0x7f45bc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 -- 172.21.15.28:0/998272276 shutdown_connections 2024-09-13T17:22:20.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.596+0000 7f45c3e00640 1 -- 172.21.15.28:0/998272276 wait complete. 2024-09-13T17:22:20.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.598+0000 7f45c3e00640 1 Processor -- start 2024-09-13T17:22:20.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.598+0000 7f45c3e00640 1 -- start start 2024-09-13T17:22:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 0x7f45bc14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc152380 con 0x7f45bc070cc0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc151a60 con 0x7f45bc15a540 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c3e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45bc151ce0 con 0x7f45bc0cabc0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52370/0 (socket says 172.21.15.28:52370) 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.599+0000 7f45c1400640 1 -- 172.21.15.28:0/3126976461 learned_addr learned my addr 172.21.15.28:0/3126976461 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c0a00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 0x7f45bc14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 -- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 msgr2=0x7f45bc14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 -- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 msgr2=0x7f45bc14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 0x7f45bc14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 -- 172.21.15.28:0/3126976461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45a8009400 con 0x7f45bc070cc0 2024-09-13T17:22:20.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c0a00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 0x7f45bc14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c1400640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f45b000c480 tx=0x7f45b000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b0005ce0 con 0x7f45bc070cc0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b0005e40 con 0x7f45bc070cc0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b00072b0 con 0x7f45bc070cc0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.600+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45bc143c70 con 0x7f45bc070cc0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.601+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45bc152ce0 con 0x7f45bc070cc0 2024-09-13T17:22:20.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.602+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45bc1524c0 con 0x7f45bc070cc0 2024-09-13T17:22:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.602+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f45b0007410 con 0x7f45bc070cc0 2024-09-13T17:22:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.603+0000 7f45b9600640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 0x7f459c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:20.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.603+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45b0088900 con 0x7f45bc070cc0 2024-09-13T17:22:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.604+0000 7f45c0a00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 0x7f459c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:20.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.605+0000 7f45c0a00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 0x7f459c06d0c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f45a8034000 tx=0x7f45a802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.607+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f45bc1524c0 con 0x7f45bc070cc0 2024-09-13T17:22:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.823+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45bc1524c0 con 0x7f45bc070cc0 2024-09-13T17:22:20.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.824+0000 7f45b9600640 1 -- 172.21.15.28:0/3126976461 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f45bc1524c0 con 0x7f45bc070cc0 2024-09-13T17:22:20.826 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:20.826 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:22:20.826 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:22:20.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 msgr2=0x7f459c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 0x7f459c06d0c0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f45a8034000 tx=0x7f45a802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 msgr2=0x7f45bc1527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f45b000c480 tx=0x7f45b000c940 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 shutdown_connections 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f459c06ac00 0x7f459c06d0c0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45bc15a540 0x7f45bc14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f45bc0cabc0 0x7f45bc14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 --2- 172.21.15.28:0/3126976461 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45bc070cc0 0x7f45bc1527a0 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 >> 172.21.15.28:0/3126976461 conn(0x7f45bc069e80 msgr2=0x7f45bc06f340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.826+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 shutdown_connections 2024-09-13T17:22:20.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.827+0000 7f45c3e00640 1 -- 172.21.15.28:0/3126976461 wait complete. 2024-09-13T17:22:20.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:21.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.999+0000 7f7c31200640 1 Processor -- start 2024-09-13T17:22:21.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:20.999+0000 7f7c31200640 1 -- start start 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c31200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c06f970 0x7f7c2c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c31200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7c2c06c2d0 0x7f7c2c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c31200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c2c159d90 con 0x7f7c2c1592a0 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c31200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2c15b0b0 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c31200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c06cca0 con 0x7f7c2c06f970 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c2a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7c2c06c2d0 0x7f7c2c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c29e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7c2c1592a0 0x7f7c2c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59854/0 (socket says 172.21.15.28:59854) 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.000+0000 7f7c29e00640 1 -- 172.21.15.28:0/3890983850 learned_addr learned my addr 172.21.15.28:0/3890983850 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:21.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c23e00640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c06f970 0x7f7c2c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c2a800640 1 -- 172.21.15.28:0/3890983850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c06f970 msgr2=0x7f7c2c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c2a800640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c06f970 0x7f7c2c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c2a800640 1 -- 172.21.15.28:0/3890983850 >> v1:172.21.15.28:6789/0 conn(0x7f7c2c1592a0 legacy=0x7f7c2c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c29400640 1 -- 172.21.15.28:0/3890983850 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623681181 0 0) 0x7f7c2c159d90 con 0x7f7c2c1592a0 2024-09-13T17:22:21.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.001+0000 7f7c2a800640 1 -- 172.21.15.28:0/3890983850 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2c0b2040 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.002+0000 7f7c2a800640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c06c760 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009d90 tx=0x7f7c2402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de752809415de3ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.002+0000 7f7c29400640 1 -- 172.21.15.28:0/3890983850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24007740 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.002+0000 7f7c29400640 1 -- 172.21.15.28:0/3890983850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c2402d050 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.002+0000 7f7c29400640 1 -- 172.21.15.28:0/3890983850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24007ac0 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 -- 172.21.15.28:0/3890983850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06c2d0 msgr2=0x7f7c2c06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c06c760 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009d90 tx=0x7f7c2402a660 comp rx=0 tx=0).stop 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 -- 172.21.15.28:0/3890983850 shutdown_connections 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c06c760 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 --2- 172.21.15.28:0/3890983850 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c06f970 0x7f7c2c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 -- 172.21.15.28:0/3890983850 >> 172.21.15.28:0/3890983850 conn(0x7f7c2c069e80 msgr2=0x7f7c2c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 -- 172.21.15.28:0/3890983850 shutdown_connections 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.003+0000 7f7c31200640 1 -- 172.21.15.28:0/3890983850 wait complete. 2024-09-13T17:22:21.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.004+0000 7f7c31200640 1 Processor -- start 2024-09-13T17:22:21.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.005+0000 7f7c31200640 1 -- start start 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.005+0000 7f7c31200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c1f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c31200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c31200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 0x7f7c2c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c31200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c13dd00 con 0x7f7c2c06f970 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c31200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c13d080 con 0x7f7c2c06c2d0 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c31200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2c13d360 con 0x7f7c2c1592a0 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52392/0 (socket says 172.21.15.28:52392) 2024-09-13T17:22:21.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 -- 172.21.15.28:0/3318368365 learned_addr learned my addr 172.21.15.28:0/3318368365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:21.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c29e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c1f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c2a800640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 0x7f7c2c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 -- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 msgr2=0x7f7c2c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 0x7f7c2c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 -- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 msgr2=0x7f7c2c1f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c1f4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c23e00640 1 -- 172.21.15.28:0/3318368365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c24009a70 con 0x7f7c2c06f970 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c2a800640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 0x7f7c2c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.006+0000 7f7c29e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c23e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400e9b0 tx=0x7f7c1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c14009670 con 0x7f7c2c06f970 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1400c9a0 con 0x7f7c2c06f970 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1400cd50 con 0x7f7c2c06f970 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2c13d640 con 0x7f7c2c06f970 2024-09-13T17:22:21.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.007+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2c14c7a0 con 0x7f7c2c06f970 2024-09-13T17:22:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.008+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2c0b3740 con 0x7f7c2c06f970 2024-09-13T17:22:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.009+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7c14013470 con 0x7f7c2c06f970 2024-09-13T17:22:21.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.010+0000 7f7c22000640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 0x7f7c0806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.010+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c14089600 con 0x7f7c2c06f970 2024-09-13T17:22:21.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.010+0000 7f7c29e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 0x7f7c0806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.011+0000 7f7c29e00640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 0x7f7c0806d0f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000f60 tx=0x7f7c1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.014+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7c14054740 con 0x7f7c2c06f970 2024-09-13T17:22:21.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.194+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c2c13df70 con 0x7f7c2c06f970 2024-09-13T17:22:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.196+0000 7f7c22000640 1 -- 172.21.15.28:0/3318368365 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16583 (secure 0 0 0) 0x7f7c2c13df70 con 0x7f7c2c06f970 2024-09-13T17:22:21.199 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:21.199 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:37.723750+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:21.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 msgr2=0x7f7c0806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 0x7f7c0806d0f0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c000f60 tx=0x7f7c1c009040 comp rx=0 tx=0).stop 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 msgr2=0x7f7c2c157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 secure :-1 s=READY pgs=2392 cs=0 l=1 rev1=1 crypto rx=0x7f7c1400e9b0 tx=0x7f7c1400ee70 comp rx=0 tx=0).stop 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 shutdown_connections 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c0806ac30 0x7f7c0806d0f0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7c2c1592a0 0x7f7c2c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7c2c06f970 0x7f7c2c157ee0 unknown :-1 s=CLOSED pgs=2392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 --2- 172.21.15.28:0/3318368365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c2c06c2d0 0x7f7c2c1f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 >> 172.21.15.28:0/3318368365 conn(0x7f7c2c069e80 msgr2=0x7f7c2c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 shutdown_connections 2024-09-13T17:22:21.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.198+0000 7f7c31200640 1 -- 172.21.15.28:0/3318368365 wait complete. 2024-09-13T17:22:21.217 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:22:21.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:21.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.373+0000 7fe94f000640 1 Processor -- start 2024-09-13T17:22:21.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.373+0000 7fe94f000640 1 -- start start 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe9480b4930 0x7fe9480d2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480d1190 con 0x7fe9480d0b00 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9480d1470 con 0x7fe94815c870 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9480d0f70 con 0x7fe9480b4930 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94e600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe9480d0b00 0x7fe9480b58d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59864/0 (socket says 172.21.15.28:59864) 2024-09-13T17:22:21.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94e600640 1 -- 172.21.15.28:0/1083215702 learned_addr learned my addr 172.21.15.28:0/1083215702 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe94d200640 1 --2- 172.21.15.28:0/1083215702 >> v2:172.21.15.189:3300/0 conn(0x7fe9480b4930 0x7fe9480d2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.374+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317546345 0 0) 0x7fe9480d1190 con 0x7fe9480d0b00 2024-09-13T17:22:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe930003520 con 0x7fe9480d0b00 2024-09-13T17:22:21.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3039701442 0 0) 0x7fe930003520 con 0x7fe9480d0b00 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 >> v2:172.21.15.189:3300/0 conn(0x7fe9480b4930 msgr2=0x7fe9480d2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 --2- 172.21.15.28:0/1083215702 >> v2:172.21.15.189:3300/0 conn(0x7fe9480b4930 0x7fe9480d2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 >> v1:172.21.15.189:6789/0 conn(0x7fe94815c870 legacy=0x7fe9480b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9480d1750 con 0x7fe9480d0b00 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fe9400019f0 con 0x7fe9480d0b00 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 >> v1:172.21.15.28:6789/0 conn(0x7fe9480d0b00 legacy=0x7fe9480b58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe930003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9300047c0 0x7fe930004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe930005180 0x7fe93000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480d0f70 con 0x7fe9300039c0 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93000c060 con 0x7fe9300047c0 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93000c220 con 0x7fe930005180 2024-09-13T17:22:21.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe94dc00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe930003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.375+0000 7fe94d200640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9300047c0 0x7fe930004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94e600640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe930005180 0x7fe93000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94dc00640 1 -- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe930005180 msgr2=0x7fe93000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94dc00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe930005180 0x7fe93000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94dc00640 1 -- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9300047c0 msgr2=0x7fe930004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94dc00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe9300047c0 0x7fe930004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94dc00640 1 -- 172.21.15.28:0/1083215702 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930003520 con 0x7fe9300039c0 2024-09-13T17:22:21.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.376+0000 7fe94e600640 1 -- 172.21.15.28:0/1083215702 reap_dead start 2024-09-13T17:22:21.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.377+0000 7fe94dc00640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe930003de0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fe93c001750 tx=0x7fe9300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=734317492650d493 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.377+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c0052d0 con 0x7fe9300039c0 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.377+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c005430 con 0x7fe9300039c0 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.377+0000 7fe947e00640 1 -- 172.21.15.28:0/1083215702 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c02abe0 con 0x7fe9300039c0 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 -- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 msgr2=0x7fe930003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe930003de0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7fe93c001750 tx=0x7fe9300057f0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 -- 172.21.15.28:0/1083215702 shutdown_connections 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 --2- 172.21.15.28:0/1083215702 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe930003de0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 -- 172.21.15.28:0/1083215702 >> 172.21.15.28:0/1083215702 conn(0x7fe9480cf400 msgr2=0x7fe9480cf810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 -- 172.21.15.28:0/1083215702 shutdown_connections 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.378+0000 7fe94f000640 1 -- 172.21.15.28:0/1083215702 wait complete. 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.380+0000 7fe94f000640 1 Processor -- start 2024-09-13T17:22:21.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.380+0000 7fe94f000640 1 -- start start 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 0x7fe9481f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 0x7fe94815eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9480cb100 con 0x7fe9300039c0 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94815d310 con 0x7fe94814fd10 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9481f69c0 con 0x7fe9480c8db0 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52408/0 (socket says 172.21.15.28:52408) 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 -- 172.21.15.28:0/2635374417 learned_addr learned my addr 172.21.15.28:0/2635374417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94d200640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 0x7fe9481f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94e600640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 0x7fe94815eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 -- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 msgr2=0x7fe9481f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 0x7fe9481f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 -- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 msgr2=0x7fe94815eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 0x7fe94815eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94dc00640 1 -- 172.21.15.28:0/2635374417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94815e910 con 0x7fe9300039c0 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94d200640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 0x7fe9481f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.381+0000 7fe94e600640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 0x7fe94815eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe94dc00640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fe93c02a4c0 tx=0x7fe93c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c03f070 con 0x7fe9300039c0 2024-09-13T17:22:21.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe93c007a30 con 0x7fe9300039c0 2024-09-13T17:22:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe93c007670 con 0x7fe9300039c0 2024-09-13T17:22:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe94815ebd0 con 0x7fe9300039c0 2024-09-13T17:22:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.382+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9481501c0 con 0x7fe9300039c0 2024-09-13T17:22:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.383+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9480b8210 con 0x7fe9300039c0 2024-09-13T17:22:21.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.384+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe93c0067d0 con 0x7fe9300039c0 2024-09-13T17:22:21.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.386+0000 7fe945600640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 0x7fe92806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.386+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe93c0a8b90 con 0x7fe9300039c0 2024-09-13T17:22:21.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.386+0000 7fe94d200640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 0x7fe92806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.387+0000 7fe94d200640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 0x7fe92806d0c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe934006d40 tx=0x7fe934008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.390+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe93c0777f0 con 0x7fe9300039c0 2024-09-13T17:22:21.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.614+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9480b8210 con 0x7fe9300039c0 2024-09-13T17:22:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.615+0000 7fe945600640 1 -- 172.21.15.28:0/2635374417 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fe93c077380 con 0x7fe9300039c0 2024-09-13T17:22:21.617 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:21.617 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:22:21.617 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:22:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 msgr2=0x7fe92806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 0x7fe92806d0c0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fe934006d40 tx=0x7fe934008040 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 msgr2=0x7fe94815cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 secure :-1 s=READY pgs=2395 cs=0 l=1 rev1=1 crypto rx=0x7fe93c02a4c0 tx=0x7fe93c001120 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 shutdown_connections 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe92806ac00 0x7fe92806d0c0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe94814fd10 0x7fe94815eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe9480c8db0 0x7fe9481f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 --2- 172.21.15.28:0/2635374417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe9300039c0 0x7fe94815cdd0 unknown :-1 s=CLOSED pgs=2395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 >> 172.21.15.28:0/2635374417 conn(0x7fe9480cf400 msgr2=0x7fe9480704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.617+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 shutdown_connections 2024-09-13T17:22:21.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.618+0000 7fe94f000640 1 -- 172.21.15.28:0/2635374417 wait complete. 2024-09-13T17:22:21.635 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.788+0000 7f89ae200640 1 Processor -- start 2024-09-13T17:22:21.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.788+0000 7f89ae200640 1 -- start start 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89ae200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89a806f970 0x7f89a815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89ae200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f89a806c2d0 0x7f89a806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89ae200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a806cc70 con 0x7f89a81592a0 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89ae200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a815b0b0 con 0x7f89a806f970 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89ae200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a8159d90 con 0x7f89a806c2d0 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.789+0000 7f89a6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89a806f970 0x7f89a815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f89a806f970 0x7f89a815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52424/0 (socket says 172.21.15.28:52424) 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 -- 172.21.15.28:0/826577975 learned_addr learned my addr 172.21.15.28:0/826577975 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a7e00640 1 --2- 172.21.15.28:0/826577975 >> v2:172.21.15.189:3300/0 conn(0x7f89a806c2d0 0x7f89a806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 -- 172.21.15.28:0/826577975 >> v2:172.21.15.189:3300/0 conn(0x7f89a806c2d0 msgr2=0x7f89a806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:21.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 --2- 172.21.15.28:0/826577975 >> v2:172.21.15.189:3300/0 conn(0x7f89a806c2d0 0x7f89a806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 -- 172.21.15.28:0/826577975 >> v1:172.21.15.189:6789/0 conn(0x7f89a81592a0 legacy=0x7f89a8159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.790+0000 7f89a6a00640 1 -- 172.21.15.28:0/826577975 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a80b2040 con 0x7f89a806f970 2024-09-13T17:22:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.791+0000 7f89a6a00640 1 --2- 172.21.15.28:0/826577975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806f970 0x7f89a815ab70 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b1dd0 tx=0x7f899402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b98845d85d24c595 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.791+0000 7f89a6000640 1 -- 172.21.15.28:0/826577975 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994032440 con 0x7f89a806f970 2024-09-13T17:22:21.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.791+0000 7f89a6000640 1 -- 172.21.15.28:0/826577975 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89940325a0 con 0x7f89a806f970 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.791+0000 7f89a6000640 1 -- 172.21.15.28:0/826577975 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994032930 con 0x7f89a806f970 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 -- 172.21.15.28:0/826577975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806f970 msgr2=0x7f89a815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 --2- 172.21.15.28:0/826577975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806f970 0x7f89a815ab70 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f89a80b1dd0 tx=0x7f899402a270 comp rx=0 tx=0).stop 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 -- 172.21.15.28:0/826577975 shutdown_connections 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 --2- 172.21.15.28:0/826577975 >> v2:172.21.15.189:3300/0 conn(0x7f89a806c2d0 0x7f89a806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 --2- 172.21.15.28:0/826577975 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a806f970 0x7f89a815ab70 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 -- 172.21.15.28:0/826577975 >> 172.21.15.28:0/826577975 conn(0x7f89a8069e80 msgr2=0x7f89a806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:21.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 -- 172.21.15.28:0/826577975 shutdown_connections 2024-09-13T17:22:21.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.792+0000 7f89ae200640 1 -- 172.21.15.28:0/826577975 wait complete. 2024-09-13T17:22:21.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.794+0000 7f89ae200640 1 Processor -- start 2024-09-13T17:22:21.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.794+0000 7f89ae200640 1 -- start start 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.794+0000 7f89ae200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a806c2d0 0x7f89a8144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.794+0000 7f89ae200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a806f970 0x7f89a81453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89ae200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89ae200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a80cb8d0 con 0x7f89a81592a0 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89ae200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a81527e0 con 0x7f89a806c2d0 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89ae200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a81441b0 con 0x7f89a806f970 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52440/0 (socket says 172.21.15.28:52440) 2024-09-13T17:22:21.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 -- 172.21.15.28:0/1209333724 learned_addr learned my addr 172.21.15.28:0/1209333724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a6a00640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a806f970 0x7f89a81453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7400640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a806c2d0 0x7f89a8144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 -- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a806f970 msgr2=0x7f89a81453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a806f970 0x7f89a81453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 -- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a806c2d0 msgr2=0x7f89a8144e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a806c2d0 0x7f89a8144e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:21.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.795+0000 7f89a7e00640 1 -- 172.21.15.28:0/1209333724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89940094d0 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f89a7e00640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007e60 tx=0x7f89a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0014420 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a00145a0 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0014950 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a81435a0 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.796+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a8143b60 con 0x7f89a81592a0 2024-09-13T17:22:21.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.797+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a8149af0 con 0x7f89a81592a0 2024-09-13T17:22:21.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.798+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f89a0014ab0 con 0x7f89a81592a0 2024-09-13T17:22:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.799+0000 7f899f400640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 0x7f898806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.799+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89a0089bb0 con 0x7f89a81592a0 2024-09-13T17:22:21.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.799+0000 7f89a7400640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 0x7f898806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:21.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.800+0000 7f89a7400640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 0x7f898806d0c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f89a80cf610 tx=0x7f8998009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:21.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:21.803+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f89a8149af0 con 0x7f89a81592a0 2024-09-13T17:22:22.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.017+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89a8149af0 con 0x7f89a81592a0 2024-09-13T17:22:22.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.017+0000 7f899f400640 1 -- 172.21.15.28:0/1209333724 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f89a8149af0 con 0x7f89a81592a0 2024-09-13T17:22:22.019 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:22.019 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":165,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:21:36.721122+0000","modified":"2024-09-13T17:21:39.737482+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11049":{"gid":11049,"name":"c","rank":0,"incarnation":161,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/762464937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":762464937},{"type":"v1","addr":"172.21.15.28:6835","nonce":762464937}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2024-09-13T17:22:22.020 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 165 2024-09-13T17:22:22.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 msgr2=0x7f898806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 0x7f898806d0c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f89a80cf610 tx=0x7f8998009040 comp rx=0 tx=0).stop 2024-09-13T17:22:22.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 msgr2=0x7f89a8142780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f89a0007e60 tx=0x7f89a000e4b0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 shutdown_connections 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f898806ac00 0x7f898806d0c0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f89a81592a0 0x7f89a8142780 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f89a806f970 0x7f89a81453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 --2- 172.21.15.28:0/1209333724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89a806c2d0 0x7f89a8144e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 >> 172.21.15.28:0/1209333724 conn(0x7f89a8069e80 msgr2=0x7f89a806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 shutdown_connections 2024-09-13T17:22:22.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.019+0000 7f89ae200640 1 -- 172.21.15.28:0/1209333724 wait complete. 2024-09-13T17:22:22.037 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:22.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.190+0000 7fc728400640 1 Processor -- start 2024-09-13T17:22:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.190+0000 7fc728400640 1 -- start start 2024-09-13T17:22:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.191+0000 7fc728400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc720146170 0x7fc72015d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.191+0000 7fc728400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc720160310 0x7fc720160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc728400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc720146550 con 0x7fc7200ac850 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc728400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc72015cd20 con 0x7fc720146170 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc728400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72015d280 con 0x7fc720160310 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc725a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc720146170 0x7fc72015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc725000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc720160310 0x7fc720160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc725a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc720146170 0x7fc72015d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:47764/0 (socket says 172.21.15.28:47764) 2024-09-13T17:22:22.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.192+0000 7fc725a00640 1 -- 172.21.15.28:0/2591961892 learned_addr learned my addr 172.21.15.28:0/2591961892 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.193+0000 7fc725a00640 1 -- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc720160310 msgr2=0x7fc720160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.193+0000 7fc725a00640 1 --2- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc720160310 0x7fc720160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.193+0000 7fc725a00640 1 -- 172.21.15.28:0/2591961892 >> v1:172.21.15.189:6789/0 conn(0x7fc7200ac850 legacy=0x7fc72006b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.193+0000 7fc725a00640 1 -- 172.21.15.28:0/2591961892 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72015dc50 con 0x7fc720146170 2024-09-13T17:22:22.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc725a00640 1 --2- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc72015d710 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc714009870 tx=0x7fc71402a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f391614fcceebd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc71fe00640 1 -- 172.21.15.28:0/2591961892 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714032440 con 0x7fc720146170 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc71fe00640 1 -- 172.21.15.28:0/2591961892 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7140325a0 con 0x7fc720146170 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc71fe00640 1 -- 172.21.15.28:0/2591961892 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714032930 con 0x7fc720146170 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc728400640 1 -- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 msgr2=0x7fc72015d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.194+0000 7fc728400640 1 --2- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc72015d710 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fc714009870 tx=0x7fc71402a250 comp rx=0 tx=0).stop 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 -- 172.21.15.28:0/2591961892 shutdown_connections 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 --2- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc720160310 0x7fc720160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 --2- 172.21.15.28:0/2591961892 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc72015d710 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 -- 172.21.15.28:0/2591961892 >> 172.21.15.28:0/2591961892 conn(0x7fc720069e80 msgr2=0x7fc72006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 -- 172.21.15.28:0/2591961892 shutdown_connections 2024-09-13T17:22:22.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.195+0000 7fc728400640 1 -- 172.21.15.28:0/2591961892 wait complete. 2024-09-13T17:22:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.196+0000 7fc728400640 1 Processor -- start 2024-09-13T17:22:22.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.196+0000 7fc728400640 1 -- start start 2024-09-13T17:22:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7200ac850 0x7fc72013f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc720131ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72012f090 con 0x7fc720160310 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc720132020 con 0x7fc720146170 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.197+0000 7fc728400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc720154130 con 0x7fc7200ac850 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc726400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc726400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52444/0 (socket says 172.21.15.28:52444) 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc726400640 1 -- 172.21.15.28:0/2074157173 learned_addr learned my addr 172.21.15.28:0/2074157173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725a00640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7200ac850 0x7fc72013f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc720131ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 -- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7200ac850 msgr2=0x7fc72013f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7200ac850 0x7fc72013f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 -- 172.21.15.28:0/2074157173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 msgr2=0x7fc72006baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 -- 172.21.15.28:0/2074157173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140094d0 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc726400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.198+0000 7fc725000640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc720131ae0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e820 tx=0x7fc70c00ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.199+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c00c940 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.199+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70c00caa0 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.199+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c00ce30 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.199+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc720154410 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.199+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72012f550 con 0x7fc720146170 2024-09-13T17:22:22.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.200+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7201545d0 con 0x7fc720146170 2024-09-13T17:22:22.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.201+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc70c013470 con 0x7fc720146170 2024-09-13T17:22:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.202+0000 7fc71d600640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 0x7fc70006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.202+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc70c089530 con 0x7fc720146170 2024-09-13T17:22:22.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.202+0000 7fc725a00640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 0x7fc70006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.203+0000 7fc725a00640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 0x7fc70006d0c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc714034000 tx=0x7fc71402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.207+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc70c054720 con 0x7fc720146170 2024-09-13T17:22:22.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.391+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc72015d6f0 con 0x7fc720146170 2024-09-13T17:22:22.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.392+0000 7fc71d600640 1 -- 172.21.15.28:0/2074157173 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v193) v1 ==== 74+0+16583 (secure 0 0 0) 0x7fc70c057d40 con 0x7fc720146170 2024-09-13T17:22:22.394 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:22.395 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":193,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:21:37.723750+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:21:34.559482+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2024-09-13T17:21:35.719796+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"193","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 msgr2=0x7fc70006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 0x7fc70006d0c0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc714034000 tx=0x7fc71402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 msgr2=0x7fc720131ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc720131ae0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fc70c00e820 tx=0x7fc70c00ece0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 shutdown_connections 2024-09-13T17:22:22.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc70006ac00 0x7fc70006d0c0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc720160310 0x7fc72006baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc720146170 0x7fc720131ae0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 --2- 172.21.15.28:0/2074157173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7200ac850 0x7fc72013f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 >> 172.21.15.28:0/2074157173 conn(0x7fc720069e80 msgr2=0x7fc7200aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 shutdown_connections 2024-09-13T17:22:22.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.395+0000 7fc728400640 1 -- 172.21.15.28:0/2074157173 wait complete. 2024-09-13T17:22:22.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:22:22.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.566+0000 7f9b84c00640 1 Processor -- start 2024-09-13T17:22:22.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.566+0000 7f9b84c00640 1 -- start start 2024-09-13T17:22:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b84c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9b80071490 0x7f9b8006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b84c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8006c8b0 con 0x7f9b8015a780 2024-09-13T17:22:22.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b84c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b8006cb90 con 0x7f9b8015ab60 2024-09-13T17:22:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b84c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b800cc980 con 0x7f9b80071490 2024-09-13T17:22:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b7de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9b8015a780 0x7f9b80070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59880/0 (socket says 172.21.15.28:59880) 2024-09-13T17:22:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.567+0000 7f9b7de00640 1 -- 172.21.15.28:0/360956172 learned_addr learned my addr 172.21.15.28:0/360956172 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:22.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7e800640 1 --2- 172.21.15.28:0/360956172 >> v2:172.21.15.189:3300/0 conn(0x7f9b80071490 0x7f9b8006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046773034 0 0) 0x7f9b8006c8b0 con 0x7f9b8015a780 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b8015a780 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 639686029 0 0) 0x7f9b64003520 con 0x7f9b8015a780 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7e800640 1 -- 172.21.15.28:0/360956172 >> v1:172.21.15.189:6789/0 conn(0x7f9b8015ab60 legacy=0x7f9b8006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7e800640 1 -- 172.21.15.28:0/360956172 >> v1:172.21.15.28:6789/0 conn(0x7f9b8015a780 legacy=0x7f9b80070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.568+0000 7f9b7e800640 1 -- 172.21.15.28:0/360956172 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b800ccc60 con 0x7f9b80071490 2024-09-13T17:22:22.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.569+0000 7f9b7e800640 1 --2- 172.21.15.28:0/360956172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b80071490 0x7f9b8006c250 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009d90 tx=0x7f9b7802a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29931dca4cc5a855 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70001670 con 0x7f9b80071490 2024-09-13T17:22:22.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b7802d050 con 0x7f9b80071490 2024-09-13T17:22:22.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b7ca00640 1 -- 172.21.15.28:0/360956172 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b78007760 con 0x7f9b80071490 2024-09-13T17:22:22.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 -- 172.21.15.28:0/360956172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b80071490 msgr2=0x7f9b8006c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 --2- 172.21.15.28:0/360956172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b80071490 0x7f9b8006c250 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9b78009d90 tx=0x7f9b7802a680 comp rx=0 tx=0).stop 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 -- 172.21.15.28:0/360956172 shutdown_connections 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 --2- 172.21.15.28:0/360956172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b80071490 0x7f9b8006c250 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 -- 172.21.15.28:0/360956172 >> 172.21.15.28:0/360956172 conn(0x7f9b800b0270 msgr2=0x7f9b800b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 -- 172.21.15.28:0/360956172 shutdown_connections 2024-09-13T17:22:22.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.570+0000 7f9b84c00640 1 -- 172.21.15.28:0/360956172 wait complete. 2024-09-13T17:22:22.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.572+0000 7f9b84c00640 1 Processor -- start 2024-09-13T17:22:22.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.572+0000 7f9b84c00640 1 -- start start 2024-09-13T17:22:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b8015a780 0x7f9b800acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8015ab60 0x7f9b80151110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8015bc70 con 0x7f9b80071490 2024-09-13T17:22:22.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8015b010 con 0x7f9b8015ab60 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b84c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b8015b2d0 con 0x7f9b8015a780 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b7de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52446/0 (socket says 172.21.15.28:52446) 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.573+0000 7f9b7de00640 1 -- 172.21.15.28:0/2429094324 learned_addr learned my addr 172.21.15.28:0/2429094324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7e800640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8015ab60 0x7f9b80151110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7d400640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b8015a780 0x7f9b800acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 -- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b8015a780 msgr2=0x7f9b800acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b8015a780 0x7f9b800acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 -- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8015ab60 msgr2=0x7f9b80151110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8015ab60 0x7f9b80151110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 -- 172.21.15.28:0/2429094324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b78009a70 con 0x7f9b80071490 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b7de00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000c750 tx=0x7f9b7000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b70005ad0 con 0x7f9b80071490 2024-09-13T17:22:22.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b70005c30 con 0x7f9b80071490 2024-09-13T17:22:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b700072b0 con 0x7f9b80071490 2024-09-13T17:22:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.574+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b8015b5b0 con 0x7f9b80071490 2024-09-13T17:22:22.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.575+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b80152430 con 0x7f9b80071490 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.576+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b800b1670 con 0x7f9b80071490 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.577+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9b70007410 con 0x7f9b80071490 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.577+0000 7f9b76000640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 0x7f9b6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.577+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b70088b00 con 0x7f9b80071490 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.578+0000 7f9b7d400640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 0x7f9b6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.579+0000 7f9b7d400640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 0x7f9b6006d0c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9b801521b0 tx=0x7f9b6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.581+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9b7008ae60 con 0x7f9b80071490 2024-09-13T17:22:22.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.802+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9b800afc70 con 0x7f9b80071490 2024-09-13T17:22:22.804 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:22:22.803+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:22:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.816+0000 7f9b76000640 1 -- 172.21.15.28:0/2429094324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v166) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9b70017020 con 0x7f9b80071490 2024-09-13T17:22:22.818 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:22:22.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 msgr2=0x7f9b6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 0x7f9b6006d0c0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9b801521b0 tx=0x7f9b6c009040 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 msgr2=0x7f9b80158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f9b7000c750 tx=0x7f9b7000cc10 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 shutdown_connections 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9b6006ac00 0x7f9b6006d0c0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9b8015ab60 0x7f9b80151110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9b8015a780 0x7f9b800acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 --2- 172.21.15.28:0/2429094324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9b80071490 0x7f9b80158790 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 >> 172.21.15.28:0/2429094324 conn(0x7f9b800b0270 msgr2=0x7f9b800cebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 shutdown_connections 2024-09-13T17:22:22.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.818+0000 7f9b84c00640 1 -- 172.21.15.28:0/2429094324 wait complete. 2024-09-13T17:22:22.824 INFO:tasks.ceph.mds.c.smithi028.stderr: -3185> 2024-09-13T17:21:42.277+0000 7f9981400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:22:22.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:22:22.873 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:22:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.993+0000 7f8c07200640 1 Processor -- start 2024-09-13T17:22:22.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.993+0000 7f8c07200640 1 -- start start 2024-09-13T17:22:22.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.994+0000 7f8c07200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c08143320 0x7f8c08070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.994+0000 7f8c07200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8c080cd120 0x7f8c080cd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:22.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.994+0000 7f8c07200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c08071630 con 0x7f8c08143910 2024-09-13T17:22:22.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.994+0000 7f8c07200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c080cdad0 con 0x7f8c08143320 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.994+0000 7f8c07200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c08143700 con 0x7f8c080cd120 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8c08143320 0x7f8c08070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8bfd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8c08143910 0x7f8c08071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:59882/0 (socket says 172.21.15.28:59882) 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8bfd400640 1 -- 172.21.15.28:0/3818559138 learned_addr learned my addr 172.21.15.28:0/3818559138 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c06800640 1 --2- 172.21.15.28:0/3818559138 >> v2:172.21.15.189:3300/0 conn(0x7f8c080cd120 0x7f8c080cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05e00640 1 -- 172.21.15.28:0/3818559138 >> v2:172.21.15.189:3300/0 conn(0x7f8c080cd120 msgr2=0x7f8c080cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05e00640 1 --2- 172.21.15.28:0/3818559138 >> v2:172.21.15.189:3300/0 conn(0x7f8c080cd120 0x7f8c080cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05e00640 1 -- 172.21.15.28:0/3818559138 >> v1:172.21.15.28:6789/0 conn(0x7f8c08143910 legacy=0x7f8c08071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05e00640 1 -- 172.21.15.28:0/3818559138 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c08070a20 con 0x7f8c08143320 2024-09-13T17:22:22.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.995+0000 7f8c05400640 1 -- 172.21.15.28:0/3818559138 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756374700 0 0) 0x7f8c08071630 con 0x7f8c08143910 2024-09-13T17:22:22.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.996+0000 7f8c05e00640 1 --2- 172.21.15.28:0/3818559138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143320 0x7f8c08070440 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8c080ce320 tx=0x7f8bf802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd05ace5cb7be8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:22.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.996+0000 7f8c05400640 1 -- 172.21.15.28:0/3818559138 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8032440 con 0x7f8c08143320 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c05400640 1 -- 172.21.15.28:0/3818559138 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf80325a0 con 0x7f8c08143320 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c05400640 1 -- 172.21.15.28:0/3818559138 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf8032930 con 0x7f8c08143320 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 -- 172.21.15.28:0/3818559138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143320 msgr2=0x7f8c08070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 --2- 172.21.15.28:0/3818559138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143320 0x7f8c08070440 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7f8c080ce320 tx=0x7f8bf802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 -- 172.21.15.28:0/3818559138 shutdown_connections 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 --2- 172.21.15.28:0/3818559138 >> v2:172.21.15.189:3300/0 conn(0x7f8c080cd120 0x7f8c080cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 --2- 172.21.15.28:0/3818559138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143320 0x7f8c08070440 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 -- 172.21.15.28:0/3818559138 >> 172.21.15.28:0/3818559138 conn(0x7f8c08069e80 msgr2=0x7f8c0806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 -- 172.21.15.28:0/3818559138 shutdown_connections 2024-09-13T17:22:23.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.997+0000 7f8c07200640 1 -- 172.21.15.28:0/3818559138 wait complete. 2024-09-13T17:22:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.999+0000 7f8c07200640 1 Processor -- start 2024-09-13T17:22:23.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:22.999+0000 7f8c07200640 1 -- start start 2024-09-13T17:22:23.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c080cd120 0x7f8c081f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:23.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 0x7f8c0813e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:23.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:23.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080a8b60 con 0x7f8c08143910 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c081f2d80 con 0x7f8c080cd120 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c07200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c080a85e0 con 0x7f8c08143320 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c06800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52490/0 (socket says 172.21.15.28:52490) 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.000+0000 7f8c06800640 1 -- 172.21.15.28:0/3649848592 learned_addr learned my addr 172.21.15.28:0/3649848592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c05e00640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c080cd120 0x7f8c081f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8bfd400640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 0x7f8c0813e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 -- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 msgr2=0x7f8c0813e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 0x7f8c0813e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 -- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c080cd120 msgr2=0x7f8c081f2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c080cd120 0x7f8c081f2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 -- 172.21.15.28:0/3649848592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf8009400 con 0x7f8c08143910 2024-09-13T17:22:23.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8bfd400640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 0x7f8c0813e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c06800640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d30 tx=0x7f8c0000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c00009ba0 con 0x7f8c08143910 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c0000f040 con 0x7f8c08143910 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c000146c0 con 0x7f8c08143910 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.001+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c080a9380 con 0x7f8c08143910 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.002+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c081e0900 con 0x7f8c08143910 2024-09-13T17:22:23.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.003+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c080b3800 con 0x7f8c08143910 2024-09-13T17:22:23.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.004+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8c00014820 con 0x7f8c08143910 2024-09-13T17:22:23.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.005+0000 7f8bfea00640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 0x7f8be406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:23.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.005+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8c00089e00 con 0x7f8c08143910 2024-09-13T17:22:23.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.005+0000 7f8c05e00640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 0x7f8be406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:23.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.006+0000 7f8c05e00640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 0x7f8be406d0c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8034000 tx=0x7f8bf802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:23.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.009+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8c00054f70 con 0x7f8c08143910 2024-09-13T17:22:23.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.230+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8c081e0af0 con 0x7f8c08143910 2024-09-13T17:22:23.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.818+0000 7f8bfea00640 1 -- 172.21.15.28:0/3649848592 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v167) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8c00058590 con 0x7f8c08143910 2024-09-13T17:22:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 msgr2=0x7f8be406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 0x7f8be406d0c0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f8bf8034000 tx=0x7f8bf802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 msgr2=0x7f8c081e2290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:23.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 secure :-1 s=READY pgs=2404 cs=0 l=1 rev1=1 crypto rx=0x7f8c00009d30 tx=0x7f8c0000c490 comp rx=0 tx=0).stop 2024-09-13T17:22:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 shutdown_connections 2024-09-13T17:22:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8be406ac00 0x7f8be406d0c0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8c08143910 0x7f8c081e2290 unknown :-1 s=CLOSED pgs=2404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8c08143320 0x7f8c0813e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 --2- 172.21.15.28:0/3649848592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8c080cd120 0x7f8c081f2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.821+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 >> 172.21.15.28:0/3649848592 conn(0x7f8c08069e80 msgr2=0x7f8c0806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.822+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 shutdown_connections 2024-09-13T17:22:23.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:23.822+0000 7f8c07200640 1 -- 172.21.15.28:0/3649848592 wait complete. 2024-09-13T17:22:23.841 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:22:24.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.042+0000 7fa3cf400640 1 Processor -- start 2024-09-13T17:22:24.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.042+0000 7fa3cf400640 1 -- start start 2024-09-13T17:22:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.043+0000 7fa3cf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159590 0x7fa3c8070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cf400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3c80b2800 0x7fa3c815a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cf400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c815a4c0 con 0x7fa3c8159970 2024-09-13T17:22:24.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cf400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c8071580 con 0x7fa3c80b2800 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cf400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c8159e70 con 0x7fa3c8159590 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3c80b2800 0x7fa3c815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa3c80b2800 0x7fa3c815a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50666/0 (socket says 172.21.15.28:50666) 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 -- 172.21.15.28:0/3879722781 learned_addr learned my addr 172.21.15.28:0/3879722781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cca00640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159590 0x7fa3c8070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 -- 172.21.15.28:0/3879722781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159590 msgr2=0x7fa3c8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159590 0x7fa3c8070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 -- 172.21.15.28:0/3879722781 >> v1:172.21.15.189:6789/0 conn(0x7fa3c8159970 legacy=0x7fa3c80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.044+0000 7fa3cd400640 1 -- 172.21.15.28:0/3879722781 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c815a150 con 0x7fa3c80b2800 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.045+0000 7fa3cd400640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c80b2800 0x7fa3c815a950 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000d150 tx=0x7fa3c002c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd6e5c2abb4a976 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:24.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3c7400640 1 -- 172.21.15.28:0/3879722781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0009600 con 0x7fa3c80b2800 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3c7400640 1 -- 172.21.15.28:0/3879722781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c0009760 con 0x7fa3c80b2800 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3c7400640 1 -- 172.21.15.28:0/3879722781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0009af0 con 0x7fa3c80b2800 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 -- 172.21.15.28:0/3879722781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c80b2800 msgr2=0x7fa3c815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c80b2800 0x7fa3c815a950 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fa3c000d150 tx=0x7fa3c002c930 comp rx=0 tx=0).stop 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 -- 172.21.15.28:0/3879722781 shutdown_connections 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c80b2800 0x7fa3c815a950 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 --2- 172.21.15.28:0/3879722781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159590 0x7fa3c8070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 -- 172.21.15.28:0/3879722781 >> 172.21.15.28:0/3879722781 conn(0x7fa3c8069e80 msgr2=0x7fa3c806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 -- 172.21.15.28:0/3879722781 shutdown_connections 2024-09-13T17:22:24.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.046+0000 7fa3cf400640 1 -- 172.21.15.28:0/3879722781 wait complete. 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.048+0000 7fa3cf400640 1 Processor -- start 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.048+0000 7fa3cf400640 1 -- start start 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3c80b2800 0x7fa3c8145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159970 0x7fa3c8142dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81462f0 con 0x7fa3c8159590 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81465d0 con 0x7fa3c80b2800 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cf400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c81468b0 con 0x7fa3c8159970 2024-09-13T17:22:24.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50682/0 (socket says 172.21.15.28:50682) 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3c7e00640 1 -- 172.21.15.28:0/1451007632 learned_addr learned my addr 172.21.15.28:0/1451007632 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cd400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159970 0x7fa3c8142dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.049+0000 7fa3cca00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3c80b2800 0x7fa3c8145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 -- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159970 msgr2=0x7fa3c8142dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159970 0x7fa3c8142dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 -- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3c80b2800 msgr2=0x7fa3c8145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3c80b2800 0x7fa3c8145ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 -- 172.21.15.28:0/1451007632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c000bd40 con 0x7fa3c8159590 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c7e00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400c3c0 tx=0x7fa3b400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b40052d0 con 0x7fa3c8159590 2024-09-13T17:22:24.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b4005430 con 0x7fa3c8159590 2024-09-13T17:22:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4003830 con 0x7fa3c8159590 2024-09-13T17:22:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.050+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c80cd980 con 0x7fa3c8159590 2024-09-13T17:22:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.051+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c80cd5e0 con 0x7fa3c8159590 2024-09-13T17:22:24.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.051+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c815a950 con 0x7fa3c8159590 2024-09-13T17:22:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.052+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa3b4003990 con 0x7fa3c8159590 2024-09-13T17:22:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.054+0000 7fa3c4c00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 0x7fa3a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.054+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa3b4088840 con 0x7fa3c8159590 2024-09-13T17:22:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.054+0000 7fa3cca00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 0x7fa3a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:24.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.055+0000 7fa3cca00640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 0x7fa3a806d0c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009b40 tx=0x7fa3bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:24.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.058+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa3b40539d0 con 0x7fa3c8159590 2024-09-13T17:22:24.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:24.239+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa3c8070bd0 con 0x7fa3c8159590 2024-09-13T17:22:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.242+0000 7fa3c4c00640 1 -- 172.21.15.28:0/1451007632 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v195) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa3b4056ff0 con 0x7fa3c8159590 2024-09-13T17:22:25.244 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:22:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 msgr2=0x7fa3a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 0x7fa3a806d0c0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc009b40 tx=0x7fa3bc009190 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 msgr2=0x7fa3c806bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fa3b400c3c0 tx=0x7fa3b400c880 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 shutdown_connections 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa3a806ac00 0x7fa3a806d0c0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa3c8159970 0x7fa3c8142dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3c8159590 0x7fa3c806bd80 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 --2- 172.21.15.28:0/1451007632 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3c80b2800 0x7fa3c8145ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.246+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 >> 172.21.15.28:0/1451007632 conn(0x7fa3c8069e80 msgr2=0x7fa3c806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.247+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 shutdown_connections 2024-09-13T17:22:25.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.247+0000 7fa3cf400640 1 -- 172.21.15.28:0/1451007632 wait complete. 2024-09-13T17:22:25.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:22:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.430+0000 7f2c5ee00640 1 Processor -- start 2024-09-13T17:22:25.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.430+0000 7f2c5ee00640 1 -- start start 2024-09-13T17:22:25.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c5ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c5815a7a0 0x7f2c580ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c5ee00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c580ce9d0 0x7f2c580b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c5ee00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c5806c950 con 0x7f2c580b1ab0 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c5ee00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c580cee10 con 0x7f2c580ce9d0 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c5ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c580b1ec0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c57400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c580ce9d0 0x7f2c580b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c57400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2c580ce9d0 0x7f2c580b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50690/0 (socket says 172.21.15.28:50690) 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.431+0000 7f2c57400640 1 -- 172.21.15.28:0/3212484665 learned_addr learned my addr 172.21.15.28:0/3212484665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.432+0000 7f2c57e00640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c5815a7a0 0x7f2c580ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.432+0000 7f2c57400640 1 -- 172.21.15.28:0/3212484665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c5815a7a0 msgr2=0x7f2c580ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.432+0000 7f2c57400640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c5815a7a0 0x7f2c580ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.432+0000 7f2c57400640 1 -- 172.21.15.28:0/3212484665 >> v1:172.21.15.189:6789/0 conn(0x7f2c580b1ab0 legacy=0x7f2c5806b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.432+0000 7f2c57400640 1 -- 172.21.15.28:0/3212484665 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c580b2150 con 0x7f2c580ce9d0 2024-09-13T17:22:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.433+0000 7f2c57400640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c580ce9d0 0x7f2c580b1570 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009870 tx=0x7f2c4402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd4b7e7d99c44af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.433+0000 7f2c56a00640 1 -- 172.21.15.28:0/3212484665 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44032440 con 0x7f2c580ce9d0 2024-09-13T17:22:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.433+0000 7f2c56a00640 1 -- 172.21.15.28:0/3212484665 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c440325a0 con 0x7f2c580ce9d0 2024-09-13T17:22:25.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.433+0000 7f2c56a00640 1 -- 172.21.15.28:0/3212484665 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c44032930 con 0x7f2c580ce9d0 2024-09-13T17:22:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 -- 172.21.15.28:0/3212484665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c580ce9d0 msgr2=0x7f2c580b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c580ce9d0 0x7f2c580b1570 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f2c44009870 tx=0x7f2c4402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 -- 172.21.15.28:0/3212484665 shutdown_connections 2024-09-13T17:22:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c580ce9d0 0x7f2c580b1570 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/3212484665 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c5815a7a0 0x7f2c580ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 -- 172.21.15.28:0/3212484665 >> 172.21.15.28:0/3212484665 conn(0x7f2c58069e80 msgr2=0x7f2c5806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 -- 172.21.15.28:0/3212484665 shutdown_connections 2024-09-13T17:22:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.434+0000 7f2c5ee00640 1 -- 172.21.15.28:0/3212484665 wait complete. 2024-09-13T17:22:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.436+0000 7f2c5ee00640 1 Processor -- start 2024-09-13T17:22:25.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.436+0000 7f2c5ee00640 1 -- start start 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.436+0000 7f2c5ee00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c580b1ab0 0x7f2c5815db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c580ce9d0 0x7f2c581563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c58155f80 con 0x7f2c5815a7a0 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5815bec0 con 0x7f2c580ce9d0 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ee00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5815c0f0 con 0x7f2c580b1ab0 2024-09-13T17:22:25.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c57e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c580b1ab0 0x7f2c5815db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50692/0 (socket says 172.21.15.28:50692) 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 -- 172.21.15.28:0/2693466826 learned_addr learned my addr 172.21.15.28:0/2693466826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c57400640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c580ce9d0 0x7f2c581563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 -- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c580b1ab0 msgr2=0x7f2c5815db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c580b1ab0 0x7f2c5815db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 -- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c580ce9d0 msgr2=0x7f2c581563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c580ce9d0 0x7f2c581563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:25.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.437+0000 7f2c5ce00640 1 -- 172.21.15.28:0/2693466826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c440094d0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c5ce00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c6b0 tx=0x7f2c5000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:25.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50003b70 con 0x7f2c5815a7a0 2024-09-13T17:22:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c50003cf0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c50003ea0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c5815c3d0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.438+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c58155ba0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.439+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c581560c0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.440+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2c50005310 con 0x7f2c5815a7a0 2024-09-13T17:22:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.441+0000 7f2c4fe00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 0x7f2c3806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.441+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f2c50088cf0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.441+0000 7f2c57e00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 0x7f2c3806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:25.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.442+0000 7f2c57e00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 0x7f2c3806d0c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009b40 tx=0x7f2c48009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:25.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.445+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2c581560c0 con 0x7f2c5815a7a0 2024-09-13T17:22:25.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:25.621+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2c581560c0 con 0x7f2c5815a7a0 2024-09-13T17:22:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.245+0000 7f2c4fe00640 1 -- 172.21.15.28:0/2693466826 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v196) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2c581560c0 con 0x7f2c5815a7a0 2024-09-13T17:22:26.247 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:22:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.248+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 msgr2=0x7f2c3806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.248+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 0x7f2c3806d0c0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f2c48009b40 tx=0x7f2c48009190 comp rx=0 tx=0).stop 2024-09-13T17:22:26.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 msgr2=0x7f2c5815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000c6b0 tx=0x7f2c5000cb70 comp rx=0 tx=0).stop 2024-09-13T17:22:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 shutdown_connections 2024-09-13T17:22:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2c3806ac00 0x7f2c3806d0c0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c5815a7a0 0x7f2c5815b980 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2c580ce9d0 0x7f2c581563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 --2- 172.21.15.28:0/2693466826 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2c580b1ab0 0x7f2c5815db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 >> 172.21.15.28:0/2693466826 conn(0x7f2c58069e80 msgr2=0x7f2c580ca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 shutdown_connections 2024-09-13T17:22:26.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.249+0000 7f2c5ee00640 1 -- 172.21.15.28:0/2693466826 wait complete. 2024-09-13T17:22:26.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:22:26.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.427+0000 7fecbcc00640 1 Processor -- start 2024-09-13T17:22:26.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.428+0000 7fecbcc00640 1 -- start start 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecbcc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fecb8071490 0x7fecb806c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecbcc00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb806c8b0 con 0x7fecb815ab60 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecbcc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecb806cb90 con 0x7fecb815a780 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecbcc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecb80cc980 con 0x7fecb8071490 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecafe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fecb815ab60 0x7fecb806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60740/0 (socket says 172.21.15.28:60740) 2024-09-13T17:22:26.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecafe00640 1 -- 172.21.15.28:0/2897237818 learned_addr learned my addr 172.21.15.28:0/2897237818 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecb6800640 1 --2- 172.21.15.28:0/2897237818 >> v2:172.21.15.189:3300/0 conn(0x7fecb8071490 0x7fecb806c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.429+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156499257 0 0) 0x7fecb806c8b0 con 0x7fecb815ab60 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec9c003520 con 0x7fecb815ab60 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58205099 0 0) 0x7fec9c003520 con 0x7fecb815ab60 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 >> v2:172.21.15.189:3300/0 conn(0x7fecb8071490 msgr2=0x7fecb806c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 --2- 172.21.15.28:0/2897237818 >> v2:172.21.15.189:3300/0 conn(0x7fecb8071490 0x7fecb806c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 >> v1:172.21.15.189:6789/0 conn(0x7fecb815a780 legacy=0x7fecb8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb80ccc60 con 0x7fecb815ab60 2024-09-13T17:22:26.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7feca0001670 con 0x7fecb815ab60 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 >> v1:172.21.15.28:6789/0 conn(0x7fecb815ab60 legacy=0x7fecb806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 0x7fec9c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c0047c0 0x7fec9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c005180 0x7fec9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80cc980 con 0x7fec9c003970 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c00c020 con 0x7fec9c005180 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9c00c1e0 con 0x7fec9c0047c0 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.430+0000 7fecb5e00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 0x7fec9c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb6800640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c005180 0x7fec9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecafe00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c0047c0 0x7fec9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb5e00640 1 -- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c0047c0 msgr2=0x7fec9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb5e00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c0047c0 0x7fec9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb5e00640 1 -- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c005180 msgr2=0x7fec9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:26.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb5e00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec9c005180 0x7fec9c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb5e00640 1 -- 172.21.15.28:0/2897237818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec9c003520 con 0x7fec9c003970 2024-09-13T17:22:26.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.431+0000 7fecb6800640 1 -- 172.21.15.28:0/2897237818 reap_dead start 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecb5e00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 0x7fec9c003d90 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7feca8001750 tx=0x7fec9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceb53f5c8009a35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca80052d0 con 0x7fec9c003970 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca8005430 con 0x7fec9c003970 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecb5400640 1 -- 172.21.15.28:0/2897237818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca802abc0 con 0x7fec9c003970 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecbcc00640 1 -- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 msgr2=0x7fec9c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.432+0000 7fecbcc00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 0x7fec9c003d90 secure :-1 s=READY pgs=2410 cs=0 l=1 rev1=1 crypto rx=0x7feca8001750 tx=0x7fec9c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.433+0000 7fecbcc00640 1 -- 172.21.15.28:0/2897237818 shutdown_connections 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.433+0000 7fecbcc00640 1 --2- 172.21.15.28:0/2897237818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec9c003970 0x7fec9c003d90 unknown :-1 s=CLOSED pgs=2410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.433+0000 7fecbcc00640 1 -- 172.21.15.28:0/2897237818 >> 172.21.15.28:0/2897237818 conn(0x7fecb80b0270 msgr2=0x7fecb80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.433+0000 7fecbcc00640 1 -- 172.21.15.28:0/2897237818 shutdown_connections 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.433+0000 7fecbcc00640 1 -- 172.21.15.28:0/2897237818 wait complete. 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.434+0000 7fecbcc00640 1 Processor -- start 2024-09-13T17:22:26.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.434+0000 7fecbcc00640 1 -- start start 2024-09-13T17:22:26.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c003970 0x7fecb80c77f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecb80b1bb0 0x7fecb80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80c34e0 con 0x7fecb81534f0 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b2170 con 0x7fecb80b1bb0 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.435+0000 7fecbcc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecb80b2450 con 0x7fec9c003970 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50714/0 (socket says 172.21.15.28:50714) 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 -- 172.21.15.28:0/1313068588 learned_addr learned my addr 172.21.15.28:0/1313068588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecb6800640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecb80b1bb0 0x7fecb80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecb5e00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c003970 0x7fecb80c77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 -- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c003970 msgr2=0x7fecb80c77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c003970 0x7fecb80c77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 -- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecb80b1bb0 msgr2=0x7fecb80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecb80b1bb0 0x7fecb80ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 -- 172.21.15.28:0/1313068588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb80c36c0 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.436+0000 7fecafe00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7feca000e730 tx=0x7feca000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.437+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000c6c0 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.437+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca000c820 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.437+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000cbd0 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.437+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecb80c39a0 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.437+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecb80c7d30 con 0x7fecb81534f0 2024-09-13T17:22:26.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.438+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecb80c3be0 con 0x7fecb81534f0 2024-09-13T17:22:26.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.439+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feca001d460 con 0x7fecb81534f0 2024-09-13T17:22:26.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.439+0000 7fecae000640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 0x7fec9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.439+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7feca0088fc0 con 0x7fecb81534f0 2024-09-13T17:22:26.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.440+0000 7fecb5e00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 0x7fec9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.440+0000 7fecb5e00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 0x7fec9406d0f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7feca8032000 tx=0x7feca802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.444+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fecb80c3be0 con 0x7fecb81534f0 2024-09-13T17:22:26.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.625+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fecb80c3be0 con 0x7fecb81534f0 2024-09-13T17:22:26.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.629+0000 7fecae000640 1 -- 172.21.15.28:0/1313068588 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7fecb80c3be0 con 0x7fecb81534f0 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 msgr2=0x7fec9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 0x7fec9406d0f0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7feca8032000 tx=0x7feca802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 msgr2=0x7fecb80b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 secure :-1 s=READY pgs=2411 cs=0 l=1 rev1=1 crypto rx=0x7feca000e730 tx=0x7feca000ebf0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 shutdown_connections 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec9406ac30 0x7fec9406d0f0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecb80b1bb0 0x7fecb80ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fecb81534f0 0x7fecb80b1670 unknown :-1 s=CLOSED pgs=2411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 --2- 172.21.15.28:0/1313068588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec9c003970 0x7fecb80c77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.631+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 >> 172.21.15.28:0/1313068588 conn(0x7fecb80b0270 msgr2=0x7fecb8070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.632+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 shutdown_connections 2024-09-13T17:22:26.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.632+0000 7fecbcc00640 1 -- 172.21.15.28:0/1313068588 wait complete. 2024-09-13T17:22:26.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace' 2024-09-13T17:22:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.805+0000 7f3217200640 1 Processor -- start 2024-09-13T17:22:26.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.805+0000 7f3217200640 1 -- start start 2024-09-13T17:22:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f3217200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3218159440 0x7f32180b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f3217200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 0x7f32180b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f3217200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32180c3850 con 0x7f3218159820 2024-09-13T17:22:26.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f3217200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f321815a930 con 0x7f3218159440 2024-09-13T17:22:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f3217200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3218159d20 con 0x7f32180b29f0 2024-09-13T17:22:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f320d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3218159820 0x7f32180b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60742/0 (socket says 172.21.15.28:60742) 2024-09-13T17:22:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.806+0000 7f320d400640 1 -- 172.21.15.28:0/3175814326 learned_addr learned my addr 172.21.15.28:0/3175814326 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3216800640 1 --2- 172.21.15.28:0/3175814326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 0x7f32180b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3215e00640 1 --2- 172.21.15.28:0/3175814326 >> v2:172.21.15.189:3300/0 conn(0x7f3218159440 0x7f32180b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3215400640 1 -- 172.21.15.28:0/3175814326 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990735208 0 0) 0x7f32180c3850 con 0x7f3218159820 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3215400640 1 -- 172.21.15.28:0/3175814326 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31fc003520 con 0x7f3218159820 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3216800640 1 -- 172.21.15.28:0/3175814326 >> v2:172.21.15.189:3300/0 conn(0x7f3218159440 msgr2=0x7f32180b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3216800640 1 --2- 172.21.15.28:0/3175814326 >> v2:172.21.15.189:3300/0 conn(0x7f3218159440 0x7f32180b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3216800640 1 -- 172.21.15.28:0/3175814326 >> v1:172.21.15.28:6789/0 conn(0x7f3218159820 legacy=0x7f32180b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.807+0000 7f3216800640 1 -- 172.21.15.28:0/3175814326 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f321815a000 con 0x7f32180b29f0 2024-09-13T17:22:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.808+0000 7f3216800640 1 --2- 172.21.15.28:0/3175814326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 0x7f32180b0980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3210009dc0 tx=0x7f321002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ddb605cc3b98c686 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3215400640 1 -- 172.21.15.28:0/3175814326 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210007740 con 0x7f32180b29f0 2024-09-13T17:22:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3215400640 1 -- 172.21.15.28:0/3175814326 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321002d050 con 0x7f32180b29f0 2024-09-13T17:22:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3215400640 1 -- 172.21.15.28:0/3175814326 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3210007ac0 con 0x7f32180b29f0 2024-09-13T17:22:26.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 -- 172.21.15.28:0/3175814326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 msgr2=0x7f32180b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 --2- 172.21.15.28:0/3175814326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 0x7f32180b0980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3210009dc0 tx=0x7f321002a710 comp rx=0 tx=0).stop 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 -- 172.21.15.28:0/3175814326 shutdown_connections 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 --2- 172.21.15.28:0/3175814326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32180b29f0 0x7f32180b0980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 --2- 172.21.15.28:0/3175814326 >> v2:172.21.15.189:3300/0 conn(0x7f3218159440 0x7f32180b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 -- 172.21.15.28:0/3175814326 >> 172.21.15.28:0/3175814326 conn(0x7f3218072e40 msgr2=0x7f3218073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 -- 172.21.15.28:0/3175814326 shutdown_connections 2024-09-13T17:22:26.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.809+0000 7f3217200640 1 -- 172.21.15.28:0/3175814326 wait complete. 2024-09-13T17:22:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.811+0000 7f3217200640 1 Processor -- start 2024-09-13T17:22:26.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.811+0000 7f3217200640 1 -- start start 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3218159440 0x7f32181571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 0x7f32180adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218156750 con 0x7f32180b29f0 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3218150600 con 0x7f3218159820 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3217200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3218156be0 con 0x7f3218159440 2024-09-13T17:22:26.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3215e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3215e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50720/0 (socket says 172.21.15.28:50720) 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3215e00640 1 -- 172.21.15.28:0/3639453195 learned_addr learned my addr 172.21.15.28:0/3639453195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f320d400640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3218159440 0x7f32181571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.812+0000 7f3216800640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 0x7f32180adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 -- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3218159440 msgr2=0x7f32181571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3218159440 0x7f32181571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 -- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 msgr2=0x7f32180adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 0x7f32180adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 -- 172.21.15.28:0/3639453195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3210009a70 con 0x7f32180b29f0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3216800640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 0x7f32180adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3215e00640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f320800e9b0 tx=0x7f320800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3208009670 con 0x7f32180b29f0 2024-09-13T17:22:26.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320800c9a0 con 0x7f32180b29f0 2024-09-13T17:22:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320800cd50 con 0x7f32180b29f0 2024-09-13T17:22:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3218155b40 con 0x7f32180b29f0 2024-09-13T17:22:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.813+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3218156100 con 0x7f32180b29f0 2024-09-13T17:22:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.815+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32180b22e0 con 0x7f32180b29f0 2024-09-13T17:22:26.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.815+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3208013470 con 0x7f32180b29f0 2024-09-13T17:22:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.816+0000 7f320ea00640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 0x7f31f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:26.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.816+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3208089390 con 0x7f32180b29f0 2024-09-13T17:22:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.817+0000 7f320d400640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 0x7f31f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:26.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.818+0000 7f320d400640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 0x7f31f406d0c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f321815af00 tx=0x7f3200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:26.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:26.821+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32080547d0 con 0x7f32180b29f0 2024-09-13T17:22:27.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.067+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace"]} v 0) v1 -- 0x7f321815b870 con 0x7f32180b29f0 2024-09-13T17:22:27.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.255+0000 7f320ea00640 1 -- 172.21.15.28:0/3639453195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_mdsdir_scrub_backtrace"]}]=0 v709) v1 ==== 140+0+0 (secure 0 0 0) 0x7f3208057df0 con 0x7f32180b29f0 2024-09-13T17:22:27.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 msgr2=0x7f31f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 0x7f31f406d0c0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f321815af00 tx=0x7f3200009040 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 msgr2=0x7f3218155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 secure :-1 s=READY pgs=2413 cs=0 l=1 rev1=1 crypto rx=0x7f320800e9b0 tx=0x7f320800ee70 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 shutdown_connections 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f31f406ac00 0x7f31f406d0c0 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3218159820 0x7f32180adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3218159440 0x7f32181571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.257+0000 7f3217200640 1 --2- 172.21.15.28:0/3639453195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32180b29f0 0x7f3218155440 unknown :-1 s=CLOSED pgs=2413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.258+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 >> 172.21.15.28:0/3639453195 conn(0x7f3218072e40 msgr2=0x7f3218073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.258+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 shutdown_connections 2024-09-13T17:22:27.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.258+0000 7f3217200640 1 -- 172.21.15.28:0/3639453195 wait complete. 2024-09-13T17:22:27.275 INFO:tasks.cephfs_test_runner:test_mdsdir_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) ... ok 2024-09-13T17:22:27.276 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) 2024-09-13T17:22:27.276 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.438+0000 7faad7e00640 1 Processor -- start 2024-09-13T17:22:27.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.438+0000 7faad7e00640 1 -- start start 2024-09-13T17:22:27.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faad006c9e0 0x7faad015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad7e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad0159d90 con 0x7faad01592a0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad7e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faad00b2040 con 0x7faad006f970 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faad015b0b0 con 0x7faad006c9e0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad5400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7faad01592a0 0x7faad0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60744/0 (socket says 172.21.15.28:60744) 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.439+0000 7faad5400640 1 -- 172.21.15.28:0/2422124861 learned_addr learned my addr 172.21.15.28:0/2422124861 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faad5e00640 1 --2- 172.21.15.28:0/2422124861 >> v2:172.21.15.189:3300/0 conn(0x7faad006c9e0 0x7faad015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661416892 0 0) 0x7faad0159d90 con 0x7faad01592a0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab8005730 con 0x7faad01592a0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220807758 0 0) 0x7faab8005730 con 0x7faad01592a0 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 >> v2:172.21.15.189:3300/0 conn(0x7faad006c9e0 msgr2=0x7faad015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:27.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 --2- 172.21.15.28:0/2422124861 >> v2:172.21.15.189:3300/0 conn(0x7faad006c9e0 0x7faad015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 >> v1:172.21.15.189:6789/0 conn(0x7faad006f970 legacy=0x7faad006c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad0159ff0 con 0x7faad01592a0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7faac4001690 con 0x7faad01592a0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 >> v1:172.21.15.28:6789/0 conn(0x7faad01592a0 legacy=0x7faad0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faab8005bd0 0x7faab8005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab80069d0 0x7faab8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 0x7faab800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad015b0b0 con 0x7faab8007390 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab800e290 con 0x7faab80069d0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.440+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab800e450 con 0x7faab8005bd0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 0x7faab800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad4a00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab80069d0 0x7faab8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5400640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faab8005bd0 0x7faab8005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 -- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faab8005bd0 msgr2=0x7faab8005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faab8005bd0 0x7faab8005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 -- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab80069d0 msgr2=0x7faab8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab80069d0 0x7faab8006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5e00640 1 -- 172.21.15.28:0/2422124861 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab8005730 con 0x7faab8007390 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.441+0000 7faad5400640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faab8005bd0 0x7faab8005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:27.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.442+0000 7faad5e00640 1 -- 172.21.15.28:0/2422124861 reap_dead start 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.442+0000 7faad5e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 0x7faab800db90 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7faac8009fc0 tx=0x7faab8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2b949fdba99512 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.442+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8003b90 con 0x7faab8007390 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.442+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac800f040 con 0x7faab8007390 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.442+0000 7faacfe00640 1 -- 172.21.15.28:0/2422124861 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac8004020 con 0x7faab8007390 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 -- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 msgr2=0x7faab800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 0x7faab800db90 secure :-1 s=READY pgs=2415 cs=0 l=1 rev1=1 crypto rx=0x7faac8009fc0 tx=0x7faab8007a00 comp rx=0 tx=0).stop 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 -- 172.21.15.28:0/2422124861 shutdown_connections 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 --2- 172.21.15.28:0/2422124861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faab8007390 0x7faab800db90 unknown :-1 s=CLOSED pgs=2415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 -- 172.21.15.28:0/2422124861 >> 172.21.15.28:0/2422124861 conn(0x7faad0069e80 msgr2=0x7faad006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:27.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 -- 172.21.15.28:0/2422124861 shutdown_connections 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.443+0000 7faad7e00640 1 -- 172.21.15.28:0/2422124861 wait complete. 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.445+0000 7faad7e00640 1 Processor -- start 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.445+0000 7faad7e00640 1 -- start start 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 0x7faad01f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faad015c700 0x7faad014dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad01f2ed0 con 0x7faad01f5be0 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faad01f2630 con 0x7faab8007390 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad01f2910 con 0x7faad015c700 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50724/0 (socket says 172.21.15.28:50724) 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 -- 172.21.15.28:0/1429581916 learned_addr learned my addr 172.21.15.28:0/1429581916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:27.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5400640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 0x7faad01f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad4a00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faad015c700 0x7faad014dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 -- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faad015c700 msgr2=0x7faad014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faad015c700 0x7faad014dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 -- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 msgr2=0x7faad01f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 0x7faad01f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5e00640 1 -- 172.21.15.28:0/1429581916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad01f2bf0 con 0x7faad01f5be0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.446+0000 7faad5400640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 0x7faad01f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faad5e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7faac800e230 tx=0x7faac8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac803c470 con 0x7faad01f5be0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faac803c5f0 con 0x7faad01f5be0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faac803c9a0 con 0x7faad01f5be0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faad01f56c0 con 0x7faad01f5be0 2024-09-13T17:22:27.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.447+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad015cb60 con 0x7faad01f5be0 2024-09-13T17:22:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.448+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faad01f5520 con 0x7faad01f5be0 2024-09-13T17:22:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.449+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faac803cb00 con 0x7faad01f5be0 2024-09-13T17:22:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.450+0000 7faacd600640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 0x7faab006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.451+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faac80a9e00 con 0x7faad01f5be0 2024-09-13T17:22:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.451+0000 7faad5400640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 0x7faab006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.452+0000 7faad5400640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 0x7faab006d0f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faac4001010 tx=0x7faac4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.455+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faad01f5520 con 0x7faad01f5be0 2024-09-13T17:22:27.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.635+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faad01f5520 con 0x7faad01f5be0 2024-09-13T17:22:27.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.636+0000 7faacd600640 1 -- 172.21.15.28:0/1429581916 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+12994 (secure 0 0 0) 0x7faad01f5520 con 0x7faad01f5be0 2024-09-13T17:22:27.639 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:27.639 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":196,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:26.241532+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/762464937":"2024-09-14T17:22:22.803971+0000","172.21.15.28:6834/762464937":"2024-09-14T17:22:22.803971+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 msgr2=0x7faab006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 0x7faab006d0f0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7faac4001010 tx=0x7faac4009040 comp rx=0 tx=0).stop 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 msgr2=0x7faad01f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7faac800e230 tx=0x7faac8009750 comp rx=0 tx=0).stop 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 shutdown_connections 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faab006ac30 0x7faab006d0f0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faad01f5be0 0x7faad01f6060 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faad015c700 0x7faad014dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 --2- 172.21.15.28:0/1429581916 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab8007390 0x7faad01f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.638+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 >> 172.21.15.28:0/1429581916 conn(0x7faad0069e80 msgr2=0x7faad0157410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.639+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 shutdown_connections 2024-09-13T17:22:27.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.639+0000 7faad7e00640 1 -- 172.21.15.28:0/1429581916 wait complete. 2024-09-13T17:22:27.656 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:22:27.657 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:22:27.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.814+0000 7f46f9400640 1 Processor -- start 2024-09-13T17:22:27.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.814+0000 7f46f9400640 1 -- start start 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.815+0000 7f46f9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.815+0000 7f46f9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46f4071490 0x7f46f406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.815+0000 7f46f9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f406c8b0 con 0x7f46f415a780 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.815+0000 7f46f9400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f406cb90 con 0x7f46f4071490 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.815+0000 7f46f9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40cc980 con 0x7f46f415ab60 2024-09-13T17:22:27.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f46f4071490 0x7f46f406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f46f415a780 0x7f46f4070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60760/0 (socket says 172.21.15.28:60760) 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f2800640 1 -- 172.21.15.28:0/3505307113 learned_addr learned my addr 172.21.15.28:0/3505307113 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46e9e00640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f3200640 1 -- 172.21.15.28:0/3505307113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 msgr2=0x7f46f406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f3200640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f3200640 1 -- 172.21.15.28:0/3505307113 >> v1:172.21.15.28:6789/0 conn(0x7f46f415a780 legacy=0x7f46f4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f3200640 1 -- 172.21.15.28:0/3505307113 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f40ccc60 con 0x7f46f4071490 2024-09-13T17:22:27.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.816+0000 7f46f1e00640 1 -- 172.21.15.28:0/3505307113 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117616341 0 0) 0x7f46f406c8b0 con 0x7f46f415a780 2024-09-13T17:22:27.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.817+0000 7f46f3200640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4071490 0x7f46f406c080 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009d90 tx=0x7f46ec02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19af9a88c8d7229a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f1e00640 1 -- 172.21.15.28:0/3505307113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec007740 con 0x7f46f4071490 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f1e00640 1 -- 172.21.15.28:0/3505307113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec02d050 con 0x7f46f4071490 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f1e00640 1 -- 172.21.15.28:0/3505307113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec007ac0 con 0x7f46f4071490 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 -- 172.21.15.28:0/3505307113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4071490 msgr2=0x7f46f406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4071490 0x7f46f406c080 secure :-1 s=READY pgs=2418 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009d90 tx=0x7f46ec02a660 comp rx=0 tx=0).stop 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 -- 172.21.15.28:0/3505307113 shutdown_connections 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f4071490 0x7f46f406c080 unknown :-1 s=CLOSED pgs=2418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 --2- 172.21.15.28:0/3505307113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 -- 172.21.15.28:0/3505307113 >> 172.21.15.28:0/3505307113 conn(0x7f46f40b0270 msgr2=0x7f46f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 -- 172.21.15.28:0/3505307113 shutdown_connections 2024-09-13T17:22:27.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.818+0000 7f46f9400640 1 -- 172.21.15.28:0/3505307113 wait complete. 2024-09-13T17:22:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.820+0000 7f46f9400640 1 Processor -- start 2024-09-13T17:22:27.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.820+0000 7f46f9400640 1 -- start start 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f4071490 0x7f46f40c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f40b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f413f380 con 0x7f46f415a780 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40ca530 con 0x7f46f4071490 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f40b2bf0 con 0x7f46f415ab60 2024-09-13T17:22:27.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46e9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50746/0 (socket says 172.21.15.28:50746) 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46e9e00640 1 -- 172.21.15.28:0/2146831034 learned_addr learned my addr 172.21.15.28:0/2146831034 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f3200640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f40b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.821+0000 7f46f2800640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f4071490 0x7f46f40c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 -- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 msgr2=0x7f46f40b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f40b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 -- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f4071490 msgr2=0x7f46f40c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:27.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f4071490 0x7f46f40c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 -- 172.21.15.28:0/2146831034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec009a70 con 0x7f46f415a780 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46e9e00640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00d5f0 tx=0x7f46dc00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc0125f0 con 0x7f46f415a780 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46dc012750 con 0x7f46f415a780 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc012b00 con 0x7f46f415a780 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f40b2ed0 con 0x7f46f415a780 2024-09-13T17:22:27.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.822+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f413f010 con 0x7f46f415a780 2024-09-13T17:22:27.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.823+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f4158bf0 con 0x7f46f415a780 2024-09-13T17:22:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.824+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f46dc012c60 con 0x7f46f415a780 2024-09-13T17:22:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.825+0000 7f46eb400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.825+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f46dc089580 con 0x7f46f415a780 2024-09-13T17:22:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.826+0000 7f46f2800640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:27.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.826+0000 7f46f2800640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f46e4000f60 tx=0x7f46e4007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:27.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:27.829+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46dc0549c0 con 0x7f46f415a780 2024-09-13T17:22:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.006+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f46f415a2c0 con 0x7f46f415a780 2024-09-13T17:22:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.006+0000 7f46eb400640 1 -- 172.21.15.28:0/2146831034 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46dc057fe0 con 0x7f46f415a780 2024-09-13T17:22:28.009 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.008+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 msgr2=0x7f46d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.008+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7f46e4000f60 tx=0x7f46e4007b50 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.008+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 msgr2=0x7f46f40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.008+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00d5f0 tx=0x7f46dc00dab0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 shutdown_connections 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f46d006ac00 0x7f46d006d0c0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46f415ab60 0x7f46f40b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f46f415a780 0x7f46f40b2080 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 --2- 172.21.15.28:0/2146831034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46f4071490 0x7f46f40c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 >> 172.21.15.28:0/2146831034 conn(0x7f46f40b0270 msgr2=0x7f46f40cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 shutdown_connections 2024-09-13T17:22:28.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.009+0000 7f46f9400640 1 -- 172.21.15.28:0/2146831034 wait complete. 2024-09-13T17:22:28.026 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace' 2024-09-13T17:22:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.179+0000 7f60e3000640 1 Processor -- start 2024-09-13T17:22:28.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.179+0000 7f60e3000640 1 -- start start 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.180+0000 7f60e3000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc15a780 0x7f60dc070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60e3000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60dc06c660 con 0x7f60dc15ab60 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60e3000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60dc06c8b0 con 0x7f60dc06c250 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60e3000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60dc06cb90 con 0x7f60dc15a780 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc15a780 0x7f60dc070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60dc15a780 0x7f60dc070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50756/0 (socket says 172.21.15.28:50756) 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 -- 172.21.15.28:0/3704926149 learned_addr learned my addr 172.21.15.28:0/3704926149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:28.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60d3e00640 1 --1- 172.21.15.28:0/3704926149 >> v1:172.21.15.28:6789/0 conn(0x7f60dc15ab60 0x7f60dc06bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60762/0 (socket says 172.21.15.28:60762) 2024-09-13T17:22:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 -- 172.21.15.28:0/3704926149 >> v1:172.21.15.189:6789/0 conn(0x7f60dc06c250 legacy=0x7f60dc0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 -- 172.21.15.28:0/3704926149 >> v1:172.21.15.28:6789/0 conn(0x7f60dc15ab60 legacy=0x7f60dc06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60db400640 1 -- 172.21.15.28:0/3704926149 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691908587 0 0) 0x7f60dc06c660 con 0x7f60dc15ab60 2024-09-13T17:22:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.181+0000 7f60dbe00640 1 -- 172.21.15.28:0/3704926149 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc0cc980 con 0x7f60dc15a780 2024-09-13T17:22:28.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.182+0000 7f60dbe00640 1 --2- 172.21.15.28:0/3704926149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15a780 0x7f60dc070d80 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009870 tx=0x7f60cc02a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b7f8763553d54e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:28.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60db400640 1 -- 172.21.15.28:0/3704926149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc032440 con 0x7f60dc15a780 2024-09-13T17:22:28.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60db400640 1 -- 172.21.15.28:0/3704926149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60cc0325a0 con 0x7f60dc15a780 2024-09-13T17:22:28.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60db400640 1 -- 172.21.15.28:0/3704926149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60cc032930 con 0x7f60dc15a780 2024-09-13T17:22:28.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 -- 172.21.15.28:0/3704926149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15a780 msgr2=0x7f60dc070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 --2- 172.21.15.28:0/3704926149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15a780 0x7f60dc070d80 secure :-1 s=READY pgs=2421 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009870 tx=0x7f60cc02a2a0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 -- 172.21.15.28:0/3704926149 shutdown_connections 2024-09-13T17:22:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 --2- 172.21.15.28:0/3704926149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15a780 0x7f60dc070d80 unknown :-1 s=CLOSED pgs=2421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 -- 172.21.15.28:0/3704926149 >> 172.21.15.28:0/3704926149 conn(0x7f60dc0b0270 msgr2=0x7f60dc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 -- 172.21.15.28:0/3704926149 shutdown_connections 2024-09-13T17:22:28.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.183+0000 7f60e3000640 1 -- 172.21.15.28:0/3704926149 wait complete. 2024-09-13T17:22:28.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.185+0000 7f60e3000640 1 Processor -- start 2024-09-13T17:22:28.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.185+0000 7f60e3000640 1 -- start start 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60dc06c250 0x7f60dc0b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60dc15a780 0x7f60dc0acac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc0b1bb0 con 0x7f60dc15ab60 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc0b2e40 con 0x7f60dc06c250 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e3000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60dc0b1da0 con 0x7f60dc15a780 2024-09-13T17:22:28.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:28.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50762/0 (socket says 172.21.15.28:50762) 2024-09-13T17:22:28.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60e1000640 1 -- 172.21.15.28:0/2555223901 learned_addr learned my addr 172.21.15.28:0/2555223901 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:28.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.186+0000 7f60dbe00640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60dc06c250 0x7f60dc0b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:28.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60d3e00640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60dc15a780 0x7f60dc0acac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 -- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60dc15a780 msgr2=0x7f60dc0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60dc15a780 0x7f60dc0acac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 -- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60dc06c250 msgr2=0x7f60dc0b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60dc06c250 0x7f60dc0b1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 -- 172.21.15.28:0/2555223901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc0094d0 con 0x7f60dc15ab60 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e1000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c780 tx=0x7f60d400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4004b60 con 0x7f60dc15ab60 2024-09-13T17:22:28.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d400ce90 con 0x7f60dc15ab60 2024-09-13T17:22:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d400a850 con 0x7f60dc15ab60 2024-09-13T17:22:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60dc0b2080 con 0x7f60dc15ab60 2024-09-13T17:22:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.187+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60dc0b2640 con 0x7f60dc15ab60 2024-09-13T17:22:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.188+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc0ad6c0 con 0x7f60dc15ab60 2024-09-13T17:22:28.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.189+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f60d400a9b0 con 0x7f60dc15ab60 2024-09-13T17:22:28.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.191+0000 7f60d8c00640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 0x7f60b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:28.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.191+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f60d40892c0 con 0x7f60dc15ab60 2024-09-13T17:22:28.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.191+0000 7f60dbe00640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 0x7f60b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:28.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.192+0000 7f60dbe00640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 0x7f60b806d0c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f60cc034000 tx=0x7f60cc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:28.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.195+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f60d4054700 con 0x7f60dc15ab60 2024-09-13T17:22:28.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:28.437+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace"]} v 0) v1 -- 0x7f60dc071680 con 0x7f60dc15ab60 2024-09-13T17:22:29.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.263+0000 7f60d8c00640 1 -- 172.21.15.28:0/2555223901 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace"]}]=0 v711) v1 ==== 136+0+0 (secure 0 0 0) 0x7f60d4057d20 con 0x7f60dc15ab60 2024-09-13T17:22:29.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 msgr2=0x7f60b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 0x7f60b806d0c0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f60cc034000 tx=0x7f60cc02e000 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 msgr2=0x7f60dc152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 secure :-1 s=READY pgs=2422 cs=0 l=1 rev1=1 crypto rx=0x7f60d400c780 tx=0x7f60d400cc40 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 shutdown_connections 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f60b806ac00 0x7f60b806d0c0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60dc15ab60 0x7f60dc152b00 unknown :-1 s=CLOSED pgs=2422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60dc15a780 0x7f60dc0acac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 --2- 172.21.15.28:0/2555223901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60dc06c250 0x7f60dc0b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.266+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 >> 172.21.15.28:0/2555223901 conn(0x7f60dc0b0270 msgr2=0x7f60dc158490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.267+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 shutdown_connections 2024-09-13T17:22:29.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.267+0000 7f60e3000640 1 -- 172.21.15.28:0/2555223901 wait complete. 2024-09-13T17:22:29.287 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:22:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.448+0000 7f23dfe00640 1 Processor -- start 2024-09-13T17:22:29.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.448+0000 7f23dfe00640 1 -- start start 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23d8071490 0x7f23d806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dfe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d806c8b0 con 0x7f23d815a780 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dfe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23d806cb90 con 0x7f23d8071490 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d80cc980 con 0x7f23d815ab60 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23d8071490 0x7f23d806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dde00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f23d8071490 0x7f23d806c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50770/0 (socket says 172.21.15.28:50770) 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.449+0000 7f23dde00640 1 -- 172.21.15.28:0/2859945395 learned_addr learned my addr 172.21.15.28:0/2859945395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.450+0000 7f23dca00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.450+0000 7f23dde00640 1 -- 172.21.15.28:0/2859945395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 msgr2=0x7f23d806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.450+0000 7f23dde00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d806bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.450+0000 7f23dde00640 1 -- 172.21.15.28:0/2859945395 >> v1:172.21.15.189:6789/0 conn(0x7f23d815a780 legacy=0x7f23d8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.450+0000 7f23dde00640 1 -- 172.21.15.28:0/2859945395 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d80ccc60 con 0x7f23d8071490 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.451+0000 7f23dde00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d8071490 0x7f23d806c080 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009e50 tx=0x7f23d002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a939518f41ff3a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.451+0000 7f23d7e00640 1 -- 172.21.15.28:0/2859945395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0031600 con 0x7f23d8071490 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.451+0000 7f23d7e00640 1 -- 172.21.15.28:0/2859945395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23d002d050 con 0x7f23d8071490 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.451+0000 7f23d7e00640 1 -- 172.21.15.28:0/2859945395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23d0031980 con 0x7f23d8071490 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 -- 172.21.15.28:0/2859945395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d8071490 msgr2=0x7f23d806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d8071490 0x7f23d806c080 secure :-1 s=READY pgs=2423 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009e50 tx=0x7f23d002a730 comp rx=0 tx=0).stop 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 -- 172.21.15.28:0/2859945395 shutdown_connections 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d8071490 0x7f23d806c080 unknown :-1 s=CLOSED pgs=2423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2859945395 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d806bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 -- 172.21.15.28:0/2859945395 >> 172.21.15.28:0/2859945395 conn(0x7f23d80b0270 msgr2=0x7f23d80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:29.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 -- 172.21.15.28:0/2859945395 shutdown_connections 2024-09-13T17:22:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.452+0000 7f23dfe00640 1 -- 172.21.15.28:0/2859945395 wait complete. 2024-09-13T17:22:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.453+0000 7f23dfe00640 1 Processor -- start 2024-09-13T17:22:29.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.453+0000 7f23dfe00640 1 -- start start 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 0x7f23d814e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d80ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d8150cc0 con 0x7f23d815a780 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d814fc20 con 0x7f23d8071490 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dfe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23d814ff00 con 0x7f23d815ab60 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.454+0000 7f23dca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50784/0 (socket says 172.21.15.28:50784) 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 -- 172.21.15.28:0/2443763231 learned_addr learned my addr 172.21.15.28:0/2443763231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dde00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d80ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dd400640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 0x7f23d814e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 -- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 msgr2=0x7f23d80ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d80ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 -- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 msgr2=0x7f23d814e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 0x7f23d814e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 -- 172.21.15.28:0/2443763231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0009b30 con 0x7f23d815a780 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dd400640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 0x7f23d814e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23dca00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f23c400d5d0 tx=0x7f23c400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c40125f0 con 0x7f23d815a780 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.455+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23c4012770 con 0x7f23d815a780 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.456+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23c4012b20 con 0x7f23d815a780 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.456+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d81501e0 con 0x7f23d815a780 2024-09-13T17:22:29.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.456+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d81507a0 con 0x7f23d815a780 2024-09-13T17:22:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.457+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d81509c0 con 0x7f23d815a780 2024-09-13T17:22:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.458+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f23c4012c80 con 0x7f23d815a780 2024-09-13T17:22:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.458+0000 7f23d5600640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 0x7f23b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:29.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.458+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f23c4089520 con 0x7f23d815a780 2024-09-13T17:22:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.459+0000 7f23dd400640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 0x7f23b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:29.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.459+0000 7f23dd400640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 0x7f23b806d0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f23d8150f70 tx=0x7f23cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:29.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.462+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f23c40549f0 con 0x7f23d815a780 2024-09-13T17:22:29.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.641+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f23d80cb8e0 con 0x7f23d815a780 2024-09-13T17:22:29.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.645+0000 7f23d5600640 1 -- 172.21.15.28:0/2443763231 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v48)=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7f23c4058010 con 0x7f23d815a780 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 msgr2=0x7f23b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 0x7f23b806d0c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f23d8150f70 tx=0x7f23cc009040 comp rx=0 tx=0).stop 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 msgr2=0x7f23d814cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 secure :-1 s=READY pgs=2424 cs=0 l=1 rev1=1 crypto rx=0x7f23c400d5d0 tx=0x7f23c400da90 comp rx=0 tx=0).stop 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 shutdown_connections 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f23b806ac00 0x7f23b806d0c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f23d815ab60 0x7f23d80ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f23d815a780 0x7f23d814cfd0 unknown :-1 s=CLOSED pgs=2424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 --2- 172.21.15.28:0/2443763231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23d8071490 0x7f23d814e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 >> 172.21.15.28:0/2443763231 conn(0x7f23d80b0270 msgr2=0x7f23d81585b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 shutdown_connections 2024-09-13T17:22:29.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:29.648+0000 7f23dfe00640 1 -- 172.21.15.28:0/2443763231 wait complete. 2024-09-13T17:22:29.665 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:22:29.666 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:22:29.756 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:22:29.847 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:22:29.847 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:29.847 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:29.883 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:22:29.883 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:22:29.883 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:29.921 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:22:29.921 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:29.921 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:29.957 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:22:29.958 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:22:29.958 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:29.998 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:22:29.998 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:22:29.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.151+0000 7f2ec8800640 1 Processor -- start 2024-09-13T17:22:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.151+0000 7f2ec8800640 1 -- start start 2024-09-13T17:22:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 0x7f2ec006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec8800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ec0070cc0 0x7f2ec0071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec8800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec015e100 con 0x7f2ec00cabc0 2024-09-13T17:22:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec8800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec00716b0 con 0x7f2ec0070cc0 2024-09-13T17:22:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec00adb00 con 0x7f2ec015a540 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec5e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 0x7f2ec006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec6800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ec0070cc0 0x7f2ec0071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec5e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 0x7f2ec006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:55856/0 (socket says 172.21.15.28:55856) 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec6800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2ec0070cc0 0x7f2ec0071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:42646/0 (socket says 172.21.15.28:42646) 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.153+0000 7f2ec5e00640 1 -- 172.21.15.28:0/155006215 learned_addr learned my addr 172.21.15.28:0/155006215 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.154+0000 7f2ec6800640 1 -- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 msgr2=0x7f2ec006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.154+0000 7f2ec6800640 1 --2- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 0x7f2ec006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.154+0000 7f2ec6800640 1 -- 172.21.15.28:0/155006215 >> v1:172.21.15.189:6789/0 conn(0x7f2ec00cabc0 legacy=0x7f2ec015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.154+0000 7f2ec6800640 1 -- 172.21.15.28:0/155006215 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec015e3e0 con 0x7f2ec0070cc0 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec6800640 1 --2- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec0071170 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009dc0 tx=0x7f2ebc02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32e1606f897fa638 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec4a00640 1 -- 172.21.15.28:0/155006215 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc007740 con 0x7f2ec0070cc0 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec4a00640 1 -- 172.21.15.28:0/155006215 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc02d050 con 0x7f2ec0070cc0 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec4a00640 1 -- 172.21.15.28:0/155006215 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc007ac0 con 0x7f2ec0070cc0 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec8800640 1 -- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 msgr2=0x7f2ec0071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.155+0000 7f2ec8800640 1 --2- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec0071170 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc009dc0 tx=0x7f2ebc02a5f0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 -- 172.21.15.28:0/155006215 shutdown_connections 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 --2- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec0071170 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 --2- 172.21.15.28:0/155006215 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec015a540 0x7f2ec006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 -- 172.21.15.28:0/155006215 >> 172.21.15.28:0/155006215 conn(0x7f2ec0069e80 msgr2=0x7f2ec006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:30.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 -- 172.21.15.28:0/155006215 shutdown_connections 2024-09-13T17:22:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.156+0000 7f2ec8800640 1 -- 172.21.15.28:0/155006215 wait complete. 2024-09-13T17:22:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.157+0000 7f2ec8800640 1 Processor -- start 2024-09-13T17:22:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.157+0000 7f2ec8800640 1 -- start start 2024-09-13T17:22:30.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec01440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 0x7f2ec0151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0150a60 con 0x7f2ec015a540 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec014fe50 con 0x7f2ec0070cc0 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.158+0000 7f2ec8800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0150130 con 0x7f2ec00cabc0 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50798/0 (socket says 172.21.15.28:50798) 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec6800640 1 -- 172.21.15.28:0/1013899140 learned_addr learned my addr 172.21.15.28:0/1013899140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5e00640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec01440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 0x7f2ec0151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 -- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 msgr2=0x7f2ec01440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec01440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 -- 172.21.15.28:0/1013899140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 msgr2=0x7f2ec015ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5400640 1 -- 172.21.15.28:0/1013899140 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc009a70 con 0x7f2ec00cabc0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec5e00640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec01440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.159+0000 7f2ec6800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2ec5400640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 0x7f2ec0151e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00ae250 tx=0x7f2eb000c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb00052d0 con 0x7f2ec00cabc0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eb0005430 con 0x7f2ec00cabc0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0003830 con 0x7f2ec00cabc0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec0150410 con 0x7f2ec00cabc0 2024-09-13T17:22:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.160+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec014fab0 con 0x7f2ec00cabc0 2024-09-13T17:22:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.161+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec00ce360 con 0x7f2ec00cabc0 2024-09-13T17:22:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.162+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2eb0003990 con 0x7f2ec00cabc0 2024-09-13T17:22:30.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.163+0000 7f2eba000640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 0x7f2ea406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.163+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2eb00884d0 con 0x7f2ec00cabc0 2024-09-13T17:22:30.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.163+0000 7f2ec5e00640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 0x7f2ea406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.164+0000 7f2ec5e00640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 0x7f2ea406d110 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009f20 tx=0x7f2eb4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.168+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2eb0053ad0 con 0x7f2ec00cabc0 2024-09-13T17:22:30.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.383+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ec0071170 con 0x7f2ec00cabc0 2024-09-13T17:22:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.384+0000 7f2eba000640 1 -- 172.21.15.28:0/1013899140 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+3540 (secure 0 0 0) 0x7f2eb0017df0 con 0x7f2ec00cabc0 2024-09-13T17:22:30.386 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:30.386 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":167,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8726,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/2120983097","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":2120983097},{"type":"v1","addr":"172.21.15.189:6835","nonce":2120983097}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11031,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6833/2999515596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2999515596},{"type":"v1","addr":"172.21.15.28:6833","nonce":2999515596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11043,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/348921750","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":348921750},{"type":"v1","addr":"172.21.15.189:6837","nonce":348921750}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":159},{"gid":11403,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1991103190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1991103190},{"type":"v1","addr":"172.21.15.28:6835","nonce":1991103190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":167}],"filesystems":[]} 2024-09-13T17:22:30.386 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 167 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 msgr2=0x7f2ea406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 0x7f2ea406d110 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f2eb4009f20 tx=0x7f2eb4009240 comp rx=0 tx=0).stop 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 msgr2=0x7f2ec0151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 0x7f2ec0151e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00ae250 tx=0x7f2eb000c820 comp rx=0 tx=0).stop 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 shutdown_connections 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2ea406ac50 0x7f2ea406d110 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ec015a540 0x7f2ec015ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2ec00cabc0 0x7f2ec0151e70 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 --2- 172.21.15.28:0/1013899140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ec0070cc0 0x7f2ec01440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 >> 172.21.15.28:0/1013899140 conn(0x7f2ec0069e80 msgr2=0x7f2ec00ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:30.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 shutdown_connections 2024-09-13T17:22:30.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.386+0000 7f2ec8800640 1 -- 172.21.15.28:0/1013899140 wait complete. 2024-09-13T17:22:30.404 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:22:30.404 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:22:30.405 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:22:30.405 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:22:30.405 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:22:30.405 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:22:30.405 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:22:30.405 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:22:30.406 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:22:30.406 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:22:30.406 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:22:30.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:22:30.406 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:22:30.406 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:22:30.406 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:22:30.406 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:22:30.419 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:22:30.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:22:30.420 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:22:30.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:22:30.425 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:22:30.426 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:22:30.426 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:22:30.427 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:22:30.464 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:22:30.464 DEBUG:teuthology.parallel:result is None 2024-09-13T17:22:30.466 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:22:30.466 DEBUG:teuthology.parallel:result is None 2024-09-13T17:22:30.470 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:22:30.470 DEBUG:teuthology.parallel:result is None 2024-09-13T17:22:30.472 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:22:30.472 DEBUG:teuthology.parallel:result is None 2024-09-13T17:22:30.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:22:30.594 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:22:30.594 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:22:30.604 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:22:30.604 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:22:30.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.639+0000 7fcc8aa00640 1 Processor -- start 2024-09-13T17:22:30.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.639+0000 7fcc8aa00640 1 -- start start 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc8aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc840c40a0 0x7fcc840b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc8aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fcc840b1ba0 0x7fcc8415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc8aa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840b2090 con 0x7fcc8415b0b0 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc8aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840b2370 con 0x7fcc840c40a0 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc8aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840b2650 con 0x7fcc840b1ba0 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc89600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc840c40a0 0x7fcc840b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc89600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcc840c40a0 0x7fcc840b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50856/0 (socket says 172.21.15.28:50856) 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc89600640 1 -- 172.21.15.28:0/2770738713 learned_addr learned my addr 172.21.15.28:0/2770738713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:30.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.640+0000 7fcc88c00640 1 --2- 172.21.15.28:0/2770738713 >> v2:172.21.15.189:3300/0 conn(0x7fcc840b1ba0 0x7fcc8415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.641+0000 7fcc89600640 1 -- 172.21.15.28:0/2770738713 >> v2:172.21.15.189:3300/0 conn(0x7fcc840b1ba0 msgr2=0x7fcc8415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.641+0000 7fcc89600640 1 --2- 172.21.15.28:0/2770738713 >> v2:172.21.15.189:3300/0 conn(0x7fcc840b1ba0 0x7fcc8415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.641+0000 7fcc89600640 1 -- 172.21.15.28:0/2770738713 >> v1:172.21.15.189:6789/0 conn(0x7fcc8415b0b0 legacy=0x7fcc8406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.641+0000 7fcc89600640 1 -- 172.21.15.28:0/2770738713 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840b2930 con 0x7fcc840c40a0 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.642+0000 7fcc89600640 1 --2- 172.21.15.28:0/2770738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840b1660 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fcc84144500 tx=0x7fcc7802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6b9137f867fff7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.642+0000 7fcc83e00640 1 -- 172.21.15.28:0/2770738713 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78032440 con 0x7fcc840c40a0 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.642+0000 7fcc83e00640 1 -- 172.21.15.28:0/2770738713 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc780325a0 con 0x7fcc840c40a0 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.642+0000 7fcc83e00640 1 -- 172.21.15.28:0/2770738713 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc78032930 con 0x7fcc840c40a0 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2770738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 msgr2=0x7fcc840b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2770738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840b1660 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7fcc84144500 tx=0x7fcc7802a270 comp rx=0 tx=0).stop 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2770738713 shutdown_connections 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2770738713 >> v2:172.21.15.189:3300/0 conn(0x7fcc840b1ba0 0x7fcc8415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2770738713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840b1660 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2770738713 >> 172.21.15.28:0/2770738713 conn(0x7fcc84073470 msgr2=0x7fcc84069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2770738713 shutdown_connections 2024-09-13T17:22:30.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.643+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2770738713 wait complete. 2024-09-13T17:22:30.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.645+0000 7fcc8aa00640 1 Processor -- start 2024-09-13T17:22:30.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.645+0000 7fcc8aa00640 1 -- start start 2024-09-13T17:22:30.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc840b1ba0 0x7fcc841f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc8415b0b0 0x7fcc840afcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840c7120 con 0x7fcc840c40a0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc841f29f0 con 0x7fcc840b1ba0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8413c920 con 0x7fcc8415b0b0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc88c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc88c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50870/0 (socket says 172.21.15.28:50870) 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc88c00640 1 -- 172.21.15.28:0/2955242852 learned_addr learned my addr 172.21.15.28:0/2955242852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc89600640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc840b1ba0 0x7fcc841f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.646+0000 7fcc8a000640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc8415b0b0 0x7fcc840afcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 -- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc8415b0b0 msgr2=0x7fcc840afcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc8415b0b0 0x7fcc840afcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 -- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc840b1ba0 msgr2=0x7fcc841f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc840b1ba0 0x7fcc841f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 -- 172.21.15.28:0/2955242852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc780094d0 con 0x7fcc840c40a0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc88c00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000e980 tx=0x7fcc7000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc70009670 con 0x7fcc840c40a0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7000ca00 con 0x7fcc840c40a0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc840c6230 con 0x7fcc840c40a0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.647+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000cd90 con 0x7fcc840c40a0 2024-09-13T17:22:30.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.648+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc840c67f0 con 0x7fcc840c40a0 2024-09-13T17:22:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.649+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8415ab70 con 0x7fcc840c40a0 2024-09-13T17:22:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.650+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcc70013470 con 0x7fcc840c40a0 2024-09-13T17:22:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.650+0000 7fcc81600640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 0x7fcc6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:30.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.650+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fcc700895b0 con 0x7fcc840c40a0 2024-09-13T17:22:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.651+0000 7fcc89600640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 0x7fcc6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:30.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.652+0000 7fcc89600640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 0x7fcc6406d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcc78034000 tx=0x7fcc7802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:30.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.654+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcc70054b30 con 0x7fcc840c40a0 2024-09-13T17:22:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:30.846+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcc841f5b30 con 0x7fcc840c40a0 2024-09-13T17:22:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.648+0000 7fcc81600640 1 -- 172.21.15.28:0/2955242852 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v201) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcc70058150 con 0x7fcc840c40a0 2024-09-13T17:22:31.650 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:22:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 msgr2=0x7fcc6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 0x7fcc6406d0c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcc78034000 tx=0x7fcc7802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 msgr2=0x7fcc840cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 secure :-1 s=READY pgs=2430 cs=0 l=1 rev1=1 crypto rx=0x7fcc7000e980 tx=0x7fcc7000ee40 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 shutdown_connections 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcc6406ac00 0x7fcc6406d0c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcc8415b0b0 0x7fcc840afcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcc840c40a0 0x7fcc840cca40 unknown :-1 s=CLOSED pgs=2430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 --2- 172.21.15.28:0/2955242852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc840b1ba0 0x7fcc841f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 >> 172.21.15.28:0/2955242852 conn(0x7fcc84073470 msgr2=0x7fcc840b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 shutdown_connections 2024-09-13T17:22:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.651+0000 7fcc8aa00640 1 -- 172.21.15.28:0/2955242852 wait complete. 2024-09-13T17:22:31.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:22:31.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.830+0000 7f2feae00640 1 Processor -- start 2024-09-13T17:22:31.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.831+0000 7f2feae00640 1 -- start start 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2feae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2fe415dcb0 0x7f2fe406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2feae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe415d820 con 0x7f2fe41594a0 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2feae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fe406b870 con 0x7f2fe40c5820 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2feae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe406b590 con 0x7f2fe415dcb0 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2fe9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2fe41594a0 0x7f2fe40c5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60766/0 (socket says 172.21.15.28:60766) 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2fe9a00640 1 -- 172.21.15.28:0/4034869200 learned_addr learned my addr 172.21.15.28:0/4034869200 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:31.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2fea400640 1 --2- 172.21.15.28:0/4034869200 >> v2:172.21.15.28:3300/0 conn(0x7f2fe415dcb0 0x7f2fe406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.832+0000 7f2fea400640 1 -- 172.21.15.28:0/4034869200 >> v1:172.21.15.189:6789/0 conn(0x7f2fe40c5820 legacy=0x7f2fe4159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.833+0000 7f2fea400640 1 -- 172.21.15.28:0/4034869200 >> v1:172.21.15.28:6789/0 conn(0x7f2fe41594a0 legacy=0x7f2fe40c5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.833+0000 7f2fea400640 1 -- 172.21.15.28:0/4034869200 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe415d1d0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.833+0000 7f2fea400640 1 --2- 172.21.15.28:0/4034869200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe406ae90 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c519103ece6d37 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:31.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2fe3e00640 1 -- 172.21.15.28:0/4034869200 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007740 con 0x7f2fe415dcb0 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2fe3e00640 1 -- 172.21.15.28:0/4034869200 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc02d050 con 0x7f2fe415dcb0 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2fe3e00640 1 -- 172.21.15.28:0/4034869200 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc007ac0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 -- 172.21.15.28:0/4034869200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 msgr2=0x7f2fe406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 --2- 172.21.15.28:0/4034869200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe406ae90 secure :-1 s=READY pgs=2432 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc02a6a0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 -- 172.21.15.28:0/4034869200 shutdown_connections 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 --2- 172.21.15.28:0/4034869200 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe406ae90 unknown :-1 s=CLOSED pgs=2432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 -- 172.21.15.28:0/4034869200 >> 172.21.15.28:0/4034869200 conn(0x7f2fe40703a0 msgr2=0x7f2fe40707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 -- 172.21.15.28:0/4034869200 shutdown_connections 2024-09-13T17:22:31.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.834+0000 7f2feae00640 1 -- 172.21.15.28:0/4034869200 wait complete. 2024-09-13T17:22:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.836+0000 7f2feae00640 1 Processor -- start 2024-09-13T17:22:31.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.836+0000 7f2feae00640 1 -- start start 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe40c5820 0x7f2fe40c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2fe41594a0 0x7f2fe41f01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe41f06f0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe41558f0 con 0x7f2fe40c5820 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2feae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe40c7bc0 con 0x7f2fe41594a0 2024-09-13T17:22:31.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2fea400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2fea400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50888/0 (socket says 172.21.15.28:50888) 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.837+0000 7f2fea400640 1 -- 172.21.15.28:0/1795239185 learned_addr learned my addr 172.21.15.28:0/1795239185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fe9a00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe40c5820 0x7f2fe40c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fe9000640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2fe41594a0 0x7f2fe41f01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 -- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2fe41594a0 msgr2=0x7f2fe41f01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2fe41594a0 0x7f2fe41f01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 -- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe40c5820 msgr2=0x7f2fe40c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe40c5820 0x7f2fe40c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 -- 172.21.15.28:0/1795239185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc009a70 con 0x7f2fe415dcb0 2024-09-13T17:22:31.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.838+0000 7f2fea400640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.839+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc031910 con 0x7f2fe415dcb0 2024-09-13T17:22:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.839+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fdc031a70 con 0x7f2fe415dcb0 2024-09-13T17:22:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.839+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc031de0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.839+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe40c7ea0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.839+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe40c8460 con 0x7f2fe415dcb0 2024-09-13T17:22:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.840+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe406ff30 con 0x7f2fe415dcb0 2024-09-13T17:22:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.841+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2fdc03b410 con 0x7f2fe415dcb0 2024-09-13T17:22:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.841+0000 7f2fe1600640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 0x7f2fc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:31.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.842+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2fdc0aa4e0 con 0x7f2fe415dcb0 2024-09-13T17:22:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.842+0000 7f2fe9a00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 0x7f2fc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:31.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.843+0000 7f2fe9a00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 0x7f2fc406d0c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8005880 tx=0x7f2fd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:31.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:31.846+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2fdc075ad0 con 0x7f2fe415dcb0 2024-09-13T17:22:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.090+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f2fe4155a60 con 0x7f2fe415dcb0 2024-09-13T17:22:32.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.091+0000 7f2fe1600640 1 -- 172.21.15.28:0/1795239185 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v26) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f2fdc0790f0 con 0x7f2fe415dcb0 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:22:32.093 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:22:32.094 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:22:32.096 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.097 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:22:32.098 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.099 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:22:32.100 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.101 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:22:32.102 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:22:32.103 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:22:32.104 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:22:32.106 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 msgr2=0x7f2fc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 0x7f2fc406d0c0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8005880 tx=0x7f2fd8009040 comp rx=0 tx=0).stop 2024-09-13T17:22:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 msgr2=0x7f2fe40c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 secure :-1 s=READY pgs=2433 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009d90 tx=0x7f2fdc004a40 comp rx=0 tx=0).stop 2024-09-13T17:22:32.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 shutdown_connections 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fc406ac00 0x7f2fc406d0c0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2fe415dcb0 0x7f2fe40c92d0 unknown :-1 s=CLOSED pgs=2433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2fe41594a0 0x7f2fe41f01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 --2- 172.21.15.28:0/1795239185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe40c5820 0x7f2fe40c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 >> 172.21.15.28:0/1795239185 conn(0x7f2fe40703a0 msgr2=0x7f2fe4159fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 shutdown_connections 2024-09-13T17:22:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.093+0000 7f2feae00640 1 -- 172.21.15.28:0/1795239185 wait complete. 2024-09-13T17:22:32.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.265+0000 7f8e55000640 1 Processor -- start 2024-09-13T17:22:32.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.265+0000 7f8e55000640 1 -- start start 2024-09-13T17:22:32.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.266+0000 7f8e55000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e50160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.266+0000 7f8e55000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e50146550 con 0x7f8e500ac850 2024-09-13T17:22:32.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.266+0000 7f8e55000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5015d280 con 0x7f8e50146170 2024-09-13T17:22:32.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.266+0000 7f8e55000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5015cd20 con 0x7f8e50160310 2024-09-13T17:22:32.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e500ac850 0x7f8e5006b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60770/0 (socket says 172.21.15.28:60770) 2024-09-13T17:22:32.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4e800640 1 -- 172.21.15.28:0/3183844288 learned_addr learned my addr 172.21.15.28:0/3183844288 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:32.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4d400640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e50160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816914162 0 0) 0x7f8e50146550 con 0x7f8e500ac850 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e34003520 con 0x7f8e500ac850 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2761439144 0 0) 0x7f8e34003520 con 0x7f8e500ac850 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 msgr2=0x7f8e50160730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e50160730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 >> v1:172.21.15.189:6789/0 conn(0x7f8e50146170 legacy=0x7f8e5015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e50160d60 con 0x7f8e500ac850 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.267+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f8e480019f0 con 0x7f8e500ac850 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 >> v1:172.21.15.28:6789/0 conn(0x7f8e500ac850 legacy=0x7f8e5006b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e340039a0 0x7f8e34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e34004d00 0x7f8e34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3400ba30 con 0x7f8e340039a0 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3400bba0 con 0x7f8e34004d00 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3400bd60 con 0x7f8e50160310 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e340039a0 0x7f8e34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4d400640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e34004d00 0x7f8e34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 -- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 msgr2=0x7f8e340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e50160310 0x7f8e340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 -- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e34004d00 msgr2=0x7f8e34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e34004d00 0x7f8e34005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4de00640 1 -- 172.21.15.28:0/3183844288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34003520 con 0x7f8e340039a0 2024-09-13T17:22:32.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.268+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e34004d00 0x7f8e34005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:32.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.269+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e340039a0 0x7f8e34003dc0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f8e40001720 tx=0x7f8e340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af4a3f23a9c82bc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.272+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e400052d0 con 0x7f8e340039a0 2024-09-13T17:22:32.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.272+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e40005430 con 0x7f8e340039a0 2024-09-13T17:22:32.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.272+0000 7f8e4ca00640 1 -- 172.21.15.28:0/3183844288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4002abc0 con 0x7f8e340039a0 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 -- 172.21.15.28:0/3183844288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e340039a0 msgr2=0x7f8e34003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 --2- 172.21.15.28:0/3183844288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e340039a0 0x7f8e34003dc0 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f8e40001720 tx=0x7f8e340057f0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e4e800640 1 -- 172.21.15.28:0/3183844288 reap_dead start 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 -- 172.21.15.28:0/3183844288 shutdown_connections 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 -- 172.21.15.28:0/3183844288 >> 172.21.15.28:0/3183844288 conn(0x7f8e50069e80 msgr2=0x7f8e5006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 -- 172.21.15.28:0/3183844288 shutdown_connections 2024-09-13T17:22:32.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.273+0000 7f8e55000640 1 -- 172.21.15.28:0/3183844288 wait complete. 2024-09-13T17:22:32.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.275+0000 7f8e55000640 1 Processor -- start 2024-09-13T17:22:32.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.275+0000 7f8e55000640 1 -- start start 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e340039a0 0x7f8e5013de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e500b0e00 0x7f8e5013d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500aeb10 con 0x7f8e500af210 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5013d840 con 0x7f8e500b0e00 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e55000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5013dbf0 con 0x7f8e340039a0 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50898/0 (socket says 172.21.15.28:50898) 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e4e800640 1 -- 172.21.15.28:0/3835921625 learned_addr learned my addr 172.21.15.28:0/3835921625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e340039a0 0x7f8e5013de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.276+0000 7f8e4d400640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e500b0e00 0x7f8e5013d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 -- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e340039a0 msgr2=0x7f8e5013de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e340039a0 0x7f8e5013de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 -- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e500b0e00 msgr2=0x7f8e5013d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e500b0e00 0x7f8e5013d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 -- 172.21.15.28:0/3835921625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5013fb90 con 0x7f8e500af210 2024-09-13T17:22:32.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8e5006d4b0 tx=0x7f8e4800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800ec10 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4800ed70 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e500aecf0 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e48013660 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.277+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e501e7050 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.278+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5013fd80 con 0x7f8e500af210 2024-09-13T17:22:32.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.279+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e480137c0 con 0x7f8e500af210 2024-09-13T17:22:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.280+0000 7f8e46000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 0x7f8e3006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.280+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8e48089310 con 0x7f8e500af210 2024-09-13T17:22:32.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.281+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 0x7f8e3006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.281+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 0x7f8e3006d0f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f8e40032000 tx=0x7f8e4002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.284+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e48054830 con 0x7f8e500af210 2024-09-13T17:22:32.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.462+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e500aeee0 con 0x7f8e500af210 2024-09-13T17:22:32.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.463+0000 7f8e46000640 1 -- 172.21.15.28:0/3835921625 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f8e48057e50 con 0x7f8e500af210 2024-09-13T17:22:32.465 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:32.465 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":201,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:31.642971+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 msgr2=0x7f8e3006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 0x7f8e3006d0f0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7f8e40032000 tx=0x7f8e4002e000 comp rx=0 tx=0).stop 2024-09-13T17:22:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 msgr2=0x7f8e500af690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 secure :-1 s=READY pgs=2436 cs=0 l=1 rev1=1 crypto rx=0x7f8e5006d4b0 tx=0x7f8e4800e530 comp rx=0 tx=0).stop 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 shutdown_connections 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e3006ac30 0x7f8e3006d0f0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e500af210 0x7f8e500af690 unknown :-1 s=CLOSED pgs=2436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e500b0e00 0x7f8e5013d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 --2- 172.21.15.28:0/3835921625 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e340039a0 0x7f8e5013de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 >> 172.21.15.28:0/3835921625 conn(0x7f8e50069e80 msgr2=0x7f8e5015ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 shutdown_connections 2024-09-13T17:22:32.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.465+0000 7f8e55000640 1 -- 172.21.15.28:0/3835921625 wait complete. 2024-09-13T17:22:32.483 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:22:32.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:22:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.641+0000 7f7caca00640 1 Processor -- start 2024-09-13T17:22:32.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.641+0000 7f7caca00640 1 -- start start 2024-09-13T17:22:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.642+0000 7f7caca00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca815ab60 0x7f7ca806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.642+0000 7f7caca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca8071490 0x7f7ca806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.642+0000 7f7caca00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca806c8b0 con 0x7f7ca815a780 2024-09-13T17:22:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.642+0000 7f7caca00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca806cb90 con 0x7f7ca815ab60 2024-09-13T17:22:32.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.642+0000 7f7caca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca80cc980 con 0x7f7ca8071490 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca815ab60 0x7f7ca806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7ca6800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca8071490 0x7f7ca806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7ca815ab60 0x7f7ca806bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:42780/0 (socket says 172.21.15.28:42780) 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 -- 172.21.15.28:0/255678436 learned_addr learned my addr 172.21.15.28:0/255678436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7ca5e00640 1 --1- 172.21.15.28:0/255678436 >> v1:172.21.15.189:6789/0 conn(0x7f7ca815a780 0x7f7ca8070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:54744/0 (socket says 172.21.15.28:54744) 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 -- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca8071490 msgr2=0x7f7ca806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:32.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 --2- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca8071490 0x7f7ca806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 -- 172.21.15.28:0/255678436 >> v1:172.21.15.189:6789/0 conn(0x7f7ca815a780 legacy=0x7f7ca8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.643+0000 7f7c9d400640 1 -- 172.21.15.28:0/255678436 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca80ccc60 con 0x7f7ca815ab60 2024-09-13T17:22:32.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.644+0000 7f7c9d400640 1 --2- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca806bb40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009870 tx=0x7f7c9002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55c3067996b16a30 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7ca5400640 1 -- 172.21.15.28:0/255678436 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90032440 con 0x7f7ca815ab60 2024-09-13T17:22:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7ca5400640 1 -- 172.21.15.28:0/255678436 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c900325a0 con 0x7f7ca815ab60 2024-09-13T17:22:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7ca5400640 1 -- 172.21.15.28:0/255678436 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c90032930 con 0x7f7ca815ab60 2024-09-13T17:22:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 -- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 msgr2=0x7f7ca806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 --2- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca806bb40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f7c90009870 tx=0x7f7c9002a270 comp rx=0 tx=0).stop 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 -- 172.21.15.28:0/255678436 shutdown_connections 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 --2- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca8071490 0x7f7ca806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 --2- 172.21.15.28:0/255678436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca806bb40 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 -- 172.21.15.28:0/255678436 >> 172.21.15.28:0/255678436 conn(0x7f7ca80b0270 msgr2=0x7f7ca80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 -- 172.21.15.28:0/255678436 shutdown_connections 2024-09-13T17:22:32.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.645+0000 7f7caca00640 1 -- 172.21.15.28:0/255678436 wait complete. 2024-09-13T17:22:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.647+0000 7f7caca00640 1 Processor -- start 2024-09-13T17:22:32.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.647+0000 7f7caca00640 1 -- start start 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca815a780 0x7f7ca80ca1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca8156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca8157430 con 0x7f7ca8071490 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca814e3f0 con 0x7f7ca815ab60 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7caca00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca80c91a0 con 0x7f7ca815a780 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7ca5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7ca5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50906/0 (socket says 172.21.15.28:50906) 2024-09-13T17:22:32.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.648+0000 7f7ca5e00640 1 -- 172.21.15.28:0/2698306471 learned_addr learned my addr 172.21.15.28:0/2698306471 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7c9d400640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca815a780 0x7f7ca80ca1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca6800640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca8156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 -- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca815a780 msgr2=0x7f7ca80ca1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca815a780 0x7f7ca80ca1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 -- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 msgr2=0x7f7ca8156850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca8156850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 -- 172.21.15.28:0/2698306471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c900094d0 con 0x7f7ca8071490 2024-09-13T17:22:32.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7ca5e00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800c4d0 tx=0x7f7c9800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.649+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98007480 con 0x7f7ca8071490 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.650+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c980075e0 con 0x7f7ca8071490 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.650+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980049e0 con 0x7f7ca8071490 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.650+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca80c9440 con 0x7f7ca8071490 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.650+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca80c99c0 con 0x7f7ca8071490 2024-09-13T17:22:32.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.651+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca8070d80 con 0x7f7ca8071490 2024-09-13T17:22:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.652+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7c98004b40 con 0x7f7ca8071490 2024-09-13T17:22:32.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.653+0000 7f7c9ea00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 0x7f7c8406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:32.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.653+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7c98088120 con 0x7f7ca8071490 2024-09-13T17:22:32.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.653+0000 7f7c9d400640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 0x7f7c8406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:32.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.654+0000 7f7c9d400640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 0x7f7c8406d0f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7c90034000 tx=0x7f7c9002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:32.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.657+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7c98053660 con 0x7f7ca8071490 2024-09-13T17:22:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.836+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ca80cbc30 con 0x7f7ca8071490 2024-09-13T17:22:32.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.836+0000 7f7c9ea00640 1 -- 172.21.15.28:0/2698306471 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7c98056c80 con 0x7f7ca8071490 2024-09-13T17:22:32.838 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:22:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 msgr2=0x7f7c8406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 0x7f7c8406d0f0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f7c90034000 tx=0x7f7c9002e000 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 msgr2=0x7f7ca814ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 secure :-1 s=READY pgs=2437 cs=0 l=1 rev1=1 crypto rx=0x7f7c9800c4d0 tx=0x7f7c9800c990 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 shutdown_connections 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7c8406ac30 0x7f7c8406d0f0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ca815ab60 0x7f7ca8156850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ca815a780 0x7f7ca80ca1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 --2- 172.21.15.28:0/2698306471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ca8071490 0x7f7ca814ab60 unknown :-1 s=CLOSED pgs=2437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 >> 172.21.15.28:0/2698306471 conn(0x7f7ca80b0270 msgr2=0x7f7ca80cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 shutdown_connections 2024-09-13T17:22:32.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:32.838+0000 7f7caca00640 1 -- 172.21.15.28:0/2698306471 wait complete. 2024-09-13T17:22:32.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:22:33.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.010+0000 7fad02600640 1 Processor -- start 2024-09-13T17:22:33.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.010+0000 7fad02600640 1 -- start start 2024-09-13T17:22:33.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 0x7facfc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facfc0add00 0x7facfc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc06f920 con 0x7facfc0add00 2024-09-13T17:22:33.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facfc15b290 con 0x7facfc159770 2024-09-13T17:22:33.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7fad02600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc15b570 con 0x7facfc06f3f0 2024-09-13T17:22:33.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7facfbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facfc0add00 0x7facfc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7facfbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7facfc0add00 0x7facfc159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50910/0 (socket says 172.21.15.28:50910) 2024-09-13T17:22:33.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.011+0000 7facfbe00640 1 -- 172.21.15.28:0/3459035321 learned_addr learned my addr 172.21.15.28:0/3459035321 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfaa00640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfb400640 1 --2- 172.21.15.28:0/3459035321 >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 0x7facfc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfbe00640 1 -- 172.21.15.28:0/3459035321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 msgr2=0x7facfc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfbe00640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfbe00640 1 -- 172.21.15.28:0/3459035321 >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 msgr2=0x7facfc06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfbe00640 1 --2- 172.21.15.28:0/3459035321 >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 0x7facfc06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfbe00640 1 -- 172.21.15.28:0/3459035321 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facfc15b850 con 0x7facfc0add00 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfb400640 1 --2- 172.21.15.28:0/3459035321 >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 0x7facfc06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.012+0000 7facfaa00640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.013+0000 7facfbe00640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc159b50 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7facfc068040 tx=0x7facf402a600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8516b9ef19f49cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.013+0000 7facfa000640 1 -- 172.21.15.28:0/3459035321 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4031600 con 0x7facfc0add00 2024-09-13T17:22:33.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.013+0000 7facfa000640 1 -- 172.21.15.28:0/3459035321 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf402d050 con 0x7facfc0add00 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.013+0000 7facfa000640 1 -- 172.21.15.28:0/3459035321 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf4031980 con 0x7facfc0add00 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 -- 172.21.15.28:0/3459035321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 msgr2=0x7facfc159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc159b50 secure :-1 s=READY pgs=2438 cs=0 l=1 rev1=1 crypto rx=0x7facfc068040 tx=0x7facf402a600 comp rx=0 tx=0).stop 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 -- 172.21.15.28:0/3459035321 shutdown_connections 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc159b50 unknown :-1 s=CLOSED pgs=2438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 --2- 172.21.15.28:0/3459035321 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 --2- 172.21.15.28:0/3459035321 >> v2:172.21.15.189:3300/0 conn(0x7facfc159770 0x7facfc06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 -- 172.21.15.28:0/3459035321 >> 172.21.15.28:0/3459035321 conn(0x7facfc0b0270 msgr2=0x7facfc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 -- 172.21.15.28:0/3459035321 shutdown_connections 2024-09-13T17:22:33.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.014+0000 7fad02600640 1 -- 172.21.15.28:0/3459035321 wait complete. 2024-09-13T17:22:33.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.016+0000 7fad02600640 1 Processor -- start 2024-09-13T17:22:33.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.016+0000 7fad02600640 1 -- start start 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc1e1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facfc159770 0x7facfc157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1e78c0 con 0x7facfc0add00 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1e1a40 con 0x7facfc159770 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7fad02600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facfc1e6d60 con 0x7facfc06f3f0 2024-09-13T17:22:33.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50922/0 (socket says 172.21.15.28:50922) 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 -- 172.21.15.28:0/2619600115 learned_addr learned my addr 172.21.15.28:0/2619600115 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfb400640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc1e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfbe00640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facfc159770 0x7facfc157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 -- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 msgr2=0x7facfc1e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc1e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 -- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facfc159770 msgr2=0x7facfc157c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facfc159770 0x7facfc157c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.017+0000 7facfaa00640 1 -- 172.21.15.28:0/2619600115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf4009b30 con 0x7facfc0add00 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7facfaa00640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7face400c820 tx=0x7face400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face4007a30 con 0x7facfc0add00 2024-09-13T17:22:33.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face4007b90 con 0x7facfc0add00 2024-09-13T17:22:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face400ce70 con 0x7facfc0add00 2024-09-13T17:22:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facfc1e7020 con 0x7facfc0add00 2024-09-13T17:22:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.018+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facfc1e75e0 con 0x7facfc0add00 2024-09-13T17:22:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.019+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facfc159b50 con 0x7facfc0add00 2024-09-13T17:22:33.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.020+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7face40113f0 con 0x7facfc0add00 2024-09-13T17:22:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.021+0000 7facf3400640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 0x7facd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.022+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 4857+0+0 (secure 0 0 0) 0x7face4089120 con 0x7facfc0add00 2024-09-13T17:22:33.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.022+0000 7facfb400640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 0x7facd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.023+0000 7facfb400640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 0x7facd806d0c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7facec00aa40 tx=0x7facec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.026+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7face40546f0 con 0x7facfc0add00 2024-09-13T17:22:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.209+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7facfc0ad0b0 con 0x7facfc0add00 2024-09-13T17:22:33.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.209+0000 7facf3400640 1 -- 172.21.15.28:0/2619600115 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v201) v1 ==== 84+0+65 (secure 0 0 0) 0x7face401f060 con 0x7facfc0add00 2024-09-13T17:22:33.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:33.212 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 msgr2=0x7facd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 0x7facd806d0c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7facec00aa40 tx=0x7facec008040 comp rx=0 tx=0).stop 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 msgr2=0x7facfc1e6820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7face400c820 tx=0x7face400cc20 comp rx=0 tx=0).stop 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 shutdown_connections 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7facd806ac00 0x7facd806d0c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7facfc159770 0x7facfc157c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7facfc0add00 0x7facfc1e6820 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 --2- 172.21.15.28:0/2619600115 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7facfc06f3f0 0x7facfc1e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 >> 172.21.15.28:0/2619600115 conn(0x7facfc0b0270 msgr2=0x7facfc070750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 shutdown_connections 2024-09-13T17:22:33.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.212+0000 7fad02600640 1 -- 172.21.15.28:0/2619600115 wait complete. 2024-09-13T17:22:33.229 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:22:33.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:22:33.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.390+0000 7f08a3200640 1 Processor -- start 2024-09-13T17:22:33.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.391+0000 7f08a3200640 1 -- start start 2024-09-13T17:22:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a3200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f08a406b9c0 0x7f08a40b2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08a40b2dd0 0x7f08a4159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a40b3600 con 0x7f08a4159440 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a40b46a0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a3200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a40b38e0 con 0x7f08a406b9c0 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a2800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f08a40b2dd0 0x7f08a4159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a1e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f08a4159440 0x7f08a406b2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:60784/0 (socket says 172.21.15.28:60784) 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a1e00640 1 -- 172.21.15.28:0/2532338455 learned_addr learned my addr 172.21.15.28:0/2532338455 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f089be00640 1 --2- 172.21.15.28:0/2532338455 >> v2:172.21.15.189:3300/0 conn(0x7f08a406b9c0 0x7f08a40b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.392+0000 7f08a1400640 1 -- 172.21.15.28:0/2532338455 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588192115 0 0) 0x7f08a40b3600 con 0x7f08a4159440 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.393+0000 7f08a2800640 1 -- 172.21.15.28:0/2532338455 >> v2:172.21.15.189:3300/0 conn(0x7f08a406b9c0 msgr2=0x7f08a40b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.393+0000 7f08a2800640 1 --2- 172.21.15.28:0/2532338455 >> v2:172.21.15.189:3300/0 conn(0x7f08a406b9c0 0x7f08a40b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.393+0000 7f08a2800640 1 -- 172.21.15.28:0/2532338455 >> v1:172.21.15.28:6789/0 conn(0x7f08a4159440 legacy=0x7f08a406b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.393+0000 7f08a2800640 1 -- 172.21.15.28:0/2532338455 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a40b3bc0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a2800640 1 --2- 172.21.15.28:0/2532338455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a4159820 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f08a406acd0 tx=0x7f089c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7bc51fcfa7b361 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a1400640 1 -- 172.21.15.28:0/2532338455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c02aea0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a1400640 1 -- 172.21.15.28:0/2532338455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f089c02d050 con 0x7f08a40b2dd0 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a1400640 1 -- 172.21.15.28:0/2532338455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089c007960 con 0x7f08a40b2dd0 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 -- 172.21.15.28:0/2532338455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 msgr2=0x7f08a4159820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 --2- 172.21.15.28:0/2532338455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a4159820 secure :-1 s=READY pgs=2441 cs=0 l=1 rev1=1 crypto rx=0x7f08a406acd0 tx=0x7f089c02a680 comp rx=0 tx=0).stop 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 -- 172.21.15.28:0/2532338455 shutdown_connections 2024-09-13T17:22:33.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 --2- 172.21.15.28:0/2532338455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a4159820 unknown :-1 s=CLOSED pgs=2441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 --2- 172.21.15.28:0/2532338455 >> v2:172.21.15.189:3300/0 conn(0x7f08a406b9c0 0x7f08a40b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 -- 172.21.15.28:0/2532338455 >> 172.21.15.28:0/2532338455 conn(0x7f08a4069e80 msgr2=0x7f08a406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.394+0000 7f08a3200640 1 -- 172.21.15.28:0/2532338455 shutdown_connections 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.395+0000 7f08a3200640 1 -- 172.21.15.28:0/2532338455 wait complete. 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.396+0000 7f08a3200640 1 Processor -- start 2024-09-13T17:22:33.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.396+0000 7f08a3200640 1 -- start start 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 0x7f08a413ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 0x7f08a406f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a41504c0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08a414f840 con 0x7f08a406b9c0 2024-09-13T17:22:33.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a414fb20 con 0x7f08a4159440 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f089be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f089be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50948/0 (socket says 172.21.15.28:50948) 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f089be00640 1 -- 172.21.15.28:0/1871407965 learned_addr learned my addr 172.21.15.28:0/1871407965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.397+0000 7f08a1e00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 0x7f08a413ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f08a2800640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 0x7f08a406f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 -- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 msgr2=0x7f08a406f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 0x7f08a406f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 -- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 msgr2=0x7f08a413ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 0x7f08a413ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 -- 172.21.15.28:0/1871407965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089c009a70 con 0x7f08a40b2dd0 2024-09-13T17:22:33.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f08a1e00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 0x7f08a413ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f08a2800640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 0x7f08a406f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089be00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d800 tx=0x7f088c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00b6c0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f088c00b820 con 0x7f08a40b2dd0 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088c00bbb0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a414fe00 con 0x7f08a40b2dd0 2024-09-13T17:22:33.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.398+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a414f4d0 con 0x7f08a40b2dd0 2024-09-13T17:22:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.399+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a4150730 con 0x7f08a40b2dd0 2024-09-13T17:22:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.400+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f088c00bd10 con 0x7f08a40b2dd0 2024-09-13T17:22:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.401+0000 7f089a000640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 0x7f088006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:33.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.401+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f088c089560 con 0x7f08a40b2dd0 2024-09-13T17:22:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.402+0000 7f08a1e00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 0x7f088006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:33.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.402+0000 7f08a1e00640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 0x7f088006d0c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0894000f60 tx=0x7f0894007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:33.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.405+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f088c054a80 con 0x7f08a40b2dd0 2024-09-13T17:22:33.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:33.584+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f08a4067a10 con 0x7f08a40b2dd0 2024-09-13T17:22:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.285+0000 7f089a000640 1 -- 172.21.15.28:0/1871407965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v202) v1 ==== 138+0+0 (secure 0 0 0) 0x7f088c0580a0 con 0x7f08a40b2dd0 2024-09-13T17:22:34.286 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:22:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 msgr2=0x7f088006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 0x7f088006d0c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f0894000f60 tx=0x7f0894007ca0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 msgr2=0x7f08a414e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:34.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 secure :-1 s=READY pgs=2442 cs=0 l=1 rev1=1 crypto rx=0x7f088c00d800 tx=0x7f088c00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 shutdown_connections 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f088006ac00 0x7f088006d0c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f08a4159440 0x7f08a406f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f08a40b2dd0 0x7f08a414e4f0 unknown :-1 s=CLOSED pgs=2442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.290+0000 7f08a3200640 1 --2- 172.21.15.28:0/1871407965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08a406b9c0 0x7f08a413ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.291+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 >> 172.21.15.28:0/1871407965 conn(0x7f08a4069e80 msgr2=0x7f08a4157a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.292+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 shutdown_connections 2024-09-13T17:22:34.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.293+0000 7f08a3200640 1 -- 172.21.15.28:0/1871407965 wait complete. 2024-09-13T17:22:34.315 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:22:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.480+0000 7f121a800640 1 Processor -- start 2024-09-13T17:22:34.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.481+0000 7f121a800640 1 -- start start 2024-09-13T17:22:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f121a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f121415a780 0x7f1214070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f121a800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f121415ab60 0x7f12140712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f121a800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f121406c660 con 0x7f121406bb40 2024-09-13T17:22:34.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f121a800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121406c8b0 con 0x7f121415a780 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f121a800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f121406cb90 con 0x7f121415ab60 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f1213400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f121415a780 0x7f1214070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f1213400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f121415a780 0x7f1214070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52774/0 (socket says 172.21.15.28:52774) 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f1213400640 1 -- 172.21.15.28:0/650074028 learned_addr learned my addr 172.21.15.28:0/650074028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.482+0000 7f120aa00640 1 --2- 172.21.15.28:0/650074028 >> v2:172.21.15.189:3300/0 conn(0x7f121415ab60 0x7f12140712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.483+0000 7f1213400640 1 -- 172.21.15.28:0/650074028 >> v2:172.21.15.189:3300/0 conn(0x7f121415ab60 msgr2=0x7f12140712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.483+0000 7f1213400640 1 --2- 172.21.15.28:0/650074028 >> v2:172.21.15.189:3300/0 conn(0x7f121415ab60 0x7f12140712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.483+0000 7f1213400640 1 -- 172.21.15.28:0/650074028 >> v1:172.21.15.189:6789/0 conn(0x7f121406bb40 legacy=0x7f121406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:34.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.483+0000 7f1213400640 1 -- 172.21.15.28:0/650074028 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140cc980 con 0x7f121415a780 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f1213400640 1 --2- 172.21.15.28:0/650074028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415a780 0x7f1214070d80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f121406f380 tx=0x7f120402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14982e2e539f7bb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f1212a00640 1 -- 172.21.15.28:0/650074028 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204032440 con 0x7f121415a780 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f1212a00640 1 -- 172.21.15.28:0/650074028 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12040325a0 con 0x7f121415a780 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f1212a00640 1 -- 172.21.15.28:0/650074028 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1204032930 con 0x7f121415a780 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f121a800640 1 -- 172.21.15.28:0/650074028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415a780 msgr2=0x7f1214070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:34.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.484+0000 7f121a800640 1 --2- 172.21.15.28:0/650074028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415a780 0x7f1214070d80 secure :-1 s=READY pgs=2443 cs=0 l=1 rev1=1 crypto rx=0x7f121406f380 tx=0x7f120402a230 comp rx=0 tx=0).stop 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 -- 172.21.15.28:0/650074028 shutdown_connections 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 --2- 172.21.15.28:0/650074028 >> v2:172.21.15.189:3300/0 conn(0x7f121415ab60 0x7f12140712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 --2- 172.21.15.28:0/650074028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415a780 0x7f1214070d80 unknown :-1 s=CLOSED pgs=2443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 -- 172.21.15.28:0/650074028 >> 172.21.15.28:0/650074028 conn(0x7f12140b0270 msgr2=0x7f12140b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 -- 172.21.15.28:0/650074028 shutdown_connections 2024-09-13T17:22:34.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.485+0000 7f121a800640 1 -- 172.21.15.28:0/650074028 wait complete. 2024-09-13T17:22:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.486+0000 7f121a800640 1 Processor -- start 2024-09-13T17:22:34.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.486+0000 7f121a800640 1 -- start start 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f121406bb40 0x7f121414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121415a780 0x7f12141556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140b2b30 con 0x7f121415ab60 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140b1eb0 con 0x7f121415a780 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f121a800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12140b2190 con 0x7f121406bb40 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.487+0000 7f1213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52780/0 (socket says 172.21.15.28:52780) 2024-09-13T17:22:34.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 -- 172.21.15.28:0/3333366690 learned_addr learned my addr 172.21.15.28:0/3333366690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f120aa00640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121415a780 0x7f12141556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213400640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f121406bb40 0x7f121414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 -- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f121406bb40 msgr2=0x7f121414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f121406bb40 0x7f121414ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 -- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121415a780 msgr2=0x7f12141556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121415a780 0x7f12141556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 -- 172.21.15.28:0/3333366690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040094d0 con 0x7f121415ab60 2024-09-13T17:22:34.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.488+0000 7f1213e00640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c780 tx=0x7f120c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.489+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c004b60 con 0x7f121415ab60 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.489+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00ce90 con 0x7f121415ab60 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.489+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c00a850 con 0x7f121415ab60 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.489+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12140b2470 con 0x7f121415ab60 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.489+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f121414b3c0 con 0x7f121415ab60 2024-09-13T17:22:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.490+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12140b2c70 con 0x7f121415ab60 2024-09-13T17:22:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.491+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f120c00a9b0 con 0x7f121415ab60 2024-09-13T17:22:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.492+0000 7f120be00640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 0x7f11f006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:34.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.492+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(202..202 src has 1..202) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f120c089210 con 0x7f121415ab60 2024-09-13T17:22:34.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.492+0000 7f1213400640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 0x7f11f006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.493+0000 7f1213400640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 0x7f11f006d0f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1204034000 tx=0x7f120402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:34.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.497+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f12140b2c70 con 0x7f121415ab60 2024-09-13T17:22:34.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:34.676+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f12140b2c70 con 0x7f121415ab60 2024-09-13T17:22:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.288+0000 7f120be00640 1 -- 172.21.15.28:0/3333366690 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v203) v1 ==== 170+0+0 (secure 0 0 0) 0x7f12140b2c70 con 0x7f121415ab60 2024-09-13T17:22:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:22:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.296+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 msgr2=0x7f11f006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.296+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 0x7f11f006d0f0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1204034000 tx=0x7f120402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.296+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 msgr2=0x7f12140b2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.296+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 secure :-1 s=READY pgs=2444 cs=0 l=1 rev1=1 crypto rx=0x7f120c00c780 tx=0x7f120c00cc40 comp rx=0 tx=0).stop 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 shutdown_connections 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f11f006ac30 0x7f11f006d0f0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f121415ab60 0x7f12140b2f50 unknown :-1 s=CLOSED pgs=2444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f121415a780 0x7f12141556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 --2- 172.21.15.28:0/3333366690 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f121406bb40 0x7f121414ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.297+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 >> 172.21.15.28:0/3333366690 conn(0x7f12140b0270 msgr2=0x7f12140ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.300+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 shutdown_connections 2024-09-13T17:22:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.300+0000 7f121a800640 1 -- 172.21.15.28:0/3333366690 wait complete. 2024-09-13T17:22:35.325 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:22:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.485+0000 7f4d82e00640 1 Processor -- start 2024-09-13T17:22:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.485+0000 7f4d82e00640 1 -- start start 2024-09-13T17:22:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d82e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 0x7f4d7c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d82e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7c06c8b0 con 0x7f4d7c071490 2024-09-13T17:22:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d82e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7c06cb90 con 0x7f4d7c15a780 2024-09-13T17:22:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d82e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c0cc980 con 0x7f4d7c15ab60 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d80e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d7c071490 0x7f4d7c06c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40232/0 (socket says 172.21.15.28:40232) 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.486+0000 7f4d80e00640 1 -- 172.21.15.28:0/2455455560 learned_addr learned my addr 172.21.15.28:0/2455455560 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d73400640 1 --2- 172.21.15.28:0/2455455560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 0x7f4d7c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636981074 0 0) 0x7f4d7c06c8b0 con 0x7f4d7c071490 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d60003520 con 0x7f4d7c071490 2024-09-13T17:22:35.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739499222 0 0) 0x7f4d7c06cb90 con 0x7f4d7c15a780 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c06c8b0 con 0x7f4d7c15a780 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d73400640 1 -- 172.21.15.28:0/2455455560 >> v1:172.21.15.189:6789/0 conn(0x7f4d7c15a780 legacy=0x7f4d7c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d73400640 1 -- 172.21.15.28:0/2455455560 >> v1:172.21.15.28:6789/0 conn(0x7f4d7c071490 legacy=0x7f4d7c06c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.487+0000 7f4d73400640 1 -- 172.21.15.28:0/2455455560 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c0ccc60 con 0x7f4d7c15ab60 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.488+0000 7f4d73400640 1 --2- 172.21.15.28:0/2455455560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 0x7f4d7c06bb40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d640097d0 tx=0x7f4d6402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84af2b787de38dfa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64032440 con 0x7f4d7c15ab60 2024-09-13T17:22:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d640325a0 con 0x7f4d7c15ab60 2024-09-13T17:22:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d7b400640 1 -- 172.21.15.28:0/2455455560 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64032930 con 0x7f4d7c15ab60 2024-09-13T17:22:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 -- 172.21.15.28:0/2455455560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 msgr2=0x7f4d7c06bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 --2- 172.21.15.28:0/2455455560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 0x7f4d7c06bb40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4d640097d0 tx=0x7f4d6402a200 comp rx=0 tx=0).stop 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 -- 172.21.15.28:0/2455455560 shutdown_connections 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 --2- 172.21.15.28:0/2455455560 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c15ab60 0x7f4d7c06bb40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 -- 172.21.15.28:0/2455455560 >> 172.21.15.28:0/2455455560 conn(0x7f4d7c0b0270 msgr2=0x7f4d7c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 -- 172.21.15.28:0/2455455560 shutdown_connections 2024-09-13T17:22:35.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.489+0000 7f4d82e00640 1 -- 172.21.15.28:0/2455455560 wait complete. 2024-09-13T17:22:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.491+0000 7f4d82e00640 1 Processor -- start 2024-09-13T17:22:35.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.491+0000 7f4d82e00640 1 -- start start 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c071490 0x7f4d7c14fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d7c15ab60 0x7f4d7c14c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c14ae90 con 0x7f4d7c15a780 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c14b110 con 0x7f4d7c15ab60 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d82e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c14b3f0 con 0x7f4d7c071490 2024-09-13T17:22:35.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d73400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52788/0 (socket says 172.21.15.28:52788) 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.492+0000 7f4d73400640 1 -- 172.21.15.28:0/4280649002 learned_addr learned my addr 172.21.15.28:0/4280649002 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d80e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d7c15ab60 0x7f4d7c14c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d7be00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c071490 0x7f4d7c14fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 -- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c071490 msgr2=0x7f4d7c14fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c071490 0x7f4d7c14fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 -- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d7c15ab60 msgr2=0x7f4d7c14c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d7c15ab60 0x7f4d7c14c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 -- 172.21.15.28:0/4280649002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d64009400 con 0x7f4d7c15a780 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d73400640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f4d64035040 tx=0x7f4d640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6403e920 con 0x7f4d7c15a780 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6403ea80 con 0x7f4d7c15a780 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6403ee30 con 0x7f4d7c15a780 2024-09-13T17:22:35.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7c14b6d0 con 0x7f4d7c15a780 2024-09-13T17:22:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.493+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7c14bc90 con 0x7f4d7c15a780 2024-09-13T17:22:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.495+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7c14be80 con 0x7f4d7c15a780 2024-09-13T17:22:35.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.495+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4d6402ad50 con 0x7f4d7c15a780 2024-09-13T17:22:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.496+0000 7f4d78c00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 0x7f4d5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.496+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(203..203 src has 1..203) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f4d640a9f00 con 0x7f4d7c15a780 2024-09-13T17:22:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.497+0000 7f4d7be00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 0x7f4d5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.498+0000 7f4d7be00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 0x7f4d5806d0f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0011e0 tx=0x7f4d6c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.501+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4d64075260 con 0x7f4d7c15a780 2024-09-13T17:22:35.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.716+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4d7c070d80 con 0x7f4d7c15a780 2024-09-13T17:22:35.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.722+0000 7f4d78c00640 1 -- 172.21.15.28:0/4280649002 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 28 and data pool 29 v169) v1 ==== 174+0+0 (secure 0 0 0) 0x7f4d64078880 con 0x7f4d7c15a780 2024-09-13T17:22:35.724 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 28 and data pool 29 2024-09-13T17:22:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.727+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 msgr2=0x7f4d5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.727+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 0x7f4d5806d0f0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0011e0 tx=0x7f4d6c009040 comp rx=0 tx=0).stop 2024-09-13T17:22:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.727+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 msgr2=0x7f4d7c14a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.727+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f4d64035040 tx=0x7f4d640046c0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 shutdown_connections 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d5806ac30 0x7f4d5806d0f0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d7c15ab60 0x7f4d7c14c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d7c15a780 0x7f4d7c14a950 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 --2- 172.21.15.28:0/4280649002 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d7c071490 0x7f4d7c14fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.728+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 >> 172.21.15.28:0/4280649002 conn(0x7f4d7c0b0270 msgr2=0x7f4d7c1549d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.729+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 shutdown_connections 2024-09-13T17:22:35.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.729+0000 7f4d82e00640 1 -- 172.21.15.28:0/4280649002 wait complete. 2024-09-13T17:22:35.758 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:35.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.914+0000 7f1b78800640 1 Processor -- start 2024-09-13T17:22:35.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.914+0000 7f1b78800640 1 -- start start 2024-09-13T17:22:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b78800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 0x7f1b7015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b78800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b70159d90 con 0x7f1b7006f970 2024-09-13T17:22:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b78800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b700b2040 con 0x7f1b7006c2d0 2024-09-13T17:22:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b78800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7015b0b0 con 0x7f1b701592a0 2024-09-13T17:22:35.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b75400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1b7006f970 0x7f1b70159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40238/0 (socket says 172.21.15.28:40238) 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.915+0000 7f1b75400640 1 -- 172.21.15.28:0/3731258723 learned_addr learned my addr 172.21.15.28:0/3731258723 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b75e00640 1 --2- 172.21.15.28:0/3731258723 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 0x7f1b7015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753966542 0 0) 0x7f1b70159d90 con 0x7f1b7006f970 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58003520 con 0x7f1b7006f970 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565481063 0 0) 0x7f1b700b2040 con 0x7f1b7006c2d0 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b70159d90 con 0x7f1b7006c2d0 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1843271149 0 0) 0x7f1b58003520 con 0x7f1b7006f970 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b75e00640 1 -- 172.21.15.28:0/3731258723 >> v1:172.21.15.189:6789/0 conn(0x7f1b7006c2d0 legacy=0x7f1b700b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b75e00640 1 -- 172.21.15.28:0/3731258723 >> v1:172.21.15.28:6789/0 conn(0x7f1b7006f970 legacy=0x7f1b70159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.916+0000 7f1b75e00640 1 -- 172.21.15.28:0/3731258723 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b70159ff0 con 0x7f1b701592a0 2024-09-13T17:22:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.917+0000 7f1b75e00640 1 --2- 172.21.15.28:0/3731258723 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 0x7f1b7015ab70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b640097d0 tx=0x7f1b6402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71fe36300937f70c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60001690 con 0x7f1b701592a0 2024-09-13T17:22:35.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b6402ac30 con 0x7f1b701592a0 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b74a00640 1 -- 172.21.15.28:0/3731258723 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64032440 con 0x7f1b701592a0 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 -- 172.21.15.28:0/3731258723 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 msgr2=0x7f1b7015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 --2- 172.21.15.28:0/3731258723 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 0x7f1b7015ab70 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b640097d0 tx=0x7f1b6402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 -- 172.21.15.28:0/3731258723 shutdown_connections 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 --2- 172.21.15.28:0/3731258723 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b701592a0 0x7f1b7015ab70 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 -- 172.21.15.28:0/3731258723 >> 172.21.15.28:0/3731258723 conn(0x7f1b70069e80 msgr2=0x7f1b7006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 -- 172.21.15.28:0/3731258723 shutdown_connections 2024-09-13T17:22:35.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.918+0000 7f1b78800640 1 -- 172.21.15.28:0/3731258723 wait complete. 2024-09-13T17:22:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.920+0000 7f1b78800640 1 Processor -- start 2024-09-13T17:22:35.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.920+0000 7f1b78800640 1 -- start start 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7006c2d0 0x7f1b701571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b701592a0 0x7f1b700b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701e3730 con 0x7f1b7006f970 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701e3960 con 0x7f1b701592a0 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b78800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b701e3c40 con 0x7f1b7006c2d0 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b75400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b75400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52798/0 (socket says 172.21.15.28:52798) 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b75400640 1 -- 172.21.15.28:0/1064632374 learned_addr learned my addr 172.21.15.28:0/1064632374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.921+0000 7f1b76800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b701592a0 0x7f1b700b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75e00640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7006c2d0 0x7f1b701571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 -- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7006c2d0 msgr2=0x7f1b701571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7006c2d0 0x7f1b701571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 -- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b701592a0 msgr2=0x7f1b700b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b701592a0 0x7f1b700b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 -- 172.21.15.28:0/1064632374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b64009400 con 0x7f1b7006f970 2024-09-13T17:22:35.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b75400640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f1b60001200 tx=0x7f1b6000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60005ad0 con 0x7f1b7006f970 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b60005c30 con 0x7f1b7006f970 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b701e3f20 con 0x7f1b7006f970 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b60007070 con 0x7f1b7006f970 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.922+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b701e44e0 con 0x7f1b7006f970 2024-09-13T17:22:35.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.924+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7015ab70 con 0x7f1b7006f970 2024-09-13T17:22:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.925+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1b600071d0 con 0x7f1b7006f970 2024-09-13T17:22:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.925+0000 7f1b6a000640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 0x7f1b5406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.926+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b60088900 con 0x7f1b7006f970 2024-09-13T17:22:35.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.926+0000 7f1b75e00640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 0x7f1b5406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:35.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.927+0000 7f1b75e00640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 0x7f1b5406d0c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f1b64034000 tx=0x7f1b6402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:35.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:35.930+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1b60053b80 con 0x7f1b7006f970 2024-09-13T17:22:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.109+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b7006c940 con 0x7f1b7006f970 2024-09-13T17:22:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.111+0000 7f1b6a000640 1 -- 172.21.15.28:0/1064632374 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16592 (secure 0 0 0) 0x7f1b60017020 con 0x7f1b7006f970 2024-09-13T17:22:36.112 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:36.112 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":204,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:35.717063+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 msgr2=0x7f1b5406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 0x7f1b5406d0c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f1b64034000 tx=0x7f1b6402e000 comp rx=0 tx=0).stop 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 msgr2=0x7f1b700cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 secure :-1 s=READY pgs=2448 cs=0 l=1 rev1=1 crypto rx=0x7f1b60001200 tx=0x7f1b6000cb60 comp rx=0 tx=0).stop 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 shutdown_connections 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1b5406ac00 0x7f1b5406d0c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1b701592a0 0x7f1b700b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1b7006f970 0x7f1b700cdb10 unknown :-1 s=CLOSED pgs=2448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 --2- 172.21.15.28:0/1064632374 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1b7006c2d0 0x7f1b701571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 >> 172.21.15.28:0/1064632374 conn(0x7f1b70069e80 msgr2=0x7f1b700b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 shutdown_connections 2024-09-13T17:22:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.113+0000 7f1b78800640 1 -- 172.21.15.28:0/1064632374 wait complete. 2024-09-13T17:22:36.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:36.275 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:22:36.274+0000 7f97f7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:22:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.283+0000 7fae8fa00640 1 Processor -- start 2024-09-13T17:22:36.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.283+0000 7fae8fa00640 1 -- start start 2024-09-13T17:22:36.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8fa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae880acf60 0x7fae88160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8fa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae88146550 con 0x7fae88160310 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8fa00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8815d280 con 0x7fae88146170 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8fa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8815cd20 con 0x7fae880acf60 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fae880acf60 0x7fae88160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae7fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fae88160310 0x7fae880ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40242/0 (socket says 172.21.15.28:40242) 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae7fe00640 1 -- 172.21.15.28:0/214708837 learned_addr learned my addr 172.21.15.28:0/214708837 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:36.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8da00640 1 -- 172.21.15.28:0/214708837 >> v1:172.21.15.189:6789/0 conn(0x7fae88146170 legacy=0x7fae8815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8da00640 1 -- 172.21.15.28:0/214708837 >> v1:172.21.15.28:6789/0 conn(0x7fae88160310 legacy=0x7fae880ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae8da00640 1 -- 172.21.15.28:0/214708837 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae88160d60 con 0x7fae880acf60 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.285+0000 7fae87e00640 1 -- 172.21.15.28:0/214708837 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104497081 0 0) 0x7fae88146550 con 0x7fae88160310 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.286+0000 7fae8da00640 1 --2- 172.21.15.28:0/214708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae880acf60 0x7fae88160730 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fae80009d90 tx=0x7fae8002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c4cf81c915fb43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae87e00640 1 -- 172.21.15.28:0/214708837 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80007740 con 0x7fae880acf60 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae87e00640 1 -- 172.21.15.28:0/214708837 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8002d050 con 0x7fae880acf60 2024-09-13T17:22:36.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae87e00640 1 -- 172.21.15.28:0/214708837 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae80007ac0 con 0x7fae880acf60 2024-09-13T17:22:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 -- 172.21.15.28:0/214708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae880acf60 msgr2=0x7fae88160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 --2- 172.21.15.28:0/214708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae880acf60 0x7fae88160730 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7fae80009d90 tx=0x7fae8002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 -- 172.21.15.28:0/214708837 shutdown_connections 2024-09-13T17:22:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 --2- 172.21.15.28:0/214708837 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae880acf60 0x7fae88160730 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 -- 172.21.15.28:0/214708837 >> 172.21.15.28:0/214708837 conn(0x7fae88069e80 msgr2=0x7fae8806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 -- 172.21.15.28:0/214708837 shutdown_connections 2024-09-13T17:22:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.287+0000 7fae8fa00640 1 -- 172.21.15.28:0/214708837 wait complete. 2024-09-13T17:22:36.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.289+0000 7fae8fa00640 1 Processor -- start 2024-09-13T17:22:36.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.289+0000 7fae8fa00640 1 -- start start 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 0x7fae881449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae88160310 0x7fae88159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae88156150 con 0x7fae88146170 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae881e7240 con 0x7fae880acf60 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8fa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae881e7520 con 0x7fae88160310 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52818/0 (socket says 172.21.15.28:52818) 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 -- 172.21.15.28:0/2585356752 learned_addr learned my addr 172.21.15.28:0/2585356752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8d000640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 0x7fae881449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8da00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae88160310 0x7fae88159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 -- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae88160310 msgr2=0x7fae88159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae88160310 0x7fae88159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 -- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 msgr2=0x7fae881449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 0x7fae881449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae7fe00640 1 -- 172.21.15.28:0/2585356752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80009a70 con 0x7fae88146170 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.290+0000 7fae8d000640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 0x7fae881449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae7fe00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c4d0 tx=0x7fae7000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae700074e0 con 0x7fae88146170 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae70007660 con 0x7fae88146170 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae70004a20 con 0x7fae88146170 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae881e7800 con 0x7fae88146170 2024-09-13T17:22:36.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.291+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae88159860 con 0x7fae88146170 2024-09-13T17:22:36.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.292+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae881e7d20 con 0x7fae88146170 2024-09-13T17:22:36.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.293+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fae70004b80 con 0x7fae88146170 2024-09-13T17:22:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.294+0000 7fae85600640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 0x7fae6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.294+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae70088510 con 0x7fae88146170 2024-09-13T17:22:36.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.294+0000 7fae8d000640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 0x7fae6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.295+0000 7fae8d000640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 0x7fae6406d0c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fae881f5d70 tx=0x7fae78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.299+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fae881e7d20 con 0x7fae88146170 2024-09-13T17:22:36.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.480+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae881e7d20 con 0x7fae88146170 2024-09-13T17:22:36.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.481+0000 7fae85600640 1 -- 172.21.15.28:0/2585356752 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v204) v1 ==== 74+0+16592 (secure 0 0 0) 0x7fae881e7d20 con 0x7fae88146170 2024-09-13T17:22:36.482 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:36.482 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":204,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:35.717063+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 msgr2=0x7fae6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 0x7fae6406d0c0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7fae881f5d70 tx=0x7fae78009040 comp rx=0 tx=0).stop 2024-09-13T17:22:36.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 msgr2=0x7fae88144f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 secure :-1 s=READY pgs=2451 cs=0 l=1 rev1=1 crypto rx=0x7fae7000c4d0 tx=0x7fae7000c990 comp rx=0 tx=0).stop 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 shutdown_connections 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fae6406ac00 0x7fae6406d0c0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fae88160310 0x7fae88159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fae88146170 0x7fae88144f50 unknown :-1 s=CLOSED pgs=2451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 --2- 172.21.15.28:0/2585356752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fae880acf60 0x7fae881449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 >> 172.21.15.28:0/2585356752 conn(0x7fae88069e80 msgr2=0x7fae880ce2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 shutdown_connections 2024-09-13T17:22:36.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.483+0000 7fae8fa00640 1 -- 172.21.15.28:0/2585356752 wait complete. 2024-09-13T17:22:36.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:22:36.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.659+0000 7f3007200640 1 Processor -- start 2024-09-13T17:22:36.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.659+0000 7f3007200640 1 -- start start 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3007200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30080b3e20 0x7f3008070e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3007200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30080cefd0 0x7f300815d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3007200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3008071350 con 0x7f300806cbe0 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3007200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3008070ac0 con 0x7f30080b3e20 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3007200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f300806d830 con 0x7f30080cefd0 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3005e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30080b3e20 0x7f3008070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3005e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f30080b3e20 0x7f3008070e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52822/0 (socket says 172.21.15.28:52822) 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.660+0000 7f3005e00640 1 -- 172.21.15.28:0/1514622809 learned_addr learned my addr 172.21.15.28:0/1514622809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.661+0000 7f3006800640 1 --2- 172.21.15.28:0/1514622809 >> v2:172.21.15.189:3300/0 conn(0x7f30080cefd0 0x7f300815d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.661+0000 7f3005e00640 1 -- 172.21.15.28:0/1514622809 >> v2:172.21.15.189:3300/0 conn(0x7f30080cefd0 msgr2=0x7f300815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.661+0000 7f3005e00640 1 --2- 172.21.15.28:0/1514622809 >> v2:172.21.15.189:3300/0 conn(0x7f30080cefd0 0x7f300815d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.661+0000 7f3005e00640 1 -- 172.21.15.28:0/1514622809 >> v1:172.21.15.189:6789/0 conn(0x7f300806cbe0 legacy=0x7f30080ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.661+0000 7f3005e00640 1 -- 172.21.15.28:0/1514622809 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f300806feb0 con 0x7f30080b3e20 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.662+0000 7f3005e00640 1 --2- 172.21.15.28:0/1514622809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f3008070e10 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f30080b2040 tx=0x7f2ff802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9151b6fa52437484 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.662+0000 7f3005400640 1 -- 172.21.15.28:0/1514622809 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8032440 con 0x7f30080b3e20 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.662+0000 7f3005400640 1 -- 172.21.15.28:0/1514622809 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff80325a0 con 0x7f30080b3e20 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.662+0000 7f3005400640 1 -- 172.21.15.28:0/1514622809 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8032930 con 0x7f30080b3e20 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 -- 172.21.15.28:0/1514622809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 msgr2=0x7f3008070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 --2- 172.21.15.28:0/1514622809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f3008070e10 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f30080b2040 tx=0x7f2ff802a230 comp rx=0 tx=0).stop 2024-09-13T17:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 -- 172.21.15.28:0/1514622809 shutdown_connections 2024-09-13T17:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 --2- 172.21.15.28:0/1514622809 >> v2:172.21.15.189:3300/0 conn(0x7f30080cefd0 0x7f300815d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 --2- 172.21.15.28:0/1514622809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f3008070e10 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 -- 172.21.15.28:0/1514622809 >> 172.21.15.28:0/1514622809 conn(0x7f3008069e80 msgr2=0x7f300806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 -- 172.21.15.28:0/1514622809 shutdown_connections 2024-09-13T17:22:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.663+0000 7f3007200640 1 -- 172.21.15.28:0/1514622809 wait complete. 2024-09-13T17:22:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.664+0000 7f3007200640 1 Processor -- start 2024-09-13T17:22:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 -- start start 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300806cbe0 0x7f3008151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30080cefd0 0x7f300809c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30081f2300 con 0x7f30080b3e20 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3008099fa0 con 0x7f30080cefd0 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.665+0000 7f3007200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f300809a280 con 0x7f300806cbe0 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52834/0 (socket says 172.21.15.28:52834) 2024-09-13T17:22:36.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 -- 172.21.15.28:0/2859536196 learned_addr learned my addr 172.21.15.28:0/2859536196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f3005e00640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300806cbe0 0x7f3008151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f3006800640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30080cefd0 0x7f300809c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 -- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300806cbe0 msgr2=0x7f3008151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300806cbe0 0x7f3008151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 -- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30080cefd0 msgr2=0x7f300809c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30080cefd0 0x7f300809c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 -- 172.21.15.28:0/2859536196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff80094d0 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.666+0000 7f2ffd400640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b3f0 tx=0x7f2ff000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.667+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0011830 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.667+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff0011990 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.667+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f300809a560 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.667+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0011d40 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.667+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f300809ab20 con 0x7f30080b3e20 2024-09-13T17:22:36.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.668+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f300806d370 con 0x7f30080b3e20 2024-09-13T17:22:36.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.669+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ff001a400 con 0x7f30080b3e20 2024-09-13T17:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.670+0000 7f2ffea00640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 0x7f2fe406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.670+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(204..204 src has 1..204) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ff0089120 con 0x7f30080b3e20 2024-09-13T17:22:36.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.670+0000 7f3005e00640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 0x7f2fe406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:36.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.671+0000 7f3005e00640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 0x7f2fe406d0c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8034000 tx=0x7f2ff802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:36.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.674+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2ff0054310 con 0x7f30080b3e20 2024-09-13T17:22:36.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:36.894+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f300809ad40 con 0x7f30080b3e20 2024-09-13T17:22:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.728+0000 7f2ffea00640 1 -- 172.21.15.28:0/2859536196 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v171) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2ff0057930 con 0x7f30080b3e20 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 msgr2=0x7f2fe406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 0x7f2fe406d0c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8034000 tx=0x7f2ff802e000 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 msgr2=0x7f30081486b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 secure :-1 s=READY pgs=2453 cs=0 l=1 rev1=1 crypto rx=0x7f2ff000b3f0 tx=0x7f2ff000b8b0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 shutdown_connections 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2fe406ac00 0x7f2fe406d0c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30080cefd0 0x7f300809c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30080b3e20 0x7f30081486b0 unknown :-1 s=CLOSED pgs=2453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 --2- 172.21.15.28:0/2859536196 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f300806cbe0 0x7f3008151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.731+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 >> 172.21.15.28:0/2859536196 conn(0x7f3008069e80 msgr2=0x7f30080ad7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.732+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 shutdown_connections 2024-09-13T17:22:37.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.732+0000 7f3007200640 1 -- 172.21.15.28:0/2859536196 wait complete. 2024-09-13T17:22:37.750 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:22:37.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:22:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.915+0000 7fdeef200640 1 Processor -- start 2024-09-13T17:22:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.915+0000 7fdeef200640 1 -- start start 2024-09-13T17:22:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdeef200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdee815ab60 0x7fdee806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdeef200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 0x7fdee806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdeef200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee806c8b0 con 0x7fdee815a780 2024-09-13T17:22:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdeef200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee806cb90 con 0x7fdee815ab60 2024-09-13T17:22:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdeef200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee80cc980 con 0x7fdee8071490 2024-09-13T17:22:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdedfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdee815ab60 0x7fdee806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdedfe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdee815ab60 0x7fdee806bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52844/0 (socket says 172.21.15.28:52844) 2024-09-13T17:22:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.916+0000 7fdedfe00640 1 -- 172.21.15.28:0/2073734394 learned_addr learned my addr 172.21.15.28:0/2073734394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdeed200640 1 --2- 172.21.15.28:0/2073734394 >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 0x7fdee806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdedfe00640 1 -- 172.21.15.28:0/2073734394 >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 msgr2=0x7fdee806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdedfe00640 1 --2- 172.21.15.28:0/2073734394 >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 0x7fdee806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdedfe00640 1 -- 172.21.15.28:0/2073734394 >> v1:172.21.15.189:6789/0 conn(0x7fdee815a780 legacy=0x7fdee8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdedfe00640 1 -- 172.21.15.28:0/2073734394 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee80ccc60 con 0x7fdee815ab60 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.917+0000 7fdeed200640 1 --2- 172.21.15.28:0/2073734394 >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 0x7fdee806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.918+0000 7fdedfe00640 1 --2- 172.21.15.28:0/2073734394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee806bb40 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fdee80cc370 tx=0x7fded002a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e6c018dd036e1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.918+0000 7fdee7400640 1 -- 172.21.15.28:0/2073734394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0032440 con 0x7fdee815ab60 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.918+0000 7fdee7400640 1 -- 172.21.15.28:0/2073734394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded00325a0 con 0x7fdee815ab60 2024-09-13T17:22:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.918+0000 7fdee7400640 1 -- 172.21.15.28:0/2073734394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0032930 con 0x7fdee815ab60 2024-09-13T17:22:37.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 -- 172.21.15.28:0/2073734394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 msgr2=0x7fdee806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 --2- 172.21.15.28:0/2073734394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee806bb40 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fdee80cc370 tx=0x7fded002a280 comp rx=0 tx=0).stop 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 -- 172.21.15.28:0/2073734394 shutdown_connections 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 --2- 172.21.15.28:0/2073734394 >> v2:172.21.15.189:3300/0 conn(0x7fdee8071490 0x7fdee806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 --2- 172.21.15.28:0/2073734394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee806bb40 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 -- 172.21.15.28:0/2073734394 >> 172.21.15.28:0/2073734394 conn(0x7fdee80b0270 msgr2=0x7fdee80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 -- 172.21.15.28:0/2073734394 shutdown_connections 2024-09-13T17:22:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.919+0000 7fdeef200640 1 -- 172.21.15.28:0/2073734394 wait complete. 2024-09-13T17:22:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.920+0000 7fdeef200640 1 Processor -- start 2024-09-13T17:22:37.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 -- start start 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 0x7fdee8157260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee815a780 0x7fdee814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80c9be0 con 0x7fdee815ab60 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.921+0000 7fdeef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee814ab30 con 0x7fdee815a780 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee80c8fd0 con 0x7fdee8071490 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52856/0 (socket says 172.21.15.28:52856) 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 -- 172.21.15.28:0/3417102074 learned_addr learned my addr 172.21.15.28:0/3417102074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdedfe00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee815a780 0x7fdee814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdee7e00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 0x7fdee8157260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 -- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 msgr2=0x7fdee8157260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 0x7fdee8157260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 -- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee815a780 msgr2=0x7fdee814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee815a780 0x7fdee814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdeed200640 1 -- 172.21.15.28:0/3417102074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00094d0 con 0x7fdee815ab60 2024-09-13T17:22:37.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.922+0000 7fdee7e00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 0x7fdee8157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdeed200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009fa0 tx=0x7fdee000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0003b90 con 0x7fdee815ab60 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdee000f040 con 0x7fdee815ab60 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdee0004030 con 0x7fdee815ab60 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee80c92b0 con 0x7fdee815ab60 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.923+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee80c9870 con 0x7fdee815ab60 2024-09-13T17:22:37.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.924+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee80c9d50 con 0x7fdee815ab60 2024-09-13T17:22:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.925+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdee0004190 con 0x7fdee815ab60 2024-09-13T17:22:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.926+0000 7fdee4c00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 0x7fdec406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:37.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.926+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdee0089780 con 0x7fdee815ab60 2024-09-13T17:22:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.926+0000 7fdee7e00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 0x7fdec406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:37.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.927+0000 7fdee7e00640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 0x7fdec406d0c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fded8007fd0 tx=0x7fded8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:37.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:37.930+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdee00548e0 con 0x7fdee815ab60 2024-09-13T17:22:38.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.143+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fdee8070d80 con 0x7fdee815ab60 2024-09-13T17:22:38.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.730+0000 7fdee4c00640 1 -- 172.21.15.28:0/3417102074 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v172) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdee001a0c0 con 0x7fdee815ab60 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 msgr2=0x7fdec406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 0x7fdec406d0c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7fded8007fd0 tx=0x7fded8009040 comp rx=0 tx=0).stop 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 msgr2=0x7fdee80ca070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009fa0 tx=0x7fdee000d560 comp rx=0 tx=0).stop 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 shutdown_connections 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdec406ac00 0x7fdec406d0c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdee815ab60 0x7fdee80ca070 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee815a780 0x7fdee814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 --2- 172.21.15.28:0/3417102074 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdee8071490 0x7fdee8157260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 >> 172.21.15.28:0/3417102074 conn(0x7fdee80b0270 msgr2=0x7fdee80ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 shutdown_connections 2024-09-13T17:22:38.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.733+0000 7fdeef200640 1 -- 172.21.15.28:0/3417102074 wait complete. 2024-09-13T17:22:38.751 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:22:38.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:38.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.908+0000 7feb9dc00640 1 Processor -- start 2024-09-13T17:22:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.908+0000 7feb9dc00640 1 -- start start 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb9815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb9806f970 0x7feb98159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7feb9806c2d0 0x7feb9806c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9815b0b0 con 0x7feb9806f970 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9806ccc0 con 0x7feb9806c2d0 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb9dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980b2040 con 0x7feb981592a0 2024-09-13T17:22:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb8e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb9806f970 0x7feb98159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb8e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feb9806f970 0x7feb98159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52868/0 (socket says 172.21.15.28:52868) 2024-09-13T17:22:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb8e000640 1 -- 172.21.15.28:0/2410531604 learned_addr learned my addr 172.21.15.28:0/2410531604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb97400640 1 --2- 172.21.15.28:0/2410531604 >> v2:172.21.15.189:3300/0 conn(0x7feb9806c2d0 0x7feb9806c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.910+0000 7feb96a00640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb9815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.911+0000 7feb8e000640 1 -- 172.21.15.28:0/2410531604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 msgr2=0x7feb9815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.911+0000 7feb8e000640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb9815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.911+0000 7feb8e000640 1 -- 172.21.15.28:0/2410531604 >> v2:172.21.15.189:3300/0 conn(0x7feb9806c2d0 msgr2=0x7feb9806c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.911+0000 7feb8e000640 1 --2- 172.21.15.28:0/2410531604 >> v2:172.21.15.189:3300/0 conn(0x7feb9806c2d0 0x7feb9806c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.911+0000 7feb8e000640 1 -- 172.21.15.28:0/2410531604 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb98159ff0 con 0x7feb9806f970 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb8e000640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806f970 0x7feb98159680 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7feb9806f700 tx=0x7feb8002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addad20a3abe9469 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb96000640 1 -- 172.21.15.28:0/2410531604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80032440 con 0x7feb9806f970 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb96000640 1 -- 172.21.15.28:0/2410531604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb800325a0 con 0x7feb9806f970 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb96000640 1 -- 172.21.15.28:0/2410531604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb80032930 con 0x7feb9806f970 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb9dc00640 1 -- 172.21.15.28:0/2410531604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806f970 msgr2=0x7feb98159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.912+0000 7feb9dc00640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806f970 0x7feb98159680 secure :-1 s=READY pgs=2456 cs=0 l=1 rev1=1 crypto rx=0x7feb9806f700 tx=0x7feb8002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 -- 172.21.15.28:0/2410531604 shutdown_connections 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806f970 0x7feb98159680 unknown :-1 s=CLOSED pgs=2456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 --2- 172.21.15.28:0/2410531604 >> v2:172.21.15.189:3300/0 conn(0x7feb9806c2d0 0x7feb9806c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 --2- 172.21.15.28:0/2410531604 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb9815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 -- 172.21.15.28:0/2410531604 >> 172.21.15.28:0/2410531604 conn(0x7feb98069e80 msgr2=0x7feb9806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 -- 172.21.15.28:0/2410531604 shutdown_connections 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.913+0000 7feb9dc00640 1 -- 172.21.15.28:0/2410531604 wait complete. 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.914+0000 7feb9dc00640 1 Processor -- start 2024-09-13T17:22:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.914+0000 7feb9dc00640 1 -- start start 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9806f970 0x7feb9814f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb981e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980cc210 con 0x7feb9806c2d0 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb980cd500 con 0x7feb9806f970 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb9dc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb980cc460 con 0x7feb981592a0 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.915+0000 7feb96a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52876/0 (socket says 172.21.15.28:52876) 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 -- 172.21.15.28:0/64709504 learned_addr learned my addr 172.21.15.28:0/64709504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb97400640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb981e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb8e000640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9806f970 0x7feb9814f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 -- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 msgr2=0x7feb981e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb981e5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 -- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9806f970 msgr2=0x7feb9814f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9806f970 0x7feb9814f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 -- 172.21.15.28:0/64709504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb800094d0 con 0x7feb9806c2d0 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb97400640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb981e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb96a00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7feb8800e870 tx=0x7feb8800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.916+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb880094f0 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.917+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb8800c940 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.917+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8800ccf0 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.917+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb980cc740 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.917+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb980ccd00 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.918+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb98067720 con 0x7feb9806c2d0 2024-09-13T17:22:38.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.919+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feb88013470 con 0x7feb9806c2d0 2024-09-13T17:22:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.920+0000 7feb8f400640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 0x7feb7406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.920+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feb88089430 con 0x7feb9806c2d0 2024-09-13T17:22:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.920+0000 7feb8e000640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 0x7feb7406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.921+0000 7feb8e000640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 0x7feb7406d0c0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7feb80034000 tx=0x7feb8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:38.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:38.924+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7feb88054600 con 0x7feb9806c2d0 2024-09-13T17:22:39.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.138+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb9806c780 con 0x7feb9806c2d0 2024-09-13T17:22:39.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.138+0000 7feb8f400640 1 -- 172.21.15.28:0/64709504 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7feb88057c20 con 0x7feb9806c2d0 2024-09-13T17:22:39.139 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:39.139 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:22:39.140 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 msgr2=0x7feb7406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 0x7feb7406d0c0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7feb80034000 tx=0x7feb8002e000 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 msgr2=0x7feb9814eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7feb8800e870 tx=0x7feb8800ed30 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 shutdown_connections 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7feb7406ac00 0x7feb7406d0c0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feb981592a0 0x7feb981e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feb9806f970 0x7feb9814f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 --2- 172.21.15.28:0/64709504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feb9806c2d0 0x7feb9814eed0 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 >> 172.21.15.28:0/64709504 conn(0x7feb98069e80 msgr2=0x7feb980cb550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:39.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.140+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 shutdown_connections 2024-09-13T17:22:39.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.141+0000 7feb9dc00640 1 -- 172.21.15.28:0/64709504 wait complete. 2024-09-13T17:22:39.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:22:39.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.311+0000 7f85efe00640 1 Processor -- start 2024-09-13T17:22:39.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.312+0000 7f85efe00640 1 -- start start 2024-09-13T17:22:39.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85efe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00c3850 con 0x7f85f0159820 2024-09-13T17:22:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85efe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f015a930 con 0x7f85f0159440 2024-09-13T17:22:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85efe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0159d20 con 0x7f85f00b0b50 2024-09-13T17:22:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85ef400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f85f0159820 0x7f85f015adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40244/0 (socket says 172.21.15.28:40244) 2024-09-13T17:22:39.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85ef400640 1 -- 172.21.15.28:0/683846167 learned_addr learned my addr 172.21.15.28:0/683846167 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85e7e00640 1 --2- 172.21.15.28:0/683846167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.313+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90640989 0 0) 0x7f85f00c3850 con 0x7f85f0159820 2024-09-13T17:22:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d4003520 con 0x7f85f0159820 2024-09-13T17:22:39.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351239362 0 0) 0x7f85f015a930 con 0x7f85f0159440 2024-09-13T17:22:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85f00c3850 con 0x7f85f0159440 2024-09-13T17:22:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85e7e00640 1 -- 172.21.15.28:0/683846167 >> v1:172.21.15.189:6789/0 conn(0x7f85f0159440 legacy=0x7f85f00b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85e7e00640 1 -- 172.21.15.28:0/683846167 >> v1:172.21.15.28:6789/0 conn(0x7f85f0159820 legacy=0x7f85f015adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1093767812 0 0) 0x7f85d4003520 con 0x7f85f0159820 2024-09-13T17:22:39.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.314+0000 7f85e7e00640 1 -- 172.21.15.28:0/683846167 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f015a000 con 0x7f85f00b0b50 2024-09-13T17:22:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.315+0000 7f85e7e00640 1 --2- 172.21.15.28:0/683846167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00b22e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097d0 tx=0x7f85d802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db021aeb627b049 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.315+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e80019f0 con 0x7f85f00b0b50 2024-09-13T17:22:39.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.315+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85d802acc0 con 0x7f85f00b0b50 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.315+0000 7f85ee000640 1 -- 172.21.15.28:0/683846167 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d8032640 con 0x7f85f00b0b50 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 -- 172.21.15.28:0/683846167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 msgr2=0x7f85f00b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 --2- 172.21.15.28:0/683846167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00b22e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f85d80097d0 tx=0x7f85d802a230 comp rx=0 tx=0).stop 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 -- 172.21.15.28:0/683846167 shutdown_connections 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 --2- 172.21.15.28:0/683846167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00b22e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 -- 172.21.15.28:0/683846167 >> 172.21.15.28:0/683846167 conn(0x7f85f0072e40 msgr2=0x7f85f0073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 -- 172.21.15.28:0/683846167 shutdown_connections 2024-09-13T17:22:39.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.316+0000 7f85efe00640 1 -- 172.21.15.28:0/683846167 wait complete. 2024-09-13T17:22:39.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.318+0000 7f85efe00640 1 Processor -- start 2024-09-13T17:22:39.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.318+0000 7f85efe00640 1 -- start start 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.318+0000 7f85efe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 0x7f85f01f2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0157fa0 con 0x7f85f0159820 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85efe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00cce60 con 0x7f85f0159440 2024-09-13T17:22:39.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85efe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0157420 con 0x7f85f00b0b50 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52888/0 (socket says 172.21.15.28:52888) 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 -- 172.21.15.28:0/1607927517 learned_addr learned my addr 172.21.15.28:0/1607927517 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85eea00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85e7e00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 0x7f85f01f2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 -- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 msgr2=0x7f85f00af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 -- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 msgr2=0x7f85f01f2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 0x7f85f01f2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85ef400640 1 -- 172.21.15.28:0/1607927517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8009400 con 0x7f85f0159820 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85eea00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:39.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.319+0000 7f85e7e00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 0x7f85f01f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85ef400640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f85e800cb60 tx=0x7f85e800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800ace0 con 0x7f85f0159820 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e800ae60 con 0x7f85f0159820 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8012640 con 0x7f85f0159820 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f0157700 con 0x7f85f0159820 2024-09-13T17:22:39.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.320+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f0157cc0 con 0x7f85f0159820 2024-09-13T17:22:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.321+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f00b07c0 con 0x7f85f0159820 2024-09-13T17:22:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.322+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f85e8007070 con 0x7f85f0159820 2024-09-13T17:22:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.323+0000 7f85e6a00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 0x7f85cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.323+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85e80893b0 con 0x7f85f0159820 2024-09-13T17:22:39.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.323+0000 7f85eea00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 0x7f85cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.324+0000 7f85eea00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 0x7f85cc06d0c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f85f00aea90 tx=0x7f85e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.327+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f85e80544f0 con 0x7f85f0159820 2024-09-13T17:22:39.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.509+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f00acc40 con 0x7f85f0159820 2024-09-13T17:22:39.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.510+0000 7f85e6a00640 1 -- 172.21.15.28:0/1607927517 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v205) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f85e8057b10 con 0x7f85f0159820 2024-09-13T17:22:39.511 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:39.512 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":205,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:36.719413+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:22:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 msgr2=0x7f85cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 0x7f85cc06d0c0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f85f00aea90 tx=0x7f85e0009040 comp rx=0 tx=0).stop 2024-09-13T17:22:39.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 msgr2=0x7f85f01f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7f85e800cb60 tx=0x7f85e800a430 comp rx=0 tx=0).stop 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 shutdown_connections 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f85cc06ac00 0x7f85cc06d0c0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f85f0159820 0x7f85f01f2660 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f85f0159440 0x7f85f01f2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 --2- 172.21.15.28:0/1607927517 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f85f00b0b50 0x7f85f00af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 >> 172.21.15.28:0/1607927517 conn(0x7f85f0072e40 msgr2=0x7f85f0070080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 shutdown_connections 2024-09-13T17:22:39.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.512+0000 7f85efe00640 1 -- 172.21.15.28:0/1607927517 wait complete. 2024-09-13T17:22:39.529 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:22:39.529 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:22:39.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.683+0000 7fe2ef000640 1 Processor -- start 2024-09-13T17:22:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.684+0000 7fe2ef000640 1 -- start start 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2ef000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe2e80acf60 0x7fe2e8160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2ef000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e8146550 con 0x7fe2e8160310 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2ef000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e815d280 con 0x7fe2e8146170 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2ef000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e815cd20 con 0x7fe2e80acf60 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2df400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe2e8160310 0x7fe2e80ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40260/0 (socket says 172.21.15.28:40260) 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2df400640 1 -- 172.21.15.28:0/780083390 learned_addr learned my addr 172.21.15.28:0/780083390 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:39.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2ed000640 1 --2- 172.21.15.28:0/780083390 >> v2:172.21.15.189:3300/0 conn(0x7fe2e80acf60 0x7fe2e8160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.685+0000 7fe2e7400640 1 -- 172.21.15.28:0/780083390 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134681959 0 0) 0x7fe2e8146550 con 0x7fe2e8160310 2024-09-13T17:22:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.686+0000 7fe2e7400640 1 -- 172.21.15.28:0/780083390 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc003520 con 0x7fe2e8160310 2024-09-13T17:22:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.686+0000 7fe2ed000640 1 -- 172.21.15.28:0/780083390 >> v1:172.21.15.189:6789/0 conn(0x7fe2e8146170 legacy=0x7fe2e815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.686+0000 7fe2ed000640 1 -- 172.21.15.28:0/780083390 >> v1:172.21.15.28:6789/0 conn(0x7fe2e8160310 legacy=0x7fe2e80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.686+0000 7fe2ed000640 1 -- 172.21.15.28:0/780083390 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e8160d60 con 0x7fe2e80acf60 2024-09-13T17:22:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.687+0000 7fe2ed000640 1 --2- 172.21.15.28:0/780083390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e80acf60 0x7fe2e8160730 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009d90 tx=0x7fe2e002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76587bac7cdec700 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.687+0000 7fe2e7400640 1 -- 172.21.15.28:0/780083390 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0007740 con 0x7fe2e80acf60 2024-09-13T17:22:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.687+0000 7fe2e7400640 1 -- 172.21.15.28:0/780083390 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e002d050 con 0x7fe2e80acf60 2024-09-13T17:22:39.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.687+0000 7fe2e7400640 1 -- 172.21.15.28:0/780083390 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0007ac0 con 0x7fe2e80acf60 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 -- 172.21.15.28:0/780083390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e80acf60 msgr2=0x7fe2e8160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 --2- 172.21.15.28:0/780083390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e80acf60 0x7fe2e8160730 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009d90 tx=0x7fe2e002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 -- 172.21.15.28:0/780083390 shutdown_connections 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 --2- 172.21.15.28:0/780083390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e80acf60 0x7fe2e8160730 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 -- 172.21.15.28:0/780083390 >> 172.21.15.28:0/780083390 conn(0x7fe2e8069e80 msgr2=0x7fe2e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 -- 172.21.15.28:0/780083390 shutdown_connections 2024-09-13T17:22:39.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.688+0000 7fe2ef000640 1 -- 172.21.15.28:0/780083390 wait complete. 2024-09-13T17:22:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.689+0000 7fe2ef000640 1 Processor -- start 2024-09-13T17:22:39.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.690+0000 7fe2ef000640 1 -- start start 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.690+0000 7fe2ef000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2ef000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e8146170 0x7fe2e8143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2ef000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2e8160310 0x7fe2e815ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2ef000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e813d930 con 0x7fe2e80acf60 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2ef000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8140160 con 0x7fe2e8146170 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2ef000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8140440 con 0x7fe2e8160310 2024-09-13T17:22:39.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52904/0 (socket says 172.21.15.28:52904) 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 -- 172.21.15.28:0/4189110336 learned_addr learned my addr 172.21.15.28:0/4189110336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2df400640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e8146170 0x7fe2e8143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 -- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2e8160310 msgr2=0x7fe2e815ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2e8160310 0x7fe2e815ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 -- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e8146170 msgr2=0x7fe2e8143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e8146170 0x7fe2e8143990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 -- 172.21.15.28:0/4189110336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e0009a70 con 0x7fe2e80acf60 2024-09-13T17:22:39.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.691+0000 7fe2e7e00640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800c450 tx=0x7fe2d800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.692+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d80052d0 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.692+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8005430 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.692+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8012870 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.692+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e8140720 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.692+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e8140ce0 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.693+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e8140f00 con 0x7fe2e80acf60 2024-09-13T17:22:39.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.694+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe2d80129d0 con 0x7fe2e80acf60 2024-09-13T17:22:39.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.695+0000 7fe2e4c00640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 0x7fe2c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.695+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe2d8088f10 con 0x7fe2e80acf60 2024-09-13T17:22:39.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.695+0000 7fe2df400640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 0x7fe2c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:39.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.696+0000 7fe2df400640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 0x7fe2c406d0c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe2e81458a0 tx=0x7fe2d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:39.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.699+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe2d8054050 con 0x7fe2e80acf60 2024-09-13T17:22:39.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.876+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe2e81405b0 con 0x7fe2c406ac00 2024-09-13T17:22:39.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.885+0000 7fe2e4c00640 1 -- 172.21.15.28:0/4189110336 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289538 (secure 0 0 0) 0x7fe2e81405b0 con 0x7fe2c406ac00 2024-09-13T17:22:39.889 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:39.890 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:22:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 msgr2=0x7fe2c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 0x7fe2c406d0c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fe2e81458a0 tx=0x7fe2d0009040 comp rx=0 tx=0).stop 2024-09-13T17:22:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 msgr2=0x7fe2e813d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:39.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800c450 tx=0x7fe2d800c910 comp rx=0 tx=0).stop 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 shutdown_connections 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe2c406ac00 0x7fe2c406d0c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe2e8160310 0x7fe2e815ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe2e8146170 0x7fe2e8143990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 --2- 172.21.15.28:0/4189110336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2e80acf60 0x7fe2e813d3f0 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 >> 172.21.15.28:0/4189110336 conn(0x7fe2e8069e80 msgr2=0x7fe2e806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 shutdown_connections 2024-09-13T17:22:39.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:39.888+0000 7fe2ef000640 1 -- 172.21.15.28:0/4189110336 wait complete. 2024-09-13T17:22:39.905 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-13T17:22:39.104553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":552956,"kb_used_data":21348,"kb_used_omap":12,"kb_used_meta":531443,"kb_avail":754421764,"statfs":{"total":773094113280,"available":772527886336,"internally_reserved":0,"allocated":21860352,"data_stored":10287282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":544198201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.821232"},"pg_stats":[{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899536+0000","last_change":"2024-09-13T17:22:35.747008+0000","last_active":"2024-09-13T17:22:37.899536+0000","last_peered":"2024-09-13T17:22:37.899536+0000","last_clean":"2024-09-13T17:22:37.899536+0000","last_became_active":"2024-09-13T17:22:35.746755+0000","last_became_peered":"2024-09-13T17:22:35.746755+0000","last_unstale":"2024-09-13T17:22:37.899536+0000","last_undegraded":"2024-09-13T17:22:37.899536+0000","last_fullsized":"2024-09-13T17:22:37.899536+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:01.527960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725581+0000","last_change":"2024-09-13T17:22:35.736781+0000","last_active":"2024-09-13T17:22:36.725581+0000","last_peered":"2024-09-13T17:22:36.725581+0000","last_clean":"2024-09-13T17:22:36.725581+0000","last_became_active":"2024-09-13T17:22:35.736519+0000","last_became_peered":"2024-09-13T17:22:35.736519+0000","last_unstale":"2024-09-13T17:22:36.725581+0000","last_undegraded":"2024-09-13T17:22:36.725581+0000","last_fullsized":"2024-09-13T17:22:36.725581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:21.616718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898657+0000","last_change":"2024-09-13T17:22:35.744547+0000","last_active":"2024-09-13T17:22:37.898657+0000","last_peered":"2024-09-13T17:22:37.898657+0000","last_clean":"2024-09-13T17:22:37.898657+0000","last_became_active":"2024-09-13T17:22:35.744287+0000","last_became_peered":"2024-09-13T17:22:35.744287+0000","last_unstale":"2024-09-13T17:22:37.898657+0000","last_undegraded":"2024-09-13T17:22:37.898657+0000","last_fullsized":"2024-09-13T17:22:37.898657+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723575+0000","last_change":"2024-09-13T17:22:35.745384+0000","last_active":"2024-09-13T17:22:36.723575+0000","last_peered":"2024-09-13T17:22:36.723575+0000","last_clean":"2024-09-13T17:22:36.723575+0000","last_became_active":"2024-09-13T17:22:35.745141+0000","last_became_peered":"2024-09-13T17:22:35.745141+0000","last_unstale":"2024-09-13T17:22:36.723575+0000","last_undegraded":"2024-09-13T17:22:36.723575+0000","last_fullsized":"2024-09-13T17:22:36.723575+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723647+0000","last_change":"2024-09-13T17:22:35.747831+0000","last_active":"2024-09-13T17:22:36.723647+0000","last_peered":"2024-09-13T17:22:36.723647+0000","last_clean":"2024-09-13T17:22:36.723647+0000","last_became_active":"2024-09-13T17:22:35.747566+0000","last_became_peered":"2024-09-13T17:22:35.747566+0000","last_unstale":"2024-09-13T17:22:36.723647+0000","last_undegraded":"2024-09-13T17:22:36.723647+0000","last_fullsized":"2024-09-13T17:22:36.723647+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:15.361193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898125+0000","last_change":"2024-09-13T17:22:35.727313+0000","last_active":"2024-09-13T17:22:37.898125+0000","last_peered":"2024-09-13T17:22:37.898125+0000","last_clean":"2024-09-13T17:22:37.898125+0000","last_became_active":"2024-09-13T17:22:35.727050+0000","last_became_peered":"2024-09-13T17:22:35.727050+0000","last_unstale":"2024-09-13T17:22:37.898125+0000","last_undegraded":"2024-09-13T17:22:37.898125+0000","last_fullsized":"2024-09-13T17:22:37.898125+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:03.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898335+0000","last_change":"2024-09-13T17:22:35.744870+0000","last_active":"2024-09-13T17:22:37.898335+0000","last_peered":"2024-09-13T17:22:37.898335+0000","last_clean":"2024-09-13T17:22:37.898335+0000","last_became_active":"2024-09-13T17:22:35.744453+0000","last_became_peered":"2024-09-13T17:22:35.744453+0000","last_unstale":"2024-09-13T17:22:37.898335+0000","last_undegraded":"2024-09-13T17:22:37.898335+0000","last_fullsized":"2024-09-13T17:22:37.898335+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:47.004276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898031+0000","last_change":"2024-09-13T17:22:35.750935+0000","last_active":"2024-09-13T17:22:37.898031+0000","last_peered":"2024-09-13T17:22:37.898031+0000","last_clean":"2024-09-13T17:22:37.898031+0000","last_became_active":"2024-09-13T17:22:35.750066+0000","last_became_peered":"2024-09-13T17:22:35.750066+0000","last_unstale":"2024-09-13T17:22:37.898031+0000","last_undegraded":"2024-09-13T17:22:37.898031+0000","last_fullsized":"2024-09-13T17:22:37.898031+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:22.082049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898832+0000","last_change":"2024-09-13T17:22:35.746371+0000","last_active":"2024-09-13T17:22:37.898832+0000","last_peered":"2024-09-13T17:22:37.898832+0000","last_clean":"2024-09-13T17:22:37.898832+0000","last_became_active":"2024-09-13T17:22:35.746088+0000","last_became_peered":"2024-09-13T17:22:35.746088+0000","last_unstale":"2024-09-13T17:22:37.898832+0000","last_undegraded":"2024-09-13T17:22:37.898832+0000","last_fullsized":"2024-09-13T17:22:37.898832+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:27.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898526+0000","last_change":"2024-09-13T17:22:35.740455+0000","last_active":"2024-09-13T17:22:37.898526+0000","last_peered":"2024-09-13T17:22:37.898526+0000","last_clean":"2024-09-13T17:22:37.898526+0000","last_became_active":"2024-09-13T17:22:35.740209+0000","last_became_peered":"2024-09-13T17:22:35.740209+0000","last_unstale":"2024-09-13T17:22:37.898526+0000","last_undegraded":"2024-09-13T17:22:37.898526+0000","last_fullsized":"2024-09-13T17:22:37.898526+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.727927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898958+0000","last_change":"2024-09-13T17:22:35.749707+0000","last_active":"2024-09-13T17:22:37.898958+0000","last_peered":"2024-09-13T17:22:37.898958+0000","last_clean":"2024-09-13T17:22:37.898958+0000","last_became_active":"2024-09-13T17:22:35.749441+0000","last_became_peered":"2024-09-13T17:22:35.749441+0000","last_unstale":"2024-09-13T17:22:37.898958+0000","last_undegraded":"2024-09-13T17:22:37.898958+0000","last_fullsized":"2024-09-13T17:22:37.898958+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.032697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725459+0000","last_change":"2024-09-13T17:22:35.735954+0000","last_active":"2024-09-13T17:22:36.725459+0000","last_peered":"2024-09-13T17:22:36.725459+0000","last_clean":"2024-09-13T17:22:36.725459+0000","last_became_active":"2024-09-13T17:22:35.734897+0000","last_became_peered":"2024-09-13T17:22:35.734897+0000","last_unstale":"2024-09-13T17:22:36.725459+0000","last_undegraded":"2024-09-13T17:22:36.725459+0000","last_fullsized":"2024-09-13T17:22:36.725459+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:01.527960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725051+0000","last_change":"2024-09-13T17:22:35.734515+0000","last_active":"2024-09-13T17:22:36.725051+0000","last_peered":"2024-09-13T17:22:36.725051+0000","last_clean":"2024-09-13T17:22:36.725051+0000","last_became_active":"2024-09-13T17:22:35.734250+0000","last_became_peered":"2024-09-13T17:22:35.734250+0000","last_unstale":"2024-09-13T17:22:36.725051+0000","last_undegraded":"2024-09-13T17:22:36.725051+0000","last_fullsized":"2024-09-13T17:22:36.725051+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:40.351477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725560+0000","last_change":"2024-09-13T17:22:35.738211+0000","last_active":"2024-09-13T17:22:36.725560+0000","last_peered":"2024-09-13T17:22:36.725560+0000","last_clean":"2024-09-13T17:22:36.725560+0000","last_became_active":"2024-09-13T17:22:35.737899+0000","last_became_peered":"2024-09-13T17:22:35.737899+0000","last_unstale":"2024-09-13T17:22:36.725560+0000","last_undegraded":"2024-09-13T17:22:36.725560+0000","last_fullsized":"2024-09-13T17:22:36.725560+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897310+0000","last_change":"2024-09-13T17:22:35.742600+0000","last_active":"2024-09-13T17:22:37.897310+0000","last_peered":"2024-09-13T17:22:37.897310+0000","last_clean":"2024-09-13T17:22:37.897310+0000","last_became_active":"2024-09-13T17:22:35.742313+0000","last_became_peered":"2024-09-13T17:22:35.742313+0000","last_unstale":"2024-09-13T17:22:37.897310+0000","last_undegraded":"2024-09-13T17:22:37.897310+0000","last_fullsized":"2024-09-13T17:22:37.897310+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:29:04.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724725+0000","last_change":"2024-09-13T17:22:35.733414+0000","last_active":"2024-09-13T17:22:36.724725+0000","last_peered":"2024-09-13T17:22:36.724725+0000","last_clean":"2024-09-13T17:22:36.724725+0000","last_became_active":"2024-09-13T17:22:35.733127+0000","last_became_peered":"2024-09-13T17:22:35.733127+0000","last_unstale":"2024-09-13T17:22:36.724725+0000","last_undegraded":"2024-09-13T17:22:36.724725+0000","last_fullsized":"2024-09-13T17:22:36.724725+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:00.326830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725824+0000","last_change":"2024-09-13T17:22:35.741391+0000","last_active":"2024-09-13T17:22:36.725824+0000","last_peered":"2024-09-13T17:22:36.725824+0000","last_clean":"2024-09-13T17:22:36.725824+0000","last_became_active":"2024-09-13T17:22:35.736783+0000","last_became_peered":"2024-09-13T17:22:35.736783+0000","last_unstale":"2024-09-13T17:22:36.725824+0000","last_undegraded":"2024-09-13T17:22:36.725824+0000","last_fullsized":"2024-09-13T17:22:36.725824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898629+0000","last_change":"2024-09-13T17:22:35.738314+0000","last_active":"2024-09-13T17:22:37.898629+0000","last_peered":"2024-09-13T17:22:37.898629+0000","last_clean":"2024-09-13T17:22:37.898629+0000","last_became_active":"2024-09-13T17:22:35.738035+0000","last_became_peered":"2024-09-13T17:22:35.738035+0000","last_unstale":"2024-09-13T17:22:37.898629+0000","last_undegraded":"2024-09-13T17:22:37.898629+0000","last_fullsized":"2024-09-13T17:22:37.898629+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:47.789552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723854+0000","last_change":"2024-09-13T17:22:35.741091+0000","last_active":"2024-09-13T17:22:36.723854+0000","last_peered":"2024-09-13T17:22:36.723854+0000","last_clean":"2024-09-13T17:22:36.723854+0000","last_became_active":"2024-09-13T17:22:35.740832+0000","last_became_peered":"2024-09-13T17:22:35.740832+0000","last_unstale":"2024-09-13T17:22:36.723854+0000","last_undegraded":"2024-09-13T17:22:36.723854+0000","last_fullsized":"2024-09-13T17:22:36.723854+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898416+0000","last_change":"2024-09-13T17:22:35.730293+0000","last_active":"2024-09-13T17:22:37.898416+0000","last_peered":"2024-09-13T17:22:37.898416+0000","last_clean":"2024-09-13T17:22:37.898416+0000","last_became_active":"2024-09-13T17:22:35.730049+0000","last_became_peered":"2024-09-13T17:22:35.730049+0000","last_unstale":"2024-09-13T17:22:37.898416+0000","last_undegraded":"2024-09-13T17:22:37.898416+0000","last_fullsized":"2024-09-13T17:22:37.898416+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:19.815948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723452+0000","last_change":"2024-09-13T17:22:35.740999+0000","last_active":"2024-09-13T17:22:36.723452+0000","last_peered":"2024-09-13T17:22:36.723452+0000","last_clean":"2024-09-13T17:22:36.723452+0000","last_became_active":"2024-09-13T17:22:35.740656+0000","last_became_peered":"2024-09-13T17:22:35.740656+0000","last_unstale":"2024-09-13T17:22:36.723452+0000","last_undegraded":"2024-09-13T17:22:36.723452+0000","last_fullsized":"2024-09-13T17:22:36.723452+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897530+0000","last_change":"2024-09-13T17:22:35.734233+0000","last_active":"2024-09-13T17:22:37.897530+0000","last_peered":"2024-09-13T17:22:37.897530+0000","last_clean":"2024-09-13T17:22:37.897530+0000","last_became_active":"2024-09-13T17:22:35.733600+0000","last_became_peered":"2024-09-13T17:22:35.733600+0000","last_unstale":"2024-09-13T17:22:37.897530+0000","last_undegraded":"2024-09-13T17:22:37.897530+0000","last_fullsized":"2024-09-13T17:22:37.897530+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:25.672327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898004+0000","last_change":"2024-09-13T17:22:35.738172+0000","last_active":"2024-09-13T17:22:37.898004+0000","last_peered":"2024-09-13T17:22:37.898004+0000","last_clean":"2024-09-13T17:22:37.898004+0000","last_became_active":"2024-09-13T17:22:35.737818+0000","last_became_peered":"2024-09-13T17:22:35.737818+0000","last_unstale":"2024-09-13T17:22:37.898004+0000","last_undegraded":"2024-09-13T17:22:37.898004+0000","last_fullsized":"2024-09-13T17:22:37.898004+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723358+0000","last_change":"2024-09-13T17:22:35.747843+0000","last_active":"2024-09-13T17:22:36.723358+0000","last_peered":"2024-09-13T17:22:36.723358+0000","last_clean":"2024-09-13T17:22:36.723358+0000","last_became_active":"2024-09-13T17:22:35.747555+0000","last_became_peered":"2024-09-13T17:22:35.747555+0000","last_unstale":"2024-09-13T17:22:36.723358+0000","last_undegraded":"2024-09-13T17:22:36.723358+0000","last_fullsized":"2024-09-13T17:22:36.723358+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899536+0000","last_change":"2024-09-13T17:22:35.745430+0000","last_active":"2024-09-13T17:22:37.899536+0000","last_peered":"2024-09-13T17:22:37.899536+0000","last_clean":"2024-09-13T17:22:37.899536+0000","last_became_active":"2024-09-13T17:22:35.744580+0000","last_became_peered":"2024-09-13T17:22:35.744580+0000","last_unstale":"2024-09-13T17:22:37.899536+0000","last_undegraded":"2024-09-13T17:22:37.899536+0000","last_fullsized":"2024-09-13T17:22:37.899536+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:35.139446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898543+0000","last_change":"2024-09-13T17:22:35.747792+0000","last_active":"2024-09-13T17:22:37.898543+0000","last_peered":"2024-09-13T17:22:37.898543+0000","last_clean":"2024-09-13T17:22:37.898543+0000","last_became_active":"2024-09-13T17:22:35.747544+0000","last_became_peered":"2024-09-13T17:22:35.747544+0000","last_unstale":"2024-09-13T17:22:37.898543+0000","last_undegraded":"2024-09-13T17:22:37.898543+0000","last_fullsized":"2024-09-13T17:22:37.898543+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:27:57.647264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899163+0000","last_change":"2024-09-13T17:22:35.749403+0000","last_active":"2024-09-13T17:22:37.899163+0000","last_peered":"2024-09-13T17:22:37.899163+0000","last_clean":"2024-09-13T17:22:37.899163+0000","last_became_active":"2024-09-13T17:22:35.749132+0000","last_became_peered":"2024-09-13T17:22:35.749132+0000","last_unstale":"2024-09-13T17:22:37.899163+0000","last_undegraded":"2024-09-13T17:22:37.899163+0000","last_fullsized":"2024-09-13T17:22:37.899163+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:54.045591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898454+0000","last_change":"2024-09-13T17:22:35.738905+0000","last_active":"2024-09-13T17:22:37.898454+0000","last_peered":"2024-09-13T17:22:37.898454+0000","last_clean":"2024-09-13T17:22:37.898454+0000","last_became_active":"2024-09-13T17:22:35.738655+0000","last_became_peered":"2024-09-13T17:22:35.738655+0000","last_unstale":"2024-09-13T17:22:37.898454+0000","last_undegraded":"2024-09-13T17:22:37.898454+0000","last_fullsized":"2024-09-13T17:22:37.898454+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:20.592424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725561+0000","last_change":"2024-09-13T17:22:35.748735+0000","last_active":"2024-09-13T17:22:36.725561+0000","last_peered":"2024-09-13T17:22:36.725561+0000","last_clean":"2024-09-13T17:22:36.725561+0000","last_became_active":"2024-09-13T17:22:35.748483+0000","last_became_peered":"2024-09-13T17:22:35.748483+0000","last_unstale":"2024-09-13T17:22:36.725561+0000","last_undegraded":"2024-09-13T17:22:36.725561+0000","last_fullsized":"2024-09-13T17:22:36.725561+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:53.627485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898006+0000","last_change":"2024-09-13T17:22:35.745720+0000","last_active":"2024-09-13T17:22:37.898006+0000","last_peered":"2024-09-13T17:22:37.898006+0000","last_clean":"2024-09-13T17:22:37.898006+0000","last_became_active":"2024-09-13T17:22:35.745457+0000","last_became_peered":"2024-09-13T17:22:35.745457+0000","last_unstale":"2024-09-13T17:22:37.898006+0000","last_undegraded":"2024-09-13T17:22:37.898006+0000","last_fullsized":"2024-09-13T17:22:37.898006+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:19.936403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723299+0000","last_change":"2024-09-13T17:22:35.742999+0000","last_active":"2024-09-13T17:22:36.723299+0000","last_peered":"2024-09-13T17:22:36.723299+0000","last_clean":"2024-09-13T17:22:36.723299+0000","last_became_active":"2024-09-13T17:22:35.739528+0000","last_became_peered":"2024-09-13T17:22:35.739528+0000","last_unstale":"2024-09-13T17:22:36.723299+0000","last_undegraded":"2024-09-13T17:22:36.723299+0000","last_fullsized":"2024-09-13T17:22:36.723299+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:45.650239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897377+0000","last_change":"2024-09-13T17:22:35.743379+0000","last_active":"2024-09-13T17:22:37.897377+0000","last_peered":"2024-09-13T17:22:37.897377+0000","last_clean":"2024-09-13T17:22:37.897377+0000","last_became_active":"2024-09-13T17:22:35.743148+0000","last_became_peered":"2024-09-13T17:22:35.743148+0000","last_unstale":"2024-09-13T17:22:37.897377+0000","last_undegraded":"2024-09-13T17:22:37.897377+0000","last_fullsized":"2024-09-13T17:22:37.897377+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:47.789552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898938+0000","last_change":"2024-09-13T17:22:35.747097+0000","last_active":"2024-09-13T17:22:37.898938+0000","last_peered":"2024-09-13T17:22:37.898938+0000","last_clean":"2024-09-13T17:22:37.898938+0000","last_became_active":"2024-09-13T17:22:35.746343+0000","last_became_peered":"2024-09-13T17:22:35.746343+0000","last_unstale":"2024-09-13T17:22:37.898938+0000","last_undegraded":"2024-09-13T17:22:37.898938+0000","last_fullsized":"2024-09-13T17:22:37.898938+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:22.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"204'4","reported_seq":15,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897746+0000","last_change":"2024-09-13T17:22:35.307764+0000","last_active":"2024-09-13T17:22:37.897746+0000","last_peered":"2024-09-13T17:22:37.897746+0000","last_clean":"2024-09-13T17:22:37.897746+0000","last_became_active":"2024-09-13T17:22:35.307441+0000","last_became_peered":"2024-09-13T17:22:35.307441+0000","last_unstale":"2024-09-13T17:22:37.897746+0000","last_undegraded":"2024-09-13T17:22:37.897746+0000","last_fullsized":"2024-09-13T17:22:37.897746+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:54.686512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898831+0000","last_change":"2024-09-13T17:22:35.741073+0000","last_active":"2024-09-13T17:22:37.898831+0000","last_peered":"2024-09-13T17:22:37.898831+0000","last_clean":"2024-09-13T17:22:37.898831+0000","last_became_active":"2024-09-13T17:22:35.740766+0000","last_became_peered":"2024-09-13T17:22:35.740766+0000","last_unstale":"2024-09-13T17:22:37.898831+0000","last_undegraded":"2024-09-13T17:22:37.898831+0000","last_fullsized":"2024-09-13T17:22:37.898831+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:52.634635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725766+0000","last_change":"2024-09-13T17:22:35.309913+0000","last_active":"2024-09-13T17:22:36.725766+0000","last_peered":"2024-09-13T17:22:36.725766+0000","last_clean":"2024-09-13T17:22:36.725766+0000","last_became_active":"2024-09-13T17:22:35.309599+0000","last_became_peered":"2024-09-13T17:22:35.309599+0000","last_unstale":"2024-09-13T17:22:36.725766+0000","last_undegraded":"2024-09-13T17:22:36.725766+0000","last_fullsized":"2024-09-13T17:22:36.725766+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:23.635744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.726012+0000","last_change":"2024-09-13T17:22:35.733816+0000","last_active":"2024-09-13T17:22:36.726012+0000","last_peered":"2024-09-13T17:22:36.726012+0000","last_clean":"2024-09-13T17:22:36.726012+0000","last_became_active":"2024-09-13T17:22:35.733556+0000","last_became_peered":"2024-09-13T17:22:35.733556+0000","last_unstale":"2024-09-13T17:22:36.726012+0000","last_undegraded":"2024-09-13T17:22:36.726012+0000","last_fullsized":"2024-09-13T17:22:36.726012+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1c","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725551+0000","last_change":"2024-09-13T17:22:35.308418+0000","last_active":"2024-09-13T17:22:36.725551+0000","last_peered":"2024-09-13T17:22:36.725551+0000","last_clean":"2024-09-13T17:22:36.725551+0000","last_became_active":"2024-09-13T17:22:35.308136+0000","last_became_peered":"2024-09-13T17:22:35.308136+0000","last_unstale":"2024-09-13T17:22:36.725551+0000","last_undegraded":"2024-09-13T17:22:36.725551+0000","last_fullsized":"2024-09-13T17:22:36.725551+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:14.358311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898044+0000","last_change":"2024-09-13T17:22:35.752345+0000","last_active":"2024-09-13T17:22:37.898044+0000","last_peered":"2024-09-13T17:22:37.898044+0000","last_clean":"2024-09-13T17:22:37.898044+0000","last_became_active":"2024-09-13T17:22:35.752092+0000","last_became_peered":"2024-09-13T17:22:35.752092+0000","last_unstale":"2024-09-13T17:22:37.898044+0000","last_undegraded":"2024-09-13T17:22:37.898044+0000","last_fullsized":"2024-09-13T17:22:37.898044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:40.351477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898240+0000","last_change":"2024-09-13T17:22:35.301231+0000","last_active":"2024-09-13T17:22:37.898240+0000","last_peered":"2024-09-13T17:22:37.898240+0000","last_clean":"2024-09-13T17:22:37.898240+0000","last_became_active":"2024-09-13T17:22:35.300369+0000","last_became_peered":"2024-09-13T17:22:35.300369+0000","last_unstale":"2024-09-13T17:22:37.898240+0000","last_undegraded":"2024-09-13T17:22:37.898240+0000","last_fullsized":"2024-09-13T17:22:37.898240+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:32.155000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723574+0000","last_change":"2024-09-13T17:22:35.745074+0000","last_active":"2024-09-13T17:22:36.723574+0000","last_peered":"2024-09-13T17:22:36.723574+0000","last_clean":"2024-09-13T17:22:36.723574+0000","last_became_active":"2024-09-13T17:22:35.744800+0000","last_became_peered":"2024-09-13T17:22:35.744800+0000","last_unstale":"2024-09-13T17:22:36.723574+0000","last_undegraded":"2024-09-13T17:22:36.723574+0000","last_fullsized":"2024-09-13T17:22:36.723574+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:54.045591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1a","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898426+0000","last_change":"2024-09-13T17:22:35.310902+0000","last_active":"2024-09-13T17:22:37.898426+0000","last_peered":"2024-09-13T17:22:37.898426+0000","last_clean":"2024-09-13T17:22:37.898426+0000","last_became_active":"2024-09-13T17:22:35.310593+0000","last_became_peered":"2024-09-13T17:22:35.310593+0000","last_unstale":"2024-09-13T17:22:37.898426+0000","last_undegraded":"2024-09-13T17:22:37.898426+0000","last_fullsized":"2024-09-13T17:22:37.898426+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:37.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897749+0000","last_change":"2024-09-13T17:22:35.736675+0000","last_active":"2024-09-13T17:22:37.897749+0000","last_peered":"2024-09-13T17:22:37.897749+0000","last_clean":"2024-09-13T17:22:37.897749+0000","last_became_active":"2024-09-13T17:22:35.735106+0000","last_became_peered":"2024-09-13T17:22:35.735106+0000","last_unstale":"2024-09-13T17:22:37.897749+0000","last_undegraded":"2024-09-13T17:22:37.897749+0000","last_fullsized":"2024-09-13T17:22:37.897749+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:51.481580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725750+0000","last_change":"2024-09-13T17:22:35.307843+0000","last_active":"2024-09-13T17:22:36.725750+0000","last_peered":"2024-09-13T17:22:36.725750+0000","last_clean":"2024-09-13T17:22:36.725750+0000","last_became_active":"2024-09-13T17:22:35.307484+0000","last_became_peered":"2024-09-13T17:22:35.307484+0000","last_unstale":"2024-09-13T17:22:36.725750+0000","last_undegraded":"2024-09-13T17:22:36.725750+0000","last_fullsized":"2024-09-13T17:22:36.725750+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:51.984618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724869+0000","last_change":"2024-09-13T17:22:35.742152+0000","last_active":"2024-09-13T17:22:36.724869+0000","last_peered":"2024-09-13T17:22:36.724869+0000","last_clean":"2024-09-13T17:22:36.724869+0000","last_became_active":"2024-09-13T17:22:35.741746+0000","last_became_peered":"2024-09-13T17:22:35.741746+0000","last_unstale":"2024-09-13T17:22:36.724869+0000","last_undegraded":"2024-09-13T17:22:36.724869+0000","last_fullsized":"2024-09-13T17:22:36.724869+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.032697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897820+0000","last_change":"2024-09-13T17:22:35.305822+0000","last_active":"2024-09-13T17:22:37.897820+0000","last_peered":"2024-09-13T17:22:37.897820+0000","last_clean":"2024-09-13T17:22:37.897820+0000","last_became_active":"2024-09-13T17:22:35.305512+0000","last_became_peered":"2024-09-13T17:22:35.305512+0000","last_unstale":"2024-09-13T17:22:37.897820+0000","last_undegraded":"2024-09-13T17:22:37.897820+0000","last_fullsized":"2024-09-13T17:22:37.897820+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:18.523079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725734+0000","last_change":"2024-09-13T17:22:35.739477+0000","last_active":"2024-09-13T17:22:36.725734+0000","last_peered":"2024-09-13T17:22:36.725734+0000","last_clean":"2024-09-13T17:22:36.725734+0000","last_became_active":"2024-09-13T17:22:35.737306+0000","last_became_peered":"2024-09-13T17:22:35.737306+0000","last_unstale":"2024-09-13T17:22:36.725734+0000","last_undegraded":"2024-09-13T17:22:36.725734+0000","last_fullsized":"2024-09-13T17:22:36.725734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:03.255217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899613+0000","last_change":"2024-09-13T17:22:35.301090+0000","last_active":"2024-09-13T17:22:37.899613+0000","last_peered":"2024-09-13T17:22:37.899613+0000","last_clean":"2024-09-13T17:22:37.899613+0000","last_became_active":"2024-09-13T17:22:35.300652+0000","last_became_peered":"2024-09-13T17:22:35.300652+0000","last_unstale":"2024-09-13T17:22:37.899613+0000","last_undegraded":"2024-09-13T17:22:37.899613+0000","last_fullsized":"2024-09-13T17:22:37.899613+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:01.543679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899619+0000","last_change":"2024-09-13T17:22:35.741998+0000","last_active":"2024-09-13T17:22:37.899619+0000","last_peered":"2024-09-13T17:22:37.899619+0000","last_clean":"2024-09-13T17:22:37.899619+0000","last_became_active":"2024-09-13T17:22:35.741723+0000","last_became_peered":"2024-09-13T17:22:35.741723+0000","last_unstale":"2024-09-13T17:22:37.899619+0000","last_undegraded":"2024-09-13T17:22:37.899619+0000","last_fullsized":"2024-09-13T17:22:37.899619+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:03.239874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897888+0000","last_change":"2024-09-13T17:22:35.302438+0000","last_active":"2024-09-13T17:22:37.897888+0000","last_peered":"2024-09-13T17:22:37.897888+0000","last_clean":"2024-09-13T17:22:37.897888+0000","last_became_active":"2024-09-13T17:22:35.302196+0000","last_became_peered":"2024-09-13T17:22:35.302196+0000","last_unstale":"2024-09-13T17:22:37.897888+0000","last_undegraded":"2024-09-13T17:22:37.897888+0000","last_fullsized":"2024-09-13T17:22:37.897888+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:45.784941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725919+0000","last_change":"2024-09-13T17:22:35.739448+0000","last_active":"2024-09-13T17:22:36.725919+0000","last_peered":"2024-09-13T17:22:36.725919+0000","last_clean":"2024-09-13T17:22:36.725919+0000","last_became_active":"2024-09-13T17:22:35.739167+0000","last_became_peered":"2024-09-13T17:22:35.739167+0000","last_unstale":"2024-09-13T17:22:36.725919+0000","last_undegraded":"2024-09-13T17:22:36.725919+0000","last_fullsized":"2024-09-13T17:22:36.725919+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:27.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.15","version":"204'3","reported_seq":14,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723873+0000","last_change":"2024-09-13T17:22:35.298661+0000","last_active":"2024-09-13T17:22:36.723873+0000","last_peered":"2024-09-13T17:22:36.723873+0000","last_clean":"2024-09-13T17:22:36.723873+0000","last_became_active":"2024-09-13T17:22:35.298396+0000","last_became_peered":"2024-09-13T17:22:35.298396+0000","last_unstale":"2024-09-13T17:22:36.723873+0000","last_undegraded":"2024-09-13T17:22:36.723873+0000","last_fullsized":"2024-09-13T17:22:36.723873+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:51.984618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899065+0000","last_change":"2024-09-13T17:22:35.741773+0000","last_active":"2024-09-13T17:22:37.899065+0000","last_peered":"2024-09-13T17:22:37.899065+0000","last_clean":"2024-09-13T17:22:37.899065+0000","last_became_active":"2024-09-13T17:22:35.741532+0000","last_became_peered":"2024-09-13T17:22:35.741532+0000","last_unstale":"2024-09-13T17:22:37.899065+0000","last_undegraded":"2024-09-13T17:22:37.899065+0000","last_fullsized":"2024-09-13T17:22:37.899065+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:03.255217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"204'6","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898519+0000","last_change":"2024-09-13T17:22:35.305124+0000","last_active":"2024-09-13T17:22:37.898519+0000","last_peered":"2024-09-13T17:22:37.898519+0000","last_clean":"2024-09-13T17:22:37.898519+0000","last_became_active":"2024-09-13T17:22:35.304822+0000","last_became_peered":"2024-09-13T17:22:35.304822+0000","last_unstale":"2024-09-13T17:22:37.898519+0000","last_undegraded":"2024-09-13T17:22:37.898519+0000","last_fullsized":"2024-09-13T17:22:37.898519+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:50.621747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897843+0000","last_change":"2024-09-13T17:22:35.735874+0000","last_active":"2024-09-13T17:22:37.897843+0000","last_peered":"2024-09-13T17:22:37.897843+0000","last_clean":"2024-09-13T17:22:37.897843+0000","last_became_active":"2024-09-13T17:22:35.735418+0000","last_became_peered":"2024-09-13T17:22:35.735418+0000","last_unstale":"2024-09-13T17:22:37.897843+0000","last_undegraded":"2024-09-13T17:22:37.897843+0000","last_fullsized":"2024-09-13T17:22:37.897843+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:03.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898535+0000","last_change":"2024-09-13T17:22:35.310730+0000","last_active":"2024-09-13T17:22:37.898535+0000","last_peered":"2024-09-13T17:22:37.898535+0000","last_clean":"2024-09-13T17:22:37.898535+0000","last_became_active":"2024-09-13T17:22:35.310422+0000","last_became_peered":"2024-09-13T17:22:35.310422+0000","last_unstale":"2024-09-13T17:22:37.898535+0000","last_undegraded":"2024-09-13T17:22:37.898535+0000","last_fullsized":"2024-09-13T17:22:37.898535+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:25.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898581+0000","last_change":"2024-09-13T17:22:35.751448+0000","last_active":"2024-09-13T17:22:37.898581+0000","last_peered":"2024-09-13T17:22:37.898581+0000","last_clean":"2024-09-13T17:22:37.898581+0000","last_became_active":"2024-09-13T17:22:35.747909+0000","last_became_peered":"2024-09-13T17:22:35.747909+0000","last_unstale":"2024-09-13T17:22:37.898581+0000","last_undegraded":"2024-09-13T17:22:37.898581+0000","last_fullsized":"2024-09-13T17:22:37.898581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.818485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897726+0000","last_change":"2024-09-13T17:22:35.300098+0000","last_active":"2024-09-13T17:22:37.897726+0000","last_peered":"2024-09-13T17:22:37.897726+0000","last_clean":"2024-09-13T17:22:37.897726+0000","last_became_active":"2024-09-13T17:22:35.299874+0000","last_became_peered":"2024-09-13T17:22:35.299874+0000","last_unstale":"2024-09-13T17:22:37.897726+0000","last_undegraded":"2024-09-13T17:22:37.897726+0000","last_fullsized":"2024-09-13T17:22:37.897726+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:47.059931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898061+0000","last_change":"2024-09-13T17:22:35.736993+0000","last_active":"2024-09-13T17:22:37.898061+0000","last_peered":"2024-09-13T17:22:37.898061+0000","last_clean":"2024-09-13T17:22:37.898061+0000","last_became_active":"2024-09-13T17:22:35.736667+0000","last_became_peered":"2024-09-13T17:22:35.736667+0000","last_unstale":"2024-09-13T17:22:37.898061+0000","last_undegraded":"2024-09-13T17:22:37.898061+0000","last_fullsized":"2024-09-13T17:22:37.898061+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723516+0000","last_change":"2024-09-13T17:22:35.300153+0000","last_active":"2024-09-13T17:22:36.723516+0000","last_peered":"2024-09-13T17:22:36.723516+0000","last_clean":"2024-09-13T17:22:36.723516+0000","last_became_active":"2024-09-13T17:22:35.299865+0000","last_became_peered":"2024-09-13T17:22:35.299865+0000","last_unstale":"2024-09-13T17:22:36.723516+0000","last_undegraded":"2024-09-13T17:22:36.723516+0000","last_fullsized":"2024-09-13T17:22:36.723516+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:59.466116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725159+0000","last_change":"2024-09-13T17:22:35.731600+0000","last_active":"2024-09-13T17:22:36.725159+0000","last_peered":"2024-09-13T17:22:36.725159+0000","last_clean":"2024-09-13T17:22:36.725159+0000","last_became_active":"2024-09-13T17:22:35.731316+0000","last_became_peered":"2024-09-13T17:22:35.731316+0000","last_unstale":"2024-09-13T17:22:36.725159+0000","last_undegraded":"2024-09-13T17:22:36.725159+0000","last_fullsized":"2024-09-13T17:22:36.725159+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.10","version":"204'8","reported_seq":19,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724771+0000","last_change":"2024-09-13T17:22:35.308164+0000","last_active":"2024-09-13T17:22:36.724771+0000","last_peered":"2024-09-13T17:22:36.724771+0000","last_clean":"2024-09-13T17:22:36.724771+0000","last_became_active":"2024-09-13T17:22:35.307885+0000","last_became_peered":"2024-09-13T17:22:35.307885+0000","last_unstale":"2024-09-13T17:22:36.724771+0000","last_undegraded":"2024-09-13T17:22:36.724771+0000","last_fullsized":"2024-09-13T17:22:36.724771+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:57.846901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898635+0000","last_change":"2024-09-13T17:22:35.748387+0000","last_active":"2024-09-13T17:22:37.898635+0000","last_peered":"2024-09-13T17:22:37.898635+0000","last_clean":"2024-09-13T17:22:37.898635+0000","last_became_active":"2024-09-13T17:22:35.747839+0000","last_became_peered":"2024-09-13T17:22:35.747839+0000","last_unstale":"2024-09-13T17:22:37.898635+0000","last_undegraded":"2024-09-13T17:22:37.898635+0000","last_fullsized":"2024-09-13T17:22:37.898635+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:26.415107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899015+0000","last_change":"2024-09-13T17:22:35.311457+0000","last_active":"2024-09-13T17:22:37.899015+0000","last_peered":"2024-09-13T17:22:37.899015+0000","last_clean":"2024-09-13T17:22:37.899015+0000","last_became_active":"2024-09-13T17:22:35.310899+0000","last_became_peered":"2024-09-13T17:22:35.310899+0000","last_unstale":"2024-09-13T17:22:37.899015+0000","last_undegraded":"2024-09-13T17:22:37.899015+0000","last_fullsized":"2024-09-13T17:22:37.899015+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:08.704735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898633+0000","last_change":"2024-09-13T17:22:35.742509+0000","last_active":"2024-09-13T17:22:37.898633+0000","last_peered":"2024-09-13T17:22:37.898633+0000","last_clean":"2024-09-13T17:22:37.898633+0000","last_became_active":"2024-09-13T17:22:35.742245+0000","last_became_peered":"2024-09-13T17:22:35.742245+0000","last_unstale":"2024-09-13T17:22:37.898633+0000","last_undegraded":"2024-09-13T17:22:37.898633+0000","last_fullsized":"2024-09-13T17:22:37.898633+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:29:04.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897951+0000","last_change":"2024-09-13T17:22:35.308074+0000","last_active":"2024-09-13T17:22:37.897951+0000","last_peered":"2024-09-13T17:22:37.897951+0000","last_clean":"2024-09-13T17:22:37.897951+0000","last_became_active":"2024-09-13T17:22:35.307425+0000","last_became_peered":"2024-09-13T17:22:35.307425+0000","last_unstale":"2024-09-13T17:22:37.897951+0000","last_undegraded":"2024-09-13T17:22:37.897951+0000","last_fullsized":"2024-09-13T17:22:37.897951+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:00.990352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898633+0000","last_change":"2024-09-13T17:22:35.740876+0000","last_active":"2024-09-13T17:22:37.898633+0000","last_peered":"2024-09-13T17:22:37.898633+0000","last_clean":"2024-09-13T17:22:37.898633+0000","last_became_active":"2024-09-13T17:22:35.740579+0000","last_became_peered":"2024-09-13T17:22:35.740579+0000","last_unstale":"2024-09-13T17:22:37.898633+0000","last_undegraded":"2024-09-13T17:22:37.898633+0000","last_fullsized":"2024-09-13T17:22:37.898633+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:15.361193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.d","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899256+0000","last_change":"2024-09-13T17:22:35.303117+0000","last_active":"2024-09-13T17:22:37.899256+0000","last_peered":"2024-09-13T17:22:37.899256+0000","last_clean":"2024-09-13T17:22:37.899256+0000","last_became_active":"2024-09-13T17:22:35.302851+0000","last_became_peered":"2024-09-13T17:22:35.302851+0000","last_unstale":"2024-09-13T17:22:37.899256+0000","last_undegraded":"2024-09-13T17:22:37.899256+0000","last_fullsized":"2024-09-13T17:22:37.899256+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:22.612572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899110+0000","last_change":"2024-09-13T17:22:35.740937+0000","last_active":"2024-09-13T17:22:37.899110+0000","last_peered":"2024-09-13T17:22:37.899110+0000","last_clean":"2024-09-13T17:22:37.899110+0000","last_became_active":"2024-09-13T17:22:35.740664+0000","last_became_peered":"2024-09-13T17:22:35.740664+0000","last_unstale":"2024-09-13T17:22:37.899110+0000","last_undegraded":"2024-09-13T17:22:37.899110+0000","last_fullsized":"2024-09-13T17:22:37.899110+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898566+0000","last_change":"2024-09-13T17:22:35.301632+0000","last_active":"2024-09-13T17:22:37.898566+0000","last_peered":"2024-09-13T17:22:37.898566+0000","last_clean":"2024-09-13T17:22:37.898566+0000","last_became_active":"2024-09-13T17:22:35.300947+0000","last_became_peered":"2024-09-13T17:22:35.300947+0000","last_unstale":"2024-09-13T17:22:37.898566+0000","last_undegraded":"2024-09-13T17:22:37.898566+0000","last_fullsized":"2024-09-13T17:22:37.898566+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:28.037447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723332+0000","last_change":"2024-09-13T17:22:35.732821+0000","last_active":"2024-09-13T17:22:36.723332+0000","last_peered":"2024-09-13T17:22:36.723332+0000","last_clean":"2024-09-13T17:22:36.723332+0000","last_became_active":"2024-09-13T17:22:35.732571+0000","last_became_peered":"2024-09-13T17:22:35.732571+0000","last_unstale":"2024-09-13T17:22:36.723332+0000","last_undegraded":"2024-09-13T17:22:36.723332+0000","last_fullsized":"2024-09-13T17:22:36.723332+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:47.004276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.b","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899076+0000","last_change":"2024-09-13T17:22:35.304871+0000","last_active":"2024-09-13T17:22:37.899076+0000","last_peered":"2024-09-13T17:22:37.899076+0000","last_clean":"2024-09-13T17:22:37.899076+0000","last_became_active":"2024-09-13T17:22:35.304338+0000","last_became_peered":"2024-09-13T17:22:35.304338+0000","last_unstale":"2024-09-13T17:22:37.899076+0000","last_undegraded":"2024-09-13T17:22:37.899076+0000","last_fullsized":"2024-09-13T17:22:37.899076+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:24.505838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898632+0000","last_change":"2024-09-13T17:22:35.750909+0000","last_active":"2024-09-13T17:22:37.898632+0000","last_peered":"2024-09-13T17:22:37.898632+0000","last_clean":"2024-09-13T17:22:37.898632+0000","last_became_active":"2024-09-13T17:22:35.746385+0000","last_became_peered":"2024-09-13T17:22:35.746385+0000","last_unstale":"2024-09-13T17:22:37.898632+0000","last_undegraded":"2024-09-13T17:22:37.898632+0000","last_fullsized":"2024-09-13T17:22:37.898632+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:53.627485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725647+0000","last_change":"2024-09-13T17:22:35.307125+0000","last_active":"2024-09-13T17:22:36.725647+0000","last_peered":"2024-09-13T17:22:36.725647+0000","last_clean":"2024-09-13T17:22:36.725647+0000","last_became_active":"2024-09-13T17:22:35.306846+0000","last_became_peered":"2024-09-13T17:22:35.306846+0000","last_unstale":"2024-09-13T17:22:36.725647+0000","last_undegraded":"2024-09-13T17:22:36.725647+0000","last_fullsized":"2024-09-13T17:22:36.725647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:13.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898133+0000","last_change":"2024-09-13T17:22:35.745947+0000","last_active":"2024-09-13T17:22:37.898133+0000","last_peered":"2024-09-13T17:22:37.898133+0000","last_clean":"2024-09-13T17:22:37.898133+0000","last_became_active":"2024-09-13T17:22:35.745697+0000","last_became_peered":"2024-09-13T17:22:35.745697+0000","last_unstale":"2024-09-13T17:22:37.898133+0000","last_undegraded":"2024-09-13T17:22:37.898133+0000","last_fullsized":"2024-09-13T17:22:37.898133+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:07.078447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725896+0000","last_change":"2024-09-13T17:22:35.306034+0000","last_active":"2024-09-13T17:22:36.725896+0000","last_peered":"2024-09-13T17:22:36.725896+0000","last_clean":"2024-09-13T17:22:36.725896+0000","last_became_active":"2024-09-13T17:22:35.305762+0000","last_became_peered":"2024-09-13T17:22:35.305762+0000","last_unstale":"2024-09-13T17:22:36.725896+0000","last_undegraded":"2024-09-13T17:22:36.725896+0000","last_fullsized":"2024-09-13T17:22:36.725896+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:53:34.294075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723405+0000","last_change":"2024-09-13T17:22:35.750742+0000","last_active":"2024-09-13T17:22:36.723405+0000","last_peered":"2024-09-13T17:22:36.723405+0000","last_clean":"2024-09-13T17:22:36.723405+0000","last_became_active":"2024-09-13T17:22:35.750452+0000","last_became_peered":"2024-09-13T17:22:35.750452+0000","last_unstale":"2024-09-13T17:22:36.723405+0000","last_undegraded":"2024-09-13T17:22:36.723405+0000","last_fullsized":"2024-09-13T17:22:36.723405+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.818485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898147+0000","last_change":"2024-09-13T17:22:35.309572+0000","last_active":"2024-09-13T17:22:37.898147+0000","last_peered":"2024-09-13T17:22:37.898147+0000","last_clean":"2024-09-13T17:22:37.898147+0000","last_became_active":"2024-09-13T17:22:35.309265+0000","last_became_peered":"2024-09-13T17:22:35.309265+0000","last_unstale":"2024-09-13T17:22:37.898147+0000","last_undegraded":"2024-09-13T17:22:37.898147+0000","last_fullsized":"2024-09-13T17:22:37.898147+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:54.554007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723672+0000","last_change":"2024-09-13T17:22:35.744675+0000","last_active":"2024-09-13T17:22:36.723672+0000","last_peered":"2024-09-13T17:22:36.723672+0000","last_clean":"2024-09-13T17:22:36.723672+0000","last_became_active":"2024-09-13T17:22:35.743943+0000","last_became_peered":"2024-09-13T17:22:35.743943+0000","last_unstale":"2024-09-13T17:22:36.723672+0000","last_undegraded":"2024-09-13T17:22:36.723672+0000","last_fullsized":"2024-09-13T17:22:36.723672+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:20.076222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.7","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723952+0000","last_change":"2024-09-13T17:22:35.309954+0000","last_active":"2024-09-13T17:22:36.723952+0000","last_peered":"2024-09-13T17:22:36.723952+0000","last_clean":"2024-09-13T17:22:36.723952+0000","last_became_active":"2024-09-13T17:22:35.309609+0000","last_became_peered":"2024-09-13T17:22:35.309609+0000","last_unstale":"2024-09-13T17:22:36.723952+0000","last_undegraded":"2024-09-13T17:22:36.723952+0000","last_fullsized":"2024-09-13T17:22:36.723952+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:15.870289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898910+0000","last_change":"2024-09-13T17:22:35.754217+0000","last_active":"2024-09-13T17:22:37.898910+0000","last_peered":"2024-09-13T17:22:37.898910+0000","last_clean":"2024-09-13T17:22:37.898910+0000","last_became_active":"2024-09-13T17:22:35.753624+0000","last_became_peered":"2024-09-13T17:22:35.753624+0000","last_unstale":"2024-09-13T17:22:37.898910+0000","last_undegraded":"2024-09-13T17:22:37.898910+0000","last_fullsized":"2024-09-13T17:22:37.898910+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899218+0000","last_change":"2024-09-13T17:22:35.303858+0000","last_active":"2024-09-13T17:22:37.899218+0000","last_peered":"2024-09-13T17:22:37.899218+0000","last_clean":"2024-09-13T17:22:37.899218+0000","last_became_active":"2024-09-13T17:22:35.303616+0000","last_became_peered":"2024-09-13T17:22:35.303616+0000","last_unstale":"2024-09-13T17:22:37.899218+0000","last_undegraded":"2024-09-13T17:22:37.899218+0000","last_fullsized":"2024-09-13T17:22:37.899218+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:52.641424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898734+0000","last_change":"2024-09-13T17:22:35.749712+0000","last_active":"2024-09-13T17:22:37.898734+0000","last_peered":"2024-09-13T17:22:37.898734+0000","last_clean":"2024-09-13T17:22:37.898734+0000","last_became_active":"2024-09-13T17:22:35.747984+0000","last_became_peered":"2024-09-13T17:22:35.747984+0000","last_unstale":"2024-09-13T17:22:37.898734+0000","last_undegraded":"2024-09-13T17:22:37.898734+0000","last_fullsized":"2024-09-13T17:22:37.898734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:04.322380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899402+0000","last_change":"2024-09-13T17:22:35.306389+0000","last_active":"2024-09-13T17:22:37.899402+0000","last_peered":"2024-09-13T17:22:37.899402+0000","last_clean":"2024-09-13T17:22:37.899402+0000","last_became_active":"2024-09-13T17:22:35.305034+0000","last_became_peered":"2024-09-13T17:22:35.305034+0000","last_unstale":"2024-09-13T17:22:37.899402+0000","last_undegraded":"2024-09-13T17:22:37.899402+0000","last_fullsized":"2024-09-13T17:22:37.899402+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:06.360515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723779+0000","last_change":"2024-09-13T17:22:35.735804+0000","last_active":"2024-09-13T17:22:36.723779+0000","last_peered":"2024-09-13T17:22:36.723779+0000","last_clean":"2024-09-13T17:22:36.723779+0000","last_became_active":"2024-09-13T17:22:35.735543+0000","last_became_peered":"2024-09-13T17:22:35.735543+0000","last_unstale":"2024-09-13T17:22:36.723779+0000","last_undegraded":"2024-09-13T17:22:36.723779+0000","last_fullsized":"2024-09-13T17:22:36.723779+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898840+0000","last_change":"2024-09-13T17:22:35.307024+0000","last_active":"2024-09-13T17:22:37.898840+0000","last_peered":"2024-09-13T17:22:37.898840+0000","last_clean":"2024-09-13T17:22:37.898840+0000","last_became_active":"2024-09-13T17:22:35.306767+0000","last_became_peered":"2024-09-13T17:22:35.306767+0000","last_unstale":"2024-09-13T17:22:37.898840+0000","last_undegraded":"2024-09-13T17:22:37.898840+0000","last_fullsized":"2024-09-13T17:22:37.898840+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:19.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725009+0000","last_change":"2024-09-13T17:22:35.730715+0000","last_active":"2024-09-13T17:22:36.725009+0000","last_peered":"2024-09-13T17:22:36.725009+0000","last_clean":"2024-09-13T17:22:36.725009+0000","last_became_active":"2024-09-13T17:22:35.730032+0000","last_became_peered":"2024-09-13T17:22:35.730032+0000","last_unstale":"2024-09-13T17:22:36.725009+0000","last_undegraded":"2024-09-13T17:22:36.725009+0000","last_fullsized":"2024-09-13T17:22:36.725009+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:19.936403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725647+0000","last_change":"2024-09-13T17:22:35.294050+0000","last_active":"2024-09-13T17:22:36.725647+0000","last_peered":"2024-09-13T17:22:36.725647+0000","last_clean":"2024-09-13T17:22:36.725647+0000","last_became_active":"2024-09-13T17:22:35.293757+0000","last_became_peered":"2024-09-13T17:22:35.293757+0000","last_unstale":"2024-09-13T17:22:36.725647+0000","last_undegraded":"2024-09-13T17:22:36.725647+0000","last_fullsized":"2024-09-13T17:22:36.725647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:15.870289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898909+0000","last_change":"2024-09-13T17:22:35.752284+0000","last_active":"2024-09-13T17:22:37.898909+0000","last_peered":"2024-09-13T17:22:37.898909+0000","last_clean":"2024-09-13T17:22:37.898909+0000","last_became_active":"2024-09-13T17:22:35.747576+0000","last_became_peered":"2024-09-13T17:22:35.747576+0000","last_unstale":"2024-09-13T17:22:37.898909+0000","last_undegraded":"2024-09-13T17:22:37.898909+0000","last_fullsized":"2024-09-13T17:22:37.898909+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:45.650239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897642+0000","last_change":"2024-09-13T17:22:35.303704+0000","last_active":"2024-09-13T17:22:37.897642+0000","last_peered":"2024-09-13T17:22:37.897642+0000","last_clean":"2024-09-13T17:22:37.897642+0000","last_became_active":"2024-09-13T17:22:35.301222+0000","last_became_peered":"2024-09-13T17:22:35.301222+0000","last_unstale":"2024-09-13T17:22:37.897642+0000","last_undegraded":"2024-09-13T17:22:37.897642+0000","last_fullsized":"2024-09-13T17:22:37.897642+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:05.823909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898125+0000","last_change":"2024-09-13T17:22:35.753631+0000","last_active":"2024-09-13T17:22:37.898125+0000","last_peered":"2024-09-13T17:22:37.898125+0000","last_clean":"2024-09-13T17:22:37.898125+0000","last_became_active":"2024-09-13T17:22:35.753250+0000","last_became_peered":"2024-09-13T17:22:35.753250+0000","last_unstale":"2024-09-13T17:22:37.898125+0000","last_undegraded":"2024-09-13T17:22:37.898125+0000","last_fullsized":"2024-09-13T17:22:37.898125+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723609+0000","last_change":"2024-09-13T17:22:35.309663+0000","last_active":"2024-09-13T17:22:36.723609+0000","last_peered":"2024-09-13T17:22:36.723609+0000","last_clean":"2024-09-13T17:22:36.723609+0000","last_became_active":"2024-09-13T17:22:35.309103+0000","last_became_peered":"2024-09-13T17:22:35.309103+0000","last_unstale":"2024-09-13T17:22:36.723609+0000","last_undegraded":"2024-09-13T17:22:36.723609+0000","last_fullsized":"2024-09-13T17:22:36.723609+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:57.167129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898006+0000","last_change":"2024-09-13T17:22:35.744716+0000","last_active":"2024-09-13T17:22:37.898006+0000","last_peered":"2024-09-13T17:22:37.898006+0000","last_clean":"2024-09-13T17:22:37.898006+0000","last_became_active":"2024-09-13T17:22:35.744397+0000","last_became_peered":"2024-09-13T17:22:35.744397+0000","last_unstale":"2024-09-13T17:22:37.898006+0000","last_undegraded":"2024-09-13T17:22:37.898006+0000","last_fullsized":"2024-09-13T17:22:37.898006+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:00.326830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898210+0000","last_change":"2024-09-13T17:22:35.307927+0000","last_active":"2024-09-13T17:22:37.898210+0000","last_peered":"2024-09-13T17:22:37.898210+0000","last_clean":"2024-09-13T17:22:37.898210+0000","last_became_active":"2024-09-13T17:22:35.307680+0000","last_became_peered":"2024-09-13T17:22:35.307680+0000","last_unstale":"2024-09-13T17:22:37.898210+0000","last_undegraded":"2024-09-13T17:22:37.898210+0000","last_fullsized":"2024-09-13T17:22:37.898210+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:22.736023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898715+0000","last_change":"2024-09-13T17:22:35.743292+0000","last_active":"2024-09-13T17:22:37.898715+0000","last_peered":"2024-09-13T17:22:37.898715+0000","last_clean":"2024-09-13T17:22:37.898715+0000","last_became_active":"2024-09-13T17:22:35.743045+0000","last_became_peered":"2024-09-13T17:22:35.743045+0000","last_unstale":"2024-09-13T17:22:37.898715+0000","last_undegraded":"2024-09-13T17:22:37.898715+0000","last_fullsized":"2024-09-13T17:22:37.898715+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:20.076222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.726050+0000","last_change":"2024-09-13T17:22:35.306449+0000","last_active":"2024-09-13T17:22:36.726050+0000","last_peered":"2024-09-13T17:22:36.726050+0000","last_clean":"2024-09-13T17:22:36.726050+0000","last_became_active":"2024-09-13T17:22:35.306096+0000","last_became_peered":"2024-09-13T17:22:35.306096+0000","last_unstale":"2024-09-13T17:22:36.726050+0000","last_undegraded":"2024-09-13T17:22:36.726050+0000","last_fullsized":"2024-09-13T17:22:36.726050+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:54.367489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-13T17:22:35.728472+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:22:35.728472+0000","last_peered":"2024-09-13T17:22:35.728472+0000","last_clean":"2024-09-13T17:22:35.728472+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:22:35.728472+0000","last_undegraded":"2024-09-13T17:22:35.728472+0000","last_fullsized":"2024-09-13T17:22:35.728472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575027,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70252,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301588,"statfs":{"total":96636764160,"available":96564826112,"internally_reserved":0,"allocated":3170304,"data_stored":1726058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63300000000000001,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.85199999999999998,"5min":1.008,"15min":2.0880000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.316},"max":{"1min":0.873,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.91800000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.66900000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89500000000000002,"5min":1.161,"15min":2.5030000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.52800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86099999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.85499999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.68799999999999994},"min":{"1min":0.54900000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89100000000000001,"5min":1.169,"15min":1.544},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68100000000000005,"15min":0.69599999999999995},"min":{"1min":0.621,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.90100000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.95399999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.499,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81999999999999995,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.746,"5min":0.70799999999999996,"15min":0.73399999999999999},"min":{"1min":0.59999999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":1.274,"15min":2.8239999999999998},"last":0.81699999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73299999999999998,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64000000000000001,"15min":0.65500000000000003},"min":{"1min":0.44900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84799999999999998,"5min":1.234,"15min":1.234},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.88200000000000001,"5min":1.135,"15min":1.143},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":1.915,"15min":1.915},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.93300000000000005,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70699999999999996,"15min":0.73099999999999998},"min":{"1min":0.56499999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.94499999999999995,"5min":1.607,"15min":1.607},"last":0.70699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575028,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65580,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306260,"statfs":{"total":96636764160,"available":96569610240,"internally_reserved":0,"allocated":2580480,"data_stored":1135676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.47199999999999998,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.97099999999999997,"5min":1.0209999999999999,"15min":3.6629999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.74099999999999999},"min":{"1min":0.61199999999999999,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.95599999999999996,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.68300000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63200000000000001,"15min":0.65800000000000003},"min":{"1min":0.496,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.998,"5min":0.998,"15min":3.7040000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.67100000000000004},"min":{"1min":0.59199999999999997,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.871,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71699999999999997,"15min":0.68200000000000005},"min":{"1min":0.56699999999999995,"5min":0.44,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.58599999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.92000000000000004,"5min":0.93300000000000005,"15min":3.2629999999999999},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69099999999999995,"15min":0.70799999999999996},"min":{"1min":0.57099999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93500000000000005,"5min":1.167,"15min":3.6520000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.51200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.262,"15min":1.6439999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61799999999999999,"15min":0.64300000000000002},"min":{"1min":0.497,"5min":0.40400000000000003,"15min":0.29199999999999998},"max":{"1min":0.88400000000000001,"5min":0.90900000000000003,"15min":4.7750000000000004},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.52400000000000002,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":0.83799999999999997,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.625,"15min":0.66400000000000003},"min":{"1min":0.434,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":1.1559999999999999,"15min":3.609},"last":0.45600000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.72499999999999998},"min":{"1min":0.55400000000000005,"5min":0.379,"15min":0.378},"max":{"1min":0.86799999999999999,"5min":0.93700000000000006,"15min":4.9669999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76100000000000001,"15min":0.78300000000000003},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":0.88100000000000001,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575027,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63984,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307856,"statfs":{"total":96636764160,"available":96571244544,"internally_reserved":0,"allocated":2584576,"data_stored":1141920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.76300000000000001},"min":{"1min":0.43099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1220000000000001,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67900000000000005,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.73599999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67800000000000005,"15min":0.72399999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.755},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68700000000000006,"15min":0.69399999999999995},"min":{"1min":0.48199999999999998,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":1.169,"5min":1.482,"15min":7.7370000000000001},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.73699999999999999},"min":{"1min":0.47599999999999998,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67600000000000005,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.98599999999999999,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69999999999999996,"15min":1.081},"min":{"1min":0.505,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":113.363},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.72299999999999998,"15min":0.74099999999999999},"min":{"1min":0.55500000000000005,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.97599999999999998,"5min":1.113,"15min":4.1180000000000003},"last":0.93799999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.89200000000000002,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65100000000000002,"15min":0.67800000000000005},"min":{"1min":0.40000000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95599999999999996,"5min":1.03,"15min":2.827},"last":0.90200000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67000000000000004,"15min":0.69099999999999995},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":0.99099999999999999,"5min":1.73,"15min":2.2320000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.54100000000000004,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":1.032,"5min":1.532,"15min":4.1500000000000004},"last":0.86899999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.51800000000000002,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":1.079,"5min":1.079,"15min":4.1029999999999998},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":0.92800000000000005,"5min":1.363,"15min":2.02},"last":1.169}]}]},{"osd":4,"up_from":13,"seq":55834575026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73780,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298060,"statfs":{"total":96636764160,"available":96561213440,"internally_reserved":0,"allocated":2588672,"data_stored":1142502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.1870000000000001,"15min":0.83399999999999996},"min":{"1min":0.498,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1.2729999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.755,"5min":1.1899999999999999,"15min":0.873},"min":{"1min":0.47699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97399999999999998,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.75700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.85099999999999998,"15min":0.71999999999999997},"min":{"1min":0.49099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94599999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.72999999999999998,"15min":0.67200000000000004},"min":{"1min":0.55800000000000005,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":1.3160000000000001,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.68200000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.70299999999999996,"15min":0.67400000000000004},"min":{"1min":0.53200000000000003,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":1.333,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.58799999999999997,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.095,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.437,"15min":0.378},"max":{"1min":1.24,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.76000000000000001,"15min":0.72099999999999997},"min":{"1min":0.49399999999999999,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.96399999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69799999999999995,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83999999999999997,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63200000000000001,"15min":0.64400000000000002},"min":{"1min":0.501,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.77200000000000002,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":1.498,"5min":2.976,"15min":2.976},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1850000000000001,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.873}]},{"osd":7,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.70699999999999996},"min":{"1min":0.52900000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.76500000000000001,"15min":0.748},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.91200000000000003,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.89400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69676,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302164,"statfs":{"total":96636764160,"available":96565415936,"internally_reserved":0,"allocated":2588672,"data_stored":1137254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.308,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":0.70499999999999996,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.36399999999999999,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":2.96},"last":0.502}]},{"osd":1,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47299999999999998,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":3.5600000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.3819999999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.73399999999999999,"15min":0.747},"min":{"1min":0.60499999999999998,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.208,"5min":1.21,"15min":2.8610000000000002},"last":0.626},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.60199999999999998,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.169,"5min":1.23,"15min":1.3640000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.228,"5min":1.323,"15min":1.391},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.60099999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.401},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.79600000000000004,"15min":0.78500000000000003},"min":{"1min":0.68100000000000005,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.78300000000000003,"15min":0.79400000000000004},"min":{"1min":0.60899999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1679999999999999,"5min":1.27,"15min":2.899},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80300000000000005,"15min":0.80000000000000004},"min":{"1min":0.66700000000000004,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.264,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.68300000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47604,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324236,"statfs":{"total":96636764160,"available":96588017664,"internally_reserved":0,"allocated":2588672,"data_stored":1136221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47299999999999998,"15min":0.436},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.67500000000000004,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":1.593,"15min":30.023},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53700000000000003,"15min":1.367},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.79900000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52500000000000002,"15min":1.4650000000000001},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.316},"max":{"1min":0.94099999999999995,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.438}]},{"osd":3,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":0.62},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.84199999999999997,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.498},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67200000000000004,"15min":0.61199999999999999},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91100000000000003,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78200000000000003,"15min":0.81000000000000005},"min":{"1min":0.64800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.98799999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.85899999999999999},"min":{"1min":0.65800000000000003,"5min":0.48999999999999999,"15min":0.371},"max":{"1min":1.075,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76400000000000001,"15min":0.876},"min":{"1min":0.66100000000000003,"5min":0.44500000000000001,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79300000000000004,"15min":0.78100000000000003},"min":{"1min":0.65300000000000002,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.014,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1339999999999999,"5min":6.4800000000000004,"15min":10.664},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78700000000000003,"15min":0.89000000000000001},"min":{"1min":0.63900000000000001,"5min":0.503,"15min":0.47999999999999998},"max":{"1min":1.121,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.77800000000000002,"15min":0.82299999999999995},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.194,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.80500000000000005,"15min":0.90600000000000003},"min":{"1min":0.71299999999999997,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.095,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.71799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91828,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280012,"statfs":{"total":96636764160,"available":96542732288,"internally_reserved":0,"allocated":2588672,"data_stored":1142010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.434,"15min":0.46700000000000003},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.67000000000000004,"15min":5.4939999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.41999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.438}]},{"osd":2,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.45900000000000002},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.629,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.65600000000000003},"min":{"1min":0.47999999999999998,"5min":0.42699999999999999,"15min":0.376},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":16.382999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.75700000000000001,"15min":0.78800000000000003},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.03,"5min":1.28,"15min":6.1630000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79500000000000004,"15min":0.80400000000000005},"min":{"1min":0.625,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0009999999999999,"5min":1.23,"15min":1.78},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":1.077,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.755,"15min":0.78400000000000003},"min":{"1min":0.56100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.073,"5min":1.175,"15min":6.1189999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76700000000000002,"15min":0.78100000000000003},"min":{"1min":0.57299999999999995,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0309999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77400000000000002,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.135,"5min":1.246,"15min":6.1970000000000001},"last":0.628}]},{"osd":7,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.79200000000000004},"min":{"1min":0.58799999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.996,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80200000000000005,"15min":0.81299999999999994},"min":{"1min":0.61099999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.1830000000000001,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575028,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70252,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301588,"statfs":{"total":96636764160,"available":96564826112,"internally_reserved":0,"allocated":3170304,"data_stored":1725641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.29599999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64900000000000002,"5min":1.986,"15min":5.5259999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48599999999999999,"15min":0.53500000000000003},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.81599999999999995,"5min":2.371,"15min":5.4889999999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.49299999999999999,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.56100000000000005},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47699999999999998,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.66600000000000004,"5min":0.878,"15min":4.7549999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55100000000000005,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77000000000000002,"15min":0.79900000000000004},"min":{"1min":0.61299999999999999,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1,"5min":3.8849999999999998,"15min":4.734},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72199999999999998,"15min":0.78300000000000003},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":0.98799999999999999,"5min":1.137,"15min":4.8079999999999998},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73599999999999999,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":0.91200000000000003,"5min":1.0980000000000001,"15min":4.944},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76800000000000002,"15min":0.80600000000000005},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.026,"5min":1.139,"15min":4.9710000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.76300000000000001,"15min":0.83199999999999996},"min":{"1min":0.57999999999999996,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.0229999999999999,"5min":1.161,"15min":4.8470000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.82099999999999995},"min":{"1min":0.55000000000000004,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.1339999999999999,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.77300000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82499999999999996,"15min":0.84599999999999997},"min":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.0609999999999999,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77400000000000002,"15min":0.82699999999999996},"min":{"1min":0.56899999999999995,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":0.97399999999999998,"5min":1.1220000000000001,"15min":4.9950000000000001},"last":0.86699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:22:39.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:22:40.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.101+0000 7f56fcc00640 1 Processor -- start 2024-09-13T17:22:40.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.101+0000 7f56fcc00640 1 -- start start 2024-09-13T17:22:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56fcc00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f8154a90 0x7f56f815b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56fcc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f806b360 0x7f56f80b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56fcc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f815bb10 con 0x7f56f80b0980 2024-09-13T17:22:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56fcc00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f80b0df0 con 0x7f56f8154a90 2024-09-13T17:22:40.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56fcc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f806b900 con 0x7f56f806b360 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56f7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f8154a90 0x7f56f815b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56f7400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f56f8154a90 0x7f56f815b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52918/0 (socket says 172.21.15.28:52918) 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.102+0000 7f56f7400640 1 -- 172.21.15.28:0/838539638 learned_addr learned my addr 172.21.15.28:0/838539638 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.103+0000 7f56f7400640 1 -- 172.21.15.28:0/838539638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f806b360 msgr2=0x7f56f80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.103+0000 7f56f7400640 1 --2- 172.21.15.28:0/838539638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f806b360 0x7f56f80b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.103+0000 7f56f7400640 1 -- 172.21.15.28:0/838539638 >> v1:172.21.15.189:6789/0 conn(0x7f56f80b0980 legacy=0x7f56f80af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:40.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.103+0000 7f56f7400640 1 -- 172.21.15.28:0/838539638 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f806bbe0 con 0x7f56f8154a90 2024-09-13T17:22:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.104+0000 7f56f7400640 1 --2- 172.21.15.28:0/838539638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f815b5d0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009e50 tx=0x7f56e802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68c47202cf2379e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.104+0000 7f56f6a00640 1 -- 172.21.15.28:0/838539638 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e802aea0 con 0x7f56f8154a90 2024-09-13T17:22:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.104+0000 7f56f6a00640 1 -- 172.21.15.28:0/838539638 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e802d050 con 0x7f56f8154a90 2024-09-13T17:22:40.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.104+0000 7f56f6a00640 1 -- 172.21.15.28:0/838539638 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8031820 con 0x7f56f8154a90 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 -- 172.21.15.28:0/838539638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 msgr2=0x7f56f815b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 --2- 172.21.15.28:0/838539638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f815b5d0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009e50 tx=0x7f56e802a6c0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 -- 172.21.15.28:0/838539638 shutdown_connections 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 --2- 172.21.15.28:0/838539638 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f806b360 0x7f56f80b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 --2- 172.21.15.28:0/838539638 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f815b5d0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 -- 172.21.15.28:0/838539638 >> 172.21.15.28:0/838539638 conn(0x7f56f8072e40 msgr2=0x7f56f8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 -- 172.21.15.28:0/838539638 shutdown_connections 2024-09-13T17:22:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.105+0000 7f56fcc00640 1 -- 172.21.15.28:0/838539638 wait complete. 2024-09-13T17:22:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.106+0000 7f56fcc00640 1 Processor -- start 2024-09-13T17:22:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 -- start start 2024-09-13T17:22:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f806b360 0x7f56f81f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f80b0980 0x7f56f8138fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8156610 con 0x7f56f8154a90 2024-09-13T17:22:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f8139a40 con 0x7f56f806b360 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.107+0000 7f56fcc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f81f2ab0 con 0x7f56f80b0980 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52934/0 (socket says 172.21.15.28:52934) 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 -- 172.21.15.28:0/3792073908 learned_addr learned my addr 172.21.15.28:0/3792073908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7400640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f806b360 0x7f56f81f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56eea00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f80b0980 0x7f56f8138fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 -- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f80b0980 msgr2=0x7f56f8138fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f80b0980 0x7f56f8138fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 -- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f806b360 msgr2=0x7f56f81f2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f806b360 0x7f56f81f2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.108+0000 7f56f7e00640 1 -- 172.21.15.28:0/3792073908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e8009b30 con 0x7f56f8154a90 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56f7e00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c410 tx=0x7f56f000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f00035f0 con 0x7f56f8154a90 2024-09-13T17:22:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f0003770 con 0x7f56f8154a90 2024-09-13T17:22:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f81f2d30 con 0x7f56f8154a90 2024-09-13T17:22:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f0003b60 con 0x7f56f8154a90 2024-09-13T17:22:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.109+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f81f32f0 con 0x7f56f8154a90 2024-09-13T17:22:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.110+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f80b0440 con 0x7f56f8154a90 2024-09-13T17:22:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.111+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f56f0003cc0 con 0x7f56f8154a90 2024-09-13T17:22:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.112+0000 7f56efe00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 0x7f56d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.112+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f56f0088730 con 0x7f56f8154a90 2024-09-13T17:22:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.112+0000 7f56f7400640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 0x7f56d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.113+0000 7f56f7400640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 0x7f56d406d0c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f56e8033000 tx=0x7f56e8009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.116+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f56f00539b0 con 0x7f56f8154a90 2024-09-13T17:22:40.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.292+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56f806c1a0 con 0x7f56d406ac00 2024-09-13T17:22:40.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.300+0000 7f56efe00640 1 -- 172.21.15.28:0/3792073908 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289538 (secure 0 0 0) 0x7f56e8040070 con 0x7f56d406ac00 2024-09-13T17:22:40.305 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:40.306 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.304+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 msgr2=0x7f56d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 0x7f56d406d0c0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f56e8033000 tx=0x7f56e8009000 comp rx=0 tx=0).stop 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 msgr2=0x7f56f8139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 secure :-1 s=READY pgs=2463 cs=0 l=1 rev1=1 crypto rx=0x7f56f000c410 tx=0x7f56f000c810 comp rx=0 tx=0).stop 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 shutdown_connections 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f56d406ac00 0x7f56d406d0c0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56f8154a90 0x7f56f8139500 unknown :-1 s=CLOSED pgs=2463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f56f80b0980 0x7f56f8138fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 --2- 172.21.15.28:0/3792073908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f806b360 0x7f56f81f2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 >> 172.21.15.28:0/3792073908 conn(0x7f56f8072e40 msgr2=0x7f56f80b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 shutdown_connections 2024-09-13T17:22:40.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.305+0000 7f56fcc00640 1 -- 172.21.15.28:0/3792073908 wait complete. 2024-09-13T17:22:40.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-13T17:22:39.104553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":156,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":552956,"kb_used_data":21348,"kb_used_omap":12,"kb_used_meta":531443,"kb_avail":754421764,"statfs":{"total":773094113280,"available":772527886336,"internally_reserved":0,"allocated":21860352,"data_stored":10287282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":544198201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.821232"},"pg_stats":[{"pgid":"29.3f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899536+0000","last_change":"2024-09-13T17:22:35.747008+0000","last_active":"2024-09-13T17:22:37.899536+0000","last_peered":"2024-09-13T17:22:37.899536+0000","last_clean":"2024-09-13T17:22:37.899536+0000","last_became_active":"2024-09-13T17:22:35.746755+0000","last_became_peered":"2024-09-13T17:22:35.746755+0000","last_unstale":"2024-09-13T17:22:37.899536+0000","last_undegraded":"2024-09-13T17:22:37.899536+0000","last_fullsized":"2024-09-13T17:22:37.899536+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:01.527960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725581+0000","last_change":"2024-09-13T17:22:35.736781+0000","last_active":"2024-09-13T17:22:36.725581+0000","last_peered":"2024-09-13T17:22:36.725581+0000","last_clean":"2024-09-13T17:22:36.725581+0000","last_became_active":"2024-09-13T17:22:35.736519+0000","last_became_peered":"2024-09-13T17:22:35.736519+0000","last_unstale":"2024-09-13T17:22:36.725581+0000","last_undegraded":"2024-09-13T17:22:36.725581+0000","last_fullsized":"2024-09-13T17:22:36.725581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:52:21.616718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898657+0000","last_change":"2024-09-13T17:22:35.744547+0000","last_active":"2024-09-13T17:22:37.898657+0000","last_peered":"2024-09-13T17:22:37.898657+0000","last_clean":"2024-09-13T17:22:37.898657+0000","last_became_active":"2024-09-13T17:22:35.744287+0000","last_became_peered":"2024-09-13T17:22:35.744287+0000","last_unstale":"2024-09-13T17:22:37.898657+0000","last_undegraded":"2024-09-13T17:22:37.898657+0000","last_fullsized":"2024-09-13T17:22:37.898657+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723575+0000","last_change":"2024-09-13T17:22:35.745384+0000","last_active":"2024-09-13T17:22:36.723575+0000","last_peered":"2024-09-13T17:22:36.723575+0000","last_clean":"2024-09-13T17:22:36.723575+0000","last_became_active":"2024-09-13T17:22:35.745141+0000","last_became_peered":"2024-09-13T17:22:35.745141+0000","last_unstale":"2024-09-13T17:22:36.723575+0000","last_undegraded":"2024-09-13T17:22:36.723575+0000","last_fullsized":"2024-09-13T17:22:36.723575+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723647+0000","last_change":"2024-09-13T17:22:35.747831+0000","last_active":"2024-09-13T17:22:36.723647+0000","last_peered":"2024-09-13T17:22:36.723647+0000","last_clean":"2024-09-13T17:22:36.723647+0000","last_became_active":"2024-09-13T17:22:35.747566+0000","last_became_peered":"2024-09-13T17:22:35.747566+0000","last_unstale":"2024-09-13T17:22:36.723647+0000","last_undegraded":"2024-09-13T17:22:36.723647+0000","last_fullsized":"2024-09-13T17:22:36.723647+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:15.361193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898125+0000","last_change":"2024-09-13T17:22:35.727313+0000","last_active":"2024-09-13T17:22:37.898125+0000","last_peered":"2024-09-13T17:22:37.898125+0000","last_clean":"2024-09-13T17:22:37.898125+0000","last_became_active":"2024-09-13T17:22:35.727050+0000","last_became_peered":"2024-09-13T17:22:35.727050+0000","last_unstale":"2024-09-13T17:22:37.898125+0000","last_undegraded":"2024-09-13T17:22:37.898125+0000","last_fullsized":"2024-09-13T17:22:37.898125+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:03.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898335+0000","last_change":"2024-09-13T17:22:35.744870+0000","last_active":"2024-09-13T17:22:37.898335+0000","last_peered":"2024-09-13T17:22:37.898335+0000","last_clean":"2024-09-13T17:22:37.898335+0000","last_became_active":"2024-09-13T17:22:35.744453+0000","last_became_peered":"2024-09-13T17:22:35.744453+0000","last_unstale":"2024-09-13T17:22:37.898335+0000","last_undegraded":"2024-09-13T17:22:37.898335+0000","last_fullsized":"2024-09-13T17:22:37.898335+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:47.004276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898031+0000","last_change":"2024-09-13T17:22:35.750935+0000","last_active":"2024-09-13T17:22:37.898031+0000","last_peered":"2024-09-13T17:22:37.898031+0000","last_clean":"2024-09-13T17:22:37.898031+0000","last_became_active":"2024-09-13T17:22:35.750066+0000","last_became_peered":"2024-09-13T17:22:35.750066+0000","last_unstale":"2024-09-13T17:22:37.898031+0000","last_undegraded":"2024-09-13T17:22:37.898031+0000","last_fullsized":"2024-09-13T17:22:37.898031+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:59:22.082049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898832+0000","last_change":"2024-09-13T17:22:35.746371+0000","last_active":"2024-09-13T17:22:37.898832+0000","last_peered":"2024-09-13T17:22:37.898832+0000","last_clean":"2024-09-13T17:22:37.898832+0000","last_became_active":"2024-09-13T17:22:35.746088+0000","last_became_peered":"2024-09-13T17:22:35.746088+0000","last_unstale":"2024-09-13T17:22:37.898832+0000","last_undegraded":"2024-09-13T17:22:37.898832+0000","last_fullsized":"2024-09-13T17:22:37.898832+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:27.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898526+0000","last_change":"2024-09-13T17:22:35.740455+0000","last_active":"2024-09-13T17:22:37.898526+0000","last_peered":"2024-09-13T17:22:37.898526+0000","last_clean":"2024-09-13T17:22:37.898526+0000","last_became_active":"2024-09-13T17:22:35.740209+0000","last_became_peered":"2024-09-13T17:22:35.740209+0000","last_unstale":"2024-09-13T17:22:37.898526+0000","last_undegraded":"2024-09-13T17:22:37.898526+0000","last_fullsized":"2024-09-13T17:22:37.898526+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.727927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898958+0000","last_change":"2024-09-13T17:22:35.749707+0000","last_active":"2024-09-13T17:22:37.898958+0000","last_peered":"2024-09-13T17:22:37.898958+0000","last_clean":"2024-09-13T17:22:37.898958+0000","last_became_active":"2024-09-13T17:22:35.749441+0000","last_became_peered":"2024-09-13T17:22:35.749441+0000","last_unstale":"2024-09-13T17:22:37.898958+0000","last_undegraded":"2024-09-13T17:22:37.898958+0000","last_fullsized":"2024-09-13T17:22:37.898958+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.032697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725459+0000","last_change":"2024-09-13T17:22:35.735954+0000","last_active":"2024-09-13T17:22:36.725459+0000","last_peered":"2024-09-13T17:22:36.725459+0000","last_clean":"2024-09-13T17:22:36.725459+0000","last_became_active":"2024-09-13T17:22:35.734897+0000","last_became_peered":"2024-09-13T17:22:35.734897+0000","last_unstale":"2024-09-13T17:22:36.725459+0000","last_undegraded":"2024-09-13T17:22:36.725459+0000","last_fullsized":"2024-09-13T17:22:36.725459+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:31:01.527960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725051+0000","last_change":"2024-09-13T17:22:35.734515+0000","last_active":"2024-09-13T17:22:36.725051+0000","last_peered":"2024-09-13T17:22:36.725051+0000","last_clean":"2024-09-13T17:22:36.725051+0000","last_became_active":"2024-09-13T17:22:35.734250+0000","last_became_peered":"2024-09-13T17:22:35.734250+0000","last_unstale":"2024-09-13T17:22:36.725051+0000","last_undegraded":"2024-09-13T17:22:36.725051+0000","last_fullsized":"2024-09-13T17:22:36.725051+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:40.351477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725560+0000","last_change":"2024-09-13T17:22:35.738211+0000","last_active":"2024-09-13T17:22:36.725560+0000","last_peered":"2024-09-13T17:22:36.725560+0000","last_clean":"2024-09-13T17:22:36.725560+0000","last_became_active":"2024-09-13T17:22:35.737899+0000","last_became_peered":"2024-09-13T17:22:35.737899+0000","last_unstale":"2024-09-13T17:22:36.725560+0000","last_undegraded":"2024-09-13T17:22:36.725560+0000","last_fullsized":"2024-09-13T17:22:36.725560+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897310+0000","last_change":"2024-09-13T17:22:35.742600+0000","last_active":"2024-09-13T17:22:37.897310+0000","last_peered":"2024-09-13T17:22:37.897310+0000","last_clean":"2024-09-13T17:22:37.897310+0000","last_became_active":"2024-09-13T17:22:35.742313+0000","last_became_peered":"2024-09-13T17:22:35.742313+0000","last_unstale":"2024-09-13T17:22:37.897310+0000","last_undegraded":"2024-09-13T17:22:37.897310+0000","last_fullsized":"2024-09-13T17:22:37.897310+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:29:04.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724725+0000","last_change":"2024-09-13T17:22:35.733414+0000","last_active":"2024-09-13T17:22:36.724725+0000","last_peered":"2024-09-13T17:22:36.724725+0000","last_clean":"2024-09-13T17:22:36.724725+0000","last_became_active":"2024-09-13T17:22:35.733127+0000","last_became_peered":"2024-09-13T17:22:35.733127+0000","last_unstale":"2024-09-13T17:22:36.724725+0000","last_undegraded":"2024-09-13T17:22:36.724725+0000","last_fullsized":"2024-09-13T17:22:36.724725+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:00.326830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725824+0000","last_change":"2024-09-13T17:22:35.741391+0000","last_active":"2024-09-13T17:22:36.725824+0000","last_peered":"2024-09-13T17:22:36.725824+0000","last_clean":"2024-09-13T17:22:36.725824+0000","last_became_active":"2024-09-13T17:22:35.736783+0000","last_became_peered":"2024-09-13T17:22:35.736783+0000","last_unstale":"2024-09-13T17:22:36.725824+0000","last_undegraded":"2024-09-13T17:22:36.725824+0000","last_fullsized":"2024-09-13T17:22:36.725824+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898629+0000","last_change":"2024-09-13T17:22:35.738314+0000","last_active":"2024-09-13T17:22:37.898629+0000","last_peered":"2024-09-13T17:22:37.898629+0000","last_clean":"2024-09-13T17:22:37.898629+0000","last_became_active":"2024-09-13T17:22:35.738035+0000","last_became_peered":"2024-09-13T17:22:35.738035+0000","last_unstale":"2024-09-13T17:22:37.898629+0000","last_undegraded":"2024-09-13T17:22:37.898629+0000","last_fullsized":"2024-09-13T17:22:37.898629+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:47.789552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723854+0000","last_change":"2024-09-13T17:22:35.741091+0000","last_active":"2024-09-13T17:22:36.723854+0000","last_peered":"2024-09-13T17:22:36.723854+0000","last_clean":"2024-09-13T17:22:36.723854+0000","last_became_active":"2024-09-13T17:22:35.740832+0000","last_became_peered":"2024-09-13T17:22:35.740832+0000","last_unstale":"2024-09-13T17:22:36.723854+0000","last_undegraded":"2024-09-13T17:22:36.723854+0000","last_fullsized":"2024-09-13T17:22:36.723854+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898416+0000","last_change":"2024-09-13T17:22:35.730293+0000","last_active":"2024-09-13T17:22:37.898416+0000","last_peered":"2024-09-13T17:22:37.898416+0000","last_clean":"2024-09-13T17:22:37.898416+0000","last_became_active":"2024-09-13T17:22:35.730049+0000","last_became_peered":"2024-09-13T17:22:35.730049+0000","last_unstale":"2024-09-13T17:22:37.898416+0000","last_undegraded":"2024-09-13T17:22:37.898416+0000","last_fullsized":"2024-09-13T17:22:37.898416+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:19.815948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723452+0000","last_change":"2024-09-13T17:22:35.740999+0000","last_active":"2024-09-13T17:22:36.723452+0000","last_peered":"2024-09-13T17:22:36.723452+0000","last_clean":"2024-09-13T17:22:36.723452+0000","last_became_active":"2024-09-13T17:22:35.740656+0000","last_became_peered":"2024-09-13T17:22:35.740656+0000","last_unstale":"2024-09-13T17:22:36.723452+0000","last_undegraded":"2024-09-13T17:22:36.723452+0000","last_fullsized":"2024-09-13T17:22:36.723452+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897530+0000","last_change":"2024-09-13T17:22:35.734233+0000","last_active":"2024-09-13T17:22:37.897530+0000","last_peered":"2024-09-13T17:22:37.897530+0000","last_clean":"2024-09-13T17:22:37.897530+0000","last_became_active":"2024-09-13T17:22:35.733600+0000","last_became_peered":"2024-09-13T17:22:35.733600+0000","last_unstale":"2024-09-13T17:22:37.897530+0000","last_undegraded":"2024-09-13T17:22:37.897530+0000","last_fullsized":"2024-09-13T17:22:37.897530+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:26:25.672327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898004+0000","last_change":"2024-09-13T17:22:35.738172+0000","last_active":"2024-09-13T17:22:37.898004+0000","last_peered":"2024-09-13T17:22:37.898004+0000","last_clean":"2024-09-13T17:22:37.898004+0000","last_became_active":"2024-09-13T17:22:35.737818+0000","last_became_peered":"2024-09-13T17:22:35.737818+0000","last_unstale":"2024-09-13T17:22:37.898004+0000","last_undegraded":"2024-09-13T17:22:37.898004+0000","last_fullsized":"2024-09-13T17:22:37.898004+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:50.208014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723358+0000","last_change":"2024-09-13T17:22:35.747843+0000","last_active":"2024-09-13T17:22:36.723358+0000","last_peered":"2024-09-13T17:22:36.723358+0000","last_clean":"2024-09-13T17:22:36.723358+0000","last_became_active":"2024-09-13T17:22:35.747555+0000","last_became_peered":"2024-09-13T17:22:35.747555+0000","last_unstale":"2024-09-13T17:22:36.723358+0000","last_undegraded":"2024-09-13T17:22:36.723358+0000","last_fullsized":"2024-09-13T17:22:36.723358+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899536+0000","last_change":"2024-09-13T17:22:35.745430+0000","last_active":"2024-09-13T17:22:37.899536+0000","last_peered":"2024-09-13T17:22:37.899536+0000","last_clean":"2024-09-13T17:22:37.899536+0000","last_became_active":"2024-09-13T17:22:35.744580+0000","last_became_peered":"2024-09-13T17:22:35.744580+0000","last_unstale":"2024-09-13T17:22:37.899536+0000","last_undegraded":"2024-09-13T17:22:37.899536+0000","last_fullsized":"2024-09-13T17:22:37.899536+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:02:35.139446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898543+0000","last_change":"2024-09-13T17:22:35.747792+0000","last_active":"2024-09-13T17:22:37.898543+0000","last_peered":"2024-09-13T17:22:37.898543+0000","last_clean":"2024-09-13T17:22:37.898543+0000","last_became_active":"2024-09-13T17:22:35.747544+0000","last_became_peered":"2024-09-13T17:22:35.747544+0000","last_unstale":"2024-09-13T17:22:37.898543+0000","last_undegraded":"2024-09-13T17:22:37.898543+0000","last_fullsized":"2024-09-13T17:22:37.898543+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:27:57.647264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899163+0000","last_change":"2024-09-13T17:22:35.749403+0000","last_active":"2024-09-13T17:22:37.899163+0000","last_peered":"2024-09-13T17:22:37.899163+0000","last_clean":"2024-09-13T17:22:37.899163+0000","last_became_active":"2024-09-13T17:22:35.749132+0000","last_became_peered":"2024-09-13T17:22:35.749132+0000","last_unstale":"2024-09-13T17:22:37.899163+0000","last_undegraded":"2024-09-13T17:22:37.899163+0000","last_fullsized":"2024-09-13T17:22:37.899163+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:54.045591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898454+0000","last_change":"2024-09-13T17:22:35.738905+0000","last_active":"2024-09-13T17:22:37.898454+0000","last_peered":"2024-09-13T17:22:37.898454+0000","last_clean":"2024-09-13T17:22:37.898454+0000","last_became_active":"2024-09-13T17:22:35.738655+0000","last_became_peered":"2024-09-13T17:22:35.738655+0000","last_unstale":"2024-09-13T17:22:37.898454+0000","last_undegraded":"2024-09-13T17:22:37.898454+0000","last_fullsized":"2024-09-13T17:22:37.898454+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:20.592424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725561+0000","last_change":"2024-09-13T17:22:35.748735+0000","last_active":"2024-09-13T17:22:36.725561+0000","last_peered":"2024-09-13T17:22:36.725561+0000","last_clean":"2024-09-13T17:22:36.725561+0000","last_became_active":"2024-09-13T17:22:35.748483+0000","last_became_peered":"2024-09-13T17:22:35.748483+0000","last_unstale":"2024-09-13T17:22:36.725561+0000","last_undegraded":"2024-09-13T17:22:36.725561+0000","last_fullsized":"2024-09-13T17:22:36.725561+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:53.627485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898006+0000","last_change":"2024-09-13T17:22:35.745720+0000","last_active":"2024-09-13T17:22:37.898006+0000","last_peered":"2024-09-13T17:22:37.898006+0000","last_clean":"2024-09-13T17:22:37.898006+0000","last_became_active":"2024-09-13T17:22:35.745457+0000","last_became_peered":"2024-09-13T17:22:35.745457+0000","last_unstale":"2024-09-13T17:22:37.898006+0000","last_undegraded":"2024-09-13T17:22:37.898006+0000","last_fullsized":"2024-09-13T17:22:37.898006+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:19.936403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723299+0000","last_change":"2024-09-13T17:22:35.742999+0000","last_active":"2024-09-13T17:22:36.723299+0000","last_peered":"2024-09-13T17:22:36.723299+0000","last_clean":"2024-09-13T17:22:36.723299+0000","last_became_active":"2024-09-13T17:22:35.739528+0000","last_became_peered":"2024-09-13T17:22:35.739528+0000","last_unstale":"2024-09-13T17:22:36.723299+0000","last_undegraded":"2024-09-13T17:22:36.723299+0000","last_fullsized":"2024-09-13T17:22:36.723299+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:45.650239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897377+0000","last_change":"2024-09-13T17:22:35.743379+0000","last_active":"2024-09-13T17:22:37.897377+0000","last_peered":"2024-09-13T17:22:37.897377+0000","last_clean":"2024-09-13T17:22:37.897377+0000","last_became_active":"2024-09-13T17:22:35.743148+0000","last_became_peered":"2024-09-13T17:22:35.743148+0000","last_unstale":"2024-09-13T17:22:37.897377+0000","last_undegraded":"2024-09-13T17:22:37.897377+0000","last_fullsized":"2024-09-13T17:22:37.897377+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:47.789552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898938+0000","last_change":"2024-09-13T17:22:35.747097+0000","last_active":"2024-09-13T17:22:37.898938+0000","last_peered":"2024-09-13T17:22:37.898938+0000","last_clean":"2024-09-13T17:22:37.898938+0000","last_became_active":"2024-09-13T17:22:35.746343+0000","last_became_peered":"2024-09-13T17:22:35.746343+0000","last_unstale":"2024-09-13T17:22:37.898938+0000","last_undegraded":"2024-09-13T17:22:37.898938+0000","last_fullsized":"2024-09-13T17:22:37.898938+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:22.233339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"204'4","reported_seq":15,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897746+0000","last_change":"2024-09-13T17:22:35.307764+0000","last_active":"2024-09-13T17:22:37.897746+0000","last_peered":"2024-09-13T17:22:37.897746+0000","last_clean":"2024-09-13T17:22:37.897746+0000","last_became_active":"2024-09-13T17:22:35.307441+0000","last_became_peered":"2024-09-13T17:22:35.307441+0000","last_unstale":"2024-09-13T17:22:37.897746+0000","last_undegraded":"2024-09-13T17:22:37.897746+0000","last_fullsized":"2024-09-13T17:22:37.897746+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:54.686512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898831+0000","last_change":"2024-09-13T17:22:35.741073+0000","last_active":"2024-09-13T17:22:37.898831+0000","last_peered":"2024-09-13T17:22:37.898831+0000","last_clean":"2024-09-13T17:22:37.898831+0000","last_became_active":"2024-09-13T17:22:35.740766+0000","last_became_peered":"2024-09-13T17:22:35.740766+0000","last_unstale":"2024-09-13T17:22:37.898831+0000","last_undegraded":"2024-09-13T17:22:37.898831+0000","last_fullsized":"2024-09-13T17:22:37.898831+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:52.634635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725766+0000","last_change":"2024-09-13T17:22:35.309913+0000","last_active":"2024-09-13T17:22:36.725766+0000","last_peered":"2024-09-13T17:22:36.725766+0000","last_clean":"2024-09-13T17:22:36.725766+0000","last_became_active":"2024-09-13T17:22:35.309599+0000","last_became_peered":"2024-09-13T17:22:35.309599+0000","last_unstale":"2024-09-13T17:22:36.725766+0000","last_undegraded":"2024-09-13T17:22:36.725766+0000","last_fullsized":"2024-09-13T17:22:36.725766+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:23.635744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.726012+0000","last_change":"2024-09-13T17:22:35.733816+0000","last_active":"2024-09-13T17:22:36.726012+0000","last_peered":"2024-09-13T17:22:36.726012+0000","last_clean":"2024-09-13T17:22:36.726012+0000","last_became_active":"2024-09-13T17:22:35.733556+0000","last_became_peered":"2024-09-13T17:22:35.733556+0000","last_unstale":"2024-09-13T17:22:36.726012+0000","last_undegraded":"2024-09-13T17:22:36.726012+0000","last_fullsized":"2024-09-13T17:22:36.726012+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.1c","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725551+0000","last_change":"2024-09-13T17:22:35.308418+0000","last_active":"2024-09-13T17:22:36.725551+0000","last_peered":"2024-09-13T17:22:36.725551+0000","last_clean":"2024-09-13T17:22:36.725551+0000","last_became_active":"2024-09-13T17:22:35.308136+0000","last_became_peered":"2024-09-13T17:22:35.308136+0000","last_unstale":"2024-09-13T17:22:36.725551+0000","last_undegraded":"2024-09-13T17:22:36.725551+0000","last_fullsized":"2024-09-13T17:22:36.725551+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:14.358311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898044+0000","last_change":"2024-09-13T17:22:35.752345+0000","last_active":"2024-09-13T17:22:37.898044+0000","last_peered":"2024-09-13T17:22:37.898044+0000","last_clean":"2024-09-13T17:22:37.898044+0000","last_became_active":"2024-09-13T17:22:35.752092+0000","last_became_peered":"2024-09-13T17:22:35.752092+0000","last_unstale":"2024-09-13T17:22:37.898044+0000","last_undegraded":"2024-09-13T17:22:37.898044+0000","last_fullsized":"2024-09-13T17:22:37.898044+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:40.351477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1b","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898240+0000","last_change":"2024-09-13T17:22:35.301231+0000","last_active":"2024-09-13T17:22:37.898240+0000","last_peered":"2024-09-13T17:22:37.898240+0000","last_clean":"2024-09-13T17:22:37.898240+0000","last_became_active":"2024-09-13T17:22:35.300369+0000","last_became_peered":"2024-09-13T17:22:35.300369+0000","last_unstale":"2024-09-13T17:22:37.898240+0000","last_undegraded":"2024-09-13T17:22:37.898240+0000","last_fullsized":"2024-09-13T17:22:37.898240+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:32.155000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723574+0000","last_change":"2024-09-13T17:22:35.745074+0000","last_active":"2024-09-13T17:22:36.723574+0000","last_peered":"2024-09-13T17:22:36.723574+0000","last_clean":"2024-09-13T17:22:36.723574+0000","last_became_active":"2024-09-13T17:22:35.744800+0000","last_became_peered":"2024-09-13T17:22:35.744800+0000","last_unstale":"2024-09-13T17:22:36.723574+0000","last_undegraded":"2024-09-13T17:22:36.723574+0000","last_fullsized":"2024-09-13T17:22:36.723574+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:54.045591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.1a","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898426+0000","last_change":"2024-09-13T17:22:35.310902+0000","last_active":"2024-09-13T17:22:37.898426+0000","last_peered":"2024-09-13T17:22:37.898426+0000","last_clean":"2024-09-13T17:22:37.898426+0000","last_became_active":"2024-09-13T17:22:35.310593+0000","last_became_peered":"2024-09-13T17:22:35.310593+0000","last_unstale":"2024-09-13T17:22:37.898426+0000","last_undegraded":"2024-09-13T17:22:37.898426+0000","last_fullsized":"2024-09-13T17:22:37.898426+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:37.482489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897749+0000","last_change":"2024-09-13T17:22:35.736675+0000","last_active":"2024-09-13T17:22:37.897749+0000","last_peered":"2024-09-13T17:22:37.897749+0000","last_clean":"2024-09-13T17:22:37.897749+0000","last_became_active":"2024-09-13T17:22:35.735106+0000","last_became_peered":"2024-09-13T17:22:35.735106+0000","last_unstale":"2024-09-13T17:22:37.897749+0000","last_undegraded":"2024-09-13T17:22:37.897749+0000","last_fullsized":"2024-09-13T17:22:37.897749+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:51.481580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725750+0000","last_change":"2024-09-13T17:22:35.307843+0000","last_active":"2024-09-13T17:22:36.725750+0000","last_peered":"2024-09-13T17:22:36.725750+0000","last_clean":"2024-09-13T17:22:36.725750+0000","last_became_active":"2024-09-13T17:22:35.307484+0000","last_became_peered":"2024-09-13T17:22:35.307484+0000","last_unstale":"2024-09-13T17:22:36.725750+0000","last_undegraded":"2024-09-13T17:22:36.725750+0000","last_fullsized":"2024-09-13T17:22:36.725750+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:51.984618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724869+0000","last_change":"2024-09-13T17:22:35.742152+0000","last_active":"2024-09-13T17:22:36.724869+0000","last_peered":"2024-09-13T17:22:36.724869+0000","last_clean":"2024-09-13T17:22:36.724869+0000","last_became_active":"2024-09-13T17:22:35.741746+0000","last_became_peered":"2024-09-13T17:22:35.741746+0000","last_unstale":"2024-09-13T17:22:36.724869+0000","last_undegraded":"2024-09-13T17:22:36.724869+0000","last_fullsized":"2024-09-13T17:22:36.724869+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:18.032697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897820+0000","last_change":"2024-09-13T17:22:35.305822+0000","last_active":"2024-09-13T17:22:37.897820+0000","last_peered":"2024-09-13T17:22:37.897820+0000","last_clean":"2024-09-13T17:22:37.897820+0000","last_became_active":"2024-09-13T17:22:35.305512+0000","last_became_peered":"2024-09-13T17:22:35.305512+0000","last_unstale":"2024-09-13T17:22:37.897820+0000","last_undegraded":"2024-09-13T17:22:37.897820+0000","last_fullsized":"2024-09-13T17:22:37.897820+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:46:18.523079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725734+0000","last_change":"2024-09-13T17:22:35.739477+0000","last_active":"2024-09-13T17:22:36.725734+0000","last_peered":"2024-09-13T17:22:36.725734+0000","last_clean":"2024-09-13T17:22:36.725734+0000","last_became_active":"2024-09-13T17:22:35.737306+0000","last_became_peered":"2024-09-13T17:22:35.737306+0000","last_unstale":"2024-09-13T17:22:36.725734+0000","last_undegraded":"2024-09-13T17:22:36.725734+0000","last_fullsized":"2024-09-13T17:22:36.725734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:03.255217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899613+0000","last_change":"2024-09-13T17:22:35.301090+0000","last_active":"2024-09-13T17:22:37.899613+0000","last_peered":"2024-09-13T17:22:37.899613+0000","last_clean":"2024-09-13T17:22:37.899613+0000","last_became_active":"2024-09-13T17:22:35.300652+0000","last_became_peered":"2024-09-13T17:22:35.300652+0000","last_unstale":"2024-09-13T17:22:37.899613+0000","last_undegraded":"2024-09-13T17:22:37.899613+0000","last_fullsized":"2024-09-13T17:22:37.899613+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:01.543679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899619+0000","last_change":"2024-09-13T17:22:35.741998+0000","last_active":"2024-09-13T17:22:37.899619+0000","last_peered":"2024-09-13T17:22:37.899619+0000","last_clean":"2024-09-13T17:22:37.899619+0000","last_became_active":"2024-09-13T17:22:35.741723+0000","last_became_peered":"2024-09-13T17:22:35.741723+0000","last_unstale":"2024-09-13T17:22:37.899619+0000","last_undegraded":"2024-09-13T17:22:37.899619+0000","last_fullsized":"2024-09-13T17:22:37.899619+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:03.239874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897888+0000","last_change":"2024-09-13T17:22:35.302438+0000","last_active":"2024-09-13T17:22:37.897888+0000","last_peered":"2024-09-13T17:22:37.897888+0000","last_clean":"2024-09-13T17:22:37.897888+0000","last_became_active":"2024-09-13T17:22:35.302196+0000","last_became_peered":"2024-09-13T17:22:35.302196+0000","last_unstale":"2024-09-13T17:22:37.897888+0000","last_undegraded":"2024-09-13T17:22:37.897888+0000","last_fullsized":"2024-09-13T17:22:37.897888+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:45.784941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725919+0000","last_change":"2024-09-13T17:22:35.739448+0000","last_active":"2024-09-13T17:22:36.725919+0000","last_peered":"2024-09-13T17:22:36.725919+0000","last_clean":"2024-09-13T17:22:36.725919+0000","last_became_active":"2024-09-13T17:22:35.739167+0000","last_became_peered":"2024-09-13T17:22:35.739167+0000","last_unstale":"2024-09-13T17:22:36.725919+0000","last_undegraded":"2024-09-13T17:22:36.725919+0000","last_fullsized":"2024-09-13T17:22:36.725919+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:27.052948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.15","version":"204'3","reported_seq":14,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723873+0000","last_change":"2024-09-13T17:22:35.298661+0000","last_active":"2024-09-13T17:22:36.723873+0000","last_peered":"2024-09-13T17:22:36.723873+0000","last_clean":"2024-09-13T17:22:36.723873+0000","last_became_active":"2024-09-13T17:22:35.298396+0000","last_became_peered":"2024-09-13T17:22:35.298396+0000","last_unstale":"2024-09-13T17:22:36.723873+0000","last_undegraded":"2024-09-13T17:22:36.723873+0000","last_fullsized":"2024-09-13T17:22:36.723873+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:51.984618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899065+0000","last_change":"2024-09-13T17:22:35.741773+0000","last_active":"2024-09-13T17:22:37.899065+0000","last_peered":"2024-09-13T17:22:37.899065+0000","last_clean":"2024-09-13T17:22:37.899065+0000","last_became_active":"2024-09-13T17:22:35.741532+0000","last_became_peered":"2024-09-13T17:22:35.741532+0000","last_unstale":"2024-09-13T17:22:37.899065+0000","last_undegraded":"2024-09-13T17:22:37.899065+0000","last_fullsized":"2024-09-13T17:22:37.899065+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:41:03.255217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.14","version":"204'6","reported_seq":17,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898519+0000","last_change":"2024-09-13T17:22:35.305124+0000","last_active":"2024-09-13T17:22:37.898519+0000","last_peered":"2024-09-13T17:22:37.898519+0000","last_clean":"2024-09-13T17:22:37.898519+0000","last_became_active":"2024-09-13T17:22:35.304822+0000","last_became_peered":"2024-09-13T17:22:35.304822+0000","last_unstale":"2024-09-13T17:22:37.898519+0000","last_undegraded":"2024-09-13T17:22:37.898519+0000","last_fullsized":"2024-09-13T17:22:37.898519+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:50.621747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897843+0000","last_change":"2024-09-13T17:22:35.735874+0000","last_active":"2024-09-13T17:22:37.897843+0000","last_peered":"2024-09-13T17:22:37.897843+0000","last_clean":"2024-09-13T17:22:37.897843+0000","last_became_active":"2024-09-13T17:22:35.735418+0000","last_became_peered":"2024-09-13T17:22:35.735418+0000","last_unstale":"2024-09-13T17:22:37.897843+0000","last_undegraded":"2024-09-13T17:22:37.897843+0000","last_fullsized":"2024-09-13T17:22:37.897843+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:03.343508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.13","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898535+0000","last_change":"2024-09-13T17:22:35.310730+0000","last_active":"2024-09-13T17:22:37.898535+0000","last_peered":"2024-09-13T17:22:37.898535+0000","last_clean":"2024-09-13T17:22:37.898535+0000","last_became_active":"2024-09-13T17:22:35.310422+0000","last_became_peered":"2024-09-13T17:22:35.310422+0000","last_unstale":"2024-09-13T17:22:37.898535+0000","last_undegraded":"2024-09-13T17:22:37.898535+0000","last_fullsized":"2024-09-13T17:22:37.898535+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:25.376497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898581+0000","last_change":"2024-09-13T17:22:35.751448+0000","last_active":"2024-09-13T17:22:37.898581+0000","last_peered":"2024-09-13T17:22:37.898581+0000","last_clean":"2024-09-13T17:22:37.898581+0000","last_became_active":"2024-09-13T17:22:35.747909+0000","last_became_peered":"2024-09-13T17:22:35.747909+0000","last_unstale":"2024-09-13T17:22:37.898581+0000","last_undegraded":"2024-09-13T17:22:37.898581+0000","last_fullsized":"2024-09-13T17:22:37.898581+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.818485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897726+0000","last_change":"2024-09-13T17:22:35.300098+0000","last_active":"2024-09-13T17:22:37.897726+0000","last_peered":"2024-09-13T17:22:37.897726+0000","last_clean":"2024-09-13T17:22:37.897726+0000","last_became_active":"2024-09-13T17:22:35.299874+0000","last_became_peered":"2024-09-13T17:22:35.299874+0000","last_unstale":"2024-09-13T17:22:37.897726+0000","last_undegraded":"2024-09-13T17:22:37.897726+0000","last_fullsized":"2024-09-13T17:22:37.897726+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:12:47.059931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898061+0000","last_change":"2024-09-13T17:22:35.736993+0000","last_active":"2024-09-13T17:22:37.898061+0000","last_peered":"2024-09-13T17:22:37.898061+0000","last_clean":"2024-09-13T17:22:37.898061+0000","last_became_active":"2024-09-13T17:22:35.736667+0000","last_became_peered":"2024-09-13T17:22:35.736667+0000","last_unstale":"2024-09-13T17:22:37.898061+0000","last_undegraded":"2024-09-13T17:22:37.898061+0000","last_fullsized":"2024-09-13T17:22:37.898061+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:16:24.638626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723516+0000","last_change":"2024-09-13T17:22:35.300153+0000","last_active":"2024-09-13T17:22:36.723516+0000","last_peered":"2024-09-13T17:22:36.723516+0000","last_clean":"2024-09-13T17:22:36.723516+0000","last_became_active":"2024-09-13T17:22:35.299865+0000","last_became_peered":"2024-09-13T17:22:35.299865+0000","last_unstale":"2024-09-13T17:22:36.723516+0000","last_undegraded":"2024-09-13T17:22:36.723516+0000","last_fullsized":"2024-09-13T17:22:36.723516+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:49:59.466116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725159+0000","last_change":"2024-09-13T17:22:35.731600+0000","last_active":"2024-09-13T17:22:36.725159+0000","last_peered":"2024-09-13T17:22:36.725159+0000","last_clean":"2024-09-13T17:22:36.725159+0000","last_became_active":"2024-09-13T17:22:35.731316+0000","last_became_peered":"2024-09-13T17:22:35.731316+0000","last_unstale":"2024-09-13T17:22:36.725159+0000","last_undegraded":"2024-09-13T17:22:36.725159+0000","last_fullsized":"2024-09-13T17:22:36.725159+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.10","version":"204'8","reported_seq":19,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.724771+0000","last_change":"2024-09-13T17:22:35.308164+0000","last_active":"2024-09-13T17:22:36.724771+0000","last_peered":"2024-09-13T17:22:36.724771+0000","last_clean":"2024-09-13T17:22:36.724771+0000","last_became_active":"2024-09-13T17:22:35.307885+0000","last_became_peered":"2024-09-13T17:22:35.307885+0000","last_unstale":"2024-09-13T17:22:36.724771+0000","last_undegraded":"2024-09-13T17:22:36.724771+0000","last_fullsized":"2024-09-13T17:22:36.724771+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:57.846901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898635+0000","last_change":"2024-09-13T17:22:35.748387+0000","last_active":"2024-09-13T17:22:37.898635+0000","last_peered":"2024-09-13T17:22:37.898635+0000","last_clean":"2024-09-13T17:22:37.898635+0000","last_became_active":"2024-09-13T17:22:35.747839+0000","last_became_peered":"2024-09-13T17:22:35.747839+0000","last_unstale":"2024-09-13T17:22:37.898635+0000","last_undegraded":"2024-09-13T17:22:37.898635+0000","last_fullsized":"2024-09-13T17:22:37.898635+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:26.415107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.f","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899015+0000","last_change":"2024-09-13T17:22:35.311457+0000","last_active":"2024-09-13T17:22:37.899015+0000","last_peered":"2024-09-13T17:22:37.899015+0000","last_clean":"2024-09-13T17:22:37.899015+0000","last_became_active":"2024-09-13T17:22:35.310899+0000","last_became_peered":"2024-09-13T17:22:35.310899+0000","last_unstale":"2024-09-13T17:22:37.899015+0000","last_undegraded":"2024-09-13T17:22:37.899015+0000","last_fullsized":"2024-09-13T17:22:37.899015+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:08.704735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898633+0000","last_change":"2024-09-13T17:22:35.742509+0000","last_active":"2024-09-13T17:22:37.898633+0000","last_peered":"2024-09-13T17:22:37.898633+0000","last_clean":"2024-09-13T17:22:37.898633+0000","last_became_active":"2024-09-13T17:22:35.742245+0000","last_became_peered":"2024-09-13T17:22:35.742245+0000","last_unstale":"2024-09-13T17:22:37.898633+0000","last_undegraded":"2024-09-13T17:22:37.898633+0000","last_fullsized":"2024-09-13T17:22:37.898633+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:29:04.063052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.e","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897951+0000","last_change":"2024-09-13T17:22:35.308074+0000","last_active":"2024-09-13T17:22:37.897951+0000","last_peered":"2024-09-13T17:22:37.897951+0000","last_clean":"2024-09-13T17:22:37.897951+0000","last_became_active":"2024-09-13T17:22:35.307425+0000","last_became_peered":"2024-09-13T17:22:35.307425+0000","last_unstale":"2024-09-13T17:22:37.897951+0000","last_undegraded":"2024-09-13T17:22:37.897951+0000","last_fullsized":"2024-09-13T17:22:37.897951+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:00.990352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898633+0000","last_change":"2024-09-13T17:22:35.740876+0000","last_active":"2024-09-13T17:22:37.898633+0000","last_peered":"2024-09-13T17:22:37.898633+0000","last_clean":"2024-09-13T17:22:37.898633+0000","last_became_active":"2024-09-13T17:22:35.740579+0000","last_became_peered":"2024-09-13T17:22:35.740579+0000","last_unstale":"2024-09-13T17:22:37.898633+0000","last_undegraded":"2024-09-13T17:22:37.898633+0000","last_fullsized":"2024-09-13T17:22:37.898633+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:15.361193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.d","version":"204'1","reported_seq":12,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899256+0000","last_change":"2024-09-13T17:22:35.303117+0000","last_active":"2024-09-13T17:22:37.899256+0000","last_peered":"2024-09-13T17:22:37.899256+0000","last_clean":"2024-09-13T17:22:37.899256+0000","last_became_active":"2024-09-13T17:22:35.302851+0000","last_became_peered":"2024-09-13T17:22:35.302851+0000","last_unstale":"2024-09-13T17:22:37.899256+0000","last_undegraded":"2024-09-13T17:22:37.899256+0000","last_fullsized":"2024-09-13T17:22:37.899256+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:48:22.612572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899110+0000","last_change":"2024-09-13T17:22:35.740937+0000","last_active":"2024-09-13T17:22:37.899110+0000","last_peered":"2024-09-13T17:22:37.899110+0000","last_clean":"2024-09-13T17:22:37.899110+0000","last_became_active":"2024-09-13T17:22:35.740664+0000","last_became_peered":"2024-09-13T17:22:35.740664+0000","last_unstale":"2024-09-13T17:22:37.899110+0000","last_undegraded":"2024-09-13T17:22:37.899110+0000","last_fullsized":"2024-09-13T17:22:37.899110+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898566+0000","last_change":"2024-09-13T17:22:35.301632+0000","last_active":"2024-09-13T17:22:37.898566+0000","last_peered":"2024-09-13T17:22:37.898566+0000","last_clean":"2024-09-13T17:22:37.898566+0000","last_became_active":"2024-09-13T17:22:35.300947+0000","last_became_peered":"2024-09-13T17:22:35.300947+0000","last_unstale":"2024-09-13T17:22:37.898566+0000","last_undegraded":"2024-09-13T17:22:37.898566+0000","last_fullsized":"2024-09-13T17:22:37.898566+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:28.037447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723332+0000","last_change":"2024-09-13T17:22:35.732821+0000","last_active":"2024-09-13T17:22:36.723332+0000","last_peered":"2024-09-13T17:22:36.723332+0000","last_clean":"2024-09-13T17:22:36.723332+0000","last_became_active":"2024-09-13T17:22:35.732571+0000","last_became_peered":"2024-09-13T17:22:35.732571+0000","last_unstale":"2024-09-13T17:22:36.723332+0000","last_undegraded":"2024-09-13T17:22:36.723332+0000","last_fullsized":"2024-09-13T17:22:36.723332+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:47.004276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.b","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899076+0000","last_change":"2024-09-13T17:22:35.304871+0000","last_active":"2024-09-13T17:22:37.899076+0000","last_peered":"2024-09-13T17:22:37.899076+0000","last_clean":"2024-09-13T17:22:37.899076+0000","last_became_active":"2024-09-13T17:22:35.304338+0000","last_became_peered":"2024-09-13T17:22:35.304338+0000","last_unstale":"2024-09-13T17:22:37.899076+0000","last_undegraded":"2024-09-13T17:22:37.899076+0000","last_fullsized":"2024-09-13T17:22:37.899076+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:24.505838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898632+0000","last_change":"2024-09-13T17:22:35.750909+0000","last_active":"2024-09-13T17:22:37.898632+0000","last_peered":"2024-09-13T17:22:37.898632+0000","last_clean":"2024-09-13T17:22:37.898632+0000","last_became_active":"2024-09-13T17:22:35.746385+0000","last_became_peered":"2024-09-13T17:22:35.746385+0000","last_unstale":"2024-09-13T17:22:37.898632+0000","last_undegraded":"2024-09-13T17:22:37.898632+0000","last_fullsized":"2024-09-13T17:22:37.898632+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:53.627485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725647+0000","last_change":"2024-09-13T17:22:35.307125+0000","last_active":"2024-09-13T17:22:36.725647+0000","last_peered":"2024-09-13T17:22:36.725647+0000","last_clean":"2024-09-13T17:22:36.725647+0000","last_became_active":"2024-09-13T17:22:35.306846+0000","last_became_peered":"2024-09-13T17:22:35.306846+0000","last_unstale":"2024-09-13T17:22:36.725647+0000","last_undegraded":"2024-09-13T17:22:36.725647+0000","last_fullsized":"2024-09-13T17:22:36.725647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:13.856467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898133+0000","last_change":"2024-09-13T17:22:35.745947+0000","last_active":"2024-09-13T17:22:37.898133+0000","last_peered":"2024-09-13T17:22:37.898133+0000","last_clean":"2024-09-13T17:22:37.898133+0000","last_became_active":"2024-09-13T17:22:35.745697+0000","last_became_peered":"2024-09-13T17:22:35.745697+0000","last_unstale":"2024-09-13T17:22:37.898133+0000","last_undegraded":"2024-09-13T17:22:37.898133+0000","last_fullsized":"2024-09-13T17:22:37.898133+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:07.078447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725896+0000","last_change":"2024-09-13T17:22:35.306034+0000","last_active":"2024-09-13T17:22:36.725896+0000","last_peered":"2024-09-13T17:22:36.725896+0000","last_clean":"2024-09-13T17:22:36.725896+0000","last_became_active":"2024-09-13T17:22:35.305762+0000","last_became_peered":"2024-09-13T17:22:35.305762+0000","last_unstale":"2024-09-13T17:22:36.725896+0000","last_undegraded":"2024-09-13T17:22:36.725896+0000","last_fullsized":"2024-09-13T17:22:36.725896+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:53:34.294075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723405+0000","last_change":"2024-09-13T17:22:35.750742+0000","last_active":"2024-09-13T17:22:36.723405+0000","last_peered":"2024-09-13T17:22:36.723405+0000","last_clean":"2024-09-13T17:22:36.723405+0000","last_became_active":"2024-09-13T17:22:35.750452+0000","last_became_peered":"2024-09-13T17:22:35.750452+0000","last_unstale":"2024-09-13T17:22:36.723405+0000","last_undegraded":"2024-09-13T17:22:36.723405+0000","last_fullsized":"2024-09-13T17:22:36.723405+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:49:03.818485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898147+0000","last_change":"2024-09-13T17:22:35.309572+0000","last_active":"2024-09-13T17:22:37.898147+0000","last_peered":"2024-09-13T17:22:37.898147+0000","last_clean":"2024-09-13T17:22:37.898147+0000","last_became_active":"2024-09-13T17:22:35.309265+0000","last_became_peered":"2024-09-13T17:22:35.309265+0000","last_unstale":"2024-09-13T17:22:37.898147+0000","last_undegraded":"2024-09-13T17:22:37.898147+0000","last_fullsized":"2024-09-13T17:22:37.898147+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:54.554007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723672+0000","last_change":"2024-09-13T17:22:35.744675+0000","last_active":"2024-09-13T17:22:36.723672+0000","last_peered":"2024-09-13T17:22:36.723672+0000","last_clean":"2024-09-13T17:22:36.723672+0000","last_became_active":"2024-09-13T17:22:35.743943+0000","last_became_peered":"2024-09-13T17:22:35.743943+0000","last_unstale":"2024-09-13T17:22:36.723672+0000","last_undegraded":"2024-09-13T17:22:36.723672+0000","last_fullsized":"2024-09-13T17:22:36.723672+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:20.076222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.7","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723952+0000","last_change":"2024-09-13T17:22:35.309954+0000","last_active":"2024-09-13T17:22:36.723952+0000","last_peered":"2024-09-13T17:22:36.723952+0000","last_clean":"2024-09-13T17:22:36.723952+0000","last_became_active":"2024-09-13T17:22:35.309609+0000","last_became_peered":"2024-09-13T17:22:35.309609+0000","last_unstale":"2024-09-13T17:22:36.723952+0000","last_undegraded":"2024-09-13T17:22:36.723952+0000","last_fullsized":"2024-09-13T17:22:36.723952+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:15.870289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898910+0000","last_change":"2024-09-13T17:22:35.754217+0000","last_active":"2024-09-13T17:22:37.898910+0000","last_peered":"2024-09-13T17:22:37.898910+0000","last_clean":"2024-09-13T17:22:37.898910+0000","last_became_active":"2024-09-13T17:22:35.753624+0000","last_became_peered":"2024-09-13T17:22:35.753624+0000","last_unstale":"2024-09-13T17:22:37.898910+0000","last_undegraded":"2024-09-13T17:22:37.898910+0000","last_fullsized":"2024-09-13T17:22:37.898910+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:10:58.849782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899218+0000","last_change":"2024-09-13T17:22:35.303858+0000","last_active":"2024-09-13T17:22:37.899218+0000","last_peered":"2024-09-13T17:22:37.899218+0000","last_clean":"2024-09-13T17:22:37.899218+0000","last_became_active":"2024-09-13T17:22:35.303616+0000","last_became_peered":"2024-09-13T17:22:35.303616+0000","last_unstale":"2024-09-13T17:22:37.899218+0000","last_undegraded":"2024-09-13T17:22:37.899218+0000","last_fullsized":"2024-09-13T17:22:37.899218+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:52.641424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898734+0000","last_change":"2024-09-13T17:22:35.749712+0000","last_active":"2024-09-13T17:22:37.898734+0000","last_peered":"2024-09-13T17:22:37.898734+0000","last_clean":"2024-09-13T17:22:37.898734+0000","last_became_active":"2024-09-13T17:22:35.747984+0000","last_became_peered":"2024-09-13T17:22:35.747984+0000","last_unstale":"2024-09-13T17:22:37.898734+0000","last_undegraded":"2024-09-13T17:22:37.898734+0000","last_fullsized":"2024-09-13T17:22:37.898734+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:57:04.322380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.899402+0000","last_change":"2024-09-13T17:22:35.306389+0000","last_active":"2024-09-13T17:22:37.899402+0000","last_peered":"2024-09-13T17:22:37.899402+0000","last_clean":"2024-09-13T17:22:37.899402+0000","last_became_active":"2024-09-13T17:22:35.305034+0000","last_became_peered":"2024-09-13T17:22:35.305034+0000","last_unstale":"2024-09-13T17:22:37.899402+0000","last_undegraded":"2024-09-13T17:22:37.899402+0000","last_fullsized":"2024-09-13T17:22:37.899402+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:06.360515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723779+0000","last_change":"2024-09-13T17:22:35.735804+0000","last_active":"2024-09-13T17:22:36.723779+0000","last_peered":"2024-09-13T17:22:36.723779+0000","last_clean":"2024-09-13T17:22:36.723779+0000","last_became_active":"2024-09-13T17:22:35.735543+0000","last_became_peered":"2024-09-13T17:22:35.735543+0000","last_unstale":"2024-09-13T17:22:36.723779+0000","last_undegraded":"2024-09-13T17:22:36.723779+0000","last_fullsized":"2024-09-13T17:22:36.723779+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:39:07.285604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898840+0000","last_change":"2024-09-13T17:22:35.307024+0000","last_active":"2024-09-13T17:22:37.898840+0000","last_peered":"2024-09-13T17:22:37.898840+0000","last_clean":"2024-09-13T17:22:37.898840+0000","last_became_active":"2024-09-13T17:22:35.306767+0000","last_became_peered":"2024-09-13T17:22:35.306767+0000","last_unstale":"2024-09-13T17:22:37.898840+0000","last_undegraded":"2024-09-13T17:22:37.898840+0000","last_fullsized":"2024-09-13T17:22:37.898840+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:19.210144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725009+0000","last_change":"2024-09-13T17:22:35.730715+0000","last_active":"2024-09-13T17:22:36.725009+0000","last_peered":"2024-09-13T17:22:36.725009+0000","last_clean":"2024-09-13T17:22:36.725009+0000","last_became_active":"2024-09-13T17:22:35.730032+0000","last_became_peered":"2024-09-13T17:22:35.730032+0000","last_unstale":"2024-09-13T17:22:36.725009+0000","last_undegraded":"2024-09-13T17:22:36.725009+0000","last_fullsized":"2024-09-13T17:22:36.725009+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:59:19.936403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.725647+0000","last_change":"2024-09-13T17:22:35.294050+0000","last_active":"2024-09-13T17:22:36.725647+0000","last_peered":"2024-09-13T17:22:36.725647+0000","last_clean":"2024-09-13T17:22:36.725647+0000","last_became_active":"2024-09-13T17:22:35.293757+0000","last_became_peered":"2024-09-13T17:22:35.293757+0000","last_unstale":"2024-09-13T17:22:36.725647+0000","last_undegraded":"2024-09-13T17:22:36.725647+0000","last_fullsized":"2024-09-13T17:22:36.725647+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:15.870289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898909+0000","last_change":"2024-09-13T17:22:35.752284+0000","last_active":"2024-09-13T17:22:37.898909+0000","last_peered":"2024-09-13T17:22:37.898909+0000","last_clean":"2024-09-13T17:22:37.898909+0000","last_became_active":"2024-09-13T17:22:35.747576+0000","last_became_peered":"2024-09-13T17:22:35.747576+0000","last_unstale":"2024-09-13T17:22:37.898909+0000","last_undegraded":"2024-09-13T17:22:37.898909+0000","last_fullsized":"2024-09-13T17:22:37.898909+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:45.650239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.897642+0000","last_change":"2024-09-13T17:22:35.303704+0000","last_active":"2024-09-13T17:22:37.897642+0000","last_peered":"2024-09-13T17:22:37.897642+0000","last_clean":"2024-09-13T17:22:37.897642+0000","last_became_active":"2024-09-13T17:22:35.301222+0000","last_became_peered":"2024-09-13T17:22:35.301222+0000","last_unstale":"2024-09-13T17:22:37.897642+0000","last_undegraded":"2024-09-13T17:22:37.897642+0000","last_fullsized":"2024-09-13T17:22:37.897642+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:14:05.823909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898125+0000","last_change":"2024-09-13T17:22:35.753631+0000","last_active":"2024-09-13T17:22:37.898125+0000","last_peered":"2024-09-13T17:22:37.898125+0000","last_clean":"2024-09-13T17:22:37.898125+0000","last_became_active":"2024-09-13T17:22:35.753250+0000","last_became_peered":"2024-09-13T17:22:35.753250+0000","last_unstale":"2024-09-13T17:22:37.898125+0000","last_undegraded":"2024-09-13T17:22:37.898125+0000","last_fullsized":"2024-09-13T17:22:37.898125+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:24.227699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.723609+0000","last_change":"2024-09-13T17:22:35.309663+0000","last_active":"2024-09-13T17:22:36.723609+0000","last_peered":"2024-09-13T17:22:36.723609+0000","last_clean":"2024-09-13T17:22:36.723609+0000","last_became_active":"2024-09-13T17:22:35.309103+0000","last_became_peered":"2024-09-13T17:22:35.309103+0000","last_unstale":"2024-09-13T17:22:36.723609+0000","last_undegraded":"2024-09-13T17:22:36.723609+0000","last_fullsized":"2024-09-13T17:22:36.723609+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:57.167129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898006+0000","last_change":"2024-09-13T17:22:35.744716+0000","last_active":"2024-09-13T17:22:37.898006+0000","last_peered":"2024-09-13T17:22:37.898006+0000","last_clean":"2024-09-13T17:22:37.898006+0000","last_became_active":"2024-09-13T17:22:35.744397+0000","last_became_peered":"2024-09-13T17:22:35.744397+0000","last_unstale":"2024-09-13T17:22:37.898006+0000","last_undegraded":"2024-09-13T17:22:37.898006+0000","last_fullsized":"2024-09-13T17:22:37.898006+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:00.326830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":11,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898210+0000","last_change":"2024-09-13T17:22:35.307927+0000","last_active":"2024-09-13T17:22:37.898210+0000","last_peered":"2024-09-13T17:22:37.898210+0000","last_clean":"2024-09-13T17:22:37.898210+0000","last_became_active":"2024-09-13T17:22:35.307680+0000","last_became_peered":"2024-09-13T17:22:35.307680+0000","last_unstale":"2024-09-13T17:22:37.898210+0000","last_undegraded":"2024-09-13T17:22:37.898210+0000","last_fullsized":"2024-09-13T17:22:37.898210+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:22.736023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1c","version":"0'0","reported_seq":10,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:37.898715+0000","last_change":"2024-09-13T17:22:35.743292+0000","last_active":"2024-09-13T17:22:37.898715+0000","last_peered":"2024-09-13T17:22:37.898715+0000","last_clean":"2024-09-13T17:22:37.898715+0000","last_became_active":"2024-09-13T17:22:35.743045+0000","last_became_peered":"2024-09-13T17:22:35.743045+0000","last_unstale":"2024-09-13T17:22:37.898715+0000","last_undegraded":"2024-09-13T17:22:37.898715+0000","last_fullsized":"2024-09-13T17:22:37.898715+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":203,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:35.281613+0000","last_clean_scrub_stamp":"2024-09-13T17:22:35.281613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:20.076222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1d","version":"204'2","reported_seq":13,"reported_epoch":205,"state":"active+clean","last_fresh":"2024-09-13T17:22:36.726050+0000","last_change":"2024-09-13T17:22:35.306449+0000","last_active":"2024-09-13T17:22:36.726050+0000","last_peered":"2024-09-13T17:22:36.726050+0000","last_clean":"2024-09-13T17:22:36.726050+0000","last_became_active":"2024-09-13T17:22:35.306096+0000","last_became_peered":"2024-09-13T17:22:35.306096+0000","last_unstale":"2024-09-13T17:22:36.726050+0000","last_undegraded":"2024-09-13T17:22:36.726050+0000","last_fullsized":"2024-09-13T17:22:36.726050+0000","mapping_epoch":202,"log_start":"0'0","ondisk_log_start":"0'0","created":202,"last_epoch_clean":203,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:22:34.278732+0000","last_clean_scrub_stamp":"2024-09-13T17:22:34.278732+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:43:54.367489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":322,"reported_epoch":204,"state":"active+clean","last_fresh":"2024-09-13T17:22:35.728472+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:22:35.728472+0000","last_peered":"2024-09-13T17:22:35.728472+0000","last_clean":"2024-09-13T17:22:35.728472+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:22:35.728472+0000","last_undegraded":"2024-09-13T17:22:35.728472+0000","last_fullsized":"2024-09-13T17:22:35.728472+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575027,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70252,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301588,"statfs":{"total":96636764160,"available":96564826112,"internally_reserved":0,"allocated":3170304,"data_stored":1726058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.63300000000000001,"15min":0.64500000000000002},"min":{"1min":0.45800000000000002,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.85199999999999998,"5min":1.008,"15min":2.0880000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.66200000000000003,"15min":0.67300000000000004},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.316},"max":{"1min":0.873,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.91800000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.66900000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.89500000000000002,"5min":1.161,"15min":2.5030000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.65600000000000003,"15min":0.69899999999999995},"min":{"1min":0.52800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86099999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.85499999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.68000000000000005,"15min":0.68799999999999994},"min":{"1min":0.54900000000000004,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.89100000000000001,"5min":1.169,"15min":1.544},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68100000000000005,"15min":0.69599999999999995},"min":{"1min":0.621,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.90100000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.95399999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66500000000000004,"15min":0.67400000000000004},"min":{"1min":0.499,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81999999999999995,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.746,"5min":0.70799999999999996,"15min":0.73399999999999999},"min":{"1min":0.59999999999999998,"5min":0.437,"15min":0.437},"max":{"1min":0.92700000000000005,"5min":1.274,"15min":2.8239999999999998},"last":0.81699999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57899999999999996,"15min":0.60799999999999998},"min":{"1min":0.40400000000000003,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73299999999999998,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.627,"5min":0.64000000000000001,"15min":0.65500000000000003},"min":{"1min":0.44900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84799999999999998,"5min":1.234,"15min":1.234},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.63,"15min":0.65400000000000003},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.88200000000000001,"5min":1.135,"15min":1.143},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.72299999999999998,"15min":0.746},"min":{"1min":0.48499999999999999,"5min":0.44800000000000001,"15min":0.42599999999999999},"max":{"1min":0.90800000000000003,"5min":1.915,"15min":1.915},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:22:22 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.51100000000000001,"5min":0.39000000000000001,"15min":0.38200000000000001},"max":{"1min":0.93300000000000005,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.70699999999999996,"15min":0.73099999999999998},"min":{"1min":0.56499999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.94499999999999995,"5min":1.607,"15min":1.607},"last":0.70699999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575028,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65580,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306260,"statfs":{"total":96636764160,"available":96569610240,"internally_reserved":0,"allocated":2580480,"data_stored":1135676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.67100000000000004,"15min":0.67500000000000004},"min":{"1min":0.47199999999999998,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.97099999999999997,"5min":1.0209999999999999,"15min":3.6629999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.77200000000000002,"15min":0.74099999999999999},"min":{"1min":0.61199999999999999,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.95599999999999996,"5min":5.9800000000000004,"15min":5.9800000000000004},"last":0.68300000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.63200000000000001,"15min":0.65800000000000003},"min":{"1min":0.496,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.998,"5min":0.998,"15min":3.7040000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.67100000000000004},"min":{"1min":0.59199999999999997,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.871,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.71699999999999997,"15min":0.68200000000000005},"min":{"1min":0.56699999999999995,"5min":0.44,"15min":0.35599999999999998},"max":{"1min":0.79800000000000004,"5min":5.9240000000000004,"15min":5.9240000000000004},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.58599999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.92000000000000004,"5min":0.93300000000000005,"15min":3.2629999999999999},"last":0.61799999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69099999999999995,"15min":0.70799999999999996},"min":{"1min":0.57099999999999995,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93500000000000005,"5min":1.167,"15min":3.6520000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.748,"5min":0.68200000000000005,"15min":0.70299999999999996},"min":{"1min":0.51200000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.63400000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.46800000000000003,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":0.78200000000000003,"5min":1.262,"15min":1.6439999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61799999999999999,"15min":0.64300000000000002},"min":{"1min":0.497,"5min":0.40400000000000003,"15min":0.29199999999999998},"max":{"1min":0.88400000000000001,"5min":0.90900000000000003,"15min":4.7750000000000004},"last":0.50800000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66200000000000003,"15min":0.68700000000000006},"min":{"1min":0.52400000000000002,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":0.83799999999999997,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.625,"15min":0.66400000000000003},"min":{"1min":0.434,"5min":0.376,"15min":0.32700000000000001},"max":{"1min":0.76000000000000001,"5min":1.1559999999999999,"15min":3.609},"last":0.45600000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:22:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.72499999999999998},"min":{"1min":0.55400000000000005,"5min":0.379,"15min":0.378},"max":{"1min":0.86799999999999999,"5min":0.93700000000000006,"15min":4.9669999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.76100000000000001,"15min":0.78300000000000003},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":0.88100000000000001,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.60499999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575027,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63984,"kb_used_data":2524,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307856,"statfs":{"total":96636764160,"available":96571244544,"internally_reserved":0,"allocated":2584576,"data_stored":1141920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64800000000000002,"15min":0.76300000000000001},"min":{"1min":0.43099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.1220000000000001,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.67900000000000005,"15min":0.73699999999999999},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.73599999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.67800000000000005,"15min":0.72399999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.93400000000000005,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.755},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68700000000000006,"15min":0.69399999999999995},"min":{"1min":0.48199999999999998,"5min":0.45800000000000002,"15min":0.41099999999999998},"max":{"1min":1.169,"5min":1.482,"15min":7.7370000000000001},"last":0.71399999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.68999999999999995,"15min":0.73699999999999999},"min":{"1min":0.47599999999999998,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.67600000000000005,"15min":0.72299999999999998},"min":{"1min":0.46300000000000002,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.98599999999999999,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.69999999999999996,"15min":1.081},"min":{"1min":0.505,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":113.363},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.72299999999999998,"15min":0.74099999999999999},"min":{"1min":0.55500000000000005,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.97599999999999998,"5min":1.113,"15min":4.1180000000000003},"last":0.93799999999999994}]},{"osd":4,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61799999999999999,"15min":0.621},"min":{"1min":0.38300000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.89200000000000002,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.65100000000000002,"15min":0.67800000000000005},"min":{"1min":0.40000000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95599999999999996,"5min":1.03,"15min":2.827},"last":0.90200000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.67000000000000004,"15min":0.69099999999999995},"min":{"1min":0.49099999999999999,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":0.99099999999999999,"5min":1.73,"15min":2.2320000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70599999999999996,"15min":0.71599999999999997},"min":{"1min":0.54100000000000004,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":1.032,"5min":1.532,"15min":4.1500000000000004},"last":0.86899999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:22:27 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.73399999999999999,"15min":0.752},"min":{"1min":0.51800000000000002,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":1.079,"5min":1.079,"15min":4.1029999999999998},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.746,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":0.92800000000000005,"5min":1.363,"15min":2.02},"last":1.169}]}]},{"osd":4,"up_from":13,"seq":55834575026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73780,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94298060,"statfs":{"total":96636764160,"available":96561213440,"internally_reserved":0,"allocated":2588672,"data_stored":1142502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":1.1870000000000001,"15min":0.83399999999999996},"min":{"1min":0.498,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":1.2729999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.755,"5min":1.1899999999999999,"15min":0.873},"min":{"1min":0.47699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.97399999999999998,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.75700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.85099999999999998,"15min":0.71999999999999997},"min":{"1min":0.49099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.94599999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.72999999999999998,"15min":0.67200000000000004},"min":{"1min":0.55800000000000005,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":1.3160000000000001,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.68200000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.70299999999999996,"15min":0.67400000000000004},"min":{"1min":0.53200000000000003,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":1.333,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.68999999999999995,"15min":0.70399999999999996},"min":{"1min":0.58799999999999997,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0109999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.90900000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":1.095,"15min":0.81799999999999995},"min":{"1min":0.58799999999999997,"5min":0.437,"15min":0.378},"max":{"1min":1.24,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.76000000000000001,"15min":0.72099999999999997},"min":{"1min":0.49399999999999999,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.96399999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.69799999999999995,"15min":0.65200000000000002},"min":{"1min":0.41599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.83999999999999997,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.63200000000000001,"15min":0.64400000000000002},"min":{"1min":0.501,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.77200000000000002,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.69399999999999995,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":1.498,"5min":2.976,"15min":2.976},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.77600000000000002,"15min":0.73799999999999999},"min":{"1min":0.48499999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1850000000000001,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.873}]},{"osd":7,"last update":"Fri Sep 13 17:22:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.70699999999999996},"min":{"1min":0.52900000000000003,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.76500000000000001,"15min":0.748},"min":{"1min":0.45600000000000002,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.91200000000000003,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.89400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575027,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69676,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94302164,"statfs":{"total":96636764160,"available":96565415936,"internally_reserved":0,"allocated":2588672,"data_stored":1137254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42299999999999999,"15min":0.433},"min":{"1min":0.308,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":0.70499999999999996,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.46200000000000002,"15min":0.48199999999999998},"min":{"1min":0.36399999999999999,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":2.96},"last":0.502}]},{"osd":1,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47299999999999998,"15min":0.502},"min":{"1min":0.33000000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":3.5600000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56299999999999994,"15min":0.60199999999999998},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.32900000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.56599999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52700000000000002,"15min":0.52900000000000003},"min":{"1min":0.45100000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53400000000000003,"15min":0.55500000000000005},"min":{"1min":0.42699999999999999,"5min":0.378,"15min":0.34499999999999997},"max":{"1min":0.875,"5min":0.875,"15min":1.3819999999999999},"last":0.52100000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.73399999999999999,"15min":0.747},"min":{"1min":0.60499999999999998,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.208,"5min":1.21,"15min":2.8610000000000002},"last":0.626},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.76200000000000001,"15min":0.76000000000000001},"min":{"1min":0.60199999999999998,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.169,"5min":1.23,"15min":1.3640000000000001},"last":0.58899999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.76200000000000001,"15min":0.76500000000000001},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.228,"5min":1.323,"15min":1.391},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.871,"5min":0.754,"15min":0.76500000000000001},"min":{"1min":0.60099999999999998,"5min":0.442,"15min":0.442},"max":{"1min":1.3200000000000001,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.54700000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.76800000000000002,"15min":0.76700000000000002},"min":{"1min":0.56299999999999994,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.218,"5min":1.218,"15min":1.401},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.79600000000000004,"15min":0.78500000000000003},"min":{"1min":0.68100000000000005,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.2949999999999999,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:22:26 2024","interfaces":[{"interface":"back","average":{"1min":0.85499999999999998,"5min":0.78300000000000003,"15min":0.79400000000000004},"min":{"1min":0.60899999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1679999999999999,"5min":1.27,"15min":2.899},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.80300000000000005,"15min":0.80000000000000004},"min":{"1min":0.66700000000000004,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.264,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.68300000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575027,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47604,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324236,"statfs":{"total":96636764160,"available":96588017664,"internally_reserved":0,"allocated":2588672,"data_stored":1136221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.47299999999999998,"15min":0.436},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.67500000000000004,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.57399999999999995},"min":{"1min":0.375,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.748,"5min":1.593,"15min":30.023},"last":0.48299999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.53700000000000003,"15min":1.367},"min":{"1min":0.36299999999999999,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.79900000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.52500000000000002,"15min":1.4650000000000001},"min":{"1min":0.38,"5min":0.32500000000000001,"15min":0.316},"max":{"1min":0.94099999999999995,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.438}]},{"osd":3,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55200000000000005,"15min":0.62},"min":{"1min":0.38900000000000001,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.84199999999999997,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.498},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.67200000000000004,"15min":0.61199999999999999},"min":{"1min":0.51400000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91100000000000003,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.54300000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.78200000000000003,"15min":0.81000000000000005},"min":{"1min":0.64800000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.98799999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79100000000000004,"15min":0.85899999999999999},"min":{"1min":0.65800000000000003,"5min":0.48999999999999999,"15min":0.371},"max":{"1min":1.075,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76400000000000001,"15min":0.876},"min":{"1min":0.66100000000000003,"5min":0.44500000000000001,"15min":0.376},"max":{"1min":0.89800000000000002,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.79300000000000004,"15min":0.78100000000000003},"min":{"1min":0.65300000000000002,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.014,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.81399999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.79400000000000004,"15min":0.82199999999999995},"min":{"1min":0.61799999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1339999999999999,"5min":6.4800000000000004,"15min":10.664},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.78700000000000003,"15min":0.89000000000000001},"min":{"1min":0.63900000000000001,"5min":0.503,"15min":0.47999999999999998},"max":{"1min":1.121,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.76100000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.77800000000000002,"15min":0.82299999999999995},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.194,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.80500000000000005,"15min":0.90600000000000003},"min":{"1min":0.71299999999999997,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.095,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.71799999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575026,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":91828,"kb_used_data":2528,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94280012,"statfs":{"total":96636764160,"available":96542732288,"internally_reserved":0,"allocated":2588672,"data_stored":1142010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.434,"15min":0.46700000000000003},"min":{"1min":0.33100000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.63600000000000001,"5min":0.67000000000000004,"15min":5.4939999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.52000000000000002},"min":{"1min":0.41999999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.438}]},{"osd":2,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.42799999999999999,"15min":0.45900000000000002},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.5,"15min":0.52200000000000002},"min":{"1min":0.34200000000000003,"5min":0.308,"15min":0.308},"max":{"1min":0.629,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.46000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58199999999999996,"15min":0.65600000000000003},"min":{"1min":0.47999999999999998,"5min":0.42699999999999999,"15min":0.376},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":16.382999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.79800000000000004,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.57099999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.75700000000000001,"15min":0.78800000000000003},"min":{"1min":0.51600000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.03,"5min":1.28,"15min":6.1630000000000003},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.79500000000000004,"15min":0.80400000000000005},"min":{"1min":0.625,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.0009999999999999,"5min":1.23,"15min":1.78},"last":0.73199999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.76800000000000002,"15min":0.76900000000000002},"min":{"1min":0.502,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":1.077,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.755,"15min":0.78400000000000003},"min":{"1min":0.56100000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.073,"5min":1.175,"15min":6.1189999999999998},"last":0.58099999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.76700000000000002,"15min":0.78100000000000003},"min":{"1min":0.57299999999999995,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.0309999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.77400000000000002,"15min":0.80300000000000005},"min":{"1min":0.59299999999999997,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.135,"5min":1.246,"15min":6.1970000000000001},"last":0.628}]},{"osd":7,"last update":"Fri Sep 13 17:22:24 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78300000000000003,"15min":0.79200000000000004},"min":{"1min":0.58799999999999997,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.996,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.80200000000000005,"15min":0.81299999999999994},"min":{"1min":0.61099999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.1830000000000001,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575028,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70252,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301588,"statfs":{"total":96636764160,"available":96564826112,"internally_reserved":0,"allocated":3170304,"data_stored":1725641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.432,"15min":0.48999999999999999},"min":{"1min":0.29599999999999999,"5min":0.247,"15min":0.247},"max":{"1min":0.64900000000000002,"5min":1.986,"15min":5.5259999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.48599999999999999,"15min":0.53500000000000003},"min":{"1min":0.32100000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.81599999999999995,"5min":2.371,"15min":5.4889999999999999},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.49299999999999999,"15min":0.55700000000000005},"min":{"1min":0.35799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.56100000000000005},"min":{"1min":0.38900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68000000000000005,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47699999999999998,"15min":0.53100000000000003},"min":{"1min":0.33500000000000002,"5min":0.32400000000000001,"15min":0.314},"max":{"1min":0.66600000000000004,"5min":0.878,"15min":4.7549999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.55100000000000005,"15min":0.60299999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.42199999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77000000000000002,"15min":0.79900000000000004},"min":{"1min":0.61299999999999999,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1,"5min":3.8849999999999998,"15min":4.734},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.73499999999999999,"5min":0.72199999999999998,"15min":0.78300000000000003},"min":{"1min":0.53900000000000003,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":0.98799999999999999,"5min":1.137,"15min":4.8079999999999998},"last":0.70899999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73599999999999999,"15min":0.79400000000000004},"min":{"1min":0.58499999999999996,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":0.91200000000000003,"5min":1.0980000000000001,"15min":4.944},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.76800000000000002,"15min":0.80600000000000005},"min":{"1min":0.56200000000000006,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.026,"5min":1.139,"15min":4.9710000000000001},"last":0.83299999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.76300000000000001,"15min":0.83199999999999996},"min":{"1min":0.57999999999999996,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.0229999999999999,"5min":1.161,"15min":4.8470000000000004},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.78700000000000003,"15min":0.82099999999999995},"min":{"1min":0.55000000000000004,"5min":0.495,"15min":0.48599999999999999},"max":{"1min":1.1339999999999999,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.77300000000000002}]},{"osd":7,"last update":"Fri Sep 13 17:22:21 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.82499999999999996,"15min":0.84599999999999997},"min":{"1min":0.57899999999999996,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.0609999999999999,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.77400000000000002,"15min":0.82699999999999996},"min":{"1min":0.56899999999999995,"5min":0.53100000000000003,"15min":0.505},"max":{"1min":0.97399999999999998,"5min":1.1220000000000001,"15min":4.9950000000000001},"last":0.86699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:22:40.326 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:22:40.326 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:22:40.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.522+0000 7f0b2f800640 1 Processor -- start 2024-09-13T17:22:40.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.522+0000 7f0b2f800640 1 -- start start 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.523+0000 7f0b2f800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b28070430 0x7f0b2815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b2815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2f800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2815ab60 con 0x7f0b28159770 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2f800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2806f920 con 0x7f0b28070430 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815b290 con 0x7f0b2806f3f0 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2d800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b2815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2ce00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f0b28159770 0x7f0b2806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:32930/0 (socket says 172.21.15.28:32930) 2024-09-13T17:22:40.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b27e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0b28070430 0x7f0b2815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.524+0000 7f0b2ce00640 1 -- 172.21.15.28:0/4271409521 learned_addr learned my addr 172.21.15.28:0/4271409521 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:40.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.525+0000 7f0b2d800640 1 -- 172.21.15.28:0/4271409521 >> v2:172.21.15.189:3300/0 conn(0x7f0b28070430 msgr2=0x7f0b2815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:40.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.525+0000 7f0b2d800640 1 --2- 172.21.15.28:0/4271409521 >> v2:172.21.15.189:3300/0 conn(0x7f0b28070430 0x7f0b2815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.525+0000 7f0b2d800640 1 -- 172.21.15.28:0/4271409521 >> v1:172.21.15.189:6789/0 conn(0x7f0b28159770 legacy=0x7f0b2806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.525+0000 7f0b2d800640 1 -- 172.21.15.28:0/4271409521 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2815b570 con 0x7f0b2806f3f0 2024-09-13T17:22:40.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.526+0000 7f0b2d800640 1 --2- 172.21.15.28:0/4271409521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b2815c2b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009d90 tx=0x7f0b2002c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2ae03c3323ca3d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.526+0000 7f0b27400640 1 -- 172.21.15.28:0/4271409521 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20007740 con 0x7f0b2806f3f0 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.526+0000 7f0b27400640 1 -- 172.21.15.28:0/4271409521 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2002f050 con 0x7f0b2806f3f0 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.526+0000 7f0b27400640 1 -- 172.21.15.28:0/4271409521 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20007ac0 con 0x7f0b2806f3f0 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.526+0000 7f0b2f800640 1 -- 172.21.15.28:0/4271409521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 msgr2=0x7f0b2815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 --2- 172.21.15.28:0/4271409521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b2815c2b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009d90 tx=0x7f0b2002c7e0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 -- 172.21.15.28:0/4271409521 shutdown_connections 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 --2- 172.21.15.28:0/4271409521 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b2815c2b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 --2- 172.21.15.28:0/4271409521 >> v2:172.21.15.189:3300/0 conn(0x7f0b28070430 0x7f0b2815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 -- 172.21.15.28:0/4271409521 >> 172.21.15.28:0/4271409521 conn(0x7f0b280b0270 msgr2=0x7f0b280b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 -- 172.21.15.28:0/4271409521 shutdown_connections 2024-09-13T17:22:40.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.527+0000 7f0b2f800640 1 -- 172.21.15.28:0/4271409521 wait complete. 2024-09-13T17:22:40.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.528+0000 7f0b2f800640 1 Processor -- start 2024-09-13T17:22:40.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.528+0000 7f0b2f800640 1 -- start start 2024-09-13T17:22:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b281563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28159770 0x7f0b280b26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281e31c0 con 0x7f0b28070430 2024-09-13T17:22:40.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28155f30 con 0x7f0b28159770 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.529+0000 7f0b2f800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28156e40 con 0x7f0b2806f3f0 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52938/0 (socket says 172.21.15.28:52938) 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 -- 172.21.15.28:0/2785919740 learned_addr learned my addr 172.21.15.28:0/2785919740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b2ce00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b281563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b2d800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28159770 0x7f0b280b26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 -- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 msgr2=0x7f0b281563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b281563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 -- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28159770 msgr2=0x7f0b280b26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28159770 0x7f0b280b26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 -- 172.21.15.28:0/2785919740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009a70 con 0x7f0b28070430 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.530+0000 7f0b27e00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e9b0 tx=0x7f0b1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.531+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b14009670 con 0x7f0b28070430 2024-09-13T17:22:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.531+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1400c9a0 con 0x7f0b28070430 2024-09-13T17:22:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.531+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1400cd50 con 0x7f0b28070430 2024-09-13T17:22:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.531+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28157120 con 0x7f0b28070430 2024-09-13T17:22:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.531+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28155bc0 con 0x7f0b28070430 2024-09-13T17:22:40.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.532+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b281560a0 con 0x7f0b28070430 2024-09-13T17:22:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.533+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0b14013470 con 0x7f0b28070430 2024-09-13T17:22:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.534+0000 7f0b24c00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.534+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b14089630 con 0x7f0b28070430 2024-09-13T17:22:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.534+0000 7f0b2ce00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.535+0000 7f0b2ce00640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c005880 tx=0x7f0b1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.538+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0b14054830 con 0x7f0b28070430 2024-09-13T17:22:40.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.783+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0b2806fd20 con 0x7f0b28070430 2024-09-13T17:22:40.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.788+0000 7f0b24c00640 1 -- 172.21.15.28:0/2785919740 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v27) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0b14057e50 con 0x7f0b28070430 2024-09-13T17:22:40.788 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 msgr2=0x7f0b0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c005880 tx=0x7f0b1c009040 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 msgr2=0x7f0b281e2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400e9b0 tx=0x7f0b1400ee70 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 shutdown_connections 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28159770 0x7f0b280b26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b28070430 0x7f0b281e2c80 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 --2- 172.21.15.28:0/2785919740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b2806f3f0 0x7f0b281563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 >> 172.21.15.28:0/2785919740 conn(0x7f0b280b0270 msgr2=0x7f0b2806bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:40.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 shutdown_connections 2024-09-13T17:22:40.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.790+0000 7f0b2f800640 1 -- 172.21.15.28:0/2785919740 wait complete. 2024-09-13T17:22:40.806 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.961+0000 7f7eaa400640 1 Processor -- start 2024-09-13T17:22:40.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.962+0000 7f7eaa400640 1 -- start start 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7eaa400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea40b0b50 0x7f7ea40af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7eaa400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea406b900 con 0x7f7ea4154a90 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7eaa400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ea406bbe0 con 0x7f7ea406b360 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7eaa400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ea406bec0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7ea9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea40b0b50 0x7f7ea40af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7ea9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7ea40b0b50 0x7f7ea40af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52948/0 (socket says 172.21.15.28:52948) 2024-09-13T17:22:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7ea9000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7ea4154a90 0x7f7ea415b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40276/0 (socket says 172.21.15.28:40276) 2024-09-13T17:22:40.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.963+0000 7f7ea9a00640 1 -- 172.21.15.28:0/3040381356 learned_addr learned my addr 172.21.15.28:0/3040381356 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.964+0000 7f7ea9a00640 1 -- 172.21.15.28:0/3040381356 >> v1:172.21.15.189:6789/0 conn(0x7f7ea406b360 legacy=0x7f7ea40b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.964+0000 7f7ea9a00640 1 -- 172.21.15.28:0/3040381356 >> v1:172.21.15.28:6789/0 conn(0x7f7ea4154a90 legacy=0x7f7ea415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.964+0000 7f7ea9a00640 1 -- 172.21.15.28:0/3040381356 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea406c1a0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.964+0000 7f7ea3e00640 1 -- 172.21.15.28:0/3040381356 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63733951 0 0) 0x7f7ea406b900 con 0x7f7ea4154a90 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7ea9a00640 1 --2- 172.21.15.28:0/3040381356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea40af4b0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009e50 tx=0x7f7e9c02a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b4078347e53b02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7ea3e00640 1 -- 172.21.15.28:0/3040381356 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c031600 con 0x7f7ea40b0b50 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7ea3e00640 1 -- 172.21.15.28:0/3040381356 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e9c02d050 con 0x7f7ea40b0b50 2024-09-13T17:22:40.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7ea3e00640 1 -- 172.21.15.28:0/3040381356 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e9c0319a0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7eaa400640 1 -- 172.21.15.28:0/3040381356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 msgr2=0x7f7ea40af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.965+0000 7f7eaa400640 1 --2- 172.21.15.28:0/3040381356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea40af4b0 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7f7e9c009e50 tx=0x7f7e9c02a7c0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.966+0000 7f7eaa400640 1 -- 172.21.15.28:0/3040381356 shutdown_connections 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.966+0000 7f7eaa400640 1 --2- 172.21.15.28:0/3040381356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea40af4b0 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.966+0000 7f7eaa400640 1 -- 172.21.15.28:0/3040381356 >> 172.21.15.28:0/3040381356 conn(0x7f7ea4072e40 msgr2=0x7f7ea4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:40.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.966+0000 7f7eaa400640 1 -- 172.21.15.28:0/3040381356 shutdown_connections 2024-09-13T17:22:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.966+0000 7f7eaa400640 1 -- 172.21.15.28:0/3040381356 wait complete. 2024-09-13T17:22:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.967+0000 7f7eaa400640 1 Processor -- start 2024-09-13T17:22:40.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.967+0000 7f7eaa400640 1 -- start start 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 0x7f7ea40c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 0x7f7ea4132190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4156af0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea413d500 con 0x7f7ea406b360 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.968+0000 7f7eaa400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea4156200 con 0x7f7ea4154a90 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52954/0 (socket says 172.21.15.28:52954) 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 -- 172.21.15.28:0/1114504423 learned_addr learned my addr 172.21.15.28:0/1114504423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7ea9000640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 0x7f7ea40c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7ea9a00640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 0x7f7ea4132190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 -- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 msgr2=0x7f7ea4132190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 0x7f7ea4132190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 -- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 msgr2=0x7f7ea40c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 0x7f7ea40c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 -- 172.21.15.28:0/1114504423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9c009b30 con 0x7f7ea40b0b50 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7ea9a00640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 0x7f7ea4132190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7ea9000640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 0x7f7ea40c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:40.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.969+0000 7f7e9be00640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c450 tx=0x7f7e8c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.970+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0052d0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.970+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c005430 con 0x7f7ea40b0b50 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.970+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea41564a0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.970+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c012870 con 0x7f7ea40b0b50 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.970+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea40c4f70 con 0x7f7ea40b0b50 2024-09-13T17:22:40.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.971+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea415a740 con 0x7f7ea40b0b50 2024-09-13T17:22:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.972+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e8c0129d0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.973+0000 7f7ea1600640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 0x7f7e8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:40.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.973+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e8c088ed0 con 0x7f7ea40b0b50 2024-09-13T17:22:40.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.973+0000 7f7ea9000640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 0x7f7e8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:40.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.974+0000 7f7ea9000640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 0x7f7e8006d0c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7e94007fd0 tx=0x7f7e94007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:40.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:40.977+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e8c0540a0 con 0x7f7ea40b0b50 2024-09-13T17:22:41.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.193+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ea4067580 con 0x7f7ea40b0b50 2024-09-13T17:22:41.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.194+0000 7f7ea1600640 1 -- 172.21.15.28:0/1114504423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7f7e8c0576c0 con 0x7f7ea40b0b50 2024-09-13T17:22:41.196 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:41.196 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:22:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:22:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 msgr2=0x7f7e8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 0x7f7e8006d0c0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f7e94007fd0 tx=0x7f7e94007dc0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 msgr2=0x7f7ea412f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c00c450 tx=0x7f7e8c00c910 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 shutdown_connections 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e8006ac00 0x7f7e8006d0c0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7ea4154a90 0x7f7ea4132190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ea40b0b50 0x7f7ea412f810 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 --2- 172.21.15.28:0/1114504423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ea406b360 0x7f7ea40c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 >> 172.21.15.28:0/1114504423 conn(0x7f7ea4072e40 msgr2=0x7f7ea40b2640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 shutdown_connections 2024-09-13T17:22:41.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.196+0000 7f7eaa400640 1 -- 172.21.15.28:0/1114504423 wait complete. 2024-09-13T17:22:41.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 172, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:22:35.717384+0000', 'modified': '2024-09-13T17:22:38.727402+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11478': {'gid': 11478, 'name': 'c', 'rank': 0, 'incarnation': 169, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6834/299660514', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 299660514}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 299660514}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:22:41.616 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:22:41.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:22:41.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:22:41.718 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:22:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.776+0000 7f5e8ea00640 1 Processor -- start 2024-09-13T17:22:41.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.776+0000 7f5e8ea00640 1 -- start start 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.777+0000 7f5e8ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e8815ab60 0x7f5e8806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.777+0000 7f5e8ea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8806c8b0 con 0x7f5e8815a780 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e8ea00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e8806cb90 con 0x7f5e88071490 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e8ea00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e880cc980 con 0x7f5e8815ab60 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e8815ab60 0x7f5e8806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5e8815ab60 0x7f5e8806bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52958/0 (socket says 172.21.15.28:52958) 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 -- 172.21.15.28:0/210569916 learned_addr learned my addr 172.21.15.28:0/210569916 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e87e00640 1 --1- 172.21.15.28:0/210569916 >> v1:172.21.15.28:6789/0 conn(0x7f5e8815a780 0x7f5e88070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40292/0 (socket says 172.21.15.28:40292) 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 -- 172.21.15.28:0/210569916 >> v1:172.21.15.189:6789/0 conn(0x7f5e88071490 legacy=0x7f5e8806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 -- 172.21.15.28:0/210569916 >> v1:172.21.15.28:6789/0 conn(0x7f5e8815a780 legacy=0x7f5e88070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.778+0000 7f5e7f400640 1 -- 172.21.15.28:0/210569916 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880ccc60 con 0x7f5e8815ab60 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.779+0000 7f5e7f400640 1 --2- 172.21.15.28:0/210569916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e8815ab60 0x7f5e8806bb40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009870 tx=0x7f5e7002a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36faca9a3a2c5bdb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.779+0000 7f5e87400640 1 -- 172.21.15.28:0/210569916 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70032440 con 0x7f5e8815ab60 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e87400640 1 -- 172.21.15.28:0/210569916 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e700325a0 con 0x7f5e8815ab60 2024-09-13T17:22:41.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e87400640 1 -- 172.21.15.28:0/210569916 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70032930 con 0x7f5e8815ab60 2024-09-13T17:22:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 -- 172.21.15.28:0/210569916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e8815ab60 msgr2=0x7f5e8806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/210569916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e8815ab60 0x7f5e8806bb40 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009870 tx=0x7f5e7002a2a0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 -- 172.21.15.28:0/210569916 shutdown_connections 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/210569916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e8815ab60 0x7f5e8806bb40 unknown :-1 s=CLOSED pgs=2469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 -- 172.21.15.28:0/210569916 >> 172.21.15.28:0/210569916 conn(0x7f5e880b0270 msgr2=0x7f5e880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 -- 172.21.15.28:0/210569916 shutdown_connections 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.780+0000 7f5e8ea00640 1 -- 172.21.15.28:0/210569916 wait complete. 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.782+0000 7f5e8ea00640 1 Processor -- start 2024-09-13T17:22:41.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.782+0000 7f5e8ea00640 1 -- start start 2024-09-13T17:22:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e8815a780 0x7f5e880c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e8815ab60 0x7f5e880c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b2410 con 0x7f5e88071490 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e881f3970 con 0x7f5e8815ab60 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b1a70 con 0x7f5e8815a780 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52962/0 (socket says 172.21.15.28:52962) 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 -- 172.21.15.28:0/3056541795 learned_addr learned my addr 172.21.15.28:0/3056541795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e8815ab60 0x7f5e880c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e7f400640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e8815a780 0x7f5e880c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 -- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e8815a780 msgr2=0x7f5e880c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e8815a780 0x7f5e880c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 -- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e8815ab60 msgr2=0x7f5e880c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e8815ab60 0x7f5e880c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.783+0000 7f5e87e00640 1 -- 172.21.15.28:0/3056541795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e700094d0 con 0x7f5e88071490 2024-09-13T17:22:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e87e00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c4d0 tx=0x7f5e7800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e84c00640 1 -- 172.21.15.28:0/3056541795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78007480 con 0x7f5e88071490 2024-09-13T17:22:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e84c00640 1 -- 172.21.15.28:0/3056541795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e780075e0 con 0x7f5e88071490 2024-09-13T17:22:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e84c00640 1 -- 172.21.15.28:0/3056541795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e780049e0 con 0x7f5e88071490 2024-09-13T17:22:41.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880b1d50 con 0x7f5e88071490 2024-09-13T17:22:41.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.784+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e881f3bd0 con 0x7f5e88071490 2024-09-13T17:22:41.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.787+0000 7f5e84c00640 1 -- 172.21.15.28:0/3056541795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5e78004b40 con 0x7f5e88071490 2024-09-13T17:22:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.787+0000 7f5e84c00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.787+0000 7f5e84c00640 1 -- 172.21.15.28:0/3056541795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e780883c0 con 0x7f5e88071490 2024-09-13T17:22:41.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.788+0000 7f5e7f400640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.789+0000 7f5e7f400640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 0x7f5e6406d0c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5e70034000 tx=0x7f5e7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.789+0000 7f5e9186d740 1 Processor -- start 2024-09-13T17:22:41.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.789+0000 7f5e9186d740 1 -- start start 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 0x5624a239eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23c5b80 0x5624a23ec0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a2307ab0 con 0x5624a239be90 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a2305240 con 0x5624a23c5b80 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624a2300b30 con 0x5624a239e6d0 2024-09-13T17:22:41.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e87e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 0x5624a239eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e8ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23c5b80 0x5624a23ec0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52976/0 (socket says 172.21.15.28:52976) 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.790+0000 7f5e7f400640 1 -- 172.21.15.28:0/631253504 learned_addr learned my addr 172.21.15.28:0/631253504 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 -- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 msgr2=0x5624a239eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 0x5624a239eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 -- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23c5b80 msgr2=0x5624a23ec0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23c5b80 0x5624a23ec0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 -- 172.21.15.28:0/631253504 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a22cbbd0 con 0x5624a239be90 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e87e00640 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 0x5624a239eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7f400640 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5e70035040 tx=0x7f5e7002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e92e5fd02296055 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7cc00640 1 -- 172.21.15.28:0/631253504 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7003c8c0 con 0x5624a239be90 2024-09-13T17:22:41.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7cc00640 1 -- 172.21.15.28:0/631253504 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7003ca20 con 0x5624a239be90 2024-09-13T17:22:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.791+0000 7f5e7cc00640 1 -- 172.21.15.28:0/631253504 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7003cdd0 con 0x5624a239be90 2024-09-13T17:22:41.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 -- 172.21.15.28:0/631253504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 msgr2=0x5624a23ebba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 secure :-1 s=READY pgs=2471 cs=0 l=1 rev1=1 crypto rx=0x7f5e70035040 tx=0x7f5e7002c000 comp rx=0 tx=0).stop 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 -- 172.21.15.28:0/631253504 shutdown_connections 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23c5b80 0x5624a23ec0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239be90 0x5624a23ebba0 unknown :-1 s=CLOSED pgs=2471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 --2- 172.21.15.28:0/631253504 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a239e6d0 0x5624a239eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 -- 172.21.15.28:0/631253504 >> 172.21.15.28:0/631253504 conn(0x5624a239ae00 msgr2=0x5624a23e32b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 -- 172.21.15.28:0/631253504 shutdown_connections 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.792+0000 7f5e9186d740 1 -- 172.21.15.28:0/631253504 wait complete. 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.793+0000 7f5e9186d740 1 -- start start 2024-09-13T17:22:41.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.793+0000 7f5e9186d740 1 Processor -- start 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23c4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23e8770 0x5624a23ebb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a22a2260 con 0x5624a239e6d0 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a22a1f00 con 0x5624a23e8770 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624a2067f20 con 0x5624a23c5b80 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e9186d740 10 client.0 _ll_register_callbacks cb 0x5624a22fa1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.794+0000 7f5e87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52978/0 (socket says 172.21.15.28:52978) 2024-09-13T17:22:41.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 -- 172.21.15.28:0/788764821 learned_addr learned my addr 172.21.15.28:0/788764821 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e7f400640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23e8770 0x5624a23ebb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 -- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 msgr2=0x5624a23c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 -- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23e8770 msgr2=0x5624a23ebb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23e8770 0x5624a23ebb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 -- 172.21.15.28:0/788764821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e880ccc60 con 0x5624a239e6d0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e87e00640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f5e78001eb0 tx=0x7f5e78057410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f9df844d9b771ce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e75600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7806d020 con 0x5624a239e6d0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.795+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e780567f0 con 0x5624a239e6d0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5624a232acd0 con 0x5624a239e6d0 2024-09-13T17:22:41.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e78055500 con 0x5624a239e6d0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 10 client.11622 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x5624a22ff1b0 con 0x5624a239e6d0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5624a232acd0 con 0x5624a239e6d0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e7805f1b0 con 0x5624a239e6d0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 10 client.11622 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 10 client.11622 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 20 client.11622 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a23eb600 0x5624a23c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 4 client.11622 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.796+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624a1ff43c0 con 0x5624a23eb600 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.797+0000 7f5e7f400640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a23eb600 0x5624a23c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.797+0000 7f5e7f400640 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a23eb600 0x5624a23c70d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5bbda0628e434d server_cookie=87790a5699499702 in_seq=0 out_seq=0 2024-09-13T17:22:41.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.797+0000 7f5e75600640 10 client.11622 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.798+0000 7f5e75600640 1 -- 172.21.15.28:0/788764821 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5624a1ff43c0 con 0x5624a23eb600 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.798+0000 7f5e75600640 10 client.11622 handle_command_reply: tid=0 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.798+0000 7f5e9186d740 1 client.11622 shutdown 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.798+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 msgr2=0x5624a23e9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f5e78001eb0 tx=0x7f5e78057410 comp rx=0 tx=0).stop 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 shutdown_connections 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a23eb600 0x5624a23c70d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a23e8770 0x5624a23ebb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 --2- 172.21.15.28:0/788764821 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a239e6d0 0x5624a23e9ff0 unknown :-1 s=CLOSED pgs=2472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 >> 172.21.15.28:0/788764821 conn(0x5624a239be90 msgr2=0x5624a23e6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e75600640 0 client.11622 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 shutdown_connections 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 1 -- 172.21.15.28:0/788764821 wait complete. 2024-09-13T17:22:41.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.799+0000 7f5e9186d740 20 client.11622 trim_cache size 0 max 16384 2024-09-13T17:22:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 Processor -- start 2024-09-13T17:22:41.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 -- start start 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a2405560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a2445870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a2405aa0 con 0x5624a23e8770 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a23c8bd0 con 0x5624a239e6d0 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.815+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624a23e8340 con 0x5624a23c5b80 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52982/0 (socket says 172.21.15.28:52982) 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 -- 172.21.15.28:0/1948551094 learned_addr learned my addr 172.21.15.28:0/1948551094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e87e00640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a2405560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e7f400640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a2445870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 -- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 msgr2=0x5624a2445870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a2445870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 -- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 msgr2=0x5624a2405560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a2405560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 -- 172.21.15.28:0/1948551094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a2334170 con 0x5624a23e8770 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e7f400640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a2445870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.816+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5e80005880 tx=0x7f5e80015780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f2f8a26b7857e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e7d600640 1 -- 172.21.15.28:0/1948551094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80015ed0 con 0x5624a23e8770 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e7d600640 1 -- 172.21.15.28:0/1948551094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e80016630 con 0x5624a23e8770 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e7d600640 1 -- 172.21.15.28:0/1948551094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e800169e0 con 0x5624a23e8770 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 -- 172.21.15.28:0/1948551094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 msgr2=0x5624a23e1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5e80005880 tx=0x7f5e80015780 comp rx=0 tx=0).stop 2024-09-13T17:22:41.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 -- 172.21.15.28:0/1948551094 shutdown_connections 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23e1560 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a2445870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 --2- 172.21.15.28:0/1948551094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a2405560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 -- 172.21.15.28:0/1948551094 >> 172.21.15.28:0/1948551094 conn(0x5624a239be90 msgr2=0x5624a23ea550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 -- 172.21.15.28:0/1948551094 shutdown_connections 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.817+0000 7f5e9186d740 1 -- 172.21.15.28:0/1948551094 wait complete. 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.818+0000 7f5e9186d740 1 -- start start 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.818+0000 7f5e9186d740 1 Processor -- start 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a239d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23ea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a239c6f0 con 0x5624a23e8770 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a239c890 con 0x5624a239e6d0 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5624a23e7580 con 0x5624a23c5b80 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e9186d740 10 client.0 _ll_register_callbacks cb 0x7f5e70035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e8ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52988/0 (socket says 172.21.15.28:52988) 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e8ca00640 1 -- 172.21.15.28:0/739813412 learned_addr learned my addr 172.21.15.28:0/739813412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e7f400640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23ea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.819+0000 7f5e87e00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a239d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 -- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 msgr2=0x5624a23ea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23ea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 -- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 msgr2=0x5624a239d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a239d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 -- 172.21.15.28:0/739813412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e80005560 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e8ca00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f5e80015c80 tx=0x7f5e80003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c76263f4fb9aa0d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e75600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e80006d50 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e80028630 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.820+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5e7003ca20 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e800287f0 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 10 client.11628 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x5624a22a1470 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7003ca20 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5e80026040 con 0x5624a23e8770 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 10 client.11628 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 10 client.11628 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:22:41.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 20 client.11628 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a2451290 0x5624a2451670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 4 client.11628 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5624a20056c0 con 0x5624a2451290 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.821+0000 7f5e87e00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a2451290 0x5624a2451670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e87e00640 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a2451290 0x5624a2451670 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7fcf4ce6067dd3b server_cookie=75416c07df3312d5 in_seq=0 out_seq=0 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e75600640 10 client.11628 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e75600640 1 -- 172.21.15.28:0/739813412 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5624a20056c0 con 0x5624a2451290 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e75600640 10 client.11628 handle_command_reply: tid=0 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 client.11628 shutdown 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 msgr2=0x5624a23ea990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f5e80015c80 tx=0x7f5e80003a70 comp rx=0 tx=0).stop 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 shutdown_connections 2024-09-13T17:22:41.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5624a2451290 0x5624a2451670 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5624a23e8770 0x5624a23ea990 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5624a23c5b80 0x5624a23ea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 --2- 172.21.15.28:0/739813412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5624a239e6d0 0x5624a239d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e75600640 0 client.11628 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 >> 172.21.15.28:0/739813412 conn(0x5624a239be90 msgr2=0x5624a2394300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.822+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 shutdown_connections 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e9186d740 1 -- 172.21.15.28:0/739813412 wait complete. 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e9186d740 20 client.11628 trim_cache size 0 max 16384 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:41.828 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 11478, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.1027478650000004, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 172, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 205, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 205, 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.225659604000001 2024-09-13T17:22:41.829 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 msgr2=0x7f5e6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 0x7f5e6406d0c0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f5e70034000 tx=0x7f5e7002e000 comp rx=0 tx=0).stop 2024-09-13T17:22:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 msgr2=0x7f5e880ad1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f5e7800c4d0 tx=0x7f5e7800c990 comp rx=0 tx=0).stop 2024-09-13T17:22:41.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 shutdown_connections 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5e8815ab60 0x7f5e880c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5e8815a780 0x7f5e880c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 --2- 172.21.15.28:0/3056541795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e88071490 0x7f5e880ad1b0 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 >> 172.21.15.28:0/3056541795 conn(0x7f5e880b0270 msgr2=0x7f5e881590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 shutdown_connections 2024-09-13T17:22:41.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:41.823+0000 7f5e8ea00640 1 -- 172.21.15.28:0/3056541795 wait complete. 2024-09-13T17:22:41.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:22:42.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.035+0000 7f3e86000640 1 Processor -- start 2024-09-13T17:22:42.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.036+0000 7f3e86000640 1 -- start start 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e86000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e8006b360 0x7f3e800b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e86000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800b0d80 con 0x7f3e800b0980 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e86000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e8006b900 con 0x7f3e80154a90 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e86000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8006bbe0 con 0x7f3e8006b360 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e77e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3e8006b360 0x7f3e800b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e85600640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3e800b0980 0x7f3e800af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:40298/0 (socket says 172.21.15.28:40298) 2024-09-13T17:22:42.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e85600640 1 -- 172.21.15.28:0/2808555354 learned_addr learned my addr 172.21.15.28:0/2808555354 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.037+0000 7f3e7fe00640 1 -- 172.21.15.28:0/2808555354 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374057312 0 0) 0x7f3e800b0d80 con 0x7f3e800b0980 2024-09-13T17:22:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.038+0000 7f3e77e00640 1 -- 172.21.15.28:0/2808555354 >> v1:172.21.15.189:6789/0 conn(0x7f3e80154a90 legacy=0x7f3e8015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.038+0000 7f3e77e00640 1 -- 172.21.15.28:0/2808555354 >> v1:172.21.15.28:6789/0 conn(0x7f3e800b0980 legacy=0x7f3e800af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.038+0000 7f3e77e00640 1 -- 172.21.15.28:0/2808555354 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8006bec0 con 0x7f3e8006b360 2024-09-13T17:22:42.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e77e00640 1 --2- 172.21.15.28:0/2808555354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e8006b360 0x7f3e800b0440 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f3e680097a0 tx=0x7f3e6802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25de78e7ca2e138f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:42.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e7fe00640 1 -- 172.21.15.28:0/2808555354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68032440 con 0x7f3e8006b360 2024-09-13T17:22:42.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e7fe00640 1 -- 172.21.15.28:0/2808555354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e680325a0 con 0x7f3e8006b360 2024-09-13T17:22:42.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e7fe00640 1 -- 172.21.15.28:0/2808555354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68032930 con 0x7f3e8006b360 2024-09-13T17:22:42.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 -- 172.21.15.28:0/2808555354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e8006b360 msgr2=0x7f3e800b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 --2- 172.21.15.28:0/2808555354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e8006b360 0x7f3e800b0440 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7f3e680097a0 tx=0x7f3e6802a230 comp rx=0 tx=0).stop 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 -- 172.21.15.28:0/2808555354 shutdown_connections 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 --2- 172.21.15.28:0/2808555354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e8006b360 0x7f3e800b0440 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 -- 172.21.15.28:0/2808555354 >> 172.21.15.28:0/2808555354 conn(0x7f3e80072e40 msgr2=0x7f3e80073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.039+0000 7f3e86000640 1 -- 172.21.15.28:0/2808555354 shutdown_connections 2024-09-13T17:22:42.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.040+0000 7f3e86000640 1 -- 172.21.15.28:0/2808555354 wait complete. 2024-09-13T17:22:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.041+0000 7f3e86000640 1 Processor -- start 2024-09-13T17:22:42.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.041+0000 7f3e86000640 1 -- start start 2024-09-13T17:22:42.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e8006b360 0x7f3e8015da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 0x7f3e801571b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e801560c0 con 0x7f3e800b0980 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e80156320 con 0x7f3e8006b360 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e86000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8015b890 con 0x7f3e80154a90 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e77e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53000/0 (socket says 172.21.15.28:53000) 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 -- 172.21.15.28:0/1777420195 learned_addr learned my addr 172.21.15.28:0/1777420195 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.042+0000 7f3e84c00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e8006b360 0x7f3e8015da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e85600640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 0x7f3e801571b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 -- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 msgr2=0x7f3e801571b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 0x7f3e801571b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 -- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e8006b360 msgr2=0x7f3e8015da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e8006b360 0x7f3e8015da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 -- 172.21.15.28:0/1777420195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e68009400 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e85600640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 0x7f3e801571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.043+0000 7f3e77e00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f3e68000c00 tx=0x7f3e680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.044+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e68030b80 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.044+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e68030ce0 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.044+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6803d680 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.044+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e8015bb70 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.044+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e80156600 con 0x7f3e800b0980 2024-09-13T17:22:42.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.045+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e800b0440 con 0x7f3e800b0980 2024-09-13T17:22:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.046+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3e68039040 con 0x7f3e800b0980 2024-09-13T17:22:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.046+0000 7f3e7d600640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 0x7f3e5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:42.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.046+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3e680a9f10 con 0x7f3e800b0980 2024-09-13T17:22:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.047+0000 7f3e84c00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 0x7f3e5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:42.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.048+0000 7f3e84c00640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 0x7f3e5c06d0c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000f60 tx=0x7f3e70009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:42.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.051+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3e68075050 con 0x7f3e800b0980 2024-09-13T17:22:42.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.268+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3e8015a6e0 con 0x7f3e800b0980 2024-09-13T17:22:42.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.269+0000 7f3e7d600640 1 -- 172.21.15.28:0/1777420195 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v172) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3e68041070 con 0x7f3e800b0980 2024-09-13T17:22:42.270 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 msgr2=0x7f3e5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 0x7f3e5c06d0c0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f3e70000f60 tx=0x7f3e70009040 comp rx=0 tx=0).stop 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 msgr2=0x7f3e80155b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 secure :-1 s=READY pgs=2477 cs=0 l=1 rev1=1 crypto rx=0x7f3e68000c00 tx=0x7f3e680046c0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 shutdown_connections 2024-09-13T17:22:42.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3e5c06ac00 0x7f3e5c06d0c0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3e80154a90 0x7f3e801571b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3e800b0980 0x7f3e80155b80 unknown :-1 s=CLOSED pgs=2477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 --2- 172.21.15.28:0/1777420195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e8006b360 0x7f3e8015da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 >> 172.21.15.28:0/1777420195 conn(0x7f3e80072e40 msgr2=0x7f3e800ccc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 shutdown_connections 2024-09-13T17:22:42.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:42.271+0000 7f3e86000640 1 -- 172.21.15.28:0/1777420195 wait complete. 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:22:42.288 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:22:42.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.289 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:22:42.324 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:22:42.324 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:22:42.325 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41819sec preferred_lft 41819sec 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.326 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:22:42.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.327 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:22:42.418 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:22:42.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.422 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:22:42.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:22:42.459 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:22:42.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.604 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:22:42.639 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:22:42.639 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:22:42.639 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:22:42.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:22:42.736 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:22:42.736 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:22:42.771 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:22:42.772 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:22:42.772 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:22:42.847 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:22:42.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:42.848 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:22:42.976 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:22:42.976 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:22:43.002 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:22:43.002 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:22:43.003 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:22:43.003 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:22:43.003 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:22:43.005 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:22:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:22:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:22:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:22:43.005 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:22:43.008 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:22:43.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:43.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:22:43.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:43.065 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:22:43.110 INFO:teuthology.orchestra.run.smithi028.stdout:client11640 2024-09-13T17:22:43.110 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:43.110 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:43.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:43.185 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-09-13T17:22:43.267 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:22:43.267 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:22:43.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:22:43.343 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=subdir/sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T17:22:43.483 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records in 2024-09-13T17:22:43.483 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records out 2024-09-13T17:22:43.483 INFO:teuthology.orchestra.run.smithi028.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0631935 s, 99.6 MB/s 2024-09-13T17:22:43.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:22:43.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:22:43.585 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T17:22:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T17:22:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:22:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T17:22:43.585 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T17:22:43.586 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:22:43.587 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T17:22:43.588 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T17:22:43.589 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.590 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-13T17:22:43.591 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:22:43.592 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T17:22:43.593 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.594 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.595 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: mds cache max size = 1000 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T17:22:43.596 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = True 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: mds log max segments = 1 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-13T17:22:43.597 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-13T17:22:43.598 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:22:43.599 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:22:43.600 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-13T17:22:43.601 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-13T17:22:43.602 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore allocator = bitmap 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 1/20 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 1/20 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 4/10 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:22:43.603 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-13T17:22:43.604 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.605 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-09-13T17:22:43.606 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-13T17:22:43.607 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds cache max size = 1000 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = True 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds log max segments = 1 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-09-13T17:22:43.608 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = False 2024-09-13T17:22:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:22:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.609 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.617 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:22:43.621 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T17:22:43.716 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T17:22:43.717 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.718 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T17:22:43.719 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T17:22:43.720 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T17:22:43.721 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-13T17:22:43.722 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.723 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:22:43.724 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:22:43.725 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T17:22:43.726 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:22:43.727 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds cache max size = 1000 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = False 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds log max segments = 1 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-09-13T17:22:43.728 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-13T17:22:43.729 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:22:43.730 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-13T17:22:43.731 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:22:43.732 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-13T17:22:43.733 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-13T17:22:43.734 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore allocator = bitmap 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 1/20 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 1/20 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 4/10 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:22:43.735 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:22:43.736 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:22:43.737 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-13T17:22:43.738 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-13T17:22:43.739 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds cache max size = 1000 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = False 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds log max segments = 1 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = False 2024-09-13T17:22:43.740 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:22:43.741 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:22:43.741 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:22:43.745 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:22:43.745 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-13T17:22:43.746 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-09-13T17:22:43.747 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:22:43.848 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mode": 33188, 2024-09-13T17:22:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ino": 1099511627777, 2024-09-13T17:22:43.848 INFO:teuthology.orchestra.run.smithi028.stdout: "st_dev": 42, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_nlink": 1, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_uid": 1000, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_gid": 1333, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_size": 6291456, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_atime": 1726248163.4181972, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mtime": 1726248163.4541962, 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ctime": 1726248163.4541962 2024-09-13T17:22:43.849 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:44.033 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:22:44.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.228+0000 7f3571000640 1 Processor -- start 2024-09-13T17:22:44.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.228+0000 7f3571000640 1 -- start start 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.229+0000 7f3571000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c06f970 0x7f356c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.229+0000 7f3571000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f356c06c2d0 0x7f356c06c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.229+0000 7f3571000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c159d90 con 0x7f356c1592a0 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.229+0000 7f3571000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c15b0b0 con 0x7f356c06c2d0 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.229+0000 7f3571000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c06cca0 con 0x7f356c06f970 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f3569e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f356c1592a0 0x7f356c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51920/0 (socket says 172.21.15.28:51920) 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f3569e00640 1 -- 172.21.15.28:0/2316449227 learned_addr learned my addr 172.21.15.28:0/2316449227 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f356a800640 1 --2- 172.21.15.28:0/2316449227 >> v2:172.21.15.189:3300/0 conn(0x7f356c06c2d0 0x7f356c06c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f3569400640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c06f970 0x7f356c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f3568a00640 1 -- 172.21.15.28:0/2316449227 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215636713 0 0) 0x7f356c159d90 con 0x7f356c1592a0 2024-09-13T17:22:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f3568a00640 1 -- 172.21.15.28:0/2316449227 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550003520 con 0x7f356c1592a0 2024-09-13T17:22:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f356a800640 1 -- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c06f970 msgr2=0x7f356c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f356a800640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c06f970 0x7f356c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.230+0000 7f356a800640 1 -- 172.21.15.28:0/2316449227 >> v1:172.21.15.28:6789/0 conn(0x7f356c1592a0 legacy=0x7f356c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.231+0000 7f356a800640 1 -- 172.21.15.28:0/2316449227 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0b2040 con 0x7f356c06c2d0 2024-09-13T17:22:44.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f356a800640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06c2d0 0x7f356c06c760 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f3564009d90 tx=0x7f356402c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f570db9db30ea09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3568a00640 1 -- 172.21.15.28:0/2316449227 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564032770 con 0x7f356c06c2d0 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3568a00640 1 -- 172.21.15.28:0/2316449227 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35640328d0 con 0x7f356c06c2d0 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3568a00640 1 -- 172.21.15.28:0/2316449227 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564032c60 con 0x7f356c06c2d0 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 -- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06c2d0 msgr2=0x7f356c06c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06c2d0 0x7f356c06c760 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f3564009d90 tx=0x7f356402c870 comp rx=0 tx=0).stop 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 -- 172.21.15.28:0/2316449227 shutdown_connections 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06c2d0 0x7f356c06c760 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 --2- 172.21.15.28:0/2316449227 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c06f970 0x7f356c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.232+0000 7f3571000640 1 -- 172.21.15.28:0/2316449227 >> 172.21.15.28:0/2316449227 conn(0x7f356c069e80 msgr2=0x7f356c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.233+0000 7f3571000640 1 -- 172.21.15.28:0/2316449227 shutdown_connections 2024-09-13T17:22:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.233+0000 7f3571000640 1 -- 172.21.15.28:0/2316449227 wait complete. 2024-09-13T17:22:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.234+0000 7f3571000640 1 Processor -- start 2024-09-13T17:22:44.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.234+0000 7f3571000640 1 -- start start 2024-09-13T17:22:44.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 0x7f356c157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 0x7f356c158020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c144680 con 0x7f356c06c2d0 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c145b20 con 0x7f356c06f970 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3571000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c143a70 con 0x7f356c1592a0 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.235+0000 7f3569e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38770/0 (socket says 172.21.15.28:38770) 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 -- 172.21.15.28:0/2157276226 learned_addr learned my addr 172.21.15.28:0/2157276226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f356a800640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 0x7f356c158020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569400640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 0x7f356c157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 -- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 msgr2=0x7f356c158020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 0x7f356c158020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 -- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 msgr2=0x7f356c157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 0x7f356c157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 -- 172.21.15.28:0/2157276226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3564009a70 con 0x7f356c06c2d0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569400640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 0x7f356c157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f356a800640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 0x7f356c158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3569e00640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c430 tx=0x7f355c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c005ce0 con 0x7f356c06c2d0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c005e40 con 0x7f356c06c2d0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c0072d0 con 0x7f356c06c2d0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.236+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c143d50 con 0x7f356c06c2d0 2024-09-13T17:22:44.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.237+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c144310 con 0x7f356c06c2d0 2024-09-13T17:22:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.238+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c1447f0 con 0x7f356c06c2d0 2024-09-13T17:22:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.239+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f355c0074f0 con 0x7f356c06c2d0 2024-09-13T17:22:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.239+0000 7f3562000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 0x7f354c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.240+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f355c0888e0 con 0x7f356c06c2d0 2024-09-13T17:22:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.240+0000 7f3569400640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 0x7f354c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.241+0000 7f3569400640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 0x7f354c06d0c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f356c0cacb0 tx=0x7f3558008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.244+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f355c053ad0 con 0x7f356c06c2d0 2024-09-13T17:22:44.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.459+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f356c0b3740 con 0x7f356c06c2d0 2024-09-13T17:22:44.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.460+0000 7f3562000640 1 -- 172.21.15.28:0/2157276226 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7f355c017020 con 0x7f356c06c2d0 2024-09-13T17:22:44.478 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:22:44.478 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:22:44.479 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:22:44.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 msgr2=0x7f354c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 0x7f354c06d0c0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f356c0cacb0 tx=0x7f3558008040 comp rx=0 tx=0).stop 2024-09-13T17:22:44.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 msgr2=0x7f356c1455e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 secure :-1 s=READY pgs=2480 cs=0 l=1 rev1=1 crypto rx=0x7f355c00c430 tx=0x7f355c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 shutdown_connections 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f354c06ac00 0x7f354c06d0c0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f356c1592a0 0x7f356c158020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f356c06f970 0x7f356c157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 --2- 172.21.15.28:0/2157276226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f356c06c2d0 0x7f356c1455e0 unknown :-1 s=CLOSED pgs=2480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 >> 172.21.15.28:0/2157276226 conn(0x7f356c069e80 msgr2=0x7f356c0ae140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 shutdown_connections 2024-09-13T17:22:44.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.462+0000 7f3571000640 1 -- 172.21.15.28:0/2157276226 wait complete. 2024-09-13T17:22:44.481 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T17:22:44.661 INFO:teuthology.orchestra.run.smithi028.stdout:{"message":"","return_code":0} 2024-09-13T17:22:44.661 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T17:22:44.661 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent '' 2024-09-13T17:22:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.738+0000 7f9dd1743f40 1 Processor -- start 2024-09-13T17:22:44.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.738+0000 7f9dd1743f40 1 -- start start 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.738+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edc6f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.738+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd16c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.738+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edc6e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8edbec400 con 0x55d8edcfdc40 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8edb87fe0 con 0x55d8edc8c100 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d8edb87c80 con 0x55d8edc458a0 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dcf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edc6e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dcf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edc6e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38794/0 (socket says 172.21.15.28:38794) 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dcea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edc6f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd16c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dcf400640 1 -- 172.21.15.28:0/780859893 learned_addr learned my addr 172.21.15.28:0/780859893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 -- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 msgr2=0x55d8edc6f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edc6f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 -- 172.21.15.28:0/780859893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 msgr2=0x55d8edc6e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edc6e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.739+0000 7f9dce000640 1 -- 172.21.15.28:0/780859893 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dc800a1c0 con 0x55d8edc8c100 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dce000640 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd16c60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009820 tx=0x7f9dbc005a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff22ff2b28b4dceb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dccc00640 1 -- 172.21.15.28:0/780859893 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc00f040 con 0x55d8edc8c100 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dccc00640 1 -- 172.21.15.28:0/780859893 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0088d0 con 0x55d8edc8c100 2024-09-13T17:22:44.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dccc00640 1 -- 172.21.15.28:0/780859893 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc008c60 con 0x55d8edc8c100 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 -- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 msgr2=0x55d8edd16c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd16c60 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc009820 tx=0x7f9dbc005a00 comp rx=0 tx=0).stop 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 -- 172.21.15.28:0/780859893 shutdown_connections 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edc6e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd16c60 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/780859893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edc6f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 -- 172.21.15.28:0/780859893 >> 172.21.15.28:0/780859893 conn(0x55d8edc70270 msgr2=0x55d8edc72b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 -- 172.21.15.28:0/780859893 shutdown_connections 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.740+0000 7f9dd1743f40 1 -- 172.21.15.28:0/780859893 wait complete. 2024-09-13T17:22:44.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.741+0000 7f9dd1743f40 1 Processor -- start 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.741+0000 7f9dd1743f40 1 -- start start 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edd14750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd11800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edd105d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8edb7e8f0 con 0x55d8edcfdc40 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8edda7390 con 0x55d8edc8c100 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dd1743f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d8edda7c30 con 0x55d8edc458a0 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.742+0000 7f9dcf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edd105d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd11800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dcea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edd14750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd11800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46426/0 (socket says 172.21.15.28:46426) 2024-09-13T17:22:44.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 -- 172.21.15.28:0/1562907745 learned_addr learned my addr 172.21.15.28:0/1562907745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 -- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 msgr2=0x55d8edd14750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55d8edc458a0 0x55d8edd14750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 -- 172.21.15.28:0/1562907745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 msgr2=0x55d8edd105d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edd105d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 -- 172.21.15.28:0/1562907745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc009450 con 0x55d8edc8c100 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dcf400640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d8edcfdc40 0x55d8edd105d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.743+0000 7f9dce000640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd11800 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc005fd0 tx=0x7f9dbc003020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.744+0000 7f9dc7400640 1 -- 172.21.15.28:0/1562907745 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc00f070 con 0x55d8edc8c100 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.744+0000 7f9dc7400640 1 -- 172.21.15.28:0/1562907745 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dbc013830 con 0x55d8edc8c100 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.744+0000 7f9dc7400640 1 -- 172.21.15.28:0/1562907745 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc013be0 con 0x55d8edc8c100 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.744+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d8edb96b20 con 0x55d8edc8c100 2024-09-13T17:22:44.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.744+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8edb89f60 con 0x55d8edc8c100 2024-09-13T17:22:44.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.746+0000 7f9dc7400640 1 -- 172.21.15.28:0/1562907745 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9dbc023070 con 0x55d8edc8c100 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dc7400640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9db00695e0 0x7f9db006baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dc7400640 1 -- 172.21.15.28:0/1562907745 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9dbc0899c0 con 0x55d8edc8c100 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dcea00640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9db00695e0 0x7f9db006baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55d8edcf9a40 0x55d8edc6e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- osd_op(unknown.0.0:1 29.7 29:e2e07930:::10000000001.00000000:head [setxattr parent in=6b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205) v8 -- 0x55d8edda80a0 con 0x55d8edcf9a40 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.747+0000 7f9dcf400640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55d8edcf9a40 0x55d8edc6e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.748+0000 7f9dcea00640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9db00695e0 0x7f9db006baa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x55d8edcfc2c0 tx=0x7f9dc0001770 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.748+0000 7f9dcf400640 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55d8edcf9a40 0x55d8edc6e760 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.749+0000 7f9dcf400640 1 -- 172.21.15.28:0/1562907745 <== osd.6 v2:172.21.15.189:6816/2946605912 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr] v205'4 uv4 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f9dc8057070 con 0x55d8edcf9a40 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.749+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55d8edcf9a40 msgr2=0x55d8edc6e760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.749+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55d8edcf9a40 0x55d8edc6e760 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9db00695e0 msgr2=0x7f9db006baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9db00695e0 0x7f9db006baa0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x55d8edcfc2c0 tx=0x7f9dc0001770 comp rx=0 tx=0).stop 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 msgr2=0x55d8edd11800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 --2- 172.21.15.28:0/1562907745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d8edc8c100 0x55d8edd11800 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc005fd0 tx=0x7f9dbc003020 comp rx=0 tx=0).stop 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dcf400640 1 -- 172.21.15.28:0/1562907745 reap_dead start 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 shutdown_connections 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 >> 172.21.15.28:0/1562907745 conn(0x55d8edc70270 msgr2=0x55d8edc6f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 shutdown_connections 2024-09-13T17:22:44.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.750+0000 7f9dd1743f40 1 -- 172.21.15.28:0/1562907745 wait complete. 2024-09-13T17:22:44.755 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.14:0 scrub start / recursive,repair 2024-09-13T17:22:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.913+0000 7fdb92e00640 1 Processor -- start 2024-09-13T17:22:44.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.913+0000 7fdb92e00640 1 -- start start 2024-09-13T17:22:44.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb92e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb92e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb8c159970 0x7fdb8c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb92e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb8c15a4c0 con 0x7fdb8c0b2630 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb92e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb8c0b2ae0 con 0x7fdb8c159970 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb92e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c071580 con 0x7fdb8c159590 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.914+0000 7fdb8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb8c159970 0x7fdb8c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fdb8c159970 0x7fdb8c0b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38822/0 (socket says 172.21.15.28:38822) 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb90e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdb8c0b2630 0x7fdb8c06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:51922/0 (socket says 172.21.15.28:51922) 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 -- 172.21.15.28:0/3450814703 learned_addr learned my addr 172.21.15.28:0/3450814703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8be00640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 -- 172.21.15.28:0/3450814703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 msgr2=0x7fdb8c070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 -- 172.21.15.28:0/3450814703 >> v1:172.21.15.28:6789/0 conn(0x7fdb8c0b2630 legacy=0x7fdb8c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8b400640 1 -- 172.21.15.28:0/3450814703 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb8c159e70 con 0x7fdb8c159970 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.915+0000 7fdb8aa00640 1 -- 172.21.15.28:0/3450814703 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123142745 0 0) 0x7fdb8c15a4c0 con 0x7fdb8c0b2630 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.916+0000 7fdb8b400640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0b20f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009870 tx=0x7fdb7802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab20f05a528644bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.916+0000 7fdb8aa00640 1 -- 172.21.15.28:0/3450814703 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78032440 con 0x7fdb8c159970 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.916+0000 7fdb8aa00640 1 -- 172.21.15.28:0/3450814703 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb780325a0 con 0x7fdb8c159970 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.916+0000 7fdb8aa00640 1 -- 172.21.15.28:0/3450814703 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb78032930 con 0x7fdb8c159970 2024-09-13T17:22:44.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 -- 172.21.15.28:0/3450814703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 msgr2=0x7fdb8c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0b20f0 secure :-1 s=READY pgs=2483 cs=0 l=1 rev1=1 crypto rx=0x7fdb78009870 tx=0x7fdb7802a250 comp rx=0 tx=0).stop 2024-09-13T17:22:44.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 -- 172.21.15.28:0/3450814703 shutdown_connections 2024-09-13T17:22:44.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0b20f0 unknown :-1 s=CLOSED pgs=2483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 --2- 172.21.15.28:0/3450814703 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 -- 172.21.15.28:0/3450814703 >> 172.21.15.28:0/3450814703 conn(0x7fdb8c069e80 msgr2=0x7fdb8c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 -- 172.21.15.28:0/3450814703 shutdown_connections 2024-09-13T17:22:44.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.917+0000 7fdb92e00640 1 -- 172.21.15.28:0/3450814703 wait complete. 2024-09-13T17:22:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.919+0000 7fdb92e00640 1 Processor -- start 2024-09-13T17:22:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.919+0000 7fdb92e00640 1 -- start start 2024-09-13T17:22:44.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.919+0000 7fdb92e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb8c0b2630 0x7fdb8c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb92e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c0cb540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb92e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb92e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0cd420 con 0x7fdb8c159970 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb92e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0cbe40 con 0x7fdb8c0b2630 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb92e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8c0cc120 con 0x7fdb8c159590 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38828/0 (socket says 172.21.15.28:38828) 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 -- 172.21.15.28:0/1016300089 learned_addr learned my addr 172.21.15.28:0/1016300089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb8b400640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c0cb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb8be00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb8c0b2630 0x7fdb8c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 -- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 msgr2=0x7fdb8c0cb540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c0cb540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 -- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb8c0b2630 msgr2=0x7fdb8c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb8c0b2630 0x7fdb8c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.920+0000 7fdb90e00640 1 -- 172.21.15.28:0/1016300089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb780094d0 con 0x7fdb8c159970 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb90e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400c7a0 tx=0x7fdb8400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb83e00640 1 -- 172.21.15.28:0/1016300089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8400a640 con 0x7fdb8c159970 2024-09-13T17:22:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb83e00640 1 -- 172.21.15.28:0/1016300089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb8400a7a0 con 0x7fdb8c159970 2024-09-13T17:22:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb83e00640 1 -- 172.21.15.28:0/1016300089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8400ab90 con 0x7fdb8c159970 2024-09-13T17:22:44.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8c0cc400 con 0x7fdb8c159970 2024-09-13T17:22:44.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.921+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8c0cc9c0 con 0x7fdb8c159970 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.923+0000 7fdb83e00640 1 -- 172.21.15.28:0/1016300089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdb8400acf0 con 0x7fdb8c159970 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.924+0000 7fdb83e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 0x7fdb6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.924+0000 7fdb83e00640 1 -- 172.21.15.28:0/1016300089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb84088970 con 0x7fdb8c159970 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.925+0000 7fdb8be00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 0x7fdb6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.925+0000 7fdb95c76740 1 Processor -- start 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.925+0000 7fdb95c76740 1 -- start start 2024-09-13T17:22:44.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.925+0000 7fdb8be00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 0x7fdb6c06d0c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009b40 tx=0x7fdb7c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106ad70 0x565371067ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x565371029870 con 0x56537106d5b0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x565370fd6b90 con 0x56537106ad70 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x565370fd4320 con 0x5653710b0ae0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38834/0 (socket says 172.21.15.28:38834) 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb8be00640 1 -- 172.21.15.28:0/991919483 learned_addr learned my addr 172.21.15.28:0/991919483 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb8b400640 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106ad70 0x565371067ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.926+0000 7fdb90e00640 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 -- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 msgr2=0x5653710b0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 -- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106ad70 msgr2=0x565371067ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106ad70 0x565371067ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 -- 172.21.15.28:0/991919483 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565370f71800 con 0x56537106d5b0 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb8be00640 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011ee0 tx=0x7fdb7c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33a71c5371fb8a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb80c00640 1 -- 172.21.15.28:0/991919483 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c01c040 con 0x56537106d5b0 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb80c00640 1 -- 172.21.15.28:0/991919483 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c003820 con 0x56537106d5b0 2024-09-13T17:22:44.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.927+0000 7fdb80c00640 1 -- 172.21.15.28:0/991919483 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c003bb0 con 0x56537106d5b0 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 -- 172.21.15.28:0/991919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 msgr2=0x56537106d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 secure :-1 s=READY pgs=2485 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011ee0 tx=0x7fdb7c014000 comp rx=0 tx=0).stop 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 -- 172.21.15.28:0/991919483 shutdown_connections 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106ad70 0x565371067ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 --2- 172.21.15.28:0/991919483 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x56537106d990 unknown :-1 s=CLOSED pgs=2485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 -- 172.21.15.28:0/991919483 >> 172.21.15.28:0/991919483 conn(0x565371069d70 msgr2=0x5653710b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 -- 172.21.15.28:0/991919483 shutdown_connections 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.928+0000 7fdb95c76740 1 -- 172.21.15.28:0/991919483 wait complete. 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.929+0000 7fdb95c76740 1 -- start start 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.929+0000 7fdb95c76740 1 Processor -- start 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106d5b0 0x5653710b7f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x565370fcfc10 con 0x5653710b9e30 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x565370f8dfd0 con 0x56537106d5b0 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x565370f8dc70 con 0x5653710b0ae0 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb95c76740 10 client.0 _ll_register_callbacks cb 0x565370fc9280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb90e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb90e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38844/0 (socket says 172.21.15.28:38844) 2024-09-13T17:22:44.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb90e00640 1 -- 172.21.15.28:0/594702256 learned_addr learned my addr 172.21.15.28:0/594702256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb8b400640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.930+0000 7fdb8be00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106d5b0 0x5653710b7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 -- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 msgr2=0x5653710b98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b98f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 -- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106d5b0 msgr2=0x5653710b7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106d5b0 0x5653710b7f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 -- 172.21.15.28:0/594702256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb7c011bc0 con 0x5653710b9e30 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb90e00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fdb84086e00 tx=0x7fdb840570f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989b155ef8c6328a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb75600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb8405b320 con 0x5653710b9e30 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb8405fb60 con 0x5653710b9e30 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565370ff90f0 con 0x5653710b9e30 2024-09-13T17:22:44.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb8406f180 con 0x5653710b9e30 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb95c76740 10 client.11670 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.931+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x565370fce290 con 0x5653710b9e30 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x565370ff90f0 con 0x5653710b9e30 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdb8405f6d0 con 0x5653710b9e30 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 10 client.11670 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 10 client.11670 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 20 client.11670 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710aa4b0 0x565371095fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 4 client.11670 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565370fc43f0 con 0x5653710aa4b0 2024-09-13T17:22:44.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.932+0000 7fdb8be00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710aa4b0 0x565371095fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.933+0000 7fdb8be00640 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710aa4b0 0x565371095fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95e87469c8c649f4 server_cookie=821ac16a4cb777a3 in_seq=0 out_seq=0 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.933+0000 7fdb75600640 10 client.11670 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.933+0000 7fdb75600640 1 -- 172.21.15.28:0/594702256 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x565370fc43f0 con 0x5653710aa4b0 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.933+0000 7fdb75600640 10 client.11670 handle_command_reply: tid=0 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.933+0000 7fdb95c76740 1 client.11670 shutdown 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 msgr2=0x5653710b5b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 secure :-1 s=READY pgs=2486 cs=0 l=1 rev1=1 crypto rx=0x7fdb84086e00 tx=0x7fdb840570f0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 shutdown_connections 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710aa4b0 0x565371095fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5653710b9e30 0x5653710b5b20 unknown :-1 s=CLOSED pgs=2486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710b98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb75600640 0 client.11670 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 --2- 172.21.15.28:0/594702256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56537106d5b0 0x5653710b7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 >> 172.21.15.28:0/594702256 conn(0x56537106ad70 msgr2=0x565371068c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 shutdown_connections 2024-09-13T17:22:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 1 -- 172.21.15.28:0/594702256 wait complete. 2024-09-13T17:22:44.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.934+0000 7fdb95c76740 20 client.11670 trim_cache size 0 max 16384 2024-09-13T17:22:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.949+0000 7fdb95c76740 1 Processor -- start 2024-09-13T17:22:44.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.949+0000 7fdb95c76740 1 -- start start 2024-09-13T17:22:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653710a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x565370d370c0 con 0x56537106d5b0 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653710b71e0 con 0x5653710b9e30 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5653710b1090 con 0x5653710b0ae0 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38852/0 (socket says 172.21.15.28:38852) 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb8be00640 1 -- 172.21.15.28:0/2032314451 learned_addr learned my addr 172.21.15.28:0/2032314451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb8be00640 1 -- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 msgr2=0x5653710a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.950+0000 7fdb8be00640 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb90e00640 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653710a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb8be00640 1 -- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 msgr2=0x5653710a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb8be00640 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653710a89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb8be00640 1 -- 172.21.15.28:0/2032314451 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565370f70040 con 0x56537106d5b0 2024-09-13T17:22:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb90e00640 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653710a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:44.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb8be00640 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011690 tx=0x7fdb7c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7349f1c22af849 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb81600640 1 -- 172.21.15.28:0/2032314451 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c031040 con 0x56537106d5b0 2024-09-13T17:22:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb81600640 1 -- 172.21.15.28:0/2032314451 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c003bd0 con 0x56537106d5b0 2024-09-13T17:22:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.951+0000 7fdb81600640 1 -- 172.21.15.28:0/2032314451 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c037410 con 0x56537106d5b0 2024-09-13T17:22:44.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- 172.21.15.28:0/2032314451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 msgr2=0x5653710d4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 secure :-1 s=READY pgs=2487 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011690 tx=0x7fdb7c016000 comp rx=0 tx=0).stop 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- 172.21.15.28:0/2032314451 shutdown_connections 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653710a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x5653710a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 --2- 172.21.15.28:0/2032314451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x5653710d4950 unknown :-1 s=CLOSED pgs=2487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- 172.21.15.28:0/2032314451 >> 172.21.15.28:0/2032314451 conn(0x56537106ad70 msgr2=0x565371095c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- 172.21.15.28:0/2032314451 shutdown_connections 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- 172.21.15.28:0/2032314451 wait complete. 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.952+0000 7fdb95c76740 1 -- start start 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.953+0000 7fdb95c76740 1 Processor -- start 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.953+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.953+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x565371106c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb95c76740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653711219c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb95c76740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x565371120fe0 con 0x56537106d5b0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56537107fea0 con 0x5653710b9e30 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb95c76740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x565371095c20 con 0x5653710b0ae0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb95c76740 10 client.0 _ll_register_callbacks cb 0x7fdb78035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38866/0 (socket says 172.21.15.28:38866) 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 -- 172.21.15.28:0/29630212 learned_addr learned my addr 172.21.15.28:0/29630212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8b400640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x565371106c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb90e00640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653711219c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 -- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 msgr2=0x565371106c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x565371106c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 -- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 msgr2=0x5653711219c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653711219c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.954+0000 7fdb8be00640 1 -- 172.21.15.28:0/29630212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565370f71800 con 0x56537106d5b0 2024-09-13T17:22:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb8be00640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011690 tx=0x7fdb7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc31366eb517a961 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:44.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7c031040 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb7c003e40 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb7c003820 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb7c037620 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 10 client.11676 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x565370f95610 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7c003820 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fdb7c037780 con 0x56537106d5b0 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 10 client.11676 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 10 client.11676 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:22:44.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 20 client.11676 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.955+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710a84b0 0x5653710a8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.956+0000 7fdb95c76740 4 client.11676 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}] 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.956+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}) v1 -- 0x565370fc1d00 con 0x5653710a84b0 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.956+0000 7fdb8b400640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710a84b0 0x5653710a8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.956+0000 7fdb8b400640 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710a84b0 0x5653710a8890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39b2768a803fe204 server_cookie=ba38cacfe435dc55 in_seq=0 out_seq=0 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.956+0000 7fdb75600640 10 client.11676 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb75600640 1 -- 172.21.15.28:0/29630212 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x565370fc1d00 con 0x5653710a84b0 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb75600640 10 client.11676 handle_command_reply: tid=0 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 client.11676 shutdown 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 msgr2=0x565371096580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 secure :-1 s=READY pgs=2488 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c011690 tx=0x7fdb7c004730 comp rx=0 tx=0).stop 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 shutdown_connections 2024-09-13T17:22:44.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x5653710a84b0 0x5653710a8890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5653710b9e30 0x5653711219c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5653710b0ae0 0x565371106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 --2- 172.21.15.28:0/29630212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56537106d5b0 0x565371096580 unknown :-1 s=CLOSED pgs=2488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb75600640 0 client.11676 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 >> 172.21.15.28:0/29630212 conn(0x56537106ad70 msgr2=0x5653710b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.957+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 shutdown_connections 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb95c76740 1 -- 172.21.15.28:0/29630212 wait complete. 2024-09-13T17:22:44.962 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:22:44.958+0000 7f97f2e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v6 ap=1 DIRTYPARENT s=5242880 n(v0 rc2024-09-13T17:22:43.449196+0000 b5242880 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={11640=0-12582912@1} caps={11640=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@4},l=11640 | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x5611462cd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(29)0x10000000001:[<0x10000000000/sixmegs v6>,<0x1/subdir v8>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T17:22:44.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb95c76740 20 client.11676 trim_cache size 0 max 16384 2024-09-13T17:22:44.963 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:44.963 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:22:44.963 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "caab1373-a73e-4d6d-a8e8-0179e4417548", 2024-09-13T17:22:44.963 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:22:44.963 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 msgr2=0x7fdb6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 0x7fdb6c06d0c0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c009b40 tx=0x7fdb7c009190 comp rx=0 tx=0).stop 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 msgr2=0x7fdb8c0cec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 secure :-1 s=READY pgs=2484 cs=0 l=1 rev1=1 crypto rx=0x7fdb8400c7a0 tx=0x7fdb8400cc60 comp rx=0 tx=0).stop 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 shutdown_connections 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdb6c06ac00 0x7fdb6c06d0c0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdb8c159970 0x7fdb8c0cec50 unknown :-1 s=CLOSED pgs=2484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdb8c159590 0x7fdb8c0cb540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 --2- 172.21.15.28:0/1016300089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdb8c0b2630 0x7fdb8c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 >> 172.21.15.28:0/1016300089 conn(0x7fdb8c069e80 msgr2=0x7fdb8c1588a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 shutdown_connections 2024-09-13T17:22:44.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:44.958+0000 7fdb92e00640 1 -- 172.21.15.28:0/1016300089 wait complete. 2024-09-13T17:22:44.975 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.14:0 scrub status 2024-09-13T17:22:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.131+0000 7fda37200640 1 Processor -- start 2024-09-13T17:22:45.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.131+0000 7fda37200640 1 -- start start 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.132+0000 7fda37200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fda30159770 0x7fda3006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.132+0000 7fda37200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda3015cd80 0x7fda30070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.132+0000 7fda37200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3006f920 con 0x7fda3006f3f0 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.132+0000 7fda37200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3015b290 con 0x7fda3015cd80 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.132+0000 7fda37200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3015b570 con 0x7fda30159770 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda3015cd80 0x7fda30070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fda3015cd80 0x7fda30070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38874/0 (socket says 172.21.15.28:38874) 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 -- 172.21.15.28:0/2943287642 learned_addr learned my addr 172.21.15.28:0/2943287642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda2fe00640 1 --2- 172.21.15.28:0/2943287642 >> v2:172.21.15.189:3300/0 conn(0x7fda30159770 0x7fda3006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 -- 172.21.15.28:0/2943287642 >> v2:172.21.15.189:3300/0 conn(0x7fda30159770 msgr2=0x7fda3006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 --2- 172.21.15.28:0/2943287642 >> v2:172.21.15.189:3300/0 conn(0x7fda30159770 0x7fda3006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 -- 172.21.15.28:0/2943287642 >> v1:172.21.15.189:6789/0 conn(0x7fda3006f3f0 legacy=0x7fda30159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.133+0000 7fda35200640 1 -- 172.21.15.28:0/2943287642 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3015b850 con 0x7fda3015cd80 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.134+0000 7fda35200640 1 --2- 172.21.15.28:0/2943287642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30070260 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fda30067f80 tx=0x7fda2802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909079c03812d154 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.134+0000 7fda2ea00640 1 -- 172.21.15.28:0/2943287642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28031600 con 0x7fda3015cd80 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.134+0000 7fda2ea00640 1 -- 172.21.15.28:0/2943287642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2802d050 con 0x7fda3015cd80 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.134+0000 7fda2ea00640 1 -- 172.21.15.28:0/2943287642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28031980 con 0x7fda3015cd80 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 -- 172.21.15.28:0/2943287642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 msgr2=0x7fda30070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 --2- 172.21.15.28:0/2943287642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30070260 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7fda30067f80 tx=0x7fda2802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 -- 172.21.15.28:0/2943287642 shutdown_connections 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 --2- 172.21.15.28:0/2943287642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30070260 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 --2- 172.21.15.28:0/2943287642 >> v2:172.21.15.189:3300/0 conn(0x7fda30159770 0x7fda3006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 -- 172.21.15.28:0/2943287642 >> 172.21.15.28:0/2943287642 conn(0x7fda300b0270 msgr2=0x7fda300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 -- 172.21.15.28:0/2943287642 shutdown_connections 2024-09-13T17:22:45.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.135+0000 7fda37200640 1 -- 172.21.15.28:0/2943287642 wait complete. 2024-09-13T17:22:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.137+0000 7fda37200640 1 Processor -- start 2024-09-13T17:22:45.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.137+0000 7fda37200640 1 -- start start 2024-09-13T17:22:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 0x7fda30155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 0x7fda30156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda301478a0 con 0x7fda3015cd80 2024-09-13T17:22:45.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda30151b60 con 0x7fda30159770 2024-09-13T17:22:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda37200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda30151e40 con 0x7fda3006f3f0 2024-09-13T17:22:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38878/0 (socket says 172.21.15.28:38878) 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 -- 172.21.15.28:0/1673434470 learned_addr learned my addr 172.21.15.28:0/1673434470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 0x7fda30155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda2f400640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 0x7fda30156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 -- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 msgr2=0x7fda30155160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 0x7fda30155160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 -- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 msgr2=0x7fda30156ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 0x7fda30156ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda35200640 1 -- 172.21.15.28:0/1673434470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda28009b30 con 0x7fda3015cd80 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.138+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 0x7fda30155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda2f400640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 0x7fda30156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:45.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda35200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fda28034040 tx=0x7fda28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda27e00640 1 -- 172.21.15.28:0/1673434470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2803d860 con 0x7fda3015cd80 2024-09-13T17:22:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda27e00640 1 -- 172.21.15.28:0/1673434470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2803d9c0 con 0x7fda3015cd80 2024-09-13T17:22:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda27e00640 1 -- 172.21.15.28:0/1673434470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2803dd70 con 0x7fda3015cd80 2024-09-13T17:22:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda30152120 con 0x7fda3015cd80 2024-09-13T17:22:45.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.139+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda301526e0 con 0x7fda3015cd80 2024-09-13T17:22:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.141+0000 7fda27e00640 1 -- 172.21.15.28:0/1673434470 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fda28046400 con 0x7fda3015cd80 2024-09-13T17:22:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.142+0000 7fda27e00640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 0x7fda1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.142+0000 7fda27e00640 1 -- 172.21.15.28:0/1673434470 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda280aa5b0 con 0x7fda3015cd80 2024-09-13T17:22:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.142+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 0x7fda1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.143+0000 7fda39fd9740 1 Processor -- start 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.143+0000 7fda39fd9740 1 -- start start 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.143+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 0x7fda1006d0c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fda20009ac0 tx=0x7fda20008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x55957580d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55957580a420 0x55957585a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595757136a0 con 0x559575834110 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559575776cc0 con 0x55957580cc60 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559575774450 con 0x55957580a420 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38884/0 (socket says 172.21.15.28:38884) 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 -- 172.21.15.28:0/2102531089 learned_addr learned my addr 172.21.15.28:0/2102531089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda2fe00640 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x55957580d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda2f400640 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55957580a420 0x55957585a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 -- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55957580a420 msgr2=0x55957585a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55957580a420 0x55957585a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 -- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 msgr2=0x55957580d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x55957580d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.144+0000 7fda35200640 1 -- 172.21.15.28:0/2102531089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595757402e0 con 0x559575834110 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda35200640 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fda2802ac90 tx=0x7fda28079ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d233d7b34d4e2f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda24c00640 1 -- 172.21.15.28:0/2102531089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280a8ea0 con 0x559575834110 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda24c00640 1 -- 172.21.15.28:0/2102531089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda280817a0 con 0x559575834110 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda24c00640 1 -- 172.21.15.28:0/2102531089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda28075ac0 con 0x559575834110 2024-09-13T17:22:45.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 -- 172.21.15.28:0/2102531089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 msgr2=0x55957585a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 secure :-1 s=READY pgs=2491 cs=0 l=1 rev1=1 crypto rx=0x7fda2802ac90 tx=0x7fda28079ab0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 -- 172.21.15.28:0/2102531089 shutdown_connections 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559575834110 0x55957585a670 unknown :-1 s=CLOSED pgs=2491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55957580a420 0x55957585a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 --2- 172.21.15.28:0/2102531089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x55957580d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.145+0000 7fda39fd9740 1 -- 172.21.15.28:0/2102531089 >> 172.21.15.28:0/2102531089 conn(0x559575809390 msgr2=0x559575851840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.146+0000 7fda39fd9740 1 -- 172.21.15.28:0/2102531089 shutdown_connections 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.146+0000 7fda39fd9740 1 -- 172.21.15.28:0/2102531089 wait complete. 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.146+0000 7fda39fd9740 1 -- start start 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.146+0000 7fda39fd9740 1 Processor -- start 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x559575834630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559575834110 0x55957582b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55957576fd40 con 0x5595758082c0 2024-09-13T17:22:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559575711280 con 0x55957580cc60 2024-09-13T17:22:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x559575710f20 con 0x559575834110 2024-09-13T17:22:45.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.147+0000 7fda39fd9740 10 client.0 _ll_register_callbacks cb 0x5595757693b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda2fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x559575834630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38894/0 (socket says 172.21.15.28:38894) 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 -- 172.21.15.28:0/1421855627 learned_addr learned my addr 172.21.15.28:0/1421855627 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda2f400640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559575834110 0x55957582b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 -- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559575834110 msgr2=0x55957582b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559575834110 0x55957582b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 -- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 msgr2=0x559575834630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x559575834630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 -- 172.21.15.28:0/1421855627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda301526e0 con 0x5595758082c0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda35200640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fda28079c60 tx=0x7fda280aa040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc9af3f9cfc81c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.148+0000 7fda19600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:45.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda280a8ea0 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5595754634e0 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda280788a0 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda28078a60 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 10 client.11688 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x55957576e3c0 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5595754634e0 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fda28081310 con 0x5595758082c0 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 10 client.11688 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 10 client.11688 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 20 client.11688 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55957582d310 0x559575832f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 4 client.11688 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595754747e0 con 0x55957582d310 2024-09-13T17:22:45.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.149+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55957582d310 0x559575832f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.150+0000 7fda2fe00640 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55957582d310 0x559575832f10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8632f2fc5d13860a server_cookie=4cac2e7bc58ff771 in_seq=0 out_seq=0 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.150+0000 7fda19600640 10 client.11688 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda19600640 1 -- 172.21.15.28:0/1421855627 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5595754747e0 con 0x55957582d310 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda19600640 10 client.11688 handle_command_reply: tid=0 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 client.11688 shutdown 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 msgr2=0x55957582cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fda28079c60 tx=0x7fda280aa040 comp rx=0 tx=0).stop 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 shutdown_connections 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55957582d310 0x559575832f10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5595758082c0 0x55957582cdd0 unknown :-1 s=CLOSED pgs=2492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x559575834110 0x55957582b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda19600640 0 client.11688 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:45.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 --2- 172.21.15.28:0/1421855627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55957580cc60 0x559575834630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 >> 172.21.15.28:0/1421855627 conn(0x55957580a420 msgr2=0x5595758524f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 shutdown_connections 2024-09-13T17:22:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 1 -- 172.21.15.28:0/1421855627 wait complete. 2024-09-13T17:22:45.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.151+0000 7fda39fd9740 20 client.11688 trim_cache size 0 max 16384 2024-09-13T17:22:45.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.166+0000 7fda39fd9740 1 Processor -- start 2024-09-13T17:22:45.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.166+0000 7fda39fd9740 1 -- start start 2024-09-13T17:22:45.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x559575851fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595754d7040 con 0x55957580cc60 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559575835f60 con 0x559575834110 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55957582cf60 con 0x5595758082c0 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.167+0000 7fda2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38908/0 (socket says 172.21.15.28:38908) 2024-09-13T17:22:45.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 -- 172.21.15.28:0/3908583789 learned_addr learned my addr 172.21.15.28:0/3908583789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2fe00640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda35200640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x559575851fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 -- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 msgr2=0x55957582a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 -- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 msgr2=0x559575851fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x559575851fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 -- 172.21.15.28:0/3908583789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559575710610 con 0x55957580cc60 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2fe00640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda35200640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x559575851fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.168+0000 7fda2f400640 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fda1c001500 tx=0x7fda1c0019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183630fababd42f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda25600640 1 -- 172.21.15.28:0/3908583789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c01a030 con 0x55957580cc60 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda25600640 1 -- 172.21.15.28:0/3908583789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda1c00d070 con 0x55957580cc60 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda25600640 1 -- 172.21.15.28:0/3908583789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c007030 con 0x55957580cc60 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 -- 172.21.15.28:0/3908583789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 msgr2=0x55957586acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fda1c001500 tx=0x7fda1c0019c0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 -- 172.21.15.28:0/3908583789 shutdown_connections 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x559575851fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x55957586acc0 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3908583789 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 -- 172.21.15.28:0/3908583789 >> 172.21.15.28:0/3908583789 conn(0x55957580a420 msgr2=0x5595758356b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 -- 172.21.15.28:0/3908583789 shutdown_connections 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.169+0000 7fda39fd9740 1 -- 172.21.15.28:0/3908583789 wait complete. 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.170+0000 7fda39fd9740 1 -- start start 2024-09-13T17:22:45.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.170+0000 7fda39fd9740 1 Processor -- start 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x55957580a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x559575863a90 con 0x55957580cc60 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559575850640 con 0x559575834110 2024-09-13T17:22:45.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55957582e910 con 0x5595758082c0 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.171+0000 7fda39fd9740 10 client.0 _ll_register_callbacks cb 0x7fda20009730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38910/0 (socket says 172.21.15.28:38910) 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 -- 172.21.15.28:0/3152528257 learned_addr learned my addr 172.21.15.28:0/3152528257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2fe00640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda35200640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x55957580a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 -- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 msgr2=0x55957582eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 -- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 msgr2=0x55957580a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x55957580a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 -- 172.21.15.28:0/3152528257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3006f920 con 0x55957580cc60 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda35200640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x55957580a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:22:45.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda2f400640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fda1c017040 tx=0x7fda1c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b5d6e35f8a0253 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.172+0000 7fda19600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda1c007230 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(205..205 src has 1..205) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda1c01b070 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda280817a0 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda1c003e10 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 10 client.11694 fetch_fsmap learned FSMap version 172 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x559575735b20 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda280817a0 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fda1c007390 con 0x55957580cc60 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 10 client.11694 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 10 client.11694 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 20 client.11694 populate_metadata read hostname 'smithi028' 2024-09-13T17:22:45.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x559575862d70 0x559575863150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 4 client.11694 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.173+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5595754c7e30 con 0x559575862d70 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda2fe00640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x559575862d70 0x559575863150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda2fe00640 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x559575862d70 0x559575863150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fbff1e426449e3e server_cookie=f2d562ec63516916 in_seq=0 out_seq=0 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda19600640 10 client.11694 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda19600640 1 -- 172.21.15.28:0/3152528257 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x5595754c7e30 con 0x559575862d70 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda19600640 10 client.11694 handle_command_reply: tid=0 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.174+0000 7fda39fd9740 1 client.11694 shutdown 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 msgr2=0x5595758a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fda1c017040 tx=0x7fda1c000ed0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 shutdown_connections 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x559575862d70 0x559575863150 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559575834110 0x55957580a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55957580cc60 0x5595758a49b0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 --2- 172.21.15.28:0/3152528257 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5595758082c0 0x55957582eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda19600640 0 client.11694 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 >> 172.21.15.28:0/3152528257 conn(0x55957580a420 msgr2=0x5595758086a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 shutdown_connections 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 1 -- 172.21.15.28:0/3152528257 wait complete. 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.175+0000 7fda39fd9740 20 client.11694 trim_cache size 0 max 16384 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:22:45.181 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "caab1373-a73e-4d6d-a8e8-0179e4417548": { 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "caab1373-a73e-4d6d-a8e8-0179e4417548", 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,repair" 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:22:45.182 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 msgr2=0x7fda1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 0x7fda1006d0c0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fda20009ac0 tx=0x7fda20008040 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 msgr2=0x7fda30146b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7fda28034040 tx=0x7fda28004a40 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 shutdown_connections 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fda1006ac00 0x7fda1006d0c0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fda3015cd80 0x7fda30146b60 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda30159770 0x7fda30156ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 --2- 172.21.15.28:0/1673434470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fda3006f3f0 0x7fda30155160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:22:45.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 >> 172.21.15.28:0/1673434470 conn(0x7fda300b0270 msgr2=0x7fda30158930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:22:45.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 shutdown_connections 2024-09-13T17:22:45.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:22:45.176+0000 7fda37200640 1 -- 172.21.15.28:0/1673434470 wait complete. 2024-09-13T17:22:45.193 INFO:tasks.cephfs.filesystem:scrub status for tag:caab1373-a73e-4d6d-a8e8-0179e4417548 - {'path': '/', 'tag': 'caab1373-a73e-4d6d-a8e8-0179e4417548', 'options': 'recursive,repair'} 2024-09-13T17:23:11.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:23:11.838 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:23:15.195 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.14:0 scrub status 2024-09-13T17:23:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.406+0000 7faf6ae00640 1 Processor -- start 2024-09-13T17:23:15.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.406+0000 7faf6ae00640 1 -- start start 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf6ae00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf6415ab60 0x7faf6406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf6ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf64071490 0x7faf6406c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf6ae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6406c8b0 con 0x7faf6415a780 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf6ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6406cb90 con 0x7faf64071490 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf6ae00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf640cc980 con 0x7faf6415ab60 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.407+0000 7faf68e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf64071490 0x7faf6406c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faf64071490 0x7faf6406c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51848/0 (socket says 172.21.15.28:51848) 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 -- 172.21.15.28:0/3598989604 learned_addr learned my addr 172.21.15.28:0/3598989604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf63400640 1 --2- 172.21.15.28:0/3598989604 >> v2:172.21.15.189:3300/0 conn(0x7faf6415ab60 0x7faf6406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 -- 172.21.15.28:0/3598989604 >> v2:172.21.15.189:3300/0 conn(0x7faf6415ab60 msgr2=0x7faf6406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 --2- 172.21.15.28:0/3598989604 >> v2:172.21.15.189:3300/0 conn(0x7faf6415ab60 0x7faf6406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 -- 172.21.15.28:0/3598989604 >> v1:172.21.15.189:6789/0 conn(0x7faf6415a780 legacy=0x7faf64070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.408+0000 7faf68e00640 1 -- 172.21.15.28:0/3598989604 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf640ccc60 con 0x7faf64071490 2024-09-13T17:23:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.409+0000 7faf68e00640 1 --2- 172.21.15.28:0/3598989604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64071490 0x7faf6406c080 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7faf6406f380 tx=0x7faf5c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246e5a3d0013f54d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.409+0000 7faf62a00640 1 -- 172.21.15.28:0/3598989604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c031600 con 0x7faf64071490 2024-09-13T17:23:15.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.409+0000 7faf62a00640 1 -- 172.21.15.28:0/3598989604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c02d050 con 0x7faf64071490 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.409+0000 7faf62a00640 1 -- 172.21.15.28:0/3598989604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c031980 con 0x7faf64071490 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 -- 172.21.15.28:0/3598989604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64071490 msgr2=0x7faf6406c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 --2- 172.21.15.28:0/3598989604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64071490 0x7faf6406c080 secure :-1 s=READY pgs=2495 cs=0 l=1 rev1=1 crypto rx=0x7faf6406f380 tx=0x7faf5c02a730 comp rx=0 tx=0).stop 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 -- 172.21.15.28:0/3598989604 shutdown_connections 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 --2- 172.21.15.28:0/3598989604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf64071490 0x7faf6406c080 unknown :-1 s=CLOSED pgs=2495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 --2- 172.21.15.28:0/3598989604 >> v2:172.21.15.189:3300/0 conn(0x7faf6415ab60 0x7faf6406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 -- 172.21.15.28:0/3598989604 >> 172.21.15.28:0/3598989604 conn(0x7faf640b0270 msgr2=0x7faf640b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 -- 172.21.15.28:0/3598989604 shutdown_connections 2024-09-13T17:23:15.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.410+0000 7faf6ae00640 1 -- 172.21.15.28:0/3598989604 wait complete. 2024-09-13T17:23:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.412+0000 7faf6ae00640 1 Processor -- start 2024-09-13T17:23:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.412+0000 7faf6ae00640 1 -- start start 2024-09-13T17:23:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64071490 0x7faf6415bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf6415a780 0x7faf641f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf64153c60 con 0x7faf6415ab60 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf641f3610 con 0x7faf64071490 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf6ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6415c500 con 0x7faf6415a780 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51856/0 (socket says 172.21.15.28:51856) 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 -- 172.21.15.28:0/2089319519 learned_addr learned my addr 172.21.15.28:0/2089319519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf63400640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf6415a780 0x7faf641f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf63e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64071490 0x7faf6415bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 -- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf6415a780 msgr2=0x7faf641f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf6415a780 0x7faf641f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 -- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64071490 msgr2=0x7faf6415bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64071490 0x7faf6415bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.413+0000 7faf68e00640 1 -- 172.21.15.28:0/2089319519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf5c009b30 con 0x7faf6415ab60 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf68e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009f40 tx=0x7faf5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf5be00640 1 -- 172.21.15.28:0/2089319519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c03b810 con 0x7faf6415ab60 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf5be00640 1 -- 172.21.15.28:0/2089319519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c03b970 con 0x7faf6415ab60 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf5be00640 1 -- 172.21.15.28:0/2089319519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c03bd20 con 0x7faf6415ab60 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf641517e0 con 0x7faf6415ab60 2024-09-13T17:23:15.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.414+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf641f3050 con 0x7faf6415ab60 2024-09-13T17:23:15.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.417+0000 7faf5be00640 1 -- 172.21.15.28:0/2089319519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faf5c03a3a0 con 0x7faf6415ab60 2024-09-13T17:23:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.418+0000 7faf5be00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 0x7faf4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.418+0000 7faf5be00640 1 -- 172.21.15.28:0/2089319519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7faf5c0a9f00 con 0x7faf6415ab60 2024-09-13T17:23:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.418+0000 7faf6ddf2740 1 Processor -- start 2024-09-13T17:23:15.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.418+0000 7faf63e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 0x7faf4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.419+0000 7faf6ddf2740 1 -- start start 2024-09-13T17:23:15.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.419+0000 7faf63e00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 0x7faf4406d0c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7faf54000f60 tx=0x7faf54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.419+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11e8f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.419+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.419+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11edc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11d956a0 con 0x55cf11e8c420 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11df8cc0 con 0x55cf11e8ec60 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf11df6450 con 0x55cf11eb6110 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51872/0 (socket says 172.21.15.28:51872) 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 -- 172.21.15.28:0/2416444760 learned_addr learned my addr 172.21.15.28:0/2416444760 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63e00640 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11e8f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf68e00640 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11edc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 -- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 msgr2=0x55cf11edc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11edc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 -- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 msgr2=0x55cf11e8f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11e8f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.420+0000 7faf63400640 1 -- 172.21.15.28:0/2416444760 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf11dc22e0 con 0x55cf11e8c420 2024-09-13T17:23:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf63400640 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7faf50009b70 tx=0x7faf50011690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d32c7dbccf4af61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf58c00640 1 -- 172.21.15.28:0/2416444760 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf50009e70 con 0x55cf11e8c420 2024-09-13T17:23:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf58c00640 1 -- 172.21.15.28:0/2416444760 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf50004030 con 0x55cf11e8c420 2024-09-13T17:23:15.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf58c00640 1 -- 172.21.15.28:0/2416444760 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf500043e0 con 0x55cf11e8c420 2024-09-13T17:23:15.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 -- 172.21.15.28:0/2416444760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 msgr2=0x55cf11edc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7faf50009b70 tx=0x7faf50011690 comp rx=0 tx=0).stop 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 -- 172.21.15.28:0/2416444760 shutdown_connections 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11edc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8c420 0x55cf11edc130 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/2416444760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11e8f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.421+0000 7faf6ddf2740 1 -- 172.21.15.28:0/2416444760 >> 172.21.15.28:0/2416444760 conn(0x55cf11e8b390 msgr2=0x55cf11ed3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.422+0000 7faf6ddf2740 1 -- 172.21.15.28:0/2416444760 shutdown_connections 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.422+0000 7faf6ddf2740 1 -- 172.21.15.28:0/2416444760 wait complete. 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.422+0000 7faf6ddf2740 1 -- start start 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.422+0000 7faf6ddf2740 1 Processor -- start 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11eb7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11eb53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11df1d40 con 0x55cf11e8ec60 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11d93280 con 0x55cf11eb4f20 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.423+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf11d92f20 con 0x55cf11eb6110 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf6ddf2740 10 client.0 _ll_register_callbacks cb 0x55cf11deb3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51878/0 (socket says 172.21.15.28:51878) 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 -- 172.21.15.28:0/3009606340 learned_addr learned my addr 172.21.15.28:0/3009606340 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63400640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11eb7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 -- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 msgr2=0x55cf11eb7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11eb7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 -- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 msgr2=0x55cf11eb53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf68e00640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11eb53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11eb53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 -- 172.21.15.28:0/3009606340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf500097a0 con 0x55cf11e8ec60 2024-09-13T17:23:15.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63400640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11eb7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf68e00640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11eb53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf63e00640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7faf54014730 tx=0x7faf54001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18632a14cb3343bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.424+0000 7faf4d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54013730 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7faf5401c040 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cf11ae54e0 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf54013890 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 10 client.11706 fetch_fsmap learned FSMap version 172 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x55cf11df03c0 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cf11ae54e0 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faf54012070 con 0x55cf11e8ec60 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 10 client.11706 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:23:15.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 10 client.11706 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:23:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 20 client.11706 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11edb930 0x55cf11ea2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 4 client.11706 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:23:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.425+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf11af67e0 con 0x55cf11edb930 2024-09-13T17:23:15.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.426+0000 7faf63400640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11edb930 0x55cf11ea2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.426+0000 7faf63400640 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11edb930 0x55cf11ea2370 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e85b6fe674b50c9b server_cookie=e7d5f64f80e7a16c in_seq=0 out_seq=0 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.426+0000 7faf4d600640 10 client.11706 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf4d600640 1 -- 172.21.15.28:0/3009606340 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55cf11af67e0 con 0x55cf11edb930 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf4d600640 10 client.11706 handle_command_reply: tid=0 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 client.11706 shutdown 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 msgr2=0x55cf11ed08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7faf54014730 tx=0x7faf54001120 comp rx=0 tx=0).stop 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 shutdown_connections 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11edb930 0x55cf11ea2370 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11eb53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11eb7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3009606340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11ed08f0 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf4d600640 0 client.11706 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 >> 172.21.15.28:0/3009606340 conn(0x55cf11e8c420 msgr2=0x55cf11ed4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 shutdown_connections 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.427+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3009606340 wait complete. 2024-09-13T17:23:15.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.428+0000 7faf6ddf2740 20 client.11706 trim_cache size 0 max 16384 2024-09-13T17:23:15.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.443+0000 7faf6ddf2740 1 Processor -- start 2024-09-13T17:23:15.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.443+0000 7faf6ddf2740 1 -- start start 2024-09-13T17:23:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.443+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11ed4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb4f20 0x55cf11ed3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11b59040 con 0x55cf11eb6110 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11e8aa50 con 0x55cf11e8ec60 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf11eba6d0 con 0x55cf11eb4f20 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51890/0 (socket says 172.21.15.28:51890) 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 -- 172.21.15.28:0/847000204 learned_addr learned my addr 172.21.15.28:0/847000204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf63e00640 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11ed4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf63400640 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb4f20 0x55cf11ed3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 -- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb4f20 msgr2=0x55cf11ed3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb4f20 0x55cf11ed3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 -- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 msgr2=0x55cf11ed4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11ed4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 -- 172.21.15.28:0/847000204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf11d92610 con 0x55cf11eb6110 2024-09-13T17:23:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.444+0000 7faf68e00640 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7faf5c031fd0 tx=0x7faf5c078740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c09233e47ad89fa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf59600640 1 -- 172.21.15.28:0/847000204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c07caa0 con 0x55cf11eb6110 2024-09-13T17:23:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf59600640 1 -- 172.21.15.28:0/847000204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf5c081060 con 0x55cf11eb6110 2024-09-13T17:23:15.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf59600640 1 -- 172.21.15.28:0/847000204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf5c075430 con 0x55cf11eb6110 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 -- 172.21.15.28:0/847000204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 msgr2=0x55cf11ed3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7faf5c031fd0 tx=0x7faf5c078740 comp rx=0 tx=0).stop 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 -- 172.21.15.28:0/847000204 shutdown_connections 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11eb6110 0x55cf11ed3840 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb4f20 0x55cf11ed3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/847000204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11e8ec60 0x55cf11ed4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 -- 172.21.15.28:0/847000204 >> 172.21.15.28:0/847000204 conn(0x55cf11e8c420 msgr2=0x55cf11eb57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 -- 172.21.15.28:0/847000204 shutdown_connections 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.445+0000 7faf6ddf2740 1 -- 172.21.15.28:0/847000204 wait complete. 2024-09-13T17:23:15.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.446+0000 7faf6ddf2740 1 -- start start 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.446+0000 7faf6ddf2740 1 Processor -- start 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11f3d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11f26940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11f3e270 con 0x55cf11e8ec60 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf11e8db50 con 0x55cf11eb4f20 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cf11e8dda0 con 0x55cf11eb6110 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf6ddf2740 10 client.0 _ll_register_callbacks cb 0x7faf54001df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51902/0 (socket says 172.21.15.28:51902) 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf63e00640 1 -- 172.21.15.28:0/3665946339 learned_addr learned my addr 172.21.15.28:0/3665946339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.447+0000 7faf63400640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11f3d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf68e00640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11f26940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 -- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 msgr2=0x55cf11f26940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11f26940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 -- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 msgr2=0x55cf11f3d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11f3d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 -- 172.21.15.28:0/3665946339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf641f3050 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf63e00640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7faf5400fc70 tx=0x7faf54014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda846652eadbdb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf4d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf54007ea0 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf50004030 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.448+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7faf54021c00 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=172) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf54021dc0 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 10 client.11712 fetch_fsmap learned FSMap version 172 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=172,monmap=2+}) v3 -- 0x55cf11db7b20 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf50004030 con 0x55cf11e8ec60 2024-09-13T17:23:15.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 172) v1 ==== 3114+0+0 (secure 0 0 0) 0x7faf5402a3f0 con 0x55cf11e8ec60 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 10 client.11712 fetch_fsmap finished waiting for FSMap version 172 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 10 client.11712 resolve_mds: resolved 14:0 to role '14:0' aka daemon mds.c 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 20 client.11712 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11e9f920 0x55cf11ed2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 4 client.11712 mds_command: new command op to 11478 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:23:15.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 --> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cf11b49e30 con 0x55cf11e9f920 2024-09-13T17:23:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.449+0000 7faf63400640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11e9f920 0x55cf11ed2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf63400640 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11e9f920 0x55cf11ed2fa0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5818d1562a4ce5e server_cookie=6bcf0fc19eec4943 in_seq=0 out_seq=0 2024-09-13T17:23:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf4d600640 10 client.11712 ms_handle_connect on v2:172.21.15.28:6832/299660514 2024-09-13T17:23:15.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf4d600640 1 -- 172.21.15.28:0/3665946339 <== mds.0 v2:172.21.15.28:6832/299660514 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cf11b49e30 con 0x55cf11e9f920 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf4d600640 10 client.11712 handle_command_reply: tid=0 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 client.11712 shutdown 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 msgr2=0x55cf11e8d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 secure :-1 s=READY pgs=2500 cs=0 l=1 rev1=1 crypto rx=0x7faf5400fc70 tx=0x7faf54014000 comp rx=0 tx=0).stop 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 shutdown_connections 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:6832/299660514,v1:172.21.15.28:6834/299660514] conn(0x55cf11e9f920 0x55cf11ed2fa0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cf11eb6110 0x55cf11f26940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cf11eb4f20 0x55cf11f3d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 --2- 172.21.15.28:0/3665946339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cf11e8ec60 0x55cf11e8d340 unknown :-1 s=CLOSED pgs=2500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf4d600640 0 client.11712 ms_handle_reset on v2:172.21.15.28:6832/299660514 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 >> 172.21.15.28:0/3665946339 conn(0x55cf11e8c420 msgr2=0x55cf11ed4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 shutdown_connections 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.450+0000 7faf6ddf2740 1 -- 172.21.15.28:0/3665946339 wait complete. 2024-09-13T17:23:15.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ddf2740 20 client.11712 trim_cache size 0 max 16384 2024-09-13T17:23:15.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:23:15.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:23:15.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 msgr2=0x7faf4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 0x7faf4406d0c0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7faf54000f60 tx=0x7faf54009040 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 msgr2=0x7faf6406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7faf5c009f40 tx=0x7faf5c004a40 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 shutdown_connections 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf4406ac00 0x7faf4406d0c0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf6415ab60 0x7faf6406ba80 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf6415a780 0x7faf641f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 --2- 172.21.15.28:0/2089319519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf64071490 0x7faf6415bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 >> 172.21.15.28:0/2089319519 conn(0x7faf640b0270 msgr2=0x7faf64158440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 shutdown_connections 2024-09-13T17:23:15.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.451+0000 7faf6ae00640 1 -- 172.21.15.28:0/2089319519 wait complete. 2024-09-13T17:23:15.467 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> import stat 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> import json 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> import sys 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/sixmegs'"'"') 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> except OSError as e: 2024-09-13T17:23:15.468 DEBUG:teuthology.orchestra.run.smithi028:> sys.exit(e.errno) 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> print(json.dumps( 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> indent=2)) 2024-09-13T17:23:15.469 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mode": 33188, 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ino": 1099511627777, 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_dev": 42, 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_nlink": 1, 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_uid": 1000, 2024-09-13T17:23:15.571 INFO:teuthology.orchestra.run.smithi028.stdout: "st_gid": 1333, 2024-09-13T17:23:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "st_size": 6291456, 2024-09-13T17:23:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "st_atime": 1726248163.4181972, 2024-09-13T17:23:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "st_mtime": 1726248163.4541962, 2024-09-13T17:23:15.572 INFO:teuthology.orchestra.run.smithi028.stdout: "st_ctime": 1726248163.4541962 2024-09-13T17:23:15.572 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:15.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.951+0000 7fd6fea00640 1 Processor -- start 2024-09-13T17:23:15.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.951+0000 7fd6fea00640 1 -- start start 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6fea00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd6f80713d0 0x7fd6f806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6fea00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f815e100 con 0x7fd6f80cabc0 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6fea00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f815e3e0 con 0x7fd6f815a540 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6fea00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f80adb00 con 0x7fd6f80713d0 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fd6f80cabc0 0x7fd6f8070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55886/0 (socket says 172.21.15.28:55886) 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.952+0000 7fd6efe00640 1 -- 172.21.15.28:0/3791573423 learned_addr learned my addr 172.21.15.28:0/3791573423 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6fca00640 1 --2- 172.21.15.28:0/3791573423 >> v2:172.21.15.189:3300/0 conn(0x7fd6f80713d0 0x7fd6f806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6f7400640 1 -- 172.21.15.28:0/3791573423 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329834568 0 0) 0x7fd6f815e100 con 0x7fd6f80cabc0 2024-09-13T17:23:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6f7400640 1 -- 172.21.15.28:0/3791573423 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6dc003520 con 0x7fd6f80cabc0 2024-09-13T17:23:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6fca00640 1 -- 172.21.15.28:0/3791573423 >> v1:172.21.15.189:6789/0 conn(0x7fd6f815a540 legacy=0x7fd6f815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6fca00640 1 -- 172.21.15.28:0/3791573423 >> v1:172.21.15.28:6789/0 conn(0x7fd6f80cabc0 legacy=0x7fd6f8070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.953+0000 7fd6fca00640 1 -- 172.21.15.28:0/3791573423 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f815ef80 con 0x7fd6f80713d0 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.954+0000 7fd6fca00640 1 --2- 172.21.15.28:0/3791573423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80713d0 0x7fd6f806c430 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009d90 tx=0x7fd6f002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e2732f8a6a5725 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6f7400640 1 -- 172.21.15.28:0/3791573423 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0007740 con 0x7fd6f80713d0 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6f7400640 1 -- 172.21.15.28:0/3791573423 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f002d050 con 0x7fd6f80713d0 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6f7400640 1 -- 172.21.15.28:0/3791573423 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0007ac0 con 0x7fd6f80713d0 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 -- 172.21.15.28:0/3791573423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80713d0 msgr2=0x7fd6f806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 --2- 172.21.15.28:0/3791573423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80713d0 0x7fd6f806c430 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0009d90 tx=0x7fd6f002a660 comp rx=0 tx=0).stop 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 -- 172.21.15.28:0/3791573423 shutdown_connections 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 --2- 172.21.15.28:0/3791573423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f80713d0 0x7fd6f806c430 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 -- 172.21.15.28:0/3791573423 >> 172.21.15.28:0/3791573423 conn(0x7fd6f8069e80 msgr2=0x7fd6f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:15.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 -- 172.21.15.28:0/3791573423 shutdown_connections 2024-09-13T17:23:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.955+0000 7fd6fea00640 1 -- 172.21.15.28:0/3791573423 wait complete. 2024-09-13T17:23:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.957+0000 7fd6fea00640 1 Processor -- start 2024-09-13T17:23:15.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.957+0000 7fd6fea00640 1 -- start start 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6f80cabc0 0x7fd6f8151c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 0x7fd6f8145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8153260 con 0x7fd6f80713d0 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81525e0 con 0x7fd6f815a540 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6fea00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f81528c0 con 0x7fd6f80cabc0 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6f7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51914/0 (socket says 172.21.15.28:51914) 2024-09-13T17:23:15.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.958+0000 7fd6f7e00640 1 -- 172.21.15.28:0/1385608089 learned_addr learned my addr 172.21.15.28:0/1385608089 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6fca00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 0x7fd6f8145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6efe00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6f80cabc0 0x7fd6f8151c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 -- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6f80cabc0 msgr2=0x7fd6f8151c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6f80cabc0 0x7fd6f8151c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 -- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 msgr2=0x7fd6f8145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 0x7fd6f8145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 -- 172.21.15.28:0/1385608089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f0009a70 con 0x7fd6f80713d0 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6fca00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 0x7fd6f8145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f7e00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c400 tx=0x7fd6e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80052d0 con 0x7fd6f80713d0 2024-09-13T17:23:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8005430 con 0x7fd6f80713d0 2024-09-13T17:23:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8003830 con 0x7fd6f80713d0 2024-09-13T17:23:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.959+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8152ba0 con 0x7fd6f80713d0 2024-09-13T17:23:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.960+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8153680 con 0x7fd6f80713d0 2024-09-13T17:23:15.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.961+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8153810 con 0x7fd6f80713d0 2024-09-13T17:23:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.962+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd6e8003990 con 0x7fd6f80713d0 2024-09-13T17:23:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.963+0000 7fd6f4c00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 0x7fd6d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:15.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.963+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd6e8088970 con 0x7fd6f80713d0 2024-09-13T17:23:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.963+0000 7fd6efe00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 0x7fd6d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:15.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.964+0000 7fd6efe00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 0x7fd6d406d0c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd6f81f5a30 tx=0x7fd6e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:15.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:15.967+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd6e8053ab0 con 0x7fd6f80713d0 2024-09-13T17:23:16.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.184+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6f81533a0 con 0x7fd6f80713d0 2024-09-13T17:23:16.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.184+0000 7fd6f4c00640 1 -- 172.21.15.28:0/1385608089 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7fd6e8017090 con 0x7fd6f80713d0 2024-09-13T17:23:16.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:16.185 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:23:16.186 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 msgr2=0x7fd6d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 0x7fd6d406d0c0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd6f81f5a30 tx=0x7fd6e0009040 comp rx=0 tx=0).stop 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 msgr2=0x7fd6f8155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 secure :-1 s=READY pgs=2502 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800c400 tx=0x7fd6e800c8c0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 shutdown_connections 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd6d406ac00 0x7fd6d406d0c0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6f815a540 0x7fd6f8145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd6f80cabc0 0x7fd6f8151c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 --2- 172.21.15.28:0/1385608089 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd6f80713d0 0x7fd6f8155b20 unknown :-1 s=CLOSED pgs=2502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 >> 172.21.15.28:0/1385608089 conn(0x7fd6f8069e80 msgr2=0x7fd6f80ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:16.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 shutdown_connections 2024-09-13T17:23:16.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.187+0000 7fd6fea00640 1 -- 172.21.15.28:0/1385608089 wait complete. 2024-09-13T17:23:16.204 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T17:23:16.383 INFO:teuthology.orchestra.run.smithi028.stdout:{"message":"","return_code":0} 2024-09-13T17:23:16.383 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T17:23:16.384 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-13T17:23:16.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.459+0000 7fcd38b73f40 1 Processor -- start 2024-09-13T17:23:16.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.459+0000 7fcd38b73f40 1 -- start start 2024-09-13T17:23:16.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 0x55c8260d0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c826060100 0x55c8260e8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c825fc4b10 con 0x55c8260d1c40 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c825fc0400 con 0x55c8260198a0 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.460+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c825f5bfe0 con 0x55c826060100 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51916/0 (socket says 172.21.15.28:51916) 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 -- 172.21.15.28:0/215370931 learned_addr learned my addr 172.21.15.28:0/215370931 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd35e00640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 0x55c8260d0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd35400640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c826060100 0x55c8260e8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 -- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c826060100 msgr2=0x55c8260e8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c826060100 0x55c8260e8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 -- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 msgr2=0x55c8260d0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:16.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 0x55c8260d0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 -- 172.21.15.28:0/215370931 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28005d20 con 0x55c8260d1c40 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd35e00640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 0x55c8260d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.461+0000 7fcd36800640 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fcd30006e50 tx=0x7fcd30001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e355e964ad624b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd2fe00640 1 -- 172.21.15.28:0/215370931 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30004520 con 0x55c8260d1c40 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd2fe00640 1 -- 172.21.15.28:0/215370931 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd30004680 con 0x55c8260d1c40 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd2fe00640 1 -- 172.21.15.28:0/215370931 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd30004a30 con 0x55c8260d1c40 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 -- 172.21.15.28:0/215370931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 msgr2=0x55c8260ed130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7fcd30006e50 tx=0x7fcd30001a20 comp rx=0 tx=0).stop 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 -- 172.21.15.28:0/215370931 shutdown_connections 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c8260d1c40 0x55c8260ed130 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c826060100 0x55c8260e8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/215370931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260198a0 0x55c8260d0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 -- 172.21.15.28:0/215370931 >> 172.21.15.28:0/215370931 conn(0x55c826044270 msgr2=0x55c8260e93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 -- 172.21.15.28:0/215370931 shutdown_connections 2024-09-13T17:23:16.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.462+0000 7fcd38b73f40 1 -- 172.21.15.28:0/215370931 wait complete. 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.463+0000 7fcd38b73f40 1 Processor -- start 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.463+0000 7fcd38b73f40 1 -- start start 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c8260198a0 0x55c8260e6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 0x55c826181680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260d1c40 0x55c8260120e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c825f5bc80 con 0x55c826060100 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c825f528f0 con 0x55c8260d1c40 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd38b73f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c826012fd0 con 0x55c8260198a0 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 0x55c826181680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 0x55c826181680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51928/0 (socket says 172.21.15.28:51928) 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.464+0000 7fcd35400640 1 -- 172.21.15.28:0/281803025 learned_addr learned my addr 172.21.15.28:0/281803025 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:16.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35e00640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c8260198a0 0x55c8260e6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd36800640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260d1c40 0x55c8260120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 -- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c8260198a0 msgr2=0x55c8260e6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c8260198a0 0x55c8260e6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 -- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260d1c40 msgr2=0x55c8260120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260d1c40 0x55c8260120e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 -- 172.21.15.28:0/281803025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c001e10 con 0x55c826060100 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd36800640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8260d1c40 0x55c8260120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd35400640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 0x55c826181680 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400deb0 tx=0x7fcd2400b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd2ea00640 1 -- 172.21.15.28:0/281803025 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240032d0 con 0x55c826060100 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd2ea00640 1 -- 172.21.15.28:0/281803025 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400a040 con 0x55c826060100 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c825f5df60 con 0x55c826060100 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.465+0000 7fcd2ea00640 1 -- 172.21.15.28:0/281803025 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24001780 con 0x55c826060100 2024-09-13T17:23:16.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.466+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c825f5d9c0 con 0x55c826060100 2024-09-13T17:23:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.467+0000 7fcd2ea00640 1 -- 172.21.15.28:0/281803025 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcd240018e0 con 0x55c826060100 2024-09-13T17:23:16.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.468+0000 7fcd2ea00640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd180695e0 0x7fcd1806baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.468+0000 7fcd2ea00640 1 -- 172.21.15.28:0/281803025 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fcd24009050 con 0x55c826060100 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.469+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55c826181bc0 0x55c8260cef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.469+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- osd_op(unknown.0.0:1 29.7 29:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e207) v8 -- 0x55c8260e7b10 con 0x55c826181bc0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.469+0000 7fcd35e00640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd180695e0 0x7fcd1806baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.469+0000 7fcd36800640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55c826181bc0 0x55c8260cef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.469+0000 7fcd35e00640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd180695e0 0x7fcd1806baa0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcd28057000 tx=0x7fcd28052000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd36800640 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55c826181bc0 0x55c8260cef10 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd36800640 1 -- 172.21.15.28:0/281803025 <== osd.6 v2:172.21.15.189:6816/2946605912 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fcd300047f0 con 0x55c826181bc0 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55c826181bc0 msgr2=0x55c8260cef10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x55c826181bc0 0x55c8260cef10 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd180695e0 msgr2=0x7fcd1806baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd180695e0 0x7fcd1806baa0 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fcd28057000 tx=0x7fcd28052000 comp rx=0 tx=0).stop 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 msgr2=0x55c826181680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 --2- 172.21.15.28:0/281803025 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c826060100 0x55c826181680 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400deb0 tx=0x7fcd2400b590 comp rx=0 tx=0).stop 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd36800640 1 -- 172.21.15.28:0/281803025 reap_dead start 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 shutdown_connections 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 >> 172.21.15.28:0/281803025 conn(0x55c826044270 msgr2=0x55c8260e93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 shutdown_connections 2024-09-13T17:23:16.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:16.470+0000 7fcd38b73f40 1 -- 172.21.15.28:0/281803025 wait complete. 2024-09-13T17:23:16.476 INFO:teuthology.orchestra.run.smithi028.stdout:Ysixmegssubdir  2024-09-13T17:23:16.476 DEBUG:teuthology.orchestra.run.smithi028:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-13T17:23:16.602 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 1099511627777, 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "ancestors": [ 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dirino": 1099511627776, 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dname": "sixmegs", 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 8 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dirino": 1, 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "dname": "subdir", 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: "version": 11 2024-09-13T17:23:16.603 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:16.604 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-09-13T17:23:16.604 INFO:teuthology.orchestra.run.smithi028.stdout: "pool": 29, 2024-09-13T17:23:16.604 INFO:teuthology.orchestra.run.smithi028.stdout: "old_pools": [] 2024-09-13T17:23:16.604 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:16.604 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:16.613 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:23:16.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:16.702 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:16.787 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:23:16.787 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:16.787 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:16.822 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:16.823 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:16.898 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:17.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.092+0000 7fcdb4a00640 1 Processor -- start 2024-09-13T17:23:17.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.092+0000 7fcdb4a00640 1 -- start start 2024-09-13T17:23:17.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdb4a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcdb015a540 0x7fcdb006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdb4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 0x7fcdb015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdb4a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdb015e100 con 0x7fcdb015edb0 2024-09-13T17:23:17.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdb4a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdb00adb00 con 0x7fcdb015a540 2024-09-13T17:23:17.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdb4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb015e3e0 con 0x7fcdb00cabc0 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcdb015a540 0x7fcdb006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcdb015a540 0x7fcdb006c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51940/0 (socket says 172.21.15.28:51940) 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 -- 172.21.15.28:0/3053558337 learned_addr learned my addr 172.21.15.28:0/3053558337 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcda5400640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 0x7fcdb015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 -- 172.21.15.28:0/3053558337 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 msgr2=0x7fcdb015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 0x7fcdb015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 -- 172.21.15.28:0/3053558337 >> v1:172.21.15.189:6789/0 conn(0x7fcdb015edb0 legacy=0x7fcdb0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcdade00640 1 -- 172.21.15.28:0/3053558337 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb006ce80 con 0x7fcdb015a540 2024-09-13T17:23:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.094+0000 7fcda5400640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 0x7fcdb015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.095+0000 7fcdade00640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb015a540 0x7fcdb006c430 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009870 tx=0x7fcda002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9126bbaac73d20dd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdad400640 1 -- 172.21.15.28:0/3053558337 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda002ae90 con 0x7fcdb015a540 2024-09-13T17:23:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdad400640 1 -- 172.21.15.28:0/3053558337 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda0032440 con 0x7fcdb015a540 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdad400640 1 -- 172.21.15.28:0/3053558337 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda00327d0 con 0x7fcdb015a540 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 -- 172.21.15.28:0/3053558337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb015a540 msgr2=0x7fcdb006c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb015a540 0x7fcdb006c430 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009870 tx=0x7fcda002a180 comp rx=0 tx=0).stop 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 -- 172.21.15.28:0/3053558337 shutdown_connections 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb00cabc0 0x7fcdb015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/3053558337 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb015a540 0x7fcdb006c430 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 -- 172.21.15.28:0/3053558337 >> 172.21.15.28:0/3053558337 conn(0x7fcdb0069e80 msgr2=0x7fcdb006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.096+0000 7fcdb4a00640 1 -- 172.21.15.28:0/3053558337 shutdown_connections 2024-09-13T17:23:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.097+0000 7fcdb4a00640 1 -- 172.21.15.28:0/3053558337 wait complete. 2024-09-13T17:23:17.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.098+0000 7fcdb4a00640 1 Processor -- start 2024-09-13T17:23:17.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.098+0000 7fcdb4a00640 1 -- start start 2024-09-13T17:23:17.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 0x7fcdb01416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 0x7fcdb0144060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb015d410 con 0x7fcdb00cabc0 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb0147f50 con 0x7fcdb015a540 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdb4a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb00a8980 con 0x7fcdb0158dd0 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdade00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51954/0 (socket says 172.21.15.28:51954) 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.099+0000 7fcdade00640 1 -- 172.21.15.28:0/1725285980 learned_addr learned my addr 172.21.15.28:0/1725285980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdae800640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 0x7fcdb0144060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcda5400640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 0x7fcdb01416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 -- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 msgr2=0x7fcdb0144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 0x7fcdb0144060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 -- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 msgr2=0x7fcdb01416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 0x7fcdb01416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 -- 172.21.15.28:0/1725285980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda00094d0 con 0x7fcdb00cabc0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdae800640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 0x7fcdb0144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcda5400640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 0x7fcdb01416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdade00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009870 tx=0x7fcda00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda0030b80 con 0x7fcdb00cabc0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcda0030ce0 con 0x7fcdb00cabc0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb0159360 con 0x7fcdb00cabc0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.100+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda003d660 con 0x7fcdb00cabc0 2024-09-13T17:23:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.101+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb015ca70 con 0x7fcdb00cabc0 2024-09-13T17:23:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.102+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb015e870 con 0x7fcdb00cabc0 2024-09-13T17:23:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.102+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcda0039040 con 0x7fcdb00cabc0 2024-09-13T17:23:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.103+0000 7fcda6a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 0x7fcd8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.103+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fcda00aa190 con 0x7fcdb00cabc0 2024-09-13T17:23:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.104+0000 7fcda5400640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 0x7fcd8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.105+0000 7fcda5400640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 0x7fcd8c06d0c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0148760 tx=0x7fcd98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.108+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcda00752d0 con 0x7fcdb00cabc0 2024-09-13T17:23:17.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.288+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdb00b0f80 con 0x7fcdb00cabc0 2024-09-13T17:23:17.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.289+0000 7fcda6a00640 1 -- 172.21.15.28:0/1725285980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16819 (secure 0 0 0) 0x7fcda00788f0 con 0x7fcdb00cabc0 2024-09-13T17:23:17.290 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:17.291 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:55.171778+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.19","mappings":[{"from":6,"to":0}]},{"pgid":"29.39","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 msgr2=0x7fcd8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 0x7fcd8c06d0c0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0148760 tx=0x7fcd98008040 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 msgr2=0x7fcdb00a8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 secure :-1 s=READY pgs=2507 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009870 tx=0x7fcda00046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 shutdown_connections 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcd8c06ac00 0x7fcd8c06d0c0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcdb0158dd0 0x7fcdb0144060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdb015a540 0x7fcdb01416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 --2- 172.21.15.28:0/1725285980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcdb00cabc0 0x7fcdb00a8440 unknown :-1 s=CLOSED pgs=2507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 >> 172.21.15.28:0/1725285980 conn(0x7fcdb0069e80 msgr2=0x7fcdb00cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.291+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 shutdown_connections 2024-09-13T17:23:17.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.292+0000 7fcdb4a00640 1 -- 172.21.15.28:0/1725285980 wait complete. 2024-09-13T17:23:17.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:22:33.628286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:22:34.718902+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:17.310 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.467+0000 7fd9f9600640 1 Processor -- start 2024-09-13T17:23:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.467+0000 7fd9f9600640 1 -- start start 2024-09-13T17:23:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.468+0000 7fd9f9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 0x7fd9f4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.468+0000 7fd9f9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9f415ab60 0x7fd9f40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.468+0000 7fd9f9600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9f406c8b0 con 0x7fd9f406bb40 2024-09-13T17:23:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.468+0000 7fd9f9600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9f406c6a0 con 0x7fd9f415ab60 2024-09-13T17:23:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.468+0000 7fd9f9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f406cb90 con 0x7fd9f415a780 2024-09-13T17:23:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9f415ab60 0x7fd9f40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 0x7fd9f4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd9f415ab60 0x7fd9f40712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:46522/0 (socket says 172.21.15.28:46522) 2024-09-13T17:23:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 -- 172.21.15.28:0/1789370986 learned_addr learned my addr 172.21.15.28:0/1789370986 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 -- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 msgr2=0x7fd9f4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 0x7fd9f4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 -- 172.21.15.28:0/1789370986 >> v1:172.21.15.189:6789/0 conn(0x7fd9f406bb40 legacy=0x7fd9f406bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2000640 1 -- 172.21.15.28:0/1789370986 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f40cc980 con 0x7fd9f415ab60 2024-09-13T17:23:17.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.469+0000 7fd9f2a00640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 0x7fd9f4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.470+0000 7fd9f2000640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f415ab60 0x7fd9f40712c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009870 tx=0x7fd9e002a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79a19324d7023cb4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f1600640 1 -- 172.21.15.28:0/1789370986 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0032440 con 0x7fd9f415ab60 2024-09-13T17:23:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f1600640 1 -- 172.21.15.28:0/1789370986 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e00325a0 con 0x7fd9f415ab60 2024-09-13T17:23:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f1600640 1 -- 172.21.15.28:0/1789370986 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0032930 con 0x7fd9f415ab60 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 -- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f415ab60 msgr2=0x7fd9f40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f415ab60 0x7fd9f40712c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0009870 tx=0x7fd9e002a250 comp rx=0 tx=0).stop 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 -- 172.21.15.28:0/1789370986 shutdown_connections 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f415ab60 0x7fd9f40712c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 --2- 172.21.15.28:0/1789370986 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415a780 0x7fd9f4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 -- 172.21.15.28:0/1789370986 >> 172.21.15.28:0/1789370986 conn(0x7fd9f40b0270 msgr2=0x7fd9f40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 -- 172.21.15.28:0/1789370986 shutdown_connections 2024-09-13T17:23:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.471+0000 7fd9f9600640 1 -- 172.21.15.28:0/1789370986 wait complete. 2024-09-13T17:23:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.473+0000 7fd9f9600640 1 Processor -- start 2024-09-13T17:23:17.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.473+0000 7fd9f9600640 1 -- start start 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f406bb40 0x7fd9f4147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415ab60 0x7fd9f40a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41e3220 con 0x7fd9f415a780 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41e3500 con 0x7fd9f406bb40 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f41e37e0 con 0x7fd9f415ab60 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51970/0 (socket says 172.21.15.28:51970) 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.474+0000 7fd9f2000640 1 -- 172.21.15.28:0/2831258795 learned_addr learned my addr 172.21.15.28:0/2831258795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2a00640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f406bb40 0x7fd9f4147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f3400640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415ab60 0x7fd9f40a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 -- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415ab60 msgr2=0x7fd9f40a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415ab60 0x7fd9f40a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 -- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f406bb40 msgr2=0x7fd9f4147e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f406bb40 0x7fd9f4147e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 -- 172.21.15.28:0/2831258795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e400d8a0 con 0x7fd9f415a780 2024-09-13T17:23:17.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f2000640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fd9e002a290 tx=0x7fd9e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e0032440 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e0030ba0 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e00094d0 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.475+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e003d440 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.476+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f41e3da0 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.477+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f406bf90 con 0x7fd9f415a780 2024-09-13T17:23:17.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.477+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd9e0039040 con 0x7fd9f415a780 2024-09-13T17:23:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.478+0000 7fd9eaa00640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 0x7fd9d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.478+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd9e00a9fa0 con 0x7fd9f415a780 2024-09-13T17:23:17.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.479+0000 7fd9f2a00640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 0x7fd9d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.479+0000 7fd9f2a00640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 0x7fd9d406d0c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd9f41e23c0 tx=0x7fd9e400f040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.482+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd9e0075050 con 0x7fd9f415a780 2024-09-13T17:23:17.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.659+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9f40aecb0 con 0x7fd9f415a780 2024-09-13T17:23:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.660+0000 7fd9eaa00640 1 -- 172.21.15.28:0/2831258795 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v207) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9e0078670 con 0x7fd9f415a780 2024-09-13T17:23:17.661 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 msgr2=0x7fd9d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 0x7fd9d406d0c0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fd9f41e23c0 tx=0x7fd9e400f040 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 msgr2=0x7fd9f40af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7fd9e002a290 tx=0x7fd9e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 shutdown_connections 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd9d406ac00 0x7fd9d406d0c0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd9f415ab60 0x7fd9f40a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd9f415a780 0x7fd9f40af610 unknown :-1 s=CLOSED pgs=2508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 --2- 172.21.15.28:0/2831258795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd9f406bb40 0x7fd9f4147e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 >> 172.21.15.28:0/2831258795 conn(0x7fd9f40b0270 msgr2=0x7fd9f40b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 shutdown_connections 2024-09-13T17:23:17.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.662+0000 7fd9f9600640 1 -- 172.21.15.28:0/2831258795 wait complete. 2024-09-13T17:23:17.679 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:23:17.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.836+0000 7f72dd000640 1 Processor -- start 2024-09-13T17:23:17.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.836+0000 7f72dd000640 1 -- start start 2024-09-13T17:23:17.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72dd000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f72d8070430 0x7f72d815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72dd000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d815ab60 con 0x7f72d8159770 2024-09-13T17:23:17.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72dd000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72d806f920 con 0x7f72d806f3f0 2024-09-13T17:23:17.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72dd000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72d815b290 con 0x7f72d8070430 2024-09-13T17:23:17.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72d5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f72d8159770 0x7f72d806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55902/0 (socket says 172.21.15.28:55902) 2024-09-13T17:23:17.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72d5e00640 1 -- 172.21.15.28:0/2085323819 learned_addr learned my addr 172.21.15.28:0/2085323819 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72cd400640 1 --2- 172.21.15.28:0/2085323819 >> v2:172.21.15.189:3300/0 conn(0x7f72d8070430 0x7f72d815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72d5400640 1 -- 172.21.15.28:0/2085323819 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3443747323 0 0) 0x7f72c8001110 con 0x7f72d8159770 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.838+0000 7f72d5400640 1 -- 172.21.15.28:0/2085323819 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72bc003520 con 0x7f72d8159770 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.839+0000 7f72cd400640 1 -- 172.21.15.28:0/2085323819 >> v1:172.21.15.189:6789/0 conn(0x7f72d806f3f0 legacy=0x7f72d815c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.839+0000 7f72cd400640 1 -- 172.21.15.28:0/2085323819 >> v1:172.21.15.28:6789/0 conn(0x7f72d8159770 legacy=0x7f72d806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.839+0000 7f72cd400640 1 -- 172.21.15.28:0/2085323819 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d815b570 con 0x7f72d8070430 2024-09-13T17:23:17.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72cd400640 1 --2- 172.21.15.28:0/2085323819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8070430 0x7f72d815bd70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f72c00097a0 tx=0x7f72c002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f42242c61a3a944 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72d5400640 1 -- 172.21.15.28:0/2085323819 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0032440 con 0x7f72d8070430 2024-09-13T17:23:17.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72d5400640 1 -- 172.21.15.28:0/2085323819 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c00325a0 con 0x7f72d8070430 2024-09-13T17:23:17.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72d5400640 1 -- 172.21.15.28:0/2085323819 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c0032930 con 0x7f72d8070430 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72dd000640 1 -- 172.21.15.28:0/2085323819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8070430 msgr2=0x7f72d815bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.840+0000 7f72dd000640 1 --2- 172.21.15.28:0/2085323819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8070430 0x7f72d815bd70 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f72c00097a0 tx=0x7f72c002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.841+0000 7f72dd000640 1 -- 172.21.15.28:0/2085323819 shutdown_connections 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.841+0000 7f72dd000640 1 --2- 172.21.15.28:0/2085323819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8070430 0x7f72d815bd70 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.841+0000 7f72dd000640 1 -- 172.21.15.28:0/2085323819 >> 172.21.15.28:0/2085323819 conn(0x7f72d80b0270 msgr2=0x7f72d80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.841+0000 7f72dd000640 1 -- 172.21.15.28:0/2085323819 shutdown_connections 2024-09-13T17:23:17.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.841+0000 7f72dd000640 1 -- 172.21.15.28:0/2085323819 wait complete. 2024-09-13T17:23:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.842+0000 7f72dd000640 1 Processor -- start 2024-09-13T17:23:17.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.842+0000 7f72dd000640 1 -- start start 2024-09-13T17:23:17.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d8070430 0x7f72d81e1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8159770 0x7f72d81f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81e1a40 con 0x7f72d806f3f0 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81f6850 con 0x7f72d8159770 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.843+0000 7f72dd000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81f6b30 con 0x7f72d8070430 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51978/0 (socket says 172.21.15.28:51978) 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 -- 172.21.15.28:0/1198290816 learned_addr learned my addr 172.21.15.28:0/1198290816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72cd400640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d8070430 0x7f72d81e1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d6800640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8159770 0x7f72d81f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 -- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d8070430 msgr2=0x7f72d81e1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d8070430 0x7f72d81e1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 -- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8159770 msgr2=0x7f72d81f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:17.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8159770 0x7f72d81f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 -- 172.21.15.28:0/1198290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c0009400 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.844+0000 7f72d5e00640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f72c800c420 tx=0x7f72c800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.845+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c8005ce0 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.845+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72c8005e40 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.845+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d81f6e10 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.845+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72c80072f0 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.845+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d81f2c30 con 0x7f72d806f3f0 2024-09-13T17:23:17.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.846+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d81f7330 con 0x7f72d806f3f0 2024-09-13T17:23:17.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.847+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f72c8007510 con 0x7f72d806f3f0 2024-09-13T17:23:17.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.848+0000 7f72cea00640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 0x7f72b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:17.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.848+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f72c8088890 con 0x7f72d806f3f0 2024-09-13T17:23:17.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.849+0000 7f72cd400640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 0x7f72b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:17.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.850+0000 7f72cd400640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 0x7f72b406d0c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f72c0034000 tx=0x7f72c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:17.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:17.852+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f72c80539f0 con 0x7f72d806f3f0 2024-09-13T17:23:18.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.030+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72d806fd20 con 0x7f72d806f3f0 2024-09-13T17:23:18.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.031+0000 7f72cea00640 1 -- 172.21.15.28:0/1198290816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v207) v1 ==== 104+0+11 (secure 0 0 0) 0x7f72c80170a0 con 0x7f72d806f3f0 2024-09-13T17:23:18.032 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:23:18.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 msgr2=0x7f72b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 0x7f72b406d0c0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f72c0034000 tx=0x7f72c002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 msgr2=0x7f72d8155c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 secure :-1 s=READY pgs=2510 cs=0 l=1 rev1=1 crypto rx=0x7f72c800c420 tx=0x7f72c800c8e0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 shutdown_connections 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f72b406ac00 0x7f72b406d0c0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72d8159770 0x7f72d81f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f72d8070430 0x7f72d81e1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 --2- 172.21.15.28:0/1198290816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72d806f3f0 0x7f72d8155c60 unknown :-1 s=CLOSED pgs=2510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 >> 172.21.15.28:0/1198290816 conn(0x7f72d80b0270 msgr2=0x7f72d806c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 shutdown_connections 2024-09-13T17:23:18.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.033+0000 7f72dd000640 1 -- 172.21.15.28:0/1198290816 wait complete. 2024-09-13T17:23:18.050 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:23:18.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.207+0000 7f5ce6c00640 1 Processor -- start 2024-09-13T17:23:18.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.207+0000 7f5ce6c00640 1 -- start start 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce6c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ce006f3f0 0x7f5ce015c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce6c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce015ab60 con 0x7f5ce0070430 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce6c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ce006f920 con 0x7f5ce0159770 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce6c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ce015b290 con 0x7f5ce006f3f0 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce4c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5ce006f3f0 0x7f5ce015c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5cd7400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ce0070430 0x7f5ce015bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55916/0 (socket says 172.21.15.28:55916) 2024-09-13T17:23:18.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5cd7400640 1 -- 172.21.15.28:0/361338118 learned_addr learned my addr 172.21.15.28:0/361338118 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce4c00640 1 -- 172.21.15.28:0/361338118 >> v1:172.21.15.189:6789/0 conn(0x7f5ce0159770 legacy=0x7f5ce006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce4c00640 1 -- 172.21.15.28:0/361338118 >> v1:172.21.15.28:6789/0 conn(0x7f5ce0070430 legacy=0x7f5ce015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.209+0000 7f5ce4c00640 1 -- 172.21.15.28:0/361338118 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ce015b570 con 0x7f5ce006f3f0 2024-09-13T17:23:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.210+0000 7f5ce4c00640 1 --2- 172.21.15.28:0/361338118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce006f3f0 0x7f5ce015c480 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009d90 tx=0x7f5cd802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285de61837e377a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5cdf400640 1 -- 172.21.15.28:0/361338118 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8031600 con 0x7f5ce006f3f0 2024-09-13T17:23:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5cdf400640 1 -- 172.21.15.28:0/361338118 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cd802d050 con 0x7f5ce006f3f0 2024-09-13T17:23:18.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5cdf400640 1 -- 172.21.15.28:0/361338118 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd8031980 con 0x7f5ce006f3f0 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 -- 172.21.15.28:0/361338118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce006f3f0 msgr2=0x7f5ce015c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/361338118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce006f3f0 0x7f5ce015c480 secure :-1 s=READY pgs=2512 cs=0 l=1 rev1=1 crypto rx=0x7f5cd8009d90 tx=0x7f5cd802a610 comp rx=0 tx=0).stop 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 -- 172.21.15.28:0/361338118 shutdown_connections 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/361338118 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce006f3f0 0x7f5ce015c480 unknown :-1 s=CLOSED pgs=2512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 -- 172.21.15.28:0/361338118 >> 172.21.15.28:0/361338118 conn(0x7f5ce00b0270 msgr2=0x7f5ce00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 -- 172.21.15.28:0/361338118 shutdown_connections 2024-09-13T17:23:18.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.211+0000 7f5ce6c00640 1 -- 172.21.15.28:0/361338118 wait complete. 2024-09-13T17:23:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.213+0000 7f5ce6c00640 1 Processor -- start 2024-09-13T17:23:18.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.213+0000 7f5ce6c00640 1 -- start start 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ce006f3f0 0x7f5ce014e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 0x7f5ce00b2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce014e7b0 con 0x7f5ce0070430 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce0152b40 con 0x7f5ce0159770 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5ce6c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce01f6330 con 0x7f5ce006f3f0 2024-09-13T17:23:18.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5cd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5cd7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51988/0 (socket says 172.21.15.28:51988) 2024-09-13T17:23:18.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.214+0000 7f5cd7400640 1 -- 172.21.15.28:0/681034963 learned_addr learned my addr 172.21.15.28:0/681034963 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5ce4c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 0x7f5ce00b2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cdfe00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ce006f3f0 0x7f5ce014e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 -- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ce006f3f0 msgr2=0x7f5ce014e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ce006f3f0 0x7f5ce014e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 -- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 msgr2=0x7f5ce00b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 0x7f5ce00b2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 -- 172.21.15.28:0/681034963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd8009a70 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5ce4c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 0x7f5ce00b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cd7400640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c4a0 tx=0x7f5cc800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8005ce0 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc8005e40 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc8007270 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce01f6610 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.215+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce01f6bd0 con 0x7f5ce0070430 2024-09-13T17:23:18.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.216+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ce006fd20 con 0x7f5ce0070430 2024-09-13T17:23:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.217+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5cc8007490 con 0x7f5ce0070430 2024-09-13T17:23:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.218+0000 7f5cdcc00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 0x7f5cbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.218+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f5cc8088700 con 0x7f5ce0070430 2024-09-13T17:23:18.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.219+0000 7f5cdfe00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 0x7f5cbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.219+0000 7f5cdfe00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 0x7f5cbc06d0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0006fd0 tx=0x7f5cd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.222+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5cc8053870 con 0x7f5ce0070430 2024-09-13T17:23:18.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.399+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ce0157d50 con 0x7f5ce0070430 2024-09-13T17:23:18.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.400+0000 7f5cdcc00640 1 -- 172.21.15.28:0/681034963 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v207) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5cc8056e90 con 0x7f5ce0070430 2024-09-13T17:23:18.401 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:23:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 msgr2=0x7f5cbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 0x7f5cbc06d0c0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5cd0006fd0 tx=0x7f5cd0008040 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 msgr2=0x7f5ce0152600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7f5cc800c4a0 tx=0x7f5cc800c960 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 shutdown_connections 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5cbc06ac00 0x7f5cbc06d0c0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ce0159770 0x7f5ce00b2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ce0070430 0x7f5ce0152600 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 --2- 172.21.15.28:0/681034963 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ce006f3f0 0x7f5ce014e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 >> 172.21.15.28:0/681034963 conn(0x7f5ce00b0270 msgr2=0x7f5ce00ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 shutdown_connections 2024-09-13T17:23:18.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.402+0000 7f5ce6c00640 1 -- 172.21.15.28:0/681034963 wait complete. 2024-09-13T17:23:18.420 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:23:18.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.574+0000 7f8ba9600640 1 Processor -- start 2024-09-13T17:23:18.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.575+0000 7f8ba9600640 1 -- start start 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba9600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8ba40c40a0 0x7f8ba40b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba9600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ba40b1ba0 0x7f8ba415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba9600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40b2090 con 0x7f8ba415b0b0 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba9600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40b2370 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba9600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40b2650 con 0x7f8ba40c40a0 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba8c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8ba415b0b0 0x7f8ba406c370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55924/0 (socket says 172.21.15.28:55924) 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8b9b400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f8ba40b1ba0 0x7f8ba415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba8c00640 1 -- 172.21.15.28:0/2710728551 learned_addr learned my addr 172.21.15.28:0/2710728551 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:18.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.576+0000 7f8ba3e00640 1 --2- 172.21.15.28:0/2710728551 >> v2:172.21.15.189:3300/0 conn(0x7f8ba40c40a0 0x7f8ba40b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.577+0000 7f8b9b400640 1 -- 172.21.15.28:0/2710728551 >> v2:172.21.15.189:3300/0 conn(0x7f8ba40c40a0 msgr2=0x7f8ba40b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.577+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2710728551 >> v2:172.21.15.189:3300/0 conn(0x7f8ba40c40a0 0x7f8ba40b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.577+0000 7f8b9b400640 1 -- 172.21.15.28:0/2710728551 >> v1:172.21.15.28:6789/0 conn(0x7f8ba415b0b0 legacy=0x7f8ba406c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.577+0000 7f8b9b400640 1 -- 172.21.15.28:0/2710728551 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40b2930 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.577+0000 7f8ba3400640 1 -- 172.21.15.28:0/2710728551 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211345726 0 0) 0x7f8ba40b2090 con 0x7f8ba415b0b0 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2710728551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba415ab70 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40ce360 tx=0x7f8b8c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bed26cd0d6853857 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8ba3400640 1 -- 172.21.15.28:0/2710728551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c032440 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8ba3400640 1 -- 172.21.15.28:0/2710728551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8c0325a0 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8ba3400640 1 -- 172.21.15.28:0/2710728551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8c032930 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8ba9600640 1 -- 172.21.15.28:0/2710728551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 msgr2=0x7f8ba415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.578+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2710728551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba415ab70 secure :-1 s=READY pgs=2515 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40ce360 tx=0x7f8b8c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 -- 172.21.15.28:0/2710728551 shutdown_connections 2024-09-13T17:23:18.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2710728551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba415ab70 unknown :-1 s=CLOSED pgs=2515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2710728551 >> v2:172.21.15.189:3300/0 conn(0x7f8ba40c40a0 0x7f8ba40b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 -- 172.21.15.28:0/2710728551 >> 172.21.15.28:0/2710728551 conn(0x7f8ba4073470 msgr2=0x7f8ba4069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 -- 172.21.15.28:0/2710728551 shutdown_connections 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.579+0000 7f8ba9600640 1 -- 172.21.15.28:0/2710728551 wait complete. 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.580+0000 7f8ba9600640 1 Processor -- start 2024-09-13T17:23:18.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.580+0000 7f8ba9600640 1 -- start start 2024-09-13T17:23:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 0x7f8ba40c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 0x7f8ba4136f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba40add20 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba4135620 con 0x7f8ba415b0b0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.581+0000 7f8ba9600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba4137fb0 con 0x7f8ba40c40a0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52006/0 (socket says 172.21.15.28:52006) 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 -- 172.21.15.28:0/2350134583 learned_addr learned my addr 172.21.15.28:0/2350134583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba8c00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 0x7f8ba4136f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 -- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 msgr2=0x7f8ba40c82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 0x7f8ba40c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 0x7f8ba40c82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 -- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 msgr2=0x7f8ba4136f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 0x7f8ba4136f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 -- 172.21.15.28:0/2350134583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c009400 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba8c00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 0x7f8ba4136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:18.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 0x7f8ba40c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.582+0000 7f8ba3e00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400e9e0 tx=0x7f8b9400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.583+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b94009670 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.583+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9400c9d0 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.583+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba4138290 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.583+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9400cd80 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.583+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba40ad940 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.584+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba40adf90 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.585+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8b94013470 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.586+0000 7f8ba0c00640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 0x7f8b8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:18.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.586+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8b94089690 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.586+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 0x7f8b8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:18.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.587+0000 7f8b9b400640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 0x7f8b8006d0c0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034000 tx=0x7f8b8c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:18.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.590+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8b940547d0 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.779+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8ba415ab70 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.779+0000 7f8ba0c00640 1 -- 172.21.15.28:0/2350134583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v207) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8b94057df0 con 0x7f8ba40b1ba0 2024-09-13T17:23:18.781 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 msgr2=0x7f8b8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 0x7f8b8006d0c0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c034000 tx=0x7f8b8c02e000 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 msgr2=0x7f8ba40c7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f8b9400e9e0 tx=0x7f8b9400eea0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 shutdown_connections 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8b8006ac00 0x7f8b8006d0c0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba415b0b0 0x7f8ba4136f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8ba40c40a0 0x7f8ba40c82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 --2- 172.21.15.28:0/2350134583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ba40b1ba0 0x7f8ba40c7d60 unknown :-1 s=CLOSED pgs=2516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.781+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 >> 172.21.15.28:0/2350134583 conn(0x7f8ba4073470 msgr2=0x7f8ba40b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:18.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.782+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 shutdown_connections 2024-09-13T17:23:18.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:18.782+0000 7f8ba9600640 1 -- 172.21.15.28:0/2350134583 wait complete. 2024-09-13T17:23:18.801 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:23:18.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:18.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:18.883 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:23:18.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:18.883 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:23:18.933 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:18.933 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:18.970 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:18.970 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:18.971 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:19.010 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:23:19.010 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:19.010 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:19.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:19.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.205+0000 7f37e1a00640 1 Processor -- start 2024-09-13T17:23:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.205+0000 7f37e1a00640 1 -- start start 2024-09-13T17:23:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37e1a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 0x7f37dc15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37e1a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f37dc06c2d0 0x7f37dc06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37e1a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc06cc70 con 0x7f37dc1592a0 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37e1a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37dc15b0b0 con 0x7f37dc06f970 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37e1a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37dc159d90 con 0x7f37dc06c2d0 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.206+0000 7f37d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 0x7f37dc15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37d3e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 0x7f37dc15ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52016/0 (socket says 172.21.15.28:52016) 2024-09-13T17:23:19.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37d3e00640 1 -- 172.21.15.28:0/3017137006 learned_addr learned my addr 172.21.15.28:0/3017137006 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37db400640 1 --2- 172.21.15.28:0/3017137006 >> v2:172.21.15.189:3300/0 conn(0x7f37dc06c2d0 0x7f37dc06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37db400640 1 -- 172.21.15.28:0/3017137006 >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 msgr2=0x7f37dc15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37db400640 1 --2- 172.21.15.28:0/3017137006 >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 0x7f37dc15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37db400640 1 -- 172.21.15.28:0/3017137006 >> v1:172.21.15.189:6789/0 conn(0x7f37dc1592a0 legacy=0x7f37dc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37db400640 1 -- 172.21.15.28:0/3017137006 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc0b2040 con 0x7f37dc06c2d0 2024-09-13T17:23:19.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.207+0000 7f37da000640 1 -- 172.21.15.28:0/3017137006 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4232792109 0 0) 0x7f37dc06cc70 con 0x7f37dc1592a0 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.208+0000 7f37db400640 1 --2- 172.21.15.28:0/3017137006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc06c730 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b1dd0 tx=0x7f37d402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bd9802819758c7e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37da000640 1 -- 172.21.15.28:0/3017137006 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d402aea0 con 0x7f37dc06c2d0 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37da000640 1 -- 172.21.15.28:0/3017137006 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d402d050 con 0x7f37dc06c2d0 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37da000640 1 -- 172.21.15.28:0/3017137006 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d4007960 con 0x7f37dc06c2d0 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 -- 172.21.15.28:0/3017137006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 msgr2=0x7f37dc06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 --2- 172.21.15.28:0/3017137006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc06c730 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f37dc0b1dd0 tx=0x7f37d402a680 comp rx=0 tx=0).stop 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 -- 172.21.15.28:0/3017137006 shutdown_connections 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 --2- 172.21.15.28:0/3017137006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc06c730 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 --2- 172.21.15.28:0/3017137006 >> v2:172.21.15.28:3300/0 conn(0x7f37dc06f970 0x7f37dc15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 -- 172.21.15.28:0/3017137006 >> 172.21.15.28:0/3017137006 conn(0x7f37dc069e80 msgr2=0x7f37dc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 -- 172.21.15.28:0/3017137006 shutdown_connections 2024-09-13T17:23:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.209+0000 7f37e1a00640 1 -- 172.21.15.28:0/3017137006 wait complete. 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.211+0000 7f37e1a00640 1 Processor -- start 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.211+0000 7f37e1a00640 1 -- start start 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc0c3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37dc06f970 0x7f37dc14f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc0cce30 con 0x7f37dc1592a0 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc0c56c0 con 0x7f37dc06c2d0 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37e1a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37dc0c4620 con 0x7f37dc06f970 2024-09-13T17:23:19.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52020/0 (socket says 172.21.15.28:52020) 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37db400640 1 -- 172.21.15.28:0/458550811 learned_addr learned my addr 172.21.15.28:0/458550811 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37daa00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc0c3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.212+0000 7f37d3e00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37dc06f970 0x7f37dc14f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 -- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37dc06f970 msgr2=0x7f37dc14f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37dc06f970 0x7f37dc14f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 -- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 msgr2=0x7f37dc0c3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc0c3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 -- 172.21.15.28:0/458550811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d4009a70 con 0x7f37dc1592a0 2024-09-13T17:23:19.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37db400640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f37d402ac70 tx=0x7f37d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d403c830 con 0x7f37dc1592a0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d403c990 con 0x7f37dc1592a0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc0c4900 con 0x7f37dc1592a0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.213+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d403cd00 con 0x7f37dc1592a0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.214+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc0c4ec0 con 0x7f37dc1592a0 2024-09-13T17:23:19.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.215+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc0b3740 con 0x7f37dc1592a0 2024-09-13T17:23:19.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.215+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f37d4031400 con 0x7f37dc1592a0 2024-09-13T17:23:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.216+0000 7f37d2a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 0x7f37b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.216+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f37d40aae70 con 0x7f37dc1592a0 2024-09-13T17:23:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.217+0000 7f37daa00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 0x7f37b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.217+0000 7f37daa00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 0x7f37b806d0c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.221+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f37d4075fe0 con 0x7f37dc1592a0 2024-09-13T17:23:19.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.437+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37dc157390 con 0x7f37dc1592a0 2024-09-13T17:23:19.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.438+0000 7f37d2a00640 1 -- 172.21.15.28:0/458550811 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7f37d403ce60 con 0x7f37dc1592a0 2024-09-13T17:23:19.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:19.439 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:23:19.440 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:23:19.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 msgr2=0x7f37b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 0x7f37b806d0c0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f37cc000f60 tx=0x7f37cc009040 comp rx=0 tx=0).stop 2024-09-13T17:23:19.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 msgr2=0x7f37dc0cc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f37d402ac70 tx=0x7f37d4004a40 comp rx=0 tx=0).stop 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 shutdown_connections 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f37b806ac00 0x7f37b806d0c0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f37dc1592a0 0x7f37dc0cc8f0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f37dc06f970 0x7f37dc14f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 --2- 172.21.15.28:0/458550811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37dc06c2d0 0x7f37dc0c3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 >> 172.21.15.28:0/458550811 conn(0x7f37dc069e80 msgr2=0x7f37dc159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 shutdown_connections 2024-09-13T17:23:19.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.440+0000 7f37e1a00640 1 -- 172.21.15.28:0/458550811 wait complete. 2024-09-13T17:23:19.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.610+0000 7f7e11c00640 1 Processor -- start 2024-09-13T17:23:19.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.610+0000 7f7e11c00640 1 -- start start 2024-09-13T17:23:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e11c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e0c0cabc0 0x7f7e0c15e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e11c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e0c15e100 con 0x7f7e0c15edb0 2024-09-13T17:23:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e11c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0c0adb00 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e11c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c15e3e0 con 0x7f7e0c15a540 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e0b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e0c15edb0 0x7f7e0c070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55938/0 (socket says 172.21.15.28:55938) 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.611+0000 7f7e0b400640 1 -- 172.21.15.28:0/19362704 learned_addr learned my addr 172.21.15.28:0/19362704 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e03e00640 1 --2- 172.21.15.28:0/19362704 >> v2:172.21.15.189:3300/0 conn(0x7f7e0c0cabc0 0x7f7e0c15e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e0aa00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e0a000640 1 -- 172.21.15.28:0/19362704 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349334042 0 0) 0x7f7e0c15e100 con 0x7f7e0c15edb0 2024-09-13T17:23:19.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e03e00640 1 -- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 msgr2=0x7f7e0c06c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e03e00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c06c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e03e00640 1 -- 172.21.15.28:0/19362704 >> v1:172.21.15.28:6789/0 conn(0x7f7e0c15edb0 legacy=0x7f7e0c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e03e00640 1 -- 172.21.15.28:0/19362704 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0c06ce80 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.612+0000 7f7e0aa00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.613+0000 7f7e03e00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c15e870 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7df40097a0 tx=0x7f7df402a0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29edaf8cbf8a526b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e0a000640 1 -- 172.21.15.28:0/19362704 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df402ae40 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e0a000640 1 -- 172.21.15.28:0/19362704 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df4032440 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e0a000640 1 -- 172.21.15.28:0/19362704 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df40327d0 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 -- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 msgr2=0x7f7e0c15e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c15e870 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f7df40097a0 tx=0x7f7df402a0f0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 -- 172.21.15.28:0/19362704 shutdown_connections 2024-09-13T17:23:19.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c15e870 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 --2- 172.21.15.28:0/19362704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c06c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 -- 172.21.15.28:0/19362704 >> 172.21.15.28:0/19362704 conn(0x7f7e0c069e80 msgr2=0x7f7e0c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 -- 172.21.15.28:0/19362704 shutdown_connections 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.614+0000 7f7e11c00640 1 -- 172.21.15.28:0/19362704 wait complete. 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.616+0000 7f7e11c00640 1 Processor -- start 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.616+0000 7f7e11c00640 1 -- start start 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c13a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c156710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c13aee0 con 0x7f7e0c0cce30 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c13ddf0 con 0x7f7e0c0cabc0 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e11c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0c13b130 con 0x7f7e0c15a540 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e0b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52024/0 (socket says 172.21.15.28:52024) 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e0b400640 1 -- 172.21.15.28:0/894342014 learned_addr learned my addr 172.21.15.28:0/894342014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e03e00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c156710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.617+0000 7f7e0aa00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c13a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 -- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 msgr2=0x7f7e0c156710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c156710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 -- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 msgr2=0x7f7e0c13a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c13a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 -- 172.21.15.28:0/894342014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dfc009400 con 0x7f7e0c0cce30 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e0b400640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c7a0 tx=0x7f7e0400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0400a640 con 0x7f7e0c0cce30 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df4009400 con 0x7f7e0c0cce30 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0400a7a0 con 0x7f7e0c0cce30 2024-09-13T17:23:19.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e0400abd0 con 0x7f7e0c0cce30 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.618+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0c13b6f0 con 0x7f7e0c0cce30 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.620+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0c15e870 con 0x7f7e0c0cce30 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.620+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e0400ad30 con 0x7f7e0c0cce30 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.621+0000 7f7e02a00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 0x7f7de806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.621+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f7e04088880 con 0x7f7e0c0cce30 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.622+0000 7f7e0aa00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 0x7f7de806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.622+0000 7f7e0aa00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 0x7f7de806d0f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009cd0 tx=0x7f7dfc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.625+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e04053960 con 0x7f7e0c0cce30 2024-09-13T17:23:19.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.804+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e0c06c100 con 0x7f7e0c0cce30 2024-09-13T17:23:19.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.805+0000 7f7e02a00640 1 -- 172.21.15.28:0/894342014 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16819 (secure 0 0 0) 0x7f7e04056f80 con 0x7f7e0c0cce30 2024-09-13T17:23:19.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:19.806 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:55.171778+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.19","mappings":[{"from":6,"to":0}]},{"pgid":"29.39","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 msgr2=0x7f7de806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 0x7f7de806d0f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f7dfc009cd0 tx=0x7f7dfc009190 comp rx=0 tx=0).stop 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 msgr2=0x7f7e0c13cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400c7a0 tx=0x7f7e0400cc60 comp rx=0 tx=0).stop 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 shutdown_connections 2024-09-13T17:23:19.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7de806ac30 0x7f7de806d0f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e0c0cce30 0x7f7e0c13cf20 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e0c15a540 0x7f7e0c156710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 --2- 172.21.15.28:0/894342014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e0c0cabc0 0x7f7e0c13a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 >> 172.21.15.28:0/894342014 conn(0x7f7e0c069e80 msgr2=0x7f7e0c15b5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 shutdown_connections 2024-09-13T17:23:19.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.807+0000 7f7e11c00640 1 -- 172.21.15.28:0/894342014 wait complete. 2024-09-13T17:23:19.824 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:22:33.628286+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '204', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:22:34.718902+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '205', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:19.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:19.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.982+0000 7f2d5d400640 1 Processor -- start 2024-09-13T17:23:19.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.983+0000 7f2d5d400640 1 -- start start 2024-09-13T17:23:19.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2d580cde30 0x7f2d58071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d58070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d58071100 0x7f2d5815cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d580715b0 con 0x7f2d58071100 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d580ce210 con 0x7f2d580cde30 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d58159e50 con 0x7f2d58071f60 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d58071100 0x7f2d5815cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5ca00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2d58071100 0x7f2d5815cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52030/0 (socket says 172.21.15.28:52030) 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d5ca00640 1 -- 172.21.15.28:0/3027880326 learned_addr learned my addr 172.21.15.28:0/3027880326 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d4f400640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d58070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.984+0000 7f2d57e00640 1 --2- 172.21.15.28:0/3027880326 >> v2:172.21.15.189:3300/0 conn(0x7f2d580cde30 0x7f2d58071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.985+0000 7f2d5ca00640 1 -- 172.21.15.28:0/3027880326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 msgr2=0x7f2d58070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.985+0000 7f2d5ca00640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d58070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.985+0000 7f2d5ca00640 1 -- 172.21.15.28:0/3027880326 >> v2:172.21.15.189:3300/0 conn(0x7f2d580cde30 msgr2=0x7f2d58071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.985+0000 7f2d5ca00640 1 --2- 172.21.15.28:0/3027880326 >> v2:172.21.15.189:3300/0 conn(0x7f2d580cde30 0x7f2d58071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.985+0000 7f2d5ca00640 1 -- 172.21.15.28:0/3027880326 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5815a130 con 0x7f2d58071100 2024-09-13T17:23:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d5ca00640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d5815cff0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806ad40 tx=0x7f2d5002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47fda7bf1c2a8aa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d57400640 1 -- 172.21.15.28:0/3027880326 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50031600 con 0x7f2d58071100 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d57400640 1 -- 172.21.15.28:0/3027880326 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d5002d050 con 0x7f2d58071100 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d57400640 1 -- 172.21.15.28:0/3027880326 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d50031980 con 0x7f2d58071100 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d5d400640 1 -- 172.21.15.28:0/3027880326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 msgr2=0x7f2d5815cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d5d400640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d5815cff0 secure :-1 s=READY pgs=2520 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806ad40 tx=0x7f2d5002a6a0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.986+0000 7f2d5d400640 1 -- 172.21.15.28:0/3027880326 shutdown_connections 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d5815cff0 unknown :-1 s=CLOSED pgs=2520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 --2- 172.21.15.28:0/3027880326 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d58070bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 --2- 172.21.15.28:0/3027880326 >> v2:172.21.15.189:3300/0 conn(0x7f2d580cde30 0x7f2d58071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 -- 172.21.15.28:0/3027880326 >> 172.21.15.28:0/3027880326 conn(0x7f2d58069e80 msgr2=0x7f2d5806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 -- 172.21.15.28:0/3027880326 shutdown_connections 2024-09-13T17:23:19.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.987+0000 7f2d5d400640 1 -- 172.21.15.28:0/3027880326 wait complete. 2024-09-13T17:23:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.988+0000 7f2d5d400640 1 Processor -- start 2024-09-13T17:23:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.988+0000 7f2d5d400640 1 -- start start 2024-09-13T17:23:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d581474a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d580cde30 0x7f2d581542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d581479e0 con 0x7f2d58071100 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580af680 con 0x7f2d580cde30 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.989+0000 7f2d5d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d580af960 con 0x7f2d58071f60 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52038/0 (socket says 172.21.15.28:52038) 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 -- 172.21.15.28:0/1368459743 learned_addr learned my addr 172.21.15.28:0/1368459743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d4f400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d581474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d5ca00640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d580cde30 0x7f2d581542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 -- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 msgr2=0x7f2d581474a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d581474a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 -- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d580cde30 msgr2=0x7f2d581542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d580cde30 0x7f2d581542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 -- 172.21.15.28:0/1368459743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d50009b30 con 0x7f2d58071100 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.990+0000 7f2d57e00640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800e9b0 tx=0x7f2d4800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.991+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48009670 con 0x7f2d58071100 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.991+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d4800c9a0 con 0x7f2d58071100 2024-09-13T17:23:19.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.991+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4800cd50 con 0x7f2d58071100 2024-09-13T17:23:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.991+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d580afc40 con 0x7f2d58071100 2024-09-13T17:23:19.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.991+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d580b0200 con 0x7f2d58071100 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.992+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d58070bc0 con 0x7f2d58071100 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.993+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2d48013470 con 0x7f2d58071100 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.994+0000 7f2d54c00640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 0x7f2d3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.994+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2d480895e0 con 0x7f2d58071100 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.994+0000 7f2d4f400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 0x7f2d3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:19.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.995+0000 7f2d4f400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 0x7f2d3406d0c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2d581f5ad0 tx=0x7f2d40009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:19.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:19.998+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2d480547d0 con 0x7f2d58071100 2024-09-13T17:23:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.174+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d5806f700 con 0x7f2d58071100 2024-09-13T17:23:20.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.174+0000 7f2d54c00640 1 -- 172.21.15.28:0/1368459743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v207) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d48057df0 con 0x7f2d58071100 2024-09-13T17:23:20.175 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 msgr2=0x7f2d3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 0x7f2d3406d0c0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2d581f5ad0 tx=0x7f2d40009190 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 msgr2=0x7f2d58148150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800e9b0 tx=0x7f2d4800ee70 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 shutdown_connections 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2d3406ac00 0x7f2d3406d0c0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d580cde30 0x7f2d581542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2d58071f60 0x7f2d581474a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 --2- 172.21.15.28:0/1368459743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2d58071100 0x7f2d58148150 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 >> 172.21.15.28:0/1368459743 conn(0x7f2d58069e80 msgr2=0x7f2d5806fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 shutdown_connections 2024-09-13T17:23:20.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.176+0000 7f2d5d400640 1 -- 172.21.15.28:0/1368459743 wait complete. 2024-09-13T17:23:20.193 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:23:20.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.350+0000 7f2a59a00640 1 Processor -- start 2024-09-13T17:23:20.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.350+0000 7f2a59a00640 1 -- start start 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a59a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a540ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a59a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2a540b2400 0x7f2a5406bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a59a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a54070470 con 0x7f2a540ad940 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a59a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a540ace30 con 0x7f2a540b2400 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a59a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a540ad110 con 0x7f2a5415c9a0 2024-09-13T17:23:20.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a4a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2a540ad940 0x7f2a540b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55954/0 (socket says 172.21.15.28:55954) 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a4a000640 1 -- 172.21.15.28:0/3950070793 learned_addr learned my addr 172.21.15.28:0/3950070793 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a53400640 1 --2- 172.21.15.28:0/3950070793 >> v2:172.21.15.189:3300/0 conn(0x7f2a540b2400 0x7f2a5406bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a52a00640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a540ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691652153 0 0) 0x7f2a54070470 con 0x7f2a540ad940 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.352+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a38003520 con 0x7f2a540ad940 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.353+0000 7f2a53400640 1 -- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 msgr2=0x7f2a540ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.353+0000 7f2a53400640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a540ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.353+0000 7f2a53400640 1 -- 172.21.15.28:0/3950070793 >> v1:172.21.15.28:6789/0 conn(0x7f2a540ad940 legacy=0x7f2a540b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.353+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745305212 0 0) 0x7f2a38003520 con 0x7f2a540ad940 2024-09-13T17:23:20.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.353+0000 7f2a53400640 1 -- 172.21.15.28:0/3950070793 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5406b170 con 0x7f2a540b2400 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.354+0000 7f2a53400640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540b2400 0x7f2a5406bc50 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009d90 tx=0x7f2a4c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d02de51529815d19 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.354+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c001670 con 0x7f2a540b2400 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.354+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c02d050 con 0x7f2a540b2400 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.354+0000 7f2a52000640 1 -- 172.21.15.28:0/3950070793 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c007760 con 0x7f2a540b2400 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 -- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540b2400 msgr2=0x7f2a5406bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540b2400 0x7f2a5406bc50 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c009d90 tx=0x7f2a4c02a680 comp rx=0 tx=0).stop 2024-09-13T17:23:20.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 -- 172.21.15.28:0/3950070793 shutdown_connections 2024-09-13T17:23:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540b2400 0x7f2a5406bc50 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 --2- 172.21.15.28:0/3950070793 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a540ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 -- 172.21.15.28:0/3950070793 >> 172.21.15.28:0/3950070793 conn(0x7f2a540b0270 msgr2=0x7f2a540b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 -- 172.21.15.28:0/3950070793 shutdown_connections 2024-09-13T17:23:20.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.355+0000 7f2a59a00640 1 -- 172.21.15.28:0/3950070793 wait complete. 2024-09-13T17:23:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.357+0000 7f2a59a00640 1 Processor -- start 2024-09-13T17:23:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.357+0000 7f2a59a00640 1 -- start start 2024-09-13T17:23:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540ad940 0x7f2a54152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a54153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a541f1f10 con 0x7f2a540b2400 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5406c6c0 con 0x7f2a540ad940 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a59a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a541f21e0 con 0x7f2a5415c9a0 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52048/0 (socket says 172.21.15.28:52048) 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 -- 172.21.15.28:0/1464373788 learned_addr learned my addr 172.21.15.28:0/1464373788 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a52a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540ad940 0x7f2a54152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a53400640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a54153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 -- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 msgr2=0x7f2a54153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a54153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 -- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540ad940 msgr2=0x7f2a54152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540ad940 0x7f2a54152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.358+0000 7f2a4a000640 1 -- 172.21.15.28:0/1464373788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c009a70 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a4a000640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0011e0 tx=0x7f2a3c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c005ad0 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c005c30 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c007290 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a54070930 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.359+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a541f27a0 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.360+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a540676b0 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.361+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2a3c0073f0 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.362+0000 7f2a4b400640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 0x7f2a3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.362+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f2a3c088b00 con 0x7f2a540b2400 2024-09-13T17:23:20.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.363+0000 7f2a52a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 0x7f2a3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.363+0000 7f2a52a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 0x7f2a3006d0c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2a5415a2c0 tx=0x7f2a44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.366+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2a3c08ae60 con 0x7f2a540b2400 2024-09-13T17:23:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.546+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a541f2050 con 0x7f2a540b2400 2024-09-13T17:23:20.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.546+0000 7f2a4b400640 1 -- 172.21.15.28:0/1464373788 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v207) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2a3c053c60 con 0x7f2a540b2400 2024-09-13T17:23:20.548 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 msgr2=0x7f2a3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 0x7f2a3006d0c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2a5415a2c0 tx=0x7f2a44009190 comp rx=0 tx=0).stop 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 msgr2=0x7f2a5406bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 secure :-1 s=READY pgs=2523 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0011e0 tx=0x7f2a3c00cc00 comp rx=0 tx=0).stop 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 shutdown_connections 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2a3006ac00 0x7f2a3006d0c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2a5415c9a0 0x7f2a54153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a540b2400 0x7f2a5406bce0 unknown :-1 s=CLOSED pgs=2523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 --2- 172.21.15.28:0/1464373788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a540ad940 0x7f2a54152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 >> 172.21.15.28:0/1464373788 conn(0x7f2a540b0270 msgr2=0x7f2a540ae030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 shutdown_connections 2024-09-13T17:23:20.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.549+0000 7f2a59a00640 1 -- 172.21.15.28:0/1464373788 wait complete. 2024-09-13T17:23:20.565 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:23:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.724+0000 7f8393000640 1 Processor -- start 2024-09-13T17:23:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.724+0000 7f8393000640 1 -- start start 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.725+0000 7f8393000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f838c159770 0x7f838c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.725+0000 7f8393000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f838c15cd80 0x7f838c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.725+0000 7f8393000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c06f920 con 0x7f838c06f3f0 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.725+0000 7f8393000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c15b290 con 0x7f838c15cd80 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.725+0000 7f8393000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c15b570 con 0x7f838c159770 2024-09-13T17:23:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f838b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f838c06f3f0 0x7f838c159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55966/0 (socket says 172.21.15.28:55966) 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f838b400640 1 -- 172.21.15.28:0/829192154 learned_addr learned my addr 172.21.15.28:0/829192154 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f8391000640 1 --2- 172.21.15.28:0/829192154 >> v2:172.21.15.28:3300/0 conn(0x7f838c15cd80 0x7f838c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f838be00640 1 --2- 172.21.15.28:0/829192154 >> v2:172.21.15.189:3300/0 conn(0x7f838c159770 0x7f838c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f838aa00640 1 -- 172.21.15.28:0/829192154 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199908807 0 0) 0x7f838c06f920 con 0x7f838c06f3f0 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f838aa00640 1 -- 172.21.15.28:0/829192154 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8370003520 con 0x7f838c06f3f0 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f8391000640 1 -- 172.21.15.28:0/829192154 >> v2:172.21.15.189:3300/0 conn(0x7f838c159770 msgr2=0x7f838c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f8391000640 1 --2- 172.21.15.28:0/829192154 >> v2:172.21.15.189:3300/0 conn(0x7f838c159770 0x7f838c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f8391000640 1 -- 172.21.15.28:0/829192154 >> v1:172.21.15.28:6789/0 conn(0x7f838c06f3f0 legacy=0x7f838c159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.726+0000 7f8391000640 1 -- 172.21.15.28:0/829192154 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c15b850 con 0x7f838c15cd80 2024-09-13T17:23:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.727+0000 7f8391000640 1 --2- 172.21.15.28:0/829192154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c15cd80 0x7f838c070260 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f838c067f80 tx=0x7f838402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=697af13366f8c7d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.727+0000 7f838aa00640 1 -- 172.21.15.28:0/829192154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384007740 con 0x7f838c15cd80 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.727+0000 7f838aa00640 1 -- 172.21.15.28:0/829192154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838402d050 con 0x7f838c15cd80 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.727+0000 7f838aa00640 1 -- 172.21.15.28:0/829192154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384007ac0 con 0x7f838c15cd80 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 -- 172.21.15.28:0/829192154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c15cd80 msgr2=0x7f838c070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 --2- 172.21.15.28:0/829192154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c15cd80 0x7f838c070260 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f838c067f80 tx=0x7f838402a5f0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 -- 172.21.15.28:0/829192154 shutdown_connections 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 --2- 172.21.15.28:0/829192154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c15cd80 0x7f838c070260 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 --2- 172.21.15.28:0/829192154 >> v2:172.21.15.189:3300/0 conn(0x7f838c159770 0x7f838c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 -- 172.21.15.28:0/829192154 >> 172.21.15.28:0/829192154 conn(0x7f838c0b0270 msgr2=0x7f838c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 -- 172.21.15.28:0/829192154 shutdown_connections 2024-09-13T17:23:20.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.728+0000 7f8393000640 1 -- 172.21.15.28:0/829192154 wait complete. 2024-09-13T17:23:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.730+0000 7f8393000640 1 Processor -- start 2024-09-13T17:23:20.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.730+0000 7f8393000640 1 -- start start 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 0x7f838c154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 0x7f838c148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c14e9e0 con 0x7f838c159770 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c14ecc0 con 0x7f838c15cd80 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8393000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c14efa0 con 0x7f838c06f3f0 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52068/0 (socket says 172.21.15.28:52068) 2024-09-13T17:23:20.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 -- 172.21.15.28:0/3998387825 learned_addr learned my addr 172.21.15.28:0/3998387825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 -- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 msgr2=0x7f838c154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8391000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 0x7f838c148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838be00640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 0x7f838c154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 0x7f838c154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 -- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 msgr2=0x7f838c148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 0x7f838c148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838b400640 1 -- 172.21.15.28:0/3998387825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384009a70 con 0x7f838c159770 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f8391000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 0x7f838c148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.731+0000 7f838be00640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 0x7f838c154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f838b400640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f837800c410 tx=0x7f837800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378005ce0 con 0x7f838c159770 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8378005e40 con 0x7f838c159770 2024-09-13T17:23:20.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83780072f0 con 0x7f838c159770 2024-09-13T17:23:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c0ae1e0 con 0x7f838c159770 2024-09-13T17:23:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.732+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c14e6b0 con 0x7f838c159770 2024-09-13T17:23:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.733+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c0ae400 con 0x7f838c159770 2024-09-13T17:23:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.734+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8378007510 con 0x7f838c159770 2024-09-13T17:23:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.735+0000 7f8383e00640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 0x7f836c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.735+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f8378088af0 con 0x7f838c159770 2024-09-13T17:23:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.735+0000 7f838be00640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 0x7f836c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.736+0000 7f838be00640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 0x7f836c06d0c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f837c009ac0 tx=0x7f837c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:20.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.739+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8378053c60 con 0x7f838c159770 2024-09-13T17:23:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.918+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f838c14eb20 con 0x7f838c159770 2024-09-13T17:23:20.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.918+0000 7f8383e00640 1 -- 172.21.15.28:0/3998387825 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v207) v1 ==== 100+0+11 (secure 0 0 0) 0x7f838c14eb20 con 0x7f838c159770 2024-09-13T17:23:20.919 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 msgr2=0x7f836c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 0x7f836c06d0c0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f837c009ac0 tx=0x7f837c008040 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 msgr2=0x7f838c0cc860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7f837800c410 tx=0x7f837800c8d0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 shutdown_connections 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f836c06ac00 0x7f836c06d0c0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f838c15cd80 0x7f838c148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f838c159770 0x7f838c0cc860 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 --2- 172.21.15.28:0/3998387825 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f838c06f3f0 0x7f838c154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 >> 172.21.15.28:0/3998387825 conn(0x7f838c0b0270 msgr2=0x7f838c06c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:20.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 shutdown_connections 2024-09-13T17:23:20.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:20.921+0000 7f8393000640 1 -- 172.21.15.28:0/3998387825 wait complete. 2024-09-13T17:23:20.938 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.092+0000 7ff611e00640 1 Processor -- start 2024-09-13T17:23:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.093+0000 7ff611e00640 1 -- start start 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff611e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c158b20 0x7ff60c0afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff611e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 0x7ff60c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff611e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff60c0b39d0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff611e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0b2dc0 con 0x7ff60c158b20 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff611e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c0b30a0 con 0x7ff60c06ba70 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff610a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c158b20 0x7ff60c0afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff610a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff60c158b20 0x7ff60c0afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52078/0 (socket says 172.21.15.28:52078) 2024-09-13T17:23:21.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff610a00640 1 -- 172.21.15.28:0/2870951850 learned_addr learned my addr 172.21.15.28:0/2870951850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.094+0000 7ff60be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff60c06c3f0 0x7ff60c06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55982/0 (socket says 172.21.15.28:55982) 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff611400640 1 --2- 172.21.15.28:0/2870951850 >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 0x7ff60c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff610a00640 1 -- 172.21.15.28:0/2870951850 >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 msgr2=0x7ff60c158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff610a00640 1 --2- 172.21.15.28:0/2870951850 >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 0x7ff60c158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff610a00640 1 -- 172.21.15.28:0/2870951850 >> v1:172.21.15.28:6789/0 conn(0x7ff60c06c3f0 legacy=0x7ff60c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff610a00640 1 -- 172.21.15.28:0/2870951850 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0b3380 con 0x7ff60c158b20 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff60b400640 1 -- 172.21.15.28:0/2870951850 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 179091145 0 0) 0x7ff60c0b39d0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.095+0000 7ff611400640 1 --2- 172.21.15.28:0/2870951850 >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 0x7ff60c158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff610a00640 1 --2- 172.21.15.28:0/2870951850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c158b20 0x7ff60c0afbd0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff600009870 tx=0x7ff60002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7991fda158feba21 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff60b400640 1 -- 172.21.15.28:0/2870951850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600032440 con 0x7ff60c158b20 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff60b400640 1 -- 172.21.15.28:0/2870951850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000325a0 con 0x7ff60c158b20 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff60b400640 1 -- 172.21.15.28:0/2870951850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600032930 con 0x7ff60c158b20 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff611e00640 1 -- 172.21.15.28:0/2870951850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c158b20 msgr2=0x7ff60c0afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.096+0000 7ff611e00640 1 --2- 172.21.15.28:0/2870951850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c158b20 0x7ff60c0afbd0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7ff600009870 tx=0x7ff60002a230 comp rx=0 tx=0).stop 2024-09-13T17:23:21.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 -- 172.21.15.28:0/2870951850 shutdown_connections 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 --2- 172.21.15.28:0/2870951850 >> v2:172.21.15.189:3300/0 conn(0x7ff60c06ba70 0x7ff60c158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 --2- 172.21.15.28:0/2870951850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c158b20 0x7ff60c0afbd0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 -- 172.21.15.28:0/2870951850 >> 172.21.15.28:0/2870951850 conn(0x7ff60c072e40 msgr2=0x7ff60c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 -- 172.21.15.28:0/2870951850 shutdown_connections 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.097+0000 7ff611e00640 1 -- 172.21.15.28:0/2870951850 wait complete. 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.098+0000 7ff611e00640 1 Processor -- start 2024-09-13T17:23:21.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.099+0000 7ff611e00640 1 -- start start 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.099+0000 7ff611e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff60c06ba70 0x7ff60c1f1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.099+0000 7ff611e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff611e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff60c158b20 0x7ff60c146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff611e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c144810 con 0x7ff60c06c3f0 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff611e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c155670 con 0x7ff60c158b20 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff611e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1550b0 con 0x7ff60c06ba70 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52084/0 (socket says 172.21.15.28:52084) 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 -- 172.21.15.28:0/101365941 learned_addr learned my addr 172.21.15.28:0/101365941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff610a00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff60c06ba70 0x7ff60c1f1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff611400640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff60c158b20 0x7ff60c146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 -- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff60c06ba70 msgr2=0x7ff60c1f1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff60c06ba70 0x7ff60c1f1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 -- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff60c158b20 msgr2=0x7ff60c146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff60c158b20 0x7ff60c146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.100+0000 7ff60be00640 1 -- 172.21.15.28:0/101365941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000094d0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff60be00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c4d0 tx=0x7ff5f800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f80074e0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5f8007660 con 0x7ff60c06c3f0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8004a20 con 0x7ff60c06c3f0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c155390 con 0x7ff60c06c3f0 2024-09-13T17:23:21.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.101+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c144ec0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.102+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c1450b0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.103+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff5f8004b80 con 0x7ff60c06c3f0 2024-09-13T17:23:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.104+0000 7ff608c00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 0x7ff5ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.104+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7ff5f80884f0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.104+0000 7ff610a00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 0x7ff5ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.105+0000 7ff610a00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 0x7ff5ec06d0c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff600034000 tx=0x7ff60002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.108+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff5f80536e0 con 0x7ff60c06c3f0 2024-09-13T17:23:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.323+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60c0b0980 con 0x7ff60c06c3f0 2024-09-13T17:23:21.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.324+0000 7ff608c00640 1 -- 172.21.15.28:0/101365941 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7ff5f8056d00 con 0x7ff60c06c3f0 2024-09-13T17:23:21.325 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:21.325 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:23:21.325 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 msgr2=0x7ff5ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 0x7ff5ec06d0c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7ff600034000 tx=0x7ff60002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 msgr2=0x7ff60c154630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800c4d0 tx=0x7ff5f800c990 comp rx=0 tx=0).stop 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 shutdown_connections 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff5ec06ac00 0x7ff5ec06d0c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff60c158b20 0x7ff60c146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff60c06c3f0 0x7ff60c154630 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 --2- 172.21.15.28:0/101365941 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff60c06ba70 0x7ff60c1f1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 >> 172.21.15.28:0/101365941 conn(0x7ff60c072e40 msgr2=0x7ff60c0b1740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 shutdown_connections 2024-09-13T17:23:21.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.326+0000 7ff611e00640 1 -- 172.21.15.28:0/101365941 wait complete. 2024-09-13T17:23:21.342 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.496+0000 7f60a7000640 1 Processor -- start 2024-09-13T17:23:21.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.496+0000 7f60a7000640 1 -- start start 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f60a7000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60a0146170 0x7f60a015d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f60a7000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f60a00acf60 0x7f60a0160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f60a7000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a0146550 con 0x7f60a0160310 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f60a7000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a015cd20 con 0x7f60a0146170 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f60a7000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a015d280 con 0x7f60a00acf60 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f609fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f60a0146170 0x7f60a015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f609f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f60a0160310 0x7f60a00ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55986/0 (socket says 172.21.15.28:55986) 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.497+0000 7f609f400640 1 -- 172.21.15.28:0/3950323354 learned_addr learned my addr 172.21.15.28:0/3950323354 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f60a5000640 1 --2- 172.21.15.28:0/3950323354 >> v2:172.21.15.189:3300/0 conn(0x7f60a00acf60 0x7f60a0160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f609fe00640 1 -- 172.21.15.28:0/3950323354 >> v2:172.21.15.189:3300/0 conn(0x7f60a00acf60 msgr2=0x7f60a0160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:21.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f609fe00640 1 --2- 172.21.15.28:0/3950323354 >> v2:172.21.15.189:3300/0 conn(0x7f60a00acf60 0x7f60a0160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f609fe00640 1 -- 172.21.15.28:0/3950323354 >> v1:172.21.15.28:6789/0 conn(0x7f60a0160310 legacy=0x7f60a00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f609fe00640 1 -- 172.21.15.28:0/3950323354 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a015dc50 con 0x7f60a0146170 2024-09-13T17:23:21.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.498+0000 7f609ea00640 1 -- 172.21.15.28:0/3950323354 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689988890 0 0) 0x7f60a0146550 con 0x7f60a0160310 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.499+0000 7f609fe00640 1 --2- 172.21.15.28:0/3950323354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a015d710 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f60a006d4b0 tx=0x7f609002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1ba9dbfdeb0b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.499+0000 7f609ea00640 1 -- 172.21.15.28:0/3950323354 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090032440 con 0x7f60a0146170 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.499+0000 7f609ea00640 1 -- 172.21.15.28:0/3950323354 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60900325a0 con 0x7f60a0146170 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.499+0000 7f609ea00640 1 -- 172.21.15.28:0/3950323354 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090032930 con 0x7f60a0146170 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 -- 172.21.15.28:0/3950323354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 msgr2=0x7f60a015d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 --2- 172.21.15.28:0/3950323354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a015d710 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f60a006d4b0 tx=0x7f609002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 -- 172.21.15.28:0/3950323354 shutdown_connections 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 --2- 172.21.15.28:0/3950323354 >> v2:172.21.15.189:3300/0 conn(0x7f60a00acf60 0x7f60a0160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 --2- 172.21.15.28:0/3950323354 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a015d710 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 -- 172.21.15.28:0/3950323354 >> 172.21.15.28:0/3950323354 conn(0x7f60a0069e80 msgr2=0x7f60a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 -- 172.21.15.28:0/3950323354 shutdown_connections 2024-09-13T17:23:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.500+0000 7f60a7000640 1 -- 172.21.15.28:0/3950323354 wait complete. 2024-09-13T17:23:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.501+0000 7f60a7000640 1 Processor -- start 2024-09-13T17:23:21.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 -- start start 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a00acf60 0x7f60a0153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60a0160310 0x7f60a00705a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a013fe50 con 0x7f60a0146170 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0153680 con 0x7f60a00acf60 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.502+0000 7f60a7000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a0131b40 con 0x7f60a0160310 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52102/0 (socket says 172.21.15.28:52102) 2024-09-13T17:23:21.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 -- 172.21.15.28:0/825701131 learned_addr learned my addr 172.21.15.28:0/825701131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609fe00640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a00acf60 0x7f60a0153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f60a5000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60a0160310 0x7f60a00705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 -- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60a0160310 msgr2=0x7f60a00705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60a0160310 0x7f60a00705a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 -- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a00acf60 msgr2=0x7f60a0153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a00acf60 0x7f60a0153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 -- 172.21.15.28:0/825701131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6090009400 con 0x7f60a0146170 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.503+0000 7f609f400640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c500 tx=0x7f608c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.504+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c007560 con 0x7f60a0146170 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.504+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f608c0047e0 con 0x7f60a0146170 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.504+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f608c004b90 con 0x7f60a0146170 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.504+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a0154580 con 0x7f60a0146170 2024-09-13T17:23:21.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.504+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a0154820 con 0x7f60a0146170 2024-09-13T17:23:21.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.505+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a0141490 con 0x7f60a0146170 2024-09-13T17:23:21.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.506+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f608c004cf0 con 0x7f60a0146170 2024-09-13T17:23:21.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.506+0000 7f6097e00640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 0x7f608006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.507+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f608c0884d0 con 0x7f60a0146170 2024-09-13T17:23:21.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.507+0000 7f609fe00640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 0x7f608006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.508+0000 7f609fe00640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 0x7f608006d0c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6090034000 tx=0x7f609002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.511+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f608c0536c0 con 0x7f60a0146170 2024-09-13T17:23:21.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.688+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60a0160750 con 0x7f60a0146170 2024-09-13T17:23:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.690+0000 7f6097e00640 1 -- 172.21.15.28:0/825701131 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16819 (secure 0 0 0) 0x7f608c056ce0 con 0x7f60a0146170 2024-09-13T17:23:21.691 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:21.691 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:55.171778+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.19","mappings":[{"from":6,"to":0}]},{"pgid":"29.39","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 msgr2=0x7f608006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 0x7f608006d0c0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f6090034000 tx=0x7f609002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 msgr2=0x7f60a0131600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f608c00c500 tx=0x7f608c00c9c0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 shutdown_connections 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f608006ac00 0x7f608006d0c0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60a0160310 0x7f60a00705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f60a0146170 0x7f60a0131600 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 --2- 172.21.15.28:0/825701131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60a00acf60 0x7f60a0153140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 >> 172.21.15.28:0/825701131 conn(0x7f60a0069e80 msgr2=0x7f60a00ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:21.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 shutdown_connections 2024-09-13T17:23:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.692+0000 7f60a7000640 1 -- 172.21.15.28:0/825701131 wait complete. 2024-09-13T17:23:21.709 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:23:21.709 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.862+0000 7fd58a400640 1 Processor -- start 2024-09-13T17:23:21.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.863+0000 7fd58a400640 1 -- start start 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd58a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd58406f970 0x7fd58415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd58a400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd58406c2d0 0x7fd58406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd58a400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd58406cc70 con 0x7fd5841592a0 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd58a400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd58415b0b0 con 0x7fd58406f970 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd58a400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd584159d90 con 0x7fd58406c2d0 2024-09-13T17:23:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd57aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd58406f970 0x7fd58415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd57aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd58406f970 0x7fd58415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52114/0 (socket says 172.21.15.28:52114) 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd57aa00640 1 -- 172.21.15.28:0/1652637253 learned_addr learned my addr 172.21.15.28:0/1652637253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.864+0000 7fd583e00640 1 --2- 172.21.15.28:0/1652637253 >> v2:172.21.15.189:3300/0 conn(0x7fd58406c2d0 0x7fd58406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.865+0000 7fd57aa00640 1 -- 172.21.15.28:0/1652637253 >> v2:172.21.15.189:3300/0 conn(0x7fd58406c2d0 msgr2=0x7fd58406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.865+0000 7fd57aa00640 1 --2- 172.21.15.28:0/1652637253 >> v2:172.21.15.189:3300/0 conn(0x7fd58406c2d0 0x7fd58406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.865+0000 7fd57aa00640 1 -- 172.21.15.28:0/1652637253 >> v1:172.21.15.189:6789/0 conn(0x7fd5841592a0 legacy=0x7fd584159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.865+0000 7fd57aa00640 1 -- 172.21.15.28:0/1652637253 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5840b2040 con 0x7fd58406f970 2024-09-13T17:23:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.866+0000 7fd57aa00640 1 --2- 172.21.15.28:0/1652637253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58406f970 0x7fd58415ab70 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fd5840b1dd0 tx=0x7fd56c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8095333f86e670 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.866+0000 7fd582a00640 1 -- 172.21.15.28:0/1652637253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c032440 con 0x7fd58406f970 2024-09-13T17:23:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.866+0000 7fd582a00640 1 -- 172.21.15.28:0/1652637253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56c0325a0 con 0x7fd58406f970 2024-09-13T17:23:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.866+0000 7fd582a00640 1 -- 172.21.15.28:0/1652637253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c032930 con 0x7fd58406f970 2024-09-13T17:23:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 -- 172.21.15.28:0/1652637253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58406f970 msgr2=0x7fd58415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 --2- 172.21.15.28:0/1652637253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58406f970 0x7fd58415ab70 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7fd5840b1dd0 tx=0x7fd56c02a270 comp rx=0 tx=0).stop 2024-09-13T17:23:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 -- 172.21.15.28:0/1652637253 shutdown_connections 2024-09-13T17:23:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 --2- 172.21.15.28:0/1652637253 >> v2:172.21.15.189:3300/0 conn(0x7fd58406c2d0 0x7fd58406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 --2- 172.21.15.28:0/1652637253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58406f970 0x7fd58415ab70 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 -- 172.21.15.28:0/1652637253 >> 172.21.15.28:0/1652637253 conn(0x7fd584069e80 msgr2=0x7fd58406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 -- 172.21.15.28:0/1652637253 shutdown_connections 2024-09-13T17:23:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.867+0000 7fd58a400640 1 -- 172.21.15.28:0/1652637253 wait complete. 2024-09-13T17:23:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.869+0000 7fd58a400640 1 Processor -- start 2024-09-13T17:23:21.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.869+0000 7fd58a400640 1 -- start start 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.869+0000 7fd58a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd58406c2d0 0x7fd5841edae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.869+0000 7fd58a400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd58406f970 0x7fd5840cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd58a400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd58a400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd58415c7f0 con 0x7fd5841592a0 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd58a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5841f1bf0 con 0x7fd58406c2d0 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd58a400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5841f1ed0 con 0x7fd58406f970 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52122/0 (socket says 172.21.15.28:52122) 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 -- 172.21.15.28:0/958003482 learned_addr learned my addr 172.21.15.28:0/958003482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd58406c2d0 0x7fd5841edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd57aa00640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd58406f970 0x7fd5840cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 -- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd58406f970 msgr2=0x7fd5840cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd58406f970 0x7fd5840cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 -- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd58406c2d0 msgr2=0x7fd5841edae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd58406c2d0 0x7fd5841edae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.870+0000 7fd583e00640 1 -- 172.21.15.28:0/958003482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c0094d0 con 0x7fd5841592a0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd583e00640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00efc0 tx=0x7fd57c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c014480 con 0x7fd5841592a0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd57c00f040 con 0x7fd5841592a0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5841f21b0 con 0x7fd5841592a0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd57c014820 con 0x7fd5841592a0 2024-09-13T17:23:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.871+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5840cf300 con 0x7fd5841592a0 2024-09-13T17:23:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.872+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5840b3740 con 0x7fd5841592a0 2024-09-13T17:23:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.873+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd57c014980 con 0x7fd5841592a0 2024-09-13T17:23:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.874+0000 7fd57be00640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 0x7fd56006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.874+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd57c089f30 con 0x7fd5841592a0 2024-09-13T17:23:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.874+0000 7fd583400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 0x7fd56006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.875+0000 7fd583400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 0x7fd56006d0c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd574000f60 tx=0x7fd574009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:21.878+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd57c055090 con 0x7fd5841592a0 2024-09-13T17:23:22.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.090+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd584159820 con 0x7fd5841592a0 2024-09-13T17:23:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.091+0000 7fd57be00640 1 -- 172.21.15.28:0/958003482 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7fd57c018020 con 0x7fd5841592a0 2024-09-13T17:23:22.092 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:22.092 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:23:22.092 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 msgr2=0x7fd56006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 0x7fd56006d0c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fd574000f60 tx=0x7fd574009040 comp rx=0 tx=0).stop 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 msgr2=0x7fd5841f26d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00efc0 tx=0x7fd57c00c490 comp rx=0 tx=0).stop 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 shutdown_connections 2024-09-13T17:23:22.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd56006ac00 0x7fd56006d0c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5841592a0 0x7fd5841f26d0 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd58406f970 0x7fd5840cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 --2- 172.21.15.28:0/958003482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd58406c2d0 0x7fd5841edae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 >> 172.21.15.28:0/958003482 conn(0x7fd584069e80 msgr2=0x7fd5840af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 shutdown_connections 2024-09-13T17:23:22.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.094+0000 7fd58a400640 1 -- 172.21.15.28:0/958003482 wait complete. 2024-09-13T17:23:22.110 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.266+0000 7f3469800640 1 Processor -- start 2024-09-13T17:23:22.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.266+0000 7f3469800640 1 -- start start 2024-09-13T17:23:22.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.267+0000 7f3469800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f346415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.267+0000 7f3469800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3464159d90 con 0x7f346406c2d0 2024-09-13T17:23:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.267+0000 7f3469800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34640b2040 con 0x7f34641592a0 2024-09-13T17:23:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.267+0000 7f3469800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346415b0b0 con 0x7f346406f970 2024-09-13T17:23:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3463400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f346406c2d0 0x7f34640b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55994/0 (socket says 172.21.15.28:55994) 2024-09-13T17:23:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3463400640 1 -- 172.21.15.28:0/464420342 learned_addr learned my addr 172.21.15.28:0/464420342 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:22.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3462000640 1 --2- 172.21.15.28:0/464420342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f346415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903683292 0 0) 0x7f3464159d90 con 0x7f346406c2d0 2024-09-13T17:23:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3448003520 con 0x7f346406c2d0 2024-09-13T17:23:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989344430 0 0) 0x7f34640b2040 con 0x7f34641592a0 2024-09-13T17:23:22.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3464159d90 con 0x7f34641592a0 2024-09-13T17:23:22.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3462000640 1 -- 172.21.15.28:0/464420342 >> v1:172.21.15.189:6789/0 conn(0x7f34641592a0 legacy=0x7f3464159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3462000640 1 -- 172.21.15.28:0/464420342 >> v1:172.21.15.28:6789/0 conn(0x7f346406c2d0 legacy=0x7f34640b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.268+0000 7f3462000640 1 -- 172.21.15.28:0/464420342 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3464159ff0 con 0x7f346406f970 2024-09-13T17:23:22.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.269+0000 7f3462000640 1 --2- 172.21.15.28:0/464420342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f346415ab70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f34500097d0 tx=0x7f345002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff6d3220363af709 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450032440 con 0x7f346406f970 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34500325a0 con 0x7f346406f970 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3461600640 1 -- 172.21.15.28:0/464420342 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3450032930 con 0x7f346406f970 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 -- 172.21.15.28:0/464420342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 msgr2=0x7f346415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 --2- 172.21.15.28:0/464420342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f346415ab70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f34500097d0 tx=0x7f345002a210 comp rx=0 tx=0).stop 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 -- 172.21.15.28:0/464420342 shutdown_connections 2024-09-13T17:23:22.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 --2- 172.21.15.28:0/464420342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f346415ab70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 -- 172.21.15.28:0/464420342 >> 172.21.15.28:0/464420342 conn(0x7f3464069e80 msgr2=0x7f346406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:22.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.270+0000 7f3469800640 1 -- 172.21.15.28:0/464420342 shutdown_connections 2024-09-13T17:23:22.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.271+0000 7f3469800640 1 -- 172.21.15.28:0/464420342 wait complete. 2024-09-13T17:23:22.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.272+0000 7f3469800640 1 Processor -- start 2024-09-13T17:23:22.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.272+0000 7f3469800640 1 -- start start 2024-09-13T17:23:22.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346406c2d0 0x7f34640ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f34641edbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641ee430 con 0x7f34641592a0 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34641ee710 con 0x7f346406c2d0 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3469800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34641ee9f0 con 0x7f346406f970 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3463400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3463400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52138/0 (socket says 172.21.15.28:52138) 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.273+0000 7f3463400640 1 -- 172.21.15.28:0/2948088996 learned_addr learned my addr 172.21.15.28:0/2948088996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3462a00640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346406c2d0 0x7f34640ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3462000640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f34641edbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 -- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 msgr2=0x7f34641edbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f34641edbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 -- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346406c2d0 msgr2=0x7f34640ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346406c2d0 0x7f34640ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 -- 172.21.15.28:0/2948088996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3450009400 con 0x7f34641592a0 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f3463400640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f345c00cbc0 tx=0x7f345c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.274+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c00ad40 con 0x7f34641592a0 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.275+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345c012400 con 0x7f34641592a0 2024-09-13T17:23:22.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.275+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34641f11b0 con 0x7f34641592a0 2024-09-13T17:23:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.275+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345c0127f0 con 0x7f34641592a0 2024-09-13T17:23:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.275+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34641ee120 con 0x7f34641592a0 2024-09-13T17:23:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.276+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34641f16d0 con 0x7f34641592a0 2024-09-13T17:23:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.277+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f345c007070 con 0x7f34641592a0 2024-09-13T17:23:22.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.278+0000 7f345aa00640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 0x7f344406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.278+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f345c089330 con 0x7f34641592a0 2024-09-13T17:23:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.278+0000 7f3462a00640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 0x7f344406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.279+0000 7f3462a00640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 0x7f344406d0c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f34540011e0 tx=0x7f3454009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.282+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f34641f16d0 con 0x7f34641592a0 2024-09-13T17:23:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.498+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34641f16d0 con 0x7f34641592a0 2024-09-13T17:23:22.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.499+0000 7f345aa00640 1 -- 172.21.15.28:0/2948088996 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+4752 (secure 0 0 0) 0x7f34641f16d0 con 0x7f34641592a0 2024-09-13T17:23:22.500 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:22.500 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":172,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:22:35.717384+0000","modified":"2024-09-13T17:22:38.727402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11478":{"gid":11478,"name":"c","rank":0,"incarnation":169,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6834/299660514","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":299660514},{"type":"v1","addr":"172.21.15.28:6834","nonce":299660514}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2024-09-13T17:23:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 172 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 msgr2=0x7f344406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 0x7f344406d0c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f34540011e0 tx=0x7f3454009040 comp rx=0 tx=0).stop 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 msgr2=0x7f3464154130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f345c00cbc0 tx=0x7f345c00a430 comp rx=0 tx=0).stop 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 shutdown_connections 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f344406ac00 0x7f344406d0c0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34641592a0 0x7f3464154130 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f346406f970 0x7f34641edbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 --2- 172.21.15.28:0/2948088996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f346406c2d0 0x7f34640ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 >> 172.21.15.28:0/2948088996 conn(0x7f3464069e80 msgr2=0x7f346406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 shutdown_connections 2024-09-13T17:23:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.501+0000 7f3469800640 1 -- 172.21.15.28:0/2948088996 wait complete. 2024-09-13T17:23:22.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:22.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.671+0000 7f36f8200640 1 Processor -- start 2024-09-13T17:23:22.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.671+0000 7f36f8200640 1 -- start start 2024-09-13T17:23:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 0x7f36f006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f00cabc0 0x7f36f015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015edb0 0x7f36f0070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f00adb00 con 0x7f36f00cabc0 2024-09-13T17:23:22.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36f015e100 con 0x7f36f015a540 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f8200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f015e3e0 con 0x7f36f015edb0 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f00cabc0 0x7f36f015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f36f00cabc0 0x7f36f015e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52150/0 (socket says 172.21.15.28:52150) 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 -- 172.21.15.28:0/870234341 learned_addr learned my addr 172.21.15.28:0/870234341 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f5800640 1 --2- 172.21.15.28:0/870234341 >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 0x7f36f006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f6200640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015edb0 0x7f36f0070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 -- 172.21.15.28:0/870234341 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015edb0 msgr2=0x7f36f0070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015edb0 0x7f36f0070cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 -- 172.21.15.28:0/870234341 >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 msgr2=0x7f36f006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 --2- 172.21.15.28:0/870234341 >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 0x7f36f006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f4e00640 1 -- 172.21.15.28:0/870234341 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36f006ce80 con 0x7f36f00cabc0 2024-09-13T17:23:22.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.673+0000 7f36f5800640 1 --2- 172.21.15.28:0/870234341 >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 0x7f36f006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:22.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.674+0000 7f36f4e00640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f00cabc0 0x7f36f015e870 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f36f00b3a00 tx=0x7f36dc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc60101a1c3d704 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36efe00640 1 -- 172.21.15.28:0/870234341 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc02ae90 con 0x7f36f00cabc0 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36efe00640 1 -- 172.21.15.28:0/870234341 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36dc032440 con 0x7f36f00cabc0 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36efe00640 1 -- 172.21.15.28:0/870234341 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36dc0327d0 con 0x7f36f00cabc0 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 -- 172.21.15.28:0/870234341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f00cabc0 msgr2=0x7f36f015e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f00cabc0 0x7f36f015e870 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7f36f00b3a00 tx=0x7f36dc02a140 comp rx=0 tx=0).stop 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 -- 172.21.15.28:0/870234341 shutdown_connections 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f00cabc0 0x7f36f015e870 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 --2- 172.21.15.28:0/870234341 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015edb0 0x7f36f0070cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 --2- 172.21.15.28:0/870234341 >> v2:172.21.15.189:3300/0 conn(0x7f36f015a540 0x7f36f006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 -- 172.21.15.28:0/870234341 >> 172.21.15.28:0/870234341 conn(0x7f36f0069e80 msgr2=0x7f36f006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 -- 172.21.15.28:0/870234341 shutdown_connections 2024-09-13T17:23:22.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.675+0000 7f36f8200640 1 -- 172.21.15.28:0/870234341 wait complete. 2024-09-13T17:23:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.677+0000 7f36f8200640 1 Processor -- start 2024-09-13T17:23:22.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.677+0000 7f36f8200640 1 -- start start 2024-09-13T17:23:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 0x7f36f01f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 0x7f36f01f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f014d510 con 0x7f36f015c630 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36f01f5d60 con 0x7f36f00cabc0 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f8200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36f00b1890 con 0x7f36f015a540 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52158/0 (socket says 172.21.15.28:52158) 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.678+0000 7f36f6200640 1 -- 172.21.15.28:0/702764367 learned_addr learned my addr 172.21.15.28:0/702764367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f5800640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 0x7f36f01f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f4e00640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 0x7f36f01f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 -- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 msgr2=0x7f36f01f5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 0x7f36f01f5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 -- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 msgr2=0x7f36f01f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 0x7f36f01f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 -- 172.21.15.28:0/702764367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36dc0094d0 con 0x7f36f015c630 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f4e00640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 0x7f36f01f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:22.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f5800640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 0x7f36f01f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f6200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009fd0 tx=0x7f36e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8014420 con 0x7f36f015c630 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36e8014580 con 0x7f36f015c630 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36e8014930 con 0x7f36f015c630 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.679+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36f00b1b70 con 0x7f36f015c630 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.680+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36f00b2130 con 0x7f36f015c630 2024-09-13T17:23:22.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.680+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f00aed30 con 0x7f36f015c630 2024-09-13T17:23:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.681+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f36e8014a90 con 0x7f36f015c630 2024-09-13T17:23:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.682+0000 7f36ed600640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 0x7f36d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.682+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7f36e8089cc0 con 0x7f36f015c630 2024-09-13T17:23:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.683+0000 7f36f5800640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 0x7f36d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:22.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.683+0000 7f36f5800640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 0x7f36d006d0f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f36e4006fd0 tx=0x7f36e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:22.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.686+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f36e8090080 con 0x7f36f015c630 2024-09-13T17:23:22.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.865+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36f015e870 con 0x7f36f015c630 2024-09-13T17:23:22.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.867+0000 7f36ed600640 1 -- 172.21.15.28:0/702764367 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v207) v1 ==== 74+0+16819 (secure 0 0 0) 0x7f36e8054d70 con 0x7f36f015c630 2024-09-13T17:23:22.868 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:22.868 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":207,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:22:55.171778+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:22:33.628286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"204","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"cephfs_data","create_time":"2024-09-13T17:22:34.718902+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"205","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"29.3","mappings":[{"from":6,"to":0}]},{"pgid":"29.f","mappings":[{"from":6,"to":0}]},{"pgid":"29.16","mappings":[{"from":6,"to":0}]},{"pgid":"29.19","mappings":[{"from":6,"to":0}]},{"pgid":"29.39","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 msgr2=0x7f36d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 0x7f36d006d0f0 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f36e4006fd0 tx=0x7f36e4008040 comp rx=0 tx=0).stop 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 msgr2=0x7f36f00ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 secure :-1 s=READY pgs=2538 cs=0 l=1 rev1=1 crypto rx=0x7f36e8009fd0 tx=0x7f36e800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 shutdown_connections 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f36d006ac30 0x7f36d006d0f0 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f015c630 0x7f36f00ac850 unknown :-1 s=CLOSED pgs=2538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36f015a540 0x7f36f01f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 --2- 172.21.15.28:0/702764367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36f00cabc0 0x7f36f01f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 >> 172.21.15.28:0/702764367 conn(0x7f36f0069e80 msgr2=0x7f36f00ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 shutdown_connections 2024-09-13T17:23:22.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:22.869+0000 7f36f8200640 1 -- 172.21.15.28:0/702764367 wait complete. 2024-09-13T17:23:22.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:23:23.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.045+0000 7fd696800640 1 Processor -- start 2024-09-13T17:23:23.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.045+0000 7fd696800640 1 -- start start 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.046+0000 7fd696800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd69006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.046+0000 7fd696800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd69015cd80 0x7fd690070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd696800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd69006f920 con 0x7fd69006f3f0 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd696800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd69015b290 con 0x7fd69015cd80 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd696800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69015b570 con 0x7fd690159770 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd69015cd80 0x7fd690070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd69015cd80 0x7fd690070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52168/0 (socket says 172.21.15.28:52168) 2024-09-13T17:23:23.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 -- 172.21.15.28:0/3465027260 learned_addr learned my addr 172.21.15.28:0/3465027260 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd686a00640 1 --1- 172.21.15.28:0/3465027260 >> v1:172.21.15.28:6789/0 conn(0x7fd69006f3f0 0x7fd690159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55996/0 (socket says 172.21.15.28:55996) 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68f400640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd69006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 -- 172.21.15.28:0/3465027260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 msgr2=0x7fd69006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd69006fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 -- 172.21.15.28:0/3465027260 >> v1:172.21.15.28:6789/0 conn(0x7fd69006f3f0 legacy=0x7fd690159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68fe00640 1 -- 172.21.15.28:0/3465027260 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69015b850 con 0x7fd69015cd80 2024-09-13T17:23:23.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.047+0000 7fd68f400640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd69006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.048+0000 7fd68fe00640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69015cd80 0x7fd690070260 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc87e68a0a441077 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd68ea00640 1 -- 172.21.15.28:0/3465027260 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688031600 con 0x7fd69015cd80 2024-09-13T17:23:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd68ea00640 1 -- 172.21.15.28:0/3465027260 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd68802d050 con 0x7fd69015cd80 2024-09-13T17:23:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd68ea00640 1 -- 172.21.15.28:0/3465027260 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd688031980 con 0x7fd69015cd80 2024-09-13T17:23:23.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 -- 172.21.15.28:0/3465027260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69015cd80 msgr2=0x7fd690070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69015cd80 0x7fd690070260 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7fd688009e50 tx=0x7fd68802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 -- 172.21.15.28:0/3465027260 shutdown_connections 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69015cd80 0x7fd690070260 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 --2- 172.21.15.28:0/3465027260 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd69006fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 -- 172.21.15.28:0/3465027260 >> 172.21.15.28:0/3465027260 conn(0x7fd6900b0270 msgr2=0x7fd6900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 -- 172.21.15.28:0/3465027260 shutdown_connections 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.049+0000 7fd696800640 1 -- 172.21.15.28:0/3465027260 wait complete. 2024-09-13T17:23:23.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.051+0000 7fd696800640 1 Processor -- start 2024-09-13T17:23:23.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.051+0000 7fd696800640 1 -- start start 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd6900cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd69015cd80 0x7fd6900ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900cb960 con 0x7fd69006f3f0 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6900b1990 con 0x7fd69015cd80 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd696800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd690148810 con 0x7fd690159770 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd68f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd68f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52170/0 (socket says 172.21.15.28:52170) 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd68f400640 1 -- 172.21.15.28:0/1397625198 learned_addr learned my addr 172.21.15.28:0/1397625198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68fe00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd69015cd80 0x7fd6900ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.052+0000 7fd686a00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd6900cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 -- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 msgr2=0x7fd6900cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd6900cbf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 -- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd69015cd80 msgr2=0x7fd6900ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd69015cd80 0x7fd6900ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 -- 172.21.15.28:0/1397625198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688009b30 con 0x7fd69006f3f0 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd686a00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd6900cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:23.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd68f400640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd68000e730 tx=0x7fd68000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd680004230 con 0x7fd69006f3f0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd680004390 con 0x7fd69006f3f0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd68000c8f0 con 0x7fd69006f3f0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.053+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6900b1c30 con 0x7fd69006f3f0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.054+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6900b21f0 con 0x7fd69006f3f0 2024-09-13T17:23:23.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.055+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6900704e0 con 0x7fd69006f3f0 2024-09-13T17:23:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.056+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd68000ca50 con 0x7fd69006f3f0 2024-09-13T17:23:23.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.057+0000 7fd687e00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 0x7fd66c06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.057+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5854+0+0 (secure 0 0 0) 0x7fd680088f80 con 0x7fd69006f3f0 2024-09-13T17:23:23.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.057+0000 7fd686a00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 0x7fd66c06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.058+0000 7fd686a00640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 0x7fd66c06d110 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd678007fd0 tx=0x7fd678007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.061+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd680054160 con 0x7fd69006f3f0 2024-09-13T17:23:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.275+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd6900b3850 con 0x7fd69006f3f0 2024-09-13T17:23:23.277 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:23:23.276+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:23:23.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.283+0000 7fd687e00640 1 -- 172.21.15.28:0/1397625198 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v173) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd680057780 con 0x7fd69006f3f0 2024-09-13T17:23:23.284 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:23:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 msgr2=0x7fd66c06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 0x7fd66c06d110 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fd678007fd0 tx=0x7fd678007d40 comp rx=0 tx=0).stop 2024-09-13T17:23:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 msgr2=0x7fd6900b29f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 secure :-1 s=READY pgs=2541 cs=0 l=1 rev1=1 crypto rx=0x7fd68000e730 tx=0x7fd68000ebf0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 shutdown_connections 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd66c06ac50 0x7fd66c06d110 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd69015cd80 0x7fd6900ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd690159770 0x7fd6900cbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 --2- 172.21.15.28:0/1397625198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd69006f3f0 0x7fd6900b29f0 unknown :-1 s=CLOSED pgs=2541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 >> 172.21.15.28:0/1397625198 conn(0x7fd6900b0270 msgr2=0x7fd69015bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 shutdown_connections 2024-09-13T17:23:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.285+0000 7fd696800640 1 -- 172.21.15.28:0/1397625198 wait complete. 2024-09-13T17:23:23.295 INFO:tasks.ceph.mds.c.smithi028.stderr: -6162> 2024-09-13T17:22:36.274+0000 7f97f7400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:23:23.298 INFO:tasks.ceph.mds.c.smithi028.stderr: -3173> 2024-09-13T17:22:44.958+0000 7f97f2e00640 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /subdir/sixmegs auth v6 ap=1 DIRTYPARENT s=5242880 n(v0 rc2024-09-13T17:22:43.449196+0000 b5242880 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={11640=0-12582912@1} caps={11640=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@4},l=11640 | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x5611462cd700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(29)0x10000000001:[<0x10000000000/sixmegs v6>,<0x1/subdir v8>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T17:23:23.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:23:23.348 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:23:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.469+0000 7fafeae00640 1 Processor -- start 2024-09-13T17:23:23.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.469+0000 7fafeae00640 1 -- start start 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafeae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 0x7fafe40ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafeae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe4070470 con 0x7fafe40b2400 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafeae00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe40ace30 con 0x7fafe415c9a0 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafeae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40ad110 con 0x7fafe40ad940 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafe3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fafe40b2400 0x7fafe406bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:56002/0 (socket says 172.21.15.28:56002) 2024-09-13T17:23:23.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.470+0000 7fafe3400640 1 -- 172.21.15.28:0/3498791212 learned_addr learned my addr 172.21.15.28:0/3498791212 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe8e00640 1 --2- 172.21.15.28:0/3498791212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 0x7fafe40ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016838509 0 0) 0x7fafe4070470 con 0x7fafe40b2400 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8003520 con 0x7fafe40b2400 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873224223 0 0) 0x7fafe40ace30 con 0x7fafe415c9a0 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe4070470 con 0x7fafe415c9a0 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe8e00640 1 -- 172.21.15.28:0/3498791212 >> v1:172.21.15.189:6789/0 conn(0x7fafe415c9a0 legacy=0x7fafe40b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe8e00640 1 -- 172.21.15.28:0/3498791212 >> v1:172.21.15.28:6789/0 conn(0x7fafe40b2400 legacy=0x7fafe406bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.471+0000 7fafe8e00640 1 -- 172.21.15.28:0/3498791212 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe406b170 con 0x7fafe40ad940 2024-09-13T17:23:23.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.473+0000 7fafe8e00640 1 --2- 172.21.15.28:0/3498791212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 0x7fafe40ac850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009dc0 tx=0x7fafdc02a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc89c75080d49d5a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.473+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc007740 con 0x7fafe40ad940 2024-09-13T17:23:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.473+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafdc02d050 con 0x7fafe40ad940 2024-09-13T17:23:23.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.473+0000 7fafe2a00640 1 -- 172.21.15.28:0/3498791212 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafdc007ac0 con 0x7fafe40ad940 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 -- 172.21.15.28:0/3498791212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 msgr2=0x7fafe40ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 --2- 172.21.15.28:0/3498791212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 0x7fafe40ac850 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fafdc009dc0 tx=0x7fafdc02a760 comp rx=0 tx=0).stop 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 -- 172.21.15.28:0/3498791212 shutdown_connections 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 --2- 172.21.15.28:0/3498791212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe40ad940 0x7fafe40ac850 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 -- 172.21.15.28:0/3498791212 >> 172.21.15.28:0/3498791212 conn(0x7fafe40b0270 msgr2=0x7fafe40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 -- 172.21.15.28:0/3498791212 shutdown_connections 2024-09-13T17:23:23.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.474+0000 7fafeae00640 1 -- 172.21.15.28:0/3498791212 wait complete. 2024-09-13T17:23:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.475+0000 7fafeae00640 1 Processor -- start 2024-09-13T17:23:23.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.475+0000 7fafeae00640 1 -- start start 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe40b2400 0x7fafe4143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe415c9a0 0x7fafe415d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe4144010 con 0x7fafe40ad940 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe415faa0 con 0x7fafe40b2400 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.476+0000 7fafeae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe415ec30 con 0x7fafe415c9a0 2024-09-13T17:23:23.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52206/0 (socket says 172.21.15.28:52206) 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 -- 172.21.15.28:0/3800175990 learned_addr learned my addr 172.21.15.28:0/3800175990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3400640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe40b2400 0x7fafe4143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe8e00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe415c9a0 0x7fafe415d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 -- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe415c9a0 msgr2=0x7fafe415d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe415c9a0 0x7fafe415d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 -- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe40b2400 msgr2=0x7fafe4143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe40b2400 0x7fafe4143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 -- 172.21.15.28:0/3800175990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc009a70 con 0x7fafe40ad940 2024-09-13T17:23:23.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.477+0000 7fafe3e00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c430 tx=0x7fafd400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.478+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd40052d0 con 0x7fafe40ad940 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.478+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe4070930 con 0x7fafe40ad940 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.478+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd4005450 con 0x7fafe40ad940 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.478+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003830 con 0x7fafe40ad940 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.478+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe415f170 con 0x7fafe40ad940 2024-09-13T17:23:23.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.479+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe415cd80 con 0x7fafe40ad940 2024-09-13T17:23:23.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.480+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fafd4003990 con 0x7fafe40ad940 2024-09-13T17:23:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.481+0000 7fafdbe00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 0x7fafc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.481+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fafd4088b10 con 0x7fafe40ad940 2024-09-13T17:23:23.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.481+0000 7fafe3400640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 0x7fafc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:23.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.482+0000 7fafe3400640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 0x7fafc406d0c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fafd00011e0 tx=0x7fafd0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:23.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.485+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fafd4053b70 con 0x7fafe40ad940 2024-09-13T17:23:23.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:23.698+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fafe4070d10 con 0x7fafe40ad940 2024-09-13T17:23:24.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.285+0000 7fafdbe00640 1 -- 172.21.15.28:0/3800175990 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v174) v1 ==== 104+0+0 (secure 0 0 0) 0x7fafd4057190 con 0x7fafe40ad940 2024-09-13T17:23:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 msgr2=0x7fafc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:24.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 0x7fafc406d0c0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fafd00011e0 tx=0x7fafd0009040 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 msgr2=0x7fafe415ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 secure :-1 s=READY pgs=2545 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c430 tx=0x7fafd400c8f0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 shutdown_connections 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fafc406ac00 0x7fafc406d0c0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fafe415c9a0 0x7fafe415d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafe40b2400 0x7fafe4143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 --2- 172.21.15.28:0/3800175990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fafe40ad940 0x7fafe415ff30 unknown :-1 s=CLOSED pgs=2545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.287+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 >> 172.21.15.28:0/3800175990 conn(0x7fafe40b0270 msgr2=0x7fafe406bb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.288+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 shutdown_connections 2024-09-13T17:23:24.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.288+0000 7fafeae00640 1 -- 172.21.15.28:0/3800175990 wait complete. 2024-09-13T17:23:24.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:23:24.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.507+0000 7fbee7c00640 1 Processor -- start 2024-09-13T17:23:24.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.507+0000 7fbee7c00640 1 -- start start 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.508+0000 7fbee7c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbee0157000 0x7fbee006b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee7c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbee006b6b0 0x7fbee00b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee7c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee7c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee015d950 con 0x7fbee006b6b0 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee7c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbee015dc30 con 0x7fbee0157000 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee7c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee00b1b80 con 0x7fbee00b28c0 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbee006b6b0 0x7fbee00b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbee006b6b0 0x7fbee00b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54174/0 (socket says 172.21.15.28:54174) 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 -- 172.21.15.28:0/1046886334 learned_addr learned my addr 172.21.15.28:0/1046886334 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee5200640 1 --2- 172.21.15.28:0/1046886334 >> v2:172.21.15.189:3300/0 conn(0x7fbee0157000 0x7fbee006b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee5c00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 -- 172.21.15.28:0/1046886334 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 msgr2=0x7fbee015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:24.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 -- 172.21.15.28:0/1046886334 >> v2:172.21.15.189:3300/0 conn(0x7fbee0157000 msgr2=0x7fbee006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:24.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 --2- 172.21.15.28:0/1046886334 >> v2:172.21.15.189:3300/0 conn(0x7fbee0157000 0x7fbee006b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbed7e00640 1 -- 172.21.15.28:0/1046886334 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee00b1e60 con 0x7fbee006b6b0 2024-09-13T17:23:24.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.509+0000 7fbee5c00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.510+0000 7fbed7e00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee00b2380 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fbee006d570 tx=0x7fbec802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7801791f71a45cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbedfe00640 1 -- 172.21.15.28:0/1046886334 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8032440 con 0x7fbee006b6b0 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbedfe00640 1 -- 172.21.15.28:0/1046886334 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbec80325a0 con 0x7fbee006b6b0 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbedfe00640 1 -- 172.21.15.28:0/1046886334 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbec8032930 con 0x7fbee006b6b0 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 -- 172.21.15.28:0/1046886334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 msgr2=0x7fbee00b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee00b2380 secure :-1 s=READY pgs=2546 cs=0 l=1 rev1=1 crypto rx=0x7fbee006d570 tx=0x7fbec802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 -- 172.21.15.28:0/1046886334 shutdown_connections 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee00b2380 unknown :-1 s=CLOSED pgs=2546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 --2- 172.21.15.28:0/1046886334 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 --2- 172.21.15.28:0/1046886334 >> v2:172.21.15.189:3300/0 conn(0x7fbee0157000 0x7fbee006b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 -- 172.21.15.28:0/1046886334 >> 172.21.15.28:0/1046886334 conn(0x7fbee0069e80 msgr2=0x7fbee006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 -- 172.21.15.28:0/1046886334 shutdown_connections 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.511+0000 7fbee7c00640 1 -- 172.21.15.28:0/1046886334 wait complete. 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.513+0000 7fbee7c00640 1 Processor -- start 2024-09-13T17:23:24.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.513+0000 7fbee7c00640 1 -- start start 2024-09-13T17:23:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 0x7fbee00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0159d40 con 0x7fbee006b6b0 2024-09-13T17:23:24.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbee0158eb0 con 0x7fbee0157000 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee7c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbee01505f0 con 0x7fbee00b28c0 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54186/0 (socket says 172.21.15.28:54186) 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.514+0000 7fbee5200640 1 -- 172.21.15.28:0/268806249 learned_addr learned my addr 172.21.15.28:0/268806249 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbed7e00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 -- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 msgr2=0x7fbee015a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 0x7fbee00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 -- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 msgr2=0x7fbee00cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:24.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 0x7fbee00cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 -- 172.21.15.28:0/268806249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec80094d0 con 0x7fbee006b6b0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbed7e00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 0x7fbee00cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee5200640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fbed000e9b0 tx=0x7fbed000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed0009670 con 0x7fbee006b6b0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed000c9a0 con 0x7fbee006b6b0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed000cd50 con 0x7fbee006b6b0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbee015a6a0 con 0x7fbee006b6b0 2024-09-13T17:23:24.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.515+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbee01593a0 con 0x7fbee006b6b0 2024-09-13T17:23:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.516+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbee015cdd0 con 0x7fbee006b6b0 2024-09-13T17:23:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.517+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbed0013470 con 0x7fbee006b6b0 2024-09-13T17:23:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.518+0000 7fbedd600640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 0x7fbebc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.518+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5940+0+0 (secure 0 0 0) 0x7fbed0089650 con 0x7fbee006b6b0 2024-09-13T17:23:24.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.519+0000 7fbed7e00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 0x7fbebc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:24.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.519+0000 7fbed7e00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 0x7fbebc06d0c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbec8034000 tx=0x7fbec802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:24.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.523+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbed0054730 con 0x7fbee006b6b0 2024-09-13T17:23:24.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:24.701+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbee006c510 con 0x7fbee006b6b0 2024-09-13T17:23:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.290+0000 7fbedd600640 1 -- 172.21.15.28:0/268806249 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v209) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbed0057d50 con 0x7fbee006b6b0 2024-09-13T17:23:25.292 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:23:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.293+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 msgr2=0x7fbebc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.293+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 0x7fbebc06d0c0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fbec8034000 tx=0x7fbec802e000 comp rx=0 tx=0).stop 2024-09-13T17:23:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.293+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 msgr2=0x7fbee01500b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.293+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 secure :-1 s=READY pgs=2547 cs=0 l=1 rev1=1 crypto rx=0x7fbed000e9b0 tx=0x7fbed000ee70 comp rx=0 tx=0).stop 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 shutdown_connections 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbebc06ac00 0x7fbebc06d0c0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0157000 0x7fbee00cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbee00b28c0 0x7fbee015a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 --2- 172.21.15.28:0/268806249 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbee006b6b0 0x7fbee01500b0 unknown :-1 s=CLOSED pgs=2547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 >> 172.21.15.28:0/268806249 conn(0x7fbee0069e80 msgr2=0x7fbee006f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:25.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 shutdown_connections 2024-09-13T17:23:25.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.294+0000 7fbee7c00640 1 -- 172.21.15.28:0/268806249 wait complete. 2024-09-13T17:23:25.312 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:23:25.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.473+0000 7fa1f8000640 1 Processor -- start 2024-09-13T17:23:25.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.473+0000 7fa1f8000640 1 -- start start 2024-09-13T17:23:25.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.474+0000 7fa1f8000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa1f015a7a0 0x7fa1f00ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.474+0000 7fa1f8000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1f00b1c80 0x7fa1f00cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.474+0000 7fa1f8000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f006c950 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.474+0000 7fa1f8000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f00cee40 con 0x7fa1f00b1c80 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.474+0000 7fa1f8000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f00b2150 con 0x7fa1f015a7a0 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1f00b1c80 0x7fa1f00cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa1f00b1c80 0x7fa1f00cd160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54196/0 (socket says 172.21.15.28:54196) 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 -- 172.21.15.28:0/951286684 learned_addr learned my addr 172.21.15.28:0/951286684 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f5600640 1 --2- 172.21.15.28:0/951286684 >> v2:172.21.15.189:3300/0 conn(0x7fa1f015a7a0 0x7fa1f00ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 -- 172.21.15.28:0/951286684 >> v2:172.21.15.189:3300/0 conn(0x7fa1f015a7a0 msgr2=0x7fa1f00ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 --2- 172.21.15.28:0/951286684 >> v2:172.21.15.189:3300/0 conn(0x7fa1f015a7a0 0x7fa1f00ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 -- 172.21.15.28:0/951286684 >> v1:172.21.15.189:6789/0 conn(0x7fa1f00ce9d0 legacy=0x7fa1f00b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.475+0000 7fa1f6000640 1 -- 172.21.15.28:0/951286684 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f00b2430 con 0x7fa1f00b1c80 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.476+0000 7fa1f6000640 1 --2- 172.21.15.28:0/951286684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00b1c80 0x7fa1f00cd160 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00716c0 tx=0x7fa1e802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb0aa1adc206048 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.476+0000 7fa1efe00640 1 -- 172.21.15.28:0/951286684 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8031600 con 0x7fa1f00b1c80 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.476+0000 7fa1efe00640 1 -- 172.21.15.28:0/951286684 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e802d050 con 0x7fa1f00b1c80 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.476+0000 7fa1efe00640 1 -- 172.21.15.28:0/951286684 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8031980 con 0x7fa1f00b1c80 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 -- 172.21.15.28:0/951286684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00b1c80 msgr2=0x7fa1f00cd160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 --2- 172.21.15.28:0/951286684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00b1c80 0x7fa1f00cd160 secure :-1 s=READY pgs=2548 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00716c0 tx=0x7fa1e802a6a0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 -- 172.21.15.28:0/951286684 shutdown_connections 2024-09-13T17:23:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 --2- 172.21.15.28:0/951286684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00b1c80 0x7fa1f00cd160 unknown :-1 s=CLOSED pgs=2548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 --2- 172.21.15.28:0/951286684 >> v2:172.21.15.189:3300/0 conn(0x7fa1f015a7a0 0x7fa1f00ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 -- 172.21.15.28:0/951286684 >> 172.21.15.28:0/951286684 conn(0x7fa1f0069e80 msgr2=0x7fa1f006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 -- 172.21.15.28:0/951286684 shutdown_connections 2024-09-13T17:23:25.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.477+0000 7fa1f8000640 1 -- 172.21.15.28:0/951286684 wait complete. 2024-09-13T17:23:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.479+0000 7fa1f8000640 1 Processor -- start 2024-09-13T17:23:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.479+0000 7fa1f8000640 1 -- start start 2024-09-13T17:23:25.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 0x7fa1f014edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f015a7a0 0x7fa1f01e4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f01e5d80 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00ade70 con 0x7fa1f015a7a0 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f8000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f00ae150 con 0x7fa1f00b1c80 2024-09-13T17:23:25.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54210/0 (socket says 172.21.15.28:54210) 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 -- 172.21.15.28:0/1686183235 learned_addr learned my addr 172.21.15.28:0/1686183235 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f6000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f015a7a0 0x7fa1f01e4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f5600640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 0x7fa1f014edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 -- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 msgr2=0x7fa1f014edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 0x7fa1f014edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 -- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f015a7a0 msgr2=0x7fa1f01e4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f015a7a0 0x7fa1f01e4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f4c00640 1 -- 172.21.15.28:0/1686183235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8009b30 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.480+0000 7fa1f5600640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 0x7fa1f014edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1f4c00640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c450 tx=0x7fa1dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:25.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0052d0 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc005430 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc012870 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f00ae430 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.481+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f006f6d0 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.482+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00ce490 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.483+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa1dc0129d0 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.484+0000 7fa1ed600640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 0x7fa1d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.484+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5502+0+0 (secure 0 0 0) 0x7fa1dc088c20 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.484+0000 7fa1f5600640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 0x7fa1d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:25.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.485+0000 7fa1f5600640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 0x7fa1d006d0c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009af0 tx=0x7fa1e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:25.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.488+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa1dc053e30 con 0x7fa1f00ce9d0 2024-09-13T17:23:25.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:25.669+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa1f006f890 con 0x7fa1f00ce9d0 2024-09-13T17:23:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.294+0000 7fa1ed600640 1 -- 172.21.15.28:0/1686183235 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v210) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa1dc057450 con 0x7fa1f00ce9d0 2024-09-13T17:23:26.297 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 msgr2=0x7fa1d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 0x7fa1d006d0c0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009af0 tx=0x7fa1e4008040 comp rx=0 tx=0).stop 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 msgr2=0x7fa1f01e4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c450 tx=0x7fa1dc00c910 comp rx=0 tx=0).stop 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 shutdown_connections 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa1d006ac00 0x7fa1d006d0c0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.298+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1f015a7a0 0x7fa1f01e4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.299+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1f00ce9d0 0x7fa1f01e4540 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.299+0000 7fa1f8000640 1 --2- 172.21.15.28:0/1686183235 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa1f00b1c80 0x7fa1f014edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.299+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 >> 172.21.15.28:0/1686183235 conn(0x7fa1f0069e80 msgr2=0x7fa1f0157c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.299+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 shutdown_connections 2024-09-13T17:23:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.299+0000 7fa1f8000640 1 -- 172.21.15.28:0/1686183235 wait complete. 2024-09-13T17:23:26.320 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:23:26.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.475+0000 7f78c1800640 1 Processor -- start 2024-09-13T17:23:26.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.475+0000 7f78c1800640 1 -- start start 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78c1800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78bc143320 0x7f78bc070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78c1800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc071ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78c1800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc071630 con 0x7f78bc072000 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78c1800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc143700 con 0x7f78bc143320 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78c1800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc070a20 con 0x7f78bc143910 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78bc143320 0x7f78bc070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f78bc143320 0x7f78bc070440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54212/0 (socket says 172.21.15.28:54212) 2024-09-13T17:23:26.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 -- 172.21.15.28:0/2221290950 learned_addr learned my addr 172.21.15.28:0/2221290950 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 -- 172.21.15.28:0/2221290950 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 msgr2=0x7f78bc071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 --2- 172.21.15.28:0/2221290950 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc071ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 -- 172.21.15.28:0/2221290950 >> v1:172.21.15.189:6789/0 conn(0x7f78bc072000 legacy=0x7f78bc0cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.477+0000 7f78bbe00640 1 -- 172.21.15.28:0/2221290950 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc070d00 con 0x7f78bc143320 2024-09-13T17:23:26.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.478+0000 7f78bbe00640 1 --2- 172.21.15.28:0/2221290950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc070440 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009870 tx=0x7f78ac02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343b8b349172d38d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78bb400640 1 -- 172.21.15.28:0/2221290950 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac032440 con 0x7f78bc143320 2024-09-13T17:23:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78bb400640 1 -- 172.21.15.28:0/2221290950 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac0325a0 con 0x7f78bc143320 2024-09-13T17:23:26.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78bb400640 1 -- 172.21.15.28:0/2221290950 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac032930 con 0x7f78bc143320 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 -- 172.21.15.28:0/2221290950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 msgr2=0x7f78bc070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 --2- 172.21.15.28:0/2221290950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc070440 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f78ac009870 tx=0x7f78ac02a210 comp rx=0 tx=0).stop 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 -- 172.21.15.28:0/2221290950 shutdown_connections 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 --2- 172.21.15.28:0/2221290950 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc071ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 --2- 172.21.15.28:0/2221290950 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc070440 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 -- 172.21.15.28:0/2221290950 >> 172.21.15.28:0/2221290950 conn(0x7f78bc069e80 msgr2=0x7f78bc06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 -- 172.21.15.28:0/2221290950 shutdown_connections 2024-09-13T17:23:26.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.479+0000 7f78c1800640 1 -- 172.21.15.28:0/2221290950 wait complete. 2024-09-13T17:23:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.481+0000 7f78c1800640 1 Processor -- start 2024-09-13T17:23:26.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.481+0000 7f78c1800640 1 -- start start 2024-09-13T17:23:26.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 0x7f78bc15d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc136f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc13b310 con 0x7f78bc143320 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc1f4a50 con 0x7f78bc072000 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78c1800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc13acc0 con 0x7f78bc143910 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54214/0 (socket says 172.21.15.28:54214) 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78b3e00640 1 -- 172.21.15.28:0/923481349 learned_addr learned my addr 172.21.15.28:0/923481349 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.482+0000 7f78bbe00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 0x7f78bc15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78c0e00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc136f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 -- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 msgr2=0x7f78bc136f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc136f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 -- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 msgr2=0x7f78bc15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 0x7f78bc15d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 -- 172.21.15.28:0/923481349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0094d0 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78bbe00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 0x7f78bc15d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b3e00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f78a400b410 tx=0x7f78a400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4011830 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a4011990 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4011d40 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc13afa0 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.483+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc15da30 con 0x7f78bc143320 2024-09-13T17:23:26.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.484+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc0b3800 con 0x7f78bc143320 2024-09-13T17:23:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.485+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f78a401a400 con 0x7f78bc143320 2024-09-13T17:23:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.486+0000 7f78b8c00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 0x7f789806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.486+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f78a4089080 con 0x7f78bc143320 2024-09-13T17:23:26.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.487+0000 7f78bbe00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 0x7f789806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.487+0000 7f78bbe00640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 0x7f789806d0c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f78ac034000 tx=0x7f78ac02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.491+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f78a40bc9c0 con 0x7f78bc143320 2024-09-13T17:23:26.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.669+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f78bc15da30 con 0x7f78bc143320 2024-09-13T17:23:26.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.673+0000 7f78b8c00640 1 -- 172.21.15.28:0/923481349 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v49) v1 ==== 104+0+0 (secure 0 0 0) 0x7f78bc15da30 con 0x7f78bc143320 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 msgr2=0x7f789806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 0x7f789806d0c0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f78ac034000 tx=0x7f78ac02e000 comp rx=0 tx=0).stop 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 msgr2=0x7f78bc136a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f78a400b410 tx=0x7f78a400b8d0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 shutdown_connections 2024-09-13T17:23:26.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f789806ac00 0x7f789806d0c0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f78bc143910 0x7f78bc136f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc143320 0x7f78bc136a10 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 --2- 172.21.15.28:0/923481349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78bc072000 0x7f78bc15d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 >> 172.21.15.28:0/923481349 conn(0x7f78bc069e80 msgr2=0x7f78bc0b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 shutdown_connections 2024-09-13T17:23:26.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.676+0000 7f78c1800640 1 -- 172.21.15.28:0/923481349 wait complete. 2024-09-13T17:23:26.695 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace' 2024-09-13T17:23:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.849+0000 7f38ea200640 1 Processor -- start 2024-09-13T17:23:26.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.849+0000 7f38ea200640 1 -- start start 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38ea200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 0x7f38e40d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38ea200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 0x7f38e40b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38ea200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e40b4d50 con 0x7f38e40b4900 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38ea200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e40d0d80 con 0x7f38e40d1d10 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38ea200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40d0960 con 0x7f38e415c750 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38e9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f38e40b4900 0x7f38e4069e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42614/0 (socket says 172.21.15.28:42614) 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.850+0000 7f38e9800640 1 -- 172.21.15.28:0/2540499499 learned_addr learned my addr 172.21.15.28:0/2540499499 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e3e00640 1 --2- 172.21.15.28:0/2540499499 >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 0x7f38e40b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2540499499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 0x7f38e40d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e3400640 1 -- 172.21.15.28:0/2540499499 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2885095630 0 0) 0x7f38dc001490 con 0x7f38e40b4900 2024-09-13T17:23:26.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e3400640 1 -- 172.21.15.28:0/2540499499 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38cc003520 con 0x7f38e40b4900 2024-09-13T17:23:26.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e8e00640 1 -- 172.21.15.28:0/2540499499 >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 msgr2=0x7f38e40b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2540499499 >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 0x7f38e40b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e8e00640 1 -- 172.21.15.28:0/2540499499 >> v1:172.21.15.28:6789/0 conn(0x7f38e40b4900 legacy=0x7f38e4069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e8e00640 1 -- 172.21.15.28:0/2540499499 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e406aa60 con 0x7f38e415c750 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.851+0000 7f38e3e00640 1 --2- 172.21.15.28:0/2540499499 >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 0x7f38e40b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.852+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2540499499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 0x7f38e40d17d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f38d80097a0 tx=0x7f38d802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9f0494854f86670 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38e3400640 1 -- 172.21.15.28:0/2540499499 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d802ae50 con 0x7f38e415c750 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38e3400640 1 -- 172.21.15.28:0/2540499499 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d8032440 con 0x7f38e415c750 2024-09-13T17:23:26.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38e3400640 1 -- 172.21.15.28:0/2540499499 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80327b0 con 0x7f38e415c750 2024-09-13T17:23:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 -- 172.21.15.28:0/2540499499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 msgr2=0x7f38e40d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 --2- 172.21.15.28:0/2540499499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 0x7f38e40d17d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f38d80097a0 tx=0x7f38d802a100 comp rx=0 tx=0).stop 2024-09-13T17:23:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 -- 172.21.15.28:0/2540499499 shutdown_connections 2024-09-13T17:23:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 --2- 172.21.15.28:0/2540499499 >> v2:172.21.15.189:3300/0 conn(0x7f38e40d1d10 0x7f38e40b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 --2- 172.21.15.28:0/2540499499 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e415c750 0x7f38e40d17d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 -- 172.21.15.28:0/2540499499 >> 172.21.15.28:0/2540499499 conn(0x7f38e40cf5a0 msgr2=0x7f38e40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 -- 172.21.15.28:0/2540499499 shutdown_connections 2024-09-13T17:23:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.853+0000 7f38ea200640 1 -- 172.21.15.28:0/2540499499 wait complete. 2024-09-13T17:23:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.855+0000 7f38ea200640 1 Processor -- start 2024-09-13T17:23:26.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.855+0000 7f38ea200640 1 -- start start 2024-09-13T17:23:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e40b4900 0x7f38e40cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e415c750 0x7f38e415e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e415d1a0 con 0x7f38e4149c30 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40cca50 con 0x7f38e415c750 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38ea200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e40cc5c0 con 0x7f38e40b4900 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38e9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38e9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54220/0 (socket says 172.21.15.28:54220) 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38e9800640 1 -- 172.21.15.28:0/2457012629 learned_addr learned my addr 172.21.15.28:0/2457012629 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38e3e00640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e415c750 0x7f38e415e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.856+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e40b4900 0x7f38e40cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 -- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e40b4900 msgr2=0x7f38e40cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e40b4900 0x7f38e40cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 -- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e415c750 msgr2=0x7f38e415e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e415c750 0x7f38e415e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:26.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 -- 172.21.15.28:0/2457012629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d0009400 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e9800640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00cb40 tx=0x7f38dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc00aca0 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc00ae00 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc012620 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d8009400 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.857+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e40cbc90 con 0x7f38e4149c30 2024-09-13T17:23:26.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.858+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e40b43c0 con 0x7f38e4149c30 2024-09-13T17:23:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.859+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f38dc007070 con 0x7f38e4149c30 2024-09-13T17:23:26.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.861+0000 7f38e0c00640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 0x7f38c406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.861+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f38dc088f40 con 0x7f38e4149c30 2024-09-13T17:23:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.861+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 0x7f38c406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:26.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.862+0000 7f38e8e00640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 0x7f38c406d0f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f38d8034000 tx=0x7f38d802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:26.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:26.865+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f38dc054380 con 0x7f38e4149c30 2024-09-13T17:23:27.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.125+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace"]} v 0) v1 -- 0x7f38e40b4d50 con 0x7f38e4149c30 2024-09-13T17:23:27.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.297+0000 7f38e0c00640 1 -- 172.21.15.28:0/2457012629 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace"]}]=0 v754) v1 ==== 133+0+0 (secure 0 0 0) 0x7f38e40b4d50 con 0x7f38e4149c30 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 msgr2=0x7f38c406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 0x7f38c406d0f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f38d8034000 tx=0x7f38d802e000 comp rx=0 tx=0).stop 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 msgr2=0x7f38e415ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 secure :-1 s=READY pgs=2553 cs=0 l=1 rev1=1 crypto rx=0x7f38dc00cb40 tx=0x7f38dc00a060 comp rx=0 tx=0).stop 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 shutdown_connections 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f38c406ac30 0x7f38c406d0f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f38e4149c30 0x7f38e415ad00 unknown :-1 s=CLOSED pgs=2553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38e415c750 0x7f38e415e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 --2- 172.21.15.28:0/2457012629 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f38e40b4900 0x7f38e40cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 >> 172.21.15.28:0/2457012629 conn(0x7f38e40cf5a0 msgr2=0x7f38e40b77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 shutdown_connections 2024-09-13T17:23:27.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.300+0000 7f38ea200640 1 -- 172.21.15.28:0/2457012629 wait complete. 2024-09-13T17:23:27.318 INFO:tasks.cephfs_test_runner:test_scrub_backtrace (tasks.cephfs.test_scrub.TestScrub) ... ok 2024-09-13T17:23:27.318 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_backtrace_for_new_files (tasks.cephfs.test_scrub.TestScrub) 2024-09-13T17:23:27.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.474+0000 7fec52e00640 1 Processor -- start 2024-09-13T17:23:27.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.474+0000 7fec52e00640 1 -- start start 2024-09-13T17:23:27.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec52e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec4c15c750 0x7fec4c0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec52e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fec4c0b4ad0 0x7fec4c069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec52e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0d0d80 con 0x7fec4c0d1d10 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec52e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c0d0960 con 0x7fec4c15c750 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec52e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4c06aa60 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec51a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec4c15c750 0x7fec4c0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec51a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fec4c15c750 0x7fec4c0d17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54234/0 (socket says 172.21.15.28:54234) 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.475+0000 7fec51a00640 1 -- 172.21.15.28:0/996560494 learned_addr learned my addr 172.21.15.28:0/996560494 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.476+0000 7fec52400640 1 --2- 172.21.15.28:0/996560494 >> v2:172.21.15.189:3300/0 conn(0x7fec4c0b4ad0 0x7fec4c069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.476+0000 7fec51a00640 1 -- 172.21.15.28:0/996560494 >> v2:172.21.15.189:3300/0 conn(0x7fec4c0b4ad0 msgr2=0x7fec4c069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.476+0000 7fec51a00640 1 --2- 172.21.15.28:0/996560494 >> v2:172.21.15.189:3300/0 conn(0x7fec4c0b4ad0 0x7fec4c069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.476+0000 7fec51a00640 1 -- 172.21.15.28:0/996560494 >> v1:172.21.15.189:6789/0 conn(0x7fec4c0d1d10 legacy=0x7fec4c0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:27.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.476+0000 7fec51a00640 1 -- 172.21.15.28:0/996560494 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c06ad40 con 0x7fec4c15c750 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.477+0000 7fec51a00640 1 --2- 172.21.15.28:0/996560494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c15c750 0x7fec4c0d17d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fec4c071420 tx=0x7fec4002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3b19b771422b6e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.477+0000 7fec4be00640 1 -- 172.21.15.28:0/996560494 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40032440 con 0x7fec4c15c750 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.477+0000 7fec4be00640 1 -- 172.21.15.28:0/996560494 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec400325a0 con 0x7fec4c15c750 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.477+0000 7fec4be00640 1 -- 172.21.15.28:0/996560494 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40032930 con 0x7fec4c15c750 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 -- 172.21.15.28:0/996560494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c15c750 msgr2=0x7fec4c0d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 --2- 172.21.15.28:0/996560494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c15c750 0x7fec4c0d17d0 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fec4c071420 tx=0x7fec4002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 -- 172.21.15.28:0/996560494 shutdown_connections 2024-09-13T17:23:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 --2- 172.21.15.28:0/996560494 >> v2:172.21.15.189:3300/0 conn(0x7fec4c0b4ad0 0x7fec4c069e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 --2- 172.21.15.28:0/996560494 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c15c750 0x7fec4c0d17d0 unknown :-1 s=CLOSED pgs=2554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 -- 172.21.15.28:0/996560494 >> 172.21.15.28:0/996560494 conn(0x7fec4c0cf5a0 msgr2=0x7fec4c0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 -- 172.21.15.28:0/996560494 shutdown_connections 2024-09-13T17:23:27.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.478+0000 7fec52e00640 1 -- 172.21.15.28:0/996560494 wait complete. 2024-09-13T17:23:27.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.479+0000 7fec52e00640 1 Processor -- start 2024-09-13T17:23:27.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 -- start start 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c15c750 0x7fec4c1ef300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c1ef840 0x7fec4c1f5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c1efc80 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c0c9420 con 0x7fec4c15c750 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.480+0000 7fec52e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4c15af80 con 0x7fec4c1ef840 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54242/0 (socket says 172.21.15.28:54242) 2024-09-13T17:23:27.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 -- 172.21.15.28:0/2374866128 learned_addr learned my addr 172.21.15.28:0/2374866128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51000640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c15c750 0x7fec4c1ef300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec52400640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c1ef840 0x7fec4c1f5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 -- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c1ef840 msgr2=0x7fec4c1f5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c1ef840 0x7fec4c1f5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 -- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c15c750 msgr2=0x7fec4c1ef300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c15c750 0x7fec4c1ef300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 -- 172.21.15.28:0/2374866128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec400094d0 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.481+0000 7fec51a00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fec40000c00 tx=0x7fec400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.482+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec40032440 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.482+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec40030b80 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.482+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec4003d420 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.482+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c15b260 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.482+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c15b820 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.483+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4c15b0c0 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.484+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fec40039040 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.485+0000 7fec49600640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 0x7fec2c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.485+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fec400a9c60 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.486+0000 7fec51000640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 0x7fec2c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.486+0000 7fec51000640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 0x7fec2c06d0c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0c5f00 tx=0x7fec38009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.489+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fec4c15b0c0 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.670+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec4c15b0c0 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.671+0000 7fec49600640 1 -- 172.21.15.28:0/2374866128 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+12994 (secure 0 0 0) 0x7fec4c15b0c0 con 0x7fec4c0b4ad0 2024-09-13T17:23:27.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:27.674 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":210,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:26.289168+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/299660514":"2024-09-14T17:23:23.276429+0000","172.21.15.28:6832/299660514":"2024-09-14T17:23:23.276429+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 msgr2=0x7fec2c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 0x7fec2c06d0c0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fec4c0c5f00 tx=0x7fec38009040 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 msgr2=0x7fec4c0c8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fec40000c00 tx=0x7fec400046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 shutdown_connections 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fec2c06ac00 0x7fec2c06d0c0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fec4c1ef840 0x7fec4c1f5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fec4c15c750 0x7fec4c1ef300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 --2- 172.21.15.28:0/2374866128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fec4c0b4ad0 0x7fec4c0c8e30 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 >> 172.21.15.28:0/2374866128 conn(0x7fec4c0cf5a0 msgr2=0x7fec4c0b7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:27.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 shutdown_connections 2024-09-13T17:23:27.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.674+0000 7fec52e00640 1 -- 172.21.15.28:0/2374866128 wait complete. 2024-09-13T17:23:27.692 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:27.692 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:27.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.848+0000 7fe413e00640 1 Processor -- start 2024-09-13T17:23:27.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.848+0000 7fe413e00640 1 -- start start 2024-09-13T17:23:27.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.849+0000 7fe413e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe414070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.849+0000 7fe413e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe4140cd120 0x7fe4140cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.849+0000 7fe413e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe414071630 con 0x7fe414143320 2024-09-13T17:23:27.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.849+0000 7fe413e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4140cdb10 con 0x7fe4140cd120 2024-09-13T17:23:27.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.849+0000 7fe413e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe414143700 con 0x7fe414143910 2024-09-13T17:23:27.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe412a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe414143320 0x7fe414071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42618/0 (socket says 172.21.15.28:42618) 2024-09-13T17:23:27.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe412a00640 1 -- 172.21.15.28:0/972910901 learned_addr learned my addr 172.21.15.28:0/972910901 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe40be00640 1 --2- 172.21.15.28:0/972910901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe414070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe413400640 1 --2- 172.21.15.28:0/972910901 >> v2:172.21.15.189:3300/0 conn(0x7fe4140cd120 0x7fe4140cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236227894 0 0) 0x7fe414071630 con 0x7fe414143320 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8003520 con 0x7fe414143320 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe40be00640 1 -- 172.21.15.28:0/972910901 >> v2:172.21.15.189:3300/0 conn(0x7fe4140cd120 msgr2=0x7fe4140cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.850+0000 7fe40be00640 1 --2- 172.21.15.28:0/972910901 >> v2:172.21.15.189:3300/0 conn(0x7fe4140cd120 0x7fe4140cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.851+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1600644893 0 0) 0x7fe3f8003520 con 0x7fe414143320 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.851+0000 7fe40be00640 1 -- 172.21.15.28:0/972910901 >> v1:172.21.15.28:6789/0 conn(0x7fe414143320 legacy=0x7fe414071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.851+0000 7fe40be00640 1 -- 172.21.15.28:0/972910901 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe414070a20 con 0x7fe414143910 2024-09-13T17:23:27.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.852+0000 7fe40be00640 1 --2- 172.21.15.28:0/972910901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe414070440 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097a0 tx=0x7fe3fc02a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d4ee8c3f9ed4d06 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.852+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe404001670 con 0x7fe414143910 2024-09-13T17:23:27.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.852+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc02aba0 con 0x7fe414143910 2024-09-13T17:23:27.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.852+0000 7fe412000640 1 -- 172.21.15.28:0/972910901 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc032440 con 0x7fe414143910 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 -- 172.21.15.28:0/972910901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 msgr2=0x7fe414070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 --2- 172.21.15.28:0/972910901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe414070440 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0097a0 tx=0x7fe3fc02a110 comp rx=0 tx=0).stop 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 -- 172.21.15.28:0/972910901 shutdown_connections 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 --2- 172.21.15.28:0/972910901 >> v2:172.21.15.189:3300/0 conn(0x7fe4140cd120 0x7fe4140cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 --2- 172.21.15.28:0/972910901 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe414070440 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 -- 172.21.15.28:0/972910901 >> 172.21.15.28:0/972910901 conn(0x7fe414069e80 msgr2=0x7fe41406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 -- 172.21.15.28:0/972910901 shutdown_connections 2024-09-13T17:23:27.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.853+0000 7fe413e00640 1 -- 172.21.15.28:0/972910901 wait complete. 2024-09-13T17:23:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.854+0000 7fe413e00640 1 Processor -- start 2024-09-13T17:23:27.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.854+0000 7fe413e00640 1 -- start start 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140cd120 0x7fe41413c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe41413eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41413ca40 con 0x7fe414143320 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140cc020 con 0x7fe4140cd120 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.855+0000 7fe413e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4140cc300 con 0x7fe414143910 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54248/0 (socket says 172.21.15.28:54248) 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 -- 172.21.15.28:0/1950817075 learned_addr learned my addr 172.21.15.28:0/1950817075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe413400640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe41413eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe412a00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140cd120 0x7fe41413c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 -- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 msgr2=0x7fe41413eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe41413eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 -- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140cd120 msgr2=0x7fe41413c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140cd120 0x7fe41413c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 -- 172.21.15.28:0/1950817075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc009400 con 0x7fe414143320 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.856+0000 7fe40be00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.857+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc03c820 con 0x7fe414143320 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.857+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc03c980 con 0x7fe414143320 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.857+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc03ccf0 con 0x7fe414143320 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.857+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4140cc5e0 con 0x7fe414143320 2024-09-13T17:23:27.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.857+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe414159f00 con 0x7fe414143320 2024-09-13T17:23:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.858+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4140b3800 con 0x7fe414143320 2024-09-13T17:23:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.859+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe3fc039050 con 0x7fe414143320 2024-09-13T17:23:27.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.860+0000 7fe40aa00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 0x7fe3f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:27.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.860+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe3fc0a9980 con 0x7fe414143320 2024-09-13T17:23:27.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.860+0000 7fe412a00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 0x7fe3f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:27.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.861+0000 7fe412a00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 0x7fe3f006d0c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe404005880 tx=0x7fe404001ce0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:27.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:27.864+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe3fc075020 con 0x7fe414143320 2024-09-13T17:23:28.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.045+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe414144010 con 0x7fe414143320 2024-09-13T17:23:28.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.045+0000 7fe40aa00640 1 -- 172.21.15.28:0/1950817075 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v210) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3fc037020 con 0x7fe414143320 2024-09-13T17:23:28.047 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:28.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 msgr2=0x7fe3f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 0x7fe3f006d0c0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fe404005880 tx=0x7fe404001ce0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 msgr2=0x7fe4140c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098c0 tx=0x7fe3fc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 shutdown_connections 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe3f006ac00 0x7fe3f006d0c0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe414143910 0x7fe41413eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe414143320 0x7fe4140c9b60 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 --2- 172.21.15.28:0/1950817075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4140cd120 0x7fe41413c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 >> 172.21.15.28:0/1950817075 conn(0x7fe414069e80 msgr2=0x7fe414071ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 shutdown_connections 2024-09-13T17:23:28.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.047+0000 7fe413e00640 1 -- 172.21.15.28:0/1950817075 wait complete. 2024-09-13T17:23:28.065 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files' 2024-09-13T17:23:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.219+0000 7efc5f200640 1 Processor -- start 2024-09-13T17:23:28.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.219+0000 7efc5f200640 1 -- start start 2024-09-13T17:23:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.220+0000 7efc5f200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc6015fb90 0x7efc600b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.220+0000 7efc5f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600cad20 con 0x7efc600b0440 2024-09-13T17:23:28.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.220+0000 7efc5f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc6006bb40 con 0x7efc600b21c0 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.220+0000 7efc5f200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc6006be20 con 0x7efc6015fb90 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.220+0000 7efc5de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc6015fb90 0x7efc600b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efc600b0440 0x7efc600cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42622/0 (socket says 172.21.15.28:42622) 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5e800640 1 -- 172.21.15.28:0/892040286 learned_addr learned my addr 172.21.15.28:0/892040286 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5de00640 1 -- 172.21.15.28:0/892040286 >> v1:172.21.15.189:6789/0 conn(0x7efc600b21c0 legacy=0x7efc6006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5de00640 1 -- 172.21.15.28:0/892040286 >> v1:172.21.15.28:6789/0 conn(0x7efc600b0440 legacy=0x7efc600cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5de00640 1 -- 172.21.15.28:0/892040286 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc6006c100 con 0x7efc6015fb90 2024-09-13T17:23:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.221+0000 7efc5ca00640 1 -- 172.21.15.28:0/892040286 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819350809 0 0) 0x7efc600cad20 con 0x7efc600b0440 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.222+0000 7efc5de00640 1 --2- 172.21.15.28:0/892040286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc600b19d0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7efc500097a0 tx=0x7efc5002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f9c2d0dce3ae12 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.222+0000 7efc5ca00640 1 -- 172.21.15.28:0/892040286 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5002adc0 con 0x7efc6015fb90 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.222+0000 7efc5ca00640 1 -- 172.21.15.28:0/892040286 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50032440 con 0x7efc6015fb90 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.222+0000 7efc5ca00640 1 -- 172.21.15.28:0/892040286 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500327d0 con 0x7efc6015fb90 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 -- 172.21.15.28:0/892040286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 msgr2=0x7efc600b19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 --2- 172.21.15.28:0/892040286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc600b19d0 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7efc500097a0 tx=0x7efc5002a0b0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 -- 172.21.15.28:0/892040286 shutdown_connections 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 --2- 172.21.15.28:0/892040286 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc600b19d0 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 -- 172.21.15.28:0/892040286 >> 172.21.15.28:0/892040286 conn(0x7efc60072e40 msgr2=0x7efc60073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 -- 172.21.15.28:0/892040286 shutdown_connections 2024-09-13T17:23:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.223+0000 7efc5f200640 1 -- 172.21.15.28:0/892040286 wait complete. 2024-09-13T17:23:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.225+0000 7efc5f200640 1 Processor -- start 2024-09-13T17:23:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.225+0000 7efc5f200640 1 -- start start 2024-09-13T17:23:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc600b0440 0x7efc601f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc600b21c0 0x7efc60155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6015f410 con 0x7efc6015fb90 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc60155b70 con 0x7efc600b0440 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc60152d30 con 0x7efc600b21c0 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54256/0 (socket says 172.21.15.28:54256) 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 -- 172.21.15.28:0/387293649 learned_addr learned my addr 172.21.15.28:0/387293649 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5de00640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc600b0440 0x7efc601f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5d400640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc600b21c0 0x7efc60155630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 -- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc600b21c0 msgr2=0x7efc60155630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc600b21c0 0x7efc60155630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 -- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc600b0440 msgr2=0x7efc601f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc600b0440 0x7efc601f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.226+0000 7efc5e800640 1 -- 172.21.15.28:0/387293649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc50009400 con 0x7efc6015fb90 2024-09-13T17:23:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc5e800640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7efc58002230 tx=0x7efc5800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5800ce30 con 0x7efc6015fb90 2024-09-13T17:23:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc5800a640 con 0x7efc6015fb90 2024-09-13T17:23:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc5800a9f0 con 0x7efc6015fb90 2024-09-13T17:23:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc60155da0 con 0x7efc6015fb90 2024-09-13T17:23:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.227+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc60152f80 con 0x7efc6015fb90 2024-09-13T17:23:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.228+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc600b0c00 con 0x7efc6015fb90 2024-09-13T17:23:28.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.229+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efc5800ab50 con 0x7efc6015fb90 2024-09-13T17:23:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.230+0000 7efc56000640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 0x7efc4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:28.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.231+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efc580884d0 con 0x7efc6015fb90 2024-09-13T17:23:28.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.231+0000 7efc5de00640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 0x7efc4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:28.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.232+0000 7efc5de00640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 0x7efc4006d0c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efc50034000 tx=0x7efc5002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:28.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.235+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efc58053a50 con 0x7efc6015fb90 2024-09-13T17:23:28.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:28.479+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files"]} v 0) v1 -- 0x7efc600afd20 con 0x7efc6015fb90 2024-09-13T17:23:29.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.305+0000 7efc56000640 1 -- 172.21.15.28:0/387293649 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files"]}]=0 v756) v1 ==== 150+0+0 (secure 0 0 0) 0x7efc58018020 con 0x7efc6015fb90 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.307+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 msgr2=0x7efc4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.307+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 0x7efc4006d0c0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7efc50034000 tx=0x7efc5002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 msgr2=0x7efc60158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7efc58002230 tx=0x7efc5800cbe0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 shutdown_connections 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc4006ac00 0x7efc4006d0c0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc6015fb90 0x7efc60158750 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc600b21c0 0x7efc60155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 --2- 172.21.15.28:0/387293649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc600b0440 0x7efc601f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 >> 172.21.15.28:0/387293649 conn(0x7efc60072e40 msgr2=0x7efc6006fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 shutdown_connections 2024-09-13T17:23:29.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.308+0000 7efc5f200640 1 -- 172.21.15.28:0/387293649 wait complete. 2024-09-13T17:23:29.328 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:23:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.492+0000 7f488d200640 1 Processor -- start 2024-09-13T17:23:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.492+0000 7f488d200640 1 -- start start 2024-09-13T17:23:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.493+0000 7f488d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.493+0000 7f488d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 0x7f4888070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.493+0000 7f488d200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488806f920 con 0x7f48880add00 2024-09-13T17:23:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.493+0000 7f488d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488815b290 con 0x7f4888159770 2024-09-13T17:23:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.493+0000 7f488d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488815b570 con 0x7f488806f3f0 2024-09-13T17:23:29.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 0x7f4888070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:58244/0 (socket says 172.21.15.28:58244) 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885e00640 1 -- 172.21.15.28:0/260762264 learned_addr learned my addr 172.21.15.28:0/260762264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885400640 1 -- 172.21.15.28:0/260762264 >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 msgr2=0x7f488806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885400640 1 --2- 172.21.15.28:0/260762264 >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885400640 1 -- 172.21.15.28:0/260762264 >> v1:172.21.15.189:6789/0 conn(0x7f48880add00 legacy=0x7f4888159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885400640 1 -- 172.21.15.28:0/260762264 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488815b850 con 0x7f488806f3f0 2024-09-13T17:23:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.494+0000 7f4885e00640 1 --2- 172.21.15.28:0/260762264 >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.495+0000 7f4885400640 1 --2- 172.21.15.28:0/260762264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 0x7f4888070260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f48740097a0 tx=0x7f487402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=537d4fc2c20b339b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f4884a00640 1 -- 172.21.15.28:0/260762264 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487402ae50 con 0x7f488806f3f0 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f4884a00640 1 -- 172.21.15.28:0/260762264 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4874032440 con 0x7f488806f3f0 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f4884a00640 1 -- 172.21.15.28:0/260762264 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48740327d0 con 0x7f488806f3f0 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 -- 172.21.15.28:0/260762264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 msgr2=0x7f4888070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 --2- 172.21.15.28:0/260762264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 0x7f4888070260 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f48740097a0 tx=0x7f487402a100 comp rx=0 tx=0).stop 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 -- 172.21.15.28:0/260762264 shutdown_connections 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 --2- 172.21.15.28:0/260762264 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f488806f3f0 0x7f4888070260 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 --2- 172.21.15.28:0/260762264 >> v2:172.21.15.189:3300/0 conn(0x7f4888159770 0x7f488806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 -- 172.21.15.28:0/260762264 >> 172.21.15.28:0/260762264 conn(0x7f48880b0270 msgr2=0x7f48880b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 -- 172.21.15.28:0/260762264 shutdown_connections 2024-09-13T17:23:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.496+0000 7f488d200640 1 -- 172.21.15.28:0/260762264 wait complete. 2024-09-13T17:23:29.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.498+0000 7f488d200640 1 Processor -- start 2024-09-13T17:23:29.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.498+0000 7f488d200640 1 -- start start 2024-09-13T17:23:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488806f3f0 0x7f488814cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48880add00 0x7f4888157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48880c42d0 con 0x7f4888159770 2024-09-13T17:23:29.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4888158210 con 0x7f488806f3f0 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f488d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48880c3930 con 0x7f48880add00 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f4886800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f4886800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54272/0 (socket says 172.21.15.28:54272) 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f4885e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488806f3f0 0x7f488814cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f4885400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48880add00 0x7f4888157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.499+0000 7f4886800640 1 -- 172.21.15.28:0/1991902332 learned_addr learned my addr 172.21.15.28:0/1991902332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 -- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48880add00 msgr2=0x7f4888157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48880add00 0x7f4888157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 -- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488806f3f0 msgr2=0x7f488814cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488806f3f0 0x7f488814cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 -- 172.21.15.28:0/1991902332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48780094d0 con 0x7f4888159770 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f4886800640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f488000c6b0 tx=0x7f488000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880003b70 con 0x7f4888159770 2024-09-13T17:23:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4880003cf0 con 0x7f4888159770 2024-09-13T17:23:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4874009400 con 0x7f4888159770 2024-09-13T17:23:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.500+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4880005310 con 0x7f4888159770 2024-09-13T17:23:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.501+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48880c3ef0 con 0x7f4888159770 2024-09-13T17:23:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.502+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48880c4410 con 0x7f4888159770 2024-09-13T17:23:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.503+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4880005470 con 0x7f4888159770 2024-09-13T17:23:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.503+0000 7f487e000640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 0x7f486806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.504+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4880088d20 con 0x7f4888159770 2024-09-13T17:23:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.504+0000 7f4885e00640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 0x7f486806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.505+0000 7f4885e00640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 0x7f486806d0c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4878006ca0 tx=0x7f4878009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.507+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f48880c4410 con 0x7f4888159770 2024-09-13T17:23:29.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.684+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f48880c4410 con 0x7f4888159770 2024-09-13T17:23:29.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.688+0000 7f487e000640 1 -- 172.21.15.28:0/1991902332 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v50)=0 v50) v1 ==== 122+0+0 (secure 0 0 0) 0x7f48880c4410 con 0x7f4888159770 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 msgr2=0x7f486806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 0x7f486806d0c0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f4878006ca0 tx=0x7f4878009240 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 msgr2=0x7f48880c60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f488000c6b0 tx=0x7f488000cb70 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 shutdown_connections 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.690+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f486806ac00 0x7f486806d0c0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4888159770 0x7f48880c60b0 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f48880add00 0x7f4888157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 --2- 172.21.15.28:0/1991902332 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f488806f3f0 0x7f488814cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 >> 172.21.15.28:0/1991902332 conn(0x7f48880b0270 msgr2=0x7f48880ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 shutdown_connections 2024-09-13T17:23:29.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:29.691+0000 7f488d200640 1 -- 172.21.15.28:0/1991902332 wait complete. 2024-09-13T17:23:29.708 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:23:29.708 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:29.792 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:29.901 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:23:29.902 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:29.902 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:29.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:23:29.938 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:29.938 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:29.976 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:29.977 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:29.977 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:30.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:23:30.013 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:30.013 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:30.053 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:30.053 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:23:30.053 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.206+0000 7f5ecb200640 1 Processor -- start 2024-09-13T17:23:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.207+0000 7f5ecb200640 1 -- start start 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5ecc0b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5ecb200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0cad20 con 0x7f5ecc0b0440 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5ecb200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc06bb40 con 0x7f5ecc0b21c0 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5ecb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc06be20 con 0x7f5ecc15fb90 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5eca800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5ecc0b0440 0x7f5ecc0cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42632/0 (socket says 172.21.15.28:42632) 2024-09-13T17:23:30.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5eca800640 1 -- 172.21.15.28:0/2142854102 learned_addr learned my addr 172.21.15.28:0/2142854102 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.208+0000 7f5ec9e00640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5ecc0b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159591837 0 0) 0x7f5ecc0cad20 con 0x7f5ecc0b0440 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb0003520 con 0x7f5ecc0b0440 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1774645176 0 0) 0x7f5eb0003520 con 0x7f5ecc0b0440 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 msgr2=0x7f5ecc0b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:30.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5ecc0b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 >> v1:172.21.15.189:6789/0 conn(0x7f5ecc0b21c0 legacy=0x7f5ecc06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc06c100 con 0x7f5ecc0b0440 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f5ec40019f0 con 0x7f5ecc0b0440 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 >> v1:172.21.15.28:6789/0 conn(0x7f5ecc0b0440 legacy=0x7f5ecc0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb00039a0 0x7f5eb0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5eb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb0004d00 0x7f5eb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc06be20 con 0x7f5eb0004d00 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb000bb40 con 0x7f5eb00039a0 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb000bd00 con 0x7f5ecc15fb90 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5eca800640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb0004d00 0x7f5eb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec9e00640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb00039a0 0x7f5eb0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.209+0000 7f5ec3e00640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5eb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.210+0000 7f5eca800640 1 -- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 msgr2=0x7f5eb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.210+0000 7f5eca800640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc15fb90 0x7f5eb00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.210+0000 7f5eca800640 1 -- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb00039a0 msgr2=0x7f5eb0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.210+0000 7f5eca800640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb00039a0 0x7f5eb0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.210+0000 7f5eca800640 1 -- 172.21.15.28:0/2142854102 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0003520 con 0x7f5eb0004d00 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.211+0000 7f5eca800640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb0004d00 0x7f5eb0005180 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400a830 tx=0x7f5eb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f540ae0cff6f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.211+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec402ba50 con 0x7f5eb0004d00 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.211+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec402bbb0 con 0x7f5eb0004d00 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.211+0000 7f5ec9400640 1 -- 172.21.15.28:0/2142854102 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec4034470 con 0x7f5eb0004d00 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 -- 172.21.15.28:0/2142854102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb0004d00 msgr2=0x7f5eb0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 --2- 172.21.15.28:0/2142854102 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5eb0004d00 0x7f5eb0005180 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f5ec400a830 tx=0x7f5eb00057f0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5eca800640 1 -- 172.21.15.28:0/2142854102 reap_dead start 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 -- 172.21.15.28:0/2142854102 shutdown_connections 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 -- 172.21.15.28:0/2142854102 >> 172.21.15.28:0/2142854102 conn(0x7f5ecc072e40 msgr2=0x7f5ecc073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 -- 172.21.15.28:0/2142854102 shutdown_connections 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.212+0000 7f5ecb200640 1 -- 172.21.15.28:0/2142854102 wait complete. 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.213+0000 7f5ecb200640 1 Processor -- start 2024-09-13T17:23:30.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 -- start start 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb0004d00 0x7f5ecc1f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc150b90 0x7f5ecc151040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc157fa0 con 0x7f5ecc1518a0 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc1f31d0 con 0x7f5eb0004d00 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.214+0000 7f5ecb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc151580 con 0x7f5ecc150b90 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54294/0 (socket says 172.21.15.28:54294) 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 -- 172.21.15.28:0/1818918773 learned_addr learned my addr 172.21.15.28:0/1818918773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec9e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb0004d00 0x7f5ecc1f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5eca800640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc150b90 0x7f5ecc151040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 -- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc150b90 msgr2=0x7f5ecc151040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc150b90 0x7f5ecc151040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 -- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb0004d00 msgr2=0x7f5ecc1f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb0004d00 0x7f5ecc1f2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 -- 172.21.15.28:0/1818918773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc151d60 con 0x7f5ecc1518a0 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.215+0000 7f5ec3e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400f7f0 tx=0x7f5eb400fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:30.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.216+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb40050d0 con 0x7f5ecc1518a0 2024-09-13T17:23:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.216+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb4005230 con 0x7f5ecc1518a0 2024-09-13T17:23:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.216+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb400d730 con 0x7f5ecc1518a0 2024-09-13T17:23:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.216+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc152040 con 0x7f5ecc1518a0 2024-09-13T17:23:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.216+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc152600 con 0x7f5ecc1518a0 2024-09-13T17:23:30.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.217+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc0b08d0 con 0x7f5ecc1518a0 2024-09-13T17:23:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.218+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5eb400d890 con 0x7f5ecc1518a0 2024-09-13T17:23:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.219+0000 7f5ec2000640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 0x7f5ea806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.219+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5eb4088ea0 con 0x7f5ecc1518a0 2024-09-13T17:23:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.219+0000 7f5ec9e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 0x7f5ea806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:30.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.220+0000 7f5ec9e00640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 0x7f5ea806d0c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc153080 tx=0x7f5ebc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:30.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.223+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5eb40542e0 con 0x7f5ecc1518a0 2024-09-13T17:23:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.437+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ecc15b230 con 0x7f5ecc1518a0 2024-09-13T17:23:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.437+0000 7f5ec2000640 1 -- 172.21.15.28:0/1818918773 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+3543 (secure 0 0 0) 0x7f5eb4057900 con 0x7f5ecc1518a0 2024-09-13T17:23:30.439 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:30.440 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":174,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9029,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2561687696","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2561687696},{"type":"v1","addr":"172.21.15.189:6837","nonce":2561687696}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10543,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3340307573","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3340307573},{"type":"v1","addr":"172.21.15.28:6835","nonce":3340307573}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":10546,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/3851840632","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3851840632},{"type":"v1","addr":"172.21.15.189:6836","nonce":3851840632}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":168},{"gid":11844,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/1336167386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":1336167386},{"type":"v1","addr":"172.21.15.28:6834","nonce":1336167386}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":174}],"filesystems":[]} 2024-09-13T17:23:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 174 2024-09-13T17:23:30.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.439+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 msgr2=0x7f5ea806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.439+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 0x7f5ea806d0c0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc153080 tx=0x7f5ebc009190 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.439+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 msgr2=0x7f5ecc0c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.439+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400f7f0 tx=0x7f5eb400fbf0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 shutdown_connections 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ea806ac00 0x7f5ea806d0c0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5ecc150b90 0x7f5ecc151040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5ecc1518a0 0x7f5ecc0c8d80 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 --2- 172.21.15.28:0/1818918773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5eb0004d00 0x7f5ecc1f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 >> 172.21.15.28:0/1818918773 conn(0x7f5ecc072e40 msgr2=0x7f5ecc0cb390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 shutdown_connections 2024-09-13T17:23:30.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:30.440+0000 7f5ecb200640 1 -- 172.21.15.28:0/1818918773 wait complete. 2024-09-13T17:23:30.457 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:23:30.458 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:23:30.458 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:23:30.458 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:23:30.458 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:23:30.458 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:23:30.458 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:23:30.458 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:23:30.459 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:23:30.459 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:23:30.459 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:23:30.459 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:23:30.459 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:23:30.459 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:23:30.459 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:23:30.460 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:23:30.473 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:23:30.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:23:30.474 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:23:30.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:23:30.477 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:23:30.477 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:23:30.518 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:23:30.518 DEBUG:teuthology.parallel:result is None 2024-09-13T17:23:30.520 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:23:30.520 DEBUG:teuthology.parallel:result is None 2024-09-13T17:23:30.522 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:23:30.522 DEBUG:teuthology.parallel:result is None 2024-09-13T17:23:30.646 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:23:30.646 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:23:30.649 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:23:36.279 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-09-13T17:23:36.477 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:23:36.478 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:23:36.483 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:23:36.483 DEBUG:teuthology.parallel:result is None 2024-09-13T17:23:36.483 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:23:36.617 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:23:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.650+0000 7f4fda800640 1 Processor -- start 2024-09-13T17:23:36.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.650+0000 7f4fda800640 1 -- start start 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.651+0000 7f4fda800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4fd415a780 0x7f4fd4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.651+0000 7f4fda800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fd415ab60 0x7f4fd40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.651+0000 7f4fda800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd406c660 con 0x7f4fd406bb40 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.651+0000 7f4fda800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd406c8b0 con 0x7f4fd415ab60 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.651+0000 7f4fda800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fd406cb90 con 0x7f4fd415a780 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fd415ab60 0x7f4fd40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4fd415ab60 0x7f4fd40712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34232/0 (socket says 172.21.15.28:34232) 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 -- 172.21.15.28:0/132773683 learned_addr learned my addr 172.21.15.28:0/132773683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:36.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fd3400640 1 --2- 172.21.15.28:0/132773683 >> v2:172.21.15.189:3300/0 conn(0x7f4fd415a780 0x7f4fd4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 -- 172.21.15.28:0/132773683 >> v2:172.21.15.189:3300/0 conn(0x7f4fd415a780 msgr2=0x7f4fd4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 --2- 172.21.15.28:0/132773683 >> v2:172.21.15.189:3300/0 conn(0x7f4fd415a780 0x7f4fd4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 -- 172.21.15.28:0/132773683 >> v1:172.21.15.28:6789/0 conn(0x7f4fd406bb40 legacy=0x7f4fd406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.652+0000 7f4fcbe00640 1 -- 172.21.15.28:0/132773683 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd40cc980 con 0x7f4fd415ab60 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.653+0000 7f4fcbe00640 1 --2- 172.21.15.28:0/132773683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415ab60 0x7f4fd40712c0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f4fd40b1330 tx=0x7f4fbc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b883daedcf27dd41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.653+0000 7f4fd2a00640 1 -- 172.21.15.28:0/132773683 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc032440 con 0x7f4fd415ab60 2024-09-13T17:23:36.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fd2a00640 1 -- 172.21.15.28:0/132773683 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc0325a0 con 0x7f4fd415ab60 2024-09-13T17:23:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fd2a00640 1 -- 172.21.15.28:0/132773683 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc032930 con 0x7f4fd415ab60 2024-09-13T17:23:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 -- 172.21.15.28:0/132773683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415ab60 msgr2=0x7f4fd40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 --2- 172.21.15.28:0/132773683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415ab60 0x7f4fd40712c0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7f4fd40b1330 tx=0x7f4fbc02a270 comp rx=0 tx=0).stop 2024-09-13T17:23:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 -- 172.21.15.28:0/132773683 shutdown_connections 2024-09-13T17:23:36.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 --2- 172.21.15.28:0/132773683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415ab60 0x7f4fd40712c0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 --2- 172.21.15.28:0/132773683 >> v2:172.21.15.189:3300/0 conn(0x7f4fd415a780 0x7f4fd4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 -- 172.21.15.28:0/132773683 >> 172.21.15.28:0/132773683 conn(0x7f4fd40b0270 msgr2=0x7f4fd40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 -- 172.21.15.28:0/132773683 shutdown_connections 2024-09-13T17:23:36.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.654+0000 7f4fda800640 1 -- 172.21.15.28:0/132773683 wait complete. 2024-09-13T17:23:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.656+0000 7f4fda800640 1 Processor -- start 2024-09-13T17:23:36.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.656+0000 7f4fda800640 1 -- start start 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 0x7f4fd41540e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 0x7f4fd414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd414f620 con 0x7f4fd415a780 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd4156810 con 0x7f4fd406bb40 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fda800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd4155800 con 0x7f4fd415ab60 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34234/0 (socket says 172.21.15.28:34234) 2024-09-13T17:23:36.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.657+0000 7f4fcbe00640 1 -- 172.21.15.28:0/3745661597 learned_addr learned my addr 172.21.15.28:0/3745661597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fd3400640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 0x7f4fd41540e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fd3e00640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 0x7f4fd414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 -- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 msgr2=0x7f4fd414f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 0x7f4fd414f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 -- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 msgr2=0x7f4fd41540e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 0x7f4fd41540e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 -- 172.21.15.28:0/3745661597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fbc0094d0 con 0x7f4fd415a780 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fd3e00640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 0x7f4fd414f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fd3400640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 0x7f4fd41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcbe00640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc032440 con 0x7f4fd415a780 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc030ba0 con 0x7f4fd415a780 2024-09-13T17:23:36.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc03d440 con 0x7f4fd415a780 2024-09-13T17:23:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.658+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd4155a50 con 0x7f4fd415a780 2024-09-13T17:23:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.659+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd4156010 con 0x7f4fd415a780 2024-09-13T17:23:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.660+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd406bf50 con 0x7f4fd415a780 2024-09-13T17:23:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.660+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4fbc039040 con 0x7f4fd415a780 2024-09-13T17:23:36.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.662+0000 7f4fcb400640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 0x7f4fb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.662+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4fbc0a9c70 con 0x7f4fd415a780 2024-09-13T17:23:36.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.662+0000 7f4fd3400640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 0x7f4fb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:36.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.663+0000 7f4fd3400640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 0x7f4fb006d0c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:36.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.666+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4fbc0750c0 con 0x7f4fd415a780 2024-09-13T17:23:36.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:36.859+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f4fd40afef0 con 0x7f4fd415a780 2024-09-13T17:23:37.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.636+0000 7f4fcb400640 1 -- 172.21.15.28:0/3745661597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v215) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4fbc0786e0 con 0x7f4fd415a780 2024-09-13T17:23:37.638 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 msgr2=0x7f4fb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 0x7f4fb006d0c0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4006fd0 tx=0x7f4fc4008040 comp rx=0 tx=0).stop 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 msgr2=0x7f4fd414eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc000c00 tx=0x7f4fbc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 shutdown_connections 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4fb006ac00 0x7f4fb006d0c0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4fd415ab60 0x7f4fd414f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4fd415a780 0x7f4fd414eba0 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 --2- 172.21.15.28:0/3745661597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fd406bb40 0x7f4fd41540e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 >> 172.21.15.28:0/3745661597 conn(0x7f4fd40b0270 msgr2=0x7f4fd40caa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 shutdown_connections 2024-09-13T17:23:37.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.639+0000 7f4fda800640 1 -- 172.21.15.28:0/3745661597 wait complete. 2024-09-13T17:23:37.661 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:23:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.819+0000 7fd3f9e00640 1 Processor -- start 2024-09-13T17:23:37.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.819+0000 7fd3f9e00640 1 -- start start 2024-09-13T17:23:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.820+0000 7fd3f9e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.820+0000 7fd3f9e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3f406c2d0 0x7fd3f406c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.820+0000 7fd3f9e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f4159d90 con 0x7fd3f41592a0 2024-09-13T17:23:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.820+0000 7fd3f9e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f415b0b0 con 0x7fd3f406c2d0 2024-09-13T17:23:37.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.820+0000 7fd3f9e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f406cca0 con 0x7fd3f406f970 2024-09-13T17:23:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f3400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3f406c2d0 0x7fd3f406c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:33538/0 (socket says 172.21.15.28:33538) 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 -- 172.21.15.28:0/1224184446 learned_addr learned my addr 172.21.15.28:0/1224184446 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 -- 172.21.15.28:0/1224184446 >> v2:172.21.15.189:3300/0 conn(0x7fd3f406c2d0 msgr2=0x7fd3f406c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 --2- 172.21.15.28:0/1224184446 >> v2:172.21.15.189:3300/0 conn(0x7fd3f406c2d0 0x7fd3f406c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 -- 172.21.15.28:0/1224184446 >> v1:172.21.15.189:6789/0 conn(0x7fd3f41592a0 legacy=0x7fd3f4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.821+0000 7fd3f2000640 1 -- 172.21.15.28:0/1224184446 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f40b2040 con 0x7fd3f406f970 2024-09-13T17:23:37.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.822+0000 7fd3f2000640 1 --2- 172.21.15.28:0/1224184446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009870 tx=0x7fd3e002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=787a37edddf4b89f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f1600640 1 -- 172.21.15.28:0/1224184446 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032440 con 0x7fd3f406f970 2024-09-13T17:23:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f1600640 1 -- 172.21.15.28:0/1224184446 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e00325a0 con 0x7fd3f406f970 2024-09-13T17:23:37.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f1600640 1 -- 172.21.15.28:0/1224184446 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032930 con 0x7fd3f406f970 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 -- 172.21.15.28:0/1224184446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 msgr2=0x7fd3f415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/1224184446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009870 tx=0x7fd3e002a250 comp rx=0 tx=0).stop 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 -- 172.21.15.28:0/1224184446 shutdown_connections 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/1224184446 >> v2:172.21.15.189:3300/0 conn(0x7fd3f406c2d0 0x7fd3f406c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/1224184446 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f406f970 0x7fd3f415ab70 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.823+0000 7fd3f9e00640 1 -- 172.21.15.28:0/1224184446 >> 172.21.15.28:0/1224184446 conn(0x7fd3f4069e80 msgr2=0x7fd3f406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.824+0000 7fd3f9e00640 1 -- 172.21.15.28:0/1224184446 shutdown_connections 2024-09-13T17:23:37.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.824+0000 7fd3f9e00640 1 -- 172.21.15.28:0/1224184446 wait complete. 2024-09-13T17:23:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.825+0000 7fd3f9e00640 1 Processor -- start 2024-09-13T17:23:37.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.825+0000 7fd3f9e00640 1 -- start start 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f406c2d0 0x7fd3f4153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f41592a0 0x7fd3f40c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41f5a50 con 0x7fd3f406f970 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41f2960 con 0x7fd3f406c2d0 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.826+0000 7fd3f9e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f41f2c40 con 0x7fd3f41592a0 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34238/0 (socket says 172.21.15.28:34238) 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 -- 172.21.15.28:0/3259723935 learned_addr learned my addr 172.21.15.28:0/3259723935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:37.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2a00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f406c2d0 0x7fd3f4153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f3400640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f41592a0 0x7fd3f40c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 -- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f41592a0 msgr2=0x7fd3f40c3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f41592a0 0x7fd3f40c3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 -- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f406c2d0 msgr2=0x7fd3f4153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f406c2d0 0x7fd3f4153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:37.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 -- 172.21.15.28:0/3259723935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00094d0 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.827+0000 7fd3f2000640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0000c00 tx=0x7fd3e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.828+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032440 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.828+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0030980 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.828+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f41f2f20 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.828+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0030d30 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.828+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f4154240 con 0x7fd3f406f970 2024-09-13T17:23:37.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.829+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f41f5bc0 con 0x7fd3f406f970 2024-09-13T17:23:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.830+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3e0039040 con 0x7fd3f406f970 2024-09-13T17:23:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.831+0000 7fd3eb400640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:37.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.831+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd3e00a9a20 con 0x7fd3f406f970 2024-09-13T17:23:37.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.831+0000 7fd3f2a00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:37.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.832+0000 7fd3f2a00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4007fd0 tx=0x7fd3e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:37.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:37.835+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd3e0074ec0 con 0x7fd3f406f970 2024-09-13T17:23:38.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.075+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd3f40b3740 con 0x7fd3f406f970 2024-09-13T17:23:38.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.075+0000 7fd3eb400640 1 -- 172.21.15.28:0/3259723935 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd3e00784e0 con 0x7fd3f406f970 2024-09-13T17:23:38.077 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:23:38.078 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:23:38.079 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.080 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.081 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:23:38.082 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.083 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:23:38.084 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:23:38.086 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:23:38.087 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:23:38.088 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:23:38.089 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:23:38.090 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:23:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:38.091 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:23:38.091 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 msgr2=0x7fd3d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4007fd0 tx=0x7fd3e4009040 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 msgr2=0x7fd3f40ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0000c00 tx=0x7fd3e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 shutdown_connections 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f41592a0 0x7fd3f40c3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f406f970 0x7fd3f40ced30 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 --2- 172.21.15.28:0/3259723935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f406c2d0 0x7fd3f4153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.077+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 >> 172.21.15.28:0/3259723935 conn(0x7fd3f4069e80 msgr2=0x7fd3f4159680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:38.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.078+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 shutdown_connections 2024-09-13T17:23:38.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.078+0000 7fd3f9e00640 1 -- 172.21.15.28:0/3259723935 wait complete. 2024-09-13T17:23:38.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.249+0000 7fc379800640 1 Processor -- start 2024-09-13T17:23:38.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.249+0000 7fc379800640 1 -- start start 2024-09-13T17:23:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc379800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc3740acf60 0x7fc374160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc379800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc374146550 con 0x7fc374146170 2024-09-13T17:23:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc379800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37415d280 con 0x7fc374160310 2024-09-13T17:23:38.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc379800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37415cd20 con 0x7fc3740acf60 2024-09-13T17:23:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc372a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc374146170 0x7fc37415d710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38534/0 (socket says 172.21.15.28:38534) 2024-09-13T17:23:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.250+0000 7fc372a00640 1 -- 172.21.15.28:0/2529826012 learned_addr learned my addr 172.21.15.28:0/2529826012 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc373400640 1 --2- 172.21.15.28:0/2529826012 >> v2:172.21.15.189:3300/0 conn(0x7fc3740acf60 0x7fc374160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc371600640 1 -- 172.21.15.28:0/2529826012 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634178666 0 0) 0x7fc374146550 con 0x7fc374146170 2024-09-13T17:23:38.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc371600640 1 -- 172.21.15.28:0/2529826012 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc358003520 con 0x7fc374146170 2024-09-13T17:23:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc373400640 1 -- 172.21.15.28:0/2529826012 >> v1:172.21.15.189:6789/0 conn(0x7fc374160310 legacy=0x7fc3740ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc373400640 1 -- 172.21.15.28:0/2529826012 >> v1:172.21.15.28:6789/0 conn(0x7fc374146170 legacy=0x7fc37415d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.251+0000 7fc373400640 1 -- 172.21.15.28:0/2529826012 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374160d60 con 0x7fc3740acf60 2024-09-13T17:23:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.252+0000 7fc373400640 1 --2- 172.21.15.28:0/2529826012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3740acf60 0x7fc374160730 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d90 tx=0x7fc36c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad373bf4d5a872b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc371600640 1 -- 172.21.15.28:0/2529826012 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c007740 con 0x7fc3740acf60 2024-09-13T17:23:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc371600640 1 -- 172.21.15.28:0/2529826012 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc36c02d050 con 0x7fc3740acf60 2024-09-13T17:23:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc371600640 1 -- 172.21.15.28:0/2529826012 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc36c007aa0 con 0x7fc3740acf60 2024-09-13T17:23:38.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 -- 172.21.15.28:0/2529826012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3740acf60 msgr2=0x7fc374160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 --2- 172.21.15.28:0/2529826012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3740acf60 0x7fc374160730 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009d90 tx=0x7fc36c02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 -- 172.21.15.28:0/2529826012 shutdown_connections 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 --2- 172.21.15.28:0/2529826012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3740acf60 0x7fc374160730 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 -- 172.21.15.28:0/2529826012 >> 172.21.15.28:0/2529826012 conn(0x7fc374069e80 msgr2=0x7fc37406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 -- 172.21.15.28:0/2529826012 shutdown_connections 2024-09-13T17:23:38.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.253+0000 7fc379800640 1 -- 172.21.15.28:0/2529826012 wait complete. 2024-09-13T17:23:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.255+0000 7fc379800640 1 Processor -- start 2024-09-13T17:23:38.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.255+0000 7fc379800640 1 -- start start 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc374146170 0x7fc37412d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc374160310 0x7fc37412dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374153760 con 0x7fc3740acf60 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37412e660 con 0x7fc374146170 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc379800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc374141d00 con 0x7fc374160310 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc372a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.256+0000 7fc372a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34240/0 (socket says 172.21.15.28:34240) 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 -- 172.21.15.28:0/678951624 learned_addr learned my addr 172.21.15.28:0/678951624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:38.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc373400640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc374160310 0x7fc37412dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372000640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc374146170 0x7fc37412d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 -- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc374160310 msgr2=0x7fc37412dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc374160310 0x7fc37412dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 -- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc374146170 msgr2=0x7fc37412d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc374146170 0x7fc37412d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 -- 172.21.15.28:0/678951624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c009a70 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.257+0000 7fc372a00640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fc36400c500 tx=0x7fc36400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.258+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364005ce0 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.258+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc364005e40 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.258+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc364007330 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.258+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc374140c60 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.258+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc374141220 con 0x7fc3740acf60 2024-09-13T17:23:38.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.259+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37406d020 con 0x7fc3740acf60 2024-09-13T17:23:38.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.260+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc364007550 con 0x7fc3740acf60 2024-09-13T17:23:38.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.260+0000 7fc36aa00640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 0x7fc35406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.261+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc364088650 con 0x7fc3740acf60 2024-09-13T17:23:38.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.261+0000 7fc372000640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 0x7fc35406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.262+0000 7fc372000640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 0x7fc35406d0c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc3741452d0 tx=0x7fc360009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.265+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc364053c50 con 0x7fc3740acf60 2024-09-13T17:23:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.447+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc374160730 con 0x7fc3740acf60 2024-09-13T17:23:38.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.448+0000 7fc36aa00640 1 -- 172.21.15.28:0/678951624 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+12867 (secure 0 0 0) 0x7fc364017090 con 0x7fc3740acf60 2024-09-13T17:23:38.450 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:38.450 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":215,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:37.629924+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":206,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":203,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 msgr2=0x7fc35406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 0x7fc35406d0c0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc3741452d0 tx=0x7fc360009040 comp rx=0 tx=0).stop 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 msgr2=0x7fc374153220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 secure :-1 s=READY pgs=2574 cs=0 l=1 rev1=1 crypto rx=0x7fc36400c500 tx=0x7fc36400c9c0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 shutdown_connections 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc35406ac00 0x7fc35406d0c0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc374160310 0x7fc37412dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc374146170 0x7fc37412d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 --2- 172.21.15.28:0/678951624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3740acf60 0x7fc374153220 unknown :-1 s=CLOSED pgs=2574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 >> 172.21.15.28:0/678951624 conn(0x7fc374069e80 msgr2=0x7fc3740b1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 shutdown_connections 2024-09-13T17:23:38.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.450+0000 7fc379800640 1 -- 172.21.15.28:0/678951624 wait complete. 2024-09-13T17:23:38.468 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:38.469 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:38.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.625+0000 7fd891000640 1 Processor -- start 2024-09-13T17:23:38.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.625+0000 7fd891000640 1 -- start start 2024-09-13T17:23:38.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.626+0000 7fd891000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd88c15c9a0 0x7fd88c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd891000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd891000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c0b2230 0x7fd88c06f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd891000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c0b26c0 con 0x7fd88c0b2230 2024-09-13T17:23:38.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd891000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd88c0ace30 con 0x7fd88c15c9a0 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd891000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c0ad110 con 0x7fd88c0ad940 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c0b2230 0x7fd88c06f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd88c0b2230 0x7fd88c06f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34246/0 (socket says 172.21.15.28:34246) 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 -- 172.21.15.28:0/1190386477 learned_addr learned my addr 172.21.15.28:0/1190386477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd881400640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd889e00640 1 --2- 172.21.15.28:0/1190386477 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15c9a0 0x7fd88c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 -- 172.21.15.28:0/1190386477 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 msgr2=0x7fd88c0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 -- 172.21.15.28:0/1190386477 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15c9a0 msgr2=0x7fd88c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 --2- 172.21.15.28:0/1190386477 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15c9a0 0x7fd88c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.627+0000 7fd88a800640 1 -- 172.21.15.28:0/1190386477 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88c06b170 con 0x7fd88c0b2230 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.628+0000 7fd88a800640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c06f460 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fd88c074440 tx=0x7fd88402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290d13bf0ee31f9e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd889400640 1 -- 172.21.15.28:0/1190386477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031600 con 0x7fd88c0b2230 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd889400640 1 -- 172.21.15.28:0/1190386477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88402d050 con 0x7fd88c0b2230 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd889400640 1 -- 172.21.15.28:0/1190386477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884031980 con 0x7fd88c0b2230 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 -- 172.21.15.28:0/1190386477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 msgr2=0x7fd88c06f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c06f460 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7fd88c074440 tx=0x7fd88402a730 comp rx=0 tx=0).stop 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 -- 172.21.15.28:0/1190386477 shutdown_connections 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c06f460 unknown :-1 s=CLOSED pgs=2575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 --2- 172.21.15.28:0/1190386477 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 --2- 172.21.15.28:0/1190386477 >> v2:172.21.15.189:3300/0 conn(0x7fd88c15c9a0 0x7fd88c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 -- 172.21.15.28:0/1190386477 >> 172.21.15.28:0/1190386477 conn(0x7fd88c0b0270 msgr2=0x7fd88c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.629+0000 7fd891000640 1 -- 172.21.15.28:0/1190386477 shutdown_connections 2024-09-13T17:23:38.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.630+0000 7fd891000640 1 -- 172.21.15.28:0/1190386477 wait complete. 2024-09-13T17:23:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.631+0000 7fd891000640 1 Processor -- start 2024-09-13T17:23:38.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.631+0000 7fd891000640 1 -- start start 2024-09-13T17:23:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15c9a0 0x7fd88c15e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1f2430 con 0x7fd88c0b2230 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1f26b0 con 0x7fd88c15c9a0 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd891000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c1f2990 con 0x7fd88c0ad940 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd881400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd881400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34260/0 (socket says 172.21.15.28:34260) 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd881400640 1 -- 172.21.15.28:0/2006055290 learned_addr learned my addr 172.21.15.28:0/2006055290 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.632+0000 7fd889e00640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd88a800640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15c9a0 0x7fd88c15e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 -- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 msgr2=0x7fd88c0ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 -- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15c9a0 msgr2=0x7fd88c15e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15c9a0 0x7fd88c15e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 -- 172.21.15.28:0/2006055290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd884009b30 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd881400640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e870 tx=0x7fd87400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8740094f0 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87400c940 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c070930 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.633+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87400ccf0 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.634+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c1f2f50 con 0x7fd88c0b2230 2024-09-13T17:23:38.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.635+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c0b1b30 con 0x7fd88c0b2230 2024-09-13T17:23:38.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.636+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd874013470 con 0x7fd88c0b2230 2024-09-13T17:23:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.637+0000 7fd882a00640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.637+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd8740890e0 con 0x7fd88c0b2230 2024-09-13T17:23:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.638+0000 7fd889e00640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.638+0000 7fd889e00640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fd88c160440 tx=0x7fd87c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.641+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd874054620 con 0x7fd88c0b2230 2024-09-13T17:23:38.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.817+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd88c0aef70 con 0x7fd88c0b2230 2024-09-13T17:23:38.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.818+0000 7fd882a00640 1 -- 172.21.15.28:0/2006055290 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v215) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd874057c40 con 0x7fd88c0b2230 2024-09-13T17:23:38.820 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 msgr2=0x7fd86806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7fd88c160440 tx=0x7fd87c009190 comp rx=0 tx=0).stop 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 msgr2=0x7fd88c15b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7fd87400e870 tx=0x7fd87400ed30 comp rx=0 tx=0).stop 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 shutdown_connections 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86806ac00 0x7fd86806d0c0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd88c15c9a0 0x7fd88c15e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd88c0b2230 0x7fd88c15b6f0 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 --2- 172.21.15.28:0/2006055290 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd88c0ad940 0x7fd88c0ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 >> 172.21.15.28:0/2006055290 conn(0x7fd88c0b0270 msgr2=0x7fd88c06c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 shutdown_connections 2024-09-13T17:23:38.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.820+0000 7fd891000640 1 -- 172.21.15.28:0/2006055290 wait complete. 2024-09-13T17:23:38.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:23:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.992+0000 7f830fe00640 1 Processor -- start 2024-09-13T17:23:38.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.992+0000 7f830fe00640 1 -- start start 2024-09-13T17:23:38.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.993+0000 7f830fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f8308070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830806c8b0 con 0x7f830815ab60 2024-09-13T17:23:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f830806cb90 con 0x7f830806c250 2024-09-13T17:23:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f830806c6a0 con 0x7f830815a780 2024-09-13T17:23:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830ca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f830815ab60 0x7f830806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38542/0 (socket says 172.21.15.28:38542) 2024-09-13T17:23:38.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830ca00640 1 -- 172.21.15.28:0/713249544 learned_addr learned my addr 172.21.15.28:0/713249544 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f830d400640 1 --2- 172.21.15.28:0/713249544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f8308070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826366338 0 0) 0x7f830806c8b0 con 0x7f830815ab60 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.994+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003520 con 0x7f830815ab60 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728578761 0 0) 0x7f830806cb90 con 0x7f830806c250 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f830806c8b0 con 0x7f830806c250 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f830d400640 1 -- 172.21.15.28:0/713249544 >> v1:172.21.15.189:6789/0 conn(0x7f830806c250 legacy=0x7f83080cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1393152114 0 0) 0x7f82f0003520 con 0x7f830815ab60 2024-09-13T17:23:38.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f830d400640 1 -- 172.21.15.28:0/713249544 >> v1:172.21.15.28:6789/0 conn(0x7f830815ab60 legacy=0x7f830806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.995+0000 7f830d400640 1 -- 172.21.15.28:0/713249544 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080cc980 con 0x7f830815a780 2024-09-13T17:23:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.996+0000 7f830d400640 1 --2- 172.21.15.28:0/713249544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f8308070d80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097d0 tx=0x7f82fc02a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcb9f539f64103aa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.996+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4001670 con 0x7f830815a780 2024-09-13T17:23:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.996+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc02ac30 con 0x7f830815a780 2024-09-13T17:23:38.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.996+0000 7f8307e00640 1 -- 172.21.15.28:0/713249544 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc032440 con 0x7f830815a780 2024-09-13T17:23:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 -- 172.21.15.28:0/713249544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 msgr2=0x7f8308070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 --2- 172.21.15.28:0/713249544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f8308070d80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097d0 tx=0x7f82fc02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 -- 172.21.15.28:0/713249544 shutdown_connections 2024-09-13T17:23:38.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 --2- 172.21.15.28:0/713249544 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f8308070d80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 -- 172.21.15.28:0/713249544 >> 172.21.15.28:0/713249544 conn(0x7f83080b0270 msgr2=0x7f83080b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 -- 172.21.15.28:0/713249544 shutdown_connections 2024-09-13T17:23:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.997+0000 7f830fe00640 1 -- 172.21.15.28:0/713249544 wait complete. 2024-09-13T17:23:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.999+0000 7f830fe00640 1 Processor -- start 2024-09-13T17:23:39.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:38.999+0000 7f830fe00640 1 -- start start 2024-09-13T17:23:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f830813de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f830815ab60 0x7f8308150780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080abfa0 con 0x7f830806c250 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83080a8820 con 0x7f830815ab60 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83080a8b00 con 0x7f830815a780 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34274/0 (socket says 172.21.15.28:34274) 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 -- 172.21.15.28:0/3648859537 learned_addr learned my addr 172.21.15.28:0/3648859537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830de00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f830815ab60 0x7f8308150780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 -- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 msgr2=0x7f830813de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f830813de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 -- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f830815ab60 msgr2=0x7f8308150780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f830815ab60 0x7f8308150780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.000+0000 7f830d400640 1 -- 172.21.15.28:0/3648859537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc009400 con 0x7f830806c250 2024-09-13T17:23:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f830d400640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097d0 tx=0x7f82fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc03c820 con 0x7f830806c250 2024-09-13T17:23:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc03c980 con 0x7f830806c250 2024-09-13T17:23:39.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc03ccf0 con 0x7f830806c250 2024-09-13T17:23:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83080a8de0 con 0x7f830806c250 2024-09-13T17:23:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.001+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83080a93a0 con 0x7f830806c250 2024-09-13T17:23:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.002+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83080af9a0 con 0x7f830806c250 2024-09-13T17:23:39.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.003+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f82fc039050 con 0x7f830806c250 2024-09-13T17:23:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.004+0000 7f8305600640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 0x7f82e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.004+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f82fc0a9980 con 0x7f830806c250 2024-09-13T17:23:39.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.004+0000 7f830ca00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 0x7f82e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.005+0000 7f830ca00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 0x7f82e806d0c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001340 tx=0x7f82f4000f50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:39.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.008+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f82fc074f30 con 0x7f830806c250 2024-09-13T17:23:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.190+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f83080b0bb0 con 0x7f830806c250 2024-09-13T17:23:39.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.190+0000 7f8305600640 1 -- 172.21.15.28:0/3648859537 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v215) v1 ==== 84+0+65 (secure 0 0 0) 0x7f82fc037020 con 0x7f830806c250 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:23:39.193 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 msgr2=0x7f82e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 0x7f82e806d0c0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f82f4001340 tx=0x7f82f4000f50 comp rx=0 tx=0).stop 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 msgr2=0x7f8308153420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0097d0 tx=0x7f82fc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 shutdown_connections 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f82e806ac00 0x7f82e806d0c0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f830815ab60 0x7f8308150780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f830815a780 0x7f830813de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 --2- 172.21.15.28:0/3648859537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f830806c250 0x7f8308153420 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 >> 172.21.15.28:0/3648859537 conn(0x7f83080b0270 msgr2=0x7f8308157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 shutdown_connections 2024-09-13T17:23:39.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.193+0000 7f830fe00640 1 -- 172.21.15.28:0/3648859537 wait complete. 2024-09-13T17:23:39.210 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:23:39.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:23:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.369+0000 7f17c9800640 1 Processor -- start 2024-09-13T17:23:39.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.369+0000 7f17c9800640 1 -- start start 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c9800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17c415a7a0 0x7f17c40ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c9800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 0x7f17c40cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c9800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c406c950 con 0x7f17c40ce9d0 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c9800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c40cee40 con 0x7f17c415a7a0 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c9800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c40b2150 con 0x7f17c40b1c80 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.370+0000 7f17c2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17c415a7a0 0x7f17c40ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f17c40ce9d0 0x7f17c40b1570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38552/0 (socket says 172.21.15.28:38552) 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2000640 1 -- 172.21.15.28:0/2163233976 learned_addr learned my addr 172.21.15.28:0/2163233976 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:39.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f17c415a7a0 0x7f17c40ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34276/0 (socket says 172.21.15.28:34276) 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c3400640 1 --2- 172.21.15.28:0/2163233976 >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 0x7f17c40cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2a00640 1 -- 172.21.15.28:0/2163233976 >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 msgr2=0x7f17c40cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2a00640 1 --2- 172.21.15.28:0/2163233976 >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 0x7f17c40cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2a00640 1 -- 172.21.15.28:0/2163233976 >> v1:172.21.15.28:6789/0 conn(0x7f17c40ce9d0 legacy=0x7f17c40b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c2a00640 1 -- 172.21.15.28:0/2163233976 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c40b2430 con 0x7f17c415a7a0 2024-09-13T17:23:39.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.371+0000 7f17c3400640 1 --2- 172.21.15.28:0/2163233976 >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 0x7f17c40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.372+0000 7f17c2a00640 1 --2- 172.21.15.28:0/2163233976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40ce490 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f17c40b3c30 tx=0x7f17b402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4909fc8b3ad2e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.372+0000 7f17c1600640 1 -- 172.21.15.28:0/2163233976 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b402ae60 con 0x7f17c415a7a0 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.372+0000 7f17c1600640 1 -- 172.21.15.28:0/2163233976 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b4032440 con 0x7f17c415a7a0 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.372+0000 7f17c1600640 1 -- 172.21.15.28:0/2163233976 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17b40327d0 con 0x7f17c415a7a0 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 -- 172.21.15.28:0/2163233976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 msgr2=0x7f17c40ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 --2- 172.21.15.28:0/2163233976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40ce490 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7f17c40b3c30 tx=0x7f17b402a110 comp rx=0 tx=0).stop 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 -- 172.21.15.28:0/2163233976 shutdown_connections 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 --2- 172.21.15.28:0/2163233976 >> v2:172.21.15.189:3300/0 conn(0x7f17c40b1c80 0x7f17c40cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 --2- 172.21.15.28:0/2163233976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40ce490 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 -- 172.21.15.28:0/2163233976 >> 172.21.15.28:0/2163233976 conn(0x7f17c4069e80 msgr2=0x7f17c406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 -- 172.21.15.28:0/2163233976 shutdown_connections 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.373+0000 7f17c9800640 1 -- 172.21.15.28:0/2163233976 wait complete. 2024-09-13T17:23:39.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.375+0000 7f17c9800640 1 Processor -- start 2024-09-13T17:23:39.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.375+0000 7f17c9800640 1 -- start start 2024-09-13T17:23:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40b1c80 0x7f17c40c58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f17c40ce9d0 0x7f17c4154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40c5e20 con 0x7f17c415a7a0 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c41545a0 con 0x7f17c40b1c80 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c9800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c40c7f00 con 0x7f17c40ce9d0 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34278/0 (socket says 172.21.15.28:34278) 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c3400640 1 -- 172.21.15.28:0/278068052 learned_addr learned my addr 172.21.15.28:0/278068052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c2a00640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40b1c80 0x7f17c40c58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.376+0000 7f17c2000640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f17c40ce9d0 0x7f17c4154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 -- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f17c40ce9d0 msgr2=0x7f17c4154060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f17c40ce9d0 0x7f17c4154060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 -- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40b1c80 msgr2=0x7f17c40c58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:39.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40b1c80 0x7f17c40c58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 -- 172.21.15.28:0/278068052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b4009400 con 0x7f17c415a7a0 2024-09-13T17:23:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c3400640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f17bc007ef0 tx=0x7f17bc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:39.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003990 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17bc00f040 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003cf0 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c40c4040 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.377+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c40c4570 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.378+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c406f480 con 0x7f17c415a7a0 2024-09-13T17:23:39.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.379+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f17bc004030 con 0x7f17c415a7a0 2024-09-13T17:23:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.380+0000 7f17baa00640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 0x7f17a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.380+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17bc089270 con 0x7f17c415a7a0 2024-09-13T17:23:39.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.380+0000 7f17c2a00640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 0x7f17a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:39.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.381+0000 7f17c2a00640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 0x7f17a406d0c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f17b4034000 tx=0x7f17b402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:39.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.384+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f17c406f480 con 0x7f17c415a7a0 2024-09-13T17:23:39.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:39.565+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f17c406f480 con 0x7f17c415a7a0 2024-09-13T17:23:40.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.194+0000 7f17baa00640 1 -- 172.21.15.28:0/278068052 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v216) v1 ==== 138+0+0 (secure 0 0 0) 0x7f17c406f480 con 0x7f17c415a7a0 2024-09-13T17:23:40.197 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:23:40.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 msgr2=0x7f17a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 0x7f17a406d0c0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f17b4034000 tx=0x7f17b402e000 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 msgr2=0x7f17c40c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7f17bc007ef0 tx=0x7f17bc00d4a0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 shutdown_connections 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f17a406ac00 0x7f17a406d0c0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c415a7a0 0x7f17c40c5fc0 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f17c40ce9d0 0x7f17c4154060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 --2- 172.21.15.28:0/278068052 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17c40b1c80 0x7f17c40c58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.199+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 >> 172.21.15.28:0/278068052 conn(0x7f17c4069e80 msgr2=0x7f17c415cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.200+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 shutdown_connections 2024-09-13T17:23:40.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.200+0000 7f17c9800640 1 -- 172.21.15.28:0/278068052 wait complete. 2024-09-13T17:23:40.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:23:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.382+0000 7f7934c00640 1 Processor -- start 2024-09-13T17:23:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.382+0000 7f7934c00640 1 -- start start 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7934c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f793015a780 0x7f7930070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7934c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 0x7f79300712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7934c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f793006c8b0 con 0x7f793006bb40 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7934c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f793006cb90 con 0x7f793015a780 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7934c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79300cc980 con 0x7f793015ab60 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f792e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f793006bb40 0x7f793006bfd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38556/0 (socket says 172.21.15.28:38556) 2024-09-13T17:23:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f792e800640 1 -- 172.21.15.28:0/4233155144 learned_addr learned my addr 172.21.15.28:0/4233155144 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f7927e00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 0x7f79300712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f792de00640 1 --2- 172.21.15.28:0/4233155144 >> v2:172.21.15.189:3300/0 conn(0x7f793015a780 0x7f7930070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 241716943 0 0) 0x7f793006c8b0 con 0x7f793006bb40 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.384+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7914003520 con 0x7f793006bb40 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f792de00640 1 -- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 msgr2=0x7f79300712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f792de00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 0x7f79300712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712535400 0 0) 0x7f7914003520 con 0x7f793006bb40 2024-09-13T17:23:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f792de00640 1 -- 172.21.15.28:0/4233155144 >> v1:172.21.15.28:6789/0 conn(0x7f793006bb40 legacy=0x7f793006bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f792de00640 1 -- 172.21.15.28:0/4233155144 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79300ccc60 con 0x7f793015a780 2024-09-13T17:23:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.385+0000 7f7927e00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 0x7f79300712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.386+0000 7f792de00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015a780 0x7f7930070d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f79200097a0 tx=0x7f792002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca3b8ea86f3ad623 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.386+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920032440 con 0x7f793015a780 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.386+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79200325a0 con 0x7f793015a780 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.386+0000 7f792d400640 1 -- 172.21.15.28:0/4233155144 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7920032930 con 0x7f793015a780 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 -- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015a780 msgr2=0x7f7930070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015a780 0x7f7930070d80 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f79200097a0 tx=0x7f792002a180 comp rx=0 tx=0).stop 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 -- 172.21.15.28:0/4233155144 shutdown_connections 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015ab60 0x7f79300712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 --2- 172.21.15.28:0/4233155144 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015a780 0x7f7930070d80 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 -- 172.21.15.28:0/4233155144 >> 172.21.15.28:0/4233155144 conn(0x7f79300b0270 msgr2=0x7f79300b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 -- 172.21.15.28:0/4233155144 shutdown_connections 2024-09-13T17:23:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.387+0000 7f7934c00640 1 -- 172.21.15.28:0/4233155144 wait complete. 2024-09-13T17:23:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.389+0000 7f7934c00640 1 Processor -- start 2024-09-13T17:23:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.389+0000 7f7934c00640 1 -- start start 2024-09-13T17:23:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.389+0000 7f7934c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7934c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015a780 0x7f7930156170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7934c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015ab60 0x7f79300af470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7934c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7930155ef0 con 0x7f793006bb40 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7934c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79300c4560 con 0x7f793015ab60 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7934c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79301474b0 con 0x7f793015a780 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34294/0 (socket says 172.21.15.28:34294) 2024-09-13T17:23:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 -- 172.21.15.28:0/465716856 learned_addr learned my addr 172.21.15.28:0/465716856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f7927e00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015a780 0x7f7930156170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792e800640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015ab60 0x7f79300af470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 -- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015a780 msgr2=0x7f7930156170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015a780 0x7f7930156170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 -- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015ab60 msgr2=0x7f79300af470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015ab60 0x7f79300af470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.390+0000 7f792de00640 1 -- 172.21.15.28:0/465716856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918009400 con 0x7f793006bb40 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f792de00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f7920035040 tx=0x7f79200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792003d8f0 con 0x7f793006bb40 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792003da70 con 0x7f793006bb40 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7920009400 con 0x7f793006bb40 2024-09-13T17:23:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792003de20 con 0x7f793006bb40 2024-09-13T17:23:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.391+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79300c3cc0 con 0x7f793006bb40 2024-09-13T17:23:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.392+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793006c8b0 con 0x7f793006bb40 2024-09-13T17:23:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.393+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7920039040 con 0x7f793006bb40 2024-09-13T17:23:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.394+0000 7f7926000640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 0x7f790c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.394+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f79200a9d90 con 0x7f793006bb40 2024-09-13T17:23:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.395+0000 7f7927e00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 0x7f790c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.395+0000 7f7927e00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 0x7f790c06d0f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f79300c6660 tx=0x7f7918008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.398+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f79200750a0 con 0x7f793006bb40 2024-09-13T17:23:40.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:40.580+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f793006b430 con 0x7f793006bb40 2024-09-13T17:23:41.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.202+0000 7f7926000640 1 -- 172.21.15.28:0/465716856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v217) v1 ==== 170+0+0 (secure 0 0 0) 0x7f792004a0d0 con 0x7f793006bb40 2024-09-13T17:23:41.205 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.210+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 msgr2=0x7f790c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.210+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 0x7f790c06d0f0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f79300c6660 tx=0x7f7918008040 comp rx=0 tx=0).stop 2024-09-13T17:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.210+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 msgr2=0x7f7930146f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.210+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f7920035040 tx=0x7f79200046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 shutdown_connections 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f790c06ac30 0x7f790c06d0f0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f793015ab60 0x7f79300af470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f793015a780 0x7f7930156170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 --2- 172.21.15.28:0/465716856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f793006bb40 0x7f7930146f70 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 >> 172.21.15.28:0/465716856 conn(0x7f79300b0270 msgr2=0x7f79300cd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:41.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 shutdown_connections 2024-09-13T17:23:41.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.211+0000 7f7934c00640 1 -- 172.21.15.28:0/465716856 wait complete. 2024-09-13T17:23:41.240 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:23:41.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.397+0000 7f7e88e00640 1 Processor -- start 2024-09-13T17:23:41.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.398+0000 7f7e88e00640 1 -- start start 2024-09-13T17:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 0x7f7e840cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e840cd5f0 0x7f7e840c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e840c3ba0 0x7f7e8406f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e840cda50 con 0x7f7e840cd5f0 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e8415ef70 con 0x7f7e840c3ba0 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e88e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840c4320 con 0x7f7e84143f70 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e7aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e840cd5f0 0x7f7e840c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e7aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e840cd5f0 0x7f7e840c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34310/0 (socket says 172.21.15.28:34310) 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e7aa00640 1 -- 172.21.15.28:0/2178689207 learned_addr learned my addr 172.21.15.28:0/2178689207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e83e00640 1 --2- 172.21.15.28:0/2178689207 >> v2:172.21.15.189:3300/0 conn(0x7f7e840c3ba0 0x7f7e8406f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.399+0000 7f7e83400640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 0x7f7e840cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e7aa00640 1 -- 172.21.15.28:0/2178689207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 msgr2=0x7f7e840cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e7aa00640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 0x7f7e840cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e7aa00640 1 -- 172.21.15.28:0/2178689207 >> v2:172.21.15.189:3300/0 conn(0x7f7e840c3ba0 msgr2=0x7f7e8406f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e7aa00640 1 --2- 172.21.15.28:0/2178689207 >> v2:172.21.15.189:3300/0 conn(0x7f7e840c3ba0 0x7f7e8406f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e7aa00640 1 -- 172.21.15.28:0/2178689207 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8415e1b0 con 0x7f7e840cd5f0 2024-09-13T17:23:41.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.400+0000 7f7e83400640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 0x7f7e840cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.401+0000 7f7e7aa00640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e840cd5f0 0x7f7e840c3660 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f7e840b14e0 tx=0x7f7e6c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=459acda5d5fc7055 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.401+0000 7f7e82a00640 1 -- 172.21.15.28:0/2178689207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c032440 con 0x7f7e840cd5f0 2024-09-13T17:23:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.401+0000 7f7e82a00640 1 -- 172.21.15.28:0/2178689207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c0325a0 con 0x7f7e840cd5f0 2024-09-13T17:23:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.401+0000 7f7e82a00640 1 -- 172.21.15.28:0/2178689207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c032930 con 0x7f7e840cd5f0 2024-09-13T17:23:41.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 -- 172.21.15.28:0/2178689207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e840cd5f0 msgr2=0x7f7e840c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e840cd5f0 0x7f7e840c3660 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f7e840b14e0 tx=0x7f7e6c02a1e0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 -- 172.21.15.28:0/2178689207 shutdown_connections 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e840cd5f0 0x7f7e840c3660 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 --2- 172.21.15.28:0/2178689207 >> v2:172.21.15.189:3300/0 conn(0x7f7e840c3ba0 0x7f7e8406f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 --2- 172.21.15.28:0/2178689207 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e84143f70 0x7f7e840cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 -- 172.21.15.28:0/2178689207 >> 172.21.15.28:0/2178689207 conn(0x7f7e84069e80 msgr2=0x7f7e8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 -- 172.21.15.28:0/2178689207 shutdown_connections 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.402+0000 7f7e88e00640 1 -- 172.21.15.28:0/2178689207 wait complete. 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.403+0000 7f7e88e00640 1 Processor -- start 2024-09-13T17:23:41.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.403+0000 7f7e88e00640 1 -- start start 2024-09-13T17:23:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e840c3ba0 0x7f7e841f3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e840cd5f0 0x7f7e840c6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f34b0 con 0x7f7e84143f70 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e841f3a70 con 0x7f7e840c3ba0 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.404+0000 7f7e88e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840c6bc0 con 0x7f7e840cd5f0 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34314/0 (socket says 172.21.15.28:34314) 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 -- 172.21.15.28:0/4094001270 learned_addr learned my addr 172.21.15.28:0/4094001270 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e7aa00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e840cd5f0 0x7f7e840c6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83400640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e840c3ba0 0x7f7e841f3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 -- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e840cd5f0 msgr2=0x7f7e840c6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e840cd5f0 0x7f7e840c6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 -- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e840c3ba0 msgr2=0x7f7e841f3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e840c3ba0 0x7f7e841f3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 -- 172.21.15.28:0/4094001270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0094d0 con 0x7f7e84143f70 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.405+0000 7f7e83e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009d30 tx=0x7f7e7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.406+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c009ba0 con 0x7f7e84143f70 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.406+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c00f040 con 0x7f7e84143f70 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.406+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0146c0 con 0x7f7e84143f70 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.406+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e840c6ea0 con 0x7f7e84143f70 2024-09-13T17:23:41.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.406+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8413ac80 con 0x7f7e84143f70 2024-09-13T17:23:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.407+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e841f3c00 con 0x7f7e84143f70 2024-09-13T17:23:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.408+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e7c014820 con 0x7f7e84143f70 2024-09-13T17:23:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.409+0000 7f7e7be00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 0x7f7e60071500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.409+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f7e7c089cf0 con 0x7f7e84143f70 2024-09-13T17:23:41.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.409+0000 7f7e83400640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 0x7f7e60071500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.410+0000 7f7e83400640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 0x7f7e60071500 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006e80 tx=0x7f7e74008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.413+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e7c054f40 con 0x7f7e84143f70 2024-09-13T17:23:41.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.626+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7e840c3660 con 0x7f7e84143f70 2024-09-13T17:23:41.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.633+0000 7f7e7be00640 1 -- 172.21.15.28:0/4094001270 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v177) v1 ==== 174+0+0 (secure 0 0 0) 0x7f7e7c0180c0 con 0x7f7e84143f70 2024-09-13T17:23:41.636 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 30 and data pool 31 2024-09-13T17:23:41.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 msgr2=0x7f7e60071500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 0x7f7e60071500 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006e80 tx=0x7f7e74008040 comp rx=0 tx=0).stop 2024-09-13T17:23:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 msgr2=0x7f7e8415c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 secure :-1 s=READY pgs=2585 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009d30 tx=0x7f7e7c00c490 comp rx=0 tx=0).stop 2024-09-13T17:23:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 shutdown_connections 2024-09-13T17:23:41.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e6006f040 0x7f7e60071500 secure :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f7e74006e80 tx=0x7f7e74008040 comp rx=0 tx=0).stop 2024-09-13T17:23:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e84143f70 0x7f7e8415c6e0 unknown :-1 s=CLOSED pgs=2585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.638+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e840cd5f0 0x7f7e840c6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.639+0000 7f7e88e00640 1 --2- 172.21.15.28:0/4094001270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e840c3ba0 0x7f7e841f3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.639+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 >> 172.21.15.28:0/4094001270 conn(0x7f7e84069e80 msgr2=0x7f7e84142ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:41.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.640+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 shutdown_connections 2024-09-13T17:23:41.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.641+0000 7f7e88e00640 1 -- 172.21.15.28:0/4094001270 wait complete. 2024-09-13T17:23:41.676 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.832+0000 7f1964a00640 1 Processor -- start 2024-09-13T17:23:41.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.832+0000 7f1964a00640 1 -- start start 2024-09-13T17:23:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.833+0000 7f1964a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f196015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.833+0000 7f1964a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f196006c2d0 0x7f196006c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.833+0000 7f1964a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1960159d90 con 0x7f19601592a0 2024-09-13T17:23:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.833+0000 7f1964a00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f196015b0b0 con 0x7f196006c2d0 2024-09-13T17:23:41.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.833+0000 7f1964a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f196006cca0 con 0x7f196006f970 2024-09-13T17:23:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195e800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f196006c2d0 0x7f196006c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f1955400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f196015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195de00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f19601592a0 0x7f1960159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:43938/0 (socket says 172.21.15.28:43938) 2024-09-13T17:23:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195de00640 1 -- 172.21.15.28:0/523517760 learned_addr learned my addr 172.21.15.28:0/523517760 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:41.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195d400640 1 -- 172.21.15.28:0/523517760 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458726542 0 0) 0x7f1960159d90 con 0x7f19601592a0 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195d400640 1 -- 172.21.15.28:0/523517760 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1944003520 con 0x7f19601592a0 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195e800640 1 -- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 msgr2=0x7f196015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195e800640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f196015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195e800640 1 -- 172.21.15.28:0/523517760 >> v1:172.21.15.189:6789/0 conn(0x7f19601592a0 legacy=0x7f1960159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f195e800640 1 -- 172.21.15.28:0/523517760 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600b2040 con 0x7f196006c2d0 2024-09-13T17:23:41.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.834+0000 7f1955400640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f196015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.835+0000 7f195e800640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006c760 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1958009d90 tx=0x7f195802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f406658a4fd305fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.835+0000 7f195d400640 1 -- 172.21.15.28:0/523517760 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958007740 con 0x7f196006c2d0 2024-09-13T17:23:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f195d400640 1 -- 172.21.15.28:0/523517760 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195802d050 con 0x7f196006c2d0 2024-09-13T17:23:41.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f195d400640 1 -- 172.21.15.28:0/523517760 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958007ac0 con 0x7f196006c2d0 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 -- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 msgr2=0x7f196006c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006c760 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f1958009d90 tx=0x7f195802a660 comp rx=0 tx=0).stop 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 -- 172.21.15.28:0/523517760 shutdown_connections 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006c760 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 --2- 172.21.15.28:0/523517760 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f196015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 -- 172.21.15.28:0/523517760 >> 172.21.15.28:0/523517760 conn(0x7f1960069e80 msgr2=0x7f196006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 -- 172.21.15.28:0/523517760 shutdown_connections 2024-09-13T17:23:41.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.836+0000 7f1964a00640 1 -- 172.21.15.28:0/523517760 wait complete. 2024-09-13T17:23:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.838+0000 7f1964a00640 1 Processor -- start 2024-09-13T17:23:41.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.838+0000 7f1964a00640 1 -- start start 2024-09-13T17:23:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f19601507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19600ca520 con 0x7f19601592a0 2024-09-13T17:23:41.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19601f1f40 con 0x7f196006c2d0 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1964a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19600cc770 con 0x7f196006f970 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f195e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f195e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34316/0 (socket says 172.21.15.28:34316) 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f195e800640 1 -- 172.21.15.28:0/4158606158 learned_addr learned my addr 172.21.15.28:0/4158606158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f1955400640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f19601507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.839+0000 7f195de00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 -- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 msgr2=0x7f196006b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006b9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 -- 172.21.15.28:0/4158606158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 msgr2=0x7f19601f2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:41.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 -- 172.21.15.28:0/4158606158 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1948009400 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f195e800640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1955400640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f19601507a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f194800e900 tx=0x7f194800edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194800ca00 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194800cb60 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.840+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1948013470 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.841+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1958009a70 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.841+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19600ccd30 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.842+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19600ccef0 con 0x7f196006f970 2024-09-13T17:23:41.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.843+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1948005990 con 0x7f196006f970 2024-09-13T17:23:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.844+0000 7f1956a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 0x7f193c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.844+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19480894f0 con 0x7f196006f970 2024-09-13T17:23:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.844+0000 7f195de00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 0x7f193c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.845+0000 7f195de00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 0x7f193c06d0c0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1950005880 tx=0x7f1950009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:41.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:41.847+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f19480547f0 con 0x7f196006f970 2024-09-13T17:23:41.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:23:41.953 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:23:42.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.025+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19600b3740 con 0x7f196006f970 2024-09-13T17:23:42.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.027+0000 7f1956a00640 1 -- 172.21.15.28:0/4158606158 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f1948017090 con 0x7f196006f970 2024-09-13T17:23:42.030 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:42.030 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":218,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:41.628703+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 msgr2=0x7f193c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 0x7f193c06d0c0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f1950005880 tx=0x7f1950009040 comp rx=0 tx=0).stop 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 msgr2=0x7f19601507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f19601507a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f194800e900 tx=0x7f194800edc0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 shutdown_connections 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f193c06ac00 0x7f193c06d0c0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f19601592a0 0x7f19601f2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f196006f970 0x7f19601507a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 --2- 172.21.15.28:0/4158606158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f196006c2d0 0x7f196006b9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 >> 172.21.15.28:0/4158606158 conn(0x7f1960069e80 msgr2=0x7f1960157aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:42.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 shutdown_connections 2024-09-13T17:23:42.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.029+0000 7f1964a00640 1 -- 172.21.15.28:0/4158606158 wait complete. 2024-09-13T17:23:42.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.201+0000 7f1365600640 1 Processor -- start 2024-09-13T17:23:42.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.201+0000 7f1365600640 1 -- start start 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1365600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f13600acf60 0x7f1360160730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1365600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1360146550 con 0x7f1360160310 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1365600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f136015d280 con 0x7f1360146170 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1365600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f136015cd20 con 0x7f13600acf60 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1357e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1360160310 0x7f13600ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38562/0 (socket says 172.21.15.28:38562) 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f1357e00640 1 -- 172.21.15.28:0/3037458025 learned_addr learned my addr 172.21.15.28:0/3037458025 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:42.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.203+0000 7f135f400640 1 --2- 172.21.15.28:0/3037458025 >> v2:172.21.15.189:3300/0 conn(0x7f13600acf60 0x7f1360160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.204+0000 7f135f400640 1 -- 172.21.15.28:0/3037458025 >> v1:172.21.15.189:6789/0 conn(0x7f1360146170 legacy=0x7f136015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.204+0000 7f135f400640 1 -- 172.21.15.28:0/3037458025 >> v1:172.21.15.28:6789/0 conn(0x7f1360160310 legacy=0x7f13600ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.204+0000 7f135f400640 1 -- 172.21.15.28:0/3037458025 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360160d60 con 0x7f13600acf60 2024-09-13T17:23:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.205+0000 7f135f400640 1 --2- 172.21.15.28:0/3037458025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13600acf60 0x7f1360160730 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1358009d90 tx=0x7f135802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65c9a2521a5a925b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.205+0000 7f135e000640 1 -- 172.21.15.28:0/3037458025 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358007740 con 0x7f13600acf60 2024-09-13T17:23:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.205+0000 7f135e000640 1 -- 172.21.15.28:0/3037458025 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135802d050 con 0x7f13600acf60 2024-09-13T17:23:42.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.205+0000 7f135e000640 1 -- 172.21.15.28:0/3037458025 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1358007ac0 con 0x7f13600acf60 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 -- 172.21.15.28:0/3037458025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13600acf60 msgr2=0x7f1360160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 --2- 172.21.15.28:0/3037458025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13600acf60 0x7f1360160730 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f1358009d90 tx=0x7f135802a6f0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 -- 172.21.15.28:0/3037458025 shutdown_connections 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 --2- 172.21.15.28:0/3037458025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13600acf60 0x7f1360160730 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 -- 172.21.15.28:0/3037458025 >> 172.21.15.28:0/3037458025 conn(0x7f1360069e80 msgr2=0x7f136006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 -- 172.21.15.28:0/3037458025 shutdown_connections 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.206+0000 7f1365600640 1 -- 172.21.15.28:0/3037458025 wait complete. 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.208+0000 7f1365600640 1 Processor -- start 2024-09-13T17:23:42.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.208+0000 7f1365600640 1 -- start start 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1360146170 0x7f136015ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1360160310 0x7f136015bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360157640 con 0x7f13600acf60 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360157920 con 0x7f1360146170 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1365600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1360157c00 con 0x7f1360160310 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34330/0 (socket says 172.21.15.28:34330) 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 -- 172.21.15.28:0/2266754423 learned_addr learned my addr 172.21.15.28:0/2266754423 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f1357e00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1360146170 0x7f136015ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135f400640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1360160310 0x7f136015bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 -- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1360160310 msgr2=0x7f136015bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1360160310 0x7f136015bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 -- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1360146170 msgr2=0x7f136015ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1360146170 0x7f136015ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.209+0000 7f135ea00640 1 -- 172.21.15.28:0/2266754423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1358009a70 con 0x7f13600acf60 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f135ea00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f135000c450 tx=0x7f135000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13500052d0 con 0x7f13600acf60 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1350005430 con 0x7f13600acf60 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1350012870 con 0x7f13600acf60 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360157ee0 con 0x7f13600acf60 2024-09-13T17:23:42.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.210+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136015a4f0 con 0x7f13600acf60 2024-09-13T17:23:42.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.211+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360160730 con 0x7f13600acf60 2024-09-13T17:23:42.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.212+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f13500129d0 con 0x7f13600acf60 2024-09-13T17:23:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.213+0000 7f1356a00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 0x7f133c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.213+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1350088d70 con 0x7f13600acf60 2024-09-13T17:23:42.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.213+0000 7f1357e00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 0x7f133c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.214+0000 7f1357e00640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 0x7f133c06d0f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f136006bb50 tx=0x7f1348009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.217+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1350053f60 con 0x7f13600acf60 2024-09-13T17:23:42.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.399+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f136006b8a0 con 0x7f13600acf60 2024-09-13T17:23:42.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.401+0000 7f1356a00640 1 -- 172.21.15.28:0/2266754423 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v218) v1 ==== 74+0+16532 (secure 0 0 0) 0x7f1350057580 con 0x7f13600acf60 2024-09-13T17:23:42.403 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:42.403 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":218,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:41.628703+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:42.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 msgr2=0x7f133c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 0x7f133c06d0f0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f136006bb50 tx=0x7f1348009040 comp rx=0 tx=0).stop 2024-09-13T17:23:42.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 msgr2=0x7f1360159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7f135000c450 tx=0x7f135000c910 comp rx=0 tx=0).stop 2024-09-13T17:23:42.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 shutdown_connections 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f133c06ac30 0x7f133c06d0f0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1360160310 0x7f136015bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1360146170 0x7f136015ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 --2- 172.21.15.28:0/2266754423 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13600acf60 0x7f1360159fb0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 >> 172.21.15.28:0/2266754423 conn(0x7f1360069e80 msgr2=0x7f13600af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 shutdown_connections 2024-09-13T17:23:42.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.403+0000 7f1365600640 1 -- 172.21.15.28:0/2266754423 wait complete. 2024-09-13T17:23:42.421 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:23:42.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.579+0000 7f05d9e00640 1 Processor -- start 2024-09-13T17:23:42.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.579+0000 7f05d9e00640 1 -- start start 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05d9e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f05d40cbec0 0x7f05d41653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05d9e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d41550c0 con 0x7f05d415b7e0 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05d9e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05d406e690 con 0x7f05d4154ce0 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05d9e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05d415bc40 con 0x7f05d40cbec0 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05ca000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05d415b7e0 0x7f05d40cb7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:38566/0 (socket says 172.21.15.28:38566) 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05ca000640 1 -- 172.21.15.28:0/2429403560 learned_addr learned my addr 172.21.15.28:0/2429403560 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:42.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.580+0000 7f05d3400640 1 --2- 172.21.15.28:0/2429403560 >> v2:172.21.15.189:3300/0 conn(0x7f05d40cbec0 0x7f05d41653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.581+0000 7f05d2000640 1 -- 172.21.15.28:0/2429403560 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601006827 0 0) 0x7f05d41550c0 con 0x7f05d415b7e0 2024-09-13T17:23:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.581+0000 7f05d2000640 1 -- 172.21.15.28:0/2429403560 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b8003520 con 0x7f05d415b7e0 2024-09-13T17:23:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.581+0000 7f05d3400640 1 -- 172.21.15.28:0/2429403560 >> v1:172.21.15.189:6789/0 conn(0x7f05d4154ce0 legacy=0x7f05d406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.581+0000 7f05d3400640 1 -- 172.21.15.28:0/2429403560 >> v1:172.21.15.28:6789/0 conn(0x7f05d415b7e0 legacy=0x7f05d40cb7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.581+0000 7f05d3400640 1 -- 172.21.15.28:0/2429403560 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05d406ddf0 con 0x7f05d40cbec0 2024-09-13T17:23:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.582+0000 7f05d3400640 1 --2- 172.21.15.28:0/2429403560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d40cbec0 0x7f05d41653c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d90 tx=0x7f05cc02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4e397fac0932ce3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.582+0000 7f05d2000640 1 -- 172.21.15.28:0/2429403560 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc007740 con 0x7f05d40cbec0 2024-09-13T17:23:42.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.582+0000 7f05d2000640 1 -- 172.21.15.28:0/2429403560 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05cc02d050 con 0x7f05d40cbec0 2024-09-13T17:23:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.582+0000 7f05d2000640 1 -- 172.21.15.28:0/2429403560 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc007ac0 con 0x7f05d40cbec0 2024-09-13T17:23:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 -- 172.21.15.28:0/2429403560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d40cbec0 msgr2=0x7f05d41653c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2429403560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d40cbec0 0x7f05d41653c0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009d90 tx=0x7f05cc02a660 comp rx=0 tx=0).stop 2024-09-13T17:23:42.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 -- 172.21.15.28:0/2429403560 shutdown_connections 2024-09-13T17:23:42.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2429403560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d40cbec0 0x7f05d41653c0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 -- 172.21.15.28:0/2429403560 >> 172.21.15.28:0/2429403560 conn(0x7f05d40b0af0 msgr2=0x7f05d40b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:42.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 -- 172.21.15.28:0/2429403560 shutdown_connections 2024-09-13T17:23:42.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.583+0000 7f05d9e00640 1 -- 172.21.15.28:0/2429403560 wait complete. 2024-09-13T17:23:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.585+0000 7f05d9e00640 1 Processor -- start 2024-09-13T17:23:42.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.585+0000 7f05d9e00640 1 -- start start 2024-09-13T17:23:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d40cbec0 0x7f05d4139060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d415b7e0 0x7f05d415e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d413c880 con 0x7f05d4154ce0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d413b390 con 0x7f05d415b7e0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05d9e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d415ecc0 con 0x7f05d40cbec0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34332/0 (socket says 172.21.15.28:34332) 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.586+0000 7f05ca000640 1 -- 172.21.15.28:0/2447073756 learned_addr learned my addr 172.21.15.28:0/2447073756 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05d3400640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d415b7e0 0x7f05d415e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05d2a00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d40cbec0 0x7f05d4139060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 -- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d40cbec0 msgr2=0x7f05d4139060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d40cbec0 0x7f05d4139060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 -- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d415b7e0 msgr2=0x7f05d415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d415b7e0 0x7f05d415e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:42.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 -- 172.21.15.28:0/2447073756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc009a70 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05ca000640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c410 tx=0x7f05bc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc005ce0 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05bc005e40 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d415efa0 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05bc0072f0 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.587+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d413c510 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.588+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d40caf60 con 0x7f05d4154ce0 2024-09-13T17:23:42.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.589+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f05bc007450 con 0x7f05d4154ce0 2024-09-13T17:23:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.590+0000 7f05cbe00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 0x7f05b006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.590+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05bc088a60 con 0x7f05d4154ce0 2024-09-13T17:23:42.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.591+0000 7f05d2a00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 0x7f05b006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:42.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.591+0000 7f05d2a00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 0x7f05b006d0f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f60 tx=0x7f05c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:42.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.594+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f05bc0576f0 con 0x7f05d4154ce0 2024-09-13T17:23:42.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:42.808+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f05d413ca30 con 0x7f05d4154ce0 2024-09-13T17:23:42.989 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-09-13T17:23:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.642+0000 7f05cbe00640 1 -- 172.21.15.28:0/2447073756 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v180) v1 ==== 118+0+0 (secure 0 0 0) 0x7f05d413ca30 con 0x7f05d4154ce0 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 msgr2=0x7f05b006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 0x7f05b006d0f0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f05c4000f60 tx=0x7f05c4009040 comp rx=0 tx=0).stop 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 msgr2=0x7f05d413ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f05bc00c410 tx=0x7f05bc00c8d0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 shutdown_connections 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f05b006ac30 0x7f05b006d0f0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05d415b7e0 0x7f05d415e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f05d4154ce0 0x7f05d413ae50 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 --2- 172.21.15.28:0/2447073756 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f05d40cbec0 0x7f05d4139060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 >> 172.21.15.28:0/2447073756 conn(0x7f05d40b0af0 msgr2=0x7f05d40cee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 shutdown_connections 2024-09-13T17:23:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.645+0000 7f05d9e00640 1 -- 172.21.15.28:0/2447073756 wait complete. 2024-09-13T17:23:43.667 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:23:43.667 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:23:43.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.828+0000 7ff1ef200640 1 Processor -- start 2024-09-13T17:23:43.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.829+0000 7ff1ef200640 1 -- start start 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ef200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff1f00cc910 0x7ff1f0072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ef200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00c78e0 con 0x7ff1f00cc530 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ef200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00cbe40 con 0x7ff1f00b06f0 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ef200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1f0070c90 con 0x7ff1f00cc910 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ede00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff1f00cc530 0x7ff1f015d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49318/0 (socket says 172.21.15.28:49318) 2024-09-13T17:23:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ede00640 1 -- 172.21.15.28:0/433797588 learned_addr learned my addr 172.21.15.28:0/433797588 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.830+0000 7ff1ed400640 1 --2- 172.21.15.28:0/433797588 >> v2:172.21.15.189:3300/0 conn(0x7ff1f00cc910 0x7ff1f0072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192187506 0 0) 0x7ff1f00c78e0 con 0x7ff1f00cc530 2024-09-13T17:23:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d4003520 con 0x7ff1f00cc530 2024-09-13T17:23:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1ed400640 1 -- 172.21.15.28:0/433797588 >> v1:172.21.15.189:6789/0 conn(0x7ff1f00b06f0 legacy=0x7ff1f00b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3601471896 0 0) 0x7ff1d4003520 con 0x7ff1f00cc530 2024-09-13T17:23:43.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1ed400640 1 -- 172.21.15.28:0/433797588 >> v1:172.21.15.28:6789/0 conn(0x7ff1f00cc530 legacy=0x7ff1f015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.831+0000 7ff1ed400640 1 -- 172.21.15.28:0/433797588 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0071000 con 0x7ff1f00cc910 2024-09-13T17:23:43.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.832+0000 7ff1ed400640 1 --2- 172.21.15.28:0/433797588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc910 0x7ff1f0072060 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097a0 tx=0x7ff1dc02a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cdf97fc8369ecd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:43.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0001670 con 0x7ff1f00cc910 2024-09-13T17:23:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc02ace0 con 0x7ff1f00cc910 2024-09-13T17:23:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1eca00640 1 -- 172.21.15.28:0/433797588 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc032640 con 0x7ff1f00cc910 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 -- 172.21.15.28:0/433797588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc910 msgr2=0x7ff1f0072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 --2- 172.21.15.28:0/433797588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc910 0x7ff1f0072060 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0097a0 tx=0x7ff1dc02a250 comp rx=0 tx=0).stop 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 -- 172.21.15.28:0/433797588 shutdown_connections 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 --2- 172.21.15.28:0/433797588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc910 0x7ff1f0072060 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 -- 172.21.15.28:0/433797588 >> 172.21.15.28:0/433797588 conn(0x7ff1f0069e80 msgr2=0x7ff1f006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 -- 172.21.15.28:0/433797588 shutdown_connections 2024-09-13T17:23:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.833+0000 7ff1ef200640 1 -- 172.21.15.28:0/433797588 wait complete. 2024-09-13T17:23:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.835+0000 7ff1ef200640 1 Processor -- start 2024-09-13T17:23:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.835+0000 7ff1ef200640 1 -- start start 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1f00b06f0 0x7ff1f00c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc530 0x7ff1f0152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f015e110 con 0x7ff1f00cc910 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f015e3f0 con 0x7ff1f00cc530 2024-09-13T17:23:43.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ef200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f015e870 con 0x7ff1f00b06f0 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35094/0 (socket says 172.21.15.28:35094) 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.836+0000 7ff1ee800640 1 -- 172.21.15.28:0/2107438050 learned_addr learned my addr 172.21.15.28:0/2107438050 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ede00640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1f00b06f0 0x7ff1f00c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ed400640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc530 0x7ff1f0152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 -- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1f00b06f0 msgr2=0x7ff1f00c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1f00b06f0 0x7ff1f00c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 -- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc530 msgr2=0x7ff1f0152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc530 0x7ff1f0152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 -- 172.21.15.28:0/2107438050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc009400 con 0x7ff1f00cc910 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ee800640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c730 tx=0x7ff1e800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003b70 con 0x7ff1f00cc910 2024-09-13T17:23:43.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e8003cd0 con 0x7ff1f00cc910 2024-09-13T17:23:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f015eb50 con 0x7ff1f00cc910 2024-09-13T17:23:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800aa90 con 0x7ff1f00cc910 2024-09-13T17:23:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.837+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f015ddc0 con 0x7ff1f00cc910 2024-09-13T17:23:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.838+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f0072060 con 0x7ff1f00cc910 2024-09-13T17:23:43.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.839+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff1e800abf0 con 0x7ff1f00cc910 2024-09-13T17:23:43.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.840+0000 7ff1e6000640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 0x7ff1d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:43.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.840+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff1e8089230 con 0x7ff1f00cc910 2024-09-13T17:23:43.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.841+0000 7ff1ede00640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 0x7ff1d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:43.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.841+0000 7ff1ede00640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 0x7ff1d006d0c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0000f60 tx=0x7ff1e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:43.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:43.844+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff1e8054370 con 0x7ff1f00cc910 2024-09-13T17:23:44.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.059+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff1f015e250 con 0x7ff1f00cc910 2024-09-13T17:23:44.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.644+0000 7ff1e6000640 1 -- 172.21.15.28:0/2107438050 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v181) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff1f015e250 con 0x7ff1f00cc910 2024-09-13T17:23:44.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 msgr2=0x7ff1d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 0x7ff1d006d0c0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0000f60 tx=0x7ff1e0009040 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 msgr2=0x7ff1f00b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 secure :-1 s=READY pgs=2591 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c730 tx=0x7ff1e800cbf0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 shutdown_connections 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff1d006ac00 0x7ff1d006d0c0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff1f00cc910 0x7ff1f00b2840 unknown :-1 s=CLOSED pgs=2591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1f00cc530 0x7ff1f0152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 --2- 172.21.15.28:0/2107438050 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff1f00b06f0 0x7ff1f00c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 >> 172.21.15.28:0/2107438050 conn(0x7ff1f0069e80 msgr2=0x7ff1f00b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.647+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 shutdown_connections 2024-09-13T17:23:44.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.648+0000 7ff1ef200640 1 -- 172.21.15.28:0/2107438050 wait complete. 2024-09-13T17:23:44.668 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:23:44.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:44.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.825+0000 7fc48ce00640 1 Processor -- start 2024-09-13T17:23:44.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.826+0000 7fc48ce00640 1 -- start start 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc48ce00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 0x7fc4880b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc48ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4880b0980 0x7fc4880af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc48ce00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4880b0d80 con 0x7fc488154a90 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc48ce00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48806b900 con 0x7fc4880b0980 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc48ce00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48806bbe0 con 0x7fc48806b360 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc487e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4880b0980 0x7fc4880af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc487e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc4880b0980 0x7fc4880af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35098/0 (socket says 172.21.15.28:35098) 2024-09-13T17:23:44.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.827+0000 7fc487e00640 1 -- 172.21.15.28:0/733011271 learned_addr learned my addr 172.21.15.28:0/733011271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc47ea00640 1 --2- 172.21.15.28:0/733011271 >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 0x7fc4880b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc487e00640 1 -- 172.21.15.28:0/733011271 >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 msgr2=0x7fc4880b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:44.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc487e00640 1 --2- 172.21.15.28:0/733011271 >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 0x7fc4880b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc487e00640 1 -- 172.21.15.28:0/733011271 >> v1:172.21.15.189:6789/0 conn(0x7fc488154a90 legacy=0x7fc48815b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc487e00640 1 -- 172.21.15.28:0/733011271 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48806bec0 con 0x7fc4880b0980 2024-09-13T17:23:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.828+0000 7fc47ea00640 1 --2- 172.21.15.28:0/733011271 >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 0x7fc4880b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:44.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.829+0000 7fc487e00640 1 --2- 172.21.15.28:0/733011271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc4880af4b0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fc48813dab0 tx=0x7fc48002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1ba39942b85ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.829+0000 7fc486a00640 1 -- 172.21.15.28:0/733011271 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480031600 con 0x7fc4880b0980 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.829+0000 7fc486a00640 1 -- 172.21.15.28:0/733011271 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48002d050 con 0x7fc4880b0980 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.829+0000 7fc486a00640 1 -- 172.21.15.28:0/733011271 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480031980 con 0x7fc4880b0980 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 -- 172.21.15.28:0/733011271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 msgr2=0x7fc4880af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 --2- 172.21.15.28:0/733011271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc4880af4b0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7fc48813dab0 tx=0x7fc48002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 -- 172.21.15.28:0/733011271 shutdown_connections 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 --2- 172.21.15.28:0/733011271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc4880af4b0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 --2- 172.21.15.28:0/733011271 >> v2:172.21.15.189:3300/0 conn(0x7fc48806b360 0x7fc4880b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 -- 172.21.15.28:0/733011271 >> 172.21.15.28:0/733011271 conn(0x7fc488072e40 msgr2=0x7fc488073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 -- 172.21.15.28:0/733011271 shutdown_connections 2024-09-13T17:23:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.830+0000 7fc48ce00640 1 -- 172.21.15.28:0/733011271 wait complete. 2024-09-13T17:23:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.831+0000 7fc48ce00640 1 Processor -- start 2024-09-13T17:23:44.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.831+0000 7fc48ce00640 1 -- start start 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 0x7fc4881f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc488154a90 0x7fc488156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881495e0 con 0x7fc4880b0980 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4881498c0 con 0x7fc488154a90 2024-09-13T17:23:44.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc48ce00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc488149ba0 con 0x7fc48806b360 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35112/0 (socket says 172.21.15.28:35112) 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 -- 172.21.15.28:0/2330739534 learned_addr learned my addr 172.21.15.28:0/2330739534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc487e00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc488154a90 0x7fc488156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc487400640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 0x7fc4881f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 -- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 msgr2=0x7fc4881f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 0x7fc4881f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 -- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc488154a90 msgr2=0x7fc488156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc488154a90 0x7fc488156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc47ea00640 1 -- 172.21.15.28:0/2330739534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480009b30 con 0x7fc4880b0980 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.833+0000 7fc487400640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 0x7fc4881f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:44.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc47ea00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c7f0 tx=0x7fc47000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470007a30 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc470007b90 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000ce40 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc488149e80 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.834+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48814ad80 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.835+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48814afa0 con 0x7fc4880b0980 2024-09-13T17:23:44.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.836+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc4700113f0 con 0x7fc4880b0980 2024-09-13T17:23:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.837+0000 7fc47fe00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 0x7fc46406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.837+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc470089500 con 0x7fc4880b0980 2024-09-13T17:23:44.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.838+0000 7fc487400640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 0x7fc46406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:44.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.838+0000 7fc487400640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 0x7fc46406d0c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc478006fd0 tx=0x7fc478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:44.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:44.842+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc4700546f0 con 0x7fc4880b0980 2024-09-13T17:23:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.057+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc48813d640 con 0x7fc4880b0980 2024-09-13T17:23:45.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.058+0000 7fc47fe00640 1 -- 172.21.15.28:0/2330739534 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7fc470057d10 con 0x7fc4880b0980 2024-09-13T17:23:45.060 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:45.060 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:45.060 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 msgr2=0x7fc46406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 0x7fc46406d0c0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fc478006fd0 tx=0x7fc478008040 comp rx=0 tx=0).stop 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 msgr2=0x7fc488156760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7fc47000c7f0 tx=0x7fc47000cbf0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 shutdown_connections 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc46406ac00 0x7fc46406d0c0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc488154a90 0x7fc488156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4880b0980 0x7fc488156760 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 --2- 172.21.15.28:0/2330739534 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc48806b360 0x7fc4881f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 >> 172.21.15.28:0/2330739534 conn(0x7fc488072e40 msgr2=0x7fc488158730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 shutdown_connections 2024-09-13T17:23:45.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.060+0000 7fc48ce00640 1 -- 172.21.15.28:0/2330739534 wait complete. 2024-09-13T17:23:45.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.234+0000 7fd672000640 1 Processor -- start 2024-09-13T17:23:45.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.234+0000 7fd672000640 1 -- start start 2024-09-13T17:23:45.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd672000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd66c0cabc0 0x7fd66c06c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd672000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd66c070cc0 0x7fd66c071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd672000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd66c15e100 con 0x7fd66c15a540 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd672000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0adb00 con 0x7fd66c0cabc0 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd672000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c071670 con 0x7fd66c070cc0 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd66c0cabc0 0x7fd66c06c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd66c0cabc0 0x7fd66c06c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35118/0 (socket says 172.21.15.28:35118) 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 -- 172.21.15.28:0/1599551996 learned_addr learned my addr 172.21.15.28:0/1599551996 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:45.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66be00640 1 --2- 172.21.15.28:0/1599551996 >> v2:172.21.15.189:3300/0 conn(0x7fd66c070cc0 0x7fd66c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 -- 172.21.15.28:0/1599551996 >> v2:172.21.15.189:3300/0 conn(0x7fd66c070cc0 msgr2=0x7fd66c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 --2- 172.21.15.28:0/1599551996 >> v2:172.21.15.189:3300/0 conn(0x7fd66c070cc0 0x7fd66c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 -- 172.21.15.28:0/1599551996 >> v1:172.21.15.189:6789/0 conn(0x7fd66c15a540 legacy=0x7fd66c15e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.235+0000 7fd66aa00640 1 -- 172.21.15.28:0/1599551996 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c15e3e0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.236+0000 7fd66aa00640 1 --2- 172.21.15.28:0/1599551996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c06c430 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fd66c070900 tx=0x7fd65802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84dbe7506ef16298 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.236+0000 7fd66a000640 1 -- 172.21.15.28:0/1599551996 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658032440 con 0x7fd66c0cabc0 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.236+0000 7fd66a000640 1 -- 172.21.15.28:0/1599551996 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6580325a0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.236+0000 7fd66a000640 1 -- 172.21.15.28:0/1599551996 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658032930 con 0x7fd66c0cabc0 2024-09-13T17:23:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 -- 172.21.15.28:0/1599551996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 msgr2=0x7fd66c06c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 --2- 172.21.15.28:0/1599551996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c06c430 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7fd66c070900 tx=0x7fd65802a210 comp rx=0 tx=0).stop 2024-09-13T17:23:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 -- 172.21.15.28:0/1599551996 shutdown_connections 2024-09-13T17:23:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 --2- 172.21.15.28:0/1599551996 >> v2:172.21.15.189:3300/0 conn(0x7fd66c070cc0 0x7fd66c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 --2- 172.21.15.28:0/1599551996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c06c430 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 -- 172.21.15.28:0/1599551996 >> 172.21.15.28:0/1599551996 conn(0x7fd66c069e80 msgr2=0x7fd66c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 -- 172.21.15.28:0/1599551996 shutdown_connections 2024-09-13T17:23:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.237+0000 7fd672000640 1 -- 172.21.15.28:0/1599551996 wait complete. 2024-09-13T17:23:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.239+0000 7fd672000640 1 Processor -- start 2024-09-13T17:23:45.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.239+0000 7fd672000640 1 -- start start 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 0x7fd66c14ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd66c15a540 0x7fd66c1f3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c14b680 con 0x7fd66c0cabc0 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c14c480 con 0x7fd66c070cc0 2024-09-13T17:23:45.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd672000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c151cf0 con 0x7fd66c15a540 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35132/0 (socket says 172.21.15.28:35132) 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 -- 172.21.15.28:0/2133804092 learned_addr learned my addr 172.21.15.28:0/2133804092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66be00640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd66c15a540 0x7fd66c1f3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66b400640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 0x7fd66c14ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 -- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd66c15a540 msgr2=0x7fd66c1f3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd66c15a540 0x7fd66c1f3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 -- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 msgr2=0x7fd66c14ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 0x7fd66c14ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66aa00640 1 -- 172.21.15.28:0/2133804092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580094d0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.240+0000 7fd66b400640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 0x7fd66c14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd66aa00640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd6580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658032440 con 0x7fd66c0cabc0 2024-09-13T17:23:45.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd658030980 con 0x7fd66c0cabc0 2024-09-13T17:23:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658030d30 con 0x7fd66c0cabc0 2024-09-13T17:23:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c151fd0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.241+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0af140 con 0x7fd66c0cabc0 2024-09-13T17:23:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.242+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c071130 con 0x7fd66c0cabc0 2024-09-13T17:23:45.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.243+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd658039040 con 0x7fd66c0cabc0 2024-09-13T17:23:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.244+0000 7fd663400640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 0x7fd64c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.244+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6580a9bb0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.244+0000 7fd66b400640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 0x7fd64c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.245+0000 7fd66b400640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 0x7fd64c06d0f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd65c007fd0 tx=0x7fd65c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.249+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd658074d80 con 0x7fd66c0cabc0 2024-09-13T17:23:45.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.428+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd66c15e910 con 0x7fd66c0cabc0 2024-09-13T17:23:45.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.430+0000 7fd663400640 1 -- 172.21.15.28:0/2133804092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+16519 (secure 0 0 0) 0x7fd6580783a0 con 0x7fd66c0cabc0 2024-09-13T17:23:45.431 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:45.431 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":219,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:42.631521+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:45.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.432+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 msgr2=0x7fd64c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.432+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 0x7fd64c06d0f0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd65c007fd0 tx=0x7fd65c009040 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.432+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 msgr2=0x7fd66c14b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.432+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7fd658000c00 tx=0x7fd6580046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 shutdown_connections 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd64c06ac30 0x7fd64c06d0f0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd66c15a540 0x7fd66c1f3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd66c0cabc0 0x7fd66c14b140 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 --2- 172.21.15.28:0/2133804092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd66c070cc0 0x7fd66c14ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 >> 172.21.15.28:0/2133804092 conn(0x7fd66c069e80 msgr2=0x7fd66c0cd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 shutdown_connections 2024-09-13T17:23:45.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.433+0000 7fd672000640 1 -- 172.21.15.28:0/2133804092 wait complete. 2024-09-13T17:23:45.450 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:23:45.450 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:23:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.604+0000 7f0b30600640 1 Processor -- start 2024-09-13T17:23:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.604+0000 7f0b30600640 1 -- start start 2024-09-13T17:23:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b30600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b2815d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b30600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b280acf60 0x7f0b28160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b30600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28146550 con 0x7f0b28160310 2024-09-13T17:23:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b30600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2815cd20 con 0x7f0b280acf60 2024-09-13T17:23:45.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b30600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2815d280 con 0x7f0b28146170 2024-09-13T17:23:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b2e600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0b280acf60 0x7f0b28160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b2d200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0b28160310 0x7f0b280ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49320/0 (socket says 172.21.15.28:49320) 2024-09-13T17:23:45.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.605+0000 7f0b2d200640 1 -- 172.21.15.28:0/3988134113 learned_addr learned my addr 172.21.15.28:0/3988134113 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2e600640 1 -- 172.21.15.28:0/3988134113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 msgr2=0x7f0b2815d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2dc00640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b2815d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b27e00640 1 -- 172.21.15.28:0/3988134113 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931871400 0 0) 0x7f0b28146550 con 0x7f0b28160310 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2e600640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b2815d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2e600640 1 -- 172.21.15.28:0/3988134113 >> v1:172.21.15.28:6789/0 conn(0x7f0b28160310 legacy=0x7f0b280ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2e600640 1 -- 172.21.15.28:0/3988134113 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2815dc50 con 0x7f0b280acf60 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.606+0000 7f0b2dc00640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b2815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.607+0000 7f0b2e600640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b28160750 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009d90 tx=0x7f0b2002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65127c99ad45489 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.607+0000 7f0b27e00640 1 -- 172.21.15.28:0/3988134113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20007740 con 0x7f0b280acf60 2024-09-13T17:23:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.607+0000 7f0b27e00640 1 -- 172.21.15.28:0/3988134113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2002d050 con 0x7f0b280acf60 2024-09-13T17:23:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.607+0000 7f0b27e00640 1 -- 172.21.15.28:0/3988134113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20007ac0 con 0x7f0b280acf60 2024-09-13T17:23:45.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 -- 172.21.15.28:0/3988134113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 msgr2=0x7f0b28160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b28160750 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009d90 tx=0x7f0b2002a660 comp rx=0 tx=0).stop 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 -- 172.21.15.28:0/3988134113 shutdown_connections 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b28160750 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 --2- 172.21.15.28:0/3988134113 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b2815d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 -- 172.21.15.28:0/3988134113 >> 172.21.15.28:0/3988134113 conn(0x7f0b28069e80 msgr2=0x7f0b2806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 -- 172.21.15.28:0/3988134113 shutdown_connections 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.608+0000 7f0b30600640 1 -- 172.21.15.28:0/3988134113 wait complete. 2024-09-13T17:23:45.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.609+0000 7f0b30600640 1 Processor -- start 2024-09-13T17:23:45.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.610+0000 7f0b30600640 1 -- start start 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.610+0000 7f0b30600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.610+0000 7f0b30600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b280ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.610+0000 7f0b30600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 0x7f0b280ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b30600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b281556a0 con 0x7f0b280acf60 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b30600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28145330 con 0x7f0b28160310 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b30600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b28144da0 con 0x7f0b28146170 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35160/0 (socket says 172.21.15.28:35160) 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 -- 172.21.15.28:0/112347327 learned_addr learned my addr 172.21.15.28:0/112347327 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:45.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2d200640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b280ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2e600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 0x7f0b280ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 -- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 msgr2=0x7f0b280ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b280ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 -- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 msgr2=0x7f0b280ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 0x7f0b280ce7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 -- 172.21.15.28:0/112347327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009a70 con 0x7f0b280acf60 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2e600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 0x7f0b280ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.611+0000 7f0b2dc00640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c7f0 tx=0x7f0b1c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.612+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c0077c0 con 0x7f0b280acf60 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.612+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1c007920 con 0x7f0b280acf60 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.612+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1c007c90 con 0x7f0b280acf60 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.612+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28154a20 con 0x7f0b280acf60 2024-09-13T17:23:45.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.612+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28154fe0 con 0x7f0b280acf60 2024-09-13T17:23:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.613+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b281f23f0 con 0x7f0b280acf60 2024-09-13T17:23:45.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.614+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0b1c0113f0 con 0x7f0b280acf60 2024-09-13T17:23:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.615+0000 7f0b25600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.615+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0b1c0895b0 con 0x7f0b280acf60 2024-09-13T17:23:45.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.616+0000 7f0b2d200640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:45.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.616+0000 7f0b2d200640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0b2815c610 tx=0x7f0b14007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:45.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.619+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0b1c054830 con 0x7f0b280acf60 2024-09-13T17:23:45.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.793+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0b2815cf10 con 0x7f0b0806ac00 2024-09-13T17:23:45.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.802+0000 7f0b25600640 1 -- 172.21.15.28:0/112347327 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289528 (secure 0 0 0) 0x7f0b14004c70 con 0x7f0b0806ac00 2024-09-13T17:23:45.808 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:45.810 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 msgr2=0x7f0b0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f0b2815c610 tx=0x7f0b14007d50 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 msgr2=0x7f0b281568b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7f0b1c00c7f0 tx=0x7f0b1c00ccb0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 shutdown_connections 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0b0806ac00 0x7f0b0806d0c0 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b28160310 0x7f0b280ce7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0b28146170 0x7f0b280ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 --2- 172.21.15.28:0/112347327 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b280acf60 0x7f0b281568b0 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:45.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.806+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 >> 172.21.15.28:0/112347327 conn(0x7f0b28069e80 msgr2=0x7f0b280af2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.807+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 shutdown_connections 2024-09-13T17:23:45.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:45.807+0000 7f0b30600640 1 -- 172.21.15.28:0/112347327 wait complete. 2024-09-13T17:23:45.824 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2024-09-13T17:23:45.122078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":558448,"kb_used_data":22756,"kb_used_omap":12,"kb_used_meta":535539,"kb_avail":754416272,"statfs":{"total":773094113280,"available":772522262528,"internally_reserved":0,"allocated":23302144,"data_stored":10936618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":548392505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.925949"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660902+0000","last_change":"2024-09-13T17:23:41.651300+0000","last_active":"2024-09-13T17:23:41.660902+0000","last_peered":"2024-09-13T17:23:41.660902+0000","last_clean":"2024-09-13T17:23:41.660902+0000","last_became_active":"2024-09-13T17:23:41.651060+0000","last_became_peered":"2024-09-13T17:23:41.651060+0000","last_unstale":"2024-09-13T17:23:41.660902+0000","last_undegraded":"2024-09-13T17:23:41.660902+0000","last_fullsized":"2024-09-13T17:23:41.660902+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:26.860796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660733+0000","last_change":"2024-09-13T17:23:41.653072+0000","last_active":"2024-09-13T17:23:41.660733+0000","last_peered":"2024-09-13T17:23:41.660733+0000","last_clean":"2024-09-13T17:23:41.660733+0000","last_became_active":"2024-09-13T17:23:41.652481+0000","last_became_peered":"2024-09-13T17:23:41.652481+0000","last_unstale":"2024-09-13T17:23:41.660733+0000","last_undegraded":"2024-09-13T17:23:41.660733+0000","last_fullsized":"2024-09-13T17:23:41.660733+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:35.831865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658919+0000","last_change":"2024-09-13T17:23:41.652700+0000","last_active":"2024-09-13T17:23:41.658919+0000","last_peered":"2024-09-13T17:23:41.658919+0000","last_clean":"2024-09-13T17:23:41.658919+0000","last_became_active":"2024-09-13T17:23:41.652355+0000","last_became_peered":"2024-09-13T17:23:41.652355+0000","last_unstale":"2024-09-13T17:23:41.658919+0000","last_undegraded":"2024-09-13T17:23:41.658919+0000","last_fullsized":"2024-09-13T17:23:41.658919+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:16.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.646223+0000","last_change":"2024-09-13T17:23:41.645536+0000","last_active":"2024-09-13T17:23:41.646223+0000","last_peered":"2024-09-13T17:23:41.646223+0000","last_clean":"2024-09-13T17:23:41.646223+0000","last_became_active":"2024-09-13T17:23:41.643906+0000","last_became_peered":"2024-09-13T17:23:41.643906+0000","last_unstale":"2024-09-13T17:23:41.646223+0000","last_undegraded":"2024-09-13T17:23:41.646223+0000","last_fullsized":"2024-09-13T17:23:41.646223+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:54.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639827+0000","last_change":"2024-09-13T17:23:41.643574+0000","last_active":"2024-09-13T17:23:42.639827+0000","last_peered":"2024-09-13T17:23:42.639827+0000","last_clean":"2024-09-13T17:23:42.639827+0000","last_became_active":"2024-09-13T17:23:41.643298+0000","last_became_peered":"2024-09-13T17:23:41.643298+0000","last_unstale":"2024-09-13T17:23:42.639827+0000","last_undegraded":"2024-09-13T17:23:42.639827+0000","last_fullsized":"2024-09-13T17:23:42.639827+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:39.164404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662443+0000","last_change":"2024-09-13T17:23:41.653886+0000","last_active":"2024-09-13T17:23:41.662443+0000","last_peered":"2024-09-13T17:23:41.662443+0000","last_clean":"2024-09-13T17:23:41.662443+0000","last_became_active":"2024-09-13T17:23:41.653639+0000","last_became_peered":"2024-09-13T17:23:41.653639+0000","last_unstale":"2024-09-13T17:23:41.662443+0000","last_undegraded":"2024-09-13T17:23:41.662443+0000","last_fullsized":"2024-09-13T17:23:41.662443+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:55:23.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665098+0000","last_change":"2024-09-13T17:23:41.664121+0000","last_active":"2024-09-13T17:23:41.665098+0000","last_peered":"2024-09-13T17:23:41.665098+0000","last_clean":"2024-09-13T17:23:41.665098+0000","last_became_active":"2024-09-13T17:23:41.663791+0000","last_became_peered":"2024-09-13T17:23:41.663791+0000","last_unstale":"2024-09-13T17:23:41.665098+0000","last_undegraded":"2024-09-13T17:23:41.665098+0000","last_fullsized":"2024-09-13T17:23:41.665098+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:46.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639414+0000","last_change":"2024-09-13T17:23:41.650518+0000","last_active":"2024-09-13T17:23:42.639414+0000","last_peered":"2024-09-13T17:23:42.639414+0000","last_clean":"2024-09-13T17:23:42.639414+0000","last_became_active":"2024-09-13T17:23:41.650185+0000","last_became_peered":"2024-09-13T17:23:41.650185+0000","last_unstale":"2024-09-13T17:23:42.639414+0000","last_undegraded":"2024-09-13T17:23:42.639414+0000","last_fullsized":"2024-09-13T17:23:42.639414+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:38.751185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683081+0000","last_change":"2024-09-13T17:23:41.664949+0000","last_active":"2024-09-13T17:23:42.683081+0000","last_peered":"2024-09-13T17:23:42.683081+0000","last_clean":"2024-09-13T17:23:42.683081+0000","last_became_active":"2024-09-13T17:23:41.664644+0000","last_became_peered":"2024-09-13T17:23:41.664644+0000","last_unstale":"2024-09-13T17:23:42.683081+0000","last_undegraded":"2024-09-13T17:23:42.683081+0000","last_fullsized":"2024-09-13T17:23:42.683081+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:15.211955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.655744+0000","last_change":"2024-09-13T17:23:41.654505+0000","last_active":"2024-09-13T17:23:41.655744+0000","last_peered":"2024-09-13T17:23:41.655744+0000","last_clean":"2024-09-13T17:23:41.655744+0000","last_became_active":"2024-09-13T17:23:41.654170+0000","last_became_peered":"2024-09-13T17:23:41.654170+0000","last_unstale":"2024-09-13T17:23:41.655744+0000","last_undegraded":"2024-09-13T17:23:41.655744+0000","last_fullsized":"2024-09-13T17:23:41.655744+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:35.831865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639388+0000","last_change":"2024-09-13T17:23:41.662856+0000","last_active":"2024-09-13T17:23:42.639388+0000","last_peered":"2024-09-13T17:23:42.639388+0000","last_clean":"2024-09-13T17:23:42.639388+0000","last_became_active":"2024-09-13T17:23:41.662578+0000","last_became_peered":"2024-09-13T17:23:41.662578+0000","last_unstale":"2024-09-13T17:23:42.639388+0000","last_undegraded":"2024-09-13T17:23:42.639388+0000","last_fullsized":"2024-09-13T17:23:42.639388+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:31.029743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640090+0000","last_change":"2024-09-13T17:23:41.660452+0000","last_active":"2024-09-13T17:23:42.640090+0000","last_peered":"2024-09-13T17:23:42.640090+0000","last_clean":"2024-09-13T17:23:42.640090+0000","last_became_active":"2024-09-13T17:23:41.660115+0000","last_became_peered":"2024-09-13T17:23:41.660115+0000","last_unstale":"2024-09-13T17:23:42.640090+0000","last_undegraded":"2024-09-13T17:23:42.640090+0000","last_fullsized":"2024-09-13T17:23:42.640090+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:31:30.719669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658526+0000","last_change":"2024-09-13T17:23:41.656591+0000","last_active":"2024-09-13T17:23:41.658526+0000","last_peered":"2024-09-13T17:23:41.658526+0000","last_clean":"2024-09-13T17:23:41.658526+0000","last_became_active":"2024-09-13T17:23:41.656245+0000","last_became_peered":"2024-09-13T17:23:41.656245+0000","last_unstale":"2024-09-13T17:23:41.658526+0000","last_undegraded":"2024-09-13T17:23:41.658526+0000","last_fullsized":"2024-09-13T17:23:41.658526+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:29.643143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665141+0000","last_change":"2024-09-13T17:23:41.661067+0000","last_active":"2024-09-13T17:23:41.665141+0000","last_peered":"2024-09-13T17:23:41.665141+0000","last_clean":"2024-09-13T17:23:41.665141+0000","last_became_active":"2024-09-13T17:23:41.660248+0000","last_became_peered":"2024-09-13T17:23:41.660248+0000","last_unstale":"2024-09-13T17:23:41.665141+0000","last_undegraded":"2024-09-13T17:23:41.665141+0000","last_fullsized":"2024-09-13T17:23:41.665141+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:52.341148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.682914+0000","last_change":"2024-09-13T17:23:41.655686+0000","last_active":"2024-09-13T17:23:42.682914+0000","last_peered":"2024-09-13T17:23:42.682914+0000","last_clean":"2024-09-13T17:23:42.682914+0000","last_became_active":"2024-09-13T17:23:41.655246+0000","last_became_peered":"2024-09-13T17:23:41.655246+0000","last_unstale":"2024-09-13T17:23:42.682914+0000","last_undegraded":"2024-09-13T17:23:42.682914+0000","last_fullsized":"2024-09-13T17:23:42.682914+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660950+0000","last_change":"2024-09-13T17:23:41.649591+0000","last_active":"2024-09-13T17:23:41.660950+0000","last_peered":"2024-09-13T17:23:41.660950+0000","last_clean":"2024-09-13T17:23:41.660950+0000","last_became_active":"2024-09-13T17:23:41.648957+0000","last_became_peered":"2024-09-13T17:23:41.648957+0000","last_unstale":"2024-09-13T17:23:41.660950+0000","last_undegraded":"2024-09-13T17:23:41.660950+0000","last_fullsized":"2024-09-13T17:23:41.660950+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:56.708929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640115+0000","last_change":"2024-09-13T17:23:41.663023+0000","last_active":"2024-09-13T17:23:42.640115+0000","last_peered":"2024-09-13T17:23:42.640115+0000","last_clean":"2024-09-13T17:23:42.640115+0000","last_became_active":"2024-09-13T17:23:41.662618+0000","last_became_peered":"2024-09-13T17:23:41.662618+0000","last_unstale":"2024-09-13T17:23:42.640115+0000","last_undegraded":"2024-09-13T17:23:42.640115+0000","last_fullsized":"2024-09-13T17:23:42.640115+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:33.823085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660696+0000","last_change":"2024-09-13T17:23:41.652724+0000","last_active":"2024-09-13T17:23:41.660696+0000","last_peered":"2024-09-13T17:23:41.660696+0000","last_clean":"2024-09-13T17:23:41.660696+0000","last_became_active":"2024-09-13T17:23:41.651872+0000","last_became_peered":"2024-09-13T17:23:41.651872+0000","last_unstale":"2024-09-13T17:23:41.660696+0000","last_undegraded":"2024-09-13T17:23:41.660696+0000","last_fullsized":"2024-09-13T17:23:41.660696+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:57.781369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640125+0000","last_change":"2024-09-13T17:23:41.665294+0000","last_active":"2024-09-13T17:23:42.640125+0000","last_peered":"2024-09-13T17:23:42.640125+0000","last_clean":"2024-09-13T17:23:42.640125+0000","last_became_active":"2024-09-13T17:23:41.664893+0000","last_became_peered":"2024-09-13T17:23:41.664893+0000","last_unstale":"2024-09-13T17:23:42.640125+0000","last_undegraded":"2024-09-13T17:23:42.640125+0000","last_fullsized":"2024-09-13T17:23:42.640125+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:38.177856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656455+0000","last_change":"2024-09-13T17:23:41.648334+0000","last_active":"2024-09-13T17:23:41.656455+0000","last_peered":"2024-09-13T17:23:41.656455+0000","last_clean":"2024-09-13T17:23:41.656455+0000","last_became_active":"2024-09-13T17:23:41.648088+0000","last_became_peered":"2024-09-13T17:23:41.648088+0000","last_unstale":"2024-09-13T17:23:41.656455+0000","last_undegraded":"2024-09-13T17:23:41.656455+0000","last_fullsized":"2024-09-13T17:23:41.656455+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:23.657853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639684+0000","last_change":"2024-09-13T17:23:41.648748+0000","last_active":"2024-09-13T17:23:42.639684+0000","last_peered":"2024-09-13T17:23:42.639684+0000","last_clean":"2024-09-13T17:23:42.639684+0000","last_became_active":"2024-09-13T17:23:41.648349+0000","last_became_peered":"2024-09-13T17:23:41.648349+0000","last_unstale":"2024-09-13T17:23:42.639684+0000","last_undegraded":"2024-09-13T17:23:42.639684+0000","last_fullsized":"2024-09-13T17:23:42.639684+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:54.836377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661582+0000","last_change":"2024-09-13T17:23:41.660829+0000","last_active":"2024-09-13T17:23:41.661582+0000","last_peered":"2024-09-13T17:23:41.661582+0000","last_clean":"2024-09-13T17:23:41.661582+0000","last_became_active":"2024-09-13T17:23:41.660392+0000","last_became_peered":"2024-09-13T17:23:41.660392+0000","last_unstale":"2024-09-13T17:23:41.661582+0000","last_undegraded":"2024-09-13T17:23:41.661582+0000","last_fullsized":"2024-09-13T17:23:41.661582+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:46.760609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639498+0000","last_change":"2024-09-13T17:23:41.659060+0000","last_active":"2024-09-13T17:23:42.639498+0000","last_peered":"2024-09-13T17:23:42.639498+0000","last_clean":"2024-09-13T17:23:42.639498+0000","last_became_active":"2024-09-13T17:23:41.658709+0000","last_became_peered":"2024-09-13T17:23:41.658709+0000","last_unstale":"2024-09-13T17:23:42.639498+0000","last_undegraded":"2024-09-13T17:23:42.639498+0000","last_fullsized":"2024-09-13T17:23:42.639498+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:45.955447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660504+0000","last_change":"2024-09-13T17:23:41.654993+0000","last_active":"2024-09-13T17:23:41.660504+0000","last_peered":"2024-09-13T17:23:41.660504+0000","last_clean":"2024-09-13T17:23:41.660504+0000","last_became_active":"2024-09-13T17:23:41.654723+0000","last_became_peered":"2024-09-13T17:23:41.654723+0000","last_unstale":"2024-09-13T17:23:41.660504+0000","last_undegraded":"2024-09-13T17:23:41.660504+0000","last_fullsized":"2024-09-13T17:23:41.660504+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:54.666406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.646339+0000","last_change":"2024-09-13T17:23:41.642871+0000","last_active":"2024-09-13T17:23:41.646339+0000","last_peered":"2024-09-13T17:23:41.646339+0000","last_clean":"2024-09-13T17:23:41.646339+0000","last_became_active":"2024-09-13T17:23:41.642627+0000","last_became_peered":"2024-09-13T17:23:41.642627+0000","last_unstale":"2024-09-13T17:23:41.646339+0000","last_undegraded":"2024-09-13T17:23:41.646339+0000","last_fullsized":"2024-09-13T17:23:41.646339+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:12.989466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.657642+0000","last_change":"2024-09-13T17:23:41.655302+0000","last_active":"2024-09-13T17:23:41.657642+0000","last_peered":"2024-09-13T17:23:41.657642+0000","last_clean":"2024-09-13T17:23:41.657642+0000","last_became_active":"2024-09-13T17:23:41.654824+0000","last_became_peered":"2024-09-13T17:23:41.654824+0000","last_unstale":"2024-09-13T17:23:41.657642+0000","last_undegraded":"2024-09-13T17:23:41.657642+0000","last_fullsized":"2024-09-13T17:23:41.657642+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:52.341148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639419+0000","last_change":"2024-09-13T17:23:41.660753+0000","last_active":"2024-09-13T17:23:42.639419+0000","last_peered":"2024-09-13T17:23:42.639419+0000","last_clean":"2024-09-13T17:23:42.639419+0000","last_became_active":"2024-09-13T17:23:41.660451+0000","last_became_peered":"2024-09-13T17:23:41.660451+0000","last_unstale":"2024-09-13T17:23:42.639419+0000","last_undegraded":"2024-09-13T17:23:42.639419+0000","last_fullsized":"2024-09-13T17:23:42.639419+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:47.304458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661737+0000","last_change":"2024-09-13T17:23:41.657866+0000","last_active":"2024-09-13T17:23:41.661737+0000","last_peered":"2024-09-13T17:23:41.661737+0000","last_clean":"2024-09-13T17:23:41.661737+0000","last_became_active":"2024-09-13T17:23:41.657509+0000","last_became_peered":"2024-09-13T17:23:41.657509+0000","last_unstale":"2024-09-13T17:23:41.661737+0000","last_undegraded":"2024-09-13T17:23:41.661737+0000","last_fullsized":"2024-09-13T17:23:41.661737+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639257+0000","last_change":"2024-09-13T17:23:41.658664+0000","last_active":"2024-09-13T17:23:42.639257+0000","last_peered":"2024-09-13T17:23:42.639257+0000","last_clean":"2024-09-13T17:23:42.639257+0000","last_became_active":"2024-09-13T17:23:41.653139+0000","last_became_peered":"2024-09-13T17:23:41.653139+0000","last_unstale":"2024-09-13T17:23:42.639257+0000","last_undegraded":"2024-09-13T17:23:42.639257+0000","last_fullsized":"2024-09-13T17:23:42.639257+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:56.708929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639805+0000","last_change":"2024-09-13T17:23:41.647731+0000","last_active":"2024-09-13T17:23:42.639805+0000","last_peered":"2024-09-13T17:23:42.639805+0000","last_clean":"2024-09-13T17:23:42.639805+0000","last_became_active":"2024-09-13T17:23:41.647313+0000","last_became_peered":"2024-09-13T17:23:41.647313+0000","last_unstale":"2024-09-13T17:23:42.639805+0000","last_undegraded":"2024-09-13T17:23:42.639805+0000","last_fullsized":"2024-09-13T17:23:42.639805+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:42.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661387+0000","last_change":"2024-09-13T17:23:41.650632+0000","last_active":"2024-09-13T17:23:41.661387+0000","last_peered":"2024-09-13T17:23:41.661387+0000","last_clean":"2024-09-13T17:23:41.661387+0000","last_became_active":"2024-09-13T17:23:41.648143+0000","last_became_peered":"2024-09-13T17:23:41.648143+0000","last_unstale":"2024-09-13T17:23:41.661387+0000","last_undegraded":"2024-09-13T17:23:41.661387+0000","last_fullsized":"2024-09-13T17:23:41.661387+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:54.536858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656234+0000","last_change":"2024-09-13T17:23:41.646966+0000","last_active":"2024-09-13T17:23:41.656234+0000","last_peered":"2024-09-13T17:23:41.656234+0000","last_clean":"2024-09-13T17:23:41.656234+0000","last_became_active":"2024-09-13T17:23:41.646664+0000","last_became_peered":"2024-09-13T17:23:41.646664+0000","last_unstale":"2024-09-13T17:23:41.656234+0000","last_undegraded":"2024-09-13T17:23:41.656234+0000","last_fullsized":"2024-09-13T17:23:41.656234+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:17:27.661820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.670021+0000","last_change":"2024-09-13T17:23:41.667798+0000","last_active":"2024-09-13T17:23:41.670021+0000","last_peered":"2024-09-13T17:23:41.670021+0000","last_clean":"2024-09-13T17:23:41.670021+0000","last_became_active":"2024-09-13T17:23:41.666086+0000","last_became_peered":"2024-09-13T17:23:41.666086+0000","last_unstale":"2024-09-13T17:23:41.670021+0000","last_undegraded":"2024-09-13T17:23:41.670021+0000","last_fullsized":"2024-09-13T17:23:41.670021+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:50.550423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658269+0000","last_change":"2024-09-13T17:23:41.218763+0000","last_active":"2024-09-13T17:23:41.658269+0000","last_peered":"2024-09-13T17:23:41.658269+0000","last_clean":"2024-09-13T17:23:41.658269+0000","last_became_active":"2024-09-13T17:23:41.218450+0000","last_became_peered":"2024-09-13T17:23:41.218450+0000","last_unstale":"2024-09-13T17:23:41.658269+0000","last_undegraded":"2024-09-13T17:23:41.658269+0000","last_fullsized":"2024-09-13T17:23:41.658269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:23.885396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639194+0000","last_change":"2024-09-13T17:23:41.661650+0000","last_active":"2024-09-13T17:23:42.639194+0000","last_peered":"2024-09-13T17:23:42.639194+0000","last_clean":"2024-09-13T17:23:42.639194+0000","last_became_active":"2024-09-13T17:23:41.660515+0000","last_became_peered":"2024-09-13T17:23:41.660515+0000","last_unstale":"2024-09-13T17:23:42.639194+0000","last_undegraded":"2024-09-13T17:23:42.639194+0000","last_fullsized":"2024-09-13T17:23:42.639194+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:23.657853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669191+0000","last_change":"2024-09-13T17:23:41.222255+0000","last_active":"2024-09-13T17:23:41.669191+0000","last_peered":"2024-09-13T17:23:41.669191+0000","last_clean":"2024-09-13T17:23:41.669191+0000","last_became_active":"2024-09-13T17:23:41.221982+0000","last_became_peered":"2024-09-13T17:23:41.221982+0000","last_unstale":"2024-09-13T17:23:41.669191+0000","last_undegraded":"2024-09-13T17:23:41.669191+0000","last_fullsized":"2024-09-13T17:23:41.669191+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:35.121129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640250+0000","last_change":"2024-09-13T17:23:41.663289+0000","last_active":"2024-09-13T17:23:42.640250+0000","last_peered":"2024-09-13T17:23:42.640250+0000","last_clean":"2024-09-13T17:23:42.640250+0000","last_became_active":"2024-09-13T17:23:41.662870+0000","last_became_peered":"2024-09-13T17:23:41.662870+0000","last_unstale":"2024-09-13T17:23:42.640250+0000","last_undegraded":"2024-09-13T17:23:42.640250+0000","last_fullsized":"2024-09-13T17:23:42.640250+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:44.440920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.667149+0000","last_change":"2024-09-13T17:23:41.221819+0000","last_active":"2024-09-13T17:23:41.667149+0000","last_peered":"2024-09-13T17:23:41.667149+0000","last_clean":"2024-09-13T17:23:41.667149+0000","last_became_active":"2024-09-13T17:23:41.221433+0000","last_became_peered":"2024-09-13T17:23:41.221433+0000","last_unstale":"2024-09-13T17:23:41.667149+0000","last_undegraded":"2024-09-13T17:23:41.667149+0000","last_fullsized":"2024-09-13T17:23:41.667149+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:54.050404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639091+0000","last_change":"2024-09-13T17:23:41.655695+0000","last_active":"2024-09-13T17:23:42.639091+0000","last_peered":"2024-09-13T17:23:42.639091+0000","last_clean":"2024-09-13T17:23:42.639091+0000","last_became_active":"2024-09-13T17:23:41.655375+0000","last_became_peered":"2024-09-13T17:23:41.655375+0000","last_unstale":"2024-09-13T17:23:42.639091+0000","last_undegraded":"2024-09-13T17:23:42.639091+0000","last_fullsized":"2024-09-13T17:23:42.639091+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:03.503645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662931+0000","last_change":"2024-09-13T17:23:41.215473+0000","last_active":"2024-09-13T17:23:41.662931+0000","last_peered":"2024-09-13T17:23:41.662931+0000","last_clean":"2024-09-13T17:23:41.662931+0000","last_became_active":"2024-09-13T17:23:41.215169+0000","last_became_peered":"2024-09-13T17:23:41.215169+0000","last_unstale":"2024-09-13T17:23:41.662931+0000","last_undegraded":"2024-09-13T17:23:41.662931+0000","last_fullsized":"2024-09-13T17:23:41.662931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:31.364289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683202+0000","last_change":"2024-09-13T17:23:41.651615+0000","last_active":"2024-09-13T17:23:42.683202+0000","last_peered":"2024-09-13T17:23:42.683202+0000","last_clean":"2024-09-13T17:23:42.683202+0000","last_became_active":"2024-09-13T17:23:41.651204+0000","last_became_peered":"2024-09-13T17:23:41.651204+0000","last_unstale":"2024-09-13T17:23:42.683202+0000","last_undegraded":"2024-09-13T17:23:42.683202+0000","last_fullsized":"2024-09-13T17:23:42.683202+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:50.579369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656102+0000","last_change":"2024-09-13T17:23:41.211244+0000","last_active":"2024-09-13T17:23:41.656102+0000","last_peered":"2024-09-13T17:23:41.656102+0000","last_clean":"2024-09-13T17:23:41.656102+0000","last_became_active":"2024-09-13T17:23:41.210982+0000","last_became_peered":"2024-09-13T17:23:41.210982+0000","last_unstale":"2024-09-13T17:23:41.656102+0000","last_undegraded":"2024-09-13T17:23:41.656102+0000","last_fullsized":"2024-09-13T17:23:41.656102+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:23.885396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658591+0000","last_change":"2024-09-13T17:23:41.650569+0000","last_active":"2024-09-13T17:23:41.658591+0000","last_peered":"2024-09-13T17:23:41.658591+0000","last_clean":"2024-09-13T17:23:41.658591+0000","last_became_active":"2024-09-13T17:23:41.649276+0000","last_became_peered":"2024-09-13T17:23:41.649276+0000","last_unstale":"2024-09-13T17:23:41.658591+0000","last_undegraded":"2024-09-13T17:23:41.658591+0000","last_fullsized":"2024-09-13T17:23:41.658591+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:56:32.718664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.640588+0000","last_change":"2024-09-13T17:23:41.223475+0000","last_active":"2024-09-13T17:23:41.640588+0000","last_peered":"2024-09-13T17:23:41.640588+0000","last_clean":"2024-09-13T17:23:41.640588+0000","last_became_active":"2024-09-13T17:23:41.223112+0000","last_became_peered":"2024-09-13T17:23:41.223112+0000","last_unstale":"2024-09-13T17:23:41.640588+0000","last_undegraded":"2024-09-13T17:23:41.640588+0000","last_fullsized":"2024-09-13T17:23:41.640588+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:51.276487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656774+0000","last_change":"2024-09-13T17:23:41.649173+0000","last_active":"2024-09-13T17:23:41.656774+0000","last_peered":"2024-09-13T17:23:41.656774+0000","last_clean":"2024-09-13T17:23:41.656774+0000","last_became_active":"2024-09-13T17:23:41.648851+0000","last_became_peered":"2024-09-13T17:23:41.648851+0000","last_unstale":"2024-09-13T17:23:41.656774+0000","last_undegraded":"2024-09-13T17:23:41.656774+0000","last_fullsized":"2024-09-13T17:23:41.656774+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:46.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643109+0000","last_change":"2024-09-13T17:23:41.220852+0000","last_active":"2024-09-13T17:23:41.643109+0000","last_peered":"2024-09-13T17:23:41.643109+0000","last_clean":"2024-09-13T17:23:41.643109+0000","last_became_active":"2024-09-13T17:23:41.220582+0000","last_became_peered":"2024-09-13T17:23:41.220582+0000","last_unstale":"2024-09-13T17:23:41.643109+0000","last_undegraded":"2024-09-13T17:23:41.643109+0000","last_fullsized":"2024-09-13T17:23:41.643109+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:18.526170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660712+0000","last_change":"2024-09-13T17:23:41.652918+0000","last_active":"2024-09-13T17:23:41.660712+0000","last_peered":"2024-09-13T17:23:41.660712+0000","last_clean":"2024-09-13T17:23:41.660712+0000","last_became_active":"2024-09-13T17:23:41.652183+0000","last_became_peered":"2024-09-13T17:23:41.652183+0000","last_unstale":"2024-09-13T17:23:41.660712+0000","last_undegraded":"2024-09-13T17:23:41.660712+0000","last_fullsized":"2024-09-13T17:23:41.660712+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:48.715837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640140+0000","last_change":"2024-09-13T17:23:41.221596+0000","last_active":"2024-09-13T17:23:42.640140+0000","last_peered":"2024-09-13T17:23:42.640140+0000","last_clean":"2024-09-13T17:23:42.640140+0000","last_became_active":"2024-09-13T17:23:41.221305+0000","last_became_peered":"2024-09-13T17:23:41.221305+0000","last_unstale":"2024-09-13T17:23:42.640140+0000","last_undegraded":"2024-09-13T17:23:42.640140+0000","last_fullsized":"2024-09-13T17:23:42.640140+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:14.204122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639556+0000","last_change":"2024-09-13T17:23:41.652093+0000","last_active":"2024-09-13T17:23:42.639556+0000","last_peered":"2024-09-13T17:23:42.639556+0000","last_clean":"2024-09-13T17:23:42.639556+0000","last_became_active":"2024-09-13T17:23:41.651846+0000","last_became_peered":"2024-09-13T17:23:41.651846+0000","last_unstale":"2024-09-13T17:23:42.639556+0000","last_undegraded":"2024-09-13T17:23:42.639556+0000","last_fullsized":"2024-09-13T17:23:42.639556+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:50.550423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643289+0000","last_change":"2024-09-13T17:23:41.220880+0000","last_active":"2024-09-13T17:23:41.643289+0000","last_peered":"2024-09-13T17:23:41.643289+0000","last_clean":"2024-09-13T17:23:41.643289+0000","last_became_active":"2024-09-13T17:23:41.220592+0000","last_became_peered":"2024-09-13T17:23:41.220592+0000","last_unstale":"2024-09-13T17:23:41.643289+0000","last_undegraded":"2024-09-13T17:23:41.643289+0000","last_fullsized":"2024-09-13T17:23:41.643289+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:05.196577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658929+0000","last_change":"2024-09-13T17:23:41.655000+0000","last_active":"2024-09-13T17:23:41.658929+0000","last_peered":"2024-09-13T17:23:41.658929+0000","last_clean":"2024-09-13T17:23:41.658929+0000","last_became_active":"2024-09-13T17:23:41.654708+0000","last_became_peered":"2024-09-13T17:23:41.654708+0000","last_unstale":"2024-09-13T17:23:41.658929+0000","last_undegraded":"2024-09-13T17:23:41.658929+0000","last_fullsized":"2024-09-13T17:23:41.658929+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:13.182568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"218'3","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.663447+0000","last_change":"2024-09-13T17:23:41.211352+0000","last_active":"2024-09-13T17:23:41.663447+0000","last_peered":"2024-09-13T17:23:41.663447+0000","last_clean":"2024-09-13T17:23:41.663447+0000","last_became_active":"2024-09-13T17:23:41.211049+0000","last_became_peered":"2024-09-13T17:23:41.211049+0000","last_unstale":"2024-09-13T17:23:41.663447+0000","last_undegraded":"2024-09-13T17:23:41.663447+0000","last_fullsized":"2024-09-13T17:23:41.663447+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:51.276487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.653345+0000","last_change":"2024-09-13T17:23:41.650056+0000","last_active":"2024-09-13T17:23:41.653345+0000","last_peered":"2024-09-13T17:23:41.653345+0000","last_clean":"2024-09-13T17:23:41.653345+0000","last_became_active":"2024-09-13T17:23:41.648322+0000","last_became_peered":"2024-09-13T17:23:41.648322+0000","last_unstale":"2024-09-13T17:23:41.653345+0000","last_undegraded":"2024-09-13T17:23:41.653345+0000","last_fullsized":"2024-09-13T17:23:41.653345+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"218'6","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669533+0000","last_change":"2024-09-13T17:23:41.214296+0000","last_active":"2024-09-13T17:23:41.669533+0000","last_peered":"2024-09-13T17:23:41.669533+0000","last_clean":"2024-09-13T17:23:41.669533+0000","last_became_active":"2024-09-13T17:23:41.214046+0000","last_became_peered":"2024-09-13T17:23:41.214046+0000","last_unstale":"2024-09-13T17:23:41.669533+0000","last_undegraded":"2024-09-13T17:23:41.669533+0000","last_fullsized":"2024-09-13T17:23:41.669533+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:45.265127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.663232+0000","last_change":"2024-09-13T17:23:41.662642+0000","last_active":"2024-09-13T17:23:41.663232+0000","last_peered":"2024-09-13T17:23:41.663232+0000","last_clean":"2024-09-13T17:23:41.663232+0000","last_became_active":"2024-09-13T17:23:41.662336+0000","last_became_peered":"2024-09-13T17:23:41.662336+0000","last_unstale":"2024-09-13T17:23:41.663232+0000","last_undegraded":"2024-09-13T17:23:41.663232+0000","last_fullsized":"2024-09-13T17:23:41.663232+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:48.715837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665624+0000","last_change":"2024-09-13T17:23:41.224408+0000","last_active":"2024-09-13T17:23:41.665624+0000","last_peered":"2024-09-13T17:23:41.665624+0000","last_clean":"2024-09-13T17:23:41.665624+0000","last_became_active":"2024-09-13T17:23:41.223968+0000","last_became_peered":"2024-09-13T17:23:41.223968+0000","last_unstale":"2024-09-13T17:23:41.665624+0000","last_undegraded":"2024-09-13T17:23:41.665624+0000","last_fullsized":"2024-09-13T17:23:41.665624+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:10.543493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683152+0000","last_change":"2024-09-13T17:23:41.649950+0000","last_active":"2024-09-13T17:23:42.683152+0000","last_peered":"2024-09-13T17:23:42.683152+0000","last_clean":"2024-09-13T17:23:42.683152+0000","last_became_active":"2024-09-13T17:23:41.649645+0000","last_became_peered":"2024-09-13T17:23:41.649645+0000","last_unstale":"2024-09-13T17:23:42.683152+0000","last_undegraded":"2024-09-13T17:23:42.683152+0000","last_fullsized":"2024-09-13T17:23:42.683152+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:41.713454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643116+0000","last_change":"2024-09-13T17:23:41.218589+0000","last_active":"2024-09-13T17:23:41.643116+0000","last_peered":"2024-09-13T17:23:41.643116+0000","last_clean":"2024-09-13T17:23:41.643116+0000","last_became_active":"2024-09-13T17:23:41.218264+0000","last_became_peered":"2024-09-13T17:23:41.218264+0000","last_unstale":"2024-09-13T17:23:41.643116+0000","last_undegraded":"2024-09-13T17:23:41.643116+0000","last_fullsized":"2024-09-13T17:23:41.643116+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:59.974787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.647521+0000","last_change":"2024-09-13T17:23:41.646671+0000","last_active":"2024-09-13T17:23:41.647521+0000","last_peered":"2024-09-13T17:23:41.647521+0000","last_clean":"2024-09-13T17:23:41.647521+0000","last_became_active":"2024-09-13T17:23:41.646243+0000","last_became_peered":"2024-09-13T17:23:41.646243+0000","last_unstale":"2024-09-13T17:23:41.647521+0000","last_undegraded":"2024-09-13T17:23:41.647521+0000","last_fullsized":"2024-09-13T17:23:41.647521+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:44.988720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.640677+0000","last_change":"2024-09-13T17:23:41.224054+0000","last_active":"2024-09-13T17:23:41.640677+0000","last_peered":"2024-09-13T17:23:41.640677+0000","last_clean":"2024-09-13T17:23:41.640677+0000","last_became_active":"2024-09-13T17:23:41.223469+0000","last_became_peered":"2024-09-13T17:23:41.223469+0000","last_unstale":"2024-09-13T17:23:41.640677+0000","last_undegraded":"2024-09-13T17:23:41.640677+0000","last_fullsized":"2024-09-13T17:23:41.640677+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:29.791684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658516+0000","last_change":"2024-09-13T17:23:41.656735+0000","last_active":"2024-09-13T17:23:41.658516+0000","last_peered":"2024-09-13T17:23:41.658516+0000","last_clean":"2024-09-13T17:23:41.658516+0000","last_became_active":"2024-09-13T17:23:41.655485+0000","last_became_peered":"2024-09-13T17:23:41.655485+0000","last_unstale":"2024-09-13T17:23:41.658516+0000","last_undegraded":"2024-09-13T17:23:41.658516+0000","last_fullsized":"2024-09-13T17:23:41.658516+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:26.860796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"218'8","reported_seq":19,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.638917+0000","last_change":"2024-09-13T17:23:41.217988+0000","last_active":"2024-09-13T17:23:42.638917+0000","last_peered":"2024-09-13T17:23:42.638917+0000","last_clean":"2024-09-13T17:23:42.638917+0000","last_became_active":"2024-09-13T17:23:41.217724+0000","last_became_peered":"2024-09-13T17:23:41.217724+0000","last_unstale":"2024-09-13T17:23:42.638917+0000","last_undegraded":"2024-09-13T17:23:42.638917+0000","last_fullsized":"2024-09-13T17:23:42.638917+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:44.412141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643860+0000","last_change":"2024-09-13T17:23:41.640787+0000","last_active":"2024-09-13T17:23:41.643860+0000","last_peered":"2024-09-13T17:23:41.643860+0000","last_clean":"2024-09-13T17:23:41.643860+0000","last_became_active":"2024-09-13T17:23:41.640508+0000","last_became_peered":"2024-09-13T17:23:41.640508+0000","last_unstale":"2024-09-13T17:23:41.643860+0000","last_undegraded":"2024-09-13T17:23:41.643860+0000","last_fullsized":"2024-09-13T17:23:41.643860+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:57.496659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660410+0000","last_change":"2024-09-13T17:23:41.222159+0000","last_active":"2024-09-13T17:23:41.660410+0000","last_peered":"2024-09-13T17:23:41.660410+0000","last_clean":"2024-09-13T17:23:41.660410+0000","last_became_active":"2024-09-13T17:23:41.221910+0000","last_became_peered":"2024-09-13T17:23:41.221910+0000","last_unstale":"2024-09-13T17:23:41.660410+0000","last_undegraded":"2024-09-13T17:23:41.660410+0000","last_fullsized":"2024-09-13T17:23:41.660410+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:05.289194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658859+0000","last_change":"2024-09-13T17:23:41.654148+0000","last_active":"2024-09-13T17:23:41.658859+0000","last_peered":"2024-09-13T17:23:41.658859+0000","last_clean":"2024-09-13T17:23:41.658859+0000","last_became_active":"2024-09-13T17:23:41.653895+0000","last_became_peered":"2024-09-13T17:23:41.653895+0000","last_unstale":"2024-09-13T17:23:41.658859+0000","last_undegraded":"2024-09-13T17:23:41.658859+0000","last_fullsized":"2024-09-13T17:23:41.658859+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:13.325786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669746+0000","last_change":"2024-09-13T17:23:41.222511+0000","last_active":"2024-09-13T17:23:41.669746+0000","last_peered":"2024-09-13T17:23:41.669746+0000","last_clean":"2024-09-13T17:23:41.669746+0000","last_became_active":"2024-09-13T17:23:41.222259+0000","last_became_peered":"2024-09-13T17:23:41.222259+0000","last_unstale":"2024-09-13T17:23:41.669746+0000","last_undegraded":"2024-09-13T17:23:41.669746+0000","last_fullsized":"2024-09-13T17:23:41.669746+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:30.077962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683037+0000","last_change":"2024-09-13T17:23:41.655170+0000","last_active":"2024-09-13T17:23:42.683037+0000","last_peered":"2024-09-13T17:23:42.683037+0000","last_clean":"2024-09-13T17:23:42.683037+0000","last_became_active":"2024-09-13T17:23:41.654653+0000","last_became_peered":"2024-09-13T17:23:41.654653+0000","last_unstale":"2024-09-13T17:23:42.683037+0000","last_undegraded":"2024-09-13T17:23:42.683037+0000","last_fullsized":"2024-09-13T17:23:42.683037+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:25.253708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"218'1","reported_seq":12,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683069+0000","last_change":"2024-09-13T17:23:41.219784+0000","last_active":"2024-09-13T17:23:42.683069+0000","last_peered":"2024-09-13T17:23:42.683069+0000","last_clean":"2024-09-13T17:23:42.683069+0000","last_became_active":"2024-09-13T17:23:41.219451+0000","last_became_peered":"2024-09-13T17:23:41.219451+0000","last_unstale":"2024-09-13T17:23:42.683069+0000","last_undegraded":"2024-09-13T17:23:42.683069+0000","last_fullsized":"2024-09-13T17:23:42.683069+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:43.099661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640343+0000","last_change":"2024-09-13T17:23:41.662847+0000","last_active":"2024-09-13T17:23:42.640343+0000","last_peered":"2024-09-13T17:23:42.640343+0000","last_clean":"2024-09-13T17:23:42.640343+0000","last_became_active":"2024-09-13T17:23:41.662470+0000","last_became_peered":"2024-09-13T17:23:41.662470+0000","last_unstale":"2024-09-13T17:23:42.640343+0000","last_undegraded":"2024-09-13T17:23:42.640343+0000","last_fullsized":"2024-09-13T17:23:42.640343+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:53.634317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639283+0000","last_change":"2024-09-13T17:23:41.223642+0000","last_active":"2024-09-13T17:23:42.639283+0000","last_peered":"2024-09-13T17:23:42.639283+0000","last_clean":"2024-09-13T17:23:42.639283+0000","last_became_active":"2024-09-13T17:23:41.223363+0000","last_became_peered":"2024-09-13T17:23:41.223363+0000","last_unstale":"2024-09-13T17:23:42.639283+0000","last_undegraded":"2024-09-13T17:23:42.639283+0000","last_fullsized":"2024-09-13T17:23:42.639283+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639683+0000","last_change":"2024-09-13T17:23:41.660463+0000","last_active":"2024-09-13T17:23:42.639683+0000","last_peered":"2024-09-13T17:23:42.639683+0000","last_clean":"2024-09-13T17:23:42.639683+0000","last_became_active":"2024-09-13T17:23:41.660093+0000","last_became_peered":"2024-09-13T17:23:41.660093+0000","last_unstale":"2024-09-13T17:23:42.639683+0000","last_undegraded":"2024-09-13T17:23:42.639683+0000","last_fullsized":"2024-09-13T17:23:42.639683+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:47.353460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669219+0000","last_change":"2024-09-13T17:23:41.224936+0000","last_active":"2024-09-13T17:23:41.669219+0000","last_peered":"2024-09-13T17:23:41.669219+0000","last_clean":"2024-09-13T17:23:41.669219+0000","last_became_active":"2024-09-13T17:23:41.224357+0000","last_became_peered":"2024-09-13T17:23:41.224357+0000","last_unstale":"2024-09-13T17:23:41.669219+0000","last_undegraded":"2024-09-13T17:23:41.669219+0000","last_fullsized":"2024-09-13T17:23:41.669219+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:27.600246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660665+0000","last_change":"2024-09-13T17:23:41.650512+0000","last_active":"2024-09-13T17:23:41.660665+0000","last_peered":"2024-09-13T17:23:41.660665+0000","last_clean":"2024-09-13T17:23:41.660665+0000","last_became_active":"2024-09-13T17:23:41.650268+0000","last_became_peered":"2024-09-13T17:23:41.650268+0000","last_unstale":"2024-09-13T17:23:41.660665+0000","last_undegraded":"2024-09-13T17:23:41.660665+0000","last_fullsized":"2024-09-13T17:23:41.660665+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:29.643143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639421+0000","last_change":"2024-09-13T17:23:41.223914+0000","last_active":"2024-09-13T17:23:42.639421+0000","last_peered":"2024-09-13T17:23:42.639421+0000","last_clean":"2024-09-13T17:23:42.639421+0000","last_became_active":"2024-09-13T17:23:41.222778+0000","last_became_peered":"2024-09-13T17:23:41.222778+0000","last_unstale":"2024-09-13T17:23:42.639421+0000","last_undegraded":"2024-09-13T17:23:42.639421+0000","last_fullsized":"2024-09-13T17:23:42.639421+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:27.600246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.638857+0000","last_change":"2024-09-13T17:23:41.649907+0000","last_active":"2024-09-13T17:23:42.638857+0000","last_peered":"2024-09-13T17:23:42.638857+0000","last_clean":"2024-09-13T17:23:42.638857+0000","last_became_active":"2024-09-13T17:23:41.649643+0000","last_became_peered":"2024-09-13T17:23:41.649643+0000","last_unstale":"2024-09-13T17:23:42.638857+0000","last_undegraded":"2024-09-13T17:23:42.638857+0000","last_fullsized":"2024-09-13T17:23:42.638857+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:56:32.718664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683055+0000","last_change":"2024-09-13T17:23:41.220607+0000","last_active":"2024-09-13T17:23:42.683055+0000","last_peered":"2024-09-13T17:23:42.683055+0000","last_clean":"2024-09-13T17:23:42.683055+0000","last_became_active":"2024-09-13T17:23:41.220347+0000","last_became_peered":"2024-09-13T17:23:41.220347+0000","last_unstale":"2024-09-13T17:23:42.683055+0000","last_undegraded":"2024-09-13T17:23:42.683055+0000","last_fullsized":"2024-09-13T17:23:42.683055+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:11.371958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639513+0000","last_change":"2024-09-13T17:23:41.662762+0000","last_active":"2024-09-13T17:23:42.639513+0000","last_peered":"2024-09-13T17:23:42.639513+0000","last_clean":"2024-09-13T17:23:42.639513+0000","last_became_active":"2024-09-13T17:23:41.662478+0000","last_became_peered":"2024-09-13T17:23:41.662478+0000","last_unstale":"2024-09-13T17:23:42.639513+0000","last_undegraded":"2024-09-13T17:23:42.639513+0000","last_fullsized":"2024-09-13T17:23:42.639513+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:08.721032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.641583+0000","last_change":"2024-09-13T17:23:41.219347+0000","last_active":"2024-09-13T17:23:41.641583+0000","last_peered":"2024-09-13T17:23:41.641583+0000","last_clean":"2024-09-13T17:23:41.641583+0000","last_became_active":"2024-09-13T17:23:41.219104+0000","last_became_peered":"2024-09-13T17:23:41.219104+0000","last_unstale":"2024-09-13T17:23:41.641583+0000","last_undegraded":"2024-09-13T17:23:41.641583+0000","last_fullsized":"2024-09-13T17:23:41.641583+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:45.265127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.649757+0000","last_change":"2024-09-13T17:23:41.646887+0000","last_active":"2024-09-13T17:23:41.649757+0000","last_peered":"2024-09-13T17:23:41.649757+0000","last_clean":"2024-09-13T17:23:41.649757+0000","last_became_active":"2024-09-13T17:23:41.646571+0000","last_became_peered":"2024-09-13T17:23:41.646571+0000","last_unstale":"2024-09-13T17:23:41.649757+0000","last_undegraded":"2024-09-13T17:23:41.649757+0000","last_fullsized":"2024-09-13T17:23:41.649757+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:57.781369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.666813+0000","last_change":"2024-09-13T17:23:41.218908+0000","last_active":"2024-09-13T17:23:41.666813+0000","last_peered":"2024-09-13T17:23:41.666813+0000","last_clean":"2024-09-13T17:23:41.666813+0000","last_became_active":"2024-09-13T17:23:41.218636+0000","last_became_peered":"2024-09-13T17:23:41.218636+0000","last_unstale":"2024-09-13T17:23:41.666813+0000","last_undegraded":"2024-09-13T17:23:41.666813+0000","last_fullsized":"2024-09-13T17:23:41.666813+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:47.952641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662666+0000","last_change":"2024-09-13T17:23:41.659475+0000","last_active":"2024-09-13T17:23:41.662666+0000","last_peered":"2024-09-13T17:23:41.662666+0000","last_clean":"2024-09-13T17:23:41.662666+0000","last_became_active":"2024-09-13T17:23:41.659208+0000","last_became_peered":"2024-09-13T17:23:41.659208+0000","last_unstale":"2024-09-13T17:23:41.662666+0000","last_undegraded":"2024-09-13T17:23:41.662666+0000","last_fullsized":"2024-09-13T17:23:41.662666+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:46.760609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640492+0000","last_change":"2024-09-13T17:23:41.219933+0000","last_active":"2024-09-13T17:23:42.640492+0000","last_peered":"2024-09-13T17:23:42.640492+0000","last_clean":"2024-09-13T17:23:42.640492+0000","last_became_active":"2024-09-13T17:23:41.219593+0000","last_became_peered":"2024-09-13T17:23:41.219593+0000","last_unstale":"2024-09-13T17:23:42.640492+0000","last_undegraded":"2024-09-13T17:23:42.640492+0000","last_fullsized":"2024-09-13T17:23:42.640492+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:17.395117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639234+0000","last_change":"2024-09-13T17:23:41.656791+0000","last_active":"2024-09-13T17:23:42.639234+0000","last_peered":"2024-09-13T17:23:42.639234+0000","last_clean":"2024-09-13T17:23:42.639234+0000","last_became_active":"2024-09-13T17:23:41.656558+0000","last_became_peered":"2024-09-13T17:23:41.656558+0000","last_unstale":"2024-09-13T17:23:42.639234+0000","last_undegraded":"2024-09-13T17:23:42.639234+0000","last_fullsized":"2024-09-13T17:23:42.639234+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:16.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683154+0000","last_change":"2024-09-13T17:23:41.222323+0000","last_active":"2024-09-13T17:23:42.683154+0000","last_peered":"2024-09-13T17:23:42.683154+0000","last_clean":"2024-09-13T17:23:42.683154+0000","last_became_active":"2024-09-13T17:23:41.222026+0000","last_became_peered":"2024-09-13T17:23:41.222026+0000","last_unstale":"2024-09-13T17:23:42.683154+0000","last_undegraded":"2024-09-13T17:23:42.683154+0000","last_fullsized":"2024-09-13T17:23:42.683154+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:05.289194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639486+0000","last_change":"2024-09-13T17:23:41.653884+0000","last_active":"2024-09-13T17:23:42.639486+0000","last_peered":"2024-09-13T17:23:42.639486+0000","last_clean":"2024-09-13T17:23:42.639486+0000","last_became_active":"2024-09-13T17:23:41.653601+0000","last_became_peered":"2024-09-13T17:23:41.653601+0000","last_unstale":"2024-09-13T17:23:42.639486+0000","last_undegraded":"2024-09-13T17:23:42.639486+0000","last_fullsized":"2024-09-13T17:23:42.639486+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:20.279302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639332+0000","last_change":"2024-09-13T17:23:41.223547+0000","last_active":"2024-09-13T17:23:42.639332+0000","last_peered":"2024-09-13T17:23:42.639332+0000","last_clean":"2024-09-13T17:23:42.639332+0000","last_became_active":"2024-09-13T17:23:41.222840+0000","last_became_peered":"2024-09-13T17:23:41.222840+0000","last_unstale":"2024-09-13T17:23:42.639332+0000","last_undegraded":"2024-09-13T17:23:42.639332+0000","last_fullsized":"2024-09-13T17:23:42.639332+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:35.121129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.653375+0000","last_change":"2024-09-13T17:23:41.651660+0000","last_active":"2024-09-13T17:23:41.653375+0000","last_peered":"2024-09-13T17:23:41.653375+0000","last_clean":"2024-09-13T17:23:41.653375+0000","last_became_active":"2024-09-13T17:23:41.651181+0000","last_became_peered":"2024-09-13T17:23:41.651181+0000","last_unstale":"2024-09-13T17:23:41.653375+0000","last_undegraded":"2024-09-13T17:23:41.653375+0000","last_fullsized":"2024-09-13T17:23:41.653375+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:50.579369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.641651+0000","last_change":"2024-09-13T17:23:41.222904+0000","last_active":"2024-09-13T17:23:41.641651+0000","last_peered":"2024-09-13T17:23:41.641651+0000","last_clean":"2024-09-13T17:23:41.641651+0000","last_became_active":"2024-09-13T17:23:41.222632+0000","last_became_peered":"2024-09-13T17:23:41.222632+0000","last_unstale":"2024-09-13T17:23:41.641651+0000","last_undegraded":"2024-09-13T17:23:41.641651+0000","last_fullsized":"2024-09-13T17:23:41.641651+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:55.070253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639774+0000","last_change":"2024-09-13T17:23:41.658707+0000","last_active":"2024-09-13T17:23:42.639774+0000","last_peered":"2024-09-13T17:23:42.639774+0000","last_clean":"2024-09-13T17:23:42.639774+0000","last_became_active":"2024-09-13T17:23:41.658166+0000","last_became_peered":"2024-09-13T17:23:41.658166+0000","last_unstale":"2024-09-13T17:23:42.639774+0000","last_undegraded":"2024-09-13T17:23:42.639774+0000","last_fullsized":"2024-09-13T17:23:42.639774+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.819173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.633506+0000","last_change":"2024-09-13T17:23:41.212544+0000","last_active":"2024-09-13T17:23:41.633506+0000","last_peered":"2024-09-13T17:23:41.633506+0000","last_clean":"2024-09-13T17:23:41.633506+0000","last_became_active":"2024-09-13T17:23:41.212034+0000","last_became_peered":"2024-09-13T17:23:41.212034+0000","last_unstale":"2024-09-13T17:23:41.633506+0000","last_undegraded":"2024-09-13T17:23:41.633506+0000","last_fullsized":"2024-09-13T17:23:41.633506+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:54.522241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639533+0000","last_change":"2024-09-13T17:23:41.650353+0000","last_active":"2024-09-13T17:23:42.639533+0000","last_peered":"2024-09-13T17:23:42.639533+0000","last_clean":"2024-09-13T17:23:42.639533+0000","last_became_active":"2024-09-13T17:23:41.650048+0000","last_became_peered":"2024-09-13T17:23:41.650048+0000","last_unstale":"2024-09-13T17:23:42.639533+0000","last_undegraded":"2024-09-13T17:23:42.639533+0000","last_fullsized":"2024-09-13T17:23:42.639533+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:25.022757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.642941+0000","last_change":"2024-09-13T17:23:41.220622+0000","last_active":"2024-09-13T17:23:41.642941+0000","last_peered":"2024-09-13T17:23:41.642941+0000","last_clean":"2024-09-13T17:23:41.642941+0000","last_became_active":"2024-09-13T17:23:41.220299+0000","last_became_peered":"2024-09-13T17:23:41.220299+0000","last_unstale":"2024-09-13T17:23:41.642941+0000","last_undegraded":"2024-09-13T17:23:41.642941+0000","last_fullsized":"2024-09-13T17:23:41.642941+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:44.412141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656755+0000","last_change":"2024-09-13T17:23:41.648897+0000","last_active":"2024-09-13T17:23:41.656755+0000","last_peered":"2024-09-13T17:23:41.656755+0000","last_clean":"2024-09-13T17:23:41.656755+0000","last_became_active":"2024-09-13T17:23:41.648189+0000","last_became_peered":"2024-09-13T17:23:41.648189+0000","last_unstale":"2024-09-13T17:23:41.656755+0000","last_undegraded":"2024-09-13T17:23:41.656755+0000","last_fullsized":"2024-09-13T17:23:41.656755+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:25.253708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643149+0000","last_change":"2024-09-13T17:23:41.226165+0000","last_active":"2024-09-13T17:23:41.643149+0000","last_peered":"2024-09-13T17:23:41.643149+0000","last_clean":"2024-09-13T17:23:41.643149+0000","last_became_active":"2024-09-13T17:23:41.225875+0000","last_became_peered":"2024-09-13T17:23:41.225875+0000","last_unstale":"2024-09-13T17:23:41.643149+0000","last_undegraded":"2024-09-13T17:23:41.643149+0000","last_fullsized":"2024-09-13T17:23:41.643149+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:40.611462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640319+0000","last_change":"2024-09-13T17:23:41.651563+0000","last_active":"2024-09-13T17:23:42.640319+0000","last_peered":"2024-09-13T17:23:42.640319+0000","last_clean":"2024-09-13T17:23:42.640319+0000","last_became_active":"2024-09-13T17:23:41.651247+0000","last_became_peered":"2024-09-13T17:23:41.651247+0000","last_unstale":"2024-09-13T17:23:42.640319+0000","last_undegraded":"2024-09-13T17:23:42.640319+0000","last_fullsized":"2024-09-13T17:23:42.640319+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:48.396974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"218'4","reported_seq":14,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.668347+0000","last_change":"2024-09-13T17:23:41.226322+0000","last_active":"2024-09-13T17:23:41.668347+0000","last_peered":"2024-09-13T17:23:41.668347+0000","last_clean":"2024-09-13T17:23:41.668347+0000","last_became_active":"2024-09-13T17:23:41.226067+0000","last_became_peered":"2024-09-13T17:23:41.226067+0000","last_unstale":"2024-09-13T17:23:41.668347+0000","last_undegraded":"2024-09-13T17:23:41.668347+0000","last_fullsized":"2024-09-13T17:23:41.668347+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":333,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.639126+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:23:41.639126+0000","last_peered":"2024-09-13T17:23:41.639126+0000","last_clean":"2024-09-13T17:23:41.639126+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:23:41.639126+0000","last_undegraded":"2024-09-13T17:23:41.639126+0000","last_fullsized":"2024-09-13T17:23:41.639126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575040,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70432,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301408,"statfs":{"total":96636764160,"available":96564641792,"internally_reserved":0,"allocated":3362816,"data_stored":1813525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.92800000000000005,"5min":1.008,"15min":2.0880000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.316},"max":{"1min":0.97599999999999998,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.55600000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96899999999999997,"5min":1.161,"15min":2.5030000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.63200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86799999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":1.105,"15min":1.544},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69099999999999995,"15min":0.69699999999999995},"min":{"1min":0.45500000000000002,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":1.0269999999999999,"15min":1.605},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.443,"15min":0.34999999999999998},"max":{"1min":0.93799999999999994,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.88800000000000001,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.90300000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.629,"15min":0.65000000000000002},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":1.2170000000000001,"15min":1.234},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.47599999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.92200000000000004,"5min":1.135,"15min":1.143},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.60499999999999998,"5min":0.47999999999999998,"15min":0.42599999999999999},"max":{"1min":0.90500000000000003,"5min":1.915,"15min":1.915},"last":0.86299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.39300000000000002,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.55500000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":1.607,"15min":1.607},"last":0.94999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65756,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306084,"statfs":{"total":96636764160,"available":96569430016,"internally_reserved":0,"allocated":2760704,"data_stored":1217391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":3.6629999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72499999999999998,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.92400000000000004,"5min":0.97399999999999998,"15min":5.9800000000000004},"last":0.63400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.874,"5min":0.998,"15min":3.7040000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.88500000000000001,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.68400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.88,"5min":0.88,"15min":5.9240000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.2629999999999999},"last":0.72899999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.53800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":3.6520000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.70599999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.006,"5min":1.262,"15min":1.6439999999999999},"last":0.439},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.626,"15min":0.64200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":4.7750000000000004},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.68300000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":1.0129999999999999,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":1.089,"5min":1.1559999999999999,"15min":3.609},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70199999999999996,"15min":0.72399999999999998},"min":{"1min":0.504,"5min":0.379,"15min":0.378},"max":{"1min":1.163,"5min":1.163,"15min":4.9669999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":1.087,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.66000000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":2764800,"data_stored":1223087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.76100000000000001},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86399999999999999,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.76200000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.871,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":1.482,"15min":7.7370000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.73499999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.67000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":1.0780000000000001},"min":{"1min":0.56100000000000005,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.042,"5min":1.042,"15min":113.363},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.73799999999999999},"min":{"1min":0.63400000000000001,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":1.113,"15min":4.1180000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":1.03,"15min":2.827},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.48199999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.0609999999999999,"5min":1.73,"15min":2.2320000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":1.532,"15min":4.1500000000000004},"last":0.73199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.59899999999999998,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.169,"5min":1.363,"15min":2.02},"last":0.78100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575039,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73968,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297872,"statfs":{"total":96636764160,"available":96561020928,"internally_reserved":0,"allocated":2785280,"data_stored":1224743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.2150000000000001,"15min":0.84099999999999997},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.2010000000000001,"15min":0.875},"min":{"1min":0.46600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93200000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.85299999999999998,"15min":0.72199999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96799999999999997,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.753,"15min":0.68200000000000005},"min":{"1min":0.53800000000000003,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.97599999999999998,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71799999999999997,"15min":0.67900000000000005},"min":{"1min":0.49299999999999999,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.996,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91500000000000004,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.109,"15min":0.82599999999999996},"min":{"1min":0.61799999999999999,"5min":0.437,"15min":0.378},"max":{"1min":0.95499999999999996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.72599999999999998},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.92600000000000005,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93799999999999994,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.496,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.88200000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.755}]},{"osd":6,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":0.92700000000000005,"5min":2.976,"15min":2.976},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.73399999999999999},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96399999999999997,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.85199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54400000000000004,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.872},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.749},"min":{"1min":0.55400000000000005,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.95099999999999996,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.83399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575040,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2756608,"data_stored":1217369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":0.624,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.373},{"interface":"front","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":2.96},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.503},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":3.5600000000000001},"last":0.495},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.379,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.38900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.875,"15min":1.3819999999999999},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.1479999999999999,"5min":1.21,"15min":2.8610000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.48199999999999998},"max":{"1min":1.0860000000000001,"5min":1.23,"15min":1.3640000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.071,"5min":1.323,"15min":1.391},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.499,"15min":0.442},"max":{"1min":1.05,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.78600000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77800000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.1559999999999999,"5min":1.218,"15min":1.401},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81699999999999995,"15min":0.78600000000000003},"min":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.175,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78900000000000003},"min":{"1min":0.624,"5min":0.503,"15min":0.42699999999999999},"max":{"1min":1.2130000000000001,"5min":1.27,"15min":2.899},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.208,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.66000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47764,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324076,"statfs":{"total":96636764160,"available":96587853824,"internally_reserved":0,"allocated":2752512,"data_stored":1215858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47599999999999998,"15min":0.436},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.309},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48499999999999999,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69599999999999995,"5min":1.593,"15min":30.023},"last":0.40000000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":1.371},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.67000000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":1.4690000000000001},"min":{"1min":0.38500000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74299999999999999,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.53000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.371},"max":{"1min":1.264,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79300000000000004,"15min":0.878},"min":{"1min":0.52300000000000002,"5min":0.48299999999999998,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82199999999999995,"15min":0.78400000000000003},"min":{"1min":0.61399999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1930000000000001,"5min":6.4800000000000004,"15min":10.664},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81499999999999995,"15min":0.89100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.23,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.872}]},{"osd":7,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1299999999999999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82999999999999996,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.93400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92004,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94279836,"statfs":{"total":96636764160,"available":96542552064,"internally_reserved":0,"allocated":2772992,"data_stored":1218475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.441,"15min":0.46300000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.63300000000000001,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.52800000000000002,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.74199999999999999,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.28,"15min":6.1630000000000003},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.23,"15min":1.78},"last":0.81799999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59599999999999997,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":0.93000000000000005,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75800000000000001,"15min":0.78200000000000003},"min":{"1min":0.57299999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99299999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.873},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78300000000000003,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0489999999999999,"5min":1.246,"15min":6.1970000000000001},"last":0.83999999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.59999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":0.81200000000000006},"min":{"1min":0.61299999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.02,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.79000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74520,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297320,"statfs":{"total":96636764160,"available":96560455680,"internally_reserved":0,"allocated":3346432,"data_stored":1806170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":1.986,"15min":5.5259999999999998},"last":0.318},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.69999999999999996,"5min":2.371,"15min":5.4889999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.501,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.434}]},{"osd":3,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.878,"15min":4.7549999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.498}]},{"osd":4,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80300000000000005,"15min":0.79100000000000004},"min":{"1min":0.629,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.119,"5min":3.8849999999999998,"15min":4.734},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.70899999999999996,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":4.8079999999999998},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.66500000000000004,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":1.141,"5min":1.141,"15min":4.944},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.78800000000000003,"15min":0.79800000000000004},"min":{"1min":0.65100000000000002,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.246,"5min":1.246,"15min":4.9710000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.78400000000000003,"15min":0.80800000000000005},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":4.8470000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.65100000000000002,"5min":0.497,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85599999999999998,"15min":0.83399999999999996},"min":{"1min":0.68799999999999994,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.2150000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.67400000000000004,"5min":0.56899999999999995,"15min":0.505},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.9950000000000001},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:23:45.830 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:23:46.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.018+0000 7fa0ebe00640 1 Processor -- start 2024-09-13T17:23:46.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.019+0000 7fa0ebe00640 1 -- start start 2024-09-13T17:23:46.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0ebe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0b0b50 0x7fa0ec0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0ebe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0c3850 con 0x7fa0ec159820 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0ebe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec15a930 con 0x7fa0ec159440 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0ebe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ec159d20 con 0x7fa0ec0b0b50 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0eb400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0ec159820 0x7fa0ec15adc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49326/0 (socket says 172.21.15.28:49326) 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0eb400640 1 -- 172.21.15.28:0/3391227211 learned_addr learned my addr 172.21.15.28:0/3391227211 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0e2000640 1 --2- 172.21.15.28:0/3391227211 >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0b0b50 0x7fa0ec0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.020+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 759913538 0 0) 0x7fa0ec0c3850 con 0x7fa0ec159820 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d0003520 con 0x7fa0ec159820 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423061345 0 0) 0x7fa0d0003520 con 0x7fa0ec159820 2024-09-13T17:23:46.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0b0b50 msgr2=0x7fa0ec0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 --2- 172.21.15.28:0/3391227211 >> v2:172.21.15.189:3300/0 conn(0x7fa0ec0b0b50 0x7fa0ec0b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 >> v1:172.21.15.189:6789/0 conn(0x7fa0ec159440 legacy=0x7fa0ec0b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec15a000 con 0x7fa0ec159820 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa0e40019f0 con 0x7fa0ec159820 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 >> v1:172.21.15.28:6789/0 conn(0x7fa0ec159820 legacy=0x7fa0ec15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d00047c0 0x7fa0d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d0005180 0x7fa0d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec159d20 con 0x7fa0d0005180 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000c030 con 0x7fa0d00047c0 2024-09-13T17:23:46.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d000c1f0 con 0x7fa0d0003970 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0eb400640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d0005180 0x7fa0d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.021+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0e2000640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d00047c0 0x7fa0d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eaa00640 1 -- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d00047c0 msgr2=0x7fa0d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0d00047c0 0x7fa0d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eaa00640 1 -- 172.21.15.28:0/3391227211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d0005180 msgr2=0x7fa0d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0d0005180 0x7fa0d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eaa00640 1 -- 172.21.15.28:0/3391227211 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d0003520 con 0x7fa0d0003970 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.022+0000 7fa0eb400640 1 -- 172.21.15.28:0/3391227211 reap_dead start 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.023+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0d0003d90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc001750 tx=0x7fa0d00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=338458671d7efaea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.023+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0052d0 con 0x7fa0d0003970 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc005430 con 0x7fa0d0003970 2024-09-13T17:23:46.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ea000640 1 -- 172.21.15.28:0/3391227211 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc02abc0 con 0x7fa0d0003970 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 -- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 msgr2=0x7fa0d0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0d0003d90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc001750 tx=0x7fa0d00057f0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 -- 172.21.15.28:0/3391227211 shutdown_connections 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/3391227211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0d0003d90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 -- 172.21.15.28:0/3391227211 >> 172.21.15.28:0/3391227211 conn(0x7fa0ec072e40 msgr2=0x7fa0ec073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 -- 172.21.15.28:0/3391227211 shutdown_connections 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.024+0000 7fa0ebe00640 1 -- 172.21.15.28:0/3391227211 wait complete. 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.026+0000 7fa0ebe00640 1 Processor -- start 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.026+0000 7fa0ebe00640 1 -- start start 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0ec149090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 0x7fa0ec149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec156530 con 0x7fa0ec156950 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec1495d0 con 0x7fa0ec1586e0 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0ebe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec158b60 con 0x7fa0d0003970 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0e2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35170/0 (socket says 172.21.15.28:35170) 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0e2000640 1 -- 172.21.15.28:0/1917720695 learned_addr learned my addr 172.21.15.28:0/1917720695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0ec149090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.027+0000 7fa0eb400640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 0x7fa0ec149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 -- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 msgr2=0x7fa0ec149090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0ec149090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 -- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 msgr2=0x7fa0ec149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 0x7fa0ec149840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 -- 172.21.15.28:0/1917720695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec1558b0 con 0x7fa0ec156950 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0eb400640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 0x7fa0ec149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e2000640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4011890 tx=0x7fa0d4011d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d4004240 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d40043a0 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d400fa20 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec155b90 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.028+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec156150 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.030+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec156670 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.031+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa0d400fb80 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.031+0000 7fa0e3400640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 0x7fa0c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.032+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0d4089820 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.032+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 0x7fa0c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.033+0000 7fa0eaa00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 0x7fa0c806d0c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc032000 tx=0x7fa0dc02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.036+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0ec156670 con 0x7fa0ec156950 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.214+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0ec15a5c0 con 0x7fa0c806ac00 2024-09-13T17:23:46.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.222+0000 7fa0e3400640 1 -- 172.21.15.28:0/1917720695 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289528 (secure 0 0 0) 0x7fa0dc007040 con 0x7fa0c806ac00 2024-09-13T17:23:46.595 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:46.596 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:23:46.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 msgr2=0x7fa0c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 0x7fa0c806d0c0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc032000 tx=0x7fa0dc02e000 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 msgr2=0x7fa0ec1581a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa0d4011890 tx=0x7fa0d4011d50 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 shutdown_connections 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa0c806ac00 0x7fa0c806d0c0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa0ec1586e0 0x7fa0ec149840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0ec156950 0x7fa0ec1581a0 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 --2- 172.21.15.28:0/1917720695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa0d0003970 0x7fa0ec149090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 >> 172.21.15.28:0/1917720695 conn(0x7fa0ec072e40 msgr2=0x7fa0ec06c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 shutdown_connections 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.227+0000 7fa0ebe00640 1 -- 172.21.15.28:0/1917720695 wait complete. 2024-09-13T17:23:46.599 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2024-09-13T17:23:45.122078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":136,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":558448,"kb_used_data":22756,"kb_used_omap":12,"kb_used_meta":535539,"kb_avail":754416272,"statfs":{"total":773094113280,"available":772522262528,"internally_reserved":0,"allocated":23302144,"data_stored":10936618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":548392505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.925949"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660902+0000","last_change":"2024-09-13T17:23:41.651300+0000","last_active":"2024-09-13T17:23:41.660902+0000","last_peered":"2024-09-13T17:23:41.660902+0000","last_clean":"2024-09-13T17:23:41.660902+0000","last_became_active":"2024-09-13T17:23:41.651060+0000","last_became_peered":"2024-09-13T17:23:41.651060+0000","last_unstale":"2024-09-13T17:23:41.660902+0000","last_undegraded":"2024-09-13T17:23:41.660902+0000","last_fullsized":"2024-09-13T17:23:41.660902+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:26.860796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660733+0000","last_change":"2024-09-13T17:23:41.653072+0000","last_active":"2024-09-13T17:23:41.660733+0000","last_peered":"2024-09-13T17:23:41.660733+0000","last_clean":"2024-09-13T17:23:41.660733+0000","last_became_active":"2024-09-13T17:23:41.652481+0000","last_became_peered":"2024-09-13T17:23:41.652481+0000","last_unstale":"2024-09-13T17:23:41.660733+0000","last_undegraded":"2024-09-13T17:23:41.660733+0000","last_fullsized":"2024-09-13T17:23:41.660733+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:35.831865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658919+0000","last_change":"2024-09-13T17:23:41.652700+0000","last_active":"2024-09-13T17:23:41.658919+0000","last_peered":"2024-09-13T17:23:41.658919+0000","last_clean":"2024-09-13T17:23:41.658919+0000","last_became_active":"2024-09-13T17:23:41.652355+0000","last_became_peered":"2024-09-13T17:23:41.652355+0000","last_unstale":"2024-09-13T17:23:41.658919+0000","last_undegraded":"2024-09-13T17:23:41.658919+0000","last_fullsized":"2024-09-13T17:23:41.658919+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:16.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.646223+0000","last_change":"2024-09-13T17:23:41.645536+0000","last_active":"2024-09-13T17:23:41.646223+0000","last_peered":"2024-09-13T17:23:41.646223+0000","last_clean":"2024-09-13T17:23:41.646223+0000","last_became_active":"2024-09-13T17:23:41.643906+0000","last_became_peered":"2024-09-13T17:23:41.643906+0000","last_unstale":"2024-09-13T17:23:41.646223+0000","last_undegraded":"2024-09-13T17:23:41.646223+0000","last_fullsized":"2024-09-13T17:23:41.646223+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:54.502951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639827+0000","last_change":"2024-09-13T17:23:41.643574+0000","last_active":"2024-09-13T17:23:42.639827+0000","last_peered":"2024-09-13T17:23:42.639827+0000","last_clean":"2024-09-13T17:23:42.639827+0000","last_became_active":"2024-09-13T17:23:41.643298+0000","last_became_peered":"2024-09-13T17:23:41.643298+0000","last_unstale":"2024-09-13T17:23:42.639827+0000","last_undegraded":"2024-09-13T17:23:42.639827+0000","last_fullsized":"2024-09-13T17:23:42.639827+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:39.164404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662443+0000","last_change":"2024-09-13T17:23:41.653886+0000","last_active":"2024-09-13T17:23:41.662443+0000","last_peered":"2024-09-13T17:23:41.662443+0000","last_clean":"2024-09-13T17:23:41.662443+0000","last_became_active":"2024-09-13T17:23:41.653639+0000","last_became_peered":"2024-09-13T17:23:41.653639+0000","last_unstale":"2024-09-13T17:23:41.662443+0000","last_undegraded":"2024-09-13T17:23:41.662443+0000","last_fullsized":"2024-09-13T17:23:41.662443+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:55:23.368841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665098+0000","last_change":"2024-09-13T17:23:41.664121+0000","last_active":"2024-09-13T17:23:41.665098+0000","last_peered":"2024-09-13T17:23:41.665098+0000","last_clean":"2024-09-13T17:23:41.665098+0000","last_became_active":"2024-09-13T17:23:41.663791+0000","last_became_peered":"2024-09-13T17:23:41.663791+0000","last_unstale":"2024-09-13T17:23:41.665098+0000","last_undegraded":"2024-09-13T17:23:41.665098+0000","last_fullsized":"2024-09-13T17:23:41.665098+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:46.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639414+0000","last_change":"2024-09-13T17:23:41.650518+0000","last_active":"2024-09-13T17:23:42.639414+0000","last_peered":"2024-09-13T17:23:42.639414+0000","last_clean":"2024-09-13T17:23:42.639414+0000","last_became_active":"2024-09-13T17:23:41.650185+0000","last_became_peered":"2024-09-13T17:23:41.650185+0000","last_unstale":"2024-09-13T17:23:42.639414+0000","last_undegraded":"2024-09-13T17:23:42.639414+0000","last_fullsized":"2024-09-13T17:23:42.639414+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:31:38.751185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683081+0000","last_change":"2024-09-13T17:23:41.664949+0000","last_active":"2024-09-13T17:23:42.683081+0000","last_peered":"2024-09-13T17:23:42.683081+0000","last_clean":"2024-09-13T17:23:42.683081+0000","last_became_active":"2024-09-13T17:23:41.664644+0000","last_became_peered":"2024-09-13T17:23:41.664644+0000","last_unstale":"2024-09-13T17:23:42.683081+0000","last_undegraded":"2024-09-13T17:23:42.683081+0000","last_fullsized":"2024-09-13T17:23:42.683081+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:15.211955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.655744+0000","last_change":"2024-09-13T17:23:41.654505+0000","last_active":"2024-09-13T17:23:41.655744+0000","last_peered":"2024-09-13T17:23:41.655744+0000","last_clean":"2024-09-13T17:23:41.655744+0000","last_became_active":"2024-09-13T17:23:41.654170+0000","last_became_peered":"2024-09-13T17:23:41.654170+0000","last_unstale":"2024-09-13T17:23:41.655744+0000","last_undegraded":"2024-09-13T17:23:41.655744+0000","last_fullsized":"2024-09-13T17:23:41.655744+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:35.831865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639388+0000","last_change":"2024-09-13T17:23:41.662856+0000","last_active":"2024-09-13T17:23:42.639388+0000","last_peered":"2024-09-13T17:23:42.639388+0000","last_clean":"2024-09-13T17:23:42.639388+0000","last_became_active":"2024-09-13T17:23:41.662578+0000","last_became_peered":"2024-09-13T17:23:41.662578+0000","last_unstale":"2024-09-13T17:23:42.639388+0000","last_undegraded":"2024-09-13T17:23:42.639388+0000","last_fullsized":"2024-09-13T17:23:42.639388+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:31.029743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640090+0000","last_change":"2024-09-13T17:23:41.660452+0000","last_active":"2024-09-13T17:23:42.640090+0000","last_peered":"2024-09-13T17:23:42.640090+0000","last_clean":"2024-09-13T17:23:42.640090+0000","last_became_active":"2024-09-13T17:23:41.660115+0000","last_became_peered":"2024-09-13T17:23:41.660115+0000","last_unstale":"2024-09-13T17:23:42.640090+0000","last_undegraded":"2024-09-13T17:23:42.640090+0000","last_fullsized":"2024-09-13T17:23:42.640090+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:31:30.719669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658526+0000","last_change":"2024-09-13T17:23:41.656591+0000","last_active":"2024-09-13T17:23:41.658526+0000","last_peered":"2024-09-13T17:23:41.658526+0000","last_clean":"2024-09-13T17:23:41.658526+0000","last_became_active":"2024-09-13T17:23:41.656245+0000","last_became_peered":"2024-09-13T17:23:41.656245+0000","last_unstale":"2024-09-13T17:23:41.658526+0000","last_undegraded":"2024-09-13T17:23:41.658526+0000","last_fullsized":"2024-09-13T17:23:41.658526+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:29.643143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665141+0000","last_change":"2024-09-13T17:23:41.661067+0000","last_active":"2024-09-13T17:23:41.665141+0000","last_peered":"2024-09-13T17:23:41.665141+0000","last_clean":"2024-09-13T17:23:41.665141+0000","last_became_active":"2024-09-13T17:23:41.660248+0000","last_became_peered":"2024-09-13T17:23:41.660248+0000","last_unstale":"2024-09-13T17:23:41.665141+0000","last_undegraded":"2024-09-13T17:23:41.665141+0000","last_fullsized":"2024-09-13T17:23:41.665141+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:52.341148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.682914+0000","last_change":"2024-09-13T17:23:41.655686+0000","last_active":"2024-09-13T17:23:42.682914+0000","last_peered":"2024-09-13T17:23:42.682914+0000","last_clean":"2024-09-13T17:23:42.682914+0000","last_became_active":"2024-09-13T17:23:41.655246+0000","last_became_peered":"2024-09-13T17:23:41.655246+0000","last_unstale":"2024-09-13T17:23:42.682914+0000","last_undegraded":"2024-09-13T17:23:42.682914+0000","last_fullsized":"2024-09-13T17:23:42.682914+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660950+0000","last_change":"2024-09-13T17:23:41.649591+0000","last_active":"2024-09-13T17:23:41.660950+0000","last_peered":"2024-09-13T17:23:41.660950+0000","last_clean":"2024-09-13T17:23:41.660950+0000","last_became_active":"2024-09-13T17:23:41.648957+0000","last_became_peered":"2024-09-13T17:23:41.648957+0000","last_unstale":"2024-09-13T17:23:41.660950+0000","last_undegraded":"2024-09-13T17:23:41.660950+0000","last_fullsized":"2024-09-13T17:23:41.660950+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:56.708929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640115+0000","last_change":"2024-09-13T17:23:41.663023+0000","last_active":"2024-09-13T17:23:42.640115+0000","last_peered":"2024-09-13T17:23:42.640115+0000","last_clean":"2024-09-13T17:23:42.640115+0000","last_became_active":"2024-09-13T17:23:41.662618+0000","last_became_peered":"2024-09-13T17:23:41.662618+0000","last_unstale":"2024-09-13T17:23:42.640115+0000","last_undegraded":"2024-09-13T17:23:42.640115+0000","last_fullsized":"2024-09-13T17:23:42.640115+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:33.823085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660696+0000","last_change":"2024-09-13T17:23:41.652724+0000","last_active":"2024-09-13T17:23:41.660696+0000","last_peered":"2024-09-13T17:23:41.660696+0000","last_clean":"2024-09-13T17:23:41.660696+0000","last_became_active":"2024-09-13T17:23:41.651872+0000","last_became_peered":"2024-09-13T17:23:41.651872+0000","last_unstale":"2024-09-13T17:23:41.660696+0000","last_undegraded":"2024-09-13T17:23:41.660696+0000","last_fullsized":"2024-09-13T17:23:41.660696+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:57.781369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640125+0000","last_change":"2024-09-13T17:23:41.665294+0000","last_active":"2024-09-13T17:23:42.640125+0000","last_peered":"2024-09-13T17:23:42.640125+0000","last_clean":"2024-09-13T17:23:42.640125+0000","last_became_active":"2024-09-13T17:23:41.664893+0000","last_became_peered":"2024-09-13T17:23:41.664893+0000","last_unstale":"2024-09-13T17:23:42.640125+0000","last_undegraded":"2024-09-13T17:23:42.640125+0000","last_fullsized":"2024-09-13T17:23:42.640125+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:49:38.177856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656455+0000","last_change":"2024-09-13T17:23:41.648334+0000","last_active":"2024-09-13T17:23:41.656455+0000","last_peered":"2024-09-13T17:23:41.656455+0000","last_clean":"2024-09-13T17:23:41.656455+0000","last_became_active":"2024-09-13T17:23:41.648088+0000","last_became_peered":"2024-09-13T17:23:41.648088+0000","last_unstale":"2024-09-13T17:23:41.656455+0000","last_undegraded":"2024-09-13T17:23:41.656455+0000","last_fullsized":"2024-09-13T17:23:41.656455+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:23.657853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639684+0000","last_change":"2024-09-13T17:23:41.648748+0000","last_active":"2024-09-13T17:23:42.639684+0000","last_peered":"2024-09-13T17:23:42.639684+0000","last_clean":"2024-09-13T17:23:42.639684+0000","last_became_active":"2024-09-13T17:23:41.648349+0000","last_became_peered":"2024-09-13T17:23:41.648349+0000","last_unstale":"2024-09-13T17:23:42.639684+0000","last_undegraded":"2024-09-13T17:23:42.639684+0000","last_fullsized":"2024-09-13T17:23:42.639684+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:54.836377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661582+0000","last_change":"2024-09-13T17:23:41.660829+0000","last_active":"2024-09-13T17:23:41.661582+0000","last_peered":"2024-09-13T17:23:41.661582+0000","last_clean":"2024-09-13T17:23:41.661582+0000","last_became_active":"2024-09-13T17:23:41.660392+0000","last_became_peered":"2024-09-13T17:23:41.660392+0000","last_unstale":"2024-09-13T17:23:41.661582+0000","last_undegraded":"2024-09-13T17:23:41.661582+0000","last_fullsized":"2024-09-13T17:23:41.661582+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:46.760609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639498+0000","last_change":"2024-09-13T17:23:41.659060+0000","last_active":"2024-09-13T17:23:42.639498+0000","last_peered":"2024-09-13T17:23:42.639498+0000","last_clean":"2024-09-13T17:23:42.639498+0000","last_became_active":"2024-09-13T17:23:41.658709+0000","last_became_peered":"2024-09-13T17:23:41.658709+0000","last_unstale":"2024-09-13T17:23:42.639498+0000","last_undegraded":"2024-09-13T17:23:42.639498+0000","last_fullsized":"2024-09-13T17:23:42.639498+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:34:45.955447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660504+0000","last_change":"2024-09-13T17:23:41.654993+0000","last_active":"2024-09-13T17:23:41.660504+0000","last_peered":"2024-09-13T17:23:41.660504+0000","last_clean":"2024-09-13T17:23:41.660504+0000","last_became_active":"2024-09-13T17:23:41.654723+0000","last_became_peered":"2024-09-13T17:23:41.654723+0000","last_unstale":"2024-09-13T17:23:41.660504+0000","last_undegraded":"2024-09-13T17:23:41.660504+0000","last_fullsized":"2024-09-13T17:23:41.660504+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:54.666406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.646339+0000","last_change":"2024-09-13T17:23:41.642871+0000","last_active":"2024-09-13T17:23:41.646339+0000","last_peered":"2024-09-13T17:23:41.646339+0000","last_clean":"2024-09-13T17:23:41.646339+0000","last_became_active":"2024-09-13T17:23:41.642627+0000","last_became_peered":"2024-09-13T17:23:41.642627+0000","last_unstale":"2024-09-13T17:23:41.646339+0000","last_undegraded":"2024-09-13T17:23:41.646339+0000","last_fullsized":"2024-09-13T17:23:41.646339+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:14:12.989466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.657642+0000","last_change":"2024-09-13T17:23:41.655302+0000","last_active":"2024-09-13T17:23:41.657642+0000","last_peered":"2024-09-13T17:23:41.657642+0000","last_clean":"2024-09-13T17:23:41.657642+0000","last_became_active":"2024-09-13T17:23:41.654824+0000","last_became_peered":"2024-09-13T17:23:41.654824+0000","last_unstale":"2024-09-13T17:23:41.657642+0000","last_undegraded":"2024-09-13T17:23:41.657642+0000","last_fullsized":"2024-09-13T17:23:41.657642+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:45:52.341148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639419+0000","last_change":"2024-09-13T17:23:41.660753+0000","last_active":"2024-09-13T17:23:42.639419+0000","last_peered":"2024-09-13T17:23:42.639419+0000","last_clean":"2024-09-13T17:23:42.639419+0000","last_became_active":"2024-09-13T17:23:41.660451+0000","last_became_peered":"2024-09-13T17:23:41.660451+0000","last_unstale":"2024-09-13T17:23:42.639419+0000","last_undegraded":"2024-09-13T17:23:42.639419+0000","last_fullsized":"2024-09-13T17:23:42.639419+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:47.304458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661737+0000","last_change":"2024-09-13T17:23:41.657866+0000","last_active":"2024-09-13T17:23:41.661737+0000","last_peered":"2024-09-13T17:23:41.661737+0000","last_clean":"2024-09-13T17:23:41.661737+0000","last_became_active":"2024-09-13T17:23:41.657509+0000","last_became_peered":"2024-09-13T17:23:41.657509+0000","last_unstale":"2024-09-13T17:23:41.661737+0000","last_undegraded":"2024-09-13T17:23:41.661737+0000","last_fullsized":"2024-09-13T17:23:41.661737+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639257+0000","last_change":"2024-09-13T17:23:41.658664+0000","last_active":"2024-09-13T17:23:42.639257+0000","last_peered":"2024-09-13T17:23:42.639257+0000","last_clean":"2024-09-13T17:23:42.639257+0000","last_became_active":"2024-09-13T17:23:41.653139+0000","last_became_peered":"2024-09-13T17:23:41.653139+0000","last_unstale":"2024-09-13T17:23:42.639257+0000","last_undegraded":"2024-09-13T17:23:42.639257+0000","last_fullsized":"2024-09-13T17:23:42.639257+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:56.708929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639805+0000","last_change":"2024-09-13T17:23:41.647731+0000","last_active":"2024-09-13T17:23:42.639805+0000","last_peered":"2024-09-13T17:23:42.639805+0000","last_clean":"2024-09-13T17:23:42.639805+0000","last_became_active":"2024-09-13T17:23:41.647313+0000","last_became_peered":"2024-09-13T17:23:41.647313+0000","last_unstale":"2024-09-13T17:23:42.639805+0000","last_undegraded":"2024-09-13T17:23:42.639805+0000","last_fullsized":"2024-09-13T17:23:42.639805+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:42.423401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.661387+0000","last_change":"2024-09-13T17:23:41.650632+0000","last_active":"2024-09-13T17:23:41.661387+0000","last_peered":"2024-09-13T17:23:41.661387+0000","last_clean":"2024-09-13T17:23:41.661387+0000","last_became_active":"2024-09-13T17:23:41.648143+0000","last_became_peered":"2024-09-13T17:23:41.648143+0000","last_unstale":"2024-09-13T17:23:41.661387+0000","last_undegraded":"2024-09-13T17:23:41.661387+0000","last_fullsized":"2024-09-13T17:23:41.661387+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:19:54.536858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656234+0000","last_change":"2024-09-13T17:23:41.646966+0000","last_active":"2024-09-13T17:23:41.656234+0000","last_peered":"2024-09-13T17:23:41.656234+0000","last_clean":"2024-09-13T17:23:41.656234+0000","last_became_active":"2024-09-13T17:23:41.646664+0000","last_became_peered":"2024-09-13T17:23:41.646664+0000","last_unstale":"2024-09-13T17:23:41.656234+0000","last_undegraded":"2024-09-13T17:23:41.656234+0000","last_fullsized":"2024-09-13T17:23:41.656234+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:17:27.661820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.670021+0000","last_change":"2024-09-13T17:23:41.667798+0000","last_active":"2024-09-13T17:23:41.670021+0000","last_peered":"2024-09-13T17:23:41.670021+0000","last_clean":"2024-09-13T17:23:41.670021+0000","last_became_active":"2024-09-13T17:23:41.666086+0000","last_became_peered":"2024-09-13T17:23:41.666086+0000","last_unstale":"2024-09-13T17:23:41.670021+0000","last_undegraded":"2024-09-13T17:23:41.670021+0000","last_fullsized":"2024-09-13T17:23:41.670021+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:50.550423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658269+0000","last_change":"2024-09-13T17:23:41.218763+0000","last_active":"2024-09-13T17:23:41.658269+0000","last_peered":"2024-09-13T17:23:41.658269+0000","last_clean":"2024-09-13T17:23:41.658269+0000","last_became_active":"2024-09-13T17:23:41.218450+0000","last_became_peered":"2024-09-13T17:23:41.218450+0000","last_unstale":"2024-09-13T17:23:41.658269+0000","last_undegraded":"2024-09-13T17:23:41.658269+0000","last_fullsized":"2024-09-13T17:23:41.658269+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:23.885396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639194+0000","last_change":"2024-09-13T17:23:41.661650+0000","last_active":"2024-09-13T17:23:42.639194+0000","last_peered":"2024-09-13T17:23:42.639194+0000","last_clean":"2024-09-13T17:23:42.639194+0000","last_became_active":"2024-09-13T17:23:41.660515+0000","last_became_peered":"2024-09-13T17:23:41.660515+0000","last_unstale":"2024-09-13T17:23:42.639194+0000","last_undegraded":"2024-09-13T17:23:42.639194+0000","last_fullsized":"2024-09-13T17:23:42.639194+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:54:23.657853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669191+0000","last_change":"2024-09-13T17:23:41.222255+0000","last_active":"2024-09-13T17:23:41.669191+0000","last_peered":"2024-09-13T17:23:41.669191+0000","last_clean":"2024-09-13T17:23:41.669191+0000","last_became_active":"2024-09-13T17:23:41.221982+0000","last_became_peered":"2024-09-13T17:23:41.221982+0000","last_unstale":"2024-09-13T17:23:41.669191+0000","last_undegraded":"2024-09-13T17:23:41.669191+0000","last_fullsized":"2024-09-13T17:23:41.669191+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:35.121129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640250+0000","last_change":"2024-09-13T17:23:41.663289+0000","last_active":"2024-09-13T17:23:42.640250+0000","last_peered":"2024-09-13T17:23:42.640250+0000","last_clean":"2024-09-13T17:23:42.640250+0000","last_became_active":"2024-09-13T17:23:41.662870+0000","last_became_peered":"2024-09-13T17:23:41.662870+0000","last_unstale":"2024-09-13T17:23:42.640250+0000","last_undegraded":"2024-09-13T17:23:42.640250+0000","last_fullsized":"2024-09-13T17:23:42.640250+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:44.440920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.667149+0000","last_change":"2024-09-13T17:23:41.221819+0000","last_active":"2024-09-13T17:23:41.667149+0000","last_peered":"2024-09-13T17:23:41.667149+0000","last_clean":"2024-09-13T17:23:41.667149+0000","last_became_active":"2024-09-13T17:23:41.221433+0000","last_became_peered":"2024-09-13T17:23:41.221433+0000","last_unstale":"2024-09-13T17:23:41.667149+0000","last_undegraded":"2024-09-13T17:23:41.667149+0000","last_fullsized":"2024-09-13T17:23:41.667149+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:54.050404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639091+0000","last_change":"2024-09-13T17:23:41.655695+0000","last_active":"2024-09-13T17:23:42.639091+0000","last_peered":"2024-09-13T17:23:42.639091+0000","last_clean":"2024-09-13T17:23:42.639091+0000","last_became_active":"2024-09-13T17:23:41.655375+0000","last_became_peered":"2024-09-13T17:23:41.655375+0000","last_unstale":"2024-09-13T17:23:42.639091+0000","last_undegraded":"2024-09-13T17:23:42.639091+0000","last_fullsized":"2024-09-13T17:23:42.639091+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:21:03.503645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662931+0000","last_change":"2024-09-13T17:23:41.215473+0000","last_active":"2024-09-13T17:23:41.662931+0000","last_peered":"2024-09-13T17:23:41.662931+0000","last_clean":"2024-09-13T17:23:41.662931+0000","last_became_active":"2024-09-13T17:23:41.215169+0000","last_became_peered":"2024-09-13T17:23:41.215169+0000","last_unstale":"2024-09-13T17:23:41.662931+0000","last_undegraded":"2024-09-13T17:23:41.662931+0000","last_fullsized":"2024-09-13T17:23:41.662931+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:31.364289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683202+0000","last_change":"2024-09-13T17:23:41.651615+0000","last_active":"2024-09-13T17:23:42.683202+0000","last_peered":"2024-09-13T17:23:42.683202+0000","last_clean":"2024-09-13T17:23:42.683202+0000","last_became_active":"2024-09-13T17:23:41.651204+0000","last_became_peered":"2024-09-13T17:23:41.651204+0000","last_unstale":"2024-09-13T17:23:42.683202+0000","last_undegraded":"2024-09-13T17:23:42.683202+0000","last_fullsized":"2024-09-13T17:23:42.683202+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:50.579369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656102+0000","last_change":"2024-09-13T17:23:41.211244+0000","last_active":"2024-09-13T17:23:41.656102+0000","last_peered":"2024-09-13T17:23:41.656102+0000","last_clean":"2024-09-13T17:23:41.656102+0000","last_became_active":"2024-09-13T17:23:41.210982+0000","last_became_peered":"2024-09-13T17:23:41.210982+0000","last_unstale":"2024-09-13T17:23:41.656102+0000","last_undegraded":"2024-09-13T17:23:41.656102+0000","last_fullsized":"2024-09-13T17:23:41.656102+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:23.885396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658591+0000","last_change":"2024-09-13T17:23:41.650569+0000","last_active":"2024-09-13T17:23:41.658591+0000","last_peered":"2024-09-13T17:23:41.658591+0000","last_clean":"2024-09-13T17:23:41.658591+0000","last_became_active":"2024-09-13T17:23:41.649276+0000","last_became_peered":"2024-09-13T17:23:41.649276+0000","last_unstale":"2024-09-13T17:23:41.658591+0000","last_undegraded":"2024-09-13T17:23:41.658591+0000","last_fullsized":"2024-09-13T17:23:41.658591+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:56:32.718664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.640588+0000","last_change":"2024-09-13T17:23:41.223475+0000","last_active":"2024-09-13T17:23:41.640588+0000","last_peered":"2024-09-13T17:23:41.640588+0000","last_clean":"2024-09-13T17:23:41.640588+0000","last_became_active":"2024-09-13T17:23:41.223112+0000","last_became_peered":"2024-09-13T17:23:41.223112+0000","last_unstale":"2024-09-13T17:23:41.640588+0000","last_undegraded":"2024-09-13T17:23:41.640588+0000","last_fullsized":"2024-09-13T17:23:41.640588+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:51.276487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656774+0000","last_change":"2024-09-13T17:23:41.649173+0000","last_active":"2024-09-13T17:23:41.656774+0000","last_peered":"2024-09-13T17:23:41.656774+0000","last_clean":"2024-09-13T17:23:41.656774+0000","last_became_active":"2024-09-13T17:23:41.648851+0000","last_became_peered":"2024-09-13T17:23:41.648851+0000","last_unstale":"2024-09-13T17:23:41.656774+0000","last_undegraded":"2024-09-13T17:23:41.656774+0000","last_fullsized":"2024-09-13T17:23:41.656774+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:46.685287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643109+0000","last_change":"2024-09-13T17:23:41.220852+0000","last_active":"2024-09-13T17:23:41.643109+0000","last_peered":"2024-09-13T17:23:41.643109+0000","last_clean":"2024-09-13T17:23:41.643109+0000","last_became_active":"2024-09-13T17:23:41.220582+0000","last_became_peered":"2024-09-13T17:23:41.220582+0000","last_unstale":"2024-09-13T17:23:41.643109+0000","last_undegraded":"2024-09-13T17:23:41.643109+0000","last_fullsized":"2024-09-13T17:23:41.643109+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:18.526170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660712+0000","last_change":"2024-09-13T17:23:41.652918+0000","last_active":"2024-09-13T17:23:41.660712+0000","last_peered":"2024-09-13T17:23:41.660712+0000","last_clean":"2024-09-13T17:23:41.660712+0000","last_became_active":"2024-09-13T17:23:41.652183+0000","last_became_peered":"2024-09-13T17:23:41.652183+0000","last_unstale":"2024-09-13T17:23:41.660712+0000","last_undegraded":"2024-09-13T17:23:41.660712+0000","last_fullsized":"2024-09-13T17:23:41.660712+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:48.715837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640140+0000","last_change":"2024-09-13T17:23:41.221596+0000","last_active":"2024-09-13T17:23:42.640140+0000","last_peered":"2024-09-13T17:23:42.640140+0000","last_clean":"2024-09-13T17:23:42.640140+0000","last_became_active":"2024-09-13T17:23:41.221305+0000","last_became_peered":"2024-09-13T17:23:41.221305+0000","last_unstale":"2024-09-13T17:23:42.640140+0000","last_undegraded":"2024-09-13T17:23:42.640140+0000","last_fullsized":"2024-09-13T17:23:42.640140+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:14.204122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639556+0000","last_change":"2024-09-13T17:23:41.652093+0000","last_active":"2024-09-13T17:23:42.639556+0000","last_peered":"2024-09-13T17:23:42.639556+0000","last_clean":"2024-09-13T17:23:42.639556+0000","last_became_active":"2024-09-13T17:23:41.651846+0000","last_became_peered":"2024-09-13T17:23:41.651846+0000","last_unstale":"2024-09-13T17:23:42.639556+0000","last_undegraded":"2024-09-13T17:23:42.639556+0000","last_fullsized":"2024-09-13T17:23:42.639556+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:00:50.550423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643289+0000","last_change":"2024-09-13T17:23:41.220880+0000","last_active":"2024-09-13T17:23:41.643289+0000","last_peered":"2024-09-13T17:23:41.643289+0000","last_clean":"2024-09-13T17:23:41.643289+0000","last_became_active":"2024-09-13T17:23:41.220592+0000","last_became_peered":"2024-09-13T17:23:41.220592+0000","last_unstale":"2024-09-13T17:23:41.643289+0000","last_undegraded":"2024-09-13T17:23:41.643289+0000","last_fullsized":"2024-09-13T17:23:41.643289+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:05.196577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658929+0000","last_change":"2024-09-13T17:23:41.655000+0000","last_active":"2024-09-13T17:23:41.658929+0000","last_peered":"2024-09-13T17:23:41.658929+0000","last_clean":"2024-09-13T17:23:41.658929+0000","last_became_active":"2024-09-13T17:23:41.654708+0000","last_became_peered":"2024-09-13T17:23:41.654708+0000","last_unstale":"2024-09-13T17:23:41.658929+0000","last_undegraded":"2024-09-13T17:23:41.658929+0000","last_fullsized":"2024-09-13T17:23:41.658929+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:13.182568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"218'3","reported_seq":13,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.663447+0000","last_change":"2024-09-13T17:23:41.211352+0000","last_active":"2024-09-13T17:23:41.663447+0000","last_peered":"2024-09-13T17:23:41.663447+0000","last_clean":"2024-09-13T17:23:41.663447+0000","last_became_active":"2024-09-13T17:23:41.211049+0000","last_became_peered":"2024-09-13T17:23:41.211049+0000","last_unstale":"2024-09-13T17:23:41.663447+0000","last_undegraded":"2024-09-13T17:23:41.663447+0000","last_fullsized":"2024-09-13T17:23:41.663447+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:51.276487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.653345+0000","last_change":"2024-09-13T17:23:41.650056+0000","last_active":"2024-09-13T17:23:41.653345+0000","last_peered":"2024-09-13T17:23:41.653345+0000","last_clean":"2024-09-13T17:23:41.653345+0000","last_became_active":"2024-09-13T17:23:41.648322+0000","last_became_peered":"2024-09-13T17:23:41.648322+0000","last_unstale":"2024-09-13T17:23:41.653345+0000","last_undegraded":"2024-09-13T17:23:41.653345+0000","last_fullsized":"2024-09-13T17:23:41.653345+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:25.632329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"218'6","reported_seq":16,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669533+0000","last_change":"2024-09-13T17:23:41.214296+0000","last_active":"2024-09-13T17:23:41.669533+0000","last_peered":"2024-09-13T17:23:41.669533+0000","last_clean":"2024-09-13T17:23:41.669533+0000","last_became_active":"2024-09-13T17:23:41.214046+0000","last_became_peered":"2024-09-13T17:23:41.214046+0000","last_unstale":"2024-09-13T17:23:41.669533+0000","last_undegraded":"2024-09-13T17:23:41.669533+0000","last_fullsized":"2024-09-13T17:23:41.669533+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:45.265127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.663232+0000","last_change":"2024-09-13T17:23:41.662642+0000","last_active":"2024-09-13T17:23:41.663232+0000","last_peered":"2024-09-13T17:23:41.663232+0000","last_clean":"2024-09-13T17:23:41.663232+0000","last_became_active":"2024-09-13T17:23:41.662336+0000","last_became_peered":"2024-09-13T17:23:41.662336+0000","last_unstale":"2024-09-13T17:23:41.663232+0000","last_undegraded":"2024-09-13T17:23:41.663232+0000","last_fullsized":"2024-09-13T17:23:41.663232+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:53:48.715837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.665624+0000","last_change":"2024-09-13T17:23:41.224408+0000","last_active":"2024-09-13T17:23:41.665624+0000","last_peered":"2024-09-13T17:23:41.665624+0000","last_clean":"2024-09-13T17:23:41.665624+0000","last_became_active":"2024-09-13T17:23:41.223968+0000","last_became_peered":"2024-09-13T17:23:41.223968+0000","last_unstale":"2024-09-13T17:23:41.665624+0000","last_undegraded":"2024-09-13T17:23:41.665624+0000","last_fullsized":"2024-09-13T17:23:41.665624+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:01:10.543493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683152+0000","last_change":"2024-09-13T17:23:41.649950+0000","last_active":"2024-09-13T17:23:42.683152+0000","last_peered":"2024-09-13T17:23:42.683152+0000","last_clean":"2024-09-13T17:23:42.683152+0000","last_became_active":"2024-09-13T17:23:41.649645+0000","last_became_peered":"2024-09-13T17:23:41.649645+0000","last_unstale":"2024-09-13T17:23:42.683152+0000","last_undegraded":"2024-09-13T17:23:42.683152+0000","last_fullsized":"2024-09-13T17:23:42.683152+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:30:41.713454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643116+0000","last_change":"2024-09-13T17:23:41.218589+0000","last_active":"2024-09-13T17:23:41.643116+0000","last_peered":"2024-09-13T17:23:41.643116+0000","last_clean":"2024-09-13T17:23:41.643116+0000","last_became_active":"2024-09-13T17:23:41.218264+0000","last_became_peered":"2024-09-13T17:23:41.218264+0000","last_unstale":"2024-09-13T17:23:41.643116+0000","last_undegraded":"2024-09-13T17:23:41.643116+0000","last_fullsized":"2024-09-13T17:23:41.643116+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:22:59.974787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.647521+0000","last_change":"2024-09-13T17:23:41.646671+0000","last_active":"2024-09-13T17:23:41.647521+0000","last_peered":"2024-09-13T17:23:41.647521+0000","last_clean":"2024-09-13T17:23:41.647521+0000","last_became_active":"2024-09-13T17:23:41.646243+0000","last_became_peered":"2024-09-13T17:23:41.646243+0000","last_unstale":"2024-09-13T17:23:41.647521+0000","last_undegraded":"2024-09-13T17:23:41.647521+0000","last_fullsized":"2024-09-13T17:23:41.647521+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:28:44.988720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.640677+0000","last_change":"2024-09-13T17:23:41.224054+0000","last_active":"2024-09-13T17:23:41.640677+0000","last_peered":"2024-09-13T17:23:41.640677+0000","last_clean":"2024-09-13T17:23:41.640677+0000","last_became_active":"2024-09-13T17:23:41.223469+0000","last_became_peered":"2024-09-13T17:23:41.223469+0000","last_unstale":"2024-09-13T17:23:41.640677+0000","last_undegraded":"2024-09-13T17:23:41.640677+0000","last_fullsized":"2024-09-13T17:23:41.640677+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:46:29.791684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658516+0000","last_change":"2024-09-13T17:23:41.656735+0000","last_active":"2024-09-13T17:23:41.658516+0000","last_peered":"2024-09-13T17:23:41.658516+0000","last_clean":"2024-09-13T17:23:41.658516+0000","last_became_active":"2024-09-13T17:23:41.655485+0000","last_became_peered":"2024-09-13T17:23:41.655485+0000","last_unstale":"2024-09-13T17:23:41.658516+0000","last_undegraded":"2024-09-13T17:23:41.658516+0000","last_fullsized":"2024-09-13T17:23:41.658516+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:12:26.860796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"218'8","reported_seq":19,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.638917+0000","last_change":"2024-09-13T17:23:41.217988+0000","last_active":"2024-09-13T17:23:42.638917+0000","last_peered":"2024-09-13T17:23:42.638917+0000","last_clean":"2024-09-13T17:23:42.638917+0000","last_became_active":"2024-09-13T17:23:41.217724+0000","last_became_peered":"2024-09-13T17:23:41.217724+0000","last_unstale":"2024-09-13T17:23:42.638917+0000","last_undegraded":"2024-09-13T17:23:42.638917+0000","last_fullsized":"2024-09-13T17:23:42.638917+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:44.412141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643860+0000","last_change":"2024-09-13T17:23:41.640787+0000","last_active":"2024-09-13T17:23:41.643860+0000","last_peered":"2024-09-13T17:23:41.643860+0000","last_clean":"2024-09-13T17:23:41.643860+0000","last_became_active":"2024-09-13T17:23:41.640508+0000","last_became_peered":"2024-09-13T17:23:41.640508+0000","last_unstale":"2024-09-13T17:23:41.643860+0000","last_undegraded":"2024-09-13T17:23:41.643860+0000","last_fullsized":"2024-09-13T17:23:41.643860+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:17:57.496659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"218'1","reported_seq":11,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660410+0000","last_change":"2024-09-13T17:23:41.222159+0000","last_active":"2024-09-13T17:23:41.660410+0000","last_peered":"2024-09-13T17:23:41.660410+0000","last_clean":"2024-09-13T17:23:41.660410+0000","last_became_active":"2024-09-13T17:23:41.221910+0000","last_became_peered":"2024-09-13T17:23:41.221910+0000","last_unstale":"2024-09-13T17:23:41.660410+0000","last_undegraded":"2024-09-13T17:23:41.660410+0000","last_fullsized":"2024-09-13T17:23:41.660410+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:05.289194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.658859+0000","last_change":"2024-09-13T17:23:41.654148+0000","last_active":"2024-09-13T17:23:41.658859+0000","last_peered":"2024-09-13T17:23:41.658859+0000","last_clean":"2024-09-13T17:23:41.658859+0000","last_became_active":"2024-09-13T17:23:41.653895+0000","last_became_peered":"2024-09-13T17:23:41.653895+0000","last_unstale":"2024-09-13T17:23:41.658859+0000","last_undegraded":"2024-09-13T17:23:41.658859+0000","last_fullsized":"2024-09-13T17:23:41.658859+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:13.325786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669746+0000","last_change":"2024-09-13T17:23:41.222511+0000","last_active":"2024-09-13T17:23:41.669746+0000","last_peered":"2024-09-13T17:23:41.669746+0000","last_clean":"2024-09-13T17:23:41.669746+0000","last_became_active":"2024-09-13T17:23:41.222259+0000","last_became_peered":"2024-09-13T17:23:41.222259+0000","last_unstale":"2024-09-13T17:23:41.669746+0000","last_undegraded":"2024-09-13T17:23:41.669746+0000","last_fullsized":"2024-09-13T17:23:41.669746+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:36:30.077962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683037+0000","last_change":"2024-09-13T17:23:41.655170+0000","last_active":"2024-09-13T17:23:42.683037+0000","last_peered":"2024-09-13T17:23:42.683037+0000","last_clean":"2024-09-13T17:23:42.683037+0000","last_became_active":"2024-09-13T17:23:41.654653+0000","last_became_peered":"2024-09-13T17:23:41.654653+0000","last_unstale":"2024-09-13T17:23:42.683037+0000","last_undegraded":"2024-09-13T17:23:42.683037+0000","last_fullsized":"2024-09-13T17:23:42.683037+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:25.253708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"218'1","reported_seq":12,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683069+0000","last_change":"2024-09-13T17:23:41.219784+0000","last_active":"2024-09-13T17:23:42.683069+0000","last_peered":"2024-09-13T17:23:42.683069+0000","last_clean":"2024-09-13T17:23:42.683069+0000","last_became_active":"2024-09-13T17:23:41.219451+0000","last_became_peered":"2024-09-13T17:23:41.219451+0000","last_unstale":"2024-09-13T17:23:42.683069+0000","last_undegraded":"2024-09-13T17:23:42.683069+0000","last_fullsized":"2024-09-13T17:23:42.683069+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:43.099661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640343+0000","last_change":"2024-09-13T17:23:41.662847+0000","last_active":"2024-09-13T17:23:42.640343+0000","last_peered":"2024-09-13T17:23:42.640343+0000","last_clean":"2024-09-13T17:23:42.640343+0000","last_became_active":"2024-09-13T17:23:41.662470+0000","last_became_peered":"2024-09-13T17:23:41.662470+0000","last_unstale":"2024-09-13T17:23:42.640343+0000","last_undegraded":"2024-09-13T17:23:42.640343+0000","last_fullsized":"2024-09-13T17:23:42.640343+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:53.634317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639283+0000","last_change":"2024-09-13T17:23:41.223642+0000","last_active":"2024-09-13T17:23:42.639283+0000","last_peered":"2024-09-13T17:23:42.639283+0000","last_clean":"2024-09-13T17:23:42.639283+0000","last_became_active":"2024-09-13T17:23:41.223363+0000","last_became_peered":"2024-09-13T17:23:41.223363+0000","last_unstale":"2024-09-13T17:23:42.639283+0000","last_undegraded":"2024-09-13T17:23:42.639283+0000","last_fullsized":"2024-09-13T17:23:42.639283+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639683+0000","last_change":"2024-09-13T17:23:41.660463+0000","last_active":"2024-09-13T17:23:42.639683+0000","last_peered":"2024-09-13T17:23:42.639683+0000","last_clean":"2024-09-13T17:23:42.639683+0000","last_became_active":"2024-09-13T17:23:41.660093+0000","last_became_peered":"2024-09-13T17:23:41.660093+0000","last_unstale":"2024-09-13T17:23:42.639683+0000","last_undegraded":"2024-09-13T17:23:42.639683+0000","last_fullsized":"2024-09-13T17:23:42.639683+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:47.353460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.669219+0000","last_change":"2024-09-13T17:23:41.224936+0000","last_active":"2024-09-13T17:23:41.669219+0000","last_peered":"2024-09-13T17:23:41.669219+0000","last_clean":"2024-09-13T17:23:41.669219+0000","last_became_active":"2024-09-13T17:23:41.224357+0000","last_became_peered":"2024-09-13T17:23:41.224357+0000","last_unstale":"2024-09-13T17:23:41.669219+0000","last_undegraded":"2024-09-13T17:23:41.669219+0000","last_fullsized":"2024-09-13T17:23:41.669219+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:27.600246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.660665+0000","last_change":"2024-09-13T17:23:41.650512+0000","last_active":"2024-09-13T17:23:41.660665+0000","last_peered":"2024-09-13T17:23:41.660665+0000","last_clean":"2024-09-13T17:23:41.660665+0000","last_became_active":"2024-09-13T17:23:41.650268+0000","last_became_peered":"2024-09-13T17:23:41.650268+0000","last_unstale":"2024-09-13T17:23:41.660665+0000","last_undegraded":"2024-09-13T17:23:41.660665+0000","last_fullsized":"2024-09-13T17:23:41.660665+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:29.643143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639421+0000","last_change":"2024-09-13T17:23:41.223914+0000","last_active":"2024-09-13T17:23:42.639421+0000","last_peered":"2024-09-13T17:23:42.639421+0000","last_clean":"2024-09-13T17:23:42.639421+0000","last_became_active":"2024-09-13T17:23:41.222778+0000","last_became_peered":"2024-09-13T17:23:41.222778+0000","last_unstale":"2024-09-13T17:23:42.639421+0000","last_undegraded":"2024-09-13T17:23:42.639421+0000","last_fullsized":"2024-09-13T17:23:42.639421+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:27.600246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.638857+0000","last_change":"2024-09-13T17:23:41.649907+0000","last_active":"2024-09-13T17:23:42.638857+0000","last_peered":"2024-09-13T17:23:42.638857+0000","last_clean":"2024-09-13T17:23:42.638857+0000","last_became_active":"2024-09-13T17:23:41.649643+0000","last_became_peered":"2024-09-13T17:23:41.649643+0000","last_unstale":"2024-09-13T17:23:42.638857+0000","last_undegraded":"2024-09-13T17:23:42.638857+0000","last_fullsized":"2024-09-13T17:23:42.638857+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:56:32.718664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683055+0000","last_change":"2024-09-13T17:23:41.220607+0000","last_active":"2024-09-13T17:23:42.683055+0000","last_peered":"2024-09-13T17:23:42.683055+0000","last_clean":"2024-09-13T17:23:42.683055+0000","last_became_active":"2024-09-13T17:23:41.220347+0000","last_became_peered":"2024-09-13T17:23:41.220347+0000","last_unstale":"2024-09-13T17:23:42.683055+0000","last_undegraded":"2024-09-13T17:23:42.683055+0000","last_fullsized":"2024-09-13T17:23:42.683055+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:11.371958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639513+0000","last_change":"2024-09-13T17:23:41.662762+0000","last_active":"2024-09-13T17:23:42.639513+0000","last_peered":"2024-09-13T17:23:42.639513+0000","last_clean":"2024-09-13T17:23:42.639513+0000","last_became_active":"2024-09-13T17:23:41.662478+0000","last_became_peered":"2024-09-13T17:23:41.662478+0000","last_unstale":"2024-09-13T17:23:42.639513+0000","last_undegraded":"2024-09-13T17:23:42.639513+0000","last_fullsized":"2024-09-13T17:23:42.639513+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:08.721032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.641583+0000","last_change":"2024-09-13T17:23:41.219347+0000","last_active":"2024-09-13T17:23:41.641583+0000","last_peered":"2024-09-13T17:23:41.641583+0000","last_clean":"2024-09-13T17:23:41.641583+0000","last_became_active":"2024-09-13T17:23:41.219104+0000","last_became_peered":"2024-09-13T17:23:41.219104+0000","last_unstale":"2024-09-13T17:23:41.641583+0000","last_undegraded":"2024-09-13T17:23:41.641583+0000","last_fullsized":"2024-09-13T17:23:41.641583+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:29:45.265127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.649757+0000","last_change":"2024-09-13T17:23:41.646887+0000","last_active":"2024-09-13T17:23:41.649757+0000","last_peered":"2024-09-13T17:23:41.649757+0000","last_clean":"2024-09-13T17:23:41.649757+0000","last_became_active":"2024-09-13T17:23:41.646571+0000","last_became_peered":"2024-09-13T17:23:41.646571+0000","last_unstale":"2024-09-13T17:23:41.649757+0000","last_undegraded":"2024-09-13T17:23:41.649757+0000","last_fullsized":"2024-09-13T17:23:41.649757+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:02:57.781369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"218'2","reported_seq":12,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.666813+0000","last_change":"2024-09-13T17:23:41.218908+0000","last_active":"2024-09-13T17:23:41.666813+0000","last_peered":"2024-09-13T17:23:41.666813+0000","last_clean":"2024-09-13T17:23:41.666813+0000","last_became_active":"2024-09-13T17:23:41.218636+0000","last_became_peered":"2024-09-13T17:23:41.218636+0000","last_unstale":"2024-09-13T17:23:41.666813+0000","last_undegraded":"2024-09-13T17:23:41.666813+0000","last_fullsized":"2024-09-13T17:23:41.666813+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:43:47.952641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.662666+0000","last_change":"2024-09-13T17:23:41.659475+0000","last_active":"2024-09-13T17:23:41.662666+0000","last_peered":"2024-09-13T17:23:41.662666+0000","last_clean":"2024-09-13T17:23:41.662666+0000","last_became_active":"2024-09-13T17:23:41.659208+0000","last_became_peered":"2024-09-13T17:23:41.659208+0000","last_unstale":"2024-09-13T17:23:41.662666+0000","last_undegraded":"2024-09-13T17:23:41.662666+0000","last_fullsized":"2024-09-13T17:23:41.662666+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:46.760609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640492+0000","last_change":"2024-09-13T17:23:41.219933+0000","last_active":"2024-09-13T17:23:42.640492+0000","last_peered":"2024-09-13T17:23:42.640492+0000","last_clean":"2024-09-13T17:23:42.640492+0000","last_became_active":"2024-09-13T17:23:41.219593+0000","last_became_peered":"2024-09-13T17:23:41.219593+0000","last_unstale":"2024-09-13T17:23:42.640492+0000","last_undegraded":"2024-09-13T17:23:42.640492+0000","last_fullsized":"2024-09-13T17:23:42.640492+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:23:17.395117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639234+0000","last_change":"2024-09-13T17:23:41.656791+0000","last_active":"2024-09-13T17:23:42.639234+0000","last_peered":"2024-09-13T17:23:42.639234+0000","last_clean":"2024-09-13T17:23:42.639234+0000","last_became_active":"2024-09-13T17:23:41.656558+0000","last_became_peered":"2024-09-13T17:23:41.656558+0000","last_unstale":"2024-09-13T17:23:42.639234+0000","last_undegraded":"2024-09-13T17:23:42.639234+0000","last_fullsized":"2024-09-13T17:23:42.639234+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:37:16.342864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.683154+0000","last_change":"2024-09-13T17:23:41.222323+0000","last_active":"2024-09-13T17:23:42.683154+0000","last_peered":"2024-09-13T17:23:42.683154+0000","last_clean":"2024-09-13T17:23:42.683154+0000","last_became_active":"2024-09-13T17:23:41.222026+0000","last_became_peered":"2024-09-13T17:23:41.222026+0000","last_unstale":"2024-09-13T17:23:42.683154+0000","last_undegraded":"2024-09-13T17:23:42.683154+0000","last_fullsized":"2024-09-13T17:23:42.683154+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:44:05.289194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639486+0000","last_change":"2024-09-13T17:23:41.653884+0000","last_active":"2024-09-13T17:23:42.639486+0000","last_peered":"2024-09-13T17:23:42.639486+0000","last_clean":"2024-09-13T17:23:42.639486+0000","last_became_active":"2024-09-13T17:23:41.653601+0000","last_became_peered":"2024-09-13T17:23:41.653601+0000","last_unstale":"2024-09-13T17:23:42.639486+0000","last_undegraded":"2024-09-13T17:23:42.639486+0000","last_fullsized":"2024-09-13T17:23:42.639486+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:29:20.279302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":11,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639332+0000","last_change":"2024-09-13T17:23:41.223547+0000","last_active":"2024-09-13T17:23:42.639332+0000","last_peered":"2024-09-13T17:23:42.639332+0000","last_clean":"2024-09-13T17:23:42.639332+0000","last_became_active":"2024-09-13T17:23:41.222840+0000","last_became_peered":"2024-09-13T17:23:41.222840+0000","last_unstale":"2024-09-13T17:23:42.639332+0000","last_undegraded":"2024-09-13T17:23:42.639332+0000","last_fullsized":"2024-09-13T17:23:42.639332+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:48:35.121129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.653375+0000","last_change":"2024-09-13T17:23:41.651660+0000","last_active":"2024-09-13T17:23:41.653375+0000","last_peered":"2024-09-13T17:23:41.653375+0000","last_clean":"2024-09-13T17:23:41.653375+0000","last_became_active":"2024-09-13T17:23:41.651181+0000","last_became_peered":"2024-09-13T17:23:41.651181+0000","last_unstale":"2024-09-13T17:23:41.653375+0000","last_undegraded":"2024-09-13T17:23:41.653375+0000","last_fullsized":"2024-09-13T17:23:41.653375+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:50.579369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.641651+0000","last_change":"2024-09-13T17:23:41.222904+0000","last_active":"2024-09-13T17:23:41.641651+0000","last_peered":"2024-09-13T17:23:41.641651+0000","last_clean":"2024-09-13T17:23:41.641651+0000","last_became_active":"2024-09-13T17:23:41.222632+0000","last_became_peered":"2024-09-13T17:23:41.222632+0000","last_unstale":"2024-09-13T17:23:41.641651+0000","last_undegraded":"2024-09-13T17:23:41.641651+0000","last_fullsized":"2024-09-13T17:23:41.641651+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:55.070253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639774+0000","last_change":"2024-09-13T17:23:41.658707+0000","last_active":"2024-09-13T17:23:42.639774+0000","last_peered":"2024-09-13T17:23:42.639774+0000","last_clean":"2024-09-13T17:23:42.639774+0000","last_became_active":"2024-09-13T17:23:41.658166+0000","last_became_peered":"2024-09-13T17:23:41.658166+0000","last_unstale":"2024-09-13T17:23:42.639774+0000","last_undegraded":"2024-09-13T17:23:42.639774+0000","last_fullsized":"2024-09-13T17:23:42.639774+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:04:24.819173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.633506+0000","last_change":"2024-09-13T17:23:41.212544+0000","last_active":"2024-09-13T17:23:41.633506+0000","last_peered":"2024-09-13T17:23:41.633506+0000","last_clean":"2024-09-13T17:23:41.633506+0000","last_became_active":"2024-09-13T17:23:41.212034+0000","last_became_peered":"2024-09-13T17:23:41.212034+0000","last_unstale":"2024-09-13T17:23:41.633506+0000","last_undegraded":"2024-09-13T17:23:41.633506+0000","last_fullsized":"2024-09-13T17:23:41.633506+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:54.522241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.639533+0000","last_change":"2024-09-13T17:23:41.650353+0000","last_active":"2024-09-13T17:23:42.639533+0000","last_peered":"2024-09-13T17:23:42.639533+0000","last_clean":"2024-09-13T17:23:42.639533+0000","last_became_active":"2024-09-13T17:23:41.650048+0000","last_became_peered":"2024-09-13T17:23:41.650048+0000","last_unstale":"2024-09-13T17:23:42.639533+0000","last_undegraded":"2024-09-13T17:23:42.639533+0000","last_fullsized":"2024-09-13T17:23:42.639533+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:25.022757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.642941+0000","last_change":"2024-09-13T17:23:41.220622+0000","last_active":"2024-09-13T17:23:41.642941+0000","last_peered":"2024-09-13T17:23:41.642941+0000","last_clean":"2024-09-13T17:23:41.642941+0000","last_became_active":"2024-09-13T17:23:41.220299+0000","last_became_peered":"2024-09-13T17:23:41.220299+0000","last_unstale":"2024-09-13T17:23:41.642941+0000","last_undegraded":"2024-09-13T17:23:41.642941+0000","last_fullsized":"2024-09-13T17:23:41.642941+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:44.412141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":9,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.656755+0000","last_change":"2024-09-13T17:23:41.648897+0000","last_active":"2024-09-13T17:23:41.656755+0000","last_peered":"2024-09-13T17:23:41.656755+0000","last_clean":"2024-09-13T17:23:41.656755+0000","last_became_active":"2024-09-13T17:23:41.648189+0000","last_became_peered":"2024-09-13T17:23:41.648189+0000","last_unstale":"2024-09-13T17:23:41.656755+0000","last_undegraded":"2024-09-13T17:23:41.656755+0000","last_fullsized":"2024-09-13T17:23:41.656755+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:25.253708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":10,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.643149+0000","last_change":"2024-09-13T17:23:41.226165+0000","last_active":"2024-09-13T17:23:41.643149+0000","last_peered":"2024-09-13T17:23:41.643149+0000","last_clean":"2024-09-13T17:23:41.643149+0000","last_became_active":"2024-09-13T17:23:41.225875+0000","last_became_peered":"2024-09-13T17:23:41.225875+0000","last_unstale":"2024-09-13T17:23:41.643149+0000","last_undegraded":"2024-09-13T17:23:41.643149+0000","last_fullsized":"2024-09-13T17:23:41.643149+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:40.611462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":10,"reported_epoch":219,"state":"active+clean","last_fresh":"2024-09-13T17:23:42.640319+0000","last_change":"2024-09-13T17:23:41.651563+0000","last_active":"2024-09-13T17:23:42.640319+0000","last_peered":"2024-09-13T17:23:42.640319+0000","last_clean":"2024-09-13T17:23:42.640319+0000","last_became_active":"2024-09-13T17:23:41.651247+0000","last_became_peered":"2024-09-13T17:23:41.651247+0000","last_unstale":"2024-09-13T17:23:42.640319+0000","last_undegraded":"2024-09-13T17:23:42.640319+0000","last_fullsized":"2024-09-13T17:23:42.640319+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":217,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:41.194954+0000","last_clean_scrub_stamp":"2024-09-13T17:23:41.194954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:20:48.396974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"218'4","reported_seq":14,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.668347+0000","last_change":"2024-09-13T17:23:41.226322+0000","last_active":"2024-09-13T17:23:41.668347+0000","last_peered":"2024-09-13T17:23:41.668347+0000","last_clean":"2024-09-13T17:23:41.668347+0000","last_became_active":"2024-09-13T17:23:41.226067+0000","last_became_peered":"2024-09-13T17:23:41.226067+0000","last_unstale":"2024-09-13T17:23:41.668347+0000","last_undegraded":"2024-09-13T17:23:41.668347+0000","last_fullsized":"2024-09-13T17:23:41.668347+0000","mapping_epoch":216,"log_start":"0'0","ondisk_log_start":"0'0","created":216,"last_epoch_clean":217,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:23:40.187121+0000","last_clean_scrub_stamp":"2024-09-13T17:23:40.187121+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:37.463970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":333,"reported_epoch":218,"state":"active+clean","last_fresh":"2024-09-13T17:23:41.639126+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:23:41.639126+0000","last_peered":"2024-09-13T17:23:41.639126+0000","last_clean":"2024-09-13T17:23:41.639126+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:23:41.639126+0000","last_undegraded":"2024-09-13T17:23:41.639126+0000","last_fullsized":"2024-09-13T17:23:41.639126+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575040,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70432,"kb_used_data":3284,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301408,"statfs":{"total":96636764160,"available":96564641792,"internally_reserved":0,"allocated":3362816,"data_stored":1813525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.92800000000000005,"5min":1.008,"15min":2.0880000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.316},"max":{"1min":0.97599999999999998,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.55600000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96899999999999997,"5min":1.161,"15min":2.5030000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.63200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86799999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.48399999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":1.105,"15min":1.544},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69099999999999995,"15min":0.69699999999999995},"min":{"1min":0.45500000000000002,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":1.0269999999999999,"15min":1.605},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.443,"15min":0.34999999999999998},"max":{"1min":0.93799999999999994,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.754,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.88800000000000001,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.90300000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.629,"15min":0.65000000000000002},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":1.2170000000000001,"15min":1.234},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.47599999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.92200000000000004,"5min":1.135,"15min":1.143},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.60499999999999998,"5min":0.47999999999999998,"15min":0.42599999999999999},"max":{"1min":0.90500000000000003,"5min":1.915,"15min":1.915},"last":0.86299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.39300000000000002,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.55500000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":1.607,"15min":1.607},"last":0.94999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834575041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65756,"kb_used_data":2696,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94306084,"statfs":{"total":96636764160,"available":96569430016,"internally_reserved":0,"allocated":2760704,"data_stored":1217391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":3.6629999999999998},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72499999999999998,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.92400000000000004,"5min":0.97399999999999998,"15min":5.9800000000000004},"last":0.63400000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.874,"5min":0.998,"15min":3.7040000000000002},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.88500000000000001,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.68400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.88,"5min":0.88,"15min":5.9240000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.2629999999999999},"last":0.72899999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.53800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":3.6520000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.70599999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.006,"5min":1.262,"15min":1.6439999999999999},"last":0.439},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.626,"15min":0.64200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":4.7750000000000004},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.68300000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":1.0129999999999999,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":1.089,"5min":1.1559999999999999,"15min":3.609},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70199999999999996,"15min":0.72399999999999998},"min":{"1min":0.504,"5min":0.379,"15min":0.378},"max":{"1min":1.163,"5min":1.163,"15min":4.9669999999999996},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":1.087,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.66000000000000003}]}]},{"osd":5,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64156,"kb_used_data":2700,"kb_used_omap":1,"kb_used_meta":61438,"kb_avail":94307684,"statfs":{"total":96636764160,"available":96571068416,"internally_reserved":0,"allocated":2764800,"data_stored":1223087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":62912966},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.76100000000000001},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86399999999999999,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.76200000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.871,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":1.482,"15min":7.7370000000000001},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.73499999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.67000000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":1.0780000000000001},"min":{"1min":0.56100000000000005,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.042,"5min":1.042,"15min":113.363},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.73799999999999999},"min":{"1min":0.63400000000000001,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":1.113,"15min":4.1180000000000003},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":1.03,"15min":2.827},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.48199999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.0609999999999999,"5min":1.73,"15min":2.2320000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":1.532,"15min":4.1500000000000004},"last":0.73199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.59899999999999998,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.169,"5min":1.363,"15min":2.02},"last":0.78100000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575039,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73968,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297872,"statfs":{"total":96636764160,"available":96561020928,"internally_reserved":0,"allocated":2785280,"data_stored":1224743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.2150000000000001,"15min":0.84099999999999997},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.2010000000000001,"15min":0.875},"min":{"1min":0.46600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93200000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.85299999999999998,"15min":0.72199999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96799999999999997,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.753,"15min":0.68200000000000005},"min":{"1min":0.53800000000000003,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.97599999999999998,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71799999999999997,"15min":0.67900000000000005},"min":{"1min":0.49299999999999999,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.996,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91500000000000004,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.78700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.109,"15min":0.82599999999999996},"min":{"1min":0.61799999999999999,"5min":0.437,"15min":0.378},"max":{"1min":0.95499999999999996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.72599999999999998},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.92600000000000005,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.65600000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93799999999999994,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.496,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.88200000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.755}]},{"osd":6,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":0.92700000000000005,"5min":2.976,"15min":2.976},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.73399999999999999},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96399999999999997,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.85199999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54400000000000004,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.872},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.749},"min":{"1min":0.55400000000000005,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.95099999999999996,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.83399999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575040,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69848,"kb_used_data":2692,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301992,"statfs":{"total":96636764160,"available":96565239808,"internally_reserved":0,"allocated":2756608,"data_stored":1217369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":0.624,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.373},{"interface":"front","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":2.96},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.503},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":3.5600000000000001},"last":0.495},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.379,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.38900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.875,"15min":1.3819999999999999},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.1479999999999999,"5min":1.21,"15min":2.8610000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.48199999999999998},"max":{"1min":1.0860000000000001,"5min":1.23,"15min":1.3640000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.071,"5min":1.323,"15min":1.391},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.499,"15min":0.442},"max":{"1min":1.05,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.78600000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77800000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.1559999999999999,"5min":1.218,"15min":1.401},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81699999999999995,"15min":0.78600000000000003},"min":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.175,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.73599999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78900000000000003},"min":{"1min":0.624,"5min":0.503,"15min":0.42699999999999999},"max":{"1min":1.2130000000000001,"5min":1.27,"15min":2.899},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.208,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.66000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834575040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47764,"kb_used_data":2688,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324076,"statfs":{"total":96636764160,"available":96587853824,"internally_reserved":0,"allocated":2752512,"data_stored":1215858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47599999999999998,"15min":0.436},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.309},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48499999999999999,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69599999999999995,"5min":1.593,"15min":30.023},"last":0.40000000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":1.371},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.67000000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":1.4690000000000001},"min":{"1min":0.38500000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74299999999999999,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.53000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.371},"max":{"1min":1.264,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.88900000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79300000000000004,"15min":0.878},"min":{"1min":0.52300000000000002,"5min":0.48299999999999998,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82199999999999995,"15min":0.78400000000000003},"min":{"1min":0.61399999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1930000000000001,"5min":6.4800000000000004,"15min":10.664},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81499999999999995,"15min":0.89100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.23,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.872}]},{"osd":7,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1299999999999999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82999999999999996,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.93400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575039,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92004,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":89278,"kb_avail":94279836,"statfs":{"total":96636764160,"available":96542552064,"internally_reserved":0,"allocated":2772992,"data_stored":1218475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91421126},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.441,"15min":0.46300000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.63300000000000001,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.52800000000000002,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.74199999999999999,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.28,"15min":6.1630000000000003},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.23,"15min":1.78},"last":0.81799999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59599999999999997,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":0.93000000000000005,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75800000000000001,"15min":0.78200000000000003},"min":{"1min":0.57299999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99299999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.873},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78300000000000003,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0489999999999999,"5min":1.246,"15min":6.1970000000000001},"last":0.83999999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.59999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":0.81200000000000006},"min":{"1min":0.61299999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.02,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.79000000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575041,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74520,"kb_used_data":3268,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297320,"statfs":{"total":96636764160,"available":96560455680,"internally_reserved":0,"allocated":3346432,"data_stored":1806170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":1.986,"15min":5.5259999999999998},"last":0.318},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.69999999999999996,"5min":2.371,"15min":5.4889999999999999},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.501,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.434}]},{"osd":3,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.878,"15min":4.7549999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.498}]},{"osd":4,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80300000000000005,"15min":0.79100000000000004},"min":{"1min":0.629,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.119,"5min":3.8849999999999998,"15min":4.734},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.70899999999999996,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":4.8079999999999998},"last":0.66900000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.66500000000000004,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":1.141,"5min":1.141,"15min":4.944},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.78800000000000003,"15min":0.79800000000000004},"min":{"1min":0.65100000000000002,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.246,"5min":1.246,"15min":4.9710000000000001},"last":0.66300000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.78400000000000003,"15min":0.80800000000000005},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":4.8470000000000004},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.65100000000000002,"5min":0.497,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85599999999999998,"15min":0.83399999999999996},"min":{"1min":0.68799999999999994,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.2150000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.67400000000000004,"5min":0.56899999999999995,"15min":0.505},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.9950000000000001},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:23:46.604 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:23:46.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:23:46.605 INFO:tasks.ceph.mds.c.smithi028.stderr:2024-09-13T17:23:46.315+0000 7f569d200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:23:46.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.771+0000 7f4c1b200640 1 Processor -- start 2024-09-13T17:23:46.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.771+0000 7f4c1b200640 1 -- start start 2024-09-13T17:23:46.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c1b200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c14159770 0x7f4c1406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c1b200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4c1406f3f0 0x7f4c14070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c1b200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1406f920 con 0x7f4c140add00 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c1b200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1415b290 con 0x7f4c14159770 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c1b200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c1415b570 con 0x7f4c1406f3f0 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c14159770 0x7f4c1406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c13e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4c14159770 0x7f4c1406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35172/0 (socket says 172.21.15.28:35172) 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c13e00640 1 -- 172.21.15.28:0/2883467733 learned_addr learned my addr 172.21.15.28:0/2883467733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.772+0000 7f4c19200640 1 --1- 172.21.15.28:0/2883467733 >> v1:172.21.15.28:6789/0 conn(0x7f4c140add00 0x7f4c14159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49330/0 (socket says 172.21.15.28:49330) 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.773+0000 7f4c0b400640 1 --2- 172.21.15.28:0/2883467733 >> v2:172.21.15.189:3300/0 conn(0x7f4c1406f3f0 0x7f4c14070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.773+0000 7f4c13e00640 1 -- 172.21.15.28:0/2883467733 >> v2:172.21.15.189:3300/0 conn(0x7f4c1406f3f0 msgr2=0x7f4c14070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.773+0000 7f4c13e00640 1 --2- 172.21.15.28:0/2883467733 >> v2:172.21.15.189:3300/0 conn(0x7f4c1406f3f0 0x7f4c14070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.773+0000 7f4c13e00640 1 -- 172.21.15.28:0/2883467733 >> v1:172.21.15.28:6789/0 conn(0x7f4c140add00 legacy=0x7f4c14159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.773+0000 7f4c13e00640 1 -- 172.21.15.28:0/2883467733 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1415b850 con 0x7f4c14159770 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.774+0000 7f4c13e00640 1 --2- 172.21.15.28:0/2883467733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c1406fd20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f4c1406b950 tx=0x7f4c0402a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d178b3aad0d7d7c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.774+0000 7f4c13400640 1 -- 172.21.15.28:0/2883467733 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04031600 con 0x7f4c14159770 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.774+0000 7f4c13400640 1 -- 172.21.15.28:0/2883467733 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0402d050 con 0x7f4c14159770 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.774+0000 7f4c13400640 1 -- 172.21.15.28:0/2883467733 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04031980 con 0x7f4c14159770 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 -- 172.21.15.28:0/2883467733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 msgr2=0x7f4c1406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 --2- 172.21.15.28:0/2883467733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c1406fd20 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f4c1406b950 tx=0x7f4c0402a620 comp rx=0 tx=0).stop 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 -- 172.21.15.28:0/2883467733 shutdown_connections 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 --2- 172.21.15.28:0/2883467733 >> v2:172.21.15.189:3300/0 conn(0x7f4c1406f3f0 0x7f4c14070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 --2- 172.21.15.28:0/2883467733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c1406fd20 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 -- 172.21.15.28:0/2883467733 >> 172.21.15.28:0/2883467733 conn(0x7f4c140b0270 msgr2=0x7f4c140b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 -- 172.21.15.28:0/2883467733 shutdown_connections 2024-09-13T17:23:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.775+0000 7f4c1b200640 1 -- 172.21.15.28:0/2883467733 wait complete. 2024-09-13T17:23:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.776+0000 7f4c1b200640 1 Processor -- start 2024-09-13T17:23:46.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.777+0000 7f4c1b200640 1 -- start start 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.777+0000 7f4c1b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1406f3f0 0x7f4c1414f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c1b200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c140add00 0x7f4c141f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c1b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c1b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f5b40 con 0x7f4c14159770 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c1b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f2de0 con 0x7f4c1406f3f0 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c1b200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141f30c0 con 0x7f4c140add00 2024-09-13T17:23:46.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35180/0 (socket says 172.21.15.28:35180) 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 -- 172.21.15.28:0/3981375699 learned_addr learned my addr 172.21.15.28:0/3981375699 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c13e00640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1406f3f0 0x7f4c1414f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c0b400640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c140add00 0x7f4c141f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 -- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c140add00 msgr2=0x7f4c141f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c140add00 0x7f4c141f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 -- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1406f3f0 msgr2=0x7f4c1414f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1406f3f0 0x7f4c1414f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.778+0000 7f4c19200640 1 -- 172.21.15.28:0/3981375699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c04009b30 con 0x7f4c14159770 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c19200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00c400 tx=0x7f4c0c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c0074e0 con 0x7f4c14159770 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0c007640 con 0x7f4c14159770 2024-09-13T17:23:46.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0c0049e0 con 0x7f4c14159770 2024-09-13T17:23:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c141f5db0 con 0x7f4c14159770 2024-09-13T17:23:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.779+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1414f6c0 con 0x7f4c14159770 2024-09-13T17:23:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.780+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c141f5fd0 con 0x7f4c14159770 2024-09-13T17:23:46.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.781+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4c0c004b40 con 0x7f4c14159770 2024-09-13T17:23:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.782+0000 7f4c10c00640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 0x7f4bf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.782+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4c0c088c40 con 0x7f4c14159770 2024-09-13T17:23:46.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.782+0000 7f4c13e00640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 0x7f4bf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:46.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.783+0000 7f4c13e00640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 0x7f4bf006d0c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4c04033000 tx=0x7f4c04009000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:46.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:46.786+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4c0c053ed0 con 0x7f4c14159770 2024-09-13T17:23:47.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.032+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4c1406cb10 con 0x7f4c14159770 2024-09-13T17:23:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.036+0000 7f4c10c00640 1 -- 172.21.15.28:0/3981375699 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v28) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4c0c0574f0 con 0x7f4c14159770 2024-09-13T17:23:47.037 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:23:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 msgr2=0x7f4bf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 0x7f4bf006d0c0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f4c04033000 tx=0x7f4c04009000 comp rx=0 tx=0).stop 2024-09-13T17:23:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 msgr2=0x7f4c141f62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c00c400 tx=0x7f4c0c00c8c0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 shutdown_connections 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4bf006ac00 0x7f4bf006d0c0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4c14159770 0x7f4c141f62d0 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4c140add00 0x7f4c141f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 --2- 172.21.15.28:0/3981375699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c1406f3f0 0x7f4c1414f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 >> 172.21.15.28:0/3981375699 conn(0x7f4c140b0270 msgr2=0x7f4c140afa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 shutdown_connections 2024-09-13T17:23:47.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.038+0000 7f4c1b200640 1 -- 172.21.15.28:0/3981375699 wait complete. 2024-09-13T17:23:47.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.210+0000 7f1aa6400640 1 Processor -- start 2024-09-13T17:23:47.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.210+0000 7f1aa6400640 1 -- start start 2024-09-13T17:23:47.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.211+0000 7f1aa6400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1aa01592a0 0x7f1aa015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.211+0000 7f1aa6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.211+0000 7f1aa6400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa006cc70 con 0x7f1aa006f970 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.211+0000 7f1aa6400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa015b0b0 con 0x7f1aa01592a0 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.211+0000 7f1aa6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0159d90 con 0x7f1aa006c2d0 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a96a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1aa006f970 0x7f1aa0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49346/0 (socket says 172.21.15.28:49346) 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a96a00640 1 -- 172.21.15.28:0/2805992076 learned_addr learned my addr 172.21.15.28:0/2805992076 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9f400640 1 --2- 172.21.15.28:0/2805992076 >> v2:172.21.15.189:3300/0 conn(0x7f1aa01592a0 0x7f1aa015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9fe00640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9ea00640 1 -- 172.21.15.28:0/2805992076 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977399075 0 0) 0x7f1aa006cc70 con 0x7f1aa006f970 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9ea00640 1 -- 172.21.15.28:0/2805992076 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a84003520 con 0x7f1aa006f970 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9f400640 1 -- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 msgr2=0x7f1aa006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9f400640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9f400640 1 -- 172.21.15.28:0/2805992076 >> v1:172.21.15.28:6789/0 conn(0x7f1aa006f970 legacy=0x7f1aa0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.212+0000 7f1a9f400640 1 -- 172.21.15.28:0/2805992076 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa00b2040 con 0x7f1aa01592a0 2024-09-13T17:23:47.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.213+0000 7f1a9f400640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa01592a0 0x7f1aa015ab70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=477318ff8d11483e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1a9ea00640 1 -- 172.21.15.28:0/2805992076 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90032440 con 0x7f1aa01592a0 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1a9ea00640 1 -- 172.21.15.28:0/2805992076 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a900325a0 con 0x7f1aa01592a0 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1a9ea00640 1 -- 172.21.15.28:0/2805992076 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a90032910 con 0x7f1aa01592a0 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 -- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa01592a0 msgr2=0x7f1aa015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa01592a0 0x7f1aa015ab70 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f1a900097a0 tx=0x7f1a9002a180 comp rx=0 tx=0).stop 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 -- 172.21.15.28:0/2805992076 shutdown_connections 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 --2- 172.21.15.28:0/2805992076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa01592a0 0x7f1aa015ab70 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 -- 172.21.15.28:0/2805992076 >> 172.21.15.28:0/2805992076 conn(0x7f1aa0069e80 msgr2=0x7f1aa006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 -- 172.21.15.28:0/2805992076 shutdown_connections 2024-09-13T17:23:47.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.214+0000 7f1aa6400640 1 -- 172.21.15.28:0/2805992076 wait complete. 2024-09-13T17:23:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.216+0000 7f1aa6400640 1 Processor -- start 2024-09-13T17:23:47.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.216+0000 7f1aa6400640 1 -- start start 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa01ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 0x7f1aa0154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa01eeb40 con 0x7f1aa01592a0 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0154280 con 0x7f1aa006f970 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1aa6400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa0153670 con 0x7f1aa006c2d0 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1a9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1a9fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35196/0 (socket says 172.21.15.28:35196) 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.217+0000 7f1a9fe00640 1 -- 172.21.15.28:0/4267043297 learned_addr learned my addr 172.21.15.28:0/4267043297 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9f400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa01ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a96a00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 0x7f1aa0154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 -- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 msgr2=0x7f1aa01ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa01ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 -- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 msgr2=0x7f1aa0154710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 0x7f1aa0154710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 -- 172.21.15.28:0/4267043297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a90009400 con 0x7f1aa01592a0 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9f400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa01ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a96a00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 0x7f1aa0154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:47.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a9fe00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800efc0 tx=0x7f1a9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98014480 con 0x7f1aa01592a0 2024-09-13T17:23:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a9800f040 con 0x7f1aa01592a0 2024-09-13T17:23:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa0153950 con 0x7f1aa01592a0 2024-09-13T17:23:47.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a98014820 con 0x7f1aa01592a0 2024-09-13T17:23:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.218+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa014e650 con 0x7f1aa01592a0 2024-09-13T17:23:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.220+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa0154410 con 0x7f1aa01592a0 2024-09-13T17:23:47.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.220+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1a98014980 con 0x7f1aa01592a0 2024-09-13T17:23:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.221+0000 7f1a97e00640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 0x7f1a7c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.221+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a98089fc0 con 0x7f1aa01592a0 2024-09-13T17:23:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.222+0000 7f1a9f400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 0x7f1a7c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:47.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.223+0000 7f1a9f400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 0x7f1a7c06d0c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1a90034000 tx=0x7f1a9002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:47.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.225+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1a98055100 con 0x7f1aa01592a0 2024-09-13T17:23:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.438+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aa00b3740 con 0x7f1aa01592a0 2024-09-13T17:23:47.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.439+0000 7f1a97e00640 1 -- 172.21.15.28:0/4267043297 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f1a98018090 con 0x7f1aa01592a0 2024-09-13T17:23:47.440 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:47.440 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:47.440 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 msgr2=0x7f1a7c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 0x7f1a7c06d0c0 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f1a90034000 tx=0x7f1a9002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 msgr2=0x7f1aa006b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7f1a9800efc0 tx=0x7f1a9800c490 comp rx=0 tx=0).stop 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 shutdown_connections 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1a7c06ac00 0x7f1a7c06d0c0 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1aa01592a0 0x7f1aa006b430 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1aa006f970 0x7f1aa0154710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 --2- 172.21.15.28:0/4267043297 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1aa006c2d0 0x7f1aa01ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 >> 172.21.15.28:0/4267043297 conn(0x7f1aa0069e80 msgr2=0x7f1aa00ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 shutdown_connections 2024-09-13T17:23:47.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:47.441+0000 7f1aa6400640 1 -- 172.21.15.28:0/4267043297 wait complete. 2024-09-13T17:23:47.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 181, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:23:41.629044+0000', 'modified': '2024-09-13T17:23:44.642067+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11916}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11916': {'gid': 11916, 'name': 'c', 'rank': 0, 'incarnation': 177, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6834/3884798333', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6832', 'nonce': 3884798333}, {'type': 'v1', 'addr': '172.21.15.28:6834', 'nonce': 3884798333}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:23:47.860 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:23:47.860 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T17:23:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.021+0000 7fc777600640 1 Processor -- start 2024-09-13T17:23:48.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.021+0000 7fc777600640 1 -- start start 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc777600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc77015ab60 0x7fc77006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc777600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc77006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc777600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc77006c8b0 con 0x7fc77015a780 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc777600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc77006cb90 con 0x7fc77015ab60 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc777600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700cc980 con 0x7fc770071490 2024-09-13T17:23:48.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc774c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc77015a780 0x7fc770070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49352/0 (socket says 172.21.15.28:49352) 2024-09-13T17:23:48.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.022+0000 7fc774c00640 1 -- 172.21.15.28:0/2939250945 learned_addr learned my addr 172.21.15.28:0/2939250945 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc767e00640 1 --2- 172.21.15.28:0/2939250945 >> v2:172.21.15.189:3300/0 conn(0x7fc77015ab60 0x7fc77006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc775600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc77006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480304622 0 0) 0x7fc77006c8b0 con 0x7fc77015a780 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc754003520 con 0x7fc77015a780 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1244672748 0 0) 0x7fc754003520 con 0x7fc77015a780 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 msgr2=0x7fc77006c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc77006c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 >> v2:172.21.15.189:3300/0 conn(0x7fc77015ab60 msgr2=0x7fc77006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 --2- 172.21.15.28:0/2939250945 >> v2:172.21.15.189:3300/0 conn(0x7fc77015ab60 0x7fc77006bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7700ccc60 con 0x7fc77015a780 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fc760001670 con 0x7fc77015a780 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 >> v1:172.21.15.28:6789/0 conn(0x7fc77015a780 legacy=0x7fc770070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754003970 0x7fc754003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc7540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc754004d00 0x7fc754005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77006cb90 con 0x7fc754004d00 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700cc980 con 0x7fc754003970 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.023+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc75400b980 con 0x7fc770071490 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc754004d00 0x7fc754005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc774c00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754003970 0x7fc754003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc767e00640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc7540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 -- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 msgr2=0x7fc7540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc770071490 0x7fc7540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 -- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754003970 msgr2=0x7fc754003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754003970 0x7fc754003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.024+0000 7fc775600640 1 -- 172.21.15.28:0/2939250945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc754003520 con 0x7fc754004d00 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.025+0000 7fc775600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc754004d00 0x7fc754005180 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fc768009fc0 tx=0x7fc7540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01d1faa4fa228a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.025+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768031640 con 0x7fc754004d00 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.025+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc76802d070 con 0x7fc754004d00 2024-09-13T17:23:48.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.025+0000 7fc76fe00640 1 -- 172.21.15.28:0/2939250945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680319a0 con 0x7fc754004d00 2024-09-13T17:23:48.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 -- 172.21.15.28:0/2939250945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc754004d00 msgr2=0x7fc754005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 --2- 172.21.15.28:0/2939250945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc754004d00 0x7fc754005180 secure :-1 s=READY pgs=2607 cs=0 l=1 rev1=1 crypto rx=0x7fc768009fc0 tx=0x7fc7540057f0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc775600640 1 -- 172.21.15.28:0/2939250945 reap_dead start 2024-09-13T17:23:48.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 -- 172.21.15.28:0/2939250945 shutdown_connections 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 -- 172.21.15.28:0/2939250945 >> 172.21.15.28:0/2939250945 conn(0x7fc7700b0270 msgr2=0x7fc7700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 -- 172.21.15.28:0/2939250945 shutdown_connections 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.026+0000 7fc777600640 1 -- 172.21.15.28:0/2939250945 wait complete. 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.028+0000 7fc777600640 1 Processor -- start 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.028+0000 7fc777600640 1 -- start start 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754004d00 0x7fc770156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 0x7fc7700c51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7700c5720 con 0x7fc77014ef80 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc770154960 con 0x7fc754004d00 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc777600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc770153d50 con 0x7fc7700c4d60 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35222/0 (socket says 172.21.15.28:35222) 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 -- 172.21.15.28:0/1616470362 learned_addr learned my addr 172.21.15.28:0/1616470362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc774c00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754004d00 0x7fc770156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc775600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 0x7fc7700c51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 -- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 msgr2=0x7fc7700c51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 0x7fc7700c51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 -- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754004d00 msgr2=0x7fc770156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754004d00 0x7fc770156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc767e00640 1 -- 172.21.15.28:0/1616470362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc770154030 con 0x7fc77014ef80 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.029+0000 7fc775600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 0x7fc7700c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc767e00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fc758011aa0 tx=0x7fc758011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc76d600640 1 -- 172.21.15.28:0/1616470362 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758009680 con 0x7fc77014ef80 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc76d600640 1 -- 172.21.15.28:0/1616470362 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75800fb80 con 0x7fc77014ef80 2024-09-13T17:23:48.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc76d600640 1 -- 172.21.15.28:0/1616470362 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758014440 con 0x7fc77014ef80 2024-09-13T17:23:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc770154310 con 0x7fc77014ef80 2024-09-13T17:23:48.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.030+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77014f400 con 0x7fc77014ef80 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.032+0000 7fc76d600640 1 -- 172.21.15.28:0/1616470362 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc7580145a0 con 0x7fc77014ef80 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.033+0000 7fc76d600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 0x7fc74c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.033+0000 7fc76d600640 1 -- 172.21.15.28:0/1616470362 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc758089d80 con 0x7fc77014ef80 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.033+0000 7fc774c00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 0x7fc74c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.033+0000 7fc77a3f6740 1 Processor -- start 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.034+0000 7fc77a3f6740 1 -- start start 2024-09-13T17:23:48.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.034+0000 7fc774c00640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 0x7fc74c06d0c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc760007cd0 tx=0x7fc760009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.034+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb17236ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.034+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.034+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb172840e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb1719fab0 con 0x55cb17233e90 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb1719d240 con 0x55cb172366d0 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb17198b30 con 0x55cb1725db80 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35232/0 (socket says 172.21.15.28:35232) 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 -- 172.21.15.28:0/1318332173 learned_addr learned my addr 172.21.15.28:0/1318332173 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc774c00640 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb17236ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc775600640 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb172840e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 -- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 msgr2=0x55cb172840e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb172840e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 -- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 msgr2=0x55cb17236ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb17236ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 -- 172.21.15.28:0/1318332173 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb17163bd0 con 0x55cb17233e90 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.035+0000 7fc767e00640 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fc758055080 tx=0x7fc758058b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649642c3f79b4814 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.037+0000 7fc765600640 1 -- 172.21.15.28:0/1318332173 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758056f20 con 0x55cb17233e90 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.037+0000 7fc765600640 1 -- 172.21.15.28:0/1318332173 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc758055290 con 0x55cb17233e90 2024-09-13T17:23:48.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.037+0000 7fc765600640 1 -- 172.21.15.28:0/1318332173 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758059020 con 0x55cb17233e90 2024-09-13T17:23:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 -- 172.21.15.28:0/1318332173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 msgr2=0x55cb17283ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7fc758055080 tx=0x7fc758058b80 comp rx=0 tx=0).stop 2024-09-13T17:23:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 -- 172.21.15.28:0/1318332173 shutdown_connections 2024-09-13T17:23:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb172840e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb17233e90 0x55cb17283ba0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/1318332173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb17236ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 -- 172.21.15.28:0/1318332173 >> 172.21.15.28:0/1318332173 conn(0x55cb17232e00 msgr2=0x55cb1727b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 -- 172.21.15.28:0/1318332173 shutdown_connections 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.038+0000 7fc77a3f6740 1 -- 172.21.15.28:0/1318332173 wait complete. 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.039+0000 7fc77a3f6740 1 -- start start 2024-09-13T17:23:48.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.039+0000 7fc77a3f6740 1 Processor -- start 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17278110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 0x55cb1727d360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb1713a260 con 0x55cb172366d0 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb17139f00 con 0x55cb1727cee0 2024-09-13T17:23:48.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb16efff20 con 0x55cb1725db80 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.040+0000 7fc77a3f6740 10 client.0 _ll_register_callbacks cb 0x55cb171921a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35236/0 (socket says 172.21.15.28:35236) 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 -- 172.21.15.28:0/2304093875 learned_addr learned my addr 172.21.15.28:0/2304093875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc775600640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 0x55cb1727d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc767e00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17278110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 -- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 msgr2=0x55cb17278110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17278110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 -- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 msgr2=0x55cb1727d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 0x55cb1727d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 -- 172.21.15.28:0/2304093875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77014f400 con 0x55cb172366d0 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc775600640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 0x55cb1727d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:48.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc767e00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17278110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc774c00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fc760015a10 tx=0x7fc760001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c457793c0456193b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.041+0000 7fc75e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760014a60 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc76001a040 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb171c2cd0 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc760014bc0 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 10 client.12057 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x55cb171971b0 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cb171c2cd0 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc760014d20 con 0x55cb172366d0 2024-09-13T17:23:48.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 10 client.12057 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 10 client.12057 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 20 client.12057 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb17256280 0x55cb17256660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 4 client.12057 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.042+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cb16e8c3c0 con 0x55cb17256280 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.043+0000 7fc767e00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb17256280 0x55cb17256660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.043+0000 7fc767e00640 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb17256280 0x55cb17256660 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302280fe91fc39bd server_cookie=82bb7718b8244839 in_seq=0 out_seq=0 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.043+0000 7fc75e000640 10 client.12057 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc75e000640 1 -- 172.21.15.28:0/2304093875 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55cb16e8c3c0 con 0x55cb17256280 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc75e000640 10 client.12057 handle_command_reply: tid=0 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 client.12057 shutdown 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 msgr2=0x55cb172344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 secure :-1 s=READY pgs=2610 cs=0 l=1 rev1=1 crypto rx=0x7fc760015a10 tx=0x7fc760001a00 comp rx=0 tx=0).stop 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 shutdown_connections 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb17256280 0x55cb17256660 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1727cee0 0x55cb1727d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17278110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2304093875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172344c0 unknown :-1 s=CLOSED pgs=2610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 >> 172.21.15.28:0/2304093875 conn(0x55cb17233e90 msgr2=0x55cb172795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc75e000640 0 client.12057 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 shutdown_connections 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.044+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2304093875 wait complete. 2024-09-13T17:23:48.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.045+0000 7fc77a3f6740 20 client.12057 trim_cache size 0 max 16384 2024-09-13T17:23:48.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.061+0000 7fc77a3f6740 1 Processor -- start 2024-09-13T17:23:48.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.061+0000 7fc77a3f6740 1 -- start start 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb1725b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17236170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb17258a80 con 0x55cb1727cee0 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb172577b0 con 0x55cb172366d0 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb17236b50 con 0x55cb1725db80 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc775600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb1725b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc775600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35246/0 (socket says 172.21.15.28:35246) 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc775600640 1 -- 172.21.15.28:0/2691539362 learned_addr learned my addr 172.21.15.28:0/2691539362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc767e00640 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17236170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 -- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 msgr2=0x55cb17236170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17236170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 -- 172.21.15.28:0/2691539362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 msgr2=0x55cb1727a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc774c00640 1 -- 172.21.15.28:0/2691539362 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cb171cc170 con 0x55cb172366d0 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.062+0000 7fc775600640 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:48.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc774c00640 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb1725b7f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc7600078c0 tx=0x7fc760023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=801bbafbb8b50e26 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc766000640 1 -- 172.21.15.28:0/2691539362 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760014c60 con 0x55cb172366d0 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc766000640 1 -- 172.21.15.28:0/2691539362 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc760014dc0 con 0x55cb172366d0 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc766000640 1 -- 172.21.15.28:0/2691539362 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760022900 con 0x55cb172366d0 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 msgr2=0x55cb1725b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.063+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb1725b7f0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc7600078c0 tx=0x7fc760023000 comp rx=0 tx=0).stop 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2691539362 shutdown_connections 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb1727cee0 0x55cb1727a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1725db80 0x55cb17236170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/2691539362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb172366d0 0x55cb1725b7f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2691539362 >> 172.21.15.28:0/2691539362 conn(0x55cb17233e90 msgr2=0x55cb17279560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2691539362 shutdown_connections 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 -- 172.21.15.28:0/2691539362 wait complete. 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 -- start start 2024-09-13T17:23:48.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.064+0000 7fc77a3f6740 1 Processor -- start 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 0x55cb172540f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 0x55cb17254630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb1725fd80 con 0x55cb172366d0 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb17277500 con 0x55cb1725db80 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb17277750 con 0x55cb1727cee0 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc77a3f6740 10 client.0 _ll_register_callbacks cb 0x7fc760015660 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.065+0000 7fc774c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35256/0 (socket says 172.21.15.28:35256) 2024-09-13T17:23:48.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 -- 172.21.15.28:0/3822051875 learned_addr learned my addr 172.21.15.28:0/3822051875 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc767e00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 0x55cb172540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 -- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 msgr2=0x55cb17254630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc775600640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 0x55cb17254630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 0x55cb17254630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 -- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 msgr2=0x55cb172540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 0x55cb172540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 -- 172.21.15.28:0/3822051875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc754003520 con 0x55cb172366d0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc767e00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 0x55cb172540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc775600640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 0x55cb17254630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc774c00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fc760007650 tx=0x7fc760016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c295d3dceae19a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.066+0000 7fc75e000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc760001a00 con 0x55cb172366d0 2024-09-13T17:23:48.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cb172582a0 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc76001a070 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc76002c450 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 10 client.12063 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x55cb171971b0 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55cb172582a0 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc76002c5b0 con 0x55cb172366d0 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 10 client.12063 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 10 client.12063 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 20 client.12063 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb1725c470 0x55cb1725c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 4 client.12063 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fc758055290 con 0x55cb1725c470 2024-09-13T17:23:48.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.067+0000 7fc767e00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb1725c470 0x55cb1725c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc767e00640 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb1725c470 0x55cb1725c850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ffc928e1e075fe server_cookie=ccbaf898db4589b8 in_seq=0 out_seq=0 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc75e000640 10 client.12063 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc75e000640 1 -- 172.21.15.28:0/3822051875 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7fc758055290 con 0x55cb1725c470 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc75e000640 10 client.12063 handle_command_reply: tid=0 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 client.12063 shutdown 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 msgr2=0x55cb172ce520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7fc760007650 tx=0x7fc760016000 comp rx=0 tx=0).stop 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 shutdown_connections 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x55cb1725c470 0x55cb1725c850 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb1727cee0 0x55cb17254630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb1725db80 0x55cb172540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 --2- 172.21.15.28:0/3822051875 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb172366d0 0x55cb172ce520 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc75e000640 0 client.12063 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:48.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.068+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 >> 172.21.15.28:0/3822051875 conn(0x55cb17233e90 msgr2=0x55cb1727ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 shutdown_connections 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc77a3f6740 1 -- 172.21.15.28:0/3822051875 wait complete. 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc77a3f6740 20 client.12063 trim_cache size 0 max 16384 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 11916, 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:23:48.074 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.437186874, 2024-09-13T17:23:48.075 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 181, 2024-09-13T17:23:48.075 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 219, 2024-09-13T17:23:48.075 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 219, 2024-09-13T17:23:48.075 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 17.419710963 2024-09-13T17:23:48.075 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 msgr2=0x7fc74c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 0x7fc74c06d0c0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc760007cd0 tx=0x7fc760009040 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 msgr2=0x7fc770154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7fc758011aa0 tx=0x7fc758011f60 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 shutdown_connections 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74c06ac00 0x7fc74c06d0c0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc7700c4d60 0x7fc7700c51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc77014ef80 0x7fc770154df0 unknown :-1 s=CLOSED pgs=2608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 --2- 172.21.15.28:0/1616470362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc754004d00 0x7fc770156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.069+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 >> 172.21.15.28:0/1616470362 conn(0x7fc7700b0270 msgr2=0x7fc770070910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.070+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 shutdown_connections 2024-09-13T17:23:48.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.070+0000 7fc777600640 1 -- 172.21.15.28:0/1616470362 wait complete. 2024-09-13T17:23:48.086 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:23:48.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.240+0000 7fcaa0a00640 1 Processor -- start 2024-09-13T17:23:48.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.240+0000 7fcaa0a00640 1 -- start start 2024-09-13T17:23:48.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fcaa0a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca9c15c750 0x7fca9c0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fcaa0a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c0d0d80 con 0x7fca9c0d1d10 2024-09-13T17:23:48.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fcaa0a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9c0d0960 con 0x7fca9c0b4ad0 2024-09-13T17:23:48.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fcaa0a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca9c06aa60 con 0x7fca9c15c750 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fca9a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca9c15c750 0x7fca9c0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fca99e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fca9c0d1d10 0x7fca9c0b43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49366/0 (socket says 172.21.15.28:49366) 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.241+0000 7fca99e00640 1 -- 172.21.15.28:0/415178843 learned_addr learned my addr 172.21.15.28:0/415178843 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.242+0000 7fca9a800640 1 -- 172.21.15.28:0/415178843 >> v1:172.21.15.189:6789/0 conn(0x7fca9c0b4ad0 legacy=0x7fca9c069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.242+0000 7fca9a800640 1 -- 172.21.15.28:0/415178843 >> v1:172.21.15.28:6789/0 conn(0x7fca9c0d1d10 legacy=0x7fca9c0b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.242+0000 7fca9a800640 1 -- 172.21.15.28:0/415178843 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9c06ad40 con 0x7fca9c15c750 2024-09-13T17:23:48.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.243+0000 7fca9a800640 1 --2- 172.21.15.28:0/415178843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c15c750 0x7fca9c0d17d0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0097a0 tx=0x7fca8c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5887f6972ab2c1b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.243+0000 7fca99400640 1 -- 172.21.15.28:0/415178843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c02ae50 con 0x7fca9c15c750 2024-09-13T17:23:48.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.243+0000 7fca99400640 1 -- 172.21.15.28:0/415178843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c032440 con 0x7fca9c15c750 2024-09-13T17:23:48.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.243+0000 7fca99400640 1 -- 172.21.15.28:0/415178843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca8c0327d0 con 0x7fca9c15c750 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 -- 172.21.15.28:0/415178843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c15c750 msgr2=0x7fca9c0d17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/415178843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c15c750 0x7fca9c0d17d0 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fca8c0097a0 tx=0x7fca8c02a140 comp rx=0 tx=0).stop 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 -- 172.21.15.28:0/415178843 shutdown_connections 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/415178843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c15c750 0x7fca9c0d17d0 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 -- 172.21.15.28:0/415178843 >> 172.21.15.28:0/415178843 conn(0x7fca9c0cf5a0 msgr2=0x7fca9c0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 -- 172.21.15.28:0/415178843 shutdown_connections 2024-09-13T17:23:48.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.244+0000 7fcaa0a00640 1 -- 172.21.15.28:0/415178843 wait complete. 2024-09-13T17:23:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.245+0000 7fcaa0a00640 1 Processor -- start 2024-09-13T17:23:48.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.245+0000 7fcaa0a00640 1 -- start start 2024-09-13T17:23:48.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0b4ad0 0x7fca9c14de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 0x7fca9c1f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c156fb0 con 0x7fca9c1565c0 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c15a330 con 0x7fca9c15c750 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.246+0000 7fcaa0a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9c150210 con 0x7fca9c0b4ad0 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35274/0 (socket says 172.21.15.28:35274) 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 -- 172.21.15.28:0/3555265500 learned_addr learned my addr 172.21.15.28:0/3555265500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9a800640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0b4ad0 0x7fca9c14de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca99e00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 0x7fca9c1f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 -- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0b4ad0 msgr2=0x7fca9c14de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0b4ad0 0x7fca9c14de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 -- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 msgr2=0x7fca9c1f3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 0x7fca9c1f3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca9b200640 1 -- 172.21.15.28:0/3555265500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca8c009400 con 0x7fca9c1565c0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.247+0000 7fca99e00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 0x7fca9c1f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fca9b200640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fca9400b970 tx=0x7fca9400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca9400cae0 con 0x7fca9c1565c0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca9400cc40 con 0x7fca9c1565c0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca94014680 con 0x7fca9c1565c0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9c1504f0 con 0x7fca9c1565c0 2024-09-13T17:23:48.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.248+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9c159be0 con 0x7fca9c1565c0 2024-09-13T17:23:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.249+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9c06a620 con 0x7fca9c1565c0 2024-09-13T17:23:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.250+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca9401d3b0 con 0x7fca9c1565c0 2024-09-13T17:23:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.251+0000 7fca92a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 0x7fca7c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:48.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.251+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca94089090 con 0x7fca9c1565c0 2024-09-13T17:23:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.251+0000 7fca9a800640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 0x7fca7c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:48.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.252+0000 7fca9a800640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 0x7fca7c06d0c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fca8c034000 tx=0x7fca8c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:48.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.255+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fca94054320 con 0x7fca9c1565c0 2024-09-13T17:23:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.468+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fca9c0d0b60 con 0x7fca9c1565c0 2024-09-13T17:23:48.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.469+0000 7fca92a00640 1 -- 172.21.15.28:0/3555265500 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v181) v1 ==== 53+0+73 (secure 0 0 0) 0x7fca94057940 con 0x7fca9c1565c0 2024-09-13T17:23:48.471 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 msgr2=0x7fca7c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 0x7fca7c06d0c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fca8c034000 tx=0x7fca8c02e000 comp rx=0 tx=0).stop 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 msgr2=0x7fca9c156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fca9400b970 tx=0x7fca9400be30 comp rx=0 tx=0).stop 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 shutdown_connections 2024-09-13T17:23:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca7c06ac00 0x7fca7c06d0c0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca9c1565c0 0x7fca9c156a70 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca9c15c750 0x7fca9c1f3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 --2- 172.21.15.28:0/3555265500 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca9c0b4ad0 0x7fca9c14de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 >> 172.21.15.28:0/3555265500 conn(0x7fca9c0cf5a0 msgr2=0x7fca9c157420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 shutdown_connections 2024-09-13T17:23:48.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:48.471+0000 7fcaa0a00640 1 -- 172.21.15.28:0/3555265500 wait complete. 2024-09-13T17:23:48.488 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:23:48.488 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:23:48.488 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:23:48.488 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:23:48.489 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:48.489 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:23:48.489 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:23:48.489 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:23:48.489 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:23:48.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.490 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:23:48.526 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41753sec preferred_lft 41753sec 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:23:48.527 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:23:48.528 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:23:48.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.529 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:23:48.619 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:23:48.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.622 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:23:48.658 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:23:48.658 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:23:48.659 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:23:48.659 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:23:48.659 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:23:48.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:23:48.659 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:23:48.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.804 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:23:48.840 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:23:48.840 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:23:48.840 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:23:48.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:48.840 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:23:48.936 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:23:48.936 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:48.971 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:23:48.972 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:23:48.972 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:49.048 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:23:49.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:49.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:23:49.176 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:23:49.176 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:23:49.203 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:23:49.204 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:23:49.204 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:23:49.204 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:23:49.204 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:23:49.206 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:23:49.206 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:23:49.206 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:23:49.206 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:23:49.206 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:23:49.209 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:23:49.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:49.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:49.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:49.268 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:23:49.312 INFO:teuthology.orchestra.run.smithi028.stdout:client12075 2024-09-13T17:23:49.312 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:49.313 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:49.388 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:49.389 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:49.464 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T17:23:49.464 DEBUG:teuthology.orchestra.run.smithi028:> import os 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> import uuid 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> n = 1000 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = False 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> if 0 > 0: 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> create_hard_links = True 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> path = "/home/ubuntu/cephtest/mnt.0/scrub-new-files/file" 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> dpath = os.path.dirname(path) 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> fnameprefix = os.path.basename(path) 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> os.makedirs(dpath, exist_ok=True) 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> try: 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-09-13T17:23:49.465 DEBUG:teuthology.orchestra.run.smithi028:> 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> for i in range(n): 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> with open(fpath, '"'"'w'"'"') as f: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> f.write(f"{i}") 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> f.flush() 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(f.fileno()) 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> os.unlink(fpath) 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> if create_hard_links: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> for j in range(0): 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> if False: 2024-09-13T17:23:49.466 DEBUG:teuthology.orchestra.run.smithi028:> os.fsync(dirfd) 2024-09-13T17:23:49.467 DEBUG:teuthology.orchestra.run.smithi028:> finally: 2024-09-13T17:23:49.467 DEBUG:teuthology.orchestra.run.smithi028:> os.close(dirfd) 2024-09-13T17:23:49.467 DEBUG:teuthology.orchestra.run.smithi028:> ' 2024-09-13T17:23:50.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.15:0 scrub status 2024-09-13T17:23:50.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.598+0000 7fce69000640 1 Processor -- start 2024-09-13T17:23:50.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.599+0000 7fce69000640 1 -- start start 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce69000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fce64073280 0x7fce640b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce69000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce640ce250 con 0x7fce64072e40 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce69000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6406cec0 con 0x7fce6415aa20 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce69000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce640cf240 con 0x7fce64073280 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce5be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fce64072e40 0x7fce640b2410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49376/0 (socket says 172.21.15.28:49376) 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce5be00640 1 -- 172.21.15.28:0/4127496500 learned_addr learned my addr 172.21.15.28:0/4127496500 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:50.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce63e00640 1 --2- 172.21.15.28:0/4127496500 >> v2:172.21.15.28:3300/0 conn(0x7fce64073280 0x7fce640b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.600+0000 7fce62a00640 1 -- 172.21.15.28:0/4127496500 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255689210 0 0) 0x7fce640ce250 con 0x7fce64072e40 2024-09-13T17:23:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.601+0000 7fce62a00640 1 -- 172.21.15.28:0/4127496500 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce48003520 con 0x7fce64072e40 2024-09-13T17:23:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.601+0000 7fce63e00640 1 -- 172.21.15.28:0/4127496500 >> v1:172.21.15.189:6789/0 conn(0x7fce6415aa20 legacy=0x7fce6406c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.601+0000 7fce63e00640 1 -- 172.21.15.28:0/4127496500 >> v1:172.21.15.28:6789/0 conn(0x7fce64072e40 legacy=0x7fce640b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.601+0000 7fce63e00640 1 -- 172.21.15.28:0/4127496500 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640ce480 con 0x7fce64073280 2024-09-13T17:23:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce63e00640 1 --2- 172.21.15.28:0/4127496500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce64073280 0x7fce640b2b20 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009d90 tx=0x7fce5c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2a00dfac99affa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce62a00640 1 -- 172.21.15.28:0/4127496500 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c007740 con 0x7fce64073280 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce62a00640 1 -- 172.21.15.28:0/4127496500 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c02d050 con 0x7fce64073280 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce62a00640 1 -- 172.21.15.28:0/4127496500 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c007ac0 con 0x7fce64073280 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce69000640 1 -- 172.21.15.28:0/4127496500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce64073280 msgr2=0x7fce640b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.602+0000 7fce69000640 1 --2- 172.21.15.28:0/4127496500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce64073280 0x7fce640b2b20 secure :-1 s=READY pgs=2617 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009d90 tx=0x7fce5c02a6f0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.603+0000 7fce69000640 1 -- 172.21.15.28:0/4127496500 shutdown_connections 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.603+0000 7fce69000640 1 --2- 172.21.15.28:0/4127496500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce64073280 0x7fce640b2b20 unknown :-1 s=CLOSED pgs=2617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.603+0000 7fce69000640 1 -- 172.21.15.28:0/4127496500 >> 172.21.15.28:0/4127496500 conn(0x7fce640cbec0 msgr2=0x7fce640cc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.603+0000 7fce69000640 1 -- 172.21.15.28:0/4127496500 shutdown_connections 2024-09-13T17:23:50.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.603+0000 7fce69000640 1 -- 172.21.15.28:0/4127496500 wait complete. 2024-09-13T17:23:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.604+0000 7fce69000640 1 Processor -- start 2024-09-13T17:23:50.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.604+0000 7fce69000640 1 -- start start 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce64072e40 0x7fce640b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce64073280 0x7fce6415d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce640b53f0 con 0x7fce6414bec0 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce640c9b30 con 0x7fce64073280 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce69000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce640c9e10 con 0x7fce64072e40 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.605+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35306/0 (socket says 172.21.15.28:35306) 2024-09-13T17:23:50.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 -- 172.21.15.28:0/1399486764 learned_addr learned my addr 172.21.15.28:0/1399486764 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce5be00640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce64073280 0x7fce6415d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63400640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce64072e40 0x7fce640b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 -- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce64072e40 msgr2=0x7fce640b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce64072e40 0x7fce640b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 -- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce64073280 msgr2=0x7fce6415d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce64073280 0x7fce6415d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 -- 172.21.15.28:0/1399486764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5c009a70 con 0x7fce6414bec0 2024-09-13T17:23:50.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.606+0000 7fce63e00640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009eb0 tx=0x7fce5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.607+0000 7fce5b400640 1 -- 172.21.15.28:0/1399486764 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c03c830 con 0x7fce6414bec0 2024-09-13T17:23:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.607+0000 7fce5b400640 1 -- 172.21.15.28:0/1399486764 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c03c990 con 0x7fce6414bec0 2024-09-13T17:23:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.607+0000 7fce5b400640 1 -- 172.21.15.28:0/1399486764 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c03cd00 con 0x7fce6414bec0 2024-09-13T17:23:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.607+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce640ca0f0 con 0x7fce6414bec0 2024-09-13T17:23:50.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.607+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce640ca8f0 con 0x7fce6414bec0 2024-09-13T17:23:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.609+0000 7fce5b400640 1 -- 172.21.15.28:0/1399486764 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fce5c031400 con 0x7fce6414bec0 2024-09-13T17:23:50.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.610+0000 7fce5b400640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 0x7fce4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.610+0000 7fce5b400640 1 -- 172.21.15.28:0/1399486764 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce5c0ab070 con 0x7fce6414bec0 2024-09-13T17:23:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.610+0000 7fce6d5ee740 1 Processor -- start 2024-09-13T17:23:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.610+0000 7fce63400640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 0x7fce4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.610+0000 7fce6d5ee740 1 -- start start 2024-09-13T17:23:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce63400640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 0x7fce4006d0c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fce54000f60 tx=0x7fce54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba17ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cb590 0x557ba181b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba16d46a0 con 0x557ba17f5280 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba1737cc0 con 0x557ba17cb590 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ba1735450 con 0x557ba17cddd0 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.611+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35314/0 (socket says 172.21.15.28:35314) 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 -- 172.21.15.28:0/61803795 learned_addr learned my addr 172.21.15.28:0/61803795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce5be00640 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cb590 0x557ba181b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63400640 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba17ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 -- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 msgr2=0x557ba17ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba17ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 -- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cb590 msgr2=0x557ba181b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cb590 0x557ba181b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 -- 172.21.15.28:0/61803795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ba17012e0 con 0x557ba17f5280 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce63e00640 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fce5c079ca0 tx=0x7fce5c0443c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19b607279447a40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.612+0000 7fce53e00640 1 -- 172.21.15.28:0/61803795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c0800c0 con 0x557ba17f5280 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce53e00640 1 -- 172.21.15.28:0/61803795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c07ddd0 con 0x557ba17f5280 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce53e00640 1 -- 172.21.15.28:0/61803795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c082390 con 0x557ba17f5280 2024-09-13T17:23:50.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 -- 172.21.15.28:0/61803795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 msgr2=0x557ba181b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fce5c079ca0 tx=0x7fce5c0443c0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 -- 172.21.15.28:0/61803795 shutdown_connections 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17f5280 0x557ba181b7e0 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cb590 0x557ba181b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/61803795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba17ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 -- 172.21.15.28:0/61803795 >> 172.21.15.28:0/61803795 conn(0x557ba17ca500 msgr2=0x557ba18129b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 -- 172.21.15.28:0/61803795 shutdown_connections 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.613+0000 7fce6d5ee740 1 -- 172.21.15.28:0/61803795 wait complete. 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.614+0000 7fce6d5ee740 1 -- start start 2024-09-13T17:23:50.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.614+0000 7fce6d5ee740 1 Processor -- start 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba180fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 0x557ba17c8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba1730d40 con 0x557ba17e1b50 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba16d2280 con 0x557ba17f5280 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ba16d1f20 con 0x557ba17cddd0 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce6d5ee740 10 client.0 _ll_register_callbacks cb 0x557ba172a3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35326/0 (socket says 172.21.15.28:35326) 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.615+0000 7fce63e00640 1 -- 172.21.15.28:0/3604004695 learned_addr learned my addr 172.21.15.28:0/3604004695 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63400640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba180fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce5be00640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 0x557ba17c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 -- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 msgr2=0x557ba180fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba180fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 -- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 msgr2=0x557ba17c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 0x557ba17c8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 -- 172.21.15.28:0/3604004695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce640ca8f0 con 0x557ba17e1b50 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce5be00640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 0x557ba17c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce63e00640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fce5c079ee0 tx=0x7fce5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d117cfe3a0677aa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce50c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5c082390 con 0x557ba17e1b50 2024-09-13T17:23:50.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce5c07a200 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557ba14244e0 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.616+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce5c0a9260 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 10 client.12090 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x557ba172f3c0 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557ba14244e0 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fce5c04d070 con 0x557ba17e1b50 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 10 client.12090 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 10 client.12090 resolve_mds: resolved 15:0 to role '15:0' aka daemon mds.c 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 20 client.12090 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba17e0710 0x557ba17e0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 4 client.12090 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557ba14357e0 con 0x557ba17e0710 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.617+0000 7fce63400640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba17e0710 0x557ba17e0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.618+0000 7fce63400640 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba17e0710 0x557ba17e0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148b8857571a8685 server_cookie=a68cf8d24817216 in_seq=0 out_seq=0 2024-09-13T17:23:50.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.618+0000 7fce50c00640 10 client.12090 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.618+0000 7fce50c00640 1 -- 172.21.15.28:0/3604004695 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x557ba14357e0 con 0x557ba17e0710 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.618+0000 7fce50c00640 10 client.12090 handle_command_reply: tid=0 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 client.12090 shutdown 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 msgr2=0x557ba17c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 secure :-1 s=READY pgs=2620 cs=0 l=1 rev1=1 crypto rx=0x7fce5c079ee0 tx=0x7fce5c009750 comp rx=0 tx=0).stop 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 shutdown_connections 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba17e0710 0x557ba17e0af0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17c8740 unknown :-1 s=CLOSED pgs=2620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17f5280 0x557ba17c8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/3604004695 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17cddd0 0x557ba180fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce50c00640 0 client.12090 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 >> 172.21.15.28:0/3604004695 conn(0x557ba17cb590 msgr2=0x557ba1810cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 shutdown_connections 2024-09-13T17:23:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 1 -- 172.21.15.28:0/3604004695 wait complete. 2024-09-13T17:23:50.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.619+0000 7fce6d5ee740 20 client.12090 trim_cache size 0 max 16384 2024-09-13T17:23:50.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.634+0000 7fce6d5ee740 1 Processor -- start 2024-09-13T17:23:50.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.634+0000 7fce6d5ee740 1 -- start start 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17e1b50 0x557ba1812a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba1810c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba17fa190 con 0x557ba17cddd0 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba1498040 con 0x557ba17e1b50 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ba17e2590 con 0x557ba17f5280 2024-09-13T17:23:50.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35342/0 (socket says 172.21.15.28:35342) 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 -- 172.21.15.28:0/958269307 learned_addr learned my addr 172.21.15.28:0/958269307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 -- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 msgr2=0x557ba1810c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba1810c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 -- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17e1b50 msgr2=0x557ba1812a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17e1b50 0x557ba1812a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.635+0000 7fce63400640 1 -- 172.21.15.28:0/958269307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557ba16d1610 con 0x557ba17cddd0 2024-09-13T17:23:50.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce63400640 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e530 tx=0x7fce54011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ad14341562412b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce58c00640 1 -- 172.21.15.28:0/958269307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400e8a0 con 0x557ba17cddd0 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce58c00640 1 -- 172.21.15.28:0/958269307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5400ea00 con 0x557ba17cddd0 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce58c00640 1 -- 172.21.15.28:0/958269307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5400eb70 con 0x557ba17cddd0 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 -- 172.21.15.28:0/958269307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 msgr2=0x557ba18124d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fce5400e530 tx=0x7fce54011000 comp rx=0 tx=0).stop 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 -- 172.21.15.28:0/958269307 shutdown_connections 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba1810c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17e1b50 0x557ba1812a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/958269307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17cddd0 0x557ba18124d0 unknown :-1 s=CLOSED pgs=2621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.636+0000 7fce6d5ee740 1 -- 172.21.15.28:0/958269307 >> 172.21.15.28:0/958269307 conn(0x557ba17cb590 msgr2=0x557ba17ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.637+0000 7fce6d5ee740 1 -- 172.21.15.28:0/958269307 shutdown_connections 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.637+0000 7fce6d5ee740 1 -- 172.21.15.28:0/958269307 wait complete. 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.637+0000 7fce6d5ee740 1 -- start start 2024-09-13T17:23:50.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.637+0000 7fce6d5ee740 1 Processor -- start 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cddd0 0x557ba1866a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba18124d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba187ef40 con 0x557ba17e1b50 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557ba187db10 con 0x557ba17cddd0 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557ba1866fb0 con 0x557ba17f5280 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce6d5ee740 10 client.0 _ll_register_callbacks cb 0x7fce4c007d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.638+0000 7fce5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35346/0 (socket says 172.21.15.28:35346) 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 -- 172.21.15.28:0/2429376054 learned_addr learned my addr 172.21.15.28:0/2429376054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce63400640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cddd0 0x557ba1866a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce63e00640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba18124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 -- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 msgr2=0x557ba18124d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba18124d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 -- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cddd0 msgr2=0x557ba1866a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cddd0 0x557ba1866a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 -- 172.21.15.28:0/2429376054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5400f040 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce5be00640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0011b0 tx=0x7fce4c007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39183853e63a13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce50c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.639+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4c001d30 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce4c016450 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce5c07ddd0 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce4c016610 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 10 client.12096 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x557ba16f6b20 con 0x557ba17e1b50 2024-09-13T17:23:50.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce5c07ddd0 con 0x557ba17e1b50 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fce4c016770 con 0x557ba17e1b50 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 10 client.12096 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 10 client.12096 resolve_mds: resolved 15:0 to role '15:0' aka daemon mds.c 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 20 client.12096 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba1812a10 0x557ba17c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 4 client.12096 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557ba1488e30 con 0x557ba1812a10 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.640+0000 7fce63400640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba1812a10 0x557ba17c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce63400640 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba1812a10 0x557ba17c75b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b94c85bfb8b5950 server_cookie=b44f0c7d92409aeb in_seq=0 out_seq=0 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce50c00640 10 client.12096 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce50c00640 1 -- 172.21.15.28:0/2429376054 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557ba1488e30 con 0x557ba1812a10 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce50c00640 10 client.12096 handle_command_reply: tid=0 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 client.12096 shutdown 2024-09-13T17:23:50.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 msgr2=0x557ba17de620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fce4c0011b0 tx=0x7fce4c007830 comp rx=0 tx=0).stop 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 shutdown_connections 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x557ba1812a10 0x557ba17c75b0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557ba17f5280 0x557ba18124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557ba17e1b50 0x557ba17de620 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 --2- 172.21.15.28:0/2429376054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557ba17cddd0 0x557ba1866a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 >> 172.21.15.28:0/2429376054 conn(0x557ba17cb590 msgr2=0x557ba17e1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.641+0000 7fce50c00640 0 client.12096 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 shutdown_connections 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce6d5ee740 1 -- 172.21.15.28:0/2429376054 wait complete. 2024-09-13T17:23:50.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce6d5ee740 20 client.12096 trim_cache size 0 max 16384 2024-09-13T17:23:50.650 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:23:50.650 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:23:50.650 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:23:50.650 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 msgr2=0x7fce4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 0x7fce4006d0c0 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fce54000f60 tx=0x7fce54009040 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 msgr2=0x7fce640c8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fce5c009eb0 tx=0x7fce5c004a40 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 shutdown_connections 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fce4006ac00 0x7fce4006d0c0 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce6414bec0 0x7fce640c8790 unknown :-1 s=CLOSED pgs=2618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce64073280 0x7fce6415d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 --2- 172.21.15.28:0/1399486764 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fce64072e40 0x7fce640b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 >> 172.21.15.28:0/1399486764 conn(0x7fce640cbec0 msgr2=0x7fce64159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.642+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 shutdown_connections 2024-09-13T17:23:50.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.643+0000 7fce69000640 1 -- 172.21.15.28:0/1399486764 wait complete. 2024-09-13T17:23:50.663 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:23:50.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.15:0 damage ls 2024-09-13T17:23:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.864+0000 7f64a0400640 1 Processor -- start 2024-09-13T17:23:50.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.864+0000 7f64a0400640 1 -- start start 2024-09-13T17:23:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.865+0000 7f64a0400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f649815c9a0 0x7f64980ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.865+0000 7f64a0400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6498070470 con 0x7f64980b2400 2024-09-13T17:23:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f64a0400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64980ace30 con 0x7f64980ad940 2024-09-13T17:23:50.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f64a0400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64980ad110 con 0x7f649815c9a0 2024-09-13T17:23:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649da00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f649815c9a0 0x7f64980ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649e400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f64980b2400 0x7f649806bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49388/0 (socket says 172.21.15.28:49388) 2024-09-13T17:23:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649e400640 1 -- 172.21.15.28:0/1263136838 learned_addr learned my addr 172.21.15.28:0/1263136838 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649da00640 1 -- 172.21.15.28:0/1263136838 >> v1:172.21.15.189:6789/0 conn(0x7f64980ad940 legacy=0x7f64980b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649da00640 1 -- 172.21.15.28:0/1263136838 >> v1:172.21.15.28:6789/0 conn(0x7f64980b2400 legacy=0x7f649806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.866+0000 7f649da00640 1 -- 172.21.15.28:0/1263136838 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649806b170 con 0x7f649815c9a0 2024-09-13T17:23:50.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.867+0000 7f649da00640 1 --2- 172.21.15.28:0/1263136838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f649815c9a0 0x7f64980ac850 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f648c0097d0 tx=0x7f648c02a2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1add5a3b0951ecb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f6497e00640 1 -- 172.21.15.28:0/1263136838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c032440 con 0x7f649815c9a0 2024-09-13T17:23:50.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f6497e00640 1 -- 172.21.15.28:0/1263136838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c0325a0 con 0x7f649815c9a0 2024-09-13T17:23:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f6497e00640 1 -- 172.21.15.28:0/1263136838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c032930 con 0x7f649815c9a0 2024-09-13T17:23:50.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 -- 172.21.15.28:0/1263136838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f649815c9a0 msgr2=0x7f64980ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 --2- 172.21.15.28:0/1263136838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f649815c9a0 0x7f64980ac850 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7f648c0097d0 tx=0x7f648c02a2e0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 -- 172.21.15.28:0/1263136838 shutdown_connections 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 --2- 172.21.15.28:0/1263136838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f649815c9a0 0x7f64980ac850 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 -- 172.21.15.28:0/1263136838 >> 172.21.15.28:0/1263136838 conn(0x7f64980b0270 msgr2=0x7f64980b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 -- 172.21.15.28:0/1263136838 shutdown_connections 2024-09-13T17:23:50.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.868+0000 7f64a0400640 1 -- 172.21.15.28:0/1263136838 wait complete. 2024-09-13T17:23:50.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.870+0000 7f64a0400640 1 Processor -- start 2024-09-13T17:23:50.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.870+0000 7f64a0400640 1 -- start start 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64980ad940 0x7f6498146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f649815c9a0 0x7f6498143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498143aa0 con 0x7f64980b2400 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498144270 con 0x7f64980ad940 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f64a0400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6498144510 con 0x7f649815c9a0 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f649d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f649d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35360/0 (socket says 172.21.15.28:35360) 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f649d000640 1 -- 172.21.15.28:0/2959550747 learned_addr learned my addr 172.21.15.28:0/2959550747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f649da00640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64980ad940 0x7f6498146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.871+0000 7f649e400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f649815c9a0 0x7f6498143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 -- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f649815c9a0 msgr2=0x7f6498143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f649815c9a0 0x7f6498143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 -- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64980ad940 msgr2=0x7f6498146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64980ad940 0x7f6498146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 -- 172.21.15.28:0/2959550747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f648c009400 con 0x7f64980b2400 2024-09-13T17:23:50.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f649d000640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f648400c450 tx=0x7f648400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f6495600640 1 -- 172.21.15.28:0/2959550747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840052d0 con 0x7f64980b2400 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f6495600640 1 -- 172.21.15.28:0/2959550747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6484005430 con 0x7f64980b2400 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f6495600640 1 -- 172.21.15.28:0/2959550747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484012870 con 0x7f64980b2400 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.872+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6498070930 con 0x7f64980b2400 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.873+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6498144ad0 con 0x7f64980b2400 2024-09-13T17:23:50.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.874+0000 7f6495600640 1 -- 172.21.15.28:0/2959550747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f64840129d0 con 0x7f64980b2400 2024-09-13T17:23:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.875+0000 7f6495600640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 0x7f647806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.875+0000 7f6495600640 1 -- 172.21.15.28:0/2959550747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6484088df0 con 0x7f64980b2400 2024-09-13T17:23:50.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.876+0000 7f649da00640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 0x7f647806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.876+0000 7f64a32b7740 1 Processor -- start 2024-09-13T17:23:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.876+0000 7f64a32b7740 1 -- start start 2024-09-13T17:23:50.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.876+0000 7f649da00640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 0x7f647806d0c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f648c034000 tx=0x7f648c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be4241f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be4215d0 0x5615be4712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be330020 con 0x5615be44b2c0 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be38dba0 con 0x5615be4215d0 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615be38b330 con 0x5615be423e10 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f649da00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be4241f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35372/0 (socket says 172.21.15.28:35372) 2024-09-13T17:23:50.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f649e400640 1 -- 172.21.15.28:0/3916757164 learned_addr learned my addr 172.21.15.28:0/3916757164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.877+0000 7f649d000640 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be4215d0 0x5615be4712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 -- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be4215d0 msgr2=0x5615be4712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be4215d0 0x5615be4712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 -- 172.21.15.28:0/3916757164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 msgr2=0x5615be471820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 -- 172.21.15.28:0/3916757164 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615be338cf0 con 0x5615be423e10 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649e400640 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:50.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.878+0000 7f649da00640 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be4241f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f648c031000 tx=0x7f648c02c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e1749489d71833 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f648a000640 1 -- 172.21.15.28:0/3916757164 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c03c8c0 con 0x5615be423e10 2024-09-13T17:23:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f648a000640 1 -- 172.21.15.28:0/3916757164 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c03ca20 con 0x5615be423e10 2024-09-13T17:23:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f648a000640 1 -- 172.21.15.28:0/3916757164 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648c03cd90 con 0x5615be423e10 2024-09-13T17:23:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 -- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 msgr2=0x5615be4241f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be4241f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f648c031000 tx=0x7f648c02c000 comp rx=0 tx=0).stop 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 -- 172.21.15.28:0/3916757164 shutdown_connections 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be471820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be4215d0 0x5615be4712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 --2- 172.21.15.28:0/3916757164 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be4241f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 -- 172.21.15.28:0/3916757164 >> 172.21.15.28:0/3916757164 conn(0x5615be420540 msgr2=0x5615be4689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 -- 172.21.15.28:0/3916757164 shutdown_connections 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.879+0000 7f64a32b7740 1 -- 172.21.15.28:0/3916757164 wait complete. 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.880+0000 7f64a32b7740 1 -- start start 2024-09-13T17:23:50.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.880+0000 7f64a32b7740 1 Processor -- start 2024-09-13T17:23:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be44eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 0x5615be448c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be386c20 con 0x5615be470a10 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be3281b0 con 0x5615be44b2c0 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615be327e50 con 0x5615be423e10 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f64a32b7740 10 client.0 _ll_register_callbacks cb 0x5615be380290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.881+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35378/0 (socket says 172.21.15.28:35378) 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 -- 172.21.15.28:0/1282281026 learned_addr learned my addr 172.21.15.28:0/1282281026 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649d000640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 0x5615be448c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649da00640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be44eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 -- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 msgr2=0x5615be44eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be44eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 -- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 msgr2=0x5615be448c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 0x5615be448c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 -- 172.21.15.28:0/1282281026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649806b170 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649da00640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be44eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649d000640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 0x5615be448c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f649e400640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f6490012d30 tx=0x7f6490010610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c734dea3a34eab7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f647ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.882+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490001ee0 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5615be3afeb0 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6490002040 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6490005be0 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 10 client.12111 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x5615be3852a0 con 0x5615be470a10 2024-09-13T17:23:50.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5615be3afeb0 con 0x5615be470a10 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f64900070d0 con 0x5615be470a10 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 10 client.12111 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 10 client.12111 resolve_mds: resolved 15:0 to role '15:0' aka daemon mds.c 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 20 client.12111 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be41e690 0x5615be41eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 4 client.12111 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.883+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5615be37b340 con 0x5615be41e690 2024-09-13T17:23:50.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.884+0000 7f649da00640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be41e690 0x5615be41eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.884+0000 7f649da00640 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be41e690 0x5615be41eaa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ac3bb4e056ec44e server_cookie=ded588a22ea81b89 in_seq=0 out_seq=0 2024-09-13T17:23:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.884+0000 7f647ea00640 10 client.12111 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f647ea00640 1 -- 172.21.15.28:0/1282281026 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5615be37b340 con 0x5615be41e690 2024-09-13T17:23:50.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f647ea00640 10 client.12111 handle_command_reply: tid=0 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 client.12111 shutdown 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 msgr2=0x5615be449490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f6490012d30 tx=0x7f6490010610 comp rx=0 tx=0).stop 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 shutdown_connections 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be41e690 0x5615be41eaa0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be449490 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be44b2c0 0x5615be448c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 --2- 172.21.15.28:0/1282281026 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be44eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 >> 172.21.15.28:0/1282281026 conn(0x5615be4215d0 msgr2=0x5615be46bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f647ea00640 0 client.12111 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 shutdown_connections 2024-09-13T17:23:50.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 1 -- 172.21.15.28:0/1282281026 wait complete. 2024-09-13T17:23:50.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.885+0000 7f64a32b7740 20 client.12111 trim_cache size 0 max 16384 2024-09-13T17:23:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.897+0000 7f64a32b7740 1 Processor -- start 2024-09-13T17:23:50.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.897+0000 7f64a32b7740 1 -- start start 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be423540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 0x5615be46bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be0ee040 con 0x5615be44b2c0 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be46ba20 con 0x5615be470a10 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615be47b250 con 0x5615be423e10 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f649d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f649d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35380/0 (socket says 172.21.15.28:35380) 2024-09-13T17:23:50.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.898+0000 7f649d000640 1 -- 172.21.15.28:0/37111031 learned_addr learned my addr 172.21.15.28:0/37111031 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649da00640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be423540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 -- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 msgr2=0x5615be423540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be423540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 -- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 msgr2=0x5615be46bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649e400640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 0x5615be46bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 0x5615be46bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 -- 172.21.15.28:0/37111031 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5615be3b9200 con 0x5615be44b2c0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649da00640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be423540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649e400640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 0x5615be46bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f649d000640 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f648405fab0 tx=0x7f648405ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdca3f6fdb2301a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f648aa00640 1 -- 172.21.15.28:0/37111031 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484055de0 con 0x5615be44b2c0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f648aa00640 1 -- 172.21.15.28:0/37111031 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648406f070 con 0x5615be44b2c0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.899+0000 7f648aa00640 1 -- 172.21.15.28:0/37111031 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840543b0 con 0x5615be44b2c0 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- 172.21.15.28:0/37111031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 msgr2=0x5615be4cce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f648405fab0 tx=0x7f648405ba20 comp rx=0 tx=0).stop 2024-09-13T17:23:50.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- 172.21.15.28:0/37111031 shutdown_connections 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be470a10 0x5615be46bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be44b2c0 0x5615be4cce10 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 --2- 172.21.15.28:0/37111031 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be423e10 0x5615be423540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- 172.21.15.28:0/37111031 >> 172.21.15.28:0/37111031 conn(0x5615be4215d0 msgr2=0x5615be44ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- 172.21.15.28:0/37111031 shutdown_connections 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- 172.21.15.28:0/37111031 wait complete. 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.900+0000 7f64a32b7740 1 -- start start 2024-09-13T17:23:50.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.901+0000 7f64a32b7740 1 Processor -- start 2024-09-13T17:23:50.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be423e10 0x5615be46cad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be44b2c0 0x5615be4b1a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be4b2440 con 0x5615be470a10 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5615be481d10 con 0x5615be423e10 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5615be4b1f70 con 0x5615be44b2c0 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f64a32b7740 10 client.0 _ll_register_callbacks cb 0x7f6490001e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35384/0 (socket says 172.21.15.28:35384) 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 -- 172.21.15.28:0/2152021147 learned_addr learned my addr 172.21.15.28:0/2152021147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649da00640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be423e10 0x5615be46cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649d000640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be44b2c0 0x5615be4b1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 -- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be44b2c0 msgr2=0x5615be4b1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be44b2c0 0x5615be4b1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 -- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be423e10 msgr2=0x5615be46cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be423e10 0x5615be46cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.902+0000 7f649e400640 1 -- 172.21.15.28:0/2152021147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6498144ad0 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f649e400640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f6490000f80 tx=0x7f6490005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3039723ad60fead2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490010b10 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6490007630 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f648c03ca20 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=181) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6490005df0 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 10 client.12117 fetch_fsmap learned FSMap version 181 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=181,monmap=2+}) v3 -- 0x5615be3275f0 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648c03ca20 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 181) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6490007840 con 0x5615be470a10 2024-09-13T17:23:50.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 10 client.12117 fetch_fsmap finished waiting for FSMap version 181 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 10 client.12117 resolve_mds: resolved 15:0 to role '15:0' aka daemon mds.c 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 20 client.12117 populate_metadata read hostname 'smithi028' 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be481880 0x5615be4227b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 4 client.12117 mds_command: new command op to 11916 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.903+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 --> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5615be07a4e0 con 0x5615be481880 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f649da00640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be481880 0x5615be4227b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f649da00640 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be481880 0x5615be4227b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fb06c83f08852e8 server_cookie=29e69f5da03797ce in_seq=0 out_seq=0 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f647ea00640 10 client.12117 ms_handle_connect on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f647ea00640 1 -- 172.21.15.28:0/2152021147 <== mds.0 v2:172.21.15.28:6832/3884798333 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5615be07a4e0 con 0x5615be481880 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f647ea00640 10 client.12117 handle_command_reply: tid=0 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f64a32b7740 1 client.12117 shutdown 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 msgr2=0x5615be4ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.904+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 secure :-1 s=READY pgs=2628 cs=0 l=1 rev1=1 crypto rx=0x7f6490000f80 tx=0x7f6490005b70 comp rx=0 tx=0).stop 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 shutdown_connections 2024-09-13T17:23:50.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:6832/3884798333,v1:172.21.15.28:6834/3884798333] conn(0x5615be481880 0x5615be4227b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5615be470a10 0x5615be4ca350 unknown :-1 s=CLOSED pgs=2628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5615be44b2c0 0x5615be4b1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 --2- 172.21.15.28:0/2152021147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5615be423e10 0x5615be46cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f647ea00640 0 client.12117 ms_handle_reset on v2:172.21.15.28:6832/3884798333 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 >> 172.21.15.28:0/2152021147 conn(0x5615be4215d0 msgr2=0x5615be46bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 shutdown_connections 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 1 -- 172.21.15.28:0/2152021147 wait complete. 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a32b7740 20 client.12117 trim_cache size 0 max 16384 2024-09-13T17:23:50.910 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 msgr2=0x7f647806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 0x7f647806d0c0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f648c034000 tx=0x7f648c02e000 comp rx=0 tx=0).stop 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 msgr2=0x7f64980cd110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7f648400c450 tx=0x7f648400c910 comp rx=0 tx=0).stop 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 shutdown_connections 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f647806ac00 0x7f647806d0c0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f649815c9a0 0x7f6498143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f64980b2400 0x7f64980cd110 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.905+0000 7f64a0400640 1 --2- 172.21.15.28:0/2959550747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64980ad940 0x7f6498146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.906+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 >> 172.21.15.28:0/2959550747 conn(0x7f64980b0270 msgr2=0x7f64980af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.906+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 shutdown_connections 2024-09-13T17:23:50.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:50.906+0000 7f64a0400640 1 -- 172.21.15.28:0/2959550747 wait complete. 2024-09-13T17:23:50.921 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:23:50.922 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:51.009 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:23:51.105 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:23:51.105 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:51.105 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:51.142 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:51.142 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:51.217 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.413+0000 7f8af7800640 1 Processor -- start 2024-09-13T17:23:51.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.413+0000 7f8af7800640 1 -- start start 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af7800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8af01592a0 0x7f8af015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af7800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af0159d90 con 0x7f8af006c2d0 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af7800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00b2040 con 0x7f8af006f970 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af7800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af015b0b0 con 0x7f8af01592a0 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af5800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8af006c2d0 0x7f8af00b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49404/0 (socket says 172.21.15.28:49404) 2024-09-13T17:23:51.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af5800640 1 -- 172.21.15.28:0/1384354460 learned_addr learned my addr 172.21.15.28:0/1384354460 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8af4e00640 1 --2- 172.21.15.28:0/1384354460 >> v2:172.21.15.189:3300/0 conn(0x7f8af01592a0 0x7f8af015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.415+0000 7f8aefe00640 1 -- 172.21.15.28:0/1384354460 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727228689 0 0) 0x7f8af0159d90 con 0x7f8af006c2d0 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.416+0000 7f8aefe00640 1 -- 172.21.15.28:0/1384354460 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad4003520 con 0x7f8af006c2d0 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.416+0000 7f8af4e00640 1 -- 172.21.15.28:0/1384354460 >> v1:172.21.15.189:6789/0 conn(0x7f8af006f970 legacy=0x7f8af0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.416+0000 7f8af4e00640 1 -- 172.21.15.28:0/1384354460 >> v1:172.21.15.28:6789/0 conn(0x7f8af006c2d0 legacy=0x7f8af00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.416+0000 7f8af4e00640 1 -- 172.21.15.28:0/1384354460 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0159ff0 con 0x7f8af01592a0 2024-09-13T17:23:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.417+0000 7f8af4e00640 1 --2- 172.21.15.28:0/1384354460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af01592a0 0x7f8af015ab70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00097a0 tx=0x7f8ae002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f2a79df9ea6f780 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.417+0000 7f8aefe00640 1 -- 172.21.15.28:0/1384354460 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0032440 con 0x7f8af01592a0 2024-09-13T17:23:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.417+0000 7f8aefe00640 1 -- 172.21.15.28:0/1384354460 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae00325a0 con 0x7f8af01592a0 2024-09-13T17:23:51.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.417+0000 7f8aefe00640 1 -- 172.21.15.28:0/1384354460 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae0032930 con 0x7f8af01592a0 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 -- 172.21.15.28:0/1384354460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af01592a0 msgr2=0x7f8af015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 --2- 172.21.15.28:0/1384354460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af01592a0 0x7f8af015ab70 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00097a0 tx=0x7f8ae002a230 comp rx=0 tx=0).stop 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 -- 172.21.15.28:0/1384354460 shutdown_connections 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 --2- 172.21.15.28:0/1384354460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af01592a0 0x7f8af015ab70 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 -- 172.21.15.28:0/1384354460 >> 172.21.15.28:0/1384354460 conn(0x7f8af0069e80 msgr2=0x7f8af006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 -- 172.21.15.28:0/1384354460 shutdown_connections 2024-09-13T17:23:51.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.418+0000 7f8af7800640 1 -- 172.21.15.28:0/1384354460 wait complete. 2024-09-13T17:23:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.420+0000 7f8af7800640 1 Processor -- start 2024-09-13T17:23:51.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.420+0000 7f8af7800640 1 -- start start 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8af006c2d0 0x7f8af0151570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af006f970 0x7f8af01f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0151210 con 0x7f8af01592a0 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0150c90 con 0x7f8af006f970 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af7800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af01f2a70 con 0x7f8af006c2d0 2024-09-13T17:23:51.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35386/0 (socket says 172.21.15.28:35386) 2024-09-13T17:23:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af5800640 1 -- 172.21.15.28:0/2269486275 learned_addr learned my addr 172.21.15.28:0/2269486275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8ae7e00640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af006f970 0x7f8af01f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.421+0000 7f8af4e00640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8af006c2d0 0x7f8af0151570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 -- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8af006c2d0 msgr2=0x7f8af0151570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8af006c2d0 0x7f8af0151570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 -- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af006f970 msgr2=0x7f8af01f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af006f970 0x7f8af01f3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 -- 172.21.15.28:0/2269486275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0009400 con 0x7f8af01592a0 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af5800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800cae0 tx=0x7f8ae800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae800ac40 con 0x7f8af01592a0 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae800ada0 con 0x7f8af01592a0 2024-09-13T17:23:51.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0150f30 con 0x7f8af01592a0 2024-09-13T17:23:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae8012600 con 0x7f8af01592a0 2024-09-13T17:23:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.422+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af0151ab0 con 0x7f8af01592a0 2024-09-13T17:23:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.423+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af01f28d0 con 0x7f8af01592a0 2024-09-13T17:23:51.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.424+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8ae8007070 con 0x7f8af01592a0 2024-09-13T17:23:51.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.426+0000 7f8aed600640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 0x7f8acc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.426+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8ae8089150 con 0x7f8af01592a0 2024-09-13T17:23:51.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.426+0000 7f8af4e00640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 0x7f8acc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.427+0000 7f8af4e00640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 0x7f8acc06d0c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0034000 tx=0x7f8ae002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.430+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8af01f28d0 con 0x7f8af01592a0 2024-09-13T17:23:51.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.611+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8af01f2f50 con 0x7f8af01592a0 2024-09-13T17:23:51.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.613+0000 7f8aed600640 1 -- 172.21.15.28:0/2269486275 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f8af01f2f50 con 0x7f8af01592a0 2024-09-13T17:23:51.614 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:51.614 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":219,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:42.631521+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:51.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 msgr2=0x7f8acc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 0x7f8acc06d0c0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f8ae0034000 tx=0x7f8ae002e000 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 msgr2=0x7f8af0150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f8ae800cae0 tx=0x7f8ae800a060 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 shutdown_connections 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8acc06ac00 0x7f8acc06d0c0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8af01592a0 0x7f8af0150330 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8af006f970 0x7f8af01f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 --2- 172.21.15.28:0/2269486275 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8af006c2d0 0x7f8af0151570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 >> 172.21.15.28:0/2269486275 conn(0x7f8af0069e80 msgr2=0x7f8af015c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 shutdown_connections 2024-09-13T17:23:51.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.615+0000 7f8af7800640 1 -- 172.21.15.28:0/2269486275 wait complete. 2024-09-13T17:23:51.632 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:23:39.609113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:23:40.623957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:51.633 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.790+0000 7f24eb200640 1 Processor -- start 2024-09-13T17:23:51.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.791+0000 7f24eb200640 1 -- start start 2024-09-13T17:23:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24eb200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24ec0cc880 0x7f24ec0b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24eb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24eb200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24ec0722e0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24eb200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24ec0cb3e0 con 0x7f24ec0cc880 2024-09-13T17:23:51.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24eb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec072000 con 0x7f24ec0b0c30 2024-09-13T17:23:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24e3e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f24ec0cc880 0x7f24ec0b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.792+0000 7f24ea800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:49896/0 (socket says 172.21.15.28:49896) 2024-09-13T17:23:51.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 -- 172.21.15.28:0/1542994588 learned_addr learned my addr 172.21.15.28:0/1542994588 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 -- 172.21.15.28:0/1542994588 >> v2:172.21.15.189:3300/0 conn(0x7f24ec0cc880 msgr2=0x7f24ec0b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 --2- 172.21.15.28:0/1542994588 >> v2:172.21.15.189:3300/0 conn(0x7f24ec0cc880 0x7f24ec0b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24e9400640 1 -- 172.21.15.28:0/1542994588 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240599428 0 0) 0x7f24ec0722e0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 -- 172.21.15.28:0/1542994588 >> v1:172.21.15.189:6789/0 conn(0x7f24ec0cc4a0 legacy=0x7f24ec0cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.793+0000 7f24ea800640 1 -- 172.21.15.28:0/1542994588 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24ec0cad90 con 0x7f24ec0b0c30 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.794+0000 7f24ea800640 1 --2- 172.21.15.28:0/1542994588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009e50 tx=0x7f24e402a5f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c62f7c11963a2d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.794+0000 7f24e9400640 1 -- 172.21.15.28:0/1542994588 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4031600 con 0x7f24ec0b0c30 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.794+0000 7f24e9400640 1 -- 172.21.15.28:0/1542994588 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e402d050 con 0x7f24ec0b0c30 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.794+0000 7f24e9400640 1 -- 172.21.15.28:0/1542994588 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e4031980 con 0x7f24ec0b0c30 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 -- 172.21.15.28:0/1542994588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 msgr2=0x7f24ec0712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 --2- 172.21.15.28:0/1542994588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f24e4009e50 tx=0x7f24e402a5f0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 -- 172.21.15.28:0/1542994588 shutdown_connections 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 --2- 172.21.15.28:0/1542994588 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0b0c30 0x7f24ec0712d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 --2- 172.21.15.28:0/1542994588 >> v2:172.21.15.189:3300/0 conn(0x7f24ec0cc880 0x7f24ec0b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 -- 172.21.15.28:0/1542994588 >> 172.21.15.28:0/1542994588 conn(0x7f24ec069e80 msgr2=0x7f24ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 -- 172.21.15.28:0/1542994588 shutdown_connections 2024-09-13T17:23:51.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.795+0000 7f24eb200640 1 -- 172.21.15.28:0/1542994588 wait complete. 2024-09-13T17:23:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.797+0000 7f24eb200640 1 Processor -- start 2024-09-13T17:23:51.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.797+0000 7f24eb200640 1 -- start start 2024-09-13T17:23:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0b0c30 0x7f24ec152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0cc880 0x7f24ec15c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1f6560 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1f5690 con 0x7f24ec0b0c30 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24eb200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24ec1f58e0 con 0x7f24ec0cc880 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35400/0 (socket says 172.21.15.28:35400) 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24e3e00640 1 -- 172.21.15.28:0/1595672657 learned_addr learned my addr 172.21.15.28:0/1595672657 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24e9e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0b0c30 0x7f24ec152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.798+0000 7f24ea800640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0cc880 0x7f24ec15c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 -- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0cc880 msgr2=0x7f24ec15c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0cc880 0x7f24ec15c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 -- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0b0c30 msgr2=0x7f24ec152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0b0c30 0x7f24ec152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 -- 172.21.15.28:0/1595672657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e4009b30 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e3e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f24d400e870 tx=0x7f24d400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d40094f0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d400c940 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d400ccf0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24ec1f5bc0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.799+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24ec1f6180 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.801+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24ec0712d0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.801+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f24d4013470 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.802+0000 7f24e2000640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 0x7f24c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.802+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f24d4089360 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.803+0000 7f24e9e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 0x7f24c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:51.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.803+0000 7f24e9e00640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 0x7f24c806d0c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f24dc000f60 tx=0x7f24dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:51.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.806+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f24d40545e0 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.985+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24ec157300 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.985+0000 7f24e2000640 1 -- 172.21.15.28:0/1595672657 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24d4057c00 con 0x7f24ec0cc4a0 2024-09-13T17:23:51.986 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 msgr2=0x7f24c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 0x7f24c806d0c0 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f24dc000f60 tx=0x7f24dc009040 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 msgr2=0x7f24ec15bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f24d400e870 tx=0x7f24d400ed30 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 shutdown_connections 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f24c806ac00 0x7f24c806d0c0 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f24ec0cc880 0x7f24ec15c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f24ec0cc4a0 0x7f24ec15bb60 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 --2- 172.21.15.28:0/1595672657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f24ec0b0c30 0x7f24ec152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.987+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 >> 172.21.15.28:0/1595672657 conn(0x7f24ec069e80 msgr2=0x7f24ec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.988+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 shutdown_connections 2024-09-13T17:23:51.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:51.988+0000 7f24eb200640 1 -- 172.21.15.28:0/1595672657 wait complete. 2024-09-13T17:23:52.005 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:23:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.163+0000 7f5debe00640 1 Processor -- start 2024-09-13T17:23:52.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.163+0000 7f5debe00640 1 -- start start 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5debe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 0x7f5dec0cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5debe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5dec0cddb0 0x7f5dec15aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5debe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dec159e10 con 0x7f5dec070e30 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5debe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec0ce240 con 0x7f5dec0cddb0 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5debe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dec15a0f0 con 0x7f5dec071210 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5deb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5dec0cddb0 0x7f5dec15aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5deaa00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5dec070e30 0x7f5dec071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49414/0 (socket says 172.21.15.28:49414) 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.164+0000 7f5deaa00640 1 -- 172.21.15.28:0/672710828 learned_addr learned my addr 172.21.15.28:0/672710828 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5de3e00640 1 --2- 172.21.15.28:0/672710828 >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 0x7f5dec0cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5deb400640 1 -- 172.21.15.28:0/672710828 >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 msgr2=0x7f5dec0cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5deb400640 1 --2- 172.21.15.28:0/672710828 >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 0x7f5dec0cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5deb400640 1 -- 172.21.15.28:0/672710828 >> v1:172.21.15.28:6789/0 conn(0x7f5dec070e30 legacy=0x7f5dec071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5dea000640 1 -- 172.21.15.28:0/672710828 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239527161 0 0) 0x7f5dec159e10 con 0x7f5dec070e30 2024-09-13T17:23:52.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5deb400640 1 -- 172.21.15.28:0/672710828 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec15a3d0 con 0x7f5dec0cddb0 2024-09-13T17:23:52.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.165+0000 7f5de3e00640 1 --2- 172.21.15.28:0/672710828 >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 0x7f5dec0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5deb400640 1 --2- 172.21.15.28:0/672710828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec0cddb0 0x7f5dec15aed0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0b3870 tx=0x7f5de402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c228cfc9022f91e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5dea000640 1 -- 172.21.15.28:0/672710828 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de402aea0 con 0x7f5dec0cddb0 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5dea000640 1 -- 172.21.15.28:0/672710828 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de402d050 con 0x7f5dec0cddb0 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5dea000640 1 -- 172.21.15.28:0/672710828 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de4007960 con 0x7f5dec0cddb0 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5debe00640 1 -- 172.21.15.28:0/672710828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec0cddb0 msgr2=0x7f5dec15aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.166+0000 7f5debe00640 1 --2- 172.21.15.28:0/672710828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec0cddb0 0x7f5dec15aed0 secure :-1 s=READY pgs=2633 cs=0 l=1 rev1=1 crypto rx=0x7f5dec0b3870 tx=0x7f5de402a680 comp rx=0 tx=0).stop 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 -- 172.21.15.28:0/672710828 shutdown_connections 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 --2- 172.21.15.28:0/672710828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec0cddb0 0x7f5dec15aed0 unknown :-1 s=CLOSED pgs=2633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 --2- 172.21.15.28:0/672710828 >> v2:172.21.15.189:3300/0 conn(0x7f5dec071210 0x7f5dec0cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 -- 172.21.15.28:0/672710828 >> 172.21.15.28:0/672710828 conn(0x7f5dec069e80 msgr2=0x7f5dec06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:52.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 -- 172.21.15.28:0/672710828 shutdown_connections 2024-09-13T17:23:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.167+0000 7f5debe00640 1 -- 172.21.15.28:0/672710828 wait complete. 2024-09-13T17:23:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.168+0000 7f5debe00640 1 Processor -- start 2024-09-13T17:23:52.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.168+0000 7f5debe00640 1 -- start start 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 0x7f5dec0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5dec0cddb0 0x7f5dec1f1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec144870 con 0x7f5dec070e30 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0c6a80 con 0x7f5dec071210 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5debe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec0c6d60 con 0x7f5dec0cddb0 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.169+0000 7f5deaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35414/0 (socket says 172.21.15.28:35414) 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 -- 172.21.15.28:0/861990597 learned_addr learned my addr 172.21.15.28:0/861990597 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5de3e00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 0x7f5dec0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deb400640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5dec0cddb0 0x7f5dec1f1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 -- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5dec0cddb0 msgr2=0x7f5dec1f1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5dec0cddb0 0x7f5dec1f1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 -- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 msgr2=0x7f5dec0cc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 0x7f5dec0cc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 -- 172.21.15.28:0/861990597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de4009a70 con 0x7f5dec070e30 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5de3e00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 0x7f5dec0cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5deaa00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00c430 tx=0x7f5ddc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.170+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc0074e0 con 0x7f5dec070e30 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.171+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ddc007660 con 0x7f5dec070e30 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.171+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc004a20 con 0x7f5dec070e30 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.171+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec0c7040 con 0x7f5dec070e30 2024-09-13T17:23:52.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.171+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec153210 con 0x7f5dec070e30 2024-09-13T17:23:52.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.172+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec0cd870 con 0x7f5dec070e30 2024-09-13T17:23:52.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.173+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5ddc004b80 con 0x7f5dec070e30 2024-09-13T17:23:52.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.173+0000 7f5de2a00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 0x7f5dc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.174+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ddc089210 con 0x7f5dec070e30 2024-09-13T17:23:52.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.174+0000 7f5de3e00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 0x7f5dc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.175+0000 7f5de3e00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 0x7f5dc806d0c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5dec1597c0 tx=0x7f5dd4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.177+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5ddc054350 con 0x7f5dec070e30 2024-09-13T17:23:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.353+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5dec0719e0 con 0x7f5dec070e30 2024-09-13T17:23:52.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.354+0000 7f5de2a00640 1 -- 172.21.15.28:0/861990597 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v219) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ddc057970 con 0x7f5dec070e30 2024-09-13T17:23:52.355 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.355+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 msgr2=0x7f5dc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.355+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 0x7f5dc806d0c0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f5dec1597c0 tx=0x7f5dd4008040 comp rx=0 tx=0).stop 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 msgr2=0x7f5dec152cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc00c430 tx=0x7f5ddc00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 shutdown_connections 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5dc806ac00 0x7f5dc806d0c0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5dec0cddb0 0x7f5dec1f1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec071210 0x7f5dec0cc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 --2- 172.21.15.28:0/861990597 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5dec070e30 0x7f5dec152cd0 unknown :-1 s=CLOSED pgs=2634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 >> 172.21.15.28:0/861990597 conn(0x7f5dec069e80 msgr2=0x7f5dec071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 shutdown_connections 2024-09-13T17:23:52.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.356+0000 7f5debe00640 1 -- 172.21.15.28:0/861990597 wait complete. 2024-09-13T17:23:52.372 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:23:52.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.529+0000 7f343de00640 1 Processor -- start 2024-09-13T17:23:52.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.529+0000 7f343de00640 1 -- start start 2024-09-13T17:23:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.530+0000 7f343de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.530+0000 7f343de00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3438071490 0x7f343806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.530+0000 7f343de00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f343806c8b0 con 0x7f343815a780 2024-09-13T17:23:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.530+0000 7f343de00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343806cb90 con 0x7f3438071490 2024-09-13T17:23:52.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.530+0000 7f343de00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380cc980 con 0x7f343815ab60 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3436a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f343815a780 0x7f3438070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:49426/0 (socket says 172.21.15.28:49426) 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3436a00640 1 -- 172.21.15.28:0/1381217093 learned_addr learned my addr 172.21.15.28:0/1381217093 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3437400640 1 --2- 172.21.15.28:0/1381217093 >> v2:172.21.15.28:3300/0 conn(0x7f3438071490 0x7f343806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f342e000640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3436000640 1 -- 172.21.15.28:0/1381217093 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764574803 0 0) 0x7f343806c8b0 con 0x7f343815a780 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3436000640 1 -- 172.21.15.28:0/1381217093 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f341c003520 con 0x7f343815a780 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3437400640 1 -- 172.21.15.28:0/1381217093 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 msgr2=0x7f343806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3437400640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343806bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3437400640 1 -- 172.21.15.28:0/1381217093 >> v1:172.21.15.28:6789/0 conn(0x7f343815a780 legacy=0x7f3438070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.531+0000 7f3437400640 1 -- 172.21.15.28:0/1381217093 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380ccc60 con 0x7f3438071490 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.532+0000 7f3437400640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f343806c080 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f3430009d90 tx=0x7f343002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a901a29cb0c1f93 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f3436000640 1 -- 172.21.15.28:0/1381217093 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430007740 con 0x7f3438071490 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f3436000640 1 -- 172.21.15.28:0/1381217093 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343002d050 con 0x7f3438071490 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f3436000640 1 -- 172.21.15.28:0/1381217093 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3430007ac0 con 0x7f3438071490 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 -- 172.21.15.28:0/1381217093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 msgr2=0x7f343806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f343806c080 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f3430009d90 tx=0x7f343002a660 comp rx=0 tx=0).stop 2024-09-13T17:23:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 -- 172.21.15.28:0/1381217093 shutdown_connections 2024-09-13T17:23:52.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f343806c080 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 --2- 172.21.15.28:0/1381217093 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343806bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 -- 172.21.15.28:0/1381217093 >> 172.21.15.28:0/1381217093 conn(0x7f34380b0270 msgr2=0x7f34380b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:52.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 -- 172.21.15.28:0/1381217093 shutdown_connections 2024-09-13T17:23:52.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.533+0000 7f343de00640 1 -- 172.21.15.28:0/1381217093 wait complete. 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.535+0000 7f343de00640 1 Processor -- start 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.535+0000 7f343de00640 1 -- start start 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 0x7f3438150d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343814cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3438150ba0 con 0x7f3438071490 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3438151240 con 0x7f343815a780 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f343de00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3438152040 con 0x7f343815ab60 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f3436a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f3436a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35428/0 (socket says 172.21.15.28:35428) 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f3436a00640 1 -- 172.21.15.28:0/797382112 learned_addr learned my addr 172.21.15.28:0/797382112 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f3437400640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343814cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.536+0000 7f342e000640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 0x7f3438150d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 -- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 msgr2=0x7f343814cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343814cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 -- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 msgr2=0x7f3438150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 0x7f3438150d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 -- 172.21.15.28:0/797382112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3430009a70 con 0x7f3438071490 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3437400640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f342e000640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 0x7f3438150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f3436a00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f342800c530 tx=0x7f342800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428005ce0 con 0x7f3438071490 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3428005e40 con 0x7f3438071490 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34280072b0 con 0x7f3438071490 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343814d3e0 con 0x7f3438071490 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.537+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3438151710 con 0x7f3438071490 2024-09-13T17:23:52.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.538+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34380cb320 con 0x7f3438071490 2024-09-13T17:23:52.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.539+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3428007410 con 0x7f3438071490 2024-09-13T17:23:52.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.540+0000 7f342fe00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 0x7f341406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.540+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3428088900 con 0x7f3438071490 2024-09-13T17:23:52.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.541+0000 7f342e000640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 0x7f341406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.541+0000 7f342e000640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 0x7f341406d0c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f34381587f0 tx=0x7f3420008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.544+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3428053ba0 con 0x7f3438071490 2024-09-13T17:23:52.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.723+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f34380ad5e0 con 0x7f3438071490 2024-09-13T17:23:52.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.724+0000 7f342fe00640 1 -- 172.21.15.28:0/797382112 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v219) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3428017020 con 0x7f3438071490 2024-09-13T17:23:52.725 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 msgr2=0x7f341406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 0x7f341406d0c0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f34381587f0 tx=0x7f3420008040 comp rx=0 tx=0).stop 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 msgr2=0x7f3438150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f342800c530 tx=0x7f342800c9f0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 shutdown_connections 2024-09-13T17:23:52.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f341406ac00 0x7f341406d0c0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f343815ab60 0x7f343814cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f343815a780 0x7f3438150d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 --2- 172.21.15.28:0/797382112 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3438071490 0x7f3438150660 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 >> 172.21.15.28:0/797382112 conn(0x7f34380b0270 msgr2=0x7f34380cda90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 shutdown_connections 2024-09-13T17:23:52.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.726+0000 7f343de00640 1 -- 172.21.15.28:0/797382112 wait complete. 2024-09-13T17:23:52.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:23:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.900+0000 7fd3fd800640 1 Processor -- start 2024-09-13T17:23:52.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.901+0000 7fd3fd800640 1 -- start start 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3fd800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd3f80c40a0 0x7fd3f80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3fd800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd3f80b1ba0 0x7fd3f815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3fd800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f80b2090 con 0x7fd3f815b0b0 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3fd800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f80b2370 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3fd800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f80b2650 con 0x7fd3f80c40a0 2024-09-13T17:23:52.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd3f80b1ba0 0x7fd3f815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3efe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd3f80b1ba0 0x7fd3f815ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35436/0 (socket says 172.21.15.28:35436) 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3efe00640 1 -- 172.21.15.28:0/2441425073 learned_addr learned my addr 172.21.15.28:0/2441425073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3f7e00640 1 --2- 172.21.15.28:0/2441425073 >> v2:172.21.15.189:3300/0 conn(0x7fd3f80c40a0 0x7fd3f80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3efe00640 1 -- 172.21.15.28:0/2441425073 >> v2:172.21.15.189:3300/0 conn(0x7fd3f80c40a0 msgr2=0x7fd3f80b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.902+0000 7fd3efe00640 1 --2- 172.21.15.28:0/2441425073 >> v2:172.21.15.189:3300/0 conn(0x7fd3f80c40a0 0x7fd3f80b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.903+0000 7fd3efe00640 1 -- 172.21.15.28:0/2441425073 >> v1:172.21.15.189:6789/0 conn(0x7fd3f815b0b0 legacy=0x7fd3f806c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.903+0000 7fd3efe00640 1 -- 172.21.15.28:0/2441425073 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f80b2930 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3efe00640 1 --2- 172.21.15.28:0/2441425073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f815ab70 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80ce360 tx=0x7fd3e002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc059c93b8f9729 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3f7400640 1 -- 172.21.15.28:0/2441425073 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032440 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3f7400640 1 -- 172.21.15.28:0/2441425073 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e00325a0 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3f7400640 1 -- 172.21.15.28:0/2441425073 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032930 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 -- 172.21.15.28:0/2441425073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80b1ba0 msgr2=0x7fd3f815ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2441425073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f815ab70 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7fd3f80ce360 tx=0x7fd3e002a270 comp rx=0 tx=0).stop 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 -- 172.21.15.28:0/2441425073 shutdown_connections 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2441425073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f815ab70 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2441425073 >> v2:172.21.15.189:3300/0 conn(0x7fd3f80c40a0 0x7fd3f80b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.904+0000 7fd3fd800640 1 -- 172.21.15.28:0/2441425073 >> 172.21.15.28:0/2441425073 conn(0x7fd3f8073470 msgr2=0x7fd3f8069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.905+0000 7fd3fd800640 1 -- 172.21.15.28:0/2441425073 shutdown_connections 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.905+0000 7fd3fd800640 1 -- 172.21.15.28:0/2441425073 wait complete. 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.906+0000 7fd3fd800640 1 Processor -- start 2024-09-13T17:23:52.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.906+0000 7fd3fd800640 1 -- start start 2024-09-13T17:23:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f80a8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 0x7fd3f815a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80c69f0 con 0x7fd3f80c40a0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80a9bb0 con 0x7fd3f80b1ba0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3fd800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80a95f0 con 0x7fd3f815b0b0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35446/0 (socket says 172.21.15.28:35446) 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.907+0000 7fd3efe00640 1 -- 172.21.15.28:0/2884380415 learned_addr learned my addr 172.21.15.28:0/2884380415 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3f7e00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f80a8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 -- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 msgr2=0x7fd3f815a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3fce00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 0x7fd3f815a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 0x7fd3f815a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 -- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 msgr2=0x7fd3f80a8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f80a8ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 -- 172.21.15.28:0/2884380415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00094d0 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3f7e00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f80a8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3fce00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 0x7fd3f815a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3efe00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd3e002a2b0 tx=0x7fd3e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.908+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032440 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.909+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0030ba0 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.909+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e003d440 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.909+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f80a98d0 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.909+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f80c76e0 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.910+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f80c7260 con 0x7fd3f80c40a0 2024-09-13T17:23:52.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.911+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd3e0039040 con 0x7fd3f80c40a0 2024-09-13T17:23:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.911+0000 7fd3f4c00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.912+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3e00a9f80 con 0x7fd3f80c40a0 2024-09-13T17:23:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.912+0000 7fd3f7e00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:52.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.913+0000 7fd3f7e00640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8006fd0 tx=0x7fd3e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:52.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:52.916+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fd3f80c7260 con 0x7fd3f80c40a0 2024-09-13T17:23:53.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.108+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3f80c7260 con 0x7fd3f80c40a0 2024-09-13T17:23:53.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.108+0000 7fd3f4c00640 1 -- 172.21.15.28:0/2884380415 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v219) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd3f80c7260 con 0x7fd3f80c40a0 2024-09-13T17:23:53.109 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:23:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.110+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 msgr2=0x7fd3d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.110+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8006fd0 tx=0x7fd3e8008040 comp rx=0 tx=0).stop 2024-09-13T17:23:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.110+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 msgr2=0x7fd3f80c64b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.110+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd3e002a2b0 tx=0x7fd3e00046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 shutdown_connections 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd3d406ac00 0x7fd3d406d0c0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd3f815b0b0 0x7fd3f815a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd3f80c40a0 0x7fd3f80c64b0 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 --2- 172.21.15.28:0/2884380415 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd3f80b1ba0 0x7fd3f80a8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 >> 172.21.15.28:0/2884380415 conn(0x7fd3f8073470 msgr2=0x7fd3f80c95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 shutdown_connections 2024-09-13T17:23:53.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.111+0000 7fd3fd800640 1 -- 172.21.15.28:0/2884380415 wait complete. 2024-09-13T17:23:53.128 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:23:53.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:53.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:23:53.558 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:23:53.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:23:53.559 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:23:53.595 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:53.596 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:53.631 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:53.632 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:23:53.632 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:53.671 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:23:53.671 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:23:53.671 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:23:53.706 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:23:53.706 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.866+0000 7fa45f000640 1 Processor -- start 2024-09-13T17:23:53.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.866+0000 7fa45f000640 1 -- start start 2024-09-13T17:23:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45f000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa4580cabc0 0x7fa45806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45f000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45815e100 con 0x7fa458070cc0 2024-09-13T17:23:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45f000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45815e3e0 con 0x7fa45815a540 2024-09-13T17:23:53.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45f000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4580adb00 con 0x7fa4580cabc0 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa458070cc0 0x7fa45806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37842/0 (socket says 172.21.15.28:37842) 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa45d000640 1 -- 172.21.15.28:0/3994847149 learned_addr learned my addr 172.21.15.28:0/3994847149 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.867+0000 7fa44f400640 1 --2- 172.21.15.28:0/3994847149 >> v2:172.21.15.189:3300/0 conn(0x7fa4580cabc0 0x7fa45806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562215869 0 0) 0x7fa45815e100 con 0x7fa458070cc0 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa43c003520 con 0x7fa458070cc0 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa44f400640 1 -- 172.21.15.28:0/3994847149 >> v1:172.21.15.189:6789/0 conn(0x7fa45815a540 legacy=0x7fa45815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235318053 0 0) 0x7fa45815e3e0 con 0x7fa45815a540 2024-09-13T17:23:53.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa44f400640 1 -- 172.21.15.28:0/3994847149 >> v1:172.21.15.28:6789/0 conn(0x7fa458070cc0 legacy=0x7fa45806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.868+0000 7fa44f400640 1 -- 172.21.15.28:0/3994847149 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45815ef80 con 0x7fa4580cabc0 2024-09-13T17:23:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.869+0000 7fa44f400640 1 --2- 172.21.15.28:0/3994847149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4580cabc0 0x7fa45806c430 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa4400097a0 tx=0x7fa44002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2e53e8865465f97 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.869+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440032440 con 0x7fa4580cabc0 2024-09-13T17:23:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.869+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4400325a0 con 0x7fa4580cabc0 2024-09-13T17:23:53.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.869+0000 7fa457400640 1 -- 172.21.15.28:0/3994847149 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440032930 con 0x7fa4580cabc0 2024-09-13T17:23:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 -- 172.21.15.28:0/3994847149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4580cabc0 msgr2=0x7fa45806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 --2- 172.21.15.28:0/3994847149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4580cabc0 0x7fa45806c430 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa4400097a0 tx=0x7fa44002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 -- 172.21.15.28:0/3994847149 shutdown_connections 2024-09-13T17:23:53.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 --2- 172.21.15.28:0/3994847149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa4580cabc0 0x7fa45806c430 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 -- 172.21.15.28:0/3994847149 >> 172.21.15.28:0/3994847149 conn(0x7fa458069e80 msgr2=0x7fa45806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 -- 172.21.15.28:0/3994847149 shutdown_connections 2024-09-13T17:23:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.870+0000 7fa45f000640 1 -- 172.21.15.28:0/3994847149 wait complete. 2024-09-13T17:23:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.872+0000 7fa45f000640 1 Processor -- start 2024-09-13T17:23:53.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.872+0000 7fa45f000640 1 -- start start 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa458070cc0 0x7fa458146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa45815a540 0x7fa4580aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa458151db0 con 0x7fa4580cabc0 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa458150570 con 0x7fa458070cc0 2024-09-13T17:23:53.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45f000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa458147400 con 0x7fa45815a540 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44516/0 (socket says 172.21.15.28:44516) 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa45d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa45815a540 0x7fa4580aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 -- 172.21.15.28:0/3974600656 learned_addr learned my addr 172.21.15.28:0/3974600656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa457e00640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa458070cc0 0x7fa458146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 -- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa45815a540 msgr2=0x7fa4580aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa45815a540 0x7fa4580aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 -- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa458070cc0 msgr2=0x7fa458146ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa458070cc0 0x7fa458146ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:53.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.873+0000 7fa44f400640 1 -- 172.21.15.28:0/3974600656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa440009400 con 0x7fa4580cabc0 2024-09-13T17:23:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa44f400640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fa440035040 tx=0x7fa4400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440032440 con 0x7fa4580cabc0 2024-09-13T17:23:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa440030b00 con 0x7fa4580cabc0 2024-09-13T17:23:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440030e90 con 0x7fa4580cabc0 2024-09-13T17:23:53.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4580b1f60 con 0x7fa4580cabc0 2024-09-13T17:23:53.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.874+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4580b2520 con 0x7fa4580cabc0 2024-09-13T17:23:53.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.875+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45806c960 con 0x7fa4580cabc0 2024-09-13T17:23:53.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.876+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa440039040 con 0x7fa4580cabc0 2024-09-13T17:23:53.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.877+0000 7fa454c00640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 0x7fa43406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:53.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.877+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4400a9db0 con 0x7fa4580cabc0 2024-09-13T17:23:53.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.877+0000 7fa457e00640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 0x7fa43406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:53.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.878+0000 7fa457e00640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 0x7fa43406d0f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa448000f60 tx=0x7fa448009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:53.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:53.881+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa4400ae070 con 0x7fa4580cabc0 2024-09-13T17:23:54.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.093+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa458151ef0 con 0x7fa4580cabc0 2024-09-13T17:23:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.094+0000 7fa454c00640 1 -- 172.21.15.28:0/3974600656 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7fa458151ef0 con 0x7fa4580cabc0 2024-09-13T17:23:54.095 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:54.095 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:54.095 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 msgr2=0x7fa43406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 0x7fa43406d0f0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7fa448000f60 tx=0x7fa448009040 comp rx=0 tx=0).stop 2024-09-13T17:23:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 msgr2=0x7fa4580b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7fa440035040 tx=0x7fa4400046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 shutdown_connections 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa43406ac30 0x7fa43406d0f0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa45815a540 0x7fa4580aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa4580cabc0 0x7fa4580b1710 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 --2- 172.21.15.28:0/3974600656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa458070cc0 0x7fa458146ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 >> 172.21.15.28:0/3974600656 conn(0x7fa458069e80 msgr2=0x7fa4580ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 shutdown_connections 2024-09-13T17:23:54.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.096+0000 7fa45f000640 1 -- 172.21.15.28:0/3974600656 wait complete. 2024-09-13T17:23:54.112 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.267+0000 7f67cd200640 1 Processor -- start 2024-09-13T17:23:54.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.267+0000 7f67cd200640 1 -- start start 2024-09-13T17:23:54.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.268+0000 7f67cd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67c80713d0 0x7f67c806c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.268+0000 7f67cd200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c815e100 con 0x7f67c80cabc0 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.268+0000 7f67cd200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67c815e3e0 con 0x7f67c815a540 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.268+0000 7f67cd200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67c80adb00 con 0x7f67c80713d0 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67c80713d0 0x7f67c806c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f67c80713d0 0x7f67c806c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44530/0 (socket says 172.21.15.28:44530) 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67bd400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f67c80cabc0 0x7f67c8070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37858/0 (socket says 172.21.15.28:37858) 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 -- 172.21.15.28:0/245340332 learned_addr learned my addr 172.21.15.28:0/245340332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:54.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 -- 172.21.15.28:0/245340332 >> v1:172.21.15.189:6789/0 conn(0x7f67c815a540 legacy=0x7f67c815e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 -- 172.21.15.28:0/245340332 >> v1:172.21.15.28:6789/0 conn(0x7f67c80cabc0 legacy=0x7f67c8070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.269+0000 7f67c6800640 1 -- 172.21.15.28:0/245340332 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c815ef80 con 0x7f67c80713d0 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.270+0000 7f67c6800640 1 --2- 172.21.15.28:0/245340332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80713d0 0x7f67c806c430 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009e50 tx=0x7f67c002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da99f159f7100e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.270+0000 7f67c5400640 1 -- 172.21.15.28:0/245340332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0031600 con 0x7f67c80713d0 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67c5400640 1 -- 172.21.15.28:0/245340332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67c002d050 con 0x7f67c80713d0 2024-09-13T17:23:54.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67c5400640 1 -- 172.21.15.28:0/245340332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67c0031980 con 0x7f67c80713d0 2024-09-13T17:23:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 -- 172.21.15.28:0/245340332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80713d0 msgr2=0x7f67c806c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 --2- 172.21.15.28:0/245340332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80713d0 0x7f67c806c430 secure :-1 s=READY pgs=2643 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009e50 tx=0x7f67c002a730 comp rx=0 tx=0).stop 2024-09-13T17:23:54.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 -- 172.21.15.28:0/245340332 shutdown_connections 2024-09-13T17:23:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 --2- 172.21.15.28:0/245340332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80713d0 0x7f67c806c430 unknown :-1 s=CLOSED pgs=2643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 -- 172.21.15.28:0/245340332 >> 172.21.15.28:0/245340332 conn(0x7f67c8069e80 msgr2=0x7f67c806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 -- 172.21.15.28:0/245340332 shutdown_connections 2024-09-13T17:23:54.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.271+0000 7f67cd200640 1 -- 172.21.15.28:0/245340332 wait complete. 2024-09-13T17:23:54.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.273+0000 7f67cd200640 1 Processor -- start 2024-09-13T17:23:54.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.273+0000 7f67cd200640 1 -- start start 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67c80713d0 0x7f67c814dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67c815a540 0x7f67c80b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c8155120 con 0x7f67c80cabc0 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c80b2dd0 con 0x7f67c815a540 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67cd200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67c8154360 con 0x7f67c80713d0 2024-09-13T17:23:54.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44534/0 (socket says 172.21.15.28:44534) 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67bd400640 1 -- 172.21.15.28:0/2355973779 learned_addr learned my addr 172.21.15.28:0/2355973779 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67c6800640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67c815a540 0x7f67c80b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.274+0000 7f67c5e00640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67c80713d0 0x7f67c814dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 -- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67c80713d0 msgr2=0x7f67c814dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67c80713d0 0x7f67c814dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 -- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67c815a540 msgr2=0x7f67c80b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67c815a540 0x7f67c80b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 -- 172.21.15.28:0/2355973779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c0009b30 con 0x7f67c80cabc0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bd400640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f67b000c4d0 tx=0x7f67b000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0007480 con 0x7f67c80cabc0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b00075e0 con 0x7f67c80cabc0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c8154640 con 0x7f67c80cabc0 2024-09-13T17:23:54.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b00049e0 con 0x7f67c80cabc0 2024-09-13T17:23:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.275+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c8154c00 con 0x7f67c80cabc0 2024-09-13T17:23:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.276+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c806c430 con 0x7f67c80cabc0 2024-09-13T17:23:54.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.277+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f67b0004b40 con 0x7f67c80cabc0 2024-09-13T17:23:54.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.278+0000 7f67bea00640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 0x7f67a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.278+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f67b0088950 con 0x7f67c80cabc0 2024-09-13T17:23:54.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.278+0000 7f67c5e00640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 0x7f67a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.279+0000 7f67c5e00640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 0x7f67a406d0c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f67b8000f60 tx=0x7f67b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.282+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f67b0053a90 con 0x7f67c80cabc0 2024-09-13T17:23:54.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.461+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67c8155c50 con 0x7f67c80cabc0 2024-09-13T17:23:54.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.462+0000 7f67bea00640 1 -- 172.21.15.28:0/2355973779 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v219) v1 ==== 74+0+16519 (secure 0 0 0) 0x7f67b0018090 con 0x7f67c80cabc0 2024-09-13T17:23:54.464 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:54.464 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":219,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:42.631521+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.464+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 msgr2=0x7f67a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.464+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 0x7f67a406d0c0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f67b8000f60 tx=0x7f67b8009040 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.464+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 msgr2=0x7f67c80b0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.464+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f67b000c4d0 tx=0x7f67b000c990 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 shutdown_connections 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67a406ac00 0x7f67a406d0c0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67c815a540 0x7f67c80b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f67c80cabc0 0x7f67c80b0450 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 --2- 172.21.15.28:0/2355973779 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f67c80713d0 0x7f67c814dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 >> 172.21.15.28:0/2355973779 conn(0x7f67c8069e80 msgr2=0x7f67c80ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 shutdown_connections 2024-09-13T17:23:54.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.465+0000 7f67cd200640 1 -- 172.21.15.28:0/2355973779 wait complete. 2024-09-13T17:23:54.481 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:23:39.609113+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '218', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:23:40.623957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '219', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:23:54.482 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:23:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.639+0000 7f2b01200640 1 Processor -- start 2024-09-13T17:23:54.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.639+0000 7f2b01200640 1 -- start start 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 0x7f2afc158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 0x7f2afc15c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2afc15cde0 0x7f2afc15d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc159630 con 0x7f2afc15cde0 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2afc1598c0 con 0x7f2afc06ccd0 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.640+0000 7f2b01200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc159ba0 con 0x7f2afc159210 2024-09-13T17:23:54.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2afc15cde0 0x7f2afc15d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2afc15cde0 0x7f2afc15d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44538/0 (socket says 172.21.15.28:44538) 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 -- 172.21.15.28:0/2213880221 learned_addr learned my addr 172.21.15.28:0/2213880221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2213880221 >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 0x7f2afc158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2af9400640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 0x7f2afc15c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 -- 172.21.15.28:0/2213880221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 msgr2=0x7f2afc15c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 0x7f2afc15c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 -- 172.21.15.28:0/2213880221 >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 msgr2=0x7f2afc158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 --2- 172.21.15.28:0/2213880221 >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 0x7f2afc158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2afa800640 1 -- 172.21.15.28:0/2213880221 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2afc159e80 con 0x7f2afc15cde0 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2213880221 >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 0x7f2afc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:54.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.641+0000 7f2af9400640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 0x7f2afc15c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.642+0000 7f2afa800640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc15cde0 0x7f2afc15d270 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f2afc067f80 tx=0x7f2af402c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6382c812b030fb2a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.642+0000 7f2af8a00640 1 -- 172.21.15.28:0/2213880221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af402f070 con 0x7f2afc15cde0 2024-09-13T17:23:54.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.642+0000 7f2af8a00640 1 -- 172.21.15.28:0/2213880221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af400b2e0 con 0x7f2afc15cde0 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.642+0000 7f2af8a00640 1 -- 172.21.15.28:0/2213880221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af4009820 con 0x7f2afc15cde0 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 -- 172.21.15.28:0/2213880221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc15cde0 msgr2=0x7f2afc15d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc15cde0 0x7f2afc15d270 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f2afc067f80 tx=0x7f2af402c8a0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 -- 172.21.15.28:0/2213880221 shutdown_connections 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc15cde0 0x7f2afc15d270 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 --2- 172.21.15.28:0/2213880221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc159210 0x7f2afc15c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 --2- 172.21.15.28:0/2213880221 >> v2:172.21.15.189:3300/0 conn(0x7f2afc06ccd0 0x7f2afc158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 -- 172.21.15.28:0/2213880221 >> 172.21.15.28:0/2213880221 conn(0x7f2afc0b0270 msgr2=0x7f2afc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 -- 172.21.15.28:0/2213880221 shutdown_connections 2024-09-13T17:23:54.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.643+0000 7f2b01200640 1 -- 172.21.15.28:0/2213880221 wait complete. 2024-09-13T17:23:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.645+0000 7f2b01200640 1 Processor -- start 2024-09-13T17:23:54.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.645+0000 7f2b01200640 1 -- start start 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 0x7f2afc147070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 0x7f2afc06b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc1e4430 con 0x7f2afc159210 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0cce60 con 0x7f2afc15cde0 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2b01200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2afc0cd0b0 con 0x7f2afc06ccd0 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2af9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2af9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44546/0 (socket says 172.21.15.28:44546) 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2af9400640 1 -- 172.21.15.28:0/2959786149 learned_addr learned my addr 172.21.15.28:0/2959786149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 0x7f2afc147070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.646+0000 7f2af9400640 1 -- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 msgr2=0x7f2afc147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9400640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 0x7f2afc147070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2afa800640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 0x7f2afc06b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9400640 1 -- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 msgr2=0x7f2afc06b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9400640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 0x7f2afc06b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9400640 1 -- 172.21.15.28:0/2959786149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af400bd40 con 0x7f2afc159210 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 0x7f2afc147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2afa800640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 0x7f2afc06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:54.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af9400640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e810 tx=0x7f2ae800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8004030 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae8004190 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8004520 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2afc0cd390 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.647+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2afc153dd0 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.648+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2afc06ffa0 con 0x7f2afc159210 2024-09-13T17:23:54.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.649+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ae800c6d0 con 0x7f2afc159210 2024-09-13T17:23:54.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.650+0000 7f2af2000640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 0x7f2adc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:54.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.650+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ae8089100 con 0x7f2afc159210 2024-09-13T17:23:54.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.651+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 0x7f2adc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:54.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.651+0000 7f2af9e00640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 0x7f2adc06d0f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2aec006fd0 tx=0x7f2aec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:54.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.654+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2ae80541b0 con 0x7f2afc159210 2024-09-13T17:23:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.832+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2afc158830 con 0x7f2afc159210 2024-09-13T17:23:54.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.832+0000 7f2af2000640 1 -- 172.21.15.28:0/2959786149 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v219) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ae80577d0 con 0x7f2afc159210 2024-09-13T17:23:54.833 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:23:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 msgr2=0x7f2adc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 0x7f2adc06d0f0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2aec006fd0 tx=0x7f2aec008040 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 msgr2=0x7f2afc0ca670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f2ae800e810 tx=0x7f2ae800ecd0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 shutdown_connections 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2adc06ac30 0x7f2adc06d0f0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2afc15cde0 0x7f2afc06b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2afc159210 0x7f2afc0ca670 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 --2- 172.21.15.28:0/2959786149 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2afc06ccd0 0x7f2afc147070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 >> 172.21.15.28:0/2959786149 conn(0x7f2afc0b0270 msgr2=0x7f2afc0b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 shutdown_connections 2024-09-13T17:23:54.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:54.834+0000 7f2b01200640 1 -- 172.21.15.28:0/2959786149 wait complete. 2024-09-13T17:23:54.850 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:23:55.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.007+0000 7f0529800640 1 Processor -- start 2024-09-13T17:23:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.007+0000 7f0529800640 1 -- start start 2024-09-13T17:23:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.008+0000 7f0529800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f052415c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.008+0000 7f0529800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052415ab60 con 0x7f0524159770 2024-09-13T17:23:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.008+0000 7f0529800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f052406f920 con 0x7f0524070430 2024-09-13T17:23:55.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.008+0000 7f0529800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052415b290 con 0x7f052406f3f0 2024-09-13T17:23:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0522a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0524159770 0x7f052406fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37860/0 (socket says 172.21.15.28:37860) 2024-09-13T17:23:55.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0522a00640 1 -- 172.21.15.28:0/2555875012 learned_addr learned my addr 172.21.15.28:0/2555875012 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0523400640 1 --2- 172.21.15.28:0/2555875012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f052415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074921352 0 0) 0x7f0514001110 con 0x7f0524159770 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508003520 con 0x7f0524159770 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838977673 0 0) 0x7f052406f920 con 0x7f0524070430 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514001110 con 0x7f0524070430 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1406612554 0 0) 0x7f0508003520 con 0x7f0524159770 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0523400640 1 -- 172.21.15.28:0/2555875012 >> v1:172.21.15.189:6789/0 conn(0x7f0524070430 legacy=0x7f052415bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0523400640 1 -- 172.21.15.28:0/2555875012 >> v1:172.21.15.28:6789/0 conn(0x7f0524159770 legacy=0x7f052406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.009+0000 7f0523400640 1 -- 172.21.15.28:0/2555875012 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052415b570 con 0x7f052406f3f0 2024-09-13T17:23:55.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.010+0000 7f0523400640 1 --2- 172.21.15.28:0/2555875012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f052415c480 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f051c009dc0 tx=0x7f051c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8612cb36df6df652 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0514001800 con 0x7f052406f3f0 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c02d050 con 0x7f052406f3f0 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0521600640 1 -- 172.21.15.28:0/2555875012 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c007760 con 0x7f052406f3f0 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0529800640 1 -- 172.21.15.28:0/2555875012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 msgr2=0x7f052415c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0529800640 1 --2- 172.21.15.28:0/2555875012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f052415c480 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f051c009dc0 tx=0x7f051c02a5d0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0529800640 1 -- 172.21.15.28:0/2555875012 shutdown_connections 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0529800640 1 --2- 172.21.15.28:0/2555875012 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f052415c480 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.011+0000 7f0529800640 1 -- 172.21.15.28:0/2555875012 >> 172.21.15.28:0/2555875012 conn(0x7f05240b0270 msgr2=0x7f05240b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.012+0000 7f0529800640 1 -- 172.21.15.28:0/2555875012 shutdown_connections 2024-09-13T17:23:55.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.012+0000 7f0529800640 1 -- 172.21.15.28:0/2555875012 wait complete. 2024-09-13T17:23:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.013+0000 7f0529800640 1 Processor -- start 2024-09-13T17:23:55.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.013+0000 7f0529800640 1 -- start start 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f0524155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0524159770 0x7f05240af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05240b2970 con 0x7f0524070430 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05240b2040 con 0x7f0524159770 2024-09-13T17:23:55.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0529800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05240b2320 con 0x7f052406f3f0 2024-09-13T17:23:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0522000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0522000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44558/0 (socket says 172.21.15.28:44558) 2024-09-13T17:23:55.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.014+0000 7f0522000640 1 -- 172.21.15.28:0/1429533599 learned_addr learned my addr 172.21.15.28:0/1429533599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0523400640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0524159770 0x7f05240af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 -- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 msgr2=0x7f0524155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f0524155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 -- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0524159770 msgr2=0x7f05240af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0524159770 0x7f05240af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 -- 172.21.15.28:0/1429533599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c009a70 con 0x7f0524070430 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0522000640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f051000c490 tx=0x7f051000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05100052d0 con 0x7f0524070430 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510005430 con 0x7f0524070430 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0510012870 con 0x7f0524070430 2024-09-13T17:23:55.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.015+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05240b2600 con 0x7f0524070430 2024-09-13T17:23:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.016+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052414a5b0 con 0x7f0524070430 2024-09-13T17:23:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.016+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f052414a7a0 con 0x7f0524070430 2024-09-13T17:23:55.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.017+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f05100129d0 con 0x7f0524070430 2024-09-13T17:23:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.018+0000 7f051aa00640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 0x7f050406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.018+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0510088ff0 con 0x7f0524070430 2024-09-13T17:23:55.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.019+0000 7f0522a00640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 0x7f050406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.019+0000 7f0522a00640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 0x7f050406d0c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0514001510 tx=0x7f05140013e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.023+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f05100bc9c0 con 0x7f0524070430 2024-09-13T17:23:55.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.203+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0524070d10 con 0x7f0524070430 2024-09-13T17:23:55.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.204+0000 7f051aa00640 1 -- 172.21.15.28:0/1429533599 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v220) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0510054150 con 0x7f0524070430 2024-09-13T17:23:55.205 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:23:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 msgr2=0x7f050406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 0x7f050406d0c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f0514001510 tx=0x7f05140013e0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 msgr2=0x7f052414a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f051000c490 tx=0x7f051000c950 comp rx=0 tx=0).stop 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 shutdown_connections 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f050406ac00 0x7f050406d0c0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0524159770 0x7f05240af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0524070430 0x7f052414a070 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 --2- 172.21.15.28:0/1429533599 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f052406f3f0 0x7f0524155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 >> 172.21.15.28:0/1429533599 conn(0x7f05240b0270 msgr2=0x7f05240ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 shutdown_connections 2024-09-13T17:23:55.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.206+0000 7f0529800640 1 -- 172.21.15.28:0/1429533599 wait complete. 2024-09-13T17:23:55.223 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:23:55.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.382+0000 7f6c7fe00640 1 Processor -- start 2024-09-13T17:23:55.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.383+0000 7f6c7fe00640 1 -- start start 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 0x7f6c80072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800c78e0 con 0x7f6c800cc530 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7fe00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800cbe40 con 0x7f6c800cc910 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80070c90 con 0x7f6c800b0e00 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6c800cc530 0x7f6c8015d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37864/0 (socket says 172.21.15.28:37864) 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7ea00640 1 -- 172.21.15.28:0/642502237 learned_addr learned my addr 172.21.15.28:0/642502237 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:55.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7f400640 1 --2- 172.21.15.28:0/642502237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 0x7f6c80072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.384+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208223667 0 0) 0x7f6c800c78e0 con 0x7f6c800cc530 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c64003520 con 0x7f6c800cc530 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925306673 0 0) 0x7f6c800cbe40 con 0x7f6c800cc910 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c800c78e0 con 0x7f6c800cc910 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7f400640 1 -- 172.21.15.28:0/642502237 >> v1:172.21.15.189:6789/0 conn(0x7f6c800cc910 legacy=0x7f6c800b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7f400640 1 -- 172.21.15.28:0/642502237 >> v1:172.21.15.28:6789/0 conn(0x7f6c800cc530 legacy=0x7f6c8015d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.385+0000 7f6c7f400640 1 -- 172.21.15.28:0/642502237 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c80071000 con 0x7f6c800b0e00 2024-09-13T17:23:55.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.386+0000 7f6c7f400640 1 --2- 172.21.15.28:0/642502237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 0x7f6c80072060 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009dc0 tx=0x7f6c7802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5ea25e94719b35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.386+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78007740 con 0x7f6c800b0e00 2024-09-13T17:23:55.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.386+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7802d050 con 0x7f6c800b0e00 2024-09-13T17:23:55.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.386+0000 7f6c7e000640 1 -- 172.21.15.28:0/642502237 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78007ac0 con 0x7f6c800b0e00 2024-09-13T17:23:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 -- 172.21.15.28:0/642502237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 msgr2=0x7f6c80072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/642502237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 0x7f6c80072060 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009dc0 tx=0x7f6c7802a760 comp rx=0 tx=0).stop 2024-09-13T17:23:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 -- 172.21.15.28:0/642502237 shutdown_connections 2024-09-13T17:23:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/642502237 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800b0e00 0x7f6c80072060 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 -- 172.21.15.28:0/642502237 >> 172.21.15.28:0/642502237 conn(0x7f6c80069e80 msgr2=0x7f6c8006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 -- 172.21.15.28:0/642502237 shutdown_connections 2024-09-13T17:23:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.387+0000 7f6c7fe00640 1 -- 172.21.15.28:0/642502237 wait complete. 2024-09-13T17:23:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.389+0000 7f6c7fe00640 1 Processor -- start 2024-09-13T17:23:55.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.389+0000 7f6c7fe00640 1 -- start start 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800cc530 0x7f6c801603f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c800cc910 0x7f6c80160930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80159f00 con 0x7f6c800b0e00 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80159280 con 0x7f6c800cc910 2024-09-13T17:23:55.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c80159560 con 0x7f6c800cc530 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44562/0 (socket says 172.21.15.28:44562) 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 -- 172.21.15.28:0/2718247578 learned_addr learned my addr 172.21.15.28:0/2718247578 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7f400640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c800cc910 0x7f6c80160930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c77e00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800cc530 0x7f6c801603f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 -- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800cc530 msgr2=0x7f6c801603f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800cc530 0x7f6c801603f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 -- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c800cc910 msgr2=0x7f6c80160930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.390+0000 7f6c7ea00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c800cc910 0x7f6c80160930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c7ea00640 1 -- 172.21.15.28:0/2718247578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78009a70 con 0x7f6c800b0e00 2024-09-13T17:23:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c7ea00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000c500 tx=0x7f6c7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70005ce0 con 0x7f6c800b0e00 2024-09-13T17:23:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c70005e60 con 0x7f6c800b0e00 2024-09-13T17:23:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80159840 con 0x7f6c800b0e00 2024-09-13T17:23:55.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c70007330 con 0x7f6c800b0e00 2024-09-13T17:23:55.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.391+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c801584c0 con 0x7f6c800b0e00 2024-09-13T17:23:55.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.392+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80071860 con 0x7f6c800b0e00 2024-09-13T17:23:55.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.393+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c70007490 con 0x7f6c800b0e00 2024-09-13T17:23:55.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.394+0000 7f6c77400640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 0x7f6c5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.394+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6c70088980 con 0x7f6c800b0e00 2024-09-13T17:23:55.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.395+0000 7f6c77e00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 0x7f6c5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.395+0000 7f6c77e00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 0x7f6c5c06d0c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6c8006bf30 tx=0x7f6c68009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.398+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6c70053c80 con 0x7f6c800b0e00 2024-09-13T17:23:55.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.581+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c80072060 con 0x7f6c800b0e00 2024-09-13T17:23:55.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.582+0000 7f6c77400640 1 -- 172.21.15.28:0/2718247578 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v220) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c70017020 con 0x7f6c800b0e00 2024-09-13T17:23:55.583 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:23:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.583+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 msgr2=0x7f6c5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 0x7f6c5c06d0c0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f6c8006bf30 tx=0x7f6c68009040 comp rx=0 tx=0).stop 2024-09-13T17:23:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 msgr2=0x7f6c801439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7f6c7000c500 tx=0x7f6c7000c9c0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 shutdown_connections 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c5c06ac00 0x7f6c5c06d0c0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c800cc910 0x7f6c80160930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c800cc530 0x7f6c801603f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 --2- 172.21.15.28:0/2718247578 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c800b0e00 0x7f6c801439b0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 >> 172.21.15.28:0/2718247578 conn(0x7f6c80069e80 msgr2=0x7f6c800cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 shutdown_connections 2024-09-13T17:23:55.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.584+0000 7f6c7fe00640 1 -- 172.21.15.28:0/2718247578 wait complete. 2024-09-13T17:23:55.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:55.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.784+0000 7f34bbc00640 1 Processor -- start 2024-09-13T17:23:55.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.784+0000 7f34bbc00640 1 -- start start 2024-09-13T17:23:55.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.785+0000 7f34bbc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 0x7f34b406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.785+0000 7f34bbc00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34b4070cc0 0x7f34b4071170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.785+0000 7f34bbc00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b415e100 con 0x7f34b415a540 2024-09-13T17:23:55.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.785+0000 7f34bbc00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b40716b0 con 0x7f34b4070cc0 2024-09-13T17:23:55.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.785+0000 7f34bbc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40adb00 con 0x7f34b40cabc0 2024-09-13T17:23:55.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34b4070cc0 0x7f34b4071170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b3e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 0x7f34b406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f34b4070cc0 0x7f34b4071170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:50352/0 (socket says 172.21.15.28:50352) 2024-09-13T17:23:55.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f34b415a540 0x7f34b415e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:32880/0 (socket says 172.21.15.28:32880) 2024-09-13T17:23:55.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 -- 172.21.15.28:0/596261309 learned_addr learned my addr 172.21.15.28:0/596261309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 -- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 msgr2=0x7f34b406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 0x7f34b406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 -- 172.21.15.28:0/596261309 >> v1:172.21.15.189:6789/0 conn(0x7f34b415a540 legacy=0x7f34b415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b9c00640 1 -- 172.21.15.28:0/596261309 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b415e3e0 con 0x7f34b4070cc0 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b3e00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 0x7f34b406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:55.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.786+0000 7f34b3400640 1 -- 172.21.15.28:0/596261309 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559462048 0 0) 0x7f34b415e100 con 0x7f34b415a540 2024-09-13T17:23:55.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.787+0000 7f34b9c00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4070cc0 0x7f34b4071170 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009e80 tx=0x7f34ac02a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=661c8de220dd5a50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34b3400640 1 -- 172.21.15.28:0/596261309 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac031600 con 0x7f34b4070cc0 2024-09-13T17:23:55.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34b3400640 1 -- 172.21.15.28:0/596261309 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ac02d050 con 0x7f34b4070cc0 2024-09-13T17:23:55.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34b3400640 1 -- 172.21.15.28:0/596261309 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac031980 con 0x7f34b4070cc0 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 -- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4070cc0 msgr2=0x7f34b4071170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4070cc0 0x7f34b4071170 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f34ac009e80 tx=0x7f34ac02a6b0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 -- 172.21.15.28:0/596261309 shutdown_connections 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b4070cc0 0x7f34b4071170 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 --2- 172.21.15.28:0/596261309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b40cabc0 0x7f34b406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 -- 172.21.15.28:0/596261309 >> 172.21.15.28:0/596261309 conn(0x7f34b4069e80 msgr2=0x7f34b406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 -- 172.21.15.28:0/596261309 shutdown_connections 2024-09-13T17:23:55.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.788+0000 7f34bbc00640 1 -- 172.21.15.28:0/596261309 wait complete. 2024-09-13T17:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.790+0000 7f34bbc00640 1 Processor -- start 2024-09-13T17:23:55.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.790+0000 7f34bbc00640 1 -- start start 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b4070cc0 0x7f34b40b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b415a540 0x7f34b4127a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40aff80 con 0x7f34b40cabc0 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40af370 con 0x7f34b415a540 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34bbc00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b40af650 con 0x7f34b4070cc0 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44578/0 (socket says 172.21.15.28:44578) 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34b3e00640 1 -- 172.21.15.28:0/4173256620 learned_addr learned my addr 172.21.15.28:0/4173256620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34b9200640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b4070cc0 0x7f34b40b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.791+0000 7f34b9c00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b415a540 0x7f34b4127a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 -- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b4070cc0 msgr2=0x7f34b40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b4070cc0 0x7f34b40b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 -- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b415a540 msgr2=0x7f34b4127a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b415a540 0x7f34b4127a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 -- 172.21.15.28:0/4173256620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a0009400 con 0x7f34b40cabc0 2024-09-13T17:23:55.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b3e00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f34a000ebb0 tx=0x7f34a000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000eeb0 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a000cc30 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0013470 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ac009b30 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.792+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b40afc10 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.793+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b4067720 con 0x7f34b40cabc0 2024-09-13T17:23:55.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.794+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f34a00135d0 con 0x7f34b40cabc0 2024-09-13T17:23:55.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.795+0000 7f34b0c00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 0x7f349406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:55.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.795+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f34a0089680 con 0x7f34b40cabc0 2024-09-13T17:23:55.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.796+0000 7f34b9200640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 0x7f349406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:55.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.796+0000 7f34b9200640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 0x7f349406d0c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f34a8007fd0 tx=0x7f34a8007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:55.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:55.799+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f34a0054830 con 0x7f34b40cabc0 2024-09-13T17:23:56.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.012+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b4071170 con 0x7f34b40cabc0 2024-09-13T17:23:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.013+0000 7f34b0c00640 1 -- 172.21.15.28:0/4173256620 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f34a0057e50 con 0x7f34b40cabc0 2024-09-13T17:23:56.014 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:56.014 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:56.014 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 msgr2=0x7f349406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 0x7f349406d0c0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f34a8007fd0 tx=0x7f34a8007de0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 msgr2=0x7f34b4127510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7f34a000ebb0 tx=0x7f34a000c520 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 shutdown_connections 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f349406ac00 0x7f349406d0c0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b415a540 0x7f34b4127a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b40cabc0 0x7f34b4127510 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 --2- 172.21.15.28:0/4173256620 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f34b4070cc0 0x7f34b40b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 >> 172.21.15.28:0/4173256620 conn(0x7f34b4069e80 msgr2=0x7f34b406b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 shutdown_connections 2024-09-13T17:23:56.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.015+0000 7f34bbc00640 1 -- 172.21.15.28:0/4173256620 wait complete. 2024-09-13T17:23:56.032 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:56.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.192+0000 7f7f68a00640 1 Processor -- start 2024-09-13T17:23:56.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.192+0000 7f7f68a00640 1 -- start start 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f68a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f640ce560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f68a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f640b3970 con 0x7f7f640cdfa0 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f68a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f64159430 con 0x7f7f6406f440 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f68a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6415d760 con 0x7f7f6415d340 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f61e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f640cdfa0 0x7f7f640b3160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37874/0 (socket says 172.21.15.28:37874) 2024-09-13T17:23:56.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.193+0000 7f7f61e00640 1 -- 172.21.15.28:0/2630524342 learned_addr learned my addr 172.21.15.28:0/2630524342 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f5be00640 1 --2- 172.21.15.28:0/2630524342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f640ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867457767 0 0) 0x7f7f54001110 con 0x7f7f640cdfa0 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f48003520 con 0x7f7f640cdfa0 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f5be00640 1 -- 172.21.15.28:0/2630524342 >> v1:172.21.15.189:6789/0 conn(0x7f7f6406f440 legacy=0x7f7f64163ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241680450 0 0) 0x7f7f64159430 con 0x7f7f6406f440 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f5be00640 1 -- 172.21.15.28:0/2630524342 >> v1:172.21.15.28:6789/0 conn(0x7f7f640cdfa0 legacy=0x7f7f640b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.194+0000 7f7f5be00640 1 -- 172.21.15.28:0/2630524342 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f64159710 con 0x7f7f6415d340 2024-09-13T17:23:56.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.195+0000 7f7f5be00640 1 --2- 172.21.15.28:0/2630524342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f640ce560 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0097d0 tx=0x7f7f4c02a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e25827148ab96d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.195+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c032440 con 0x7f7f6415d340 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.195+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c0325a0 con 0x7f7f6415d340 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.195+0000 7f7f61400640 1 -- 172.21.15.28:0/2630524342 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c032910 con 0x7f7f6415d340 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 -- 172.21.15.28:0/2630524342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 msgr2=0x7f7f640ce560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 --2- 172.21.15.28:0/2630524342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f640ce560 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0097d0 tx=0x7f7f4c02a200 comp rx=0 tx=0).stop 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 -- 172.21.15.28:0/2630524342 shutdown_connections 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 --2- 172.21.15.28:0/2630524342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f640ce560 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 -- 172.21.15.28:0/2630524342 >> 172.21.15.28:0/2630524342 conn(0x7f7f64069e80 msgr2=0x7f7f6406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 -- 172.21.15.28:0/2630524342 shutdown_connections 2024-09-13T17:23:56.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.196+0000 7f7f68a00640 1 -- 172.21.15.28:0/2630524342 wait complete. 2024-09-13T17:23:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.198+0000 7f7f68a00640 1 Processor -- start 2024-09-13T17:23:56.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.198+0000 7f7f68a00640 1 -- start start 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f6406f440 0x7f7f64134120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f641353f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64135930 con 0x7f7f640cdfa0 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64135be0 con 0x7f7f6406f440 2024-09-13T17:23:56.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f68a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f64135ec0 con 0x7f7f6415d340 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44580/0 (socket says 172.21.15.28:44580) 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 -- 172.21.15.28:0/1432649129 learned_addr learned my addr 172.21.15.28:0/1432649129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f61e00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f6406f440 0x7f7f64134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f62800640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f641353f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 -- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 msgr2=0x7f7f641353f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f641353f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 -- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f6406f440 msgr2=0x7f7f64134120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f6406f440 0x7f7f64134120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f5be00640 1 -- 172.21.15.28:0/1432649129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4c009400 con 0x7f7f640cdfa0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.199+0000 7f7f62800640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f641353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f5be00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c000c00 tx=0x7f7f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c030b80 con 0x7f7f640cdfa0 2024-09-13T17:23:56.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c032e90 con 0x7f7f640cdfa0 2024-09-13T17:23:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c006e10 con 0x7f7f640cdfa0 2024-09-13T17:23:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f641361a0 con 0x7f7f640cdfa0 2024-09-13T17:23:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.200+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f64136760 con 0x7f7f640cdfa0 2024-09-13T17:23:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.201+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f640ce560 con 0x7f7f640cdfa0 2024-09-13T17:23:56.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.202+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7f4c02ad50 con 0x7f7f640cdfa0 2024-09-13T17:23:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.203+0000 7f7f5a000640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 0x7f7f4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.203+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7f4c0a9da0 con 0x7f7f640cdfa0 2024-09-13T17:23:56.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.203+0000 7f7f61e00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 0x7f7f4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.204+0000 7f7f61e00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 0x7f7f4006d0c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005880 tx=0x7f7f54009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.207+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7f4c074fe0 con 0x7f7f640cdfa0 2024-09-13T17:23:56.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.385+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f640cf280 con 0x7f7f640cdfa0 2024-09-13T17:23:56.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.386+0000 7f7f5a000640 1 -- 172.21.15.28:0/1432649129 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+16617 (secure 0 0 0) 0x7f7f4c037090 con 0x7f7f640cdfa0 2024-09-13T17:23:56.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:56.388 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:56.063188+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"31.17","mappings":[{"from":0,"to":7}]},{"pgid":"31.33","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 msgr2=0x7f7f4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 0x7f7f4006d0c0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f7f54005880 tx=0x7f7f54009040 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 msgr2=0x7f7f64134eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c000c00 tx=0x7f7f4c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 shutdown_connections 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7f4006ac00 0x7f7f4006d0c0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f6415d340 0x7f7f641353f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f640cdfa0 0x7f7f64134eb0 unknown :-1 s=CLOSED pgs=2653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 --2- 172.21.15.28:0/1432649129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f6406f440 0x7f7f64134120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.388+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 >> 172.21.15.28:0/1432649129 conn(0x7f7f64069e80 msgr2=0x7f7f64158e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.389+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 shutdown_connections 2024-09-13T17:23:56.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.389+0000 7f7f68a00640 1 -- 172.21.15.28:0/1432649129 wait complete. 2024-09-13T17:23:56.406 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:23:56.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:56.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.560+0000 7f6580e00640 1 Processor -- start 2024-09-13T17:23:56.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.560+0000 7f6580e00640 1 -- start start 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c1592a0 0x7f657c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 0x7f657c159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f657c06c2d0 0x7f657c06c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c15b0b0 con 0x7f657c06c2d0 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c06ccc0 con 0x7f657c06f970 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.561+0000 7f6580e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0b2040 con 0x7f657c1592a0 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f657c06c2d0 0x7f657c06c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f657c06c2d0 0x7f657c06c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44590/0 (socket says 172.21.15.28:44590) 2024-09-13T17:23:56.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 -- 172.21.15.28:0/3638345479 learned_addr learned my addr 172.21.15.28:0/3638345479 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f6579400640 1 --2- 172.21.15.28:0/3638345479 >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 0x7f657c159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f6579e00640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c1592a0 0x7f657c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 -- 172.21.15.28:0/3638345479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c1592a0 msgr2=0x7f657c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c1592a0 0x7f657c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 -- 172.21.15.28:0/3638345479 >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 msgr2=0x7f657c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 --2- 172.21.15.28:0/3638345479 >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 0x7f657c159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f657a800640 1 -- 172.21.15.28:0/3638345479 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c159ff0 con 0x7f657c06c2d0 2024-09-13T17:23:56.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.562+0000 7f6579400640 1 --2- 172.21.15.28:0/3638345479 >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 0x7f657c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.563+0000 7f657a800640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06c2d0 0x7f657c06c780 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f657c06ae00 tx=0x7f657402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de944902b83d18ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.563+0000 7f6578a00640 1 -- 172.21.15.28:0/3638345479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574031600 con 0x7f657c06c2d0 2024-09-13T17:23:56.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6578a00640 1 -- 172.21.15.28:0/3638345479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657402d050 con 0x7f657c06c2d0 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6578a00640 1 -- 172.21.15.28:0/3638345479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574031980 con 0x7f657c06c2d0 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 -- 172.21.15.28:0/3638345479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06c2d0 msgr2=0x7f657c06c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06c2d0 0x7f657c06c780 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7f657c06ae00 tx=0x7f657402a6a0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 -- 172.21.15.28:0/3638345479 shutdown_connections 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06c2d0 0x7f657c06c780 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 --2- 172.21.15.28:0/3638345479 >> v2:172.21.15.189:3300/0 conn(0x7f657c06f970 0x7f657c159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 --2- 172.21.15.28:0/3638345479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c1592a0 0x7f657c15ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 -- 172.21.15.28:0/3638345479 >> 172.21.15.28:0/3638345479 conn(0x7f657c069e80 msgr2=0x7f657c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 -- 172.21.15.28:0/3638345479 shutdown_connections 2024-09-13T17:23:56.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.564+0000 7f6580e00640 1 -- 172.21.15.28:0/3638345479 wait complete. 2024-09-13T17:23:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.566+0000 7f6580e00640 1 Processor -- start 2024-09-13T17:23:56.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.566+0000 7f6580e00640 1 -- start start 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c06c2d0 0x7f657c144650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f657c1592a0 0x7f657c0b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c1444e0 con 0x7f657c06f970 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0af070 con 0x7f657c1592a0 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6580e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0af350 con 0x7f657c06c2d0 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6579400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6579400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44604/0 (socket says 172.21.15.28:44604) 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6579400640 1 -- 172.21.15.28:0/1920314694 learned_addr learned my addr 172.21.15.28:0/1920314694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f6579e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c06c2d0 0x7f657c144650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.567+0000 7f657a800640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f657c1592a0 0x7f657c0b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 -- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c06c2d0 msgr2=0x7f657c144650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c06c2d0 0x7f657c144650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 -- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f657c1592a0 msgr2=0x7f657c0b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f657c1592a0 0x7f657c0b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 -- 172.21.15.28:0/1920314694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574009b30 con 0x7f657c06f970 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6579400640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f656800e7e0 tx=0x7f656800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800c8d0 con 0x7f657c06f970 2024-09-13T17:23:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656800ee90 con 0x7f657c06f970 2024-09-13T17:23:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c149c70 con 0x7f657c06f970 2024-09-13T17:23:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656800cc70 con 0x7f657c06f970 2024-09-13T17:23:56.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.568+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c0af630 con 0x7f657c06f970 2024-09-13T17:23:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.569+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c06c780 con 0x7f657c06f970 2024-09-13T17:23:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.570+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6568013410 con 0x7f657c06f970 2024-09-13T17:23:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.571+0000 7f6572000640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.571+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6568089190 con 0x7f657c06f970 2024-09-13T17:23:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.572+0000 7f6579e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.572+0000 7f6579e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.575+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6568054320 con 0x7f657c06f970 2024-09-13T17:23:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.791+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f657c14df60 con 0x7f657c06f970 2024-09-13T17:23:56.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.791+0000 7f6572000640 1 -- 172.21.15.28:0/1920314694 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f6568057940 con 0x7f657c06f970 2024-09-13T17:23:56.793 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:56.793 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:56.793 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 msgr2=0x7f655c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f656c006fd0 tx=0x7f656c008040 comp rx=0 tx=0).stop 2024-09-13T17:23:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 msgr2=0x7f657c143fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 secure :-1 s=READY pgs=2655 cs=0 l=1 rev1=1 crypto rx=0x7f656800e7e0 tx=0x7f656800eca0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 shutdown_connections 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f655c06ac00 0x7f655c06d0c0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f657c1592a0 0x7f657c0b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f657c06f970 0x7f657c143fa0 unknown :-1 s=CLOSED pgs=2655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 --2- 172.21.15.28:0/1920314694 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f657c06c2d0 0x7f657c144650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 >> 172.21.15.28:0/1920314694 conn(0x7f657c069e80 msgr2=0x7f657c14e760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 shutdown_connections 2024-09-13T17:23:56.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.794+0000 7f6580e00640 1 -- 172.21.15.28:0/1920314694 wait complete. 2024-09-13T17:23:56.810 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:23:56.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.966+0000 7f32ea200640 1 Processor -- start 2024-09-13T17:23:56.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.966+0000 7f32ea200640 1 -- start start 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32ea200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f32e40b21c0 0x7f32e40b19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32ea200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e40cad20 con 0x7f32e40b0440 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32ea200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e406bb40 con 0x7f32e415fb90 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32ea200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e406be20 con 0x7f32e40b21c0 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32e9800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f32e40b0440 0x7f32e40cb490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37888/0 (socket says 172.21.15.28:37888) 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32e9800640 1 -- 172.21.15.28:0/533724993 learned_addr learned my addr 172.21.15.28:0/533724993 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:56.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> v2:172.21.15.189:3300/0 conn(0x7f32e40b21c0 0x7f32e40b19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.968+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479374828 0 0) 0x7f32e40cad20 con 0x7f32e40b0440 2024-09-13T17:23:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c8003520 con 0x7f32e40b0440 2024-09-13T17:23:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997883165 0 0) 0x7f32c8003520 con 0x7f32e40b0440 2024-09-13T17:23:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 >> v2:172.21.15.189:3300/0 conn(0x7f32e40b21c0 msgr2=0x7f32e40b19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:56.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 --2- 172.21.15.28:0/533724993 >> v2:172.21.15.189:3300/0 conn(0x7f32e40b21c0 0x7f32e40b19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 >> v1:172.21.15.189:6789/0 conn(0x7f32e415fb90 legacy=0x7f32e406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e406c100 con 0x7f32e40b0440 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f32dc0019f0 con 0x7f32e40b0440 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> v2:172.21.15.189:3300/0 conn(0x7f32e40b21c0 0x7f32e40b19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 >> v1:172.21.15.28:6789/0 conn(0x7f32e40b0440 legacy=0x7f32e40cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c8003970 0x7f32c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c80052a0 0x7f32c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e406be20 con 0x7f32c80047c0 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c800c170 con 0x7f32c8003970 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c800c330 con 0x7f32c80052a0 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.969+0000 7f32e8e00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c8003970 0x7f32c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32e9800640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c80052a0 0x7f32c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32dbe00640 1 -- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c80052a0 msgr2=0x7f32c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c80052a0 0x7f32c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32dbe00640 1 -- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c8003970 msgr2=0x7f32c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32c8003970 0x7f32c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32dbe00640 1 -- 172.21.15.28:0/533724993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c8003520 con 0x7f32c80047c0 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32e9800640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32c80052a0 0x7f32c800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:56.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.970+0000 7f32e9800640 1 -- 172.21.15.28:0/533724993 reap_dead start 2024-09-13T17:23:56.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.971+0000 7f32dbe00640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32c8004c40 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009c60 tx=0x7f32c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2af0e453b200da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.971+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc031400 con 0x7f32c80047c0 2024-09-13T17:23:56.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.971+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32cc02d050 con 0x7f32c80047c0 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.971+0000 7f32e3e00640 1 -- 172.21.15.28:0/533724993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32cc031760 con 0x7f32c80047c0 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 -- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 msgr2=0x7f32c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32c8004c40 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009c60 tx=0x7f32c80057f0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 -- 172.21.15.28:0/533724993 shutdown_connections 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 --2- 172.21.15.28:0/533724993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32c8004c40 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 -- 172.21.15.28:0/533724993 >> 172.21.15.28:0/533724993 conn(0x7f32e4072e40 msgr2=0x7f32e4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 -- 172.21.15.28:0/533724993 shutdown_connections 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.972+0000 7f32ea200640 1 -- 172.21.15.28:0/533724993 wait complete. 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.973+0000 7f32ea200640 1 Processor -- start 2024-09-13T17:23:56.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.973+0000 7f32ea200640 1 -- start start 2024-09-13T17:23:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32e40ae980 0x7f32e40aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32e414fc60 0x7f32e41500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40af300 con 0x7f32c80047c0 2024-09-13T17:23:56.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40addb0 con 0x7f32e414fc60 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.974+0000 7f32ea200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e4150620 con 0x7f32e40ae980 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44626/0 (socket says 172.21.15.28:44626) 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 -- 172.21.15.28:0/3218775108 learned_addr learned my addr 172.21.15.28:0/3218775108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e9800640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32e414fc60 0x7f32e41500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32dbe00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32e40ae980 0x7f32e40aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 -- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32e40ae980 msgr2=0x7f32e40aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32e40ae980 0x7f32e40aedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 -- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32e414fc60 msgr2=0x7f32e41500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32e414fc60 0x7f32e41500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 -- 172.21.15.28:0/3218775108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e40af520 con 0x7f32c80047c0 2024-09-13T17:23:56.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.975+0000 7f32e8e00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f32d400f820 tx=0x7f32d400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.976+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d40050d0 con 0x7f32c80047c0 2024-09-13T17:23:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.976+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d4005230 con 0x7f32c80047c0 2024-09-13T17:23:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.976+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d400d730 con 0x7f32c80047c0 2024-09-13T17:23:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.976+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e41575f0 con 0x7f32c80047c0 2024-09-13T17:23:56.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.976+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e4157bb0 con 0x7f32c80047c0 2024-09-13T17:23:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.977+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e40b0870 con 0x7f32c80047c0 2024-09-13T17:23:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.978+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f32d400d890 con 0x7f32c80047c0 2024-09-13T17:23:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.979+0000 7f32e1600640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 0x7f32c006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.979+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f32d40891f0 con 0x7f32c80047c0 2024-09-13T17:23:56.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.979+0000 7f32dbe00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 0x7f32c006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:56.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.980+0000 7f32dbe00640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 0x7f32c006d0f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f32cc033000 tx=0x7f32cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:56.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:56.983+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f32d40543a0 con 0x7f32c80047c0 2024-09-13T17:23:57.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.198+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32e40c8020 con 0x7f32c80047c0 2024-09-13T17:23:57.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.199+0000 7f32e1600640 1 -- 172.21.15.28:0/3218775108 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+4755 (secure 0 0 0) 0x7f32d40579c0 con 0x7f32c80047c0 2024-09-13T17:23:57.200 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:57.200 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":181,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:23:41.629044+0000","modified":"2024-09-13T17:23:44.642067+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":11916},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11916":{"gid":11916,"name":"c","rank":0,"incarnation":177,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6834/3884798333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3884798333},{"type":"v1","addr":"172.21.15.28:6834","nonce":3884798333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2024-09-13T17:23:57.200 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 181 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.200+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 msgr2=0x7f32c006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.200+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 0x7f32c006d0f0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f32cc033000 tx=0x7f32cc008030 comp rx=0 tx=0).stop 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 msgr2=0x7f32e40c87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f32d400f820 tx=0x7f32d400fce0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 shutdown_connections 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f32c006ac30 0x7f32c006d0f0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f32e414fc60 0x7f32e41500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f32e40ae980 0x7f32e40aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 --2- 172.21.15.28:0/3218775108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f32c80047c0 0x7f32e40c87b0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 >> 172.21.15.28:0/3218775108 conn(0x7f32e4072e40 msgr2=0x7f32e40afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 shutdown_connections 2024-09-13T17:23:57.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.201+0000 7f32ea200640 1 -- 172.21.15.28:0/3218775108 wait complete. 2024-09-13T17:23:57.218 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:23:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.373+0000 7fa824c00640 1 Processor -- start 2024-09-13T17:23:57.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.373+0000 7fa824c00640 1 -- start start 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa824c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa820159440 0x7fa8200b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa824c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa820159820 0x7fa8200b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa824c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8200c3850 con 0x7fa8200b22e0 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa824c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82015a930 con 0x7fa820159820 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa824c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa820159d20 con 0x7fa820159440 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa816a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa820159820 0x7fa8200b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa816a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa820159820 0x7fa8200b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44632/0 (socket says 172.21.15.28:44632) 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.374+0000 7fa816a00640 1 -- 172.21.15.28:0/480361180 learned_addr learned my addr 172.21.15.28:0/480361180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.375+0000 7fa81f400640 1 --2- 172.21.15.28:0/480361180 >> v2:172.21.15.189:3300/0 conn(0x7fa820159440 0x7fa8200b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.375+0000 7fa816a00640 1 -- 172.21.15.28:0/480361180 >> v2:172.21.15.189:3300/0 conn(0x7fa820159440 msgr2=0x7fa8200b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:57.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.375+0000 7fa816a00640 1 --2- 172.21.15.28:0/480361180 >> v2:172.21.15.189:3300/0 conn(0x7fa820159440 0x7fa8200b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.375+0000 7fa816a00640 1 -- 172.21.15.28:0/480361180 >> v1:172.21.15.189:6789/0 conn(0x7fa8200b22e0 legacy=0x7fa8200b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:57.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.375+0000 7fa816a00640 1 -- 172.21.15.28:0/480361180 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82015a000 con 0x7fa820159820 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.376+0000 7fa816a00640 1 --2- 172.21.15.28:0/480361180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8200b0980 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fa808009870 tx=0x7fa80802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126cd5ab84ccdaa0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.376+0000 7fa81ea00640 1 -- 172.21.15.28:0/480361180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808032640 con 0x7fa820159820 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.376+0000 7fa81ea00640 1 -- 172.21.15.28:0/480361180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8080327a0 con 0x7fa820159820 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.376+0000 7fa81ea00640 1 -- 172.21.15.28:0/480361180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa808032b30 con 0x7fa820159820 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 -- 172.21.15.28:0/480361180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 msgr2=0x7fa8200b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 --2- 172.21.15.28:0/480361180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8200b0980 secure :-1 s=READY pgs=2659 cs=0 l=1 rev1=1 crypto rx=0x7fa808009870 tx=0x7fa80802a300 comp rx=0 tx=0).stop 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 -- 172.21.15.28:0/480361180 shutdown_connections 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 --2- 172.21.15.28:0/480361180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8200b0980 unknown :-1 s=CLOSED pgs=2659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 --2- 172.21.15.28:0/480361180 >> v2:172.21.15.189:3300/0 conn(0x7fa820159440 0x7fa8200b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 -- 172.21.15.28:0/480361180 >> 172.21.15.28:0/480361180 conn(0x7fa820072e40 msgr2=0x7fa820073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:57.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 -- 172.21.15.28:0/480361180 shutdown_connections 2024-09-13T17:23:57.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.377+0000 7fa824c00640 1 -- 172.21.15.28:0/480361180 wait complete. 2024-09-13T17:23:57.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.378+0000 7fa824c00640 1 Processor -- start 2024-09-13T17:23:57.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.379+0000 7fa824c00640 1 -- start start 2024-09-13T17:23:57.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.379+0000 7fa824c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8200b22e0 0x7fa8200c39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.379+0000 7fa824c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa820159440 0x7fa8201f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.379+0000 7fa824c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa824c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8201564b0 con 0x7fa820159820 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa824c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa820156790 con 0x7fa820159440 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa824c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa820156a70 con 0x7fa8200b22e0 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44648/0 (socket says 172.21.15.28:44648) 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 -- 172.21.15.28:0/2970911458 learned_addr learned my addr 172.21.15.28:0/2970911458 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa816a00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa820159440 0x7fa8201f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81f400640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8200b22e0 0x7fa8200c39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 -- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8200b22e0 msgr2=0x7fa8200c39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8200b22e0 0x7fa8200c39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 -- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa820159440 msgr2=0x7fa8201f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa820159440 0x7fa8201f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.380+0000 7fa81fe00640 1 -- 172.21.15.28:0/2970911458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8080094d0 con 0x7fa820159820 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa81fe00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c780 tx=0x7fa81800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa818004b10 con 0x7fa820159820 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa818004c70 con 0x7fa820159820 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81800aa90 con 0x7fa820159820 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa820156d50 con 0x7fa820159820 2024-09-13T17:23:57.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.381+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8200c3f20 con 0x7fa820159820 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.382+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa820157270 con 0x7fa820159820 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.383+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa81800abf0 con 0x7fa820159820 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.384+0000 7fa817e00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 0x7fa7fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.384+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa818089370 con 0x7fa820159820 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.384+0000 7fa81f400640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 0x7fa7fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.385+0000 7fa81f400640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 0x7fa7fc06d0c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa810006fd0 tx=0x7fa810008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.388+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa820157270 con 0x7fa820159820 2024-09-13T17:23:57.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.568+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa820157270 con 0x7fa820159820 2024-09-13T17:23:57.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.569+0000 7fa817e00640 1 -- 172.21.15.28:0/2970911458 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v221) v1 ==== 74+0+16617 (secure 0 0 0) 0x7fa820157270 con 0x7fa820159820 2024-09-13T17:23:57.571 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:23:57.571 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":221,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:23:56.063188+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:23:39.609113+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"218","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2024-09-13T17:23:40.623957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"219","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"31.17","mappings":[{"from":0,"to":7}]},{"pgid":"31.33","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:23:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 msgr2=0x7fa7fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 0x7fa7fc06d0c0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fa810006fd0 tx=0x7fa810008040 comp rx=0 tx=0).stop 2024-09-13T17:23:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 msgr2=0x7fa8201f5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fa81800c780 tx=0x7fa81800cc40 comp rx=0 tx=0).stop 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 shutdown_connections 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa7fc06ac00 0x7fa7fc06d0c0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa820159820 0x7fa8201f5e60 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa820159440 0x7fa8201f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 --2- 172.21.15.28:0/2970911458 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa8200b22e0 0x7fa8200c39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 >> 172.21.15.28:0/2970911458 conn(0x7fa820072e40 msgr2=0x7fa82015b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 shutdown_connections 2024-09-13T17:23:57.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.572+0000 7fa824c00640 1 -- 172.21.15.28:0/2970911458 wait complete. 2024-09-13T17:23:57.589 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:23:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.748+0000 7f7e98c00640 1 Processor -- start 2024-09-13T17:23:57.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.748+0000 7f7e98c00640 1 -- start start 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e98c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e94070430 0x7f7e9415bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e98c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9415ab60 con 0x7f7e9406f3f0 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e98c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9406f920 con 0x7f7e94159770 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e98c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9415b290 con 0x7f7e94070430 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e8be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f7e94070430 0x7f7e9415bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e92800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e9406f3f0 0x7f7e9415c2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37892/0 (socket says 172.21.15.28:37892) 2024-09-13T17:23:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.749+0000 7f7e92800640 1 -- 172.21.15.28:0/3464791921 learned_addr learned my addr 172.21.15.28:0/3464791921 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.750+0000 7f7e8be00640 1 -- 172.21.15.28:0/3464791921 >> v1:172.21.15.189:6789/0 conn(0x7f7e94159770 legacy=0x7f7e9406fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.750+0000 7f7e8be00640 1 -- 172.21.15.28:0/3464791921 >> v1:172.21.15.28:6789/0 conn(0x7f7e9406f3f0 legacy=0x7f7e9415c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.750+0000 7f7e8be00640 1 -- 172.21.15.28:0/3464791921 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9415b570 con 0x7f7e94070430 2024-09-13T17:23:57.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.750+0000 7f7e91400640 1 -- 172.21.15.28:0/3464791921 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196655564 0 0) 0x7f7e8c001490 con 0x7f7e9406f3f0 2024-09-13T17:23:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.751+0000 7f7e8be00640 1 --2- 172.21.15.28:0/3464791921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e9415bd70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0097a0 tx=0x7f7e7c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1744eb527e5796de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.751+0000 7f7e91400640 1 -- 172.21.15.28:0/3464791921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c032440 con 0x7f7e94070430 2024-09-13T17:23:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.751+0000 7f7e91400640 1 -- 172.21.15.28:0/3464791921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c0325a0 con 0x7f7e94070430 2024-09-13T17:23:57.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.751+0000 7f7e91400640 1 -- 172.21.15.28:0/3464791921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c032930 con 0x7f7e94070430 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 -- 172.21.15.28:0/3464791921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 msgr2=0x7f7e9415bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3464791921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e9415bd70 secure :-1 s=READY pgs=2662 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0097a0 tx=0x7f7e7c02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 -- 172.21.15.28:0/3464791921 shutdown_connections 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3464791921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e9415bd70 unknown :-1 s=CLOSED pgs=2662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 -- 172.21.15.28:0/3464791921 >> 172.21.15.28:0/3464791921 conn(0x7f7e940b0270 msgr2=0x7f7e940b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 -- 172.21.15.28:0/3464791921 shutdown_connections 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.752+0000 7f7e98c00640 1 -- 172.21.15.28:0/3464791921 wait complete. 2024-09-13T17:23:57.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.754+0000 7f7e98c00640 1 Processor -- start 2024-09-13T17:23:57.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.754+0000 7f7e98c00640 1 -- start start 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.754+0000 7f7e98c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e9406f3f0 0x7f7e9414eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.754+0000 7f7e98c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.754+0000 7f7e98c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 0x7f7e940b25c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e98c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941f32a0 con 0x7f7e94070430 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e98c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941f3580 con 0x7f7e94159770 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e98c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941f3860 con 0x7f7e9406f3f0 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44674/0 (socket says 172.21.15.28:44674) 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 -- 172.21.15.28:0/3305271860 learned_addr learned my addr 172.21.15.28:0/3305271860 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:57.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e91e00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e9406f3f0 0x7f7e9414eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e92800640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 0x7f7e940b25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 -- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e9406f3f0 msgr2=0x7f7e9414eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e9406f3f0 0x7f7e9414eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 -- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 msgr2=0x7f7e940b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 0x7f7e940b25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 -- 172.21.15.28:0/3305271860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c009400 con 0x7f7e94070430 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e92800640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 0x7f7e940b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.755+0000 7f7e8be00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c000c00 tx=0x7f7e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.756+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c004230 con 0x7f7e94070430 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.756+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c004390 con 0x7f7e94070430 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.756+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c030ac0 con 0x7f7e94070430 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.756+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e941f3b40 con 0x7f7e94070430 2024-09-13T17:23:57.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.756+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e941e4770 con 0x7f7e94070430 2024-09-13T17:23:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.757+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e941e4900 con 0x7f7e94070430 2024-09-13T17:23:57.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.758+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e7c039050 con 0x7f7e94070430 2024-09-13T17:23:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.759+0000 7f7e8a000640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 0x7f7e7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:57.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.759+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(221..221 src has 1..221) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7e7c0a9dc0 con 0x7f7e94070430 2024-09-13T17:23:57.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.760+0000 7f7e91e00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 0x7f7e7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:57.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.760+0000 7f7e91e00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 0x7f7e7006d0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7e940b3110 tx=0x7f7e84009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:57.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.764+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7e941e4900 con 0x7f7e94070430 2024-09-13T17:23:57.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.978+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7e941f4060 con 0x7f7e94070430 2024-09-13T17:23:57.980 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:23:57.979+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:23:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.988+0000 7f7e8a000640 1 -- 172.21.15.28:0/3305271860 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v182) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7e941f4060 con 0x7f7e94070430 2024-09-13T17:23:57.990 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 msgr2=0x7f7e7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 0x7f7e7006d0c0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f7e940b3110 tx=0x7f7e84009040 comp rx=0 tx=0).stop 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 msgr2=0x7f7e941e4230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c000c00 tx=0x7f7e7c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 shutdown_connections 2024-09-13T17:23:57.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7e7006ac00 0x7f7e7006d0c0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e94159770 0x7f7e940b25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e94070430 0x7f7e941e4230 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 --2- 172.21.15.28:0/3305271860 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e9406f3f0 0x7f7e9414eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 >> 172.21.15.28:0/3305271860 conn(0x7f7e940b0270 msgr2=0x7f7e94158810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 shutdown_connections 2024-09-13T17:23:57.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:57.991+0000 7f7e98c00640 1 -- 172.21.15.28:0/3305271860 wait complete. 2024-09-13T17:23:58.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:23:58.060 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:23:58.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.171+0000 7f5c2b400640 1 Processor -- start 2024-09-13T17:23:58.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.171+0000 7f5c2b400640 1 -- start start 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c2b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 0x7f5c2406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c2b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 0x7f5c24070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c2b400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c2406f920 con 0x7f5c240add00 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c2b400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c2415b290 con 0x7f5c2406f3f0 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c2b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2415b570 con 0x7f5c24159770 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c29400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5c240add00 0x7f5c24159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37906/0 (socket says 172.21.15.28:37906) 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c29400640 1 -- 172.21.15.28:0/1047621982 learned_addr learned my addr 172.21.15.28:0/1047621982 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1047621982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 0x7f5c2406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c1be00640 1 --2- 172.21.15.28:0/1047621982 >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 0x7f5c24070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.173+0000 7f5c23e00640 1 -- 172.21.15.28:0/1047621982 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776773109 0 0) 0x7f5c2406f920 con 0x7f5c240add00 2024-09-13T17:23:58.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c23e00640 1 -- 172.21.15.28:0/1047621982 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c08003520 con 0x7f5c240add00 2024-09-13T17:23:58.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c28a00640 1 -- 172.21.15.28:0/1047621982 >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 msgr2=0x7f5c24070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1047621982 >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 0x7f5c24070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c28a00640 1 -- 172.21.15.28:0/1047621982 >> v1:172.21.15.28:6789/0 conn(0x7f5c240add00 legacy=0x7f5c24159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c28a00640 1 -- 172.21.15.28:0/1047621982 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2415b850 con 0x7f5c24159770 2024-09-13T17:23:58.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.174+0000 7f5c1be00640 1 --2- 172.21.15.28:0/1047621982 >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 0x7f5c24070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.175+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1047621982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 0x7f5c2406fd20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4424de5c9072e8bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.175+0000 7f5c23e00640 1 -- 172.21.15.28:0/1047621982 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1402ae50 con 0x7f5c24159770 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.175+0000 7f5c23e00640 1 -- 172.21.15.28:0/1047621982 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14032440 con 0x7f5c24159770 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.175+0000 7f5c23e00640 1 -- 172.21.15.28:0/1047621982 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c140327d0 con 0x7f5c24159770 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 -- 172.21.15.28:0/1047621982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 msgr2=0x7f5c2406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1047621982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 0x7f5c2406fd20 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402a100 comp rx=0 tx=0).stop 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 -- 172.21.15.28:0/1047621982 shutdown_connections 2024-09-13T17:23:58.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1047621982 >> v2:172.21.15.189:3300/0 conn(0x7f5c2406f3f0 0x7f5c24070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1047621982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c24159770 0x7f5c2406fd20 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 -- 172.21.15.28:0/1047621982 >> 172.21.15.28:0/1047621982 conn(0x7f5c240b0270 msgr2=0x7f5c240b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 -- 172.21.15.28:0/1047621982 shutdown_connections 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.176+0000 7f5c2b400640 1 -- 172.21.15.28:0/1047621982 wait complete. 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.178+0000 7f5c2b400640 1 Processor -- start 2024-09-13T17:23:58.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.178+0000 7f5c2b400640 1 -- start start 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2406f3f0 0x7f5c241f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 0x7f5c241f53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c241f58e0 con 0x7f5c24159770 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c24153af0 con 0x7f5c2406f3f0 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c2b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c241570d0 con 0x7f5c240add00 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44708/0 (socket says 172.21.15.28:44708) 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 -- 172.21.15.28:0/1437170781 learned_addr learned my addr 172.21.15.28:0/1437170781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c1be00640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 0x7f5c241f53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2406f3f0 0x7f5c241f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 -- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 msgr2=0x7f5c241f53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 0x7f5c241f53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 -- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2406f3f0 msgr2=0x7f5c241f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2406f3f0 0x7f5c241f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c29400640 1 -- 172.21.15.28:0/1437170781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0c009400 con 0x7f5c24159770 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.179+0000 7f5c1be00640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 0x7f5c241f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c29400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00cbb0 tx=0x7f5c1c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00ad40 con 0x7f5c24159770 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c1c012400 con 0x7f5c24159770 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c14009400 con 0x7f5c24159770 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1c00aea0 con 0x7f5c24159770 2024-09-13T17:23:58.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.180+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c24157690 con 0x7f5c24159770 2024-09-13T17:23:58.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.181+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c2406fd20 con 0x7f5c24159770 2024-09-13T17:23:58.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.182+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5c1c007070 con 0x7f5c24159770 2024-09-13T17:23:58.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.183+0000 7f5c21600640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 0x7f5c0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:58.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.183+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f5c1c089210 con 0x7f5c24159770 2024-09-13T17:23:58.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.183+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 0x7f5c0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:58.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.184+0000 7f5c28a00640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 0x7f5c0006d0c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5c14034000 tx=0x7f5c1402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:58.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.187+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5c1c0544a0 con 0x7f5c24159770 2024-09-13T17:23:58.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.402+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5c240accd0 con 0x7f5c24159770 2024-09-13T17:23:58.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.990+0000 7f5c21600640 1 -- 172.21.15.28:0/1437170781 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v183) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5c1c057ac0 con 0x7f5c24159770 2024-09-13T17:23:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 msgr2=0x7f5c0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 0x7f5c0006d0c0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f5c14034000 tx=0x7f5c1402e000 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 msgr2=0x7f5c24150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 secure :-1 s=READY pgs=2668 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c00cbb0 tx=0x7f5c1c00a430 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 shutdown_connections 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5c0006ac00 0x7f5c0006d0c0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c24159770 0x7f5c24150fa0 unknown :-1 s=CLOSED pgs=2668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5c240add00 0x7f5c241f53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 --2- 172.21.15.28:0/1437170781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c2406f3f0 0x7f5c241f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 >> 172.21.15.28:0/1437170781 conn(0x7f5c240b0270 msgr2=0x7f5c240ad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 shutdown_connections 2024-09-13T17:23:58.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:58.993+0000 7f5c2b400640 1 -- 172.21.15.28:0/1437170781 wait complete. 2024-09-13T17:23:59.013 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:23:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.176+0000 7f6823c00640 1 Processor -- start 2024-09-13T17:23:59.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.176+0000 7f6823c00640 1 -- start start 2024-09-13T17:23:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f681c15a7a0 0x7f681c0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f681c0ce9d0 0x7f681c0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 0x7f681c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0cee40 con 0x7f681c0ce9d0 2024-09-13T17:23:59.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c06c950 con 0x7f681c15a7a0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6823c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0b1ed0 con 0x7f681c0b1ab0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f681c0ce9d0 0x7f681c0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f681c0ce9d0 0x7f681c0b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44720/0 (socket says 172.21.15.28:44720) 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 -- 172.21.15.28:0/1387930972 learned_addr learned my addr 172.21.15.28:0/1387930972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6821200640 1 --2- 172.21.15.28:0/1387930972 >> v2:172.21.15.189:3300/0 conn(0x7f681c15a7a0 0x7f681c0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6821c00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 0x7f681c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 -- 172.21.15.28:0/1387930972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 msgr2=0x7f681c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 0x7f681c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 -- 172.21.15.28:0/1387930972 >> v2:172.21.15.189:3300/0 conn(0x7f681c15a7a0 msgr2=0x7f681c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 --2- 172.21.15.28:0/1387930972 >> v2:172.21.15.189:3300/0 conn(0x7f681c15a7a0 0x7f681c0ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6813e00640 1 -- 172.21.15.28:0/1387930972 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0b2150 con 0x7f681c0ce9d0 2024-09-13T17:23:59.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.178+0000 7f6821c00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 0x7f681c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:23:59.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.179+0000 7f6813e00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0ce9d0 0x7f681c0b1570 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f681c0716c0 tx=0x7f680402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a08d8170d239e99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f681be00640 1 -- 172.21.15.28:0/1387930972 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680402ae90 con 0x7f681c0ce9d0 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f681be00640 1 -- 172.21.15.28:0/1387930972 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6804032440 con 0x7f681c0ce9d0 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f681be00640 1 -- 172.21.15.28:0/1387930972 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68040327d0 con 0x7f681c0ce9d0 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 -- 172.21.15.28:0/1387930972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0ce9d0 msgr2=0x7f681c0b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0ce9d0 0x7f681c0b1570 secure :-1 s=READY pgs=2669 cs=0 l=1 rev1=1 crypto rx=0x7f681c0716c0 tx=0x7f680402a140 comp rx=0 tx=0).stop 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 -- 172.21.15.28:0/1387930972 shutdown_connections 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0ce9d0 0x7f681c0b1570 unknown :-1 s=CLOSED pgs=2669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 --2- 172.21.15.28:0/1387930972 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0b1ab0 0x7f681c06b170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 --2- 172.21.15.28:0/1387930972 >> v2:172.21.15.189:3300/0 conn(0x7f681c15a7a0 0x7f681c0ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 -- 172.21.15.28:0/1387930972 >> 172.21.15.28:0/1387930972 conn(0x7f681c069e80 msgr2=0x7f681c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 -- 172.21.15.28:0/1387930972 shutdown_connections 2024-09-13T17:23:59.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.180+0000 7f6823c00640 1 -- 172.21.15.28:0/1387930972 wait complete. 2024-09-13T17:23:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.182+0000 7f6823c00640 1 Processor -- start 2024-09-13T17:23:59.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.182+0000 7f6823c00640 1 -- start start 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0ce9d0 0x7f681c143d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f681c15a7a0 0x7f681c144250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c158960 con 0x7f681c0b1ab0 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1578c0 con 0x7f681c15a7a0 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6823c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c157ba0 con 0x7f681c0ce9d0 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6821200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6821200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44734/0 (socket says 172.21.15.28:44734) 2024-09-13T17:23:59.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6821200640 1 -- 172.21.15.28:0/3182737714 learned_addr learned my addr 172.21.15.28:0/3182737714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6813e00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0ce9d0 0x7f681c143d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.183+0000 7f6821c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f681c15a7a0 0x7f681c144250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 -- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0ce9d0 msgr2=0x7f681c143d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0ce9d0 0x7f681c143d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 -- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f681c15a7a0 msgr2=0x7f681c144250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f681c15a7a0 0x7f681c144250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 -- 172.21.15.28:0/3182737714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68040094d0 con 0x7f681c0b1ab0 2024-09-13T17:23:59.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6821200640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f680c00d5f0 tx=0x7f680c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.184+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0125f0 con 0x7f681c0b1ab0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.185+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c012750 con 0x7f681c0b1ab0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.185+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c012b00 con 0x7f681c0b1ab0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.185+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c157e80 con 0x7f681c0b1ab0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.185+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c158440 con 0x7f681c0b1ab0 2024-09-13T17:23:59.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.186+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c15ab80 con 0x7f681c0b1ab0 2024-09-13T17:23:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.187+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f680c012c60 con 0x7f681c0b1ab0 2024-09-13T17:23:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.187+0000 7f6819600640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 0x7f67f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:23:59.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.188+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(222..222 src has 1..222) v4 ==== 5853+0+0 (secure 0 0 0) 0x7f680c089820 con 0x7f681c0b1ab0 2024-09-13T17:23:59.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.188+0000 7f6813e00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 0x7f67f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:23:59.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.189+0000 7f6813e00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 0x7f67f806d0c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6804034000 tx=0x7f680402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:23:59.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.192+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f680c054980 con 0x7f681c0b1ab0 2024-09-13T17:23:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:23:59.370+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f681c06bb70 con 0x7f681c0b1ab0 2024-09-13T17:24:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.083+0000 7f6819600640 1 -- 172.21.15.28:0/3182737714 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v223) v1 ==== 181+0+0 (secure 0 0 0) 0x7f680c057fa0 con 0x7f681c0b1ab0 2024-09-13T17:24:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:24:00.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.087+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 msgr2=0x7f67f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.087+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 0x7f67f806d0c0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f6804034000 tx=0x7f680402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.087+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 msgr2=0x7f681c0c8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.087+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 secure :-1 s=READY pgs=2670 cs=0 l=1 rev1=1 crypto rx=0x7f680c00d5f0 tx=0x7f680c00dab0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 shutdown_connections 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f67f806ac00 0x7f67f806d0c0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f681c15a7a0 0x7f681c144250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f681c0ce9d0 0x7f681c143d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 --2- 172.21.15.28:0/3182737714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f681c0b1ab0 0x7f681c0c8fb0 unknown :-1 s=CLOSED pgs=2670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 >> 172.21.15.28:0/3182737714 conn(0x7f681c069e80 msgr2=0x7f681c154860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 shutdown_connections 2024-09-13T17:24:00.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.088+0000 7f6823c00640 1 -- 172.21.15.28:0/3182737714 wait complete. 2024-09-13T17:24:00.109 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:24:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.273+0000 7f6666a00640 1 Processor -- start 2024-09-13T17:24:00.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.273+0000 7f6666a00640 1 -- start start 2024-09-13T17:24:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6666a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66600ccd70 0x7f66600cf300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6666a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f666006c1f0 0x7f666006c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6666a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666006bd60 con 0x7f66600cc7b0 2024-09-13T17:24:00.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6666a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66600b34a0 con 0x7f66600ccd70 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6666a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666006b9f0 con 0x7f666006c1f0 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6665600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66600ccd70 0x7f66600cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6665600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f66600ccd70 0x7f66600cf300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44740/0 (socket says 172.21.15.28:44740) 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6664c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f66600cc7b0 0x7f66600cd150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37912/0 (socket says 172.21.15.28:37912) 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.274+0000 7f6665600640 1 -- 172.21.15.28:0/3452465470 learned_addr learned my addr 172.21.15.28:0/3452465470 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:00.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.275+0000 7f6666000640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f666006c1f0 0x7f666006c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.275+0000 7f6665600640 1 -- 172.21.15.28:0/3452465470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f666006c1f0 msgr2=0x7f666006c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.275+0000 7f6665600640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f666006c1f0 0x7f666006c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.275+0000 7f6665600640 1 -- 172.21.15.28:0/3452465470 >> v1:172.21.15.28:6789/0 conn(0x7f66600cc7b0 legacy=0x7f66600cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.275+0000 7f6665600640 1 -- 172.21.15.28:0/3452465470 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66600b31c0 con 0x7f66600ccd70 2024-09-13T17:24:00.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.276+0000 7f6665600640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66600ccd70 0x7f66600cf300 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6654009870 tx=0x7f665402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad7e3d9fc043e74 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.276+0000 7f665fe00640 1 -- 172.21.15.28:0/3452465470 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654032440 con 0x7f66600ccd70 2024-09-13T17:24:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.276+0000 7f665fe00640 1 -- 172.21.15.28:0/3452465470 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66540325a0 con 0x7f66600ccd70 2024-09-13T17:24:00.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.276+0000 7f665fe00640 1 -- 172.21.15.28:0/3452465470 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654032930 con 0x7f66600ccd70 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 -- 172.21.15.28:0/3452465470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66600ccd70 msgr2=0x7f66600cf300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66600ccd70 0x7f66600cf300 secure :-1 s=READY pgs=2672 cs=0 l=1 rev1=1 crypto rx=0x7f6654009870 tx=0x7f665402a250 comp rx=0 tx=0).stop 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 -- 172.21.15.28:0/3452465470 shutdown_connections 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f666006c1f0 0x7f666006c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 --2- 172.21.15.28:0/3452465470 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f66600ccd70 0x7f66600cf300 unknown :-1 s=CLOSED pgs=2672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 -- 172.21.15.28:0/3452465470 >> 172.21.15.28:0/3452465470 conn(0x7f66600b0930 msgr2=0x7f66600b0d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 -- 172.21.15.28:0/3452465470 shutdown_connections 2024-09-13T17:24:00.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.277+0000 7f6666a00640 1 -- 172.21.15.28:0/3452465470 wait complete. 2024-09-13T17:24:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.278+0000 7f6666a00640 1 Processor -- start 2024-09-13T17:24:00.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.279+0000 7f6666a00640 1 -- start start 2024-09-13T17:24:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.279+0000 7f6666a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666006c1f0 0x7f666015b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.279+0000 7f6666a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 0x7f666015b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6660153d30 con 0x7f6660154f00 2024-09-13T17:24:00.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666015cd00 con 0x7f666006c1f0 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6660155c10 con 0x7f66600ccd70 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44750/0 (socket says 172.21.15.28:44750) 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 -- 172.21.15.28:0/1105778583 learned_addr learned my addr 172.21.15.28:0/1105778583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6664c00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 0x7f666015b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6665600640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666006c1f0 0x7f666015b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 -- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 msgr2=0x7f666015b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 0x7f666015b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 -- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666006c1f0 msgr2=0x7f666015b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666006c1f0 0x7f666015b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6666000640 1 -- 172.21.15.28:0/1105778583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66540094d0 con 0x7f6660154f00 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.280+0000 7f6664c00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 0x7f666015b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:00.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f6666000640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f6658007e40 tx=0x7f665800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658014620 con 0x7f6660154f00 2024-09-13T17:24:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665800ee90 con 0x7f6660154f00 2024-09-13T17:24:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66580149e0 con 0x7f6660154f00 2024-09-13T17:24:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6660155ef0 con 0x7f6660154f00 2024-09-13T17:24:00.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.281+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666015cfc0 con 0x7f6660154f00 2024-09-13T17:24:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.282+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666006c660 con 0x7f6660154f00 2024-09-13T17:24:00.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.283+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6658014b40 con 0x7f6660154f00 2024-09-13T17:24:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.284+0000 7f665d600640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 0x7f664006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:00.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.284+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(223..223 src has 1..223) v4 ==== 5415+0+0 (secure 0 0 0) 0x7f6658089b10 con 0x7f6660154f00 2024-09-13T17:24:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.285+0000 7f6665600640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 0x7f664006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:00.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.286+0000 7f6665600640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 0x7f664006d0c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6654034000 tx=0x7f665402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:00.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.289+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6658054e10 con 0x7f6660154f00 2024-09-13T17:24:00.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:00.468+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f666009d080 con 0x7f6660154f00 2024-09-13T17:24:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.094+0000 7f665d600640 1 -- 172.21.15.28:0/1105778583 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v224) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6658058430 con 0x7f6660154f00 2024-09-13T17:24:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:24:01.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.098+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 msgr2=0x7f664006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.098+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 0x7f664006d0c0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f6654034000 tx=0x7f665402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:01.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.098+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 msgr2=0x7f66601553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.098+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 secure :-1 s=READY pgs=2673 cs=0 l=1 rev1=1 crypto rx=0x7f6658007e40 tx=0x7f665800e4b0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 shutdown_connections 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f664006ac00 0x7f664006d0c0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6660154f00 0x7f66601553b0 unknown :-1 s=CLOSED pgs=2673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f66600ccd70 0x7f666015b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 --2- 172.21.15.28:0/1105778583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666006c1f0 0x7f666015b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 >> 172.21.15.28:0/1105778583 conn(0x7f66600b0930 msgr2=0x7f66600cbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 shutdown_connections 2024-09-13T17:24:01.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.099+0000 7f6666a00640 1 -- 172.21.15.28:0/1105778583 wait complete. 2024-09-13T17:24:01.119 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:24:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.288+0000 7f473d200640 1 Processor -- start 2024-09-13T17:24:01.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.288+0000 7f473d200640 1 -- start start 2024-09-13T17:24:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f473d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f473815a780 0x7f4738070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f473d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473806c250 0x7f47380712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f473d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f473806c680 con 0x7f473815ab60 2024-09-13T17:24:01.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f473d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f473806c8b0 con 0x7f473815a780 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f473d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473806cb90 con 0x7f473806c250 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f4735e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f473815a780 0x7f4738070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f4735e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f473815a780 0x7f4738070d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44762/0 (socket says 172.21.15.28:44762) 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f472d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f473815ab60 0x7f473806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37928/0 (socket says 172.21.15.28:37928) 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f4735e00640 1 -- 172.21.15.28:0/94612664 learned_addr learned my addr 172.21.15.28:0/94612664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.290+0000 7f4736800640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473806c250 0x7f47380712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.291+0000 7f4735e00640 1 -- 172.21.15.28:0/94612664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473806c250 msgr2=0x7f47380712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.291+0000 7f4735e00640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473806c250 0x7f47380712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.291+0000 7f4735e00640 1 -- 172.21.15.28:0/94612664 >> v1:172.21.15.28:6789/0 conn(0x7f473815ab60 legacy=0x7f473806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.291+0000 7f4735e00640 1 -- 172.21.15.28:0/94612664 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47380cc980 con 0x7f473815a780 2024-09-13T17:24:01.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.291+0000 7f4735400640 1 -- 172.21.15.28:0/94612664 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194168389 0 0) 0x7f473806c680 con 0x7f473815ab60 2024-09-13T17:24:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f4735e00640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473815a780 0x7f4738070d80 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f4728009870 tx=0x7f472802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a507801f0abd9c6e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f4735400640 1 -- 172.21.15.28:0/94612664 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728032440 con 0x7f473815a780 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f4735400640 1 -- 172.21.15.28:0/94612664 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47280325a0 con 0x7f473815a780 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f4735400640 1 -- 172.21.15.28:0/94612664 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728032930 con 0x7f473815a780 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f473d200640 1 -- 172.21.15.28:0/94612664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473815a780 msgr2=0x7f4738070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.292+0000 7f473d200640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473815a780 0x7f4738070d80 secure :-1 s=READY pgs=2675 cs=0 l=1 rev1=1 crypto rx=0x7f4728009870 tx=0x7f472802a270 comp rx=0 tx=0).stop 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 -- 172.21.15.28:0/94612664 shutdown_connections 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473806c250 0x7f47380712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 --2- 172.21.15.28:0/94612664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473815a780 0x7f4738070d80 unknown :-1 s=CLOSED pgs=2675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 -- 172.21.15.28:0/94612664 >> 172.21.15.28:0/94612664 conn(0x7f47380b0270 msgr2=0x7f47380b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 -- 172.21.15.28:0/94612664 shutdown_connections 2024-09-13T17:24:01.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.293+0000 7f473d200640 1 -- 172.21.15.28:0/94612664 wait complete. 2024-09-13T17:24:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.294+0000 7f473d200640 1 Processor -- start 2024-09-13T17:24:01.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 -- start start 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473815a780 0x7f47381e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473815ab60 0x7f47380ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47381e4c20 con 0x7f473806c250 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.295+0000 7f473d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47381f59d0 con 0x7f473815a780 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f473d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47381e3fa0 con 0x7f473815ab60 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44778/0 (socket says 172.21.15.28:44778) 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 -- 172.21.15.28:0/1428082055 learned_addr learned my addr 172.21.15.28:0/1428082055 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:01.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f472d400640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473815a780 0x7f47381e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4736800640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473815ab60 0x7f47380ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 -- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473815ab60 msgr2=0x7f47380ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473815ab60 0x7f47380ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 -- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473815a780 msgr2=0x7f47381e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473815a780 0x7f47381e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 -- 172.21.15.28:0/1428082055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47280094d0 con 0x7f473806c250 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.296+0000 7f4735e00640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f4728000c00 tx=0x7f47280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.297+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728032440 con 0x7f473806c250 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.297+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4728030980 con 0x7f473806c250 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.297+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728030d30 con 0x7f473806c250 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.297+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47381e4280 con 0x7f473806c250 2024-09-13T17:24:01.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.297+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47381e4840 con 0x7f473806c250 2024-09-13T17:24:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.298+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47380b0020 con 0x7f473806c250 2024-09-13T17:24:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.299+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4728039040 con 0x7f473806c250 2024-09-13T17:24:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.300+0000 7f472ea00640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.300+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f47280a9a10 con 0x7f473806c250 2024-09-13T17:24:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.300+0000 7f472d400640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.301+0000 7f472d400640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f47380ca590 tx=0x7f4720007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.304+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4728074ee0 con 0x7f473806c250 2024-09-13T17:24:01.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.482+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f473806bb40 con 0x7f473806c250 2024-09-13T17:24:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.486+0000 7f472ea00640 1 -- 172.21.15.28:0/1428082055 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4728078500 con 0x7f473806c250 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 msgr2=0x7f471406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f47380ca590 tx=0x7f4720007c00 comp rx=0 tx=0).stop 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 msgr2=0x7f47381f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 secure :-1 s=READY pgs=2676 cs=0 l=1 rev1=1 crypto rx=0x7f4728000c00 tx=0x7f47280046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 shutdown_connections 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f471406ac00 0x7f471406d0c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f473815ab60 0x7f47380ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f473815a780 0x7f47381e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 --2- 172.21.15.28:0/1428082055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f473806c250 0x7f47381f5490 unknown :-1 s=CLOSED pgs=2676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 >> 172.21.15.28:0/1428082055 conn(0x7f47380b0270 msgr2=0x7f47380b0650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 shutdown_connections 2024-09-13T17:24:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.488+0000 7f473d200640 1 -- 172.21.15.28:0/1428082055 wait complete. 2024-09-13T17:24:01.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files' 2024-09-13T17:24:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.660+0000 7efc5c000640 1 Processor -- start 2024-09-13T17:24:01.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.660+0000 7efc5c000640 1 -- start start 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.661+0000 7efc5c000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 0x7efc5406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc5c000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc540cabc0 0x7efc5415e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc5c000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5415e100 con 0x7efc5415edb0 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc5c000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc540adb00 con 0x7efc540cabc0 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc5c000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc5415e3e0 con 0x7efc5415a540 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc540cabc0 0x7efc5415e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7efc540cabc0 0x7efc5415e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44784/0 (socket says 172.21.15.28:44784) 2024-09-13T17:24:01.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 -- 172.21.15.28:0/2879668020 learned_addr learned my addr 172.21.15.28:0/2879668020 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:01.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 -- 172.21.15.28:0/2879668020 >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 msgr2=0x7efc5406c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc59600640 1 --2- 172.21.15.28:0/2879668020 >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 0x7efc5406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 --2- 172.21.15.28:0/2879668020 >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 0x7efc5406c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 -- 172.21.15.28:0/2879668020 >> v1:172.21.15.189:6789/0 conn(0x7efc5415edb0 legacy=0x7efc54070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc58c00640 1 -- 172.21.15.28:0/2879668020 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc5406ce80 con 0x7efc540cabc0 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.662+0000 7efc59600640 1 --2- 172.21.15.28:0/2879668020 >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 0x7efc5406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.663+0000 7efc58c00640 1 --2- 172.21.15.28:0/2879668020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc540cabc0 0x7efc5415e870 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7efc540b3a00 tx=0x7efc4002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13763c71211ba147 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc53e00640 1 -- 172.21.15.28:0/2879668020 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40032440 con 0x7efc540cabc0 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc53e00640 1 -- 172.21.15.28:0/2879668020 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc400325a0 con 0x7efc540cabc0 2024-09-13T17:24:01.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc53e00640 1 -- 172.21.15.28:0/2879668020 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc40032930 con 0x7efc540cabc0 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 -- 172.21.15.28:0/2879668020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc540cabc0 msgr2=0x7efc5415e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 --2- 172.21.15.28:0/2879668020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc540cabc0 0x7efc5415e870 secure :-1 s=READY pgs=2677 cs=0 l=1 rev1=1 crypto rx=0x7efc540b3a00 tx=0x7efc4002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 -- 172.21.15.28:0/2879668020 shutdown_connections 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 --2- 172.21.15.28:0/2879668020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc540cabc0 0x7efc5415e870 unknown :-1 s=CLOSED pgs=2677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 --2- 172.21.15.28:0/2879668020 >> v2:172.21.15.189:3300/0 conn(0x7efc5415a540 0x7efc5406c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 -- 172.21.15.28:0/2879668020 >> 172.21.15.28:0/2879668020 conn(0x7efc54069e80 msgr2=0x7efc5406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:01.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 -- 172.21.15.28:0/2879668020 shutdown_connections 2024-09-13T17:24:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.664+0000 7efc5c000640 1 -- 172.21.15.28:0/2879668020 wait complete. 2024-09-13T17:24:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.666+0000 7efc5c000640 1 Processor -- start 2024-09-13T17:24:01.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.666+0000 7efc5c000640 1 -- start start 2024-09-13T17:24:01.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 0x7efc540b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc5415a540 0x7efc541f3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc541f4990 con 0x7efc5413bd10 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc541f4c70 con 0x7efc540cabc0 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.667+0000 7efc5c000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc540b0100 con 0x7efc5415a540 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44786/0 (socket says 172.21.15.28:44786) 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 -- 172.21.15.28:0/1096892063 learned_addr learned my addr 172.21.15.28:0/1096892063 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc59600640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 0x7efc540b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc58c00640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc5415a540 0x7efc541f3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 -- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc5415a540 msgr2=0x7efc541f3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc5415a540 0x7efc541f3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 -- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 msgr2=0x7efc540b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 0x7efc540b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc5a000640 1 -- 172.21.15.28:0/1096892063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc400094d0 con 0x7efc5413bd10 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.668+0000 7efc59600640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 0x7efc540b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc5a000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00b9e0 tx=0x7efc4c00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c003b70 con 0x7efc5413bd10 2024-09-13T17:24:01.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4c003cd0 con 0x7efc5413bd10 2024-09-13T17:24:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc540b03e0 con 0x7efc5413bd10 2024-09-13T17:24:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4c00cb60 con 0x7efc5413bd10 2024-09-13T17:24:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.669+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc54159120 con 0x7efc5413bd10 2024-09-13T17:24:01.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.670+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5414c470 con 0x7efc5413bd10 2024-09-13T17:24:01.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.671+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efc4c00ccc0 con 0x7efc5413bd10 2024-09-13T17:24:01.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.672+0000 7efc51600640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 0x7efc3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:01.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.672+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7efc4c088d90 con 0x7efc5413bd10 2024-09-13T17:24:01.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.672+0000 7efc59600640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 0x7efc3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:01.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.673+0000 7efc59600640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 0x7efc3406d0c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efc540afbe0 tx=0x7efc48008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:01.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.676+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efc4c054280 con 0x7efc5413bd10 2024-09-13T17:24:01.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:01.920+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files"]} v 0) v1 -- 0x7efc540b05d0 con 0x7efc5413bd10 2024-09-13T17:24:02.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.102+0000 7efc51600640 1 -- 172.21.15.28:0/1096892063 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_backtrace_for_new_files"]}]=0 v785) v1 ==== 147+0+0 (secure 0 0 0) 0x7efc4c0578a0 con 0x7efc5413bd10 2024-09-13T17:24:02.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 msgr2=0x7efc3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 0x7efc3406d0c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7efc540afbe0 tx=0x7efc48008040 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 msgr2=0x7efc54157cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 secure :-1 s=READY pgs=2678 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00b9e0 tx=0x7efc4c00bea0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 shutdown_connections 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efc3406ac00 0x7efc3406d0c0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efc5413bd10 0x7efc54157cb0 unknown :-1 s=CLOSED pgs=2678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efc5415a540 0x7efc541f3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 --2- 172.21.15.28:0/1096892063 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc540cabc0 0x7efc540b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 >> 172.21.15.28:0/1096892063 conn(0x7efc54069e80 msgr2=0x7efc5406c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 shutdown_connections 2024-09-13T17:24:02.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.104+0000 7efc5c000640 1 -- 172.21.15.28:0/1096892063 wait complete. 2024-09-13T17:24:02.123 INFO:tasks.cephfs_test_runner:test_scrub_backtrace_for_new_files (tasks.cephfs.test_scrub.TestScrub) ... ok 2024-09-13T17:24:02.123 INFO:tasks.cephfs_test_runner:Starting test: test_scrub_dup_inode (tasks.cephfs.test_scrub.TestScrub) 2024-09-13T17:24:02.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:02.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.278+0000 7faf8ba00640 1 Processor -- start 2024-09-13T17:24:02.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.278+0000 7faf8ba00640 1 -- start start 2024-09-13T17:24:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.279+0000 7faf8ba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf840cabc0 0x7faf8406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.279+0000 7faf8ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.279+0000 7faf8ba00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf8415e100 con 0x7faf8415a540 2024-09-13T17:24:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.279+0000 7faf8ba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf840adb00 con 0x7faf840cabc0 2024-09-13T17:24:02.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.279+0000 7faf8ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf84071670 con 0x7faf84070cc0 2024-09-13T17:24:02.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf840cabc0 0x7faf8406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf89a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7faf840cabc0 0x7faf8406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:50540/0 (socket says 172.21.15.28:50540) 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 -- 172.21.15.28:0/2562753336 learned_addr learned my addr 172.21.15.28:0/2562753336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 -- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 msgr2=0x7faf84071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 -- 172.21.15.28:0/2562753336 >> v1:172.21.15.189:6789/0 conn(0x7faf8415a540 legacy=0x7faf8415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf83e00640 1 -- 172.21.15.28:0/2562753336 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf8415e3e0 con 0x7faf840cabc0 2024-09-13T17:24:02.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.280+0000 7faf89a00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:02.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.281+0000 7faf83e00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf840cabc0 0x7faf8406c430 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7faf70009870 tx=0x7faf7002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6032c79fcad5fd86 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf83400640 1 -- 172.21.15.28:0/2562753336 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70032440 con 0x7faf840cabc0 2024-09-13T17:24:02.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf83400640 1 -- 172.21.15.28:0/2562753336 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf700325a0 con 0x7faf840cabc0 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf83400640 1 -- 172.21.15.28:0/2562753336 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70032930 con 0x7faf840cabc0 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 -- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf840cabc0 msgr2=0x7faf8406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf840cabc0 0x7faf8406c430 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7faf70009870 tx=0x7faf7002a1e0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 -- 172.21.15.28:0/2562753336 shutdown_connections 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 --2- 172.21.15.28:0/2562753336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf840cabc0 0x7faf8406c430 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 -- 172.21.15.28:0/2562753336 >> 172.21.15.28:0/2562753336 conn(0x7faf84069e80 msgr2=0x7faf8406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 -- 172.21.15.28:0/2562753336 shutdown_connections 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.282+0000 7faf8ba00640 1 -- 172.21.15.28:0/2562753336 wait complete. 2024-09-13T17:24:02.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.284+0000 7faf8ba00640 1 Processor -- start 2024-09-13T17:24:02.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.284+0000 7faf8ba00640 1 -- start start 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf8415a540 0x7faf8413fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf841522f0 con 0x7faf840cabc0 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8414c160 con 0x7faf8415a540 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf8ba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf84140a10 con 0x7faf84070cc0 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf83e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44796/0 (socket says 172.21.15.28:44796) 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.285+0000 7faf83e00640 1 -- 172.21.15.28:0/3529315510 learned_addr learned my addr 172.21.15.28:0/3529315510 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf89000640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf89a00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf8415a540 0x7faf8413fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 -- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 msgr2=0x7faf84150080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84150080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 -- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf8415a540 msgr2=0x7faf8413fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf8415a540 0x7faf8413fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 -- 172.21.15.28:0/3529315510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c009a70 con 0x7faf840cabc0 2024-09-13T17:24:02.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf83e00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7faf70000c00 tx=0x7faf700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70032440 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf70030980 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf700094d0 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.286+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf70030d10 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.287+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf84151f10 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.288+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf84152430 con 0x7faf840cabc0 2024-09-13T17:24:02.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.288+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faf70039040 con 0x7faf840cabc0 2024-09-13T17:24:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.289+0000 7faf80c00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 0x7faf6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.289+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faf700a9a40 con 0x7faf840cabc0 2024-09-13T17:24:02.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.290+0000 7faf89000640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 0x7faf6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.291+0000 7faf89000640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 0x7faf6406d0c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7faf78007fd0 tx=0x7faf78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.293+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faf84152430 con 0x7faf840cabc0 2024-09-13T17:24:02.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.473+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf84152430 con 0x7faf840cabc0 2024-09-13T17:24:02.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.474+0000 7faf80c00640 1 -- 172.21.15.28:0/3529315510 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v224) v1 ==== 74+0+12996 (secure 0 0 0) 0x7faf84152430 con 0x7faf840cabc0 2024-09-13T17:24:02.476 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:02.476 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":224,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:01.087529+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6834/3884798333":"2024-09-14T17:23:57.979229+0000","172.21.15.28:6832/3884798333":"2024-09-14T17:23:57.979229+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 msgr2=0x7faf6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 0x7faf6406d0c0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7faf78007fd0 tx=0x7faf78009040 comp rx=0 tx=0).stop 2024-09-13T17:24:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 msgr2=0x7faf84152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 secure :-1 s=READY pgs=2679 cs=0 l=1 rev1=1 crypto rx=0x7faf70000c00 tx=0x7faf700046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 shutdown_connections 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faf6406ac00 0x7faf6406d0c0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf8415a540 0x7faf8413fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faf840cabc0 0x7faf84152710 unknown :-1 s=CLOSED pgs=2679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 --2- 172.21.15.28:0/3529315510 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faf84070cc0 0x7faf84150080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 >> 172.21.15.28:0/3529315510 conn(0x7faf84069e80 msgr2=0x7faf840701a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 shutdown_connections 2024-09-13T17:24:02.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.476+0000 7faf8ba00640 1 -- 172.21.15.28:0/3529315510 wait complete. 2024-09-13T17:24:02.494 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:24:02.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:24:02.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.650+0000 7ff489e00640 1 Processor -- start 2024-09-13T17:24:02.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.651+0000 7ff489e00640 1 -- start start 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 0x7ff4840b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840722e0 con 0x7ff4840b0c30 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840cb3e0 con 0x7ff4840cc4a0 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4840b1080 con 0x7ff4840cc880 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff4840b0c30 0x7ff4840712d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37936/0 (socket says 172.21.15.28:37936) 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff489400640 1 -- 172.21.15.28:0/3230972618 learned_addr learned my addr 172.21.15.28:0/3230972618 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.652+0000 7ff47be00640 1 --2- 172.21.15.28:0/3230972618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 0x7ff4840b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111299226 0 0) 0x7ff4840722e0 con 0x7ff4840b0c30 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff468003520 con 0x7ff4840b0c30 2024-09-13T17:24:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636051369 0 0) 0x7ff4840cb3e0 con 0x7ff4840cc4a0 2024-09-13T17:24:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4840722e0 con 0x7ff4840cc4a0 2024-09-13T17:24:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff47be00640 1 -- 172.21.15.28:0/3230972618 >> v1:172.21.15.189:6789/0 conn(0x7ff4840cc4a0 legacy=0x7ff4840cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff47be00640 1 -- 172.21.15.28:0/3230972618 >> v1:172.21.15.28:6789/0 conn(0x7ff4840b0c30 legacy=0x7ff4840712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.653+0000 7ff47be00640 1 -- 172.21.15.28:0/3230972618 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484072000 con 0x7ff4840cc880 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.654+0000 7ff47be00640 1 --2- 172.21.15.28:0/3230972618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 0x7ff4840b06f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0097d0 tx=0x7ff46c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ce21f494f94783 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.654+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c032440 con 0x7ff4840cc880 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.654+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c0325a0 con 0x7ff4840cc880 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.654+0000 7ff483e00640 1 -- 172.21.15.28:0/3230972618 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c032910 con 0x7ff4840cc880 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 -- 172.21.15.28:0/3230972618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 msgr2=0x7ff4840b06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 --2- 172.21.15.28:0/3230972618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 0x7ff4840b06f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0097d0 tx=0x7ff46c02a180 comp rx=0 tx=0).stop 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 -- 172.21.15.28:0/3230972618 shutdown_connections 2024-09-13T17:24:02.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 --2- 172.21.15.28:0/3230972618 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc880 0x7ff4840b06f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 -- 172.21.15.28:0/3230972618 >> 172.21.15.28:0/3230972618 conn(0x7ff484069e80 msgr2=0x7ff48406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 -- 172.21.15.28:0/3230972618 shutdown_connections 2024-09-13T17:24:02.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.655+0000 7ff489e00640 1 -- 172.21.15.28:0/3230972618 wait complete. 2024-09-13T17:24:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.657+0000 7ff489e00640 1 Processor -- start 2024-09-13T17:24:02.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.657+0000 7ff489e00640 1 -- start start 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc4a0 0x7ff484151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4840cc880 0x7ff4841515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484153de0 con 0x7ff4840b0c30 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff484151c30 con 0x7ff4840cc880 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff489e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4841541f0 con 0x7ff4840cc4a0 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44806/0 (socket says 172.21.15.28:44806) 2024-09-13T17:24:02.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 -- 172.21.15.28:0/2013340781 learned_addr learned my addr 172.21.15.28:0/2013340781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 -- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc4a0 msgr2=0x7ff484151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc4a0 0x7ff484151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 -- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4840cc880 msgr2=0x7ff4841515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4840cc880 0x7ff4841515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.658+0000 7ff488a00640 1 -- 172.21.15.28:0/2013340781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c009400 con 0x7ff4840b0c30 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff488a00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7ff47400c490 tx=0x7ff47400c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4740052d0 con 0x7ff4840b0c30 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff474005430 con 0x7ff4840b0c30 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474012870 con 0x7ff4840b0c30 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4841544d0 con 0x7ff4840b0c30 2024-09-13T17:24:02.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.659+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff484154a90 con 0x7ff4840b0c30 2024-09-13T17:24:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.660+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4840b06f0 con 0x7ff4840b0c30 2024-09-13T17:24:02.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.661+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff4740129d0 con 0x7ff4840b0c30 2024-09-13T17:24:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.662+0000 7ff481600640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.662+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff474088d90 con 0x7ff4840b0c30 2024-09-13T17:24:02.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.663+0000 7ff47be00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:02.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.663+0000 7ff47be00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff46c034000 tx=0x7ff46c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:02.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.667+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff474054150 con 0x7ff4840b0c30 2024-09-13T17:24:02.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.843+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff48415cea0 con 0x7ff4840b0c30 2024-09-13T17:24:02.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.844+0000 7ff481600640 1 -- 172.21.15.28:0/2013340781 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v224) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff474057770 con 0x7ff4840b0c30 2024-09-13T17:24:02.846 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 msgr2=0x7ff46006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7ff46c034000 tx=0x7ff46c02c000 comp rx=0 tx=0).stop 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 msgr2=0x7ff48414ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 secure :-1 s=READY pgs=2681 cs=0 l=1 rev1=1 crypto rx=0x7ff47400c490 tx=0x7ff47400c950 comp rx=0 tx=0).stop 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 shutdown_connections 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff46006ac00 0x7ff46006d0c0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4840cc880 0x7ff4841515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff4840cc4a0 0x7ff484151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 --2- 172.21.15.28:0/2013340781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4840b0c30 0x7ff48414ef00 unknown :-1 s=CLOSED pgs=2681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 >> 172.21.15.28:0/2013340781 conn(0x7ff484069e80 msgr2=0x7ff4840ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 shutdown_connections 2024-09-13T17:24:02.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:02.846+0000 7ff489e00640 1 -- 172.21.15.28:0/2013340781 wait complete. 2024-09-13T17:24:02.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode' 2024-09-13T17:24:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.018+0000 7ff2a9200640 1 Processor -- start 2024-09-13T17:24:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.018+0000 7ff2a9200640 1 -- start start 2024-09-13T17:24:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a9200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2a4159770 0x7ff2a406fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a9200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 0x7ff2a4070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a406f920 con 0x7ff2a40add00 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a9200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a415b290 con 0x7ff2a4159770 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a9200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a415b570 con 0x7ff2a406f3f0 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2a4159770 0x7ff2a406fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a2800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff2a40add00 0x7ff2a4159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:37948/0 (socket says 172.21.15.28:37948) 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7ff2a4159770 0x7ff2a406fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44816/0 (socket says 172.21.15.28:44816) 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 -- 172.21.15.28:0/1954563534 learned_addr learned my addr 172.21.15.28:0/1954563534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff299400640 1 --2- 172.21.15.28:0/1954563534 >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 0x7ff2a4070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 -- 172.21.15.28:0/1954563534 >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 msgr2=0x7ff2a4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 --2- 172.21.15.28:0/1954563534 >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 0x7ff2a4070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 -- 172.21.15.28:0/1954563534 >> v1:172.21.15.28:6789/0 conn(0x7ff2a40add00 legacy=0x7ff2a4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.020+0000 7ff2a1e00640 1 -- 172.21.15.28:0/1954563534 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a415b850 con 0x7ff2a4159770 2024-09-13T17:24:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.021+0000 7ff299400640 1 --2- 172.21.15.28:0/1954563534 >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 0x7ff2a4070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.021+0000 7ff2a1e00640 1 --2- 172.21.15.28:0/1954563534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a4159770 0x7ff2a406fd20 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7ff2a406b950 tx=0x7ff29402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ed67ab48b08772 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a1400640 1 -- 172.21.15.28:0/1954563534 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294032440 con 0x7ff2a4159770 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a1400640 1 -- 172.21.15.28:0/1954563534 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2940325a0 con 0x7ff2a4159770 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a1400640 1 -- 172.21.15.28:0/1954563534 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294032930 con 0x7ff2a4159770 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 -- 172.21.15.28:0/1954563534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a4159770 msgr2=0x7ff2a406fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1954563534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a4159770 0x7ff2a406fd20 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7ff2a406b950 tx=0x7ff29402a160 comp rx=0 tx=0).stop 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 -- 172.21.15.28:0/1954563534 shutdown_connections 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1954563534 >> v2:172.21.15.189:3300/0 conn(0x7ff2a406f3f0 0x7ff2a4070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1954563534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a4159770 0x7ff2a406fd20 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.022+0000 7ff2a9200640 1 -- 172.21.15.28:0/1954563534 >> 172.21.15.28:0/1954563534 conn(0x7ff2a40b0270 msgr2=0x7ff2a40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.023+0000 7ff2a9200640 1 -- 172.21.15.28:0/1954563534 shutdown_connections 2024-09-13T17:24:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.023+0000 7ff2a9200640 1 -- 172.21.15.28:0/1954563534 wait complete. 2024-09-13T17:24:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.024+0000 7ff2a9200640 1 Processor -- start 2024-09-13T17:24:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.024+0000 7ff2a9200640 1 -- start start 2024-09-13T17:24:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a406f3f0 0x7ff2a40b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2a4159770 0x7ff2a4146d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4144a60 con 0x7ff2a40add00 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4144d40 con 0x7ff2a406f3f0 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.025+0000 7ff2a9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a4147620 con 0x7ff2a4159770 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44830/0 (socket says 172.21.15.28:44830) 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 -- 172.21.15.28:0/1174832982 learned_addr learned my addr 172.21.15.28:0/1174832982 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff2a1e00640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a406f3f0 0x7ff2a40b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff2a2800640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2a4159770 0x7ff2a4146d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 -- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2a4159770 msgr2=0x7ff2a4146d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:03.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2a4159770 0x7ff2a4146d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 -- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a406f3f0 msgr2=0x7ff2a40b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a406f3f0 0x7ff2a40b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 -- 172.21.15.28:0/1174832982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2940094d0 con 0x7ff2a40add00 2024-09-13T17:24:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.026+0000 7ff299400640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e730 tx=0x7ff28c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.027+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c004230 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.027+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28c004390 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.027+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28c00c890 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.027+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a4147900 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.027+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a4144fe0 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.028+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a4145170 con 0x7ff2a40add00 2024-09-13T17:24:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.029+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff28c00c9f0 con 0x7ff2a40add00 2024-09-13T17:24:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.029+0000 7ff29aa00640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 0x7ff28006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.030+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff28c088a50 con 0x7ff2a40add00 2024-09-13T17:24:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.030+0000 7ff2a1e00640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 0x7ff28006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.031+0000 7ff2a1e00640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 0x7ff28006d0f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff294034000 tx=0x7ff29402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:03.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.033+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff28c054070 con 0x7ff2a40add00 2024-09-13T17:24:03.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:03.277+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode"]} v 0) v1 -- 0x7ff2a406b230 con 0x7ff2a40add00 2024-09-13T17:24:04.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.112+0000 7ff29aa00640 1 -- 172.21.15.28:0/1174832982 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode"]}]=0 v787) v1 ==== 136+0+0 (secure 0 0 0) 0x7ff28c057690 con 0x7ff2a40add00 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.114+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 msgr2=0x7ff28006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.114+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 0x7ff28006d0f0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7ff294034000 tx=0x7ff29402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.114+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 msgr2=0x7ff2a4144520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 secure :-1 s=READY pgs=2684 cs=0 l=1 rev1=1 crypto rx=0x7ff28c00e730 tx=0x7ff28c00ebf0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 shutdown_connections 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff28006ac30 0x7ff28006d0f0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff2a4159770 0x7ff2a4146d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2a40add00 0x7ff2a4144520 unknown :-1 s=CLOSED pgs=2684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 --2- 172.21.15.28:0/1174832982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff2a406f3f0 0x7ff2a40b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 >> 172.21.15.28:0/1174832982 conn(0x7ff2a40b0270 msgr2=0x7ff2a40ad360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:04.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 shutdown_connections 2024-09-13T17:24:04.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.115+0000 7ff2a9200640 1 -- 172.21.15.28:0/1174832982 wait complete. 2024-09-13T17:24:04.136 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T17:24:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.297+0000 7f811f200640 1 Processor -- start 2024-09-13T17:24:04.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.297+0000 7f811f200640 1 -- start start 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811f200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8120071210 0x7f81200cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811f200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8120159e10 con 0x7f81200cddb0 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811f200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812015a0f0 con 0x7f8120070e30 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811f200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200ce240 con 0x7f8120071210 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f81200cddb0 0x7f812015aed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42222/0 (socket says 172.21.15.28:42222) 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811e800640 1 -- 172.21.15.28:0/2900636414 learned_addr learned my addr 172.21.15.28:0/2900636414 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:04.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f8115400640 1 --2- 172.21.15.28:0/2900636414 >> v2:172.21.15.189:3300/0 conn(0x7f8120071210 0x7f81200cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811d400640 1 -- 172.21.15.28:0/2900636414 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738368079 0 0) 0x7f8120159e10 con 0x7f81200cddb0 2024-09-13T17:24:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.299+0000 7f811d400640 1 -- 172.21.15.28:0/2900636414 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8104003520 con 0x7f81200cddb0 2024-09-13T17:24:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.300+0000 7f8115400640 1 -- 172.21.15.28:0/2900636414 >> v1:172.21.15.189:6789/0 conn(0x7f8120070e30 legacy=0x7f8120071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.300+0000 7f8115400640 1 -- 172.21.15.28:0/2900636414 >> v1:172.21.15.28:6789/0 conn(0x7f81200cddb0 legacy=0x7f812015aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.300+0000 7f8115400640 1 -- 172.21.15.28:0/2900636414 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812015a3d0 con 0x7f8120071210 2024-09-13T17:24:04.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f8115400640 1 --2- 172.21.15.28:0/2900636414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120071210 0x7f81200cd870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f81080097a0 tx=0x7f810802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef49ade7df4dc4a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f811d400640 1 -- 172.21.15.28:0/2900636414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108032440 con 0x7f8120071210 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f811d400640 1 -- 172.21.15.28:0/2900636414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81080325a0 con 0x7f8120071210 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f811d400640 1 -- 172.21.15.28:0/2900636414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108032930 con 0x7f8120071210 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f811f200640 1 -- 172.21.15.28:0/2900636414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120071210 msgr2=0x7f81200cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.301+0000 7f811f200640 1 --2- 172.21.15.28:0/2900636414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120071210 0x7f81200cd870 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f81080097a0 tx=0x7f810802a230 comp rx=0 tx=0).stop 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.302+0000 7f811f200640 1 -- 172.21.15.28:0/2900636414 shutdown_connections 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.302+0000 7f811f200640 1 --2- 172.21.15.28:0/2900636414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120071210 0x7f81200cd870 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.302+0000 7f811f200640 1 -- 172.21.15.28:0/2900636414 >> 172.21.15.28:0/2900636414 conn(0x7f8120069e80 msgr2=0x7f812006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.302+0000 7f811f200640 1 -- 172.21.15.28:0/2900636414 shutdown_connections 2024-09-13T17:24:04.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.302+0000 7f811f200640 1 -- 172.21.15.28:0/2900636414 wait complete. 2024-09-13T17:24:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.303+0000 7f811f200640 1 Processor -- start 2024-09-13T17:24:04.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.303+0000 7f811f200640 1 -- start start 2024-09-13T17:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120070e30 0x7f81201f6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81200cddb0 0x7f8120158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200a9390 con 0x7f8120071210 2024-09-13T17:24:04.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81200a9580 con 0x7f8120070e30 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.304+0000 7f811f200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81200cb170 con 0x7f81200cddb0 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46068/0 (socket says 172.21.15.28:46068) 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 -- 172.21.15.28:0/1845670381 learned_addr learned my addr 172.21.15.28:0/1845670381 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f811de00640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120070e30 0x7f81201f6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f811e800640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81200cddb0 0x7f8120158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 -- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81200cddb0 msgr2=0x7f8120158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81200cddb0 0x7f8120158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 -- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120070e30 msgr2=0x7f81201f6410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120070e30 0x7f81201f6410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.305+0000 7f8115400640 1 -- 172.21.15.28:0/1845670381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108009400 con 0x7f8120071210 2024-09-13T17:24:04.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f8115400640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f8108035040 tx=0x7f81080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108006e10 con 0x7f8120071210 2024-09-13T17:24:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108032e90 con 0x7f8120071210 2024-09-13T17:24:04.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108030be0 con 0x7f8120071210 2024-09-13T17:24:04.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81200cb450 con 0x7f8120071210 2024-09-13T17:24:04.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.306+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81200a9860 con 0x7f8120071210 2024-09-13T17:24:04.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.307+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81200cb970 con 0x7f8120071210 2024-09-13T17:24:04.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.308+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8108039040 con 0x7f8120071210 2024-09-13T17:24:04.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.309+0000 7f8116a00640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 0x7f80fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.309+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f81080a9850 con 0x7f8120071210 2024-09-13T17:24:04.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.309+0000 7f811de00640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 0x7f80fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.310+0000 7f811de00640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 0x7f80fc06d0c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8110000f60 tx=0x7f8110009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:04.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.313+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f81200cb970 con 0x7f8120071210 2024-09-13T17:24:04.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.493+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f81200cb970 con 0x7f8120071210 2024-09-13T17:24:04.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.497+0000 7f8116a00640 1 -- 172.21.15.28:0/1845670381 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f81200cb970 con 0x7f8120071210 2024-09-13T17:24:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.499+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 msgr2=0x7f80fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.499+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 0x7f80fc06d0c0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f8110000f60 tx=0x7f8110009040 comp rx=0 tx=0).stop 2024-09-13T17:24:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.499+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 msgr2=0x7f81200a8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.499+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 secure :-1 s=READY pgs=2686 cs=0 l=1 rev1=1 crypto rx=0x7f8108035040 tx=0x7f81080046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 shutdown_connections 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f80fc06ac00 0x7f80fc06d0c0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f81200cddb0 0x7f8120158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8120071210 0x7f81200a8e50 unknown :-1 s=CLOSED pgs=2686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 --2- 172.21.15.28:0/1845670381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8120070e30 0x7f81201f6410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 >> 172.21.15.28:0/1845670381 conn(0x7f8120069e80 msgr2=0x7f81200b2140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 shutdown_connections 2024-09-13T17:24:04.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.500+0000 7f811f200640 1 -- 172.21.15.28:0/1845670381 wait complete. 2024-09-13T17:24:04.517 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:24:04.517 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:24:04.592 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:24:04.681 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:24:04.681 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:04.681 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:04.718 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:24:04.718 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:24:04.719 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:04.757 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:24:04.758 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:04.758 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:04.795 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:24:04.796 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:24:04.796 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:04.834 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:24:04.834 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T17:24:04.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.988+0000 7f5b48600640 1 Processor -- start 2024-09-13T17:24:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.988+0000 7f5b48600640 1 -- start start 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.989+0000 7f5b48600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b48600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 0x7f5b4006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b48600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4006cc70 con 0x7f5b4006f970 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b48600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b4015b0b0 con 0x7f5b401592a0 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b48600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b40159d90 con 0x7f5b4006c2d0 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b45c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b46600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 0x7f5b4006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b45c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:36842/0 (socket says 172.21.15.28:36842) 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b45c00640 1 -- 172.21.15.28:0/467544253 learned_addr learned my addr 172.21.15.28:0/467544253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:04.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b45200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f5b4006f970 0x7f5b40159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:39840/0 (socket says 172.21.15.28:39840) 2024-09-13T17:24:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b3fe00640 1 -- 172.21.15.28:0/467544253 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217053711 0 0) 0x7f5b4006cc70 con 0x7f5b4006f970 2024-09-13T17:24:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.990+0000 7f5b3fe00640 1 -- 172.21.15.28:0/467544253 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b28003520 con 0x7f5b4006f970 2024-09-13T17:24:04.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.991+0000 7f5b46600640 1 -- 172.21.15.28:0/467544253 >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 msgr2=0x7f5b4015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.991+0000 7f5b46600640 1 --2- 172.21.15.28:0/467544253 >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.991+0000 7f5b46600640 1 -- 172.21.15.28:0/467544253 >> v1:172.21.15.189:6789/0 conn(0x7f5b4006f970 legacy=0x7f5b40159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.991+0000 7f5b46600640 1 -- 172.21.15.28:0/467544253 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b400b2040 con 0x7f5b4006c2d0 2024-09-13T17:24:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.991+0000 7f5b45c00640 1 --2- 172.21.15.28:0/467544253 >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:04.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.992+0000 7f5b46600640 1 --2- 172.21.15.28:0/467544253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 0x7f5b4006c730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009d90 tx=0x7f5b3802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4998ba3a6d731680 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.992+0000 7f5b3fe00640 1 -- 172.21.15.28:0/467544253 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38007740 con 0x7f5b4006c2d0 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.992+0000 7f5b3fe00640 1 -- 172.21.15.28:0/467544253 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3802d050 con 0x7f5b4006c2d0 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.992+0000 7f5b3fe00640 1 -- 172.21.15.28:0/467544253 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38007aa0 con 0x7f5b4006c2d0 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 -- 172.21.15.28:0/467544253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 msgr2=0x7f5b4006c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 --2- 172.21.15.28:0/467544253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 0x7f5b4006c730 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5b38009d90 tx=0x7f5b3802a660 comp rx=0 tx=0).stop 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 -- 172.21.15.28:0/467544253 shutdown_connections 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 --2- 172.21.15.28:0/467544253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006c2d0 0x7f5b4006c730 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 --2- 172.21.15.28:0/467544253 >> v2:172.21.15.189:3300/0 conn(0x7f5b401592a0 0x7f5b4015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 -- 172.21.15.28:0/467544253 >> 172.21.15.28:0/467544253 conn(0x7f5b40069e80 msgr2=0x7f5b4006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 -- 172.21.15.28:0/467544253 shutdown_connections 2024-09-13T17:24:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.993+0000 7f5b48600640 1 -- 172.21.15.28:0/467544253 wait complete. 2024-09-13T17:24:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.994+0000 7f5b48600640 1 Processor -- start 2024-09-13T17:24:04.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.995+0000 7f5b48600640 1 -- start start 2024-09-13T17:24:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006f970 0x7f5b401f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b400cb8e0 con 0x7f5b401592a0 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401f6260 con 0x7f5b4006c2d0 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b48600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b401f6540 con 0x7f5b4006f970 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b46600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:36846/0 (socket says 172.21.15.28:36846) 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b46600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46084/0 (socket says 172.21.15.28:46084) 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 -- 172.21.15.28:0/3043608148 learned_addr learned my addr 172.21.15.28:0/3043608148 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45200640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006f970 0x7f5b401f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 -- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006f970 msgr2=0x7f5b401f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:04.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006f970 0x7f5b401f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 -- 172.21.15.28:0/3043608148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 msgr2=0x7f5b4015b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b45c00640 1 -- 172.21.15.28:0/3043608148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3400b770 con 0x7f5b4006c2d0 2024-09-13T17:24:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.996+0000 7f5b46600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:05.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b45c00640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400ed40 tx=0x7f5b3400d720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b34019680 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b340197e0 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b34019b50 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b38009a70 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.997+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b400cbe10 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.998+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b40159bb0 con 0x7f5b4006c2d0 2024-09-13T17:24:05.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:04.999+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5b34010440 con 0x7f5b4006c2d0 2024-09-13T17:24:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.000+0000 7f5b3d600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 0x7f5b2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.000+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5b34014070 con 0x7f5b4006c2d0 2024-09-13T17:24:05.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.000+0000 7f5b45200640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 0x7f5b2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.001+0000 7f5b45200640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 0x7f5b2006d0c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5b400cb800 tx=0x7f5b2c007cf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:05.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.005+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5b34055060 con 0x7f5b4006c2d0 2024-09-13T17:24:05.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.221+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b400b3740 con 0x7f5b4006c2d0 2024-09-13T17:24:05.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.222+0000 7f5b3d600640 1 -- 172.21.15.28:0/3043608148 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+3543 (secure 0 0 0) 0x7f5b3401e090 con 0x7f5b4006c2d0 2024-09-13T17:24:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:05.224 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":183,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":9287,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2685906056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":2685906056},{"type":"v1","addr":"172.21.15.189:6837","nonce":2685906056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":176},{"gid":10924,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3652516572","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3652516572},{"type":"v1","addr":"172.21.15.28:6835","nonce":3652516572}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":10927,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/3831628329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":3831628329},{"type":"v1","addr":"172.21.15.189:6835","nonce":3831628329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":175},{"gid":12228,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3384165913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3384165913},{"type":"v1","addr":"172.21.15.28:6834","nonce":3384165913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":183}],"filesystems":[]} 2024-09-13T17:24:05.224 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 183 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 msgr2=0x7f5b2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 0x7f5b2006d0c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f5b400cb800 tx=0x7f5b2c007cf0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 msgr2=0x7f5b4014bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400ed40 tx=0x7f5b3400d720 comp rx=0 tx=0).stop 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 shutdown_connections 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5b2006ac00 0x7f5b2006d0c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5b401592a0 0x7f5b4015b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5b4006f970 0x7f5b401f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 --2- 172.21.15.28:0/3043608148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b4006c2d0 0x7f5b4014bfd0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 >> 172.21.15.28:0/3043608148 conn(0x7f5b40069e80 msgr2=0x7f5b4006b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 shutdown_connections 2024-09-13T17:24:05.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.224+0000 7f5b48600640 1 -- 172.21.15.28:0/3043608148 wait complete. 2024-09-13T17:24:05.242 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T17:24:05.243 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T17:24:05.243 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:24:05.243 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:24:05.243 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T17:24:05.243 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T17:24:05.243 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:24:05.243 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:24:05.244 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T17:24:05.244 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T17:24:05.244 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:24:05.244 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:24:05.244 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T17:24:05.244 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T17:24:05.244 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:24:05.244 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:24:05.257 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:24:05.258 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T17:24:05.258 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:24:05.259 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T17:24:05.260 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:24:05.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T17:24:05.261 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:24:05.261 DEBUG:teuthology.orchestra.run.smithi189:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T17:24:05.303 INFO:tasks.ceph.mds.c:Started 2024-09-13T17:24:05.303 DEBUG:teuthology.parallel:result is None 2024-09-13T17:24:05.305 INFO:tasks.ceph.mds.b:Started 2024-09-13T17:24:05.305 DEBUG:teuthology.parallel:result is None 2024-09-13T17:24:05.305 INFO:tasks.ceph.mds.a:Started 2024-09-13T17:24:05.305 DEBUG:teuthology.parallel:result is None 2024-09-13T17:24:05.307 INFO:tasks.ceph.mds.d:Started 2024-09-13T17:24:05.307 DEBUG:teuthology.parallel:result is None 2024-09-13T17:24:05.307 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T17:24:05.436 INFO:tasks.ceph.mds.b.smithi189.stdout:starting mds.b at 2024-09-13T17:24:05.437 INFO:tasks.ceph.mds.d.smithi189.stdout:starting mds.d at 2024-09-13T17:24:05.438 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:24:05.438 INFO:tasks.ceph.mds.c.smithi028.stdout:starting mds.c at 2024-09-13T17:24:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.470+0000 7ffbe2800640 1 Processor -- start 2024-09-13T17:24:05.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.470+0000 7ffbe2800640 1 -- start start 2024-09-13T17:24:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.471+0000 7ffbe2800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ffbdc0b1d70 0x7ffbdc164820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.471+0000 7ffbe2800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc0b2590 con 0x7ffbdc06ca80 2024-09-13T17:24:05.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.471+0000 7ffbe2800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbdc0b22b0 con 0x7ffbdc146f20 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.471+0000 7ffbe2800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbdc0ce3e0 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe0a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ffbdc06ca80 0x7ffbdc0b1660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42246/0 (socket says 172.21.15.28:42246) 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe0a00640 1 -- 172.21.15.28:0/611611121 learned_addr learned my addr 172.21.15.28:0/611611121 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe1e00640 1 --2- 172.21.15.28:0/611611121 >> v2:172.21.15.189:3300/0 conn(0x7ffbdc0b1d70 0x7ffbdc164820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3362883356 0 0) 0x7ffbdc0b2590 con 0x7ffbdc06ca80 2024-09-13T17:24:05.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbc4003520 con 0x7ffbdc06ca80 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257243675 0 0) 0x7ffbdc0b22b0 con 0x7ffbdc146f20 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbdc0b2590 con 0x7ffbdc146f20 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe1e00640 1 -- 172.21.15.28:0/611611121 >> v1:172.21.15.189:6789/0 conn(0x7ffbdc146f20 legacy=0x7ffbdc06c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe1e00640 1 -- 172.21.15.28:0/611611121 >> v1:172.21.15.28:6789/0 conn(0x7ffbdc06ca80 legacy=0x7ffbdc0b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.472+0000 7ffbe1e00640 1 -- 172.21.15.28:0/611611121 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc1613d0 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.473+0000 7ffbe1e00640 1 --2- 172.21.15.28:0/611611121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc164820 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009d90 tx=0x7ffbd402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=231034656d7f7b75 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:05.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4007740 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd402d050 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbdbe00640 1 -- 172.21.15.28:0/611611121 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4007ac0 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 -- 172.21.15.28:0/611611121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 msgr2=0x7ffbdc164820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 --2- 172.21.15.28:0/611611121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc164820 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7ffbd4009d90 tx=0x7ffbd402a6f0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 -- 172.21.15.28:0/611611121 shutdown_connections 2024-09-13T17:24:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 --2- 172.21.15.28:0/611611121 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc164820 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 -- 172.21.15.28:0/611611121 >> 172.21.15.28:0/611611121 conn(0x7ffbdc073470 msgr2=0x7ffbdc069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 -- 172.21.15.28:0/611611121 shutdown_connections 2024-09-13T17:24:05.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.474+0000 7ffbe2800640 1 -- 172.21.15.28:0/611611121 wait complete. 2024-09-13T17:24:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.476+0000 7ffbe2800640 1 Processor -- start 2024-09-13T17:24:05.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.476+0000 7ffbe2800640 1 -- start start 2024-09-13T17:24:05.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc15d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbdc146f20 0x7ffbdc0c35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc15d660 con 0x7ffbdc06ca80 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc131f20 con 0x7ffbdc0b1d70 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe2800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc132200 con 0x7ffbdc146f20 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46126/0 (socket says 172.21.15.28:46126) 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe1400640 1 -- 172.21.15.28:0/3929061658 learned_addr learned my addr 172.21.15.28:0/3929061658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe0a00640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc15d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.477+0000 7ffbe1e00640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbdc146f20 0x7ffbdc0c35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 -- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbdc146f20 msgr2=0x7ffbdc0c35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbdc146f20 0x7ffbdc0c35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 -- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 msgr2=0x7ffbdc15d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc15d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 -- 172.21.15.28:0/3929061658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd4009a70 con 0x7ffbdc06ca80 2024-09-13T17:24:05.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbe1400640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c430 tx=0x7ffbd000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd00052d0 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd0005450 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.478+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd0003830 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.479+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc1324e0 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.479+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc132aa0 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.480+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc0cd8a0 con 0x7ffbdc06ca80 2024-09-13T17:24:05.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.481+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ffbd0003990 con 0x7ffbdc06ca80 2024-09-13T17:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.482+0000 7ffbd9600640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 0x7ffbbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:05.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.482+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffbd0088610 con 0x7ffbdc06ca80 2024-09-13T17:24:05.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.482+0000 7ffbe0a00640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 0x7ffbbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:05.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.483+0000 7ffbe0a00640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 0x7ffbbc06d0c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc1622e0 tx=0x7ffbc8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:05.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.487+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ffbd0053b90 con 0x7ffbdc06ca80 2024-09-13T17:24:05.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:05.677+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ffbdc161eb0 con 0x7ffbdc06ca80 2024-09-13T17:24:06.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.465+0000 7ffbd9600640 1 -- 172.21.15.28:0/3929061658 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v229) v1 ==== 97+0+0 (secure 0 0 0) 0x7ffbd00571b0 con 0x7ffbdc06ca80 2024-09-13T17:24:06.467 INFO:teuthology.orchestra.run.smithi028.stderr: removed all blocklist entries 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 msgr2=0x7ffbbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 0x7ffbbc06d0c0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffbdc1622e0 tx=0x7ffbc8007d30 comp rx=0 tx=0).stop 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 msgr2=0x7ffbdc15de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7ffbd000c430 tx=0x7ffbd000c8f0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 shutdown_connections 2024-09-13T17:24:06.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ffbbc06ac00 0x7ffbbc06d0c0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ffbdc146f20 0x7ffbdc0c35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffbdc0b1d70 0x7ffbdc15d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 --2- 172.21.15.28:0/3929061658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffbdc06ca80 0x7ffbdc15de20 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 >> 172.21.15.28:0/3929061658 conn(0x7ffbdc073470 msgr2=0x7ffbdc0b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 shutdown_connections 2024-09-13T17:24:06.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.468+0000 7ffbe2800640 1 -- 172.21.15.28:0/3929061658 wait complete. 2024-09-13T17:24:06.489 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T17:24:06.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.647+0000 7fab5c600640 1 Processor -- start 2024-09-13T17:24:06.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.647+0000 7fab5c600640 1 -- start start 2024-09-13T17:24:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.648+0000 7fab5c600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab5415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab5c600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab5406f970 0x7fab54159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab5c600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab540b2040 con 0x7fab5406c2d0 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab5c600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab5415b0b0 con 0x7fab5406f970 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab5c600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab54159ff0 con 0x7fab541592a0 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab5406f970 0x7fab54159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fab5406f970 0x7fab54159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46134/0 (socket says 172.21.15.28:46134) 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 -- 172.21.15.28:0/2596442373 learned_addr learned my addr 172.21.15.28:0/2596442373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59c00640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab5415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 -- 172.21.15.28:0/2596442373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 msgr2=0x7fab5415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab5415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 -- 172.21.15.28:0/2596442373 >> v1:172.21.15.189:6789/0 conn(0x7fab5406c2d0 legacy=0x7fab540b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.649+0000 7fab59200640 1 -- 172.21.15.28:0/2596442373 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab5415a2d0 con 0x7fab5406f970 2024-09-13T17:24:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.650+0000 7fab59200640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406f970 0x7fab54159680 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fab40009870 tx=0x7fab4002c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e99b0ba05dc83fa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab53e00640 1 -- 172.21.15.28:0/2596442373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40031710 con 0x7fab5406f970 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab53e00640 1 -- 172.21.15.28:0/2596442373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4002f050 con 0x7fab5406f970 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab53e00640 1 -- 172.21.15.28:0/2596442373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40031a90 con 0x7fab5406f970 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 -- 172.21.15.28:0/2596442373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406f970 msgr2=0x7fab54159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406f970 0x7fab54159680 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7fab40009870 tx=0x7fab4002c480 comp rx=0 tx=0).stop 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 -- 172.21.15.28:0/2596442373 shutdown_connections 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406f970 0x7fab54159680 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 --2- 172.21.15.28:0/2596442373 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab5415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 -- 172.21.15.28:0/2596442373 >> 172.21.15.28:0/2596442373 conn(0x7fab54069e80 msgr2=0x7fab5406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 -- 172.21.15.28:0/2596442373 shutdown_connections 2024-09-13T17:24:06.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.651+0000 7fab5c600640 1 -- 172.21.15.28:0/2596442373 wait complete. 2024-09-13T17:24:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.653+0000 7fab5c600640 1 Processor -- start 2024-09-13T17:24:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.653+0000 7fab5c600640 1 -- start start 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab5406f970 0x7fab54152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab54154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab540cd4b0 con 0x7fab5406c2d0 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab541f6150 con 0x7fab5406f970 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab5c600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab541f6430 con 0x7fab541592a0 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab59c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab59c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46148/0 (socket says 172.21.15.28:46148) 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.654+0000 7fab59c00640 1 -- 172.21.15.28:0/500421965 learned_addr learned my addr 172.21.15.28:0/500421965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab5a600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab54154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59200640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab5406f970 0x7fab54152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 -- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 msgr2=0x7fab54154c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab54154c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 -- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab5406f970 msgr2=0x7fab54152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab5406f970 0x7fab54152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 -- 172.21.15.28:0/500421965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab400094d0 con 0x7fab5406c2d0 2024-09-13T17:24:06.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab59c00640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fab4800d5f0 tx=0x7fab4800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab480125f0 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab48012750 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab48012b00 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.655+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab541f6710 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.656+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab541527a0 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.657+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab540cd720 con 0x7fab5406c2d0 2024-09-13T17:24:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.658+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fab48012c60 con 0x7fab5406c2d0 2024-09-13T17:24:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.659+0000 7fab51600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 0x7fab3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.659+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fab48089500 con 0x7fab5406c2d0 2024-09-13T17:24:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.659+0000 7fab59200640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 0x7fab3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:06.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.660+0000 7fab59200640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 0x7fab3406d0c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fab40035000 tx=0x7fab40008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:06.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.663+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fab48054a20 con 0x7fab5406c2d0 2024-09-13T17:24:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.908+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fab5415ab70 con 0x7fab5406c2d0 2024-09-13T17:24:06.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.909+0000 7fab51600640 1 -- 172.21.15.28:0/500421965 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v29) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fab48058040 con 0x7fab5406c2d0 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "auth_dump": [ 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.a", 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRm4epQOBAA4xEyGsOTIZIdGirN6zhJoA==", 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:24:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.b", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmu4VXEBAAqiXiiJd3NSm1Zfwf2/1QlA==", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.c", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmrHGDBhAAbjQA71xZYAGda00KiTcINw==", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mds.d", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm3jsjGhAAoAYj+AFGZ8aPRkE8BHeKFA==", 2024-09-13T17:24:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.0", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBCceRm+tqEDRAAajL/SV2E2nj27xeO4x1o7g==", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.1", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBFceRmtNYSExAAsHUt6QpN1ToBq7+yGa42wA==", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.2", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBIceRmQRSwFxAAPu5RuJLlyLx+6OJUaF1Pnw==", 2024-09-13T17:24:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.3", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBLceRmT/voGhAAaaQwWv3igd1FCEaW+FfGxg==", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.4", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBRceRmGidIFhAAFLIVde7NYWz3wJDz47tp3g==", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.5", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBUceRmH90IHBAAZ4hJSg2OutA/sGdxyq1j/A==", 2024-09-13T17:24:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.6", 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBXceRmFzY7IBAAr3j86aeCSu+hTPOAxhNiTA==", 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "osd.7", 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBaceRmvTzuJBAAgeEHLfKkEPjA2sm135QfZA==", 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow profile osd", 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile osd", 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.0", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmAArEIhAA1c4oWrwsDgjFw9HmpMnzOA==", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow r", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.1", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmutC8LxAA/J104TRiEhbIhPhJdUp6Pg==", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.2", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRmnOhFKRAAmEgQyz+5C5xLaoDCIF/xsA==", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:24:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.3", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQBAceRm4VctNhAAuK1OwJyAw2Olbjv7Yo6gkQ==", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow r", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow rw", 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow rwx" 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "client.admin", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmItQGCRAAFDs5qyKr/4IyRVC4qrPF0w==", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr": "allow *", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow *", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.x", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmEhSWMRAAU0Krjy915oZiHSknh6q3Vg==", 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "entity": "mgr.y", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "key": "AQA/ceRmgM3BKhAAKpWpM92ky0FkXQCYzjoDtg==", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": { 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": "allow *", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "mon": "allow profile mgr", 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "osd": "allow *" 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-09-13T17:24:06.925 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:06.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 msgr2=0x7fab3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 0x7fab3406d0c0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fab40035000 tx=0x7fab40008030 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 msgr2=0x7fab540c3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7fab4800d5f0 tx=0x7fab4800dab0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 shutdown_connections 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fab3406ac00 0x7fab3406d0c0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fab541592a0 0x7fab54154c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab5406f970 0x7fab54152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 --2- 172.21.15.28:0/500421965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab5406c2d0 0x7fab540c3580 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 >> 172.21.15.28:0/500421965 conn(0x7fab54069e80 msgr2=0x7fab540ae1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 shutdown_connections 2024-09-13T17:24:06.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:06.911+0000 7fab5c600640 1 -- 172.21.15.28:0/500421965 wait complete. 2024-09-13T17:24:06.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.083+0000 7f0ab7200640 1 Processor -- start 2024-09-13T17:24:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.084+0000 7f0ab7200640 1 -- start start 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab80b0b50 0x7f0ab80b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab7200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ab8159820 0x7f0ab80b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ab80c3850 con 0x7f0ab8159440 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab7200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab80b2d60 con 0x7f0ab8159820 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab815a930 con 0x7f0ab80b0b50 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab5e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0ab8159440 0x7f0ab80b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42262/0 (socket says 172.21.15.28:42262) 2024-09-13T17:24:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab5e00640 1 -- 172.21.15.28:0/21670995 learned_addr learned my addr 172.21.15.28:0/21670995 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0ab6800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0ab8159820 0x7f0ab80b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.085+0000 7f0aafe00640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab80b0b50 0x7f0ab80b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.086+0000 7f0ab6800640 1 -- 172.21.15.28:0/21670995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab80b0b50 msgr2=0x7f0ab80b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.086+0000 7f0ab6800640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab80b0b50 0x7f0ab80b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.086+0000 7f0ab6800640 1 -- 172.21.15.28:0/21670995 >> v1:172.21.15.28:6789/0 conn(0x7f0ab8159440 legacy=0x7f0ab80b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.086+0000 7f0ab6800640 1 -- 172.21.15.28:0/21670995 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab8159d20 con 0x7f0ab8159820 2024-09-13T17:24:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.086+0000 7f0ab5400640 1 -- 172.21.15.28:0/21670995 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379788932 0 0) 0x7f0ab80c3850 con 0x7f0ab8159440 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.087+0000 7f0ab6800640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8159820 0x7f0ab80b2820 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009d90 tx=0x7f0ab002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9a09ef659c8e25a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.087+0000 7f0ab5400640 1 -- 172.21.15.28:0/21670995 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0007740 con 0x7f0ab8159820 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.087+0000 7f0ab5400640 1 -- 172.21.15.28:0/21670995 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab002d050 con 0x7f0ab8159820 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.087+0000 7f0ab5400640 1 -- 172.21.15.28:0/21670995 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab0007ac0 con 0x7f0ab8159820 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 -- 172.21.15.28:0/21670995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8159820 msgr2=0x7f0ab80b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8159820 0x7f0ab80b2820 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0009d90 tx=0x7f0ab002a6f0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 -- 172.21.15.28:0/21670995 shutdown_connections 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab8159820 0x7f0ab80b2820 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 --2- 172.21.15.28:0/21670995 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab80b0b50 0x7f0ab80b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 -- 172.21.15.28:0/21670995 >> 172.21.15.28:0/21670995 conn(0x7f0ab8072e40 msgr2=0x7f0ab8073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 -- 172.21.15.28:0/21670995 shutdown_connections 2024-09-13T17:24:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.088+0000 7f0ab7200640 1 -- 172.21.15.28:0/21670995 wait complete. 2024-09-13T17:24:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.089+0000 7f0ab7200640 1 Processor -- start 2024-09-13T17:24:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 -- start start 2024-09-13T17:24:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159440 0x7f0ab81583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 0x7f0ab81f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8158900 con 0x7f0ab80b0b50 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8158b60 con 0x7f0ab8159820 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.090+0000 7f0ab7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80c3da0 con 0x7f0ab8159440 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46178/0 (socket says 172.21.15.28:46178) 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 -- 172.21.15.28:0/3491013893 learned_addr learned my addr 172.21.15.28:0/3491013893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0aafe00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159440 0x7f0ab81583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab6800640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 0x7f0ab81f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 -- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159440 msgr2=0x7f0ab81583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159440 0x7f0ab81583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 -- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 msgr2=0x7f0ab81f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 0x7f0ab81f1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 -- 172.21.15.28:0/3491013893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab0009a70 con 0x7f0ab80b0b50 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab6800640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 0x7f0ab81f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.091+0000 7f0ab5e00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c4d0 tx=0x7f0aa800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.092+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa80074e0 con 0x7f0ab80b0b50 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.092+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa8007660 con 0x7f0ab80b0b50 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.092+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8004a20 con 0x7f0ab80b0b50 2024-09-13T17:24:07.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.092+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab80c4080 con 0x7f0ab80b0b50 2024-09-13T17:24:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.092+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab80c4640 con 0x7f0ab80b0b50 2024-09-13T17:24:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.093+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab806b2e0 con 0x7f0ab80b0b50 2024-09-13T17:24:07.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.094+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0aa8004b80 con 0x7f0ab80b0b50 2024-09-13T17:24:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.094+0000 7f0aae000640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 0x7f0a9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.095+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0aa8088160 con 0x7f0ab80b0b50 2024-09-13T17:24:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.095+0000 7f0aafe00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 0x7f0a9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.096+0000 7f0aafe00640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 0x7f0a9406d0f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80adce0 tx=0x7f0aa0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.098+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0aa80536a0 con 0x7f0ab80b0b50 2024-09-13T17:24:07.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.281+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ab8154d50 con 0x7f0ab80b0b50 2024-09-13T17:24:07.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.282+0000 7f0aae000640 1 -- 172.21.15.28:0/3491013893 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+12867 (secure 0 0 0) 0x7f0aa8056cc0 con 0x7f0ab80b0b50 2024-09-13T17:24:07.284 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:07.284 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":229,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:06.462405+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":217,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":220,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 msgr2=0x7f0a9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 0x7f0a9406d0f0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80adce0 tx=0x7f0aa0009190 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 msgr2=0x7f0ab81f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800c4d0 tx=0x7f0aa800c990 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 shutdown_connections 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a9406ac30 0x7f0a9406d0f0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ab8159820 0x7f0ab81f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.284+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0ab8159440 0x7f0ab81583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.285+0000 7f0ab7200640 1 --2- 172.21.15.28:0/3491013893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0ab80b0b50 0x7f0ab81f2b30 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.285+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 >> 172.21.15.28:0/3491013893 conn(0x7f0ab8072e40 msgr2=0x7f0ab806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.285+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 shutdown_connections 2024-09-13T17:24:07.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.285+0000 7f0ab7200640 1 -- 172.21.15.28:0/3491013893 wait complete. 2024-09-13T17:24:07.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:24:07.303 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:24:07.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.461+0000 7f0a80400640 1 Processor -- start 2024-09-13T17:24:07.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.461+0000 7f0a80400640 1 -- start start 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a80400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0a7815a780 0x7f0a78070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a80400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806c660 con 0x7f0a7815ab60 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a80400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7806c8b0 con 0x7f0a7806c250 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a80400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7806cb90 con 0x7f0a7815a780 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a7d000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0a7815ab60 0x7f0a7806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42276/0 (socket says 172.21.15.28:42276) 2024-09-13T17:24:07.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.462+0000 7f0a7d000640 1 -- 172.21.15.28:0/515997146 learned_addr learned my addr 172.21.15.28:0/515997146 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a7da00640 1 --2- 172.21.15.28:0/515997146 >> v2:172.21.15.189:3300/0 conn(0x7f0a7815a780 0x7f0a78070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605226521 0 0) 0x7f0a7806c660 con 0x7f0a7815ab60 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a60003520 con 0x7f0a7815ab60 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700860258 0 0) 0x7f0a60003520 con 0x7f0a7815ab60 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 >> v2:172.21.15.189:3300/0 conn(0x7f0a7815a780 msgr2=0x7f0a78070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 --2- 172.21.15.28:0/515997146 >> v2:172.21.15.189:3300/0 conn(0x7f0a7815a780 0x7f0a78070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 >> v1:172.21.15.189:6789/0 conn(0x7f0a7806c250 legacy=0x7f0a780cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780cc980 con 0x7f0a7815ab60 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f0a64001670 con 0x7f0a7815ab60 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 >> v1:172.21.15.28:6789/0 conn(0x7f0a7815ab60 legacy=0x7f0a7806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a60003970 0x7f0a60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 0x7f0a60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a60005180 0x7f0a6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7806cb90 con 0x7f0a600047c0 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6000c020 con 0x7f0a60005180 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.463+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6000c1e0 con 0x7f0a60003970 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 0x7f0a60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7e400640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a60005180 0x7f0a6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7da00640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a60003970 0x7f0a60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 -- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a60003970 msgr2=0x7f0a60003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a60003970 0x7f0a60003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 -- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a60005180 msgr2=0x7f0a6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a60005180 0x7f0a6000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7d000640 1 -- 172.21.15.28:0/515997146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60003520 con 0x7f0a600047c0 2024-09-13T17:24:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.464+0000 7f0a7e400640 1 -- 172.21.15.28:0/515997146 reap_dead start 2024-09-13T17:24:07.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.465+0000 7f0a7d000640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 0x7f0a60004c40 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c240 tx=0x7f0a600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea812089dd19bc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.465+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6402e070 con 0x7f0a600047c0 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.465+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a640058d0 con 0x7f0a600047c0 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.465+0000 7f0a77e00640 1 -- 172.21.15.28:0/515997146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64005d00 con 0x7f0a600047c0 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 -- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 msgr2=0x7f0a60004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 0x7f0a60004c40 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f0a6400c240 tx=0x7f0a600057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 -- 172.21.15.28:0/515997146 shutdown_connections 2024-09-13T17:24:07.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 --2- 172.21.15.28:0/515997146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a600047c0 0x7f0a60004c40 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 -- 172.21.15.28:0/515997146 >> 172.21.15.28:0/515997146 conn(0x7f0a780b0270 msgr2=0x7f0a780b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 -- 172.21.15.28:0/515997146 shutdown_connections 2024-09-13T17:24:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.466+0000 7f0a80400640 1 -- 172.21.15.28:0/515997146 wait complete. 2024-09-13T17:24:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.468+0000 7f0a80400640 1 Processor -- start 2024-09-13T17:24:07.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.468+0000 7f0a80400640 1 -- start start 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a600047c0 0x7f0a78144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a78147370 0x7f0a781477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78148380 con 0x7f0a78070cb0 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78071670 con 0x7f0a600047c0 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a80400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780a8440 con 0x7f0a78147370 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a7d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a7d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46184/0 (socket says 172.21.15.28:46184) 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a7d000640 1 -- 172.21.15.28:0/2526671473 learned_addr learned my addr 172.21.15.28:0/2526671473 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a7e400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a78147370 0x7f0a781477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.469+0000 7f0a7da00640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a600047c0 0x7f0a78144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 -- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a78147370 msgr2=0x7f0a781477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a78147370 0x7f0a781477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 -- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a600047c0 msgr2=0x7f0a78144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a600047c0 0x7f0a78144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 -- 172.21.15.28:0/2526671473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a781480a0 con 0x7f0a78070cb0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a7d000640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f0a64001930 tx=0x7f0a64001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64004e10 con 0x7f0a78070cb0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a64000ed0 con 0x7f0a78070cb0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a781454c0 con 0x7f0a78070cb0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64005280 con 0x7f0a78070cb0 2024-09-13T17:24:07.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.470+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a78145a80 con 0x7f0a78070cb0 2024-09-13T17:24:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.472+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a78145320 con 0x7f0a78070cb0 2024-09-13T17:24:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.472+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0a640053e0 con 0x7f0a78070cb0 2024-09-13T17:24:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.473+0000 7f0a75600640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 0x7f0a5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.473+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a640a8c30 con 0x7f0a78070cb0 2024-09-13T17:24:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.474+0000 7f0a7da00640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 0x7f0a5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.474+0000 7f0a7da00640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 0x7f0a5806d0f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0a78067520 tx=0x7f0a6c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.477+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0a78145320 con 0x7f0a78070cb0 2024-09-13T17:24:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.653+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a78145320 con 0x7f0a78070cb0 2024-09-13T17:24:07.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.653+0000 7f0a75600640 1 -- 172.21.15.28:0/2526671473 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v229) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a78145320 con 0x7f0a78070cb0 2024-09-13T17:24:07.656 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.655+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 msgr2=0x7f0a5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.655+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 0x7f0a5806d0f0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f0a78067520 tx=0x7f0a6c00b040 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.655+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 msgr2=0x7f0a78071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.655+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f0a64001930 tx=0x7f0a64001a00 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 shutdown_connections 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0a5806ac30 0x7f0a5806d0f0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0a78147370 0x7f0a781477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0a78070cb0 0x7f0a78071130 unknown :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 --2- 172.21.15.28:0/2526671473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a600047c0 0x7f0a78144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 >> 172.21.15.28:0/2526671473 conn(0x7f0a780b0270 msgr2=0x7f0a780ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:07.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 shutdown_connections 2024-09-13T17:24:07.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.656+0000 7f0a80400640 1 -- 172.21.15.28:0/2526671473 wait complete. 2024-09-13T17:24:07.674 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T17:24:07.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.827+0000 7f4843800640 1 Processor -- start 2024-09-13T17:24:07.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.827+0000 7f4843800640 1 -- start start 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c159770 0x7f483c06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f483c06f3f0 0x7f483c070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f483c0add00 0x7f483c159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c06f920 con 0x7f483c0add00 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f483c15b290 con 0x7f483c06f3f0 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.828+0000 7f4843800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c15b570 con 0x7f483c159770 2024-09-13T17:24:07.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f483c0add00 0x7f483c159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f483c0add00 0x7f483c159b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46196/0 (socket says 172.21.15.28:46196) 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 -- 172.21.15.28:0/396672422 learned_addr learned my addr 172.21.15.28:0/396672422 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4840e00640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c159770 0x7f483c06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f483be00640 1 --2- 172.21.15.28:0/396672422 >> v2:172.21.15.189:3300/0 conn(0x7f483c06f3f0 0x7f483c070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 -- 172.21.15.28:0/396672422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c159770 msgr2=0x7f483c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c159770 0x7f483c06fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 -- 172.21.15.28:0/396672422 >> v2:172.21.15.189:3300/0 conn(0x7f483c06f3f0 msgr2=0x7f483c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 --2- 172.21.15.28:0/396672422 >> v2:172.21.15.189:3300/0 conn(0x7f483c06f3f0 0x7f483c070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.829+0000 7f4841800640 1 -- 172.21.15.28:0/396672422 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c15b850 con 0x7f483c0add00 2024-09-13T17:24:07.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.830+0000 7f4841800640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c159b50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f483c0cf5c0 tx=0x7f483402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c3aaa9dce449ab1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f483b400640 1 -- 172.21.15.28:0/396672422 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834031600 con 0x7f483c0add00 2024-09-13T17:24:07.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f483b400640 1 -- 172.21.15.28:0/396672422 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f483402d050 con 0x7f483c0add00 2024-09-13T17:24:07.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f483b400640 1 -- 172.21.15.28:0/396672422 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4834031980 con 0x7f483c0add00 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 -- 172.21.15.28:0/396672422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 msgr2=0x7f483c159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c159b50 secure :-1 s=READY pgs=2703 cs=0 l=1 rev1=1 crypto rx=0x7f483c0cf5c0 tx=0x7f483402a610 comp rx=0 tx=0).stop 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 -- 172.21.15.28:0/396672422 shutdown_connections 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c159b50 unknown :-1 s=CLOSED pgs=2703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 --2- 172.21.15.28:0/396672422 >> v2:172.21.15.189:3300/0 conn(0x7f483c06f3f0 0x7f483c070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 --2- 172.21.15.28:0/396672422 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c159770 0x7f483c06fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 -- 172.21.15.28:0/396672422 >> 172.21.15.28:0/396672422 conn(0x7f483c0b0270 msgr2=0x7f483c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 -- 172.21.15.28:0/396672422 shutdown_connections 2024-09-13T17:24:07.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.831+0000 7f4843800640 1 -- 172.21.15.28:0/396672422 wait complete. 2024-09-13T17:24:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.833+0000 7f4843800640 1 Processor -- start 2024-09-13T17:24:07.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.833+0000 7f4843800640 1 -- start start 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c06f3f0 0x7f483c14f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c159770 0x7f483c1e5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c156380 con 0x7f483c0add00 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c155700 con 0x7f483c159770 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4843800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1559e0 con 0x7f483c06f3f0 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f483be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f483be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46206/0 (socket says 172.21.15.28:46206) 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f483be00640 1 -- 172.21.15.28:0/3645487194 learned_addr learned my addr 172.21.15.28:0/3645487194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4841800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c159770 0x7f483c1e5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.834+0000 7f4840e00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c06f3f0 0x7f483c14f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 -- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c06f3f0 msgr2=0x7f483c14f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c06f3f0 0x7f483c14f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 -- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c159770 msgr2=0x7f483c1e5720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:07.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c159770 0x7f483c1e5720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:07.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 -- 172.21.15.28:0/3645487194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4834009b30 con 0x7f483c0add00 2024-09-13T17:24:07.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f483be00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f482800e980 tx=0x7f482800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828009670 con 0x7f483c0add00 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f482800ca00 con 0x7f483c0add00 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482800cd90 con 0x7f483c0add00 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c155cc0 con 0x7f483c0add00 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.835+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c155320 con 0x7f483c0add00 2024-09-13T17:24:07.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.837+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f483c150a80 con 0x7f483c0add00 2024-09-13T17:24:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.837+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4828013470 con 0x7f483c0add00 2024-09-13T17:24:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.838+0000 7f4838c00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 0x7f481c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.838+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4828089180 con 0x7f483c0add00 2024-09-13T17:24:07.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.839+0000 7f4840e00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 0x7f481c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:07.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.839+0000 7f4840e00640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 0x7f481c06d0c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f483c1f5ac0 tx=0x7f4830009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:07.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:07.842+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f483c150a80 con 0x7f483c0add00 2024-09-13T17:24:08.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.026+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f483c150a80 con 0x7f483c0add00 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.027+0000 7f4838c00640 1 -- 172.21.15.28:0/3645487194 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v229) v1 ==== 84+0+65 (secure 0 0 0) 0x7f483c150a80 con 0x7f483c0add00 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:08.029 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-09-13T17:24:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 msgr2=0x7f481c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 0x7f481c06d0c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f483c1f5ac0 tx=0x7f4830009190 comp rx=0 tx=0).stop 2024-09-13T17:24:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 msgr2=0x7f483c1e51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 secure :-1 s=READY pgs=2704 cs=0 l=1 rev1=1 crypto rx=0x7f482800e980 tx=0x7f482800ee40 comp rx=0 tx=0).stop 2024-09-13T17:24:08.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 shutdown_connections 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f481c06ac00 0x7f481c06d0c0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f483c159770 0x7f483c1e5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f483c0add00 0x7f483c1e51e0 unknown :-1 s=CLOSED pgs=2704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 --2- 172.21.15.28:0/3645487194 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f483c06f3f0 0x7f483c14f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 >> 172.21.15.28:0/3645487194 conn(0x7f483c0b0270 msgr2=0x7f483c070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 shutdown_connections 2024-09-13T17:24:08.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.029+0000 7f4843800640 1 -- 172.21.15.28:0/3645487194 wait complete. 2024-09-13T17:24:08.047 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T17:24:08.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T17:24:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.205+0000 7f4f72200640 1 Processor -- start 2024-09-13T17:24:08.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.205+0000 7f4f72200640 1 -- start start 2024-09-13T17:24:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.206+0000 7f4f72200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f6c06b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.206+0000 7f4f72200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c15d950 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.206+0000 7f4f72200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c15dc30 con 0x7f4f6c15cdd0 2024-09-13T17:24:08.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.206+0000 7f4f72200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0b1b80 con 0x7f4f6c157000 2024-09-13T17:24:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f62a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4f6c06b6b0 0x7f4f6c0b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42290/0 (socket says 172.21.15.28:42290) 2024-09-13T17:24:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f62a00640 1 -- 172.21.15.28:0/4063293167 learned_addr learned my addr 172.21.15.28:0/4063293167 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:08.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6b400640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f6c06b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399142437 0 0) 0x7f4f6c15d950 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f50003520 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471225248 0 0) 0x7f4f6c15dc30 con 0x7f4f6c15cdd0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6c15d950 con 0x7f4f6c15cdd0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269774720 0 0) 0x7f4f50003520 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 msgr2=0x7f4f6c06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f6c06b170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 >> v1:172.21.15.189:6789/0 conn(0x7f4f6c15cdd0 legacy=0x7f4f6c0ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c0b1e60 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4f54001670 con 0x7f4f6c06b6b0 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 >> v1:172.21.15.28:6789/0 conn(0x7f4f6c06b6b0 legacy=0x7f4f6c0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f50003b30 0x7f4f50003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f500047c0 0x7f4f50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c0b1b80 con 0x7f4f500047c0 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000ba30 con 0x7f4f50003b30 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.207+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5000bbc0 con 0x7f4f6c157000 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f500047c0 0x7f4f50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f6b400640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f50003b30 0x7f4f50003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f6be00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 -- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 msgr2=0x7f4f50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157000 0x7f4f50005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 -- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f50003b30 msgr2=0x7f4f50003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f50003b30 0x7f4f50003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.208+0000 7f4f62a00640 1 -- 172.21.15.28:0/4063293167 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f50003520 con 0x7f4f500047c0 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.209+0000 7f4f62a00640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f500047c0 0x7f4f50004c40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c070c20 tx=0x7f4f500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb48992ec963287a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.209+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54005ad0 con 0x7f4f500047c0 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.209+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5402d070 con 0x7f4f500047c0 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.209+0000 7f4f6aa00640 1 -- 172.21.15.28:0/4063293167 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f54005e90 con 0x7f4f500047c0 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 -- 172.21.15.28:0/4063293167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f500047c0 msgr2=0x7f4f50004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 --2- 172.21.15.28:0/4063293167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f500047c0 0x7f4f50004c40 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c070c20 tx=0x7f4f500057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f6be00640 1 -- 172.21.15.28:0/4063293167 reap_dead start 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 -- 172.21.15.28:0/4063293167 shutdown_connections 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 -- 172.21.15.28:0/4063293167 >> 172.21.15.28:0/4063293167 conn(0x7f4f6c069e80 msgr2=0x7f4f6c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 -- 172.21.15.28:0/4063293167 shutdown_connections 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.210+0000 7f4f72200640 1 -- 172.21.15.28:0/4063293167 wait complete. 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.212+0000 7f4f72200640 1 Processor -- start 2024-09-13T17:24:08.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.212+0000 7f4f72200640 1 -- start start 2024-09-13T17:24:08.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 0x7f4f6c1594b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157df0 0x7f4f6c1f2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c1f2f20 con 0x7f4f6c1f2560 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c158e30 con 0x7f4f500047c0 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f72200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c158590 con 0x7f4f6c157df0 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46220/0 (socket says 172.21.15.28:46220) 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.213+0000 7f4f6be00640 1 -- 172.21.15.28:0/670559092 learned_addr learned my addr 172.21.15.28:0/670559092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6b400640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 0x7f4f6c1594b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 -- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157df0 msgr2=0x7f4f6c1f2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157df0 0x7f4f6c1f2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 -- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 msgr2=0x7f4f6c1594b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 0x7f4f6c1594b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:08.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 -- 172.21.15.28:0/670559092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c158870 con 0x7f4f6c1f2560 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6b400640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 0x7f4f6c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f6be00640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c15d1b0 tx=0x7f4f6400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64003970 con 0x7f4f6c1f2560 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f64003ad0 con 0x7f4f6c1f2560 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64003e80 con 0x7f4f6c1f2560 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c158b50 con 0x7f4f6c1f2560 2024-09-13T17:24:08.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.214+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c158270 con 0x7f4f6c1f2560 2024-09-13T17:24:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.215+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c0c3850 con 0x7f4f6c1f2560 2024-09-13T17:24:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.216+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4f640050d0 con 0x7f4f6c1f2560 2024-09-13T17:24:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.217+0000 7f4f63e00640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 0x7f4f4806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.217+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4f64088e80 con 0x7f4f6c1f2560 2024-09-13T17:24:08.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.217+0000 7f4f6b400640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 0x7f4f4806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:08.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.218+0000 7f4f6b400640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 0x7f4f4806d0c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0099e0 tx=0x7f4f5c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:08.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.221+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4f6c0c3850 con 0x7f4f6c1f2560 2024-09-13T17:24:08.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:08.398+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4f6c0c3850 con 0x7f4f6c1f2560 2024-09-13T17:24:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.137+0000 7f4f63e00640 1 -- 172.21.15.28:0/670559092 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v230) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4f6c0c3850 con 0x7f4f6c1f2560 2024-09-13T17:24:09.139 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-09-13T17:24:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 msgr2=0x7f4f4806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 0x7f4f4806d0c0 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0099e0 tx=0x7f4f5c008040 comp rx=0 tx=0).stop 2024-09-13T17:24:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 msgr2=0x7f4f6c1f29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c15d1b0 tx=0x7f4f6400eb20 comp rx=0 tx=0).stop 2024-09-13T17:24:09.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 shutdown_connections 2024-09-13T17:24:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4f4806ac00 0x7f4f4806d0c0 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4f6c1f2560 0x7f4f6c1f29e0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4f6c157df0 0x7f4f6c1f2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 --2- 172.21.15.28:0/670559092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f500047c0 0x7f4f6c1594b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.142+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 >> 172.21.15.28:0/670559092 conn(0x7f4f6c069e80 msgr2=0x7f4f6c06cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:09.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.143+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 shutdown_connections 2024-09-13T17:24:09.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.144+0000 7f4f72200640 1 -- 172.21.15.28:0/670559092 wait complete. 2024-09-13T17:24:09.166 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T17:24:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.332+0000 7faa44400640 1 Processor -- start 2024-09-13T17:24:09.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.332+0000 7faa44400640 1 -- start start 2024-09-13T17:24:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa44400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15ab60 0x7faa3c06bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa44400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3c071490 0x7faa3c06c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa44400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3c06c8b0 con 0x7faa3c15a780 2024-09-13T17:24:09.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa44400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3c06cb90 con 0x7faa3c071490 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa44400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c0cc980 con 0x7faa3c15ab60 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa42400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3c071490 0x7faa3c06c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa42400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7faa3c071490 0x7faa3c06c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46224/0 (socket says 172.21.15.28:46224) 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.333+0000 7faa42400640 1 -- 172.21.15.28:0/3473621365 learned_addr learned my addr 172.21.15.28:0/3473621365 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.334+0000 7faa41000640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15ab60 0x7faa3c06bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:09.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.334+0000 7faa42400640 1 -- 172.21.15.28:0/3473621365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15ab60 msgr2=0x7faa3c06bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.334+0000 7faa42400640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15ab60 0x7faa3c06bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.334+0000 7faa42400640 1 -- 172.21.15.28:0/3473621365 >> v1:172.21.15.189:6789/0 conn(0x7faa3c15a780 legacy=0x7faa3c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.334+0000 7faa42400640 1 -- 172.21.15.28:0/3473621365 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c0ccc60 con 0x7faa3c071490 2024-09-13T17:24:09.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.335+0000 7faa42400640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c06c080 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7faa34009e50 tx=0x7faa3402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb095d41bb82298 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.335+0000 7faa3be00640 1 -- 172.21.15.28:0/3473621365 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34031600 con 0x7faa3c071490 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.335+0000 7faa3be00640 1 -- 172.21.15.28:0/3473621365 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa3402d050 con 0x7faa3c071490 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.335+0000 7faa3be00640 1 -- 172.21.15.28:0/3473621365 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34031980 con 0x7faa3c071490 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 -- 172.21.15.28:0/3473621365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 msgr2=0x7faa3c06c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c06c080 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7faa34009e50 tx=0x7faa3402a710 comp rx=0 tx=0).stop 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 -- 172.21.15.28:0/3473621365 shutdown_connections 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c06c080 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 --2- 172.21.15.28:0/3473621365 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15ab60 0x7faa3c06bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 -- 172.21.15.28:0/3473621365 >> 172.21.15.28:0/3473621365 conn(0x7faa3c0b0270 msgr2=0x7faa3c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 -- 172.21.15.28:0/3473621365 shutdown_connections 2024-09-13T17:24:09.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.336+0000 7faa44400640 1 -- 172.21.15.28:0/3473621365 wait complete. 2024-09-13T17:24:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.337+0000 7faa44400640 1 Processor -- start 2024-09-13T17:24:09.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.338+0000 7faa44400640 1 -- start start 2024-09-13T17:24:09.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.338+0000 7faa44400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.338+0000 7faa44400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15a780 0x7faa3c1509d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.338+0000 7faa44400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 0x7faa3c142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa44400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c1433e0 con 0x7faa3c071490 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa44400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c158960 con 0x7faa3c15ab60 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa44400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa3c14fef0 con 0x7faa3c15a780 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46238/0 (socket says 172.21.15.28:46238) 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 -- 172.21.15.28:0/3520871463 learned_addr learned my addr 172.21.15.28:0/3520871463 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa42400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 0x7faa3c142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 -- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15a780 msgr2=0x7faa3c1509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15a780 0x7faa3c1509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 -- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 msgr2=0x7faa3c142ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:09.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 0x7faa3c142ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 -- 172.21.15.28:0/3520871463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa34009b30 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa42400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 0x7faa3c142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.339+0000 7faa41a00640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7faa3000c450 tx=0x7faa3000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.340+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa300052d0 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.340+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30005430 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.340+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30012870 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.340+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa3c1501d0 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.340+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa3c14d780 con 0x7faa3c071490 2024-09-13T17:24:09.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.341+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa3c1506f0 con 0x7faa3c071490 2024-09-13T17:24:09.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.342+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7faa300129d0 con 0x7faa3c071490 2024-09-13T17:24:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.344+0000 7faa39600640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 0x7faa1c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.344+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 5227+0+0 (secure 0 0 0) 0x7faa300896c0 con 0x7faa3c071490 2024-09-13T17:24:09.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.345+0000 7faa41000640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 0x7faa1c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:09.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.345+0000 7faa41000640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 0x7faa1c06d0f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7faa28006a90 tx=0x7faa28006a20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:09.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.348+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7faa30054af0 con 0x7faa3c071490 2024-09-13T17:24:09.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:09.528+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faa3c070d80 con 0x7faa3c071490 2024-09-13T17:24:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.146+0000 7faa39600640 1 -- 172.21.15.28:0/3520871463 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v231) v1 ==== 170+0+0 (secure 0 0 0) 0x7faa30019300 con 0x7faa3c071490 2024-09-13T17:24:10.148 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 msgr2=0x7faa1c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 0x7faa1c06d0f0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7faa28006a90 tx=0x7faa28006a20 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 msgr2=0x7faa3c158310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7faa3000c450 tx=0x7faa3000c910 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 shutdown_connections 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faa1c06ac30 0x7faa1c06d0f0 secure :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7faa28006a90 tx=0x7faa28006a20 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa3c15ab60 0x7faa3c142ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7faa3c15a780 0x7faa3c1509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 --2- 172.21.15.28:0/3520871463 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa3c071490 0x7faa3c158310 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.152+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 >> 172.21.15.28:0/3520871463 conn(0x7faa3c0b0270 msgr2=0x7faa3c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.153+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 shutdown_connections 2024-09-13T17:24:10.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.153+0000 7faa44400640 1 -- 172.21.15.28:0/3520871463 wait complete. 2024-09-13T17:24:10.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T17:24:10.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.342+0000 7fb5fe800640 1 Processor -- start 2024-09-13T17:24:10.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.343+0000 7fb5fe800640 1 -- start start 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fe800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5f80cc530 0x7fb5f8072060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fe800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb5f80cc910 0x7fb5f815d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fe800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5f80c78e0 con 0x7fb5f80b06f0 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fe800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f8070c90 con 0x7fb5f80cc530 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fe800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5f80cbe40 con 0x7fb5f80cc910 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5f80cc530 0x7fb5f8072060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fd400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb5f80cc530 0x7fb5f8072060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46240/0 (socket says 172.21.15.28:46240) 2024-09-13T17:24:10.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fd400640 1 -- 172.21.15.28:0/3545309596 learned_addr learned my addr 172.21.15.28:0/3545309596 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.344+0000 7fb5fca00640 1 --2- 172.21.15.28:0/3545309596 >> v2:172.21.15.189:3300/0 conn(0x7fb5f80cc910 0x7fb5f815d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.345+0000 7fb5fd400640 1 -- 172.21.15.28:0/3545309596 >> v2:172.21.15.189:3300/0 conn(0x7fb5f80cc910 msgr2=0x7fb5f815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.345+0000 7fb5fd400640 1 --2- 172.21.15.28:0/3545309596 >> v2:172.21.15.189:3300/0 conn(0x7fb5f80cc910 0x7fb5f815d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.345+0000 7fb5fd400640 1 -- 172.21.15.28:0/3545309596 >> v1:172.21.15.189:6789/0 conn(0x7fb5f80b06f0 legacy=0x7fb5f80b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.345+0000 7fb5fd400640 1 -- 172.21.15.28:0/3545309596 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5f8071000 con 0x7fb5f80cc530 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5fd400640 1 --2- 172.21.15.28:0/3545309596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8072060 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80ce370 tx=0x7fb5ec02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e987af4b6ff01001 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5f7e00640 1 -- 172.21.15.28:0/3545309596 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec032440 con 0x7fb5f80cc530 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5f7e00640 1 -- 172.21.15.28:0/3545309596 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ec0325a0 con 0x7fb5f80cc530 2024-09-13T17:24:10.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5f7e00640 1 -- 172.21.15.28:0/3545309596 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec032930 con 0x7fb5f80cc530 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5fe800640 1 -- 172.21.15.28:0/3545309596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 msgr2=0x7fb5f8072060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.346+0000 7fb5fe800640 1 --2- 172.21.15.28:0/3545309596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8072060 secure :-1 s=READY pgs=2710 cs=0 l=1 rev1=1 crypto rx=0x7fb5f80ce370 tx=0x7fb5ec02a2c0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 -- 172.21.15.28:0/3545309596 shutdown_connections 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 --2- 172.21.15.28:0/3545309596 >> v2:172.21.15.189:3300/0 conn(0x7fb5f80cc910 0x7fb5f815d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 --2- 172.21.15.28:0/3545309596 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8072060 unknown :-1 s=CLOSED pgs=2710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 -- 172.21.15.28:0/3545309596 >> 172.21.15.28:0/3545309596 conn(0x7fb5f8069e80 msgr2=0x7fb5f806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 -- 172.21.15.28:0/3545309596 shutdown_connections 2024-09-13T17:24:10.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.347+0000 7fb5fe800640 1 -- 172.21.15.28:0/3545309596 wait complete. 2024-09-13T17:24:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.348+0000 7fb5fe800640 1 Processor -- start 2024-09-13T17:24:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.349+0000 7fb5fe800640 1 -- start start 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.349+0000 7fb5fe800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 0x7fb5f8145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8146520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5f80cc910 0x7fb5f8160700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fe800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f8146a60 con 0x7fb5f80cc530 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80c7e20 con 0x7fb5f80cc910 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fe800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5f80c8100 con 0x7fb5f80b06f0 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8146520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fd400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 0x7fb5f8145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fd400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 0x7fb5f8145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:33240/0 (socket says 172.21.15.28:33240) 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fd400640 1 -- 172.21.15.28:0/499597587 learned_addr learned my addr 172.21.15.28:0/499597587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fde00640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5f80cc910 0x7fb5f8160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 -- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 msgr2=0x7fb5f8145e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 0x7fb5f8145e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 -- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5f80cc910 msgr2=0x7fb5f8160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5f80cc910 0x7fb5f8160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.350+0000 7fb5fca00640 1 -- 172.21.15.28:0/499597587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec0094d0 con 0x7fb5f80cc530 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5fca00640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8146520 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400d5f0 tx=0x7fb5e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40125f0 con 0x7fb5f80cc530 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5e4012750 con 0x7fb5f80cc530 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4012b00 con 0x7fb5f80cc530 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5f80c83e0 con 0x7fb5f80cc530 2024-09-13T17:24:10.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.351+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5f80c89a0 con 0x7fb5f80cc530 2024-09-13T17:24:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.352+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5f8072060 con 0x7fb5f80cc530 2024-09-13T17:24:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.354+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb5e4012c60 con 0x7fb5f80cc530 2024-09-13T17:24:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.355+0000 7fb5f5600640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 0x7fb5d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.355+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb5e408a1a0 con 0x7fb5f80cc530 2024-09-13T17:24:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.356+0000 7fb5fd400640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 0x7fb5d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.356+0000 7fb5fd400640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 0x7fb5d806d0f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0090b0 tx=0x7fb5ec02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.359+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb5e4055470 con 0x7fb5f80cc530 2024-09-13T17:24:10.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.575+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb5f806c170 con 0x7fb5f80cc530 2024-09-13T17:24:10.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.583+0000 7fb5f5600640 1 -- 172.21.15.28:0/499597587 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 32 and data pool 33 v185) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb5e4058a90 con 0x7fb5f80cc530 2024-09-13T17:24:10.588 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 32 and data pool 33 2024-09-13T17:24:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.592+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 msgr2=0x7fb5d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.592+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 0x7fb5d806d0f0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0090b0 tx=0x7fb5ec02e000 comp rx=0 tx=0).stop 2024-09-13T17:24:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.592+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 msgr2=0x7fb5f8146520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.592+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8146520 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7fb5e400d5f0 tx=0x7fb5e400dab0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 shutdown_connections 2024-09-13T17:24:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb5d806ac30 0x7fb5d806d0f0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5f80cc910 0x7fb5f8160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb5f80cc530 0x7fb5f8146520 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 --2- 172.21.15.28:0/499597587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb5f80b06f0 0x7fb5f8145e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.594+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 >> 172.21.15.28:0/499597587 conn(0x7fb5f8069e80 msgr2=0x7fb5f80b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.595+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 shutdown_connections 2024-09-13T17:24:10.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.595+0000 7fb5fe800640 1 -- 172.21.15.28:0/499597587 wait complete. 2024-09-13T17:24:10.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.779+0000 7f093a600640 1 Processor -- start 2024-09-13T17:24:10.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.779+0000 7f093a600640 1 -- start start 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f093a600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f093415a540 0x7f093406c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f093a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0934070cc0 0x7f0934071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f093a600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f093415e100 con 0x7f09340cabc0 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f093a600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09340adb00 con 0x7f093415a540 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f093a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0934071670 con 0x7f0934070cc0 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f0933400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f093415a540 0x7f093406c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.780+0000 7f0933400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f093415a540 0x7f093406c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46252/0 (socket says 172.21.15.28:46252) 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933400640 1 -- 172.21.15.28:0/2776839072 learned_addr learned my addr 172.21.15.28:0/2776839072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933e00640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0934070cc0 0x7f0934071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933400640 1 -- 172.21.15.28:0/2776839072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0934070cc0 msgr2=0x7f0934071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:10.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933400640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0934070cc0 0x7f0934071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933400640 1 -- 172.21.15.28:0/2776839072 >> v1:172.21.15.189:6789/0 conn(0x7f09340cabc0 legacy=0x7f093415e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.781+0000 7f0933400640 1 -- 172.21.15.28:0/2776839072 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093415e3e0 con 0x7f093415a540 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.782+0000 7f0933400640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093415a540 0x7f093406c430 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f0924009870 tx=0x7f092402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dbdae49e907929 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.782+0000 7f0932a00640 1 -- 172.21.15.28:0/2776839072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924032440 con 0x7f093415a540 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.782+0000 7f0932a00640 1 -- 172.21.15.28:0/2776839072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09240325a0 con 0x7f093415a540 2024-09-13T17:24:10.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.782+0000 7f0932a00640 1 -- 172.21.15.28:0/2776839072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924032930 con 0x7f093415a540 2024-09-13T17:24:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 -- 172.21.15.28:0/2776839072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093415a540 msgr2=0x7f093406c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093415a540 0x7f093406c430 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f0924009870 tx=0x7f092402a1c0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 -- 172.21.15.28:0/2776839072 shutdown_connections 2024-09-13T17:24:10.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0934070cc0 0x7f0934071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 --2- 172.21.15.28:0/2776839072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f093415a540 0x7f093406c430 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 -- 172.21.15.28:0/2776839072 >> 172.21.15.28:0/2776839072 conn(0x7f0934069e80 msgr2=0x7f093406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 -- 172.21.15.28:0/2776839072 shutdown_connections 2024-09-13T17:24:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.783+0000 7f093a600640 1 -- 172.21.15.28:0/2776839072 wait complete. 2024-09-13T17:24:10.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.785+0000 7f093a600640 1 Processor -- start 2024-09-13T17:24:10.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.785+0000 7f093a600640 1 -- start start 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0934070cc0 0x7f09340b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093415a540 0x7f093413bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093413b770 con 0x7f09340cabc0 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09340aff80 con 0x7f0934070cc0 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f093a600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340af370 con 0x7f093415a540 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46266/0 (socket says 172.21.15.28:46266) 2024-09-13T17:24:10.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 -- 172.21.15.28:0/271149554 learned_addr learned my addr 172.21.15.28:0/271149554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f0933400640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0934070cc0 0x7f09340b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f0933e00640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093415a540 0x7f093413bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 -- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093415a540 msgr2=0x7f093413bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093415a540 0x7f093413bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 -- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0934070cc0 msgr2=0x7f09340b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0934070cc0 0x7f09340b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.786+0000 7f092aa00640 1 -- 172.21.15.28:0/271149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09240094d0 con 0x7f09340cabc0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f092aa00640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f091c00c450 tx=0x7f091c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c0052d0 con 0x7f09340cabc0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091c005430 con 0x7f09340cabc0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091c012870 con 0x7f09340cabc0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09340af650 con 0x7f09340cabc0 2024-09-13T17:24:10.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.787+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09340afc10 con 0x7f09340cabc0 2024-09-13T17:24:10.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.788+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09340b0110 con 0x7f09340cabc0 2024-09-13T17:24:10.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.789+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f091c0129d0 con 0x7f09340cabc0 2024-09-13T17:24:10.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.790+0000 7f092be00640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 0x7f091006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:10.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.791+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f091c088da0 con 0x7f09340cabc0 2024-09-13T17:24:10.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.791+0000 7f0933400640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 0x7f091006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:10.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.792+0000 7f0933400640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 0x7f091006d0c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0924034000 tx=0x7f092402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:10.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.795+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f091c053f90 con 0x7f09340cabc0 2024-09-13T17:24:10.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.974+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0934071130 con 0x7f09340cabc0 2024-09-13T17:24:10.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.976+0000 7f092be00640 1 -- 172.21.15.28:0/271149554 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+16536 (secure 0 0 0) 0x7f091c0575b0 con 0x7f09340cabc0 2024-09-13T17:24:10.978 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:10.978 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":232,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:10.577578+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 msgr2=0x7f091006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 0x7f091006d0c0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0924034000 tx=0x7f092402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 msgr2=0x7f093413ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 secure :-1 s=READY pgs=2713 cs=0 l=1 rev1=1 crypto rx=0x7f091c00c450 tx=0x7f091c00c910 comp rx=0 tx=0).stop 2024-09-13T17:24:10.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 shutdown_connections 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f091006ac00 0x7f091006d0c0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f093415a540 0x7f093413bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09340cabc0 0x7f093413ba10 unknown :-1 s=CLOSED pgs=2713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 --2- 172.21.15.28:0/271149554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0934070cc0 0x7f09340b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 >> 172.21.15.28:0/271149554 conn(0x7f0934069e80 msgr2=0x7f09340cd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 shutdown_connections 2024-09-13T17:24:10.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:10.978+0000 7f093a600640 1 -- 172.21.15.28:0/271149554 wait complete. 2024-09-13T17:24:10.997 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:11.126 INFO:tasks.ceph.mds.a.smithi028.stderr:2024-09-13T17:24:11.125+0000 7f564a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:24:11.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.153+0000 7f73a5200640 1 Processor -- start 2024-09-13T17:24:11.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.153+0000 7f73a5200640 1 -- start start 2024-09-13T17:24:11.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.154+0000 7f73a5200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 0x7f73a015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.154+0000 7f73a5200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a0159d90 con 0x7f73a006f970 2024-09-13T17:24:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.154+0000 7f73a5200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a00b2040 con 0x7f73a006c2d0 2024-09-13T17:24:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.154+0000 7f73a5200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a015b0b0 con 0x7f73a01592a0 2024-09-13T17:24:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.154+0000 7f739d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f73a006f970 0x7f73a0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42298/0 (socket says 172.21.15.28:42298) 2024-09-13T17:24:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739d400640 1 -- 172.21.15.28:0/636431473 learned_addr learned my addr 172.21.15.28:0/636431473 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:11.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739de00640 1 --2- 172.21.15.28:0/636431473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 0x7f73a015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552812730 0 0) 0x7f73a0159d90 con 0x7f73a006f970 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7384003520 con 0x7f73a006f970 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214663560 0 0) 0x7f73a00b2040 con 0x7f73a006c2d0 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73a0159d90 con 0x7f73a006c2d0 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739de00640 1 -- 172.21.15.28:0/636431473 >> v1:172.21.15.189:6789/0 conn(0x7f73a006c2d0 legacy=0x7f73a00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739de00640 1 -- 172.21.15.28:0/636431473 >> v1:172.21.15.28:6789/0 conn(0x7f73a006f970 legacy=0x7f73a0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.155+0000 7f739de00640 1 -- 172.21.15.28:0/636431473 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a0159ff0 con 0x7f73a01592a0 2024-09-13T17:24:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.156+0000 7f739de00640 1 --2- 172.21.15.28:0/636431473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 0x7f73a015ab70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f73900097d0 tx=0x7f739002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644d34c5805400b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7390032440 con 0x7f73a01592a0 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73900325a0 con 0x7f73a01592a0 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f739ca00640 1 -- 172.21.15.28:0/636431473 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7390032930 con 0x7f73a01592a0 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f73a5200640 1 -- 172.21.15.28:0/636431473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 msgr2=0x7f73a015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f73a5200640 1 --2- 172.21.15.28:0/636431473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 0x7f73a015ab70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f73900097d0 tx=0x7f739002a210 comp rx=0 tx=0).stop 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f73a5200640 1 -- 172.21.15.28:0/636431473 shutdown_connections 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f73a5200640 1 --2- 172.21.15.28:0/636431473 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a01592a0 0x7f73a015ab70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.157+0000 7f73a5200640 1 -- 172.21.15.28:0/636431473 >> 172.21.15.28:0/636431473 conn(0x7f73a0069e80 msgr2=0x7f73a006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.158+0000 7f73a5200640 1 -- 172.21.15.28:0/636431473 shutdown_connections 2024-09-13T17:24:11.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.158+0000 7f73a5200640 1 -- 172.21.15.28:0/636431473 wait complete. 2024-09-13T17:24:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.159+0000 7f73a5200640 1 Processor -- start 2024-09-13T17:24:11.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.159+0000 7f73a5200640 1 -- start start 2024-09-13T17:24:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a006f970 0x7f73a0150e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a01592a0 0x7f73a014acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a014fa30 con 0x7f73a006c2d0 2024-09-13T17:24:11.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a0151680 con 0x7f73a01592a0 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f73a5200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a014a1d0 con 0x7f73a006f970 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f739de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f739de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46280/0 (socket says 172.21.15.28:46280) 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.160+0000 7f739de00640 1 -- 172.21.15.28:0/3029619882 learned_addr learned my addr 172.21.15.28:0/3029619882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739d400640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a006f970 0x7f73a0150e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739e800640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a01592a0 0x7f73a014acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 -- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a006f970 msgr2=0x7f73a0150e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a006f970 0x7f73a0150e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 -- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a01592a0 msgr2=0x7f73a014acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a01592a0 0x7f73a014acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 -- 172.21.15.28:0/3029619882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390009400 con 0x7f73a006c2d0 2024-09-13T17:24:11.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.161+0000 7f739de00640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f73900097d0 tx=0x7f73900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.162+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739003d830 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.162+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a014a4b0 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.162+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739003d9b0 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.162+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739003dd60 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.162+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a0151390 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.163+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a015ab70 con 0x7f73a006c2d0 2024-09-13T17:24:11.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.164+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7390039040 con 0x7f73a006c2d0 2024-09-13T17:24:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.164+0000 7f7396000640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 0x7f738006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.164+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73900a9e70 con 0x7f73a006c2d0 2024-09-13T17:24:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.165+0000 7f739d400640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 0x7f738006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.166+0000 7f739d400640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 0x7f738006d0c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f738c000f60 tx=0x7f738c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.169+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7390074fc0 con 0x7f73a006c2d0 2024-09-13T17:24:11.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.348+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73a006a300 con 0x7f73a006c2d0 2024-09-13T17:24:11.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.349+0000 7f7396000640 1 -- 172.21.15.28:0/3029619882 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+16536 (secure 0 0 0) 0x7f73900785e0 con 0x7f73a006c2d0 2024-09-13T17:24:11.351 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:11.351 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":232,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:10.577578+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:11.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 msgr2=0x7f738006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 0x7f738006d0c0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f738c000f60 tx=0x7f738c009040 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 msgr2=0x7f73a014f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f73900097d0 tx=0x7f73900046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 shutdown_connections 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f738006ac00 0x7f738006d0c0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f73a01592a0 0x7f73a014acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f73a006f970 0x7f73a0150e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 --2- 172.21.15.28:0/3029619882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f73a006c2d0 0x7f73a014f4f0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 >> 172.21.15.28:0/3029619882 conn(0x7f73a0069e80 msgr2=0x7f73a00afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 shutdown_connections 2024-09-13T17:24:11.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.351+0000 7f73a5200640 1 -- 172.21.15.28:0/3029619882 wait complete. 2024-09-13T17:24:11.370 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T17:24:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.526+0000 7f9960800640 1 Processor -- start 2024-09-13T17:24:11.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.526+0000 7f9960800640 1 -- start start 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f9960800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 0x7f9958070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f9960800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f995815ab60 0x7f99580712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f9960800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995806c8b0 con 0x7f995806bb40 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f9960800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995806c6a0 con 0x7f995815ab60 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f9960800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995806cb90 con 0x7f995815a780 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f995d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f995815ab60 0x7f99580712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f995d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f995815ab60 0x7f99580712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46294/0 (socket says 172.21.15.28:46294) 2024-09-13T17:24:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.527+0000 7f995d400640 1 -- 172.21.15.28:0/1764476587 learned_addr learned my addr 172.21.15.28:0/1764476587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995de00640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 0x7f9958070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995d400640 1 -- 172.21.15.28:0/1764476587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 msgr2=0x7f9958070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995d400640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 0x7f9958070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995d400640 1 -- 172.21.15.28:0/1764476587 >> v1:172.21.15.189:6789/0 conn(0x7f995806bb40 legacy=0x7f995806bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995d400640 1 -- 172.21.15.28:0/1764476587 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580cc980 con 0x7f995815ab60 2024-09-13T17:24:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.528+0000 7f995de00640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 0x7f9958070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.529+0000 7f995d400640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815ab60 0x7f99580712c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f9948009870 tx=0x7f994802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e793b9f78936cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.529+0000 7f995ca00640 1 -- 172.21.15.28:0/1764476587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948032440 con 0x7f995815ab60 2024-09-13T17:24:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.529+0000 7f995ca00640 1 -- 172.21.15.28:0/1764476587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99480325a0 con 0x7f995815ab60 2024-09-13T17:24:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.529+0000 7f995ca00640 1 -- 172.21.15.28:0/1764476587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948032930 con 0x7f995815ab60 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 -- 172.21.15.28:0/1764476587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815ab60 msgr2=0x7f99580712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815ab60 0x7f99580712c0 secure :-1 s=READY pgs=2716 cs=0 l=1 rev1=1 crypto rx=0x7f9948009870 tx=0x7f994802a210 comp rx=0 tx=0).stop 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 -- 172.21.15.28:0/1764476587 shutdown_connections 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815ab60 0x7f99580712c0 unknown :-1 s=CLOSED pgs=2716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 --2- 172.21.15.28:0/1764476587 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815a780 0x7f9958070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 -- 172.21.15.28:0/1764476587 >> 172.21.15.28:0/1764476587 conn(0x7f99580b0270 msgr2=0x7f99580b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 -- 172.21.15.28:0/1764476587 shutdown_connections 2024-09-13T17:24:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.530+0000 7f9960800640 1 -- 172.21.15.28:0/1764476587 wait complete. 2024-09-13T17:24:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.531+0000 7f9960800640 1 Processor -- start 2024-09-13T17:24:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.531+0000 7f9960800640 1 -- start start 2024-09-13T17:24:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 0x7f995814d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 0x7f99580ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995814f5a0 con 0x7f995815a780 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995814f880 con 0x7f995806bb40 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.532+0000 7f9960800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995814fb60 con 0x7f995815ab60 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46310/0 (socket says 172.21.15.28:46310) 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 -- 172.21.15.28:0/3680935233 learned_addr learned my addr 172.21.15.28:0/3680935233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995de00640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 0x7f995814d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995e800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 0x7f99580ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 -- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 msgr2=0x7f99580ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 0x7f99580ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 -- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 msgr2=0x7f995814d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 0x7f995814d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 -- 172.21.15.28:0/3680935233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480094d0 con 0x7f995815a780 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995de00640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 0x7f995814d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995e800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 0x7f99580ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.533+0000 7f995d400640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f99480090b0 tx=0x7f99480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.534+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9948030b00 con 0x7f995815a780 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.534+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9948030c80 con 0x7f995815a780 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.534+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994803d420 con 0x7f995815a780 2024-09-13T17:24:11.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.534+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995814fe40 con 0x7f995815a780 2024-09-13T17:24:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.534+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99580b2650 con 0x7f995815a780 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.535+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995806bf90 con 0x7f995815a780 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.536+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9948039040 con 0x7f995815a780 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.536+0000 7f9952000640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 0x7f993c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.537+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99480aa0b0 con 0x7f995815a780 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.537+0000 7f995de00640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 0x7f993c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:11.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.538+0000 7f995de00640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 0x7f993c06d0c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f994c00aa30 tx=0x7f994c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:11.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.541+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9948075280 con 0x7f995815a780 2024-09-13T17:24:11.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:11.759+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f99580ae6d0 con 0x7f995815a780 2024-09-13T17:24:12.055 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:24:12.060 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:24:12.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.584+0000 7f9952000640 1 -- 172.21.15.28:0/3680935233 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v187) v1 ==== 118+0+0 (secure 0 0 0) 0x7f99480788a0 con 0x7f995815a780 2024-09-13T17:24:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 msgr2=0x7f993c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 0x7f993c06d0c0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f994c00aa30 tx=0x7f994c008040 comp rx=0 tx=0).stop 2024-09-13T17:24:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 msgr2=0x7f995814eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7f99480090b0 tx=0x7f99480046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 shutdown_connections 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f993c06ac00 0x7f993c06d0c0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f995815ab60 0x7f99580ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f995815a780 0x7f995814eb30 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 --2- 172.21.15.28:0/3680935233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f995806bb40 0x7f995814d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 >> 172.21.15.28:0/3680935233 conn(0x7f99580b0270 msgr2=0x7f99580cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 shutdown_connections 2024-09-13T17:24:12.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.586+0000 7f9960800640 1 -- 172.21.15.28:0/3680935233 wait complete. 2024-09-13T17:24:12.605 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T17:24:12.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T17:24:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.762+0000 7fadad000640 1 Processor -- start 2024-09-13T17:24:12.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.762+0000 7fadad000640 1 -- start start 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fadad000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fadad000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8070470 con 0x7fada80b2400 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fadad000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80ace30 con 0x7fada815c9a0 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fadad000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada80ad110 con 0x7fada80ad940 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fad9d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fada80b2400 0x7fada806bc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42312/0 (socket says 172.21.15.28:42312) 2024-09-13T17:24:12.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.763+0000 7fad9d400640 1 -- 172.21.15.28:0/246078459 learned_addr learned my addr 172.21.15.28:0/246078459 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:12.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada6800640 1 --2- 172.21.15.28:0/246078459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2868158599 0 0) 0x7fada8070470 con 0x7fada80b2400 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8c003520 con 0x7fada80b2400 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157040669 0 0) 0x7fada80ace30 con 0x7fada815c9a0 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada8070470 con 0x7fada815c9a0 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada6800640 1 -- 172.21.15.28:0/246078459 >> v1:172.21.15.189:6789/0 conn(0x7fada815c9a0 legacy=0x7fada80b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada6800640 1 -- 172.21.15.28:0/246078459 >> v1:172.21.15.28:6789/0 conn(0x7fada80b2400 legacy=0x7fada806bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.764+0000 7fada6800640 1 -- 172.21.15.28:0/246078459 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada806b170 con 0x7fada80ad940 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.765+0000 7fada6800640 1 --2- 172.21.15.28:0/246078459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada80ac850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fada0009dc0 tx=0x7fada002a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d10fd74b89d994b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0007740 con 0x7fada80ad940 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada002d050 con 0x7fada80ad940 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fada5400640 1 -- 172.21.15.28:0/246078459 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0007ac0 con 0x7fada80ad940 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 -- 172.21.15.28:0/246078459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 msgr2=0x7fada80ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 --2- 172.21.15.28:0/246078459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada80ac850 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fada0009dc0 tx=0x7fada002a760 comp rx=0 tx=0).stop 2024-09-13T17:24:12.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 -- 172.21.15.28:0/246078459 shutdown_connections 2024-09-13T17:24:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 --2- 172.21.15.28:0/246078459 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada80ac850 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 -- 172.21.15.28:0/246078459 >> 172.21.15.28:0/246078459 conn(0x7fada80b0270 msgr2=0x7fada80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 -- 172.21.15.28:0/246078459 shutdown_connections 2024-09-13T17:24:12.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.766+0000 7fadad000640 1 -- 172.21.15.28:0/246078459 wait complete. 2024-09-13T17:24:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.768+0000 7fadad000640 1 Processor -- start 2024-09-13T17:24:12.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.768+0000 7fadad000640 1 -- start start 2024-09-13T17:24:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada81492b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:12.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fada815c9a0 0x7fada81502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:12.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada815e030 con 0x7fada80b2400 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada815cf90 con 0x7fada815c9a0 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fadad000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada815d270 con 0x7fada80ad940 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fad9d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fad9d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46316/0 (socket says 172.21.15.28:46316) 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fad9d400640 1 -- 172.21.15.28:0/1068382519 learned_addr learned my addr 172.21.15.28:0/1068382519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fada5e00640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada81492b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.769+0000 7fada6800640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fada815c9a0 0x7fada81502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 -- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 msgr2=0x7fada81492b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada81492b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 -- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fada815c9a0 msgr2=0x7fada81502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fada815c9a0 0x7fada81502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 -- 172.21.15.28:0/1068382519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0009a70 con 0x7fada80b2400 2024-09-13T17:24:12.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9d400640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fad9000c500 tx=0x7fad9000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90005ce0 con 0x7fada80b2400 2024-09-13T17:24:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad90005e40 con 0x7fada80b2400 2024-09-13T17:24:12.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad900072f0 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada8070930 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.770+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada815d830 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.772+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada815cd80 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.772+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fad90007450 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.773+0000 7fad9ea00640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 0x7fad8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.773+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fad900888c0 con 0x7fada80b2400 2024-09-13T17:24:12.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.774+0000 7fada5e00640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 0x7fad8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:12.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.774+0000 7fada5e00640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 0x7fad8406d0c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fad980011e0 tx=0x7fad98009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:12.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.777+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fad900575d0 con 0x7fada80b2400 2024-09-13T17:24:12.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:12.989+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fada80ca920 con 0x7fada80b2400 2024-09-13T17:24:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.590+0000 7fad9ea00640 1 -- 172.21.15.28:0/1068382519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v188) v1 ==== 122+0+0 (secure 0 0 0) 0x7fad90017020 con 0x7fada80b2400 2024-09-13T17:24:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 msgr2=0x7fad8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 0x7fad8406d0c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fad980011e0 tx=0x7fad98009040 comp rx=0 tx=0).stop 2024-09-13T17:24:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 msgr2=0x7fada80cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 secure :-1 s=READY pgs=2719 cs=0 l=1 rev1=1 crypto rx=0x7fad9000c500 tx=0x7fad9000c9c0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 shutdown_connections 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fad8406ac00 0x7fad8406d0c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fada815c9a0 0x7fada81502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fada80b2400 0x7fada80cd710 unknown :-1 s=CLOSED pgs=2719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 --2- 172.21.15.28:0/1068382519 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fada80ad940 0x7fada81492b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 >> 172.21.15.28:0/1068382519 conn(0x7fada80b0270 msgr2=0x7fada80b1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 shutdown_connections 2024-09-13T17:24:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.592+0000 7fadad000640 1 -- 172.21.15.28:0/1068382519 wait complete. 2024-09-13T17:24:13.613 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T17:24:13.613 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.768+0000 7fca7b600640 1 Processor -- start 2024-09-13T17:24:13.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.768+0000 7fca7b600640 1 -- start start 2024-09-13T17:24:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.769+0000 7fca7b600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca74146170 0x7fca7415d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca7b600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca740acf60 0x7fca74160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca7b600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca74146550 con 0x7fca74160310 2024-09-13T17:24:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca7b600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca7415cd20 con 0x7fca74146170 2024-09-13T17:24:13.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca7b600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca7415d280 con 0x7fca740acf60 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca78c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fca74146170 0x7fca7415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca73e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fca74160310 0x7fca740ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:42318/0 (socket says 172.21.15.28:42318) 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca73e00640 1 -- 172.21.15.28:0/248476385 learned_addr learned my addr 172.21.15.28:0/248476385 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca79600640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca740acf60 0x7fca74160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca78c00640 1 -- 172.21.15.28:0/248476385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca740acf60 msgr2=0x7fca74160750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca78c00640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca740acf60 0x7fca74160750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca78c00640 1 -- 172.21.15.28:0/248476385 >> v1:172.21.15.28:6789/0 conn(0x7fca74160310 legacy=0x7fca740ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca78c00640 1 -- 172.21.15.28:0/248476385 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7415dc50 con 0x7fca74146170 2024-09-13T17:24:13.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.770+0000 7fca73400640 1 -- 172.21.15.28:0/248476385 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000290925 0 0) 0x7fca74146550 con 0x7fca74160310 2024-09-13T17:24:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.771+0000 7fca78c00640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca7415d710 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fca680097a0 tx=0x7fca6802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c62f9ae3a77dab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca73400640 1 -- 172.21.15.28:0/248476385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68032440 con 0x7fca74146170 2024-09-13T17:24:13.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca73400640 1 -- 172.21.15.28:0/248476385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca680325a0 con 0x7fca74146170 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca73400640 1 -- 172.21.15.28:0/248476385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68032930 con 0x7fca74146170 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 -- 172.21.15.28:0/248476385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 msgr2=0x7fca7415d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca7415d710 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fca680097a0 tx=0x7fca6802a180 comp rx=0 tx=0).stop 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 -- 172.21.15.28:0/248476385 shutdown_connections 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca740acf60 0x7fca74160750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 --2- 172.21.15.28:0/248476385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca7415d710 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 -- 172.21.15.28:0/248476385 >> 172.21.15.28:0/248476385 conn(0x7fca74069e80 msgr2=0x7fca7406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 -- 172.21.15.28:0/248476385 shutdown_connections 2024-09-13T17:24:13.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.772+0000 7fca7b600640 1 -- 172.21.15.28:0/248476385 wait complete. 2024-09-13T17:24:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.774+0000 7fca7b600640 1 Processor -- start 2024-09-13T17:24:13.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.774+0000 7fca7b600640 1 -- start start 2024-09-13T17:24:13.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca740acf60 0x7fca740ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca74160310 0x7fca74144580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca741560c0 con 0x7fca74146170 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca74155810 con 0x7fca740acf60 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca7b600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca74157c10 con 0x7fca74160310 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca73e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46342/0 (socket says 172.21.15.28:46342) 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.775+0000 7fca73e00640 1 -- 172.21.15.28:0/788787101 learned_addr learned my addr 172.21.15.28:0/788787101 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 -- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca74160310 msgr2=0x7fca74144580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca74160310 0x7fca74144580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 -- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca740acf60 msgr2=0x7fca740ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca740acf60 0x7fca740ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 -- 172.21.15.28:0/788787101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca68009400 con 0x7fca74146170 2024-09-13T17:24:13.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca73e00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c450 tx=0x7fca6000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:13.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca600074e0 con 0x7fca74146170 2024-09-13T17:24:13.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca60007660 con 0x7fca74146170 2024-09-13T17:24:13.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca60004a20 con 0x7fca74146170 2024-09-13T17:24:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca74157930 con 0x7fca74146170 2024-09-13T17:24:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.776+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca74155ce0 con 0x7fca74146170 2024-09-13T17:24:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.777+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca74156200 con 0x7fca74146170 2024-09-13T17:24:13.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.778+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fca60004b80 con 0x7fca74146170 2024-09-13T17:24:13.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.779+0000 7fca70c00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 0x7fca5406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:13.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.779+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca60089200 con 0x7fca74146170 2024-09-13T17:24:13.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.779+0000 7fca78c00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 0x7fca5406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:13.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.781+0000 7fca78c00640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 0x7fca5406d0c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fca68034000 tx=0x7fca6802c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:13.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.783+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fca74156200 con 0x7fca74146170 2024-09-13T17:24:14.000 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.998+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca74156200 con 0x7fca74146170 2024-09-13T17:24:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:13.999+0000 7fca70c00640 1 -- 172.21.15.28:0/788787101 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4750 (secure 0 0 0) 0x7fca74156200 con 0x7fca74146170 2024-09-13T17:24:14.001 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:14.001 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":188,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:13.584112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1114231130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1114231130},{"type":"v1","addr":"172.21.15.28:6835","nonce":1114231130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:14.001 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 188 2024-09-13T17:24:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 msgr2=0x7fca5406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.003 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 0x7fca5406d0c0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fca68034000 tx=0x7fca6802c000 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 msgr2=0x7fca740ce7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 secure :-1 s=READY pgs=2722 cs=0 l=1 rev1=1 crypto rx=0x7fca6000c450 tx=0x7fca6000c910 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 shutdown_connections 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fca5406ac00 0x7fca5406d0c0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fca74160310 0x7fca74144580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca74146170 0x7fca740ce7e0 unknown :-1 s=CLOSED pgs=2722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 --2- 172.21.15.28:0/788787101 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca740acf60 0x7fca740ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 >> 172.21.15.28:0/788787101 conn(0x7fca74069e80 msgr2=0x7fca740aeed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.001+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 shutdown_connections 2024-09-13T17:24:14.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.002+0000 7fca7b600640 1 -- 172.21.15.28:0/788787101 wait complete. 2024-09-13T17:24:14.019 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.171+0000 7f592ae00640 1 Processor -- start 2024-09-13T17:24:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.171+0000 7f592ae00640 1 -- start start 2024-09-13T17:24:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592ae00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59240b0b50 0x7f59240b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592ae00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5924159820 0x7f59240b2820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240c3850 con 0x7f5924159440 2024-09-13T17:24:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592ae00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59240b2d60 con 0x7f5924159820 2024-09-13T17:24:14.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592ae00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f592415a930 con 0x7f59240b0b50 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592a400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5924159820 0x7f59240b2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f5929a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5924159440 0x7f59240b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44398/0 (socket says 172.21.15.28:44398) 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f5929a00640 1 -- 172.21.15.28:0/4064913642 learned_addr learned my addr 172.21.15.28:0/4064913642 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f5929000640 1 --2- 172.21.15.28:0/4064913642 >> v2:172.21.15.189:3300/0 conn(0x7f59240b0b50 0x7f59240b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592a400640 1 -- 172.21.15.28:0/4064913642 >> v2:172.21.15.189:3300/0 conn(0x7f59240b0b50 msgr2=0x7f59240b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592a400640 1 --2- 172.21.15.28:0/4064913642 >> v2:172.21.15.189:3300/0 conn(0x7f59240b0b50 0x7f59240b22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592a400640 1 -- 172.21.15.28:0/4064913642 >> v1:172.21.15.28:6789/0 conn(0x7f5924159440 legacy=0x7f59240b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f592a400640 1 -- 172.21.15.28:0/4064913642 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924159d20 con 0x7f5924159820 2024-09-13T17:24:14.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.173+0000 7f5923e00640 1 -- 172.21.15.28:0/4064913642 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007923443 0 0) 0x7f59240c3850 con 0x7f5924159440 2024-09-13T17:24:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.174+0000 7f592a400640 1 --2- 172.21.15.28:0/4064913642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159820 0x7f59240b2820 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f592406d620 tx=0x7f591c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef934c7ba0d644d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f5923e00640 1 -- 172.21.15.28:0/4064913642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c007740 con 0x7f5924159820 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f5923e00640 1 -- 172.21.15.28:0/4064913642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591c02d050 con 0x7f5924159820 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f5923e00640 1 -- 172.21.15.28:0/4064913642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591c007ac0 con 0x7f5924159820 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 -- 172.21.15.28:0/4064913642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159820 msgr2=0x7f59240b2820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 --2- 172.21.15.28:0/4064913642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159820 0x7f59240b2820 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f592406d620 tx=0x7f591c02a710 comp rx=0 tx=0).stop 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 -- 172.21.15.28:0/4064913642 shutdown_connections 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 --2- 172.21.15.28:0/4064913642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159820 0x7f59240b2820 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 --2- 172.21.15.28:0/4064913642 >> v2:172.21.15.189:3300/0 conn(0x7f59240b0b50 0x7f59240b22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 -- 172.21.15.28:0/4064913642 >> 172.21.15.28:0/4064913642 conn(0x7f5924072e40 msgr2=0x7f5924073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 -- 172.21.15.28:0/4064913642 shutdown_connections 2024-09-13T17:24:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.175+0000 7f592ae00640 1 -- 172.21.15.28:0/4064913642 wait complete. 2024-09-13T17:24:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.177+0000 7f592ae00640 1 Processor -- start 2024-09-13T17:24:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.177+0000 7f592ae00640 1 -- start start 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59240b0b50 0x7f59240cc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5924159820 0x7f59240ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924149d90 con 0x7f5924159440 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924146eb0 con 0x7f59240b0b50 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f592ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5924147190 con 0x7f5924159820 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f5929000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f5929000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33196/0 (socket says 172.21.15.28:33196) 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f5929000640 1 -- 172.21.15.28:0/2343188308 learned_addr learned my addr 172.21.15.28:0/2343188308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.178+0000 7f5929a00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59240b0b50 0x7f59240cc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f592a400640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5924159820 0x7f59240ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 -- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5924159820 msgr2=0x7f59240ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5924159820 0x7f59240ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 -- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59240b0b50 msgr2=0x7f59240cc6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59240b0b50 0x7f59240cc6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 -- 172.21.15.28:0/2343188308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c009a70 con 0x7f5924159440 2024-09-13T17:24:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5929000640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f591000d800 tx=0x7f591000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591000b6c0 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591000b820 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591000bbb0 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.179+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5924147470 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.180+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59240c37e0 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.181+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f592415ada0 con 0x7f5924159440 2024-09-13T17:24:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.181+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f591000bd10 con 0x7f5924159440 2024-09-13T17:24:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.182+0000 7f5921600640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 0x7f590406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.182+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5910089820 con 0x7f5924159440 2024-09-13T17:24:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.183+0000 7f5929a00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 0x7f590406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.184+0000 7f5929a00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 0x7f590406d0c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5918000f60 tx=0x7f5918007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.186+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f592415ada0 con 0x7f5924159440 2024-09-13T17:24:14.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.366+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f592415ada0 con 0x7f5924159440 2024-09-13T17:24:14.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.367+0000 7f5921600640 1 -- 172.21.15.28:0/2343188308 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+16523 (secure 0 0 0) 0x7f592415ada0 con 0x7f5924159440 2024-09-13T17:24:14.369 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:14.370 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":233,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:11.579976+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 msgr2=0x7f590406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 0x7f590406d0c0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5918000f60 tx=0x7f5918007d50 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 msgr2=0x7f59240ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f591000d800 tx=0x7f591000dcc0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 shutdown_connections 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f590406ac00 0x7f590406d0c0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5924159820 0x7f59240ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5924159440 0x7f59240ac8c0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.369+0000 7f592ae00640 1 --2- 172.21.15.28:0/2343188308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59240b0b50 0x7f59240cc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.370+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 >> 172.21.15.28:0/2343188308 conn(0x7f5924072e40 msgr2=0x7f592415d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.370+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 shutdown_connections 2024-09-13T17:24:14.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.370+0000 7f592ae00640 1 -- 172.21.15.28:0/2343188308 wait complete. 2024-09-13T17:24:14.387 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T17:24:14.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:24:14.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.540+0000 7fe0b9400640 1 Processor -- start 2024-09-13T17:24:14.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.541+0000 7fe0b9400640 1 -- start start 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b9400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe0b406b360 0x7fe0b40b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b40b0980 0x7fe0b40af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b9400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b40b0dd0 con 0x7fe0b4154a90 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b9400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0b406b900 con 0x7fe0b406b360 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b406bbe0 con 0x7fe0b40b0980 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe0b4154a90 0x7fe0b415b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44412/0 (socket says 172.21.15.28:44412) 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b3e00640 1 -- 172.21.15.28:0/911355479 learned_addr learned my addr 172.21.15.28:0/911355479 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:14.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0ab400640 1 --2- 172.21.15.28:0/911355479 >> v2:172.21.15.28:3300/0 conn(0x7fe0b406b360 0x7fe0b40b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.542+0000 7fe0b8a00640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b40b0980 0x7fe0b40af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.543+0000 7fe0ab400640 1 -- 172.21.15.28:0/911355479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b40b0980 msgr2=0x7fe0b40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.543+0000 7fe0ab400640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b40b0980 0x7fe0b40af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.543+0000 7fe0ab400640 1 -- 172.21.15.28:0/911355479 >> v1:172.21.15.28:6789/0 conn(0x7fe0b4154a90 legacy=0x7fe0b415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.543+0000 7fe0ab400640 1 -- 172.21.15.28:0/911355479 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b406bec0 con 0x7fe0b406b360 2024-09-13T17:24:14.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.543+0000 7fe0b3400640 1 -- 172.21.15.28:0/911355479 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027970085 0 0) 0x7fe0a4001110 con 0x7fe0b4154a90 2024-09-13T17:24:14.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.544+0000 7fe0ab400640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b406b360 0x7fe0b40b0440 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0097a0 tx=0x7fe09c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5778b9b60fef30ee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.544+0000 7fe0b3400640 1 -- 172.21.15.28:0/911355479 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c032440 con 0x7fe0b406b360 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.544+0000 7fe0b3400640 1 -- 172.21.15.28:0/911355479 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c0325a0 con 0x7fe0b406b360 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.544+0000 7fe0b3400640 1 -- 172.21.15.28:0/911355479 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c032930 con 0x7fe0b406b360 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 -- 172.21.15.28:0/911355479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b406b360 msgr2=0x7fe0b40b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b406b360 0x7fe0b40b0440 secure :-1 s=READY pgs=2727 cs=0 l=1 rev1=1 crypto rx=0x7fe09c0097a0 tx=0x7fe09c02a180 comp rx=0 tx=0).stop 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 -- 172.21.15.28:0/911355479 shutdown_connections 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b40b0980 0x7fe0b40af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 --2- 172.21.15.28:0/911355479 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b406b360 0x7fe0b40b0440 unknown :-1 s=CLOSED pgs=2727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 -- 172.21.15.28:0/911355479 >> 172.21.15.28:0/911355479 conn(0x7fe0b4072e40 msgr2=0x7fe0b4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 -- 172.21.15.28:0/911355479 shutdown_connections 2024-09-13T17:24:14.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.545+0000 7fe0b9400640 1 -- 172.21.15.28:0/911355479 wait complete. 2024-09-13T17:24:14.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.546+0000 7fe0b9400640 1 Processor -- start 2024-09-13T17:24:14.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.546+0000 7fe0b9400640 1 -- start start 2024-09-13T17:24:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b406b360 0x7fe0b40aa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0b40b0980 0x7fe0b4153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b40aa260 con 0x7fe0b4154a90 2024-09-13T17:24:14.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b40aa9a0 con 0x7fe0b40b0980 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.547+0000 7fe0b9400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b4155600 con 0x7fe0b406b360 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33218/0 (socket says 172.21.15.28:33218) 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 -- 172.21.15.28:0/117610342 learned_addr learned my addr 172.21.15.28:0/117610342 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b3e00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b406b360 0x7fe0b40aa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0ab400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0b40b0980 0x7fe0b4153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 -- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b406b360 msgr2=0x7fe0b40aa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b406b360 0x7fe0b40aa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 -- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0b40b0980 msgr2=0x7fe0b4153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0b40b0980 0x7fe0b4153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.548+0000 7fe0b8a00640 1 -- 172.21.15.28:0/117610342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c009400 con 0x7fe0b4154a90 2024-09-13T17:24:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b8a00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00efc0 tx=0x7fe0ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac009b10 con 0x7fe0b4154a90 2024-09-13T17:24:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0ac00f040 con 0x7fe0b4154a90 2024-09-13T17:24:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac0146c0 con 0x7fe0b4154a90 2024-09-13T17:24:14.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b41558e0 con 0x7fe0b4154a90 2024-09-13T17:24:14.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.549+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b4155ea0 con 0x7fe0b4154a90 2024-09-13T17:24:14.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.550+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b415b0f0 con 0x7fe0b4154a90 2024-09-13T17:24:14.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.551+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe0ac014820 con 0x7fe0b4154a90 2024-09-13T17:24:14.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.552+0000 7fe0b0c00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 0x7fe09006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.552+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0ac089cf0 con 0x7fe0b4154a90 2024-09-13T17:24:14.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.553+0000 7fe0b3e00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 0x7fe09006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.553+0000 7fe0b3e00640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 0x7fe09006d0c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4000f60 tx=0x7fe0a4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.557+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe0ac054f70 con 0x7fe0b4154a90 2024-09-13T17:24:14.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.731+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe0b4158ba0 con 0x7fe09006ac00 2024-09-13T17:24:14.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.740+0000 7fe0b0c00640 1 -- 172.21.15.28:0/117610342 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289471 (secure 0 0 0) 0x7fe0b4158ba0 con 0x7fe09006ac00 2024-09-13T17:24:14.745 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:14.747 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:24:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.743+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 msgr2=0x7fe09006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.743+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 0x7fe09006d0c0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7fe0a4000f60 tx=0x7fe0a4009040 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.743+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 msgr2=0x7fe0b40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.743+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 secure :-1 s=READY pgs=2728 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac00efc0 tx=0x7fe0ac00c490 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 shutdown_connections 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe09006ac00 0x7fe09006d0c0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0b4154a90 0x7fe0b40702e0 unknown :-1 s=CLOSED pgs=2728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0b40b0980 0x7fe0b4153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 --2- 172.21.15.28:0/117610342 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe0b406b360 0x7fe0b40aa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 >> 172.21.15.28:0/117610342 conn(0x7fe0b4072e40 msgr2=0x7fe0b4159dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 shutdown_connections 2024-09-13T17:24:14.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.744+0000 7fe0b9400640 1 -- 172.21.15.28:0/117610342 wait complete. 2024-09-13T17:24:14.762 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":714,"stamp":"2024-09-13T17:24:13.129147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":555820,"kb_used_data":24224,"kb_used_omap":12,"kb_used_meta":531443,"kb_avail":754418900,"statfs":{"total":773094113280,"available":772524953600,"internally_reserved":0,"allocated":24805376,"data_stored":11613502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":544198201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001062"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588365+0000","last_change":"2024-09-13T17:24:10.600258+0000","last_active":"2024-09-13T17:24:11.588365+0000","last_peered":"2024-09-13T17:24:11.588365+0000","last_clean":"2024-09-13T17:24:11.588365+0000","last_became_active":"2024-09-13T17:24:10.599973+0000","last_became_peered":"2024-09-13T17:24:10.599973+0000","last_unstale":"2024-09-13T17:24:11.588365+0000","last_undegraded":"2024-09-13T17:24:11.588365+0000","last_fullsized":"2024-09-13T17:24:11.588365+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:08.924474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154450+0000","last_change":"2024-09-13T17:24:10.605961+0000","last_active":"2024-09-13T17:24:12.154450+0000","last_peered":"2024-09-13T17:24:12.154450+0000","last_clean":"2024-09-13T17:24:12.154450+0000","last_became_active":"2024-09-13T17:24:10.605559+0000","last_became_peered":"2024-09-13T17:24:10.605559+0000","last_unstale":"2024-09-13T17:24:12.154450+0000","last_undegraded":"2024-09-13T17:24:12.154450+0000","last_fullsized":"2024-09-13T17:24:12.154450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:42.124789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154510+0000","last_change":"2024-09-13T17:24:10.601232+0000","last_active":"2024-09-13T17:24:12.154510+0000","last_peered":"2024-09-13T17:24:12.154510+0000","last_clean":"2024-09-13T17:24:12.154510+0000","last_became_active":"2024-09-13T17:24:10.600971+0000","last_became_peered":"2024-09-13T17:24:10.600971+0000","last_unstale":"2024-09-13T17:24:12.154510+0000","last_undegraded":"2024-09-13T17:24:12.154510+0000","last_fullsized":"2024-09-13T17:24:12.154510+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:14.362195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154422+0000","last_change":"2024-09-13T17:24:10.596699+0000","last_active":"2024-09-13T17:24:12.154422+0000","last_peered":"2024-09-13T17:24:12.154422+0000","last_clean":"2024-09-13T17:24:12.154422+0000","last_became_active":"2024-09-13T17:24:10.596434+0000","last_became_peered":"2024-09-13T17:24:10.596434+0000","last_unstale":"2024-09-13T17:24:12.154422+0000","last_undegraded":"2024-09-13T17:24:12.154422+0000","last_fullsized":"2024-09-13T17:24:12.154422+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:00.028016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153796+0000","last_change":"2024-09-13T17:24:10.596600+0000","last_active":"2024-09-13T17:24:12.153796+0000","last_peered":"2024-09-13T17:24:12.153796+0000","last_clean":"2024-09-13T17:24:12.153796+0000","last_became_active":"2024-09-13T17:24:10.596321+0000","last_became_peered":"2024-09-13T17:24:10.596321+0000","last_unstale":"2024-09-13T17:24:12.153796+0000","last_undegraded":"2024-09-13T17:24:12.153796+0000","last_fullsized":"2024-09-13T17:24:12.153796+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:07.414024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153277+0000","last_change":"2024-09-13T17:24:10.602768+0000","last_active":"2024-09-13T17:24:12.153277+0000","last_peered":"2024-09-13T17:24:12.153277+0000","last_clean":"2024-09-13T17:24:12.153277+0000","last_became_active":"2024-09-13T17:24:10.602379+0000","last_became_peered":"2024-09-13T17:24:10.602379+0000","last_unstale":"2024-09-13T17:24:12.153277+0000","last_undegraded":"2024-09-13T17:24:12.153277+0000","last_fullsized":"2024-09-13T17:24:12.153277+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:48.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586951+0000","last_change":"2024-09-13T17:24:10.600787+0000","last_active":"2024-09-13T17:24:11.586951+0000","last_peered":"2024-09-13T17:24:11.586951+0000","last_clean":"2024-09-13T17:24:11.586951+0000","last_became_active":"2024-09-13T17:24:10.600449+0000","last_became_peered":"2024-09-13T17:24:10.600449+0000","last_unstale":"2024-09-13T17:24:11.586951+0000","last_undegraded":"2024-09-13T17:24:11.586951+0000","last_fullsized":"2024-09-13T17:24:11.586951+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:53.636468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586956+0000","last_change":"2024-09-13T17:24:10.609262+0000","last_active":"2024-09-13T17:24:11.586956+0000","last_peered":"2024-09-13T17:24:11.586956+0000","last_clean":"2024-09-13T17:24:11.586956+0000","last_became_active":"2024-09-13T17:24:10.608736+0000","last_became_peered":"2024-09-13T17:24:10.608736+0000","last_unstale":"2024-09-13T17:24:11.586956+0000","last_undegraded":"2024-09-13T17:24:11.586956+0000","last_fullsized":"2024-09-13T17:24:11.586956+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:43.230913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154098+0000","last_change":"2024-09-13T17:24:10.594650+0000","last_active":"2024-09-13T17:24:12.154098+0000","last_peered":"2024-09-13T17:24:12.154098+0000","last_clean":"2024-09-13T17:24:12.154098+0000","last_became_active":"2024-09-13T17:24:10.594245+0000","last_became_peered":"2024-09-13T17:24:10.594245+0000","last_unstale":"2024-09-13T17:24:12.154098+0000","last_undegraded":"2024-09-13T17:24:12.154098+0000","last_fullsized":"2024-09-13T17:24:12.154098+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:25.126873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587794+0000","last_change":"2024-09-13T17:24:10.609386+0000","last_active":"2024-09-13T17:24:11.587794+0000","last_peered":"2024-09-13T17:24:11.587794+0000","last_clean":"2024-09-13T17:24:11.587794+0000","last_became_active":"2024-09-13T17:24:10.609097+0000","last_became_peered":"2024-09-13T17:24:10.609097+0000","last_unstale":"2024-09-13T17:24:11.587794+0000","last_undegraded":"2024-09-13T17:24:11.587794+0000","last_fullsized":"2024-09-13T17:24:11.587794+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:42.252875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587513+0000","last_change":"2024-09-13T17:24:10.616431+0000","last_active":"2024-09-13T17:24:11.587513+0000","last_peered":"2024-09-13T17:24:11.587513+0000","last_clean":"2024-09-13T17:24:11.587513+0000","last_became_active":"2024-09-13T17:24:10.616028+0000","last_became_peered":"2024-09-13T17:24:10.616028+0000","last_unstale":"2024-09-13T17:24:11.587513+0000","last_undegraded":"2024-09-13T17:24:11.587513+0000","last_fullsized":"2024-09-13T17:24:11.587513+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:33.776565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587970+0000","last_change":"2024-09-13T17:24:10.592092+0000","last_active":"2024-09-13T17:24:11.587970+0000","last_peered":"2024-09-13T17:24:11.587970+0000","last_clean":"2024-09-13T17:24:11.587970+0000","last_became_active":"2024-09-13T17:24:10.591821+0000","last_became_peered":"2024-09-13T17:24:10.591821+0000","last_unstale":"2024-09-13T17:24:11.587970+0000","last_undegraded":"2024-09-13T17:24:11.587970+0000","last_fullsized":"2024-09-13T17:24:11.587970+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:45.474245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653263+0000","last_change":"2024-09-13T17:24:10.613652+0000","last_active":"2024-09-13T17:24:11.653263+0000","last_peered":"2024-09-13T17:24:11.653263+0000","last_clean":"2024-09-13T17:24:11.653263+0000","last_became_active":"2024-09-13T17:24:10.611859+0000","last_became_peered":"2024-09-13T17:24:10.611859+0000","last_unstale":"2024-09-13T17:24:11.653263+0000","last_undegraded":"2024-09-13T17:24:11.653263+0000","last_fullsized":"2024-09-13T17:24:11.653263+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:24.000458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.652964+0000","last_change":"2024-09-13T17:24:10.594151+0000","last_active":"2024-09-13T17:24:11.652964+0000","last_peered":"2024-09-13T17:24:11.652964+0000","last_clean":"2024-09-13T17:24:11.652964+0000","last_became_active":"2024-09-13T17:24:10.593898+0000","last_became_peered":"2024-09-13T17:24:10.593898+0000","last_unstale":"2024-09-13T17:24:11.652964+0000","last_undegraded":"2024-09-13T17:24:11.652964+0000","last_fullsized":"2024-09-13T17:24:11.652964+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:23:29.924841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653009+0000","last_change":"2024-09-13T17:24:10.601302+0000","last_active":"2024-09-13T17:24:11.653009+0000","last_peered":"2024-09-13T17:24:11.653009+0000","last_clean":"2024-09-13T17:24:11.653009+0000","last_became_active":"2024-09-13T17:24:10.601010+0000","last_became_peered":"2024-09-13T17:24:10.601010+0000","last_unstale":"2024-09-13T17:24:11.653009+0000","last_undegraded":"2024-09-13T17:24:11.653009+0000","last_fullsized":"2024-09-13T17:24:11.653009+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:14.897668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153197+0000","last_change":"2024-09-13T17:24:10.615920+0000","last_active":"2024-09-13T17:24:12.153197+0000","last_peered":"2024-09-13T17:24:12.153197+0000","last_clean":"2024-09-13T17:24:12.153197+0000","last_became_active":"2024-09-13T17:24:10.615521+0000","last_became_peered":"2024-09-13T17:24:10.615521+0000","last_unstale":"2024-09-13T17:24:12.153197+0000","last_undegraded":"2024-09-13T17:24:12.153197+0000","last_fullsized":"2024-09-13T17:24:12.153197+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.933226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584875+0000","last_change":"2024-09-13T17:24:10.606893+0000","last_active":"2024-09-13T17:24:11.584875+0000","last_peered":"2024-09-13T17:24:11.584875+0000","last_clean":"2024-09-13T17:24:11.584875+0000","last_became_active":"2024-09-13T17:24:10.606448+0000","last_became_peered":"2024-09-13T17:24:10.606448+0000","last_unstale":"2024-09-13T17:24:11.584875+0000","last_undegraded":"2024-09-13T17:24:11.584875+0000","last_fullsized":"2024-09-13T17:24:11.584875+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:45.103302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588085+0000","last_change":"2024-09-13T17:24:10.607018+0000","last_active":"2024-09-13T17:24:11.588085+0000","last_peered":"2024-09-13T17:24:11.588085+0000","last_clean":"2024-09-13T17:24:11.588085+0000","last_became_active":"2024-09-13T17:24:10.606614+0000","last_became_peered":"2024-09-13T17:24:10.606614+0000","last_unstale":"2024-09-13T17:24:11.588085+0000","last_undegraded":"2024-09-13T17:24:11.588085+0000","last_fullsized":"2024-09-13T17:24:11.588085+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:21.640431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587520+0000","last_change":"2024-09-13T17:24:10.609108+0000","last_active":"2024-09-13T17:24:11.587520+0000","last_peered":"2024-09-13T17:24:11.587520+0000","last_clean":"2024-09-13T17:24:11.587520+0000","last_became_active":"2024-09-13T17:24:10.608836+0000","last_became_peered":"2024-09-13T17:24:10.608836+0000","last_unstale":"2024-09-13T17:24:11.587520+0000","last_undegraded":"2024-09-13T17:24:11.587520+0000","last_fullsized":"2024-09-13T17:24:11.587520+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:36.118776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153601+0000","last_change":"2024-09-13T17:24:10.610351+0000","last_active":"2024-09-13T17:24:12.153601+0000","last_peered":"2024-09-13T17:24:12.153601+0000","last_clean":"2024-09-13T17:24:12.153601+0000","last_became_active":"2024-09-13T17:24:10.610035+0000","last_became_peered":"2024-09-13T17:24:10.610035+0000","last_unstale":"2024-09-13T17:24:12.153601+0000","last_undegraded":"2024-09-13T17:24:12.153601+0000","last_fullsized":"2024-09-13T17:24:12.153601+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:21.640431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153484+0000","last_change":"2024-09-13T17:24:10.609011+0000","last_active":"2024-09-13T17:24:12.153484+0000","last_peered":"2024-09-13T17:24:12.153484+0000","last_clean":"2024-09-13T17:24:12.153484+0000","last_became_active":"2024-09-13T17:24:10.608740+0000","last_became_peered":"2024-09-13T17:24:10.608740+0000","last_unstale":"2024-09-13T17:24:12.153484+0000","last_undegraded":"2024-09-13T17:24:12.153484+0000","last_fullsized":"2024-09-13T17:24:12.153484+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:16.297416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153476+0000","last_change":"2024-09-13T17:24:10.612570+0000","last_active":"2024-09-13T17:24:12.153476+0000","last_peered":"2024-09-13T17:24:12.153476+0000","last_clean":"2024-09-13T17:24:12.153476+0000","last_became_active":"2024-09-13T17:24:10.612288+0000","last_became_peered":"2024-09-13T17:24:10.612288+0000","last_unstale":"2024-09-13T17:24:12.153476+0000","last_undegraded":"2024-09-13T17:24:12.153476+0000","last_fullsized":"2024-09-13T17:24:12.153476+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587063+0000","last_change":"2024-09-13T17:24:10.598390+0000","last_active":"2024-09-13T17:24:11.587063+0000","last_peered":"2024-09-13T17:24:11.587063+0000","last_clean":"2024-09-13T17:24:11.587063+0000","last_became_active":"2024-09-13T17:24:10.598151+0000","last_became_peered":"2024-09-13T17:24:10.598151+0000","last_unstale":"2024-09-13T17:24:11.587063+0000","last_undegraded":"2024-09-13T17:24:11.587063+0000","last_fullsized":"2024-09-13T17:24:11.587063+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:30.009252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584198+0000","last_change":"2024-09-13T17:24:10.605233+0000","last_active":"2024-09-13T17:24:11.584198+0000","last_peered":"2024-09-13T17:24:11.584198+0000","last_clean":"2024-09-13T17:24:11.584198+0000","last_became_active":"2024-09-13T17:24:10.604454+0000","last_became_peered":"2024-09-13T17:24:10.604454+0000","last_unstale":"2024-09-13T17:24:11.584198+0000","last_undegraded":"2024-09-13T17:24:11.584198+0000","last_fullsized":"2024-09-13T17:24:11.584198+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:18.471672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653348+0000","last_change":"2024-09-13T17:24:10.595916+0000","last_active":"2024-09-13T17:24:11.653348+0000","last_peered":"2024-09-13T17:24:11.653348+0000","last_clean":"2024-09-13T17:24:11.653348+0000","last_became_active":"2024-09-13T17:24:10.595623+0000","last_became_peered":"2024-09-13T17:24:10.595623+0000","last_unstale":"2024-09-13T17:24:11.653348+0000","last_undegraded":"2024-09-13T17:24:11.653348+0000","last_fullsized":"2024-09-13T17:24:11.653348+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:00.028016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153815+0000","last_change":"2024-09-13T17:24:10.597754+0000","last_active":"2024-09-13T17:24:12.153815+0000","last_peered":"2024-09-13T17:24:12.153815+0000","last_clean":"2024-09-13T17:24:12.153815+0000","last_became_active":"2024-09-13T17:24:10.597466+0000","last_became_peered":"2024-09-13T17:24:10.597466+0000","last_unstale":"2024-09-13T17:24:12.153815+0000","last_undegraded":"2024-09-13T17:24:12.153815+0000","last_fullsized":"2024-09-13T17:24:12.153815+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:05.071183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587636+0000","last_change":"2024-09-13T17:24:10.593257+0000","last_active":"2024-09-13T17:24:11.587636+0000","last_peered":"2024-09-13T17:24:11.587636+0000","last_clean":"2024-09-13T17:24:11.587636+0000","last_became_active":"2024-09-13T17:24:10.592990+0000","last_became_peered":"2024-09-13T17:24:10.592990+0000","last_unstale":"2024-09-13T17:24:11.587636+0000","last_undegraded":"2024-09-13T17:24:11.587636+0000","last_fullsized":"2024-09-13T17:24:11.587636+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.933226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653456+0000","last_change":"2024-09-13T17:24:10.601276+0000","last_active":"2024-09-13T17:24:11.653456+0000","last_peered":"2024-09-13T17:24:11.653456+0000","last_clean":"2024-09-13T17:24:11.653456+0000","last_became_active":"2024-09-13T17:24:10.598273+0000","last_became_peered":"2024-09-13T17:24:10.598273+0000","last_unstale":"2024-09-13T17:24:11.653456+0000","last_undegraded":"2024-09-13T17:24:11.653456+0000","last_fullsized":"2024-09-13T17:24:11.653456+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:16:29.406341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584619+0000","last_change":"2024-09-13T17:24:10.614554+0000","last_active":"2024-09-13T17:24:11.584619+0000","last_peered":"2024-09-13T17:24:11.584619+0000","last_clean":"2024-09-13T17:24:11.584619+0000","last_became_active":"2024-09-13T17:24:10.614283+0000","last_became_peered":"2024-09-13T17:24:10.614283+0000","last_unstale":"2024-09-13T17:24:11.584619+0000","last_undegraded":"2024-09-13T17:24:11.584619+0000","last_fullsized":"2024-09-13T17:24:11.584619+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:56.093781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653111+0000","last_change":"2024-09-13T17:24:10.594132+0000","last_active":"2024-09-13T17:24:11.653111+0000","last_peered":"2024-09-13T17:24:11.653111+0000","last_clean":"2024-09-13T17:24:11.653111+0000","last_became_active":"2024-09-13T17:24:10.593876+0000","last_became_peered":"2024-09-13T17:24:10.593876+0000","last_unstale":"2024-09-13T17:24:11.653111+0000","last_undegraded":"2024-09-13T17:24:11.653111+0000","last_fullsized":"2024-09-13T17:24:11.653111+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:07.120077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587061+0000","last_change":"2024-09-13T17:24:10.597332+0000","last_active":"2024-09-13T17:24:11.587061+0000","last_peered":"2024-09-13T17:24:11.587061+0000","last_clean":"2024-09-13T17:24:11.587061+0000","last_became_active":"2024-09-13T17:24:10.596977+0000","last_became_peered":"2024-09-13T17:24:10.596977+0000","last_unstale":"2024-09-13T17:24:11.587061+0000","last_undegraded":"2024-09-13T17:24:11.587061+0000","last_fullsized":"2024-09-13T17:24:11.587061+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:00.767994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587895+0000","last_change":"2024-09-13T17:24:10.598328+0000","last_active":"2024-09-13T17:24:11.587895+0000","last_peered":"2024-09-13T17:24:11.587895+0000","last_clean":"2024-09-13T17:24:11.587895+0000","last_became_active":"2024-09-13T17:24:10.598059+0000","last_became_peered":"2024-09-13T17:24:10.598059+0000","last_unstale":"2024-09-13T17:24:11.587895+0000","last_undegraded":"2024-09-13T17:24:11.587895+0000","last_fullsized":"2024-09-13T17:24:11.587895+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.168528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"232'4","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584983+0000","last_change":"2024-09-13T17:24:10.166081+0000","last_active":"2024-09-13T17:24:11.584983+0000","last_peered":"2024-09-13T17:24:11.584983+0000","last_clean":"2024-09-13T17:24:11.584983+0000","last_became_active":"2024-09-13T17:24:10.164208+0000","last_became_peered":"2024-09-13T17:24:10.164208+0000","last_unstale":"2024-09-13T17:24:11.584983+0000","last_undegraded":"2024-09-13T17:24:11.584983+0000","last_fullsized":"2024-09-13T17:24:11.584983+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:47.844554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653369+0000","last_change":"2024-09-13T17:24:10.602308+0000","last_active":"2024-09-13T17:24:11.653369+0000","last_peered":"2024-09-13T17:24:11.653369+0000","last_clean":"2024-09-13T17:24:11.653369+0000","last_became_active":"2024-09-13T17:24:10.600996+0000","last_became_peered":"2024-09-13T17:24:10.600996+0000","last_unstale":"2024-09-13T17:24:11.653369+0000","last_undegraded":"2024-09-13T17:24:11.653369+0000","last_fullsized":"2024-09-13T17:24:11.653369+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:23.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653235+0000","last_change":"2024-09-13T17:24:10.164589+0000","last_active":"2024-09-13T17:24:11.653235+0000","last_peered":"2024-09-13T17:24:11.653235+0000","last_clean":"2024-09-13T17:24:11.653235+0000","last_became_active":"2024-09-13T17:24:10.164313+0000","last_became_peered":"2024-09-13T17:24:10.164313+0000","last_unstale":"2024-09-13T17:24:11.653235+0000","last_undegraded":"2024-09-13T17:24:11.653235+0000","last_fullsized":"2024-09-13T17:24:11.653235+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:57.577314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587925+0000","last_change":"2024-09-13T17:24:10.606967+0000","last_active":"2024-09-13T17:24:11.587925+0000","last_peered":"2024-09-13T17:24:11.587925+0000","last_clean":"2024-09-13T17:24:11.587925+0000","last_became_active":"2024-09-13T17:24:10.606674+0000","last_became_peered":"2024-09-13T17:24:10.606674+0000","last_unstale":"2024-09-13T17:24:11.587925+0000","last_undegraded":"2024-09-13T17:24:11.587925+0000","last_fullsized":"2024-09-13T17:24:11.587925+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:45.474245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153901+0000","last_change":"2024-09-13T17:24:10.164923+0000","last_active":"2024-09-13T17:24:12.153901+0000","last_peered":"2024-09-13T17:24:12.153901+0000","last_clean":"2024-09-13T17:24:12.153901+0000","last_became_active":"2024-09-13T17:24:10.164636+0000","last_became_peered":"2024-09-13T17:24:10.164636+0000","last_unstale":"2024-09-13T17:24:12.153901+0000","last_undegraded":"2024-09-13T17:24:12.153901+0000","last_fullsized":"2024-09-13T17:24:12.153901+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:29.403573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588083+0000","last_change":"2024-09-13T17:24:10.597639+0000","last_active":"2024-09-13T17:24:11.588083+0000","last_peered":"2024-09-13T17:24:11.588083+0000","last_clean":"2024-09-13T17:24:11.588083+0000","last_became_active":"2024-09-13T17:24:10.596401+0000","last_became_peered":"2024-09-13T17:24:10.596401+0000","last_unstale":"2024-09-13T17:24:11.588083+0000","last_undegraded":"2024-09-13T17:24:11.588083+0000","last_fullsized":"2024-09-13T17:24:11.588083+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:33.776565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153689+0000","last_change":"2024-09-13T17:24:10.162847+0000","last_active":"2024-09-13T17:24:12.153689+0000","last_peered":"2024-09-13T17:24:12.153689+0000","last_clean":"2024-09-13T17:24:12.153689+0000","last_became_active":"2024-09-13T17:24:10.161943+0000","last_became_peered":"2024-09-13T17:24:10.161943+0000","last_unstale":"2024-09-13T17:24:12.153689+0000","last_undegraded":"2024-09-13T17:24:12.153689+0000","last_fullsized":"2024-09-13T17:24:12.153689+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.799697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153374+0000","last_change":"2024-09-13T17:24:10.608931+0000","last_active":"2024-09-13T17:24:12.153374+0000","last_peered":"2024-09-13T17:24:12.153374+0000","last_clean":"2024-09-13T17:24:12.153374+0000","last_became_active":"2024-09-13T17:24:10.608646+0000","last_became_peered":"2024-09-13T17:24:10.608646+0000","last_unstale":"2024-09-13T17:24:12.153374+0000","last_undegraded":"2024-09-13T17:24:12.153374+0000","last_fullsized":"2024-09-13T17:24:12.153374+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:08.724701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587341+0000","last_change":"2024-09-13T17:24:10.166535+0000","last_active":"2024-09-13T17:24:11.587341+0000","last_peered":"2024-09-13T17:24:11.587341+0000","last_clean":"2024-09-13T17:24:11.587341+0000","last_became_active":"2024-09-13T17:24:10.165741+0000","last_became_peered":"2024-09-13T17:24:10.165741+0000","last_unstale":"2024-09-13T17:24:11.587341+0000","last_undegraded":"2024-09-13T17:24:11.587341+0000","last_fullsized":"2024-09-13T17:24:11.587341+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:39:24.643100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153553+0000","last_change":"2024-09-13T17:24:10.604093+0000","last_active":"2024-09-13T17:24:12.153553+0000","last_peered":"2024-09-13T17:24:12.153553+0000","last_clean":"2024-09-13T17:24:12.153553+0000","last_became_active":"2024-09-13T17:24:10.603857+0000","last_became_peered":"2024-09-13T17:24:10.603857+0000","last_unstale":"2024-09-13T17:24:12.153553+0000","last_undegraded":"2024-09-13T17:24:12.153553+0000","last_fullsized":"2024-09-13T17:24:12.153553+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.168528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584763+0000","last_change":"2024-09-13T17:24:10.153073+0000","last_active":"2024-09-13T17:24:11.584763+0000","last_peered":"2024-09-13T17:24:11.584763+0000","last_clean":"2024-09-13T17:24:11.584763+0000","last_became_active":"2024-09-13T17:24:10.152795+0000","last_became_peered":"2024-09-13T17:24:10.152795+0000","last_unstale":"2024-09-13T17:24:11.584763+0000","last_undegraded":"2024-09-13T17:24:11.584763+0000","last_fullsized":"2024-09-13T17:24:11.584763+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:36.005955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153298+0000","last_change":"2024-09-13T17:24:10.602855+0000","last_active":"2024-09-13T17:24:12.153298+0000","last_peered":"2024-09-13T17:24:12.153298+0000","last_clean":"2024-09-13T17:24:12.153298+0000","last_became_active":"2024-09-13T17:24:10.602604+0000","last_became_peered":"2024-09-13T17:24:10.602604+0000","last_unstale":"2024-09-13T17:24:12.153298+0000","last_undegraded":"2024-09-13T17:24:12.153298+0000","last_fullsized":"2024-09-13T17:24:12.153298+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:36.118776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153740+0000","last_change":"2024-09-13T17:24:10.168440+0000","last_active":"2024-09-13T17:24:12.153740+0000","last_peered":"2024-09-13T17:24:12.153740+0000","last_clean":"2024-09-13T17:24:12.153740+0000","last_became_active":"2024-09-13T17:24:10.168161+0000","last_became_peered":"2024-09-13T17:24:10.168161+0000","last_unstale":"2024-09-13T17:24:12.153740+0000","last_undegraded":"2024-09-13T17:24:12.153740+0000","last_fullsized":"2024-09-13T17:24:12.153740+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:22.770548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587270+0000","last_change":"2024-09-13T17:24:10.600654+0000","last_active":"2024-09-13T17:24:11.587270+0000","last_peered":"2024-09-13T17:24:11.587270+0000","last_clean":"2024-09-13T17:24:11.587270+0000","last_became_active":"2024-09-13T17:24:10.600383+0000","last_became_peered":"2024-09-13T17:24:10.600383+0000","last_unstale":"2024-09-13T17:24:11.587270+0000","last_undegraded":"2024-09-13T17:24:11.587270+0000","last_fullsized":"2024-09-13T17:24:11.587270+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:26.579180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587043+0000","last_change":"2024-09-13T17:24:10.171987+0000","last_active":"2024-09-13T17:24:11.587043+0000","last_peered":"2024-09-13T17:24:11.587043+0000","last_clean":"2024-09-13T17:24:11.587043+0000","last_became_active":"2024-09-13T17:24:10.171717+0000","last_became_peered":"2024-09-13T17:24:10.171717+0000","last_unstale":"2024-09-13T17:24:11.587043+0000","last_undegraded":"2024-09-13T17:24:11.587043+0000","last_fullsized":"2024-09-13T17:24:11.587043+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:41.259957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588021+0000","last_change":"2024-09-13T17:24:10.607207+0000","last_active":"2024-09-13T17:24:11.588021+0000","last_peered":"2024-09-13T17:24:11.588021+0000","last_clean":"2024-09-13T17:24:11.588021+0000","last_became_active":"2024-09-13T17:24:10.606649+0000","last_became_peered":"2024-09-13T17:24:10.606649+0000","last_unstale":"2024-09-13T17:24:11.588021+0000","last_undegraded":"2024-09-13T17:24:11.588021+0000","last_fullsized":"2024-09-13T17:24:11.588021+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:48.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584969+0000","last_change":"2024-09-13T17:24:10.168009+0000","last_active":"2024-09-13T17:24:11.584969+0000","last_peered":"2024-09-13T17:24:11.584969+0000","last_clean":"2024-09-13T17:24:11.584969+0000","last_became_active":"2024-09-13T17:24:10.167731+0000","last_became_peered":"2024-09-13T17:24:10.167731+0000","last_unstale":"2024-09-13T17:24:11.584969+0000","last_undegraded":"2024-09-13T17:24:11.584969+0000","last_fullsized":"2024-09-13T17:24:11.584969+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:35.999463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584897+0000","last_change":"2024-09-13T17:24:10.614744+0000","last_active":"2024-09-13T17:24:11.584897+0000","last_peered":"2024-09-13T17:24:11.584897+0000","last_clean":"2024-09-13T17:24:11.584897+0000","last_became_active":"2024-09-13T17:24:10.614432+0000","last_became_peered":"2024-09-13T17:24:10.614432+0000","last_unstale":"2024-09-13T17:24:11.584897+0000","last_undegraded":"2024-09-13T17:24:11.584897+0000","last_fullsized":"2024-09-13T17:24:11.584897+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:48.823884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153558+0000","last_change":"2024-09-13T17:24:10.170053+0000","last_active":"2024-09-13T17:24:12.153558+0000","last_peered":"2024-09-13T17:24:12.153558+0000","last_clean":"2024-09-13T17:24:12.153558+0000","last_became_active":"2024-09-13T17:24:10.169769+0000","last_became_peered":"2024-09-13T17:24:10.169769+0000","last_unstale":"2024-09-13T17:24:12.153558+0000","last_undegraded":"2024-09-13T17:24:12.153558+0000","last_fullsized":"2024-09-13T17:24:12.153558+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:54:20.173529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653048+0000","last_change":"2024-09-13T17:24:10.593000+0000","last_active":"2024-09-13T17:24:11.653048+0000","last_peered":"2024-09-13T17:24:11.653048+0000","last_clean":"2024-09-13T17:24:11.653048+0000","last_became_active":"2024-09-13T17:24:10.592762+0000","last_became_peered":"2024-09-13T17:24:10.592762+0000","last_unstale":"2024-09-13T17:24:11.653048+0000","last_undegraded":"2024-09-13T17:24:11.653048+0000","last_fullsized":"2024-09-13T17:24:11.653048+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:58.029629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"232'3","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153949+0000","last_change":"2024-09-13T17:24:10.170259+0000","last_active":"2024-09-13T17:24:12.153949+0000","last_peered":"2024-09-13T17:24:12.153949+0000","last_clean":"2024-09-13T17:24:12.153949+0000","last_became_active":"2024-09-13T17:24:10.170014+0000","last_became_peered":"2024-09-13T17:24:10.170014+0000","last_unstale":"2024-09-13T17:24:12.153949+0000","last_undegraded":"2024-09-13T17:24:12.153949+0000","last_fullsized":"2024-09-13T17:24:12.153949+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:32.986051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584695+0000","last_change":"2024-09-13T17:24:10.617281+0000","last_active":"2024-09-13T17:24:11.584695+0000","last_peered":"2024-09-13T17:24:11.584695+0000","last_clean":"2024-09-13T17:24:11.584695+0000","last_became_active":"2024-09-13T17:24:10.616691+0000","last_became_peered":"2024-09-13T17:24:10.616691+0000","last_unstale":"2024-09-13T17:24:11.584695+0000","last_undegraded":"2024-09-13T17:24:11.584695+0000","last_fullsized":"2024-09-13T17:24:11.584695+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:56.815950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"232'6","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584698+0000","last_change":"2024-09-13T17:24:10.162890+0000","last_active":"2024-09-13T17:24:11.584698+0000","last_peered":"2024-09-13T17:24:11.584698+0000","last_clean":"2024-09-13T17:24:11.584698+0000","last_became_active":"2024-09-13T17:24:10.160626+0000","last_became_peered":"2024-09-13T17:24:10.160626+0000","last_unstale":"2024-09-13T17:24:11.584698+0000","last_undegraded":"2024-09-13T17:24:11.584698+0000","last_fullsized":"2024-09-13T17:24:11.584698+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:58.971790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584574+0000","last_change":"2024-09-13T17:24:10.603088+0000","last_active":"2024-09-13T17:24:11.584574+0000","last_peered":"2024-09-13T17:24:11.584574+0000","last_clean":"2024-09-13T17:24:11.584574+0000","last_became_active":"2024-09-13T17:24:10.602839+0000","last_became_peered":"2024-09-13T17:24:10.602839+0000","last_unstale":"2024-09-13T17:24:11.584574+0000","last_undegraded":"2024-09-13T17:24:11.584574+0000","last_fullsized":"2024-09-13T17:24:11.584574+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:02.695710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587883+0000","last_change":"2024-09-13T17:24:10.165606+0000","last_active":"2024-09-13T17:24:11.587883+0000","last_peered":"2024-09-13T17:24:11.587883+0000","last_clean":"2024-09-13T17:24:11.587883+0000","last_became_active":"2024-09-13T17:24:10.165288+0000","last_became_peered":"2024-09-13T17:24:10.165288+0000","last_unstale":"2024-09-13T17:24:11.587883+0000","last_undegraded":"2024-09-13T17:24:11.587883+0000","last_fullsized":"2024-09-13T17:24:11.587883+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:36.533122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153652+0000","last_change":"2024-09-13T17:24:10.604730+0000","last_active":"2024-09-13T17:24:12.153652+0000","last_peered":"2024-09-13T17:24:12.153652+0000","last_clean":"2024-09-13T17:24:12.153652+0000","last_became_active":"2024-09-13T17:24:10.604467+0000","last_became_peered":"2024-09-13T17:24:10.604467+0000","last_unstale":"2024-09-13T17:24:12.153652+0000","last_undegraded":"2024-09-13T17:24:12.153652+0000","last_fullsized":"2024-09-13T17:24:12.153652+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:42.252875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153556+0000","last_change":"2024-09-13T17:24:10.162284+0000","last_active":"2024-09-13T17:24:12.153556+0000","last_peered":"2024-09-13T17:24:12.153556+0000","last_clean":"2024-09-13T17:24:12.153556+0000","last_became_active":"2024-09-13T17:24:10.162038+0000","last_became_peered":"2024-09-13T17:24:10.162038+0000","last_unstale":"2024-09-13T17:24:12.153556+0000","last_undegraded":"2024-09-13T17:24:12.153556+0000","last_fullsized":"2024-09-13T17:24:12.153556+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:01.002427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584270+0000","last_change":"2024-09-13T17:24:10.599364+0000","last_active":"2024-09-13T17:24:11.584270+0000","last_peered":"2024-09-13T17:24:11.584270+0000","last_clean":"2024-09-13T17:24:11.584270+0000","last_became_active":"2024-09-13T17:24:10.599050+0000","last_became_peered":"2024-09-13T17:24:10.599050+0000","last_unstale":"2024-09-13T17:24:11.584270+0000","last_undegraded":"2024-09-13T17:24:11.584270+0000","last_fullsized":"2024-09-13T17:24:11.584270+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:25.354254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154264+0000","last_change":"2024-09-13T17:24:10.162253+0000","last_active":"2024-09-13T17:24:12.154264+0000","last_peered":"2024-09-13T17:24:12.154264+0000","last_clean":"2024-09-13T17:24:12.154264+0000","last_became_active":"2024-09-13T17:24:10.161951+0000","last_became_peered":"2024-09-13T17:24:10.161951+0000","last_unstale":"2024-09-13T17:24:12.154264+0000","last_undegraded":"2024-09-13T17:24:12.154264+0000","last_fullsized":"2024-09-13T17:24:12.154264+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:58.963914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584178+0000","last_change":"2024-09-13T17:24:10.599667+0000","last_active":"2024-09-13T17:24:11.584178+0000","last_peered":"2024-09-13T17:24:11.584178+0000","last_clean":"2024-09-13T17:24:11.584178+0000","last_became_active":"2024-09-13T17:24:10.599421+0000","last_became_peered":"2024-09-13T17:24:10.599421+0000","last_unstale":"2024-09-13T17:24:11.584178+0000","last_undegraded":"2024-09-13T17:24:11.584178+0000","last_fullsized":"2024-09-13T17:24:11.584178+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:41.441391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"232'8","reported_seq":19,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153390+0000","last_change":"2024-09-13T17:24:10.159134+0000","last_active":"2024-09-13T17:24:12.153390+0000","last_peered":"2024-09-13T17:24:12.153390+0000","last_clean":"2024-09-13T17:24:12.153390+0000","last_became_active":"2024-09-13T17:24:10.158699+0000","last_became_peered":"2024-09-13T17:24:10.158699+0000","last_unstale":"2024-09-13T17:24:12.153390+0000","last_undegraded":"2024-09-13T17:24:12.153390+0000","last_fullsized":"2024-09-13T17:24:12.153390+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:45.093617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653059+0000","last_change":"2024-09-13T17:24:10.606193+0000","last_active":"2024-09-13T17:24:11.653059+0000","last_peered":"2024-09-13T17:24:11.653059+0000","last_clean":"2024-09-13T17:24:11.653059+0000","last_became_active":"2024-09-13T17:24:10.605633+0000","last_became_peered":"2024-09-13T17:24:10.605633+0000","last_unstale":"2024-09-13T17:24:11.653059+0000","last_undegraded":"2024-09-13T17:24:11.653059+0000","last_fullsized":"2024-09-13T17:24:11.653059+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:07.414024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588369+0000","last_change":"2024-09-13T17:24:10.168701+0000","last_active":"2024-09-13T17:24:11.588369+0000","last_peered":"2024-09-13T17:24:11.588369+0000","last_clean":"2024-09-13T17:24:11.588369+0000","last_became_active":"2024-09-13T17:24:10.167344+0000","last_became_peered":"2024-09-13T17:24:10.167344+0000","last_unstale":"2024-09-13T17:24:11.588369+0000","last_undegraded":"2024-09-13T17:24:11.588369+0000","last_fullsized":"2024-09-13T17:24:11.588369+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:09.135618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154293+0000","last_change":"2024-09-13T17:24:10.612462+0000","last_active":"2024-09-13T17:24:12.154293+0000","last_peered":"2024-09-13T17:24:12.154293+0000","last_clean":"2024-09-13T17:24:12.154293+0000","last_became_active":"2024-09-13T17:24:10.612183+0000","last_became_peered":"2024-09-13T17:24:10.612183+0000","last_unstale":"2024-09-13T17:24:12.154293+0000","last_undegraded":"2024-09-13T17:24:12.154293+0000","last_fullsized":"2024-09-13T17:24:12.154293+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:42.268007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153650+0000","last_change":"2024-09-13T17:24:10.162625+0000","last_active":"2024-09-13T17:24:12.153650+0000","last_peered":"2024-09-13T17:24:12.153650+0000","last_clean":"2024-09-13T17:24:12.153650+0000","last_became_active":"2024-09-13T17:24:10.162310+0000","last_became_peered":"2024-09-13T17:24:10.162310+0000","last_unstale":"2024-09-13T17:24:12.153650+0000","last_undegraded":"2024-09-13T17:24:12.153650+0000","last_fullsized":"2024-09-13T17:24:12.153650+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:00.385884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653154+0000","last_change":"2024-09-13T17:24:10.601820+0000","last_active":"2024-09-13T17:24:11.653154+0000","last_peered":"2024-09-13T17:24:11.653154+0000","last_clean":"2024-09-13T17:24:11.653154+0000","last_became_active":"2024-09-13T17:24:10.601536+0000","last_became_peered":"2024-09-13T17:24:10.601536+0000","last_unstale":"2024-09-13T17:24:11.653154+0000","last_undegraded":"2024-09-13T17:24:11.653154+0000","last_fullsized":"2024-09-13T17:24:11.653154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:13.970855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154343+0000","last_change":"2024-09-13T17:24:10.164555+0000","last_active":"2024-09-13T17:24:12.154343+0000","last_peered":"2024-09-13T17:24:12.154343+0000","last_clean":"2024-09-13T17:24:12.154343+0000","last_became_active":"2024-09-13T17:24:10.164194+0000","last_became_peered":"2024-09-13T17:24:10.164194+0000","last_unstale":"2024-09-13T17:24:12.154343+0000","last_undegraded":"2024-09-13T17:24:12.154343+0000","last_fullsized":"2024-09-13T17:24:12.154343+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:18.484595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154548+0000","last_change":"2024-09-13T17:24:10.600664+0000","last_active":"2024-09-13T17:24:12.154548+0000","last_peered":"2024-09-13T17:24:12.154548+0000","last_clean":"2024-09-13T17:24:12.154548+0000","last_became_active":"2024-09-13T17:24:10.597527+0000","last_became_peered":"2024-09-13T17:24:10.597527+0000","last_unstale":"2024-09-13T17:24:12.154548+0000","last_undegraded":"2024-09-13T17:24:12.154548+0000","last_fullsized":"2024-09-13T17:24:12.154548+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:58.029629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653284+0000","last_change":"2024-09-13T17:24:10.170331+0000","last_active":"2024-09-13T17:24:11.653284+0000","last_peered":"2024-09-13T17:24:11.653284+0000","last_clean":"2024-09-13T17:24:11.653284+0000","last_became_active":"2024-09-13T17:24:10.169843+0000","last_became_peered":"2024-09-13T17:24:10.169843+0000","last_unstale":"2024-09-13T17:24:11.653284+0000","last_undegraded":"2024-09-13T17:24:11.653284+0000","last_fullsized":"2024-09-13T17:24:11.653284+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:14.619458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153303+0000","last_change":"2024-09-13T17:24:10.602619+0000","last_active":"2024-09-13T17:24:12.153303+0000","last_peered":"2024-09-13T17:24:12.153303+0000","last_clean":"2024-09-13T17:24:12.153303+0000","last_became_active":"2024-09-13T17:24:10.602056+0000","last_became_peered":"2024-09-13T17:24:10.602056+0000","last_unstale":"2024-09-13T17:24:12.153303+0000","last_undegraded":"2024-09-13T17:24:12.153303+0000","last_fullsized":"2024-09-13T17:24:12.153303+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:37.541172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153692+0000","last_change":"2024-09-13T17:24:10.157895+0000","last_active":"2024-09-13T17:24:12.153692+0000","last_peered":"2024-09-13T17:24:12.153692+0000","last_clean":"2024-09-13T17:24:12.153692+0000","last_became_active":"2024-09-13T17:24:10.157277+0000","last_became_peered":"2024-09-13T17:24:10.157277+0000","last_unstale":"2024-09-13T17:24:12.153692+0000","last_undegraded":"2024-09-13T17:24:12.153692+0000","last_fullsized":"2024-09-13T17:24:12.153692+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:21.568488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153732+0000","last_change":"2024-09-13T17:24:10.611408+0000","last_active":"2024-09-13T17:24:12.153732+0000","last_peered":"2024-09-13T17:24:12.153732+0000","last_clean":"2024-09-13T17:24:12.153732+0000","last_became_active":"2024-09-13T17:24:10.611166+0000","last_became_peered":"2024-09-13T17:24:10.611166+0000","last_unstale":"2024-09-13T17:24:12.153732+0000","last_undegraded":"2024-09-13T17:24:12.153732+0000","last_fullsized":"2024-09-13T17:24:12.153732+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:30.009252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587824+0000","last_change":"2024-09-13T17:24:10.165683+0000","last_active":"2024-09-13T17:24:11.587824+0000","last_peered":"2024-09-13T17:24:11.587824+0000","last_clean":"2024-09-13T17:24:11.587824+0000","last_became_active":"2024-09-13T17:24:10.165437+0000","last_became_peered":"2024-09-13T17:24:10.165437+0000","last_unstale":"2024-09-13T17:24:11.587824+0000","last_undegraded":"2024-09-13T17:24:11.587824+0000","last_fullsized":"2024-09-13T17:24:11.587824+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:20.632382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584331+0000","last_change":"2024-09-13T17:24:10.600609+0000","last_active":"2024-09-13T17:24:11.584331+0000","last_peered":"2024-09-13T17:24:11.584331+0000","last_clean":"2024-09-13T17:24:11.584331+0000","last_became_active":"2024-09-13T17:24:10.600301+0000","last_became_peered":"2024-09-13T17:24:10.600301+0000","last_unstale":"2024-09-13T17:24:11.584331+0000","last_undegraded":"2024-09-13T17:24:11.584331+0000","last_fullsized":"2024-09-13T17:24:11.584331+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:00.894395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154328+0000","last_change":"2024-09-13T17:24:10.164654+0000","last_active":"2024-09-13T17:24:12.154328+0000","last_peered":"2024-09-13T17:24:12.154328+0000","last_clean":"2024-09-13T17:24:12.154328+0000","last_became_active":"2024-09-13T17:24:10.164398+0000","last_became_peered":"2024-09-13T17:24:10.164398+0000","last_unstale":"2024-09-13T17:24:12.154328+0000","last_undegraded":"2024-09-13T17:24:12.154328+0000","last_fullsized":"2024-09-13T17:24:12.154328+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:00.652836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153373+0000","last_change":"2024-09-13T17:24:10.597555+0000","last_active":"2024-09-13T17:24:12.153373+0000","last_peered":"2024-09-13T17:24:12.153373+0000","last_clean":"2024-09-13T17:24:12.153373+0000","last_became_active":"2024-09-13T17:24:10.597334+0000","last_became_peered":"2024-09-13T17:24:10.597334+0000","last_unstale":"2024-09-13T17:24:12.153373+0000","last_undegraded":"2024-09-13T17:24:12.153373+0000","last_fullsized":"2024-09-13T17:24:12.153373+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:53.011193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587052+0000","last_change":"2024-09-13T17:24:10.160038+0000","last_active":"2024-09-13T17:24:11.587052+0000","last_peered":"2024-09-13T17:24:11.587052+0000","last_clean":"2024-09-13T17:24:11.587052+0000","last_became_active":"2024-09-13T17:24:10.159232+0000","last_became_peered":"2024-09-13T17:24:10.159232+0000","last_unstale":"2024-09-13T17:24:11.587052+0000","last_undegraded":"2024-09-13T17:24:11.587052+0000","last_fullsized":"2024-09-13T17:24:11.587052+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:52.628418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653301+0000","last_change":"2024-09-13T17:24:10.605898+0000","last_active":"2024-09-13T17:24:11.653301+0000","last_peered":"2024-09-13T17:24:11.653301+0000","last_clean":"2024-09-13T17:24:11.653301+0000","last_became_active":"2024-09-13T17:24:10.605270+0000","last_became_peered":"2024-09-13T17:24:10.605270+0000","last_unstale":"2024-09-13T17:24:11.653301+0000","last_undegraded":"2024-09-13T17:24:11.653301+0000","last_fullsized":"2024-09-13T17:24:11.653301+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:14.362195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653428+0000","last_change":"2024-09-13T17:24:10.170300+0000","last_active":"2024-09-13T17:24:11.653428+0000","last_peered":"2024-09-13T17:24:11.653428+0000","last_clean":"2024-09-13T17:24:11.653428+0000","last_became_active":"2024-09-13T17:24:10.168746+0000","last_became_peered":"2024-09-13T17:24:10.168746+0000","last_unstale":"2024-09-13T17:24:11.653428+0000","last_undegraded":"2024-09-13T17:24:11.653428+0000","last_fullsized":"2024-09-13T17:24:11.653428+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:20.275319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584924+0000","last_change":"2024-09-13T17:24:10.600902+0000","last_active":"2024-09-13T17:24:11.584924+0000","last_peered":"2024-09-13T17:24:11.584924+0000","last_clean":"2024-09-13T17:24:11.584924+0000","last_became_active":"2024-09-13T17:24:10.600071+0000","last_became_peered":"2024-09-13T17:24:10.600071+0000","last_unstale":"2024-09-13T17:24:11.584924+0000","last_undegraded":"2024-09-13T17:24:11.584924+0000","last_fullsized":"2024-09-13T17:24:11.584924+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:12.019907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153826+0000","last_change":"2024-09-13T17:24:10.169837+0000","last_active":"2024-09-13T17:24:12.153826+0000","last_peered":"2024-09-13T17:24:12.153826+0000","last_clean":"2024-09-13T17:24:12.153826+0000","last_became_active":"2024-09-13T17:24:10.169558+0000","last_became_peered":"2024-09-13T17:24:10.169558+0000","last_unstale":"2024-09-13T17:24:12.153826+0000","last_undegraded":"2024-09-13T17:24:12.153826+0000","last_fullsized":"2024-09-13T17:24:12.153826+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:36.655203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153754+0000","last_change":"2024-09-13T17:24:10.598827+0000","last_active":"2024-09-13T17:24:12.153754+0000","last_peered":"2024-09-13T17:24:12.153754+0000","last_clean":"2024-09-13T17:24:12.153754+0000","last_became_active":"2024-09-13T17:24:10.598569+0000","last_became_peered":"2024-09-13T17:24:10.598569+0000","last_unstale":"2024-09-13T17:24:12.153754+0000","last_undegraded":"2024-09-13T17:24:12.153754+0000","last_fullsized":"2024-09-13T17:24:12.153754+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:23.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587735+0000","last_change":"2024-09-13T17:24:10.163976+0000","last_active":"2024-09-13T17:24:11.587735+0000","last_peered":"2024-09-13T17:24:11.587735+0000","last_clean":"2024-09-13T17:24:11.587735+0000","last_became_active":"2024-09-13T17:24:10.163456+0000","last_became_peered":"2024-09-13T17:24:10.163456+0000","last_unstale":"2024-09-13T17:24:11.587735+0000","last_undegraded":"2024-09-13T17:24:11.587735+0000","last_fullsized":"2024-09-13T17:24:11.587735+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:41.116739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154568+0000","last_change":"2024-09-13T17:24:10.597585+0000","last_active":"2024-09-13T17:24:12.154568+0000","last_peered":"2024-09-13T17:24:12.154568+0000","last_clean":"2024-09-13T17:24:12.154568+0000","last_became_active":"2024-09-13T17:24:10.597256+0000","last_became_peered":"2024-09-13T17:24:10.597256+0000","last_unstale":"2024-09-13T17:24:12.154568+0000","last_undegraded":"2024-09-13T17:24:12.154568+0000","last_fullsized":"2024-09-13T17:24:12.154568+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:57.550300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584728+0000","last_change":"2024-09-13T17:24:10.167794+0000","last_active":"2024-09-13T17:24:11.584728+0000","last_peered":"2024-09-13T17:24:11.584728+0000","last_clean":"2024-09-13T17:24:11.584728+0000","last_became_active":"2024-09-13T17:24:10.167533+0000","last_became_peered":"2024-09-13T17:24:10.167533+0000","last_unstale":"2024-09-13T17:24:11.584728+0000","last_undegraded":"2024-09-13T17:24:11.584728+0000","last_fullsized":"2024-09-13T17:24:11.584728+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:47.468554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653225+0000","last_change":"2024-09-13T17:24:10.605798+0000","last_active":"2024-09-13T17:24:11.653225+0000","last_peered":"2024-09-13T17:24:11.653225+0000","last_clean":"2024-09-13T17:24:11.653225+0000","last_became_active":"2024-09-13T17:24:10.605527+0000","last_became_peered":"2024-09-13T17:24:10.605527+0000","last_unstale":"2024-09-13T17:24:11.653225+0000","last_undegraded":"2024-09-13T17:24:11.653225+0000","last_fullsized":"2024-09-13T17:24:11.653225+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:05.071183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153555+0000","last_change":"2024-09-13T17:24:10.161964+0000","last_active":"2024-09-13T17:24:12.153555+0000","last_peered":"2024-09-13T17:24:12.153555+0000","last_clean":"2024-09-13T17:24:12.153555+0000","last_became_active":"2024-09-13T17:24:10.161665+0000","last_became_peered":"2024-09-13T17:24:10.161665+0000","last_unstale":"2024-09-13T17:24:12.153555+0000","last_undegraded":"2024-09-13T17:24:12.153555+0000","last_fullsized":"2024-09-13T17:24:12.153555+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:36.941285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653271+0000","last_change":"2024-09-13T17:24:10.602372+0000","last_active":"2024-09-13T17:24:11.653271+0000","last_peered":"2024-09-13T17:24:11.653271+0000","last_clean":"2024-09-13T17:24:11.653271+0000","last_became_active":"2024-09-13T17:24:10.600175+0000","last_became_peered":"2024-09-13T17:24:10.600175+0000","last_unstale":"2024-09-13T17:24:11.653271+0000","last_undegraded":"2024-09-13T17:24:11.653271+0000","last_fullsized":"2024-09-13T17:24:11.653271+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:57.550300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153627+0000","last_change":"2024-09-13T17:24:10.165872+0000","last_active":"2024-09-13T17:24:12.153627+0000","last_peered":"2024-09-13T17:24:12.153627+0000","last_clean":"2024-09-13T17:24:12.153627+0000","last_became_active":"2024-09-13T17:24:10.164792+0000","last_became_peered":"2024-09-13T17:24:10.164792+0000","last_unstale":"2024-09-13T17:24:12.153627+0000","last_undegraded":"2024-09-13T17:24:12.153627+0000","last_fullsized":"2024-09-13T17:24:12.153627+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:31.026549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653112+0000","last_change":"2024-09-13T17:24:10.602015+0000","last_active":"2024-09-13T17:24:11.653112+0000","last_peered":"2024-09-13T17:24:11.653112+0000","last_clean":"2024-09-13T17:24:11.653112+0000","last_became_active":"2024-09-13T17:24:10.598297+0000","last_became_peered":"2024-09-13T17:24:10.598297+0000","last_unstale":"2024-09-13T17:24:11.653112+0000","last_undegraded":"2024-09-13T17:24:11.653112+0000","last_fullsized":"2024-09-13T17:24:11.653112+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:10.561516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653068+0000","last_change":"2024-09-13T17:24:10.165380+0000","last_active":"2024-09-13T17:24:11.653068+0000","last_peered":"2024-09-13T17:24:11.653068+0000","last_clean":"2024-09-13T17:24:11.653068+0000","last_became_active":"2024-09-13T17:24:10.163630+0000","last_became_peered":"2024-09-13T17:24:10.163630+0000","last_unstale":"2024-09-13T17:24:11.653068+0000","last_undegraded":"2024-09-13T17:24:11.653068+0000","last_fullsized":"2024-09-13T17:24:11.653068+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:25.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153663+0000","last_change":"2024-09-13T17:24:10.589698+0000","last_active":"2024-09-13T17:24:12.153663+0000","last_peered":"2024-09-13T17:24:12.153663+0000","last_clean":"2024-09-13T17:24:12.153663+0000","last_became_active":"2024-09-13T17:24:10.589411+0000","last_became_peered":"2024-09-13T17:24:10.589411+0000","last_unstale":"2024-09-13T17:24:12.153663+0000","last_undegraded":"2024-09-13T17:24:12.153663+0000","last_fullsized":"2024-09-13T17:24:12.153663+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:23:29.924841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584350+0000","last_change":"2024-09-13T17:24:10.171531+0000","last_active":"2024-09-13T17:24:11.584350+0000","last_peered":"2024-09-13T17:24:11.584350+0000","last_clean":"2024-09-13T17:24:11.584350+0000","last_became_active":"2024-09-13T17:24:10.171131+0000","last_became_peered":"2024-09-13T17:24:10.171131+0000","last_unstale":"2024-09-13T17:24:11.584350+0000","last_undegraded":"2024-09-13T17:24:11.584350+0000","last_fullsized":"2024-09-13T17:24:11.584350+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:20.440578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586955+0000","last_change":"2024-09-13T17:24:10.594323+0000","last_active":"2024-09-13T17:24:11.586955+0000","last_peered":"2024-09-13T17:24:11.586955+0000","last_clean":"2024-09-13T17:24:11.586955+0000","last_became_active":"2024-09-13T17:24:10.594028+0000","last_became_peered":"2024-09-13T17:24:10.594028+0000","last_unstale":"2024-09-13T17:24:11.586955+0000","last_undegraded":"2024-09-13T17:24:11.586955+0000","last_fullsized":"2024-09-13T17:24:11.586955+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:00.767994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":347,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T17:24:10.595532+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:24:10.595532+0000","last_peered":"2024-09-13T17:24:10.595532+0000","last_clean":"2024-09-13T17:24:10.595532+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:24:10.595532+0000","last_undegraded":"2024-09-13T17:24:10.595532+0000","last_fullsized":"2024-09-13T17:24:10.595532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70616,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301224,"statfs":{"total":96636764160,"available":96564453376,"internally_reserved":0,"allocated":3543040,"data_stored":1894198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.92800000000000005,"5min":1.008,"15min":2.0880000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.316},"max":{"1min":0.97599999999999998,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.55600000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96899999999999997,"5min":1.161,"15min":2.5030000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.63200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86799999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":1.105,"15min":1.544},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69099999999999995,"15min":0.69699999999999995},"min":{"1min":0.45500000000000002,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":1.0269999999999999,"15min":1.605},"last":0.75}]},{"osd":3,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.443,"15min":0.34999999999999998},"max":{"1min":0.93799999999999994,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.88800000000000001,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.629,"15min":0.65000000000000002},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":1.2170000000000001,"15min":1.234},"last":0.60799999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.47599999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.92200000000000004,"5min":1.135,"15min":1.143},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.60499999999999998,"5min":0.47999999999999998,"15min":0.42599999999999999},"max":{"1min":0.90500000000000003,"5min":1.915,"15min":1.915},"last":0.80800000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.39300000000000002,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.55500000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":1.607,"15min":1.607},"last":0.77000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70040,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301800,"statfs":{"total":96636764160,"available":96565043200,"internally_reserved":0,"allocated":2953216,"data_stored":1303872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":3.6629999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72499999999999998,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.92400000000000004,"5min":0.97399999999999998,"15min":5.9800000000000004},"last":0.86499999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.874,"5min":0.998,"15min":3.7040000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.88500000000000001,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.68400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.88,"5min":0.88,"15min":5.9240000000000004},"last":0.872},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.2629999999999999},"last":0.76600000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.53800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":3.6520000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.006,"5min":1.262,"15min":1.6439999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.626,"15min":0.64200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":4.7750000000000004},"last":0.623}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.68300000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":1.0129999999999999,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":1.089,"5min":1.1559999999999999,"15min":3.609},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70199999999999996,"15min":0.72399999999999998},"min":{"1min":0.504,"5min":0.379,"15min":0.378},"max":{"1min":1.163,"5min":1.163,"15min":4.9669999999999996},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":1.087,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.92600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":2945024,"data_stored":1303760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.76100000000000001},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86399999999999999,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.871,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":1.482,"15min":7.7370000000000001},"last":0.498}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.73499999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":1.0780000000000001},"min":{"1min":0.56100000000000005,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.042,"5min":1.042,"15min":113.363},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.73799999999999999},"min":{"1min":0.63400000000000001,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":1.113,"15min":4.1180000000000003},"last":0.71499999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":1.03,"15min":2.827},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.48199999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.0609999999999999,"5min":1.73,"15min":2.2320000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":1.532,"15min":4.1500000000000004},"last":0.68700000000000006}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.59899999999999998,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.169,"5min":1.363,"15min":2.02},"last":0.82199999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575045,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74140,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297700,"statfs":{"total":96636764160,"available":96560844800,"internally_reserved":0,"allocated":2965504,"data_stored":1305380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.2150000000000001,"15min":0.84099999999999997},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.2010000000000001,"15min":0.875},"min":{"1min":0.46600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93200000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.85299999999999998,"15min":0.72199999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96799999999999997,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.753,"15min":0.68200000000000005},"min":{"1min":0.53800000000000003,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.97599999999999998,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71799999999999997,"15min":0.67900000000000005},"min":{"1min":0.49299999999999999,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.996,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91500000000000004,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.501}]},{"osd":3,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.109,"15min":0.82599999999999996},"min":{"1min":0.61799999999999999,"5min":0.437,"15min":0.378},"max":{"1min":0.95499999999999996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.72599999999999998},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.92600000000000005,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.628}]},{"osd":5,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93799999999999994,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.496,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.88200000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":0.92700000000000005,"5min":2.976,"15min":2.976},"last":0.438},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.73399999999999999},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96399999999999997,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54400000000000004,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.749},"min":{"1min":0.55400000000000005,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.95099999999999996,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.65800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575046,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70052,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301788,"statfs":{"total":96636764160,"available":96565030912,"internally_reserved":0,"allocated":2961408,"data_stored":1305906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":0.624,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":2.96},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.503},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":3.5600000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.379,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.38900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.875,"15min":1.3819999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.1479999999999999,"5min":1.21,"15min":2.8610000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.48199999999999998},"max":{"1min":1.0860000000000001,"5min":1.23,"15min":1.3640000000000001},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.071,"5min":1.323,"15min":1.391},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.499,"15min":0.442},"max":{"1min":1.05,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.81100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77800000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.1559999999999999,"5min":1.218,"15min":1.401},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81699999999999995,"15min":0.78600000000000003},"min":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.175,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78900000000000003},"min":{"1min":0.624,"5min":0.503,"15min":0.42699999999999999},"max":{"1min":1.2130000000000001,"5min":1.27,"15min":2.899},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.208,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.73999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575046,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52044,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319796,"statfs":{"total":96636764160,"available":96583471104,"internally_reserved":0,"allocated":2940928,"data_stored":1302741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47599999999999998,"15min":0.436},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.314},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48499999999999999,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69599999999999995,"5min":1.593,"15min":30.023},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":1.371},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.67000000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":1.4690000000000001},"min":{"1min":0.38500000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.40100000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.443},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74299999999999999,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.53000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.371},"max":{"1min":1.264,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79300000000000004,"15min":0.878},"min":{"1min":0.52300000000000002,"5min":0.48299999999999998,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.623},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82199999999999995,"15min":0.78400000000000003},"min":{"1min":0.61399999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1930000000000001,"5min":6.4800000000000004,"15min":10.664},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81499999999999995,"15min":0.89100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.23,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.94999999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1299999999999999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":1.099},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82999999999999996,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.84799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96280,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275560,"statfs":{"total":96636764160,"available":96538173440,"internally_reserved":0,"allocated":2961408,"data_stored":1304536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.441,"15min":0.46300000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.312},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.63300000000000001,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.443},{"interface":"front","average":{"1min":0.502,"5min":0.52800000000000002,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.74199999999999999,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.28,"15min":6.1630000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.23,"15min":1.78},"last":0.81499999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59599999999999997,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":0.93000000000000005,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75800000000000001,"15min":0.78200000000000003},"min":{"1min":0.57299999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99299999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78300000000000003,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0489999999999999,"5min":1.246,"15min":6.1970000000000001},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.59999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":0.81200000000000006},"min":{"1min":0.61299999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.02,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.88500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74704,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297136,"statfs":{"total":96636764160,"available":96560267264,"internally_reserved":0,"allocated":3534848,"data_stored":1893109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":1.986,"15min":5.5259999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.69999999999999996,"5min":2.371,"15min":5.4889999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.501,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.39100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.878,"15min":4.7549999999999999},"last":0.372},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.432}]},{"osd":4,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80300000000000005,"15min":0.79100000000000004},"min":{"1min":0.629,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.119,"5min":3.8849999999999998,"15min":4.734},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.70899999999999996,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":4.8079999999999998},"last":0.499}]},{"osd":5,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.66500000000000004,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":1.141,"5min":1.141,"15min":4.944},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.78800000000000003,"15min":0.79800000000000004},"min":{"1min":0.65100000000000002,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.246,"5min":1.246,"15min":4.9710000000000001},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.78400000000000003,"15min":0.80800000000000005},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":4.8470000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.65100000000000002,"5min":0.497,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85599999999999998,"15min":0.83399999999999996},"min":{"1min":0.68799999999999994,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.2150000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.67400000000000004,"5min":0.56899999999999995,"15min":0.505},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.9950000000000001},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:24:14.768 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:24:14.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.957+0000 7f7f10a00640 1 Processor -- start 2024-09-13T17:24:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.958+0000 7f7f10a00640 1 -- start start 2024-09-13T17:24:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c0b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f10a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0c0c3850 con 0x7f7f0c159820 2024-09-13T17:24:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f10a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0c15a930 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f10a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c159d20 con 0x7f7f0c159440 2024-09-13T17:24:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f01e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7f0c159820 0x7f7f0c0b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44420/0 (socket says 172.21.15.28:44420) 2024-09-13T17:24:14.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f01e00640 1 -- 172.21.15.28:0/270851571 learned_addr learned my addr 172.21.15.28:0/270851571 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.959+0000 7f7f0a800640 1 --2- 172.21.15.28:0/270851571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c0b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636063214 0 0) 0x7f7f0c0c3850 con 0x7f7f0c159820 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef0005730 con 0x7f7f0c159820 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402865537 0 0) 0x7f7f0c15a930 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f0a800640 1 -- 172.21.15.28:0/270851571 >> v1:172.21.15.189:6789/0 conn(0x7f7f0c0b29f0 legacy=0x7f7f0c15adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f0a800640 1 -- 172.21.15.28:0/270851571 >> v1:172.21.15.28:6789/0 conn(0x7f7f0c159820 legacy=0x7f7f0c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.960+0000 7f7f0a800640 1 -- 172.21.15.28:0/270851571 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0c15a000 con 0x7f7f0c159440 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.961+0000 7f7f0a800640 1 --2- 172.21.15.28:0/270851571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c0b0440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0097d0 tx=0x7f7efc02a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e96297b8ea38768 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.961+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4001670 con 0x7f7f0c159440 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.961+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc02ac90 con 0x7f7f0c159440 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.961+0000 7f7f09e00640 1 -- 172.21.15.28:0/270851571 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc032440 con 0x7f7f0c159440 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 -- 172.21.15.28:0/270851571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 msgr2=0x7f7f0c0b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 --2- 172.21.15.28:0/270851571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c0b0440 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7efc0097d0 tx=0x7f7efc02a200 comp rx=0 tx=0).stop 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 -- 172.21.15.28:0/270851571 shutdown_connections 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 --2- 172.21.15.28:0/270851571 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c0b0440 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 -- 172.21.15.28:0/270851571 >> 172.21.15.28:0/270851571 conn(0x7f7f0c072e40 msgr2=0x7f7f0c073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 -- 172.21.15.28:0/270851571 shutdown_connections 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.962+0000 7f7f10a00640 1 -- 172.21.15.28:0/270851571 wait complete. 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.964+0000 7f7f10a00640 1 Processor -- start 2024-09-13T17:24:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.964+0000 7f7f10a00640 1 -- start start 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c14bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f0c159820 0x7f7f0c14c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c1499a0 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c14c0b0 con 0x7f7f0c159820 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f10a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0c1496c0 con 0x7f7f0c159440 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33220/0 (socket says 172.21.15.28:33220) 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 -- 172.21.15.28:0/2134623245 learned_addr learned my addr 172.21.15.28:0/2134623245 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0b200640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f0c159820 0x7f7f0c14c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f01e00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c14bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 -- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 msgr2=0x7f7f0c14bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c14bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 -- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f0c159820 msgr2=0x7f7f0c14c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f0c159820 0x7f7f0c14c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.965+0000 7f7f0a800640 1 -- 172.21.15.28:0/2134623245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc009400 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f0a800640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f7efc02a7f0 tx=0x7f7efc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc02ae00 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0c156a80 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7efc003c50 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7efc032440 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.966+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f0c157040 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.967+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0c1568e0 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.968+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7efc0325a0 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.969+0000 7f7f03400640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 0x7f7ee806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.969+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7efc0a9d30 con 0x7f7f0c0b29f0 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.969+0000 7f7f01e00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 0x7f7ee806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.970+0000 7f7f01e00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 0x7f7ee806d0c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4001600 tx=0x7f7ef4001570 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:14.973+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7f0c1568e0 con 0x7f7f0c0b29f0 2024-09-13T17:24:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.148+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f0c15a5c0 con 0x7f7ee806ac00 2024-09-13T17:24:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.157+0000 7f7f03400640 1 -- 172.21.15.28:0/2134623245 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289471 (secure 0 0 0) 0x7f7ef400f860 con 0x7f7ee806ac00 2024-09-13T17:24:15.162 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:24:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 msgr2=0x7f7ee806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 0x7f7ee806d0c0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7ef4001600 tx=0x7f7ef4001570 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 msgr2=0x7f7f0c149e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 secure :-1 s=READY pgs=2730 cs=0 l=1 rev1=1 crypto rx=0x7f7efc02a7f0 tx=0x7f7efc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 shutdown_connections 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7ee806ac00 0x7f7ee806d0c0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f0c159820 0x7f7f0c14c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7f0c159440 0x7f7f0c14bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 --2- 172.21.15.28:0/2134623245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f0c0b29f0 0x7f7f0c149e30 unknown :-1 s=CLOSED pgs=2730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 >> 172.21.15.28:0/2134623245 conn(0x7f7f0c072e40 msgr2=0x7f7f0c070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 shutdown_connections 2024-09-13T17:24:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.161+0000 7f7f10a00640 1 -- 172.21.15.28:0/2134623245 wait complete. 2024-09-13T17:24:15.179 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-09-13T17:24:15.129875+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":555820,"kb_used_data":24224,"kb_used_omap":12,"kb_used_meta":531443,"kb_avail":754418900,"statfs":{"total":773094113280,"available":772524953600,"internally_reserved":0,"allocated":24805376,"data_stored":11613502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":544198201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.992347"},"pg_stats":[{"pgid":"33.3f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588365+0000","last_change":"2024-09-13T17:24:10.600258+0000","last_active":"2024-09-13T17:24:11.588365+0000","last_peered":"2024-09-13T17:24:11.588365+0000","last_clean":"2024-09-13T17:24:11.588365+0000","last_became_active":"2024-09-13T17:24:10.599973+0000","last_became_peered":"2024-09-13T17:24:10.599973+0000","last_unstale":"2024-09-13T17:24:11.588365+0000","last_undegraded":"2024-09-13T17:24:11.588365+0000","last_fullsized":"2024-09-13T17:24:11.588365+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:26:08.924474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154450+0000","last_change":"2024-09-13T17:24:10.605961+0000","last_active":"2024-09-13T17:24:12.154450+0000","last_peered":"2024-09-13T17:24:12.154450+0000","last_clean":"2024-09-13T17:24:12.154450+0000","last_became_active":"2024-09-13T17:24:10.605559+0000","last_became_peered":"2024-09-13T17:24:10.605559+0000","last_unstale":"2024-09-13T17:24:12.154450+0000","last_undegraded":"2024-09-13T17:24:12.154450+0000","last_fullsized":"2024-09-13T17:24:12.154450+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:42.124789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154510+0000","last_change":"2024-09-13T17:24:10.601232+0000","last_active":"2024-09-13T17:24:12.154510+0000","last_peered":"2024-09-13T17:24:12.154510+0000","last_clean":"2024-09-13T17:24:12.154510+0000","last_became_active":"2024-09-13T17:24:10.600971+0000","last_became_peered":"2024-09-13T17:24:10.600971+0000","last_unstale":"2024-09-13T17:24:12.154510+0000","last_undegraded":"2024-09-13T17:24:12.154510+0000","last_fullsized":"2024-09-13T17:24:12.154510+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:14.362195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154422+0000","last_change":"2024-09-13T17:24:10.596699+0000","last_active":"2024-09-13T17:24:12.154422+0000","last_peered":"2024-09-13T17:24:12.154422+0000","last_clean":"2024-09-13T17:24:12.154422+0000","last_became_active":"2024-09-13T17:24:10.596434+0000","last_became_peered":"2024-09-13T17:24:10.596434+0000","last_unstale":"2024-09-13T17:24:12.154422+0000","last_undegraded":"2024-09-13T17:24:12.154422+0000","last_fullsized":"2024-09-13T17:24:12.154422+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:00.028016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153796+0000","last_change":"2024-09-13T17:24:10.596600+0000","last_active":"2024-09-13T17:24:12.153796+0000","last_peered":"2024-09-13T17:24:12.153796+0000","last_clean":"2024-09-13T17:24:12.153796+0000","last_became_active":"2024-09-13T17:24:10.596321+0000","last_became_peered":"2024-09-13T17:24:10.596321+0000","last_unstale":"2024-09-13T17:24:12.153796+0000","last_undegraded":"2024-09-13T17:24:12.153796+0000","last_fullsized":"2024-09-13T17:24:12.153796+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:07.414024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153277+0000","last_change":"2024-09-13T17:24:10.602768+0000","last_active":"2024-09-13T17:24:12.153277+0000","last_peered":"2024-09-13T17:24:12.153277+0000","last_clean":"2024-09-13T17:24:12.153277+0000","last_became_active":"2024-09-13T17:24:10.602379+0000","last_became_peered":"2024-09-13T17:24:10.602379+0000","last_unstale":"2024-09-13T17:24:12.153277+0000","last_undegraded":"2024-09-13T17:24:12.153277+0000","last_fullsized":"2024-09-13T17:24:12.153277+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:48.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586951+0000","last_change":"2024-09-13T17:24:10.600787+0000","last_active":"2024-09-13T17:24:11.586951+0000","last_peered":"2024-09-13T17:24:11.586951+0000","last_clean":"2024-09-13T17:24:11.586951+0000","last_became_active":"2024-09-13T17:24:10.600449+0000","last_became_peered":"2024-09-13T17:24:10.600449+0000","last_unstale":"2024-09-13T17:24:11.586951+0000","last_undegraded":"2024-09-13T17:24:11.586951+0000","last_fullsized":"2024-09-13T17:24:11.586951+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:53.636468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586956+0000","last_change":"2024-09-13T17:24:10.609262+0000","last_active":"2024-09-13T17:24:11.586956+0000","last_peered":"2024-09-13T17:24:11.586956+0000","last_clean":"2024-09-13T17:24:11.586956+0000","last_became_active":"2024-09-13T17:24:10.608736+0000","last_became_peered":"2024-09-13T17:24:10.608736+0000","last_unstale":"2024-09-13T17:24:11.586956+0000","last_undegraded":"2024-09-13T17:24:11.586956+0000","last_fullsized":"2024-09-13T17:24:11.586956+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:56:43.230913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154098+0000","last_change":"2024-09-13T17:24:10.594650+0000","last_active":"2024-09-13T17:24:12.154098+0000","last_peered":"2024-09-13T17:24:12.154098+0000","last_clean":"2024-09-13T17:24:12.154098+0000","last_became_active":"2024-09-13T17:24:10.594245+0000","last_became_peered":"2024-09-13T17:24:10.594245+0000","last_unstale":"2024-09-13T17:24:12.154098+0000","last_undegraded":"2024-09-13T17:24:12.154098+0000","last_fullsized":"2024-09-13T17:24:12.154098+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:35:25.126873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587794+0000","last_change":"2024-09-13T17:24:10.609386+0000","last_active":"2024-09-13T17:24:11.587794+0000","last_peered":"2024-09-13T17:24:11.587794+0000","last_clean":"2024-09-13T17:24:11.587794+0000","last_became_active":"2024-09-13T17:24:10.609097+0000","last_became_peered":"2024-09-13T17:24:10.609097+0000","last_unstale":"2024-09-13T17:24:11.587794+0000","last_undegraded":"2024-09-13T17:24:11.587794+0000","last_fullsized":"2024-09-13T17:24:11.587794+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:42.252875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587513+0000","last_change":"2024-09-13T17:24:10.616431+0000","last_active":"2024-09-13T17:24:11.587513+0000","last_peered":"2024-09-13T17:24:11.587513+0000","last_clean":"2024-09-13T17:24:11.587513+0000","last_became_active":"2024-09-13T17:24:10.616028+0000","last_became_peered":"2024-09-13T17:24:10.616028+0000","last_unstale":"2024-09-13T17:24:11.587513+0000","last_undegraded":"2024-09-13T17:24:11.587513+0000","last_fullsized":"2024-09-13T17:24:11.587513+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:33.776565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587970+0000","last_change":"2024-09-13T17:24:10.592092+0000","last_active":"2024-09-13T17:24:11.587970+0000","last_peered":"2024-09-13T17:24:11.587970+0000","last_clean":"2024-09-13T17:24:11.587970+0000","last_became_active":"2024-09-13T17:24:10.591821+0000","last_became_peered":"2024-09-13T17:24:10.591821+0000","last_unstale":"2024-09-13T17:24:11.587970+0000","last_undegraded":"2024-09-13T17:24:11.587970+0000","last_fullsized":"2024-09-13T17:24:11.587970+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:45.474245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653263+0000","last_change":"2024-09-13T17:24:10.613652+0000","last_active":"2024-09-13T17:24:11.653263+0000","last_peered":"2024-09-13T17:24:11.653263+0000","last_clean":"2024-09-13T17:24:11.653263+0000","last_became_active":"2024-09-13T17:24:10.611859+0000","last_became_peered":"2024-09-13T17:24:10.611859+0000","last_unstale":"2024-09-13T17:24:11.653263+0000","last_undegraded":"2024-09-13T17:24:11.653263+0000","last_fullsized":"2024-09-13T17:24:11.653263+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:24.000458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.652964+0000","last_change":"2024-09-13T17:24:10.594151+0000","last_active":"2024-09-13T17:24:11.652964+0000","last_peered":"2024-09-13T17:24:11.652964+0000","last_clean":"2024-09-13T17:24:11.652964+0000","last_became_active":"2024-09-13T17:24:10.593898+0000","last_became_peered":"2024-09-13T17:24:10.593898+0000","last_unstale":"2024-09-13T17:24:11.652964+0000","last_undegraded":"2024-09-13T17:24:11.652964+0000","last_fullsized":"2024-09-13T17:24:11.652964+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:23:29.924841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653009+0000","last_change":"2024-09-13T17:24:10.601302+0000","last_active":"2024-09-13T17:24:11.653009+0000","last_peered":"2024-09-13T17:24:11.653009+0000","last_clean":"2024-09-13T17:24:11.653009+0000","last_became_active":"2024-09-13T17:24:10.601010+0000","last_became_peered":"2024-09-13T17:24:10.601010+0000","last_unstale":"2024-09-13T17:24:11.653009+0000","last_undegraded":"2024-09-13T17:24:11.653009+0000","last_fullsized":"2024-09-13T17:24:11.653009+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:14.897668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153197+0000","last_change":"2024-09-13T17:24:10.615920+0000","last_active":"2024-09-13T17:24:12.153197+0000","last_peered":"2024-09-13T17:24:12.153197+0000","last_clean":"2024-09-13T17:24:12.153197+0000","last_became_active":"2024-09-13T17:24:10.615521+0000","last_became_peered":"2024-09-13T17:24:10.615521+0000","last_unstale":"2024-09-13T17:24:12.153197+0000","last_undegraded":"2024-09-13T17:24:12.153197+0000","last_fullsized":"2024-09-13T17:24:12.153197+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.933226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584875+0000","last_change":"2024-09-13T17:24:10.606893+0000","last_active":"2024-09-13T17:24:11.584875+0000","last_peered":"2024-09-13T17:24:11.584875+0000","last_clean":"2024-09-13T17:24:11.584875+0000","last_became_active":"2024-09-13T17:24:10.606448+0000","last_became_peered":"2024-09-13T17:24:10.606448+0000","last_unstale":"2024-09-13T17:24:11.584875+0000","last_undegraded":"2024-09-13T17:24:11.584875+0000","last_fullsized":"2024-09-13T17:24:11.584875+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:48:45.103302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588085+0000","last_change":"2024-09-13T17:24:10.607018+0000","last_active":"2024-09-13T17:24:11.588085+0000","last_peered":"2024-09-13T17:24:11.588085+0000","last_clean":"2024-09-13T17:24:11.588085+0000","last_became_active":"2024-09-13T17:24:10.606614+0000","last_became_peered":"2024-09-13T17:24:10.606614+0000","last_unstale":"2024-09-13T17:24:11.588085+0000","last_undegraded":"2024-09-13T17:24:11.588085+0000","last_fullsized":"2024-09-13T17:24:11.588085+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:21.640431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587520+0000","last_change":"2024-09-13T17:24:10.609108+0000","last_active":"2024-09-13T17:24:11.587520+0000","last_peered":"2024-09-13T17:24:11.587520+0000","last_clean":"2024-09-13T17:24:11.587520+0000","last_became_active":"2024-09-13T17:24:10.608836+0000","last_became_peered":"2024-09-13T17:24:10.608836+0000","last_unstale":"2024-09-13T17:24:11.587520+0000","last_undegraded":"2024-09-13T17:24:11.587520+0000","last_fullsized":"2024-09-13T17:24:11.587520+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:36.118776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153601+0000","last_change":"2024-09-13T17:24:10.610351+0000","last_active":"2024-09-13T17:24:12.153601+0000","last_peered":"2024-09-13T17:24:12.153601+0000","last_clean":"2024-09-13T17:24:12.153601+0000","last_became_active":"2024-09-13T17:24:10.610035+0000","last_became_peered":"2024-09-13T17:24:10.610035+0000","last_unstale":"2024-09-13T17:24:12.153601+0000","last_undegraded":"2024-09-13T17:24:12.153601+0000","last_fullsized":"2024-09-13T17:24:12.153601+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:21.640431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153484+0000","last_change":"2024-09-13T17:24:10.609011+0000","last_active":"2024-09-13T17:24:12.153484+0000","last_peered":"2024-09-13T17:24:12.153484+0000","last_clean":"2024-09-13T17:24:12.153484+0000","last_became_active":"2024-09-13T17:24:10.608740+0000","last_became_peered":"2024-09-13T17:24:10.608740+0000","last_unstale":"2024-09-13T17:24:12.153484+0000","last_undegraded":"2024-09-13T17:24:12.153484+0000","last_fullsized":"2024-09-13T17:24:12.153484+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:03:16.297416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153476+0000","last_change":"2024-09-13T17:24:10.612570+0000","last_active":"2024-09-13T17:24:12.153476+0000","last_peered":"2024-09-13T17:24:12.153476+0000","last_clean":"2024-09-13T17:24:12.153476+0000","last_became_active":"2024-09-13T17:24:10.612288+0000","last_became_peered":"2024-09-13T17:24:10.612288+0000","last_unstale":"2024-09-13T17:24:12.153476+0000","last_undegraded":"2024-09-13T17:24:12.153476+0000","last_fullsized":"2024-09-13T17:24:12.153476+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:15.501365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587063+0000","last_change":"2024-09-13T17:24:10.598390+0000","last_active":"2024-09-13T17:24:11.587063+0000","last_peered":"2024-09-13T17:24:11.587063+0000","last_clean":"2024-09-13T17:24:11.587063+0000","last_became_active":"2024-09-13T17:24:10.598151+0000","last_became_peered":"2024-09-13T17:24:10.598151+0000","last_unstale":"2024-09-13T17:24:11.587063+0000","last_undegraded":"2024-09-13T17:24:11.587063+0000","last_fullsized":"2024-09-13T17:24:11.587063+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:30.009252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584198+0000","last_change":"2024-09-13T17:24:10.605233+0000","last_active":"2024-09-13T17:24:11.584198+0000","last_peered":"2024-09-13T17:24:11.584198+0000","last_clean":"2024-09-13T17:24:11.584198+0000","last_became_active":"2024-09-13T17:24:10.604454+0000","last_became_peered":"2024-09-13T17:24:10.604454+0000","last_unstale":"2024-09-13T17:24:11.584198+0000","last_undegraded":"2024-09-13T17:24:11.584198+0000","last_fullsized":"2024-09-13T17:24:11.584198+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:07:18.471672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653348+0000","last_change":"2024-09-13T17:24:10.595916+0000","last_active":"2024-09-13T17:24:11.653348+0000","last_peered":"2024-09-13T17:24:11.653348+0000","last_clean":"2024-09-13T17:24:11.653348+0000","last_became_active":"2024-09-13T17:24:10.595623+0000","last_became_peered":"2024-09-13T17:24:10.595623+0000","last_unstale":"2024-09-13T17:24:11.653348+0000","last_undegraded":"2024-09-13T17:24:11.653348+0000","last_fullsized":"2024-09-13T17:24:11.653348+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:00.028016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153815+0000","last_change":"2024-09-13T17:24:10.597754+0000","last_active":"2024-09-13T17:24:12.153815+0000","last_peered":"2024-09-13T17:24:12.153815+0000","last_clean":"2024-09-13T17:24:12.153815+0000","last_became_active":"2024-09-13T17:24:10.597466+0000","last_became_peered":"2024-09-13T17:24:10.597466+0000","last_unstale":"2024-09-13T17:24:12.153815+0000","last_undegraded":"2024-09-13T17:24:12.153815+0000","last_fullsized":"2024-09-13T17:24:12.153815+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:05.071183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587636+0000","last_change":"2024-09-13T17:24:10.593257+0000","last_active":"2024-09-13T17:24:11.587636+0000","last_peered":"2024-09-13T17:24:11.587636+0000","last_clean":"2024-09-13T17:24:11.587636+0000","last_became_active":"2024-09-13T17:24:10.592990+0000","last_became_peered":"2024-09-13T17:24:10.592990+0000","last_unstale":"2024-09-13T17:24:11.587636+0000","last_undegraded":"2024-09-13T17:24:11.587636+0000","last_fullsized":"2024-09-13T17:24:11.587636+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:52:10.933226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653456+0000","last_change":"2024-09-13T17:24:10.601276+0000","last_active":"2024-09-13T17:24:11.653456+0000","last_peered":"2024-09-13T17:24:11.653456+0000","last_clean":"2024-09-13T17:24:11.653456+0000","last_became_active":"2024-09-13T17:24:10.598273+0000","last_became_peered":"2024-09-13T17:24:10.598273+0000","last_unstale":"2024-09-13T17:24:11.653456+0000","last_undegraded":"2024-09-13T17:24:11.653456+0000","last_fullsized":"2024-09-13T17:24:11.653456+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:16:29.406341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584619+0000","last_change":"2024-09-13T17:24:10.614554+0000","last_active":"2024-09-13T17:24:11.584619+0000","last_peered":"2024-09-13T17:24:11.584619+0000","last_clean":"2024-09-13T17:24:11.584619+0000","last_became_active":"2024-09-13T17:24:10.614283+0000","last_became_peered":"2024-09-13T17:24:10.614283+0000","last_unstale":"2024-09-13T17:24:11.584619+0000","last_undegraded":"2024-09-13T17:24:11.584619+0000","last_fullsized":"2024-09-13T17:24:11.584619+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:56.093781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653111+0000","last_change":"2024-09-13T17:24:10.594132+0000","last_active":"2024-09-13T17:24:11.653111+0000","last_peered":"2024-09-13T17:24:11.653111+0000","last_clean":"2024-09-13T17:24:11.653111+0000","last_became_active":"2024-09-13T17:24:10.593876+0000","last_became_peered":"2024-09-13T17:24:10.593876+0000","last_unstale":"2024-09-13T17:24:11.653111+0000","last_undegraded":"2024-09-13T17:24:11.653111+0000","last_fullsized":"2024-09-13T17:24:11.653111+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:07.120077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587061+0000","last_change":"2024-09-13T17:24:10.597332+0000","last_active":"2024-09-13T17:24:11.587061+0000","last_peered":"2024-09-13T17:24:11.587061+0000","last_clean":"2024-09-13T17:24:11.587061+0000","last_became_active":"2024-09-13T17:24:10.596977+0000","last_became_peered":"2024-09-13T17:24:10.596977+0000","last_unstale":"2024-09-13T17:24:11.587061+0000","last_undegraded":"2024-09-13T17:24:11.587061+0000","last_fullsized":"2024-09-13T17:24:11.587061+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:00.767994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587895+0000","last_change":"2024-09-13T17:24:10.598328+0000","last_active":"2024-09-13T17:24:11.587895+0000","last_peered":"2024-09-13T17:24:11.587895+0000","last_clean":"2024-09-13T17:24:11.587895+0000","last_became_active":"2024-09-13T17:24:10.598059+0000","last_became_peered":"2024-09-13T17:24:10.598059+0000","last_unstale":"2024-09-13T17:24:11.587895+0000","last_undegraded":"2024-09-13T17:24:11.587895+0000","last_fullsized":"2024-09-13T17:24:11.587895+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.168528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"232'4","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584983+0000","last_change":"2024-09-13T17:24:10.166081+0000","last_active":"2024-09-13T17:24:11.584983+0000","last_peered":"2024-09-13T17:24:11.584983+0000","last_clean":"2024-09-13T17:24:11.584983+0000","last_became_active":"2024-09-13T17:24:10.164208+0000","last_became_peered":"2024-09-13T17:24:10.164208+0000","last_unstale":"2024-09-13T17:24:11.584983+0000","last_undegraded":"2024-09-13T17:24:11.584983+0000","last_fullsized":"2024-09-13T17:24:11.584983+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:35:47.844554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653369+0000","last_change":"2024-09-13T17:24:10.602308+0000","last_active":"2024-09-13T17:24:11.653369+0000","last_peered":"2024-09-13T17:24:11.653369+0000","last_clean":"2024-09-13T17:24:11.653369+0000","last_became_active":"2024-09-13T17:24:10.600996+0000","last_became_peered":"2024-09-13T17:24:10.600996+0000","last_unstale":"2024-09-13T17:24:11.653369+0000","last_undegraded":"2024-09-13T17:24:11.653369+0000","last_fullsized":"2024-09-13T17:24:11.653369+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:23.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653235+0000","last_change":"2024-09-13T17:24:10.164589+0000","last_active":"2024-09-13T17:24:11.653235+0000","last_peered":"2024-09-13T17:24:11.653235+0000","last_clean":"2024-09-13T17:24:11.653235+0000","last_became_active":"2024-09-13T17:24:10.164313+0000","last_became_peered":"2024-09-13T17:24:10.164313+0000","last_unstale":"2024-09-13T17:24:11.653235+0000","last_undegraded":"2024-09-13T17:24:11.653235+0000","last_fullsized":"2024-09-13T17:24:11.653235+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:57.577314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587925+0000","last_change":"2024-09-13T17:24:10.606967+0000","last_active":"2024-09-13T17:24:11.587925+0000","last_peered":"2024-09-13T17:24:11.587925+0000","last_clean":"2024-09-13T17:24:11.587925+0000","last_became_active":"2024-09-13T17:24:10.606674+0000","last_became_peered":"2024-09-13T17:24:10.606674+0000","last_unstale":"2024-09-13T17:24:11.587925+0000","last_undegraded":"2024-09-13T17:24:11.587925+0000","last_fullsized":"2024-09-13T17:24:11.587925+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:45.474245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153901+0000","last_change":"2024-09-13T17:24:10.164923+0000","last_active":"2024-09-13T17:24:12.153901+0000","last_peered":"2024-09-13T17:24:12.153901+0000","last_clean":"2024-09-13T17:24:12.153901+0000","last_became_active":"2024-09-13T17:24:10.164636+0000","last_became_peered":"2024-09-13T17:24:10.164636+0000","last_unstale":"2024-09-13T17:24:12.153901+0000","last_undegraded":"2024-09-13T17:24:12.153901+0000","last_fullsized":"2024-09-13T17:24:12.153901+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:29.403573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588083+0000","last_change":"2024-09-13T17:24:10.597639+0000","last_active":"2024-09-13T17:24:11.588083+0000","last_peered":"2024-09-13T17:24:11.588083+0000","last_clean":"2024-09-13T17:24:11.588083+0000","last_became_active":"2024-09-13T17:24:10.596401+0000","last_became_peered":"2024-09-13T17:24:10.596401+0000","last_unstale":"2024-09-13T17:24:11.588083+0000","last_undegraded":"2024-09-13T17:24:11.588083+0000","last_fullsized":"2024-09-13T17:24:11.588083+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:12:33.776565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153689+0000","last_change":"2024-09-13T17:24:10.162847+0000","last_active":"2024-09-13T17:24:12.153689+0000","last_peered":"2024-09-13T17:24:12.153689+0000","last_clean":"2024-09-13T17:24:12.153689+0000","last_became_active":"2024-09-13T17:24:10.161943+0000","last_became_peered":"2024-09-13T17:24:10.161943+0000","last_unstale":"2024-09-13T17:24:12.153689+0000","last_undegraded":"2024-09-13T17:24:12.153689+0000","last_fullsized":"2024-09-13T17:24:12.153689+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:42.799697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153374+0000","last_change":"2024-09-13T17:24:10.608931+0000","last_active":"2024-09-13T17:24:12.153374+0000","last_peered":"2024-09-13T17:24:12.153374+0000","last_clean":"2024-09-13T17:24:12.153374+0000","last_became_active":"2024-09-13T17:24:10.608646+0000","last_became_peered":"2024-09-13T17:24:10.608646+0000","last_unstale":"2024-09-13T17:24:12.153374+0000","last_undegraded":"2024-09-13T17:24:12.153374+0000","last_fullsized":"2024-09-13T17:24:12.153374+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:47:08.724701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587341+0000","last_change":"2024-09-13T17:24:10.166535+0000","last_active":"2024-09-13T17:24:11.587341+0000","last_peered":"2024-09-13T17:24:11.587341+0000","last_clean":"2024-09-13T17:24:11.587341+0000","last_became_active":"2024-09-13T17:24:10.165741+0000","last_became_peered":"2024-09-13T17:24:10.165741+0000","last_unstale":"2024-09-13T17:24:11.587341+0000","last_undegraded":"2024-09-13T17:24:11.587341+0000","last_fullsized":"2024-09-13T17:24:11.587341+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:39:24.643100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153553+0000","last_change":"2024-09-13T17:24:10.604093+0000","last_active":"2024-09-13T17:24:12.153553+0000","last_peered":"2024-09-13T17:24:12.153553+0000","last_clean":"2024-09-13T17:24:12.153553+0000","last_became_active":"2024-09-13T17:24:10.603857+0000","last_became_peered":"2024-09-13T17:24:10.603857+0000","last_unstale":"2024-09-13T17:24:12.153553+0000","last_undegraded":"2024-09-13T17:24:12.153553+0000","last_fullsized":"2024-09-13T17:24:12.153553+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:03.168528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584763+0000","last_change":"2024-09-13T17:24:10.153073+0000","last_active":"2024-09-13T17:24:11.584763+0000","last_peered":"2024-09-13T17:24:11.584763+0000","last_clean":"2024-09-13T17:24:11.584763+0000","last_became_active":"2024-09-13T17:24:10.152795+0000","last_became_peered":"2024-09-13T17:24:10.152795+0000","last_unstale":"2024-09-13T17:24:11.584763+0000","last_undegraded":"2024-09-13T17:24:11.584763+0000","last_fullsized":"2024-09-13T17:24:11.584763+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:36.005955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153298+0000","last_change":"2024-09-13T17:24:10.602855+0000","last_active":"2024-09-13T17:24:12.153298+0000","last_peered":"2024-09-13T17:24:12.153298+0000","last_clean":"2024-09-13T17:24:12.153298+0000","last_became_active":"2024-09-13T17:24:10.602604+0000","last_became_peered":"2024-09-13T17:24:10.602604+0000","last_unstale":"2024-09-13T17:24:12.153298+0000","last_undegraded":"2024-09-13T17:24:12.153298+0000","last_fullsized":"2024-09-13T17:24:12.153298+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:36.118776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153740+0000","last_change":"2024-09-13T17:24:10.168440+0000","last_active":"2024-09-13T17:24:12.153740+0000","last_peered":"2024-09-13T17:24:12.153740+0000","last_clean":"2024-09-13T17:24:12.153740+0000","last_became_active":"2024-09-13T17:24:10.168161+0000","last_became_peered":"2024-09-13T17:24:10.168161+0000","last_unstale":"2024-09-13T17:24:12.153740+0000","last_undegraded":"2024-09-13T17:24:12.153740+0000","last_fullsized":"2024-09-13T17:24:12.153740+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:53:22.770548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587270+0000","last_change":"2024-09-13T17:24:10.600654+0000","last_active":"2024-09-13T17:24:11.587270+0000","last_peered":"2024-09-13T17:24:11.587270+0000","last_clean":"2024-09-13T17:24:11.587270+0000","last_became_active":"2024-09-13T17:24:10.600383+0000","last_became_peered":"2024-09-13T17:24:10.600383+0000","last_unstale":"2024-09-13T17:24:11.587270+0000","last_undegraded":"2024-09-13T17:24:11.587270+0000","last_fullsized":"2024-09-13T17:24:11.587270+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:32:26.579180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587043+0000","last_change":"2024-09-13T17:24:10.171987+0000","last_active":"2024-09-13T17:24:11.587043+0000","last_peered":"2024-09-13T17:24:11.587043+0000","last_clean":"2024-09-13T17:24:11.587043+0000","last_became_active":"2024-09-13T17:24:10.171717+0000","last_became_peered":"2024-09-13T17:24:10.171717+0000","last_unstale":"2024-09-13T17:24:11.587043+0000","last_undegraded":"2024-09-13T17:24:11.587043+0000","last_fullsized":"2024-09-13T17:24:11.587043+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:41.259957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588021+0000","last_change":"2024-09-13T17:24:10.607207+0000","last_active":"2024-09-13T17:24:11.588021+0000","last_peered":"2024-09-13T17:24:11.588021+0000","last_clean":"2024-09-13T17:24:11.588021+0000","last_became_active":"2024-09-13T17:24:10.606649+0000","last_became_peered":"2024-09-13T17:24:10.606649+0000","last_unstale":"2024-09-13T17:24:11.588021+0000","last_undegraded":"2024-09-13T17:24:11.588021+0000","last_fullsized":"2024-09-13T17:24:11.588021+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:54:48.624555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584969+0000","last_change":"2024-09-13T17:24:10.168009+0000","last_active":"2024-09-13T17:24:11.584969+0000","last_peered":"2024-09-13T17:24:11.584969+0000","last_clean":"2024-09-13T17:24:11.584969+0000","last_became_active":"2024-09-13T17:24:10.167731+0000","last_became_peered":"2024-09-13T17:24:10.167731+0000","last_unstale":"2024-09-13T17:24:11.584969+0000","last_undegraded":"2024-09-13T17:24:11.584969+0000","last_fullsized":"2024-09-13T17:24:11.584969+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:28:35.999463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584897+0000","last_change":"2024-09-13T17:24:10.614744+0000","last_active":"2024-09-13T17:24:11.584897+0000","last_peered":"2024-09-13T17:24:11.584897+0000","last_clean":"2024-09-13T17:24:11.584897+0000","last_became_active":"2024-09-13T17:24:10.614432+0000","last_became_peered":"2024-09-13T17:24:10.614432+0000","last_unstale":"2024-09-13T17:24:11.584897+0000","last_undegraded":"2024-09-13T17:24:11.584897+0000","last_fullsized":"2024-09-13T17:24:11.584897+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:48.823884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153558+0000","last_change":"2024-09-13T17:24:10.170053+0000","last_active":"2024-09-13T17:24:12.153558+0000","last_peered":"2024-09-13T17:24:12.153558+0000","last_clean":"2024-09-13T17:24:12.153558+0000","last_became_active":"2024-09-13T17:24:10.169769+0000","last_became_peered":"2024-09-13T17:24:10.169769+0000","last_unstale":"2024-09-13T17:24:12.153558+0000","last_undegraded":"2024-09-13T17:24:12.153558+0000","last_fullsized":"2024-09-13T17:24:12.153558+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:54:20.173529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653048+0000","last_change":"2024-09-13T17:24:10.593000+0000","last_active":"2024-09-13T17:24:11.653048+0000","last_peered":"2024-09-13T17:24:11.653048+0000","last_clean":"2024-09-13T17:24:11.653048+0000","last_became_active":"2024-09-13T17:24:10.592762+0000","last_became_peered":"2024-09-13T17:24:10.592762+0000","last_unstale":"2024-09-13T17:24:11.653048+0000","last_undegraded":"2024-09-13T17:24:11.653048+0000","last_fullsized":"2024-09-13T17:24:11.653048+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:58.029629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"232'3","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153949+0000","last_change":"2024-09-13T17:24:10.170259+0000","last_active":"2024-09-13T17:24:12.153949+0000","last_peered":"2024-09-13T17:24:12.153949+0000","last_clean":"2024-09-13T17:24:12.153949+0000","last_became_active":"2024-09-13T17:24:10.170014+0000","last_became_peered":"2024-09-13T17:24:10.170014+0000","last_unstale":"2024-09-13T17:24:12.153949+0000","last_undegraded":"2024-09-13T17:24:12.153949+0000","last_fullsized":"2024-09-13T17:24:12.153949+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:55:32.986051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584695+0000","last_change":"2024-09-13T17:24:10.617281+0000","last_active":"2024-09-13T17:24:11.584695+0000","last_peered":"2024-09-13T17:24:11.584695+0000","last_clean":"2024-09-13T17:24:11.584695+0000","last_became_active":"2024-09-13T17:24:10.616691+0000","last_became_peered":"2024-09-13T17:24:10.616691+0000","last_unstale":"2024-09-13T17:24:11.584695+0000","last_undegraded":"2024-09-13T17:24:11.584695+0000","last_fullsized":"2024-09-13T17:24:11.584695+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:34:56.815950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"232'6","reported_seq":17,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584698+0000","last_change":"2024-09-13T17:24:10.162890+0000","last_active":"2024-09-13T17:24:11.584698+0000","last_peered":"2024-09-13T17:24:11.584698+0000","last_clean":"2024-09-13T17:24:11.584698+0000","last_became_active":"2024-09-13T17:24:10.160626+0000","last_became_peered":"2024-09-13T17:24:10.160626+0000","last_unstale":"2024-09-13T17:24:11.584698+0000","last_undegraded":"2024-09-13T17:24:11.584698+0000","last_fullsized":"2024-09-13T17:24:11.584698+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:58.971790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584574+0000","last_change":"2024-09-13T17:24:10.603088+0000","last_active":"2024-09-13T17:24:11.584574+0000","last_peered":"2024-09-13T17:24:11.584574+0000","last_clean":"2024-09-13T17:24:11.584574+0000","last_became_active":"2024-09-13T17:24:10.602839+0000","last_became_peered":"2024-09-13T17:24:10.602839+0000","last_unstale":"2024-09-13T17:24:11.584574+0000","last_undegraded":"2024-09-13T17:24:11.584574+0000","last_fullsized":"2024-09-13T17:24:11.584574+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:02.695710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587883+0000","last_change":"2024-09-13T17:24:10.165606+0000","last_active":"2024-09-13T17:24:11.587883+0000","last_peered":"2024-09-13T17:24:11.587883+0000","last_clean":"2024-09-13T17:24:11.587883+0000","last_became_active":"2024-09-13T17:24:10.165288+0000","last_became_peered":"2024-09-13T17:24:10.165288+0000","last_unstale":"2024-09-13T17:24:11.587883+0000","last_undegraded":"2024-09-13T17:24:11.587883+0000","last_fullsized":"2024-09-13T17:24:11.587883+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:36.533122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153652+0000","last_change":"2024-09-13T17:24:10.604730+0000","last_active":"2024-09-13T17:24:12.153652+0000","last_peered":"2024-09-13T17:24:12.153652+0000","last_clean":"2024-09-13T17:24:12.153652+0000","last_became_active":"2024-09-13T17:24:10.604467+0000","last_became_peered":"2024-09-13T17:24:10.604467+0000","last_unstale":"2024-09-13T17:24:12.153652+0000","last_undegraded":"2024-09-13T17:24:12.153652+0000","last_fullsized":"2024-09-13T17:24:12.153652+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:39:42.252875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153556+0000","last_change":"2024-09-13T17:24:10.162284+0000","last_active":"2024-09-13T17:24:12.153556+0000","last_peered":"2024-09-13T17:24:12.153556+0000","last_clean":"2024-09-13T17:24:12.153556+0000","last_became_active":"2024-09-13T17:24:10.162038+0000","last_became_peered":"2024-09-13T17:24:10.162038+0000","last_unstale":"2024-09-13T17:24:12.153556+0000","last_undegraded":"2024-09-13T17:24:12.153556+0000","last_fullsized":"2024-09-13T17:24:12.153556+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:22:01.002427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584270+0000","last_change":"2024-09-13T17:24:10.599364+0000","last_active":"2024-09-13T17:24:11.584270+0000","last_peered":"2024-09-13T17:24:11.584270+0000","last_clean":"2024-09-13T17:24:11.584270+0000","last_became_active":"2024-09-13T17:24:10.599050+0000","last_became_peered":"2024-09-13T17:24:10.599050+0000","last_unstale":"2024-09-13T17:24:11.584270+0000","last_undegraded":"2024-09-13T17:24:11.584270+0000","last_fullsized":"2024-09-13T17:24:11.584270+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:25.354254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154264+0000","last_change":"2024-09-13T17:24:10.162253+0000","last_active":"2024-09-13T17:24:12.154264+0000","last_peered":"2024-09-13T17:24:12.154264+0000","last_clean":"2024-09-13T17:24:12.154264+0000","last_became_active":"2024-09-13T17:24:10.161951+0000","last_became_peered":"2024-09-13T17:24:10.161951+0000","last_unstale":"2024-09-13T17:24:12.154264+0000","last_undegraded":"2024-09-13T17:24:12.154264+0000","last_fullsized":"2024-09-13T17:24:12.154264+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:57:58.963914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584178+0000","last_change":"2024-09-13T17:24:10.599667+0000","last_active":"2024-09-13T17:24:11.584178+0000","last_peered":"2024-09-13T17:24:11.584178+0000","last_clean":"2024-09-13T17:24:11.584178+0000","last_became_active":"2024-09-13T17:24:10.599421+0000","last_became_peered":"2024-09-13T17:24:10.599421+0000","last_unstale":"2024-09-13T17:24:11.584178+0000","last_undegraded":"2024-09-13T17:24:11.584178+0000","last_fullsized":"2024-09-13T17:24:11.584178+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:30:41.441391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"232'8","reported_seq":19,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153390+0000","last_change":"2024-09-13T17:24:10.159134+0000","last_active":"2024-09-13T17:24:12.153390+0000","last_peered":"2024-09-13T17:24:12.153390+0000","last_clean":"2024-09-13T17:24:12.153390+0000","last_became_active":"2024-09-13T17:24:10.158699+0000","last_became_peered":"2024-09-13T17:24:10.158699+0000","last_unstale":"2024-09-13T17:24:12.153390+0000","last_undegraded":"2024-09-13T17:24:12.153390+0000","last_fullsized":"2024-09-13T17:24:12.153390+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:45.093617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653059+0000","last_change":"2024-09-13T17:24:10.606193+0000","last_active":"2024-09-13T17:24:11.653059+0000","last_peered":"2024-09-13T17:24:11.653059+0000","last_clean":"2024-09-13T17:24:11.653059+0000","last_became_active":"2024-09-13T17:24:10.605633+0000","last_became_peered":"2024-09-13T17:24:10.605633+0000","last_unstale":"2024-09-13T17:24:11.653059+0000","last_undegraded":"2024-09-13T17:24:11.653059+0000","last_fullsized":"2024-09-13T17:24:11.653059+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:52:07.414024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.588369+0000","last_change":"2024-09-13T17:24:10.168701+0000","last_active":"2024-09-13T17:24:11.588369+0000","last_peered":"2024-09-13T17:24:11.588369+0000","last_clean":"2024-09-13T17:24:11.588369+0000","last_became_active":"2024-09-13T17:24:10.167344+0000","last_became_peered":"2024-09-13T17:24:10.167344+0000","last_unstale":"2024-09-13T17:24:11.588369+0000","last_undegraded":"2024-09-13T17:24:11.588369+0000","last_fullsized":"2024-09-13T17:24:11.588369+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:09.135618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154293+0000","last_change":"2024-09-13T17:24:10.612462+0000","last_active":"2024-09-13T17:24:12.154293+0000","last_peered":"2024-09-13T17:24:12.154293+0000","last_clean":"2024-09-13T17:24:12.154293+0000","last_became_active":"2024-09-13T17:24:10.612183+0000","last_became_peered":"2024-09-13T17:24:10.612183+0000","last_unstale":"2024-09-13T17:24:12.154293+0000","last_undegraded":"2024-09-13T17:24:12.154293+0000","last_fullsized":"2024-09-13T17:24:12.154293+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:42.268007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153650+0000","last_change":"2024-09-13T17:24:10.162625+0000","last_active":"2024-09-13T17:24:12.153650+0000","last_peered":"2024-09-13T17:24:12.153650+0000","last_clean":"2024-09-13T17:24:12.153650+0000","last_became_active":"2024-09-13T17:24:10.162310+0000","last_became_peered":"2024-09-13T17:24:10.162310+0000","last_unstale":"2024-09-13T17:24:12.153650+0000","last_undegraded":"2024-09-13T17:24:12.153650+0000","last_fullsized":"2024-09-13T17:24:12.153650+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:55:00.385884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653154+0000","last_change":"2024-09-13T17:24:10.601820+0000","last_active":"2024-09-13T17:24:11.653154+0000","last_peered":"2024-09-13T17:24:11.653154+0000","last_clean":"2024-09-13T17:24:11.653154+0000","last_became_active":"2024-09-13T17:24:10.601536+0000","last_became_peered":"2024-09-13T17:24:10.601536+0000","last_unstale":"2024-09-13T17:24:11.653154+0000","last_undegraded":"2024-09-13T17:24:11.653154+0000","last_fullsized":"2024-09-13T17:24:11.653154+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:30:13.970855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"232'1","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154343+0000","last_change":"2024-09-13T17:24:10.164555+0000","last_active":"2024-09-13T17:24:12.154343+0000","last_peered":"2024-09-13T17:24:12.154343+0000","last_clean":"2024-09-13T17:24:12.154343+0000","last_became_active":"2024-09-13T17:24:10.164194+0000","last_became_peered":"2024-09-13T17:24:10.164194+0000","last_unstale":"2024-09-13T17:24:12.154343+0000","last_undegraded":"2024-09-13T17:24:12.154343+0000","last_fullsized":"2024-09-13T17:24:12.154343+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:18.484595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154548+0000","last_change":"2024-09-13T17:24:10.600664+0000","last_active":"2024-09-13T17:24:12.154548+0000","last_peered":"2024-09-13T17:24:12.154548+0000","last_clean":"2024-09-13T17:24:12.154548+0000","last_became_active":"2024-09-13T17:24:10.597527+0000","last_became_peered":"2024-09-13T17:24:10.597527+0000","last_unstale":"2024-09-13T17:24:12.154548+0000","last_undegraded":"2024-09-13T17:24:12.154548+0000","last_fullsized":"2024-09-13T17:24:12.154548+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:58.029629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653284+0000","last_change":"2024-09-13T17:24:10.170331+0000","last_active":"2024-09-13T17:24:11.653284+0000","last_peered":"2024-09-13T17:24:11.653284+0000","last_clean":"2024-09-13T17:24:11.653284+0000","last_became_active":"2024-09-13T17:24:10.169843+0000","last_became_peered":"2024-09-13T17:24:10.169843+0000","last_unstale":"2024-09-13T17:24:11.653284+0000","last_undegraded":"2024-09-13T17:24:11.653284+0000","last_fullsized":"2024-09-13T17:24:11.653284+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:49:14.619458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153303+0000","last_change":"2024-09-13T17:24:10.602619+0000","last_active":"2024-09-13T17:24:12.153303+0000","last_peered":"2024-09-13T17:24:12.153303+0000","last_clean":"2024-09-13T17:24:12.153303+0000","last_became_active":"2024-09-13T17:24:10.602056+0000","last_became_peered":"2024-09-13T17:24:10.602056+0000","last_unstale":"2024-09-13T17:24:12.153303+0000","last_undegraded":"2024-09-13T17:24:12.153303+0000","last_fullsized":"2024-09-13T17:24:12.153303+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:37.541172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153692+0000","last_change":"2024-09-13T17:24:10.157895+0000","last_active":"2024-09-13T17:24:12.153692+0000","last_peered":"2024-09-13T17:24:12.153692+0000","last_clean":"2024-09-13T17:24:12.153692+0000","last_became_active":"2024-09-13T17:24:10.157277+0000","last_became_peered":"2024-09-13T17:24:10.157277+0000","last_unstale":"2024-09-13T17:24:12.153692+0000","last_undegraded":"2024-09-13T17:24:12.153692+0000","last_fullsized":"2024-09-13T17:24:12.153692+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:51:21.568488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153732+0000","last_change":"2024-09-13T17:24:10.611408+0000","last_active":"2024-09-13T17:24:12.153732+0000","last_peered":"2024-09-13T17:24:12.153732+0000","last_clean":"2024-09-13T17:24:12.153732+0000","last_became_active":"2024-09-13T17:24:10.611166+0000","last_became_peered":"2024-09-13T17:24:10.611166+0000","last_unstale":"2024-09-13T17:24:12.153732+0000","last_undegraded":"2024-09-13T17:24:12.153732+0000","last_fullsized":"2024-09-13T17:24:12.153732+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:30.009252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587824+0000","last_change":"2024-09-13T17:24:10.165683+0000","last_active":"2024-09-13T17:24:11.587824+0000","last_peered":"2024-09-13T17:24:11.587824+0000","last_clean":"2024-09-13T17:24:11.587824+0000","last_became_active":"2024-09-13T17:24:10.165437+0000","last_became_peered":"2024-09-13T17:24:10.165437+0000","last_unstale":"2024-09-13T17:24:11.587824+0000","last_undegraded":"2024-09-13T17:24:11.587824+0000","last_fullsized":"2024-09-13T17:24:11.587824+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:36:20.632382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584331+0000","last_change":"2024-09-13T17:24:10.600609+0000","last_active":"2024-09-13T17:24:11.584331+0000","last_peered":"2024-09-13T17:24:11.584331+0000","last_clean":"2024-09-13T17:24:11.584331+0000","last_became_active":"2024-09-13T17:24:10.600301+0000","last_became_peered":"2024-09-13T17:24:10.600301+0000","last_unstale":"2024-09-13T17:24:11.584331+0000","last_undegraded":"2024-09-13T17:24:11.584331+0000","last_fullsized":"2024-09-13T17:24:11.584331+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:00.894395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154328+0000","last_change":"2024-09-13T17:24:10.164654+0000","last_active":"2024-09-13T17:24:12.154328+0000","last_peered":"2024-09-13T17:24:12.154328+0000","last_clean":"2024-09-13T17:24:12.154328+0000","last_became_active":"2024-09-13T17:24:10.164398+0000","last_became_peered":"2024-09-13T17:24:10.164398+0000","last_unstale":"2024-09-13T17:24:12.154328+0000","last_undegraded":"2024-09-13T17:24:12.154328+0000","last_fullsized":"2024-09-13T17:24:12.154328+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:00.652836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153373+0000","last_change":"2024-09-13T17:24:10.597555+0000","last_active":"2024-09-13T17:24:12.153373+0000","last_peered":"2024-09-13T17:24:12.153373+0000","last_clean":"2024-09-13T17:24:12.153373+0000","last_became_active":"2024-09-13T17:24:10.597334+0000","last_became_peered":"2024-09-13T17:24:10.597334+0000","last_unstale":"2024-09-13T17:24:12.153373+0000","last_undegraded":"2024-09-13T17:24:12.153373+0000","last_fullsized":"2024-09-13T17:24:12.153373+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:53.011193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587052+0000","last_change":"2024-09-13T17:24:10.160038+0000","last_active":"2024-09-13T17:24:11.587052+0000","last_peered":"2024-09-13T17:24:11.587052+0000","last_clean":"2024-09-13T17:24:11.587052+0000","last_became_active":"2024-09-13T17:24:10.159232+0000","last_became_peered":"2024-09-13T17:24:10.159232+0000","last_unstale":"2024-09-13T17:24:11.587052+0000","last_undegraded":"2024-09-13T17:24:11.587052+0000","last_fullsized":"2024-09-13T17:24:11.587052+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:26:52.628418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653301+0000","last_change":"2024-09-13T17:24:10.605898+0000","last_active":"2024-09-13T17:24:11.653301+0000","last_peered":"2024-09-13T17:24:11.653301+0000","last_clean":"2024-09-13T17:24:11.653301+0000","last_became_active":"2024-09-13T17:24:10.605270+0000","last_became_peered":"2024-09-13T17:24:10.605270+0000","last_unstale":"2024-09-13T17:24:11.653301+0000","last_undegraded":"2024-09-13T17:24:11.653301+0000","last_fullsized":"2024-09-13T17:24:11.653301+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:14.362195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"232'2","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653428+0000","last_change":"2024-09-13T17:24:10.170300+0000","last_active":"2024-09-13T17:24:11.653428+0000","last_peered":"2024-09-13T17:24:11.653428+0000","last_clean":"2024-09-13T17:24:11.653428+0000","last_became_active":"2024-09-13T17:24:10.168746+0000","last_became_peered":"2024-09-13T17:24:10.168746+0000","last_unstale":"2024-09-13T17:24:11.653428+0000","last_undegraded":"2024-09-13T17:24:11.653428+0000","last_fullsized":"2024-09-13T17:24:11.653428+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:46:20.275319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584924+0000","last_change":"2024-09-13T17:24:10.600902+0000","last_active":"2024-09-13T17:24:11.584924+0000","last_peered":"2024-09-13T17:24:11.584924+0000","last_clean":"2024-09-13T17:24:11.584924+0000","last_became_active":"2024-09-13T17:24:10.600071+0000","last_became_peered":"2024-09-13T17:24:10.600071+0000","last_unstale":"2024-09-13T17:24:11.584924+0000","last_undegraded":"2024-09-13T17:24:11.584924+0000","last_fullsized":"2024-09-13T17:24:11.584924+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:12.019907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153826+0000","last_change":"2024-09-13T17:24:10.169837+0000","last_active":"2024-09-13T17:24:12.153826+0000","last_peered":"2024-09-13T17:24:12.153826+0000","last_clean":"2024-09-13T17:24:12.153826+0000","last_became_active":"2024-09-13T17:24:10.169558+0000","last_became_peered":"2024-09-13T17:24:10.169558+0000","last_unstale":"2024-09-13T17:24:12.153826+0000","last_undegraded":"2024-09-13T17:24:12.153826+0000","last_fullsized":"2024-09-13T17:24:12.153826+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:36.655203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153754+0000","last_change":"2024-09-13T17:24:10.598827+0000","last_active":"2024-09-13T17:24:12.153754+0000","last_peered":"2024-09-13T17:24:12.153754+0000","last_clean":"2024-09-13T17:24:12.153754+0000","last_became_active":"2024-09-13T17:24:10.598569+0000","last_became_peered":"2024-09-13T17:24:10.598569+0000","last_unstale":"2024-09-13T17:24:12.153754+0000","last_undegraded":"2024-09-13T17:24:12.153754+0000","last_fullsized":"2024-09-13T17:24:12.153754+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:56:23.533299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.587735+0000","last_change":"2024-09-13T17:24:10.163976+0000","last_active":"2024-09-13T17:24:11.587735+0000","last_peered":"2024-09-13T17:24:11.587735+0000","last_clean":"2024-09-13T17:24:11.587735+0000","last_became_active":"2024-09-13T17:24:10.163456+0000","last_became_peered":"2024-09-13T17:24:10.163456+0000","last_unstale":"2024-09-13T17:24:11.587735+0000","last_undegraded":"2024-09-13T17:24:11.587735+0000","last_fullsized":"2024-09-13T17:24:11.587735+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:02:41.116739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.154568+0000","last_change":"2024-09-13T17:24:10.597585+0000","last_active":"2024-09-13T17:24:12.154568+0000","last_peered":"2024-09-13T17:24:12.154568+0000","last_clean":"2024-09-13T17:24:12.154568+0000","last_became_active":"2024-09-13T17:24:10.597256+0000","last_became_peered":"2024-09-13T17:24:10.597256+0000","last_unstale":"2024-09-13T17:24:12.154568+0000","last_undegraded":"2024-09-13T17:24:12.154568+0000","last_fullsized":"2024-09-13T17:24:12.154568+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:57.550300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584728+0000","last_change":"2024-09-13T17:24:10.167794+0000","last_active":"2024-09-13T17:24:11.584728+0000","last_peered":"2024-09-13T17:24:11.584728+0000","last_clean":"2024-09-13T17:24:11.584728+0000","last_became_active":"2024-09-13T17:24:10.167533+0000","last_became_peered":"2024-09-13T17:24:10.167533+0000","last_unstale":"2024-09-13T17:24:11.584728+0000","last_undegraded":"2024-09-13T17:24:11.584728+0000","last_fullsized":"2024-09-13T17:24:11.584728+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:47.468554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653225+0000","last_change":"2024-09-13T17:24:10.605798+0000","last_active":"2024-09-13T17:24:11.653225+0000","last_peered":"2024-09-13T17:24:11.653225+0000","last_clean":"2024-09-13T17:24:11.653225+0000","last_became_active":"2024-09-13T17:24:10.605527+0000","last_became_peered":"2024-09-13T17:24:10.605527+0000","last_unstale":"2024-09-13T17:24:11.653225+0000","last_undegraded":"2024-09-13T17:24:11.653225+0000","last_fullsized":"2024-09-13T17:24:11.653225+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:49:05.071183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153555+0000","last_change":"2024-09-13T17:24:10.161964+0000","last_active":"2024-09-13T17:24:12.153555+0000","last_peered":"2024-09-13T17:24:12.153555+0000","last_clean":"2024-09-13T17:24:12.153555+0000","last_became_active":"2024-09-13T17:24:10.161665+0000","last_became_peered":"2024-09-13T17:24:10.161665+0000","last_unstale":"2024-09-13T17:24:12.153555+0000","last_undegraded":"2024-09-13T17:24:12.153555+0000","last_fullsized":"2024-09-13T17:24:12.153555+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:37:36.941285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653271+0000","last_change":"2024-09-13T17:24:10.602372+0000","last_active":"2024-09-13T17:24:11.653271+0000","last_peered":"2024-09-13T17:24:11.653271+0000","last_clean":"2024-09-13T17:24:11.653271+0000","last_became_active":"2024-09-13T17:24:10.600175+0000","last_became_peered":"2024-09-13T17:24:10.600175+0000","last_unstale":"2024-09-13T17:24:11.653271+0000","last_undegraded":"2024-09-13T17:24:11.653271+0000","last_fullsized":"2024-09-13T17:24:11.653271+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:57.550300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153627+0000","last_change":"2024-09-13T17:24:10.165872+0000","last_active":"2024-09-13T17:24:12.153627+0000","last_peered":"2024-09-13T17:24:12.153627+0000","last_clean":"2024-09-13T17:24:12.153627+0000","last_became_active":"2024-09-13T17:24:10.164792+0000","last_became_peered":"2024-09-13T17:24:10.164792+0000","last_unstale":"2024-09-13T17:24:12.153627+0000","last_undegraded":"2024-09-13T17:24:12.153627+0000","last_fullsized":"2024-09-13T17:24:12.153627+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:31.026549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653112+0000","last_change":"2024-09-13T17:24:10.602015+0000","last_active":"2024-09-13T17:24:11.653112+0000","last_peered":"2024-09-13T17:24:11.653112+0000","last_clean":"2024-09-13T17:24:11.653112+0000","last_became_active":"2024-09-13T17:24:10.598297+0000","last_became_peered":"2024-09-13T17:24:10.598297+0000","last_unstale":"2024-09-13T17:24:11.653112+0000","last_undegraded":"2024-09-13T17:24:11.653112+0000","last_fullsized":"2024-09-13T17:24:11.653112+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:10.561516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.653068+0000","last_change":"2024-09-13T17:24:10.165380+0000","last_active":"2024-09-13T17:24:11.653068+0000","last_peered":"2024-09-13T17:24:11.653068+0000","last_clean":"2024-09-13T17:24:11.653068+0000","last_became_active":"2024-09-13T17:24:10.163630+0000","last_became_peered":"2024-09-13T17:24:10.163630+0000","last_unstale":"2024-09-13T17:24:11.653068+0000","last_undegraded":"2024-09-13T17:24:11.653068+0000","last_fullsized":"2024-09-13T17:24:11.653068+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:03:25.715540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:12.153663+0000","last_change":"2024-09-13T17:24:10.589698+0000","last_active":"2024-09-13T17:24:12.153663+0000","last_peered":"2024-09-13T17:24:12.153663+0000","last_clean":"2024-09-13T17:24:12.153663+0000","last_became_active":"2024-09-13T17:24:10.589411+0000","last_became_peered":"2024-09-13T17:24:10.589411+0000","last_unstale":"2024-09-13T17:24:12.153663+0000","last_undegraded":"2024-09-13T17:24:12.153663+0000","last_fullsized":"2024-09-13T17:24:12.153663+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T05:23:29.924841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.584350+0000","last_change":"2024-09-13T17:24:10.171531+0000","last_active":"2024-09-13T17:24:11.584350+0000","last_peered":"2024-09-13T17:24:11.584350+0000","last_clean":"2024-09-13T17:24:11.584350+0000","last_became_active":"2024-09-13T17:24:10.171131+0000","last_became_peered":"2024-09-13T17:24:10.171131+0000","last_unstale":"2024-09-13T17:24:11.584350+0000","last_undegraded":"2024-09-13T17:24:11.584350+0000","last_fullsized":"2024-09-13T17:24:11.584350+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:09.129126+0000","last_clean_scrub_stamp":"2024-09-13T17:24:09.129126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:20.440578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":10,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-13T17:24:11.586955+0000","last_change":"2024-09-13T17:24:10.594323+0000","last_active":"2024-09-13T17:24:11.586955+0000","last_peered":"2024-09-13T17:24:11.586955+0000","last_clean":"2024-09-13T17:24:11.586955+0000","last_became_active":"2024-09-13T17:24:10.594028+0000","last_became_peered":"2024-09-13T17:24:10.594028+0000","last_unstale":"2024-09-13T17:24:11.586955+0000","last_undegraded":"2024-09-13T17:24:11.586955+0000","last_fullsized":"2024-09-13T17:24:11.586955+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:24:10.137175+0000","last_clean_scrub_stamp":"2024-09-13T17:24:10.137175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:00.767994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":347,"reported_epoch":232,"state":"active+clean","last_fresh":"2024-09-13T17:24:10.595532+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:24:10.595532+0000","last_peered":"2024-09-13T17:24:10.595532+0000","last_clean":"2024-09-13T17:24:10.595532+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:24:10.595532+0000","last_undegraded":"2024-09-13T17:24:10.595532+0000","last_fullsized":"2024-09-13T17:24:10.595532+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70616,"kb_used_data":3460,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301224,"statfs":{"total":96636764160,"available":96564453376,"internally_reserved":0,"allocated":3543040,"data_stored":1894198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.63200000000000001,"15min":0.64700000000000002},"min":{"1min":0.47999999999999998,"5min":0.33900000000000002,"15min":0.33700000000000002},"max":{"1min":0.92800000000000005,"5min":1.008,"15min":2.0880000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.67100000000000004,"15min":0.67600000000000005},"min":{"1min":0.41699999999999998,"5min":0.373,"15min":0.316},"max":{"1min":0.97599999999999998,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.64300000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.55600000000000005,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.96899999999999997,"5min":1.161,"15min":2.5030000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.67000000000000004,"15min":0.70299999999999996},"min":{"1min":0.63200000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86799999999999999,"5min":1.399,"15min":6.4909999999999997},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.58999999999999997,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93799999999999994,"5min":1.105,"15min":1.544},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.69099999999999995,"15min":0.69699999999999995},"min":{"1min":0.45500000000000002,"5min":0.434,"15min":0.39900000000000002},"max":{"1min":0.95399999999999996,"5min":1.0269999999999999,"15min":1.605},"last":0.75}]},{"osd":3,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67400000000000004,"15min":0.67700000000000005},"min":{"1min":0.52400000000000002,"5min":0.443,"15min":0.34999999999999998},"max":{"1min":0.93799999999999994,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.71199999999999997,"15min":0.73599999999999999},"min":{"1min":0.623,"5min":0.437,"15min":0.437},"max":{"1min":0.88800000000000001,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.82799999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58199999999999996,"15min":0.60299999999999998},"min":{"1min":0.41399999999999998,"5min":0.35799999999999998,"15min":0.30499999999999999},"max":{"1min":0.73799999999999999,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.629,"15min":0.65000000000000002},"min":{"1min":0.38900000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":1.2170000000000001,"15min":1.234},"last":0.60799999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63400000000000001,"15min":0.65000000000000002},"min":{"1min":0.47599999999999998,"5min":0.35399999999999998,"15min":0.33800000000000002},"max":{"1min":0.92200000000000004,"5min":1.135,"15min":1.143},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.74399999999999999},"min":{"1min":0.60499999999999998,"5min":0.47999999999999998,"15min":0.42599999999999999},"max":{"1min":0.90500000000000003,"5min":1.915,"15min":1.915},"last":0.80800000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:24 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.70499999999999996,"15min":0.72099999999999997},"min":{"1min":0.498,"5min":0.39300000000000002,"15min":0.38200000000000001},"max":{"1min":0.91800000000000004,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.70699999999999996,"15min":0.72599999999999998},"min":{"1min":0.55500000000000005,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95899999999999996,"5min":1.607,"15min":1.607},"last":0.77000000000000002}]}]},{"osd":6,"up_from":13,"seq":55834575047,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70040,"kb_used_data":2884,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301800,"statfs":{"total":96636764160,"available":96565043200,"internally_reserved":0,"allocated":2953216,"data_stored":1303872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68300000000000005,"15min":0.67900000000000005},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.32900000000000001},"max":{"1min":0.89700000000000002,"5min":0.97099999999999997,"15min":3.6629999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.72499999999999998,"15min":0.74299999999999999},"min":{"1min":0.51300000000000001,"5min":0.41699999999999998,"15min":0.36099999999999999},"max":{"1min":0.92400000000000004,"5min":0.97399999999999998,"15min":5.9800000000000004},"last":0.86499999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.63800000000000001,"15min":0.66000000000000003},"min":{"1min":0.44400000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.874,"5min":0.998,"15min":3.7040000000000002},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67800000000000005,"15min":0.67600000000000005},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.36299999999999999},"max":{"1min":0.88500000000000001,"5min":0.96899999999999997,"15min":3.3079999999999998},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.67800000000000005,"15min":0.68400000000000005},"min":{"1min":0.372,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.88,"5min":0.88,"15min":5.9240000000000004},"last":0.872},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.67700000000000005,"15min":0.68899999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":3.2629999999999999},"last":0.76600000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.53800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.94099999999999995,"5min":0.95199999999999996,"15min":3.6520000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.69199999999999995,"15min":0.70299999999999996},"min":{"1min":0.46999999999999997,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95899999999999996,"15min":3.742},"last":0.94999999999999996}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.628,"15min":0.624},"min":{"1min":0.40100000000000002,"5min":0.375,"15min":0.32600000000000001},"max":{"1min":1.006,"5min":1.262,"15min":1.6439999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.626,"15min":0.64200000000000002},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":4.7750000000000004},"last":0.623}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.68300000000000005,"15min":0.68600000000000005},"min":{"1min":0.48399999999999999,"5min":0.39800000000000002,"15min":0.374},"max":{"1min":1.0129999999999999,"5min":1.0509999999999999,"15min":2.3700000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64400000000000002,"15min":0.66100000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":1.089,"5min":1.1559999999999999,"15min":3.609},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.70199999999999996,"15min":0.72399999999999998},"min":{"1min":0.504,"5min":0.379,"15min":0.378},"max":{"1min":1.163,"5min":1.163,"15min":4.9669999999999996},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.78100000000000003,"15min":0.78500000000000003},"min":{"1min":0.60499999999999998,"5min":0.55900000000000005,"15min":0.46800000000000003},"max":{"1min":1.087,"5min":1.0940000000000001,"15min":3.8100000000000001},"last":0.92600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575046,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47944,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323896,"statfs":{"total":96636764160,"available":96587669504,"internally_reserved":0,"allocated":2945024,"data_stored":1303760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.76100000000000001},"min":{"1min":0.39600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86399999999999999,"5min":1.4390000000000001,"15min":38.005000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.68600000000000005,"15min":0.73599999999999999},"min":{"1min":0.41099999999999998,"5min":0.377,"15min":0.33400000000000002},"max":{"1min":0.91600000000000004,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.68200000000000005,"15min":0.72199999999999998},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.871,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.69299999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.41099999999999998},"max":{"1min":0.90700000000000003,"5min":1.482,"15min":7.7370000000000001},"last":0.498}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.69099999999999995,"15min":0.73499999999999999},"min":{"1min":0.45600000000000002,"5min":0.40600000000000003,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.68300000000000005,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.439,"15min":0.39800000000000002},"max":{"1min":0.83699999999999997,"5min":1.0649999999999999,"15min":9.6549999999999994},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":1.0780000000000001},"min":{"1min":0.56100000000000005,"5min":0.45700000000000002,"15min":0.435},"max":{"1min":1.042,"5min":1.042,"15min":113.363},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.72799999999999998,"15min":0.73799999999999999},"min":{"1min":0.63400000000000001,"5min":0.45800000000000002,"15min":0.433},"max":{"1min":0.93799999999999994,"5min":1.113,"15min":4.1180000000000003},"last":0.71499999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.61699999999999999,"15min":0.61899999999999999},"min":{"1min":0.38700000000000001,"5min":0.36099999999999999,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":1.5669999999999999,"15min":2.0590000000000002},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66300000000000003,"15min":0.67500000000000004},"min":{"1min":0.52400000000000002,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.90200000000000002,"5min":1.03,"15min":2.827},"last":0.74299999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68000000000000005,"15min":0.68999999999999995},"min":{"1min":0.48199999999999998,"5min":0.40400000000000003,"15min":0.34799999999999998},"max":{"1min":1.0609999999999999,"5min":1.73,"15min":2.2320000000000002},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.70499999999999996,"15min":0.71299999999999997},"min":{"1min":0.48799999999999999,"5min":0.44800000000000001,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":1.532,"15min":4.1500000000000004},"last":0.68700000000000006}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.73799999999999999,"15min":0.752},"min":{"1min":0.59899999999999998,"5min":0.442,"15min":0.40999999999999998},"max":{"1min":0.97299999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.53700000000000003,"5min":0.47099999999999997,"15min":0.44},"max":{"1min":1.169,"5min":1.363,"15min":2.02},"last":0.82199999999999995}]}]},{"osd":4,"up_from":13,"seq":55834575045,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74140,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297700,"statfs":{"total":96636764160,"available":96560844800,"internally_reserved":0,"allocated":2965504,"data_stored":1305380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":1.2150000000000001,"15min":0.84099999999999997},"min":{"1min":0.45000000000000001,"5min":0.34999999999999998,"15min":0.32200000000000001},"max":{"1min":0.94499999999999995,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.77600000000000002,"5min":1.2010000000000001,"15min":0.875},"min":{"1min":0.46600000000000003,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93200000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.60399999999999998}]},{"osd":1,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.85299999999999998,"15min":0.72199999999999998},"min":{"1min":0.40899999999999997,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.96799999999999997,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.753,"15min":0.68200000000000005},"min":{"1min":0.53800000000000003,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.97599999999999998,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71799999999999997,"15min":0.67900000000000005},"min":{"1min":0.49299999999999999,"5min":0.38300000000000001,"15min":0.34000000000000002},"max":{"1min":0.996,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.71199999999999997,"15min":0.70699999999999996},"min":{"1min":0.57599999999999996,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.91500000000000004,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.501}]},{"osd":3,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.109,"15min":0.82599999999999996},"min":{"1min":0.61799999999999999,"5min":0.437,"15min":0.378},"max":{"1min":0.95499999999999996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.77600000000000002,"15min":0.72599999999999998},"min":{"1min":0.51100000000000001,"5min":0.42099999999999999,"15min":0.40600000000000003},"max":{"1min":0.92600000000000005,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.628}]},{"osd":5,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.69299999999999995,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.93799999999999994,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.63400000000000001,"15min":0.64400000000000002},"min":{"1min":0.496,"5min":0.32600000000000001,"15min":0.30299999999999999},"max":{"1min":0.88200000000000001,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.45200000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.32200000000000001},"max":{"1min":0.92700000000000005,"5min":2.976,"15min":2.976},"last":0.438},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.76200000000000001,"15min":0.73399999999999999},"min":{"1min":0.48799999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.96399999999999997,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54400000000000004,"5min":0.374,"15min":0.374},"max":{"1min":0.92200000000000004,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.76200000000000001,"15min":0.749},"min":{"1min":0.55400000000000005,"5min":0.38100000000000001,"15min":0.371},"max":{"1min":0.95099999999999996,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.65800000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575046,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70052,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301788,"statfs":{"total":96636764160,"available":96565030912,"internally_reserved":0,"allocated":2961408,"data_stored":1305906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.435,"15min":0.434},"min":{"1min":0.29199999999999998,"5min":0.27700000000000002,"15min":0.25},"max":{"1min":0.624,"5min":0.82599999999999996,"15min":1.9810000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.38600000000000001,"5min":0.28199999999999997,"15min":0.26200000000000001},"max":{"1min":0.73199999999999998,"5min":0.76400000000000001,"15min":2.96},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.48099999999999998,"15min":0.503},"min":{"1min":0.312,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":3.5600000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63,"5min":0.57799999999999996,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.379,"15min":0.32900000000000001},"max":{"1min":0.78800000000000003,"5min":0.92900000000000005,"15min":3.6829999999999998},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53700000000000003,"15min":0.53200000000000003},"min":{"1min":0.38900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.82499999999999996,"5min":0.86699999999999999,"15min":1.9590000000000001},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.41499999999999998,"5min":0.38,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.875,"15min":1.3819999999999999},"last":0.45700000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75,"15min":0.74099999999999999},"min":{"1min":0.48699999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"max":{"1min":1.1479999999999999,"5min":1.21,"15min":2.8610000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.76900000000000002,"15min":0.75900000000000001},"min":{"1min":0.5,"5min":0.5,"15min":0.48199999999999998},"max":{"1min":1.0860000000000001,"5min":1.23,"15min":1.3640000000000001},"last":0.69699999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.76500000000000001},"min":{"1min":0.60799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.071,"5min":1.323,"15min":1.391},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77000000000000002,"15min":0.75900000000000001},"min":{"1min":0.54500000000000004,"5min":0.499,"15min":0.442},"max":{"1min":1.05,"5min":1.3200000000000001,"15min":2.9300000000000002},"last":0.81100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.77800000000000002,"15min":0.77000000000000002},"min":{"1min":0.53600000000000003,"5min":0.51100000000000001,"15min":0.47199999999999998},"max":{"1min":1.1559999999999999,"5min":1.218,"15min":1.401},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81699999999999995,"15min":0.78600000000000003},"min":{"1min":0.64700000000000002,"5min":0.53500000000000003,"15min":0.46300000000000002},"max":{"1min":1.175,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.59899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.80200000000000005,"15min":0.78900000000000003},"min":{"1min":0.624,"5min":0.503,"15min":0.42699999999999999},"max":{"1min":1.2130000000000001,"5min":1.27,"15min":2.899},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.81499999999999995,"15min":0.80100000000000005},"min":{"1min":0.63800000000000001,"5min":0.495,"15min":0.47599999999999998},"max":{"1min":1.208,"5min":1.2749999999999999,"15min":1.4359999999999999},"last":0.73999999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575046,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52044,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319796,"statfs":{"total":96636764160,"available":96583471104,"internally_reserved":0,"allocated":2940928,"data_stored":1302741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.47599999999999998,"15min":0.436},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.60499999999999998,"5min":5.6769999999999996,"15min":5.6769999999999996},"last":0.314},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.48499999999999999,"15min":0.58099999999999996},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.69599999999999995,"5min":1.593,"15min":30.023},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54800000000000004,"15min":1.371},"min":{"1min":0.39600000000000002,"5min":0.32000000000000001,"15min":0.307},"max":{"1min":0.67000000000000004,"5min":5.7409999999999997,"15min":260.53399999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":1.4690000000000001},"min":{"1min":0.38500000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":3.3620000000000001,"15min":262.61399999999998},"last":0.40100000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56499999999999995,"15min":0.624},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.29099999999999998},"max":{"1min":0.78400000000000003,"5min":5.0229999999999997,"15min":27.271999999999998},"last":0.443},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.68300000000000005,"15min":0.61499999999999999},"min":{"1min":0.50800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.74299999999999999,"5min":8.4149999999999991,"15min":8.4149999999999991},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.53000000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1379999999999999,"5min":6.0039999999999996,"15min":10.598000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.81399999999999995,"15min":0.85999999999999999},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.371},"max":{"1min":1.264,"5min":6.7080000000000002,"15min":21.213000000000001},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.79300000000000004,"15min":0.878},"min":{"1min":0.52300000000000002,"5min":0.48299999999999998,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":3.8279999999999998,"15min":27.135000000000002},"last":0.623},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.82199999999999995,"15min":0.78400000000000003},"min":{"1min":0.61399999999999999,"5min":0.5,"15min":0.47599999999999998},"max":{"1min":1.1619999999999999,"5min":5.8890000000000002,"15min":5.8890000000000002},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81899999999999995,"15min":0.82599999999999996},"min":{"1min":0.61099999999999999,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.1930000000000001,"5min":6.4800000000000004,"15min":10.664},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.81499999999999995,"15min":0.89100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.23,"5min":4.9480000000000004,"15min":27.236999999999998},"last":0.94999999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.80700000000000005,"15min":0.82799999999999996},"min":{"1min":0.61299999999999999,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.1299999999999999,"5min":3.3929999999999998,"15min":10.712999999999999},"last":1.099},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.82999999999999996,"15min":0.90800000000000003},"min":{"1min":0.55200000000000005,"5min":0.501,"15min":0.46100000000000002},"max":{"1min":1.1799999999999999,"5min":4.9809999999999999,"15min":27.199999999999999},"last":0.84799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96280,"kb_used_data":2892,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275560,"statfs":{"total":96636764160,"available":96538173440,"internally_reserved":0,"allocated":2961408,"data_stored":1304536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.441,"15min":0.46300000000000002},"min":{"1min":0.25700000000000001,"5min":0.23899999999999999,"15min":0.23400000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.312},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.35999999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.68700000000000006,"5min":0.70199999999999996,"15min":6.7240000000000002},"last":0.36599999999999999}]},{"osd":2,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42999999999999999,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.69599999999999995,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.308,"15min":0.308},"max":{"1min":0.63300000000000001,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.41699999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.65300000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.443},{"interface":"front","average":{"1min":0.502,"5min":0.52800000000000002,"15min":0.59799999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.74199999999999999,"5min":0.83899999999999997,"15min":18.318000000000001},"last":0.46800000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.76200000000000001,"15min":0.78500000000000003},"min":{"1min":0.56000000000000005,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.92400000000000004,"5min":1.28,"15min":6.1630000000000003},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80000000000000004,"15min":0.80300000000000005},"min":{"1min":0.65300000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.95799999999999996,"5min":1.23,"15min":1.78},"last":0.81499999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.59599999999999997,"5min":0.502,"15min":0.48199999999999998},"max":{"1min":0.93000000000000005,"5min":1.2649999999999999,"15min":1.6870000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75800000000000001,"15min":0.78200000000000003},"min":{"1min":0.57299999999999995,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.77000000000000002,"15min":0.77800000000000002},"min":{"1min":0.54300000000000004,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.99299999999999999,"5min":1.2330000000000001,"15min":1.7310000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.78300000000000003,"15min":0.80200000000000005},"min":{"1min":0.628,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0489999999999999,"5min":1.246,"15min":6.1970000000000001},"last":0.85299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78100000000000003,"15min":0.78800000000000003},"min":{"1min":0.59999999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.81000000000000005,"15min":0.81200000000000006},"min":{"1min":0.61299999999999999,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.02,"5min":1.2609999999999999,"15min":1.6359999999999999},"last":0.88500000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575047,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74704,"kb_used_data":3452,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297136,"statfs":{"total":96636764160,"available":96560267264,"internally_reserved":0,"allocated":3534848,"data_stored":1893109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.46700000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.72599999999999998,"5min":1.986,"15min":5.5259999999999998},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.504,"15min":0.51800000000000002},"min":{"1min":0.31900000000000001,"5min":0.27300000000000002,"15min":0.27000000000000002},"max":{"1min":0.69999999999999996,"5min":2.371,"15min":5.4889999999999999},"last":0.28000000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.501,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.72099999999999997,"5min":0.89900000000000002,"15min":6.0430000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.34200000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76200000000000001,"5min":2.4039999999999999,"15min":4.6760000000000002},"last":0.39100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.48199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.878,"15min":4.7549999999999999},"last":0.372},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.56399999999999995,"15min":0.58999999999999997},"min":{"1min":0.42199999999999999,"5min":0.38,"15min":0.34999999999999998},"max":{"1min":0.79600000000000004,"5min":0.94899999999999995,"15min":4.5940000000000003},"last":0.432}]},{"osd":4,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.80300000000000005,"15min":0.79100000000000004},"min":{"1min":0.629,"5min":0.46200000000000002,"15min":0.443},"max":{"1min":1.119,"5min":3.8849999999999998,"15min":4.734},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.75,"15min":0.77400000000000002},"min":{"1min":0.70899999999999996,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":4.8079999999999998},"last":0.499}]},{"osd":5,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.76500000000000001,"15min":0.78500000000000003},"min":{"1min":0.66500000000000004,"5min":0.51600000000000001,"15min":0.44500000000000001},"max":{"1min":1.141,"5min":1.141,"15min":4.944},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.78800000000000003,"15min":0.79800000000000004},"min":{"1min":0.65100000000000002,"5min":0.50700000000000001,"15min":0.432},"max":{"1min":1.246,"5min":1.246,"15min":4.9710000000000001},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.78400000000000003,"15min":0.80800000000000005},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.46700000000000003},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":4.8470000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.81899999999999995,"15min":0.81200000000000006},"min":{"1min":0.65100000000000002,"5min":0.497,"15min":0.48599999999999999},"max":{"1min":1.274,"5min":2.8380000000000001,"15min":5.0170000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:23:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.85599999999999998,"15min":0.83399999999999996},"min":{"1min":0.68799999999999994,"5min":0.53400000000000003,"15min":0.47399999999999998},"max":{"1min":1.2150000000000001,"5min":3.9209999999999998,"15min":5.0540000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.80300000000000005,"15min":0.81699999999999995},"min":{"1min":0.67400000000000004,"5min":0.56899999999999995,"15min":0.505},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":4.9950000000000001},"last":0.56699999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:24:15.183 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T17:24:15.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T17:24:15.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.379+0000 7f9193e00640 1 Processor -- start 2024-09-13T17:24:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.380+0000 7f9193e00640 1 -- start start 2024-09-13T17:24:15.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 0x7f9194071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9194159e50 con 0x7f91940712d0 2024-09-13T17:24:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919415a130 con 0x7f9194071f60 2024-09-13T17:24:15.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91940ce210 con 0x7f91940cde30 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f91940712d0 0x7f919415af10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44432/0 (socket says 172.21.15.28:44432) 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9193400640 1 -- 172.21.15.28:0/1350266889 learned_addr learned my addr 172.21.15.28:0/1350266889 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.381+0000 7f9192a00640 1 --2- 172.21.15.28:0/1350266889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 0x7f9194071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703294875 0 0) 0x7f9194159e50 con 0x7f91940712d0 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9178003520 con 0x7f91940712d0 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935377269 0 0) 0x7f919415a130 con 0x7f9194071f60 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9194159e50 con 0x7f9194071f60 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192a00640 1 -- 172.21.15.28:0/1350266889 >> v1:172.21.15.189:6789/0 conn(0x7f9194071f60 legacy=0x7f9194070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192a00640 1 -- 172.21.15.28:0/1350266889 >> v1:172.21.15.28:6789/0 conn(0x7f91940712d0 legacy=0x7f919415af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.382+0000 7f9192a00640 1 -- 172.21.15.28:0/1350266889 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919415a410 con 0x7f91940cde30 2024-09-13T17:24:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.383+0000 7f9192a00640 1 --2- 172.21.15.28:0/1350266889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 0x7f9194071a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f91840097d0 tx=0x7f918402a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b36def2d1f4e3f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.383+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184032440 con 0x7f91940cde30 2024-09-13T17:24:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.383+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91840325a0 con 0x7f91940cde30 2024-09-13T17:24:15.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.383+0000 7f9192000640 1 -- 172.21.15.28:0/1350266889 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184032910 con 0x7f91940cde30 2024-09-13T17:24:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 -- 172.21.15.28:0/1350266889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 msgr2=0x7f9194071a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 --2- 172.21.15.28:0/1350266889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 0x7f9194071a20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f91840097d0 tx=0x7f918402a200 comp rx=0 tx=0).stop 2024-09-13T17:24:15.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 -- 172.21.15.28:0/1350266889 shutdown_connections 2024-09-13T17:24:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 --2- 172.21.15.28:0/1350266889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940cde30 0x7f9194071a20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 -- 172.21.15.28:0/1350266889 >> 172.21.15.28:0/1350266889 conn(0x7f9194069e80 msgr2=0x7f919406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 -- 172.21.15.28:0/1350266889 shutdown_connections 2024-09-13T17:24:15.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.384+0000 7f9193e00640 1 -- 172.21.15.28:0/1350266889 wait complete. 2024-09-13T17:24:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.386+0000 7f9193e00640 1 Processor -- start 2024-09-13T17:24:15.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.386+0000 7f9193e00640 1 -- start start 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940712d0 0x7f9194154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91940cde30 0x7f91941502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194155250 con 0x7f9194071f60 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9194155530 con 0x7f91940cde30 2024-09-13T17:24:15.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f9193e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9194155810 con 0x7f91940712d0 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f918a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f918a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33222/0 (socket says 172.21.15.28:33222) 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.387+0000 7f918a000640 1 -- 172.21.15.28:0/3103788616 learned_addr learned my addr 172.21.15.28:0/3103788616 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f9193400640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91940cde30 0x7f91941502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f9192a00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940712d0 0x7f9194154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 -- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940712d0 msgr2=0x7f9194154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940712d0 0x7f9194154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 -- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91940cde30 msgr2=0x7f91941502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91940cde30 0x7f91941502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 -- 172.21.15.28:0/3103788616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184009400 con 0x7f9194071f60 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918a000640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c430 tx=0x7f917c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c0052d0 con 0x7f9194071f60 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917c005450 con 0x7f9194071f60 2024-09-13T17:24:15.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9194155af0 con 0x7f9194071f60 2024-09-13T17:24:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917c003830 con 0x7f9194071f60 2024-09-13T17:24:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.388+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91941507e0 con 0x7f9194071f60 2024-09-13T17:24:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.389+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9194156010 con 0x7f9194071f60 2024-09-13T17:24:15.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.390+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f917c003990 con 0x7f9194071f60 2024-09-13T17:24:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.391+0000 7f918b400640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.391+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f917c0889f0 con 0x7f9194071f60 2024-09-13T17:24:15.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.392+0000 7f9192a00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.392+0000 7f9192a00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 0x7f917006d0c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f9184034000 tx=0x7f918402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.395+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9194156010 con 0x7f9194071f60 2024-09-13T17:24:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.644+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9194156010 con 0x7f9194071f60 2024-09-13T17:24:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.648+0000 7f918b400640 1 -- 172.21.15.28:0/3103788616 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v30) v1 ==== 190+0+0 (secure 0 0 0) 0x7f9194156010 con 0x7f9194071f60 2024-09-13T17:24:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:updated caps for client.0 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.650+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 msgr2=0x7f917006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.650+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 0x7f917006d0c0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f9184034000 tx=0x7f918402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.650+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 msgr2=0x7f9194150e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.650+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 secure :-1 s=READY pgs=2732 cs=0 l=1 rev1=1 crypto rx=0x7f917c00c430 tx=0x7f917c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 shutdown_connections 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91940cde30 0x7f91941502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9194071f60 0x7f9194150e00 unknown :-1 s=CLOSED pgs=2732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 --2- 172.21.15.28:0/3103788616 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f91940712d0 0x7f9194154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 >> 172.21.15.28:0/3103788616 conn(0x7f9194069e80 msgr2=0x7f9194158ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 shutdown_connections 2024-09-13T17:24:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.651+0000 7f9193e00640 1 -- 172.21.15.28:0/3103788616 wait complete. 2024-09-13T17:24:15.668 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.821+0000 7f5f58200640 1 Processor -- start 2024-09-13T17:24:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.821+0000 7f5f58200640 1 -- start start 2024-09-13T17:24:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f5f501592a0 0x7f5f5015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f5006f970 0x7f5f50159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5006c2d0 0x7f5f5006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5015b0b0 con 0x7f5f5006f970 2024-09-13T17:24:15.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f5006cc70 con 0x7f5f501592a0 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f58200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500b2040 con 0x7f5f5006c2d0 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f5006f970 0x7f5f50159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5f5006f970 0x7f5f50159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33236/0 (socket says 172.21.15.28:33236) 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 -- 172.21.15.28:0/741016673 learned_addr learned my addr 172.21.15.28:0/741016673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f55800640 1 --2- 172.21.15.28:0/741016673 >> v2:172.21.15.189:3300/0 conn(0x7f5f501592a0 0x7f5f5015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f56200640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5006c2d0 0x7f5f5006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 -- 172.21.15.28:0/741016673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5006c2d0 msgr2=0x7f5f5006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5006c2d0 0x7f5f5006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 -- 172.21.15.28:0/741016673 >> v2:172.21.15.189:3300/0 conn(0x7f5f501592a0 msgr2=0x7f5f5015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 --2- 172.21.15.28:0/741016673 >> v2:172.21.15.189:3300/0 conn(0x7f5f501592a0 0x7f5f5015ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.823+0000 7f5f54e00640 1 -- 172.21.15.28:0/741016673 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50159ff0 con 0x7f5f5006f970 2024-09-13T17:24:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.824+0000 7f5f54e00640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006f970 0x7f5f50159680 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f5f5006d370 tx=0x7f5f3c02c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf48296d384c8d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f4fe00640 1 -- 172.21.15.28:0/741016673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c032790 con 0x7f5f5006f970 2024-09-13T17:24:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f4fe00640 1 -- 172.21.15.28:0/741016673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3c0328f0 con 0x7f5f5006f970 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f4fe00640 1 -- 172.21.15.28:0/741016673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c032c80 con 0x7f5f5006f970 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 -- 172.21.15.28:0/741016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006f970 msgr2=0x7f5f50159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006f970 0x7f5f50159680 secure :-1 s=READY pgs=2733 cs=0 l=1 rev1=1 crypto rx=0x7f5f5006d370 tx=0x7f5f3c02c890 comp rx=0 tx=0).stop 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 -- 172.21.15.28:0/741016673 shutdown_connections 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006f970 0x7f5f50159680 unknown :-1 s=CLOSED pgs=2733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 --2- 172.21.15.28:0/741016673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f5006c2d0 0x7f5f5006c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 --2- 172.21.15.28:0/741016673 >> v2:172.21.15.189:3300/0 conn(0x7f5f501592a0 0x7f5f5015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 -- 172.21.15.28:0/741016673 >> 172.21.15.28:0/741016673 conn(0x7f5f50069e80 msgr2=0x7f5f5006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 -- 172.21.15.28:0/741016673 shutdown_connections 2024-09-13T17:24:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.825+0000 7f5f58200640 1 -- 172.21.15.28:0/741016673 wait complete. 2024-09-13T17:24:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.827+0000 7f5f58200640 1 Processor -- start 2024-09-13T17:24:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.827+0000 7f5f58200640 1 -- start start 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f5006f970 0x7f5f5006b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f501592a0 0x7f5f5006b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500cc170 con 0x7f5f5006c2d0 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500ccfb0 con 0x7f5f5006f970 2024-09-13T17:24:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f58200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f500cc3a0 con 0x7f5f501592a0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f55800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f55800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33242/0 (socket says 172.21.15.28:33242) 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f55800640 1 -- 172.21.15.28:0/854705704 learned_addr learned my addr 172.21.15.28:0/854705704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f56200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f501592a0 0x7f5f5006b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.828+0000 7f5f54e00640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f5006f970 0x7f5f5006b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 -- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f501592a0 msgr2=0x7f5f5006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f501592a0 0x7f5f5006b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 -- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f5006f970 msgr2=0x7f5f5006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f5006f970 0x7f5f5006b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 -- 172.21.15.28:0/854705704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3c00b770 con 0x7f5f5006c2d0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f55800640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e870 tx=0x7f5f4400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f440094f0 con 0x7f5f5006c2d0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4400c940 con 0x7f5f5006c2d0 2024-09-13T17:24:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400ccf0 con 0x7f5f5006c2d0 2024-09-13T17:24:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f500cc680 con 0x7f5f5006c2d0 2024-09-13T17:24:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.829+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f500ccc40 con 0x7f5f5006c2d0 2024-09-13T17:24:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.830+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5006c730 con 0x7f5f5006c2d0 2024-09-13T17:24:15.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.831+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5f44013470 con 0x7f5f5006c2d0 2024-09-13T17:24:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.832+0000 7f5f4d600640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 0x7f5f3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.832+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f44089470 con 0x7f5f5006c2d0 2024-09-13T17:24:15.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.833+0000 7f5f54e00640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 0x7f5f3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.834+0000 7f5f54e00640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 0x7f5f3006d0c0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c037000 tx=0x7f5f3c03e040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:15.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:15.837+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5f440545b0 con 0x7f5f5006c2d0 2024-09-13T17:24:16.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.052+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f500b18b0 con 0x7f5f5006c2d0 2024-09-13T17:24:16.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.053+0000 7f5f4d600640 1 -- 172.21.15.28:0/854705704 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f5f44057bd0 con 0x7f5f5006c2d0 2024-09-13T17:24:16.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:16.056 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":188,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:13.584112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1114231130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1114231130},{"type":"v1","addr":"172.21.15.28:6835","nonce":1114231130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:16.056 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 188 2024-09-13T17:24:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 msgr2=0x7f5f3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 0x7f5f3006d0c0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c037000 tx=0x7f5f3c03e040 comp rx=0 tx=0).stop 2024-09-13T17:24:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 msgr2=0x7f5f50151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 secure :-1 s=READY pgs=2734 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e870 tx=0x7f5f4400ed30 comp rx=0 tx=0).stop 2024-09-13T17:24:16.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 shutdown_connections 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5f3006ac00 0x7f5f3006d0c0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5f501592a0 0x7f5f5006b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f5006f970 0x7f5f5006b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 --2- 172.21.15.28:0/854705704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5f5006c2d0 0x7f5f50151190 unknown :-1 s=CLOSED pgs=2734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.055+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 >> 172.21.15.28:0/854705704 conn(0x7f5f50069e80 msgr2=0x7f5f5015beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.056+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 shutdown_connections 2024-09-13T17:24:16.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.056+0000 7f5f58200640 1 -- 172.21.15.28:0/854705704 wait complete. 2024-09-13T17:24:16.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 188, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:24:10.577893+0000', 'modified': '2024-09-13T17:24:13.584112+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12318}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12318': {'gid': 12318, 'name': 'a', 'rank': 0, 'incarnation': 185, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.28:6835/1114231130', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 1114231130}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 1114231130}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:24:16.478 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:24:16.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:24:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.635+0000 7fc2a0000640 1 Processor -- start 2024-09-13T17:24:16.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.635+0000 7fc2a0000640 1 -- start start 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc2a0000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2980cbdf0 0x7fc298167650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc2a0000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2980b4350 con 0x7fc29815dbf0 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc2a0000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2980b4070 con 0x7fc2980ae580 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc2a0000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29815cf00 con 0x7fc2980cbdf0 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc29e000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fc2980cbdf0 0x7fc298167650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc29cc00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fc29815dbf0 0x7fc2980cb6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44442/0 (socket says 172.21.15.28:44442) 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc29cc00640 1 -- 172.21.15.28:0/3340025310 learned_addr learned my addr 172.21.15.28:0/3340025310 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:16.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.636+0000 7fc29e000640 1 --2- 172.21.15.28:0/3340025310 >> v2:172.21.15.28:3300/0 conn(0x7fc2980cbdf0 0x7fc298167650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33254/0 (socket says 172.21.15.28:33254) 2024-09-13T17:24:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.637+0000 7fc29e000640 1 -- 172.21.15.28:0/3340025310 >> v1:172.21.15.189:6789/0 conn(0x7fc2980ae580 legacy=0x7fc2980b3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.637+0000 7fc29e000640 1 -- 172.21.15.28:0/3340025310 >> v1:172.21.15.28:6789/0 conn(0x7fc29815dbf0 legacy=0x7fc2980cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.637+0000 7fc29e000640 1 -- 172.21.15.28:0/3340025310 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29815d110 con 0x7fc2980cbdf0 2024-09-13T17:24:16.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.637+0000 7fc297e00640 1 -- 172.21.15.28:0/3340025310 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834609338 0 0) 0x7fc284001110 con 0x7fc29815dbf0 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc29e000640 1 --2- 172.21.15.28:0/3340025310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2980cbdf0 0x7fc298167650 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fc29815d040 tx=0x7fc29002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85741096f847a1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc297e00640 1 -- 172.21.15.28:0/3340025310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290007740 con 0x7fc2980cbdf0 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc297e00640 1 -- 172.21.15.28:0/3340025310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29002d050 con 0x7fc2980cbdf0 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc297e00640 1 -- 172.21.15.28:0/3340025310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290007ac0 con 0x7fc2980cbdf0 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc2a0000640 1 -- 172.21.15.28:0/3340025310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2980cbdf0 msgr2=0x7fc298167650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.638+0000 7fc2a0000640 1 --2- 172.21.15.28:0/3340025310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2980cbdf0 0x7fc298167650 secure :-1 s=READY pgs=2736 cs=0 l=1 rev1=1 crypto rx=0x7fc29815d040 tx=0x7fc29002a660 comp rx=0 tx=0).stop 2024-09-13T17:24:16.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.639+0000 7fc2a0000640 1 -- 172.21.15.28:0/3340025310 shutdown_connections 2024-09-13T17:24:16.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.639+0000 7fc2a0000640 1 --2- 172.21.15.28:0/3340025310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc2980cbdf0 0x7fc298167650 unknown :-1 s=CLOSED pgs=2736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.639+0000 7fc2a0000640 1 -- 172.21.15.28:0/3340025310 >> 172.21.15.28:0/3340025310 conn(0x7fc2980b0af0 msgr2=0x7fc2980b0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.639+0000 7fc2a0000640 1 -- 172.21.15.28:0/3340025310 shutdown_connections 2024-09-13T17:24:16.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.639+0000 7fc2a0000640 1 -- 172.21.15.28:0/3340025310 wait complete. 2024-09-13T17:24:16.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.640+0000 7fc2a0000640 1 Processor -- start 2024-09-13T17:24:16.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 -- start start 2024-09-13T17:24:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2980ae580 0x7fc298154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2980cbdf0 0x7fc2981563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc298154550 con 0x7fc29815dbf0 2024-09-13T17:24:16.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc298156920 con 0x7fc2980cbdf0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.641+0000 7fc2a0000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc298155340 con 0x7fc2980ae580 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29d600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2980ae580 0x7fc298154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33264/0 (socket says 172.21.15.28:33264) 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 -- 172.21.15.28:0/1983504428 learned_addr learned my addr 172.21.15.28:0/1983504428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29cc00640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2980cbdf0 0x7fc2981563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 -- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2980ae580 msgr2=0x7fc298154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2980ae580 0x7fc298154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 -- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2980cbdf0 msgr2=0x7fc2981563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2980cbdf0 0x7fc2981563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.642+0000 7fc29e000640 1 -- 172.21.15.28:0/1983504428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290009a70 con 0x7fc29815dbf0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc29e000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fc29002ac50 tx=0x7fc290004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc295600640 1 -- 172.21.15.28:0/1983504428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290031910 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc295600640 1 -- 172.21.15.28:0/1983504428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc290031a70 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc295600640 1 -- 172.21.15.28:0/1983504428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290031e20 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc298155620 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.643+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc298155be0 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.645+0000 7fc295600640 1 -- 172.21.15.28:0/1983504428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc290045460 con 0x7fc29815dbf0 2024-09-13T17:24:16.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.646+0000 7fc295600640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 0x7fc27806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.646+0000 7fc295600640 1 -- 172.21.15.28:0/1983504428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2900aafe0 con 0x7fc29815dbf0 2024-09-13T17:24:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.646+0000 7fc2a2e57740 1 Processor -- start 2024-09-13T17:24:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc29d600640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 0x7fc27806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 -- start start 2024-09-13T17:24:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43826d0 0x5574d4382ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d437fe90 0x5574d43cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d42ebab0 con 0x5574d43a9b80 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d42e9240 con 0x5574d43826d0 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574d42e4b30 con 0x5574d437fe90 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.647+0000 7fc29d600640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 0x7fc27806d0c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000f60 tx=0x7fc28c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43826d0 0x5574d4382ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33274/0 (socket says 172.21.15.28:33274) 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 -- 172.21.15.28:0/1756334267 learned_addr learned my addr 172.21.15.28:0/1756334267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29cc00640 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d437fe90 0x5574d43cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 -- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d437fe90 msgr2=0x5574d43cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d437fe90 0x5574d43cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 -- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43826d0 msgr2=0x5574d4382ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43826d0 0x5574d4382ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 -- 172.21.15.28:0/1756334267 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574d42afbd0 con 0x5574d43a9b80 2024-09-13T17:24:16.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.648+0000 7fc29e000640 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fc290081d50 tx=0x7fc29007a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b738f65d1f6464 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc28a000640 1 -- 172.21.15.28:0/1756334267 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29007aea0 con 0x5574d43a9b80 2024-09-13T17:24:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc28a000640 1 -- 172.21.15.28:0/1756334267 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900782e0 con 0x5574d43a9b80 2024-09-13T17:24:16.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc28a000640 1 -- 172.21.15.28:0/1756334267 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29007ebe0 con 0x5574d43a9b80 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 -- 172.21.15.28:0/1756334267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 msgr2=0x5574d43d00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 secure :-1 s=READY pgs=2738 cs=0 l=1 rev1=1 crypto rx=0x7fc290081d50 tx=0x7fc29007a000 comp rx=0 tx=0).stop 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 -- 172.21.15.28:0/1756334267 shutdown_connections 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43a9b80 0x5574d43d00e0 unknown :-1 s=CLOSED pgs=2738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d437fe90 0x5574d43cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/1756334267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43826d0 0x5574d4382ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 -- 172.21.15.28:0/1756334267 >> 172.21.15.28:0/1756334267 conn(0x5574d437ee00 msgr2=0x5574d43c72b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 -- 172.21.15.28:0/1756334267 shutdown_connections 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.649+0000 7fc2a2e57740 1 -- 172.21.15.28:0/1756334267 wait complete. 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.650+0000 7fc2a2e57740 1 -- start start 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.650+0000 7fc2a2e57740 1 Processor -- start 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43826d0 0x5574d43a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 0x5574d43ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d4286260 con 0x5574d43ad660 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d4285f00 con 0x5574d43a9b80 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574d404bf20 con 0x5574d43826d0 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.651+0000 7fc2a2e57740 10 client.0 _ll_register_callbacks cb 0x5574d42de1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33286/0 (socket says 172.21.15.28:33286) 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 -- 172.21.15.28:0/3074974428 learned_addr learned my addr 172.21.15.28:0/3074974428 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29d600640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43826d0 0x5574d43a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29cc00640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 0x5574d43ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 -- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43826d0 msgr2=0x5574d43a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43826d0 0x5574d43a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 -- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 msgr2=0x5574d43ab0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 0x5574d43ab0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 -- 172.21.15.28:0/3074974428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc298155be0 con 0x5574d43ad660 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29cc00640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 0x5574d43ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc29e000640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fc290081d50 tx=0x7fc290009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71c2bfc18a6d2f5e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc27ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.652+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29007ede0 con 0x5574d43ad660 2024-09-13T17:24:16.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574d430ecd0 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc29007a1a0 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=188) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2900a9240 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 10 client.12453 fetch_fsmap learned FSMap version 188 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=188,monmap=2+}) v3 -- 0x5574d42e31b0 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5574d430ecd0 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 188) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc290038070 con 0x5574d43ad660 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 10 client.12453 fetch_fsmap finished waiting for FSMap version 188 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 10 client.12453 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 20 client.12453 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d43a8600 0x5574d43e4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 4 client.12453 mds_command: new command op to 12318 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 --> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574d3fd83c0 con 0x5574d43a8600 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.653+0000 7fc29d600640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d43a8600 0x5574d43e4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.654+0000 7fc29d600640 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d43a8600 0x5574d43e4350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3669c2ed59529c44 server_cookie=11b54723e1ce5039 in_seq=0 out_seq=0 2024-09-13T17:24:16.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.654+0000 7fc27ea00640 10 client.12453 ms_handle_connect on v2:172.21.15.28:6833/1114231130 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc27ea00640 1 -- 172.21.15.28:0/3074974428 <== mds.0 v2:172.21.15.28:6833/1114231130 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5574d3fd83c0 con 0x5574d43a8600 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc27ea00640 10 client.12453 handle_command_reply: tid=0 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 client.12453 shutdown 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 msgr2=0x5574d43ab630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 secure :-1 s=READY pgs=2739 cs=0 l=1 rev1=1 crypto rx=0x7fc290081d50 tx=0x7fc290009750 comp rx=0 tx=0).stop 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 shutdown_connections 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43ad660 0x5574d43ab630 unknown :-1 s=CLOSED pgs=2739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43a9b80 0x5574d43ab0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d43a8600 0x5574d43e4350 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/3074974428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43826d0 0x5574d43a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc27ea00640 0 client.12453 ms_handle_reset on v2:172.21.15.28:6833/1114231130 2024-09-13T17:24:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 >> 172.21.15.28:0/3074974428 conn(0x5574d437fe90 msgr2=0x5574d43c7da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 shutdown_connections 2024-09-13T17:24:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.655+0000 7fc2a2e57740 1 -- 172.21.15.28:0/3074974428 wait complete. 2024-09-13T17:24:16.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.656+0000 7fc2a2e57740 20 client.12453 trim_cache size 0 max 16384 2024-09-13T17:24:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.671+0000 7fc2a2e57740 1 Processor -- start 2024-09-13T17:24:16.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.671+0000 7fc2a2e57740 1 -- start start 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43e0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43e0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d42ddaf0 con 0x5574d43826d0 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d442b450 con 0x5574d43ad660 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574d4378df0 con 0x5574d43a9b80 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc29d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc29d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33302/0 (socket says 172.21.15.28:33302) 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc29d600640 1 -- 172.21.15.28:0/785088727 learned_addr learned my addr 172.21.15.28:0/785088727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc29e000640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43e0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.672+0000 7fc29cc00640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 -- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 msgr2=0x5574d43e0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43e0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 -- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 msgr2=0x5574d43e0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43e0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 -- 172.21.15.28:0/785088727 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574d4318170 con 0x5574d43826d0 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29cc00640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc29d600640 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x5574d43adaa0 tx=0x7fc28c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6206fee5ea76cbb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc28aa00640 1 -- 172.21.15.28:0/785088727 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c028070 con 0x5574d43826d0 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc28aa00640 1 -- 172.21.15.28:0/785088727 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c016190 con 0x5574d43826d0 2024-09-13T17:24:16.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.673+0000 7fc28aa00640 1 -- 172.21.15.28:0/785088727 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c016ac0 con 0x5574d43826d0 2024-09-13T17:24:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- 172.21.15.28:0/785088727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 msgr2=0x5574d43ab1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 secure :-1 s=READY pgs=2740 cs=0 l=1 rev1=1 crypto rx=0x5574d43adaa0 tx=0x7fc28c012000 comp rx=0 tx=0).stop 2024-09-13T17:24:16.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- 172.21.15.28:0/785088727 shutdown_connections 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43e0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/785088727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d43ab1e0 unknown :-1 s=CLOSED pgs=2740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- 172.21.15.28:0/785088727 >> 172.21.15.28:0/785088727 conn(0x5574d437fe90 msgr2=0x5574d43c7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- 172.21.15.28:0/785088727 shutdown_connections 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- 172.21.15.28:0/785088727 wait complete. 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.674+0000 7fc2a2e57740 1 -- start start 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.675+0000 7fc2a2e57740 1 Processor -- start 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.675+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43a4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d43e61f0 con 0x5574d43826d0 2024-09-13T17:24:16.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574d43c3af0 con 0x5574d43ad660 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5574d43e0970 con 0x5574d43a9b80 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc2a2e57740 10 client.0 _ll_register_callbacks cb 0x7fc290091070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33312/0 (socket says 172.21.15.28:33312) 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 -- 172.21.15.28:0/2110743539 learned_addr learned my addr 172.21.15.28:0/2110743539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29e000640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29cc00640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 -- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 msgr2=0x5574d43c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43c7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 -- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 msgr2=0x5574d43a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43a4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 -- 172.21.15.28:0/2110743539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c019040 con 0x5574d43826d0 2024-09-13T17:24:16.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29e000640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc29d600640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00f9a0 tx=0x7fc28c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f96f7cb62722bf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.676+0000 7fc27ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c028070 con 0x5574d43826d0 2024-09-13T17:24:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc28c01c040 con 0x5574d43826d0 2024-09-13T17:24:16.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2900782e0 con 0x5574d43826d0 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=188) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc28c007ea0 con 0x5574d43826d0 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 10 client.12459 fetch_fsmap learned FSMap version 188 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=188,monmap=2+}) v3 -- 0x5574d4285470 con 0x5574d43826d0 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900782e0 con 0x5574d43826d0 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 188) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fc28c012070 con 0x5574d43826d0 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 10 client.12459 fetch_fsmap finished waiting for FSMap version 188 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 10 client.12459 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 20 client.12459 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d4382ea0 0x5574d43a4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 4 client.12459 mds_command: new command op to 12318 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:24:16.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.677+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 --> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5574d3fe96c0 con 0x5574d4382ea0 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc29cc00640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d4382ea0 0x5574d43a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc29cc00640 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d4382ea0 0x5574d43a4760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=209224e933228b0f server_cookie=26eeb25fe6475afc in_seq=0 out_seq=0 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc27ea00640 10 client.12459 ms_handle_connect on v2:172.21.15.28:6833/1114231130 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc27ea00640 1 -- 172.21.15.28:0/2110743539 <== mds.0 v2:172.21.15.28:6833/1114231130 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x5574d3fe96c0 con 0x5574d4382ea0 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc27ea00640 10 client.12459 handle_command_reply: tid=0 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc2a2e57740 1 client.12459 shutdown 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 msgr2=0x5574d4380c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.678+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 secure :-1 s=READY pgs=2741 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00f9a0 tx=0x7fc28c001120 comp rx=0 tx=0).stop 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 shutdown_connections 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574d43ad660 0x5574d43a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:6833/1114231130,v1:172.21.15.28:6835/1114231130] conn(0x5574d4382ea0 0x5574d43a4760 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5574d43a9b80 0x5574d43c7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 --2- 172.21.15.28:0/2110743539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5574d43826d0 0x5574d4380c30 unknown :-1 s=CLOSED pgs=2741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 >> 172.21.15.28:0/2110743539 conn(0x5574d437fe90 msgr2=0x5574d43ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc27ea00640 0 client.12459 ms_handle_reset on v2:172.21.15.28:6833/1114231130 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 shutdown_connections 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 1 -- 172.21.15.28:0/2110743539 wait complete. 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a2e57740 20 client.12459 trim_cache size 0 max 16384 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 12318, 2024-09-13T17:24:16.685 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 6.0994752280000002, 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 188, 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 233, 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 233, 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 11.236771000999999 2024-09-13T17:24:16.686 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 msgr2=0x7fc27806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 0x7fc27806d0c0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000f60 tx=0x7fc28c009040 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 msgr2=0x7fc298150330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.679+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 secure :-1 s=READY pgs=2737 cs=0 l=1 rev1=1 crypto rx=0x7fc29002ac50 tx=0x7fc290004a40 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 shutdown_connections 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc27806ac00 0x7fc27806d0c0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc29815dbf0 0x7fc298150330 unknown :-1 s=CLOSED pgs=2737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2980cbdf0 0x7fc2981563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 --2- 172.21.15.28:0/1983504428 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc2980ae580 0x7fc298154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 >> 172.21.15.28:0/1983504428 conn(0x7fc2980b0af0 msgr2=0x7fc2980ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 shutdown_connections 2024-09-13T17:24:16.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.680+0000 7fc2a0000640 1 -- 172.21.15.28:0/1983504428 wait complete. 2024-09-13T17:24:16.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T17:24:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.850+0000 7f9bfa000640 1 Processor -- start 2024-09-13T17:24:16.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.850+0000 7f9bfa000640 1 -- start start 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bfa000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f9bf41592a0 0x7f9bf415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bfa000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4159d90 con 0x7f9bf406f970 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bfa000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40b2040 con 0x7f9bf406c2d0 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bfa000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf415b0b0 con 0x7f9bf41592a0 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bebe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f9bf406f970 0x7f9bf4159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44458/0 (socket says 172.21.15.28:44458) 2024-09-13T17:24:16.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.851+0000 7f9bebe00640 1 -- 172.21.15.28:0/310695639 learned_addr learned my addr 172.21.15.28:0/310695639 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:16.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf3400640 1 --2- 172.21.15.28:0/310695639 >> v2:172.21.15.189:3300/0 conn(0x7f9bf41592a0 0x7f9bf415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442405849 0 0) 0x7f9bf4159d90 con 0x7f9bf406f970 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003520 con 0x7f9bf406f970 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4130393599 0 0) 0x7f9bd8003520 con 0x7f9bf406f970 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 >> v2:172.21.15.189:3300/0 conn(0x7f9bf41592a0 msgr2=0x7f9bf415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 --2- 172.21.15.28:0/310695639 >> v2:172.21.15.189:3300/0 conn(0x7f9bf41592a0 0x7f9bf415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 >> v1:172.21.15.189:6789/0 conn(0x7f9bf406c2d0 legacy=0x7f9bf40b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4159ff0 con 0x7f9bf406f970 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f9bdc001690 con 0x7f9bf406f970 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 >> v1:172.21.15.28:6789/0 conn(0x7f9bf406f970 legacy=0x7f9bf4159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd8003970 0x7f9bd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 0x7f9bd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bd8005180 0x7f9bd800ff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf415b0b0 con 0x7f9bd80047c0 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd80104d0 con 0x7f9bd8003970 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.852+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8010610 con 0x7f9bd8005180 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 0x7f9bd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bf3400640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd8003970 0x7f9bd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bf3e00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bd8005180 0x7f9bd800ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 -- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bd8005180 msgr2=0x7f9bd800ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bd8005180 0x7f9bd800ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 -- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd8003970 msgr2=0x7f9bd8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd8003970 0x7f9bd8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bebe00640 1 -- 172.21.15.28:0/310695639 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd8003520 con 0x7f9bd80047c0 2024-09-13T17:24:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.853+0000 7f9bf3e00640 1 -- 172.21.15.28:0/310695639 reap_dead start 2024-09-13T17:24:16.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.854+0000 7f9bebe00640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 0x7f9bd8004c40 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4159f20 tx=0x7f9bd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aac7000532ab9c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.854+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc005ad0 con 0x7f9bd80047c0 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.854+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bdc02d070 con 0x7f9bd80047c0 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.854+0000 7f9bf2a00640 1 -- 172.21.15.28:0/310695639 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc005e70 con 0x7f9bd80047c0 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 -- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 msgr2=0x7f9bd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 0x7f9bd8004c40 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f9bf4159f20 tx=0x7f9bd80057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 -- 172.21.15.28:0/310695639 shutdown_connections 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 --2- 172.21.15.28:0/310695639 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bd80047c0 0x7f9bd8004c40 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 -- 172.21.15.28:0/310695639 >> 172.21.15.28:0/310695639 conn(0x7f9bf4069e80 msgr2=0x7f9bf406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 -- 172.21.15.28:0/310695639 shutdown_connections 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.855+0000 7f9bfa000640 1 -- 172.21.15.28:0/310695639 wait complete. 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.857+0000 7f9bfa000640 1 Processor -- start 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.857+0000 7f9bfa000640 1 -- start start 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.857+0000 7f9bfa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd80047c0 0x7f9bf40cc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bfa000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bf40ce400 0x7f9bf41ed8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bfa000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bfa000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf40cca20 con 0x7f9bf41eddf0 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bfa000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4153560 con 0x7f9bd80047c0 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bfa000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4152c30 con 0x7f9bf40ce400 2024-09-13T17:24:16.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33332/0 (socket says 172.21.15.28:33332) 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 -- 172.21.15.28:0/1836385241 learned_addr learned my addr 172.21.15.28:0/1836385241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 -- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bf40ce400 msgr2=0x7f9bf41ed8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bf40ce400 0x7f9bf41ed8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 -- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd80047c0 msgr2=0x7f9bf40cc0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd80047c0 0x7f9bf40cc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 -- 172.21.15.28:0/1836385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4152f10 con 0x7f9bf41eddf0 2024-09-13T17:24:16.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.858+0000 7f9bf3e00640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f9bf406d340 tx=0x7f9bec00fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.859+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec00fec0 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.859+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec003b80 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.859+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0050d0 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.859+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf41531f0 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.859+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf40cc7d0 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.860+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf415ab70 con 0x7f9bf41eddf0 2024-09-13T17:24:16.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.861+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9bec005230 con 0x7f9bf41eddf0 2024-09-13T17:24:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.862+0000 7f9beb400640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 0x7f9bd006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.862+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bec0891e0 con 0x7f9bf41eddf0 2024-09-13T17:24:16.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.862+0000 7f9bf3400640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 0x7f9bd006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:16.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.863+0000 7f9bf3400640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 0x7f9bd006d0c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006e80 tx=0x7f9be4006df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:16.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:16.866+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9bec054450 con 0x7f9bf41eddf0 2024-09-13T17:24:17.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.083+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9bf41536a0 con 0x7f9bf41eddf0 2024-09-13T17:24:17.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.084+0000 7f9beb400640 1 -- 172.21.15.28:0/1836385241 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v188) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9bec05ce60 con 0x7f9bf41eddf0 2024-09-13T17:24:17.086 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T17:24:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 msgr2=0x7f9bd006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 0x7f9bd006d0c0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f9be4006e80 tx=0x7f9be4006df0 comp rx=0 tx=0).stop 2024-09-13T17:24:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 msgr2=0x7f9bf41539f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:17.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 secure :-1 s=READY pgs=2744 cs=0 l=1 rev1=1 crypto rx=0x7f9bf406d340 tx=0x7f9bec00fc70 comp rx=0 tx=0).stop 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 shutdown_connections 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f9bd006ac00 0x7f9bd006d0c0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bf41eddf0 0x7f9bf41539f0 unknown :-1 s=CLOSED pgs=2744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f9bf40ce400 0x7f9bf41ed8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 --2- 172.21.15.28:0/1836385241 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9bd80047c0 0x7f9bf40cc0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 >> 172.21.15.28:0/1836385241 conn(0x7f9bf4069e80 msgr2=0x7f9bf40af240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 shutdown_connections 2024-09-13T17:24:17.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:17.086+0000 7f9bfa000640 1 -- 172.21.15.28:0/1836385241 wait complete. 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T17:24:17.104 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:24:17.105 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.105 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-09-13T17:24:17.141 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41724sec preferred_lft 41724sec 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T17:24:17.142 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3401:aaff:fef3:2a67/64 scope link 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 7a:27:b8:47:18:bf brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::7827:b8ff:fe47:18bf/64 scope link 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.143 INFO:teuthology.orchestra.run.smithi028.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T17:24:17.144 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 3e:b6:10:58:96:9d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:24:17.144 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::3cb6:10ff:fe58:969d/64 scope link 2024-09-13T17:24:17.144 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-09-13T17:24:17.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.145 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T17:24:17.234 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-09-13T17:24:17.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.237 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-09-13T17:24:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-09-13T17:24:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T17:24:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-09-13T17:24:17.273 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-09-13T17:24:17.274 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T17:24:17.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec bash -c ' 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-09-13T17:24:17.274 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-09-13T17:24:17.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.419 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:24:17.455 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:24:17.455 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:24:17.455 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:24:17.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.455 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-09-13T17:24:17.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:24:17.552 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T17:24:17.588 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T17:24:17.588 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T17:24:17.588 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:24:17.663 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-09-13T17:24:17.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.663 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:24:17.792 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:24:17.792 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:24:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-09-13T17:24:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-09-13T17:24:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.189:3300,172.21.15.189:3301" 2024-09-13T17:24:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with new device syntax: 0@99459b14-8688-430f-9c7e-baba0c3bf859.cephfs=/ 2024-09-13T17:24:17.819 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.28:3300/172.21.15.189:3300/172.21.15.189:3301" will pass to kernel 2024-09-13T17:24:17.821 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-09-13T17:24:17.821 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-09-13T17:24:17.821 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-09-13T17:24:17.821 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-09-13T17:24:17.822 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-09-13T17:24:17.824 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-09-13T17:24:17.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.825 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T17:24:17.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:17.882 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-09-13T17:24:17.926 INFO:teuthology.orchestra.run.smithi028.stdout:client12468 2024-09-13T17:24:17.926 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:17.926 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:18.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:18.001 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent) 2024-09-13T17:24:18.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:18.086 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/child) 2024-09-13T17:24:18.163 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:18.164 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:18.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:18.239 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=parent/parentfile bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T17:24:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records in 2024-09-13T17:24:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records out 2024-09-13T17:24:18.389 INFO:teuthology.orchestra.run.smithi028.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0731164 s, 86.0 MB/s 2024-09-13T17:24:18.390 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:18.390 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:18.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:18.426 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=parent/child/childfile bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T17:24:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records in 2024-09-13T17:24:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:6+0 records out 2024-09-13T17:24:18.566 INFO:teuthology.orchestra.run.smithi028.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0624033 s, 101 MB/s 2024-09-13T17:24:18.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:24:18.573 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T17:24:18.670 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.671 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T17:24:18.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T17:24:18.673 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T17:24:18.674 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-13T17:24:18.675 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:24:18.676 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.677 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:24:18.678 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:24:18.679 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:24:18.680 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: mds cache max size = 1000 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = False 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: mds log max segments = 1 2024-09-13T17:24:18.681 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T17:24:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-09-13T17:24:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:24:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.682 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.682 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.683 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.684 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-13T17:24:18.685 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-09-13T17:24:18.686 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.687 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.688 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore allocator = bitmap 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 1/20 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 1/20 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-13T17:24:18.689 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 4/10 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.690 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.691 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.692 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.693 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-09-13T17:24:18.694 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: mds cache max size = 1000 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-09-13T17:24:18.695 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = False 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: mds log max segments = 1 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = False 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.696 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:24:18.707 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: chdir = "" 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: auth supported = cephx 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: filestore xattr use omap = true 2024-09-13T17:24:18.802 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: mon clock drift allowed = 1.000 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: osd crush chooseleaf type = 0 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: auth debug = true 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on old message = true 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on bug = true 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: mon pg warn max object skew = 0 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:24:18.803 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default size = 2 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow primary affinity = true 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd allow pg remap = true 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on no sortbitwise = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on too few osds = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: mon_allow_pool_size_one = true 2024-09-13T17:24:18.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: osd default data pool replay window = 5 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: mon allow pool delete = true 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: mon cluster log file level = debug 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: debug asserts on shutdown = true 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: mon health detail to clog = false 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: ms die on skipped message = False 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.805 INFO:teuthology.orchestra.run.smithi028.stdout:[osd] 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd journal size = 100 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub load threshold = 5.0 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd scrub max interval = 600 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd recover clone overlap = true 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd recovery max chunk = 1048576 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug shutdown = true 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug op order = true 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug verify stray on activate = true 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.806 INFO:teuthology.orchestra.run.smithi028.stdout: osd open classes on start = true 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug pg log writeout = true 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: osd map max advance = 10 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: journal zero on create = true 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: filestore apply finisher threads = 3 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: bdev debug aio = true 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: osd debug misdirected ops = true 2024-09-13T17:24:18.807 INFO:teuthology.orchestra.run.smithi028.stdout: bdev async discard = True 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: bdev enable discard = True 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore allocator = bitmap 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore block size = 96636764160 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: bluestore fsck on mount = True 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluefs = 1/20 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: debug bluestore = 1/20 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: debug osd = 20 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: debug rocksdb = 4/10 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd full ratio = 0.9 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:24:18.808 INFO:teuthology.orchestra.run.smithi028.stdout: osd objectstore = bluestore 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout:[mgr] 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: debug mgr = 20 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: mgr/telemetry/nag = false 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.809 INFO:teuthology.orchestra.run.smithi028.stdout:[mon] 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: debug mon = 20 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: debug paxos = 20 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: debug auth = 20 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon data avail warn = 5 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon mgr mkfs grace = 240 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd reporter subtree level = osd 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon osd prime pg temp = true 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:24:18.810 INFO:teuthology.orchestra.run.smithi028.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: # 1m isn't quite enough 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon op complaint time = 120 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: mon warn on pool no app = False 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout:[client] 2024-09-13T17:24:18.811 INFO:teuthology.orchestra.run.smithi028.stdout: rgw cache enabled = true 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable ops log = true 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: rgw enable usage log = true 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: client mount timeout = 600 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: debug client = 20 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: pid file = "" 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.a] 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.b] 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout:[mon.c] 2024-09-13T17:24:18.812 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a] 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c] 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b] 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d] 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout:[mds] 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds = 20 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: debug mds balancer = 20 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: debug ms = 1 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds cache max size = 1000 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug frag = True 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds debug scatterstat = False 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds log max segments = 1 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds op complaint time = 180 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: mds verify scatter = False 2024-09-13T17:24:18.813 INFO:teuthology.orchestra.run.smithi028.stdout: osd op complaint time = 180 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi028.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi028.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout:[global] 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout: chdir = "" 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout: auth supported = cephx 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.814 INFO:teuthology.orchestra.run.smithi189.stdout: filestore xattr use omap = true 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: mon clock drift allowed = 1.000 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: osd crush chooseleaf type = 0 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: auth debug = true 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on old message = true 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on bug = true 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: mon pg warn max object skew = 0 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T17:24:18.815 INFO:teuthology.orchestra.run.smithi189.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default size = 2 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow primary affinity = true 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd allow pg remap = true 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on legacy crush tunables = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on crush straw calc version zero = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on no sortbitwise = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on osd down out interval zero = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on too few osds = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: mon_allow_pool_size_one = true 2024-09-13T17:24:18.816 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: osd default data pool replay window = 5 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: mon allow pool delete = true 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: mon cluster log file level = debug 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: debug asserts on shutdown = true 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: mon health detail to clog = false 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: mon host = "172.21.15.28,172.21.15.189,[v2:172.21.15.189:3301,v1:172.21.15.189:6790]" 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: ms die on skipped message = False 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: fsid = 99459b14-8688-430f-9c7e-baba0c3bf859 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.817 INFO:teuthology.orchestra.run.smithi189.stdout:[osd] 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd journal size = 100 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub load threshold = 5.0 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd scrub max interval = 600 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd mclock profile = high_recovery_ops 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd recover clone overlap = true 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd recovery max chunk = 1048576 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug shutdown = true 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug op order = true 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug verify stray on activate = true 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd open classes on start = true 2024-09-13T17:24:18.818 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug pg log writeout = true 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: osd deep scrub update digest min age = 30 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: osd map max advance = 10 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: journal zero on create = true 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: filestore ondisk finisher threads = 3 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: filestore apply finisher threads = 3 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: bdev debug aio = true 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: osd debug misdirected ops = true 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: bdev async discard = True 2024-09-13T17:24:18.819 INFO:teuthology.orchestra.run.smithi189.stdout: bdev enable discard = True 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore allocator = bitmap 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore block size = 96636764160 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: bluestore fsck on mount = True 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluefs = 1/20 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: debug bluestore = 1/20 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: debug osd = 20 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: debug rocksdb = 4/10 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd full ratio = 0.9 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd nearfull ratio = 0.8 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: osd failsafe full ratio = 0.95 2024-09-13T17:24:18.820 INFO:teuthology.orchestra.run.smithi189.stdout: osd objectstore = bluestore 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout:[mgr] 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug mgr = 20 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: mgr/telemetry/nag = false 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout:[mon] 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.821 INFO:teuthology.orchestra.run.smithi189.stdout: debug mon = 20 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: debug paxos = 20 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: debug auth = 20 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon data avail warn = 5 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon mgr mkfs grace = 240 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min pgs per osd = 4 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd reporter subtree level = osd 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon osd prime pg temp = true 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: mon reweight min bytes per osd = 10 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.822 INFO:teuthology.orchestra.run.smithi189.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: # 1m isn't quite enough 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon_down_mkfs_grace = 2m 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon_warn_on_filestore_osds = false 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon op complaint time = 120 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: mon warn on pool no app = False 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout:[client] 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: rgw cache enabled = true 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable ops log = true 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: rgw enable usage log = true 2024-09-13T17:24:18.823 INFO:teuthology.orchestra.run.smithi189.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: client mount timeout = 600 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: debug client = 20 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: pid file = "" 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.a] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.b] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mon.c] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.a] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.c] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.b] 2024-09-13T17:24:18.824 INFO:teuthology.orchestra.run.smithi189.stdout:[mds.d] 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout:[mds] 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds = 20 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: debug mds balancer = 20 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: debug ms = 1 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds cache max size = 1000 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug frag = True 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds debug scatterstat = False 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds log max segments = 1 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds op complaint time = 180 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: mds verify scatter = False 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: osd op complaint time = 180 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: rados mon op timeout = 900 2024-09-13T17:24:18.825 INFO:teuthology.orchestra.run.smithi189.stdout: rados osd op timeout = 900 2024-09-13T17:24:18.831 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:18.831 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:18.867 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:24:18.867 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:24:18.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:24:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.137+0000 7f3627200640 1 Processor -- start 2024-09-13T17:24:19.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.137+0000 7f3627200640 1 -- start start 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f362015a540 0x7f362006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 0x7f362015e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f362015edb0 0x7f3620070cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36200adb00 con 0x7f362015edb0 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362015e100 con 0x7f362015a540 2024-09-13T17:24:19.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.138+0000 7f3627200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362015e3e0 con 0x7f36200cabc0 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f362015edb0 0x7f3620070cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f362015edb0 0x7f3620070cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33346/0 (socket says 172.21.15.28:33346) 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 -- 172.21.15.28:0/3210547281 learned_addr learned my addr 172.21.15.28:0/3210547281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f361fe00640 1 --2- 172.21.15.28:0/3210547281 >> v2:172.21.15.189:3300/0 conn(0x7f362015a540 0x7f362006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f361f400640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 0x7f362015e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 -- 172.21.15.28:0/3210547281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 msgr2=0x7f362015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 0x7f362015e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 -- 172.21.15.28:0/3210547281 >> v2:172.21.15.189:3300/0 conn(0x7f362015a540 msgr2=0x7f362006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 --2- 172.21.15.28:0/3210547281 >> v2:172.21.15.189:3300/0 conn(0x7f362015a540 0x7f362006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f3625200640 1 -- 172.21.15.28:0/3210547281 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362006ce80 con 0x7f362015edb0 2024-09-13T17:24:19.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.139+0000 7f361f400640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 0x7f362015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.140+0000 7f3625200640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015edb0 0x7f3620070cc0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f36200b3a00 tx=0x7f361802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42efc46c0ad45d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.140+0000 7f361ea00640 1 -- 172.21.15.28:0/3210547281 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618031600 con 0x7f362015edb0 2024-09-13T17:24:19.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.140+0000 7f361ea00640 1 -- 172.21.15.28:0/3210547281 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361802d050 con 0x7f362015edb0 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.140+0000 7f361ea00640 1 -- 172.21.15.28:0/3210547281 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618031980 con 0x7f362015edb0 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 -- 172.21.15.28:0/3210547281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015edb0 msgr2=0x7f3620070cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015edb0 0x7f3620070cc0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f36200b3a00 tx=0x7f361802a610 comp rx=0 tx=0).stop 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 -- 172.21.15.28:0/3210547281 shutdown_connections 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015edb0 0x7f3620070cc0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 --2- 172.21.15.28:0/3210547281 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200cabc0 0x7f362015e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 --2- 172.21.15.28:0/3210547281 >> v2:172.21.15.189:3300/0 conn(0x7f362015a540 0x7f362006c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 -- 172.21.15.28:0/3210547281 >> 172.21.15.28:0/3210547281 conn(0x7f3620069e80 msgr2=0x7f362006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 -- 172.21.15.28:0/3210547281 shutdown_connections 2024-09-13T17:24:19.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.141+0000 7f3627200640 1 -- 172.21.15.28:0/3210547281 wait complete. 2024-09-13T17:24:19.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.143+0000 7f3627200640 1 Processor -- start 2024-09-13T17:24:19.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.143+0000 7f3627200640 1 -- start start 2024-09-13T17:24:19.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36200cabc0 0x7f362014b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200b33c0 0x7f36200acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362014d130 con 0x7f362015a540 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362014bde0 con 0x7f36200cabc0 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3627200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362014c520 con 0x7f36200b33c0 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36200cabc0 0x7f362014b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33354/0 (socket says 172.21.15.28:33354) 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 -- 172.21.15.28:0/3564761856 learned_addr learned my addr 172.21.15.28:0/3564761856 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f3625200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200b33c0 0x7f36200acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 -- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200b33c0 msgr2=0x7f36200acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200b33c0 0x7f36200acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 -- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36200cabc0 msgr2=0x7f362014b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36200cabc0 0x7f362014b8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.144+0000 7f361f400640 1 -- 172.21.15.28:0/3564761856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3618009b30 con 0x7f362015a540 2024-09-13T17:24:19.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f361f400640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f360c00d800 tx=0x7f360c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c00b6c0 con 0x7f362015a540 2024-09-13T17:24:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f360c00b820 con 0x7f362015a540 2024-09-13T17:24:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c00bbd0 con 0x7f362015a540 2024-09-13T17:24:19.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f362014c800 con 0x7f362015a540 2024-09-13T17:24:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.145+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362014cdc0 con 0x7f362015a540 2024-09-13T17:24:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.146+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362014d2e0 con 0x7f362015a540 2024-09-13T17:24:19.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.147+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f360c00bd30 con 0x7f362015a540 2024-09-13T17:24:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.148+0000 7f3617e00640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 0x7f360006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.148+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f360c0897b0 con 0x7f362015a540 2024-09-13T17:24:19.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.148+0000 7f361fe00640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 0x7f360006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.149+0000 7f361fe00640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 0x7f360006d0c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3620159df0 tx=0x7f3610009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.152+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f362014d2e0 con 0x7f362015a540 2024-09-13T17:24:19.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.333+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f362014d2e0 con 0x7f362015a540 2024-09-13T17:24:19.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.334+0000 7f3617e00640 1 -- 172.21.15.28:0/3564761856 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+16523 (secure 0 0 0) 0x7f362014d2e0 con 0x7f362015a540 2024-09-13T17:24:19.336 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:19.336 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":233,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:11.579976+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:24:19.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 msgr2=0x7f360006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 0x7f360006d0c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f3620159df0 tx=0x7f3610009190 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 msgr2=0x7f36200ac850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 secure :-1 s=READY pgs=2747 cs=0 l=1 rev1=1 crypto rx=0x7f360c00d800 tx=0x7f360c00dcc0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 shutdown_connections 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f360006ac00 0x7f360006d0c0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f36200b33c0 0x7f36200acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f362015a540 0x7f36200ac850 unknown :-1 s=CLOSED pgs=2747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 --2- 172.21.15.28:0/3564761856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36200cabc0 0x7f362014b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.336+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 >> 172.21.15.28:0/3564761856 conn(0x7f3620069e80 msgr2=0x7f36200cc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.337+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 shutdown_connections 2024-09-13T17:24:19.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.337+0000 7f3627200640 1 -- 172.21.15.28:0/3564761856 wait complete. 2024-09-13T17:24:19.355 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:24:08.441956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:24:09.571290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:24:19.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:24:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.511+0000 7fc765c00640 1 Processor -- start 2024-09-13T17:24:19.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.511+0000 7fc765c00640 1 -- start start 2024-09-13T17:24:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.512+0000 7fc765c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015ab60 0x7fc76006bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.512+0000 7fc765c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc760071490 0x7fc76006c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.512+0000 7fc765c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc76006c8b0 con 0x7fc76015a780 2024-09-13T17:24:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.512+0000 7fc765c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc76006cb90 con 0x7fc760071490 2024-09-13T17:24:19.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.512+0000 7fc765c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7600cc980 con 0x7fc76015ab60 2024-09-13T17:24:19.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75f400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fc760071490 0x7fc76006c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75ea00640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7fc76015a780 0x7fc760070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:47852/0 (socket says 172.21.15.28:47852) 2024-09-13T17:24:19.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75ea00640 1 -- 172.21.15.28:0/2100910279 learned_addr learned my addr 172.21.15.28:0/2100910279 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:19.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75e000640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015ab60 0x7fc76006bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75f400640 1 -- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015ab60 msgr2=0x7fc76006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.513+0000 7fc75f400640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015ab60 0x7fc76006bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.514+0000 7fc75f400640 1 -- 172.21.15.28:0/2100910279 >> v1:172.21.15.189:6789/0 conn(0x7fc76015a780 legacy=0x7fc760070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.514+0000 7fc75f400640 1 -- 172.21.15.28:0/2100910279 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7600ccc60 con 0x7fc760071490 2024-09-13T17:24:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.515+0000 7fc75f400640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc76006c080 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc758009d90 tx=0x7fc75802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e95b1edc952582bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.515+0000 7fc75d600640 1 -- 172.21.15.28:0/2100910279 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758007740 con 0x7fc760071490 2024-09-13T17:24:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.515+0000 7fc75d600640 1 -- 172.21.15.28:0/2100910279 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc75802d050 con 0x7fc760071490 2024-09-13T17:24:19.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.515+0000 7fc75d600640 1 -- 172.21.15.28:0/2100910279 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc758007ac0 con 0x7fc760071490 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 -- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 msgr2=0x7fc76006c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc76006c080 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fc758009d90 tx=0x7fc75802a660 comp rx=0 tx=0).stop 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 -- 172.21.15.28:0/2100910279 shutdown_connections 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc76006c080 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 --2- 172.21.15.28:0/2100910279 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015ab60 0x7fc76006bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 -- 172.21.15.28:0/2100910279 >> 172.21.15.28:0/2100910279 conn(0x7fc7600b0270 msgr2=0x7fc7600b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 -- 172.21.15.28:0/2100910279 shutdown_connections 2024-09-13T17:24:19.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.516+0000 7fc765c00640 1 -- 172.21.15.28:0/2100910279 wait complete. 2024-09-13T17:24:19.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.518+0000 7fc765c00640 1 Processor -- start 2024-09-13T17:24:19.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.518+0000 7fc765c00640 1 -- start start 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 0x7fc760154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc76015ab60 0x7fc76006b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760142010 con 0x7fc76015ab60 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760142280 con 0x7fc760071490 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc765c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc760142e00 con 0x7fc76015a780 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 0x7fc760154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc76015ab60 0x7fc76006b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75e000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 0x7fc760154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:35034/0 (socket says 172.21.15.28:35034) 2024-09-13T17:24:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:47506/0 (socket says 172.21.15.28:47506) 2024-09-13T17:24:19.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.519+0000 7fc75e000640 1 -- 172.21.15.28:0/704643512 learned_addr learned my addr 172.21.15.28:0/704643512 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 -- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 msgr2=0x7fc760154580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 0x7fc760154580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 -- 172.21.15.28:0/704643512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc76015ab60 msgr2=0x7fc76006b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc76015ab60 0x7fc76006b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 -- 172.21.15.28:0/704643512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc758009a70 con 0x7fc760071490 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc75ea00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c470 tx=0x7fc750001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc750005ce0 con 0x7fc760071490 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc750005e40 con 0x7fc760071490 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7601424d0 con 0x7fc760071490 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7500072d0 con 0x7fc760071490 2024-09-13T17:24:19.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.520+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc760142a90 con 0x7fc760071490 2024-09-13T17:24:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.521+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc760070d80 con 0x7fc760071490 2024-09-13T17:24:19.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.522+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fc750007430 con 0x7fc760071490 2024-09-13T17:24:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.523+0000 7fc756a00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 0x7fc74006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.523+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc7500887d0 con 0x7fc760071490 2024-09-13T17:24:19.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.523+0000 7fc75e000640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 0x7fc74006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.524+0000 7fc75e000640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 0x7fc74006d0f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fc7600703f0 tx=0x7fc74c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.529+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fc750053a50 con 0x7fc760071490 2024-09-13T17:24:19.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.707+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc760157920 con 0x7fc760071490 2024-09-13T17:24:19.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.708+0000 7fc756a00640 1 -- 172.21.15.28:0/704643512 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v233) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc750057070 con 0x7fc760071490 2024-09-13T17:24:19.710 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.710+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 msgr2=0x7fc74006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.710+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 0x7fc74006d0f0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fc7600703f0 tx=0x7fc74c009190 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 msgr2=0x7fc760141ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fc75000c470 tx=0x7fc750001e70 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 shutdown_connections 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fc74006ac30 0x7fc74006d0f0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc76015ab60 0x7fc76006b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fc76015a780 0x7fc760154580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 --2- 172.21.15.28:0/704643512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc760071490 0x7fc760141ad0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 >> 172.21.15.28:0/704643512 conn(0x7fc7600b0270 msgr2=0x7fc7600cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 shutdown_connections 2024-09-13T17:24:19.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.711+0000 7fc765c00640 1 -- 172.21.15.28:0/704643512 wait complete. 2024-09-13T17:24:19.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:24:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.884+0000 7fb8f7200640 1 Processor -- start 2024-09-13T17:24:19.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.884+0000 7fb8f7200640 1 -- start start 2024-09-13T17:24:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.885+0000 7fb8f7200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fb8f0070430 0x7fb8f015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.885+0000 7fb8f7200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f015ab60 con 0x7fb8f0159770 2024-09-13T17:24:19.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.885+0000 7fb8f7200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f006f920 con 0x7fb8f006f3f0 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.885+0000 7fb8f7200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f015b290 con 0x7fb8f0070430 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8efe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb8f0159770 0x7fb8f006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44462/0 (socket says 172.21.15.28:44462) 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8efe00640 1 -- 172.21.15.28:0/3159349690 learned_addr learned my addr 172.21.15.28:0/3159349690 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8ef400640 1 --2- 172.21.15.28:0/3159349690 >> v2:172.21.15.189:3300/0 conn(0x7fb8f0070430 0x7fb8f015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172843348 0 0) 0x7fb8e0001110 con 0x7fb8f0159770 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8d4003520 con 0x7fb8f0159770 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875806500 0 0) 0x7fb8d4003520 con 0x7fb8f0159770 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8ef400640 1 -- 172.21.15.28:0/3159349690 >> v1:172.21.15.189:6789/0 conn(0x7fb8f006f3f0 legacy=0x7fb8f015c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8ef400640 1 -- 172.21.15.28:0/3159349690 >> v1:172.21.15.28:6789/0 conn(0x7fb8f0159770 legacy=0x7fb8f006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.886+0000 7fb8ef400640 1 -- 172.21.15.28:0/3159349690 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f015b570 con 0x7fb8f0070430 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.887+0000 7fb8ef400640 1 --2- 172.21.15.28:0/3159349690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0070430 0x7fb8f015bd70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0097a0 tx=0x7fb8dc02c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e9527792a50f6a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e0001800 con 0x7fb8f0070430 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8dc02cdd0 con 0x7fb8f0070430 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8eea00640 1 -- 172.21.15.28:0/3159349690 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc034660 con 0x7fb8f0070430 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 -- 172.21.15.28:0/3159349690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0070430 msgr2=0x7fb8f015bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 --2- 172.21.15.28:0/3159349690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0070430 0x7fb8f015bd70 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0097a0 tx=0x7fb8dc02c340 comp rx=0 tx=0).stop 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 -- 172.21.15.28:0/3159349690 shutdown_connections 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 --2- 172.21.15.28:0/3159349690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f0070430 0x7fb8f015bd70 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 -- 172.21.15.28:0/3159349690 >> 172.21.15.28:0/3159349690 conn(0x7fb8f00b0270 msgr2=0x7fb8f00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 -- 172.21.15.28:0/3159349690 shutdown_connections 2024-09-13T17:24:19.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.888+0000 7fb8f7200640 1 -- 172.21.15.28:0/3159349690 wait complete. 2024-09-13T17:24:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.890+0000 7fb8f7200640 1 Processor -- start 2024-09-13T17:24:19.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.890+0000 7fb8f7200640 1 -- start start 2024-09-13T17:24:19.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f006f3f0 0x7fb8f0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8f0070430 0x7fb8f01ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0154bd0 con 0x7fb8f0159770 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0151550 con 0x7fb8f006f3f0 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f7200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01ef030 con 0x7fb8f0070430 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f5200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33374/0 (socket says 172.21.15.28:33374) 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.891+0000 7fb8f5200640 1 -- 172.21.15.28:0/1896604150 learned_addr learned my addr 172.21.15.28:0/1896604150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8ef400640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8f0070430 0x7fb8f01ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8efe00640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f006f3f0 0x7fb8f0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 -- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8f0070430 msgr2=0x7fb8f01ee6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8f0070430 0x7fb8f01ee6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 -- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f006f3f0 msgr2=0x7fb8f0151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f006f3f0 0x7fb8f0151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 -- 172.21.15.28:0/1896604150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc009400 con 0x7fb8f0159770 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f5200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c7a0 tx=0x7fb8e800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e8003b70 con 0x7fb8f0159770 2024-09-13T17:24:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e8003cf0 con 0x7fb8f0159770 2024-09-13T17:24:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f01ef310 con 0x7fb8f0159770 2024-09-13T17:24:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e800aa90 con 0x7fb8f0159770 2024-09-13T17:24:19.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.892+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f01547f0 con 0x7fb8f0159770 2024-09-13T17:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.894+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f0154d10 con 0x7fb8f0159770 2024-09-13T17:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.894+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb8e800abf0 con 0x7fb8f0159770 2024-09-13T17:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.895+0000 7fb8e7e00640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 0x7fb8d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.895+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8e8089100 con 0x7fb8f0159770 2024-09-13T17:24:19.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.896+0000 7fb8efe00640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 0x7fb8d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:19.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.896+0000 7fb8efe00640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 0x7fb8d006d0c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0001930 tx=0x7fb8e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:19.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:19.899+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fb8f0154d10 con 0x7fb8f0159770 2024-09-13T17:24:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.081+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8f0154d10 con 0x7fb8f0159770 2024-09-13T17:24:20.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.081+0000 7fb8e7e00640 1 -- 172.21.15.28:0/1896604150 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v233) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb8f0154d10 con 0x7fb8f0159770 2024-09-13T17:24:20.087 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 msgr2=0x7fb8d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 0x7fb8d006d0c0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0001930 tx=0x7fb8e0009040 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 msgr2=0x7fb8f00af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c7a0 tx=0x7fb8e800cc60 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 shutdown_connections 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb8d006ac00 0x7fb8d006d0c0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb8f0159770 0x7fb8f00af270 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb8f0070430 0x7fb8f01ee6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 --2- 172.21.15.28:0/1896604150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8f006f3f0 0x7fb8f0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.083+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 >> 172.21.15.28:0/1896604150 conn(0x7fb8f00b0270 msgr2=0x7fb8f00b2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:20.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.084+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 shutdown_connections 2024-09-13T17:24:20.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.084+0000 7fb8f7200640 1 -- 172.21.15.28:0/1896604150 wait complete. 2024-09-13T17:24:20.111 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:24:20.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.279+0000 7f2ee2200640 1 Processor -- start 2024-09-13T17:24:20.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.279+0000 7f2ee2200640 1 -- start start 2024-09-13T17:24:20.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2ee2200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2edc159770 0x7f2edc06fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2ee2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc15cd80 0x7f2edc070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2ee2200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc06f920 con 0x7f2edc06f3f0 2024-09-13T17:24:20.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2ee2200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc15b290 con 0x7f2edc159770 2024-09-13T17:24:20.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2ee2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc15b570 con 0x7f2edc15cd80 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2edb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2edc159770 0x7f2edc06fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2edb400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2edc159770 0x7f2edc06fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33388/0 (socket says 172.21.15.28:33388) 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.280+0000 7f2edb400640 1 -- 172.21.15.28:0/2846959895 learned_addr learned my addr 172.21.15.28:0/2846959895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.281+0000 7f2edbe00640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc15cd80 0x7f2edc070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.281+0000 7f2edb400640 1 -- 172.21.15.28:0/2846959895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc15cd80 msgr2=0x7f2edc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.281+0000 7f2edb400640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc15cd80 0x7f2edc070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.281+0000 7f2edb400640 1 -- 172.21.15.28:0/2846959895 >> v1:172.21.15.189:6789/0 conn(0x7f2edc06f3f0 legacy=0x7f2edc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.281+0000 7f2edb400640 1 -- 172.21.15.28:0/2846959895 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc15b850 con 0x7f2edc159770 2024-09-13T17:24:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.282+0000 7f2edb400640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc159770 0x7f2edc06fd20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009870 tx=0x7f2ecc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2301ee351b74a187 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.282+0000 7f2edaa00640 1 -- 172.21.15.28:0/2846959895 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc032440 con 0x7f2edc159770 2024-09-13T17:24:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.282+0000 7f2edaa00640 1 -- 172.21.15.28:0/2846959895 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc0325a0 con 0x7f2edc159770 2024-09-13T17:24:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.282+0000 7f2edaa00640 1 -- 172.21.15.28:0/2846959895 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc032930 con 0x7f2edc159770 2024-09-13T17:24:20.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 -- 172.21.15.28:0/2846959895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc159770 msgr2=0x7f2edc06fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc159770 0x7f2edc06fd20 secure :-1 s=READY pgs=2750 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc009870 tx=0x7f2ecc02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 -- 172.21.15.28:0/2846959895 shutdown_connections 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc15cd80 0x7f2edc070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 --2- 172.21.15.28:0/2846959895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc159770 0x7f2edc06fd20 unknown :-1 s=CLOSED pgs=2750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 -- 172.21.15.28:0/2846959895 >> 172.21.15.28:0/2846959895 conn(0x7f2edc0b0270 msgr2=0x7f2edc0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 -- 172.21.15.28:0/2846959895 shutdown_connections 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.283+0000 7f2ee2200640 1 -- 172.21.15.28:0/2846959895 wait complete. 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.284+0000 7f2ee2200640 1 Processor -- start 2024-09-13T17:24:20.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 -- start start 2024-09-13T17:24:20.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc159770 0x7f2edc1f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 0x7f2edc0b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1f33b0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc156150 con 0x7f2edc15cd80 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.285+0000 7f2ee2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc1f2df0 con 0x7f2edc159770 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33398/0 (socket says 172.21.15.28:33398) 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 -- 172.21.15.28:0/486624782 learned_addr learned my addr 172.21.15.28:0/486624782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2ed2a00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc159770 0x7f2edc1f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edbe00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 0x7f2edc0b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 -- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc159770 msgr2=0x7f2edc1f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc159770 0x7f2edc1f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 -- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 msgr2=0x7f2edc0b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 0x7f2edc0b2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 -- 172.21.15.28:0/486624782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc0094d0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edbe00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 0x7f2edc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.286+0000 7f2edb400640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc02a200 tx=0x7f2ecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.287+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc032440 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.287+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc030b80 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.287+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc03d440 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.287+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc1f30d0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.287+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc1e22a0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.288+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc0008d0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.289+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2ecc039040 con 0x7f2edc06f3f0 2024-09-13T17:24:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.290+0000 7f2ed3e00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 0x7f2eb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.290+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2ecc0a9df0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.290+0000 7f2ed2a00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 0x7f2eb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.291+0000 7f2ed2a00640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 0x7f2eb806d0c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0cdee0 tx=0x7f2ec4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.294+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2ecc074fc0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.475+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2edc1e2490 con 0x7f2edc06f3f0 2024-09-13T17:24:20.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.475+0000 7f2ed3e00640 1 -- 172.21.15.28:0/486624782 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v233) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ecc0785e0 con 0x7f2edc06f3f0 2024-09-13T17:24:20.478 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.477+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 msgr2=0x7f2eb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.477+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 0x7f2eb806d0c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0cdee0 tx=0x7f2ec4009040 comp rx=0 tx=0).stop 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.477+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 msgr2=0x7f2edc155c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.477+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc02a200 tx=0x7f2ecc0046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 shutdown_connections 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2eb806ac00 0x7f2eb806d0c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2edc15cd80 0x7f2edc0b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2edc159770 0x7f2edc1f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 --2- 172.21.15.28:0/486624782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2edc06f3f0 0x7f2edc155c10 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 >> 172.21.15.28:0/486624782 conn(0x7f2edc0b0270 msgr2=0x7f2edc15a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 shutdown_connections 2024-09-13T17:24:20.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.478+0000 7f2ee2200640 1 -- 172.21.15.28:0/486624782 wait complete. 2024-09-13T17:24:20.495 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T17:24:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.652+0000 7f6bc5000640 1 Processor -- start 2024-09-13T17:24:20.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.652+0000 7f6bc5000640 1 -- start start 2024-09-13T17:24:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bc5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 0x7f6bc0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bc5000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc006c8b0 con 0x7f6bc006c250 2024-09-13T17:24:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bc5000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bc006cb90 con 0x7f6bc015ab60 2024-09-13T17:24:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bc5000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc006c6a0 con 0x7f6bc015a780 2024-09-13T17:24:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbe800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6bc006c250 0x7f6bc00cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44468/0 (socket says 172.21.15.28:44468) 2024-09-13T17:24:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbe800640 1 -- 172.21.15.28:0/358104981 learned_addr learned my addr 172.21.15.28:0/358104981 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbde00640 1 --2- 172.21.15.28:0/358104981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 0x7f6bc0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607049500 0 0) 0x7f6bc006c8b0 con 0x7f6bc006c250 2024-09-13T17:24:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ba4003520 con 0x7f6bc006c250 2024-09-13T17:24:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660901799 0 0) 0x7f6bc006cb90 con 0x7f6bc015ab60 2024-09-13T17:24:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bc006c8b0 con 0x7f6bc015ab60 2024-09-13T17:24:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbde00640 1 -- 172.21.15.28:0/358104981 >> v1:172.21.15.189:6789/0 conn(0x7f6bc015ab60 legacy=0x7f6bc006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbde00640 1 -- 172.21.15.28:0/358104981 >> v1:172.21.15.28:6789/0 conn(0x7f6bc006c250 legacy=0x7f6bc00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059536123 0 0) 0x7f6ba4003520 con 0x7f6bc006c250 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.654+0000 7f6bbde00640 1 -- 172.21.15.28:0/358104981 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc00cc980 con 0x7f6bc015a780 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.655+0000 7f6bbde00640 1 --2- 172.21.15.28:0/358104981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 0x7f6bc0070d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00097d0 tx=0x7f6bb002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7761bd0f860157ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb80019f0 con 0x7f6bc015a780 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb002ac30 con 0x7f6bc015a780 2024-09-13T17:24:20.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bbd400640 1 -- 172.21.15.28:0/358104981 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0032440 con 0x7f6bc015a780 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 -- 172.21.15.28:0/358104981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 msgr2=0x7f6bc0070d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 --2- 172.21.15.28:0/358104981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 0x7f6bc0070d80 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00097d0 tx=0x7f6bb002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 -- 172.21.15.28:0/358104981 shutdown_connections 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 --2- 172.21.15.28:0/358104981 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015a780 0x7f6bc0070d80 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 -- 172.21.15.28:0/358104981 >> 172.21.15.28:0/358104981 conn(0x7f6bc00b0270 msgr2=0x7f6bc00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 -- 172.21.15.28:0/358104981 shutdown_connections 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.656+0000 7f6bc5000640 1 -- 172.21.15.28:0/358104981 wait complete. 2024-09-13T17:24:20.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.658+0000 7f6bc5000640 1 Processor -- start 2024-09-13T17:24:20.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.658+0000 7f6bc5000640 1 -- start start 2024-09-13T17:24:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 0x7f6bc0141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015ab60 0x7f6bc0143060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0141650 con 0x7f6bc015a780 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc01438d0 con 0x7f6bc006c250 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bc5000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc0143bb0 con 0x7f6bc015ab60 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bb5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bb5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33414/0 (socket says 172.21.15.28:33414) 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.659+0000 7f6bb5400640 1 -- 172.21.15.28:0/1653830078 learned_addr learned my addr 172.21.15.28:0/1653830078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bbde00640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 0x7f6bc0141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bbe800640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015ab60 0x7f6bc0143060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 -- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015ab60 msgr2=0x7f6bc0143060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015ab60 0x7f6bc0143060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 -- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 msgr2=0x7f6bc0141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 0x7f6bc0141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 -- 172.21.15.28:0/1653830078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0009400 con 0x7f6bc015a780 2024-09-13T17:24:20.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bbde00640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 0x7f6bc0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb5400640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c3c0 tx=0x7f6ba800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba80052d0 con 0x7f6bc015a780 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba8005450 con 0x7f6bc015a780 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8003830 con 0x7f6bc015a780 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.660+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc0143e90 con 0x7f6bc015a780 2024-09-13T17:24:20.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.661+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc0144450 con 0x7f6bc015a780 2024-09-13T17:24:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.663+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc0144670 con 0x7f6bc015a780 2024-09-13T17:24:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.664+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6ba8003990 con 0x7f6bc015a780 2024-09-13T17:24:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.664+0000 7f6bb6a00640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 0x7f6b9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:20.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.664+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ba8088900 con 0x7f6bc015a780 2024-09-13T17:24:20.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.664+0000 7f6bbde00640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 0x7f6b9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:20.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.666+0000 7f6bbde00640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 0x7f6b9c06d0c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0034000 tx=0x7f6bb002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.667+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6ba8053ba0 con 0x7f6bc015a780 2024-09-13T17:24:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.857+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6bc00676b0 con 0x7f6bc015a780 2024-09-13T17:24:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.857+0000 7f6bb6a00640 1 -- 172.21.15.28:0/1653830078 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v233) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6ba8017080 con 0x7f6bc015a780 2024-09-13T17:24:20.860 INFO:teuthology.orchestra.run.smithi028.stderr:listed 0 entries 2024-09-13T17:24:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 msgr2=0x7f6b9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 0x7f6b9c06d0c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0034000 tx=0x7f6bb002e000 comp rx=0 tx=0).stop 2024-09-13T17:24:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 msgr2=0x7f6bc0142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 secure :-1 s=READY pgs=2753 cs=0 l=1 rev1=1 crypto rx=0x7f6ba800c3c0 tx=0x7f6ba800c880 comp rx=0 tx=0).stop 2024-09-13T17:24:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 shutdown_connections 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6b9c06ac00 0x7f6b9c06d0c0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6bc015ab60 0x7f6bc0143060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6bc015a780 0x7f6bc0142b20 unknown :-1 s=CLOSED pgs=2753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 --2- 172.21.15.28:0/1653830078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bc006c250 0x7f6bc0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 >> 172.21.15.28:0/1653830078 conn(0x7f6bc00b0270 msgr2=0x7f6bc00cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 shutdown_connections 2024-09-13T17:24:20.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:20.859+0000 7f6bc5000640 1 -- 172.21.15.28:0/1653830078 wait complete. 2024-09-13T17:24:20.878 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-09-13T17:24:20.878 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:20.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-09-13T17:24:20.977 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:24:20.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:24:20.978 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T17:24:21.022 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.219+0000 7f682ae00640 1 Processor -- start 2024-09-13T17:24:21.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.219+0000 7f682ae00640 1 -- start start 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.220+0000 7f682ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 0x7f682415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f682ae00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f682406c2d0 0x7f682406c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f682ae00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6824159d90 con 0x7f682406f970 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f682ae00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f682415b0b0 con 0x7f682406c2d0 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f682ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682406ccc0 con 0x7f68241592a0 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f681be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f682406f970 0x7f6824159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44474/0 (socket says 172.21.15.28:44474) 2024-09-13T17:24:21.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f681be00640 1 -- 172.21.15.28:0/2094193732 learned_addr learned my addr 172.21.15.28:0/2094193732 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f6828e00640 1 --2- 172.21.15.28:0/2094193732 >> v2:172.21.15.189:3300/0 conn(0x7f682406c2d0 0x7f682406c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f6823e00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 0x7f682415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585409207 0 0) 0x7f6824159d90 con 0x7f682406f970 2024-09-13T17:24:21.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.221+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6808005730 con 0x7f682406f970 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3862049122 0 0) 0x7f6808005730 con 0x7f682406f970 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6828e00640 1 -- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 msgr2=0x7f682415ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6828e00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 0x7f682415ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6828e00640 1 -- 172.21.15.28:0/2094193732 >> v1:172.21.15.28:6789/0 conn(0x7f682406f970 legacy=0x7f6824159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6828e00640 1 -- 172.21.15.28:0/2094193732 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68240b2040 con 0x7f682406c2d0 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.222+0000 7f6823e00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 0x7f682415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.223+0000 7f6828e00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406c2d0 0x7f682406c780 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f681c009d90 tx=0x7f681c02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ed1104fe3edbac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.223+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c001690 con 0x7f682406c2d0 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.223+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c02d050 con 0x7f682406c2d0 2024-09-13T17:24:21.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.223+0000 7f6823400640 1 -- 172.21.15.28:0/2094193732 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c007760 con 0x7f682406c2d0 2024-09-13T17:24:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 -- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406c2d0 msgr2=0x7f682406c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406c2d0 0x7f682406c780 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f681c009d90 tx=0x7f681c02a5f0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 -- 172.21.15.28:0/2094193732 shutdown_connections 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406c2d0 0x7f682406c780 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 --2- 172.21.15.28:0/2094193732 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f68241592a0 0x7f682415ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 -- 172.21.15.28:0/2094193732 >> 172.21.15.28:0/2094193732 conn(0x7f6824069e80 msgr2=0x7f682406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 -- 172.21.15.28:0/2094193732 shutdown_connections 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.224+0000 7f682ae00640 1 -- 172.21.15.28:0/2094193732 wait complete. 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.225+0000 7f682ae00640 1 Processor -- start 2024-09-13T17:24:21.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.226+0000 7f682ae00640 1 -- start start 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.226+0000 7f682ae00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f682406c2d0 0x7f6824151150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f682ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406f970 0x7f6824150530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f682ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f682ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824151a10 con 0x7f68241592a0 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f682ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682414fa50 con 0x7f682406f970 2024-09-13T17:24:21.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f682ae00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682414fd30 con 0x7f682406c2d0 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33416/0 (socket says 172.21.15.28:33416) 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 -- 172.21.15.28:0/2208052474 learned_addr learned my addr 172.21.15.28:0/2208052474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6823e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f682406c2d0 0x7f6824151150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f681be00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406f970 0x7f6824150530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 -- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f682406c2d0 msgr2=0x7f6824151150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f682406c2d0 0x7f6824151150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 -- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406f970 msgr2=0x7f6824150530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406f970 0x7f6824150530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.227+0000 7f6828e00640 1 -- 172.21.15.28:0/2208052474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814009400 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f6828e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f681c009d90 tx=0x7f681c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c03c830 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681c03c990 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681c03cd00 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c009a70 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.228+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6824151cf0 con 0x7f68241592a0 2024-09-13T17:24:21.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.229+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6824150250 con 0x7f68241592a0 2024-09-13T17:24:21.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.230+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f681c02ad50 con 0x7f68241592a0 2024-09-13T17:24:21.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.231+0000 7f6820c00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 0x7f680006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.231+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f681c0aa7c0 con 0x7f68241592a0 2024-09-13T17:24:21.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.231+0000 7f6823e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 0x7f680006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.232+0000 7f6823e00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 0x7f680006d0c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f6814009c30 tx=0x7f6814009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.235+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f6824150250 con 0x7f68241592a0 2024-09-13T17:24:21.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.451+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6824150250 con 0x7f68241592a0 2024-09-13T17:24:21.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.451+0000 7f6820c00640 1 -- 172.21.15.28:0/2208052474 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+4750 (secure 0 0 0) 0x7f6824150250 con 0x7f68241592a0 2024-09-13T17:24:21.453 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:21.453 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":188,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:13.584112+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12318},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12318":{"gid":12318,"name":"a","rank":0,"incarnation":185,"state":"up:active","state_seq":3,"addr":"172.21.15.28:6835/1114231130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":1114231130},{"type":"v1","addr":"172.21.15.28:6835","nonce":1114231130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:21.453 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 188 2024-09-13T17:24:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 msgr2=0x7f680006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 0x7f680006d0c0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f6814009c30 tx=0x7f6814009190 comp rx=0 tx=0).stop 2024-09-13T17:24:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 msgr2=0x7f6824148c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f681c009d90 tx=0x7f681c004a40 comp rx=0 tx=0).stop 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 shutdown_connections 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f680006ac00 0x7f680006d0c0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f68241592a0 0x7f6824148c30 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f682406f970 0x7f6824150530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 --2- 172.21.15.28:0/2208052474 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f682406c2d0 0x7f6824151150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 >> 172.21.15.28:0/2208052474 conn(0x7f6824069e80 msgr2=0x7f682406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 shutdown_connections 2024-09-13T17:24:21.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.454+0000 7f682ae00640 1 -- 172.21.15.28:0/2208052474 wait complete. 2024-09-13T17:24:21.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-13T17:24:21.653 INFO:teuthology.orchestra.run.smithi028.stdout:{"message":"","return_code":0} 2024-09-13T17:24:21.654 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T17:24:21.654 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:24:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.809+0000 7f1aefe00640 1 Processor -- start 2024-09-13T17:24:21.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.809+0000 7f1aefe00640 1 -- start start 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aefe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1af00b0e10 0x7f1af0159a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aefe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1af00b2870 0x7f1af00b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aefe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af00b32f0 con 0x7f1af006c370 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aefe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af00cbac0 con 0x7f1af00b2870 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aefe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1af00b35d0 con 0x7f1af00b0e10 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aef400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1af006c370 0x7f1af006c780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44490/0 (socket says 172.21.15.28:44490) 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1aef400640 1 -- 172.21.15.28:0/2146252081 learned_addr learned my addr 172.21.15.28:0/2146252081 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:21.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.810+0000 7f1ae6000640 1 --2- 172.21.15.28:0/2146252081 >> v2:172.21.15.28:3300/0 conn(0x7f1af00b2870 0x7f1af00b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1aeea00640 1 --2- 172.21.15.28:0/2146252081 >> v2:172.21.15.189:3300/0 conn(0x7f1af00b0e10 0x7f1af0159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1aee000640 1 -- 172.21.15.28:0/2146252081 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1213898494 0 0) 0x7f1af00b32f0 con 0x7f1af006c370 2024-09-13T17:24:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1ae6000640 1 -- 172.21.15.28:0/2146252081 >> v2:172.21.15.189:3300/0 conn(0x7f1af00b0e10 msgr2=0x7f1af0159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1ae6000640 1 --2- 172.21.15.28:0/2146252081 >> v2:172.21.15.189:3300/0 conn(0x7f1af00b0e10 0x7f1af0159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1ae6000640 1 -- 172.21.15.28:0/2146252081 >> v1:172.21.15.28:6789/0 conn(0x7f1af006c370 legacy=0x7f1af006c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.811+0000 7f1ae6000640 1 -- 172.21.15.28:0/2146252081 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af00b38b0 con 0x7f1af00b2870 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.812+0000 7f1ae6000640 1 --2- 172.21.15.28:0/2146252081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b2870 0x7f1af00b2ce0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f1af006cf00 tx=0x7f1ad802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=480194971f7f891b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.812+0000 7f1aee000640 1 -- 172.21.15.28:0/2146252081 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad802ae60 con 0x7f1af00b2870 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.812+0000 7f1aee000640 1 -- 172.21.15.28:0/2146252081 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad8032440 con 0x7f1af00b2870 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.812+0000 7f1aee000640 1 -- 172.21.15.28:0/2146252081 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80327d0 con 0x7f1af00b2870 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 -- 172.21.15.28:0/2146252081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b2870 msgr2=0x7f1af00b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 --2- 172.21.15.28:0/2146252081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b2870 0x7f1af00b2ce0 secure :-1 s=READY pgs=2757 cs=0 l=1 rev1=1 crypto rx=0x7f1af006cf00 tx=0x7f1ad802a110 comp rx=0 tx=0).stop 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 -- 172.21.15.28:0/2146252081 shutdown_connections 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 --2- 172.21.15.28:0/2146252081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b2870 0x7f1af00b2ce0 unknown :-1 s=CLOSED pgs=2757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 --2- 172.21.15.28:0/2146252081 >> v2:172.21.15.189:3300/0 conn(0x7f1af00b0e10 0x7f1af0159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 -- 172.21.15.28:0/2146252081 >> 172.21.15.28:0/2146252081 conn(0x7f1af0073470 msgr2=0x7f1af0069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 -- 172.21.15.28:0/2146252081 shutdown_connections 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.813+0000 7f1aefe00640 1 -- 172.21.15.28:0/2146252081 wait complete. 2024-09-13T17:24:21.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.814+0000 7f1aefe00640 1 Processor -- start 2024-09-13T17:24:21.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.815+0000 7f1aefe00640 1 -- start start 2024-09-13T17:24:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.815+0000 7f1aefe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af006c370 0x7f1af013d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.815+0000 7f1aefe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.815+0000 7f1aefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af00b2870 0x7f1af00c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1aefe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af00c9f00 con 0x7f1af00b0e10 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1aefe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1af00ca1e0 con 0x7f1af00b2870 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1aefe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1af00ca4c0 con 0x7f1af006c370 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33446/0 (socket says 172.21.15.28:33446) 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 -- 172.21.15.28:0/3526455094 learned_addr learned my addr 172.21.15.28:0/3526455094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1aef400640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af00b2870 0x7f1af00c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1aeea00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af006c370 0x7f1af013d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 -- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af006c370 msgr2=0x7f1af013d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af006c370 0x7f1af013d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 -- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af00b2870 msgr2=0x7f1af00c9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:21.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af00b2870 0x7f1af00c9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.816+0000 7f1ae6000640 1 -- 172.21.15.28:0/3526455094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8009400 con 0x7f1af00b0e10 2024-09-13T17:24:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1ae6000640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000c00 tx=0x7f1ad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803d8f0 con 0x7f1af00b0e10 2024-09-13T17:24:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad803da70 con 0x7f1af00b0e10 2024-09-13T17:24:21.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803de20 con 0x7f1af00b0e10 2024-09-13T17:24:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af015a330 con 0x7f1af00b0e10 2024-09-13T17:24:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.817+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af00c9bd0 con 0x7f1af00b0e10 2024-09-13T17:24:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.818+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af015a7f0 con 0x7f1af00b0e10 2024-09-13T17:24:21.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.819+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1ad8039040 con 0x7f1af00b0e10 2024-09-13T17:24:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.820+0000 7f1ae7400640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 0x7f1acc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.820+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ad80a9f10 con 0x7f1af00b0e10 2024-09-13T17:24:21.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.820+0000 7f1aeea00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 0x7f1acc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:21.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.821+0000 7f1aeea00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 0x7f1acc06d0f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009ad0 tx=0x7f1ae0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:21.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:21.824+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1ad8075050 con 0x7f1af00b0e10 2024-09-13T17:24:22.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.036+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1af0159a60 con 0x7f1af00b0e10 2024-09-13T17:24:22.038 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:24:22.037+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:24:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.047+0000 7f1ae7400640 1 -- 172.21.15.28:0/3526455094 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v189) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1ad8078670 con 0x7f1af00b0e10 2024-09-13T17:24:22.048 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 msgr2=0x7f1acc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 0x7f1acc06d0f0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009ad0 tx=0x7f1ae0008040 comp rx=0 tx=0).stop 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 msgr2=0x7f1af013a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000c00 tx=0x7f1ad80046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 shutdown_connections 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1acc06ac30 0x7f1acc06d0f0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1af00b2870 0x7f1af00c9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1af00b0e10 0x7f1af013a0e0 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 --2- 172.21.15.28:0/3526455094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1af006c370 0x7f1af013d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 >> 172.21.15.28:0/3526455094 conn(0x7f1af0073470 msgr2=0x7f1af015cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.049+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 shutdown_connections 2024-09-13T17:24:22.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.050+0000 7f1aefe00640 1 -- 172.21.15.28:0/3526455094 wait complete. 2024-09-13T17:24:22.057 INFO:tasks.ceph.mds.a.smithi028.stderr: -5468> 2024-09-13T17:24:11.125+0000 7f564a800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T17:24:22.067 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getomapval 10000000000.00000000 parentfile_head - 2024-09-13T17:24:22.108 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:24:22.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.145+0000 7f3550941f40 1 Processor -- start 2024-09-13T17:24:22.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 -- start start 2024-09-13T17:24:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 0x558e59a4ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 0x558e59a47820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e59924400 con 0x558e59975c00 2024-09-13T17:24:22.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e598bffe0 con 0x558e5997d8a0 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.146+0000 7f3550941f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e598bfc80 con 0x558e59a35c40 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33466/0 (socket says 172.21.15.28:33466) 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 -- 172.21.15.28:0/891686096 learned_addr learned my addr 172.21.15.28:0/891686096 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354e600640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 0x558e59a47820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354d200640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 0x558e59a4ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 -- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 msgr2=0x558e59a47820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 0x558e59a47820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 -- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 msgr2=0x558e59a4ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 0x558e59a4ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 -- 172.21.15.28:0/891686096 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f353c005d20 con 0x558e59975c00 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354d200640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 0x558e59a4ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354e600640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 0x558e59a47820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.147+0000 7f354dc00640 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f35400098b0 tx=0x7f3540005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddba9cb216241707 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3547400640 1 -- 172.21.15.28:0/891686096 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354000f040 con 0x558e59975c00 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3547400640 1 -- 172.21.15.28:0/891686096 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3540008930 con 0x558e59975c00 2024-09-13T17:24:22.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3547400640 1 -- 172.21.15.28:0/891686096 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3540008ca0 con 0x558e59975c00 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 -- 172.21.15.28:0/891686096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 msgr2=0x558e599aa2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 secure :-1 s=READY pgs=2762 cs=0 l=1 rev1=1 crypto rx=0x7f35400098b0 tx=0x7f3540005a00 comp rx=0 tx=0).stop 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 -- 172.21.15.28:0/891686096 shutdown_connections 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59a35c40 0x558e59a47820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e5997d8a0 0x558e59a4ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 --2- 172.21.15.28:0/891686096 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e59975c00 0x558e599aa2e0 unknown :-1 s=CLOSED pgs=2762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 -- 172.21.15.28:0/891686096 >> 172.21.15.28:0/891686096 conn(0x558e599a8270 msgr2=0x558e599a9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 -- 172.21.15.28:0/891686096 shutdown_connections 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.148+0000 7f3550941f40 1 -- 172.21.15.28:0/891686096 wait complete. 2024-09-13T17:24:22.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.149+0000 7f3550941f40 1 Processor -- start 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.149+0000 7f3550941f40 1 -- start start 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59975c00 0x558e599c5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 0x558e59a42b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e59a35c40 0x558e59a491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e598b68f0 con 0x558e5997d8a0 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e59a2c610 con 0x558e59a35c40 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f3550941f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x558e599a9480 con 0x558e59975c00 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f354d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 0x558e59a42b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f354d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 0x558e59a42b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33480/0 (socket says 172.21.15.28:33480) 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f354d200640 1 -- 172.21.15.28:0/711417095 learned_addr learned my addr 172.21.15.28:0/711417095 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f354e600640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e59a35c40 0x558e59a491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.150+0000 7f354dc00640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59975c00 0x558e599c5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 -- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59975c00 msgr2=0x558e599c5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59975c00 0x558e599c5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 -- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e59a35c40 msgr2=0x558e59a491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558e59a35c40 0x558e59a491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 -- 172.21.15.28:0/711417095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3540009510 con 0x558e5997d8a0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354dc00640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x558e59975c00 0x558e599c5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f354d200640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 0x558e59a42b60 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f353c057040 tx=0x7f353c002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f3546000640 1 -- 172.21.15.28:0/711417095 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c054b10 con 0x558e5997d8a0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f3546000640 1 -- 172.21.15.28:0/711417095 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f353c054c70 con 0x558e5997d8a0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558e598ceb20 con 0x558e5997d8a0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f3546000640 1 -- 172.21.15.28:0/711417095 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f353c05f480 con 0x558e5997d8a0 2024-09-13T17:24:22.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.151+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e598c1f60 con 0x558e5997d8a0 2024-09-13T17:24:22.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.154+0000 7f3546000640 1 -- 172.21.15.28:0/711417095 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f353c05b040 con 0x558e5997d8a0 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.155+0000 7f3546000640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35300695e0 0x7f353006baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.155+0000 7f3546000640 1 -- 172.21.15.28:0/711417095 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f353c0cbca0 con 0x558e5997d8a0 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.155+0000 7f3550941f40 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x558e59a4f350 0x558e59a2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.155+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- osd_op(unknown.0.0:1 32.10 32:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=23b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e234) v8 -- 0x558e59a4a680 con 0x558e59a4f350 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.156+0000 7f354dc00640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35300695e0 0x7f353006baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.156+0000 7f354e600640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x558e59a4f350 0x558e59a2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.156+0000 7f354e600640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x558e59a4f350 0x558e59a2cf40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.156+0000 7f354dc00640 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35300695e0 0x7f353006baa0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3540015040 tx=0x7f354000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.158+0000 7f354e600640 1 -- 172.21.15.28:0/711417095 <== osd.1 v2:172.21.15.28:6808/2722659727 1 ==== osd_op_reply(1 10000000000.00000000 [omap-get-vals-by-keys out=596b] v0'0 uv10 ondisk = 0) v8 ==== 164+0+596 (crc 0 0 0) 0x7f35480035c0 con 0x558e59a4f350 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:Writing to - 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.158+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x558e59a4f350 msgr2=0x558e59a2cf40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.158+0000 7f3550941f40 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x558e59a4f350 0x558e59a2cf40 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.158+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35300695e0 msgr2=0x7f353006baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f35300695e0 0x7f353006baa0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3540015040 tx=0x7f354000b040 comp rx=0 tx=0).stop 2024-09-13T17:24:22.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 msgr2=0x558e59a42b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 --2- 172.21.15.28:0/711417095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558e5997d8a0 0x558e59a42b60 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7f353c057040 tx=0x7f353c002fb0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f354e600640 1 -- 172.21.15.28:0/711417095 reap_dead start 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 shutdown_connections 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 >> 172.21.15.28:0/711417095 conn(0x558e599a8270 msgr2=0x558e59975fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 shutdown_connections 2024-09-13T17:24:22.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.159+0000 7f3550941f40 1 -- 172.21.15.28:0/711417095 wait complete. 2024-09-13T17:24:22.164 INFO:teuthology.orchestra.run.smithi028.stdout:i* �Bu�f�;����5@@!`��������Bu�f�;�Bu�f�0�(8`Bu�f�;�8`Bu�f�;���������Bu�f�0�����`security.selinux%unconfined_u:object_r:unlabeled_t:s0�������� 2024-09-13T17:24:22.164 DEBUG:teuthology.orchestra.run.smithi028:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 shadow_head 2024-09-13T17:24:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.280+0000 7f3083b29f40 1 Processor -- start 2024-09-13T17:24:22.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.280+0000 7f3083b29f40 1 -- start start 2024-09-13T17:24:22.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049ff100 0x55cb049e6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb04a70c40 0x55cb04a6f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb0495f400 con 0x55cb049b88a0 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb048fafe0 con 0x55cb04a70c40 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.281+0000 7f3083b29f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb048fac80 con 0x55cb049ff100 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33498/0 (socket says 172.21.15.28:33498) 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 -- 172.21.15.28:0/3805891886 learned_addr learned my addr 172.21.15.28:0/3805891886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 -- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049ff100 msgr2=0x55cb049e6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049ff100 0x55cb049e6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 -- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb04a70c40 msgr2=0x55cb04a6f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb04a70c40 0x55cb04a6f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 -- 172.21.15.28:0/3805891886 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c005d20 con 0x55cb049b88a0 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f3080e00640 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f307400b240 tx=0x7f307400b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=219dbd186f5c96b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f307aa00640 1 -- 172.21.15.28:0/3805891886 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074014070 con 0x55cb049b88a0 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f307aa00640 1 -- 172.21.15.28:0/3805891886 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307400d500 con 0x55cb049b88a0 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.282+0000 7f307aa00640 1 -- 172.21.15.28:0/3805891886 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307400d8b0 con 0x55cb049b88a0 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 -- 172.21.15.28:0/3805891886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 msgr2=0x55cb04a88510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f307400b240 tx=0x7f307400b700 comp rx=0 tx=0).stop 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 -- 172.21.15.28:0/3805891886 shutdown_connections 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb04a70c40 0x55cb04a6f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049ff100 0x55cb049e6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 --2- 172.21.15.28:0/3805891886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb049b88a0 0x55cb04a88510 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 -- 172.21.15.28:0/3805891886 >> 172.21.15.28:0/3805891886 conn(0x55cb049e3270 msgr2=0x55cb049e23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 -- 172.21.15.28:0/3805891886 shutdown_connections 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.283+0000 7f3083b29f40 1 -- 172.21.15.28:0/3805891886 wait complete. 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.284+0000 7f3083b29f40 1 Processor -- start 2024-09-13T17:24:22.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.284+0000 7f3083b29f40 1 -- start start 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049b88a0 0x55cb049e4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb049ff100 0x55cb04b27630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 0x55cb049b71d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb048f18f0 con 0x55cb04a70c40 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cb049e4b80 con 0x55cb049ff100 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3083b29f40 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cb04b271a0 con 0x55cb049b88a0 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3081800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 0x55cb049b71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3081800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 0x55cb049b71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33500/0 (socket says 172.21.15.28:33500) 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3081800640 1 -- 172.21.15.28:0/1536846878 learned_addr learned my addr 172.21.15.28:0/1536846878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3080e00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049b88a0 0x55cb049e4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f307be00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb049ff100 0x55cb04b27630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3081800640 1 -- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049b88a0 msgr2=0x55cb049e4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.285+0000 7f3081800640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cb049b88a0 0x55cb049e4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3081800640 1 -- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb049ff100 msgr2=0x55cb04b27630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3081800640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cb049ff100 0x55cb04b27630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3081800640 1 -- 172.21.15.28:0/1536846878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3074001b40 con 0x55cb04a70c40 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3081800640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 0x55cb049b71d0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f307c00a410 tx=0x7f307c00a8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3079600640 1 -- 172.21.15.28:0/1536846878 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c011630 con 0x55cb04a70c40 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3079600640 1 -- 172.21.15.28:0/1536846878 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307c011790 con 0x55cb04a70c40 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3079600640 1 -- 172.21.15.28:0/1536846878 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307c011b40 con 0x55cb04a70c40 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cb04909b20 con 0x55cb04a70c40 2024-09-13T17:24:22.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.286+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cb048fcf60 con 0x55cb04a70c40 2024-09-13T17:24:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.288+0000 7f3079600640 1 -- 172.21.15.28:0/1536846878 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f307c011ca0 con 0x55cb04a70c40 2024-09-13T17:24:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f3079600640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30600695e0 0x7f306006baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f3079600640 1 -- 172.21.15.28:0/1536846878 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f307c088300 con 0x55cb04a70c40 2024-09-13T17:24:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f3083b29f40 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x55cb049b5480 0x55cb04a86230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- osd_op(unknown.0.0:1 32.10 32:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=592b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234) v8 -- 0x55cb04a86770 con 0x55cb049b5480 2024-09-13T17:24:22.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f3080e00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30600695e0 0x7f306006baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.289+0000 7f307be00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x55cb049b5480 0x55cb04a86230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.290+0000 7f307be00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x55cb049b5480 0x55cb04a86230 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.290+0000 7f3080e00640 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30600695e0 0x7f306006baa0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f307400b240 tx=0x7f3074008000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f307be00640 1 -- 172.21.15.28:0/1536846878 <== osd.1 v2:172.21.15.28:6808/2722659727 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v234'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f306c054540 con 0x55cb049b5480 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x55cb049b5480 msgr2=0x55cb04a86230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x55cb049b5480 0x55cb04a86230 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30600695e0 msgr2=0x7f306006baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30600695e0 0x7f306006baa0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f307400b240 tx=0x7f3074008000 comp rx=0 tx=0).stop 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 msgr2=0x55cb049b71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 --2- 172.21.15.28:0/1536846878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cb04a70c40 0x55cb049b71d0 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f307c00a410 tx=0x7f307c00a8d0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3081800640 1 -- 172.21.15.28:0/1536846878 reap_dead start 2024-09-13T17:24:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 shutdown_connections 2024-09-13T17:24:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 >> 172.21.15.28:0/1536846878 conn(0x55cb049e3270 msgr2=0x55cb049e1c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 shutdown_connections 2024-09-13T17:24:22.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.292+0000 7f3083b29f40 1 -- 172.21.15.28:0/1536846878 wait complete. 2024-09-13T17:24:22.297 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_hack_allow_loading_invalid_metadata True 2024-09-13T17:24:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.495+0000 7f8e4e800640 1 Processor -- start 2024-09-13T17:24:22.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.496+0000 7f8e4e800640 1 -- start start 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e480cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4e800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f8e480cddb0 0x7f8e4815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4e800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e480ce1d0 con 0x7f8e48070e30 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4e800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e48159e10 con 0x7f8e480cddb0 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4815a0f0 con 0x7f8e48071210 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4d400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f8e48070e30 0x7f8e48071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44528/0 (socket says 172.21.15.28:44528) 2024-09-13T17:24:22.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4d400640 1 -- 172.21.15.28:0/3490642943 learned_addr learned my addr 172.21.15.28:0/3490642943 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> v2:172.21.15.189:3300/0 conn(0x7f8e480cddb0 0x7f8e4815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e480cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.497+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865674351 0 0) 0x7f8e480ce1d0 con 0x7f8e48070e30 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e30003520 con 0x7f8e48070e30 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 610316476 0 0) 0x7f8e30003520 con 0x7f8e48070e30 2024-09-13T17:24:22.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 msgr2=0x7f8e480cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e480cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 >> v2:172.21.15.189:3300/0 conn(0x7f8e480cddb0 msgr2=0x7f8e4815aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 --2- 172.21.15.28:0/3490642943 >> v2:172.21.15.189:3300/0 conn(0x7f8e480cddb0 0x7f8e4815aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4815a3d0 con 0x7f8e48070e30 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f8e3c001670 con 0x7f8e48070e30 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> v2:172.21.15.189:3300/0 conn(0x7f8e480cddb0 0x7f8e4815aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 >> v1:172.21.15.28:6789/0 conn(0x7f8e48070e30 legacy=0x7f8e48071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e30003970 0x7f8e30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e48159e10 con 0x7f8e30004d00 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4815a0f0 con 0x7f8e30003970 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e3000b980 con 0x7f8e48071210 2024-09-13T17:24:22.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.498+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4d400640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e30003970 0x7f8e30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4de00640 1 -- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 msgr2=0x7f8e300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e48071210 0x7f8e300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4de00640 1 -- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e30003970 msgr2=0x7f8e30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e30003970 0x7f8e30003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.499+0000 7f8e4de00640 1 -- 172.21.15.28:0/3490642943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30003520 con 0x7f8e30004d00 2024-09-13T17:24:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.500+0000 7f8e4de00640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000e380 tx=0x7f8e300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b46bb6c5c5aced7e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.500+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4002c620 con 0x7f8e30004d00 2024-09-13T17:24:22.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.500+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4000f040 con 0x7f8e30004d00 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.500+0000 7f8e47e00640 1 -- 172.21.15.28:0/3490642943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4002c980 con 0x7f8e30004d00 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 -- 172.21.15.28:0/3490642943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e30004d00 msgr2=0x7f8e30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 --2- 172.21.15.28:0/3490642943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e30004d00 0x7f8e30005180 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8e4000e380 tx=0x7f8e300057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4de00640 1 -- 172.21.15.28:0/3490642943 reap_dead start 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 -- 172.21.15.28:0/3490642943 shutdown_connections 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 -- 172.21.15.28:0/3490642943 >> 172.21.15.28:0/3490642943 conn(0x7f8e48069e80 msgr2=0x7f8e4806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 -- 172.21.15.28:0/3490642943 shutdown_connections 2024-09-13T17:24:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.501+0000 7f8e4e800640 1 -- 172.21.15.28:0/3490642943 wait complete. 2024-09-13T17:24:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.502+0000 7f8e4e800640 1 Processor -- start 2024-09-13T17:24:22.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.503+0000 7f8e4e800640 1 -- start start 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.503+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4814d4d0 0x7f8e4814e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 0x7f8e4814dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4814c650 con 0x7f8e481f2ae0 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4814c920 con 0x7f8e4814d8b0 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e4814cc00 con 0x7f8e4814d4d0 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33530/0 (socket says 172.21.15.28:33530) 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 -- 172.21.15.28:0/2795326775 learned_addr learned my addr 172.21.15.28:0/2795326775 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4d400640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4814d4d0 0x7f8e4814e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4de00640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 0x7f8e4814dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 -- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4814d4d0 msgr2=0x7f8e4814e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4814d4d0 0x7f8e4814e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 -- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 msgr2=0x7f8e4814dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 0x7f8e4814dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 -- 172.21.15.28:0/2795326775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e480cae30 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4de00640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 0x7f8e4814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.504+0000 7f8e4ca00640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f8e34011600 tx=0x7f8e34011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.505+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400f920 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.505+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3400fa80 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.505+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3400fdf0 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.505+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e480cb110 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.505+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4814e280 con 0x7f8e481f2ae0 2024-09-13T17:24:22.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.506+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e480cd870 con 0x7f8e481f2ae0 2024-09-13T17:24:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.507+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f8e34015460 con 0x7f8e481f2ae0 2024-09-13T17:24:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.508+0000 7f8e45600640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 0x7f8e2806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.508+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8e340898f0 con 0x7f8e481f2ae0 2024-09-13T17:24:22.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.508+0000 7f8e4d400640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 0x7f8e2806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.509+0000 7f8e4d400640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 0x7f8e2806d0c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001050 tx=0x7f8e3c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.512+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f8e34054a60 con 0x7f8e481f2ae0 2024-09-13T17:24:22.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.691+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mds_hack_allow_loading_invalid_metadata}] v 0) v1 -- 0x7f8e480719e0 con 0x7f8e481f2ae0 2024-09-13T17:24:22.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.695+0000 7f8e45600640 1 -- 172.21.15.28:0/2795326775 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_hack_allow_loading_invalid_metadata}]=0 v53)=0 v53) v1 ==== 140+0+0 (secure 0 0 0) 0x7f8e340220d0 con 0x7f8e481f2ae0 2024-09-13T17:24:22.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 msgr2=0x7f8e2806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 0x7f8e2806d0c0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c001050 tx=0x7f8e3c009040 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 msgr2=0x7f8e4814c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 secure :-1 s=READY pgs=2770 cs=0 l=1 rev1=1 crypto rx=0x7f8e34011600 tx=0x7f8e34011ac0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 shutdown_connections 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f8e2806ac00 0x7f8e2806d0c0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e4814d8b0 0x7f8e4814dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e481f2ae0 0x7f8e4814c110 unknown :-1 s=CLOSED pgs=2770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 --2- 172.21.15.28:0/2795326775 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f8e4814d4d0 0x7f8e4814e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 >> 172.21.15.28:0/2795326775 conn(0x7f8e48069e80 msgr2=0x7f8e4806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 shutdown_connections 2024-09-13T17:24:22.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.698+0000 7f8e4e800640 1 -- 172.21.15.28:0/2795326775 wait complete. 2024-09-13T17:24:22.716 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T17:24:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.873+0000 7f9782400640 1 Processor -- start 2024-09-13T17:24:22.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.873+0000 7f9782400640 1 -- start start 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f9782400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f977c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f9782400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c15ab60 con 0x7f977c159770 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f9782400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c06f920 con 0x7f977c06f3f0 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f9782400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c15b290 con 0x7f977c070430 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f977b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f977c159770 0x7f977c06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:44538/0 (socket says 172.21.15.28:44538) 2024-09-13T17:24:22.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f977b400640 1 -- 172.21.15.28:0/3435453176 learned_addr learned my addr 172.21.15.28:0/3435453176 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f9772a00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f977c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390583050 0 0) 0x7f976c001110 con 0x7f977c159770 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.875+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9760003520 con 0x7f977c159770 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451335921 0 0) 0x7f9760003520 con 0x7f977c159770 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 msgr2=0x7f977c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f977c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 >> v1:172.21.15.189:6789/0 conn(0x7f977c06f3f0 legacy=0x7f977c15c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c15b570 con 0x7f977c159770 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f976c001800 con 0x7f977c159770 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 >> v1:172.21.15.28:6789/0 conn(0x7f977c159770 legacy=0x7f977c06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f9760003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f97600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9760004d00 0x7f9760005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c15b290 con 0x7f97600039a0 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976000ba30 con 0x7f9760004d00 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976000de00 con 0x7f977c070430 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f9772a00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f97600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.876+0000 7f977b400640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f9760003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977be00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9760004d00 0x7f9760005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977b400640 1 -- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 msgr2=0x7f97600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977b400640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c070430 0x7f97600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977b400640 1 -- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9760004d00 msgr2=0x7f9760005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977b400640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9760004d00 0x7f9760005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977b400640 1 -- 172.21.15.28:0/3435453176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9760003520 con 0x7f97600039a0 2024-09-13T17:24:22.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.877+0000 7f977be00640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9760004d00 0x7f9760005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.878+0000 7f977b400640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f9760003d80 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f976c00c190 tx=0x7f97600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b58c89af664745 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.878+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c005ad0 con 0x7f97600039a0 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.878+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c02d070 con 0x7f97600039a0 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.878+0000 7f977aa00640 1 -- 172.21.15.28:0/3435453176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c005e90 con 0x7f97600039a0 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 -- 172.21.15.28:0/3435453176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 msgr2=0x7f9760003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 --2- 172.21.15.28:0/3435453176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f9760003d80 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f976c00c190 tx=0x7f97600057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f977be00640 1 -- 172.21.15.28:0/3435453176 reap_dead start 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 -- 172.21.15.28:0/3435453176 shutdown_connections 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 -- 172.21.15.28:0/3435453176 >> 172.21.15.28:0/3435453176 conn(0x7f977c0b0270 msgr2=0x7f977c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 -- 172.21.15.28:0/3435453176 shutdown_connections 2024-09-13T17:24:22.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.879+0000 7f9782400640 1 -- 172.21.15.28:0/3435453176 wait complete. 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.880+0000 7f9782400640 1 Processor -- start 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.881+0000 7f9782400640 1 -- start start 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 0x7f977c14d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c14d780 0x7f977c0b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c14b5e0 con 0x7f97600039a0 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c14dd60 con 0x7f977c1562e0 2024-09-13T17:24:22.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9782400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c14e040 con 0x7f977c14d780 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33546/0 (socket says 172.21.15.28:33546) 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 -- 172.21.15.28:0/2658758839 learned_addr learned my addr 172.21.15.28:0/2658758839 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9772a00640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 0x7f977c14d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 -- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c14d780 msgr2=0x7f977c0b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c14d780 0x7f977c0b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 -- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 msgr2=0x7f977c14d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 0x7f977c14d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 -- 172.21.15.28:0/2658758839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c0c8f40 con 0x7f97600039a0 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f9772a00640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 0x7f977c14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:22.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.882+0000 7f977b400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f976c02ad00 tx=0x7f976c001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.883+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c0049e0 con 0x7f97600039a0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.883+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f976c004b40 con 0x7f97600039a0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.883+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c007070 con 0x7f97600039a0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.883+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c0c9220 con 0x7f97600039a0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.883+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c151010 con 0x7f97600039a0 2024-09-13T17:24:22.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.884+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c14e200 con 0x7f97600039a0 2024-09-13T17:24:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.885+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f976c02f030 con 0x7f97600039a0 2024-09-13T17:24:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.886+0000 7f9773e00640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 0x7f975806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:22.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.886+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f976c0a8ab0 con 0x7f97600039a0 2024-09-13T17:24:22.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.886+0000 7f9772a00640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 0x7f975806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:22.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.887+0000 7f9772a00640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 0x7f975806d0c0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f977c157b90 tx=0x7f9764008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:22.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:22.890+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f976c073ca0 con 0x7f97600039a0 2024-09-13T17:24:23.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:23.105+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f977c15ab60 con 0x7f97600039a0 2024-09-13T17:24:24.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.060+0000 7f9773e00640 1 -- 172.21.15.28:0/2658758839 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v191) v1 ==== 162+0+0 (secure 0 0 0) 0x7f977c15ab60 con 0x7f97600039a0 2024-09-13T17:24:24.061 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 msgr2=0x7f975806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 0x7f975806d0c0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f977c157b90 tx=0x7f9764008040 comp rx=0 tx=0).stop 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 msgr2=0x7f977c155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 secure :-1 s=READY pgs=2773 cs=0 l=1 rev1=1 crypto rx=0x7f976c02ad00 tx=0x7f976c001b10 comp rx=0 tx=0).stop 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 shutdown_connections 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f975806ac00 0x7f975806d0c0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f977c14d780 0x7f977c0b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c1562e0 0x7f977c14d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 --2- 172.21.15.28:0/2658758839 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f97600039a0 0x7f977c155da0 unknown :-1 s=CLOSED pgs=2773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 >> 172.21.15.28:0/2658758839 conn(0x7f977c0b0270 msgr2=0x7f977c157bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 shutdown_connections 2024-09-13T17:24:24.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.063+0000 7f9782400640 1 -- 172.21.15.28:0/2658758839 wait complete. 2024-09-13T17:24:24.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:24.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.246+0000 7f059d200640 1 Processor -- start 2024-09-13T17:24:24.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.246+0000 7f059d200640 1 -- start start 2024-09-13T17:24:24.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f059d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0598159440 0x7f05980b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f059d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0598159820 0x7f05980b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f059d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05980c3850 con 0x7f05980b22e0 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f059d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f059815a930 con 0x7f0598159440 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f059d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0598159d20 con 0x7f0598159820 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f0597400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0598159440 0x7f05980b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f0597e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f05980b22e0 0x7f05980b2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43344/0 (socket says 172.21.15.28:43344) 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f0597e00640 1 -- 172.21.15.28:0/3203343621 learned_addr learned my addr 172.21.15.28:0/3203343621 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.247+0000 7f058ea00640 1 --2- 172.21.15.28:0/3203343621 >> v2:172.21.15.189:3300/0 conn(0x7f0598159820 0x7f05980b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.248+0000 7f0597400640 1 -- 172.21.15.28:0/3203343621 >> v2:172.21.15.189:3300/0 conn(0x7f0598159820 msgr2=0x7f05980b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.248+0000 7f0597400640 1 --2- 172.21.15.28:0/3203343621 >> v2:172.21.15.189:3300/0 conn(0x7f0598159820 0x7f05980b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.248+0000 7f0597400640 1 -- 172.21.15.28:0/3203343621 >> v1:172.21.15.28:6789/0 conn(0x7f05980b22e0 legacy=0x7f05980b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.248+0000 7f0596a00640 1 -- 172.21.15.28:0/3203343621 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1976592927 0 0) 0x7f05980c3850 con 0x7f05980b22e0 2024-09-13T17:24:24.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.248+0000 7f0597400640 1 -- 172.21.15.28:0/3203343621 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059815a000 con 0x7f0598159440 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.249+0000 7f0597400640 1 --2- 172.21.15.28:0/3203343621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159440 0x7f05980b0440 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f05880097a0 tx=0x7f058802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cab9042bc6a0d781 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.249+0000 7f0596a00640 1 -- 172.21.15.28:0/3203343621 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588032440 con 0x7f0598159440 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.249+0000 7f0596a00640 1 -- 172.21.15.28:0/3203343621 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05880325a0 con 0x7f0598159440 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.249+0000 7f0596a00640 1 -- 172.21.15.28:0/3203343621 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0588032930 con 0x7f0598159440 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 -- 172.21.15.28:0/3203343621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159440 msgr2=0x7f05980b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 --2- 172.21.15.28:0/3203343621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159440 0x7f05980b0440 secure :-1 s=READY pgs=2775 cs=0 l=1 rev1=1 crypto rx=0x7f05880097a0 tx=0x7f058802a270 comp rx=0 tx=0).stop 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 -- 172.21.15.28:0/3203343621 shutdown_connections 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 --2- 172.21.15.28:0/3203343621 >> v2:172.21.15.189:3300/0 conn(0x7f0598159820 0x7f05980b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 --2- 172.21.15.28:0/3203343621 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159440 0x7f05980b0440 unknown :-1 s=CLOSED pgs=2775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 -- 172.21.15.28:0/3203343621 >> 172.21.15.28:0/3203343621 conn(0x7f0598072e40 msgr2=0x7f0598073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 -- 172.21.15.28:0/3203343621 shutdown_connections 2024-09-13T17:24:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.250+0000 7f059d200640 1 -- 172.21.15.28:0/3203343621 wait complete. 2024-09-13T17:24:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.251+0000 7f059d200640 1 Processor -- start 2024-09-13T17:24:24.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 -- start start 2024-09-13T17:24:24.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05980b22e0 0x7f05981f5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0598159440 0x7f05981f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f059815d3a0 con 0x7f0598159820 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05980ada40 con 0x7f05980b22e0 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.252+0000 7f059d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f059815d0c0 con 0x7f0598159440 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56100/0 (socket says 172.21.15.28:56100) 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 -- 172.21.15.28:0/1771004253 learned_addr learned my addr 172.21.15.28:0/1771004253 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597400640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05980b22e0 0x7f05981f5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f058ea00640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0598159440 0x7f05981f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 -- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0598159440 msgr2=0x7f05981f3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0598159440 0x7f05981f3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 -- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05980b22e0 msgr2=0x7f05981f5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05980b22e0 0x7f05981f5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 -- 172.21.15.28:0/1771004253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0588009400 con 0x7f0598159820 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.253+0000 7f0597e00640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f0590002230 tx=0x7f059000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.254+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059000ce30 con 0x7f0598159820 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.254+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f059000a640 con 0x7f0598159820 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.254+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f059000a9f0 con 0x7f0598159820 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.254+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05980adce0 con 0x7f0598159820 2024-09-13T17:24:24.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.254+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05980c4c80 con 0x7f0598159820 2024-09-13T17:24:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.255+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05980ad8a0 con 0x7f0598159820 2024-09-13T17:24:24.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.256+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f059000ab50 con 0x7f0598159820 2024-09-13T17:24:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.257+0000 7f058fe00640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 0x7f057406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.257+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05900886d0 con 0x7f0598159820 2024-09-13T17:24:24.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.258+0000 7f0597400640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 0x7f057406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:24.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.258+0000 7f0597400640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 0x7f057406d0c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0588034000 tx=0x7f058802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:24.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.261+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f05980ad8a0 con 0x7f0598159820 2024-09-13T17:24:24.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.473+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05980ae200 con 0x7f0598159820 2024-09-13T17:24:24.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.474+0000 7f058fe00640 1 -- 172.21.15.28:0/1771004253 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f05980ae200 con 0x7f0598159820 2024-09-13T17:24:24.475 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:24.475 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":192,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":192,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:24.060488+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:replay","state_seq":1,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:24.475 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 192 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 msgr2=0x7f057406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 0x7f057406d0c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0588034000 tx=0x7f058802e000 comp rx=0 tx=0).stop 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 msgr2=0x7f05980c4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 secure :-1 s=READY pgs=2776 cs=0 l=1 rev1=1 crypto rx=0x7f0590002230 tx=0x7f059000cbe0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 shutdown_connections 2024-09-13T17:24:24.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f057406ac00 0x7f057406d0c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0598159820 0x7f05980c4740 unknown :-1 s=CLOSED pgs=2776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0598159440 0x7f05981f3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 --2- 172.21.15.28:0/1771004253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05980b22e0 0x7f05981f5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 >> 172.21.15.28:0/1771004253 conn(0x7f0598072e40 msgr2=0x7f059806f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 shutdown_connections 2024-09-13T17:24:24.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:24.476+0000 7f059d200640 1 -- 172.21.15.28:0/1771004253 wait complete. 2024-09-13T17:24:24.895 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 192, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:24:10.577893+0000', 'modified': '2024-09-13T17:24:24.060488+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 234, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12516}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12516': {'gid': 12516, 'name': 'a', 'rank': 0, 'incarnation': 192, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.28:6835/994067792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 994067792}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 994067792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:24:24.896 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12516:up:replay 2024-09-13T17:24:25.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:26.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.064+0000 7f61f8400640 1 Processor -- start 2024-09-13T17:24:26.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.064+0000 7f61f8400640 1 -- start start 2024-09-13T17:24:26.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.065+0000 7f61f8400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61f01592a0 0x7f61f015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.065+0000 7f61f8400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 0x7f61f0159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.065+0000 7f61f8400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61f00b2040 con 0x7f61f006c2d0 2024-09-13T17:24:26.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.065+0000 7f61f8400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f015b0b0 con 0x7f61f01592a0 2024-09-13T17:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.065+0000 7f61f8400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61f0159ff0 con 0x7f61f006f970 2024-09-13T17:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f6400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f61f006c2d0 0x7f61f00b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43358/0 (socket says 172.21.15.28:43358) 2024-09-13T17:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f61f01592a0 0x7f61f015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f6400640 1 -- 172.21.15.28:0/3089446017 learned_addr learned my addr 172.21.15.28:0/3089446017 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:26.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5000640 1 --2- 172.21.15.28:0/3089446017 >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 0x7f61f0159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5a00640 1 -- 172.21.15.28:0/3089446017 >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 msgr2=0x7f61f0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5a00640 1 --2- 172.21.15.28:0/3089446017 >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 0x7f61f0159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5a00640 1 -- 172.21.15.28:0/3089446017 >> v1:172.21.15.28:6789/0 conn(0x7f61f006c2d0 legacy=0x7f61f00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61efe00640 1 -- 172.21.15.28:0/3089446017 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033828121 0 0) 0x7f61f00b2040 con 0x7f61f006c2d0 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5a00640 1 -- 172.21.15.28:0/3089446017 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f015a2d0 con 0x7f61f01592a0 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.066+0000 7f61f5000640 1 --2- 172.21.15.28:0/3089446017 >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 0x7f61f0159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:26.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.067+0000 7f61f5a00640 1 --2- 172.21.15.28:0/3089446017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f01592a0 0x7f61f015ab70 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f61f006ac10 tx=0x7f61e402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd00c35b26336c13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.069+0000 7f61efe00640 1 -- 172.21.15.28:0/3089446017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4032440 con 0x7f61f01592a0 2024-09-13T17:24:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.069+0000 7f61efe00640 1 -- 172.21.15.28:0/3089446017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e40325a0 con 0x7f61f01592a0 2024-09-13T17:24:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.069+0000 7f61efe00640 1 -- 172.21.15.28:0/3089446017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e4032930 con 0x7f61f01592a0 2024-09-13T17:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 -- 172.21.15.28:0/3089446017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f01592a0 msgr2=0x7f61f015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 --2- 172.21.15.28:0/3089446017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f01592a0 0x7f61f015ab70 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7f61f006ac10 tx=0x7f61e402a1a0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 -- 172.21.15.28:0/3089446017 shutdown_connections 2024-09-13T17:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 --2- 172.21.15.28:0/3089446017 >> v2:172.21.15.189:3300/0 conn(0x7f61f006f970 0x7f61f0159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 --2- 172.21.15.28:0/3089446017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f01592a0 0x7f61f015ab70 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 -- 172.21.15.28:0/3089446017 >> 172.21.15.28:0/3089446017 conn(0x7f61f0069e80 msgr2=0x7f61f006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 -- 172.21.15.28:0/3089446017 shutdown_connections 2024-09-13T17:24:26.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.070+0000 7f61f8400640 1 -- 172.21.15.28:0/3089446017 wait complete. 2024-09-13T17:24:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.072+0000 7f61f8400640 1 Processor -- start 2024-09-13T17:24:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.072+0000 7f61f8400640 1 -- start start 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f006c2d0 0x7f61f0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61f01592a0 0x7f61f00cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00cb5b0 con 0x7f61f006f970 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00cb800 con 0x7f61f006c2d0 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.073+0000 7f61f8400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61f00cbae0 con 0x7f61f01592a0 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56118/0 (socket says 172.21.15.28:56118) 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 -- 172.21.15.28:0/4272981247 learned_addr learned my addr 172.21.15.28:0/4272981247 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5a00640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f006c2d0 0x7f61f0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f6400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61f01592a0 0x7f61f00cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 -- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61f01592a0 msgr2=0x7f61f00cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61f01592a0 0x7f61f00cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 -- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f006c2d0 msgr2=0x7f61f0152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f006c2d0 0x7f61f0152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 -- 172.21.15.28:0/4272981247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e4009400 con 0x7f61f006f970 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.074+0000 7f61f5000640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e730 tx=0x7f61dc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.075+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc004230 con 0x7f61f006f970 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.075+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61dc004390 con 0x7f61f006f970 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.075+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61dc00c8f0 con 0x7f61f006f970 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.075+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61f00cbdc0 con 0x7f61f006f970 2024-09-13T17:24:26.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.075+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61f01525c0 con 0x7f61f006f970 2024-09-13T17:24:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.076+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61f015ab70 con 0x7f61f006f970 2024-09-13T17:24:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.077+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f61dc00ca50 con 0x7f61f006f970 2024-09-13T17:24:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.078+0000 7f61ed600640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 0x7f61d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.078+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f61dc088f40 con 0x7f61f006f970 2024-09-13T17:24:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.079+0000 7f61f5a00640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 0x7f61d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:26.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.079+0000 7f61f5a00640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 0x7f61d006d0f0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f61e4034000 tx=0x7f61e402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:26.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.083+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f61dc054160 con 0x7f61f006f970 2024-09-13T17:24:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.298+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61f0155f00 con 0x7f61f006f970 2024-09-13T17:24:26.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.299+0000 7f61ed600640 1 -- 172.21.15.28:0/4272981247 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f61dc057780 con 0x7f61f006f970 2024-09-13T17:24:26.300 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:26.300 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":194,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:26.066647+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:26.300 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 194 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 msgr2=0x7f61d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 0x7f61d006d0f0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f61e4034000 tx=0x7f61e402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 msgr2=0x7f61f0155400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 secure :-1 s=READY pgs=2779 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00e730 tx=0x7f61dc00ebf0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 shutdown_connections 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f61d006ac30 0x7f61d006d0f0 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f61f01592a0 0x7f61f00cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f61f006f970 0x7f61f0155400 unknown :-1 s=CLOSED pgs=2779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 --2- 172.21.15.28:0/4272981247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61f006c2d0 0x7f61f0152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 >> 172.21.15.28:0/4272981247 conn(0x7f61f0069e80 msgr2=0x7f61f00ada80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:26.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 shutdown_connections 2024-09-13T17:24:26.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:26.301+0000 7f61f8400640 1 -- 172.21.15.28:0/4272981247 wait complete. 2024-09-13T17:24:26.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 194, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:24:10.577893+0000', 'modified': '2024-09-13T17:24:26.066647+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 234, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12516}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12516': {'gid': 12516, 'name': 'a', 'rank': 0, 'incarnation': 192, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.28:6835/994067792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 994067792}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 994067792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:24:26.720 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_12516:up:rejoin 2024-09-13T17:24:27.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:24:27.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.888+0000 7f1723400640 1 Processor -- start 2024-09-13T17:24:27.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.889+0000 7f1723400640 1 -- start start 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1723400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 0x7f171c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1723400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f171c06f980 0x7f171c06b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1723400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c0b3970 con 0x7f171c0cdfa0 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1723400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c15d760 con 0x7f171c06f980 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1723400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c159230 con 0x7f171c15d340 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1720a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f171c0cdfa0 0x7f171c0b3160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43362/0 (socket says 172.21.15.28:43362) 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1720a00640 1 -- 172.21.15.28:0/3846186740 learned_addr learned my addr 172.21.15.28:0/3846186740 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:27.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1713e00640 1 --2- 172.21.15.28:0/3846186740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 0x7f171c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.890+0000 7f1721400640 1 --2- 172.21.15.28:0/3846186740 >> v2:172.21.15.189:3300/0 conn(0x7f171c06f980 0x7f171c06b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f171be00640 1 -- 172.21.15.28:0/3846186740 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333761281 0 0) 0x7f170c001110 con 0x7f171c0cdfa0 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f171be00640 1 -- 172.21.15.28:0/3846186740 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1700003520 con 0x7f171c0cdfa0 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f1713e00640 1 -- 172.21.15.28:0/3846186740 >> v2:172.21.15.189:3300/0 conn(0x7f171c06f980 msgr2=0x7f171c06b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f1713e00640 1 --2- 172.21.15.28:0/3846186740 >> v2:172.21.15.189:3300/0 conn(0x7f171c06f980 0x7f171c06b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f1713e00640 1 -- 172.21.15.28:0/3846186740 >> v1:172.21.15.28:6789/0 conn(0x7f171c0cdfa0 legacy=0x7f171c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:27.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.891+0000 7f1713e00640 1 -- 172.21.15.28:0/3846186740 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c159430 con 0x7f171c15d340 2024-09-13T17:24:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.892+0000 7f1713e00640 1 --2- 172.21.15.28:0/3846186740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 0x7f171c06f440 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17040097a0 tx=0x7f170402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6df5046b2a17c0f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.892+0000 7f171be00640 1 -- 172.21.15.28:0/3846186740 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704032440 con 0x7f171c15d340 2024-09-13T17:24:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.892+0000 7f171be00640 1 -- 172.21.15.28:0/3846186740 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17040325a0 con 0x7f171c15d340 2024-09-13T17:24:27.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.892+0000 7f171be00640 1 -- 172.21.15.28:0/3846186740 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1704032930 con 0x7f171c15d340 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 -- 172.21.15.28:0/3846186740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 msgr2=0x7f171c06f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 --2- 172.21.15.28:0/3846186740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 0x7f171c06f440 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17040097a0 tx=0x7f170402a180 comp rx=0 tx=0).stop 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 -- 172.21.15.28:0/3846186740 shutdown_connections 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 --2- 172.21.15.28:0/3846186740 >> v2:172.21.15.189:3300/0 conn(0x7f171c06f980 0x7f171c06b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 --2- 172.21.15.28:0/3846186740 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c15d340 0x7f171c06f440 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 -- 172.21.15.28:0/3846186740 >> 172.21.15.28:0/3846186740 conn(0x7f171c069e80 msgr2=0x7f171c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 -- 172.21.15.28:0/3846186740 shutdown_connections 2024-09-13T17:24:27.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.893+0000 7f1723400640 1 -- 172.21.15.28:0/3846186740 wait complete. 2024-09-13T17:24:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.895+0000 7f1723400640 1 Processor -- start 2024-09-13T17:24:27.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.895+0000 7f1723400640 1 -- start start 2024-09-13T17:24:27.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 0x7f171c1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c0cdfa0 0x7f171c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c1500a0 con 0x7f171c15d340 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c150380 con 0x7f171c06f980 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1723400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c150660 con 0x7f171c0cdfa0 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1721400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1721400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56128/0 (socket says 172.21.15.28:56128) 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1721400640 1 -- 172.21.15.28:0/1740061641 learned_addr learned my addr 172.21.15.28:0/1740061641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:27.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1720a00640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 0x7f171c1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.896+0000 7f1713e00640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c0cdfa0 0x7f171c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 -- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c0cdfa0 msgr2=0x7f171c155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c0cdfa0 0x7f171c155d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 -- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 msgr2=0x7f171c1557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 0x7f171c1557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 -- 172.21.15.28:0/1740061641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1704009400 con 0x7f171c15d340 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1720a00640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 0x7f171c1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:27.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1721400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f171400efc0 tx=0x7f171400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714014480 con 0x7f171c15d340 2024-09-13T17:24:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171400f040 con 0x7f171c15d340 2024-09-13T17:24:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714014820 con 0x7f171c15d340 2024-09-13T17:24:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c150940 con 0x7f171c15d340 2024-09-13T17:24:27.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.897+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c1564b0 con 0x7f171c15d340 2024-09-13T17:24:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.898+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c150e60 con 0x7f171c15d340 2024-09-13T17:24:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.899+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1714014980 con 0x7f171c15d340 2024-09-13T17:24:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.900+0000 7f1719600640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.900+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1714089fe0 con 0x7f171c15d340 2024-09-13T17:24:27.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.901+0000 7f1720a00640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:27.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.901+0000 7f1720a00640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 0x7f16f806d0c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f170c001270 tx=0x7f170c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:27.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:27.904+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f171c150e60 con 0x7f171c15d340 2024-09-13T17:24:28.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.119+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f171c150e60 con 0x7f171c15d340 2024-09-13T17:24:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.120+0000 7f1719600640 1 -- 172.21.15.28:0/1740061641 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f171c150e60 con 0x7f171c15d340 2024-09-13T17:24:28.121 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:24:28.121 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:24:28.121 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:24:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 msgr2=0x7f16f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 0x7f16f806d0c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f170c001270 tx=0x7f170c009040 comp rx=0 tx=0).stop 2024-09-13T17:24:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 msgr2=0x7f171c14a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f171400efc0 tx=0x7f171400c490 comp rx=0 tx=0).stop 2024-09-13T17:24:28.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 shutdown_connections 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f171c15d340 0x7f171c14a5e0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f171c0cdfa0 0x7f171c155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 --2- 172.21.15.28:0/1740061641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f171c06f980 0x7f171c1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 >> 172.21.15.28:0/1740061641 conn(0x7f171c069e80 msgr2=0x7f171c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 shutdown_connections 2024-09-13T17:24:28.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.122+0000 7f1723400640 1 -- 172.21.15.28:0/1740061641 wait complete. 2024-09-13T17:24:28.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:24:10.577893+0000', 'modified': '2024-09-13T17:24:27.074178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 234, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12516}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12516': {'gid': 12516, 'name': 'a', 'rank': 0, 'incarnation': 192, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/994067792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 994067792}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 994067792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:24:28.542 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:24:28.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:24:28.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.698+0000 7f3874a00640 1 Processor -- start 2024-09-13T17:24:28.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.698+0000 7f3874a00640 1 -- start start 2024-09-13T17:24:28.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.699+0000 7f3874a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3870070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.699+0000 7f3874a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387006c8b0 con 0x7f387006c250 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.699+0000 7f3874a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f387006cb90 con 0x7f387015ab60 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.699+0000 7f3874a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387006c6a0 con 0x7f387015a780 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386e800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f387006c250 0x7f38700cd180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:43370/0 (socket says 172.21.15.28:43370) 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386e800640 1 -- 172.21.15.28:0/1079144350 learned_addr learned my addr 172.21.15.28:0/1079144350 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:28.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386de00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3870070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722195560 0 0) 0x7f387006c8b0 con 0x7f387006c250 2024-09-13T17:24:28.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3854003520 con 0x7f387006c250 2024-09-13T17:24:28.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792877593 0 0) 0x7f387006cb90 con 0x7f387015ab60 2024-09-13T17:24:28.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f387006c8b0 con 0x7f387015ab60 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480789702 0 0) 0x7f3854003520 con 0x7f387006c250 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 msgr2=0x7f3870070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3870070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 >> v1:172.21.15.189:6789/0 conn(0x7f387015ab60 legacy=0x7f387006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.700+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38700cc980 con 0x7f387006c250 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f38680019f0 con 0x7f387006c250 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 >> v1:172.21.15.28:6789/0 conn(0x7f387006c250 legacy=0x7f38700cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f3854003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38540047c0 0x7f3854004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3854005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387006c6a0 con 0x7f3854003b30 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385400ba30 con 0x7f38540047c0 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385400bbf0 con 0x7f387015a780 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f3854003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386d400640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38540047c0 0x7f3854004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386e800640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3854005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 -- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 msgr2=0x7f3854005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f387015a780 0x7f3854005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 -- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38540047c0 msgr2=0x7f3854004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38540047c0 0x7f3854004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.701+0000 7f386de00640 1 -- 172.21.15.28:0/1079144350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3854003520 con 0x7f3854003b30 2024-09-13T17:24:28.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.702+0000 7f386de00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f3854003f30 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f3860009aa0 tx=0x7f38540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f067c7309be009 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3860031400 con 0x7f3854003b30 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386002d050 con 0x7f3854003b30 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f386ca00640 1 -- 172.21.15.28:0/1079144350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38600317a0 con 0x7f3854003b30 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 -- 172.21.15.28:0/1079144350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 msgr2=0x7f3854003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 --2- 172.21.15.28:0/1079144350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f3854003f30 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f3860009aa0 tx=0x7f38540057f0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f386e800640 1 -- 172.21.15.28:0/1079144350 reap_dead start 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 -- 172.21.15.28:0/1079144350 shutdown_connections 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 -- 172.21.15.28:0/1079144350 >> 172.21.15.28:0/1079144350 conn(0x7f38700b0270 msgr2=0x7f38700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 -- 172.21.15.28:0/1079144350 shutdown_connections 2024-09-13T17:24:28.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.703+0000 7f3874a00640 1 -- 172.21.15.28:0/1079144350 wait complete. 2024-09-13T17:24:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.705+0000 7f3874a00640 1 Processor -- start 2024-09-13T17:24:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.705+0000 7f3874a00640 1 -- start start 2024-09-13T17:24:28.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38700b1670 0x7f38700b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3870151030 0x7f38701514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38701f3270 con 0x7f3854003b30 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38701f23e0 con 0x7f38700b1670 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f3874a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38700b2030 con 0x7f3870151030 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f386de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f386de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56144/0 (socket says 172.21.15.28:56144) 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f386de00640 1 -- 172.21.15.28:0/3218375965 learned_addr learned my addr 172.21.15.28:0/3218375965 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f386d400640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38700b1670 0x7f38700b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.706+0000 7f386e800640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3870151030 0x7f38701514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 -- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3870151030 msgr2=0x7f38701514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3870151030 0x7f38701514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 -- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38700b1670 msgr2=0x7f38700b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38700b1670 0x7f38700b1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 -- 172.21.15.28:0/3218375965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38701f25f0 con 0x7f3854003b30 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f386de00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f386002aaa0 tx=0x7f3860004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f3866000640 1 -- 172.21.15.28:0/3218375965 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386003c680 con 0x7f3854003b30 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f3866000640 1 -- 172.21.15.28:0/3218375965 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f386003c820 con 0x7f3854003b30 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f3866000640 1 -- 172.21.15.28:0/3218375965 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386003cbd0 con 0x7f3854003b30 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.707+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38701f28d0 con 0x7f3854003b30 2024-09-13T17:24:28.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.708+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38701f2e90 con 0x7f3854003b30 2024-09-13T17:24:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.710+0000 7f3866000640 1 -- 172.21.15.28:0/3218375965 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3860038040 con 0x7f3854003b30 2024-09-13T17:24:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.711+0000 7f3866000640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 0x7f385006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.711+0000 7f3866000640 1 -- 172.21.15.28:0/3218375965 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f38600a9e30 con 0x7f3854003b30 2024-09-13T17:24:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.712+0000 7f386d400640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 0x7f385006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.712+0000 7f387789f740 1 Processor -- start 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.712+0000 7f387789f740 1 -- start start 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.712+0000 7f386d400640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 0x7f385006d0c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f38700adf70 tx=0x7f385c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a62dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a654b80 0x55c60a67b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a596ab0 con 0x55c60a62ae90 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a594240 con 0x55c60a62d6d0 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c60a58fb30 con 0x55c60a654b80 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56146/0 (socket says 172.21.15.28:56146) 2024-09-13T17:24:28.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 -- 172.21.15.28:0/2037492673 learned_addr learned my addr 172.21.15.28:0/2037492673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386de00640 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a62dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386e800640 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a654b80 0x55c60a67b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 -- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a654b80 msgr2=0x55c60a67b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a654b80 0x55c60a67b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 -- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 msgr2=0x55c60a62dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a62dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.713+0000 7f386d400640 1 -- 172.21.15.28:0/2037492673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c60a55abd0 con 0x55c60a62ae90 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f386d400640 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f385c012a50 tx=0x7f385c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21500644945782c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f385aa00640 1 -- 172.21.15.28:0/2037492673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c007260 con 0x55c60a62ae90 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f385aa00640 1 -- 172.21.15.28:0/2037492673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c0073c0 con 0x55c60a62ae90 2024-09-13T17:24:28.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f385aa00640 1 -- 172.21.15.28:0/2037492673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c0067d0 con 0x55c60a62ae90 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f387789f740 1 -- 172.21.15.28:0/2037492673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 msgr2=0x55c60a67aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.714+0000 7f387789f740 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 secure :-1 s=READY pgs=2785 cs=0 l=1 rev1=1 crypto rx=0x7f385c012a50 tx=0x7f385c015000 comp rx=0 tx=0).stop 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 -- 172.21.15.28:0/2037492673 shutdown_connections 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a654b80 0x55c60a67b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a62ae90 0x55c60a67aba0 unknown :-1 s=CLOSED pgs=2785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 --2- 172.21.15.28:0/2037492673 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a62dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 -- 172.21.15.28:0/2037492673 >> 172.21.15.28:0/2037492673 conn(0x55c60a629e00 msgr2=0x55c60a6722b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 -- 172.21.15.28:0/2037492673 shutdown_connections 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 -- 172.21.15.28:0/2037492673 wait complete. 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.715+0000 7f387789f740 1 -- start start 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 Processor -- start 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a62cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a6732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a531260 con 0x55c60a654b80 2024-09-13T17:24:28.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a530f00 con 0x55c60a64d470 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.716+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c60a2f6f20 con 0x55c60a62d6d0 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f387789f740 10 client.0 _ll_register_callbacks cb 0x55c60a5891a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56148/0 (socket says 172.21.15.28:56148) 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386de00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a62cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 -- 172.21.15.28:0/2402073299 learned_addr learned my addr 172.21.15.28:0/2402073299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386e800640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a6732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 -- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 msgr2=0x55c60a62cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a62cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 -- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 msgr2=0x55c60a6732e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a6732e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386d400640 1 -- 172.21.15.28:0/2402073299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387006c8b0 con 0x55c60a654b80 2024-09-13T17:24:28.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.717+0000 7f386de00640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a62cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f386d400640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f385c008fd0 tx=0x7f385c0012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2c85f5636b79715 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385c015780 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f385c0158e0 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c60a5b9cd0 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f385c015aa0 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 10 client.12585 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55c60a58e1b0 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c60a5b9cd0 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f385c015c00 con 0x55c60a654b80 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 10 client.12585 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:28.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 10 client.12585 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 20 client.12585 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a639490 0x55c60a636cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 4 client.12585 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.718+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c60a2833c0 con 0x55c60a639490 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.719+0000 7f386de00640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a639490 0x55c60a636cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.719+0000 7f386de00640 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a639490 0x55c60a636cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59e5f52728ed76bb server_cookie=1d4970168b968b58 in_seq=0 out_seq=0 2024-09-13T17:24:28.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.719+0000 7f384f400640 10 client.12585 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f384f400640 1 -- 172.21.15.28:0/2402073299 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55c60a2833c0 con 0x55c60a639490 2024-09-13T17:24:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f384f400640 10 client.12585 handle_command_reply: tid=0 2024-09-13T17:24:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 client.12585 shutdown 2024-09-13T17:24:28.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 msgr2=0x55c60a64cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7f385c008fd0 tx=0x7f385c0012d0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 shutdown_connections 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a639490 0x55c60a636cf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a6732e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a64cf30 unknown :-1 s=CLOSED pgs=2786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 --2- 172.21.15.28:0/2402073299 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a62cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f384f400640 0 client.12585 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.720+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 >> 172.21.15.28:0/2402073299 conn(0x55c60a62ae90 msgr2=0x55c60a6705f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.721+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 shutdown_connections 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.721+0000 7f387789f740 1 -- 172.21.15.28:0/2402073299 wait complete. 2024-09-13T17:24:28.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.721+0000 7f387789f740 20 client.12585 trim_cache size 0 max 16384 2024-09-13T17:24:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.736+0000 7f387789f740 1 Processor -- start 2024-09-13T17:24:28.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.736+0000 7f387789f740 1 -- start start 2024-09-13T17:24:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a64bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 0x55c60a62cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a64ce60 con 0x55c60a654b80 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a6387c0 con 0x55c60a62d6d0 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c60a2c9940 con 0x55c60a64d470 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f386e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f386e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56156/0 (socket says 172.21.15.28:56156) 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.737+0000 7f386e800640 1 -- 172.21.15.28:0/1107609239 learned_addr learned my addr 172.21.15.28:0/1107609239 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386de00640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a64bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386d400640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 0x55c60a62cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 -- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 msgr2=0x55c60a62cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 0x55c60a62cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 -- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 msgr2=0x55c60a64bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a64bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 -- 172.21.15.28:0/1107609239 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c60a5c3170 con 0x55c60a654b80 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386d400640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 0x55c60a62cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f386e800640 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f387006d800 tx=0x7f3868001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f29a656222863f50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f385b400640 1 -- 172.21.15.28:0/1107609239 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38680078c0 con 0x55c60a654b80 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f385b400640 1 -- 172.21.15.28:0/1107609239 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3868007a20 con 0x55c60a654b80 2024-09-13T17:24:28.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.738+0000 7f385b400640 1 -- 172.21.15.28:0/1107609239 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386801f660 con 0x55c60a654b80 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 -- 172.21.15.28:0/1107609239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 msgr2=0x55c60a62db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7f387006d800 tx=0x7f3868001e80 comp rx=0 tx=0).stop 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 -- 172.21.15.28:0/1107609239 shutdown_connections 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a62db50 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a64d470 0x55c60a62cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 --2- 172.21.15.28:0/1107609239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a62d6d0 0x55c60a64bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 -- 172.21.15.28:0/1107609239 >> 172.21.15.28:0/1107609239 conn(0x55c60a62ae90 msgr2=0x55c60a638010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 -- 172.21.15.28:0/1107609239 shutdown_connections 2024-09-13T17:24:28.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.739+0000 7f387789f740 1 -- 172.21.15.28:0/1107609239 wait complete. 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.740+0000 7f387789f740 1 -- start start 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.740+0000 7f387789f740 1 Processor -- start 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a628de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a636cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a6c3130 con 0x55c60a654b80 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c60a6d5d40 con 0x55c60a64d470 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c60a6d5f80 con 0x55c60a62d6d0 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f387789f740 10 client.0 _ll_register_callbacks cb 0x7f385c0067d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56170/0 (socket says 172.21.15.28:56170) 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 -- 172.21.15.28:0/3460784377 learned_addr learned my addr 172.21.15.28:0/3460784377 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386de00640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a628de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386d400640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a636cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 -- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 msgr2=0x55c60a628de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a628de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 -- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 msgr2=0x55c60a636cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a636cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.741+0000 7f386e800640 1 -- 172.21.15.28:0/3460784377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38700cc980 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f386e800640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f3868001ec0 tx=0x7f3868018040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=208977cfc0104df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f384f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3868026070 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f386801fde0 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f385c0073c0 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38680078c0 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f387789f740 10 client.12591 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55c60a530470 con 0x55c60a654b80 2024-09-13T17:24:28.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.742+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385c0073c0 con 0x55c60a654b80 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3868012040 con 0x55c60a654b80 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 10 client.12591 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 10 client.12591 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 20 client.12591 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a66e500 0x55c60a66e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 4 client.12591 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c60a2946c0 con 0x55c60a66e500 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f386de00640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a66e500 0x55c60a66e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f386de00640 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a66e500 0x55c60a66e8e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb14fd75b1c827bc server_cookie=b17d68827abdc079 in_seq=0 out_seq=0 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.743+0000 7f384f400640 10 client.12591 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f384f400640 1 -- 172.21.15.28:0/3460784377 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55c60a2946c0 con 0x55c60a66e500 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f384f400640 10 client.12591 handle_command_reply: tid=0 2024-09-13T17:24:28.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 client.12591 shutdown 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 msgr2=0x55c60a6371f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 secure :-1 s=READY pgs=2788 cs=0 l=1 rev1=1 crypto rx=0x7f3868001ec0 tx=0x7f3868018040 comp rx=0 tx=0).stop 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 shutdown_connections 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55c60a66e500 0x55c60a66e8e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c60a654b80 0x55c60a6371f0 unknown :-1 s=CLOSED pgs=2788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c60a64d470 0x55c60a636cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 --2- 172.21.15.28:0/3460784377 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55c60a62d6d0 0x55c60a628de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f384f400640 0 client.12591 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 >> 172.21.15.28:0/3460784377 conn(0x55c60a62ae90 msgr2=0x55c60a6541b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 shutdown_connections 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 1 -- 172.21.15.28:0/3460784377 wait complete. 2024-09-13T17:24:28.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.744+0000 7f387789f740 20 client.12591 trim_cache size 0 max 16384 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 12516, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 4.6787784510000003, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 195, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 234, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 234, 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 6.6327353809999998 2024-09-13T17:24:28.751 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 msgr2=0x7f385006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 0x7f385006d0c0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f38700adf70 tx=0x7f385c009040 comp rx=0 tx=0).stop 2024-09-13T17:24:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 msgr2=0x7f387014afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f386002aaa0 tx=0x7f3860004730 comp rx=0 tx=0).stop 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 shutdown_connections 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f385006ac00 0x7f385006d0c0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3870151030 0x7f38701514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38700b1670 0x7f38700b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 --2- 172.21.15.28:0/3218375965 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3854003b30 0x7f387014afb0 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 >> 172.21.15.28:0/3218375965 conn(0x7f38700b0270 msgr2=0x7f3870157820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 shutdown_connections 2024-09-13T17:24:28.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.745+0000 7f3874a00640 1 -- 172.21.15.28:0/3218375965 wait complete. 2024-09-13T17:24:28.761 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub start / recursive,repair 2024-09-13T17:24:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.919+0000 7f3a5c000640 1 Processor -- start 2024-09-13T17:24:28.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.919+0000 7f3a5c000640 1 -- start start 2024-09-13T17:24:28.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5c000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a540ceba0 0x7f3a540b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5c000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a540b1ab0 0x7f3a540cd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5c000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5406c950 con 0x7f3a5415a7a0 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5c000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a540b1ef0 con 0x7f3a540ceba0 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5c000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a540b2150 con 0x7f3a540b1ab0 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a58c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a540ceba0 0x7f3a540b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a58c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3a540ceba0 0x7f3a540b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56180/0 (socket says 172.21.15.28:56180) 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a58c00640 1 -- 172.21.15.28:0/2458388899 learned_addr learned my addr 172.21.15.28:0/2458388899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.920+0000 7f3a5a000640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a540b1ab0 0x7f3a540cd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.921+0000 7f3a58c00640 1 -- 172.21.15.28:0/2458388899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a540b1ab0 msgr2=0x7f3a540cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.921+0000 7f3a58c00640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a540b1ab0 0x7f3a540cd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.921+0000 7f3a58c00640 1 -- 172.21.15.28:0/2458388899 >> v1:172.21.15.189:6789/0 conn(0x7f3a5415a7a0 legacy=0x7f3a540ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.921+0000 7f3a58c00640 1 -- 172.21.15.28:0/2458388899 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a540b2430 con 0x7f3a540ceba0 2024-09-13T17:24:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.922+0000 7f3a58c00640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540ceba0 0x7f3a540b1570 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009870 tx=0x7f3a4002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39abb44188e05de3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.922+0000 7f3a53e00640 1 -- 172.21.15.28:0/2458388899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40032440 con 0x7f3a540ceba0 2024-09-13T17:24:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.922+0000 7f3a53e00640 1 -- 172.21.15.28:0/2458388899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a400325a0 con 0x7f3a540ceba0 2024-09-13T17:24:28.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.922+0000 7f3a53e00640 1 -- 172.21.15.28:0/2458388899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40032930 con 0x7f3a540ceba0 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 -- 172.21.15.28:0/2458388899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540ceba0 msgr2=0x7f3a540b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540ceba0 0x7f3a540b1570 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009870 tx=0x7f3a4002a1c0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 -- 172.21.15.28:0/2458388899 shutdown_connections 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a540b1ab0 0x7f3a540cd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 --2- 172.21.15.28:0/2458388899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540ceba0 0x7f3a540b1570 unknown :-1 s=CLOSED pgs=2789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 -- 172.21.15.28:0/2458388899 >> 172.21.15.28:0/2458388899 conn(0x7f3a54069e80 msgr2=0x7f3a5406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 -- 172.21.15.28:0/2458388899 shutdown_connections 2024-09-13T17:24:28.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.923+0000 7f3a5c000640 1 -- 172.21.15.28:0/2458388899 wait complete. 2024-09-13T17:24:28.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.924+0000 7f3a5c000640 1 Processor -- start 2024-09-13T17:24:28.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 -- start start 2024-09-13T17:24:28.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a540ceba0 0x7f3a54152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a5415a7a0 0x7f3a5415b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5415d5a0 con 0x7f3a540b1ab0 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5415b000 con 0x7f3a540ceba0 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.925+0000 7f3a5c000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5415cfe0 con 0x7f3a5415a7a0 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56186/0 (socket says 172.21.15.28:56186) 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 -- 172.21.15.28:0/1761689816 learned_addr learned my addr 172.21.15.28:0/1761689816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a58c00640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a540ceba0 0x7f3a54152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 -- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a5415a7a0 msgr2=0x7f3a5415b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a5415a7a0 0x7f3a5415b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 -- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a540ceba0 msgr2=0x7f3a54152570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a540ceba0 0x7f3a54152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 -- 172.21.15.28:0/1761689816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400094d0 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.926+0000 7f3a59600640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800c380 tx=0x7f3a4800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.927+0000 7f3a51600640 1 -- 172.21.15.28:0/1761689816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a480052d0 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.927+0000 7f3a51600640 1 -- 172.21.15.28:0/1761689816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a48005430 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.927+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5415d2c0 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.927+0000 7f3a51600640 1 -- 172.21.15.28:0/1761689816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a48003830 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.927+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a541474d0 con 0x7f3a540b1ab0 2024-09-13T17:24:28.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.929+0000 7f3a51600640 1 -- 172.21.15.28:0/1761689816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a48003990 con 0x7f3a540b1ab0 2024-09-13T17:24:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.930+0000 7f3a51600640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 0x7f3a3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.930+0000 7f3a51600640 1 -- 172.21.15.28:0/1761689816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a48088740 con 0x7f3a540b1ab0 2024-09-13T17:24:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.931+0000 7f3a58c00640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 0x7f3a3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.931+0000 7f3a5edd4740 1 Processor -- start 2024-09-13T17:24:28.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.931+0000 7f3a5edd4740 1 -- start start 2024-09-13T17:24:28.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.931+0000 7f3a58c00640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 0x7f3a3406d0c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3a40034000 tx=0x7f3a4002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490cf45b0 0x561490cf4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d37f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490cb0870 con 0x561490cf1d70 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490c5db90 con 0x561490d37ae0 2024-09-13T17:24:28.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561490c5b320 con 0x561490cf45b0 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a58c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56200/0 (socket says 172.21.15.28:56200) 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a58c00640 1 -- 172.21.15.28:0/199895670 learned_addr learned my addr 172.21.15.28:0/199895670 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.932+0000 7f3a59600640 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490cf45b0 0x561490cf4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a5a000640 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d37f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 -- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490cf45b0 msgr2=0x561490cf4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490cf45b0 0x561490cf4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 -- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 msgr2=0x561490d37f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d37f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 -- 172.21.15.28:0/199895670 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561490bf8800 con 0x561490cf1d70 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a58c00640 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f3a40035040 tx=0x7f3a4002c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51d51aff346ba53f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a46000640 1 -- 172.21.15.28:0/199895670 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a40048630 con 0x561490cf1d70 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a46000640 1 -- 172.21.15.28:0/199895670 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40042070 con 0x561490cf1d70 2024-09-13T17:24:28.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.933+0000 7f3a46000640 1 -- 172.21.15.28:0/199895670 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a400489b0 con 0x561490cf1d70 2024-09-13T17:24:28.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 -- 172.21.15.28:0/199895670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 msgr2=0x561490ceeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f3a40035040 tx=0x7f3a4002c000 comp rx=0 tx=0).stop 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 -- 172.21.15.28:0/199895670 shutdown_connections 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d37f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf1d70 0x561490ceeba0 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/199895670 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490cf45b0 0x561490cf4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 -- 172.21.15.28:0/199895670 >> 172.21.15.28:0/199895670 conn(0x561490cf0d70 msgr2=0x561490d39430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 -- 172.21.15.28:0/199895670 shutdown_connections 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.934+0000 7f3a5edd4740 1 -- 172.21.15.28:0/199895670 wait complete. 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.935+0000 7f3a5edd4740 1 -- start start 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.935+0000 7f3a5edd4740 1 Processor -- start 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d3a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490d09ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490c56c10 con 0x561490cf45b0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490c14fd0 con 0x561490d37ae0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561490c14c70 con 0x561490d3abc0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5edd4740 10 client.0 _ll_register_callbacks cb 0x561490c50280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56202/0 (socket says 172.21.15.28:56202) 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a59600640 1 -- 172.21.15.28:0/3344206562 learned_addr learned my addr 172.21.15.28:0/3344206562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a5a000640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490d09ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.936+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d3a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 -- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 msgr2=0x561490d09ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490d09ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 -- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 msgr2=0x561490d3a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d3a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 -- 172.21.15.28:0/3344206562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a540b2430 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d3a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a59600640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f3a48057500 tx=0x7f3a48057570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=602e7b6716970bad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4805f960 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a4805f4d0 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561490c800f0 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a4805f6b0 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a5edd4740 10 client.12603 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.937+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x561490c55290 con 0x561490cf45b0 2024-09-13T17:24:28.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561490c800f0 con 0x561490cf45b0 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3a48080b00 con 0x561490cf45b0 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 10 client.12603 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 10 client.12603 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 20 client.12603 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d4c290 0x561490d4c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 4 client.12603 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561490c4b3f0 con 0x561490d4c290 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.938+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d4c290 0x561490d4c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.939+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d4c290 0x561490d4c6a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4f6cbb719c3220 server_cookie=a3ca375c52cac9ca in_seq=0 out_seq=0 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.939+0000 7f3a3aa00640 10 client.12603 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.939+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3344206562 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x561490c4b3f0 con 0x561490d4c290 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.939+0000 7f3a3aa00640 10 client.12603 handle_command_reply: tid=0 2024-09-13T17:24:28.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.939+0000 7f3a5edd4740 1 client.12603 shutdown 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 msgr2=0x561490d1cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f3a48057500 tx=0x7f3a48057570 comp rx=0 tx=0).stop 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 shutdown_connections 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d4c290 0x561490d4c6a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490d09ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d3a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3344206562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1cb30 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 >> 172.21.15.28:0/3344206562 conn(0x561490cf1d70 msgr2=0x561490d3c930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a3aa00640 0 client.12603 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 shutdown_connections 2024-09-13T17:24:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3344206562 wait complete. 2024-09-13T17:24:28.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.940+0000 7f3a5edd4740 20 client.12603 trim_cache size 0 max 16384 2024-09-13T17:24:28.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.955+0000 7f3a5edd4740 1 Processor -- start 2024-09-13T17:24:28.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.955+0000 7f3a5edd4740 1 -- start start 2024-09-13T17:24:28.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d37ae0 0x561490d1e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d3abc0 0x561490d1c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5614909be0c0 con 0x561490cf45b0 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490cf0b60 con 0x561490d3abc0 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561490d5b9d0 con 0x561490d37ae0 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56218/0 (socket says 172.21.15.28:56218) 2024-09-13T17:24:28.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.956+0000 7f3a59600640 1 -- 172.21.15.28:0/404327904 learned_addr learned my addr 172.21.15.28:0/404327904 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 -- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d37ae0 msgr2=0x561490d1e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d37ae0 0x561490d1e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 -- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d3abc0 msgr2=0x561490d1c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d3abc0 0x561490d1c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 -- 172.21.15.28:0/404327904 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561490cf2800 con 0x561490cf45b0 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a59600640 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f3a48001eb0 tx=0x7f3a48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcf092f6b8c2b5fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a46a00640 1 -- 172.21.15.28:0/404327904 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800e040 con 0x561490cf45b0 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a46a00640 1 -- 172.21.15.28:0/404327904 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a480538b0 con 0x561490cf45b0 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.957+0000 7f3a46a00640 1 -- 172.21.15.28:0/404327904 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4805f960 con 0x561490cf45b0 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 -- 172.21.15.28:0/404327904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 msgr2=0x561490d1e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 secure :-1 s=READY pgs=2793 cs=0 l=1 rev1=1 crypto rx=0x7f3a48001eb0 tx=0x7f3a48014000 comp rx=0 tx=0).stop 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 -- 172.21.15.28:0/404327904 shutdown_connections 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d3abc0 0x561490d1c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d37ae0 0x561490d1e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/404327904 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d1e860 unknown :-1 s=CLOSED pgs=2793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 -- 172.21.15.28:0/404327904 >> 172.21.15.28:0/404327904 conn(0x561490cf1d70 msgr2=0x561490d333f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 -- 172.21.15.28:0/404327904 shutdown_connections 2024-09-13T17:24:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.958+0000 7f3a5edd4740 1 -- 172.21.15.28:0/404327904 wait complete. 2024-09-13T17:24:28.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.959+0000 7f3a5edd4740 1 -- start start 2024-09-13T17:24:28.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.959+0000 7f3a5edd4740 1 Processor -- start 2024-09-13T17:24:28.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d1da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490cefc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490d08680 con 0x561490cf45b0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561490d3d8a0 con 0x561490d37ae0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x561490d38550 con 0x561490d3abc0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a5edd4740 10 client.0 _ll_register_callbacks cb 0x7f3a48056f40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a59600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56222/0 (socket says 172.21.15.28:56222) 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.960+0000 7f3a59600640 1 -- 172.21.15.28:0/3571591350 learned_addr learned my addr 172.21.15.28:0/3571591350 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d1da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a5a000640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490cefc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 -- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 msgr2=0x561490cefc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490cefc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 -- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 msgr2=0x561490d1da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d1da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 -- 172.21.15.28:0/3571591350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561490c55290 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a5a000640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490cefc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a59600640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f3a4806f0f0 tx=0x7f3a48086ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=806b060363059522 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a3aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4800e040 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3a4805f960 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a40042070 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a4805fbb0 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a5edd4740 10 client.12609 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.961+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x561490c88ee0 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40042070 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3a4805fd10 con 0x561490cf45b0 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 10 client.12609 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 10 client.12609 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 20 client.12609 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:28.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d1df50 0x561490cf0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 4 client.12609 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}] 2024-09-13T17:24:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}) v1 -- 0x561490c48d00 con 0x561490d1df50 2024-09-13T17:24:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.962+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d1df50 0x561490cf0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.963+0000 7f3a58c00640 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d1df50 0x561490cf0190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64d66fc5e1300f44 server_cookie=8500222d03893bfa in_seq=0 out_seq=0 2024-09-13T17:24:28.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.963+0000 7f3a3aa00640 10 client.12609 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.963+0000 7f3a3aa00640 1 -- 172.21.15.28:0/3571591350 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561490c48d00 con 0x561490d1df50 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.963+0000 7f3a3aa00640 10 client.12609 handle_command_reply: tid=0 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.963+0000 7f3a5edd4740 1 client.12609 shutdown 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 msgr2=0x561490d89690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7f3a4806f0f0 tx=0x7f3a48086ad0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 shutdown_connections 2024-09-13T17:24:28.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x561490d1df50 0x561490cf0190 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x561490d3abc0 0x561490cefc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561490d37ae0 0x561490d1da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 --2- 172.21.15.28:0/3571591350 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561490cf45b0 0x561490d89690 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 >> 172.21.15.28:0/3571591350 conn(0x561490cf1d70 msgr2=0x561490d1c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a3aa00640 0 client.12609 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:28.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 shutdown_connections 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 1 -- 172.21.15.28:0/3571591350 wait complete. 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.964+0000 7f3a5edd4740 20 client.12609 trim_cache size 0 max 16384 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "fd51e398-8268-473c-bfdb-21169bb2df6e", 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:24:28.969 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 msgr2=0x7f3a3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 0x7f3a3406d0c0 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f3a40034000 tx=0x7f3a4002e000 comp rx=0 tx=0).stop 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 msgr2=0x7f3a54146f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7f3a4800c380 tx=0x7f3a4800c840 comp rx=0 tx=0).stop 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 shutdown_connections 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a3406ac00 0x7f3a3406d0c0 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a5415a7a0 0x7f3a5415b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a540ceba0 0x7f3a54152570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 --2- 172.21.15.28:0/1761689816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a540b1ab0 0x7f3a54146f90 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 >> 172.21.15.28:0/1761689816 conn(0x7f3a54069e80 msgr2=0x7f3a540b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 shutdown_connections 2024-09-13T17:24:28.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:28.965+0000 7f3a5c000640 1 -- 172.21.15.28:0/1761689816 wait complete. 2024-09-13T17:24:28.981 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub status 2024-09-13T17:24:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.139+0000 7f4e7d400640 1 Processor -- start 2024-09-13T17:24:29.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.139+0000 7f4e7d400640 1 -- start start 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e78159770 0x7f4e7806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e7806f3f0 0x7f4e78070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f4e780add00 0x7f4e78159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7806f920 con 0x7f4e7806f3f0 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e7815b290 con 0x7f4e780add00 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.140+0000 7f4e7d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7815b570 con 0x7f4e78159770 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e7806f3f0 0x7f4e78070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f4e7806f3f0 0x7f4e78070260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56224/0 (socket says 172.21.15.28:56224) 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 -- 172.21.15.28:0/3183707794 learned_addr learned my addr 172.21.15.28:0/3183707794 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e77200640 1 --2- 172.21.15.28:0/3183707794 >> v2:172.21.15.189:3300/0 conn(0x7f4e780add00 0x7f4e78159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e76800640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e78159770 0x7f4e7806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 -- 172.21.15.28:0/3183707794 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e78159770 msgr2=0x7f4e7806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e78159770 0x7f4e7806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 -- 172.21.15.28:0/3183707794 >> v2:172.21.15.189:3300/0 conn(0x7f4e780add00 msgr2=0x7f4e78159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3183707794 >> v2:172.21.15.189:3300/0 conn(0x7f4e780add00 0x7f4e78159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.141+0000 7f4e75e00640 1 -- 172.21.15.28:0/3183707794 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7815b850 con 0x7f4e7806f3f0 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.142+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e78070260 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f4e7806ce40 tx=0x7f4e6402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3700f1eb62122c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.142+0000 7f4e75400640 1 -- 172.21.15.28:0/3183707794 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6402ae90 con 0x7f4e7806f3f0 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.142+0000 7f4e75400640 1 -- 172.21.15.28:0/3183707794 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64032440 con 0x7f4e7806f3f0 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.142+0000 7f4e75400640 1 -- 172.21.15.28:0/3183707794 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e640327d0 con 0x7f4e7806f3f0 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 -- 172.21.15.28:0/3183707794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 msgr2=0x7f4e78070260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e78070260 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7f4e7806ce40 tx=0x7f4e6402a140 comp rx=0 tx=0).stop 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 -- 172.21.15.28:0/3183707794 shutdown_connections 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e78070260 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 --2- 172.21.15.28:0/3183707794 >> v2:172.21.15.189:3300/0 conn(0x7f4e780add00 0x7f4e78159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 --2- 172.21.15.28:0/3183707794 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e78159770 0x7f4e7806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 -- 172.21.15.28:0/3183707794 >> 172.21.15.28:0/3183707794 conn(0x7f4e780b0270 msgr2=0x7f4e780b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 -- 172.21.15.28:0/3183707794 shutdown_connections 2024-09-13T17:24:29.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.143+0000 7f4e7d400640 1 -- 172.21.15.28:0/3183707794 wait complete. 2024-09-13T17:24:29.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.144+0000 7f4e7d400640 1 Processor -- start 2024-09-13T17:24:29.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.145+0000 7f4e7d400640 1 -- start start 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.145+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.145+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e780add00 0x7f4e780c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.145+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 0x7f4e7814ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e7d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780c3a20 con 0x7f4e7806f3f0 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e7d400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780aab30 con 0x7f4e78159770 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e7d400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780ab0f0 con 0x7f4e780add00 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56232/0 (socket says 172.21.15.28:56232) 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 -- 172.21.15.28:0/2562790908 learned_addr learned my addr 172.21.15.28:0/2562790908 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e75e00640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e780add00 0x7f4e780c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e77200640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 0x7f4e7814ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 -- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e780add00 msgr2=0x7f4e780c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e780add00 0x7f4e780c34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 -- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 msgr2=0x7f4e7814ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 0x7f4e7814ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e76800640 1 -- 172.21.15.28:0/2562790908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e640094d0 con 0x7f4e7806f3f0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.146+0000 7f4e77200640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 0x7f4e7814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e76800640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800d5f0 tx=0x7f4e6800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e6ea00640 1 -- 172.21.15.28:0/2562790908 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680125f0 con 0x7f4e7806f3f0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e6ea00640 1 -- 172.21.15.28:0/2562790908 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68012750 con 0x7f4e7806f3f0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e780ad510 con 0x7f4e7806f3f0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e6ea00640 1 -- 172.21.15.28:0/2562790908 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68012b00 con 0x7f4e7806f3f0 2024-09-13T17:24:29.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.147+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e780aad80 con 0x7f4e7806f3f0 2024-09-13T17:24:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.149+0000 7f4e6ea00640 1 -- 172.21.15.28:0/2562790908 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4e68012c60 con 0x7f4e7806f3f0 2024-09-13T17:24:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.150+0000 7f4e6ea00640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 0x7f4e5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.150+0000 7f4e6ea00640 1 -- 172.21.15.28:0/2562790908 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e68089710 con 0x7f4e7806f3f0 2024-09-13T17:24:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.150+0000 7f4e75e00640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 0x7f4e5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.151+0000 7f4e80233740 1 Processor -- start 2024-09-13T17:24:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.151+0000 7f4e80233740 1 -- start start 2024-09-13T17:24:29.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.151+0000 7f4e75e00640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 0x7f4e5806d0f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f4e64034000 tx=0x7f4e6402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.151+0000 7f4e80233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f41420 0x55cda7f91130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f91670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e80233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7e4a6a0 con 0x55cda7f43c60 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7eadcc0 con 0x55cda7f6b110 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cda7eab450 con 0x55cda7f41420 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56234/0 (socket says 172.21.15.28:56234) 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 -- 172.21.15.28:0/1026541259 learned_addr learned my addr 172.21.15.28:0/1026541259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e75e00640 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f41420 0x55cda7f91130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e77200640 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f91670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 -- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f41420 msgr2=0x55cda7f91130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f41420 0x55cda7f91130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 -- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 msgr2=0x55cda7f91670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f91670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 -- 172.21.15.28:0/1026541259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda7e772e0 con 0x55cda7f43c60 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.152+0000 7f4e76800640 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f4e68087b80 tx=0x7f4e68058450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3122f1e115e0c3bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e63400640 1 -- 172.21.15.28:0/1026541259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6805c170 con 0x55cda7f43c60 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e63400640 1 -- 172.21.15.28:0/1026541259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6805c380 con 0x55cda7f43c60 2024-09-13T17:24:29.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e63400640 1 -- 172.21.15.28:0/1026541259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68060940 con 0x55cda7f43c60 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 -- 172.21.15.28:0/1026541259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 msgr2=0x55cda7f44040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f4e68087b80 tx=0x7f4e68058450 comp rx=0 tx=0).stop 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 -- 172.21.15.28:0/1026541259 shutdown_connections 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f91670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f41420 0x55cda7f91130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 --2- 172.21.15.28:0/1026541259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f44040 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 -- 172.21.15.28:0/1026541259 >> 172.21.15.28:0/1026541259 conn(0x55cda7f40390 msgr2=0x55cda7f88840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 -- 172.21.15.28:0/1026541259 shutdown_connections 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.153+0000 7f4e80233740 1 -- 172.21.15.28:0/1026541259 wait complete. 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.154+0000 7f4e80233740 1 -- start start 2024-09-13T17:24:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.154+0000 7f4e80233740 1 Processor -- start 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f6b110 0x55cda7f579a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f85d70 0x55cda7f69a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7ea6d40 con 0x55cda7f43c60 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7e48280 con 0x55cda7f85d70 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cda7e47f20 con 0x55cda7f6b110 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e80233740 10 client.0 _ll_register_callbacks cb 0x55cda7ea03b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56242/0 (socket says 172.21.15.28:56242) 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.155+0000 7f4e76800640 1 -- 172.21.15.28:0/3191964058 learned_addr learned my addr 172.21.15.28:0/3191964058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f6b110 0x55cda7f579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e77200640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f85d70 0x55cda7f69a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 -- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f6b110 msgr2=0x55cda7f579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f6b110 0x55cda7f579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 -- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f85d70 msgr2=0x55cda7f69a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f85d70 0x55cda7f69a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 -- 172.21.15.28:0/3191964058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e780aad80 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e76800640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f4e68087c30 tx=0x7f4e68020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9355b314716ba419 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68060940 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e680604b0 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.156+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cda7b9a4e0 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e68060670 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 10 client.12621 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55cda7ea53c0 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6807e020 con 0x55cda7f43c60 2024-09-13T17:24:29.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4e68074020 con 0x55cda7f43c60 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 10 client.12621 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 10 client.12621 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 20 client.12621 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f89330 0x55cda7f89740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 4 client.12621 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cda7bab7e0 con 0x55cda7f89330 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.157+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f89330 0x55cda7f89740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.158+0000 7f4e75e00640 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f89330 0x55cda7f89740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8d845f06a4e62815 server_cookie=d94004858ed50e6c in_seq=0 out_seq=0 2024-09-13T17:24:29.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.158+0000 7f4e57e00640 10 client.12621 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e57e00640 1 -- 172.21.15.28:0/3191964058 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55cda7bab7e0 con 0x55cda7f89330 2024-09-13T17:24:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e57e00640 10 client.12621 handle_command_reply: tid=0 2024-09-13T17:24:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 client.12621 shutdown 2024-09-13T17:24:29.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 msgr2=0x55cda7f54f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f4e68087c30 tx=0x7f4e68020070 comp rx=0 tx=0).stop 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 shutdown_connections 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f89330 0x55cda7f89740 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f85d70 0x55cda7f69a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f6b110 0x55cda7f579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e57e00640 0 client.12621 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 --2- 172.21.15.28:0/3191964058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7f54f00 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 >> 172.21.15.28:0/3191964058 conn(0x55cda7f41420 msgr2=0x55cda7f89f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 shutdown_connections 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 1 -- 172.21.15.28:0/3191964058 wait complete. 2024-09-13T17:24:29.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.159+0000 7f4e80233740 20 client.12621 trim_cache size 0 max 16384 2024-09-13T17:24:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.174+0000 7f4e80233740 1 Processor -- start 2024-09-13T17:24:29.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.174+0000 7f4e80233740 1 -- start start 2024-09-13T17:24:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 0x55cda7f89ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7c0e040 con 0x55cda7f6b110 2024-09-13T17:24:29.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7fa21d0 con 0x55cda7f43c60 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cda7f579d0 con 0x55cda7f85d70 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e75e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e75e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56254/0 (socket says 172.21.15.28:56254) 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e75e00640 1 -- 172.21.15.28:0/4229686797 learned_addr learned my addr 172.21.15.28:0/4229686797 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e77200640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.175+0000 7f4e76800640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 0x55cda7f89ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 -- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 msgr2=0x55cda7f8f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 -- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 msgr2=0x55cda7f89ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 0x55cda7f89ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 -- 172.21.15.28:0/4229686797 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cda7e47610 con 0x55cda7f6b110 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e77200640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e76800640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 0x55cda7f89ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:29.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f4e64032bf0 tx=0x7f4e6403e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9914cabfe3ef941 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e63e00640 1 -- 172.21.15.28:0/4229686797 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64032ea0 con 0x55cda7f6b110 2024-09-13T17:24:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e63e00640 1 -- 172.21.15.28:0/4229686797 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64040070 con 0x55cda7f6b110 2024-09-13T17:24:29.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.176+0000 7f4e63e00640 1 -- 172.21.15.28:0/4229686797 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6403cb30 con 0x55cda7f6b110 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 -- 172.21.15.28:0/4229686797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 msgr2=0x55cda7f688a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f4e64032bf0 tx=0x7f4e6403e040 comp rx=0 tx=0).stop 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 -- 172.21.15.28:0/4229686797 shutdown_connections 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f6b110 0x55cda7f688a0 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 --2- 172.21.15.28:0/4229686797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f43c60 0x55cda7f89ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 -- 172.21.15.28:0/4229686797 >> 172.21.15.28:0/4229686797 conn(0x55cda7f41420 msgr2=0x55cda7faaa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 -- 172.21.15.28:0/4229686797 shutdown_connections 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.177+0000 7f4e80233740 1 -- 172.21.15.28:0/4229686797 wait complete. 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.178+0000 7f4e80233740 1 -- start start 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.178+0000 7f4e80233740 1 Processor -- start 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f84500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7f85800 con 0x55cda7f43c60 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cda7fd81e0 con 0x55cda7f6b110 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cda7f689c0 con 0x55cda7f85d70 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e80233740 10 client.0 _ll_register_callbacks cb 0x7f4e680046a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56260/0 (socket says 172.21.15.28:56260) 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 -- 172.21.15.28:0/4265805600 learned_addr learned my addr 172.21.15.28:0/4265805600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:29.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 -- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 msgr2=0x55cda7f8f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 -- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 msgr2=0x55cda7f84500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f84500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f84500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.179+0000 7f4e76800640 1 -- 172.21.15.28:0/4265805600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e7815b850 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e76800640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f4e68012f60 tx=0x7f4e68012f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=848fdc5bef5b5bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68058c50 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e6807f080 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e6805c380 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e68060940 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 10 client.12627 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55cda7e6cb20 con 0x55cda7f43c60 2024-09-13T17:24:29.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6805c380 con 0x55cda7f43c60 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4e68074040 con 0x55cda7f43c60 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 10 client.12627 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 10 client.12627 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 20 client.12627 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f8ff00 0x55cda7f8ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 4 client.12627 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.180+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cda7bfee30 con 0x55cda7f8ff00 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f8ff00 0x55cda7f8ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e75e00640 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f8ff00 0x55cda7f8ba70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b7830437183d5e server_cookie=7d7a29b0c8a7691e in_seq=0 out_seq=0 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e57e00640 10 client.12627 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e57e00640 1 -- 172.21.15.28:0/4265805600 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x55cda7bfee30 con 0x55cda7f8ff00 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e57e00640 10 client.12627 handle_command_reply: tid=0 2024-09-13T17:24:29.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.181+0000 7f4e80233740 1 client.12627 shutdown 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 msgr2=0x55cda7fd7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 secure :-1 s=READY pgs=2800 cs=0 l=1 rev1=1 crypto rx=0x7f4e68012f60 tx=0x7f4e68012f90 comp rx=0 tx=0).stop 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 shutdown_connections 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55cda7f8ff00 0x55cda7f8ba70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55cda7f85d70 0x55cda7f8f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55cda7f6b110 0x55cda7f84500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 --2- 172.21.15.28:0/4265805600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cda7f43c60 0x55cda7fd7910 unknown :-1 s=CLOSED pgs=2800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e57e00640 0 client.12627 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 >> 172.21.15.28:0/4265805600 conn(0x55cda7f41420 msgr2=0x55cda7f886a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 shutdown_connections 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 1 -- 172.21.15.28:0/4265805600 wait complete. 2024-09-13T17:24:29.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e80233740 20 client.12627 trim_cache size 0 max 16384 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "fd51e398-8268-473c-bfdb-21169bb2df6e": { 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "fd51e398-8268-473c-bfdb-21169bb2df6e", 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,repair" 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:29.188 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 msgr2=0x7f4e5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 0x7f4e5806d0f0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f4e64034000 tx=0x7f4e6402e000 comp rx=0 tx=0).stop 2024-09-13T17:24:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 msgr2=0x7f4e781f23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.182+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 secure :-1 s=READY pgs=2796 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800d5f0 tx=0x7f4e6800dab0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 shutdown_connections 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4e5806ac30 0x7f4e5806d0f0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e78159770 0x7f4e7814ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4e780add00 0x7f4e780c34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 --2- 172.21.15.28:0/2562790908 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e7806f3f0 0x7f4e781f23b0 unknown :-1 s=CLOSED pgs=2796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 >> 172.21.15.28:0/2562790908 conn(0x7f4e780b0270 msgr2=0x7f4e780b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 shutdown_connections 2024-09-13T17:24:29.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:29.183+0000 7f4e7d400640 1 -- 172.21.15.28:0/2562790908 wait complete. 2024-09-13T17:24:29.199 INFO:tasks.cephfs.filesystem:scrub status for tag:fd51e398-8268-473c-bfdb-21169bb2df6e - {'path': '/', 'tag': 'fd51e398-8268-473c-bfdb-21169bb2df6e', 'options': 'recursive,repair'} 2024-09-13T17:24:42.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:24:42.158 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:24:59.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub status 2024-09-13T17:24:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.370+0000 7f1bdba00640 1 Processor -- start 2024-09-13T17:24:59.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.370+0000 7f1bdba00640 1 -- start start 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bdba00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 0x7f1bd4070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bdba00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1bd415ab60 0x7f1bd40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bdba00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bd406c660 con 0x7f1bd406bb40 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bdba00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd406c8b0 con 0x7f1bd415ab60 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bdba00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bd406cb90 con 0x7f1bd415a780 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bd9a00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f1bd406bb40 0x7f1bd406bf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:55774/0 (socket says 172.21.15.28:55774) 2024-09-13T17:24:59.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bcbe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1bd415ab60 0x7f1bd40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.371+0000 7f1bd9a00640 1 -- 172.21.15.28:0/2736258003 learned_addr learned my addr 172.21.15.28:0/2736258003 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bd9000640 1 --2- 172.21.15.28:0/2736258003 >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 0x7f1bd4070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bcbe00640 1 -- 172.21.15.28:0/2736258003 >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 msgr2=0x7f1bd4070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2736258003 >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 0x7f1bd4070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bcbe00640 1 -- 172.21.15.28:0/2736258003 >> v1:172.21.15.28:6789/0 conn(0x7f1bd406bb40 legacy=0x7f1bd406bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bcbe00640 1 -- 172.21.15.28:0/2736258003 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40cc980 con 0x7f1bd415ab60 2024-09-13T17:24:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bd3e00640 1 -- 172.21.15.28:0/2736258003 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950670340 0 0) 0x7f1bd406c660 con 0x7f1bd406bb40 2024-09-13T17:24:59.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.372+0000 7f1bd9000640 1 --2- 172.21.15.28:0/2736258003 >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 0x7f1bd4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.373+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2736258003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40712c0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b1330 tx=0x7f1bbc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93fe92dc0cbf6af1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.373+0000 7f1bd3e00640 1 -- 172.21.15.28:0/2736258003 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc032440 con 0x7f1bd415ab60 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.373+0000 7f1bd3e00640 1 -- 172.21.15.28:0/2736258003 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0325a0 con 0x7f1bd415ab60 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.373+0000 7f1bd3e00640 1 -- 172.21.15.28:0/2736258003 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc032930 con 0x7f1bd415ab60 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 -- 172.21.15.28:0/2736258003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 msgr2=0x7f1bd40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 --2- 172.21.15.28:0/2736258003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40712c0 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b1330 tx=0x7f1bbc02a1a0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 -- 172.21.15.28:0/2736258003 shutdown_connections 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 --2- 172.21.15.28:0/2736258003 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40712c0 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 --2- 172.21.15.28:0/2736258003 >> v2:172.21.15.189:3300/0 conn(0x7f1bd415a780 0x7f1bd4070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 -- 172.21.15.28:0/2736258003 >> 172.21.15.28:0/2736258003 conn(0x7f1bd40b0270 msgr2=0x7f1bd40b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 -- 172.21.15.28:0/2736258003 shutdown_connections 2024-09-13T17:24:59.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.374+0000 7f1bdba00640 1 -- 172.21.15.28:0/2736258003 wait complete. 2024-09-13T17:24:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.375+0000 7f1bdba00640 1 Processor -- start 2024-09-13T17:24:59.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.376+0000 7f1bdba00640 1 -- start start 2024-09-13T17:24:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 0x7f1bd4149020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bd415a780 0x7f1bd40ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd4149560 con 0x7f1bd415ab60 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40aa4c0 con 0x7f1bd406bb40 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bdba00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd40aa7a0 con 0x7f1bd415a780 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 0x7f1bd4149020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 0x7f1bd4149020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:35268/0 (socket says 172.21.15.28:35268) 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59118/0 (socket says 172.21.15.28:59118) 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9000640 1 -- 172.21.15.28:0/1500526778 learned_addr learned my addr 172.21.15.28:0/1500526778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bd415a780 0x7f1bd40ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 -- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bd415a780 msgr2=0x7f1bd40ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bd415a780 0x7f1bd40ab560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 -- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 msgr2=0x7f1bd4149020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 0x7f1bd4149020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.377+0000 7f1bd9a00640 1 -- 172.21.15.28:0/1500526778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bbc009400 con 0x7f1bd415ab60 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc001560 tx=0x7f1bcc0021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bd1600640 1 -- 172.21.15.28:0/1500526778 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc013070 con 0x7f1bd415ab60 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bd1600640 1 -- 172.21.15.28:0/1500526778 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bcc004b60 con 0x7f1bd415ab60 2024-09-13T17:24:59.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bd1600640 1 -- 172.21.15.28:0/1500526778 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc010810 con 0x7f1bd415ab60 2024-09-13T17:24:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd40aaa80 con 0x7f1bd415ab60 2024-09-13T17:24:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.378+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd40ab040 con 0x7f1bd415ab60 2024-09-13T17:24:59.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.380+0000 7f1bd1600640 1 -- 172.21.15.28:0/1500526778 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1bcc010970 con 0x7f1bd415ab60 2024-09-13T17:24:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.381+0000 7f1bd1600640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 0x7f1bb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.381+0000 7f1bd1600640 1 -- 172.21.15.28:0/1500526778 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1bcc08cd10 con 0x7f1bd415ab60 2024-09-13T17:24:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.382+0000 7f1bd9000640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 0x7f1bb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.382+0000 7f1bde7e2740 1 Processor -- start 2024-09-13T17:24:59.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.382+0000 7f1bde7e2740 1 -- start start 2024-09-13T17:24:59.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bd9000640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 0x7f1bb006d0c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006d40 tx=0x7f1bc4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbbc040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcc09670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcac26a0 con 0x55a9dcbb9420 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcb25cc0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcb23450 con 0x55a9dcbe3110 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59128/0 (socket says 172.21.15.28:59128) 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1314359007 learned_addr learned my addr 172.21.15.28:0/1314359007 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bd9000640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbbc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.383+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcc09670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 msgr2=0x55a9dcc09670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcc09670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 msgr2=0x55a9dcbbc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbbc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1314359007 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9dcaef2e0 con 0x55a9dcbb9420 2024-09-13T17:24:59.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcc09670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc035040 tx=0x7f1bbc02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c462f9816a7376d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bc9600640 1 -- 172.21.15.28:0/1314359007 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc032a90 con 0x55a9dcbb9420 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bc9600640 1 -- 172.21.15.28:0/1314359007 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc032c10 con 0x55a9dcbb9420 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.384+0000 7f1bc9600640 1 -- 172.21.15.28:0/1314359007 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc03c440 con 0x55a9dcbb9420 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1314359007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 msgr2=0x55a9dcc09130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc035040 tx=0x7f1bbc02c000 comp rx=0 tx=0).stop 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1314359007 shutdown_connections 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcc09670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbb9420 0x55a9dcc09130 unknown :-1 s=CLOSED pgs=2804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1314359007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbbc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1314359007 >> 172.21.15.28:0/1314359007 conn(0x55a9dcbb8390 msgr2=0x55a9dcc00840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1314359007 shutdown_connections 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.385+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1314359007 wait complete. 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.386+0000 7f1bde7e2740 1 -- start start 2024-09-13T17:24:59.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.386+0000 7f1bde7e2740 1 Processor -- start 2024-09-13T17:24:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcbcd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbb68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcb1ed40 con 0x55a9dcbbbc60 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcac0280 con 0x55a9dcbe6bf0 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcabff20 con 0x55a9dcbe3110 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bde7e2740 10 client.0 _ll_register_callbacks cb 0x55a9dcb183b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bd9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59138/0 (socket says 172.21.15.28:59138) 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bd9000640 1 -- 172.21.15.28:0/2975777592 learned_addr learned my addr 172.21.15.28:0/2975777592 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcbcd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.387+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbb68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 -- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 msgr2=0x55a9dcbcd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcbcd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 -- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 msgr2=0x55a9dcbb68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbb68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 -- 172.21.15.28:0/2975777592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd40cc980 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcbcd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bd9000640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40157c0 tx=0x7f1bc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f539f18109589ff7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bc2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc4014960 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1bc4014ac0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9dc8124e0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bc4014c80 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bde7e2740 10 client.12642 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.388+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55a9dcb1d3c0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a9dc8124e0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1bc4028020 con 0x55a9dcbbbc60 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 10 client.12642 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 10 client.12642 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 20 client.12642 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe1600 0x55a9dcbe19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 4 client.12642 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9dc8237e0 con 0x55a9dcbe1600 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.389+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe1600 0x55a9dcbe19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe1600 0x55a9dcbe19e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e140e4c751fb54e server_cookie=eeb220802622abfa in_seq=0 out_seq=0 2024-09-13T17:24:59.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bc2000640 10 client.12642 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bc2000640 1 -- 172.21.15.28:0/2975777592 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55a9dc8237e0 con 0x55a9dcbe1600 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bc2000640 10 client.12642 handle_command_reply: tid=0 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 client.12642 shutdown 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 msgr2=0x55a9dcbfdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f1bc40157c0 tx=0x7f1bc40046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 shutdown_connections 2024-09-13T17:24:59.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe1600 0x55a9dcbe19e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbb68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe3110 0x55a9dcbcd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bc2000640 0 client.12642 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/2975777592 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbbbc60 0x55a9dcbfdf90 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.390+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 >> 172.21.15.28:0/2975777592 conn(0x55a9dcbb9420 msgr2=0x55a9dcbfeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.391+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 shutdown_connections 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.391+0000 7f1bde7e2740 1 -- 172.21.15.28:0/2975777592 wait complete. 2024-09-13T17:24:59.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.391+0000 7f1bde7e2740 20 client.12642 trim_cache size 0 max 16384 2024-09-13T17:24:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 Processor -- start 2024-09-13T17:24:59.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 -- start start 2024-09-13T17:24:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcc22c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 0x55a9dcc028a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dc886040 con 0x55a9dcbe3110 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcbe0b80 con 0x55a9dcbbbc60 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.406+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcbbd5a0 con 0x55a9dcbe6bf0 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59146/0 (socket says 172.21.15.28:59146) 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1396483274 learned_addr learned my addr 172.21.15.28:0/1396483274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bd9000640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcc22c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 0x55a9dcc028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 msgr2=0x55a9dcc028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 0x55a9dcc028a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 msgr2=0x55a9dcc22c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcc22c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 -- 172.21.15.28:0/1396483274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9dcabf610 con 0x55a9dcbe3110 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 0x55a9dcc028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bd9000640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcc22c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.407+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc000c00 tx=0x7f1bbc008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc5828bd2e57dfba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bca000640 1 -- 172.21.15.28:0/1396483274 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0040c0 con 0x55a9dcbe3110 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bca000640 1 -- 172.21.15.28:0/1396483274 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc004240 con 0x55a9dcbe3110 2024-09-13T17:24:59.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bca000640 1 -- 172.21.15.28:0/1396483274 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0043f0 con 0x55a9dcbe3110 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1396483274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 msgr2=0x55a9dcc07cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc000c00 tx=0x7f1bbc008030 comp rx=0 tx=0).stop 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1396483274 shutdown_connections 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbe6bf0 0x55a9dcc028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc07cb0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/1396483274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbbbc60 0x55a9dcc22c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1396483274 >> 172.21.15.28:0/1396483274 conn(0x55a9dcbb9420 msgr2=0x55a9dcbcf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1396483274 shutdown_connections 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.408+0000 7f1bde7e2740 1 -- 172.21.15.28:0/1396483274 wait complete. 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.409+0000 7f1bde7e2740 1 -- start start 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.409+0000 7f1bde7e2740 1 Processor -- start 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbbbc60 0x55a9dcc53a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbe56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcc53f70 con 0x55a9dcbe3110 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcc6bb70 con 0x55a9dcbe6bf0 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a9dcbcbac0 con 0x55a9dcbbbc60 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bde7e2740 10 client.0 _ll_register_callbacks cb 0x55a9dcc08fa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bcbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59150/0 (socket says 172.21.15.28:59150) 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bcbe00640 1 -- 172.21.15.28:0/4131605865 learned_addr learned my addr 172.21.15.28:0/4131605865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bd9000640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbbbc60 0x55a9dcc53a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.410+0000 7f1bd9a00640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbe56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 -- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbbbc60 msgr2=0x55a9dcc53a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbbbc60 0x55a9dcc53a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 -- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 msgr2=0x55a9dcbe56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbe56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 -- 172.21.15.28:0/4131605865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bbc04a040 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bcbe00640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009890 tx=0x7f1bbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10df45e1a14ceaa4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bc2000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc003820 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1bbc003980 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.411+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1bbc032c10 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1bbc003b40 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 10 client.12648 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55a9dcae4b20 con 0x55a9dcbe3110 2024-09-13T17:24:59.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc032c10 con 0x55a9dcbe3110 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1bbc003ca0 con 0x55a9dcbe3110 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 10 client.12648 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 10 client.12648 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 20 client.12648 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe5c30 0x55a9dcbfe830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 4 client.12648 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a9dc876e30 con 0x55a9dcbe5c30 2024-09-13T17:24:59.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.412+0000 7f1bd9000640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe5c30 0x55a9dcbfe830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bd9000640 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe5c30 0x55a9dcbfe830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1ca08c0e6e46069 server_cookie=fa6cc9f10ecc85ac in_seq=0 out_seq=0 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bc2000640 10 client.12648 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bc2000640 1 -- 172.21.15.28:0/4131605865 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a9dc876e30 con 0x55a9dcbe5c30 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bc2000640 10 client.12648 handle_command_reply: tid=0 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 client.12648 shutdown 2024-09-13T17:24:59.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 msgr2=0x55a9dcc01300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc009890 tx=0x7f1bbc004730 comp rx=0 tx=0).stop 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 shutdown_connections 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55a9dcbe5c30 0x55a9dcbfe830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a9dcbe6bf0 0x55a9dcbe56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a9dcbe3110 0x55a9dcc01300 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bc2000640 0 client.12648 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 --2- 172.21.15.28:0/4131605865 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55a9dcbbbc60 0x55a9dcc53a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 >> 172.21.15.28:0/4131605865 conn(0x55a9dcbb9420 msgr2=0x55a9dcbbb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.413+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 shutdown_connections 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bde7e2740 1 -- 172.21.15.28:0/4131605865 wait complete. 2024-09-13T17:24:59.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bde7e2740 20 client.12648 trim_cache size 0 max 16384 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 msgr2=0x7f1bb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 0x7f1bb006d0c0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4006d40 tx=0x7f1bc4008040 comp rx=0 tx=0).stop 2024-09-13T17:24:59.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 msgr2=0x7f1bd40cdb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 secure :-1 s=READY pgs=2803 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc001560 tx=0x7f1bcc0021d0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 shutdown_connections 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1bb006ac00 0x7f1bb006d0c0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd415ab60 0x7f1bd40cdb30 unknown :-1 s=CLOSED pgs=2803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1bd415a780 0x7f1bd40ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 --2- 172.21.15.28:0/1500526778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1bd406bb40 0x7f1bd4149020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 >> 172.21.15.28:0/1500526778 conn(0x7f1bd40b0270 msgr2=0x7f1bd4154490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 shutdown_connections 2024-09-13T17:24:59.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.414+0000 7f1bdba00640 1 -- 172.21.15.28:0/1500526778 wait complete. 2024-09-13T17:24:59.431 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:24:59.431 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub start / recursive,repair 2024-09-13T17:24:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.589+0000 7feabf200640 1 Processor -- start 2024-09-13T17:24:59.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.589+0000 7feabf200640 1 -- start start 2024-09-13T17:24:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabf200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159970 0x7feab80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabf200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feab80b2630 0x7feab815a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabf200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feab815a4c0 con 0x7feab8159590 2024-09-13T17:24:59.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabf200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab80b2ae0 con 0x7feab80b2630 2024-09-13T17:24:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabf200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8071390 con 0x7feab8159970 2024-09-13T17:24:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feab80b2630 0x7feab815a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabd200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feab80b2630 0x7feab815a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59166/0 (socket says 172.21.15.28:59166) 2024-09-13T17:24:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.590+0000 7feabd200640 1 -- 172.21.15.28:0/3634374302 learned_addr learned my addr 172.21.15.28:0/3634374302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.591+0000 7feab7400640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159970 0x7feab80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.591+0000 7feabd200640 1 -- 172.21.15.28:0/3634374302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159970 msgr2=0x7feab80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.591+0000 7feabd200640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159970 0x7feab80b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.591+0000 7feabd200640 1 -- 172.21.15.28:0/3634374302 >> v1:172.21.15.189:6789/0 conn(0x7feab8159590 legacy=0x7feab8070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.591+0000 7feabd200640 1 -- 172.21.15.28:0/3634374302 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8071580 con 0x7feab80b2630 2024-09-13T17:24:59.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.592+0000 7feabd200640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab815a950 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7feab000d150 tx=0x7feab002c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c12f3388ff075740 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.592+0000 7feab6a00640 1 -- 172.21.15.28:0/3634374302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0009600 con 0x7feab80b2630 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.592+0000 7feab6a00640 1 -- 172.21.15.28:0/3634374302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0009760 con 0x7feab80b2630 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.592+0000 7feab6a00640 1 -- 172.21.15.28:0/3634374302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0009af0 con 0x7feab80b2630 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 -- 172.21.15.28:0/3634374302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 msgr2=0x7feab815a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab815a950 secure :-1 s=READY pgs=2808 cs=0 l=1 rev1=1 crypto rx=0x7feab000d150 tx=0x7feab002c930 comp rx=0 tx=0).stop 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 -- 172.21.15.28:0/3634374302 shutdown_connections 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab815a950 unknown :-1 s=CLOSED pgs=2808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 --2- 172.21.15.28:0/3634374302 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159970 0x7feab80b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 -- 172.21.15.28:0/3634374302 >> 172.21.15.28:0/3634374302 conn(0x7feab8069e80 msgr2=0x7feab806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 -- 172.21.15.28:0/3634374302 shutdown_connections 2024-09-13T17:24:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.593+0000 7feabf200640 1 -- 172.21.15.28:0/3634374302 wait complete. 2024-09-13T17:24:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.595+0000 7feabf200640 1 Processor -- start 2024-09-13T17:24:59.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.595+0000 7feabf200640 1 -- start start 2024-09-13T17:24:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159590 0x7feab80ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab8159970 0x7feab80acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab815b0d0 con 0x7feab80b2630 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab815c7a0 con 0x7feab8159970 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabf200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab815c4c0 con 0x7feab8159590 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59170/0 (socket says 172.21.15.28:59170) 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feab7e00640 1 -- 172.21.15.28:0/275737866 learned_addr learned my addr 172.21.15.28:0/275737866 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feab7400640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159590 0x7feab80ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.596+0000 7feabd200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab8159970 0x7feab80acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 -- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159590 msgr2=0x7feab80ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159590 0x7feab80ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 -- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab8159970 msgr2=0x7feab80acd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab8159970 0x7feab80acd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 -- 172.21.15.28:0/275737866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab000bd40 con 0x7feab80b2630 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feab7e00640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7feaa800c480 tx=0x7feaa800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feaafe00640 1 -- 172.21.15.28:0/275737866 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa80052d0 con 0x7feab80b2630 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feaafe00640 1 -- 172.21.15.28:0/275737866 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa8005430 con 0x7feab80b2630 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feaafe00640 1 -- 172.21.15.28:0/275737866 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8012870 con 0x7feab80b2630 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab815aa80 con 0x7feab80b2630 2024-09-13T17:24:59.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.597+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab815c0e0 con 0x7feab80b2630 2024-09-13T17:24:59.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.599+0000 7feaafe00640 1 -- 172.21.15.28:0/275737866 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7feaa80129d0 con 0x7feab80b2630 2024-09-13T17:24:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.600+0000 7feaafe00640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 0x7fea9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.600+0000 7feaafe00640 1 -- 172.21.15.28:0/275737866 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feaa8088e40 con 0x7feab80b2630 2024-09-13T17:24:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.601+0000 7feab7400640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 0x7fea9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.601+0000 7feac201a740 1 Processor -- start 2024-09-13T17:24:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.601+0000 7feac201a740 1 -- start start 2024-09-13T17:24:59.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.601+0000 7feab7400640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 0x7fea9806d0f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7feab814f280 tx=0x7feaa4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5d6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5d3d70 0x555f5a5d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a592870 con 0x555f5a619ae0 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a53fb90 con 0x555f5a5d65b0 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f5a53d320 con 0x555f5a5d3d70 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feabd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feabd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59178/0 (socket says 172.21.15.28:59178) 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feabd200640 1 -- 172.21.15.28:0/2030510368 learned_addr learned my addr 172.21.15.28:0/2030510368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feab7e00640 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5d6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.602+0000 7feab7400640 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5d3d70 0x555f5a5d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 -- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5d3d70 msgr2=0x555f5a5d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5d3d70 0x555f5a5d0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 -- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 msgr2=0x555f5a5d6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5d6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 -- 172.21.15.28:0/2030510368 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f5a4da800 con 0x555f5a619ae0 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feabd200640 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7feab000d150 tx=0x7feab0031000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d64db7a7bb164e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feaacc00640 1 -- 172.21.15.28:0/2030510368 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab00093d0 con 0x555f5a619ae0 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feaacc00640 1 -- 172.21.15.28:0/2030510368 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0009530 con 0x555f5a619ae0 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.603+0000 7feaacc00640 1 -- 172.21.15.28:0/2030510368 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0009af0 con 0x555f5a619ae0 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 -- 172.21.15.28:0/2030510368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 msgr2=0x555f5a619f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7feab000d150 tx=0x7feab0031000 comp rx=0 tx=0).stop 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 -- 172.21.15.28:0/2030510368 shutdown_connections 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a619f90 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5d3d70 0x555f5a5d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 --2- 172.21.15.28:0/2030510368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5d6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 -- 172.21.15.28:0/2030510368 >> 172.21.15.28:0/2030510368 conn(0x555f5a5d2d70 msgr2=0x555f5a61b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 -- 172.21.15.28:0/2030510368 shutdown_connections 2024-09-13T17:24:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.604+0000 7feac201a740 1 -- 172.21.15.28:0/2030510368 wait complete. 2024-09-13T17:24:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.605+0000 7feac201a740 1 -- start start 2024-09-13T17:24:59.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.605+0000 7feac201a740 1 Processor -- start 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a538c10 con 0x555f5a619ae0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a4f6fd0 con 0x555f5a5d65b0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f5a4f6c70 con 0x555f5a5ebcb0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feac201a740 10 client.0 _ll_register_callbacks cb 0x555f5a532280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59190/0 (socket says 172.21.15.28:59190) 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 -- 172.21.15.28:0/1215861266 learned_addr learned my addr 172.21.15.28:0/1215861266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feabd200640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7e00640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 -- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 msgr2=0x555f5a61c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 -- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 msgr2=0x555f5a5e9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5e9fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7400640 1 -- 172.21.15.28:0/1215861266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab8071580 con 0x555f5a619ae0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.606+0000 7feab7e00640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feab7400640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7feaa4014770 tx=0x7feaa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20369f2a433d247b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feaa1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa4014ea0 con 0x555f5a619ae0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f5a5620f0 con 0x555f5a619ae0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feaa401c040 con 0x555f5a619ae0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7feaa4022620 con 0x555f5a619ae0 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feac201a740 10 client.12660 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.607+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x555f5a537290 con 0x555f5a619ae0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555f5a5620f0 con 0x555f5a619ae0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7feaa4022780 con 0x555f5a619ae0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 10 client.12660 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 10 client.12660 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 20 client.12660 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a5fe440 0x555f5a5fe820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 4 client.12660 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f5a52d3f0 con 0x555f5a5fe440 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feab7e00640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a5fe440 0x555f5a5fe820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feab7e00640 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a5fe440 0x555f5a5fe820 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb1d120496a2e3a2 server_cookie=2a57411ee38c5eb1 in_seq=0 out_seq=0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.608+0000 7feaa1600640 10 client.12660 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feaa1600640 1 -- 172.21.15.28:0/1215861266 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x555f5a52d3f0 con 0x555f5a5fe440 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feaa1600640 10 client.12660 handle_command_reply: tid=0 2024-09-13T17:24:59.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 client.12660 shutdown 2024-09-13T17:24:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 msgr2=0x555f5a61c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 secure :-1 s=READY pgs=2811 cs=0 l=1 rev1=1 crypto rx=0x7feaa4014770 tx=0x7feaa40046c0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 shutdown_connections 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a5fe440 0x555f5a5fe820 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a61c4a0 unknown :-1 s=CLOSED pgs=2811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 --2- 172.21.15.28:0/1215861266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a5e9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feaa1600640 0 client.12660 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.609+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 >> 172.21.15.28:0/1215861266 conn(0x555f5a5d3d70 msgr2=0x555f5a5d1e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.610+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 shutdown_connections 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.610+0000 7feac201a740 1 -- 172.21.15.28:0/1215861266 wait complete. 2024-09-13T17:24:59.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.610+0000 7feac201a740 20 client.12660 trim_cache size 0 max 16384 2024-09-13T17:24:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 Processor -- start 2024-09-13T17:24:59.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 -- start start 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a600330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a2a00c0 con 0x555f5a619ae0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a600bc0 con 0x555f5a5d65b0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.625+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f5a61e4b0 con 0x555f5a5ebcb0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59200/0 (socket says 172.21.15.28:59200) 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 -- 172.21.15.28:0/2901580574 learned_addr learned my addr 172.21.15.28:0/2901580574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feab7e00640 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a600330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feab7400640 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 -- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 msgr2=0x555f5a61a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 -- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 msgr2=0x555f5a600330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a600330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 -- 172.21.15.28:0/2901580574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f5a56aee0 con 0x555f5a619ae0 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.626+0000 7feabd200640 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7feab0000f80 tx=0x7feab0032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d54898facd4814d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feaad600640 1 -- 172.21.15.28:0/2901580574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab00093d0 con 0x555f5a619ae0 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feaad600640 1 -- 172.21.15.28:0/2901580574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0003990 con 0x555f5a619ae0 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feaad600640 1 -- 172.21.15.28:0/2901580574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0003d20 con 0x555f5a619ae0 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 -- 172.21.15.28:0/2901580574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 msgr2=0x555f5a67c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 secure :-1 s=READY pgs=2812 cs=0 l=1 rev1=1 crypto rx=0x7feab0000f80 tx=0x7feab0032000 comp rx=0 tx=0).stop 2024-09-13T17:24:59.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 -- 172.21.15.28:0/2901580574 shutdown_connections 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a619ae0 0x555f5a67c410 unknown :-1 s=CLOSED pgs=2812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 --2- 172.21.15.28:0/2901580574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a5d65b0 0x555f5a600330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 -- 172.21.15.28:0/2901580574 >> 172.21.15.28:0/2901580574 conn(0x555f5a5d3d70 msgr2=0x555f5a61efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 -- 172.21.15.28:0/2901580574 shutdown_connections 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.627+0000 7feac201a740 1 -- 172.21.15.28:0/2901580574 wait complete. 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.628+0000 7feac201a740 1 -- start start 2024-09-13T17:24:59.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.628+0000 7feac201a740 1 Processor -- start 2024-09-13T17:24:59.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 0x555f5a6299b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a62a3a0 con 0x555f5a5d65b0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f5a62cea0 con 0x555f5a619ae0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x555f5a5fece0 con 0x555f5a5ebcb0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feac201a740 10 client.0 _ll_register_callbacks cb 0x7feaa4014440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feab7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59210/0 (socket says 172.21.15.28:59210) 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.629+0000 7feab7e00640 1 -- 172.21.15.28:0/4205819933 learned_addr learned my addr 172.21.15.28:0/4205819933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feabd200640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 0x555f5a6299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7400640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 -- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 msgr2=0x555f5a61a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 -- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 msgr2=0x555f5a6299b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 0x555f5a6299b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 -- 172.21.15.28:0/4205819933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f5a4da800 con 0x555f5a5d65b0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7400640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feabd200640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 0x555f5a6299b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feab7e00640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7feaa8057cb0 tx=0x7feaa8005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e9634421462e4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feaa1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaa8012e80 con 0x555f5a5d65b0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feaa805fbd0 con 0x555f5a5d65b0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.630+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feab0009530 con 0x555f5a5d65b0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7feaa805fd90 con 0x555f5a5d65b0 2024-09-13T17:24:59.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 10 client.12666 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x555f5a4d9040 con 0x555f5a5d65b0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaa807a020 con 0x555f5a5d65b0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7feaa8019020 con 0x555f5a5d65b0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 10 client.12666 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 10 client.12666 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 20 client.12666 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a61a5f0 0x555f5a67a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 4 client.12666 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}] 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive", "repair"]}) v1 -- 0x555f5a52ad00 con 0x555f5a61a5f0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.631+0000 7feab7400640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a61a5f0 0x555f5a67a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.632+0000 7feab7400640 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a61a5f0 0x555f5a67a7a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbe91875d8e8e62c server_cookie=7195e6e10a7b9ccb in_seq=0 out_seq=0 2024-09-13T17:24:59.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.632+0000 7feaa1600640 10 client.12666 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.632+0000 7feaa1600640 1 -- 172.21.15.28:0/4205819933 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555f5a52ad00 con 0x555f5a61a5f0 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.632+0000 7feaa1600640 10 client.12666 handle_command_reply: tid=0 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.632+0000 7feac201a740 1 client.12666 shutdown 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 msgr2=0x555f5a5d1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 secure :-1 s=READY pgs=2813 cs=0 l=1 rev1=1 crypto rx=0x7feaa8057cb0 tx=0x7feaa8005040 comp rx=0 tx=0).stop 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 shutdown_connections 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x555f5a61a5f0 0x555f5a67a7a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f5a619ae0 0x555f5a6299b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x555f5a5ebcb0 0x555f5a61a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feaa1600640 0 client.12666 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 --2- 172.21.15.28:0/4205819933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555f5a5d65b0 0x555f5a5d1310 unknown :-1 s=CLOSED pgs=2813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 >> 172.21.15.28:0/4205819933 conn(0x555f5a5d3d70 msgr2=0x555f5a61aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 shutdown_connections 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 1 -- 172.21.15.28:0/4205819933 wait complete. 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.633+0000 7feac201a740 20 client.12666 trim_cache size 0 max 16384 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "dbadb92f-b94f-4a59-90ad-9de809d28bb9", 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-09-13T17:24:59.638 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 msgr2=0x7fea9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 0x7fea9806d0f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7feab814f280 tx=0x7feaa4009190 comp rx=0 tx=0).stop 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 msgr2=0x7feab8148f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7feaa800c480 tx=0x7feaa800c940 comp rx=0 tx=0).stop 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 shutdown_connections 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fea9806ac30 0x7fea9806d0f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feab8159970 0x7feab80acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feab8159590 0x7feab80ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 --2- 172.21.15.28:0/275737866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feab80b2630 0x7feab8148f20 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 >> 172.21.15.28:0/275737866 conn(0x7feab8069e80 msgr2=0x7feab80ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 shutdown_connections 2024-09-13T17:24:59.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.634+0000 7feabf200640 1 -- 172.21.15.28:0/275737866 wait complete. 2024-09-13T17:24:59.650 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub status 2024-09-13T17:24:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.806+0000 7f0e13200640 1 Processor -- start 2024-09-13T17:24:59.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.806+0000 7f0e13200640 1 -- start start 2024-09-13T17:24:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e13200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 0x7f0e0c071a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e13200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e0c0712d0 0x7f0e0c15af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e13200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e0c159e50 con 0x7f0e0c071f60 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e13200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0c0ce210 con 0x7f0e0c0712d0 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e13200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e0c15a130 con 0x7f0e0c0cde30 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e12800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e0c0712d0 0x7f0e0c15af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e12800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e0c0712d0 0x7f0e0c15af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59218/0 (socket says 172.21.15.28:59218) 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e12800640 1 -- 172.21.15.28:0/1153108441 learned_addr learned my addr 172.21.15.28:0/1153108441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.808+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1153108441 >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 0x7f0e0c071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.809+0000 7f0e12800640 1 -- 172.21.15.28:0/1153108441 >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 msgr2=0x7f0e0c071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.809+0000 7f0e12800640 1 --2- 172.21.15.28:0/1153108441 >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 0x7f0e0c071a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.809+0000 7f0e12800640 1 -- 172.21.15.28:0/1153108441 >> v1:172.21.15.189:6789/0 conn(0x7f0e0c071f60 legacy=0x7f0e0c070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.809+0000 7f0e12800640 1 -- 172.21.15.28:0/1153108441 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c15a410 con 0x7f0e0c0712d0 2024-09-13T17:24:59.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.809+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1153108441 >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 0x7f0e0c071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e12800640 1 --2- 172.21.15.28:0/1153108441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c15af10 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c06ad40 tx=0x7f0e0802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c4c91dabceed20 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e10a00640 1 -- 172.21.15.28:0/1153108441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08031600 con 0x7f0e0c0712d0 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e10a00640 1 -- 172.21.15.28:0/1153108441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0802d050 con 0x7f0e0c0712d0 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e10a00640 1 -- 172.21.15.28:0/1153108441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08031980 con 0x7f0e0c0712d0 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e13200640 1 -- 172.21.15.28:0/1153108441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0712d0 msgr2=0x7f0e0c15af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.810+0000 7f0e13200640 1 --2- 172.21.15.28:0/1153108441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c15af10 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c06ad40 tx=0x7f0e0802a730 comp rx=0 tx=0).stop 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 -- 172.21.15.28:0/1153108441 shutdown_connections 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 --2- 172.21.15.28:0/1153108441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c15af10 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 --2- 172.21.15.28:0/1153108441 >> v2:172.21.15.189:3300/0 conn(0x7f0e0c0cde30 0x7f0e0c071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 -- 172.21.15.28:0/1153108441 >> 172.21.15.28:0/1153108441 conn(0x7f0e0c069e80 msgr2=0x7f0e0c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 -- 172.21.15.28:0/1153108441 shutdown_connections 2024-09-13T17:24:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.811+0000 7f0e13200640 1 -- 172.21.15.28:0/1153108441 wait complete. 2024-09-13T17:24:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.812+0000 7f0e13200640 1 Processor -- start 2024-09-13T17:24:59.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.812+0000 7f0e13200640 1 -- start start 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c13ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e0c071f60 0x7f0e0c0adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c13b270 con 0x7f0e0c0cde30 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0af890 con 0x7f0e0c0712d0 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.813+0000 7f0e13200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c0ae7f0 con 0x7f0e0c071f60 2024-09-13T17:24:59.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e11e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c13ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59234/0 (socket says 172.21.15.28:59234) 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 -- 172.21.15.28:0/2339644795 learned_addr learned my addr 172.21.15.28:0/2339644795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e11400640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e0c071f60 0x7f0e0c0adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 -- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e0c071f60 msgr2=0x7f0e0c0adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e0c071f60 0x7f0e0c0adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 -- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0c0712d0 msgr2=0x7f0e0c13ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c13ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 -- 172.21.15.28:0/2339644795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08009b30 con 0x7f0e0c0cde30 2024-09-13T17:24:59.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.814+0000 7f0e12800640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f0e08034040 tx=0x7f0e08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.815+0000 7f0e06000640 1 -- 172.21.15.28:0/2339644795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0803d860 con 0x7f0e0c0cde30 2024-09-13T17:24:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.815+0000 7f0e06000640 1 -- 172.21.15.28:0/2339644795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0803d9c0 con 0x7f0e0c0cde30 2024-09-13T17:24:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.815+0000 7f0e06000640 1 -- 172.21.15.28:0/2339644795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0803dd70 con 0x7f0e0c0cde30 2024-09-13T17:24:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.815+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c0aead0 con 0x7f0e0c0cde30 2024-09-13T17:24:59.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.815+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c0af090 con 0x7f0e0c0cde30 2024-09-13T17:24:59.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.817+0000 7f0e06000640 1 -- 172.21.15.28:0/2339644795 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e0803b400 con 0x7f0e0c0cde30 2024-09-13T17:24:59.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.818+0000 7f0e06000640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 0x7f0df006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.818+0000 7f0e06000640 1 -- 172.21.15.28:0/2339644795 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0e080aa190 con 0x7f0e0c0cde30 2024-09-13T17:24:59.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.818+0000 7f0e17633740 1 Processor -- start 2024-09-13T17:24:59.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.818+0000 7f0e11e00640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 0x7f0df006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.819+0000 7f0e17633740 1 -- start start 2024-09-13T17:24:59.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.819+0000 7f0e11e00640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 0x7f0df006d0c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0e00006fd0 tx=0x7f0e00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f6590 0x5572576462a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x5572576467e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572574ff6a0 con 0x5572575f8dd0 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557257562cc0 con 0x557257620280 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x557257560450 con 0x5572575f6590 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59250/0 (socket says 172.21.15.28:59250) 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 -- 172.21.15.28:0/1598803674 learned_addr learned my addr 172.21.15.28:0/1598803674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11400640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f6590 0x5572576462a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e12800640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x5572576467e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 -- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f6590 msgr2=0x5572576462a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f6590 0x5572576462a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 -- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 msgr2=0x5572576467e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x5572576467e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e11e00640 1 -- 172.21.15.28:0/1598803674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55725752c2e0 con 0x5572575f8dd0 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.820+0000 7f0e12800640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x5572576467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.821+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000b920 tx=0x7f0e00004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beecf22cf06dd90a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.821+0000 7f0dfaa00640 1 -- 172.21.15.28:0/1598803674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00016c10 con 0x5572575f8dd0 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.821+0000 7f0dfaa00640 1 -- 172.21.15.28:0/1598803674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0001a070 con 0x5572575f8dd0 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.821+0000 7f0dfaa00640 1 -- 172.21.15.28:0/1598803674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00021440 con 0x5572575f8dd0 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- 172.21.15.28:0/1598803674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 msgr2=0x5572575f91b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f0e0000b920 tx=0x7f0e00004610 comp rx=0 tx=0).stop 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- 172.21.15.28:0/1598803674 shutdown_connections 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x5572576467e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f6590 0x5572576462a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 --2- 172.21.15.28:0/1598803674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x5572575f91b0 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- 172.21.15.28:0/1598803674 >> 172.21.15.28:0/1598803674 conn(0x5572575f5500 msgr2=0x55725763d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- 172.21.15.28:0/1598803674 shutdown_connections 2024-09-13T17:24:59.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- 172.21.15.28:0/1598803674 wait complete. 2024-09-13T17:24:59.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.822+0000 7f0e17633740 1 -- start start 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.823+0000 7f0e17633740 1 Processor -- start 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.823+0000 7f0e17633740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257623d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x5572576437b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725755bd40 con 0x5572575f8dd0 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572574fd280 con 0x557257620280 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572574fcf20 con 0x557257643330 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e17633740 10 client.0 _ll_register_callbacks cb 0x5572575553b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59262/0 (socket says 172.21.15.28:59262) 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 -- 172.21.15.28:0/1376318743 learned_addr learned my addr 172.21.15.28:0/1376318743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e12800640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x5572576437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11400640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257623d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 -- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 msgr2=0x5572576437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x5572576437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 -- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 msgr2=0x557257623d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257623d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11e00640 1 -- 172.21.15.28:0/1376318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0000e040 con 0x5572575f8dd0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.824+0000 7f0e11400640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257623d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e11e00640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f0e00008010 tx=0x7f0e00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24efeaf572bdf43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e00021a80 con 0x5572575f8dd0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0e00031070 con 0x5572575f8dd0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55725724f4e0 con 0x5572575f8dd0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e00021be0 con 0x5572575f8dd0 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 10 client.12678 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55725755a3c0 con 0x5572575f8dd0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0003d020 con 0x5572575f8dd0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0e00036480 con 0x5572575f8dd0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 10 client.12678 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 10 client.12678 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.825+0000 7f0e17633740 20 client.12678 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x557257621240 0x557257621650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0e17633740 4 client.12678 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5572572607e0 con 0x557257621240 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0e11400640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x557257621240 0x557257621650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0e11400640 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x557257621240 0x557257621650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0433ce33b9b393 server_cookie=f1a57bad882352a8 in_seq=0 out_seq=0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.826+0000 7f0def400640 10 client.12678 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0def400640 1 -- 172.21.15.28:0/1376318743 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x5572572607e0 con 0x557257621240 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0def400640 10 client.12678 handle_command_reply: tid=0 2024-09-13T17:24:59.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 client.12678 shutdown 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 msgr2=0x557257645770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f0e00008010 tx=0x7f0e00004730 comp rx=0 tx=0).stop 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 shutdown_connections 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x557257621240 0x557257621650 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x5572576437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257623d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 --2- 172.21.15.28:0/1376318743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572575f8dd0 0x557257645770 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0def400640 0 client.12678 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 >> 172.21.15.28:0/1376318743 conn(0x5572575f6590 msgr2=0x55725763f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 shutdown_connections 2024-09-13T17:24:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.827+0000 7f0e17633740 1 -- 172.21.15.28:0/1376318743 wait complete. 2024-09-13T17:24:59.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.828+0000 7f0e17633740 20 client.12678 trim_cache size 0 max 16384 2024-09-13T17:24:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.842+0000 7f0e17633740 1 Processor -- start 2024-09-13T17:24:59.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 -- start start 2024-09-13T17:24:59.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572575f8dd0 0x557257644d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x55725761c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572572c3040 con 0x557257620280 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725765f0c0 con 0x5572575f8dd0 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.843+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x5572576230c0 con 0x557257643330 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59272/0 (socket says 172.21.15.28:59272) 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 -- 172.21.15.28:0/3203987221 learned_addr learned my addr 172.21.15.28:0/3203987221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11e00640 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572575f8dd0 0x557257644d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e12800640 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x55725761c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 -- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 msgr2=0x55725761c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x55725761c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 -- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572575f8dd0 msgr2=0x557257644d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572575f8dd0 0x557257644d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 -- 172.21.15.28:0/3203987221 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572574fc610 con 0x557257620280 2024-09-13T17:24:59.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.844+0000 7f0e11400640 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000f60 tx=0x7f0dfc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74efdc9a8b727bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0dfb400640 1 -- 172.21.15.28:0/3203987221 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc0166e0 con 0x557257620280 2024-09-13T17:24:59.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0dfb400640 1 -- 172.21.15.28:0/3203987221 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dfc016840 con 0x557257620280 2024-09-13T17:24:59.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0dfb400640 1 -- 172.21.15.28:0/3203987221 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc016bb0 con 0x557257620280 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 -- 172.21.15.28:0/3203987221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 msgr2=0x55725763fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 secure :-1 s=READY pgs=2818 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000f60 tx=0x7f0dfc009040 comp rx=0 tx=0).stop 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 -- 172.21.15.28:0/3203987221 shutdown_connections 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x557257643330 0x55725761c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257620280 0x55725763fe60 unknown :-1 s=CLOSED pgs=2818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 --2- 172.21.15.28:0/3203987221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5572575f8dd0 0x557257644d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 -- 172.21.15.28:0/3203987221 >> 172.21.15.28:0/3203987221 conn(0x5572575f6590 msgr2=0x557257620660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 -- 172.21.15.28:0/3203987221 shutdown_connections 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.845+0000 7f0e17633740 1 -- 172.21.15.28:0/3203987221 wait complete. 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.846+0000 7f0e17633740 1 -- start start 2024-09-13T17:24:59.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.846+0000 7f0e17633740 1 Processor -- start 2024-09-13T17:24:59.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f8dd0 0x557257697bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257640e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725769ff90 con 0x557257643330 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557257698110 con 0x557257620280 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55725763f320 con 0x5572575f8dd0 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e17633740 10 client.0 _ll_register_callbacks cb 0x557257646110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.847+0000 7f0e12800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59278/0 (socket says 172.21.15.28:59278) 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 -- 172.21.15.28:0/4029583229 learned_addr learned my addr 172.21.15.28:0/4029583229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:24:59.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e11e00640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f8dd0 0x557257697bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e11400640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257640e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 -- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f8dd0 msgr2=0x557257697bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f8dd0 0x557257697bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 -- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 msgr2=0x557257640e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257640e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 -- 172.21.15.28:0/4029583229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c159e50 con 0x557257643330 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e12800640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f0e080a85c0 tx=0x7f0e08009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87463ca60b58d049 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0def400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e080786d0 con 0x557257643330 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0e08081320 con 0x557257643330 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.848+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e0001a070 con 0x557257643330 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e080a8250 con 0x557257643330 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 10 client.12684 fetch_fsmap learned FSMap version 195 2024-09-13T17:24:59.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x557257521b20 con 0x557257643330 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0001a070 con 0x557257643330 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0e08094070 con 0x557257643330 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 10 client.12684 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 10 client.12684 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 20 client.12684 populate_metadata read hostname 'smithi028' 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x5572576422b0 0x55725763e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 4 client.12684 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5572572b3e30 con 0x5572576422b0 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.849+0000 7f0e11e00640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x5572576422b0 0x55725763e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e11e00640 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x5572576422b0 0x55725763e470 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac266d9d6d3f82d server_cookie=c7edfc6312f4fb2f in_seq=0 out_seq=0 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0def400640 10 client.12684 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0def400640 1 -- 172.21.15.28:0/4029583229 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x5572572b3e30 con 0x5572576422b0 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0def400640 10 client.12684 handle_command_reply: tid=0 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 client.12684 shutdown 2024-09-13T17:24:59.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 msgr2=0x557257641360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f0e080a85c0 tx=0x7f0e08009510 comp rx=0 tx=0).stop 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 shutdown_connections 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x5572576422b0 0x55725763e470 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557257643330 0x557257641360 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557257620280 0x557257640e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 --2- 172.21.15.28:0/4029583229 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x5572575f8dd0 0x557257697bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0def400640 0 client.12684 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.850+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 >> 172.21.15.28:0/4029583229 conn(0x5572575f6590 msgr2=0x557257623e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 shutdown_connections 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e17633740 1 -- 172.21.15.28:0/4029583229 wait complete. 2024-09-13T17:24:59.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e17633740 20 client.12684 trim_cache size 0 max 16384 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "dbadb92f-b94f-4a59-90ad-9de809d28bb9": { 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "dbadb92f-b94f-4a59-90ad-9de809d28bb9", 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,repair" 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-09-13T17:24:59.856 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:24:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 msgr2=0x7f0df006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 0x7f0df006d0c0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f0e00006fd0 tx=0x7f0e00008040 comp rx=0 tx=0).stop 2024-09-13T17:24:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 msgr2=0x7f0e0c0ae1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:24:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f0e08034040 tx=0x7f0e08004a40 comp rx=0 tx=0).stop 2024-09-13T17:24:59.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 shutdown_connections 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0df006ac00 0x7f0df006d0c0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e0c0cde30 0x7f0e0c0ae1b0 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e0c071f60 0x7f0e0c0adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 --2- 172.21.15.28:0/2339644795 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e0c0712d0 0x7f0e0c13ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 >> 172.21.15.28:0/2339644795 conn(0x7f0e0c069e80 msgr2=0x7f0e0c06fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 shutdown_connections 2024-09-13T17:24:59.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:24:59.851+0000 7f0e13200640 1 -- 172.21.15.28:0/2339644795 wait complete. 2024-09-13T17:24:59.868 INFO:tasks.cephfs.filesystem:scrub status for tag:dbadb92f-b94f-4a59-90ad-9de809d28bb9 - {'path': '/', 'tag': 'dbadb92f-b94f-4a59-90ad-9de809d28bb9', 'options': 'recursive,repair'} 2024-09-13T17:25:12.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:25:12.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:25:29.869 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.16:0 scrub status 2024-09-13T17:25:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.036+0000 7fb795e00640 1 Processor -- start 2024-09-13T17:25:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.036+0000 7fb795e00640 1 -- start start 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb795e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb790159770 0x7fb79006fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb795e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb79006f3f0 0x7fb790070260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb795e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb79006f920 con 0x7fb7900add00 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb795e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb79015b290 con 0x7fb790159770 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb795e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb79015b570 con 0x7fb79006f3f0 2024-09-13T17:25:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb78ea00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fb790159770 0x7fb79006fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb78f400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fb7900add00 0x7fb790159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47262/0 (socket says 172.21.15.28:47262) 2024-09-13T17:25:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.037+0000 7fb78f400640 1 -- 172.21.15.28:0/2121252562 learned_addr learned my addr 172.21.15.28:0/2121252562 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.038+0000 7fb786000640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb79006f3f0 0x7fb790070260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.038+0000 7fb78ea00640 1 -- 172.21.15.28:0/2121252562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb79006f3f0 msgr2=0x7fb790070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.038+0000 7fb78ea00640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb79006f3f0 0x7fb790070260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.038+0000 7fb78ea00640 1 -- 172.21.15.28:0/2121252562 >> v1:172.21.15.28:6789/0 conn(0x7fb7900add00 legacy=0x7fb790159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.038+0000 7fb78ea00640 1 -- 172.21.15.28:0/2121252562 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb79015b850 con 0x7fb790159770 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.039+0000 7fb78ea00640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb790159770 0x7fb79006fd20 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7fb78000ba40 tx=0x7fb78002c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbf685617b8db08 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.039+0000 7fb78e000640 1 -- 172.21.15.28:0/2121252562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780031710 con 0x7fb790159770 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.039+0000 7fb78e000640 1 -- 172.21.15.28:0/2121252562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb78002f050 con 0x7fb790159770 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.039+0000 7fb78e000640 1 -- 172.21.15.28:0/2121252562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780031a90 con 0x7fb790159770 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 -- 172.21.15.28:0/2121252562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb790159770 msgr2=0x7fb79006fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb790159770 0x7fb79006fd20 secure :-1 s=READY pgs=2821 cs=0 l=1 rev1=1 crypto rx=0x7fb78000ba40 tx=0x7fb78002c7b0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 -- 172.21.15.28:0/2121252562 shutdown_connections 2024-09-13T17:25:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb79006f3f0 0x7fb790070260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 --2- 172.21.15.28:0/2121252562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb790159770 0x7fb79006fd20 unknown :-1 s=CLOSED pgs=2821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 -- 172.21.15.28:0/2121252562 >> 172.21.15.28:0/2121252562 conn(0x7fb7900b0270 msgr2=0x7fb7900b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 -- 172.21.15.28:0/2121252562 shutdown_connections 2024-09-13T17:25:30.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.040+0000 7fb795e00640 1 -- 172.21.15.28:0/2121252562 wait complete. 2024-09-13T17:25:30.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.042+0000 7fb795e00640 1 Processor -- start 2024-09-13T17:25:30.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.042+0000 7fb795e00640 1 -- start start 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7900add00 0x7fb7900abaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb790159770 0x7fb7900ac030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900aa2a0 con 0x7fb79006f3f0 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900ac570 con 0x7fb790159770 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb795e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7900a9620 con 0x7fb7900add00 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45604/0 (socket says 172.21.15.28:45604) 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 -- 172.21.15.28:0/644747041 learned_addr learned my addr 172.21.15.28:0/644747041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78f400640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb790159770 0x7fb7900ac030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb786000640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7900add00 0x7fb7900abaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 -- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7900add00 msgr2=0x7fb7900abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7900add00 0x7fb7900abaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 -- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb790159770 msgr2=0x7fb7900ac030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb790159770 0x7fb7900ac030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.043+0000 7fb78ea00640 1 -- 172.21.15.28:0/644747041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78000b6a0 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb78ea00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fb78000bb60 tx=0x7fb7800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb787e00640 1 -- 172.21.15.28:0/644747041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780033990 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb787e00640 1 -- 172.21.15.28:0/644747041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb780033af0 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb787e00640 1 -- 172.21.15.28:0/644747041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb780033ea0 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7900a9900 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.044+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7900a9ec0 con 0x7fb79006f3f0 2024-09-13T17:25:30.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.047+0000 7fb787e00640 1 -- 172.21.15.28:0/644747041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fb78003e420 con 0x7fb79006f3f0 2024-09-13T17:25:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.048+0000 7fb787e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 0x7fb76c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.048+0000 7fb787e00640 1 -- 172.21.15.28:0/644747041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb7800acdb0 con 0x7fb79006f3f0 2024-09-13T17:25:30.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.048+0000 7fb786000640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 0x7fb76c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.048+0000 7fb798c9d740 1 Processor -- start 2024-09-13T17:25:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.049+0000 7fb798c9d740 1 -- start start 2024-09-13T17:25:30.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.049+0000 7fb786000640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 0x7fb76c06d0c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fb7900cad70 tx=0x7fb778008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x55698ffea450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x556990037a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698feec6a0 con 0x55698ffe7830 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698ff4fcc0 con 0x55698ffea070 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55698ff4d450 con 0x556990011520 2024-09-13T17:25:30.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45606/0 (socket says 172.21.15.28:45606) 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 -- 172.21.15.28:0/504837816 learned_addr learned my addr 172.21.15.28:0/504837816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb78ea00640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x55698ffea450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb78f400640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x556990037a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 -- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 msgr2=0x556990037a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x556990037a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 -- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 msgr2=0x55698ffea450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x55698ffea450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb786000640 1 -- 172.21.15.28:0/504837816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55698ff192e0 con 0x55698ffe7830 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.050+0000 7fb78f400640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x556990037a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb786000640 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fb7780074a0 tx=0x7fb778013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fe94e637c8ff26c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb77fe00640 1 -- 172.21.15.28:0/504837816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778016c10 con 0x55698ffe7830 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb77fe00640 1 -- 172.21.15.28:0/504837816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77801a070 con 0x55698ffe7830 2024-09-13T17:25:30.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb77fe00640 1 -- 172.21.15.28:0/504837816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778021440 con 0x55698ffe7830 2024-09-13T17:25:30.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb798c9d740 1 -- 172.21.15.28:0/504837816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 msgr2=0x556990037540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.051+0000 7fb798c9d740 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fb7780074a0 tx=0x7fb778013000 comp rx=0 tx=0).stop 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 -- 172.21.15.28:0/504837816 shutdown_connections 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x556990037a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55698ffe7830 0x556990037540 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 --2- 172.21.15.28:0/504837816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x55698ffea450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 -- 172.21.15.28:0/504837816 >> 172.21.15.28:0/504837816 conn(0x55698ffe67a0 msgr2=0x55699002ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 -- 172.21.15.28:0/504837816 shutdown_connections 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 -- 172.21.15.28:0/504837816 wait complete. 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.052+0000 7fb798c9d740 1 -- start start 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 Processor -- start 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990006c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x556990009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698ff48d40 con 0x556990011520 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698feea280 con 0x556990008520 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.053+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55698fee9f20 con 0x55698ffea070 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb798c9d740 10 client.0 _ll_register_callbacks cb 0x55698ff423b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45620/0 (socket says 172.21.15.28:45620) 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 -- 172.21.15.28:0/174558139 learned_addr learned my addr 172.21.15.28:0/174558139 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb78f400640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x556990009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb78ea00640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990006c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 -- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 msgr2=0x556990006c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990006c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 -- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 msgr2=0x556990009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x556990009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.054+0000 7fb786000640 1 -- 172.21.15.28:0/174558139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77800e040 con 0x556990011520 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb786000640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fb778007450 tx=0x7fb778004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14f5b487ca681e77 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb77cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778021b50 con 0x556990011520 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb778021cb0 con 0x556990011520 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55698fc3c4e0 con 0x556990011520 2024-09-13T17:25:30.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb778021e70 con 0x556990011520 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb798c9d740 10 client.12699 fetch_fsmap learned FSMap version 195 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55698ff473c0 con 0x556990011520 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.055+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55698fc3c4e0 con 0x556990011520 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb778035470 con 0x556990011520 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 10 client.12699 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 10 client.12699 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 20 client.12699 populate_metadata read hostname 'smithi028' 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55699000cd30 0x55699002f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 4 client.12699 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55698fc4d7e0 con 0x55699000cd30 2024-09-13T17:25:30.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb78ea00640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55699000cd30 0x55699002f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb78ea00640 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55699000cd30 0x55699002f740 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba238847fd39594 server_cookie=40b49aa19f56a027 in_seq=0 out_seq=0 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.056+0000 7fb77cc00640 10 client.12699 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb77cc00640 1 -- 172.21.15.28:0/174558139 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55698fc4d7e0 con 0x55699000cd30 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb77cc00640 10 client.12699 handle_command_reply: tid=0 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 client.12699 shutdown 2024-09-13T17:25:30.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 msgr2=0x5569900092e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fb778007450 tx=0x7fb778004730 comp rx=0 tx=0).stop 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 shutdown_connections 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55699000cd30 0x55699002f740 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x556990009820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x5569900092e0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb77cc00640 0 client.12699 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 --2- 172.21.15.28:0/174558139 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990006c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.057+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 >> 172.21.15.28:0/174558139 conn(0x55698ffe7830 msgr2=0x55698ffe4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.058+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 shutdown_connections 2024-09-13T17:25:30.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.058+0000 7fb798c9d740 1 -- 172.21.15.28:0/174558139 wait complete. 2024-09-13T17:25:30.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.058+0000 7fb798c9d740 20 client.12699 trim_cache size 0 max 16384 2024-09-13T17:25:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.073+0000 7fb798c9d740 1 Processor -- start 2024-09-13T17:25:30.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.073+0000 7fb798c9d740 1 -- start start 2024-09-13T17:25:30.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.073+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x556990006870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.073+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.073+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x55698ffe5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb798c9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698fcb0040 con 0x556990008520 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699002c8b0 con 0x55698ffea070 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x556990031520 con 0x556990011520 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45624/0 (socket says 172.21.15.28:45624) 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 -- 172.21.15.28:0/2373517211 learned_addr learned my addr 172.21.15.28:0/2373517211 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb78f400640 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x55698ffe5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb78ea00640 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x556990006870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 -- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 msgr2=0x55698ffe5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x55698ffe5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 -- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 msgr2=0x556990006870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x556990006870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.074+0000 7fb786000640 1 -- 172.21.15.28:0/2373517211 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55698fee9610 con 0x556990008520 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb786000640 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7fb778007fd0 tx=0x7fb77802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897b267542495562 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb784c00640 1 -- 172.21.15.28:0/2373517211 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778035930 con 0x556990008520 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb784c00640 1 -- 172.21.15.28:0/2373517211 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb778016850 con 0x556990008520 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb784c00640 1 -- 172.21.15.28:0/2373517211 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb778035b10 con 0x556990008520 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 -- 172.21.15.28:0/2373517211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 msgr2=0x556990006db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 secure :-1 s=READY pgs=2825 cs=0 l=1 rev1=1 crypto rx=0x7fb778007fd0 tx=0x7fb77802e070 comp rx=0 tx=0).stop 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 -- 172.21.15.28:0/2373517211 shutdown_connections 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x556990011520 0x55698ffe5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990008520 0x556990006db0 unknown :-1 s=CLOSED pgs=2825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 --2- 172.21.15.28:0/2373517211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55698ffea070 0x556990006870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.075+0000 7fb798c9d740 1 -- 172.21.15.28:0/2373517211 >> 172.21.15.28:0/2373517211 conn(0x55698ffe7830 msgr2=0x55699000b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.076+0000 7fb798c9d740 1 -- 172.21.15.28:0/2373517211 shutdown_connections 2024-09-13T17:25:30.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.076+0000 7fb798c9d740 1 -- 172.21.15.28:0/2373517211 wait complete. 2024-09-13T17:25:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.076+0000 7fb798c9d740 1 -- start start 2024-09-13T17:25:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.076+0000 7fb798c9d740 1 Processor -- start 2024-09-13T17:25:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990097fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x55699009baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55698ffe55d0 con 0x556990011520 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55699009c030 con 0x556990008520 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55699009c1d0 con 0x55698ffea070 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.077+0000 7fb798c9d740 10 client.0 _ll_register_callbacks cb 0x7fb778021440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78ea00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990097fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45638/0 (socket says 172.21.15.28:45638) 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 -- 172.21.15.28:0/1854194308 learned_addr learned my addr 172.21.15.28:0/1854194308 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb786000640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x55699009baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 -- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 msgr2=0x556990097fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990097fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 -- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 msgr2=0x55699009baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x55699009baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 -- 172.21.15.28:0/1854194308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55698ff473c0 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb786000640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x55699009baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb78f400640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7fb788016c00 tx=0x7fb788016cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f45f97ad8fdd38 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.078+0000 7fb77cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788016ee0 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb788007780 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb77801a070 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb788007940 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 10 client.12705 fetch_fsmap learned FSMap version 195 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55698ff0eb20 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb77801a070 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fb788017430 con 0x556990011520 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 10 client.12705 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:25:30.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 10 client.12705 resolve_mds: resolved 16:0 to role '16:0' aka daemon mds.a 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 20 client.12705 populate_metadata read hostname 'smithi028' 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x556990013aa0 0x55699002dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 4 client.12705 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.079+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55698fca0e30 con 0x556990013aa0 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb78ea00640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x556990013aa0 0x55699002dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb78ea00640 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x556990013aa0 0x55699002dd50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b6f14ff05db4ef9 server_cookie=bcc1bc223348d848 in_seq=0 out_seq=0 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb77cc00640 10 client.12705 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb77cc00640 1 -- 172.21.15.28:0/1854194308 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55698fca0e30 con 0x556990013aa0 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb77cc00640 10 client.12705 handle_command_reply: tid=0 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb798c9d740 1 client.12705 shutdown 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 msgr2=0x556990040630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7fb788016c00 tx=0x7fb788016cf0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 shutdown_connections 2024-09-13T17:25:30.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.080+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x556990013aa0 0x55699002dd50 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556990011520 0x556990040630 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556990008520 0x55699009baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 --2- 172.21.15.28:0/1854194308 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55698ffea070 0x556990097fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb77cc00640 0 client.12705 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 >> 172.21.15.28:0/1854194308 conn(0x55698ffe7830 msgr2=0x5569900510e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 shutdown_connections 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 1 -- 172.21.15.28:0/1854194308 wait complete. 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb798c9d740 20 client.12705 trim_cache size 0 max 16384 2024-09-13T17:25:30.088 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-09-13T17:25:30.089 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-09-13T17:25:30.089 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:30.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 msgr2=0x7fb76c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 0x7fb76c06d0c0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fb7900cad70 tx=0x7fb778008040 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 msgr2=0x7fb790157bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fb78000bb60 tx=0x7fb7800046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 shutdown_connections 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fb76c06ac00 0x7fb76c06d0c0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb790159770 0x7fb7900ac030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fb7900add00 0x7fb7900abaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 --2- 172.21.15.28:0/644747041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb79006f3f0 0x7fb790157bf0 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.081+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 >> 172.21.15.28:0/644747041 conn(0x7fb7900b0270 msgr2=0x7fb79015a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.082+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 shutdown_connections 2024-09-13T17:25:30.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.082+0000 7fb795e00640 1 -- 172.21.15.28:0/644747041 wait complete. 2024-09-13T17:25:30.099 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T17:25:30.502 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:25:30.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.661+0000 7f597aa00640 1 Processor -- start 2024-09-13T17:25:30.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.662+0000 7f597aa00640 1 -- start start 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f597aa00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5974159440 0x7f59740b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f597aa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f59740b29f0 0x7f59740b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f597aa00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59740c3850 con 0x7f5974159820 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f597aa00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f597415a930 con 0x7f5974159440 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f597aa00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5974159d20 con 0x7f59740b29f0 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f5979600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f5974159440 0x7f59740b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f5978c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f5974159820 0x7f59740b22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47270/0 (socket says 172.21.15.28:47270) 2024-09-13T17:25:30.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f5978c00640 1 -- 172.21.15.28:0/3894651019 learned_addr learned my addr 172.21.15.28:0/3894651019 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:30.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.663+0000 7f5979600640 1 --2- 172.21.15.28:0/3894651019 >> v2:172.21.15.28:3300/0 conn(0x7f5974159440 0x7f59740b0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45644/0 (socket says 172.21.15.28:45644) 2024-09-13T17:25:30.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f597a000640 1 --2- 172.21.15.28:0/3894651019 >> v2:172.21.15.189:3300/0 conn(0x7f59740b29f0 0x7f59740b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f5979600640 1 -- 172.21.15.28:0/3894651019 >> v2:172.21.15.189:3300/0 conn(0x7f59740b29f0 msgr2=0x7f59740b0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:30.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f5979600640 1 --2- 172.21.15.28:0/3894651019 >> v2:172.21.15.189:3300/0 conn(0x7f59740b29f0 0x7f59740b0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f5979600640 1 -- 172.21.15.28:0/3894651019 >> v1:172.21.15.28:6789/0 conn(0x7f5974159820 legacy=0x7f59740b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f5979600640 1 -- 172.21.15.28:0/3894651019 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597415a000 con 0x7f5974159440 2024-09-13T17:25:30.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.664+0000 7f5973e00640 1 -- 172.21.15.28:0/3894651019 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441664405 0 0) 0x7f59740c3850 con 0x7f5974159820 2024-09-13T17:25:30.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f5979600640 1 --2- 172.21.15.28:0/3894651019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159440 0x7f59740b0440 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f596802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2680c73507b1c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f5973e00640 1 -- 172.21.15.28:0/3894651019 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968032440 con 0x7f5974159440 2024-09-13T17:25:30.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f5973e00640 1 -- 172.21.15.28:0/3894651019 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59680325a0 con 0x7f5974159440 2024-09-13T17:25:30.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f5973e00640 1 -- 172.21.15.28:0/3894651019 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968032930 con 0x7f5974159440 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f597aa00640 1 -- 172.21.15.28:0/3894651019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159440 msgr2=0x7f59740b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.665+0000 7f597aa00640 1 --2- 172.21.15.28:0/3894651019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159440 0x7f59740b0440 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f596802a270 comp rx=0 tx=0).stop 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 -- 172.21.15.28:0/3894651019 shutdown_connections 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 --2- 172.21.15.28:0/3894651019 >> v2:172.21.15.189:3300/0 conn(0x7f59740b29f0 0x7f59740b0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 --2- 172.21.15.28:0/3894651019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159440 0x7f59740b0440 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 -- 172.21.15.28:0/3894651019 >> 172.21.15.28:0/3894651019 conn(0x7f5974072e40 msgr2=0x7f5974073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 -- 172.21.15.28:0/3894651019 shutdown_connections 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.666+0000 7f597aa00640 1 -- 172.21.15.28:0/3894651019 wait complete. 2024-09-13T17:25:30.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.667+0000 7f597aa00640 1 Processor -- start 2024-09-13T17:25:30.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 -- start start 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59740b29f0 0x7f59741f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5974159440 0x7f59741f57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974155d20 con 0x7f5974159820 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5974158550 con 0x7f59740b29f0 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.668+0000 7f597aa00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59741f2f10 con 0x7f5974159440 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f5979600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59740b29f0 0x7f59741f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45650/0 (socket says 172.21.15.28:45650) 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 -- 172.21.15.28:0/2059489047 learned_addr learned my addr 172.21.15.28:0/2059489047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f5978c00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5974159440 0x7f59741f57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 -- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5974159440 msgr2=0x7f59741f57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5974159440 0x7f59741f57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 -- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59740b29f0 msgr2=0x7f59741f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59740b29f0 0x7f59741f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.669+0000 7f597a000640 1 -- 172.21.15.28:0/2059489047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5968009400 con 0x7f5974159820 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f597a000640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f596c007d90 tx=0x7f596c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c014420 con 0x7f5974159820 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596c0145a0 con 0x7f5974159820 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5974155960 con 0x7f5974159820 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596c014950 con 0x7f5974159820 2024-09-13T17:25:30.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.670+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5974158a80 con 0x7f5974159820 2024-09-13T17:25:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.671+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5974155e60 con 0x7f5974159820 2024-09-13T17:25:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.672+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f596c014ab0 con 0x7f5974159820 2024-09-13T17:25:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.673+0000 7f5971600640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 0x7f595406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:30.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.673+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f596c089b80 con 0x7f5974159820 2024-09-13T17:25:30.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.673+0000 7f5979600640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 0x7f595406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:30.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.674+0000 7f5979600640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 0x7f595406d0c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f5968034000 tx=0x7f596802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:30.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.677+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5974155e60 con 0x7f5974159820 2024-09-13T17:25:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.891+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59740b22e0 con 0x7f5974159820 2024-09-13T17:25:30.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.891+0000 7f5971600640 1 -- 172.21.15.28:0/2059489047 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f596c089e60 con 0x7f5974159820 2024-09-13T17:25:30.894 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:30.894 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:25:30.894 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 msgr2=0x7f595406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 0x7f595406d0c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f5968034000 tx=0x7f596802e000 comp rx=0 tx=0).stop 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 msgr2=0x7f59741f1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f596c007d90 tx=0x7f596c00e4b0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 shutdown_connections 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f595406ac00 0x7f595406d0c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5974159820 0x7f59741f1020 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5974159440 0x7f59741f57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 --2- 172.21.15.28:0/2059489047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59740b29f0 0x7f59741f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 >> 172.21.15.28:0/2059489047 conn(0x7f5974072e40 msgr2=0x7f597415cff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 shutdown_connections 2024-09-13T17:25:30.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:30.894+0000 7f597aa00640 1 -- 172.21.15.28:0/2059489047 wait complete. 2024-09-13T17:25:30.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 195, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T17:24:10.577893+0000', 'modified': '2024-09-13T17:24:27.074178+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 234, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12516}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12516': {'gid': 12516, 'name': 'a', 'rank': 0, 'incarnation': 192, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.28:6835/994067792', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6833', 'nonce': 994067792}, {'type': 'v1', 'addr': '172.21.15.28:6835', 'nonce': 994067792}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T17:25:30.912 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T17:25:30.912 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T17:25:31.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.067+0000 7ff518800640 1 Processor -- start 2024-09-13T17:25:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.067+0000 7ff518800640 1 -- start start 2024-09-13T17:25:31.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff518800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff510070430 0x7ff51015bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff518800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 0x7ff51015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff518800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51015ab60 con 0x7ff510159770 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff518800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51006f920 con 0x7ff510070430 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff518800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff51015b290 con 0x7ff51006f3f0 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff515e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff510159770 0x7ff51006fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47276/0 (socket says 172.21.15.28:47276) 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.068+0000 7ff515e00640 1 -- 172.21.15.28:0/4067999384 learned_addr learned my addr 172.21.15.28:0/4067999384 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff516800640 1 --2- 172.21.15.28:0/4067999384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 0x7ff51015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff515400640 1 --2- 172.21.15.28:0/4067999384 >> v2:172.21.15.189:3300/0 conn(0x7ff510070430 0x7ff51015bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526870668 0 0) 0x7ff504001110 con 0x7ff510159770 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4f8003520 con 0x7ff510159770 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff516800640 1 -- 172.21.15.28:0/4067999384 >> v2:172.21.15.189:3300/0 conn(0x7ff510070430 msgr2=0x7ff51015bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff516800640 1 --2- 172.21.15.28:0/4067999384 >> v2:172.21.15.189:3300/0 conn(0x7ff510070430 0x7ff51015bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1706029198 0 0) 0x7ff4f8003520 con 0x7ff510159770 2024-09-13T17:25:31.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff516800640 1 -- 172.21.15.28:0/4067999384 >> v1:172.21.15.28:6789/0 conn(0x7ff510159770 legacy=0x7ff51006fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.069+0000 7ff516800640 1 -- 172.21.15.28:0/4067999384 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51015b570 con 0x7ff51006f3f0 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.070+0000 7ff516800640 1 --2- 172.21.15.28:0/4067999384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 0x7ff51015c2b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009d90 tx=0x7ff50c02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53b9abf9b6ae00bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.070+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504001800 con 0x7ff51006f3f0 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50c02d050 con 0x7ff51006f3f0 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff514a00640 1 -- 172.21.15.28:0/4067999384 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50c007740 con 0x7ff51006f3f0 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 -- 172.21.15.28:0/4067999384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 msgr2=0x7ff51015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 --2- 172.21.15.28:0/4067999384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 0x7ff51015c2b0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009d90 tx=0x7ff50c02a560 comp rx=0 tx=0).stop 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 -- 172.21.15.28:0/4067999384 shutdown_connections 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 --2- 172.21.15.28:0/4067999384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff51006f3f0 0x7ff51015c2b0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 --2- 172.21.15.28:0/4067999384 >> v2:172.21.15.189:3300/0 conn(0x7ff510070430 0x7ff51015bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 -- 172.21.15.28:0/4067999384 >> 172.21.15.28:0/4067999384 conn(0x7ff5100b0270 msgr2=0x7ff5100b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 -- 172.21.15.28:0/4067999384 shutdown_connections 2024-09-13T17:25:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.071+0000 7ff518800640 1 -- 172.21.15.28:0/4067999384 wait complete. 2024-09-13T17:25:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.073+0000 7ff518800640 1 Processor -- start 2024-09-13T17:25:31.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.073+0000 7ff518800640 1 -- start start 2024-09-13T17:25:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 0x7ff5101575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 0x7ff5100cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510157ae0 con 0x7ff510070430 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510151cf0 con 0x7ff51006f3f0 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff518800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5101ee010 con 0x7ff510159770 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff515400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff515400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45664/0 (socket says 172.21.15.28:45664) 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.074+0000 7ff515400640 1 -- 172.21.15.28:0/4133601407 learned_addr learned my addr 172.21.15.28:0/4133601407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff516800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 0x7ff5100cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 -- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 msgr2=0x7ff5100cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515e00640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 0x7ff5101575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 0x7ff5100cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 -- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 msgr2=0x7ff5101575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 0x7ff5101575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 -- 172.21.15.28:0/4133601407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c009a70 con 0x7ff510070430 2024-09-13T17:25:31.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff516800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 0x7ff5100cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515e00640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 0x7ff5101575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff515400640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7ff50000e9b0 tx=0x7ff50000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff50a000640 1 -- 172.21.15.28:0/4133601407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500009670 con 0x7ff510070430 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff50a000640 1 -- 172.21.15.28:0/4133601407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50000c9a0 con 0x7ff510070430 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff50a000640 1 -- 172.21.15.28:0/4133601407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50000cd50 con 0x7ff510070430 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5101ee2f0 con 0x7ff510070430 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.075+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5101ee8b0 con 0x7ff510070430 2024-09-13T17:25:31.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.077+0000 7ff50a000640 1 -- 172.21.15.28:0/4133601407 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff500013470 con 0x7ff510070430 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.078+0000 7ff50a000640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 0x7ff4f406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.078+0000 7ff50a000640 1 -- 172.21.15.28:0/4133601407 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff500089660 con 0x7ff510070430 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.079+0000 7ff51b69d740 1 Processor -- start 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.079+0000 7ff515e00640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 0x7ff4f406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.079+0000 7ff51b69d740 1 -- start start 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 0x7ff4f406d0f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff504001930 tx=0x7ff504009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2315a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23075ab0 con 0x55ce2310c6d0 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23073240 con 0x55ce23109e90 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce2306eb30 con 0x55ce23133b80 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45678/0 (socket says 172.21.15.28:45678) 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 -- 172.21.15.28:0/3307954648 learned_addr learned my addr 172.21.15.28:0/3307954648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff516800640 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2315a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515400640 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 -- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 msgr2=0x55ce2315a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2315a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 -- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 msgr2=0x55ce23159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23159ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.080+0000 7ff515e00640 1 -- 172.21.15.28:0/3307954648 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce23039bd0 con 0x55ce2310c6d0 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.081+0000 7ff515e00640 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7ff504005bd0 tx=0x7ff504010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3b0a11f33b93a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.081+0000 7ff4fea00640 1 -- 172.21.15.28:0/3307954648 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50400dbb0 con 0x55ce2310c6d0 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.081+0000 7ff4fea00640 1 -- 172.21.15.28:0/3307954648 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400dd10 con 0x55ce2310c6d0 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.081+0000 7ff4fea00640 1 -- 172.21.15.28:0/3307954648 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504020650 con 0x55ce2310c6d0 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- 172.21.15.28:0/3307954648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 msgr2=0x55ce2310cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 secure :-1 s=READY pgs=2832 cs=0 l=1 rev1=1 crypto rx=0x7ff504005bd0 tx=0x7ff504010000 comp rx=0 tx=0).stop 2024-09-13T17:25:31.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- 172.21.15.28:0/3307954648 shutdown_connections 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2315a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 --2- 172.21.15.28:0/3307954648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce2310cab0 unknown :-1 s=CLOSED pgs=2832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- 172.21.15.28:0/3307954648 >> 172.21.15.28:0/3307954648 conn(0x55ce23108e00 msgr2=0x55ce231512b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- 172.21.15.28:0/3307954648 shutdown_connections 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- 172.21.15.28:0/3307954648 wait complete. 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.082+0000 7ff51b69d740 1 -- start start 2024-09-13T17:25:31.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.083+0000 7ff51b69d740 1 Processor -- start 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.083+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.083+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 0x55ce23132970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 0x55ce23158ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff51b69d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23010260 con 0x55ce2310c6d0 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce2300ff00 con 0x55ce23133b80 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce22dd5f20 con 0x55ce231566f0 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff51b69d740 10 client.0 _ll_register_callbacks cb 0x55ce230681a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:25:31.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45688/0 (socket says 172.21.15.28:45688) 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 -- 172.21.15.28:0/209137180 learned_addr learned my addr 172.21.15.28:0/209137180 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515400640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 0x55ce23132970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff516800640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 0x55ce23158ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 -- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 msgr2=0x55ce23158ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 0x55ce23158ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 -- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 msgr2=0x55ce23132970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 0x55ce23132970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515e00640 1 -- 172.21.15.28:0/209137180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff504005910 con 0x55ce2310c6d0 2024-09-13T17:25:31.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff516800640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 0x55ce23158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.084+0000 7ff515400640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 0x55ce23132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff515e00640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7ff504005880 tx=0x7ff504007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55ea48c6b8f83d40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff4f3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504027040 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ce23098cd0 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff50400de80 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff504020c90 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff51b69d740 10 client.12726 fetch_fsmap learned FSMap version 195 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55ce2306d1b0 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.085+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ce23098cd0 con 0x55ce2310c6d0 2024-09-13T17:25:31.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff504012410 con 0x55ce2310c6d0 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 10 client.12726 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 10 client.12726 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 20 client.12726 populate_metadata read hostname 'smithi028' 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce2315a150 0x55ce23135050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 4 client.12726 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce22d623c0 con 0x55ce2315a150 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff515400640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce2315a150 0x55ce23135050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff515400640 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce2315a150 0x55ce23135050 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bfffc3d3613d7a8 server_cookie=a8c8039aa3abd47a in_seq=0 out_seq=0 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.086+0000 7ff4f3400640 10 client.12726 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff4f3400640 1 -- 172.21.15.28:0/209137180 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19071 (crc 0 0 0) 0x55ce22d623c0 con 0x55ce2315a150 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff4f3400640 10 client.12726 handle_command_reply: tid=0 2024-09-13T17:25:31.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 client.12726 shutdown 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 msgr2=0x55ce23157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7ff504005880 tx=0x7ff504007000 comp rx=0 tx=0).stop 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 shutdown_connections 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce2315a150 0x55ce23135050 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce231566f0 0x55ce23158ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23133b80 0x55ce23132970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 --2- 172.21.15.28:0/209137180 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23157e90 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff4f3400640 0 client.12726 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.087+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 >> 172.21.15.28:0/209137180 conn(0x55ce23109e90 msgr2=0x55ce23154fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.088+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 shutdown_connections 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.088+0000 7ff51b69d740 1 -- 172.21.15.28:0/209137180 wait complete. 2024-09-13T17:25:31.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.088+0000 7ff51b69d740 20 client.12726 trim_cache size 0 max 16384 2024-09-13T17:25:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.103+0000 7ff51b69d740 1 Processor -- start 2024-09-13T17:25:31.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.103+0000 7ff51b69d740 1 -- start start 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce2310c6d0 0x55ce23132500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2314f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23169fe0 con 0x55ce23109e90 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23158c40 con 0x55ce2310c6d0 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce22ff57c0 con 0x55ce23133b80 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff515e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45702/0 (socket says 172.21.15.28:45702) 2024-09-13T17:25:31.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.104+0000 7ff515e00640 1 -- 172.21.15.28:0/1499772848 learned_addr learned my addr 172.21.15.28:0/1499772848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 -- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 msgr2=0x55ce2314f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2314f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 -- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce2310c6d0 msgr2=0x55ce23132500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce2310c6d0 0x55ce23132500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 -- 172.21.15.28:0/1499772848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce230a2170 con 0x55ce23109e90 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff515e00640 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7ff50400dce0 tx=0x7ff504011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a489e0c39de1b8fa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff4ff400640 1 -- 172.21.15.28:0/1499772848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504027070 con 0x55ce23109e90 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff4ff400640 1 -- 172.21.15.28:0/1499772848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504018070 con 0x55ce23109e90 2024-09-13T17:25:31.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.105+0000 7ff4ff400640 1 -- 172.21.15.28:0/1499772848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504012920 con 0x55ce23109e90 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 -- 172.21.15.28:0/1499772848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 msgr2=0x55ce23158700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7ff50400dce0 tx=0x7ff504011000 comp rx=0 tx=0).stop 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 -- 172.21.15.28:0/1499772848 shutdown_connections 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce2314f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce2310c6d0 0x55ce23132500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 --2- 172.21.15.28:0/1499772848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce23109e90 0x55ce23158700 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 -- 172.21.15.28:0/1499772848 >> 172.21.15.28:0/1499772848 conn(0x55ce2310ce20 msgr2=0x55ce2316a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 -- 172.21.15.28:0/1499772848 shutdown_connections 2024-09-13T17:25:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.106+0000 7ff51b69d740 1 -- 172.21.15.28:0/1499772848 wait complete. 2024-09-13T17:25:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.107+0000 7ff51b69d740 1 -- start start 2024-09-13T17:25:31.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.107+0000 7ff51b69d740 1 Processor -- start 2024-09-13T17:25:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23162bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce23172a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23172120 con 0x55ce2310c6d0 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce23154990 con 0x55ce23109e90 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ce23170050 con 0x55ce23133b80 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff51b69d740 10 client.0 _ll_register_callbacks cb 0x7ff5000046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45708/0 (socket says 172.21.15.28:45708) 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 -- 172.21.15.28:0/2111475977 learned_addr learned my addr 172.21.15.28:0/2111475977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515e00640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23162bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff516800640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce23172a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 -- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 msgr2=0x55ce23172a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce23172a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 -- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 msgr2=0x55ce23162bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23162bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.108+0000 7ff515400640 1 -- 172.21.15.28:0/2111475977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce2306d1b0 con 0x55ce2310c6d0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff515400640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7ff5000580b0 tx=0x7ff50000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af9406077ffb35e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500054b10 con 0x55ce2310c6d0 2024-09-13T17:25:31.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff500054c70 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff50400dd10 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=195) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff500060360 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 10 client.12732 fetch_fsmap learned FSMap version 195 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=195,monmap=2+}) v3 -- 0x55ce2300f470 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50400dd10 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 195) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff500087020 con 0x55ce2310c6d0 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 10 client.12732 fetch_fsmap finished waiting for FSMap version 195 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 10 client.12732 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 20 client.12732 populate_metadata read hostname 'smithi028' 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.109+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce231566f0 0x55ce231505f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff51b69d740 4 client.12732 mds_command: new command op to 12516 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 --> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ce22d736c0 con 0x55ce231566f0 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff515e00640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce231566f0 0x55ce231505f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff515e00640 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce231566f0 0x55ce231505f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0dc97f5618a049b server_cookie=556a7471aa5d5d2b in_seq=0 out_seq=0 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff4f3400640 10 client.12732 ms_handle_connect on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff4f3400640 1 -- 172.21.15.28:0/2111475977 <== mds.0 v2:172.21.15.28:6833/994067792 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55ce22d736c0 con 0x55ce231566f0 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff4f3400640 10 client.12732 handle_command_reply: tid=0 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.110+0000 7ff51b69d740 1 client.12732 shutdown 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 msgr2=0x55ce23172520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7ff5000580b0 tx=0x7ff50000ceb0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 shutdown_connections 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:6833/994067792,v1:172.21.15.28:6835/994067792] conn(0x55ce231566f0 0x55ce231505f0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x55ce23133b80 0x55ce23172a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce2310c6d0 0x55ce23172520 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 --2- 172.21.15.28:0/2111475977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ce23109e90 0x55ce23162bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff4f3400640 0 client.12732 ms_handle_reset on v2:172.21.15.28:6833/994067792 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 >> 172.21.15.28:0/2111475977 conn(0x55ce2310ce20 msgr2=0x55ce2316a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 shutdown_connections 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 1 -- 172.21.15.28:0/2111475977 wait complete. 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff51b69d740 20 client.12732 trim_cache size 0 max 16384 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "99459b14-8688-430f-9c7e-baba0c3bf859", 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-09-13T17:25:31.118 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 12516, 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 67.046599835999999, 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 195, 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 234, 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 234, 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 69.000544602999994 2024-09-13T17:25:31.119 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.111+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 msgr2=0x7ff4f406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 0x7ff4f406d0f0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7ff504001930 tx=0x7ff504009040 comp rx=0 tx=0).stop 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 msgr2=0x7ff510151230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7ff50000e9b0 tx=0x7ff50000ee70 comp rx=0 tx=0).stop 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 shutdown_connections 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff4f406ac30 0x7ff4f406d0f0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff510159770 0x7ff5100cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff510070430 0x7ff510151230 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 --2- 172.21.15.28:0/4133601407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff51006f3f0 0x7ff5101575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 >> 172.21.15.28:0/4133601407 conn(0x7ff5100b0270 msgr2=0x7ff510070810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 shutdown_connections 2024-09-13T17:25:31.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.112+0000 7ff518800640 1 -- 172.21.15.28:0/4133601407 wait complete. 2024-09-13T17:25:31.129 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T17:25:31.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:25:31.224 DEBUG:teuthology.orchestra.run.smithi189:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T17:25:31.321 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T17:25:31.322 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:25:31.322 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:31.358 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:25:31.359 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:25:31.359 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:31.396 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:25:31.396 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:25:31.396 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:31.434 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:25:31.435 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:25:31.435 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:31.473 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:25:31.473 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:25:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.629+0000 7f2e23200640 1 Processor -- start 2024-09-13T17:25:31.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.629+0000 7f2e23200640 1 -- start start 2024-09-13T17:25:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.630+0000 7f2e23200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 0x7f2e240b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.630+0000 7f2e23200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e24159820 0x7f2e240b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.630+0000 7f2e23200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240c3850 con 0x7f2e240b22e0 2024-09-13T17:25:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.630+0000 7f2e23200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2415a930 con 0x7f2e24159820 2024-09-13T17:25:31.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.630+0000 7f2e23200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24159d20 con 0x7f2e24159440 2024-09-13T17:25:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e24159820 0x7f2e240b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e21e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 0x7f2e240b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f2e24159820 0x7f2e240b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:37118/0 (socket says 172.21.15.28:37118) 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 -- 172.21.15.28:0/501914994 learned_addr learned my addr 172.21.15.28:0/501914994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e21400640 1 -- 172.21.15.28:0/501914994 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547054379 0 0) 0x7f2e240c3850 con 0x7f2e240b22e0 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e21400640 1 -- 172.21.15.28:0/501914994 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e08005730 con 0x7f2e240b22e0 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 -- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 msgr2=0x7f2e240b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 0x7f2e240b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 -- 172.21.15.28:0/501914994 >> v1:172.21.15.189:6789/0 conn(0x7f2e240b22e0 legacy=0x7f2e240b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e19400640 1 -- 172.21.15.28:0/501914994 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2415a000 con 0x7f2e24159820 2024-09-13T17:25:31.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.631+0000 7f2e21e00640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 0x7f2e240b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.632+0000 7f2e19400640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159820 0x7f2e240b0980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009870 tx=0x7f2e0c02a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b61aa918498b4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.632+0000 7f2e21400640 1 -- 172.21.15.28:0/501914994 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c032640 con 0x7f2e24159820 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.632+0000 7f2e21400640 1 -- 172.21.15.28:0/501914994 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e0c0327a0 con 0x7f2e24159820 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.632+0000 7f2e21400640 1 -- 172.21.15.28:0/501914994 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c032b30 con 0x7f2e24159820 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 -- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159820 msgr2=0x7f2e240b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159820 0x7f2e240b0980 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009870 tx=0x7f2e0c02a300 comp rx=0 tx=0).stop 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 -- 172.21.15.28:0/501914994 shutdown_connections 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159820 0x7f2e240b0980 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 --2- 172.21.15.28:0/501914994 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159440 0x7f2e240b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 -- 172.21.15.28:0/501914994 >> 172.21.15.28:0/501914994 conn(0x7f2e24072e40 msgr2=0x7f2e24073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 -- 172.21.15.28:0/501914994 shutdown_connections 2024-09-13T17:25:31.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.633+0000 7f2e23200640 1 -- 172.21.15.28:0/501914994 wait complete. 2024-09-13T17:25:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.635+0000 7f2e23200640 1 Processor -- start 2024-09-13T17:25:31.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.635+0000 7f2e23200640 1 -- start start 2024-09-13T17:25:31.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159440 0x7f2e241edb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159820 0x7f2e240a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24150540 con 0x7f2e240b22e0 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24156870 con 0x7f2e24159440 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e23200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24156b50 con 0x7f2e24159820 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e21e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45710/0 (socket says 172.21.15.28:45710) 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e21e00640 1 -- 172.21.15.28:0/477291642 learned_addr learned my addr 172.21.15.28:0/477291642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e19400640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159440 0x7f2e241edb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.636+0000 7f2e22800640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159820 0x7f2e240a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 -- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159820 msgr2=0x7f2e240a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159820 0x7f2e240a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 -- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159440 msgr2=0x7f2e241edb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159440 0x7f2e241edb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 -- 172.21.15.28:0/477291642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14009400 con 0x7f2e240b22e0 2024-09-13T17:25:31.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e21e00640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400ebe0 tx=0x7f2e1400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14009760 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1400cc30 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400eee0 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c0094d0 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.637+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24157110 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.638+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240b2790 con 0x7f2e240b22e0 2024-09-13T17:25:31.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.639+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2e14013470 con 0x7f2e240b22e0 2024-09-13T17:25:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.640+0000 7f2e1aa00640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 0x7f2e0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.640+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2e140896e0 con 0x7f2e240b22e0 2024-09-13T17:25:31.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.640+0000 7f2e19400640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 0x7f2e0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:31.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.641+0000 7f2e19400640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 0x7f2e0006d0c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c034000 tx=0x7f2e0c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:31.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.645+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2e14054850 con 0x7f2e240b22e0 2024-09-13T17:25:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.857+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e24154240 con 0x7f2e240b22e0 2024-09-13T17:25:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.858+0000 7f2e1aa00640 1 -- 172.21.15.28:0/477291642 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7f2e14057e70 con 0x7f2e240b22e0 2024-09-13T17:25:31.860 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:31.860 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:25:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:25:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 msgr2=0x7f2e0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 0x7f2e0006d0c0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c034000 tx=0x7f2e0c02e000 comp rx=0 tx=0).stop 2024-09-13T17:25:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 msgr2=0x7f2e24150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400ebe0 tx=0x7f2e1400c520 comp rx=0 tx=0).stop 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 shutdown_connections 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2e0006ac00 0x7f2e0006d0c0 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2e24159820 0x7f2e240a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2e24159440 0x7f2e241edb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 --2- 172.21.15.28:0/477291642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2e240b22e0 0x7f2e24150000 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 >> 172.21.15.28:0/477291642 conn(0x7f2e24072e40 msgr2=0x7f2e24070b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 shutdown_connections 2024-09-13T17:25:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:31.860+0000 7f2e23200640 1 -- 172.21.15.28:0/477291642 wait complete. 2024-09-13T17:25:31.878 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:25:32.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.033+0000 7efe14400640 1 Processor -- start 2024-09-13T17:25:32.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.033+0000 7efe14400640 1 -- start start 2024-09-13T17:25:32.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.033+0000 7efe14400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7efe0c06fb50 0x7efe0c163ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.033+0000 7efe14400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c0b3970 con 0x7efe0c15d340 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe14400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe0c159430 con 0x7efe0c0cdfa0 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe14400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe0c159230 con 0x7efe0c06fb50 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe11000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7efe0c15d340 0x7efe0c06f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47282/0 (socket says 172.21.15.28:47282) 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe11000640 1 -- 172.21.15.28:0/3924748541 learned_addr learned my addr 172.21.15.28:0/3924748541 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:32.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe12400640 1 --2- 172.21.15.28:0/3924748541 >> v2:172.21.15.189:3300/0 conn(0x7efe0c06fb50 0x7efe0c163ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533623954 0 0) 0x7efdf8001110 con 0x7efe0c15d340 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.034+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdf4003520 con 0x7efe0c15d340 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938567426 0 0) 0x7efe0c159430 con 0x7efe0c0cdfa0 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdf8001110 con 0x7efe0c0cdfa0 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe12400640 1 -- 172.21.15.28:0/3924748541 >> v1:172.21.15.189:6789/0 conn(0x7efe0c0cdfa0 legacy=0x7efe0c0b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe12400640 1 -- 172.21.15.28:0/3924748541 >> v1:172.21.15.28:6789/0 conn(0x7efe0c15d340 legacy=0x7efe0c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469373836 0 0) 0x7efdf4003520 con 0x7efe0c15d340 2024-09-13T17:25:32.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.035+0000 7efe12400640 1 -- 172.21.15.28:0/3924748541 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe0c159710 con 0x7efe0c06fb50 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.036+0000 7efe12400640 1 --2- 172.21.15.28:0/3924748541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c06fb50 0x7efe0c163ba0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7efe04009d90 tx=0x7efe0402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2fbdef07fe63764 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.036+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8001800 con 0x7efe0c06fb50 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.036+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0402d050 con 0x7efe0c06fb50 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.036+0000 7efe0be00640 1 -- 172.21.15.28:0/3924748541 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04007760 con 0x7efe0c06fb50 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 -- 172.21.15.28:0/3924748541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c06fb50 msgr2=0x7efe0c163ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 --2- 172.21.15.28:0/3924748541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c06fb50 0x7efe0c163ba0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7efe04009d90 tx=0x7efe0402a680 comp rx=0 tx=0).stop 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 -- 172.21.15.28:0/3924748541 shutdown_connections 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 --2- 172.21.15.28:0/3924748541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c06fb50 0x7efe0c163ba0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 -- 172.21.15.28:0/3924748541 >> 172.21.15.28:0/3924748541 conn(0x7efe0c069e80 msgr2=0x7efe0c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 -- 172.21.15.28:0/3924748541 shutdown_connections 2024-09-13T17:25:32.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.037+0000 7efe14400640 1 -- 172.21.15.28:0/3924748541 wait complete. 2024-09-13T17:25:32.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.038+0000 7efe14400640 1 Processor -- start 2024-09-13T17:25:32.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 -- start start 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe0c0cdfa0 0x7efe0c0ce500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c15d340 0x7efe0c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c14a250 con 0x7efe0c06fb50 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c137430 con 0x7efe0c15d340 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.039+0000 7efe14400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0c149680 con 0x7efe0c0cdfa0 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45716/0 (socket says 172.21.15.28:45716) 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 -- 172.21.15.28:0/433224689 learned_addr learned my addr 172.21.15.28:0/433224689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:32.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11000640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe0c0cdfa0 0x7efe0c0ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe12400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c15d340 0x7efe0c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 -- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe0c0cdfa0 msgr2=0x7efe0c0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe0c0cdfa0 0x7efe0c0ce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 -- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c15d340 msgr2=0x7efe0c0cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c15d340 0x7efe0c0cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.040+0000 7efe11a00640 1 -- 172.21.15.28:0/433224689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe04009a70 con 0x7efe0c06fb50 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe11a00640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7efe0000c490 tx=0x7efe0000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe000052d0 con 0x7efe0c06fb50 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe00005430 con 0x7efe0c06fb50 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe00012870 con 0x7efe0c06fb50 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0c149920 con 0x7efe0c06fb50 2024-09-13T17:25:32.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.041+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0c149ee0 con 0x7efe0c06fb50 2024-09-13T17:25:32.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.042+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0c0b3160 con 0x7efe0c06fb50 2024-09-13T17:25:32.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.043+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7efe000129d0 con 0x7efe0c06fb50 2024-09-13T17:25:32.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.045+0000 7efe09600640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 0x7efdec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.045+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7efe00088ed0 con 0x7efe0c06fb50 2024-09-13T17:25:32.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.045+0000 7efe11000640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 0x7efdec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.045+0000 7efe11000640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 0x7efdec06d0f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7efe0c1f5df0 tx=0x7efdf8000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.049+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7efe00054060 con 0x7efe0c06fb50 2024-09-13T17:25:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.226+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe0c158af0 con 0x7efe0c06fb50 2024-09-13T17:25:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.227+0000 7efe09600640 1 -- 172.21.15.28:0/433224689 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+16652 (secure 0 0 0) 0x7efe00057680 con 0x7efe0c06fb50 2024-09-13T17:25:32.229 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:32.230 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:22.037713+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1114231130":"2024-09-14T17:24:22.037690+0000","172.21.15.28:6833/1114231130":"2024-09-14T17:24:22.037690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 msgr2=0x7efdec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 0x7efdec06d0f0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7efe0c1f5df0 tx=0x7efdf8000ed0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 msgr2=0x7efe0c136ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 secure :-1 s=READY pgs=2838 cs=0 l=1 rev1=1 crypto rx=0x7efe0000c490 tx=0x7efe0000c950 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 shutdown_connections 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7efdec06ac30 0x7efdec06d0f0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe0c15d340 0x7efe0c0cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.229+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7efe0c0cdfa0 0x7efe0c0ce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.230+0000 7efe14400640 1 --2- 172.21.15.28:0/433224689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efe0c06fb50 0x7efe0c136ef0 unknown :-1 s=CLOSED pgs=2838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.230+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 >> 172.21.15.28:0/433224689 conn(0x7efe0c069e80 msgr2=0x7efe0c0b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.230+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 shutdown_connections 2024-09-13T17:25:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.230+0000 7efe14400640 1 -- 172.21.15.28:0/433224689 wait complete. 2024-09-13T17:25:32.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T17:07:58.938630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T17:24:08.441956+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T17:24:09.571290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T17:25:32.248 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T17:25:32.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.404+0000 7f30e5800640 1 Processor -- start 2024-09-13T17:25:32.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.406+0000 7f30e5800640 1 -- start start 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30e5800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30e0070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30e5800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f30e015ab60 0x7f30e00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30e5800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30e006c8b0 con 0x7f30e006bb40 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30e5800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30e006c6a0 con 0x7f30e015ab60 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30e5800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e006cb90 con 0x7f30e015a780 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30df400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f30e006bb40 0x7f30e006bf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47294/0 (socket says 172.21.15.28:47294) 2024-09-13T17:25:32.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.407+0000 7f30df400640 1 -- 172.21.15.28:0/671422210 learned_addr learned my addr 172.21.15.28:0/671422210 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30d6000640 1 --2- 172.21.15.28:0/671422210 >> v2:172.21.15.189:3300/0 conn(0x7f30e015ab60 0x7f30e00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30dea00640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30e0070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910853894 0 0) 0x7f30e006c8b0 con 0x7f30e006bb40 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c4003520 con 0x7f30e006bb40 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1373863558 0 0) 0x7f30c4003520 con 0x7f30e006bb40 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 msgr2=0x7f30e0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30e0070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 >> v2:172.21.15.189:3300/0 conn(0x7f30e015ab60 msgr2=0x7f30e00712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 --2- 172.21.15.28:0/671422210 >> v2:172.21.15.189:3300/0 conn(0x7f30e015ab60 0x7f30e00712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00cc980 con 0x7f30e006bb40 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f30d80019f0 con 0x7f30e006bb40 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 >> v1:172.21.15.28:6789/0 conn(0x7f30e006bb40 legacy=0x7f30e006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30c4003970 0x7f30c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c4004d00 0x7f30c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e006c6a0 con 0x7f30c4003970 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e006cb90 con 0x7f30c4004d00 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c400b980 con 0x7f30e015a780 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.408+0000 7f30d6000640 1 --2- 172.21.15.28:0/671422210 >> v2:172.21.15.189:3300/0 conn(0x7f30e015ab60 0x7f30e00712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30c4003970 0x7f30c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30df400640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c4004d00 0x7f30c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30d6000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 -- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 msgr2=0x7f30c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30c40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 -- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c4004d00 msgr2=0x7f30c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30c4004d00 0x7f30c4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30dea00640 1 -- 172.21.15.28:0/671422210 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c4003520 con 0x7f30c4003970 2024-09-13T17:25:32.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.409+0000 7f30d6000640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30e015a780 0x7f30c40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:32.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30dea00640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30c4003970 0x7f30c4003d90 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009ac0 tx=0x7f30c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b768a1499045cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0031400 con 0x7f30c4003970 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d002d050 con 0x7f30c4003970 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30de000640 1 -- 172.21.15.28:0/671422210 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0031780 con 0x7f30c4003970 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 -- 172.21.15.28:0/671422210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30c4003970 msgr2=0x7f30c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 --2- 172.21.15.28:0/671422210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30c4003970 0x7f30c4003d90 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7f30d0009ac0 tx=0x7f30c40057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30df400640 1 -- 172.21.15.28:0/671422210 reap_dead start 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 -- 172.21.15.28:0/671422210 shutdown_connections 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 -- 172.21.15.28:0/671422210 >> 172.21.15.28:0/671422210 conn(0x7f30e00b0270 msgr2=0x7f30e00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 -- 172.21.15.28:0/671422210 shutdown_connections 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.410+0000 7f30e5800640 1 -- 172.21.15.28:0/671422210 wait complete. 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.413+0000 7f30e5800640 1 Processor -- start 2024-09-13T17:25:32.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.413+0000 7f30e5800640 1 -- start start 2024-09-13T17:25:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30c4003970 0x7f30e00c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e00c7b80 0x7f30e00c8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c6ae0 con 0x7f30e00c5410 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c6dc0 con 0x7f30e00c7b80 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30e5800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30e00c70a0 con 0x7f30c4003970 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45734/0 (socket says 172.21.15.28:45734) 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 -- 172.21.15.28:0/1842583493 learned_addr learned my addr 172.21.15.28:0/1842583493 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30dea00640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30c4003970 0x7f30e00c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30df400640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e00c7b80 0x7f30e00c8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 -- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30c4003970 msgr2=0x7f30e00c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30c4003970 0x7f30e00c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 -- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e00c7b80 msgr2=0x7f30e00c8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e00c7b80 0x7f30e00c8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.414+0000 7f30d6000640 1 -- 172.21.15.28:0/1842583493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e00c7380 con 0x7f30e00c5410 2024-09-13T17:25:32.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30d6000640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f30c80118d0 tx=0x7f30c8011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c800fbd0 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c800fd30 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c8015660 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30e00c7660 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.415+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30e00c9b70 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.416+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30e006c8b0 con 0x7f30e00c5410 2024-09-13T17:25:32.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.417+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f30c80157c0 con 0x7f30e00c5410 2024-09-13T17:25:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.418+0000 7f30d7400640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 0x7f30bc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.419+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f30c80898d0 con 0x7f30e00c5410 2024-09-13T17:25:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.419+0000 7f30dea00640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 0x7f30bc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.420+0000 7f30dea00640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 0x7f30bc06d0f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f30d0033000 tx=0x7f30d0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.423+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f30c8054a60 con 0x7f30e00c5410 2024-09-13T17:25:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.600+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30e0071510 con 0x7f30e00c5410 2024-09-13T17:25:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.600+0000 7f30d7400640 1 -- 172.21.15.28:0/1842583493 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v234) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30c80220a0 con 0x7f30e00c5410 2024-09-13T17:25:32.601 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.602+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 msgr2=0x7f30bc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.602+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 0x7f30bc06d0f0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f30d0033000 tx=0x7f30d0008030 comp rx=0 tx=0).stop 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.602+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 msgr2=0x7f30e00c5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 secure :-1 s=READY pgs=2841 cs=0 l=1 rev1=1 crypto rx=0x7f30c80118d0 tx=0x7f30c8011d90 comp rx=0 tx=0).stop 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 shutdown_connections 2024-09-13T17:25:32.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f30bc06ac30 0x7f30bc06d0f0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30e00c7b80 0x7f30e00c8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f30e00c5410 0x7f30e00c5890 unknown :-1 s=CLOSED pgs=2841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 --2- 172.21.15.28:0/1842583493 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f30c4003970 0x7f30e00c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 >> 172.21.15.28:0/1842583493 conn(0x7f30e00b0270 msgr2=0x7f30e0060980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 shutdown_connections 2024-09-13T17:25:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.603+0000 7f30e5800640 1 -- 172.21.15.28:0/1842583493 wait complete. 2024-09-13T17:25:32.620 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T17:25:32.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.776+0000 7f1d16600640 1 Processor -- start 2024-09-13T17:25:32.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.777+0000 7f1d16600640 1 -- start start 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d16600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d10143f70 0x7f1d100cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d16600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 0x7f1d100c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d16600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d1015ef70 con 0x7f1d100c3ba0 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d16600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d100cda80 con 0x7f1d10143f70 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d16600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d100c4320 con 0x7f1d100cd5f0 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d15200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d10143f70 0x7f1d100cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d15200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f1d10143f70 0x7f1d100cd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45746/0 (socket says 172.21.15.28:45746) 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d15200640 1 -- 172.21.15.28:0/3873126296 learned_addr learned my addr 172.21.15.28:0/3873126296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.778+0000 7f1d07e00640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 0x7f1d100c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.779+0000 7f1d15200640 1 -- 172.21.15.28:0/3873126296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 msgr2=0x7f1d100c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.779+0000 7f1d15200640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 0x7f1d100c3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.779+0000 7f1d15200640 1 -- 172.21.15.28:0/3873126296 >> v1:172.21.15.189:6789/0 conn(0x7f1d100c3ba0 legacy=0x7f1d1006f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.779+0000 7f1d15200640 1 -- 172.21.15.28:0/3873126296 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1015e1b0 con 0x7f1d10143f70 2024-09-13T17:25:32.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.779+0000 7f1d07e00640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 0x7f1d100c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.780+0000 7f1d15200640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10143f70 0x7f1d100cd0b0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f1d00009870 tx=0x7f1d0002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57af789623ab82f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.780+0000 7f1d0fe00640 1 -- 172.21.15.28:0/3873126296 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00032440 con 0x7f1d10143f70 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.780+0000 7f1d0fe00640 1 -- 172.21.15.28:0/3873126296 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d000325a0 con 0x7f1d10143f70 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.780+0000 7f1d0fe00640 1 -- 172.21.15.28:0/3873126296 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d00032930 con 0x7f1d10143f70 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 -- 172.21.15.28:0/3873126296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10143f70 msgr2=0x7f1d100cd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10143f70 0x7f1d100cd0b0 secure :-1 s=READY pgs=2842 cs=0 l=1 rev1=1 crypto rx=0x7f1d00009870 tx=0x7f1d0002a270 comp rx=0 tx=0).stop 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 -- 172.21.15.28:0/3873126296 shutdown_connections 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100cd5f0 0x7f1d100c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 --2- 172.21.15.28:0/3873126296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d10143f70 0x7f1d100cd0b0 unknown :-1 s=CLOSED pgs=2842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 -- 172.21.15.28:0/3873126296 >> 172.21.15.28:0/3873126296 conn(0x7f1d10069e80 msgr2=0x7f1d1006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 -- 172.21.15.28:0/3873126296 shutdown_connections 2024-09-13T17:25:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.781+0000 7f1d16600640 1 -- 172.21.15.28:0/3873126296 wait complete. 2024-09-13T17:25:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.783+0000 7f1d16600640 1 Processor -- start 2024-09-13T17:25:32.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.783+0000 7f1d16600640 1 -- start start 2024-09-13T17:25:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100c3ba0 0x7f1d1014fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d10143f70 0x7f1d10140930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10141c10 con 0x7f1d100cd5f0 2024-09-13T17:25:32.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1015aeb0 con 0x7f1d10143f70 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d16600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d10150130 con 0x7f1d100c3ba0 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45750/0 (socket says 172.21.15.28:45750) 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 -- 172.21.15.28:0/579919126 learned_addr learned my addr 172.21.15.28:0/579919126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d15200640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100c3ba0 0x7f1d1014fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d15c00640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d10143f70 0x7f1d10140930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 -- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100c3ba0 msgr2=0x7f1d1014fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100c3ba0 0x7f1d1014fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 -- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d10143f70 msgr2=0x7f1d10140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d10143f70 0x7f1d10140930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.784+0000 7f1d07e00640 1 -- 172.21.15.28:0/579919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000094d0 con 0x7f1d100cd5f0 2024-09-13T17:25:32.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d07e00640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d7d0 tx=0x7f1cf800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8004030 con 0x7f1d100cd5f0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf8004190 con 0x7f1d100cd5f0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8004520 con 0x7f1d100cd5f0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d10141000 con 0x7f1d100cd5f0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.785+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d101415c0 con 0x7f1d100cd5f0 2024-09-13T17:25:32.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.786+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d100c3660 con 0x7f1d100cd5f0 2024-09-13T17:25:32.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.787+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f1cf800b6c0 con 0x7f1d100cd5f0 2024-09-13T17:25:32.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.788+0000 7f1d0d600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 0x7f1cec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:32.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.788+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1cf8089500 con 0x7f1d100cd5f0 2024-09-13T17:25:32.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.788+0000 7f1d15200640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 0x7f1cec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:32.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.789+0000 7f1d15200640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 0x7f1cec06d0c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f1d00034000 tx=0x7f1d0002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:32.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.792+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f1cf8054600 con 0x7f1d100cd5f0 2024-09-13T17:25:32.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.971+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1d10142da0 con 0x7f1d100cd5f0 2024-09-13T17:25:32.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.972+0000 7f1d0d600640 1 -- 172.21.15.28:0/579919126 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v234) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1cf8057c20 con 0x7f1d100cd5f0 2024-09-13T17:25:32.973 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 msgr2=0x7f1cec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 0x7f1cec06d0c0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f1d00034000 tx=0x7f1d0002e000 comp rx=0 tx=0).stop 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 msgr2=0x7f1d101403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 secure :-1 s=READY pgs=2843 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800d7d0 tx=0x7f1cf800dc90 comp rx=0 tx=0).stop 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 shutdown_connections 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f1cec06ac00 0x7f1cec06d0c0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d10143f70 0x7f1d10140930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1d100cd5f0 0x7f1d101403f0 unknown :-1 s=CLOSED pgs=2843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 --2- 172.21.15.28:0/579919126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f1d100c3ba0 0x7f1d1014fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 >> 172.21.15.28:0/579919126 conn(0x7f1d10069e80 msgr2=0x7f1d1006f970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 shutdown_connections 2024-09-13T17:25:32.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:32.974+0000 7f1d16600640 1 -- 172.21.15.28:0/579919126 wait complete. 2024-09-13T17:25:32.990 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T17:25:33.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.147+0000 7fa350200640 1 Processor -- start 2024-09-13T17:25:33.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.147+0000 7fa350200640 1 -- start start 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa350200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34815ab60 0x7fa34806bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa350200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa348071490 0x7fa34806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa350200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34806c8b0 con 0x7fa34815a780 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa350200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34806cb90 con 0x7fa34815ab60 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa350200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3480cc980 con 0x7fa348071490 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.148+0000 7fa34ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34815ab60 0x7fa34806bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa34815ab60 0x7fa34806bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45756/0 (socket says 172.21.15.28:45756) 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 -- 172.21.15.28:0/555021575 learned_addr learned my addr 172.21.15.28:0/555021575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34e200640 1 --2- 172.21.15.28:0/555021575 >> v2:172.21.15.189:3300/0 conn(0x7fa348071490 0x7fa34806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 -- 172.21.15.28:0/555021575 >> v2:172.21.15.189:3300/0 conn(0x7fa348071490 msgr2=0x7fa34806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:33.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 --2- 172.21.15.28:0/555021575 >> v2:172.21.15.189:3300/0 conn(0x7fa348071490 0x7fa34806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 -- 172.21.15.28:0/555021575 >> v1:172.21.15.189:6789/0 conn(0x7fa34815a780 legacy=0x7fa348070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.149+0000 7fa34ce00640 1 -- 172.21.15.28:0/555021575 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3480ccc60 con 0x7fa34815ab60 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.150+0000 7fa34ce00640 1 --2- 172.21.15.28:0/555021575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34815ab60 0x7fa34806bb40 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7fa3480cc370 tx=0x7fa33402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786986c0ce79c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.150+0000 7fa347e00640 1 -- 172.21.15.28:0/555021575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334032440 con 0x7fa34815ab60 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.150+0000 7fa347e00640 1 -- 172.21.15.28:0/555021575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3340325a0 con 0x7fa34815ab60 2024-09-13T17:25:33.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.150+0000 7fa347e00640 1 -- 172.21.15.28:0/555021575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa334032930 con 0x7fa34815ab60 2024-09-13T17:25:33.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 -- 172.21.15.28:0/555021575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34815ab60 msgr2=0x7fa34806bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 --2- 172.21.15.28:0/555021575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34815ab60 0x7fa34806bb40 secure :-1 s=READY pgs=2844 cs=0 l=1 rev1=1 crypto rx=0x7fa3480cc370 tx=0x7fa33402a270 comp rx=0 tx=0).stop 2024-09-13T17:25:33.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 -- 172.21.15.28:0/555021575 shutdown_connections 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 --2- 172.21.15.28:0/555021575 >> v2:172.21.15.189:3300/0 conn(0x7fa348071490 0x7fa34806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 --2- 172.21.15.28:0/555021575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa34815ab60 0x7fa34806bb40 unknown :-1 s=CLOSED pgs=2844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 -- 172.21.15.28:0/555021575 >> 172.21.15.28:0/555021575 conn(0x7fa3480b0270 msgr2=0x7fa3480b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 -- 172.21.15.28:0/555021575 shutdown_connections 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.151+0000 7fa350200640 1 -- 172.21.15.28:0/555021575 wait complete. 2024-09-13T17:25:33.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.153+0000 7fa350200640 1 Processor -- start 2024-09-13T17:25:33.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.153+0000 7fa350200640 1 -- start start 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa34815a780 0x7fa3480b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa34815ab60 0x7fa3480ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480b31d0 con 0x7fa348071490 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa348147680 con 0x7fa34815a780 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa350200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3480b2550 con 0x7fa34815ab60 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45758/0 (socket says 172.21.15.28:45758) 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 -- 172.21.15.28:0/3632278013 learned_addr learned my addr 172.21.15.28:0/3632278013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34ce00640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa34815a780 0x7fa3480b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 -- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa34815ab60 msgr2=0x7fa3480ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa34815ab60 0x7fa3480ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 -- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa34815a780 msgr2=0x7fa3480b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa34815a780 0x7fa3480b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.154+0000 7fa34d800640 1 -- 172.21.15.28:0/3632278013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3340094d0 con 0x7fa348071490 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa34d800640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c450 tx=0x7fa33c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c0052d0 con 0x7fa348071490 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa33c005430 con 0x7fa348071490 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3480b2830 con 0x7fa348071490 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa33c012870 con 0x7fa348071490 2024-09-13T17:25:33.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.155+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3480b2df0 con 0x7fa348071490 2024-09-13T17:25:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.156+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3480b3440 con 0x7fa348071490 2024-09-13T17:25:33.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.157+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa33c0129d0 con 0x7fa348071490 2024-09-13T17:25:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.158+0000 7fa345600640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 0x7fa32806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.158+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa33c088f80 con 0x7fa348071490 2024-09-13T17:25:33.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.158+0000 7fa34ce00640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 0x7fa32806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.159+0000 7fa34ce00640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 0x7fa32806d0c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa334034000 tx=0x7fa33402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.162+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa33c054060 con 0x7fa348071490 2024-09-13T17:25:33.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.340+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa3480adf50 con 0x7fa348071490 2024-09-13T17:25:33.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.341+0000 7fa345600640 1 -- 172.21.15.28:0/3632278013 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v234) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa33c057680 con 0x7fa348071490 2024-09-13T17:25:33.342 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.343+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 msgr2=0x7fa32806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.343+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 0x7fa32806d0c0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fa334034000 tx=0x7fa33402e000 comp rx=0 tx=0).stop 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.343+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 msgr2=0x7fa348147140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.343+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 secure :-1 s=READY pgs=2845 cs=0 l=1 rev1=1 crypto rx=0x7fa33c00c450 tx=0x7fa33c00c910 comp rx=0 tx=0).stop 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 shutdown_connections 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa32806ac00 0x7fa32806d0c0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa34815ab60 0x7fa3480ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa34815a780 0x7fa3480b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 --2- 172.21.15.28:0/3632278013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa348071490 0x7fa348147140 unknown :-1 s=CLOSED pgs=2845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 >> 172.21.15.28:0/3632278013 conn(0x7fa3480b0270 msgr2=0x7fa34806f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 shutdown_connections 2024-09-13T17:25:33.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.344+0000 7fa350200640 1 -- 172.21.15.28:0/3632278013 wait complete. 2024-09-13T17:25:33.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:25:33.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.515+0000 7ff3d7400640 1 Processor -- start 2024-09-13T17:25:33.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.515+0000 7ff3d7400640 1 -- start start 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.516+0000 7ff3d7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7ff3d01592a0 0x7ff3d015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3d7400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d0159d90 con 0x7ff3d006f970 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3d7400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3d00b2040 con 0x7ff3d006c2d0 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3d7400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3d015b0b0 con 0x7ff3d01592a0 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3c7e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7ff3d006f970 0x7ff3d0159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:47296/0 (socket says 172.21.15.28:47296) 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3c7e00640 1 -- 172.21.15.28:0/490821016 learned_addr learned my addr 172.21.15.28:0/490821016 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:33.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3d4a00640 1 --2- 172.21.15.28:0/490821016 >> v2:172.21.15.189:3300/0 conn(0x7ff3d01592a0 0x7ff3d015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 560247300 0 0) 0x7ff3d0159d90 con 0x7ff3d006f970 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.517+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b4003520 con 0x7ff3d006f970 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.518+0000 7ff3d4a00640 1 -- 172.21.15.28:0/490821016 >> v1:172.21.15.189:6789/0 conn(0x7ff3d006c2d0 legacy=0x7ff3d00b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.518+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058811601 0 0) 0x7ff3d00b2040 con 0x7ff3d006c2d0 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.518+0000 7ff3d4a00640 1 -- 172.21.15.28:0/490821016 >> v1:172.21.15.28:6789/0 conn(0x7ff3d006f970 legacy=0x7ff3d0159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.518+0000 7ff3d4a00640 1 -- 172.21.15.28:0/490821016 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d0159ff0 con 0x7ff3d01592a0 2024-09-13T17:25:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.519+0000 7ff3d4a00640 1 --2- 172.21.15.28:0/490821016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d01592a0 0x7ff3d015ab70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00097a0 tx=0x7ff3c002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd7a3944f0a64034 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.519+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8001690 con 0x7ff3d01592a0 2024-09-13T17:25:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.519+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c002ac30 con 0x7ff3d01592a0 2024-09-13T17:25:33.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.519+0000 7ff3cfe00640 1 -- 172.21.15.28:0/490821016 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0032440 con 0x7ff3d01592a0 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 -- 172.21.15.28:0/490821016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d01592a0 msgr2=0x7ff3d015ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 --2- 172.21.15.28:0/490821016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d01592a0 0x7ff3d015ab70 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00097a0 tx=0x7ff3c002a1a0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 -- 172.21.15.28:0/490821016 shutdown_connections 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 --2- 172.21.15.28:0/490821016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d01592a0 0x7ff3d015ab70 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 -- 172.21.15.28:0/490821016 >> 172.21.15.28:0/490821016 conn(0x7ff3d0069e80 msgr2=0x7ff3d006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:33.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 -- 172.21.15.28:0/490821016 shutdown_connections 2024-09-13T17:25:33.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.520+0000 7ff3d7400640 1 -- 172.21.15.28:0/490821016 wait complete. 2024-09-13T17:25:33.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.522+0000 7ff3d7400640 1 Processor -- start 2024-09-13T17:25:33.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.522+0000 7ff3d7400640 1 -- start start 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3d006c2d0 0x7ff3d0143a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d006f970 0x7ff3d0151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0151ed0 con 0x7ff3d01592a0 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d01521b0 con 0x7ff3d006f970 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d7400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3d0152490 con 0x7ff3d006c2d0 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45768/0 (socket says 172.21.15.28:45768) 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 -- 172.21.15.28:0/3008928743 learned_addr learned my addr 172.21.15.28:0/3008928743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:33.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3c7e00640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d006f970 0x7ff3d0151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d4a00640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3d006c2d0 0x7ff3d0143a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 -- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3d006c2d0 msgr2=0x7ff3d0143a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3d006c2d0 0x7ff3d0143a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 -- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d006f970 msgr2=0x7ff3d0151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d006f970 0x7ff3d0151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.523+0000 7ff3d5400640 1 -- 172.21.15.28:0/3008928743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0009400 con 0x7ff3d01592a0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3d5400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8001560 tx=0x7ff3c800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8003da0 con 0x7ff3d01592a0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3c8005070 con 0x7ff3d01592a0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3d0149960 con 0x7ff3d01592a0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8005420 con 0x7ff3d01592a0 2024-09-13T17:25:33.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.524+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3d014b600 con 0x7ff3d01592a0 2024-09-13T17:25:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.525+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3d0149e80 con 0x7ff3d01592a0 2024-09-13T17:25:33.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.526+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7ff3c8005580 con 0x7ff3d01592a0 2024-09-13T17:25:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.527+0000 7ff3cd600640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 0x7ff3ac06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.527+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff3c8088cb0 con 0x7ff3d01592a0 2024-09-13T17:25:33.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.528+0000 7ff3d4a00640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 0x7ff3ac06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.528+0000 7ff3d4a00640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 0x7ff3ac06d0f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0034000 tx=0x7ff3c002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.531+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7ff3d0149e80 con 0x7ff3d01592a0 2024-09-13T17:25:33.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.746+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3d0149e80 con 0x7ff3d01592a0 2024-09-13T17:25:33.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.747+0000 7ff3cd600640 1 -- 172.21.15.28:0/3008928743 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7ff3d0149e80 con 0x7ff3d01592a0 2024-09-13T17:25:33.748 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:33.748 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:25:33.749 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:25:33.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 msgr2=0x7ff3ac06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 0x7ff3ac06d0f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0034000 tx=0x7ff3c002e000 comp rx=0 tx=0).stop 2024-09-13T17:25:33.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 msgr2=0x7ff3d01442f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8001560 tx=0x7ff3c800cc70 comp rx=0 tx=0).stop 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 shutdown_connections 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7ff3ac06ac30 0x7ff3ac06d0f0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d01592a0 0x7ff3d01442f0 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3d006f970 0x7ff3d0151410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 --2- 172.21.15.28:0/3008928743 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7ff3d006c2d0 0x7ff3d0143a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 >> 172.21.15.28:0/3008928743 conn(0x7ff3d0069e80 msgr2=0x7ff3d00ace10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 shutdown_connections 2024-09-13T17:25:33.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.749+0000 7ff3d7400640 1 -- 172.21.15.28:0/3008928743 wait complete. 2024-09-13T17:25:33.766 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:25:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.921+0000 7f4d2c200640 1 Processor -- start 2024-09-13T17:25:33.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.921+0000 7f4d2c200640 1 -- start start 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d2c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 0x7f4d2415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d2c200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d24159d90 con 0x7f4d2406f970 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d2c200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d240b2040 con 0x7f4d2406c2d0 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d2c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2415b0b0 con 0x7f4d241592a0 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d28e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4d2406f970 0x7f4d24159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58088/0 (socket says 172.21.15.28:58088) 2024-09-13T17:25:33.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.922+0000 7f4d28e00640 1 -- 172.21.15.28:0/345946273 learned_addr learned my addr 172.21.15.28:0/345946273 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d29800640 1 --2- 172.21.15.28:0/345946273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 0x7f4d2415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825908155 0 0) 0x7f4d24159d90 con 0x7f4d2406f970 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d0c003520 con 0x7f4d2406f970 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883012836 0 0) 0x7f4d240b2040 con 0x7f4d2406c2d0 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24159d90 con 0x7f4d2406c2d0 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d29800640 1 -- 172.21.15.28:0/345946273 >> v1:172.21.15.189:6789/0 conn(0x7f4d2406c2d0 legacy=0x7f4d240b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d29800640 1 -- 172.21.15.28:0/345946273 >> v1:172.21.15.28:6789/0 conn(0x7f4d2406f970 legacy=0x7f4d24159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.923+0000 7f4d29800640 1 -- 172.21.15.28:0/345946273 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24159ff0 con 0x7f4d241592a0 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.924+0000 7f4d29800640 1 --2- 172.21.15.28:0/345946273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 0x7f4d2415ab70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097d0 tx=0x7f4d1802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4466313410ba287 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18032440 con 0x7f4d241592a0 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d180325a0 con 0x7f4d241592a0 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d23e00640 1 -- 172.21.15.28:0/345946273 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18032930 con 0x7f4d241592a0 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 -- 172.21.15.28:0/345946273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 msgr2=0x7f4d2415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 --2- 172.21.15.28:0/345946273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 0x7f4d2415ab70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097d0 tx=0x7f4d1802a210 comp rx=0 tx=0).stop 2024-09-13T17:25:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 -- 172.21.15.28:0/345946273 shutdown_connections 2024-09-13T17:25:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 --2- 172.21.15.28:0/345946273 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d241592a0 0x7f4d2415ab70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 -- 172.21.15.28:0/345946273 >> 172.21.15.28:0/345946273 conn(0x7f4d24069e80 msgr2=0x7f4d2406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 -- 172.21.15.28:0/345946273 shutdown_connections 2024-09-13T17:25:33.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.925+0000 7f4d2c200640 1 -- 172.21.15.28:0/345946273 wait complete. 2024-09-13T17:25:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.927+0000 7f4d2c200640 1 Processor -- start 2024-09-13T17:25:33.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.927+0000 7f4d2c200640 1 -- start start 2024-09-13T17:25:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d2406f970 0x7f4d240aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 0x7f4d240a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240ac800 con 0x7f4d2406c2d0 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240acae0 con 0x7f4d241592a0 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2c200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d240acdc0 con 0x7f4d2406f970 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d29800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58110/0 (socket says 172.21.15.28:58110) 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d29800640 1 -- 172.21.15.28:0/3208042960 learned_addr learned my addr 172.21.15.28:0/3208042960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d28e00640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d2406f970 0x7f4d240aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.928+0000 7f4d2a200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 0x7f4d240a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 -- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d2406f970 msgr2=0x7f4d240aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d2406f970 0x7f4d240aa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 -- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 msgr2=0x7f4d240a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 0x7f4d240a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:33.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 -- 172.21.15.28:0/3208042960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d18009400 con 0x7f4d2406c2d0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d2a200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 0x7f4d240a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d29800640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097d0 tx=0x7f4d180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1803d830 con 0x7f4d2406c2d0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1803d9b0 con 0x7f4d2406c2d0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d240ad0a0 con 0x7f4d2406c2d0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1803dd60 con 0x7f4d2406c2d0 2024-09-13T17:25:33.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.929+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d240ac4d0 con 0x7f4d2406c2d0 2024-09-13T17:25:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.930+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d24152ab0 con 0x7f4d2406c2d0 2024-09-13T17:25:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.931+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f4d18039040 con 0x7f4d2406c2d0 2024-09-13T17:25:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.932+0000 7f4d21600640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 0x7f4d0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:33.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.933+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4d180a9e60 con 0x7f4d2406c2d0 2024-09-13T17:25:33.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.933+0000 7f4d28e00640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 0x7f4d0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:33.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.934+0000 7f4d28e00640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 0x7f4d0406d0c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4d240cde20 tx=0x7f4d10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:33.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:33.937+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f4d18074ff0 con 0x7f4d2406c2d0 2024-09-13T17:25:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.116+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d2415ab70 con 0x7f4d2406c2d0 2024-09-13T17:25:34.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.117+0000 7f4d21600640 1 -- 172.21.15.28:0/3208042960 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f4d18078610 con 0x7f4d2406c2d0 2024-09-13T17:25:34.118 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:34.119 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:22.037713+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1114231130":"2024-09-14T17:24:22.037690+0000","172.21.15.28:6833/1114231130":"2024-09-14T17:24:22.037690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:25:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 msgr2=0x7f4d0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 0x7f4d0406d0c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f4d240cde20 tx=0x7f4d10009040 comp rx=0 tx=0).stop 2024-09-13T17:25:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 msgr2=0x7f4d240abf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7f4d180097d0 tx=0x7f4d180046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 shutdown_connections 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f4d0406ac00 0x7f4d0406d0c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d241592a0 0x7f4d240a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4d2406f970 0x7f4d240aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 --2- 172.21.15.28:0/3208042960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d2406c2d0 0x7f4d240abf90 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 >> 172.21.15.28:0/3208042960 conn(0x7f4d24069e80 msgr2=0x7f4d240ad970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 shutdown_connections 2024-09-13T17:25:34.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.120+0000 7f4d2c200640 1 -- 172.21.15.28:0/3208042960 wait complete. 2024-09-13T17:25:34.136 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:25:34.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:25:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.292+0000 7fbf74800640 1 Processor -- start 2024-09-13T17:25:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.292+0000 7fbf74800640 1 -- start start 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf74800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fbf6c06c9e0 0x7fbf6c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf74800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c159d90 con 0x7fbf6c1592a0 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf74800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf6c0b2040 con 0x7fbf6c06f970 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf74800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf6c15b0b0 con 0x7fbf6c06c9e0 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf71e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbf6c1592a0 0x7fbf6c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58098/0 (socket says 172.21.15.28:58098) 2024-09-13T17:25:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf71e00640 1 -- 172.21.15.28:0/335921233 learned_addr learned my addr 172.21.15.28:0/335921233 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106465446 0 0) 0x7fbf6c159d90 con 0x7fbf6c1592a0 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf54003520 con 0x7fbf6c1592a0 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.294+0000 7fbf72800640 1 --2- 172.21.15.28:0/335921233 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c06c9e0 0x7fbf6c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186763852 0 0) 0x7fbf54003520 con 0x7fbf6c1592a0 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c06c9e0 msgr2=0x7fbf6c15ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 --2- 172.21.15.28:0/335921233 >> v2:172.21.15.189:3300/0 conn(0x7fbf6c06c9e0 0x7fbf6c15ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 >> v1:172.21.15.189:6789/0 conn(0x7fbf6c06f970 legacy=0x7fbf6c06c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c159ff0 con 0x7fbf6c1592a0 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fbf60001690 con 0x7fbf6c1592a0 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 >> v1:172.21.15.28:6789/0 conn(0x7fbf6c1592a0 legacy=0x7fbf6c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 0x7fbf54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf540047c0 0x7fbf54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54005180 0x7fbf5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c15b0b0 con 0x7fbf54005180 2024-09-13T17:25:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400c080 con 0x7fbf540039c0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5400c240 con 0x7fbf540047c0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf71e00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 0x7fbf54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf72800640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54005180 0x7fbf5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.295+0000 7fbf71400640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf540047c0 0x7fbf54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf71e00640 1 -- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf540047c0 msgr2=0x7fbf54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf71e00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf540047c0 0x7fbf54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf71e00640 1 -- 172.21.15.28:0/335921233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54005180 msgr2=0x7fbf5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf71e00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54005180 0x7fbf5400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf71e00640 1 -- 172.21.15.28:0/335921233 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf54003520 con 0x7fbf540039c0 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf72800640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf54005180 0x7fbf5400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.296+0000 7fbf72800640 1 -- 172.21.15.28:0/335921233 reap_dead start 2024-09-13T17:25:34.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.297+0000 7fbf71e00640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 0x7fbf54003de0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c160 tx=0x7fbf540057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48039cd0aa4b563c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.297+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60005ad0 con 0x7fbf540039c0 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.297+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf6002d070 con 0x7fbf540039c0 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.297+0000 7fbf70a00640 1 -- 172.21.15.28:0/335921233 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60005e70 con 0x7fbf540039c0 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 -- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 msgr2=0x7fbf54003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 0x7fbf54003de0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbf6000c160 tx=0x7fbf540057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 -- 172.21.15.28:0/335921233 shutdown_connections 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 --2- 172.21.15.28:0/335921233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf540039c0 0x7fbf54003de0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 -- 172.21.15.28:0/335921233 >> 172.21.15.28:0/335921233 conn(0x7fbf6c069e80 msgr2=0x7fbf6c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 -- 172.21.15.28:0/335921233 shutdown_connections 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.298+0000 7fbf74800640 1 -- 172.21.15.28:0/335921233 wait complete. 2024-09-13T17:25:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.300+0000 7fbf74800640 1 Processor -- start 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.300+0000 7fbf74800640 1 -- start start 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c0cecf0 0x7fbf6c15b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 0x7fbf6c15b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0ccc10 con 0x7fbf540039c0 2024-09-13T17:25:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0cc090 con 0x7fbf6c0cecf0 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf74800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf6c0cc370 con 0x7fbf6c0cf170 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58140/0 (socket says 172.21.15.28:58140) 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf71e00640 1 -- 172.21.15.28:0/1487497358 learned_addr learned my addr 172.21.15.28:0/1487497358 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf71400640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c0cecf0 0x7fbf6c15b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.301+0000 7fbf72800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 0x7fbf6c15b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 -- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 msgr2=0x7fbf6c15b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 0x7fbf6c15b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 -- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c0cecf0 msgr2=0x7fbf6c15b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c0cecf0 0x7fbf6c15b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 -- 172.21.15.28:0/1487497358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c0cc650 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf72800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 0x7fbf6c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf71e00640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7fbf6002abb0 tx=0x7fbf60001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf600049f0 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf60004b50 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf60007070 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.302+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf6c0cc930 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.303+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf6c1485d0 con 0x7fbf540039c0 2024-09-13T17:25:34.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.304+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf6c15ab70 con 0x7fbf540039c0 2024-09-13T17:25:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.304+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbf6002f030 con 0x7fbf540039c0 2024-09-13T17:25:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.305+0000 7fbf66000640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 0x7fbf5006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.305+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbf600a8a20 con 0x7fbf540039c0 2024-09-13T17:25:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.306+0000 7fbf71400640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 0x7fbf5006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.306+0000 7fbf71400640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 0x7fbf5006d0c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c15c370 tx=0x7fbf5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.309+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbf60073b80 con 0x7fbf540039c0 2024-09-13T17:25:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.524+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf6c0ccd80 con 0x7fbf540039c0 2024-09-13T17:25:34.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.524+0000 7fbf66000640 1 -- 172.21.15.28:0/1487497358 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fbf600771a0 con 0x7fbf540039c0 2024-09-13T17:25:34.525 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:34.526 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:25:34.526 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 msgr2=0x7fbf5006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 0x7fbf5006d0c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fbf6c15c370 tx=0x7fbf5c009040 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 msgr2=0x7fbf6c148090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7fbf6002abb0 tx=0x7fbf60001950 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 shutdown_connections 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbf5006ac00 0x7fbf5006d0c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbf6c0cf170 0x7fbf6c15b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf6c0cecf0 0x7fbf6c15b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 --2- 172.21.15.28:0/1487497358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbf540039c0 0x7fbf6c148090 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.526+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 >> 172.21.15.28:0/1487497358 conn(0x7fbf6c069e80 msgr2=0x7fbf6c0ac850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.527+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 shutdown_connections 2024-09-13T17:25:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.527+0000 7fbf74800640 1 -- 172.21.15.28:0/1487497358 wait complete. 2024-09-13T17:25:34.543 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:25:34.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.697+0000 7fabc9200640 1 Processor -- start 2024-09-13T17:25:34.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.697+0000 7fabc9200640 1 -- start start 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc406c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc9200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40b39d0 con 0x7fabc4158b20 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc9200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc40b2dc0 con 0x7fabc406ba70 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc406c930 con 0x7fabc40afbd0 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc3400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fabc4158b20 0x7fabc406b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58102/0 (socket says 172.21.15.28:58102) 2024-09-13T17:25:34.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.698+0000 7fabc3400640 1 -- 172.21.15.28:0/3323548384 learned_addr learned my addr 172.21.15.28:0/3323548384 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc3e00640 1 --2- 172.21.15.28:0/3323548384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc406c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144192670 0 0) 0x7fabc40b39d0 con 0x7fabc4158b20 2024-09-13T17:25:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faba8003440 con 0x7fabc4158b20 2024-09-13T17:25:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108591137 0 0) 0x7fabc40b2dc0 con 0x7fabc406ba70 2024-09-13T17:25:34.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc40b39d0 con 0x7fabc406ba70 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc3e00640 1 -- 172.21.15.28:0/3323548384 >> v1:172.21.15.189:6789/0 conn(0x7fabc406ba70 legacy=0x7fabc4158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc3e00640 1 -- 172.21.15.28:0/3323548384 >> v1:172.21.15.28:6789/0 conn(0x7fabc4158b20 legacy=0x7fabc406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846450096 0 0) 0x7faba8003440 con 0x7fabc4158b20 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.699+0000 7fabc3e00640 1 -- 172.21.15.28:0/3323548384 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc40b30a0 con 0x7fabc40afbd0 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.700+0000 7fabc3e00640 1 --2- 172.21.15.28:0/3323548384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc406c3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009dc0 tx=0x7fabbc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b98261c057b88e9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.700+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4001670 con 0x7fabc40afbd0 2024-09-13T17:25:34.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc02d050 con 0x7fabc40afbd0 2024-09-13T17:25:34.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc2a00640 1 -- 172.21.15.28:0/3323548384 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc007760 con 0x7fabc40afbd0 2024-09-13T17:25:34.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 -- 172.21.15.28:0/3323548384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 msgr2=0x7fabc406c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 --2- 172.21.15.28:0/3323548384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc406c3f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009dc0 tx=0x7fabbc02a660 comp rx=0 tx=0).stop 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 -- 172.21.15.28:0/3323548384 shutdown_connections 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 --2- 172.21.15.28:0/3323548384 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc406c3f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 -- 172.21.15.28:0/3323548384 >> 172.21.15.28:0/3323548384 conn(0x7fabc4072e40 msgr2=0x7fabc4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 -- 172.21.15.28:0/3323548384 shutdown_connections 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.701+0000 7fabc9200640 1 -- 172.21.15.28:0/3323548384 wait complete. 2024-09-13T17:25:34.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.703+0000 7fabc9200640 1 Processor -- start 2024-09-13T17:25:34.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.703+0000 7fabc9200640 1 -- start start 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabc406ba70 0x7fabc41e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc41e6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc4139530 con 0x7fabc4158b20 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc40af280 con 0x7fabc406ba70 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc9200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc4138c40 con 0x7fabc40afbd0 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58142/0 (socket says 172.21.15.28:58142) 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc3e00640 1 -- 172.21.15.28:0/3482546280 learned_addr learned my addr 172.21.15.28:0/3482546280 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabc3400640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabc406ba70 0x7fabc41e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.704+0000 7fabbaa00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc41e6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 -- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 msgr2=0x7fabc41e6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc41e6080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 -- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabc406ba70 msgr2=0x7fabc41e5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabc406ba70 0x7fabc41e5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 -- 172.21.15.28:0/3482546280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabbc009a70 con 0x7fabc4158b20 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabbaa00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc41e6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:34.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc3e00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009ee0 tx=0x7fabbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc03c830 con 0x7fabc4158b20 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabbc03c990 con 0x7fabc4158b20 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc4138ee0 con 0x7fabc4158b20 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.705+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabbc03cd00 con 0x7fabc4158b20 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.706+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc4154920 con 0x7fabc4158b20 2024-09-13T17:25:34.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.707+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc4154ae0 con 0x7fabc4158b20 2024-09-13T17:25:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.707+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fabbc045460 con 0x7fabc4158b20 2024-09-13T17:25:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.708+0000 7fabbbe00640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 0x7faba006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.708+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fabbc0aadb0 con 0x7fabc4158b20 2024-09-13T17:25:34.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.709+0000 7fabc3400640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 0x7faba006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:34.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.709+0000 7fabc3400640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 0x7faba006d0c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fabb4000f60 tx=0x7fabb4001690 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:34.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.712+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fabc4154ae0 con 0x7fabc4158b20 2024-09-13T17:25:34.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.925+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabc4154ae0 con 0x7fabc4158b20 2024-09-13T17:25:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.926+0000 7fabbbe00640 1 -- 172.21.15.28:0/3482546280 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+4749 (secure 0 0 0) 0x7fabc4154ae0 con 0x7fabc4158b20 2024-09-13T17:25:34.927 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:34.927 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":195,"default_fscid":16,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T17:24:10.577893+0000","modified":"2024-09-13T17:24:27.074178+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":234,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":12516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12516":{"gid":12516,"name":"a","rank":0,"incarnation":192,"state":"up:active","state_seq":4,"addr":"172.21.15.28:6835/994067792","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":994067792},{"type":"v1","addr":"172.21.15.28:6835","nonce":994067792}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2024-09-13T17:25:34.927 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 195 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.928+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 msgr2=0x7faba006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.928+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 0x7faba006d0c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fabb4000f60 tx=0x7fabb4001690 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.928+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 msgr2=0x7fabc413aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.928+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fabbc009ee0 tx=0x7fabbc004a40 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 shutdown_connections 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7faba006ac00 0x7faba006d0c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fabc4158b20 0x7fabc413aba0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fabc40afbd0 0x7fabc41e6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 --2- 172.21.15.28:0/3482546280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fabc406ba70 0x7fabc41e5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 >> 172.21.15.28:0/3482546280 conn(0x7fabc4072e40 msgr2=0x7fabc406b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 shutdown_connections 2024-09-13T17:25:34.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:34.929+0000 7fabc9200640 1 -- 172.21.15.28:0/3482546280 wait complete. 2024-09-13T17:25:34.945 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:25:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.099+0000 7f988b400640 1 Processor -- start 2024-09-13T17:25:35.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.099+0000 7f988b400640 1 -- start start 2024-09-13T17:25:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.100+0000 7f988b400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f98841592a0 0x7f988415ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.100+0000 7f988b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406f970 0x7f9884159680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.100+0000 7f988b400640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98840b2040 con 0x7f988406c2d0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.100+0000 7f988b400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988415b0b0 con 0x7f98841592a0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.100+0000 7f988b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9884159ff0 con 0x7f988406f970 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.101+0000 7f987be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406f970 0x7f9884159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.101+0000 7f9888a00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f98841592a0 0x7f988415ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.101+0000 7f9889400640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f988406c2d0 0x7f98840b2540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:59828/0 (socket says 172.21.15.28:59828) 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.101+0000 7f9889400640 1 -- 172.21.15.28:0/3469571484 learned_addr learned my addr 172.21.15.28:0/3469571484 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.102+0000 7f9888a00640 1 -- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406f970 msgr2=0x7f9884159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.102+0000 7f9888a00640 1 --2- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406f970 0x7f9884159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.102+0000 7f9888a00640 1 -- 172.21.15.28:0/3469571484 >> v1:172.21.15.189:6789/0 conn(0x7f988406c2d0 legacy=0x7f98840b2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.102+0000 7f9888a00640 1 -- 172.21.15.28:0/3469571484 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988415a2d0 con 0x7f98841592a0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.103+0000 7f9888a00640 1 --2- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98841592a0 0x7f988415ab70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f987400ba40 tx=0x7f987402c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=babc5cf15e053f0b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.103+0000 7f9883e00640 1 -- 172.21.15.28:0/3469571484 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874032750 con 0x7f98841592a0 2024-09-13T17:25:35.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.103+0000 7f9883e00640 1 -- 172.21.15.28:0/3469571484 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98740328b0 con 0x7f98841592a0 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.103+0000 7f9883e00640 1 -- 172.21.15.28:0/3469571484 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9874032c40 con 0x7f98841592a0 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 -- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98841592a0 msgr2=0x7f988415ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 --2- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98841592a0 0x7f988415ab70 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f987400ba40 tx=0x7f987402c850 comp rx=0 tx=0).stop 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 -- 172.21.15.28:0/3469571484 shutdown_connections 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 --2- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406f970 0x7f9884159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 --2- 172.21.15.28:0/3469571484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f98841592a0 0x7f988415ab70 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 -- 172.21.15.28:0/3469571484 >> 172.21.15.28:0/3469571484 conn(0x7f9884069e80 msgr2=0x7f988406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 -- 172.21.15.28:0/3469571484 shutdown_connections 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.104+0000 7f988b400640 1 -- 172.21.15.28:0/3469571484 wait complete. 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.105+0000 7f988b400640 1 Processor -- start 2024-09-13T17:25:35.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.106+0000 7f988b400640 1 -- start start 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.106+0000 7f988b400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406c2d0 0x7f9884150c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.106+0000 7f988b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988406f970 0x7f98840ab630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.106+0000 7f988b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f988b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98840aa210 con 0x7f98841592a0 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f988b400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98840abf80 con 0x7f988406f970 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f988b400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98840ac260 con 0x7f988406c2d0 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58144/0 (socket says 172.21.15.28:58144) 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 -- 172.21.15.28:0/1529952691 learned_addr learned my addr 172.21.15.28:0/1529952691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f987be00640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988406f970 0x7f98840ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9888a00640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406c2d0 0x7f9884150c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 -- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406c2d0 msgr2=0x7f9884150c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406c2d0 0x7f9884150c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 -- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988406f970 msgr2=0x7f98840ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988406f970 0x7f98840ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.107+0000 7f9889400640 1 -- 172.21.15.28:0/1529952691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987400b6a0 con 0x7f98841592a0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f9889400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f987c00cbc0 tx=0x7f987c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00ad40 con 0x7f98841592a0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c012400 con 0x7f98841592a0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98840ac540 con 0x7f98841592a0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c00aea0 con 0x7f98841592a0 2024-09-13T17:25:35.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.108+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98840acb00 con 0x7f98841592a0 2024-09-13T17:25:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.109+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988415ab70 con 0x7f98841592a0 2024-09-13T17:25:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.110+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f987c007070 con 0x7f98841592a0 2024-09-13T17:25:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.111+0000 7f9881600640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 0x7f986006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.111+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f987c089230 con 0x7f98841592a0 2024-09-13T17:25:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.112+0000 7f9888a00640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 0x7f986006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.112+0000 7f9888a00640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 0x7f986006d0c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f9874037000 tx=0x7f9874036040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.116+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f987c054450 con 0x7f98841592a0 2024-09-13T17:25:35.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.295+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98840accf0 con 0x7f98841592a0 2024-09-13T17:25:35.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.296+0000 7f9881600640 1 -- 172.21.15.28:0/1529952691 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+16652 (secure 0 0 0) 0x7f987c057a70 con 0x7f98841592a0 2024-09-13T17:25:35.297 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:35.297 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":234,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:24:22.037713+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"cephfs_metadata","create_time":"2024-09-13T17:24:08.441956+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"cephfs_data","create_time":"2024-09-13T17:24:09.571290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/1114231130":"2024-09-14T17:24:22.037690+0000","172.21.15.28:6833/1114231130":"2024-09-14T17:24:22.037690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:25:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 msgr2=0x7f986006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 0x7f986006d0c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f9874037000 tx=0x7f9874036040 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 msgr2=0x7f9884155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7f987c00cbc0 tx=0x7f987c00a430 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 shutdown_connections 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f986006ac00 0x7f986006d0c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f98841592a0 0x7f9884155600 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f988406f970 0x7f98840ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 --2- 172.21.15.28:0/1529952691 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f988406c2d0 0x7f9884150c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 >> 172.21.15.28:0/1529952691 conn(0x7f9884069e80 msgr2=0x7f98840b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.298+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 shutdown_connections 2024-09-13T17:25:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.299+0000 7f988b400640 1 -- 172.21.15.28:0/1529952691 wait complete. 2024-09-13T17:25:35.316 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T17:25:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.472+0000 7fdda5e00640 1 Processor -- start 2024-09-13T17:25:35.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.472+0000 7fdda5e00640 1 -- start start 2024-09-13T17:25:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.473+0000 7fdda5e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fdda0071490 0x7fdda006c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.473+0000 7fdda5e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda006c8b0 con 0x7fdda015a780 2024-09-13T17:25:35.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.473+0000 7fdda5e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda006cb90 con 0x7fdda015ab60 2024-09-13T17:25:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.473+0000 7fdda5e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda00cc980 con 0x7fdda0071490 2024-09-13T17:25:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fdda015a780 0x7fdda0070d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58108/0 (socket says 172.21.15.28:58108) 2024-09-13T17:25:35.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9ea00640 1 -- 172.21.15.28:0/4194433094 learned_addr learned my addr 172.21.15.28:0/4194433094 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9f400640 1 --2- 172.21.15.28:0/4194433094 >> v2:172.21.15.189:3300/0 conn(0x7fdda0071490 0x7fdda006c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580714540 0 0) 0x7fdda006c8b0 con 0x7fdda015a780 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84003520 con 0x7fdda015a780 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 949000989 0 0) 0x7fdd84003520 con 0x7fdda015a780 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 >> v2:172.21.15.189:3300/0 conn(0x7fdda0071490 msgr2=0x7fdda006c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 --2- 172.21.15.28:0/4194433094 >> v2:172.21.15.189:3300/0 conn(0x7fdda0071490 0x7fdda006c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.474+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 >> v1:172.21.15.189:6789/0 conn(0x7fdda015ab60 legacy=0x7fdda006bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda00ccc60 con 0x7fdda015a780 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fdd90001670 con 0x7fdda015a780 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 >> v1:172.21.15.28:6789/0 conn(0x7fdda015a780 legacy=0x7fdda0070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd84003970 0x7fdd84003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 0x7fdd84004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84005040 0x7fdd8400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda00cc980 con 0x7fdd84004370 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8400bfa0 con 0x7fdd84005040 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8400c160 con 0x7fdd84003970 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9f400640 1 --2- 172.21.15.28:0/4194433094 >> v2:172.21.15.189:3300/0 conn(0x7fdda0071490 0x7fdda006c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 0x7fdd84004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9ea00640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd84003970 0x7fdd84003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9f400640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84005040 0x7fdd8400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 -- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd84003970 msgr2=0x7fdd84003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:35.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdd84003970 0x7fdd84003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 -- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84005040 msgr2=0x7fdd8400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84005040 0x7fdd8400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9e000640 1 -- 172.21.15.28:0/4194433094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd84003520 con 0x7fdd84004370 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9f400640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84005040 0x7fdd8400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.475+0000 7fdd9f400640 1 -- 172.21.15.28:0/4194433094 reap_dead start 2024-09-13T17:25:35.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.476+0000 7fdd9e000640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 0x7fdd84004840 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c001860 tx=0x7fdd840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e50024fac4fa9cf1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c02abc0 con 0x7fdd84004370 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c02ad40 con 0x7fdd84004370 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdd9d600640 1 -- 172.21.15.28:0/4194433094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c032680 con 0x7fdd84004370 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 -- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 msgr2=0x7fdd84004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 0x7fdd84004840 secure :-1 s=READY pgs=2856 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c001860 tx=0x7fdd840057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.480 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 -- 172.21.15.28:0/4194433094 shutdown_connections 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 --2- 172.21.15.28:0/4194433094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdd84004370 0x7fdd84004840 unknown :-1 s=CLOSED pgs=2856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 -- 172.21.15.28:0/4194433094 >> 172.21.15.28:0/4194433094 conn(0x7fdda00b0270 msgr2=0x7fdda00b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 -- 172.21.15.28:0/4194433094 shutdown_connections 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.477+0000 7fdda5e00640 1 -- 172.21.15.28:0/4194433094 wait complete. 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.479+0000 7fdda5e00640 1 Processor -- start 2024-09-13T17:25:35.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.479+0000 7fdda5e00640 1 -- start start 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84004370 0x7fdda0142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdda0145790 0x7fdda0143bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda0146440 con 0x7fdda01427b0 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda0142c30 con 0x7fdd84004370 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdda5e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdda00c4bf0 con 0x7fdda0145790 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdd9e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdd9e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58164/0 (socket says 172.21.15.28:58164) 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.480+0000 7fdd9e000640 1 -- 172.21.15.28:0/582369309 learned_addr learned my addr 172.21.15.28:0/582369309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:35.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9ea00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84004370 0x7fdda0142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9f400640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdda0145790 0x7fdda0143bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 -- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdda0145790 msgr2=0x7fdda0143bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdda0145790 0x7fdda0143bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 -- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84004370 msgr2=0x7fdda0142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84004370 0x7fdda0142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 -- 172.21.15.28:0/582369309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda01430f0 con 0x7fdda01427b0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.481+0000 7fdd9e000640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c02a5a0 tx=0x7fdd8c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.482+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c03e650 con 0x7fdda01427b0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.482+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c02abd0 con 0x7fdda01427b0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.482+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd8c03ea10 con 0x7fdda01427b0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.482+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda01433d0 con 0x7fdda01427b0 2024-09-13T17:25:35.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.482+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda0145c10 con 0x7fdda01427b0 2024-09-13T17:25:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.483+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda0142f50 con 0x7fdda01427b0 2024-09-13T17:25:35.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.484+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fdd8c03eb70 con 0x7fdda01427b0 2024-09-13T17:25:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.485+0000 7fdd97400640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 0x7fdd8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.485+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fdd8c0a9c60 con 0x7fdda01427b0 2024-09-13T17:25:35.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.486+0000 7fdd9ea00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 0x7fdd8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.486+0000 7fdd9ea00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 0x7fdd8006d0c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fdd90001050 tx=0x7fdd90000f80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.489+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fdda0142f50 con 0x7fdda01427b0 2024-09-13T17:25:35.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.703+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdda0142f50 con 0x7fdda01427b0 2024-09-13T17:25:35.705 INFO:tasks.ceph.mon.a.smithi028.stderr:2024-09-13T17:25:35.704+0000 7f58f3200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T17:25:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.716+0000 7fdd97400640 1 -- 172.21.15.28:0/582369309 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v196) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdda0142f50 con 0x7fdda01427b0 2024-09-13T17:25:35.717 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T17:25:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 msgr2=0x7fdd8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 0x7fdd8006d0c0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7fdd90001050 tx=0x7fdd90000f80 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 msgr2=0x7fdda0145250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c02a5a0 tx=0x7fdd8c001230 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 shutdown_connections 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fdd8006ac00 0x7fdd8006d0c0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fdda0145790 0x7fdda0143bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdda01427b0 0x7fdda0145250 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 --2- 172.21.15.28:0/582369309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd84004370 0x7fdda0142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 >> 172.21.15.28:0/582369309 conn(0x7fdda00b0270 msgr2=0x7fdda00ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 shutdown_connections 2024-09-13T17:25:35.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.718+0000 7fdda5e00640 1 -- 172.21.15.28:0/582369309 wait complete. 2024-09-13T17:25:35.738 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T17:25:35.774 INFO:tasks.ceph.mds.a.smithi028.stdout:starting mds.a at 2024-09-13T17:25:35.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.896+0000 7f65d7e00640 1 Processor -- start 2024-09-13T17:25:35.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.897+0000 7f65d7e00640 1 -- start start 2024-09-13T17:25:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 0x7f65d8070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 0x7f65d80cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7e00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d8071630 con 0x7f65d8143910 2024-09-13T17:25:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d80cdb10 con 0x7f65d80cd120 2024-09-13T17:25:35.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d8143700 con 0x7f65d8143320 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 0x7f65d80cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 0x7f65d8070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 0x7f65d80cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:56094/0 (socket says 172.21.15.28:56094) 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65d7400640 1 -- 172.21.15.28:0/72926801 learned_addr learned my addr 172.21.15.28:0/72926801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.898+0000 7f65cfe00640 1 --1- 172.21.15.28:0/72926801 >> v1:172.21.15.189:6789/0 conn(0x7f65d8143910 0x7f65d8071ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:59846/0 (socket says 172.21.15.28:59846) 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.899+0000 7f65d6a00640 1 -- 172.21.15.28:0/72926801 >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 msgr2=0x7f65d80cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:35.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.899+0000 7f65d6a00640 1 --2- 172.21.15.28:0/72926801 >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 0x7f65d80cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.899+0000 7f65d6a00640 1 -- 172.21.15.28:0/72926801 >> v1:172.21.15.189:6789/0 conn(0x7f65d8143910 legacy=0x7f65d8071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.899+0000 7f65d6a00640 1 -- 172.21.15.28:0/72926801 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d8070a20 con 0x7f65d8143320 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.900+0000 7f65d6a00640 1 --2- 172.21.15.28:0/72926801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 0x7f65d8070440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097a0 tx=0x7f65c802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=197428decc7e4939 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.900+0000 7f65d6000640 1 -- 172.21.15.28:0/72926801 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8032440 con 0x7f65d8143320 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.900+0000 7f65d6000640 1 -- 172.21.15.28:0/72926801 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c80325a0 con 0x7f65d8143320 2024-09-13T17:25:35.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.900+0000 7f65d6000640 1 -- 172.21.15.28:0/72926801 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8032930 con 0x7f65d8143320 2024-09-13T17:25:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 -- 172.21.15.28:0/72926801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 msgr2=0x7f65d8070440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 --2- 172.21.15.28:0/72926801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 0x7f65d8070440 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f65c80097a0 tx=0x7f65c802a180 comp rx=0 tx=0).stop 2024-09-13T17:25:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 -- 172.21.15.28:0/72926801 shutdown_connections 2024-09-13T17:25:35.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 --2- 172.21.15.28:0/72926801 >> v2:172.21.15.189:3300/0 conn(0x7f65d80cd120 0x7f65d80cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 --2- 172.21.15.28:0/72926801 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d8143320 0x7f65d8070440 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 -- 172.21.15.28:0/72926801 >> 172.21.15.28:0/72926801 conn(0x7f65d8069e80 msgr2=0x7f65d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 -- 172.21.15.28:0/72926801 shutdown_connections 2024-09-13T17:25:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.901+0000 7f65d7e00640 1 -- 172.21.15.28:0/72926801 wait complete. 2024-09-13T17:25:35.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.903+0000 7f65d7e00640 1 Processor -- start 2024-09-13T17:25:35.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.903+0000 7f65d7e00640 1 -- start start 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8143320 0x7f65d81f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d81f3270 con 0x7f65d8143910 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d81f3d00 con 0x7f65d8143320 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d80a8600 con 0x7f65d80cd120 2024-09-13T17:25:35.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58192/0 (socket says 172.21.15.28:58192) 2024-09-13T17:25:35.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d6a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:44988/0 (socket says 172.21.15.28:44988) 2024-09-13T17:25:35.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65d7400640 1 -- 172.21.15.28:0/404223990 learned_addr learned my addr 172.21.15.28:0/404223990 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:35.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.904+0000 7f65cfe00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8143320 0x7f65d81f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 -- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8143320 msgr2=0x7f65d81f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8143320 0x7f65d81f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 -- 172.21.15.28:0/404223990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 msgr2=0x7f65d815cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 -- 172.21.15.28:0/404223990 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8009400 con 0x7f65d80cd120 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d7400640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:35.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65d6a00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65d8143840 tx=0x7f65c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.905+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8032440 con 0x7f65d80cd120 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.906+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c8030b80 con 0x7f65d80cd120 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.906+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d81f3460 con 0x7f65d80cd120 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.906+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c8006e10 con 0x7f65d80cd120 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.906+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d81f3a20 con 0x7f65d80cd120 2024-09-13T17:25:35.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.907+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d80b3800 con 0x7f65d80cd120 2024-09-13T17:25:35.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.908+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f65c803c400 con 0x7f65d80cd120 2024-09-13T17:25:35.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.909+0000 7f65cea00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 0x7f65b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:35.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.909+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5881+0+0 (secure 0 0 0) 0x7f65c80aa180 con 0x7f65d80cd120 2024-09-13T17:25:35.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.909+0000 7f65cfe00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 0x7f65b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:35.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.910+0000 7f65cfe00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 0x7f65b406d0c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f65d8141ab0 tx=0x7f65c0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:35.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:35.914+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f65c8075350 con 0x7f65d80cd120 2024-09-13T17:25:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.130+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f65d8067580 con 0x7f65d80cd120 2024-09-13T17:25:36.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.719+0000 7f65cea00640 1 -- 172.21.15.28:0/404223990 <== mon.2 v2:172.21.15.189:3301/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v197) v1 ==== 104+0+0 (secure 0 0 0) 0x7f65c8078970 con 0x7f65d80cd120 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.722+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 msgr2=0x7f65b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.722+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 0x7f65b406d0c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f65d8141ab0 tx=0x7f65c0007e80 comp rx=0 tx=0).stop 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.722+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 msgr2=0x7f65d80cbc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.722+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f65d8143840 tx=0x7f65c80046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 shutdown_connections 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f65b406ac00 0x7f65b406d0c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f65d8143910 0x7f65d815cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65d8143320 0x7f65d81f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 --2- 172.21.15.28:0/404223990 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f65d80cd120 0x7f65d80cbc60 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 >> 172.21.15.28:0/404223990 conn(0x7f65d8069e80 msgr2=0x7f65d80b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 shutdown_connections 2024-09-13T17:25:36.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.723+0000 7f65d7e00640 1 -- 172.21.15.28:0/404223990 wait complete. 2024-09-13T17:25:36.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T17:25:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.941+0000 7fe220600640 1 Processor -- start 2024-09-13T17:25:36.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.941+0000 7fe220600640 1 -- start start 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.942+0000 7fe220600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159590 0x7fe218070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.942+0000 7fe220600640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe218159970 0x7fe2180b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe220600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe21815a4c0 con 0x7fe2180b2630 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe220600640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2180b2ae0 con 0x7fe218159970 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe220600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe218071580 con 0x7fe218159590 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe218159970 0x7fe2180b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe218159970 0x7fe2180b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58194/0 (socket says 172.21.15.28:58194) 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 -- 172.21.15.28:0/2906889520 learned_addr learned my addr 172.21.15.28:0/2906889520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:36.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21dc00640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159590 0x7fe218070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 -- 172.21.15.28:0/2906889520 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159590 msgr2=0x7fe218070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159590 0x7fe218070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 -- 172.21.15.28:0/2906889520 >> v1:172.21.15.189:6789/0 conn(0x7fe2180b2630 legacy=0x7fe21806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.943+0000 7fe21d200640 1 -- 172.21.15.28:0/2906889520 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe218159e70 con 0x7fe218159970 2024-09-13T17:25:36.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.944+0000 7fe21d200640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218159970 0x7fe2180b20f0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fe204009870 tx=0x7fe20402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5fc5065781699d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe217e00640 1 -- 172.21.15.28:0/2906889520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204032440 con 0x7fe218159970 2024-09-13T17:25:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe217e00640 1 -- 172.21.15.28:0/2906889520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2040325a0 con 0x7fe218159970 2024-09-13T17:25:36.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe217e00640 1 -- 172.21.15.28:0/2906889520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204032930 con 0x7fe218159970 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 -- 172.21.15.28:0/2906889520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218159970 msgr2=0x7fe2180b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218159970 0x7fe2180b20f0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7fe204009870 tx=0x7fe20402a210 comp rx=0 tx=0).stop 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 -- 172.21.15.28:0/2906889520 shutdown_connections 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe218159970 0x7fe2180b20f0 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 --2- 172.21.15.28:0/2906889520 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159590 0x7fe218070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 -- 172.21.15.28:0/2906889520 >> 172.21.15.28:0/2906889520 conn(0x7fe218069e80 msgr2=0x7fe21806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 -- 172.21.15.28:0/2906889520 shutdown_connections 2024-09-13T17:25:36.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.945+0000 7fe220600640 1 -- 172.21.15.28:0/2906889520 wait complete. 2024-09-13T17:25:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.947+0000 7fe220600640 1 Processor -- start 2024-09-13T17:25:36.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.947+0000 7fe220600640 1 -- start start 2024-09-13T17:25:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe218159590 0x7fe21814c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159970 0x7fe21815b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe218147080 con 0x7fe2180b2630 2024-09-13T17:25:36.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe218149f40 con 0x7fe218159590 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe220600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe21814aac0 con 0x7fe218159970 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe21dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe21dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58210/0 (socket says 172.21.15.28:58210) 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe21dc00640 1 -- 172.21.15.28:0/167824816 learned_addr learned my addr 172.21.15.28:0/167824816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe21d200640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe218159590 0x7fe21814c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.948+0000 7fe21e600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159970 0x7fe21815b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 -- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159970 msgr2=0x7fe21815b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159970 0x7fe21815b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 -- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe218159590 msgr2=0x7fe21814c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe218159590 0x7fe21814c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 -- 172.21.15.28:0/167824816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2040094d0 con 0x7fe2180b2630 2024-09-13T17:25:36.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe21dc00640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00d830 tx=0x7fe20c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c00b6c0 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe20c00b820 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20c00bbb0 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21814cce0 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.949+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21814a470 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.951+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21806fb00 con 0x7fe2180b2630 2024-09-13T17:25:36.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.952+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe20c01e460 con 0x7fe2180b2630 2024-09-13T17:25:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.953+0000 7fe215600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 0x7fe1f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.953+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 5881+0+0 (secure 0 0 0) 0x7fe20c089ac0 con 0x7fe2180b2630 2024-09-13T17:25:36.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.954+0000 7fe21d200640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 0x7fe1f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:36.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.954+0000 7fe21d200640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 0x7fe1f806d0c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe204034000 tx=0x7fe20402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:36.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:36.958+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe20c054c00 con 0x7fe2180b2630 2024-09-13T17:25:37.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.135+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe218070c80 con 0x7fe2180b2630 2024-09-13T17:25:37.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.724+0000 7fe215600640 1 -- 172.21.15.28:0/167824816 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v236) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe20c058220 con 0x7fe2180b2630 2024-09-13T17:25:37.725 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.727+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 msgr2=0x7fe1f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.727+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 0x7fe1f806d0c0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fe204034000 tx=0x7fe20402e000 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.727+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 msgr2=0x7fe218146b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.727+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7fe20c00d830 tx=0x7fe20c00dcf0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 shutdown_connections 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe1f806ac00 0x7fe1f806d0c0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe218159970 0x7fe21815b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe218159590 0x7fe21814c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 --2- 172.21.15.28:0/167824816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe2180b2630 0x7fe218146b40 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 >> 172.21.15.28:0/167824816 conn(0x7fe218069e80 msgr2=0x7fe21806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:37.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 shutdown_connections 2024-09-13T17:25:37.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.728+0000 7fe220600640 1 -- 172.21.15.28:0/167824816 wait complete. 2024-09-13T17:25:37.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T17:25:37.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.905+0000 7f9417200640 1 Processor -- start 2024-09-13T17:25:37.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.906+0000 7f9417200640 1 -- start start 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9417200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9418143ea0 0x7f94180b1b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9417200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9418144280 con 0x7f94180cd8a0 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9417200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94180b23e0 con 0x7f941815ebd0 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9417200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94180b3060 con 0x7f9418143ea0 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9415e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f9418143ea0 0x7f94180b1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9416800640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f94180cd8a0 0x7f94180b3480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58122/0 (socket says 172.21.15.28:58122) 2024-09-13T17:25:37.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.907+0000 7f9416800640 1 -- 172.21.15.28:0/3484994830 learned_addr learned my addr 172.21.15.28:0/3484994830 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.908+0000 7f9415400640 1 -- 172.21.15.28:0/3484994830 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057798141 0 0) 0x7f9410001490 con 0x7f94180cd8a0 2024-09-13T17:25:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.908+0000 7f9415400640 1 -- 172.21.15.28:0/3484994830 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93fc003520 con 0x7f94180cd8a0 2024-09-13T17:25:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.908+0000 7f9415e00640 1 -- 172.21.15.28:0/3484994830 >> v1:172.21.15.189:6789/0 conn(0x7f941815ebd0 legacy=0x7f94180cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.908+0000 7f9415e00640 1 -- 172.21.15.28:0/3484994830 >> v1:172.21.15.28:6789/0 conn(0x7f94180cd8a0 legacy=0x7f94180b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.908+0000 7f9415e00640 1 -- 172.21.15.28:0/3484994830 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180b26c0 con 0x7f9418143ea0 2024-09-13T17:25:37.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.909+0000 7f9415e00640 1 --2- 172.21.15.28:0/3484994830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9418143ea0 0x7f94180b1b00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f94080097a0 tx=0x7f940802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f4e405d57bc184 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:37.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.909+0000 7f9415400640 1 -- 172.21.15.28:0/3484994830 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408032440 con 0x7f9418143ea0 2024-09-13T17:25:37.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.909+0000 7f9415400640 1 -- 172.21.15.28:0/3484994830 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94080325a0 con 0x7f9418143ea0 2024-09-13T17:25:37.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.909+0000 7f9415400640 1 -- 172.21.15.28:0/3484994830 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408032930 con 0x7f9418143ea0 2024-09-13T17:25:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 -- 172.21.15.28:0/3484994830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9418143ea0 msgr2=0x7f94180b1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 --2- 172.21.15.28:0/3484994830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9418143ea0 0x7f94180b1b00 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f94080097a0 tx=0x7f940802a1a0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 -- 172.21.15.28:0/3484994830 shutdown_connections 2024-09-13T17:25:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 --2- 172.21.15.28:0/3484994830 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9418143ea0 0x7f94180b1b00 unknown :-1 s=CLOSED pgs=2864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 -- 172.21.15.28:0/3484994830 >> 172.21.15.28:0/3484994830 conn(0x7f9418069e80 msgr2=0x7f941806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 -- 172.21.15.28:0/3484994830 shutdown_connections 2024-09-13T17:25:37.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.910+0000 7f9417200640 1 -- 172.21.15.28:0/3484994830 wait complete. 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.912+0000 7f9417200640 1 Processor -- start 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.912+0000 7f9417200640 1 -- start start 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9418143ea0 0x7f941815abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f941815ebd0 0x7f941815b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94181f3340 con 0x7f94180cd8a0 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94180ca800 con 0x7f9418143ea0 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9417200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f941813c330 con 0x7f941815ebd0 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:37.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58220/0 (socket says 172.21.15.28:58220) 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 -- 172.21.15.28:0/3895796964 learned_addr learned my addr 172.21.15.28:0/3895796964 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f940d400640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9418143ea0 0x7f941815abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9416800640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f941815ebd0 0x7f941815b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 -- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f941815ebd0 msgr2=0x7f941815b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f941815ebd0 0x7f941815b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 -- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9418143ea0 msgr2=0x7f941815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9418143ea0 0x7f941815abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.913+0000 7f9415e00640 1 -- 172.21.15.28:0/3895796964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408009400 con 0x7f94180cd8a0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f9415e00640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f9408035040 tx=0x7f94080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408006e10 con 0x7f94180cd8a0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408032e90 con 0x7f94180cd8a0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94181f3590 con 0x7f94180cd8a0 2024-09-13T17:25:37.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408030be0 con 0x7f94180cd8a0 2024-09-13T17:25:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.914+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94181f3b50 con 0x7f94180cd8a0 2024-09-13T17:25:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.915+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9418067a10 con 0x7f94180cd8a0 2024-09-13T17:25:37.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.916+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f9408039040 con 0x7f94180cd8a0 2024-09-13T17:25:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.917+0000 7f940ea00640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 0x7f93f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.917+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 5443+0+0 (secure 0 0 0) 0x7f94080a9c20 con 0x7f94180cd8a0 2024-09-13T17:25:37.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.917+0000 7f940d400640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 0x7f93f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:37.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.918+0000 7f940d400640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 0x7f93f406d0c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f94181419f0 tx=0x7f9400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:37.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:37.921+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f9408074f80 con 0x7f94180cd8a0 2024-09-13T17:25:38.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.103+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f94181efc00 con 0x7f94180cd8a0 2024-09-13T17:25:38.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.727+0000 7f940ea00640 1 -- 172.21.15.28:0/3895796964 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v237) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9408030d40 con 0x7f94180cd8a0 2024-09-13T17:25:38.729 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-09-13T17:25:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 msgr2=0x7f93f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 0x7f93f406d0c0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f94181419f0 tx=0x7f9400009040 comp rx=0 tx=0).stop 2024-09-13T17:25:38.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 msgr2=0x7f94181ef6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f9408035040 tx=0x7f94080046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 shutdown_connections 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f93f406ac00 0x7f93f406d0c0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f941815ebd0 0x7f941815b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9418143ea0 0x7f941815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 --2- 172.21.15.28:0/3895796964 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f94180cd8a0 0x7f94181ef6c0 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.731+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 >> 172.21.15.28:0/3895796964 conn(0x7f9418069e80 msgr2=0x7f941806f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.732+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 shutdown_connections 2024-09-13T17:25:38.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.732+0000 7f9417200640 1 -- 172.21.15.28:0/3895796964 wait complete. 2024-09-13T17:25:38.751 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T17:25:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.909+0000 7f7047200640 1 Processor -- start 2024-09-13T17:25:38.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.909+0000 7f7047200640 1 -- start start 2024-09-13T17:25:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7047200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f7048069e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7047200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f704806a440 0x7f704806a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7047200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70480ae200 con 0x7f70480b2dc0 2024-09-13T17:25:38.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7047200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704806ae10 con 0x7f704806a440 2024-09-13T17:25:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7047200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70480ae4e0 con 0x7f704815c900 2024-09-13T17:25:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7045400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f70480b2dc0 0x7f70480af2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58130/0 (socket says 172.21.15.28:58130) 2024-09-13T17:25:38.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.910+0000 7f7045400640 1 -- 172.21.15.28:0/3894911552 learned_addr learned my addr 172.21.15.28:0/3894911552 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383600336 0 0) 0x7f70480ae200 con 0x7f70480b2dc0 2024-09-13T17:25:38.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f702c003520 con 0x7f70480b2dc0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7045e00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f7048069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7046800640 1 --2- 172.21.15.28:0/3894911552 >> v2:172.21.15.189:3300/0 conn(0x7f704806a440 0x7f704806a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2257876115 0 0) 0x7f702c003520 con 0x7f70480b2dc0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 msgr2=0x7f7048069e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f7048069e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 >> v2:172.21.15.189:3300/0 conn(0x7f704806a440 msgr2=0x7f704806a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 --2- 172.21.15.28:0/3894911552 >> v2:172.21.15.189:3300/0 conn(0x7f704806a440 0x7f704806a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70480ae7c0 con 0x7f70480b2dc0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7034001670 con 0x7f70480b2dc0 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 >> v1:172.21.15.28:6789/0 conn(0x7f70480b2dc0 legacy=0x7f70480af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f702c003970 0x7f702c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c0047c0 0x7f702c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f702c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704806ae10 con 0x7f702c003970 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70480ae4e0 con 0x7f702c0047c0 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.911+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702c00b980 con 0x7f704815c900 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f702c003970 0x7f702c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045400640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c0047c0 0x7f702c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7046800640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f702c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 -- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 msgr2=0x7f702c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815c900 0x7f702c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 -- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c0047c0 msgr2=0x7f702c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c0047c0 0x7f702c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.912+0000 7f7045e00640 1 -- 172.21.15.28:0/3894911552 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c003520 con 0x7f702c003970 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.913+0000 7f7045e00640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f702c003970 0x7f702c003d90 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f7038009ac0 tx=0x7f702c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d8ae1cffbce761 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.913+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031400 con 0x7f702c003970 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703802d050 con 0x7f702c003970 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7044a00640 1 -- 172.21.15.28:0/3894911552 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7038031780 con 0x7f702c003970 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 -- 172.21.15.28:0/3894911552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f702c003970 msgr2=0x7f702c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 --2- 172.21.15.28:0/3894911552 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f702c003970 0x7f702c003d90 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f7038009ac0 tx=0x7f702c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7046800640 1 -- 172.21.15.28:0/3894911552 reap_dead start 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 -- 172.21.15.28:0/3894911552 shutdown_connections 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 -- 172.21.15.28:0/3894911552 >> 172.21.15.28:0/3894911552 conn(0x7f70480b2150 msgr2=0x7f70480b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:38.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 -- 172.21.15.28:0/3894911552 shutdown_connections 2024-09-13T17:25:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.914+0000 7f7047200640 1 -- 172.21.15.28:0/3894911552 wait complete. 2024-09-13T17:25:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.916+0000 7f7047200640 1 Processor -- start 2024-09-13T17:25:38.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.916+0000 7f7047200640 1 -- start start 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c003970 0x7f70481522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815d900 0x7f70480b5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704815fed0 con 0x7f704815e7d0 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f704815f130 con 0x7f702c003970 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7047200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704815f3f0 con 0x7f704815d900 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.917+0000 7f7046800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58240/0 (socket says 172.21.15.28:58240) 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 -- 172.21.15.28:0/4087307675 learned_addr learned my addr 172.21.15.28:0/4087307675 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:38.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7045400640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815d900 0x7f70480b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7045e00640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c003970 0x7f70481522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 -- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815d900 msgr2=0x7f70480b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815d900 0x7f70480b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 -- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c003970 msgr2=0x7f70481522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c003970 0x7f70481522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 -- 172.21.15.28:0/4087307675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704815f6d0 con 0x7f704815e7d0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f7046800640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f70480d1690 tx=0x7f704000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.918+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f704000e040 con 0x7f704815e7d0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.919+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f704000f040 con 0x7f704815e7d0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.919+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7040015670 con 0x7f704815e7d0 2024-09-13T17:25:38.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.919+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f704815f9b0 con 0x7f704815e7d0 2024-09-13T17:25:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.919+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70480c71b0 con 0x7f704815e7d0 2024-09-13T17:25:38.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.920+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704815cce0 con 0x7f704815e7d0 2024-09-13T17:25:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.921+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f70400157d0 con 0x7f704815e7d0 2024-09-13T17:25:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.921+0000 7f703e000640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 0x7f702806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.921+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7040089740 con 0x7f704815e7d0 2024-09-13T17:25:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.922+0000 7f7045e00640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 0x7f702806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:38.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.923+0000 7f7045e00640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 0x7f702806d0c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f7038033000 tx=0x7f7038008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:38.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:38.926+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f7040054bd0 con 0x7f704815e7d0 2024-09-13T17:25:39.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.105+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f70480b65c0 con 0x7f704815e7d0 2024-09-13T17:25:39.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.110+0000 7f703e000640 1 -- 172.21.15.28:0/4087307675 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v54) v1 ==== 104+0+0 (secure 0 0 0) 0x7f70400581f0 con 0x7f704815e7d0 2024-09-13T17:25:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 msgr2=0x7f702806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 0x7f702806d0c0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f7038033000 tx=0x7f7038008030 comp rx=0 tx=0).stop 2024-09-13T17:25:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 msgr2=0x7f70480b5b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f70480d1690 tx=0x7f704000f580 comp rx=0 tx=0).stop 2024-09-13T17:25:39.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 shutdown_connections 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f702806ac00 0x7f702806d0c0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f704815e7d0 0x7f70480b5b80 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f704815d900 0x7f70480b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 --2- 172.21.15.28:0/4087307675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f702c003970 0x7f70481522d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 >> 172.21.15.28:0/4087307675 conn(0x7f70480b2150 msgr2=0x7f704815b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 shutdown_connections 2024-09-13T17:25:39.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.112+0000 7f7047200640 1 -- 172.21.15.28:0/4087307675 wait complete. 2024-09-13T17:25:39.129 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_hack_allow_loading_invalid_metadata 2024-09-13T17:25:39.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.286+0000 7f096d200640 1 Processor -- start 2024-09-13T17:25:39.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.286+0000 7f096d200640 1 -- start start 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.287+0000 7f096d200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f09680cc880 0x7f09680b06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f096d200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09680b0c30 0x7f09680712d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f096d200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680722e0 con 0x7f09680cc4a0 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f096d200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09680cb3e0 con 0x7f09680b0c30 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f096d200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0968072000 con 0x7f09680cc880 2024-09-13T17:25:39.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f09680b0c30 0x7f09680712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f09680cc4a0 0x7f09680cb870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58140/0 (socket says 172.21.15.28:58140) 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967400640 1 -- 172.21.15.28:0/2782837213 learned_addr learned my addr 172.21.15.28:0/2782837213 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f095ea00640 1 --2- 172.21.15.28:0/2782837213 >> v2:172.21.15.189:3300/0 conn(0x7f09680cc880 0x7f09680b06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967e00640 1 -- 172.21.15.28:0/2782837213 >> v2:172.21.15.189:3300/0 conn(0x7f09680cc880 msgr2=0x7f09680b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967e00640 1 --2- 172.21.15.28:0/2782837213 >> v2:172.21.15.189:3300/0 conn(0x7f09680cc880 0x7f09680b06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967e00640 1 -- 172.21.15.28:0/2782837213 >> v1:172.21.15.28:6789/0 conn(0x7f09680cc4a0 legacy=0x7f09680cb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0967e00640 1 -- 172.21.15.28:0/2782837213 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680cad90 con 0x7f09680b0c30 2024-09-13T17:25:39.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.288+0000 7f0966a00640 1 -- 172.21.15.28:0/2782837213 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659775974 0 0) 0x7f09680722e0 con 0x7f09680cc4a0 2024-09-13T17:25:39.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.289+0000 7f0967e00640 1 --2- 172.21.15.28:0/2782837213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680b0c30 0x7f09680712d0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f09680b3b70 tx=0x7f096002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=165d43f930beab37 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f0966a00640 1 -- 172.21.15.28:0/2782837213 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960007740 con 0x7f09680b0c30 2024-09-13T17:25:39.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f0966a00640 1 -- 172.21.15.28:0/2782837213 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096002d050 con 0x7f09680b0c30 2024-09-13T17:25:39.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f0966a00640 1 -- 172.21.15.28:0/2782837213 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0960007ac0 con 0x7f09680b0c30 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 -- 172.21.15.28:0/2782837213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680b0c30 msgr2=0x7f09680712d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 --2- 172.21.15.28:0/2782837213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680b0c30 0x7f09680712d0 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7f09680b3b70 tx=0x7f096002a5f0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 -- 172.21.15.28:0/2782837213 shutdown_connections 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 --2- 172.21.15.28:0/2782837213 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680b0c30 0x7f09680712d0 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 --2- 172.21.15.28:0/2782837213 >> v2:172.21.15.189:3300/0 conn(0x7f09680cc880 0x7f09680b06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 -- 172.21.15.28:0/2782837213 >> 172.21.15.28:0/2782837213 conn(0x7f0968069e80 msgr2=0x7f096806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 -- 172.21.15.28:0/2782837213 shutdown_connections 2024-09-13T17:25:39.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.290+0000 7f096d200640 1 -- 172.21.15.28:0/2782837213 wait complete. 2024-09-13T17:25:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.292+0000 7f096d200640 1 Processor -- start 2024-09-13T17:25:39.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.292+0000 7f096d200640 1 -- start start 2024-09-13T17:25:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09680b0c30 0x7f096815b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 0x7f09680a8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0968157c90 con 0x7f09680cc4a0 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096815be60 con 0x7f09680cc880 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f096d200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0968156410 con 0x7f09680b0c30 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f095ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f095ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58258/0 (socket says 172.21.15.28:58258) 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.293+0000 7f095ea00640 1 -- 172.21.15.28:0/1456804258 learned_addr learned my addr 172.21.15.28:0/1456804258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f0967400640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09680b0c30 0x7f096815b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f0967e00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 0x7f09680a8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 -- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09680b0c30 msgr2=0x7f096815b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09680b0c30 0x7f096815b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 -- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 msgr2=0x7f09680a8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 0x7f09680a8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 -- 172.21.15.28:0/1456804258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0960009a70 con 0x7f09680cc4a0 2024-09-13T17:25:39.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f0967e00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 0x7f09680a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095ea00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f095000e870 tx=0x7f095000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500094f0 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095000c940 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095000ccf0 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.294+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09681578f0 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.295+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f096815c330 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.295+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0968157dd0 con 0x7f09680cc4a0 2024-09-13T17:25:39.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.296+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0950013470 con 0x7f09680cc4a0 2024-09-13T17:25:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.297+0000 7f095fe00640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 0x7f094406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.297+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0950089550 con 0x7f09680cc4a0 2024-09-13T17:25:39.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.298+0000 7f0967400640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 0x7f094406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.299+0000 7f0967400640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 0x7f094406d0c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0958000f60 tx=0x7f0958007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.302+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0968157dd0 con 0x7f09680cc4a0 2024-09-13T17:25:39.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.491+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_hack_allow_loading_invalid_metadata"} v 0) v1 -- 0x7f0968157dd0 con 0x7f09680cc4a0 2024-09-13T17:25:39.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.495+0000 7f095fe00640 1 -- 172.21.15.28:0/1456804258 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_hack_allow_loading_invalid_metadata"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7f09500549e0 con 0x7f09680cc4a0 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.497+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 msgr2=0x7f094406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.497+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 0x7f094406d0c0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f0958000f60 tx=0x7f0958007d50 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.497+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 msgr2=0x7f096814bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.497+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 secure :-1 s=READY pgs=2871 cs=0 l=1 rev1=1 crypto rx=0x7f095000e870 tx=0x7f095000ed30 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 shutdown_connections 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f094406ac00 0x7f094406d0c0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09680cc880 0x7f09680a8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09680cc4a0 0x7f096814bd40 unknown :-1 s=CLOSED pgs=2871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 --2- 172.21.15.28:0/1456804258 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f09680b0c30 0x7f096815b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 >> 172.21.15.28:0/1456804258 conn(0x7f0968069e80 msgr2=0x7f09680cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 shutdown_connections 2024-09-13T17:25:39.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.498+0000 7f096d200640 1 -- 172.21.15.28:0/1456804258 wait complete. 2024-09-13T17:25:39.514 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode' 2024-09-13T17:25:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.669+0000 7f4289200640 1 Processor -- start 2024-09-13T17:25:39.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.669+0000 7f4289200640 1 -- start start 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f4289200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 0x7f428415b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f4289200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428415bb10 con 0x7f428406b360 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f4289200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f428406b900 con 0x7f42840b0b50 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f4289200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428406bbe0 con 0x7f4284154a90 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f427be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f428406b360 0x7f42840b0440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58152/0 (socket says 172.21.15.28:58152) 2024-09-13T17:25:39.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.670+0000 7f427be00640 1 -- 172.21.15.28:0/3632284214 learned_addr learned my addr 172.21.15.28:0/3632284214 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:39.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4283400640 1 --2- 172.21.15.28:0/3632284214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 0x7f428415b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692188659 0 0) 0x7f428415bb10 con 0x7f428406b360 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4268003520 con 0x7f428406b360 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1079395475 0 0) 0x7f428406b900 con 0x7f42840b0b50 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 --> v1:172.21.15.189:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f428415bb10 con 0x7f42840b0b50 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4283400640 1 -- 172.21.15.28:0/3632284214 >> v1:172.21.15.189:6789/0 conn(0x7f42840b0b50 legacy=0x7f42840af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4283400640 1 -- 172.21.15.28:0/3632284214 >> v1:172.21.15.28:6789/0 conn(0x7f428406b360 legacy=0x7f42840b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.671+0000 7f4283400640 1 -- 172.21.15.28:0/3632284214 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f428406bec0 con 0x7f4284154a90 2024-09-13T17:25:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.672+0000 7f4283400640 1 --2- 172.21.15.28:0/3632284214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 0x7f428415b5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42740097d0 tx=0x7f427402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbd7ee186b44be2a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274032440 con 0x7f4284154a90 2024-09-13T17:25:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42740325a0 con 0x7f4284154a90 2024-09-13T17:25:39.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4282a00640 1 -- 172.21.15.28:0/3632284214 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4274032930 con 0x7f4284154a90 2024-09-13T17:25:39.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 -- 172.21.15.28:0/3632284214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 msgr2=0x7f428415b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 --2- 172.21.15.28:0/3632284214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 0x7f428415b5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f42740097d0 tx=0x7f427402a210 comp rx=0 tx=0).stop 2024-09-13T17:25:39.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 -- 172.21.15.28:0/3632284214 shutdown_connections 2024-09-13T17:25:39.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 --2- 172.21.15.28:0/3632284214 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f4284154a90 0x7f428415b5d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 -- 172.21.15.28:0/3632284214 >> 172.21.15.28:0/3632284214 conn(0x7f4284072e40 msgr2=0x7f4284073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:39.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.673+0000 7f4289200640 1 -- 172.21.15.28:0/3632284214 shutdown_connections 2024-09-13T17:25:39.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.674+0000 7f4289200640 1 -- 172.21.15.28:0/3632284214 wait complete. 2024-09-13T17:25:39.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.675+0000 7f4289200640 1 Processor -- start 2024-09-13T17:25:39.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.675+0000 7f4289200640 1 -- start start 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42840b0b50 0x7f42841575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 0x7f42840a8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42840b0240 con 0x7f428406b360 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42841f3cf0 con 0x7f4284154a90 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4289200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840a8fb0 con 0x7f42840b0b50 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4283400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4283400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58270/0 (socket says 172.21.15.28:58270) 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.676+0000 7f4283400640 1 -- 172.21.15.28:0/3294595898 learned_addr learned my addr 172.21.15.28:0/3294595898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f427be00640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42840b0b50 0x7f42841575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283e00640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 0x7f42840a8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 -- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42840b0b50 msgr2=0x7f42841575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42840b0b50 0x7f42841575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 -- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 msgr2=0x7f42840a8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 0x7f42840a8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 -- 172.21.15.28:0/3294595898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4274009400 con 0x7f428406b360 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283e00640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 0x7f42840a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4283400640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f4274035040 tx=0x7f42740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427403e920 con 0x7f428406b360 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427403ea80 con 0x7f428406b360 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427403ee30 con 0x7f428406b360 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.677+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42840a9290 con 0x7f428406b360 2024-09-13T17:25:39.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.678+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42840ac050 con 0x7f428406b360 2024-09-13T17:25:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.679+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42840ac1e0 con 0x7f428406b360 2024-09-13T17:25:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.679+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f427403c400 con 0x7f428406b360 2024-09-13T17:25:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.680+0000 7f427b400640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 0x7f426006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.680+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f42740a9fc0 con 0x7f428406b360 2024-09-13T17:25:39.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.681+0000 7f427be00640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 0x7f426006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:39.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.681+0000 7f427be00640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 0x7f426006d0c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f428413ca80 tx=0x7f426c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:39.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.684+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f42740753a0 con 0x7f428406b360 2024-09-13T17:25:39.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:39.929+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode"]} v 0) v1 -- 0x7f42840a94b0 con 0x7f428406b360 2024-09-13T17:25:40.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.730+0000 7f427b400640 1 -- 172.21.15.28:0/3294595898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_scrub.TestScrub.test_scrub_dup_inode"]}]=0 v850) v1 ==== 133+0+0 (secure 0 0 0) 0x7f42740789c0 con 0x7f428406b360 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 msgr2=0x7f426006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 0x7f426006d0c0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f428413ca80 tx=0x7f426c009040 comp rx=0 tx=0).stop 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 msgr2=0x7f42840a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f4274035040 tx=0x7f42740046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 shutdown_connections 2024-09-13T17:25:40.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f426006ac00 0x7f426006d0c0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4284154a90 0x7f42840a8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f42840b0b50 0x7f42841575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 --2- 172.21.15.28:0/3294595898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f428406b360 0x7f42840a9d50 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 >> 172.21.15.28:0/3294595898 conn(0x7f4284072e40 msgr2=0x7f428415aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 shutdown_connections 2024-09-13T17:25:40.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:40.733+0000 7f4289200640 1 -- 172.21.15.28:0/3294595898 wait complete. 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner:test_scrub_dup_inode (tasks.cephfs.test_scrub.TestScrub) ... ok 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner: 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner:Ran 15 tests in 1016.560s 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner: 2024-09-13T17:25:40.752 INFO:tasks.cephfs_test_runner:OK 2024-09-13T17:25:40.753 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-13T17:25:40.763 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-09-13T17:25:40.774 INFO:tasks.kclient:Unmounting kernel clients... 2024-09-13T17:25:40.774 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:25:40.774 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:40.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:25:40.813 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-09-13T17:25:40.813 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:40.890 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053402,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249604k,nr_inodes=812401,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499212k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T17:25:40.890 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:25:40.890 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:40.930 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:25:40.931 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-09-13T17:25:40.931 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T17:25:41.009 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4052629,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248984k,nr_inodes=812246,mode=700,uid=1000,gid=1333 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497976k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T17:25:41.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.009 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:25:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T17:25:41.047 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T17:25:41.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T17:25:41.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.148 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T17:25:41.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.404 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T17:25:41.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.709 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-09-13T17:25:41.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.746 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-09-13T17:25:41.846 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-09-13T17:25:41.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:25:41.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.850 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:25:41.890 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T17:25:41.890 INFO:teuthology.orchestra.run.smithi189.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T17:25:41.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.891 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T17:25:41.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:41.992 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T17:25:42.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:42.252 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-09-13T17:25:42.387 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:25:42.392 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:25:42.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:42.557 DEBUG:teuthology.orchestra.run.smithi189:> ip netns list 2024-09-13T17:25:42.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T17:25:42.593 DEBUG:teuthology.orchestra.run.smithi189:> sudo ip link delete ceph-brx 2024-09-13T17:25:42.693 INFO:teuthology.orchestra.run.smithi189.stderr:Cannot find device "ceph-brx" 2024-09-13T17:25:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:25:42.696 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T17:25:42.705 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T17:25:42.705 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:25:42.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.861+0000 7f3f3d000640 1 Processor -- start 2024-09-13T17:25:42.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.861+0000 7f3f3d000640 1 -- start start 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f3d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f3815ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f3d000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3f3806c2d0 0x7f3f3806c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f3d000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f38159d90 con 0x7f3f3806f970 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f3d000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f3815b0b0 con 0x7f3f3806c2d0 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f3d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3806ccc0 con 0x7f3f381592a0 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f35400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3f3806f970 0x7f3f38159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58160/0 (socket says 172.21.15.28:58160) 2024-09-13T17:25:42.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.862+0000 7f3f35400640 1 -- 172.21.15.28:0/2433227067 learned_addr learned my addr 172.21.15.28:0/2433227067 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:42.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f35e00640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f3815ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f36800640 1 --2- 172.21.15.28:0/2433227067 >> v2:172.21.15.189:3300/0 conn(0x7f3f3806c2d0 0x7f3f3806c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f34a00640 1 -- 172.21.15.28:0/2433227067 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137840212 0 0) 0x7f3f38159d90 con 0x7f3f3806f970 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f34a00640 1 -- 172.21.15.28:0/2433227067 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c003520 con 0x7f3f3806f970 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f36800640 1 -- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 msgr2=0x7f3f3815ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f36800640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f3815ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f36800640 1 -- 172.21.15.28:0/2433227067 >> v1:172.21.15.28:6789/0 conn(0x7f3f3806f970 legacy=0x7f3f38159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f36800640 1 -- 172.21.15.28:0/2433227067 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f380b2040 con 0x7f3f3806c2d0 2024-09-13T17:25:42.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.863+0000 7f3f35e00640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f3815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.864+0000 7f3f36800640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f3806c780 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009d90 tx=0x7f3f3002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b97239a5101d9b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.864+0000 7f3f34a00640 1 -- 172.21.15.28:0/2433227067 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30007740 con 0x7f3f3806c2d0 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.864+0000 7f3f34a00640 1 -- 172.21.15.28:0/2433227067 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3002d050 con 0x7f3f3806c2d0 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.864+0000 7f3f34a00640 1 -- 172.21.15.28:0/2433227067 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f30007aa0 con 0x7f3f3806c2d0 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 -- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 msgr2=0x7f3f3806c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f3806c780 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f3f30009d90 tx=0x7f3f3002a660 comp rx=0 tx=0).stop 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 -- 172.21.15.28:0/2433227067 shutdown_connections 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f3806c780 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 --2- 172.21.15.28:0/2433227067 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f3815ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 -- 172.21.15.28:0/2433227067 >> 172.21.15.28:0/2433227067 conn(0x7f3f38069e80 msgr2=0x7f3f3806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 -- 172.21.15.28:0/2433227067 shutdown_connections 2024-09-13T17:25:42.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.865+0000 7f3f3d000640 1 -- 172.21.15.28:0/2433227067 wait complete. 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.867+0000 7f3f3d000640 1 Processor -- start 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.867+0000 7f3f3d000640 1 -- start start 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f38166d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f380af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380b04e0 con 0x7f3f3806f970 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3814a410 con 0x7f3f3806c2d0 2024-09-13T17:25:42.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f3d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f380b0200 con 0x7f3f381592a0 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58278/0 (socket says 172.21.15.28:58278) 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 -- 172.21.15.28:0/860236359 learned_addr learned my addr 172.21.15.28:0/860236359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35e00640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f38166d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f36800640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f380af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 -- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 msgr2=0x7f3f380af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f380af690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 -- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 msgr2=0x7f3f38166d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f38166d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.868+0000 7f3f35400640 1 -- 172.21.15.28:0/860236359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f28009400 con 0x7f3f3806f970 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f35400640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f3f2400c3e0 tx=0x7f3f2400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:42.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f24005ce0 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f24005e40 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f240072f0 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f30009a70 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.869+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f38166570 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.870+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f380b3740 con 0x7f3f3806f970 2024-09-13T17:25:42.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.871+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3f24007510 con 0x7f3f3806f970 2024-09-13T17:25:42.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.872+0000 7f3f2e000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 0x7f3f1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:42.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.872+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3f24088650 con 0x7f3f3806f970 2024-09-13T17:25:42.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.872+0000 7f3f35e00640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 0x7f3f1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:42.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.873+0000 7f3f35e00640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 0x7f3f1806d0c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009c30 tx=0x7f3f28009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:42.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:42.876+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3f24053b80 con 0x7f3f3806f970 2024-09-13T17:25:43.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.051+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3f3815a5b0 con 0x7f3f1806ac00 2024-09-13T17:25:43.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.053+0000 7f3f2e000640 1 -- 172.21.15.28:0/860236359 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50674 (secure 0 0 0) 0x7f3f28009880 con 0x7f3f1806ac00 2024-09-13T17:25:43.055 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:43.055 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.055+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 msgr2=0x7f3f1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.055+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 0x7f3f1806d0c0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009c30 tx=0x7f3f28009190 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.055+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 msgr2=0x7f3f3814c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.055+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7f3f2400c3e0 tx=0x7f3f2400c8a0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 shutdown_connections 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3f1806ac00 0x7f3f1806d0c0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3f381592a0 0x7f3f380af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f3806f970 0x7f3f3814c250 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 --2- 172.21.15.28:0/860236359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f3806c2d0 0x7f3f38166d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 >> 172.21.15.28:0/860236359 conn(0x7f3f38069e80 msgr2=0x7f3f3806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 shutdown_connections 2024-09-13T17:25:43.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.056+0000 7f3f3d000640 1 -- 172.21.15.28:0/860236359 wait complete. 2024-09-13T17:25:43.073 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-09-13T17:25:41.152806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":584712,"kb_used_data":49064,"kb_used_omap":12,"kb_used_meta":535539,"kb_avail":754390008,"statfs":{"total":773094113280,"available":772495368192,"internally_reserved":0,"allocated":50241536,"data_stored":36955556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":548392505},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12624397,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-99,"num_read_kb":-105,"num_write":-94,"num_write_kb":-12360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001276"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":349,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-13T17:24:22.655571+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:24:22.655571+0000","last_peered":"2024-09-13T17:24:22.655571+0000","last_clean":"2024-09-13T17:24:22.655571+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:24:22.655571+0000","last_undegraded":"2024-09-13T17:24:22.655571+0000","last_fullsized":"2024-09-13T17:24:22.655571+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575063,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70632,"kb_used_data":3484,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301208,"statfs":{"total":96636764160,"available":96564436992,"internally_reserved":0,"allocated":3567616,"data_stored":1906446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33800000000000002},"max":{"1min":0.87,"5min":1.008,"15min":2.0880000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.57599999999999996,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":0.93700000000000006,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.91100000000000003,"5min":1.161,"15min":2.5030000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.65000000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":1.399,"15min":6.4909999999999997},"last":0.82499999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71899999999999997,"15min":0.70299999999999996},"min":{"1min":0.58899999999999997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":0.90300000000000002,"5min":1.105,"15min":1.544},"last":0.747},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.70599999999999996},"min":{"1min":0.61799999999999999,"5min":0.42399999999999999,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.76600000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.45400000000000001,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.437},"max":{"1min":0.91200000000000003,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.71399999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":1.2170000000000001,"15min":1.234},"last":0.57299999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.43099999999999999,"5min":0.42699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":1.143},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.57399999999999995,"5min":0.48499999999999999,"15min":0.44800000000000001},"max":{"1min":0.79000000000000004,"5min":1.915,"15min":1.915},"last":0.79100000000000004}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.49299999999999999,"5min":0.41799999999999998,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.607,"15min":1.607},"last":0.73099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834575064,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78248,"kb_used_data":11100,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94293592,"statfs":{"total":96636764160,"available":96556638208,"internally_reserved":0,"allocated":11366400,"data_stored":9704728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70999999999999996,"15min":0.67600000000000005},"min":{"1min":0.58399999999999996,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":0.96399999999999997,"5min":0.97099999999999997,"15min":1.298},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.67000000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":5.9800000000000004},"last":0.81999999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.373,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.998,"15min":2.3100000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.57499999999999996,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90700000000000003,"15min":1.46},"last":0.95299999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.60899999999999999,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.89800000000000002,"5min":1.107,"15min":5.9240000000000004},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71099999999999997,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.872,"5min":0.92700000000000005,"15min":1.012},"last":0.91400000000000003}]},{"osd":3,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73599999999999999,"15min":0.72199999999999998},"min":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":1.151,"15min":3.6520000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72499999999999998,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92400000000000004,"5min":1.085,"15min":3.6720000000000002},"last":0.84299999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.33800000000000002},"max":{"1min":0.92600000000000005,"5min":1.262,"15min":1.6439999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.075},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.374},"max":{"1min":1.0209999999999999,"5min":1.202,"15min":1.8260000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.47699999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":0.90200000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.504}]},{"osd":7,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.46999999999999997,"5min":0.379,"15min":0.378},"max":{"1min":1.052,"5min":1.163,"15min":1.601},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79400000000000004,"15min":0.78000000000000003},"min":{"1min":0.66600000000000004,"5min":0.58999999999999997,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.5720000000000001},"last":0.68799999999999994}]}]},{"osd":5,"up_from":13,"seq":55834575063,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47968,"kb_used_data":2900,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323872,"statfs":{"total":96636764160,"available":96587644928,"internally_reserved":0,"allocated":2969600,"data_stored":1316008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.76800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91200000000000003,"5min":1.1220000000000001,"15min":38.005000000000003},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.41099999999999998,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.48899999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.69299999999999995,"15min":0.70199999999999996},"min":{"1min":0.58299999999999996,"5min":0.44700000000000001,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.169,"15min":7.7370000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.379},"max":{"1min":0.93000000000000005,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98599999999999999,"15min":9.6549999999999994},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.72499999999999998,"15min":1.0820000000000001},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.435},"max":{"1min":0.98099999999999998,"5min":1.042,"15min":113.363},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.61899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":4.1180000000000003},"last":0.77400000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":1.0760000000000001,"15min":2.0590000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":2.827},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.68600000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.34799999999999998},"max":{"1min":0.85199999999999998,"5min":1.0609999999999999,"15min":2.2320000000000002},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.71299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":1.032,"15min":4.1500000000000004},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.746,"15min":0.755},"min":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.40999999999999998},"max":{"1min":0.98199999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.748,"15min":0.75700000000000001},"min":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.44},"max":{"1min":0.95799999999999996,"5min":1.169,"15min":2.02},"last":0.84299999999999997}]}]},{"osd":4,"up_from":13,"seq":55834575062,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78296,"kb_used_data":7052,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94293544,"statfs":{"total":96636764160,"available":96556589056,"internally_reserved":0,"allocated":7221248,"data_stored":5551055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.26,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.379,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2350000000000001,"15min":0.88400000000000001},"min":{"1min":0.63100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.77600000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.88600000000000001,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.40899999999999997,"15min":0.35099999999999998},"max":{"1min":0.94399999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.95999999999999996,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.72699999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72999999999999998,"15min":0.68600000000000005},"min":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.98599999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.74099999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.1259999999999999,"15min":0.83599999999999997},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.378},"max":{"1min":0.996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.73399999999999999},"min":{"1min":0.499,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.046,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.89400000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":0.76000000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.51000000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65700000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.32500000000000001},"max":{"1min":0.85499999999999998,"5min":2.976,"15min":2.976},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.75,"15min":0.73199999999999998},"min":{"1min":0.58099999999999996,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79600000000000004,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.753,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96599999999999997,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.75900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575063,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82388,"kb_used_data":11144,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94289452,"statfs":{"total":96636764160,"available":96552398848,"internally_reserved":0,"allocated":11411456,"data_stored":9745869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.65200000000000002,"5min":0.76400000000000001,"15min":0.97099999999999997},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.32900000000000001},"max":{"1min":0.872,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.439}]},{"osd":2,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":1.4930000000000001,"15min":1.9590000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34499999999999997},"max":{"1min":0.79700000000000004,"5min":0.875,"15min":1.3819999999999999},"last":0.40500000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.748},"min":{"1min":0.63900000000000001,"5min":0.45600000000000002,"15min":0.39700000000000002},"max":{"1min":1.05,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.498},"max":{"1min":0.99399999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.54600000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.76900000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0389999999999999,"5min":1.228,"15min":1.391},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.76400000000000001},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.442},"max":{"1min":1.022,"5min":2.347,"15min":2.347},"last":0.65900000000000003}]},{"osd":6,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80400000000000005,"15min":0.78000000000000003},"min":{"1min":0.63,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.81000000000000005,"15min":0.79000000000000004},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46300000000000002},"max":{"1min":1.069,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.621}]},{"osd":7,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.81399999999999995,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.42699999999999999},"max":{"1min":1.036,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83799999999999997,"15min":0.80800000000000005},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.47599999999999998},"max":{"1min":1.0529999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.68600000000000005}]}]},{"osd":2,"up_from":13,"seq":55834575063,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52060,"kb_used_data":2896,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94319780,"statfs":{"total":96636764160,"available":96583454720,"internally_reserved":0,"allocated":2965504,"data_stored":1314989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":5.6769999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.58699999999999997},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.27100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":30.023},"last":0.315}]},{"osd":1,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":1.3720000000000001},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.307},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":260.53399999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":1.4710000000000001},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.94099999999999995,"15min":262.61399999999998},"last":0.40999999999999998}]},{"osd":3,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51000000000000001,"15min":0.624},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.84199999999999997,"15min":27.271999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.61799999999999999},"min":{"1min":0.52400000000000002,"5min":0.45000000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91100000000000003,"15min":8.4149999999999991},"last":0.441}]},{"osd":4,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78200000000000003,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.53000000000000003,"15min":0.45500000000000002},"max":{"1min":1.331,"5min":1.331,"15min":10.598000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.78600000000000003,"15min":0.86599999999999999},"min":{"1min":0.60899999999999999,"5min":0.51200000000000001,"15min":0.371},"max":{"1min":1.1100000000000001,"5min":1.264,"15min":21.213000000000001},"last":0.621}]},{"osd":5,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.78400000000000003,"15min":0.88400000000000001},"min":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.40300000000000002},"max":{"1min":1.198,"5min":1.198,"15min":27.135000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80000000000000004,"15min":0.79100000000000004},"min":{"1min":0.63100000000000001,"5min":0.52400000000000002,"15min":0.47599999999999998},"max":{"1min":1.167,"5min":1.167,"15min":5.8890000000000002},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.79900000000000004,"15min":0.83599999999999997},"min":{"1min":0.68000000000000005,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":10.664},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78200000000000003,"15min":0.89100000000000001},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.222,"5min":1.23,"15min":27.236999999999998},"last":0.58999999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.68100000000000005,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.369,"5min":1.369,"15min":10.712999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80300000000000005,"15min":0.91000000000000003},"min":{"1min":0.64400000000000002,"5min":0.52300000000000002,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.1799999999999999,"15min":27.199999999999999},"last":0.76000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96304,"kb_used_data":2916,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275536,"statfs":{"total":96636764160,"available":96538148864,"internally_reserved":0,"allocated":2985984,"data_stored":1316800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23400000000000001},"max":{"1min":0.61099999999999999,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.47899999999999998,"5min":0.35999999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":6.7240000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64300000000000002,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.308,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.65300000000000002},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54200000000000004,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":18.318000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.28,"15min":6.1630000000000003},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.67000000000000004,"5min":0.56299999999999994,"15min":0.42599999999999999},"max":{"1min":1.0129999999999999,"5min":1.23,"15min":1.78},"last":0.96299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.0389999999999999,"5min":1.159,"15min":1.6020000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.498,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77900000000000003},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.2330000000000001,"15min":1.6599999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.65300000000000002,"5min":0.503,"15min":0.41399999999999998},"max":{"1min":1.079,"5min":1.246,"15min":6.1970000000000001},"last":0.90500000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.79100000000000004},"min":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.46600000000000003},"max":{"1min":1.0209999999999999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81100000000000005},"min":{"1min":0.64300000000000002,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.097,"5min":1.2609999999999999,"15min":1.544},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575064,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78816,"kb_used_data":7572,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94293024,"statfs":{"total":96636764160,"available":96556056576,"internally_reserved":0,"allocated":7753728,"data_stored":6099661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.433,"15min":0.442},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":0.72599999999999998,"15min":1.986},"last":0.373},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":0.73399999999999999,"5min":0.81599999999999995,"15min":3.004},"last":0.33200000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":0.877,"15min":3.605},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.45800000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":2.4039999999999999},"last":0.40699999999999997}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":2.6400000000000001},"last":0.314},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":3.7709999999999999},"last":0.44800000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.65400000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.034,"5min":1.1599999999999999,"15min":3.8849999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.64400000000000002,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.002,"5min":1.2969999999999999,"15min":2.6760000000000002},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.44500000000000001},"max":{"1min":1.1299999999999999,"5min":1.1799999999999999,"15min":2.7320000000000002},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80500000000000005,"15min":0.79100000000000004},"min":{"1min":0.65900000000000003,"5min":0.51800000000000002,"15min":0.432},"max":{"1min":1.006,"5min":1.276,"15min":1.8740000000000001},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.70999999999999996,"5min":0.52600000000000002,"15min":0.46700000000000003},"max":{"1min":1.0349999999999999,"5min":1.1619999999999999,"15min":1.242},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.70199999999999996,"5min":0.50600000000000001,"15min":0.48599999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":2.8380000000000001},"last":0.72599999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.73899999999999999,"5min":0.52100000000000002,"15min":0.47399999999999998},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":3.9209999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0760000000000001,"5min":1.242,"15min":2.706},"last":0.67600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:25:43.075 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.234+0000 7f4616600640 1 Processor -- start 2024-09-13T17:25:43.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.234+0000 7f4616600640 1 -- start start 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4616600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f46101602a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4616600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f461006bde0 con 0x7f4610156a20 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4616600640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46100b3a80 con 0x7f4610156e00 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4616600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461006b820 con 0x7f46100b3620 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4615200640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f4610156a20 0x7f4610071dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58174/0 (socket says 172.21.15.28:58174) 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.235+0000 7f4615200640 1 -- 172.21.15.28:0/4249265291 learned_addr learned my addr 172.21.15.28:0/4249265291 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f4615c00640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f46101602a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377336583 0 0) 0x7f461006bde0 con 0x7f4610156a20 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f8003520 con 0x7f4610156a20 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3066006681 0 0) 0x7f45f8003520 con 0x7f4610156a20 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 msgr2=0x7f46101602a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f46101602a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 >> v1:172.21.15.189:6789/0 conn(0x7f4610156e00 legacy=0x7f461015d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461006bb00 con 0x7f4610156a20 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f4604001670 con 0x7f4610156a20 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 >> v1:172.21.15.28:6789/0 conn(0x7f4610156a20 legacy=0x7f4610071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f45f80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f8003f20 0x7f45f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f8004d00 0x7f45f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461006b820 con 0x7f45f8004d00 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f800ba60 con 0x7f45f8003f20 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.236+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f800bc20 con 0x7f46100b3620 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f45f80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f460fe00640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f8003f20 0x7f45f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615c00640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f8004d00 0x7f45f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 -- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f8003f20 msgr2=0x7f45f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f8003f20 0x7f45f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 -- 172.21.15.28:0/4249265291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f8004d00 msgr2=0x7f45f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:43.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45f8004d00 0x7f45f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.237+0000 7f4615200640 1 -- 172.21.15.28:0/4249265291 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f8003520 con 0x7f46100b3620 2024-09-13T17:25:43.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.238+0000 7f4615200640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f45f80039a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f460400c240 tx=0x7f45f80057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abb823e3dd9ac93d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.238+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460402e070 con 0x7f46100b3620 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.238+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46040058d0 con 0x7f46100b3620 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.238+0000 7f460f400640 1 -- 172.21.15.28:0/4249265291 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604005d00 con 0x7f46100b3620 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 -- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 msgr2=0x7f45f80039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 --2- 172.21.15.28:0/4249265291 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b3620 0x7f45f80039a0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f460400c240 tx=0x7f45f80057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4615c00640 1 -- 172.21.15.28:0/4249265291 reap_dead start 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 -- 172.21.15.28:0/4249265291 shutdown_connections 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 -- 172.21.15.28:0/4249265291 >> 172.21.15.28:0/4249265291 conn(0x7f461006d4e0 msgr2=0x7f461006d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 -- 172.21.15.28:0/4249265291 shutdown_connections 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.239+0000 7f4616600640 1 -- 172.21.15.28:0/4249265291 wait complete. 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.241+0000 7f4616600640 1 Processor -- start 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.241+0000 7f4616600640 1 -- start start 2024-09-13T17:25:43.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46100b3620 0x7f4610147fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b19d0 0x7f4610158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46100b3050 con 0x7f4610152130 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46100b2840 con 0x7f46100b3620 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4616600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46100b2a90 con 0x7f46100b19d0 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f460fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f460fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58298/0 (socket says 172.21.15.28:58298) 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f460fe00640 1 -- 172.21.15.28:0/524192844 learned_addr learned my addr 172.21.15.28:0/524192844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4615200640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46100b3620 0x7f4610147fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.242+0000 7f4615c00640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b19d0 0x7f4610158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 -- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b19d0 msgr2=0x7f4610158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b19d0 0x7f4610158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 -- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46100b3620 msgr2=0x7f4610147fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46100b3620 0x7f4610147fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 -- 172.21.15.28:0/524192844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46100b2d70 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460fe00640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00f5f0 tx=0x7f45fc00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc0052e0 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45fc005440 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.243+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46100b0c90 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.244+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc003810 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.244+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4610148510 con 0x7f4610152130 2024-09-13T17:25:43.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.245+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46100b0af0 con 0x7f4610152130 2024-09-13T17:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.246+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f45fc003970 con 0x7f4610152130 2024-09-13T17:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.246+0000 7f460cc00640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 0x7f45f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.247+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f45fc088de0 con 0x7f4610152130 2024-09-13T17:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.247+0000 7f4615200640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 0x7f45f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.247+0000 7f4615200640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 0x7f45f006d0c0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4604033000 tx=0x7f460402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.255+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f46100b0af0 con 0x7f4610152130 2024-09-13T17:25:43.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.432+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f461015da80 con 0x7f45f006ac00 2024-09-13T17:25:43.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.435+0000 7f460cc00640 1 -- 172.21.15.28:0/524192844 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50654 (secure 0 0 0) 0x7f4604001710 con 0x7f45f006ac00 2024-09-13T17:25:43.437 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:43.437 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:25:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 msgr2=0x7f45f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 0x7f45f006d0c0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f4604033000 tx=0x7f460402e000 comp rx=0 tx=0).stop 2024-09-13T17:25:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 msgr2=0x7f46100b1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00f5f0 tx=0x7f45fc00fab0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 shutdown_connections 2024-09-13T17:25:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f45f006ac00 0x7f45f006d0c0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f46100b19d0 0x7f4610158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4610152130 0x7f46100b1490 unknown :-1 s=CLOSED pgs=2877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 --2- 172.21.15.28:0/524192844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46100b3620 0x7f4610147fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.437+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 >> 172.21.15.28:0/524192844 conn(0x7f461006d4e0 msgr2=0x7f461015ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.438+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 shutdown_connections 2024-09-13T17:25:43.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.438+0000 7f4616600640 1 -- 172.21.15.28:0/524192844 wait complete. 2024-09-13T17:25:43.455 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2024-09-13T17:25:43.153166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":578240,"kb_used_data":24608,"kb_used_omap":12,"kb_used_meta":553523,"kb_avail":754396480,"statfs":{"total":773094113280,"available":772501995520,"internally_reserved":0,"allocated":25198592,"data_stored":11800682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":566808121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12624397,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-99,"num_read_kb":-105,"num_write":-94,"num_write_kb":-12360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.441587"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T17:25:35.713540+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:25:35.713540+0000","last_peered":"2024-09-13T17:25:35.713540+0000","last_clean":"2024-09-13T17:25:35.713540+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:25:35.713540+0000","last_undegraded":"2024-09-13T17:25:35.713540+0000","last_fullsized":"2024-09-13T17:25:35.713540+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70660,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301180,"statfs":{"total":96636764160,"available":96564408320,"internally_reserved":0,"allocated":3596288,"data_stored":1918116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33800000000000002},"max":{"1min":0.87,"5min":1.008,"15min":2.0880000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.57599999999999996,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":0.93700000000000006,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.91100000000000003,"5min":1.161,"15min":2.5030000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.65000000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":1.399,"15min":6.4909999999999997},"last":0.72899999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71899999999999997,"15min":0.70299999999999996},"min":{"1min":0.58899999999999997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":0.90300000000000002,"5min":1.105,"15min":1.544},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.70599999999999996},"min":{"1min":0.61799999999999999,"5min":0.42399999999999999,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.77500000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.45400000000000001,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.437},"max":{"1min":0.91200000000000003,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.97499999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":1.2170000000000001,"15min":1.234},"last":1.079}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.43099999999999999,"5min":0.42699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":1.143},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.57399999999999995,"5min":0.48499999999999999,"15min":0.44800000000000001},"max":{"1min":0.79000000000000004,"5min":1.915,"15min":1.915},"last":0.99399999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.49299999999999999,"5min":0.41799999999999998,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.607,"15min":1.607},"last":0.94499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70084,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301756,"statfs":{"total":96636764160,"available":96564998144,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70999999999999996,"15min":0.67600000000000005},"min":{"1min":0.58399999999999996,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":0.96399999999999997,"5min":0.97099999999999997,"15min":1.298},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.67000000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":5.9800000000000004},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.373,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.998,"15min":2.3100000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.57499999999999996,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90700000000000003,"15min":1.46},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.60899999999999999,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.89800000000000002,"5min":1.107,"15min":5.9240000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71099999999999997,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.872,"5min":0.92700000000000005,"15min":1.012},"last":0.61699999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73599999999999999,"15min":0.72199999999999998},"min":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":1.151,"15min":3.6520000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72499999999999998,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92400000000000004,"5min":1.085,"15min":3.6720000000000002},"last":0.81100000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.33800000000000002},"max":{"1min":0.92600000000000005,"5min":1.262,"15min":1.6439999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.075},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.374},"max":{"1min":1.0209999999999999,"5min":1.202,"15min":1.8260000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.47699999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":0.90200000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.46999999999999997,"5min":0.379,"15min":0.378},"max":{"1min":1.052,"5min":1.163,"15min":1.601},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79400000000000004,"15min":0.78000000000000003},"min":{"1min":0.66600000000000004,"5min":0.58999999999999997,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.5720000000000001},"last":0.88400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575064,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48000,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323840,"statfs":{"total":96636764160,"available":96587612160,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.76800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91200000000000003,"5min":1.1220000000000001,"15min":38.005000000000003},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.41099999999999998,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.879}]},{"osd":1,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.69299999999999995,"15min":0.70199999999999996},"min":{"1min":0.58299999999999996,"5min":0.44700000000000001,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.169,"15min":7.7370000000000001},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.379},"max":{"1min":0.93000000000000005,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98599999999999999,"15min":9.6549999999999994},"last":0.89100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.72499999999999998,"15min":1.0820000000000001},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.435},"max":{"1min":0.98099999999999998,"5min":1.042,"15min":113.363},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.61899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":4.1180000000000003},"last":0.84699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":1.0760000000000001,"15min":2.0590000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":2.827},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.68600000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.34799999999999998},"max":{"1min":0.85199999999999998,"5min":1.0609999999999999,"15min":2.2320000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.71299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":1.032,"15min":4.1500000000000004},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.746,"15min":0.755},"min":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.40999999999999998},"max":{"1min":0.98199999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.748},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.748,"15min":0.75700000000000001},"min":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.44},"max":{"1min":0.95799999999999996,"5min":1.169,"15min":2.02},"last":0.79400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.26,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.379,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2350000000000001,"15min":0.88400000000000001},"min":{"1min":0.63100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.88600000000000001,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.40899999999999997,"15min":0.35099999999999998},"max":{"1min":0.94399999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.95999999999999996,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.70999999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72999999999999998,"15min":0.68600000000000005},"min":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.98599999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.1259999999999999,"15min":0.83599999999999997},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.378},"max":{"1min":0.996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.73399999999999999},"min":{"1min":0.499,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.046,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":0.76000000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65700000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.32500000000000001},"max":{"1min":0.85499999999999998,"5min":2.976,"15min":2.976},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.75,"15min":0.73199999999999998},"min":{"1min":0.58099999999999996,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79600000000000004,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.54100000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.753,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96599999999999997,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.57399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.65200000000000002,"5min":0.76400000000000001,"15min":0.97099999999999997},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.32900000000000001},"max":{"1min":0.872,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.621}]},{"osd":2,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":1.4930000000000001,"15min":1.9590000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34499999999999997},"max":{"1min":0.79700000000000004,"5min":0.875,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.748},"min":{"1min":0.63900000000000001,"5min":0.45600000000000002,"15min":0.39700000000000002},"max":{"1min":1.05,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.498},"max":{"1min":0.99399999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.76900000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0389999999999999,"5min":1.228,"15min":1.391},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.76400000000000001},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.442},"max":{"1min":1.022,"5min":2.347,"15min":2.347},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80400000000000005,"15min":0.78000000000000003},"min":{"1min":0.63,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.81000000000000005,"15min":0.79000000000000004},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46300000000000002},"max":{"1min":1.069,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.81399999999999995,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.42699999999999999},"max":{"1min":1.036,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83799999999999997,"15min":0.80800000000000005},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.47599999999999998},"max":{"1min":1.0529999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70076,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":2998272,"data_stored":1326729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":5.6769999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.58699999999999997},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.27100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":30.023},"last":0.64400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":1.3720000000000001},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.307},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":260.53399999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":1.4710000000000001},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.94099999999999995,"15min":262.61399999999998},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51000000000000001,"15min":0.624},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.84199999999999997,"15min":27.271999999999998},"last":0.435},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.61799999999999999},"min":{"1min":0.52400000000000002,"5min":0.45000000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91100000000000003,"15min":8.4149999999999991},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78200000000000003,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.53000000000000003,"15min":0.45500000000000002},"max":{"1min":1.331,"5min":1.331,"15min":10.598000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.78600000000000003,"15min":0.86599999999999999},"min":{"1min":0.60899999999999999,"5min":0.51200000000000001,"15min":0.371},"max":{"1min":1.1100000000000001,"5min":1.264,"15min":21.213000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.78400000000000003,"15min":0.88400000000000001},"min":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.40300000000000002},"max":{"1min":1.198,"5min":1.198,"15min":27.135000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80000000000000004,"15min":0.79100000000000004},"min":{"1min":0.63100000000000001,"5min":0.52400000000000002,"15min":0.47599999999999998},"max":{"1min":1.167,"5min":1.167,"15min":5.8890000000000002},"last":1.01}]},{"osd":6,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.79900000000000004,"15min":0.83599999999999997},"min":{"1min":0.68000000000000005,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":10.664},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78200000000000003,"15min":0.89100000000000001},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.222,"5min":1.23,"15min":27.236999999999998},"last":0.88}]},{"osd":7,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.68100000000000005,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.369,"5min":1.369,"15min":10.712999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80300000000000005,"15min":0.91000000000000003},"min":{"1min":0.64400000000000002,"5min":0.52300000000000002,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.1799999999999999,"15min":27.199999999999999},"last":0.80200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96320,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275520,"statfs":{"total":96636764160,"available":96538132480,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23400000000000001},"max":{"1min":0.61099999999999999,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.47899999999999998,"5min":0.35999999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":6.7240000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64300000000000002,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.308,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.65300000000000002},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54200000000000004,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":18.318000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.28,"15min":6.1630000000000003},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.67000000000000004,"5min":0.56299999999999994,"15min":0.42599999999999999},"max":{"1min":1.0129999999999999,"5min":1.23,"15min":1.78},"last":0.96299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.0389999999999999,"5min":1.159,"15min":1.6020000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.498,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77900000000000003},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.2330000000000001,"15min":1.6599999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.65300000000000002,"5min":0.503,"15min":0.41399999999999998},"max":{"1min":1.079,"5min":1.246,"15min":6.1970000000000001},"last":0.90500000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.79100000000000004},"min":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.46600000000000003},"max":{"1min":1.0209999999999999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81100000000000005},"min":{"1min":0.64300000000000002,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.097,"5min":1.2609999999999999,"15min":1.544},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74748,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297092,"statfs":{"total":96636764160,"available":96560222208,"internally_reserved":0,"allocated":3592192,"data_stored":1917097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.433,"15min":0.442},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":0.72599999999999998,"15min":1.986},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":0.73399999999999999,"5min":0.81599999999999995,"15min":3.004},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":0.877,"15min":3.605},"last":0.379},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.45800000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":2.4039999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":2.6400000000000001},"last":0.435},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":3.7709999999999999},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.65400000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.034,"5min":1.1599999999999999,"15min":3.8849999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.64400000000000002,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.002,"5min":1.2969999999999999,"15min":2.6760000000000002},"last":0.48699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.44500000000000001},"max":{"1min":1.1299999999999999,"5min":1.1799999999999999,"15min":2.7320000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80500000000000005,"15min":0.79100000000000004},"min":{"1min":0.65900000000000003,"5min":0.51800000000000002,"15min":0.432},"max":{"1min":1.006,"5min":1.276,"15min":1.8740000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.70999999999999996,"5min":0.52600000000000002,"15min":0.46700000000000003},"max":{"1min":1.0349999999999999,"5min":1.1619999999999999,"15min":1.242},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.70199999999999996,"5min":0.50600000000000001,"15min":0.48599999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":2.8380000000000001},"last":0.499}]},{"osd":7,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.73899999999999999,"5min":0.52100000000000002,"15min":0.47399999999999998},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":3.9209999999999998},"last":0.626},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0760000000000001,"5min":1.242,"15min":2.706},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:25:43.457 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T17:25:43.457 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:25:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.612+0000 7fa07fe00640 1 Processor -- start 2024-09-13T17:25:43.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.612+0000 7fa07fe00640 1 -- start start 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.613+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07815e860 0x7fa078161d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.613+0000 7fa07fe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fa07806cde0 0x7fa078070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07fe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0780b35c0 con 0x7fa0780711d0 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07fe00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0781618f0 con 0x7fa07806cde0 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0780b38a0 con 0x7fa07815e860 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa0780711d0 0x7fa07806b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:58186/0 (socket says 172.21.15.28:58186) 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07de00640 1 -- 172.21.15.28:0/2466798987 learned_addr learned my addr 172.21.15.28:0/2466798987 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07ca00640 1 --2- 172.21.15.28:0/2466798987 >> v2:172.21.15.28:3300/0 conn(0x7fa07806cde0 0x7fa078070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07d400640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07815e860 0x7fa078161d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa077e00640 1 -- 172.21.15.28:0/2466798987 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3624089790 0 0) 0x7fa0780b35c0 con 0x7fa0780711d0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa077e00640 1 -- 172.21.15.28:0/2466798987 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa060003520 con 0x7fa0780711d0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07ca00640 1 -- 172.21.15.28:0/2466798987 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07815e860 msgr2=0x7fa078161d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07ca00640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07815e860 0x7fa078161d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.614+0000 7fa07ca00640 1 -- 172.21.15.28:0/2466798987 >> v1:172.21.15.28:6789/0 conn(0x7fa0780711d0 legacy=0x7fa07806b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.615+0000 7fa07ca00640 1 -- 172.21.15.28:0/2466798987 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa078161230 con 0x7fa07806cde0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa07ca00640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07806cde0 0x7fa078070c90 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7fa0640097a0 tx=0x7fa06402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865505945670efd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa077e00640 1 -- 172.21.15.28:0/2466798987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa06402ae50 con 0x7fa07806cde0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa077e00640 1 -- 172.21.15.28:0/2466798987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa064032440 con 0x7fa07806cde0 2024-09-13T17:25:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa077e00640 1 -- 172.21.15.28:0/2466798987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0640327b0 con 0x7fa07806cde0 2024-09-13T17:25:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa07fe00640 1 -- 172.21.15.28:0/2466798987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07806cde0 msgr2=0x7fa078070c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.616+0000 7fa07fe00640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07806cde0 0x7fa078070c90 secure :-1 s=READY pgs=2879 cs=0 l=1 rev1=1 crypto rx=0x7fa0640097a0 tx=0x7fa06402a100 comp rx=0 tx=0).stop 2024-09-13T17:25:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 -- 172.21.15.28:0/2466798987 shutdown_connections 2024-09-13T17:25:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa07806cde0 0x7fa078070c90 unknown :-1 s=CLOSED pgs=2879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 --2- 172.21.15.28:0/2466798987 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07815e860 0x7fa078161d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 -- 172.21.15.28:0/2466798987 >> 172.21.15.28:0/2466798987 conn(0x7fa078069e80 msgr2=0x7fa07806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 -- 172.21.15.28:0/2466798987 shutdown_connections 2024-09-13T17:25:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.617+0000 7fa07fe00640 1 -- 172.21.15.28:0/2466798987 wait complete. 2024-09-13T17:25:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.618+0000 7fa07fe00640 1 Processor -- start 2024-09-13T17:25:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.618+0000 7fa07fe00640 1 -- start start 2024-09-13T17:25:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07806cde0 0x7fa078149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 0x7fa07815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa078149f70 con 0x7fa0780711d0 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781e9450 con 0x7fa07815e860 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.619+0000 7fa07fe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0781eca30 con 0x7fa07806cde0 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58308/0 (socket says 172.21.15.28:58308) 2024-09-13T17:25:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 -- 172.21.15.28:0/109216817 learned_addr learned my addr 172.21.15.28:0/109216817 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07d400640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07806cde0 0x7fa078149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07de00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 0x7fa07815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 -- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07806cde0 msgr2=0x7fa078149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07806cde0 0x7fa078149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 -- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 msgr2=0x7fa07815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 0x7fa07815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07ca00640 1 -- 172.21.15.28:0/109216817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa064009400 con 0x7fa0780711d0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.620+0000 7fa07de00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 0x7fa07815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa07ca00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fa0640098c0 tx=0x7fa0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa064003e50 con 0x7fa0780711d0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0640044c0 con 0x7fa0780711d0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0640327d0 con 0x7fa0780711d0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0781e9e20 con 0x7fa0780711d0 2024-09-13T17:25:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.621+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0781e8b20 con 0x7fa0780711d0 2024-09-13T17:25:43.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.623+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0781e9600 con 0x7fa0780711d0 2024-09-13T17:25:43.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.623+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa064032930 con 0x7fa0780711d0 2024-09-13T17:25:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.624+0000 7fa075600640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 0x7fa05806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.624+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa0640a9970 con 0x7fa0780711d0 2024-09-13T17:25:43.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.625+0000 7fa07d400640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 0x7fa05806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:43.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.625+0000 7fa07d400640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 0x7fa05806d0c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006fd0 tx=0x7fa06c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:43.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.633+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa0781e9600 con 0x7fa0780711d0 2024-09-13T17:25:43.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.831+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa07806d240 con 0x7fa05806ac00 2024-09-13T17:25:43.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.833+0000 7fa075600640 1 -- 172.21.15.28:0/109216817 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50654 (secure 0 0 0) 0x7fa06c010990 con 0x7fa05806ac00 2024-09-13T17:25:43.835 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:43.835 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:25:43.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.835+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 msgr2=0x7fa05806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.835+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 0x7fa05806d0c0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fa06c006fd0 tx=0x7fa06c008040 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.835+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 msgr2=0x7fa0781e98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.835+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fa0640098c0 tx=0x7fa0640046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 shutdown_connections 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa05806ac00 0x7fa05806d0c0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa07815e860 0x7fa07815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa0780711d0 0x7fa0781e98e0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 --2- 172.21.15.28:0/109216817 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa07806cde0 0x7fa078149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 >> 172.21.15.28:0/109216817 conn(0x7fa078069e80 msgr2=0x7fa0780a8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 shutdown_connections 2024-09-13T17:25:43.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:43.836+0000 7fa07fe00640 1 -- 172.21.15.28:0/109216817 wait complete. 2024-09-13T17:25:43.852 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2024-09-13T17:25:43.153166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":47,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":578240,"kb_used_data":24608,"kb_used_omap":12,"kb_used_meta":553523,"kb_avail":754396480,"statfs":{"total":773094113280,"available":772501995520,"internally_reserved":0,"allocated":25198592,"data_stored":11800682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":566808121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12624397,"num_objects":-28,"num_object_clones":0,"num_object_copies":-56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-28,"num_whiteouts":0,"num_read":-99,"num_read_kb":-105,"num_write":-94,"num_write_kb":-12360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.441587"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T17:25:35.713540+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:25:35.713540+0000","last_peered":"2024-09-13T17:25:35.713540+0000","last_clean":"2024-09-13T17:25:35.713540+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:25:35.713540+0000","last_undegraded":"2024-09-13T17:25:35.713540+0000","last_fullsized":"2024-09-13T17:25:35.713540+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575064,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70660,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301180,"statfs":{"total":96636764160,"available":96564408320,"internally_reserved":0,"allocated":3596288,"data_stored":1918116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33800000000000002},"max":{"1min":0.87,"5min":1.008,"15min":2.0880000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.57599999999999996,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":0.93700000000000006,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.81000000000000005}]},{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.91100000000000003,"5min":1.161,"15min":2.5030000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.65000000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":1.399,"15min":6.4909999999999997},"last":0.72899999999999998}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71899999999999997,"15min":0.70299999999999996},"min":{"1min":0.58899999999999997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":0.90300000000000002,"5min":1.105,"15min":1.544},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.70599999999999996},"min":{"1min":0.61799999999999999,"5min":0.42399999999999999,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.77500000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.45400000000000001,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.437},"max":{"1min":0.91200000000000003,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.97499999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":1.2170000000000001,"15min":1.234},"last":1.079}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.43099999999999999,"5min":0.42699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":1.143},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.57399999999999995,"5min":0.48499999999999999,"15min":0.44800000000000001},"max":{"1min":0.79000000000000004,"5min":1.915,"15min":1.915},"last":0.99399999999999999}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.49299999999999999,"5min":0.41799999999999998,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.607,"15min":1.607},"last":0.94499999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70084,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301756,"statfs":{"total":96636764160,"available":96564998144,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70999999999999996,"15min":0.67600000000000005},"min":{"1min":0.58399999999999996,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":0.96399999999999997,"5min":0.97099999999999997,"15min":1.298},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.67000000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":5.9800000000000004},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.373,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.998,"15min":2.3100000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.57499999999999996,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90700000000000003,"15min":1.46},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.60899999999999999,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.89800000000000002,"5min":1.107,"15min":5.9240000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71099999999999997,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.872,"5min":0.92700000000000005,"15min":1.012},"last":0.61699999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73599999999999999,"15min":0.72199999999999998},"min":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":1.151,"15min":3.6520000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72499999999999998,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92400000000000004,"5min":1.085,"15min":3.6720000000000002},"last":0.81100000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.33800000000000002},"max":{"1min":0.92600000000000005,"5min":1.262,"15min":1.6439999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.075},"last":0.70299999999999996}]},{"osd":5,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.374},"max":{"1min":1.0209999999999999,"5min":1.202,"15min":1.8260000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.47699999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":0.90200000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.84699999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.46999999999999997,"5min":0.379,"15min":0.378},"max":{"1min":1.052,"5min":1.163,"15min":1.601},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79400000000000004,"15min":0.78000000000000003},"min":{"1min":0.66600000000000004,"5min":0.58999999999999997,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.5720000000000001},"last":0.88400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575064,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48000,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323840,"statfs":{"total":96636764160,"available":96587612160,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.76800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91200000000000003,"5min":1.1220000000000001,"15min":38.005000000000003},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.41099999999999998,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.879}]},{"osd":1,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.69299999999999995,"15min":0.70199999999999996},"min":{"1min":0.58299999999999996,"5min":0.44700000000000001,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.169,"15min":7.7370000000000001},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.379},"max":{"1min":0.93000000000000005,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98599999999999999,"15min":9.6549999999999994},"last":0.89100000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.72499999999999998,"15min":1.0820000000000001},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.435},"max":{"1min":0.98099999999999998,"5min":1.042,"15min":113.363},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.61899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":4.1180000000000003},"last":0.84699999999999998}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":1.0760000000000001,"15min":2.0590000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":2.827},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.68600000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.34799999999999998},"max":{"1min":0.85199999999999998,"5min":1.0609999999999999,"15min":2.2320000000000002},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.71299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":1.032,"15min":4.1500000000000004},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.746,"15min":0.755},"min":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.40999999999999998},"max":{"1min":0.98199999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.748},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.748,"15min":0.75700000000000001},"min":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.44},"max":{"1min":0.95799999999999996,"5min":1.169,"15min":2.02},"last":0.79400000000000004}]}]},{"osd":4,"up_from":13,"seq":55834575063,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.26,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.379,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2350000000000001,"15min":0.88400000000000001},"min":{"1min":0.63100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.88600000000000001,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.40899999999999997,"15min":0.35099999999999998},"max":{"1min":0.94399999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.95999999999999996,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.70999999999999996}]},{"osd":2,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72999999999999998,"15min":0.68600000000000005},"min":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.98599999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.61899999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.1259999999999999,"15min":0.83599999999999997},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.378},"max":{"1min":0.996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.73399999999999999},"min":{"1min":0.499,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.046,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":0.76000000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.82299999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65700000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.32500000000000001},"max":{"1min":0.85499999999999998,"5min":2.976,"15min":2.976},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.75,"15min":0.73199999999999998},"min":{"1min":0.58099999999999996,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79600000000000004,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.54100000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.753,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96599999999999997,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.57399999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.65200000000000002,"5min":0.76400000000000001,"15min":0.97099999999999997},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.32900000000000001},"max":{"1min":0.872,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.621}]},{"osd":2,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":1.4930000000000001,"15min":1.9590000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34499999999999997},"max":{"1min":0.79700000000000004,"5min":0.875,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.748},"min":{"1min":0.63900000000000001,"5min":0.45600000000000002,"15min":0.39700000000000002},"max":{"1min":1.05,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.498},"max":{"1min":0.99399999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.76900000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0389999999999999,"5min":1.228,"15min":1.391},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.76400000000000001},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.442},"max":{"1min":1.022,"5min":2.347,"15min":2.347},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80400000000000005,"15min":0.78000000000000003},"min":{"1min":0.63,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.81000000000000005,"15min":0.79000000000000004},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46300000000000002},"max":{"1min":1.069,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.81399999999999995,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.42699999999999999},"max":{"1min":1.036,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83799999999999997,"15min":0.80800000000000005},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.47599999999999998},"max":{"1min":1.0529999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70076,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":2998272,"data_stored":1326729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":5.6769999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.58699999999999997},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.27100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":30.023},"last":0.64400000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":1.3720000000000001},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.307},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":260.53399999999999},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":1.4710000000000001},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.94099999999999995,"15min":262.61399999999998},"last":0.52000000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51000000000000001,"15min":0.624},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.84199999999999997,"15min":27.271999999999998},"last":0.435},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.61799999999999999},"min":{"1min":0.52400000000000002,"5min":0.45000000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91100000000000003,"15min":8.4149999999999991},"last":0.93600000000000005}]},{"osd":4,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78200000000000003,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.53000000000000003,"15min":0.45500000000000002},"max":{"1min":1.331,"5min":1.331,"15min":10.598000000000001},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.78600000000000003,"15min":0.86599999999999999},"min":{"1min":0.60899999999999999,"5min":0.51200000000000001,"15min":0.371},"max":{"1min":1.1100000000000001,"5min":1.264,"15min":21.213000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.78400000000000003,"15min":0.88400000000000001},"min":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.40300000000000002},"max":{"1min":1.198,"5min":1.198,"15min":27.135000000000002},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80000000000000004,"15min":0.79100000000000004},"min":{"1min":0.63100000000000001,"5min":0.52400000000000002,"15min":0.47599999999999998},"max":{"1min":1.167,"5min":1.167,"15min":5.8890000000000002},"last":1.01}]},{"osd":6,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.79900000000000004,"15min":0.83599999999999997},"min":{"1min":0.68000000000000005,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":10.664},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78200000000000003,"15min":0.89100000000000001},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.222,"5min":1.23,"15min":27.236999999999998},"last":0.88}]},{"osd":7,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.68100000000000005,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.369,"5min":1.369,"15min":10.712999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80300000000000005,"15min":0.91000000000000003},"min":{"1min":0.64400000000000002,"5min":0.52300000000000002,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.1799999999999999,"15min":27.199999999999999},"last":0.80200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834575063,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96320,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275520,"statfs":{"total":96636764160,"available":96538132480,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23400000000000001},"max":{"1min":0.61099999999999999,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.47899999999999998,"5min":0.35999999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":6.7240000000000002},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64300000000000002,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.438},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.308,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.65300000000000002},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54200000000000004,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":18.318000000000001},"last":0.45900000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.28,"15min":6.1630000000000003},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.67000000000000004,"5min":0.56299999999999994,"15min":0.42599999999999999},"max":{"1min":1.0129999999999999,"5min":1.23,"15min":1.78},"last":0.96299999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.0389999999999999,"5min":1.159,"15min":1.6020000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.498,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.81599999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77900000000000003},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.2330000000000001,"15min":1.6599999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.65300000000000002,"5min":0.503,"15min":0.41399999999999998},"max":{"1min":1.079,"5min":1.246,"15min":6.1970000000000001},"last":0.90500000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.79100000000000004},"min":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.46600000000000003},"max":{"1min":1.0209999999999999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81100000000000005},"min":{"1min":0.64300000000000002,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.097,"5min":1.2609999999999999,"15min":1.544},"last":0.79500000000000004}]}]},{"osd":0,"up_from":13,"seq":55834575065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74748,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297092,"statfs":{"total":96636764160,"available":96560222208,"internally_reserved":0,"allocated":3592192,"data_stored":1917097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.433,"15min":0.442},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":0.72599999999999998,"15min":1.986},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":0.73399999999999999,"5min":0.81599999999999995,"15min":3.004},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":0.877,"15min":3.605},"last":0.379},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.45800000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":2.4039999999999999},"last":0.39500000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":2.6400000000000001},"last":0.435},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":3.7709999999999999},"last":0.42099999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.65400000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.034,"5min":1.1599999999999999,"15min":3.8849999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.64400000000000002,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.002,"5min":1.2969999999999999,"15min":2.6760000000000002},"last":0.48699999999999999}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.44500000000000001},"max":{"1min":1.1299999999999999,"5min":1.1799999999999999,"15min":2.7320000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80500000000000005,"15min":0.79100000000000004},"min":{"1min":0.65900000000000003,"5min":0.51800000000000002,"15min":0.432},"max":{"1min":1.006,"5min":1.276,"15min":1.8740000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.70999999999999996,"5min":0.52600000000000002,"15min":0.46700000000000003},"max":{"1min":1.0349999999999999,"5min":1.1619999999999999,"15min":1.242},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.70199999999999996,"5min":0.50600000000000001,"15min":0.48599999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":2.8380000000000001},"last":0.499}]},{"osd":7,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.73899999999999999,"5min":0.52100000000000002,"15min":0.47399999999999998},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":3.9209999999999998},"last":0.626},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0760000000000001,"5min":1.242,"15min":2.706},"last":0.54200000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:25:43.854 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T17:25:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.009+0000 7f3bb9a00640 1 Processor -- start 2024-09-13T17:25:44.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.010+0000 7f3bb9a00640 1 -- start start 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb9a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bb406b360 0x7f3bb40b0440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb9a00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40b0d80 con 0x7f3bb4154a90 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb9a00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb406b900 con 0x7f3bb40b0980 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb9a00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb406bbe0 con 0x7f3bb406b360 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb3e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3bb4154a90 0x7f3bb415b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53182/0 (socket says 172.21.15.28:53182) 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3bb3e00640 1 -- 172.21.15.28:0/3843003387 learned_addr learned my addr 172.21.15.28:0/3843003387 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:44.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.011+0000 7f3babe00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f3bb406b360 0x7f3bb40b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.012+0000 7f3bb3400640 1 -- 172.21.15.28:0/3843003387 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14755752 0 0) 0x7f3bb40b0d80 con 0x7f3bb4154a90 2024-09-13T17:25:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.012+0000 7f3babe00640 1 -- 172.21.15.28:0/3843003387 >> v1:172.21.15.189:6789/0 conn(0x7f3bb40b0980 legacy=0x7f3bb40af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.012+0000 7f3babe00640 1 -- 172.21.15.28:0/3843003387 >> v1:172.21.15.28:6789/0 conn(0x7f3bb4154a90 legacy=0x7f3bb415b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.012+0000 7f3babe00640 1 -- 172.21.15.28:0/3843003387 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb406bec0 con 0x7f3bb406b360 2024-09-13T17:25:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.013+0000 7f3babe00640 1 --2- 172.21.15.28:0/3843003387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb406b360 0x7f3bb40b0440 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0097a0 tx=0x7f3b9c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f12c2aaed09fad1b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:44.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb3400640 1 -- 172.21.15.28:0/3843003387 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c032440 con 0x7f3bb406b360 2024-09-13T17:25:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb3400640 1 -- 172.21.15.28:0/3843003387 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c0325a0 con 0x7f3bb406b360 2024-09-13T17:25:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb3400640 1 -- 172.21.15.28:0/3843003387 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c032930 con 0x7f3bb406b360 2024-09-13T17:25:44.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 -- 172.21.15.28:0/3843003387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb406b360 msgr2=0x7f3bb40b0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/3843003387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb406b360 0x7f3bb40b0440 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0097a0 tx=0x7f3b9c02a230 comp rx=0 tx=0).stop 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 -- 172.21.15.28:0/3843003387 shutdown_connections 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/3843003387 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb406b360 0x7f3bb40b0440 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 -- 172.21.15.28:0/3843003387 >> 172.21.15.28:0/3843003387 conn(0x7f3bb4072e40 msgr2=0x7f3bb4073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 -- 172.21.15.28:0/3843003387 shutdown_connections 2024-09-13T17:25:44.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.014+0000 7f3bb9a00640 1 -- 172.21.15.28:0/3843003387 wait complete. 2024-09-13T17:25:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.016+0000 7f3bb9a00640 1 Processor -- start 2024-09-13T17:25:44.017 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.016+0000 7f3bb9a00640 1 -- start start 2024-09-13T17:25:44.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 0x7f3bb4137f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb4154a90 0x7f3bb413ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40ad670 con 0x7f3bb40b0980 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40ac9f0 con 0x7f3bb4154a90 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb9a00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40accd0 con 0x7f3bb406b360 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3babe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53098/0 (socket says 172.21.15.28:53098) 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3babe00640 1 -- 172.21.15.28:0/2976189835 learned_addr learned my addr 172.21.15.28:0/2976189835 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:44.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.017+0000 7f3bb3e00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 0x7f3bb4137f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb9000640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb4154a90 0x7f3bb413ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 -- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 msgr2=0x7f3bb4137f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 0x7f3bb4137f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 -- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb4154a90 msgr2=0x7f3bb413ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb4154a90 0x7f3bb413ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 -- 172.21.15.28:0/2976189835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c009400 con 0x7f3bb40b0980 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb3e00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 0x7f3bb4137f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3babe00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c004230 con 0x7f3bb40b0980 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9c004390 con 0x7f3bb40b0980 2024-09-13T17:25:44.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c030ac0 con 0x7f3bb40b0980 2024-09-13T17:25:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.018+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb40acfb0 con 0x7f3bb40b0980 2024-09-13T17:25:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.019+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb4138450 con 0x7f3bb40b0980 2024-09-13T17:25:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.020+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb40ad7b0 con 0x7f3bb40b0980 2024-09-13T17:25:44.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.021+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3b9c039050 con 0x7f3bb40b0980 2024-09-13T17:25:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.022+0000 7f3bb0c00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 0x7f3b9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.022+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3b9c0a9ba0 con 0x7f3bb40b0980 2024-09-13T17:25:44.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.023+0000 7f3bb3e00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 0x7f3b9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:44.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.024+0000 7f3bb3e00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 0x7f3b9006d0c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4007fd0 tx=0x7f3ba4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:44.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.027+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f3bb40ad7b0 con 0x7f3bb40b0980 2024-09-13T17:25:44.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.206+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bb40ad7b0 con 0x7f3bb40b0980 2024-09-13T17:25:44.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.207+0000 7f3bb0c00640 1 -- 172.21.15.28:0/2976189835 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v237) v1 ==== 74+0+13124 (secure 0 0 0) 0x7f3bb40ad7b0 con 0x7f3bb40b0980 2024-09-13T17:25:44.208 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:44.208 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":237,"fsid":"99459b14-8688-430f-9c7e-baba0c3bf859","created":"2024-09-13T17:07:46.106817+0000","modified":"2024-09-13T17:25:38.722899+0000","last_up_change":"2024-09-13T17:07:57.601553+0000","last_in_change":"2024-09-13T17:07:48.915782+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T17:07:58.938630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"83067adb-3344-4f9d-b7f5-33d98663ee01","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6801","nonce":3372255106}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6803","nonce":3372255106}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6807","nonce":3372255106}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":3372255106},{"type":"v1","addr":"172.21.15.28:6805","nonce":3372255106}]},"public_addr":"172.21.15.28:6801/3372255106","cluster_addr":"172.21.15.28:6803/3372255106","heartbeat_back_addr":"172.21.15.28:6807/3372255106","heartbeat_front_addr":"172.21.15.28:6805/3372255106","state":["exists","up"]},{"osd":1,"uuid":"ae399b9b-4170-40ef-9121-629474c54fb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6809","nonce":2722659727}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6811","nonce":2722659727}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6815","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6818","nonce":2722659727}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":2722659727},{"type":"v1","addr":"172.21.15.28:6813","nonce":2722659727}]},"public_addr":"172.21.15.28:6809/2722659727","cluster_addr":"172.21.15.28:6811/2722659727","heartbeat_back_addr":"172.21.15.28:6818/2722659727","heartbeat_front_addr":"172.21.15.28:6813/2722659727","state":["exists","up"]},{"osd":2,"uuid":"8638f528-5f30-4d64-ad6f-25c517197b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6819","nonce":2959928040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6821","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6823","nonce":2959928040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6829","nonce":2959928040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":2959928040},{"type":"v1","addr":"172.21.15.28:6826","nonce":2959928040}]},"public_addr":"172.21.15.28:6819/2959928040","cluster_addr":"172.21.15.28:6823/2959928040","heartbeat_back_addr":"172.21.15.28:6829/2959928040","heartbeat_front_addr":"172.21.15.28:6826/2959928040","state":["exists","up"]},{"osd":3,"uuid":"00d0c5cc-4bae-44a8-97bd-42c0ac7e9d58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6817","nonce":3941122226}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6822","nonce":3941122226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6831","nonce":3941122226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6825","nonce":3941122226},{"type":"v1","addr":"172.21.15.28:6827","nonce":3941122226}]},"public_addr":"172.21.15.28:6817/3941122226","cluster_addr":"172.21.15.28:6822/3941122226","heartbeat_back_addr":"172.21.15.28:6831/3941122226","heartbeat_front_addr":"172.21.15.28:6827/3941122226","state":["exists","up"]},{"osd":4,"uuid":"f390dbdf-1494-4416-9b4e-50c14a064a5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6801","nonce":531106775}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6803","nonce":531106775}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6807","nonce":531106775}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":531106775},{"type":"v1","addr":"172.21.15.189:6805","nonce":531106775}]},"public_addr":"172.21.15.189:6801/531106775","cluster_addr":"172.21.15.189:6803/531106775","heartbeat_back_addr":"172.21.15.189:6807/531106775","heartbeat_front_addr":"172.21.15.189:6805/531106775","state":["exists","up"]},{"osd":5,"uuid":"fc031241-63f7-4113-a561-8bc3012719bc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6809","nonce":1971214256}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6811","nonce":1971214256}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6815","nonce":1971214256}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":1971214256},{"type":"v1","addr":"172.21.15.189:6813","nonce":1971214256}]},"public_addr":"172.21.15.189:6809/1971214256","cluster_addr":"172.21.15.189:6811/1971214256","heartbeat_back_addr":"172.21.15.189:6815/1971214256","heartbeat_front_addr":"172.21.15.189:6813/1971214256","state":["exists","up"]},{"osd":6,"uuid":"8cd60534-2b08-47c1-8cd7-69c2e3ac1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6817","nonce":2946605912}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6819","nonce":2946605912}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6823","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6825","nonce":2946605912}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":2946605912},{"type":"v1","addr":"172.21.15.189:6821","nonce":2946605912}]},"public_addr":"172.21.15.189:6817/2946605912","cluster_addr":"172.21.15.189:6819/2946605912","heartbeat_back_addr":"172.21.15.189:6825/2946605912","heartbeat_front_addr":"172.21.15.189:6821/2946605912","state":["exists","up"]},{"osd":7,"uuid":"e1be57a8-fb3f-4809-88b1-009581c546b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6824","nonce":2656371491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6827","nonce":2656371491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6831","nonce":2656371491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2656371491},{"type":"v1","addr":"172.21.15.189:6829","nonce":2656371491}]},"public_addr":"172.21.15.189:6824/2656371491","cluster_addr":"172.21.15.189:6827/2656371491","heartbeat_back_addr":"172.21.15.189:6831/2656371491","heartbeat_front_addr":"172.21.15.189:6829/2656371491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.422875+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.267403+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.484490+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.308652+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.428349+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.271763+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.291533+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T17:07:54.360025+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6835/994067792":"2024-09-14T17:25:35.704376+0000","172.21.15.28:6833/994067792":"2024-09-14T17:25:35.704376+0000","172.21.15.28:6835/1114231130":"2024-09-14T17:24:22.037690+0000","172.21.15.28:6833/1114231130":"2024-09-14T17:24:22.037690+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 msgr2=0x7f3b9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 0x7f3b9006d0c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4007fd0 tx=0x7f3ba4007dc0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 msgr2=0x7f3bb40b2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c000c00 tx=0x7f3b9c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 shutdown_connections 2024-09-13T17:25:44.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3b9006ac00 0x7f3b9006d0c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bb4154a90 0x7f3bb413ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3bb40b0980 0x7f3bb40b2dd0 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 --2- 172.21.15.28:0/2976189835 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3bb406b360 0x7f3bb4137f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 >> 172.21.15.28:0/2976189835 conn(0x7f3bb4072e40 msgr2=0x7f3bb415d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 shutdown_connections 2024-09-13T17:25:44.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:44.209+0000 7f3bb9a00640 1 -- 172.21.15.28:0/2976189835 wait complete. 2024-09-13T17:25:45.228 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T17:25:45.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.391+0000 7f3aa2200640 1 Processor -- start 2024-09-13T17:25:45.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.391+0000 7f3aa2200640 1 -- start start 2024-09-13T17:25:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.392+0000 7f3aa2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a9c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.392+0000 7f3aa2200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f3a9c0b2630 0x7f3a9c15a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.392+0000 7f3aa2200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a9c15a4c0 con 0x7f3a9c159590 2024-09-13T17:25:45.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.392+0000 7f3aa2200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a9c0b2ae0 con 0x7f3a9c0b2630 2024-09-13T17:25:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.392+0000 7f3aa2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c071390 con 0x7f3a9c159970 2024-09-13T17:25:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9b400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f3a9c159590 0x7f3a9c070c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53186/0 (socket says 172.21.15.28:53186) 2024-09-13T17:25:45.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9b400640 1 -- 172.21.15.28:0/3127646699 learned_addr learned my addr 172.21.15.28:0/3127646699 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:45.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3127646699 >> v2:172.21.15.189:3300/0 conn(0x7f3a9c0b2630 0x7f3a9c15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a9c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625864546 0 0) 0x7f3a9c15a4c0 con 0x7f3a9c159590 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a80003520 con 0x7f3a9c159590 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27369992 0 0) 0x7f3a80003520 con 0x7f3a9c159590 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 msgr2=0x7f3a9c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a9c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 >> v2:172.21.15.189:3300/0 conn(0x7f3a9c0b2630 msgr2=0x7f3a9c15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 --2- 172.21.15.28:0/3127646699 >> v2:172.21.15.189:3300/0 conn(0x7f3a9c0b2630 0x7f3a9c15a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c071580 con 0x7f3a9c159590 2024-09-13T17:25:45.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.393+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f3a8c001670 con 0x7f3a9c159590 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 >> v1:172.21.15.28:6789/0 conn(0x7f3a9c159590 legacy=0x7f3a9c070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80003970 0x7f3a80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a800047c0 0x7f3a80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c0b2ae0 con 0x7f3a80003970 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8000b980 con 0x7f3a800047c0 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8000bac0 con 0x7f3a9c159970 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80003970 0x7f3a80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a800047c0 0x7f3a80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 -- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 msgr2=0x7f3a80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c159970 0x7f3a80005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 -- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a800047c0 msgr2=0x7f3a80004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a800047c0 0x7f3a80004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9b400640 1 -- 172.21.15.28:0/3127646699 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a80003520 con 0x7f3a80003970 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.394+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a800047c0 0x7f3a80004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.395+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80003970 0x7f3a80003d90 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00c210 tx=0x7f3a800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903e1aaff99e2660 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c02e070 con 0x7f3a80003970 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c0058d0 con 0x7f3a80003970 2024-09-13T17:25:45.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3a9a000640 1 -- 172.21.15.28:0/3127646699 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c005c80 con 0x7f3a80003970 2024-09-13T17:25:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 -- 172.21.15.28:0/3127646699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80003970 msgr2=0x7f3a80003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 --2- 172.21.15.28:0/3127646699 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a80003970 0x7f3a80003d90 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c00c210 tx=0x7f3a800057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3a9be00640 1 -- 172.21.15.28:0/3127646699 reap_dead start 2024-09-13T17:25:45.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 -- 172.21.15.28:0/3127646699 shutdown_connections 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 -- 172.21.15.28:0/3127646699 >> 172.21.15.28:0/3127646699 conn(0x7f3a9c069e80 msgr2=0x7f3a9c06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 -- 172.21.15.28:0/3127646699 shutdown_connections 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.396+0000 7f3aa2200640 1 -- 172.21.15.28:0/3127646699 wait complete. 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.398+0000 7f3aa2200640 1 Processor -- start 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.398+0000 7f3aa2200640 1 -- start start 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a80003970 0x7f3a9c14c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 0x7f3a9c1f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c1f24c0 0x7f3a9c14f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c14c5b0 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1f2f20 con 0x7f3a80003970 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3aa2200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9c1f2960 con 0x7f3a9c1f24c0 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3a9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 0x7f3a9c1f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3a9aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 0x7f3a9c1f1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53110/0 (socket says 172.21.15.28:53110) 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.399+0000 7f3a9aa00640 1 -- 172.21.15.28:0/3928217159 learned_addr learned my addr 172.21.15.28:0/3928217159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c1f24c0 0x7f3a9c14f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 -- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c1f24c0 msgr2=0x7f3a9c14f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f3a9c1f24c0 0x7f3a9c14f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a80003970 0x7f3a9c14c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 -- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a80003970 msgr2=0x7f3a9c14c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a80003970 0x7f3a9c14c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a9c1f2c40 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a80003970 0x7f3a9c14c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a9aa00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 0x7f3a9c1f1f80 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f3a88006c00 tx=0x7f3a88011ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8800fbb0 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8800fd10 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a88014680 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.400+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9c0cb740 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.401+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a9c0c34e0 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.401+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3a9c15a150 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.402+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f3a880147e0 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.403+0000 7f3a93400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7c06ac00 0x7f3a7c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.403+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f3a88089a10 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a93400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f3a7c070b00 0x7f3a7c072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3a7c0735d0 con 0x7f3a7c070b00 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a8808e2a0 con 0x7f3a9c0cdeb0 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7c06ac00 0x7f3a7c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f3a7c070b00 0x7f3a7c072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a9be00640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f3a7c070b00 0x7f3a7c072f20 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.404+0000 7f3a9b400640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7c06ac00 0x7f3a7c06d0c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c033000 tx=0x7f3a8c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.405+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== osd.0 v2:172.21.15.28:6800/3372255106 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f3a7c0735d0 con 0x7f3a7c070b00 2024-09-13T17:25:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.418+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 --> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3a9c0b20f0 con 0x7f3a7c070b00 2024-09-13T17:25:45.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3a93400640 1 -- 172.21.15.28:0/3928217159 <== osd.0 v2:172.21.15.28:6800/3372255106 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3a9c0b20f0 con 0x7f3a7c070b00 2024-09-13T17:25:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:45.420 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:45.420 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f3a7c070b00 msgr2=0x7f3a7c072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:6800/3372255106,v1:172.21.15.28:6801/3372255106] conn(0x7f3a7c070b00 0x7f3a7c072f20 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7c06ac00 msgr2=0x7f3a7c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f3a7c06ac00 0x7f3a7c06d0c0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f3a8c033000 tx=0x7f3a8c009030 comp rx=0 tx=0).stop 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 msgr2=0x7f3a9c1f1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 --2- 172.21.15.28:0/3928217159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a9c0cdeb0 0x7f3a9c1f1f80 secure :-1 s=READY pgs=2886 cs=0 l=1 rev1=1 crypto rx=0x7f3a88006c00 tx=0x7f3a88011ee0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3a9be00640 1 -- 172.21.15.28:0/3928217159 reap_dead start 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 shutdown_connections 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 >> 172.21.15.28:0/3928217159 conn(0x7f3a9c069e80 msgr2=0x7f3a9c06bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 shutdown_connections 2024-09-13T17:25:45.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.419+0000 7f3aa2200640 1 -- 172.21.15.28:0/3928217159 wait complete. 2024-09-13T17:25:45.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T17:25:45.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.590+0000 7f226e600640 1 Processor -- start 2024-09-13T17:25:45.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.591+0000 7f226e600640 1 -- start start 2024-09-13T17:25:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f226e600640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f226815a780 0x7f2268070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f226e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f22680712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f226e600640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226806c680 con 0x7f226815ab60 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f226e600640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226806c8b0 con 0x7f226815a780 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f226e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226806cb90 con 0x7f226806c250 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f225fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f226815ab60 0x7f226806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53190/0 (socket says 172.21.15.28:53190) 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f225fe00640 1 -- 172.21.15.28:0/3265021390 learned_addr learned my addr 172.21.15.28:0/3265021390 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> v2:172.21.15.189:3300/0 conn(0x7f226815a780 0x7f2268070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.592+0000 7f2267e00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f22680712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382400968 0 0) 0x7f226806c680 con 0x7f226815ab60 2024-09-13T17:25:45.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f224c003520 con 0x7f226815ab60 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3910269807 0 0) 0x7f224c003520 con 0x7f226815ab60 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 msgr2=0x7f22680712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f22680712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 >> v2:172.21.15.189:3300/0 conn(0x7f226815a780 msgr2=0x7f2268070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 --2- 172.21.15.28:0/3265021390 >> v2:172.21.15.189:3300/0 conn(0x7f226815a780 0x7f2268070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22680cc980 con 0x7f226815ab60 2024-09-13T17:25:45.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f2250001670 con 0x7f226815ab60 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> v2:172.21.15.189:3300/0 conn(0x7f226815a780 0x7f2268070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2267e00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f22680712c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 >> v1:172.21.15.28:6789/0 conn(0x7f226815ab60 legacy=0x7f226806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f224c003970 0x7f224c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f224c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f224c004d00 0x7f224c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226806c8b0 con 0x7f224c003970 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226806cb90 con 0x7f224c004d00 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c00b980 con 0x7f226806c250 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.593+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f224c003970 0x7f224c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f225fe00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f224c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267e00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f224c004d00 0x7f224c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267400640 1 -- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 msgr2=0x7f224c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f226806c250 0x7f224c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267400640 1 -- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f224c004d00 msgr2=0x7f224c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f224c004d00 0x7f224c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267400640 1 -- 172.21.15.28:0/3265021390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2260009a70 con 0x7f224c003970 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.594+0000 7f2267e00640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f224c004d00 0x7f224c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.595+0000 7f2267400640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f224c003970 0x7f224c003d90 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f2258009c90 tx=0x7f224c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7579ee2d5bbf44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.595+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258031400 con 0x7f224c003970 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.595+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225802d050 con 0x7f224c003970 2024-09-13T17:25:45.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.595+0000 7f2266a00640 1 -- 172.21.15.28:0/3265021390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258031760 con 0x7f224c003970 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 -- 172.21.15.28:0/3265021390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f224c003970 msgr2=0x7f224c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 --2- 172.21.15.28:0/3265021390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f224c003970 0x7f224c003d90 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7f2258009c90 tx=0x7f224c0057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f2267e00640 1 -- 172.21.15.28:0/3265021390 reap_dead start 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 -- 172.21.15.28:0/3265021390 shutdown_connections 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 -- 172.21.15.28:0/3265021390 >> 172.21.15.28:0/3265021390 conn(0x7f22680b0270 msgr2=0x7f22680b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 -- 172.21.15.28:0/3265021390 shutdown_connections 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.596+0000 7f226e600640 1 -- 172.21.15.28:0/3265021390 wait complete. 2024-09-13T17:25:45.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.598+0000 7f226e600640 1 Processor -- start 2024-09-13T17:25:45.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.598+0000 7f226e600640 1 -- start start 2024-09-13T17:25:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f224c003970 0x7f22681f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2268151880 0x7f22680ca760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22680ad210 con 0x7f22680ac850 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22681dfe20 con 0x7f2268151880 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f226e600640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22681e0100 con 0x7f224c003970 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f2267e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f2267e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53138/0 (socket says 172.21.15.28:53138) 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f2267e00640 1 -- 172.21.15.28:0/1956082309 learned_addr learned my addr 172.21.15.28:0/1956082309 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f2267400640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f224c003970 0x7f22681f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.599+0000 7f225fe00640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2268151880 0x7f22680ca760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 -- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f224c003970 msgr2=0x7f22681f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f224c003970 0x7f22681f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 -- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2268151880 msgr2=0x7f22680ca760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2268151880 0x7f22680ca760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 -- 172.21.15.28:0/1956082309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22681e03e0 con 0x7f22680ac850 2024-09-13T17:25:45.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f2267e00640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f22680b1600 tx=0x7f22600115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226000e070 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2260011ed0 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2260003990 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22681e06c0 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.600+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2268151d00 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.601+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22681e0be0 con 0x7f22680ac850 2024-09-13T17:25:45.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.602+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2260003af0 con 0x7f22680ac850 2024-09-13T17:25:45.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.603+0000 7f225f400640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 0x7f224406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.603+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2260089720 con 0x7f22680ac850 2024-09-13T17:25:45.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.603+0000 7f2267400640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 0x7f224406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.604+0000 7f2267400640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 0x7f224406d0c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f2258033000 tx=0x7f2258008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.607+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f22681e0be0 con 0x7f22680ac850 2024-09-13T17:25:45.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.784+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f22680712c0 con 0x7f224406ac00 2024-09-13T17:25:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.785+0000 7f225f400640 1 -- 172.21.15.28:0/1956082309 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f225803f070 con 0x7f224406ac00 2024-09-13T17:25:45.786 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 msgr2=0x7f224406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 0x7f224406d0c0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f2258033000 tx=0x7f2258008030 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 msgr2=0x7f22680accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7f22680b1600 tx=0x7f22600115a0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 shutdown_connections 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f224406ac00 0x7f224406d0c0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22680ac850 0x7f22680accd0 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2268151880 0x7f22680ca760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 --2- 172.21.15.28:0/1956082309 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f224c003970 0x7f22681f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.787+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 >> 172.21.15.28:0/1956082309 conn(0x7f22680b0270 msgr2=0x7f22680ad8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.788+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 shutdown_connections 2024-09-13T17:25:45.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.788+0000 7f226e600640 1 -- 172.21.15.28:0/1956082309 wait complete. 2024-09-13T17:25:45.804 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T17:25:45.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.964+0000 7fe1a5c00640 1 Processor -- start 2024-09-13T17:25:45.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.964+0000 7fe1a5c00640 1 -- start start 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.965+0000 7fe1a5c00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe1a00b2380 0x7fe1a00b27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.965+0000 7fe1a5c00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a015d950 con 0x7fe1a0157000 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.965+0000 7fe1a5c00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a015dc30 con 0x7fe1a015cdd0 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.965+0000 7fe1a5c00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a00b1b80 con 0x7fe1a00b2380 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe19ea00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fe1a0157000 0x7fe1a006b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53204/0 (socket says 172.21.15.28:53204) 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe19ea00640 1 -- 172.21.15.28:0/1273020557 learned_addr learned my addr 172.21.15.28:0/1273020557 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe197e00640 1 --2- 172.21.15.28:0/1273020557 >> v2:172.21.15.189:3300/0 conn(0x7fe1a00b2380 0x7fe1a00b27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe19e000640 1 -- 172.21.15.28:0/1273020557 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113183144 0 0) 0x7fe1a015d950 con 0x7fe1a0157000 2024-09-13T17:25:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe19e000640 1 -- 172.21.15.28:0/1273020557 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe184003520 con 0x7fe1a0157000 2024-09-13T17:25:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe197e00640 1 -- 172.21.15.28:0/1273020557 >> v1:172.21.15.189:6789/0 conn(0x7fe1a015cdd0 legacy=0x7fe1a00ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe197e00640 1 -- 172.21.15.28:0/1273020557 >> v1:172.21.15.28:6789/0 conn(0x7fe1a0157000 legacy=0x7fe1a006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.966+0000 7fe197e00640 1 -- 172.21.15.28:0/1273020557 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00b1e60 con 0x7fe1a00b2380 2024-09-13T17:25:45.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.968+0000 7fe197e00640 1 --2- 172.21.15.28:0/1273020557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00b2380 0x7fe1a00b27e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe18802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4394048d471ec3ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.968+0000 7fe19e000640 1 -- 172.21.15.28:0/1273020557 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188032440 con 0x7fe1a00b2380 2024-09-13T17:25:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.968+0000 7fe19e000640 1 -- 172.21.15.28:0/1273020557 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1880325a0 con 0x7fe1a00b2380 2024-09-13T17:25:45.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.968+0000 7fe19e000640 1 -- 172.21.15.28:0/1273020557 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188032930 con 0x7fe1a00b2380 2024-09-13T17:25:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 -- 172.21.15.28:0/1273020557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00b2380 msgr2=0x7fe1a00b27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 --2- 172.21.15.28:0/1273020557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00b2380 0x7fe1a00b27e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe1880097a0 tx=0x7fe18802a230 comp rx=0 tx=0).stop 2024-09-13T17:25:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 -- 172.21.15.28:0/1273020557 shutdown_connections 2024-09-13T17:25:45.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 --2- 172.21.15.28:0/1273020557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a00b2380 0x7fe1a00b27e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 -- 172.21.15.28:0/1273020557 >> 172.21.15.28:0/1273020557 conn(0x7fe1a0069e80 msgr2=0x7fe1a006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 -- 172.21.15.28:0/1273020557 shutdown_connections 2024-09-13T17:25:45.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.969+0000 7fe1a5c00640 1 -- 172.21.15.28:0/1273020557 wait complete. 2024-09-13T17:25:45.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.971+0000 7fe1a5c00640 1 Processor -- start 2024-09-13T17:25:45.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.971+0000 7fe1a5c00640 1 -- start start 2024-09-13T17:25:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 0x7fe1a00cd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a0157000 0x7fe1a00cd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015cdd0 0x7fe1a0140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a01503f0 con 0x7fe1a00b2380 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0154990 con 0x7fe1a0157000 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe1a5c00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a0154c70 con 0x7fe1a015cdd0 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe19ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 0x7fe1a00cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe19ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 0x7fe1a00cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53150/0 (socket says 172.21.15.28:53150) 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe19ea00640 1 -- 172.21.15.28:0/713904658 learned_addr learned my addr 172.21.15.28:0/713904658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe197e00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a0157000 0x7fe1a00cd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.972+0000 7fe19f400640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015cdd0 0x7fe1a0140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 -- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015cdd0 msgr2=0x7fe1a0140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe1a015cdd0 0x7fe1a0140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 -- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a0157000 msgr2=0x7fe1a00cd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a0157000 0x7fe1a00cd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe188009400 con 0x7fe1a00b2380 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe197e00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe1a0157000 0x7fe1a00cd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe19ea00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 0x7fe1a00cd070 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c500 tx=0x7fe19000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190005ce0 con 0x7fe1a00b2380 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe190005e40 con 0x7fe1a00b2380 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe190007330 con 0x7fe1a00b2380 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a0154f50 con 0x7fe1a00b2380 2024-09-13T17:25:45.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.973+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a0155750 con 0x7fe1a00b2380 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.974+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe1a00b1850 con 0x7fe1a00b2380 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.975+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe190007550 con 0x7fe1a00b2380 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe196a00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac00 0x7fe17c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe1900887a0 con 0x7fe1a00b2380 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe196a00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7fe17c070b00 0x7fe17c072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe17c0735d0 con 0x7fe17c070b00 2024-09-13T17:25:45.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe19008ad80 con 0x7fe1a00b2380 2024-09-13T17:25:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe197e00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac00 0x7fe17c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.976+0000 7fe19f400640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7fe17c070b00 0x7fe17c072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.977+0000 7fe19f400640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7fe17c070b00 0x7fe17c072f20 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.977+0000 7fe197e00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac00 0x7fe17c06d0c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fe188034000 tx=0x7fe18802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:45.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.978+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== osd.1 v2:172.21.15.28:6808/2722659727 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fe17c0735d0 con 0x7fe17c070b00 2024-09-13T17:25:45.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.990+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 --> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe1a015d600 con 0x7fe17c070b00 2024-09-13T17:25:45.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe196a00640 1 -- 172.21.15.28:0/713904658 <== osd.1 v2:172.21.15.28:6808/2722659727 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe1a015d600 con 0x7fe17c070b00 2024-09-13T17:25:45.992 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:45.992 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:45.992 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:45.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7fe17c070b00 msgr2=0x7fe17c072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:6808/2722659727,v1:172.21.15.28:6809/2722659727] conn(0x7fe17c070b00 0x7fe17c072f20 crc :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac00 msgr2=0x7fe17c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe17c06ac00 0x7fe17c06d0c0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7fe188034000 tx=0x7fe18802e000 comp rx=0 tx=0).stop 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 msgr2=0x7fe1a00cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 --2- 172.21.15.28:0/713904658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1a00b2380 0x7fe1a00cd070 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fe19000c500 tx=0x7fe19000c9c0 comp rx=0 tx=0).stop 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe19f400640 1 -- 172.21.15.28:0/713904658 reap_dead start 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 shutdown_connections 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 >> 172.21.15.28:0/713904658 conn(0x7fe1a0069e80 msgr2=0x7fe1a015b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 shutdown_connections 2024-09-13T17:25:45.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:45.991+0000 7fe1a5c00640 1 -- 172.21.15.28:0/713904658 wait complete. 2024-09-13T17:25:46.007 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T17:25:46.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.162+0000 7fa2efe00640 1 Processor -- start 2024-09-13T17:25:46.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.162+0000 7fa2efe00640 1 -- start start 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.163+0000 7fa2efe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2e8070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2efe00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fa2e806c250 0x7fa2e80712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2efe00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e806c8b0 con 0x7fa2e815ab60 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2efe00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e806c6a0 con 0x7fa2e806c250 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2efe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e806cb90 con 0x7fa2e815a780 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2eca00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fa2e815ab60 0x7fa2e806bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53212/0 (socket says 172.21.15.28:53212) 2024-09-13T17:25:46.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2eca00640 1 -- 172.21.15.28:0/3131607889 learned_addr learned my addr 172.21.15.28:0/3131607889 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:46.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2e8070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2ede00640 1 --2- 172.21.15.28:0/3131607889 >> v2:172.21.15.189:3300/0 conn(0x7fa2e806c250 0x7fa2e80712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632885470 0 0) 0x7fa2e806c8b0 con 0x7fa2e815ab60 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.164+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d0003520 con 0x7fa2e815ab60 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864365334 0 0) 0x7fa2d0003520 con 0x7fa2e815ab60 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 msgr2=0x7fa2e8070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2e8070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 >> v2:172.21.15.189:3300/0 conn(0x7fa2e806c250 msgr2=0x7fa2e80712c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 --2- 172.21.15.28:0/3131607889 >> v2:172.21.15.189:3300/0 conn(0x7fa2e806c250 0x7fa2e80712c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e80cc980 con 0x7fa2e815ab60 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7fa2d4001670 con 0x7fa2e815ab60 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 >> v1:172.21.15.28:6789/0 conn(0x7fa2e815ab60 legacy=0x7fa2e806bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0003970 0x7fa2d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d0004d00 0x7fa2d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e806c6a0 con 0x7fa2d0003970 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e806cb90 con 0x7fa2d0004d00 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d000b980 con 0x7fa2e815a780 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0003970 0x7fa2d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2eca00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.165+0000 7fa2ede00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d0004d00 0x7fa2d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.166+0000 7fa2ed400640 1 -- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 msgr2=0x7fa2d00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.166+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2e815a780 0x7fa2d00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.166+0000 7fa2ed400640 1 -- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d0004d00 msgr2=0x7fa2d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.166+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2d0004d00 0x7fa2d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.166+0000 7fa2ed400640 1 -- 172.21.15.28:0/3131607889 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d0003520 con 0x7fa2d0003970 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0003970 0x7fa2d0003d90 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009ac0 tx=0x7fa2d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f1e650aacbd5db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc031400 con 0x7fa2d0003970 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc02d050 con 0x7fa2d0003970 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2e7e00640 1 -- 172.21.15.28:0/3131607889 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc031780 con 0x7fa2d0003970 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2efe00640 1 -- 172.21.15.28:0/3131607889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0003970 msgr2=0x7fa2d0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3131607889 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2d0003970 0x7fa2d0003d90 secure :-1 s=READY pgs=2893 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc009ac0 tx=0x7fa2d00057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.167+0000 7fa2ede00640 1 -- 172.21.15.28:0/3131607889 reap_dead start 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.168+0000 7fa2efe00640 1 -- 172.21.15.28:0/3131607889 shutdown_connections 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.168+0000 7fa2efe00640 1 -- 172.21.15.28:0/3131607889 >> 172.21.15.28:0/3131607889 conn(0x7fa2e80b0270 msgr2=0x7fa2e80b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.168+0000 7fa2efe00640 1 -- 172.21.15.28:0/3131607889 shutdown_connections 2024-09-13T17:25:46.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.168+0000 7fa2efe00640 1 -- 172.21.15.28:0/3131607889 wait complete. 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.169+0000 7fa2efe00640 1 Processor -- start 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.170+0000 7fa2efe00640 1 -- start start 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.170+0000 7fa2efe00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2d0003970 0x7fa2e80b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.170+0000 7fa2efe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2efe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e814b150 0x7fa2e8152280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2efe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e80b2720 con 0x7fa2e80b29f0 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2efe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e814c660 con 0x7fa2e814b150 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2efe00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81534a0 con 0x7fa2d0003970 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53168/0 (socket says 172.21.15.28:53168) 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 -- 172.21.15.28:0/3627478946 learned_addr learned my addr 172.21.15.28:0/3627478946 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2d0003970 0x7fa2e80b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2ede00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e814b150 0x7fa2e8152280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 -- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2d0003970 msgr2=0x7fa2e80b2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2d0003970 0x7fa2e80b2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 -- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e814b150 msgr2=0x7fa2e8152280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e814b150 0x7fa2e8152280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.171+0000 7fa2eca00640 1 -- 172.21.15.28:0/3627478946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8153780 con 0x7fa2e80b29f0 2024-09-13T17:25:46.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2eca00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007340 tx=0x7fa2d4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d400f8c0 con 0x7fa2e80b29f0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d400fa60 con 0x7fa2e80b29f0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d400fe10 con 0x7fa2e80b29f0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e8153a60 con 0x7fa2e80b29f0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.172+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e814b5d0 con 0x7fa2e80b29f0 2024-09-13T17:25:46.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.173+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e8061090 con 0x7fa2e80b29f0 2024-09-13T17:25:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.174+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fa2d401d460 con 0x7fa2e80b29f0 2024-09-13T17:25:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.175+0000 7fa2e5600640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 0x7fa2c806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.175+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fa2d40891b0 con 0x7fa2e80b29f0 2024-09-13T17:25:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.175+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 0x7fa2c806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.176+0000 7fa2ed400640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 0x7fa2c806d0f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc033000 tx=0x7fa2dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.180+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fa2d40546d0 con 0x7fa2e80b29f0 2024-09-13T17:25:46.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.358+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2e8153610 con 0x7fa2c806ac30 2024-09-13T17:25:46.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.359+0000 7fa2e5600640 1 -- 172.21.15.28:0/3627478946 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa2e8153610 con 0x7fa2c806ac30 2024-09-13T17:25:46.360 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 msgr2=0x7fa2c806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 0x7fa2c806d0f0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc033000 tx=0x7fa2dc008030 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 msgr2=0x7fa2e81f2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fa2d4007340 tx=0x7fa2d4007800 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 shutdown_connections 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fa2c806ac30 0x7fa2c806d0f0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa2e814b150 0x7fa2e8152280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa2e80b29f0 0x7fa2e81f2280 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 --2- 172.21.15.28:0/3627478946 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fa2d0003970 0x7fa2e80b2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 >> 172.21.15.28:0/3627478946 conn(0x7fa2e80b0270 msgr2=0x7fa2e80ae1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.361+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 shutdown_connections 2024-09-13T17:25:46.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.362+0000 7fa2efe00640 1 -- 172.21.15.28:0/3627478946 wait complete. 2024-09-13T17:25:46.379 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T17:25:46.379 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.538+0000 7fd896800640 1 Processor -- start 2024-09-13T17:25:46.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.538+0000 7fd896800640 1 -- start start 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd896800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fd89015a7a0 0x7fd8900ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd896800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8900ce9d0 0x7fd8900b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd896800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89006c950 con 0x7fd8900b1ab0 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd896800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900cee40 con 0x7fd8900ce9d0 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd896800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8900b1ed0 con 0x7fd89015a7a0 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8900ce9d0 0x7fd8900b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fd8900ce9d0 0x7fd8900b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53178/0 (socket says 172.21.15.28:53178) 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 -- 172.21.15.28:0/900430617 learned_addr learned my addr 172.21.15.28:0/900430617 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:46.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd88f400640 1 --2- 172.21.15.28:0/900430617 >> v2:172.21.15.189:3300/0 conn(0x7fd89015a7a0 0x7fd8900ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 -- 172.21.15.28:0/900430617 >> v2:172.21.15.189:3300/0 conn(0x7fd89015a7a0 msgr2=0x7fd8900ce490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 --2- 172.21.15.28:0/900430617 >> v2:172.21.15.189:3300/0 conn(0x7fd89015a7a0 0x7fd8900ce490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 -- 172.21.15.28:0/900430617 >> v1:172.21.15.189:6789/0 conn(0x7fd8900b1ab0 legacy=0x7fd89006b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.540+0000 7fd887e00640 1 -- 172.21.15.28:0/900430617 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8900b2150 con 0x7fd8900ce9d0 2024-09-13T17:25:46.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.541+0000 7fd887e00640 1 --2- 172.21.15.28:0/900430617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900ce9d0 0x7fd8900b1570 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fd8900716c0 tx=0x7fd87802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad23861b3a08a9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd88ea00640 1 -- 172.21.15.28:0/900430617 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878032440 con 0x7fd8900ce9d0 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd88ea00640 1 -- 172.21.15.28:0/900430617 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8780325a0 con 0x7fd8900ce9d0 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd88ea00640 1 -- 172.21.15.28:0/900430617 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd878032930 con 0x7fd8900ce9d0 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 -- 172.21.15.28:0/900430617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900ce9d0 msgr2=0x7fd8900b1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 --2- 172.21.15.28:0/900430617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900ce9d0 0x7fd8900b1570 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fd8900716c0 tx=0x7fd87802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 -- 172.21.15.28:0/900430617 shutdown_connections 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 --2- 172.21.15.28:0/900430617 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900ce9d0 0x7fd8900b1570 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 --2- 172.21.15.28:0/900430617 >> v2:172.21.15.189:3300/0 conn(0x7fd89015a7a0 0x7fd8900ce490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 -- 172.21.15.28:0/900430617 >> 172.21.15.28:0/900430617 conn(0x7fd890069e80 msgr2=0x7fd89006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 -- 172.21.15.28:0/900430617 shutdown_connections 2024-09-13T17:25:46.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.542+0000 7fd896800640 1 -- 172.21.15.28:0/900430617 wait complete. 2024-09-13T17:25:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.544+0000 7fd896800640 1 Processor -- start 2024-09-13T17:25:46.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.544+0000 7fd896800640 1 -- start start 2024-09-13T17:25:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 0x7fd8901526b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8900ce9d0 0x7fd8901f3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89015a7a0 0x7fd8901f4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890150240 con 0x7fd8900b1ab0 2024-09-13T17:25:46.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890157d40 con 0x7fd8900ce9d0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd896800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd890158020 con 0x7fd89015a7a0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd88f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 0x7fd8901526b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd88f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 0x7fd8901526b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53192/0 (socket says 172.21.15.28:53192) 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.545+0000 7fd88f400640 1 -- 172.21.15.28:0/2064476802 learned_addr learned my addr 172.21.15.28:0/2064476802 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89015a7a0 0x7fd8901f4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd887e00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8900ce9d0 0x7fd8901f3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 -- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89015a7a0 msgr2=0x7fd8901f4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89015a7a0 0x7fd8901f4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 -- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8900ce9d0 msgr2=0x7fd8901f3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8900ce9d0 0x7fd8901f3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8780094d0 con 0x7fd8900b1ab0 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd887e00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8900ce9d0 0x7fd8901f3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fd89015a7a0 0x7fd8901f4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:46.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd88f400640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 0x7fd8901526b0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9b0 tx=0x7fd88000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880009670 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd88000c9a0 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd890150420 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88000cd50 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.546+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd890152bf0 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.547+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd89006caf0 con 0x7fd8900b1ab0 2024-09-13T17:25:46.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.548+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fd880013470 con 0x7fd8900b1ab0 2024-09-13T17:25:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.549+0000 7fd887400640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.549+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fd8800893a0 con 0x7fd8900b1ab0 2024-09-13T17:25:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.549+0000 7fd887400640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7fd86c070b00 0x7fd86c072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.549+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd86c0735d0 con 0x7fd86c070b00 2024-09-13T17:25:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.549+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd88008b980 con 0x7fd8900b1ab0 2024-09-13T17:25:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.550+0000 7fd887e00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.550+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7fd86c070b00 0x7fd86c072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.550+0000 7fd88fe00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7fd86c070b00 0x7fd86c072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.550+0000 7fd887e00640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd878034000 tx=0x7fd87802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.551+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== osd.2 v2:172.21.15.28:6816/2959928040 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fd86c0735d0 con 0x7fd86c070b00 2024-09-13T17:25:46.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 --> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd8900ccc40 con 0x7fd86c070b00 2024-09-13T17:25:46.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd887400640 1 -- 172.21.15.28:0/2064476802 <== osd.2 v2:172.21.15.28:6816/2959928040 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd888001e00 con 0x7fd86c070b00 2024-09-13T17:25:46.565 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:46.565 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:46.565 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7fd86c070b00 msgr2=0x7fd86c072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:6816/2959928040,v1:172.21.15.28:6819/2959928040] conn(0x7fd86c070b00 0x7fd86c072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 msgr2=0x7fd86c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fd86c06ac00 0x7fd86c06d0c0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7fd878034000 tx=0x7fd87802e000 comp rx=0 tx=0).stop 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 msgr2=0x7fd8901526b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd896800640 1 --2- 172.21.15.28:0/2064476802 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8900b1ab0 0x7fd8901526b0 secure :-1 s=READY pgs=2896 cs=0 l=1 rev1=1 crypto rx=0x7fd88000e9b0 tx=0x7fd88000ee70 comp rx=0 tx=0).stop 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.564+0000 7fd88fe00640 1 -- 172.21.15.28:0/2064476802 reap_dead start 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.565+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 shutdown_connections 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.565+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 >> 172.21.15.28:0/2064476802 conn(0x7fd890069e80 msgr2=0x7fd8900cd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.565+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 shutdown_connections 2024-09-13T17:25:46.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.565+0000 7fd896800640 1 -- 172.21.15.28:0/2064476802 wait complete. 2024-09-13T17:25:46.580 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T17:25:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.735+0000 7f5175800640 1 Processor -- start 2024-09-13T17:25:46.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.735+0000 7f5175800640 1 -- start start 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f517015a780 0x7f5170070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517015ab60 0x7f51700712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517006bb40 0x7f517006bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517006c8b0 con 0x7f517015ab60 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f517006cb90 con 0x7f517015a780 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5175800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51700cc980 con 0x7f517006bb40 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5167e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517015ab60 0x7f51700712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5167e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f517015ab60 0x7f51700712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53204/0 (socket says 172.21.15.28:53204) 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.736+0000 7f5167e00640 1 -- 172.21.15.28:0/604552992 learned_addr learned my addr 172.21.15.28:0/604552992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f516f400640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517006bb40 0x7f517006bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f516ea00640 1 --2- 172.21.15.28:0/604552992 >> v2:172.21.15.189:3300/0 conn(0x7f517015a780 0x7f5170070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f5167e00640 1 -- 172.21.15.28:0/604552992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517006bb40 msgr2=0x7f517006bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f5167e00640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517006bb40 0x7f517006bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f5167e00640 1 -- 172.21.15.28:0/604552992 >> v2:172.21.15.189:3300/0 conn(0x7f517015a780 msgr2=0x7f5170070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:46.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f5167e00640 1 --2- 172.21.15.28:0/604552992 >> v2:172.21.15.189:3300/0 conn(0x7f517015a780 0x7f5170070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.737+0000 7f5167e00640 1 -- 172.21.15.28:0/604552992 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51700ccc60 con 0x7f517015ab60 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.738+0000 7f5167e00640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015ab60 0x7f51700712c0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f51700b1330 tx=0x7f515802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b0151fa9ba304 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.738+0000 7f516e000640 1 -- 172.21.15.28:0/604552992 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158032440 con 0x7f517015ab60 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.738+0000 7f516e000640 1 -- 172.21.15.28:0/604552992 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51580325a0 con 0x7f517015ab60 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.738+0000 7f516e000640 1 -- 172.21.15.28:0/604552992 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5158032930 con 0x7f517015ab60 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 -- 172.21.15.28:0/604552992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015ab60 msgr2=0x7f51700712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015ab60 0x7f51700712c0 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f51700b1330 tx=0x7f515802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 -- 172.21.15.28:0/604552992 shutdown_connections 2024-09-13T17:25:46.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015ab60 0x7f51700712c0 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 --2- 172.21.15.28:0/604552992 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517006bb40 0x7f517006bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 --2- 172.21.15.28:0/604552992 >> v2:172.21.15.189:3300/0 conn(0x7f517015a780 0x7f5170070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 -- 172.21.15.28:0/604552992 >> 172.21.15.28:0/604552992 conn(0x7f51700b0270 msgr2=0x7f51700b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 -- 172.21.15.28:0/604552992 shutdown_connections 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.739+0000 7f5175800640 1 -- 172.21.15.28:0/604552992 wait complete. 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.741+0000 7f5175800640 1 Processor -- start 2024-09-13T17:25:46.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.741+0000 7f5175800640 1 -- start start 2024-09-13T17:25:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 0x7f5170143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 0x7f51701f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5170149470 con 0x7f517015a780 2024-09-13T17:25:46.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5170158360 con 0x7f517006bb40 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5175800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5170157750 con 0x7f517015ab60 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53208/0 (socket says 172.21.15.28:53208) 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 -- 172.21.15.28:0/3850267888 learned_addr learned my addr 172.21.15.28:0/3850267888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f516f400640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 0x7f51701f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f516ea00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 0x7f5170143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 -- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 msgr2=0x7f51701f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 0x7f51701f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 -- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 msgr2=0x7f5170143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 0x7f5170143740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f5167e00640 1 -- 172.21.15.28:0/3850267888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580094d0 con 0x7f517015a780 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f516f400640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 0x7f51701f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:46.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.742+0000 7f516ea00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 0x7f5170143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5167e00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f5158000c00 tx=0x7f51580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515803d830 con 0x7f517015a780 2024-09-13T17:25:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515803d990 con 0x7f517015a780 2024-09-13T17:25:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515803dd40 con 0x7f517015a780 2024-09-13T17:25:46.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5170157a30 con 0x7f517015a780 2024-09-13T17:25:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.743+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5170157ff0 con 0x7f517015a780 2024-09-13T17:25:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.744+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5170158180 con 0x7f517015a780 2024-09-13T17:25:46.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.745+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f5158039040 con 0x7f517015a780 2024-09-13T17:25:46.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.746+0000 7f5166a00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 0x7f514c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:46.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.746+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f51580a9b30 con 0x7f517015a780 2024-09-13T17:25:46.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.746+0000 7f516ea00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 0x7f514c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:46.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.747+0000 7f516ea00640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 0x7f514c06d0c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f51701449d0 tx=0x7f5160008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:46.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.750+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5158074fa0 con 0x7f517015a780 2024-09-13T17:25:46.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.929+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f517006f3f0 con 0x7f514c06ac00 2024-09-13T17:25:46.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.930+0000 7f5166a00640 1 -- 172.21.15.28:0/3850267888 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5160004030 con 0x7f514c06ac00 2024-09-13T17:25:46.931 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 msgr2=0x7f514c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 0x7f514c06d0c0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f51701449d0 tx=0x7f5160008040 comp rx=0 tx=0).stop 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 msgr2=0x7f5170149d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f5158000c00 tx=0x7f51580046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 shutdown_connections 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f514c06ac00 0x7f514c06d0c0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f517015ab60 0x7f51701f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f517015a780 0x7f5170149d70 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 --2- 172.21.15.28:0/3850267888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f517006bb40 0x7f5170143740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 >> 172.21.15.28:0/3850267888 conn(0x7f51700b0270 msgr2=0x7f51700b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 shutdown_connections 2024-09-13T17:25:46.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:46.932+0000 7f5175800640 1 -- 172.21.15.28:0/3850267888 wait complete. 2024-09-13T17:25:46.948 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T17:25:46.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:47.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.107+0000 7f7e23000640 1 Processor -- start 2024-09-13T17:25:47.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.107+0000 7f7e23000640 1 -- start start 2024-09-13T17:25:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e23000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cd300 0x7f7e1c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e23000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e1c0b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e23000640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1c06ca10 con 0x7f7e1c143d90 2024-09-13T17:25:47.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e23000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1c144170 con 0x7f7e1c0cd300 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e23000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c06be00 con 0x7f7e1c0b1400 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e21c00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f7e1c143d90 0x7f7e1c06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53218/0 (socket says 172.21.15.28:53218) 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e21c00640 1 -- 172.21.15.28:0/490835594 learned_addr learned my addr 172.21.15.28:0/490835594 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e21200640 1 --2- 172.21.15.28:0/490835594 >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cd300 0x7f7e1c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e22600640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e1c0b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374829179 0 0) 0x7f7e1c06ca10 con 0x7f7e1c143d90 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.109+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e04003520 con 0x7f7e1c143d90 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2693425236 0 0) 0x7f7e04003520 con 0x7f7e1c143d90 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 msgr2=0x7f7e1c0b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e1c0b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cd300 msgr2=0x7f7e1c0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 --2- 172.21.15.28:0/490835594 >> v2:172.21.15.189:3300/0 conn(0x7f7e1c0cd300 0x7f7e1c0cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c06c0e0 con 0x7f7e1c143d90 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f7e10001670 con 0x7f7e1c143d90 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 >> v1:172.21.15.28:6789/0 conn(0x7f7e1c143d90 legacy=0x7f7e1c06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e04003970 0x7f7e04003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e040047c0 0x7f7e04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c144170 con 0x7f7e040047c0 2024-09-13T17:25:47.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c06be00 con 0x7f7e04003970 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0400b980 con 0x7f7e1c0b1400 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e21200640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e040047c0 0x7f7e04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e21c00640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e04003970 0x7f7e04003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.110+0000 7f7e22600640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.111+0000 7f7e21200640 1 -- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 msgr2=0x7f7e04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.111+0000 7f7e21200640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e1c0b1400 0x7f7e04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.111+0000 7f7e21200640 1 -- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e04003970 msgr2=0x7f7e04003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.111+0000 7f7e21200640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e04003970 0x7f7e04003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.111+0000 7f7e21200640 1 -- 172.21.15.28:0/490835594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e04003520 con 0x7f7e040047c0 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e21200640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e040047c0 0x7f7e04004c40 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009ac0 tx=0x7f7e040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ef6cf8c9fbffb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08031400 con 0x7f7e040047c0 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e0802d050 con 0x7f7e040047c0 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e1be00640 1 -- 172.21.15.28:0/490835594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e08031780 con 0x7f7e040047c0 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e23000640 1 -- 172.21.15.28:0/490835594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e040047c0 msgr2=0x7f7e04004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.112+0000 7f7e23000640 1 --2- 172.21.15.28:0/490835594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e040047c0 0x7f7e04004c40 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7f7e08009ac0 tx=0x7f7e040057f0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.113+0000 7f7e22600640 1 -- 172.21.15.28:0/490835594 reap_dead start 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.113+0000 7f7e23000640 1 -- 172.21.15.28:0/490835594 shutdown_connections 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.113+0000 7f7e23000640 1 -- 172.21.15.28:0/490835594 >> 172.21.15.28:0/490835594 conn(0x7f7e1c073470 msgr2=0x7f7e1c069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.113+0000 7f7e23000640 1 -- 172.21.15.28:0/490835594 shutdown_connections 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.113+0000 7f7e23000640 1 -- 172.21.15.28:0/490835594 wait complete. 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.114+0000 7f7e23000640 1 Processor -- start 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.115+0000 7f7e23000640 1 -- start start 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.115+0000 7f7e23000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e040047c0 0x7f7e1c13e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.115+0000 7f7e23000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c1414e0 0x7f7e1c159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e23000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 0x7f7e1c0accd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e23000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c0ad210 con 0x7f7e1c0ac850 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e23000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c1424f0 con 0x7f7e1c1414e0 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e23000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1c06f4c0 con 0x7f7e040047c0 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 0x7f7e1c0accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 0x7f7e1c0accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53232/0 (socket says 172.21.15.28:53232) 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 -- 172.21.15.28:0/3517470583 learned_addr learned my addr 172.21.15.28:0/3517470583 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e21c00640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e040047c0 0x7f7e1c13e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e21200640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c1414e0 0x7f7e1c159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 -- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e040047c0 msgr2=0x7f7e1c13e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f7e040047c0 0x7f7e1c13e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 -- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c1414e0 msgr2=0x7f7e1c159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c1414e0 0x7f7e1c159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e22600640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1c141c50 con 0x7f7e1c0ac850 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.116+0000 7f7e21200640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e1c1414e0 0x7f7e1c159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e22600640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 0x7f7e1c0accd0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0b3360 tx=0x7f7e1400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e1400e040 con 0x7f7e1c0ac850 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e1400f040 con 0x7f7e1c0ac850 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e14015430 con 0x7f7e1c0ac850 2024-09-13T17:25:47.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1c141f30 con 0x7f7e1c0ac850 2024-09-13T17:25:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.117+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1c141960 con 0x7f7e1c0ac850 2024-09-13T17:25:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.118+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e1c0cdcb0 con 0x7f7e1c0ac850 2024-09-13T17:25:47.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.119+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f7e14015590 con 0x7f7e1c0ac850 2024-09-13T17:25:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e19600640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7dfc06ac00 0x7f7dfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f7e14089650 con 0x7f7e1c0ac850 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e19600640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f7dfc070b00 0x7f7dfc072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7dfc0735d0 con 0x7f7dfc070b00 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e1408bc30 con 0x7f7e1c0ac850 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e21c00640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7dfc06ac00 0x7f7dfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.120+0000 7f7e21200640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f7dfc070b00 0x7f7dfc072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.121+0000 7f7e21200640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f7dfc070b00 0x7f7dfc072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.121+0000 7f7e21c00640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7dfc06ac00 0x7f7dfc06d0c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001050 tx=0x7f7e100078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.122+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== osd.3 v2:172.21.15.28:6814/3941122226 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f7dfc0735d0 con 0x7f7dfc070b00 2024-09-13T17:25:47.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.134+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 --> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7e1c06c6a0 con 0x7f7dfc070b00 2024-09-13T17:25:47.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e19600640 1 -- 172.21.15.28:0/3517470583 <== osd.3 v2:172.21.15.28:6814/3941122226 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f7e08004350 con 0x7f7dfc070b00 2024-09-13T17:25:47.136 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:47.136 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:47.136 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f7dfc070b00 msgr2=0x7f7dfc072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:6814/3941122226,v1:172.21.15.28:6817/3941122226] conn(0x7f7dfc070b00 0x7f7dfc072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7dfc06ac00 msgr2=0x7f7dfc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f7dfc06ac00 0x7f7dfc06d0c0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f7e10001050 tx=0x7f7e100078f0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 msgr2=0x7f7e1c0accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 --2- 172.21.15.28:0/3517470583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e1c0ac850 0x7f7e1c0accd0 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c0b3360 tx=0x7f7e1400f1b0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e22600640 1 -- 172.21.15.28:0/3517470583 reap_dead start 2024-09-13T17:25:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 shutdown_connections 2024-09-13T17:25:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 >> 172.21.15.28:0/3517470583 conn(0x7f7e1c073470 msgr2=0x7f7e1c15d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.135+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 shutdown_connections 2024-09-13T17:25:47.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.136+0000 7f7e23000640 1 -- 172.21.15.28:0/3517470583 wait complete. 2024-09-13T17:25:47.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T17:25:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.308+0000 7fe89be00640 1 Processor -- start 2024-09-13T17:25:47.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.308+0000 7fe89be00640 1 -- start start 2024-09-13T17:25:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe89be00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe89c0cd7c0 0x7fe89c0c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe89be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0c3ba0 0x7fe89c070440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe89be00640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c15ef70 con 0x7fe89c143f70 2024-09-13T17:25:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe89be00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0c4320 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe89be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c15e1b0 con 0x7fe89c0c3ba0 2024-09-13T17:25:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe893e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe89c0cd7c0 0x7fe89c0c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe893e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe89c0cd7c0 0x7fe89c0c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53242/0 (socket says 172.21.15.28:53242) 2024-09-13T17:25:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.309+0000 7fe893e00640 1 -- 172.21.15.28:0/844407094 learned_addr learned my addr 172.21.15.28:0/844407094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.310+0000 7fe89b400640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0c3ba0 0x7fe89c070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.310+0000 7fe893e00640 1 -- 172.21.15.28:0/844407094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0c3ba0 msgr2=0x7fe89c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.310+0000 7fe893e00640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0c3ba0 0x7fe89c070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.310+0000 7fe893e00640 1 -- 172.21.15.28:0/844407094 >> v1:172.21.15.189:6789/0 conn(0x7fe89c143f70 legacy=0x7fe89c0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.310+0000 7fe893e00640 1 -- 172.21.15.28:0/844407094 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c15e490 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.311+0000 7fe893e00640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c0cd7c0 0x7fe89c0c3660 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fe884009870 tx=0x7fe88402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e088868ac46a0d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.311+0000 7fe89a000640 1 -- 172.21.15.28:0/844407094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884032440 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.311+0000 7fe89a000640 1 -- 172.21.15.28:0/844407094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8840325a0 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.311+0000 7fe89a000640 1 -- 172.21.15.28:0/844407094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe884032930 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 -- 172.21.15.28:0/844407094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c0cd7c0 msgr2=0x7fe89c0c3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c0cd7c0 0x7fe89c0c3660 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fe884009870 tx=0x7fe88402a250 comp rx=0 tx=0).stop 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 -- 172.21.15.28:0/844407094 shutdown_connections 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0c3ba0 0x7fe89c070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 --2- 172.21.15.28:0/844407094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c0cd7c0 0x7fe89c0c3660 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 -- 172.21.15.28:0/844407094 >> 172.21.15.28:0/844407094 conn(0x7fe89c069e80 msgr2=0x7fe89c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 -- 172.21.15.28:0/844407094 shutdown_connections 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.312+0000 7fe89be00640 1 -- 172.21.15.28:0/844407094 wait complete. 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.314+0000 7fe89be00640 1 Processor -- start 2024-09-13T17:25:47.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.314+0000 7fe89be00640 1 -- start start 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 0x7fe89c0ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 0x7fe89c159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c13ae10 con 0x7fe89c143f70 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c13c350 con 0x7fe89c0c3ba0 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89be00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0b2e50 con 0x7fe89c0cd7c0 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53246/0 (socket says 172.21.15.28:53246) 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 -- 172.21.15.28:0/1127845126 learned_addr learned my addr 172.21.15.28:0/1127845126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe893e00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 0x7fe89c159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89aa00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 0x7fe89c0ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 -- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 msgr2=0x7fe89c159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 0x7fe89c159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 -- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 msgr2=0x7fe89c0ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 0x7fe89c0ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89b400640 1 -- 172.21.15.28:0/1127845126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8840094d0 con 0x7fe89c143f70 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe893e00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 0x7fe89c159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.315+0000 7fe89aa00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 0x7fe89c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe89b400640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fe894007e40 tx=0x7fe89400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe894014620 con 0x7fe89c143f70 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89400ee90 con 0x7fe89c143f70 2024-09-13T17:25:47.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8940149e0 con 0x7fe89c143f70 2024-09-13T17:25:47.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0cc210 con 0x7fe89c143f70 2024-09-13T17:25:47.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.316+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c0b2b70 con 0x7fe89c143f70 2024-09-13T17:25:47.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.318+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c140230 con 0x7fe89c143f70 2024-09-13T17:25:47.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.318+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe894014b40 con 0x7fe89c143f70 2024-09-13T17:25:47.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.319+0000 7fe892a00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 0x7fe87806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.319+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe894089ae0 con 0x7fe89c143f70 2024-09-13T17:25:47.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.320+0000 7fe89aa00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 0x7fe87806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.320+0000 7fe89aa00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 0x7fe87806d0c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe88c007fd0 tx=0x7fe88c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.323+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe894054ec0 con 0x7fe89c143f70 2024-09-13T17:25:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.506+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe89c0c3fd0 con 0x7fe87806ac00 2024-09-13T17:25:47.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.506+0000 7fe892a00640 1 -- 172.21.15.28:0/1127845126 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe88c001c90 con 0x7fe87806ac00 2024-09-13T17:25:47.508 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T17:25:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 msgr2=0x7fe87806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 0x7fe87806d0c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe88c007fd0 tx=0x7fe88c009040 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 msgr2=0x7fe89c06b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 secure :-1 s=READY pgs=2903 cs=0 l=1 rev1=1 crypto rx=0x7fe894007e40 tx=0x7fe89400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 shutdown_connections 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe87806ac00 0x7fe87806d0c0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe89c143f70 0x7fe89c06b330 unknown :-1 s=CLOSED pgs=2903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe89c0cd7c0 0x7fe89c159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 --2- 172.21.15.28:0/1127845126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe89c0c3ba0 0x7fe89c0ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 >> 172.21.15.28:0/1127845126 conn(0x7fe89c069e80 msgr2=0x7fe89c070db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 shutdown_connections 2024-09-13T17:25:47.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.509+0000 7fe89be00640 1 -- 172.21.15.28:0/1127845126 wait complete. 2024-09-13T17:25:47.527 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T17:25:47.527 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.685+0000 7f601e800640 1 Processor -- start 2024-09-13T17:25:47.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.685+0000 7f601e800640 1 -- start start 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f601e800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6018159590 0x7f6018070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f601e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6018159970 0x7f60180b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f601e800640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601815a4c0 con 0x7f60180b2630 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f601e800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180b2ae0 con 0x7f6018159590 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f601e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6018071580 con 0x7f6018159970 2024-09-13T17:25:47.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6018159590 0x7f6018070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f6018159590 0x7f6018070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53262/0 (socket says 172.21.15.28:53262) 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 -- 172.21.15.28:0/2485543880 learned_addr learned my addr 172.21.15.28:0/2485543880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f600ea00640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6018159970 0x7f60180b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 -- 172.21.15.28:0/2485543880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6018159970 msgr2=0x7f60180b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6018159970 0x7f60180b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 -- 172.21.15.28:0/2485543880 >> v1:172.21.15.189:6789/0 conn(0x7f60180b2630 legacy=0x7f601806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.687+0000 7f6017400640 1 -- 172.21.15.28:0/2485543880 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018159e70 con 0x7f6018159590 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.688+0000 7f6017400640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159590 0x7f6018070c80 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f6008009870 tx=0x7f600802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8327723998dcc332 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f6016a00640 1 -- 172.21.15.28:0/2485543880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008032440 con 0x7f6018159590 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f6016a00640 1 -- 172.21.15.28:0/2485543880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60080325a0 con 0x7f6018159590 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f6016a00640 1 -- 172.21.15.28:0/2485543880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008032930 con 0x7f6018159590 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 -- 172.21.15.28:0/2485543880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159590 msgr2=0x7f6018070c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159590 0x7f6018070c80 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7f6008009870 tx=0x7f600802a270 comp rx=0 tx=0).stop 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 -- 172.21.15.28:0/2485543880 shutdown_connections 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6018159970 0x7f60180b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 --2- 172.21.15.28:0/2485543880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159590 0x7f6018070c80 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.689+0000 7f601e800640 1 -- 172.21.15.28:0/2485543880 >> 172.21.15.28:0/2485543880 conn(0x7f6018069e80 msgr2=0x7f601806a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.690+0000 7f601e800640 1 -- 172.21.15.28:0/2485543880 shutdown_connections 2024-09-13T17:25:47.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.690+0000 7f601e800640 1 -- 172.21.15.28:0/2485543880 wait complete. 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.691+0000 7f601e800640 1 Processor -- start 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.691+0000 7f601e800640 1 -- start start 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60180b2630 0x7f601814e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6018159590 0x7f601814e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 0x7f6018146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6018144810 con 0x7f6018159970 2024-09-13T17:25:47.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60181653b0 con 0x7f6018159590 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f601e800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6018165650 con 0x7f60180b2630 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f6017e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 0x7f6018146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f6017400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60180b2630 0x7f601814e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f6017e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 0x7f6018146d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53264/0 (socket says 172.21.15.28:53264) 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.692+0000 7f6017e00640 1 -- 172.21.15.28:0/4173018511 learned_addr learned my addr 172.21.15.28:0/4173018511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f600ea00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6018159590 0x7f601814e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 -- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60180b2630 msgr2=0x7f601814e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60180b2630 0x7f601814e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 -- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6018159590 msgr2=0x7f601814e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6018159590 0x7f601814e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60080094d0 con 0x7f6018159970 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f6017e00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 0x7f6018146d00 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f601000c7a0 tx=0x7f601000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.693+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010003b70 con 0x7f6018159970 2024-09-13T17:25:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.694+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6010003cf0 con 0x7f6018159970 2024-09-13T17:25:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.694+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010003ea0 con 0x7f6018159970 2024-09-13T17:25:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.694+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6018165930 con 0x7f6018159970 2024-09-13T17:25:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.694+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6018165ef0 con 0x7f6018159970 2024-09-13T17:25:47.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.695+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f601806fb80 con 0x7f6018159970 2024-09-13T17:25:47.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.696+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f601000aa90 con 0x7f6018159970 2024-09-13T17:25:47.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.696+0000 7f600fe00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ff406ac30 0x7f5ff406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6010088e30 con 0x7f6018159970 2024-09-13T17:25:47.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f600fe00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7f5ff4070b30 0x7f5ff4072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ff4073600 con 0x7f5ff4070b30 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f601008b410 con 0x7f6018159970 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f6017400640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ff406ac30 0x7f5ff406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.697+0000 7f600ea00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7f5ff4070b30 0x7f5ff4072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.698+0000 7f6017400640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ff406ac30 0x7f5ff406d0f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6008034000 tx=0x7f600802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.698+0000 7f600ea00640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7f5ff4070b30 0x7f5ff4072f50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.699+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== osd.4 v2:172.21.15.189:6800/531106775 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f5ff4073600 con 0x7f5ff4070b30 2024-09-13T17:25:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.711+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 --> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6018070c80 con 0x7f5ff4070b30 2024-09-13T17:25:47.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f600fe00640 1 -- 172.21.15.28:0/4173018511 <== osd.4 v2:172.21.15.189:6800/531106775 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6018070c80 con 0x7f5ff4070b30 2024-09-13T17:25:47.714 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:47.714 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:47.714 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:47.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7f5ff4070b30 msgr2=0x7f5ff4072f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6800/531106775,v1:172.21.15.189:6801/531106775] conn(0x7f5ff4070b30 0x7f5ff4072f50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ff406ac30 msgr2=0x7f5ff406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f5ff406ac30 0x7f5ff406d0f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6008034000 tx=0x7f600802e000 comp rx=0 tx=0).stop 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 msgr2=0x7f6018146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.712+0000 7f601e800640 1 --2- 172.21.15.28:0/4173018511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6018159970 0x7f6018146d00 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7f601000c7a0 tx=0x7f601000cc60 comp rx=0 tx=0).stop 2024-09-13T17:25:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.713+0000 7f6017e00640 1 -- 172.21.15.28:0/4173018511 reap_dead start 2024-09-13T17:25:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.713+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 shutdown_connections 2024-09-13T17:25:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.713+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 >> 172.21.15.28:0/4173018511 conn(0x7f6018069e80 msgr2=0x7f601814fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.713+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 shutdown_connections 2024-09-13T17:25:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.713+0000 7f601e800640 1 -- 172.21.15.28:0/4173018511 wait complete. 2024-09-13T17:25:47.728 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T17:25:47.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.884+0000 7f582d000640 1 Processor -- start 2024-09-13T17:25:47.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.884+0000 7f582d000640 1 -- start start 2024-09-13T17:25:47.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f582d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f58280cd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f582d000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58280cddb0 0x7f582815aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f582d000640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58280ce1d0 con 0x7f5828070e30 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f582d000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5828159e10 con 0x7f58280cddb0 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f582d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582815a0f0 con 0x7f5828071210 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f5827e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58280cddb0 0x7f582815aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f5827e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f58280cddb0 0x7f582815aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53270/0 (socket says 172.21.15.28:53270) 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f5827e00640 1 -- 172.21.15.28:0/1497451212 learned_addr learned my addr 172.21.15.28:0/1497451212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.885+0000 7f581ea00640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f58280cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.886+0000 7f5827e00640 1 -- 172.21.15.28:0/1497451212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 msgr2=0x7f58280cd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.886+0000 7f5827e00640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f58280cd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.886+0000 7f5827e00640 1 -- 172.21.15.28:0/1497451212 >> v1:172.21.15.189:6789/0 conn(0x7f5828070e30 legacy=0x7f5828071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.886+0000 7f5827e00640 1 -- 172.21.15.28:0/1497451212 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582815a3d0 con 0x7f58280cddb0 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.887+0000 7f5827e00640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58280cddb0 0x7f582815aed0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f5820009e50 tx=0x7f582002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fd8c0cc5f7e893 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.887+0000 7f5826a00640 1 -- 172.21.15.28:0/1497451212 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820031600 con 0x7f58280cddb0 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.887+0000 7f5826a00640 1 -- 172.21.15.28:0/1497451212 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582002d050 con 0x7f58280cddb0 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.887+0000 7f5826a00640 1 -- 172.21.15.28:0/1497451212 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820031980 con 0x7f58280cddb0 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 -- 172.21.15.28:0/1497451212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58280cddb0 msgr2=0x7f582815aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58280cddb0 0x7f582815aed0 secure :-1 s=READY pgs=2906 cs=0 l=1 rev1=1 crypto rx=0x7f5820009e50 tx=0x7f582002a710 comp rx=0 tx=0).stop 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 -- 172.21.15.28:0/1497451212 shutdown_connections 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58280cddb0 0x7f582815aed0 unknown :-1 s=CLOSED pgs=2906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 --2- 172.21.15.28:0/1497451212 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f58280cd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 -- 172.21.15.28:0/1497451212 >> 172.21.15.28:0/1497451212 conn(0x7f5828069e80 msgr2=0x7f582806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 -- 172.21.15.28:0/1497451212 shutdown_connections 2024-09-13T17:25:47.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.888+0000 7f582d000640 1 -- 172.21.15.28:0/1497451212 wait complete. 2024-09-13T17:25:47.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.890+0000 7f582d000640 1 Processor -- start 2024-09-13T17:25:47.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.890+0000 7f582d000640 1 -- start start 2024-09-13T17:25:47.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f582814e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58280cddb0 0x7f5828156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828155be0 con 0x7f5828070e30 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5828159850 con 0x7f58280cddb0 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f582d000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5828158d10 con 0x7f5828071210 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53282/0 (socket says 172.21.15.28:53282) 2024-09-13T17:25:47.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 -- 172.21.15.28:0/1390224838 learned_addr learned my addr 172.21.15.28:0/1390224838 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827e00640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58280cddb0 0x7f5828156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f581ea00640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f582814e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 -- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 msgr2=0x7f582814e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f582814e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 -- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58280cddb0 msgr2=0x7f5828156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58280cddb0 0x7f5828156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.891+0000 7f5827400640 1 -- 172.21.15.28:0/1390224838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5820009b30 con 0x7f5828070e30 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f5827400640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f581800c450 tx=0x7f581800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58180052d0 con 0x7f5828070e30 2024-09-13T17:25:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818005430 con 0x7f5828070e30 2024-09-13T17:25:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5818012870 con 0x7f5828070e30 2024-09-13T17:25:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5828158fb0 con 0x7f5828070e30 2024-09-13T17:25:47.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.892+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5828159570 con 0x7f5828070e30 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.893+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582804e210 con 0x7f5828070e30 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.894+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f58180129d0 con 0x7f5828070e30 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.895+0000 7f581fe00640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 0x7f580406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.895+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f5818088a90 con 0x7f5828070e30 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.895+0000 7f581ea00640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 0x7f580406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:47.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.896+0000 7f581ea00640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 0x7f580406d0c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f58281f3c50 tx=0x7f5810009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:47.899+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f5818053fb0 con 0x7f5828070e30 2024-09-13T17:25:48.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.080+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5828072310 con 0x7f580406ac00 2024-09-13T17:25:48.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.080+0000 7f581fe00640 1 -- 172.21.15.28:0/1390224838 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5810011a30 con 0x7f580406ac00 2024-09-13T17:25:48.081 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.082+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 msgr2=0x7f580406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 0x7f580406d0c0 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f58281f3c50 tx=0x7f5810009190 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 msgr2=0x7f58281556a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f581800c450 tx=0x7f581800c910 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 shutdown_connections 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f580406ac00 0x7f580406d0c0 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f58280cddb0 0x7f5828156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f5828071210 0x7f582814e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 --2- 172.21.15.28:0/1390224838 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5828070e30 0x7f58281556a0 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 >> 172.21.15.28:0/1390224838 conn(0x7f5828069e80 msgr2=0x7f582815bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 shutdown_connections 2024-09-13T17:25:48.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.083+0000 7f582d000640 1 -- 172.21.15.28:0/1390224838 wait complete. 2024-09-13T17:25:48.099 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T17:25:48.099 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.258+0000 7f53c4200640 1 Processor -- start 2024-09-13T17:25:48.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.258+0000 7f53c4200640 1 -- start start 2024-09-13T17:25:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.259+0000 7f53c4200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.259+0000 7f53c4200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.259+0000 7f53c4200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53bc06c950 con 0x7f53bc0b1ab0 2024-09-13T17:25:48.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c4200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53bc0cee10 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c4200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc0b1ec0 con 0x7f53bc15a7a0 2024-09-13T17:25:48.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c0e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c1800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c2200640 1 --1- >> v1:172.21.15.189:6789/0 conn(0x7f53bc0b1ab0 0x7f53bc06b170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.189:6789/0 says I am v1:172.21.15.28:54470/0 (socket says 172.21.15.28:54470) 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c2200640 1 -- 172.21.15.28:0/3343255716 learned_addr learned my addr 172.21.15.28:0/3343255716 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c0e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.28:50924/0 (socket says 172.21.15.28:50924) 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.260+0000 7f53c1800640 1 --2- 172.21.15.28:0/3343255716 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:49156/0 (socket says 172.21.15.28:49156) 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53c1800640 1 -- 172.21.15.28:0/3343255716 >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 msgr2=0x7f53bc0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53c1800640 1 --2- 172.21.15.28:0/3343255716 >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53c1800640 1 -- 172.21.15.28:0/3343255716 >> v1:172.21.15.189:6789/0 conn(0x7f53bc0b1ab0 legacy=0x7f53bc06b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53c1800640 1 -- 172.21.15.28:0/3343255716 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53bc0b2150 con 0x7f53bc15a7a0 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53bbe00640 1 -- 172.21.15.28:0/3343255716 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439040145 0 0) 0x7f53b4001490 con 0x7f53bc0b1ab0 2024-09-13T17:25:48.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.261+0000 7f53c0e00640 1 --2- 172.21.15.28:0/3343255716 >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.262+0000 7f53c1800640 1 --2- 172.21.15.28:0/3343255716 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f53b00097a0 tx=0x7f53b002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c5ac38e644f33e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.262+0000 7f53bbe00640 1 -- 172.21.15.28:0/3343255716 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b002ae50 con 0x7f53bc15a7a0 2024-09-13T17:25:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.262+0000 7f53bbe00640 1 -- 172.21.15.28:0/3343255716 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0032440 con 0x7f53bc15a7a0 2024-09-13T17:25:48.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.262+0000 7f53bbe00640 1 -- 172.21.15.28:0/3343255716 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b00327d0 con 0x7f53bc15a7a0 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 -- 172.21.15.28:0/3343255716 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 msgr2=0x7f53bc0ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 --2- 172.21.15.28:0/3343255716 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f53b00097a0 tx=0x7f53b002a100 comp rx=0 tx=0).stop 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 -- 172.21.15.28:0/3343255716 shutdown_connections 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 --2- 172.21.15.28:0/3343255716 >> v2:172.21.15.189:3300/0 conn(0x7f53bc0ce9d0 0x7f53bc0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 --2- 172.21.15.28:0/3343255716 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc0ce490 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 -- 172.21.15.28:0/3343255716 >> 172.21.15.28:0/3343255716 conn(0x7f53bc069e80 msgr2=0x7f53bc06a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 -- 172.21.15.28:0/3343255716 shutdown_connections 2024-09-13T17:25:48.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.263+0000 7f53c4200640 1 -- 172.21.15.28:0/3343255716 wait complete. 2024-09-13T17:25:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.265+0000 7f53c4200640 1 Processor -- start 2024-09-13T17:25:48.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.265+0000 7f53c4200640 1 -- start start 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc0b1ab0 0x7f53bc15dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 0x7f53bc1e3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc145ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1f3150 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc15ab80 con 0x7f53bc0b1ab0 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c4200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1e4060 con 0x7f53bc15a7a0 2024-09-13T17:25:48.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 0x7f53bc1e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c0e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 0x7f53bc1e3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53284/0 (socket says 172.21.15.28:53284) 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c0e00640 1 -- 172.21.15.28:0/2007756704 learned_addr learned my addr 172.21.15.28:0/2007756704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c1800640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc0b1ab0 0x7f53bc15dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.266+0000 7f53c2200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc145ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 -- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 msgr2=0x7f53bc145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f53bc15a7a0 0x7f53bc145ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 -- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc0b1ab0 msgr2=0x7f53bc15dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc0b1ab0 0x7f53bc15dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8009400 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c0e00640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 0x7f53bc1e3960 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f53a800e960 tx=0x7f53a800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a800ca00 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800cb60 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8013470 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b0009400 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.267+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc1e4620 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.268+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53bc0ccc40 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.269+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f53a80135d0 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.270+0000 7f53b9600640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f539c06ac00 0x7f539c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.270+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f53a8089300 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.270+0000 7f53b9600640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f539c070b00 0x7f539c072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.270+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f539c0735d0 con 0x7f539c070b00 2024-09-13T17:25:48.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.270+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53a808b8e0 con 0x7f53bc0ce9d0 2024-09-13T17:25:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.271+0000 7f53c1800640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f539c06ac00 0x7f539c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.271+0000 7f53c2200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f539c070b00 0x7f539c072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.271+0000 7f53c1800640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f539c06ac00 0x7f539c06d0c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f53b0034000 tx=0x7f53b002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.271+0000 7f53c2200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f539c070b00 0x7f539c072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.272+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== osd.5 v2:172.21.15.189:6808/1971214256 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f539c0735d0 con 0x7f539c070b00 2024-09-13T17:25:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.285+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 --> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f53bc0cc960 con 0x7f539c070b00 2024-09-13T17:25:48.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.285+0000 7f53b9600640 1 -- 172.21.15.28:0/2007756704 <== osd.5 v2:172.21.15.189:6808/1971214256 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f53b4004dc0 con 0x7f539c070b00 2024-09-13T17:25:48.286 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:48.287 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:48.287 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:48.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f539c070b00 msgr2=0x7f539c072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6808/1971214256,v1:172.21.15.189:6809/1971214256] conn(0x7f539c070b00 0x7f539c072f20 crc :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f539c06ac00 msgr2=0x7f539c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f539c06ac00 0x7f539c06d0c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f53b0034000 tx=0x7f53b002e000 comp rx=0 tx=0).stop 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 msgr2=0x7f53bc1e3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 --2- 172.21.15.28:0/2007756704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f53bc0ce9d0 0x7f53bc1e3960 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f53a800e960 tx=0x7f53a800ee20 comp rx=0 tx=0).stop 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c2200640 1 -- 172.21.15.28:0/2007756704 reap_dead start 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 shutdown_connections 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 >> 172.21.15.28:0/2007756704 conn(0x7f53bc069e80 msgr2=0x7f53bc0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 shutdown_connections 2024-09-13T17:25:48.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.286+0000 7f53c4200640 1 -- 172.21.15.28:0/2007756704 wait complete. 2024-09-13T17:25:48.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T17:25:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.457+0000 7fe554000640 1 Processor -- start 2024-09-13T17:25:48.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.457+0000 7fe554000640 1 -- start start 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7fe54c15a780 0x7fe54c070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c15ab60 0x7fe54c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c06bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c06c8b0 con 0x7fe54c15ab60 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54c06cb90 con 0x7fe54c15a780 2024-09-13T17:25:48.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe554000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c0cc980 con 0x7fe54c06bb40 2024-09-13T17:25:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe550c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c15ab60 0x7fe54c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe550c00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fe54c15ab60 0x7fe54c0712c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53290/0 (socket says 172.21.15.28:53290) 2024-09-13T17:25:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.458+0000 7fe550c00640 1 -- 172.21.15.28:0/783271909 learned_addr learned my addr 172.21.15.28:0/783271909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:48.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe552000640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c06bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe551600640 1 --2- 172.21.15.28:0/783271909 >> v2:172.21.15.189:3300/0 conn(0x7fe54c15a780 0x7fe54c070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe550c00640 1 -- 172.21.15.28:0/783271909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 msgr2=0x7fe54c06bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe550c00640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c06bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe550c00640 1 -- 172.21.15.28:0/783271909 >> v2:172.21.15.189:3300/0 conn(0x7fe54c15a780 msgr2=0x7fe54c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe550c00640 1 --2- 172.21.15.28:0/783271909 >> v2:172.21.15.189:3300/0 conn(0x7fe54c15a780 0x7fe54c070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.459+0000 7fe550c00640 1 -- 172.21.15.28:0/783271909 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c0ccc60 con 0x7fe54c15ab60 2024-09-13T17:25:48.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.460+0000 7fe550c00640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15ab60 0x7fe54c0712c0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0b1330 tx=0x7fe53802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38364cde093f6ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.460+0000 7fe54be00640 1 -- 172.21.15.28:0/783271909 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538032440 con 0x7fe54c15ab60 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.460+0000 7fe54be00640 1 -- 172.21.15.28:0/783271909 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5380325a0 con 0x7fe54c15ab60 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.460+0000 7fe54be00640 1 -- 172.21.15.28:0/783271909 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538032930 con 0x7fe54c15ab60 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 -- 172.21.15.28:0/783271909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15ab60 msgr2=0x7fe54c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15ab60 0x7fe54c0712c0 secure :-1 s=READY pgs=2909 cs=0 l=1 rev1=1 crypto rx=0x7fe54c0b1330 tx=0x7fe53802a1e0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 -- 172.21.15.28:0/783271909 shutdown_connections 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15ab60 0x7fe54c0712c0 unknown :-1 s=CLOSED pgs=2909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 --2- 172.21.15.28:0/783271909 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c06bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 --2- 172.21.15.28:0/783271909 >> v2:172.21.15.189:3300/0 conn(0x7fe54c15a780 0x7fe54c070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 -- 172.21.15.28:0/783271909 >> 172.21.15.28:0/783271909 conn(0x7fe54c0b0270 msgr2=0x7fe54c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 -- 172.21.15.28:0/783271909 shutdown_connections 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.461+0000 7fe554000640 1 -- 172.21.15.28:0/783271909 wait complete. 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.462+0000 7fe554000640 1 Processor -- start 2024-09-13T17:25:48.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.463+0000 7fe554000640 1 -- start start 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.463+0000 7fe554000640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.463+0000 7fe554000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe554000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c15ab60 0x7fe54c1583b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe554000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c156750 con 0x7fe54c15a780 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe554000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c157070 con 0x7fe54c15ab60 2024-09-13T17:25:48.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe554000640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54c157330 con 0x7fe54c06bb40 2024-09-13T17:25:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53294/0 (socket says 172.21.15.28:53294) 2024-09-13T17:25:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 -- 172.21.15.28:0/3457900777 learned_addr learned my addr 172.21.15.28:0/3457900777 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe551600640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe552000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c15ab60 0x7fe54c1583b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 -- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 msgr2=0x7fe54c156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 -- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c15ab60 msgr2=0x7fe54c1583b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c15ab60 0x7fe54c1583b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.464+0000 7fe550c00640 1 -- 172.21.15.28:0/3457900777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380094d0 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe550c00640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7fe53802a220 tx=0x7fe5380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538032440 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538030ba0 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c157610 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53803d440 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.465+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c157bd0 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.466+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c06bf70 con 0x7fe54c15a780 2024-09-13T17:25:48.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.467+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fe538039040 con 0x7fe54c15a780 2024-09-13T17:25:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.468+0000 7fe549600640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.468+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fe5380a9c00 con 0x7fe54c15a780 2024-09-13T17:25:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.468+0000 7fe551600640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.469+0000 7fe551600640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe540006fd0 tx=0x7fe540008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.472+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fe538075070 con 0x7fe54c15a780 2024-09-13T17:25:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.651+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fe54c06f3f0 con 0x7fe52c06ac00 2024-09-13T17:25:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.652+0000 7fe549600640 1 -- 172.21.15.28:0/3457900777 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe540004020 con 0x7fe52c06ac00 2024-09-13T17:25:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 msgr2=0x7fe52c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7fe540006fd0 tx=0x7fe540008040 comp rx=0 tx=0).stop 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 msgr2=0x7fe54c141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7fe53802a220 tx=0x7fe5380046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 shutdown_connections 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fe52c06ac00 0x7fe52c06d0c0 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe54c15ab60 0x7fe54c1583b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe54c15a780 0x7fe54c141e90 unknown :-1 s=CLOSED pgs=2910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 --2- 172.21.15.28:0/3457900777 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fe54c06bb40 0x7fe54c156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 >> 172.21.15.28:0/3457900777 conn(0x7fe54c0b0270 msgr2=0x7fe54c0cd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 shutdown_connections 2024-09-13T17:25:48.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.654+0000 7fe554000640 1 -- 172.21.15.28:0/3457900777 wait complete. 2024-09-13T17:25:48.671 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T17:25:48.671 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.828+0000 7f70ca200640 1 Processor -- start 2024-09-13T17:25:48.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.828+0000 7f70ca200640 1 -- start start 2024-09-13T17:25:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.829+0000 7f70ca200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 0x7f70c4070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.829+0000 7f70ca200640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70c4159970 0x7f70c40b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.829+0000 7f70ca200640 1 -- --> v1:172.21.15.189:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70c415a4c0 con 0x7f70c40b2630 2024-09-13T17:25:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.829+0000 7f70ca200640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70c40b2ae0 con 0x7f70c4159970 2024-09-13T17:25:48.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.829+0000 7f70ca200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4071580 con 0x7f70c4159590 2024-09-13T17:25:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70c3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 0x7f70c4070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70baa00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f70c4159970 0x7f70c40b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70c3400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 0x7f70c4070c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3301/0 says I am v2:172.21.15.28:49186/0 (socket says 172.21.15.28:49186) 2024-09-13T17:25:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70c3400640 1 -- 172.21.15.28:0/194573482 learned_addr learned my addr 172.21.15.28:0/194573482 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:48.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70baa00640 1 -- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 msgr2=0x7f70c4070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70baa00640 1 --2- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 0x7f70c4070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70baa00640 1 -- 172.21.15.28:0/194573482 >> v1:172.21.15.189:6789/0 conn(0x7f70c40b2630 legacy=0x7f70c406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70c2a00640 1 -- 172.21.15.28:0/194573482 <== mon.1 v1:172.21.15.189:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018307033 0 0) 0x7f70c415a4c0 con 0x7f70c40b2630 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.830+0000 7f70baa00640 1 -- 172.21.15.28:0/194573482 --> v2:172.21.15.189:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4159e70 con 0x7f70c4159970 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.831+0000 7f70baa00640 1 --2- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4159970 0x7f70c40b20f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0097a0 tx=0x7f70ac02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80148efd8b1f5cb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70c2a00640 1 -- 172.21.15.28:0/194573482 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac032440 con 0x7f70c4159970 2024-09-13T17:25:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70c2a00640 1 -- 172.21.15.28:0/194573482 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac0325a0 con 0x7f70c4159970 2024-09-13T17:25:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70c2a00640 1 -- 172.21.15.28:0/194573482 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac032930 con 0x7f70c4159970 2024-09-13T17:25:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 -- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4159970 msgr2=0x7f70c40b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 --2- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4159970 0x7f70c40b20f0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f70ac0097a0 tx=0x7f70ac02a180 comp rx=0 tx=0).stop 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 -- 172.21.15.28:0/194573482 shutdown_connections 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 --2- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c4159970 0x7f70c40b20f0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 --2- 172.21.15.28:0/194573482 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159590 0x7f70c4070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 -- 172.21.15.28:0/194573482 >> 172.21.15.28:0/194573482 conn(0x7f70c4069e80 msgr2=0x7f70c406a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 -- 172.21.15.28:0/194573482 shutdown_connections 2024-09-13T17:25:48.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.832+0000 7f70ca200640 1 -- 172.21.15.28:0/194573482 wait complete. 2024-09-13T17:25:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.834+0000 7f70ca200640 1 Processor -- start 2024-09-13T17:25:48.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.834+0000 7f70ca200640 1 -- start start 2024-09-13T17:25:48.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c40b2630 0x7f70c4145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 0x7f70c40abe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159970 0x7f70c4155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c40ac7a0 con 0x7f70c4159590 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c4145ae0 con 0x7f70c40b2630 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70ca200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70c41457b0 con 0x7f70c4159970 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 0x7f70c40abe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70baa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 0x7f70c40abe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53310/0 (socket says 172.21.15.28:53310) 2024-09-13T17:25:48.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70baa00640 1 -- 172.21.15.28:0/4249322518 learned_addr learned my addr 172.21.15.28:0/4249322518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70c3400640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c40b2630 0x7f70c4145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.835+0000 7f70c3e00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159970 0x7f70c4155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 -- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159970 msgr2=0x7f70c4155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f70c4159970 0x7f70c4155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 -- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c40b2630 msgr2=0x7f70c4145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70c40b2630 0x7f70c4145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac009400 con 0x7f70c4159590 2024-09-13T17:25:48.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70baa00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 0x7f70c40abe70 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f70ac000c00 tx=0x7f70ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac030b80 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70ac032e90 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70ac006e10 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.836+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c4145d30 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.837+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c41462f0 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.837+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f70c406fb80 con 0x7f70c4159590 2024-09-13T17:25:48.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.838+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f70ac039040 con 0x7f70c4159590 2024-09-13T17:25:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.839+0000 7f70bbe00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70a006ac00 0x7f70a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.839+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f70ac0a9970 con 0x7f70c4159590 2024-09-13T17:25:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70bbe00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f70a0070b00 0x7f70a0072f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f70a00735d0 con 0x7f70a0070b00 2024-09-13T17:25:48.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70ac0ae2a0 con 0x7f70c4159590 2024-09-13T17:25:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70c3400640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70a006ac00 0x7f70a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70c3e00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f70a0070b00 0x7f70a0072f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.840+0000 7f70c3400640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70a006ac00 0x7f70a006d0c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009bd0 tx=0x7f70b4009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.841+0000 7f70c3e00640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f70a0070b00 0x7f70a0072f20 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:48.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.842+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== osd.6 v2:172.21.15.189:6816/2946605912 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f70a00735d0 con 0x7f70a0070b00 2024-09-13T17:25:48.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.855+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 --> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f70c4070c80 con 0x7f70a0070b00 2024-09-13T17:25:48.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.855+0000 7f70bbe00640 1 -- 172.21.15.28:0/4249322518 <== osd.6 v2:172.21.15.189:6816/2946605912 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f70c4070c80 con 0x7f70a0070b00 2024-09-13T17:25:48.857 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:48.858 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:48.858 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f70a0070b00 msgr2=0x7f70a0072f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6816/2946605912,v1:172.21.15.189:6817/2946605912] conn(0x7f70a0070b00 0x7f70a0072f20 crc :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70a006ac00 msgr2=0x7f70a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f70a006ac00 0x7f70a006d0c0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f70b4009bd0 tx=0x7f70b4009240 comp rx=0 tx=0).stop 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 msgr2=0x7f70c40abe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 --2- 172.21.15.28:0/4249322518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f70c4159590 0x7f70c40abe70 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f70ac000c00 tx=0x7f70ac0046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70c3e00640 1 -- 172.21.15.28:0/4249322518 reap_dead start 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 shutdown_connections 2024-09-13T17:25:48.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 >> 172.21.15.28:0/4249322518 conn(0x7f70c4069e80 msgr2=0x7f70c406a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 shutdown_connections 2024-09-13T17:25:48.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:48.856+0000 7f70ca200640 1 -- 172.21.15.28:0/4249322518 wait complete. 2024-09-13T17:25:48.873 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T17:25:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.027+0000 7fcb68e00640 1 Processor -- start 2024-09-13T17:25:49.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.027+0000 7fcb68e00640 1 -- start start 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.028+0000 7fcb68e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb64070d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.028+0000 7fcb68e00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcb6406c250 0x7fcb640712c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.028+0000 7fcb68e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6406c8b0 con 0x7fcb6415ab60 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb68e00640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6406c6a0 con 0x7fcb6406c250 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb68e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6406cb90 con 0x7fcb6415a780 2024-09-13T17:25:49.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb62800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fcb6406c250 0x7fcb640712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb59400640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fcb6415ab60 0x7fcb6406bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53234/0 (socket says 172.21.15.28:53234) 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb59400640 1 -- 172.21.15.28:0/3020805932 learned_addr learned my addr 172.21.15.28:0/3020805932 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb61e00640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb64070d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb62800640 1 -- 172.21.15.28:0/3020805932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 msgr2=0x7fcb64070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb62800640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb64070d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb62800640 1 -- 172.21.15.28:0/3020805932 >> v1:172.21.15.28:6789/0 conn(0x7fcb6415ab60 legacy=0x7fcb6406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb62800640 1 -- 172.21.15.28:0/3020805932 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb640cc980 con 0x7fcb6406c250 2024-09-13T17:25:49.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.029+0000 7fcb61400640 1 -- 172.21.15.28:0/3020805932 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191792373 0 0) 0x7fcb6406c8b0 con 0x7fcb6415ab60 2024-09-13T17:25:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.030+0000 7fcb62800640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb640712c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009d90 tx=0x7fcb5c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac99ffb63260cad8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb61400640 1 -- 172.21.15.28:0/3020805932 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c007740 con 0x7fcb6406c250 2024-09-13T17:25:49.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb61400640 1 -- 172.21.15.28:0/3020805932 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c02d050 con 0x7fcb6406c250 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb61400640 1 -- 172.21.15.28:0/3020805932 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c007ac0 con 0x7fcb6406c250 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 -- 172.21.15.28:0/3020805932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 msgr2=0x7fcb640712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb640712c0 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c009d90 tx=0x7fcb5c02a660 comp rx=0 tx=0).stop 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 -- 172.21.15.28:0/3020805932 shutdown_connections 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb640712c0 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3020805932 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb64070d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 -- 172.21.15.28:0/3020805932 >> 172.21.15.28:0/3020805932 conn(0x7fcb640b0270 msgr2=0x7fcb640b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 -- 172.21.15.28:0/3020805932 shutdown_connections 2024-09-13T17:25:49.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.031+0000 7fcb68e00640 1 -- 172.21.15.28:0/3020805932 wait complete. 2024-09-13T17:25:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.033+0000 7fcb68e00640 1 Processor -- start 2024-09-13T17:25:49.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.033+0000 7fcb68e00640 1 -- start start 2024-09-13T17:25:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb641463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6415ab60 0x7fcb64146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb64146f60 con 0x7fcb6406c250 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb64147240 con 0x7fcb6415ab60 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb68e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb64147520 con 0x7fcb6415a780 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb61e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53324/0 (socket says 172.21.15.28:53324) 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb61e00640 1 -- 172.21.15.28:0/3253666114 learned_addr learned my addr 172.21.15.28:0/3253666114 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb59400640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb641463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.034+0000 7fcb62800640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6415ab60 0x7fcb64146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 -- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 msgr2=0x7fcb641463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb641463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 -- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6415ab60 msgr2=0x7fcb64146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6415ab60 0x7fcb64146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 -- 172.21.15.28:0/3253666114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c009a70 con 0x7fcb6406c250 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb59400640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb641463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb61e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e9b0 tx=0x7fcb5400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb54009670 con 0x7fcb6406c250 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5400c9a0 con 0x7fcb6406c250 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5400cd50 con 0x7fcb6406c250 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb64147800 con 0x7fcb6406c250 2024-09-13T17:25:49.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.035+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb64148700 con 0x7fcb6406c250 2024-09-13T17:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.036+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6406c8b0 con 0x7fcb6406c250 2024-09-13T17:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.037+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fcb54013470 con 0x7fcb6406c250 2024-09-13T17:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.038+0000 7fcb5aa00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 0x7fcb4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.038+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fcb540893f0 con 0x7fcb6406c250 2024-09-13T17:25:49.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.039+0000 7fcb59400640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 0x7fcb4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.039+0000 7fcb59400640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 0x7fcb4006d0c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcb64154e60 tx=0x7fcb4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.042+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fcb540547d0 con 0x7fcb6406c250 2024-09-13T17:25:49.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.219+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb640cd180 con 0x7fcb4006ac00 2024-09-13T17:25:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.220+0000 7fcb5aa00640 1 -- 172.21.15.28:0/3253666114 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcb4c001c90 con 0x7fcb4006ac00 2024-09-13T17:25:49.222 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T17:25:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 msgr2=0x7fcb4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 0x7fcb4006d0c0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fcb64154e60 tx=0x7fcb4c009040 comp rx=0 tx=0).stop 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 msgr2=0x7fcb64145a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400e9b0 tx=0x7fcb5400ee70 comp rx=0 tx=0).stop 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 shutdown_connections 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fcb4006ac00 0x7fcb4006d0c0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcb6415ab60 0x7fcb64146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fcb6415a780 0x7fcb641463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 --2- 172.21.15.28:0/3253666114 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb6406c250 0x7fcb64145a80 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.222+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 >> 172.21.15.28:0/3253666114 conn(0x7fcb640b0270 msgr2=0x7fcb640af040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.223+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 shutdown_connections 2024-09-13T17:25:49.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.223+0000 7fcb68e00640 1 -- 172.21.15.28:0/3253666114 wait complete. 2024-09-13T17:25:49.241 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T17:25:49.241 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T17:25:49.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.398+0000 7f6c53e00640 1 Processor -- start 2024-09-13T17:25:49.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.399+0000 7f6c53e00640 1 -- start start 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c53e00640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f6c54159190 0x7f6c5406fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c540b1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c53e00640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c540cd1d0 con 0x7f6c540cdf10 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c53e00640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c540cd660 con 0x7f6c54159190 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c53e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540cc5c0 con 0x7f6c5406d410 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c4a000640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f6c540cdf10 0x7f6c5406cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53236/0 (socket says 172.21.15.28:53236) 2024-09-13T17:25:49.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.400+0000 7f6c4a000640 1 -- 172.21.15.28:0/3047069407 learned_addr learned my addr 172.21.15.28:0/3047069407 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c53400640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c540b1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52a00640 1 --2- 172.21.15.28:0/3047069407 >> v2:172.21.15.189:3300/0 conn(0x7f6c54159190 0x7f6c5406fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451317134 0 0) 0x7f6c540cd1d0 con 0x7f6c540cdf10 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c38005730 con 0x7f6c540cdf10 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1461884419 0 0) 0x7f6c38005730 con 0x7f6c540cdf10 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 msgr2=0x7f6c540b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c540b1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 >> v2:172.21.15.189:3300/0 conn(0x7f6c54159190 msgr2=0x7f6c5406fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 --2- 172.21.15.28:0/3047069407 >> v2:172.21.15.189:3300/0 conn(0x7f6c54159190 0x7f6c5406fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c540cc8a0 con 0x7f6c540cdf10 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f6c3c001670 con 0x7f6c540cdf10 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 >> v1:172.21.15.28:6789/0 conn(0x7f6c540cdf10 legacy=0x7f6c5406cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c38005b80 0x7f6c38005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c380069d0 0x7f6c38006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c38007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540cd660 con 0x7f6c380069d0 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c540cc5c0 con 0x7f6c38005b80 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3800db90 con 0x7f6c5406d410 2024-09-13T17:25:49.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c4a000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c380069d0 0x7f6c38006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.401+0000 7f6c52a00640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c38005b80 0x7f6c38005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c53400640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c38007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c4a000640 1 -- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 msgr2=0x7f6c38007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c4a000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c38007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c4a000640 1 -- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c38005b80 msgr2=0x7f6c38005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c4a000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c38005b80 0x7f6c38005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c4a000640 1 -- 172.21.15.28:0/3047069407 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38005730 con 0x7f6c380069d0 2024-09-13T17:25:49.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.402+0000 7f6c53400640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5406d410 0x7f6c38007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:25:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c4a000640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c380069d0 0x7f6c38006e50 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c110 tx=0x7f6c38007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e930707d356273 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c005ad0 con 0x7f6c380069d0 2024-09-13T17:25:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c02d070 con 0x7f6c380069d0 2024-09-13T17:25:49.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c52000640 1 -- 172.21.15.28:0/3047069407 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c005e90 con 0x7f6c380069d0 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c53e00640 1 -- 172.21.15.28:0/3047069407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c380069d0 msgr2=0x7f6c38006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.403+0000 7f6c53e00640 1 --2- 172.21.15.28:0/3047069407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c380069d0 0x7f6c38006e50 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c00c110 tx=0x7f6c38007a00 comp rx=0 tx=0).stop 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.404+0000 7f6c53400640 1 -- 172.21.15.28:0/3047069407 reap_dead start 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.404+0000 7f6c53e00640 1 -- 172.21.15.28:0/3047069407 shutdown_connections 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.404+0000 7f6c53e00640 1 -- 172.21.15.28:0/3047069407 >> 172.21.15.28:0/3047069407 conn(0x7f6c54073470 msgr2=0x7f6c54069e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.404+0000 7f6c53e00640 1 -- 172.21.15.28:0/3047069407 shutdown_connections 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.404+0000 7f6c53e00640 1 -- 172.21.15.28:0/3047069407 wait complete. 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.405+0000 7f6c53e00640 1 Processor -- start 2024-09-13T17:25:49.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.405+0000 7f6c53e00640 1 -- start start 2024-09-13T17:25:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c380069d0 0x7f6c54157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5415b190 0x7f6c5415b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 0x7f6c541ef210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5415bb30 con 0x7f6c541eed90 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c541523c0 con 0x7f6c380069d0 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.406+0000 7f6c53e00640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c541575f0 con 0x7f6c5415b190 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 0x7f6c541ef210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 0x7f6c541ef210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53344/0 (socket says 172.21.15.28:53344) 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 -- 172.21.15.28:0/1632190376 learned_addr learned my addr 172.21.15.28:0/1632190376 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c52a00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c380069d0 0x7f6c54157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c4a000640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5415b190 0x7f6c5415b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 -- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5415b190 msgr2=0x7f6c5415b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6c5415b190 0x7f6c5415b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 -- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c380069d0 msgr2=0x7f6c54157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c380069d0 0x7f6c54157a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c541569e0 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c52a00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c380069d0 0x7f6c54157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.407+0000 7f6c53400640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 0x7f6c541ef210 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f6c5406d810 tx=0x7f6c4c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.408+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c00e030 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.408+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c011de0 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.408+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c0166b0 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.408+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c54156cc0 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.408+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c54157280 con 0x7f6c541eed90 2024-09-13T17:25:49.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.409+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c54070250 con 0x7f6c541eed90 2024-09-13T17:25:49.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.410+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f6c4c016810 con 0x7f6c541eed90 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c4b400640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c3006ac30 0x7f6c3006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f6c4c00f070 con 0x7f6c541eed90 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c4b400640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f6c30070b30 0x7f6c30072f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c30073600 con 0x7f6c30070b30 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=237) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c4c08e2a0 con 0x7f6c541eed90 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.411+0000 7f6c52a00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c3006ac30 0x7f6c3006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.412+0000 7f6c4a000640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f6c30070b30 0x7f6c30072f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.412+0000 7f6c52a00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c3006ac30 0x7f6c3006d0f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6c44014400 tx=0x7f6c44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.412+0000 7f6c4a000640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f6c30070b30 0x7f6c30072f50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.414+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== osd.7 v2:172.21.15.189:6822/2656371491 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f6c30073600 con 0x7f6c30070b30 2024-09-13T17:25:49.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.426+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 --> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6c540cce60 con 0x7f6c30070b30 2024-09-13T17:25:49.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c4b400640 1 -- 172.21.15.28:0/1632190376 <== osd.7 v2:172.21.15.189:6822/2656371491 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6c3c001720 con 0x7f6c30070b30 2024-09-13T17:25:49.429 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-09-13T17:25:49.429 INFO:teuthology.orchestra.run.smithi028.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T17:25:49.429 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f6c30070b30 msgr2=0x7f6c30072f50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6822/2656371491,v1:172.21.15.189:6824/2656371491] conn(0x7f6c30070b30 0x7f6c30072f50 crc :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c3006ac30 msgr2=0x7f6c3006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f6c3006ac30 0x7f6c3006d0f0 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f6c44014400 tx=0x7f6c44009190 comp rx=0 tx=0).stop 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 msgr2=0x7f6c541ef210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.427+0000 7f6c53e00640 1 --2- 172.21.15.28:0/1632190376 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c541eed90 0x7f6c541ef210 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f6c5406d810 tx=0x7f6c4c0115a0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.428+0000 7f6c53400640 1 -- 172.21.15.28:0/1632190376 reap_dead start 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.428+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 shutdown_connections 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.428+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 >> 172.21.15.28:0/1632190376 conn(0x7f6c54073470 msgr2=0x7f6c540704f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.428+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 shutdown_connections 2024-09-13T17:25:49.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.428+0000 7f6c53e00640 1 -- 172.21.15.28:0/1632190376 wait complete. 2024-09-13T17:25:49.445 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T17:25:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.599+0000 7fbd20800640 1 Processor -- start 2024-09-13T17:25:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.599+0000 7fbd20800640 1 -- start start 2024-09-13T17:25:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd20800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18070430 0x7fbd1815bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd20800640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7fbd1806f3f0 0x7fbd1815c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd20800640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1815ab60 con 0x7fbd18159770 2024-09-13T17:25:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd20800640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd1806f920 con 0x7fbd1806f3f0 2024-09-13T17:25:49.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd20800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1815b290 con 0x7fbd18070430 2024-09-13T17:25:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd1de00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7fbd18159770 0x7fbd1806fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53248/0 (socket says 172.21.15.28:53248) 2024-09-13T17:25:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd1de00640 1 -- 172.21.15.28:0/1186640927 learned_addr learned my addr 172.21.15.28:0/1186640927 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.600+0000 7fbd1e800640 1 --2- 172.21.15.28:0/1186640927 >> v2:172.21.15.28:3300/0 conn(0x7fbd1806f3f0 0x7fbd1815c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1d400640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18070430 0x7fbd1815bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1e800640 1 -- 172.21.15.28:0/1186640927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18070430 msgr2=0x7fbd1815bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1ca00640 1 -- 172.21.15.28:0/1186640927 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177501720 0 0) 0x7fbd0c001110 con 0x7fbd18159770 2024-09-13T17:25:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1e800640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18070430 0x7fbd1815bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1e800640 1 -- 172.21.15.28:0/1186640927 >> v1:172.21.15.28:6789/0 conn(0x7fbd18159770 legacy=0x7fbd1806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.601+0000 7fbd1e800640 1 -- 172.21.15.28:0/1186640927 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1815b570 con 0x7fbd1806f3f0 2024-09-13T17:25:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.602+0000 7fbd1e800640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd1806f3f0 0x7fbd1815c2b0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009d90 tx=0x7fbd1402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb42b5f4c468cbb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.602+0000 7fbd1ca00640 1 -- 172.21.15.28:0/1186640927 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14031600 con 0x7fbd1806f3f0 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.602+0000 7fbd1ca00640 1 -- 172.21.15.28:0/1186640927 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1402d050 con 0x7fbd1806f3f0 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.602+0000 7fbd1ca00640 1 -- 172.21.15.28:0/1186640927 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd14031980 con 0x7fbd1806f3f0 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 -- 172.21.15.28:0/1186640927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd1806f3f0 msgr2=0x7fbd1815c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd1806f3f0 0x7fbd1815c2b0 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7fbd14009d90 tx=0x7fbd1402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 -- 172.21.15.28:0/1186640927 shutdown_connections 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd1806f3f0 0x7fbd1815c2b0 unknown :-1 s=CLOSED pgs=2919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 --2- 172.21.15.28:0/1186640927 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18070430 0x7fbd1815bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 -- 172.21.15.28:0/1186640927 >> 172.21.15.28:0/1186640927 conn(0x7fbd180b0270 msgr2=0x7fbd180b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 -- 172.21.15.28:0/1186640927 shutdown_connections 2024-09-13T17:25:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.603+0000 7fbd20800640 1 -- 172.21.15.28:0/1186640927 wait complete. 2024-09-13T17:25:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.604+0000 7fbd20800640 1 Processor -- start 2024-09-13T17:25:49.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 -- start start 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd1806f3f0 0x7fbd180aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18159770 0x7fbd180ad220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180a8e50 con 0x7fbd18070430 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180a90f0 con 0x7fbd1806f3f0 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.605+0000 7fbd20800640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd180a93d0 con 0x7fbd18159770 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53358/0 (socket says 172.21.15.28:53358) 2024-09-13T17:25:49.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 -- 172.21.15.28:0/472537846 learned_addr learned my addr 172.21.15.28:0/472537846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1de00640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd1806f3f0 0x7fbd180aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 -- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18159770 msgr2=0x7fbd180ad220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18159770 0x7fbd180ad220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 -- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd1806f3f0 msgr2=0x7fbd180aacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd1806f3f0 0x7fbd180aacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 -- 172.21.15.28:0/472537846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd14009a70 con 0x7fbd18070430 2024-09-13T17:25:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd1d400640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800d800 tx=0x7fbd0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.606+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800b6c0 con 0x7fbd18070430 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.607+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0800b820 con 0x7fbd18070430 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.607+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800bbb0 con 0x7fbd18070430 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.607+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd180a96b0 con 0x7fbd18070430 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.607+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd180a9c70 con 0x7fbd18070430 2024-09-13T17:25:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.608+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd180a9e00 con 0x7fbd18070430 2024-09-13T17:25:49.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.609+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fbd0800bd10 con 0x7fbd18070430 2024-09-13T17:25:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.610+0000 7fbd12000640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 0x7fbcfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.610+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fbd08089700 con 0x7fbd18070430 2024-09-13T17:25:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.611+0000 7fbd1de00640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 0x7fbcfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.611+0000 7fbd1de00640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 0x7fbcfc06d0c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c000f60 tx=0x7fbd0c007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.614+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fbd08054ae0 con 0x7fbd18070430 2024-09-13T17:25:49.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.792+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fbd180add00 con 0x7fbcfc06ac00 2024-09-13T17:25:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.793+0000 7fbd12000640 1 -- 172.21.15.28:0/472537846 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fbd0c007270 con 0x7fbcfc06ac00 2024-09-13T17:25:49.795 INFO:teuthology.orchestra.run.smithi028.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T17:25:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 msgr2=0x7fbcfc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 0x7fbcfc06d0c0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c000f60 tx=0x7fbd0c007c00 comp rx=0 tx=0).stop 2024-09-13T17:25:49.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 msgr2=0x7fbd180acce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800d800 tx=0x7fbd0800dcc0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 shutdown_connections 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fbcfc06ac00 0x7fbcfc06d0c0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7fbd18159770 0x7fbd180ad220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbd18070430 0x7fbd180acce0 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 --2- 172.21.15.28:0/472537846 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd1806f3f0 0x7fbd180aacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 >> 172.21.15.28:0/472537846 conn(0x7fbd180b0270 msgr2=0x7fbd1806ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 shutdown_connections 2024-09-13T17:25:49.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.796+0000 7fbd20800640 1 -- 172.21.15.28:0/472537846 wait complete. 2024-09-13T17:25:49.814 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:25:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.968+0000 7f0e39400640 1 Processor -- start 2024-09-13T17:25:49.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.968+0000 7f0e39400640 1 -- start start 2024-09-13T17:25:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e39400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f0e3406ba70 0x7f0e340afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e39400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406c3f0 0x7f0e34158f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e39400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e340b39d0 con 0x7f0e34158b20 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e39400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e3406c880 con 0x7f0e3406ba70 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e39400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e340b2dc0 con 0x7f0e3406c3f0 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e33e00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f0e34158b20 0x7f0e3406b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:53256/0 (socket says 172.21.15.28:53256) 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e33e00640 1 -- 172.21.15.28:0/811047769 learned_addr learned my addr 172.21.15.28:0/811047769 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.969+0000 7f0e2be00640 1 --2- 172.21.15.28:0/811047769 >> v2:172.21.15.28:3300/0 conn(0x7f0e3406ba70 0x7f0e340afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e38a00640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406c3f0 0x7f0e34158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e2be00640 1 -- 172.21.15.28:0/811047769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406c3f0 msgr2=0x7f0e34158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e33400640 1 -- 172.21.15.28:0/811047769 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 995959772 0 0) 0x7f0e340b39d0 con 0x7f0e34158b20 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e2be00640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406c3f0 0x7f0e34158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e2be00640 1 -- 172.21.15.28:0/811047769 >> v1:172.21.15.28:6789/0 conn(0x7f0e34158b20 legacy=0x7f0e3406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.970+0000 7f0e2be00640 1 -- 172.21.15.28:0/811047769 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e340b30a0 con 0x7f0e3406ba70 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.971+0000 7f0e2be00640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406ba70 0x7f0e340afbd0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0097d0 tx=0x7f0e1c02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66c456022cffca7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.971+0000 7f0e33400640 1 -- 172.21.15.28:0/811047769 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c032440 con 0x7f0e3406ba70 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.971+0000 7f0e33400640 1 -- 172.21.15.28:0/811047769 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1c0325a0 con 0x7f0e3406ba70 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.971+0000 7f0e33400640 1 -- 172.21.15.28:0/811047769 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c032930 con 0x7f0e3406ba70 2024-09-13T17:25:49.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 -- 172.21.15.28:0/811047769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406ba70 msgr2=0x7f0e340afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406ba70 0x7f0e340afbd0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0097d0 tx=0x7f0e1c02a1c0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 -- 172.21.15.28:0/811047769 shutdown_connections 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406c3f0 0x7f0e34158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 --2- 172.21.15.28:0/811047769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406ba70 0x7f0e340afbd0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 -- 172.21.15.28:0/811047769 >> 172.21.15.28:0/811047769 conn(0x7f0e34072e40 msgr2=0x7f0e34073220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 -- 172.21.15.28:0/811047769 shutdown_connections 2024-09-13T17:25:49.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.972+0000 7f0e39400640 1 -- 172.21.15.28:0/811047769 wait complete. 2024-09-13T17:25:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.973+0000 7f0e39400640 1 Processor -- start 2024-09-13T17:25:49.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 -- start start 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406ba70 0x7f0e340af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e34158b20 0x7f0e341436e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3414eda0 con 0x7f0e3406c3f0 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3414dd00 con 0x7f0e34158b20 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.974+0000 7f0e39400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3414dfe0 con 0x7f0e3406ba70 2024-09-13T17:25:49.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53388/0 (socket says 172.21.15.28:53388) 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 -- 172.21.15.28:0/1641120920 learned_addr learned my addr 172.21.15.28:0/1641120920 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e38a00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e34158b20 0x7f0e341436e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e33e00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406ba70 0x7f0e340af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 -- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406ba70 msgr2=0x7f0e340af0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406ba70 0x7f0e340af0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 -- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e34158b20 msgr2=0x7f0e341436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e34158b20 0x7f0e341436e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 -- 172.21.15.28:0/1641120920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c009400 con 0x7f0e3406c3f0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.975+0000 7f0e2be00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0098f0 tx=0x7f0e1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.976+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c032440 con 0x7f0e3406c3f0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.976+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1c004230 con 0x7f0e3406c3f0 2024-09-13T17:25:49.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.976+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3414e2c0 con 0x7f0e3406c3f0 2024-09-13T17:25:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.976+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1c006e10 con 0x7f0e3406c3f0 2024-09-13T17:25:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.976+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3414e880 con 0x7f0e3406c3f0 2024-09-13T17:25:49.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.977+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e340b09a0 con 0x7f0e3406c3f0 2024-09-13T17:25:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.978+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f0e1c039050 con 0x7f0e3406c3f0 2024-09-13T17:25:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.979+0000 7f0e30c00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 0x7f0e1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:25:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.979+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f0e1c0a9a20 con 0x7f0e3406c3f0 2024-09-13T17:25:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.979+0000 7f0e33e00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 0x7f0e1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:25:49.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.980+0000 7f0e33e00640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 0x7f0e1006d0c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0e24000f60 tx=0x7f0e24007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:25:49.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:49.983+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f0e1c074e00 con 0x7f0e3406c3f0 2024-09-13T17:25:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.156+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0e340b3660 con 0x7f0e1006ac00 2024-09-13T17:25:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.159+0000 7f0e30c00640 1 -- 172.21.15.28:0/1641120920 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50531 (secure 0 0 0) 0x7f0e24007270 con 0x7f0e1006ac00 2024-09-13T17:25:50.162 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:25:50.162 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:25:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 msgr2=0x7f0e1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 0x7f0e1006d0c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f0e24000f60 tx=0x7f0e24007c00 comp rx=0 tx=0).stop 2024-09-13T17:25:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 msgr2=0x7f0e3414d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:25:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c0098f0 tx=0x7f0e1c0046c0 comp rx=0 tx=0).stop 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 shutdown_connections 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f0e1006ac00 0x7f0e1006d0c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0e34158b20 0x7f0e341436e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0e3406c3f0 0x7f0e3414d320 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 --2- 172.21.15.28:0/1641120920 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f0e3406ba70 0x7f0e340af0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 >> 172.21.15.28:0/1641120920 conn(0x7f0e34072e40 msgr2=0x7f0e340707b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 shutdown_connections 2024-09-13T17:25:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:25:50.161+0000 7f0e39400640 1 -- 172.21.15.28:0/1641120920 wait complete. 2024-09-13T17:25:50.178 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-13T17:25:49.154400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":578240,"kb_used_data":24608,"kb_used_omap":12,"kb_used_meta":553523,"kb_avail":754396480,"statfs":{"total":773094113280,"available":772501995520,"internally_reserved":0,"allocated":25198592,"data_stored":11800682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":566808121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.426623"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":350,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-09-13T17:25:35.713540+0000","last_change":"2024-09-13T17:08:00.482036+0000","last_active":"2024-09-13T17:25:35.713540+0000","last_peered":"2024-09-13T17:25:35.713540+0000","last_clean":"2024-09-13T17:25:35.713540+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:25:35.713540+0000","last_undegraded":"2024-09-13T17:25:35.713540+0000","last_fullsized":"2024-09-13T17:25:35.713540+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T17:07:58.938764+0000","last_clean_scrub_stamp":"2024-09-13T17:07:58.938764+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:24:39.899083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575065,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70660,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301180,"statfs":{"total":96636764160,"available":96564408320,"internally_reserved":0,"allocated":3596288,"data_stored":1918116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33800000000000002},"max":{"1min":0.87,"5min":1.008,"15min":2.0880000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.57599999999999996,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":0.93700000000000006,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.70299999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.91100000000000003,"5min":1.161,"15min":2.5030000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.65000000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":1.399,"15min":6.4909999999999997},"last":0.78800000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71899999999999997,"15min":0.70299999999999996},"min":{"1min":0.58899999999999997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":0.90300000000000002,"5min":1.105,"15min":1.544},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.70599999999999996},"min":{"1min":0.61799999999999999,"5min":0.42399999999999999,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.64400000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.45400000000000001,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.75,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.437},"max":{"1min":0.91200000000000003,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.77000000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":1.2170000000000001,"15min":1.234},"last":0.67800000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.43099999999999999,"5min":0.42699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":1.143},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.57399999999999995,"5min":0.48499999999999999,"15min":0.44800000000000001},"max":{"1min":0.79000000000000004,"5min":1.915,"15min":1.915},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.49299999999999999,"5min":0.41799999999999998,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.607,"15min":1.607},"last":0.71999999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575066,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70084,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301756,"statfs":{"total":96636764160,"available":96564998144,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70999999999999996,"15min":0.67600000000000005},"min":{"1min":0.58399999999999996,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":0.96399999999999997,"5min":0.97099999999999997,"15min":1.298},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.67000000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":5.9800000000000004},"last":0.63100000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.373,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.998,"15min":2.3100000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.57499999999999996,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90700000000000003,"15min":1.46},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.60899999999999999,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.89800000000000002,"5min":1.107,"15min":5.9240000000000004},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71099999999999997,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.872,"5min":0.92700000000000005,"15min":1.012},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73599999999999999,"15min":0.72199999999999998},"min":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":1.151,"15min":3.6520000000000001},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72499999999999998,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92400000000000004,"5min":1.085,"15min":3.6720000000000002},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.33800000000000002},"max":{"1min":0.92600000000000005,"5min":1.262,"15min":1.6439999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.075},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.374},"max":{"1min":1.0209999999999999,"5min":1.202,"15min":1.8260000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.47699999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":0.90200000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.46999999999999997,"5min":0.379,"15min":0.378},"max":{"1min":1.052,"5min":1.163,"15min":1.601},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79400000000000004,"15min":0.78000000000000003},"min":{"1min":0.66600000000000004,"5min":0.58999999999999997,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.5720000000000001},"last":0.77600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48000,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323840,"statfs":{"total":96636764160,"available":96587612160,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.76800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91200000000000003,"5min":1.1220000000000001,"15min":38.005000000000003},"last":0.377},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.41099999999999998,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.745,"5min":0.69299999999999995,"15min":0.70199999999999996},"min":{"1min":0.58299999999999996,"5min":0.44700000000000001,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.169,"15min":7.7370000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.379},"max":{"1min":0.93000000000000005,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98599999999999999,"15min":9.6549999999999994},"last":0.57999999999999996}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.72499999999999998,"15min":1.0820000000000001},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.435},"max":{"1min":0.98099999999999998,"5min":1.042,"15min":113.363},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.61899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":4.1180000000000003},"last":0.501}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":1.0760000000000001,"15min":2.0590000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":2.827},"last":0.40000000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.68600000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.34799999999999998},"max":{"1min":0.85199999999999998,"5min":1.0609999999999999,"15min":2.2320000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.71299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":1.032,"15min":4.1500000000000004},"last":0.43099999999999999}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.746,"15min":0.755},"min":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.40999999999999998},"max":{"1min":0.98199999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.748,"15min":0.75700000000000001},"min":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.44},"max":{"1min":0.95799999999999996,"5min":1.169,"15min":2.02},"last":0.53400000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.26,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.379,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2350000000000001,"15min":0.88400000000000001},"min":{"1min":0.63100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.81699999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.88600000000000001,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.40899999999999997,"15min":0.35099999999999998},"max":{"1min":0.94399999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.627},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.95999999999999996,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.54200000000000004}]},{"osd":2,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72999999999999998,"15min":0.68600000000000005},"min":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.98599999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.76700000000000002}]},{"osd":3,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.1259999999999999,"15min":0.83599999999999997},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.378},"max":{"1min":0.996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.73399999999999999},"min":{"1min":0.499,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.046,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":0.76000000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.70199999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65700000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.32500000000000001},"max":{"1min":0.85499999999999998,"5min":2.976,"15min":2.976},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.75,"15min":0.73199999999999998},"min":{"1min":0.58099999999999996,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79600000000000004,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.92900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.752},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.753,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96599999999999997,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.90700000000000003}]}]},{"osd":3,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.65200000000000002,"5min":0.76400000000000001,"15min":0.97099999999999997},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.32900000000000001},"max":{"1min":0.872,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.621}]},{"osd":2,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":1.4930000000000001,"15min":1.9590000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34499999999999997},"max":{"1min":0.79700000000000004,"5min":0.875,"15min":1.3819999999999999},"last":0.53000000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.748},"min":{"1min":0.63900000000000001,"5min":0.45600000000000002,"15min":0.39700000000000002},"max":{"1min":1.05,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.498},"max":{"1min":0.99399999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.76900000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0389999999999999,"5min":1.228,"15min":1.391},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.76400000000000001},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.442},"max":{"1min":1.022,"5min":2.347,"15min":2.347},"last":0.65300000000000002}]},{"osd":6,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80400000000000005,"15min":0.78000000000000003},"min":{"1min":0.63,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.81000000000000005,"15min":0.79000000000000004},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46300000000000002},"max":{"1min":1.069,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.81399999999999995,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.42699999999999999},"max":{"1min":1.036,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83799999999999997,"15min":0.80800000000000005},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.47599999999999998},"max":{"1min":1.0529999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.70099999999999996}]}]},{"osd":2,"up_from":13,"seq":55834575065,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70076,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":2998272,"data_stored":1326729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":5.6769999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.58699999999999997},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.27100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":30.023},"last":0.34200000000000003}]},{"osd":1,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":1.3720000000000001},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.307},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":260.53399999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":1.4710000000000001},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.94099999999999995,"15min":262.61399999999998},"last":0.38600000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51000000000000001,"15min":0.624},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.84199999999999997,"15min":27.271999999999998},"last":0.501},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.61799999999999999},"min":{"1min":0.52400000000000002,"5min":0.45000000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91100000000000003,"15min":8.4149999999999991},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78200000000000003,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.53000000000000003,"15min":0.45500000000000002},"max":{"1min":1.331,"5min":1.331,"15min":10.598000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.78600000000000003,"15min":0.86599999999999999},"min":{"1min":0.60899999999999999,"5min":0.51200000000000001,"15min":0.371},"max":{"1min":1.1100000000000001,"5min":1.264,"15min":21.213000000000001},"last":0.79300000000000004}]},{"osd":5,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.78400000000000003,"15min":0.88400000000000001},"min":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.40300000000000002},"max":{"1min":1.198,"5min":1.198,"15min":27.135000000000002},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80000000000000004,"15min":0.79100000000000004},"min":{"1min":0.63100000000000001,"5min":0.52400000000000002,"15min":0.47599999999999998},"max":{"1min":1.167,"5min":1.167,"15min":5.8890000000000002},"last":0.81000000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.79900000000000004,"15min":0.83599999999999997},"min":{"1min":0.68000000000000005,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":10.664},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78200000000000003,"15min":0.89100000000000001},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.222,"5min":1.23,"15min":27.236999999999998},"last":0.75900000000000001}]},{"osd":7,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.68100000000000005,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.369,"5min":1.369,"15min":10.712999999999999},"last":0.746},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80300000000000005,"15min":0.91000000000000003},"min":{"1min":0.64400000000000002,"5min":0.52300000000000002,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.1799999999999999,"15min":27.199999999999999},"last":0.72799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575064,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96320,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275520,"statfs":{"total":96636764160,"available":96538132480,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23400000000000001},"max":{"1min":0.61099999999999999,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.47899999999999998,"5min":0.35999999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":6.7240000000000002},"last":0.46700000000000003}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64300000000000002,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.308,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.441}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.65300000000000002},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54200000000000004,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":18.318000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.28,"15min":6.1630000000000003},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.67000000000000004,"5min":0.56299999999999994,"15min":0.42599999999999999},"max":{"1min":1.0129999999999999,"5min":1.23,"15min":1.78},"last":0.68600000000000005}]},{"osd":5,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.0389999999999999,"5min":1.159,"15min":1.6020000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.498,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.69899999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77900000000000003},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.2330000000000001,"15min":1.6599999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.65300000000000002,"5min":0.503,"15min":0.41399999999999998},"max":{"1min":1.079,"5min":1.246,"15min":6.1970000000000001},"last":0.67200000000000004}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.79100000000000004},"min":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.46600000000000003},"max":{"1min":1.0209999999999999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81100000000000005},"min":{"1min":0.64300000000000002,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.097,"5min":1.2609999999999999,"15min":1.544},"last":0.622}]}]},{"osd":0,"up_from":13,"seq":55834575066,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74748,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297092,"statfs":{"total":96636764160,"available":96560222208,"internally_reserved":0,"allocated":3592192,"data_stored":1917097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.433,"15min":0.442},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":0.72599999999999998,"15min":1.986},"last":0.309},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":0.73399999999999999,"5min":0.81599999999999995,"15min":3.004},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":0.877,"15min":3.605},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.45800000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":2.4039999999999999},"last":0.63}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":2.6400000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":3.7709999999999999},"last":0.38500000000000001}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.65400000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.034,"5min":1.1599999999999999,"15min":3.8849999999999998},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.64400000000000002,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.002,"5min":1.2969999999999999,"15min":2.6760000000000002},"last":0.69899999999999995}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.44500000000000001},"max":{"1min":1.1299999999999999,"5min":1.1799999999999999,"15min":2.7320000000000002},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80500000000000005,"15min":0.79100000000000004},"min":{"1min":0.65900000000000003,"5min":0.51800000000000002,"15min":0.432},"max":{"1min":1.006,"5min":1.276,"15min":1.8740000000000001},"last":0.51600000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.70999999999999996,"5min":0.52600000000000002,"15min":0.46700000000000003},"max":{"1min":1.0349999999999999,"5min":1.1619999999999999,"15min":1.242},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.70199999999999996,"5min":0.50600000000000001,"15min":0.48599999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":2.8380000000000001},"last":0.91100000000000003}]},{"osd":7,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.73899999999999999,"5min":0.52100000000000002,"15min":0.47399999999999998},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":3.9209999999999998},"last":1.02},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0760000000000001,"5min":1.242,"15min":2.706},"last":0.79900000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:25:50.182 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T17:07:58.938764+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=17, tm_min=7, tm_sec=58, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=17, tm_min=25, tm_sec=44, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T17:25:50.182 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T17:26:10.183 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T17:26:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.350+0000 7f6057400640 1 Processor -- start 2024-09-13T17:26:10.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.350+0000 7f6057400640 1 -- start start 2024-09-13T17:26:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f6057400640 1 --2- >> v2:172.21.15.189:3300/0 conn(0x7f605015a540 0x7f605006c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f6057400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6050071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f6057400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605015e100 con 0x7f60500cabc0 2024-09-13T17:26:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f6057400640 1 -- --> v2:172.21.15.189:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60500adb00 con 0x7f605015a540 2024-09-13T17:26:10.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f6057400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6050071670 con 0x7f6050070cc0 2024-09-13T17:26:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f604fe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f60500cabc0 0x7f605015e870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54938/0 (socket says 172.21.15.28:54938) 2024-09-13T17:26:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.351+0000 7f604fe00640 1 -- 172.21.15.28:0/1635697628 learned_addr learned my addr 172.21.15.28:0/1635697628 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:26:10.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f6055400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6050071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f6054a00640 1 --2- 172.21.15.28:0/1635697628 >> v2:172.21.15.189:3300/0 conn(0x7f605015a540 0x7f605006c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912414893 0 0) 0x7f605015e100 con 0x7f60500cabc0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 --> v1:172.21.15.28:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6038003520 con 0x7f60500cabc0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.0 v1:172.21.15.28:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3480971040 0 0) 0x7f6038003520 con 0x7f60500cabc0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 msgr2=0x7f6050071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6050071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 >> v2:172.21.15.189:3300/0 conn(0x7f605015a540 msgr2=0x7f605006c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 --2- 172.21.15.28:0/1635697628 >> v2:172.21.15.189:3300/0 conn(0x7f605015a540 0x7f605006c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 --> v1:172.21.15.28:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605015e3e0 con 0x7f60500cabc0 2024-09-13T17:26:10.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.0 v1:172.21.15.28:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2583757678 0 0) 0x7f603c001670 con 0x7f60500cabc0 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 >> v1:172.21.15.28:6789/0 conn(0x7f60500cabc0 legacy=0x7f605015e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6038003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6038003f40 0x7f60380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.352+0000 7f604f400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6038004d00 0x7f6038005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6050071670 con 0x7f6038004d00 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603800b9b0 con 0x7f6038003f40 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603800bb70 con 0x7f6050070cc0 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6038003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f604fe00640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6038003f40 0x7f60380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6055400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6038004d00 0x7f6038005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 -- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6038003f40 msgr2=0x7f60380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6038003f40 0x7f60380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 -- 172.21.15.28:0/1635697628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6038004d00 msgr2=0x7f6038005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6038004d00 0x7f6038005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6054a00640 1 -- 172.21.15.28:0/1635697628 --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038003520 con 0x7f6050070cc0 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.353+0000 7f6055400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6038004d00 0x7f6038005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.354+0000 7f6054a00640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6038003970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6044009ac0 tx=0x7f60380057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=982c0fad88c4b849 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.2 v2:172.21.15.189:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044031400 con 0x7f6050070cc0 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.2 v2:172.21.15.189:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604402d050 con 0x7f6050070cc0 2024-09-13T17:26:10.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f604f400640 1 -- 172.21.15.28:0/1635697628 <== mon.2 v2:172.21.15.189:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6044031780 con 0x7f6050070cc0 2024-09-13T17:26:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 -- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 msgr2=0x7f6038003970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 --2- 172.21.15.28:0/1635697628 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f6050070cc0 0x7f6038003970 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6044009ac0 tx=0x7f60380057f0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6055400640 1 -- 172.21.15.28:0/1635697628 reap_dead start 2024-09-13T17:26:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 -- 172.21.15.28:0/1635697628 shutdown_connections 2024-09-13T17:26:10.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 -- 172.21.15.28:0/1635697628 >> 172.21.15.28:0/1635697628 conn(0x7f6050069e80 msgr2=0x7f605006a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 -- 172.21.15.28:0/1635697628 shutdown_connections 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.355+0000 7f6057400640 1 -- 172.21.15.28:0/1635697628 wait complete. 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.357+0000 7f6057400640 1 Processor -- start 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.357+0000 7f6057400640 1 -- start start 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6050070cc0 0x7f60500b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 0x7f6050157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6050159770 con 0x7f605014abe0 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60500cdfd0 con 0x7f6050070cc0 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6057400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6050154d20 con 0x7f60500cdb50 2024-09-13T17:26:10.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f604fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f604fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46930/0 (socket says 172.21.15.28:46930) 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f604fe00640 1 -- 172.21.15.28:0/1899474573 learned_addr learned my addr 172.21.15.28:0/1899474573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.358+0000 7f6054a00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6050070cc0 0x7f60500b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f6055400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 0x7f6050157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 -- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 msgr2=0x7f6050157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 0x7f6050157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 -- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6050070cc0 msgr2=0x7f60500b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6050070cc0 0x7f60500b03a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 -- 172.21.15.28:0/1899474573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500af300 con 0x7f605014abe0 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f6055400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 0x7f6050157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T17:26:10.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604fe00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f603c007340 tx=0x7f603c007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c001700 con 0x7f605014abe0 2024-09-13T17:26:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.359+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c00f960 con 0x7f605014abe0 2024-09-13T17:26:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.360+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c00fd30 con 0x7f605014abe0 2024-09-13T17:26:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.360+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60500af5e0 con 0x7f605014abe0 2024-09-13T17:26:10.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.360+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60500afba0 con 0x7f605014abe0 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.361+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6050071130 con 0x7f605014abe0 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.362+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f603c01d360 con 0x7f605014abe0 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.362+0000 7f604cc00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 0x7f603006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.363+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f603c0888b0 con 0x7f605014abe0 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.363+0000 7f6054a00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 0x7f603006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.364+0000 7f6054a00640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 0x7f603006d0c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f6044033000 tx=0x7f6044008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.367+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f603c053d40 con 0x7f605014abe0 2024-09-13T17:26:10.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.544+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 --> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f605006d160 con 0x7f603006ac00 2024-09-13T17:26:10.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.547+0000 7f604cc00640 1 -- 172.21.15.28:0/1899474573 <== mgr.4105 v2:172.21.15.189:6832/19110189 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+50560 (secure 0 0 0) 0x7f6044004220 con 0x7f603006ac00 2024-09-13T17:26:10.548 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:26:10.549 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 msgr2=0x7f603006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 0x7f603006d0c0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f6044033000 tx=0x7f6044008030 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 msgr2=0x7f60500cd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f603c007340 tx=0x7f603c007740 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 shutdown_connections 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f603006ac00 0x7f603006d0c0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f60500cdb50 0x7f6050157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f605014abe0 0x7f60500cd610 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 --2- 172.21.15.28:0/1899474573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6050070cc0 0x7f60500b03a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 >> 172.21.15.28:0/1899474573 conn(0x7f6050069e80 msgr2=0x7f605015ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:10.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 shutdown_connections 2024-09-13T17:26:10.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.550+0000 7f6057400640 1 -- 172.21.15.28:0/1899474573 wait complete. 2024-09-13T17:26:10.566 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-13T17:26:09.158774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":578236,"kb_used_data":24608,"kb_used_omap":12,"kb_used_meta":553523,"kb_avail":754396484,"statfs":{"total":773094113280,"available":772501999616,"internally_reserved":0,"allocated":25198592,"data_stored":11800682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":566808121},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"1.0","version":"237'119","reported_seq":365,"reported_epoch":237,"state":"active+clean","last_fresh":"2024-09-13T17:25:52.611034+0000","last_change":"2024-09-13T17:25:52.611034+0000","last_active":"2024-09-13T17:25:52.611034+0000","last_peered":"2024-09-13T17:25:52.611034+0000","last_clean":"2024-09-13T17:25:52.611034+0000","last_became_active":"2024-09-13T17:08:00.480891+0000","last_became_peered":"2024-09-13T17:08:00.480891+0000","last_unstale":"2024-09-13T17:25:52.611034+0000","last_undegraded":"2024-09-13T17:25:52.611034+0000","last_fullsized":"2024-09-13T17:25:52.611034+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'119","last_scrub_stamp":"2024-09-13T17:25:52.610888+0000","last_deep_scrub":"237'119","last_deep_scrub_stamp":"2024-09-13T17:25:52.610888+0000","last_clean_scrub_stamp":"2024-09-13T17:25:52.610888+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:22:33.311322+0000","scrub_duration":0.0058776779999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575069,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70656,"kb_used_data":3512,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301184,"statfs":{"total":96636764160,"available":96564412416,"internally_reserved":0,"allocated":3596288,"data_stored":1918116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":68745675},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.66100000000000003,"15min":0.65900000000000003},"min":{"1min":0.54000000000000004,"5min":0.33900000000000002,"15min":0.33800000000000002},"max":{"1min":0.87,"5min":1.008,"15min":2.0880000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.69899999999999995,"15min":0.68600000000000005},"min":{"1min":0.57599999999999996,"5min":0.41699999999999998,"15min":0.316},"max":{"1min":0.93700000000000006,"5min":1.5169999999999999,"15min":2.6739999999999999},"last":0.77300000000000002}]},{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70299999999999996,"15min":0.68700000000000006},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.33400000000000002},"max":{"1min":0.91100000000000003,"5min":1.161,"15min":2.5030000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.65000000000000002,"5min":0.39900000000000002,"15min":0.38900000000000001},"max":{"1min":0.89600000000000002,"5min":1.399,"15min":6.4909999999999997},"last":0.81899999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.71899999999999997,"15min":0.70299999999999996},"min":{"1min":0.58899999999999997,"5min":0.436,"15min":0.40600000000000003},"max":{"1min":0.90300000000000002,"5min":1.105,"15min":1.544},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71899999999999997,"15min":0.70599999999999996},"min":{"1min":0.61799999999999999,"5min":0.42399999999999999,"15min":0.41799999999999998},"max":{"1min":0.89500000000000002,"5min":1.0269999999999999,"15min":1.605},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69499999999999995,"15min":0.68600000000000005},"min":{"1min":0.53400000000000003,"5min":0.45400000000000001,"15min":0.34999999999999998},"max":{"1min":0.874,"5min":1.6539999999999999,"15min":2.4249999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.51400000000000001,"5min":0.47499999999999998,"15min":0.437},"max":{"1min":0.91200000000000003,"5min":1.2649999999999999,"15min":2.8239999999999998},"last":0.56999999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.59599999999999997,"15min":0.59799999999999998},"min":{"1min":0.39900000000000002,"5min":0.38900000000000001,"15min":0.30499999999999999},"max":{"1min":0.629,"5min":2.4910000000000001,"15min":2.4910000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.628,"15min":0.64700000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34200000000000003},"max":{"1min":0.82099999999999995,"5min":1.2170000000000001,"15min":1.234},"last":0.44700000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.64300000000000002,"15min":0.64500000000000002},"min":{"1min":0.43099999999999999,"5min":0.42699999999999999,"15min":0.35399999999999998},"max":{"1min":0.80400000000000005,"5min":1.135,"15min":1.143},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.746,"15min":0.74299999999999999},"min":{"1min":0.57399999999999995,"5min":0.48499999999999999,"15min":0.44800000000000001},"max":{"1min":0.79000000000000004,"5min":1.915,"15min":1.915},"last":0.76300000000000001}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.70399999999999996,"15min":0.71599999999999997},"min":{"1min":0.49299999999999999,"5min":0.41799999999999998,"15min":0.39000000000000001},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.1930000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.71999999999999997,"15min":0.72199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.44500000000000001},"max":{"1min":0.85899999999999999,"5min":1.607,"15min":1.607},"last":0.58899999999999997}]}]},{"osd":6,"up_from":13,"seq":55834575070,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70084,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301756,"statfs":{"total":96636764160,"available":96564998144,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.70999999999999996,"15min":0.67600000000000005},"min":{"1min":0.58399999999999996,"5min":0.38400000000000001,"15min":0.33900000000000002},"max":{"1min":0.96399999999999997,"5min":0.97099999999999997,"15min":1.298},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.75600000000000001,"15min":0.747},"min":{"1min":0.67000000000000004,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93899999999999995,"5min":0.97899999999999998,"15min":5.9800000000000004},"last":0.70499999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.67000000000000004,"15min":0.65600000000000003},"min":{"1min":0.50600000000000001,"5min":0.373,"15min":0.27800000000000002},"max":{"1min":0.92700000000000005,"5min":0.998,"15min":2.3100000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69499999999999995,"15min":0.68000000000000005},"min":{"1min":0.57499999999999996,"5min":0.45300000000000001,"15min":0.36299999999999999},"max":{"1min":0.90200000000000002,"5min":0.90700000000000003,"15min":1.46},"last":0.56799999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.70999999999999996,"15min":0.69699999999999995},"min":{"1min":0.60899999999999999,"5min":0.372,"15min":0.35599999999999998},"max":{"1min":0.89800000000000002,"5min":1.107,"15min":5.9240000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.71099999999999997,"15min":0.69199999999999995},"min":{"1min":0.55500000000000005,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.872,"5min":0.92700000000000005,"15min":1.012},"last":0.63200000000000001}]},{"osd":3,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.73599999999999999,"15min":0.72199999999999998},"min":{"1min":0.67500000000000004,"5min":0.51900000000000002,"15min":0.41099999999999998},"max":{"1min":0.91300000000000003,"5min":1.151,"15min":3.6520000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.72499999999999998,"15min":0.70199999999999996},"min":{"1min":0.55300000000000005,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.92400000000000004,"5min":1.085,"15min":3.6720000000000002},"last":0.64500000000000002}]},{"osd":4,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63400000000000001,"15min":0.624},"min":{"1min":0.46700000000000003,"5min":0.40100000000000002,"15min":0.33800000000000002},"max":{"1min":0.92600000000000005,"5min":1.262,"15min":1.6439999999999999},"last":0.438},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.628,"15min":0.63},"min":{"1min":0.47099999999999997,"5min":0.38600000000000001,"15min":0.29199999999999998},"max":{"1min":0.98899999999999999,"5min":0.99299999999999999,"15min":1.075},"last":0.46300000000000002}]},{"osd":5,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.68500000000000005,"15min":0.67900000000000005},"min":{"1min":0.51000000000000001,"5min":0.48299999999999998,"15min":0.374},"max":{"1min":1.0209999999999999,"5min":1.202,"15min":1.8260000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65400000000000003,"15min":0.64600000000000002},"min":{"1min":0.47699999999999998,"5min":0.34799999999999998,"15min":0.32700000000000001},"max":{"1min":0.90200000000000002,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.48299999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:32 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.46999999999999997,"5min":0.379,"15min":0.378},"max":{"1min":1.052,"5min":1.163,"15min":1.601},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79400000000000004,"15min":0.78000000000000003},"min":{"1min":0.66600000000000004,"5min":0.58999999999999997,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.5720000000000001},"last":0.73799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48000,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323840,"statfs":{"total":96636764160,"available":96587612160,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.66500000000000004,"15min":0.76800000000000002},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.91200000000000003,"5min":1.1220000000000001,"15min":38.005000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.70499999999999996,"15min":0.74199999999999999},"min":{"1min":0.432,"5min":0.41099999999999998,"15min":0.33400000000000002},"max":{"1min":0.94899999999999995,"5min":0.97199999999999998,"15min":21.376999999999999},"last":0.70299999999999996}]},{"osd":1,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.69599999999999995,"15min":0.72899999999999998},"min":{"1min":0.55900000000000005,"5min":0.41899999999999998,"15min":0.30499999999999999},"max":{"1min":0.96199999999999997,"5min":1.0169999999999999,"15min":17.231000000000002},"last":0.877},{"interface":"front","average":{"1min":0.745,"5min":0.69299999999999995,"15min":0.70199999999999996},"min":{"1min":0.58299999999999996,"5min":0.44700000000000001,"15min":0.41699999999999998},"max":{"1min":0.94699999999999995,"5min":1.169,"15min":7.7370000000000001},"last":0.84499999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.71399999999999997,"15min":0.74299999999999999},"min":{"1min":0.59799999999999998,"5min":0.45600000000000002,"15min":0.379},"max":{"1min":0.93000000000000005,"5min":1.0720000000000001,"15min":17.268000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.69999999999999996,"15min":0.72499999999999998},"min":{"1min":0.57399999999999995,"5min":0.46300000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98599999999999999,"15min":9.6549999999999994},"last":0.82399999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.72499999999999998,"15min":1.0820000000000001},"min":{"1min":0.60399999999999998,"5min":0.45300000000000001,"15min":0.435},"max":{"1min":0.98099999999999998,"5min":1.042,"15min":113.363},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.61899999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.94999999999999996,"5min":0.97599999999999998,"15min":4.1180000000000003},"last":0.93999999999999995}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59999999999999998,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":1.0760000000000001,"15min":2.0590000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.48199999999999998,"5min":0.38800000000000001,"15min":0.34799999999999998},"max":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":2.827},"last":0.56100000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.66400000000000003,"15min":0.68600000000000005},"min":{"1min":0.44600000000000001,"5min":0.42699999999999999,"15min":0.34799999999999998},"max":{"1min":0.85199999999999998,"5min":1.0609999999999999,"15min":2.2320000000000002},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69499999999999995,"15min":0.71299999999999997},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.40200000000000002},"max":{"1min":0.91800000000000004,"5min":1.032,"15min":4.1500000000000004},"last":0.80700000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.746,"15min":0.755},"min":{"1min":0.55800000000000005,"5min":0.51800000000000002,"15min":0.40999999999999998},"max":{"1min":0.98199999999999998,"5min":1.079,"15min":4.1029999999999998},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.748,"15min":0.75700000000000001},"min":{"1min":0.58199999999999996,"5min":0.53700000000000003,"15min":0.44},"max":{"1min":0.95799999999999996,"5min":1.169,"15min":2.02},"last":0.91200000000000003}]}]},{"osd":4,"up_from":13,"seq":55834575068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":72939974},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":1.26,"15min":0.85799999999999998},"min":{"1min":0.57599999999999996,"5min":0.379,"15min":0.32200000000000001},"max":{"1min":1.0049999999999999,"5min":46.860999999999997,"15min":46.860999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.78600000000000003,"5min":1.2350000000000001,"15min":0.88400000000000001},"min":{"1min":0.63100000000000001,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93500000000000005,"5min":45.140000000000001,"15min":45.140000000000001},"last":0.74199999999999999}]},{"osd":1,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.88600000000000001,"15min":0.73299999999999998},"min":{"1min":0.52400000000000002,"5min":0.40899999999999997,"15min":0.35099999999999998},"max":{"1min":0.94399999999999995,"5min":18.071000000000002,"15min":18.071000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.78900000000000003,"15min":0.69799999999999995},"min":{"1min":0.51200000000000001,"5min":0.39300000000000002,"15min":0.35999999999999999},"max":{"1min":0.95999999999999996,"5min":8.3320000000000007,"15min":8.3320000000000007},"last":0.81799999999999995}]},{"osd":2,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.72999999999999998,"15min":0.68600000000000005},"min":{"1min":0.54200000000000004,"5min":0.44900000000000001,"15min":0.34000000000000002},"max":{"1min":0.88,"5min":3.1190000000000002,"15min":3.1190000000000002},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.56000000000000005,"5min":0.44600000000000001,"15min":0.41299999999999998},"max":{"1min":0.98599999999999999,"5min":1.7729999999999999,"15min":2.1259999999999999},"last":0.69499999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":1.1259999999999999,"15min":0.83599999999999997},"min":{"1min":0.50900000000000001,"5min":0.437,"15min":0.378},"max":{"1min":0.996,"5min":36.725999999999999,"15min":36.725999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.79000000000000004,"15min":0.73399999999999999},"min":{"1min":0.499,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.046,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.625}]},{"osd":5,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.68899999999999995,"15min":0.65200000000000002},"min":{"1min":0.47499999999999998,"5min":0.40000000000000002,"15min":0.34300000000000003},"max":{"1min":0.76000000000000001,"5min":7.2610000000000001,"15min":7.2610000000000001},"last":0.502},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.62,"15min":0.64200000000000002},"min":{"1min":0.45700000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70599999999999996,"5min":1.1870000000000001,"15min":1.3240000000000001},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.65700000000000003,"15min":0.67000000000000004},"min":{"1min":0.438,"5min":0.39400000000000002,"15min":0.32500000000000001},"max":{"1min":0.85499999999999998,"5min":2.976,"15min":2.976},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.75,"15min":0.73199999999999998},"min":{"1min":0.58099999999999996,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79600000000000004,"5min":6.4370000000000003,"15min":6.4370000000000003},"last":0.55900000000000005}]},{"osd":7,"last update":"Fri Sep 13 17:25:23 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.70299999999999996,"15min":0.71199999999999997},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.80800000000000005,"5min":2.3780000000000001,"15min":2.3780000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.753,"15min":0.755},"min":{"1min":0.48799999999999999,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.96599999999999997,"5min":2.9129999999999998,"15min":2.9129999999999998},"last":0.77400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":74176,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297664,"statfs":{"total":96636764160,"available":96560807936,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46899999999999997,"15min":0.44800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":0.64200000000000002,"5min":4.3550000000000004,"15min":4.3550000000000004},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.499,"5min":0.47599999999999998,"15min":0.47099999999999997},"min":{"1min":0.34699999999999998,"5min":0.30599999999999999,"15min":0.26200000000000001},"max":{"1min":0.65200000000000002,"5min":0.76400000000000001,"15min":0.97099999999999997},"last":0.57099999999999995}]},{"osd":1,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51700000000000002,"15min":0.499},"min":{"1min":0.374,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.65400000000000003,"5min":3.1779999999999999,"15min":3.1779999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60299999999999998,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.40100000000000002,"15min":0.32900000000000001},"max":{"1min":0.872,"5min":2.9430000000000001,"15min":2.9430000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.38100000000000001,"15min":0.31900000000000001},"max":{"1min":0.61899999999999999,"5min":1.4930000000000001,"15min":1.9590000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56000000000000005,"15min":0.55400000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34499999999999997},"max":{"1min":0.79700000000000004,"5min":0.875,"15min":1.3819999999999999},"last":0.78800000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.77800000000000002,"15min":0.748},"min":{"1min":0.63900000000000001,"5min":0.45600000000000002,"15min":0.39700000000000002},"max":{"1min":1.05,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.78800000000000003,"15min":0.76700000000000002},"min":{"1min":0.54600000000000004,"5min":0.5,"15min":0.498},"max":{"1min":0.99399999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.78200000000000003,"15min":0.76900000000000002},"min":{"1min":0.53600000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0389999999999999,"5min":1.228,"15min":1.391},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.79600000000000004,"15min":0.76400000000000001},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.442},"max":{"1min":1.022,"5min":2.347,"15min":2.347},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.80400000000000005,"15min":0.78000000000000003},"min":{"1min":0.63,"5min":0.52900000000000003,"15min":0.47199999999999998},"max":{"1min":1.0549999999999999,"5min":1.7290000000000001,"15min":1.7290000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.81000000000000005,"15min":0.79000000000000004},"min":{"1min":0.621,"5min":0.57699999999999996,"15min":0.46300000000000002},"max":{"1min":1.069,"5min":1.2949999999999999,"15min":1.3300000000000001},"last":0.95299999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:25:33 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.81399999999999995,"15min":0.79100000000000004},"min":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.42699999999999999},"max":{"1min":1.036,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83799999999999997,"15min":0.80800000000000005},"min":{"1min":0.66300000000000003,"5min":0.55400000000000005,"15min":0.47599999999999998},"max":{"1min":1.0529999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.81799999999999995}]}]},{"osd":2,"up_from":13,"seq":55834575069,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":70076,"kb_used_data":2928,"kb_used_omap":1,"kb_used_meta":67134,"kb_avail":94301764,"statfs":{"total":96636764160,"available":96565006336,"internally_reserved":0,"allocated":2998272,"data_stored":1326729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68745671},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42499999999999999,"15min":0.438},"min":{"1min":0.32500000000000001,"5min":0.27300000000000002,"15min":0.25700000000000001},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":5.6769999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48899999999999999,"15min":0.58699999999999997},"min":{"1min":0.32700000000000001,"5min":0.29799999999999999,"15min":0.27100000000000002},"max":{"1min":1.05,"5min":1.05,"15min":30.023},"last":0.38700000000000001}]},{"osd":1,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.496,"15min":1.3720000000000001},"min":{"1min":0.39100000000000001,"5min":0.34300000000000003,"15min":0.307},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":260.53399999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.51000000000000001,"15min":1.4710000000000001},"min":{"1min":0.38800000000000001,"5min":0.34999999999999998,"15min":0.316},"max":{"1min":0.64700000000000002,"5min":0.94099999999999995,"15min":262.61399999999998},"last":0.443}]},{"osd":3,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.51000000000000001,"15min":0.624},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.29099999999999998},"max":{"1min":0.71399999999999997,"5min":0.84199999999999997,"15min":27.271999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.60199999999999998,"15min":0.61799999999999999},"min":{"1min":0.52400000000000002,"5min":0.45000000000000001,"15min":0.38800000000000001},"max":{"1min":0.77500000000000002,"5min":0.91100000000000003,"15min":8.4149999999999991},"last":0.65800000000000003}]},{"osd":4,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.78200000000000003,"15min":0.81899999999999995},"min":{"1min":0.65500000000000003,"5min":0.53000000000000003,"15min":0.45500000000000002},"max":{"1min":1.331,"5min":1.331,"15min":10.598000000000001},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.78600000000000003,"15min":0.86599999999999999},"min":{"1min":0.60899999999999999,"5min":0.51200000000000001,"15min":0.371},"max":{"1min":1.1100000000000001,"5min":1.264,"15min":21.213000000000001},"last":0.63}]},{"osd":5,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.78400000000000003,"15min":0.88400000000000001},"min":{"1min":0.67300000000000004,"5min":0.52300000000000002,"15min":0.40300000000000002},"max":{"1min":1.198,"5min":1.198,"15min":27.135000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.80000000000000004,"15min":0.79100000000000004},"min":{"1min":0.63100000000000001,"5min":0.52400000000000002,"15min":0.47599999999999998},"max":{"1min":1.167,"5min":1.167,"15min":5.8890000000000002},"last":0.68899999999999995}]},{"osd":6,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.79900000000000004,"15min":0.83599999999999997},"min":{"1min":0.68000000000000005,"5min":0.47699999999999998,"15min":0.44700000000000001},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":10.664},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.78200000000000003,"15min":0.89100000000000001},"min":{"1min":0.59899999999999998,"5min":0.501,"15min":0.47999999999999998},"max":{"1min":1.222,"5min":1.23,"15min":27.236999999999998},"last":0.60499999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88100000000000001,"5min":0.80600000000000005,"15min":0.83599999999999997},"min":{"1min":0.68100000000000005,"5min":0.56499999999999995,"15min":0.52100000000000002},"max":{"1min":1.369,"5min":1.369,"15min":10.712999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.80300000000000005,"15min":0.91000000000000003},"min":{"1min":0.64400000000000002,"5min":0.52300000000000002,"15min":0.501},"max":{"1min":1.0600000000000001,"5min":1.1799999999999999,"15min":27.199999999999999},"last":0.70599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":96320,"kb_used_data":2932,"kb_used_omap":1,"kb_used_meta":93374,"kb_avail":94275520,"statfs":{"total":96636764160,"available":96538132480,"internally_reserved":0,"allocated":3002368,"data_stored":1327748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":95615430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.46000000000000002,"15min":0.46300000000000002},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23400000000000001},"max":{"1min":0.61099999999999999,"5min":0.67900000000000005,"15min":5.4939999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.47899999999999998,"5min":0.35999999999999999,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":6.7240000000000002},"last":0.45900000000000002}]},{"osd":2,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.39200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64300000000000002,"5min":0.81000000000000005,"15min":2.3119999999999998},"last":0.435},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.378,"5min":0.308,"15min":0.308},"max":{"1min":0.68600000000000005,"5min":0.81699999999999995,"15min":1.5089999999999999},"last":0.501}]},{"osd":3,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.61699999999999999,"15min":0.65300000000000002},"min":{"1min":0.51200000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.81000000000000005,"5min":0.82199999999999995,"15min":16.382999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.54200000000000004,"15min":0.59899999999999998},"min":{"1min":0.39600000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":18.318000000000001},"last":0.58999999999999997}]},{"osd":4,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.55300000000000005,"5min":0.47199999999999998,"15min":0.41299999999999998},"max":{"1min":0.93500000000000005,"5min":1.28,"15min":6.1630000000000003},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.81999999999999995,"15min":0.80600000000000005},"min":{"1min":0.67000000000000004,"5min":0.56299999999999994,"15min":0.42599999999999999},"max":{"1min":1.0129999999999999,"5min":1.23,"15min":1.78},"last":0.71599999999999997}]},{"osd":5,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.78300000000000003,"15min":0.77400000000000002},"min":{"1min":0.498,"5min":0.498,"15min":0.48199999999999998},"max":{"1min":1.0389999999999999,"5min":1.159,"15min":1.6020000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.58199999999999996,"5min":0.498,"15min":0.44700000000000001},"max":{"1min":0.99199999999999999,"5min":1.175,"15min":6.1189999999999998},"last":0.84399999999999997}]},{"osd":6,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.78700000000000003,"15min":0.77900000000000003},"min":{"1min":0.69499999999999995,"5min":0.51700000000000002,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.2330000000000001,"15min":1.6599999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.80200000000000005,"15min":0.80600000000000005},"min":{"1min":0.65300000000000002,"5min":0.503,"15min":0.41399999999999998},"max":{"1min":1.079,"5min":1.246,"15min":6.1970000000000001},"last":0.57499999999999996}]},{"osd":7,"last update":"Fri Sep 13 17:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.80700000000000005,"15min":0.79100000000000004},"min":{"1min":0.59299999999999997,"5min":0.58799999999999997,"15min":0.46600000000000003},"max":{"1min":1.0209999999999999,"5min":1.1359999999999999,"15min":1.8320000000000001},"last":0.755},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.81999999999999995,"15min":0.81100000000000005},"min":{"1min":0.64300000000000002,"5min":0.54300000000000004,"15min":0.53800000000000003},"max":{"1min":1.097,"5min":1.2609999999999999,"15min":1.544},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575070,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74748,"kb_used_data":3508,"kb_used_omap":1,"kb_used_meta":71230,"kb_avail":94297092,"statfs":{"total":96636764160,"available":96560222208,"internally_reserved":0,"allocated":3592192,"data_stored":1917097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":72939975},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.433,"15min":0.442},"min":{"1min":0.308,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":0.67100000000000004,"5min":0.72599999999999998,"15min":1.986},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.501,"15min":0.50600000000000001},"min":{"1min":0.33200000000000002,"5min":0.28000000000000003,"15min":0.27000000000000002},"max":{"1min":0.73399999999999999,"5min":0.81599999999999995,"15min":3.004},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.502,"15min":0.51700000000000002},"min":{"1min":0.35899999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.877,"5min":0.877,"15min":3.605},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.53500000000000003,"15min":0.52800000000000002},"min":{"1min":0.45800000000000002,"5min":0.34200000000000003,"15min":0.33000000000000002},"max":{"1min":0.75700000000000001,"5min":0.76200000000000001,"15min":2.4039999999999999},"last":0.69699999999999995}]},{"osd":3,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.315,"15min":0.314},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":2.6400000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.57899999999999996,"15min":0.57799999999999996},"min":{"1min":0.433,"5min":0.39000000000000001,"15min":0.34999999999999998},"max":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":3.7709999999999999},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78400000000000003},"min":{"1min":0.65400000000000003,"5min":0.53100000000000003,"15min":0.443},"max":{"1min":1.034,"5min":1.1599999999999999,"15min":3.8849999999999998},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.77400000000000002,"15min":0.76400000000000001},"min":{"1min":0.64400000000000002,"5min":0.46999999999999997,"15min":0.39300000000000002},"max":{"1min":1.002,"5min":1.2969999999999999,"15min":2.6760000000000002},"last":0.90700000000000003}]},{"osd":5,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.80400000000000005,"15min":0.77800000000000002},"min":{"1min":0.59699999999999998,"5min":0.503,"15min":0.44500000000000001},"max":{"1min":1.1299999999999999,"5min":1.1799999999999999,"15min":2.7320000000000002},"last":0.879},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.80500000000000005,"15min":0.79100000000000004},"min":{"1min":0.65900000000000003,"5min":0.51800000000000002,"15min":0.432},"max":{"1min":1.006,"5min":1.276,"15min":1.8740000000000001},"last":1.022}]},{"osd":6,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.81299999999999994,"15min":0.78900000000000003},"min":{"1min":0.70999999999999996,"5min":0.52600000000000002,"15min":0.46700000000000003},"max":{"1min":1.0349999999999999,"5min":1.1619999999999999,"15min":1.242},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.871,"5min":0.80900000000000005,"15min":0.80200000000000005},"min":{"1min":0.70199999999999996,"5min":0.50600000000000001,"15min":0.48599999999999999},"max":{"1min":1.0940000000000001,"5min":1.274,"15min":2.8380000000000001},"last":0.84899999999999998}]},{"osd":7,"last update":"Fri Sep 13 17:25:27 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.83099999999999996,"15min":0.82299999999999995},"min":{"1min":0.73899999999999999,"5min":0.52100000000000002,"15min":0.47399999999999998},"max":{"1min":1.093,"5min":1.3300000000000001,"15min":3.9209999999999998},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.81899999999999995,"15min":0.80700000000000005},"min":{"1min":0.67700000000000005,"5min":0.56699999999999995,"15min":0.505},"max":{"1min":1.0760000000000001,"5min":1.242,"15min":2.706},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T17:26:10.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T17:26:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.718+0000 7feeb3200640 1 Processor -- start 2024-09-13T17:26:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.718+0000 7feeb3200640 1 -- start start 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.719+0000 7feeb3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.719+0000 7feeb3200640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7feeac06f3f0 0x7feeac15c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.719+0000 7feeb3200640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac15ab60 con 0x7feeac159770 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.719+0000 7feeb3200640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac06f920 con 0x7feeac06f3f0 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.719+0000 7feeb3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15b290 con 0x7feeac070430 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeabe00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7feeac159770 0x7feeac06fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54944/0 (socket says 172.21.15.28:54944) 2024-09-13T17:26:10.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeabe00640 1 -- 172.21.15.28:0/1240316554 learned_addr learned my addr 172.21.15.28:0/1240316554 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeb1200640 1 --2- 172.21.15.28:0/1240316554 >> v2:172.21.15.28:3300/0 conn(0x7feeac06f3f0 0x7feeac15c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feea3400640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeb1200640 1 -- 172.21.15.28:0/1240316554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 msgr2=0x7feeac15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeb1200640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeab400640 1 -- 172.21.15.28:0/1240316554 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101341228 0 0) 0x7fee9c001110 con 0x7feeac159770 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeb1200640 1 -- 172.21.15.28:0/1240316554 >> v1:172.21.15.28:6789/0 conn(0x7feeac159770 legacy=0x7feeac06fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.720+0000 7feeb1200640 1 -- 172.21.15.28:0/1240316554 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac15b570 con 0x7feeac06f3f0 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.721+0000 7feeb1200640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac15c2b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7feea4009d90 tx=0x7feea402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c98a8881a5e91927 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeab400640 1 -- 172.21.15.28:0/1240316554 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4031600 con 0x7feeac06f3f0 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeab400640 1 -- 172.21.15.28:0/1240316554 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea402d050 con 0x7feeac06f3f0 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeab400640 1 -- 172.21.15.28:0/1240316554 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4031980 con 0x7feeac06f3f0 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 -- 172.21.15.28:0/1240316554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 msgr2=0x7feeac15c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac15c2b0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7feea4009d90 tx=0x7feea402a5d0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 -- 172.21.15.28:0/1240316554 shutdown_connections 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac15c2b0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 --2- 172.21.15.28:0/1240316554 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 -- 172.21.15.28:0/1240316554 >> 172.21.15.28:0/1240316554 conn(0x7feeac0b0270 msgr2=0x7feeac0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 -- 172.21.15.28:0/1240316554 shutdown_connections 2024-09-13T17:26:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.722+0000 7feeb3200640 1 -- 172.21.15.28:0/1240316554 wait complete. 2024-09-13T17:26:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.724+0000 7feeb3200640 1 Processor -- start 2024-09-13T17:26:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.724+0000 7feeb3200640 1 -- start start 2024-09-13T17:26:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac159770 0x7feeac156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac14d810 con 0x7feeac06f3f0 2024-09-13T17:26:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1f0400 con 0x7feeac159770 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb3200640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac157700 con 0x7feeac070430 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46952/0 (socket says 172.21.15.28:46952) 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeabe00640 1 -- 172.21.15.28:0/2356529985 learned_addr learned my addr 172.21.15.28:0/2356529985 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feea3400640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.725+0000 7feeb1200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac159770 0x7feeac156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 -- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 msgr2=0x7feeac1547f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac1547f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 -- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac159770 msgr2=0x7feeac156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac159770 0x7feeac156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 -- 172.21.15.28:0/2356529985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea4009a70 con 0x7feeac06f3f0 2024-09-13T17:26:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeabe00640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00c430 tx=0x7fee9c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0074e0 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c007660 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c004a20 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac1579e0 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.726+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac14dd40 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.727+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac06fd20 con 0x7feeac06f3f0 2024-09-13T17:26:10.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.728+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7fee9c004b80 con 0x7feeac06f3f0 2024-09-13T17:26:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.729+0000 7feea8c00640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 0x7fee8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.729+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7fee9c088c00 con 0x7feeac06f3f0 2024-09-13T17:26:10.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.730+0000 7feea3400640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 0x7fee8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:10.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.730+0000 7feea3400640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 0x7fee8806d0c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7feeac1566a0 tx=0x7fee94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:10.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.733+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7fee9c054120 con 0x7feeac06f3f0 2024-09-13T17:26:10.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.916+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7feeac15ab60 con 0x7feeac06f3f0 2024-09-13T17:26:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.920+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fee9c001170 con 0x7feeac06f3f0 2024-09-13T17:26:10.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.920+0000 7feea8c00640 1 -- 172.21.15.28:0/2356529985 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v56)=0 v56) v1 ==== 123+0+0 (secure 0 0 0) 0x7feeac15ab60 con 0x7feeac06f3f0 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 msgr2=0x7fee8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 0x7fee8806d0c0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7feeac1566a0 tx=0x7fee94008040 comp rx=0 tx=0).stop 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 msgr2=0x7feeac1e9fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7fee9c00c430 tx=0x7fee9c00c8f0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 shutdown_connections 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7fee8806ac00 0x7fee8806d0c0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feeac159770 0x7feeac156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7feeac070430 0x7feeac1547f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 --2- 172.21.15.28:0/2356529985 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feeac06f3f0 0x7feeac1e9fb0 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 >> 172.21.15.28:0/2356529985 conn(0x7feeac0b0270 msgr2=0x7feeac0cd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 shutdown_connections 2024-09-13T17:26:10.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:10.924+0000 7feeb3200640 1 -- 172.21.15.28:0/2356529985 wait complete. 2024-09-13T17:26:10.941 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T17:26:10.941 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T17:26:11.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.096+0000 7f2f93400640 1 Processor -- start 2024-09-13T17:26:11.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.096+0000 7f2f93400640 1 -- start start 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f93400640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f8c1592a0 0x7f2f8c15ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f93400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f93400640 1 -- --> v1:172.21.15.28:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8c06cc70 con 0x7f2f8c06f970 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f93400640 1 -- --> v2:172.21.15.28:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8c15b0b0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f93400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c159d90 con 0x7f2f8c06c2d0 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f8be00640 1 --1- >> v1:172.21.15.28:6789/0 conn(0x7f2f8c06f970 0x7f2f8c159680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.28:6789/0 says I am v1:172.21.15.28:54946/0 (socket says 172.21.15.28:54946) 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f90a00640 1 --2- >> v2:172.21.15.28:3300/0 conn(0x7f2f8c1592a0 0x7f2f8c15ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.097+0000 7f2f8be00640 1 -- 172.21.15.28:0/4132557284 learned_addr learned my addr 172.21.15.28:0/4132557284 (peer_addr_for_me v1:172.21.15.28:0/0) 2024-09-13T17:26:11.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f91400640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f90a00640 1 -- 172.21.15.28:0/4132557284 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 msgr2=0x7f2f8c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T17:26:11.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f90a00640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f90a00640 1 -- 172.21.15.28:0/4132557284 >> v1:172.21.15.28:6789/0 conn(0x7f2f8c06f970 legacy=0x7f2f8c159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f90a00640 1 -- 172.21.15.28:0/4132557284 --> v2:172.21.15.28:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f8c0b2040 con 0x7f2f8c1592a0 2024-09-13T17:26:11.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.098+0000 7f2f8b400640 1 -- 172.21.15.28:0/4132557284 <== mon.0 v1:172.21.15.28:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326629225 0 0) 0x7f2f8c06cc70 con 0x7f2f8c06f970 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.099+0000 7f2f90a00640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c15ab70 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f2f800097a0 tx=0x7f2f8002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea50fe382d83e6ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.099+0000 7f2f8b400640 1 -- 172.21.15.28:0/4132557284 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80032440 con 0x7f2f8c1592a0 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.099+0000 7f2f8b400640 1 -- 172.21.15.28:0/4132557284 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2f800325a0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.099+0000 7f2f8b400640 1 -- 172.21.15.28:0/4132557284 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f80032950 con 0x7f2f8c1592a0 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 -- 172.21.15.28:0/4132557284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 msgr2=0x7f2f8c15ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c15ab70 secure :-1 s=READY pgs=2930 cs=0 l=1 rev1=1 crypto rx=0x7f2f800097a0 tx=0x7f2f8002a180 comp rx=0 tx=0).stop 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 -- 172.21.15.28:0/4132557284 shutdown_connections 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 --2- 172.21.15.28:0/4132557284 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c15ab70 unknown :-1 s=CLOSED pgs=2930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 -- 172.21.15.28:0/4132557284 >> 172.21.15.28:0/4132557284 conn(0x7f2f8c069e80 msgr2=0x7f2f8c06a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 -- 172.21.15.28:0/4132557284 shutdown_connections 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.100+0000 7f2f93400640 1 -- 172.21.15.28:0/4132557284 wait complete. 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.102+0000 7f2f93400640 1 Processor -- start 2024-09-13T17:26:11.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.102+0000 7f2f93400640 1 -- start start 2024-09-13T17:26:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.102+0000 7f2f93400640 1 --2- >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.102+0000 7f2f93400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 0x7f2f8c14f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f93400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c14de60 con 0x7f2f8c1592a0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f93400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c14d8e0 con 0x7f2f8c06f970 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f93400640 1 -- --> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c14a9c0 con 0x7f2f8c06c2d0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46962/0 (socket says 172.21.15.28:46962) 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 -- 172.21.15.28:0/2193458814 learned_addr learned my addr 172.21.15.28:0/2193458814 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f90a00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 -- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 msgr2=0x7f2f8c154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f8be00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 0x7f2f8c14f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 -- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 msgr2=0x7f2f8c14f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 0x7f2f8c14f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f91400640 1 -- 172.21.15.28:0/2193458814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f80009400 con 0x7f2f8c1592a0 2024-09-13T17:26:11.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f90a00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:26:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.103+0000 7f2f8be00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 0x7f2f8c14f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T17:26:11.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f91400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007e40 tx=0x7f2f8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84014620 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2f8400ee90 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84014a00 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c14db80 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.104+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c14b190 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.105+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c14a6e0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.106+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) v1 ==== 89145+0+0 (secure 0 0 0) 0x7f2f84014b60 con 0x7f2f8c1592a0 2024-09-13T17:26:11.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.107+0000 7f2f88c00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 0x7f2f6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T17:26:11.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.107+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 5029+0+0 (secure 0 0 0) 0x7f2f84089b20 con 0x7f2f8c1592a0 2024-09-13T17:26:11.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.107+0000 7f2f90a00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 0x7f2f6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T17:26:11.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.108+0000 7f2f90a00640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 0x7f2f6c06d0c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2f80034000 tx=0x7f2f8002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T17:26:11.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.111+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182695 (secure 0 0 0) 0x7f2f8c14a6e0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.324+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f8c14a6e0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.325+0000 7f2f88c00640 1 -- 172.21.15.28:0/2193458814 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+3538 (secure 0 0 0) 0x7f2f8c14a6e0 con 0x7f2f8c1592a0 2024-09-13T17:26:11.326 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-09-13T17:26:11.326 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":197,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":11272,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6836/59419557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":59419557},{"type":"v1","addr":"172.21.15.189:6836","nonce":59419557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":11275,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/2287710859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6835","nonce":2287710859},{"type":"v1","addr":"172.21.15.189:6837","nonce":2287710859}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12315,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6834/3631927863","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":3631927863},{"type":"v1","addr":"172.21.15.28:6834","nonce":3631927863}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":184},{"gid":12801,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/3311834202","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6833","nonce":3311834202},{"type":"v1","addr":"172.21.15.28:6835","nonce":3311834202}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":197}],"filesystems":[]} 2024-09-13T17:26:11.326 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 197 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 msgr2=0x7f2f6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 0x7f2f6c06d0c0 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f2f80034000 tx=0x7f2f8002e000 comp rx=0 tx=0).stop 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 msgr2=0x7f2f8c14f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f2f84007e40 tx=0x7f2f8400e4b0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 shutdown_connections 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:6832/19110189,v1:172.21.15.189:6833/19110189] conn(0x7f2f6c06ac00 0x7f2f6c06d0c0 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f8c1592a0 0x7f2f8c14f730 unknown :-1 s=CLOSED pgs=2931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f8c06f970 0x7f2f8c14f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 --2- 172.21.15.28:0/2193458814 >> [v2:172.21.15.189:3301/0,v1:172.21.15.189:6790/0] conn(0x7f2f8c06c2d0 0x7f2f8c154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T17:26:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 >> 172.21.15.28:0/2193458814 conn(0x7f2f8c069e80 msgr2=0x7f2f8c06fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T17:26:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 shutdown_connections 2024-09-13T17:26:11.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-09-13T17:26:11.327+0000 7f2f93400640 1 -- 172.21.15.28:0/2193458814 wait complete. 2024-09-13T17:26:11.343 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-13T17:26:11.343 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T17:26:11.343 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T17:26:11.344 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.358 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T17:26:11.358 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T17:26:11.358 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.411 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T17:26:11.412 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T17:26:11.412 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.429 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T17:26:11.429 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T17:26:11.429 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.486 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T17:26:11.486 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T17:26:11.487 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T17:26:11.487 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.544 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T17:26:11.545 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T17:26:11.545 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.602 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T17:26:11.602 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T17:26:11.602 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.670 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T17:26:11.671 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T17:26:11.671 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.740 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T17:26:11.740 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T17:26:11.740 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.806 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T17:26:11.806 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T17:26:11.806 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.874 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T17:26:11.874 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T17:26:11.874 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.937 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T17:26:11.937 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T17:26:11.938 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:11.999 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T17:26:12.000 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T17:26:12.000 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T17:26:12.000 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:12.063 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T17:26:12.063 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T17:26:12.063 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:12.134 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T17:26:12.134 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T17:26:12.134 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T17:26:12.134 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:12.156 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T17:26:12.157 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T17:26:12.157 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:12.175 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T17:26:12.176 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T17:26:12.176 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T17:26:12.239 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T17:26:12.239 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T17:26:12.239 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'Replacing daemon mds' | egrep -v 'Scrub error on inode' | egrep -v 'Behind on trimming' | egrep -v 'Metadata damage detected' | egrep -v 'bad backtrace on inode' | egrep -v 'overall HEALTH_' | egrep -v '\(MDS_TRIM\)' | head -n 1 2024-09-13T17:26:12.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:26:12.329 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T17:26:12.573 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:12.577 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:12.590 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:26:12.590 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T17:26:12.838 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:26:12.838 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T17:26:12.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:26:12.977 DEBUG:teuthology.orchestra.run.smithi028:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T17:26:13.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:26:13.119 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T17:26:13.357 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:26:13.357 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T17:26:13.496 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:26:13.496 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T17:26:13.624 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:26:13.624 DEBUG:teuthology.orchestra.run.smithi189:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T17:26:13.791 INFO:tasks.ceph:Archiving mon data... 2024-09-13T17:26:14.006 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/data/mon.a.tgz 2024-09-13T17:26:14.006 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-09-13T17:26:14.045 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.p5otqTUJIf 2024-09-13T17:26:14.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.p5otqTUJIf 2024-09-13T17:26:14.255 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.p5otqTUJIf 2024-09-13T17:26:14.350 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.p5otqTUJIf is 487KB 2024-09-13T17:26:14.441 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.p5otqTUJIf 2024-09-13T17:26:14.476 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/data/mon.b.tgz 2024-09-13T17:26:14.476 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-13T17:26:14.513 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.uvGHam3qEE 2024-09-13T17:26:14.513 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uvGHam3qEE 2024-09-13T17:26:14.718 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.uvGHam3qEE 2024-09-13T17:26:14.812 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.uvGHam3qEE is 492KB 2024-09-13T17:26:14.901 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.uvGHam3qEE 2024-09-13T17:26:14.936 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/data/mon.c.tgz 2024-09-13T17:26:14.936 DEBUG:teuthology.orchestra.run.smithi189:> mktemp 2024-09-13T17:26:15.011 INFO:teuthology.orchestra.run.smithi189.stdout:/tmp/tmp.9a9GCHb7Dx 2024-09-13T17:26:15.011 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.9a9GCHb7Dx 2024-09-13T17:26:15.199 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 0666 /tmp/tmp.9a9GCHb7Dx 2024-09-13T17:26:15.292 DEBUG:teuthology.orchestra.remote:smithi189:/tmp/tmp.9a9GCHb7Dx is 492KB 2024-09-13T17:26:15.383 DEBUG:teuthology.orchestra.run.smithi189:> rm -fr /tmp/tmp.9a9GCHb7Dx 2024-09-13T17:26:15.417 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T17:26:15.418 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T17:26:15.422 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T17:26:15.513 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T17:26:15.513 DEBUG:teuthology.orchestra.run.smithi028:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T17:26:15.518 DEBUG:teuthology.orchestra.run.smithi189:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T17:26:15.580 INFO:teuthology.orchestra.run.smithi189.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T17:26:15.580 INFO:teuthology.orchestra.run.smithi028.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T17:26:15.591 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T17:26:15.592 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi028/crash 2024-09-13T17:26:15.593 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T17:26:15.649 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi189/crash 2024-09-13T17:26:15.650 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T17:26:15.704 INFO:tasks.ceph:Shutting down logrotate 2024-09-13T17:26:15.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:15.708 DEBUG:teuthology.orchestra.run.smithi189:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:15.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:15.858 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T17:26:15.913 INFO:tasks.ceph:Compressing logs... 2024-09-13T17:26:15.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T17:26:15.917 DEBUG:teuthology.orchestra.run.smithi189:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T17:27:35.799 INFO:tasks.ceph:Archiving logs... 2024-09-13T17:27:35.800 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi028/log 2024-09-13T17:27:35.801 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T17:27:44.489 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi189/log 2024-09-13T17:27:44.490 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T17:27:46.144 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T17:27:46.493 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T17:27:46.494 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T17:27:46.498 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T17:27:46.626 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T17:27:46.696 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T17:27:46.697 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T17:27:46.702 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T17:27:46.733 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-09-13T17:27:46.738 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-09-13T17:27:46.740 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T17:27:46.741 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-09-13T17:27:46.741 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv01.front.sepia.ceph.com 3 7 377 105 -160us[ -154us] +/- 38ms 2024-09-13T17:27:46.741 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 41 +1051us[+1051us] +/- 73ms 2024-09-13T17:27:46.741 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 41 -724us[ -724us] +/- 71ms 2024-09-13T17:27:46.741 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 -199us[ -190us] +/- 39ms 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 40 +1012us[+1021us] +/- 73ms 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -770us[ -760us] +/- 71ms 2024-09-13T17:27:46.745 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-13T17:27:46.746 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T17:27:46.755 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T17:27:46.755 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T17:27:46.764 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T17:27:46.942 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-13T17:27:47.061 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T17:27:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:27:47.125 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-13T17:27:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:27:47.186 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T17:27:47.200 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T17:27:47.209 INFO:teuthology.task.internal:Duration was 1623.088657 seconds 2024-09-13T17:27:47.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T17:27:47.218 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T17:27:47.219 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T17:27:47.223 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T17:27:47.307 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T17:27:47.310 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-13T17:27:47.609 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T17:27:47.610 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-09-13T17:27:47.610 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T17:27:47.659 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-09-13T17:27:47.659 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T17:27:47.711 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T17:27:47.711 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T17:27:47.715 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T17:27:47.774 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T17:27:47.774 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T17:27:47.778 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T17:27:47.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T17:27:47.982 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T17:27:47.983 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T17:27:48.039 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T17:27:48.095 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T17:27:48.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T17:27:48.110 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T17:27:48.177 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-09-13T17:27:48.180 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-09-13T17:27:48.210 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T17:27:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:27:48.244 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T17:27:48.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T17:27:48.277 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T17:27:48.288 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T17:27:48.289 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi028 2024-09-13T17:27:48.289 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T17:27:48.356 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:42:16-fs-wip-rishabh-testing-20240910.100730-reef-distro-default-smithi/7903869/remote/smithi189 2024-09-13T17:27:48.356 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T17:27:48.420 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T17:27:48.421 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T17:27:48.429 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T17:27:48.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T17:27:48.475 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T17:27:48.475 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T17:27:48.529 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T17:27:48.529 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T17:27:48.533 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T17:27:48.564 INFO:teuthology.orchestra.run.smithi028.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 17:27 /home/ubuntu/cephtest 2024-09-13T17:27:48.567 INFO:teuthology.orchestra.run.smithi189.stdout: 393233 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 17:27 /home/ubuntu/cephtest 2024-09-13T17:27:48.570 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T17:27:48.580 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T17:27:48.601 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a client.1-kernel-sha1: 2ca44a2f35ca8f36983aaaa46d49312239ef519a description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/scrub} duration: 1623.0886571407318 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-09-13T17:27:48.602 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T17:27:48.694 INFO:teuthology.run:pass